2023-10-30T13:56:53.288 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-10-30T13:56:53.289 DEBUG:teuthology.run:Teuthology command: teuthology --name leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi --archive /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441053 --verbose --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} --owner scheduled_leonidus@teuthology -- /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441053/orig.config.yaml 2023-10-30T13:56:53.339 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T13:56:53.417 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441053 branch: wip-lusov-balance-after-stop description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: null first_in_suite: false job_id: '7441053' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-lusov-balance-after-stop ceph: cephfs: max_mds: 3 session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: false mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - Replacing daemon mds sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 kclient: syntax: v1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-balance-after-stop sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 owner: scheduled_leonidus@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 8397 sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 sleep_before_teardown: 0 subset: 464/1024 suite: fs suite_branch: wip-lusov-balance-after-stop suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi057.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIMwmje95SbNyfN/QhIqW+xTGxgbFqqhqVuHX1ojoZwDT7O/oIBYa4ThMFPh2Waxbow/mVMbY8cM9aUazaL1ENE= smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPPr4j6c4Elj4JMggLxzU+oXj+Zbox5W/SOrDnjdPhsyNdfo/+FImaEbo+jin9aqDlqe5Rnspl/1O6/Th177wnE= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-10-30_06:34:16 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-10-30T13:56:53.417 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa; will attempt to use it 2023-10-30T13:56:53.418 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks 2023-10-30T13:56:53.418 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-30T13:56:53.420 INFO:teuthology.task.internal:Checking packages... 2023-10-30T13:56:53.441 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '00c1c13cea21a686b046584596fbeac5315a7d71' 2023-10-30T13:56:53.442 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-30T13:56:53.442 INFO:teuthology.packaging:ref: None 2023-10-30T13:56:53.442 INFO:teuthology.packaging:tag: None 2023-10-30T13:56:53.443 INFO:teuthology.packaging:branch: wip-lusov-balance-after-stop 2023-10-30T13:56:53.443 INFO:teuthology.packaging:sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:56:53.443 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-lusov-balance-after-stop 2023-10-30T13:56:53.593 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:56:53.594 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-30T13:56:53.600 INFO:teuthology.task.internal:no buildpackages task found 2023-10-30T13:56:53.600 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-30T13:56:53.607 INFO:teuthology.task.internal:Saving configuration 2023-10-30T13:56:53.624 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-30T13:56:53.630 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T13:56:53.655 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441053', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 13:50:53.741350', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM0txIcit+ASzNXsPWGVW6YveoIisW0DLjvEwkzCKIr4'} 2023-10-30T13:56:53.675 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441053', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 13:50:53.739374', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA//zu1amkK2rU/PGIUZGSZDpTrjqCk9+aK4kusoQqv+'} 2023-10-30T13:56:53.675 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-30T13:56:53.681 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-10-30T13:56:53.681 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-10-30T13:56:53.681 INFO:teuthology.run_tasks:Running task console_log... 2023-10-30T13:56:53.751 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd7230b6c10>, signals=[15]) 2023-10-30T13:56:53.752 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-30T13:56:53.758 INFO:teuthology.task.internal:Opening connections... 2023-10-30T13:56:53.759 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi057.front.sepia.ceph.com 2023-10-30T13:56:53.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T13:56:53.836 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2023-10-30T13:56:53.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T13:56:53.912 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-30T13:56:53.918 DEBUG:teuthology.orchestra.run.smithi057:> uname -m 2023-10-30T13:56:53.924 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64 2023-10-30T13:56:53.924 DEBUG:teuthology.orchestra.run.smithi057:> cat /etc/os-release 2023-10-30T13:56:53.972 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-30T13:56:53.972 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="Ubuntu" 2023-10-30T13:56:53.972 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="22.04" 2023-10-30T13:56:53.972 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-30T13:56:53.973 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_CODENAME=jammy 2023-10-30T13:56:53.973 INFO:teuthology.orchestra.run.smithi057.stdout:ID=ubuntu 2023-10-30T13:56:53.973 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE=debian 2023-10-30T13:56:53.973 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-30T13:56:53.973 INFO:teuthology.orchestra.run.smithi057.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-30T13:56:53.973 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-30T13:56:53.974 INFO:teuthology.orchestra.run.smithi057.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-30T13:56:53.974 INFO:teuthology.orchestra.run.smithi057.stdout:UBUNTU_CODENAME=jammy 2023-10-30T13:56:53.974 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server 2023-10-30T13:56:53.998 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2023-10-30T13:56:54.003 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2023-10-30T13:56:54.004 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2023-10-30T13:56:54.049 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-30T13:56:54.050 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="Ubuntu" 2023-10-30T13:56:54.050 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="22.04" 2023-10-30T13:56:54.050 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-30T13:56:54.050 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_CODENAME=jammy 2023-10-30T13:56:54.050 INFO:teuthology.orchestra.run.smithi088.stdout:ID=ubuntu 2023-10-30T13:56:54.050 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE=debian 2023-10-30T13:56:54.051 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-30T13:56:54.051 INFO:teuthology.orchestra.run.smithi088.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-30T13:56:54.051 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-30T13:56:54.051 INFO:teuthology.orchestra.run.smithi088.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-30T13:56:54.051 INFO:teuthology.orchestra.run.smithi088.stdout:UBUNTU_CODENAME=jammy 2023-10-30T13:56:54.052 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2023-10-30T13:56:54.073 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-30T13:56:54.082 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-30T13:56:54.088 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-30T13:56:54.088 DEBUG:teuthology.orchestra.run.smithi057:> test '!' -e /home/ubuntu/cephtest 2023-10-30T13:56:54.091 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2023-10-30T13:56:54.095 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-30T13:56:54.101 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-30T13:56:54.102 DEBUG:teuthology.orchestra.run.smithi057:> test -z $(ls -A /var/lib/ceph) 2023-10-30T13:56:54.137 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2023-10-30T13:56:54.153 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-30T13:56:54.198 INFO:teuthology.run_tasks:Running task kernel... 2023-10-30T13:56:54.212 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2023-10-30T13:56:54.213 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2023-10-30T13:56:54.213 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2023-10-30T13:56:54.214 DEBUG:teuthology.orchestra.run.smithi057:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T13:56:54.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:56:54.219 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing 2023-10-30T13:56:54.385 DEBUG:teuthology.task.kernel:sha1 for client.0 is eacf00cc51e204f401b4d5d4f84825a68f373c74 2023-10-30T13:56:54.386 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "eacf00cc51e204f401b4d5d4f84825a68f373c74"... 2023-10-30T13:56:54.386 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2023-10-30T13:56:54.392 INFO:teuthology.orchestra.run.smithi057.stdout:5.15.0-56-generic 2023-10-30T13:56:54.392 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs eacf00cc51e204f401b4d5d4f84825a68f373c74 2023-10-30T13:56:54.393 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-10-30T13:56:54.393 INFO:teuthology.task.kernel:Installing linux-firmware on client.0... 2023-10-30T13:56:54.393 DEBUG:teuthology.orchestra.run.smithi057:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2023-10-30T13:56:54.444 INFO:teuthology.orchestra.run.smithi057.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2023-10-30T13:56:54.585 INFO:teuthology.orchestra.run.smithi057.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2023-10-30T13:56:54.585 INFO:teuthology.orchestra.run.smithi057.stderr:hint: is subject to change. To configure the initial branch name to use in all 2023-10-30T13:56:54.585 INFO:teuthology.orchestra.run.smithi057.stderr:hint: of your new repositories, which will suppress this warning, call: 2023-10-30T13:56:54.585 INFO:teuthology.orchestra.run.smithi057.stderr:hint: 2023-10-30T13:56:54.585 INFO:teuthology.orchestra.run.smithi057.stderr:hint: git config --global init.defaultBranch 2023-10-30T13:56:54.586 INFO:teuthology.orchestra.run.smithi057.stderr:hint: 2023-10-30T13:56:54.586 INFO:teuthology.orchestra.run.smithi057.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2023-10-30T13:56:54.586 INFO:teuthology.orchestra.run.smithi057.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2023-10-30T13:56:54.586 INFO:teuthology.orchestra.run.smithi057.stderr:hint: 2023-10-30T13:56:54.586 INFO:teuthology.orchestra.run.smithi057.stderr:hint: git branch -m 2023-10-30T13:56:54.587 INFO:teuthology.orchestra.run.smithi057.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2023-10-30T13:56:54.588 DEBUG:teuthology.orchestra.run.smithi057:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2023-10-30T13:56:54.654 DEBUG:teuthology.orchestra.run.smithi057:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2023-10-30T13:56:54.709 DEBUG:teuthology.orchestra.run.smithi057:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2023-10-30T13:58:25.106 INFO:teuthology.orchestra.run.smithi057.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2023-10-30T13:58:25.107 INFO:teuthology.orchestra.run.smithi057.stderr: * [new branch] main -> origin/main 2023-10-30T13:58:25.139 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20190312 -> 20190312 2023-10-30T13:58:25.139 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20190416 -> 20190416 2023-10-30T13:58:25.139 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20190514 -> 20190514 2023-10-30T13:58:25.139 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20190618 -> 20190618 2023-10-30T13:58:25.140 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20190717 -> 20190717 2023-10-30T13:58:25.140 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20190815 -> 20190815 2023-10-30T13:58:25.141 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20190923 -> 20190923 2023-10-30T13:58:25.141 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20191022 -> 20191022 2023-10-30T13:58:25.142 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20191215 -> 20191215 2023-10-30T13:58:25.142 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20200122 -> 20200122 2023-10-30T13:58:25.142 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20200316 -> 20200316 2023-10-30T13:58:25.142 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20200421 -> 20200421 2023-10-30T13:58:25.142 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20200519 -> 20200519 2023-10-30T13:58:25.143 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20200619 -> 20200619 2023-10-30T13:58:25.143 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20200721 -> 20200721 2023-10-30T13:58:25.143 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20200817 -> 20200817 2023-10-30T13:58:25.146 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20200918 -> 20200918 2023-10-30T13:58:25.147 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20201022 -> 20201022 2023-10-30T13:58:25.147 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20201118 -> 20201118 2023-10-30T13:58:25.147 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20201218 -> 20201218 2023-10-30T13:58:25.147 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20210208 -> 20210208 2023-10-30T13:58:25.147 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20210315 -> 20210315 2023-10-30T13:58:25.148 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20210511 -> 20210511 2023-10-30T13:58:25.148 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20210716 -> 20210716 2023-10-30T13:58:25.148 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20210818 -> 20210818 2023-10-30T13:58:25.148 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20210919 -> 20210919 2023-10-30T13:58:25.148 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20211027 -> 20211027 2023-10-30T13:58:25.149 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20211216 -> 20211216 2023-10-30T13:58:25.149 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20220209 -> 20220209 2023-10-30T13:58:25.149 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20220310 -> 20220310 2023-10-30T13:58:25.149 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20220411 -> 20220411 2023-10-30T13:58:25.149 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20220509 -> 20220509 2023-10-30T13:58:25.149 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20220610 -> 20220610 2023-10-30T13:58:25.150 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20220708 -> 20220708 2023-10-30T13:58:25.150 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20220815 -> 20220815 2023-10-30T13:58:25.150 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20220913 -> 20220913 2023-10-30T13:58:25.150 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20221012 -> 20221012 2023-10-30T13:58:25.150 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20221109 -> 20221109 2023-10-30T13:58:25.151 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20221214 -> 20221214 2023-10-30T13:58:25.151 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20230117 -> 20230117 2023-10-30T13:58:25.151 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20230210 -> 20230210 2023-10-30T13:58:25.151 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20230310 -> 20230310 2023-10-30T13:58:25.151 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20230404 -> 20230404 2023-10-30T13:58:25.151 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20230515 -> 20230515 2023-10-30T13:58:25.152 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20230625 -> 20230625 2023-10-30T13:58:25.152 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20230804 -> 20230804 2023-10-30T13:58:25.152 INFO:teuthology.orchestra.run.smithi057.stderr: * [new tag] 20230919 -> 20230919 2023-10-30T13:58:29.062 INFO:teuthology.orchestra.run.smithi057.stderr:Updating files: 49% (1553/3168) Updating files: 50% (1584/3168) Updating files: 51% (1616/3168) Updating files: 52% (1648/3168) Updating files: 53% (1680/3168) Updating files: 54% (1711/3168) Updating files: 55% (1743/3168) Updating files: 56% (1775/3168) Updating files: 57% (1806/3168) Updating files: 58% (1838/3168) Updating files: 59% (1870/3168) Updating files: 60% (1901/3168) Updating files: 61% (1933/3168) Updating files: 62% (1965/3168) Updating files: 63% (1996/3168) Updating files: 64% (2028/3168) Updating files: 64% (2035/3168) Updating files: 65% (2060/3168) Updating files: 66% (2091/3168) Updating files: 67% (2123/3168) Updating files: 68% (2155/3168) Updating files: 69% (2186/3168) Updating files: 70% (2218/3168) Updating files: 71% (2250/3168) Updating files: 72% (2281/3168) Updating files: 73% (2313/3168) Updating files: 74% (2345/3168) Updating files: 75% (2376/3168) Updating files: 76% (2408/3168) Updating files: 77% (2440/3168) Updating files: 78% (2472/3168) Updating files: 79% (2503/3168) Updating files: 80% (2535/3168) Updating files: 81% (2567/3168) Updating files: 81% (2580/3168) Updating files: 82% (2598/3168) Updating files: 83% (2630/3168) Updating files: 84% (2662/3168) Updating files: 85% (2693/3168) Updating files: 86% (2725/3168) Updating files: 87% (2757/3168) Updating files: 88% (2788/3168) Updating files: 89% (2820/3168) Updating files: 90% (2852/3168) Updating files: 91% (2883/3168) Updating files: 92% (2915/3168) Updating files: 93% (2947/3168) Updating files: 94% (2978/3168) Updating files: 95% (3010/3168) Updating files: 96% (3042/3168) Updating files: 97% (3073/3168) Updating files: 98% (3105/3168) Updating files: 99% (3137/3168) Updating files: 100% (3168/3168) Updating files: 100% (3168/3168), done. 2023-10-30T13:58:29.067 INFO:teuthology.orchestra.run.smithi057.stdout:HEAD is now at 2b304bfe Merge branch 'main' into 'main' 2023-10-30T13:58:29.088 INFO:teuthology.task.kernel:Downloading kernel eacf00cc51e204f401b4d5d4f84825a68f373c74 on client.0... 2023-10-30T13:58:29.088 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=eacf00cc51e204f401b4d5d4f84825a68f373c74 2023-10-30T13:58:29.238 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/eacf00cc51e204f401b4d5d4f84825a68f373c74/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-geacf00cc51e2/ 2023-10-30T13:58:29.238 DEBUG:teuthology.orchestra.run.smithi057:> rm -f /tmp/linux-image.deb && echo linux-image-6.6.0-rc5-geacf00cc51e2_6.6.0-rc5-geacf00cc51e2-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testing/eacf00cc51e204f401b4d5d4f84825a68f373c74/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-geacf00cc51e2/ --input-file=- 2023-10-30T13:58:29.241 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi057.front.sepia.ceph.com to complete... 2023-10-30T13:58:30.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30 13:58:31 URL:https://4.chacra.ceph.com/r/kernel/testing/eacf00cc51e204f401b4d5d4f84825a68f373c74/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-geacf00cc51e2/linux-image-6.6.0-rc5-geacf00cc51e2_6.6.0-rc5-geacf00cc51e2-1_amd64.deb [38909658/38909658] -> "/tmp/linux-image.deb" [1] 2023-10-30T13:58:30.729 INFO:teuthology.orchestra.run.smithi057.stderr:FINISHED --2023-10-30 13:58:31-- 2023-10-30T13:58:30.729 INFO:teuthology.orchestra.run.smithi057.stderr:Total wall clock time: 1.5s 2023-10-30T13:58:30.729 INFO:teuthology.orchestra.run.smithi057.stderr:Downloaded: 1 files, 37M in 1.3s (27.6 MB/s) 2023-10-30T13:58:30.731 INFO:teuthology.task.kernel:Installing kernel eacf00cc51e204f401b4d5d4f84825a68f373c74 on client.0... 2023-10-30T13:58:30.732 DEBUG:teuthology.orchestra.run.smithi057:> sudo dpkg -i /tmp/linux-image.deb 2023-10-30T13:58:30.940 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package linux-image-6.6.0-rc5-geacf00cc51e2. 2023-10-30T13:58:32.864 INFO:teuthology.orchestra.run.smithi057.stdout:(Reading database ... 129542 files and directories currently installed.) 2023-10-30T13:58:32.868 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack /tmp/linux-image.deb ... 2023-10-30T13:58:32.999 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking linux-image-6.6.0-rc5-geacf00cc51e2 (6.6.0-rc5-geacf00cc51e2-1) ... 2023-10-30T13:58:36.004 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up linux-image-6.6.0-rc5-geacf00cc51e2 (6.6.0-rc5-geacf00cc51e2-1) ... 2023-10-30T13:58:36.113 INFO:teuthology.orchestra.run.smithi057.stderr:update-initramfs: Generating /boot/initrd.img-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:58:55.574 INFO:teuthology.orchestra.run.smithi057.stderr:Sourcing file `/etc/default/grub' 2023-10-30T13:58:55.594 INFO:teuthology.orchestra.run.smithi057.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-30T13:58:55.613 INFO:teuthology.orchestra.run.smithi057.stderr:Generating grub configuration file ... 2023-10-30T13:58:55.923 INFO:teuthology.orchestra.run.smithi057.stderr:Found linux image: /boot/vmlinuz-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:58:55.937 INFO:teuthology.orchestra.run.smithi057.stderr:Found initrd image: /boot/initrd.img-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:58:56.211 INFO:teuthology.orchestra.run.smithi057.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-30T13:58:56.223 INFO:teuthology.orchestra.run.smithi057.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-30T13:58:56.282 INFO:teuthology.orchestra.run.smithi057.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-30T13:58:56.288 INFO:teuthology.orchestra.run.smithi057.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-30T13:58:56.448 INFO:teuthology.orchestra.run.smithi057.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-30T13:58:56.449 INFO:teuthology.orchestra.run.smithi057.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-30T13:58:56.449 INFO:teuthology.orchestra.run.smithi057.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-30T13:58:56.478 INFO:teuthology.orchestra.run.smithi057.stderr:done 2023-10-30T13:58:56.634 DEBUG:teuthology.orchestra.run.smithi057:> dpkg-deb -c /tmp/linux-image.deb 2023-10-30T13:58:56.682 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./ 2023-10-30T13:58:56.682 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./boot/ 2023-10-30T13:58:56.682 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 4415097 2023-10-30 03:46 ./boot/System.map-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:58:56.695 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 169233 2023-10-30 03:46 ./boot/config-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:58:56.695 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11352928 2023-10-30 03:46 ./boot/vmlinuz-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:58:56.716 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/ 2023-10-30T13:58:56.717 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/ 2023-10-30T13:58:56.717 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/postinst.d/ 2023-10-30T13:58:56.717 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/postrm.d/ 2023-10-30T13:58:56.717 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/preinst.d/ 2023-10-30T13:58:56.717 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/prerm.d/ 2023-10-30T13:58:56.718 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/ 2023-10-30T13:58:56.718 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/ 2023-10-30T13:58:56.718 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/ 2023-10-30T13:58:56.718 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/ 2023-10-30T13:58:56.719 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/ 2023-10-30T13:58:56.719 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/ 2023-10-30T13:58:56.719 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/ 2023-10-30T13:58:56.719 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 490233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/aesni-intel.ko 2023-10-30T13:58:56.722 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-10-30T13:58:56.722 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-10-30T13:58:56.722 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-10-30T13:58:56.723 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 93113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-10-30T13:58:56.723 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 106729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-10-30T13:58:56.723 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 152241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-10-30T13:58:56.723 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-10-30T13:58:56.724 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-10-30T13:58:56.724 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 83889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-10-30T13:58:56.724 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-10-30T13:58:56.724 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 47225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-10-30T13:58:56.724 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 47209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/serpent-avx2.ko 2023-10-30T13:58:56.725 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 51801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-10-30T13:58:56.725 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 38937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-10-30T13:58:56.725 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 40529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-10-30T13:58:56.725 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 58585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-10-30T13:58:56.725 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 50497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-10-30T13:58:56.726 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 32065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-10-30T13:58:56.726 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-10-30T13:58:56.726 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/ 2023-10-30T13:58:56.726 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpu/ 2023-10-30T13:58:56.726 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpu/mce/ 2023-10-30T13:58:56.727 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 40113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-10-30T13:58:56.727 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpuid.ko 2023-10-30T13:58:56.727 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/msr.ko 2023-10-30T13:58:56.727 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/ 2023-10-30T13:58:56.728 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 353761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/kvm-amd.ko 2023-10-30T13:58:56.728 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 881713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/kvm-intel.ko 2023-10-30T13:58:56.728 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 2276817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/kvm.ko 2023-10-30T13:58:56.736 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/ 2023-10-30T13:58:56.737 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 53761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/af_alg.ko 2023-10-30T13:58:56.737 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/algif_hash.ko 2023-10-30T13:58:56.737 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/algif_skcipher.ko 2023-10-30T13:58:56.737 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/ansi_cprng.ko 2023-10-30T13:58:56.737 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/anubis.ko 2023-10-30T13:58:56.738 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/arc4.ko 2023-10-30T13:58:56.738 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/asymmetric_keys/ 2023-10-30T13:58:56.738 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2023-10-30T13:58:56.738 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/ 2023-10-30T13:58:56.738 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_memcpy.ko 2023-10-30T13:58:56.739 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_pq.ko 2023-10-30T13:58:56.739 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_raid6_recov.ko 2023-10-30T13:58:56.739 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_tx.ko 2023-10-30T13:58:56.739 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_xor.ko 2023-10-30T13:58:56.740 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/raid6test.ko 2023-10-30T13:58:56.740 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/authenc.ko 2023-10-30T13:58:56.740 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/authencesn.ko 2023-10-30T13:58:56.740 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/blake2b_generic.ko 2023-10-30T13:58:56.740 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/blowfish_common.ko 2023-10-30T13:58:56.740 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/blowfish_generic.ko 2023-10-30T13:58:56.741 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 38873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/camellia_generic.ko 2023-10-30T13:58:56.741 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cast5_generic.ko 2023-10-30T13:58:56.741 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cast6_generic.ko 2023-10-30T13:58:56.741 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cast_common.ko 2023-10-30T13:58:56.741 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 35097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/ccm.ko 2023-10-30T13:58:56.742 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cmac.ko 2023-10-30T13:58:56.742 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crc32_generic.ko 2023-10-30T13:58:56.742 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 47505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cryptd.ko 2023-10-30T13:58:56.742 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 39505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crypto_engine.ko 2023-10-30T13:58:56.742 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crypto_simd.ko 2023-10-30T13:58:56.743 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crypto_user.ko 2023-10-30T13:58:56.743 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cts.ko 2023-10-30T13:58:56.743 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/des_generic.ko 2023-10-30T13:58:56.743 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/dh_generic.ko 2023-10-30T13:58:56.743 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/echainiv.ko 2023-10-30T13:58:56.744 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/essiv.ko 2023-10-30T13:58:56.744 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/fcrypt.ko 2023-10-30T13:58:56.744 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/khazad.ko 2023-10-30T13:58:56.744 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/lrw.ko 2023-10-30T13:58:56.744 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/lz4.ko 2023-10-30T13:58:56.745 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/lz4hc.ko 2023-10-30T13:58:56.745 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/md4.ko 2023-10-30T13:58:56.745 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/michael_mic.ko 2023-10-30T13:58:56.745 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/pcbc.ko 2023-10-30T13:58:56.745 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/pcrypt.ko 2023-10-30T13:58:56.745 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/rmd160.ko 2023-10-30T13:58:56.746 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/seed.ko 2023-10-30T13:58:56.746 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/serpent_generic.ko 2023-10-30T13:58:56.746 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 149193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/tcrypt.ko 2023-10-30T13:58:56.746 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/tea.ko 2023-10-30T13:58:56.747 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/twofish_common.ko 2023-10-30T13:58:56.747 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/twofish_generic.ko 2023-10-30T13:58:56.747 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/vmac.ko 2023-10-30T13:58:56.747 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/wp512.ko 2023-10-30T13:58:56.747 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xcbc.ko 2023-10-30T13:58:56.748 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xor.ko 2023-10-30T13:58:56.748 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xts.ko 2023-10-30T13:58:56.748 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xxhash_generic.ko 2023-10-30T13:58:56.748 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ 2023-10-30T13:58:56.748 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/ 2023-10-30T13:58:56.749 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/acpi_extlog.ko 2023-10-30T13:58:56.749 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/acpi_ipmi.ko 2023-10-30T13:58:56.749 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/acpi_pad.ko 2023-10-30T13:58:56.749 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/apei/ 2023-10-30T13:58:56.749 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 31241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/apei/einj.ko 2023-10-30T13:58:56.749 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/ec_sys.ko 2023-10-30T13:58:56.750 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/platform_profile.ko 2023-10-30T13:58:56.750 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/sbs.ko 2023-10-30T13:58:56.750 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/sbshc.ko 2023-10-30T13:58:56.750 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 103025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/video.ko 2023-10-30T13:58:56.750 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/ 2023-10-30T13:58:56.750 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/acard-ahci.ko 2023-10-30T13:58:56.751 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 88865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/ahci.ko 2023-10-30T13:58:56.751 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/ahci_platform.ko 2023-10-30T13:58:56.751 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 88289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/libahci.ko 2023-10-30T13:58:56.751 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/libahci_platform.ko 2023-10-30T13:58:56.751 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_acpi.ko 2023-10-30T13:58:56.752 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ali.ko 2023-10-30T13:58:56.752 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_amd.ko 2023-10-30T13:58:56.752 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_artop.ko 2023-10-30T13:58:56.752 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_atiixp.ko 2023-10-30T13:58:56.753 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_atp867x.ko 2023-10-30T13:58:56.753 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_cmd640.ko 2023-10-30T13:58:56.753 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_cmd64x.ko 2023-10-30T13:58:56.753 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_cypress.ko 2023-10-30T13:58:56.753 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_efar.ko 2023-10-30T13:58:56.754 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt366.ko 2023-10-30T13:58:56.754 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt37x.ko 2023-10-30T13:58:56.754 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt3x2n.ko 2023-10-30T13:58:56.754 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt3x3.ko 2023-10-30T13:58:56.754 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_it8213.ko 2023-10-30T13:58:56.755 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 35273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_it821x.ko 2023-10-30T13:58:56.755 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_jmicron.ko 2023-10-30T13:58:56.755 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 51481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_legacy.ko 2023-10-30T13:58:56.755 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_marvell.ko 2023-10-30T13:58:56.755 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_mpiix.ko 2023-10-30T13:58:56.756 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_netcell.ko 2023-10-30T13:58:56.756 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ninja32.ko 2023-10-30T13:58:56.756 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ns87410.ko 2023-10-30T13:58:56.756 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ns87415.ko 2023-10-30T13:58:56.756 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_oldpiix.ko 2023-10-30T13:58:56.756 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_opti.ko 2023-10-30T13:58:56.757 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_optidma.ko 2023-10-30T13:58:56.757 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 32585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_pcmcia.ko 2023-10-30T13:58:56.757 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 37793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_pdc2027x.ko 2023-10-30T13:58:56.757 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-10-30T13:58:56.757 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_piccolo.ko 2023-10-30T13:58:56.758 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_radisys.ko 2023-10-30T13:58:56.758 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_rdc.ko 2023-10-30T13:58:56.758 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_rz1000.ko 2023-10-30T13:58:56.758 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_sch.ko 2023-10-30T13:58:56.758 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_serverworks.ko 2023-10-30T13:58:56.759 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_sil680.ko 2023-10-30T13:58:56.759 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_sl82c105.ko 2023-10-30T13:58:56.759 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_triflex.ko 2023-10-30T13:58:56.759 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_via.ko 2023-10-30T13:58:56.759 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pdc_adma.ko 2023-10-30T13:58:56.759 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_inic162x.ko 2023-10-30T13:58:56.760 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 83345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_mv.ko 2023-10-30T13:58:56.760 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 70393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_nv.ko 2023-10-30T13:58:56.760 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 37377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_promise.ko 2023-10-30T13:58:56.760 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_qstor.ko 2023-10-30T13:58:56.760 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sil.ko 2023-10-30T13:58:56.761 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 38337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sil24.ko 2023-10-30T13:58:56.761 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sis.ko 2023-10-30T13:58:56.761 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_svw.ko 2023-10-30T13:58:56.761 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 44793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sx4.ko 2023-10-30T13:58:56.761 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_uli.ko 2023-10-30T13:58:56.761 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 35065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_via.ko 2023-10-30T13:58:56.762 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_vsc.ko 2023-10-30T13:58:56.762 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/auxdisplay/ 2023-10-30T13:58:56.762 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/auxdisplay/ks0108.ko 2023-10-30T13:58:56.762 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/bcma/ 2023-10-30T13:58:56.762 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 124473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/bcma/bcma.ko 2023-10-30T13:58:56.762 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/ 2023-10-30T13:58:56.763 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/aoe/ 2023-10-30T13:58:56.763 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 113665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/aoe/aoe.ko 2023-10-30T13:58:56.763 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/drbd/ 2023-10-30T13:58:56.763 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 965585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/drbd/drbd.ko 2023-10-30T13:58:56.810 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 173225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/floppy.ko 2023-10-30T13:58:56.810 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/mtip32xx/ 2023-10-30T13:58:56.811 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 94297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-10-30T13:58:56.811 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 115793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/nbd.ko 2023-10-30T13:58:56.811 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/null_blk/ 2023-10-30T13:58:56.811 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 110449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/null_blk/null_blk.ko 2023-10-30T13:58:56.811 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 98193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/pktcdvd.ko 2023-10-30T13:58:56.812 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 263769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/rbd.ko 2023-10-30T13:58:56.812 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/xen-blkback/ 2023-10-30T13:58:56.812 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 98145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-10-30T13:58:56.812 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/zram/ 2023-10-30T13:58:56.812 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 60217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/zram/zram.ko 2023-10-30T13:58:56.813 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ 2023-10-30T13:58:56.813 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/applicom.ko 2023-10-30T13:58:56.813 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hangcheck-timer.ko 2023-10-30T13:58:56.813 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/ 2023-10-30T13:58:56.813 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/amd-rng.ko 2023-10-30T13:58:56.814 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/intel-rng.ko 2023-10-30T13:58:56.814 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-10-30T13:58:56.814 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/via-rng.ko 2023-10-30T13:58:56.814 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/virtio-rng.ko 2023-10-30T13:58:56.814 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ 2023-10-30T13:58:56.815 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-10-30T13:58:56.815 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 148601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-10-30T13:58:56.815 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-10-30T13:58:56.815 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 139313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_si.ko 2023-10-30T13:58:56.815 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 70921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-10-30T13:58:56.816 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 53513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-10-30T13:58:56.816 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/lp.ko 2023-10-30T13:58:56.816 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/mwave/ 2023-10-30T13:58:56.816 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 84097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/mwave/mwave.ko 2023-10-30T13:58:56.816 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/nvram.ko 2023-10-30T13:58:56.817 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 32169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ppdev.ko 2023-10-30T13:58:56.817 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 51113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tlclk.ko 2023-10-30T13:58:56.817 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/ 2023-10-30T13:58:56.817 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_atmel.ko 2023-10-30T13:58:56.817 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2023-10-30T13:58:56.818 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2023-10-30T13:58:56.818 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 32985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2023-10-30T13:58:56.818 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_infineon.ko 2023-10-30T13:58:56.818 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_nsc.ko 2023-10-30T13:58:56.818 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/xen-tpmfront.ko 2023-10-30T13:58:56.819 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/ 2023-10-30T13:58:56.819 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/xillybus_class.ko 2023-10-30T13:58:56.819 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 50945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/xillybus_core.ko 2023-10-30T13:58:56.819 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/xillybus_pcie.ko 2023-10-30T13:58:56.819 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/ 2023-10-30T13:58:56.820 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-palmas.ko 2023-10-30T13:58:56.820 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 47849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-si5351.ko 2023-10-30T13:58:56.820 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-twl6040.ko 2023-10-30T13:58:56.820 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-wm831x.ko 2023-10-30T13:58:56.820 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/ 2023-10-30T13:58:56.821 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-10-30T13:58:56.821 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/p4-clockmod.ko 2023-10-30T13:58:56.821 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/speedstep-lib.ko 2023-10-30T13:58:56.821 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ 2023-10-30T13:58:56.821 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ccp/ 2023-10-30T13:58:56.822 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 70801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-10-30T13:58:56.822 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 243025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ccp/ccp.ko 2023-10-30T13:58:56.822 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/ 2023-10-30T13:58:56.822 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/ 2023-10-30T13:58:56.822 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_common/ 2023-10-30T13:58:56.822 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 438761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-10-30T13:58:56.823 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2023-10-30T13:58:56.823 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-10-30T13:58:56.823 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/padlock-aes.ko 2023-10-30T13:58:56.823 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/padlock-sha.ko 2023-10-30T13:58:56.824 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/virtio/ 2023-10-30T13:58:56.824 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 68857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-10-30T13:58:56.824 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dca/ 2023-10-30T13:58:56.824 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dca/dca.ko 2023-10-30T13:58:56.824 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/ 2023-10-30T13:58:56.825 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/dw/ 2023-10-30T13:58:56.825 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/dw/dw_dmac.ko 2023-10-30T13:58:56.825 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/ioat/ 2023-10-30T13:58:56.825 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 122569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/ioat/ioatdma.ko 2023-10-30T13:58:56.825 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/ 2023-10-30T13:58:56.825 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 79953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/amd64_edac.ko 2023-10-30T13:58:56.826 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 35769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/e752x_edac.ko 2023-10-30T13:58:56.826 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 63617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/edac_mce_amd.ko 2023-10-30T13:58:56.826 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i3000_edac.ko 2023-10-30T13:58:56.826 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i3200_edac.ko 2023-10-30T13:58:56.827 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i5100_edac.ko 2023-10-30T13:58:56.827 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i5400_edac.ko 2023-10-30T13:58:56.827 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i7300_edac.ko 2023-10-30T13:58:56.827 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 52041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i7core_edac.ko 2023-10-30T13:58:56.827 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i82975x_edac.ko 2023-10-30T13:58:56.828 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/ie31200_edac.ko 2023-10-30T13:58:56.828 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 56825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/sb_edac.ko 2023-10-30T13:58:56.828 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/x38_edac.ko 2023-10-30T13:58:56.828 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/ 2023-10-30T13:58:56.828 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-gpio.ko 2023-10-30T13:58:56.828 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-max14577.ko 2023-10-30T13:58:56.829 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-max77693.ko 2023-10-30T13:58:56.829 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-max8997.ko 2023-10-30T13:58:56.829 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-palmas.ko 2023-10-30T13:58:56.829 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-rt8973a.ko 2023-10-30T13:58:56.829 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-sm5502.ko 2023-10-30T13:58:56.830 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/ 2023-10-30T13:58:56.830 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/dmi-sysfs.ko 2023-10-30T13:58:56.830 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/efi/ 2023-10-30T13:58:56.830 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/efi/efi-pstore.ko 2023-10-30T13:58:56.830 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/iscsi_ibft.ko 2023-10-30T13:58:56.831 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/ 2023-10-30T13:58:56.831 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-adp5520.ko 2023-10-30T13:58:56.831 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-amd8111.ko 2023-10-30T13:58:56.831 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-arizona.ko 2023-10-30T13:58:56.831 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-da9052.ko 2023-10-30T13:58:56.832 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-da9055.ko 2023-10-30T13:58:56.832 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-dln2.ko 2023-10-30T13:58:56.832 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 77849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-f7188x.ko 2023-10-30T13:58:56.832 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-generic.ko 2023-10-30T13:58:56.832 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-ich.ko 2023-10-30T13:58:56.833 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-janz-ttl.ko 2023-10-30T13:58:56.833 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-kempld.ko 2023-10-30T13:58:56.833 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-lp3943.ko 2023-10-30T13:58:56.833 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-max7300.ko 2023-10-30T13:58:56.833 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-max730x.ko 2023-10-30T13:58:56.833 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-max732x.ko 2023-10-30T13:58:56.834 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-ml-ioh.ko 2023-10-30T13:58:56.834 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 47073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-pca953x.ko 2023-10-30T13:58:56.834 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-pcf857x.ko 2023-10-30T13:58:56.834 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-rdc321x.ko 2023-10-30T13:58:56.834 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-sch.ko 2023-10-30T13:58:56.835 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-sch311x.ko 2023-10-30T13:58:56.835 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-tps65912.ko 2023-10-30T13:58:56.835 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-twl4030.ko 2023-10-30T13:58:56.835 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-twl6040.ko 2023-10-30T13:58:56.835 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-viperboard.ko 2023-10-30T13:58:56.836 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-vx855.ko 2023-10-30T13:58:56.836 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-wm831x.ko 2023-10-30T13:58:56.836 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-wm8350.ko 2023-10-30T13:58:56.836 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-wm8994.ko 2023-10-30T13:58:56.836 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/ 2023-10-30T13:58:56.836 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-a4tech.ko 2023-10-30T13:58:56.837 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 39169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-apple.ko 2023-10-30T13:58:56.837 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-appleir.ko 2023-10-30T13:58:56.837 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-aureal.ko 2023-10-30T13:58:56.837 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-axff.ko 2023-10-30T13:58:56.838 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-belkin.ko 2023-10-30T13:58:56.838 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-cherry.ko 2023-10-30T13:58:56.838 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-chicony.ko 2023-10-30T13:58:56.838 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 60649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-cp2112.ko 2023-10-30T13:58:56.838 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-cypress.ko 2023-10-30T13:58:56.839 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-dr.ko 2023-10-30T13:58:56.839 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-elecom.ko 2023-10-30T13:58:56.839 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-elo.ko 2023-10-30T13:58:56.839 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-emsff.ko 2023-10-30T13:58:56.839 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-ezkey.ko 2023-10-30T13:58:56.840 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-gaff.ko 2023-10-30T13:58:56.840 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-generic.ko 2023-10-30T13:58:56.840 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-gt683r.ko 2023-10-30T13:58:56.840 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-gyration.ko 2023-10-30T13:58:56.840 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-holtek-kbd.ko 2023-10-30T13:58:56.841 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-holtek-mouse.ko 2023-10-30T13:58:56.841 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-holtekff.ko 2023-10-30T13:58:56.841 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-hyperv.ko 2023-10-30T13:58:56.841 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-icade.ko 2023-10-30T13:58:56.841 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-kensington.ko 2023-10-30T13:58:56.842 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-keytouch.ko 2023-10-30T13:58:56.842 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-kye.ko 2023-10-30T13:58:56.842 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-lcpower.ko 2023-10-30T13:58:56.842 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-led.ko 2023-10-30T13:58:56.842 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 42361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-lenovo.ko 2023-10-30T13:58:56.843 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-lg-g15.ko 2023-10-30T13:58:56.843 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 58185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-logitech-dj.ko 2023-10-30T13:58:56.843 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 101369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-10-30T13:58:56.843 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 80993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-logitech.ko 2023-10-30T13:58:56.843 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-magicmouse.ko 2023-10-30T13:58:56.844 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-microsoft.ko 2023-10-30T13:58:56.844 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-monterey.ko 2023-10-30T13:58:56.844 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 50161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-multitouch.ko 2023-10-30T13:58:56.844 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 32593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-ntrig.ko 2023-10-30T13:58:56.844 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-ortek.ko 2023-10-30T13:58:56.845 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-penmount.ko 2023-10-30T13:58:56.845 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-petalynx.ko 2023-10-30T13:58:56.845 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 72937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-picolcd.ko 2023-10-30T13:58:56.845 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-pl.ko 2023-10-30T13:58:56.845 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-plantronics.ko 2023-10-30T13:58:56.846 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-primax.ko 2023-10-30T13:58:56.846 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 35457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-rmi.ko 2023-10-30T13:58:56.846 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-arvo.ko 2023-10-30T13:58:56.846 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-common.ko 2023-10-30T13:58:56.846 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 31425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-isku.ko 2023-10-30T13:58:56.847 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-kone.ko 2023-10-30T13:58:56.847 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 31129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-10-30T13:58:56.847 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-konepure.ko 2023-10-30T13:58:56.847 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-10-30T13:58:56.847 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-lua.ko 2023-10-30T13:58:56.847 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-pyra.ko 2023-10-30T13:58:56.848 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-ryos.ko 2023-10-30T13:58:56.848 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-savu.ko 2023-10-30T13:58:56.848 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat.ko 2023-10-30T13:58:56.848 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-saitek.ko 2023-10-30T13:58:56.848 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-samsung.ko 2023-10-30T13:58:56.849 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sensor-hub.ko 2023-10-30T13:58:56.849 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sjoy.ko 2023-10-30T13:58:56.849 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 51249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sony.ko 2023-10-30T13:58:56.849 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-speedlink.ko 2023-10-30T13:58:56.849 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-steelseries.ko 2023-10-30T13:58:56.850 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sunplus.ko 2023-10-30T13:58:56.850 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-thrustmaster.ko 2023-10-30T13:58:56.850 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-tivo.ko 2023-10-30T13:58:56.850 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-tmff.ko 2023-10-30T13:58:56.850 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-topseed.ko 2023-10-30T13:58:56.851 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-twinhan.ko 2023-10-30T13:58:56.853 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 71073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-uclogic.ko 2023-10-30T13:58:56.853 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-waltop.ko 2023-10-30T13:58:56.853 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 143473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-wiimote.ko 2023-10-30T13:58:56.854 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-xinmo.ko 2023-10-30T13:58:56.854 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-zpff.ko 2023-10-30T13:58:56.854 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-zydacron.ko 2023-10-30T13:58:56.854 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 267825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid.ko 2023-10-30T13:58:56.854 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 32321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/uhid.ko 2023-10-30T13:58:56.855 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/ 2023-10-30T13:58:56.855 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 125721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/usbhid.ko 2023-10-30T13:58:56.855 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/usbkbd.ko 2023-10-30T13:58:56.855 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/usbmouse.ko 2023-10-30T13:58:56.855 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 217873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/wacom.ko 2023-10-30T13:58:56.856 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/ 2023-10-30T13:58:56.856 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 64201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/hv_balloon.ko 2023-10-30T13:58:56.856 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 89625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/hv_utils.ko 2023-10-30T13:58:56.856 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 305465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/hv_vmbus.ko 2023-10-30T13:58:56.856 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ 2023-10-30T13:58:56.857 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 66521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/abituguru.ko 2023-10-30T13:58:56.857 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 61393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/abituguru3.ko 2023-10-30T13:58:56.857 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/acpi_power_meter.ko 2023-10-30T13:58:56.857 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ad7414.ko 2023-10-30T13:58:56.857 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ad7418.ko 2023-10-30T13:58:56.857 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adc128d818.ko 2023-10-30T13:58:56.858 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 31729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1025.ko 2023-10-30T13:58:56.858 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 97977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1026.ko 2023-10-30T13:58:56.858 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1029.ko 2023-10-30T13:58:56.858 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 45553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1031.ko 2023-10-30T13:58:56.858 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm9240.ko 2023-10-30T13:58:56.859 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ads7828.ko 2023-10-30T13:58:56.859 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7410.ko 2023-10-30T13:58:56.859 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7411.ko 2023-10-30T13:58:56.859 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 74433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7462.ko 2023-10-30T13:58:56.859 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 39585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7470.ko 2023-10-30T13:58:56.860 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 67929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7475.ko 2023-10-30T13:58:56.860 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7x10.ko 2023-10-30T13:58:56.860 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 39073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/amc6821.ko 2023-10-30T13:58:56.860 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 55265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/applesmc.ko 2023-10-30T13:58:56.860 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 49913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/asb100.ko 2023-10-30T13:58:56.861 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 58617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/asc7621.ko 2023-10-30T13:58:56.861 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 40889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/asus_atk0110.ko 2023-10-30T13:58:56.861 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/atxp1.ko 2023-10-30T13:58:56.861 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/coretemp.ko 2023-10-30T13:58:56.861 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/da9052-hwmon.ko 2023-10-30T13:58:56.862 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/da9055-hwmon.ko 2023-10-30T13:58:56.862 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 44465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-10-30T13:58:56.862 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 99601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/dme1737.ko 2023-10-30T13:58:56.862 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ds1621.ko 2023-10-30T13:58:56.862 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ds620.ko 2023-10-30T13:58:56.862 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/emc1403.ko 2023-10-30T13:58:56.863 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/emc2103.ko 2023-10-30T13:58:56.863 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/emc6w201.ko 2023-10-30T13:58:56.863 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 66985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/f71805f.ko 2023-10-30T13:58:56.863 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 94457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/f71882fg.ko 2023-10-30T13:58:56.863 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 41401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/f75375s.ko 2023-10-30T13:58:56.864 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/fam15h_power.ko 2023-10-30T13:58:56.864 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 55449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/fschmd.ko 2023-10-30T13:58:56.864 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/g760a.ko 2023-10-30T13:58:56.864 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/g762.ko 2023-10-30T13:58:56.864 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 46801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/gl518sm.ko 2023-10-30T13:58:56.864 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 45529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/gl520sm.ko 2023-10-30T13:58:56.865 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/hih6130.ko 2023-10-30T13:58:56.865 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/hwmon-vid.ko 2023-10-30T13:58:56.865 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/i5500_temp.ko 2023-10-30T13:58:56.865 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/i5k_amb.ko 2023-10-30T13:58:56.865 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ibmaem.ko 2023-10-30T13:58:56.866 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ibmpex.ko 2023-10-30T13:58:56.866 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ina209.ko 2023-10-30T13:58:56.866 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ina2xx.ko 2023-10-30T13:58:56.866 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 130449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/it87.ko 2023-10-30T13:58:56.866 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/jc42.ko 2023-10-30T13:58:56.866 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/k10temp.ko 2023-10-30T13:58:56.867 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/k8temp.ko 2023-10-30T13:58:56.867 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lineage-pem.ko 2023-10-30T13:58:56.867 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 48601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm63.ko 2023-10-30T13:58:56.867 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm73.ko 2023-10-30T13:58:56.867 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm75.ko 2023-10-30T13:58:56.868 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm77.ko 2023-10-30T13:58:56.868 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm78.ko 2023-10-30T13:58:56.868 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 31745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm80.ko 2023-10-30T13:58:56.868 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm83.ko 2023-10-30T13:58:56.868 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 72833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm85.ko 2023-10-30T13:58:56.869 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 46153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm87.ko 2023-10-30T13:58:56.869 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 37913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm90.ko 2023-10-30T13:58:56.869 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm92.ko 2023-10-30T13:58:56.869 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 108841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm93.ko 2023-10-30T13:58:56.869 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm95234.ko 2023-10-30T13:58:56.870 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm95241.ko 2023-10-30T13:58:56.870 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm95245.ko 2023-10-30T13:58:56.870 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc2945.ko 2023-10-30T13:58:56.870 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4151.ko 2023-10-30T13:58:56.870 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4215.ko 2023-10-30T13:58:56.871 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4222.ko 2023-10-30T13:58:56.871 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4245.ko 2023-10-30T13:58:56.871 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4260.ko 2023-10-30T13:58:56.871 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4261.ko 2023-10-30T13:58:56.871 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max16065.ko 2023-10-30T13:58:56.872 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max1619.ko 2023-10-30T13:58:56.872 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max1668.ko 2023-10-30T13:58:56.872 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max197.ko 2023-10-30T13:58:56.872 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max6639.ko 2023-10-30T13:58:56.873 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max6650.ko 2023-10-30T13:58:56.873 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 35777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max6697.ko 2023-10-30T13:58:56.873 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/mc13783-adc.ko 2023-10-30T13:58:56.873 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/mcp3021.ko 2023-10-30T13:58:56.873 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2023-10-30T13:58:56.873 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 44129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct6683.ko 2023-10-30T13:58:56.874 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 119817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct6775-core.ko 2023-10-30T13:58:56.874 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 60705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct6775.ko 2023-10-30T13:58:56.874 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 59737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct7802.ko 2023-10-30T13:58:56.874 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 82337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pc87360.ko 2023-10-30T13:58:56.874 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 54433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pc87427.ko 2023-10-30T13:58:56.875 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pcf8591.ko 2023-10-30T13:58:56.875 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ 2023-10-30T13:58:56.875 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-10-30T13:58:56.875 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-10-30T13:58:56.875 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-10-30T13:58:56.876 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/max16064.ko 2023-10-30T13:58:56.876 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/max34440.ko 2023-10-30T13:58:56.876 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/max8688.ko 2023-10-30T13:58:56.876 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-10-30T13:58:56.876 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 82577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-10-30T13:58:56.876 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-10-30T13:58:56.877 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-10-30T13:58:56.877 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-10-30T13:58:56.877 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-10-30T13:58:56.877 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/powr1220.ko 2023-10-30T13:58:56.877 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sch5627.ko 2023-10-30T13:58:56.877 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sch5636.ko 2023-10-30T13:58:56.878 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sch56xx-common.ko 2023-10-30T13:58:56.878 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 32137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sht15.ko 2023-10-30T13:58:56.878 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sht21.ko 2023-10-30T13:58:56.878 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/shtc1.ko 2023-10-30T13:58:56.878 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 40105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sis5595.ko 2023-10-30T13:58:56.879 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/smsc47b397.ko 2023-10-30T13:58:56.879 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 39193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/smsc47m1.ko 2023-10-30T13:58:56.879 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 37385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/smsc47m192.ko 2023-10-30T13:58:56.879 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/thmc50.ko 2023-10-30T13:58:56.879 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp102.ko 2023-10-30T13:58:56.880 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp103.ko 2023-10-30T13:58:56.880 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp401.ko 2023-10-30T13:58:56.880 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp421.ko 2023-10-30T13:58:56.880 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/via-cputemp.ko 2023-10-30T13:58:56.880 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 41945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/via686a.ko 2023-10-30T13:58:56.880 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 52457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/vt1211.ko 2023-10-30T13:58:56.881 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 51545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/vt8231.ko 2023-10-30T13:58:56.881 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 57513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83627ehf.ko 2023-10-30T13:58:56.881 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 78873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83627hf.ko 2023-10-30T13:58:56.881 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 71345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83781d.ko 2023-10-30T13:58:56.881 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 63289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83791d.ko 2023-10-30T13:58:56.881 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 71321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83792d.ko 2023-10-30T13:58:56.882 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 101473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83793.ko 2023-10-30T13:58:56.882 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 119505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83795.ko 2023-10-30T13:58:56.882 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83l785ts.ko 2023-10-30T13:58:56.882 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 35625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83l786ng.ko 2023-10-30T13:58:56.882 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/wm831x-hwmon.ko 2023-10-30T13:58:56.883 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/wm8350-hwmon.ko 2023-10-30T13:58:56.883 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/ 2023-10-30T13:58:56.883 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/algos/ 2023-10-30T13:58:56.883 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-10-30T13:58:56.883 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-10-30T13:58:56.883 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/ 2023-10-30T13:58:56.884 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ali1535.ko 2023-10-30T13:58:56.884 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ali1563.ko 2023-10-30T13:58:56.884 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2023-10-30T13:58:56.884 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-10-30T13:58:56.884 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-10-30T13:58:56.885 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-10-30T13:58:56.885 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2023-10-30T13:58:56.885 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 6929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-10-30T13:58:56.885 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2023-10-30T13:58:56.885 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 50201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-10-30T13:58:56.885 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-10-30T13:58:56.886 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-10-30T13:58:56.886 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-10-30T13:58:56.886 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-dln2.ko 2023-10-30T13:58:56.886 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-gpio.ko 2023-10-30T13:58:56.886 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 62817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-i801.ko 2023-10-30T13:58:56.887 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-isch.ko 2023-10-30T13:58:56.887 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 47889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-10-30T13:58:56.887 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-kempld.ko 2023-10-30T13:58:56.887 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-10-30T13:58:56.887 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-10-30T13:58:56.888 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ocores.ko 2023-10-30T13:58:56.888 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-parport.ko 2023-10-30T13:58:56.888 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-10-30T13:58:56.888 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 47241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-10-30T13:58:56.888 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2023-10-30T13:58:56.888 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-10-30T13:58:56.889 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-10-30T13:58:56.889 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-sis5595.ko 2023-10-30T13:58:56.889 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-sis630.ko 2023-10-30T13:58:56.889 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-10-30T13:58:56.889 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2023-10-30T13:58:56.890 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-10-30T13:58:56.890 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-via.ko 2023-10-30T13:58:56.890 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-10-30T13:58:56.890 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-10-30T13:58:56.890 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 47657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-xiic.ko 2023-10-30T13:58:56.891 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/i2c-mux.ko 2023-10-30T13:58:56.891 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/i2c-smbus.ko 2023-10-30T13:58:56.891 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/i2c-stub.ko 2023-10-30T13:58:56.891 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/ 2023-10-30T13:58:56.891 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2023-10-30T13:58:56.892 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2023-10-30T13:58:56.892 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2023-10-30T13:58:56.892 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ 2023-10-30T13:58:56.892 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ 2023-10-30T13:58:56.892 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 273609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_cm.ko 2023-10-30T13:58:56.892 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 879969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_core.ko 2023-10-30T13:58:56.893 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 69337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_umad.ko 2023-10-30T13:58:56.893 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 324217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-10-30T13:58:56.893 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 97985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/iw_cm.ko 2023-10-30T13:58:56.893 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 282273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/rdma_cm.ko 2023-10-30T13:58:56.893 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 66201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-10-30T13:58:56.894 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/ 2023-10-30T13:58:56.894 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/cxgb4/ 2023-10-30T13:58:56.894 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 493073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-10-30T13:58:56.894 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx4/ 2023-10-30T13:58:56.894 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 419737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-10-30T13:58:56.895 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx5/ 2023-10-30T13:58:56.895 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 739025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-10-30T13:58:56.895 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mthca/ 2023-10-30T13:58:56.895 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 296625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-10-30T13:58:56.895 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/ocrdma/ 2023-10-30T13:58:56.895 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 153697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2023-10-30T13:58:56.896 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/usnic/ 2023-10-30T13:58:56.896 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 168969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2023-10-30T13:58:56.896 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/ 2023-10-30T13:58:56.896 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/ipoib/ 2023-10-30T13:58:56.896 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 263081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-10-30T13:58:56.896 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/iser/ 2023-10-30T13:58:56.897 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 100241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-10-30T13:58:56.897 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/isert/ 2023-10-30T13:58:56.897 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 111425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-10-30T13:58:56.897 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srp/ 2023-10-30T13:58:56.897 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 128753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-10-30T13:58:56.897 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srpt/ 2023-10-30T13:58:56.898 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 129529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-10-30T13:58:56.898 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/ 2023-10-30T13:58:56.898 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/evbug.ko 2023-10-30T13:58:56.898 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/ff-memless.ko 2023-10-30T13:58:56.898 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/ 2023-10-30T13:58:56.899 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-10-30T13:58:56.899 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/fm801-gp.ko 2023-10-30T13:58:56.899 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 41241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/gameport.ko 2023-10-30T13:58:56.899 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 41801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/joydev.ko 2023-10-30T13:58:56.899 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/ 2023-10-30T13:58:56.899 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/adp5520-keys.ko 2023-10-30T13:58:56.900 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/adp5588-keys.ko 2023-10-30T13:58:56.900 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/adp5589-keys.ko 2023-10-30T13:58:56.900 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2023-10-30T13:58:56.900 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/gpio_keys.ko 2023-10-30T13:58:56.900 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2023-10-30T13:58:56.900 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/lkkbd.ko 2023-10-30T13:58:56.901 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/lm8323.ko 2023-10-30T13:58:56.901 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/lm8333.ko 2023-10-30T13:58:56.901 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/matrix_keypad.ko 2023-10-30T13:58:56.901 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/max7359_keypad.ko 2023-10-30T13:58:56.901 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/mcs_touchkey.ko 2023-10-30T13:58:56.901 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2023-10-30T13:58:56.902 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/newtonkbd.ko 2023-10-30T13:58:56.902 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/opencores-kbd.ko 2023-10-30T13:58:56.902 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/qt1070.ko 2023-10-30T13:58:56.902 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/qt2160.ko 2023-10-30T13:58:56.902 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/samsung-keypad.ko 2023-10-30T13:58:56.903 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/stowaway.ko 2023-10-30T13:58:56.903 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/sunkbd.ko 2023-10-30T13:58:56.903 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/tca6416-keypad.ko 2023-10-30T13:58:56.903 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/tca8418_keypad.ko 2023-10-30T13:58:56.903 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/twl4030_keypad.ko 2023-10-30T13:58:56.904 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/xtkbd.ko 2023-10-30T13:58:56.904 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/matrix-keymap.ko 2023-10-30T13:58:56.904 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/ 2023-10-30T13:58:56.904 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/appletouch.ko 2023-10-30T13:58:56.904 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/bcm5974.ko 2023-10-30T13:58:56.905 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 123681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/cyapatp.ko 2023-10-30T13:58:56.905 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 89449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/elan_i2c.ko 2023-10-30T13:58:56.905 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/gpio_mouse.ko 2023-10-30T13:58:56.905 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 335977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/psmouse.ko 2023-10-30T13:58:56.905 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/sermouse.ko 2023-10-30T13:58:56.905 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-10-30T13:58:56.906 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/synaptics_usb.ko 2023-10-30T13:58:56.906 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/vsxxxaa.ko 2023-10-30T13:58:56.906 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/rmi4/ 2023-10-30T13:58:56.906 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 109873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/rmi4/rmi_core.ko 2023-10-30T13:58:56.906 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/ 2023-10-30T13:58:56.907 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/altera_ps2.ko 2023-10-30T13:58:56.907 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/arc_ps2.ko 2023-10-30T13:58:56.907 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/ct82c710.ko 2023-10-30T13:58:56.907 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-10-30T13:58:56.907 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/parkbd.ko 2023-10-30T13:58:56.908 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/pcips2.ko 2023-10-30T13:58:56.908 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/ps2mult.ko 2023-10-30T13:58:56.908 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/serio_raw.ko 2023-10-30T13:58:56.908 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/serport.ko 2023-10-30T13:58:56.908 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/sparse-keymap.ko 2023-10-30T13:58:56.909 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/iommu/ 2023-10-30T13:58:56.909 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/iommu/amd/ 2023-10-30T13:58:56.909 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 39289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/iommu/amd/iommu_v2.ko 2023-10-30T13:58:56.909 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/ 2023-10-30T13:58:56.909 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-88pm860x.ko 2023-10-30T13:58:56.910 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-adp5520.ko 2023-10-30T13:58:56.910 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 59753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-bd2802.ko 2023-10-30T13:58:56.910 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-blinkm.ko 2023-10-30T13:58:56.910 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-da903x.ko 2023-10-30T13:58:56.910 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-da9052.ko 2023-10-30T13:58:56.910 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-gpio.ko 2023-10-30T13:58:56.911 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm3530.ko 2023-10-30T13:58:56.911 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm3533.ko 2023-10-30T13:58:56.911 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm355x.ko 2023-10-30T13:58:56.911 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm3642.ko 2023-10-30T13:58:56.911 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lp3944.ko 2023-10-30T13:58:56.912 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lp8788.ko 2023-10-30T13:58:56.912 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-max8997.ko 2023-10-30T13:58:56.912 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-mc13783.ko 2023-10-30T13:58:56.912 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-menf21bmc.ko 2023-10-30T13:58:56.912 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-pca9532.ko 2023-10-30T13:58:56.912 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-pca955x.ko 2023-10-30T13:58:56.913 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-pca963x.ko 2023-10-30T13:58:56.913 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-ss4200.ko 2023-10-30T13:58:56.913 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-tca6507.ko 2023-10-30T13:58:56.913 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-wm831x-status.ko 2023-10-30T13:58:56.913 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-wm8350.ko 2023-10-30T13:58:56.914 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ 2023-10-30T13:58:56.914 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-10-30T13:58:56.914 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-10-30T13:58:56.914 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-10-30T13:58:56.914 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-10-30T13:58:56.915 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-10-30T13:58:56.915 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-10-30T13:58:56.915 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-10-30T13:58:56.915 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-10-30T13:58:56.915 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/ 2023-10-30T13:58:56.915 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/bcache/ 2023-10-30T13:58:56.916 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 654969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/bcache/bcache.ko 2023-10-30T13:58:56.916 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 40577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-bio-prison.ko 2023-10-30T13:58:56.916 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 89601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-bufio.ko 2023-10-30T13:58:56.916 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 47529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-cache-smq.ko 2023-10-30T13:58:56.916 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 156009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-cache.ko 2023-10-30T13:58:56.917 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 106897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-crypt.ko 2023-10-30T13:58:56.917 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-delay.ko 2023-10-30T13:58:56.917 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 53857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-era.ko 2023-10-30T13:58:56.917 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-flakey.ko 2023-10-30T13:58:56.917 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 41593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-log-userspace.ko 2023-10-30T13:58:56.917 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 32737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-log.ko 2023-10-30T13:58:56.918 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 49545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-mirror.ko 2023-10-30T13:58:56.918 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 78033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-multipath.ko 2023-10-30T13:58:56.918 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-queue-length.ko 2023-10-30T13:58:56.918 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 70825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-raid.ko 2023-10-30T13:58:56.918 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-region-hash.ko 2023-10-30T13:58:56.919 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-round-robin.ko 2023-10-30T13:58:56.919 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-service-time.ko 2023-10-30T13:58:56.919 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 109593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-snapshot.ko 2023-10-30T13:58:56.919 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-switch.ko 2023-10-30T13:58:56.919 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 181865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-thin-pool.ko 2023-10-30T13:58:56.920 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 51737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-verity.ko 2023-10-30T13:58:56.920 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-zero.ko 2023-10-30T13:58:56.920 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/faulty.ko 2023-10-30T13:58:56.920 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/linear.ko 2023-10-30T13:58:56.920 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 32417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/multipath.ko 2023-10-30T13:58:56.920 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/persistent-data/ 2023-10-30T13:58:56.921 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 199937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-10-30T13:58:56.921 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 37337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid0.ko 2023-10-30T13:58:56.921 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 113073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid1.ko 2023-10-30T13:58:56.921 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 158681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid10.ko 2023-10-30T13:58:56.921 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 395721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid456.ko 2023-10-30T13:58:56.921 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/ 2023-10-30T13:58:56.922 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/ 2023-10-30T13:58:56.922 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 178137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptbase.ko 2023-10-30T13:58:56.922 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 63385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptctl.ko 2023-10-30T13:58:56.922 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 44665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptfc.ko 2023-10-30T13:58:56.922 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 37225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptlan.ko 2023-10-30T13:58:56.923 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 123321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptsas.ko 2023-10-30T13:58:56.923 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 79041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptscsih.ko 2023-10-30T13:58:56.923 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 53081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptspi.ko 2023-10-30T13:58:56.923 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/ 2023-10-30T13:58:56.923 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/88pm800.ko 2023-10-30T13:58:56.924 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/88pm805.ko 2023-10-30T13:58:56.924 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/88pm80x.ko 2023-10-30T13:58:56.924 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/arizona-i2c.ko 2023-10-30T13:58:56.924 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 123721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/arizona.ko 2023-10-30T13:58:56.924 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/bcm590xx.ko 2023-10-30T13:58:56.924 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/cros_ec_dev.ko 2023-10-30T13:58:56.925 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/dln2.ko 2023-10-30T13:58:56.925 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/janz-cmodio.ko 2023-10-30T13:58:56.925 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 53585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/kempld-core.ko 2023-10-30T13:58:56.925 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lm3533-core.ko 2023-10-30T13:58:56.925 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lm3533-ctrlbank.ko 2023-10-30T13:58:56.925 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lp3943.ko 2023-10-30T13:58:56.926 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 54569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lpc_ich.ko 2023-10-30T13:58:56.926 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lpc_sch.ko 2023-10-30T13:58:56.926 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/max8907.ko 2023-10-30T13:58:56.926 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/mc13xxx-core.ko 2023-10-30T13:58:56.926 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/mc13xxx-i2c.ko 2023-10-30T13:58:56.927 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/menf21bmc.ko 2023-10-30T13:58:56.927 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/pcf50633-adc.ko 2023-10-30T13:58:56.927 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/pcf50633-gpio.ko 2023-10-30T13:58:56.927 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/pcf50633.ko 2023-10-30T13:58:56.927 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/rdc321x-southbridge.ko 2023-10-30T13:58:56.927 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/retu-mfd.ko 2023-10-30T13:58:56.928 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 80425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/si476x-core.ko 2023-10-30T13:58:56.928 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 48457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/sm501.ko 2023-10-30T13:58:56.928 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/ti_am335x_tscadc.ko 2023-10-30T13:58:56.928 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 51137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/tps65010.ko 2023-10-30T13:58:56.928 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/tps6507x.ko 2023-10-30T13:58:56.929 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/viperboard.ko 2023-10-30T13:58:56.929 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/vx855.ko 2023-10-30T13:58:56.929 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/wl1273-core.ko 2023-10-30T13:58:56.929 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 44345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/wm8994.ko 2023-10-30T13:58:56.929 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ 2023-10-30T13:58:56.929 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ad525x_dpot-i2c.ko 2023-10-30T13:58:56.930 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 48489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ad525x_dpot.ko 2023-10-30T13:58:56.930 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/altera-stapl/ 2023-10-30T13:58:56.930 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 46585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-10-30T13:58:56.930 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/apds9802als.ko 2023-10-30T13:58:56.930 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/apds990x.ko 2023-10-30T13:58:56.931 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 45593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/bh1770glc.ko 2023-10-30T13:58:56.931 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/c2port/ 2023-10-30T13:58:56.931 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2023-10-30T13:58:56.931 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 35281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/c2port/core.ko 2023-10-30T13:58:56.931 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/cb710/ 2023-10-30T13:58:56.931 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/cb710/cb710.ko 2023-10-30T13:58:56.932 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ds1682.ko 2023-10-30T13:58:56.932 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/dummy-irq.ko 2023-10-30T13:58:56.932 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/echo/ 2023-10-30T13:58:56.932 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/echo/echo.ko 2023-10-30T13:58:56.932 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/ 2023-10-30T13:58:56.932 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 37849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/at24.ko 2023-10-30T13:58:56.933 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/eeprom.ko 2023-10-30T13:58:56.933 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-10-30T13:58:56.933 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/max6875.ko 2023-10-30T13:58:56.933 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/enclosure.ko 2023-10-30T13:58:56.933 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/genwqe/ 2023-10-30T13:58:56.933 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 152201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/genwqe/genwqe_card.ko 2023-10-30T13:58:56.934 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/hmc6352.ko 2023-10-30T13:58:56.934 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 31321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/hpilo.ko 2023-10-30T13:58:56.934 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ibmasm/ 2023-10-30T13:58:56.934 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 72129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ibmasm/ibmasm.ko 2023-10-30T13:58:56.934 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ics932s401.ko 2023-10-30T13:58:56.935 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/isl29003.ko 2023-10-30T13:58:56.935 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/isl29020.ko 2023-10-30T13:58:56.935 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/lis3lv02d/ 2023-10-30T13:58:56.935 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 42513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-10-30T13:58:56.935 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-10-30T13:58:56.936 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/ 2023-10-30T13:58:56.936 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 125209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/mei-me.ko 2023-10-30T13:58:56.936 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 56521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/mei-txe.ko 2023-10-30T13:58:56.936 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 310113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/mei.ko 2023-10-30T13:58:56.936 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 31329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/phantom.ko 2023-10-30T13:58:56.936 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ti-st/ 2023-10-30T13:58:56.937 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 74265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ti-st/st_drv.ko 2023-10-30T13:58:56.937 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/tifm_7xx1.ko 2023-10-30T13:58:56.937 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/tifm_core.ko 2023-10-30T13:58:56.937 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/tsl2550.ko 2023-10-30T13:58:56.937 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 48697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/vmw_balloon.ko 2023-10-30T13:58:56.937 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/vmw_vmci/ 2023-10-30T13:58:56.938 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 183609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-10-30T13:58:56.938 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ 2023-10-30T13:58:56.938 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/bonding/ 2023-10-30T13:58:56.938 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 490129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/bonding/bonding.ko 2023-10-30T13:58:56.938 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/ 2023-10-30T13:58:56.938 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/b53/ 2023-10-30T13:58:56.939 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 106033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/b53/b53_common.ko 2023-10-30T13:58:56.939 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 40297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/bcm-sf2.ko 2023-10-30T13:58:56.939 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/mv88e6060.ko 2023-10-30T13:58:56.939 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/mv88e6xxx/ 2023-10-30T13:58:56.939 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 453737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-10-30T13:58:56.940 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dummy.ko 2023-10-30T13:58:56.940 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/eql.ko 2023-10-30T13:58:56.940 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ 2023-10-30T13:58:56.940 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/ 2023-10-30T13:58:56.940 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 45785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-10-30T13:58:56.940 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 42105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-10-30T13:58:56.941 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 130553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-10-30T13:58:56.941 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 68849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-10-30T13:58:56.941 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/ 2023-10-30T13:58:56.941 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 37649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/8390.ko 2023-10-30T13:58:56.941 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 59689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-10-30T13:58:56.941 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-10-30T13:58:56.942 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 99113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-10-30T13:58:56.942 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/adaptec/ 2023-10-30T13:58:56.942 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 54521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-10-30T13:58:56.942 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/agere/ 2023-10-30T13:58:56.942 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 69201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/agere/et131x.ko 2023-10-30T13:58:56.943 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/alteon/ 2023-10-30T13:58:56.943 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 59585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-10-30T13:58:56.943 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/altera/ 2023-10-30T13:58:56.943 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 41409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-10-30T13:58:56.943 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/ 2023-10-30T13:58:56.943 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 54609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-10-30T13:58:56.944 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 49289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-10-30T13:58:56.944 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 87297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-10-30T13:58:56.944 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/ 2023-10-30T13:58:56.944 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/alx/ 2023-10-30T13:58:56.944 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 97337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-10-30T13:58:56.945 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1c/ 2023-10-30T13:58:56.945 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 104673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-10-30T13:58:56.945 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1e/ 2023-10-30T13:58:56.945 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 90257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-10-30T13:58:56.945 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atlx/ 2023-10-30T13:58:56.946 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 81569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-10-30T13:58:56.946 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 69729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-10-30T13:58:56.946 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/ 2023-10-30T13:58:56.946 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 78401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-10-30T13:58:56.946 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 176761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-10-30T13:58:56.947 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2023-10-30T13:58:56.947 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 1421497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-10-30T13:58:56.947 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 128569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-10-30T13:58:56.947 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/genet/ 2023-10-30T13:58:56.947 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 132089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-10-30T13:58:56.948 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 312081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-10-30T13:58:56.948 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/brocade/ 2023-10-30T13:58:56.948 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/brocade/bna/ 2023-10-30T13:58:56.948 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 323513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-10-30T13:58:56.948 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/ 2023-10-30T13:58:56.949 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb/ 2023-10-30T13:58:56.949 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 193497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-10-30T13:58:56.949 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2023-10-30T13:58:56.949 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 347417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-10-30T13:58:57.037 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2023-10-30T13:58:57.038 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 1044433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-10-30T13:58:57.038 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2023-10-30T13:58:57.038 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 166193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-10-30T13:58:57.038 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2023-10-30T13:58:57.039 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-10-30T13:58:57.039 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/cisco/ 2023-10-30T13:58:57.039 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/cisco/enic/ 2023-10-30T13:58:57.039 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 204345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-10-30T13:58:57.039 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/ 2023-10-30T13:58:57.040 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/ 2023-10-30T13:58:57.040 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 69449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-10-30T13:58:57.040 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 61489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-10-30T13:58:57.040 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 141225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-10-30T13:58:57.040 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 51497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-10-30T13:58:57.041 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 62817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-10-30T13:58:57.041 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 31369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-10-30T13:58:57.041 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dlink/ 2023-10-30T13:58:57.041 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 60873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-10-30T13:58:57.042 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 62833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-10-30T13:58:57.042 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 45721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dnet.ko 2023-10-30T13:58:57.042 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ec_bhf.ko 2023-10-30T13:58:57.042 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/emulex/ 2023-10-30T13:58:57.042 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/emulex/benet/ 2023-10-30T13:58:57.043 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 281137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-10-30T13:58:57.043 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 49457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ethoc.ko 2023-10-30T13:58:57.043 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 51737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/fealnx.ko 2023-10-30T13:58:57.043 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/fujitsu/ 2023-10-30T13:58:57.043 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 45345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2023-10-30T13:58:57.044 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ 2023-10-30T13:58:57.044 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 90465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e100.ko 2023-10-30T13:58:57.044 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000/ 2023-10-30T13:58:57.044 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 295577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-10-30T13:58:57.044 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000e/ 2023-10-30T13:58:57.044 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 584497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-10-30T13:58:57.045 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/fm10k/ 2023-10-30T13:58:57.045 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 246209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-10-30T13:58:57.045 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/i40e/ 2023-10-30T13:58:57.045 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 925697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-10-30T13:58:57.045 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/iavf/ 2023-10-30T13:58:57.046 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 302849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-10-30T13:58:57.046 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igb/ 2023-10-30T13:58:57.046 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 514681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-10-30T13:58:57.046 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igbvf/ 2023-10-30T13:58:57.046 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 93593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-10-30T13:58:57.047 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbe/ 2023-10-30T13:58:57.047 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 734305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-10-30T13:58:57.047 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbevf/ 2023-10-30T13:58:57.047 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 151769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-10-30T13:58:57.047 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 86921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/jme.ko 2023-10-30T13:58:57.047 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/ 2023-10-30T13:58:57.048 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-10-30T13:58:57.048 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 102289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/skge.ko 2023-10-30T13:58:57.048 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 122649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-10-30T13:58:57.048 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/ 2023-10-30T13:58:57.048 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx4/ 2023-10-30T13:58:57.049 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 727601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-10-30T13:58:57.049 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 278937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-10-30T13:58:57.049 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx5/ 2023-10-30T13:58:57.049 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2023-10-30T13:58:57.049 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 1113993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-10-30T13:58:57.050 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ 2023-10-30T13:58:57.050 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 53105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ks8842.ko 2023-10-30T13:58:57.050 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 40993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2023-10-30T13:58:57.050 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2023-10-30T13:58:57.050 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 95665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-10-30T13:58:57.051 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/myricom/ 2023-10-30T13:58:57.051 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/myricom/myri10ge/ 2023-10-30T13:58:57.051 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 108649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-10-30T13:58:57.051 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/natsemi/ 2023-10-30T13:58:57.051 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 73441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-10-30T13:58:57.052 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 47777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-10-30T13:58:57.052 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/neterion/ 2023-10-30T13:58:57.052 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 150905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-10-30T13:58:57.052 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/nvidia/ 2023-10-30T13:58:57.052 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 136721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-10-30T13:58:57.052 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/packetengines/ 2023-10-30T13:58:57.053 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 50481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-10-30T13:58:57.053 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 43145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-10-30T13:58:57.053 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/ 2023-10-30T13:58:57.053 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/netxen/ 2023-10-30T13:58:57.053 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 227897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-10-30T13:58:57.053 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 87081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-10-30T13:58:57.054 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2023-10-30T13:58:57.054 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 672049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-10-30T13:58:57.054 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/rdc/ 2023-10-30T13:58:57.054 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 43017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-10-30T13:58:57.054 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/ 2023-10-30T13:58:57.057 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 67385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-10-30T13:58:57.057 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 83985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-10-30T13:58:57.057 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/atp.ko 2023-10-30T13:58:57.057 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 189801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-10-30T13:58:57.058 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/samsung/ 2023-10-30T13:58:57.058 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/samsung/sxgbe/ 2023-10-30T13:58:57.058 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 104241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2023-10-30T13:58:57.058 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sfc/ 2023-10-30T13:58:57.058 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 1053921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-10-30T13:58:57.059 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/silan/ 2023-10-30T13:58:57.059 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 47097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-10-30T13:58:57.059 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sis/ 2023-10-30T13:58:57.059 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 54817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sis/sis190.ko 2023-10-30T13:58:57.059 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 63337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sis/sis900.ko 2023-10-30T13:58:57.059 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/ 2023-10-30T13:58:57.060 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 61601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-10-30T13:58:57.060 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 67193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-10-30T13:58:57.060 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 73409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-10-30T13:58:57.060 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 68177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-10-30T13:58:57.060 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/ 2023-10-30T13:58:57.060 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/ 2023-10-30T13:58:57.061 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2023-10-30T13:58:57.061 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 39905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-10-30T13:58:57.061 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2023-10-30T13:58:57.061 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 486961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-10-30T13:58:57.061 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/ 2023-10-30T13:58:57.061 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 103337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/cassini.ko 2023-10-30T13:58:57.062 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 166161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/niu.ko 2023-10-30T13:58:57.062 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 66329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/sungem.ko 2023-10-30T13:58:57.062 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 78801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-10-30T13:58:57.062 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/tehuti/ 2023-10-30T13:58:57.062 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 50057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-10-30T13:58:57.062 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ti/ 2023-10-30T13:58:57.063 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 86529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ti/tlan.ko 2023-10-30T13:58:57.063 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/via/ 2023-10-30T13:58:57.063 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 80273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-10-30T13:58:57.063 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 83633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-10-30T13:58:57.063 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/wiznet/ 2023-10-30T13:58:57.063 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 46241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-10-30T13:58:57.064 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-10-30T13:58:57.064 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/xircom/ 2023-10-30T13:58:57.064 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 60057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-10-30T13:58:57.064 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/ 2023-10-30T13:58:57.064 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 41065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/defxx.ko 2023-10-30T13:58:57.064 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/skfp/ 2023-10-30T13:58:57.065 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 216985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/skfp/skfp.ko 2023-10-30T13:58:57.065 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 77345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/geneve.ko 2023-10-30T13:58:57.065 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/hyperv/ 2023-10-30T13:58:57.065 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 235521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-10-30T13:58:57.065 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ieee802154/ 2023-10-30T13:58:57.065 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ieee802154/fakelb.ko 2023-10-30T13:58:57.066 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ifb.ko 2023-10-30T13:58:57.066 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ipvlan/ 2023-10-30T13:58:57.066 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 83769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ipvlan/ipvlan.ko 2023-10-30T13:58:57.066 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 62209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/macvlan.ko 2023-10-30T13:58:57.066 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/macvtap.ko 2023-10-30T13:58:57.066 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/ 2023-10-30T13:58:57.067 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-10-30T13:58:57.067 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-10-30T13:58:57.067 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-gpio.ko 2023-10-30T13:58:57.067 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-regmap.ko 2023-10-30T13:58:57.067 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio.ko 2023-10-30T13:58:57.067 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mii.ko 2023-10-30T13:58:57.068 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 43953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/netconsole.ko 2023-10-30T13:58:57.068 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/nlmon.ko 2023-10-30T13:58:57.068 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/pcs/ 2023-10-30T13:58:57.068 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/pcs/pcs-lynx.ko 2023-10-30T13:58:57.068 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 38977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-10-30T13:58:57.068 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/ 2023-10-30T13:58:57.069 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/amd.ko 2023-10-30T13:58:57.069 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 50097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-10-30T13:58:57.069 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 44505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/bcm7xxx.ko 2023-10-30T13:58:57.069 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/bcm87xx.ko 2023-10-30T13:58:57.069 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 42241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/broadcom.ko 2023-10-30T13:58:57.070 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/cicada.ko 2023-10-30T13:58:57.070 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/davicom.ko 2023-10-30T13:58:57.070 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/et1011c.ko 2023-10-30T13:58:57.070 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/icplus.ko 2023-10-30T13:58:57.070 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/lxt.ko 2023-10-30T13:58:57.070 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 82049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/marvell.ko 2023-10-30T13:58:57.071 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 76177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/micrel.ko 2023-10-30T13:58:57.071 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/national.ko 2023-10-30T13:58:57.071 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 97857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/phylink.ko 2023-10-30T13:58:57.071 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/qsemi.ko 2023-10-30T13:58:57.071 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 50065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/realtek.ko 2023-10-30T13:58:57.071 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/smsc.ko 2023-10-30T13:58:57.072 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/ste10Xp.ko 2023-10-30T13:58:57.072 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/vitesse.ko 2023-10-30T13:58:57.072 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/plip/ 2023-10-30T13:58:57.072 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 43569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/plip/plip.ko 2023-10-30T13:58:57.072 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ 2023-10-30T13:58:57.072 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/bsd_comp.ko 2023-10-30T13:58:57.073 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_async.ko 2023-10-30T13:58:57.073 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_deflate.ko 2023-10-30T13:58:57.073 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_mppe.ko 2023-10-30T13:58:57.073 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_synctty.ko 2023-10-30T13:58:57.073 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 42545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/pppoe.ko 2023-10-30T13:58:57.073 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/pppox.ko 2023-10-30T13:58:57.074 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 32689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/pptp.ko 2023-10-30T13:58:57.074 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/sb1000.ko 2023-10-30T13:58:57.074 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/slip/ 2023-10-30T13:58:57.074 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 42465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/slip/slip.ko 2023-10-30T13:58:57.074 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/sungem_phy.ko 2023-10-30T13:58:57.074 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 64385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/tap.ko 2023-10-30T13:58:57.075 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/ 2023-10-30T13:58:57.075 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 106169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team.ko 2023-10-30T13:58:57.075 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_activebackup.ko 2023-10-30T13:58:57.075 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_broadcast.ko 2023-10-30T13:58:57.075 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-10-30T13:58:57.075 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_random.ko 2023-10-30T13:58:57.076 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-10-30T13:58:57.076 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 87017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/veth.ko 2023-10-30T13:58:57.076 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vmxnet3/ 2023-10-30T13:58:57.076 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 144297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-10-30T13:58:57.076 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vxlan/ 2023-10-30T13:58:57.076 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 297329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vxlan/vxlan.ko 2023-10-30T13:58:57.076 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/xen-netback/ 2023-10-30T13:58:57.077 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 144649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/xen-netback/xen-netback.ko 2023-10-30T13:58:57.077 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ntb/ 2023-10-30T13:58:57.077 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ntb/ntb.ko 2023-10-30T13:58:57.077 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/ 2023-10-30T13:58:57.077 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/host/ 2023-10-30T13:58:57.077 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 264937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/host/nvme-core.ko 2023-10-30T13:58:57.078 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 102209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/host/nvme.ko 2023-10-30T13:58:57.078 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/ 2023-10-30T13:58:57.078 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 133841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport.ko 2023-10-30T13:58:57.078 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport_cs.ko 2023-10-30T13:58:57.078 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 84465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport_pc.ko 2023-10-30T13:58:57.079 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport_serial.ko 2023-10-30T13:58:57.079 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/ 2023-10-30T13:58:57.079 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/ 2023-10-30T13:58:57.079 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-10-30T13:58:57.079 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2023-10-30T13:58:57.079 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2023-10-30T13:58:57.080 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/pci-stub.ko 2023-10-30T13:58:57.080 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 46529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/xen-pcifront.ko 2023-10-30T13:58:57.080 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/ 2023-10-30T13:58:57.080 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/i82092.ko 2023-10-30T13:58:57.080 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 145809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pcmcia.ko 2023-10-30T13:58:57.080 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 62137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pcmcia_core.ko 2023-10-30T13:58:57.081 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2023-10-30T13:58:57.081 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pd6729.ko 2023-10-30T13:58:57.081 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 112889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/yenta_socket.ko 2023-10-30T13:58:57.081 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/ 2023-10-30T13:58:57.081 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/broadcom/ 2023-10-30T13:58:57.081 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2023-10-30T13:58:57.082 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/samsung/ 2023-10-30T13:58:57.082 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2023-10-30T13:58:57.082 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pinctrl/ 2023-10-30T13:58:57.082 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pinctrl/intel/ 2023-10-30T13:58:57.082 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 70153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-10-30T13:58:57.082 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/ 2023-10-30T13:58:57.083 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/ 2023-10-30T13:58:57.083 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 50889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2023-10-30T13:58:57.083 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec.ko 2023-10-30T13:58:57.083 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2023-10-30T13:58:57.083 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2023-10-30T13:58:57.084 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2023-10-30T13:58:57.084 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2023-10-30T13:58:57.084 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2023-10-30T13:58:57.084 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/ 2023-10-30T13:58:57.084 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 67097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/acer-wmi.ko 2023-10-30T13:58:57.084 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 39145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/acerhdf.ko 2023-10-30T13:58:57.085 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/apple-gmux.ko 2023-10-30T13:58:57.085 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 59665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/asus-laptop.ko 2023-10-30T13:58:57.085 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 31097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-10-30T13:58:57.085 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 127569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/asus-wmi.ko 2023-10-30T13:58:57.085 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/classmate-laptop.ko 2023-10-30T13:58:57.085 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 49121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-10-30T13:58:57.085 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-10-30T13:58:57.086 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 40265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-10-30T13:58:57.086 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-10-30T13:58:57.086 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/hdaps.ko 2023-10-30T13:58:57.086 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/ibm_rtl.ko 2023-10-30T13:58:57.086 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel/ 2023-10-30T13:58:57.087 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-10-30T13:58:57.087 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2023-10-30T13:58:57.087 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 43497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel_ips.ko 2023-10-30T13:58:57.087 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/msi-wmi.ko 2023-10-30T13:58:57.087 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/mxm-wmi.ko 2023-10-30T13:58:57.087 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-10-30T13:58:57.088 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 42329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/samsung-laptop.ko 2023-10-30T13:58:57.088 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/samsung-q10.ko 2023-10-30T13:58:57.088 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/topstar-laptop.ko 2023-10-30T13:58:57.088 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-10-30T13:58:57.088 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/toshiba_haps.ko 2023-10-30T13:58:57.088 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/wmi-bmof.ko 2023-10-30T13:58:57.089 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 57569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/wmi.ko 2023-10-30T13:58:57.089 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/ 2023-10-30T13:58:57.089 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ 2023-10-30T13:58:57.089 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/88pm860x_battery.ko 2023-10-30T13:58:57.089 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/88pm860x_charger.ko 2023-10-30T13:58:57.089 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 55057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/bq2415x_charger.ko 2023-10-30T13:58:57.090 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 54081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/bq24190_charger.ko 2023-10-30T13:58:57.090 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/bq24735-charger.ko 2023-10-30T13:58:57.090 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/cros_peripheral_charger.ko 2023-10-30T13:58:57.090 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/da9030_battery.ko 2023-10-30T13:58:57.090 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/da9052-battery.ko 2023-10-30T13:58:57.091 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2760_battery.ko 2023-10-30T13:58:57.091 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2780_battery.ko 2023-10-30T13:58:57.091 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2781_battery.ko 2023-10-30T13:58:57.091 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2782_battery.ko 2023-10-30T13:58:57.091 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/gpio-charger.ko 2023-10-30T13:58:57.091 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/isp1704_charger.ko 2023-10-30T13:58:57.092 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/lp8727_charger.ko 2023-10-30T13:58:57.092 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max14577_charger.ko 2023-10-30T13:58:57.092 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max17040_battery.ko 2023-10-30T13:58:57.092 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max17042_battery.ko 2023-10-30T13:58:57.092 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max8903_charger.ko 2023-10-30T13:58:57.093 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max8925_power.ko 2023-10-30T13:58:57.093 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/pcf50633-charger.ko 2023-10-30T13:58:57.093 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 38009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/sbs-battery.ko 2023-10-30T13:58:57.093 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/test_power.ko 2023-10-30T13:58:57.093 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/tps65090-charger.ko 2023-10-30T13:58:57.093 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/wm831x_backup.ko 2023-10-30T13:58:57.094 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/wm831x_power.ko 2023-10-30T13:58:57.094 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/wm8350_power.ko 2023-10-30T13:58:57.094 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/powercap/ 2023-10-30T13:58:57.094 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 57729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/powercap/intel_rapl_common.ko 2023-10-30T13:58:57.094 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/powercap/intel_rapl_msr.ko 2023-10-30T13:58:57.094 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/ 2023-10-30T13:58:57.095 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/ 2023-10-30T13:58:57.095 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/pps-gpio.ko 2023-10-30T13:58:57.095 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/pps-ldisc.ko 2023-10-30T13:58:57.095 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/pps_parport.ko 2023-10-30T13:58:57.095 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 41873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/pps_core.ko 2023-10-30T13:58:57.095 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ptp/ 2023-10-30T13:58:57.096 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 66385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ptp/ptp.ko 2023-10-30T13:58:57.096 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ptp/ptp_kvm.ko 2023-10-30T13:58:57.096 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/ 2023-10-30T13:58:57.096 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-88pm80x.ko 2023-10-30T13:58:57.096 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-88pm860x.ko 2023-10-30T13:58:57.096 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-bq32k.ko 2023-10-30T13:58:57.097 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-da9052.ko 2023-10-30T13:58:57.097 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-da9055.ko 2023-10-30T13:58:57.097 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-da9063.ko 2023-10-30T13:58:57.097 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1286.ko 2023-10-30T13:58:57.097 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 68577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1307.ko 2023-10-30T13:58:57.097 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1374.ko 2023-10-30T13:58:57.098 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1511.ko 2023-10-30T13:58:57.098 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1553.ko 2023-10-30T13:58:57.098 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1672.ko 2023-10-30T13:58:57.098 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1742.ko 2023-10-30T13:58:57.098 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds2404.ko 2023-10-30T13:58:57.098 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds3232.ko 2023-10-30T13:58:57.099 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-em3027.ko 2023-10-30T13:58:57.099 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-fm3130.ko 2023-10-30T13:58:57.099 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-isl12022.ko 2023-10-30T13:58:57.099 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 35969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-isl1208.ko 2023-10-30T13:58:57.099 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-lp8788.ko 2023-10-30T13:58:57.099 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 38097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m41t80.ko 2023-10-30T13:58:57.100 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m48t35.ko 2023-10-30T13:58:57.100 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m48t59.ko 2023-10-30T13:58:57.100 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m48t86.ko 2023-10-30T13:58:57.100 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max6900.ko 2023-10-30T13:58:57.100 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8907.ko 2023-10-30T13:58:57.100 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8925.ko 2023-10-30T13:58:57.101 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8997.ko 2023-10-30T13:58:57.101 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8998.ko 2023-10-30T13:58:57.101 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-mc13xxx.ko 2023-10-30T13:58:57.101 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-msm6242.ko 2023-10-30T13:58:57.101 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-palmas.ko 2023-10-30T13:58:57.101 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 46633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf2127.ko 2023-10-30T13:58:57.102 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf50633.ko 2023-10-30T13:58:57.102 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf85063.ko 2023-10-30T13:58:57.102 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf8523.ko 2023-10-30T13:58:57.102 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf8563.ko 2023-10-30T13:58:57.102 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf8583.ko 2023-10-30T13:58:57.103 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rc5t583.ko 2023-10-30T13:58:57.103 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rp5c01.ko 2023-10-30T13:58:57.103 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rs5c372.ko 2023-10-30T13:58:57.103 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-10-30T13:58:57.103 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rx8025.ko 2023-10-30T13:58:57.103 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rx8581.ko 2023-10-30T13:58:57.104 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-s35390a.ko 2023-10-30T13:58:57.104 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-10-30T13:58:57.104 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-tps6586x.ko 2023-10-30T13:58:57.104 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-tps65910.ko 2023-10-30T13:58:57.104 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-wm831x.ko 2023-10-30T13:58:57.104 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-wm8350.ko 2023-10-30T13:58:57.104 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-x1205.ko 2023-10-30T13:58:57.105 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ 2023-10-30T13:58:57.105 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 77617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/3w-9xxx.ko 2023-10-30T13:58:57.105 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 57665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/3w-sas.ko 2023-10-30T13:58:57.105 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 55169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/3w-xxxx.ko 2023-10-30T13:58:57.105 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 135185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/BusLogic.ko 2023-10-30T13:58:57.105 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/a100u2w.ko 2023-10-30T13:58:57.106 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aacraid/ 2023-10-30T13:58:57.106 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 272305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aacraid/aacraid.ko 2023-10-30T13:58:57.106 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 100617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/advansys.ko 2023-10-30T13:58:57.106 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic7xxx/ 2023-10-30T13:58:57.106 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 356265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-10-30T13:58:57.107 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 295801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-10-30T13:58:57.107 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic94xx/ 2023-10-30T13:58:57.107 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 189185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic94xx/aic94xx.ko 2023-10-30T13:58:57.107 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/am53c974.ko 2023-10-30T13:58:57.107 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/arcmsr/ 2023-10-30T13:58:57.107 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 106881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-10-30T13:58:57.108 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 38185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/atp870u.ko 2023-10-30T13:58:57.108 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/be2iscsi/ 2023-10-30T13:58:57.108 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 246985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-10-30T13:58:57.108 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bfa/ 2023-10-30T13:58:57.108 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 1171945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bfa/bfa.ko 2023-10-30T13:58:57.108 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2fc/ 2023-10-30T13:58:57.108 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 237273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-10-30T13:58:57.109 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2i/ 2023-10-30T13:58:57.109 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 115369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-10-30T13:58:57.109 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 40585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ch.ko 2023-10-30T13:58:57.109 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/csiostor/ 2023-10-30T13:58:57.110 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 308921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/csiostor/csiostor.ko 2023-10-30T13:58:57.110 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/ 2023-10-30T13:58:57.110 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb3i/ 2023-10-30T13:58:57.110 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 76145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-10-30T13:58:57.110 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb4i/ 2023-10-30T13:58:57.110 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 101449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-10-30T13:58:57.111 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 146689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-10-30T13:58:57.111 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 75553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/dc395x.ko 2023-10-30T13:58:57.111 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 38137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/dmx3191d.ko 2023-10-30T13:58:57.111 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/esas2r/ 2023-10-30T13:58:57.111 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 212425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/esas2r/esas2r.ko 2023-10-30T13:58:57.111 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 60505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/esp_scsi.ko 2023-10-30T13:58:57.112 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fcoe/ 2023-10-30T13:58:57.112 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 78209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fcoe/fcoe.ko 2023-10-30T13:58:57.112 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 158505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-10-30T13:58:57.112 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fdomain.ko 2023-10-30T13:58:57.112 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fnic/ 2023-10-30T13:58:57.112 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 263489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fnic/fnic.ko 2023-10-30T13:58:57.113 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 201817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/hpsa.ko 2023-10-30T13:58:57.113 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 51209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/hptiop.ko 2023-10-30T13:58:57.113 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 49041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/hv_storvsc.ko 2023-10-30T13:58:57.113 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 35337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/imm.ko 2023-10-30T13:58:57.113 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 44057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/initio.ko 2023-10-30T13:58:57.114 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 266537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ipr.ko 2023-10-30T13:58:57.114 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 101881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ips.ko 2023-10-30T13:58:57.114 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/isci/ 2023-10-30T13:58:57.114 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 317153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/isci/isci.ko 2023-10-30T13:58:57.114 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-10-30T13:58:57.114 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 52961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/iscsi_tcp.ko 2023-10-30T13:58:57.115 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libfc/ 2023-10-30T13:58:57.115 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 314865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libfc/libfc.ko 2023-10-30T13:58:57.115 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 145585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libiscsi.ko 2023-10-30T13:58:57.115 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 53521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libiscsi_tcp.ko 2023-10-30T13:58:57.115 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libsas/ 2023-10-30T13:58:57.115 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 216665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libsas/libsas.ko 2023-10-30T13:58:57.116 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/lpfc/ 2023-10-30T13:58:57.116 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 2197249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/lpfc/lpfc.ko 2023-10-30T13:58:57.116 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/ 2023-10-30T13:58:57.116 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 73569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-10-30T13:58:57.116 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-10-30T13:58:57.116 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 310617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-10-30T13:58:57.116 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 84721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid.ko 2023-10-30T13:58:57.117 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mpt3sas/ 2023-10-30T13:58:57.117 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 659777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-10-30T13:58:57.117 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mvsas/ 2023-10-30T13:58:57.117 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 119561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mvsas/mvsas.ko 2023-10-30T13:58:57.117 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 78129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mvumi.ko 2023-10-30T13:58:57.117 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/ 2023-10-30T13:58:57.118 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 81625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2023-10-30T13:58:57.118 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2023-10-30T13:58:57.118 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2023-10-30T13:58:57.118 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2023-10-30T13:58:57.118 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pm8001/ 2023-10-30T13:58:57.119 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 364105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-10-30T13:58:57.119 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 123617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pmcraid.ko 2023-10-30T13:58:57.119 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ppa.ko 2023-10-30T13:58:57.119 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 55361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla1280.ko 2023-10-30T13:58:57.119 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla2xxx/ 2023-10-30T13:58:57.119 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 1876625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-10-30T13:58:57.120 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 85289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-10-30T13:58:57.120 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla4xxx/ 2023-10-30T13:58:57.120 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 559033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-10-30T13:58:57.120 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qlogicfas408.ko 2023-10-30T13:58:57.120 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/raid_class.ko 2023-10-30T13:58:57.120 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 238825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_debug.ko 2023-10-30T13:58:57.121 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 173937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_fc.ko 2023-10-30T13:58:57.121 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 279313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-10-30T13:58:57.121 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 97729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_sas.ko 2023-10-30T13:58:57.121 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 73881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_spi.ko 2023-10-30T13:58:57.121 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 44993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_srp.ko 2023-10-30T13:58:57.121 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ses.ko 2023-10-30T13:58:57.122 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 115889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/st.ko 2023-10-30T13:58:57.122 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 46825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/stex.ko 2023-10-30T13:58:57.122 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/sym53c8xx_2/ 2023-10-30T13:58:57.122 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 162121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-10-30T13:58:57.122 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 42297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/virtio_scsi.ko 2023-10-30T13:58:57.122 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 50321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/vmw_pvscsi.ko 2023-10-30T13:58:57.122 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 40025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/wd719x.ko 2023-10-30T13:58:57.123 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 50025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/xen-scsifront.ko 2023-10-30T13:58:57.123 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ssb/ 2023-10-30T13:58:57.123 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 129441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ssb/ssb.ko 2023-10-30T13:58:57.123 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/ 2023-10-30T13:58:57.123 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/iscsi/ 2023-10-30T13:58:57.123 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 736601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-10-30T13:58:57.124 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/loopback/ 2023-10-30T13:58:57.124 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 49985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/loopback/tcm_loop.ko 2023-10-30T13:58:57.124 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 39713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_file.ko 2023-10-30T13:58:57.124 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 43025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_iblock.ko 2023-10-30T13:58:57.124 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 930321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_mod.ko 2023-10-30T13:58:57.124 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 44305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_pscsi.ko 2023-10-30T13:58:57.125 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 122521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_user.ko 2023-10-30T13:58:57.125 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/tcm_fc/ 2023-10-30T13:58:57.125 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 75593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-10-30T13:58:57.125 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/ 2023-10-30T13:58:57.125 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/ 2023-10-30T13:58:57.125 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/ 2023-10-30T13:58:57.126 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-10-30T13:58:57.126 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-10-30T13:58:57.126 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-10-30T13:58:57.126 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-10-30T13:58:57.126 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-10-30T13:58:57.126 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-10-30T13:58:57.127 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-10-30T13:58:57.127 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-10-30T13:58:57.127 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-10-30T13:58:57.127 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-10-30T13:58:57.127 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-10-30T13:58:57.127 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 37065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-10-30T13:58:57.128 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/intel_powerclamp.ko 2023-10-30T13:58:57.128 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-10-30T13:58:57.128 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-10-30T13:58:57.128 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-10-30T13:58:57.128 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/ 2023-10-30T13:58:57.129 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/ipwireless/ 2023-10-30T13:58:57.129 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 78425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-10-30T13:58:57.129 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 57865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/moxa.ko 2023-10-30T13:58:57.129 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 50953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/mxser.ko 2023-10-30T13:58:57.129 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 108601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/n_gsm.ko 2023-10-30T13:58:57.129 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 31961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/n_hdlc.ko 2023-10-30T13:58:57.130 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 67929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/nozomi.ko 2023-10-30T13:58:57.130 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/ 2023-10-30T13:58:57.130 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/8250/ 2023-10-30T13:58:57.130 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 35753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/8250/8250_dw.ko 2023-10-30T13:58:57.130 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 59497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-10-30T13:58:57.130 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/altera_jtaguart.ko 2023-10-30T13:58:57.131 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/altera_uart.ko 2023-10-30T13:58:57.131 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/arc_uart.ko 2023-10-30T13:58:57.131 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 72425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/fsl_lpuart.ko 2023-10-30T13:58:57.131 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/jsm/ 2023-10-30T13:58:57.131 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 103073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/jsm/jsm.ko 2023-10-30T13:58:57.132 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/rp2.ko 2023-10-30T13:58:57.132 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 52817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/sc16is7xx.ko 2023-10-30T13:58:57.132 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/ 2023-10-30T13:58:57.132 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/core/ 2023-10-30T13:58:57.132 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 531265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/core/ufshcd-core.ko 2023-10-30T13:58:57.133 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/host/ 2023-10-30T13:58:57.133 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-10-30T13:58:57.133 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2023-10-30T13:58:57.133 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/ 2023-10-30T13:58:57.133 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 44513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio.ko 2023-10-30T13:58:57.133 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_aec.ko 2023-10-30T13:58:57.134 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_cif.ko 2023-10-30T13:58:57.134 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_dmem_genirq.ko 2023-10-30T13:58:57.134 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_mf624.ko 2023-10-30T13:58:57.134 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_netx.ko 2023-10-30T13:58:57.134 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_pci_generic.ko 2023-10-30T13:58:57.135 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_pdrv_genirq.ko 2023-10-30T13:58:57.135 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_sercos3.ko 2023-10-30T13:58:57.135 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/ 2023-10-30T13:58:57.135 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/c67x00/ 2023-10-30T13:58:57.135 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 73297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/c67x00/c67x00.ko 2023-10-30T13:58:57.136 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ 2023-10-30T13:58:57.136 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 144377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc.ko 2023-10-30T13:58:57.136 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2023-10-30T13:58:57.136 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2023-10-30T13:58:57.136 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2023-10-30T13:58:57.137 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/ 2023-10-30T13:58:57.137 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 87089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/cdc-acm.ko 2023-10-30T13:58:57.137 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 51409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/cdc-wdm.ko 2023-10-30T13:58:57.137 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 47369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/usblp.ko 2023-10-30T13:58:57.137 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 89857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/usbtmc.ko 2023-10-30T13:58:57.137 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/common/ 2023-10-30T13:58:57.138 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/common/ulpi.ko 2023-10-30T13:58:57.138 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc2/ 2023-10-30T13:58:57.138 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc2/dwc2_pci.ko 2023-10-30T13:58:57.138 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/ 2023-10-30T13:58:57.138 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/dwc3-haps.ko 2023-10-30T13:58:57.139 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/dwc3-pci.ko 2023-10-30T13:58:57.139 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 444553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/dwc3.ko 2023-10-30T13:58:57.139 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/ 2023-10-30T13:58:57.139 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/ 2023-10-30T13:58:57.139 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 60177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/u_ether.ko 2023-10-30T13:58:57.140 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 51881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/u_serial.ko 2023-10-30T13:58:57.140 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2023-10-30T13:58:57.140 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 42449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2023-10-30T13:58:57.140 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2023-10-30T13:58:57.140 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2023-10-30T13:58:57.141 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 92489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2023-10-30T13:58:57.141 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 51049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2023-10-30T13:58:57.141 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 122081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2023-10-30T13:58:57.141 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 57617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2023-10-30T13:58:57.141 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2023-10-30T13:58:57.141 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 31265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2023-10-30T13:58:57.142 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 57729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2023-10-30T13:58:57.142 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 85609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2023-10-30T13:58:57.142 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2023-10-30T13:58:57.142 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 58113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2023-10-30T13:58:57.142 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 67337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2023-10-30T13:58:57.143 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/ 2023-10-30T13:58:57.143 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2023-10-30T13:58:57.143 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2023-10-30T13:58:57.143 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2023-10-30T13:58:57.143 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_ether.ko 2023-10-30T13:58:57.144 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2023-10-30T13:58:57.144 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_hid.ko 2023-10-30T13:58:57.144 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2023-10-30T13:58:57.144 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2023-10-30T13:58:57.144 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 32073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2023-10-30T13:58:57.144 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_printer.ko 2023-10-30T13:58:57.145 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_serial.ko 2023-10-30T13:58:57.145 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_zero.ko 2023-10-30T13:58:57.145 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 62185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2023-10-30T13:58:57.145 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2023-10-30T13:58:57.145 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 161225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/libcomposite.ko 2023-10-30T13:58:57.146 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/ 2023-10-30T13:58:57.146 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2023-10-30T13:58:57.146 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/bdc/ 2023-10-30T13:58:57.146 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 157681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2023-10-30T13:58:57.146 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/goku_udc.ko 2023-10-30T13:58:57.147 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 52721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/gr_udc.ko 2023-10-30T13:58:57.147 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 69705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2023-10-30T13:58:57.147 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 56033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/mv_udc.ko 2023-10-30T13:58:57.147 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 76057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/net2272.ko 2023-10-30T13:58:57.147 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 75073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/net2280.ko 2023-10-30T13:58:57.147 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 73169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/pch_udc.ko 2023-10-30T13:58:57.148 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 65809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2023-10-30T13:58:57.148 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 66097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2023-10-30T13:58:57.148 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 81505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2023-10-30T13:58:57.148 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 156073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/udc-core.ko 2023-10-30T13:58:57.148 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/ 2023-10-30T13:58:57.149 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/bcma-hcd.ko 2023-10-30T13:58:57.149 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 80113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/isp116x-hcd.ko 2023-10-30T13:58:57.149 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 87353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/oxu210hp-hcd.ko 2023-10-30T13:58:57.149 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 78873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/r8a66597-hcd.ko 2023-10-30T13:58:57.149 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 51329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/sl811-hcd.ko 2023-10-30T13:58:57.150 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/sl811_cs.ko 2023-10-30T13:58:57.150 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/ssb-hcd.ko 2023-10-30T13:58:57.150 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/xhci-plat-hcd.ko 2023-10-30T13:58:57.150 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/image/ 2023-10-30T13:58:57.150 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/image/mdc800.ko 2023-10-30T13:58:57.151 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/image/microtek.ko 2023-10-30T13:58:57.151 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ 2023-10-30T13:58:57.151 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 41721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/adutux.ko 2023-10-30T13:58:57.151 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/appledisplay.ko 2023-10-30T13:58:57.151 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/cypress_cy7c63.ko 2023-10-30T13:58:57.152 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/cytherm.ko 2023-10-30T13:58:57.152 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ehset.ko 2023-10-30T13:58:57.152 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/emi26.ko 2023-10-30T13:58:57.152 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/emi62.ko 2023-10-30T13:58:57.152 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ezusb.ko 2023-10-30T13:58:57.153 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/idmouse.ko 2023-10-30T13:58:57.153 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/iowarrior.ko 2023-10-30T13:58:57.153 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/isight_firmware.ko 2023-10-30T13:58:57.153 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ldusb.ko 2023-10-30T13:58:57.153 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/legousbtower.ko 2023-10-30T13:58:57.153 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/lvstest.ko 2023-10-30T13:58:57.154 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/sisusbvga/ 2023-10-30T13:58:57.154 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 64841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-10-30T13:58:57.154 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/trancevibrator.ko 2023-10-30T13:58:57.154 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usb3503.ko 2023-10-30T13:58:57.154 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usblcd.ko 2023-10-30T13:58:57.155 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usbsevseg.ko 2023-10-30T13:58:57.155 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 70873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usbtest.ko 2023-10-30T13:58:57.155 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 41969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/uss720.ko 2023-10-30T13:58:57.155 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/yurex.ko 2023-10-30T13:58:57.155 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/mon/ 2023-10-30T13:58:57.155 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 72953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/mon/usbmon.ko 2023-10-30T13:58:57.156 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/musb/ 2023-10-30T13:58:57.156 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 301545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/musb/musb_hdrc.ko 2023-10-30T13:58:57.156 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/ 2023-10-30T13:58:57.156 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-generic.ko 2023-10-30T13:58:57.156 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2023-10-30T13:58:57.156 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-isp1301.ko 2023-10-30T13:58:57.157 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-tahvo.ko 2023-10-30T13:58:57.157 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ 2023-10-30T13:58:57.157 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/aircable.ko 2023-10-30T13:58:57.157 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ark3116.ko 2023-10-30T13:58:57.157 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/belkin_sa.ko 2023-10-30T13:58:57.157 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 38105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ch341.ko 2023-10-30T13:58:57.158 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 70257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/cp210x.ko 2023-10-30T13:58:57.158 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 31057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/cyberjack.ko 2023-10-30T13:58:57.158 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 58217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/cypress_m8.ko 2023-10-30T13:58:57.158 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 49817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/digi_acceleport.ko 2023-10-30T13:58:57.158 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/empeg.ko 2023-10-30T13:58:57.159 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 44337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/f81232.ko 2023-10-30T13:58:57.159 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 175137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ftdi_sio.ko 2023-10-30T13:58:57.159 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 53113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/garmin_gps.ko 2023-10-30T13:58:57.159 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 137081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/io_edgeport.ko 2023-10-30T13:58:57.160 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 124833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/io_ti.ko 2023-10-30T13:58:57.160 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 67769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ipaq.ko 2023-10-30T13:58:57.160 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ipw.ko 2023-10-30T13:58:57.160 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ir-usb.ko 2023-10-30T13:58:57.160 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 58233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-10-30T13:58:57.160 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 107521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/keyspan.ko 2023-10-30T13:58:57.161 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 35617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/keyspan_pda.ko 2023-10-30T13:58:57.161 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/kl5kusb105.ko 2023-10-30T13:58:57.161 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/kobil_sct.ko 2023-10-30T13:58:57.161 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 32953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mct_u232.ko 2023-10-30T13:58:57.161 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/metro-usb.ko 2023-10-30T13:58:57.161 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 66449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mos7720.ko 2023-10-30T13:58:57.162 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 78217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mos7840.ko 2023-10-30T13:58:57.162 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 56337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mxuport.ko 2023-10-30T13:58:57.162 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/navman.ko 2023-10-30T13:58:57.162 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/omninet.ko 2023-10-30T13:58:57.162 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/opticon.ko 2023-10-30T13:58:57.163 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 190633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/option.ko 2023-10-30T13:58:57.163 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 39073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/oti6858.ko 2023-10-30T13:58:57.163 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 54353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/pl2303.ko 2023-10-30T13:58:57.163 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/qcaux.ko 2023-10-30T13:58:57.163 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 37825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/qcserial.ko 2023-10-30T13:58:57.163 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/quatech2.ko 2023-10-30T13:58:57.164 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/safe_serial.ko 2023-10-30T13:58:57.164 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 51625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/sierra.ko 2023-10-30T13:58:57.164 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/spcp8x5.ko 2023-10-30T13:58:57.164 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ssu100.ko 2023-10-30T13:58:57.164 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/symbolserial.ko 2023-10-30T13:58:57.164 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 61489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-10-30T13:58:57.165 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-10-30T13:58:57.165 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usb_debug.ko 2023-10-30T13:58:57.165 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usb_wwan.ko 2023-10-30T13:58:57.165 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 125609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usbserial.ko 2023-10-30T13:58:57.165 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/visor.ko 2023-10-30T13:58:57.165 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 37409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/whiteheat.ko 2023-10-30T13:58:57.166 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/wishbone-serial.ko 2023-10-30T13:58:57.166 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/xsens_mt.ko 2023-10-30T13:58:57.166 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ 2023-10-30T13:58:57.166 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 52913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/uas.ko 2023-10-30T13:58:57.166 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-alauda.ko 2023-10-30T13:58:57.166 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-cypress.ko 2023-10-30T13:58:57.167 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-datafab.ko 2023-10-30T13:58:57.167 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-10-30T13:58:57.167 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-freecom.ko 2023-10-30T13:58:57.167 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-isd200.ko 2023-10-30T13:58:57.167 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-10-30T13:58:57.168 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-karma.ko 2023-10-30T13:58:57.168 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-onetouch.ko 2023-10-30T13:58:57.168 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-realtek.ko 2023-10-30T13:58:57.168 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-sddr09.ko 2023-10-30T13:58:57.168 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-sddr55.ko 2023-10-30T13:58:57.168 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 35337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-usbat.ko 2023-10-30T13:58:57.169 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 179969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/usb-storage.ko 2023-10-30T13:58:57.169 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/ 2023-10-30T13:58:57.169 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 67481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/usbip-core.ko 2023-10-30T13:58:57.169 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 72177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/usbip-host.ko 2023-10-30T13:58:57.169 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 107001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-10-30T13:58:57.169 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/ 2023-10-30T13:58:57.170 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/pci/ 2023-10-30T13:58:57.170 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 154297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-10-30T13:58:57.170 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/pci/vfio-pci.ko 2023-10-30T13:58:57.170 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 108737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/vfio.ko 2023-10-30T13:58:57.170 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 77609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-10-30T13:58:57.170 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/ 2023-10-30T13:58:57.171 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 113873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost.ko 2023-10-30T13:58:57.171 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost_iotlb.ko 2023-10-30T13:58:57.171 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 58089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost_net.ko 2023-10-30T13:58:57.171 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 89249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost_scsi.ko 2023-10-30T13:58:57.171 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/ 2023-10-30T13:58:57.171 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/ 2023-10-30T13:58:57.172 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/88pm860x_bl.ko 2023-10-30T13:58:57.172 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/aat2870_bl.ko 2023-10-30T13:58:57.172 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/adp5520_bl.ko 2023-10-30T13:58:57.172 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 32609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/adp8860_bl.ko 2023-10-30T13:58:57.172 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 39993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/adp8870_bl.ko 2023-10-30T13:58:57.172 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/apple_bl.ko 2023-10-30T13:58:57.173 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/as3711_bl.ko 2023-10-30T13:58:57.173 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/bd6107.ko 2023-10-30T13:58:57.173 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/da903x_bl.ko 2023-10-30T13:58:57.173 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/da9052_bl.ko 2023-10-30T13:58:57.173 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/gpio_backlight.ko 2023-10-30T13:58:57.173 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/kb3886_bl.ko 2023-10-30T13:58:57.174 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lcd.ko 2023-10-30T13:58:57.174 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lm3533_bl.ko 2023-10-30T13:58:57.174 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lm3639_bl.ko 2023-10-30T13:58:57.174 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lv5207lp.ko 2023-10-30T13:58:57.174 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/max8925_bl.ko 2023-10-30T13:58:57.174 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/pandora_bl.ko 2023-10-30T13:58:57.175 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/pcf50633-backlight.ko 2023-10-30T13:58:57.175 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/platform_lcd.ko 2023-10-30T13:58:57.175 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/wm831x_bl.ko 2023-10-30T13:58:57.175 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/ 2023-10-30T13:58:57.175 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/ 2023-10-30T13:58:57.175 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/ds2482.ko 2023-10-30T13:58:57.176 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 38985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/ds2490.ko 2023-10-30T13:58:57.176 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/matrox_w1.ko 2023-10-30T13:58:57.176 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/w1-gpio.ko 2023-10-30T13:58:57.176 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/ 2023-10-30T13:58:57.176 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-10-30T13:58:57.176 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-10-30T13:58:57.177 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-10-30T13:58:57.177 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-10-30T13:58:57.177 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-10-30T13:58:57.177 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-10-30T13:58:57.177 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-10-30T13:58:57.177 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-10-30T13:58:57.178 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-10-30T13:58:57.178 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_smem.ko 2023-10-30T13:58:57.178 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 56425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_therm.ko 2023-10-30T13:58:57.178 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 102673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/wire.ko 2023-10-30T13:58:57.178 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ 2023-10-30T13:58:57.179 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/acquirewdt.ko 2023-10-30T13:58:57.179 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/advantechwdt.ko 2023-10-30T13:58:57.179 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/alim1535_wdt.ko 2023-10-30T13:58:57.179 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/alim7101_wdt.ko 2023-10-30T13:58:57.179 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/cpu5wdt.ko 2023-10-30T13:58:57.179 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/da9052_wdt.ko 2023-10-30T13:58:57.180 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/da9055_wdt.ko 2023-10-30T13:58:57.180 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/da9063_wdt.ko 2023-10-30T13:58:57.180 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/dw_wdt.ko 2023-10-30T13:58:57.180 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/eurotechwdt.ko 2023-10-30T13:58:57.180 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/f71808e_wdt.ko 2023-10-30T13:58:57.180 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/hpwdt.ko 2023-10-30T13:58:57.181 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/i6300esb.ko 2023-10-30T13:58:57.181 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-10-30T13:58:57.181 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/iTCO_wdt.ko 2023-10-30T13:58:57.181 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ib700wdt.ko 2023-10-30T13:58:57.181 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ibmasr.ko 2023-10-30T13:58:57.181 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-10-30T13:58:57.182 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/it8712f_wdt.ko 2023-10-30T13:58:57.182 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/it87_wdt.ko 2023-10-30T13:58:57.182 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/kempld_wdt.ko 2023-10-30T13:58:57.182 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/machzwd.ko 2023-10-30T13:58:57.182 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/mena21_wdt.ko 2023-10-30T13:58:57.182 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/menf21bmc_wdt.ko 2023-10-30T13:58:57.183 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/nv_tco.ko 2023-10-30T13:58:57.183 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/of_xilinx_wdt.ko 2023-10-30T13:58:57.183 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/pc87413_wdt.ko 2023-10-30T13:58:57.183 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 38657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/pcwd_pci.ko 2023-10-30T13:58:57.183 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 31825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/pcwd_usb.ko 2023-10-30T13:58:57.183 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/retu_wdt.ko 2023-10-30T13:58:57.184 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sbc60xxwdt.ko 2023-10-30T13:58:57.184 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sbc_epx_c3.ko 2023-10-30T13:58:57.184 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-10-30T13:58:57.184 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sc1200wdt.ko 2023-10-30T13:58:57.184 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sch311x_wdt.ko 2023-10-30T13:58:57.184 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/smsc37b787_wdt.ko 2023-10-30T13:58:57.184 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/softdog.ko 2023-10-30T13:58:57.185 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sp5100_tco.ko 2023-10-30T13:58:57.185 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/twl4030_wdt.ko 2023-10-30T13:58:57.185 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/via_wdt.ko 2023-10-30T13:58:57.185 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-10-30T13:58:57.185 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/w83877f_wdt.ko 2023-10-30T13:58:57.185 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/w83977f_wdt.ko 2023-10-30T13:58:57.186 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wafer5823wdt.ko 2023-10-30T13:58:57.186 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wdt_pci.ko 2023-10-30T13:58:57.186 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wm831x_wdt.ko 2023-10-30T13:58:57.186 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wm8350_wdt.ko 2023-10-30T13:58:57.186 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/xen_wdt.ko 2023-10-30T13:58:57.186 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/ 2023-10-30T13:58:57.187 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-evtchn.ko 2023-10-30T13:58:57.187 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-gntalloc.ko 2023-10-30T13:58:57.187 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 47865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-gntdev.ko 2023-10-30T13:58:57.187 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-pciback/ 2023-10-30T13:58:57.187 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 176441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-10-30T13:58:57.187 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 40121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-privcmd.ko 2023-10-30T13:58:57.188 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 71249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-scsiback.ko 2023-10-30T13:58:57.188 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xenfs/ 2023-10-30T13:58:57.188 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xenfs/xenfs.ko 2023-10-30T13:58:57.188 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ 2023-10-30T13:58:57.188 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/9p/ 2023-10-30T13:58:57.188 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 166425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/9p/9p.ko 2023-10-30T13:58:57.189 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/adfs/ 2023-10-30T13:58:57.189 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 61073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/adfs/adfs.ko 2023-10-30T13:58:57.189 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/affs/ 2023-10-30T13:58:57.189 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 163593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/affs/affs.ko 2023-10-30T13:58:57.189 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/afs/ 2023-10-30T13:58:57.189 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 932817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/afs/kafs.ko 2023-10-30T13:58:57.190 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/autofs/ 2023-10-30T13:58:57.190 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 101057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/autofs/autofs4.ko 2023-10-30T13:58:57.190 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/befs/ 2023-10-30T13:58:57.190 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 64545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/befs/befs.ko 2023-10-30T13:58:57.190 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/bfs/ 2023-10-30T13:58:57.191 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 41257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/bfs/bfs.ko 2023-10-30T13:58:57.191 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/binfmt_misc.ko 2023-10-30T13:58:57.191 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/btrfs/ 2023-10-30T13:58:57.191 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 3402585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/btrfs/btrfs.ko 2023-10-30T13:58:57.191 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cachefiles/ 2023-10-30T13:58:57.191 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 306073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cachefiles/cachefiles.ko 2023-10-30T13:58:57.192 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ceph/ 2023-10-30T13:58:57.192 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 1323945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ceph/ceph.ko 2023-10-30T13:58:57.192 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/coda/ 2023-10-30T13:58:57.192 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 100801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/coda/coda.ko 2023-10-30T13:58:57.192 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/configfs/ 2023-10-30T13:58:57.192 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 108393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/configfs/configfs.ko 2023-10-30T13:58:57.193 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cramfs/ 2023-10-30T13:58:57.193 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 29625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cramfs/cramfs.ko 2023-10-30T13:58:57.193 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/dlm/ 2023-10-30T13:58:57.193 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 642577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/dlm/dlm.ko 2023-10-30T13:58:57.193 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/efs/ 2023-10-30T13:58:57.193 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/efs/efs.ko 2023-10-30T13:58:57.194 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/f2fs/ 2023-10-30T13:58:57.194 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 1505025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/f2fs/f2fs.ko 2023-10-30T13:58:57.194 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fat/ 2023-10-30T13:58:57.194 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fat/msdos.ko 2023-10-30T13:58:57.194 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/freevxfs/ 2023-10-30T13:58:57.195 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 42545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/freevxfs/freevxfs.ko 2023-10-30T13:58:57.195 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fscache/ 2023-10-30T13:58:57.195 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 226481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fscache/fscache.ko 2023-10-30T13:58:57.195 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fuse/ 2023-10-30T13:58:57.195 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fuse/cuse.ko 2023-10-30T13:58:57.195 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/gfs2/ 2023-10-30T13:58:57.196 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 1231529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/gfs2/gfs2.ko 2023-10-30T13:58:57.196 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfs/ 2023-10-30T13:58:57.196 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 141209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfs/hfs.ko 2023-10-30T13:58:57.196 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfsplus/ 2023-10-30T13:58:57.196 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 222817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfsplus/hfsplus.ko 2023-10-30T13:58:57.197 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hpfs/ 2023-10-30T13:58:57.197 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 192329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hpfs/hpfs.ko 2023-10-30T13:58:57.197 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/isofs/ 2023-10-30T13:58:57.197 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 74065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/isofs/isofs.ko 2023-10-30T13:58:57.197 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/jfs/ 2023-10-30T13:58:57.197 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 419497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/jfs/jfs.ko 2023-10-30T13:58:57.198 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/lockd/ 2023-10-30T13:58:57.198 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 280313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/lockd/lockd.ko 2023-10-30T13:58:57.198 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/minix/ 2023-10-30T13:58:57.198 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 89193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/minix/minix.ko 2023-10-30T13:58:57.198 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/netfs/ 2023-10-30T13:58:57.199 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 124649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/netfs/netfs.ko 2023-10-30T13:58:57.199 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/ 2023-10-30T13:58:57.199 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/blocklayout/ 2023-10-30T13:58:57.199 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 81449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-10-30T13:58:57.199 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/filelayout/ 2023-10-30T13:58:57.200 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 59449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-10-30T13:58:57.200 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/flexfilelayout/ 2023-10-30T13:58:57.200 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 117513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-10-30T13:58:57.200 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 1055649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfs.ko 2023-10-30T13:58:57.200 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 66441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfsv2.ko 2023-10-30T13:58:57.200 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 120177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfsv3.ko 2023-10-30T13:58:57.201 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 2041689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfsv4.ko 2023-10-30T13:58:57.201 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs_common/ 2023-10-30T13:58:57.201 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs_common/grace.ko 2023-10-30T13:58:57.201 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs_common/nfs_acl.ko 2023-10-30T13:58:57.201 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfsd/ 2023-10-30T13:58:57.202 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 1516809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfsd/nfsd.ko 2023-10-30T13:58:57.202 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nilfs2/ 2023-10-30T13:58:57.202 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 592833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nilfs2/nilfs2.ko 2023-10-30T13:58:57.202 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/ 2023-10-30T13:58:57.202 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-celtic.ko 2023-10-30T13:58:57.203 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-centeuro.ko 2023-10-30T13:58:57.203 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-croatian.ko 2023-10-30T13:58:57.203 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-cyrillic.ko 2023-10-30T13:58:57.203 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-gaelic.ko 2023-10-30T13:58:57.203 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-greek.ko 2023-10-30T13:58:57.203 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-iceland.ko 2023-10-30T13:58:57.204 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-inuit.ko 2023-10-30T13:58:57.204 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-roman.ko 2023-10-30T13:58:57.204 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-romanian.ko 2023-10-30T13:58:57.204 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-turkish.ko 2023-10-30T13:58:57.204 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_ascii.ko 2023-10-30T13:58:57.205 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp1250.ko 2023-10-30T13:58:57.205 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp1251.ko 2023-10-30T13:58:57.205 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp1255.ko 2023-10-30T13:58:57.205 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp737.ko 2023-10-30T13:58:57.205 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp775.ko 2023-10-30T13:58:57.205 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp850.ko 2023-10-30T13:58:57.206 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp852.ko 2023-10-30T13:58:57.206 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp855.ko 2023-10-30T13:58:57.206 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp857.ko 2023-10-30T13:58:57.206 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp860.ko 2023-10-30T13:58:57.206 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp861.ko 2023-10-30T13:58:57.207 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp862.ko 2023-10-30T13:58:57.207 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp863.ko 2023-10-30T13:58:57.207 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp864.ko 2023-10-30T13:58:57.207 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp865.ko 2023-10-30T13:58:57.207 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp866.ko 2023-10-30T13:58:57.207 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp869.ko 2023-10-30T13:58:57.208 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp874.ko 2023-10-30T13:58:57.208 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 94137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp932.ko 2023-10-30T13:58:57.208 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 142673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp936.ko 2023-10-30T13:58:57.208 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 165745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp949.ko 2023-10-30T13:58:57.208 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 118121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp950.ko 2023-10-30T13:58:57.209 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_euc-jp.ko 2023-10-30T13:58:57.209 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-1.ko 2023-10-30T13:58:57.209 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-13.ko 2023-10-30T13:58:57.209 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-14.ko 2023-10-30T13:58:57.209 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-15.ko 2023-10-30T13:58:57.209 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-2.ko 2023-10-30T13:58:57.210 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-3.ko 2023-10-30T13:58:57.210 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-4.ko 2023-10-30T13:58:57.210 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-5.ko 2023-10-30T13:58:57.210 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-6.ko 2023-10-30T13:58:57.210 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-7.ko 2023-10-30T13:58:57.211 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-9.ko 2023-10-30T13:58:57.211 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_koi8-r.ko 2023-10-30T13:58:57.211 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_koi8-ru.ko 2023-10-30T13:58:57.211 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_koi8-u.ko 2023-10-30T13:58:57.211 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_ucs2_utils.ko 2023-10-30T13:58:57.211 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_utf8.ko 2023-10-30T13:58:57.212 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ntfs/ 2023-10-30T13:58:57.212 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 219417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ntfs/ntfs.ko 2023-10-30T13:58:57.212 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ 2023-10-30T13:58:57.212 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/cluster/ 2023-10-30T13:58:57.212 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 244409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-10-30T13:58:57.212 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlm/ 2023-10-30T13:58:57.213 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 440737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-10-30T13:58:57.213 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlmfs/ 2023-10-30T13:58:57.213 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 54049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-10-30T13:58:57.213 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 2592433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2.ko 2023-10-30T13:58:57.213 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-10-30T13:58:57.213 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 31729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-10-30T13:58:57.214 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 35897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-10-30T13:58:57.214 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/omfs/ 2023-10-30T13:58:57.214 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 53433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/omfs/omfs.ko 2023-10-30T13:58:57.214 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/overlayfs/ 2023-10-30T13:58:57.214 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 352153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/overlayfs/overlay.ko 2023-10-30T13:58:57.214 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/pstore/ 2023-10-30T13:58:57.215 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 54809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/pstore/ramoops.ko 2023-10-30T13:58:57.215 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx4/ 2023-10-30T13:58:57.215 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx4/qnx4.ko 2023-10-30T13:58:57.215 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx6/ 2023-10-30T13:58:57.215 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 46689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx6/qnx6.ko 2023-10-30T13:58:57.215 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/ 2023-10-30T13:58:57.216 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/quota_tree.ko 2023-10-30T13:58:57.216 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/quota_v1.ko 2023-10-30T13:58:57.216 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/quota_v2.ko 2023-10-30T13:58:57.216 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/reiserfs/ 2023-10-30T13:58:57.216 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 532673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/reiserfs/reiserfs.ko 2023-10-30T13:58:57.216 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/romfs/ 2023-10-30T13:58:57.217 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/romfs/romfs.ko 2023-10-30T13:58:57.217 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/ 2023-10-30T13:58:57.217 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/client/ 2023-10-30T13:58:57.217 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 2668121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/client/cifs.ko 2023-10-30T13:58:57.217 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/common/ 2023-10-30T13:58:57.217 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/common/cifs_arc4.ko 2023-10-30T13:58:57.218 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/common/cifs_md4.ko 2023-10-30T13:58:57.218 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/squashfs/ 2023-10-30T13:58:57.218 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 134641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/squashfs/squashfs.ko 2023-10-30T13:58:57.218 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/sysv/ 2023-10-30T13:58:57.218 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 90721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/sysv/sysv.ko 2023-10-30T13:58:57.218 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/udf/ 2023-10-30T13:58:57.219 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 247737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/udf/udf.ko 2023-10-30T13:58:57.219 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ufs/ 2023-10-30T13:58:57.219 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 162497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ufs/ufs.ko 2023-10-30T13:58:57.219 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/xfs/ 2023-10-30T13:58:57.219 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 4010729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/xfs/xfs.ko 2023-10-30T13:58:57.219 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/ 2023-10-30T13:58:57.220 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/locking/ 2023-10-30T13:58:57.220 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 75489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/locking/locktorture.ko 2023-10-30T13:58:57.220 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/time/ 2023-10-30T13:58:57.220 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/time/test_udelay.ko 2023-10-30T13:58:57.220 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 60553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/torture.ko 2023-10-30T13:58:57.220 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ 2023-10-30T13:58:57.221 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 6833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crc-itu-t.ko 2023-10-30T13:58:57.221 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 6529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crc7.ko 2023-10-30T13:58:57.221 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crc8.ko 2023-10-30T13:58:57.221 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crypto/ 2023-10-30T13:58:57.221 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crypto/libarc4.ko 2023-10-30T13:58:57.221 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 37081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crypto/libdes.ko 2023-10-30T13:58:57.222 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/interval_tree_test.ko 2023-10-30T13:58:57.222 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/libcrc32c.ko 2023-10-30T13:58:57.222 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lru_cache.ko 2023-10-30T13:58:57.222 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lz4/ 2023-10-30T13:58:57.222 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lz4/lz4_compress.ko 2023-10-30T13:58:57.223 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lz4/lz4hc_compress.ko 2023-10-30T13:58:57.223 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/math/ 2023-10-30T13:58:57.223 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 6617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/math/cordic.ko 2023-10-30T13:58:57.223 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/memory-notifier-error-inject.ko 2023-10-30T13:58:57.223 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/notifier-error-inject.ko 2023-10-30T13:58:57.223 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/percpu_test.ko 2023-10-30T13:58:57.223 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/pm-notifier-error-inject.ko 2023-10-30T13:58:57.224 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/raid6/ 2023-10-30T13:58:57.224 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 143433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/raid6/raid6_pq.ko 2023-10-30T13:58:57.224 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/rbtree_test.ko 2023-10-30T13:58:57.224 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/reed_solomon/ 2023-10-30T13:58:57.224 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/reed_solomon/reed_solomon.ko 2023-10-30T13:58:57.224 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 38841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test-kstrtox.ko 2023-10-30T13:58:57.225 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test-string_helpers.ko 2023-10-30T13:58:57.225 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 4760425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_bpf.ko 2023-10-30T13:58:57.225 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 49089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_firmware.ko 2023-10-30T13:58:57.225 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 6369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_module.ko 2023-10-30T13:58:57.225 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_user_copy.ko 2023-10-30T13:58:57.225 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ts_bm.ko 2023-10-30T13:58:57.226 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ts_fsm.ko 2023-10-30T13:58:57.226 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ts_kmp.ko 2023-10-30T13:58:57.226 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/xz/ 2023-10-30T13:58:57.226 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/xz/xz_dec_test.ko 2023-10-30T13:58:57.226 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/mm/ 2023-10-30T13:58:57.227 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/mm/hwpoison-inject.ko 2023-10-30T13:58:57.227 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ 2023-10-30T13:58:57.227 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/ 2023-10-30T13:58:57.227 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 55737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/6lowpan.ko 2023-10-30T13:58:57.227 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_dest.ko 2023-10-30T13:58:57.228 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_fragment.ko 2023-10-30T13:58:57.228 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_hop.ko 2023-10-30T13:58:57.228 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 6985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_ipv6.ko 2023-10-30T13:58:57.228 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_mobility.ko 2023-10-30T13:58:57.228 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_routing.ko 2023-10-30T13:58:57.229 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_udp.ko 2023-10-30T13:58:57.229 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/ 2023-10-30T13:58:57.229 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/garp.ko 2023-10-30T13:58:57.229 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/mrp.ko 2023-10-30T13:58:57.229 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/p8022.ko 2023-10-30T13:58:57.229 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/psnap.ko 2023-10-30T13:58:57.230 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/stp.ko 2023-10-30T13:58:57.230 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/8021q/ 2023-10-30T13:58:57.230 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 81737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/8021q/8021q.ko 2023-10-30T13:58:57.230 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/ 2023-10-30T13:58:57.231 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 198337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet.ko 2023-10-30T13:58:57.231 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 35977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet_fd.ko 2023-10-30T13:58:57.231 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet_rdma.ko 2023-10-30T13:58:57.231 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 32337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet_virtio.ko 2023-10-30T13:58:57.231 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/batman-adv/ 2023-10-30T13:58:57.232 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 649297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/batman-adv/batman-adv.ko 2023-10-30T13:58:57.232 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/ 2023-10-30T13:58:57.232 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 63809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/br_netfilter.ko 2023-10-30T13:58:57.232 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 777689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/bridge.ko 2023-10-30T13:58:57.232 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ 2023-10-30T13:58:57.233 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-10-30T13:58:57.233 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_among.ko 2023-10-30T13:58:57.233 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_arp.ko 2023-10-30T13:58:57.233 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-10-30T13:58:57.233 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-10-30T13:58:57.234 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_ip.ko 2023-10-30T13:58:57.234 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-10-30T13:58:57.234 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_limit.ko 2023-10-30T13:58:57.234 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_log.ko 2023-10-30T13:58:57.234 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_mark.ko 2023-10-30T13:58:57.235 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-10-30T13:58:57.235 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-10-30T13:58:57.235 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-10-30T13:58:57.235 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-10-30T13:58:57.235 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_snat.ko 2023-10-30T13:58:57.235 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_stp.ko 2023-10-30T13:58:57.236 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-10-30T13:58:57.236 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-10-30T13:58:57.236 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-10-30T13:58:57.236 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-10-30T13:58:57.236 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 55929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtables.ko 2023-10-30T13:58:57.237 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-10-30T13:58:57.237 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-10-30T13:58:57.237 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ceph/ 2023-10-30T13:58:57.237 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 1047497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ceph/libceph.ko 2023-10-30T13:58:57.237 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/core/ 2023-10-30T13:58:57.238 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 125697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/core/pktgen.ko 2023-10-30T13:58:57.238 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/ 2023-10-30T13:58:57.238 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 194209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp.ko 2023-10-30T13:58:57.238 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp_diag.ko 2023-10-30T13:58:57.238 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 57881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp_ipv4.ko 2023-10-30T13:58:57.239 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 60337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp_ipv6.ko 2023-10-30T13:58:57.239 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/ 2023-10-30T13:58:57.239 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 390473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/dsa_core.ko 2023-10-30T13:58:57.239 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_brcm.ko 2023-10-30T13:58:57.239 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_dsa.ko 2023-10-30T13:58:57.240 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_none.ko 2023-10-30T13:58:57.240 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_trailer.ko 2023-10-30T13:58:57.240 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/hsr/ 2023-10-30T13:58:57.240 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 99729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/hsr/hsr.ko 2023-10-30T13:58:57.240 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/ 2023-10-30T13:58:57.240 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/6lowpan/ 2023-10-30T13:58:57.241 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 45233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-10-30T13:58:57.241 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 254625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/ieee802154.ko 2023-10-30T13:58:57.241 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 47289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/ieee802154_socket.ko 2023-10-30T13:58:57.241 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ 2023-10-30T13:58:57.241 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ah4.ko 2023-10-30T13:58:57.241 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 38905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/esp4.ko 2023-10-30T13:58:57.242 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 48401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/fou.ko 2023-10-30T13:58:57.242 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/gre.ko 2023-10-30T13:58:57.242 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 48657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/inet_diag.ko 2023-10-30T13:58:57.242 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 60633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ip_gre.ko 2023-10-30T13:58:57.242 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 61225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ip_tunnel.ko 2023-10-30T13:58:57.242 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ip_vti.ko 2023-10-30T13:58:57.243 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ipcomp.ko 2023-10-30T13:58:57.243 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ipip.ko 2023-10-30T13:58:57.243 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ 2023-10-30T13:58:57.243 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 37641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/arp_tables.ko 2023-10-30T13:58:57.243 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-10-30T13:58:57.244 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-10-30T13:58:57.244 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 41673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ip_tables.ko 2023-10-30T13:58:57.244 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-10-30T13:58:57.244 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2023-10-30T13:58:57.244 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-10-30T13:58:57.244 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-10-30T13:58:57.244 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-10-30T13:58:57.245 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_filter.ko 2023-10-30T13:58:57.245 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-10-30T13:58:57.245 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-10-30T13:58:57.245 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-10-30T13:58:57.245 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_security.ko 2023-10-30T13:58:57.245 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-10-30T13:58:57.246 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-10-30T13:58:57.246 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-10-30T13:58:57.246 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-10-30T13:58:57.246 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-10-30T13:58:57.246 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2023-10-30T13:58:57.247 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-10-30T13:58:57.247 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-10-30T13:58:57.247 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-10-30T13:58:57.247 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-10-30T13:58:57.247 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-10-30T13:58:57.247 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_bic.ko 2023-10-30T13:58:57.248 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_dctcp.ko 2023-10-30T13:58:57.248 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_diag.ko 2023-10-30T13:58:57.248 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_highspeed.ko 2023-10-30T13:58:57.248 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_htcp.ko 2023-10-30T13:58:57.248 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_hybla.ko 2023-10-30T13:58:57.248 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_illinois.ko 2023-10-30T13:58:57.249 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_lp.ko 2023-10-30T13:58:57.249 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_scalable.ko 2023-10-30T13:58:57.249 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_vegas.ko 2023-10-30T13:58:57.249 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_veno.ko 2023-10-30T13:58:57.249 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_westwood.ko 2023-10-30T13:58:57.249 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_yeah.ko 2023-10-30T13:58:57.250 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tunnel4.ko 2023-10-30T13:58:57.250 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/udp_diag.ko 2023-10-30T13:58:57.250 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 41665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/udp_tunnel.ko 2023-10-30T13:58:57.250 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/xfrm4_tunnel.ko 2023-10-30T13:58:57.250 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ 2023-10-30T13:58:57.250 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ah6.ko 2023-10-30T13:58:57.251 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 39537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/esp6.ko 2023-10-30T13:58:57.251 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/fou6.ko 2023-10-30T13:58:57.251 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 80897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_gre.ko 2023-10-30T13:58:57.251 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 93577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_tunnel.ko 2023-10-30T13:58:57.251 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-10-30T13:58:57.252 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 53273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_vti.ko 2023-10-30T13:58:57.252 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ipcomp6.ko 2023-10-30T13:58:57.252 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/mip6.ko 2023-10-30T13:58:57.252 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ 2023-10-30T13:58:57.252 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 42169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-10-30T13:58:57.253 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2023-10-30T13:58:57.253 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-10-30T13:58:57.253 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-10-30T13:58:57.253 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-10-30T13:58:57.253 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-10-30T13:58:57.253 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-10-30T13:58:57.254 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-10-30T13:58:57.254 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-10-30T13:58:57.254 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-10-30T13:58:57.254 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-10-30T13:58:57.254 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-10-30T13:58:57.255 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-10-30T13:58:57.255 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-10-30T13:58:57.255 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-10-30T13:58:57.255 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-10-30T13:58:57.255 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-10-30T13:58:57.255 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 39113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-10-30T13:58:57.256 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-10-30T13:58:57.256 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-10-30T13:58:57.256 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-10-30T13:58:57.256 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-10-30T13:58:57.256 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-10-30T13:58:57.256 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-10-30T13:58:57.256 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-10-30T13:58:57.257 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 74073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/sit.ko 2023-10-30T13:58:57.257 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/tunnel6.ko 2023-10-30T13:58:57.257 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/xfrm6_tunnel.ko 2023-10-30T13:58:57.257 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/key/ 2023-10-30T13:58:57.257 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 82833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/key/af_key.ko 2023-10-30T13:58:57.257 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/ 2023-10-30T13:58:57.258 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 125281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_core.ko 2023-10-30T13:58:57.258 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_debugfs.ko 2023-10-30T13:58:57.258 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 25953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_eth.ko 2023-10-30T13:58:57.258 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 35569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_ip.ko 2023-10-30T13:58:57.258 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_ip6.ko 2023-10-30T13:58:57.259 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_netlink.ko 2023-10-30T13:58:57.259 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 53241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_ppp.ko 2023-10-30T13:58:57.259 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/lapb/ 2023-10-30T13:58:57.259 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 47977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/lapb/lapb.ko 2023-10-30T13:58:57.259 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/llc/ 2023-10-30T13:58:57.260 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/llc/llc.ko 2023-10-30T13:58:57.260 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 267777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/llc/llc2.ko 2023-10-30T13:58:57.260 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mac802154/ 2023-10-30T13:58:57.260 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 290417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mac802154/mac802154.ko 2023-10-30T13:58:57.260 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mpls/ 2023-10-30T13:58:57.260 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mpls/mpls_gso.ko 2023-10-30T13:58:57.260 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ 2023-10-30T13:58:57.261 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ 2023-10-30T13:58:57.261 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 119585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set.ko 2023-10-30T13:58:57.261 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-10-30T13:58:57.261 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-10-30T13:58:57.261 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-10-30T13:58:57.261 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 100297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-10-30T13:58:57.262 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 101441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-10-30T13:58:57.262 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 102817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-10-30T13:58:57.262 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 103481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-10-30T13:58:57.262 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 112849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-10-30T13:58:57.262 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 60329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-10-30T13:58:57.263 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 109593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-10-30T13:58:57.263 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 113129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-10-30T13:58:57.263 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 112585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-10-30T13:58:57.263 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 111585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-10-30T13:58:57.263 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 114465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-10-30T13:58:57.263 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-10-30T13:58:57.264 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ 2023-10-30T13:58:57.264 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 411729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs.ko 2023-10-30T13:58:57.264 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-10-30T13:58:57.264 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-10-30T13:58:57.264 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-10-30T13:58:57.265 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-10-30T13:58:57.265 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-10-30T13:58:57.265 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-10-30T13:58:57.265 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-10-30T13:58:57.265 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-10-30T13:58:57.265 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-10-30T13:58:57.266 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-10-30T13:58:57.266 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-10-30T13:58:57.266 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-10-30T13:58:57.266 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-10-30T13:58:57.266 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 49257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conncount.ko 2023-10-30T13:58:57.266 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 408257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack.ko 2023-10-30T13:58:57.267 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-10-30T13:58:57.267 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-10-30T13:58:57.267 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-10-30T13:58:57.267 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 118145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_h323.ko 2023-10-30T13:58:57.267 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_irc.ko 2023-10-30T13:58:57.268 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-10-30T13:58:57.268 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 107297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-10-30T13:58:57.268 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-10-30T13:58:57.268 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_sane.ko 2023-10-30T13:58:57.268 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 66641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_sip.ko 2023-10-30T13:58:57.268 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-10-30T13:58:57.269 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-10-30T13:58:57.269 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_dup_netdev.ko 2023-10-30T13:58:57.269 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 89473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_flow_table.ko 2023-10-30T13:58:57.269 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_flow_table_inet.ko 2023-10-30T13:58:57.269 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 37129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_log_syslog.ko 2023-10-30T13:58:57.269 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 95417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat.ko 2023-10-30T13:58:57.270 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_amanda.ko 2023-10-30T13:58:57.270 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_ftp.ko 2023-10-30T13:58:57.270 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_irc.ko 2023-10-30T13:58:57.270 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_sip.ko 2023-10-30T13:58:57.270 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_tftp.ko 2023-10-30T13:58:57.271 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 44065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_synproxy_core.ko 2023-10-30T13:58:57.271 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 615313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_tables.ko 2023-10-30T13:58:57.271 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 42561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink.ko 2023-10-30T13:58:57.271 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_acct.ko 2023-10-30T13:58:57.271 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_cttimeout.ko 2023-10-30T13:58:57.271 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 39785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_log.ko 2023-10-30T13:58:57.272 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_osf.ko 2023-10-30T13:58:57.272 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 49377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_queue.ko 2023-10-30T13:58:57.272 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_chain_nat.ko 2023-10-30T13:58:57.272 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_compat.ko 2023-10-30T13:58:57.272 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 40721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_ct.ko 2023-10-30T13:58:57.272 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_dup_netdev.ko 2023-10-30T13:58:57.273 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_fib.ko 2023-10-30T13:58:57.273 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_fwd_netdev.ko 2023-10-30T13:58:57.273 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_hash.ko 2023-10-30T13:58:57.273 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_limit.ko 2023-10-30T13:58:57.273 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_log.ko 2023-10-30T13:58:57.273 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_masq.ko 2023-10-30T13:58:57.274 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_nat.ko 2023-10-30T13:58:57.274 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_numgen.ko 2023-10-30T13:58:57.274 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_osf.ko 2023-10-30T13:58:57.274 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_queue.ko 2023-10-30T13:58:57.275 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_redir.ko 2023-10-30T13:58:57.275 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_reject.ko 2023-10-30T13:58:57.275 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_reject_inet.ko 2023-10-30T13:58:57.275 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_socket.ko 2023-10-30T13:58:57.275 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_synproxy.ko 2023-10-30T13:58:57.276 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_tproxy.ko 2023-10-30T13:58:57.276 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 89313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/x_tables.ko 2023-10-30T13:58:57.276 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_AUDIT.ko 2023-10-30T13:58:57.276 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CHECKSUM.ko 2023-10-30T13:58:57.277 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CLASSIFY.ko 2023-10-30T13:58:57.277 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-10-30T13:58:57.277 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 18169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CT.ko 2023-10-30T13:58:57.277 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_DSCP.ko 2023-10-30T13:58:57.277 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_HL.ko 2023-10-30T13:58:57.278 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_HMARK.ko 2023-10-30T13:58:57.278 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 38873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_IDLETIMER.ko 2023-10-30T13:58:57.278 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_LED.ko 2023-10-30T13:58:57.278 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_LOG.ko 2023-10-30T13:58:57.278 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_MASQUERADE.ko 2023-10-30T13:58:57.279 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_NETMAP.ko 2023-10-30T13:58:57.279 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_NFLOG.ko 2023-10-30T13:58:57.279 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_NFQUEUE.ko 2023-10-30T13:58:57.279 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_RATEEST.ko 2023-10-30T13:58:57.279 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_REDIRECT.ko 2023-10-30T13:58:57.279 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 14337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_SECMARK.ko 2023-10-30T13:58:57.280 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TCPMSS.ko 2023-10-30T13:58:57.280 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-10-30T13:58:57.280 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TEE.ko 2023-10-30T13:58:57.280 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TPROXY.ko 2023-10-30T13:58:57.280 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TRACE.ko 2023-10-30T13:58:57.281 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_addrtype.ko 2023-10-30T13:58:57.281 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_bpf.ko 2023-10-30T13:58:57.281 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_cgroup.ko 2023-10-30T13:58:57.281 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_cluster.ko 2023-10-30T13:58:57.281 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_comment.ko 2023-10-30T13:58:57.282 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connbytes.ko 2023-10-30T13:58:57.282 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connlabel.ko 2023-10-30T13:58:57.282 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connlimit.ko 2023-10-30T13:58:57.282 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connmark.ko 2023-10-30T13:58:57.282 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_conntrack.ko 2023-10-30T13:58:57.282 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_cpu.ko 2023-10-30T13:58:57.283 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_dccp.ko 2023-10-30T13:58:57.283 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_devgroup.ko 2023-10-30T13:58:57.283 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_dscp.ko 2023-10-30T13:58:57.283 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_ecn.ko 2023-10-30T13:58:57.283 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_esp.ko 2023-10-30T13:58:57.284 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 43289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_hashlimit.ko 2023-10-30T13:58:57.284 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_helper.ko 2023-10-30T13:58:57.284 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_hl.ko 2023-10-30T13:58:57.284 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_ipcomp.ko 2023-10-30T13:58:57.284 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_iprange.ko 2023-10-30T13:58:57.285 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_ipvs.ko 2023-10-30T13:58:57.285 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_l2tp.ko 2023-10-30T13:58:57.285 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_length.ko 2023-10-30T13:58:57.285 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_limit.ko 2023-10-30T13:58:57.285 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_mac.ko 2023-10-30T13:58:57.286 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_mark.ko 2023-10-30T13:58:57.286 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_multiport.ko 2023-10-30T13:58:57.286 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_nat.ko 2023-10-30T13:58:57.286 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_nfacct.ko 2023-10-30T13:58:57.286 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_osf.ko 2023-10-30T13:58:57.287 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_owner.ko 2023-10-30T13:58:57.287 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_physdev.ko 2023-10-30T13:58:57.287 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_pkttype.ko 2023-10-30T13:58:57.287 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_policy.ko 2023-10-30T13:58:57.287 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_quota.ko 2023-10-30T13:58:57.287 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_rateest.ko 2023-10-30T13:58:57.288 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_realm.ko 2023-10-30T13:58:57.288 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_recent.ko 2023-10-30T13:58:57.288 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_sctp.ko 2023-10-30T13:58:57.288 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_set.ko 2023-10-30T13:58:57.289 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 16089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_socket.ko 2023-10-30T13:58:57.289 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_state.ko 2023-10-30T13:58:57.289 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_statistic.ko 2023-10-30T13:58:57.289 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_string.ko 2023-10-30T13:58:57.289 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 8553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_tcpmss.ko 2023-10-30T13:58:57.290 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_tcpudp.ko 2023-10-30T13:58:57.290 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_time.ko 2023-10-30T13:58:57.290 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_u32.ko 2023-10-30T13:58:57.290 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netlink/ 2023-10-30T13:58:57.290 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netlink/netlink_diag.ko 2023-10-30T13:58:57.291 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/nsh/ 2023-10-30T13:58:57.291 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 12425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/nsh/nsh.ko 2023-10-30T13:58:57.291 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/ 2023-10-30T13:58:57.291 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 380033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/openvswitch.ko 2023-10-30T13:58:57.292 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/vport-geneve.ko 2023-10-30T13:58:57.292 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/vport-gre.ko 2023-10-30T13:58:57.292 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 11793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/vport-vxlan.ko 2023-10-30T13:58:57.292 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/packet/ 2023-10-30T13:58:57.292 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/packet/af_packet_diag.ko 2023-10-30T13:58:57.293 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/phonet/ 2023-10-30T13:58:57.293 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 110201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/phonet/phonet.ko 2023-10-30T13:58:57.293 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 54697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/phonet/pn_pep.ko 2023-10-30T13:58:57.293 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/ 2023-10-30T13:58:57.293 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 251481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/rds.ko 2023-10-30T13:58:57.294 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 164937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/rds_rdma.ko 2023-10-30T13:58:57.294 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 67377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/rds_tcp.ko 2023-10-30T13:58:57.294 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rxrpc/ 2023-10-30T13:58:57.294 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 819657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rxrpc/rxrpc.ko 2023-10-30T13:58:57.294 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/ 2023-10-30T13:58:57.295 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 32241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_csum.ko 2023-10-30T13:58:57.295 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_gact.ko 2023-10-30T13:58:57.295 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_ipt.ko 2023-10-30T13:58:57.295 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_mirred.ko 2023-10-30T13:58:57.295 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_nat.ko 2023-10-30T13:58:57.296 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 43449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_pedit.ko 2023-10-30T13:58:57.296 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 31961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_police.ko 2023-10-30T13:58:57.296 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_simple.ko 2023-10-30T13:58:57.296 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 35297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_skbedit.ko 2023-10-30T13:58:57.296 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_vlan.ko 2023-10-30T13:58:57.297 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_basic.ko 2023-10-30T13:58:57.297 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_bpf.ko 2023-10-30T13:58:57.297 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_cgroup.ko 2023-10-30T13:58:57.297 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 31401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_flow.ko 2023-10-30T13:58:57.297 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 31769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_fw.ko 2023-10-30T13:58:57.298 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 38489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_route.ko 2023-10-30T13:58:57.298 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 61409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_u32.ko 2023-10-30T13:58:57.298 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_cmp.ko 2023-10-30T13:58:57.298 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 13409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_ipset.ko 2023-10-30T13:58:57.298 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_meta.ko 2023-10-30T13:58:57.299 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_nbyte.ko 2023-10-30T13:58:57.299 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_text.ko 2023-10-30T13:58:57.299 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_u32.ko 2023-10-30T13:58:57.299 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 21649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_choke.ko 2023-10-30T13:58:57.299 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_codel.ko 2023-10-30T13:58:57.299 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 34633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_drr.ko 2023-10-30T13:58:57.300 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 33609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_fq.ko 2023-10-30T13:58:57.300 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 31617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_fq_codel.ko 2023-10-30T13:58:57.300 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 35433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_gred.ko 2023-10-30T13:58:57.300 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 49601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_hfsc.ko 2023-10-30T13:58:57.300 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_hhf.ko 2023-10-30T13:58:57.300 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 66393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_htb.ko 2023-10-30T13:58:57.301 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_ingress.ko 2023-10-30T13:58:57.301 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 30273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_mqprio.ko 2023-10-30T13:58:57.301 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_mqprio_lib.ko 2023-10-30T13:58:57.301 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_multiq.ko 2023-10-30T13:58:57.301 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 36137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_netem.ko 2023-10-30T13:58:57.301 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 27497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_pie.ko 2023-10-30T13:58:57.302 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_plug.ko 2023-10-30T13:58:57.302 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 24729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_prio.ko 2023-10-30T13:58:57.302 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 48793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_qfq.ko 2023-10-30T13:58:57.302 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 32905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_red.ko 2023-10-30T13:58:57.302 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 26881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_sfb.ko 2023-10-30T13:58:57.302 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 31353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_sfq.ko 2023-10-30T13:58:57.303 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_tbf.ko 2023-10-30T13:58:57.303 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 28105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_teql.ko 2023-10-30T13:58:57.303 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sctp/ 2023-10-30T13:58:57.303 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 929745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sctp/sctp.ko 2023-10-30T13:58:57.303 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 19033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sctp/sctp_diag.ko 2023-10-30T13:58:57.303 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/ 2023-10-30T13:58:57.304 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/auth_gss/ 2023-10-30T13:58:57.304 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 336529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-10-30T13:58:57.304 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 54737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-10-30T13:58:57.304 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 1520161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/sunrpc.ko 2023-10-30T13:58:57.304 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/xprtrdma/ 2023-10-30T13:58:57.304 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 707721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-10-30T13:58:57.305 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/tipc/ 2023-10-30T13:58:57.305 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/tipc/diag.ko 2023-10-30T13:58:57.305 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 862641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/tipc/tipc.ko 2023-10-30T13:58:57.305 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/unix/ 2023-10-30T13:58:57.305 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/unix/unix_diag.ko 2023-10-30T13:58:57.305 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/ 2023-10-30T13:58:57.306 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 85337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-10-30T13:58:57.306 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 68049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-10-30T13:58:57.306 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 96193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vsock.ko 2023-10-30T13:58:57.306 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 10705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vsock_diag.ko 2023-10-30T13:58:57.306 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 17521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vsock_loopback.ko 2023-10-30T13:58:57.306 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/x25/ 2023-10-30T13:58:57.307 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 125177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/x25/x25.ko 2023-10-30T13:58:57.307 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/ 2023-10-30T13:58:57.307 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 22657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/xfrm_algo.ko 2023-10-30T13:58:57.307 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 20649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/xfrm_ipcomp.ko 2023-10-30T13:58:57.307 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 102257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/xfrm_user.ko 2023-10-30T13:58:57.307 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/virt/ 2023-10-30T13:58:57.308 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/virt/lib/ 2023-10-30T13:58:57.308 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 15537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/virt/lib/irqbypass.ko 2023-10-30T13:58:57.308 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 639863 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.alias 2023-10-30T13:58:57.309 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 623919 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.alias.bin 2023-10-30T13:58:57.309 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 7663 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin 2023-10-30T13:58:57.309 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 23428 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin.alias.bin 2023-10-30T13:58:57.309 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 9821 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin.bin 2023-10-30T13:58:57.309 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 70004 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin.modinfo 2023-10-30T13:58:57.309 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 146313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.dep 2023-10-30T13:58:57.310 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 232548 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.dep.bin 2023-10-30T13:58:57.310 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 210 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.devname 2023-10-30T13:58:57.310 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 72606 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.order 2023-10-30T13:58:57.310 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 879 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.softdep 2023-10-30T13:58:57.310 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 254438 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.symbols 2023-10-30T13:58:57.311 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 309363 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.symbols.bin 2023-10-30T13:58:57.311 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/ 2023-10-30T13:58:57.311 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/share/ 2023-10-30T13:58:57.311 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/share/doc/ 2023-10-30T13:58:57.311 INFO:teuthology.orchestra.run.smithi057.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/share/doc/linux-image-6.6.0-rc5-geacf00cc51e2/ 2023-10-30T13:58:57.311 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 173 2023-10-30 03:46 ./usr/share/doc/linux-image-6.6.0-rc5-geacf00cc51e2/changelog.Debian.gz 2023-10-30T13:58:57.311 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r-- root/root 691 2023-10-30 03:46 ./usr/share/doc/linux-image-6.6.0-rc5-geacf00cc51e2/copyright 2023-10-30T13:58:57.312 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-rc5-geacf00cc51e2 2023-10-30T13:58:57.312 INFO:teuthology.task.kernel:searching for kernel 6.6.0-rc5-geacf00cc51e2 2023-10-30T13:58:57.313 DEBUG:teuthology.orchestra.run.smithi057:> egrep '(submenu|menuentry.*6.6.0-rc5-geacf00cc51e2).*{' /boot/grub/grub.cfg 2023-10-30T13:58:57.320 INFO:teuthology.orchestra.run.smithi057.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-30T13:58:57.320 INFO:teuthology.orchestra.run.smithi057.stdout: menuentry 'Ubuntu, with Linux 6.6.0-rc5-geacf00cc51e2' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-rc5-geacf00cc51e2-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-30T13:58:57.321 INFO:teuthology.orchestra.run.smithi057.stdout: menuentry 'Ubuntu, with Linux 6.6.0-rc5-geacf00cc51e2 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-rc5-geacf00cc51e2-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-30T13:58:57.321 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2023-10-30T13:58:57.321 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.6.0-rc5-geacf00cc51e2 2023-10-30T13:58:57.322 DEBUG:teuthology.orchestra.run.smithi057:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2023-10-30T13:58:57.368 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi057.front.sepia.ceph.com to complete... 2023-10-30T13:58:57.734 INFO:teuthology.orchestra.run.smithi057.stderr:Sourcing file `/etc/default/grub' 2023-10-30T13:58:57.735 INFO:teuthology.orchestra.run.smithi057.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-30T13:58:57.757 INFO:teuthology.orchestra.run.smithi057.stderr:Generating grub configuration file ... 2023-10-30T13:58:57.946 INFO:teuthology.orchestra.run.smithi057.stderr:Found linux image: /boot/vmlinuz-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:58:57.952 INFO:teuthology.orchestra.run.smithi057.stderr:Found initrd image: /boot/initrd.img-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:58:58.202 INFO:teuthology.orchestra.run.smithi057.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-30T13:58:58.207 INFO:teuthology.orchestra.run.smithi057.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-30T13:58:58.247 INFO:teuthology.orchestra.run.smithi057.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-30T13:58:58.252 INFO:teuthology.orchestra.run.smithi057.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-30T13:58:58.493 INFO:teuthology.orchestra.run.smithi057.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-30T13:58:58.493 INFO:teuthology.orchestra.run.smithi057.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-30T13:58:58.494 INFO:teuthology.orchestra.run.smithi057.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-30T13:58:58.507 INFO:teuthology.orchestra.run.smithi057.stderr:done 2023-10-30T13:59:29.637 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-30T13:59:29.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-10-30T13:59:29.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T14:00:04.218 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2023-10-30T14:00:13.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-10-30T14:00:13.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T14:01:13.223 DEBUG:teuthology.orchestra.remote:timed out 2023-10-30T14:01:25.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-10-30T14:01:25.226 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T14:01:25.522 DEBUG:teuthology.orchestra.run.smithi057:> true 2023-10-30T14:01:26.172 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-10-30T14:01:26.172 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.6.0-rc5-geacf00cc51e2"... 2023-10-30T14:01:26.173 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2023-10-30T14:01:26.222 INFO:teuthology.orchestra.run.smithi057.stdout:6.6.0-rc5-geacf00cc51e2 2023-10-30T14:01:26.222 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-rc5-geacf00cc51e2 vs 6.6.0-rc5-geacf00cc51e2 2023-10-30T14:01:26.222 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-30T14:01:26.223 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-30T14:01:27.224 DEBUG:teuthology.parallel:result is None 2023-10-30T14:01:27.224 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-30T14:01:27.233 INFO:teuthology.task.internal:Creating test directory... 2023-10-30T14:01:27.233 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T14:01:27.237 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T14:01:27.244 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-30T14:01:27.319 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-30T14:01:27.327 INFO:teuthology.task.internal:Creating archive directory... 2023-10-30T14:01:27.327 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T14:01:27.331 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T14:01:27.351 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-30T14:01:27.358 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-30T14:01:27.358 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T14:01:27.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T14:01:27.391 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T14:01:27.395 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T14:01:27.464 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T14:01:27.475 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T14:01:27.481 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T14:01:27.499 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T14:01:27.502 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-30T14:01:27.509 INFO:teuthology.task.internal:Configuring sudo... 2023-10-30T14:01:27.509 DEBUG:teuthology.orchestra.run.smithi057:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T14:01:27.513 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T14:01:27.535 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-30T14:01:27.544 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-30T14:01:27.544 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T14:01:27.577 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T14:01:27.584 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T14:01:27.639 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T14:01:27.695 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:01:27.695 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T14:01:27.768 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T14:01:27.774 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T14:01:27.820 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:01:27.820 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T14:01:27.883 DEBUG:teuthology.orchestra.run.smithi057:> sudo service rsyslog restart 2023-10-30T14:01:27.887 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2023-10-30T14:01:27.984 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-30T14:01:27.992 INFO:teuthology.task.internal:Starting timer... 2023-10-30T14:01:27.992 INFO:teuthology.run_tasks:Running task pcp... 2023-10-30T14:01:28.002 INFO:teuthology.run_tasks:Running task selinux... 2023-10-30T14:01:28.010 DEBUG:teuthology.task.selinux:Excluding smithi057: OS 'ubuntu' does not support SELinux 2023-10-30T14:01:28.011 DEBUG:teuthology.task.selinux:Excluding smithi088: OS 'ubuntu' does not support SELinux 2023-10-30T14:01:28.011 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-30T14:01:28.011 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-30T14:01:28.012 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-30T14:01:28.012 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-30T14:01:28.021 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-30T14:01:28.027 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-30T14:01:28.176 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-30T14:01:28.192 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-30T14:01:28.194 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi057.front.sepia.ceph.com,smithi088.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-30T14:09:59.265 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi057.front.sepia.ceph.com'), Remote(name='ubuntu@smithi088.front.sepia.ceph.com')] 2023-10-30T14:09:59.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-10-30T14:09:59.267 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T14:09:59.348 DEBUG:teuthology.orchestra.run.smithi057:> true 2023-10-30T14:09:59.474 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-10-30T14:09:59.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-10-30T14:09:59.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T14:09:59.551 DEBUG:teuthology.orchestra.run.smithi088:> true 2023-10-30T14:09:59.662 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-10-30T14:09:59.662 INFO:teuthology.run_tasks:Running task clock... 2023-10-30T14:09:59.674 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-30T14:09:59.675 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T14:09:59.675 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T14:09:59.679 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T14:09:59.679 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T14:09:59.716 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-30T14:09:59.716 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: Command line: ntpd -gq 2023-10-30T14:09:59.716 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: ---------------------------------------------------- 2023-10-30T14:09:59.716 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: ntp-4 is maintained by Network Time Foundation, 2023-10-30T14:09:59.717 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-30T14:09:59.717 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: corporation. Support and training for ntp-4 are 2023-10-30T14:09:59.717 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: available at https://www.nwtime.org/support 2023-10-30T14:09:59.717 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: ---------------------------------------------------- 2023-10-30T14:09:59.718 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: proto: precision = 0.043 usec (-24) 2023-10-30T14:09:59.718 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: basedate set to 2022-02-04 2023-10-30T14:09:59.718 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: gps base set to 2022-02-06 (week 2196) 2023-10-30T14:09:59.719 INFO:teuthology.orchestra.run.smithi057.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T14:09:59.719 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T14:09:59.719 INFO:teuthology.orchestra.run.smithi057.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-30T14:09:59.720 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: restrict ::: KOD does nothing without LIMITED. 2023-10-30T14:09:59.720 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: Listen and drop on 0 v6wildcard [::]:123 2023-10-30T14:09:59.721 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-30T14:09:59.721 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: Listen normally on 2 lo 127.0.0.1:123 2023-10-30T14:09:59.721 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: Listen normally on 3 ens1f0 172.21.15.57:123 2023-10-30T14:09:59.721 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: Listen normally on 4 lo [::1]:123 2023-10-30T14:09:59.721 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1562%4]:123 2023-10-30T14:09:59.722 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:09:59 ntpd[27872]: Listening on routing socket on fd #22 for interface updates 2023-10-30T14:09:59.732 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-30T14:09:59.732 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: Command line: ntpd -gq 2023-10-30T14:09:59.732 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: ---------------------------------------------------- 2023-10-30T14:09:59.733 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: ntp-4 is maintained by Network Time Foundation, 2023-10-30T14:09:59.733 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-30T14:09:59.733 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: corporation. Support and training for ntp-4 are 2023-10-30T14:09:59.733 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: available at https://www.nwtime.org/support 2023-10-30T14:09:59.733 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: ---------------------------------------------------- 2023-10-30T14:09:59.733 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: proto: precision = 0.043 usec (-24) 2023-10-30T14:09:59.734 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: basedate set to 2022-02-04 2023-10-30T14:09:59.734 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: gps base set to 2022-02-06 (week 2196) 2023-10-30T14:09:59.734 INFO:teuthology.orchestra.run.smithi088.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T14:09:59.735 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T14:09:59.735 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: restrict ::: KOD does nothing without LIMITED. 2023-10-30T14:09:59.735 INFO:teuthology.orchestra.run.smithi088.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-30T14:09:59.736 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: Listen and drop on 0 v6wildcard [::]:123 2023-10-30T14:09:59.736 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-30T14:09:59.737 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: Listen normally on 2 lo 127.0.0.1:123 2023-10-30T14:09:59.737 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: Listen normally on 3 enp3s0f1 172.21.15.88:123 2023-10-30T14:09:59.737 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: Listen normally on 4 lo [::1]:123 2023-10-30T14:09:59.737 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d42d%5]:123 2023-10-30T14:09:59.737 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:09:59 ntpd[28392]: Listening on routing socket on fd #22 for interface updates 2023-10-30T14:10:00.732 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:00 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:00.732 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:00 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:00.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:00 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:00.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:00 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:00.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:00 ntpd[28392]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T14:10:00.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:00 ntpd[28392]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T14:10:01.717 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:01 ntpd[27872]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:01.717 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:01 ntpd[27872]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:01.717 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:01 ntpd[27872]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:01.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:01 ntpd[27872]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:01.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:01 ntpd[27872]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:01.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:01 ntpd[27872]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:01.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:01 ntpd[27872]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T14:10:01.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:01 ntpd[27872]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T14:10:01.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:01 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:01.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:01 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:01.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:01 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:01.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:01 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:01.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:01 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:01.734 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:01 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:01.734 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:01 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:01.734 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:01 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:01.734 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:01 ntpd[28392]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T14:10:01.734 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:01 ntpd[28392]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T14:10:01.734 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:01 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:01.735 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:01 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:01.735 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:01 ntpd[28392]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T14:10:01.735 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:01 ntpd[28392]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T14:10:02.732 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:02 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:02.732 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:02 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:02.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:02 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:02.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:02 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:02.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:02 ntpd[28392]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T14:10:02.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:02 ntpd[28392]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T14:10:03.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:03 ntpd[27872]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:03.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:03 ntpd[27872]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:03.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:03 ntpd[27872]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:03.719 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:03 ntpd[27872]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:03.719 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:03 ntpd[27872]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:03.719 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:03 ntpd[27872]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:03.719 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:03 ntpd[27872]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T14:10:03.719 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:03 ntpd[27872]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T14:10:03.719 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:03 ntpd[27872]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:03.720 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:03 ntpd[27872]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:03.720 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:03 ntpd[27872]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T14:10:03.720 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:03 ntpd[27872]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T14:10:03.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:03 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:03.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:03 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:03.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:03 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:03.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:03 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:03.734 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:03 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:03.734 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:03 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:03.734 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:03 ntpd[28392]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T14:10:03.734 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:03 ntpd[28392]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T14:10:03.735 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:03 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:03.735 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:03 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:03.735 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:03 ntpd[28392]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T14:10:03.735 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:03 ntpd[28392]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T14:10:04.717 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:04 ntpd[27872]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:04.717 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:04 ntpd[27872]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:04.717 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:04 ntpd[27872]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:04.717 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:04 ntpd[27872]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:04.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:04 ntpd[27872]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T14:10:04.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:04 ntpd[27872]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T14:10:04.732 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:04 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:04.732 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:04 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:04.732 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:04 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:04.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:04 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:04.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:04 ntpd[28392]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T14:10:04.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:04 ntpd[28392]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T14:10:05.717 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:05 ntpd[27872]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:05.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:05 ntpd[27872]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:05.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:05 ntpd[27872]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:05.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:05 ntpd[27872]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:05.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:05 ntpd[27872]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:05.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:05 ntpd[27872]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:05.719 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:05 ntpd[27872]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T14:10:05.719 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:05 ntpd[27872]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T14:10:05.719 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:05 ntpd[27872]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:05.719 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:05 ntpd[27872]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:05.719 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:05 ntpd[27872]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T14:10:05.719 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:05 ntpd[27872]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T14:10:05.732 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:05 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:05.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:05 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:05.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:05 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:05.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:05 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:05.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:05 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:05.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:05 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:05.734 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:05 ntpd[28392]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T14:10:05.734 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:05 ntpd[28392]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T14:10:05.734 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:05 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:05.734 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:05 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:05.734 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:05 ntpd[28392]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T14:10:05.735 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:05 ntpd[28392]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T14:10:06.717 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:06 ntpd[27872]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:06.717 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:06 ntpd[27872]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:06.717 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:06 ntpd[27872]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:06.717 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:06 ntpd[27872]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:06.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:06 ntpd[27872]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T14:10:06.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:06 ntpd[27872]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T14:10:06.732 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:06 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:06.732 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:06 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:06.732 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:06 ntpd[28392]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:06.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:06 ntpd[28392]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:06.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:06 ntpd[28392]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T14:10:06.733 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:06 ntpd[28392]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T14:10:06.733 INFO:teuthology.orchestra.run.smithi088.stdout:30 Oct 14:10:06 ntpd[28392]: ntpd: time slew -0.004636 s 2023-10-30T14:10:06.734 INFO:teuthology.orchestra.run.smithi088.stdout:ntpd: time slew -0.004636s 2023-10-30T14:10:06.734 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:06 ntpd[28392]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-30T14:10:06.734 INFO:teuthology.orchestra.run.smithi088.stderr:30 Oct 14:10:06 ntpd[28392]: can't open /var/log/ntpstats/loopstats.20231030: Permission denied 2023-10-30T14:10:06.814 INFO:teuthology.orchestra.run.smithi088.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T14:10:06.814 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================== 2023-10-30T14:10:06.814 INFO:teuthology.orchestra.run.smithi088.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T14:10:06.815 INFO:teuthology.orchestra.run.smithi088.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T14:10:06.815 INFO:teuthology.orchestra.run.smithi088.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T14:10:06.815 INFO:teuthology.orchestra.run.smithi088.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T14:10:07.717 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:07 ntpd[27872]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:07.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:07 ntpd[27872]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:07.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:07 ntpd[27872]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:07.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:07 ntpd[27872]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:07.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:07 ntpd[27872]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:07.718 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:07 ntpd[27872]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:07.719 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:07 ntpd[27872]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T14:10:07.719 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:07 ntpd[27872]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T14:10:07.719 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:07 ntpd[27872]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T14:10:07.719 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:07 ntpd[27872]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T14:10:07.719 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:07 ntpd[27872]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T14:10:07.719 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:07 ntpd[27872]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T14:10:07.720 INFO:teuthology.orchestra.run.smithi057.stdout:30 Oct 14:10:07 ntpd[27872]: ntpd: time slew -0.000596 s 2023-10-30T14:10:07.720 INFO:teuthology.orchestra.run.smithi057.stdout:ntpd: time slew -0.000596s 2023-10-30T14:10:07.721 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:07 ntpd[27872]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-30T14:10:07.721 INFO:teuthology.orchestra.run.smithi057.stderr:30 Oct 14:10:07 ntpd[27872]: can't open /var/log/ntpstats/loopstats.20231030: Permission denied 2023-10-30T14:10:07.808 INFO:teuthology.orchestra.run.smithi057.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T14:10:07.808 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================== 2023-10-30T14:10:07.809 INFO:teuthology.orchestra.run.smithi057.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T14:10:07.809 INFO:teuthology.orchestra.run.smithi057.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T14:10:07.809 INFO:teuthology.orchestra.run.smithi057.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T14:10:07.809 INFO:teuthology.orchestra.run.smithi057.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T14:10:07.810 INFO:teuthology.run_tasks:Running task install... 2023-10-30T14:10:07.819 DEBUG:teuthology.task.install:project ceph 2023-10-30T14:10:07.819 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}} 2023-10-30T14:10:07.819 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'} 2023-10-30T14:10:07.820 INFO:teuthology.task.install:Using flavor: default 2023-10-30T14:10:07.825 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-30T14:10:07.825 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-30T14:10:07.826 DEBUG:teuthology.orchestra.run.smithi057:> sudo apt-key list | grep Ceph 2023-10-30T14:10:07.826 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-key list | grep Ceph 2023-10-30T14:10:07.887 INFO:teuthology.orchestra.run.smithi088.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-30T14:10:07.914 INFO:teuthology.orchestra.run.smithi088.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-30T14:10:07.914 INFO:teuthology.orchestra.run.smithi088.stdout:uid [ unknown] Ceph.com (release key) 2023-10-30T14:10:07.915 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-10-30T14:10:07.916 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-10-30T14:10:07.916 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T14:10:07.962 INFO:teuthology.orchestra.run.smithi057.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-30T14:10:08.017 INFO:teuthology.orchestra.run.smithi057.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-30T14:10:08.017 INFO:teuthology.orchestra.run.smithi057.stdout:uid [ unknown] Ceph.com (release key) 2023-10-30T14:10:08.018 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-10-30T14:10:08.018 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-10-30T14:10:08.018 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T14:10:08.066 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default/ 2023-10-30T14:10:08.067 INFO:teuthology.task.install.deb:Package version is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T14:10:08.258 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:10:08.259 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-30T14:10:08.272 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get update 2023-10-30T14:10:08.279 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default/ 2023-10-30T14:10:08.280 INFO:teuthology.task.install.deb:Package version is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T14:10:08.383 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:10:08.383 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-30T14:10:08.404 DEBUG:teuthology.orchestra.run.smithi057:> sudo apt-get update 2023-10-30T14:10:08.440 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-30T14:10:08.449 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-30T14:10:08.466 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-30T14:10:08.542 INFO:teuthology.orchestra.run.smithi088.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy InRelease 2023-10-30T14:10:08.577 INFO:teuthology.orchestra.run.smithi088.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-30T14:10:08.594 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-30T14:10:08.597 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-30T14:10:08.615 INFO:teuthology.orchestra.run.smithi088.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-30T14:10:08.664 INFO:teuthology.orchestra.run.smithi088.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-30T14:10:08.709 INFO:teuthology.orchestra.run.smithi057.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy InRelease 2023-10-30T14:10:08.711 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-30T14:10:08.752 INFO:teuthology.orchestra.run.smithi057.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-30T14:10:08.755 INFO:teuthology.orchestra.run.smithi088.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2023-10-30T14:10:08.783 INFO:teuthology.orchestra.run.smithi057.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-30T14:10:08.793 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-30T14:10:08.810 INFO:teuthology.orchestra.run.smithi057.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-30T14:10:08.829 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 28.5 kB in 0s (75.9 kB/s) 2023-10-30T14:10:08.848 INFO:teuthology.orchestra.run.smithi057.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2023-10-30T14:10:08.874 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-30T14:10:09.220 INFO:teuthology.orchestra.run.smithi057.stdout:Fetched 28.5 kB in 1s (48.5 kB/s) 2023-10-30T14:10:09.730 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2023-10-30T14:10:09.744 DEBUG:teuthology.orchestra.run.smithi088:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6835-g00c1c13c-1jammy cephadm=18.0.0-6835-g00c1c13c-1jammy ceph-mds=18.0.0-6835-g00c1c13c-1jammy ceph-mgr=18.0.0-6835-g00c1c13c-1jammy ceph-common=18.0.0-6835-g00c1c13c-1jammy ceph-fuse=18.0.0-6835-g00c1c13c-1jammy ceph-test=18.0.0-6835-g00c1c13c-1jammy ceph-volume=18.0.0-6835-g00c1c13c-1jammy radosgw=18.0.0-6835-g00c1c13c-1jammy python3-rados=18.0.0-6835-g00c1c13c-1jammy python3-rgw=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy python3-rbd=18.0.0-6835-g00c1c13c-1jammy libcephfs2=18.0.0-6835-g00c1c13c-1jammy libcephfs-dev=18.0.0-6835-g00c1c13c-1jammy librados2=18.0.0-6835-g00c1c13c-1jammy librbd1=18.0.0-6835-g00c1c13c-1jammy rbd-fuse=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy cephfs-shell=18.0.0-6835-g00c1c13c-1jammy cephfs-top=18.0.0-6835-g00c1c13c-1jammy cephfs-mirror=18.0.0-6835-g00c1c13c-1jammy 2023-10-30T14:10:09.796 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2023-10-30T14:10:09.961 INFO:teuthology.orchestra.run.smithi088.stdout:Building dependency tree... 2023-10-30T14:10:09.962 INFO:teuthology.orchestra.run.smithi088.stdout:Reading state information... 2023-10-30T14:10:10.109 INFO:teuthology.orchestra.run.smithi088.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T14:10:10.109 INFO:teuthology.orchestra.run.smithi088.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T14:10:10.109 INFO:teuthology.orchestra.run.smithi088.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T14:10:10.110 INFO:teuthology.orchestra.run.smithi088.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T14:10:10.111 INFO:teuthology.orchestra.run.smithi088.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-30T14:10:10.111 INFO:teuthology.orchestra.run.smithi088.stdout: python2.7-minimal 2023-10-30T14:10:10.111 INFO:teuthology.orchestra.run.smithi088.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T14:10:10.111 INFO:teuthology.orchestra.run.smithi088.stdout:The following additional packages will be installed: 2023-10-30T14:10:10.111 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-30T14:10:10.112 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-30T14:10:10.112 INFO:teuthology.orchestra.run.smithi088.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T14:10:10.112 INFO:teuthology.orchestra.run.smithi088.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-30T14:10:10.113 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-30T14:10:10.113 INFO:teuthology.orchestra.run.smithi088.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-30T14:10:10.114 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-30T14:10:10.114 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-30T14:10:10.114 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-10-30T14:10:10.114 INFO:teuthology.orchestra.run.smithi088.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-30T14:10:10.114 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-30T14:10:10.114 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-30T14:10:10.115 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-30T14:10:10.115 INFO:teuthology.orchestra.run.smithi088.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-30T14:10:10.115 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-30T14:10:10.115 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-30T14:10:10.115 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-10-30T14:10:10.116 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-10-30T14:10:10.116 INFO:teuthology.orchestra.run.smithi088.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-10-30T14:10:10.116 INFO:teuthology.orchestra.run.smithi088.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-10-30T14:10:10.116 INFO:teuthology.orchestra.run.smithi088.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-10-30T14:10:10.116 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-10-30T14:10:10.117 INFO:teuthology.orchestra.run.smithi088.stdout: qttranslations5-l10n socat xmlstarlet 2023-10-30T14:10:10.117 INFO:teuthology.orchestra.run.smithi088.stdout:Suggested packages: 2023-10-30T14:10:10.117 INFO:teuthology.orchestra.run.smithi088.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-30T14:10:10.117 INFO:teuthology.orchestra.run.smithi088.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-30T14:10:10.117 INFO:teuthology.orchestra.run.smithi088.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-30T14:10:10.117 INFO:teuthology.orchestra.run.smithi088.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-30T14:10:10.118 INFO:teuthology.orchestra.run.smithi088.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-30T14:10:10.118 INFO:teuthology.orchestra.run.smithi088.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-30T14:10:10.118 INFO:teuthology.orchestra.run.smithi088.stdout: python3-watchdog 2023-10-30T14:10:10.118 INFO:teuthology.orchestra.run.smithi088.stdout:Recommended packages: 2023-10-30T14:10:10.118 INFO:teuthology.orchestra.run.smithi088.stdout: btrfs-tools 2023-10-30T14:10:10.175 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2023-10-30T14:10:10.189 DEBUG:teuthology.orchestra.run.smithi057:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6835-g00c1c13c-1jammy cephadm=18.0.0-6835-g00c1c13c-1jammy ceph-mds=18.0.0-6835-g00c1c13c-1jammy ceph-mgr=18.0.0-6835-g00c1c13c-1jammy ceph-common=18.0.0-6835-g00c1c13c-1jammy ceph-fuse=18.0.0-6835-g00c1c13c-1jammy ceph-test=18.0.0-6835-g00c1c13c-1jammy ceph-volume=18.0.0-6835-g00c1c13c-1jammy radosgw=18.0.0-6835-g00c1c13c-1jammy python3-rados=18.0.0-6835-g00c1c13c-1jammy python3-rgw=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy python3-rbd=18.0.0-6835-g00c1c13c-1jammy libcephfs2=18.0.0-6835-g00c1c13c-1jammy libcephfs-dev=18.0.0-6835-g00c1c13c-1jammy librados2=18.0.0-6835-g00c1c13c-1jammy librbd1=18.0.0-6835-g00c1c13c-1jammy rbd-fuse=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy cephfs-shell=18.0.0-6835-g00c1c13c-1jammy cephfs-top=18.0.0-6835-g00c1c13c-1jammy cephfs-mirror=18.0.0-6835-g00c1c13c-1jammy 2023-10-30T14:10:10.210 INFO:teuthology.orchestra.run.smithi088.stdout:The following NEW packages will be installed: 2023-10-30T14:10:10.211 INFO:teuthology.orchestra.run.smithi088.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-30T14:10:10.211 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-30T14:10:10.211 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-30T14:10:10.213 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-10-30T14:10:10.213 INFO:teuthology.orchestra.run.smithi088.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T14:10:10.213 INFO:teuthology.orchestra.run.smithi088.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-10-30T14:10:10.213 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-30T14:10:10.213 INFO:teuthology.orchestra.run.smithi088.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-30T14:10:10.214 INFO:teuthology.orchestra.run.smithi088.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-30T14:10:10.214 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-30T14:10:10.214 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-10-30T14:10:10.214 INFO:teuthology.orchestra.run.smithi088.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-10-30T14:10:10.214 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-30T14:10:10.215 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-30T14:10:10.215 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-30T14:10:10.215 INFO:teuthology.orchestra.run.smithi088.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-30T14:10:10.215 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-30T14:10:10.215 INFO:teuthology.orchestra.run.smithi088.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-30T14:10:10.216 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-10-30T14:10:10.216 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-30T14:10:10.216 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-30T14:10:10.216 INFO:teuthology.orchestra.run.smithi088.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-30T14:10:10.216 INFO:teuthology.orchestra.run.smithi088.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-30T14:10:10.217 INFO:teuthology.orchestra.run.smithi088.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-30T14:10:10.217 INFO:teuthology.orchestra.run.smithi088.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-30T14:10:10.217 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-30T14:10:10.217 INFO:teuthology.orchestra.run.smithi088.stdout: socat xmlstarlet 2023-10-30T14:10:10.267 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2023-10-30T14:10:10.398 INFO:teuthology.orchestra.run.smithi088.stdout:0 upgraded, 115 newly installed, 0 to remove and 260 not upgraded. 2023-10-30T14:10:10.398 INFO:teuthology.orchestra.run.smithi088.stdout:Need to get 184 MB of archives. 2023-10-30T14:10:10.399 INFO:teuthology.orchestra.run.smithi088.stdout:After this operation, 826 MB of additional disk space will be used. 2023-10-30T14:10:10.399 INFO:teuthology.orchestra.run.smithi088.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6835-g00c1c13c-1jammy [3,450 kB] 2023-10-30T14:10:10.400 INFO:teuthology.orchestra.run.smithi088.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-10-30T14:10:10.439 INFO:teuthology.orchestra.run.smithi057.stdout:Building dependency tree... 2023-10-30T14:10:10.440 INFO:teuthology.orchestra.run.smithi057.stdout:Reading state information... 2023-10-30T14:10:10.483 INFO:teuthology.orchestra.run.smithi088.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-30T14:10:10.528 INFO:teuthology.orchestra.run.smithi088.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-10-30T14:10:10.596 INFO:teuthology.orchestra.run.smithi057.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T14:10:10.597 INFO:teuthology.orchestra.run.smithi057.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T14:10:10.597 INFO:teuthology.orchestra.run.smithi057.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T14:10:10.597 INFO:teuthology.orchestra.run.smithi057.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T14:10:10.597 INFO:teuthology.orchestra.run.smithi057.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-30T14:10:10.599 INFO:teuthology.orchestra.run.smithi057.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-30T14:10:10.599 INFO:teuthology.orchestra.run.smithi057.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T14:10:10.599 INFO:teuthology.orchestra.run.smithi057.stdout:The following additional packages will be installed: 2023-10-30T14:10:10.600 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-30T14:10:10.600 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-30T14:10:10.600 INFO:teuthology.orchestra.run.smithi057.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T14:10:10.600 INFO:teuthology.orchestra.run.smithi057.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-30T14:10:10.600 INFO:teuthology.orchestra.run.smithi057.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-30T14:10:10.601 INFO:teuthology.orchestra.run.smithi057.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-30T14:10:10.601 INFO:teuthology.orchestra.run.smithi057.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-30T14:10:10.601 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-30T14:10:10.601 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-10-30T14:10:10.601 INFO:teuthology.orchestra.run.smithi057.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-30T14:10:10.602 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-30T14:10:10.602 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-30T14:10:10.602 INFO:teuthology.orchestra.run.smithi057.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-30T14:10:10.602 INFO:teuthology.orchestra.run.smithi057.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-30T14:10:10.602 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-30T14:10:10.603 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-30T14:10:10.603 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-10-30T14:10:10.603 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-10-30T14:10:10.603 INFO:teuthology.orchestra.run.smithi057.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-10-30T14:10:10.603 INFO:teuthology.orchestra.run.smithi057.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-10-30T14:10:10.604 INFO:teuthology.orchestra.run.smithi057.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-10-30T14:10:10.604 INFO:teuthology.orchestra.run.smithi057.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-10-30T14:10:10.604 INFO:teuthology.orchestra.run.smithi057.stdout: qttranslations5-l10n socat xmlstarlet 2023-10-30T14:10:10.604 INFO:teuthology.orchestra.run.smithi057.stdout:Suggested packages: 2023-10-30T14:10:10.605 INFO:teuthology.orchestra.run.smithi057.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-30T14:10:10.605 INFO:teuthology.orchestra.run.smithi057.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-30T14:10:10.605 INFO:teuthology.orchestra.run.smithi057.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-30T14:10:10.605 INFO:teuthology.orchestra.run.smithi057.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-30T14:10:10.605 INFO:teuthology.orchestra.run.smithi057.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-30T14:10:10.606 INFO:teuthology.orchestra.run.smithi057.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-30T14:10:10.606 INFO:teuthology.orchestra.run.smithi057.stdout: python3-watchdog 2023-10-30T14:10:10.606 INFO:teuthology.orchestra.run.smithi057.stdout:Recommended packages: 2023-10-30T14:10:10.606 INFO:teuthology.orchestra.run.smithi057.stdout: btrfs-tools 2023-10-30T14:10:10.620 INFO:teuthology.orchestra.run.smithi088.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6835-g00c1c13c-1jammy [3,131 kB] 2023-10-30T14:10:10.716 INFO:teuthology.orchestra.run.smithi088.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6835-g00c1c13c-1jammy [689 kB] 2023-10-30T14:10:10.723 INFO:teuthology.orchestra.run.smithi088.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-30T14:10:10.742 INFO:teuthology.orchestra.run.smithi088.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6835-g00c1c13c-1jammy [356 kB] 2023-10-30T14:10:10.749 INFO:teuthology.orchestra.run.smithi088.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6835-g00c1c13c-1jammy [32.8 kB] 2023-10-30T14:10:10.749 INFO:teuthology.orchestra.run.smithi057.stdout:The following NEW packages will be installed: 2023-10-30T14:10:10.749 INFO:teuthology.orchestra.run.smithi057.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-30T14:10:10.750 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-30T14:10:10.750 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-30T14:10:10.750 INFO:teuthology.orchestra.run.smithi088.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6835-g00c1c13c-1jammy [180 kB] 2023-10-30T14:10:10.752 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-10-30T14:10:10.752 INFO:teuthology.orchestra.run.smithi057.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T14:10:10.752 INFO:teuthology.orchestra.run.smithi057.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-10-30T14:10:10.753 INFO:teuthology.orchestra.run.smithi057.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-30T14:10:10.753 INFO:teuthology.orchestra.run.smithi057.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-30T14:10:10.753 INFO:teuthology.orchestra.run.smithi057.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-30T14:10:10.753 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-30T14:10:10.754 INFO:teuthology.orchestra.run.smithi057.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-10-30T14:10:10.754 INFO:teuthology.orchestra.run.smithi057.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-10-30T14:10:10.754 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-30T14:10:10.754 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-30T14:10:10.755 INFO:teuthology.orchestra.run.smithi057.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-30T14:10:10.755 INFO:teuthology.orchestra.run.smithi057.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-30T14:10:10.755 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-30T14:10:10.755 INFO:teuthology.orchestra.run.smithi057.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-30T14:10:10.756 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-10-30T14:10:10.756 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-30T14:10:10.756 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-30T14:10:10.757 INFO:teuthology.orchestra.run.smithi057.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-30T14:10:10.757 INFO:teuthology.orchestra.run.smithi057.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-30T14:10:10.757 INFO:teuthology.orchestra.run.smithi057.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-30T14:10:10.758 INFO:teuthology.orchestra.run.smithi057.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-30T14:10:10.758 INFO:teuthology.orchestra.run.smithi057.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-30T14:10:10.758 INFO:teuthology.orchestra.run.smithi057.stdout: socat xmlstarlet 2023-10-30T14:10:10.759 INFO:teuthology.orchestra.run.smithi088.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6835-g00c1c13c-1jammy [62.4 kB] 2023-10-30T14:10:10.760 INFO:teuthology.orchestra.run.smithi088.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6835-g00c1c13c-1jammy [330 kB] 2023-10-30T14:10:10.766 INFO:teuthology.orchestra.run.smithi088.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-30T14:10:10.769 INFO:teuthology.orchestra.run.smithi088.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6835-g00c1c13c-1jammy [6,226 kB] 2023-10-30T14:10:10.841 INFO:teuthology.orchestra.run.smithi088.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-30T14:10:10.852 INFO:teuthology.orchestra.run.smithi088.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-30T14:10:10.897 INFO:teuthology.orchestra.run.smithi088.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-30T14:10:10.949 INFO:teuthology.orchestra.run.smithi057.stdout:0 upgraded, 115 newly installed, 0 to remove and 260 not upgraded. 2023-10-30T14:10:10.949 INFO:teuthology.orchestra.run.smithi057.stdout:Need to get 184 MB of archives. 2023-10-30T14:10:10.949 INFO:teuthology.orchestra.run.smithi057.stdout:After this operation, 826 MB of additional disk space will be used. 2023-10-30T14:10:10.949 INFO:teuthology.orchestra.run.smithi057.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-10-30T14:10:10.993 INFO:teuthology.orchestra.run.smithi088.stdout:Get:18 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6835-g00c1c13c-1jammy [112 kB] 2023-10-30T14:10:10.997 INFO:teuthology.orchestra.run.smithi088.stdout:Get:19 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6835-g00c1c13c-1jammy [439 kB] 2023-10-30T14:10:11.018 INFO:teuthology.orchestra.run.smithi088.stdout:Get:20 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6835-g00c1c13c-1jammy [22.6 MB] 2023-10-30T14:10:11.031 INFO:teuthology.orchestra.run.smithi057.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-30T14:10:11.062 INFO:teuthology.orchestra.run.smithi057.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6835-g00c1c13c-1jammy [3,450 kB] 2023-10-30T14:10:11.063 INFO:teuthology.orchestra.run.smithi088.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-30T14:10:11.071 INFO:teuthology.orchestra.run.smithi088.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-30T14:10:11.077 INFO:teuthology.orchestra.run.smithi057.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-10-30T14:10:11.098 INFO:teuthology.orchestra.run.smithi088.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-30T14:10:11.270 INFO:teuthology.orchestra.run.smithi088.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-30T14:10:11.282 INFO:teuthology.orchestra.run.smithi057.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-30T14:10:11.327 INFO:teuthology.orchestra.run.smithi057.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-30T14:10:11.402 INFO:teuthology.orchestra.run.smithi057.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-30T14:10:11.414 INFO:teuthology.orchestra.run.smithi057.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-30T14:10:11.461 INFO:teuthology.orchestra.run.smithi057.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-30T14:10:11.465 INFO:teuthology.orchestra.run.smithi088.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-30T14:10:11.626 INFO:teuthology.orchestra.run.smithi057.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-30T14:10:11.634 INFO:teuthology.orchestra.run.smithi057.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-30T14:10:11.661 INFO:teuthology.orchestra.run.smithi057.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-30T14:10:11.834 INFO:teuthology.orchestra.run.smithi057.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-30T14:10:11.835 INFO:teuthology.orchestra.run.smithi088.stdout:Get:26 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6835-g00c1c13c-1jammy [6,858 kB] 2023-10-30T14:10:11.856 INFO:teuthology.orchestra.run.smithi088.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-30T14:10:11.858 INFO:teuthology.orchestra.run.smithi088.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-30T14:10:11.859 INFO:teuthology.orchestra.run.smithi088.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-30T14:10:11.861 INFO:teuthology.orchestra.run.smithi088.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-30T14:10:11.861 INFO:teuthology.orchestra.run.smithi088.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-30T14:10:11.862 INFO:teuthology.orchestra.run.smithi088.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-30T14:10:11.863 INFO:teuthology.orchestra.run.smithi088.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-30T14:10:11.863 INFO:teuthology.orchestra.run.smithi088.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-30T14:10:11.882 INFO:teuthology.orchestra.run.smithi088.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-30T14:10:11.995 INFO:teuthology.orchestra.run.smithi088.stdout:Get:36 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6835-g00c1c13c-1jammy [238 kB] 2023-10-30T14:10:12.001 INFO:teuthology.orchestra.run.smithi088.stdout:Get:37 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6835-g00c1c13c-1jammy [128 kB] 2023-10-30T14:10:12.005 INFO:teuthology.orchestra.run.smithi088.stdout:Get:38 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6835-g00c1c13c-1jammy [1,496 kB] 2023-10-30T14:10:12.026 INFO:teuthology.orchestra.run.smithi057.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-30T14:10:12.047 INFO:teuthology.orchestra.run.smithi088.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-30T14:10:12.057 INFO:teuthology.orchestra.run.smithi088.stdout:Get:40 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6835-g00c1c13c-1jammy [9,892 kB] 2023-10-30T14:10:12.442 INFO:teuthology.orchestra.run.smithi057.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-30T14:10:12.449 INFO:teuthology.orchestra.run.smithi057.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-30T14:10:12.451 INFO:teuthology.orchestra.run.smithi057.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-30T14:10:12.455 INFO:teuthology.orchestra.run.smithi088.stdout:Get:41 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6835-g00c1c13c-1jammy [31.7 MB] 2023-10-30T14:10:12.463 INFO:teuthology.orchestra.run.smithi057.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-30T14:10:12.463 INFO:teuthology.orchestra.run.smithi057.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-30T14:10:12.465 INFO:teuthology.orchestra.run.smithi057.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-30T14:10:12.467 INFO:teuthology.orchestra.run.smithi057.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-30T14:10:12.468 INFO:teuthology.orchestra.run.smithi057.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-30T14:10:12.470 INFO:teuthology.orchestra.run.smithi057.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-30T14:10:12.479 INFO:teuthology.orchestra.run.smithi088.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-30T14:10:12.488 INFO:teuthology.orchestra.run.smithi088.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-30T14:10:12.494 INFO:teuthology.orchestra.run.smithi088.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-30T14:10:12.498 INFO:teuthology.orchestra.run.smithi088.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-30T14:10:12.516 INFO:teuthology.orchestra.run.smithi088.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-30T14:10:12.519 INFO:teuthology.orchestra.run.smithi088.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-30T14:10:12.521 INFO:teuthology.orchestra.run.smithi088.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-30T14:10:12.563 INFO:teuthology.orchestra.run.smithi088.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-30T14:10:12.577 INFO:teuthology.orchestra.run.smithi088.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-30T14:10:12.581 INFO:teuthology.orchestra.run.smithi088.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-30T14:10:12.635 INFO:teuthology.orchestra.run.smithi057.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-30T14:10:12.835 INFO:teuthology.orchestra.run.smithi088.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-30T14:10:12.918 INFO:teuthology.orchestra.run.smithi088.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-30T14:10:13.008 INFO:teuthology.orchestra.run.smithi088.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-30T14:10:13.065 INFO:teuthology.orchestra.run.smithi088.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-30T14:10:13.080 INFO:teuthology.orchestra.run.smithi057.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-30T14:10:13.090 INFO:teuthology.orchestra.run.smithi057.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-30T14:10:13.096 INFO:teuthology.orchestra.run.smithi057.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-30T14:10:13.100 INFO:teuthology.orchestra.run.smithi057.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-30T14:10:13.119 INFO:teuthology.orchestra.run.smithi057.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-30T14:10:13.122 INFO:teuthology.orchestra.run.smithi057.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-30T14:10:13.124 INFO:teuthology.orchestra.run.smithi057.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-30T14:10:13.164 INFO:teuthology.orchestra.run.smithi057.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-30T14:10:13.179 INFO:teuthology.orchestra.run.smithi057.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-30T14:10:13.183 INFO:teuthology.orchestra.run.smithi088.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-30T14:10:13.184 INFO:teuthology.orchestra.run.smithi057.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-30T14:10:13.269 INFO:teuthology.orchestra.run.smithi088.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-30T14:10:13.287 INFO:teuthology.orchestra.run.smithi088.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-30T14:10:13.316 INFO:teuthology.orchestra.run.smithi088.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-30T14:10:13.328 INFO:teuthology.orchestra.run.smithi088.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-30T14:10:13.366 INFO:teuthology.orchestra.run.smithi088.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-30T14:10:13.374 INFO:teuthology.orchestra.run.smithi088.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-30T14:10:13.438 INFO:teuthology.orchestra.run.smithi057.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-30T14:10:13.521 INFO:teuthology.orchestra.run.smithi057.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-30T14:10:13.565 INFO:teuthology.orchestra.run.smithi088.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-30T14:10:13.614 INFO:teuthology.orchestra.run.smithi057.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-30T14:10:13.674 INFO:teuthology.orchestra.run.smithi057.stdout:Get:38 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6835-g00c1c13c-1jammy [3,131 kB] 2023-10-30T14:10:13.689 INFO:teuthology.orchestra.run.smithi057.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-30T14:10:13.701 INFO:teuthology.orchestra.run.smithi088.stdout:Get:64 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6835-g00c1c13c-1jammy [14.1 kB] 2023-10-30T14:10:13.702 INFO:teuthology.orchestra.run.smithi088.stdout:Get:65 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [839 kB] 2023-10-30T14:10:13.718 INFO:teuthology.orchestra.run.smithi088.stdout:Get:66 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6835-g00c1c13c-1jammy [2,148 kB] 2023-10-30T14:10:13.756 INFO:teuthology.orchestra.run.smithi088.stdout:Get:67 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6835-g00c1c13c-1jammy [284 kB] 2023-10-30T14:10:13.760 INFO:teuthology.orchestra.run.smithi088.stdout:Get:68 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6835-g00c1c13c-1jammy [138 kB] 2023-10-30T14:10:13.762 INFO:teuthology.orchestra.run.smithi088.stdout:Get:69 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6835-g00c1c13c-1jammy [1,907 kB] 2023-10-30T14:10:13.783 INFO:teuthology.orchestra.run.smithi088.stdout:Get:70 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6835-g00c1c13c-1jammy [8,625 kB] 2023-10-30T14:10:13.791 INFO:teuthology.orchestra.run.smithi057.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-30T14:10:13.879 INFO:teuthology.orchestra.run.smithi057.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-30T14:10:13.898 INFO:teuthology.orchestra.run.smithi057.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-30T14:10:13.928 INFO:teuthology.orchestra.run.smithi057.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-30T14:10:13.940 INFO:teuthology.orchestra.run.smithi057.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-30T14:10:13.977 INFO:teuthology.orchestra.run.smithi057.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-30T14:10:13.985 INFO:teuthology.orchestra.run.smithi057.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-30T14:10:14.069 INFO:teuthology.orchestra.run.smithi088.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6835-g00c1c13c-1jammy [14.2 kB] 2023-10-30T14:10:14.069 INFO:teuthology.orchestra.run.smithi088.stdout:Get:72 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6835-g00c1c13c-1jammy [47.8 MB] 2023-10-30T14:10:14.177 INFO:teuthology.orchestra.run.smithi057.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-30T14:10:14.374 INFO:teuthology.orchestra.run.smithi088.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-30T14:10:14.376 INFO:teuthology.orchestra.run.smithi088.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-30T14:10:14.378 INFO:teuthology.orchestra.run.smithi088.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-30T14:10:14.379 INFO:teuthology.orchestra.run.smithi088.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-30T14:10:14.380 INFO:teuthology.orchestra.run.smithi088.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-30T14:10:14.382 INFO:teuthology.orchestra.run.smithi088.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-30T14:10:14.421 INFO:teuthology.orchestra.run.smithi088.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-30T14:10:14.425 INFO:teuthology.orchestra.run.smithi088.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-30T14:10:14.426 INFO:teuthology.orchestra.run.smithi088.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-30T14:10:14.485 INFO:teuthology.orchestra.run.smithi088.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-30T14:10:14.650 INFO:teuthology.orchestra.run.smithi088.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-30T14:10:14.912 INFO:teuthology.orchestra.run.smithi088.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-30T14:10:14.945 INFO:teuthology.orchestra.run.smithi088.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-30T14:10:14.972 INFO:teuthology.orchestra.run.smithi088.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-30T14:10:15.008 INFO:teuthology.orchestra.run.smithi057.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-30T14:10:15.009 INFO:teuthology.orchestra.run.smithi057.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-30T14:10:15.011 INFO:teuthology.orchestra.run.smithi057.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-30T14:10:15.011 INFO:teuthology.orchestra.run.smithi057.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-30T14:10:15.012 INFO:teuthology.orchestra.run.smithi057.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-30T14:10:15.013 INFO:teuthology.orchestra.run.smithi057.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-30T14:10:15.022 INFO:teuthology.orchestra.run.smithi088.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-30T14:10:15.036 INFO:teuthology.orchestra.run.smithi057.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-30T14:10:15.039 INFO:teuthology.orchestra.run.smithi057.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-30T14:10:15.039 INFO:teuthology.orchestra.run.smithi057.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-30T14:10:15.043 INFO:teuthology.orchestra.run.smithi088.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-30T14:10:15.071 INFO:teuthology.orchestra.run.smithi088.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-30T14:10:15.109 INFO:teuthology.orchestra.run.smithi057.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-30T14:10:15.199 INFO:teuthology.orchestra.run.smithi088.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-30T14:10:15.231 INFO:teuthology.orchestra.run.smithi088.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-30T14:10:15.251 INFO:teuthology.orchestra.run.smithi088.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-30T14:10:15.252 INFO:teuthology.orchestra.run.smithi057.stdout:Get:58 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6835-g00c1c13c-1jammy [689 kB] 2023-10-30T14:10:15.259 INFO:teuthology.orchestra.run.smithi088.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-30T14:10:15.274 INFO:teuthology.orchestra.run.smithi057.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-30T14:10:15.463 INFO:teuthology.orchestra.run.smithi088.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-30T14:10:15.533 INFO:teuthology.orchestra.run.smithi057.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-30T14:10:15.565 INFO:teuthology.orchestra.run.smithi057.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-30T14:10:15.587 INFO:teuthology.orchestra.run.smithi057.stdout:Get:62 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6835-g00c1c13c-1jammy [356 kB] 2023-10-30T14:10:15.628 INFO:teuthology.orchestra.run.smithi057.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-30T14:10:15.662 INFO:teuthology.orchestra.run.smithi057.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-30T14:10:15.680 INFO:teuthology.orchestra.run.smithi057.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-30T14:10:15.707 INFO:teuthology.orchestra.run.smithi057.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-30T14:10:15.842 INFO:teuthology.orchestra.run.smithi057.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-30T14:10:15.854 INFO:teuthology.orchestra.run.smithi057.stdout:Get:68 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6835-g00c1c13c-1jammy [32.8 kB] 2023-10-30T14:10:15.863 INFO:teuthology.orchestra.run.smithi057.stdout:Get:69 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6835-g00c1c13c-1jammy [180 kB] 2023-10-30T14:10:15.876 INFO:teuthology.orchestra.run.smithi057.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-30T14:10:15.897 INFO:teuthology.orchestra.run.smithi057.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-30T14:10:15.904 INFO:teuthology.orchestra.run.smithi057.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-30T14:10:15.924 INFO:teuthology.orchestra.run.smithi088.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-10-30T14:10:15.929 INFO:teuthology.orchestra.run.smithi088.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-30T14:10:16.006 INFO:teuthology.orchestra.run.smithi057.stdout:Get:73 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6835-g00c1c13c-1jammy [62.4 kB] 2023-10-30T14:10:16.041 INFO:teuthology.orchestra.run.smithi057.stdout:Get:74 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6835-g00c1c13c-1jammy [330 kB] 2023-10-30T14:10:16.101 INFO:teuthology.orchestra.run.smithi088.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-30T14:10:16.108 INFO:teuthology.orchestra.run.smithi057.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-30T14:10:16.176 INFO:teuthology.orchestra.run.smithi088.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6835-g00c1c13c-1jammy [120 kB] 2023-10-30T14:10:16.177 INFO:teuthology.orchestra.run.smithi088.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-30T14:10:16.177 INFO:teuthology.orchestra.run.smithi088.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-30T14:10:16.178 INFO:teuthology.orchestra.run.smithi088.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-30T14:10:16.179 INFO:teuthology.orchestra.run.smithi088.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-6835-g00c1c13c-1jammy [227 kB] 2023-10-30T14:10:16.179 INFO:teuthology.orchestra.run.smithi088.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-30T14:10:16.183 INFO:teuthology.orchestra.run.smithi088.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-30T14:10:16.183 INFO:teuthology.orchestra.run.smithi088.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-6835-g00c1c13c-1jammy [34.0 kB] 2023-10-30T14:10:16.184 INFO:teuthology.orchestra.run.smithi088.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-6835-g00c1c13c-1jammy [23.4 kB] 2023-10-30T14:10:16.185 INFO:teuthology.orchestra.run.smithi088.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-30T14:10:16.185 INFO:teuthology.orchestra.run.smithi088.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6835-g00c1c13c-1jammy [38.3 kB] 2023-10-30T14:10:16.186 INFO:teuthology.orchestra.run.smithi088.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6835-g00c1c13c-1jammy [12.3 MB] 2023-10-30T14:10:16.196 INFO:teuthology.orchestra.run.smithi057.stdout:Get:76 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6835-g00c1c13c-1jammy [6,226 kB] 2023-10-30T14:10:16.202 INFO:teuthology.orchestra.run.smithi088.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-30T14:10:16.367 INFO:teuthology.orchestra.run.smithi088.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-30T14:10:16.533 INFO:teuthology.orchestra.run.smithi088.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-30T14:10:16.585 INFO:teuthology.orchestra.run.smithi057.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-10-30T14:10:16.590 INFO:teuthology.orchestra.run.smithi057.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-30T14:10:16.719 INFO:teuthology.orchestra.run.smithi057.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-30T14:10:16.765 INFO:teuthology.orchestra.run.smithi057.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-30T14:10:16.766 INFO:teuthology.orchestra.run.smithi057.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-30T14:10:16.772 INFO:teuthology.orchestra.run.smithi057.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-30T14:10:16.773 INFO:teuthology.orchestra.run.smithi057.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-30T14:10:16.789 INFO:teuthology.orchestra.run.smithi057.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-30T14:10:16.796 INFO:teuthology.orchestra.run.smithi057.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-30T14:10:16.810 INFO:teuthology.orchestra.run.smithi088.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-30T14:10:16.859 INFO:teuthology.orchestra.run.smithi057.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-30T14:10:16.870 INFO:teuthology.orchestra.run.smithi088.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-30T14:10:16.870 INFO:teuthology.orchestra.run.smithi088.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [91.6 kB] 2023-10-30T14:10:17.024 INFO:teuthology.orchestra.run.smithi057.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-30T14:10:17.190 INFO:teuthology.orchestra.run.smithi057.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-30T14:10:17.479 INFO:teuthology.orchestra.run.smithi057.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-30T14:10:17.505 INFO:teuthology.orchestra.run.smithi057.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-30T14:10:17.634 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 184 MB in 7s (26.9 MB/s) 2023-10-30T14:10:17.774 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pyperclip. 2023-10-30T14:10:17.852 INFO:teuthology.orchestra.run.smithi088.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 130506 files and directories currently installed.) 2023-10-30T14:10:17.857 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-10-30T14:10:17.897 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-10-30T14:10:18.159 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-30T14:10:18.179 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-30T14:10:18.208 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T14:10:18.470 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cmd2. 2023-10-30T14:10:18.490 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-10-30T14:10:18.528 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-10-30T14:10:18.823 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-30T14:10:18.843 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../003-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-30T14:10:18.923 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T14:10:18.953 INFO:teuthology.orchestra.run.smithi057.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6835-g00c1c13c-1jammy [112 kB] 2023-10-30T14:10:19.006 INFO:teuthology.orchestra.run.smithi057.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6835-g00c1c13c-1jammy [439 kB] 2023-10-30T14:10:19.211 INFO:teuthology.orchestra.run.smithi057.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6835-g00c1c13c-1jammy [22.6 MB] 2023-10-30T14:10:19.267 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-30T14:10:19.287 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../004-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-30T14:10:19.326 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T14:10:19.637 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-30T14:10:19.658 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../005-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-30T14:10:19.696 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T14:10:20.041 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-30T14:10:20.061 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../006-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-30T14:10:20.099 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T14:10:20.844 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-30T14:10:20.864 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../007-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T14:10:20.910 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T14:10:21.432 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-30T14:10:21.453 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../008-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T14:10:21.490 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T14:10:21.835 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-30T14:10:21.855 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../009-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T14:10:21.893 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T14:10:22.188 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-30T14:10:22.199 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../010-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-30T14:10:22.229 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T14:10:22.549 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package librados2. 2023-10-30T14:10:22.569 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../011-librados2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:22.607 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:22.794 INFO:teuthology.orchestra.run.smithi057.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6835-g00c1c13c-1jammy [6,858 kB] 2023-10-30T14:10:23.060 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package librbd1. 2023-10-30T14:10:23.080 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../012-librbd1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:23.135 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:23.534 INFO:teuthology.orchestra.run.smithi057.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6835-g00c1c13c-1jammy [238 kB] 2023-10-30T14:10:23.546 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libcephfs2. 2023-10-30T14:10:23.558 INFO:teuthology.orchestra.run.smithi057.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6835-g00c1c13c-1jammy [128 kB] 2023-10-30T14:10:23.566 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../013-libcephfs2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:23.577 INFO:teuthology.orchestra.run.smithi057.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6835-g00c1c13c-1jammy [1,496 kB] 2023-10-30T14:10:23.595 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:23.723 INFO:teuthology.orchestra.run.smithi057.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6835-g00c1c13c-1jammy [9,892 kB] 2023-10-30T14:10:23.840 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rados. 2023-10-30T14:10:23.852 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../014-python3-rados_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:23.890 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:24.251 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-30T14:10:24.271 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../015-python3-ceph-argparse_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:10:24.309 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:24.561 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cephfs. 2023-10-30T14:10:24.573 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../016-python3-cephfs_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:24.611 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:24.662 INFO:teuthology.orchestra.run.smithi057.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6835-g00c1c13c-1jammy [31.7 MB] 2023-10-30T14:10:24.914 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-30T14:10:24.926 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../017-python3-ceph-common_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:10:24.956 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:25.225 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-prettytable. 2023-10-30T14:10:25.246 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2023-10-30T14:10:25.290 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-30T14:10:25.545 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rbd. 2023-10-30T14:10:25.565 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../019-python3-rbd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:25.603 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:26.131 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-30T14:10:26.151 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-30T14:10:26.189 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T14:10:26.525 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package librgw2. 2023-10-30T14:10:26.545 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../021-librgw2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:26.692 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:27.187 INFO:teuthology.orchestra.run.smithi057.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6835-g00c1c13c-1jammy [14.1 kB] 2023-10-30T14:10:27.187 INFO:teuthology.orchestra.run.smithi057.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [839 kB] 2023-10-30T14:10:27.219 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rgw. 2023-10-30T14:10:27.236 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../022-python3-rgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:27.247 INFO:teuthology.orchestra.run.smithi057.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6835-g00c1c13c-1jammy [2,148 kB] 2023-10-30T14:10:27.277 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:27.413 INFO:teuthology.orchestra.run.smithi057.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6835-g00c1c13c-1jammy [284 kB] 2023-10-30T14:10:27.435 INFO:teuthology.orchestra.run.smithi057.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6835-g00c1c13c-1jammy [138 kB] 2023-10-30T14:10:27.439 INFO:teuthology.orchestra.run.smithi057.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6835-g00c1c13c-1jammy [1,907 kB] 2023-10-30T14:10:27.547 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-30T14:10:27.567 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../023-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-30T14:10:27.570 INFO:teuthology.orchestra.run.smithi057.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6835-g00c1c13c-1jammy [8,625 kB] 2023-10-30T14:10:27.596 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T14:10:27.949 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libradosstriper1. 2023-10-30T14:10:27.970 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../024-libradosstriper1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:28.008 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:28.150 INFO:teuthology.orchestra.run.smithi057.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6835-g00c1c13c-1jammy [14.2 kB] 2023-10-30T14:10:28.150 INFO:teuthology.orchestra.run.smithi057.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6835-g00c1c13c-1jammy [47.8 MB] 2023-10-30T14:10:28.536 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-common. 2023-10-30T14:10:28.557 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../025-ceph-common_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:28.602 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:30.025 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-base. 2023-10-30T14:10:30.045 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../026-ceph-base_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:30.109 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:30.828 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-30T14:10:30.849 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../027-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-30T14:10:30.887 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-30T14:10:30.999 INFO:teuthology.orchestra.run.smithi057.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6835-g00c1c13c-1jammy [120 kB] 2023-10-30T14:10:31.000 INFO:teuthology.orchestra.run.smithi057.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-6835-g00c1c13c-1jammy [227 kB] 2023-10-30T14:10:31.004 INFO:teuthology.orchestra.run.smithi057.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-6835-g00c1c13c-1jammy [34.0 kB] 2023-10-30T14:10:31.004 INFO:teuthology.orchestra.run.smithi057.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-6835-g00c1c13c-1jammy [23.4 kB] 2023-10-30T14:10:31.005 INFO:teuthology.orchestra.run.smithi057.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6835-g00c1c13c-1jammy [38.3 kB] 2023-10-30T14:10:31.023 INFO:teuthology.orchestra.run.smithi057.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6835-g00c1c13c-1jammy [12.3 MB] 2023-10-30T14:10:31.148 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cheroot. 2023-10-30T14:10:31.159 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../028-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-30T14:10:31.189 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T14:10:31.509 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-30T14:10:31.529 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../029-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-30T14:10:31.567 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-30T14:10:31.663 INFO:teuthology.orchestra.run.smithi057.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [91.6 kB] 2023-10-30T14:10:31.870 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-30T14:10:31.890 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../030-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-30T14:10:31.928 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-30T14:10:32.248 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-30T14:10:32.252 INFO:teuthology.orchestra.run.smithi057.stdout:Fetched 184 MB in 21s (8,816 kB/s) 2023-10-30T14:10:32.268 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../031-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-30T14:10:32.306 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-30T14:10:32.425 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pyperclip. 2023-10-30T14:10:32.495 INFO:teuthology.orchestra.run.smithi057.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132932 files and directories currently installed.) 2023-10-30T14:10:32.499 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-10-30T14:10:32.540 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-10-30T14:10:32.609 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-tempora. 2023-10-30T14:10:32.629 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../032-python3-tempora_4.1.2-1_all.deb ... 2023-10-30T14:10:32.667 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-30T14:10:32.843 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-30T14:10:32.857 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-30T14:10:32.926 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T14:10:32.970 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-portend. 2023-10-30T14:10:32.990 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../033-python3-portend_3.0.0-1_all.deb ... 2023-10-30T14:10:33.028 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-30T14:10:33.238 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-cmd2. 2023-10-30T14:10:33.252 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-10-30T14:10:33.296 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-10-30T14:10:33.339 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-30T14:10:33.360 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../034-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-30T14:10:33.397 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-30T14:10:33.641 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-30T14:10:33.655 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../003-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-30T14:10:33.692 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-30T14:10:33.713 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../035-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-30T14:10:33.741 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T14:10:33.742 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-30T14:10:34.028 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-30T14:10:34.042 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../004-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-30T14:10:34.095 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-natsort. 2023-10-30T14:10:34.103 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T14:10:34.116 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../036-python3-natsort_8.0.2-1_all.deb ... 2023-10-30T14:10:34.153 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-30T14:10:34.423 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-30T14:10:34.438 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../005-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-30T14:10:34.473 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-logutils. 2023-10-30T14:10:34.481 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T14:10:34.494 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../037-python3-logutils_0.3.3-8_all.deb ... 2023-10-30T14:10:34.531 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-30T14:10:34.801 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-30T14:10:34.813 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../038-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-30T14:10:34.842 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-30T14:10:34.851 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-30T14:10:34.866 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../006-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-30T14:10:34.910 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T14:10:35.120 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-mako. 2023-10-30T14:10:35.141 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../039-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-30T14:10:35.178 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T14:10:35.347 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-30T14:10:35.362 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../007-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T14:10:35.405 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T14:10:35.607 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-30T14:10:35.627 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../040-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-30T14:10:35.673 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-30T14:10:35.825 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-30T14:10:35.840 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../008-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T14:10:35.883 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T14:10:36.154 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-30T14:10:36.175 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../041-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-30T14:10:36.212 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-30T14:10:36.227 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../009-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T14:10:36.268 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-30T14:10:36.270 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T14:10:36.582 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-30T14:10:36.596 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../010-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-30T14:10:36.604 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-webob. 2023-10-30T14:10:36.624 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../042-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-30T14:10:36.640 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T14:10:36.662 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-30T14:10:36.952 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package librados2. 2023-10-30T14:10:36.967 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../011-librados2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:37.040 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-waitress. 2023-10-30T14:10:37.061 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../043-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-30T14:10:37.101 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T14:10:37.226 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:37.459 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-tempita. 2023-10-30T14:10:37.480 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-30T14:10:37.526 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T14:10:37.721 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package librbd1. 2023-10-30T14:10:37.736 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../012-librbd1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:37.779 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:37.895 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-paste. 2023-10-30T14:10:37.916 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../045-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-30T14:10:37.962 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T14:10:38.226 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libcephfs2. 2023-10-30T14:10:38.241 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../013-libcephfs2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:38.284 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:38.307 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-30T14:10:38.327 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-30T14:10:38.365 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T14:10:38.570 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-rados. 2023-10-30T14:10:38.585 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../014-python3-rados_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:38.626 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-30T14:10:38.628 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:38.638 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-30T14:10:38.676 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-30T14:10:38.979 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-webtest. 2023-10-30T14:10:38.989 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-30T14:10:38.999 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../048-python3-webtest_2.0.35-1_all.deb ... 2023-10-30T14:10:39.004 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../015-python3-ceph-argparse_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:10:39.038 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-30T14:10:39.047 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:39.325 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-cephfs. 2023-10-30T14:10:39.340 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../016-python3-cephfs_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:39.375 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:39.379 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pecan. 2023-10-30T14:10:39.400 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-30T14:10:39.440 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T14:10:39.644 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-30T14:10:39.659 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../017-python3-ceph-common_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:10:39.702 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:39.819 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-certifi. 2023-10-30T14:10:39.840 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../050-python3-certifi_2020.6.20-1_all.deb ... 2023-10-30T14:10:39.878 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-30T14:10:40.055 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-prettytable. 2023-10-30T14:10:40.070 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2023-10-30T14:10:40.113 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-30T14:10:40.222 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-idna. 2023-10-30T14:10:40.243 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../051-python3-idna_3.3-1_all.deb ... 2023-10-30T14:10:40.280 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-30T14:10:40.349 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-rbd. 2023-10-30T14:10:40.364 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../019-python3-rbd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:40.416 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:40.550 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-urllib3. 2023-10-30T14:10:40.562 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../052-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-30T14:10:40.600 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T14:10:40.785 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-30T14:10:40.801 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-30T14:10:40.843 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T14:10:40.886 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-requests. 2023-10-30T14:10:40.898 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../053-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-30T14:10:40.953 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T14:10:41.154 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package librgw2. 2023-10-30T14:10:41.169 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../021-librgw2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:41.204 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:41.256 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-30T14:10:41.277 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-30T14:10:41.314 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T14:10:41.583 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-30T14:10:41.595 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:10:41.633 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:41.682 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-rgw. 2023-10-30T14:10:41.697 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../022-python3-rgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:41.740 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:41.996 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-30T14:10:42.017 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:42.054 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:42.076 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-30T14:10:42.091 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../023-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-30T14:10:42.134 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T14:10:42.349 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-openssl. 2023-10-30T14:10:42.370 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../057-python3-openssl_21.0.0-1_all.deb ... 2023-10-30T14:10:42.399 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-30T14:10:42.487 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libradosstriper1. 2023-10-30T14:10:42.502 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../024-libradosstriper1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:42.537 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:42.907 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-common. 2023-10-30T14:10:42.922 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../025-ceph-common_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:42.965 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:43.211 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr. 2023-10-30T14:10:43.232 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../058-ceph-mgr_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:43.594 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:44.006 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mon. 2023-10-30T14:10:44.027 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../059-ceph-mon_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:44.064 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:44.679 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-base. 2023-10-30T14:10:44.695 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../026-ceph-base_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:44.747 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:44.959 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-osd. 2023-10-30T14:10:44.980 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../060-ceph-osd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:45.017 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:45.483 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-30T14:10:45.497 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../027-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-30T14:10:45.542 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-30T14:10:45.819 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-cheroot. 2023-10-30T14:10:45.834 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../028-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-30T14:10:45.869 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T14:10:46.181 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-30T14:10:46.195 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../029-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-30T14:10:46.230 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-30T14:10:46.525 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-30T14:10:46.540 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../030-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-30T14:10:46.583 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-30T14:10:46.902 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-30T14:10:46.918 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../031-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-30T14:10:46.961 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-30T14:10:47.095 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph. 2023-10-30T14:10:47.113 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../061-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:47.153 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:47.280 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-tempora. 2023-10-30T14:10:47.296 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../032-python3-tempora_4.1.2-1_all.deb ... 2023-10-30T14:10:47.330 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-30T14:10:47.481 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-fuse. 2023-10-30T14:10:47.502 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../062-ceph-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:47.539 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:47.791 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-portend. 2023-10-30T14:10:47.807 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../033-python3-portend_3.0.0-1_all.deb ... 2023-10-30T14:10:47.849 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-30T14:10:47.917 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mds. 2023-10-30T14:10:47.939 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../063-ceph-mds_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:47.984 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:48.152 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-30T14:10:48.168 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../034-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-30T14:10:48.210 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-30T14:10:48.362 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package cephadm. 2023-10-30T14:10:48.383 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../064-cephadm_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:48.428 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:48.538 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-30T14:10:48.553 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../035-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-30T14:10:48.596 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-30T14:10:48.798 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-30T14:10:48.819 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-30T14:10:48.856 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-30T14:10:48.908 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-natsort. 2023-10-30T14:10:48.923 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../036-python3-natsort_8.0.2-1_all.deb ... 2023-10-30T14:10:48.983 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-30T14:10:49.184 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-30T14:10:49.205 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../066-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T14:10:49.242 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T14:10:49.295 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-logutils. 2023-10-30T14:10:49.309 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../037-python3-logutils_0.3.3-8_all.deb ... 2023-10-30T14:10:49.353 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-30T14:10:49.672 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-30T14:10:49.687 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../038-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-30T14:10:49.730 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-30T14:10:50.041 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-mako. 2023-10-30T14:10:50.056 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../039-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-30T14:10:50.100 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T14:10:50.197 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-babel. 2023-10-30T14:10:50.210 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../067-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T14:10:50.238 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T14:10:50.411 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-30T14:10:50.426 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../040-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-30T14:10:50.470 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-30T14:10:50.575 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jinja2. 2023-10-30T14:10:50.596 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../068-python3-jinja2_3.0.3-1_all.deb ... 2023-10-30T14:10:50.633 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-30T14:10:50.781 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-30T14:10:50.796 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../041-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-30T14:10:50.838 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-30T14:10:51.003 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-30T14:10:51.024 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../069-ceph-mgr-cephadm_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:10:51.061 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:51.141 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-webob. 2023-10-30T14:10:51.157 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../042-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-30T14:10:51.208 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-30T14:10:51.390 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-30T14:10:51.410 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../070-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-30T14:10:51.448 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-30T14:10:51.586 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-waitress. 2023-10-30T14:10:51.601 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../043-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-30T14:10:51.647 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T14:10:51.709 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-30T14:10:51.730 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../071-python3-repoze.lru_0.7-2_all.deb ... 2023-10-30T14:10:51.759 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-30T14:10:51.963 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-tempita. 2023-10-30T14:10:51.979 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-30T14:10:52.022 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T14:10:52.045 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-routes. 2023-10-30T14:10:52.066 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../072-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-30T14:10:52.103 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T14:10:52.308 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-paste. 2023-10-30T14:10:52.324 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../045-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-30T14:10:52.357 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T14:10:52.407 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-30T14:10:52.428 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../073-ceph-mgr-dashboard_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:10:52.465 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:52.686 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-30T14:10:52.700 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-30T14:10:52.744 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T14:10:52.910 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-30T14:10:52.922 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-30T14:10:52.960 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T14:10:53.022 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-30T14:10:53.037 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-30T14:10:53.096 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-30T14:10:53.358 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-webtest. 2023-10-30T14:10:53.373 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../048-python3-webtest_2.0.35-1_all.deb ... 2023-10-30T14:10:53.407 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-30T14:10:53.472 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-joblib. 2023-10-30T14:10:53.493 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../075-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-30T14:10:53.530 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T14:10:53.736 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pecan. 2023-10-30T14:10:53.751 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-30T14:10:53.793 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T14:10:53.817 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-30T14:10:53.829 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../076-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-30T14:10:53.858 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-30T14:10:54.123 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-certifi. 2023-10-30T14:10:54.138 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../050-python3-certifi_2020.6.20-1_all.deb ... 2023-10-30T14:10:54.153 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-sklearn. 2023-10-30T14:10:54.174 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../077-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-30T14:10:54.181 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-30T14:10:54.211 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T14:10:54.576 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-idna. 2023-10-30T14:10:54.591 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../051-python3-idna_3.3-1_all.deb ... 2023-10-30T14:10:54.634 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-30T14:10:54.970 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-urllib3. 2023-10-30T14:10:54.986 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../052-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-30T14:10:55.020 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-30T14:10:55.028 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T14:10:55.041 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../078-ceph-mgr-diskprediction-local_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:10:55.086 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:55.394 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-requests. 2023-10-30T14:10:55.410 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../053-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-30T14:10:55.456 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T14:10:55.751 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-30T14:10:55.766 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-30T14:10:55.801 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T14:10:56.056 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cachetools. 2023-10-30T14:10:56.077 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../079-python3-cachetools_5.0.0-1_all.deb ... 2023-10-30T14:10:56.112 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-30T14:10:56.114 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-30T14:10:56.127 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:10:56.170 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:56.367 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-30T14:10:56.388 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../080-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-30T14:10:56.417 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-30T14:10:56.526 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-30T14:10:56.541 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:56.574 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:56.728 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-30T14:10:56.749 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../081-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-30T14:10:56.786 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T14:10:56.878 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-openssl. 2023-10-30T14:10:56.893 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../057-python3-openssl_21.0.0-1_all.deb ... 2023-10-30T14:10:56.936 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-30T14:10:57.272 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-mgr. 2023-10-30T14:10:57.287 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../058-ceph-mgr_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:57.330 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:57.331 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rsa. 2023-10-30T14:10:57.352 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../082-python3-rsa_4.8-1_all.deb ... 2023-10-30T14:10:57.389 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-30T14:10:57.683 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-mon. 2023-10-30T14:10:57.698 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../059-ceph-mon_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:57.749 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:57.778 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-google-auth. 2023-10-30T14:10:57.799 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../083-python3-google-auth_1.5.1-3_all.deb ... 2023-10-30T14:10:57.836 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-30T14:10:58.131 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-30T14:10:58.152 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../084-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-30T14:10:58.181 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T14:10:58.450 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-websocket. 2023-10-30T14:10:58.471 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../085-python3-websocket_1.2.3-1_all.deb ... 2023-10-30T14:10:58.508 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-30T14:10:58.528 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-osd. 2023-10-30T14:10:58.543 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../060-ceph-osd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:58.586 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:58.828 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-30T14:10:58.841 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../086-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-30T14:10:58.878 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T14:10:59.609 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-30T14:10:59.622 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../087-ceph-mgr-k8sevents_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:10:59.659 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:59.840 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph. 2023-10-30T14:10:59.855 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../061-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:10:59.898 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:10:59.928 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-30T14:10:59.950 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../088-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-30T14:10:59.986 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T14:11:00.251 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-fuse. 2023-10-30T14:11:00.266 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../062-ceph-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:11:00.301 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:00.342 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-30T14:11:00.363 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../089-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T14:11:00.408 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T14:11:00.644 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package jq. 2023-10-30T14:11:00.654 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-mds. 2023-10-30T14:11:00.656 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../090-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T14:11:00.669 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../063-ceph-mds_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:11:00.694 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-30T14:11:00.712 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:00.946 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package socat. 2023-10-30T14:11:00.959 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../091-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-30T14:11:00.996 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-30T14:11:01.098 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package cephadm. 2023-10-30T14:11:01.114 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../064-cephadm_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:11:01.148 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:01.274 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package xmlstarlet. 2023-10-30T14:11:01.287 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../092-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-30T14:11:01.324 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-30T14:11:01.460 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-30T14:11:01.475 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-30T14:11:01.518 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-30T14:11:01.645 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-test. 2023-10-30T14:11:01.657 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../093-ceph-test_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:11:01.686 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:01.812 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-30T14:11:01.827 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../066-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T14:11:01.862 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T14:11:02.867 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-babel. 2023-10-30T14:11:02.881 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../067-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T14:11:02.933 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T14:11:03.270 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-jinja2. 2023-10-30T14:11:03.284 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../068-python3-jinja2_3.0.3-1_all.deb ... 2023-10-30T14:11:03.328 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-30T14:11:03.656 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-30T14:11:03.670 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../069-ceph-mgr-cephadm_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:11:03.750 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:04.016 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-volume. 2023-10-30T14:11:04.037 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../094-ceph-volume_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:11:04.065 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:04.084 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-30T14:11:04.099 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../070-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-30T14:11:04.142 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-30T14:11:04.470 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-30T14:11:04.486 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../071-python3-repoze.lru_0.7-2_all.deb ... 2023-10-30T14:11:04.520 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-30T14:11:04.594 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package cephfs-mirror. 2023-10-30T14:11:04.616 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../095-cephfs-mirror_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:11:04.653 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:04.765 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-routes. 2023-10-30T14:11:04.780 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../072-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-30T14:11:04.822 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T14:11:04.964 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-colorama. 2023-10-30T14:11:04.985 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../096-python3-colorama_0.4.4-1_all.deb ... 2023-10-30T14:11:05.030 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-10-30T14:11:05.118 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-30T14:11:05.133 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../073-ceph-mgr-dashboard_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:11:05.168 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:05.292 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package cephfs-shell. 2023-10-30T14:11:05.313 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../097-cephfs-shell_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:11:05.358 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:05.579 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-30T14:11:05.594 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-30T14:11:05.627 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package cephfs-top. 2023-10-30T14:11:05.629 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T14:11:05.639 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../098-cephfs-top_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:11:05.677 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:05.921 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-30T14:11:05.933 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../099-libcephfs-dev_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:11:05.971 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:06.116 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-joblib. 2023-10-30T14:11:06.130 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../075-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-30T14:11:06.165 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T14:11:06.358 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package nvme-cli. 2023-10-30T14:11:06.379 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../100-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-30T14:11:06.424 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T14:11:06.503 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-30T14:11:06.517 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../076-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-30T14:11:06.561 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-30T14:11:06.779 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-30T14:11:06.801 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../101-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-30T14:11:06.854 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-30T14:11:06.872 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-sklearn. 2023-10-30T14:11:06.887 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../077-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-30T14:11:06.930 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T14:11:07.324 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-30T14:11:07.346 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../102-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-30T14:11:07.391 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-30T14:11:07.630 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-30T14:11:07.644 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../078-ceph-mgr-diskprediction-local_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:11:07.688 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:08.043 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pastescript. 2023-10-30T14:11:08.065 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-4_all.deb ... 2023-10-30T14:11:08.118 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-30T14:11:08.521 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pluggy. 2023-10-30T14:11:08.543 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../104-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-30T14:11:08.588 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-30T14:11:08.716 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-cachetools. 2023-10-30T14:11:08.732 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../079-python3-cachetools_5.0.0-1_all.deb ... 2023-10-30T14:11:08.774 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-30T14:11:08.974 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-psutil. 2023-10-30T14:11:08.996 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../105-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-30T14:11:09.041 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-30T14:11:09.085 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-30T14:11:09.100 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../080-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-30T14:11:09.144 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-30T14:11:09.410 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-py. 2023-10-30T14:11:09.432 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../106-python3-py_1.10.0-1_all.deb ... 2023-10-30T14:11:09.438 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-30T14:11:09.453 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../081-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-30T14:11:09.477 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-30T14:11:09.488 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T14:11:09.797 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pygments. 2023-10-30T14:11:09.799 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-rsa. 2023-10-30T14:11:09.815 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../082-python3-rsa_4.8-1_all.deb ... 2023-10-30T14:11:09.818 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../107-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-30T14:11:09.849 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-30T14:11:09.863 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T14:11:10.254 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-google-auth. 2023-10-30T14:11:10.270 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../083-python3-google-auth_1.5.1-3_all.deb ... 2023-10-30T14:11:10.312 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-30T14:11:10.367 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-30T14:11:10.389 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../108-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-30T14:11:10.434 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-30T14:11:10.615 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-30T14:11:10.631 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../084-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-30T14:11:10.673 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T14:11:10.703 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-toml. 2023-10-30T14:11:10.725 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../109-python3-toml_0.10.2-1_all.deb ... 2023-10-30T14:11:10.761 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-30T14:11:10.993 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-websocket. 2023-10-30T14:11:11.009 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../085-python3-websocket_1.2.3-1_all.deb ... 2023-10-30T14:11:11.043 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-30T14:11:11.064 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pytest. 2023-10-30T14:11:11.086 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../110-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-30T14:11:11.139 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T14:11:11.349 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-30T14:11:11.365 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../086-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-30T14:11:11.404 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T14:11:11.451 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-simplejson. 2023-10-30T14:11:11.472 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../111-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-30T14:11:11.509 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-30T14:11:11.761 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-30T14:11:11.774 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../112-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-30T14:11:11.811 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-30T14:11:12.118 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-30T14:11:12.133 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../087-ceph-mgr-k8sevents_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:11:12.176 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:12.545 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-30T14:11:12.559 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../088-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-30T14:11:12.604 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T14:11:12.624 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package radosgw. 2023-10-30T14:11:12.646 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../113-radosgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:11:12.698 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:12.882 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-30T14:11:12.896 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../089-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T14:11:12.932 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T14:11:13.202 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package jq. 2023-10-30T14:11:13.216 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../090-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T14:11:13.259 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-30T14:11:13.477 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package rbd-fuse. 2023-10-30T14:11:13.498 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../114-rbd-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:11:13.512 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package socat. 2023-10-30T14:11:13.527 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../091-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-30T14:11:13.543 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:13.570 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-30T14:11:13.895 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-30T14:11:14.140 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package xmlstarlet. 2023-10-30T14:11:14.156 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../092-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-30T14:11:14.182 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T14:11:14.198 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-30T14:11:14.283 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T14:11:14.544 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-30T14:11:14.554 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-test. 2023-10-30T14:11:14.568 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../093-ceph-test_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:11:14.612 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:14.874 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-30T14:11:15.346 INFO:teuthology.orchestra.run.smithi088.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-30T14:11:15.593 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:16.211 INFO:teuthology.orchestra.run.smithi088.stdout:Adding system user cephadm....done 2023-10-30T14:11:16.349 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T14:11:16.508 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-volume. 2023-10-30T14:11:16.524 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../094-ceph-volume_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:11:16.557 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:16.601 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-30T14:11:16.852 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-30T14:11:16.903 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package cephfs-mirror. 2023-10-30T14:11:16.919 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../095-cephfs-mirror_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:11:16.953 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:16.978 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-30T14:11:17.229 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-30T14:11:17.248 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-colorama. 2023-10-30T14:11:17.263 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../096-python3-colorama_0.4.4-1_all.deb ... 2023-10-30T14:11:17.306 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-10-30T14:11:17.480 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T14:11:17.590 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-30T14:11:17.592 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package cephfs-shell. 2023-10-30T14:11:17.608 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../097-cephfs-shell_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:11:17.650 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:17.894 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package cephfs-top. 2023-10-30T14:11:17.899 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T14:11:17.910 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../098-cephfs-top_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T14:11:17.953 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:18.189 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-30T14:11:18.205 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../099-libcephfs-dev_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:11:18.238 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:18.301 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-30T14:11:18.533 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package nvme-cli. 2023-10-30T14:11:18.549 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../100-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-30T14:11:18.560 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-10-30T14:11:18.583 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T14:11:18.820 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-30T14:11:18.913 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-30T14:11:18.928 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../101-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-30T14:11:18.963 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-30T14:11:19.121 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-30T14:11:19.381 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-30T14:11:19.399 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-30T14:11:19.414 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../102-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-30T14:11:19.448 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-30T14:11:19.632 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-30T14:11:19.735 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pastescript. 2023-10-30T14:11:19.750 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-4_all.deb ... 2023-10-30T14:11:19.818 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-30T14:11:19.966 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:20.138 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pluggy. 2023-10-30T14:11:20.154 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../104-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-30T14:11:20.196 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-30T14:11:20.225 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T14:11:20.351 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-10-30T14:11:20.532 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-psutil. 2023-10-30T14:11:20.548 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../105-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-30T14:11:20.590 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-30T14:11:20.628 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-30T14:11:20.879 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T14:11:20.927 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-py. 2023-10-30T14:11:20.943 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../106-python3-py_1.10.0-1_all.deb ... 2023-10-30T14:11:20.993 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-30T14:11:21.005 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-30T14:11:21.346 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pygments. 2023-10-30T14:11:21.357 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T14:11:21.362 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../107-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-30T14:11:21.404 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T14:11:21.465 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-30T14:11:21.708 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T14:11:21.883 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-30T14:11:21.898 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../108-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-30T14:11:21.933 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-30T14:11:21.951 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-30T14:11:22.244 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-toml. 2023-10-30T14:11:22.252 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T14:11:22.259 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../109-python3-toml_0.10.2-1_all.deb ... 2023-10-30T14:11:22.302 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-30T14:11:22.588 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pytest. 2023-10-30T14:11:22.603 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../110-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-30T14:11:22.646 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T14:11:22.929 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T14:11:22.999 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-simplejson. 2023-10-30T14:11:23.015 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../111-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-30T14:11:23.058 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-30T14:11:23.181 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T14:11:23.281 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-30T14:11:23.327 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-30T14:11:23.343 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../112-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-30T14:11:23.393 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-30T14:11:23.407 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T14:11:23.700 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-30T14:11:24.076 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-30T14:11:24.319 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T14:11:24.347 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package radosgw. 2023-10-30T14:11:24.363 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../113-radosgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:11:24.405 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:24.704 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T14:11:24.972 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-10-30T14:11:25.016 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package rbd-fuse. 2023-10-30T14:11:25.031 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../114-rbd-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T14:11:25.066 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:25.265 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T14:11:25.360 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-30T14:11:25.605 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T14:11:25.608 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-30T14:11:25.705 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T14:11:25.859 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-30T14:11:25.985 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:25.994 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-30T14:11:26.295 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-idna (3.3-1) ... 2023-10-30T14:11:26.471 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-30T14:11:26.613 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T14:11:27.060 INFO:teuthology.orchestra.run.smithi057.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-30T14:11:27.100 INFO:teuthology.orchestra.run.smithi057.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2023-10-30T14:11:27.140 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:27.744 INFO:teuthology.orchestra.run.smithi057.stdout:Adding system user cephadm....done 2023-10-30T14:11:27.802 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T14:11:27.889 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T14:11:27.936 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-30T14:11:28.187 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T14:11:28.214 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-30T14:11:28.288 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-30T14:11:28.389 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T14:11:28.465 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-30T14:11:28.591 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-30T14:11:28.674 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-30T14:11:28.893 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-30T14:11:28.925 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-30T14:11:29.169 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T14:11:29.176 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T14:11:29.327 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T14:11:29.453 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-30T14:11:29.511 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-30T14:11:29.746 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-30T14:11:29.844 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T14:11:30.006 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-30T14:11:30.223 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-30T14:11:30.257 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-30T14:11:30.499 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-10-30T14:11:30.508 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-30T14:11:30.759 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-30T14:11:30.767 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-30T14:11:31.018 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T14:11:31.126 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-30T14:11:31.145 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-30T14:11:31.403 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-30T14:11:31.421 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T14:11:31.530 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T14:11:31.678 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-30T14:11:31.798 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-30T14:11:31.929 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:32.074 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-30T14:11:32.214 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T14:11:32.317 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T14:11:32.339 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-10-30T14:11:32.607 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-30T14:11:32.714 INFO:teuthology.orchestra.run.smithi088.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-30T14:11:32.875 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T14:11:32.969 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T14:11:33.001 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-30T14:11:33.071 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T14:11:33.377 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T14:11:33.495 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-30T14:11:33.648 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-30T14:11:33.746 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T14:11:33.899 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-30T14:11:34.005 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-30T14:11:34.150 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T14:11:34.292 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-30T14:11:34.315 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T14:11:34.610 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T14:11:34.920 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T14:11:35.034 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T14:11:35.204 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-30T14:11:35.339 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-30T14:11:35.359 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T14:11:35.485 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-30T14:11:35.582 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-30T14:11:35.619 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T14:11:35.849 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T14:11:35.962 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-30T14:11:36.109 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-30T14:11:36.380 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-30T14:11:36.485 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-30T14:11:36.648 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T14:11:36.770 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T14:11:37.032 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T14:11:37.105 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T14:11:37.206 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:37.317 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-10-30T14:11:37.324 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:37.535 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:37.643 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T14:11:37.661 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:37.788 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:37.897 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-30T14:11:38.036 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-30T14:11:38.157 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:38.283 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:38.304 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-30T14:11:38.409 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:38.430 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:38.622 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T14:11:38.622 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T14:11:38.739 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-idna (3.3-1) ... 2023-10-30T14:11:39.090 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T14:11:39.144 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:39.287 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:39.405 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:39.506 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T14:11:40.404 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T14:11:40.514 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-30T14:11:40.689 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:40.781 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T14:11:40.807 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:40.908 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:40.932 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-30T14:11:41.017 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:41.033 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T14:11:41.143 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:41.269 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:41.334 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-30T14:11:41.611 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-30T14:11:41.812 INFO:teuthology.orchestra.run.smithi088.stdout:Adding group ceph....done 2023-10-30T14:11:41.895 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T14:11:42.021 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T14:11:42.155 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-30T14:11:42.163 INFO:teuthology.orchestra.run.smithi088.stdout:Adding system user ceph....done 2023-10-30T14:11:42.290 INFO:teuthology.orchestra.run.smithi088.stdout:Setting system user ceph properties....done 2023-10-30T14:11:42.306 INFO:teuthology.orchestra.run.smithi088.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-30T14:11:42.442 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-30T14:11:42.465 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-30T14:11:42.733 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-30T14:11:42.779 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-30T14:11:43.009 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-30T14:11:43.293 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-30T14:11:43.329 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:43.540 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T14:11:43.540 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T14:11:43.560 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-30T14:11:43.870 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T14:11:43.996 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-30T14:11:44.039 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:44.165 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:44.264 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T14:11:44.423 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T14:11:44.660 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T14:11:44.660 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T14:11:44.724 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-30T14:11:45.042 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-30T14:11:45.166 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:45.318 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T14:11:45.516 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-30T14:11:45.736 INFO:teuthology.orchestra.run.smithi057.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-30T14:11:46.012 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T14:11:46.043 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:46.164 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T14:11:46.248 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T14:11:46.249 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T14:11:46.573 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-30T14:11:46.761 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:46.841 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-30T14:11:46.988 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T14:11:46.989 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T14:11:47.100 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T14:11:47.226 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-30T14:11:47.529 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:47.602 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T14:11:47.844 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T14:11:47.844 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T14:11:47.928 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T14:11:48.213 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-30T14:11:48.314 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-30T14:11:48.348 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:48.494 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:48.589 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-30T14:11:48.649 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:48.866 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T14:11:48.880 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T14:11:48.881 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T14:11:49.142 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-30T14:11:49.376 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:49.502 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:49.543 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-30T14:11:49.639 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:49.790 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:49.878 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T14:11:50.126 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T14:11:50.272 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T14:11:50.397 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:50.532 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:50.734 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:50.894 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:50.976 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T14:11:51.020 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:51.146 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-30T14:11:51.455 INFO:teuthology.orchestra.run.smithi088.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T14:11:51.458 DEBUG:teuthology.orchestra.run.smithi088:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-10-30T14:11:51.464 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:51.553 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2023-10-30T14:11:51.590 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:51.716 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:51.740 INFO:teuthology.orchestra.run.smithi088.stdout:Building dependency tree... 2023-10-30T14:11:51.741 INFO:teuthology.orchestra.run.smithi088.stdout:Reading state information... 2023-10-30T14:11:51.893 INFO:teuthology.orchestra.run.smithi088.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-10-30T14:11:51.893 INFO:teuthology.orchestra.run.smithi088.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-10-30T14:11:51.894 INFO:teuthology.orchestra.run.smithi088.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-10-30T14:11:51.894 INFO:teuthology.orchestra.run.smithi088.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-10-30T14:11:51.894 INFO:teuthology.orchestra.run.smithi088.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-10-30T14:11:51.894 INFO:teuthology.orchestra.run.smithi088.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T14:11:51.894 INFO:teuthology.orchestra.run.smithi088.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T14:11:51.895 INFO:teuthology.orchestra.run.smithi088.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T14:11:51.896 INFO:teuthology.orchestra.run.smithi088.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T14:11:51.896 INFO:teuthology.orchestra.run.smithi088.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-30T14:11:51.896 INFO:teuthology.orchestra.run.smithi088.stdout: python2.7-minimal 2023-10-30T14:11:51.897 INFO:teuthology.orchestra.run.smithi088.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T14:11:51.897 INFO:teuthology.orchestra.run.smithi088.stdout:The following additional packages will be installed: 2023-10-30T14:11:51.897 INFO:teuthology.orchestra.run.smithi088.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-10-30T14:11:51.897 INFO:teuthology.orchestra.run.smithi088.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-10-30T14:11:51.897 INFO:teuthology.orchestra.run.smithi088.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-10-30T14:11:51.898 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-10-30T14:11:51.898 INFO:teuthology.orchestra.run.smithi088.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T14:11:51.898 INFO:teuthology.orchestra.run.smithi088.stdout:Suggested packages: 2023-10-30T14:11:51.899 INFO:teuthology.orchestra.run.smithi088.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-10-30T14:11:51.899 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-10-30T14:11:51.899 INFO:teuthology.orchestra.run.smithi088.stdout: libengine-pkcs11-openssl 2023-10-30T14:11:51.931 INFO:teuthology.orchestra.run.smithi088.stdout:The following NEW packages will be installed: 2023-10-30T14:11:51.932 INFO:teuthology.orchestra.run.smithi088.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-10-30T14:11:51.932 INFO:teuthology.orchestra.run.smithi088.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-10-30T14:11:51.932 INFO:teuthology.orchestra.run.smithi088.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-10-30T14:11:51.933 INFO:teuthology.orchestra.run.smithi088.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-10-30T14:11:51.933 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-10-30T14:11:51.933 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-10-30T14:11:51.933 INFO:teuthology.orchestra.run.smithi088.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T14:11:51.934 INFO:teuthology.orchestra.run.smithi088.stdout:The following packages will be upgraded: 2023-10-30T14:11:51.934 INFO:teuthology.orchestra.run.smithi088.stdout: libssl3 2023-10-30T14:11:51.935 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T14:11:51.935 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T14:11:51.984 INFO:teuthology.orchestra.run.smithi088.stdout:1 upgraded, 39 newly installed, 0 to remove and 259 not upgraded. 2023-10-30T14:11:51.984 INFO:teuthology.orchestra.run.smithi088.stdout:Need to get 54.6 MB of archives. 2023-10-30T14:11:51.984 INFO:teuthology.orchestra.run.smithi088.stdout:After this operation, 202 MB of additional disk space will be used. 2023-10-30T14:11:51.985 INFO:teuthology.orchestra.run.smithi088.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-10-30T14:11:52.083 INFO:teuthology.orchestra.run.smithi088.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-10-30T14:11:52.143 INFO:teuthology.orchestra.run.smithi088.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-10-30T14:11:52.157 INFO:teuthology.orchestra.run.smithi088.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-10-30T14:11:52.158 INFO:teuthology.orchestra.run.smithi088.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-10-30T14:11:52.161 INFO:teuthology.orchestra.run.smithi088.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-10-30T14:11:52.162 INFO:teuthology.orchestra.run.smithi088.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-10-30T14:11:52.163 INFO:teuthology.orchestra.run.smithi088.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-10-30T14:11:52.165 INFO:teuthology.orchestra.run.smithi088.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-10-30T14:11:52.166 INFO:teuthology.orchestra.run.smithi088.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-10-30T14:11:52.167 INFO:teuthology.orchestra.run.smithi088.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-10-30T14:11:52.167 INFO:teuthology.orchestra.run.smithi088.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-10-30T14:11:52.181 INFO:teuthology.orchestra.run.smithi088.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-10-30T14:11:52.498 INFO:teuthology.orchestra.run.smithi088.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-10-30T14:11:52.500 INFO:teuthology.orchestra.run.smithi088.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-10-30T14:11:52.500 INFO:teuthology.orchestra.run.smithi088.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-10-30T14:11:52.503 INFO:teuthology.orchestra.run.smithi088.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-10-30T14:11:52.503 INFO:teuthology.orchestra.run.smithi088.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-10-30T14:11:52.507 INFO:teuthology.orchestra.run.smithi088.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-10-30T14:11:52.508 INFO:teuthology.orchestra.run.smithi088.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-10-30T14:11:52.515 INFO:teuthology.orchestra.run.smithi088.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-10-30T14:11:52.516 INFO:teuthology.orchestra.run.smithi088.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-10-30T14:11:52.537 INFO:teuthology.orchestra.run.smithi088.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-10-30T14:11:52.537 INFO:teuthology.orchestra.run.smithi088.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-10-30T14:11:52.546 INFO:teuthology.orchestra.run.smithi088.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-10-30T14:11:52.566 INFO:teuthology.orchestra.run.smithi088.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-10-30T14:11:52.569 INFO:teuthology.orchestra.run.smithi088.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-10-30T14:11:52.570 INFO:teuthology.orchestra.run.smithi088.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-10-30T14:11:52.570 INFO:teuthology.orchestra.run.smithi088.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-10-30T14:11:52.571 INFO:teuthology.orchestra.run.smithi088.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-10-30T14:11:52.581 INFO:teuthology.orchestra.run.smithi088.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-10-30T14:11:52.582 INFO:teuthology.orchestra.run.smithi088.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-10-30T14:11:52.618 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:52.777 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:52.778 INFO:teuthology.orchestra.run.smithi088.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-10-30T14:11:52.779 INFO:teuthology.orchestra.run.smithi088.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-10-30T14:11:52.779 INFO:teuthology.orchestra.run.smithi088.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-10-30T14:11:52.779 INFO:teuthology.orchestra.run.smithi088.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-10-30T14:11:52.779 INFO:teuthology.orchestra.run.smithi088.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-10-30T14:11:52.780 INFO:teuthology.orchestra.run.smithi088.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-10-30T14:11:52.791 INFO:teuthology.orchestra.run.smithi088.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-10-30T14:11:52.791 INFO:teuthology.orchestra.run.smithi088.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-10-30T14:11:52.912 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:53.047 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T14:11:53.192 INFO:teuthology.orchestra.run.smithi088.stdout:Preconfiguring packages ... 2023-10-30T14:11:53.345 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 54.6 MB in 1s (64.1 MB/s) 2023-10-30T14:11:53.484 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package flex. 2023-10-30T14:11:53.571 INFO:teuthology.orchestra.run.smithi088.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137425 files and directories currently installed.) 2023-10-30T14:11:53.575 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-10-30T14:11:53.616 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking flex (2.6.4-8build2) ... 2023-10-30T14:11:54.268 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T14:11:54.435 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:54.473 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-10-30T14:11:54.594 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:54.720 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:54.846 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:54.912 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T14:11:54.997 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:55.123 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:55.466 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package bison. 2023-10-30T14:11:55.555 INFO:teuthology.orchestra.run.smithi088.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137498 files and directories currently installed.) 2023-10-30T14:11:55.559 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-10-30T14:11:55.609 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T14:11:55.635 INFO:teuthology.orchestra.run.smithi057.stdout:Adding group ceph....done 2023-10-30T14:11:55.986 INFO:teuthology.orchestra.run.smithi057.stdout:Adding system user ceph....done 2023-10-30T14:11:56.061 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-10-30T14:11:56.082 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-10-30T14:11:56.134 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T14:11:56.205 INFO:teuthology.orchestra.run.smithi057.stdout:Setting system user ceph properties....done 2023-10-30T14:11:56.225 INFO:teuthology.orchestra.run.smithi057.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-30T14:11:56.375 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-30T14:11:56.563 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package dump. 2023-10-30T14:11:56.585 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-10-30T14:11:56.637 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking dump (0.4b47-3) ... 2023-10-30T14:11:56.805 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-30T14:11:56.924 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package indent. 2023-10-30T14:11:56.937 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T14:11:56.974 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T14:11:57.243 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-10-30T14:11:57.256 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-10-30T14:11:57.294 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T14:11:57.499 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:57.581 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-10-30T14:11:57.602 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-10-30T14:11:57.647 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T14:11:57.698 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T14:11:57.698 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T14:11:57.908 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-10-30T14:11:57.920 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-10-30T14:11:57.958 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T14:11:58.292 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:58.303 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libfl2:amd64. 2023-10-30T14:11:58.324 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-10-30T14:11:58.369 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T14:11:58.451 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:11:58.630 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-10-30T14:11:58.643 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-10-30T14:11:58.672 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T14:11:58.908 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libjson-perl. 2023-10-30T14:11:58.929 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-10-30T14:11:58.999 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-10-30T14:11:59.038 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T14:11:59.038 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T14:11:59.437 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libllvm14:amd64. 2023-10-30T14:11:59.459 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-10-30T14:11:59.529 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T14:11:59.721 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:12:00.693 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-30T14:12:01.107 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-10-30T14:12:01.128 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T14:12:01.165 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T14:12:01.383 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:12:01.425 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libmbim-proxy. 2023-10-30T14:12:01.447 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T14:12:01.492 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T14:12:01.629 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T14:12:01.630 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T14:12:01.828 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-10-30T14:12:01.850 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T14:12:01.894 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T14:12:02.248 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libndp0:amd64. 2023-10-30T14:12:02.252 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:12:02.269 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-10-30T14:12:02.314 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T14:12:02.471 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T14:12:02.471 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T14:12:02.592 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libnm0:amd64. 2023-10-30T14:12:02.614 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T14:12:02.658 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T14:12:03.020 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libpq5:amd64. 2023-10-30T14:12:03.041 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T14:12:03.094 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T14:12:03.111 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:12:03.448 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-10-30T14:12:03.469 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T14:12:03.477 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T14:12:03.478 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T14:12:03.514 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T14:12:03.808 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libqmi-proxy. 2023-10-30T14:12:03.830 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T14:12:03.883 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T14:12:04.111 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-10-30T14:12:04.123 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T14:12:04.138 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:12:04.160 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T14:12:04.264 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:12:04.440 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:12:04.658 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T14:12:04.658 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T14:12:04.831 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-10-30T14:12:04.854 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-10-30T14:12:05.190 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T14:12:05.324 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:12:05.542 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package modemmanager. 2023-10-30T14:12:05.565 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T14:12:05.601 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T14:12:05.671 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:12:05.797 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:12:05.956 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T14:12:06.240 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package network-manager. 2023-10-30T14:12:06.262 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T14:12:06.315 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-10-30T14:12:06.326 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T14:12:06.939 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ppp. 2023-10-30T14:12:06.961 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-10-30T14:12:07.008 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T14:12:07.283 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T14:12:07.480 INFO:teuthology.orchestra.run.smithi057.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T14:12:07.484 DEBUG:teuthology.orchestra.run.smithi057:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-10-30T14:12:07.561 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2023-10-30T14:12:07.610 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package pptp-linux. 2023-10-30T14:12:07.632 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-10-30T14:12:07.668 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-10-30T14:12:07.731 INFO:teuthology.orchestra.run.smithi057.stdout:Building dependency tree... 2023-10-30T14:12:07.732 INFO:teuthology.orchestra.run.smithi057.stdout:Reading state information... 2023-10-30T14:12:07.888 INFO:teuthology.orchestra.run.smithi057.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-10-30T14:12:07.888 INFO:teuthology.orchestra.run.smithi057.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-10-30T14:12:07.889 INFO:teuthology.orchestra.run.smithi057.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-10-30T14:12:07.889 INFO:teuthology.orchestra.run.smithi057.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-10-30T14:12:07.889 INFO:teuthology.orchestra.run.smithi057.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-10-30T14:12:07.889 INFO:teuthology.orchestra.run.smithi057.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T14:12:07.889 INFO:teuthology.orchestra.run.smithi057.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T14:12:07.889 INFO:teuthology.orchestra.run.smithi057.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T14:12:07.890 INFO:teuthology.orchestra.run.smithi057.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T14:12:07.890 INFO:teuthology.orchestra.run.smithi057.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-30T14:12:07.891 INFO:teuthology.orchestra.run.smithi057.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-30T14:12:07.891 INFO:teuthology.orchestra.run.smithi057.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T14:12:07.892 INFO:teuthology.orchestra.run.smithi057.stdout:The following additional packages will be installed: 2023-10-30T14:12:07.892 INFO:teuthology.orchestra.run.smithi057.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-10-30T14:12:07.892 INFO:teuthology.orchestra.run.smithi057.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-10-30T14:12:07.892 INFO:teuthology.orchestra.run.smithi057.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-10-30T14:12:07.892 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-10-30T14:12:07.892 INFO:teuthology.orchestra.run.smithi057.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T14:12:07.893 INFO:teuthology.orchestra.run.smithi057.stdout:Suggested packages: 2023-10-30T14:12:07.893 INFO:teuthology.orchestra.run.smithi057.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-10-30T14:12:07.893 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-10-30T14:12:07.893 INFO:teuthology.orchestra.run.smithi057.stdout: libengine-pkcs11-openssl 2023-10-30T14:12:07.913 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package network-manager-pptp. 2023-10-30T14:12:07.934 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-10-30T14:12:07.941 INFO:teuthology.orchestra.run.smithi057.stdout:The following NEW packages will be installed: 2023-10-30T14:12:07.942 INFO:teuthology.orchestra.run.smithi057.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-10-30T14:12:07.942 INFO:teuthology.orchestra.run.smithi057.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-10-30T14:12:07.943 INFO:teuthology.orchestra.run.smithi057.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-10-30T14:12:07.943 INFO:teuthology.orchestra.run.smithi057.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-10-30T14:12:07.943 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-10-30T14:12:07.943 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-10-30T14:12:07.943 INFO:teuthology.orchestra.run.smithi057.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T14:12:07.944 INFO:teuthology.orchestra.run.smithi057.stdout:The following packages will be upgraded: 2023-10-30T14:12:07.944 INFO:teuthology.orchestra.run.smithi057.stdout: libssl3 2023-10-30T14:12:07.979 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-10-30T14:12:08.283 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package postgresql-client-common. 2023-10-30T14:12:08.304 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-10-30T14:12:08.349 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking postgresql-client-common (238) ... 2023-10-30T14:12:08.653 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package postgresql-client-14. 2023-10-30T14:12:08.675 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T14:12:08.719 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T14:12:09.299 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package postgresql-common. 2023-10-30T14:12:09.321 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-10-30T14:12:09.383 INFO:teuthology.orchestra.run.smithi088.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-10-30T14:12:09.413 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking postgresql-common (238) ... 2023-10-30T14:12:09.882 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package postgresql-14. 2023-10-30T14:12:09.904 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T14:12:09.952 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T14:12:11.561 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package postgresql. 2023-10-30T14:12:11.584 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-10-30T14:12:11.636 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking postgresql (14+238) ... 2023-10-30T14:12:11.939 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package postgresql-client. 2023-10-30T14:12:11.961 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-10-30T14:12:12.005 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking postgresql-client (14+238) ... 2023-10-30T14:12:12.266 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package postgresql-contrib. 2023-10-30T14:12:12.288 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-10-30T14:12:12.340 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking postgresql-contrib (14+238) ... 2023-10-30T14:12:12.718 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package tcl8.6. 2023-10-30T14:12:12.740 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-10-30T14:12:12.793 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T14:12:13.138 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package tcl. 2023-10-30T14:12:13.159 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-10-30T14:12:13.217 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-10-30T14:12:13.591 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package wpasupplicant. 2023-10-30T14:12:13.612 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-10-30T14:12:13.665 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T14:12:14.070 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-10-30T14:12:14.092 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-10-30T14:12:14.144 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-10-30T14:12:14.145 INFO:teuthology.orchestra.run.smithi057.stdout:1 upgraded, 39 newly installed, 0 to remove and 259 not upgraded. 2023-10-30T14:12:14.145 INFO:teuthology.orchestra.run.smithi057.stdout:Need to get 54.6 MB of archives. 2023-10-30T14:12:14.146 INFO:teuthology.orchestra.run.smithi057.stdout:After this operation, 202 MB of additional disk space will be used. 2023-10-30T14:12:14.146 INFO:teuthology.orchestra.run.smithi057.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-10-30T14:12:14.266 INFO:teuthology.orchestra.run.smithi057.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-10-30T14:12:14.338 INFO:teuthology.orchestra.run.smithi057.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-10-30T14:12:14.353 INFO:teuthology.orchestra.run.smithi057.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-10-30T14:12:14.354 INFO:teuthology.orchestra.run.smithi057.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-10-30T14:12:14.355 INFO:teuthology.orchestra.run.smithi057.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-10-30T14:12:14.357 INFO:teuthology.orchestra.run.smithi057.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-10-30T14:12:14.357 INFO:teuthology.orchestra.run.smithi057.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-10-30T14:12:14.358 INFO:teuthology.orchestra.run.smithi057.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-10-30T14:12:14.359 INFO:teuthology.orchestra.run.smithi057.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-10-30T14:12:14.360 INFO:teuthology.orchestra.run.smithi057.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-10-30T14:12:14.367 INFO:teuthology.orchestra.run.smithi057.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-10-30T14:12:14.383 INFO:teuthology.orchestra.run.smithi057.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-10-30T14:12:14.615 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package usb-modeswitch. 2023-10-30T14:12:14.637 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-10-30T14:12:14.698 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T14:12:14.711 INFO:teuthology.orchestra.run.smithi057.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-10-30T14:12:14.713 INFO:teuthology.orchestra.run.smithi057.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-10-30T14:12:14.713 INFO:teuthology.orchestra.run.smithi057.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-10-30T14:12:14.716 INFO:teuthology.orchestra.run.smithi057.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-10-30T14:12:14.716 INFO:teuthology.orchestra.run.smithi057.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-10-30T14:12:14.721 INFO:teuthology.orchestra.run.smithi057.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-10-30T14:12:14.722 INFO:teuthology.orchestra.run.smithi057.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-10-30T14:12:14.731 INFO:teuthology.orchestra.run.smithi057.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-10-30T14:12:14.731 INFO:teuthology.orchestra.run.smithi057.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-10-30T14:12:14.753 INFO:teuthology.orchestra.run.smithi057.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-10-30T14:12:14.757 INFO:teuthology.orchestra.run.smithi057.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-10-30T14:12:14.767 INFO:teuthology.orchestra.run.smithi057.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-10-30T14:12:14.793 INFO:teuthology.orchestra.run.smithi057.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-10-30T14:12:14.797 INFO:teuthology.orchestra.run.smithi057.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-10-30T14:12:14.797 INFO:teuthology.orchestra.run.smithi057.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-10-30T14:12:14.798 INFO:teuthology.orchestra.run.smithi057.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-10-30T14:12:14.799 INFO:teuthology.orchestra.run.smithi057.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-10-30T14:12:14.811 INFO:teuthology.orchestra.run.smithi057.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-10-30T14:12:14.816 INFO:teuthology.orchestra.run.smithi057.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-10-30T14:12:14.990 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up postgresql-client-common (238) ... 2023-10-30T14:12:15.041 INFO:teuthology.orchestra.run.smithi057.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-10-30T14:12:15.041 INFO:teuthology.orchestra.run.smithi057.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-10-30T14:12:15.041 INFO:teuthology.orchestra.run.smithi057.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-10-30T14:12:15.042 INFO:teuthology.orchestra.run.smithi057.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-10-30T14:12:15.042 INFO:teuthology.orchestra.run.smithi057.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-10-30T14:12:15.042 INFO:teuthology.orchestra.run.smithi057.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-10-30T14:12:15.056 INFO:teuthology.orchestra.run.smithi057.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-10-30T14:12:15.056 INFO:teuthology.orchestra.run.smithi057.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-10-30T14:12:15.196 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up flex (2.6.4-8build2) ... 2023-10-30T14:12:15.314 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T14:12:15.454 INFO:teuthology.orchestra.run.smithi057.stdout:Preconfiguring packages ... 2023-10-30T14:12:15.618 INFO:teuthology.orchestra.run.smithi057.stdout:Fetched 54.6 MB in 7s (7,684 kB/s) 2023-10-30T14:12:15.640 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T14:12:15.767 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package flex. 2023-10-30T14:12:15.783 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T14:12:15.842 INFO:teuthology.orchestra.run.smithi057.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 139851 files and directories currently installed.) 2023-10-30T14:12:15.845 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-10-30T14:12:15.893 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking flex (2.6.4-8build2) ... 2023-10-30T14:12:15.909 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T14:12:16.035 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T14:12:16.436 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T14:12:16.571 INFO:teuthology.orchestra.run.smithi088.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-10-30T14:12:16.639 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-10-30T14:12:16.666 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T14:12:17.045 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T14:12:17.054 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T14:12:17.055 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T14:12:17.560 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T14:12:17.598 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package bison. 2023-10-30T14:12:17.674 INFO:teuthology.orchestra.run.smithi057.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 139924 files and directories currently installed.) 2023-10-30T14:12:17.678 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-10-30T14:12:17.686 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T14:12:17.726 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T14:12:17.812 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-10-30T14:12:17.947 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T14:12:18.073 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T14:12:18.199 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T14:12:18.226 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-10-30T14:12:18.241 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-10-30T14:12:18.285 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T14:12:18.300 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T14:12:18.672 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package dump. 2023-10-30T14:12:18.687 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-10-30T14:12:18.730 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking dump (0.4b47-3) ... 2023-10-30T14:12:18.939 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T14:12:19.025 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package indent. 2023-10-30T14:12:19.040 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T14:12:19.065 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up dump (0.4b47-3) ... 2023-10-30T14:12:19.083 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T14:12:19.159 INFO:teuthology.orchestra.run.smithi088.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-10-30T14:12:19.241 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T14:12:19.343 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T14:12:19.344 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-10-30T14:12:19.359 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-10-30T14:12:19.403 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T14:12:19.468 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up tcl (8.6.11+1build2) ... 2023-10-30T14:12:19.594 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libjson-perl (4.04000-1) ... 2023-10-30T14:12:19.720 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T14:12:19.757 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-10-30T14:12:19.773 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-10-30T14:12:19.811 INFO:teuthology.orchestra.run.smithi088.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-10-30T14:12:19.815 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T14:12:19.872 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T14:12:19.998 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T14:12:20.060 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-10-30T14:12:20.076 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-10-30T14:12:20.126 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T14:12:20.174 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T14:12:20.284 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T14:12:20.393 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-10-30T14:12:20.463 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libfl2:amd64. 2023-10-30T14:12:20.479 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-10-30T14:12:20.521 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T14:12:20.561 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up postgresql-client (14+238) ... 2023-10-30T14:12:20.687 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-10-30T14:12:20.808 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-10-30T14:12:20.823 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-10-30T14:12:20.866 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T14:12:21.118 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libjson-perl. 2023-10-30T14:12:21.134 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-10-30T14:12:21.177 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-10-30T14:12:21.276 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:12:21.276 INFO:teuthology.orchestra.run.smithi088.stdout:The following network interfaces were found in /etc/network/interfaces 2023-10-30T14:12:21.276 INFO:teuthology.orchestra.run.smithi088.stdout:which means they are currently configured by ifupdown: 2023-10-30T14:12:21.276 INFO:teuthology.orchestra.run.smithi088.stdout:- enp3s0f1 2023-10-30T14:12:21.277 INFO:teuthology.orchestra.run.smithi088.stdout:If you want to manage those interfaces with NetworkManager instead 2023-10-30T14:12:21.277 INFO:teuthology.orchestra.run.smithi088.stdout:remove their configuration from /etc/network/interfaces. 2023-10-30T14:12:21.277 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:12:21.393 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-10-30T14:12:21.505 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libllvm14:amd64. 2023-10-30T14:12:21.521 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-10-30T14:12:21.563 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T14:12:21.760 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-10-30T14:12:22.103 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-10-30T14:12:22.631 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T14:12:22.791 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T14:12:22.892 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-10-30T14:12:22.992 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-10-30T14:12:23.008 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T14:12:23.018 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up postgresql-common (238) ... 2023-10-30T14:12:23.050 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T14:12:23.344 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libmbim-proxy. 2023-10-30T14:12:23.360 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T14:12:23.402 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T14:12:23.739 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-10-30T14:12:23.754 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T14:12:23.797 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T14:12:23.951 INFO:teuthology.orchestra.run.smithi088.stdout:Adding user postgres to group ssl-cert 2023-10-30T14:12:24.131 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:12:24.131 INFO:teuthology.orchestra.run.smithi088.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-10-30T14:12:24.142 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libndp0:amd64. 2023-10-30T14:12:24.158 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-10-30T14:12:24.200 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T14:12:24.293 INFO:teuthology.orchestra.run.smithi088.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-10-30T14:12:24.294 INFO:teuthology.orchestra.run.smithi088.stdout:Removing obsolete dictionary files: 2023-10-30T14:12:24.536 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libnm0:amd64. 2023-10-30T14:12:24.552 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T14:12:24.594 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T14:12:24.873 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-10-30T14:12:24.914 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libpq5:amd64. 2023-10-30T14:12:24.930 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T14:12:24.972 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T14:12:25.270 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T14:12:25.325 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-10-30T14:12:25.341 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T14:12:25.383 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T14:12:25.587 INFO:teuthology.orchestra.run.smithi088.stdout:Creating new PostgreSQL cluster 14/main ... 2023-10-30T14:12:25.612 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-10-30T14:12:25.650 INFO:teuthology.orchestra.run.smithi088.stdout:The files belonging to this database system will be owned by user "postgres". 2023-10-30T14:12:25.650 INFO:teuthology.orchestra.run.smithi088.stdout:This user must also own the server process. 2023-10-30T14:12:25.651 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:12:25.651 INFO:teuthology.orchestra.run.smithi088.stdout:The database cluster will be initialized with locale "en_US". 2023-10-30T14:12:25.651 INFO:teuthology.orchestra.run.smithi088.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-10-30T14:12:25.651 INFO:teuthology.orchestra.run.smithi088.stdout:The default text search configuration will be set to "english". 2023-10-30T14:12:25.651 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:12:25.652 INFO:teuthology.orchestra.run.smithi088.stdout:Data page checksums are disabled. 2023-10-30T14:12:25.652 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:12:25.652 INFO:teuthology.orchestra.run.smithi088.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-10-30T14:12:25.652 INFO:teuthology.orchestra.run.smithi088.stdout:creating subdirectories ... ok 2023-10-30T14:12:25.652 INFO:teuthology.orchestra.run.smithi088.stdout:selecting dynamic shared memory implementation ... posix 2023-10-30T14:12:25.663 INFO:teuthology.orchestra.run.smithi088.stdout:selecting default max_connections ... 100 2023-10-30T14:12:25.683 INFO:teuthology.orchestra.run.smithi088.stdout:selecting default shared_buffers ... 128MB 2023-10-30T14:12:25.692 INFO:teuthology.orchestra.run.smithi088.stdout:selecting default time zone ... Etc/UTC 2023-10-30T14:12:25.693 INFO:teuthology.orchestra.run.smithi088.stdout:creating configuration files ... ok 2023-10-30T14:12:25.694 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libqmi-proxy. 2023-10-30T14:12:25.710 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T14:12:25.753 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T14:12:25.805 INFO:teuthology.orchestra.run.smithi088.stdout:running bootstrap script ... ok 2023-10-30T14:12:25.964 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-10-30T14:12:25.980 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T14:12:26.014 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T14:12:26.440 INFO:teuthology.orchestra.run.smithi088.stdout:performing post-bootstrap initialization ... ok 2023-10-30T14:12:26.576 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-10-30T14:12:26.591 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-10-30T14:12:26.667 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T14:12:27.020 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package modemmanager. 2023-10-30T14:12:27.036 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T14:12:27.078 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T14:12:27.432 INFO:teuthology.orchestra.run.smithi088.stdout:syncing data to disk ... ok 2023-10-30T14:12:27.702 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package network-manager. 2023-10-30T14:12:27.718 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T14:12:27.768 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-10-30T14:12:27.836 INFO:teuthology.orchestra.run.smithi088.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-10-30T14:12:28.335 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ppp. 2023-10-30T14:12:28.351 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-10-30T14:12:28.725 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T14:12:29.106 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package pptp-linux. 2023-10-30T14:12:29.121 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-10-30T14:12:29.164 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-10-30T14:12:29.417 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package network-manager-pptp. 2023-10-30T14:12:29.432 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-10-30T14:12:29.475 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-10-30T14:12:29.778 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package postgresql-client-common. 2023-10-30T14:12:29.793 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-10-30T14:12:29.836 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking postgresql-client-common (238) ... 2023-10-30T14:12:30.157 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package postgresql-client-14. 2023-10-30T14:12:30.173 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T14:12:30.215 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T14:12:30.627 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T14:12:30.769 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up postgresql-contrib (14+238) ... 2023-10-30T14:12:30.821 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package postgresql-common. 2023-10-30T14:12:30.836 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-10-30T14:12:30.895 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up postgresql (14+238) ... 2023-10-30T14:12:30.898 INFO:teuthology.orchestra.run.smithi057.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-10-30T14:12:30.928 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking postgresql-common (238) ... 2023-10-30T14:12:31.021 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T14:12:31.147 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T14:12:31.403 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package postgresql-14. 2023-10-30T14:12:31.404 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T14:12:31.404 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T14:12:31.420 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T14:12:31.473 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T14:12:32.050 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-10-30T14:12:32.360 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T14:12:32.520 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T14:12:32.633 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package postgresql. 2023-10-30T14:12:32.648 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-10-30T14:12:32.682 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking postgresql (14+238) ... 2023-10-30T14:12:32.918 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package postgresql-client. 2023-10-30T14:12:32.933 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-10-30T14:12:32.976 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking postgresql-client (14+238) ... 2023-10-30T14:12:33.254 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package postgresql-contrib. 2023-10-30T14:12:33.269 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-10-30T14:12:33.312 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking postgresql-contrib (14+238) ... 2023-10-30T14:12:33.631 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package tcl8.6. 2023-10-30T14:12:33.647 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-10-30T14:12:33.690 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T14:12:33.832 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-10-30T14:12:33.932 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-10-30T14:12:33.959 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package tcl. 2023-10-30T14:12:33.975 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-10-30T14:12:34.023 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-10-30T14:12:34.404 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package wpasupplicant. 2023-10-30T14:12:34.420 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-10-30T14:12:34.462 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T14:12:34.593 INFO:teuthology.orchestra.run.smithi088.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T14:12:34.597 DEBUG:teuthology.parallel:result is None 2023-10-30T14:12:34.800 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-10-30T14:12:34.816 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-10-30T14:12:34.858 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-10-30T14:12:35.195 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package usb-modeswitch. 2023-10-30T14:12:35.212 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-10-30T14:12:35.262 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T14:12:35.572 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up postgresql-client-common (238) ... 2023-10-30T14:12:35.785 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up flex (2.6.4-8build2) ... 2023-10-30T14:12:35.927 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T14:12:36.062 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T14:12:36.179 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T14:12:36.314 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T14:12:36.440 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T14:12:37.466 INFO:teuthology.orchestra.run.smithi057.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-10-30T14:12:37.572 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T14:12:37.992 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T14:12:37.993 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T14:12:38.717 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T14:12:38.842 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T14:12:38.968 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-10-30T14:12:39.120 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T14:12:39.221 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T14:12:39.347 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T14:12:39.473 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T14:12:40.179 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T14:12:40.280 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up dump (0.4b47-3) ... 2023-10-30T14:12:40.384 INFO:teuthology.orchestra.run.smithi057.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-10-30T14:12:40.448 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T14:12:40.574 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T14:12:40.700 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up tcl (8.6.11+1build2) ... 2023-10-30T14:12:40.851 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libjson-perl (4.04000-1) ... 2023-10-30T14:12:40.977 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T14:12:41.067 INFO:teuthology.orchestra.run.smithi057.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-10-30T14:12:41.137 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T14:12:41.247 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T14:12:41.449 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T14:12:41.575 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T14:12:41.897 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-10-30T14:12:42.227 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up postgresql-client (14+238) ... 2023-10-30T14:12:42.353 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-10-30T14:12:42.947 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:12:42.947 INFO:teuthology.orchestra.run.smithi057.stdout:The following network interfaces were found in /etc/network/interfaces 2023-10-30T14:12:42.947 INFO:teuthology.orchestra.run.smithi057.stdout:which means they are currently configured by ifupdown: 2023-10-30T14:12:42.948 INFO:teuthology.orchestra.run.smithi057.stdout:- ens1f0 2023-10-30T14:12:42.948 INFO:teuthology.orchestra.run.smithi057.stdout:If you want to manage those interfaces with NetworkManager instead 2023-10-30T14:12:42.948 INFO:teuthology.orchestra.run.smithi057.stdout:remove their configuration from /etc/network/interfaces. 2023-10-30T14:12:42.948 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:12:43.087 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-10-30T14:12:43.496 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-10-30T14:12:43.918 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-10-30T14:12:44.614 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T14:12:44.741 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T14:12:44.867 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-10-30T14:12:45.001 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up postgresql-common (238) ... 2023-10-30T14:12:45.961 INFO:teuthology.orchestra.run.smithi057.stdout:Adding user postgres to group ssl-cert 2023-10-30T14:12:46.198 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:12:46.198 INFO:teuthology.orchestra.run.smithi057.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-10-30T14:12:46.484 INFO:teuthology.orchestra.run.smithi057.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-10-30T14:12:46.484 INFO:teuthology.orchestra.run.smithi057.stdout:Removing obsolete dictionary files: 2023-10-30T14:12:47.219 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-10-30T14:12:47.687 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T14:12:48.032 INFO:teuthology.orchestra.run.smithi057.stdout:Creating new PostgreSQL cluster 14/main ... 2023-10-30T14:12:48.057 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-10-30T14:12:48.151 INFO:teuthology.orchestra.run.smithi057.stdout:The files belonging to this database system will be owned by user "postgres". 2023-10-30T14:12:48.152 INFO:teuthology.orchestra.run.smithi057.stdout:This user must also own the server process. 2023-10-30T14:12:48.152 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:12:48.152 INFO:teuthology.orchestra.run.smithi057.stdout:The database cluster will be initialized with locale "en_US". 2023-10-30T14:12:48.152 INFO:teuthology.orchestra.run.smithi057.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-10-30T14:12:48.152 INFO:teuthology.orchestra.run.smithi057.stdout:The default text search configuration will be set to "english". 2023-10-30T14:12:48.153 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:12:48.153 INFO:teuthology.orchestra.run.smithi057.stdout:Data page checksums are disabled. 2023-10-30T14:12:48.153 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:12:48.153 INFO:teuthology.orchestra.run.smithi057.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-10-30T14:12:48.154 INFO:teuthology.orchestra.run.smithi057.stdout:creating subdirectories ... ok 2023-10-30T14:12:48.154 INFO:teuthology.orchestra.run.smithi057.stdout:selecting dynamic shared memory implementation ... posix 2023-10-30T14:12:48.178 INFO:teuthology.orchestra.run.smithi057.stdout:selecting default max_connections ... 100 2023-10-30T14:12:48.215 INFO:teuthology.orchestra.run.smithi057.stdout:selecting default shared_buffers ... 128MB 2023-10-30T14:12:48.229 INFO:teuthology.orchestra.run.smithi057.stdout:selecting default time zone ... Etc/UTC 2023-10-30T14:12:48.230 INFO:teuthology.orchestra.run.smithi057.stdout:creating configuration files ... ok 2023-10-30T14:12:48.408 INFO:teuthology.orchestra.run.smithi057.stdout:running bootstrap script ... ok 2023-10-30T14:12:49.050 INFO:teuthology.orchestra.run.smithi057.stdout:performing post-bootstrap initialization ... ok 2023-10-30T14:12:49.601 INFO:teuthology.orchestra.run.smithi057.stdout:syncing data to disk ... ok 2023-10-30T14:12:50.145 INFO:teuthology.orchestra.run.smithi057.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-10-30T14:12:53.520 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T14:12:53.646 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up postgresql-contrib (14+238) ... 2023-10-30T14:12:53.781 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up postgresql (14+238) ... 2023-10-30T14:12:53.907 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T14:12:54.041 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T14:12:54.310 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T14:12:54.310 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T14:12:55.043 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-10-30T14:12:55.370 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T14:12:55.539 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T14:12:57.018 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-10-30T14:12:57.118 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-10-30T14:12:57.847 INFO:teuthology.orchestra.run.smithi057.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T14:12:57.852 DEBUG:teuthology.parallel:result is None 2023-10-30T14:12:57.853 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T14:12:58.182 DEBUG:teuthology.orchestra.run.smithi057:> dpkg-query -W -f '${Version}' ceph 2023-10-30T14:12:58.205 INFO:teuthology.orchestra.run.smithi057.stdout:18.0.0-6835-g00c1c13c-1jammy 2023-10-30T14:12:58.205 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T14:12:58.206 INFO:teuthology.task.install:The correct ceph version 18.0.0-6835-g00c1c13c-1jammy is installed. 2023-10-30T14:12:58.206 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T14:12:58.352 DEBUG:teuthology.orchestra.run.smithi088:> dpkg-query -W -f '${Version}' ceph 2023-10-30T14:12:58.372 INFO:teuthology.orchestra.run.smithi088.stdout:18.0.0-6835-g00c1c13c-1jammy 2023-10-30T14:12:58.372 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T14:12:58.372 INFO:teuthology.task.install:The correct ceph version 18.0.0-6835-g00c1c13c-1jammy is installed. 2023-10-30T14:12:58.373 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-30T14:12:58.373 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:12:58.374 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T14:12:58.395 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:12:58.396 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T14:12:58.428 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-30T14:12:58.429 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:12:58.429 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T14:12:58.457 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T14:12:58.525 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:12:58.525 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T14:12:58.538 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T14:12:58.592 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-30T14:12:58.592 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:12:58.592 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T14:12:58.613 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T14:12:58.677 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:12:58.677 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T14:12:58.690 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T14:12:58.743 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-30T14:12:58.744 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:12:58.744 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T14:12:58.765 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T14:12:58.824 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:12:58.825 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T14:12:58.838 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T14:12:58.891 INFO:teuthology.run_tasks:Running task ceph... 2023-10-30T14:12:58.948 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-30T14:12:58.949 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 777 /var/log/ceph 2023-10-30T14:12:58.953 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 777 /var/log/ceph 2023-10-30T14:12:58.970 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-30T14:12:58.970 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T14:12:59.016 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T14:12:59.033 INFO:tasks.ceph:Creating extra log directories... 2023-10-30T14:12:59.033 DEBUG:teuthology.orchestra.run.smithi057:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T14:12:59.080 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T14:12:59.103 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-30T14:12:59.103 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-30T14:12:59.104 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-30T14:12:59.104 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:12:59.104 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-30T14:12:59.149 DEBUG:teuthology.orchestra.run.smithi057:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-30T14:12:59.241 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:12:59.241 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-30T14:12:59.247 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-30T14:12:59.325 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-30T14:12:59.325 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-30T14:12:59.326 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441053', 'branch': 'wip-lusov-balance-after-stop', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': None, 'first_in_suite': False, 'job_id': '7441053', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-lusov-balance-after-stop'}, 'ceph': {'cephfs': {'max_mds': 3, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}}, 'kclient': {'syntax': 'v1'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-balance-after-stop', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 8397, 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71', 'sleep_before_teardown': 0, 'subset': '464/1024', 'suite': 'fs', 'suite_branch': 'wip-lusov-balance-after-stop', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '00c1c13cea21a686b046584596fbeac5315a7d71', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi057.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIMwmje95SbNyfN/QhIqW+xTGxgbFqqhqVuHX1ojoZwDT7O/oIBYa4ThMFPh2Waxbow/mVMbY8cM9aUazaL1ENE=', 'smithi088.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPPr4j6c4Elj4JMggLxzU+oXj+Zbox5W/SOrDnjdPhsyNdfo/+FImaEbo+jin9aqDlqe5Rnspl/1O6/Th177wnE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71', 'cluster': 'ceph'}}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-10-30_06:34:16', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-10-30T14:12:59.327 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T14:12:59.330 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T14:12:59.369 DEBUG:teuthology.orchestra.run.smithi057:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T14:12:59.384 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T14:12:59.425 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:12:59.425 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T14:12:59.457 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T14:12:59.458 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_1 2023-10-30T14:12:59.513 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T14:12:59.514 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T14:12:59.514 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 879 Links: 1 2023-10-30T14:12:59.514 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T14:12:59.514 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-10-30 14:06:25.561835711 +0000 2023-10-30T14:12:59.515 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-10-30 14:06:25.181839104 +0000 2023-10-30T14:12:59.515 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-10-30 14:06:25.181839104 +0000 2023-10-30T14:12:59.515 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2023-10-30 14:06:25.181839104 +0000 2023-10-30T14:12:59.515 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T14:12:59.576 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-10-30T14:12:59.576 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-10-30T14:12:59.576 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000678279 s, 755 kB/s 2023-10-30T14:12:59.578 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T14:12:59.635 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_2 2023-10-30T14:12:59.689 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T14:12:59.690 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T14:12:59.690 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 894 Links: 1 2023-10-30T14:12:59.690 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T14:12:59.690 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-10-30 14:06:26.217829851 +0000 2023-10-30T14:12:59.690 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-10-30 14:06:25.829833316 +0000 2023-10-30T14:12:59.691 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-10-30 14:06:25.829833316 +0000 2023-10-30T14:12:59.691 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2023-10-30 14:06:25.829833316 +0000 2023-10-30T14:12:59.691 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T14:12:59.752 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-10-30T14:12:59.752 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-10-30T14:12:59.752 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000407667 s, 1.3 MB/s 2023-10-30T14:12:59.754 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T14:12:59.812 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_3 2023-10-30T14:12:59.866 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T14:12:59.866 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T14:12:59.866 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 911 Links: 1 2023-10-30T14:12:59.866 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T14:12:59.867 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-10-30 14:06:26.901823741 +0000 2023-10-30T14:12:59.867 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-10-30 14:06:26.497827349 +0000 2023-10-30T14:12:59.867 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-10-30 14:06:26.497827349 +0000 2023-10-30T14:12:59.867 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2023-10-30 14:06:26.497827349 +0000 2023-10-30T14:12:59.868 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T14:12:59.928 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-10-30T14:12:59.928 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-10-30T14:12:59.928 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000410763 s, 1.2 MB/s 2023-10-30T14:12:59.930 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T14:12:59.983 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_4 2023-10-30T14:13:00.034 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T14:13:00.034 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T14:13:00.035 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 930 Links: 1 2023-10-30T14:13:00.035 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T14:13:00.035 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-10-30 14:06:27.157821454 +0000 2023-10-30T14:13:00.035 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-10-30 14:06:27.153821490 +0000 2023-10-30T14:13:00.035 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-10-30 14:06:27.153821490 +0000 2023-10-30T14:13:00.036 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2023-10-30 14:06:27.153821490 +0000 2023-10-30T14:13:00.036 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T14:13:00.100 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-10-30T14:13:00.100 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-10-30T14:13:00.100 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000428447 s, 1.2 MB/s 2023-10-30T14:13:00.102 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T14:13:00.159 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T14:13:00.159 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:13:00.160 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T14:13:00.166 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T14:13:00.166 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_1 2023-10-30T14:13:00.213 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T14:13:00.213 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T14:13:00.213 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 889 Links: 1 2023-10-30T14:13:00.214 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T14:13:00.214 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-10-30 14:05:54.917479678 +0000 2023-10-30T14:13:00.214 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-10-30 14:05:54.557483098 +0000 2023-10-30T14:13:00.214 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-10-30 14:05:54.557483098 +0000 2023-10-30T14:13:00.214 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-10-30T14:13:00.215 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T14:13:00.268 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-10-30T14:13:00.268 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-10-30T14:13:00.268 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000522007 s, 981 kB/s 2023-10-30T14:13:00.269 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T14:13:00.318 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_2 2023-10-30T14:13:00.365 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T14:13:00.365 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T14:13:00.365 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 907 Links: 1 2023-10-30T14:13:00.365 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T14:13:00.366 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-10-30 14:05:55.509474055 +0000 2023-10-30T14:13:00.366 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-10-30 14:05:55.161477361 +0000 2023-10-30T14:13:00.366 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-10-30 14:05:55.161477361 +0000 2023-10-30T14:13:00.366 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-10-30T14:13:00.366 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T14:13:00.419 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-10-30T14:13:00.419 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-10-30T14:13:00.419 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000478959 s, 1.1 MB/s 2023-10-30T14:13:00.420 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T14:13:00.470 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_3 2023-10-30T14:13:00.516 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T14:13:00.517 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T14:13:00.517 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 922 Links: 1 2023-10-30T14:13:00.517 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T14:13:00.517 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-10-30 14:05:56.093468508 +0000 2023-10-30T14:13:00.518 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-10-30 14:05:55.745471813 +0000 2023-10-30T14:13:00.518 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-10-30 14:05:55.745471813 +0000 2023-10-30T14:13:00.518 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-10-30T14:13:00.518 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T14:13:00.571 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-10-30T14:13:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-10-30T14:13:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000534633 s, 958 kB/s 2023-10-30T14:13:00.573 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T14:13:00.622 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_4 2023-10-30T14:13:00.669 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T14:13:00.669 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T14:13:00.669 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 946 Links: 1 2023-10-30T14:13:00.670 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T14:13:00.670 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-10-30 14:05:56.689462849 +0000 2023-10-30T14:13:00.670 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-10-30 14:05:56.361465964 +0000 2023-10-30T14:13:00.670 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-10-30 14:05:56.361465964 +0000 2023-10-30T14:13:00.671 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-10-30T14:13:00.671 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T14:13:00.723 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-10-30T14:13:00.723 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-10-30T14:13:00.723 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000513601 s, 997 kB/s 2023-10-30T14:13:00.725 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T14:13:00.774 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T14:13:00.775 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi057.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T14:13:00.775 INFO:tasks.ceph:Generating config... 2023-10-30T14:13:00.776 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-30T14:13:00.777 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2023-10-30T14:13:00.777 INFO:tasks.ceph:[client] debug client = 20 2023-10-30T14:13:00.777 INFO:tasks.ceph:[client] debug ms = 1 2023-10-30T14:13:00.777 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-30T14:13:00.778 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-30T14:13:00.778 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-10-30T14:13:00.778 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-30T14:13:00.779 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-10-30T14:13:00.779 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-30T14:13:00.779 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2023-10-30T14:13:00.779 INFO:tasks.ceph:[mds] mds bal merge size = 5 2023-10-30T14:13:00.779 INFO:tasks.ceph:[mds] mds bal split bits = 3 2023-10-30T14:13:00.780 INFO:tasks.ceph:[mds] mds bal split size = 100 2023-10-30T14:13:00.780 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-30T14:13:00.780 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-30T14:13:00.780 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2023-10-30T14:13:00.781 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-30T14:13:00.781 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-30T14:13:00.781 INFO:tasks.ceph:[mds] mds_dir_prefetch = False 2023-10-30T14:13:00.781 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2023-10-30T14:13:00.782 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2023-10-30T14:13:00.782 INFO:tasks.ceph:[mds] ms inject delay max = 1 2023-10-30T14:13:00.782 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2023-10-30T14:13:00.782 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2023-10-30T14:13:00.783 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2023-10-30T14:13:00.783 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-30T14:13:00.783 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-30T14:13:00.783 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-30T14:13:00.784 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-30T14:13:00.784 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-30T14:13:00.784 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-30T14:13:00.784 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-30T14:13:00.785 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-30T14:13:00.785 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-30T14:13:00.785 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-30T14:13:00.785 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-30T14:13:00.786 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-10-30T14:13:00.786 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-30T14:13:00.786 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-30T14:13:00.786 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-30T14:13:00.787 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-30T14:13:00.787 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-30T14:13:00.787 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-30T14:13:00.788 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-30T14:13:00.788 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2023-10-30T14:13:00.788 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-30T14:13:00.788 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-30T14:13:00.789 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-30T14:13:00.789 INFO:tasks.ceph:[osd] ms inject delay max = 1 2023-10-30T14:13:00.789 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2023-10-30T14:13:00.789 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2023-10-30T14:13:00.789 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2023-10-30T14:13:00.790 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-10-30T14:13:00.790 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-30T14:13:00.790 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-10-30T14:13:00.790 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-30T14:13:00.791 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-30T14:13:00.791 INFO:tasks.ceph:Setting up mon.a... 2023-10-30T14:13:00.791 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-30T14:13:00.841 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/ceph.keyring 2023-10-30T14:13:00.944 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-30T14:13:00.995 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T14:13:01.016 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.57'), ('mon.b', '172.21.15.88'), ('mon.c', '[v2:172.21.15.88:3301,v1:172.21.15.88:6790]')] 2023-10-30T14:13:01.017 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.57,172.21.15.88,[v2:172.21.15.88:3301,v1:172.21.15.88:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-30T14:13:01.018 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:13:01.018 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T14:13:01.066 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.57 --add b 172.21.15.88 --addv c '[v2:172.21.15.88:3301,v1:172.21.15.88:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-30T14:13:01.157 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-30T14:13:01.158 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool: generated fsid e4ade530-0fca-4330-9824-3adb64478f80 2023-10-30T14:13:01.158 INFO:teuthology.orchestra.run.smithi057.stdout:setting min_mon_release = pacific 2023-10-30T14:13:01.158 INFO:teuthology.orchestra.run.smithi057.stdout:epoch 0 2023-10-30T14:13:01.159 INFO:teuthology.orchestra.run.smithi057.stdout:fsid e4ade530-0fca-4330-9824-3adb64478f80 2023-10-30T14:13:01.159 INFO:teuthology.orchestra.run.smithi057.stdout:last_changed 2023-10-30T14:13:01.158480+0000 2023-10-30T14:13:01.159 INFO:teuthology.orchestra.run.smithi057.stdout:created 2023-10-30T14:13:01.158480+0000 2023-10-30T14:13:01.159 INFO:teuthology.orchestra.run.smithi057.stdout:min_mon_release 16 (pacific) 2023-10-30T14:13:01.159 INFO:teuthology.orchestra.run.smithi057.stdout:election_strategy: 1 2023-10-30T14:13:01.160 INFO:teuthology.orchestra.run.smithi057.stdout:0: [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] mon.a 2023-10-30T14:13:01.160 INFO:teuthology.orchestra.run.smithi057.stdout:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2023-10-30T14:13:01.160 INFO:teuthology.orchestra.run.smithi057.stdout:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.c 2023-10-30T14:13:01.160 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-30T14:13:01.162 DEBUG:teuthology.orchestra.run.smithi057:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T14:13:01.171 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e4ade530-0fca-4330-9824-3adb64478f80... 2023-10-30T14:13:01.172 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T14:13:01.216 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T14:13:01.250 INFO:teuthology.orchestra.run.smithi088.stdout:[global] 2023-10-30T14:13:01.250 INFO:teuthology.orchestra.run.smithi088.stdout: chdir = "" 2023-10-30T14:13:01.250 INFO:teuthology.orchestra.run.smithi088.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T14:13:01.250 INFO:teuthology.orchestra.run.smithi088.stdout: auth supported = cephx 2023-10-30T14:13:01.251 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.251 INFO:teuthology.orchestra.run.smithi088.stdout: filestore xattr use omap = true 2023-10-30T14:13:01.251 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.251 INFO:teuthology.orchestra.run.smithi088.stdout: mon clock drift allowed = 1.000 2023-10-30T14:13:01.251 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.252 INFO:teuthology.orchestra.run.smithi088.stdout: osd crush chooseleaf type = 0 2023-10-30T14:13:01.252 INFO:teuthology.orchestra.run.smithi088.stdout: auth debug = true 2023-10-30T14:13:01.252 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.252 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on old message = true 2023-10-30T14:13:01.252 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on bug = true 2023-10-30T14:13:01.253 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.253 INFO:teuthology.orchestra.run.smithi088.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T14:13:01.253 INFO:teuthology.orchestra.run.smithi088.stdout: mon pg warn max object skew = 0 2023-10-30T14:13:01.253 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.253 INFO:teuthology.orchestra.run.smithi088.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T14:13:01.254 INFO:teuthology.orchestra.run.smithi088.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T14:13:01.254 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.254 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default size = 2 2023-10-30T14:13:01.254 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.254 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow primary affinity = true 2023-10-30T14:13:01.254 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow pg remap = true 2023-10-30T14:13:01.255 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on legacy crush tunables = false 2023-10-30T14:13:01.255 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on crush straw calc version zero = false 2023-10-30T14:13:01.255 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on no sortbitwise = false 2023-10-30T14:13:01.255 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on osd down out interval zero = false 2023-10-30T14:13:01.255 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on too few osds = false 2023-10-30T14:13:01.256 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T14:13:01.256 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T14:13:01.256 INFO:teuthology.orchestra.run.smithi088.stdout: mon_allow_pool_size_one = true 2023-10-30T14:13:01.256 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.257 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T14:13:01.257 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.257 INFO:teuthology.orchestra.run.smithi088.stdout: osd default data pool replay window = 5 2023-10-30T14:13:01.257 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.257 INFO:teuthology.orchestra.run.smithi088.stdout: mon allow pool delete = true 2023-10-30T14:13:01.258 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.258 INFO:teuthology.orchestra.run.smithi088.stdout: mon cluster log file level = debug 2023-10-30T14:13:01.258 INFO:teuthology.orchestra.run.smithi088.stdout: debug asserts on shutdown = true 2023-10-30T14:13:01.258 INFO:teuthology.orchestra.run.smithi088.stdout: mon health detail to clog = false 2023-10-30T14:13:01.258 INFO:teuthology.orchestra.run.smithi088.stdout: mon host = "172.21.15.57,172.21.15.88,[v2:172.21.15.88:3301,v1:172.21.15.88:6790]" 2023-10-30T14:13:01.259 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on skipped message = False 2023-10-30T14:13:01.259 INFO:teuthology.orchestra.run.smithi088.stdout: fsid = e4ade530-0fca-4330-9824-3adb64478f80 2023-10-30T14:13:01.259 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.259 INFO:teuthology.orchestra.run.smithi088.stdout:[osd] 2023-10-30T14:13:01.259 INFO:teuthology.orchestra.run.smithi088.stdout: osd journal size = 100 2023-10-30T14:13:01.260 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.260 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub load threshold = 5.0 2023-10-30T14:13:01.260 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub max interval = 600 2023-10-30T14:13:01.260 INFO:teuthology.orchestra.run.smithi088.stdout: osd mclock profile = high_recovery_ops 2023-10-30T14:13:01.260 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.261 INFO:teuthology.orchestra.run.smithi088.stdout: osd recover clone overlap = true 2023-10-30T14:13:01.261 INFO:teuthology.orchestra.run.smithi088.stdout: osd recovery max chunk = 1048576 2023-10-30T14:13:01.261 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.261 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug shutdown = true 2023-10-30T14:13:01.261 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug op order = true 2023-10-30T14:13:01.261 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify stray on activate = true 2023-10-30T14:13:01.262 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.262 INFO:teuthology.orchestra.run.smithi088.stdout: osd open classes on start = true 2023-10-30T14:13:01.262 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug pg log writeout = true 2023-10-30T14:13:01.262 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.262 INFO:teuthology.orchestra.run.smithi088.stdout: osd deep scrub update digest min age = 30 2023-10-30T14:13:01.263 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.263 INFO:teuthology.orchestra.run.smithi088.stdout: osd map max advance = 10 2023-10-30T14:13:01.263 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.263 INFO:teuthology.orchestra.run.smithi088.stdout: journal zero on create = true 2023-10-30T14:13:01.263 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.264 INFO:teuthology.orchestra.run.smithi088.stdout: filestore ondisk finisher threads = 3 2023-10-30T14:13:01.264 INFO:teuthology.orchestra.run.smithi088.stdout: filestore apply finisher threads = 3 2023-10-30T14:13:01.264 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.264 INFO:teuthology.orchestra.run.smithi088.stdout: bdev debug aio = true 2023-10-30T14:13:01.264 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug misdirected ops = true 2023-10-30T14:13:01.265 INFO:teuthology.orchestra.run.smithi088.stdout: bdev async discard = True 2023-10-30T14:13:01.265 INFO:teuthology.orchestra.run.smithi088.stdout: bdev enable discard = True 2023-10-30T14:13:01.265 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore allocator = bitmap 2023-10-30T14:13:01.265 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore block size = 96636764160 2023-10-30T14:13:01.265 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore fsck on mount = True 2023-10-30T14:13:01.265 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluefs = 1/20 2023-10-30T14:13:01.266 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluestore = 1/20 2023-10-30T14:13:01.266 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-10-30T14:13:01.266 INFO:teuthology.orchestra.run.smithi088.stdout: debug osd = 20 2023-10-30T14:13:01.266 INFO:teuthology.orchestra.run.smithi088.stdout: debug rocksdb = 4/10 2023-10-30T14:13:01.266 INFO:teuthology.orchestra.run.smithi088.stdout: mon client directed command retry = 5 2023-10-30T14:13:01.267 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T14:13:01.267 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd full ratio = 0.9 2023-10-30T14:13:01.267 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd nearfull ratio = 0.8 2023-10-30T14:13:01.267 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject delay max = 1 2023-10-30T14:13:01.267 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject delay probability = 0.005 2023-10-30T14:13:01.267 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject delay type = client mds 2023-10-30T14:13:01.268 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject socket failures = 2500 2023-10-30T14:13:01.268 INFO:teuthology.orchestra.run.smithi088.stdout: osd blocked scrub grace period = 3600 2023-10-30T14:13:01.268 INFO:teuthology.orchestra.run.smithi088.stdout: osd failsafe full ratio = 0.95 2023-10-30T14:13:01.268 INFO:teuthology.orchestra.run.smithi088.stdout: osd max markdown count = 1000 2023-10-30T14:13:01.268 INFO:teuthology.orchestra.run.smithi088.stdout: osd objectstore = bluestore 2023-10-30T14:13:01.269 INFO:teuthology.orchestra.run.smithi088.stdout: osd op complaint time = 180 2023-10-30T14:13:01.269 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.269 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr] 2023-10-30T14:13:01.269 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-10-30T14:13:01.269 INFO:teuthology.orchestra.run.smithi088.stdout: debug mgr = 20 2023-10-30T14:13:01.269 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2023-10-30T14:13:01.270 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2023-10-30T14:13:01.270 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2023-10-30T14:13:01.270 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2023-10-30T14:13:01.270 INFO:teuthology.orchestra.run.smithi088.stdout: mgr/telemetry/nag = false 2023-10-30T14:13:01.270 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.271 INFO:teuthology.orchestra.run.smithi088.stdout:[mon] 2023-10-30T14:13:01.271 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-10-30T14:13:01.271 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2023-10-30T14:13:01.271 INFO:teuthology.orchestra.run.smithi088.stdout: debug paxos = 20 2023-10-30T14:13:01.271 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2023-10-30T14:13:01.271 INFO:teuthology.orchestra.run.smithi088.stdout: mon data avail warn = 5 2023-10-30T14:13:01.272 INFO:teuthology.orchestra.run.smithi088.stdout: mon mgr mkfs grace = 240 2023-10-30T14:13:01.272 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2023-10-30T14:13:01.272 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd reporter subtree level = osd 2023-10-30T14:13:01.272 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd prime pg temp = true 2023-10-30T14:13:01.272 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2023-10-30T14:13:01.272 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.273 INFO:teuthology.orchestra.run.smithi088.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T14:13:01.273 INFO:teuthology.orchestra.run.smithi088.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T14:13:01.273 INFO:teuthology.orchestra.run.smithi088.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T14:13:01.273 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.273 INFO:teuthology.orchestra.run.smithi088.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T14:13:01.273 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T14:13:01.274 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T14:13:01.274 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.274 INFO:teuthology.orchestra.run.smithi088.stdout: # 1m isn't quite enough 2023-10-30T14:13:01.274 INFO:teuthology.orchestra.run.smithi088.stdout: mon_down_mkfs_grace = 2m 2023-10-30T14:13:01.274 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.274 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_filestore_osds = false 2023-10-30T14:13:01.275 INFO:teuthology.orchestra.run.smithi088.stdout: mon op complaint time = 120 2023-10-30T14:13:01.275 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-30T14:13:01.275 INFO:teuthology.orchestra.run.smithi088.stdout:[client] 2023-10-30T14:13:01.275 INFO:teuthology.orchestra.run.smithi088.stdout: rgw cache enabled = true 2023-10-30T14:13:01.275 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable ops log = true 2023-10-30T14:13:01.275 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable usage log = true 2023-10-30T14:13:01.276 INFO:teuthology.orchestra.run.smithi088.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T14:13:01.276 INFO:teuthology.orchestra.run.smithi088.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T14:13:01.276 INFO:teuthology.orchestra.run.smithi088.stdout: client mount timeout = 600 2023-10-30T14:13:01.276 INFO:teuthology.orchestra.run.smithi088.stdout: client_shutdown_timeout = 120 2023-10-30T14:13:01.276 INFO:teuthology.orchestra.run.smithi088.stdout: debug client = 20 2023-10-30T14:13:01.276 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-10-30T14:13:01.277 INFO:teuthology.orchestra.run.smithi088.stdout: rados mon op timeout = 900 2023-10-30T14:13:01.277 INFO:teuthology.orchestra.run.smithi088.stdout: rados osd op timeout = 900 2023-10-30T14:13:01.277 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.a] 2023-10-30T14:13:01.277 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.b] 2023-10-30T14:13:01.277 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.c] 2023-10-30T14:13:01.277 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.a] 2023-10-30T14:13:01.278 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.c] 2023-10-30T14:13:01.278 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.e] 2023-10-30T14:13:01.278 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.b] 2023-10-30T14:13:01.278 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.d] 2023-10-30T14:13:01.278 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.f] 2023-10-30T14:13:01.279 INFO:teuthology.orchestra.run.smithi088.stdout:[mds] 2023-10-30T14:13:01.279 INFO:teuthology.orchestra.run.smithi088.stdout: debug mds = 20 2023-10-30T14:13:01.279 INFO:teuthology.orchestra.run.smithi088.stdout: debug mds balancer = 20 2023-10-30T14:13:01.279 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-10-30T14:13:01.279 INFO:teuthology.orchestra.run.smithi088.stdout: mds bal fragment size max = 10000 2023-10-30T14:13:01.279 INFO:teuthology.orchestra.run.smithi088.stdout: mds bal merge size = 5 2023-10-30T14:13:01.280 INFO:teuthology.orchestra.run.smithi088.stdout: mds bal split bits = 3 2023-10-30T14:13:01.280 INFO:teuthology.orchestra.run.smithi088.stdout: mds bal split size = 100 2023-10-30T14:13:01.280 INFO:teuthology.orchestra.run.smithi088.stdout: mds debug frag = True 2023-10-30T14:13:01.280 INFO:teuthology.orchestra.run.smithi088.stdout: mds debug scatterstat = True 2023-10-30T14:13:01.280 INFO:teuthology.orchestra.run.smithi088.stdout: mds oft prefetch dirfrags = True 2023-10-30T14:13:01.280 INFO:teuthology.orchestra.run.smithi088.stdout: mds op complaint time = 180 2023-10-30T14:13:01.281 INFO:teuthology.orchestra.run.smithi088.stdout: mds verify scatter = True 2023-10-30T14:13:01.281 INFO:teuthology.orchestra.run.smithi088.stdout: mds_dir_prefetch = False 2023-10-30T14:13:01.281 INFO:teuthology.orchestra.run.smithi088.stdout: mds_sleep_rank_change = 5000000.0 2023-10-30T14:13:01.281 INFO:teuthology.orchestra.run.smithi088.stdout: mon client directed command retry = 5 2023-10-30T14:13:01.281 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject delay max = 1 2023-10-30T14:13:01.281 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject delay probability = 0.005 2023-10-30T14:13:01.282 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject delay type = client mds osd 2023-10-30T14:13:01.282 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject socket failures = 2500 2023-10-30T14:13:01.282 INFO:teuthology.orchestra.run.smithi088.stdout: osd op complaint time = 180 2023-10-30T14:13:01.282 INFO:teuthology.orchestra.run.smithi088.stdout: rados mon op timeout = 900 2023-10-30T14:13:01.282 INFO:teuthology.orchestra.run.smithi088.stdout: rados osd op timeout = 900 2023-10-30T14:13:01.284 INFO:teuthology.orchestra.run.smithi057.stdout:[global] 2023-10-30T14:13:01.284 INFO:teuthology.orchestra.run.smithi057.stdout: chdir = "" 2023-10-30T14:13:01.284 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T14:13:01.284 INFO:teuthology.orchestra.run.smithi057.stdout: auth supported = cephx 2023-10-30T14:13:01.284 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.285 INFO:teuthology.orchestra.run.smithi057.stdout: filestore xattr use omap = true 2023-10-30T14:13:01.285 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.285 INFO:teuthology.orchestra.run.smithi057.stdout: mon clock drift allowed = 1.000 2023-10-30T14:13:01.285 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.285 INFO:teuthology.orchestra.run.smithi057.stdout: osd crush chooseleaf type = 0 2023-10-30T14:13:01.286 INFO:teuthology.orchestra.run.smithi057.stdout: auth debug = true 2023-10-30T14:13:01.286 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.286 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on old message = true 2023-10-30T14:13:01.286 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on bug = true 2023-10-30T14:13:01.286 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.286 INFO:teuthology.orchestra.run.smithi057.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T14:13:01.287 INFO:teuthology.orchestra.run.smithi057.stdout: mon pg warn max object skew = 0 2023-10-30T14:13:01.287 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.287 INFO:teuthology.orchestra.run.smithi057.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T14:13:01.287 INFO:teuthology.orchestra.run.smithi057.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T14:13:01.287 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.288 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default size = 2 2023-10-30T14:13:01.288 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.288 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow primary affinity = true 2023-10-30T14:13:01.288 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow pg remap = true 2023-10-30T14:13:01.288 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on legacy crush tunables = false 2023-10-30T14:13:01.288 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on crush straw calc version zero = false 2023-10-30T14:13:01.289 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on no sortbitwise = false 2023-10-30T14:13:01.289 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on osd down out interval zero = false 2023-10-30T14:13:01.289 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on too few osds = false 2023-10-30T14:13:01.289 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T14:13:01.289 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T14:13:01.290 INFO:teuthology.orchestra.run.smithi057.stdout: mon_allow_pool_size_one = true 2023-10-30T14:13:01.290 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.290 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T14:13:01.290 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.290 INFO:teuthology.orchestra.run.smithi057.stdout: osd default data pool replay window = 5 2023-10-30T14:13:01.291 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.291 INFO:teuthology.orchestra.run.smithi057.stdout: mon allow pool delete = true 2023-10-30T14:13:01.291 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.291 INFO:teuthology.orchestra.run.smithi057.stdout: mon cluster log file level = debug 2023-10-30T14:13:01.291 INFO:teuthology.orchestra.run.smithi057.stdout: debug asserts on shutdown = true 2023-10-30T14:13:01.291 INFO:teuthology.orchestra.run.smithi057.stdout: mon health detail to clog = false 2023-10-30T14:13:01.292 INFO:teuthology.orchestra.run.smithi057.stdout: mon host = "172.21.15.57,172.21.15.88,[v2:172.21.15.88:3301,v1:172.21.15.88:6790]" 2023-10-30T14:13:01.292 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on skipped message = False 2023-10-30T14:13:01.292 INFO:teuthology.orchestra.run.smithi057.stdout: fsid = e4ade530-0fca-4330-9824-3adb64478f80 2023-10-30T14:13:01.292 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.292 INFO:teuthology.orchestra.run.smithi057.stdout:[osd] 2023-10-30T14:13:01.292 INFO:teuthology.orchestra.run.smithi057.stdout: osd journal size = 100 2023-10-30T14:13:01.293 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.293 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub load threshold = 5.0 2023-10-30T14:13:01.293 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub max interval = 600 2023-10-30T14:13:01.293 INFO:teuthology.orchestra.run.smithi057.stdout: osd mclock profile = high_recovery_ops 2023-10-30T14:13:01.293 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.293 INFO:teuthology.orchestra.run.smithi057.stdout: osd recover clone overlap = true 2023-10-30T14:13:01.294 INFO:teuthology.orchestra.run.smithi057.stdout: osd recovery max chunk = 1048576 2023-10-30T14:13:01.294 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.294 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug shutdown = true 2023-10-30T14:13:01.294 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug op order = true 2023-10-30T14:13:01.294 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug verify stray on activate = true 2023-10-30T14:13:01.295 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.295 INFO:teuthology.orchestra.run.smithi057.stdout: osd open classes on start = true 2023-10-30T14:13:01.295 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug pg log writeout = true 2023-10-30T14:13:01.295 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.295 INFO:teuthology.orchestra.run.smithi057.stdout: osd deep scrub update digest min age = 30 2023-10-30T14:13:01.296 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.296 INFO:teuthology.orchestra.run.smithi057.stdout: osd map max advance = 10 2023-10-30T14:13:01.296 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.296 INFO:teuthology.orchestra.run.smithi057.stdout: journal zero on create = true 2023-10-30T14:13:01.297 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.297 INFO:teuthology.orchestra.run.smithi057.stdout: filestore ondisk finisher threads = 3 2023-10-30T14:13:01.297 INFO:teuthology.orchestra.run.smithi057.stdout: filestore apply finisher threads = 3 2023-10-30T14:13:01.297 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.297 INFO:teuthology.orchestra.run.smithi057.stdout: bdev debug aio = true 2023-10-30T14:13:01.298 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug misdirected ops = true 2023-10-30T14:13:01.298 INFO:teuthology.orchestra.run.smithi057.stdout: bdev async discard = True 2023-10-30T14:13:01.298 INFO:teuthology.orchestra.run.smithi057.stdout: bdev enable discard = True 2023-10-30T14:13:01.298 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore allocator = bitmap 2023-10-30T14:13:01.298 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore block size = 96636764160 2023-10-30T14:13:01.299 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore fsck on mount = True 2023-10-30T14:13:01.299 INFO:teuthology.orchestra.run.smithi057.stdout: debug bluefs = 1/20 2023-10-30T14:13:01.299 INFO:teuthology.orchestra.run.smithi057.stdout: debug bluestore = 1/20 2023-10-30T14:13:01.299 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-10-30T14:13:01.299 INFO:teuthology.orchestra.run.smithi057.stdout: debug osd = 20 2023-10-30T14:13:01.300 INFO:teuthology.orchestra.run.smithi057.stdout: debug rocksdb = 4/10 2023-10-30T14:13:01.300 INFO:teuthology.orchestra.run.smithi057.stdout: mon client directed command retry = 5 2023-10-30T14:13:01.300 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T14:13:01.300 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd full ratio = 0.9 2023-10-30T14:13:01.301 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd nearfull ratio = 0.8 2023-10-30T14:13:01.301 INFO:teuthology.orchestra.run.smithi057.stdout: ms inject delay max = 1 2023-10-30T14:13:01.301 INFO:teuthology.orchestra.run.smithi057.stdout: ms inject delay probability = 0.005 2023-10-30T14:13:01.301 INFO:teuthology.orchestra.run.smithi057.stdout: ms inject delay type = client mds 2023-10-30T14:13:01.301 INFO:teuthology.orchestra.run.smithi057.stdout: ms inject socket failures = 2500 2023-10-30T14:13:01.302 INFO:teuthology.orchestra.run.smithi057.stdout: osd blocked scrub grace period = 3600 2023-10-30T14:13:01.302 INFO:teuthology.orchestra.run.smithi057.stdout: osd failsafe full ratio = 0.95 2023-10-30T14:13:01.302 INFO:teuthology.orchestra.run.smithi057.stdout: osd max markdown count = 1000 2023-10-30T14:13:01.302 INFO:teuthology.orchestra.run.smithi057.stdout: osd objectstore = bluestore 2023-10-30T14:13:01.302 INFO:teuthology.orchestra.run.smithi057.stdout: osd op complaint time = 180 2023-10-30T14:13:01.303 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.303 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr] 2023-10-30T14:13:01.303 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-10-30T14:13:01.303 INFO:teuthology.orchestra.run.smithi057.stdout: debug mgr = 20 2023-10-30T14:13:01.304 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2023-10-30T14:13:01.304 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2023-10-30T14:13:01.304 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2023-10-30T14:13:01.304 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2023-10-30T14:13:01.304 INFO:teuthology.orchestra.run.smithi057.stdout: mgr/telemetry/nag = false 2023-10-30T14:13:01.305 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.305 INFO:teuthology.orchestra.run.smithi057.stdout:[mon] 2023-10-30T14:13:01.305 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-10-30T14:13:01.305 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2023-10-30T14:13:01.305 INFO:teuthology.orchestra.run.smithi057.stdout: debug paxos = 20 2023-10-30T14:13:01.306 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2023-10-30T14:13:01.306 INFO:teuthology.orchestra.run.smithi057.stdout: mon data avail warn = 5 2023-10-30T14:13:01.306 INFO:teuthology.orchestra.run.smithi057.stdout: mon mgr mkfs grace = 240 2023-10-30T14:13:01.306 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2023-10-30T14:13:01.307 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd reporter subtree level = osd 2023-10-30T14:13:01.307 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd prime pg temp = true 2023-10-30T14:13:01.307 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2023-10-30T14:13:01.307 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.307 INFO:teuthology.orchestra.run.smithi057.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T14:13:01.308 INFO:teuthology.orchestra.run.smithi057.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T14:13:01.308 INFO:teuthology.orchestra.run.smithi057.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T14:13:01.308 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.308 INFO:teuthology.orchestra.run.smithi057.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T14:13:01.308 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T14:13:01.309 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T14:13:01.309 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.309 INFO:teuthology.orchestra.run.smithi057.stdout: # 1m isn't quite enough 2023-10-30T14:13:01.309 INFO:teuthology.orchestra.run.smithi057.stdout: mon_down_mkfs_grace = 2m 2023-10-30T14:13:01.309 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.310 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_filestore_osds = false 2023-10-30T14:13:01.310 INFO:teuthology.orchestra.run.smithi057.stdout: mon op complaint time = 120 2023-10-30T14:13:01.310 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:01.310 INFO:teuthology.orchestra.run.smithi057.stdout:[client] 2023-10-30T14:13:01.310 INFO:teuthology.orchestra.run.smithi057.stdout: rgw cache enabled = true 2023-10-30T14:13:01.311 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable ops log = true 2023-10-30T14:13:01.311 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable usage log = true 2023-10-30T14:13:01.311 INFO:teuthology.orchestra.run.smithi057.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T14:13:01.311 INFO:teuthology.orchestra.run.smithi057.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T14:13:01.311 INFO:teuthology.orchestra.run.smithi057.stdout: client mount timeout = 600 2023-10-30T14:13:01.312 INFO:teuthology.orchestra.run.smithi057.stdout: client_shutdown_timeout = 120 2023-10-30T14:13:01.312 INFO:teuthology.orchestra.run.smithi057.stdout: debug client = 20 2023-10-30T14:13:01.312 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-10-30T14:13:01.312 INFO:teuthology.orchestra.run.smithi057.stdout: rados mon op timeout = 900 2023-10-30T14:13:01.312 INFO:teuthology.orchestra.run.smithi057.stdout: rados osd op timeout = 900 2023-10-30T14:13:01.313 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.a] 2023-10-30T14:13:01.313 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.b] 2023-10-30T14:13:01.313 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.c] 2023-10-30T14:13:01.313 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.a] 2023-10-30T14:13:01.313 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.c] 2023-10-30T14:13:01.314 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.e] 2023-10-30T14:13:01.314 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.b] 2023-10-30T14:13:01.314 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.d] 2023-10-30T14:13:01.314 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.f] 2023-10-30T14:13:01.314 INFO:teuthology.orchestra.run.smithi057.stdout:[mds] 2023-10-30T14:13:01.314 INFO:teuthology.orchestra.run.smithi057.stdout: debug mds = 20 2023-10-30T14:13:01.315 INFO:teuthology.orchestra.run.smithi057.stdout: debug mds balancer = 20 2023-10-30T14:13:01.315 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-10-30T14:13:01.315 INFO:teuthology.orchestra.run.smithi057.stdout: mds bal fragment size max = 10000 2023-10-30T14:13:01.315 INFO:teuthology.orchestra.run.smithi057.stdout: mds bal merge size = 5 2023-10-30T14:13:01.315 INFO:teuthology.orchestra.run.smithi057.stdout: mds bal split bits = 3 2023-10-30T14:13:01.315 INFO:teuthology.orchestra.run.smithi057.stdout: mds bal split size = 100 2023-10-30T14:13:01.316 INFO:teuthology.orchestra.run.smithi057.stdout: mds debug frag = True 2023-10-30T14:13:01.316 INFO:teuthology.orchestra.run.smithi057.stdout: mds debug scatterstat = True 2023-10-30T14:13:01.316 INFO:teuthology.orchestra.run.smithi057.stdout: mds oft prefetch dirfrags = True 2023-10-30T14:13:01.316 INFO:teuthology.orchestra.run.smithi057.stdout: mds op complaint time = 180 2023-10-30T14:13:01.316 INFO:teuthology.orchestra.run.smithi057.stdout: mds verify scatter = True 2023-10-30T14:13:01.317 INFO:teuthology.orchestra.run.smithi057.stdout: mds_dir_prefetch = False 2023-10-30T14:13:01.317 INFO:teuthology.orchestra.run.smithi057.stdout: mds_sleep_rank_change = 5000000.0 2023-10-30T14:13:01.317 INFO:teuthology.orchestra.run.smithi057.stdout: mon client directed command retry = 5 2023-10-30T14:13:01.317 INFO:teuthology.orchestra.run.smithi057.stdout: ms inject delay max = 1 2023-10-30T14:13:01.317 INFO:teuthology.orchestra.run.smithi057.stdout: ms inject delay probability = 0.005 2023-10-30T14:13:01.318 INFO:teuthology.orchestra.run.smithi057.stdout: ms inject delay type = client mds osd 2023-10-30T14:13:01.318 INFO:teuthology.orchestra.run.smithi057.stdout: ms inject socket failures = 2500 2023-10-30T14:13:01.318 INFO:teuthology.orchestra.run.smithi057.stdout: osd op complaint time = 180 2023-10-30T14:13:01.318 INFO:teuthology.orchestra.run.smithi057.stdout: rados mon op timeout = 900 2023-10-30T14:13:01.318 INFO:teuthology.orchestra.run.smithi057.stdout: rados osd op timeout = 900 2023-10-30T14:13:01.319 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-30T14:13:01.319 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-30T14:13:01.383 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-30T14:13:01.383 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:13:01.384 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-30T14:13:01.393 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:13:01.393 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-30T14:13:01.441 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi057.front.sepia.ceph.com 2023-10-30T14:13:01.441 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:13:01.442 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T14:13:01.442 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T14:13:01.519 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:13:01.519 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T14:13:01.574 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi088.front.sepia.ceph.com 2023-10-30T14:13:01.574 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:13:01.574 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T14:13:01.575 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T14:13:01.597 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:13:01.597 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T14:13:01.645 INFO:tasks.ceph:Setting up mon nodes... 2023-10-30T14:13:01.645 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-30T14:13:01.645 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T14:13:01.704 INFO:teuthology.orchestra.run.smithi057.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T14:13:01.709 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T14:13:01.753 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T14:13:01.759 INFO:tasks.ceph:Setting up mds nodes... 2023-10-30T14:13:01.759 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-30T14:13:01.817 INFO:teuthology.orchestra.run.smithi057.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-30T14:13:01.821 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-30T14:13:01.843 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-30T14:13:01.942 INFO:teuthology.orchestra.run.smithi057.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-30T14:13:01.946 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-30T14:13:01.968 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2023-10-30T14:13:02.066 INFO:teuthology.orchestra.run.smithi057.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2023-10-30T14:13:02.070 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2023-10-30T14:13:02.091 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-30T14:13:02.135 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-30T14:13:02.138 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-30T14:13:02.196 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-30T14:13:02.281 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-30T14:13:02.284 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-30T14:13:02.340 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2023-10-30T14:13:02.424 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2023-10-30T14:13:02.427 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2023-10-30T14:13:02.480 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-30T14:13:02.480 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-30T14:13:02.526 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-30T14:13:02.544 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-30T14:13:02.544 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi057.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T14:13:02.544 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-30T14:13:02.565 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T14:13:02.565 INFO:tasks.ceph:role: osd.0 2023-10-30T14:13:02.566 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi057.front.sepia.ceph.com 2023-10-30T14:13:02.566 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T14:13:02.657 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T14:13:02.657 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T14:13:02.658 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T14:13:02.658 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T14:13:02.658 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T14:13:02.658 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2023-10-30T14:13:02.658 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T14:13:02.659 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T14:13:02.659 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T14:13:02.659 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T14:13:03.966 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2023-10-30T14:13:03.969 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2023-10-30T14:13:03.969 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-30T14:13:04.070 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-30T14:13:04.088 INFO:teuthology.orchestra.run.smithi057.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T14:13:04.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T14:13:04.090 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-30T14:13:04.159 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T14:13:04.159 INFO:tasks.ceph:role: osd.1 2023-10-30T14:13:04.160 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi057.front.sepia.ceph.com 2023-10-30T14:13:04.160 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T14:13:04.251 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T14:13:04.251 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T14:13:04.251 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T14:13:04.251 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T14:13:04.252 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T14:13:04.252 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2023-10-30T14:13:04.252 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T14:13:04.252 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T14:13:04.252 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T14:13:04.252 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T14:13:05.541 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2023-10-30T14:13:05.543 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2023-10-30T14:13:05.544 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-30T14:13:05.620 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-30T14:13:05.682 INFO:teuthology.orchestra.run.smithi057.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T14:13:05.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T14:13:05.683 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-30T14:13:05.746 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T14:13:05.747 INFO:tasks.ceph:role: osd.2 2023-10-30T14:13:05.747 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi057.front.sepia.ceph.com 2023-10-30T14:13:05.747 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T14:13:05.839 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T14:13:05.839 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T14:13:05.840 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T14:13:05.840 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T14:13:05.840 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T14:13:05.840 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2023-10-30T14:13:05.840 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T14:13:05.841 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T14:13:05.841 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T14:13:05.841 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T14:13:07.127 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2023-10-30T14:13:07.130 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2023-10-30T14:13:07.130 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-30T14:13:07.163 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-30T14:13:07.223 INFO:teuthology.orchestra.run.smithi057.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T14:13:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T14:13:07.224 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-30T14:13:07.287 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T14:13:07.287 INFO:tasks.ceph:role: osd.3 2023-10-30T14:13:07.288 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi057.front.sepia.ceph.com 2023-10-30T14:13:07.288 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T14:13:07.379 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T14:13:07.380 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T14:13:07.380 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T14:13:07.380 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T14:13:07.380 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T14:13:07.380 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2023-10-30T14:13:07.381 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T14:13:07.381 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T14:13:07.381 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T14:13:07.381 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T14:13:08.674 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2023-10-30T14:13:08.676 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2023-10-30T14:13:08.677 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-30T14:13:08.709 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-30T14:13:08.771 INFO:teuthology.orchestra.run.smithi057.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T14:13:08.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T14:13:08.772 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T14:13:08.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:08.911+0000 7ff9bfc6e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-30T14:13:08.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:08.915+0000 7ff9bfc6e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-30T14:13:08.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:08.915+0000 7ff9bfc6e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T14:13:08.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:08.915+0000 7ff9bfc6e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T14:13:08.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:08.915+0000 7ff9bfc6e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T14:13:08.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:08.915+0000 7ff9bfc6e640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-30T14:13:11.922 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-30T14:13:11.944 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T14:13:12.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:12.039+0000 7f51ee8e8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-30T14:13:12.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:12.043+0000 7f51ee8e8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-30T14:13:12.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:12.043+0000 7f51ee8e8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T14:13:12.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:12.043+0000 7f51ee8e8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T14:13:12.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:12.043+0000 7f51ee8e8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T14:13:12.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:12.043+0000 7f51ee8e8640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-30T14:13:15.045 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-30T14:13:15.069 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T14:13:15.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:15.167+0000 7f3ce422b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-30T14:13:15.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:15.167+0000 7f3ce422b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-30T14:13:15.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:15.167+0000 7f3ce422b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T14:13:15.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:15.167+0000 7f3ce422b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T14:13:15.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:15.167+0000 7f3ce422b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T14:13:15.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:15.171+0000 7f3ce422b640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-30T14:13:18.173 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-30T14:13:18.195 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T14:13:18.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:18.299+0000 7f6f7062e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-30T14:13:18.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:18.299+0000 7f6f7062e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-30T14:13:18.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:18.299+0000 7f6f7062e640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T14:13:18.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:18.299+0000 7f6f7062e640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T14:13:18.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:18.299+0000 7f6f7062e640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T14:13:18.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:18.299+0000 7f6f7062e640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-30T14:13:21.296 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-30T14:13:21.318 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-30T14:13:21.333 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T14:13:21.333 INFO:tasks.ceph:role: osd.4 2023-10-30T14:13:21.333 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-30T14:13:21.334 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T14:13:21.397 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T14:13:21.397 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T14:13:21.397 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T14:13:21.397 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T14:13:21.397 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T14:13:21.398 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2023-10-30T14:13:21.398 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T14:13:21.398 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T14:13:21.398 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T14:13:21.398 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T14:13:21.849 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2023-10-30T14:13:21.851 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2023-10-30T14:13:21.852 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-30T14:13:21.943 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-30T14:13:21.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T14:13:21.954 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T14:13:21.954 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-30T14:13:22.005 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T14:13:22.006 INFO:tasks.ceph:role: osd.5 2023-10-30T14:13:22.006 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-30T14:13:22.006 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T14:13:22.069 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T14:13:22.069 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T14:13:22.069 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T14:13:22.070 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T14:13:22.070 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T14:13:22.070 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2023-10-30T14:13:22.070 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T14:13:22.070 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T14:13:22.071 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T14:13:22.071 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T14:13:22.540 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2023-10-30T14:13:22.542 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2023-10-30T14:13:22.543 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-30T14:13:22.688 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-30T14:13:22.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T14:13:22.700 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T14:13:22.700 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-30T14:13:22.756 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T14:13:22.757 INFO:tasks.ceph:role: osd.6 2023-10-30T14:13:22.757 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-30T14:13:22.757 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T14:13:22.822 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T14:13:22.822 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T14:13:22.822 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T14:13:22.822 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T14:13:22.823 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T14:13:22.823 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2023-10-30T14:13:22.823 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T14:13:22.823 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T14:13:22.823 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T14:13:22.823 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T14:13:23.296 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2023-10-30T14:13:23.298 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2023-10-30T14:13:23.298 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-30T14:13:23.427 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-30T14:13:23.435 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T14:13:23.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T14:13:23.437 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-30T14:13:23.492 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T14:13:23.492 INFO:tasks.ceph:role: osd.7 2023-10-30T14:13:23.492 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-30T14:13:23.492 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T14:13:23.560 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T14:13:23.561 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T14:13:23.561 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T14:13:23.561 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T14:13:23.562 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T14:13:23.562 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2023-10-30T14:13:23.562 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T14:13:23.562 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T14:13:23.562 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T14:13:23.563 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T14:13:23.977 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2023-10-30T14:13:23.980 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2023-10-30T14:13:23.980 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-30T14:13:24.055 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-30T14:13:24.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T14:13:24.105 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T14:13:24.106 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T14:13:24.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:24.216+0000 7fc93e92b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-30T14:13:24.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:24.220+0000 7fc93e92b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-30T14:13:24.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:24.220+0000 7fc93e92b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T14:13:24.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:24.220+0000 7fc93e92b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T14:13:24.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:24.220+0000 7fc93e92b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T14:13:24.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:24.220+0000 7fc93e92b640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-30T14:13:27.245 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-30T14:13:27.259 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T14:13:27.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:27.336+0000 7f406448a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-30T14:13:27.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:27.336+0000 7f406448a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-30T14:13:27.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:27.336+0000 7f406448a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T14:13:27.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:27.336+0000 7f406448a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T14:13:27.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:27.336+0000 7f406448a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T14:13:27.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:27.340+0000 7f406448a640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-30T14:13:29.328 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:13:29.333 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:13:30.349 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-30T14:13:30.361 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T14:13:30.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:30.440+0000 7f92c05e5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-30T14:13:30.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:30.440+0000 7f92c05e5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-30T14:13:30.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:30.440+0000 7f92c05e5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T14:13:30.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:30.440+0000 7f92c05e5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T14:13:30.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:30.440+0000 7f92c05e5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T14:13:30.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:30.440+0000 7f92c05e5640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-30T14:13:33.450 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-30T14:13:33.505 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T14:13:33.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:33.587+0000 7f5f74cbd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-30T14:13:33.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:33.587+0000 7f5f74cbd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-30T14:13:33.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:33.587+0000 7f5f74cbd640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T14:13:33.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:33.587+0000 7f5f74cbd640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T14:13:33.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:33.587+0000 7f5f74cbd640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T14:13:33.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:33.591+0000 7f5f74cbd640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-30T14:13:36.576 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-30T14:13:36.630 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-30T14:13:36.630 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:13:36.631 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-30T14:13:36.654 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:13:36.655 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-30T14:13:36.724 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:13:36.724 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-30T14:13:36.788 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:13:36.788 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2023-10-30T14:13:36.852 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:13:36.852 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-30T14:13:36.915 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:13:36.916 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-30T14:13:36.980 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:13:36.980 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-30T14:13:37.047 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:13:37.048 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-30T14:13:37.115 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:13:37.116 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-30T14:13:37.129 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:13:37.129 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-30T14:13:37.188 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:13:37.188 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-30T14:13:37.244 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:13:37.245 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2023-10-30T14:13:37.300 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:13:37.301 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-30T14:13:37.356 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:13:37.357 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-30T14:13:37.412 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:13:37.413 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-30T14:13:37.468 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:13:37.469 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-30T14:13:37.524 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:13:37.525 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-30T14:13:37.535 INFO:tasks.ceph:Adding keys to all mons... 2023-10-30T14:13:37.535 DEBUG:teuthology.orchestra.run.smithi057:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T14:13:37.581 DEBUG:teuthology.orchestra.run.smithi088:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T14:13:37.592 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr.y] 2023-10-30T14:13:37.592 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDtuT9lDZQWKhAASr6Y3aSAp5qy+xQCS/Zt5w== 2023-10-30T14:13:37.593 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.a] 2023-10-30T14:13:37.593 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDtuT9lkL7RMBAAWCPsmOE9HQz1itj82a5AmQ== 2023-10-30T14:13:37.593 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.c] 2023-10-30T14:13:37.593 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDtuT9lpdBAOBAA9nzEtD8Gx2Brxu/6GnN8lg== 2023-10-30T14:13:37.593 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.e] 2023-10-30T14:13:37.594 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDuuT9lS/QMBBAAbEh/N1uRoExaq8mIS+q2Tw== 2023-10-30T14:13:37.594 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.0] 2023-10-30T14:13:37.594 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQD0uT9lrZqFNhAAbnzxlSCyFGGaNTactOwK2w== 2023-10-30T14:13:37.594 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.1] 2023-10-30T14:13:37.594 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQD4uT9lXqqEAhAAg6B4XT2YgRqqdOXQtKLPbA== 2023-10-30T14:13:37.595 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.2] 2023-10-30T14:13:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQD7uT9lIKYTChAA/SIaL74McirPyQL8gw/mvw== 2023-10-30T14:13:37.595 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.3] 2023-10-30T14:13:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQD+uT9l8XHcERAAkjh2tiezkXFosmJ3GQ98vA== 2023-10-30T14:13:37.595 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr.x] 2023-10-30T14:13:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDtuT9lsw/aLBAAP6mRHs8uvXDARiRA3Cp+xg== 2023-10-30T14:13:37.596 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.b] 2023-10-30T14:13:37.596 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDuuT9lFdEhCBAAu4SNh3kMj2ktTG5/IPdjjQ== 2023-10-30T14:13:37.596 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.d] 2023-10-30T14:13:37.596 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDuuT9liTPhEBAAdhpTQDhBBQWK9vXcbnS6aw== 2023-10-30T14:13:37.596 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.f] 2023-10-30T14:13:37.596 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDuuT9lLw9iGRAAxFwqa3jKbSGR3VFaJICUQQ== 2023-10-30T14:13:37.597 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.4] 2023-10-30T14:13:37.597 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAEuj9lTBwQDRAA0MsEUnmsRD+/Amf9Nzihng== 2023-10-30T14:13:37.597 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.5] 2023-10-30T14:13:37.597 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAHuj9lBU5TFBAAkLcMFrzaZ/ZVuf++oitClg== 2023-10-30T14:13:37.597 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.6] 2023-10-30T14:13:37.597 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAKuj9lhqZ+GhAAqD7I+wAtQGMYYNvwf4g2fQ== 2023-10-30T14:13:37.598 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.7] 2023-10-30T14:13:37.598 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQANuj9lX51YIxAASWHzNEo10bssHVZQm4CLmw== 2023-10-30T14:13:37.598 INFO:teuthology.orchestra.run.smithi088.stdout:[client.0] 2023-10-30T14:13:37.598 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDuuT9lsdR1HxAAsfBbJIY/CibbrIheYWsUpg== 2023-10-30T14:13:37.599 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr.y] 2023-10-30T14:13:37.599 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDtuT9lDZQWKhAASr6Y3aSAp5qy+xQCS/Zt5w== 2023-10-30T14:13:37.599 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.a] 2023-10-30T14:13:37.599 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDtuT9lkL7RMBAAWCPsmOE9HQz1itj82a5AmQ== 2023-10-30T14:13:37.600 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.c] 2023-10-30T14:13:37.600 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDtuT9lpdBAOBAA9nzEtD8Gx2Brxu/6GnN8lg== 2023-10-30T14:13:37.600 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.e] 2023-10-30T14:13:37.600 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDuuT9lS/QMBBAAbEh/N1uRoExaq8mIS+q2Tw== 2023-10-30T14:13:37.600 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.0] 2023-10-30T14:13:37.601 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQD0uT9lrZqFNhAAbnzxlSCyFGGaNTactOwK2w== 2023-10-30T14:13:37.601 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.1] 2023-10-30T14:13:37.601 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQD4uT9lXqqEAhAAg6B4XT2YgRqqdOXQtKLPbA== 2023-10-30T14:13:37.601 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.2] 2023-10-30T14:13:37.601 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQD7uT9lIKYTChAA/SIaL74McirPyQL8gw/mvw== 2023-10-30T14:13:37.601 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.3] 2023-10-30T14:13:37.602 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQD+uT9l8XHcERAAkjh2tiezkXFosmJ3GQ98vA== 2023-10-30T14:13:37.602 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr.x] 2023-10-30T14:13:37.602 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDtuT9lsw/aLBAAP6mRHs8uvXDARiRA3Cp+xg== 2023-10-30T14:13:37.602 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.b] 2023-10-30T14:13:37.602 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDuuT9lFdEhCBAAu4SNh3kMj2ktTG5/IPdjjQ== 2023-10-30T14:13:37.603 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.d] 2023-10-30T14:13:37.603 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDuuT9liTPhEBAAdhpTQDhBBQWK9vXcbnS6aw== 2023-10-30T14:13:37.603 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.f] 2023-10-30T14:13:37.603 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDuuT9lLw9iGRAAxFwqa3jKbSGR3VFaJICUQQ== 2023-10-30T14:13:37.603 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.4] 2023-10-30T14:13:37.603 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAEuj9lTBwQDRAA0MsEUnmsRD+/Amf9Nzihng== 2023-10-30T14:13:37.604 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.5] 2023-10-30T14:13:37.604 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAHuj9lBU5TFBAAkLcMFrzaZ/ZVuf++oitClg== 2023-10-30T14:13:37.604 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.6] 2023-10-30T14:13:37.604 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAKuj9lhqZ+GhAAqD7I+wAtQGMYYNvwf4g2fQ== 2023-10-30T14:13:37.604 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.7] 2023-10-30T14:13:37.605 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQANuj9lX51YIxAASWHzNEo10bssHVZQm4CLmw== 2023-10-30T14:13:37.605 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2023-10-30T14:13:37.605 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDuuT9lsdR1HxAAsfBbJIY/CibbrIheYWsUpg== 2023-10-30T14:13:37.606 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T14:13:37.657 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T14:13:37.714 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T14:13:37.719 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T14:13:37.777 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T14:13:37.781 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T14:13:37.849 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T14:13:37.853 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T14:13:37.907 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T14:13:37.911 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T14:13:37.979 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T14:13:37.983 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T14:13:38.060 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T14:13:38.064 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T14:13:38.142 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T14:13:38.146 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T14:13:38.224 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T14:13:38.228 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T14:13:38.312 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T14:13:38.316 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T14:13:38.397 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T14:13:38.401 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T14:13:38.486 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T14:13:38.490 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T14:13:38.571 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T14:13:38.575 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T14:13:38.654 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T14:13:38.658 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T14:13:38.741 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T14:13:38.745 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T14:13:38.797 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T14:13:38.801 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T14:13:38.865 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T14:13:38.869 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T14:13:38.956 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-30T14:13:38.956 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-30T14:13:38.977 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T14:13:39.101 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-30T14:13:39.122 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-30T14:13:39.137 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T14:13:39.236 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-30T14:13:39.246 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-30T14:13:39.298 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T14:13:39.395 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-30T14:13:39.405 DEBUG:teuthology.orchestra.run.smithi057:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T14:13:39.408 DEBUG:teuthology.orchestra.run.smithi088:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T14:13:39.453 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-30T14:13:39.453 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T14:13:39.454 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T14:13:39.458 INFO:tasks.ceph.mon.a:Started 2023-10-30T14:13:39.458 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-30T14:13:39.458 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-30T14:13:39.498 INFO:tasks.ceph.mon.b:Started 2023-10-30T14:13:39.499 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-30T14:13:39.499 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-30T14:13:39.501 INFO:tasks.ceph.mon.c:Started 2023-10-30T14:13:39.502 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-30T14:13:39.502 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-30T14:13:39.503 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-30T14:13:39.506 INFO:tasks.ceph.mgr.y:Started 2023-10-30T14:13:39.507 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-30T14:13:39.507 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-30T14:13:39.509 INFO:tasks.ceph.mgr.x:Started 2023-10-30T14:13:39.509 INFO:tasks.ceph:Setting crush tunables to default 2023-10-30T14:13:39.510 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph osd crush tunables default 2023-10-30T14:13:39.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.649+0000 7fde5bf65640 1 Processor -- start 2023-10-30T14:13:39.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.649+0000 7fde5bf65640 1 -- start start 2023-10-30T14:13:39.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.653+0000 7fde5bf65640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fde54158070 0x7fde541611b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:39.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.653+0000 7fde5bf65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde54071c30 0x7fde540720c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:39.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.653+0000 7fde5bf65640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde540717a0 con 0x7fde5415e650 2023-10-30T14:13:39.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.653+0000 7fde5bf65640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde54162170 con 0x7fde54158070 2023-10-30T14:13:39.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.653+0000 7fde5bf65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde541616f0 con 0x7fde54071c30 2023-10-30T14:13:39.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.653+0000 7fde5a502640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde54071c30 0x7fde540720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:39.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.653+0000 7fde59d01640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fde54158070 0x7fde541611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:39.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.653+0000 7fde5a502640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde54071c30 0x7fde540720c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:56898/0 (socket says 172.21.15.57:56898) 2023-10-30T14:13:39.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.653+0000 7fde5a502640 1 -- 172.21.15.57:0/3310222933 learned_addr learned my addr 172.21.15.57:0/3310222933 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:13:39.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.653+0000 7fde5a502640 1 -- 172.21.15.57:0/3310222933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde54071c30 msgr2=0x7fde540720c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T14:13:39.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.653+0000 7fde5a502640 1 -- 172.21.15.57:0/3310222933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde54071c30 msgr2=0x7fde540720c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T14:13:39.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.653+0000 7fde5a502640 1 --2- 172.21.15.57:0/3310222933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde54071c30 0x7fde540720c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T14:13:39.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.653+0000 7fde59d01640 1 -- 172.21.15.57:0/3310222933 >> v2:172.21.15.88:3300/0 conn(0x7fde54158070 msgr2=0x7fde541611b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T14:13:39.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.653+0000 7fde59d01640 1 -- 172.21.15.57:0/3310222933 >> v2:172.21.15.88:3300/0 conn(0x7fde54158070 msgr2=0x7fde541611b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T14:13:39.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.653+0000 7fde59d01640 1 --2- 172.21.15.57:0/3310222933 >> v2:172.21.15.88:3300/0 conn(0x7fde54158070 0x7fde541611b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T14:13:39.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.653+0000 7fde5a502640 1 --2- 172.21.15.57:0/3310222933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde54071c30 0x7fde540720c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T14:13:39.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.653+0000 7fde59d01640 1 --2- 172.21.15.57:0/3310222933 >> v2:172.21.15.88:3300/0 conn(0x7fde54158070 0x7fde541611b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T14:13:39.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.857+0000 7fde59d01640 1 --2- 172.21.15.57:0/3310222933 >> v2:172.21.15.88:3300/0 conn(0x7fde54158070 0x7fde541611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:39.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.857+0000 7fde5a502640 1 --2- 172.21.15.57:0/3310222933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde54071c30 0x7fde540720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:39.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.857+0000 7fde59d01640 1 -- 172.21.15.57:0/3310222933 >> v2:172.21.15.88:3300/0 conn(0x7fde54158070 msgr2=0x7fde541611b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T14:13:39.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.857+0000 7fde59d01640 1 -- 172.21.15.57:0/3310222933 >> v2:172.21.15.88:3300/0 conn(0x7fde54158070 msgr2=0x7fde541611b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T14:13:39.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.857+0000 7fde59d01640 1 --2- 172.21.15.57:0/3310222933 >> v2:172.21.15.88:3300/0 conn(0x7fde54158070 0x7fde541611b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T14:13:39.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.857+0000 7fde5a502640 1 -- 172.21.15.57:0/3310222933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde54071c30 msgr2=0x7fde540720c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T14:13:39.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.857+0000 7fde5a502640 1 -- 172.21.15.57:0/3310222933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde54071c30 msgr2=0x7fde540720c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T14:13:39.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.857+0000 7fde5a502640 1 --2- 172.21.15.57:0/3310222933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde54071c30 0x7fde540720c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T14:13:39.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.857+0000 7fde59d01640 1 --2- 172.21.15.57:0/3310222933 >> v2:172.21.15.88:3300/0 conn(0x7fde54158070 0x7fde541611b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T14:13:39.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.857+0000 7fde5a502640 1 --2- 172.21.15.57:0/3310222933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde54071c30 0x7fde540720c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T14:13:39.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.877+0000 7fde58cff640 1 -- 172.21.15.57:0/3310222933 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996988500 0 0) 0x7fde540717a0 con 0x7fde5415e650 2023-10-30T14:13:39.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.877+0000 7fde58cff640 1 -- 172.21.15.57:0/3310222933 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde40003530 con 0x7fde5415e650 2023-10-30T14:13:39.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde58cff640 1 -- 172.21.15.57:0/3310222933 <== mon.1 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1782557359 0 0) 0x7fde40003530 con 0x7fde5415e650 2023-10-30T14:13:39.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde58cff640 1 -- 172.21.15.57:0/3310222933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde54071c30 msgr2=0x7fde540720c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T14:13:39.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde58cff640 1 --2- 172.21.15.57:0/3310222933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde54071c30 0x7fde540720c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:39.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde58cff640 1 -- 172.21.15.57:0/3310222933 >> v2:172.21.15.88:3300/0 conn(0x7fde54158070 msgr2=0x7fde541611b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T14:13:39.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde58cff640 1 --2- 172.21.15.57:0/3310222933 >> v2:172.21.15.88:3300/0 conn(0x7fde54158070 0x7fde541611b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:39.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde58cff640 1 -- 172.21.15.57:0/3310222933 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde54072600 con 0x7fde5415e650 2023-10-30T14:13:39.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde58cff640 1 -- 172.21.15.57:0/3310222933 <== mon.1 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fde44001680 con 0x7fde5415e650 2023-10-30T14:13:39.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde58cff640 1 -- 172.21.15.57:0/3310222933 >> v1:172.21.15.88:6789/0 conn(0x7fde5415e650 legacy=0x7fde540c0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:39.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde58cff640 1 --2- 172.21.15.57:0/3310222933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde54071c30 0x7fde40003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:39.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde58cff640 1 --2- 172.21.15.57:0/3310222933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde40003fb0 0x7fde400047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:39.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde58cff640 1 --2- 172.21.15.57:0/3310222933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde40004d10 0x7fde40005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:39.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde58cff640 1 -- 172.21.15.57:0/3310222933 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde54162170 con 0x7fde40003fb0 2023-10-30T14:13:39.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde58cff640 1 -- 172.21.15.57:0/3310222933 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde541616f0 con 0x7fde40004d10 2023-10-30T14:13:39.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde58cff640 1 -- 172.21.15.57:0/3310222933 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde4000bb20 con 0x7fde54071c30 2023-10-30T14:13:39.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde59500640 1 --2- 172.21.15.57:0/3310222933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde40003fb0 0x7fde400047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:39.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde5a502640 1 --2- 172.21.15.57:0/3310222933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde40004d10 0x7fde40005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:39.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde59d01640 1 --2- 172.21.15.57:0/3310222933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde54071c30 0x7fde40003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:39.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde59500640 1 -- 172.21.15.57:0/3310222933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde54071c30 msgr2=0x7fde40003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:13:39.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde59500640 1 --2- 172.21.15.57:0/3310222933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde54071c30 0x7fde40003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:39.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde59500640 1 -- 172.21.15.57:0/3310222933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde40004d10 msgr2=0x7fde40005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:13:39.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde59500640 1 --2- 172.21.15.57:0/3310222933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde40004d10 0x7fde40005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:39.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.885+0000 7fde59500640 1 -- 172.21.15.57:0/3310222933 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde40003530 con 0x7fde40003fb0 2023-10-30T14:13:39.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.889+0000 7fde59500640 1 --2- 172.21.15.57:0/3310222933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde40003fb0 0x7fde400047d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fde4400c110 tx=0x7fde4401cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f219faa0c6c6a003 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:39.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.889+0000 7fde58cff640 1 -- 172.21.15.57:0/3310222933 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde440a5aa0 con 0x7fde40003fb0 2023-10-30T14:13:39.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde58cff640 1 -- 172.21.15.57:0/3310222933 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde440a5c20 con 0x7fde40003fb0 2023-10-30T14:13:39.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde58cff640 1 -- 172.21.15.57:0/3310222933 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde440ad440 con 0x7fde40003fb0 2023-10-30T14:13:39.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde5bf65640 1 -- 172.21.15.57:0/3310222933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde40003fb0 msgr2=0x7fde400047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:39.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde5bf65640 1 --2- 172.21.15.57:0/3310222933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde40003fb0 0x7fde400047d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fde4400c110 tx=0x7fde4401cba0 comp rx=0 tx=0).stop 2023-10-30T14:13:39.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde5a502640 1 -- 172.21.15.57:0/3310222933 reap_dead start 2023-10-30T14:13:39.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde5bf65640 1 -- 172.21.15.57:0/3310222933 shutdown_connections 2023-10-30T14:13:39.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde5bf65640 1 -- 172.21.15.57:0/3310222933 >> 172.21.15.57:0/3310222933 conn(0x7fde5406da40 msgr2=0x7fde5406de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:39.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde5bf65640 1 -- 172.21.15.57:0/3310222933 shutdown_connections 2023-10-30T14:13:39.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde5bf65640 1 -- 172.21.15.57:0/3310222933 wait complete. 2023-10-30T14:13:39.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde5bf65640 1 Processor -- start 2023-10-30T14:13:39.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde5bf65640 1 -- start start 2023-10-30T14:13:39.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde5bf65640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde40003fb0 0x7fde541f6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:39.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde5bf65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde5414f1a0 0x7fde540c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:39.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde5bf65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde540c6b30 0x7fde54158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:39.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde5bf65640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5414ffc0 con 0x7fde40003fb0 2023-10-30T14:13:39.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde5bf65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde540ca140 con 0x7fde5414f1a0 2023-10-30T14:13:39.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde5bf65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde540cad20 con 0x7fde540c6b30 2023-10-30T14:13:39.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde59d01640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde40003fb0 0x7fde541f6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:39.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde59d01640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde40003fb0 0x7fde541f6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33764/0 (socket says 172.21.15.57:33764) 2023-10-30T14:13:39.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.893+0000 7fde59d01640 1 -- 172.21.15.57:0/2084519923 learned_addr learned my addr 172.21.15.57:0/2084519923 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:13:39.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.897+0000 7fde5a502640 1 --2- 172.21.15.57:0/2084519923 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde540c6b30 0x7fde54158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:39.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.897+0000 7fde59500640 1 --2- 172.21.15.57:0/2084519923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde5414f1a0 0x7fde540c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:39.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.897+0000 7fde59d01640 1 -- 172.21.15.57:0/2084519923 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde540c6b30 msgr2=0x7fde54158d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:39.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.897+0000 7fde59d01640 1 --2- 172.21.15.57:0/2084519923 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde540c6b30 0x7fde54158d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:39.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.897+0000 7fde59d01640 1 -- 172.21.15.57:0/2084519923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde5414f1a0 msgr2=0x7fde540c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:39.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.897+0000 7fde59d01640 1 --2- 172.21.15.57:0/2084519923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde5414f1a0 0x7fde540c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:39.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.897+0000 7fde59d01640 1 -- 172.21.15.57:0/2084519923 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde540cb180 con 0x7fde40003fb0 2023-10-30T14:13:39.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.897+0000 7fde59d01640 1 --2- 172.21.15.57:0/2084519923 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde40003fb0 0x7fde541f6ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fde500128c0 tx=0x7fde50012d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:39.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.897+0000 7fde4a7fc640 1 -- 172.21.15.57:0/2084519923 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde50010980 con 0x7fde40003fb0 2023-10-30T14:13:39.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.897+0000 7fde4a7fc640 1 -- 172.21.15.57:0/2084519923 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5000c040 con 0x7fde40003fb0 2023-10-30T14:13:39.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.897+0000 7fde4a7fc640 1 -- 172.21.15.57:0/2084519923 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde50010ce0 con 0x7fde40003fb0 2023-10-30T14:13:39.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.897+0000 7fde5bf65640 1 -- 172.21.15.57:0/2084519923 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde54158570 con 0x7fde40003fb0 2023-10-30T14:13:39.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.897+0000 7fde5bf65640 1 -- 172.21.15.57:0/2084519923 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde540cb440 con 0x7fde40003fb0 2023-10-30T14:13:39.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.897+0000 7fde4a7fc640 1 -- 172.21.15.57:0/2084519923 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fde5001e710 con 0x7fde40003fb0 2023-10-30T14:13:39.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.897+0000 7fde4a7fc640 1 -- 172.21.15.57:0/2084519923 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fde50015a30 con 0x7fde40003fb0 2023-10-30T14:13:39.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.897+0000 7fde5bf65640 1 -- 172.21.15.57:0/2084519923 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde5404e170 con 0x7fde40003fb0 2023-10-30T14:13:39.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.901+0000 7fde4a7fc640 1 -- 172.21.15.57:0/2084519923 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fde50023060 con 0x7fde40003fb0 2023-10-30T14:13:39.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.961+0000 7fde5bf65640 1 -- 172.21.15.57:0/2084519923 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fde540c0e40 con 0x7fde40003fb0 2023-10-30T14:13:39.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.965+0000 7fde4a7fc640 1 -- 172.21.15.57:0/2084519923 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fde5002c020 con 0x7fde40003fb0 2023-10-30T14:13:39.968 INFO:teuthology.orchestra.run.smithi057.stderr:adjusted tunables profile to default 2023-10-30T14:13:39.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.965+0000 7fde5bf65640 1 -- 172.21.15.57:0/2084519923 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde40003fb0 msgr2=0x7fde541f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:39.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.965+0000 7fde5bf65640 1 --2- 172.21.15.57:0/2084519923 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde40003fb0 0x7fde541f6ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fde500128c0 tx=0x7fde50012d80 comp rx=0 tx=0).stop 2023-10-30T14:13:39.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.965+0000 7fde5bf65640 1 -- 172.21.15.57:0/2084519923 shutdown_connections 2023-10-30T14:13:39.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.965+0000 7fde5bf65640 1 --2- 172.21.15.57:0/2084519923 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde540c6b30 0x7fde54158d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:39.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.965+0000 7fde5bf65640 1 --2- 172.21.15.57:0/2084519923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde5414f1a0 0x7fde540c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:39.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.965+0000 7fde5bf65640 1 --2- 172.21.15.57:0/2084519923 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde40003fb0 0x7fde541f6ea0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:39.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.965+0000 7fde5bf65640 1 -- 172.21.15.57:0/2084519923 >> 172.21.15.57:0/2084519923 conn(0x7fde5406da40 msgr2=0x7fde540c02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:39.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.965+0000 7fde5bf65640 1 -- 172.21.15.57:0/2084519923 shutdown_connections 2023-10-30T14:13:39.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:39.965+0000 7fde5bf65640 1 -- 172.21.15.57:0/2084519923 wait complete. 2023-10-30T14:13:39.989 INFO:tasks.ceph:check_enable_crimson: False 2023-10-30T14:13:39.989 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-30T14:13:39.989 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:13:39.989 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-30T14:13:40.010 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:13:40.011 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-30T14:13:40.074 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:13:40.075 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-30T14:13:40.085 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:40.081+0000 7fb5e302d180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T14:13:40.086 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:40.083+0000 7f49e8105180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T14:13:40.106 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:13:40.106 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-30T14:13:40.171 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:13:40.171 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-30T14:13:40.184 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:13:40.184 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-30T14:13:40.236 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:13:40.237 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-30T14:13:40.291 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-30T14:13:40.292 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-30T14:13:40.343 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new febf18a3-5dae-4a27-855b-c0ec470c8488 0 2023-10-30T14:13:40.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.499+0000 7fd8e68d1640 1 Processor -- start 2023-10-30T14:13:40.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.499+0000 7fd8e68d1640 1 -- start start 2023-10-30T14:13:40.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.503+0000 7fd8e68d1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8e00cb6c0 0x7fd8e006e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:40.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.503+0000 7fd8e68d1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd8e0157a50 0x7fd8e0157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:40.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.503+0000 7fd8e68d1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e0151fd0 con 0x7fd8e00c90e0 2023-10-30T14:13:40.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.503+0000 7fd8e68d1640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e015bc10 con 0x7fd8e0157a50 2023-10-30T14:13:40.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.503+0000 7fd8e68d1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e015ad80 con 0x7fd8e00cb6c0 2023-10-30T14:13:40.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.503+0000 7fd8d77fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd8e00c90e0 0x7fd8e00c5400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50190/0 (socket says 172.21.15.88:50190) 2023-10-30T14:13:40.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.503+0000 7fd8dffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8e00cb6c0 0x7fd8e006e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:40.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.503+0000 7fd8d77fe640 1 -- 172.21.15.88:0/326565350 learned_addr learned my addr 172.21.15.88:0/326565350 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-30T14:13:40.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.503+0000 7fd8e4e6e640 1 --2- 172.21.15.88:0/326565350 >> v2:172.21.15.57:3300/0 conn(0x7fd8e0157a50 0x7fd8e0157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:40.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.503+0000 7fd8df7fe640 1 -- 172.21.15.88:0/326565350 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082177185 0 0) 0x7fd8e0151fd0 con 0x7fd8e00c90e0 2023-10-30T14:13:40.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.503+0000 7fd8df7fe640 1 -- 172.21.15.88:0/326565350 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8cc007940 con 0x7fd8e00c90e0 2023-10-30T14:13:40.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.503+0000 7fd8dffff640 1 -- 172.21.15.88:0/326565350 >> v2:172.21.15.57:3300/0 conn(0x7fd8e0157a50 msgr2=0x7fd8e0157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:13:40.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.503+0000 7fd8dffff640 1 --2- 172.21.15.88:0/326565350 >> v2:172.21.15.57:3300/0 conn(0x7fd8e0157a50 0x7fd8e0157ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.503+0000 7fd8dffff640 1 -- 172.21.15.88:0/326565350 >> v1:172.21.15.88:6789/0 conn(0x7fd8e00c90e0 legacy=0x7fd8e00c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:40.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.503+0000 7fd8dffff640 1 -- 172.21.15.88:0/326565350 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e0158420 con 0x7fd8e00cb6c0 2023-10-30T14:13:40.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.503+0000 7fd8dffff640 1 --2- 172.21.15.88:0/326565350 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8e00cb6c0 0x7fd8e006e6a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd8d00097c0 tx=0x7fd8d001ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a922e8634104904c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:40.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.503+0000 7fd8df7fe640 1 -- 172.21.15.88:0/326565350 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d00a8970 con 0x7fd8e00cb6c0 2023-10-30T14:13:40.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.503+0000 7fd8df7fe640 1 -- 172.21.15.88:0/326565350 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d00a8ad0 con 0x7fd8e00cb6c0 2023-10-30T14:13:40.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.503+0000 7fd8df7fe640 1 -- 172.21.15.88:0/326565350 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d00a8e60 con 0x7fd8e00cb6c0 2023-10-30T14:13:40.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e68d1640 1 -- 172.21.15.88:0/326565350 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8e00cb6c0 msgr2=0x7fd8e006e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:40.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e68d1640 1 --2- 172.21.15.88:0/326565350 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8e00cb6c0 0x7fd8e006e6a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd8d00097c0 tx=0x7fd8d001ec70 comp rx=0 tx=0).stop 2023-10-30T14:13:40.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e68d1640 1 -- 172.21.15.88:0/326565350 shutdown_connections 2023-10-30T14:13:40.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e68d1640 1 --2- 172.21.15.88:0/326565350 >> v2:172.21.15.57:3300/0 conn(0x7fd8e0157a50 0x7fd8e0157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e68d1640 1 --2- 172.21.15.88:0/326565350 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8e00cb6c0 0x7fd8e006e6a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e68d1640 1 -- 172.21.15.88:0/326565350 >> 172.21.15.88:0/326565350 conn(0x7fd8e006afa0 msgr2=0x7fd8e006b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:40.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e68d1640 1 -- 172.21.15.88:0/326565350 shutdown_connections 2023-10-30T14:13:40.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e68d1640 1 -- 172.21.15.88:0/326565350 wait complete. 2023-10-30T14:13:40.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e68d1640 1 Processor -- start 2023-10-30T14:13:40.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e68d1640 1 -- start start 2023-10-30T14:13:40.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e68d1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8e00c90e0 0x7fd8e00c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:40.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e68d1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8e00cb6c0 0x7fd8e014f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:40.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e68d1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8e0157a50 0x7fd8e014f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:40.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e68d1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e00c1c70 con 0x7fd8e0157a50 2023-10-30T14:13:40.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e68d1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e00729f0 con 0x7fd8e00c90e0 2023-10-30T14:13:40.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e68d1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e00c1e80 con 0x7fd8e00cb6c0 2023-10-30T14:13:40.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8dffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8e00c90e0 0x7fd8e00c44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:40.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8d77fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8e00cb6c0 0x7fd8e014f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:40.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8dffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8e00c90e0 0x7fd8e00c44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41166/0 (socket says 172.21.15.88:41166) 2023-10-30T14:13:40.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8dffff640 1 -- 172.21.15.88:0/1988589795 learned_addr learned my addr 172.21.15.88:0/1988589795 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-30T14:13:40.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e4e6e640 1 --2- 172.21.15.88:0/1988589795 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8e0157a50 0x7fd8e014f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:40.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8d77fe640 1 -- 172.21.15.88:0/1988589795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8e00c90e0 msgr2=0x7fd8e00c44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:40.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8d77fe640 1 --2- 172.21.15.88:0/1988589795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8e00c90e0 0x7fd8e00c44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8d77fe640 1 -- 172.21.15.88:0/1988589795 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8e0157a50 msgr2=0x7fd8e014f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:40.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8d77fe640 1 --2- 172.21.15.88:0/1988589795 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8e0157a50 0x7fd8e014f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8d77fe640 1 -- 172.21.15.88:0/1988589795 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d0009420 con 0x7fd8e00cb6c0 2023-10-30T14:13:40.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e4e6e640 1 --2- 172.21.15.88:0/1988589795 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8e0157a50 0x7fd8e014f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:13:40.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8dffff640 1 --2- 172.21.15.88:0/1988589795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8e00c90e0 0x7fd8e00c44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:13:40.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8d77fe640 1 --2- 172.21.15.88:0/1988589795 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8e00cb6c0 0x7fd8e014f140 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800c400 tx=0x7fd8c800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:40.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8dd7fa640 1 -- 172.21.15.88:0/1988589795 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c8005cf0 con 0x7fd8e00cb6c0 2023-10-30T14:13:40.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8dd7fa640 1 -- 172.21.15.88:0/1988589795 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c8005e50 con 0x7fd8e00cb6c0 2023-10-30T14:13:40.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8dd7fa640 1 -- 172.21.15.88:0/1988589795 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c80072d0 con 0x7fd8e00cb6c0 2023-10-30T14:13:40.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e68d1640 1 -- 172.21.15.88:0/1988589795 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e00c2160 con 0x7fd8e00cb6c0 2023-10-30T14:13:40.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8e68d1640 1 -- 172.21.15.88:0/1988589795 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e00cbdd0 con 0x7fd8e00cb6c0 2023-10-30T14:13:40.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8dd7fa640 1 -- 172.21.15.88:0/1988589795 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd8c801c440 con 0x7fd8e00cb6c0 2023-10-30T14:13:40.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.507+0000 7fd8dd7fa640 1 -- 172.21.15.88:0/1988589795 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd8c8012780 con 0x7fd8e00cb6c0 2023-10-30T14:13:40.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.511+0000 7fd8e68d1640 1 -- 172.21.15.88:0/1988589795 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e014ee60 con 0x7fd8e00cb6c0 2023-10-30T14:13:40.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.519+0000 7fd8dd7fa640 1 -- 172.21.15.88:0/1988589795 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fd8e014ee60 con 0x7fd8e00cb6c0 2023-10-30T14:13:40.591 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:40.587+0000 7f49e8105180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T14:13:40.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.587+0000 7fd8e68d1640 1 -- 172.21.15.88:0/1988589795 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd new", "uuid": "febf18a3-5dae-4a27-855b-c0ec470c8488", "id": 0} v 0) v1 -- 0x7fd8e0067540 con 0x7fd8e00cb6c0 2023-10-30T14:13:40.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.595+0000 7fd8dd7fa640 1 -- 172.21.15.88:0/1988589795 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "febf18a3-5dae-4a27-855b-c0ec470c8488", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd8c8021060 con 0x7fd8e00cb6c0 2023-10-30T14:13:40.600 INFO:teuthology.orchestra.run.smithi088.stdout:0 2023-10-30T14:13:40.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.595+0000 7fd8e68d1640 1 -- 172.21.15.88:0/1988589795 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8e00cb6c0 msgr2=0x7fd8e014f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:40.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.595+0000 7fd8e68d1640 1 --2- 172.21.15.88:0/1988589795 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8e00cb6c0 0x7fd8e014f140 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800c400 tx=0x7fd8c800c8c0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.599+0000 7fd8e68d1640 1 -- 172.21.15.88:0/1988589795 shutdown_connections 2023-10-30T14:13:40.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.599+0000 7fd8e68d1640 1 --2- 172.21.15.88:0/1988589795 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8e0157a50 0x7fd8e014f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.599+0000 7fd8e68d1640 1 --2- 172.21.15.88:0/1988589795 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8e00cb6c0 0x7fd8e014f140 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.599+0000 7fd8e68d1640 1 --2- 172.21.15.88:0/1988589795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8e00c90e0 0x7fd8e00c44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.599+0000 7fd8e68d1640 1 -- 172.21.15.88:0/1988589795 >> 172.21.15.88:0/1988589795 conn(0x7fd8e006afa0 msgr2=0x7fd8e0151560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:40.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.599+0000 7fd8e68d1640 1 -- 172.21.15.88:0/1988589795 shutdown_connections 2023-10-30T14:13:40.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.599+0000 7fd8e68d1640 1 -- 172.21.15.88:0/1988589795 wait complete. 2023-10-30T14:13:40.616 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new 049efcb1-8448-4df9-9193-22747f733904 1 2023-10-30T14:13:40.656 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:40.653+0000 7fb5e302d180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T14:13:40.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.731+0000 7f23e58d1640 1 Processor -- start 2023-10-30T14:13:40.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.731+0000 7f23e58d1640 1 -- start start 2023-10-30T14:13:40.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.731+0000 7f23e58d1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23e00c5f50 0x7f23e00c7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:40.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.731+0000 7f23e58d1640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e0159fc0 con 0x7f23e006a4d0 2023-10-30T14:13:40.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.731+0000 7f23e58d1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e006f670 con 0x7f23e006a0f0 2023-10-30T14:13:40.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.731+0000 7f23e58d1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e006f0b0 con 0x7f23e00c5f50 2023-10-30T14:13:40.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.731+0000 7f23deffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f23e006a0f0 0x7f23e00c5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50192/0 (socket says 172.21.15.88:50192) 2023-10-30T14:13:40.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.731+0000 7f23deffd640 1 -- 172.21.15.88:0/560283339 learned_addr learned my addr 172.21.15.88:0/560283339 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-30T14:13:40.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.731+0000 7f23df7fe640 1 --2- 172.21.15.88:0/560283339 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23e00c5f50 0x7f23e00c7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:40.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.731+0000 7f23de7fc640 1 -- 172.21.15.88:0/560283339 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647204463 0 0) 0x7f23e006f670 con 0x7f23e006a0f0 2023-10-30T14:13:40.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.731+0000 7f23de7fc640 1 -- 172.21.15.88:0/560283339 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23cc003530 con 0x7f23e006a0f0 2023-10-30T14:13:40.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.731+0000 7f23df7fe640 1 -- 172.21.15.88:0/560283339 >> v1:172.21.15.88:6789/0 conn(0x7f23e006a0f0 legacy=0x7f23e00c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:40.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.731+0000 7f23df7fe640 1 -- 172.21.15.88:0/560283339 >> v1:172.21.15.57:6789/0 conn(0x7f23e006a4d0 legacy=0x7f23e0162160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:40.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.731+0000 7f23df7fe640 1 -- 172.21.15.88:0/560283339 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e006f390 con 0x7f23e00c5f50 2023-10-30T14:13:40.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23df7fe640 1 --2- 172.21.15.88:0/560283339 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23e00c5f50 0x7f23e00c7180 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f23d8009dc0 tx=0x7f23d801c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a38ba291bab763c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:40.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23de7fc640 1 -- 172.21.15.88:0/560283339 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d80a5920 con 0x7f23e00c5f50 2023-10-30T14:13:40.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23de7fc640 1 -- 172.21.15.88:0/560283339 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d80a5a80 con 0x7f23e00c5f50 2023-10-30T14:13:40.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23de7fc640 1 -- 172.21.15.88:0/560283339 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d80a5e10 con 0x7f23e00c5f50 2023-10-30T14:13:40.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23e58d1640 1 -- 172.21.15.88:0/560283339 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23e00c5f50 msgr2=0x7f23e00c7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:40.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23e58d1640 1 --2- 172.21.15.88:0/560283339 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23e00c5f50 0x7f23e00c7180 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f23d8009dc0 tx=0x7f23d801c970 comp rx=0 tx=0).stop 2023-10-30T14:13:40.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23e58d1640 1 -- 172.21.15.88:0/560283339 shutdown_connections 2023-10-30T14:13:40.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23e58d1640 1 --2- 172.21.15.88:0/560283339 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23e00c5f50 0x7f23e00c7180 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23e58d1640 1 -- 172.21.15.88:0/560283339 >> 172.21.15.88:0/560283339 conn(0x7f23e006afa0 msgr2=0x7f23e006b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:40.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23e58d1640 1 -- 172.21.15.88:0/560283339 shutdown_connections 2023-10-30T14:13:40.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23e58d1640 1 -- 172.21.15.88:0/560283339 wait complete. 2023-10-30T14:13:40.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23e58d1640 1 Processor -- start 2023-10-30T14:13:40.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23e58d1640 1 -- start start 2023-10-30T14:13:40.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23e58d1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23e006a0f0 0x7f23e01f5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:40.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23e58d1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23e006a4d0 0x7f23e0149970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:40.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23e58d1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23e00c5f50 0x7f23e0149eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:40.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23e58d1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e01f5f20 con 0x7f23e006a4d0 2023-10-30T14:13:40.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23e58d1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e01f6200 con 0x7f23e00c5f50 2023-10-30T14:13:40.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23e58d1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e014a730 con 0x7f23e006a0f0 2023-10-30T14:13:40.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23deffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23e006a0f0 0x7f23e01f5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:40.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23df7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23e00c5f50 0x7f23e0149eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:40.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23deffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23e006a0f0 0x7f23e01f5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:38314/0 (socket says 172.21.15.88:38314) 2023-10-30T14:13:40.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23deffd640 1 -- 172.21.15.88:0/182871397 learned_addr learned my addr 172.21.15.88:0/182871397 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-30T14:13:40.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.735+0000 7f23d7fff640 1 --2- 172.21.15.88:0/182871397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23e006a4d0 0x7f23e0149970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:40.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.739+0000 7f23df7fe640 1 -- 172.21.15.88:0/182871397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23e006a0f0 msgr2=0x7f23e01f5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:40.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.739+0000 7f23df7fe640 1 --2- 172.21.15.88:0/182871397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23e006a0f0 0x7f23e01f5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.739+0000 7f23df7fe640 1 -- 172.21.15.88:0/182871397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23e006a4d0 msgr2=0x7f23e0149970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:40.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.739+0000 7f23df7fe640 1 --2- 172.21.15.88:0/182871397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23e006a4d0 0x7f23e0149970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.739+0000 7f23df7fe640 1 -- 172.21.15.88:0/182871397 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d8009a70 con 0x7f23e00c5f50 2023-10-30T14:13:40.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.739+0000 7f23df7fe640 1 --2- 172.21.15.88:0/182871397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23e00c5f50 0x7f23e0149eb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f23d80ac000 tx=0x7f23d8019710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:40.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.739+0000 7f23dcff9640 1 -- 172.21.15.88:0/182871397 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d80bf070 con 0x7f23e00c5f50 2023-10-30T14:13:40.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.739+0000 7f23dcff9640 1 -- 172.21.15.88:0/182871397 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d80a58a0 con 0x7f23e00c5f50 2023-10-30T14:13:40.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.739+0000 7f23e58d1640 1 -- 172.21.15.88:0/182871397 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e014aa10 con 0x7f23e00c5f50 2023-10-30T14:13:40.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.739+0000 7f23dcff9640 1 -- 172.21.15.88:0/182871397 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d80ad870 con 0x7f23e00c5f50 2023-10-30T14:13:40.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.739+0000 7f23e58d1640 1 -- 172.21.15.88:0/182871397 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e014afd0 con 0x7f23e00c5f50 2023-10-30T14:13:40.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.739+0000 7f23dcff9640 1 -- 172.21.15.88:0/182871397 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f23d80b4050 con 0x7f23e00c5f50 2023-10-30T14:13:40.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.739+0000 7f23dcff9640 1 -- 172.21.15.88:0/182871397 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f23d8004620 con 0x7f23e00c5f50 2023-10-30T14:13:40.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.739+0000 7f23e58d1640 1 -- 172.21.15.88:0/182871397 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e0157800 con 0x7f23e00c5f50 2023-10-30T14:13:40.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.739+0000 7f23dcff9640 1 -- 172.21.15.88:0/182871397 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f23d80b2070 con 0x7f23e00c5f50 2023-10-30T14:13:40.784 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:40.779+0000 7f49e8105180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T14:13:40.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.807+0000 7f23e58d1640 1 -- 172.21.15.88:0/182871397 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd new", "uuid": "049efcb1-8448-4df9-9193-22747f733904", "id": 1} v 0) v1 -- 0x7f23e0067540 con 0x7f23e00c5f50 2023-10-30T14:13:40.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.815+0000 7f23dcff9640 1 -- 172.21.15.88:0/182871397 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "049efcb1-8448-4df9-9193-22747f733904", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f23d80d20b0 con 0x7f23e00c5f50 2023-10-30T14:13:40.818 INFO:teuthology.orchestra.run.smithi088.stdout:1 2023-10-30T14:13:40.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.815+0000 7f23e58d1640 1 -- 172.21.15.88:0/182871397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23e00c5f50 msgr2=0x7f23e0149eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:40.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.815+0000 7f23e58d1640 1 --2- 172.21.15.88:0/182871397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23e00c5f50 0x7f23e0149eb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f23d80ac000 tx=0x7f23d8019710 comp rx=0 tx=0).stop 2023-10-30T14:13:40.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.815+0000 7f23e58d1640 1 -- 172.21.15.88:0/182871397 shutdown_connections 2023-10-30T14:13:40.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.815+0000 7f23e58d1640 1 --2- 172.21.15.88:0/182871397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23e00c5f50 0x7f23e0149eb0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.815+0000 7f23e58d1640 1 --2- 172.21.15.88:0/182871397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23e006a4d0 0x7f23e0149970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.815+0000 7f23e58d1640 1 --2- 172.21.15.88:0/182871397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23e006a0f0 0x7f23e01f5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.815+0000 7f23e58d1640 1 -- 172.21.15.88:0/182871397 >> 172.21.15.88:0/182871397 conn(0x7f23e006afa0 msgr2=0x7f23e0157ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:40.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.815+0000 7f23e58d1640 1 -- 172.21.15.88:0/182871397 shutdown_connections 2023-10-30T14:13:40.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.815+0000 7f23e58d1640 1 -- 172.21.15.88:0/182871397 wait complete. 2023-10-30T14:13:40.835 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new 1da21529-8703-49f1-841e-e79fd29ab5b7 2 2023-10-30T14:13:40.873 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:40.869+0000 7fb5e302d180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T14:13:40.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.947+0000 7f78a710c640 1 Processor -- start 2023-10-30T14:13:40.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.947+0000 7f78a710c640 1 -- start start 2023-10-30T14:13:40.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f78a710c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a01508f0 0x7f78a00c4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:40.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f78a710c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f78a0070680 0x7f78a006e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:40.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f78a710c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a0071710 con 0x7f78a006e5b0 2023-10-30T14:13:40.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f78a710c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a00c4df0 con 0x7f78a0070680 2023-10-30T14:13:40.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f78a710c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a00c4830 con 0x7f78a01508f0 2023-10-30T14:13:40.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f78a56a9640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f78a006e5b0 0x7f78a0159bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50206/0 (socket says 172.21.15.88:50206) 2023-10-30T14:13:40.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f78a4ea8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a01508f0 0x7f78a00c4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:40.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f78a56a9640 1 -- 172.21.15.88:0/3261879115 learned_addr learned my addr 172.21.15.88:0/3261879115 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-30T14:13:40.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f789ffff640 1 --2- 172.21.15.88:0/3261879115 >> v2:172.21.15.57:3300/0 conn(0x7f78a0070680 0x7f78a006e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:40.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f78a4ea8640 1 -- 172.21.15.88:0/3261879115 >> v2:172.21.15.57:3300/0 conn(0x7f78a0070680 msgr2=0x7f78a006e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:13:40.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f78a4ea8640 1 --2- 172.21.15.88:0/3261879115 >> v2:172.21.15.57:3300/0 conn(0x7f78a0070680 0x7f78a006e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f789f7fe640 1 -- 172.21.15.88:0/3261879115 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1200547520 0 0) 0x7f78a0071710 con 0x7f78a006e5b0 2023-10-30T14:13:40.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f78a4ea8640 1 -- 172.21.15.88:0/3261879115 >> v1:172.21.15.88:6789/0 conn(0x7f78a006e5b0 legacy=0x7f78a0159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:40.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f78a4ea8640 1 -- 172.21.15.88:0/3261879115 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a00c4b10 con 0x7f78a01508f0 2023-10-30T14:13:40.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f78a4ea8640 1 --2- 172.21.15.88:0/3261879115 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a01508f0 0x7f78a00c4240 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f78940097c0 tx=0x7f789401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2144ca66d1759a8d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:40.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f789f7fe640 1 -- 172.21.15.88:0/3261879115 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78940a8970 con 0x7f78a01508f0 2023-10-30T14:13:40.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f789f7fe640 1 -- 172.21.15.88:0/3261879115 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78940a8ad0 con 0x7f78a01508f0 2023-10-30T14:13:40.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f789f7fe640 1 -- 172.21.15.88:0/3261879115 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78940a8e60 con 0x7f78a01508f0 2023-10-30T14:13:40.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f78a710c640 1 -- 172.21.15.88:0/3261879115 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a01508f0 msgr2=0x7f78a00c4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:40.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.951+0000 7f78a710c640 1 --2- 172.21.15.88:0/3261879115 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a01508f0 0x7f78a00c4240 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f78940097c0 tx=0x7f789401ec70 comp rx=0 tx=0).stop 2023-10-30T14:13:40.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a710c640 1 -- 172.21.15.88:0/3261879115 shutdown_connections 2023-10-30T14:13:40.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a710c640 1 --2- 172.21.15.88:0/3261879115 >> v2:172.21.15.57:3300/0 conn(0x7f78a0070680 0x7f78a006e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a710c640 1 --2- 172.21.15.88:0/3261879115 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a01508f0 0x7f78a00c4240 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a710c640 1 -- 172.21.15.88:0/3261879115 >> 172.21.15.88:0/3261879115 conn(0x7f78a0069e90 msgr2=0x7f78a006a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:40.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a710c640 1 -- 172.21.15.88:0/3261879115 shutdown_connections 2023-10-30T14:13:40.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a710c640 1 -- 172.21.15.88:0/3261879115 wait complete. 2023-10-30T14:13:40.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a710c640 1 Processor -- start 2023-10-30T14:13:40.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a710c640 1 -- start start 2023-10-30T14:13:40.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a710c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78a006e5b0 0x7f78a014a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a710c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a0070680 0x7f78a00cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a710c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78a01508f0 0x7f78a0148250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a710c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a01ed390 con 0x7f78a006e5b0 2023-10-30T14:13:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a710c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a014b630 con 0x7f78a01508f0 2023-10-30T14:13:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a710c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a014b910 con 0x7f78a0070680 2023-10-30T14:13:40.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f789ffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a0070680 0x7f78a00cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:40.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a56a9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78a01508f0 0x7f78a0148250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:40.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f789ffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a0070680 0x7f78a00cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:38330/0 (socket says 172.21.15.88:38330) 2023-10-30T14:13:40.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f789ffff640 1 -- 172.21.15.88:0/2302854804 learned_addr learned my addr 172.21.15.88:0/2302854804 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-30T14:13:40.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a4ea8640 1 --2- 172.21.15.88:0/2302854804 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78a006e5b0 0x7f78a014a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:40.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f789ffff640 1 -- 172.21.15.88:0/2302854804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78a01508f0 msgr2=0x7f78a0148250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:40.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f789ffff640 1 --2- 172.21.15.88:0/2302854804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78a01508f0 0x7f78a0148250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f789ffff640 1 -- 172.21.15.88:0/2302854804 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78a006e5b0 msgr2=0x7f78a014a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:40.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f789ffff640 1 --2- 172.21.15.88:0/2302854804 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78a006e5b0 0x7f78a014a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:40.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f789ffff640 1 -- 172.21.15.88:0/2302854804 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7894009420 con 0x7f78a0070680 2023-10-30T14:13:40.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a56a9640 1 --2- 172.21.15.88:0/2302854804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78a01508f0 0x7f78a0148250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:13:40.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a4ea8640 1 --2- 172.21.15.88:0/2302854804 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78a006e5b0 0x7f78a014a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:13:40.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f789ffff640 1 --2- 172.21.15.88:0/2302854804 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a0070680 0x7f78a00cb7b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f788c007f40 tx=0x7f788c00dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:40.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f789d7fa640 1 -- 172.21.15.88:0/2302854804 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c00b8c0 con 0x7f78a0070680 2023-10-30T14:13:40.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f789d7fa640 1 -- 172.21.15.88:0/2302854804 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788c016070 con 0x7f78a0070680 2023-10-30T14:13:40.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a710c640 1 -- 172.21.15.88:0/2302854804 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a014bbf0 con 0x7f78a0070680 2023-10-30T14:13:40.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f789d7fa640 1 -- 172.21.15.88:0/2302854804 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c00bc20 con 0x7f78a0070680 2023-10-30T14:13:40.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a710c640 1 -- 172.21.15.88:0/2302854804 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a01ed030 con 0x7f78a0070680 2023-10-30T14:13:40.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f789d7fa640 1 -- 172.21.15.88:0/2302854804 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f788c0128e0 con 0x7f78a0070680 2023-10-30T14:13:40.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f789d7fa640 1 -- 172.21.15.88:0/2302854804 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f788c018070 con 0x7f78a0070680 2023-10-30T14:13:40.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.955+0000 7f78a710c640 1 -- 172.21.15.88:0/2302854804 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a0067540 con 0x7f78a0070680 2023-10-30T14:13:40.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:40.959+0000 7f789d7fa640 1 -- 172.21.15.88:0/2302854804 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f788c00bd80 con 0x7f78a0070680 2023-10-30T14:13:41.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.027+0000 7f78a710c640 1 -- 172.21.15.88:0/2302854804 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1da21529-8703-49f1-841e-e79fd29ab5b7", "id": 2} v 0) v1 -- 0x7f78a00c7ae0 con 0x7f78a0070680 2023-10-30T14:13:41.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.035+0000 7f789d7fa640 1 -- 172.21.15.88:0/2302854804 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1da21529-8703-49f1-841e-e79fd29ab5b7", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f788c012b90 con 0x7f78a0070680 2023-10-30T14:13:41.037 INFO:teuthology.orchestra.run.smithi088.stdout:2 2023-10-30T14:13:41.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.035+0000 7f78a710c640 1 -- 172.21.15.88:0/2302854804 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a0070680 msgr2=0x7f78a00cb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.035+0000 7f78a710c640 1 --2- 172.21.15.88:0/2302854804 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a0070680 0x7f78a00cb7b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f788c007f40 tx=0x7f788c00dbe0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.035+0000 7f78a710c640 1 -- 172.21.15.88:0/2302854804 shutdown_connections 2023-10-30T14:13:41.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.035+0000 7f78a710c640 1 --2- 172.21.15.88:0/2302854804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78a01508f0 0x7f78a0148250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.035+0000 7f78a710c640 1 --2- 172.21.15.88:0/2302854804 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a0070680 0x7f78a00cb7b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.035+0000 7f78a710c640 1 --2- 172.21.15.88:0/2302854804 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78a006e5b0 0x7f78a014a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.035+0000 7f78a710c640 1 -- 172.21.15.88:0/2302854804 >> 172.21.15.88:0/2302854804 conn(0x7f78a0069e90 msgr2=0x7f78a014fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:41.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.035+0000 7f78a710c640 1 -- 172.21.15.88:0/2302854804 shutdown_connections 2023-10-30T14:13:41.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.035+0000 7f78a710c640 1 -- 172.21.15.88:0/2302854804 wait complete. 2023-10-30T14:13:41.054 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new e4b3c017-ffee-4175-9aae-f33482d1ba32 3 2023-10-30T14:13:41.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.163+0000 7f0d41369640 1 Processor -- start 2023-10-30T14:13:41.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.163+0000 7f0d41369640 1 -- start start 2023-10-30T14:13:41.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.163+0000 7f0d41369640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0d3c0cd560 0x7f0d3c070600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:41.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.163+0000 7f0d41369640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d3c15a490 0x7f0d3c153830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:41.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.163+0000 7f0d41369640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d3c15a9d0 con 0x7f0d3c0cd180 2023-10-30T14:13:41.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.163+0000 7f0d41369640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d3c06bce0 con 0x7f0d3c0cd560 2023-10-30T14:13:41.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.163+0000 7f0d41369640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c06c170 con 0x7f0d3c15a490 2023-10-30T14:13:41.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.163+0000 7f0d3b7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d3c15a490 0x7f0d3c153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:41.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.163+0000 7f0d3b7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d3c15a490 0x7f0d3c153830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:38340/0 (socket says 172.21.15.88:38340) 2023-10-30T14:13:41.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.163+0000 7f0d3b7fe640 1 -- 172.21.15.88:0/3889258020 learned_addr learned my addr 172.21.15.88:0/3889258020 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-30T14:13:41.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.163+0000 7f0d3a7fc640 1 --2- 172.21.15.88:0/3889258020 >> v2:172.21.15.57:3300/0 conn(0x7f0d3c0cd560 0x7f0d3c070600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:41.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.163+0000 7f0d3b7fe640 1 -- 172.21.15.88:0/3889258020 >> v2:172.21.15.57:3300/0 conn(0x7f0d3c0cd560 msgr2=0x7f0d3c070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:13:41.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.163+0000 7f0d3b7fe640 1 --2- 172.21.15.88:0/3889258020 >> v2:172.21.15.57:3300/0 conn(0x7f0d3c0cd560 0x7f0d3c070600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.163+0000 7f0d3b7fe640 1 -- 172.21.15.88:0/3889258020 >> v1:172.21.15.57:6789/0 conn(0x7f0d3c0cd180 legacy=0x7f0d3c0c2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:13:41.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.163+0000 7f0d3b7fe640 1 -- 172.21.15.88:0/3889258020 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3c06b0d0 con 0x7f0d3c15a490 2023-10-30T14:13:41.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.167+0000 7f0d3b7fe640 1 --2- 172.21.15.88:0/3889258020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d3c15a490 0x7f0d3c153830 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0d34009e50 tx=0x7f0d3401c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d54d19dc1c8878e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:41.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.167+0000 7f0d39ffb640 1 -- 172.21.15.88:0/3889258020 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d340a5820 con 0x7f0d3c15a490 2023-10-30T14:13:41.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.167+0000 7f0d39ffb640 1 -- 172.21.15.88:0/3889258020 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d340a5980 con 0x7f0d3c15a490 2023-10-30T14:13:41.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.167+0000 7f0d39ffb640 1 -- 172.21.15.88:0/3889258020 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d340a5d10 con 0x7f0d3c15a490 2023-10-30T14:13:41.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.167+0000 7f0d41369640 1 -- 172.21.15.88:0/3889258020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d3c15a490 msgr2=0x7f0d3c153830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.167+0000 7f0d41369640 1 --2- 172.21.15.88:0/3889258020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d3c15a490 0x7f0d3c153830 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0d34009e50 tx=0x7f0d3401c9a0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.167+0000 7f0d41369640 1 -- 172.21.15.88:0/3889258020 shutdown_connections 2023-10-30T14:13:41.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.167+0000 7f0d41369640 1 --2- 172.21.15.88:0/3889258020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d3c15a490 0x7f0d3c153830 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.167+0000 7f0d41369640 1 --2- 172.21.15.88:0/3889258020 >> v2:172.21.15.57:3300/0 conn(0x7f0d3c0cd560 0x7f0d3c070600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.167+0000 7f0d41369640 1 -- 172.21.15.88:0/3889258020 >> 172.21.15.88:0/3889258020 conn(0x7f0d3c072010 msgr2=0x7f0d3c072420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:41.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.167+0000 7f0d41369640 1 -- 172.21.15.88:0/3889258020 shutdown_connections 2023-10-30T14:13:41.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.167+0000 7f0d41369640 1 -- 172.21.15.88:0/3889258020 wait complete. 2023-10-30T14:13:41.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d41369640 1 Processor -- start 2023-10-30T14:13:41.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d41369640 1 -- start start 2023-10-30T14:13:41.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d41369640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d3c0cd180 0x7f0d3c1ec250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:41.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d41369640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d3c0cd560 0x7f0d3c1f5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:41.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d41369640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d3c15a490 0x7f0d3c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:41.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d41369640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c1f59d0 con 0x7f0d3c0cd560 2023-10-30T14:13:41.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d41369640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c1f2ca0 con 0x7f0d3c15a490 2023-10-30T14:13:41.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d41369640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c154520 con 0x7f0d3c0cd180 2023-10-30T14:13:41.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d3affd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d3c0cd180 0x7f0d3c1ec250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:41.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d3b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d3c15a490 0x7f0d3c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:41.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d3affd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d3c0cd180 0x7f0d3c1ec250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:38342/0 (socket says 172.21.15.88:38342) 2023-10-30T14:13:41.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d3b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d3c15a490 0x7f0d3c15ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41186/0 (socket says 172.21.15.88:41186) 2023-10-30T14:13:41.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d3affd640 1 -- 172.21.15.88:0/2229267516 learned_addr learned my addr 172.21.15.88:0/2229267516 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-30T14:13:41.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d3affd640 1 -- 172.21.15.88:0/2229267516 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d3c15a490 msgr2=0x7f0d3c15ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d3affd640 1 --2- 172.21.15.88:0/2229267516 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d3c15a490 0x7f0d3c15ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d3affd640 1 -- 172.21.15.88:0/2229267516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d3c0cd560 msgr2=0x7f0d3c1f5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d3affd640 1 --2- 172.21.15.88:0/2229267516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d3c0cd560 0x7f0d3c1f5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d3affd640 1 -- 172.21.15.88:0/2229267516 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d34009b30 con 0x7f0d3c0cd180 2023-10-30T14:13:41.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d3b7fe640 1 --2- 172.21.15.88:0/2229267516 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d3c15a490 0x7f0d3c15ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:13:41.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d3affd640 1 --2- 172.21.15.88:0/2229267516 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d3c0cd180 0x7f0d3c1ec250 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c00b410 tx=0x7f0d2c00b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:41.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d237fe640 1 -- 172.21.15.88:0/2229267516 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c004020 con 0x7f0d3c0cd180 2023-10-30T14:13:41.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d237fe640 1 -- 172.21.15.88:0/2229267516 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d2c004180 con 0x7f0d3c0cd180 2023-10-30T14:13:41.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d41369640 1 -- 172.21.15.88:0/2229267516 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d3c1f2f40 con 0x7f0d3c0cd180 2023-10-30T14:13:41.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d237fe640 1 -- 172.21.15.88:0/2229267516 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c011630 con 0x7f0d3c0cd180 2023-10-30T14:13:41.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d41369640 1 -- 172.21.15.88:0/2229267516 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d3c154800 con 0x7f0d3c0cd180 2023-10-30T14:13:41.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d237fe640 1 -- 172.21.15.88:0/2229267516 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0d2c011a90 con 0x7f0d3c0cd180 2023-10-30T14:13:41.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d237fe640 1 -- 172.21.15.88:0/2229267516 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f0d2c003dd0 con 0x7f0d3c0cd180 2023-10-30T14:13:41.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.171+0000 7f0d41369640 1 -- 172.21.15.88:0/2229267516 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d3c067540 con 0x7f0d3c0cd180 2023-10-30T14:13:41.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.175+0000 7f0d237fe640 1 -- 172.21.15.88:0/2229267516 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f0d2c015030 con 0x7f0d3c0cd180 2023-10-30T14:13:41.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.239+0000 7f0d41369640 1 -- 172.21.15.88:0/2229267516 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e4b3c017-ffee-4175-9aae-f33482d1ba32", "id": 3} v 0) v1 -- 0x7f0d3c15de20 con 0x7f0d3c0cd180 2023-10-30T14:13:41.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.243+0000 7f0d237fe640 1 -- 172.21.15.88:0/2229267516 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e4b3c017-ffee-4175-9aae-f33482d1ba32", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0d2c0036d0 con 0x7f0d3c0cd180 2023-10-30T14:13:41.249 INFO:teuthology.orchestra.run.smithi088.stdout:3 2023-10-30T14:13:41.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.247+0000 7f0d41369640 1 -- 172.21.15.88:0/2229267516 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d3c0cd180 msgr2=0x7f0d3c1ec250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.247+0000 7f0d41369640 1 --2- 172.21.15.88:0/2229267516 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d3c0cd180 0x7f0d3c1ec250 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c00b410 tx=0x7f0d2c00b8d0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.247+0000 7f0d41369640 1 -- 172.21.15.88:0/2229267516 shutdown_connections 2023-10-30T14:13:41.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.247+0000 7f0d41369640 1 --2- 172.21.15.88:0/2229267516 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d3c15a490 0x7f0d3c15ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.247+0000 7f0d41369640 1 --2- 172.21.15.88:0/2229267516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d3c0cd560 0x7f0d3c1f5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.247+0000 7f0d41369640 1 --2- 172.21.15.88:0/2229267516 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d3c0cd180 0x7f0d3c1ec250 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.247+0000 7f0d41369640 1 -- 172.21.15.88:0/2229267516 >> 172.21.15.88:0/2229267516 conn(0x7f0d3c072010 msgr2=0x7f0d3c0c2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:41.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.247+0000 7f0d41369640 1 -- 172.21.15.88:0/2229267516 shutdown_connections 2023-10-30T14:13:41.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.247+0000 7f0d41369640 1 -- 172.21.15.88:0/2229267516 wait complete. 2023-10-30T14:13:41.265 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new 1332a9e0-38b6-4d83-b2d5-f0cd2306e983 4 2023-10-30T14:13:41.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.375+0000 7fc44f8b0640 1 Processor -- start 2023-10-30T14:13:41.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.375+0000 7fc44f8b0640 1 -- start start 2023-10-30T14:13:41.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.375+0000 7fc44f8b0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4481597d0 0x7fc44815d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:41.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44f8b0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc44815da70 0x7fc448069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:41.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44f8b0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc44815cdc0 con 0x7fc4480c5dc0 2023-10-30T14:13:41.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44f8b0640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc44806f600 con 0x7fc44815da70 2023-10-30T14:13:41.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44f8b0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4480ca100 con 0x7fc4481597d0 2023-10-30T14:13:41.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44d64c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc4480c5dc0 0x7fc44806eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50222/0 (socket says 172.21.15.88:50222) 2023-10-30T14:13:41.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44d64c640 1 -- 172.21.15.88:0/1420825949 learned_addr learned my addr 172.21.15.88:0/1420825949 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-30T14:13:41.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44ce4b640 1 --2- 172.21.15.88:0/1420825949 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4481597d0 0x7fc44815d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:41.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44de4d640 1 --2- 172.21.15.88:0/1420825949 >> v2:172.21.15.57:3300/0 conn(0x7fc44815da70 0x7fc448069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:41.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc43ffff640 1 -- 172.21.15.88:0/1420825949 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222605276 0 0) 0x7fc44815cdc0 con 0x7fc4480c5dc0 2023-10-30T14:13:41.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc43ffff640 1 -- 172.21.15.88:0/1420825949 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc434003530 con 0x7fc4480c5dc0 2023-10-30T14:13:41.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44ce4b640 1 -- 172.21.15.88:0/1420825949 >> v2:172.21.15.57:3300/0 conn(0x7fc44815da70 msgr2=0x7fc448069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:13:41.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44ce4b640 1 --2- 172.21.15.88:0/1420825949 >> v2:172.21.15.57:3300/0 conn(0x7fc44815da70 0x7fc448069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44ce4b640 1 -- 172.21.15.88:0/1420825949 >> v1:172.21.15.88:6789/0 conn(0x7fc4480c5dc0 legacy=0x7fc44806eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44ce4b640 1 -- 172.21.15.88:0/1420825949 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc448159d60 con 0x7fc4481597d0 2023-10-30T14:13:41.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44de4d640 1 --2- 172.21.15.88:0/1420825949 >> v2:172.21.15.57:3300/0 conn(0x7fc44815da70 0x7fc448069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:13:41.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44ce4b640 1 --2- 172.21.15.88:0/1420825949 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4481597d0 0x7fc44815d530 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc4380097c0 tx=0x7fc43801c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b6b0f6535d4c836 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:41.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc43ffff640 1 -- 172.21.15.88:0/1420825949 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4380ad660 con 0x7fc4481597d0 2023-10-30T14:13:41.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc43ffff640 1 -- 172.21.15.88:0/1420825949 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4380ad7c0 con 0x7fc4481597d0 2023-10-30T14:13:41.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc43ffff640 1 -- 172.21.15.88:0/1420825949 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4380adb50 con 0x7fc4481597d0 2023-10-30T14:13:41.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44f8b0640 1 -- 172.21.15.88:0/1420825949 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4481597d0 msgr2=0x7fc44815d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44f8b0640 1 --2- 172.21.15.88:0/1420825949 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4481597d0 0x7fc44815d530 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc4380097c0 tx=0x7fc43801c450 comp rx=0 tx=0).stop 2023-10-30T14:13:41.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44f8b0640 1 -- 172.21.15.88:0/1420825949 shutdown_connections 2023-10-30T14:13:41.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44f8b0640 1 --2- 172.21.15.88:0/1420825949 >> v2:172.21.15.57:3300/0 conn(0x7fc44815da70 0x7fc448069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44f8b0640 1 --2- 172.21.15.88:0/1420825949 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4481597d0 0x7fc44815d530 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44f8b0640 1 -- 172.21.15.88:0/1420825949 >> 172.21.15.88:0/1420825949 conn(0x7fc44806afa0 msgr2=0x7fc44806b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:41.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44f8b0640 1 -- 172.21.15.88:0/1420825949 shutdown_connections 2023-10-30T14:13:41.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.379+0000 7fc44f8b0640 1 -- 172.21.15.88:0/1420825949 wait complete. 2023-10-30T14:13:41.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44f8b0640 1 Processor -- start 2023-10-30T14:13:41.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44f8b0640 1 -- start start 2023-10-30T14:13:41.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44f8b0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4480c5dc0 0x7fc44814b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:41.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44f8b0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4481597d0 0x7fc448157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:41.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44f8b0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc44815da70 0x7fc4481f58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:41.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44f8b0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44814d720 con 0x7fc4481597d0 2023-10-30T14:13:41.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44f8b0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44815cd70 con 0x7fc44815da70 2023-10-30T14:13:41.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44f8b0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc448158f80 con 0x7fc4480c5dc0 2023-10-30T14:13:41.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44d64c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4480c5dc0 0x7fc44814b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:41.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44d64c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4480c5dc0 0x7fc44814b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:38352/0 (socket says 172.21.15.88:38352) 2023-10-30T14:13:41.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44de4d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc44815da70 0x7fc4481f58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:41.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44d64c640 1 -- 172.21.15.88:0/3100057832 learned_addr learned my addr 172.21.15.88:0/3100057832 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-30T14:13:41.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44ce4b640 1 --2- 172.21.15.88:0/3100057832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4481597d0 0x7fc448157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:41.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44d64c640 1 -- 172.21.15.88:0/3100057832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc44815da70 msgr2=0x7fc4481f58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44d64c640 1 --2- 172.21.15.88:0/3100057832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc44815da70 0x7fc4481f58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44d64c640 1 -- 172.21.15.88:0/3100057832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4481597d0 msgr2=0x7fc448157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44d64c640 1 --2- 172.21.15.88:0/3100057832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4481597d0 0x7fc448157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44d64c640 1 -- 172.21.15.88:0/3100057832 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc438009420 con 0x7fc4480c5dc0 2023-10-30T14:13:41.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44d64c640 1 --2- 172.21.15.88:0/3100057832 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4480c5dc0 0x7fc44814b740 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc44400c480 tx=0x7fc44400c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:41.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc43dffb640 1 -- 172.21.15.88:0/3100057832 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444005cf0 con 0x7fc4480c5dc0 2023-10-30T14:13:41.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc43dffb640 1 -- 172.21.15.88:0/3100057832 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc444005e50 con 0x7fc4480c5dc0 2023-10-30T14:13:41.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44f8b0640 1 -- 172.21.15.88:0/3100057832 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4481f5df0 con 0x7fc4480c5dc0 2023-10-30T14:13:41.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc43dffb640 1 -- 172.21.15.88:0/3100057832 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444007270 con 0x7fc4480c5dc0 2023-10-30T14:13:41.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44f8b0640 1 -- 172.21.15.88:0/3100057832 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44814d440 con 0x7fc4480c5dc0 2023-10-30T14:13:41.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc43dffb640 1 -- 172.21.15.88:0/3100057832 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc44401c440 con 0x7fc4480c5dc0 2023-10-30T14:13:41.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc43dffb640 1 -- 172.21.15.88:0/3100057832 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fc444012910 con 0x7fc4480c5dc0 2023-10-30T14:13:41.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.383+0000 7fc44f8b0640 1 -- 172.21.15.88:0/3100057832 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc448067540 con 0x7fc4480c5dc0 2023-10-30T14:13:41.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.387+0000 7fc43dffb640 1 -- 172.21.15.88:0/3100057832 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fc444021060 con 0x7fc4480c5dc0 2023-10-30T14:13:41.395 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:41.387+0000 7f49e8105180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T14:13:41.445 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:41.443+0000 7f49e8105180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T14:13:41.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.455+0000 7fc44f8b0640 1 -- 172.21.15.88:0/3100057832 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1332a9e0-38b6-4d83-b2d5-f0cd2306e983", "id": 4} v 0) v1 -- 0x7fc44814b1f0 con 0x7fc4480c5dc0 2023-10-30T14:13:41.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.459+0000 7fc43dffb640 1 -- 172.21.15.88:0/3100057832 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1332a9e0-38b6-4d83-b2d5-f0cd2306e983", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc444017070 con 0x7fc4480c5dc0 2023-10-30T14:13:41.463 INFO:teuthology.orchestra.run.smithi088.stdout:4 2023-10-30T14:13:41.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.459+0000 7fc44f8b0640 1 -- 172.21.15.88:0/3100057832 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4480c5dc0 msgr2=0x7fc44814b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.459+0000 7fc44f8b0640 1 --2- 172.21.15.88:0/3100057832 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4480c5dc0 0x7fc44814b740 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc44400c480 tx=0x7fc44400c940 comp rx=0 tx=0).stop 2023-10-30T14:13:41.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.459+0000 7fc44f8b0640 1 -- 172.21.15.88:0/3100057832 shutdown_connections 2023-10-30T14:13:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.459+0000 7fc44f8b0640 1 --2- 172.21.15.88:0/3100057832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc44815da70 0x7fc4481f58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.459+0000 7fc44f8b0640 1 --2- 172.21.15.88:0/3100057832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4481597d0 0x7fc448157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.459+0000 7fc44f8b0640 1 --2- 172.21.15.88:0/3100057832 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4480c5dc0 0x7fc44814b740 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.459+0000 7fc44f8b0640 1 -- 172.21.15.88:0/3100057832 >> 172.21.15.88:0/3100057832 conn(0x7fc44806afa0 msgr2=0x7fc448071690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:41.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.459+0000 7fc44f8b0640 1 -- 172.21.15.88:0/3100057832 shutdown_connections 2023-10-30T14:13:41.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.459+0000 7fc44f8b0640 1 -- 172.21.15.88:0/3100057832 wait complete. 2023-10-30T14:13:41.480 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new d826e2f9-75cc-46db-b03c-03146644128a 5 2023-10-30T14:13:41.500 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:41.495+0000 7f49e8105180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T14:13:41.539 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:41.537+0000 7fb5e302d180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T14:13:41.580 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:41.575+0000 7f49e8105180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T14:13:41.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.587+0000 7f530a2cf640 1 Processor -- start 2023-10-30T14:13:41.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.587+0000 7f530a2cf640 1 -- start start 2023-10-30T14:13:41.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f530a2cf640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5304151a00 0x7f530415ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:41.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f530a2cf640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53040c7940 0x7f53040ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:41.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f530a2cf640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530415bcc0 con 0x7f53040cd460 2023-10-30T14:13:41.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f530a2cf640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53040c3e60 con 0x7f5304151a00 2023-10-30T14:13:41.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f530a2cf640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53040c23c0 con 0x7f53040c7940 2023-10-30T14:13:41.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f5303fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5304151a00 0x7f530415ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:41.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f52fbfff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53040c7940 0x7f53040ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:41.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f5303fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5304151a00 0x7f530415ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41202/0 (socket says 172.21.15.88:41202) 2023-10-30T14:13:41.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f5303fff640 1 -- 172.21.15.88:0/2865580584 learned_addr learned my addr 172.21.15.88:0/2865580584 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-30T14:13:41.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f52fbfff640 1 -- 172.21.15.88:0/2865580584 >> v2:172.21.15.88:3300/0 conn(0x7f5304151a00 msgr2=0x7f530415ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:13:41.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f52fbfff640 1 --2- 172.21.15.88:0/2865580584 >> v2:172.21.15.88:3300/0 conn(0x7f5304151a00 0x7f530415ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f52fbfff640 1 -- 172.21.15.88:0/2865580584 >> v1:172.21.15.57:6789/0 conn(0x7f53040cd460 legacy=0x7f53040c2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f52fbfff640 1 -- 172.21.15.88:0/2865580584 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530406f9f0 con 0x7f53040c7940 2023-10-30T14:13:41.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f5303fff640 1 --2- 172.21.15.88:0/2865580584 >> v2:172.21.15.88:3300/0 conn(0x7f5304151a00 0x7f530415ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:13:41.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f52fbfff640 1 --2- 172.21.15.88:0/2865580584 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53040c7940 0x7f53040ccf20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f52ec0097c0 tx=0x7f52ec01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd77085cadb3d803 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:41.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f53037fe640 1 -- 172.21.15.88:0/2865580584 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec0ad660 con 0x7f53040c7940 2023-10-30T14:13:41.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f53037fe640 1 -- 172.21.15.88:0/2865580584 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ec0ad7c0 con 0x7f53040c7940 2023-10-30T14:13:41.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f53037fe640 1 -- 172.21.15.88:0/2865580584 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec0adb50 con 0x7f53040c7940 2023-10-30T14:13:41.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f530a2cf640 1 -- 172.21.15.88:0/2865580584 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53040c7940 msgr2=0x7f53040ccf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f530a2cf640 1 --2- 172.21.15.88:0/2865580584 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53040c7940 0x7f53040ccf20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f52ec0097c0 tx=0x7f52ec01c450 comp rx=0 tx=0).stop 2023-10-30T14:13:41.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f530a2cf640 1 -- 172.21.15.88:0/2865580584 shutdown_connections 2023-10-30T14:13:41.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f530a2cf640 1 --2- 172.21.15.88:0/2865580584 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53040c7940 0x7f53040ccf20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f530a2cf640 1 --2- 172.21.15.88:0/2865580584 >> v2:172.21.15.88:3300/0 conn(0x7f5304151a00 0x7f530415ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.591+0000 7f530a2cf640 1 -- 172.21.15.88:0/2865580584 >> 172.21.15.88:0/2865580584 conn(0x7f5304072010 msgr2=0x7f5304072420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:41.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f530a2cf640 1 -- 172.21.15.88:0/2865580584 shutdown_connections 2023-10-30T14:13:41.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f530a2cf640 1 -- 172.21.15.88:0/2865580584 wait complete. 2023-10-30T14:13:41.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f530a2cf640 1 Processor -- start 2023-10-30T14:13:41.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f530a2cf640 1 -- start start 2023-10-30T14:13:41.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f530a2cf640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53040c7940 0x7f5304148d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:41.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f530a2cf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53040cd460 0x7f53040c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:41.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f530a2cf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5304151a00 0x7f53040c8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:41.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f530a2cf640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53040c6e00 con 0x7f53040cd460 2023-10-30T14:13:41.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f530a2cf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53040c92b0 con 0x7f5304151a00 2023-10-30T14:13:41.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f530a2cf640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53040c9b90 con 0x7f53040c7940 2023-10-30T14:13:41.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f5303fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53040c7940 0x7f5304148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:41.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f5303fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53040c7940 0x7f5304148d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:38364/0 (socket says 172.21.15.88:38364) 2023-10-30T14:13:41.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f5303fff640 1 -- 172.21.15.88:0/343413182 learned_addr learned my addr 172.21.15.88:0/343413182 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-30T14:13:41.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f530886c640 1 --2- 172.21.15.88:0/343413182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5304151a00 0x7f53040c8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:41.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f52fbfff640 1 --2- 172.21.15.88:0/343413182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53040cd460 0x7f53040c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:41.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f5303fff640 1 -- 172.21.15.88:0/343413182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5304151a00 msgr2=0x7f53040c8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f5303fff640 1 --2- 172.21.15.88:0/343413182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5304151a00 0x7f53040c8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f5303fff640 1 -- 172.21.15.88:0/343413182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53040cd460 msgr2=0x7f53040c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f5303fff640 1 --2- 172.21.15.88:0/343413182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53040cd460 0x7f53040c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f5303fff640 1 -- 172.21.15.88:0/343413182 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f40094d0 con 0x7f53040c7940 2023-10-30T14:13:41.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f530886c640 1 --2- 172.21.15.88:0/343413182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5304151a00 0x7f53040c8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:13:41.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f52fbfff640 1 --2- 172.21.15.88:0/343413182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53040cd460 0x7f53040c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:13:41.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f5303fff640 1 --2- 172.21.15.88:0/343413182 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53040c7940 0x7f5304148d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f52f400e930 tx=0x7f52f400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:41.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f53017fa640 1 -- 172.21.15.88:0/343413182 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f400ca00 con 0x7f53040c7940 2023-10-30T14:13:41.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f53017fa640 1 -- 172.21.15.88:0/343413182 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f400cb60 con 0x7f53040c7940 2023-10-30T14:13:41.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f53017fa640 1 -- 172.21.15.88:0/343413182 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f4013470 con 0x7f53040c7940 2023-10-30T14:13:41.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f530a2cf640 1 -- 172.21.15.88:0/343413182 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ec009420 con 0x7f53040c7940 2023-10-30T14:13:41.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f530a2cf640 1 -- 172.21.15.88:0/343413182 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53040c65f0 con 0x7f53040c7940 2023-10-30T14:13:41.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f53017fa640 1 -- 172.21.15.88:0/343413182 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f52f40138d0 con 0x7f53040c7940 2023-10-30T14:13:41.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f53017fa640 1 -- 172.21.15.88:0/343413182 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f52f401ce30 con 0x7f53040c7940 2023-10-30T14:13:41.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.595+0000 7f530a2cf640 1 -- 172.21.15.88:0/343413182 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5304067540 con 0x7f53040c7940 2023-10-30T14:13:41.607 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:41.601+0000 7fb5e302d180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T14:13:41.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.599+0000 7f53017fa640 1 -- 172.21.15.88:0/343413182 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f52f4021060 con 0x7f53040c7940 2023-10-30T14:13:41.635 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:41.631+0000 7f49e8105180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T14:13:41.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.663+0000 7f530a2cf640 1 -- 172.21.15.88:0/343413182 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d826e2f9-75cc-46db-b03c-03146644128a", "id": 5} v 0) v1 -- 0x7f5304160280 con 0x7f53040c7940 2023-10-30T14:13:41.667 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:41.665+0000 7fb5e302d180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T14:13:41.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.667+0000 7f53017fa640 1 -- 172.21.15.88:0/343413182 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d826e2f9-75cc-46db-b03c-03146644128a", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f52f402b0b0 con 0x7f53040c7940 2023-10-30T14:13:41.672 INFO:teuthology.orchestra.run.smithi088.stdout:5 2023-10-30T14:13:41.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.671+0000 7f530a2cf640 1 -- 172.21.15.88:0/343413182 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53040c7940 msgr2=0x7f5304148d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.671+0000 7f530a2cf640 1 --2- 172.21.15.88:0/343413182 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53040c7940 0x7f5304148d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f52f400e930 tx=0x7f52f400edf0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.671+0000 7f530a2cf640 1 -- 172.21.15.88:0/343413182 shutdown_connections 2023-10-30T14:13:41.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.671+0000 7f530a2cf640 1 --2- 172.21.15.88:0/343413182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5304151a00 0x7f53040c8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.671+0000 7f530a2cf640 1 --2- 172.21.15.88:0/343413182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53040cd460 0x7f53040c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.671+0000 7f530a2cf640 1 --2- 172.21.15.88:0/343413182 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53040c7940 0x7f5304148d30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.671+0000 7f530a2cf640 1 -- 172.21.15.88:0/343413182 >> 172.21.15.88:0/343413182 conn(0x7f5304072010 msgr2=0x7f53040c4890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:41.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.671+0000 7f530a2cf640 1 -- 172.21.15.88:0/343413182 shutdown_connections 2023-10-30T14:13:41.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.671+0000 7f530a2cf640 1 -- 172.21.15.88:0/343413182 wait complete. 2023-10-30T14:13:41.688 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new fe6fce6c-0b9a-4c80-b21c-4c17db861adb 6 2023-10-30T14:13:41.715 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:41.711+0000 7f49e8105180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T14:13:41.762 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:41.761+0000 7fb5e302d180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T14:13:41.768 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:41.763+0000 7f49e8105180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T14:13:41.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.819+0000 7ffa46cf9640 1 Processor -- start 2023-10-30T14:13:41.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.819+0000 7ffa46cf9640 1 -- start start 2023-10-30T14:13:41.825 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:41.821+0000 7fb5e302d180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T14:13:41.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.823+0000 7ffa46cf9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa4015a0a0 0x7ffa400c2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:41.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.823+0000 7ffa46cf9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa400c6ed0 con 0x7ffa40153b30 2023-10-30T14:13:41.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.823+0000 7ffa46cf9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa400c67a0 con 0x7ffa40153f10 2023-10-30T14:13:41.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.823+0000 7ffa46cf9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa400c6a40 con 0x7ffa4015a0a0 2023-10-30T14:13:41.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.823+0000 7ffa45296640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa4015a0a0 0x7ffa400c2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:41.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.823+0000 7ffa45296640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa4015a0a0 0x7ffa400c2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:38370/0 (socket says 172.21.15.88:38370) 2023-10-30T14:13:41.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.823+0000 7ffa3ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ffa40153f10 0x7ffa40159990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50238/0 (socket says 172.21.15.88:50238) 2023-10-30T14:13:41.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.823+0000 7ffa45296640 1 -- 172.21.15.88:0/3834048446 learned_addr learned my addr 172.21.15.88:0/3834048446 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-30T14:13:41.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.823+0000 7ffa45296640 1 -- 172.21.15.88:0/3834048446 >> v1:172.21.15.88:6789/0 conn(0x7ffa40153f10 legacy=0x7ffa40159990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.823+0000 7ffa3f7fe640 1 -- 172.21.15.88:0/3834048446 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2664734857 0 0) 0x7ffa400c67a0 con 0x7ffa40153f10 2023-10-30T14:13:41.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.823+0000 7ffa45296640 1 -- 172.21.15.88:0/3834048446 >> v1:172.21.15.57:6789/0 conn(0x7ffa40153b30 legacy=0x7ffa4006cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.823+0000 7ffa45296640 1 -- 172.21.15.88:0/3834048446 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa400c7400 con 0x7ffa4015a0a0 2023-10-30T14:13:41.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.823+0000 7ffa45296640 1 --2- 172.21.15.88:0/3834048446 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa4015a0a0 0x7ffa400c2bb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009e80 tx=0x7ffa3801ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=354bd13e33fe3189 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:41.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.823+0000 7ffa3f7fe640 1 -- 172.21.15.88:0/3834048446 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa380a57f0 con 0x7ffa4015a0a0 2023-10-30T14:13:41.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.823+0000 7ffa3f7fe640 1 -- 172.21.15.88:0/3834048446 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa380a5950 con 0x7ffa4015a0a0 2023-10-30T14:13:41.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.823+0000 7ffa3f7fe640 1 -- 172.21.15.88:0/3834048446 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa380a5ce0 con 0x7ffa4015a0a0 2023-10-30T14:13:41.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa46cf9640 1 -- 172.21.15.88:0/3834048446 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa4015a0a0 msgr2=0x7ffa400c2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa46cf9640 1 --2- 172.21.15.88:0/3834048446 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa4015a0a0 0x7ffa400c2bb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009e80 tx=0x7ffa3801ca30 comp rx=0 tx=0).stop 2023-10-30T14:13:41.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa46cf9640 1 -- 172.21.15.88:0/3834048446 shutdown_connections 2023-10-30T14:13:41.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa46cf9640 1 --2- 172.21.15.88:0/3834048446 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa4015a0a0 0x7ffa400c2bb0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa46cf9640 1 -- 172.21.15.88:0/3834048446 >> 172.21.15.88:0/3834048446 conn(0x7ffa4006f1b0 msgr2=0x7ffa400cbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:41.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa46cf9640 1 -- 172.21.15.88:0/3834048446 shutdown_connections 2023-10-30T14:13:41.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa46cf9640 1 -- 172.21.15.88:0/3834048446 wait complete. 2023-10-30T14:13:41.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa46cf9640 1 Processor -- start 2023-10-30T14:13:41.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa46cf9640 1 -- start start 2023-10-30T14:13:41.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa46cf9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa40153b30 0x7ffa40146060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:41.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa46cf9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa40153f10 0x7ffa401465a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:41.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa46cf9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffa4015a0a0 0x7ffa401f5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:41.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa46cf9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa401422a0 con 0x7ffa40153b30 2023-10-30T14:13:41.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa46cf9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4014eff0 con 0x7ffa4015a0a0 2023-10-30T14:13:41.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa46cf9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa40158140 con 0x7ffa40153f10 2023-10-30T14:13:41.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa3ffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa40153f10 0x7ffa401465a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:41.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa45296640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffa4015a0a0 0x7ffa401f5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:41.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa3ffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa40153f10 0x7ffa401465a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:38382/0 (socket says 172.21.15.88:38382) 2023-10-30T14:13:41.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa3ffff640 1 -- 172.21.15.88:0/3806891394 learned_addr learned my addr 172.21.15.88:0/3806891394 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-30T14:13:41.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa44a95640 1 --2- 172.21.15.88:0/3806891394 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa40153b30 0x7ffa40146060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:41.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa3ffff640 1 -- 172.21.15.88:0/3806891394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffa4015a0a0 msgr2=0x7ffa401f5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa3ffff640 1 --2- 172.21.15.88:0/3806891394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffa4015a0a0 0x7ffa401f5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa3ffff640 1 -- 172.21.15.88:0/3806891394 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa40153b30 msgr2=0x7ffa40146060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa3ffff640 1 --2- 172.21.15.88:0/3806891394 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa40153b30 0x7ffa40146060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa3ffff640 1 -- 172.21.15.88:0/3806891394 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa38009b30 con 0x7ffa40153f10 2023-10-30T14:13:41.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa44a95640 1 --2- 172.21.15.88:0/3806891394 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa40153b30 0x7ffa40146060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:13:41.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa45296640 1 --2- 172.21.15.88:0/3806891394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffa4015a0a0 0x7ffa401f5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:13:41.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa3ffff640 1 --2- 172.21.15.88:0/3806891394 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa40153f10 0x7ffa401465a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c00c4d0 tx=0x7ffa2c00c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:41.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa3d7fa640 1 -- 172.21.15.88:0/3806891394 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c0074e0 con 0x7ffa40153f10 2023-10-30T14:13:41.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa3d7fa640 1 -- 172.21.15.88:0/3806891394 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2c007640 con 0x7ffa40153f10 2023-10-30T14:13:41.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa46cf9640 1 -- 172.21.15.88:0/3806891394 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa40157e60 con 0x7ffa40153f10 2023-10-30T14:13:41.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa3d7fa640 1 -- 172.21.15.88:0/3806891394 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c004a30 con 0x7ffa40153f10 2023-10-30T14:13:41.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa46cf9640 1 -- 172.21.15.88:0/3806891394 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4014d5d0 con 0x7ffa40153f10 2023-10-30T14:13:41.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa3d7fa640 1 -- 172.21.15.88:0/3806891394 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ffa2c01d440 con 0x7ffa40153f10 2023-10-30T14:13:41.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa3d7fa640 1 -- 172.21.15.88:0/3806891394 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7ffa2c0129f0 con 0x7ffa40153f10 2023-10-30T14:13:41.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.827+0000 7ffa46cf9640 1 -- 172.21.15.88:0/3806891394 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4004e170 con 0x7ffa40153f10 2023-10-30T14:13:41.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.831+0000 7ffa3d7fa640 1 -- 172.21.15.88:0/3806891394 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7ffa2c018070 con 0x7ffa40153f10 2023-10-30T14:13:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.895+0000 7ffa46cf9640 1 -- 172.21.15.88:0/3806891394 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd new", "uuid": "fe6fce6c-0b9a-4c80-b21c-4c17db861adb", "id": 6} v 0) v1 -- 0x7ffa400caeb0 con 0x7ffa40153f10 2023-10-30T14:13:41.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.899+0000 7ffa3d7fa640 1 -- 172.21.15.88:0/3806891394 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fe6fce6c-0b9a-4c80-b21c-4c17db861adb", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7ffa2c012400 con 0x7ffa40153f10 2023-10-30T14:13:41.905 INFO:teuthology.orchestra.run.smithi088.stdout:6 2023-10-30T14:13:41.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.903+0000 7ffa46cf9640 1 -- 172.21.15.88:0/3806891394 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa40153f10 msgr2=0x7ffa401465a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:41.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.903+0000 7ffa46cf9640 1 --2- 172.21.15.88:0/3806891394 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa40153f10 0x7ffa401465a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c00c4d0 tx=0x7ffa2c00c990 comp rx=0 tx=0).stop 2023-10-30T14:13:41.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.903+0000 7ffa46cf9640 1 -- 172.21.15.88:0/3806891394 shutdown_connections 2023-10-30T14:13:41.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.903+0000 7ffa46cf9640 1 --2- 172.21.15.88:0/3806891394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffa4015a0a0 0x7ffa401f5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.903+0000 7ffa46cf9640 1 --2- 172.21.15.88:0/3806891394 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa40153f10 0x7ffa401465a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.903+0000 7ffa46cf9640 1 --2- 172.21.15.88:0/3806891394 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa40153b30 0x7ffa40146060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:41.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.903+0000 7ffa46cf9640 1 -- 172.21.15.88:0/3806891394 >> 172.21.15.88:0/3806891394 conn(0x7ffa4006f1b0 msgr2=0x7ffa4006ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:41.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.903+0000 7ffa46cf9640 1 -- 172.21.15.88:0/3806891394 shutdown_connections 2023-10-30T14:13:41.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:41.903+0000 7ffa46cf9640 1 -- 172.21.15.88:0/3806891394 wait complete. 2023-10-30T14:13:41.916 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:41.911+0000 7f49e8105180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T14:13:41.919 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:41.917+0000 7fb5e302d180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T14:13:41.920 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new 82de08f0-87e3-43fc-8fdf-79409a22ed2d 7 2023-10-30T14:13:41.977 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:41.975+0000 7f49e8105180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T14:13:41.981 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:41.977+0000 7fb5e302d180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T14:13:42.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.031+0000 7facc4583640 1 Processor -- start 2023-10-30T14:13:42.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.031+0000 7facc4583640 1 -- start start 2023-10-30T14:13:42.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.031+0000 7facc4583640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7facbc0c8150 0x7facbc157040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:42.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.031+0000 7facc4583640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7facbc0c8530 0x7facbc0c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:42.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.031+0000 7facc4583640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facbc06f6b0 con 0x7facbc0ca210 2023-10-30T14:13:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.031+0000 7facc4583640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facbc06e6a0 con 0x7facbc0c8150 2023-10-30T14:13:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.031+0000 7facc4583640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facbc157580 con 0x7facbc0c8530 2023-10-30T14:13:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.031+0000 7facc2b20640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7facbc0c8530 0x7facbc0c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.035+0000 7facc2b20640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7facbc0c8530 0x7facbc0c8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:38396/0 (socket says 172.21.15.88:38396) 2023-10-30T14:13:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.035+0000 7facc2b20640 1 -- 172.21.15.88:0/389895794 learned_addr learned my addr 172.21.15.88:0/389895794 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-30T14:13:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.035+0000 7facc231f640 1 --2- 172.21.15.88:0/389895794 >> v2:172.21.15.57:3300/0 conn(0x7facbc0c8150 0x7facbc157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.035+0000 7facc2b20640 1 -- 172.21.15.88:0/389895794 >> v2:172.21.15.57:3300/0 conn(0x7facbc0c8150 msgr2=0x7facbc157040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:13:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.035+0000 7facc2b20640 1 --2- 172.21.15.88:0/389895794 >> v2:172.21.15.57:3300/0 conn(0x7facbc0c8150 0x7facbc157040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.035+0000 7facc2b20640 1 -- 172.21.15.88:0/389895794 >> v1:172.21.15.57:6789/0 conn(0x7facbc0ca210 legacy=0x7facbc0c4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:13:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.035+0000 7facc2b20640 1 -- 172.21.15.88:0/389895794 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc0ca720 con 0x7facbc0c8530 2023-10-30T14:13:42.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.035+0000 7facc2b20640 1 --2- 172.21.15.88:0/389895794 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7facbc0c8530 0x7facbc0c8980 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7facb400d150 tx=0x7facb40213c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da000d0d8ab51e81 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:42.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.035+0000 7facc131d640 1 -- 172.21.15.88:0/389895794 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb400b2d0 con 0x7facbc0c8530 2023-10-30T14:13:42.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.035+0000 7facc131d640 1 -- 172.21.15.88:0/389895794 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb40ad050 con 0x7facbc0c8530 2023-10-30T14:13:42.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.035+0000 7facc131d640 1 -- 172.21.15.88:0/389895794 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb4009820 con 0x7facbc0c8530 2023-10-30T14:13:42.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc4583640 1 -- 172.21.15.88:0/389895794 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7facbc0c8530 msgr2=0x7facbc0c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:42.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc4583640 1 --2- 172.21.15.88:0/389895794 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7facbc0c8530 0x7facbc0c8980 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7facb400d150 tx=0x7facb40213c0 comp rx=0 tx=0).stop 2023-10-30T14:13:42.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc4583640 1 -- 172.21.15.88:0/389895794 shutdown_connections 2023-10-30T14:13:42.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc4583640 1 --2- 172.21.15.88:0/389895794 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7facbc0c8530 0x7facbc0c8980 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:42.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc4583640 1 --2- 172.21.15.88:0/389895794 >> v2:172.21.15.57:3300/0 conn(0x7facbc0c8150 0x7facbc157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc4583640 1 -- 172.21.15.88:0/389895794 >> 172.21.15.88:0/389895794 conn(0x7facbc06afa0 msgr2=0x7facbc06b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc4583640 1 -- 172.21.15.88:0/389895794 shutdown_connections 2023-10-30T14:13:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc4583640 1 -- 172.21.15.88:0/389895794 wait complete. 2023-10-30T14:13:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc4583640 1 Processor -- start 2023-10-30T14:13:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc4583640 1 -- start start 2023-10-30T14:13:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc4583640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facbc0c8150 0x7facbc1f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc4583640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facbc0c8530 0x7facbc154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc4583640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7facbc0ca210 0x7facbc151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc4583640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc1f3270 con 0x7facbc0c8150 2023-10-30T14:13:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc4583640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc152470 con 0x7facbc0c8530 2023-10-30T14:13:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc4583640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facbc154980 con 0x7facbc0ca210 2023-10-30T14:13:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc1b1e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facbc0c8530 0x7facbc154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc2b20640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7facbc0ca210 0x7facbc151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc1b1e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facbc0c8530 0x7facbc154440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41220/0 (socket says 172.21.15.88:41220) 2023-10-30T14:13:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc1b1e640 1 -- 172.21.15.88:0/1152442077 learned_addr learned my addr 172.21.15.88:0/1152442077 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-30T14:13:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc2b20640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7facbc0ca210 0x7facbc151680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:38404/0 (socket says 172.21.15.88:38404) 2023-10-30T14:13:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc231f640 1 --2- 172.21.15.88:0/1152442077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facbc0c8150 0x7facbc1f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc1b1e640 1 -- 172.21.15.88:0/1152442077 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7facbc0ca210 msgr2=0x7facbc151680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc1b1e640 1 --2- 172.21.15.88:0/1152442077 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7facbc0ca210 0x7facbc151680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc1b1e640 1 -- 172.21.15.88:0/1152442077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facbc0c8150 msgr2=0x7facbc1f2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc1b1e640 1 --2- 172.21.15.88:0/1152442077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facbc0c8150 0x7facbc1f2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc1b1e640 1 -- 172.21.15.88:0/1152442077 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb400bd40 con 0x7facbc0c8530 2023-10-30T14:13:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc2b20640 1 --2- 172.21.15.88:0/1152442077 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7facbc0ca210 0x7facbc151680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:13:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc1b1e640 1 --2- 172.21.15.88:0/1152442077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facbc0c8530 0x7facbc154440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7facac00c450 tx=0x7facac00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facb2ffd640 1 -- 172.21.15.88:0/1152442077 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac012830 con 0x7facbc0c8530 2023-10-30T14:13:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facb2ffd640 1 -- 172.21.15.88:0/1152442077 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facac0129b0 con 0x7facbc0c8530 2023-10-30T14:13:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc4583640 1 -- 172.21.15.88:0/1152442077 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facbc0c6bb0 con 0x7facbc0c8530 2023-10-30T14:13:42.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facb2ffd640 1 -- 172.21.15.88:0/1152442077 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac012d60 con 0x7facbc0c8530 2023-10-30T14:13:42.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facc4583640 1 -- 172.21.15.88:0/1152442077 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facbc1526c0 con 0x7facbc0c8530 2023-10-30T14:13:42.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facb2ffd640 1 -- 172.21.15.88:0/1152442077 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7facac01c760 con 0x7facbc0c8530 2023-10-30T14:13:42.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.039+0000 7facb2ffd640 1 -- 172.21.15.88:0/1152442077 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7facac01bde0 con 0x7facbc0c8530 2023-10-30T14:13:42.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.043+0000 7facc4583640 1 -- 172.21.15.88:0/1152442077 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facbc072d40 con 0x7facbc0c8530 2023-10-30T14:13:42.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.043+0000 7facb2ffd640 1 -- 172.21.15.88:0/1152442077 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7facac01b780 con 0x7facbc0c8530 2023-10-30T14:13:42.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.107+0000 7facc4583640 1 -- 172.21.15.88:0/1152442077 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd new", "uuid": "82de08f0-87e3-43fc-8fdf-79409a22ed2d", "id": 7} v 0) v1 -- 0x7facbc067540 con 0x7facbc0c8530 2023-10-30T14:13:42.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.115+0000 7facb2ffd640 1 -- 172.21.15.88:0/1152442077 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "82de08f0-87e3-43fc-8fdf-79409a22ed2d", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7facac01b920 con 0x7facbc0c8530 2023-10-30T14:13:42.120 INFO:teuthology.orchestra.run.smithi088.stdout:7 2023-10-30T14:13:42.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.115+0000 7facc4583640 1 -- 172.21.15.88:0/1152442077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facbc0c8530 msgr2=0x7facbc154440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:42.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.115+0000 7facc4583640 1 --2- 172.21.15.88:0/1152442077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facbc0c8530 0x7facbc154440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7facac00c450 tx=0x7facac00c910 comp rx=0 tx=0).stop 2023-10-30T14:13:42.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.115+0000 7facc4583640 1 -- 172.21.15.88:0/1152442077 shutdown_connections 2023-10-30T14:13:42.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.115+0000 7facc4583640 1 --2- 172.21.15.88:0/1152442077 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7facbc0ca210 0x7facbc151680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:42.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.115+0000 7facc4583640 1 --2- 172.21.15.88:0/1152442077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facbc0c8530 0x7facbc154440 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:42.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.115+0000 7facc4583640 1 --2- 172.21.15.88:0/1152442077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facbc0c8150 0x7facbc1f2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:42.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.115+0000 7facc4583640 1 -- 172.21.15.88:0/1152442077 >> 172.21.15.88:0/1152442077 conn(0x7facbc06afa0 msgr2=0x7facbc0c9060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:42.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.115+0000 7facc4583640 1 -- 172.21.15.88:0/1152442077 shutdown_connections 2023-10-30T14:13:42.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-30T14:13:42.115+0000 7facc4583640 1 -- 172.21.15.88:0/1152442077 wait complete. 2023-10-30T14:13:42.135 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-30T14:13:42.135 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-30T14:13:42.140 INFO:tasks.ceph.osd.0:Started 2023-10-30T14:13:42.140 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-30T14:13:42.141 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-30T14:13:42.146 INFO:tasks.ceph.osd.1:Started 2023-10-30T14:13:42.146 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-30T14:13:42.146 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-30T14:13:42.152 INFO:tasks.ceph.osd.2:Started 2023-10-30T14:13:42.152 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-30T14:13:42.152 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-30T14:13:42.153 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:42.145+0000 7fb5e302d180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T14:13:42.157 INFO:tasks.ceph.osd.3:Started 2023-10-30T14:13:42.158 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-30T14:13:42.158 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-30T14:13:42.160 INFO:tasks.ceph.osd.4:Started 2023-10-30T14:13:42.161 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-30T14:13:42.161 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-30T14:13:42.163 INFO:tasks.ceph.osd.5:Started 2023-10-30T14:13:42.165 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-30T14:13:42.165 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-30T14:13:42.168 INFO:tasks.ceph.osd.6:Started 2023-10-30T14:13:42.168 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-30T14:13:42.168 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-30T14:13:42.170 INFO:tasks.ceph.osd.7:Started 2023-10-30T14:13:42.171 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:13:42.272 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:42.269+0000 7fb5e302d180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T14:13:42.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.325+0000 7fe187965640 1 Processor -- start 2023-10-30T14:13:42.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.325+0000 7fe187965640 1 -- start start 2023-10-30T14:13:42.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.325+0000 7fe187965640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe180153c80 0x7fe18015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:42.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.325+0000 7fe187965640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1800c42a0 0x7fe1800ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:42.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.325+0000 7fe187965640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe180159f20 con 0x7fe1800c4a10 2023-10-30T14:13:42.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.325+0000 7fe187965640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe18015dec0 con 0x7fe180153c80 2023-10-30T14:13:42.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.325+0000 7fe187965640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1801599a0 con 0x7fe1800c42a0 2023-10-30T14:13:42.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.325+0000 7fe185701640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe180153c80 0x7fe18015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:42.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.325+0000 7fe184f00640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe1800c4a10 0x7fe18015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:33484/0 (socket says 172.21.15.57:33484) 2023-10-30T14:13:42.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.325+0000 7fe184f00640 1 -- 172.21.15.57:0/2164939768 learned_addr learned my addr 172.21.15.57:0/2164939768 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:13:42.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.325+0000 7fe185f02640 1 --2- 172.21.15.57:0/2164939768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1800c42a0 0x7fe1800ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:42.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.325+0000 7fe185701640 1 -- 172.21.15.57:0/2164939768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1800c42a0 msgr2=0x7fe1800ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:13:42.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.325+0000 7fe185701640 1 --2- 172.21.15.57:0/2164939768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1800c42a0 0x7fe1800ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:42.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.325+0000 7fe177fff640 1 -- 172.21.15.57:0/2164939768 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786837326 0 0) 0x7fe180159f20 con 0x7fe1800c4a10 2023-10-30T14:13:42.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.325+0000 7fe185701640 1 -- 172.21.15.57:0/2164939768 >> v1:172.21.15.88:6789/0 conn(0x7fe1800c4a10 legacy=0x7fe18015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:42.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.325+0000 7fe185701640 1 -- 172.21.15.57:0/2164939768 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18015d620 con 0x7fe180153c80 2023-10-30T14:13:42.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.329+0000 7fe185701640 1 --2- 172.21.15.57:0/2164939768 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe180153c80 0x7fe18015cf90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe17c0097c0 tx=0x7fe17c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e65ef24d2d8ea1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:42.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.333+0000 7fe177fff640 1 -- 172.21.15.57:0/2164939768 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c0a8970 con 0x7fe180153c80 2023-10-30T14:13:42.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.333+0000 7fe177fff640 1 -- 172.21.15.57:0/2164939768 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe17c0a8ad0 con 0x7fe180153c80 2023-10-30T14:13:42.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.333+0000 7fe177fff640 1 -- 172.21.15.57:0/2164939768 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c0a8e60 con 0x7fe180153c80 2023-10-30T14:13:42.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.333+0000 7fe187965640 1 -- 172.21.15.57:0/2164939768 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe180153c80 msgr2=0x7fe18015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:42.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.333+0000 7fe187965640 1 --2- 172.21.15.57:0/2164939768 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe180153c80 0x7fe18015cf90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe17c0097c0 tx=0x7fe17c01ec70 comp rx=0 tx=0).stop 2023-10-30T14:13:42.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.333+0000 7fe187965640 1 -- 172.21.15.57:0/2164939768 shutdown_connections 2023-10-30T14:13:42.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.333+0000 7fe187965640 1 --2- 172.21.15.57:0/2164939768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1800c42a0 0x7fe1800ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:42.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.333+0000 7fe187965640 1 --2- 172.21.15.57:0/2164939768 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe180153c80 0x7fe18015cf90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:42.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.333+0000 7fe187965640 1 -- 172.21.15.57:0/2164939768 >> 172.21.15.57:0/2164939768 conn(0x7fe1800715c0 msgr2=0x7fe1800719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:42.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.333+0000 7fe187965640 1 -- 172.21.15.57:0/2164939768 shutdown_connections 2023-10-30T14:13:42.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.333+0000 7fe187965640 1 -- 172.21.15.57:0/2164939768 wait complete. 2023-10-30T14:13:42.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.333+0000 7fe187965640 1 Processor -- start 2023-10-30T14:13:42.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.333+0000 7fe187965640 1 -- start start 2023-10-30T14:13:42.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.333+0000 7fe187965640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1800c42a0 0x7fe180154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:42.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.333+0000 7fe187965640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1800c4a10 0x7fe180154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:42.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.333+0000 7fe187965640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe180153c80 0x7fe180155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:42.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.333+0000 7fe187965640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe180156620 con 0x7fe1800c4a10 2023-10-30T14:13:42.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe187965640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1800ceac0 con 0x7fe1800c42a0 2023-10-30T14:13:42.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe187965640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe180154850 con 0x7fe180153c80 2023-10-30T14:13:42.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe184f00640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1800c4a10 0x7fe180154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:42.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe185701640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1800c42a0 0x7fe180154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:42.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe185701640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1800c42a0 0x7fe180154310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:55166/0 (socket says 172.21.15.57:55166) 2023-10-30T14:13:42.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe185701640 1 -- 172.21.15.57:0/353259202 learned_addr learned my addr 172.21.15.57:0/353259202 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:13:42.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe185f02640 1 --2- 172.21.15.57:0/353259202 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe180153c80 0x7fe180155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:42.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe184f00640 1 -- 172.21.15.57:0/353259202 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe180153c80 msgr2=0x7fe180155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:42.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe184f00640 1 --2- 172.21.15.57:0/353259202 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe180153c80 0x7fe180155220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:42.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe184f00640 1 -- 172.21.15.57:0/353259202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1800c42a0 msgr2=0x7fe180154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:42.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe184f00640 1 --2- 172.21.15.57:0/353259202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1800c42a0 0x7fe180154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:42.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe184f00640 1 -- 172.21.15.57:0/353259202 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17c009420 con 0x7fe1800c4a10 2023-10-30T14:13:42.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe184f00640 1 --2- 172.21.15.57:0/353259202 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1800c4a10 0x7fe180154ce0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe17000c430 tx=0x7fe17000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:42.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe175ffb640 1 -- 172.21.15.57:0/353259202 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1700074e0 con 0x7fe1800c4a10 2023-10-30T14:13:42.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe175ffb640 1 -- 172.21.15.57:0/353259202 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe170007660 con 0x7fe1800c4a10 2023-10-30T14:13:42.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe187965640 1 -- 172.21.15.57:0/353259202 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1801f29f0 con 0x7fe1800c4a10 2023-10-30T14:13:42.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe175ffb640 1 -- 172.21.15.57:0/353259202 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe170004a30 con 0x7fe1800c4a10 2023-10-30T14:13:42.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe175ffb640 1 -- 172.21.15.57:0/353259202 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe1700133c0 con 0x7fe1800c4a10 2023-10-30T14:13:42.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe187965640 1 -- 172.21.15.57:0/353259202 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1801f2fa0 con 0x7fe1800c4a10 2023-10-30T14:13:42.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe175ffb640 1 -- 172.21.15.57:0/353259202 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fe1700124a0 con 0x7fe1800c4a10 2023-10-30T14:13:42.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.337+0000 7fe187965640 1 -- 172.21.15.57:0/353259202 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe180159b60 con 0x7fe1800c4a10 2023-10-30T14:13:42.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.341+0000 7fe175ffb640 1 -- 172.21.15.57:0/353259202 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fe170013860 con 0x7fe1800c4a10 2023-10-30T14:13:42.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.397+0000 7fe187965640 1 -- 172.21.15.57:0/353259202 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe18004e170 con 0x7fe1800c4a10 2023-10-30T14:13:42.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.397+0000 7fe175ffb640 1 -- 172.21.15.57:0/353259202 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fe170013a00 con 0x7fe1800c4a10 2023-10-30T14:13:42.402 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:42.402 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":10,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:13:42.112020+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:13:42.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.401+0000 7fe187965640 1 -- 172.21.15.57:0/353259202 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1800c4a10 msgr2=0x7fe180154ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:42.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.401+0000 7fe187965640 1 --2- 172.21.15.57:0/353259202 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1800c4a10 0x7fe180154ce0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe17000c430 tx=0x7fe17000c8f0 comp rx=0 tx=0).stop 2023-10-30T14:13:42.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.401+0000 7fe187965640 1 -- 172.21.15.57:0/353259202 shutdown_connections 2023-10-30T14:13:42.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.401+0000 7fe187965640 1 --2- 172.21.15.57:0/353259202 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe180153c80 0x7fe180155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:42.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.401+0000 7fe187965640 1 --2- 172.21.15.57:0/353259202 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1800c4a10 0x7fe180154ce0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:42.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.401+0000 7fe187965640 1 --2- 172.21.15.57:0/353259202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1800c42a0 0x7fe180154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:42.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.401+0000 7fe187965640 1 -- 172.21.15.57:0/353259202 >> 172.21.15.57:0/353259202 conn(0x7fe1800715c0 msgr2=0x7fe180158450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:42.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.401+0000 7fe187965640 1 -- 172.21.15.57:0/353259202 shutdown_connections 2023-10-30T14:13:42.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:42.401+0000 7fe187965640 1 -- 172.21.15.57:0/353259202 wait complete. 2023-10-30T14:13:42.422 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-30T14:13:42.423 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-30T14:13:42.617 INFO:tasks.ceph.mgr.x.smithi088.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-30T14:13:42.617 INFO:tasks.ceph.mgr.x.smithi088.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-30T14:13:42.617 INFO:tasks.ceph.mgr.x.smithi088.stderr: from numpy import show_config as show_numpy_config 2023-10-30T14:13:42.666 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:42.663+0000 7f49e8105180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T14:13:42.778 INFO:tasks.ceph.osd.5.smithi088.stderr:2023-10-30T14:13:42.775+0000 7f667846e640 -1 Falling back to public interface 2023-10-30T14:13:42.778 INFO:tasks.ceph.osd.4.smithi088.stderr:2023-10-30T14:13:42.775+0000 7f153a827640 -1 Falling back to public interface 2023-10-30T14:13:42.785 INFO:tasks.ceph.osd.7.smithi088.stderr:2023-10-30T14:13:42.783+0000 7f965213f640 -1 Falling back to public interface 2023-10-30T14:13:42.786 INFO:tasks.ceph.osd.6.smithi088.stderr:2023-10-30T14:13:42.783+0000 7f50fef11640 -1 Falling back to public interface 2023-10-30T14:13:42.800 INFO:tasks.ceph.osd.0.smithi057.stderr:2023-10-30T14:13:42.797+0000 7ff74c9d6640 -1 Falling back to public interface 2023-10-30T14:13:42.820 INFO:tasks.ceph.osd.1.smithi057.stderr:2023-10-30T14:13:42.817+0000 7fe332495640 -1 Falling back to public interface 2023-10-30T14:13:42.830 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:42.827+0000 7f49e8105180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T14:13:42.836 INFO:tasks.ceph.osd.3.smithi057.stderr:2023-10-30T14:13:42.833+0000 7f8111a12640 -1 Falling back to public interface 2023-10-30T14:13:42.847 INFO:tasks.ceph.osd.2.smithi057.stderr:2023-10-30T14:13:42.845+0000 7efd2d85e640 -1 Falling back to public interface 2023-10-30T14:13:42.932 INFO:tasks.ceph.mgr.y.smithi057.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-30T14:13:42.932 INFO:tasks.ceph.mgr.y.smithi057.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-30T14:13:42.933 INFO:tasks.ceph.mgr.y.smithi057.stderr: from numpy import show_config as show_numpy_config 2023-10-30T14:13:42.977 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:42.973+0000 7fb5e302d180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T14:13:42.990 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:42.987+0000 7f49e8105180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T14:13:43.052 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:43.047+0000 7f49e8105180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T14:13:43.119 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:43.117+0000 7fb5e302d180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T14:13:43.176 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:43.171+0000 7f49e8105180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T14:13:43.227 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T14:13:43.309 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:43.305+0000 7fb5e302d180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T14:13:43.359 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.357+0000 7fbb63417640 1 Processor -- start 2023-10-30T14:13:43.360 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.357+0000 7fbb63417640 1 -- start start 2023-10-30T14:13:43.361 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.357+0000 7fbb63417640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb5c153b30 0x7fbb5c0c4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:43.361 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.357+0000 7fbb63417640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbb5c0ca040 0x7fbb5c15a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:43.362 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.357+0000 7fbb63417640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbb5c15d570 0x7fbb5c0cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:43.362 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.357+0000 7fbb63417640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb5c0caa30 con 0x7fbb5c15d570 2023-10-30T14:13:43.362 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.357+0000 7fbb63417640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb5c15a730 con 0x7fbb5c0ca040 2023-10-30T14:13:43.362 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.357+0000 7fbb63417640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5c153f10 con 0x7fbb5c153b30 2023-10-30T14:13:43.362 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.357+0000 7fbb619b4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbb5c15d570 0x7fbb5c0cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:43.363 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.357+0000 7fbb609b2640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbb5c0ca040 0x7fbb5c15a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:43.363 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.357+0000 7fbb619b4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbb5c15d570 0x7fbb5c0cf030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33804/0 (socket says 172.21.15.57:33804) 2023-10-30T14:13:43.363 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.357+0000 7fbb619b4640 1 -- 172.21.15.57:0/681210490 learned_addr learned my addr 172.21.15.57:0/681210490 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:13:43.363 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.357+0000 7fbb611b3640 1 --2- 172.21.15.57:0/681210490 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb5c153b30 0x7fbb5c0c4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:43.364 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.357+0000 7fbb619b4640 1 -- 172.21.15.57:0/681210490 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb5c153b30 msgr2=0x7fbb5c0c4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:13:43.364 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.357+0000 7fbb619b4640 1 --2- 172.21.15.57:0/681210490 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb5c153b30 0x7fbb5c0c4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:43.364 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.357+0000 7fbb619b4640 1 -- 172.21.15.57:0/681210490 >> v2:172.21.15.88:3300/0 conn(0x7fbb5c0ca040 msgr2=0x7fbb5c15a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:13:43.364 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.357+0000 7fbb619b4640 1 --2- 172.21.15.57:0/681210490 >> v2:172.21.15.88:3300/0 conn(0x7fbb5c0ca040 0x7fbb5c15a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:43.364 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.357+0000 7fbb619b4640 1 -- 172.21.15.57:0/681210490 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5c15cde0 con 0x7fbb5c15d570 2023-10-30T14:13:43.365 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.361+0000 7fbb619b4640 1 --2- 172.21.15.57:0/681210490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb5c15d570 0x7fbb5c0cf030 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c0cf000 tx=0x7fbb5401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40dd3a43f4e2ad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:43.365 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.361+0000 7fbb53fff640 1 -- 172.21.15.57:0/681210490 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb540af400 con 0x7fbb5c15d570 2023-10-30T14:13:43.366 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.361+0000 7fbb53fff640 1 -- 172.21.15.57:0/681210490 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb540ab050 con 0x7fbb5c15d570 2023-10-30T14:13:43.366 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.361+0000 7fbb53fff640 1 -- 172.21.15.57:0/681210490 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb540af780 con 0x7fbb5c15d570 2023-10-30T14:13:43.366 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.361+0000 7fbb63417640 1 -- 172.21.15.57:0/681210490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb5c15d570 msgr2=0x7fbb5c0cf030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:43.366 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.361+0000 7fbb63417640 1 --2- 172.21.15.57:0/681210490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb5c15d570 0x7fbb5c0cf030 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c0cf000 tx=0x7fbb5401f1c0 comp rx=0 tx=0).stop 2023-10-30T14:13:43.366 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.361+0000 7fbb63417640 1 -- 172.21.15.57:0/681210490 shutdown_connections 2023-10-30T14:13:43.367 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.361+0000 7fbb63417640 1 --2- 172.21.15.57:0/681210490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb5c15d570 0x7fbb5c0cf030 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:43.367 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.361+0000 7fbb63417640 1 --2- 172.21.15.57:0/681210490 >> v2:172.21.15.88:3300/0 conn(0x7fbb5c0ca040 0x7fbb5c15a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:43.367 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.361+0000 7fbb63417640 1 --2- 172.21.15.57:0/681210490 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb5c153b30 0x7fbb5c0c4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:43.367 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.361+0000 7fbb63417640 1 -- 172.21.15.57:0/681210490 >> 172.21.15.57:0/681210490 conn(0x7fbb5c06bfe0 msgr2=0x7fbb5c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:43.368 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.361+0000 7fbb63417640 1 -- 172.21.15.57:0/681210490 shutdown_connections 2023-10-30T14:13:43.368 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.361+0000 7fbb63417640 1 -- 172.21.15.57:0/681210490 wait complete. 2023-10-30T14:13:43.368 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb63417640 1 Processor -- start 2023-10-30T14:13:43.368 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb63417640 1 -- start start 2023-10-30T14:13:43.370 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb63417640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb5c0ca040 0x7fbb5c1ef7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:43.371 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb63417640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb5c153b30 0x7fbb5c0c5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:43.371 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb63417640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb5c15d570 0x7fbb5c0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:43.371 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb63417640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5c0cedf0 con 0x7fbb5c153b30 2023-10-30T14:13:43.371 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb63417640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5c0ce040 con 0x7fbb5c0ca040 2023-10-30T14:13:43.372 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb63417640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5c1efce0 con 0x7fbb5c15d570 2023-10-30T14:13:43.372 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb611b3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb5c0ca040 0x7fbb5c1ef7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:43.372 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb609b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb5c153b30 0x7fbb5c0c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:43.372 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb609b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb5c153b30 0x7fbb5c0c5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33806/0 (socket says 172.21.15.57:33806) 2023-10-30T14:13:43.372 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb609b2640 1 -- 172.21.15.57:0/430810895 learned_addr learned my addr 172.21.15.57:0/430810895 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:13:43.372 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb611b3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb5c0ca040 0x7fbb5c1ef7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:55190/0 (socket says 172.21.15.57:55190) 2023-10-30T14:13:43.373 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb619b4640 1 --2- 172.21.15.57:0/430810895 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb5c15d570 0x7fbb5c0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:43.373 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb609b2640 1 -- 172.21.15.57:0/430810895 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb5c15d570 msgr2=0x7fbb5c0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:43.373 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb609b2640 1 --2- 172.21.15.57:0/430810895 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb5c15d570 0x7fbb5c0cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:43.373 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb609b2640 1 -- 172.21.15.57:0/430810895 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb5c0ca040 msgr2=0x7fbb5c1ef7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:43.373 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb609b2640 1 --2- 172.21.15.57:0/430810895 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb5c0ca040 0x7fbb5c1ef7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:43.373 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb609b2640 1 -- 172.21.15.57:0/430810895 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb54009b30 con 0x7fbb5c153b30 2023-10-30T14:13:43.374 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb609b2640 1 --2- 172.21.15.57:0/430810895 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb5c153b30 0x7fbb5c0c5270 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00e8b0 tx=0x7fbb4c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:43.374 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb51ffb640 1 -- 172.21.15.57:0/430810895 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c00c8e0 con 0x7fbb5c153b30 2023-10-30T14:13:43.374 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb51ffb640 1 -- 172.21.15.57:0/430810895 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4c00ca40 con 0x7fbb5c153b30 2023-10-30T14:13:43.374 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb51ffb640 1 -- 172.21.15.57:0/430810895 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c00cdd0 con 0x7fbb5c153b30 2023-10-30T14:13:43.374 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb63417640 1 -- 172.21.15.57:0/430810895 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb5c0ce270 con 0x7fbb5c153b30 2023-10-30T14:13:43.375 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb63417640 1 -- 172.21.15.57:0/430810895 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb5c0ce830 con 0x7fbb5c153b30 2023-10-30T14:13:43.375 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.365+0000 7fbb51ffb640 1 -- 172.21.15.57:0/430810895 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fbb4c005ca0 con 0x7fbb5c153b30 2023-10-30T14:13:43.375 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.369+0000 7fbb51ffb640 1 -- 172.21.15.57:0/430810895 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fbb4c004020 con 0x7fbb5c153b30 2023-10-30T14:13:43.375 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.369+0000 7fbb63417640 1 -- 172.21.15.57:0/430810895 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb5c04e170 con 0x7fbb5c153b30 2023-10-30T14:13:43.376 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.369+0000 7fbb51ffb640 1 -- 172.21.15.57:0/430810895 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fbb4c021060 con 0x7fbb5c153b30 2023-10-30T14:13:43.388 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:43.385+0000 7fb5e302d180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T14:13:43.432 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.429+0000 7fbb63417640 1 -- 172.21.15.57:0/430810895 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb5c0cb140 con 0x7fbb5c153b30 2023-10-30T14:13:43.433 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.429+0000 7fbb51ffb640 1 -- 172.21.15.57:0/430810895 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fbb4c017070 con 0x7fbb5c153b30 2023-10-30T14:13:43.433 INFO:teuthology.misc.health.smithi057.stdout: 2023-10-30T14:13:43.433 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:13:42.112020+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:13:43.434 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.429+0000 7fbb63417640 1 -- 172.21.15.57:0/430810895 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb5c153b30 msgr2=0x7fbb5c0c5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:43.434 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.429+0000 7fbb63417640 1 --2- 172.21.15.57:0/430810895 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb5c153b30 0x7fbb5c0c5270 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00e8b0 tx=0x7fbb4c00ed70 comp rx=0 tx=0).stop 2023-10-30T14:13:43.435 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.429+0000 7fbb63417640 1 -- 172.21.15.57:0/430810895 shutdown_connections 2023-10-30T14:13:43.435 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.429+0000 7fbb63417640 1 --2- 172.21.15.57:0/430810895 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb5c15d570 0x7fbb5c0cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:43.435 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.429+0000 7fbb63417640 1 --2- 172.21.15.57:0/430810895 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb5c153b30 0x7fbb5c0c5270 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:43.435 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.429+0000 7fbb63417640 1 --2- 172.21.15.57:0/430810895 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb5c0ca040 0x7fbb5c1ef7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:43.435 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.429+0000 7fbb63417640 1 -- 172.21.15.57:0/430810895 >> 172.21.15.57:0/430810895 conn(0x7fbb5c06bfe0 msgr2=0x7fbb5c150610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:43.436 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.429+0000 7fbb63417640 1 -- 172.21.15.57:0/430810895 shutdown_connections 2023-10-30T14:13:43.436 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:43.429+0000 7fbb63417640 1 -- 172.21.15.57:0/430810895 wait complete. 2023-10-30T14:13:43.451 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T14:13:43.537 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:43.533+0000 7fb5e302d180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T14:13:43.784 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:43.779+0000 7f49e8105180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T14:13:43.858 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:43.855+0000 7f49e8105180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T14:13:44.029 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:44.023+0000 7f49e8105180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T14:13:44.102 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:44.099+0000 7f49e8105180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T14:13:44.238 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:44.233+0000 7fb5e302d180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T14:13:44.329 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:44.325+0000 7fb5e302d180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T14:13:44.390 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:44.387+0000 7f49e8105180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T14:13:44.449 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:44.443+0000 7f49e8105180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T14:13:44.536 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:44.533+0000 7fb5e302d180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T14:13:44.626 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:44.625+0000 7fb5e302d180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T14:13:44.974 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:44.969+0000 7fb5e302d180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T14:13:45.047 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-30T14:13:45.045+0000 7fb5e302d180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T14:13:45.501 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:45.499+0000 7f498f97a640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-30T14:13:45.502 INFO:tasks.ceph.mgr.x.smithi088.stderr:2023-10-30T14:13:45.499+0000 7f498f97a640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-30T14:13:46.668 INFO:tasks.ceph.osd.0.smithi057.stderr:2023-10-30T14:13:46.665+0000 7ff74c9d6640 -1 osd.0 0 log_to_monitors true 2023-10-30T14:13:46.692 INFO:tasks.ceph.osd.6.smithi088.stderr:2023-10-30T14:13:46.687+0000 7f50fef11640 -1 osd.6 0 log_to_monitors true 2023-10-30T14:13:46.696 INFO:tasks.ceph.osd.5.smithi088.stderr:2023-10-30T14:13:46.691+0000 7f667846e640 -1 osd.5 0 log_to_monitors true 2023-10-30T14:13:46.729 INFO:tasks.ceph.osd.4.smithi088.stderr:2023-10-30T14:13:46.727+0000 7f153a827640 -1 osd.4 0 log_to_monitors true 2023-10-30T14:13:46.744 INFO:tasks.ceph.osd.7.smithi088.stderr:2023-10-30T14:13:46.739+0000 7f965213f640 -1 osd.7 0 log_to_monitors true 2023-10-30T14:13:46.748 INFO:tasks.ceph.osd.3.smithi057.stderr:2023-10-30T14:13:46.745+0000 7f8111a12640 -1 osd.3 0 log_to_monitors true 2023-10-30T14:13:46.779 INFO:tasks.ceph.osd.1.smithi057.stderr:2023-10-30T14:13:46.777+0000 7fe332495640 -1 osd.1 0 log_to_monitors true 2023-10-30T14:13:46.819 INFO:tasks.ceph.osd.2.smithi057.stderr:2023-10-30T14:13:46.817+0000 7efd2d85e640 -1 osd.2 0 log_to_monitors true 2023-10-30T14:13:50.130 INFO:tasks.ceph.osd.2.smithi057.stderr:2023-10-30T14:13:50.125+0000 7efd28fe3640 -1 osd.2 0 waiting for initial osdmap 2023-10-30T14:13:50.150 INFO:tasks.ceph.osd.1.smithi057.stderr:2023-10-30T14:13:50.145+0000 7fe32dc1c640 -1 osd.1 0 waiting for initial osdmap 2023-10-30T14:13:50.169 INFO:tasks.ceph.osd.2.smithi057.stderr:2023-10-30T14:13:50.165+0000 7efd2162a640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T14:13:50.177 INFO:tasks.ceph.osd.1.smithi057.stderr:2023-10-30T14:13:50.173+0000 7fe326263640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T14:13:50.208 INFO:tasks.ceph.osd.3.smithi057.stderr:2023-10-30T14:13:50.205+0000 7f810d199640 -1 osd.3 0 waiting for initial osdmap 2023-10-30T14:13:50.214 INFO:tasks.ceph.osd.0.smithi057.stderr:2023-10-30T14:13:50.209+0000 7ff74815b640 -1 osd.0 0 waiting for initial osdmap 2023-10-30T14:13:50.233 INFO:tasks.ceph.osd.6.smithi088.stderr:2023-10-30T14:13:50.227+0000 7f50fbece640 -1 osd.6 0 waiting for initial osdmap 2023-10-30T14:13:50.236 INFO:tasks.ceph.osd.3.smithi057.stderr:2023-10-30T14:13:50.233+0000 7f81057e0640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T14:13:50.236 INFO:tasks.ceph.osd.0.smithi057.stderr:2023-10-30T14:13:50.233+0000 7ff7407a2640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T14:13:50.256 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T14:13:50.262 INFO:tasks.ceph.osd.6.smithi088.stderr:2023-10-30T14:13:50.258+0000 7f50f2cdd640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T14:13:50.299 INFO:tasks.ceph.osd.4.smithi088.stderr:2023-10-30T14:13:50.294+0000 7f1535fac640 -1 osd.4 0 waiting for initial osdmap 2023-10-30T14:13:50.330 INFO:tasks.ceph.osd.4.smithi088.stderr:2023-10-30T14:13:50.326+0000 7f152e5f3640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T14:13:50.340 INFO:tasks.ceph.osd.7.smithi088.stderr:2023-10-30T14:13:50.334+0000 7f964d8c4640 -1 osd.7 0 waiting for initial osdmap 2023-10-30T14:13:50.366 INFO:tasks.ceph.osd.7.smithi088.stderr:2023-10-30T14:13:50.362+0000 7f9645f0b640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T14:13:50.370 INFO:tasks.ceph.osd.5.smithi088.stderr:2023-10-30T14:13:50.366+0000 7f6673bf3640 -1 osd.5 0 waiting for initial osdmap 2023-10-30T14:13:50.389 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.385+0000 7fd784f80640 1 Processor -- start 2023-10-30T14:13:50.389 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.385+0000 7fd784f80640 1 -- start start 2023-10-30T14:13:50.391 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.385+0000 7fd784f80640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7800c4600 0x7fd78015a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:50.392 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.385+0000 7fd784f80640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7800caa30 con 0x7fd7800ca040 2023-10-30T14:13:50.392 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.385+0000 7fd784f80640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd78015a730 con 0x7fd780153b30 2023-10-30T14:13:50.392 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.385+0000 7fd784f80640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd780153f10 con 0x7fd7800c4600 2023-10-30T14:13:50.392 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.389+0000 7fd77e59c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd780153b30 0x7fd78015d570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:52130/0 (socket says 172.21.15.57:52130) 2023-10-30T14:13:50.392 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.389+0000 7fd77e59c640 1 -- 172.21.15.57:0/3267695908 learned_addr learned my addr 172.21.15.57:0/3267695908 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:13:50.393 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.389+0000 7fd77ed9d640 1 --2- 172.21.15.57:0/3267695908 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7800c4600 0x7fd78015a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:50.393 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.389+0000 7fd77d59a640 1 -- 172.21.15.57:0/3267695908 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779011261 0 0) 0x7fd78015a730 con 0x7fd780153b30 2023-10-30T14:13:50.393 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.389+0000 7fd77d59a640 1 -- 172.21.15.57:0/3267695908 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd768003530 con 0x7fd780153b30 2023-10-30T14:13:50.393 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.389+0000 7fd77d59a640 1 -- 172.21.15.57:0/3267695908 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949447473 0 0) 0x7fd774001120 con 0x7fd7800ca040 2023-10-30T14:13:50.393 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.389+0000 7fd77d59a640 1 -- 172.21.15.57:0/3267695908 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd78015a730 con 0x7fd7800ca040 2023-10-30T14:13:50.393 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.389+0000 7fd77ed9d640 1 -- 172.21.15.57:0/3267695908 >> v1:172.21.15.88:6789/0 conn(0x7fd780153b30 legacy=0x7fd78015d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:50.394 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.389+0000 7fd77ed9d640 1 -- 172.21.15.57:0/3267695908 >> v1:172.21.15.57:6789/0 conn(0x7fd7800ca040 legacy=0x7fd7800cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:50.394 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.389+0000 7fd77ed9d640 1 -- 172.21.15.57:0/3267695908 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd78015cde0 con 0x7fd7800c4600 2023-10-30T14:13:50.395 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.389+0000 7fd77ed9d640 1 --2- 172.21.15.57:0/3267695908 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7800c4600 0x7fd78015a1f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd778009dc0 tx=0x7fd77801f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6ce3b79e0106e90 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:50.395 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.389+0000 7fd77d59a640 1 -- 172.21.15.57:0/3267695908 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7780af600 con 0x7fd7800c4600 2023-10-30T14:13:50.395 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.389+0000 7fd77d59a640 1 -- 172.21.15.57:0/3267695908 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7780ab050 con 0x7fd7800c4600 2023-10-30T14:13:50.395 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.389+0000 7fd77d59a640 1 -- 172.21.15.57:0/3267695908 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7780af980 con 0x7fd7800c4600 2023-10-30T14:13:50.396 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd784f80640 1 -- 172.21.15.57:0/3267695908 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7800c4600 msgr2=0x7fd78015a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:50.396 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd784f80640 1 --2- 172.21.15.57:0/3267695908 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7800c4600 0x7fd78015a1f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd778009dc0 tx=0x7fd77801f150 comp rx=0 tx=0).stop 2023-10-30T14:13:50.396 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd784f80640 1 -- 172.21.15.57:0/3267695908 shutdown_connections 2023-10-30T14:13:50.396 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd784f80640 1 --2- 172.21.15.57:0/3267695908 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7800c4600 0x7fd78015a1f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:50.397 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd784f80640 1 -- 172.21.15.57:0/3267695908 >> 172.21.15.57:0/3267695908 conn(0x7fd78006bfe0 msgr2=0x7fd78006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:50.397 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd784f80640 1 -- 172.21.15.57:0/3267695908 shutdown_connections 2023-10-30T14:13:50.397 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd784f80640 1 -- 172.21.15.57:0/3267695908 wait complete. 2023-10-30T14:13:50.397 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd784f80640 1 Processor -- start 2023-10-30T14:13:50.398 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd784f80640 1 -- start start 2023-10-30T14:13:50.400 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd784f80640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7800c4600 0x7fd7801f1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:50.400 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd784f80640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7800ca040 0x7fd7801f1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:50.400 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd784f80640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd780153b30 0x7fd7801f0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:50.400 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd784f80640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7801f0250 con 0x7fd780153b30 2023-10-30T14:13:50.400 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd784f80640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7800ce960 con 0x7fd7800ca040 2023-10-30T14:13:50.401 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd784f80640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78015c260 con 0x7fd7800c4600 2023-10-30T14:13:50.401 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd77dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7800ca040 0x7fd7801f1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:50.401 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd77e59c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7800c4600 0x7fd7801f1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:50.401 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd77ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd780153b30 0x7fd7801f0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:50.401 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd77e59c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7800c4600 0x7fd7801f1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:36204/0 (socket says 172.21.15.57:36204) 2023-10-30T14:13:50.402 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd77dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7800ca040 0x7fd7801f1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:48014/0 (socket says 172.21.15.57:48014) 2023-10-30T14:13:50.402 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.393+0000 7fd77e59c640 1 -- 172.21.15.57:0/237284921 learned_addr learned my addr 172.21.15.57:0/237284921 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:13:50.402 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.397+0000 7fd77dd9b640 1 -- 172.21.15.57:0/237284921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7800c4600 msgr2=0x7fd7801f1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:50.402 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.397+0000 7fd77dd9b640 1 --2- 172.21.15.57:0/237284921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7800c4600 0x7fd7801f1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:50.402 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.397+0000 7fd77dd9b640 1 -- 172.21.15.57:0/237284921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd780153b30 msgr2=0x7fd7801f0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:50.403 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.397+0000 7fd77dd9b640 1 --2- 172.21.15.57:0/237284921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd780153b30 0x7fd7801f0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:50.403 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.397+0000 7fd77dd9b640 1 -- 172.21.15.57:0/237284921 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd778009a70 con 0x7fd7800ca040 2023-10-30T14:13:50.403 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.397+0000 7fd77e59c640 1 --2- 172.21.15.57:0/237284921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7800c4600 0x7fd7801f1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:13:50.403 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.397+0000 7fd77ed9d640 1 --2- 172.21.15.57:0/237284921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd780153b30 0x7fd7801f0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:13:50.403 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.397+0000 7fd77dd9b640 1 --2- 172.21.15.57:0/237284921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7800ca040 0x7fd7801f1ba0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd77400c730 tx=0x7fd77400cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:50.404 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.397+0000 7fd76effd640 1 -- 172.21.15.57:0/237284921 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7740050d0 con 0x7fd7800ca040 2023-10-30T14:13:50.404 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.397+0000 7fd76effd640 1 -- 172.21.15.57:0/237284921 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd774005230 con 0x7fd7800ca040 2023-10-30T14:13:50.404 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.397+0000 7fd76effd640 1 -- 172.21.15.57:0/237284921 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd774003600 con 0x7fd7800ca040 2023-10-30T14:13:50.404 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.397+0000 7fd784f80640 1 -- 172.21.15.57:0/237284921 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd78015a910 con 0x7fd7800ca040 2023-10-30T14:13:50.404 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.397+0000 7fd784f80640 1 -- 172.21.15.57:0/237284921 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd78015ae90 con 0x7fd7800ca040 2023-10-30T14:13:50.405 INFO:tasks.ceph.osd.5.smithi088.stderr:2023-10-30T14:13:50.398+0000 7f666c23a640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T14:13:50.406 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.397+0000 7fd784f80640 1 -- 172.21.15.57:0/237284921 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7801f0390 con 0x7fd7800ca040 2023-10-30T14:13:50.406 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.397+0000 7fd76effd640 1 -- 172.21.15.57:0/237284921 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 85794+0+0 (secure 0 0 0) 0x7fd774003760 con 0x7fd7800ca040 2023-10-30T14:13:50.406 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.397+0000 7fd76effd640 1 --2- 172.21.15.57:0/237284921 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd760066440 0x7fd760068900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:50.406 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.397+0000 7fd76effd640 1 -- 172.21.15.57:0/237284921 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fd774082720 con 0x7fd7800ca040 2023-10-30T14:13:50.406 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.397+0000 7fd77e59c640 1 --2- 172.21.15.57:0/237284921 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd760066440 0x7fd760068900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:50.407 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.401+0000 7fd77e59c640 1 --2- 172.21.15.57:0/237284921 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd760066440 0x7fd760068900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd770000f70 tx=0x7fd770009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:50.408 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.405+0000 7fd76effd640 1 -- 172.21.15.57:0/237284921 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd7801f0390 con 0x7fd7800ca040 2023-10-30T14:13:50.578 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.573+0000 7fd784f80640 1 -- 172.21.15.57:0/237284921 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7801f0390 con 0x7fd7800ca040 2023-10-30T14:13:50.579 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.577+0000 7fd76effd640 1 -- 172.21.15.57:0/237284921 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fd7801f0390 con 0x7fd7800ca040 2023-10-30T14:13:50.579 INFO:teuthology.misc.health.smithi057.stdout: 2023-10-30T14:13:50.579 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":12,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:13:47.909986+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:13:50.581 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.577+0000 7fd784f80640 1 -- 172.21.15.57:0/237284921 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd760066440 msgr2=0x7fd760068900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:50.581 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.577+0000 7fd784f80640 1 --2- 172.21.15.57:0/237284921 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd760066440 0x7fd760068900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd770000f70 tx=0x7fd770009040 comp rx=0 tx=0).stop 2023-10-30T14:13:50.582 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.577+0000 7fd784f80640 1 -- 172.21.15.57:0/237284921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7800ca040 msgr2=0x7fd7801f1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:50.582 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.577+0000 7fd784f80640 1 --2- 172.21.15.57:0/237284921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7800ca040 0x7fd7801f1ba0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd77400c730 tx=0x7fd77400cbf0 comp rx=0 tx=0).stop 2023-10-30T14:13:50.582 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.577+0000 7fd784f80640 1 -- 172.21.15.57:0/237284921 shutdown_connections 2023-10-30T14:13:50.582 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.577+0000 7fd784f80640 1 --2- 172.21.15.57:0/237284921 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd760066440 0x7fd760068900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:50.583 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.577+0000 7fd784f80640 1 --2- 172.21.15.57:0/237284921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd780153b30 0x7fd7801f0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:50.583 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.577+0000 7fd784f80640 1 --2- 172.21.15.57:0/237284921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7800ca040 0x7fd7801f1ba0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:50.583 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.577+0000 7fd784f80640 1 --2- 172.21.15.57:0/237284921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7800c4600 0x7fd7801f1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:50.583 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.577+0000 7fd784f80640 1 -- 172.21.15.57:0/237284921 >> 172.21.15.57:0/237284921 conn(0x7fd78006bfe0 msgr2=0x7fd7800c4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:50.583 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.577+0000 7fd784f80640 1 -- 172.21.15.57:0/237284921 shutdown_connections 2023-10-30T14:13:50.583 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:50.577+0000 7fd784f80640 1 -- 172.21.15.57:0/237284921 wait complete. 2023-10-30T14:13:50.599 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T14:13:57.405 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T14:13:57.536 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.532+0000 7f676c846640 1 Processor -- start 2023-10-30T14:13:57.536 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.532+0000 7f676c846640 1 -- start start 2023-10-30T14:13:57.538 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.532+0000 7f676c846640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6768153b30 0x7f67680c4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:57.538 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.532+0000 7f676c846640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f676815a1f0 0x7f67680cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:57.538 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.532+0000 7f676c846640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67680caa30 con 0x7f67680ca040 2023-10-30T14:13:57.538 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.532+0000 7f676c846640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6768153f10 con 0x7f676815a1f0 2023-10-30T14:13:57.539 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.532+0000 7f676c846640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f676815cde0 con 0x7f6768153b30 2023-10-30T14:13:57.539 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.532+0000 7f6766d9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f676815a1f0 0x7f67680cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:57.539 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.532+0000 7f6766d9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f676815a1f0 0x7f67680cf030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44536/0 (socket says 172.21.15.57:44536) 2023-10-30T14:13:57.539 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.532+0000 7f6766d9d640 1 -- 172.21.15.57:0/3899614700 learned_addr learned my addr 172.21.15.57:0/3899614700 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:13:57.539 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.532+0000 7f675ffff640 1 --1- 172.21.15.57:0/3899614700 >> v1:172.21.15.57:6789/0 conn(0x7f67680ca040 0x7f676815d570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:45794/0 (socket says 172.21.15.57:45794) 2023-10-30T14:13:57.540 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.532+0000 7f676659c640 1 --2- 172.21.15.57:0/3899614700 >> v2:172.21.15.88:3300/0 conn(0x7f6768153b30 0x7f67680c4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:57.540 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.532+0000 7f6766d9d640 1 -- 172.21.15.57:0/3899614700 >> v2:172.21.15.88:3300/0 conn(0x7f6768153b30 msgr2=0x7f67680c4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:13:57.540 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.532+0000 7f6766d9d640 1 --2- 172.21.15.57:0/3899614700 >> v2:172.21.15.88:3300/0 conn(0x7f6768153b30 0x7f67680c4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:57.540 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.532+0000 7f6766d9d640 1 -- 172.21.15.57:0/3899614700 >> v1:172.21.15.57:6789/0 conn(0x7f67680ca040 legacy=0x7f676815d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:57.540 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.532+0000 7f6766d9d640 1 -- 172.21.15.57:0/3899614700 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676815dc80 con 0x7f676815a1f0 2023-10-30T14:13:57.541 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.532+0000 7f676659c640 1 --2- 172.21.15.57:0/3899614700 >> v2:172.21.15.88:3300/0 conn(0x7f6768153b30 0x7f67680c4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:13:57.541 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.532+0000 7f6765d9b640 1 -- 172.21.15.57:0/3899614700 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927629729 0 0) 0x7f6750001120 con 0x7f67680ca040 2023-10-30T14:13:57.542 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.536+0000 7f6766d9d640 1 --2- 172.21.15.57:0/3899614700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f676815a1f0 0x7f67680cf030 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f676808cac0 tx=0x7f676001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59443515b51f819 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:57.542 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.536+0000 7f6765d9b640 1 -- 172.21.15.57:0/3899614700 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67600af400 con 0x7f676815a1f0 2023-10-30T14:13:57.543 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.536+0000 7f6765d9b640 1 -- 172.21.15.57:0/3899614700 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67600ab050 con 0x7f676815a1f0 2023-10-30T14:13:57.543 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.536+0000 7f6765d9b640 1 -- 172.21.15.57:0/3899614700 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67600af780 con 0x7f676815a1f0 2023-10-30T14:13:57.543 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.536+0000 7f676c846640 1 -- 172.21.15.57:0/3899614700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f676815a1f0 msgr2=0x7f67680cf030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:57.543 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.536+0000 7f676c846640 1 --2- 172.21.15.57:0/3899614700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f676815a1f0 0x7f67680cf030 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f676808cac0 tx=0x7f676001f1c0 comp rx=0 tx=0).stop 2023-10-30T14:13:57.543 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.536+0000 7f676c846640 1 -- 172.21.15.57:0/3899614700 shutdown_connections 2023-10-30T14:13:57.544 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.536+0000 7f676c846640 1 --2- 172.21.15.57:0/3899614700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f676815a1f0 0x7f67680cf030 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:57.544 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.536+0000 7f676c846640 1 --2- 172.21.15.57:0/3899614700 >> v2:172.21.15.88:3300/0 conn(0x7f6768153b30 0x7f67680c4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:57.544 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.536+0000 7f676c846640 1 -- 172.21.15.57:0/3899614700 >> 172.21.15.57:0/3899614700 conn(0x7f676806bfe0 msgr2=0x7f676806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:57.544 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.536+0000 7f676c846640 1 -- 172.21.15.57:0/3899614700 shutdown_connections 2023-10-30T14:13:57.544 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.536+0000 7f676c846640 1 -- 172.21.15.57:0/3899614700 wait complete. 2023-10-30T14:13:57.546 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f676c846640 1 Processor -- start 2023-10-30T14:13:57.547 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f676c846640 1 -- start start 2023-10-30T14:13:57.547 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f676c846640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67680ca040 0x7f67681eeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:57.547 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f676c846640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6768153b30 0x7f6768149d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:57.547 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f676c846640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f676815a1f0 0x7f67681554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:57.548 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f676c846640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67680b29e0 con 0x7f676815a1f0 2023-10-30T14:13:57.548 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f676c846640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6768155fb0 con 0x7f67680ca040 2023-10-30T14:13:57.548 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f676c846640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6768156290 con 0x7f6768153b30 2023-10-30T14:13:57.548 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f6766d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f676815a1f0 0x7f67681554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:57.548 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f6766d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f676815a1f0 0x7f67681554e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44544/0 (socket says 172.21.15.57:44544) 2023-10-30T14:13:57.549 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f6766d9d640 1 -- 172.21.15.57:0/3737271564 learned_addr learned my addr 172.21.15.57:0/3737271564 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:13:57.549 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f676659c640 1 --2- 172.21.15.57:0/3737271564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67680ca040 0x7f67681eeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:57.549 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f675ffff640 1 --2- 172.21.15.57:0/3737271564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6768153b30 0x7f6768149d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:57.549 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f6766d9d640 1 -- 172.21.15.57:0/3737271564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6768153b30 msgr2=0x7f6768149d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:57.549 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f6766d9d640 1 --2- 172.21.15.57:0/3737271564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6768153b30 0x7f6768149d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:57.549 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f6766d9d640 1 -- 172.21.15.57:0/3737271564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67680ca040 msgr2=0x7f67681eeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:57.550 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f6766d9d640 1 --2- 172.21.15.57:0/3737271564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67680ca040 0x7f67681eeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:57.550 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f6766d9d640 1 -- 172.21.15.57:0/3737271564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6760009b30 con 0x7f676815a1f0 2023-10-30T14:13:57.550 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f6766d9d640 1 --2- 172.21.15.57:0/3737271564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f676815a1f0 0x7f67681554e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6760009f10 tx=0x7f676001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:57.550 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f675effd640 1 -- 172.21.15.57:0/3737271564 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67600afe40 con 0x7f676815a1f0 2023-10-30T14:13:57.550 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f675effd640 1 -- 172.21.15.57:0/3737271564 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6760009510 con 0x7f676815a1f0 2023-10-30T14:13:57.551 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f675effd640 1 -- 172.21.15.57:0/3737271564 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6760003bc0 con 0x7f676815a1f0 2023-10-30T14:13:57.551 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f676c846640 1 -- 172.21.15.57:0/3737271564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6768156570 con 0x7f676815a1f0 2023-10-30T14:13:57.551 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f676c846640 1 -- 172.21.15.57:0/3737271564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6768156b30 con 0x7f676815a1f0 2023-10-30T14:13:57.551 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.540+0000 7f676c846640 1 -- 172.21.15.57:0/3737271564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6768157050 con 0x7f676815a1f0 2023-10-30T14:13:57.552 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.544+0000 7f675effd640 1 -- 172.21.15.57:0/3737271564 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f67600b6070 con 0x7f676815a1f0 2023-10-30T14:13:57.552 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.544+0000 7f675effd640 1 --2- 172.21.15.57:0/3737271564 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f67480664a0 0x7f6748068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:57.552 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.544+0000 7f675effd640 1 -- 172.21.15.57:0/3737271564 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f67601211a0 con 0x7f676815a1f0 2023-10-30T14:13:57.553 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.544+0000 7f676659c640 1 --2- 172.21.15.57:0/3737271564 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f67480664a0 0x7f6748068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:57.553 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.544+0000 7f676659c640 1 --2- 172.21.15.57:0/3737271564 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f67480664a0 0x7f6748068960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6758009950 tx=0x7f6758008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:57.554 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.548+0000 7f675effd640 1 -- 172.21.15.57:0/3737271564 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6768157050 con 0x7f676815a1f0 2023-10-30T14:13:57.712 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.708+0000 7f676c846640 1 -- 172.21.15.57:0/3737271564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6768157050 con 0x7f676815a1f0 2023-10-30T14:13:57.713 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.708+0000 7f675effd640 1 -- 172.21.15.57:0/3737271564 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12778 (secure 0 0 0) 0x7f6768157050 con 0x7f676815a1f0 2023-10-30T14:13:57.713 INFO:teuthology.misc.health.smithi057.stdout: 2023-10-30T14:13:57.713 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":17,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:13:55.244786+0000","last_up_change":"2023-10-30T14:13:51.171287+0000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T14:13:52.574639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6801","nonce":3287367848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6803","nonce":3287367848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6807","nonce":3287367848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6805","nonce":3287367848}]},"public_addr":"172.21.15.57:6801/3287367848","cluster_addr":"172.21.15.57:6803/3287367848","heartbeat_back_addr":"172.21.15.57:6807/3287367848","heartbeat_front_addr":"172.21.15.57:6805/3287367848","state":["exists","up"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6809","nonce":3340797836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6811","nonce":3340797836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6815","nonce":3340797836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6813","nonce":3340797836}]},"public_addr":"172.21.15.57:6809/3340797836","cluster_addr":"172.21.15.57:6811/3340797836","heartbeat_back_addr":"172.21.15.57:6815/3340797836","heartbeat_front_addr":"172.21.15.57:6813/3340797836","state":["exists","up"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6825","nonce":1700292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6827","nonce":1700292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6831","nonce":1700292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6829","nonce":1700292}]},"public_addr":"172.21.15.57:6825/1700292","cluster_addr":"172.21.15.57:6827/1700292","heartbeat_back_addr":"172.21.15.57:6831/1700292","heartbeat_front_addr":"172.21.15.57:6829/1700292","state":["exists","up"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6817","nonce":2893420631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6819","nonce":2893420631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6823","nonce":2893420631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6821","nonce":2893420631}]},"public_addr":"172.21.15.57:6817/2893420631","cluster_addr":"172.21.15.57:6819/2893420631","heartbeat_back_addr":"172.21.15.57:6823/2893420631","heartbeat_front_addr":"172.21.15.57:6821/2893420631","state":["exists","up"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6807","nonce":2784816485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6811","nonce":2784816485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6815","nonce":2784816485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6813","nonce":2784816485}]},"public_addr":"172.21.15.88:6807/2784816485","cluster_addr":"172.21.15.88:6811/2784816485","heartbeat_back_addr":"172.21.15.88:6815/2784816485","heartbeat_front_addr":"172.21.15.88:6813/2784816485","state":["exists","up"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6801","nonce":2682449812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6803","nonce":2682449812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6810","nonce":2682449812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6806","nonce":2682449812}]},"public_addr":"172.21.15.88:6801/2682449812","cluster_addr":"172.21.15.88:6803/2682449812","heartbeat_back_addr":"172.21.15.88:6810/2682449812","heartbeat_front_addr":"172.21.15.88:6806/2682449812","state":["exists","up"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6822","nonce":3340693897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6827","nonce":3340693897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6831","nonce":3340693897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6829","nonce":3340693897}]},"public_addr":"172.21.15.88:6822/3340693897","cluster_addr":"172.21.15.88:6827/3340693897","heartbeat_back_addr":"172.21.15.88:6831/3340693897","heartbeat_front_addr":"172.21.15.88:6829/3340693897","state":["exists","up"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6817","nonce":2862111199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6820","nonce":2862111199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6825","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6826","nonce":2862111199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6821","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6823","nonce":2862111199}]},"public_addr":"172.21.15.88:6817/2862111199","cluster_addr":"172.21.15.88:6820/2862111199","heartbeat_back_addr":"172.21.15.88:6826/2862111199","heartbeat_front_addr":"172.21.15.88:6823/2862111199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.634920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.773993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.725825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.761452+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.694274+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.650718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:13:57.715 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.708+0000 7f676c846640 1 -- 172.21.15.57:0/3737271564 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f67480664a0 msgr2=0x7f6748068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:57.715 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.708+0000 7f676c846640 1 --2- 172.21.15.57:0/3737271564 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f67480664a0 0x7f6748068960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6758009950 tx=0x7f6758008040 comp rx=0 tx=0).stop 2023-10-30T14:13:57.715 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.708+0000 7f676c846640 1 -- 172.21.15.57:0/3737271564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f676815a1f0 msgr2=0x7f67681554e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:57.715 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.708+0000 7f676c846640 1 --2- 172.21.15.57:0/3737271564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f676815a1f0 0x7f67681554e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6760009f10 tx=0x7f676001bf60 comp rx=0 tx=0).stop 2023-10-30T14:13:57.716 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.708+0000 7f676c846640 1 -- 172.21.15.57:0/3737271564 shutdown_connections 2023-10-30T14:13:57.716 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.708+0000 7f676c846640 1 --2- 172.21.15.57:0/3737271564 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f67480664a0 0x7f6748068960 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:57.716 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.708+0000 7f676c846640 1 --2- 172.21.15.57:0/3737271564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f676815a1f0 0x7f67681554e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:57.716 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.708+0000 7f676c846640 1 --2- 172.21.15.57:0/3737271564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6768153b30 0x7f6768149d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:57.716 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.708+0000 7f676c846640 1 --2- 172.21.15.57:0/3737271564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67680ca040 0x7f67681eeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:57.717 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.708+0000 7f676c846640 1 -- 172.21.15.57:0/3737271564 >> 172.21.15.57:0/3737271564 conn(0x7f676806bfe0 msgr2=0x7f67681524a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:57.717 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.708+0000 7f676c846640 1 -- 172.21.15.57:0/3737271564 shutdown_connections 2023-10-30T14:13:57.717 INFO:teuthology.misc.health.smithi057.stderr:2023-10-30T14:13:57.708+0000 7f676c846640 1 -- 172.21.15.57:0/3737271564 wait complete. 2023-10-30T14:13:57.733 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-30T14:13:57.733 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-30T14:13:57.734 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T14:13:57.734 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T14:13:57.738 INFO:tasks.ceph.mds.a:Started 2023-10-30T14:13:57.738 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T14:13:57.738 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T14:13:57.742 INFO:tasks.ceph.mds.c:Started 2023-10-30T14:13:57.742 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T14:13:57.742 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T14:13:57.746 INFO:tasks.ceph.mds.e:Started 2023-10-30T14:13:57.746 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-30T14:13:57.747 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-30T14:13:57.750 INFO:tasks.ceph.mds.b:Started 2023-10-30T14:13:57.750 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T14:13:57.750 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T14:13:57.753 INFO:tasks.ceph.mds.d:Started 2023-10-30T14:13:57.753 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T14:13:57.753 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T14:13:57.756 INFO:tasks.ceph.mds.f:Started 2023-10-30T14:13:57.756 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-30T14:13:57.757 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:13:57.836 INFO:tasks.ceph.mds.d.smithi088.stdout:starting mds.d at 2023-10-30T14:13:57.843 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-30T14:13:57.844 INFO:tasks.ceph.mds.e.smithi057.stdout:starting mds.e at 2023-10-30T14:13:57.844 INFO:tasks.ceph.mds.b.smithi088.stdout:starting mds.b at 2023-10-30T14:13:57.845 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-30T14:13:57.845 INFO:tasks.ceph.mds.f.smithi088.stdout:starting mds.f at 2023-10-30T14:13:57.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.908+0000 7f350c8ea640 1 Processor -- start 2023-10-30T14:13:57.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.908+0000 7f350c8ea640 1 -- start start 2023-10-30T14:13:57.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.912+0000 7f350c8ea640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3508153c80 0x7f350815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:57.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.912+0000 7f350c8ea640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3508159f20 con 0x7f35080c42a0 2023-10-30T14:13:57.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.912+0000 7f350c8ea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350815d620 con 0x7f35080c4a10 2023-10-30T14:13:57.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.912+0000 7f350c8ea640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f350815dec0 con 0x7f3508153c80 2023-10-30T14:13:57.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.912+0000 7f350659c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3508153c80 0x7f350815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:57.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.912+0000 7f3506d9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f35080c42a0 0x7f35080ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:45804/0 (socket says 172.21.15.57:45804) 2023-10-30T14:13:57.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.912+0000 7f3506d9d640 1 -- 172.21.15.57:0/16800333 learned_addr learned my addr 172.21.15.57:0/16800333 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:13:57.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.912+0000 7f350659c640 1 -- 172.21.15.57:0/16800333 >> v1:172.21.15.88:6789/0 conn(0x7f35080c4a10 legacy=0x7f350815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:57.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.912+0000 7f350659c640 1 -- 172.21.15.57:0/16800333 >> v1:172.21.15.57:6789/0 conn(0x7f35080c42a0 legacy=0x7f35080ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:57.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.912+0000 7f350659c640 1 -- 172.21.15.57:0/16800333 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350815d900 con 0x7f3508153c80 2023-10-30T14:13:57.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.916+0000 7f350659c640 1 --2- 172.21.15.57:0/16800333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3508153c80 0x7f350815cf90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f34f80097c0 tx=0x7f34f801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db24f74e43401978 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:57.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.916+0000 7f350559a640 1 -- 172.21.15.57:0/16800333 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f80a8970 con 0x7f3508153c80 2023-10-30T14:13:57.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.916+0000 7f350559a640 1 -- 172.21.15.57:0/16800333 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f80a8ad0 con 0x7f3508153c80 2023-10-30T14:13:57.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.916+0000 7f350559a640 1 -- 172.21.15.57:0/16800333 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f80a8e60 con 0x7f3508153c80 2023-10-30T14:13:57.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.916+0000 7f350c8ea640 1 -- 172.21.15.57:0/16800333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3508153c80 msgr2=0x7f350815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:57.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.916+0000 7f350c8ea640 1 --2- 172.21.15.57:0/16800333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3508153c80 0x7f350815cf90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f34f80097c0 tx=0x7f34f801ec70 comp rx=0 tx=0).stop 2023-10-30T14:13:57.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.916+0000 7f350c8ea640 1 -- 172.21.15.57:0/16800333 shutdown_connections 2023-10-30T14:13:57.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.916+0000 7f350c8ea640 1 --2- 172.21.15.57:0/16800333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3508153c80 0x7f350815cf90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:57.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.916+0000 7f350c8ea640 1 -- 172.21.15.57:0/16800333 >> 172.21.15.57:0/16800333 conn(0x7f35080715c0 msgr2=0x7f35080719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:57.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.916+0000 7f350c8ea640 1 -- 172.21.15.57:0/16800333 shutdown_connections 2023-10-30T14:13:57.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.916+0000 7f350c8ea640 1 -- 172.21.15.57:0/16800333 wait complete. 2023-10-30T14:13:57.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.916+0000 7f350c8ea640 1 Processor -- start 2023-10-30T14:13:57.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.916+0000 7f350c8ea640 1 -- start start 2023-10-30T14:13:57.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.916+0000 7f350c8ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35080c42a0 0x7f3508155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:57.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.916+0000 7f350c8ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35080c4a10 0x7f35080c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:57.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.916+0000 7f350c8ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3508153c80 0x7f3508158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:57.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f350c8ea640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3508155190 con 0x7f35080c42a0 2023-10-30T14:13:57.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f350c8ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35081f2d80 con 0x7f35080c4a10 2023-10-30T14:13:57.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f350c8ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3508154380 con 0x7f3508153c80 2023-10-30T14:13:57.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f350659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35080c42a0 0x7f3508155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:57.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f350659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35080c42a0 0x7f3508155620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44600/0 (socket says 172.21.15.57:44600) 2023-10-30T14:13:57.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f350659c640 1 -- 172.21.15.57:0/2735558147 learned_addr learned my addr 172.21.15.57:0/2735558147 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:13:57.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f3506d9d640 1 --2- 172.21.15.57:0/2735558147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3508153c80 0x7f3508158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:57.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f3505d9b640 1 --2- 172.21.15.57:0/2735558147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35080c4a10 0x7f35080c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:57.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f350659c640 1 -- 172.21.15.57:0/2735558147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3508153c80 msgr2=0x7f3508158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:57.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f350659c640 1 --2- 172.21.15.57:0/2735558147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3508153c80 0x7f3508158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:57.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f350659c640 1 -- 172.21.15.57:0/2735558147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35080c4a10 msgr2=0x7f35080c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:57.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f350659c640 1 --2- 172.21.15.57:0/2735558147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35080c4a10 0x7f35080c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:57.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f350659c640 1 -- 172.21.15.57:0/2735558147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f8009420 con 0x7f35080c42a0 2023-10-30T14:13:57.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f3506d9d640 1 --2- 172.21.15.57:0/2735558147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3508153c80 0x7f3508158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:13:57.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f3505d9b640 1 --2- 172.21.15.57:0/2735558147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35080c4a10 0x7f35080c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:13:57.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f350659c640 1 --2- 172.21.15.57:0/2735558147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35080c42a0 0x7f3508155620 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f34f80b3040 tx=0x7f34f801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:57.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f34f6ffd640 1 -- 172.21.15.57:0/2735558147 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f80b0e40 con 0x7f35080c42a0 2023-10-30T14:13:57.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f34f6ffd640 1 -- 172.21.15.57:0/2735558147 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f80b9410 con 0x7f35080c42a0 2023-10-30T14:13:57.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f34f6ffd640 1 -- 172.21.15.57:0/2735558147 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f80b97c0 con 0x7f35080c42a0 2023-10-30T14:13:57.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f350c8ea640 1 -- 172.21.15.57:0/2735558147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3508154860 con 0x7f35080c42a0 2023-10-30T14:13:57.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f350c8ea640 1 -- 172.21.15.57:0/2735558147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3508154e20 con 0x7f35080c42a0 2023-10-30T14:13:57.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f350c8ea640 1 -- 172.21.15.57:0/2735558147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350804e170 con 0x7f35080c42a0 2023-10-30T14:13:57.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.920+0000 7f34f6ffd640 1 -- 172.21.15.57:0/2735558147 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f34f80b7040 con 0x7f35080c42a0 2023-10-30T14:13:57.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.924+0000 7f34f6ffd640 1 --2- 172.21.15.57:0/2735558147 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f34e80664a0 0x7f34e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:57.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.924+0000 7f34f6ffd640 1 -- 172.21.15.57:0/2735558147 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f34f8121930 con 0x7f35080c42a0 2023-10-30T14:13:57.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.924+0000 7f3505d9b640 1 --2- 172.21.15.57:0/2735558147 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f34e80664a0 0x7f34e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:57.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.924+0000 7f3505d9b640 1 --2- 172.21.15.57:0/2735558147 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f34e80664a0 0x7f34e8068960 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f350814f060 tx=0x7f34fc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:57.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:57.928+0000 7f34f6ffd640 1 -- 172.21.15.57:0/2735558147 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f34f80eefd0 con 0x7f35080c42a0 2023-10-30T14:13:58.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.084+0000 7f350c8ea640 1 -- 172.21.15.57:0/2735558147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35080cad40 con 0x7f35080c42a0 2023-10-30T14:13:58.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.084+0000 7f34f6ffd640 1 -- 172.21.15.57:0/2735558147 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12778 (secure 0 0 0) 0x7f34f80b5070 con 0x7f35080c42a0 2023-10-30T14:13:58.090 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:58.091 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":17,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:13:55.244786+0000","last_up_change":"2023-10-30T14:13:51.171287+0000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T14:13:52.574639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6801","nonce":3287367848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6803","nonce":3287367848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6807","nonce":3287367848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6805","nonce":3287367848}]},"public_addr":"172.21.15.57:6801/3287367848","cluster_addr":"172.21.15.57:6803/3287367848","heartbeat_back_addr":"172.21.15.57:6807/3287367848","heartbeat_front_addr":"172.21.15.57:6805/3287367848","state":["exists","up"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6809","nonce":3340797836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6811","nonce":3340797836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6815","nonce":3340797836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6813","nonce":3340797836}]},"public_addr":"172.21.15.57:6809/3340797836","cluster_addr":"172.21.15.57:6811/3340797836","heartbeat_back_addr":"172.21.15.57:6815/3340797836","heartbeat_front_addr":"172.21.15.57:6813/3340797836","state":["exists","up"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6825","nonce":1700292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6827","nonce":1700292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6831","nonce":1700292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6829","nonce":1700292}]},"public_addr":"172.21.15.57:6825/1700292","cluster_addr":"172.21.15.57:6827/1700292","heartbeat_back_addr":"172.21.15.57:6831/1700292","heartbeat_front_addr":"172.21.15.57:6829/1700292","state":["exists","up"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6817","nonce":2893420631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6819","nonce":2893420631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6823","nonce":2893420631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6821","nonce":2893420631}]},"public_addr":"172.21.15.57:6817/2893420631","cluster_addr":"172.21.15.57:6819/2893420631","heartbeat_back_addr":"172.21.15.57:6823/2893420631","heartbeat_front_addr":"172.21.15.57:6821/2893420631","state":["exists","up"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6807","nonce":2784816485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6811","nonce":2784816485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6815","nonce":2784816485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6813","nonce":2784816485}]},"public_addr":"172.21.15.88:6807/2784816485","cluster_addr":"172.21.15.88:6811/2784816485","heartbeat_back_addr":"172.21.15.88:6815/2784816485","heartbeat_front_addr":"172.21.15.88:6813/2784816485","state":["exists","up"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6801","nonce":2682449812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6803","nonce":2682449812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6810","nonce":2682449812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6806","nonce":2682449812}]},"public_addr":"172.21.15.88:6801/2682449812","cluster_addr":"172.21.15.88:6803/2682449812","heartbeat_back_addr":"172.21.15.88:6810/2682449812","heartbeat_front_addr":"172.21.15.88:6806/2682449812","state":["exists","up"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6822","nonce":3340693897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6827","nonce":3340693897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6831","nonce":3340693897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6829","nonce":3340693897}]},"public_addr":"172.21.15.88:6822/3340693897","cluster_addr":"172.21.15.88:6827/3340693897","heartbeat_back_addr":"172.21.15.88:6831/3340693897","heartbeat_front_addr":"172.21.15.88:6829/3340693897","state":["exists","up"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6817","nonce":2862111199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6820","nonce":2862111199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6825","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6826","nonce":2862111199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6821","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6823","nonce":2862111199}]},"public_addr":"172.21.15.88:6817/2862111199","cluster_addr":"172.21.15.88:6820/2862111199","heartbeat_back_addr":"172.21.15.88:6826/2862111199","heartbeat_front_addr":"172.21.15.88:6823/2862111199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.634920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.773993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.725825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.761452+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.694274+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.650718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:13:58.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.088+0000 7f350c8ea640 1 -- 172.21.15.57:0/2735558147 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f34e80664a0 msgr2=0x7f34e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:58.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.088+0000 7f350c8ea640 1 --2- 172.21.15.57:0/2735558147 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f34e80664a0 0x7f34e8068960 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f350814f060 tx=0x7f34fc009040 comp rx=0 tx=0).stop 2023-10-30T14:13:58.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.088+0000 7f350c8ea640 1 -- 172.21.15.57:0/2735558147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35080c42a0 msgr2=0x7f3508155620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:58.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.088+0000 7f350c8ea640 1 --2- 172.21.15.57:0/2735558147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35080c42a0 0x7f3508155620 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f34f80b3040 tx=0x7f34f801ba10 comp rx=0 tx=0).stop 2023-10-30T14:13:58.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.088+0000 7f350c8ea640 1 -- 172.21.15.57:0/2735558147 shutdown_connections 2023-10-30T14:13:58.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.088+0000 7f350c8ea640 1 --2- 172.21.15.57:0/2735558147 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f34e80664a0 0x7f34e8068960 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.088+0000 7f350c8ea640 1 --2- 172.21.15.57:0/2735558147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3508153c80 0x7f3508158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.088+0000 7f350c8ea640 1 --2- 172.21.15.57:0/2735558147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35080c4a10 0x7f35080c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.088+0000 7f350c8ea640 1 --2- 172.21.15.57:0/2735558147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35080c42a0 0x7f3508155620 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.088+0000 7f350c8ea640 1 -- 172.21.15.57:0/2735558147 >> 172.21.15.57:0/2735558147 conn(0x7f35080715c0 msgr2=0x7f35080719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:58.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.088+0000 7f350c8ea640 1 -- 172.21.15.57:0/2735558147 shutdown_connections 2023-10-30T14:13:58.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.088+0000 7f350c8ea640 1 -- 172.21.15.57:0/2735558147 wait complete. 2023-10-30T14:13:58.112 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T14:13:52.574639+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T14:13:58.112 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T14:13:58.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.252+0000 7fae66772640 1 Processor -- start 2023-10-30T14:13:58.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.252+0000 7fae66772640 1 -- start start 2023-10-30T14:13:58.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.256+0000 7fae66772640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fae600c1430 0x7fae60159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:58.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.256+0000 7fae66772640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae60157100 con 0x7fae6015a040 2023-10-30T14:13:58.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.256+0000 7fae66772640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae60150a10 con 0x7fae60150630 2023-10-30T14:13:58.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.256+0000 7fae66772640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae60072000 con 0x7fae600c1430 2023-10-30T14:13:58.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.256+0000 7fae64d0f640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fae6015a040 0x7fae600cacb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:45816/0 (socket says 172.21.15.57:45816) 2023-10-30T14:13:58.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.256+0000 7fae64d0f640 1 -- 172.21.15.57:0/989278837 learned_addr learned my addr 172.21.15.57:0/989278837 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:13:58.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.256+0000 7fae5f7fe640 1 --2- 172.21.15.57:0/989278837 >> v2:172.21.15.88:3300/0 conn(0x7fae600c1430 0x7fae60159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:58.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.256+0000 7fae5effd640 1 -- 172.21.15.57:0/989278837 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102644720 0 0) 0x7fae60157100 con 0x7fae6015a040 2023-10-30T14:13:58.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.256+0000 7fae5effd640 1 -- 172.21.15.57:0/989278837 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae48003530 con 0x7fae6015a040 2023-10-30T14:13:58.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.256+0000 7fae5f7fe640 1 -- 172.21.15.57:0/989278837 >> v1:172.21.15.88:6789/0 conn(0x7fae60150630 legacy=0x7fae600c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:58.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.256+0000 7fae5f7fe640 1 -- 172.21.15.57:0/989278837 >> v1:172.21.15.57:6789/0 conn(0x7fae6015a040 legacy=0x7fae600cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:58.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.256+0000 7fae5f7fe640 1 -- 172.21.15.57:0/989278837 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae60156510 con 0x7fae600c1430 2023-10-30T14:13:58.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.260+0000 7fae5f7fe640 1 --2- 172.21.15.57:0/989278837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae600c1430 0x7fae60159b00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fae540097c0 tx=0x7fae5401c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b328072d454a9da server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:58.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.260+0000 7fae5effd640 1 -- 172.21.15.57:0/989278837 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae540ab600 con 0x7fae600c1430 2023-10-30T14:13:58.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.260+0000 7fae5effd640 1 -- 172.21.15.57:0/989278837 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae540a7050 con 0x7fae600c1430 2023-10-30T14:13:58.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.260+0000 7fae5effd640 1 -- 172.21.15.57:0/989278837 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae540ab980 con 0x7fae600c1430 2023-10-30T14:13:58.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.260+0000 7fae66772640 1 -- 172.21.15.57:0/989278837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae600c1430 msgr2=0x7fae60159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:58.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.260+0000 7fae66772640 1 --2- 172.21.15.57:0/989278837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae600c1430 0x7fae60159b00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fae540097c0 tx=0x7fae5401c480 comp rx=0 tx=0).stop 2023-10-30T14:13:58.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.260+0000 7fae66772640 1 -- 172.21.15.57:0/989278837 shutdown_connections 2023-10-30T14:13:58.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.260+0000 7fae66772640 1 --2- 172.21.15.57:0/989278837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae600c1430 0x7fae60159b00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.260+0000 7fae66772640 1 -- 172.21.15.57:0/989278837 >> 172.21.15.57:0/989278837 conn(0x7fae6006da40 msgr2=0x7fae6006de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:58.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.260+0000 7fae66772640 1 -- 172.21.15.57:0/989278837 shutdown_connections 2023-10-30T14:13:58.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.260+0000 7fae66772640 1 -- 172.21.15.57:0/989278837 wait complete. 2023-10-30T14:13:58.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.260+0000 7fae66772640 1 Processor -- start 2023-10-30T14:13:58.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.260+0000 7fae66772640 1 -- start start 2023-10-30T14:13:58.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae66772640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae600c1430 0x7fae6014ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:58.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae66772640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fae60150630 0x7fae6014dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:58.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae66772640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae6015a040 0x7fae6014e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:58.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae66772640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6014ff40 con 0x7fae600c1430 2023-10-30T14:13:58.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae66772640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6014c670 con 0x7fae6015a040 2023-10-30T14:13:58.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae66772640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae60155d50 con 0x7fae60150630 2023-10-30T14:13:58.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae5ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae600c1430 0x7fae6014ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:58.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae5ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae600c1430 0x7fae6014ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44610/0 (socket says 172.21.15.57:44610) 2023-10-30T14:13:58.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae5ffff640 1 -- 172.21.15.57:0/2229158734 learned_addr learned my addr 172.21.15.57:0/2229158734 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:13:58.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae64d0f640 1 --2- 172.21.15.57:0/2229158734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae6015a040 0x7fae6014e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:58.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae5f7fe640 1 --2- 172.21.15.57:0/2229158734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fae60150630 0x7fae6014dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:58.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae5ffff640 1 -- 172.21.15.57:0/2229158734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fae60150630 msgr2=0x7fae6014dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:58.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae5ffff640 1 --2- 172.21.15.57:0/2229158734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fae60150630 0x7fae6014dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae5ffff640 1 -- 172.21.15.57:0/2229158734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae6015a040 msgr2=0x7fae6014e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:58.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae5ffff640 1 --2- 172.21.15.57:0/2229158734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae6015a040 0x7fae6014e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae5ffff640 1 -- 172.21.15.57:0/2229158734 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae54009420 con 0x7fae600c1430 2023-10-30T14:13:58.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae5f7fe640 1 --2- 172.21.15.57:0/2229158734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fae60150630 0x7fae6014dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:13:58.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae5ffff640 1 --2- 172.21.15.57:0/2229158734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae600c1430 0x7fae6014ab40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fae5000c530 tx=0x7fae5000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:58.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae5cff9640 1 -- 172.21.15.57:0/2229158734 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50005220 con 0x7fae600c1430 2023-10-30T14:13:58.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae5cff9640 1 -- 172.21.15.57:0/2229158734 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae50005380 con 0x7fae600c1430 2023-10-30T14:13:58.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae5cff9640 1 -- 172.21.15.57:0/2229158734 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50007270 con 0x7fae600c1430 2023-10-30T14:13:58.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae66772640 1 -- 172.21.15.57:0/2229158734 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae60155fa0 con 0x7fae600c1430 2023-10-30T14:13:58.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae66772640 1 -- 172.21.15.57:0/2229158734 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae6014f8f0 con 0x7fae600c1430 2023-10-30T14:13:58.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae66772640 1 -- 172.21.15.57:0/2229158734 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae6004e170 con 0x7fae600c1430 2023-10-30T14:13:58.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.264+0000 7fae5cff9640 1 -- 172.21.15.57:0/2229158734 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fae500073d0 con 0x7fae600c1430 2023-10-30T14:13:58.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.268+0000 7fae5cff9640 1 --2- 172.21.15.57:0/2229158734 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fae4c0664d0 0x7fae4c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:58.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.268+0000 7fae5cff9640 1 -- 172.21.15.57:0/2229158734 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fae500832d0 con 0x7fae600c1430 2023-10-30T14:13:58.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.268+0000 7fae5f7fe640 1 --2- 172.21.15.57:0/2229158734 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fae4c0664d0 0x7fae4c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:58.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.268+0000 7fae5f7fe640 1 --2- 172.21.15.57:0/2229158734 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fae4c0664d0 0x7fae4c068990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fae540ad000 tx=0x7fae54008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:58.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.272+0000 7fae5cff9640 1 -- 172.21.15.57:0/2229158734 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fae50050b30 con 0x7fae600c1430 2023-10-30T14:13:58.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.428+0000 7fae66772640 1 -- 172.21.15.57:0/2229158734 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fae600b0c80 con 0x7fae600c1430 2023-10-30T14:13:58.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.428+0000 7fae5cff9640 1 -- 172.21.15.57:0/2229158734 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7fae50054150 con 0x7fae600c1430 2023-10-30T14:13:58.433 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-30T14:13:58.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.428+0000 7fae66772640 1 -- 172.21.15.57:0/2229158734 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fae4c0664d0 msgr2=0x7fae4c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:58.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.428+0000 7fae66772640 1 --2- 172.21.15.57:0/2229158734 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fae4c0664d0 0x7fae4c068990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fae540ad000 tx=0x7fae54008040 comp rx=0 tx=0).stop 2023-10-30T14:13:58.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.428+0000 7fae66772640 1 -- 172.21.15.57:0/2229158734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae600c1430 msgr2=0x7fae6014ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:58.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.428+0000 7fae66772640 1 --2- 172.21.15.57:0/2229158734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae600c1430 0x7fae6014ab40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fae5000c530 tx=0x7fae5000c9f0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.428+0000 7fae66772640 1 -- 172.21.15.57:0/2229158734 shutdown_connections 2023-10-30T14:13:58.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.428+0000 7fae66772640 1 --2- 172.21.15.57:0/2229158734 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fae4c0664d0 0x7fae4c068990 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.428+0000 7fae66772640 1 --2- 172.21.15.57:0/2229158734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae6015a040 0x7fae6014e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.428+0000 7fae66772640 1 --2- 172.21.15.57:0/2229158734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fae60150630 0x7fae6014dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.428+0000 7fae66772640 1 --2- 172.21.15.57:0/2229158734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae600c1430 0x7fae6014ab40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.428+0000 7fae66772640 1 -- 172.21.15.57:0/2229158734 >> 172.21.15.57:0/2229158734 conn(0x7fae6006da40 msgr2=0x7fae600c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:58.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.428+0000 7fae66772640 1 -- 172.21.15.57:0/2229158734 shutdown_connections 2023-10-30T14:13:58.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.428+0000 7fae66772640 1 -- 172.21.15.57:0/2229158734 wait complete. 2023-10-30T14:13:58.453 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:13:58.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.592+0000 7f329c572640 1 Processor -- start 2023-10-30T14:13:58.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.592+0000 7f329c572640 1 -- start start 2023-10-30T14:13:58.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.592+0000 7f329c572640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3294150830 0x7f329406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:58.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.592+0000 7f329c572640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3294070520 0x7f32940709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:58.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.592+0000 7f329c572640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32940c2310 con 0x7f32940cb7d0 2023-10-30T14:13:58.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.592+0000 7f329c572640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3294156710 con 0x7f3294070520 2023-10-30T14:13:58.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.592+0000 7f329c572640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32940c4360 con 0x7f3294150830 2023-10-30T14:13:58.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.592+0000 7f329ab0f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3294070520 0x7f32940709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:58.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.592+0000 7f3299b0d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f32940cb7d0 0x7f32940c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:45826/0 (socket says 172.21.15.57:45826) 2023-10-30T14:13:58.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.592+0000 7f3299b0d640 1 -- 172.21.15.57:0/1381851051 learned_addr learned my addr 172.21.15.57:0/1381851051 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:13:58.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.592+0000 7f329a30e640 1 --2- 172.21.15.57:0/1381851051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3294150830 0x7f329406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:58.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.592+0000 7f329ab0f640 1 -- 172.21.15.57:0/1381851051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3294150830 msgr2=0x7f329406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:13:58.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.592+0000 7f329ab0f640 1 --2- 172.21.15.57:0/1381851051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3294150830 0x7f329406b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.592+0000 7f329ab0f640 1 -- 172.21.15.57:0/1381851051 >> v1:172.21.15.57:6789/0 conn(0x7f32940cb7d0 legacy=0x7f32940c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:58.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.592+0000 7f329ab0f640 1 -- 172.21.15.57:0/1381851051 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32941569b0 con 0x7f3294070520 2023-10-30T14:13:58.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.592+0000 7f329a30e640 1 --2- 172.21.15.57:0/1381851051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3294150830 0x7f329406b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:13:58.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.592+0000 7f329930c640 1 -- 172.21.15.57:0/1381851051 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610601841 0 0) 0x7f32940c2310 con 0x7f32940cb7d0 2023-10-30T14:13:58.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.596+0000 7f329ab0f640 1 --2- 172.21.15.57:0/1381851051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3294070520 0x7f32940709b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f328c009d90 tx=0x7f328c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570be4f9bf9d7e27 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:58.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.596+0000 7f329930c640 1 -- 172.21.15.57:0/1381851051 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c0af600 con 0x7f3294070520 2023-10-30T14:13:58.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.596+0000 7f329930c640 1 -- 172.21.15.57:0/1381851051 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328c0ab050 con 0x7f3294070520 2023-10-30T14:13:58.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.596+0000 7f329930c640 1 -- 172.21.15.57:0/1381851051 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c0af980 con 0x7f3294070520 2023-10-30T14:13:58.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.596+0000 7f329c572640 1 -- 172.21.15.57:0/1381851051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3294070520 msgr2=0x7f32940709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:58.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.596+0000 7f329c572640 1 --2- 172.21.15.57:0/1381851051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3294070520 0x7f32940709b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f328c009d90 tx=0x7f328c01f130 comp rx=0 tx=0).stop 2023-10-30T14:13:58.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.596+0000 7f329c572640 1 -- 172.21.15.57:0/1381851051 shutdown_connections 2023-10-30T14:13:58.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.596+0000 7f329c572640 1 --2- 172.21.15.57:0/1381851051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3294070520 0x7f32940709b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.596+0000 7f329c572640 1 --2- 172.21.15.57:0/1381851051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3294150830 0x7f329406b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.596+0000 7f329c572640 1 -- 172.21.15.57:0/1381851051 >> 172.21.15.57:0/1381851051 conn(0x7f32940cac70 msgr2=0x7f32940cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:58.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.596+0000 7f329c572640 1 -- 172.21.15.57:0/1381851051 shutdown_connections 2023-10-30T14:13:58.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.596+0000 7f329c572640 1 -- 172.21.15.57:0/1381851051 wait complete. 2023-10-30T14:13:58.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329c572640 1 Processor -- start 2023-10-30T14:13:58.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329c572640 1 -- start start 2023-10-30T14:13:58.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329c572640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3294070520 0x7f32941eea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:58.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329c572640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32940cb7d0 0x7f32940ca280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:58.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329c572640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3294150830 0x7f32941445a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:58.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329c572640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32941474f0 con 0x7f3294070520 2023-10-30T14:13:58.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329c572640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3294146eb0 con 0x7f32940cb7d0 2023-10-30T14:13:58.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329c572640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32941500f0 con 0x7f3294150830 2023-10-30T14:13:58.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329a30e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3294070520 0x7f32941eea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:58.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329a30e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3294070520 0x7f32941eea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44638/0 (socket says 172.21.15.57:44638) 2023-10-30T14:13:58.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329a30e640 1 -- 172.21.15.57:0/1913060217 learned_addr learned my addr 172.21.15.57:0/1913060217 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:13:58.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329ab0f640 1 --2- 172.21.15.57:0/1913060217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3294150830 0x7f32941445a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:58.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f3299b0d640 1 --2- 172.21.15.57:0/1913060217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32940cb7d0 0x7f32940ca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:58.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329a30e640 1 -- 172.21.15.57:0/1913060217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3294150830 msgr2=0x7f32941445a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:58.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329a30e640 1 --2- 172.21.15.57:0/1913060217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3294150830 0x7f32941445a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329a30e640 1 -- 172.21.15.57:0/1913060217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32940cb7d0 msgr2=0x7f32940ca280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:58.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329a30e640 1 --2- 172.21.15.57:0/1913060217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32940cb7d0 0x7f32940ca280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329a30e640 1 -- 172.21.15.57:0/1913060217 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c009a70 con 0x7f3294070520 2023-10-30T14:13:58.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f3299b0d640 1 --2- 172.21.15.57:0/1913060217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32940cb7d0 0x7f32940ca280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:13:58.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329a30e640 1 --2- 172.21.15.57:0/1913060217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3294070520 0x7f32941eea80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f329000d830 tx=0x7f329000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:58.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f328affd640 1 -- 172.21.15.57:0/1913060217 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32900096c0 con 0x7f3294070520 2023-10-30T14:13:58.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f328affd640 1 -- 172.21.15.57:0/1913060217 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3290009820 con 0x7f3294070520 2023-10-30T14:13:58.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329c572640 1 -- 172.21.15.57:0/1913060217 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f329414fbc0 con 0x7f3294070520 2023-10-30T14:13:58.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f328affd640 1 -- 172.21.15.57:0/1913060217 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3290009bd0 con 0x7f3294070520 2023-10-30T14:13:58.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.600+0000 7f329c572640 1 -- 172.21.15.57:0/1913060217 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32941f2190 con 0x7f3294070520 2023-10-30T14:13:58.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.604+0000 7f329c572640 1 -- 172.21.15.57:0/1913060217 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32941f1ff0 con 0x7f3294070520 2023-10-30T14:13:58.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.604+0000 7f328affd640 1 -- 172.21.15.57:0/1913060217 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3290009d30 con 0x7f3294070520 2023-10-30T14:13:58.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.604+0000 7f328affd640 1 --2- 172.21.15.57:0/1913060217 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32780664d0 0x7f3278068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:13:58.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.604+0000 7f328affd640 1 -- 172.21.15.57:0/1913060217 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f32900842a0 con 0x7f3294070520 2023-10-30T14:13:58.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.604+0000 7f3299b0d640 1 --2- 172.21.15.57:0/1913060217 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32780664d0 0x7f3278068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:13:58.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.608+0000 7f3299b0d640 1 --2- 172.21.15.57:0/1913060217 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32780664d0 0x7f3278068990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3284007fd0 tx=0x7f3284007c20 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:13:58.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.608+0000 7f328affd640 1 -- 172.21.15.57:0/1913060217 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f32941f1ff0 con 0x7f3294070520 2023-10-30T14:13:58.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.800+0000 7f329c572640 1 -- 172.21.15.57:0/1913060217 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32941f1ff0 con 0x7f3294070520 2023-10-30T14:13:58.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.800+0000 7f328affd640 1 -- 172.21.15.57:0/1913060217 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7f32941f1ff0 con 0x7f3294070520 2023-10-30T14:13:58.806 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:13:58.806 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-10-30T14:13:58.806 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 2 2023-10-30T14:13:58.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.800+0000 7f329c572640 1 -- 172.21.15.57:0/1913060217 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32780664d0 msgr2=0x7f3278068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:58.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.800+0000 7f329c572640 1 --2- 172.21.15.57:0/1913060217 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32780664d0 0x7f3278068990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3284007fd0 tx=0x7f3284007c20 comp rx=0 tx=0).stop 2023-10-30T14:13:58.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.800+0000 7f329c572640 1 -- 172.21.15.57:0/1913060217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3294070520 msgr2=0x7f32941eea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:13:58.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.804+0000 7f329c572640 1 --2- 172.21.15.57:0/1913060217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3294070520 0x7f32941eea80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f329000d830 tx=0x7f329000dcf0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.804+0000 7f329c572640 1 -- 172.21.15.57:0/1913060217 shutdown_connections 2023-10-30T14:13:58.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.804+0000 7f329c572640 1 --2- 172.21.15.57:0/1913060217 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32780664d0 0x7f3278068990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.804+0000 7f329c572640 1 --2- 172.21.15.57:0/1913060217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3294150830 0x7f32941445a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.804+0000 7f329c572640 1 --2- 172.21.15.57:0/1913060217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32940cb7d0 0x7f32940ca280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.804+0000 7f329c572640 1 --2- 172.21.15.57:0/1913060217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3294070520 0x7f32941eea80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:13:58.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.804+0000 7f329c572640 1 -- 172.21.15.57:0/1913060217 >> 172.21.15.57:0/1913060217 conn(0x7f32940cac70 msgr2=0x7f329414cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:13:58.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.804+0000 7f329c572640 1 -- 172.21.15.57:0/1913060217 shutdown_connections 2023-10-30T14:13:58.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:13:58.804+0000 7f329c572640 1 -- 172.21.15.57:0/1913060217 wait complete. 2023-10-30T14:13:59.389 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:13:59.396 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:14:00.827 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:00.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.976+0000 7f3f25009640 1 Processor -- start 2023-10-30T14:14:00.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.976+0000 7f3f25009640 1 -- start start 2023-10-30T14:14:00.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.976+0000 7f3f25009640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3f200cb7d0 0x7f3f200c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:00.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.976+0000 7f3f25009640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f200c2310 con 0x7f3f20150830 2023-10-30T14:14:00.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.976+0000 7f3f25009640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f20156710 con 0x7f3f200c3e40 2023-10-30T14:14:00.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.976+0000 7f3f25009640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f2006bc10 con 0x7f3f200cb7d0 2023-10-30T14:14:00.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.976+0000 7f3f1ed9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f3f20150830 0x7f3f2006b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:45830/0 (socket says 172.21.15.57:45830) 2023-10-30T14:14:00.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.976+0000 7f3f1ed9d640 1 -- 172.21.15.57:0/4135943463 learned_addr learned my addr 172.21.15.57:0/4135943463 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:00.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.976+0000 7f3f17fff640 1 --2- 172.21.15.57:0/4135943463 >> v2:172.21.15.88:3300/0 conn(0x7f3f200cb7d0 0x7f3f200c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:00.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1e59c640 1 -- 172.21.15.57:0/4135943463 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963585653 0 0) 0x7f3f200c2310 con 0x7f3f20150830 2023-10-30T14:14:00.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1e59c640 1 -- 172.21.15.57:0/4135943463 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f0c005740 con 0x7f3f20150830 2023-10-30T14:14:00.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1e59c640 1 -- 172.21.15.57:0/4135943463 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1333529664 0 0) 0x7f3f0c005740 con 0x7f3f20150830 2023-10-30T14:14:00.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1e59c640 1 -- 172.21.15.57:0/4135943463 >> v2:172.21.15.88:3300/0 conn(0x7f3f200cb7d0 msgr2=0x7f3f200c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:00.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1e59c640 1 --2- 172.21.15.57:0/4135943463 >> v2:172.21.15.88:3300/0 conn(0x7f3f200cb7d0 0x7f3f200c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:00.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1e59c640 1 -- 172.21.15.57:0/4135943463 >> v1:172.21.15.88:6789/0 conn(0x7f3f200c3e40 legacy=0x7f3f20070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:00.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1e59c640 1 -- 172.21.15.57:0/4135943463 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f200c4360 con 0x7f3f20150830 2023-10-30T14:14:00.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1e59c640 1 -- 172.21.15.57:0/4135943463 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f3f100016a0 con 0x7f3f20150830 2023-10-30T14:14:00.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1e59c640 1 -- 172.21.15.57:0/4135943463 >> v1:172.21.15.57:6789/0 conn(0x7f3f20150830 legacy=0x7f3f2006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:00.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1e59c640 1 --2- 172.21.15.57:0/4135943463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f0c005b90 0x7f3f0c005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:00.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1e59c640 1 --2- 172.21.15.57:0/4135943463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f0c0069e0 0x7f3f0c006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:00.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1e59c640 1 --2- 172.21.15.57:0/4135943463 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3f0c0073a0 0x7f3f0c00dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:00.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1e59c640 1 -- 172.21.15.57:0/4135943463 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2006bc10 con 0x7f3f0c005b90 2023-10-30T14:14:00.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1e59c640 1 -- 172.21.15.57:0/4135943463 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c00e250 con 0x7f3f0c0069e0 2023-10-30T14:14:00.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1e59c640 1 -- 172.21.15.57:0/4135943463 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c00e410 con 0x7f3f0c0073a0 2023-10-30T14:14:00.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1ed9d640 1 --2- 172.21.15.57:0/4135943463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f0c005b90 0x7f3f0c005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:00.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f17fff640 1 --2- 172.21.15.57:0/4135943463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f0c0069e0 0x7f3f0c006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:00.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1f59e640 1 --2- 172.21.15.57:0/4135943463 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3f0c0073a0 0x7f3f0c00dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:00.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1ed9d640 1 -- 172.21.15.57:0/4135943463 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3f0c0073a0 msgr2=0x7f3f0c00dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:00.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1ed9d640 1 --2- 172.21.15.57:0/4135943463 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3f0c0073a0 0x7f3f0c00dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:00.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1ed9d640 1 -- 172.21.15.57:0/4135943463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f0c0069e0 msgr2=0x7f3f0c006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:00.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1ed9d640 1 --2- 172.21.15.57:0/4135943463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f0c0069e0 0x7f3f0c006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:00.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1ed9d640 1 -- 172.21.15.57:0/4135943463 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0c005740 con 0x7f3f0c005b90 2023-10-30T14:14:00.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.980+0000 7f3f1f59e640 1 -- 172.21.15.57:0/4135943463 reap_dead start 2023-10-30T14:14:00.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.984+0000 7f3f1ed9d640 1 --2- 172.21.15.57:0/4135943463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f0c005b90 0x7f3f0c005fb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3f200c24a0 tx=0x7f3f1001f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da63772506bbd032 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:00.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.984+0000 7f3f1e59c640 1 -- 172.21.15.57:0/4135943463 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f10005ae0 con 0x7f3f0c005b90 2023-10-30T14:14:00.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.984+0000 7f3f1e59c640 1 -- 172.21.15.57:0/4135943463 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f100ab070 con 0x7f3f0c005b90 2023-10-30T14:14:00.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.984+0000 7f3f1e59c640 1 -- 172.21.15.57:0/4135943463 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f10005e80 con 0x7f3f0c005b90 2023-10-30T14:14:00.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.984+0000 7f3f25009640 1 -- 172.21.15.57:0/4135943463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f0c005b90 msgr2=0x7f3f0c005fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:00.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.984+0000 7f3f25009640 1 --2- 172.21.15.57:0/4135943463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f0c005b90 0x7f3f0c005fb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3f200c24a0 tx=0x7f3f1001f330 comp rx=0 tx=0).stop 2023-10-30T14:14:00.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.984+0000 7f3f25009640 1 -- 172.21.15.57:0/4135943463 shutdown_connections 2023-10-30T14:14:00.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.984+0000 7f3f25009640 1 --2- 172.21.15.57:0/4135943463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f0c005b90 0x7f3f0c005fb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:00.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.984+0000 7f3f25009640 1 -- 172.21.15.57:0/4135943463 >> 172.21.15.57:0/4135943463 conn(0x7f3f200cac70 msgr2=0x7f3f200cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:00.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.984+0000 7f3f25009640 1 -- 172.21.15.57:0/4135943463 shutdown_connections 2023-10-30T14:14:00.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.984+0000 7f3f25009640 1 -- 172.21.15.57:0/4135943463 wait complete. 2023-10-30T14:14:00.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.984+0000 7f3f25009640 1 Processor -- start 2023-10-30T14:14:00.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.984+0000 7f3f25009640 1 -- start start 2023-10-30T14:14:00.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f25009640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3f0c005b90 0x7f3f20071570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:00.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f25009640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f20155350 0x7f3f20145990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:00.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f25009640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f20155eb0 0x7f3f201475b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:00.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f25009640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f200c2b40 con 0x7f3f20155350 2023-10-30T14:14:00.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f25009640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2014b6b0 con 0x7f3f20155eb0 2023-10-30T14:14:00.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f25009640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f201482c0 con 0x7f3f0c005b90 2023-10-30T14:14:00.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f17fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f20155350 0x7f3f20145990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:00.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f17fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f20155350 0x7f3f20145990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44658/0 (socket says 172.21.15.57:44658) 2023-10-30T14:14:00.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f17fff640 1 -- 172.21.15.57:0/3974715933 learned_addr learned my addr 172.21.15.57:0/3974715933 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:00.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f1f59e640 1 --2- 172.21.15.57:0/3974715933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f20155eb0 0x7f3f201475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:00.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f1ed9d640 1 --2- 172.21.15.57:0/3974715933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3f0c005b90 0x7f3f20071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:00.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f17fff640 1 -- 172.21.15.57:0/3974715933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3f0c005b90 msgr2=0x7f3f20071570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:00.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f17fff640 1 --2- 172.21.15.57:0/3974715933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3f0c005b90 0x7f3f20071570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:00.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f17fff640 1 -- 172.21.15.57:0/3974715933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f20155eb0 msgr2=0x7f3f201475b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:00.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f17fff640 1 --2- 172.21.15.57:0/3974715933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f20155eb0 0x7f3f201475b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:00.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f17fff640 1 -- 172.21.15.57:0/3974715933 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f201485a0 con 0x7f3f20155350 2023-10-30T14:14:00.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f17fff640 1 --2- 172.21.15.57:0/3974715933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f20155350 0x7f3f20145990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3f08011930 tx=0x7f3f08011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:00.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f177fe640 1 -- 172.21.15.57:0/3974715933 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f080095b0 con 0x7f3f20155350 2023-10-30T14:14:00.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f177fe640 1 -- 172.21.15.57:0/3974715933 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f08004240 con 0x7f3f20155350 2023-10-30T14:14:00.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f177fe640 1 -- 172.21.15.57:0/3974715933 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0800f8e0 con 0x7f3f20155350 2023-10-30T14:14:01.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f25009640 1 -- 172.21.15.57:0/3974715933 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f20148880 con 0x7f3f20155350 2023-10-30T14:14:01.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f25009640 1 -- 172.21.15.57:0/3974715933 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2014b920 con 0x7f3f20155350 2023-10-30T14:14:01.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f25009640 1 -- 172.21.15.57:0/3974715933 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f2004e170 con 0x7f3f20155350 2023-10-30T14:14:01.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.988+0000 7f3f177fe640 1 -- 172.21.15.57:0/3974715933 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3f0800fa40 con 0x7f3f20155350 2023-10-30T14:14:01.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.992+0000 7f3f177fe640 1 --2- 172.21.15.57:0/3974715933 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3f000664a0 0x7f3f00068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:01.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.992+0000 7f3f177fe640 1 -- 172.21.15.57:0/3974715933 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3f08084220 con 0x7f3f20155350 2023-10-30T14:14:01.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.992+0000 7f3f1ed9d640 1 --2- 172.21.15.57:0/3974715933 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3f000664a0 0x7f3f00068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:01.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.992+0000 7f3f1ed9d640 1 --2- 172.21.15.57:0/3974715933 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3f000664a0 0x7f3f00068960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3f100af000 tx=0x7f3f100aa000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:01.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:00.996+0000 7f3f177fe640 1 -- 172.21.15.57:0/3974715933 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3f080518c0 con 0x7f3f20155350 2023-10-30T14:14:01.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.184+0000 7f3f25009640 1 -- 172.21.15.57:0/3974715933 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f20156f40 con 0x7f3f20155350 2023-10-30T14:14:01.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.184+0000 7f3f177fe640 1 -- 172.21.15.57:0/3974715933 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5271 (secure 0 0 0) 0x7f3f08019090 con 0x7f3f20155350 2023-10-30T14:14:01.189 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:01.189 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4265,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/901968115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":901968115},{"type":"v1","addr":"172.21.15.57:6837","nonce":901968115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4282,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-10-30T14:14:01.189 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 3 2023-10-30T14:14:01.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.184+0000 7f3f25009640 1 -- 172.21.15.57:0/3974715933 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3f000664a0 msgr2=0x7f3f00068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:01.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.184+0000 7f3f25009640 1 --2- 172.21.15.57:0/3974715933 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3f000664a0 0x7f3f00068960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3f100af000 tx=0x7f3f100aa000 comp rx=0 tx=0).stop 2023-10-30T14:14:01.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.184+0000 7f3f25009640 1 -- 172.21.15.57:0/3974715933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f20155350 msgr2=0x7f3f20145990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:01.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.184+0000 7f3f25009640 1 --2- 172.21.15.57:0/3974715933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f20155350 0x7f3f20145990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3f08011930 tx=0x7f3f08011df0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.184+0000 7f3f25009640 1 -- 172.21.15.57:0/3974715933 shutdown_connections 2023-10-30T14:14:01.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.184+0000 7f3f25009640 1 --2- 172.21.15.57:0/3974715933 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3f000664a0 0x7f3f00068960 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.184+0000 7f3f25009640 1 --2- 172.21.15.57:0/3974715933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f20155eb0 0x7f3f201475b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.184+0000 7f3f25009640 1 --2- 172.21.15.57:0/3974715933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f20155350 0x7f3f20145990 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.184+0000 7f3f25009640 1 --2- 172.21.15.57:0/3974715933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3f0c005b90 0x7f3f20071570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.184+0000 7f3f25009640 1 -- 172.21.15.57:0/3974715933 >> 172.21.15.57:0/3974715933 conn(0x7f3f200cac70 msgr2=0x7f3f2014d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:01.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.184+0000 7f3f25009640 1 -- 172.21.15.57:0/3974715933 shutdown_connections 2023-10-30T14:14:01.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.184+0000 7f3f25009640 1 -- 172.21.15.57:0/3974715933 wait complete. 2023-10-30T14:14:01.210 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:14:01.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.348+0000 7fe357246640 1 Processor -- start 2023-10-30T14:14:01.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.348+0000 7fe357246640 1 -- start start 2023-10-30T14:14:01.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.348+0000 7fe357246640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe350153c80 0x7fe35015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:01.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.348+0000 7fe357246640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3500c4a10 0x7fe3500c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:01.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.348+0000 7fe357246640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe350159f20 con 0x7fe35015a3b0 2023-10-30T14:14:01.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.348+0000 7fe357246640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe35015dec0 con 0x7fe350153c80 2023-10-30T14:14:01.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.348+0000 7fe357246640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3501599a0 con 0x7fe3500c4a10 2023-10-30T14:14:01.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.348+0000 7fe3557e3640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fe35015a3b0 0x7fe3500ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:45846/0 (socket says 172.21.15.57:45846) 2023-10-30T14:14:01.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.348+0000 7fe3557e3640 1 -- 172.21.15.57:0/915417287 learned_addr learned my addr 172.21.15.57:0/915417287 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:01.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.348+0000 7fe34ffff640 1 --2- 172.21.15.57:0/915417287 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3500c4a10 0x7fe3500c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:01.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.348+0000 7fe354fe2640 1 --2- 172.21.15.57:0/915417287 >> v2:172.21.15.88:3300/0 conn(0x7fe350153c80 0x7fe35015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:01.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.348+0000 7fe34f7fe640 1 -- 172.21.15.57:0/915417287 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701490315 0 0) 0x7fe350159f20 con 0x7fe35015a3b0 2023-10-30T14:14:01.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.348+0000 7fe34f7fe640 1 -- 172.21.15.57:0/915417287 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe340003530 con 0x7fe35015a3b0 2023-10-30T14:14:01.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.348+0000 7fe34ffff640 1 -- 172.21.15.57:0/915417287 >> v2:172.21.15.88:3300/0 conn(0x7fe350153c80 msgr2=0x7fe35015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:01.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.348+0000 7fe34ffff640 1 --2- 172.21.15.57:0/915417287 >> v2:172.21.15.88:3300/0 conn(0x7fe350153c80 0x7fe35015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.348+0000 7fe34ffff640 1 -- 172.21.15.57:0/915417287 >> v1:172.21.15.57:6789/0 conn(0x7fe35015a3b0 legacy=0x7fe3500ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:01.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.348+0000 7fe34ffff640 1 -- 172.21.15.57:0/915417287 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3500c47e0 con 0x7fe3500c4a10 2023-10-30T14:14:01.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.352+0000 7fe34ffff640 1 --2- 172.21.15.57:0/915417287 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3500c4a10 0x7fe3500c42a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe33c0097c0 tx=0x7fe33c01ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c389bf7402e31a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:01.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.352+0000 7fe34f7fe640 1 -- 172.21.15.57:0/915417287 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c0a8970 con 0x7fe3500c4a10 2023-10-30T14:14:01.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.352+0000 7fe34f7fe640 1 -- 172.21.15.57:0/915417287 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33c0a8ad0 con 0x7fe3500c4a10 2023-10-30T14:14:01.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.352+0000 7fe34f7fe640 1 -- 172.21.15.57:0/915417287 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c0a8e40 con 0x7fe3500c4a10 2023-10-30T14:14:01.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.352+0000 7fe357246640 1 -- 172.21.15.57:0/915417287 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3500c4a10 msgr2=0x7fe3500c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:01.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.352+0000 7fe357246640 1 --2- 172.21.15.57:0/915417287 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3500c4a10 0x7fe3500c42a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe33c0097c0 tx=0x7fe33c01ec70 comp rx=0 tx=0).stop 2023-10-30T14:14:01.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.352+0000 7fe357246640 1 -- 172.21.15.57:0/915417287 shutdown_connections 2023-10-30T14:14:01.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.352+0000 7fe357246640 1 --2- 172.21.15.57:0/915417287 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3500c4a10 0x7fe3500c42a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.352+0000 7fe357246640 1 --2- 172.21.15.57:0/915417287 >> v2:172.21.15.88:3300/0 conn(0x7fe350153c80 0x7fe35015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.352+0000 7fe357246640 1 -- 172.21.15.57:0/915417287 >> 172.21.15.57:0/915417287 conn(0x7fe3500715c0 msgr2=0x7fe3500719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:01.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.352+0000 7fe357246640 1 -- 172.21.15.57:0/915417287 shutdown_connections 2023-10-30T14:14:01.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.352+0000 7fe357246640 1 -- 172.21.15.57:0/915417287 wait complete. 2023-10-30T14:14:01.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe357246640 1 Processor -- start 2023-10-30T14:14:01.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe357246640 1 -- start start 2023-10-30T14:14:01.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe357246640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3500c4a10 0x7fe3501f24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:01.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe357246640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe350153c80 0x7fe3500cea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:01.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe357246640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe35015a3b0 0x7fe35014d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:01.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe357246640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35014ad40 con 0x7fe350153c80 2023-10-30T14:14:01.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe357246640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35006c850 con 0x7fe35015a3b0 2023-10-30T14:14:01.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe357246640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe350150d40 con 0x7fe3500c4a10 2023-10-30T14:14:01.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe34ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe350153c80 0x7fe3500cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:01.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe34ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe350153c80 0x7fe3500cea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44662/0 (socket says 172.21.15.57:44662) 2023-10-30T14:14:01.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe34ffff640 1 -- 172.21.15.57:0/3334840823 learned_addr learned my addr 172.21.15.57:0/3334840823 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:01.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe354fe2640 1 --2- 172.21.15.57:0/3334840823 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3500c4a10 0x7fe3501f24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:01.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe3557e3640 1 --2- 172.21.15.57:0/3334840823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe35015a3b0 0x7fe35014d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:01.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe34ffff640 1 -- 172.21.15.57:0/3334840823 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3500c4a10 msgr2=0x7fe3501f24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:01.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe34ffff640 1 --2- 172.21.15.57:0/3334840823 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3500c4a10 0x7fe3501f24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe34ffff640 1 -- 172.21.15.57:0/3334840823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe35015a3b0 msgr2=0x7fe35014d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:01.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe34ffff640 1 --2- 172.21.15.57:0/3334840823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe35015a3b0 0x7fe35014d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe34ffff640 1 -- 172.21.15.57:0/3334840823 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33c009420 con 0x7fe350153c80 2023-10-30T14:14:01.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe34ffff640 1 --2- 172.21.15.57:0/3334840823 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe350153c80 0x7fe3500cea50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe33c009710 tx=0x7fe33c01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:01.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe34d7fa640 1 -- 172.21.15.57:0/3334840823 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c0b1d70 con 0x7fe350153c80 2023-10-30T14:14:01.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe34d7fa640 1 -- 172.21.15.57:0/3334840823 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33c0b9410 con 0x7fe350153c80 2023-10-30T14:14:01.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe34d7fa640 1 -- 172.21.15.57:0/3334840823 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c0b9780 con 0x7fe350153c80 2023-10-30T14:14:01.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe357246640 1 -- 172.21.15.57:0/3334840823 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35014b8c0 con 0x7fe350153c80 2023-10-30T14:14:01.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.356+0000 7fe357246640 1 -- 172.21.15.57:0/3334840823 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe35014fe50 con 0x7fe350153c80 2023-10-30T14:14:01.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.360+0000 7fe357246640 1 -- 172.21.15.57:0/3334840823 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe350150e80 con 0x7fe350153c80 2023-10-30T14:14:01.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.360+0000 7fe34d7fa640 1 -- 172.21.15.57:0/3334840823 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe33c0b7040 con 0x7fe350153c80 2023-10-30T14:14:01.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.360+0000 7fe34d7fa640 1 --2- 172.21.15.57:0/3334840823 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3340664a0 0x7fe334068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:01.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.360+0000 7fe34d7fa640 1 -- 172.21.15.57:0/3334840823 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe33c121670 con 0x7fe350153c80 2023-10-30T14:14:01.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.360+0000 7fe354fe2640 1 --2- 172.21.15.57:0/3334840823 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3340664a0 0x7fe334068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:01.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.360+0000 7fe354fe2640 1 --2- 172.21.15.57:0/3334840823 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3340664a0 0x7fe334068960 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe344009b60 tx=0x7fe344009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:01.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.364+0000 7fe34d7fa640 1 -- 172.21.15.57:0/3334840823 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe350150e80 con 0x7fe350153c80 2023-10-30T14:14:01.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.520+0000 7fe357246640 1 -- 172.21.15.57:0/3334840823 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe350150e80 con 0x7fe350153c80 2023-10-30T14:14:01.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.520+0000 7fe34d7fa640 1 -- 172.21.15.57:0/3334840823 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12778 (secure 0 0 0) 0x7fe350150e80 con 0x7fe350153c80 2023-10-30T14:14:01.527 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:01.527 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":17,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:13:55.244786+0000","last_up_change":"2023-10-30T14:13:51.171287+0000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T14:13:52.574639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6801","nonce":3287367848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6803","nonce":3287367848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6807","nonce":3287367848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6805","nonce":3287367848}]},"public_addr":"172.21.15.57:6801/3287367848","cluster_addr":"172.21.15.57:6803/3287367848","heartbeat_back_addr":"172.21.15.57:6807/3287367848","heartbeat_front_addr":"172.21.15.57:6805/3287367848","state":["exists","up"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6809","nonce":3340797836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6811","nonce":3340797836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6815","nonce":3340797836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6813","nonce":3340797836}]},"public_addr":"172.21.15.57:6809/3340797836","cluster_addr":"172.21.15.57:6811/3340797836","heartbeat_back_addr":"172.21.15.57:6815/3340797836","heartbeat_front_addr":"172.21.15.57:6813/3340797836","state":["exists","up"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6825","nonce":1700292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6827","nonce":1700292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6831","nonce":1700292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6829","nonce":1700292}]},"public_addr":"172.21.15.57:6825/1700292","cluster_addr":"172.21.15.57:6827/1700292","heartbeat_back_addr":"172.21.15.57:6831/1700292","heartbeat_front_addr":"172.21.15.57:6829/1700292","state":["exists","up"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6817","nonce":2893420631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6819","nonce":2893420631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6823","nonce":2893420631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6821","nonce":2893420631}]},"public_addr":"172.21.15.57:6817/2893420631","cluster_addr":"172.21.15.57:6819/2893420631","heartbeat_back_addr":"172.21.15.57:6823/2893420631","heartbeat_front_addr":"172.21.15.57:6821/2893420631","state":["exists","up"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6807","nonce":2784816485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6811","nonce":2784816485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6815","nonce":2784816485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6813","nonce":2784816485}]},"public_addr":"172.21.15.88:6807/2784816485","cluster_addr":"172.21.15.88:6811/2784816485","heartbeat_back_addr":"172.21.15.88:6815/2784816485","heartbeat_front_addr":"172.21.15.88:6813/2784816485","state":["exists","up"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6801","nonce":2682449812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6803","nonce":2682449812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6810","nonce":2682449812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6806","nonce":2682449812}]},"public_addr":"172.21.15.88:6801/2682449812","cluster_addr":"172.21.15.88:6803/2682449812","heartbeat_back_addr":"172.21.15.88:6810/2682449812","heartbeat_front_addr":"172.21.15.88:6806/2682449812","state":["exists","up"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6822","nonce":3340693897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6827","nonce":3340693897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6831","nonce":3340693897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6829","nonce":3340693897}]},"public_addr":"172.21.15.88:6822/3340693897","cluster_addr":"172.21.15.88:6827/3340693897","heartbeat_back_addr":"172.21.15.88:6831/3340693897","heartbeat_front_addr":"172.21.15.88:6829/3340693897","state":["exists","up"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6817","nonce":2862111199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6820","nonce":2862111199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6825","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6826","nonce":2862111199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6821","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6823","nonce":2862111199}]},"public_addr":"172.21.15.88:6817/2862111199","cluster_addr":"172.21.15.88:6820/2862111199","heartbeat_back_addr":"172.21.15.88:6826/2862111199","heartbeat_front_addr":"172.21.15.88:6823/2862111199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.634920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.773993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.725825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.761452+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.694274+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.650718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:14:01.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.524+0000 7fe357246640 1 -- 172.21.15.57:0/3334840823 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3340664a0 msgr2=0x7fe334068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:01.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.524+0000 7fe357246640 1 --2- 172.21.15.57:0/3334840823 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3340664a0 0x7fe334068960 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe344009b60 tx=0x7fe344009190 comp rx=0 tx=0).stop 2023-10-30T14:14:01.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.524+0000 7fe357246640 1 -- 172.21.15.57:0/3334840823 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe350153c80 msgr2=0x7fe3500cea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:01.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.524+0000 7fe357246640 1 --2- 172.21.15.57:0/3334840823 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe350153c80 0x7fe3500cea50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe33c009710 tx=0x7fe33c01ba10 comp rx=0 tx=0).stop 2023-10-30T14:14:01.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.524+0000 7fe357246640 1 -- 172.21.15.57:0/3334840823 shutdown_connections 2023-10-30T14:14:01.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.524+0000 7fe357246640 1 --2- 172.21.15.57:0/3334840823 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3340664a0 0x7fe334068960 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.524+0000 7fe357246640 1 --2- 172.21.15.57:0/3334840823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe35015a3b0 0x7fe35014d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.524+0000 7fe357246640 1 --2- 172.21.15.57:0/3334840823 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe350153c80 0x7fe3500cea50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.524+0000 7fe357246640 1 --2- 172.21.15.57:0/3334840823 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3500c4a10 0x7fe3501f24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.524+0000 7fe357246640 1 -- 172.21.15.57:0/3334840823 >> 172.21.15.57:0/3334840823 conn(0x7fe3500715c0 msgr2=0x7fe3500719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:01.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.524+0000 7fe357246640 1 -- 172.21.15.57:0/3334840823 shutdown_connections 2023-10-30T14:14:01.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.524+0000 7fe357246640 1 -- 172.21.15.57:0/3334840823 wait complete. 2023-10-30T14:14:01.548 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T14:13:52.574639+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T14:14:01.549 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T14:14:01.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.688+0000 7fdc3d26b640 1 Processor -- start 2023-10-30T14:14:01.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.692+0000 7fdc3d26b640 1 -- start start 2023-10-30T14:14:01.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.692+0000 7fdc3d26b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdc380c4410 0x7fdc380cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:01.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.692+0000 7fdc3d26b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fdc380c4810 0x7fdc3815a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:01.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.692+0000 7fdc3d26b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc38153f80 con 0x7fdc38153ba0 2023-10-30T14:14:01.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.692+0000 7fdc3d26b640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc38159df0 con 0x7fdc380c4810 2023-10-30T14:14:01.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.692+0000 7fdc3d26b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc3815a7c0 con 0x7fdc380c4410 2023-10-30T14:14:01.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.692+0000 7fdc377fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fdc380c4810 0x7fdc3815a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:01.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.692+0000 7fdc377fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fdc380c4810 0x7fdc3815a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44668/0 (socket says 172.21.15.57:44668) 2023-10-30T14:14:01.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.692+0000 7fdc377fe640 1 -- 172.21.15.57:0/3855372384 learned_addr learned my addr 172.21.15.57:0/3855372384 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:01.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.692+0000 7fdc2ffff640 1 --2- 172.21.15.57:0/3855372384 >> v2:172.21.15.88:3300/0 conn(0x7fdc380c4410 0x7fdc380cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:01.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.692+0000 7fdc377fe640 1 -- 172.21.15.57:0/3855372384 >> v2:172.21.15.88:3300/0 conn(0x7fdc380c4410 msgr2=0x7fdc380cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:01.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.692+0000 7fdc377fe640 1 --2- 172.21.15.57:0/3855372384 >> v2:172.21.15.88:3300/0 conn(0x7fdc380c4410 0x7fdc380cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.692+0000 7fdc377fe640 1 -- 172.21.15.57:0/3855372384 >> v1:172.21.15.88:6789/0 conn(0x7fdc38153ba0 legacy=0x7fdc3815cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:01.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.692+0000 7fdc377fe640 1 -- 172.21.15.57:0/3855372384 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc381597c0 con 0x7fdc380c4810 2023-10-30T14:14:01.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.692+0000 7fdc2ffff640 1 --2- 172.21.15.57:0/3855372384 >> v2:172.21.15.88:3300/0 conn(0x7fdc380c4410 0x7fdc380cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:01.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.696+0000 7fdc377fe640 1 --2- 172.21.15.57:0/3855372384 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc380c4810 0x7fdc3815a280 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdc38072540 tx=0x7fdc3001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b71b1249c848818 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:01.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.696+0000 7fdc367fc640 1 -- 172.21.15.57:0/3855372384 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc300a57d0 con 0x7fdc380c4810 2023-10-30T14:14:01.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.696+0000 7fdc367fc640 1 -- 172.21.15.57:0/3855372384 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc300a5930 con 0x7fdc380c4810 2023-10-30T14:14:01.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.696+0000 7fdc367fc640 1 -- 172.21.15.57:0/3855372384 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc300a5cc0 con 0x7fdc380c4810 2023-10-30T14:14:01.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.696+0000 7fdc3d26b640 1 -- 172.21.15.57:0/3855372384 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc380c4810 msgr2=0x7fdc3815a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:01.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.696+0000 7fdc3d26b640 1 --2- 172.21.15.57:0/3855372384 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc380c4810 0x7fdc3815a280 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdc38072540 tx=0x7fdc3001c910 comp rx=0 tx=0).stop 2023-10-30T14:14:01.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.696+0000 7fdc3d26b640 1 -- 172.21.15.57:0/3855372384 shutdown_connections 2023-10-30T14:14:01.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.696+0000 7fdc3d26b640 1 --2- 172.21.15.57:0/3855372384 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc380c4810 0x7fdc3815a280 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.696+0000 7fdc3d26b640 1 --2- 172.21.15.57:0/3855372384 >> v2:172.21.15.88:3300/0 conn(0x7fdc380c4410 0x7fdc380cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.696+0000 7fdc3d26b640 1 -- 172.21.15.57:0/3855372384 >> 172.21.15.57:0/3855372384 conn(0x7fdc38071790 msgr2=0x7fdc38071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:01.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.696+0000 7fdc3d26b640 1 -- 172.21.15.57:0/3855372384 shutdown_connections 2023-10-30T14:14:01.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.696+0000 7fdc3d26b640 1 -- 172.21.15.57:0/3855372384 wait complete. 2023-10-30T14:14:01.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.696+0000 7fdc3d26b640 1 Processor -- start 2023-10-30T14:14:01.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.696+0000 7fdc3d26b640 1 -- start start 2023-10-30T14:14:01.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc3d26b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc380c4410 0x7fdc380ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:01.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc3d26b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc380c4810 0x7fdc381429c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:01.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc3d26b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc38153ba0 0x7fdc38142f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:01.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc3d26b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc38146820 con 0x7fdc380c4410 2023-10-30T14:14:01.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc3d26b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3815b020 con 0x7fdc38153ba0 2023-10-30T14:14:01.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc3d26b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc381453a0 con 0x7fdc380c4810 2023-10-30T14:14:01.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc36ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc380c4410 0x7fdc380ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:01.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc36ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc380c4410 0x7fdc380ce3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44680/0 (socket says 172.21.15.57:44680) 2023-10-30T14:14:01.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc36ffd640 1 -- 172.21.15.57:0/2054521111 learned_addr learned my addr 172.21.15.57:0/2054521111 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:01.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc377fe640 1 --2- 172.21.15.57:0/2054521111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc38153ba0 0x7fdc38142f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:01.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc2ffff640 1 --2- 172.21.15.57:0/2054521111 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc380c4810 0x7fdc381429c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:01.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc36ffd640 1 -- 172.21.15.57:0/2054521111 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc380c4810 msgr2=0x7fdc381429c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:01.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc36ffd640 1 --2- 172.21.15.57:0/2054521111 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc380c4810 0x7fdc381429c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc36ffd640 1 -- 172.21.15.57:0/2054521111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc38153ba0 msgr2=0x7fdc38142f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:01.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc36ffd640 1 --2- 172.21.15.57:0/2054521111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc38153ba0 0x7fdc38142f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc36ffd640 1 -- 172.21.15.57:0/2054521111 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc30009b30 con 0x7fdc380c4410 2023-10-30T14:14:01.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc2ffff640 1 --2- 172.21.15.57:0/2054521111 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc380c4810 0x7fdc381429c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:01.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc377fe640 1 --2- 172.21.15.57:0/2054521111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc38153ba0 0x7fdc38142f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:01.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc36ffd640 1 --2- 172.21.15.57:0/2054521111 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc380c4410 0x7fdc380ce3d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800c3c0 tx=0x7fdc2800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:01.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc2f7fe640 1 -- 172.21.15.57:0/2054521111 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc280052d0 con 0x7fdc380c4410 2023-10-30T14:14:01.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc2f7fe640 1 -- 172.21.15.57:0/2054521111 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc28005430 con 0x7fdc380c4410 2023-10-30T14:14:01.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc2f7fe640 1 -- 172.21.15.57:0/2054521111 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc28003840 con 0x7fdc380c4410 2023-10-30T14:14:01.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc3d26b640 1 -- 172.21.15.57:0/2054521111 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc38145680 con 0x7fdc380c4410 2023-10-30T14:14:01.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc3d26b640 1 -- 172.21.15.57:0/2054521111 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc380ce910 con 0x7fdc380c4410 2023-10-30T14:14:01.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc3d26b640 1 -- 172.21.15.57:0/2054521111 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc3804e170 con 0x7fdc380c4410 2023-10-30T14:14:01.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.700+0000 7fdc2f7fe640 1 -- 172.21.15.57:0/2054521111 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdc280039a0 con 0x7fdc380c4410 2023-10-30T14:14:01.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.704+0000 7fdc2f7fe640 1 --2- 172.21.15.57:0/2054521111 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdc180664f0 0x7fdc180689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:01.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.704+0000 7fdc2f7fe640 1 -- 172.21.15.57:0/2054521111 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdc280830c0 con 0x7fdc380c4410 2023-10-30T14:14:01.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.704+0000 7fdc2ffff640 1 --2- 172.21.15.57:0/2054521111 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdc180664f0 0x7fdc180689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:01.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.704+0000 7fdc2ffff640 1 --2- 172.21.15.57:0/2054521111 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdc180664f0 0x7fdc180689b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdc38143e50 tx=0x7fdc20008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:01.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.708+0000 7fdc2f7fe640 1 -- 172.21.15.57:0/2054521111 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdc28050870 con 0x7fdc380c4410 2023-10-30T14:14:01.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.864+0000 7fdc3d26b640 1 -- 172.21.15.57:0/2054521111 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdc380cabb0 con 0x7fdc380c4410 2023-10-30T14:14:01.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.864+0000 7fdc2f7fe640 1 -- 172.21.15.57:0/2054521111 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdc28053e90 con 0x7fdc380c4410 2023-10-30T14:14:01.871 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-30T14:14:01.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.868+0000 7fdc3d26b640 1 -- 172.21.15.57:0/2054521111 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdc180664f0 msgr2=0x7fdc180689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:01.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.868+0000 7fdc3d26b640 1 --2- 172.21.15.57:0/2054521111 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdc180664f0 0x7fdc180689b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdc38143e50 tx=0x7fdc20008040 comp rx=0 tx=0).stop 2023-10-30T14:14:01.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.868+0000 7fdc3d26b640 1 -- 172.21.15.57:0/2054521111 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc380c4410 msgr2=0x7fdc380ce3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:01.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.868+0000 7fdc3d26b640 1 --2- 172.21.15.57:0/2054521111 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc380c4410 0x7fdc380ce3d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800c3c0 tx=0x7fdc2800c880 comp rx=0 tx=0).stop 2023-10-30T14:14:01.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.868+0000 7fdc3d26b640 1 -- 172.21.15.57:0/2054521111 shutdown_connections 2023-10-30T14:14:01.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.868+0000 7fdc3d26b640 1 --2- 172.21.15.57:0/2054521111 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdc180664f0 0x7fdc180689b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.868+0000 7fdc3d26b640 1 --2- 172.21.15.57:0/2054521111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc38153ba0 0x7fdc38142f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.868+0000 7fdc3d26b640 1 --2- 172.21.15.57:0/2054521111 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc380c4810 0x7fdc381429c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.868+0000 7fdc3d26b640 1 --2- 172.21.15.57:0/2054521111 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc380c4410 0x7fdc380ce3d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:01.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.868+0000 7fdc3d26b640 1 -- 172.21.15.57:0/2054521111 >> 172.21.15.57:0/2054521111 conn(0x7fdc38071790 msgr2=0x7fdc38147440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:01.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.868+0000 7fdc3d26b640 1 -- 172.21.15.57:0/2054521111 shutdown_connections 2023-10-30T14:14:01.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:01.868+0000 7fdc3d26b640 1 -- 172.21.15.57:0/2054521111 wait complete. 2023-10-30T14:14:01.891 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-30T14:14:02.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.028+0000 7f0dc65b5640 1 Processor -- start 2023-10-30T14:14:02.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.028+0000 7f0dc65b5640 1 -- start start 2023-10-30T14:14:02.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0dc65b5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00c8eb0 0x7f0dc015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:02.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0dc65b5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc015dcc0 con 0x7f0dc015ac70 2023-10-30T14:14:02.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0dc65b5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc0159d80 con 0x7f0dc006aeb0 2023-10-30T14:14:02.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0dc65b5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc0159800 con 0x7f0dc00c8eb0 2023-10-30T14:14:02.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0dc4db2640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f0dc015ac70 0x7f0dc015a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:45862/0 (socket says 172.21.15.57:45862) 2023-10-30T14:14:02.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0dc4db2640 1 -- 172.21.15.57:0/92141435 learned_addr learned my addr 172.21.15.57:0/92141435 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:02.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0dc55b3640 1 --2- 172.21.15.57:0/92141435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00c8eb0 0x7f0dc015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:02.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0db7fff640 1 -- 172.21.15.57:0/92141435 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171723489 0 0) 0x7f0db0001120 con 0x7f0dc015ac70 2023-10-30T14:14:02.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0db7fff640 1 -- 172.21.15.57:0/92141435 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dac003530 con 0x7f0dc015ac70 2023-10-30T14:14:02.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0db7fff640 1 -- 172.21.15.57:0/92141435 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1342058519 0 0) 0x7f0dac003530 con 0x7f0dc015ac70 2023-10-30T14:14:02.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0db7fff640 1 -- 172.21.15.57:0/92141435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00c8eb0 msgr2=0x7f0dc015cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:02.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0db7fff640 1 --2- 172.21.15.57:0/92141435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00c8eb0 0x7f0dc015cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0db7fff640 1 -- 172.21.15.57:0/92141435 >> v1:172.21.15.88:6789/0 conn(0x7f0dc006aeb0 legacy=0x7f0dc0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:02.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0db7fff640 1 -- 172.21.15.57:0/92141435 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc015d420 con 0x7f0dc015ac70 2023-10-30T14:14:02.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0db7fff640 1 -- 172.21.15.57:0/92141435 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f0db0001810 con 0x7f0dc015ac70 2023-10-30T14:14:02.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0db7fff640 1 -- 172.21.15.57:0/92141435 >> v1:172.21.15.57:6789/0 conn(0x7f0dc015ac70 legacy=0x7f0dc015a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:02.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0db7fff640 1 --2- 172.21.15.57:0/92141435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dac0039b0 0x7f0dac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:02.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0db7fff640 1 --2- 172.21.15.57:0/92141435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dac004360 0x7f0dac004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:02.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0db7fff640 1 --2- 172.21.15.57:0/92141435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00c8eb0 0x7f0dac005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:02.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0db7fff640 1 -- 172.21.15.57:0/92141435 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc0159800 con 0x7f0dac0039b0 2023-10-30T14:14:02.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0db7fff640 1 -- 172.21.15.57:0/92141435 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac00b990 con 0x7f0dac004360 2023-10-30T14:14:02.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0db7fff640 1 -- 172.21.15.57:0/92141435 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac00bb50 con 0x7f0dc00c8eb0 2023-10-30T14:14:02.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0dc55b3640 1 --2- 172.21.15.57:0/92141435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dac0039b0 0x7f0dac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:02.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0dc4db2640 1 --2- 172.21.15.57:0/92141435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dac004360 0x7f0dac004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:02.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0dc5db4640 1 --2- 172.21.15.57:0/92141435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00c8eb0 0x7f0dac005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:02.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0dc55b3640 1 -- 172.21.15.57:0/92141435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00c8eb0 msgr2=0x7f0dac005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:02.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0dc55b3640 1 --2- 172.21.15.57:0/92141435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00c8eb0 0x7f0dac005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0dc55b3640 1 -- 172.21.15.57:0/92141435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dac004360 msgr2=0x7f0dac004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:02.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0dc55b3640 1 --2- 172.21.15.57:0/92141435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dac004360 0x7f0dac004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.032+0000 7f0dc55b3640 1 -- 172.21.15.57:0/92141435 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dac003530 con 0x7f0dac0039b0 2023-10-30T14:14:02.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.036+0000 7f0dc55b3640 1 --2- 172.21.15.57:0/92141435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dac0039b0 0x7f0dac003d90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009ae0 tx=0x7f0dbc01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181b75da053ef965 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:02.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.036+0000 7f0db7fff640 1 -- 172.21.15.57:0/92141435 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc0a5640 con 0x7f0dac0039b0 2023-10-30T14:14:02.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.036+0000 7f0db7fff640 1 -- 172.21.15.57:0/92141435 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dbc0a57a0 con 0x7f0dac0039b0 2023-10-30T14:14:02.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.036+0000 7f0db7fff640 1 -- 172.21.15.57:0/92141435 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc0a5b50 con 0x7f0dac0039b0 2023-10-30T14:14:02.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.036+0000 7f0dc65b5640 1 -- 172.21.15.57:0/92141435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dac0039b0 msgr2=0x7f0dac003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:02.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.036+0000 7f0dc65b5640 1 --2- 172.21.15.57:0/92141435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dac0039b0 0x7f0dac003d90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009ae0 tx=0x7f0dbc01ca50 comp rx=0 tx=0).stop 2023-10-30T14:14:02.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.036+0000 7f0dc5db4640 1 -- 172.21.15.57:0/92141435 reap_dead start 2023-10-30T14:14:02.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.036+0000 7f0dc65b5640 1 -- 172.21.15.57:0/92141435 shutdown_connections 2023-10-30T14:14:02.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.036+0000 7f0dc65b5640 1 -- 172.21.15.57:0/92141435 >> 172.21.15.57:0/92141435 conn(0x7f0dc00cef00 msgr2=0x7f0dc00cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:02.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.036+0000 7f0dc65b5640 1 -- 172.21.15.57:0/92141435 shutdown_connections 2023-10-30T14:14:02.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.036+0000 7f0dc65b5640 1 -- 172.21.15.57:0/92141435 wait complete. 2023-10-30T14:14:02.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc65b5640 1 Processor -- start 2023-10-30T14:14:02.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc65b5640 1 -- start start 2023-10-30T14:14:02.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc65b5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dac0039b0 0x7f0dc01ef9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:02.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc65b5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dc00ce220 0x7f0dc00ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:02.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc65b5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00cd220 0x7f0dc01515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:02.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc65b5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc01eff00 con 0x7f0dc00ce220 2023-10-30T14:14:02.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc65b5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc01510d0 con 0x7f0dac0039b0 2023-10-30T14:14:02.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc65b5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc0152290 con 0x7f0dc00cd220 2023-10-30T14:14:02.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc4db2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dc00ce220 0x7f0dc00ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:02.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc4db2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dc00ce220 0x7f0dc00ccce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44696/0 (socket says 172.21.15.57:44696) 2023-10-30T14:14:02.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc4db2640 1 -- 172.21.15.57:0/4085933607 learned_addr learned my addr 172.21.15.57:0/4085933607 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:02.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc55b3640 1 --2- 172.21.15.57:0/4085933607 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dac0039b0 0x7f0dc01ef9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:02.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc5db4640 1 --2- 172.21.15.57:0/4085933607 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00cd220 0x7f0dc01515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:02.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc4db2640 1 -- 172.21.15.57:0/4085933607 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00cd220 msgr2=0x7f0dc01515b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:02.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc4db2640 1 --2- 172.21.15.57:0/4085933607 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00cd220 0x7f0dc01515b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc4db2640 1 -- 172.21.15.57:0/4085933607 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dac0039b0 msgr2=0x7f0dc01ef9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:02.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc4db2640 1 --2- 172.21.15.57:0/4085933607 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dac0039b0 0x7f0dc01ef9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc4db2640 1 -- 172.21.15.57:0/4085933607 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc0152570 con 0x7f0dc00ce220 2023-10-30T14:14:02.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc5db4640 1 --2- 172.21.15.57:0/4085933607 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00cd220 0x7f0dc01515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:02.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc55b3640 1 --2- 172.21.15.57:0/4085933607 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dac0039b0 0x7f0dc01ef9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:02.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc4db2640 1 --2- 172.21.15.57:0/4085933607 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dc00ce220 0x7f0dc00ccce0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0db0001280 tx=0x7f0db000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:02.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0db5ffb640 1 -- 172.21.15.57:0/4085933607 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db000c8d0 con 0x7f0dc00ce220 2023-10-30T14:14:02.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0db5ffb640 1 -- 172.21.15.57:0/4085933607 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db000ca30 con 0x7f0dc00ce220 2023-10-30T14:14:02.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0db5ffb640 1 -- 172.21.15.57:0/4085933607 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db000cde0 con 0x7f0dc00ce220 2023-10-30T14:14:02.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc65b5640 1 -- 172.21.15.57:0/4085933607 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc0152850 con 0x7f0dc00ce220 2023-10-30T14:14:02.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc65b5640 1 -- 172.21.15.57:0/4085933607 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc0159270 con 0x7f0dc00ce220 2023-10-30T14:14:02.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.040+0000 7f0dc65b5640 1 -- 172.21.15.57:0/4085933607 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc0150f30 con 0x7f0dc00ce220 2023-10-30T14:14:02.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.044+0000 7f0db5ffb640 1 -- 172.21.15.57:0/4085933607 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0db001d460 con 0x7f0dc00ce220 2023-10-30T14:14:02.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.044+0000 7f0db5ffb640 1 --2- 172.21.15.57:0/4085933607 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0da40664a0 0x7f0da4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:02.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.044+0000 7f0db5ffb640 1 -- 172.21.15.57:0/4085933607 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0db0083a90 con 0x7f0dc00ce220 2023-10-30T14:14:02.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.044+0000 7f0dc55b3640 1 --2- 172.21.15.57:0/4085933607 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0da40664a0 0x7f0da4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:02.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.044+0000 7f0dc55b3640 1 --2- 172.21.15.57:0/4085933607 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0da40664a0 0x7f0da4068960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009ae0 tx=0x7f0dbc0046d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:02.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.048+0000 7f0db5ffb640 1 -- 172.21.15.57:0/4085933607 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0dc0150f30 con 0x7f0dc00ce220 2023-10-30T14:14:02.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.208+0000 7f0dc65b5640 1 -- 172.21.15.57:0/4085933607 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0dc015dcc0 con 0x7f0dc00ce220 2023-10-30T14:14:02.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.208+0000 7f0db5ffb640 1 -- 172.21.15.57:0/4085933607 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0dc015dcc0 con 0x7f0dc00ce220 2023-10-30T14:14:02.214 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:02.215 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-30T14:14:02.215 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-30T14:14:02.215 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-10-30T14:14:02.215 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-10-30T14:14:02.215 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-30T14:14:02.216 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-30T14:14:02.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.212+0000 7f0dc65b5640 1 -- 172.21.15.57:0/4085933607 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0da40664a0 msgr2=0x7f0da4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:02.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.212+0000 7f0dc65b5640 1 --2- 172.21.15.57:0/4085933607 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0da40664a0 0x7f0da4068960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009ae0 tx=0x7f0dbc0046d0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.212+0000 7f0dc65b5640 1 -- 172.21.15.57:0/4085933607 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dc00ce220 msgr2=0x7f0dc00ccce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:02.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.212+0000 7f0dc65b5640 1 --2- 172.21.15.57:0/4085933607 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dc00ce220 0x7f0dc00ccce0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0db0001280 tx=0x7f0db000eca0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.212+0000 7f0dc65b5640 1 -- 172.21.15.57:0/4085933607 shutdown_connections 2023-10-30T14:14:02.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.212+0000 7f0dc65b5640 1 --2- 172.21.15.57:0/4085933607 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0da40664a0 0x7f0da4068960 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.212+0000 7f0dc65b5640 1 --2- 172.21.15.57:0/4085933607 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00cd220 0x7f0dc01515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.212+0000 7f0dc65b5640 1 --2- 172.21.15.57:0/4085933607 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dc00ce220 0x7f0dc00ccce0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.212+0000 7f0dc65b5640 1 --2- 172.21.15.57:0/4085933607 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dac0039b0 0x7f0dc01ef9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.212+0000 7f0dc65b5640 1 -- 172.21.15.57:0/4085933607 >> 172.21.15.57:0/4085933607 conn(0x7f0dc00cef00 msgr2=0x7f0dc00c7500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:02.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.212+0000 7f0dc65b5640 1 -- 172.21.15.57:0/4085933607 shutdown_connections 2023-10-30T14:14:02.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.212+0000 7f0dc65b5640 1 -- 172.21.15.57:0/4085933607 wait complete. 2023-10-30T14:14:02.235 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-30T14:14:02.235 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-30T14:14:02.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.376+0000 7fc06ce32640 1 Processor -- start 2023-10-30T14:14:02.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.376+0000 7fc06ce32640 1 -- start start 2023-10-30T14:14:02.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.376+0000 7fc06ce32640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc0680c4340 0x7fc0681599e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:02.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.376+0000 7fc06ce32640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0680ca470 0x7fc0680ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:02.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.376+0000 7fc06ce32640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc06806af00 con 0x7fc0680c4720 2023-10-30T14:14:02.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.376+0000 7fc06ce32640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0680c4cd0 con 0x7fc0680c4340 2023-10-30T14:14:02.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.376+0000 7fc06ce32640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc06806a660 con 0x7fc0680ca470 2023-10-30T14:14:02.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.380+0000 7fc066d9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0680ca470 0x7fc0680ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:02.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.380+0000 7fc06659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc0680c4340 0x7fc0681599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:02.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.380+0000 7fc066d9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0680ca470 0x7fc0680ca8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:58658/0 (socket says 172.21.15.57:58658) 2023-10-30T14:14:02.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.380+0000 7fc066d9d640 1 -- 172.21.15.57:0/2080765187 learned_addr learned my addr 172.21.15.57:0/2080765187 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:02.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.380+0000 7fc065d9b640 1 --1- 172.21.15.57:0/2080765187 >> v1:172.21.15.88:6789/0 conn(0x7fc0680c4720 0x7fc068069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:41362/0 (socket says 172.21.15.57:41362) 2023-10-30T14:14:02.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.380+0000 7fc066d9d640 1 -- 172.21.15.57:0/2080765187 >> v2:172.21.15.88:3300/0 conn(0x7fc0680c4340 msgr2=0x7fc0681599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:02.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.380+0000 7fc066d9d640 1 --2- 172.21.15.57:0/2080765187 >> v2:172.21.15.88:3300/0 conn(0x7fc0680c4340 0x7fc0681599e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.380+0000 7fc066d9d640 1 -- 172.21.15.57:0/2080765187 >> v1:172.21.15.88:6789/0 conn(0x7fc0680c4720 legacy=0x7fc068069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:02.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.380+0000 7fc066d9d640 1 -- 172.21.15.57:0/2080765187 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06806a940 con 0x7fc0680ca470 2023-10-30T14:14:02.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.380+0000 7fc06659c640 1 --2- 172.21.15.57:0/2080765187 >> v2:172.21.15.88:3300/0 conn(0x7fc0680c4340 0x7fc0681599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:14:02.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.380+0000 7fc066d9d640 1 --2- 172.21.15.57:0/2080765187 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0680ca470 0x7fc0680ca8e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc0680c4e10 tx=0x7fc06001c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93c3e6adf5ae1b69 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:02.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.384+0000 7fc06559a640 1 -- 172.21.15.57:0/2080765187 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0600a5530 con 0x7fc0680ca470 2023-10-30T14:14:02.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.384+0000 7fc06559a640 1 -- 172.21.15.57:0/2080765187 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0600a5690 con 0x7fc0680ca470 2023-10-30T14:14:02.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.384+0000 7fc06559a640 1 -- 172.21.15.57:0/2080765187 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0600a5a20 con 0x7fc0680ca470 2023-10-30T14:14:02.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.384+0000 7fc06ce32640 1 -- 172.21.15.57:0/2080765187 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0680ca470 msgr2=0x7fc0680ca8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:02.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.384+0000 7fc06ce32640 1 --2- 172.21.15.57:0/2080765187 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0680ca470 0x7fc0680ca8e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc0680c4e10 tx=0x7fc06001c670 comp rx=0 tx=0).stop 2023-10-30T14:14:02.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.384+0000 7fc06ce32640 1 -- 172.21.15.57:0/2080765187 shutdown_connections 2023-10-30T14:14:02.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.384+0000 7fc06ce32640 1 --2- 172.21.15.57:0/2080765187 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0680ca470 0x7fc0680ca8e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.384+0000 7fc06ce32640 1 --2- 172.21.15.57:0/2080765187 >> v2:172.21.15.88:3300/0 conn(0x7fc0680c4340 0x7fc0681599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.384+0000 7fc06ce32640 1 -- 172.21.15.57:0/2080765187 >> 172.21.15.57:0/2080765187 conn(0x7fc0680cf1c0 msgr2=0x7fc0680cf5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:02.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.384+0000 7fc06ce32640 1 -- 172.21.15.57:0/2080765187 shutdown_connections 2023-10-30T14:14:02.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.384+0000 7fc06ce32640 1 -- 172.21.15.57:0/2080765187 wait complete. 2023-10-30T14:14:02.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.384+0000 7fc06ce32640 1 Processor -- start 2023-10-30T14:14:02.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.384+0000 7fc06ce32640 1 -- start start 2023-10-30T14:14:02.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.384+0000 7fc06ce32640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0680c4340 0x7fc068152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:02.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.384+0000 7fc06ce32640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0680c4720 0x7fc06814add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:02.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.384+0000 7fc06ce32640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0680ca470 0x7fc06814b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:02.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.384+0000 7fc06ce32640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc068157450 con 0x7fc0680ca470 2023-10-30T14:14:02.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.384+0000 7fc06ce32640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0681551a0 con 0x7fc0680c4720 2023-10-30T14:14:02.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.384+0000 7fc06ce32640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc068156860 con 0x7fc0680c4340 2023-10-30T14:14:02.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.388+0000 7fc066d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0680ca470 0x7fc06814b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:02.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.388+0000 7fc06659c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0680c4340 0x7fc068152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:02.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.388+0000 7fc066d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0680ca470 0x7fc06814b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44710/0 (socket says 172.21.15.57:44710) 2023-10-30T14:14:02.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.388+0000 7fc066d9d640 1 -- 172.21.15.57:0/1981406636 learned_addr learned my addr 172.21.15.57:0/1981406636 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:02.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.388+0000 7fc065d9b640 1 --2- 172.21.15.57:0/1981406636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0680c4720 0x7fc06814add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:02.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.388+0000 7fc066d9d640 1 -- 172.21.15.57:0/1981406636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0680c4340 msgr2=0x7fc068152c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:02.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.388+0000 7fc066d9d640 1 --2- 172.21.15.57:0/1981406636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0680c4340 0x7fc068152c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.388+0000 7fc066d9d640 1 -- 172.21.15.57:0/1981406636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0680c4720 msgr2=0x7fc06814add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:02.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.388+0000 7fc066d9d640 1 --2- 172.21.15.57:0/1981406636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0680c4720 0x7fc06814add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.388+0000 7fc066d9d640 1 -- 172.21.15.57:0/1981406636 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc058009420 con 0x7fc0680ca470 2023-10-30T14:14:02.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.388+0000 7fc066d9d640 1 --2- 172.21.15.57:0/1981406636 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0680ca470 0x7fc06814b310 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc0600af040 tx=0x7fc060019410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:02.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.388+0000 7fc056ffd640 1 -- 172.21.15.57:0/1981406636 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0600a5530 con 0x7fc0680ca470 2023-10-30T14:14:02.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.388+0000 7fc056ffd640 1 -- 172.21.15.57:0/1981406636 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0600b8640 con 0x7fc0680ca470 2023-10-30T14:14:02.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.388+0000 7fc06ce32640 1 -- 172.21.15.57:0/1981406636 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc060009b30 con 0x7fc0680ca470 2023-10-30T14:14:02.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.388+0000 7fc056ffd640 1 -- 172.21.15.57:0/1981406636 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0600b89b0 con 0x7fc0680ca470 2023-10-30T14:14:02.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.388+0000 7fc06ce32640 1 -- 172.21.15.57:0/1981406636 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc068156e00 con 0x7fc0680ca470 2023-10-30T14:14:02.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.388+0000 7fc06ce32640 1 -- 172.21.15.57:0/1981406636 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc06804e170 con 0x7fc0680ca470 2023-10-30T14:14:02.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.388+0000 7fc056ffd640 1 -- 172.21.15.57:0/1981406636 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc0600b8b10 con 0x7fc0680ca470 2023-10-30T14:14:02.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.392+0000 7fc056ffd640 1 --2- 172.21.15.57:0/1981406636 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0480664a0 0x7fc048068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:02.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.392+0000 7fc056ffd640 1 -- 172.21.15.57:0/1981406636 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc06011ec60 con 0x7fc0680ca470 2023-10-30T14:14:02.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.392+0000 7fc06659c640 1 --2- 172.21.15.57:0/1981406636 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0480664a0 0x7fc048068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:02.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.392+0000 7fc06659c640 1 --2- 172.21.15.57:0/1981406636 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0480664a0 0x7fc048068960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc058009c50 tx=0x7fc058009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:02.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.396+0000 7fc056ffd640 1 -- 172.21.15.57:0/1981406636 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc0600ec480 con 0x7fc0680ca470 2023-10-30T14:14:02.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.548+0000 7fc06ce32640 1 -- 172.21.15.57:0/1981406636 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc06806a210 con 0x7fc0680ca470 2023-10-30T14:14:02.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.668+0000 7fc056ffd640 1 -- 172.21.15.57:0/1981406636 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc060009510 con 0x7fc0680ca470 2023-10-30T14:14:02.672 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' created 2023-10-30T14:14:02.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.668+0000 7fc06ce32640 1 -- 172.21.15.57:0/1981406636 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0480664a0 msgr2=0x7fc048068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:02.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.668+0000 7fc06ce32640 1 --2- 172.21.15.57:0/1981406636 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0480664a0 0x7fc048068960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc058009c50 tx=0x7fc058009190 comp rx=0 tx=0).stop 2023-10-30T14:14:02.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.668+0000 7fc06ce32640 1 -- 172.21.15.57:0/1981406636 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0680ca470 msgr2=0x7fc06814b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:02.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.668+0000 7fc06ce32640 1 --2- 172.21.15.57:0/1981406636 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0680ca470 0x7fc06814b310 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc0600af040 tx=0x7fc060019410 comp rx=0 tx=0).stop 2023-10-30T14:14:02.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.668+0000 7fc06ce32640 1 -- 172.21.15.57:0/1981406636 shutdown_connections 2023-10-30T14:14:02.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.668+0000 7fc06ce32640 1 --2- 172.21.15.57:0/1981406636 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0480664a0 0x7fc048068960 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.668+0000 7fc06ce32640 1 --2- 172.21.15.57:0/1981406636 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0680ca470 0x7fc06814b310 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.668+0000 7fc06ce32640 1 --2- 172.21.15.57:0/1981406636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0680c4720 0x7fc06814add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.668+0000 7fc06ce32640 1 --2- 172.21.15.57:0/1981406636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0680c4340 0x7fc068152c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.668+0000 7fc06ce32640 1 -- 172.21.15.57:0/1981406636 >> 172.21.15.57:0/1981406636 conn(0x7fc0680cf1c0 msgr2=0x7fc0680c5990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:02.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.668+0000 7fc06ce32640 1 -- 172.21.15.57:0/1981406636 shutdown_connections 2023-10-30T14:14:02.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.668+0000 7fc06ce32640 1 -- 172.21.15.57:0/1981406636 wait complete. 2023-10-30T14:14:02.700 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-30T14:14:02.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.844+0000 7f221d73e640 1 Processor -- start 2023-10-30T14:14:02.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.844+0000 7f221d73e640 1 -- start start 2023-10-30T14:14:02.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.844+0000 7f221d73e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2218153ba0 0x7f22180cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:02.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.844+0000 7f221d73e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2218153f80 con 0x7f22180c4410 2023-10-30T14:14:02.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.844+0000 7f221d73e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2218159df0 con 0x7f22180c4810 2023-10-30T14:14:02.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.844+0000 7f221d73e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22181597c0 con 0x7f2218153ba0 2023-10-30T14:14:02.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.848+0000 7f2216ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2218153ba0 0x7f22180cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:02.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.848+0000 7f22167fc640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f22180c4410 0x7f221815cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:45872/0 (socket says 172.21.15.57:45872) 2023-10-30T14:14:02.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.848+0000 7f22167fc640 1 -- 172.21.15.57:0/3146315959 learned_addr learned my addr 172.21.15.57:0/3146315959 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:02.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.848+0000 7f2216ffd640 1 -- 172.21.15.57:0/3146315959 >> v1:172.21.15.88:6789/0 conn(0x7f22180c4810 legacy=0x7f22180cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:02.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.848+0000 7f2216ffd640 1 -- 172.21.15.57:0/3146315959 >> v1:172.21.15.57:6789/0 conn(0x7f22180c4410 legacy=0x7f221815cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:02.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.848+0000 7f2216ffd640 1 -- 172.21.15.57:0/3146315959 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221815d580 con 0x7f2218153ba0 2023-10-30T14:14:02.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.848+0000 7f2215ffb640 1 -- 172.21.15.57:0/3146315959 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636913650 0 0) 0x7f2218153f80 con 0x7f22180c4410 2023-10-30T14:14:02.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.848+0000 7f2216ffd640 1 --2- 172.21.15.57:0/3146315959 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2218153ba0 0x7f22180cb4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f22080097c0 tx=0x7f220801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d3c00db680283b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:02.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.848+0000 7f2215ffb640 1 -- 172.21.15.57:0/3146315959 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22080ac620 con 0x7f2218153ba0 2023-10-30T14:14:02.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.848+0000 7f2215ffb640 1 -- 172.21.15.57:0/3146315959 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22080ac780 con 0x7f2218153ba0 2023-10-30T14:14:02.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.848+0000 7f2215ffb640 1 -- 172.21.15.57:0/3146315959 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22080acb10 con 0x7f2218153ba0 2023-10-30T14:14:02.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f221d73e640 1 -- 172.21.15.57:0/3146315959 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2218153ba0 msgr2=0x7f22180cb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:02.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f221d73e640 1 --2- 172.21.15.57:0/3146315959 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2218153ba0 0x7f22180cb4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f22080097c0 tx=0x7f220801c3c0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f221d73e640 1 -- 172.21.15.57:0/3146315959 shutdown_connections 2023-10-30T14:14:02.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f221d73e640 1 --2- 172.21.15.57:0/3146315959 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2218153ba0 0x7f22180cb4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f221d73e640 1 -- 172.21.15.57:0/3146315959 >> 172.21.15.57:0/3146315959 conn(0x7f2218071790 msgr2=0x7f2218071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:02.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f221d73e640 1 -- 172.21.15.57:0/3146315959 shutdown_connections 2023-10-30T14:14:02.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f221d73e640 1 -- 172.21.15.57:0/3146315959 wait complete. 2023-10-30T14:14:02.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f221d73e640 1 Processor -- start 2023-10-30T14:14:02.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f221d73e640 1 -- start start 2023-10-30T14:14:02.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f221d73e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22180c4410 0x7f22180c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:02.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f221d73e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22180c4810 0x7f221815c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:02.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f221d73e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2218153ba0 0x7f221815c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:02.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f221d73e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22180c8820 con 0x7f22180c4410 2023-10-30T14:14:02.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f221d73e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22180c94e0 con 0x7f22180c4810 2023-10-30T14:14:02.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f221d73e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22181f2730 con 0x7f2218153ba0 2023-10-30T14:14:02.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f2216ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22180c4410 0x7f22180c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:02.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f2216ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22180c4410 0x7f22180c82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44716/0 (socket says 172.21.15.57:44716) 2023-10-30T14:14:02.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f2216ffd640 1 -- 172.21.15.57:0/3421993977 learned_addr learned my addr 172.21.15.57:0/3421993977 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:02.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f22177fe640 1 --2- 172.21.15.57:0/3421993977 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2218153ba0 0x7f221815c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:02.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f22167fc640 1 --2- 172.21.15.57:0/3421993977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22180c4810 0x7f221815c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:02.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f2216ffd640 1 -- 172.21.15.57:0/3421993977 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2218153ba0 msgr2=0x7f221815c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:02.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f2216ffd640 1 --2- 172.21.15.57:0/3421993977 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2218153ba0 0x7f221815c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f2216ffd640 1 -- 172.21.15.57:0/3421993977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22180c4810 msgr2=0x7f221815c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:02.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f2216ffd640 1 --2- 172.21.15.57:0/3421993977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22180c4810 0x7f221815c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:02.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.852+0000 7f2216ffd640 1 -- 172.21.15.57:0/3421993977 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2208009420 con 0x7f22180c4410 2023-10-30T14:14:02.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.856+0000 7f2216ffd640 1 --2- 172.21.15.57:0/3421993977 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22180c4410 0x7f22180c82e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f22080af040 tx=0x7f2208019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:02.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.856+0000 7f21ff7fe640 1 -- 172.21.15.57:0/3421993977 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22080b5610 con 0x7f22180c4410 2023-10-30T14:14:02.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.856+0000 7f21ff7fe640 1 -- 172.21.15.57:0/3421993977 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22080b5790 con 0x7f22180c4410 2023-10-30T14:14:02.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.856+0000 7f221d73e640 1 -- 172.21.15.57:0/3421993977 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22180c8f20 con 0x7f22180c4410 2023-10-30T14:14:02.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.856+0000 7f21ff7fe640 1 -- 172.21.15.57:0/3421993977 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22080b5b40 con 0x7f22180c4410 2023-10-30T14:14:02.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.856+0000 7f221d73e640 1 -- 172.21.15.57:0/3421993977 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f221815bca0 con 0x7f22180c4410 2023-10-30T14:14:02.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.856+0000 7f221d73e640 1 -- 172.21.15.57:0/3421993977 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f221815be90 con 0x7f22180c4410 2023-10-30T14:14:02.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.856+0000 7f21ff7fe640 1 -- 172.21.15.57:0/3421993977 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f22080b3040 con 0x7f22180c4410 2023-10-30T14:14:02.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.856+0000 7f21ff7fe640 1 --2- 172.21.15.57:0/3421993977 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f22040664a0 0x7f2204068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:02.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.856+0000 7f21ff7fe640 1 -- 172.21.15.57:0/3421993977 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f220811de40 con 0x7f22180c4410 2023-10-30T14:14:02.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.856+0000 7f22167fc640 1 --2- 172.21.15.57:0/3421993977 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f22040664a0 0x7f2204068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:02.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.860+0000 7f22167fc640 1 --2- 172.21.15.57:0/3421993977 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f22040664a0 0x7f2204068960 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f220c007fd0 tx=0x7f220c007ba0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:02.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:02.860+0000 7f21ff7fe640 1 -- 172.21.15.57:0/3421993977 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f22080eb4b0 con 0x7f22180c4410 2023-10-30T14:14:03.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.020+0000 7f221d73e640 1 -- 172.21.15.57:0/3421993977 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f221804e170 con 0x7f22180c4410 2023-10-30T14:14:03.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.672+0000 7f21ff7fe640 1 -- 172.21.15.57:0/3421993977 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7f22080ac5e0 con 0x7f22180c4410 2023-10-30T14:14:03.677 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' created 2023-10-30T14:14:03.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.672+0000 7f221d73e640 1 -- 172.21.15.57:0/3421993977 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f22040664a0 msgr2=0x7f2204068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:03.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.672+0000 7f221d73e640 1 --2- 172.21.15.57:0/3421993977 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f22040664a0 0x7f2204068960 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f220c007fd0 tx=0x7f220c007ba0 comp rx=0 tx=0).stop 2023-10-30T14:14:03.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.672+0000 7f221d73e640 1 -- 172.21.15.57:0/3421993977 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22180c4410 msgr2=0x7f22180c82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:03.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.672+0000 7f221d73e640 1 --2- 172.21.15.57:0/3421993977 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22180c4410 0x7f22180c82e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f22080af040 tx=0x7f2208019160 comp rx=0 tx=0).stop 2023-10-30T14:14:03.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.672+0000 7f221d73e640 1 -- 172.21.15.57:0/3421993977 shutdown_connections 2023-10-30T14:14:03.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.672+0000 7f221d73e640 1 --2- 172.21.15.57:0/3421993977 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f22040664a0 0x7f2204068960 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:03.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.672+0000 7f221d73e640 1 --2- 172.21.15.57:0/3421993977 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2218153ba0 0x7f221815c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:03.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.672+0000 7f221d73e640 1 --2- 172.21.15.57:0/3421993977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22180c4810 0x7f221815c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:03.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.672+0000 7f221d73e640 1 --2- 172.21.15.57:0/3421993977 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22180c4410 0x7f22180c82e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:03.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.672+0000 7f221d73e640 1 -- 172.21.15.57:0/3421993977 >> 172.21.15.57:0/3421993977 conn(0x7f2218071790 msgr2=0x7f22180cd740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:03.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.672+0000 7f221d73e640 1 -- 172.21.15.57:0/3421993977 shutdown_connections 2023-10-30T14:14:03.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.672+0000 7f221d73e640 1 -- 172.21.15.57:0/3421993977 wait complete. 2023-10-30T14:14:03.725 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-30T14:14:03.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.872+0000 7f2c69809640 1 Processor -- start 2023-10-30T14:14:03.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.872+0000 7f2c69809640 1 -- start start 2023-10-30T14:14:03.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.872+0000 7f2c69809640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2c640b1c80 0x7f2c640c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:03.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.872+0000 7f2c69809640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2c640ca370 0x7f2c640b33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:03.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.872+0000 7f2c69809640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c640b2700 con 0x7f2c64153ab0 2023-10-30T14:14:03.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.872+0000 7f2c69809640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c6415a140 con 0x7f2c640b1c80 2023-10-30T14:14:03.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.872+0000 7f2c69809640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c6415a420 con 0x7f2c640ca370 2023-10-30T14:14:03.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.872+0000 7f2c62ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f2c64153ab0 0x7f2c641598b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46214/0 (socket says 172.21.15.57:46214) 2023-10-30T14:14:03.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.872+0000 7f2c62ffd640 1 -- 172.21.15.57:0/217982763 learned_addr learned my addr 172.21.15.57:0/217982763 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:03.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.872+0000 7f2c5bfff640 1 --2- 172.21.15.57:0/217982763 >> v2:172.21.15.57:3300/0 conn(0x7f2c640b1c80 0x7f2c640c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:03.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.876+0000 7f2c637fe640 1 --2- 172.21.15.57:0/217982763 >> v2:172.21.15.88:3300/0 conn(0x7f2c640ca370 0x7f2c640b33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:03.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.876+0000 7f2c627fc640 1 -- 172.21.15.57:0/217982763 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412731796 0 0) 0x7f2c640b2700 con 0x7f2c64153ab0 2023-10-30T14:14:03.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.876+0000 7f2c627fc640 1 -- 172.21.15.57:0/217982763 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c50003530 con 0x7f2c64153ab0 2023-10-30T14:14:03.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.876+0000 7f2c5bfff640 1 -- 172.21.15.57:0/217982763 >> v2:172.21.15.88:3300/0 conn(0x7f2c640ca370 msgr2=0x7f2c640b33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:03.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.876+0000 7f2c5bfff640 1 --2- 172.21.15.57:0/217982763 >> v2:172.21.15.88:3300/0 conn(0x7f2c640ca370 0x7f2c640b33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:03.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.876+0000 7f2c5bfff640 1 -- 172.21.15.57:0/217982763 >> v1:172.21.15.57:6789/0 conn(0x7f2c64153ab0 legacy=0x7f2c641598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:03.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.876+0000 7f2c5bfff640 1 -- 172.21.15.57:0/217982763 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6415a700 con 0x7f2c640b1c80 2023-10-30T14:14:03.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.876+0000 7f2c637fe640 1 --2- 172.21.15.57:0/217982763 >> v2:172.21.15.88:3300/0 conn(0x7f2c640ca370 0x7f2c640b33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:03.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.876+0000 7f2c5bfff640 1 --2- 172.21.15.57:0/217982763 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c640b1c80 0x7f2c640c9e30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2c640ca950 tx=0x7f2c4c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b06cc68ea9494a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:03.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.876+0000 7f2c627fc640 1 -- 172.21.15.57:0/217982763 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c0a8950 con 0x7f2c640b1c80 2023-10-30T14:14:03.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.876+0000 7f2c627fc640 1 -- 172.21.15.57:0/217982763 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4c0a8ab0 con 0x7f2c640b1c80 2023-10-30T14:14:03.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.876+0000 7f2c627fc640 1 -- 172.21.15.57:0/217982763 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c0a8e20 con 0x7f2c640b1c80 2023-10-30T14:14:03.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c69809640 1 -- 172.21.15.57:0/217982763 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c640b1c80 msgr2=0x7f2c640c9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:03.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c69809640 1 --2- 172.21.15.57:0/217982763 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c640b1c80 0x7f2c640c9e30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2c640ca950 tx=0x7f2c4c01ec50 comp rx=0 tx=0).stop 2023-10-30T14:14:03.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c69809640 1 -- 172.21.15.57:0/217982763 shutdown_connections 2023-10-30T14:14:03.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c69809640 1 --2- 172.21.15.57:0/217982763 >> v2:172.21.15.88:3300/0 conn(0x7f2c640ca370 0x7f2c640b33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:03.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c69809640 1 --2- 172.21.15.57:0/217982763 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c640b1c80 0x7f2c640c9e30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:03.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c69809640 1 -- 172.21.15.57:0/217982763 >> 172.21.15.57:0/217982763 conn(0x7f2c6406bef0 msgr2=0x7f2c6406c300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:03.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c69809640 1 -- 172.21.15.57:0/217982763 shutdown_connections 2023-10-30T14:14:03.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c69809640 1 -- 172.21.15.57:0/217982763 wait complete. 2023-10-30T14:14:03.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c69809640 1 Processor -- start 2023-10-30T14:14:03.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c69809640 1 -- start start 2023-10-30T14:14:03.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c69809640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c640b1c80 0x7f2c641f2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:03.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c69809640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c640ca370 0x7f2c641482f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:03.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c69809640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c64153ab0 0x7f2c64148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:03.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c69809640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c64147ed0 con 0x7f2c64153ab0 2023-10-30T14:14:03.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c69809640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c641493f0 con 0x7f2c640ca370 2023-10-30T14:14:03.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c69809640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c640c8dd0 con 0x7f2c640b1c80 2023-10-30T14:14:03.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c637fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c64153ab0 0x7f2c64148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:03.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c637fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c64153ab0 0x7f2c64148830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37770/0 (socket says 172.21.15.57:37770) 2023-10-30T14:14:03.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c637fe640 1 -- 172.21.15.57:0/2247791289 learned_addr learned my addr 172.21.15.57:0/2247791289 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:03.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c5bfff640 1 --2- 172.21.15.57:0/2247791289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c640ca370 0x7f2c641482f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:03.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c62ffd640 1 --2- 172.21.15.57:0/2247791289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c640b1c80 0x7f2c641f2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:03.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c637fe640 1 -- 172.21.15.57:0/2247791289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c640b1c80 msgr2=0x7f2c641f2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:03.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c637fe640 1 --2- 172.21.15.57:0/2247791289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c640b1c80 0x7f2c641f2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:03.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c637fe640 1 -- 172.21.15.57:0/2247791289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c640ca370 msgr2=0x7f2c641482f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:03.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c637fe640 1 --2- 172.21.15.57:0/2247791289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c640ca370 0x7f2c641482f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:03.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c637fe640 1 -- 172.21.15.57:0/2247791289 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4c009420 con 0x7f2c64153ab0 2023-10-30T14:14:03.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c62ffd640 1 --2- 172.21.15.57:0/2247791289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c640b1c80 0x7f2c641f2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:03.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.880+0000 7f2c5bfff640 1 --2- 172.21.15.57:0/2247791289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c640ca370 0x7f2c641482f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:03.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.884+0000 7f2c637fe640 1 --2- 172.21.15.57:0/2247791289 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c64153ab0 0x7f2c64148830 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c007ef0 tx=0x7f2c5c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:03.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.884+0000 7f2c60ff9640 1 -- 172.21.15.57:0/2247791289 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c003ba0 con 0x7f2c64153ab0 2023-10-30T14:14:03.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.884+0000 7f2c60ff9640 1 -- 172.21.15.57:0/2247791289 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5c00f040 con 0x7f2c64153ab0 2023-10-30T14:14:03.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.884+0000 7f2c60ff9640 1 -- 172.21.15.57:0/2247791289 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c00de80 con 0x7f2c64153ab0 2023-10-30T14:14:03.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.884+0000 7f2c69809640 1 -- 172.21.15.57:0/2247791289 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c641526d0 con 0x7f2c64153ab0 2023-10-30T14:14:03.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.884+0000 7f2c69809640 1 -- 172.21.15.57:0/2247791289 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c641f2e80 con 0x7f2c64153ab0 2023-10-30T14:14:03.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.884+0000 7f2c69809640 1 -- 172.21.15.57:0/2247791289 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c64148010 con 0x7f2c64153ab0 2023-10-30T14:14:03.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.884+0000 7f2c60ff9640 1 -- 172.21.15.57:0/2247791289 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2c5c004030 con 0x7f2c64153ab0 2023-10-30T14:14:03.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.884+0000 7f2c60ff9640 1 --2- 172.21.15.57:0/2247791289 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2c440664a0 0x7f2c44068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:03.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.884+0000 7f2c60ff9640 1 -- 172.21.15.57:0/2247791289 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f2c5c084150 con 0x7f2c64153ab0 2023-10-30T14:14:03.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.888+0000 7f2c62ffd640 1 --2- 172.21.15.57:0/2247791289 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2c440664a0 0x7f2c44068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:03.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.888+0000 7f2c62ffd640 1 --2- 172.21.15.57:0/2247791289 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2c440664a0 0x7f2c44068960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2c54005890 tx=0x7f2c54009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:03.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:03.892+0000 7f2c60ff9640 1 -- 172.21.15.57:0/2247791289 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2c64148010 con 0x7f2c64153ab0 2023-10-30T14:14:04.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.080+0000 7f2c69809640 1 -- 172.21.15.57:0/2247791289 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2c64148010 con 0x7f2c64153ab0 2023-10-30T14:14:04.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.696+0000 7f2c60ff9640 1 -- 172.21.15.57:0/2247791289 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f2c64148010 con 0x7f2c64153ab0 2023-10-30T14:14:04.700 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 2 and data pool 3 2023-10-30T14:14:04.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.696+0000 7f2c69809640 1 -- 172.21.15.57:0/2247791289 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2c440664a0 msgr2=0x7f2c44068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:04.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.696+0000 7f2c69809640 1 --2- 172.21.15.57:0/2247791289 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2c440664a0 0x7f2c44068960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2c54005890 tx=0x7f2c54009040 comp rx=0 tx=0).stop 2023-10-30T14:14:04.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.696+0000 7f2c69809640 1 -- 172.21.15.57:0/2247791289 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c64153ab0 msgr2=0x7f2c64148830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:04.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.696+0000 7f2c69809640 1 --2- 172.21.15.57:0/2247791289 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c64153ab0 0x7f2c64148830 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c007ef0 tx=0x7f2c5c00d4a0 comp rx=0 tx=0).stop 2023-10-30T14:14:04.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.696+0000 7f2c69809640 1 -- 172.21.15.57:0/2247791289 shutdown_connections 2023-10-30T14:14:04.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.700+0000 7f2c69809640 1 --2- 172.21.15.57:0/2247791289 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2c440664a0 0x7f2c44068960 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:04.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.700+0000 7f2c69809640 1 --2- 172.21.15.57:0/2247791289 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c64153ab0 0x7f2c64148830 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:04.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.700+0000 7f2c69809640 1 --2- 172.21.15.57:0/2247791289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c640ca370 0x7f2c641482f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:04.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.700+0000 7f2c69809640 1 --2- 172.21.15.57:0/2247791289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c640b1c80 0x7f2c641f2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:04.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.700+0000 7f2c69809640 1 -- 172.21.15.57:0/2247791289 >> 172.21.15.57:0/2247791289 conn(0x7f2c6406bef0 msgr2=0x7f2c64150690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:04.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.700+0000 7f2c69809640 1 -- 172.21.15.57:0/2247791289 shutdown_connections 2023-10-30T14:14:04.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.700+0000 7f2c69809640 1 -- 172.21.15.57:0/2247791289 wait complete. 2023-10-30T14:14:04.725 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:14:04.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.864+0000 7fd632bf5640 1 Processor -- start 2023-10-30T14:14:04.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.864+0000 7fd632bf5640 1 -- start start 2023-10-30T14:14:04.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.864+0000 7fd632bf5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd62c151840 0x7fd62c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:04.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.864+0000 7fd632bf5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd62c151c20 0x7fd62c072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:04.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.868+0000 7fd632bf5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd62c0c5030 con 0x7fd62c0cb8b0 2023-10-30T14:14:04.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.868+0000 7fd632bf5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd62c157830 con 0x7fd62c151c20 2023-10-30T14:14:04.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.868+0000 7fd632bf5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c157b10 con 0x7fd62c151840 2023-10-30T14:14:04.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.868+0000 7fd62bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd62c151c20 0x7fd62c072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:04.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.868+0000 7fd62bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd62c151c20 0x7fd62c072000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37780/0 (socket says 172.21.15.57:37780) 2023-10-30T14:14:04.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.868+0000 7fd62bfff640 1 -- 172.21.15.57:0/3216979088 learned_addr learned my addr 172.21.15.57:0/3216979088 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:04.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.868+0000 7fd630991640 1 --2- 172.21.15.57:0/3216979088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd62c151840 0x7fd62c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:04.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.868+0000 7fd62bfff640 1 -- 172.21.15.57:0/3216979088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd62c151840 msgr2=0x7fd62c0c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:04.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.868+0000 7fd62bfff640 1 --2- 172.21.15.57:0/3216979088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd62c151840 0x7fd62c0c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:04.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.868+0000 7fd62bfff640 1 -- 172.21.15.57:0/3216979088 >> v1:172.21.15.57:6789/0 conn(0x7fd62c0cb8b0 legacy=0x7fd62c06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:04.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.868+0000 7fd62bfff640 1 -- 172.21.15.57:0/3216979088 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd62c157df0 con 0x7fd62c151c20 2023-10-30T14:14:04.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.868+0000 7fd62bfff640 1 --2- 172.21.15.57:0/3216979088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62c151c20 0x7fd62c072000 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd618009870 tx=0x7fd61801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc298aef461dd7fe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:04.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.868+0000 7fd62b7fe640 1 -- 172.21.15.57:0/3216979088 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6180a8a30 con 0x7fd62c151c20 2023-10-30T14:14:04.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.868+0000 7fd62b7fe640 1 -- 172.21.15.57:0/3216979088 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6180a8b90 con 0x7fd62c151c20 2023-10-30T14:14:04.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.868+0000 7fd62b7fe640 1 -- 172.21.15.57:0/3216979088 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6180b0440 con 0x7fd62c151c20 2023-10-30T14:14:04.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd632bf5640 1 -- 172.21.15.57:0/3216979088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62c151c20 msgr2=0x7fd62c072000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:04.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd632bf5640 1 --2- 172.21.15.57:0/3216979088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62c151c20 0x7fd62c072000 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd618009870 tx=0x7fd61801ed30 comp rx=0 tx=0).stop 2023-10-30T14:14:04.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd632bf5640 1 -- 172.21.15.57:0/3216979088 shutdown_connections 2023-10-30T14:14:04.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd632bf5640 1 --2- 172.21.15.57:0/3216979088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62c151c20 0x7fd62c072000 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:04.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd632bf5640 1 --2- 172.21.15.57:0/3216979088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd62c151840 0x7fd62c0c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:04.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd632bf5640 1 -- 172.21.15.57:0/3216979088 >> 172.21.15.57:0/3216979088 conn(0x7fd62c0cac70 msgr2=0x7fd62c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:04.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd632bf5640 1 -- 172.21.15.57:0/3216979088 shutdown_connections 2023-10-30T14:14:04.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd632bf5640 1 -- 172.21.15.57:0/3216979088 wait complete. 2023-10-30T14:14:04.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd632bf5640 1 Processor -- start 2023-10-30T14:14:04.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd632bf5640 1 -- start start 2023-10-30T14:14:04.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd632bf5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62c0cb8b0 0x7fd62c152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:04.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd632bf5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd62c151840 0x7fd62c146500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:04.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd632bf5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd62c151c20 0x7fd62c156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:04.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd632bf5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c14a0a0 con 0x7fd62c0cb8b0 2023-10-30T14:14:04.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd632bf5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c1ec2f0 con 0x7fd62c151840 2023-10-30T14:14:04.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd632bf5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c14b0e0 con 0x7fd62c151c20 2023-10-30T14:14:04.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd630991640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62c0cb8b0 0x7fd62c152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:04.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd630991640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62c0cb8b0 0x7fd62c152030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37794/0 (socket says 172.21.15.57:37794) 2023-10-30T14:14:04.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd630991640 1 -- 172.21.15.57:0/989872169 learned_addr learned my addr 172.21.15.57:0/989872169 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:04.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd62bfff640 1 --2- 172.21.15.57:0/989872169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd62c151840 0x7fd62c146500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:04.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd631192640 1 --2- 172.21.15.57:0/989872169 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd62c151c20 0x7fd62c156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:04.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd630991640 1 -- 172.21.15.57:0/989872169 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd62c151c20 msgr2=0x7fd62c156570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:04.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd630991640 1 --2- 172.21.15.57:0/989872169 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd62c151c20 0x7fd62c156570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:04.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd630991640 1 -- 172.21.15.57:0/989872169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd62c151840 msgr2=0x7fd62c146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:04.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.872+0000 7fd630991640 1 --2- 172.21.15.57:0/989872169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd62c151840 0x7fd62c146500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:04.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.876+0000 7fd630991640 1 -- 172.21.15.57:0/989872169 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6180094d0 con 0x7fd62c0cb8b0 2023-10-30T14:14:04.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.876+0000 7fd630991640 1 --2- 172.21.15.57:0/989872169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62c0cb8b0 0x7fd62c152030 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd62000d800 tx=0x7fd62000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:04.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.876+0000 7fd6297fa640 1 -- 172.21.15.57:0/989872169 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62000b6c0 con 0x7fd62c0cb8b0 2023-10-30T14:14:04.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.876+0000 7fd6297fa640 1 -- 172.21.15.57:0/989872169 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62000b820 con 0x7fd62c0cb8b0 2023-10-30T14:14:04.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.876+0000 7fd6297fa640 1 -- 172.21.15.57:0/989872169 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62000bb90 con 0x7fd62c0cb8b0 2023-10-30T14:14:04.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.876+0000 7fd632bf5640 1 -- 172.21.15.57:0/989872169 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd62c14b3c0 con 0x7fd62c0cb8b0 2023-10-30T14:14:04.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.876+0000 7fd632bf5640 1 -- 172.21.15.57:0/989872169 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd62c1ec890 con 0x7fd62c0cb8b0 2023-10-30T14:14:04.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.876+0000 7fd632bf5640 1 -- 172.21.15.57:0/989872169 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd62c04e170 con 0x7fd62c0cb8b0 2023-10-30T14:14:04.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.876+0000 7fd6297fa640 1 -- 172.21.15.57:0/989872169 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd62000bcf0 con 0x7fd62c0cb8b0 2023-10-30T14:14:04.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.876+0000 7fd6297fa640 1 --2- 172.21.15.57:0/989872169 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd6100664d0 0x7fd610068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:04.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.876+0000 7fd6297fa640 1 -- 172.21.15.57:0/989872169 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6200843b0 con 0x7fd62c0cb8b0 2023-10-30T14:14:04.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.876+0000 7fd62bfff640 1 --2- 172.21.15.57:0/989872169 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd6100664d0 0x7fd610068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:04.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.880+0000 7fd62bfff640 1 --2- 172.21.15.57:0/989872169 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd6100664d0 0x7fd610068990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd6180b2000 tx=0x7fd618008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:04.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:04.880+0000 7fd6297fa640 1 -- 172.21.15.57:0/989872169 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd620051820 con 0x7fd62c0cb8b0 2023-10-30T14:14:05.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.040+0000 7fd632bf5640 1 -- 172.21.15.57:0/989872169 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd62c150ef0 con 0x7fd62c0cb8b0 2023-10-30T14:14:05.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.040+0000 7fd6297fa640 1 -- 172.21.15.57:0/989872169 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16446 (secure 0 0 0) 0x7fd620054e40 con 0x7fd62c0cb8b0 2023-10-30T14:14:05.045 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:05.045 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":20,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:14:04.086723+0000","last_up_change":"2023-10-30T14:13:51.171287+0000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T14:13:52.574639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T14:14:02.600122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T14:14:03.069429+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6801","nonce":3287367848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6803","nonce":3287367848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6807","nonce":3287367848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6805","nonce":3287367848}]},"public_addr":"172.21.15.57:6801/3287367848","cluster_addr":"172.21.15.57:6803/3287367848","heartbeat_back_addr":"172.21.15.57:6807/3287367848","heartbeat_front_addr":"172.21.15.57:6805/3287367848","state":["exists","up"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6809","nonce":3340797836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6811","nonce":3340797836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6815","nonce":3340797836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6813","nonce":3340797836}]},"public_addr":"172.21.15.57:6809/3340797836","cluster_addr":"172.21.15.57:6811/3340797836","heartbeat_back_addr":"172.21.15.57:6815/3340797836","heartbeat_front_addr":"172.21.15.57:6813/3340797836","state":["exists","up"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6825","nonce":1700292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6827","nonce":1700292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6831","nonce":1700292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6829","nonce":1700292}]},"public_addr":"172.21.15.57:6825/1700292","cluster_addr":"172.21.15.57:6827/1700292","heartbeat_back_addr":"172.21.15.57:6831/1700292","heartbeat_front_addr":"172.21.15.57:6829/1700292","state":["exists","up"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6817","nonce":2893420631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6819","nonce":2893420631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6823","nonce":2893420631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6821","nonce":2893420631}]},"public_addr":"172.21.15.57:6817/2893420631","cluster_addr":"172.21.15.57:6819/2893420631","heartbeat_back_addr":"172.21.15.57:6823/2893420631","heartbeat_front_addr":"172.21.15.57:6821/2893420631","state":["exists","up"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6807","nonce":2784816485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6811","nonce":2784816485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6815","nonce":2784816485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6813","nonce":2784816485}]},"public_addr":"172.21.15.88:6807/2784816485","cluster_addr":"172.21.15.88:6811/2784816485","heartbeat_back_addr":"172.21.15.88:6815/2784816485","heartbeat_front_addr":"172.21.15.88:6813/2784816485","state":["exists","up"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6801","nonce":2682449812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6803","nonce":2682449812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6810","nonce":2682449812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6806","nonce":2682449812}]},"public_addr":"172.21.15.88:6801/2682449812","cluster_addr":"172.21.15.88:6803/2682449812","heartbeat_back_addr":"172.21.15.88:6810/2682449812","heartbeat_front_addr":"172.21.15.88:6806/2682449812","state":["exists","up"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6822","nonce":3340693897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6827","nonce":3340693897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6831","nonce":3340693897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6829","nonce":3340693897}]},"public_addr":"172.21.15.88:6822/3340693897","cluster_addr":"172.21.15.88:6827/3340693897","heartbeat_back_addr":"172.21.15.88:6831/3340693897","heartbeat_front_addr":"172.21.15.88:6829/3340693897","state":["exists","up"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6817","nonce":2862111199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6820","nonce":2862111199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6825","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6826","nonce":2862111199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6821","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6823","nonce":2862111199}]},"public_addr":"172.21.15.88:6817/2862111199","cluster_addr":"172.21.15.88:6820/2862111199","heartbeat_back_addr":"172.21.15.88:6826/2862111199","heartbeat_front_addr":"172.21.15.88:6823/2862111199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.634920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.773993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.725825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.761452+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.694274+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.650718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:14:05.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.040+0000 7fd632bf5640 1 -- 172.21.15.57:0/989872169 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd6100664d0 msgr2=0x7fd610068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:05.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.040+0000 7fd632bf5640 1 --2- 172.21.15.57:0/989872169 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd6100664d0 0x7fd610068990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd6180b2000 tx=0x7fd618008040 comp rx=0 tx=0).stop 2023-10-30T14:14:05.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.040+0000 7fd632bf5640 1 -- 172.21.15.57:0/989872169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62c0cb8b0 msgr2=0x7fd62c152030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:05.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.040+0000 7fd632bf5640 1 --2- 172.21.15.57:0/989872169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62c0cb8b0 0x7fd62c152030 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd62000d800 tx=0x7fd62000dcc0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.040+0000 7fd632bf5640 1 -- 172.21.15.57:0/989872169 shutdown_connections 2023-10-30T14:14:05.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.040+0000 7fd632bf5640 1 --2- 172.21.15.57:0/989872169 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd6100664d0 0x7fd610068990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.040+0000 7fd632bf5640 1 --2- 172.21.15.57:0/989872169 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd62c151c20 0x7fd62c156570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.040+0000 7fd632bf5640 1 --2- 172.21.15.57:0/989872169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd62c151840 0x7fd62c146500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.040+0000 7fd632bf5640 1 --2- 172.21.15.57:0/989872169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62c0cb8b0 0x7fd62c152030 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.040+0000 7fd632bf5640 1 -- 172.21.15.57:0/989872169 >> 172.21.15.57:0/989872169 conn(0x7fd62c0cac70 msgr2=0x7fd62c0c7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:05.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.040+0000 7fd632bf5640 1 -- 172.21.15.57:0/989872169 shutdown_connections 2023-10-30T14:14:05.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.040+0000 7fd632bf5640 1 -- 172.21.15.57:0/989872169 wait complete. 2023-10-30T14:14:05.067 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:14:05.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.204+0000 7ff92989c640 1 Processor -- start 2023-10-30T14:14:05.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.204+0000 7ff92989c640 1 -- start start 2023-10-30T14:14:05.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.204+0000 7ff92989c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff924153c80 0x7ff92415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:05.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.204+0000 7ff92989c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff9240c42a0 0x7ff9240ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:05.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.204+0000 7ff92989c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff924159f20 con 0x7ff9240c4a10 2023-10-30T14:14:05.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.204+0000 7ff92989c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff92415dec0 con 0x7ff9240c42a0 2023-10-30T14:14:05.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.204+0000 7ff92989c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9241599a0 con 0x7ff924153c80 2023-10-30T14:14:05.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.208+0000 7ff9237fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff9240c42a0 0x7ff9240ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:05.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.208+0000 7ff922ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff924153c80 0x7ff92415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:05.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.208+0000 7ff9227fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff9240c4a10 0x7ff92415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:52508/0 (socket says 172.21.15.57:52508) 2023-10-30T14:14:05.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.208+0000 7ff9227fc640 1 -- 172.21.15.57:0/235675260 learned_addr learned my addr 172.21.15.57:0/235675260 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:05.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.208+0000 7ff9237fe640 1 -- 172.21.15.57:0/235675260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff924153c80 msgr2=0x7ff92415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:05.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.208+0000 7ff9237fe640 1 --2- 172.21.15.57:0/235675260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff924153c80 0x7ff92415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.208+0000 7ff9237fe640 1 -- 172.21.15.57:0/235675260 >> v1:172.21.15.88:6789/0 conn(0x7ff9240c4a10 legacy=0x7ff92415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:05.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.208+0000 7ff9237fe640 1 -- 172.21.15.57:0/235675260 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92415d620 con 0x7ff9240c42a0 2023-10-30T14:14:05.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.208+0000 7ff9237fe640 1 --2- 172.21.15.57:0/235675260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9240c42a0 0x7ff9240ca1e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff91c009d90 tx=0x7ff91c01f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e82ebc8407fdca74 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:05.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.208+0000 7ff921ffb640 1 -- 172.21.15.57:0/235675260 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c0af600 con 0x7ff9240c42a0 2023-10-30T14:14:05.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff921ffb640 1 -- 172.21.15.57:0/235675260 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91c0ab050 con 0x7ff9240c42a0 2023-10-30T14:14:05.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff921ffb640 1 -- 172.21.15.57:0/235675260 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c0af980 con 0x7ff9240c42a0 2023-10-30T14:14:05.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff92989c640 1 -- 172.21.15.57:0/235675260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9240c42a0 msgr2=0x7ff9240ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:05.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff92989c640 1 --2- 172.21.15.57:0/235675260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9240c42a0 0x7ff9240ca1e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff91c009d90 tx=0x7ff91c01f130 comp rx=0 tx=0).stop 2023-10-30T14:14:05.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff92989c640 1 -- 172.21.15.57:0/235675260 shutdown_connections 2023-10-30T14:14:05.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff92989c640 1 --2- 172.21.15.57:0/235675260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9240c42a0 0x7ff9240ca1e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff92989c640 1 --2- 172.21.15.57:0/235675260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff924153c80 0x7ff92415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff92989c640 1 -- 172.21.15.57:0/235675260 >> 172.21.15.57:0/235675260 conn(0x7ff9240715c0 msgr2=0x7ff9240719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:05.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff92989c640 1 -- 172.21.15.57:0/235675260 shutdown_connections 2023-10-30T14:14:05.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff92989c640 1 -- 172.21.15.57:0/235675260 wait complete. 2023-10-30T14:14:05.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff92989c640 1 Processor -- start 2023-10-30T14:14:05.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff92989c640 1 -- start start 2023-10-30T14:14:05.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff92989c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9240c42a0 0x7ff9240b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:05.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff92989c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9240c4a10 0x7ff924158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:05.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff92989c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff924153c80 0x7ff9240b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:05.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff92989c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92415c740 con 0x7ff9240c4a10 2023-10-30T14:14:05.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff92989c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9241ef880 con 0x7ff9240c42a0 2023-10-30T14:14:05.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff92989c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92415c970 con 0x7ff924153c80 2023-10-30T14:14:05.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff9227fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9240c4a10 0x7ff924158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:05.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff9227fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9240c4a10 0x7ff924158880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37810/0 (socket says 172.21.15.57:37810) 2023-10-30T14:14:05.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff9227fc640 1 -- 172.21.15.57:0/4220342950 learned_addr learned my addr 172.21.15.57:0/4220342950 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:05.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff922ffd640 1 --2- 172.21.15.57:0/4220342950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9240c42a0 0x7ff9240b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:05.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.212+0000 7ff9237fe640 1 --2- 172.21.15.57:0/4220342950 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff924153c80 0x7ff9240b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:05.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.216+0000 7ff9227fc640 1 -- 172.21.15.57:0/4220342950 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff924153c80 msgr2=0x7ff9240b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:05.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.216+0000 7ff9227fc640 1 --2- 172.21.15.57:0/4220342950 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff924153c80 0x7ff9240b3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.216+0000 7ff9227fc640 1 -- 172.21.15.57:0/4220342950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9240c42a0 msgr2=0x7ff9240b2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:05.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.216+0000 7ff9227fc640 1 --2- 172.21.15.57:0/4220342950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9240c42a0 0x7ff9240b2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.216+0000 7ff9227fc640 1 -- 172.21.15.57:0/4220342950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91c009a70 con 0x7ff9240c4a10 2023-10-30T14:14:05.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.216+0000 7ff9227fc640 1 --2- 172.21.15.57:0/4220342950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9240c4a10 0x7ff924158880 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff91800c430 tx=0x7ff91800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:05.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.216+0000 7ff90bfff640 1 -- 172.21.15.57:0/4220342950 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff918005cf0 con 0x7ff9240c4a10 2023-10-30T14:14:05.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.216+0000 7ff90bfff640 1 -- 172.21.15.57:0/4220342950 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff918005e50 con 0x7ff9240c4a10 2023-10-30T14:14:05.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.216+0000 7ff90bfff640 1 -- 172.21.15.57:0/4220342950 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9180072f0 con 0x7ff9240c4a10 2023-10-30T14:14:05.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.216+0000 7ff92989c640 1 -- 172.21.15.57:0/4220342950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9241eef50 con 0x7ff9240c4a10 2023-10-30T14:14:05.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.216+0000 7ff92989c640 1 -- 172.21.15.57:0/4220342950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9241ef510 con 0x7ff9240c4a10 2023-10-30T14:14:05.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.216+0000 7ff92989c640 1 -- 172.21.15.57:0/4220342950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff92404e170 con 0x7ff9240c4a10 2023-10-30T14:14:05.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.216+0000 7ff90bfff640 1 -- 172.21.15.57:0/4220342950 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff918007450 con 0x7ff9240c4a10 2023-10-30T14:14:05.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.216+0000 7ff90bfff640 1 --2- 172.21.15.57:0/4220342950 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff9100664a0 0x7ff910068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:05.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.216+0000 7ff90bfff640 1 -- 172.21.15.57:0/4220342950 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9180834e0 con 0x7ff9240c4a10 2023-10-30T14:14:05.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.220+0000 7ff922ffd640 1 --2- 172.21.15.57:0/4220342950 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff9100664a0 0x7ff910068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:05.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.220+0000 7ff922ffd640 1 --2- 172.21.15.57:0/4220342950 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff9100664a0 0x7ff910068960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff914009b60 tx=0x7ff914009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:05.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.220+0000 7ff90bfff640 1 -- 172.21.15.57:0/4220342950 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff9180508e0 con 0x7ff9240c4a10 2023-10-30T14:14:05.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.380+0000 7ff92989c640 1 -- 172.21.15.57:0/4220342950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9240cad40 con 0x7ff9240c4a10 2023-10-30T14:14:05.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.380+0000 7ff90bfff640 1 -- 172.21.15.57:0/4220342950 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16446 (secure 0 0 0) 0x7ff918017090 con 0x7ff9240c4a10 2023-10-30T14:14:05.386 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:05.386 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":20,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:14:04.086723+0000","last_up_change":"2023-10-30T14:13:51.171287+0000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T14:13:52.574639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T14:14:02.600122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T14:14:03.069429+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6801","nonce":3287367848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6803","nonce":3287367848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6807","nonce":3287367848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6805","nonce":3287367848}]},"public_addr":"172.21.15.57:6801/3287367848","cluster_addr":"172.21.15.57:6803/3287367848","heartbeat_back_addr":"172.21.15.57:6807/3287367848","heartbeat_front_addr":"172.21.15.57:6805/3287367848","state":["exists","up"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6809","nonce":3340797836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6811","nonce":3340797836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6815","nonce":3340797836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6813","nonce":3340797836}]},"public_addr":"172.21.15.57:6809/3340797836","cluster_addr":"172.21.15.57:6811/3340797836","heartbeat_back_addr":"172.21.15.57:6815/3340797836","heartbeat_front_addr":"172.21.15.57:6813/3340797836","state":["exists","up"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6825","nonce":1700292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6827","nonce":1700292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6831","nonce":1700292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6829","nonce":1700292}]},"public_addr":"172.21.15.57:6825/1700292","cluster_addr":"172.21.15.57:6827/1700292","heartbeat_back_addr":"172.21.15.57:6831/1700292","heartbeat_front_addr":"172.21.15.57:6829/1700292","state":["exists","up"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6817","nonce":2893420631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6819","nonce":2893420631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6823","nonce":2893420631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6821","nonce":2893420631}]},"public_addr":"172.21.15.57:6817/2893420631","cluster_addr":"172.21.15.57:6819/2893420631","heartbeat_back_addr":"172.21.15.57:6823/2893420631","heartbeat_front_addr":"172.21.15.57:6821/2893420631","state":["exists","up"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6807","nonce":2784816485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6811","nonce":2784816485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6815","nonce":2784816485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6813","nonce":2784816485}]},"public_addr":"172.21.15.88:6807/2784816485","cluster_addr":"172.21.15.88:6811/2784816485","heartbeat_back_addr":"172.21.15.88:6815/2784816485","heartbeat_front_addr":"172.21.15.88:6813/2784816485","state":["exists","up"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6801","nonce":2682449812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6803","nonce":2682449812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6810","nonce":2682449812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6806","nonce":2682449812}]},"public_addr":"172.21.15.88:6801/2682449812","cluster_addr":"172.21.15.88:6803/2682449812","heartbeat_back_addr":"172.21.15.88:6810/2682449812","heartbeat_front_addr":"172.21.15.88:6806/2682449812","state":["exists","up"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6822","nonce":3340693897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6827","nonce":3340693897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6831","nonce":3340693897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6829","nonce":3340693897}]},"public_addr":"172.21.15.88:6822/3340693897","cluster_addr":"172.21.15.88:6827/3340693897","heartbeat_back_addr":"172.21.15.88:6831/3340693897","heartbeat_front_addr":"172.21.15.88:6829/3340693897","state":["exists","up"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6817","nonce":2862111199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6820","nonce":2862111199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6825","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6826","nonce":2862111199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6821","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6823","nonce":2862111199}]},"public_addr":"172.21.15.88:6817/2862111199","cluster_addr":"172.21.15.88:6820/2862111199","heartbeat_back_addr":"172.21.15.88:6826/2862111199","heartbeat_front_addr":"172.21.15.88:6823/2862111199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.634920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.773993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.725825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.761452+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.694274+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.650718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:14:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.384+0000 7ff92989c640 1 -- 172.21.15.57:0/4220342950 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff9100664a0 msgr2=0x7ff910068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.384+0000 7ff92989c640 1 --2- 172.21.15.57:0/4220342950 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff9100664a0 0x7ff910068960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff914009b60 tx=0x7ff914009190 comp rx=0 tx=0).stop 2023-10-30T14:14:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.384+0000 7ff92989c640 1 -- 172.21.15.57:0/4220342950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9240c4a10 msgr2=0x7ff924158880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.384+0000 7ff92989c640 1 --2- 172.21.15.57:0/4220342950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9240c4a10 0x7ff924158880 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff91800c430 tx=0x7ff91800c8f0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.384+0000 7ff92989c640 1 -- 172.21.15.57:0/4220342950 shutdown_connections 2023-10-30T14:14:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.384+0000 7ff92989c640 1 --2- 172.21.15.57:0/4220342950 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff9100664a0 0x7ff910068960 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.384+0000 7ff92989c640 1 --2- 172.21.15.57:0/4220342950 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff924153c80 0x7ff9240b3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.384+0000 7ff92989c640 1 --2- 172.21.15.57:0/4220342950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9240c4a10 0x7ff924158880 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.384+0000 7ff92989c640 1 --2- 172.21.15.57:0/4220342950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9240c42a0 0x7ff9240b2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.384+0000 7ff92989c640 1 -- 172.21.15.57:0/4220342950 >> 172.21.15.57:0/4220342950 conn(0x7ff9240715c0 msgr2=0x7ff9240c5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.384+0000 7ff92989c640 1 -- 172.21.15.57:0/4220342950 shutdown_connections 2023-10-30T14:14:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.384+0000 7ff92989c640 1 -- 172.21.15.57:0/4220342950 wait complete. 2023-10-30T14:14:05.407 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-30T14:14:05.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.548+0000 7fa98946b640 1 Processor -- start 2023-10-30T14:14:05.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.548+0000 7fa98946b640 1 -- start start 2023-10-30T14:14:05.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.548+0000 7fa98946b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa984153ba0 0x7fa9840cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:05.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.548+0000 7fa98946b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa9840c4410 0x7fa98415a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:05.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.548+0000 7fa98946b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa9840c4810 0x7fa98415cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:05.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.548+0000 7fa98946b640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa984153f80 con 0x7fa984153ba0 2023-10-30T14:14:05.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.548+0000 7fa98946b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa984159df0 con 0x7fa9840c4810 2023-10-30T14:14:05.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.548+0000 7fa98946b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98415a7c0 con 0x7fa9840c4410 2023-10-30T14:14:05.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.548+0000 7fa982ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa984153ba0 0x7fa9840cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:05.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.548+0000 7fa982ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa984153ba0 0x7fa9840cb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37814/0 (socket says 172.21.15.57:37814) 2023-10-30T14:14:05.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.548+0000 7fa982ffd640 1 -- 172.21.15.57:0/1429984661 learned_addr learned my addr 172.21.15.57:0/1429984661 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:05.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.548+0000 7fa97bfff640 1 --2- 172.21.15.57:0/1429984661 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa9840c4410 0x7fa98415a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:05.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.548+0000 7fa9837fe640 1 --2- 172.21.15.57:0/1429984661 >> v2:172.21.15.88:3300/0 conn(0x7fa9840c4810 0x7fa98415cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:05.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.548+0000 7fa982ffd640 1 -- 172.21.15.57:0/1429984661 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa9840c4410 msgr2=0x7fa98415a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:05.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.548+0000 7fa982ffd640 1 --2- 172.21.15.57:0/1429984661 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa9840c4410 0x7fa98415a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.548+0000 7fa982ffd640 1 -- 172.21.15.57:0/1429984661 >> v2:172.21.15.88:3300/0 conn(0x7fa9840c4810 msgr2=0x7fa98415cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:05.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.548+0000 7fa982ffd640 1 --2- 172.21.15.57:0/1429984661 >> v2:172.21.15.88:3300/0 conn(0x7fa9840c4810 0x7fa98415cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.548+0000 7fa982ffd640 1 -- 172.21.15.57:0/1429984661 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98415d2f0 con 0x7fa984153ba0 2023-10-30T14:14:05.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.552+0000 7fa982ffd640 1 --2- 172.21.15.57:0/1429984661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984153ba0 0x7fa9840cb4e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa984066ee0 tx=0x7fa97401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d90a1782e6f643fc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:05.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.552+0000 7fa9827fc640 1 -- 172.21.15.57:0/1429984661 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9740a7070 con 0x7fa984153ba0 2023-10-30T14:14:05.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.552+0000 7fa9827fc640 1 -- 172.21.15.57:0/1429984661 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9740ab430 con 0x7fa984153ba0 2023-10-30T14:14:05.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.552+0000 7fa9827fc640 1 -- 172.21.15.57:0/1429984661 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9740ab7c0 con 0x7fa984153ba0 2023-10-30T14:14:05.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.552+0000 7fa98946b640 1 -- 172.21.15.57:0/1429984661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984153ba0 msgr2=0x7fa9840cb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:05.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.552+0000 7fa98946b640 1 --2- 172.21.15.57:0/1429984661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984153ba0 0x7fa9840cb4e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa984066ee0 tx=0x7fa97401c470 comp rx=0 tx=0).stop 2023-10-30T14:14:05.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.552+0000 7fa98946b640 1 -- 172.21.15.57:0/1429984661 shutdown_connections 2023-10-30T14:14:05.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.552+0000 7fa98946b640 1 --2- 172.21.15.57:0/1429984661 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa9840c4410 0x7fa98415a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.552+0000 7fa98946b640 1 --2- 172.21.15.57:0/1429984661 >> v2:172.21.15.88:3300/0 conn(0x7fa9840c4810 0x7fa98415cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.552+0000 7fa98946b640 1 --2- 172.21.15.57:0/1429984661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984153ba0 0x7fa9840cb4e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.552+0000 7fa98946b640 1 -- 172.21.15.57:0/1429984661 >> 172.21.15.57:0/1429984661 conn(0x7fa984071790 msgr2=0x7fa984071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:05.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.552+0000 7fa98946b640 1 -- 172.21.15.57:0/1429984661 shutdown_connections 2023-10-30T14:14:05.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.552+0000 7fa98946b640 1 -- 172.21.15.57:0/1429984661 wait complete. 2023-10-30T14:14:05.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa98946b640 1 Processor -- start 2023-10-30T14:14:05.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa98946b640 1 -- start start 2023-10-30T14:14:05.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa98946b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9840c4410 0x7fa984154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:05.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa98946b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa9840c4810 0x7fa9841f51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:05.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa98946b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984153ba0 0x7fa984156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:05.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa98946b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9840c7480 con 0x7fa984153ba0 2023-10-30T14:14:05.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa98946b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9840c7f50 con 0x7fa9840c4410 2023-10-30T14:14:05.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa98946b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9841570b0 con 0x7fa9840c4810 2023-10-30T14:14:05.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa9837fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984153ba0 0x7fa984156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:05.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa9837fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984153ba0 0x7fa984156b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37816/0 (socket says 172.21.15.57:37816) 2023-10-30T14:14:05.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa9837fe640 1 -- 172.21.15.57:0/2094287113 learned_addr learned my addr 172.21.15.57:0/2094287113 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:05.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa982ffd640 1 --2- 172.21.15.57:0/2094287113 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9840c4410 0x7fa984154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:05.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa97bfff640 1 --2- 172.21.15.57:0/2094287113 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa9840c4810 0x7fa9841f51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:05.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa9837fe640 1 -- 172.21.15.57:0/2094287113 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa9840c4810 msgr2=0x7fa9841f51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:05.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa9837fe640 1 --2- 172.21.15.57:0/2094287113 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa9840c4810 0x7fa9841f51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa9837fe640 1 -- 172.21.15.57:0/2094287113 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9840c4410 msgr2=0x7fa984154c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:05.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa9837fe640 1 --2- 172.21.15.57:0/2094287113 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9840c4410 0x7fa984154c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:05.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa9837fe640 1 -- 172.21.15.57:0/2094287113 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9740094d0 con 0x7fa984153ba0 2023-10-30T14:14:05.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa9837fe640 1 --2- 172.21.15.57:0/2094287113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984153ba0 0x7fa984156b70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa97c007e10 tx=0x7fa97c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:05.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa97b7fe640 1 -- 172.21.15.57:0/2094287113 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c014620 con 0x7fa984153ba0 2023-10-30T14:14:05.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa97b7fe640 1 -- 172.21.15.57:0/2094287113 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97c0147a0 con 0x7fa984153ba0 2023-10-30T14:14:05.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa98946b640 1 -- 172.21.15.57:0/2094287113 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9840c8300 con 0x7fa984153ba0 2023-10-30T14:14:05.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa97b7fe640 1 -- 172.21.15.57:0/2094287113 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c014b50 con 0x7fa984153ba0 2023-10-30T14:14:05.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.556+0000 7fa98946b640 1 -- 172.21.15.57:0/2094287113 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9840c88a0 con 0x7fa984153ba0 2023-10-30T14:14:05.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.560+0000 7fa98946b640 1 -- 172.21.15.57:0/2094287113 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa98404e170 con 0x7fa984153ba0 2023-10-30T14:14:05.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.560+0000 7fa97b7fe640 1 -- 172.21.15.57:0/2094287113 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa97c014cb0 con 0x7fa984153ba0 2023-10-30T14:14:05.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.560+0000 7fa97b7fe640 1 --2- 172.21.15.57:0/2094287113 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa9680664a0 0x7fa968068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:05.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.560+0000 7fa97b7fe640 1 -- 172.21.15.57:0/2094287113 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa97c084770 con 0x7fa984153ba0 2023-10-30T14:14:05.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.560+0000 7fa982ffd640 1 --2- 172.21.15.57:0/2094287113 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa9680664a0 0x7fa968068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:05.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.560+0000 7fa982ffd640 1 --2- 172.21.15.57:0/2094287113 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa9680664a0 0x7fa968068960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa9740af000 tx=0x7fa974008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:05.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.564+0000 7fa97b7fe640 1 -- 172.21.15.57:0/2094287113 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa97c051b50 con 0x7fa984153ba0 2023-10-30T14:14:05.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:05.760+0000 7fa98946b640 1 -- 172.21.15.57:0/2094287113 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa9840b1e50 con 0x7fa984153ba0 2023-10-30T14:14:06.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.720+0000 7fa97b7fe640 1 -- 172.21.15.57:0/2094287113 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa97c055170 con 0x7fa984153ba0 2023-10-30T14:14:06.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.724+0000 7fa98946b640 1 -- 172.21.15.57:0/2094287113 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa9680664a0 msgr2=0x7fa968068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:06.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.724+0000 7fa98946b640 1 --2- 172.21.15.57:0/2094287113 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa9680664a0 0x7fa968068960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa9740af000 tx=0x7fa974008040 comp rx=0 tx=0).stop 2023-10-30T14:14:06.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.724+0000 7fa98946b640 1 -- 172.21.15.57:0/2094287113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984153ba0 msgr2=0x7fa984156b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:06.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.724+0000 7fa98946b640 1 --2- 172.21.15.57:0/2094287113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984153ba0 0x7fa984156b70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa97c007e10 tx=0x7fa97c00e4b0 comp rx=0 tx=0).stop 2023-10-30T14:14:06.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.724+0000 7fa98946b640 1 -- 172.21.15.57:0/2094287113 shutdown_connections 2023-10-30T14:14:06.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.724+0000 7fa98946b640 1 --2- 172.21.15.57:0/2094287113 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa9680664a0 0x7fa968068960 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:06.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.724+0000 7fa98946b640 1 --2- 172.21.15.57:0/2094287113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984153ba0 0x7fa984156b70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:06.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.724+0000 7fa98946b640 1 --2- 172.21.15.57:0/2094287113 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa9840c4810 0x7fa9841f51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:06.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.724+0000 7fa98946b640 1 --2- 172.21.15.57:0/2094287113 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9840c4410 0x7fa984154c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:06.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.724+0000 7fa98946b640 1 -- 172.21.15.57:0/2094287113 >> 172.21.15.57:0/2094287113 conn(0x7fa984071790 msgr2=0x7fa9840c55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:06.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.724+0000 7fa98946b640 1 -- 172.21.15.57:0/2094287113 shutdown_connections 2023-10-30T14:14:06.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.724+0000 7fa98946b640 1 -- 172.21.15.57:0/2094287113 wait complete. 2023-10-30T14:14:06.752 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300} 2023-10-30T14:14:06.752 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-10-30T14:14:06.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.896+0000 7f20fa617640 1 Processor -- start 2023-10-30T14:14:06.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.896+0000 7f20fa617640 1 -- start start 2023-10-30T14:14:06.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.900+0000 7f20fa617640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20f4150630 0x7f20f40c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:06.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.900+0000 7f20fa617640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f20f40c1430 0x7f20f40cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:06.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.900+0000 7f20fa617640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f4157100 con 0x7f20f40c5be0 2023-10-30T14:14:06.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.900+0000 7f20fa617640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20f4150a10 con 0x7f20f40c1430 2023-10-30T14:14:06.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.900+0000 7f20fa617640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20f4072000 con 0x7f20f4150630 2023-10-30T14:14:06.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.900+0000 7f20f8bb4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f20f40c1430 0x7f20f40cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:06.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.900+0000 7f20f8bb4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f20f40c1430 0x7f20f40cacb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37824/0 (socket says 172.21.15.57:37824) 2023-10-30T14:14:06.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.900+0000 7f20f8bb4640 1 -- 172.21.15.57:0/497612283 learned_addr learned my addr 172.21.15.57:0/497612283 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:06.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.900+0000 7f20f3fff640 1 --2- 172.21.15.57:0/497612283 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20f4150630 0x7f20f40c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:06.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.900+0000 7f20f8bb4640 1 -- 172.21.15.57:0/497612283 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20f4150630 msgr2=0x7f20f40c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:06.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.900+0000 7f20f8bb4640 1 --2- 172.21.15.57:0/497612283 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20f4150630 0x7f20f40c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:06.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.900+0000 7f20f8bb4640 1 -- 172.21.15.57:0/497612283 >> v1:172.21.15.88:6789/0 conn(0x7f20f40c5be0 legacy=0x7f20f4159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:06.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.900+0000 7f20f8bb4640 1 -- 172.21.15.57:0/497612283 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f4156510 con 0x7f20f40c1430 2023-10-30T14:14:06.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.904+0000 7f20f8bb4640 1 --2- 172.21.15.57:0/497612283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f40c1430 0x7f20f40cacb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f20ec009e50 tx=0x7f20ec01c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=643f21fec8a6be23 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:06.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.904+0000 7f20f2ffd640 1 -- 172.21.15.57:0/497612283 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec0a5640 con 0x7f20f40c1430 2023-10-30T14:14:06.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.904+0000 7f20f2ffd640 1 -- 172.21.15.57:0/497612283 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20ec0a57a0 con 0x7f20f40c1430 2023-10-30T14:14:06.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.904+0000 7f20f2ffd640 1 -- 172.21.15.57:0/497612283 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec0a5b30 con 0x7f20f40c1430 2023-10-30T14:14:06.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.904+0000 7f20fa617640 1 -- 172.21.15.57:0/497612283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f40c1430 msgr2=0x7f20f40cacb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:06.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.904+0000 7f20fa617640 1 --2- 172.21.15.57:0/497612283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f40c1430 0x7f20f40cacb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f20ec009e50 tx=0x7f20ec01c980 comp rx=0 tx=0).stop 2023-10-30T14:14:06.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.904+0000 7f20fa617640 1 -- 172.21.15.57:0/497612283 shutdown_connections 2023-10-30T14:14:06.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.904+0000 7f20fa617640 1 --2- 172.21.15.57:0/497612283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f40c1430 0x7f20f40cacb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:06.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.904+0000 7f20fa617640 1 --2- 172.21.15.57:0/497612283 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20f4150630 0x7f20f40c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:06.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.904+0000 7f20fa617640 1 -- 172.21.15.57:0/497612283 >> 172.21.15.57:0/497612283 conn(0x7f20f406da40 msgr2=0x7f20f406de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:06.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.904+0000 7f20fa617640 1 -- 172.21.15.57:0/497612283 shutdown_connections 2023-10-30T14:14:06.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.904+0000 7f20fa617640 1 -- 172.21.15.57:0/497612283 wait complete. 2023-10-30T14:14:06.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.904+0000 7f20fa617640 1 Processor -- start 2023-10-30T14:14:06.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.904+0000 7f20fa617640 1 -- start start 2023-10-30T14:14:06.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20fa617640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20f40c1430 0x7f20f41f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:06.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20fa617640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f40c5be0 0x7f20f40c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:06.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20fa617640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f4150630 0x7f20f414f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:06.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20fa617640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f41f2f20 con 0x7f20f40c5be0 2023-10-30T14:14:06.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20fa617640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f414e680 con 0x7f20f4150630 2023-10-30T14:14:06.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20fa617640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20f414ef20 con 0x7f20f40c1430 2023-10-30T14:14:06.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20f37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f40c5be0 0x7f20f40c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:06.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20f37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f40c5be0 0x7f20f40c9db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37832/0 (socket says 172.21.15.57:37832) 2023-10-30T14:14:06.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20f37fe640 1 -- 172.21.15.57:0/1339972154 learned_addr learned my addr 172.21.15.57:0/1339972154 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:06.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20f3fff640 1 --2- 172.21.15.57:0/1339972154 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20f40c1430 0x7f20f41f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:06.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20f8bb4640 1 --2- 172.21.15.57:0/1339972154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f4150630 0x7f20f414f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:06.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20f37fe640 1 -- 172.21.15.57:0/1339972154 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20f40c1430 msgr2=0x7f20f41f29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:06.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20f37fe640 1 --2- 172.21.15.57:0/1339972154 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20f40c1430 0x7f20f41f29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:06.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20f37fe640 1 -- 172.21.15.57:0/1339972154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f4150630 msgr2=0x7f20f414f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:06.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20f37fe640 1 --2- 172.21.15.57:0/1339972154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f4150630 0x7f20f414f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:06.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20f37fe640 1 -- 172.21.15.57:0/1339972154 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ec009b30 con 0x7f20f40c5be0 2023-10-30T14:14:06.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20f37fe640 1 --2- 172.21.15.57:0/1339972154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f40c5be0 0x7f20f40c9db0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f20e800c530 tx=0x7f20e800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:06.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20f0ff9640 1 -- 172.21.15.57:0/1339972154 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e8005220 con 0x7f20f40c5be0 2023-10-30T14:14:06.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20f0ff9640 1 -- 172.21.15.57:0/1339972154 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e8005380 con 0x7f20f40c5be0 2023-10-30T14:14:06.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20f0ff9640 1 -- 172.21.15.57:0/1339972154 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e8007270 con 0x7f20f40c5be0 2023-10-30T14:14:06.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20fa617640 1 -- 172.21.15.57:0/1339972154 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20f414dde0 con 0x7f20f40c5be0 2023-10-30T14:14:06.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20fa617640 1 -- 172.21.15.57:0/1339972154 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20f414e3a0 con 0x7f20f40c5be0 2023-10-30T14:14:06.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20fa617640 1 -- 172.21.15.57:0/1339972154 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20f404e170 con 0x7f20f40c5be0 2023-10-30T14:14:06.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.908+0000 7f20f0ff9640 1 -- 172.21.15.57:0/1339972154 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f20e80073d0 con 0x7f20f40c5be0 2023-10-30T14:14:06.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.912+0000 7f20f0ff9640 1 --2- 172.21.15.57:0/1339972154 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20e00664a0 0x7f20e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:06.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.912+0000 7f20f0ff9640 1 -- 172.21.15.57:0/1339972154 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f20e8083790 con 0x7f20f40c5be0 2023-10-30T14:14:06.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.912+0000 7f20f3fff640 1 --2- 172.21.15.57:0/1339972154 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20e00664a0 0x7f20e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:06.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.912+0000 7f20f3fff640 1 --2- 172.21.15.57:0/1339972154 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20e00664a0 0x7f20e0068960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f20e400b930 tx=0x7f20e4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:06.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:06.916+0000 7f20f0ff9640 1 -- 172.21.15.57:0/1339972154 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f20e8050c00 con 0x7f20f40c5be0 2023-10-30T14:14:07.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.108+0000 7f20fa617640 1 -- 172.21.15.57:0/1339972154 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f20f40b0c80 con 0x7f20f40c5be0 2023-10-30T14:14:07.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.724+0000 7f20f0ff9640 1 -- 172.21.15.57:0/1339972154 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f20e8054220 con 0x7f20f40c5be0 2023-10-30T14:14:07.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.724+0000 7f20fa617640 1 -- 172.21.15.57:0/1339972154 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20e00664a0 msgr2=0x7f20e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:07.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.724+0000 7f20fa617640 1 --2- 172.21.15.57:0/1339972154 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20e00664a0 0x7f20e0068960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f20e400b930 tx=0x7f20e4008040 comp rx=0 tx=0).stop 2023-10-30T14:14:07.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.724+0000 7f20fa617640 1 -- 172.21.15.57:0/1339972154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f40c5be0 msgr2=0x7f20f40c9db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:07.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.724+0000 7f20fa617640 1 --2- 172.21.15.57:0/1339972154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f40c5be0 0x7f20f40c9db0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f20e800c530 tx=0x7f20e800c9f0 comp rx=0 tx=0).stop 2023-10-30T14:14:07.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.724+0000 7f20fa617640 1 -- 172.21.15.57:0/1339972154 shutdown_connections 2023-10-30T14:14:07.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.724+0000 7f20fa617640 1 --2- 172.21.15.57:0/1339972154 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20e00664a0 0x7f20e0068960 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:07.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.724+0000 7f20fa617640 1 --2- 172.21.15.57:0/1339972154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f4150630 0x7f20f414f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:07.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.724+0000 7f20fa617640 1 --2- 172.21.15.57:0/1339972154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f40c5be0 0x7f20f40c9db0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:07.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.724+0000 7f20fa617640 1 --2- 172.21.15.57:0/1339972154 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20f40c1430 0x7f20f41f29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:07.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.724+0000 7f20fa617640 1 -- 172.21.15.57:0/1339972154 >> 172.21.15.57:0/1339972154 conn(0x7f20f406da40 msgr2=0x7f20f40c09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:07.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.724+0000 7f20fa617640 1 -- 172.21.15.57:0/1339972154 shutdown_connections 2023-10-30T14:14:07.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.724+0000 7f20fa617640 1 -- 172.21.15.57:0/1339972154 wait complete. 2023-10-30T14:14:07.754 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-30T14:14:07.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.900+0000 7f9533f65640 1 Processor -- start 2023-10-30T14:14:07.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.900+0000 7f9533f65640 1 -- start start 2023-10-30T14:14:07.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.900+0000 7f9533f65640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f952c150630 0x7f952c0c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:07.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.900+0000 7f9533f65640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f952c0c5be0 0x7f952c159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:07.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.900+0000 7f9533f65640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f952c157100 con 0x7f952c0c1430 2023-10-30T14:14:07.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.900+0000 7f9533f65640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f952c150a10 con 0x7f952c0c5be0 2023-10-30T14:14:07.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.900+0000 7f9533f65640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f952c072000 con 0x7f952c150630 2023-10-30T14:14:07.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.900+0000 7f9532502640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f952c0c1430 0x7f952c0cacb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46234/0 (socket says 172.21.15.57:46234) 2023-10-30T14:14:07.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.900+0000 7f9531500640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f952c0c5be0 0x7f952c159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:07.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.900+0000 7f9532502640 1 -- 172.21.15.57:0/2029867000 learned_addr learned my addr 172.21.15.57:0/2029867000 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:07.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.900+0000 7f9531d01640 1 --2- 172.21.15.57:0/2029867000 >> v2:172.21.15.88:3300/0 conn(0x7f952c150630 0x7f952c0c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:07.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.900+0000 7f9531500640 1 -- 172.21.15.57:0/2029867000 >> v2:172.21.15.88:3300/0 conn(0x7f952c150630 msgr2=0x7f952c0c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:07.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.900+0000 7f9531500640 1 --2- 172.21.15.57:0/2029867000 >> v2:172.21.15.88:3300/0 conn(0x7f952c150630 0x7f952c0c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:07.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.900+0000 7f9530cff640 1 -- 172.21.15.57:0/2029867000 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590030856 0 0) 0x7f952c157100 con 0x7f952c0c1430 2023-10-30T14:14:07.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.900+0000 7f9531500640 1 -- 172.21.15.57:0/2029867000 >> v1:172.21.15.57:6789/0 conn(0x7f952c0c1430 legacy=0x7f952c0cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:07.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.900+0000 7f9531500640 1 -- 172.21.15.57:0/2029867000 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952c156510 con 0x7f952c0c5be0 2023-10-30T14:14:07.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.904+0000 7f9531500640 1 --2- 172.21.15.57:0/2029867000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c0c5be0 0x7f952c159b00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f952c0c13c0 tx=0x7f951c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3be6695c38adcdd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:07.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.904+0000 7f9530cff640 1 -- 172.21.15.57:0/2029867000 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c0ab600 con 0x7f952c0c5be0 2023-10-30T14:14:07.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.904+0000 7f9530cff640 1 -- 172.21.15.57:0/2029867000 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951c0a7050 con 0x7f952c0c5be0 2023-10-30T14:14:07.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.904+0000 7f9530cff640 1 -- 172.21.15.57:0/2029867000 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c0ab980 con 0x7f952c0c5be0 2023-10-30T14:14:07.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.904+0000 7f9533f65640 1 -- 172.21.15.57:0/2029867000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c0c5be0 msgr2=0x7f952c159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:07.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.904+0000 7f9533f65640 1 --2- 172.21.15.57:0/2029867000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c0c5be0 0x7f952c159b00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f952c0c13c0 tx=0x7f951c01c3d0 comp rx=0 tx=0).stop 2023-10-30T14:14:07.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.904+0000 7f9533f65640 1 -- 172.21.15.57:0/2029867000 shutdown_connections 2023-10-30T14:14:07.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.904+0000 7f9533f65640 1 --2- 172.21.15.57:0/2029867000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c0c5be0 0x7f952c159b00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:07.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.904+0000 7f9533f65640 1 --2- 172.21.15.57:0/2029867000 >> v2:172.21.15.88:3300/0 conn(0x7f952c150630 0x7f952c0c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:07.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.904+0000 7f9533f65640 1 -- 172.21.15.57:0/2029867000 >> 172.21.15.57:0/2029867000 conn(0x7f952c06da40 msgr2=0x7f952c06de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:07.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.904+0000 7f9533f65640 1 -- 172.21.15.57:0/2029867000 shutdown_connections 2023-10-30T14:14:07.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.904+0000 7f9533f65640 1 -- 172.21.15.57:0/2029867000 wait complete. 2023-10-30T14:14:07.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9533f65640 1 Processor -- start 2023-10-30T14:14:07.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9533f65640 1 -- start start 2023-10-30T14:14:07.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9533f65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f952c0c1430 0x7f952c1f5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:07.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9533f65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f952c0c5be0 0x7f952c158f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:07.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9533f65640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c150630 0x7f952c159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:07.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9533f65640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c1f6560 con 0x7f952c150630 2023-10-30T14:14:07.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9533f65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c158a90 con 0x7f952c0c5be0 2023-10-30T14:14:07.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9533f65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952c14ca30 con 0x7f952c0c1430 2023-10-30T14:14:07.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9532502640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c150630 0x7f952c159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:07.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9532502640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c150630 0x7f952c159460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37846/0 (socket says 172.21.15.57:37846) 2023-10-30T14:14:07.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9532502640 1 -- 172.21.15.57:0/2751462448 learned_addr learned my addr 172.21.15.57:0/2751462448 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:07.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9531500640 1 --2- 172.21.15.57:0/2751462448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f952c0c5be0 0x7f952c158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:07.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9531d01640 1 --2- 172.21.15.57:0/2751462448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f952c0c1430 0x7f952c1f5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:07.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9532502640 1 -- 172.21.15.57:0/2751462448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f952c0c1430 msgr2=0x7f952c1f5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:07.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9532502640 1 --2- 172.21.15.57:0/2751462448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f952c0c1430 0x7f952c1f5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:07.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9532502640 1 -- 172.21.15.57:0/2751462448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f952c0c5be0 msgr2=0x7f952c158f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:07.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9532502640 1 --2- 172.21.15.57:0/2751462448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f952c0c5be0 0x7f952c158f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:07.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9532502640 1 -- 172.21.15.57:0/2751462448 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951c009420 con 0x7f952c150630 2023-10-30T14:14:07.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9532502640 1 --2- 172.21.15.57:0/2751462448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c150630 0x7f952c159460 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f952400c7b0 tx=0x7f952400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:07.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f95227fc640 1 -- 172.21.15.57:0/2751462448 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952400a640 con 0x7f952c150630 2023-10-30T14:14:07.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f95227fc640 1 -- 172.21.15.57:0/2751462448 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f952400a7c0 con 0x7f952c150630 2023-10-30T14:14:07.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9533f65640 1 -- 172.21.15.57:0/2751462448 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f952c1f62c0 con 0x7f952c150630 2023-10-30T14:14:07.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f95227fc640 1 -- 172.21.15.57:0/2751462448 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952400ab70 con 0x7f952c150630 2023-10-30T14:14:07.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9533f65640 1 -- 172.21.15.57:0/2751462448 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f952c14c750 con 0x7f952c150630 2023-10-30T14:14:07.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.908+0000 7f9533f65640 1 -- 172.21.15.57:0/2751462448 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f952c158c40 con 0x7f952c150630 2023-10-30T14:14:07.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.912+0000 7f95227fc640 1 -- 172.21.15.57:0/2751462448 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f952400acd0 con 0x7f952c150630 2023-10-30T14:14:07.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.912+0000 7f95227fc640 1 --2- 172.21.15.57:0/2751462448 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95100664a0 0x7f9510068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:07.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.912+0000 7f95227fc640 1 -- 172.21.15.57:0/2751462448 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9524083580 con 0x7f952c150630 2023-10-30T14:14:07.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.912+0000 7f9531d01640 1 --2- 172.21.15.57:0/2751462448 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95100664a0 0x7f9510068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:07.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.912+0000 7f9531d01640 1 --2- 172.21.15.57:0/2751462448 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95100664a0 0x7f9510068960 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9528007a60 tx=0x7f9528008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:07.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:07.916+0000 7f95227fc640 1 -- 172.21.15.57:0/2751462448 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f952c158c40 con 0x7f952c150630 2023-10-30T14:14:08.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.108+0000 7f9533f65640 1 -- 172.21.15.57:0/2751462448 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f952c158c40 con 0x7f952c150630 2023-10-30T14:14:08.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.736+0000 7f95227fc640 1 -- 172.21.15.57:0/2751462448 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7f952c158c40 con 0x7f952c150630 2023-10-30T14:14:08.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.736+0000 7f9533f65640 1 -- 172.21.15.57:0/2751462448 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95100664a0 msgr2=0x7f9510068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:08.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.736+0000 7f9533f65640 1 --2- 172.21.15.57:0/2751462448 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95100664a0 0x7f9510068960 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9528007a60 tx=0x7f9528008040 comp rx=0 tx=0).stop 2023-10-30T14:14:08.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.736+0000 7f9533f65640 1 -- 172.21.15.57:0/2751462448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c150630 msgr2=0x7f952c159460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:08.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.736+0000 7f9533f65640 1 --2- 172.21.15.57:0/2751462448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c150630 0x7f952c159460 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f952400c7b0 tx=0x7f952400cc70 comp rx=0 tx=0).stop 2023-10-30T14:14:08.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.736+0000 7f9533f65640 1 -- 172.21.15.57:0/2751462448 shutdown_connections 2023-10-30T14:14:08.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.736+0000 7f9533f65640 1 --2- 172.21.15.57:0/2751462448 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95100664a0 0x7f9510068960 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:08.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.736+0000 7f9533f65640 1 --2- 172.21.15.57:0/2751462448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c150630 0x7f952c159460 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:08.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.736+0000 7f9533f65640 1 --2- 172.21.15.57:0/2751462448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f952c0c5be0 0x7f952c158f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:08.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.736+0000 7f9533f65640 1 --2- 172.21.15.57:0/2751462448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f952c0c1430 0x7f952c1f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:08.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.736+0000 7f9533f65640 1 -- 172.21.15.57:0/2751462448 >> 172.21.15.57:0/2751462448 conn(0x7f952c06da40 msgr2=0x7f952c155b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:08.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.740+0000 7f9533f65640 1 -- 172.21.15.57:0/2751462448 shutdown_connections 2023-10-30T14:14:08.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.740+0000 7f9533f65640 1 -- 172.21.15.57:0/2751462448 wait complete. 2023-10-30T14:14:08.766 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-10-30T14:14:08.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.912+0000 7f2169b65640 1 Processor -- start 2023-10-30T14:14:08.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.912+0000 7f2169b65640 1 -- start start 2023-10-30T14:14:08.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.912+0000 7f2169b65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21640c4410 0x7f21640cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:08.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.912+0000 7f2169b65640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2164153f80 con 0x7f2164153ba0 2023-10-30T14:14:08.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.912+0000 7f2169b65640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2164159df0 con 0x7f21640c4810 2023-10-30T14:14:08.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.912+0000 7f2169b65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21641597c0 con 0x7f21640c4410 2023-10-30T14:14:08.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.912+0000 7f21637fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f2164153ba0 0x7f216415cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46246/0 (socket says 172.21.15.57:46246) 2023-10-30T14:14:08.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.912+0000 7f21637fe640 1 -- 172.21.15.57:0/1164594690 learned_addr learned my addr 172.21.15.57:0/1164594690 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:08.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.912+0000 7f2162ffd640 1 --2- 172.21.15.57:0/1164594690 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21640c4410 0x7f21640cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:08.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21627fc640 1 -- 172.21.15.57:0/1164594690 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2005535084 0 0) 0x7f2164153f80 con 0x7f2164153ba0 2023-10-30T14:14:08.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21627fc640 1 -- 172.21.15.57:0/1164594690 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f214c003530 con 0x7f2164153ba0 2023-10-30T14:14:08.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21627fc640 1 -- 172.21.15.57:0/1164594690 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1029137363 0 0) 0x7f214c003530 con 0x7f2164153ba0 2023-10-30T14:14:08.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21627fc640 1 -- 172.21.15.57:0/1164594690 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21640c4410 msgr2=0x7f21640cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:08.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21627fc640 1 --2- 172.21.15.57:0/1164594690 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21640c4410 0x7f21640cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:08.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21627fc640 1 -- 172.21.15.57:0/1164594690 >> v1:172.21.15.88:6789/0 conn(0x7f21640c4810 legacy=0x7f21640cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:08.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21627fc640 1 -- 172.21.15.57:0/1164594690 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216415d580 con 0x7f2164153ba0 2023-10-30T14:14:08.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21627fc640 1 -- 172.21.15.57:0/1164594690 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f2154001680 con 0x7f2164153ba0 2023-10-30T14:14:08.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21627fc640 1 -- 172.21.15.57:0/1164594690 >> v1:172.21.15.57:6789/0 conn(0x7f2164153ba0 legacy=0x7f216415cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:08.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21627fc640 1 --2- 172.21.15.57:0/1164594690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f214c0039b0 0x7f214c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:08.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21627fc640 1 --2- 172.21.15.57:0/1164594690 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21640c4410 0x7f214c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:08.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21627fc640 1 --2- 172.21.15.57:0/1164594690 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214c004d10 0x7f214c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:08.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21627fc640 1 -- 172.21.15.57:0/1164594690 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21641597c0 con 0x7f214c0039b0 2023-10-30T14:14:08.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21627fc640 1 -- 172.21.15.57:0/1164594690 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214c00bac0 con 0x7f214c004d10 2023-10-30T14:14:08.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21627fc640 1 -- 172.21.15.57:0/1164594690 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214c00bc80 con 0x7f21640c4410 2023-10-30T14:14:08.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21637fe640 1 --2- 172.21.15.57:0/1164594690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f214c0039b0 0x7f214c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:08.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f2162ffd640 1 --2- 172.21.15.57:0/1164594690 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21640c4410 0x7f214c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:08.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f2163fff640 1 --2- 172.21.15.57:0/1164594690 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214c004d10 0x7f214c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:08.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21637fe640 1 -- 172.21.15.57:0/1164594690 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21640c4410 msgr2=0x7f214c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:08.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21637fe640 1 --2- 172.21.15.57:0/1164594690 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21640c4410 0x7f214c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:08.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21637fe640 1 -- 172.21.15.57:0/1164594690 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214c004d10 msgr2=0x7f214c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:08.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21637fe640 1 --2- 172.21.15.57:0/1164594690 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214c004d10 0x7f214c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:08.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f21637fe640 1 -- 172.21.15.57:0/1164594690 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214c003530 con 0x7f214c0039b0 2023-10-30T14:14:08.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.916+0000 7f2163fff640 1 --2- 172.21.15.57:0/1164594690 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214c004d10 0x7f214c005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:08.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.920+0000 7f21637fe640 1 --2- 172.21.15.57:0/1164594690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f214c0039b0 0x7f214c003dd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f215400c240 tx=0x7f215401cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=582efd6909435c67 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:08.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.920+0000 7f21627fc640 1 -- 172.21.15.57:0/1164594690 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2154005ae0 con 0x7f214c0039b0 2023-10-30T14:14:08.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.920+0000 7f21627fc640 1 -- 172.21.15.57:0/1164594690 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2154005c40 con 0x7f214c0039b0 2023-10-30T14:14:08.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.920+0000 7f21627fc640 1 -- 172.21.15.57:0/1164594690 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21540072b0 con 0x7f214c0039b0 2023-10-30T14:14:08.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.920+0000 7f2169b65640 1 -- 172.21.15.57:0/1164594690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f214c0039b0 msgr2=0x7f214c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:08.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.920+0000 7f2169b65640 1 --2- 172.21.15.57:0/1164594690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f214c0039b0 0x7f214c003dd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f215400c240 tx=0x7f215401cb50 comp rx=0 tx=0).stop 2023-10-30T14:14:08.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.920+0000 7f2163fff640 1 -- 172.21.15.57:0/1164594690 reap_dead start 2023-10-30T14:14:08.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.920+0000 7f2169b65640 1 -- 172.21.15.57:0/1164594690 shutdown_connections 2023-10-30T14:14:08.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.920+0000 7f2169b65640 1 -- 172.21.15.57:0/1164594690 >> 172.21.15.57:0/1164594690 conn(0x7f2164071790 msgr2=0x7f2164071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:08.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.920+0000 7f2169b65640 1 -- 172.21.15.57:0/1164594690 shutdown_connections 2023-10-30T14:14:08.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.920+0000 7f2169b65640 1 -- 172.21.15.57:0/1164594690 wait complete. 2023-10-30T14:14:08.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.920+0000 7f2169b65640 1 Processor -- start 2023-10-30T14:14:08.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.920+0000 7f2169b65640 1 -- start start 2023-10-30T14:14:08.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.920+0000 7f2169b65640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f214c0039b0 0x7f216415e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:08.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.920+0000 7f2169b65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f216415fcd0 0x7f21640ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:08.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.920+0000 7f2169b65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f216415c740 0x7f21641f5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:08.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.920+0000 7f2169b65640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216415c170 con 0x7f214c0039b0 2023-10-30T14:14:08.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.920+0000 7f2169b65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216415e3a0 con 0x7f216415fcd0 2023-10-30T14:14:08.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.920+0000 7f2169b65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f216415b220 con 0x7f216415c740 2023-10-30T14:14:08.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f21637fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f214c0039b0 0x7f216415e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:08.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f21637fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f214c0039b0 0x7f216415e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37864/0 (socket says 172.21.15.57:37864) 2023-10-30T14:14:08.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f21637fe640 1 -- 172.21.15.57:0/828070409 learned_addr learned my addr 172.21.15.57:0/828070409 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:08.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f2162ffd640 1 --2- 172.21.15.57:0/828070409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f216415fcd0 0x7f21640ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:08.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f2163fff640 1 --2- 172.21.15.57:0/828070409 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f216415c740 0x7f21641f5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:08.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f21637fe640 1 -- 172.21.15.57:0/828070409 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f216415c740 msgr2=0x7f21641f5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:08.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f21637fe640 1 --2- 172.21.15.57:0/828070409 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f216415c740 0x7f21641f5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:08.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f21637fe640 1 -- 172.21.15.57:0/828070409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f216415fcd0 msgr2=0x7f21640ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:08.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f21637fe640 1 --2- 172.21.15.57:0/828070409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f216415fcd0 0x7f21640ce2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:08.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f21637fe640 1 -- 172.21.15.57:0/828070409 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216415a7f0 con 0x7f214c0039b0 2023-10-30T14:14:08.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f2162ffd640 1 --2- 172.21.15.57:0/828070409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f216415fcd0 0x7f21640ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:08.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f21637fe640 1 --2- 172.21.15.57:0/828070409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f214c0039b0 0x7f216415e830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f215401c5c0 tx=0x7f214c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:08.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f214bfff640 1 -- 172.21.15.57:0/828070409 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21540b4070 con 0x7f214c0039b0 2023-10-30T14:14:08.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f214bfff640 1 -- 172.21.15.57:0/828070409 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2154005e70 con 0x7f214c0039b0 2023-10-30T14:14:08.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f214bfff640 1 -- 172.21.15.57:0/828070409 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2154001640 con 0x7f214c0039b0 2023-10-30T14:14:08.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f2169b65640 1 -- 172.21.15.57:0/828070409 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f216415a980 con 0x7f214c0039b0 2023-10-30T14:14:08.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f2169b65640 1 -- 172.21.15.57:0/828070409 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216415af40 con 0x7f214c0039b0 2023-10-30T14:14:08.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f2169b65640 1 -- 172.21.15.57:0/828070409 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f216404e170 con 0x7f214c0039b0 2023-10-30T14:14:08.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f214bfff640 1 -- 172.21.15.57:0/828070409 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f21540047f0 con 0x7f214c0039b0 2023-10-30T14:14:08.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.924+0000 7f214bfff640 1 --2- 172.21.15.57:0/828070409 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f21500664d0 0x7f2150068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:08.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.928+0000 7f214bfff640 1 -- 172.21.15.57:0/828070409 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f215411d2c0 con 0x7f214c0039b0 2023-10-30T14:14:08.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.928+0000 7f2162ffd640 1 --2- 172.21.15.57:0/828070409 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f21500664d0 0x7f2150068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:08.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.928+0000 7f2162ffd640 1 --2- 172.21.15.57:0/828070409 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f21500664d0 0x7f2150068990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2164149cc0 tx=0x7f2158008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:08.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:08.928+0000 7f214bfff640 1 -- 172.21.15.57:0/828070409 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f21540ea6a0 con 0x7f214c0039b0 2023-10-30T14:14:09.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.120+0000 7f2169b65640 1 -- 172.21.15.57:0/828070409 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f21640cabb0 con 0x7f214c0039b0 2023-10-30T14:14:09.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.736+0000 7f214bfff640 1 -- 172.21.15.57:0/828070409 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v8) v1 ==== 115+0+0 (secure 0 0 0) 0x7f21540edcc0 con 0x7f214c0039b0 2023-10-30T14:14:09.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.740+0000 7f2169b65640 1 -- 172.21.15.57:0/828070409 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f21500664d0 msgr2=0x7f2150068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:09.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.740+0000 7f2169b65640 1 --2- 172.21.15.57:0/828070409 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f21500664d0 0x7f2150068990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2164149cc0 tx=0x7f2158008040 comp rx=0 tx=0).stop 2023-10-30T14:14:09.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.740+0000 7f2169b65640 1 -- 172.21.15.57:0/828070409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f214c0039b0 msgr2=0x7f216415e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:09.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.740+0000 7f2169b65640 1 --2- 172.21.15.57:0/828070409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f214c0039b0 0x7f216415e830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f215401c5c0 tx=0x7f214c005800 comp rx=0 tx=0).stop 2023-10-30T14:14:09.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.740+0000 7f2169b65640 1 -- 172.21.15.57:0/828070409 shutdown_connections 2023-10-30T14:14:09.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.740+0000 7f2169b65640 1 --2- 172.21.15.57:0/828070409 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f21500664d0 0x7f2150068990 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:09.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.740+0000 7f2169b65640 1 --2- 172.21.15.57:0/828070409 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f216415c740 0x7f21641f5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:09.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.740+0000 7f2169b65640 1 --2- 172.21.15.57:0/828070409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f216415fcd0 0x7f21640ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:09.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.740+0000 7f2169b65640 1 --2- 172.21.15.57:0/828070409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f214c0039b0 0x7f216415e830 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:09.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.740+0000 7f2169b65640 1 -- 172.21.15.57:0/828070409 >> 172.21.15.57:0/828070409 conn(0x7f2164071790 msgr2=0x7f2164152ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:09.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.740+0000 7f2169b65640 1 -- 172.21.15.57:0/828070409 shutdown_connections 2023-10-30T14:14:09.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.740+0000 7f2169b65640 1 -- 172.21.15.57:0/828070409 wait complete. 2023-10-30T14:14:09.768 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-30T14:14:09.769 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:09.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.908+0000 7f1559565640 1 Processor -- start 2023-10-30T14:14:09.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.908+0000 7f1559565640 1 -- start start 2023-10-30T14:14:09.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.912+0000 7f1559565640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1554151840 0x7f15540c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:09.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.912+0000 7f1559565640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f15540cb8b0 0x7f155406b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:09.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.912+0000 7f1559565640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15540c5030 con 0x7f1554151c20 2023-10-30T14:14:09.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.912+0000 7f1559565640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1554157830 con 0x7f1554151840 2023-10-30T14:14:09.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.912+0000 7f1559565640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1554157b10 con 0x7f15540cb8b0 2023-10-30T14:14:09.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.912+0000 7f154bfff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1554151c20 0x7f1554072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46252/0 (socket says 172.21.15.57:46252) 2023-10-30T14:14:09.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.912+0000 7f1552ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1554151840 0x7f15540c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:09.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.912+0000 7f154bfff640 1 -- 172.21.15.57:0/292738739 learned_addr learned my addr 172.21.15.57:0/292738739 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:09.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.912+0000 7f15537fe640 1 --2- 172.21.15.57:0/292738739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f15540cb8b0 0x7f155406b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:09.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.912+0000 7f1552ffd640 1 -- 172.21.15.57:0/292738739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f15540cb8b0 msgr2=0x7f155406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:09.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.912+0000 7f1552ffd640 1 --2- 172.21.15.57:0/292738739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f15540cb8b0 0x7f155406b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:09.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.912+0000 7f1552ffd640 1 -- 172.21.15.57:0/292738739 >> v1:172.21.15.57:6789/0 conn(0x7f1554151c20 legacy=0x7f1554072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:09.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.912+0000 7f1552ffd640 1 -- 172.21.15.57:0/292738739 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1554157df0 con 0x7f1554151840 2023-10-30T14:14:09.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.912+0000 7f15527fc640 1 -- 172.21.15.57:0/292738739 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460486780 0 0) 0x7f15540c5030 con 0x7f1554151c20 2023-10-30T14:14:09.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f1552ffd640 1 --2- 172.21.15.57:0/292738739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1554151840 0x7f15540c49a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f15440097c0 tx=0x7f154401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4384ef2c8e1e26 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:09.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f15527fc640 1 -- 172.21.15.57:0/292738739 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15440a8970 con 0x7f1554151840 2023-10-30T14:14:09.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f15527fc640 1 -- 172.21.15.57:0/292738739 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15440a8ad0 con 0x7f1554151840 2023-10-30T14:14:09.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f15527fc640 1 -- 172.21.15.57:0/292738739 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15440a8e60 con 0x7f1554151840 2023-10-30T14:14:09.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f1559565640 1 -- 172.21.15.57:0/292738739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1554151840 msgr2=0x7f15540c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:09.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f1559565640 1 --2- 172.21.15.57:0/292738739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1554151840 0x7f15540c49a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f15440097c0 tx=0x7f154401ec70 comp rx=0 tx=0).stop 2023-10-30T14:14:09.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f1559565640 1 -- 172.21.15.57:0/292738739 shutdown_connections 2023-10-30T14:14:09.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f1559565640 1 --2- 172.21.15.57:0/292738739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f15540cb8b0 0x7f155406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:09.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f1559565640 1 --2- 172.21.15.57:0/292738739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1554151840 0x7f15540c49a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:09.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f1559565640 1 -- 172.21.15.57:0/292738739 >> 172.21.15.57:0/292738739 conn(0x7f15540cac70 msgr2=0x7f15540cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:09.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f1559565640 1 -- 172.21.15.57:0/292738739 shutdown_connections 2023-10-30T14:14:09.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f1559565640 1 -- 172.21.15.57:0/292738739 wait complete. 2023-10-30T14:14:09.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f1559565640 1 Processor -- start 2023-10-30T14:14:09.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f1559565640 1 -- start start 2023-10-30T14:14:09.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f1559565640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15540cb8b0 0x7f15541e0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:09.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f1559565640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1554151840 0x7f15541ecec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:09.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f1559565640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1554151c20 0x7f15541ed400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:09.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f1559565640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15541573f0 con 0x7f1554151c20 2023-10-30T14:14:09.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f1559565640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15541e08b0 con 0x7f15540cb8b0 2023-10-30T14:14:09.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.916+0000 7f1559565640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15541567e0 con 0x7f1554151840 2023-10-30T14:14:09.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f15537fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1554151c20 0x7f15541ed400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:09.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f15537fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1554151c20 0x7f15541ed400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37870/0 (socket says 172.21.15.57:37870) 2023-10-30T14:14:09.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f15537fe640 1 -- 172.21.15.57:0/1481446103 learned_addr learned my addr 172.21.15.57:0/1481446103 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:09.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f1552ffd640 1 --2- 172.21.15.57:0/1481446103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15540cb8b0 0x7f15541e0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:09.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f154bfff640 1 --2- 172.21.15.57:0/1481446103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1554151840 0x7f15541ecec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:09.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f15537fe640 1 -- 172.21.15.57:0/1481446103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1554151840 msgr2=0x7f15541ecec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:09.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f15537fe640 1 --2- 172.21.15.57:0/1481446103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1554151840 0x7f15541ecec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:09.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f15537fe640 1 -- 172.21.15.57:0/1481446103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15540cb8b0 msgr2=0x7f15541e0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:09.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f15537fe640 1 --2- 172.21.15.57:0/1481446103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15540cb8b0 0x7f15541e0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:09.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f15537fe640 1 -- 172.21.15.57:0/1481446103 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1544009420 con 0x7f1554151c20 2023-10-30T14:14:09.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f154bfff640 1 --2- 172.21.15.57:0/1481446103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1554151840 0x7f15541ecec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:09.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f15537fe640 1 --2- 172.21.15.57:0/1481446103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1554151c20 0x7f15541ed400 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f154c009dd0 tx=0x7f154c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:09.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f154b7fe640 1 -- 172.21.15.57:0/1481446103 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c014480 con 0x7f1554151c20 2023-10-30T14:14:09.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f154b7fe640 1 -- 172.21.15.57:0/1481446103 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154c00f040 con 0x7f1554151c20 2023-10-30T14:14:09.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f1559565640 1 -- 172.21.15.57:0/1481446103 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1554156ac0 con 0x7f1554151c20 2023-10-30T14:14:09.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f154b7fe640 1 -- 172.21.15.57:0/1481446103 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c014840 con 0x7f1554151c20 2023-10-30T14:14:09.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f1559565640 1 -- 172.21.15.57:0/1481446103 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1554157080 con 0x7f1554151c20 2023-10-30T14:14:09.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f1559565640 1 -- 172.21.15.57:0/1481446103 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15541575a0 con 0x7f1554151c20 2023-10-30T14:14:09.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f154b7fe640 1 -- 172.21.15.57:0/1481446103 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f154c0149a0 con 0x7f1554151c20 2023-10-30T14:14:09.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.920+0000 7f154b7fe640 1 --2- 172.21.15.57:0/1481446103 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f15340664a0 0x7f1534068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:09.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.924+0000 7f154b7fe640 1 -- 172.21.15.57:0/1481446103 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f154c084da0 con 0x7f1554151c20 2023-10-30T14:14:09.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.924+0000 7f1552ffd640 1 --2- 172.21.15.57:0/1481446103 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f15340664a0 0x7f1534068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:09.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.924+0000 7f1552ffd640 1 --2- 172.21.15.57:0/1481446103 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f15340664a0 0x7f1534068960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f15440b2000 tx=0x7f1544008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:09.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:09.924+0000 7f154b7fe640 1 -- 172.21.15.57:0/1481446103 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f15541575a0 con 0x7f1554151c20 2023-10-30T14:14:10.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.120+0000 7f1559565640 1 -- 172.21.15.57:0/1481446103 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f155404e170 con 0x7f1554151c20 2023-10-30T14:14:10.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.120+0000 7f154b7fe640 1 -- 172.21.15.57:0/1481446103 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6501 (secure 0 0 0) 0x7f154c018020 con 0x7f1554151c20 2023-10-30T14:14:10.126 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:10.126 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4265,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/901968115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":901968115},{"type":"v1","addr":"172.21.15.57:6837","nonce":901968115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:14:09.745168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1],"up":{"mds_0":4282,"mds_1":4266},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:creating","state_seq":1,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:10.127 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 9 2023-10-30T14:14:10.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.124+0000 7f1559565640 1 -- 172.21.15.57:0/1481446103 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f15340664a0 msgr2=0x7f1534068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:10.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.124+0000 7f1559565640 1 --2- 172.21.15.57:0/1481446103 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f15340664a0 0x7f1534068960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f15440b2000 tx=0x7f1544008040 comp rx=0 tx=0).stop 2023-10-30T14:14:10.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.124+0000 7f1559565640 1 -- 172.21.15.57:0/1481446103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1554151c20 msgr2=0x7f15541ed400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:10.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.124+0000 7f1559565640 1 --2- 172.21.15.57:0/1481446103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1554151c20 0x7f15541ed400 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f154c009dd0 tx=0x7f154c00c490 comp rx=0 tx=0).stop 2023-10-30T14:14:10.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.124+0000 7f1559565640 1 -- 172.21.15.57:0/1481446103 shutdown_connections 2023-10-30T14:14:10.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.124+0000 7f1559565640 1 --2- 172.21.15.57:0/1481446103 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f15340664a0 0x7f1534068960 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.124+0000 7f1559565640 1 --2- 172.21.15.57:0/1481446103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1554151c20 0x7f15541ed400 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.124+0000 7f1559565640 1 --2- 172.21.15.57:0/1481446103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1554151840 0x7f15541ecec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.124+0000 7f1559565640 1 --2- 172.21.15.57:0/1481446103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15540cb8b0 0x7f15541e0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.124+0000 7f1559565640 1 -- 172.21.15.57:0/1481446103 >> 172.21.15.57:0/1481446103 conn(0x7f15540cac70 msgr2=0x7f155406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:10.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.124+0000 7f1559565640 1 -- 172.21.15.57:0/1481446103 shutdown_connections 2023-10-30T14:14:10.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.124+0000 7f1559565640 1 -- 172.21.15.57:0/1481446103 wait complete. 2023-10-30T14:14:10.147 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:14:10.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.284+0000 7f2e37046640 1 Processor -- start 2023-10-30T14:14:10.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.284+0000 7f2e37046640 1 -- start start 2023-10-30T14:14:10.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.284+0000 7f2e37046640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e30153aa0 0x7f2e3015d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:10.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.284+0000 7f2e37046640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e300c7b00 con 0x7f2e3006b3b0 2023-10-30T14:14:10.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.284+0000 7f2e37046640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e300c2050 con 0x7f2e300701e0 2023-10-30T14:14:10.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.284+0000 7f2e37046640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e30070a50 con 0x7f2e30153aa0 2023-10-30T14:14:10.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.284+0000 7f2e355e3640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f2e3006b3b0 0x7f2e3006b840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46256/0 (socket says 172.21.15.57:46256) 2023-10-30T14:14:10.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.284+0000 7f2e355e3640 1 -- 172.21.15.57:0/2811680345 learned_addr learned my addr 172.21.15.57:0/2811680345 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:10.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.284+0000 7f2e34de2640 1 --2- 172.21.15.57:0/2811680345 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e30153aa0 0x7f2e3015d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:10.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.288+0000 7f2e2f7fe640 1 -- 172.21.15.57:0/2811680345 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190770312 0 0) 0x7f2e300c7b00 con 0x7f2e3006b3b0 2023-10-30T14:14:10.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.288+0000 7f2e2f7fe640 1 -- 172.21.15.57:0/2811680345 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e20003530 con 0x7f2e3006b3b0 2023-10-30T14:14:10.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.288+0000 7f2e34de2640 1 -- 172.21.15.57:0/2811680345 >> v1:172.21.15.88:6789/0 conn(0x7f2e300701e0 legacy=0x7f2e300ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:10.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.288+0000 7f2e34de2640 1 -- 172.21.15.57:0/2811680345 >> v1:172.21.15.57:6789/0 conn(0x7f2e3006b3b0 legacy=0x7f2e3006b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:10.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.288+0000 7f2e34de2640 1 -- 172.21.15.57:0/2811680345 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e300c2330 con 0x7f2e30153aa0 2023-10-30T14:14:10.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.288+0000 7f2e34de2640 1 --2- 172.21.15.57:0/2811680345 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e30153aa0 0x7f2e3015d840 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2e240097f0 tx=0x7f2e2401ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99b007e000ec4ff0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:10.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.288+0000 7f2e2f7fe640 1 -- 172.21.15.57:0/2811680345 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e240a89b0 con 0x7f2e30153aa0 2023-10-30T14:14:10.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.288+0000 7f2e2f7fe640 1 -- 172.21.15.57:0/2811680345 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e240a8b10 con 0x7f2e30153aa0 2023-10-30T14:14:10.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.288+0000 7f2e2f7fe640 1 -- 172.21.15.57:0/2811680345 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e240a8e80 con 0x7f2e30153aa0 2023-10-30T14:14:10.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e37046640 1 -- 172.21.15.57:0/2811680345 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e30153aa0 msgr2=0x7f2e3015d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:10.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e37046640 1 --2- 172.21.15.57:0/2811680345 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e30153aa0 0x7f2e3015d840 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2e240097f0 tx=0x7f2e2401ecb0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e37046640 1 -- 172.21.15.57:0/2811680345 shutdown_connections 2023-10-30T14:14:10.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e37046640 1 --2- 172.21.15.57:0/2811680345 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e30153aa0 0x7f2e3015d840 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e37046640 1 -- 172.21.15.57:0/2811680345 >> 172.21.15.57:0/2811680345 conn(0x7f2e30071fb0 msgr2=0x7f2e300723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:10.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e37046640 1 -- 172.21.15.57:0/2811680345 shutdown_connections 2023-10-30T14:14:10.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e37046640 1 -- 172.21.15.57:0/2811680345 wait complete. 2023-10-30T14:14:10.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e37046640 1 Processor -- start 2023-10-30T14:14:10.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e37046640 1 -- start start 2023-10-30T14:14:10.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e37046640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e3006b3b0 0x7f2e30154870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:10.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e37046640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e300701e0 0x7f2e301541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:10.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e37046640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e30153aa0 0x7f2e301f2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:10.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e37046640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e30154f30 con 0x7f2e3006b3b0 2023-10-30T14:14:10.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e37046640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e301447f0 con 0x7f2e30153aa0 2023-10-30T14:14:10.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e37046640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e30144ad0 con 0x7f2e300701e0 2023-10-30T14:14:10.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e34de2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e3006b3b0 0x7f2e30154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:10.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e34de2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e3006b3b0 0x7f2e30154870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37874/0 (socket says 172.21.15.57:37874) 2023-10-30T14:14:10.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e34de2640 1 -- 172.21.15.57:0/1988362242 learned_addr learned my addr 172.21.15.57:0/1988362242 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:10.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e2ffff640 1 --2- 172.21.15.57:0/1988362242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e300701e0 0x7f2e301541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:10.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e355e3640 1 --2- 172.21.15.57:0/1988362242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e30153aa0 0x7f2e301f2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:10.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e34de2640 1 -- 172.21.15.57:0/1988362242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e300701e0 msgr2=0x7f2e301541b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:10.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.292+0000 7f2e34de2640 1 --2- 172.21.15.57:0/1988362242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e300701e0 0x7f2e301541b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.296+0000 7f2e34de2640 1 -- 172.21.15.57:0/1988362242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e30153aa0 msgr2=0x7f2e301f2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:10.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.296+0000 7f2e34de2640 1 --2- 172.21.15.57:0/1988362242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e30153aa0 0x7f2e301f2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.296+0000 7f2e34de2640 1 -- 172.21.15.57:0/1988362242 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e24009420 con 0x7f2e3006b3b0 2023-10-30T14:14:10.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.296+0000 7f2e355e3640 1 --2- 172.21.15.57:0/1988362242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e30153aa0 0x7f2e301f2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:14:10.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.296+0000 7f2e34de2640 1 --2- 172.21.15.57:0/1988362242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e3006b3b0 0x7f2e30154870 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2e24000c10 tx=0x7f2e2401d4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:10.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.296+0000 7f2e2d7fa640 1 -- 172.21.15.57:0/1988362242 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e240bf070 con 0x7f2e3006b3b0 2023-10-30T14:14:10.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.296+0000 7f2e2d7fa640 1 -- 172.21.15.57:0/1988362242 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e24003620 con 0x7f2e3006b3b0 2023-10-30T14:14:10.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.296+0000 7f2e2d7fa640 1 -- 172.21.15.57:0/1988362242 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e24003990 con 0x7f2e3006b3b0 2023-10-30T14:14:10.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.296+0000 7f2e37046640 1 -- 172.21.15.57:0/1988362242 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e301568b0 con 0x7f2e3006b3b0 2023-10-30T14:14:10.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.296+0000 7f2e37046640 1 -- 172.21.15.57:0/1988362242 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e30156e70 con 0x7f2e3006b3b0 2023-10-30T14:14:10.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.296+0000 7f2e37046640 1 -- 172.21.15.57:0/1988362242 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e3004e170 con 0x7f2e3006b3b0 2023-10-30T14:14:10.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.296+0000 7f2e2d7fa640 1 -- 172.21.15.57:0/1988362242 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2e24003af0 con 0x7f2e3006b3b0 2023-10-30T14:14:10.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.296+0000 7f2e2d7fa640 1 --2- 172.21.15.57:0/1988362242 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2e140664a0 0x7f2e14068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:10.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.296+0000 7f2e2d7fa640 1 -- 172.21.15.57:0/1988362242 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e241216a0 con 0x7f2e3006b3b0 2023-10-30T14:14:10.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.296+0000 7f2e2ffff640 1 --2- 172.21.15.57:0/1988362242 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2e140664a0 0x7f2e14068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:10.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.300+0000 7f2e2ffff640 1 --2- 172.21.15.57:0/1988362242 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2e140664a0 0x7f2e14068960 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2e300cd690 tx=0x7f2e1c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:10.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.300+0000 7f2e2d7fa640 1 -- 172.21.15.57:0/1988362242 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2e240eec30 con 0x7f2e3006b3b0 2023-10-30T14:14:10.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.456+0000 7f2e37046640 1 -- 172.21.15.57:0/1988362242 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e300c28c0 con 0x7f2e3006b3b0 2023-10-30T14:14:10.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.460+0000 7f2e2d7fa640 1 -- 172.21.15.57:0/1988362242 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16433 (secure 0 0 0) 0x7f2e240f2250 con 0x7f2e3006b3b0 2023-10-30T14:14:10.465 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:10.465 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:14:05.576818+0000","last_up_change":"2023-10-30T14:13:51.171287+0000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T14:13:52.574639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T14:14:02.600122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T14:14:03.069429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6801","nonce":3287367848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6803","nonce":3287367848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6807","nonce":3287367848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6805","nonce":3287367848}]},"public_addr":"172.21.15.57:6801/3287367848","cluster_addr":"172.21.15.57:6803/3287367848","heartbeat_back_addr":"172.21.15.57:6807/3287367848","heartbeat_front_addr":"172.21.15.57:6805/3287367848","state":["exists","up"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6809","nonce":3340797836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6811","nonce":3340797836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6815","nonce":3340797836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6813","nonce":3340797836}]},"public_addr":"172.21.15.57:6809/3340797836","cluster_addr":"172.21.15.57:6811/3340797836","heartbeat_back_addr":"172.21.15.57:6815/3340797836","heartbeat_front_addr":"172.21.15.57:6813/3340797836","state":["exists","up"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6825","nonce":1700292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6827","nonce":1700292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6831","nonce":1700292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6829","nonce":1700292}]},"public_addr":"172.21.15.57:6825/1700292","cluster_addr":"172.21.15.57:6827/1700292","heartbeat_back_addr":"172.21.15.57:6831/1700292","heartbeat_front_addr":"172.21.15.57:6829/1700292","state":["exists","up"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6817","nonce":2893420631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6819","nonce":2893420631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6823","nonce":2893420631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6821","nonce":2893420631}]},"public_addr":"172.21.15.57:6817/2893420631","cluster_addr":"172.21.15.57:6819/2893420631","heartbeat_back_addr":"172.21.15.57:6823/2893420631","heartbeat_front_addr":"172.21.15.57:6821/2893420631","state":["exists","up"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6807","nonce":2784816485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6811","nonce":2784816485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6815","nonce":2784816485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6813","nonce":2784816485}]},"public_addr":"172.21.15.88:6807/2784816485","cluster_addr":"172.21.15.88:6811/2784816485","heartbeat_back_addr":"172.21.15.88:6815/2784816485","heartbeat_front_addr":"172.21.15.88:6813/2784816485","state":["exists","up"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6801","nonce":2682449812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6803","nonce":2682449812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6810","nonce":2682449812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6806","nonce":2682449812}]},"public_addr":"172.21.15.88:6801/2682449812","cluster_addr":"172.21.15.88:6803/2682449812","heartbeat_back_addr":"172.21.15.88:6810/2682449812","heartbeat_front_addr":"172.21.15.88:6806/2682449812","state":["exists","up"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6822","nonce":3340693897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6827","nonce":3340693897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6831","nonce":3340693897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6829","nonce":3340693897}]},"public_addr":"172.21.15.88:6822/3340693897","cluster_addr":"172.21.15.88:6827/3340693897","heartbeat_back_addr":"172.21.15.88:6831/3340693897","heartbeat_front_addr":"172.21.15.88:6829/3340693897","state":["exists","up"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6817","nonce":2862111199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6820","nonce":2862111199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6825","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6826","nonce":2862111199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6821","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6823","nonce":2862111199}]},"public_addr":"172.21.15.88:6817/2862111199","cluster_addr":"172.21.15.88:6820/2862111199","heartbeat_back_addr":"172.21.15.88:6826/2862111199","heartbeat_front_addr":"172.21.15.88:6823/2862111199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.634920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.773993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.725825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.761452+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.694274+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.650718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:14:10.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.460+0000 7f2e37046640 1 -- 172.21.15.57:0/1988362242 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2e140664a0 msgr2=0x7f2e14068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:10.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.460+0000 7f2e37046640 1 --2- 172.21.15.57:0/1988362242 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2e140664a0 0x7f2e14068960 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2e300cd690 tx=0x7f2e1c009040 comp rx=0 tx=0).stop 2023-10-30T14:14:10.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.460+0000 7f2e37046640 1 -- 172.21.15.57:0/1988362242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e3006b3b0 msgr2=0x7f2e30154870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:10.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.460+0000 7f2e37046640 1 --2- 172.21.15.57:0/1988362242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e3006b3b0 0x7f2e30154870 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2e24000c10 tx=0x7f2e2401d4f0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.460+0000 7f2e37046640 1 -- 172.21.15.57:0/1988362242 shutdown_connections 2023-10-30T14:14:10.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.460+0000 7f2e37046640 1 --2- 172.21.15.57:0/1988362242 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2e140664a0 0x7f2e14068960 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.460+0000 7f2e37046640 1 --2- 172.21.15.57:0/1988362242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e30153aa0 0x7f2e301f2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.460+0000 7f2e37046640 1 --2- 172.21.15.57:0/1988362242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e300701e0 0x7f2e301541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.460+0000 7f2e37046640 1 --2- 172.21.15.57:0/1988362242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e3006b3b0 0x7f2e30154870 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.460+0000 7f2e37046640 1 -- 172.21.15.57:0/1988362242 >> 172.21.15.57:0/1988362242 conn(0x7f2e30071fb0 msgr2=0x7f2e3014eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:10.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.460+0000 7f2e37046640 1 -- 172.21.15.57:0/1988362242 shutdown_connections 2023-10-30T14:14:10.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.460+0000 7f2e37046640 1 -- 172.21.15.57:0/1988362242 wait complete. 2023-10-30T14:14:10.487 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-30T14:14:10.487 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:14:10.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.624+0000 7f0f9fd94640 1 Processor -- start 2023-10-30T14:14:10.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.624+0000 7f0f9fd94640 1 -- start start 2023-10-30T14:14:10.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9fd94640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0f98070520 0x7f0f98070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:10.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9fd94640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f980c2310 con 0x7f0f98150830 2023-10-30T14:14:10.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9fd94640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f98156710 con 0x7f0f980cb7d0 2023-10-30T14:14:10.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9fd94640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f9806bc10 con 0x7f0f98070520 2023-10-30T14:14:10.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9db30640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f0f98150830 0x7f0f9806b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46268/0 (socket says 172.21.15.57:46268) 2023-10-30T14:14:10.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9db30640 1 -- 172.21.15.57:0/3334212415 learned_addr learned my addr 172.21.15.57:0/3334212415 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:10.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9e331640 1 --2- 172.21.15.57:0/3334212415 >> v2:172.21.15.88:3300/0 conn(0x7f0f98070520 0x7f0f98070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:10.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9cb2e640 1 -- 172.21.15.57:0/3334212415 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948025475 0 0) 0x7f0f980c2310 con 0x7f0f98150830 2023-10-30T14:14:10.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9cb2e640 1 -- 172.21.15.57:0/3334212415 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f84005740 con 0x7f0f98150830 2023-10-30T14:14:10.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9cb2e640 1 -- 172.21.15.57:0/3334212415 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3687325365 0 0) 0x7f0f84005740 con 0x7f0f98150830 2023-10-30T14:14:10.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9cb2e640 1 -- 172.21.15.57:0/3334212415 >> v2:172.21.15.88:3300/0 conn(0x7f0f98070520 msgr2=0x7f0f98070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:10.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9cb2e640 1 --2- 172.21.15.57:0/3334212415 >> v2:172.21.15.88:3300/0 conn(0x7f0f98070520 0x7f0f98070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9cb2e640 1 -- 172.21.15.57:0/3334212415 >> v1:172.21.15.88:6789/0 conn(0x7f0f980cb7d0 legacy=0x7f0f980c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:10.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9cb2e640 1 -- 172.21.15.57:0/3334212415 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f980c4360 con 0x7f0f98150830 2023-10-30T14:14:10.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9cb2e640 1 -- 172.21.15.57:0/3334212415 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f0f940016a0 con 0x7f0f98150830 2023-10-30T14:14:10.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9cb2e640 1 -- 172.21.15.57:0/3334212415 >> v1:172.21.15.57:6789/0 conn(0x7f0f98150830 legacy=0x7f0f9806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:10.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9cb2e640 1 --2- 172.21.15.57:0/3334212415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f84005b90 0x7f0f84005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:10.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9cb2e640 1 --2- 172.21.15.57:0/3334212415 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f840069e0 0x7f0f84006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:10.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9cb2e640 1 --2- 172.21.15.57:0/3334212415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f840073a0 0x7f0f8400dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:10.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9cb2e640 1 -- 172.21.15.57:0/3334212415 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9806bc10 con 0x7f0f84005b90 2023-10-30T14:14:10.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9cb2e640 1 -- 172.21.15.57:0/3334212415 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8400e250 con 0x7f0f840073a0 2023-10-30T14:14:10.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9cb2e640 1 -- 172.21.15.57:0/3334212415 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8400e410 con 0x7f0f840069e0 2023-10-30T14:14:10.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9db30640 1 --2- 172.21.15.57:0/3334212415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f84005b90 0x7f0f84005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:10.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9d32f640 1 --2- 172.21.15.57:0/3334212415 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f840069e0 0x7f0f84006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:10.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9e331640 1 --2- 172.21.15.57:0/3334212415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f840073a0 0x7f0f8400dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:10.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9db30640 1 -- 172.21.15.57:0/3334212415 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f840069e0 msgr2=0x7f0f84006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:10.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9db30640 1 --2- 172.21.15.57:0/3334212415 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f840069e0 0x7f0f84006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9db30640 1 -- 172.21.15.57:0/3334212415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f840073a0 msgr2=0x7f0f8400dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:10.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9db30640 1 --2- 172.21.15.57:0/3334212415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f840073a0 0x7f0f8400dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9db30640 1 -- 172.21.15.57:0/3334212415 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f84005740 con 0x7f0f84005b90 2023-10-30T14:14:10.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9e331640 1 --2- 172.21.15.57:0/3334212415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f840073a0 0x7f0f8400dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:10.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.628+0000 7f0f9e331640 1 -- 172.21.15.57:0/3334212415 reap_dead start 2023-10-30T14:14:10.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.632+0000 7f0f9db30640 1 --2- 172.21.15.57:0/3334212415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f84005b90 0x7f0f84005fb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0f980c24a0 tx=0x7f0f9401f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d88205111e52722 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:10.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.632+0000 7f0f9cb2e640 1 -- 172.21.15.57:0/3334212415 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f94005ae0 con 0x7f0f84005b90 2023-10-30T14:14:10.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.632+0000 7f0f9cb2e640 1 -- 172.21.15.57:0/3334212415 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f940ab070 con 0x7f0f84005b90 2023-10-30T14:14:10.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.632+0000 7f0f9cb2e640 1 -- 172.21.15.57:0/3334212415 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f94007070 con 0x7f0f84005b90 2023-10-30T14:14:10.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.632+0000 7f0f9fd94640 1 -- 172.21.15.57:0/3334212415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f84005b90 msgr2=0x7f0f84005fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:10.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.632+0000 7f0f9fd94640 1 --2- 172.21.15.57:0/3334212415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f84005b90 0x7f0f84005fb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0f980c24a0 tx=0x7f0f9401f330 comp rx=0 tx=0).stop 2023-10-30T14:14:10.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.632+0000 7f0f9fd94640 1 -- 172.21.15.57:0/3334212415 shutdown_connections 2023-10-30T14:14:10.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.632+0000 7f0f9fd94640 1 --2- 172.21.15.57:0/3334212415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f84005b90 0x7f0f84005fb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.632+0000 7f0f9fd94640 1 -- 172.21.15.57:0/3334212415 >> 172.21.15.57:0/3334212415 conn(0x7f0f980cac70 msgr2=0x7f0f980cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:10.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.632+0000 7f0f9fd94640 1 -- 172.21.15.57:0/3334212415 shutdown_connections 2023-10-30T14:14:10.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.632+0000 7f0f9fd94640 1 -- 172.21.15.57:0/3334212415 wait complete. 2023-10-30T14:14:10.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9fd94640 1 Processor -- start 2023-10-30T14:14:10.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9fd94640 1 -- start start 2023-10-30T14:14:10.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9fd94640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f84005b90 0x7f0f981577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:10.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9fd94640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f981f1c30 0x7f0f981ef0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:10.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9fd94640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f981538d0 0x7f0f980733c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:10.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9fd94640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f981ee6c0 con 0x7f0f981f1c30 2023-10-30T14:14:10.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9fd94640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f981eec40 con 0x7f0f84005b90 2023-10-30T14:14:10.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9fd94640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f981ef610 con 0x7f0f981538d0 2023-10-30T14:14:10.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9d32f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f981f1c30 0x7f0f981ef0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:10.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9d32f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f981f1c30 0x7f0f981ef0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37880/0 (socket says 172.21.15.57:37880) 2023-10-30T14:14:10.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9d32f640 1 -- 172.21.15.57:0/3621957469 learned_addr learned my addr 172.21.15.57:0/3621957469 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:10.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9e331640 1 --2- 172.21.15.57:0/3621957469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f981538d0 0x7f0f980733c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:10.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9db30640 1 --2- 172.21.15.57:0/3621957469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f84005b90 0x7f0f981577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:10.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9d32f640 1 -- 172.21.15.57:0/3621957469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f981538d0 msgr2=0x7f0f980733c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:10.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9d32f640 1 --2- 172.21.15.57:0/3621957469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f981538d0 0x7f0f980733c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9d32f640 1 -- 172.21.15.57:0/3621957469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f84005b90 msgr2=0x7f0f981577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:10.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9d32f640 1 --2- 172.21.15.57:0/3621957469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f84005b90 0x7f0f981577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9d32f640 1 -- 172.21.15.57:0/3621957469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f981f2370 con 0x7f0f981f1c30 2023-10-30T14:14:10.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9db30640 1 --2- 172.21.15.57:0/3621957469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f84005b90 0x7f0f981577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:10.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9d32f640 1 --2- 172.21.15.57:0/3621957469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f981f1c30 0x7f0f981ef0d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800f820 tx=0x7f0f8800fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:10.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f8e7fc640 1 -- 172.21.15.57:0/3621957469 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88001cf0 con 0x7f0f981f1c30 2023-10-30T14:14:10.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f8e7fc640 1 -- 172.21.15.57:0/3621957469 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f880052e0 con 0x7f0f981f1c30 2023-10-30T14:14:10.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f8e7fc640 1 -- 172.21.15.57:0/3621957469 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8800d8f0 con 0x7f0f981f1c30 2023-10-30T14:14:10.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9fd94640 1 -- 172.21.15.57:0/3621957469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f981f2650 con 0x7f0f981f1c30 2023-10-30T14:14:10.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9fd94640 1 -- 172.21.15.57:0/3621957469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f981ee8d0 con 0x7f0f981f1c30 2023-10-30T14:14:10.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.636+0000 7f0f9fd94640 1 -- 172.21.15.57:0/3621957469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9804e170 con 0x7f0f981f1c30 2023-10-30T14:14:10.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.640+0000 7f0f8e7fc640 1 -- 172.21.15.57:0/3621957469 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0f8800da50 con 0x7f0f981f1c30 2023-10-30T14:14:10.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.640+0000 7f0f8e7fc640 1 --2- 172.21.15.57:0/3621957469 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0f7c0664a0 0x7f0f7c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:10.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.640+0000 7f0f8e7fc640 1 -- 172.21.15.57:0/3621957469 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0f88083d20 con 0x7f0f981f1c30 2023-10-30T14:14:10.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.640+0000 7f0f9db30640 1 --2- 172.21.15.57:0/3621957469 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0f7c0664a0 0x7f0f7c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:10.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.640+0000 7f0f9db30640 1 --2- 172.21.15.57:0/3621957469 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0f7c0664a0 0x7f0f7c068960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0f940af000 tx=0x7f0f940aa000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:10.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.644+0000 7f0f8e7fc640 1 -- 172.21.15.57:0/3621957469 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0f88051100 con 0x7f0f981f1c30 2023-10-30T14:14:10.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.800+0000 7f0f9fd94640 1 -- 172.21.15.57:0/3621957469 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0f98156c90 con 0x7f0f7c0664a0 2023-10-30T14:14:10.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.824+0000 7f0f8e7fc640 1 -- 172.21.15.57:0/3621957469 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264470 (secure 0 0 0) 0x7f0f94001620 con 0x7f0f7c0664a0 2023-10-30T14:14:10.830 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:10.834 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-30T14:14:10.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.828+0000 7f0f9fd94640 1 -- 172.21.15.57:0/3621957469 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0f7c0664a0 msgr2=0x7f0f7c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:10.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.828+0000 7f0f9fd94640 1 --2- 172.21.15.57:0/3621957469 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0f7c0664a0 0x7f0f7c068960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0f940af000 tx=0x7f0f940aa000 comp rx=0 tx=0).stop 2023-10-30T14:14:10.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.828+0000 7f0f9fd94640 1 -- 172.21.15.57:0/3621957469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f981f1c30 msgr2=0x7f0f981ef0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:10.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.828+0000 7f0f9fd94640 1 --2- 172.21.15.57:0/3621957469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f981f1c30 0x7f0f981ef0d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800f820 tx=0x7f0f8800fce0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.828+0000 7f0f9fd94640 1 -- 172.21.15.57:0/3621957469 shutdown_connections 2023-10-30T14:14:10.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.828+0000 7f0f9fd94640 1 --2- 172.21.15.57:0/3621957469 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0f7c0664a0 0x7f0f7c068960 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.828+0000 7f0f9fd94640 1 --2- 172.21.15.57:0/3621957469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f981538d0 0x7f0f980733c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.828+0000 7f0f9fd94640 1 --2- 172.21.15.57:0/3621957469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f981f1c30 0x7f0f981ef0d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.828+0000 7f0f9fd94640 1 --2- 172.21.15.57:0/3621957469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f84005b90 0x7f0f981577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:10.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.828+0000 7f0f9fd94640 1 -- 172.21.15.57:0/3621957469 >> 172.21.15.57:0/3621957469 conn(0x7f0f980cac70 msgr2=0x7f0f980c54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:10.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.828+0000 7f0f9fd94640 1 -- 172.21.15.57:0/3621957469 shutdown_connections 2023-10-30T14:14:10.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:10.828+0000 7f0f9fd94640 1 -- 172.21.15.57:0/3621957469 wait complete. 2023-10-30T14:14:10.852 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-30T14:14:10.494590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227364,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754747356,"statfs":{"total":773094113280,"available":772861292544,"internally_reserved":0,"allocated":2924544,"data_stored":1785288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":229756473},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.813582"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757043+0000","last_change":"2023-10-30T14:14:04.651413+0000","last_active":"2023-10-30T14:14:05.757043+0000","last_peered":"2023-10-30T14:14:05.757043+0000","last_clean":"2023-10-30T14:14:05.757043+0000","last_became_active":"2023-10-30T14:14:04.650882+0000","last_became_peered":"2023-10-30T14:14:04.650882+0000","last_unstale":"2023-10-30T14:14:05.757043+0000","last_undegraded":"2023-10-30T14:14:05.757043+0000","last_fullsized":"2023-10-30T14:14:05.757043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:30:08.122326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585036+0000","last_change":"2023-10-30T14:14:04.588523+0000","last_active":"2023-10-30T14:14:05.585036+0000","last_peered":"2023-10-30T14:14:05.585036+0000","last_clean":"2023-10-30T14:14:05.585036+0000","last_became_active":"2023-10-30T14:14:04.587942+0000","last_became_peered":"2023-10-30T14:14:04.587942+0000","last_unstale":"2023-10-30T14:14:05.585036+0000","last_undegraded":"2023-10-30T14:14:05.585036+0000","last_fullsized":"2023-10-30T14:14:05.585036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:16.028564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586543+0000","last_change":"2023-10-30T14:14:04.590976+0000","last_active":"2023-10-30T14:14:05.586543+0000","last_peered":"2023-10-30T14:14:05.586543+0000","last_clean":"2023-10-30T14:14:05.586543+0000","last_became_active":"2023-10-30T14:14:04.590700+0000","last_became_peered":"2023-10-30T14:14:04.590700+0000","last_unstale":"2023-10-30T14:14:05.586543+0000","last_undegraded":"2023-10-30T14:14:05.586543+0000","last_fullsized":"2023-10-30T14:14:05.586543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:49.169325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586525+0000","last_change":"2023-10-30T14:14:04.618603+0000","last_active":"2023-10-30T14:14:05.586525+0000","last_peered":"2023-10-30T14:14:05.586525+0000","last_clean":"2023-10-30T14:14:05.586525+0000","last_became_active":"2023-10-30T14:14:04.617768+0000","last_became_peered":"2023-10-30T14:14:04.617768+0000","last_unstale":"2023-10-30T14:14:05.586525+0000","last_undegraded":"2023-10-30T14:14:05.586525+0000","last_fullsized":"2023-10-30T14:14:05.586525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:16.443567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756348+0000","last_change":"2023-10-30T14:14:04.647842+0000","last_active":"2023-10-30T14:14:05.756348+0000","last_peered":"2023-10-30T14:14:05.756348+0000","last_clean":"2023-10-30T14:14:05.756348+0000","last_became_active":"2023-10-30T14:14:04.644683+0000","last_became_peered":"2023-10-30T14:14:04.644683+0000","last_unstale":"2023-10-30T14:14:05.756348+0000","last_undegraded":"2023-10-30T14:14:05.756348+0000","last_fullsized":"2023-10-30T14:14:05.756348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:51:01.230722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585902+0000","last_change":"2023-10-30T14:14:04.620587+0000","last_active":"2023-10-30T14:14:05.585902+0000","last_peered":"2023-10-30T14:14:05.585902+0000","last_clean":"2023-10-30T14:14:05.585902+0000","last_became_active":"2023-10-30T14:14:04.619335+0000","last_became_peered":"2023-10-30T14:14:04.619335+0000","last_unstale":"2023-10-30T14:14:05.585902+0000","last_undegraded":"2023-10-30T14:14:05.585902+0000","last_fullsized":"2023-10-30T14:14:05.585902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:01.735027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755914+0000","last_change":"2023-10-30T14:14:04.641688+0000","last_active":"2023-10-30T14:14:05.755914+0000","last_peered":"2023-10-30T14:14:05.755914+0000","last_clean":"2023-10-30T14:14:05.755914+0000","last_became_active":"2023-10-30T14:14:04.641323+0000","last_became_peered":"2023-10-30T14:14:04.641323+0000","last_unstale":"2023-10-30T14:14:05.755914+0000","last_undegraded":"2023-10-30T14:14:05.755914+0000","last_fullsized":"2023-10-30T14:14:05.755914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:43:35.285019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805751+0000","last_change":"2023-10-30T14:14:04.618530+0000","last_active":"2023-10-30T14:14:05.805751+0000","last_peered":"2023-10-30T14:14:05.805751+0000","last_clean":"2023-10-30T14:14:05.805751+0000","last_became_active":"2023-10-30T14:14:04.618312+0000","last_became_peered":"2023-10-30T14:14:04.618312+0000","last_unstale":"2023-10-30T14:14:05.805751+0000","last_undegraded":"2023-10-30T14:14:05.805751+0000","last_fullsized":"2023-10-30T14:14:05.805751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:25.089106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757146+0000","last_change":"2023-10-30T14:14:04.650330+0000","last_active":"2023-10-30T14:14:05.757146+0000","last_peered":"2023-10-30T14:14:05.757146+0000","last_clean":"2023-10-30T14:14:05.757146+0000","last_became_active":"2023-10-30T14:14:04.647361+0000","last_became_peered":"2023-10-30T14:14:04.647361+0000","last_unstale":"2023-10-30T14:14:05.757146+0000","last_undegraded":"2023-10-30T14:14:05.757146+0000","last_fullsized":"2023-10-30T14:14:05.757146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:49:57.370980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808090+0000","last_change":"2023-10-30T14:14:04.631540+0000","last_active":"2023-10-30T14:14:05.808090+0000","last_peered":"2023-10-30T14:14:05.808090+0000","last_clean":"2023-10-30T14:14:05.808090+0000","last_became_active":"2023-10-30T14:14:04.630959+0000","last_became_peered":"2023-10-30T14:14:04.630959+0000","last_unstale":"2023-10-30T14:14:05.808090+0000","last_undegraded":"2023-10-30T14:14:05.808090+0000","last_fullsized":"2023-10-30T14:14:05.808090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:49.468429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807762+0000","last_change":"2023-10-30T14:14:04.613800+0000","last_active":"2023-10-30T14:14:05.807762+0000","last_peered":"2023-10-30T14:14:05.807762+0000","last_clean":"2023-10-30T14:14:05.807762+0000","last_became_active":"2023-10-30T14:14:04.612653+0000","last_became_peered":"2023-10-30T14:14:04.612653+0000","last_unstale":"2023-10-30T14:14:05.807762+0000","last_undegraded":"2023-10-30T14:14:05.807762+0000","last_fullsized":"2023-10-30T14:14:05.807762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:44.665588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809700+0000","last_change":"2023-10-30T14:14:04.622483+0000","last_active":"2023-10-30T14:14:05.809700+0000","last_peered":"2023-10-30T14:14:05.809700+0000","last_clean":"2023-10-30T14:14:05.809700+0000","last_became_active":"2023-10-30T14:14:04.621135+0000","last_became_peered":"2023-10-30T14:14:04.621135+0000","last_unstale":"2023-10-30T14:14:05.809700+0000","last_undegraded":"2023-10-30T14:14:05.809700+0000","last_fullsized":"2023-10-30T14:14:05.809700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:52.299412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806054+0000","last_change":"2023-10-30T14:14:04.616357+0000","last_active":"2023-10-30T14:14:05.806054+0000","last_peered":"2023-10-30T14:14:05.806054+0000","last_clean":"2023-10-30T14:14:05.806054+0000","last_became_active":"2023-10-30T14:14:04.616142+0000","last_became_peered":"2023-10-30T14:14:04.616142+0000","last_unstale":"2023-10-30T14:14:05.806054+0000","last_undegraded":"2023-10-30T14:14:05.806054+0000","last_fullsized":"2023-10-30T14:14:05.806054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:32:26.432551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586106+0000","last_change":"2023-10-30T14:14:04.588759+0000","last_active":"2023-10-30T14:14:05.586106+0000","last_peered":"2023-10-30T14:14:05.586106+0000","last_clean":"2023-10-30T14:14:05.586106+0000","last_became_active":"2023-10-30T14:14:04.588452+0000","last_became_peered":"2023-10-30T14:14:04.588452+0000","last_unstale":"2023-10-30T14:14:05.586106+0000","last_undegraded":"2023-10-30T14:14:05.586106+0000","last_fullsized":"2023-10-30T14:14:05.586106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:38:21.206284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584572+0000","last_change":"2023-10-30T14:14:04.621073+0000","last_active":"2023-10-30T14:14:05.584572+0000","last_peered":"2023-10-30T14:14:05.584572+0000","last_clean":"2023-10-30T14:14:05.584572+0000","last_became_active":"2023-10-30T14:14:04.620781+0000","last_became_peered":"2023-10-30T14:14:04.620781+0000","last_unstale":"2023-10-30T14:14:05.584572+0000","last_undegraded":"2023-10-30T14:14:05.584572+0000","last_fullsized":"2023-10-30T14:14:05.584572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:17:10.938195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585712+0000","last_change":"2023-10-30T14:14:04.608942+0000","last_active":"2023-10-30T14:14:05.585712+0000","last_peered":"2023-10-30T14:14:05.585712+0000","last_clean":"2023-10-30T14:14:05.585712+0000","last_became_active":"2023-10-30T14:14:04.608190+0000","last_became_peered":"2023-10-30T14:14:04.608190+0000","last_unstale":"2023-10-30T14:14:05.585712+0000","last_undegraded":"2023-10-30T14:14:05.585712+0000","last_fullsized":"2023-10-30T14:14:05.585712+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:18:39.846841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587056+0000","last_change":"2023-10-30T14:14:04.641876+0000","last_active":"2023-10-30T14:14:05.587056+0000","last_peered":"2023-10-30T14:14:05.587056+0000","last_clean":"2023-10-30T14:14:05.587056+0000","last_became_active":"2023-10-30T14:14:04.641306+0000","last_became_peered":"2023-10-30T14:14:04.641306+0000","last_unstale":"2023-10-30T14:14:05.587056+0000","last_undegraded":"2023-10-30T14:14:05.587056+0000","last_fullsized":"2023-10-30T14:14:05.587056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:10:08.169786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805744+0000","last_change":"2023-10-30T14:14:04.613999+0000","last_active":"2023-10-30T14:14:05.805744+0000","last_peered":"2023-10-30T14:14:05.805744+0000","last_clean":"2023-10-30T14:14:05.805744+0000","last_became_active":"2023-10-30T14:14:04.613644+0000","last_became_peered":"2023-10-30T14:14:04.613644+0000","last_unstale":"2023-10-30T14:14:05.805744+0000","last_undegraded":"2023-10-30T14:14:05.805744+0000","last_fullsized":"2023-10-30T14:14:05.805744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:14:50.232028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756929+0000","last_change":"2023-10-30T14:14:04.650393+0000","last_active":"2023-10-30T14:14:05.756929+0000","last_peered":"2023-10-30T14:14:05.756929+0000","last_clean":"2023-10-30T14:14:05.756929+0000","last_became_active":"2023-10-30T14:14:04.649850+0000","last_became_peered":"2023-10-30T14:14:04.649850+0000","last_unstale":"2023-10-30T14:14:05.756929+0000","last_undegraded":"2023-10-30T14:14:05.756929+0000","last_fullsized":"2023-10-30T14:14:05.756929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:25.174023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808107+0000","last_change":"2023-10-30T14:14:04.625744+0000","last_active":"2023-10-30T14:14:05.808107+0000","last_peered":"2023-10-30T14:14:05.808107+0000","last_clean":"2023-10-30T14:14:05.808107+0000","last_became_active":"2023-10-30T14:14:04.625042+0000","last_became_peered":"2023-10-30T14:14:04.625042+0000","last_unstale":"2023-10-30T14:14:05.808107+0000","last_undegraded":"2023-10-30T14:14:05.808107+0000","last_fullsized":"2023-10-30T14:14:05.808107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:30:36.063455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586236+0000","last_change":"2023-10-30T14:14:04.633942+0000","last_active":"2023-10-30T14:14:05.586236+0000","last_peered":"2023-10-30T14:14:05.586236+0000","last_clean":"2023-10-30T14:14:05.586236+0000","last_became_active":"2023-10-30T14:14:04.633586+0000","last_became_peered":"2023-10-30T14:14:04.633586+0000","last_unstale":"2023-10-30T14:14:05.586236+0000","last_undegraded":"2023-10-30T14:14:05.586236+0000","last_fullsized":"2023-10-30T14:14:05.586236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:42:58.067892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586050+0000","last_change":"2023-10-30T14:14:04.642847+0000","last_active":"2023-10-30T14:14:05.586050+0000","last_peered":"2023-10-30T14:14:05.586050+0000","last_clean":"2023-10-30T14:14:05.586050+0000","last_became_active":"2023-10-30T14:14:04.642244+0000","last_became_peered":"2023-10-30T14:14:04.642244+0000","last_unstale":"2023-10-30T14:14:05.586050+0000","last_undegraded":"2023-10-30T14:14:05.586050+0000","last_fullsized":"2023-10-30T14:14:05.586050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:28.780866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805762+0000","last_change":"2023-10-30T14:14:04.620651+0000","last_active":"2023-10-30T14:14:05.805762+0000","last_peered":"2023-10-30T14:14:05.805762+0000","last_clean":"2023-10-30T14:14:05.805762+0000","last_became_active":"2023-10-30T14:14:04.620343+0000","last_became_peered":"2023-10-30T14:14:04.620343+0000","last_unstale":"2023-10-30T14:14:05.805762+0000","last_undegraded":"2023-10-30T14:14:05.805762+0000","last_fullsized":"2023-10-30T14:14:05.805762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:16:00.557118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587224+0000","last_change":"2023-10-30T14:14:04.610472+0000","last_active":"2023-10-30T14:14:05.587224+0000","last_peered":"2023-10-30T14:14:05.587224+0000","last_clean":"2023-10-30T14:14:05.587224+0000","last_became_active":"2023-10-30T14:14:04.609720+0000","last_became_peered":"2023-10-30T14:14:04.609720+0000","last_unstale":"2023-10-30T14:14:05.587224+0000","last_undegraded":"2023-10-30T14:14:05.587224+0000","last_fullsized":"2023-10-30T14:14:05.587224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:44:28.391589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805906+0000","last_change":"2023-10-30T14:14:04.620548+0000","last_active":"2023-10-30T14:14:05.805906+0000","last_peered":"2023-10-30T14:14:05.805906+0000","last_clean":"2023-10-30T14:14:05.805906+0000","last_became_active":"2023-10-30T14:14:04.620217+0000","last_became_peered":"2023-10-30T14:14:04.620217+0000","last_unstale":"2023-10-30T14:14:05.805906+0000","last_undegraded":"2023-10-30T14:14:05.805906+0000","last_fullsized":"2023-10-30T14:14:05.805906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:40:22.912696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808191+0000","last_change":"2023-10-30T14:14:04.644315+0000","last_active":"2023-10-30T14:14:05.808191+0000","last_peered":"2023-10-30T14:14:05.808191+0000","last_clean":"2023-10-30T14:14:05.808191+0000","last_became_active":"2023-10-30T14:14:04.643707+0000","last_became_peered":"2023-10-30T14:14:04.643707+0000","last_unstale":"2023-10-30T14:14:05.808191+0000","last_undegraded":"2023-10-30T14:14:05.808191+0000","last_fullsized":"2023-10-30T14:14:05.808191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:20:33.544167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584874+0000","last_change":"2023-10-30T14:14:04.597855+0000","last_active":"2023-10-30T14:14:05.584874+0000","last_peered":"2023-10-30T14:14:05.584874+0000","last_clean":"2023-10-30T14:14:05.584874+0000","last_became_active":"2023-10-30T14:14:04.597590+0000","last_became_peered":"2023-10-30T14:14:04.597590+0000","last_unstale":"2023-10-30T14:14:05.584874+0000","last_undegraded":"2023-10-30T14:14:05.584874+0000","last_fullsized":"2023-10-30T14:14:05.584874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:05.905027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806303+0000","last_change":"2023-10-30T14:14:04.624133+0000","last_active":"2023-10-30T14:14:05.806303+0000","last_peered":"2023-10-30T14:14:05.806303+0000","last_clean":"2023-10-30T14:14:05.806303+0000","last_became_active":"2023-10-30T14:14:04.623902+0000","last_became_peered":"2023-10-30T14:14:04.623902+0000","last_unstale":"2023-10-30T14:14:05.806303+0000","last_undegraded":"2023-10-30T14:14:05.806303+0000","last_fullsized":"2023-10-30T14:14:05.806303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:33:10.476461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586328+0000","last_change":"2023-10-30T14:14:04.597875+0000","last_active":"2023-10-30T14:14:05.586328+0000","last_peered":"2023-10-30T14:14:05.586328+0000","last_clean":"2023-10-30T14:14:05.586328+0000","last_became_active":"2023-10-30T14:14:04.597556+0000","last_became_peered":"2023-10-30T14:14:04.597556+0000","last_unstale":"2023-10-30T14:14:05.586328+0000","last_undegraded":"2023-10-30T14:14:05.586328+0000","last_fullsized":"2023-10-30T14:14:05.586328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:08:46.407610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586842+0000","last_change":"2023-10-30T14:14:04.620110+0000","last_active":"2023-10-30T14:14:05.586842+0000","last_peered":"2023-10-30T14:14:05.586842+0000","last_clean":"2023-10-30T14:14:05.586842+0000","last_became_active":"2023-10-30T14:14:04.618430+0000","last_became_peered":"2023-10-30T14:14:04.618430+0000","last_unstale":"2023-10-30T14:14:05.586842+0000","last_undegraded":"2023-10-30T14:14:05.586842+0000","last_fullsized":"2023-10-30T14:14:05.586842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:36.966564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805526+0000","last_change":"2023-10-30T14:14:04.606692+0000","last_active":"2023-10-30T14:14:05.805526+0000","last_peered":"2023-10-30T14:14:05.805526+0000","last_clean":"2023-10-30T14:14:05.805526+0000","last_became_active":"2023-10-30T14:14:04.606340+0000","last_became_peered":"2023-10-30T14:14:04.606340+0000","last_unstale":"2023-10-30T14:14:05.805526+0000","last_undegraded":"2023-10-30T14:14:05.805526+0000","last_fullsized":"2023-10-30T14:14:05.805526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:02:24.518205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807314+0000","last_change":"2023-10-30T14:14:04.613535+0000","last_active":"2023-10-30T14:14:05.807314+0000","last_peered":"2023-10-30T14:14:05.807314+0000","last_clean":"2023-10-30T14:14:05.807314+0000","last_became_active":"2023-10-30T14:14:04.612588+0000","last_became_peered":"2023-10-30T14:14:04.612588+0000","last_unstale":"2023-10-30T14:14:05.807314+0000","last_undegraded":"2023-10-30T14:14:05.807314+0000","last_fullsized":"2023-10-30T14:14:05.807314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:58:23.057988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809168+0000","last_change":"2023-10-30T14:14:04.608434+0000","last_active":"2023-10-30T14:14:05.809168+0000","last_peered":"2023-10-30T14:14:05.809168+0000","last_clean":"2023-10-30T14:14:05.809168+0000","last_became_active":"2023-10-30T14:14:04.607683+0000","last_became_peered":"2023-10-30T14:14:04.607683+0000","last_unstale":"2023-10-30T14:14:05.809168+0000","last_undegraded":"2023-10-30T14:14:05.809168+0000","last_fullsized":"2023-10-30T14:14:05.809168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:17:27.467330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806271+0000","last_change":"2023-10-30T14:14:03.732084+0000","last_active":"2023-10-30T14:14:05.806271+0000","last_peered":"2023-10-30T14:14:05.806271+0000","last_clean":"2023-10-30T14:14:05.806271+0000","last_became_active":"2023-10-30T14:14:03.731740+0000","last_became_peered":"2023-10-30T14:14:03.731740+0000","last_unstale":"2023-10-30T14:14:05.806271+0000","last_undegraded":"2023-10-30T14:14:05.806271+0000","last_fullsized":"2023-10-30T14:14:05.806271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:34:27.462477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586305+0000","last_change":"2023-10-30T14:14:04.621869+0000","last_active":"2023-10-30T14:14:05.586305+0000","last_peered":"2023-10-30T14:14:05.586305+0000","last_clean":"2023-10-30T14:14:05.586305+0000","last_became_active":"2023-10-30T14:14:04.621526+0000","last_became_peered":"2023-10-30T14:14:04.621526+0000","last_unstale":"2023-10-30T14:14:05.586305+0000","last_undegraded":"2023-10-30T14:14:05.586305+0000","last_fullsized":"2023-10-30T14:14:05.586305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:08:21.260692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807788+0000","last_change":"2023-10-30T14:14:03.730789+0000","last_active":"2023-10-30T14:14:05.807788+0000","last_peered":"2023-10-30T14:14:05.807788+0000","last_clean":"2023-10-30T14:14:05.807788+0000","last_became_active":"2023-10-30T14:14:03.729386+0000","last_became_peered":"2023-10-30T14:14:03.729386+0000","last_unstale":"2023-10-30T14:14:05.807788+0000","last_undegraded":"2023-10-30T14:14:05.807788+0000","last_fullsized":"2023-10-30T14:14:05.807788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805855+0000","last_change":"2023-10-30T14:14:03.724379+0000","last_active":"2023-10-30T14:14:05.805855+0000","last_peered":"2023-10-30T14:14:05.805855+0000","last_clean":"2023-10-30T14:14:05.805855+0000","last_became_active":"2023-10-30T14:14:03.724078+0000","last_became_peered":"2023-10-30T14:14:03.724078+0000","last_unstale":"2023-10-30T14:14:05.805855+0000","last_undegraded":"2023-10-30T14:14:05.805855+0000","last_fullsized":"2023-10-30T14:14:05.805855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:10:17.829899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805886+0000","last_change":"2023-10-30T14:14:04.597423+0000","last_active":"2023-10-30T14:14:05.805886+0000","last_peered":"2023-10-30T14:14:05.805886+0000","last_clean":"2023-10-30T14:14:05.805886+0000","last_became_active":"2023-10-30T14:14:04.597099+0000","last_became_peered":"2023-10-30T14:14:04.597099+0000","last_unstale":"2023-10-30T14:14:05.805886+0000","last_undegraded":"2023-10-30T14:14:05.805886+0000","last_fullsized":"2023-10-30T14:14:05.805886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:54:52.346295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585964+0000","last_change":"2023-10-30T14:14:03.721030+0000","last_active":"2023-10-30T14:14:05.585964+0000","last_peered":"2023-10-30T14:14:05.585964+0000","last_clean":"2023-10-30T14:14:05.585964+0000","last_became_active":"2023-10-30T14:14:03.719816+0000","last_became_peered":"2023-10-30T14:14:03.719816+0000","last_unstale":"2023-10-30T14:14:05.585964+0000","last_undegraded":"2023-10-30T14:14:05.585964+0000","last_fullsized":"2023-10-30T14:14:05.585964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:43:50.150156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584729+0000","last_change":"2023-10-30T14:14:04.599863+0000","last_active":"2023-10-30T14:14:05.584729+0000","last_peered":"2023-10-30T14:14:05.584729+0000","last_clean":"2023-10-30T14:14:05.584729+0000","last_became_active":"2023-10-30T14:14:04.599323+0000","last_became_peered":"2023-10-30T14:14:04.599323+0000","last_unstale":"2023-10-30T14:14:05.584729+0000","last_undegraded":"2023-10-30T14:14:05.584729+0000","last_fullsized":"2023-10-30T14:14:05.584729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:42:36.939959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808018+0000","last_change":"2023-10-30T14:14:03.738694+0000","last_active":"2023-10-30T14:14:05.808018+0000","last_peered":"2023-10-30T14:14:05.808018+0000","last_clean":"2023-10-30T14:14:05.808018+0000","last_became_active":"2023-10-30T14:14:03.738132+0000","last_became_peered":"2023-10-30T14:14:03.738132+0000","last_unstale":"2023-10-30T14:14:05.808018+0000","last_undegraded":"2023-10-30T14:14:05.808018+0000","last_fullsized":"2023-10-30T14:14:05.808018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:46:59.692900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586432+0000","last_change":"2023-10-30T14:14:04.649502+0000","last_active":"2023-10-30T14:14:05.586432+0000","last_peered":"2023-10-30T14:14:05.586432+0000","last_clean":"2023-10-30T14:14:05.586432+0000","last_became_active":"2023-10-30T14:14:04.648816+0000","last_became_peered":"2023-10-30T14:14:04.648816+0000","last_unstale":"2023-10-30T14:14:05.586432+0000","last_undegraded":"2023-10-30T14:14:05.586432+0000","last_fullsized":"2023-10-30T14:14:05.586432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:23:07.149253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586445+0000","last_change":"2023-10-30T14:14:03.726112+0000","last_active":"2023-10-30T14:14:05.586445+0000","last_peered":"2023-10-30T14:14:05.586445+0000","last_clean":"2023-10-30T14:14:05.586445+0000","last_became_active":"2023-10-30T14:14:03.725272+0000","last_became_peered":"2023-10-30T14:14:03.725272+0000","last_unstale":"2023-10-30T14:14:05.586445+0000","last_undegraded":"2023-10-30T14:14:05.586445+0000","last_fullsized":"2023-10-30T14:14:05.586445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:37:20.764294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586677+0000","last_change":"2023-10-30T14:14:04.610043+0000","last_active":"2023-10-30T14:14:05.586677+0000","last_peered":"2023-10-30T14:14:05.586677+0000","last_clean":"2023-10-30T14:14:05.586677+0000","last_became_active":"2023-10-30T14:14:04.609697+0000","last_became_peered":"2023-10-30T14:14:04.609697+0000","last_unstale":"2023-10-30T14:14:05.586677+0000","last_undegraded":"2023-10-30T14:14:05.586677+0000","last_fullsized":"2023-10-30T14:14:05.586677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:28.543547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584836+0000","last_change":"2023-10-30T14:14:03.727871+0000","last_active":"2023-10-30T14:14:05.584836+0000","last_peered":"2023-10-30T14:14:05.584836+0000","last_clean":"2023-10-30T14:14:05.584836+0000","last_became_active":"2023-10-30T14:14:03.727422+0000","last_became_peered":"2023-10-30T14:14:03.727422+0000","last_unstale":"2023-10-30T14:14:05.584836+0000","last_undegraded":"2023-10-30T14:14:05.584836+0000","last_fullsized":"2023-10-30T14:14:05.584836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:54.782533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809764+0000","last_change":"2023-10-30T14:14:04.645734+0000","last_active":"2023-10-30T14:14:05.809764+0000","last_peered":"2023-10-30T14:14:05.809764+0000","last_clean":"2023-10-30T14:14:05.809764+0000","last_became_active":"2023-10-30T14:14:04.645333+0000","last_became_peered":"2023-10-30T14:14:04.645333+0000","last_unstale":"2023-10-30T14:14:05.809764+0000","last_undegraded":"2023-10-30T14:14:05.809764+0000","last_fullsized":"2023-10-30T14:14:05.809764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:36:11.322798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808319+0000","last_change":"2023-10-30T14:14:03.725554+0000","last_active":"2023-10-30T14:14:05.808319+0000","last_peered":"2023-10-30T14:14:05.808319+0000","last_clean":"2023-10-30T14:14:05.808319+0000","last_became_active":"2023-10-30T14:14:03.724806+0000","last_became_peered":"2023-10-30T14:14:03.724806+0000","last_unstale":"2023-10-30T14:14:05.808319+0000","last_undegraded":"2023-10-30T14:14:05.808319+0000","last_fullsized":"2023-10-30T14:14:05.808319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:37.823561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756505+0000","last_change":"2023-10-30T14:14:04.647122+0000","last_active":"2023-10-30T14:14:05.756505+0000","last_peered":"2023-10-30T14:14:05.756505+0000","last_clean":"2023-10-30T14:14:05.756505+0000","last_became_active":"2023-10-30T14:14:04.646543+0000","last_became_peered":"2023-10-30T14:14:04.646543+0000","last_unstale":"2023-10-30T14:14:05.756505+0000","last_undegraded":"2023-10-30T14:14:05.756505+0000","last_fullsized":"2023-10-30T14:14:05.756505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:16.158442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586349+0000","last_change":"2023-10-30T14:14:03.717903+0000","last_active":"2023-10-30T14:14:05.586349+0000","last_peered":"2023-10-30T14:14:05.586349+0000","last_clean":"2023-10-30T14:14:05.586349+0000","last_became_active":"2023-10-30T14:14:03.716941+0000","last_became_peered":"2023-10-30T14:14:03.716941+0000","last_unstale":"2023-10-30T14:14:05.586349+0000","last_undegraded":"2023-10-30T14:14:05.586349+0000","last_fullsized":"2023-10-30T14:14:05.586349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:58.178166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586173+0000","last_change":"2023-10-30T14:14:04.641757+0000","last_active":"2023-10-30T14:14:05.586173+0000","last_peered":"2023-10-30T14:14:05.586173+0000","last_clean":"2023-10-30T14:14:05.586173+0000","last_became_active":"2023-10-30T14:14:04.641494+0000","last_became_peered":"2023-10-30T14:14:04.641494+0000","last_unstale":"2023-10-30T14:14:05.586173+0000","last_undegraded":"2023-10-30T14:14:05.586173+0000","last_fullsized":"2023-10-30T14:14:05.586173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:00:18.795083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587083+0000","last_change":"2023-10-30T14:14:03.729626+0000","last_active":"2023-10-30T14:14:05.587083+0000","last_peered":"2023-10-30T14:14:05.587083+0000","last_clean":"2023-10-30T14:14:05.587083+0000","last_became_active":"2023-10-30T14:14:03.728053+0000","last_became_peered":"2023-10-30T14:14:03.728053+0000","last_unstale":"2023-10-30T14:14:05.587083+0000","last_undegraded":"2023-10-30T14:14:05.587083+0000","last_fullsized":"2023-10-30T14:14:05.587083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:44:53.362066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756773+0000","last_change":"2023-10-30T14:14:04.650513+0000","last_active":"2023-10-30T14:14:05.756773+0000","last_peered":"2023-10-30T14:14:05.756773+0000","last_clean":"2023-10-30T14:14:05.756773+0000","last_became_active":"2023-10-30T14:14:04.649896+0000","last_became_peered":"2023-10-30T14:14:04.649896+0000","last_unstale":"2023-10-30T14:14:05.756773+0000","last_undegraded":"2023-10-30T14:14:05.756773+0000","last_fullsized":"2023-10-30T14:14:05.756773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:23.100490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807384+0000","last_change":"2023-10-30T14:14:03.736824+0000","last_active":"2023-10-30T14:14:05.807384+0000","last_peered":"2023-10-30T14:14:05.807384+0000","last_clean":"2023-10-30T14:14:05.807384+0000","last_became_active":"2023-10-30T14:14:03.736155+0000","last_became_peered":"2023-10-30T14:14:03.736155+0000","last_unstale":"2023-10-30T14:14:05.807384+0000","last_undegraded":"2023-10-30T14:14:05.807384+0000","last_fullsized":"2023-10-30T14:14:05.807384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:35.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807372+0000","last_change":"2023-10-30T14:14:04.615159+0000","last_active":"2023-10-30T14:14:05.807372+0000","last_peered":"2023-10-30T14:14:05.807372+0000","last_clean":"2023-10-30T14:14:05.807372+0000","last_became_active":"2023-10-30T14:14:04.614328+0000","last_became_peered":"2023-10-30T14:14:04.614328+0000","last_unstale":"2023-10-30T14:14:05.807372+0000","last_undegraded":"2023-10-30T14:14:05.807372+0000","last_fullsized":"2023-10-30T14:14:05.807372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:16:52.999786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805450+0000","last_change":"2023-10-30T14:14:03.700690+0000","last_active":"2023-10-30T14:14:05.805450+0000","last_peered":"2023-10-30T14:14:05.805450+0000","last_clean":"2023-10-30T14:14:05.805450+0000","last_became_active":"2023-10-30T14:14:03.700352+0000","last_became_peered":"2023-10-30T14:14:03.700352+0000","last_unstale":"2023-10-30T14:14:05.805450+0000","last_undegraded":"2023-10-30T14:14:05.805450+0000","last_fullsized":"2023-10-30T14:14:05.805450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:03:01.103359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755948+0000","last_change":"2023-10-30T14:14:04.647253+0000","last_active":"2023-10-30T14:14:05.755948+0000","last_peered":"2023-10-30T14:14:05.755948+0000","last_clean":"2023-10-30T14:14:05.755948+0000","last_became_active":"2023-10-30T14:14:04.646639+0000","last_became_peered":"2023-10-30T14:14:04.646639+0000","last_unstale":"2023-10-30T14:14:05.755948+0000","last_undegraded":"2023-10-30T14:14:05.755948+0000","last_fullsized":"2023-10-30T14:14:05.755948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:22:50.893734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.810120+0000","last_change":"2023-10-30T14:14:03.710717+0000","last_active":"2023-10-30T14:14:05.810120+0000","last_peered":"2023-10-30T14:14:05.810120+0000","last_clean":"2023-10-30T14:14:05.810120+0000","last_became_active":"2023-10-30T14:14:03.707950+0000","last_became_peered":"2023-10-30T14:14:03.707950+0000","last_unstale":"2023-10-30T14:14:05.810120+0000","last_undegraded":"2023-10-30T14:14:05.810120+0000","last_fullsized":"2023-10-30T14:14:05.810120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:29:41.232902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757066+0000","last_change":"2023-10-30T14:14:04.647688+0000","last_active":"2023-10-30T14:14:05.757066+0000","last_peered":"2023-10-30T14:14:05.757066+0000","last_clean":"2023-10-30T14:14:05.757066+0000","last_became_active":"2023-10-30T14:14:04.644761+0000","last_became_peered":"2023-10-30T14:14:04.644761+0000","last_unstale":"2023-10-30T14:14:05.757066+0000","last_undegraded":"2023-10-30T14:14:05.757066+0000","last_fullsized":"2023-10-30T14:14:05.757066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:00:38.482522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805776+0000","last_change":"2023-10-30T14:14:03.705567+0000","last_active":"2023-10-30T14:14:05.805776+0000","last_peered":"2023-10-30T14:14:05.805776+0000","last_clean":"2023-10-30T14:14:05.805776+0000","last_became_active":"2023-10-30T14:14:03.705186+0000","last_became_peered":"2023-10-30T14:14:03.705186+0000","last_unstale":"2023-10-30T14:14:05.805776+0000","last_undegraded":"2023-10-30T14:14:05.805776+0000","last_fullsized":"2023-10-30T14:14:05.805776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:14.609243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757235+0000","last_change":"2023-10-30T14:14:04.647335+0000","last_active":"2023-10-30T14:14:05.757235+0000","last_peered":"2023-10-30T14:14:05.757235+0000","last_clean":"2023-10-30T14:14:05.757235+0000","last_became_active":"2023-10-30T14:14:04.646785+0000","last_became_peered":"2023-10-30T14:14:04.646785+0000","last_unstale":"2023-10-30T14:14:05.757235+0000","last_undegraded":"2023-10-30T14:14:05.757235+0000","last_fullsized":"2023-10-30T14:14:05.757235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:58:41.600761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584948+0000","last_change":"2023-10-30T14:14:03.693349+0000","last_active":"2023-10-30T14:14:05.584948+0000","last_peered":"2023-10-30T14:14:05.584948+0000","last_clean":"2023-10-30T14:14:05.584948+0000","last_became_active":"2023-10-30T14:14:03.692966+0000","last_became_peered":"2023-10-30T14:14:03.692966+0000","last_unstale":"2023-10-30T14:14:05.584948+0000","last_undegraded":"2023-10-30T14:14:05.584948+0000","last_fullsized":"2023-10-30T14:14:05.584948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:30:28.800214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587213+0000","last_change":"2023-10-30T14:14:04.617023+0000","last_active":"2023-10-30T14:14:05.587213+0000","last_peered":"2023-10-30T14:14:05.587213+0000","last_clean":"2023-10-30T14:14:05.587213+0000","last_became_active":"2023-10-30T14:14:04.615925+0000","last_became_peered":"2023-10-30T14:14:04.615925+0000","last_unstale":"2023-10-30T14:14:05.587213+0000","last_undegraded":"2023-10-30T14:14:05.587213+0000","last_fullsized":"2023-10-30T14:14:05.587213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:32:02.289324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805586+0000","last_change":"2023-10-30T14:14:03.727659+0000","last_active":"2023-10-30T14:14:05.805586+0000","last_peered":"2023-10-30T14:14:05.805586+0000","last_clean":"2023-10-30T14:14:05.805586+0000","last_became_active":"2023-10-30T14:14:03.727409+0000","last_became_peered":"2023-10-30T14:14:03.727409+0000","last_unstale":"2023-10-30T14:14:05.805586+0000","last_undegraded":"2023-10-30T14:14:05.805586+0000","last_fullsized":"2023-10-30T14:14:05.805586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:24.210635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756468+0000","last_change":"2023-10-30T14:14:04.650083+0000","last_active":"2023-10-30T14:14:05.756468+0000","last_peered":"2023-10-30T14:14:05.756468+0000","last_clean":"2023-10-30T14:14:05.756468+0000","last_became_active":"2023-10-30T14:14:04.647562+0000","last_became_peered":"2023-10-30T14:14:04.647562+0000","last_unstale":"2023-10-30T14:14:05.756468+0000","last_undegraded":"2023-10-30T14:14:05.756468+0000","last_fullsized":"2023-10-30T14:14:05.756468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:11:41.743850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T14:14:04.632968+0000","last_change":"2023-10-30T14:13:54.251572+0000","last_active":"2023-10-30T14:14:04.632968+0000","last_peered":"2023-10-30T14:14:04.632968+0000","last_clean":"2023-10-30T14:14:04.632968+0000","last_became_active":"2023-10-30T14:13:54.250907+0000","last_became_peered":"2023-10-30T14:13:54.250907+0000","last_unstale":"2023-10-30T14:14:04.632968+0000","last_undegraded":"2023-10-30T14:14:04.632968+0000","last_fullsized":"2023-10-30T14:14:04.632968+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_clean_scrub_stamp":"2023-10-30T14:13:53.220267+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:44.590852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585892+0000","last_change":"2023-10-30T14:14:03.729034+0000","last_active":"2023-10-30T14:14:05.585892+0000","last_peered":"2023-10-30T14:14:05.585892+0000","last_clean":"2023-10-30T14:14:05.585892+0000","last_became_active":"2023-10-30T14:14:03.728293+0000","last_became_peered":"2023-10-30T14:14:03.728293+0000","last_unstale":"2023-10-30T14:14:05.585892+0000","last_undegraded":"2023-10-30T14:14:05.585892+0000","last_fullsized":"2023-10-30T14:14:05.585892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806225+0000","last_change":"2023-10-30T14:14:04.630325+0000","last_active":"2023-10-30T14:14:05.806225+0000","last_peered":"2023-10-30T14:14:05.806225+0000","last_clean":"2023-10-30T14:14:05.806225+0000","last_became_active":"2023-10-30T14:14:04.630121+0000","last_became_peered":"2023-10-30T14:14:04.630121+0000","last_unstale":"2023-10-30T14:14:05.806225+0000","last_undegraded":"2023-10-30T14:14:05.806225+0000","last_fullsized":"2023-10-30T14:14:05.806225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:02:28.792879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755708+0000","last_change":"2023-10-30T14:14:03.732909+0000","last_active":"2023-10-30T14:14:05.755708+0000","last_peered":"2023-10-30T14:14:05.755708+0000","last_clean":"2023-10-30T14:14:05.755708+0000","last_became_active":"2023-10-30T14:14:03.732199+0000","last_became_peered":"2023-10-30T14:14:03.732199+0000","last_unstale":"2023-10-30T14:14:05.755708+0000","last_undegraded":"2023-10-30T14:14:05.755708+0000","last_fullsized":"2023-10-30T14:14:05.755708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:38.117750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586172+0000","last_change":"2023-10-30T14:14:04.620314+0000","last_active":"2023-10-30T14:14:05.586172+0000","last_peered":"2023-10-30T14:14:05.586172+0000","last_clean":"2023-10-30T14:14:05.586172+0000","last_became_active":"2023-10-30T14:14:04.619990+0000","last_became_peered":"2023-10-30T14:14:04.619990+0000","last_unstale":"2023-10-30T14:14:05.586172+0000","last_undegraded":"2023-10-30T14:14:05.586172+0000","last_fullsized":"2023-10-30T14:14:05.586172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:38:25.657586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805642+0000","last_change":"2023-10-30T14:14:03.702862+0000","last_active":"2023-10-30T14:14:05.805642+0000","last_peered":"2023-10-30T14:14:05.805642+0000","last_clean":"2023-10-30T14:14:05.805642+0000","last_became_active":"2023-10-30T14:14:03.702513+0000","last_became_peered":"2023-10-30T14:14:03.702513+0000","last_unstale":"2023-10-30T14:14:05.805642+0000","last_undegraded":"2023-10-30T14:14:05.805642+0000","last_fullsized":"2023-10-30T14:14:05.805642+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:25.499390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584623+0000","last_change":"2023-10-30T14:14:04.589353+0000","last_active":"2023-10-30T14:14:05.584623+0000","last_peered":"2023-10-30T14:14:05.584623+0000","last_clean":"2023-10-30T14:14:05.584623+0000","last_became_active":"2023-10-30T14:14:04.589087+0000","last_became_peered":"2023-10-30T14:14:04.589087+0000","last_unstale":"2023-10-30T14:14:05.584623+0000","last_undegraded":"2023-10-30T14:14:05.584623+0000","last_fullsized":"2023-10-30T14:14:05.584623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:20:12.808641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587305+0000","last_change":"2023-10-30T14:14:03.730048+0000","last_active":"2023-10-30T14:14:05.587305+0000","last_peered":"2023-10-30T14:14:05.587305+0000","last_clean":"2023-10-30T14:14:05.587305+0000","last_became_active":"2023-10-30T14:14:03.728925+0000","last_became_peered":"2023-10-30T14:14:03.728925+0000","last_unstale":"2023-10-30T14:14:05.587305+0000","last_undegraded":"2023-10-30T14:14:05.587305+0000","last_fullsized":"2023-10-30T14:14:05.587305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:32:05.056039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809437+0000","last_change":"2023-10-30T14:14:04.608289+0000","last_active":"2023-10-30T14:14:05.809437+0000","last_peered":"2023-10-30T14:14:05.809437+0000","last_clean":"2023-10-30T14:14:05.809437+0000","last_became_active":"2023-10-30T14:14:04.607377+0000","last_became_peered":"2023-10-30T14:14:04.607377+0000","last_unstale":"2023-10-30T14:14:05.809437+0000","last_undegraded":"2023-10-30T14:14:05.809437+0000","last_fullsized":"2023-10-30T14:14:05.809437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:43:40.869579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584795+0000","last_change":"2023-10-30T14:14:03.693753+0000","last_active":"2023-10-30T14:14:05.584795+0000","last_peered":"2023-10-30T14:14:05.584795+0000","last_clean":"2023-10-30T14:14:05.584795+0000","last_became_active":"2023-10-30T14:14:03.692621+0000","last_became_peered":"2023-10-30T14:14:03.692621+0000","last_unstale":"2023-10-30T14:14:05.584795+0000","last_undegraded":"2023-10-30T14:14:05.584795+0000","last_fullsized":"2023-10-30T14:14:05.584795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:32:28.577584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587225+0000","last_change":"2023-10-30T14:14:04.614649+0000","last_active":"2023-10-30T14:14:05.587225+0000","last_peered":"2023-10-30T14:14:05.587225+0000","last_clean":"2023-10-30T14:14:05.587225+0000","last_became_active":"2023-10-30T14:14:04.613780+0000","last_became_peered":"2023-10-30T14:14:04.613780+0000","last_unstale":"2023-10-30T14:14:05.587225+0000","last_undegraded":"2023-10-30T14:14:05.587225+0000","last_fullsized":"2023-10-30T14:14:05.587225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:47:40.354022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757037+0000","last_change":"2023-10-30T14:14:03.740213+0000","last_active":"2023-10-30T14:14:05.757037+0000","last_peered":"2023-10-30T14:14:05.757037+0000","last_clean":"2023-10-30T14:14:05.757037+0000","last_became_active":"2023-10-30T14:14:03.739650+0000","last_became_peered":"2023-10-30T14:14:03.739650+0000","last_unstale":"2023-10-30T14:14:05.757037+0000","last_undegraded":"2023-10-30T14:14:05.757037+0000","last_fullsized":"2023-10-30T14:14:05.757037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:54.120757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584691+0000","last_change":"2023-10-30T14:14:04.590569+0000","last_active":"2023-10-30T14:14:05.584691+0000","last_peered":"2023-10-30T14:14:05.584691+0000","last_clean":"2023-10-30T14:14:05.584691+0000","last_became_active":"2023-10-30T14:14:04.590138+0000","last_became_peered":"2023-10-30T14:14:04.590138+0000","last_unstale":"2023-10-30T14:14:05.584691+0000","last_undegraded":"2023-10-30T14:14:05.584691+0000","last_fullsized":"2023-10-30T14:14:05.584691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:41:30.333668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585050+0000","last_change":"2023-10-30T14:14:03.721450+0000","last_active":"2023-10-30T14:14:05.585050+0000","last_peered":"2023-10-30T14:14:05.585050+0000","last_clean":"2023-10-30T14:14:05.585050+0000","last_became_active":"2023-10-30T14:14:03.721234+0000","last_became_peered":"2023-10-30T14:14:03.721234+0000","last_unstale":"2023-10-30T14:14:05.585050+0000","last_undegraded":"2023-10-30T14:14:05.585050+0000","last_fullsized":"2023-10-30T14:14:05.585050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:26.959676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806005+0000","last_change":"2023-10-30T14:14:04.634004+0000","last_active":"2023-10-30T14:14:05.806005+0000","last_peered":"2023-10-30T14:14:05.806005+0000","last_clean":"2023-10-30T14:14:05.806005+0000","last_became_active":"2023-10-30T14:14:04.633643+0000","last_became_peered":"2023-10-30T14:14:04.633643+0000","last_unstale":"2023-10-30T14:14:05.806005+0000","last_undegraded":"2023-10-30T14:14:05.806005+0000","last_fullsized":"2023-10-30T14:14:05.806005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:05:14.572477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808140+0000","last_change":"2023-10-30T14:14:03.738557+0000","last_active":"2023-10-30T14:14:05.808140+0000","last_peered":"2023-10-30T14:14:05.808140+0000","last_clean":"2023-10-30T14:14:05.808140+0000","last_became_active":"2023-10-30T14:14:03.737801+0000","last_became_peered":"2023-10-30T14:14:03.737801+0000","last_unstale":"2023-10-30T14:14:05.808140+0000","last_undegraded":"2023-10-30T14:14:05.808140+0000","last_fullsized":"2023-10-30T14:14:05.808140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:26.938343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586581+0000","last_change":"2023-10-30T14:14:04.589273+0000","last_active":"2023-10-30T14:14:05.586581+0000","last_peered":"2023-10-30T14:14:05.586581+0000","last_clean":"2023-10-30T14:14:05.586581+0000","last_became_active":"2023-10-30T14:14:04.588826+0000","last_became_peered":"2023-10-30T14:14:04.588826+0000","last_unstale":"2023-10-30T14:14:05.586581+0000","last_undegraded":"2023-10-30T14:14:05.586581+0000","last_fullsized":"2023-10-30T14:14:05.586581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:33:52.452394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755689+0000","last_change":"2023-10-30T14:14:03.733836+0000","last_active":"2023-10-30T14:14:05.755689+0000","last_peered":"2023-10-30T14:14:05.755689+0000","last_clean":"2023-10-30T14:14:05.755689+0000","last_became_active":"2023-10-30T14:14:03.732912+0000","last_became_peered":"2023-10-30T14:14:03.732912+0000","last_unstale":"2023-10-30T14:14:05.755689+0000","last_undegraded":"2023-10-30T14:14:05.755689+0000","last_fullsized":"2023-10-30T14:14:05.755689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:50.332738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809314+0000","last_change":"2023-10-30T14:14:04.610696+0000","last_active":"2023-10-30T14:14:05.809314+0000","last_peered":"2023-10-30T14:14:05.809314+0000","last_clean":"2023-10-30T14:14:05.809314+0000","last_became_active":"2023-10-30T14:14:04.609835+0000","last_became_peered":"2023-10-30T14:14:04.609835+0000","last_unstale":"2023-10-30T14:14:05.809314+0000","last_undegraded":"2023-10-30T14:14:05.809314+0000","last_fullsized":"2023-10-30T14:14:05.809314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:04:31.270262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584669+0000","last_change":"2023-10-30T14:14:03.732539+0000","last_active":"2023-10-30T14:14:05.584669+0000","last_peered":"2023-10-30T14:14:05.584669+0000","last_clean":"2023-10-30T14:14:05.584669+0000","last_became_active":"2023-10-30T14:14:03.732265+0000","last_became_peered":"2023-10-30T14:14:03.732265+0000","last_unstale":"2023-10-30T14:14:05.584669+0000","last_undegraded":"2023-10-30T14:14:05.584669+0000","last_fullsized":"2023-10-30T14:14:05.584669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:02:00.929612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806211+0000","last_change":"2023-10-30T14:14:04.595357+0000","last_active":"2023-10-30T14:14:05.806211+0000","last_peered":"2023-10-30T14:14:05.806211+0000","last_clean":"2023-10-30T14:14:05.806211+0000","last_became_active":"2023-10-30T14:14:04.594998+0000","last_became_peered":"2023-10-30T14:14:04.594998+0000","last_unstale":"2023-10-30T14:14:05.806211+0000","last_undegraded":"2023-10-30T14:14:05.806211+0000","last_fullsized":"2023-10-30T14:14:05.806211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:44.365673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584615+0000","last_change":"2023-10-30T14:14:03.703744+0000","last_active":"2023-10-30T14:14:05.584615+0000","last_peered":"2023-10-30T14:14:05.584615+0000","last_clean":"2023-10-30T14:14:05.584615+0000","last_became_active":"2023-10-30T14:14:03.701759+0000","last_became_peered":"2023-10-30T14:14:03.701759+0000","last_unstale":"2023-10-30T14:14:05.584615+0000","last_undegraded":"2023-10-30T14:14:05.584615+0000","last_fullsized":"2023-10-30T14:14:05.584615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:22:27.948007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806186+0000","last_change":"2023-10-30T14:14:04.600759+0000","last_active":"2023-10-30T14:14:05.806186+0000","last_peered":"2023-10-30T14:14:05.806186+0000","last_clean":"2023-10-30T14:14:05.806186+0000","last_became_active":"2023-10-30T14:14:04.600100+0000","last_became_peered":"2023-10-30T14:14:04.600100+0000","last_unstale":"2023-10-30T14:14:05.806186+0000","last_undegraded":"2023-10-30T14:14:05.806186+0000","last_fullsized":"2023-10-30T14:14:05.806186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:02.831207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756778+0000","last_change":"2023-10-30T14:14:03.741268+0000","last_active":"2023-10-30T14:14:05.756778+0000","last_peered":"2023-10-30T14:14:05.756778+0000","last_clean":"2023-10-30T14:14:05.756778+0000","last_became_active":"2023-10-30T14:14:03.740648+0000","last_became_peered":"2023-10-30T14:14:03.740648+0000","last_unstale":"2023-10-30T14:14:05.756778+0000","last_undegraded":"2023-10-30T14:14:05.756778+0000","last_fullsized":"2023-10-30T14:14:05.756778+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:30.799148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807541+0000","last_change":"2023-10-30T14:14:04.623644+0000","last_active":"2023-10-30T14:14:05.807541+0000","last_peered":"2023-10-30T14:14:05.807541+0000","last_clean":"2023-10-30T14:14:05.807541+0000","last_became_active":"2023-10-30T14:14:04.622921+0000","last_became_peered":"2023-10-30T14:14:04.622921+0000","last_unstale":"2023-10-30T14:14:05.807541+0000","last_undegraded":"2023-10-30T14:14:05.807541+0000","last_fullsized":"2023-10-30T14:14:05.807541+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:11:41.810610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587103+0000","last_change":"2023-10-30T14:14:04.624799+0000","last_active":"2023-10-30T14:14:05.587103+0000","last_peered":"2023-10-30T14:14:05.587103+0000","last_clean":"2023-10-30T14:14:05.587103+0000","last_became_active":"2023-10-30T14:14:04.621885+0000","last_became_peered":"2023-10-30T14:14:04.621885+0000","last_unstale":"2023-10-30T14:14:05.587103+0000","last_undegraded":"2023-10-30T14:14:05.587103+0000","last_fullsized":"2023-10-30T14:14:05.587103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:12.055247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756286+0000","last_change":"2023-10-30T14:14:03.740897+0000","last_active":"2023-10-30T14:14:05.756286+0000","last_peered":"2023-10-30T14:14:05.756286+0000","last_clean":"2023-10-30T14:14:05.756286+0000","last_became_active":"2023-10-30T14:14:03.740247+0000","last_became_peered":"2023-10-30T14:14:03.740247+0000","last_unstale":"2023-10-30T14:14:05.756286+0000","last_undegraded":"2023-10-30T14:14:05.756286+0000","last_fullsized":"2023-10-30T14:14:05.756286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:29.601978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586861+0000","last_change":"2023-10-30T14:14:04.625498+0000","last_active":"2023-10-30T14:14:05.586861+0000","last_peered":"2023-10-30T14:14:05.586861+0000","last_clean":"2023-10-30T14:14:05.586861+0000","last_became_active":"2023-10-30T14:14:04.623638+0000","last_became_peered":"2023-10-30T14:14:04.623638+0000","last_unstale":"2023-10-30T14:14:05.586861+0000","last_undegraded":"2023-10-30T14:14:05.586861+0000","last_fullsized":"2023-10-30T14:14:05.586861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:19.717965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756898+0000","last_change":"2023-10-30T14:14:03.733233+0000","last_active":"2023-10-30T14:14:05.756898+0000","last_peered":"2023-10-30T14:14:05.756898+0000","last_clean":"2023-10-30T14:14:05.756898+0000","last_became_active":"2023-10-30T14:14:03.732538+0000","last_became_peered":"2023-10-30T14:14:03.732538+0000","last_unstale":"2023-10-30T14:14:05.756898+0000","last_undegraded":"2023-10-30T14:14:05.756898+0000","last_fullsized":"2023-10-30T14:14:05.756898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:25.696173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586619+0000","last_change":"2023-10-30T14:14:04.612104+0000","last_active":"2023-10-30T14:14:05.586619+0000","last_peered":"2023-10-30T14:14:05.586619+0000","last_clean":"2023-10-30T14:14:05.586619+0000","last_became_active":"2023-10-30T14:14:04.611773+0000","last_became_peered":"2023-10-30T14:14:04.611773+0000","last_unstale":"2023-10-30T14:14:05.586619+0000","last_undegraded":"2023-10-30T14:14:05.586619+0000","last_fullsized":"2023-10-30T14:14:05.586619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:44:21.952103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806100+0000","last_change":"2023-10-30T14:14:03.703988+0000","last_active":"2023-10-30T14:14:05.806100+0000","last_peered":"2023-10-30T14:14:05.806100+0000","last_clean":"2023-10-30T14:14:05.806100+0000","last_became_active":"2023-10-30T14:14:03.703691+0000","last_became_peered":"2023-10-30T14:14:03.703691+0000","last_unstale":"2023-10-30T14:14:05.806100+0000","last_undegraded":"2023-10-30T14:14:05.806100+0000","last_fullsized":"2023-10-30T14:14:05.806100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:34.538541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806119+0000","last_change":"2023-10-30T14:14:04.613738+0000","last_active":"2023-10-30T14:14:05.806119+0000","last_peered":"2023-10-30T14:14:05.806119+0000","last_clean":"2023-10-30T14:14:05.806119+0000","last_became_active":"2023-10-30T14:14:04.613420+0000","last_became_peered":"2023-10-30T14:14:04.613420+0000","last_unstale":"2023-10-30T14:14:05.806119+0000","last_undegraded":"2023-10-30T14:14:05.806119+0000","last_fullsized":"2023-10-30T14:14:05.806119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:01:52.695195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586668+0000","last_change":"2023-10-30T14:14:03.737319+0000","last_active":"2023-10-30T14:14:05.586668+0000","last_peered":"2023-10-30T14:14:05.586668+0000","last_clean":"2023-10-30T14:14:05.586668+0000","last_became_active":"2023-10-30T14:14:03.736964+0000","last_became_peered":"2023-10-30T14:14:03.736964+0000","last_unstale":"2023-10-30T14:14:05.586668+0000","last_undegraded":"2023-10-30T14:14:05.586668+0000","last_fullsized":"2023-10-30T14:14:05.586668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:41.469809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":665937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4039999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.458}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1639999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7909999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2829999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.222}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.276}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":75569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1519999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":811008,"data_stored":665937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:14:10.863 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:14:11.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.036+0000 7f2bac31d640 1 Processor -- start 2023-10-30T14:14:11.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.036+0000 7f2bac31d640 1 -- start start 2023-10-30T14:14:11.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.036+0000 7f2bac31d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba4151840 0x7f2ba40c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:11.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.036+0000 7f2bac31d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2ba4151c20 0x7f2ba4072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:11.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.036+0000 7f2bac31d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba40c5030 con 0x7f2ba40cb8b0 2023-10-30T14:14:11.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.036+0000 7f2bac31d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ba4157830 con 0x7f2ba4151c20 2023-10-30T14:14:11.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.036+0000 7f2bac31d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba4157b10 con 0x7f2ba4151840 2023-10-30T14:14:11.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.036+0000 7f2ba98b8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2ba4151c20 0x7f2ba4072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:11.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.036+0000 7f2baa0b9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba4151840 0x7f2ba40c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:11.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.040+0000 7f2ba98b8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2ba4151c20 0x7f2ba4072000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:44766/0 (socket says 172.21.15.57:44766) 2023-10-30T14:14:11.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.040+0000 7f2ba98b8640 1 -- 172.21.15.57:0/3384134703 learned_addr learned my addr 172.21.15.57:0/3384134703 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:11.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.040+0000 7f2baa0b9640 1 -- 172.21.15.57:0/3384134703 >> v2:172.21.15.88:3300/0 conn(0x7f2ba4151c20 msgr2=0x7f2ba4072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:11.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.040+0000 7f2baa0b9640 1 --2- 172.21.15.57:0/3384134703 >> v2:172.21.15.88:3300/0 conn(0x7f2ba4151c20 0x7f2ba4072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.040+0000 7f2baa0b9640 1 -- 172.21.15.57:0/3384134703 >> v1:172.21.15.88:6789/0 conn(0x7f2ba40cb8b0 legacy=0x7f2ba406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:11.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.040+0000 7f2baa0b9640 1 -- 172.21.15.57:0/3384134703 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba4157df0 con 0x7f2ba4151840 2023-10-30T14:14:11.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.040+0000 7f2baa0b9640 1 --2- 172.21.15.57:0/3384134703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba4151840 0x7f2ba40c49a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2ba00097c0 tx=0x7f2ba001ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b7b3ae900c6ad61 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:11.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2ba90b7640 1 -- 172.21.15.57:0/3384134703 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba00a8970 con 0x7f2ba4151840 2023-10-30T14:14:11.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2ba90b7640 1 -- 172.21.15.57:0/3384134703 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba00a8ad0 con 0x7f2ba4151840 2023-10-30T14:14:11.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2ba90b7640 1 -- 172.21.15.57:0/3384134703 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba00a8e60 con 0x7f2ba4151840 2023-10-30T14:14:11.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2bac31d640 1 -- 172.21.15.57:0/3384134703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba4151840 msgr2=0x7f2ba40c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:11.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2bac31d640 1 --2- 172.21.15.57:0/3384134703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba4151840 0x7f2ba40c49a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2ba00097c0 tx=0x7f2ba001ec70 comp rx=0 tx=0).stop 2023-10-30T14:14:11.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2bac31d640 1 -- 172.21.15.57:0/3384134703 shutdown_connections 2023-10-30T14:14:11.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2bac31d640 1 --2- 172.21.15.57:0/3384134703 >> v2:172.21.15.88:3300/0 conn(0x7f2ba4151c20 0x7f2ba4072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2bac31d640 1 --2- 172.21.15.57:0/3384134703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba4151840 0x7f2ba40c49a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2bac31d640 1 -- 172.21.15.57:0/3384134703 >> 172.21.15.57:0/3384134703 conn(0x7f2ba40cac70 msgr2=0x7f2ba40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:11.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2bac31d640 1 -- 172.21.15.57:0/3384134703 shutdown_connections 2023-10-30T14:14:11.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2bac31d640 1 -- 172.21.15.57:0/3384134703 wait complete. 2023-10-30T14:14:11.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2bac31d640 1 Processor -- start 2023-10-30T14:14:11.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2bac31d640 1 -- start start 2023-10-30T14:14:11.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2bac31d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba40cb8b0 0x7f2ba41ef300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:11.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2bac31d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ba4151840 0x7f2ba4155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:11.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2bac31d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba4151c20 0x7f2ba41566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:11.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2bac31d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba4152650 con 0x7f2ba4151840 2023-10-30T14:14:11.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2bac31d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba4146e20 con 0x7f2ba4151c20 2023-10-30T14:14:11.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2bac31d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba41ef840 con 0x7f2ba40cb8b0 2023-10-30T14:14:11.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2ba98b8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ba4151840 0x7f2ba4155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:11.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2ba98b8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ba4151840 0x7f2ba4155c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37882/0 (socket says 172.21.15.57:37882) 2023-10-30T14:14:11.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.044+0000 7f2ba98b8640 1 -- 172.21.15.57:0/2187407767 learned_addr learned my addr 172.21.15.57:0/2187407767 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:11.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.048+0000 7f2baa0b9640 1 --2- 172.21.15.57:0/2187407767 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba40cb8b0 0x7f2ba41ef300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:11.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.048+0000 7f2baa8ba640 1 --2- 172.21.15.57:0/2187407767 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba4151c20 0x7f2ba41566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:11.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.048+0000 7f2ba98b8640 1 -- 172.21.15.57:0/2187407767 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba40cb8b0 msgr2=0x7f2ba41ef300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:11.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.048+0000 7f2ba98b8640 1 --2- 172.21.15.57:0/2187407767 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba40cb8b0 0x7f2ba41ef300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.048+0000 7f2ba98b8640 1 -- 172.21.15.57:0/2187407767 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba4151c20 msgr2=0x7f2ba41566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:11.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.048+0000 7f2ba98b8640 1 --2- 172.21.15.57:0/2187407767 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba4151c20 0x7f2ba41566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.048+0000 7f2ba98b8640 1 -- 172.21.15.57:0/2187407767 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba0009420 con 0x7f2ba4151840 2023-10-30T14:14:11.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.048+0000 7f2ba98b8640 1 --2- 172.21.15.57:0/2187407767 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ba4151840 0x7f2ba4155c50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2b9400e8a0 tx=0x7f2b9400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:11.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.048+0000 7f2b9affd640 1 -- 172.21.15.57:0/2187407767 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b940094e0 con 0x7f2ba4151840 2023-10-30T14:14:11.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.048+0000 7f2b9affd640 1 -- 172.21.15.57:0/2187407767 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9400c8e0 con 0x7f2ba4151840 2023-10-30T14:14:11.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.048+0000 7f2b9affd640 1 -- 172.21.15.57:0/2187407767 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9400cc70 con 0x7f2ba4151840 2023-10-30T14:14:11.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.048+0000 7f2bac31d640 1 -- 172.21.15.57:0/2187407767 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba4146330 con 0x7f2ba4151840 2023-10-30T14:14:11.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.048+0000 7f2bac31d640 1 -- 172.21.15.57:0/2187407767 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba41557c0 con 0x7f2ba4151840 2023-10-30T14:14:11.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.048+0000 7f2bac31d640 1 -- 172.21.15.57:0/2187407767 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba4155950 con 0x7f2ba4151840 2023-10-30T14:14:11.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.048+0000 7f2b9affd640 1 -- 172.21.15.57:0/2187407767 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2b94013470 con 0x7f2ba4151840 2023-10-30T14:14:11.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.052+0000 7f2b9affd640 1 --2- 172.21.15.57:0/2187407767 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b880664a0 0x7f2b88068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:11.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.052+0000 7f2b9affd640 1 -- 172.21.15.57:0/2187407767 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b94083df0 con 0x7f2ba4151840 2023-10-30T14:14:11.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.052+0000 7f2baa0b9640 1 --2- 172.21.15.57:0/2187407767 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b880664a0 0x7f2b88068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:11.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.052+0000 7f2baa0b9640 1 --2- 172.21.15.57:0/2187407767 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b880664a0 0x7f2b88068960 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2ba00b2000 tx=0x7f2ba00aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:11.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.056+0000 7f2b9affd640 1 -- 172.21.15.57:0/2187407767 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2b94051320 con 0x7f2ba4151840 2023-10-30T14:14:11.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.208+0000 7f2bac31d640 1 -- 172.21.15.57:0/2187407767 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ba41583b0 con 0x7f2b880664a0 2023-10-30T14:14:11.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.228+0000 7f2b9affd640 1 -- 172.21.15.57:0/2187407767 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264470 (secure 0 0 0) 0x7f2ba00c0090 con 0x7f2b880664a0 2023-10-30T14:14:11.234 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:11.236 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-30T14:14:11.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.228+0000 7f2bac31d640 1 -- 172.21.15.57:0/2187407767 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b880664a0 msgr2=0x7f2b88068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:11.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.228+0000 7f2bac31d640 1 --2- 172.21.15.57:0/2187407767 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b880664a0 0x7f2b88068960 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2ba00b2000 tx=0x7f2ba00aa070 comp rx=0 tx=0).stop 2023-10-30T14:14:11.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.228+0000 7f2bac31d640 1 -- 172.21.15.57:0/2187407767 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ba4151840 msgr2=0x7f2ba4155c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:11.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.228+0000 7f2bac31d640 1 --2- 172.21.15.57:0/2187407767 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ba4151840 0x7f2ba4155c50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2b9400e8a0 tx=0x7f2b9400ed60 comp rx=0 tx=0).stop 2023-10-30T14:14:11.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.228+0000 7f2bac31d640 1 -- 172.21.15.57:0/2187407767 shutdown_connections 2023-10-30T14:14:11.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.228+0000 7f2bac31d640 1 --2- 172.21.15.57:0/2187407767 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b880664a0 0x7f2b88068960 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.228+0000 7f2bac31d640 1 --2- 172.21.15.57:0/2187407767 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba4151c20 0x7f2ba41566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.228+0000 7f2bac31d640 1 --2- 172.21.15.57:0/2187407767 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ba4151840 0x7f2ba4155c50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.228+0000 7f2bac31d640 1 --2- 172.21.15.57:0/2187407767 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba40cb8b0 0x7f2ba41ef300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.228+0000 7f2bac31d640 1 -- 172.21.15.57:0/2187407767 >> 172.21.15.57:0/2187407767 conn(0x7f2ba40cac70 msgr2=0x7f2ba4073730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:11.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.228+0000 7f2bac31d640 1 -- 172.21.15.57:0/2187407767 shutdown_connections 2023-10-30T14:14:11.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.228+0000 7f2bac31d640 1 -- 172.21.15.57:0/2187407767 wait complete. 2023-10-30T14:14:11.254 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-30T14:14:10.494590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227364,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754747356,"statfs":{"total":773094113280,"available":772861292544,"internally_reserved":0,"allocated":2924544,"data_stored":1785288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":229756473},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.813582"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757043+0000","last_change":"2023-10-30T14:14:04.651413+0000","last_active":"2023-10-30T14:14:05.757043+0000","last_peered":"2023-10-30T14:14:05.757043+0000","last_clean":"2023-10-30T14:14:05.757043+0000","last_became_active":"2023-10-30T14:14:04.650882+0000","last_became_peered":"2023-10-30T14:14:04.650882+0000","last_unstale":"2023-10-30T14:14:05.757043+0000","last_undegraded":"2023-10-30T14:14:05.757043+0000","last_fullsized":"2023-10-30T14:14:05.757043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:30:08.122326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585036+0000","last_change":"2023-10-30T14:14:04.588523+0000","last_active":"2023-10-30T14:14:05.585036+0000","last_peered":"2023-10-30T14:14:05.585036+0000","last_clean":"2023-10-30T14:14:05.585036+0000","last_became_active":"2023-10-30T14:14:04.587942+0000","last_became_peered":"2023-10-30T14:14:04.587942+0000","last_unstale":"2023-10-30T14:14:05.585036+0000","last_undegraded":"2023-10-30T14:14:05.585036+0000","last_fullsized":"2023-10-30T14:14:05.585036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:16.028564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586543+0000","last_change":"2023-10-30T14:14:04.590976+0000","last_active":"2023-10-30T14:14:05.586543+0000","last_peered":"2023-10-30T14:14:05.586543+0000","last_clean":"2023-10-30T14:14:05.586543+0000","last_became_active":"2023-10-30T14:14:04.590700+0000","last_became_peered":"2023-10-30T14:14:04.590700+0000","last_unstale":"2023-10-30T14:14:05.586543+0000","last_undegraded":"2023-10-30T14:14:05.586543+0000","last_fullsized":"2023-10-30T14:14:05.586543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:49.169325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586525+0000","last_change":"2023-10-30T14:14:04.618603+0000","last_active":"2023-10-30T14:14:05.586525+0000","last_peered":"2023-10-30T14:14:05.586525+0000","last_clean":"2023-10-30T14:14:05.586525+0000","last_became_active":"2023-10-30T14:14:04.617768+0000","last_became_peered":"2023-10-30T14:14:04.617768+0000","last_unstale":"2023-10-30T14:14:05.586525+0000","last_undegraded":"2023-10-30T14:14:05.586525+0000","last_fullsized":"2023-10-30T14:14:05.586525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:16.443567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756348+0000","last_change":"2023-10-30T14:14:04.647842+0000","last_active":"2023-10-30T14:14:05.756348+0000","last_peered":"2023-10-30T14:14:05.756348+0000","last_clean":"2023-10-30T14:14:05.756348+0000","last_became_active":"2023-10-30T14:14:04.644683+0000","last_became_peered":"2023-10-30T14:14:04.644683+0000","last_unstale":"2023-10-30T14:14:05.756348+0000","last_undegraded":"2023-10-30T14:14:05.756348+0000","last_fullsized":"2023-10-30T14:14:05.756348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:51:01.230722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585902+0000","last_change":"2023-10-30T14:14:04.620587+0000","last_active":"2023-10-30T14:14:05.585902+0000","last_peered":"2023-10-30T14:14:05.585902+0000","last_clean":"2023-10-30T14:14:05.585902+0000","last_became_active":"2023-10-30T14:14:04.619335+0000","last_became_peered":"2023-10-30T14:14:04.619335+0000","last_unstale":"2023-10-30T14:14:05.585902+0000","last_undegraded":"2023-10-30T14:14:05.585902+0000","last_fullsized":"2023-10-30T14:14:05.585902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:01.735027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755914+0000","last_change":"2023-10-30T14:14:04.641688+0000","last_active":"2023-10-30T14:14:05.755914+0000","last_peered":"2023-10-30T14:14:05.755914+0000","last_clean":"2023-10-30T14:14:05.755914+0000","last_became_active":"2023-10-30T14:14:04.641323+0000","last_became_peered":"2023-10-30T14:14:04.641323+0000","last_unstale":"2023-10-30T14:14:05.755914+0000","last_undegraded":"2023-10-30T14:14:05.755914+0000","last_fullsized":"2023-10-30T14:14:05.755914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:43:35.285019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805751+0000","last_change":"2023-10-30T14:14:04.618530+0000","last_active":"2023-10-30T14:14:05.805751+0000","last_peered":"2023-10-30T14:14:05.805751+0000","last_clean":"2023-10-30T14:14:05.805751+0000","last_became_active":"2023-10-30T14:14:04.618312+0000","last_became_peered":"2023-10-30T14:14:04.618312+0000","last_unstale":"2023-10-30T14:14:05.805751+0000","last_undegraded":"2023-10-30T14:14:05.805751+0000","last_fullsized":"2023-10-30T14:14:05.805751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:25.089106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757146+0000","last_change":"2023-10-30T14:14:04.650330+0000","last_active":"2023-10-30T14:14:05.757146+0000","last_peered":"2023-10-30T14:14:05.757146+0000","last_clean":"2023-10-30T14:14:05.757146+0000","last_became_active":"2023-10-30T14:14:04.647361+0000","last_became_peered":"2023-10-30T14:14:04.647361+0000","last_unstale":"2023-10-30T14:14:05.757146+0000","last_undegraded":"2023-10-30T14:14:05.757146+0000","last_fullsized":"2023-10-30T14:14:05.757146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:49:57.370980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808090+0000","last_change":"2023-10-30T14:14:04.631540+0000","last_active":"2023-10-30T14:14:05.808090+0000","last_peered":"2023-10-30T14:14:05.808090+0000","last_clean":"2023-10-30T14:14:05.808090+0000","last_became_active":"2023-10-30T14:14:04.630959+0000","last_became_peered":"2023-10-30T14:14:04.630959+0000","last_unstale":"2023-10-30T14:14:05.808090+0000","last_undegraded":"2023-10-30T14:14:05.808090+0000","last_fullsized":"2023-10-30T14:14:05.808090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:49.468429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807762+0000","last_change":"2023-10-30T14:14:04.613800+0000","last_active":"2023-10-30T14:14:05.807762+0000","last_peered":"2023-10-30T14:14:05.807762+0000","last_clean":"2023-10-30T14:14:05.807762+0000","last_became_active":"2023-10-30T14:14:04.612653+0000","last_became_peered":"2023-10-30T14:14:04.612653+0000","last_unstale":"2023-10-30T14:14:05.807762+0000","last_undegraded":"2023-10-30T14:14:05.807762+0000","last_fullsized":"2023-10-30T14:14:05.807762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:44.665588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809700+0000","last_change":"2023-10-30T14:14:04.622483+0000","last_active":"2023-10-30T14:14:05.809700+0000","last_peered":"2023-10-30T14:14:05.809700+0000","last_clean":"2023-10-30T14:14:05.809700+0000","last_became_active":"2023-10-30T14:14:04.621135+0000","last_became_peered":"2023-10-30T14:14:04.621135+0000","last_unstale":"2023-10-30T14:14:05.809700+0000","last_undegraded":"2023-10-30T14:14:05.809700+0000","last_fullsized":"2023-10-30T14:14:05.809700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:52.299412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806054+0000","last_change":"2023-10-30T14:14:04.616357+0000","last_active":"2023-10-30T14:14:05.806054+0000","last_peered":"2023-10-30T14:14:05.806054+0000","last_clean":"2023-10-30T14:14:05.806054+0000","last_became_active":"2023-10-30T14:14:04.616142+0000","last_became_peered":"2023-10-30T14:14:04.616142+0000","last_unstale":"2023-10-30T14:14:05.806054+0000","last_undegraded":"2023-10-30T14:14:05.806054+0000","last_fullsized":"2023-10-30T14:14:05.806054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:32:26.432551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586106+0000","last_change":"2023-10-30T14:14:04.588759+0000","last_active":"2023-10-30T14:14:05.586106+0000","last_peered":"2023-10-30T14:14:05.586106+0000","last_clean":"2023-10-30T14:14:05.586106+0000","last_became_active":"2023-10-30T14:14:04.588452+0000","last_became_peered":"2023-10-30T14:14:04.588452+0000","last_unstale":"2023-10-30T14:14:05.586106+0000","last_undegraded":"2023-10-30T14:14:05.586106+0000","last_fullsized":"2023-10-30T14:14:05.586106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:38:21.206284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584572+0000","last_change":"2023-10-30T14:14:04.621073+0000","last_active":"2023-10-30T14:14:05.584572+0000","last_peered":"2023-10-30T14:14:05.584572+0000","last_clean":"2023-10-30T14:14:05.584572+0000","last_became_active":"2023-10-30T14:14:04.620781+0000","last_became_peered":"2023-10-30T14:14:04.620781+0000","last_unstale":"2023-10-30T14:14:05.584572+0000","last_undegraded":"2023-10-30T14:14:05.584572+0000","last_fullsized":"2023-10-30T14:14:05.584572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:17:10.938195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585712+0000","last_change":"2023-10-30T14:14:04.608942+0000","last_active":"2023-10-30T14:14:05.585712+0000","last_peered":"2023-10-30T14:14:05.585712+0000","last_clean":"2023-10-30T14:14:05.585712+0000","last_became_active":"2023-10-30T14:14:04.608190+0000","last_became_peered":"2023-10-30T14:14:04.608190+0000","last_unstale":"2023-10-30T14:14:05.585712+0000","last_undegraded":"2023-10-30T14:14:05.585712+0000","last_fullsized":"2023-10-30T14:14:05.585712+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:18:39.846841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587056+0000","last_change":"2023-10-30T14:14:04.641876+0000","last_active":"2023-10-30T14:14:05.587056+0000","last_peered":"2023-10-30T14:14:05.587056+0000","last_clean":"2023-10-30T14:14:05.587056+0000","last_became_active":"2023-10-30T14:14:04.641306+0000","last_became_peered":"2023-10-30T14:14:04.641306+0000","last_unstale":"2023-10-30T14:14:05.587056+0000","last_undegraded":"2023-10-30T14:14:05.587056+0000","last_fullsized":"2023-10-30T14:14:05.587056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:10:08.169786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805744+0000","last_change":"2023-10-30T14:14:04.613999+0000","last_active":"2023-10-30T14:14:05.805744+0000","last_peered":"2023-10-30T14:14:05.805744+0000","last_clean":"2023-10-30T14:14:05.805744+0000","last_became_active":"2023-10-30T14:14:04.613644+0000","last_became_peered":"2023-10-30T14:14:04.613644+0000","last_unstale":"2023-10-30T14:14:05.805744+0000","last_undegraded":"2023-10-30T14:14:05.805744+0000","last_fullsized":"2023-10-30T14:14:05.805744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:14:50.232028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756929+0000","last_change":"2023-10-30T14:14:04.650393+0000","last_active":"2023-10-30T14:14:05.756929+0000","last_peered":"2023-10-30T14:14:05.756929+0000","last_clean":"2023-10-30T14:14:05.756929+0000","last_became_active":"2023-10-30T14:14:04.649850+0000","last_became_peered":"2023-10-30T14:14:04.649850+0000","last_unstale":"2023-10-30T14:14:05.756929+0000","last_undegraded":"2023-10-30T14:14:05.756929+0000","last_fullsized":"2023-10-30T14:14:05.756929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:25.174023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808107+0000","last_change":"2023-10-30T14:14:04.625744+0000","last_active":"2023-10-30T14:14:05.808107+0000","last_peered":"2023-10-30T14:14:05.808107+0000","last_clean":"2023-10-30T14:14:05.808107+0000","last_became_active":"2023-10-30T14:14:04.625042+0000","last_became_peered":"2023-10-30T14:14:04.625042+0000","last_unstale":"2023-10-30T14:14:05.808107+0000","last_undegraded":"2023-10-30T14:14:05.808107+0000","last_fullsized":"2023-10-30T14:14:05.808107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:30:36.063455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586236+0000","last_change":"2023-10-30T14:14:04.633942+0000","last_active":"2023-10-30T14:14:05.586236+0000","last_peered":"2023-10-30T14:14:05.586236+0000","last_clean":"2023-10-30T14:14:05.586236+0000","last_became_active":"2023-10-30T14:14:04.633586+0000","last_became_peered":"2023-10-30T14:14:04.633586+0000","last_unstale":"2023-10-30T14:14:05.586236+0000","last_undegraded":"2023-10-30T14:14:05.586236+0000","last_fullsized":"2023-10-30T14:14:05.586236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:42:58.067892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586050+0000","last_change":"2023-10-30T14:14:04.642847+0000","last_active":"2023-10-30T14:14:05.586050+0000","last_peered":"2023-10-30T14:14:05.586050+0000","last_clean":"2023-10-30T14:14:05.586050+0000","last_became_active":"2023-10-30T14:14:04.642244+0000","last_became_peered":"2023-10-30T14:14:04.642244+0000","last_unstale":"2023-10-30T14:14:05.586050+0000","last_undegraded":"2023-10-30T14:14:05.586050+0000","last_fullsized":"2023-10-30T14:14:05.586050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:28.780866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805762+0000","last_change":"2023-10-30T14:14:04.620651+0000","last_active":"2023-10-30T14:14:05.805762+0000","last_peered":"2023-10-30T14:14:05.805762+0000","last_clean":"2023-10-30T14:14:05.805762+0000","last_became_active":"2023-10-30T14:14:04.620343+0000","last_became_peered":"2023-10-30T14:14:04.620343+0000","last_unstale":"2023-10-30T14:14:05.805762+0000","last_undegraded":"2023-10-30T14:14:05.805762+0000","last_fullsized":"2023-10-30T14:14:05.805762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:16:00.557118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587224+0000","last_change":"2023-10-30T14:14:04.610472+0000","last_active":"2023-10-30T14:14:05.587224+0000","last_peered":"2023-10-30T14:14:05.587224+0000","last_clean":"2023-10-30T14:14:05.587224+0000","last_became_active":"2023-10-30T14:14:04.609720+0000","last_became_peered":"2023-10-30T14:14:04.609720+0000","last_unstale":"2023-10-30T14:14:05.587224+0000","last_undegraded":"2023-10-30T14:14:05.587224+0000","last_fullsized":"2023-10-30T14:14:05.587224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:44:28.391589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805906+0000","last_change":"2023-10-30T14:14:04.620548+0000","last_active":"2023-10-30T14:14:05.805906+0000","last_peered":"2023-10-30T14:14:05.805906+0000","last_clean":"2023-10-30T14:14:05.805906+0000","last_became_active":"2023-10-30T14:14:04.620217+0000","last_became_peered":"2023-10-30T14:14:04.620217+0000","last_unstale":"2023-10-30T14:14:05.805906+0000","last_undegraded":"2023-10-30T14:14:05.805906+0000","last_fullsized":"2023-10-30T14:14:05.805906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:40:22.912696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808191+0000","last_change":"2023-10-30T14:14:04.644315+0000","last_active":"2023-10-30T14:14:05.808191+0000","last_peered":"2023-10-30T14:14:05.808191+0000","last_clean":"2023-10-30T14:14:05.808191+0000","last_became_active":"2023-10-30T14:14:04.643707+0000","last_became_peered":"2023-10-30T14:14:04.643707+0000","last_unstale":"2023-10-30T14:14:05.808191+0000","last_undegraded":"2023-10-30T14:14:05.808191+0000","last_fullsized":"2023-10-30T14:14:05.808191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:20:33.544167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584874+0000","last_change":"2023-10-30T14:14:04.597855+0000","last_active":"2023-10-30T14:14:05.584874+0000","last_peered":"2023-10-30T14:14:05.584874+0000","last_clean":"2023-10-30T14:14:05.584874+0000","last_became_active":"2023-10-30T14:14:04.597590+0000","last_became_peered":"2023-10-30T14:14:04.597590+0000","last_unstale":"2023-10-30T14:14:05.584874+0000","last_undegraded":"2023-10-30T14:14:05.584874+0000","last_fullsized":"2023-10-30T14:14:05.584874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:05.905027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806303+0000","last_change":"2023-10-30T14:14:04.624133+0000","last_active":"2023-10-30T14:14:05.806303+0000","last_peered":"2023-10-30T14:14:05.806303+0000","last_clean":"2023-10-30T14:14:05.806303+0000","last_became_active":"2023-10-30T14:14:04.623902+0000","last_became_peered":"2023-10-30T14:14:04.623902+0000","last_unstale":"2023-10-30T14:14:05.806303+0000","last_undegraded":"2023-10-30T14:14:05.806303+0000","last_fullsized":"2023-10-30T14:14:05.806303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:33:10.476461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586328+0000","last_change":"2023-10-30T14:14:04.597875+0000","last_active":"2023-10-30T14:14:05.586328+0000","last_peered":"2023-10-30T14:14:05.586328+0000","last_clean":"2023-10-30T14:14:05.586328+0000","last_became_active":"2023-10-30T14:14:04.597556+0000","last_became_peered":"2023-10-30T14:14:04.597556+0000","last_unstale":"2023-10-30T14:14:05.586328+0000","last_undegraded":"2023-10-30T14:14:05.586328+0000","last_fullsized":"2023-10-30T14:14:05.586328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:08:46.407610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586842+0000","last_change":"2023-10-30T14:14:04.620110+0000","last_active":"2023-10-30T14:14:05.586842+0000","last_peered":"2023-10-30T14:14:05.586842+0000","last_clean":"2023-10-30T14:14:05.586842+0000","last_became_active":"2023-10-30T14:14:04.618430+0000","last_became_peered":"2023-10-30T14:14:04.618430+0000","last_unstale":"2023-10-30T14:14:05.586842+0000","last_undegraded":"2023-10-30T14:14:05.586842+0000","last_fullsized":"2023-10-30T14:14:05.586842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:36.966564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805526+0000","last_change":"2023-10-30T14:14:04.606692+0000","last_active":"2023-10-30T14:14:05.805526+0000","last_peered":"2023-10-30T14:14:05.805526+0000","last_clean":"2023-10-30T14:14:05.805526+0000","last_became_active":"2023-10-30T14:14:04.606340+0000","last_became_peered":"2023-10-30T14:14:04.606340+0000","last_unstale":"2023-10-30T14:14:05.805526+0000","last_undegraded":"2023-10-30T14:14:05.805526+0000","last_fullsized":"2023-10-30T14:14:05.805526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:02:24.518205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807314+0000","last_change":"2023-10-30T14:14:04.613535+0000","last_active":"2023-10-30T14:14:05.807314+0000","last_peered":"2023-10-30T14:14:05.807314+0000","last_clean":"2023-10-30T14:14:05.807314+0000","last_became_active":"2023-10-30T14:14:04.612588+0000","last_became_peered":"2023-10-30T14:14:04.612588+0000","last_unstale":"2023-10-30T14:14:05.807314+0000","last_undegraded":"2023-10-30T14:14:05.807314+0000","last_fullsized":"2023-10-30T14:14:05.807314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:58:23.057988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809168+0000","last_change":"2023-10-30T14:14:04.608434+0000","last_active":"2023-10-30T14:14:05.809168+0000","last_peered":"2023-10-30T14:14:05.809168+0000","last_clean":"2023-10-30T14:14:05.809168+0000","last_became_active":"2023-10-30T14:14:04.607683+0000","last_became_peered":"2023-10-30T14:14:04.607683+0000","last_unstale":"2023-10-30T14:14:05.809168+0000","last_undegraded":"2023-10-30T14:14:05.809168+0000","last_fullsized":"2023-10-30T14:14:05.809168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:17:27.467330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806271+0000","last_change":"2023-10-30T14:14:03.732084+0000","last_active":"2023-10-30T14:14:05.806271+0000","last_peered":"2023-10-30T14:14:05.806271+0000","last_clean":"2023-10-30T14:14:05.806271+0000","last_became_active":"2023-10-30T14:14:03.731740+0000","last_became_peered":"2023-10-30T14:14:03.731740+0000","last_unstale":"2023-10-30T14:14:05.806271+0000","last_undegraded":"2023-10-30T14:14:05.806271+0000","last_fullsized":"2023-10-30T14:14:05.806271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:34:27.462477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586305+0000","last_change":"2023-10-30T14:14:04.621869+0000","last_active":"2023-10-30T14:14:05.586305+0000","last_peered":"2023-10-30T14:14:05.586305+0000","last_clean":"2023-10-30T14:14:05.586305+0000","last_became_active":"2023-10-30T14:14:04.621526+0000","last_became_peered":"2023-10-30T14:14:04.621526+0000","last_unstale":"2023-10-30T14:14:05.586305+0000","last_undegraded":"2023-10-30T14:14:05.586305+0000","last_fullsized":"2023-10-30T14:14:05.586305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:08:21.260692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807788+0000","last_change":"2023-10-30T14:14:03.730789+0000","last_active":"2023-10-30T14:14:05.807788+0000","last_peered":"2023-10-30T14:14:05.807788+0000","last_clean":"2023-10-30T14:14:05.807788+0000","last_became_active":"2023-10-30T14:14:03.729386+0000","last_became_peered":"2023-10-30T14:14:03.729386+0000","last_unstale":"2023-10-30T14:14:05.807788+0000","last_undegraded":"2023-10-30T14:14:05.807788+0000","last_fullsized":"2023-10-30T14:14:05.807788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805855+0000","last_change":"2023-10-30T14:14:03.724379+0000","last_active":"2023-10-30T14:14:05.805855+0000","last_peered":"2023-10-30T14:14:05.805855+0000","last_clean":"2023-10-30T14:14:05.805855+0000","last_became_active":"2023-10-30T14:14:03.724078+0000","last_became_peered":"2023-10-30T14:14:03.724078+0000","last_unstale":"2023-10-30T14:14:05.805855+0000","last_undegraded":"2023-10-30T14:14:05.805855+0000","last_fullsized":"2023-10-30T14:14:05.805855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:10:17.829899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805886+0000","last_change":"2023-10-30T14:14:04.597423+0000","last_active":"2023-10-30T14:14:05.805886+0000","last_peered":"2023-10-30T14:14:05.805886+0000","last_clean":"2023-10-30T14:14:05.805886+0000","last_became_active":"2023-10-30T14:14:04.597099+0000","last_became_peered":"2023-10-30T14:14:04.597099+0000","last_unstale":"2023-10-30T14:14:05.805886+0000","last_undegraded":"2023-10-30T14:14:05.805886+0000","last_fullsized":"2023-10-30T14:14:05.805886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:54:52.346295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585964+0000","last_change":"2023-10-30T14:14:03.721030+0000","last_active":"2023-10-30T14:14:05.585964+0000","last_peered":"2023-10-30T14:14:05.585964+0000","last_clean":"2023-10-30T14:14:05.585964+0000","last_became_active":"2023-10-30T14:14:03.719816+0000","last_became_peered":"2023-10-30T14:14:03.719816+0000","last_unstale":"2023-10-30T14:14:05.585964+0000","last_undegraded":"2023-10-30T14:14:05.585964+0000","last_fullsized":"2023-10-30T14:14:05.585964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:43:50.150156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584729+0000","last_change":"2023-10-30T14:14:04.599863+0000","last_active":"2023-10-30T14:14:05.584729+0000","last_peered":"2023-10-30T14:14:05.584729+0000","last_clean":"2023-10-30T14:14:05.584729+0000","last_became_active":"2023-10-30T14:14:04.599323+0000","last_became_peered":"2023-10-30T14:14:04.599323+0000","last_unstale":"2023-10-30T14:14:05.584729+0000","last_undegraded":"2023-10-30T14:14:05.584729+0000","last_fullsized":"2023-10-30T14:14:05.584729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:42:36.939959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808018+0000","last_change":"2023-10-30T14:14:03.738694+0000","last_active":"2023-10-30T14:14:05.808018+0000","last_peered":"2023-10-30T14:14:05.808018+0000","last_clean":"2023-10-30T14:14:05.808018+0000","last_became_active":"2023-10-30T14:14:03.738132+0000","last_became_peered":"2023-10-30T14:14:03.738132+0000","last_unstale":"2023-10-30T14:14:05.808018+0000","last_undegraded":"2023-10-30T14:14:05.808018+0000","last_fullsized":"2023-10-30T14:14:05.808018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:46:59.692900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586432+0000","last_change":"2023-10-30T14:14:04.649502+0000","last_active":"2023-10-30T14:14:05.586432+0000","last_peered":"2023-10-30T14:14:05.586432+0000","last_clean":"2023-10-30T14:14:05.586432+0000","last_became_active":"2023-10-30T14:14:04.648816+0000","last_became_peered":"2023-10-30T14:14:04.648816+0000","last_unstale":"2023-10-30T14:14:05.586432+0000","last_undegraded":"2023-10-30T14:14:05.586432+0000","last_fullsized":"2023-10-30T14:14:05.586432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:23:07.149253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586445+0000","last_change":"2023-10-30T14:14:03.726112+0000","last_active":"2023-10-30T14:14:05.586445+0000","last_peered":"2023-10-30T14:14:05.586445+0000","last_clean":"2023-10-30T14:14:05.586445+0000","last_became_active":"2023-10-30T14:14:03.725272+0000","last_became_peered":"2023-10-30T14:14:03.725272+0000","last_unstale":"2023-10-30T14:14:05.586445+0000","last_undegraded":"2023-10-30T14:14:05.586445+0000","last_fullsized":"2023-10-30T14:14:05.586445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:37:20.764294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586677+0000","last_change":"2023-10-30T14:14:04.610043+0000","last_active":"2023-10-30T14:14:05.586677+0000","last_peered":"2023-10-30T14:14:05.586677+0000","last_clean":"2023-10-30T14:14:05.586677+0000","last_became_active":"2023-10-30T14:14:04.609697+0000","last_became_peered":"2023-10-30T14:14:04.609697+0000","last_unstale":"2023-10-30T14:14:05.586677+0000","last_undegraded":"2023-10-30T14:14:05.586677+0000","last_fullsized":"2023-10-30T14:14:05.586677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:28.543547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584836+0000","last_change":"2023-10-30T14:14:03.727871+0000","last_active":"2023-10-30T14:14:05.584836+0000","last_peered":"2023-10-30T14:14:05.584836+0000","last_clean":"2023-10-30T14:14:05.584836+0000","last_became_active":"2023-10-30T14:14:03.727422+0000","last_became_peered":"2023-10-30T14:14:03.727422+0000","last_unstale":"2023-10-30T14:14:05.584836+0000","last_undegraded":"2023-10-30T14:14:05.584836+0000","last_fullsized":"2023-10-30T14:14:05.584836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:54.782533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809764+0000","last_change":"2023-10-30T14:14:04.645734+0000","last_active":"2023-10-30T14:14:05.809764+0000","last_peered":"2023-10-30T14:14:05.809764+0000","last_clean":"2023-10-30T14:14:05.809764+0000","last_became_active":"2023-10-30T14:14:04.645333+0000","last_became_peered":"2023-10-30T14:14:04.645333+0000","last_unstale":"2023-10-30T14:14:05.809764+0000","last_undegraded":"2023-10-30T14:14:05.809764+0000","last_fullsized":"2023-10-30T14:14:05.809764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:36:11.322798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808319+0000","last_change":"2023-10-30T14:14:03.725554+0000","last_active":"2023-10-30T14:14:05.808319+0000","last_peered":"2023-10-30T14:14:05.808319+0000","last_clean":"2023-10-30T14:14:05.808319+0000","last_became_active":"2023-10-30T14:14:03.724806+0000","last_became_peered":"2023-10-30T14:14:03.724806+0000","last_unstale":"2023-10-30T14:14:05.808319+0000","last_undegraded":"2023-10-30T14:14:05.808319+0000","last_fullsized":"2023-10-30T14:14:05.808319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:37.823561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756505+0000","last_change":"2023-10-30T14:14:04.647122+0000","last_active":"2023-10-30T14:14:05.756505+0000","last_peered":"2023-10-30T14:14:05.756505+0000","last_clean":"2023-10-30T14:14:05.756505+0000","last_became_active":"2023-10-30T14:14:04.646543+0000","last_became_peered":"2023-10-30T14:14:04.646543+0000","last_unstale":"2023-10-30T14:14:05.756505+0000","last_undegraded":"2023-10-30T14:14:05.756505+0000","last_fullsized":"2023-10-30T14:14:05.756505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:16.158442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586349+0000","last_change":"2023-10-30T14:14:03.717903+0000","last_active":"2023-10-30T14:14:05.586349+0000","last_peered":"2023-10-30T14:14:05.586349+0000","last_clean":"2023-10-30T14:14:05.586349+0000","last_became_active":"2023-10-30T14:14:03.716941+0000","last_became_peered":"2023-10-30T14:14:03.716941+0000","last_unstale":"2023-10-30T14:14:05.586349+0000","last_undegraded":"2023-10-30T14:14:05.586349+0000","last_fullsized":"2023-10-30T14:14:05.586349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:58.178166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586173+0000","last_change":"2023-10-30T14:14:04.641757+0000","last_active":"2023-10-30T14:14:05.586173+0000","last_peered":"2023-10-30T14:14:05.586173+0000","last_clean":"2023-10-30T14:14:05.586173+0000","last_became_active":"2023-10-30T14:14:04.641494+0000","last_became_peered":"2023-10-30T14:14:04.641494+0000","last_unstale":"2023-10-30T14:14:05.586173+0000","last_undegraded":"2023-10-30T14:14:05.586173+0000","last_fullsized":"2023-10-30T14:14:05.586173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:00:18.795083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587083+0000","last_change":"2023-10-30T14:14:03.729626+0000","last_active":"2023-10-30T14:14:05.587083+0000","last_peered":"2023-10-30T14:14:05.587083+0000","last_clean":"2023-10-30T14:14:05.587083+0000","last_became_active":"2023-10-30T14:14:03.728053+0000","last_became_peered":"2023-10-30T14:14:03.728053+0000","last_unstale":"2023-10-30T14:14:05.587083+0000","last_undegraded":"2023-10-30T14:14:05.587083+0000","last_fullsized":"2023-10-30T14:14:05.587083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:44:53.362066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756773+0000","last_change":"2023-10-30T14:14:04.650513+0000","last_active":"2023-10-30T14:14:05.756773+0000","last_peered":"2023-10-30T14:14:05.756773+0000","last_clean":"2023-10-30T14:14:05.756773+0000","last_became_active":"2023-10-30T14:14:04.649896+0000","last_became_peered":"2023-10-30T14:14:04.649896+0000","last_unstale":"2023-10-30T14:14:05.756773+0000","last_undegraded":"2023-10-30T14:14:05.756773+0000","last_fullsized":"2023-10-30T14:14:05.756773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:23.100490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807384+0000","last_change":"2023-10-30T14:14:03.736824+0000","last_active":"2023-10-30T14:14:05.807384+0000","last_peered":"2023-10-30T14:14:05.807384+0000","last_clean":"2023-10-30T14:14:05.807384+0000","last_became_active":"2023-10-30T14:14:03.736155+0000","last_became_peered":"2023-10-30T14:14:03.736155+0000","last_unstale":"2023-10-30T14:14:05.807384+0000","last_undegraded":"2023-10-30T14:14:05.807384+0000","last_fullsized":"2023-10-30T14:14:05.807384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:35.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807372+0000","last_change":"2023-10-30T14:14:04.615159+0000","last_active":"2023-10-30T14:14:05.807372+0000","last_peered":"2023-10-30T14:14:05.807372+0000","last_clean":"2023-10-30T14:14:05.807372+0000","last_became_active":"2023-10-30T14:14:04.614328+0000","last_became_peered":"2023-10-30T14:14:04.614328+0000","last_unstale":"2023-10-30T14:14:05.807372+0000","last_undegraded":"2023-10-30T14:14:05.807372+0000","last_fullsized":"2023-10-30T14:14:05.807372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:16:52.999786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805450+0000","last_change":"2023-10-30T14:14:03.700690+0000","last_active":"2023-10-30T14:14:05.805450+0000","last_peered":"2023-10-30T14:14:05.805450+0000","last_clean":"2023-10-30T14:14:05.805450+0000","last_became_active":"2023-10-30T14:14:03.700352+0000","last_became_peered":"2023-10-30T14:14:03.700352+0000","last_unstale":"2023-10-30T14:14:05.805450+0000","last_undegraded":"2023-10-30T14:14:05.805450+0000","last_fullsized":"2023-10-30T14:14:05.805450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:03:01.103359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755948+0000","last_change":"2023-10-30T14:14:04.647253+0000","last_active":"2023-10-30T14:14:05.755948+0000","last_peered":"2023-10-30T14:14:05.755948+0000","last_clean":"2023-10-30T14:14:05.755948+0000","last_became_active":"2023-10-30T14:14:04.646639+0000","last_became_peered":"2023-10-30T14:14:04.646639+0000","last_unstale":"2023-10-30T14:14:05.755948+0000","last_undegraded":"2023-10-30T14:14:05.755948+0000","last_fullsized":"2023-10-30T14:14:05.755948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:22:50.893734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.810120+0000","last_change":"2023-10-30T14:14:03.710717+0000","last_active":"2023-10-30T14:14:05.810120+0000","last_peered":"2023-10-30T14:14:05.810120+0000","last_clean":"2023-10-30T14:14:05.810120+0000","last_became_active":"2023-10-30T14:14:03.707950+0000","last_became_peered":"2023-10-30T14:14:03.707950+0000","last_unstale":"2023-10-30T14:14:05.810120+0000","last_undegraded":"2023-10-30T14:14:05.810120+0000","last_fullsized":"2023-10-30T14:14:05.810120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:29:41.232902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757066+0000","last_change":"2023-10-30T14:14:04.647688+0000","last_active":"2023-10-30T14:14:05.757066+0000","last_peered":"2023-10-30T14:14:05.757066+0000","last_clean":"2023-10-30T14:14:05.757066+0000","last_became_active":"2023-10-30T14:14:04.644761+0000","last_became_peered":"2023-10-30T14:14:04.644761+0000","last_unstale":"2023-10-30T14:14:05.757066+0000","last_undegraded":"2023-10-30T14:14:05.757066+0000","last_fullsized":"2023-10-30T14:14:05.757066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:00:38.482522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805776+0000","last_change":"2023-10-30T14:14:03.705567+0000","last_active":"2023-10-30T14:14:05.805776+0000","last_peered":"2023-10-30T14:14:05.805776+0000","last_clean":"2023-10-30T14:14:05.805776+0000","last_became_active":"2023-10-30T14:14:03.705186+0000","last_became_peered":"2023-10-30T14:14:03.705186+0000","last_unstale":"2023-10-30T14:14:05.805776+0000","last_undegraded":"2023-10-30T14:14:05.805776+0000","last_fullsized":"2023-10-30T14:14:05.805776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:14.609243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757235+0000","last_change":"2023-10-30T14:14:04.647335+0000","last_active":"2023-10-30T14:14:05.757235+0000","last_peered":"2023-10-30T14:14:05.757235+0000","last_clean":"2023-10-30T14:14:05.757235+0000","last_became_active":"2023-10-30T14:14:04.646785+0000","last_became_peered":"2023-10-30T14:14:04.646785+0000","last_unstale":"2023-10-30T14:14:05.757235+0000","last_undegraded":"2023-10-30T14:14:05.757235+0000","last_fullsized":"2023-10-30T14:14:05.757235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:58:41.600761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584948+0000","last_change":"2023-10-30T14:14:03.693349+0000","last_active":"2023-10-30T14:14:05.584948+0000","last_peered":"2023-10-30T14:14:05.584948+0000","last_clean":"2023-10-30T14:14:05.584948+0000","last_became_active":"2023-10-30T14:14:03.692966+0000","last_became_peered":"2023-10-30T14:14:03.692966+0000","last_unstale":"2023-10-30T14:14:05.584948+0000","last_undegraded":"2023-10-30T14:14:05.584948+0000","last_fullsized":"2023-10-30T14:14:05.584948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:30:28.800214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587213+0000","last_change":"2023-10-30T14:14:04.617023+0000","last_active":"2023-10-30T14:14:05.587213+0000","last_peered":"2023-10-30T14:14:05.587213+0000","last_clean":"2023-10-30T14:14:05.587213+0000","last_became_active":"2023-10-30T14:14:04.615925+0000","last_became_peered":"2023-10-30T14:14:04.615925+0000","last_unstale":"2023-10-30T14:14:05.587213+0000","last_undegraded":"2023-10-30T14:14:05.587213+0000","last_fullsized":"2023-10-30T14:14:05.587213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:32:02.289324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805586+0000","last_change":"2023-10-30T14:14:03.727659+0000","last_active":"2023-10-30T14:14:05.805586+0000","last_peered":"2023-10-30T14:14:05.805586+0000","last_clean":"2023-10-30T14:14:05.805586+0000","last_became_active":"2023-10-30T14:14:03.727409+0000","last_became_peered":"2023-10-30T14:14:03.727409+0000","last_unstale":"2023-10-30T14:14:05.805586+0000","last_undegraded":"2023-10-30T14:14:05.805586+0000","last_fullsized":"2023-10-30T14:14:05.805586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:24.210635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756468+0000","last_change":"2023-10-30T14:14:04.650083+0000","last_active":"2023-10-30T14:14:05.756468+0000","last_peered":"2023-10-30T14:14:05.756468+0000","last_clean":"2023-10-30T14:14:05.756468+0000","last_became_active":"2023-10-30T14:14:04.647562+0000","last_became_peered":"2023-10-30T14:14:04.647562+0000","last_unstale":"2023-10-30T14:14:05.756468+0000","last_undegraded":"2023-10-30T14:14:05.756468+0000","last_fullsized":"2023-10-30T14:14:05.756468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:11:41.743850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T14:14:04.632968+0000","last_change":"2023-10-30T14:13:54.251572+0000","last_active":"2023-10-30T14:14:04.632968+0000","last_peered":"2023-10-30T14:14:04.632968+0000","last_clean":"2023-10-30T14:14:04.632968+0000","last_became_active":"2023-10-30T14:13:54.250907+0000","last_became_peered":"2023-10-30T14:13:54.250907+0000","last_unstale":"2023-10-30T14:14:04.632968+0000","last_undegraded":"2023-10-30T14:14:04.632968+0000","last_fullsized":"2023-10-30T14:14:04.632968+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_clean_scrub_stamp":"2023-10-30T14:13:53.220267+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:44.590852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585892+0000","last_change":"2023-10-30T14:14:03.729034+0000","last_active":"2023-10-30T14:14:05.585892+0000","last_peered":"2023-10-30T14:14:05.585892+0000","last_clean":"2023-10-30T14:14:05.585892+0000","last_became_active":"2023-10-30T14:14:03.728293+0000","last_became_peered":"2023-10-30T14:14:03.728293+0000","last_unstale":"2023-10-30T14:14:05.585892+0000","last_undegraded":"2023-10-30T14:14:05.585892+0000","last_fullsized":"2023-10-30T14:14:05.585892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806225+0000","last_change":"2023-10-30T14:14:04.630325+0000","last_active":"2023-10-30T14:14:05.806225+0000","last_peered":"2023-10-30T14:14:05.806225+0000","last_clean":"2023-10-30T14:14:05.806225+0000","last_became_active":"2023-10-30T14:14:04.630121+0000","last_became_peered":"2023-10-30T14:14:04.630121+0000","last_unstale":"2023-10-30T14:14:05.806225+0000","last_undegraded":"2023-10-30T14:14:05.806225+0000","last_fullsized":"2023-10-30T14:14:05.806225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:02:28.792879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755708+0000","last_change":"2023-10-30T14:14:03.732909+0000","last_active":"2023-10-30T14:14:05.755708+0000","last_peered":"2023-10-30T14:14:05.755708+0000","last_clean":"2023-10-30T14:14:05.755708+0000","last_became_active":"2023-10-30T14:14:03.732199+0000","last_became_peered":"2023-10-30T14:14:03.732199+0000","last_unstale":"2023-10-30T14:14:05.755708+0000","last_undegraded":"2023-10-30T14:14:05.755708+0000","last_fullsized":"2023-10-30T14:14:05.755708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:38.117750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586172+0000","last_change":"2023-10-30T14:14:04.620314+0000","last_active":"2023-10-30T14:14:05.586172+0000","last_peered":"2023-10-30T14:14:05.586172+0000","last_clean":"2023-10-30T14:14:05.586172+0000","last_became_active":"2023-10-30T14:14:04.619990+0000","last_became_peered":"2023-10-30T14:14:04.619990+0000","last_unstale":"2023-10-30T14:14:05.586172+0000","last_undegraded":"2023-10-30T14:14:05.586172+0000","last_fullsized":"2023-10-30T14:14:05.586172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:38:25.657586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805642+0000","last_change":"2023-10-30T14:14:03.702862+0000","last_active":"2023-10-30T14:14:05.805642+0000","last_peered":"2023-10-30T14:14:05.805642+0000","last_clean":"2023-10-30T14:14:05.805642+0000","last_became_active":"2023-10-30T14:14:03.702513+0000","last_became_peered":"2023-10-30T14:14:03.702513+0000","last_unstale":"2023-10-30T14:14:05.805642+0000","last_undegraded":"2023-10-30T14:14:05.805642+0000","last_fullsized":"2023-10-30T14:14:05.805642+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:25.499390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584623+0000","last_change":"2023-10-30T14:14:04.589353+0000","last_active":"2023-10-30T14:14:05.584623+0000","last_peered":"2023-10-30T14:14:05.584623+0000","last_clean":"2023-10-30T14:14:05.584623+0000","last_became_active":"2023-10-30T14:14:04.589087+0000","last_became_peered":"2023-10-30T14:14:04.589087+0000","last_unstale":"2023-10-30T14:14:05.584623+0000","last_undegraded":"2023-10-30T14:14:05.584623+0000","last_fullsized":"2023-10-30T14:14:05.584623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:20:12.808641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587305+0000","last_change":"2023-10-30T14:14:03.730048+0000","last_active":"2023-10-30T14:14:05.587305+0000","last_peered":"2023-10-30T14:14:05.587305+0000","last_clean":"2023-10-30T14:14:05.587305+0000","last_became_active":"2023-10-30T14:14:03.728925+0000","last_became_peered":"2023-10-30T14:14:03.728925+0000","last_unstale":"2023-10-30T14:14:05.587305+0000","last_undegraded":"2023-10-30T14:14:05.587305+0000","last_fullsized":"2023-10-30T14:14:05.587305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:32:05.056039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809437+0000","last_change":"2023-10-30T14:14:04.608289+0000","last_active":"2023-10-30T14:14:05.809437+0000","last_peered":"2023-10-30T14:14:05.809437+0000","last_clean":"2023-10-30T14:14:05.809437+0000","last_became_active":"2023-10-30T14:14:04.607377+0000","last_became_peered":"2023-10-30T14:14:04.607377+0000","last_unstale":"2023-10-30T14:14:05.809437+0000","last_undegraded":"2023-10-30T14:14:05.809437+0000","last_fullsized":"2023-10-30T14:14:05.809437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:43:40.869579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584795+0000","last_change":"2023-10-30T14:14:03.693753+0000","last_active":"2023-10-30T14:14:05.584795+0000","last_peered":"2023-10-30T14:14:05.584795+0000","last_clean":"2023-10-30T14:14:05.584795+0000","last_became_active":"2023-10-30T14:14:03.692621+0000","last_became_peered":"2023-10-30T14:14:03.692621+0000","last_unstale":"2023-10-30T14:14:05.584795+0000","last_undegraded":"2023-10-30T14:14:05.584795+0000","last_fullsized":"2023-10-30T14:14:05.584795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:32:28.577584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587225+0000","last_change":"2023-10-30T14:14:04.614649+0000","last_active":"2023-10-30T14:14:05.587225+0000","last_peered":"2023-10-30T14:14:05.587225+0000","last_clean":"2023-10-30T14:14:05.587225+0000","last_became_active":"2023-10-30T14:14:04.613780+0000","last_became_peered":"2023-10-30T14:14:04.613780+0000","last_unstale":"2023-10-30T14:14:05.587225+0000","last_undegraded":"2023-10-30T14:14:05.587225+0000","last_fullsized":"2023-10-30T14:14:05.587225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:47:40.354022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757037+0000","last_change":"2023-10-30T14:14:03.740213+0000","last_active":"2023-10-30T14:14:05.757037+0000","last_peered":"2023-10-30T14:14:05.757037+0000","last_clean":"2023-10-30T14:14:05.757037+0000","last_became_active":"2023-10-30T14:14:03.739650+0000","last_became_peered":"2023-10-30T14:14:03.739650+0000","last_unstale":"2023-10-30T14:14:05.757037+0000","last_undegraded":"2023-10-30T14:14:05.757037+0000","last_fullsized":"2023-10-30T14:14:05.757037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:54.120757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584691+0000","last_change":"2023-10-30T14:14:04.590569+0000","last_active":"2023-10-30T14:14:05.584691+0000","last_peered":"2023-10-30T14:14:05.584691+0000","last_clean":"2023-10-30T14:14:05.584691+0000","last_became_active":"2023-10-30T14:14:04.590138+0000","last_became_peered":"2023-10-30T14:14:04.590138+0000","last_unstale":"2023-10-30T14:14:05.584691+0000","last_undegraded":"2023-10-30T14:14:05.584691+0000","last_fullsized":"2023-10-30T14:14:05.584691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:41:30.333668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585050+0000","last_change":"2023-10-30T14:14:03.721450+0000","last_active":"2023-10-30T14:14:05.585050+0000","last_peered":"2023-10-30T14:14:05.585050+0000","last_clean":"2023-10-30T14:14:05.585050+0000","last_became_active":"2023-10-30T14:14:03.721234+0000","last_became_peered":"2023-10-30T14:14:03.721234+0000","last_unstale":"2023-10-30T14:14:05.585050+0000","last_undegraded":"2023-10-30T14:14:05.585050+0000","last_fullsized":"2023-10-30T14:14:05.585050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:26.959676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806005+0000","last_change":"2023-10-30T14:14:04.634004+0000","last_active":"2023-10-30T14:14:05.806005+0000","last_peered":"2023-10-30T14:14:05.806005+0000","last_clean":"2023-10-30T14:14:05.806005+0000","last_became_active":"2023-10-30T14:14:04.633643+0000","last_became_peered":"2023-10-30T14:14:04.633643+0000","last_unstale":"2023-10-30T14:14:05.806005+0000","last_undegraded":"2023-10-30T14:14:05.806005+0000","last_fullsized":"2023-10-30T14:14:05.806005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:05:14.572477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808140+0000","last_change":"2023-10-30T14:14:03.738557+0000","last_active":"2023-10-30T14:14:05.808140+0000","last_peered":"2023-10-30T14:14:05.808140+0000","last_clean":"2023-10-30T14:14:05.808140+0000","last_became_active":"2023-10-30T14:14:03.737801+0000","last_became_peered":"2023-10-30T14:14:03.737801+0000","last_unstale":"2023-10-30T14:14:05.808140+0000","last_undegraded":"2023-10-30T14:14:05.808140+0000","last_fullsized":"2023-10-30T14:14:05.808140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:26.938343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586581+0000","last_change":"2023-10-30T14:14:04.589273+0000","last_active":"2023-10-30T14:14:05.586581+0000","last_peered":"2023-10-30T14:14:05.586581+0000","last_clean":"2023-10-30T14:14:05.586581+0000","last_became_active":"2023-10-30T14:14:04.588826+0000","last_became_peered":"2023-10-30T14:14:04.588826+0000","last_unstale":"2023-10-30T14:14:05.586581+0000","last_undegraded":"2023-10-30T14:14:05.586581+0000","last_fullsized":"2023-10-30T14:14:05.586581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:33:52.452394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755689+0000","last_change":"2023-10-30T14:14:03.733836+0000","last_active":"2023-10-30T14:14:05.755689+0000","last_peered":"2023-10-30T14:14:05.755689+0000","last_clean":"2023-10-30T14:14:05.755689+0000","last_became_active":"2023-10-30T14:14:03.732912+0000","last_became_peered":"2023-10-30T14:14:03.732912+0000","last_unstale":"2023-10-30T14:14:05.755689+0000","last_undegraded":"2023-10-30T14:14:05.755689+0000","last_fullsized":"2023-10-30T14:14:05.755689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:50.332738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809314+0000","last_change":"2023-10-30T14:14:04.610696+0000","last_active":"2023-10-30T14:14:05.809314+0000","last_peered":"2023-10-30T14:14:05.809314+0000","last_clean":"2023-10-30T14:14:05.809314+0000","last_became_active":"2023-10-30T14:14:04.609835+0000","last_became_peered":"2023-10-30T14:14:04.609835+0000","last_unstale":"2023-10-30T14:14:05.809314+0000","last_undegraded":"2023-10-30T14:14:05.809314+0000","last_fullsized":"2023-10-30T14:14:05.809314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:04:31.270262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584669+0000","last_change":"2023-10-30T14:14:03.732539+0000","last_active":"2023-10-30T14:14:05.584669+0000","last_peered":"2023-10-30T14:14:05.584669+0000","last_clean":"2023-10-30T14:14:05.584669+0000","last_became_active":"2023-10-30T14:14:03.732265+0000","last_became_peered":"2023-10-30T14:14:03.732265+0000","last_unstale":"2023-10-30T14:14:05.584669+0000","last_undegraded":"2023-10-30T14:14:05.584669+0000","last_fullsized":"2023-10-30T14:14:05.584669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:02:00.929612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806211+0000","last_change":"2023-10-30T14:14:04.595357+0000","last_active":"2023-10-30T14:14:05.806211+0000","last_peered":"2023-10-30T14:14:05.806211+0000","last_clean":"2023-10-30T14:14:05.806211+0000","last_became_active":"2023-10-30T14:14:04.594998+0000","last_became_peered":"2023-10-30T14:14:04.594998+0000","last_unstale":"2023-10-30T14:14:05.806211+0000","last_undegraded":"2023-10-30T14:14:05.806211+0000","last_fullsized":"2023-10-30T14:14:05.806211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:44.365673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584615+0000","last_change":"2023-10-30T14:14:03.703744+0000","last_active":"2023-10-30T14:14:05.584615+0000","last_peered":"2023-10-30T14:14:05.584615+0000","last_clean":"2023-10-30T14:14:05.584615+0000","last_became_active":"2023-10-30T14:14:03.701759+0000","last_became_peered":"2023-10-30T14:14:03.701759+0000","last_unstale":"2023-10-30T14:14:05.584615+0000","last_undegraded":"2023-10-30T14:14:05.584615+0000","last_fullsized":"2023-10-30T14:14:05.584615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:22:27.948007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806186+0000","last_change":"2023-10-30T14:14:04.600759+0000","last_active":"2023-10-30T14:14:05.806186+0000","last_peered":"2023-10-30T14:14:05.806186+0000","last_clean":"2023-10-30T14:14:05.806186+0000","last_became_active":"2023-10-30T14:14:04.600100+0000","last_became_peered":"2023-10-30T14:14:04.600100+0000","last_unstale":"2023-10-30T14:14:05.806186+0000","last_undegraded":"2023-10-30T14:14:05.806186+0000","last_fullsized":"2023-10-30T14:14:05.806186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:02.831207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756778+0000","last_change":"2023-10-30T14:14:03.741268+0000","last_active":"2023-10-30T14:14:05.756778+0000","last_peered":"2023-10-30T14:14:05.756778+0000","last_clean":"2023-10-30T14:14:05.756778+0000","last_became_active":"2023-10-30T14:14:03.740648+0000","last_became_peered":"2023-10-30T14:14:03.740648+0000","last_unstale":"2023-10-30T14:14:05.756778+0000","last_undegraded":"2023-10-30T14:14:05.756778+0000","last_fullsized":"2023-10-30T14:14:05.756778+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:30.799148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807541+0000","last_change":"2023-10-30T14:14:04.623644+0000","last_active":"2023-10-30T14:14:05.807541+0000","last_peered":"2023-10-30T14:14:05.807541+0000","last_clean":"2023-10-30T14:14:05.807541+0000","last_became_active":"2023-10-30T14:14:04.622921+0000","last_became_peered":"2023-10-30T14:14:04.622921+0000","last_unstale":"2023-10-30T14:14:05.807541+0000","last_undegraded":"2023-10-30T14:14:05.807541+0000","last_fullsized":"2023-10-30T14:14:05.807541+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:11:41.810610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587103+0000","last_change":"2023-10-30T14:14:04.624799+0000","last_active":"2023-10-30T14:14:05.587103+0000","last_peered":"2023-10-30T14:14:05.587103+0000","last_clean":"2023-10-30T14:14:05.587103+0000","last_became_active":"2023-10-30T14:14:04.621885+0000","last_became_peered":"2023-10-30T14:14:04.621885+0000","last_unstale":"2023-10-30T14:14:05.587103+0000","last_undegraded":"2023-10-30T14:14:05.587103+0000","last_fullsized":"2023-10-30T14:14:05.587103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:12.055247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756286+0000","last_change":"2023-10-30T14:14:03.740897+0000","last_active":"2023-10-30T14:14:05.756286+0000","last_peered":"2023-10-30T14:14:05.756286+0000","last_clean":"2023-10-30T14:14:05.756286+0000","last_became_active":"2023-10-30T14:14:03.740247+0000","last_became_peered":"2023-10-30T14:14:03.740247+0000","last_unstale":"2023-10-30T14:14:05.756286+0000","last_undegraded":"2023-10-30T14:14:05.756286+0000","last_fullsized":"2023-10-30T14:14:05.756286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:29.601978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586861+0000","last_change":"2023-10-30T14:14:04.625498+0000","last_active":"2023-10-30T14:14:05.586861+0000","last_peered":"2023-10-30T14:14:05.586861+0000","last_clean":"2023-10-30T14:14:05.586861+0000","last_became_active":"2023-10-30T14:14:04.623638+0000","last_became_peered":"2023-10-30T14:14:04.623638+0000","last_unstale":"2023-10-30T14:14:05.586861+0000","last_undegraded":"2023-10-30T14:14:05.586861+0000","last_fullsized":"2023-10-30T14:14:05.586861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:19.717965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756898+0000","last_change":"2023-10-30T14:14:03.733233+0000","last_active":"2023-10-30T14:14:05.756898+0000","last_peered":"2023-10-30T14:14:05.756898+0000","last_clean":"2023-10-30T14:14:05.756898+0000","last_became_active":"2023-10-30T14:14:03.732538+0000","last_became_peered":"2023-10-30T14:14:03.732538+0000","last_unstale":"2023-10-30T14:14:05.756898+0000","last_undegraded":"2023-10-30T14:14:05.756898+0000","last_fullsized":"2023-10-30T14:14:05.756898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:25.696173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586619+0000","last_change":"2023-10-30T14:14:04.612104+0000","last_active":"2023-10-30T14:14:05.586619+0000","last_peered":"2023-10-30T14:14:05.586619+0000","last_clean":"2023-10-30T14:14:05.586619+0000","last_became_active":"2023-10-30T14:14:04.611773+0000","last_became_peered":"2023-10-30T14:14:04.611773+0000","last_unstale":"2023-10-30T14:14:05.586619+0000","last_undegraded":"2023-10-30T14:14:05.586619+0000","last_fullsized":"2023-10-30T14:14:05.586619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:44:21.952103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806100+0000","last_change":"2023-10-30T14:14:03.703988+0000","last_active":"2023-10-30T14:14:05.806100+0000","last_peered":"2023-10-30T14:14:05.806100+0000","last_clean":"2023-10-30T14:14:05.806100+0000","last_became_active":"2023-10-30T14:14:03.703691+0000","last_became_peered":"2023-10-30T14:14:03.703691+0000","last_unstale":"2023-10-30T14:14:05.806100+0000","last_undegraded":"2023-10-30T14:14:05.806100+0000","last_fullsized":"2023-10-30T14:14:05.806100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:34.538541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806119+0000","last_change":"2023-10-30T14:14:04.613738+0000","last_active":"2023-10-30T14:14:05.806119+0000","last_peered":"2023-10-30T14:14:05.806119+0000","last_clean":"2023-10-30T14:14:05.806119+0000","last_became_active":"2023-10-30T14:14:04.613420+0000","last_became_peered":"2023-10-30T14:14:04.613420+0000","last_unstale":"2023-10-30T14:14:05.806119+0000","last_undegraded":"2023-10-30T14:14:05.806119+0000","last_fullsized":"2023-10-30T14:14:05.806119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:01:52.695195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586668+0000","last_change":"2023-10-30T14:14:03.737319+0000","last_active":"2023-10-30T14:14:05.586668+0000","last_peered":"2023-10-30T14:14:05.586668+0000","last_clean":"2023-10-30T14:14:05.586668+0000","last_became_active":"2023-10-30T14:14:03.736964+0000","last_became_peered":"2023-10-30T14:14:03.736964+0000","last_unstale":"2023-10-30T14:14:05.586668+0000","last_undegraded":"2023-10-30T14:14:05.586668+0000","last_fullsized":"2023-10-30T14:14:05.586668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:41.469809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":665937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4039999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.458}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1639999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7909999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2829999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.222}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.276}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":75569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1519999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":811008,"data_stored":665937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:14:11.260 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-30T14:14:11.260 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-30T14:14:11.260 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-30T14:14:11.261 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-30T14:14:11.261 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-30T14:14:11.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.432+0000 7fe175fd3640 1 Processor -- start 2023-10-30T14:14:11.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.432+0000 7fe175fd3640 1 -- start start 2023-10-30T14:14:11.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.432+0000 7fe175fd3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe170153c80 0x7fe17015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:11.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.432+0000 7fe175fd3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe1700c4a10 0x7fe1700c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:11.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.432+0000 7fe175fd3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe17015a3b0 0x7fe1700ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:11.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.432+0000 7fe175fd3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe17015dec0 con 0x7fe1700c4a10 2023-10-30T14:14:11.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.432+0000 7fe175fd3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1701599a0 con 0x7fe17015a3b0 2023-10-30T14:14:11.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.432+0000 7fe175fd3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe170159f20 con 0x7fe170153c80 2023-10-30T14:14:11.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.432+0000 7fe16effd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe1700c4a10 0x7fe1700c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:11.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.432+0000 7fe16effd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe1700c4a10 0x7fe1700c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37888/0 (socket says 172.21.15.57:37888) 2023-10-30T14:14:11.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.432+0000 7fe16effd640 1 -- 172.21.15.57:0/2380930353 learned_addr learned my addr 172.21.15.57:0/2380930353 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:11.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.432+0000 7fe16ffff640 1 --2- 172.21.15.57:0/2380930353 >> v2:172.21.15.88:3300/0 conn(0x7fe17015a3b0 0x7fe1700ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:11.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.432+0000 7fe16f7fe640 1 --2- 172.21.15.57:0/2380930353 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe170153c80 0x7fe17015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:11.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.432+0000 7fe16effd640 1 -- 172.21.15.57:0/2380930353 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe170153c80 msgr2=0x7fe17015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:11.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.432+0000 7fe16effd640 1 --2- 172.21.15.57:0/2380930353 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe170153c80 0x7fe17015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.432+0000 7fe16effd640 1 -- 172.21.15.57:0/2380930353 >> v2:172.21.15.88:3300/0 conn(0x7fe17015a3b0 msgr2=0x7fe1700ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:11.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.432+0000 7fe16effd640 1 --2- 172.21.15.57:0/2380930353 >> v2:172.21.15.88:3300/0 conn(0x7fe17015a3b0 0x7fe1700ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.432+0000 7fe16effd640 1 -- 172.21.15.57:0/2380930353 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1700c47e0 con 0x7fe1700c4a10 2023-10-30T14:14:11.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.436+0000 7fe16effd640 1 --2- 172.21.15.57:0/2380930353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1700c4a10 0x7fe1700c42a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe1700cbbc0 tx=0x7fe16401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4054d3c280e3064f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:11.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.436+0000 7fe16e7fc640 1 -- 172.21.15.57:0/2380930353 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1640a89b0 con 0x7fe1700c4a10 2023-10-30T14:14:11.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.436+0000 7fe16e7fc640 1 -- 172.21.15.57:0/2380930353 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1640a8b10 con 0x7fe1700c4a10 2023-10-30T14:14:11.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.436+0000 7fe16e7fc640 1 -- 172.21.15.57:0/2380930353 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1640a8ea0 con 0x7fe1700c4a10 2023-10-30T14:14:11.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.436+0000 7fe175fd3640 1 -- 172.21.15.57:0/2380930353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1700c4a10 msgr2=0x7fe1700c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:11.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.436+0000 7fe175fd3640 1 --2- 172.21.15.57:0/2380930353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1700c4a10 0x7fe1700c42a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe1700cbbc0 tx=0x7fe16401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.436+0000 7fe175fd3640 1 -- 172.21.15.57:0/2380930353 shutdown_connections 2023-10-30T14:14:11.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.436+0000 7fe175fd3640 1 --2- 172.21.15.57:0/2380930353 >> v2:172.21.15.88:3300/0 conn(0x7fe17015a3b0 0x7fe1700ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.436+0000 7fe175fd3640 1 --2- 172.21.15.57:0/2380930353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1700c4a10 0x7fe1700c42a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.436+0000 7fe175fd3640 1 --2- 172.21.15.57:0/2380930353 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe170153c80 0x7fe17015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.436+0000 7fe175fd3640 1 -- 172.21.15.57:0/2380930353 >> 172.21.15.57:0/2380930353 conn(0x7fe1700715c0 msgr2=0x7fe1700719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:11.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.436+0000 7fe175fd3640 1 -- 172.21.15.57:0/2380930353 shutdown_connections 2023-10-30T14:14:11.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.436+0000 7fe175fd3640 1 -- 172.21.15.57:0/2380930353 wait complete. 2023-10-30T14:14:11.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe175fd3640 1 Processor -- start 2023-10-30T14:14:11.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe175fd3640 1 -- start start 2023-10-30T14:14:11.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe175fd3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1700c4a10 0x7fe170151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:11.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe175fd3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe170153c80 0x7fe1700b3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:11.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe175fd3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe17015a3b0 0x7fe170149b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:11.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe175fd3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1700b2b70 con 0x7fe170153c80 2023-10-30T14:14:11.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe175fd3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1701eea50 con 0x7fe1700c4a10 2023-10-30T14:14:11.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe175fd3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1700b2e30 con 0x7fe17015a3b0 2023-10-30T14:14:11.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe16effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe170153c80 0x7fe1700b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:11.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe16effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe170153c80 0x7fe1700b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37890/0 (socket says 172.21.15.57:37890) 2023-10-30T14:14:11.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe16effd640 1 -- 172.21.15.57:0/2080626619 learned_addr learned my addr 172.21.15.57:0/2080626619 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:11.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe16f7fe640 1 --2- 172.21.15.57:0/2080626619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1700c4a10 0x7fe170151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:11.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe16ffff640 1 --2- 172.21.15.57:0/2080626619 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe17015a3b0 0x7fe170149b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:11.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe16effd640 1 -- 172.21.15.57:0/2080626619 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe17015a3b0 msgr2=0x7fe170149b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:11.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe16effd640 1 --2- 172.21.15.57:0/2080626619 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe17015a3b0 0x7fe170149b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe16effd640 1 -- 172.21.15.57:0/2080626619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1700c4a10 msgr2=0x7fe170151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:11.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe16effd640 1 --2- 172.21.15.57:0/2080626619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1700c4a10 0x7fe170151fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe16effd640 1 -- 172.21.15.57:0/2080626619 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1640094d0 con 0x7fe170153c80 2023-10-30T14:14:11.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe16effd640 1 --2- 172.21.15.57:0/2080626619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe170153c80 0x7fe1700b3470 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe16401d3a0 tx=0x7fe16401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:11.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe157fff640 1 -- 172.21.15.57:0/2080626619 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1640b0d40 con 0x7fe170153c80 2023-10-30T14:14:11.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe157fff640 1 -- 172.21.15.57:0/2080626619 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1640bb460 con 0x7fe170153c80 2023-10-30T14:14:11.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe157fff640 1 -- 172.21.15.57:0/2080626619 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1640b0ea0 con 0x7fe170153c80 2023-10-30T14:14:11.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe175fd3640 1 -- 172.21.15.57:0/2080626619 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1701eed10 con 0x7fe170153c80 2023-10-30T14:14:11.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.440+0000 7fe175fd3640 1 -- 172.21.15.57:0/2080626619 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1701ef2d0 con 0x7fe170153c80 2023-10-30T14:14:11.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.444+0000 7fe175fd3640 1 -- 172.21.15.57:0/2080626619 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe17004e170 con 0x7fe170153c80 2023-10-30T14:14:11.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.444+0000 7fe157fff640 1 -- 172.21.15.57:0/2080626619 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe1640b83f0 con 0x7fe170153c80 2023-10-30T14:14:11.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.444+0000 7fe157fff640 1 --2- 172.21.15.57:0/2080626619 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe15c0664d0 0x7fe15c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:11.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.444+0000 7fe157fff640 1 -- 172.21.15.57:0/2080626619 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe164121810 con 0x7fe170153c80 2023-10-30T14:14:11.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.444+0000 7fe16f7fe640 1 --2- 172.21.15.57:0/2080626619 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe15c0664d0 0x7fe15c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:11.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.444+0000 7fe16f7fe640 1 --2- 172.21.15.57:0/2080626619 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe15c0664d0 0x7fe15c068990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe160009a70 tx=0x7fe160008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:11.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.448+0000 7fe157fff640 1 -- 172.21.15.57:0/2080626619 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe1640eeb60 con 0x7fe170153c80 2023-10-30T14:14:11.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.640+0000 7fe175fd3640 1 -- 172.21.15.57:0/2080626619 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fe1701ef4c0 con 0x7fe170153c80 2023-10-30T14:14:11.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.644+0000 7fe157fff640 1 -- 172.21.15.57:0/2080626619 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+164430 (secure 0 0 0) 0x7fe1640b5090 con 0x7fe170153c80 2023-10-30T14:14:11.651 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:11.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.648+0000 7fe175fd3640 1 -- 172.21.15.57:0/2080626619 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe15c0664d0 msgr2=0x7fe15c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:11.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.648+0000 7fe175fd3640 1 --2- 172.21.15.57:0/2080626619 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe15c0664d0 0x7fe15c068990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe160009a70 tx=0x7fe160008040 comp rx=0 tx=0).stop 2023-10-30T14:14:11.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.648+0000 7fe175fd3640 1 -- 172.21.15.57:0/2080626619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe170153c80 msgr2=0x7fe1700b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:11.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.648+0000 7fe175fd3640 1 --2- 172.21.15.57:0/2080626619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe170153c80 0x7fe1700b3470 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe16401d3a0 tx=0x7fe16401ba90 comp rx=0 tx=0).stop 2023-10-30T14:14:11.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.648+0000 7fe175fd3640 1 -- 172.21.15.57:0/2080626619 shutdown_connections 2023-10-30T14:14:11.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.648+0000 7fe175fd3640 1 --2- 172.21.15.57:0/2080626619 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe15c0664d0 0x7fe15c068990 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.648+0000 7fe175fd3640 1 --2- 172.21.15.57:0/2080626619 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe17015a3b0 0x7fe170149b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.648+0000 7fe175fd3640 1 --2- 172.21.15.57:0/2080626619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe170153c80 0x7fe1700b3470 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.648+0000 7fe175fd3640 1 --2- 172.21.15.57:0/2080626619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1700c4a10 0x7fe170151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.648+0000 7fe175fd3640 1 -- 172.21.15.57:0/2080626619 >> 172.21.15.57:0/2080626619 conn(0x7fe1700715c0 msgr2=0x7fe170158450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:11.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.648+0000 7fe175fd3640 1 -- 172.21.15.57:0/2080626619 shutdown_connections 2023-10-30T14:14:11.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.648+0000 7fe175fd3640 1 -- 172.21.15.57:0/2080626619 wait complete. 2023-10-30T14:14:11.671 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":5,"active_gid":4111,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3881513431},{"type":"v1","addr":"172.21.15.88:6833","nonce":3881513431}]},"active_addr":"172.21.15.88:6833/3881513431","active_change":"2023-10-30T14:13:44.472643+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4101,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":1274096754}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":4056339741}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":2083198484}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":4190004168}]}]} 2023-10-30T14:14:11.674 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-30T14:14:11.675 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-30T14:14:11.675 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:14:11.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.816+0000 7f0b4e679640 1 Processor -- start 2023-10-30T14:14:11.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.816+0000 7f0b4e679640 1 -- start start 2023-10-30T14:14:11.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.816+0000 7f0b4e679640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0b48153c80 0x7f0b4815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:11.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.816+0000 7f0b4e679640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0b480c42a0 0x7f0b480ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:11.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.820+0000 7f0b4e679640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b48159f20 con 0x7f0b480c4a10 2023-10-30T14:14:11.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.820+0000 7f0b4e679640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b4815dec0 con 0x7f0b48153c80 2023-10-30T14:14:11.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.820+0000 7f0b4e679640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b481599a0 con 0x7f0b480c42a0 2023-10-30T14:14:11.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.820+0000 7f0b47fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0b48153c80 0x7f0b4815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:11.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.820+0000 7f0b47fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0b48153c80 0x7f0b4815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:44816/0 (socket says 172.21.15.57:44816) 2023-10-30T14:14:11.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.820+0000 7f0b47fff640 1 -- 172.21.15.57:0/1976487618 learned_addr learned my addr 172.21.15.57:0/1976487618 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:11.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.820+0000 7f0b4cc16640 1 --2- 172.21.15.57:0/1976487618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0b480c42a0 0x7f0b480ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:11.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.820+0000 7f0b4cc16640 1 -- 172.21.15.57:0/1976487618 >> v2:172.21.15.88:3300/0 conn(0x7f0b48153c80 msgr2=0x7f0b4815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:11.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.820+0000 7f0b4cc16640 1 --2- 172.21.15.57:0/1976487618 >> v2:172.21.15.88:3300/0 conn(0x7f0b48153c80 0x7f0b4815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.820+0000 7f0b4cc16640 1 -- 172.21.15.57:0/1976487618 >> v1:172.21.15.88:6789/0 conn(0x7f0b480c4a10 legacy=0x7f0b4815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:11.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.820+0000 7f0b4cc16640 1 -- 172.21.15.57:0/1976487618 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4815d620 con 0x7f0b480c42a0 2023-10-30T14:14:11.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.820+0000 7f0b47fff640 1 --2- 172.21.15.57:0/1976487618 >> v2:172.21.15.88:3300/0 conn(0x7f0b48153c80 0x7f0b4815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:14:11.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.824+0000 7f0b4cc16640 1 --2- 172.21.15.57:0/1976487618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0b480c42a0 0x7f0b480ca1e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0b40009d90 tx=0x7f0b4001f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15ca23393c74f320 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:11.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.824+0000 7f0b46ffd640 1 -- 172.21.15.57:0/1976487618 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b400af600 con 0x7f0b480c42a0 2023-10-30T14:14:11.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.824+0000 7f0b46ffd640 1 -- 172.21.15.57:0/1976487618 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b400ab050 con 0x7f0b480c42a0 2023-10-30T14:14:11.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.824+0000 7f0b46ffd640 1 -- 172.21.15.57:0/1976487618 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b400af980 con 0x7f0b480c42a0 2023-10-30T14:14:11.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.824+0000 7f0b4e679640 1 -- 172.21.15.57:0/1976487618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0b480c42a0 msgr2=0x7f0b480ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:11.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.824+0000 7f0b4e679640 1 --2- 172.21.15.57:0/1976487618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0b480c42a0 0x7f0b480ca1e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0b40009d90 tx=0x7f0b4001f130 comp rx=0 tx=0).stop 2023-10-30T14:14:11.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.824+0000 7f0b4e679640 1 -- 172.21.15.57:0/1976487618 shutdown_connections 2023-10-30T14:14:11.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.824+0000 7f0b4e679640 1 --2- 172.21.15.57:0/1976487618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0b480c42a0 0x7f0b480ca1e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.824+0000 7f0b4e679640 1 --2- 172.21.15.57:0/1976487618 >> v2:172.21.15.88:3300/0 conn(0x7f0b48153c80 0x7f0b4815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.824+0000 7f0b4e679640 1 -- 172.21.15.57:0/1976487618 >> 172.21.15.57:0/1976487618 conn(0x7f0b480715c0 msgr2=0x7f0b480719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:11.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.824+0000 7f0b4e679640 1 -- 172.21.15.57:0/1976487618 shutdown_connections 2023-10-30T14:14:11.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.824+0000 7f0b4e679640 1 -- 172.21.15.57:0/1976487618 wait complete. 2023-10-30T14:14:11.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.824+0000 7f0b4e679640 1 Processor -- start 2023-10-30T14:14:11.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.824+0000 7f0b4e679640 1 -- start start 2023-10-30T14:14:11.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b4e679640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0b480c42a0 0x7f0b480b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:11.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b4e679640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b480c4a10 0x7f0b4814fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:11.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b4e679640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b48153c80 0x7f0b480c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:11.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b4e679640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4814f910 con 0x7f0b48153c80 2023-10-30T14:14:11.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b4e679640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4814f600 con 0x7f0b480c4a10 2023-10-30T14:14:11.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b4e679640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4815c870 con 0x7f0b480c42a0 2023-10-30T14:14:11.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b4cc16640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b48153c80 0x7f0b480c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:11.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b4cc16640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b48153c80 0x7f0b480c8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37906/0 (socket says 172.21.15.57:37906) 2023-10-30T14:14:11.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b4cc16640 1 -- 172.21.15.57:0/218453227 learned_addr learned my addr 172.21.15.57:0/218453227 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:11.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b47fff640 1 --2- 172.21.15.57:0/218453227 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0b480c42a0 0x7f0b480b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:11.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b477fe640 1 --2- 172.21.15.57:0/218453227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b480c4a10 0x7f0b4814fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:11.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b4cc16640 1 -- 172.21.15.57:0/218453227 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0b480c42a0 msgr2=0x7f0b480b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:11.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b4cc16640 1 --2- 172.21.15.57:0/218453227 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0b480c42a0 0x7f0b480b3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b4cc16640 1 -- 172.21.15.57:0/218453227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b480c4a10 msgr2=0x7f0b4814fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:11.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b4cc16640 1 --2- 172.21.15.57:0/218453227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b480c4a10 0x7f0b4814fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:11.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b4cc16640 1 -- 172.21.15.57:0/218453227 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b380094d0 con 0x7f0b48153c80 2023-10-30T14:14:11.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b4cc16640 1 --2- 172.21.15.57:0/218453227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b48153c80 0x7f0b480c8de0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0b400b2040 tx=0x7f0b4001bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:11.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b44ff9640 1 -- 172.21.15.57:0/218453227 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b40009450 con 0x7f0b48153c80 2023-10-30T14:14:11.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b44ff9640 1 -- 172.21.15.57:0/218453227 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b400b9410 con 0x7f0b48153c80 2023-10-30T14:14:11.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b44ff9640 1 -- 172.21.15.57:0/218453227 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b400b97c0 con 0x7f0b48153c80 2023-10-30T14:14:11.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b4e679640 1 -- 172.21.15.57:0/218453227 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b40009a70 con 0x7f0b48153c80 2023-10-30T14:14:11.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b4e679640 1 -- 172.21.15.57:0/218453227 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b4815c6c0 con 0x7f0b48153c80 2023-10-30T14:14:11.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b4e679640 1 -- 172.21.15.57:0/218453227 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b4814fa50 con 0x7f0b48153c80 2023-10-30T14:14:11.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.828+0000 7f0b44ff9640 1 -- 172.21.15.57:0/218453227 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0b400b6050 con 0x7f0b48153c80 2023-10-30T14:14:11.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.832+0000 7f0b44ff9640 1 --2- 172.21.15.57:0/218453227 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0b340664a0 0x7f0b34068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:11.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.832+0000 7f0b44ff9640 1 -- 172.21.15.57:0/218453227 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b40122310 con 0x7f0b48153c80 2023-10-30T14:14:11.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.832+0000 7f0b47fff640 1 --2- 172.21.15.57:0/218453227 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0b340664a0 0x7f0b34068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:11.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.832+0000 7f0b47fff640 1 --2- 172.21.15.57:0/218453227 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0b340664a0 0x7f0b34068960 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0b38006cb0 tx=0x7f0b38009240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:11.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.836+0000 7f0b44ff9640 1 -- 172.21.15.57:0/218453227 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0b4814fa50 con 0x7f0b48153c80 2023-10-30T14:14:11.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.992+0000 7f0b4e679640 1 -- 172.21.15.57:0/218453227 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b4814fa50 con 0x7f0b48153c80 2023-10-30T14:14:12.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.996+0000 7f0b44ff9640 1 -- 172.21.15.57:0/218453227 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16433 (secure 0 0 0) 0x7f0b4814fa50 con 0x7f0b48153c80 2023-10-30T14:14:12.000 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:12.001 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:14:05.576818+0000","last_up_change":"2023-10-30T14:13:51.171287+0000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T14:13:52.574639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T14:14:02.600122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T14:14:03.069429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6801","nonce":3287367848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6803","nonce":3287367848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6807","nonce":3287367848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6805","nonce":3287367848}]},"public_addr":"172.21.15.57:6801/3287367848","cluster_addr":"172.21.15.57:6803/3287367848","heartbeat_back_addr":"172.21.15.57:6807/3287367848","heartbeat_front_addr":"172.21.15.57:6805/3287367848","state":["exists","up"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6809","nonce":3340797836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6811","nonce":3340797836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6815","nonce":3340797836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6813","nonce":3340797836}]},"public_addr":"172.21.15.57:6809/3340797836","cluster_addr":"172.21.15.57:6811/3340797836","heartbeat_back_addr":"172.21.15.57:6815/3340797836","heartbeat_front_addr":"172.21.15.57:6813/3340797836","state":["exists","up"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6825","nonce":1700292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6827","nonce":1700292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6831","nonce":1700292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6829","nonce":1700292}]},"public_addr":"172.21.15.57:6825/1700292","cluster_addr":"172.21.15.57:6827/1700292","heartbeat_back_addr":"172.21.15.57:6831/1700292","heartbeat_front_addr":"172.21.15.57:6829/1700292","state":["exists","up"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6817","nonce":2893420631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6819","nonce":2893420631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6823","nonce":2893420631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6821","nonce":2893420631}]},"public_addr":"172.21.15.57:6817/2893420631","cluster_addr":"172.21.15.57:6819/2893420631","heartbeat_back_addr":"172.21.15.57:6823/2893420631","heartbeat_front_addr":"172.21.15.57:6821/2893420631","state":["exists","up"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6807","nonce":2784816485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6811","nonce":2784816485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6815","nonce":2784816485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6813","nonce":2784816485}]},"public_addr":"172.21.15.88:6807/2784816485","cluster_addr":"172.21.15.88:6811/2784816485","heartbeat_back_addr":"172.21.15.88:6815/2784816485","heartbeat_front_addr":"172.21.15.88:6813/2784816485","state":["exists","up"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6801","nonce":2682449812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6803","nonce":2682449812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6810","nonce":2682449812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6806","nonce":2682449812}]},"public_addr":"172.21.15.88:6801/2682449812","cluster_addr":"172.21.15.88:6803/2682449812","heartbeat_back_addr":"172.21.15.88:6810/2682449812","heartbeat_front_addr":"172.21.15.88:6806/2682449812","state":["exists","up"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6822","nonce":3340693897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6827","nonce":3340693897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6831","nonce":3340693897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6829","nonce":3340693897}]},"public_addr":"172.21.15.88:6822/3340693897","cluster_addr":"172.21.15.88:6827/3340693897","heartbeat_back_addr":"172.21.15.88:6831/3340693897","heartbeat_front_addr":"172.21.15.88:6829/3340693897","state":["exists","up"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6817","nonce":2862111199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6820","nonce":2862111199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6825","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6826","nonce":2862111199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6821","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6823","nonce":2862111199}]},"public_addr":"172.21.15.88:6817/2862111199","cluster_addr":"172.21.15.88:6820/2862111199","heartbeat_back_addr":"172.21.15.88:6826/2862111199","heartbeat_front_addr":"172.21.15.88:6823/2862111199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.634920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.773993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.725825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.761452+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.694274+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.650718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:14:12.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.996+0000 7f0b4e679640 1 -- 172.21.15.57:0/218453227 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0b340664a0 msgr2=0x7f0b34068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.996+0000 7f0b4e679640 1 --2- 172.21.15.57:0/218453227 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0b340664a0 0x7f0b34068960 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0b38006cb0 tx=0x7f0b38009240 comp rx=0 tx=0).stop 2023-10-30T14:14:12.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.996+0000 7f0b4e679640 1 -- 172.21.15.57:0/218453227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b48153c80 msgr2=0x7f0b480c8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.996+0000 7f0b4e679640 1 --2- 172.21.15.57:0/218453227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b48153c80 0x7f0b480c8de0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0b400b2040 tx=0x7f0b4001bed0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.996+0000 7f0b4e679640 1 -- 172.21.15.57:0/218453227 shutdown_connections 2023-10-30T14:14:12.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.996+0000 7f0b4e679640 1 --2- 172.21.15.57:0/218453227 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0b340664a0 0x7f0b34068960 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.996+0000 7f0b4e679640 1 --2- 172.21.15.57:0/218453227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b48153c80 0x7f0b480c8de0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.996+0000 7f0b4e679640 1 --2- 172.21.15.57:0/218453227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b480c4a10 0x7f0b4814fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.996+0000 7f0b4e679640 1 --2- 172.21.15.57:0/218453227 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0b480c42a0 0x7f0b480b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.996+0000 7f0b4e679640 1 -- 172.21.15.57:0/218453227 >> 172.21.15.57:0/218453227 conn(0x7f0b480715c0 msgr2=0x7f0b48157a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:12.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.996+0000 7f0b4e679640 1 -- 172.21.15.57:0/218453227 shutdown_connections 2023-10-30T14:14:12.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:11.996+0000 7f0b4e679640 1 -- 172.21.15.57:0/218453227 wait complete. 2023-10-30T14:14:12.021 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-30T14:14:12.022 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:14:12.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.160+0000 7f9a9059d640 1 Processor -- start 2023-10-30T14:14:12.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.160+0000 7f9a9059d640 1 -- start start 2023-10-30T14:14:12.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.160+0000 7f9a9059d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9a88151840 0x7f9a880c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.160+0000 7f9a9059d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a880cb8b0 0x7f9a8806b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.160+0000 7f9a9059d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a880c5030 con 0x7f9a88151c20 2023-10-30T14:14:12.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.160+0000 7f9a9059d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a88157830 con 0x7f9a88151840 2023-10-30T14:14:12.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.160+0000 7f9a9059d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a88157b10 con 0x7f9a880cb8b0 2023-10-30T14:14:12.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.164+0000 7f9a8eb3a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a880cb8b0 0x7f9a8806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.164+0000 7f9a8e339640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9a88151840 0x7f9a880c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.164+0000 7f9a8eb3a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a880cb8b0 0x7f9a8806b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:47572/0 (socket says 172.21.15.57:47572) 2023-10-30T14:14:12.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.164+0000 7f9a8db38640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9a88151c20 0x7f9a88072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:52558/0 (socket says 172.21.15.57:52558) 2023-10-30T14:14:12.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.164+0000 7f9a8e339640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9a88151840 0x7f9a880c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:44830/0 (socket says 172.21.15.57:44830) 2023-10-30T14:14:12.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.164+0000 7f9a8eb3a640 1 -- 172.21.15.57:0/4207340126 learned_addr learned my addr 172.21.15.57:0/4207340126 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:12.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.164+0000 7f9a8eb3a640 1 -- 172.21.15.57:0/4207340126 >> v2:172.21.15.88:3300/0 conn(0x7f9a88151840 msgr2=0x7f9a880c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:12.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.164+0000 7f9a8eb3a640 1 --2- 172.21.15.57:0/4207340126 >> v2:172.21.15.88:3300/0 conn(0x7f9a88151840 0x7f9a880c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.164+0000 7f9a8eb3a640 1 -- 172.21.15.57:0/4207340126 >> v1:172.21.15.88:6789/0 conn(0x7f9a88151c20 legacy=0x7f9a88072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.164+0000 7f9a8eb3a640 1 -- 172.21.15.57:0/4207340126 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a88157df0 con 0x7f9a880cb8b0 2023-10-30T14:14:12.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.164+0000 7f9a8eb3a640 1 --2- 172.21.15.57:0/4207340126 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a880cb8b0 0x7f9a8806b600 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9a80009e50 tx=0x7f9a8001f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9a5d51965fe6114 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.164+0000 7f9a8d337640 1 -- 172.21.15.57:0/4207340126 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a800af400 con 0x7f9a880cb8b0 2023-10-30T14:14:12.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.164+0000 7f9a8d337640 1 -- 172.21.15.57:0/4207340126 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a800ab050 con 0x7f9a880cb8b0 2023-10-30T14:14:12.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.164+0000 7f9a8d337640 1 -- 172.21.15.57:0/4207340126 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a800af780 con 0x7f9a880cb8b0 2023-10-30T14:14:12.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.168+0000 7f9a9059d640 1 -- 172.21.15.57:0/4207340126 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a880cb8b0 msgr2=0x7f9a8806b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.168+0000 7f9a9059d640 1 --2- 172.21.15.57:0/4207340126 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a880cb8b0 0x7f9a8806b600 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9a80009e50 tx=0x7f9a8001f1f0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.168+0000 7f9a9059d640 1 -- 172.21.15.57:0/4207340126 shutdown_connections 2023-10-30T14:14:12.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.168+0000 7f9a9059d640 1 --2- 172.21.15.57:0/4207340126 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a880cb8b0 0x7f9a8806b600 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.168+0000 7f9a9059d640 1 --2- 172.21.15.57:0/4207340126 >> v2:172.21.15.88:3300/0 conn(0x7f9a88151840 0x7f9a880c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.168+0000 7f9a9059d640 1 -- 172.21.15.57:0/4207340126 >> 172.21.15.57:0/4207340126 conn(0x7f9a880cac70 msgr2=0x7f9a880cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:12.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.168+0000 7f9a9059d640 1 -- 172.21.15.57:0/4207340126 shutdown_connections 2023-10-30T14:14:12.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.168+0000 7f9a9059d640 1 -- 172.21.15.57:0/4207340126 wait complete. 2023-10-30T14:14:12.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.168+0000 7f9a9059d640 1 Processor -- start 2023-10-30T14:14:12.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.168+0000 7f9a9059d640 1 -- start start 2023-10-30T14:14:12.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.168+0000 7f9a9059d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a880cb8b0 0x7f9a881f21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.168+0000 7f9a9059d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a88151840 0x7f9a881efe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.168+0000 7f9a9059d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a88151c20 0x7f9a881f0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.168+0000 7f9a9059d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8814fc30 con 0x7f9a880cb8b0 2023-10-30T14:14:12.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.168+0000 7f9a9059d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8814af70 con 0x7f9a88151c20 2023-10-30T14:14:12.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.168+0000 7f9a9059d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8814b250 con 0x7f9a88151840 2023-10-30T14:14:12.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.168+0000 7f9a8e339640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a880cb8b0 0x7f9a881f21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.168+0000 7f9a8e339640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a880cb8b0 0x7f9a881f21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37922/0 (socket says 172.21.15.57:37922) 2023-10-30T14:14:12.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.168+0000 7f9a8e339640 1 -- 172.21.15.57:0/2531159843 learned_addr learned my addr 172.21.15.57:0/2531159843 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:12.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.172+0000 7f9a8db38640 1 --2- 172.21.15.57:0/2531159843 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a88151840 0x7f9a881efe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.172+0000 7f9a8eb3a640 1 --2- 172.21.15.57:0/2531159843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a88151c20 0x7f9a881f0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.172+0000 7f9a8e339640 1 -- 172.21.15.57:0/2531159843 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a88151840 msgr2=0x7f9a881efe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.172+0000 7f9a8e339640 1 --2- 172.21.15.57:0/2531159843 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a88151840 0x7f9a881efe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.172+0000 7f9a8e339640 1 -- 172.21.15.57:0/2531159843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a88151c20 msgr2=0x7f9a881f0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.172+0000 7f9a8e339640 1 --2- 172.21.15.57:0/2531159843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a88151c20 0x7f9a881f0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.172+0000 7f9a8e339640 1 -- 172.21.15.57:0/2531159843 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a80009b30 con 0x7f9a880cb8b0 2023-10-30T14:14:12.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.172+0000 7f9a8db38640 1 --2- 172.21.15.57:0/2531159843 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a88151840 0x7f9a881efe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:12.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.172+0000 7f9a8eb3a640 1 --2- 172.21.15.57:0/2531159843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a88151c20 0x7f9a881f0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:12.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.172+0000 7f9a8e339640 1 --2- 172.21.15.57:0/2531159843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a880cb8b0 0x7f9a881f21a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9a8400e960 tx=0x7f9a8400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.172+0000 7f9a7effd640 1 -- 172.21.15.57:0/2531159843 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a84009690 con 0x7f9a880cb8b0 2023-10-30T14:14:12.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.172+0000 7f9a7effd640 1 -- 172.21.15.57:0/2531159843 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a8400c9a0 con 0x7f9a880cb8b0 2023-10-30T14:14:12.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.172+0000 7f9a7effd640 1 -- 172.21.15.57:0/2531159843 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8400cd10 con 0x7f9a880cb8b0 2023-10-30T14:14:12.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.172+0000 7f9a9059d640 1 -- 172.21.15.57:0/2531159843 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a88148a20 con 0x7f9a880cb8b0 2023-10-30T14:14:12.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.172+0000 7f9a9059d640 1 -- 172.21.15.57:0/2531159843 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a881f2d50 con 0x7f9a880cb8b0 2023-10-30T14:14:12.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.172+0000 7f9a9059d640 1 -- 172.21.15.57:0/2531159843 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a8804e170 con 0x7f9a880cb8b0 2023-10-30T14:14:12.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.172+0000 7f9a7effd640 1 -- 172.21.15.57:0/2531159843 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9a84013470 con 0x7f9a880cb8b0 2023-10-30T14:14:12.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.172+0000 7f9a7effd640 1 --2- 172.21.15.57:0/2531159843 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9a6c0664a0 0x7f9a6c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.172+0000 7f9a7effd640 1 -- 172.21.15.57:0/2531159843 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a84084200 con 0x7f9a880cb8b0 2023-10-30T14:14:12.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.176+0000 7f9a8db38640 1 --2- 172.21.15.57:0/2531159843 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9a6c0664a0 0x7f9a6c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.176+0000 7f9a8db38640 1 --2- 172.21.15.57:0/2531159843 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9a6c0664a0 0x7f9a6c068960 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9a8814f0e0 tx=0x7f9a78007d70 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.176+0000 7f9a7effd640 1 -- 172.21.15.57:0/2531159843 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9a84051550 con 0x7f9a880cb8b0 2023-10-30T14:14:12.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.336+0000 7f9a9059d640 1 -- 172.21.15.57:0/2531159843 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a88155060 con 0x7f9a880cb8b0 2023-10-30T14:14:12.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.336+0000 7f9a7effd640 1 -- 172.21.15.57:0/2531159843 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16433 (secure 0 0 0) 0x7f9a84054b70 con 0x7f9a880cb8b0 2023-10-30T14:14:12.341 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:12.341 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:14:05.576818+0000","last_up_change":"2023-10-30T14:13:51.171287+0000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T14:13:52.574639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T14:14:02.600122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T14:14:03.069429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6801","nonce":3287367848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6803","nonce":3287367848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6807","nonce":3287367848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6805","nonce":3287367848}]},"public_addr":"172.21.15.57:6801/3287367848","cluster_addr":"172.21.15.57:6803/3287367848","heartbeat_back_addr":"172.21.15.57:6807/3287367848","heartbeat_front_addr":"172.21.15.57:6805/3287367848","state":["exists","up"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6809","nonce":3340797836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6811","nonce":3340797836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6815","nonce":3340797836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6813","nonce":3340797836}]},"public_addr":"172.21.15.57:6809/3340797836","cluster_addr":"172.21.15.57:6811/3340797836","heartbeat_back_addr":"172.21.15.57:6815/3340797836","heartbeat_front_addr":"172.21.15.57:6813/3340797836","state":["exists","up"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6825","nonce":1700292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6827","nonce":1700292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6831","nonce":1700292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6829","nonce":1700292}]},"public_addr":"172.21.15.57:6825/1700292","cluster_addr":"172.21.15.57:6827/1700292","heartbeat_back_addr":"172.21.15.57:6831/1700292","heartbeat_front_addr":"172.21.15.57:6829/1700292","state":["exists","up"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6817","nonce":2893420631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6819","nonce":2893420631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6823","nonce":2893420631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6821","nonce":2893420631}]},"public_addr":"172.21.15.57:6817/2893420631","cluster_addr":"172.21.15.57:6819/2893420631","heartbeat_back_addr":"172.21.15.57:6823/2893420631","heartbeat_front_addr":"172.21.15.57:6821/2893420631","state":["exists","up"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6807","nonce":2784816485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6811","nonce":2784816485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6815","nonce":2784816485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6813","nonce":2784816485}]},"public_addr":"172.21.15.88:6807/2784816485","cluster_addr":"172.21.15.88:6811/2784816485","heartbeat_back_addr":"172.21.15.88:6815/2784816485","heartbeat_front_addr":"172.21.15.88:6813/2784816485","state":["exists","up"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6801","nonce":2682449812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6803","nonce":2682449812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6810","nonce":2682449812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6806","nonce":2682449812}]},"public_addr":"172.21.15.88:6801/2682449812","cluster_addr":"172.21.15.88:6803/2682449812","heartbeat_back_addr":"172.21.15.88:6810/2682449812","heartbeat_front_addr":"172.21.15.88:6806/2682449812","state":["exists","up"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6822","nonce":3340693897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6827","nonce":3340693897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6831","nonce":3340693897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6829","nonce":3340693897}]},"public_addr":"172.21.15.88:6822/3340693897","cluster_addr":"172.21.15.88:6827/3340693897","heartbeat_back_addr":"172.21.15.88:6831/3340693897","heartbeat_front_addr":"172.21.15.88:6829/3340693897","state":["exists","up"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6817","nonce":2862111199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6820","nonce":2862111199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6825","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6826","nonce":2862111199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6821","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6823","nonce":2862111199}]},"public_addr":"172.21.15.88:6817/2862111199","cluster_addr":"172.21.15.88:6820/2862111199","heartbeat_back_addr":"172.21.15.88:6826/2862111199","heartbeat_front_addr":"172.21.15.88:6823/2862111199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.634920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.773993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.725825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.761452+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.694274+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.650718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:14:12.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.336+0000 7f9a9059d640 1 -- 172.21.15.57:0/2531159843 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9a6c0664a0 msgr2=0x7f9a6c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.336+0000 7f9a9059d640 1 --2- 172.21.15.57:0/2531159843 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9a6c0664a0 0x7f9a6c068960 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9a8814f0e0 tx=0x7f9a78007d70 comp rx=0 tx=0).stop 2023-10-30T14:14:12.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.336+0000 7f9a9059d640 1 -- 172.21.15.57:0/2531159843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a880cb8b0 msgr2=0x7f9a881f21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.336+0000 7f9a9059d640 1 --2- 172.21.15.57:0/2531159843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a880cb8b0 0x7f9a881f21a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9a8400e960 tx=0x7f9a8400ee20 comp rx=0 tx=0).stop 2023-10-30T14:14:12.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.336+0000 7f9a9059d640 1 -- 172.21.15.57:0/2531159843 shutdown_connections 2023-10-30T14:14:12.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.336+0000 7f9a9059d640 1 --2- 172.21.15.57:0/2531159843 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9a6c0664a0 0x7f9a6c068960 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.336+0000 7f9a9059d640 1 --2- 172.21.15.57:0/2531159843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a88151c20 0x7f9a881f0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.336+0000 7f9a9059d640 1 --2- 172.21.15.57:0/2531159843 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a88151840 0x7f9a881efe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.336+0000 7f9a9059d640 1 --2- 172.21.15.57:0/2531159843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a880cb8b0 0x7f9a881f21a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.336+0000 7f9a9059d640 1 -- 172.21.15.57:0/2531159843 >> 172.21.15.57:0/2531159843 conn(0x7f9a880cac70 msgr2=0x7f9a8814d510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:12.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.336+0000 7f9a9059d640 1 -- 172.21.15.57:0/2531159843 shutdown_connections 2023-10-30T14:14:12.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.336+0000 7f9a9059d640 1 -- 172.21.15.57:0/2531159843 wait complete. 2023-10-30T14:14:12.363 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-30T14:14:12.363 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-30T14:14:12.364 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-30T14:14:12.364 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-30T14:14:12.365 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-30T14:14:12.365 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-30T14:14:12.366 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-30T14:14:12.367 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-30T14:14:12.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.608+0000 7f8f8cedc640 1 Processor -- start 2023-10-30T14:14:12.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.608+0000 7f8f8cedc640 1 -- start start 2023-10-30T14:14:12.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.612+0000 7f8f8cedc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f8004c570 0x7f8f800d6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.612+0000 7f8f8cedc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8f800e0990 0x7f8f800dcb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.612+0000 7f8f8cedc640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f80034dd0 con 0x7f8f800e0dd0 2023-10-30T14:14:12.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.612+0000 7f8f8cedc640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f800d73f0 con 0x7f8f800e0990 2023-10-30T14:14:12.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.612+0000 7f8f8cedc640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8004ba30 con 0x7f8f8004c570 2023-10-30T14:14:12.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.612+0000 7f8f877fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f8004c570 0x7f8f800d6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.612+0000 7f8f877fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f8004c570 0x7f8f800d6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:47590/0 (socket says 172.21.15.57:47590) 2023-10-30T14:14:12.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.612+0000 7f8f877fe640 1 -- 172.21.15.57:0/1376644860 learned_addr learned my addr 172.21.15.57:0/1376644860 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:12.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.612+0000 7f8f877fe640 1 -- 172.21.15.57:0/1376644860 >> v2:172.21.15.57:3300/0 conn(0x7f8f800e0990 msgr2=0x7f8f800dcb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:12.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.612+0000 7f8f86ffd640 1 --2- 172.21.15.57:0/1376644860 >> v2:172.21.15.57:3300/0 conn(0x7f8f800e0990 0x7f8f800dcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.612+0000 7f8f877fe640 1 --2- 172.21.15.57:0/1376644860 >> v2:172.21.15.57:3300/0 conn(0x7f8f800e0990 0x7f8f800dcb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.612+0000 7f8f877fe640 1 -- 172.21.15.57:0/1376644860 >> v1:172.21.15.57:6789/0 conn(0x7f8f800e0dd0 legacy=0x7f8f800dd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:12.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.616+0000 7f8f877fe640 1 -- 172.21.15.57:0/1376644860 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f800e0500 con 0x7f8f8004c570 2023-10-30T14:14:12.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.616+0000 7f5f939f8640 1 Processor -- start 2023-10-30T14:14:12.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.616+0000 7f5f939f8640 1 -- start start 2023-10-30T14:14:12.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.616+0000 7f5f939f8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5f8c153b10 0x7f5f8c0c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.616+0000 7f5f939f8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5f8c0ca8e0 0x7f5f8c06bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.616+0000 7f5f939f8640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5f8c06c480 0x7f5f8c159d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.620+0000 7f5f939f8640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f8c159900 con 0x7f5f8c06c480 2023-10-30T14:14:12.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.620+0000 7f5f939f8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f8c15d070 con 0x7f5f8c0ca8e0 2023-10-30T14:14:12.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.620+0000 7f5f939f8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c15d350 con 0x7f5f8c153b10 2023-10-30T14:14:12.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.620+0000 7f5f91794640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5f8c153b10 0x7f5f8c0c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.620+0000 7f5f91794640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5f8c153b10 0x7f5f8c0c9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:47600/0 (socket says 172.21.15.57:47600) 2023-10-30T14:14:12.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.620+0000 7f5f91794640 1 -- 172.21.15.57:0/156936706 learned_addr learned my addr 172.21.15.57:0/156936706 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:12.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.620+0000 7f8f877fe640 1 --2- 172.21.15.57:0/1376644860 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f8004c570 0x7f8f800d6eb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c009870 tx=0x7f8f7c01ecf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=863f6c2c42b21df7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.620+0000 7f5f90f93640 1 --2- 172.21.15.57:0/156936706 >> v2:172.21.15.88:3300/0 conn(0x7f5f8c0ca8e0 0x7f5f8c06bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.620+0000 7f5f91f95640 1 --2- 172.21.15.57:0/156936706 >> v2:172.21.15.57:3300/0 conn(0x7f5f8c06c480 0x7f5f8c159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.620+0000 7f5f91794640 1 -- 172.21.15.57:0/156936706 >> v2:172.21.15.88:3300/0 conn(0x7f5f8c0ca8e0 msgr2=0x7f5f8c06bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:12.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.620+0000 7f5f91794640 1 --2- 172.21.15.57:0/156936706 >> v2:172.21.15.88:3300/0 conn(0x7f5f8c0ca8e0 0x7f5f8c06bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.620+0000 7f5f91794640 1 -- 172.21.15.57:0/156936706 >> v2:172.21.15.57:3300/0 conn(0x7f5f8c06c480 msgr2=0x7f5f8c159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:12.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.620+0000 7f5f91794640 1 --2- 172.21.15.57:0/156936706 >> v2:172.21.15.57:3300/0 conn(0x7f5f8c06c480 0x7f5f8c159d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.620+0000 7f5f91794640 1 -- 172.21.15.57:0/156936706 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8c15d630 con 0x7f5f8c153b10 2023-10-30T14:14:12.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.620+0000 7f581493e640 1 Processor -- start 2023-10-30T14:14:12.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.620+0000 7f8f867fc640 1 -- 172.21.15.57:0/1376644860 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c0a89b0 con 0x7f8f8004c570 2023-10-30T14:14:12.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.620+0000 7f581493e640 1 -- start start 2023-10-30T14:14:12.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.620+0000 7f8f867fc640 1 -- 172.21.15.57:0/1376644860 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7c0a8b10 con 0x7f8f8004c570 2023-10-30T14:14:12.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.620+0000 7f8f867fc640 1 -- 172.21.15.57:0/1376644860 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c0a8ea0 con 0x7f8f8004c570 2023-10-30T14:14:12.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f8f8cedc640 1 -- 172.21.15.57:0/1376644860 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f8004c570 msgr2=0x7f8f800d6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f8f8cedc640 1 --2- 172.21.15.57:0/1376644860 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f8004c570 0x7f8f800d6eb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c009870 tx=0x7f8f7c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f5f91794640 1 --2- 172.21.15.57:0/156936706 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5f8c153b10 0x7f5f8c0c9b90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c06cb60 tx=0x7f5f8401c9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20977c3cf05d83b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f581493e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f581014fb70 0x7f581006fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f5f83fff640 1 -- 172.21.15.57:0/156936706 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f840a5760 con 0x7f5f8c153b10 2023-10-30T14:14:12.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f5f83fff640 1 -- 172.21.15.57:0/156936706 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f840a58c0 con 0x7f5f8c153b10 2023-10-30T14:14:12.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f5f83fff640 1 -- 172.21.15.57:0/156936706 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f840a5c50 con 0x7f5f8c153b10 2023-10-30T14:14:12.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f8f8cedc640 1 -- 172.21.15.57:0/1376644860 shutdown_connections 2023-10-30T14:14:12.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f8f8cedc640 1 --2- 172.21.15.57:0/1376644860 >> v2:172.21.15.57:3300/0 conn(0x7f8f800e0990 0x7f8f800dcb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f8f8cedc640 1 --2- 172.21.15.57:0/1376644860 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f8004c570 0x7f8f800d6eb0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f8f8cedc640 1 -- 172.21.15.57:0/1376644860 >> 172.21.15.57:0/1376644860 conn(0x7f8f80050280 msgr2=0x7f8f80050690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:12.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f581493e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f581006b500 0x7f581006b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f5f939f8640 1 -- 172.21.15.57:0/156936706 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5f8c153b10 msgr2=0x7f5f8c0c9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f5f939f8640 1 --2- 172.21.15.57:0/156936706 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5f8c153b10 0x7f5f8c0c9b90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c06cb60 tx=0x7f5f8401c9d0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f5f939f8640 1 -- 172.21.15.57:0/156936706 shutdown_connections 2023-10-30T14:14:12.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f5f939f8640 1 --2- 172.21.15.57:0/156936706 >> v2:172.21.15.57:3300/0 conn(0x7f5f8c06c480 0x7f5f8c159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f5f939f8640 1 --2- 172.21.15.57:0/156936706 >> v2:172.21.15.88:3300/0 conn(0x7f5f8c0ca8e0 0x7f5f8c06bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f5f939f8640 1 --2- 172.21.15.57:0/156936706 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5f8c153b10 0x7f5f8c0c9b90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f5f939f8640 1 -- 172.21.15.57:0/156936706 >> 172.21.15.57:0/156936706 conn(0x7f5f8c06a240 msgr2=0x7f5f8c06a650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:12.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f580e59c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f581014fb70 0x7f581006fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f580e59c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f581014fb70 0x7f581006fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:47616/0 (socket says 172.21.15.57:47616) 2023-10-30T14:14:12.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f580ed9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f581006b500 0x7f581006b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f580ed9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f581006b500 0x7f581006b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37940/0 (socket says 172.21.15.57:37940) 2023-10-30T14:14:12.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f5f939f8640 1 -- 172.21.15.57:0/156936706 shutdown_connections 2023-10-30T14:14:12.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f5f939f8640 1 -- 172.21.15.57:0/156936706 wait complete. 2023-10-30T14:14:12.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f581493e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f581014ff50 con 0x7f58100c7110 2023-10-30T14:14:12.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f580e59c640 1 -- 172.21.15.57:0/3483801705 learned_addr learned my addr 172.21.15.57:0/3483801705 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:12.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f581493e640 1 -- 172.21.15.57:0/3483801705 --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58100c6030 con 0x7f581006b500 2023-10-30T14:14:12.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f581493e640 1 -- 172.21.15.57:0/3483801705 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58100c6d10 con 0x7f581014fb70 2023-10-30T14:14:12.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f8f8cedc640 1 -- 172.21.15.57:0/1376644860 shutdown_connections 2023-10-30T14:14:12.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.624+0000 7f8f8cedc640 1 -- 172.21.15.57:0/1376644860 wait complete. 2023-10-30T14:14:12.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.628+0000 7f580e59c640 1 -- 172.21.15.57:0/3483801705 >> v2:172.21.15.57:3300/0 conn(0x7f581006b500 msgr2=0x7f581006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:12.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.628+0000 7f580e59c640 1 --2- 172.21.15.57:0/3483801705 >> v2:172.21.15.57:3300/0 conn(0x7f581006b500 0x7f581006b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.628+0000 7f580e59c640 1 -- 172.21.15.57:0/3483801705 >> v1:172.21.15.57:6789/0 conn(0x7f58100c7110 legacy=0x7f58100c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.628+0000 7f580e59c640 1 -- 172.21.15.57:0/3483801705 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58101563b0 con 0x7f581014fb70 2023-10-30T14:14:12.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.628+0000 7f5f939f8640 1 Processor -- start 2023-10-30T14:14:12.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.628+0000 7f8f8cedc640 1 Processor -- start 2023-10-30T14:14:12.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.628+0000 7f3ff3668640 1 Processor -- start 2023-10-30T14:14:12.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.628+0000 7f201d2ea640 1 Processor -- start 2023-10-30T14:14:12.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.628+0000 7f5f939f8640 1 -- start start 2023-10-30T14:14:12.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.628+0000 7f8f8cedc640 1 -- start start 2023-10-30T14:14:12.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.628+0000 7f3ff3668640 1 -- start start 2023-10-30T14:14:12.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f5f939f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f8c06c480 0x7f5f8c0c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f5f939f8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5f8c0ca8e0 0x7f5f8c1526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f5f939f8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f8c153b10 0x7f5f8c152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f3ff3668640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3fec158000 0x7f3fec0c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f3ff3668640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fec15e7d0 0x7f3fec0c45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f3ff3668640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fec0c4110 con 0x7f3fec157c20 2023-10-30T14:14:12.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f3ff3668640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fec0ccc00 con 0x7f3fec158000 2023-10-30T14:14:12.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f3ff3668640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec161370 con 0x7f3fec15e7d0 2023-10-30T14:14:12.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f5f939f8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c1f7240 con 0x7f5f8c153b10 2023-10-30T14:14:12.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f5f939f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c1f6ce0 con 0x7f5f8c06c480 2023-10-30T14:14:12.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f5f939f8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c0c9380 con 0x7f5f8c0ca8e0 2023-10-30T14:14:12.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f5f90f93640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5f8c0ca8e0 0x7f5f8c1526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f5f91794640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f8c06c480 0x7f5f8c0c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.628+0000 7f201d2ea640 1 -- start start 2023-10-30T14:14:12.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f201d2ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2018153b10 0x7f20180c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f201d2ea640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f201815d070 con 0x7f20180ca8e0 2023-10-30T14:14:12.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f201d2ea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f201815d350 con 0x7f2018159d50 2023-10-30T14:14:12.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f201d2ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2018159900 con 0x7f2018153b10 2023-10-30T14:14:12.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f3ff1404640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3fec157c20 0x7f3fec06bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:52572/0 (socket says 172.21.15.57:52572) 2023-10-30T14:14:12.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f3ff1404640 1 -- 172.21.15.57:0/3793515676 learned_addr learned my addr 172.21.15.57:0/3793515676 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:12.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f3ff0c03640 1 --2- 172.21.15.57:0/3793515676 >> v2:172.21.15.88:3300/0 conn(0x7f3fec158000 0x7f3fec0c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f8f8cedc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f8004c570 0x7f8f800d5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f8f8cedc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f800e0990 0x7f8f800c9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f8f8cedc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f800e0dd0 0x7f8f800baea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f8f8cedc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f80048d00 con 0x7f8f8004c570 2023-10-30T14:14:12.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f8f8cedc640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f800c71d0 con 0x7f8f800e0990 2023-10-30T14:14:12.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f8f8cedc640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f800c7da0 con 0x7f8f800e0dd0 2023-10-30T14:14:12.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f8f87fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f800e0dd0 0x7f8f800baea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f8f87fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f800e0dd0 0x7f8f800baea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:47642/0 (socket says 172.21.15.57:47642) 2023-10-30T14:14:12.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f8f87fff640 1 -- 172.21.15.57:0/976685183 learned_addr learned my addr 172.21.15.57:0/976685183 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:12.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f2016ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2018153b10 0x7f20180c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f8f86ffd640 1 --2- 172.21.15.57:0/976685183 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f800e0990 0x7f8f800c9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f2016ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2018153b10 0x7f20180c9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:47658/0 (socket says 172.21.15.57:47658) 2023-10-30T14:14:12.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f2016ffd640 1 -- 172.21.15.57:0/334081439 learned_addr learned my addr 172.21.15.57:0/334081439 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:12.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f5f90f93640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5f8c0ca8e0 0x7f5f8c1526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:47632/0 (socket says 172.21.15.57:47632) 2023-10-30T14:14:12.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f5f90f93640 1 -- 172.21.15.57:0/3796161123 learned_addr learned my addr 172.21.15.57:0/3796161123 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:12.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f5f91f95640 1 --2- 172.21.15.57:0/3796161123 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f8c153b10 0x7f5f8c152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f3ff1c05640 1 --2- 172.21.15.57:0/3793515676 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fec15e7d0 0x7f3fec0c45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7fc0fdbaa640 1 Processor -- start 2023-10-30T14:14:12.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f44d8394640 1 Processor -- start 2023-10-30T14:14:12.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f8f877fe640 1 --2- 172.21.15.57:0/976685183 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f8004c570 0x7f8f800d5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f8f87fff640 1 -- 172.21.15.57:0/976685183 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f800e0990 msgr2=0x7f8f800c9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f8f87fff640 1 --2- 172.21.15.57:0/976685183 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f800e0990 0x7f8f800c9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f8f87fff640 1 -- 172.21.15.57:0/976685183 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f8004c570 msgr2=0x7f8f800d5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f8f87fff640 1 --2- 172.21.15.57:0/976685183 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f8004c570 0x7f8f800d5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f8f87fff640 1 -- 172.21.15.57:0/976685183 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7c0094d0 con 0x7f8f800e0dd0 2023-10-30T14:14:12.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f20167fc640 1 -- 172.21.15.57:0/334081439 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260908128 0 0) 0x7f201815d070 con 0x7f20180ca8e0 2023-10-30T14:14:12.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f20167fc640 1 -- 172.21.15.57:0/334081439 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2004003530 con 0x7f20180ca8e0 2023-10-30T14:14:12.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7fc0fdbaa640 1 -- start start 2023-10-30T14:14:12.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f5f90f93640 1 -- 172.21.15.57:0/3796161123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f8c06c480 msgr2=0x7f5f8c0c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f5f90f93640 1 --2- 172.21.15.57:0/3796161123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f8c06c480 0x7f5f8c0c8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f5f90f93640 1 -- 172.21.15.57:0/3796161123 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f8c153b10 msgr2=0x7f5f8c152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f5f90f93640 1 --2- 172.21.15.57:0/3796161123 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f8c153b10 0x7f5f8c152be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f5f90f93640 1 -- 172.21.15.57:0/3796161123 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f84009b30 con 0x7f5f8c0ca8e0 2023-10-30T14:14:12.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f3fe3fff640 1 -- 172.21.15.57:0/3793515676 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263615070 0 0) 0x7f3fec0c4110 con 0x7f3fec157c20 2023-10-30T14:14:12.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f3fe3fff640 1 -- 172.21.15.57:0/3793515676 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fd8003530 con 0x7f3fec157c20 2023-10-30T14:14:12.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f3ff1c05640 1 -- 172.21.15.57:0/3793515676 >> v2:172.21.15.88:3300/0 conn(0x7f3fec158000 msgr2=0x7f3fec0c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:12.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f3ff1c05640 1 --2- 172.21.15.57:0/3793515676 >> v2:172.21.15.88:3300/0 conn(0x7f3fec158000 0x7f3fec0c9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f3ff1c05640 1 -- 172.21.15.57:0/3793515676 >> v1:172.21.15.88:6789/0 conn(0x7f3fec157c20 legacy=0x7f3fec06bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f3ff1c05640 1 -- 172.21.15.57:0/3793515676 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fec161610 con 0x7f3fec15e7d0 2023-10-30T14:14:12.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f8f87fff640 1 --2- 172.21.15.57:0/976685183 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f800e0dd0 0x7f8f800baea0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8f880745b0 tx=0x7f8f88066e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f44d8394640 1 -- start start 2023-10-30T14:14:12.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f5f90f93640 1 --2- 172.21.15.57:0/3796161123 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5f8c0ca8e0 0x7f5f8c1526a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c00e960 tx=0x7f5f7c00ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f2016ffd640 1 -- 172.21.15.57:0/334081439 >> v1:172.21.15.88:6789/0 conn(0x7f2018159d50 legacy=0x7f201815a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f2016ffd640 1 -- 172.21.15.57:0/334081439 >> v1:172.21.15.57:6789/0 conn(0x7f20180ca8e0 legacy=0x7f201806bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.632+0000 7f2016ffd640 1 -- 172.21.15.57:0/334081439 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201815d630 con 0x7f2018153b10 2023-10-30T14:14:12.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f5f81ffb640 1 -- 172.21.15.57:0/3796161123 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c009690 con 0x7f5f8c0ca8e0 2023-10-30T14:14:12.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f5f81ffb640 1 -- 172.21.15.57:0/3796161123 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7c00ca00 con 0x7f5f8c0ca8e0 2023-10-30T14:14:12.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f5f81ffb640 1 -- 172.21.15.57:0/3796161123 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c00cd70 con 0x7f5f8c0ca8e0 2023-10-30T14:14:12.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f580e59c640 1 --2- 172.21.15.57:0/3483801705 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f581014fb70 0x7f581006fdf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f58101500e0 tx=0x7f580401cdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cf94f7593bbd07d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f5f939f8640 1 -- 172.21.15.57:0/3796161123 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f8c1f6f60 con 0x7f5f8c0ca8e0 2023-10-30T14:14:12.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f8f73fff640 1 -- 172.21.15.57:0/976685183 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f88072560 con 0x7f8f800e0dd0 2023-10-30T14:14:12.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f8f73fff640 1 -- 172.21.15.57:0/976685183 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f88072700 con 0x7f8f800e0dd0 2023-10-30T14:14:12.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f8f73fff640 1 -- 172.21.15.57:0/976685183 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f88072ab0 con 0x7f8f800e0dd0 2023-10-30T14:14:12.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f44d8394640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f44d00cb790 0x7f44d006fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f44d8394640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f44d00c7110 0x7f44d00c6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f44d8394640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44d014f910 0x7f44d006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f44d8394640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44d00c6030 con 0x7f44d00cb790 2023-10-30T14:14:12.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f44d8394640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44d01563b0 con 0x7f44d00c7110 2023-10-30T14:14:12.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f44d8394640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d00c5690 con 0x7f44d014f910 2023-10-30T14:14:12.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7fc0fdbaa640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc0f806b760 0x7fc0f815e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7fc0fdbaa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f806bb40 0x7fc0f8157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7fc0fdbaa640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f8071e00 con 0x7fc0f806c420 2023-10-30T14:14:12.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7fc0fdbaa640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f8070020 con 0x7fc0f806b760 2023-10-30T14:14:12.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7fc0fdbaa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f815eb90 con 0x7fc0f806bb40 2023-10-30T14:14:12.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f44d6931640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44d014f910 0x7f44d006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f44d6931640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44d014f910 0x7f44d006b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:47660/0 (socket says 172.21.15.57:47660) 2023-10-30T14:14:12.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f44d6931640 1 -- 172.21.15.57:0/1733607978 learned_addr learned my addr 172.21.15.57:0/1733607978 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:12.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f8f8cedc640 1 -- 172.21.15.57:0/976685183 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f80049080 con 0x7f8f800e0dd0 2023-10-30T14:14:12.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f8f8cedc640 1 -- 172.21.15.57:0/976685183 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f800c7a50 con 0x7f8f800e0dd0 2023-10-30T14:14:12.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f5f939f8640 1 -- 172.21.15.57:0/3796161123 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f8c159350 con 0x7f5f8c0ca8e0 2023-10-30T14:14:12.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7fc0f7fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc0f806c420 0x7fc0f80c57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:52600/0 (socket says 172.21.15.57:52600) 2023-10-30T14:14:12.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7fc0f6ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f806bb40 0x7fc0f8157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7fc0f6ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f806bb40 0x7fc0f8157a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:47674/0 (socket says 172.21.15.57:47674) 2023-10-30T14:14:12.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7fc0f6ffd640 1 -- 172.21.15.57:0/1943323073 learned_addr learned my addr 172.21.15.57:0/1943323073 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:12.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f44d6130640 1 --2- 172.21.15.57:0/1733607978 >> v2:172.21.15.57:3300/0 conn(0x7f44d00cb790 0x7f44d006fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f44d6931640 1 -- 172.21.15.57:0/1733607978 >> v2:172.21.15.88:3300/0 conn(0x7f44d00c7110 msgr2=0x7f44d00c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:12.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f5f939f8640 1 -- 172.21.15.57:0/3796161123 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5f8c0ccaf0 con 0x7f5f8c0ca8e0 2023-10-30T14:14:12.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f8f8cedc640 1 -- 172.21.15.57:0/976685183 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8f8004bf60 con 0x7f8f800e0dd0 2023-10-30T14:14:12.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7fc0f77fe640 1 --2- 172.21.15.57:0/1943323073 >> v2:172.21.15.57:3300/0 conn(0x7fc0f806b760 0x7fc0f815e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f580d59a640 1 -- 172.21.15.57:0/3483801705 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58040a5c70 con 0x7f581014fb70 2023-10-30T14:14:12.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f580d59a640 1 -- 172.21.15.57:0/3483801705 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58040a5dd0 con 0x7f581014fb70 2023-10-30T14:14:12.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f580d59a640 1 -- 172.21.15.57:0/3483801705 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58040ad660 con 0x7f581014fb70 2023-10-30T14:14:12.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7fb5e60dc640 1 Processor -- start 2023-10-30T14:14:12.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7fc0f67fc640 1 -- 172.21.15.57:0/1943323073 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506980195 0 0) 0x7fc0f8071e00 con 0x7fc0f806c420 2023-10-30T14:14:12.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.636+0000 7f44d592f640 1 --2- 172.21.15.57:0/1733607978 >> v2:172.21.15.88:3300/0 conn(0x7f44d00c7110 0x7f44d00c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f44d6931640 1 --2- 172.21.15.57:0/1733607978 >> v2:172.21.15.88:3300/0 conn(0x7f44d00c7110 0x7f44d00c6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f44d6931640 1 -- 172.21.15.57:0/1733607978 >> v2:172.21.15.57:3300/0 conn(0x7f44d00cb790 msgr2=0x7f44d006fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:12.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f44d6931640 1 --2- 172.21.15.57:0/1733607978 >> v2:172.21.15.57:3300/0 conn(0x7f44d00cb790 0x7f44d006fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f44d6931640 1 -- 172.21.15.57:0/1733607978 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d00c5970 con 0x7f44d014f910 2023-10-30T14:14:12.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7fb5e60dc640 1 -- start start 2023-10-30T14:14:12.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f5f81ffb640 1 -- 172.21.15.57:0/3796161123 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5f7c013470 con 0x7f5f8c0ca8e0 2023-10-30T14:14:12.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f581493e640 1 -- 172.21.15.57:0/3483801705 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f581014fb70 msgr2=0x7f581006fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f581493e640 1 --2- 172.21.15.57:0/3483801705 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f581014fb70 0x7f581006fdf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f58101500e0 tx=0x7f580401cdc0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f2016ffd640 1 --2- 172.21.15.57:0/334081439 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2018153b10 0x7f20180c9b90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2008009e80 tx=0x7f200801ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b10374dd6fcfe34a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f581493e640 1 -- 172.21.15.57:0/3483801705 shutdown_connections 2023-10-30T14:14:12.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f581493e640 1 --2- 172.21.15.57:0/3483801705 >> v2:172.21.15.57:3300/0 conn(0x7f581006b500 0x7f581006b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f581493e640 1 --2- 172.21.15.57:0/3483801705 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f581014fb70 0x7f581006fdf0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f581493e640 1 -- 172.21.15.57:0/3483801705 >> 172.21.15.57:0/3483801705 conn(0x7f58100cac70 msgr2=0x7f58100cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:12.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f581493e640 1 -- 172.21.15.57:0/3483801705 shutdown_connections 2023-10-30T14:14:12.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f20167fc640 1 -- 172.21.15.57:0/334081439 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20080a5810 con 0x7f2018153b10 2023-10-30T14:14:12.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f581493e640 1 -- 172.21.15.57:0/3483801705 wait complete. 2023-10-30T14:14:12.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f5f81ffb640 1 --2- 172.21.15.57:0/3796161123 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5f70066570 0x7f5f70068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f44d6130640 1 --2- 172.21.15.57:0/1733607978 >> v2:172.21.15.57:3300/0 conn(0x7f44d00cb790 0x7f44d006fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:12.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f44d592f640 1 --2- 172.21.15.57:0/1733607978 >> v2:172.21.15.88:3300/0 conn(0x7f44d00c7110 0x7f44d00c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:14:12.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f5f81ffb640 1 -- 172.21.15.57:0/3796161123 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f7c0840a0 con 0x7f5f8c0ca8e0 2023-10-30T14:14:12.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f20167fc640 1 -- 172.21.15.57:0/334081439 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20080a5970 con 0x7f2018153b10 2023-10-30T14:14:12.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f20167fc640 1 -- 172.21.15.57:0/334081439 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20080a5d00 con 0x7f2018153b10 2023-10-30T14:14:12.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7fb5e60dc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb5e00ca0d0 0x7fb5e01596f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7fb5e60dc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5e00ca4b0 0x7fb5e00c4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7fb5e60dc640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e00c4b10 con 0x7fb5e00c5560 2023-10-30T14:14:12.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7fb5e60dc640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e00b27b0 con 0x7fb5e00ca0d0 2023-10-30T14:14:12.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7fb5e60dc640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e00c50d0 con 0x7fb5e00ca4b0 2023-10-30T14:14:12.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f5f81ffb640 1 --2- 172.21.15.57:0/3796161123 >> [v2:172.21.15.57:6824/1700292,v1:172.21.15.57:6825/1700292] conn(0x7f5f7006e790 0x7f5f70070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f5f81ffb640 1 -- 172.21.15.57:0/3796161123 --> [v2:172.21.15.57:6824/1700292,v1:172.21.15.57:6825/1700292] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f70071260 con 0x7f5f7006e790 2023-10-30T14:14:12.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f5f81ffb640 1 -- 172.21.15.57:0/3796161123 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f7c0866c0 con 0x7f5f8c0ca8e0 2023-10-30T14:14:12.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f5f91794640 1 --2- 172.21.15.57:0/3796161123 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5f70066570 0x7f5f70068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7fb5e50da640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb5e00ca0d0 0x7fb5e01596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7fb5e50da640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb5e00ca0d0 0x7fb5e01596f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37968/0 (socket says 172.21.15.57:37968) 2023-10-30T14:14:12.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7fb5e50da640 1 -- 172.21.15.57:0/1523866498 learned_addr learned my addr 172.21.15.57:0/1523866498 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:12.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7fc0f67fc640 1 -- 172.21.15.57:0/1943323073 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0e0003530 con 0x7fc0f806c420 2023-10-30T14:14:12.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7fb5e48d9640 1 --2- 172.21.15.57:0/1523866498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5e00ca4b0 0x7fb5e00c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f5f91f95640 1 --2- 172.21.15.57:0/3796161123 >> [v2:172.21.15.57:6824/1700292,v1:172.21.15.57:6825/1700292] conn(0x7f5f7006e790 0x7f5f70070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f3ff1c05640 1 --2- 172.21.15.57:0/3793515676 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fec15e7d0 0x7f3fec0c45a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4009d90 tx=0x7f3fe401f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4de5126e3ca6f44b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f8f73fff640 1 -- 172.21.15.57:0/976685183 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8f880a2020 con 0x7f8f800e0dd0 2023-10-30T14:14:12.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f201d2ea640 1 -- 172.21.15.57:0/334081439 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2018153b10 msgr2=0x7f20180c9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f201d2ea640 1 --2- 172.21.15.57:0/334081439 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2018153b10 0x7f20180c9b90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2008009e80 tx=0x7f200801ca30 comp rx=0 tx=0).stop 2023-10-30T14:14:12.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7fb5e48d9640 1 -- 172.21.15.57:0/1523866498 >> v2:172.21.15.57:3300/0 conn(0x7fb5e00ca0d0 msgr2=0x7fb5e01596f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:12.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7fb5e48d9640 1 --2- 172.21.15.57:0/1523866498 >> v2:172.21.15.57:3300/0 conn(0x7fb5e00ca0d0 0x7fb5e01596f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7fb5e48d9640 1 -- 172.21.15.57:0/1523866498 >> v1:172.21.15.57:6789/0 conn(0x7fb5e00c5560 legacy=0x7fb5e0153520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:12.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f201d2ea640 1 -- 172.21.15.57:0/334081439 shutdown_connections 2023-10-30T14:14:12.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7f201d2ea640 1 --2- 172.21.15.57:0/334081439 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2018153b10 0x7f20180c9b90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7fc0f6ffd640 1 -- 172.21.15.57:0/1943323073 >> v2:172.21.15.57:3300/0 conn(0x7fc0f806b760 msgr2=0x7fc0f815e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:12.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7fc0f6ffd640 1 --2- 172.21.15.57:0/1943323073 >> v2:172.21.15.57:3300/0 conn(0x7fc0f806b760 0x7fc0f815e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7fc0f6ffd640 1 -- 172.21.15.57:0/1943323073 >> v1:172.21.15.88:6789/0 conn(0x7fc0f806c420 legacy=0x7fc0f80c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7fc0f6ffd640 1 -- 172.21.15.57:0/1943323073 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f8070f10 con 0x7fc0f806bb40 2023-10-30T14:14:12.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.640+0000 7fc0f77fe640 1 --2- 172.21.15.57:0/1943323073 >> v2:172.21.15.57:3300/0 conn(0x7fc0f806b760 0x7fc0f815e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:12.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f5f91794640 1 --2- 172.21.15.57:0/3796161123 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5f70066570 0x7f5f70068a30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5f840ae000 tx=0x7f5f840a7070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7fb5e48d9640 1 -- 172.21.15.57:0/1523866498 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e0159f80 con 0x7fb5e00ca4b0 2023-10-30T14:14:12.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f201d2ea640 1 -- 172.21.15.57:0/334081439 >> 172.21.15.57:0/334081439 conn(0x7f201806a240 msgr2=0x7f201806a650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:12.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f8f73fff640 1 --2- 172.21.15.57:0/976685183 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f6c066570 0x7f8f6c068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f201d2ea640 1 -- 172.21.15.57:0/334081439 shutdown_connections 2023-10-30T14:14:12.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f201d2ea640 1 -- 172.21.15.57:0/334081439 wait complete. 2023-10-30T14:14:12.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f8f73fff640 1 -- 172.21.15.57:0/976685183 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f8810f1b0 con 0x7f8f800e0dd0 2023-10-30T14:14:12.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f3fe3fff640 1 -- 172.21.15.57:0/3793515676 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe40af600 con 0x7f3fec15e7d0 2023-10-30T14:14:12.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f3fe3fff640 1 -- 172.21.15.57:0/3793515676 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe40ab050 con 0x7f3fec15e7d0 2023-10-30T14:14:12.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f3fe3fff640 1 -- 172.21.15.57:0/3793515676 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe40af980 con 0x7f3fec15e7d0 2023-10-30T14:14:12.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f581493e640 1 Processor -- start 2023-10-30T14:14:12.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f8f73fff640 1 --2- 172.21.15.57:0/976685183 >> [v2:172.21.15.57:6808/3340797836,v1:172.21.15.57:6809/3340797836] conn(0x7f8f6c06e790 0x7f8f6c070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f8f73fff640 1 -- 172.21.15.57:0/976685183 --> [v2:172.21.15.57:6808/3340797836,v1:172.21.15.57:6809/3340797836] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8f6c071260 con 0x7f8f6c06e790 2023-10-30T14:14:12.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f8f73fff640 1 -- 172.21.15.57:0/976685183 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f8810f560 con 0x7f8f800e0dd0 2023-10-30T14:14:12.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f5f91f95640 1 --2- 172.21.15.57:0/3796161123 >> [v2:172.21.15.57:6824/1700292,v1:172.21.15.57:6825/1700292] conn(0x7f5f7006e790 0x7f5f70070bb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f8f877fe640 1 --2- 172.21.15.57:0/976685183 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f6c066570 0x7f8f6c068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f3ff3668640 1 -- 172.21.15.57:0/3793515676 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fec15e7d0 msgr2=0x7f3fec0c45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f3ff3668640 1 --2- 172.21.15.57:0/3793515676 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fec15e7d0 0x7f3fec0c45a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4009d90 tx=0x7f3fe401f130 comp rx=0 tx=0).stop 2023-10-30T14:14:12.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f3ff3668640 1 -- 172.21.15.57:0/3793515676 shutdown_connections 2023-10-30T14:14:12.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f3ff3668640 1 --2- 172.21.15.57:0/3793515676 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fec15e7d0 0x7f3fec0c45a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f3ff3668640 1 --2- 172.21.15.57:0/3793515676 >> v2:172.21.15.88:3300/0 conn(0x7f3fec158000 0x7f3fec0c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f3ff3668640 1 -- 172.21.15.57:0/3793515676 >> 172.21.15.57:0/3793515676 conn(0x7f3fec06a240 msgr2=0x7f3fec06a650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:12.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f581493e640 1 -- start start 2023-10-30T14:14:12.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f3ff3668640 1 -- 172.21.15.57:0/3793515676 shutdown_connections 2023-10-30T14:14:12.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f3ff3668640 1 -- 172.21.15.57:0/3793515676 wait complete. 2023-10-30T14:14:12.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f8f877fe640 1 --2- 172.21.15.57:0/976685183 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f6c066570 0x7f8f6c068a30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c0b2000 tx=0x7f8f7c0aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f5f81ffb640 1 -- 172.21.15.57:0/3796161123 <== osd.2 v2:172.21.15.57:6824/1700292 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f5f70071260 con 0x7f5f7006e790 2023-10-30T14:14:12.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f8f86ffd640 1 --2- 172.21.15.57:0/976685183 >> [v2:172.21.15.57:6808/3340797836,v1:172.21.15.57:6809/3340797836] conn(0x7f8f6c06e790 0x7f8f6c070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f581493e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f581006b500 0x7f5810154340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f581493e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58100c7110 0x7f5810154ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f8f86ffd640 1 --2- 172.21.15.57:0/976685183 >> [v2:172.21.15.57:6808/3340797836,v1:172.21.15.57:6809/3340797836] conn(0x7f8f6c06e790 0x7f8f6c070bb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f581493e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f581014fb70 0x7f58101f6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f581493e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58100ca9a0 con 0x7f581014fb70 2023-10-30T14:14:12.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f581493e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5810154880 con 0x7f581006b500 2023-10-30T14:14:12.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f581493e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58100c1990 con 0x7f58100c7110 2023-10-30T14:14:12.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f580e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f581006b500 0x7f5810154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f580e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f581006b500 0x7f5810154340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:44902/0 (socket says 172.21.15.57:44902) 2023-10-30T14:14:12.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.644+0000 7f580e59c640 1 -- 172.21.15.57:0/1032711669 learned_addr learned my addr 172.21.15.57:0/1032711669 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:12.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f44d6931640 1 --2- 172.21.15.57:0/1733607978 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44d014f910 0x7f44d006b500 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f44d00c25e0 tx=0x7f44c801f1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=195fa025c49818aa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f580dd9b640 1 --2- 172.21.15.57:0/1032711669 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58100c7110 0x7f5810154ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f580ed9d640 1 --2- 172.21.15.57:0/1032711669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f581014fb70 0x7f58101f6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f580e59c640 1 -- 172.21.15.57:0/1032711669 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58100c7110 msgr2=0x7f5810154ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f580e59c640 1 --2- 172.21.15.57:0/1032711669 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58100c7110 0x7f5810154ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f580e59c640 1 -- 172.21.15.57:0/1032711669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f581014fb70 msgr2=0x7f58101f6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f580e59c640 1 --2- 172.21.15.57:0/1032711669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f581014fb70 0x7f58101f6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f580e59c640 1 -- 172.21.15.57:0/1032711669 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58040094d0 con 0x7f581006b500 2023-10-30T14:14:12.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f201d2ea640 1 Processor -- start 2023-10-30T14:14:12.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f201d2ea640 1 -- start start 2023-10-30T14:14:12.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f3ff3668640 1 Processor -- start 2023-10-30T14:14:12.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f44d512e640 1 -- 172.21.15.57:0/1733607978 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c80af400 con 0x7f44d014f910 2023-10-30T14:14:12.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7fb5e48d9640 1 --2- 172.21.15.57:0/1523866498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5e00ca4b0 0x7fb5e00c4200 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb5d00097c0 tx=0x7fb5d001c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed4edaccf2c5ebd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f8f73fff640 1 -- 172.21.15.57:0/976685183 <== osd.1 v2:172.21.15.57:6808/3340797836 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f8f6c071260 con 0x7f8f6c06e790 2023-10-30T14:14:12.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f3ff3668640 1 -- start start 2023-10-30T14:14:12.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f44d512e640 1 -- 172.21.15.57:0/1733607978 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c80ab050 con 0x7f44d014f910 2023-10-30T14:14:12.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f44d512e640 1 -- 172.21.15.57:0/1733607978 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c80af780 con 0x7f44d014f910 2023-10-30T14:14:12.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7fb5d7fff640 1 -- 172.21.15.57:0/1523866498 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d00ac630 con 0x7fb5e00ca4b0 2023-10-30T14:14:12.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7fc0f6ffd640 1 --2- 172.21.15.57:0/1943323073 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f806bb40 0x7fc0f8157a20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec009870 tx=0x7fc0ec01c530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b8c55e466857185 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f580e59c640 1 --2- 172.21.15.57:0/1032711669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f581006b500 0x7f5810154340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f580401a9f0 tx=0x7f5804019b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7fb5d7fff640 1 -- 172.21.15.57:0/1523866498 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d00ac790 con 0x7fb5e00ca4b0 2023-10-30T14:14:12.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7fb5d7fff640 1 -- 172.21.15.57:0/1523866498 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d00acb20 con 0x7fb5e00ca4b0 2023-10-30T14:14:12.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7fc0f67fc640 1 -- 172.21.15.57:0/1943323073 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec0a95f0 con 0x7fc0f806bb40 2023-10-30T14:14:12.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7fc0f67fc640 1 -- 172.21.15.57:0/1943323073 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ec0a9750 con 0x7fc0f806bb40 2023-10-30T14:14:12.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7fc0f67fc640 1 -- 172.21.15.57:0/1943323073 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec0a95f0 con 0x7fc0f806bb40 2023-10-30T14:14:12.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f44d8394640 1 -- 172.21.15.57:0/1733607978 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44d014f910 msgr2=0x7f44d006b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f44d8394640 1 --2- 172.21.15.57:0/1733607978 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44d014f910 0x7f44d006b500 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f44d00c25e0 tx=0x7f44c801f1c0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f3ff3668640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fec157c20 0x7f3fec156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f3ff3668640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fec158000 0x7f3fec1f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f3ff3668640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fec15e7d0 0x7f3fec0b27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f3ff3668640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec15fa10 con 0x7f3fec15e7d0 2023-10-30T14:14:12.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f3ff3668640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec0b2ce0 con 0x7f3fec158000 2023-10-30T14:14:12.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f3ff3668640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec156480 con 0x7f3fec157c20 2023-10-30T14:14:12.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f201d2ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20180ca8e0 0x7f20180cef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f44d8394640 1 -- 172.21.15.57:0/1733607978 shutdown_connections 2023-10-30T14:14:12.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f44d8394640 1 --2- 172.21.15.57:0/1733607978 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44d014f910 0x7f44d006b500 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f44d8394640 1 --2- 172.21.15.57:0/1733607978 >> v2:172.21.15.88:3300/0 conn(0x7f44d00c7110 0x7f44d00c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f44d8394640 1 --2- 172.21.15.57:0/1733607978 >> v2:172.21.15.57:3300/0 conn(0x7f44d00cb790 0x7f44d006fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f44d8394640 1 -- 172.21.15.57:0/1733607978 >> 172.21.15.57:0/1733607978 conn(0x7f44d00cac70 msgr2=0x7f44d00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:12.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f44d8394640 1 -- 172.21.15.57:0/1733607978 shutdown_connections 2023-10-30T14:14:12.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f3ff1404640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fec157c20 0x7f3fec156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f57feffd640 1 -- 172.21.15.57:0/1032711669 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58040b6660 con 0x7f581006b500 2023-10-30T14:14:12.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f2016ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20180ca8e0 0x7f20180cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f3ff1404640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fec157c20 0x7f3fec156e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:47702/0 (socket says 172.21.15.57:47702) 2023-10-30T14:14:12.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f3ff1404640 1 -- 172.21.15.57:0/3421946460 learned_addr learned my addr 172.21.15.57:0/3421946460 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:12.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7fc0fdbaa640 1 -- 172.21.15.57:0/1943323073 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f806bb40 msgr2=0x7fc0f8157a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7fc0fdbaa640 1 --2- 172.21.15.57:0/1943323073 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f806bb40 0x7fc0f8157a20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec009870 tx=0x7fc0ec01c530 comp rx=0 tx=0).stop 2023-10-30T14:14:12.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f44d8394640 1 -- 172.21.15.57:0/1733607978 wait complete. 2023-10-30T14:14:12.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f3ff0c03640 1 --2- 172.21.15.57:0/3421946460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fec158000 0x7f3fec1f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7fb5e60dc640 1 -- 172.21.15.57:0/1523866498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5e00ca4b0 msgr2=0x7fb5e00c4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7f57feffd640 1 -- 172.21.15.57:0/1032711669 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58040b67c0 con 0x7f581006b500 2023-10-30T14:14:12.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.648+0000 7fb5e60dc640 1 --2- 172.21.15.57:0/1523866498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5e00ca4b0 0x7fb5e00c4200 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb5d00097c0 tx=0x7fb5d001c410 comp rx=0 tx=0).stop 2023-10-30T14:14:12.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f57feffd640 1 -- 172.21.15.57:0/1032711669 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58040b6b70 con 0x7f581006b500 2023-10-30T14:14:12.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f201d2ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2018153b10 0x7f20181f2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f2016ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20180ca8e0 0x7f20180cef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37992/0 (socket says 172.21.15.57:37992) 2023-10-30T14:14:12.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7fc0fdbaa640 1 -- 172.21.15.57:0/1943323073 shutdown_connections 2023-10-30T14:14:12.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7fc0fdbaa640 1 --2- 172.21.15.57:0/1943323073 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f806bb40 0x7fc0f8157a20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7fc0fdbaa640 1 --2- 172.21.15.57:0/1943323073 >> v2:172.21.15.57:3300/0 conn(0x7fc0f806b760 0x7fc0f815e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7fc0fdbaa640 1 -- 172.21.15.57:0/1943323073 >> 172.21.15.57:0/1943323073 conn(0x7fc0f80cac70 msgr2=0x7fc0f80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:12.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f2016ffd640 1 -- 172.21.15.57:0/1287137320 learned_addr learned my addr 172.21.15.57:0/1287137320 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:12.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f581493e640 1 -- 172.21.15.57:0/1032711669 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58100c1c30 con 0x7f581006b500 2023-10-30T14:14:12.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f581493e640 1 -- 172.21.15.57:0/1032711669 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58100ca100 con 0x7f581006b500 2023-10-30T14:14:12.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f201d2ea640 1 --2- 172.21.15.57:0/1287137320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2018159d50 0x7f20181f55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f201d2ea640 1 -- 172.21.15.57:0/1287137320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2018154a50 con 0x7f20180ca8e0 2023-10-30T14:14:12.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f201d2ea640 1 -- 172.21.15.57:0/1287137320 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20181f2690 con 0x7f2018159d50 2023-10-30T14:14:12.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f201d2ea640 1 -- 172.21.15.57:0/1287137320 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20181f5b10 con 0x7f2018153b10 2023-10-30T14:14:12.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f200e7fc640 1 --2- 172.21.15.57:0/1287137320 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2018153b10 0x7f20181f2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f3ff1c05640 1 --2- 172.21.15.57:0/3421946460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fec15e7d0 0x7f3fec0b27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f3ff1404640 1 -- 172.21.15.57:0/3421946460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fec158000 msgr2=0x7f3fec1f2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f3ff1404640 1 --2- 172.21.15.57:0/3421946460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fec158000 0x7f3fec1f2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f3ff1404640 1 -- 172.21.15.57:0/3421946460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fec15e7d0 msgr2=0x7f3fec0b27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f3ff1404640 1 --2- 172.21.15.57:0/3421946460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fec15e7d0 0x7f3fec0b27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f3ff1404640 1 -- 172.21.15.57:0/3421946460 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe4009a70 con 0x7f3fec157c20 2023-10-30T14:14:12.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f3ff1c05640 1 --2- 172.21.15.57:0/3421946460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fec15e7d0 0x7f3fec0b27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:14:12.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f3ff0c03640 1 --2- 172.21.15.57:0/3421946460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fec158000 0x7f3fec1f2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:12.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f3ff1404640 1 --2- 172.21.15.57:0/3421946460 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fec157c20 0x7f3fec156e90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3fe800c450 tx=0x7f3fe800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f20177fe640 1 --2- 172.21.15.57:0/1287137320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2018159d50 0x7f20181f55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f581493e640 1 -- 172.21.15.57:0/1032711669 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f57f4000fd0 con 0x7f581006b500 2023-10-30T14:14:12.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7fc0fdbaa640 1 -- 172.21.15.57:0/1943323073 shutdown_connections 2023-10-30T14:14:12.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f3fe1ffb640 1 -- 172.21.15.57:0/3421946460 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe8005cf0 con 0x7f3fec157c20 2023-10-30T14:14:12.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f3fe1ffb640 1 -- 172.21.15.57:0/3421946460 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe8005e50 con 0x7f3fec157c20 2023-10-30T14:14:12.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f3fe1ffb640 1 -- 172.21.15.57:0/3421946460 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe8007270 con 0x7f3fec157c20 2023-10-30T14:14:12.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f3ff3668640 1 -- 172.21.15.57:0/3421946460 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fec0b2f30 con 0x7f3fec157c20 2023-10-30T14:14:12.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7fc0fdbaa640 1 -- 172.21.15.57:0/1943323073 wait complete. 2023-10-30T14:14:12.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f200e7fc640 1 -- 172.21.15.57:0/1287137320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2018159d50 msgr2=0x7f20181f55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f200e7fc640 1 --2- 172.21.15.57:0/1287137320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2018159d50 0x7f20181f55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f200e7fc640 1 -- 172.21.15.57:0/1287137320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20180ca8e0 msgr2=0x7f20180cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f3ff3668640 1 -- 172.21.15.57:0/3421946460 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fec0b34f0 con 0x7f3fec157c20 2023-10-30T14:14:12.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7fb5e60dc640 1 -- 172.21.15.57:0/1523866498 shutdown_connections 2023-10-30T14:14:12.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f200e7fc640 1 --2- 172.21.15.57:0/1287137320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20180ca8e0 0x7f20180cef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7fb5e60dc640 1 --2- 172.21.15.57:0/1523866498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5e00ca4b0 0x7fb5e00c4200 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7fb5e60dc640 1 --2- 172.21.15.57:0/1523866498 >> v2:172.21.15.57:3300/0 conn(0x7fb5e00ca0d0 0x7fb5e01596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7fb5e60dc640 1 -- 172.21.15.57:0/1523866498 >> 172.21.15.57:0/1523866498 conn(0x7fb5e006bf20 msgr2=0x7fb5e006c330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:12.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f44d8394640 1 Processor -- start 2023-10-30T14:14:12.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f200e7fc640 1 -- 172.21.15.57:0/1287137320 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2008009b30 con 0x7f2018153b10 2023-10-30T14:14:12.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f200e7fc640 1 --2- 172.21.15.57:0/1287137320 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2018153b10 0x7f20181f2b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f200000c430 tx=0x7f200000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f20177fe640 1 --2- 172.21.15.57:0/1287137320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2018159d50 0x7f20181f55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:12.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7fb5e60dc640 1 -- 172.21.15.57:0/1523866498 shutdown_connections 2023-10-30T14:14:12.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f44d8394640 1 -- start start 2023-10-30T14:14:12.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7f200ffff640 1 -- 172.21.15.57:0/1287137320 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20000052d0 con 0x7f2018153b10 2023-10-30T14:14:12.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f200ffff640 1 -- 172.21.15.57:0/1287137320 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2000005430 con 0x7f2018153b10 2023-10-30T14:14:12.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f200ffff640 1 -- 172.21.15.57:0/1287137320 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20000052d0 con 0x7f2018153b10 2023-10-30T14:14:12.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.652+0000 7fb5e60dc640 1 -- 172.21.15.57:0/1523866498 wait complete. 2023-10-30T14:14:12.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f57feffd640 1 -- 172.21.15.57:0/1032711669 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f58040b4070 con 0x7f581006b500 2023-10-30T14:14:12.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f201d2ea640 1 -- 172.21.15.57:0/1287137320 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20181f2060 con 0x7f2018153b10 2023-10-30T14:14:12.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f3ff3668640 1 -- 172.21.15.57:0/3421946460 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3fec1618d0 con 0x7f3fec157c20 2023-10-30T14:14:12.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f44d8394640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44d00c7110 0x7f44d00c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f44d8394640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44d00cb790 0x7f44d014fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f44d8394640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44d014f910 0x7f44d013bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f44d8394640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d00c9ff0 con 0x7f44d00cb790 2023-10-30T14:14:12.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f44d8394640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d0151d80 con 0x7f44d00c7110 2023-10-30T14:14:12.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f44d8394640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d0150360 con 0x7f44d014f910 2023-10-30T14:14:12.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f44d6931640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44d014f910 0x7f44d013bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f44d6931640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44d014f910 0x7f44d013bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:47716/0 (socket says 172.21.15.57:47716) 2023-10-30T14:14:12.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f44d6931640 1 -- 172.21.15.57:0/3194022917 learned_addr learned my addr 172.21.15.57:0/3194022917 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:12.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7fb5e60dc640 1 Processor -- start 2023-10-30T14:14:12.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7fb5e60dc640 1 -- start start 2023-10-30T14:14:12.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7fc0fdbaa640 1 Processor -- start 2023-10-30T14:14:12.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f44d6931640 1 -- 172.21.15.57:0/3194022917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44d00c7110 msgr2=0x7f44d00c8a50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-30T14:14:12.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f44d6931640 1 --2- 172.21.15.57:0/3194022917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44d00c7110 0x7f44d00c8a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f44d6931640 1 -- 172.21.15.57:0/3194022917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44d00cb790 msgr2=0x7f44d014fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f44d6931640 1 --2- 172.21.15.57:0/3194022917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44d00cb790 0x7f44d014fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f44d6931640 1 -- 172.21.15.57:0/3194022917 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c8009b30 con 0x7f44d014f910 2023-10-30T14:14:12.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f201d2ea640 1 -- 172.21.15.57:0/1287137320 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20181f6080 con 0x7f2018153b10 2023-10-30T14:14:12.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f201d2ea640 1 -- 172.21.15.57:0/1287137320 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f20180ccaf0 con 0x7f2018153b10 2023-10-30T14:14:12.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f3fe1ffb640 1 -- 172.21.15.57:0/3421946460 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3fe80073d0 con 0x7f3fec157c20 2023-10-30T14:14:12.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f57feffd640 1 --2- 172.21.15.57:0/1032711669 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f57f00665a0 0x7f57f0068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f57feffd640 1 -- 172.21.15.57:0/1032711669 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f580411e660 con 0x7f581006b500 2023-10-30T14:14:12.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f57feffd640 1 --2- 172.21.15.57:0/1032711669 >> [v2:172.21.15.57:6800/3287367848,v1:172.21.15.57:6801/3287367848] conn(0x7f57f006e7c0 0x7f57f0070be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f57feffd640 1 -- 172.21.15.57:0/1032711669 --> [v2:172.21.15.57:6800/3287367848,v1:172.21.15.57:6801/3287367848] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57f0071290 con 0x7f57f006e7c0 2023-10-30T14:14:12.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f57feffd640 1 -- 172.21.15.57:0/1032711669 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f580411ea10 con 0x7f581006b500 2023-10-30T14:14:12.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f580dd9b640 1 --2- 172.21.15.57:0/1032711669 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f57f00665a0 0x7f57f0068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f44d6931640 1 --2- 172.21.15.57:0/3194022917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44d014f910 0x7f44d013bb30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f44c801fa60 tx=0x7f44c801bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7fb5e60dc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb5e00c5560 0x7fb5e0148300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7fb5e60dc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5e00ca0d0 0x7fb5e01524c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7fb5e60dc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5e00ca4b0 0x7fb5e01f9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7fb5e60dc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e00cdc30 con 0x7fb5e00c5560 2023-10-30T14:14:12.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7fb5e60dc640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e0147e70 con 0x7fb5e00ca4b0 2023-10-30T14:14:12.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7fb5e60dc640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e01498f0 con 0x7fb5e00ca0d0 2023-10-30T14:14:12.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7fb5e50da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb5e00c5560 0x7fb5e0148300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7fb5e50da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb5e00c5560 0x7fb5e0148300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38018/0 (socket says 172.21.15.57:38018) 2023-10-30T14:14:12.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7fb5e50da640 1 -- 172.21.15.57:0/4006507787 learned_addr learned my addr 172.21.15.57:0/4006507787 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:12.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7fc0fdbaa640 1 -- start start 2023-10-30T14:14:12.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f580ed9d640 1 --2- 172.21.15.57:0/1032711669 >> [v2:172.21.15.57:6800/3287367848,v1:172.21.15.57:6801/3287367848] conn(0x7f57f006e7c0 0x7f57f0070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.656+0000 7f44c6ffd640 1 -- 172.21.15.57:0/3194022917 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c80afd60 con 0x7f44d014f910 2023-10-30T14:14:12.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f44c6ffd640 1 -- 172.21.15.57:0/3194022917 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c8009510 con 0x7f44d014f910 2023-10-30T14:14:12.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f44c6ffd640 1 -- 172.21.15.57:0/3194022917 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c80afd60 con 0x7f44d014f910 2023-10-30T14:14:12.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f580dd9b640 1 --2- 172.21.15.57:0/1032711669 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f57f00665a0 0x7f57f0068a60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f58100c9b30 tx=0x7f5800009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7fb5e48d9640 1 --2- 172.21.15.57:0/4006507787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5e00ca0d0 0x7fb5e01524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f3fe1ffb640 1 --2- 172.21.15.57:0/3421946460 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3fd00664a0 0x7f3fd0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f3fe1ffb640 1 -- 172.21.15.57:0/3421946460 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3fe8083380 con 0x7f3fec157c20 2023-10-30T14:14:12.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f3fe1ffb640 1 --2- 172.21.15.57:0/3421946460 >> [v2:172.21.15.88:6805/2784816485,v1:172.21.15.88:6807/2784816485] conn(0x7f3fd006e640 0x7f3fd0070a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f3fe1ffb640 1 -- 172.21.15.57:0/3421946460 --> [v2:172.21.15.88:6805/2784816485,v1:172.21.15.88:6807/2784816485] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3fd0071110 con 0x7f3fd006e640 2023-10-30T14:14:12.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f3fe1ffb640 1 -- 172.21.15.57:0/3421946460 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3fe8050780 con 0x7f3fec157c20 2023-10-30T14:14:12.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7fb5e58db640 1 --2- 172.21.15.57:0/4006507787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5e00ca4b0 0x7fb5e01f9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f580ed9d640 1 --2- 172.21.15.57:0/1032711669 >> [v2:172.21.15.57:6800/3287367848,v1:172.21.15.57:6801/3287367848] conn(0x7f57f006e7c0 0x7f57f0070be0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f44d8394640 1 -- 172.21.15.57:0/3194022917 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d00c9090 con 0x7f44d014f910 2023-10-30T14:14:12.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f200ffff640 1 -- 172.21.15.57:0/1287137320 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2000003600 con 0x7f2018153b10 2023-10-30T14:14:12.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f3ff0c03640 1 --2- 172.21.15.57:0/3421946460 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3fd00664a0 0x7f3fd0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f3ff1c05640 1 --2- 172.21.15.57:0/3421946460 >> [v2:172.21.15.88:6805/2784816485,v1:172.21.15.88:6807/2784816485] conn(0x7f3fd006e640 0x7f3fd0070a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f44d8394640 1 -- 172.21.15.57:0/3194022917 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d00c9650 con 0x7f44d014f910 2023-10-30T14:14:12.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7fb5e50da640 1 -- 172.21.15.57:0/4006507787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5e00ca0d0 msgr2=0x7fb5e01524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7fb5e50da640 1 --2- 172.21.15.57:0/4006507787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5e00ca0d0 0x7fb5e01524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7fb5e50da640 1 -- 172.21.15.57:0/4006507787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5e00ca4b0 msgr2=0x7fb5e01f9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7fb5e50da640 1 --2- 172.21.15.57:0/4006507787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5e00ca4b0 0x7fb5e01f9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7fb5e50da640 1 -- 172.21.15.57:0/4006507787 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d0009420 con 0x7fb5e00c5560 2023-10-30T14:14:12.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f3ff0c03640 1 --2- 172.21.15.57:0/3421946460 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3fd00664a0 0x7f3fd0068960 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3fec15d540 tx=0x7f3fdc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7fc0fdbaa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0f806b760 0x7fc0f81f61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f200ffff640 1 --2- 172.21.15.57:0/1287137320 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1ff8066500 0x7f1ff80689c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f200ffff640 1 -- 172.21.15.57:0/1287137320 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2000083cc0 con 0x7f2018153b10 2023-10-30T14:14:12.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7fc0fdbaa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f806bb40 0x7fc0f814ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7fc0fdbaa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f806c420 0x7fc0f81503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7fc0fdbaa640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f815bf90 con 0x7fc0f806c420 2023-10-30T14:14:12.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7fc0fdbaa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f815e100 con 0x7fc0f806b760 2023-10-30T14:14:12.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7fc0fdbaa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f814bfc0 con 0x7fc0f806bb40 2023-10-30T14:14:12.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f3ff1c05640 1 --2- 172.21.15.57:0/3421946460 >> [v2:172.21.15.88:6805/2784816485,v1:172.21.15.88:6807/2784816485] conn(0x7f3fd006e640 0x7f3fd0070a60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f57feffd640 1 -- 172.21.15.57:0/1032711669 <== osd.0 v2:172.21.15.57:6800/3287367848 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f57f0071290 con 0x7f57f006e7c0 2023-10-30T14:14:12.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f2016ffd640 1 --2- 172.21.15.57:0/1287137320 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1ff8066500 0x7f1ff80689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f200ffff640 1 --2- 172.21.15.57:0/1287137320 >> [v2:172.21.15.57:6816/2893420631,v1:172.21.15.57:6817/2893420631] conn(0x7f1ff806e720 0x7f1ff8070b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f200ffff640 1 -- 172.21.15.57:0/1287137320 --> [v2:172.21.15.57:6816/2893420631,v1:172.21.15.57:6817/2893420631] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ff80711f0 con 0x7f1ff806e720 2023-10-30T14:14:12.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f200ffff640 1 -- 172.21.15.57:0/1287137320 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2000084070 con 0x7f2018153b10 2023-10-30T14:14:12.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7f20177fe640 1 --2- 172.21.15.57:0/1287137320 >> [v2:172.21.15.57:6816/2893420631,v1:172.21.15.57:6817/2893420631] conn(0x7f1ff806e720 0x7f1ff8070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7f20177fe640 1 --2- 172.21.15.57:0/1287137320 >> [v2:172.21.15.57:6816/2893420631,v1:172.21.15.57:6817/2893420631] conn(0x7f1ff806e720 0x7f1ff8070b40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7f44d8394640 1 -- 172.21.15.57:0/3194022917 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f44d00c6990 con 0x7f44d014f910 2023-10-30T14:14:12.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7fc0f7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f806c420 0x7fc0f81503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7fc0f6ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f806bb40 0x7fc0f814ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7fc0f7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f806c420 0x7fc0f81503c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38022/0 (socket says 172.21.15.57:38022) 2023-10-30T14:14:12.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7fc0f7fff640 1 -- 172.21.15.57:0/2940178373 learned_addr learned my addr 172.21.15.57:0/2940178373 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:12.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7fc0f77fe640 1 --2- 172.21.15.57:0/2940178373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0f806b760 0x7fc0f81f61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.660+0000 7fb5e50da640 1 --2- 172.21.15.57:0/4006507787 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb5e00c5560 0x7fb5e0148300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb5d80070b0 tx=0x7fb5d800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7fc0f6ffd640 1 -- 172.21.15.57:0/2940178373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0f806b760 msgr2=0x7fc0f81f61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7fc0f6ffd640 1 --2- 172.21.15.57:0/2940178373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0f806b760 0x7fc0f81f61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7f2016ffd640 1 --2- 172.21.15.57:0/1287137320 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1ff8066500 0x7f1ff80689c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f200801ca70 tx=0x7f20080a7070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7f3fe1ffb640 1 -- 172.21.15.57:0/3421946460 <== osd.4 v2:172.21.15.88:6805/2784816485 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f3fd0071110 con 0x7f3fd006e640 2023-10-30T14:14:12.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7f44c6ffd640 1 -- 172.21.15.57:0/3194022917 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f44c80b6040 con 0x7f44d014f910 2023-10-30T14:14:12.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7f200ffff640 1 -- 172.21.15.57:0/1287137320 <== osd.3 v2:172.21.15.57:6816/2893420631 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f1ff80711f0 con 0x7f1ff806e720 2023-10-30T14:14:12.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7fc0f6ffd640 1 -- 172.21.15.57:0/2940178373 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f806c420 msgr2=0x7fc0f81503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7fc0f6ffd640 1 --2- 172.21.15.57:0/2940178373 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f806c420 0x7fc0f81503c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7fc0f6ffd640 1 -- 172.21.15.57:0/2940178373 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ec0094d0 con 0x7fc0f806bb40 2023-10-30T14:14:12.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7f44c6ffd640 1 --2- 172.21.15.57:0/3194022917 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f44b4066500 0x7f44b40689c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7f44c6ffd640 1 -- 172.21.15.57:0/3194022917 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44c80ad300 con 0x7f44d014f910 2023-10-30T14:14:12.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7f44c6ffd640 1 --2- 172.21.15.57:0/3194022917 >> [v2:172.21.15.88:6800/2682449812,v1:172.21.15.88:6801/2682449812] conn(0x7f44b406f2b0 0x7f44b4071750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7fb5d5ffb640 1 -- 172.21.15.57:0/4006507787 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8009e40 con 0x7fb5e00c5560 2023-10-30T14:14:12.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7fb5d5ffb640 1 -- 172.21.15.57:0/4006507787 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d800f040 con 0x7fb5e00c5560 2023-10-30T14:14:12.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7fb5d5ffb640 1 -- 172.21.15.57:0/4006507787 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8015620 con 0x7fb5e00c5560 2023-10-30T14:14:12.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7fb5e60dc640 1 -- 172.21.15.57:0/4006507787 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e00b4d90 con 0x7fb5e00c5560 2023-10-30T14:14:12.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7fb5e60dc640 1 -- 172.21.15.57:0/4006507787 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e0147b00 con 0x7fb5e00c5560 2023-10-30T14:14:12.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7fc0f6ffd640 1 --2- 172.21.15.57:0/2940178373 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f806bb40 0x7fc0f814ba80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0aa000 tx=0x7fc0ec0192d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.664+0000 7f44d6130640 1 --2- 172.21.15.57:0/3194022917 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f44b4066500 0x7f44b40689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.668+0000 7f44c6ffd640 1 -- 172.21.15.57:0/3194022917 --> [v2:172.21.15.88:6800/2682449812,v1:172.21.15.88:6801/2682449812] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f44b4071e20 con 0x7f44b406f2b0 2023-10-30T14:14:12.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.668+0000 7f44d592f640 1 --2- 172.21.15.57:0/3194022917 >> [v2:172.21.15.88:6800/2682449812,v1:172.21.15.88:6801/2682449812] conn(0x7f44b406f2b0 0x7f44b4071750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.668+0000 7f44c6ffd640 1 -- 172.21.15.57:0/3194022917 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44c80f2bd0 con 0x7f44d014f910 2023-10-30T14:14:12.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.668+0000 7fb5e60dc640 1 -- 172.21.15.57:0/4006507787 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb5e0159c30 con 0x7fb5e00c5560 2023-10-30T14:14:12.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.668+0000 7f44d6130640 1 --2- 172.21.15.57:0/3194022917 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f44b4066500 0x7f44b40689c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00b810 tx=0x7f44cc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.668+0000 7f44d592f640 1 --2- 172.21.15.57:0/3194022917 >> [v2:172.21.15.88:6800/2682449812,v1:172.21.15.88:6801/2682449812] conn(0x7f44b406f2b0 0x7f44b4071750 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.668+0000 7fc0dffff640 1 -- 172.21.15.57:0/2940178373 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec0a9b40 con 0x7fc0f806bb40 2023-10-30T14:14:12.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.668+0000 7fc0dffff640 1 -- 172.21.15.57:0/2940178373 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ec0a9ca0 con 0x7fc0f806bb40 2023-10-30T14:14:12.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.668+0000 7fc0dffff640 1 -- 172.21.15.57:0/2940178373 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec003860 con 0x7fc0f806bb40 2023-10-30T14:14:12.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.668+0000 7f44c6ffd640 1 -- 172.21.15.57:0/3194022917 <== osd.5 v2:172.21.15.88:6800/2682449812 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f44b4071e20 con 0x7f44b406f2b0 2023-10-30T14:14:12.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.668+0000 7fc0fdbaa640 1 -- 172.21.15.57:0/2940178373 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f815d1a0 con 0x7fc0f806bb40 2023-10-30T14:14:12.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.672+0000 7fc0fdbaa640 1 -- 172.21.15.57:0/2940178373 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f815d760 con 0x7fc0f806bb40 2023-10-30T14:14:12.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.672+0000 7fb5d5ffb640 1 -- 172.21.15.57:0/4006507787 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb5d8015780 con 0x7fb5e00c5560 2023-10-30T14:14:12.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.672+0000 7fb5d5ffb640 1 --2- 172.21.15.57:0/4006507787 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb5c4066520 0x7fb5c40689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.672+0000 7fb5d5ffb640 1 -- 172.21.15.57:0/4006507787 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5d8084820 con 0x7fb5e00c5560 2023-10-30T14:14:12.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.672+0000 7fb5e48d9640 1 --2- 172.21.15.57:0/4006507787 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb5c4066520 0x7fb5c40689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7fb5d5ffb640 1 --2- 172.21.15.57:0/4006507787 >> [v2:172.21.15.88:6816/2862111199,v1:172.21.15.88:6817/2862111199] conn(0x7fb5c406e740 0x7fb5c4070b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7fb5d5ffb640 1 -- 172.21.15.57:0/4006507787 --> [v2:172.21.15.88:6816/2862111199,v1:172.21.15.88:6817/2862111199] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5c4071210 con 0x7fb5c406e740 2023-10-30T14:14:12.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7fc0fdbaa640 1 -- 172.21.15.57:0/2940178373 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc0f8071a90 con 0x7fc0f806bb40 2023-10-30T14:14:12.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7fb5d5ffb640 1 -- 172.21.15.57:0/4006507787 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5d8084b90 con 0x7fb5e00c5560 2023-10-30T14:14:12.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7fb5e48d9640 1 --2- 172.21.15.57:0/4006507787 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb5c4066520 0x7fb5c40689e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb5d00ae000 tx=0x7fb5d00a7000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.672+0000 7f8f8cedc640 1 -- 172.21.15.57:0/976685183 --> [v2:172.21.15.57:6808/3340797836,v1:172.21.15.57:6809/3340797836] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8f80047230 con 0x7f8f6c06e790 2023-10-30T14:14:12.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7f8f73fff640 1 -- 172.21.15.57:0/976685183 <== osd.1 v2:172.21.15.57:6808/3340797836 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8f7400c040 con 0x7f8f6c06e790 2023-10-30T14:14:12.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7fc0dffff640 1 -- 172.21.15.57:0/2940178373 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc0ec0b9070 con 0x7fc0f806bb40 2023-10-30T14:14:12.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7fb5e58db640 1 --2- 172.21.15.57:0/4006507787 >> [v2:172.21.15.88:6816/2862111199,v1:172.21.15.88:6817/2862111199] conn(0x7fb5c406e740 0x7fb5c4070b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7f8f8cedc640 1 -- 172.21.15.57:0/976685183 >> [v2:172.21.15.57:6808/3340797836,v1:172.21.15.57:6809/3340797836] conn(0x7f8f6c06e790 msgr2=0x7f8f6c070bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7f8f8cedc640 1 --2- 172.21.15.57:0/976685183 >> [v2:172.21.15.57:6808/3340797836,v1:172.21.15.57:6809/3340797836] conn(0x7f8f6c06e790 0x7f8f6c070bb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7f8f8cedc640 1 -- 172.21.15.57:0/976685183 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f6c066570 msgr2=0x7f8f6c068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7f8f8cedc640 1 --2- 172.21.15.57:0/976685183 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f6c066570 0x7f8f6c068a30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c0b2000 tx=0x7f8f7c0aa070 comp rx=0 tx=0).stop 2023-10-30T14:14:12.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7f8f8cedc640 1 -- 172.21.15.57:0/976685183 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f800e0dd0 msgr2=0x7f8f800baea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7f8f8cedc640 1 --2- 172.21.15.57:0/976685183 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f800e0dd0 0x7f8f800baea0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8f880745b0 tx=0x7f8f88066e60 comp rx=0 tx=0).stop 2023-10-30T14:14:12.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7f8f87fff640 1 -- 172.21.15.57:0/976685183 reap_dead start 2023-10-30T14:14:12.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7f8f8cedc640 1 -- 172.21.15.57:0/976685183 shutdown_connections 2023-10-30T14:14:12.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7f8f8cedc640 1 -- 172.21.15.57:0/976685183 >> 172.21.15.57:0/976685183 conn(0x7f8f80050280 msgr2=0x7f8f800d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:12.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7f8f8cedc640 1 -- 172.21.15.57:0/976685183 shutdown_connections 2023-10-30T14:14:12.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7f8f8cedc640 1 -- 172.21.15.57:0/976685183 wait complete. 2023-10-30T14:14:12.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7f5f939f8640 1 -- 172.21.15.57:0/3796161123 --> [v2:172.21.15.57:6824/1700292,v1:172.21.15.57:6825/1700292] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5f8c15a6a0 con 0x7f5f7006e790 2023-10-30T14:14:12.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7fc0dffff640 1 --2- 172.21.15.57:0/2940178373 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0e40665a0 0x7fc0e4068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.676+0000 7fb5e58db640 1 --2- 172.21.15.57:0/4006507787 >> [v2:172.21.15.88:6816/2862111199,v1:172.21.15.88:6817/2862111199] conn(0x7fb5c406e740 0x7fb5c4070b60 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7fc0dffff640 1 -- 172.21.15.57:0/2940178373 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0ec11e750 con 0x7fc0f806bb40 2023-10-30T14:14:12.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7fc0f77fe640 1 --2- 172.21.15.57:0/2940178373 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0e40665a0 0x7fc0e4068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7f5f81ffb640 1 -- 172.21.15.57:0/3796161123 <== osd.2 v2:172.21.15.57:6824/1700292 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5f8800c040 con 0x7f5f7006e790 2023-10-30T14:14:12.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7fc0dffff640 1 --2- 172.21.15.57:0/2940178373 >> [v2:172.21.15.88:6819/3340693897,v1:172.21.15.88:6822/3340693897] conn(0x7fc0e406e7c0 0x7fc0e4070be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:12.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7f5f939f8640 1 -- 172.21.15.57:0/3796161123 >> [v2:172.21.15.57:6824/1700292,v1:172.21.15.57:6825/1700292] conn(0x7f5f7006e790 msgr2=0x7f5f70070bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7f5f939f8640 1 --2- 172.21.15.57:0/3796161123 >> [v2:172.21.15.57:6824/1700292,v1:172.21.15.57:6825/1700292] conn(0x7f5f7006e790 0x7f5f70070bb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7f5f939f8640 1 -- 172.21.15.57:0/3796161123 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5f70066570 msgr2=0x7f5f70068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7f5f939f8640 1 --2- 172.21.15.57:0/3796161123 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5f70066570 0x7f5f70068a30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5f840ae000 tx=0x7f5f840a7070 comp rx=0 tx=0).stop 2023-10-30T14:14:12.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7f5f939f8640 1 -- 172.21.15.57:0/3796161123 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5f8c0ca8e0 msgr2=0x7f5f8c1526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7f5f939f8640 1 --2- 172.21.15.57:0/3796161123 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5f8c0ca8e0 0x7f5f8c1526a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c00e960 tx=0x7f5f7c00ee20 comp rx=0 tx=0).stop 2023-10-30T14:14:12.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7f5f91f95640 1 -- 172.21.15.57:0/3796161123 reap_dead start 2023-10-30T14:14:12.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7fc0dffff640 1 -- 172.21.15.57:0/2940178373 --> [v2:172.21.15.88:6819/3340693897,v1:172.21.15.88:6822/3340693897] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0e4071290 con 0x7fc0e406e7c0 2023-10-30T14:14:12.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7fc0dffff640 1 -- 172.21.15.57:0/2940178373 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0ec120cf0 con 0x7fc0f806bb40 2023-10-30T14:14:12.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7fc0f7fff640 1 --2- 172.21.15.57:0/2940178373 >> [v2:172.21.15.88:6819/3340693897,v1:172.21.15.88:6822/3340693897] conn(0x7fc0e406e7c0 0x7fc0e4070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:12.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7fc0f77fe640 1 --2- 172.21.15.57:0/2940178373 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0e40665a0 0x7fc0e4068a60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0003b90 tx=0x7fc0f0009890 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7f5f939f8640 1 -- 172.21.15.57:0/3796161123 shutdown_connections 2023-10-30T14:14:12.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7f5f939f8640 1 -- 172.21.15.57:0/3796161123 >> 172.21.15.57:0/3796161123 conn(0x7f5f8c06a240 msgr2=0x7f5f8c0cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:12.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7f5f939f8640 1 -- 172.21.15.57:0/3796161123 shutdown_connections 2023-10-30T14:14:12.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7f5f939f8640 1 -- 172.21.15.57:0/3796161123 wait complete. 2023-10-30T14:14:12.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7fc0f7fff640 1 --2- 172.21.15.57:0/2940178373 >> [v2:172.21.15.88:6819/3340693897,v1:172.21.15.88:6822/3340693897] conn(0x7fc0e406e7c0 0x7fc0e4070be0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:12.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.680+0000 7fb5d5ffb640 1 -- 172.21.15.57:0/4006507787 <== osd.7 v2:172.21.15.88:6816/2862111199 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fb5c4071210 con 0x7fb5c406e740 2023-10-30T14:14:12.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.688+0000 7fc0dffff640 1 -- 172.21.15.57:0/2940178373 <== osd.6 v2:172.21.15.88:6819/3340693897 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fc0e4071290 con 0x7fc0e406e7c0 2023-10-30T14:14:12.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.692+0000 7f201d2ea640 1 -- 172.21.15.57:0/1287137320 --> [v2:172.21.15.57:6816/2893420631,v1:172.21.15.57:6817/2893420631] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f20181f2840 con 0x7f1ff806e720 2023-10-30T14:14:12.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f581493e640 1 -- 172.21.15.57:0/1032711669 --> [v2:172.21.15.57:6800/3287367848,v1:172.21.15.57:6801/3287367848] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f57f4002cc0 con 0x7f57f006e7c0 2023-10-30T14:14:12.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f3ff3668640 1 -- 172.21.15.57:0/3421946460 --> [v2:172.21.15.88:6805/2784816485,v1:172.21.15.88:6807/2784816485] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3fec15ec40 con 0x7f3fd006e640 2023-10-30T14:14:12.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f200ffff640 1 -- 172.21.15.57:0/1287137320 <== osd.3 v2:172.21.15.57:6816/2893420631 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f20181f2840 con 0x7f1ff806e720 2023-10-30T14:14:12.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f201d2ea640 1 -- 172.21.15.57:0/1287137320 >> [v2:172.21.15.57:6816/2893420631,v1:172.21.15.57:6817/2893420631] conn(0x7f1ff806e720 msgr2=0x7f1ff8070b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f201d2ea640 1 --2- 172.21.15.57:0/1287137320 >> [v2:172.21.15.57:6816/2893420631,v1:172.21.15.57:6817/2893420631] conn(0x7f1ff806e720 0x7f1ff8070b40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f3fe1ffb640 1 -- 172.21.15.57:0/3421946460 <== osd.4 v2:172.21.15.88:6805/2784816485 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3fe40af5b0 con 0x7f3fd006e640 2023-10-30T14:14:12.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f201d2ea640 1 -- 172.21.15.57:0/1287137320 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1ff8066500 msgr2=0x7f1ff80689c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f201d2ea640 1 --2- 172.21.15.57:0/1287137320 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1ff8066500 0x7f1ff80689c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f200801ca70 tx=0x7f20080a7070 comp rx=0 tx=0).stop 2023-10-30T14:14:12.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f201d2ea640 1 -- 172.21.15.57:0/1287137320 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2018153b10 msgr2=0x7f20181f2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f201d2ea640 1 --2- 172.21.15.57:0/1287137320 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2018153b10 0x7f20181f2b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f200000c430 tx=0x7f200000c8f0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f20177fe640 1 -- 172.21.15.57:0/1287137320 reap_dead start 2023-10-30T14:14:12.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f201d2ea640 1 -- 172.21.15.57:0/1287137320 shutdown_connections 2023-10-30T14:14:12.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f201d2ea640 1 -- 172.21.15.57:0/1287137320 >> 172.21.15.57:0/1287137320 conn(0x7f201806a240 msgr2=0x7f20180cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:12.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f3ff3668640 1 -- 172.21.15.57:0/3421946460 >> [v2:172.21.15.88:6805/2784816485,v1:172.21.15.88:6807/2784816485] conn(0x7f3fd006e640 msgr2=0x7f3fd0070a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f3ff3668640 1 --2- 172.21.15.57:0/3421946460 >> [v2:172.21.15.88:6805/2784816485,v1:172.21.15.88:6807/2784816485] conn(0x7f3fd006e640 0x7f3fd0070a60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f57feffd640 1 -- 172.21.15.57:0/1032711669 <== osd.0 v2:172.21.15.57:6800/3287367848 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f57f4002cc0 con 0x7f57f006e7c0 2023-10-30T14:14:12.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f201d2ea640 1 -- 172.21.15.57:0/1287137320 shutdown_connections 2023-10-30T14:14:12.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f201d2ea640 1 -- 172.21.15.57:0/1287137320 wait complete. 2023-10-30T14:14:12.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f3ff3668640 1 -- 172.21.15.57:0/3421946460 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3fd00664a0 msgr2=0x7f3fd0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f3ff3668640 1 --2- 172.21.15.57:0/3421946460 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3fd00664a0 0x7f3fd0068960 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3fec15d540 tx=0x7f3fdc008040 comp rx=0 tx=0).stop 2023-10-30T14:14:12.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f3ff3668640 1 -- 172.21.15.57:0/3421946460 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fec157c20 msgr2=0x7f3fec156e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f3ff3668640 1 --2- 172.21.15.57:0/3421946460 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fec157c20 0x7f3fec156e90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3fe800c450 tx=0x7f3fe800c910 comp rx=0 tx=0).stop 2023-10-30T14:14:12.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f581493e640 1 -- 172.21.15.57:0/1032711669 >> [v2:172.21.15.57:6800/3287367848,v1:172.21.15.57:6801/3287367848] conn(0x7f57f006e7c0 msgr2=0x7f57f0070be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f3ff1c05640 1 -- 172.21.15.57:0/3421946460 reap_dead start 2023-10-30T14:14:12.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f3ff3668640 1 -- 172.21.15.57:0/3421946460 shutdown_connections 2023-10-30T14:14:12.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f3ff3668640 1 -- 172.21.15.57:0/3421946460 >> 172.21.15.57:0/3421946460 conn(0x7f3fec06a240 msgr2=0x7f3fec06a620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:12.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.696+0000 7f581493e640 1 --2- 172.21.15.57:0/1032711669 >> [v2:172.21.15.57:6800/3287367848,v1:172.21.15.57:6801/3287367848] conn(0x7f57f006e7c0 0x7f57f0070be0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.700+0000 7f3ff3668640 1 -- 172.21.15.57:0/3421946460 shutdown_connections 2023-10-30T14:14:12.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.700+0000 7f3ff3668640 1 -- 172.21.15.57:0/3421946460 wait complete. 2023-10-30T14:14:12.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.700+0000 7f581493e640 1 -- 172.21.15.57:0/1032711669 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f57f00665a0 msgr2=0x7f57f0068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.700+0000 7f581493e640 1 --2- 172.21.15.57:0/1032711669 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f57f00665a0 0x7f57f0068a60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f58100c9b30 tx=0x7f5800009040 comp rx=0 tx=0).stop 2023-10-30T14:14:12.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.700+0000 7f581493e640 1 -- 172.21.15.57:0/1032711669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f581006b500 msgr2=0x7f5810154340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.700+0000 7f581493e640 1 --2- 172.21.15.57:0/1032711669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f581006b500 0x7f5810154340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f580401a9f0 tx=0x7f5804019b60 comp rx=0 tx=0).stop 2023-10-30T14:14:12.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.700+0000 7f580ed9d640 1 -- 172.21.15.57:0/1032711669 reap_dead start 2023-10-30T14:14:12.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.700+0000 7f581493e640 1 -- 172.21.15.57:0/1032711669 shutdown_connections 2023-10-30T14:14:12.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.700+0000 7f581493e640 1 -- 172.21.15.57:0/1032711669 >> 172.21.15.57:0/1032711669 conn(0x7f58100cac70 msgr2=0x7f58101629c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:12.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.700+0000 7f581493e640 1 -- 172.21.15.57:0/1032711669 shutdown_connections 2023-10-30T14:14:12.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.700+0000 7f581493e640 1 -- 172.21.15.57:0/1032711669 wait complete. 2023-10-30T14:14:12.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.700+0000 7f44d8394640 1 -- 172.21.15.57:0/3194022917 --> [v2:172.21.15.88:6800/2682449812,v1:172.21.15.88:6801/2682449812] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f44d00c5c50 con 0x7f44b406f2b0 2023-10-30T14:14:12.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.700+0000 7f44c6ffd640 1 -- 172.21.15.57:0/3194022917 <== osd.5 v2:172.21.15.88:6800/2682449812 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f44c000a980 con 0x7f44b406f2b0 2023-10-30T14:14:12.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.704+0000 7f44d8394640 1 -- 172.21.15.57:0/3194022917 >> [v2:172.21.15.88:6800/2682449812,v1:172.21.15.88:6801/2682449812] conn(0x7f44b406f2b0 msgr2=0x7f44b4071750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.704+0000 7f44d8394640 1 --2- 172.21.15.57:0/3194022917 >> [v2:172.21.15.88:6800/2682449812,v1:172.21.15.88:6801/2682449812] conn(0x7f44b406f2b0 0x7f44b4071750 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.704+0000 7f44d8394640 1 -- 172.21.15.57:0/3194022917 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f44b4066500 msgr2=0x7f44b40689c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.704+0000 7f44d8394640 1 --2- 172.21.15.57:0/3194022917 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f44b4066500 0x7f44b40689c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00b810 tx=0x7f44cc008040 comp rx=0 tx=0).stop 2023-10-30T14:14:12.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.704+0000 7f44d8394640 1 -- 172.21.15.57:0/3194022917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44d014f910 msgr2=0x7f44d013bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.704+0000 7f44d8394640 1 --2- 172.21.15.57:0/3194022917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44d014f910 0x7f44d013bb30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f44c801fa60 tx=0x7f44c801bf60 comp rx=0 tx=0).stop 2023-10-30T14:14:12.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.704+0000 7f44d6931640 1 -- 172.21.15.57:0/3194022917 reap_dead start 2023-10-30T14:14:12.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.704+0000 7f44d8394640 1 -- 172.21.15.57:0/3194022917 shutdown_connections 2023-10-30T14:14:12.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.704+0000 7f44d8394640 1 -- 172.21.15.57:0/3194022917 >> 172.21.15.57:0/3194022917 conn(0x7f44d00cac70 msgr2=0x7f44d006fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:12.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.704+0000 7f44d8394640 1 -- 172.21.15.57:0/3194022917 shutdown_connections 2023-10-30T14:14:12.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.704+0000 7f44d8394640 1 -- 172.21.15.57:0/3194022917 wait complete. 2023-10-30T14:14:12.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.708+0000 7fb5e60dc640 1 -- 172.21.15.57:0/4006507787 --> [v2:172.21.15.88:6816/2862111199,v1:172.21.15.88:6817/2862111199] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb5e00c4d60 con 0x7fb5c406e740 2023-10-30T14:14:12.811 INFO:teuthology.orchestra.run.smithi057.stdout:55834574856 2023-10-30T14:14:12.811 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T14:14:12.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.712+0000 7fb5d5ffb640 1 -- 172.21.15.57:0/4006507787 <== osd.7 v2:172.21.15.88:6816/2862111199 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb5dc004b80 con 0x7fb5c406e740 2023-10-30T14:14:12.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fb5e60dc640 1 -- 172.21.15.57:0/4006507787 >> [v2:172.21.15.88:6816/2862111199,v1:172.21.15.88:6817/2862111199] conn(0x7fb5c406e740 msgr2=0x7fb5c4070b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fb5e60dc640 1 --2- 172.21.15.57:0/4006507787 >> [v2:172.21.15.88:6816/2862111199,v1:172.21.15.88:6817/2862111199] conn(0x7fb5c406e740 0x7fb5c4070b60 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.712+0000 7fc0fdbaa640 1 -- 172.21.15.57:0/2940178373 --> [v2:172.21.15.88:6819/3340693897,v1:172.21.15.88:6822/3340693897] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc0f80717b0 con 0x7fc0e406e7c0 2023-10-30T14:14:12.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fb5e60dc640 1 -- 172.21.15.57:0/4006507787 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb5c4066520 msgr2=0x7fb5c40689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fb5e60dc640 1 --2- 172.21.15.57:0/4006507787 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb5c4066520 0x7fb5c40689e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb5d00ae000 tx=0x7fb5d00a7000 comp rx=0 tx=0).stop 2023-10-30T14:14:12.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fb5e60dc640 1 -- 172.21.15.57:0/4006507787 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb5e00c5560 msgr2=0x7fb5e0148300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fb5e60dc640 1 --2- 172.21.15.57:0/4006507787 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb5e00c5560 0x7fb5e0148300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb5d80070b0 tx=0x7fb5d800d4a0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fb5e58db640 1 -- 172.21.15.57:0/4006507787 reap_dead start 2023-10-30T14:14:12.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fb5e60dc640 1 -- 172.21.15.57:0/4006507787 shutdown_connections 2023-10-30T14:14:12.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fb5e60dc640 1 -- 172.21.15.57:0/4006507787 >> 172.21.15.57:0/4006507787 conn(0x7fb5e006bf20 msgr2=0x7fb5e0158070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:12.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fb5e60dc640 1 -- 172.21.15.57:0/4006507787 shutdown_connections 2023-10-30T14:14:12.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fb5e60dc640 1 -- 172.21.15.57:0/4006507787 wait complete. 2023-10-30T14:14:12.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fc0dffff640 1 -- 172.21.15.57:0/2940178373 <== osd.6 v2:172.21.15.88:6819/3340693897 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc0e80072c0 con 0x7fc0e406e7c0 2023-10-30T14:14:12.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fc0fdbaa640 1 -- 172.21.15.57:0/2940178373 >> [v2:172.21.15.88:6819/3340693897,v1:172.21.15.88:6822/3340693897] conn(0x7fc0e406e7c0 msgr2=0x7fc0e4070be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.817 INFO:teuthology.orchestra.run.smithi057.stdout:55834574856 2023-10-30T14:14:12.817 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T14:14:12.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fc0fdbaa640 1 --2- 172.21.15.57:0/2940178373 >> [v2:172.21.15.88:6819/3340693897,v1:172.21.15.88:6822/3340693897] conn(0x7fc0e406e7c0 0x7fc0e4070be0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fc0fdbaa640 1 -- 172.21.15.57:0/2940178373 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0e40665a0 msgr2=0x7fc0e4068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fc0fdbaa640 1 --2- 172.21.15.57:0/2940178373 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0e40665a0 0x7fc0e4068a60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0003b90 tx=0x7fc0f0009890 comp rx=0 tx=0).stop 2023-10-30T14:14:12.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fc0fdbaa640 1 -- 172.21.15.57:0/2940178373 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f806bb40 msgr2=0x7fc0f814ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:12.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fc0fdbaa640 1 --2- 172.21.15.57:0/2940178373 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f806bb40 0x7fc0f814ba80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0aa000 tx=0x7fc0ec0192d0 comp rx=0 tx=0).stop 2023-10-30T14:14:12.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fc0f7fff640 1 -- 172.21.15.57:0/2940178373 reap_dead start 2023-10-30T14:14:12.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fc0fdbaa640 1 -- 172.21.15.57:0/2940178373 shutdown_connections 2023-10-30T14:14:12.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fc0fdbaa640 1 -- 172.21.15.57:0/2940178373 >> 172.21.15.57:0/2940178373 conn(0x7fc0f80cac70 msgr2=0x7fc0f8151530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:12.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fc0fdbaa640 1 -- 172.21.15.57:0/2940178373 shutdown_connections 2023-10-30T14:14:12.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:12.716+0000 7fc0fdbaa640 1 -- 172.21.15.57:0/2940178373 wait complete. 2023-10-30T14:14:12.821 INFO:teuthology.orchestra.run.smithi057.stdout:55834574857 2023-10-30T14:14:12.821 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T14:14:12.822 INFO:teuthology.orchestra.run.smithi057.stdout:55834574855 2023-10-30T14:14:12.822 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T14:14:12.823 INFO:teuthology.orchestra.run.smithi057.stdout:55834574856 2023-10-30T14:14:12.824 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T14:14:12.824 INFO:teuthology.orchestra.run.smithi057.stdout:55834574857 2023-10-30T14:14:12.825 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T14:14:12.826 INFO:teuthology.orchestra.run.smithi057.stdout:55834574856 2023-10-30T14:14:12.826 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T14:14:12.827 INFO:teuthology.orchestra.run.smithi057.stdout:55834574857 2023-10-30T14:14:12.827 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T14:14:13.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.092+0000 7fab8b162640 1 Processor -- start 2023-10-30T14:14:13.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.100+0000 7fab8b162640 1 -- start start 2023-10-30T14:14:13.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.100+0000 7fab8a160640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fab7c04b600 0x7fab7c005ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46316/0 (socket says 172.21.15.57:46316) 2023-10-30T14:14:13.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.100+0000 7fab8a160640 1 -- 172.21.15.57:0/2228118251 learned_addr learned my addr 172.21.15.57:0/2228118251 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:13.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.104+0000 7fab8b162640 1 --2- 172.21.15.57:0/2228118251 >> v2:172.21.15.88:3300/0 conn(0x7fab7c0dad60 0x7fab7c04b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.104+0000 7fab8b162640 1 --2- 172.21.15.57:0/2228118251 >> v2:172.21.15.57:3300/0 conn(0x7fab7c034b40 0x7fab7c034f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.104+0000 7fab8995f640 1 --2- 172.21.15.57:0/2228118251 >> v2:172.21.15.88:3300/0 conn(0x7fab7c0dad60 0x7fab7c04b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.104+0000 7fab8b162640 1 -- 172.21.15.57:0/2228118251 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7c007750 con 0x7fab7c04b600 2023-10-30T14:14:13.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.104+0000 7fab8b162640 1 -- 172.21.15.57:0/2228118251 --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab7c04bf20 con 0x7fab7c034b40 2023-10-30T14:14:13.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.104+0000 7fab8b162640 1 -- 172.21.15.57:0/2228118251 --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab7c0354b0 con 0x7fab7c0dad60 2023-10-30T14:14:13.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.104+0000 7fab8915e640 1 -- 172.21.15.57:0/2228118251 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213521835 0 0) 0x7fab7c007750 con 0x7fab7c04b600 2023-10-30T14:14:13.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.104+0000 7fab8915e640 1 -- 172.21.15.57:0/2228118251 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab70003530 con 0x7fab7c04b600 2023-10-30T14:14:13.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.104+0000 7fab8915e640 1 -- 172.21.15.57:0/2228118251 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2337883649 0 0) 0x7fab70003530 con 0x7fab7c04b600 2023-10-30T14:14:13.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.104+0000 7fab8915e640 1 -- 172.21.15.57:0/2228118251 >> v2:172.21.15.88:3300/0 conn(0x7fab7c0dad60 msgr2=0x7fab7c04b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:13.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.104+0000 7fab8915e640 1 --2- 172.21.15.57:0/2228118251 >> v2:172.21.15.88:3300/0 conn(0x7fab7c0dad60 0x7fab7c04b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.104+0000 7fab8915e640 1 -- 172.21.15.57:0/2228118251 >> v2:172.21.15.57:3300/0 conn(0x7fab7c034b40 msgr2=0x7fab7c034f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:13.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.104+0000 7fab8915e640 1 --2- 172.21.15.57:0/2228118251 >> v2:172.21.15.57:3300/0 conn(0x7fab7c034b40 0x7fab7c034f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.104+0000 7fab8915e640 1 -- 172.21.15.57:0/2228118251 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c0068f0 con 0x7fab7c04b600 2023-10-30T14:14:13.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.108+0000 7fab8915e640 1 -- 172.21.15.57:0/2228118251 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fab74001680 con 0x7fab7c04b600 2023-10-30T14:14:13.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.108+0000 7fab8915e640 1 -- 172.21.15.57:0/2228118251 >> v1:172.21.15.57:6789/0 conn(0x7fab7c04b600 legacy=0x7fab7c005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.108+0000 7fab8915e640 1 --2- 172.21.15.57:0/2228118251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab70003980 0x7fab70003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.108+0000 7fab8915e640 1 --2- 172.21.15.57:0/2228118251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab700047d0 0x7fab70004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.108+0000 7fab8915e640 1 --2- 172.21.15.57:0/2228118251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab70005190 0x7fab7000fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.108+0000 7fab8915e640 1 -- 172.21.15.57:0/2228118251 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c04bf20 con 0x7fab70003980 2023-10-30T14:14:13.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.108+0000 7fab8915e640 1 -- 172.21.15.57:0/2228118251 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c0354b0 con 0x7fab700047d0 2023-10-30T14:14:13.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.108+0000 7fab8915e640 1 -- 172.21.15.57:0/2228118251 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab700104d0 con 0x7fab70005190 2023-10-30T14:14:13.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.108+0000 7fab8a160640 1 --2- 172.21.15.57:0/2228118251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab70003980 0x7fab70003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.108+0000 7fab8a961640 1 --2- 172.21.15.57:0/2228118251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab70005190 0x7fab7000fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.108+0000 7fab8a961640 1 -- 172.21.15.57:0/2228118251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab700047d0 msgr2=0x7fab70004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:13.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.108+0000 7fab8995f640 1 --2- 172.21.15.57:0/2228118251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab700047d0 0x7fab70004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.112+0000 7fab8a961640 1 --2- 172.21.15.57:0/2228118251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab700047d0 0x7fab70004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.112+0000 7f7fe7d1d640 1 Processor -- start 2023-10-30T14:14:13.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.112+0000 7fab8a961640 1 -- 172.21.15.57:0/2228118251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab70003980 msgr2=0x7fab70003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:13.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.112+0000 7fab8a961640 1 --2- 172.21.15.57:0/2228118251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab70003980 0x7fab70003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.112+0000 7fab8a961640 1 -- 172.21.15.57:0/2228118251 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab70003530 con 0x7fab70005190 2023-10-30T14:14:13.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.112+0000 7fab8a160640 1 --2- 172.21.15.57:0/2228118251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab70003980 0x7fab70003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:14:13.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.112+0000 7fab8995f640 1 --2- 172.21.15.57:0/2228118251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab700047d0 0x7fab70004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:14:13.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.112+0000 7f7fe7d1d640 1 -- start start 2023-10-30T14:14:13.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.112+0000 7f7fe7d1d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7fe0071d50 0x7f7fe006fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.112+0000 7f7fe5ab9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7fe0071d50 0x7f7fe006fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.112+0000 7f7fe5ab9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7fe0071d50 0x7f7fe006fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:44942/0 (socket says 172.21.15.57:44942) 2023-10-30T14:14:13.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.112+0000 7f7fe52b8640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7fe00c7070 0x7f7fe00c7b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:52606/0 (socket says 172.21.15.57:52606) 2023-10-30T14:14:13.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.112+0000 7f7fe5ab9640 1 -- 172.21.15.57:0/4118674314 learned_addr learned my addr 172.21.15.57:0/4118674314 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:13.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.112+0000 7f7fe7d1d640 1 --2- 172.21.15.57:0/4118674314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fe0159920 0x7f7fe0156240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe7d1d640 1 -- 172.21.15.57:0/4118674314 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe0159490 con 0x7f7fe00c7070 2023-10-30T14:14:13.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7fab8a961640 1 -- 172.21.15.57:0/2228118251 reap_dead start 2023-10-30T14:14:13.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe62ba640 1 --2- 172.21.15.57:0/4118674314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fe0159920 0x7f7fe0156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7fab8a961640 1 --2- 172.21.15.57:0/2228118251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab70005190 0x7fab7000fda0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fab8404efc0 tx=0x7fab840972b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=409c6606b56c1a40 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe7d1d640 1 -- 172.21.15.57:0/4118674314 --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe00c67f0 con 0x7f7fe0071d50 2023-10-30T14:14:13.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7fab8915e640 1 -- 172.21.15.57:0/2228118251 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84126640 con 0x7fab70005190 2023-10-30T14:14:13.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7fab8915e640 1 -- 172.21.15.57:0/2228118251 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab84122050 con 0x7fab70005190 2023-10-30T14:14:13.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe4ab7640 1 -- 172.21.15.57:0/4118674314 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322905819 0 0) 0x7f7fe0159490 con 0x7f7fe00c7070 2023-10-30T14:14:13.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe7d1d640 1 -- 172.21.15.57:0/4118674314 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe00c6230 con 0x7f7fe0159920 2023-10-30T14:14:13.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe4ab7640 1 -- 172.21.15.57:0/4118674314 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fcc003530 con 0x7f7fe00c7070 2023-10-30T14:14:13.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7fab8b162640 1 -- 172.21.15.57:0/2228118251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab70005190 msgr2=0x7fab7000fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7fab8b162640 1 --2- 172.21.15.57:0/2228118251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab70005190 0x7fab7000fda0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fab8404efc0 tx=0x7fab840972b0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe4ab7640 1 -- 172.21.15.57:0/4118674314 <== mon.1 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1129649975 0 0) 0x7f7fcc003530 con 0x7f7fe00c7070 2023-10-30T14:14:13.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe4ab7640 1 -- 172.21.15.57:0/4118674314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fe0159920 msgr2=0x7f7fe0156240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:13.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe4ab7640 1 --2- 172.21.15.57:0/4118674314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fe0159920 0x7f7fe0156240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe4ab7640 1 -- 172.21.15.57:0/4118674314 >> v2:172.21.15.88:3300/0 conn(0x7f7fe0071d50 msgr2=0x7f7fe006fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:13.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7fab8b162640 1 -- 172.21.15.57:0/2228118251 shutdown_connections 2023-10-30T14:14:13.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7fab8b162640 1 --2- 172.21.15.57:0/2228118251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab70005190 0x7fab7000fda0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7fab8b162640 1 -- 172.21.15.57:0/2228118251 >> 172.21.15.57:0/2228118251 conn(0x7fab7c050280 msgr2=0x7fab7c050690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:13.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe4ab7640 1 --2- 172.21.15.57:0/4118674314 >> v2:172.21.15.88:3300/0 conn(0x7f7fe0071d50 0x7f7fe006fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe4ab7640 1 -- 172.21.15.57:0/4118674314 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe00c6510 con 0x7f7fe00c7070 2023-10-30T14:14:13.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe4ab7640 1 -- 172.21.15.57:0/4118674314 <== mon.1 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f7fdc001680 con 0x7f7fe00c7070 2023-10-30T14:14:13.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe4ab7640 1 -- 172.21.15.57:0/4118674314 >> v1:172.21.15.88:6789/0 conn(0x7f7fe00c7070 legacy=0x7f7fe00c7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe4ab7640 1 --2- 172.21.15.57:0/4118674314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fcc003980 0x7f7fcc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe4ab7640 1 --2- 172.21.15.57:0/4118674314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fe0159920 0x7f7fcc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe4ab7640 1 --2- 172.21.15.57:0/4118674314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fcc004370 0x7f7fcc004d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe4ab7640 1 -- 172.21.15.57:0/4118674314 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe00c67f0 con 0x7f7fcc004370 2023-10-30T14:14:13.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe4ab7640 1 -- 172.21.15.57:0/4118674314 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe00c6230 con 0x7f7fcc003980 2023-10-30T14:14:13.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7fab8b162640 1 -- 172.21.15.57:0/2228118251 shutdown_connections 2023-10-30T14:14:13.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7fab8b162640 1 -- 172.21.15.57:0/2228118251 wait complete. 2023-10-30T14:14:13.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe4ab7640 1 -- 172.21.15.57:0/4118674314 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc00bdf0 con 0x7f7fe0159920 2023-10-30T14:14:13.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.116+0000 7f7fe52b8640 1 --2- 172.21.15.57:0/4118674314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fe0159920 0x7f7fcc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7f7fe5ab9640 1 --2- 172.21.15.57:0/4118674314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fcc003980 0x7f7fcc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7f7fe62ba640 1 --2- 172.21.15.57:0/4118674314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fcc004370 0x7f7fcc004d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7f7fe52b8640 1 -- 172.21.15.57:0/4118674314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fcc003980 msgr2=0x7f7fcc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:13.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7f7fe52b8640 1 --2- 172.21.15.57:0/4118674314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fcc003980 0x7f7fcc003da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7f7fe52b8640 1 -- 172.21.15.57:0/4118674314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fcc004370 msgr2=0x7f7fcc004d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:13.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7f7fe52b8640 1 --2- 172.21.15.57:0/4118674314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fcc004370 0x7f7fcc004d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7f7fe52b8640 1 -- 172.21.15.57:0/4118674314 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe00c6510 con 0x7f7fe0159920 2023-10-30T14:14:13.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7f7fe62ba640 1 --2- 172.21.15.57:0/4118674314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fcc004370 0x7f7fcc004d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:14:13.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7f4840bb8640 1 Processor -- start 2023-10-30T14:14:13.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7f4840bb8640 1 -- start start 2023-10-30T14:14:13.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7f4840bb8640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f483c153bb0 0x7f483c0c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7f4840bb8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f483c159980 0x7f483c159d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7f4840bb8640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483c0c5f80 con 0x7f483c06c420 2023-10-30T14:14:13.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7f4840bb8640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f483c0c5690 con 0x7f483c153bb0 2023-10-30T14:14:13.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7f4840bb8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f483c15a2d0 con 0x7f483c159980 2023-10-30T14:14:13.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7f483ad9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f483c159980 0x7f483c159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7f483ad9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f483c159980 0x7f483c159d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:44956/0 (socket says 172.21.15.57:44956) 2023-10-30T14:14:13.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.120+0000 7f483ad9d640 1 -- 172.21.15.57:0/3977473974 learned_addr learned my addr 172.21.15.57:0/3977473974 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:13.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7f483a59c640 1 --2- 172.21.15.57:0/3977473974 >> v2:172.21.15.57:3300/0 conn(0x7f483c153bb0 0x7f483c0c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7ff28ffff640 1 Processor -- start 2023-10-30T14:14:13.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7f483ad9d640 1 -- 172.21.15.57:0/3977473974 >> v2:172.21.15.57:3300/0 conn(0x7f483c153bb0 msgr2=0x7f483c0c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:13.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7ff28ffff640 1 -- start start 2023-10-30T14:14:13.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7f483ad9d640 1 --2- 172.21.15.57:0/3977473974 >> v2:172.21.15.57:3300/0 conn(0x7f483c153bb0 0x7f483c0c6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7f483ad9d640 1 -- 172.21.15.57:0/3977473974 >> v1:172.21.15.57:6789/0 conn(0x7f483c06c420 legacy=0x7f483c0c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:13.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7fab8b162640 1 Processor -- start 2023-10-30T14:14:13.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7f983e33b640 1 Processor -- start 2023-10-30T14:14:13.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7fab8b162640 1 -- start start 2023-10-30T14:14:13.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7f7fe52b8640 1 --2- 172.21.15.57:0/4118674314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fe0159920 0x7f7fcc0047d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc01d270 tx=0x7f7fdc02dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e357bb219c486ed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7faba1ac9640 1 Processor -- start 2023-10-30T14:14:13.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7f983e33b640 1 -- start start 2023-10-30T14:14:13.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7ff28ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff2880e24f0 0x7ff288046860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7f205ab0f640 1 Processor -- start 2023-10-30T14:14:13.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7f205ab0f640 1 -- start start 2023-10-30T14:14:13.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7ff28ffff640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff288045850 con 0x7ff2880e28d0 2023-10-30T14:14:13.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f7fe4ab7640 1 -- 172.21.15.57:0/4118674314 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc0b6990 con 0x7f7fe0159920 2023-10-30T14:14:13.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f7fe4ab7640 1 -- 172.21.15.57:0/4118674314 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fdc0b6b10 con 0x7f7fe0159920 2023-10-30T14:14:13.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f7fe4ab7640 1 -- 172.21.15.57:0/4118674314 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc0070d0 con 0x7f7fe0159920 2023-10-30T14:14:13.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7faba1ac9640 1 -- start start 2023-10-30T14:14:13.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7faba1ac9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab9c14fb70 0x7fab9c06fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7faba1ac9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fab9c0c7110 0x7fab9c0c6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7ff28ffff640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff28804b3f0 con 0x7ff2880e2e00 2023-10-30T14:14:13.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7ff28ffff640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff288045aa0 con 0x7ff2880e24f0 2023-10-30T14:14:13.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7ff28effd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff2880e24f0 0x7ff288046860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f205ab0f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f20540cb790 0x7f2054155860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7f983e33b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9838153a20 0x7f98380bc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7f983e33b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f98380c5460 0x7f983806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7f983e33b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98380bcdc0 con 0x7f98380be140 2023-10-30T14:14:13.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7f983e33b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98380bcaf0 con 0x7f98380c5460 2023-10-30T14:14:13.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7f983e33b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98380bd0a0 con 0x7f9838153a20 2023-10-30T14:14:13.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab8b162640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab70005190 0x7fab7c0d9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7faba1ac9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab9c14ff50 con 0x7fab9c06b500 2023-10-30T14:14:13.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7faba1ac9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab9c0c6030 con 0x7fab9c0c7110 2023-10-30T14:14:13.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7faba1ac9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c0c6d10 con 0x7fab9c14fb70 2023-10-30T14:14:13.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab8b162640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab7c0d0e90 0x7fab7c0c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab8b162640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab7c04e080 0x7fab7c0c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab8a160640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab70005190 0x7fab7c0d9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab8995f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab7c0d0e90 0x7fab7c0c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab8a160640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab70005190 0x7fab7c0d9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:44970/0 (socket says 172.21.15.57:44970) 2023-10-30T14:14:13.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab8a160640 1 -- 172.21.15.57:0/1339665521 learned_addr learned my addr 172.21.15.57:0/1339665521 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:13.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab8b162640 1 -- 172.21.15.57:0/1339665521 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c0d4b80 con 0x7fab7c04e080 2023-10-30T14:14:13.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab8b162640 1 -- 172.21.15.57:0/1339665521 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c0ce310 con 0x7fab70005190 2023-10-30T14:14:13.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab8b162640 1 -- 172.21.15.57:0/1339665521 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c0ce5f0 con 0x7fab7c0d0e90 2023-10-30T14:14:13.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f7fe7d1d640 1 -- 172.21.15.57:0/4118674314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fe0159920 msgr2=0x7f7fcc0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f7fe7d1d640 1 --2- 172.21.15.57:0/4118674314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fe0159920 0x7f7fcc0047d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc01d270 tx=0x7f7fdc02dc90 comp rx=0 tx=0).stop 2023-10-30T14:14:13.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f7fe62ba640 1 -- 172.21.15.57:0/4118674314 reap_dead start 2023-10-30T14:14:13.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f7fe7d1d640 1 -- 172.21.15.57:0/4118674314 shutdown_connections 2023-10-30T14:14:13.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f7fe7d1d640 1 -- 172.21.15.57:0/4118674314 >> 172.21.15.57:0/4118674314 conn(0x7f7fe006d810 msgr2=0x7f7fe006dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:13.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f7fe7d1d640 1 -- 172.21.15.57:0/4118674314 shutdown_connections 2023-10-30T14:14:13.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.124+0000 7f483ad9d640 1 -- 172.21.15.57:0/3977473974 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c15a580 con 0x7f483c159980 2023-10-30T14:14:13.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7ff28f7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff2880e28d0 0x7ff2880d8b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46342/0 (socket says 172.21.15.57:46342) 2023-10-30T14:14:13.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7ff28f7fe640 1 -- 172.21.15.57:0/3986777076 learned_addr learned my addr 172.21.15.57:0/3986777076 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:13.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f205ab0f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20540c50d0 con 0x7f20540c6ff0 2023-10-30T14:14:13.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f205ab0f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20541567b0 con 0x7f20540c6c10 2023-10-30T14:14:13.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f205ab0f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20540c5a70 con 0x7f20540cb790 2023-10-30T14:14:13.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f20588ab640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f20540c6c10 0x7f20540c5e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:52618/0 (socket says 172.21.15.57:52618) 2023-10-30T14:14:13.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f20588ab640 1 -- 172.21.15.57:0/1528921870 learned_addr learned my addr 172.21.15.57:0/1528921870 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:13.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f20590ac640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f20540c6ff0 0x7f205414f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46358/0 (socket says 172.21.15.57:46358) 2023-10-30T14:14:13.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f983db3a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f98380c5460 0x7f983806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f983db3a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f98380c5460 0x7f983806d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:44986/0 (socket says 172.21.15.57:44986) 2023-10-30T14:14:13.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f983db3a640 1 -- 172.21.15.57:0/2211037747 learned_addr learned my addr 172.21.15.57:0/2211037747 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:13.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab8995f640 1 -- 172.21.15.57:0/1339665521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab70005190 msgr2=0x7fab7c0d9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab8995f640 1 --2- 172.21.15.57:0/1339665521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab70005190 0x7fab7c0d9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab8995f640 1 -- 172.21.15.57:0/1339665521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab7c04e080 msgr2=0x7fab7c0c9770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T14:14:13.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab8995f640 1 --2- 172.21.15.57:0/1339665521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab7c04e080 0x7fab7c0c9770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f7fe7d1d640 1 -- 172.21.15.57:0/4118674314 wait complete. 2023-10-30T14:14:13.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7ff28effd640 1 -- 172.21.15.57:0/3986777076 >> v1:172.21.15.88:6789/0 conn(0x7ff2880e2e00 legacy=0x7ff28804c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7ff28effd640 1 -- 172.21.15.57:0/3986777076 >> v1:172.21.15.57:6789/0 conn(0x7ff2880e28d0 legacy=0x7ff2880d8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7ff28effd640 1 -- 172.21.15.57:0/3986777076 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff288045d80 con 0x7ff2880e24f0 2023-10-30T14:14:13.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f982bfff640 1 -- 172.21.15.57:0/2211037747 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454964365 0 0) 0x7f98380bcdc0 con 0x7f98380be140 2023-10-30T14:14:13.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab8995f640 1 -- 172.21.15.57:0/1339665521 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c0ce8d0 con 0x7fab7c0d0e90 2023-10-30T14:14:13.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f982bfff640 1 -- 172.21.15.57:0/2211037747 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9824003530 con 0x7f98380be140 2023-10-30T14:14:13.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f983db3a640 1 -- 172.21.15.57:0/2211037747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9838153a20 msgr2=0x7f98380bc5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T14:14:13.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f983db3a640 1 --2- 172.21.15.57:0/2211037747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9838153a20 0x7f98380bc5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f983db3a640 1 -- 172.21.15.57:0/2211037747 >> v1:172.21.15.57:6789/0 conn(0x7f98380be140 legacy=0x7f983815a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab8a160640 1 --2- 172.21.15.57:0/1339665521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab70005190 0x7fab7c0d9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:14:13.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f983db3a640 1 -- 172.21.15.57:0/2211037747 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98380bd380 con 0x7f98380c5460 2023-10-30T14:14:13.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7fab8995f640 1 --2- 172.21.15.57:0/1339665521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab7c0d0e90 0x7fab7c0c9230 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fab8000dae0 tx=0x7fab8000dee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab9affd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fab9c0c7110 0x7fab9c0c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab9affd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fab9c0c7110 0x7fab9c0c6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38074/0 (socket says 172.21.15.57:38074) 2023-10-30T14:14:13.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab9affd640 1 -- 172.21.15.57:0/2685672074 learned_addr learned my addr 172.21.15.57:0/2685672074 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:13.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab9b7fe640 1 --2- 172.21.15.57:0/2685672074 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab9c14fb70 0x7fab9c06fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab9b7fe640 1 -- 172.21.15.57:0/2685672074 >> v2:172.21.15.57:3300/0 conn(0x7fab9c0c7110 msgr2=0x7fab9c0c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:13.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7fab9a7fc640 1 -- 172.21.15.57:0/2685672074 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272689994 0 0) 0x7fab9c14ff50 con 0x7fab9c06b500 2023-10-30T14:14:13.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7fab7affd640 1 -- 172.21.15.57:0/1339665521 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab80012070 con 0x7fab7c0d0e90 2023-10-30T14:14:13.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f2053fff640 1 --2- 172.21.15.57:0/1528921870 >> v2:172.21.15.88:3300/0 conn(0x7f20540cb790 0x7f2054155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f20537fe640 1 -- 172.21.15.57:0/1528921870 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132084354 0 0) 0x7f20541567b0 con 0x7f20540c6c10 2023-10-30T14:14:13.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f20537fe640 1 -- 172.21.15.57:0/1528921870 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2040003530 con 0x7f20540c6c10 2023-10-30T14:14:13.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f20537fe640 1 -- 172.21.15.57:0/1528921870 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562106333 0 0) 0x7f20540c50d0 con 0x7f20540c6ff0 2023-10-30T14:14:13.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f20537fe640 1 -- 172.21.15.57:0/1528921870 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20541567b0 con 0x7f20540c6ff0 2023-10-30T14:14:13.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f20537fe640 1 -- 172.21.15.57:0/1528921870 <== mon.1 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 445476806 0 0) 0x7f2040003530 con 0x7f20540c6c10 2023-10-30T14:14:13.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f20537fe640 1 -- 172.21.15.57:0/1528921870 >> v2:172.21.15.88:3300/0 conn(0x7f20540cb790 msgr2=0x7f2054155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:13.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f20537fe640 1 --2- 172.21.15.57:0/1528921870 >> v2:172.21.15.88:3300/0 conn(0x7f20540cb790 0x7f2054155860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f20537fe640 1 -- 172.21.15.57:0/1528921870 >> v1:172.21.15.57:6789/0 conn(0x7f20540c6ff0 legacy=0x7f205414f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f20537fe640 1 -- 172.21.15.57:0/1528921870 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2054155f10 con 0x7f20540c6c10 2023-10-30T14:14:13.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.128+0000 7f20537fe640 1 -- 172.21.15.57:0/1528921870 <== mon.1 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f203c001680 con 0x7f20540c6c10 2023-10-30T14:14:13.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f20537fe640 1 -- 172.21.15.57:0/1528921870 >> v1:172.21.15.88:6789/0 conn(0x7f20540c6c10 legacy=0x7f20540c5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f20537fe640 1 --2- 172.21.15.57:0/1528921870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2040003b60 0x7f2040003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f20537fe640 1 --2- 172.21.15.57:0/1528921870 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20400047d0 0x7f2040004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f20537fe640 1 --2- 172.21.15.57:0/1528921870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20400052b0 0x7f204000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f20537fe640 1 -- 172.21.15.57:0/1528921870 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20540c5a70 con 0x7f20400047d0 2023-10-30T14:14:13.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f20537fe640 1 -- 172.21.15.57:0/1528921870 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204000c130 con 0x7f2040003b60 2023-10-30T14:14:13.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f20537fe640 1 -- 172.21.15.57:0/1528921870 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f204000c2f0 con 0x7f20400052b0 2023-10-30T14:14:13.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f7fe7d1d640 1 Processor -- start 2023-10-30T14:14:13.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f7fe7d1d640 1 -- start start 2023-10-30T14:14:13.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f483ad9d640 1 --2- 172.21.15.57:0/3977473974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f483c159980 0x7f483c159d90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f483c0c8b20 tx=0x7f483001ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ef6c5aa9a7cba61 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7fab7affd640 1 -- 172.21.15.57:0/1339665521 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab8000cdb0 con 0x7fab7c0d0e90 2023-10-30T14:14:13.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7fab8b162640 1 -- 172.21.15.57:0/1339665521 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab7c0cebb0 con 0x7fab7c0d0e90 2023-10-30T14:14:13.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7fab8b162640 1 -- 172.21.15.57:0/1339665521 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab7c0d4820 con 0x7fab7c0d0e90 2023-10-30T14:14:13.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7fab7affd640 1 -- 172.21.15.57:0/1339665521 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8000b040 con 0x7fab7c0d0e90 2023-10-30T14:14:13.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f7a78c09640 1 Processor -- start 2023-10-30T14:14:13.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f20590ac640 1 --2- 172.21.15.57:0/1528921870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20400052b0 0x7f204000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f20588ab640 1 --2- 172.21.15.57:0/1528921870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2040003b60 0x7f2040003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7fab9b7fe640 1 --2- 172.21.15.57:0/2685672074 >> v2:172.21.15.57:3300/0 conn(0x7fab9c0c7110 0x7fab9c0c6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7fab9b7fe640 1 -- 172.21.15.57:0/2685672074 >> v1:172.21.15.57:6789/0 conn(0x7fab9c06b500 legacy=0x7fab9c06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7fab9b7fe640 1 -- 172.21.15.57:0/2685672074 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab9c1563b0 con 0x7fab9c14fb70 2023-10-30T14:14:13.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7fab9affd640 1 --2- 172.21.15.57:0/2685672074 >> v2:172.21.15.57:3300/0 conn(0x7fab9c0c7110 0x7fab9c0c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:14:13.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f483959a640 1 -- 172.21.15.57:0/3977473974 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48300a89b0 con 0x7f483c159980 2023-10-30T14:14:13.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f483959a640 1 -- 172.21.15.57:0/3977473974 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48300a8b10 con 0x7f483c159980 2023-10-30T14:14:13.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f483959a640 1 -- 172.21.15.57:0/3977473974 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48300a8ea0 con 0x7f483c159980 2023-10-30T14:14:13.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f7fe7d1d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fe0159920 0x7f7fe01527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7ff28effd640 1 --2- 172.21.15.57:0/3986777076 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2880e24f0 0x7ff288046860 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff2840097c0 tx=0x7ff28401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e3484721b19ed0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f7fe7d1d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fe01566f0 0x7f7fe0153630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f7fe7d1d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fe014a540 0x7f7fe014a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f7fe7d1d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe01547a0 con 0x7f7fe0159920 2023-10-30T14:14:13.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f7fe7d1d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe0155400 con 0x7f7fe014a540 2023-10-30T14:14:13.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f7fe7d1d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe0155cf0 con 0x7f7fe01566f0 2023-10-30T14:14:13.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f20588ab640 1 -- 172.21.15.57:0/1528921870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20400052b0 msgr2=0x7f204000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:13.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f20588ab640 1 --2- 172.21.15.57:0/1528921870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20400052b0 0x7f204000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f7a78c09640 1 -- start start 2023-10-30T14:14:13.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7ff28dffb640 1 -- 172.21.15.57:0/3986777076 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2840ab650 con 0x7ff2880e24f0 2023-10-30T14:14:13.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7ff28dffb640 1 -- 172.21.15.57:0/3986777076 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2840ab7b0 con 0x7ff2880e24f0 2023-10-30T14:14:13.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7fe52b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fe01566f0 0x7f7fe0153630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7ff28dffb640 1 -- 172.21.15.57:0/3986777076 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2840ab650 con 0x7ff2880e24f0 2023-10-30T14:14:13.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f4840bb8640 1 -- 172.21.15.57:0/3977473974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f483c159980 msgr2=0x7f483c159d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f4840bb8640 1 --2- 172.21.15.57:0/3977473974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f483c159980 0x7f483c159d90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f483c0c8b20 tx=0x7f483001ecb0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7fe62ba640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fe014a540 0x7f7fe014a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7fe5ab9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fe0159920 0x7f7fe01527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7fe5ab9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fe0159920 0x7f7fe01527b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38094/0 (socket says 172.21.15.57:38094) 2023-10-30T14:14:13.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7fe5ab9640 1 -- 172.21.15.57:0/3773429842 learned_addr learned my addr 172.21.15.57:0/3773429842 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:13.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f20588ab640 1 -- 172.21.15.57:0/1528921870 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20400047d0 msgr2=0x7f2040004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:13.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f2053fff640 1 --2- 172.21.15.57:0/1528921870 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20400047d0 0x7f2040004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f20588ab640 1 --2- 172.21.15.57:0/1528921870 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20400047d0 0x7f2040004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.132+0000 7f20588ab640 1 -- 172.21.15.57:0/1528921870 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2040003530 con 0x7f2040003b60 2023-10-30T14:14:13.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f2053fff640 1 --2- 172.21.15.57:0/1528921870 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20400047d0 0x7f2040004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:14:13.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f20590ac640 1 -- 172.21.15.57:0/1528921870 reap_dead start 2023-10-30T14:14:13.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7fab8b162640 1 -- 172.21.15.57:0/1339665521 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab6c005260 con 0x7fab7c0d0e90 2023-10-30T14:14:13.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f4840bb8640 1 -- 172.21.15.57:0/3977473974 shutdown_connections 2023-10-30T14:14:13.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7ff28ffff640 1 -- 172.21.15.57:0/3986777076 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2880e24f0 msgr2=0x7ff288046860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7ff28ffff640 1 --2- 172.21.15.57:0/3986777076 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2880e24f0 0x7ff288046860 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff2840097c0 tx=0x7ff28401c470 comp rx=0 tx=0).stop 2023-10-30T14:14:13.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7fe5ab9640 1 -- 172.21.15.57:0/3773429842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fe01566f0 msgr2=0x7f7fe0153630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7fe5ab9640 1 --2- 172.21.15.57:0/3773429842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fe01566f0 0x7f7fe0153630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7fe5ab9640 1 -- 172.21.15.57:0/3773429842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fe014a540 msgr2=0x7f7fe014a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7fe5ab9640 1 --2- 172.21.15.57:0/3773429842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fe014a540 0x7f7fe014a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7fe5ab9640 1 -- 172.21.15.57:0/3773429842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe014e280 con 0x7f7fe0159920 2023-10-30T14:14:13.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7a78c09640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a74157cb0 0x7f7a740c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f4840bb8640 1 --2- 172.21.15.57:0/3977473974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f483c159980 0x7f483c159d90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f4840bb8640 1 --2- 172.21.15.57:0/3977473974 >> v2:172.21.15.57:3300/0 conn(0x7f483c153bb0 0x7f483c0c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f4840bb8640 1 -- 172.21.15.57:0/3977473974 >> 172.21.15.57:0/3977473974 conn(0x7f483c0cac70 msgr2=0x7f483c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:13.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7ff28ffff640 1 -- 172.21.15.57:0/3986777076 shutdown_connections 2023-10-30T14:14:13.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7ff28ffff640 1 --2- 172.21.15.57:0/3986777076 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2880e24f0 0x7ff288046860 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7ff28ffff640 1 -- 172.21.15.57:0/3986777076 >> 172.21.15.57:0/3986777076 conn(0x7ff288050170 msgr2=0x7ff288050580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:13.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7a78c09640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7a7406fc70 0x7f7a740c7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7fe5ab9640 1 --2- 172.21.15.57:0/3773429842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fe0159920 0x7f7fe01527b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000f850 tx=0x7f7fd000fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7ff28ffff640 1 -- 172.21.15.57:0/3986777076 shutdown_connections 2023-10-30T14:14:13.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7fd67fc640 1 -- 172.21.15.57:0/3773429842 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd000a1a0 con 0x7f7fe0159920 2023-10-30T14:14:13.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7fd67fc640 1 -- 172.21.15.57:0/3773429842 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd0010040 con 0x7f7fe0159920 2023-10-30T14:14:13.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7fd67fc640 1 -- 172.21.15.57:0/3773429842 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd000dcd0 con 0x7f7fe0159920 2023-10-30T14:14:13.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7a78c09640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7a740c7ad0 0x7f7a74161500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7a78c09640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a7415da80 con 0x7f7a740c7ad0 2023-10-30T14:14:13.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7a78c09640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a74161070 con 0x7f7a7406fc70 2023-10-30T14:14:13.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7a78c09640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7415dc90 con 0x7f7a74157cb0 2023-10-30T14:14:13.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7a72d9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7a740c7ad0 0x7f7a74161500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7a72d9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7a740c7ad0 0x7f7a74161500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38108/0 (socket says 172.21.15.57:38108) 2023-10-30T14:14:13.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7a72d9d640 1 -- 172.21.15.57:0/2292932136 learned_addr learned my addr 172.21.15.57:0/2292932136 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:13.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7a71d9b640 1 --2- 172.21.15.57:0/2292932136 >> v2:172.21.15.88:3300/0 conn(0x7f7a7406fc70 0x7f7a740c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f4840bb8640 1 -- 172.21.15.57:0/3977473974 shutdown_connections 2023-10-30T14:14:13.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f4840bb8640 1 -- 172.21.15.57:0/3977473974 wait complete. 2023-10-30T14:14:13.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f983db3a640 1 --2- 172.21.15.57:0/2211037747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98380c5460 0x7f983806d5d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9830009e50 tx=0x7f983001ca00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e40535c29407cdc0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7ff28ffff640 1 -- 172.21.15.57:0/3986777076 wait complete. 2023-10-30T14:14:13.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f982bfff640 1 -- 172.21.15.57:0/2211037747 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98300a57e0 con 0x7f98380c5460 2023-10-30T14:14:13.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f982bfff640 1 -- 172.21.15.57:0/2211037747 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98300a5940 con 0x7f98380c5460 2023-10-30T14:14:13.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f982bfff640 1 -- 172.21.15.57:0/2211037747 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98300a5cd0 con 0x7f98380c5460 2023-10-30T14:14:13.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7fe7d1d640 1 -- 172.21.15.57:0/3773429842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe01556a0 con 0x7f7fe0159920 2023-10-30T14:14:13.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7f7fe7d1d640 1 -- 172.21.15.57:0/3773429842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe014e560 con 0x7f7fe0159920 2023-10-30T14:14:13.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7f983e33b640 1 -- 172.21.15.57:0/2211037747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98380c5460 msgr2=0x7f983806d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7f983e33b640 1 --2- 172.21.15.57:0/2211037747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98380c5460 0x7f983806d5d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9830009e50 tx=0x7f983001ca00 comp rx=0 tx=0).stop 2023-10-30T14:14:13.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7fab9b7fe640 1 --2- 172.21.15.57:0/2685672074 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab9c14fb70 0x7fab9c06fdf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fab9c1500e0 tx=0x7fab8c01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bac72699e68f3c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7a72d9d640 1 -- 172.21.15.57:0/2292932136 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a74157cb0 msgr2=0x7f7a740c6ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T14:14:13.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7fab7affd640 1 -- 172.21.15.57:0/1339665521 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fab80004260 con 0x7fab7c0d0e90 2023-10-30T14:14:13.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7f983e33b640 1 -- 172.21.15.57:0/2211037747 shutdown_connections 2023-10-30T14:14:13.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7f983e33b640 1 --2- 172.21.15.57:0/2211037747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98380c5460 0x7f983806d5d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7f983e33b640 1 --2- 172.21.15.57:0/2211037747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9838153a20 0x7f98380bc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7f983e33b640 1 -- 172.21.15.57:0/2211037747 >> 172.21.15.57:0/2211037747 conn(0x7f98380c7560 msgr2=0x7f98380c7940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:13.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7f20588ab640 1 --2- 172.21.15.57:0/1528921870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2040003b60 0x7f2040003f40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f203c00fa20 tx=0x7f203c023b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da6699322210909d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7ff28ffff640 1 Processor -- start 2023-10-30T14:14:13.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7fab9a7fc640 1 -- 172.21.15.57:0/2685672074 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8c0ab600 con 0x7fab9c14fb70 2023-10-30T14:14:13.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7fab9a7fc640 1 -- 172.21.15.57:0/2685672074 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab8c0a7050 con 0x7fab9c14fb70 2023-10-30T14:14:13.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7fab9a7fc640 1 -- 172.21.15.57:0/2685672074 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8c0ab980 con 0x7fab9c14fb70 2023-10-30T14:14:13.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7a72d9d640 1 --2- 172.21.15.57:0/2292932136 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a74157cb0 0x7f7a740c6ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7a72d9d640 1 -- 172.21.15.57:0/2292932136 >> v2:172.21.15.88:3300/0 conn(0x7f7a7406fc70 msgr2=0x7f7a740c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:13.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7a72d9d640 1 --2- 172.21.15.57:0/2292932136 >> v2:172.21.15.88:3300/0 conn(0x7f7a7406fc70 0x7f7a740c7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.136+0000 7f7a72d9d640 1 -- 172.21.15.57:0/2292932136 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7415df70 con 0x7f7a740c7ad0 2023-10-30T14:14:13.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7ff28ffff640 1 -- start start 2023-10-30T14:14:13.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7ff28ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2880e24f0 0x7ff288031f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7ff28ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2880e28d0 0x7ff2881720e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7ff28ffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2880e2e00 0x7ff288172620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7ff28ffff640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff288035440 con 0x7ff2880e24f0 2023-10-30T14:14:13.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7ff28ffff640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff288179030 con 0x7ff2880e28d0 2023-10-30T14:14:13.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7ff28ffff640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff288179310 con 0x7ff2880e2e00 2023-10-30T14:14:13.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7ff28e7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2880e28d0 0x7ff2881720e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7ff28e7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2880e28d0 0x7ff2881720e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:45032/0 (socket says 172.21.15.57:45032) 2023-10-30T14:14:13.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7ff28e7fc640 1 -- 172.21.15.57:0/17891084 learned_addr learned my addr 172.21.15.57:0/17891084 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:13.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7ff28effd640 1 --2- 172.21.15.57:0/17891084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2880e24f0 0x7ff288031f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7ff28f7fe640 1 --2- 172.21.15.57:0/17891084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2880e2e00 0x7ff288172620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7ff28e7fc640 1 -- 172.21.15.57:0/17891084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2880e2e00 msgr2=0x7ff288172620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7ff28e7fc640 1 --2- 172.21.15.57:0/17891084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2880e2e00 0x7ff288172620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7ff28e7fc640 1 -- 172.21.15.57:0/17891084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2880e24f0 msgr2=0x7ff288031f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7fab7affd640 1 --2- 172.21.15.57:0/1339665521 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fab680664d0 0x7fab68068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7fab7affd640 1 -- 172.21.15.57:0/1339665521 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab80083910 con 0x7fab7c0d0e90 2023-10-30T14:14:13.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7fab8a160640 1 --2- 172.21.15.57:0/1339665521 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fab680664d0 0x7fab68068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7faba1ac9640 1 -- 172.21.15.57:0/2685672074 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab9c14fb70 msgr2=0x7fab9c06fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7faba1ac9640 1 --2- 172.21.15.57:0/2685672074 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab9c14fb70 0x7fab9c06fdf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fab9c1500e0 tx=0x7fab8c01c480 comp rx=0 tx=0).stop 2023-10-30T14:14:13.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7faba1ac9640 1 -- 172.21.15.57:0/2685672074 shutdown_connections 2023-10-30T14:14:13.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7faba1ac9640 1 --2- 172.21.15.57:0/2685672074 >> v2:172.21.15.57:3300/0 conn(0x7fab9c0c7110 0x7fab9c0c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7faba1ac9640 1 --2- 172.21.15.57:0/2685672074 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab9c14fb70 0x7fab9c06fdf0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7faba1ac9640 1 -- 172.21.15.57:0/2685672074 >> 172.21.15.57:0/2685672074 conn(0x7fab9c0cac70 msgr2=0x7fab9c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:13.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7faba1ac9640 1 -- 172.21.15.57:0/2685672074 shutdown_connections 2023-10-30T14:14:13.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7faba1ac9640 1 -- 172.21.15.57:0/2685672074 wait complete. 2023-10-30T14:14:13.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f7a72d9d640 1 --2- 172.21.15.57:0/2292932136 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a740c7ad0 0x7f7a74161500 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7a7406aea0 tx=0x7f7a6c01f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b6d3a0812459e91 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7f4840bb8640 1 Processor -- start 2023-10-30T14:14:13.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7f4840bb8640 1 -- start start 2023-10-30T14:14:13.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7f20537fe640 1 -- 172.21.15.57:0/1528921870 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c00f2d0 con 0x7f2040003b60 2023-10-30T14:14:13.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7f20537fe640 1 -- 172.21.15.57:0/1528921870 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203c0b1070 con 0x7f2040003b60 2023-10-30T14:14:13.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7f20537fe640 1 -- 172.21.15.57:0/1528921870 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c005b20 con 0x7f2040003b60 2023-10-30T14:14:13.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7f205ab0f640 1 -- 172.21.15.57:0/1528921870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2040003b60 msgr2=0x7f2040003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7f205ab0f640 1 --2- 172.21.15.57:0/1528921870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2040003b60 0x7f2040003f40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f203c00fa20 tx=0x7f203c023b40 comp rx=0 tx=0).stop 2023-10-30T14:14:13.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7f983e33b640 1 -- 172.21.15.57:0/2211037747 shutdown_connections 2023-10-30T14:14:13.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.140+0000 7f983e33b640 1 -- 172.21.15.57:0/2211037747 wait complete. 2023-10-30T14:14:13.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f4840bb8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f483c06c420 0x7f483c1fed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f4840bb8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f483c153bb0 0x7f483c1fcac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f4840bb8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f483c159980 0x7f483c1fd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7ff28e7fc640 1 --2- 172.21.15.57:0/17891084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2880e24f0 0x7ff288031f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f4840bb8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c123f30 con 0x7f483c06c420 2023-10-30T14:14:13.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f4840bb8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c1258e0 con 0x7f483c159980 2023-10-30T14:14:13.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f4840bb8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483c125040 con 0x7f483c153bb0 2023-10-30T14:14:13.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f205ab0f640 1 -- 172.21.15.57:0/1528921870 shutdown_connections 2023-10-30T14:14:13.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f205ab0f640 1 --2- 172.21.15.57:0/1528921870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2040003b60 0x7f2040003f40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f205ab0f640 1 -- 172.21.15.57:0/1528921870 >> 172.21.15.57:0/1528921870 conn(0x7f20540cac70 msgr2=0x7f20540cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:13.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f7fe7d1d640 1 -- 172.21.15.57:0/3773429842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc8005260 con 0x7f7fe0159920 2023-10-30T14:14:13.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f983e33b640 1 Processor -- start 2023-10-30T14:14:13.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f4839d9b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f483c153bb0 0x7f483c1fcac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7ff28e7fc640 1 -- 172.21.15.57:0/17891084 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff284009420 con 0x7ff2880e28d0 2023-10-30T14:14:13.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7ff28effd640 1 --2- 172.21.15.57:0/17891084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2880e24f0 0x7ff288031f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:14:13.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7fab8a160640 1 --2- 172.21.15.57:0/1339665521 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fab680664d0 0x7fab68068990 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fab740017b0 tx=0x7fab74001e50 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f4839d9b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f483c153bb0 0x7f483c1fcac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:47822/0 (socket says 172.21.15.57:47822) 2023-10-30T14:14:13.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f4839d9b640 1 -- 172.21.15.57:0/299554229 learned_addr learned my addr 172.21.15.57:0/299554229 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:13.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f483a59c640 1 --2- 172.21.15.57:0/299554229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f483c06c420 0x7f483c1fed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7ff28e7fc640 1 --2- 172.21.15.57:0/17891084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2880e28d0 0x7ff2881720e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff29004efc0 tx=0x7ff2900674c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f483ad9d640 1 --2- 172.21.15.57:0/299554229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f483c159980 0x7f483c1fd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7ff2777fe640 1 -- 172.21.15.57:0/17891084 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2900675f0 con 0x7ff2880e28d0 2023-10-30T14:14:13.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7ff2777fe640 1 -- 172.21.15.57:0/17891084 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff290072910 con 0x7ff2880e28d0 2023-10-30T14:14:13.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f205ab0f640 1 -- 172.21.15.57:0/1528921870 shutdown_connections 2023-10-30T14:14:13.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f205ab0f640 1 -- 172.21.15.57:0/1528921870 wait complete. 2023-10-30T14:14:13.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f7a7159a640 1 -- 172.21.15.57:0/2292932136 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c0af400 con 0x7f7a740c7ad0 2023-10-30T14:14:13.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f7fd67fc640 1 -- 172.21.15.57:0/3773429842 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7fd0015480 con 0x7f7fe0159920 2023-10-30T14:14:13.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f7fd67fc640 1 --2- 172.21.15.57:0/3773429842 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7fc4066570 0x7f7fc4068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f7fd67fc640 1 -- 172.21.15.57:0/3773429842 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7fd0084240 con 0x7f7fe0159920 2023-10-30T14:14:13.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7faba1ac9640 1 Processor -- start 2023-10-30T14:14:13.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f7a7159a640 1 -- 172.21.15.57:0/2292932136 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6c0ab050 con 0x7f7a740c7ad0 2023-10-30T14:14:13.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f7a7159a640 1 -- 172.21.15.57:0/2292932136 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c0af780 con 0x7f7a740c7ad0 2023-10-30T14:14:13.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f7a78c09640 1 -- 172.21.15.57:0/2292932136 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a740c7ad0 msgr2=0x7f7a74161500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f7a78c09640 1 --2- 172.21.15.57:0/2292932136 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a740c7ad0 0x7f7a74161500 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7a7406aea0 tx=0x7f7a6c01f1e0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f7a78c09640 1 -- 172.21.15.57:0/2292932136 shutdown_connections 2023-10-30T14:14:13.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f7a78c09640 1 --2- 172.21.15.57:0/2292932136 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a740c7ad0 0x7f7a74161500 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f7a78c09640 1 --2- 172.21.15.57:0/2292932136 >> v2:172.21.15.88:3300/0 conn(0x7f7a7406fc70 0x7f7a740c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f7a78c09640 1 --2- 172.21.15.57:0/2292932136 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a74157cb0 0x7f7a740c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f7a78c09640 1 -- 172.21.15.57:0/2292932136 >> 172.21.15.57:0/2292932136 conn(0x7f7a7406d810 msgr2=0x7f7a7406dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:13.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f4839d9b640 1 -- 172.21.15.57:0/299554229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f483c159980 msgr2=0x7f483c1fd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f4839d9b640 1 --2- 172.21.15.57:0/299554229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f483c159980 0x7f483c1fd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f4839d9b640 1 -- 172.21.15.57:0/299554229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f483c06c420 msgr2=0x7f483c1fed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f4839d9b640 1 --2- 172.21.15.57:0/299554229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f483c06c420 0x7f483c1fed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f4839d9b640 1 -- 172.21.15.57:0/299554229 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48300094d0 con 0x7f483c153bb0 2023-10-30T14:14:13.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f4839d9b640 1 --2- 172.21.15.57:0/299554229 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f483c153bb0 0x7f483c1fcac0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f482c00b410 tx=0x7f482c00b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7ff2777fe640 1 -- 172.21.15.57:0/17891084 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff290072cc0 con 0x7ff2880e28d0 2023-10-30T14:14:13.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7ff28ffff640 1 -- 172.21.15.57:0/17891084 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff288035950 con 0x7ff2880e28d0 2023-10-30T14:14:13.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.144+0000 7f983e33b640 1 -- start start 2023-10-30T14:14:13.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f7fe52b8640 1 --2- 172.21.15.57:0/3773429842 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7fc4066570 0x7f7fc4068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7faba1ac9640 1 -- start start 2023-10-30T14:14:13.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f7a78c09640 1 -- 172.21.15.57:0/2292932136 shutdown_connections 2023-10-30T14:14:13.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7ff28ffff640 1 -- 172.21.15.57:0/17891084 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff288035f10 con 0x7ff2880e28d0 2023-10-30T14:14:13.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7ff28ffff640 1 -- 172.21.15.57:0/17891084 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff288005280 con 0x7ff2880e28d0 2023-10-30T14:14:13.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f205ab0f640 1 Processor -- start 2023-10-30T14:14:13.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f205ab0f640 1 -- start start 2023-10-30T14:14:13.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f983e33b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98380c5460 0x7f98380c3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f983e33b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9838153a20 0x7f983814a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f983e33b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f983815cec0 0x7f983814af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f983e33b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983815c3b0 con 0x7f983815cec0 2023-10-30T14:14:13.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f983e33b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9838156aa0 con 0x7f9838153a20 2023-10-30T14:14:13.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f983e33b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98381f2440 con 0x7f98380c5460 2023-10-30T14:14:13.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f983d339640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98380c5460 0x7f98380c3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f983d339640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98380c5460 0x7f98380c3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:47824/0 (socket says 172.21.15.57:47824) 2023-10-30T14:14:13.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f983d339640 1 -- 172.21.15.57:0/1350425449 learned_addr learned my addr 172.21.15.57:0/1350425449 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:13.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f983d339640 1 -- 172.21.15.57:0/1350425449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9838153a20 msgr2=0x7f983814a9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-30T14:14:13.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f983d339640 1 --2- 172.21.15.57:0/1350425449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9838153a20 0x7f983814a9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f983d339640 1 -- 172.21.15.57:0/1350425449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f983815cec0 msgr2=0x7f983814af00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T14:14:13.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f983d339640 1 --2- 172.21.15.57:0/1350425449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f983815cec0 0x7f983814af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f983d339640 1 -- 172.21.15.57:0/1350425449 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9830009b30 con 0x7f98380c5460 2023-10-30T14:14:13.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f7a78c09640 1 -- 172.21.15.57:0/2292932136 wait complete. 2023-10-30T14:14:13.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f983d339640 1 --2- 172.21.15.57:0/1350425449 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98380c5460 0x7f98380c3cc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f982c00b410 tx=0x7f982c00b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f7fe52b8640 1 --2- 172.21.15.57:0/3773429842 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7fc4066570 0x7f7fc4068a30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0bb000 tx=0x7f7fdc001940 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7faba1ac9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab9c06b500 0x7fab9c149e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7faba1ac9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab9c0c7110 0x7fab9c1f41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7faba1ac9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab9c14fb70 0x7fab9c1f9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7fab9affd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab9c0c7110 0x7fab9c1f41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7fab9affd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab9c0c7110 0x7fab9c1f41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:47838/0 (socket says 172.21.15.57:47838) 2023-10-30T14:14:13.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7fab9affd640 1 -- 172.21.15.57:0/2590936048 learned_addr learned my addr 172.21.15.57:0/2590936048 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:13.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7fab9bfff640 1 --2- 172.21.15.57:0/2590936048 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab9c14fb70 0x7fab9c1f9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7faba1ac9640 1 -- 172.21.15.57:0/2590936048 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c1f7230 con 0x7fab9c14fb70 2023-10-30T14:14:13.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7faba1ac9640 1 -- 172.21.15.57:0/2590936048 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c14af20 con 0x7fab9c06b500 2023-10-30T14:14:13.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7faba1ac9640 1 -- 172.21.15.57:0/2590936048 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c152bb0 con 0x7fab9c0c7110 2023-10-30T14:14:13.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7fab9affd640 1 -- 172.21.15.57:0/2590936048 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab9c06b500 msgr2=0x7fab9c149e70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-30T14:14:13.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7ff2777fe640 1 -- 172.21.15.57:0/17891084 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff290091020 con 0x7ff2880e28d0 2023-10-30T14:14:13.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f205ab0f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2040003b60 0x7f20541ee350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f205ab0f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2054070630 0x7f20541445a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f205ab0f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2054144ae0 0x7f20541eaa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f205ab0f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20541ee890 con 0x7f2054144ae0 2023-10-30T14:14:13.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f205ab0f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205414c940 con 0x7f2040003b60 2023-10-30T14:14:13.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.148+0000 7f205ab0f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205414c660 con 0x7f2054070630 2023-10-30T14:14:13.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f20588ab640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2040003b60 0x7f20541ee350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f20588ab640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2040003b60 0x7f20541ee350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:45072/0 (socket says 172.21.15.57:45072) 2023-10-30T14:14:13.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7fab9affd640 1 --2- 172.21.15.57:0/2590936048 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab9c06b500 0x7fab9c149e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7fab9affd640 1 -- 172.21.15.57:0/2590936048 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab9c14fb70 msgr2=0x7fab9c1f9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7fab9affd640 1 --2- 172.21.15.57:0/2590936048 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab9c14fb70 0x7fab9c1f9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7fab9affd640 1 -- 172.21.15.57:0/2590936048 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab900094d0 con 0x7fab9c0c7110 2023-10-30T14:14:13.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f20588ab640 1 -- 172.21.15.57:0/2230179207 learned_addr learned my addr 172.21.15.57:0/2230179207 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:13.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f2053fff640 1 --2- 172.21.15.57:0/2230179207 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2054070630 0x7f20541445a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7fab9affd640 1 --2- 172.21.15.57:0/2590936048 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab9c0c7110 0x7fab9c1f41f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fab9000e930 tx=0x7fab9000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f20590ac640 1 --2- 172.21.15.57:0/2230179207 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2054144ae0 0x7f20541eaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f20588ab640 1 -- 172.21.15.57:0/2230179207 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2054070630 msgr2=0x7f20541445a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f20588ab640 1 --2- 172.21.15.57:0/2230179207 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2054070630 0x7f20541445a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f20588ab640 1 -- 172.21.15.57:0/2230179207 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2054144ae0 msgr2=0x7f20541eaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f20588ab640 1 --2- 172.21.15.57:0/2230179207 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2054144ae0 0x7f20541eaa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f20588ab640 1 -- 172.21.15.57:0/2230179207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20541596f0 con 0x7f2040003b60 2023-10-30T14:14:13.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7ff2777fe640 1 --2- 172.21.15.57:0/17891084 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff27c066520 0x7ff27c0689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7ff2777fe640 1 -- 172.21.15.57:0/17891084 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2900fe5c0 con 0x7ff2880e28d0 2023-10-30T14:14:13.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f7a78c09640 1 Processor -- start 2023-10-30T14:14:13.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f20588ab640 1 --2- 172.21.15.57:0/2230179207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2040003b60 0x7f20541ee350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f203c007040 tx=0x7f2040005800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7ff28effd640 1 --2- 172.21.15.57:0/17891084 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff27c066520 0x7ff27c0689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f7a78c09640 1 -- start start 2023-10-30T14:14:13.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f7a78c09640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a7406fc70 0x7f7a740c3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f7a78c09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a740c7ad0 0x7f7a740c2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f7a78c09640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a74157cb0 0x7f7a7414e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f7a78c09640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a741499b0 con 0x7f7a74157cb0 2023-10-30T14:14:13.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f7a78c09640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a740c57b0 con 0x7f7a740c7ad0 2023-10-30T14:14:13.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f7a78c09640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a740c5510 con 0x7f7a7406fc70 2023-10-30T14:14:13.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7ff28effd640 1 --2- 172.21.15.57:0/17891084 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff27c066520 0x7ff27c0689e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff2840af000 tx=0x7ff2840a7000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f7a7259c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a7406fc70 0x7f7a740c3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f7a72d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a74157cb0 0x7f7a7414e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f7a7259c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a7406fc70 0x7f7a740c3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:47858/0 (socket says 172.21.15.57:47858) 2023-10-30T14:14:13.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f7a7259c640 1 -- 172.21.15.57:0/3886654798 learned_addr learned my addr 172.21.15.57:0/3886654798 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:13.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.152+0000 7f7a71d9b640 1 --2- 172.21.15.57:0/3886654798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a740c7ad0 0x7f7a740c2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f7a7259c640 1 -- 172.21.15.57:0/3886654798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a740c7ad0 msgr2=0x7f7a740c2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f7a7259c640 1 --2- 172.21.15.57:0/3886654798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a740c7ad0 0x7f7a740c2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f7a7259c640 1 -- 172.21.15.57:0/3886654798 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a74157cb0 msgr2=0x7f7a7414e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f7a7259c640 1 --2- 172.21.15.57:0/3886654798 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a74157cb0 0x7f7a7414e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f7a7259c640 1 -- 172.21.15.57:0/3886654798 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c009b30 con 0x7f7a7406fc70 2023-10-30T14:14:13.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f7a72d9d640 1 --2- 172.21.15.57:0/3886654798 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a74157cb0 0x7f7a7414e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:14:13.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f7fd67fc640 1 -- 172.21.15.57:0/3773429842 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7fd00516a0 con 0x7f7fe0159920 2023-10-30T14:14:13.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f7a7259c640 1 --2- 172.21.15.57:0/3886654798 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a7406fc70 0x7f7a740c3860 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800b4a0 tx=0x7f7a6800b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f482affd640 1 -- 172.21.15.57:0/299554229 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c004020 con 0x7f483c153bb0 2023-10-30T14:14:13.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f9829ffb640 1 -- 172.21.15.57:0/1350425449 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c011830 con 0x7f98380c5460 2023-10-30T14:14:13.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f4840bb8640 1 -- 172.21.15.57:0/299554229 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483c1ff810 con 0x7f483c153bb0 2023-10-30T14:14:13.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f983e33b640 1 -- 172.21.15.57:0/1350425449 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9838156cd0 con 0x7f98380c5460 2023-10-30T14:14:13.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7fab83fff640 1 -- 172.21.15.57:0/2590936048 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9000ca00 con 0x7fab9c0c7110 2023-10-30T14:14:13.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f9829ffb640 1 -- 172.21.15.57:0/1350425449 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982c011990 con 0x7f98380c5460 2023-10-30T14:14:13.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7faba1ac9640 1 -- 172.21.15.57:0/2590936048 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab8c009420 con 0x7fab9c0c7110 2023-10-30T14:14:13.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7faba1ac9640 1 -- 172.21.15.57:0/2590936048 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab9c147b40 con 0x7fab9c0c7110 2023-10-30T14:14:13.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7fab83fff640 1 -- 172.21.15.57:0/2590936048 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9000cb60 con 0x7fab9c0c7110 2023-10-30T14:14:13.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7fab83fff640 1 -- 172.21.15.57:0/2590936048 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab90013470 con 0x7fab9c0c7110 2023-10-30T14:14:13.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f4840bb8640 1 -- 172.21.15.57:0/299554229 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483c125570 con 0x7f483c153bb0 2023-10-30T14:14:13.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f482affd640 1 -- 172.21.15.57:0/299554229 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482c004180 con 0x7f483c153bb0 2023-10-30T14:14:13.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f482affd640 1 -- 172.21.15.57:0/299554229 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c011630 con 0x7f483c153bb0 2023-10-30T14:14:13.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f9829ffb640 1 -- 172.21.15.57:0/1350425449 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c011d00 con 0x7f98380c5460 2023-10-30T14:14:13.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f983e33b640 1 -- 172.21.15.57:0/1350425449 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98381f2670 con 0x7f98380c5460 2023-10-30T14:14:13.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f7a62ffd640 1 -- 172.21.15.57:0/3886654798 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a68010610 con 0x7f7a7406fc70 2023-10-30T14:14:13.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f7a62ffd640 1 -- 172.21.15.57:0/3886654798 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a68010790 con 0x7f7a7406fc70 2023-10-30T14:14:13.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f7a62ffd640 1 -- 172.21.15.57:0/3886654798 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a68010b40 con 0x7f7a7406fc70 2023-10-30T14:14:13.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f7a78c09640 1 -- 172.21.15.57:0/3886654798 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a7414f820 con 0x7f7a7406fc70 2023-10-30T14:14:13.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f7a78c09640 1 -- 172.21.15.57:0/3886654798 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a7414fda0 con 0x7f7a7406fc70 2023-10-30T14:14:13.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.160+0000 7fab7affd640 1 -- 172.21.15.57:0/1339665521 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fab80050cf0 con 0x7fab7c0d0e90 2023-10-30T14:14:13.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.156+0000 7f4840bb8640 1 -- 172.21.15.57:0/299554229 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4820005260 con 0x7f483c153bb0 2023-10-30T14:14:13.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.160+0000 7fab81ffb640 1 -- 172.21.15.57:0/2590936048 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab9c04e170 con 0x7fab9c0c7110 2023-10-30T14:14:13.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.160+0000 7f983e33b640 1 -- 172.21.15.57:0/1350425449 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9818005260 con 0x7f98380c5460 2023-10-30T14:14:13.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.160+0000 7f7a78c09640 1 -- 172.21.15.57:0/3886654798 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a7404e170 con 0x7f7a7406fc70 2023-10-30T14:14:13.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.160+0000 7f9829ffb640 1 -- 172.21.15.57:0/1350425449 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f982c01a400 con 0x7f98380c5460 2023-10-30T14:14:13.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.160+0000 7f9829ffb640 1 --2- 172.21.15.57:0/1350425449 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f981c0664a0 0x7f981c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.160+0000 7f9829ffb640 1 -- 172.21.15.57:0/1350425449 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f982c083ee0 con 0x7f98380c5460 2023-10-30T14:14:13.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.160+0000 7f983cb38640 1 --2- 172.21.15.57:0/1350425449 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f981c0664a0 0x7f981c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.160+0000 7fab83fff640 1 -- 172.21.15.57:0/2590936048 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fab900135d0 con 0x7fab9c0c7110 2023-10-30T14:14:13.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.160+0000 7f482affd640 1 -- 172.21.15.57:0/299554229 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f482c011790 con 0x7f483c153bb0 2023-10-30T14:14:13.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7f983cb38640 1 --2- 172.21.15.57:0/1350425449 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f981c0664a0 0x7f981c068960 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f98380bc8b0 tx=0x7f98340078b0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7f7a62ffd640 1 -- 172.21.15.57:0/3886654798 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7a68010ca0 con 0x7f7a7406fc70 2023-10-30T14:14:13.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7fab83fff640 1 --2- 172.21.15.57:0/2590936048 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fab880664d0 0x7fab88068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7fab83fff640 1 -- 172.21.15.57:0/2590936048 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab90084330 con 0x7fab9c0c7110 2023-10-30T14:14:13.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7fab9b7fe640 1 --2- 172.21.15.57:0/2590936048 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fab880664d0 0x7fab88068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7f7a62ffd640 1 --2- 172.21.15.57:0/3886654798 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7a54066520 0x7f7a540689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7f482affd640 1 --2- 172.21.15.57:0/299554229 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f481c066530 0x7f481c0689f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7f7a62ffd640 1 -- 172.21.15.57:0/3886654798 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a68083f20 con 0x7f7a7406fc70 2023-10-30T14:14:13.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7f482affd640 1 -- 172.21.15.57:0/299554229 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f482c084480 con 0x7f483c153bb0 2023-10-30T14:14:13.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7f483a59c640 1 --2- 172.21.15.57:0/299554229 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f481c066530 0x7f481c0689f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7f7a71d9b640 1 --2- 172.21.15.57:0/3886654798 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7a54066520 0x7f7a540689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7fab9b7fe640 1 --2- 172.21.15.57:0/2590936048 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fab880664d0 0x7fab88068990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fab8c0ad000 tx=0x7fab8c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7f20517fa640 1 -- 172.21.15.57:0/2230179207 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c0be070 con 0x7f2040003b60 2023-10-30T14:14:13.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7f20517fa640 1 -- 172.21.15.57:0/2230179207 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203c005980 con 0x7f2040003b60 2023-10-30T14:14:13.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7f20517fa640 1 -- 172.21.15.57:0/2230179207 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c005d30 con 0x7f2040003b60 2023-10-30T14:14:13.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7f7a71d9b640 1 --2- 172.21.15.57:0/3886654798 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7a54066520 0x7f7a540689e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7a74161dc0 tx=0x7f7a64009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7f205ab0f640 1 -- 172.21.15.57:0/2230179207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20541599d0 con 0x7f2040003b60 2023-10-30T14:14:13.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7f205ab0f640 1 -- 172.21.15.57:0/2230179207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20541ea530 con 0x7f2040003b60 2023-10-30T14:14:13.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7f483a59c640 1 --2- 172.21.15.57:0/299554229 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f481c066530 0x7f481c0689f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f48340070b0 tx=0x7f4834007040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.164+0000 7ff2777fe640 1 -- 172.21.15.57:0/17891084 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff2900cbb60 con 0x7ff2880e28d0 2023-10-30T14:14:13.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.168+0000 7f205ab0f640 1 -- 172.21.15.57:0/2230179207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2054144ec0 con 0x7f2040003b60 2023-10-30T14:14:13.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.172+0000 7f20517fa640 1 -- 172.21.15.57:0/2230179207 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f203c008120 con 0x7f2040003b60 2023-10-30T14:14:13.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.172+0000 7f20517fa640 1 --2- 172.21.15.57:0/2230179207 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20340664a0 0x7f2034068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:13.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.172+0000 7f20517fa640 1 -- 172.21.15.57:0/2230179207 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f203c124610 con 0x7f2040003b60 2023-10-30T14:14:13.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.172+0000 7f2053fff640 1 --2- 172.21.15.57:0/2230179207 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20340664a0 0x7f2034068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:13.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.172+0000 7f2053fff640 1 --2- 172.21.15.57:0/2230179207 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20340664a0 0x7f2034068960 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f205414c2b0 tx=0x7f2044007700 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:13.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.172+0000 7f482affd640 1 -- 172.21.15.57:0/299554229 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f482c051860 con 0x7f483c153bb0 2023-10-30T14:14:13.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.180+0000 7f20517fa640 1 -- 172.21.15.57:0/2230179207 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f203c0f1a90 con 0x7f2040003b60 2023-10-30T14:14:13.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.184+0000 7f9829ffb640 1 -- 172.21.15.57:0/1350425449 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f982c051360 con 0x7f98380c5460 2023-10-30T14:14:13.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.200+0000 7fab83fff640 1 -- 172.21.15.57:0/2590936048 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fab90051730 con 0x7fab9c0c7110 2023-10-30T14:14:13.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.208+0000 7f7a62ffd640 1 -- 172.21.15.57:0/3886654798 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7a68051430 con 0x7f7a7406fc70 2023-10-30T14:14:13.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.476+0000 7fab8b162640 1 -- 172.21.15.57:0/1339665521 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fab6c0050d0 con 0x7fab7c0d0e90 2023-10-30T14:14:13.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.476+0000 7fab7affd640 1 -- 172.21.15.57:0/1339665521 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fab8000a020 con 0x7fab7c0d0e90 2023-10-30T14:14:13.484 INFO:teuthology.orchestra.run.smithi057.stdout:55834574855 2023-10-30T14:14:13.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.480+0000 7fab8b162640 1 -- 172.21.15.57:0/1339665521 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fab680664d0 msgr2=0x7fab68068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.480+0000 7fab8b162640 1 --2- 172.21.15.57:0/1339665521 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fab680664d0 0x7fab68068990 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fab740017b0 tx=0x7fab74001e50 comp rx=0 tx=0).stop 2023-10-30T14:14:13.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.480+0000 7fab8b162640 1 -- 172.21.15.57:0/1339665521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab7c0d0e90 msgr2=0x7fab7c0c9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.480+0000 7fab8b162640 1 --2- 172.21.15.57:0/1339665521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab7c0d0e90 0x7fab7c0c9230 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fab8000dae0 tx=0x7fab8000dee0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.480+0000 7fab8b162640 1 -- 172.21.15.57:0/1339665521 shutdown_connections 2023-10-30T14:14:13.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.480+0000 7fab8b162640 1 --2- 172.21.15.57:0/1339665521 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fab680664d0 0x7fab68068990 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fab740017b0 tx=0x7fab74001e50 comp rx=0 tx=0).stop 2023-10-30T14:14:13.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.480+0000 7fab8b162640 1 --2- 172.21.15.57:0/1339665521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab7c04e080 0x7fab7c0c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.480+0000 7fab8b162640 1 --2- 172.21.15.57:0/1339665521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab7c0d0e90 0x7fab7c0c9230 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.480+0000 7fab8b162640 1 --2- 172.21.15.57:0/1339665521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab70005190 0x7fab7c0d9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.480+0000 7fab8b162640 1 -- 172.21.15.57:0/1339665521 >> 172.21.15.57:0/1339665521 conn(0x7fab7c050280 msgr2=0x7fab7c1658a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:13.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.480+0000 7fab8b162640 1 -- 172.21.15.57:0/1339665521 shutdown_connections 2023-10-30T14:14:13.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.480+0000 7fab8b162640 1 -- 172.21.15.57:0/1339665521 wait complete. 2023-10-30T14:14:13.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.488+0000 7f7fe7d1d640 1 -- 172.21.15.57:0/3773429842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f7fc80050d0 con 0x7f7fe0159920 2023-10-30T14:14:13.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.492+0000 7f7fd67fc640 1 -- 172.21.15.57:0/3773429842 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7fd0054cc0 con 0x7f7fe0159920 2023-10-30T14:14:13.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.492+0000 7ff28ffff640 1 -- 172.21.15.57:0/17891084 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff288000e80 con 0x7ff2880e28d0 2023-10-30T14:14:13.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.492+0000 7ff2777fe640 1 -- 172.21.15.57:0/17891084 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff290099020 con 0x7ff2880e28d0 2023-10-30T14:14:13.499 INFO:teuthology.orchestra.run.smithi057.stdout:55834574855 2023-10-30T14:14:13.499 INFO:teuthology.orchestra.run.smithi057.stdout:55834574856 2023-10-30T14:14:13.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.496+0000 7f7fe7d1d640 1 -- 172.21.15.57:0/3773429842 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7fc4066570 msgr2=0x7f7fc4068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.496+0000 7ff28ffff640 1 -- 172.21.15.57:0/17891084 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff27c066520 msgr2=0x7ff27c0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.496+0000 7ff28ffff640 1 --2- 172.21.15.57:0/17891084 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff27c066520 0x7ff27c0689e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff2840af000 tx=0x7ff2840a7000 comp rx=0 tx=0).stop 2023-10-30T14:14:13.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.496+0000 7ff28ffff640 1 -- 172.21.15.57:0/17891084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2880e28d0 msgr2=0x7ff2881720e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.496+0000 7ff28ffff640 1 --2- 172.21.15.57:0/17891084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2880e28d0 0x7ff2881720e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff29004efc0 tx=0x7ff2900674c0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.496+0000 7f7fe7d1d640 1 --2- 172.21.15.57:0/3773429842 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7fc4066570 0x7f7fc4068a30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0bb000 tx=0x7f7fdc001940 comp rx=0 tx=0).stop 2023-10-30T14:14:13.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.496+0000 7ff28ffff640 1 -- 172.21.15.57:0/17891084 shutdown_connections 2023-10-30T14:14:13.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.496+0000 7ff28ffff640 1 --2- 172.21.15.57:0/17891084 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff27c066520 0x7ff27c0689e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.496+0000 7ff28ffff640 1 --2- 172.21.15.57:0/17891084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2880e2e00 0x7ff288172620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.496+0000 7ff28ffff640 1 --2- 172.21.15.57:0/17891084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2880e28d0 0x7ff2881720e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.496+0000 7ff28ffff640 1 --2- 172.21.15.57:0/17891084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2880e24f0 0x7ff288031f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.496+0000 7ff28ffff640 1 -- 172.21.15.57:0/17891084 >> 172.21.15.57:0/17891084 conn(0x7ff288050170 msgr2=0x7ff288007080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:13.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.496+0000 7f7fe7d1d640 1 -- 172.21.15.57:0/3773429842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fe0159920 msgr2=0x7f7fe01527b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.496+0000 7f7fe7d1d640 1 --2- 172.21.15.57:0/3773429842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fe0159920 0x7f7fe01527b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000f850 tx=0x7f7fd000fd10 comp rx=0 tx=0).stop 2023-10-30T14:14:13.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.500+0000 7f7fe7d1d640 1 -- 172.21.15.57:0/3773429842 shutdown_connections 2023-10-30T14:14:13.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.500+0000 7f7fe7d1d640 1 --2- 172.21.15.57:0/3773429842 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7fc4066570 0x7f7fc4068a30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.500+0000 7f7fe7d1d640 1 --2- 172.21.15.57:0/3773429842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fe014a540 0x7f7fe014a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.500+0000 7f7fe7d1d640 1 --2- 172.21.15.57:0/3773429842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fe01566f0 0x7f7fe0153630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.500+0000 7f7fe7d1d640 1 --2- 172.21.15.57:0/3773429842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fe0159920 0x7f7fe01527b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.500+0000 7f7fe7d1d640 1 -- 172.21.15.57:0/3773429842 >> 172.21.15.57:0/3773429842 conn(0x7f7fe006d810 msgr2=0x7f7fe00c41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:13.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.500+0000 7f7fe7d1d640 1 -- 172.21.15.57:0/3773429842 shutdown_connections 2023-10-30T14:14:13.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.500+0000 7f7fe7d1d640 1 -- 172.21.15.57:0/3773429842 wait complete. 2023-10-30T14:14:13.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.500+0000 7ff28ffff640 1 -- 172.21.15.57:0/17891084 shutdown_connections 2023-10-30T14:14:13.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.500+0000 7ff28ffff640 1 -- 172.21.15.57:0/17891084 wait complete. 2023-10-30T14:14:13.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.500+0000 7f4840bb8640 1 -- 172.21.15.57:0/299554229 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f48200057e0 con 0x7f483c153bb0 2023-10-30T14:14:13.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.500+0000 7f983e33b640 1 -- 172.21.15.57:0/1350425449 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f98180057e0 con 0x7f98380c5460 2023-10-30T14:14:13.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.500+0000 7f482affd640 1 -- 172.21.15.57:0/299554229 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f482c015090 con 0x7f483c153bb0 2023-10-30T14:14:13.511 INFO:teuthology.orchestra.run.smithi057.stdout:55834574855 2023-10-30T14:14:13.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.500+0000 7f9829ffb640 1 -- 172.21.15.57:0/1350425449 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f982c015090 con 0x7f98380c5460 2023-10-30T14:14:13.512 INFO:teuthology.orchestra.run.smithi057.stdout:55834574856 2023-10-30T14:14:13.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f4840bb8640 1 -- 172.21.15.57:0/299554229 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f481c066530 msgr2=0x7f481c0689f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f4840bb8640 1 --2- 172.21.15.57:0/299554229 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f481c066530 0x7f481c0689f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f48340070b0 tx=0x7f4834007040 comp rx=0 tx=0).stop 2023-10-30T14:14:13.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f4840bb8640 1 -- 172.21.15.57:0/299554229 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f483c153bb0 msgr2=0x7f483c1fcac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f4840bb8640 1 --2- 172.21.15.57:0/299554229 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f483c153bb0 0x7f483c1fcac0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f482c00b410 tx=0x7f482c00b8d0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f4840bb8640 1 -- 172.21.15.57:0/299554229 shutdown_connections 2023-10-30T14:14:13.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f4840bb8640 1 --2- 172.21.15.57:0/299554229 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f481c066530 0x7f481c0689f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f4840bb8640 1 --2- 172.21.15.57:0/299554229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f483c159980 0x7f483c1fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f4840bb8640 1 --2- 172.21.15.57:0/299554229 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f483c153bb0 0x7f483c1fcac0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f482c00b410 tx=0x7f482c00b8d0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f4840bb8640 1 --2- 172.21.15.57:0/299554229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f483c06c420 0x7f483c1fed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f4840bb8640 1 -- 172.21.15.57:0/299554229 >> 172.21.15.57:0/299554229 conn(0x7f483c0cac70 msgr2=0x7f483c152da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:13.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f4840bb8640 1 -- 172.21.15.57:0/299554229 shutdown_connections 2023-10-30T14:14:13.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f4840bb8640 1 -- 172.21.15.57:0/299554229 wait complete. 2023-10-30T14:14:13.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f983e33b640 1 -- 172.21.15.57:0/1350425449 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f981c0664a0 msgr2=0x7f981c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f983e33b640 1 --2- 172.21.15.57:0/1350425449 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f981c0664a0 0x7f981c068960 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f98380bc8b0 tx=0x7f98340078b0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f983e33b640 1 -- 172.21.15.57:0/1350425449 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98380c5460 msgr2=0x7f98380c3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f983e33b640 1 --2- 172.21.15.57:0/1350425449 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98380c5460 0x7f98380c3cc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f982c00b410 tx=0x7f982c00b8d0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f983e33b640 1 -- 172.21.15.57:0/1350425449 shutdown_connections 2023-10-30T14:14:13.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f983e33b640 1 --2- 172.21.15.57:0/1350425449 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f981c0664a0 0x7f981c068960 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f983e33b640 1 --2- 172.21.15.57:0/1350425449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f983815cec0 0x7f983814af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f983e33b640 1 --2- 172.21.15.57:0/1350425449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9838153a20 0x7f983814a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f983e33b640 1 --2- 172.21.15.57:0/1350425449 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98380c5460 0x7f98380c3cc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.504+0000 7f983e33b640 1 -- 172.21.15.57:0/1350425449 >> 172.21.15.57:0/1350425449 conn(0x7f98380c7560 msgr2=0x7f98380733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:13.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.508+0000 7f983e33b640 1 -- 172.21.15.57:0/1350425449 shutdown_connections 2023-10-30T14:14:13.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.508+0000 7f983e33b640 1 -- 172.21.15.57:0/1350425449 wait complete. 2023-10-30T14:14:13.521 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.1 2023-10-30T14:14:13.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.516+0000 7f205ab0f640 1 -- 172.21.15.57:0/2230179207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f2054144ec0 con 0x7f2040003b60 2023-10-30T14:14:13.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.516+0000 7f20517fa640 1 -- 172.21.15.57:0/2230179207 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2054144ec0 con 0x7f2040003b60 2023-10-30T14:14:13.524 INFO:teuthology.orchestra.run.smithi057.stdout:55834574855 2023-10-30T14:14:13.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.520+0000 7f205ab0f640 1 -- 172.21.15.57:0/2230179207 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20340664a0 msgr2=0x7f2034068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.520+0000 7f205ab0f640 1 --2- 172.21.15.57:0/2230179207 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20340664a0 0x7f2034068960 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f205414c2b0 tx=0x7f2044007700 comp rx=0 tx=0).stop 2023-10-30T14:14:13.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.520+0000 7f205ab0f640 1 -- 172.21.15.57:0/2230179207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2040003b60 msgr2=0x7f20541ee350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.520+0000 7f205ab0f640 1 --2- 172.21.15.57:0/2230179207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2040003b60 0x7f20541ee350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f203c007040 tx=0x7f2040005800 comp rx=0 tx=0).stop 2023-10-30T14:14:13.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.520+0000 7f205ab0f640 1 -- 172.21.15.57:0/2230179207 shutdown_connections 2023-10-30T14:14:13.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.520+0000 7f205ab0f640 1 --2- 172.21.15.57:0/2230179207 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20340664a0 0x7f2034068960 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.520+0000 7f205ab0f640 1 --2- 172.21.15.57:0/2230179207 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2054144ae0 0x7f20541eaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.520+0000 7f205ab0f640 1 --2- 172.21.15.57:0/2230179207 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2054070630 0x7f20541445a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.520+0000 7f205ab0f640 1 --2- 172.21.15.57:0/2230179207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2040003b60 0x7f20541ee350 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.520+0000 7f205ab0f640 1 -- 172.21.15.57:0/2230179207 >> 172.21.15.57:0/2230179207 conn(0x7f20540cac70 msgr2=0x7f2054146760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:13.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.528+0000 7f205ab0f640 1 -- 172.21.15.57:0/2230179207 shutdown_connections 2023-10-30T14:14:13.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.528+0000 7f205ab0f640 1 -- 172.21.15.57:0/2230179207 wait complete. 2023-10-30T14:14:13.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.532+0000 7fab81ffb640 1 -- 172.21.15.57:0/2590936048 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fab9c147b40 con 0x7fab9c0c7110 2023-10-30T14:14:13.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.532+0000 7fab83fff640 1 -- 172.21.15.57:0/2590936048 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fab9c147b40 con 0x7fab9c0c7110 2023-10-30T14:14:13.539 INFO:teuthology.orchestra.run.smithi057.stdout:55834574856 2023-10-30T14:14:13.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.540+0000 7fab81ffb640 1 -- 172.21.15.57:0/2590936048 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fab880664d0 msgr2=0x7fab88068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.540+0000 7fab81ffb640 1 --2- 172.21.15.57:0/2590936048 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fab880664d0 0x7fab88068990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fab8c0ad000 tx=0x7fab8c008040 comp rx=0 tx=0).stop 2023-10-30T14:14:13.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.540+0000 7fab81ffb640 1 -- 172.21.15.57:0/2590936048 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab9c0c7110 msgr2=0x7fab9c1f41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.540+0000 7fab81ffb640 1 --2- 172.21.15.57:0/2590936048 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab9c0c7110 0x7fab9c1f41f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fab9000e930 tx=0x7fab9000edf0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.540+0000 7fab81ffb640 1 -- 172.21.15.57:0/2590936048 shutdown_connections 2023-10-30T14:14:13.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.540+0000 7fab81ffb640 1 --2- 172.21.15.57:0/2590936048 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fab880664d0 0x7fab88068990 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.540+0000 7fab81ffb640 1 --2- 172.21.15.57:0/2590936048 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab9c14fb70 0x7fab9c1f9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.540+0000 7fab81ffb640 1 --2- 172.21.15.57:0/2590936048 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab9c0c7110 0x7fab9c1f41f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.540+0000 7fab81ffb640 1 --2- 172.21.15.57:0/2590936048 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab9c06b500 0x7fab9c149e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.540+0000 7fab81ffb640 1 -- 172.21.15.57:0/2590936048 >> 172.21.15.57:0/2590936048 conn(0x7fab9c0cac70 msgr2=0x7fab9c071b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:13.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.540+0000 7f7a78c09640 1 -- 172.21.15.57:0/3886654798 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f7a740cb6d0 con 0x7f7a7406fc70 2023-10-30T14:14:13.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.540+0000 7fab81ffb640 1 -- 172.21.15.57:0/2590936048 shutdown_connections 2023-10-30T14:14:13.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.540+0000 7fab81ffb640 1 -- 172.21.15.57:0/2590936048 wait complete. 2023-10-30T14:14:13.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.540+0000 7f7a62ffd640 1 -- 172.21.15.57:0/3886654798 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7a68014020 con 0x7f7a7406fc70 2023-10-30T14:14:13.549 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.6 2023-10-30T14:14:13.549 INFO:teuthology.orchestra.run.smithi057.stdout:55834574854 2023-10-30T14:14:13.550 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.2 2023-10-30T14:14:13.551 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.0 2023-10-30T14:14:13.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.544+0000 7f7a78c09640 1 -- 172.21.15.57:0/3886654798 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7a54066520 msgr2=0x7f7a540689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.544+0000 7f7a78c09640 1 --2- 172.21.15.57:0/3886654798 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7a54066520 0x7f7a540689e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7a74161dc0 tx=0x7f7a64009190 comp rx=0 tx=0).stop 2023-10-30T14:14:13.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.544+0000 7f7a78c09640 1 -- 172.21.15.57:0/3886654798 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a7406fc70 msgr2=0x7f7a740c3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:13.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.544+0000 7f7a78c09640 1 --2- 172.21.15.57:0/3886654798 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a7406fc70 0x7f7a740c3860 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800b4a0 tx=0x7f7a6800b960 comp rx=0 tx=0).stop 2023-10-30T14:14:13.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.544+0000 7f7a78c09640 1 -- 172.21.15.57:0/3886654798 shutdown_connections 2023-10-30T14:14:13.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.544+0000 7f7a78c09640 1 --2- 172.21.15.57:0/3886654798 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7a54066520 0x7f7a540689e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.544+0000 7f7a78c09640 1 --2- 172.21.15.57:0/3886654798 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a74157cb0 0x7f7a7414e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.544+0000 7f7a78c09640 1 --2- 172.21.15.57:0/3886654798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a740c7ad0 0x7f7a740c2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.544+0000 7f7a78c09640 1 --2- 172.21.15.57:0/3886654798 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a7406fc70 0x7f7a740c3860 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:13.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.544+0000 7f7a78c09640 1 -- 172.21.15.57:0/3886654798 >> 172.21.15.57:0/3886654798 conn(0x7f7a7406d810 msgr2=0x7f7a740c5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:13.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.544+0000 7f7a78c09640 1 -- 172.21.15.57:0/3886654798 shutdown_connections 2023-10-30T14:14:13.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:13.544+0000 7f7a78c09640 1 -- 172.21.15.57:0/3886654798 wait complete. 2023-10-30T14:14:13.555 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.4 2023-10-30T14:14:13.560 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.7 2023-10-30T14:14:13.572 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.5 2023-10-30T14:14:13.574 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.3 2023-10-30T14:14:14.523 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T14:14:14.550 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T14:14:14.551 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T14:14:14.552 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T14:14:14.555 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T14:14:14.560 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T14:14:14.572 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T14:14:14.574 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T14:14:14.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.728+0000 7f37c8879640 1 Processor -- start 2023-10-30T14:14:14.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.728+0000 7f37c8879640 1 -- start start 2023-10-30T14:14:14.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.728+0000 7f37c8879640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37c4071d50 0x7f37c406fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.732+0000 7f37c8879640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f37c4159920 0x7f37c4156240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.732+0000 7f37c8879640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c4159490 con 0x7f37c40c7070 2023-10-30T14:14:14.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.732+0000 7f37c8879640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c40c67f0 con 0x7f37c4159920 2023-10-30T14:14:14.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.732+0000 7f37c8879640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c40c6230 con 0x7f37c4071d50 2023-10-30T14:14:14.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.732+0000 7f37b9d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f37c40c7070 0x7f37c40c7b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:46076/0 (socket says 172.21.15.57:46076) 2023-10-30T14:14:14.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.732+0000 7f37b9d9b640 1 -- 172.21.15.57:0/3011203545 learned_addr learned my addr 172.21.15.57:0/3011203545 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:14.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.732+0000 7f37c259c640 1 --2- 172.21.15.57:0/3011203545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37c4071d50 0x7f37c406fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.732+0000 7f37c1d9b640 1 -- 172.21.15.57:0/3011203545 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897351761 0 0) 0x7f37c4159490 con 0x7f37c40c7070 2023-10-30T14:14:14.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.732+0000 7f37c1d9b640 1 -- 172.21.15.57:0/3011203545 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37b0003530 con 0x7f37c40c7070 2023-10-30T14:14:14.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.732+0000 7f37c2d9d640 1 --2- 172.21.15.57:0/3011203545 >> v2:172.21.15.57:3300/0 conn(0x7f37c4159920 0x7f37c4156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.732+0000 7f37c1d9b640 1 -- 172.21.15.57:0/3011203545 <== mon.1 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4223379158 0 0) 0x7f37b0003530 con 0x7f37c40c7070 2023-10-30T14:14:14.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.732+0000 7f37c1d9b640 1 -- 172.21.15.57:0/3011203545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37c4071d50 msgr2=0x7f37c406fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:14.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.732+0000 7f37c1d9b640 1 --2- 172.21.15.57:0/3011203545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37c4071d50 0x7f37c406fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.732+0000 7f37c1d9b640 1 -- 172.21.15.57:0/3011203545 >> v2:172.21.15.57:3300/0 conn(0x7f37c4159920 msgr2=0x7f37c4156240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:14.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.732+0000 7f37c1d9b640 1 --2- 172.21.15.57:0/3011203545 >> v2:172.21.15.57:3300/0 conn(0x7f37c4159920 0x7f37c4156240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.732+0000 7f37c1d9b640 1 -- 172.21.15.57:0/3011203545 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c40c6510 con 0x7f37c40c7070 2023-10-30T14:14:14.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.732+0000 7f37c1d9b640 1 -- 172.21.15.57:0/3011203545 <== mon.1 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f37ac001680 con 0x7f37c40c7070 2023-10-30T14:14:14.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.732+0000 7f37c1d9b640 1 -- 172.21.15.57:0/3011203545 >> v1:172.21.15.88:6789/0 conn(0x7f37c40c7070 legacy=0x7f37c40c7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.736+0000 7f37c1d9b640 1 --2- 172.21.15.57:0/3011203545 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b0003770 0x7f37b0003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.736+0000 7f37c1d9b640 1 --2- 172.21.15.57:0/3011203545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37c4071d50 0x7f37b00040d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.736+0000 7f37c1d9b640 1 --2- 172.21.15.57:0/3011203545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37b000b020 0x7f37b000d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.736+0000 7f37c1d9b640 1 -- 172.21.15.57:0/3011203545 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c40c67f0 con 0x7f37b0003770 2023-10-30T14:14:14.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.736+0000 7f37c1d9b640 1 -- 172.21.15.57:0/3011203545 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c40c6230 con 0x7f37b000b020 2023-10-30T14:14:14.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.736+0000 7f37c1d9b640 1 -- 172.21.15.57:0/3011203545 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b000dbf0 con 0x7f37c4071d50 2023-10-30T14:14:14.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.736+0000 7f37b9d9b640 1 --2- 172.21.15.57:0/3011203545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37c4071d50 0x7f37b00040d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.736+0000 7f37c259c640 1 --2- 172.21.15.57:0/3011203545 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b0003770 0x7f37b0003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.736+0000 7f37c2d9d640 1 --2- 172.21.15.57:0/3011203545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37b000b020 0x7f37b000d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.736+0000 7f37c259c640 1 -- 172.21.15.57:0/3011203545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37c4071d50 msgr2=0x7f37b00040d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:14.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.736+0000 7f37c259c640 1 --2- 172.21.15.57:0/3011203545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37c4071d50 0x7f37b00040d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.736+0000 7f37c259c640 1 -- 172.21.15.57:0/3011203545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37b000b020 msgr2=0x7f37b000d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:14.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.736+0000 7f37c259c640 1 --2- 172.21.15.57:0/3011203545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37b000b020 0x7f37b000d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.736+0000 7f37c259c640 1 -- 172.21.15.57:0/3011203545 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c40c6510 con 0x7f37b0003770 2023-10-30T14:14:14.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.740+0000 7f37c259c640 1 --2- 172.21.15.57:0/3011203545 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b0003770 0x7f37b0003b90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f37b4009fc0 tx=0x7f37b4025470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b79cb21a42936cb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.740+0000 7f37c1d9b640 1 -- 172.21.15.57:0/3011203545 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b40b5410 con 0x7f37b0003770 2023-10-30T14:14:14.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.740+0000 7f37c1d9b640 1 -- 172.21.15.57:0/3011203545 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b400f040 con 0x7f37b0003770 2023-10-30T14:14:14.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.744+0000 7f37c8879640 1 -- 172.21.15.57:0/3011203545 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b0003770 msgr2=0x7f37b0003b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.744+0000 7f37c1d9b640 1 -- 172.21.15.57:0/3011203545 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b40b5410 con 0x7f37b0003770 2023-10-30T14:14:14.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.744+0000 7f37c8879640 1 --2- 172.21.15.57:0/3011203545 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b0003770 0x7f37b0003b90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f37b4009fc0 tx=0x7f37b4025470 comp rx=0 tx=0).stop 2023-10-30T14:14:14.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.744+0000 7f37c2d9d640 1 -- 172.21.15.57:0/3011203545 reap_dead start 2023-10-30T14:14:14.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.744+0000 7f37c8879640 1 -- 172.21.15.57:0/3011203545 shutdown_connections 2023-10-30T14:14:14.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.744+0000 7f37c8879640 1 -- 172.21.15.57:0/3011203545 >> 172.21.15.57:0/3011203545 conn(0x7f37c406d810 msgr2=0x7f37c406dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:14.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.744+0000 7f37c8879640 1 -- 172.21.15.57:0/3011203545 shutdown_connections 2023-10-30T14:14:14.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.744+0000 7f37c8879640 1 -- 172.21.15.57:0/3011203545 wait complete. 2023-10-30T14:14:14.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.748+0000 7f37c8879640 1 Processor -- start 2023-10-30T14:14:14.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.752+0000 7f37c8879640 1 -- start start 2023-10-30T14:14:14.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.752+0000 7f37c8879640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b0003770 0x7f37c4148a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.752+0000 7f37c8879640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37c41eb190 0x7f37c41ee420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.752+0000 7f37c8879640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37c40709f0 0x7f37c40c24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.752+0000 7f37c8879640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c40c3c70 con 0x7f37b0003770 2023-10-30T14:14:14.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.752+0000 7f37c8879640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c4148f50 con 0x7f37c40709f0 2023-10-30T14:14:14.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.752+0000 7f37c8879640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c41ea4a0 con 0x7f37c41eb190 2023-10-30T14:14:14.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.752+0000 7f37c2d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37c40709f0 0x7f37c40c24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.752+0000 7f37c2d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37c40709f0 0x7f37c40c24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:49174/0 (socket says 172.21.15.57:49174) 2023-10-30T14:14:14.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.752+0000 7f37c2d9d640 1 -- 172.21.15.57:0/3704121704 learned_addr learned my addr 172.21.15.57:0/3704121704 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:14.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.752+0000 7f37c2d9d640 1 -- 172.21.15.57:0/3704121704 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37c41eb190 msgr2=0x7f37c41ee420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T14:14:14.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.752+0000 7f37c2d9d640 1 --2- 172.21.15.57:0/3704121704 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37c41eb190 0x7f37c41ee420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.752+0000 7f37c2d9d640 1 -- 172.21.15.57:0/3704121704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b0003770 msgr2=0x7f37c4148a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T14:14:14.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.752+0000 7f37c2d9d640 1 --2- 172.21.15.57:0/3704121704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b0003770 0x7f37c4148a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.752+0000 7f37c2d9d640 1 -- 172.21.15.57:0/3704121704 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c41ee960 con 0x7f37c40709f0 2023-10-30T14:14:14.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.760+0000 7f37c2d9d640 1 --2- 172.21.15.57:0/3704121704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37c40709f0 0x7f37c40c24b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f37c40c89f0 tx=0x7f37bc00f8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.760+0000 7f37bb7fe640 1 -- 172.21.15.57:0/3704121704 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc004440 con 0x7f37c40709f0 2023-10-30T14:14:14.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.760+0000 7f37bb7fe640 1 -- 172.21.15.57:0/3704121704 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37bc003620 con 0x7f37c40709f0 2023-10-30T14:14:14.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.760+0000 7f37bb7fe640 1 -- 172.21.15.57:0/3704121704 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc0039d0 con 0x7f37c40709f0 2023-10-30T14:14:14.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.760+0000 7f37c8879640 1 -- 172.21.15.57:0/3704121704 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c41ead00 con 0x7f37c40709f0 2023-10-30T14:14:14.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.760+0000 7f37c8879640 1 -- 172.21.15.57:0/3704121704 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c414ed50 con 0x7f37c40709f0 2023-10-30T14:14:14.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.764+0000 7f37c8879640 1 -- 172.21.15.57:0/3704121704 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c404e170 con 0x7f37c40709f0 2023-10-30T14:14:14.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.772+0000 7f37bb7fe640 1 -- 172.21.15.57:0/3704121704 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f37bc003b30 con 0x7f37c40709f0 2023-10-30T14:14:14.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.772+0000 7f37bb7fe640 1 --2- 172.21.15.57:0/3704121704 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f37a4066520 0x7f37a40689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.772+0000 7f37bb7fe640 1 -- 172.21.15.57:0/3704121704 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37bc055c10 con 0x7f37c40709f0 2023-10-30T14:14:14.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.772+0000 7f37c259c640 1 --2- 172.21.15.57:0/3704121704 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f37a4066520 0x7f37a40689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.776+0000 7f37c259c640 1 --2- 172.21.15.57:0/3704121704 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f37a4066520 0x7f37a40689e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f37b40252a0 tx=0x7f37b4004ac0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.776+0000 7f37bb7fe640 1 -- 172.21.15.57:0/3704121704 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f37bc051fe0 con 0x7f37c40709f0 2023-10-30T14:14:14.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.812+0000 7f733e9f5640 1 Processor -- start 2023-10-30T14:14:14.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.812+0000 7f733e9f5640 1 -- start start 2023-10-30T14:14:14.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.812+0000 7f733e9f5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f73380c7070 0x7f733806fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.812+0000 7f7337fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7338071d50 0x7f73380c7b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:46080/0 (socket says 172.21.15.57:46080) 2023-10-30T14:14:14.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.812+0000 7f7337fff640 1 -- 172.21.15.57:0/4123013328 learned_addr learned my addr 172.21.15.57:0/4123013328 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:14.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.812+0000 7f733e9f5640 1 --2- 172.21.15.57:0/4123013328 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f733806e330 0x7f7338156240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.816+0000 7f733e9f5640 1 -- 172.21.15.57:0/4123013328 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7338159490 con 0x7f7338071d50 2023-10-30T14:14:14.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.816+0000 7f73377fe640 1 --2- 172.21.15.57:0/4123013328 >> v2:172.21.15.57:3300/0 conn(0x7f73380c7070 0x7f733806fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.816+0000 7f733e9f5640 1 -- 172.21.15.57:0/4123013328 --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73380c67f0 con 0x7f73380c7070 2023-10-30T14:14:14.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.816+0000 7f733e9f5640 1 -- 172.21.15.57:0/4123013328 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73380c6230 con 0x7f733806e330 2023-10-30T14:14:14.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.816+0000 7f73377fe640 1 -- 172.21.15.57:0/4123013328 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f733806e330 msgr2=0x7f7338156240 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T14:14:14.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.816+0000 7f73377fe640 1 --2- 172.21.15.57:0/4123013328 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f733806e330 0x7f7338156240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.816+0000 7f73377fe640 1 -- 172.21.15.57:0/4123013328 >> v1:172.21.15.88:6789/0 conn(0x7f7338071d50 legacy=0x7f73380c7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.816+0000 7f73377fe640 1 -- 172.21.15.57:0/4123013328 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73380c6510 con 0x7f73380c7070 2023-10-30T14:14:14.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.820+0000 7f73377fe640 1 --2- 172.21.15.57:0/4123013328 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73380c7070 0x7f733806fc70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7328009d90 tx=0x7f732801f120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdeda8ba040cc3c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.824+0000 7f7336ffd640 1 -- 172.21.15.57:0/4123013328 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73280b0440 con 0x7f73380c7070 2023-10-30T14:14:14.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.824+0000 7f7336ffd640 1 -- 172.21.15.57:0/4123013328 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73280b05a0 con 0x7f73380c7070 2023-10-30T14:14:14.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.824+0000 7f7336ffd640 1 -- 172.21.15.57:0/4123013328 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73280b0440 con 0x7f73380c7070 2023-10-30T14:14:14.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.824+0000 7f733e9f5640 1 -- 172.21.15.57:0/4123013328 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73380c7070 msgr2=0x7f733806fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.824+0000 7f733e9f5640 1 --2- 172.21.15.57:0/4123013328 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73380c7070 0x7f733806fc70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7328009d90 tx=0x7f732801f120 comp rx=0 tx=0).stop 2023-10-30T14:14:14.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.824+0000 7f733e9f5640 1 -- 172.21.15.57:0/4123013328 shutdown_connections 2023-10-30T14:14:14.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.824+0000 7f733e9f5640 1 --2- 172.21.15.57:0/4123013328 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f733806e330 0x7f7338156240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.824+0000 7f733e9f5640 1 --2- 172.21.15.57:0/4123013328 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73380c7070 0x7f733806fc70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.824+0000 7f733e9f5640 1 -- 172.21.15.57:0/4123013328 >> 172.21.15.57:0/4123013328 conn(0x7f733806d810 msgr2=0x7f733806dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:14.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.824+0000 7f733e9f5640 1 -- 172.21.15.57:0/4123013328 shutdown_connections 2023-10-30T14:14:14.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.824+0000 7f733e9f5640 1 -- 172.21.15.57:0/4123013328 wait complete. 2023-10-30T14:14:14.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.828+0000 7f733e9f5640 1 Processor -- start 2023-10-30T14:14:14.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.828+0000 7f733e9f5640 1 -- start start 2023-10-30T14:14:14.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.832+0000 7f733e9f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733806e330 0x7f7338143770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.832+0000 7f733e9f5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7338071d50 0x7f73381f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.832+0000 7f7337fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733806e330 0x7f7338143770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.832+0000 7f733e9f5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73380c7070 0x7f733813f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.832+0000 7f7337fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733806e330 0x7f7338143770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39812/0 (socket says 172.21.15.57:39812) 2023-10-30T14:14:14.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.832+0000 7f7337fff640 1 -- 172.21.15.57:0/1206462734 learned_addr learned my addr 172.21.15.57:0/1206462734 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:14.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.832+0000 7f733e9f5640 1 -- 172.21.15.57:0/1206462734 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7338153610 con 0x7f733806e330 2023-10-30T14:14:14.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.832+0000 7f733e9f5640 1 -- 172.21.15.57:0/1206462734 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73380cb350 con 0x7f73380c7070 2023-10-30T14:14:14.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.832+0000 7f733e9f5640 1 -- 172.21.15.57:0/1206462734 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73380cb630 con 0x7f7338071d50 2023-10-30T14:14:14.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.832+0000 7f73377fe640 1 --2- 172.21.15.57:0/1206462734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7338071d50 0x7f73381f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.832+0000 7f733cf92640 1 --2- 172.21.15.57:0/1206462734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73380c7070 0x7f733813f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.832+0000 7f7337fff640 1 -- 172.21.15.57:0/1206462734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7338071d50 msgr2=0x7f73381f26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.832+0000 7f7337fff640 1 --2- 172.21.15.57:0/1206462734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7338071d50 0x7f73381f26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.832+0000 7f7337fff640 1 -- 172.21.15.57:0/1206462734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73380c7070 msgr2=0x7f733813f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.832+0000 7f7337fff640 1 --2- 172.21.15.57:0/1206462734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73380c7070 0x7f733813f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.832+0000 7f7337fff640 1 -- 172.21.15.57:0/1206462734 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7328009a70 con 0x7f733806e330 2023-10-30T14:14:14.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.832+0000 7f733cf92640 1 --2- 172.21.15.57:0/1206462734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73380c7070 0x7f733813f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:14.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.832+0000 7f7337fff640 1 --2- 172.21.15.57:0/1206462734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733806e330 0x7f7338143770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f733000d640 tx=0x7f733000db00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.836+0000 7f7334ff9640 1 -- 172.21.15.57:0/1206462734 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73300088d0 con 0x7f733806e330 2023-10-30T14:14:14.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.836+0000 7f7334ff9640 1 -- 172.21.15.57:0/1206462734 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7330008a30 con 0x7f733806e330 2023-10-30T14:14:14.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.836+0000 7f7334ff9640 1 -- 172.21.15.57:0/1206462734 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7330008de0 con 0x7f733806e330 2023-10-30T14:14:14.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.836+0000 7f733e9f5640 1 -- 172.21.15.57:0/1206462734 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73381538b0 con 0x7f733806e330 2023-10-30T14:14:14.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.836+0000 7faceebf5640 1 Processor -- start 2023-10-30T14:14:14.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.836+0000 7faceebf5640 1 -- start start 2023-10-30T14:14:14.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.836+0000 7f733e9f5640 1 -- 172.21.15.57:0/1206462734 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73380c8c80 con 0x7f733806e330 2023-10-30T14:14:14.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.840+0000 7faceebf5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7face806fdf0 0x7face8155860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.840+0000 7f732e7fc640 1 -- 172.21.15.57:0/1206462734 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733804e170 con 0x7f733806e330 2023-10-30T14:14:14.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.840+0000 7faceebf5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face80c50d0 con 0x7face806c420 2023-10-30T14:14:14.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.840+0000 7faceebf5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face81567b0 con 0x7face80cb790 2023-10-30T14:14:14.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.840+0000 7faceebf5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face80c5a70 con 0x7face806fdf0 2023-10-30T14:14:14.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.840+0000 7faced192640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7face806c420 0x7face80c5e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:37678/0 (socket says 172.21.15.57:37678) 2023-10-30T14:14:14.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.840+0000 7faced192640 1 -- 172.21.15.57:0/1527672999 learned_addr learned my addr 172.21.15.57:0/1527672999 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:14.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.840+0000 7facdf7fe640 1 -- 172.21.15.57:0/1527672999 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202129312 0 0) 0x7face80c50d0 con 0x7face806c420 2023-10-30T14:14:14.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.840+0000 7facdf7fe640 1 -- 172.21.15.57:0/1527672999 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facd0003530 con 0x7face806c420 2023-10-30T14:14:14.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.840+0000 7facdffff640 1 --2- 172.21.15.57:0/1527672999 >> v2:172.21.15.88:3300/0 conn(0x7face806fdf0 0x7face8155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.840+0000 7f7334ff9640 1 -- 172.21.15.57:0/1206462734 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7330010420 con 0x7f733806e330 2023-10-30T14:14:14.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.840+0000 7f7334ff9640 1 --2- 172.21.15.57:0/1206462734 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f73180664f0 0x7f73180689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.844+0000 7f7334ff9640 1 -- 172.21.15.57:0/1206462734 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7330009040 con 0x7f733806e330 2023-10-30T14:14:14.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.844+0000 7f9ac60c9640 1 Processor -- start 2023-10-30T14:14:14.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.844+0000 7f73377fe640 1 --2- 172.21.15.57:0/1206462734 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f73180664f0 0x7f73180689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.844+0000 7f9ac60c9640 1 -- start start 2023-10-30T14:14:14.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.844+0000 7facdffff640 1 -- 172.21.15.57:0/1527672999 >> v1:172.21.15.88:6789/0 conn(0x7face80cb790 legacy=0x7face80c6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:14.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.844+0000 7facdffff640 1 -- 172.21.15.57:0/1527672999 >> v1:172.21.15.57:6789/0 conn(0x7face806c420 legacy=0x7face80c5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.844+0000 7facdffff640 1 -- 172.21.15.57:0/1527672999 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face8155f10 con 0x7face806fdf0 2023-10-30T14:14:14.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.844+0000 7f9ac60c9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ac0153b10 0x7f9ac00c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.844+0000 7f9ac60c9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9ac00ca8e0 0x7f9ac006bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.844+0000 7f9ac60c9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac015d070 con 0x7f9ac006c480 2023-10-30T14:14:14.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.844+0000 7f9ac60c9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ac0159900 con 0x7f9ac00ca8e0 2023-10-30T14:14:14.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.844+0000 7f9ac60c9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac015d350 con 0x7f9ac0153b10 2023-10-30T14:14:14.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.844+0000 7f73377fe640 1 --2- 172.21.15.57:0/1206462734 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f73180664f0 0x7f73180689b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f73280b2000 tx=0x7f73280aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.844+0000 7f9abffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9ac006c480 0x7f9ac0159d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:46086/0 (socket says 172.21.15.57:46086) 2023-10-30T14:14:14.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.844+0000 7f9abf7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ac0153b10 0x7f9ac00c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.844+0000 7f9abf7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ac0153b10 0x7f9ac00c9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:33238/0 (socket says 172.21.15.57:33238) 2023-10-30T14:14:14.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.844+0000 7f9abf7fe640 1 -- 172.21.15.57:0/2094904568 learned_addr learned my addr 172.21.15.57:0/2094904568 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:14.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.848+0000 7f9ab6ffd640 1 --2- 172.21.15.57:0/2094904568 >> v2:172.21.15.57:3300/0 conn(0x7f9ac00ca8e0 0x7f9ac006bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.848+0000 7f9abeffd640 1 -- 172.21.15.57:0/2094904568 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508225580 0 0) 0x7f9ac015d070 con 0x7f9ac006c480 2023-10-30T14:14:14.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.848+0000 7f9abeffd640 1 -- 172.21.15.57:0/2094904568 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aac003530 con 0x7f9ac006c480 2023-10-30T14:14:14.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.848+0000 7f9abf7fe640 1 -- 172.21.15.57:0/2094904568 >> v2:172.21.15.57:3300/0 conn(0x7f9ac00ca8e0 msgr2=0x7f9ac006bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:14.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.848+0000 7f9abf7fe640 1 --2- 172.21.15.57:0/2094904568 >> v2:172.21.15.57:3300/0 conn(0x7f9ac00ca8e0 0x7f9ac006bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.848+0000 7f9abf7fe640 1 -- 172.21.15.57:0/2094904568 >> v1:172.21.15.88:6789/0 conn(0x7f9ac006c480 legacy=0x7f9ac0159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.848+0000 7f9abf7fe640 1 -- 172.21.15.57:0/2094904568 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac015d630 con 0x7f9ac0153b10 2023-10-30T14:14:14.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7facdffff640 1 --2- 172.21.15.57:0/1527672999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face806fdf0 0x7face8155860 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7facd80097c0 tx=0x7facd801c1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e755f5455b5ef45f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7f7334ff9640 1 -- 172.21.15.57:0/1206462734 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f73300517d0 con 0x7f733806e330 2023-10-30T14:14:14.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7facdf7fe640 1 -- 172.21.15.57:0/1527672999 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd80ab600 con 0x7face806fdf0 2023-10-30T14:14:14.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7facdf7fe640 1 -- 172.21.15.57:0/1527672999 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd80a7050 con 0x7face806fdf0 2023-10-30T14:14:14.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7facdf7fe640 1 -- 172.21.15.57:0/1527672999 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd80ab980 con 0x7face806fdf0 2023-10-30T14:14:14.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7fbc914d6640 1 Processor -- start 2023-10-30T14:14:14.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7fbc914d6640 1 -- start start 2023-10-30T14:14:14.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7ff234971640 1 Processor -- start 2023-10-30T14:14:14.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7faceebf5640 1 -- 172.21.15.57:0/1527672999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face806fdf0 msgr2=0x7face8155860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7faceebf5640 1 --2- 172.21.15.57:0/1527672999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face806fdf0 0x7face8155860 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7facd80097c0 tx=0x7facd801c1a0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7faceebf5640 1 -- 172.21.15.57:0/1527672999 shutdown_connections 2023-10-30T14:14:14.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7ff234971640 1 -- start start 2023-10-30T14:14:14.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7faceebf5640 1 --2- 172.21.15.57:0/1527672999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face806fdf0 0x7face8155860 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7faceebf5640 1 -- 172.21.15.57:0/1527672999 >> 172.21.15.57:0/1527672999 conn(0x7face80cac70 msgr2=0x7face80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:14.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7f9abf7fe640 1 --2- 172.21.15.57:0/2094904568 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ac0153b10 0x7f9ac00c9b90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0009870 tx=0x7f9ab001c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18089dbc5b35684b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7faceebf5640 1 -- 172.21.15.57:0/1527672999 shutdown_connections 2023-10-30T14:14:14.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7faceebf5640 1 -- 172.21.15.57:0/1527672999 wait complete. 2023-10-30T14:14:14.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7fbc914d6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbc8c06c420 0x7fbc8c0c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7fbc914d6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc8c0c5f80 con 0x7fbc8c153bb0 2023-10-30T14:14:14.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7fbc914d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc8c0c5690 con 0x7fbc8c0c7170 2023-10-30T14:14:14.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.852+0000 7fbc914d6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc8c0c7580 con 0x7fbc8c06c420 2023-10-30T14:14:14.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7fbc8a7fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbc8c06c420 0x7fbc8c0c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7ff234971640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff2280464d0 0x7ff228047d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7ff234971640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff22804bd90 con 0x7ff22804c220 2023-10-30T14:14:14.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7ff234971640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff228047960 con 0x7ff2280468b0 2023-10-30T14:14:14.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7ff234971640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2280091a0 con 0x7ff2280464d0 2023-10-30T14:14:14.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7ff22f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff2280464d0 0x7ff228047d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7ff22f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff2280464d0 0x7ff228047d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:49206/0 (socket says 172.21.15.57:49206) 2023-10-30T14:14:14.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7ff22f7fe640 1 -- 172.21.15.57:0/3740703057 learned_addr learned my addr 172.21.15.57:0/3740703057 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:14.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7f9abeffd640 1 -- 172.21.15.57:0/2094904568 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab00ab650 con 0x7f9ac0153b10 2023-10-30T14:14:14.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7f9abeffd640 1 -- 172.21.15.57:0/2094904568 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab00ab7b0 con 0x7f9ac0153b10 2023-10-30T14:14:14.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7f9abeffd640 1 -- 172.21.15.57:0/2094904568 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab00abb40 con 0x7f9ac0153b10 2023-10-30T14:14:14.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7f9ac60c9640 1 -- 172.21.15.57:0/2094904568 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ac0153b10 msgr2=0x7f9ac00c9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7f9ac60c9640 1 --2- 172.21.15.57:0/2094904568 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ac0153b10 0x7f9ac00c9b90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0009870 tx=0x7f9ab001c540 comp rx=0 tx=0).stop 2023-10-30T14:14:14.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7fbc8a7fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbc8c06c420 0x7fbc8c0c6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39836/0 (socket says 172.21.15.57:39836) 2023-10-30T14:14:14.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7fbc8a7fc640 1 -- 172.21.15.57:0/3784809977 learned_addr learned my addr 172.21.15.57:0/3784809977 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:14.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7f9ac60c9640 1 -- 172.21.15.57:0/2094904568 shutdown_connections 2023-10-30T14:14:14.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7f9ac60c9640 1 --2- 172.21.15.57:0/2094904568 >> v2:172.21.15.57:3300/0 conn(0x7f9ac00ca8e0 0x7f9ac006bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7f9ac60c9640 1 --2- 172.21.15.57:0/2094904568 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ac0153b10 0x7f9ac00c9b90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7f9ac60c9640 1 -- 172.21.15.57:0/2094904568 >> 172.21.15.57:0/2094904568 conn(0x7f9ac006a240 msgr2=0x7f9ac006a650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:14.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7f9ac60c9640 1 -- 172.21.15.57:0/2094904568 shutdown_connections 2023-10-30T14:14:14.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7faceebf5640 1 Processor -- start 2023-10-30T14:14:14.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7ff22f7fe640 1 -- 172.21.15.57:0/3740703057 >> v1:172.21.15.88:6789/0 conn(0x7ff2280468b0 legacy=0x7ff228047250 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T14:14:14.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7fbc89ffb640 1 -- 172.21.15.57:0/3784809977 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009003542 0 0) 0x7fbc8c0c5690 con 0x7fbc8c0c7170 2023-10-30T14:14:14.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7fbc89ffb640 1 -- 172.21.15.57:0/3784809977 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc74003530 con 0x7fbc8c0c7170 2023-10-30T14:14:14.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7faceebf5640 1 -- start start 2023-10-30T14:14:14.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7faceebf5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face806c420 0x7face8148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7faceebf5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7face806fdf0 0x7face8147450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7faceebf5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7face80cb790 0x7face81409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7faceebf5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face8140260 con 0x7face80cb790 2023-10-30T14:14:14.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7faceebf5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face81462c0 con 0x7face806c420 2023-10-30T14:14:14.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7faceebf5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face80c9be0 con 0x7face806fdf0 2023-10-30T14:14:14.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7fbc89ffb640 1 -- 172.21.15.57:0/3784809977 <== mon.1 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1174491222 0 0) 0x7fbc74003530 con 0x7fbc8c0c7170 2023-10-30T14:14:14.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7fbc89ffb640 1 -- 172.21.15.57:0/3784809977 >> v2:172.21.15.57:3300/0 conn(0x7fbc8c06c420 msgr2=0x7fbc8c0c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:14.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7fbc89ffb640 1 --2- 172.21.15.57:0/3784809977 >> v2:172.21.15.57:3300/0 conn(0x7fbc8c06c420 0x7fbc8c0c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7fbc89ffb640 1 -- 172.21.15.57:0/3784809977 >> v1:172.21.15.57:6789/0 conn(0x7fbc8c153bb0 legacy=0x7fbc8c0c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7fbc89ffb640 1 -- 172.21.15.57:0/3784809977 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8c15a580 con 0x7fbc8c0c7170 2023-10-30T14:14:14.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7fbc89ffb640 1 -- 172.21.15.57:0/3784809977 <== mon.1 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fbc84001a00 con 0x7fbc8c0c7170 2023-10-30T14:14:14.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7fbc89ffb640 1 -- 172.21.15.57:0/3784809977 >> v1:172.21.15.88:6789/0 conn(0x7fbc8c0c7170 legacy=0x7fbc8c159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7fbc89ffb640 1 --2- 172.21.15.57:0/3784809977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc74003980 0x7fbc74003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7fbc89ffb640 1 --2- 172.21.15.57:0/3784809977 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc740047d0 0x7fbc74004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7fbc89ffb640 1 --2- 172.21.15.57:0/3784809977 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc74005190 0x7fbc7400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7fbc89ffb640 1 -- 172.21.15.57:0/3784809977 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c0c7580 con 0x7fbc74005190 2023-10-30T14:14:14.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7fbc89ffb640 1 -- 172.21.15.57:0/3784809977 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7400bfe0 con 0x7fbc74003980 2023-10-30T14:14:14.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7fbc89ffb640 1 -- 172.21.15.57:0/3784809977 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7400c1a0 con 0x7fbc740047d0 2023-10-30T14:14:14.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7facdffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7face806fdf0 0x7face8147450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7facdffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7face806fdf0 0x7face8147450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:33240/0 (socket says 172.21.15.57:33240) 2023-10-30T14:14:14.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7facec991640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face806c420 0x7face8148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7facec991640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face806c420 0x7face8148690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:49236/0 (socket says 172.21.15.57:49236) 2023-10-30T14:14:14.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7facec991640 1 -- 172.21.15.57:0/3050327589 learned_addr learned my addr 172.21.15.57:0/3050327589 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:14.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7facdffff640 1 -- 172.21.15.57:0/3050327589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face806c420 msgr2=0x7face8148690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7f9ac60c9640 1 -- 172.21.15.57:0/2094904568 wait complete. 2023-10-30T14:14:14.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7f9ac60c9640 1 Processor -- start 2023-10-30T14:14:14.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7fbc8b7fe640 1 --2- 172.21.15.57:0/3784809977 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc74005190 0x7fbc7400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7fbc8affd640 1 --2- 172.21.15.57:0/3784809977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc74003980 0x7fbc74003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7facdffff640 1 --2- 172.21.15.57:0/3050327589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face806c420 0x7face8148690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7facdffff640 1 -- 172.21.15.57:0/3050327589 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7face80cb790 msgr2=0x7face81409d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7facdffff640 1 --2- 172.21.15.57:0/3050327589 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7face80cb790 0x7face81409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7facdffff640 1 -- 172.21.15.57:0/3050327589 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd8009420 con 0x7face806fdf0 2023-10-30T14:14:14.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7fbc8a7fc640 1 --2- 172.21.15.57:0/3784809977 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc740047d0 0x7fbc74004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7f9ac60c9640 1 -- start start 2023-10-30T14:14:14.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7facdffff640 1 --2- 172.21.15.57:0/3050327589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7face806fdf0 0x7face8147450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7facd80ab000 tx=0x7facd8018f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7ff22f7fe640 1 -- 172.21.15.57:0/3740703057 >> v1:172.21.15.57:6789/0 conn(0x7ff22804c220 legacy=0x7ff22804c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:14.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7fbc8affd640 1 -- 172.21.15.57:0/3784809977 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc740047d0 msgr2=0x7fbc74004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:14.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7fbc8affd640 1 --2- 172.21.15.57:0/3784809977 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc740047d0 0x7fbc74004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7fbc8affd640 1 -- 172.21.15.57:0/3784809977 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc74005190 msgr2=0x7fbc7400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:14.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7fbc8affd640 1 --2- 172.21.15.57:0/3784809977 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc74005190 0x7fbc7400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7fbc8affd640 1 -- 172.21.15.57:0/3784809977 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8c15a580 con 0x7fbc74003980 2023-10-30T14:14:14.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.856+0000 7ff22f7fe640 1 -- 172.21.15.57:0/3740703057 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2280dedd0 con 0x7ff2280464d0 2023-10-30T14:14:14.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7fbc8b7fe640 1 -- 172.21.15.57:0/3784809977 reap_dead start 2023-10-30T14:14:14.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7facdd7fa640 1 -- 172.21.15.57:0/3050327589 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd80b4600 con 0x7face806fdf0 2023-10-30T14:14:14.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7ff22f7fe640 1 --2- 172.21.15.57:0/3740703057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2280464d0 0x7ff228047d90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff224009870 tx=0x7ff22401c4c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64057319337abf11 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7facdd7fa640 1 -- 172.21.15.57:0/3050327589 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd80b4760 con 0x7face806fdf0 2023-10-30T14:14:14.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7facdd7fa640 1 -- 172.21.15.57:0/3050327589 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd80b4600 con 0x7face806fdf0 2023-10-30T14:14:14.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.860+0000 7faceebf5640 1 -- 172.21.15.57:0/3050327589 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face80c9ea0 con 0x7face806fdf0 2023-10-30T14:14:14.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7faceebf5640 1 -- 172.21.15.57:0/3050327589 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face81404b0 con 0x7face806fdf0 2023-10-30T14:14:14.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7f9ac60c9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ac006c480 0x7f9ac0150980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7f9ac60c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ac00ca8e0 0x7f9ac00cd110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7f9abf7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ac006c480 0x7f9ac0150980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7f9abf7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ac006c480 0x7f9ac0150980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:33258/0 (socket says 172.21.15.57:33258) 2023-10-30T14:14:14.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7f9abf7fe640 1 -- 172.21.15.57:0/1155034739 learned_addr learned my addr 172.21.15.57:0/1155034739 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:14.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7ff22e7fc640 1 -- 172.21.15.57:0/3740703057 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2240ac410 con 0x7ff2280464d0 2023-10-30T14:14:14.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7f9ab6ffd640 1 --2- 172.21.15.57:0/1155034739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ac00ca8e0 0x7f9ac00cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7f9ac60c9640 1 --2- 172.21.15.57:0/1155034739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ac0153b10 0x7f9ac00715f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7ff22e7fc640 1 -- 172.21.15.57:0/3740703057 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2240ac570 con 0x7ff2280464d0 2023-10-30T14:14:14.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7ff22e7fc640 1 -- 172.21.15.57:0/3740703057 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2240ac410 con 0x7ff2280464d0 2023-10-30T14:14:14.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7f9ac60c9640 1 -- 172.21.15.57:0/1155034739 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac014ff50 con 0x7f9ac0153b10 2023-10-30T14:14:14.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7f9ac60c9640 1 -- 172.21.15.57:0/1155034739 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00cb710 con 0x7f9ac00ca8e0 2023-10-30T14:14:14.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7f9ac60c9640 1 -- 172.21.15.57:0/1155034739 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac0151fe0 con 0x7f9ac006c480 2023-10-30T14:14:14.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7f9abffff640 1 --2- 172.21.15.57:0/1155034739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ac0153b10 0x7f9ac00715f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7f9ab6ffd640 1 -- 172.21.15.57:0/1155034739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ac006c480 msgr2=0x7f9ac0150980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7f9ab6ffd640 1 --2- 172.21.15.57:0/1155034739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ac006c480 0x7f9ac0150980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7f9ab6ffd640 1 -- 172.21.15.57:0/1155034739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ac0153b10 msgr2=0x7f9ac00715f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7f9ab6ffd640 1 --2- 172.21.15.57:0/1155034739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ac0153b10 0x7f9ac00715f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7f9ab6ffd640 1 -- 172.21.15.57:0/1155034739 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab00094d0 con 0x7f9ac00ca8e0 2023-10-30T14:14:14.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7f9ab6ffd640 1 --2- 172.21.15.57:0/1155034739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ac00ca8e0 0x7f9ac00cd110 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9aa800d5f0 tx=0x7f9aa800dab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.864+0000 7facdd7fa640 1 -- 172.21.15.57:0/3050327589 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7facd80b2040 con 0x7face806fdf0 2023-10-30T14:14:14.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7f9abcff9640 1 -- 172.21.15.57:0/1155034739 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa80125f0 con 0x7f9ac00ca8e0 2023-10-30T14:14:14.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7f9abcff9640 1 -- 172.21.15.57:0/1155034739 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa8012750 con 0x7f9ac00ca8e0 2023-10-30T14:14:14.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7f9ac60c9640 1 -- 172.21.15.57:0/1155034739 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac0152280 con 0x7f9ac00ca8e0 2023-10-30T14:14:14.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7f9abcff9640 1 -- 172.21.15.57:0/1155034739 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8012ac0 con 0x7f9ac00ca8e0 2023-10-30T14:14:14.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7faceebf5640 1 -- 172.21.15.57:0/3050327589 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face8066510 con 0x7face806fdf0 2023-10-30T14:14:14.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7ff234971640 1 -- 172.21.15.57:0/3740703057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2280464d0 msgr2=0x7ff228047d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7ff234971640 1 --2- 172.21.15.57:0/3740703057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2280464d0 0x7ff228047d90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff224009870 tx=0x7ff22401c4c0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7facdd7fa640 1 --2- 172.21.15.57:0/3050327589 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7facd4066560 0x7facd4068a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7fbc8affd640 1 --2- 172.21.15.57:0/3784809977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc74003980 0x7fbc74003da0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c001780 tx=0x7fbc7c01c340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc6b8037f9b38d3e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7facdd7fa640 1 -- 172.21.15.57:0/3050327589 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7facd811d640 con 0x7face806fdf0 2023-10-30T14:14:14.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7fbc89ffb640 1 -- 172.21.15.57:0/3784809977 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c0052d0 con 0x7fbc74003980 2023-10-30T14:14:14.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7fbc89ffb640 1 -- 172.21.15.57:0/3784809977 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7c0a7050 con 0x7fbc74003980 2023-10-30T14:14:14.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7fbc89ffb640 1 -- 172.21.15.57:0/3784809977 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c003840 con 0x7fbc74003980 2023-10-30T14:14:14.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7f9ac60c9640 1 -- 172.21.15.57:0/1155034739 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac0150460 con 0x7f9ac00ca8e0 2023-10-30T14:14:14.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7ff234971640 1 -- 172.21.15.57:0/3740703057 shutdown_connections 2023-10-30T14:14:14.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7ff234971640 1 --2- 172.21.15.57:0/3740703057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2280464d0 0x7ff228047d90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7ff234971640 1 -- 172.21.15.57:0/3740703057 >> 172.21.15.57:0/3740703057 conn(0x7ff228051410 msgr2=0x7ff228051820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:14.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7facec991640 1 --2- 172.21.15.57:0/3050327589 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7facd4066560 0x7facd4068a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7ff234971640 1 -- 172.21.15.57:0/3740703057 shutdown_connections 2023-10-30T14:14:14.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7ff234971640 1 -- 172.21.15.57:0/3740703057 wait complete. 2023-10-30T14:14:14.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7fbc914d6640 1 -- 172.21.15.57:0/3784809977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc74003980 msgr2=0x7fbc74003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7fbc914d6640 1 --2- 172.21.15.57:0/3784809977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc74003980 0x7fbc74003da0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c001780 tx=0x7fbc7c01c340 comp rx=0 tx=0).stop 2023-10-30T14:14:14.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7fbc914d6640 1 -- 172.21.15.57:0/3784809977 shutdown_connections 2023-10-30T14:14:14.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7fbc914d6640 1 --2- 172.21.15.57:0/3784809977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc74003980 0x7fbc74003da0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7fbc914d6640 1 -- 172.21.15.57:0/3784809977 >> 172.21.15.57:0/3784809977 conn(0x7fbc8c0cac70 msgr2=0x7fbc8c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:14.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7fbc914d6640 1 -- 172.21.15.57:0/3784809977 shutdown_connections 2023-10-30T14:14:14.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7fbc914d6640 1 -- 172.21.15.57:0/3784809977 wait complete. 2023-10-30T14:14:14.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.868+0000 7f9ac60c9640 1 -- 172.21.15.57:0/1155034739 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac0067540 con 0x7f9ac00ca8e0 2023-10-30T14:14:14.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.872+0000 7facec991640 1 --2- 172.21.15.57:0/3050327589 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7facd4066560 0x7facd4068a20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7face4004ef0 tx=0x7face4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.872+0000 7fbc914d6640 1 Processor -- start 2023-10-30T14:14:14.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.872+0000 7f9abcff9640 1 -- 172.21.15.57:0/1155034739 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9aa8012c20 con 0x7f9ac00ca8e0 2023-10-30T14:14:14.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.872+0000 7fbc914d6640 1 -- start start 2023-10-30T14:14:14.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.872+0000 7ff234971640 1 Processor -- start 2023-10-30T14:14:14.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.872+0000 7ff234971640 1 -- start start 2023-10-30T14:14:14.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.872+0000 7fbc914d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc74003980 0x7fbc8c1e25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.872+0000 7f9abcff9640 1 --2- 172.21.15.57:0/1155034739 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9aa00664d0 0x7f9aa0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.872+0000 7fbc914d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc8c0c3c30 0x7fbc8c1f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.872+0000 7fbc914d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc8c1509c0 0x7fbc8c1f3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.872+0000 7fbc914d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c0c4220 con 0x7fbc8c0c3c30 2023-10-30T14:14:14.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.872+0000 7fbc914d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c0c4500 con 0x7fbc8c1509c0 2023-10-30T14:14:14.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.872+0000 7fbc914d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c1e1110 con 0x7fbc74003980 2023-10-30T14:14:14.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.872+0000 7fbc8affd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc74003980 0x7fbc8c1e25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.872+0000 7fbc8affd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc74003980 0x7fbc8c1e25f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:33260/0 (socket says 172.21.15.57:33260) 2023-10-30T14:14:14.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.872+0000 7fbc8affd640 1 -- 172.21.15.57:0/2304719666 learned_addr learned my addr 172.21.15.57:0/2304719666 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:14.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.872+0000 7f9abcff9640 1 -- 172.21.15.57:0/1155034739 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9aa8084430 con 0x7f9ac00ca8e0 2023-10-30T14:14:14.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7f9abf7fe640 1 --2- 172.21.15.57:0/1155034739 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9aa00664d0 0x7f9aa0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.872+0000 7fbc8a7fc640 1 --2- 172.21.15.57:0/2304719666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc8c0c3c30 0x7fbc8c1f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.872+0000 7fbc8b7fe640 1 --2- 172.21.15.57:0/2304719666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc8c1509c0 0x7fbc8c1f3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff234971640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2280464d0 0x7ff2280d23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7fbc8affd640 1 -- 172.21.15.57:0/2304719666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc8c1509c0 msgr2=0x7fbc8c1f3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7fbc8affd640 1 --2- 172.21.15.57:0/2304719666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc8c1509c0 0x7fbc8c1f3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7fbc8affd640 1 -- 172.21.15.57:0/2304719666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc8c0c3c30 msgr2=0x7fbc8c1f2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7fbc8affd640 1 --2- 172.21.15.57:0/2304719666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc8c0c3c30 0x7fbc8c1f2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7fbc8affd640 1 -- 172.21.15.57:0/2304719666 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8c1e13f0 con 0x7fbc74003980 2023-10-30T14:14:14.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff234971640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2280468b0 0x7ff228177ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff234971640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff22804c220 0x7ff2280dddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff22effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2280468b0 0x7ff228177ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff22f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2280464d0 0x7ff2280d23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff22effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2280468b0 0x7ff228177ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:49262/0 (socket says 172.21.15.57:49262) 2023-10-30T14:14:14.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff22effd640 1 -- 172.21.15.57:0/3431699182 learned_addr learned my addr 172.21.15.57:0/3431699182 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:14.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff234971640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2280d0b30 con 0x7ff2280464d0 2023-10-30T14:14:14.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff234971640 1 -- 172.21.15.57:0/3431699182 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2281773c0 con 0x7ff2280468b0 2023-10-30T14:14:14.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff234971640 1 -- 172.21.15.57:0/3431699182 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff228031ec0 con 0x7ff22804c220 2023-10-30T14:14:14.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7f9abf7fe640 1 --2- 172.21.15.57:0/1155034739 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9aa00664d0 0x7f9aa0068990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0009870 tx=0x7f9ab00a7000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff22ffff640 1 --2- 172.21.15.57:0/3431699182 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff22804c220 0x7ff2280dddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7fbc8affd640 1 --2- 172.21.15.57:0/2304719666 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc74003980 0x7fbc8c1e25f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c005040 tx=0x7fbc74005800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff22f7fe640 1 -- 172.21.15.57:0/3431699182 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff22804c220 msgr2=0x7ff2280dddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff22f7fe640 1 --2- 172.21.15.57:0/3431699182 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff22804c220 0x7ff2280dddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff22f7fe640 1 -- 172.21.15.57:0/3431699182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2280468b0 msgr2=0x7ff228177ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff22f7fe640 1 --2- 172.21.15.57:0/3431699182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2280468b0 0x7ff228177ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff22f7fe640 1 -- 172.21.15.57:0/3431699182 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2240094d0 con 0x7ff2280464d0 2023-10-30T14:14:14.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff22ffff640 1 --2- 172.21.15.57:0/3431699182 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff22804c220 0x7ff2280dddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:14.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff22f7fe640 1 --2- 172.21.15.57:0/3431699182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2280464d0 0x7ff2280d23b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff2240ac000 tx=0x7ff224019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff21bfff640 1 -- 172.21.15.57:0/3431699182 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2240bc070 con 0x7ff2280464d0 2023-10-30T14:14:14.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff21bfff640 1 -- 172.21.15.57:0/3431699182 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2240ab740 con 0x7ff2280464d0 2023-10-30T14:14:14.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff21bfff640 1 -- 172.21.15.57:0/3431699182 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2240ac410 con 0x7ff2280464d0 2023-10-30T14:14:14.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff234971640 1 -- 172.21.15.57:0/3431699182 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2280321a0 con 0x7ff2280464d0 2023-10-30T14:14:14.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.876+0000 7ff234971640 1 -- 172.21.15.57:0/3431699182 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff228032760 con 0x7ff2280464d0 2023-10-30T14:14:14.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.880+0000 7ff234971640 1 -- 172.21.15.57:0/3431699182 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2280069c0 con 0x7ff2280464d0 2023-10-30T14:14:14.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.880+0000 7facdd7fa640 1 -- 172.21.15.57:0/3050327589 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7facd80eaa40 con 0x7face806fdf0 2023-10-30T14:14:14.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.880+0000 7ff21bfff640 1 -- 172.21.15.57:0/3431699182 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff2240ab8d0 con 0x7ff2280464d0 2023-10-30T14:14:14.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.880+0000 7fbc737fe640 1 -- 172.21.15.57:0/2304719666 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c007440 con 0x7fbc74003980 2023-10-30T14:14:14.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.880+0000 7fbc737fe640 1 -- 172.21.15.57:0/2304719666 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7c0075a0 con 0x7fbc74003980 2023-10-30T14:14:14.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.880+0000 7fbc914d6640 1 -- 172.21.15.57:0/2304719666 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8c1e16d0 con 0x7fbc74003980 2023-10-30T14:14:14.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.880+0000 7fbc737fe640 1 -- 172.21.15.57:0/2304719666 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c006a00 con 0x7fbc74003980 2023-10-30T14:14:14.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.884+0000 7fbc914d6640 1 -- 172.21.15.57:0/2304719666 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8c1e2b30 con 0x7fbc74003980 2023-10-30T14:14:14.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.884+0000 7ff21bfff640 1 --2- 172.21.15.57:0/3431699182 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff2140664a0 0x7ff214068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.884+0000 7ff21bfff640 1 -- 172.21.15.57:0/3431699182 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff22411dd20 con 0x7ff2280464d0 2023-10-30T14:14:14.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.884+0000 7fbc914d6640 1 -- 172.21.15.57:0/2304719666 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc8c04e170 con 0x7fbc74003980 2023-10-30T14:14:14.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.884+0000 7ff22effd640 1 --2- 172.21.15.57:0/3431699182 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff2140664a0 0x7ff214068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.884+0000 7f9abcff9640 1 -- 172.21.15.57:0/1155034739 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9aa8051820 con 0x7f9ac00ca8e0 2023-10-30T14:14:14.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.884+0000 7fbc737fe640 1 -- 172.21.15.57:0/2304719666 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbc7c006b60 con 0x7fbc74003980 2023-10-30T14:14:14.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7fa6ee691640 1 Processor -- start 2023-10-30T14:14:14.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7fa6ee691640 1 -- start start 2023-10-30T14:14:14.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7fbc737fe640 1 --2- 172.21.15.57:0/2304719666 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbc78066520 0x7fbc780689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7fbc737fe640 1 -- 172.21.15.57:0/2304719666 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc7c11ecc0 con 0x7fbc74003980 2023-10-30T14:14:14.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7fbc8a7fc640 1 --2- 172.21.15.57:0/2304719666 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbc78066520 0x7fbc780689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7ff22effd640 1 --2- 172.21.15.57:0/3431699182 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff2140664a0 0x7ff214068960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff230068f50 tx=0x7ff230075610 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7fa6ee691640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e80c9b90 0x7fa6e806bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7fbc8a7fc640 1 --2- 172.21.15.57:0/2304719666 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbc78066520 0x7fbc780689e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c156d90 tx=0x7fbc80009240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7fa6ee691640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa6e806c480 0x7fa6e8159d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7fa6ee691640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e80c9fd0 con 0x7fa6e8153b10 2023-10-30T14:14:14.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7fa6ee691640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6e8159900 con 0x7fa6e806c480 2023-10-30T14:14:14.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7fa6ee691640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e815d070 con 0x7fa6e80c9b90 2023-10-30T14:14:14.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7fa6e7fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa6e8153b10 0x7fa6e80ca8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:46104/0 (socket says 172.21.15.57:46104) 2023-10-30T14:14:14.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7fa6e7fff640 1 -- 172.21.15.57:0/3640280938 learned_addr learned my addr 172.21.15.57:0/3640280938 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:14.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7fa6e77fe640 1 --2- 172.21.15.57:0/3640280938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e80c9b90 0x7fa6e806bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7fa6ecc2e640 1 --2- 172.21.15.57:0/3640280938 >> v2:172.21.15.88:3300/0 conn(0x7fa6e806c480 0x7fa6e8159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7f9743380640 1 Processor -- start 2023-10-30T14:14:14.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7fa6e6ffd640 1 -- 172.21.15.57:0/3640280938 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39048777 0 0) 0x7fa6d8001120 con 0x7fa6e8153b10 2023-10-30T14:14:14.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7fa6e6ffd640 1 -- 172.21.15.57:0/3640280938 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d0003530 con 0x7fa6e8153b10 2023-10-30T14:14:14.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.888+0000 7f9743380640 1 -- start start 2023-10-30T14:14:14.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7fa6e77fe640 1 -- 172.21.15.57:0/3640280938 >> v2:172.21.15.88:3300/0 conn(0x7fa6e806c480 msgr2=0x7fa6e8159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:14.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7fa6e77fe640 1 --2- 172.21.15.57:0/3640280938 >> v2:172.21.15.88:3300/0 conn(0x7fa6e806c480 0x7fa6e8159d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7fa6e6ffd640 1 -- 172.21.15.57:0/3640280938 <== mon.1 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1353587008 0 0) 0x7fa6d0003530 con 0x7fa6e8153b10 2023-10-30T14:14:14.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7fa6e77fe640 1 -- 172.21.15.57:0/3640280938 >> v1:172.21.15.88:6789/0 conn(0x7fa6e8153b10 legacy=0x7fa6e80ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7fa6e77fe640 1 -- 172.21.15.57:0/3640280938 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e815d350 con 0x7fa6e80c9b90 2023-10-30T14:14:14.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f9743380640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f973c06c420 0x7f973c0c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f9743380640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973c0c5f80 con 0x7f973c0c7170 2023-10-30T14:14:14.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f9743380640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973c0c5690 con 0x7f973c153bb0 2023-10-30T14:14:14.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f9743380640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f973c0c7580 con 0x7f973c06c420 2023-10-30T14:14:14.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f974111c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f973c153bb0 0x7f973c0c6410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:46112/0 (socket says 172.21.15.57:46112) 2023-10-30T14:14:14.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f974111c640 1 -- 172.21.15.57:0/1045047300 learned_addr learned my addr 172.21.15.57:0/1045047300 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:14.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f974091b640 1 --2- 172.21.15.57:0/1045047300 >> v2:172.21.15.57:3300/0 conn(0x7f973c06c420 0x7f973c0c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f973bfff640 1 -- 172.21.15.57:0/1045047300 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711480566 0 0) 0x7f973c0c5690 con 0x7f973c153bb0 2023-10-30T14:14:14.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f973bfff640 1 -- 172.21.15.57:0/1045047300 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f972c003530 con 0x7f973c153bb0 2023-10-30T14:14:14.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f973bfff640 1 -- 172.21.15.57:0/1045047300 <== mon.1 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 794955275 0 0) 0x7f972c003530 con 0x7f973c153bb0 2023-10-30T14:14:14.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f973bfff640 1 -- 172.21.15.57:0/1045047300 >> v2:172.21.15.57:3300/0 conn(0x7f973c06c420 msgr2=0x7f973c0c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:14.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f973bfff640 1 --2- 172.21.15.57:0/1045047300 >> v2:172.21.15.57:3300/0 conn(0x7f973c06c420 0x7f973c0c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f973bfff640 1 -- 172.21.15.57:0/1045047300 >> v1:172.21.15.57:6789/0 conn(0x7f973c0c7170 legacy=0x7f973c159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f973bfff640 1 -- 172.21.15.57:0/1045047300 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c15a580 con 0x7f973c153bb0 2023-10-30T14:14:14.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f973bfff640 1 -- 172.21.15.57:0/1045047300 <== mon.1 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f9730001a00 con 0x7f973c153bb0 2023-10-30T14:14:14.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f973bfff640 1 -- 172.21.15.57:0/1045047300 >> v1:172.21.15.88:6789/0 conn(0x7f973c153bb0 legacy=0x7f973c0c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f973bfff640 1 --2- 172.21.15.57:0/1045047300 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f972c003980 0x7f972c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f973bfff640 1 --2- 172.21.15.57:0/1045047300 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f972c0047d0 0x7f972c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f973bfff640 1 --2- 172.21.15.57:0/1045047300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f972c005190 0x7f972c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f973bfff640 1 -- 172.21.15.57:0/1045047300 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c0c7580 con 0x7f972c0047d0 2023-10-30T14:14:14.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f973bfff640 1 -- 172.21.15.57:0/1045047300 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972c00c070 con 0x7f972c005190 2023-10-30T14:14:14.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f973bfff640 1 -- 172.21.15.57:0/1045047300 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f972c00c230 con 0x7f972c003980 2023-10-30T14:14:14.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.892+0000 7f974091b640 1 --2- 172.21.15.57:0/1045047300 >> v2:172.21.15.57:3300/0 conn(0x7f973c06c420 0x7f973c0c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:14.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.896+0000 7f974191d640 1 --2- 172.21.15.57:0/1045047300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f972c005190 0x7f972c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.896+0000 7f974111c640 1 --2- 172.21.15.57:0/1045047300 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f972c003980 0x7f972c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.896+0000 7f974091b640 1 --2- 172.21.15.57:0/1045047300 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f972c0047d0 0x7f972c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.896+0000 7f974191d640 1 -- 172.21.15.57:0/1045047300 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f972c003980 msgr2=0x7f972c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:14.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.896+0000 7f974191d640 1 --2- 172.21.15.57:0/1045047300 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f972c003980 0x7f972c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.896+0000 7f974191d640 1 -- 172.21.15.57:0/1045047300 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f972c0047d0 msgr2=0x7f972c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:14.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.896+0000 7f974191d640 1 --2- 172.21.15.57:0/1045047300 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f972c0047d0 0x7f972c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.896+0000 7f974191d640 1 -- 172.21.15.57:0/1045047300 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c15a580 con 0x7f972c005190 2023-10-30T14:14:14.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.896+0000 7f974091b640 1 --2- 172.21.15.57:0/1045047300 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f972c0047d0 0x7f972c004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:14:14.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.896+0000 7fa6e77fe640 1 --2- 172.21.15.57:0/3640280938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e80c9b90 0x7fa6e806bf40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc0097c0 tx=0x7fa6dc01c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b53ab88cef6c46d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.896+0000 7ff21bfff640 1 -- 172.21.15.57:0/3431699182 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff2240eb100 con 0x7ff2280464d0 2023-10-30T14:14:14.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.900+0000 7fbc737fe640 1 -- 172.21.15.57:0/2304719666 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbc7c0ec150 con 0x7fbc74003980 2023-10-30T14:14:14.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.896+0000 7fa6e6ffd640 1 -- 172.21.15.57:0/3640280938 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8001810 con 0x7fa6e80c9b90 2023-10-30T14:14:14.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.896+0000 7fa6e6ffd640 1 -- 172.21.15.57:0/3640280938 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6dc0a7050 con 0x7fa6e80c9b90 2023-10-30T14:14:14.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.896+0000 7fa6e6ffd640 1 -- 172.21.15.57:0/3640280938 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc0ab620 con 0x7fa6e80c9b90 2023-10-30T14:14:14.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.900+0000 7f974191d640 1 -- 172.21.15.57:0/1045047300 reap_dead start 2023-10-30T14:14:14.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.900+0000 7fa6ee691640 1 -- 172.21.15.57:0/3640280938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e80c9b90 msgr2=0x7fa6e806bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.900+0000 7fa6ee691640 1 --2- 172.21.15.57:0/3640280938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e80c9b90 0x7fa6e806bf40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc0097c0 tx=0x7fa6dc01c330 comp rx=0 tx=0).stop 2023-10-30T14:14:14.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.900+0000 7fa6ee691640 1 -- 172.21.15.57:0/3640280938 shutdown_connections 2023-10-30T14:14:14.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.900+0000 7fa6ee691640 1 --2- 172.21.15.57:0/3640280938 >> v2:172.21.15.88:3300/0 conn(0x7fa6e806c480 0x7fa6e8159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.900+0000 7fa6ee691640 1 --2- 172.21.15.57:0/3640280938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e80c9b90 0x7fa6e806bf40 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc0097c0 tx=0x7fa6dc01c330 comp rx=0 tx=0).stop 2023-10-30T14:14:14.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.900+0000 7fa6ee691640 1 -- 172.21.15.57:0/3640280938 >> 172.21.15.57:0/3640280938 conn(0x7fa6e806a240 msgr2=0x7fa6e806a650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:14.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.900+0000 7fa6ee691640 1 -- 172.21.15.57:0/3640280938 shutdown_connections 2023-10-30T14:14:14.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.900+0000 7fa6ee691640 1 -- 172.21.15.57:0/3640280938 wait complete. 2023-10-30T14:14:14.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.900+0000 7f974191d640 1 --2- 172.21.15.57:0/1045047300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f972c005190 0x7f972c00b990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9734014dd0 tx=0x7f9734024b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32e9716b2c854173 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.900+0000 7f973bfff640 1 -- 172.21.15.57:0/1045047300 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97340052d0 con 0x7f972c005190 2023-10-30T14:14:14.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.900+0000 7f973bfff640 1 -- 172.21.15.57:0/1045047300 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9734005450 con 0x7f972c005190 2023-10-30T14:14:14.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.900+0000 7f973bfff640 1 -- 172.21.15.57:0/1045047300 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97340adc20 con 0x7f972c005190 2023-10-30T14:14:14.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.900+0000 7f9743380640 1 -- 172.21.15.57:0/1045047300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f972c005190 msgr2=0x7f972c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.900+0000 7f9743380640 1 --2- 172.21.15.57:0/1045047300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f972c005190 0x7f972c00b990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9734014dd0 tx=0x7f9734024b30 comp rx=0 tx=0).stop 2023-10-30T14:14:14.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.900+0000 7f9743380640 1 -- 172.21.15.57:0/1045047300 shutdown_connections 2023-10-30T14:14:14.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.900+0000 7f9743380640 1 --2- 172.21.15.57:0/1045047300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f972c005190 0x7f972c00b990 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.900+0000 7f9743380640 1 -- 172.21.15.57:0/1045047300 >> 172.21.15.57:0/1045047300 conn(0x7f973c0cac70 msgr2=0x7f973c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:14.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.904+0000 7fa6ee691640 1 Processor -- start 2023-10-30T14:14:14.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.904+0000 7fa6ee691640 1 -- start start 2023-10-30T14:14:14.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.904+0000 7fa6ee691640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e806c480 0x7fa6e81f2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.904+0000 7fa6ee691640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6e8153b10 0x7fa6e80b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.904+0000 7fa6ee691640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6e8155d10 0x7fa6e815bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.904+0000 7fa6ee691640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e81f5b10 con 0x7fa6e8155d10 2023-10-30T14:14:14.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.904+0000 7fa6ee691640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e81f5dd0 con 0x7fa6e8153b10 2023-10-30T14:14:14.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.904+0000 7fa6ee691640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e81f60b0 con 0x7fa6e806c480 2023-10-30T14:14:14.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.904+0000 7fa6e77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6e8153b10 0x7fa6e80b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.904+0000 7f9743380640 1 -- 172.21.15.57:0/1045047300 shutdown_connections 2023-10-30T14:14:14.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.904+0000 7fa6ecc2e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6e8155d10 0x7fa6e815bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.904+0000 7fa6ecc2e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6e8155d10 0x7fa6e815bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39916/0 (socket says 172.21.15.57:39916) 2023-10-30T14:14:14.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.904+0000 7fa6ecc2e640 1 -- 172.21.15.57:0/1894081552 learned_addr learned my addr 172.21.15.57:0/1894081552 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:14.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.904+0000 7f9743380640 1 -- 172.21.15.57:0/1045047300 wait complete. 2023-10-30T14:14:14.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.908+0000 7fa6e7fff640 1 --2- 172.21.15.57:0/1894081552 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e806c480 0x7fa6e81f2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.908+0000 7fa6ecc2e640 1 -- 172.21.15.57:0/1894081552 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e806c480 msgr2=0x7fa6e81f2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.908+0000 7fa6ecc2e640 1 --2- 172.21.15.57:0/1894081552 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e806c480 0x7fa6e81f2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.908+0000 7fa6ecc2e640 1 -- 172.21.15.57:0/1894081552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6e8153b10 msgr2=0x7fa6e80b3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.908+0000 7fa6ecc2e640 1 --2- 172.21.15.57:0/1894081552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6e8153b10 0x7fa6e80b3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.908+0000 7fa6ecc2e640 1 -- 172.21.15.57:0/1894081552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6dc009420 con 0x7fa6e8155d10 2023-10-30T14:14:14.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.908+0000 7fa6e7fff640 1 --2- 172.21.15.57:0/1894081552 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e806c480 0x7fa6e81f2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:14:14.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.908+0000 7fa6e77fe640 1 --2- 172.21.15.57:0/1894081552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6e8153b10 0x7fa6e80b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:14.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.908+0000 7f9743380640 1 Processor -- start 2023-10-30T14:14:14.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.908+0000 7f9743380640 1 -- start start 2023-10-30T14:14:14.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.908+0000 7fa6ecc2e640 1 --2- 172.21.15.57:0/1894081552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6e8155d10 0x7fa6e815bd20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa6e000efc0 tx=0x7fa6e000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.908+0000 7f9743380640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f972c005190 0x7f973c0b7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.908+0000 7fa6e4ff9640 1 -- 172.21.15.57:0/1894081552 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e0014480 con 0x7fa6e8155d10 2023-10-30T14:14:14.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.908+0000 7fa6e4ff9640 1 -- 172.21.15.57:0/1894081552 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e000f040 con 0x7fa6e8155d10 2023-10-30T14:14:14.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.908+0000 7f9743380640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f973c0b92c0 0x7f973c1ea750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.908+0000 7f9743380640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f973c0c9700 0x7f973c14ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.908+0000 7fa6e4ff9640 1 -- 172.21.15.57:0/1894081552 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e0014820 con 0x7fa6e8155d10 2023-10-30T14:14:14.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.908+0000 7fa6ee691640 1 -- 172.21.15.57:0/1894081552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e81f2100 con 0x7fa6e8155d10 2023-10-30T14:14:14.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.912+0000 7fa6ee691640 1 -- 172.21.15.57:0/1894081552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e8157120 con 0x7fa6e8155d10 2023-10-30T14:14:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.912+0000 7f9743380640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c1570e0 con 0x7f972c005190 2023-10-30T14:14:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.912+0000 7f9743380640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c0b9720 con 0x7f973c0c9700 2023-10-30T14:14:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.912+0000 7f9743380640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f973c156b20 con 0x7f973c0b92c0 2023-10-30T14:14:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.912+0000 7f974111c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f972c005190 0x7f973c0b7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.912+0000 7f974191d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f973c0c9700 0x7f973c14ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.912+0000 7f974111c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f972c005190 0x7f973c0b7f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39932/0 (socket says 172.21.15.57:39932) 2023-10-30T14:14:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.912+0000 7f974111c640 1 -- 172.21.15.57:0/1055423189 learned_addr learned my addr 172.21.15.57:0/1055423189 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.912+0000 7f974091b640 1 --2- 172.21.15.57:0/1055423189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f973c0b92c0 0x7f973c1ea750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.912+0000 7f974111c640 1 -- 172.21.15.57:0/1055423189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f973c0b92c0 msgr2=0x7f973c1ea750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.912+0000 7f974111c640 1 --2- 172.21.15.57:0/1055423189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f973c0b92c0 0x7f973c1ea750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.912+0000 7f974111c640 1 -- 172.21.15.57:0/1055423189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f973c0c9700 msgr2=0x7f973c14ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:14.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.912+0000 7f974111c640 1 --2- 172.21.15.57:0/1055423189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f973c0c9700 0x7f973c14ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:14.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.912+0000 7f974111c640 1 -- 172.21.15.57:0/1055423189 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c156e00 con 0x7f972c005190 2023-10-30T14:14:14.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.916+0000 7f974111c640 1 --2- 172.21.15.57:0/1055423189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f972c005190 0x7f973c0b7f00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f97300078c0 tx=0x7f973000a7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.916+0000 7fa6e4ff9640 1 -- 172.21.15.57:0/1894081552 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa6e0014980 con 0x7fa6e8155d10 2023-10-30T14:14:14.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.916+0000 7f9739ffb640 1 -- 172.21.15.57:0/1055423189 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97300133f0 con 0x7f972c005190 2023-10-30T14:14:14.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.916+0000 7f9739ffb640 1 -- 172.21.15.57:0/1055423189 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9730013550 con 0x7f972c005190 2023-10-30T14:14:14.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.916+0000 7f9739ffb640 1 -- 172.21.15.57:0/1055423189 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97300138c0 con 0x7f972c005190 2023-10-30T14:14:14.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.916+0000 7f9743380640 1 -- 172.21.15.57:0/1055423189 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f973c1e9f50 con 0x7f972c005190 2023-10-30T14:14:14.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.916+0000 7f9743380640 1 -- 172.21.15.57:0/1055423189 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f973c154d70 con 0x7f972c005190 2023-10-30T14:14:14.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.916+0000 7f9743380640 1 -- 172.21.15.57:0/1055423189 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f973c1e9db0 con 0x7f972c005190 2023-10-30T14:14:14.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.916+0000 7fa6e4ff9640 1 --2- 172.21.15.57:0/1894081552 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa6d40664a0 0x7fa6d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.916+0000 7fa6e4ff9640 1 -- 172.21.15.57:0/1894081552 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6e0056270 con 0x7fa6e8155d10 2023-10-30T14:14:14.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.916+0000 7fa6e7fff640 1 --2- 172.21.15.57:0/1894081552 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa6d40664a0 0x7fa6d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.920+0000 7fa6ee691640 1 -- 172.21.15.57:0/1894081552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e81f2620 con 0x7fa6e8155d10 2023-10-30T14:14:14.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.920+0000 7fa6e7fff640 1 --2- 172.21.15.57:0/1894081552 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa6d40664a0 0x7fa6d4068960 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8001380 tx=0x7fa6d8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.924+0000 7f9739ffb640 1 -- 172.21.15.57:0/1055423189 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9730013a20 con 0x7f972c005190 2023-10-30T14:14:14.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.924+0000 7f9739ffb640 1 --2- 172.21.15.57:0/1055423189 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f97200664a0 0x7f9720068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:14.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.924+0000 7f9739ffb640 1 -- 172.21.15.57:0/1055423189 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97300851a0 con 0x7f972c005190 2023-10-30T14:14:14.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.924+0000 7f974091b640 1 --2- 172.21.15.57:0/1055423189 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f97200664a0 0x7f9720068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:14.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.928+0000 7f974091b640 1 --2- 172.21.15.57:0/1055423189 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f97200664a0 0x7f9720068960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f973c14fd10 tx=0x7f9728008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:14.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.932+0000 7fa6e4ff9640 1 -- 172.21.15.57:0/1894081552 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa6e81f2620 con 0x7fa6e8155d10 2023-10-30T14:14:14.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:14.944+0000 7f9739ffb640 1 -- 172.21.15.57:0/1055423189 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9730052790 con 0x7f972c005190 2023-10-30T14:14:15.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.124+0000 7f37c8879640 1 -- 172.21.15.57:0/3704121704 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f37c41ee960 con 0x7f37c40709f0 2023-10-30T14:14:15.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.124+0000 7f37bb7fe640 1 -- 172.21.15.57:0/3704121704 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f37bc051fe0 con 0x7f37c40709f0 2023-10-30T14:14:15.130 INFO:teuthology.orchestra.run.smithi057.stdout:55834574856 2023-10-30T14:14:15.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.128+0000 7f37c8879640 1 -- 172.21.15.57:0/3704121704 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f37a4066520 msgr2=0x7f37a40689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.128+0000 7f37c8879640 1 --2- 172.21.15.57:0/3704121704 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f37a4066520 0x7f37a40689e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f37b40252a0 tx=0x7f37b4004ac0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.128+0000 7f37c8879640 1 -- 172.21.15.57:0/3704121704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37c40709f0 msgr2=0x7f37c40c24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.128+0000 7f37c8879640 1 --2- 172.21.15.57:0/3704121704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37c40709f0 0x7f37c40c24b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f37c40c89f0 tx=0x7f37bc00f8a0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.128+0000 7f37c8879640 1 -- 172.21.15.57:0/3704121704 shutdown_connections 2023-10-30T14:14:15.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.128+0000 7f37c8879640 1 --2- 172.21.15.57:0/3704121704 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f37a4066520 0x7f37a40689e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.128+0000 7f37c8879640 1 --2- 172.21.15.57:0/3704121704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37c40709f0 0x7f37c40c24b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.128+0000 7f37c8879640 1 --2- 172.21.15.57:0/3704121704 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37c41eb190 0x7f37c41ee420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.128+0000 7f37c8879640 1 --2- 172.21.15.57:0/3704121704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b0003770 0x7f37c4148a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.128+0000 7f37c8879640 1 -- 172.21.15.57:0/3704121704 >> 172.21.15.57:0/3704121704 conn(0x7f37c406d810 msgr2=0x7f37c40ca190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:15.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.128+0000 7f37c8879640 1 -- 172.21.15.57:0/3704121704 shutdown_connections 2023-10-30T14:14:15.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.128+0000 7f37c8879640 1 -- 172.21.15.57:0/3704121704 wait complete. 2023-10-30T14:14:15.165 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.1 2023-10-30T14:14:15.165 DEBUG:teuthology.parallel:result is None 2023-10-30T14:14:15.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.180+0000 7f732e7fc640 1 -- 172.21.15.57:0/1206462734 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f73380c6dc0 con 0x7f733806e330 2023-10-30T14:14:15.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.180+0000 7f7334ff9640 1 -- 172.21.15.57:0/1206462734 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7330054df0 con 0x7f733806e330 2023-10-30T14:14:15.187 INFO:teuthology.orchestra.run.smithi057.stdout:55834574857 2023-10-30T14:14:15.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.184+0000 7f732e7fc640 1 -- 172.21.15.57:0/1206462734 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f73180664f0 msgr2=0x7f73180689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.184+0000 7f732e7fc640 1 --2- 172.21.15.57:0/1206462734 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f73180664f0 0x7f73180689b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f73280b2000 tx=0x7f73280aa070 comp rx=0 tx=0).stop 2023-10-30T14:14:15.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.184+0000 7f732e7fc640 1 -- 172.21.15.57:0/1206462734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733806e330 msgr2=0x7f7338143770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.184+0000 7f732e7fc640 1 --2- 172.21.15.57:0/1206462734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733806e330 0x7f7338143770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f733000d640 tx=0x7f733000db00 comp rx=0 tx=0).stop 2023-10-30T14:14:15.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.184+0000 7f732e7fc640 1 -- 172.21.15.57:0/1206462734 shutdown_connections 2023-10-30T14:14:15.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.184+0000 7f732e7fc640 1 --2- 172.21.15.57:0/1206462734 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f73180664f0 0x7f73180689b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.184+0000 7f732e7fc640 1 --2- 172.21.15.57:0/1206462734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73380c7070 0x7f733813f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.184+0000 7f732e7fc640 1 --2- 172.21.15.57:0/1206462734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7338071d50 0x7f73381f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.184+0000 7f732e7fc640 1 --2- 172.21.15.57:0/1206462734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733806e330 0x7f7338143770 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.184+0000 7f732e7fc640 1 -- 172.21.15.57:0/1206462734 >> 172.21.15.57:0/1206462734 conn(0x7f733806d810 msgr2=0x7f73380c12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:15.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.184+0000 7f732e7fc640 1 -- 172.21.15.57:0/1206462734 shutdown_connections 2023-10-30T14:14:15.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.184+0000 7f732e7fc640 1 -- 172.21.15.57:0/1206462734 wait complete. 2023-10-30T14:14:15.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.204+0000 7faceebf5640 1 -- 172.21.15.57:0/3050327589 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7face81406a0 con 0x7face806fdf0 2023-10-30T14:14:15.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.204+0000 7facdd7fa640 1 -- 172.21.15.57:0/3050327589 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7facd80ee060 con 0x7face806fdf0 2023-10-30T14:14:15.210 INFO:teuthology.orchestra.run.smithi057.stdout:55834574857 2023-10-30T14:14:15.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.208+0000 7faceebf5640 1 -- 172.21.15.57:0/3050327589 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7facd4066560 msgr2=0x7facd4068a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.208+0000 7faceebf5640 1 --2- 172.21.15.57:0/3050327589 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7facd4066560 0x7facd4068a20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7face4004ef0 tx=0x7face4009040 comp rx=0 tx=0).stop 2023-10-30T14:14:15.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.208+0000 7faceebf5640 1 -- 172.21.15.57:0/3050327589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7face806fdf0 msgr2=0x7face8147450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.208+0000 7faceebf5640 1 --2- 172.21.15.57:0/3050327589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7face806fdf0 0x7face8147450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7facd80ab000 tx=0x7facd8018f40 comp rx=0 tx=0).stop 2023-10-30T14:14:15.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.208+0000 7fbc914d6640 1 -- 172.21.15.57:0/2304719666 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fbc8c1e1250 con 0x7fbc74003980 2023-10-30T14:14:15.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.208+0000 7faceebf5640 1 -- 172.21.15.57:0/3050327589 shutdown_connections 2023-10-30T14:14:15.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.208+0000 7faceebf5640 1 --2- 172.21.15.57:0/3050327589 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7facd4066560 0x7facd4068a20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.208+0000 7faceebf5640 1 --2- 172.21.15.57:0/3050327589 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7face80cb790 0x7face81409d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.208+0000 7faceebf5640 1 --2- 172.21.15.57:0/3050327589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7face806fdf0 0x7face8147450 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.208+0000 7faceebf5640 1 --2- 172.21.15.57:0/3050327589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face806c420 0x7face8148690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.208+0000 7faceebf5640 1 -- 172.21.15.57:0/3050327589 >> 172.21.15.57:0/3050327589 conn(0x7face80cac70 msgr2=0x7face814e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:15.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.208+0000 7faceebf5640 1 -- 172.21.15.57:0/3050327589 shutdown_connections 2023-10-30T14:14:15.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.208+0000 7faceebf5640 1 -- 172.21.15.57:0/3050327589 wait complete. 2023-10-30T14:14:15.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.208+0000 7fbc737fe640 1 -- 172.21.15.57:0/2304719666 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbc8c1e1250 con 0x7fbc74003980 2023-10-30T14:14:15.216 INFO:teuthology.orchestra.run.smithi057.stdout:55834574856 2023-10-30T14:14:15.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.212+0000 7fbc914d6640 1 -- 172.21.15.57:0/2304719666 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbc78066520 msgr2=0x7fbc780689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.212+0000 7fbc914d6640 1 --2- 172.21.15.57:0/2304719666 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbc78066520 0x7fbc780689e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c156d90 tx=0x7fbc80009240 comp rx=0 tx=0).stop 2023-10-30T14:14:15.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.212+0000 7fbc914d6640 1 -- 172.21.15.57:0/2304719666 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc74003980 msgr2=0x7fbc8c1e25f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.212+0000 7fbc914d6640 1 --2- 172.21.15.57:0/2304719666 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc74003980 0x7fbc8c1e25f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c005040 tx=0x7fbc74005800 comp rx=0 tx=0).stop 2023-10-30T14:14:15.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.212+0000 7fbc914d6640 1 -- 172.21.15.57:0/2304719666 shutdown_connections 2023-10-30T14:14:15.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.212+0000 7fbc914d6640 1 --2- 172.21.15.57:0/2304719666 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbc78066520 0x7fbc780689e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.212+0000 7fbc914d6640 1 --2- 172.21.15.57:0/2304719666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc8c1509c0 0x7fbc8c1f3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.212+0000 7fbc914d6640 1 --2- 172.21.15.57:0/2304719666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc8c0c3c30 0x7fbc8c1f2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.212+0000 7fbc914d6640 1 --2- 172.21.15.57:0/2304719666 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc74003980 0x7fbc8c1e25f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.212+0000 7fbc914d6640 1 -- 172.21.15.57:0/2304719666 >> 172.21.15.57:0/2304719666 conn(0x7fbc8c0cac70 msgr2=0x7fbc8c1513c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:15.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.212+0000 7fbc914d6640 1 -- 172.21.15.57:0/2304719666 shutdown_connections 2023-10-30T14:14:15.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.212+0000 7fbc914d6640 1 -- 172.21.15.57:0/2304719666 wait complete. 2023-10-30T14:14:15.221 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2023-10-30T14:14:15.222 DEBUG:teuthology.parallel:result is None 2023-10-30T14:14:15.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.224+0000 7f9ac60c9640 1 -- 172.21.15.57:0/1155034739 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f9ac00ca170 con 0x7f9ac00ca8e0 2023-10-30T14:14:15.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.224+0000 7f9abcff9640 1 -- 172.21.15.57:0/1155034739 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9aa8012ed0 con 0x7f9ac00ca8e0 2023-10-30T14:14:15.230 INFO:teuthology.orchestra.run.smithi057.stdout:55834574856 2023-10-30T14:14:15.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.228+0000 7f9ac60c9640 1 -- 172.21.15.57:0/1155034739 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9aa00664d0 msgr2=0x7f9aa0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.228+0000 7f9ac60c9640 1 --2- 172.21.15.57:0/1155034739 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9aa00664d0 0x7f9aa0068990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0009870 tx=0x7f9ab00a7000 comp rx=0 tx=0).stop 2023-10-30T14:14:15.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.228+0000 7f9ac60c9640 1 -- 172.21.15.57:0/1155034739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ac00ca8e0 msgr2=0x7f9ac00cd110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.228+0000 7f9ac60c9640 1 --2- 172.21.15.57:0/1155034739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ac00ca8e0 0x7f9ac00cd110 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9aa800d5f0 tx=0x7f9aa800dab0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.228+0000 7f9ac60c9640 1 -- 172.21.15.57:0/1155034739 shutdown_connections 2023-10-30T14:14:15.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.228+0000 7f9ac60c9640 1 --2- 172.21.15.57:0/1155034739 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9aa00664d0 0x7f9aa0068990 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.228+0000 7f9ac60c9640 1 --2- 172.21.15.57:0/1155034739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ac0153b10 0x7f9ac00715f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.228+0000 7f9ac60c9640 1 --2- 172.21.15.57:0/1155034739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ac00ca8e0 0x7f9ac00cd110 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9aa800d5f0 tx=0x7f9aa800dab0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.228+0000 7f9ac60c9640 1 --2- 172.21.15.57:0/1155034739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ac006c480 0x7f9ac0150980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.228+0000 7f9ac60c9640 1 -- 172.21.15.57:0/1155034739 >> 172.21.15.57:0/1155034739 conn(0x7f9ac006a240 msgr2=0x7f9ac00cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:15.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.228+0000 7f9ac60c9640 1 -- 172.21.15.57:0/1155034739 shutdown_connections 2023-10-30T14:14:15.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.228+0000 7f9ac60c9640 1 -- 172.21.15.57:0/1155034739 wait complete. 2023-10-30T14:14:15.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.232+0000 7ff234971640 1 -- 172.21.15.57:0/3431699182 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff2280328f0 con 0x7ff2280464d0 2023-10-30T14:14:15.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.232+0000 7ff21bfff640 1 -- 172.21.15.57:0/3431699182 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff2240ee720 con 0x7ff2280464d0 2023-10-30T14:14:15.237 INFO:teuthology.orchestra.run.smithi057.stdout:55834574856 2023-10-30T14:14:15.239 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.0 2023-10-30T14:14:15.240 DEBUG:teuthology.parallel:result is None 2023-10-30T14:14:15.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.232+0000 7ff234971640 1 -- 172.21.15.57:0/3431699182 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff2140664a0 msgr2=0x7ff214068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.232+0000 7ff234971640 1 --2- 172.21.15.57:0/3431699182 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff2140664a0 0x7ff214068960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff230068f50 tx=0x7ff230075610 comp rx=0 tx=0).stop 2023-10-30T14:14:15.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.236+0000 7ff234971640 1 -- 172.21.15.57:0/3431699182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2280464d0 msgr2=0x7ff2280d23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.236+0000 7ff234971640 1 --2- 172.21.15.57:0/3431699182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2280464d0 0x7ff2280d23b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff2240ac000 tx=0x7ff224019260 comp rx=0 tx=0).stop 2023-10-30T14:14:15.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.236+0000 7ff234971640 1 -- 172.21.15.57:0/3431699182 shutdown_connections 2023-10-30T14:14:15.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.236+0000 7ff234971640 1 --2- 172.21.15.57:0/3431699182 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff2140664a0 0x7ff214068960 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.236+0000 7ff234971640 1 --2- 172.21.15.57:0/3431699182 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff22804c220 0x7ff2280dddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.236+0000 7ff234971640 1 --2- 172.21.15.57:0/3431699182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2280468b0 0x7ff228177ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.236+0000 7ff234971640 1 --2- 172.21.15.57:0/3431699182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2280464d0 0x7ff2280d23b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.236+0000 7ff234971640 1 -- 172.21.15.57:0/3431699182 >> 172.21.15.57:0/3431699182 conn(0x7ff228051410 msgr2=0x7ff2280d8750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:15.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.236+0000 7ff234971640 1 -- 172.21.15.57:0/3431699182 shutdown_connections 2023-10-30T14:14:15.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.236+0000 7ff234971640 1 -- 172.21.15.57:0/3431699182 wait complete. 2023-10-30T14:14:15.245 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.7 2023-10-30T14:14:15.245 DEBUG:teuthology.parallel:result is None 2023-10-30T14:14:15.258 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.2 2023-10-30T14:14:15.258 DEBUG:teuthology.parallel:result is None 2023-10-30T14:14:15.262 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2023-10-30T14:14:15.262 DEBUG:teuthology.parallel:result is None 2023-10-30T14:14:15.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.268+0000 7fa6ee691640 1 -- 172.21.15.57:0/1894081552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fa6e81f2620 con 0x7fa6e8155d10 2023-10-30T14:14:15.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.268+0000 7fa6e4ff9640 1 -- 172.21.15.57:0/1894081552 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa6e81f2620 con 0x7fa6e8155d10 2023-10-30T14:14:15.275 INFO:teuthology.orchestra.run.smithi057.stdout:55834574857 2023-10-30T14:14:15.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.272+0000 7fa6ee691640 1 -- 172.21.15.57:0/1894081552 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa6d40664a0 msgr2=0x7fa6d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.272+0000 7fa6ee691640 1 --2- 172.21.15.57:0/1894081552 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa6d40664a0 0x7fa6d4068960 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8001380 tx=0x7fa6d8009040 comp rx=0 tx=0).stop 2023-10-30T14:14:15.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.272+0000 7fa6ee691640 1 -- 172.21.15.57:0/1894081552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6e8155d10 msgr2=0x7fa6e815bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.272+0000 7fa6ee691640 1 --2- 172.21.15.57:0/1894081552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6e8155d10 0x7fa6e815bd20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa6e000efc0 tx=0x7fa6e000c490 comp rx=0 tx=0).stop 2023-10-30T14:14:15.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.272+0000 7fa6ee691640 1 -- 172.21.15.57:0/1894081552 shutdown_connections 2023-10-30T14:14:15.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.272+0000 7fa6ee691640 1 --2- 172.21.15.57:0/1894081552 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa6d40664a0 0x7fa6d4068960 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.272+0000 7fa6ee691640 1 --2- 172.21.15.57:0/1894081552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6e8155d10 0x7fa6e815bd20 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.272+0000 7fa6ee691640 1 --2- 172.21.15.57:0/1894081552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6e8153b10 0x7fa6e80b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.272+0000 7fa6ee691640 1 --2- 172.21.15.57:0/1894081552 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e806c480 0x7fa6e81f2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.272+0000 7fa6ee691640 1 -- 172.21.15.57:0/1894081552 >> 172.21.15.57:0/1894081552 conn(0x7fa6e806a240 msgr2=0x7fa6e806a620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:15.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.272+0000 7fa6ee691640 1 -- 172.21.15.57:0/1894081552 shutdown_connections 2023-10-30T14:14:15.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.272+0000 7fa6ee691640 1 -- 172.21.15.57:0/1894081552 wait complete. 2023-10-30T14:14:15.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.280+0000 7f9743380640 1 -- 172.21.15.57:0/1055423189 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f973c1e9db0 con 0x7f972c005190 2023-10-30T14:14:15.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.280+0000 7f9739ffb640 1 -- 172.21.15.57:0/1055423189 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f973c1e9db0 con 0x7f972c005190 2023-10-30T14:14:15.284 INFO:teuthology.orchestra.run.smithi057.stdout:55834574855 2023-10-30T14:14:15.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.280+0000 7f9743380640 1 -- 172.21.15.57:0/1055423189 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f97200664a0 msgr2=0x7f9720068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.280+0000 7f9743380640 1 --2- 172.21.15.57:0/1055423189 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f97200664a0 0x7f9720068960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f973c14fd10 tx=0x7f9728008040 comp rx=0 tx=0).stop 2023-10-30T14:14:15.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.280+0000 7f9743380640 1 -- 172.21.15.57:0/1055423189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f972c005190 msgr2=0x7f973c0b7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.280+0000 7f9743380640 1 --2- 172.21.15.57:0/1055423189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f972c005190 0x7f973c0b7f00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f97300078c0 tx=0x7f973000a7b0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.280+0000 7f9743380640 1 -- 172.21.15.57:0/1055423189 shutdown_connections 2023-10-30T14:14:15.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.280+0000 7f9743380640 1 --2- 172.21.15.57:0/1055423189 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f97200664a0 0x7f9720068960 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.280+0000 7f9743380640 1 --2- 172.21.15.57:0/1055423189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f973c0c9700 0x7f973c14ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.280+0000 7f9743380640 1 --2- 172.21.15.57:0/1055423189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f973c0b92c0 0x7f973c1ea750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.280+0000 7f9743380640 1 --2- 172.21.15.57:0/1055423189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f972c005190 0x7f973c0b7f00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.280+0000 7f9743380640 1 -- 172.21.15.57:0/1055423189 >> 172.21.15.57:0/1055423189 conn(0x7f973c0cac70 msgr2=0x7f973c150890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:15.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.280+0000 7f9743380640 1 -- 172.21.15.57:0/1055423189 shutdown_connections 2023-10-30T14:14:15.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.280+0000 7f9743380640 1 -- 172.21.15.57:0/1055423189 wait complete. 2023-10-30T14:14:15.301 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.5 2023-10-30T14:14:15.301 DEBUG:teuthology.parallel:result is None 2023-10-30T14:14:15.306 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2023-10-30T14:14:15.307 DEBUG:teuthology.parallel:result is None 2023-10-30T14:14:15.307 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T14:14:15.307 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:14:15.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.488+0000 7fb74f06b640 1 Processor -- start 2023-10-30T14:14:15.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.488+0000 7fb74f06b640 1 -- start start 2023-10-30T14:14:15.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.488+0000 7fb74f06b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb748153c80 0x7fb74815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:15.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.488+0000 7fb74f06b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb7480c42a0 0x7fb7480ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:15.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.488+0000 7fb74f06b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb748159f20 con 0x7fb7480c4a10 2023-10-30T14:14:15.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.488+0000 7fb74f06b640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74815dec0 con 0x7fb748153c80 2023-10-30T14:14:15.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.488+0000 7fb74f06b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7481599a0 con 0x7fb7480c42a0 2023-10-30T14:14:15.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.488+0000 7fb74ce07640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb748153c80 0x7fb74815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:15.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.488+0000 7fb747fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb7480c4a10 0x7fb74815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:37708/0 (socket says 172.21.15.57:37708) 2023-10-30T14:14:15.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.488+0000 7fb747fff640 1 -- 172.21.15.57:0/2420363175 learned_addr learned my addr 172.21.15.57:0/2420363175 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:15.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.488+0000 7fb74d608640 1 --2- 172.21.15.57:0/2420363175 >> v2:172.21.15.88:3300/0 conn(0x7fb7480c42a0 0x7fb7480ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:15.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.488+0000 7fb74ce07640 1 -- 172.21.15.57:0/2420363175 >> v2:172.21.15.88:3300/0 conn(0x7fb7480c42a0 msgr2=0x7fb7480ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:15.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.488+0000 7fb74ce07640 1 --2- 172.21.15.57:0/2420363175 >> v2:172.21.15.88:3300/0 conn(0x7fb7480c42a0 0x7fb7480ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.488+0000 7fb74ce07640 1 -- 172.21.15.57:0/2420363175 >> v1:172.21.15.57:6789/0 conn(0x7fb7480c4a10 legacy=0x7fb74815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.488+0000 7fb74ce07640 1 -- 172.21.15.57:0/2420363175 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74815d620 con 0x7fb748153c80 2023-10-30T14:14:15.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.488+0000 7fb7477fe640 1 -- 172.21.15.57:0/2420363175 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707895541 0 0) 0x7fb748159f20 con 0x7fb7480c4a10 2023-10-30T14:14:15.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.492+0000 7fb74ce07640 1 --2- 172.21.15.57:0/2420363175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb748153c80 0x7fb74815cf90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb7480b1d40 tx=0x7fb73c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae7565bc01015c7c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:15.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.492+0000 7fb7477fe640 1 -- 172.21.15.57:0/2420363175 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c0a8950 con 0x7fb748153c80 2023-10-30T14:14:15.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.492+0000 7fb7477fe640 1 -- 172.21.15.57:0/2420363175 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c0a8ab0 con 0x7fb748153c80 2023-10-30T14:14:15.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.492+0000 7fb7477fe640 1 -- 172.21.15.57:0/2420363175 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c0a8e40 con 0x7fb748153c80 2023-10-30T14:14:15.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.492+0000 7fb74f06b640 1 -- 172.21.15.57:0/2420363175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb748153c80 msgr2=0x7fb74815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.492+0000 7fb74f06b640 1 --2- 172.21.15.57:0/2420363175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb748153c80 0x7fb74815cf90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb7480b1d40 tx=0x7fb73c01ec50 comp rx=0 tx=0).stop 2023-10-30T14:14:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.492+0000 7fb74f06b640 1 -- 172.21.15.57:0/2420363175 shutdown_connections 2023-10-30T14:14:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.492+0000 7fb74f06b640 1 --2- 172.21.15.57:0/2420363175 >> v2:172.21.15.88:3300/0 conn(0x7fb7480c42a0 0x7fb7480ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.492+0000 7fb74f06b640 1 --2- 172.21.15.57:0/2420363175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb748153c80 0x7fb74815cf90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.492+0000 7fb74f06b640 1 -- 172.21.15.57:0/2420363175 >> 172.21.15.57:0/2420363175 conn(0x7fb7480715c0 msgr2=0x7fb7480719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.492+0000 7fb74f06b640 1 -- 172.21.15.57:0/2420363175 shutdown_connections 2023-10-30T14:14:15.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.492+0000 7fb74f06b640 1 -- 172.21.15.57:0/2420363175 wait complete. 2023-10-30T14:14:15.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74f06b640 1 Processor -- start 2023-10-30T14:14:15.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74f06b640 1 -- start start 2023-10-30T14:14:15.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74f06b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7480c42a0 0x7fb74814df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:15.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74f06b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7480c4a10 0x7fb7481ef990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:15.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74f06b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb748153c80 0x7fb74814a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:15.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74f06b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74814be40 con 0x7fb7480c42a0 2023-10-30T14:14:15.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74f06b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74815c7c0 con 0x7fb7480c4a10 2023-10-30T14:14:15.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74f06b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74815caa0 con 0x7fb748153c80 2023-10-30T14:14:15.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74ce07640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7480c42a0 0x7fb74814df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:15.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74ce07640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7480c42a0 0x7fb74814df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39962/0 (socket says 172.21.15.57:39962) 2023-10-30T14:14:15.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74ce07640 1 -- 172.21.15.57:0/3023769987 learned_addr learned my addr 172.21.15.57:0/3023769987 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:15.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb747fff640 1 --2- 172.21.15.57:0/3023769987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7480c4a10 0x7fb7481ef990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:15.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74d608640 1 --2- 172.21.15.57:0/3023769987 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb748153c80 0x7fb74814a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:15.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74ce07640 1 -- 172.21.15.57:0/3023769987 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb748153c80 msgr2=0x7fb74814a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74ce07640 1 --2- 172.21.15.57:0/3023769987 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb748153c80 0x7fb74814a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74ce07640 1 -- 172.21.15.57:0/3023769987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7480c4a10 msgr2=0x7fb7481ef990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74ce07640 1 --2- 172.21.15.57:0/3023769987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7480c4a10 0x7fb7481ef990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74ce07640 1 -- 172.21.15.57:0/3023769987 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73c009420 con 0x7fb7480c42a0 2023-10-30T14:14:15.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74ce07640 1 --2- 172.21.15.57:0/3023769987 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7480c42a0 0x7fb74814df30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0b3040 tx=0x7fb73c01b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:15.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb7457fa640 1 -- 172.21.15.57:0/3023769987 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c0b0df0 con 0x7fb7480c42a0 2023-10-30T14:14:15.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb7457fa640 1 -- 172.21.15.57:0/3023769987 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c0b9410 con 0x7fb7480c42a0 2023-10-30T14:14:15.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74f06b640 1 -- 172.21.15.57:0/3023769987 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74815cd80 con 0x7fb7480c42a0 2023-10-30T14:14:15.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb7457fa640 1 -- 172.21.15.57:0/3023769987 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c0b97c0 con 0x7fb7480c42a0 2023-10-30T14:14:15.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74f06b640 1 -- 172.21.15.57:0/3023769987 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7481ef470 con 0x7fb7480c42a0 2023-10-30T14:14:15.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.496+0000 7fb74f06b640 1 -- 172.21.15.57:0/3023769987 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7481ef660 con 0x7fb7480c42a0 2023-10-30T14:14:15.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.500+0000 7fb7457fa640 1 -- 172.21.15.57:0/3023769987 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb73c0b7040 con 0x7fb7480c42a0 2023-10-30T14:14:15.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.500+0000 7fb7457fa640 1 --2- 172.21.15.57:0/3023769987 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb72c0664a0 0x7fb72c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:15.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.500+0000 7fb7457fa640 1 -- 172.21.15.57:0/3023769987 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb73c121a50 con 0x7fb7480c42a0 2023-10-30T14:14:15.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.500+0000 7fb747fff640 1 --2- 172.21.15.57:0/3023769987 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb72c0664a0 0x7fb72c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:15.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.500+0000 7fb747fff640 1 --2- 172.21.15.57:0/3023769987 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb72c0664a0 0x7fb72c068960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb74814f370 tx=0x7fb734007b60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:15.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.504+0000 7fb7457fa640 1 -- 172.21.15.57:0/3023769987 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb73c0eee30 con 0x7fb7480c42a0 2023-10-30T14:14:15.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.656+0000 7fb74f06b640 1 -- 172.21.15.57:0/3023769987 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb74815dbe0 con 0x7fb72c0664a0 2023-10-30T14:14:15.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.676+0000 7fb7457fa640 1 -- 172.21.15.57:0/3023769987 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264525 (secure 0 0 0) 0x7fb7340071f0 con 0x7fb72c0664a0 2023-10-30T14:14:15.686 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:15.687 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-30T14:14:15.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.680+0000 7fb74f06b640 1 -- 172.21.15.57:0/3023769987 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb72c0664a0 msgr2=0x7fb72c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.680+0000 7fb74f06b640 1 --2- 172.21.15.57:0/3023769987 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb72c0664a0 0x7fb72c068960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb74814f370 tx=0x7fb734007b60 comp rx=0 tx=0).stop 2023-10-30T14:14:15.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.680+0000 7fb74f06b640 1 -- 172.21.15.57:0/3023769987 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7480c42a0 msgr2=0x7fb74814df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.680+0000 7fb74f06b640 1 --2- 172.21.15.57:0/3023769987 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7480c42a0 0x7fb74814df30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0b3040 tx=0x7fb73c01b9f0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.680+0000 7fb74f06b640 1 -- 172.21.15.57:0/3023769987 shutdown_connections 2023-10-30T14:14:15.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.680+0000 7fb74f06b640 1 --2- 172.21.15.57:0/3023769987 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb72c0664a0 0x7fb72c068960 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.680+0000 7fb74f06b640 1 --2- 172.21.15.57:0/3023769987 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb748153c80 0x7fb74814a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.680+0000 7fb74f06b640 1 --2- 172.21.15.57:0/3023769987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7480c4a10 0x7fb7481ef990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.680+0000 7fb74f06b640 1 --2- 172.21.15.57:0/3023769987 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7480c42a0 0x7fb74814df30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.680+0000 7fb74f06b640 1 -- 172.21.15.57:0/3023769987 >> 172.21.15.57:0/3023769987 conn(0x7fb7480715c0 msgr2=0x7fb748157810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:15.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.680+0000 7fb74f06b640 1 -- 172.21.15.57:0/3023769987 shutdown_connections 2023-10-30T14:14:15.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.680+0000 7fb74f06b640 1 -- 172.21.15.57:0/3023769987 wait complete. 2023-10-30T14:14:15.706 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-30T14:14:14.496045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227368,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754747352,"statfs":{"total":773094113280,"available":772861288448,"internally_reserved":0,"allocated":2990080,"data_stored":1790108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":229756473},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.916605"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757043+0000","last_change":"2023-10-30T14:14:04.651413+0000","last_active":"2023-10-30T14:14:05.757043+0000","last_peered":"2023-10-30T14:14:05.757043+0000","last_clean":"2023-10-30T14:14:05.757043+0000","last_became_active":"2023-10-30T14:14:04.650882+0000","last_became_peered":"2023-10-30T14:14:04.650882+0000","last_unstale":"2023-10-30T14:14:05.757043+0000","last_undegraded":"2023-10-30T14:14:05.757043+0000","last_fullsized":"2023-10-30T14:14:05.757043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:30:08.122326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585036+0000","last_change":"2023-10-30T14:14:04.588523+0000","last_active":"2023-10-30T14:14:05.585036+0000","last_peered":"2023-10-30T14:14:05.585036+0000","last_clean":"2023-10-30T14:14:05.585036+0000","last_became_active":"2023-10-30T14:14:04.587942+0000","last_became_peered":"2023-10-30T14:14:04.587942+0000","last_unstale":"2023-10-30T14:14:05.585036+0000","last_undegraded":"2023-10-30T14:14:05.585036+0000","last_fullsized":"2023-10-30T14:14:05.585036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:16.028564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586543+0000","last_change":"2023-10-30T14:14:04.590976+0000","last_active":"2023-10-30T14:14:05.586543+0000","last_peered":"2023-10-30T14:14:05.586543+0000","last_clean":"2023-10-30T14:14:05.586543+0000","last_became_active":"2023-10-30T14:14:04.590700+0000","last_became_peered":"2023-10-30T14:14:04.590700+0000","last_unstale":"2023-10-30T14:14:05.586543+0000","last_undegraded":"2023-10-30T14:14:05.586543+0000","last_fullsized":"2023-10-30T14:14:05.586543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:49.169325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586525+0000","last_change":"2023-10-30T14:14:04.618603+0000","last_active":"2023-10-30T14:14:05.586525+0000","last_peered":"2023-10-30T14:14:05.586525+0000","last_clean":"2023-10-30T14:14:05.586525+0000","last_became_active":"2023-10-30T14:14:04.617768+0000","last_became_peered":"2023-10-30T14:14:04.617768+0000","last_unstale":"2023-10-30T14:14:05.586525+0000","last_undegraded":"2023-10-30T14:14:05.586525+0000","last_fullsized":"2023-10-30T14:14:05.586525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:16.443567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756348+0000","last_change":"2023-10-30T14:14:04.647842+0000","last_active":"2023-10-30T14:14:05.756348+0000","last_peered":"2023-10-30T14:14:05.756348+0000","last_clean":"2023-10-30T14:14:05.756348+0000","last_became_active":"2023-10-30T14:14:04.644683+0000","last_became_peered":"2023-10-30T14:14:04.644683+0000","last_unstale":"2023-10-30T14:14:05.756348+0000","last_undegraded":"2023-10-30T14:14:05.756348+0000","last_fullsized":"2023-10-30T14:14:05.756348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:51:01.230722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585902+0000","last_change":"2023-10-30T14:14:04.620587+0000","last_active":"2023-10-30T14:14:05.585902+0000","last_peered":"2023-10-30T14:14:05.585902+0000","last_clean":"2023-10-30T14:14:05.585902+0000","last_became_active":"2023-10-30T14:14:04.619335+0000","last_became_peered":"2023-10-30T14:14:04.619335+0000","last_unstale":"2023-10-30T14:14:05.585902+0000","last_undegraded":"2023-10-30T14:14:05.585902+0000","last_fullsized":"2023-10-30T14:14:05.585902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:01.735027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755914+0000","last_change":"2023-10-30T14:14:04.641688+0000","last_active":"2023-10-30T14:14:05.755914+0000","last_peered":"2023-10-30T14:14:05.755914+0000","last_clean":"2023-10-30T14:14:05.755914+0000","last_became_active":"2023-10-30T14:14:04.641323+0000","last_became_peered":"2023-10-30T14:14:04.641323+0000","last_unstale":"2023-10-30T14:14:05.755914+0000","last_undegraded":"2023-10-30T14:14:05.755914+0000","last_fullsized":"2023-10-30T14:14:05.755914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:43:35.285019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805751+0000","last_change":"2023-10-30T14:14:04.618530+0000","last_active":"2023-10-30T14:14:05.805751+0000","last_peered":"2023-10-30T14:14:05.805751+0000","last_clean":"2023-10-30T14:14:05.805751+0000","last_became_active":"2023-10-30T14:14:04.618312+0000","last_became_peered":"2023-10-30T14:14:04.618312+0000","last_unstale":"2023-10-30T14:14:05.805751+0000","last_undegraded":"2023-10-30T14:14:05.805751+0000","last_fullsized":"2023-10-30T14:14:05.805751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:25.089106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757146+0000","last_change":"2023-10-30T14:14:04.650330+0000","last_active":"2023-10-30T14:14:05.757146+0000","last_peered":"2023-10-30T14:14:05.757146+0000","last_clean":"2023-10-30T14:14:05.757146+0000","last_became_active":"2023-10-30T14:14:04.647361+0000","last_became_peered":"2023-10-30T14:14:04.647361+0000","last_unstale":"2023-10-30T14:14:05.757146+0000","last_undegraded":"2023-10-30T14:14:05.757146+0000","last_fullsized":"2023-10-30T14:14:05.757146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:49:57.370980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808090+0000","last_change":"2023-10-30T14:14:04.631540+0000","last_active":"2023-10-30T14:14:05.808090+0000","last_peered":"2023-10-30T14:14:05.808090+0000","last_clean":"2023-10-30T14:14:05.808090+0000","last_became_active":"2023-10-30T14:14:04.630959+0000","last_became_peered":"2023-10-30T14:14:04.630959+0000","last_unstale":"2023-10-30T14:14:05.808090+0000","last_undegraded":"2023-10-30T14:14:05.808090+0000","last_fullsized":"2023-10-30T14:14:05.808090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:49.468429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807762+0000","last_change":"2023-10-30T14:14:04.613800+0000","last_active":"2023-10-30T14:14:05.807762+0000","last_peered":"2023-10-30T14:14:05.807762+0000","last_clean":"2023-10-30T14:14:05.807762+0000","last_became_active":"2023-10-30T14:14:04.612653+0000","last_became_peered":"2023-10-30T14:14:04.612653+0000","last_unstale":"2023-10-30T14:14:05.807762+0000","last_undegraded":"2023-10-30T14:14:05.807762+0000","last_fullsized":"2023-10-30T14:14:05.807762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:44.665588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809700+0000","last_change":"2023-10-30T14:14:04.622483+0000","last_active":"2023-10-30T14:14:05.809700+0000","last_peered":"2023-10-30T14:14:05.809700+0000","last_clean":"2023-10-30T14:14:05.809700+0000","last_became_active":"2023-10-30T14:14:04.621135+0000","last_became_peered":"2023-10-30T14:14:04.621135+0000","last_unstale":"2023-10-30T14:14:05.809700+0000","last_undegraded":"2023-10-30T14:14:05.809700+0000","last_fullsized":"2023-10-30T14:14:05.809700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:52.299412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806054+0000","last_change":"2023-10-30T14:14:04.616357+0000","last_active":"2023-10-30T14:14:05.806054+0000","last_peered":"2023-10-30T14:14:05.806054+0000","last_clean":"2023-10-30T14:14:05.806054+0000","last_became_active":"2023-10-30T14:14:04.616142+0000","last_became_peered":"2023-10-30T14:14:04.616142+0000","last_unstale":"2023-10-30T14:14:05.806054+0000","last_undegraded":"2023-10-30T14:14:05.806054+0000","last_fullsized":"2023-10-30T14:14:05.806054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:32:26.432551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586106+0000","last_change":"2023-10-30T14:14:04.588759+0000","last_active":"2023-10-30T14:14:05.586106+0000","last_peered":"2023-10-30T14:14:05.586106+0000","last_clean":"2023-10-30T14:14:05.586106+0000","last_became_active":"2023-10-30T14:14:04.588452+0000","last_became_peered":"2023-10-30T14:14:04.588452+0000","last_unstale":"2023-10-30T14:14:05.586106+0000","last_undegraded":"2023-10-30T14:14:05.586106+0000","last_fullsized":"2023-10-30T14:14:05.586106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:38:21.206284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584572+0000","last_change":"2023-10-30T14:14:04.621073+0000","last_active":"2023-10-30T14:14:05.584572+0000","last_peered":"2023-10-30T14:14:05.584572+0000","last_clean":"2023-10-30T14:14:05.584572+0000","last_became_active":"2023-10-30T14:14:04.620781+0000","last_became_peered":"2023-10-30T14:14:04.620781+0000","last_unstale":"2023-10-30T14:14:05.584572+0000","last_undegraded":"2023-10-30T14:14:05.584572+0000","last_fullsized":"2023-10-30T14:14:05.584572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:17:10.938195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585712+0000","last_change":"2023-10-30T14:14:04.608942+0000","last_active":"2023-10-30T14:14:05.585712+0000","last_peered":"2023-10-30T14:14:05.585712+0000","last_clean":"2023-10-30T14:14:05.585712+0000","last_became_active":"2023-10-30T14:14:04.608190+0000","last_became_peered":"2023-10-30T14:14:04.608190+0000","last_unstale":"2023-10-30T14:14:05.585712+0000","last_undegraded":"2023-10-30T14:14:05.585712+0000","last_fullsized":"2023-10-30T14:14:05.585712+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:18:39.846841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587056+0000","last_change":"2023-10-30T14:14:04.641876+0000","last_active":"2023-10-30T14:14:05.587056+0000","last_peered":"2023-10-30T14:14:05.587056+0000","last_clean":"2023-10-30T14:14:05.587056+0000","last_became_active":"2023-10-30T14:14:04.641306+0000","last_became_peered":"2023-10-30T14:14:04.641306+0000","last_unstale":"2023-10-30T14:14:05.587056+0000","last_undegraded":"2023-10-30T14:14:05.587056+0000","last_fullsized":"2023-10-30T14:14:05.587056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:10:08.169786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805744+0000","last_change":"2023-10-30T14:14:04.613999+0000","last_active":"2023-10-30T14:14:05.805744+0000","last_peered":"2023-10-30T14:14:05.805744+0000","last_clean":"2023-10-30T14:14:05.805744+0000","last_became_active":"2023-10-30T14:14:04.613644+0000","last_became_peered":"2023-10-30T14:14:04.613644+0000","last_unstale":"2023-10-30T14:14:05.805744+0000","last_undegraded":"2023-10-30T14:14:05.805744+0000","last_fullsized":"2023-10-30T14:14:05.805744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:14:50.232028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756929+0000","last_change":"2023-10-30T14:14:04.650393+0000","last_active":"2023-10-30T14:14:05.756929+0000","last_peered":"2023-10-30T14:14:05.756929+0000","last_clean":"2023-10-30T14:14:05.756929+0000","last_became_active":"2023-10-30T14:14:04.649850+0000","last_became_peered":"2023-10-30T14:14:04.649850+0000","last_unstale":"2023-10-30T14:14:05.756929+0000","last_undegraded":"2023-10-30T14:14:05.756929+0000","last_fullsized":"2023-10-30T14:14:05.756929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:25.174023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808107+0000","last_change":"2023-10-30T14:14:04.625744+0000","last_active":"2023-10-30T14:14:05.808107+0000","last_peered":"2023-10-30T14:14:05.808107+0000","last_clean":"2023-10-30T14:14:05.808107+0000","last_became_active":"2023-10-30T14:14:04.625042+0000","last_became_peered":"2023-10-30T14:14:04.625042+0000","last_unstale":"2023-10-30T14:14:05.808107+0000","last_undegraded":"2023-10-30T14:14:05.808107+0000","last_fullsized":"2023-10-30T14:14:05.808107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:30:36.063455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586236+0000","last_change":"2023-10-30T14:14:04.633942+0000","last_active":"2023-10-30T14:14:05.586236+0000","last_peered":"2023-10-30T14:14:05.586236+0000","last_clean":"2023-10-30T14:14:05.586236+0000","last_became_active":"2023-10-30T14:14:04.633586+0000","last_became_peered":"2023-10-30T14:14:04.633586+0000","last_unstale":"2023-10-30T14:14:05.586236+0000","last_undegraded":"2023-10-30T14:14:05.586236+0000","last_fullsized":"2023-10-30T14:14:05.586236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:42:58.067892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586050+0000","last_change":"2023-10-30T14:14:04.642847+0000","last_active":"2023-10-30T14:14:05.586050+0000","last_peered":"2023-10-30T14:14:05.586050+0000","last_clean":"2023-10-30T14:14:05.586050+0000","last_became_active":"2023-10-30T14:14:04.642244+0000","last_became_peered":"2023-10-30T14:14:04.642244+0000","last_unstale":"2023-10-30T14:14:05.586050+0000","last_undegraded":"2023-10-30T14:14:05.586050+0000","last_fullsized":"2023-10-30T14:14:05.586050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:28.780866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805762+0000","last_change":"2023-10-30T14:14:04.620651+0000","last_active":"2023-10-30T14:14:05.805762+0000","last_peered":"2023-10-30T14:14:05.805762+0000","last_clean":"2023-10-30T14:14:05.805762+0000","last_became_active":"2023-10-30T14:14:04.620343+0000","last_became_peered":"2023-10-30T14:14:04.620343+0000","last_unstale":"2023-10-30T14:14:05.805762+0000","last_undegraded":"2023-10-30T14:14:05.805762+0000","last_fullsized":"2023-10-30T14:14:05.805762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:16:00.557118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587224+0000","last_change":"2023-10-30T14:14:04.610472+0000","last_active":"2023-10-30T14:14:05.587224+0000","last_peered":"2023-10-30T14:14:05.587224+0000","last_clean":"2023-10-30T14:14:05.587224+0000","last_became_active":"2023-10-30T14:14:04.609720+0000","last_became_peered":"2023-10-30T14:14:04.609720+0000","last_unstale":"2023-10-30T14:14:05.587224+0000","last_undegraded":"2023-10-30T14:14:05.587224+0000","last_fullsized":"2023-10-30T14:14:05.587224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:44:28.391589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805906+0000","last_change":"2023-10-30T14:14:04.620548+0000","last_active":"2023-10-30T14:14:05.805906+0000","last_peered":"2023-10-30T14:14:05.805906+0000","last_clean":"2023-10-30T14:14:05.805906+0000","last_became_active":"2023-10-30T14:14:04.620217+0000","last_became_peered":"2023-10-30T14:14:04.620217+0000","last_unstale":"2023-10-30T14:14:05.805906+0000","last_undegraded":"2023-10-30T14:14:05.805906+0000","last_fullsized":"2023-10-30T14:14:05.805906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:40:22.912696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808191+0000","last_change":"2023-10-30T14:14:04.644315+0000","last_active":"2023-10-30T14:14:05.808191+0000","last_peered":"2023-10-30T14:14:05.808191+0000","last_clean":"2023-10-30T14:14:05.808191+0000","last_became_active":"2023-10-30T14:14:04.643707+0000","last_became_peered":"2023-10-30T14:14:04.643707+0000","last_unstale":"2023-10-30T14:14:05.808191+0000","last_undegraded":"2023-10-30T14:14:05.808191+0000","last_fullsized":"2023-10-30T14:14:05.808191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:20:33.544167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584874+0000","last_change":"2023-10-30T14:14:04.597855+0000","last_active":"2023-10-30T14:14:05.584874+0000","last_peered":"2023-10-30T14:14:05.584874+0000","last_clean":"2023-10-30T14:14:05.584874+0000","last_became_active":"2023-10-30T14:14:04.597590+0000","last_became_peered":"2023-10-30T14:14:04.597590+0000","last_unstale":"2023-10-30T14:14:05.584874+0000","last_undegraded":"2023-10-30T14:14:05.584874+0000","last_fullsized":"2023-10-30T14:14:05.584874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:05.905027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806303+0000","last_change":"2023-10-30T14:14:04.624133+0000","last_active":"2023-10-30T14:14:05.806303+0000","last_peered":"2023-10-30T14:14:05.806303+0000","last_clean":"2023-10-30T14:14:05.806303+0000","last_became_active":"2023-10-30T14:14:04.623902+0000","last_became_peered":"2023-10-30T14:14:04.623902+0000","last_unstale":"2023-10-30T14:14:05.806303+0000","last_undegraded":"2023-10-30T14:14:05.806303+0000","last_fullsized":"2023-10-30T14:14:05.806303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:33:10.476461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586328+0000","last_change":"2023-10-30T14:14:04.597875+0000","last_active":"2023-10-30T14:14:05.586328+0000","last_peered":"2023-10-30T14:14:05.586328+0000","last_clean":"2023-10-30T14:14:05.586328+0000","last_became_active":"2023-10-30T14:14:04.597556+0000","last_became_peered":"2023-10-30T14:14:04.597556+0000","last_unstale":"2023-10-30T14:14:05.586328+0000","last_undegraded":"2023-10-30T14:14:05.586328+0000","last_fullsized":"2023-10-30T14:14:05.586328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:08:46.407610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586842+0000","last_change":"2023-10-30T14:14:04.620110+0000","last_active":"2023-10-30T14:14:05.586842+0000","last_peered":"2023-10-30T14:14:05.586842+0000","last_clean":"2023-10-30T14:14:05.586842+0000","last_became_active":"2023-10-30T14:14:04.618430+0000","last_became_peered":"2023-10-30T14:14:04.618430+0000","last_unstale":"2023-10-30T14:14:05.586842+0000","last_undegraded":"2023-10-30T14:14:05.586842+0000","last_fullsized":"2023-10-30T14:14:05.586842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:36.966564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805526+0000","last_change":"2023-10-30T14:14:04.606692+0000","last_active":"2023-10-30T14:14:05.805526+0000","last_peered":"2023-10-30T14:14:05.805526+0000","last_clean":"2023-10-30T14:14:05.805526+0000","last_became_active":"2023-10-30T14:14:04.606340+0000","last_became_peered":"2023-10-30T14:14:04.606340+0000","last_unstale":"2023-10-30T14:14:05.805526+0000","last_undegraded":"2023-10-30T14:14:05.805526+0000","last_fullsized":"2023-10-30T14:14:05.805526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:02:24.518205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807314+0000","last_change":"2023-10-30T14:14:04.613535+0000","last_active":"2023-10-30T14:14:05.807314+0000","last_peered":"2023-10-30T14:14:05.807314+0000","last_clean":"2023-10-30T14:14:05.807314+0000","last_became_active":"2023-10-30T14:14:04.612588+0000","last_became_peered":"2023-10-30T14:14:04.612588+0000","last_unstale":"2023-10-30T14:14:05.807314+0000","last_undegraded":"2023-10-30T14:14:05.807314+0000","last_fullsized":"2023-10-30T14:14:05.807314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:58:23.057988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809168+0000","last_change":"2023-10-30T14:14:04.608434+0000","last_active":"2023-10-30T14:14:05.809168+0000","last_peered":"2023-10-30T14:14:05.809168+0000","last_clean":"2023-10-30T14:14:05.809168+0000","last_became_active":"2023-10-30T14:14:04.607683+0000","last_became_peered":"2023-10-30T14:14:04.607683+0000","last_unstale":"2023-10-30T14:14:05.809168+0000","last_undegraded":"2023-10-30T14:14:05.809168+0000","last_fullsized":"2023-10-30T14:14:05.809168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:17:27.467330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.933073+0000","last_change":"2023-10-30T14:14:03.732084+0000","last_active":"2023-10-30T14:14:06.933073+0000","last_peered":"2023-10-30T14:14:06.933073+0000","last_clean":"2023-10-30T14:14:06.933073+0000","last_became_active":"2023-10-30T14:14:03.731740+0000","last_became_peered":"2023-10-30T14:14:03.731740+0000","last_unstale":"2023-10-30T14:14:06.933073+0000","last_undegraded":"2023-10-30T14:14:06.933073+0000","last_fullsized":"2023-10-30T14:14:06.933073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:34:27.462477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586305+0000","last_change":"2023-10-30T14:14:04.621869+0000","last_active":"2023-10-30T14:14:05.586305+0000","last_peered":"2023-10-30T14:14:05.586305+0000","last_clean":"2023-10-30T14:14:05.586305+0000","last_became_active":"2023-10-30T14:14:04.621526+0000","last_became_peered":"2023-10-30T14:14:04.621526+0000","last_unstale":"2023-10-30T14:14:05.586305+0000","last_undegraded":"2023-10-30T14:14:05.586305+0000","last_fullsized":"2023-10-30T14:14:05.586305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:08:21.260692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.964457+0000","last_change":"2023-10-30T14:14:03.730789+0000","last_active":"2023-10-30T14:14:06.964457+0000","last_peered":"2023-10-30T14:14:06.964457+0000","last_clean":"2023-10-30T14:14:06.964457+0000","last_became_active":"2023-10-30T14:14:03.729386+0000","last_became_peered":"2023-10-30T14:14:03.729386+0000","last_unstale":"2023-10-30T14:14:06.964457+0000","last_undegraded":"2023-10-30T14:14:06.964457+0000","last_fullsized":"2023-10-30T14:14:06.964457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805855+0000","last_change":"2023-10-30T14:14:03.724379+0000","last_active":"2023-10-30T14:14:05.805855+0000","last_peered":"2023-10-30T14:14:05.805855+0000","last_clean":"2023-10-30T14:14:05.805855+0000","last_became_active":"2023-10-30T14:14:03.724078+0000","last_became_peered":"2023-10-30T14:14:03.724078+0000","last_unstale":"2023-10-30T14:14:05.805855+0000","last_undegraded":"2023-10-30T14:14:05.805855+0000","last_fullsized":"2023-10-30T14:14:05.805855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:10:17.829899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805886+0000","last_change":"2023-10-30T14:14:04.597423+0000","last_active":"2023-10-30T14:14:05.805886+0000","last_peered":"2023-10-30T14:14:05.805886+0000","last_clean":"2023-10-30T14:14:05.805886+0000","last_became_active":"2023-10-30T14:14:04.597099+0000","last_became_peered":"2023-10-30T14:14:04.597099+0000","last_unstale":"2023-10-30T14:14:05.805886+0000","last_undegraded":"2023-10-30T14:14:05.805886+0000","last_fullsized":"2023-10-30T14:14:05.805886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:54:52.346295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585964+0000","last_change":"2023-10-30T14:14:03.721030+0000","last_active":"2023-10-30T14:14:05.585964+0000","last_peered":"2023-10-30T14:14:05.585964+0000","last_clean":"2023-10-30T14:14:05.585964+0000","last_became_active":"2023-10-30T14:14:03.719816+0000","last_became_peered":"2023-10-30T14:14:03.719816+0000","last_unstale":"2023-10-30T14:14:05.585964+0000","last_undegraded":"2023-10-30T14:14:05.585964+0000","last_fullsized":"2023-10-30T14:14:05.585964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:43:50.150156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584729+0000","last_change":"2023-10-30T14:14:04.599863+0000","last_active":"2023-10-30T14:14:05.584729+0000","last_peered":"2023-10-30T14:14:05.584729+0000","last_clean":"2023-10-30T14:14:05.584729+0000","last_became_active":"2023-10-30T14:14:04.599323+0000","last_became_peered":"2023-10-30T14:14:04.599323+0000","last_unstale":"2023-10-30T14:14:05.584729+0000","last_undegraded":"2023-10-30T14:14:05.584729+0000","last_fullsized":"2023-10-30T14:14:05.584729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:42:36.939959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808018+0000","last_change":"2023-10-30T14:14:03.738694+0000","last_active":"2023-10-30T14:14:05.808018+0000","last_peered":"2023-10-30T14:14:05.808018+0000","last_clean":"2023-10-30T14:14:05.808018+0000","last_became_active":"2023-10-30T14:14:03.738132+0000","last_became_peered":"2023-10-30T14:14:03.738132+0000","last_unstale":"2023-10-30T14:14:05.808018+0000","last_undegraded":"2023-10-30T14:14:05.808018+0000","last_fullsized":"2023-10-30T14:14:05.808018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:46:59.692900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586432+0000","last_change":"2023-10-30T14:14:04.649502+0000","last_active":"2023-10-30T14:14:05.586432+0000","last_peered":"2023-10-30T14:14:05.586432+0000","last_clean":"2023-10-30T14:14:05.586432+0000","last_became_active":"2023-10-30T14:14:04.648816+0000","last_became_peered":"2023-10-30T14:14:04.648816+0000","last_unstale":"2023-10-30T14:14:05.586432+0000","last_undegraded":"2023-10-30T14:14:05.586432+0000","last_fullsized":"2023-10-30T14:14:05.586432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:23:07.149253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586445+0000","last_change":"2023-10-30T14:14:03.726112+0000","last_active":"2023-10-30T14:14:05.586445+0000","last_peered":"2023-10-30T14:14:05.586445+0000","last_clean":"2023-10-30T14:14:05.586445+0000","last_became_active":"2023-10-30T14:14:03.725272+0000","last_became_peered":"2023-10-30T14:14:03.725272+0000","last_unstale":"2023-10-30T14:14:05.586445+0000","last_undegraded":"2023-10-30T14:14:05.586445+0000","last_fullsized":"2023-10-30T14:14:05.586445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:37:20.764294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586677+0000","last_change":"2023-10-30T14:14:04.610043+0000","last_active":"2023-10-30T14:14:05.586677+0000","last_peered":"2023-10-30T14:14:05.586677+0000","last_clean":"2023-10-30T14:14:05.586677+0000","last_became_active":"2023-10-30T14:14:04.609697+0000","last_became_peered":"2023-10-30T14:14:04.609697+0000","last_unstale":"2023-10-30T14:14:05.586677+0000","last_undegraded":"2023-10-30T14:14:05.586677+0000","last_fullsized":"2023-10-30T14:14:05.586677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:28.543547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.718722+0000","last_change":"2023-10-30T14:14:03.727871+0000","last_active":"2023-10-30T14:14:09.718722+0000","last_peered":"2023-10-30T14:14:09.718722+0000","last_clean":"2023-10-30T14:14:09.718722+0000","last_became_active":"2023-10-30T14:14:03.727422+0000","last_became_peered":"2023-10-30T14:14:03.727422+0000","last_unstale":"2023-10-30T14:14:09.718722+0000","last_undegraded":"2023-10-30T14:14:09.718722+0000","last_fullsized":"2023-10-30T14:14:09.718722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:54.782533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809764+0000","last_change":"2023-10-30T14:14:04.645734+0000","last_active":"2023-10-30T14:14:05.809764+0000","last_peered":"2023-10-30T14:14:05.809764+0000","last_clean":"2023-10-30T14:14:05.809764+0000","last_became_active":"2023-10-30T14:14:04.645333+0000","last_became_peered":"2023-10-30T14:14:04.645333+0000","last_unstale":"2023-10-30T14:14:05.809764+0000","last_undegraded":"2023-10-30T14:14:05.809764+0000","last_fullsized":"2023-10-30T14:14:05.809764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:36:11.322798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.722581+0000","last_change":"2023-10-30T14:14:03.725554+0000","last_active":"2023-10-30T14:14:09.722581+0000","last_peered":"2023-10-30T14:14:09.722581+0000","last_clean":"2023-10-30T14:14:09.722581+0000","last_became_active":"2023-10-30T14:14:03.724806+0000","last_became_peered":"2023-10-30T14:14:03.724806+0000","last_unstale":"2023-10-30T14:14:09.722581+0000","last_undegraded":"2023-10-30T14:14:09.722581+0000","last_fullsized":"2023-10-30T14:14:09.722581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:37.823561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756505+0000","last_change":"2023-10-30T14:14:04.647122+0000","last_active":"2023-10-30T14:14:05.756505+0000","last_peered":"2023-10-30T14:14:05.756505+0000","last_clean":"2023-10-30T14:14:05.756505+0000","last_became_active":"2023-10-30T14:14:04.646543+0000","last_became_peered":"2023-10-30T14:14:04.646543+0000","last_unstale":"2023-10-30T14:14:05.756505+0000","last_undegraded":"2023-10-30T14:14:05.756505+0000","last_fullsized":"2023-10-30T14:14:05.756505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:16.158442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.722910+0000","last_change":"2023-10-30T14:14:03.717903+0000","last_active":"2023-10-30T14:14:09.722910+0000","last_peered":"2023-10-30T14:14:09.722910+0000","last_clean":"2023-10-30T14:14:09.722910+0000","last_became_active":"2023-10-30T14:14:03.716941+0000","last_became_peered":"2023-10-30T14:14:03.716941+0000","last_unstale":"2023-10-30T14:14:09.722910+0000","last_undegraded":"2023-10-30T14:14:09.722910+0000","last_fullsized":"2023-10-30T14:14:09.722910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:58.178166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586173+0000","last_change":"2023-10-30T14:14:04.641757+0000","last_active":"2023-10-30T14:14:05.586173+0000","last_peered":"2023-10-30T14:14:05.586173+0000","last_clean":"2023-10-30T14:14:05.586173+0000","last_became_active":"2023-10-30T14:14:04.641494+0000","last_became_peered":"2023-10-30T14:14:04.641494+0000","last_unstale":"2023-10-30T14:14:05.586173+0000","last_undegraded":"2023-10-30T14:14:05.586173+0000","last_fullsized":"2023-10-30T14:14:05.586173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:00:18.795083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587083+0000","last_change":"2023-10-30T14:14:03.729626+0000","last_active":"2023-10-30T14:14:05.587083+0000","last_peered":"2023-10-30T14:14:05.587083+0000","last_clean":"2023-10-30T14:14:05.587083+0000","last_became_active":"2023-10-30T14:14:03.728053+0000","last_became_peered":"2023-10-30T14:14:03.728053+0000","last_unstale":"2023-10-30T14:14:05.587083+0000","last_undegraded":"2023-10-30T14:14:05.587083+0000","last_fullsized":"2023-10-30T14:14:05.587083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:44:53.362066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756773+0000","last_change":"2023-10-30T14:14:04.650513+0000","last_active":"2023-10-30T14:14:05.756773+0000","last_peered":"2023-10-30T14:14:05.756773+0000","last_clean":"2023-10-30T14:14:05.756773+0000","last_became_active":"2023-10-30T14:14:04.649896+0000","last_became_peered":"2023-10-30T14:14:04.649896+0000","last_unstale":"2023-10-30T14:14:05.756773+0000","last_undegraded":"2023-10-30T14:14:05.756773+0000","last_fullsized":"2023-10-30T14:14:05.756773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:23.100490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807384+0000","last_change":"2023-10-30T14:14:03.736824+0000","last_active":"2023-10-30T14:14:05.807384+0000","last_peered":"2023-10-30T14:14:05.807384+0000","last_clean":"2023-10-30T14:14:05.807384+0000","last_became_active":"2023-10-30T14:14:03.736155+0000","last_became_peered":"2023-10-30T14:14:03.736155+0000","last_unstale":"2023-10-30T14:14:05.807384+0000","last_undegraded":"2023-10-30T14:14:05.807384+0000","last_fullsized":"2023-10-30T14:14:05.807384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:35.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807372+0000","last_change":"2023-10-30T14:14:04.615159+0000","last_active":"2023-10-30T14:14:05.807372+0000","last_peered":"2023-10-30T14:14:05.807372+0000","last_clean":"2023-10-30T14:14:05.807372+0000","last_became_active":"2023-10-30T14:14:04.614328+0000","last_became_peered":"2023-10-30T14:14:04.614328+0000","last_unstale":"2023-10-30T14:14:05.807372+0000","last_undegraded":"2023-10-30T14:14:05.807372+0000","last_fullsized":"2023-10-30T14:14:05.807372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:16:52.999786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.722021+0000","last_change":"2023-10-30T14:14:03.700690+0000","last_active":"2023-10-30T14:14:09.722021+0000","last_peered":"2023-10-30T14:14:09.722021+0000","last_clean":"2023-10-30T14:14:09.722021+0000","last_became_active":"2023-10-30T14:14:03.700352+0000","last_became_peered":"2023-10-30T14:14:03.700352+0000","last_unstale":"2023-10-30T14:14:09.722021+0000","last_undegraded":"2023-10-30T14:14:09.722021+0000","last_fullsized":"2023-10-30T14:14:09.722021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:03:01.103359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755948+0000","last_change":"2023-10-30T14:14:04.647253+0000","last_active":"2023-10-30T14:14:05.755948+0000","last_peered":"2023-10-30T14:14:05.755948+0000","last_clean":"2023-10-30T14:14:05.755948+0000","last_became_active":"2023-10-30T14:14:04.646639+0000","last_became_peered":"2023-10-30T14:14:04.646639+0000","last_unstale":"2023-10-30T14:14:05.755948+0000","last_undegraded":"2023-10-30T14:14:05.755948+0000","last_fullsized":"2023-10-30T14:14:05.755948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:22:50.893734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.936522+0000","last_change":"2023-10-30T14:14:03.710717+0000","last_active":"2023-10-30T14:14:06.936522+0000","last_peered":"2023-10-30T14:14:06.936522+0000","last_clean":"2023-10-30T14:14:06.936522+0000","last_became_active":"2023-10-30T14:14:03.707950+0000","last_became_peered":"2023-10-30T14:14:03.707950+0000","last_unstale":"2023-10-30T14:14:06.936522+0000","last_undegraded":"2023-10-30T14:14:06.936522+0000","last_fullsized":"2023-10-30T14:14:06.936522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:29:41.232902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757066+0000","last_change":"2023-10-30T14:14:04.647688+0000","last_active":"2023-10-30T14:14:05.757066+0000","last_peered":"2023-10-30T14:14:05.757066+0000","last_clean":"2023-10-30T14:14:05.757066+0000","last_became_active":"2023-10-30T14:14:04.644761+0000","last_became_peered":"2023-10-30T14:14:04.644761+0000","last_unstale":"2023-10-30T14:14:05.757066+0000","last_undegraded":"2023-10-30T14:14:05.757066+0000","last_fullsized":"2023-10-30T14:14:05.757066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:00:38.482522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.716374+0000","last_change":"2023-10-30T14:14:03.705567+0000","last_active":"2023-10-30T14:14:09.716374+0000","last_peered":"2023-10-30T14:14:09.716374+0000","last_clean":"2023-10-30T14:14:09.716374+0000","last_became_active":"2023-10-30T14:14:03.705186+0000","last_became_peered":"2023-10-30T14:14:03.705186+0000","last_unstale":"2023-10-30T14:14:09.716374+0000","last_undegraded":"2023-10-30T14:14:09.716374+0000","last_fullsized":"2023-10-30T14:14:09.716374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:14.609243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757235+0000","last_change":"2023-10-30T14:14:04.647335+0000","last_active":"2023-10-30T14:14:05.757235+0000","last_peered":"2023-10-30T14:14:05.757235+0000","last_clean":"2023-10-30T14:14:05.757235+0000","last_became_active":"2023-10-30T14:14:04.646785+0000","last_became_peered":"2023-10-30T14:14:04.646785+0000","last_unstale":"2023-10-30T14:14:05.757235+0000","last_undegraded":"2023-10-30T14:14:05.757235+0000","last_fullsized":"2023-10-30T14:14:05.757235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:58:41.600761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.935812+0000","last_change":"2023-10-30T14:14:03.693349+0000","last_active":"2023-10-30T14:14:06.935812+0000","last_peered":"2023-10-30T14:14:06.935812+0000","last_clean":"2023-10-30T14:14:06.935812+0000","last_became_active":"2023-10-30T14:14:03.692966+0000","last_became_peered":"2023-10-30T14:14:03.692966+0000","last_unstale":"2023-10-30T14:14:06.935812+0000","last_undegraded":"2023-10-30T14:14:06.935812+0000","last_fullsized":"2023-10-30T14:14:06.935812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:30:28.800214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587213+0000","last_change":"2023-10-30T14:14:04.617023+0000","last_active":"2023-10-30T14:14:05.587213+0000","last_peered":"2023-10-30T14:14:05.587213+0000","last_clean":"2023-10-30T14:14:05.587213+0000","last_became_active":"2023-10-30T14:14:04.615925+0000","last_became_peered":"2023-10-30T14:14:04.615925+0000","last_unstale":"2023-10-30T14:14:05.587213+0000","last_undegraded":"2023-10-30T14:14:05.587213+0000","last_fullsized":"2023-10-30T14:14:05.587213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:32:02.289324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805586+0000","last_change":"2023-10-30T14:14:03.727659+0000","last_active":"2023-10-30T14:14:05.805586+0000","last_peered":"2023-10-30T14:14:05.805586+0000","last_clean":"2023-10-30T14:14:05.805586+0000","last_became_active":"2023-10-30T14:14:03.727409+0000","last_became_peered":"2023-10-30T14:14:03.727409+0000","last_unstale":"2023-10-30T14:14:05.805586+0000","last_undegraded":"2023-10-30T14:14:05.805586+0000","last_fullsized":"2023-10-30T14:14:05.805586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:24.210635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756468+0000","last_change":"2023-10-30T14:14:04.650083+0000","last_active":"2023-10-30T14:14:05.756468+0000","last_peered":"2023-10-30T14:14:05.756468+0000","last_clean":"2023-10-30T14:14:05.756468+0000","last_became_active":"2023-10-30T14:14:04.647562+0000","last_became_peered":"2023-10-30T14:14:04.647562+0000","last_unstale":"2023-10-30T14:14:05.756468+0000","last_undegraded":"2023-10-30T14:14:05.756468+0000","last_fullsized":"2023-10-30T14:14:05.756468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:11:41.743850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T14:14:04.632968+0000","last_change":"2023-10-30T14:13:54.251572+0000","last_active":"2023-10-30T14:14:04.632968+0000","last_peered":"2023-10-30T14:14:04.632968+0000","last_clean":"2023-10-30T14:14:04.632968+0000","last_became_active":"2023-10-30T14:13:54.250907+0000","last_became_peered":"2023-10-30T14:13:54.250907+0000","last_unstale":"2023-10-30T14:14:04.632968+0000","last_undegraded":"2023-10-30T14:14:04.632968+0000","last_fullsized":"2023-10-30T14:14:04.632968+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_clean_scrub_stamp":"2023-10-30T14:13:53.220267+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:44.590852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585892+0000","last_change":"2023-10-30T14:14:03.729034+0000","last_active":"2023-10-30T14:14:05.585892+0000","last_peered":"2023-10-30T14:14:05.585892+0000","last_clean":"2023-10-30T14:14:05.585892+0000","last_became_active":"2023-10-30T14:14:03.728293+0000","last_became_peered":"2023-10-30T14:14:03.728293+0000","last_unstale":"2023-10-30T14:14:05.585892+0000","last_undegraded":"2023-10-30T14:14:05.585892+0000","last_fullsized":"2023-10-30T14:14:05.585892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806225+0000","last_change":"2023-10-30T14:14:04.630325+0000","last_active":"2023-10-30T14:14:05.806225+0000","last_peered":"2023-10-30T14:14:05.806225+0000","last_clean":"2023-10-30T14:14:05.806225+0000","last_became_active":"2023-10-30T14:14:04.630121+0000","last_became_peered":"2023-10-30T14:14:04.630121+0000","last_unstale":"2023-10-30T14:14:05.806225+0000","last_undegraded":"2023-10-30T14:14:05.806225+0000","last_fullsized":"2023-10-30T14:14:05.806225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:02:28.792879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755708+0000","last_change":"2023-10-30T14:14:03.732909+0000","last_active":"2023-10-30T14:14:05.755708+0000","last_peered":"2023-10-30T14:14:05.755708+0000","last_clean":"2023-10-30T14:14:05.755708+0000","last_became_active":"2023-10-30T14:14:03.732199+0000","last_became_peered":"2023-10-30T14:14:03.732199+0000","last_unstale":"2023-10-30T14:14:05.755708+0000","last_undegraded":"2023-10-30T14:14:05.755708+0000","last_fullsized":"2023-10-30T14:14:05.755708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:38.117750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586172+0000","last_change":"2023-10-30T14:14:04.620314+0000","last_active":"2023-10-30T14:14:05.586172+0000","last_peered":"2023-10-30T14:14:05.586172+0000","last_clean":"2023-10-30T14:14:05.586172+0000","last_became_active":"2023-10-30T14:14:04.619990+0000","last_became_peered":"2023-10-30T14:14:04.619990+0000","last_unstale":"2023-10-30T14:14:05.586172+0000","last_undegraded":"2023-10-30T14:14:05.586172+0000","last_fullsized":"2023-10-30T14:14:05.586172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:38:25.657586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805642+0000","last_change":"2023-10-30T14:14:03.702862+0000","last_active":"2023-10-30T14:14:05.805642+0000","last_peered":"2023-10-30T14:14:05.805642+0000","last_clean":"2023-10-30T14:14:05.805642+0000","last_became_active":"2023-10-30T14:14:03.702513+0000","last_became_peered":"2023-10-30T14:14:03.702513+0000","last_unstale":"2023-10-30T14:14:05.805642+0000","last_undegraded":"2023-10-30T14:14:05.805642+0000","last_fullsized":"2023-10-30T14:14:05.805642+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:25.499390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584623+0000","last_change":"2023-10-30T14:14:04.589353+0000","last_active":"2023-10-30T14:14:05.584623+0000","last_peered":"2023-10-30T14:14:05.584623+0000","last_clean":"2023-10-30T14:14:05.584623+0000","last_became_active":"2023-10-30T14:14:04.589087+0000","last_became_peered":"2023-10-30T14:14:04.589087+0000","last_unstale":"2023-10-30T14:14:05.584623+0000","last_undegraded":"2023-10-30T14:14:05.584623+0000","last_fullsized":"2023-10-30T14:14:05.584623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:20:12.808641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587305+0000","last_change":"2023-10-30T14:14:03.730048+0000","last_active":"2023-10-30T14:14:05.587305+0000","last_peered":"2023-10-30T14:14:05.587305+0000","last_clean":"2023-10-30T14:14:05.587305+0000","last_became_active":"2023-10-30T14:14:03.728925+0000","last_became_peered":"2023-10-30T14:14:03.728925+0000","last_unstale":"2023-10-30T14:14:05.587305+0000","last_undegraded":"2023-10-30T14:14:05.587305+0000","last_fullsized":"2023-10-30T14:14:05.587305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:32:05.056039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809437+0000","last_change":"2023-10-30T14:14:04.608289+0000","last_active":"2023-10-30T14:14:05.809437+0000","last_peered":"2023-10-30T14:14:05.809437+0000","last_clean":"2023-10-30T14:14:05.809437+0000","last_became_active":"2023-10-30T14:14:04.607377+0000","last_became_peered":"2023-10-30T14:14:04.607377+0000","last_unstale":"2023-10-30T14:14:05.809437+0000","last_undegraded":"2023-10-30T14:14:05.809437+0000","last_fullsized":"2023-10-30T14:14:05.809437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:43:40.869579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584795+0000","last_change":"2023-10-30T14:14:03.693753+0000","last_active":"2023-10-30T14:14:05.584795+0000","last_peered":"2023-10-30T14:14:05.584795+0000","last_clean":"2023-10-30T14:14:05.584795+0000","last_became_active":"2023-10-30T14:14:03.692621+0000","last_became_peered":"2023-10-30T14:14:03.692621+0000","last_unstale":"2023-10-30T14:14:05.584795+0000","last_undegraded":"2023-10-30T14:14:05.584795+0000","last_fullsized":"2023-10-30T14:14:05.584795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:32:28.577584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587225+0000","last_change":"2023-10-30T14:14:04.614649+0000","last_active":"2023-10-30T14:14:05.587225+0000","last_peered":"2023-10-30T14:14:05.587225+0000","last_clean":"2023-10-30T14:14:05.587225+0000","last_became_active":"2023-10-30T14:14:04.613780+0000","last_became_peered":"2023-10-30T14:14:04.613780+0000","last_unstale":"2023-10-30T14:14:05.587225+0000","last_undegraded":"2023-10-30T14:14:05.587225+0000","last_fullsized":"2023-10-30T14:14:05.587225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:47:40.354022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757037+0000","last_change":"2023-10-30T14:14:03.740213+0000","last_active":"2023-10-30T14:14:05.757037+0000","last_peered":"2023-10-30T14:14:05.757037+0000","last_clean":"2023-10-30T14:14:05.757037+0000","last_became_active":"2023-10-30T14:14:03.739650+0000","last_became_peered":"2023-10-30T14:14:03.739650+0000","last_unstale":"2023-10-30T14:14:05.757037+0000","last_undegraded":"2023-10-30T14:14:05.757037+0000","last_fullsized":"2023-10-30T14:14:05.757037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:54.120757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584691+0000","last_change":"2023-10-30T14:14:04.590569+0000","last_active":"2023-10-30T14:14:05.584691+0000","last_peered":"2023-10-30T14:14:05.584691+0000","last_clean":"2023-10-30T14:14:05.584691+0000","last_became_active":"2023-10-30T14:14:04.590138+0000","last_became_peered":"2023-10-30T14:14:04.590138+0000","last_unstale":"2023-10-30T14:14:05.584691+0000","last_undegraded":"2023-10-30T14:14:05.584691+0000","last_fullsized":"2023-10-30T14:14:05.584691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:41:30.333668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585050+0000","last_change":"2023-10-30T14:14:03.721450+0000","last_active":"2023-10-30T14:14:05.585050+0000","last_peered":"2023-10-30T14:14:05.585050+0000","last_clean":"2023-10-30T14:14:05.585050+0000","last_became_active":"2023-10-30T14:14:03.721234+0000","last_became_peered":"2023-10-30T14:14:03.721234+0000","last_unstale":"2023-10-30T14:14:05.585050+0000","last_undegraded":"2023-10-30T14:14:05.585050+0000","last_fullsized":"2023-10-30T14:14:05.585050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:26.959676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806005+0000","last_change":"2023-10-30T14:14:04.634004+0000","last_active":"2023-10-30T14:14:05.806005+0000","last_peered":"2023-10-30T14:14:05.806005+0000","last_clean":"2023-10-30T14:14:05.806005+0000","last_became_active":"2023-10-30T14:14:04.633643+0000","last_became_peered":"2023-10-30T14:14:04.633643+0000","last_unstale":"2023-10-30T14:14:05.806005+0000","last_undegraded":"2023-10-30T14:14:05.806005+0000","last_fullsized":"2023-10-30T14:14:05.806005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:05:14.572477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.721271+0000","last_change":"2023-10-30T14:14:03.738557+0000","last_active":"2023-10-30T14:14:09.721271+0000","last_peered":"2023-10-30T14:14:09.721271+0000","last_clean":"2023-10-30T14:14:09.721271+0000","last_became_active":"2023-10-30T14:14:03.737801+0000","last_became_peered":"2023-10-30T14:14:03.737801+0000","last_unstale":"2023-10-30T14:14:09.721271+0000","last_undegraded":"2023-10-30T14:14:09.721271+0000","last_fullsized":"2023-10-30T14:14:09.721271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:26.938343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586581+0000","last_change":"2023-10-30T14:14:04.589273+0000","last_active":"2023-10-30T14:14:05.586581+0000","last_peered":"2023-10-30T14:14:05.586581+0000","last_clean":"2023-10-30T14:14:05.586581+0000","last_became_active":"2023-10-30T14:14:04.588826+0000","last_became_peered":"2023-10-30T14:14:04.588826+0000","last_unstale":"2023-10-30T14:14:05.586581+0000","last_undegraded":"2023-10-30T14:14:05.586581+0000","last_fullsized":"2023-10-30T14:14:05.586581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:33:52.452394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755689+0000","last_change":"2023-10-30T14:14:03.733836+0000","last_active":"2023-10-30T14:14:05.755689+0000","last_peered":"2023-10-30T14:14:05.755689+0000","last_clean":"2023-10-30T14:14:05.755689+0000","last_became_active":"2023-10-30T14:14:03.732912+0000","last_became_peered":"2023-10-30T14:14:03.732912+0000","last_unstale":"2023-10-30T14:14:05.755689+0000","last_undegraded":"2023-10-30T14:14:05.755689+0000","last_fullsized":"2023-10-30T14:14:05.755689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:50.332738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809314+0000","last_change":"2023-10-30T14:14:04.610696+0000","last_active":"2023-10-30T14:14:05.809314+0000","last_peered":"2023-10-30T14:14:05.809314+0000","last_clean":"2023-10-30T14:14:05.809314+0000","last_became_active":"2023-10-30T14:14:04.609835+0000","last_became_peered":"2023-10-30T14:14:04.609835+0000","last_unstale":"2023-10-30T14:14:05.809314+0000","last_undegraded":"2023-10-30T14:14:05.809314+0000","last_fullsized":"2023-10-30T14:14:05.809314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:04:31.270262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584669+0000","last_change":"2023-10-30T14:14:03.732539+0000","last_active":"2023-10-30T14:14:05.584669+0000","last_peered":"2023-10-30T14:14:05.584669+0000","last_clean":"2023-10-30T14:14:05.584669+0000","last_became_active":"2023-10-30T14:14:03.732265+0000","last_became_peered":"2023-10-30T14:14:03.732265+0000","last_unstale":"2023-10-30T14:14:05.584669+0000","last_undegraded":"2023-10-30T14:14:05.584669+0000","last_fullsized":"2023-10-30T14:14:05.584669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:02:00.929612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806211+0000","last_change":"2023-10-30T14:14:04.595357+0000","last_active":"2023-10-30T14:14:05.806211+0000","last_peered":"2023-10-30T14:14:05.806211+0000","last_clean":"2023-10-30T14:14:05.806211+0000","last_became_active":"2023-10-30T14:14:04.594998+0000","last_became_peered":"2023-10-30T14:14:04.594998+0000","last_unstale":"2023-10-30T14:14:05.806211+0000","last_undegraded":"2023-10-30T14:14:05.806211+0000","last_fullsized":"2023-10-30T14:14:05.806211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:44.365673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584615+0000","last_change":"2023-10-30T14:14:03.703744+0000","last_active":"2023-10-30T14:14:05.584615+0000","last_peered":"2023-10-30T14:14:05.584615+0000","last_clean":"2023-10-30T14:14:05.584615+0000","last_became_active":"2023-10-30T14:14:03.701759+0000","last_became_peered":"2023-10-30T14:14:03.701759+0000","last_unstale":"2023-10-30T14:14:05.584615+0000","last_undegraded":"2023-10-30T14:14:05.584615+0000","last_fullsized":"2023-10-30T14:14:05.584615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:22:27.948007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806186+0000","last_change":"2023-10-30T14:14:04.600759+0000","last_active":"2023-10-30T14:14:05.806186+0000","last_peered":"2023-10-30T14:14:05.806186+0000","last_clean":"2023-10-30T14:14:05.806186+0000","last_became_active":"2023-10-30T14:14:04.600100+0000","last_became_peered":"2023-10-30T14:14:04.600100+0000","last_unstale":"2023-10-30T14:14:05.806186+0000","last_undegraded":"2023-10-30T14:14:05.806186+0000","last_fullsized":"2023-10-30T14:14:05.806186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:02.831207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.723470+0000","last_change":"2023-10-30T14:14:03.741268+0000","last_active":"2023-10-30T14:14:09.723470+0000","last_peered":"2023-10-30T14:14:09.723470+0000","last_clean":"2023-10-30T14:14:09.723470+0000","last_became_active":"2023-10-30T14:14:03.740648+0000","last_became_peered":"2023-10-30T14:14:03.740648+0000","last_unstale":"2023-10-30T14:14:09.723470+0000","last_undegraded":"2023-10-30T14:14:09.723470+0000","last_fullsized":"2023-10-30T14:14:09.723470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:30.799148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807541+0000","last_change":"2023-10-30T14:14:04.623644+0000","last_active":"2023-10-30T14:14:05.807541+0000","last_peered":"2023-10-30T14:14:05.807541+0000","last_clean":"2023-10-30T14:14:05.807541+0000","last_became_active":"2023-10-30T14:14:04.622921+0000","last_became_peered":"2023-10-30T14:14:04.622921+0000","last_unstale":"2023-10-30T14:14:05.807541+0000","last_undegraded":"2023-10-30T14:14:05.807541+0000","last_fullsized":"2023-10-30T14:14:05.807541+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:11:41.810610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587103+0000","last_change":"2023-10-30T14:14:04.624799+0000","last_active":"2023-10-30T14:14:05.587103+0000","last_peered":"2023-10-30T14:14:05.587103+0000","last_clean":"2023-10-30T14:14:05.587103+0000","last_became_active":"2023-10-30T14:14:04.621885+0000","last_became_peered":"2023-10-30T14:14:04.621885+0000","last_unstale":"2023-10-30T14:14:05.587103+0000","last_undegraded":"2023-10-30T14:14:05.587103+0000","last_fullsized":"2023-10-30T14:14:05.587103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:12.055247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.721291+0000","last_change":"2023-10-30T14:14:03.740897+0000","last_active":"2023-10-30T14:14:09.721291+0000","last_peered":"2023-10-30T14:14:09.721291+0000","last_clean":"2023-10-30T14:14:09.721291+0000","last_became_active":"2023-10-30T14:14:03.740247+0000","last_became_peered":"2023-10-30T14:14:03.740247+0000","last_unstale":"2023-10-30T14:14:09.721291+0000","last_undegraded":"2023-10-30T14:14:09.721291+0000","last_fullsized":"2023-10-30T14:14:09.721291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:29.601978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586861+0000","last_change":"2023-10-30T14:14:04.625498+0000","last_active":"2023-10-30T14:14:05.586861+0000","last_peered":"2023-10-30T14:14:05.586861+0000","last_clean":"2023-10-30T14:14:05.586861+0000","last_became_active":"2023-10-30T14:14:04.623638+0000","last_became_peered":"2023-10-30T14:14:04.623638+0000","last_unstale":"2023-10-30T14:14:05.586861+0000","last_undegraded":"2023-10-30T14:14:05.586861+0000","last_fullsized":"2023-10-30T14:14:05.586861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:19.717965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.719605+0000","last_change":"2023-10-30T14:14:03.733233+0000","last_active":"2023-10-30T14:14:09.719605+0000","last_peered":"2023-10-30T14:14:09.719605+0000","last_clean":"2023-10-30T14:14:09.719605+0000","last_became_active":"2023-10-30T14:14:03.732538+0000","last_became_peered":"2023-10-30T14:14:03.732538+0000","last_unstale":"2023-10-30T14:14:09.719605+0000","last_undegraded":"2023-10-30T14:14:09.719605+0000","last_fullsized":"2023-10-30T14:14:09.719605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:25.696173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586619+0000","last_change":"2023-10-30T14:14:04.612104+0000","last_active":"2023-10-30T14:14:05.586619+0000","last_peered":"2023-10-30T14:14:05.586619+0000","last_clean":"2023-10-30T14:14:05.586619+0000","last_became_active":"2023-10-30T14:14:04.611773+0000","last_became_peered":"2023-10-30T14:14:04.611773+0000","last_unstale":"2023-10-30T14:14:05.586619+0000","last_undegraded":"2023-10-30T14:14:05.586619+0000","last_fullsized":"2023-10-30T14:14:05.586619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:44:21.952103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.933126+0000","last_change":"2023-10-30T14:14:03.703988+0000","last_active":"2023-10-30T14:14:06.933126+0000","last_peered":"2023-10-30T14:14:06.933126+0000","last_clean":"2023-10-30T14:14:06.933126+0000","last_became_active":"2023-10-30T14:14:03.703691+0000","last_became_peered":"2023-10-30T14:14:03.703691+0000","last_unstale":"2023-10-30T14:14:06.933126+0000","last_undegraded":"2023-10-30T14:14:06.933126+0000","last_fullsized":"2023-10-30T14:14:06.933126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:34.538541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806119+0000","last_change":"2023-10-30T14:14:04.613738+0000","last_active":"2023-10-30T14:14:05.806119+0000","last_peered":"2023-10-30T14:14:05.806119+0000","last_clean":"2023-10-30T14:14:05.806119+0000","last_became_active":"2023-10-30T14:14:04.613420+0000","last_became_peered":"2023-10-30T14:14:04.613420+0000","last_unstale":"2023-10-30T14:14:05.806119+0000","last_undegraded":"2023-10-30T14:14:05.806119+0000","last_fullsized":"2023-10-30T14:14:05.806119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:01:52.695195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.720791+0000","last_change":"2023-10-30T14:14:03.737319+0000","last_active":"2023-10-30T14:14:09.720791+0000","last_peered":"2023-10-30T14:14:09.720791+0000","last_clean":"2023-10-30T14:14:09.720791+0000","last_became_active":"2023-10-30T14:14:03.736964+0000","last_became_peered":"2023-10-30T14:14:03.736964+0000","last_unstale":"2023-10-30T14:14:09.720791+0000","last_undegraded":"2023-10-30T14:14:09.720791+0000","last_fullsized":"2023-10-30T14:14:09.720791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:41.469809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":815104,"data_stored":666519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.47}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":75659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.056}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":77181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":75569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":823296,"data_stored":666575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:14:15.711 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:14:15.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.888+0000 7f6e7390f640 1 Processor -- start 2023-10-30T14:14:15.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.888+0000 7f6e7390f640 1 -- start start 2023-10-30T14:14:15.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.892+0000 7f6e7390f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6e6c153c80 0x7f6e6c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:15.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.892+0000 7f6e7390f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e6c0c42a0 0x7f6e6c0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:15.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.892+0000 7f6e7390f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6c159f20 con 0x7f6e6c0c4a10 2023-10-30T14:14:15.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.892+0000 7f6e7390f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e6c15dec0 con 0x7f6e6c153c80 2023-10-30T14:14:15.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.892+0000 7f6e7390f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c1599a0 con 0x7f6e6c0c42a0 2023-10-30T14:14:15.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.892+0000 7f6e716ab640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6e6c153c80 0x7f6e6c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:15.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.892+0000 7f6e70eaa640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f6e6c0c4a10 0x7f6e6c15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:37712/0 (socket says 172.21.15.57:37712) 2023-10-30T14:14:15.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.892+0000 7f6e70eaa640 1 -- 172.21.15.57:0/2775769669 learned_addr learned my addr 172.21.15.57:0/2775769669 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:15.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.892+0000 7f6e71eac640 1 --2- 172.21.15.57:0/2775769669 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e6c0c42a0 0x7f6e6c0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:15.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.892+0000 7f6e716ab640 1 -- 172.21.15.57:0/2775769669 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e6c0c42a0 msgr2=0x7f6e6c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:15.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.892+0000 7f6e716ab640 1 --2- 172.21.15.57:0/2775769669 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e6c0c42a0 0x7f6e6c0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.892+0000 7f6e716ab640 1 -- 172.21.15.57:0/2775769669 >> v1:172.21.15.57:6789/0 conn(0x7f6e6c0c4a10 legacy=0x7f6e6c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.892+0000 7f6e716ab640 1 -- 172.21.15.57:0/2775769669 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c15d620 con 0x7f6e6c153c80 2023-10-30T14:14:15.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.892+0000 7f6e63fff640 1 -- 172.21.15.57:0/2775769669 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827975858 0 0) 0x7f6e6c159f20 con 0x7f6e6c0c4a10 2023-10-30T14:14:15.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e716ab640 1 --2- 172.21.15.57:0/2775769669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e6c153c80 0x7f6e6c15cf90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6e680097c0 tx=0x7f6e6801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90a427094f6f8528 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:15.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e63fff640 1 -- 172.21.15.57:0/2775769669 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e680a8970 con 0x7f6e6c153c80 2023-10-30T14:14:15.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e63fff640 1 -- 172.21.15.57:0/2775769669 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e680a8ad0 con 0x7f6e6c153c80 2023-10-30T14:14:15.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e63fff640 1 -- 172.21.15.57:0/2775769669 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e680a8e60 con 0x7f6e6c153c80 2023-10-30T14:14:15.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e7390f640 1 -- 172.21.15.57:0/2775769669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e6c153c80 msgr2=0x7f6e6c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e7390f640 1 --2- 172.21.15.57:0/2775769669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e6c153c80 0x7f6e6c15cf90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6e680097c0 tx=0x7f6e6801ec70 comp rx=0 tx=0).stop 2023-10-30T14:14:15.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e7390f640 1 -- 172.21.15.57:0/2775769669 shutdown_connections 2023-10-30T14:14:15.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e7390f640 1 --2- 172.21.15.57:0/2775769669 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e6c0c42a0 0x7f6e6c0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e7390f640 1 --2- 172.21.15.57:0/2775769669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e6c153c80 0x7f6e6c15cf90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e7390f640 1 -- 172.21.15.57:0/2775769669 >> 172.21.15.57:0/2775769669 conn(0x7f6e6c0715c0 msgr2=0x7f6e6c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:15.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e7390f640 1 -- 172.21.15.57:0/2775769669 shutdown_connections 2023-10-30T14:14:15.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e7390f640 1 -- 172.21.15.57:0/2775769669 wait complete. 2023-10-30T14:14:15.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e7390f640 1 Processor -- start 2023-10-30T14:14:15.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e7390f640 1 -- start start 2023-10-30T14:14:15.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e7390f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e6c0c42a0 0x7f6e6c154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:15.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e7390f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e6c0c4a10 0x7f6e6c154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:15.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e7390f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e6c153c80 0x7f6e6c155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:15.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e7390f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c156620 con 0x7f6e6c0c42a0 2023-10-30T14:14:15.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e7390f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c0ceac0 con 0x7f6e6c153c80 2023-10-30T14:14:15.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.896+0000 7f6e7390f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c154850 con 0x7f6e6c0c4a10 2023-10-30T14:14:15.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.900+0000 7f6e716ab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e6c0c42a0 0x7f6e6c154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:15.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.900+0000 7f6e716ab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e6c0c42a0 0x7f6e6c154310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39970/0 (socket says 172.21.15.57:39970) 2023-10-30T14:14:15.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.900+0000 7f6e716ab640 1 -- 172.21.15.57:0/3902256469 learned_addr learned my addr 172.21.15.57:0/3902256469 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:15.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.900+0000 7f6e71eac640 1 --2- 172.21.15.57:0/3902256469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e6c153c80 0x7f6e6c155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:15.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.900+0000 7f6e70eaa640 1 --2- 172.21.15.57:0/3902256469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e6c0c4a10 0x7f6e6c154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:15.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.900+0000 7f6e716ab640 1 -- 172.21.15.57:0/3902256469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e6c0c4a10 msgr2=0x7f6e6c154ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.900+0000 7f6e716ab640 1 --2- 172.21.15.57:0/3902256469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e6c0c4a10 0x7f6e6c154ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.900+0000 7f6e716ab640 1 -- 172.21.15.57:0/3902256469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e6c153c80 msgr2=0x7f6e6c155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:15.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.900+0000 7f6e716ab640 1 --2- 172.21.15.57:0/3902256469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e6c153c80 0x7f6e6c155220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:15.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.900+0000 7f6e716ab640 1 -- 172.21.15.57:0/3902256469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e68009420 con 0x7f6e6c0c42a0 2023-10-30T14:14:15.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.900+0000 7f6e716ab640 1 --2- 172.21.15.57:0/3902256469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e6c0c42a0 0x7f6e6c154310 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6e680097c0 tx=0x7f6e6801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:15.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.900+0000 7f6e61ffb640 1 -- 172.21.15.57:0/3902256469 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e680b9630 con 0x7f6e6c0c42a0 2023-10-30T14:14:15.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.900+0000 7f6e61ffb640 1 -- 172.21.15.57:0/3902256469 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e680b97b0 con 0x7f6e6c0c42a0 2023-10-30T14:14:15.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.900+0000 7f6e7390f640 1 -- 172.21.15.57:0/3902256469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6c1f29f0 con 0x7f6e6c0c42a0 2023-10-30T14:14:15.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.900+0000 7f6e61ffb640 1 -- 172.21.15.57:0/3902256469 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e680b9b60 con 0x7f6e6c0c42a0 2023-10-30T14:14:15.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.900+0000 7f6e7390f640 1 -- 172.21.15.57:0/3902256469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6c1f2fa0 con 0x7f6e6c0c42a0 2023-10-30T14:14:15.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.900+0000 7f6e7390f640 1 -- 172.21.15.57:0/3902256469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6c04e170 con 0x7f6e6c0c42a0 2023-10-30T14:14:15.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.900+0000 7f6e61ffb640 1 -- 172.21.15.57:0/3902256469 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6e680b7040 con 0x7f6e6c0c42a0 2023-10-30T14:14:15.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.904+0000 7f6e61ffb640 1 --2- 172.21.15.57:0/3902256469 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6e500664a0 0x7f6e50068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:15.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.904+0000 7f6e61ffb640 1 -- 172.21.15.57:0/3902256469 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e68121ed0 con 0x7f6e6c0c42a0 2023-10-30T14:14:15.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.904+0000 7f6e70eaa640 1 --2- 172.21.15.57:0/3902256469 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6e500664a0 0x7f6e50068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:15.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.904+0000 7f6e70eaa640 1 --2- 172.21.15.57:0/3902256469 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6e500664a0 0x7f6e50068960 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c148a00 tx=0x7f6e5c007b60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:15.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:15.904+0000 7f6e61ffb640 1 -- 172.21.15.57:0/3902256469 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6e680ef340 con 0x7f6e6c0c42a0 2023-10-30T14:14:16.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.060+0000 7f6e7390f640 1 -- 172.21.15.57:0/3902256469 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6e6c15dbe0 con 0x7f6e500664a0 2023-10-30T14:14:16.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.084+0000 7f6e61ffb640 1 -- 172.21.15.57:0/3902256469 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264525 (secure 0 0 0) 0x7f6e5c0071f0 con 0x7f6e500664a0 2023-10-30T14:14:16.088 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:16.090 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-30T14:14:16.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.084+0000 7f6e7390f640 1 -- 172.21.15.57:0/3902256469 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6e500664a0 msgr2=0x7f6e50068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:16.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.084+0000 7f6e7390f640 1 --2- 172.21.15.57:0/3902256469 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6e500664a0 0x7f6e50068960 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c148a00 tx=0x7f6e5c007b60 comp rx=0 tx=0).stop 2023-10-30T14:14:16.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.084+0000 7f6e7390f640 1 -- 172.21.15.57:0/3902256469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e6c0c42a0 msgr2=0x7f6e6c154310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:16.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.084+0000 7f6e7390f640 1 --2- 172.21.15.57:0/3902256469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e6c0c42a0 0x7f6e6c154310 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6e680097c0 tx=0x7f6e6801ba10 comp rx=0 tx=0).stop 2023-10-30T14:14:16.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.084+0000 7f6e7390f640 1 -- 172.21.15.57:0/3902256469 shutdown_connections 2023-10-30T14:14:16.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.084+0000 7f6e7390f640 1 --2- 172.21.15.57:0/3902256469 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6e500664a0 0x7f6e50068960 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.084+0000 7f6e7390f640 1 --2- 172.21.15.57:0/3902256469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e6c153c80 0x7f6e6c155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.084+0000 7f6e7390f640 1 --2- 172.21.15.57:0/3902256469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e6c0c4a10 0x7f6e6c154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.084+0000 7f6e7390f640 1 --2- 172.21.15.57:0/3902256469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e6c0c42a0 0x7f6e6c154310 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.084+0000 7f6e7390f640 1 -- 172.21.15.57:0/3902256469 >> 172.21.15.57:0/3902256469 conn(0x7f6e6c0715c0 msgr2=0x7f6e6c158450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:16.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.084+0000 7f6e7390f640 1 -- 172.21.15.57:0/3902256469 shutdown_connections 2023-10-30T14:14:16.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.084+0000 7f6e7390f640 1 -- 172.21.15.57:0/3902256469 wait complete. 2023-10-30T14:14:16.108 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-30T14:14:14.496045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227368,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754747352,"statfs":{"total":773094113280,"available":772861288448,"internally_reserved":0,"allocated":2990080,"data_stored":1790108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":229756473},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.916605"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757043+0000","last_change":"2023-10-30T14:14:04.651413+0000","last_active":"2023-10-30T14:14:05.757043+0000","last_peered":"2023-10-30T14:14:05.757043+0000","last_clean":"2023-10-30T14:14:05.757043+0000","last_became_active":"2023-10-30T14:14:04.650882+0000","last_became_peered":"2023-10-30T14:14:04.650882+0000","last_unstale":"2023-10-30T14:14:05.757043+0000","last_undegraded":"2023-10-30T14:14:05.757043+0000","last_fullsized":"2023-10-30T14:14:05.757043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:30:08.122326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585036+0000","last_change":"2023-10-30T14:14:04.588523+0000","last_active":"2023-10-30T14:14:05.585036+0000","last_peered":"2023-10-30T14:14:05.585036+0000","last_clean":"2023-10-30T14:14:05.585036+0000","last_became_active":"2023-10-30T14:14:04.587942+0000","last_became_peered":"2023-10-30T14:14:04.587942+0000","last_unstale":"2023-10-30T14:14:05.585036+0000","last_undegraded":"2023-10-30T14:14:05.585036+0000","last_fullsized":"2023-10-30T14:14:05.585036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:16.028564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586543+0000","last_change":"2023-10-30T14:14:04.590976+0000","last_active":"2023-10-30T14:14:05.586543+0000","last_peered":"2023-10-30T14:14:05.586543+0000","last_clean":"2023-10-30T14:14:05.586543+0000","last_became_active":"2023-10-30T14:14:04.590700+0000","last_became_peered":"2023-10-30T14:14:04.590700+0000","last_unstale":"2023-10-30T14:14:05.586543+0000","last_undegraded":"2023-10-30T14:14:05.586543+0000","last_fullsized":"2023-10-30T14:14:05.586543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:49.169325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586525+0000","last_change":"2023-10-30T14:14:04.618603+0000","last_active":"2023-10-30T14:14:05.586525+0000","last_peered":"2023-10-30T14:14:05.586525+0000","last_clean":"2023-10-30T14:14:05.586525+0000","last_became_active":"2023-10-30T14:14:04.617768+0000","last_became_peered":"2023-10-30T14:14:04.617768+0000","last_unstale":"2023-10-30T14:14:05.586525+0000","last_undegraded":"2023-10-30T14:14:05.586525+0000","last_fullsized":"2023-10-30T14:14:05.586525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:16.443567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756348+0000","last_change":"2023-10-30T14:14:04.647842+0000","last_active":"2023-10-30T14:14:05.756348+0000","last_peered":"2023-10-30T14:14:05.756348+0000","last_clean":"2023-10-30T14:14:05.756348+0000","last_became_active":"2023-10-30T14:14:04.644683+0000","last_became_peered":"2023-10-30T14:14:04.644683+0000","last_unstale":"2023-10-30T14:14:05.756348+0000","last_undegraded":"2023-10-30T14:14:05.756348+0000","last_fullsized":"2023-10-30T14:14:05.756348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:51:01.230722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585902+0000","last_change":"2023-10-30T14:14:04.620587+0000","last_active":"2023-10-30T14:14:05.585902+0000","last_peered":"2023-10-30T14:14:05.585902+0000","last_clean":"2023-10-30T14:14:05.585902+0000","last_became_active":"2023-10-30T14:14:04.619335+0000","last_became_peered":"2023-10-30T14:14:04.619335+0000","last_unstale":"2023-10-30T14:14:05.585902+0000","last_undegraded":"2023-10-30T14:14:05.585902+0000","last_fullsized":"2023-10-30T14:14:05.585902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:01.735027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755914+0000","last_change":"2023-10-30T14:14:04.641688+0000","last_active":"2023-10-30T14:14:05.755914+0000","last_peered":"2023-10-30T14:14:05.755914+0000","last_clean":"2023-10-30T14:14:05.755914+0000","last_became_active":"2023-10-30T14:14:04.641323+0000","last_became_peered":"2023-10-30T14:14:04.641323+0000","last_unstale":"2023-10-30T14:14:05.755914+0000","last_undegraded":"2023-10-30T14:14:05.755914+0000","last_fullsized":"2023-10-30T14:14:05.755914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:43:35.285019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805751+0000","last_change":"2023-10-30T14:14:04.618530+0000","last_active":"2023-10-30T14:14:05.805751+0000","last_peered":"2023-10-30T14:14:05.805751+0000","last_clean":"2023-10-30T14:14:05.805751+0000","last_became_active":"2023-10-30T14:14:04.618312+0000","last_became_peered":"2023-10-30T14:14:04.618312+0000","last_unstale":"2023-10-30T14:14:05.805751+0000","last_undegraded":"2023-10-30T14:14:05.805751+0000","last_fullsized":"2023-10-30T14:14:05.805751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:25.089106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757146+0000","last_change":"2023-10-30T14:14:04.650330+0000","last_active":"2023-10-30T14:14:05.757146+0000","last_peered":"2023-10-30T14:14:05.757146+0000","last_clean":"2023-10-30T14:14:05.757146+0000","last_became_active":"2023-10-30T14:14:04.647361+0000","last_became_peered":"2023-10-30T14:14:04.647361+0000","last_unstale":"2023-10-30T14:14:05.757146+0000","last_undegraded":"2023-10-30T14:14:05.757146+0000","last_fullsized":"2023-10-30T14:14:05.757146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:49:57.370980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808090+0000","last_change":"2023-10-30T14:14:04.631540+0000","last_active":"2023-10-30T14:14:05.808090+0000","last_peered":"2023-10-30T14:14:05.808090+0000","last_clean":"2023-10-30T14:14:05.808090+0000","last_became_active":"2023-10-30T14:14:04.630959+0000","last_became_peered":"2023-10-30T14:14:04.630959+0000","last_unstale":"2023-10-30T14:14:05.808090+0000","last_undegraded":"2023-10-30T14:14:05.808090+0000","last_fullsized":"2023-10-30T14:14:05.808090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:49.468429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807762+0000","last_change":"2023-10-30T14:14:04.613800+0000","last_active":"2023-10-30T14:14:05.807762+0000","last_peered":"2023-10-30T14:14:05.807762+0000","last_clean":"2023-10-30T14:14:05.807762+0000","last_became_active":"2023-10-30T14:14:04.612653+0000","last_became_peered":"2023-10-30T14:14:04.612653+0000","last_unstale":"2023-10-30T14:14:05.807762+0000","last_undegraded":"2023-10-30T14:14:05.807762+0000","last_fullsized":"2023-10-30T14:14:05.807762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:44.665588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809700+0000","last_change":"2023-10-30T14:14:04.622483+0000","last_active":"2023-10-30T14:14:05.809700+0000","last_peered":"2023-10-30T14:14:05.809700+0000","last_clean":"2023-10-30T14:14:05.809700+0000","last_became_active":"2023-10-30T14:14:04.621135+0000","last_became_peered":"2023-10-30T14:14:04.621135+0000","last_unstale":"2023-10-30T14:14:05.809700+0000","last_undegraded":"2023-10-30T14:14:05.809700+0000","last_fullsized":"2023-10-30T14:14:05.809700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:52.299412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806054+0000","last_change":"2023-10-30T14:14:04.616357+0000","last_active":"2023-10-30T14:14:05.806054+0000","last_peered":"2023-10-30T14:14:05.806054+0000","last_clean":"2023-10-30T14:14:05.806054+0000","last_became_active":"2023-10-30T14:14:04.616142+0000","last_became_peered":"2023-10-30T14:14:04.616142+0000","last_unstale":"2023-10-30T14:14:05.806054+0000","last_undegraded":"2023-10-30T14:14:05.806054+0000","last_fullsized":"2023-10-30T14:14:05.806054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:32:26.432551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586106+0000","last_change":"2023-10-30T14:14:04.588759+0000","last_active":"2023-10-30T14:14:05.586106+0000","last_peered":"2023-10-30T14:14:05.586106+0000","last_clean":"2023-10-30T14:14:05.586106+0000","last_became_active":"2023-10-30T14:14:04.588452+0000","last_became_peered":"2023-10-30T14:14:04.588452+0000","last_unstale":"2023-10-30T14:14:05.586106+0000","last_undegraded":"2023-10-30T14:14:05.586106+0000","last_fullsized":"2023-10-30T14:14:05.586106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:38:21.206284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584572+0000","last_change":"2023-10-30T14:14:04.621073+0000","last_active":"2023-10-30T14:14:05.584572+0000","last_peered":"2023-10-30T14:14:05.584572+0000","last_clean":"2023-10-30T14:14:05.584572+0000","last_became_active":"2023-10-30T14:14:04.620781+0000","last_became_peered":"2023-10-30T14:14:04.620781+0000","last_unstale":"2023-10-30T14:14:05.584572+0000","last_undegraded":"2023-10-30T14:14:05.584572+0000","last_fullsized":"2023-10-30T14:14:05.584572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:17:10.938195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585712+0000","last_change":"2023-10-30T14:14:04.608942+0000","last_active":"2023-10-30T14:14:05.585712+0000","last_peered":"2023-10-30T14:14:05.585712+0000","last_clean":"2023-10-30T14:14:05.585712+0000","last_became_active":"2023-10-30T14:14:04.608190+0000","last_became_peered":"2023-10-30T14:14:04.608190+0000","last_unstale":"2023-10-30T14:14:05.585712+0000","last_undegraded":"2023-10-30T14:14:05.585712+0000","last_fullsized":"2023-10-30T14:14:05.585712+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:18:39.846841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587056+0000","last_change":"2023-10-30T14:14:04.641876+0000","last_active":"2023-10-30T14:14:05.587056+0000","last_peered":"2023-10-30T14:14:05.587056+0000","last_clean":"2023-10-30T14:14:05.587056+0000","last_became_active":"2023-10-30T14:14:04.641306+0000","last_became_peered":"2023-10-30T14:14:04.641306+0000","last_unstale":"2023-10-30T14:14:05.587056+0000","last_undegraded":"2023-10-30T14:14:05.587056+0000","last_fullsized":"2023-10-30T14:14:05.587056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:10:08.169786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805744+0000","last_change":"2023-10-30T14:14:04.613999+0000","last_active":"2023-10-30T14:14:05.805744+0000","last_peered":"2023-10-30T14:14:05.805744+0000","last_clean":"2023-10-30T14:14:05.805744+0000","last_became_active":"2023-10-30T14:14:04.613644+0000","last_became_peered":"2023-10-30T14:14:04.613644+0000","last_unstale":"2023-10-30T14:14:05.805744+0000","last_undegraded":"2023-10-30T14:14:05.805744+0000","last_fullsized":"2023-10-30T14:14:05.805744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:14:50.232028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756929+0000","last_change":"2023-10-30T14:14:04.650393+0000","last_active":"2023-10-30T14:14:05.756929+0000","last_peered":"2023-10-30T14:14:05.756929+0000","last_clean":"2023-10-30T14:14:05.756929+0000","last_became_active":"2023-10-30T14:14:04.649850+0000","last_became_peered":"2023-10-30T14:14:04.649850+0000","last_unstale":"2023-10-30T14:14:05.756929+0000","last_undegraded":"2023-10-30T14:14:05.756929+0000","last_fullsized":"2023-10-30T14:14:05.756929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:25.174023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808107+0000","last_change":"2023-10-30T14:14:04.625744+0000","last_active":"2023-10-30T14:14:05.808107+0000","last_peered":"2023-10-30T14:14:05.808107+0000","last_clean":"2023-10-30T14:14:05.808107+0000","last_became_active":"2023-10-30T14:14:04.625042+0000","last_became_peered":"2023-10-30T14:14:04.625042+0000","last_unstale":"2023-10-30T14:14:05.808107+0000","last_undegraded":"2023-10-30T14:14:05.808107+0000","last_fullsized":"2023-10-30T14:14:05.808107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:30:36.063455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586236+0000","last_change":"2023-10-30T14:14:04.633942+0000","last_active":"2023-10-30T14:14:05.586236+0000","last_peered":"2023-10-30T14:14:05.586236+0000","last_clean":"2023-10-30T14:14:05.586236+0000","last_became_active":"2023-10-30T14:14:04.633586+0000","last_became_peered":"2023-10-30T14:14:04.633586+0000","last_unstale":"2023-10-30T14:14:05.586236+0000","last_undegraded":"2023-10-30T14:14:05.586236+0000","last_fullsized":"2023-10-30T14:14:05.586236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:42:58.067892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586050+0000","last_change":"2023-10-30T14:14:04.642847+0000","last_active":"2023-10-30T14:14:05.586050+0000","last_peered":"2023-10-30T14:14:05.586050+0000","last_clean":"2023-10-30T14:14:05.586050+0000","last_became_active":"2023-10-30T14:14:04.642244+0000","last_became_peered":"2023-10-30T14:14:04.642244+0000","last_unstale":"2023-10-30T14:14:05.586050+0000","last_undegraded":"2023-10-30T14:14:05.586050+0000","last_fullsized":"2023-10-30T14:14:05.586050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:28.780866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805762+0000","last_change":"2023-10-30T14:14:04.620651+0000","last_active":"2023-10-30T14:14:05.805762+0000","last_peered":"2023-10-30T14:14:05.805762+0000","last_clean":"2023-10-30T14:14:05.805762+0000","last_became_active":"2023-10-30T14:14:04.620343+0000","last_became_peered":"2023-10-30T14:14:04.620343+0000","last_unstale":"2023-10-30T14:14:05.805762+0000","last_undegraded":"2023-10-30T14:14:05.805762+0000","last_fullsized":"2023-10-30T14:14:05.805762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:16:00.557118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587224+0000","last_change":"2023-10-30T14:14:04.610472+0000","last_active":"2023-10-30T14:14:05.587224+0000","last_peered":"2023-10-30T14:14:05.587224+0000","last_clean":"2023-10-30T14:14:05.587224+0000","last_became_active":"2023-10-30T14:14:04.609720+0000","last_became_peered":"2023-10-30T14:14:04.609720+0000","last_unstale":"2023-10-30T14:14:05.587224+0000","last_undegraded":"2023-10-30T14:14:05.587224+0000","last_fullsized":"2023-10-30T14:14:05.587224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:44:28.391589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805906+0000","last_change":"2023-10-30T14:14:04.620548+0000","last_active":"2023-10-30T14:14:05.805906+0000","last_peered":"2023-10-30T14:14:05.805906+0000","last_clean":"2023-10-30T14:14:05.805906+0000","last_became_active":"2023-10-30T14:14:04.620217+0000","last_became_peered":"2023-10-30T14:14:04.620217+0000","last_unstale":"2023-10-30T14:14:05.805906+0000","last_undegraded":"2023-10-30T14:14:05.805906+0000","last_fullsized":"2023-10-30T14:14:05.805906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:40:22.912696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808191+0000","last_change":"2023-10-30T14:14:04.644315+0000","last_active":"2023-10-30T14:14:05.808191+0000","last_peered":"2023-10-30T14:14:05.808191+0000","last_clean":"2023-10-30T14:14:05.808191+0000","last_became_active":"2023-10-30T14:14:04.643707+0000","last_became_peered":"2023-10-30T14:14:04.643707+0000","last_unstale":"2023-10-30T14:14:05.808191+0000","last_undegraded":"2023-10-30T14:14:05.808191+0000","last_fullsized":"2023-10-30T14:14:05.808191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:20:33.544167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584874+0000","last_change":"2023-10-30T14:14:04.597855+0000","last_active":"2023-10-30T14:14:05.584874+0000","last_peered":"2023-10-30T14:14:05.584874+0000","last_clean":"2023-10-30T14:14:05.584874+0000","last_became_active":"2023-10-30T14:14:04.597590+0000","last_became_peered":"2023-10-30T14:14:04.597590+0000","last_unstale":"2023-10-30T14:14:05.584874+0000","last_undegraded":"2023-10-30T14:14:05.584874+0000","last_fullsized":"2023-10-30T14:14:05.584874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:05.905027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806303+0000","last_change":"2023-10-30T14:14:04.624133+0000","last_active":"2023-10-30T14:14:05.806303+0000","last_peered":"2023-10-30T14:14:05.806303+0000","last_clean":"2023-10-30T14:14:05.806303+0000","last_became_active":"2023-10-30T14:14:04.623902+0000","last_became_peered":"2023-10-30T14:14:04.623902+0000","last_unstale":"2023-10-30T14:14:05.806303+0000","last_undegraded":"2023-10-30T14:14:05.806303+0000","last_fullsized":"2023-10-30T14:14:05.806303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:33:10.476461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586328+0000","last_change":"2023-10-30T14:14:04.597875+0000","last_active":"2023-10-30T14:14:05.586328+0000","last_peered":"2023-10-30T14:14:05.586328+0000","last_clean":"2023-10-30T14:14:05.586328+0000","last_became_active":"2023-10-30T14:14:04.597556+0000","last_became_peered":"2023-10-30T14:14:04.597556+0000","last_unstale":"2023-10-30T14:14:05.586328+0000","last_undegraded":"2023-10-30T14:14:05.586328+0000","last_fullsized":"2023-10-30T14:14:05.586328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:08:46.407610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586842+0000","last_change":"2023-10-30T14:14:04.620110+0000","last_active":"2023-10-30T14:14:05.586842+0000","last_peered":"2023-10-30T14:14:05.586842+0000","last_clean":"2023-10-30T14:14:05.586842+0000","last_became_active":"2023-10-30T14:14:04.618430+0000","last_became_peered":"2023-10-30T14:14:04.618430+0000","last_unstale":"2023-10-30T14:14:05.586842+0000","last_undegraded":"2023-10-30T14:14:05.586842+0000","last_fullsized":"2023-10-30T14:14:05.586842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:36.966564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805526+0000","last_change":"2023-10-30T14:14:04.606692+0000","last_active":"2023-10-30T14:14:05.805526+0000","last_peered":"2023-10-30T14:14:05.805526+0000","last_clean":"2023-10-30T14:14:05.805526+0000","last_became_active":"2023-10-30T14:14:04.606340+0000","last_became_peered":"2023-10-30T14:14:04.606340+0000","last_unstale":"2023-10-30T14:14:05.805526+0000","last_undegraded":"2023-10-30T14:14:05.805526+0000","last_fullsized":"2023-10-30T14:14:05.805526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:02:24.518205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807314+0000","last_change":"2023-10-30T14:14:04.613535+0000","last_active":"2023-10-30T14:14:05.807314+0000","last_peered":"2023-10-30T14:14:05.807314+0000","last_clean":"2023-10-30T14:14:05.807314+0000","last_became_active":"2023-10-30T14:14:04.612588+0000","last_became_peered":"2023-10-30T14:14:04.612588+0000","last_unstale":"2023-10-30T14:14:05.807314+0000","last_undegraded":"2023-10-30T14:14:05.807314+0000","last_fullsized":"2023-10-30T14:14:05.807314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:58:23.057988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809168+0000","last_change":"2023-10-30T14:14:04.608434+0000","last_active":"2023-10-30T14:14:05.809168+0000","last_peered":"2023-10-30T14:14:05.809168+0000","last_clean":"2023-10-30T14:14:05.809168+0000","last_became_active":"2023-10-30T14:14:04.607683+0000","last_became_peered":"2023-10-30T14:14:04.607683+0000","last_unstale":"2023-10-30T14:14:05.809168+0000","last_undegraded":"2023-10-30T14:14:05.809168+0000","last_fullsized":"2023-10-30T14:14:05.809168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:17:27.467330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.933073+0000","last_change":"2023-10-30T14:14:03.732084+0000","last_active":"2023-10-30T14:14:06.933073+0000","last_peered":"2023-10-30T14:14:06.933073+0000","last_clean":"2023-10-30T14:14:06.933073+0000","last_became_active":"2023-10-30T14:14:03.731740+0000","last_became_peered":"2023-10-30T14:14:03.731740+0000","last_unstale":"2023-10-30T14:14:06.933073+0000","last_undegraded":"2023-10-30T14:14:06.933073+0000","last_fullsized":"2023-10-30T14:14:06.933073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:34:27.462477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586305+0000","last_change":"2023-10-30T14:14:04.621869+0000","last_active":"2023-10-30T14:14:05.586305+0000","last_peered":"2023-10-30T14:14:05.586305+0000","last_clean":"2023-10-30T14:14:05.586305+0000","last_became_active":"2023-10-30T14:14:04.621526+0000","last_became_peered":"2023-10-30T14:14:04.621526+0000","last_unstale":"2023-10-30T14:14:05.586305+0000","last_undegraded":"2023-10-30T14:14:05.586305+0000","last_fullsized":"2023-10-30T14:14:05.586305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:08:21.260692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.964457+0000","last_change":"2023-10-30T14:14:03.730789+0000","last_active":"2023-10-30T14:14:06.964457+0000","last_peered":"2023-10-30T14:14:06.964457+0000","last_clean":"2023-10-30T14:14:06.964457+0000","last_became_active":"2023-10-30T14:14:03.729386+0000","last_became_peered":"2023-10-30T14:14:03.729386+0000","last_unstale":"2023-10-30T14:14:06.964457+0000","last_undegraded":"2023-10-30T14:14:06.964457+0000","last_fullsized":"2023-10-30T14:14:06.964457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805855+0000","last_change":"2023-10-30T14:14:03.724379+0000","last_active":"2023-10-30T14:14:05.805855+0000","last_peered":"2023-10-30T14:14:05.805855+0000","last_clean":"2023-10-30T14:14:05.805855+0000","last_became_active":"2023-10-30T14:14:03.724078+0000","last_became_peered":"2023-10-30T14:14:03.724078+0000","last_unstale":"2023-10-30T14:14:05.805855+0000","last_undegraded":"2023-10-30T14:14:05.805855+0000","last_fullsized":"2023-10-30T14:14:05.805855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:10:17.829899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805886+0000","last_change":"2023-10-30T14:14:04.597423+0000","last_active":"2023-10-30T14:14:05.805886+0000","last_peered":"2023-10-30T14:14:05.805886+0000","last_clean":"2023-10-30T14:14:05.805886+0000","last_became_active":"2023-10-30T14:14:04.597099+0000","last_became_peered":"2023-10-30T14:14:04.597099+0000","last_unstale":"2023-10-30T14:14:05.805886+0000","last_undegraded":"2023-10-30T14:14:05.805886+0000","last_fullsized":"2023-10-30T14:14:05.805886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:54:52.346295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585964+0000","last_change":"2023-10-30T14:14:03.721030+0000","last_active":"2023-10-30T14:14:05.585964+0000","last_peered":"2023-10-30T14:14:05.585964+0000","last_clean":"2023-10-30T14:14:05.585964+0000","last_became_active":"2023-10-30T14:14:03.719816+0000","last_became_peered":"2023-10-30T14:14:03.719816+0000","last_unstale":"2023-10-30T14:14:05.585964+0000","last_undegraded":"2023-10-30T14:14:05.585964+0000","last_fullsized":"2023-10-30T14:14:05.585964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:43:50.150156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584729+0000","last_change":"2023-10-30T14:14:04.599863+0000","last_active":"2023-10-30T14:14:05.584729+0000","last_peered":"2023-10-30T14:14:05.584729+0000","last_clean":"2023-10-30T14:14:05.584729+0000","last_became_active":"2023-10-30T14:14:04.599323+0000","last_became_peered":"2023-10-30T14:14:04.599323+0000","last_unstale":"2023-10-30T14:14:05.584729+0000","last_undegraded":"2023-10-30T14:14:05.584729+0000","last_fullsized":"2023-10-30T14:14:05.584729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:42:36.939959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808018+0000","last_change":"2023-10-30T14:14:03.738694+0000","last_active":"2023-10-30T14:14:05.808018+0000","last_peered":"2023-10-30T14:14:05.808018+0000","last_clean":"2023-10-30T14:14:05.808018+0000","last_became_active":"2023-10-30T14:14:03.738132+0000","last_became_peered":"2023-10-30T14:14:03.738132+0000","last_unstale":"2023-10-30T14:14:05.808018+0000","last_undegraded":"2023-10-30T14:14:05.808018+0000","last_fullsized":"2023-10-30T14:14:05.808018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:46:59.692900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586432+0000","last_change":"2023-10-30T14:14:04.649502+0000","last_active":"2023-10-30T14:14:05.586432+0000","last_peered":"2023-10-30T14:14:05.586432+0000","last_clean":"2023-10-30T14:14:05.586432+0000","last_became_active":"2023-10-30T14:14:04.648816+0000","last_became_peered":"2023-10-30T14:14:04.648816+0000","last_unstale":"2023-10-30T14:14:05.586432+0000","last_undegraded":"2023-10-30T14:14:05.586432+0000","last_fullsized":"2023-10-30T14:14:05.586432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:23:07.149253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586445+0000","last_change":"2023-10-30T14:14:03.726112+0000","last_active":"2023-10-30T14:14:05.586445+0000","last_peered":"2023-10-30T14:14:05.586445+0000","last_clean":"2023-10-30T14:14:05.586445+0000","last_became_active":"2023-10-30T14:14:03.725272+0000","last_became_peered":"2023-10-30T14:14:03.725272+0000","last_unstale":"2023-10-30T14:14:05.586445+0000","last_undegraded":"2023-10-30T14:14:05.586445+0000","last_fullsized":"2023-10-30T14:14:05.586445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:37:20.764294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586677+0000","last_change":"2023-10-30T14:14:04.610043+0000","last_active":"2023-10-30T14:14:05.586677+0000","last_peered":"2023-10-30T14:14:05.586677+0000","last_clean":"2023-10-30T14:14:05.586677+0000","last_became_active":"2023-10-30T14:14:04.609697+0000","last_became_peered":"2023-10-30T14:14:04.609697+0000","last_unstale":"2023-10-30T14:14:05.586677+0000","last_undegraded":"2023-10-30T14:14:05.586677+0000","last_fullsized":"2023-10-30T14:14:05.586677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:28.543547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.718722+0000","last_change":"2023-10-30T14:14:03.727871+0000","last_active":"2023-10-30T14:14:09.718722+0000","last_peered":"2023-10-30T14:14:09.718722+0000","last_clean":"2023-10-30T14:14:09.718722+0000","last_became_active":"2023-10-30T14:14:03.727422+0000","last_became_peered":"2023-10-30T14:14:03.727422+0000","last_unstale":"2023-10-30T14:14:09.718722+0000","last_undegraded":"2023-10-30T14:14:09.718722+0000","last_fullsized":"2023-10-30T14:14:09.718722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:54.782533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809764+0000","last_change":"2023-10-30T14:14:04.645734+0000","last_active":"2023-10-30T14:14:05.809764+0000","last_peered":"2023-10-30T14:14:05.809764+0000","last_clean":"2023-10-30T14:14:05.809764+0000","last_became_active":"2023-10-30T14:14:04.645333+0000","last_became_peered":"2023-10-30T14:14:04.645333+0000","last_unstale":"2023-10-30T14:14:05.809764+0000","last_undegraded":"2023-10-30T14:14:05.809764+0000","last_fullsized":"2023-10-30T14:14:05.809764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:36:11.322798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.722581+0000","last_change":"2023-10-30T14:14:03.725554+0000","last_active":"2023-10-30T14:14:09.722581+0000","last_peered":"2023-10-30T14:14:09.722581+0000","last_clean":"2023-10-30T14:14:09.722581+0000","last_became_active":"2023-10-30T14:14:03.724806+0000","last_became_peered":"2023-10-30T14:14:03.724806+0000","last_unstale":"2023-10-30T14:14:09.722581+0000","last_undegraded":"2023-10-30T14:14:09.722581+0000","last_fullsized":"2023-10-30T14:14:09.722581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:37.823561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756505+0000","last_change":"2023-10-30T14:14:04.647122+0000","last_active":"2023-10-30T14:14:05.756505+0000","last_peered":"2023-10-30T14:14:05.756505+0000","last_clean":"2023-10-30T14:14:05.756505+0000","last_became_active":"2023-10-30T14:14:04.646543+0000","last_became_peered":"2023-10-30T14:14:04.646543+0000","last_unstale":"2023-10-30T14:14:05.756505+0000","last_undegraded":"2023-10-30T14:14:05.756505+0000","last_fullsized":"2023-10-30T14:14:05.756505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:16.158442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.722910+0000","last_change":"2023-10-30T14:14:03.717903+0000","last_active":"2023-10-30T14:14:09.722910+0000","last_peered":"2023-10-30T14:14:09.722910+0000","last_clean":"2023-10-30T14:14:09.722910+0000","last_became_active":"2023-10-30T14:14:03.716941+0000","last_became_peered":"2023-10-30T14:14:03.716941+0000","last_unstale":"2023-10-30T14:14:09.722910+0000","last_undegraded":"2023-10-30T14:14:09.722910+0000","last_fullsized":"2023-10-30T14:14:09.722910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:58.178166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586173+0000","last_change":"2023-10-30T14:14:04.641757+0000","last_active":"2023-10-30T14:14:05.586173+0000","last_peered":"2023-10-30T14:14:05.586173+0000","last_clean":"2023-10-30T14:14:05.586173+0000","last_became_active":"2023-10-30T14:14:04.641494+0000","last_became_peered":"2023-10-30T14:14:04.641494+0000","last_unstale":"2023-10-30T14:14:05.586173+0000","last_undegraded":"2023-10-30T14:14:05.586173+0000","last_fullsized":"2023-10-30T14:14:05.586173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:00:18.795083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587083+0000","last_change":"2023-10-30T14:14:03.729626+0000","last_active":"2023-10-30T14:14:05.587083+0000","last_peered":"2023-10-30T14:14:05.587083+0000","last_clean":"2023-10-30T14:14:05.587083+0000","last_became_active":"2023-10-30T14:14:03.728053+0000","last_became_peered":"2023-10-30T14:14:03.728053+0000","last_unstale":"2023-10-30T14:14:05.587083+0000","last_undegraded":"2023-10-30T14:14:05.587083+0000","last_fullsized":"2023-10-30T14:14:05.587083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:44:53.362066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756773+0000","last_change":"2023-10-30T14:14:04.650513+0000","last_active":"2023-10-30T14:14:05.756773+0000","last_peered":"2023-10-30T14:14:05.756773+0000","last_clean":"2023-10-30T14:14:05.756773+0000","last_became_active":"2023-10-30T14:14:04.649896+0000","last_became_peered":"2023-10-30T14:14:04.649896+0000","last_unstale":"2023-10-30T14:14:05.756773+0000","last_undegraded":"2023-10-30T14:14:05.756773+0000","last_fullsized":"2023-10-30T14:14:05.756773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:23.100490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807384+0000","last_change":"2023-10-30T14:14:03.736824+0000","last_active":"2023-10-30T14:14:05.807384+0000","last_peered":"2023-10-30T14:14:05.807384+0000","last_clean":"2023-10-30T14:14:05.807384+0000","last_became_active":"2023-10-30T14:14:03.736155+0000","last_became_peered":"2023-10-30T14:14:03.736155+0000","last_unstale":"2023-10-30T14:14:05.807384+0000","last_undegraded":"2023-10-30T14:14:05.807384+0000","last_fullsized":"2023-10-30T14:14:05.807384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:35.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807372+0000","last_change":"2023-10-30T14:14:04.615159+0000","last_active":"2023-10-30T14:14:05.807372+0000","last_peered":"2023-10-30T14:14:05.807372+0000","last_clean":"2023-10-30T14:14:05.807372+0000","last_became_active":"2023-10-30T14:14:04.614328+0000","last_became_peered":"2023-10-30T14:14:04.614328+0000","last_unstale":"2023-10-30T14:14:05.807372+0000","last_undegraded":"2023-10-30T14:14:05.807372+0000","last_fullsized":"2023-10-30T14:14:05.807372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:16:52.999786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.722021+0000","last_change":"2023-10-30T14:14:03.700690+0000","last_active":"2023-10-30T14:14:09.722021+0000","last_peered":"2023-10-30T14:14:09.722021+0000","last_clean":"2023-10-30T14:14:09.722021+0000","last_became_active":"2023-10-30T14:14:03.700352+0000","last_became_peered":"2023-10-30T14:14:03.700352+0000","last_unstale":"2023-10-30T14:14:09.722021+0000","last_undegraded":"2023-10-30T14:14:09.722021+0000","last_fullsized":"2023-10-30T14:14:09.722021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:03:01.103359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755948+0000","last_change":"2023-10-30T14:14:04.647253+0000","last_active":"2023-10-30T14:14:05.755948+0000","last_peered":"2023-10-30T14:14:05.755948+0000","last_clean":"2023-10-30T14:14:05.755948+0000","last_became_active":"2023-10-30T14:14:04.646639+0000","last_became_peered":"2023-10-30T14:14:04.646639+0000","last_unstale":"2023-10-30T14:14:05.755948+0000","last_undegraded":"2023-10-30T14:14:05.755948+0000","last_fullsized":"2023-10-30T14:14:05.755948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:22:50.893734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.936522+0000","last_change":"2023-10-30T14:14:03.710717+0000","last_active":"2023-10-30T14:14:06.936522+0000","last_peered":"2023-10-30T14:14:06.936522+0000","last_clean":"2023-10-30T14:14:06.936522+0000","last_became_active":"2023-10-30T14:14:03.707950+0000","last_became_peered":"2023-10-30T14:14:03.707950+0000","last_unstale":"2023-10-30T14:14:06.936522+0000","last_undegraded":"2023-10-30T14:14:06.936522+0000","last_fullsized":"2023-10-30T14:14:06.936522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:29:41.232902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757066+0000","last_change":"2023-10-30T14:14:04.647688+0000","last_active":"2023-10-30T14:14:05.757066+0000","last_peered":"2023-10-30T14:14:05.757066+0000","last_clean":"2023-10-30T14:14:05.757066+0000","last_became_active":"2023-10-30T14:14:04.644761+0000","last_became_peered":"2023-10-30T14:14:04.644761+0000","last_unstale":"2023-10-30T14:14:05.757066+0000","last_undegraded":"2023-10-30T14:14:05.757066+0000","last_fullsized":"2023-10-30T14:14:05.757066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:00:38.482522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.716374+0000","last_change":"2023-10-30T14:14:03.705567+0000","last_active":"2023-10-30T14:14:09.716374+0000","last_peered":"2023-10-30T14:14:09.716374+0000","last_clean":"2023-10-30T14:14:09.716374+0000","last_became_active":"2023-10-30T14:14:03.705186+0000","last_became_peered":"2023-10-30T14:14:03.705186+0000","last_unstale":"2023-10-30T14:14:09.716374+0000","last_undegraded":"2023-10-30T14:14:09.716374+0000","last_fullsized":"2023-10-30T14:14:09.716374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:14.609243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757235+0000","last_change":"2023-10-30T14:14:04.647335+0000","last_active":"2023-10-30T14:14:05.757235+0000","last_peered":"2023-10-30T14:14:05.757235+0000","last_clean":"2023-10-30T14:14:05.757235+0000","last_became_active":"2023-10-30T14:14:04.646785+0000","last_became_peered":"2023-10-30T14:14:04.646785+0000","last_unstale":"2023-10-30T14:14:05.757235+0000","last_undegraded":"2023-10-30T14:14:05.757235+0000","last_fullsized":"2023-10-30T14:14:05.757235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:58:41.600761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.935812+0000","last_change":"2023-10-30T14:14:03.693349+0000","last_active":"2023-10-30T14:14:06.935812+0000","last_peered":"2023-10-30T14:14:06.935812+0000","last_clean":"2023-10-30T14:14:06.935812+0000","last_became_active":"2023-10-30T14:14:03.692966+0000","last_became_peered":"2023-10-30T14:14:03.692966+0000","last_unstale":"2023-10-30T14:14:06.935812+0000","last_undegraded":"2023-10-30T14:14:06.935812+0000","last_fullsized":"2023-10-30T14:14:06.935812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:30:28.800214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587213+0000","last_change":"2023-10-30T14:14:04.617023+0000","last_active":"2023-10-30T14:14:05.587213+0000","last_peered":"2023-10-30T14:14:05.587213+0000","last_clean":"2023-10-30T14:14:05.587213+0000","last_became_active":"2023-10-30T14:14:04.615925+0000","last_became_peered":"2023-10-30T14:14:04.615925+0000","last_unstale":"2023-10-30T14:14:05.587213+0000","last_undegraded":"2023-10-30T14:14:05.587213+0000","last_fullsized":"2023-10-30T14:14:05.587213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:32:02.289324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805586+0000","last_change":"2023-10-30T14:14:03.727659+0000","last_active":"2023-10-30T14:14:05.805586+0000","last_peered":"2023-10-30T14:14:05.805586+0000","last_clean":"2023-10-30T14:14:05.805586+0000","last_became_active":"2023-10-30T14:14:03.727409+0000","last_became_peered":"2023-10-30T14:14:03.727409+0000","last_unstale":"2023-10-30T14:14:05.805586+0000","last_undegraded":"2023-10-30T14:14:05.805586+0000","last_fullsized":"2023-10-30T14:14:05.805586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:24.210635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756468+0000","last_change":"2023-10-30T14:14:04.650083+0000","last_active":"2023-10-30T14:14:05.756468+0000","last_peered":"2023-10-30T14:14:05.756468+0000","last_clean":"2023-10-30T14:14:05.756468+0000","last_became_active":"2023-10-30T14:14:04.647562+0000","last_became_peered":"2023-10-30T14:14:04.647562+0000","last_unstale":"2023-10-30T14:14:05.756468+0000","last_undegraded":"2023-10-30T14:14:05.756468+0000","last_fullsized":"2023-10-30T14:14:05.756468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:11:41.743850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T14:14:04.632968+0000","last_change":"2023-10-30T14:13:54.251572+0000","last_active":"2023-10-30T14:14:04.632968+0000","last_peered":"2023-10-30T14:14:04.632968+0000","last_clean":"2023-10-30T14:14:04.632968+0000","last_became_active":"2023-10-30T14:13:54.250907+0000","last_became_peered":"2023-10-30T14:13:54.250907+0000","last_unstale":"2023-10-30T14:14:04.632968+0000","last_undegraded":"2023-10-30T14:14:04.632968+0000","last_fullsized":"2023-10-30T14:14:04.632968+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_clean_scrub_stamp":"2023-10-30T14:13:53.220267+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:44.590852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585892+0000","last_change":"2023-10-30T14:14:03.729034+0000","last_active":"2023-10-30T14:14:05.585892+0000","last_peered":"2023-10-30T14:14:05.585892+0000","last_clean":"2023-10-30T14:14:05.585892+0000","last_became_active":"2023-10-30T14:14:03.728293+0000","last_became_peered":"2023-10-30T14:14:03.728293+0000","last_unstale":"2023-10-30T14:14:05.585892+0000","last_undegraded":"2023-10-30T14:14:05.585892+0000","last_fullsized":"2023-10-30T14:14:05.585892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806225+0000","last_change":"2023-10-30T14:14:04.630325+0000","last_active":"2023-10-30T14:14:05.806225+0000","last_peered":"2023-10-30T14:14:05.806225+0000","last_clean":"2023-10-30T14:14:05.806225+0000","last_became_active":"2023-10-30T14:14:04.630121+0000","last_became_peered":"2023-10-30T14:14:04.630121+0000","last_unstale":"2023-10-30T14:14:05.806225+0000","last_undegraded":"2023-10-30T14:14:05.806225+0000","last_fullsized":"2023-10-30T14:14:05.806225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:02:28.792879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755708+0000","last_change":"2023-10-30T14:14:03.732909+0000","last_active":"2023-10-30T14:14:05.755708+0000","last_peered":"2023-10-30T14:14:05.755708+0000","last_clean":"2023-10-30T14:14:05.755708+0000","last_became_active":"2023-10-30T14:14:03.732199+0000","last_became_peered":"2023-10-30T14:14:03.732199+0000","last_unstale":"2023-10-30T14:14:05.755708+0000","last_undegraded":"2023-10-30T14:14:05.755708+0000","last_fullsized":"2023-10-30T14:14:05.755708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:38.117750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586172+0000","last_change":"2023-10-30T14:14:04.620314+0000","last_active":"2023-10-30T14:14:05.586172+0000","last_peered":"2023-10-30T14:14:05.586172+0000","last_clean":"2023-10-30T14:14:05.586172+0000","last_became_active":"2023-10-30T14:14:04.619990+0000","last_became_peered":"2023-10-30T14:14:04.619990+0000","last_unstale":"2023-10-30T14:14:05.586172+0000","last_undegraded":"2023-10-30T14:14:05.586172+0000","last_fullsized":"2023-10-30T14:14:05.586172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:38:25.657586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805642+0000","last_change":"2023-10-30T14:14:03.702862+0000","last_active":"2023-10-30T14:14:05.805642+0000","last_peered":"2023-10-30T14:14:05.805642+0000","last_clean":"2023-10-30T14:14:05.805642+0000","last_became_active":"2023-10-30T14:14:03.702513+0000","last_became_peered":"2023-10-30T14:14:03.702513+0000","last_unstale":"2023-10-30T14:14:05.805642+0000","last_undegraded":"2023-10-30T14:14:05.805642+0000","last_fullsized":"2023-10-30T14:14:05.805642+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:25.499390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584623+0000","last_change":"2023-10-30T14:14:04.589353+0000","last_active":"2023-10-30T14:14:05.584623+0000","last_peered":"2023-10-30T14:14:05.584623+0000","last_clean":"2023-10-30T14:14:05.584623+0000","last_became_active":"2023-10-30T14:14:04.589087+0000","last_became_peered":"2023-10-30T14:14:04.589087+0000","last_unstale":"2023-10-30T14:14:05.584623+0000","last_undegraded":"2023-10-30T14:14:05.584623+0000","last_fullsized":"2023-10-30T14:14:05.584623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:20:12.808641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587305+0000","last_change":"2023-10-30T14:14:03.730048+0000","last_active":"2023-10-30T14:14:05.587305+0000","last_peered":"2023-10-30T14:14:05.587305+0000","last_clean":"2023-10-30T14:14:05.587305+0000","last_became_active":"2023-10-30T14:14:03.728925+0000","last_became_peered":"2023-10-30T14:14:03.728925+0000","last_unstale":"2023-10-30T14:14:05.587305+0000","last_undegraded":"2023-10-30T14:14:05.587305+0000","last_fullsized":"2023-10-30T14:14:05.587305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:32:05.056039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809437+0000","last_change":"2023-10-30T14:14:04.608289+0000","last_active":"2023-10-30T14:14:05.809437+0000","last_peered":"2023-10-30T14:14:05.809437+0000","last_clean":"2023-10-30T14:14:05.809437+0000","last_became_active":"2023-10-30T14:14:04.607377+0000","last_became_peered":"2023-10-30T14:14:04.607377+0000","last_unstale":"2023-10-30T14:14:05.809437+0000","last_undegraded":"2023-10-30T14:14:05.809437+0000","last_fullsized":"2023-10-30T14:14:05.809437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:43:40.869579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584795+0000","last_change":"2023-10-30T14:14:03.693753+0000","last_active":"2023-10-30T14:14:05.584795+0000","last_peered":"2023-10-30T14:14:05.584795+0000","last_clean":"2023-10-30T14:14:05.584795+0000","last_became_active":"2023-10-30T14:14:03.692621+0000","last_became_peered":"2023-10-30T14:14:03.692621+0000","last_unstale":"2023-10-30T14:14:05.584795+0000","last_undegraded":"2023-10-30T14:14:05.584795+0000","last_fullsized":"2023-10-30T14:14:05.584795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:32:28.577584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587225+0000","last_change":"2023-10-30T14:14:04.614649+0000","last_active":"2023-10-30T14:14:05.587225+0000","last_peered":"2023-10-30T14:14:05.587225+0000","last_clean":"2023-10-30T14:14:05.587225+0000","last_became_active":"2023-10-30T14:14:04.613780+0000","last_became_peered":"2023-10-30T14:14:04.613780+0000","last_unstale":"2023-10-30T14:14:05.587225+0000","last_undegraded":"2023-10-30T14:14:05.587225+0000","last_fullsized":"2023-10-30T14:14:05.587225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:47:40.354022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757037+0000","last_change":"2023-10-30T14:14:03.740213+0000","last_active":"2023-10-30T14:14:05.757037+0000","last_peered":"2023-10-30T14:14:05.757037+0000","last_clean":"2023-10-30T14:14:05.757037+0000","last_became_active":"2023-10-30T14:14:03.739650+0000","last_became_peered":"2023-10-30T14:14:03.739650+0000","last_unstale":"2023-10-30T14:14:05.757037+0000","last_undegraded":"2023-10-30T14:14:05.757037+0000","last_fullsized":"2023-10-30T14:14:05.757037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:54.120757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584691+0000","last_change":"2023-10-30T14:14:04.590569+0000","last_active":"2023-10-30T14:14:05.584691+0000","last_peered":"2023-10-30T14:14:05.584691+0000","last_clean":"2023-10-30T14:14:05.584691+0000","last_became_active":"2023-10-30T14:14:04.590138+0000","last_became_peered":"2023-10-30T14:14:04.590138+0000","last_unstale":"2023-10-30T14:14:05.584691+0000","last_undegraded":"2023-10-30T14:14:05.584691+0000","last_fullsized":"2023-10-30T14:14:05.584691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:41:30.333668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585050+0000","last_change":"2023-10-30T14:14:03.721450+0000","last_active":"2023-10-30T14:14:05.585050+0000","last_peered":"2023-10-30T14:14:05.585050+0000","last_clean":"2023-10-30T14:14:05.585050+0000","last_became_active":"2023-10-30T14:14:03.721234+0000","last_became_peered":"2023-10-30T14:14:03.721234+0000","last_unstale":"2023-10-30T14:14:05.585050+0000","last_undegraded":"2023-10-30T14:14:05.585050+0000","last_fullsized":"2023-10-30T14:14:05.585050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:26.959676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806005+0000","last_change":"2023-10-30T14:14:04.634004+0000","last_active":"2023-10-30T14:14:05.806005+0000","last_peered":"2023-10-30T14:14:05.806005+0000","last_clean":"2023-10-30T14:14:05.806005+0000","last_became_active":"2023-10-30T14:14:04.633643+0000","last_became_peered":"2023-10-30T14:14:04.633643+0000","last_unstale":"2023-10-30T14:14:05.806005+0000","last_undegraded":"2023-10-30T14:14:05.806005+0000","last_fullsized":"2023-10-30T14:14:05.806005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:05:14.572477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.721271+0000","last_change":"2023-10-30T14:14:03.738557+0000","last_active":"2023-10-30T14:14:09.721271+0000","last_peered":"2023-10-30T14:14:09.721271+0000","last_clean":"2023-10-30T14:14:09.721271+0000","last_became_active":"2023-10-30T14:14:03.737801+0000","last_became_peered":"2023-10-30T14:14:03.737801+0000","last_unstale":"2023-10-30T14:14:09.721271+0000","last_undegraded":"2023-10-30T14:14:09.721271+0000","last_fullsized":"2023-10-30T14:14:09.721271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:26.938343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586581+0000","last_change":"2023-10-30T14:14:04.589273+0000","last_active":"2023-10-30T14:14:05.586581+0000","last_peered":"2023-10-30T14:14:05.586581+0000","last_clean":"2023-10-30T14:14:05.586581+0000","last_became_active":"2023-10-30T14:14:04.588826+0000","last_became_peered":"2023-10-30T14:14:04.588826+0000","last_unstale":"2023-10-30T14:14:05.586581+0000","last_undegraded":"2023-10-30T14:14:05.586581+0000","last_fullsized":"2023-10-30T14:14:05.586581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:33:52.452394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755689+0000","last_change":"2023-10-30T14:14:03.733836+0000","last_active":"2023-10-30T14:14:05.755689+0000","last_peered":"2023-10-30T14:14:05.755689+0000","last_clean":"2023-10-30T14:14:05.755689+0000","last_became_active":"2023-10-30T14:14:03.732912+0000","last_became_peered":"2023-10-30T14:14:03.732912+0000","last_unstale":"2023-10-30T14:14:05.755689+0000","last_undegraded":"2023-10-30T14:14:05.755689+0000","last_fullsized":"2023-10-30T14:14:05.755689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:50.332738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809314+0000","last_change":"2023-10-30T14:14:04.610696+0000","last_active":"2023-10-30T14:14:05.809314+0000","last_peered":"2023-10-30T14:14:05.809314+0000","last_clean":"2023-10-30T14:14:05.809314+0000","last_became_active":"2023-10-30T14:14:04.609835+0000","last_became_peered":"2023-10-30T14:14:04.609835+0000","last_unstale":"2023-10-30T14:14:05.809314+0000","last_undegraded":"2023-10-30T14:14:05.809314+0000","last_fullsized":"2023-10-30T14:14:05.809314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:04:31.270262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584669+0000","last_change":"2023-10-30T14:14:03.732539+0000","last_active":"2023-10-30T14:14:05.584669+0000","last_peered":"2023-10-30T14:14:05.584669+0000","last_clean":"2023-10-30T14:14:05.584669+0000","last_became_active":"2023-10-30T14:14:03.732265+0000","last_became_peered":"2023-10-30T14:14:03.732265+0000","last_unstale":"2023-10-30T14:14:05.584669+0000","last_undegraded":"2023-10-30T14:14:05.584669+0000","last_fullsized":"2023-10-30T14:14:05.584669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:02:00.929612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806211+0000","last_change":"2023-10-30T14:14:04.595357+0000","last_active":"2023-10-30T14:14:05.806211+0000","last_peered":"2023-10-30T14:14:05.806211+0000","last_clean":"2023-10-30T14:14:05.806211+0000","last_became_active":"2023-10-30T14:14:04.594998+0000","last_became_peered":"2023-10-30T14:14:04.594998+0000","last_unstale":"2023-10-30T14:14:05.806211+0000","last_undegraded":"2023-10-30T14:14:05.806211+0000","last_fullsized":"2023-10-30T14:14:05.806211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:44.365673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584615+0000","last_change":"2023-10-30T14:14:03.703744+0000","last_active":"2023-10-30T14:14:05.584615+0000","last_peered":"2023-10-30T14:14:05.584615+0000","last_clean":"2023-10-30T14:14:05.584615+0000","last_became_active":"2023-10-30T14:14:03.701759+0000","last_became_peered":"2023-10-30T14:14:03.701759+0000","last_unstale":"2023-10-30T14:14:05.584615+0000","last_undegraded":"2023-10-30T14:14:05.584615+0000","last_fullsized":"2023-10-30T14:14:05.584615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:22:27.948007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806186+0000","last_change":"2023-10-30T14:14:04.600759+0000","last_active":"2023-10-30T14:14:05.806186+0000","last_peered":"2023-10-30T14:14:05.806186+0000","last_clean":"2023-10-30T14:14:05.806186+0000","last_became_active":"2023-10-30T14:14:04.600100+0000","last_became_peered":"2023-10-30T14:14:04.600100+0000","last_unstale":"2023-10-30T14:14:05.806186+0000","last_undegraded":"2023-10-30T14:14:05.806186+0000","last_fullsized":"2023-10-30T14:14:05.806186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:02.831207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.723470+0000","last_change":"2023-10-30T14:14:03.741268+0000","last_active":"2023-10-30T14:14:09.723470+0000","last_peered":"2023-10-30T14:14:09.723470+0000","last_clean":"2023-10-30T14:14:09.723470+0000","last_became_active":"2023-10-30T14:14:03.740648+0000","last_became_peered":"2023-10-30T14:14:03.740648+0000","last_unstale":"2023-10-30T14:14:09.723470+0000","last_undegraded":"2023-10-30T14:14:09.723470+0000","last_fullsized":"2023-10-30T14:14:09.723470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:30.799148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807541+0000","last_change":"2023-10-30T14:14:04.623644+0000","last_active":"2023-10-30T14:14:05.807541+0000","last_peered":"2023-10-30T14:14:05.807541+0000","last_clean":"2023-10-30T14:14:05.807541+0000","last_became_active":"2023-10-30T14:14:04.622921+0000","last_became_peered":"2023-10-30T14:14:04.622921+0000","last_unstale":"2023-10-30T14:14:05.807541+0000","last_undegraded":"2023-10-30T14:14:05.807541+0000","last_fullsized":"2023-10-30T14:14:05.807541+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:11:41.810610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587103+0000","last_change":"2023-10-30T14:14:04.624799+0000","last_active":"2023-10-30T14:14:05.587103+0000","last_peered":"2023-10-30T14:14:05.587103+0000","last_clean":"2023-10-30T14:14:05.587103+0000","last_became_active":"2023-10-30T14:14:04.621885+0000","last_became_peered":"2023-10-30T14:14:04.621885+0000","last_unstale":"2023-10-30T14:14:05.587103+0000","last_undegraded":"2023-10-30T14:14:05.587103+0000","last_fullsized":"2023-10-30T14:14:05.587103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:12.055247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.721291+0000","last_change":"2023-10-30T14:14:03.740897+0000","last_active":"2023-10-30T14:14:09.721291+0000","last_peered":"2023-10-30T14:14:09.721291+0000","last_clean":"2023-10-30T14:14:09.721291+0000","last_became_active":"2023-10-30T14:14:03.740247+0000","last_became_peered":"2023-10-30T14:14:03.740247+0000","last_unstale":"2023-10-30T14:14:09.721291+0000","last_undegraded":"2023-10-30T14:14:09.721291+0000","last_fullsized":"2023-10-30T14:14:09.721291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:29.601978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586861+0000","last_change":"2023-10-30T14:14:04.625498+0000","last_active":"2023-10-30T14:14:05.586861+0000","last_peered":"2023-10-30T14:14:05.586861+0000","last_clean":"2023-10-30T14:14:05.586861+0000","last_became_active":"2023-10-30T14:14:04.623638+0000","last_became_peered":"2023-10-30T14:14:04.623638+0000","last_unstale":"2023-10-30T14:14:05.586861+0000","last_undegraded":"2023-10-30T14:14:05.586861+0000","last_fullsized":"2023-10-30T14:14:05.586861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:19.717965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.719605+0000","last_change":"2023-10-30T14:14:03.733233+0000","last_active":"2023-10-30T14:14:09.719605+0000","last_peered":"2023-10-30T14:14:09.719605+0000","last_clean":"2023-10-30T14:14:09.719605+0000","last_became_active":"2023-10-30T14:14:03.732538+0000","last_became_peered":"2023-10-30T14:14:03.732538+0000","last_unstale":"2023-10-30T14:14:09.719605+0000","last_undegraded":"2023-10-30T14:14:09.719605+0000","last_fullsized":"2023-10-30T14:14:09.719605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:25.696173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586619+0000","last_change":"2023-10-30T14:14:04.612104+0000","last_active":"2023-10-30T14:14:05.586619+0000","last_peered":"2023-10-30T14:14:05.586619+0000","last_clean":"2023-10-30T14:14:05.586619+0000","last_became_active":"2023-10-30T14:14:04.611773+0000","last_became_peered":"2023-10-30T14:14:04.611773+0000","last_unstale":"2023-10-30T14:14:05.586619+0000","last_undegraded":"2023-10-30T14:14:05.586619+0000","last_fullsized":"2023-10-30T14:14:05.586619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:44:21.952103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.933126+0000","last_change":"2023-10-30T14:14:03.703988+0000","last_active":"2023-10-30T14:14:06.933126+0000","last_peered":"2023-10-30T14:14:06.933126+0000","last_clean":"2023-10-30T14:14:06.933126+0000","last_became_active":"2023-10-30T14:14:03.703691+0000","last_became_peered":"2023-10-30T14:14:03.703691+0000","last_unstale":"2023-10-30T14:14:06.933126+0000","last_undegraded":"2023-10-30T14:14:06.933126+0000","last_fullsized":"2023-10-30T14:14:06.933126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:34.538541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806119+0000","last_change":"2023-10-30T14:14:04.613738+0000","last_active":"2023-10-30T14:14:05.806119+0000","last_peered":"2023-10-30T14:14:05.806119+0000","last_clean":"2023-10-30T14:14:05.806119+0000","last_became_active":"2023-10-30T14:14:04.613420+0000","last_became_peered":"2023-10-30T14:14:04.613420+0000","last_unstale":"2023-10-30T14:14:05.806119+0000","last_undegraded":"2023-10-30T14:14:05.806119+0000","last_fullsized":"2023-10-30T14:14:05.806119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:01:52.695195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.720791+0000","last_change":"2023-10-30T14:14:03.737319+0000","last_active":"2023-10-30T14:14:09.720791+0000","last_peered":"2023-10-30T14:14:09.720791+0000","last_clean":"2023-10-30T14:14:09.720791+0000","last_became_active":"2023-10-30T14:14:03.736964+0000","last_became_peered":"2023-10-30T14:14:03.736964+0000","last_unstale":"2023-10-30T14:14:09.720791+0000","last_undegraded":"2023-10-30T14:14:09.720791+0000","last_fullsized":"2023-10-30T14:14:09.720791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:41.469809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":815104,"data_stored":666519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.47}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":75659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.056}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":77181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":75569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":823296,"data_stored":666575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:14:16.113 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T14:14:16.113 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-30T14:14:16.113 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-30T14:14:16.114 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-30T14:14:16.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.284+0000 7ff26e5aa640 1 Processor -- start 2023-10-30T14:14:16.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.284+0000 7ff26e5aa640 1 -- start start 2023-10-30T14:14:16.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.288+0000 7ff26e5aa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff268151950 0x7ff26806fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:16.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.288+0000 7ff26e5aa640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff268070400 0x7ff2680c57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:16.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.288+0000 7ff26e5aa640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff268158520 con 0x7ff268072a20 2023-10-30T14:14:16.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.288+0000 7ff26e5aa640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2681587e0 con 0x7ff268070400 2023-10-30T14:14:16.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.288+0000 7ff26e5aa640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff268070af0 con 0x7ff268151950 2023-10-30T14:14:16.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.288+0000 7ff2677fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff268072a20 0x7ff26806b6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:37726/0 (socket says 172.21.15.57:37726) 2023-10-30T14:14:16.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.288+0000 7ff2677fe640 1 -- 172.21.15.57:0/3010651166 learned_addr learned my addr 172.21.15.57:0/3010651166 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:16.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.288+0000 7ff26cb47640 1 --2- 172.21.15.57:0/3010651166 >> v2:172.21.15.57:3300/0 conn(0x7ff268070400 0x7ff2680c57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:16.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.288+0000 7ff267fff640 1 --2- 172.21.15.57:0/3010651166 >> v2:172.21.15.88:3300/0 conn(0x7ff268151950 0x7ff26806fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:16.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.288+0000 7ff266ffd640 1 -- 172.21.15.57:0/3010651166 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186288035 0 0) 0x7ff268158520 con 0x7ff268072a20 2023-10-30T14:14:16.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.288+0000 7ff26cb47640 1 -- 172.21.15.57:0/3010651166 >> v2:172.21.15.88:3300/0 conn(0x7ff268151950 msgr2=0x7ff26806fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:16.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.288+0000 7ff26cb47640 1 --2- 172.21.15.57:0/3010651166 >> v2:172.21.15.88:3300/0 conn(0x7ff268151950 0x7ff26806fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.288+0000 7ff26cb47640 1 -- 172.21.15.57:0/3010651166 >> v1:172.21.15.57:6789/0 conn(0x7ff268072a20 legacy=0x7ff26806b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:16.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.288+0000 7ff26cb47640 1 -- 172.21.15.57:0/3010651166 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff268070dd0 con 0x7ff268070400 2023-10-30T14:14:16.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.292+0000 7ff26cb47640 1 --2- 172.21.15.57:0/3010651166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff268070400 0x7ff2680c57e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff2680c84b0 tx=0x7ff26001f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee49e2e33079852d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:16.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.292+0000 7ff266ffd640 1 -- 172.21.15.57:0/3010651166 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2600b0440 con 0x7ff268070400 2023-10-30T14:14:16.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.292+0000 7ff266ffd640 1 -- 172.21.15.57:0/3010651166 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2600b05a0 con 0x7ff268070400 2023-10-30T14:14:16.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.292+0000 7ff266ffd640 1 -- 172.21.15.57:0/3010651166 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2600b0930 con 0x7ff268070400 2023-10-30T14:14:16.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.292+0000 7ff26e5aa640 1 -- 172.21.15.57:0/3010651166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff268070400 msgr2=0x7ff2680c57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:16.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.292+0000 7ff26e5aa640 1 --2- 172.21.15.57:0/3010651166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff268070400 0x7ff2680c57e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff2680c84b0 tx=0x7ff26001f0f0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.292+0000 7ff26e5aa640 1 -- 172.21.15.57:0/3010651166 shutdown_connections 2023-10-30T14:14:16.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.292+0000 7ff26e5aa640 1 --2- 172.21.15.57:0/3010651166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff268070400 0x7ff2680c57e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.292+0000 7ff26e5aa640 1 --2- 172.21.15.57:0/3010651166 >> v2:172.21.15.88:3300/0 conn(0x7ff268151950 0x7ff26806fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.292+0000 7ff26e5aa640 1 -- 172.21.15.57:0/3010651166 >> 172.21.15.57:0/3010651166 conn(0x7ff2680c78e0 msgr2=0x7ff2680c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:16.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.292+0000 7ff26e5aa640 1 -- 172.21.15.57:0/3010651166 shutdown_connections 2023-10-30T14:14:16.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.292+0000 7ff26e5aa640 1 -- 172.21.15.57:0/3010651166 wait complete. 2023-10-30T14:14:16.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.292+0000 7ff26e5aa640 1 Processor -- start 2023-10-30T14:14:16.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.292+0000 7ff26e5aa640 1 -- start start 2023-10-30T14:14:16.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff26e5aa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff268070400 0x7ff2680c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:16.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff26e5aa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff268072a20 0x7ff268157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:16.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff26e5aa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff268151950 0x7ff2680c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:16.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff26e5aa640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2681ee920 con 0x7ff268072a20 2023-10-30T14:14:16.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff26e5aa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2681e51b0 con 0x7ff268151950 2023-10-30T14:14:16.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff26e5aa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2680c3e70 con 0x7ff268070400 2023-10-30T14:14:16.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff2677fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff268072a20 0x7ff268157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:16.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff2677fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff268072a20 0x7ff268157380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39978/0 (socket says 172.21.15.57:39978) 2023-10-30T14:14:16.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff2677fe640 1 -- 172.21.15.57:0/2748480960 learned_addr learned my addr 172.21.15.57:0/2748480960 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:16.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff267fff640 1 --2- 172.21.15.57:0/2748480960 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff268070400 0x7ff2680c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:16.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff26cb47640 1 --2- 172.21.15.57:0/2748480960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff268151950 0x7ff2680c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:16.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff2677fe640 1 -- 172.21.15.57:0/2748480960 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff268070400 msgr2=0x7ff2680c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:16.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff2677fe640 1 --2- 172.21.15.57:0/2748480960 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff268070400 0x7ff2680c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff2677fe640 1 -- 172.21.15.57:0/2748480960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff268151950 msgr2=0x7ff2680c34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:16.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff2677fe640 1 --2- 172.21.15.57:0/2748480960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff268151950 0x7ff2680c34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff2677fe640 1 -- 172.21.15.57:0/2748480960 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff260009a70 con 0x7ff268072a20 2023-10-30T14:14:16.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff26cb47640 1 --2- 172.21.15.57:0/2748480960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff268151950 0x7ff2680c34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:16.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff267fff640 1 --2- 172.21.15.57:0/2748480960 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff268070400 0x7ff2680c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:16.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff2677fe640 1 --2- 172.21.15.57:0/2748480960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff268072a20 0x7ff268157380 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0011f0 tx=0x7ff25c00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:16.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff264ff9640 1 -- 172.21.15.57:0/2748480960 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c0074e0 con 0x7ff268072a20 2023-10-30T14:14:16.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff264ff9640 1 -- 172.21.15.57:0/2748480960 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25c007640 con 0x7ff268072a20 2023-10-30T14:14:16.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff26e5aa640 1 -- 172.21.15.57:0/2748480960 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2680c4150 con 0x7ff268072a20 2023-10-30T14:14:16.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff264ff9640 1 -- 172.21.15.57:0/2748480960 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c004a10 con 0x7ff268072a20 2023-10-30T14:14:16.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff26e5aa640 1 -- 172.21.15.57:0/2748480960 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2681e56e0 con 0x7ff268072a20 2023-10-30T14:14:16.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff26e5aa640 1 -- 172.21.15.57:0/2748480960 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2680c8840 con 0x7ff268072a20 2023-10-30T14:14:16.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.296+0000 7ff264ff9640 1 -- 172.21.15.57:0/2748480960 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff25c004b70 con 0x7ff268072a20 2023-10-30T14:14:16.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.300+0000 7ff264ff9640 1 --2- 172.21.15.57:0/2748480960 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff2540664a0 0x7ff254068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:16.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.300+0000 7ff264ff9640 1 -- 172.21.15.57:0/2748480960 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff25c083410 con 0x7ff268072a20 2023-10-30T14:14:16.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.300+0000 7ff267fff640 1 --2- 172.21.15.57:0/2748480960 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff2540664a0 0x7ff254068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:16.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.300+0000 7ff267fff640 1 --2- 172.21.15.57:0/2748480960 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff2540664a0 0x7ff254068960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff258009b10 tx=0x7ff258008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:16.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.304+0000 7ff264ff9640 1 -- 172.21.15.57:0/2748480960 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff25c050890 con 0x7ff268072a20 2023-10-30T14:14:16.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.492+0000 7ff26e5aa640 1 -- 172.21.15.57:0/2748480960 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff2681eea70 con 0x7ff268072a20 2023-10-30T14:14:16.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.492+0000 7ff264ff9640 1 -- 172.21.15.57:0/2748480960 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7ff2681eea70 con 0x7ff268072a20 2023-10-30T14:14:16.499 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:16.499 INFO:teuthology.orchestra.run.smithi057.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-30T14:14:16.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.496+0000 7ff26e5aa640 1 -- 172.21.15.57:0/2748480960 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff2540664a0 msgr2=0x7ff254068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:16.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.496+0000 7ff26e5aa640 1 --2- 172.21.15.57:0/2748480960 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff2540664a0 0x7ff254068960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff258009b10 tx=0x7ff258008040 comp rx=0 tx=0).stop 2023-10-30T14:14:16.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.496+0000 7ff26e5aa640 1 -- 172.21.15.57:0/2748480960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff268072a20 msgr2=0x7ff268157380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:16.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.496+0000 7ff26e5aa640 1 --2- 172.21.15.57:0/2748480960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff268072a20 0x7ff268157380 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0011f0 tx=0x7ff25c00c9b0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.496+0000 7ff26e5aa640 1 -- 172.21.15.57:0/2748480960 shutdown_connections 2023-10-30T14:14:16.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.496+0000 7ff26e5aa640 1 --2- 172.21.15.57:0/2748480960 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff2540664a0 0x7ff254068960 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.496+0000 7ff26e5aa640 1 --2- 172.21.15.57:0/2748480960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff268151950 0x7ff2680c34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.496+0000 7ff26e5aa640 1 --2- 172.21.15.57:0/2748480960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff268072a20 0x7ff268157380 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.496+0000 7ff26e5aa640 1 --2- 172.21.15.57:0/2748480960 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff268070400 0x7ff2680c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.496+0000 7ff26e5aa640 1 -- 172.21.15.57:0/2748480960 >> 172.21.15.57:0/2748480960 conn(0x7ff2680c78e0 msgr2=0x7ff2680c54d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:16.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.496+0000 7ff26e5aa640 1 -- 172.21.15.57:0/2748480960 shutdown_connections 2023-10-30T14:14:16.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.496+0000 7ff26e5aa640 1 -- 172.21.15.57:0/2748480960 wait complete. 2023-10-30T14:14:16.520 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-30T14:14:16.520 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:14:16.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.656+0000 7fd5428c8640 1 Processor -- start 2023-10-30T14:14:16.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.660+0000 7fd5428c8640 1 -- start start 2023-10-30T14:14:16.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.660+0000 7fd5428c8640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd53c0c8eb0 0x7fd53c15cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:16.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.660+0000 7fd5428c8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd53c06aeb0 0x7fd53c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:16.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.660+0000 7fd5428c8640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd53c15dcc0 con 0x7fd53c15ac70 2023-10-30T14:14:16.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.660+0000 7fd5428c8640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd53c159800 con 0x7fd53c0c8eb0 2023-10-30T14:14:16.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.660+0000 7fd5428c8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c159d80 con 0x7fd53c06aeb0 2023-10-30T14:14:16.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.660+0000 7fd5418c6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd53c0c8eb0 0x7fd53c15cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:16.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.660+0000 7fd5410c5640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fd53c15ac70 0x7fd53c15a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:37736/0 (socket says 172.21.15.57:37736) 2023-10-30T14:14:16.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.660+0000 7fd5410c5640 1 -- 172.21.15.57:0/885804571 learned_addr learned my addr 172.21.15.57:0/885804571 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:16.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.660+0000 7fd5420c7640 1 --2- 172.21.15.57:0/885804571 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd53c06aeb0 0x7fd53c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:16.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.660+0000 7fd5418c6640 1 -- 172.21.15.57:0/885804571 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd53c06aeb0 msgr2=0x7fd53c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:16.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.660+0000 7fd5418c6640 1 --2- 172.21.15.57:0/885804571 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd53c06aeb0 0x7fd53c069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.660+0000 7fd5418c6640 1 -- 172.21.15.57:0/885804571 >> v1:172.21.15.57:6789/0 conn(0x7fd53c15ac70 legacy=0x7fd53c15a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:16.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.660+0000 7fd5418c6640 1 -- 172.21.15.57:0/885804571 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53c15d420 con 0x7fd53c0c8eb0 2023-10-30T14:14:16.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.660+0000 7fd5408c4640 1 -- 172.21.15.57:0/885804571 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891042676 0 0) 0x7fd52c001120 con 0x7fd53c15ac70 2023-10-30T14:14:16.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.664+0000 7fd5418c6640 1 --2- 172.21.15.57:0/885804571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd53c0c8eb0 0x7fd53c15cd70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd5380097c0 tx=0x7fd53801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa88a64c58a01e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:16.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.664+0000 7fd5408c4640 1 -- 172.21.15.57:0/885804571 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5380a8970 con 0x7fd53c0c8eb0 2023-10-30T14:14:16.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.664+0000 7fd5408c4640 1 -- 172.21.15.57:0/885804571 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5380a8ad0 con 0x7fd53c0c8eb0 2023-10-30T14:14:16.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.664+0000 7fd5408c4640 1 -- 172.21.15.57:0/885804571 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5380a8e60 con 0x7fd53c0c8eb0 2023-10-30T14:14:16.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.664+0000 7fd5428c8640 1 -- 172.21.15.57:0/885804571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd53c0c8eb0 msgr2=0x7fd53c15cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:16.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.664+0000 7fd5428c8640 1 --2- 172.21.15.57:0/885804571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd53c0c8eb0 0x7fd53c15cd70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd5380097c0 tx=0x7fd53801ec70 comp rx=0 tx=0).stop 2023-10-30T14:14:16.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.664+0000 7fd5428c8640 1 -- 172.21.15.57:0/885804571 shutdown_connections 2023-10-30T14:14:16.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.664+0000 7fd5428c8640 1 --2- 172.21.15.57:0/885804571 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd53c06aeb0 0x7fd53c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.664+0000 7fd5428c8640 1 --2- 172.21.15.57:0/885804571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd53c0c8eb0 0x7fd53c15cd70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.664+0000 7fd5428c8640 1 -- 172.21.15.57:0/885804571 >> 172.21.15.57:0/885804571 conn(0x7fd53c0cef00 msgr2=0x7fd53c0cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:16.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.664+0000 7fd5428c8640 1 -- 172.21.15.57:0/885804571 shutdown_connections 2023-10-30T14:14:16.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.664+0000 7fd5428c8640 1 -- 172.21.15.57:0/885804571 wait complete. 2023-10-30T14:14:16.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.664+0000 7fd5428c8640 1 Processor -- start 2023-10-30T14:14:16.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.664+0000 7fd5428c8640 1 -- start start 2023-10-30T14:14:16.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5428c8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd53c06aeb0 0x7fd53c1578b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:16.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5428c8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd53c0c8eb0 0x7fd53c14f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:16.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5428c8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd53c15ac70 0x7fd53c14fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:16.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5428c8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c1f3390 con 0x7fd53c15ac70 2023-10-30T14:14:16.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5428c8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c156c50 con 0x7fd53c06aeb0 2023-10-30T14:14:16.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5428c8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c156f30 con 0x7fd53c0c8eb0 2023-10-30T14:14:16.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5420c7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd53c15ac70 0x7fd53c14fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:16.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5420c7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd53c15ac70 0x7fd53c14fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39998/0 (socket says 172.21.15.57:39998) 2023-10-30T14:14:16.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5420c7640 1 -- 172.21.15.57:0/2014787098 learned_addr learned my addr 172.21.15.57:0/2014787098 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:16.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5410c5640 1 --2- 172.21.15.57:0/2014787098 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd53c0c8eb0 0x7fd53c14f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:16.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5418c6640 1 --2- 172.21.15.57:0/2014787098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd53c06aeb0 0x7fd53c1578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:16.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5420c7640 1 -- 172.21.15.57:0/2014787098 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd53c0c8eb0 msgr2=0x7fd53c14f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:16.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5420c7640 1 --2- 172.21.15.57:0/2014787098 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd53c0c8eb0 0x7fd53c14f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5420c7640 1 -- 172.21.15.57:0/2014787098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd53c06aeb0 msgr2=0x7fd53c1578b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:16.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5420c7640 1 --2- 172.21.15.57:0/2014787098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd53c06aeb0 0x7fd53c1578b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5420c7640 1 -- 172.21.15.57:0/2014787098 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd538009420 con 0x7fd53c15ac70 2023-10-30T14:14:16.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5420c7640 1 --2- 172.21.15.57:0/2014787098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd53c15ac70 0x7fd53c14fdd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd534007e40 tx=0x7fd53400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:16.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5327fc640 1 -- 172.21.15.57:0/2014787098 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd534014620 con 0x7fd53c15ac70 2023-10-30T14:14:16.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5327fc640 1 -- 172.21.15.57:0/2014787098 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53400ee90 con 0x7fd53c15ac70 2023-10-30T14:14:16.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5428c8640 1 -- 172.21.15.57:0/2014787098 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd53c150a00 con 0x7fd53c15ac70 2023-10-30T14:14:16.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5327fc640 1 -- 172.21.15.57:0/2014787098 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5340149e0 con 0x7fd53c15ac70 2023-10-30T14:14:16.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5428c8640 1 -- 172.21.15.57:0/2014787098 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd53c157df0 con 0x7fd53c15ac70 2023-10-30T14:14:16.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5428c8640 1 -- 172.21.15.57:0/2014787098 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd53c150860 con 0x7fd53c15ac70 2023-10-30T14:14:16.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.668+0000 7fd5327fc640 1 -- 172.21.15.57:0/2014787098 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd534014b40 con 0x7fd53c15ac70 2023-10-30T14:14:16.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.672+0000 7fd5327fc640 1 --2- 172.21.15.57:0/2014787098 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd5200664a0 0x7fd520068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:16.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.672+0000 7fd5327fc640 1 -- 172.21.15.57:0/2014787098 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd534084700 con 0x7fd53c15ac70 2023-10-30T14:14:16.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.672+0000 7fd5418c6640 1 --2- 172.21.15.57:0/2014787098 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd5200664a0 0x7fd520068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:16.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.672+0000 7fd5418c6640 1 --2- 172.21.15.57:0/2014787098 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd5200664a0 0x7fd520068960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd5380b2000 tx=0x7fd538008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:16.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.676+0000 7fd5327fc640 1 -- 172.21.15.57:0/2014787098 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd53c150860 con 0x7fd53c15ac70 2023-10-30T14:14:16.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.836+0000 7fd5428c8640 1 -- 172.21.15.57:0/2014787098 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd53c150860 con 0x7fd53c15ac70 2023-10-30T14:14:16.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.836+0000 7fd5327fc640 1 -- 172.21.15.57:0/2014787098 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16433 (secure 0 0 0) 0x7fd53c150860 con 0x7fd53c15ac70 2023-10-30T14:14:16.841 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:16.841 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:14:05.576818+0000","last_up_change":"2023-10-30T14:13:51.171287+0000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T14:13:52.574639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T14:14:02.600122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T14:14:03.069429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6801","nonce":3287367848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6803","nonce":3287367848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6807","nonce":3287367848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6805","nonce":3287367848}]},"public_addr":"172.21.15.57:6801/3287367848","cluster_addr":"172.21.15.57:6803/3287367848","heartbeat_back_addr":"172.21.15.57:6807/3287367848","heartbeat_front_addr":"172.21.15.57:6805/3287367848","state":["exists","up"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6809","nonce":3340797836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6811","nonce":3340797836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6815","nonce":3340797836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6813","nonce":3340797836}]},"public_addr":"172.21.15.57:6809/3340797836","cluster_addr":"172.21.15.57:6811/3340797836","heartbeat_back_addr":"172.21.15.57:6815/3340797836","heartbeat_front_addr":"172.21.15.57:6813/3340797836","state":["exists","up"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6825","nonce":1700292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6827","nonce":1700292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6831","nonce":1700292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6829","nonce":1700292}]},"public_addr":"172.21.15.57:6825/1700292","cluster_addr":"172.21.15.57:6827/1700292","heartbeat_back_addr":"172.21.15.57:6831/1700292","heartbeat_front_addr":"172.21.15.57:6829/1700292","state":["exists","up"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6817","nonce":2893420631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6819","nonce":2893420631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6823","nonce":2893420631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6821","nonce":2893420631}]},"public_addr":"172.21.15.57:6817/2893420631","cluster_addr":"172.21.15.57:6819/2893420631","heartbeat_back_addr":"172.21.15.57:6823/2893420631","heartbeat_front_addr":"172.21.15.57:6821/2893420631","state":["exists","up"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6807","nonce":2784816485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6811","nonce":2784816485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6815","nonce":2784816485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6813","nonce":2784816485}]},"public_addr":"172.21.15.88:6807/2784816485","cluster_addr":"172.21.15.88:6811/2784816485","heartbeat_back_addr":"172.21.15.88:6815/2784816485","heartbeat_front_addr":"172.21.15.88:6813/2784816485","state":["exists","up"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6801","nonce":2682449812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6803","nonce":2682449812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6810","nonce":2682449812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6806","nonce":2682449812}]},"public_addr":"172.21.15.88:6801/2682449812","cluster_addr":"172.21.15.88:6803/2682449812","heartbeat_back_addr":"172.21.15.88:6810/2682449812","heartbeat_front_addr":"172.21.15.88:6806/2682449812","state":["exists","up"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6822","nonce":3340693897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6827","nonce":3340693897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6831","nonce":3340693897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6829","nonce":3340693897}]},"public_addr":"172.21.15.88:6822/3340693897","cluster_addr":"172.21.15.88:6827/3340693897","heartbeat_back_addr":"172.21.15.88:6831/3340693897","heartbeat_front_addr":"172.21.15.88:6829/3340693897","state":["exists","up"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6817","nonce":2862111199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6820","nonce":2862111199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6825","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6826","nonce":2862111199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6821","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6823","nonce":2862111199}]},"public_addr":"172.21.15.88:6817/2862111199","cluster_addr":"172.21.15.88:6820/2862111199","heartbeat_back_addr":"172.21.15.88:6826/2862111199","heartbeat_front_addr":"172.21.15.88:6823/2862111199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.634920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.773993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.725825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.761452+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.694274+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.650718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:14:16.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.836+0000 7fd5428c8640 1 -- 172.21.15.57:0/2014787098 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd5200664a0 msgr2=0x7fd520068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:16.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.836+0000 7fd5428c8640 1 --2- 172.21.15.57:0/2014787098 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd5200664a0 0x7fd520068960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd5380b2000 tx=0x7fd538008040 comp rx=0 tx=0).stop 2023-10-30T14:14:16.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.836+0000 7fd5428c8640 1 -- 172.21.15.57:0/2014787098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd53c15ac70 msgr2=0x7fd53c14fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:16.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.836+0000 7fd5428c8640 1 --2- 172.21.15.57:0/2014787098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd53c15ac70 0x7fd53c14fdd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd534007e40 tx=0x7fd53400e4b0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.836+0000 7fd5428c8640 1 -- 172.21.15.57:0/2014787098 shutdown_connections 2023-10-30T14:14:16.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.836+0000 7fd5428c8640 1 --2- 172.21.15.57:0/2014787098 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd5200664a0 0x7fd520068960 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.836+0000 7fd5428c8640 1 --2- 172.21.15.57:0/2014787098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd53c15ac70 0x7fd53c14fdd0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.836+0000 7fd5428c8640 1 --2- 172.21.15.57:0/2014787098 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd53c0c8eb0 0x7fd53c14f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.836+0000 7fd5428c8640 1 --2- 172.21.15.57:0/2014787098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd53c06aeb0 0x7fd53c1578b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:16.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.836+0000 7fd5428c8640 1 -- 172.21.15.57:0/2014787098 >> 172.21.15.57:0/2014787098 conn(0x7fd53c0cef00 msgr2=0x7fd53c0cadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:16.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.836+0000 7fd5428c8640 1 -- 172.21.15.57:0/2014787098 shutdown_connections 2023-10-30T14:14:16.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:16.836+0000 7fd5428c8640 1 -- 172.21.15.57:0/2014787098 wait complete. 2023-10-30T14:14:16.863 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T14:13:52.574639+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T14:14:02.600122+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T14:14:03.069429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T14:14:16.863 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T14:14:17.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.004+0000 7f08111c5640 1 Processor -- start 2023-10-30T14:14:17.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.004+0000 7f08111c5640 1 -- start start 2023-10-30T14:14:17.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.004+0000 7f08111c5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f080c0c61c0 0x7f080c0c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.004+0000 7f08111c5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f080c0706f0 0x7f080c0c8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.004+0000 7f08111c5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f080c0c65a0 0x7f080c153770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.004+0000 7f08111c5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f080c159a70 con 0x7f080c0c65a0 2023-10-30T14:14:17.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.004+0000 7f08111c5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f080c06e210 con 0x7f080c0706f0 2023-10-30T14:14:17.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.004+0000 7f08111c5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080c06e4f0 con 0x7f080c0c61c0 2023-10-30T14:14:17.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.004+0000 7f080b59e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f080c0c65a0 0x7f080c153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.004+0000 7f080b59e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f080c0c65a0 0x7f080c153770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40012/0 (socket says 172.21.15.57:40012) 2023-10-30T14:14:17.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.004+0000 7f080b59e640 1 -- 172.21.15.57:0/2074431646 learned_addr learned my addr 172.21.15.57:0/2074431646 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:17.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.004+0000 7f080ad9d640 1 --2- 172.21.15.57:0/2074431646 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f080c0c61c0 0x7f080c0c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.004+0000 7f0803fff640 1 --2- 172.21.15.57:0/2074431646 >> v2:172.21.15.88:3300/0 conn(0x7f080c0706f0 0x7f080c0c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.004+0000 7f080b59e640 1 -- 172.21.15.57:0/2074431646 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f080c0c61c0 msgr2=0x7f080c0c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:17.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.004+0000 7f080b59e640 1 --2- 172.21.15.57:0/2074431646 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f080c0c61c0 0x7f080c0c7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.004+0000 7f080b59e640 1 -- 172.21.15.57:0/2074431646 >> v2:172.21.15.88:3300/0 conn(0x7f080c0706f0 msgr2=0x7f080c0c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:17.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.004+0000 7f080b59e640 1 --2- 172.21.15.57:0/2074431646 >> v2:172.21.15.88:3300/0 conn(0x7f080c0706f0 0x7f080c0c8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.004+0000 7f080b59e640 1 -- 172.21.15.57:0/2074431646 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c06e7d0 con 0x7f080c0c65a0 2023-10-30T14:14:17.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.008+0000 7f080b59e640 1 --2- 172.21.15.57:0/2074431646 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080c0c65a0 0x7f080c153770 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f080c06ae10 tx=0x7f080401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37533c1d96107f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:17.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.008+0000 7f080a59c640 1 -- 172.21.15.57:0/2074431646 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08040af400 con 0x7f080c0c65a0 2023-10-30T14:14:17.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.008+0000 7f080a59c640 1 -- 172.21.15.57:0/2074431646 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08040ab050 con 0x7f080c0c65a0 2023-10-30T14:14:17.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.008+0000 7f080a59c640 1 -- 172.21.15.57:0/2074431646 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08040af780 con 0x7f080c0c65a0 2023-10-30T14:14:17.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.008+0000 7f08111c5640 1 -- 172.21.15.57:0/2074431646 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080c0c65a0 msgr2=0x7f080c153770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:17.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.008+0000 7f08111c5640 1 --2- 172.21.15.57:0/2074431646 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080c0c65a0 0x7f080c153770 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f080c06ae10 tx=0x7f080401f1c0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.008+0000 7f08111c5640 1 -- 172.21.15.57:0/2074431646 shutdown_connections 2023-10-30T14:14:17.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.008+0000 7f08111c5640 1 --2- 172.21.15.57:0/2074431646 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080c0c65a0 0x7f080c153770 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.008+0000 7f08111c5640 1 --2- 172.21.15.57:0/2074431646 >> v2:172.21.15.88:3300/0 conn(0x7f080c0706f0 0x7f080c0c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.008+0000 7f08111c5640 1 --2- 172.21.15.57:0/2074431646 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f080c0c61c0 0x7f080c0c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.008+0000 7f08111c5640 1 -- 172.21.15.57:0/2074431646 >> 172.21.15.57:0/2074431646 conn(0x7f080c069e90 msgr2=0x7f080c06a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:17.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.008+0000 7f08111c5640 1 -- 172.21.15.57:0/2074431646 shutdown_connections 2023-10-30T14:14:17.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.008+0000 7f08111c5640 1 -- 172.21.15.57:0/2074431646 wait complete. 2023-10-30T14:14:17.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f08111c5640 1 Processor -- start 2023-10-30T14:14:17.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f08111c5640 1 -- start start 2023-10-30T14:14:17.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f08111c5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080c0706f0 0x7f080c0c3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f08111c5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f080c0c61c0 0x7f080c1542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f08111c5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f080c0c65a0 0x7f080c1589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f08111c5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c148dc0 con 0x7f080c0706f0 2023-10-30T14:14:17.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f08111c5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c159680 con 0x7f080c0c61c0 2023-10-30T14:14:17.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f08111c5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080c14c780 con 0x7f080c0c65a0 2023-10-30T14:14:17.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f080ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080c0706f0 0x7f080c0c3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f080ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080c0706f0 0x7f080c0c3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40026/0 (socket says 172.21.15.57:40026) 2023-10-30T14:14:17.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f080ad9d640 1 -- 172.21.15.57:0/3728444937 learned_addr learned my addr 172.21.15.57:0/3728444937 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:17.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f0803fff640 1 --2- 172.21.15.57:0/3728444937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f080c0c61c0 0x7f080c1542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f080b59e640 1 --2- 172.21.15.57:0/3728444937 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f080c0c65a0 0x7f080c1589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f080ad9d640 1 -- 172.21.15.57:0/3728444937 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f080c0c65a0 msgr2=0x7f080c1589f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:17.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f080ad9d640 1 --2- 172.21.15.57:0/3728444937 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f080c0c65a0 0x7f080c1589f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f080ad9d640 1 -- 172.21.15.57:0/3728444937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f080c0c61c0 msgr2=0x7f080c1542c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:17.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f080ad9d640 1 --2- 172.21.15.57:0/3728444937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f080c0c61c0 0x7f080c1542c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f080ad9d640 1 -- 172.21.15.57:0/3728444937 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0804009b30 con 0x7f080c0706f0 2023-10-30T14:14:17.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f080ad9d640 1 --2- 172.21.15.57:0/3728444937 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080c0706f0 0x7f080c0c3130 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f07fc00d830 tx=0x7f07fc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:17.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f08037fe640 1 -- 172.21.15.57:0/3728444937 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc00b6c0 con 0x7f080c0706f0 2023-10-30T14:14:17.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f08037fe640 1 -- 172.21.15.57:0/3728444937 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07fc00b820 con 0x7f080c0706f0 2023-10-30T14:14:17.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f08111c5640 1 -- 172.21.15.57:0/3728444937 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080c0c52a0 con 0x7f080c0706f0 2023-10-30T14:14:17.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f08037fe640 1 -- 172.21.15.57:0/3728444937 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc00bb90 con 0x7f080c0706f0 2023-10-30T14:14:17.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.012+0000 7f08111c5640 1 -- 172.21.15.57:0/3728444937 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080c14ca00 con 0x7f080c0706f0 2023-10-30T14:14:17.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.016+0000 7f08111c5640 1 -- 172.21.15.57:0/3728444937 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f080c04e170 con 0x7f080c0706f0 2023-10-30T14:14:17.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.016+0000 7f08037fe640 1 -- 172.21.15.57:0/3728444937 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f07fc01e460 con 0x7f080c0706f0 2023-10-30T14:14:17.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.016+0000 7f08037fe640 1 --2- 172.21.15.57:0/3728444937 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07ec0664d0 0x7f07ec068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.016+0000 7f08037fe640 1 -- 172.21.15.57:0/3728444937 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f07fc084500 con 0x7f080c0706f0 2023-10-30T14:14:17.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.016+0000 7f0803fff640 1 --2- 172.21.15.57:0/3728444937 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07ec0664d0 0x7f07ec068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.016+0000 7f0803fff640 1 --2- 172.21.15.57:0/3728444937 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07ec0664d0 0x7f07ec068990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f080c14fed0 tx=0x7f07f4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:17.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.020+0000 7f08037fe640 1 -- 172.21.15.57:0/3728444937 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f07fc051a20 con 0x7f080c0706f0 2023-10-30T14:14:17.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.176+0000 7f08111c5640 1 -- 172.21.15.57:0/3728444937 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f080c0c6a30 con 0x7f080c0706f0 2023-10-30T14:14:17.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.176+0000 7f08037fe640 1 -- 172.21.15.57:0/3728444937 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f07fc055040 con 0x7f080c0706f0 2023-10-30T14:14:17.183 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-30T14:14:17.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.180+0000 7f08111c5640 1 -- 172.21.15.57:0/3728444937 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07ec0664d0 msgr2=0x7f07ec068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:17.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.180+0000 7f08111c5640 1 --2- 172.21.15.57:0/3728444937 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07ec0664d0 0x7f07ec068990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f080c14fed0 tx=0x7f07f4008040 comp rx=0 tx=0).stop 2023-10-30T14:14:17.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.180+0000 7f08111c5640 1 -- 172.21.15.57:0/3728444937 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080c0706f0 msgr2=0x7f080c0c3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:17.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.180+0000 7f08111c5640 1 --2- 172.21.15.57:0/3728444937 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080c0706f0 0x7f080c0c3130 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f07fc00d830 tx=0x7f07fc00dcf0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.180+0000 7f08111c5640 1 -- 172.21.15.57:0/3728444937 shutdown_connections 2023-10-30T14:14:17.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.180+0000 7f08111c5640 1 --2- 172.21.15.57:0/3728444937 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07ec0664d0 0x7f07ec068990 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.180+0000 7f08111c5640 1 --2- 172.21.15.57:0/3728444937 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f080c0c65a0 0x7f080c1589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.180+0000 7f08111c5640 1 --2- 172.21.15.57:0/3728444937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f080c0c61c0 0x7f080c1542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.180+0000 7f08111c5640 1 --2- 172.21.15.57:0/3728444937 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080c0706f0 0x7f080c0c3130 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.180+0000 7f08111c5640 1 -- 172.21.15.57:0/3728444937 >> 172.21.15.57:0/3728444937 conn(0x7f080c069e90 msgr2=0x7f080c151ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:17.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.180+0000 7f08111c5640 1 -- 172.21.15.57:0/3728444937 shutdown_connections 2023-10-30T14:14:17.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.180+0000 7f08111c5640 1 -- 172.21.15.57:0/3728444937 wait complete. 2023-10-30T14:14:17.205 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T14:14:17.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.344+0000 7f53804f7640 1 Processor -- start 2023-10-30T14:14:17.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.344+0000 7f53804f7640 1 -- start start 2023-10-30T14:14:17.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f53804f7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53780c4410 0x7f53780cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f53804f7640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5378153f80 con 0x7f5378153ba0 2023-10-30T14:14:17.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f53804f7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5378159df0 con 0x7f53780c4810 2023-10-30T14:14:17.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f53804f7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53781597c0 con 0x7f53780c4410 2023-10-30T14:14:17.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537e293640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f5378153ba0 0x7f537815cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:37748/0 (socket says 172.21.15.57:37748) 2023-10-30T14:14:17.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537e293640 1 -- 172.21.15.57:0/286503612 learned_addr learned my addr 172.21.15.57:0/286503612 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:17.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537da92640 1 --2- 172.21.15.57:0/286503612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53780c4410 0x7f53780cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537d291640 1 -- 172.21.15.57:0/286503612 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811421928 0 0) 0x7f5378153f80 con 0x7f5378153ba0 2023-10-30T14:14:17.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537d291640 1 -- 172.21.15.57:0/286503612 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5364003530 con 0x7f5378153ba0 2023-10-30T14:14:17.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537d291640 1 -- 172.21.15.57:0/286503612 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3036069869 0 0) 0x7f5364003530 con 0x7f5378153ba0 2023-10-30T14:14:17.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537d291640 1 -- 172.21.15.57:0/286503612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53780c4410 msgr2=0x7f53780cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:17.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537d291640 1 --2- 172.21.15.57:0/286503612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53780c4410 0x7f53780cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537d291640 1 -- 172.21.15.57:0/286503612 >> v1:172.21.15.88:6789/0 conn(0x7f53780c4810 legacy=0x7f53780cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:17.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537d291640 1 -- 172.21.15.57:0/286503612 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537815d580 con 0x7f5378153ba0 2023-10-30T14:14:17.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537d291640 1 -- 172.21.15.57:0/286503612 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f5374001680 con 0x7f5378153ba0 2023-10-30T14:14:17.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537d291640 1 -- 172.21.15.57:0/286503612 >> v1:172.21.15.57:6789/0 conn(0x7f5378153ba0 legacy=0x7f537815cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:17.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537d291640 1 --2- 172.21.15.57:0/286503612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53640039b0 0x7f5364003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537d291640 1 --2- 172.21.15.57:0/286503612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53640047d0 0x7f5364004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537d291640 1 --2- 172.21.15.57:0/286503612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5364005190 0x7f536400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537d291640 1 -- 172.21.15.57:0/286503612 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53781597c0 con 0x7f53640039b0 2023-10-30T14:14:17.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537d291640 1 -- 172.21.15.57:0/286503612 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536400c030 con 0x7f53640047d0 2023-10-30T14:14:17.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537d291640 1 -- 172.21.15.57:0/286503612 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f536400c1f0 con 0x7f5364005190 2023-10-30T14:14:17.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537da92640 1 --2- 172.21.15.57:0/286503612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53780c4410 0x7f53780cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:14:17.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537e293640 1 --2- 172.21.15.57:0/286503612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53640039b0 0x7f5364003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537ea94640 1 --2- 172.21.15.57:0/286503612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5364005190 0x7f536400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537da92640 1 --2- 172.21.15.57:0/286503612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53640047d0 0x7f5364004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537e293640 1 -- 172.21.15.57:0/286503612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5364005190 msgr2=0x7f536400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:17.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537e293640 1 --2- 172.21.15.57:0/286503612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5364005190 0x7f536400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537e293640 1 -- 172.21.15.57:0/286503612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53640047d0 msgr2=0x7f5364004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:17.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537e293640 1 --2- 172.21.15.57:0/286503612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53640047d0 0x7f5364004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537e293640 1 -- 172.21.15.57:0/286503612 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5364003530 con 0x7f53640039b0 2023-10-30T14:14:17.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.348+0000 7f537da92640 1 --2- 172.21.15.57:0/286503612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53640047d0 0x7f5364004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:14:17.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.352+0000 7f537e293640 1 --2- 172.21.15.57:0/286503612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53640039b0 0x7f5364003dd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f537400c240 tx=0x7f537401cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f86a61184013e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:17.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.352+0000 7f537d291640 1 -- 172.21.15.57:0/286503612 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5374005ae0 con 0x7f53640039b0 2023-10-30T14:14:17.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.352+0000 7f537d291640 1 -- 172.21.15.57:0/286503612 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5374005c40 con 0x7f53640039b0 2023-10-30T14:14:17.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.352+0000 7f537d291640 1 -- 172.21.15.57:0/286503612 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5374007070 con 0x7f53640039b0 2023-10-30T14:14:17.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.352+0000 7f53804f7640 1 -- 172.21.15.57:0/286503612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53640039b0 msgr2=0x7f5364003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:17.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.352+0000 7f53804f7640 1 --2- 172.21.15.57:0/286503612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53640039b0 0x7f5364003dd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f537400c240 tx=0x7f537401cb50 comp rx=0 tx=0).stop 2023-10-30T14:14:17.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.352+0000 7f537ea94640 1 -- 172.21.15.57:0/286503612 reap_dead start 2023-10-30T14:14:17.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.352+0000 7f53804f7640 1 -- 172.21.15.57:0/286503612 shutdown_connections 2023-10-30T14:14:17.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.352+0000 7f53804f7640 1 -- 172.21.15.57:0/286503612 >> 172.21.15.57:0/286503612 conn(0x7f5378071790 msgr2=0x7f5378071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:17.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.352+0000 7f53804f7640 1 -- 172.21.15.57:0/286503612 shutdown_connections 2023-10-30T14:14:17.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.352+0000 7f53804f7640 1 -- 172.21.15.57:0/286503612 wait complete. 2023-10-30T14:14:17.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f53804f7640 1 Processor -- start 2023-10-30T14:14:17.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f53804f7640 1 -- start start 2023-10-30T14:14:17.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f53804f7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53640039b0 0x7f53780b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f53804f7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f537814e9d0 0x7f5378149040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f53804f7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f537814f920 0x7f537815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f53804f7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537814e5b0 con 0x7f537814e9d0 2023-10-30T14:14:17.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f53804f7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53780b38a0 con 0x7f53640039b0 2023-10-30T14:14:17.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f53804f7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5378151880 con 0x7f537814f920 2023-10-30T14:14:17.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f537da92640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f537814e9d0 0x7f5378149040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f537da92640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f537814e9d0 0x7f5378149040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40042/0 (socket says 172.21.15.57:40042) 2023-10-30T14:14:17.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f537da92640 1 -- 172.21.15.57:0/3315908343 learned_addr learned my addr 172.21.15.57:0/3315908343 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:17.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f537ea94640 1 --2- 172.21.15.57:0/3315908343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f537814f920 0x7f537815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f537e293640 1 --2- 172.21.15.57:0/3315908343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53640039b0 0x7f53780b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f537da92640 1 -- 172.21.15.57:0/3315908343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f537814f920 msgr2=0x7f537815bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:17.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f537da92640 1 --2- 172.21.15.57:0/3315908343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f537814f920 0x7f537815bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f537da92640 1 -- 172.21.15.57:0/3315908343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53640039b0 msgr2=0x7f53780b3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:17.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f537da92640 1 --2- 172.21.15.57:0/3315908343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53640039b0 0x7f53780b3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f537da92640 1 -- 172.21.15.57:0/3315908343 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537815a7f0 con 0x7f537814e9d0 2023-10-30T14:14:17.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f537da92640 1 --2- 172.21.15.57:0/3315908343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f537814e9d0 0x7f5378149040 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f53680108f0 tx=0x7f5368010db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:17.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f536effd640 1 -- 172.21.15.57:0/3315908343 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536800eb00 con 0x7f537814e9d0 2023-10-30T14:14:17.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f536effd640 1 -- 172.21.15.57:0/3315908343 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536800ec60 con 0x7f537814e9d0 2023-10-30T14:14:17.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f53804f7640 1 -- 172.21.15.57:0/3315908343 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5378151630 con 0x7f537814e9d0 2023-10-30T14:14:17.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f536effd640 1 -- 172.21.15.57:0/3315908343 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536801c400 con 0x7f537814e9d0 2023-10-30T14:14:17.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f53804f7640 1 -- 172.21.15.57:0/3315908343 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f537814def0 con 0x7f537814e9d0 2023-10-30T14:14:17.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.356+0000 7f53804f7640 1 -- 172.21.15.57:0/3315908343 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f537814dd50 con 0x7f537814e9d0 2023-10-30T14:14:17.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.360+0000 7f536effd640 1 -- 172.21.15.57:0/3315908343 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f536801c560 con 0x7f537814e9d0 2023-10-30T14:14:17.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.360+0000 7f536effd640 1 --2- 172.21.15.57:0/3315908343 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f535c0664a0 0x7f535c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.360+0000 7f536effd640 1 -- 172.21.15.57:0/3315908343 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5368084380 con 0x7f537814e9d0 2023-10-30T14:14:17.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.360+0000 7f537e293640 1 --2- 172.21.15.57:0/3315908343 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f535c0664a0 0x7f535c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.360+0000 7f537e293640 1 --2- 172.21.15.57:0/3315908343 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f535c0664a0 0x7f535c068960 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f53740ab000 tx=0x7f53740a7070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:17.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.364+0000 7f536effd640 1 -- 172.21.15.57:0/3315908343 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f537814dd50 con 0x7f537814e9d0 2023-10-30T14:14:17.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.520+0000 7f53804f7640 1 -- 172.21.15.57:0/3315908343 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f537814dd50 con 0x7f537814e9d0 2023-10-30T14:14:17.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.520+0000 7f536effd640 1 -- 172.21.15.57:0/3315908343 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f537814dd50 con 0x7f537814e9d0 2023-10-30T14:14:17.525 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-30T14:14:17.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.520+0000 7f53804f7640 1 -- 172.21.15.57:0/3315908343 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f535c0664a0 msgr2=0x7f535c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:17.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.520+0000 7f53804f7640 1 --2- 172.21.15.57:0/3315908343 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f535c0664a0 0x7f535c068960 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f53740ab000 tx=0x7f53740a7070 comp rx=0 tx=0).stop 2023-10-30T14:14:17.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.520+0000 7f53804f7640 1 -- 172.21.15.57:0/3315908343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f537814e9d0 msgr2=0x7f5378149040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:17.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.520+0000 7f53804f7640 1 --2- 172.21.15.57:0/3315908343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f537814e9d0 0x7f5378149040 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f53680108f0 tx=0x7f5368010db0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.520+0000 7f53804f7640 1 -- 172.21.15.57:0/3315908343 shutdown_connections 2023-10-30T14:14:17.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.520+0000 7f53804f7640 1 --2- 172.21.15.57:0/3315908343 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f535c0664a0 0x7f535c068960 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.520+0000 7f53804f7640 1 --2- 172.21.15.57:0/3315908343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f537814f920 0x7f537815bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.520+0000 7f53804f7640 1 --2- 172.21.15.57:0/3315908343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f537814e9d0 0x7f5378149040 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.520+0000 7f53804f7640 1 --2- 172.21.15.57:0/3315908343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53640039b0 0x7f53780b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.520+0000 7f53804f7640 1 -- 172.21.15.57:0/3315908343 >> 172.21.15.57:0/3315908343 conn(0x7f5378071790 msgr2=0x7f5378158250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:17.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.520+0000 7f53804f7640 1 -- 172.21.15.57:0/3315908343 shutdown_connections 2023-10-30T14:14:17.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.520+0000 7f53804f7640 1 -- 172.21.15.57:0/3315908343 wait complete. 2023-10-30T14:14:17.545 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T14:14:17.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.684+0000 7f746a09e640 1 Processor -- start 2023-10-30T14:14:17.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.684+0000 7f746a09e640 1 -- start start 2023-10-30T14:14:17.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.688+0000 7f746a09e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f74641500e0 0x7f74640b0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.688+0000 7f746a09e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74641504c0 con 0x7f7464156a60 2023-10-30T14:14:17.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.688+0000 7f746a09e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74640c5e90 con 0x7f7464072520 2023-10-30T14:14:17.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.688+0000 7f746a09e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74640c5ca0 con 0x7f74641500e0 2023-10-30T14:14:17.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.688+0000 7f74637fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f74641500e0 0x7f74640b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.688+0000 7f7463fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f7464156a60 0x7f74641605e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:37758/0 (socket says 172.21.15.57:37758) 2023-10-30T14:14:17.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.688+0000 7f7463fff640 1 -- 172.21.15.57:0/743045574 learned_addr learned my addr 172.21.15.57:0/743045574 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:17.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.688+0000 7f74637fe640 1 -- 172.21.15.57:0/743045574 >> v1:172.21.15.88:6789/0 conn(0x7f7464072520 legacy=0x7f7464156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:17.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.688+0000 7f74637fe640 1 -- 172.21.15.57:0/743045574 >> v1:172.21.15.57:6789/0 conn(0x7f7464156a60 legacy=0x7f74641605e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:17.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.688+0000 7f74637fe640 1 -- 172.21.15.57:0/743045574 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74640c6170 con 0x7f74641500e0 2023-10-30T14:14:17.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.688+0000 7f74627fc640 1 -- 172.21.15.57:0/743045574 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698223412 0 0) 0x7f74641504c0 con 0x7f7464156a60 2023-10-30T14:14:17.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.692+0000 7f74637fe640 1 --2- 172.21.15.57:0/743045574 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74641500e0 0x7f74640b0710 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f74540097c0 tx=0x7f745401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7788a51579fb7ba server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:17.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.692+0000 7f74627fc640 1 -- 172.21.15.57:0/743045574 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74540a8970 con 0x7f74641500e0 2023-10-30T14:14:17.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.692+0000 7f74627fc640 1 -- 172.21.15.57:0/743045574 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74540a8ad0 con 0x7f74641500e0 2023-10-30T14:14:17.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.692+0000 7f74627fc640 1 -- 172.21.15.57:0/743045574 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74540a8e60 con 0x7f74641500e0 2023-10-30T14:14:17.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.692+0000 7f746a09e640 1 -- 172.21.15.57:0/743045574 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74641500e0 msgr2=0x7f74640b0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:17.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.692+0000 7f746a09e640 1 --2- 172.21.15.57:0/743045574 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74641500e0 0x7f74640b0710 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f74540097c0 tx=0x7f745401ec70 comp rx=0 tx=0).stop 2023-10-30T14:14:17.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.692+0000 7f746a09e640 1 -- 172.21.15.57:0/743045574 shutdown_connections 2023-10-30T14:14:17.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.692+0000 7f746a09e640 1 --2- 172.21.15.57:0/743045574 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74641500e0 0x7f74640b0710 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.692+0000 7f746a09e640 1 -- 172.21.15.57:0/743045574 >> 172.21.15.57:0/743045574 conn(0x7f746406da40 msgr2=0x7f746406de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:17.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.692+0000 7f746a09e640 1 -- 172.21.15.57:0/743045574 shutdown_connections 2023-10-30T14:14:17.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.692+0000 7f746a09e640 1 -- 172.21.15.57:0/743045574 wait complete. 2023-10-30T14:14:17.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.692+0000 7f746a09e640 1 Processor -- start 2023-10-30T14:14:17.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.692+0000 7f746a09e640 1 -- start start 2023-10-30T14:14:17.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f746a09e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7464072520 0x7f74641ed740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f746a09e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f74641500e0 0x7f746414e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f746a09e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7464156a60 0x7f746414bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f746a09e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746414d330 con 0x7f7464156a60 2023-10-30T14:14:17.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f746a09e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74641edc80 con 0x7f7464072520 2023-10-30T14:14:17.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f746a09e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74640c3be0 con 0x7f74641500e0 2023-10-30T14:14:17.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f7463fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7464156a60 0x7f746414bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f7463fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7464156a60 0x7f746414bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40062/0 (socket says 172.21.15.57:40062) 2023-10-30T14:14:17.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f7463fff640 1 -- 172.21.15.57:0/1729466222 learned_addr learned my addr 172.21.15.57:0/1729466222 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:17.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f74637fe640 1 --2- 172.21.15.57:0/1729466222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7464072520 0x7f74641ed740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f7462ffd640 1 --2- 172.21.15.57:0/1729466222 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f74641500e0 0x7f746414e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f7463fff640 1 -- 172.21.15.57:0/1729466222 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f74641500e0 msgr2=0x7f746414e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:17.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f7463fff640 1 --2- 172.21.15.57:0/1729466222 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f74641500e0 0x7f746414e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f7463fff640 1 -- 172.21.15.57:0/1729466222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7464072520 msgr2=0x7f74641ed740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:17.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f7463fff640 1 --2- 172.21.15.57:0/1729466222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7464072520 0x7f74641ed740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f7463fff640 1 -- 172.21.15.57:0/1729466222 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7454009420 con 0x7f7464156a60 2023-10-30T14:14:17.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f7463fff640 1 --2- 172.21.15.57:0/1729466222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7464156a60 0x7f746414bef0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f745c00c7b0 tx=0x7f745c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:17.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f744bfff640 1 -- 172.21.15.57:0/1729466222 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745c00ce60 con 0x7f7464156a60 2023-10-30T14:14:17.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f744bfff640 1 -- 172.21.15.57:0/1729466222 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f745c00a5e0 con 0x7f7464156a60 2023-10-30T14:14:17.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f746a09e640 1 -- 172.21.15.57:0/1729466222 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74640c53b0 con 0x7f7464156a60 2023-10-30T14:14:17.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f744bfff640 1 -- 172.21.15.57:0/1729466222 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745c00a9b0 con 0x7f7464156a60 2023-10-30T14:14:17.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f746a09e640 1 -- 172.21.15.57:0/1729466222 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74640c32b0 con 0x7f7464156a60 2023-10-30T14:14:17.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f746a09e640 1 -- 172.21.15.57:0/1729466222 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f746404e170 con 0x7f7464156a60 2023-10-30T14:14:17.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.696+0000 7f744bfff640 1 -- 172.21.15.57:0/1729466222 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f745c00ab10 con 0x7f7464156a60 2023-10-30T14:14:17.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.700+0000 7f744bfff640 1 --2- 172.21.15.57:0/1729466222 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f74500664d0 0x7f7450068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:17.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.700+0000 7f744bfff640 1 -- 172.21.15.57:0/1729466222 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f745c083570 con 0x7f7464156a60 2023-10-30T14:14:17.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.700+0000 7f74637fe640 1 --2- 172.21.15.57:0/1729466222 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f74500664d0 0x7f7450068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:17.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.700+0000 7f74637fe640 1 --2- 172.21.15.57:0/1729466222 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f74500664d0 0x7f7450068990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f74540b2000 tx=0x7f7454008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:17.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.704+0000 7f744bfff640 1 -- 172.21.15.57:0/1729466222 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f745c0509f0 con 0x7f7464156a60 2023-10-30T14:14:17.864 INFO:tasks.ceph.mds.f.smithi088.stderr:2023-10-30T14:14:17.858+0000 7f1418be9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T14:14:17.865 INFO:tasks.ceph.mds.f.smithi088.stderr:2023-10-30T14:14:17.858+0000 7f1418be9640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:14:17.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.860+0000 7f746a09e640 1 -- 172.21.15.57:0/1729466222 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f74640c3d20 con 0x7f7464156a60 2023-10-30T14:14:17.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.860+0000 7f744bfff640 1 -- 172.21.15.57:0/1729466222 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f745c054010 con 0x7f7464156a60 2023-10-30T14:14:17.866 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-30T14:14:17.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.864+0000 7f746a09e640 1 -- 172.21.15.57:0/1729466222 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f74500664d0 msgr2=0x7f7450068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:17.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.864+0000 7f746a09e640 1 --2- 172.21.15.57:0/1729466222 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f74500664d0 0x7f7450068990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f74540b2000 tx=0x7f7454008040 comp rx=0 tx=0).stop 2023-10-30T14:14:17.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.864+0000 7f746a09e640 1 -- 172.21.15.57:0/1729466222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7464156a60 msgr2=0x7f746414bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:17.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.864+0000 7f746a09e640 1 --2- 172.21.15.57:0/1729466222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7464156a60 0x7f746414bef0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f745c00c7b0 tx=0x7f745c00cc70 comp rx=0 tx=0).stop 2023-10-30T14:14:17.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.864+0000 7f746a09e640 1 -- 172.21.15.57:0/1729466222 shutdown_connections 2023-10-30T14:14:17.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.864+0000 7f746a09e640 1 --2- 172.21.15.57:0/1729466222 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f74500664d0 0x7f7450068990 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.864+0000 7f746a09e640 1 --2- 172.21.15.57:0/1729466222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7464156a60 0x7f746414bef0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.864+0000 7f746a09e640 1 --2- 172.21.15.57:0/1729466222 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f74641500e0 0x7f746414e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.864+0000 7f746a09e640 1 --2- 172.21.15.57:0/1729466222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7464072520 0x7f74641ed740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:17.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.864+0000 7f746a09e640 1 -- 172.21.15.57:0/1729466222 >> 172.21.15.57:0/1729466222 conn(0x7f746406da40 msgr2=0x7f7464156350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:17.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.864+0000 7f746a09e640 1 -- 172.21.15.57:0/1729466222 shutdown_connections 2023-10-30T14:14:17.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:17.864+0000 7f746a09e640 1 -- 172.21.15.57:0/1729466222 wait complete. 2023-10-30T14:14:17.887 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:18.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.024+0000 7f37b6e74640 1 Processor -- start 2023-10-30T14:14:18.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.024+0000 7f37b6e74640 1 -- start start 2023-10-30T14:14:18.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.024+0000 7f37b6e74640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37b00c8eb0 0x7f37b015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:18.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.024+0000 7f37b6e74640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f37b015ac70 0x7f37b015a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:18.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.028+0000 7f37b6e74640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b015dcc0 con 0x7f37b006aeb0 2023-10-30T14:14:18.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.028+0000 7f37b6e74640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37b0159800 con 0x7f37b015ac70 2023-10-30T14:14:18.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.028+0000 7f37b6e74640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b0159d80 con 0x7f37b00c8eb0 2023-10-30T14:14:18.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.028+0000 7f37b5671640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f37b015ac70 0x7f37b015a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:18.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.028+0000 7f37b5671640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f37b015ac70 0x7f37b015a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40064/0 (socket says 172.21.15.57:40064) 2023-10-30T14:14:18.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.028+0000 7f37b5671640 1 -- 172.21.15.57:0/2893551017 learned_addr learned my addr 172.21.15.57:0/2893551017 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:18.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.028+0000 7f37b5e72640 1 --2- 172.21.15.57:0/2893551017 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37b00c8eb0 0x7f37b015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:18.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.028+0000 7f37b5671640 1 -- 172.21.15.57:0/2893551017 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37b00c8eb0 msgr2=0x7f37b015cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:18.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.028+0000 7f37b5671640 1 --2- 172.21.15.57:0/2893551017 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37b00c8eb0 0x7f37b015cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.028+0000 7f37b5671640 1 -- 172.21.15.57:0/2893551017 >> v1:172.21.15.88:6789/0 conn(0x7f37b006aeb0 legacy=0x7f37b0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:18.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.028+0000 7f37b5671640 1 -- 172.21.15.57:0/2893551017 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b015a750 con 0x7f37b015ac70 2023-10-30T14:14:18.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.028+0000 7f37b5671640 1 --2- 172.21.15.57:0/2893551017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b015ac70 0x7f37b015a210 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f37a0009870 tx=0x7f37a001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e8d44f21f2073c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:18.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.028+0000 7f37b4e70640 1 -- 172.21.15.57:0/2893551017 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a00a8a30 con 0x7f37b015ac70 2023-10-30T14:14:18.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.028+0000 7f37b4e70640 1 -- 172.21.15.57:0/2893551017 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a00a8b90 con 0x7f37b015ac70 2023-10-30T14:14:18.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.028+0000 7f37b4e70640 1 -- 172.21.15.57:0/2893551017 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a00b0440 con 0x7f37b015ac70 2023-10-30T14:14:18.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b6e74640 1 -- 172.21.15.57:0/2893551017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b015ac70 msgr2=0x7f37b015a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:18.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b6e74640 1 --2- 172.21.15.57:0/2893551017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b015ac70 0x7f37b015a210 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f37a0009870 tx=0x7f37a001ed30 comp rx=0 tx=0).stop 2023-10-30T14:14:18.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b6e74640 1 -- 172.21.15.57:0/2893551017 shutdown_connections 2023-10-30T14:14:18.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b6e74640 1 --2- 172.21.15.57:0/2893551017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b015ac70 0x7f37b015a210 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b6e74640 1 --2- 172.21.15.57:0/2893551017 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37b00c8eb0 0x7f37b015cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b6e74640 1 -- 172.21.15.57:0/2893551017 >> 172.21.15.57:0/2893551017 conn(0x7f37b00cef00 msgr2=0x7f37b00cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:18.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b6e74640 1 -- 172.21.15.57:0/2893551017 shutdown_connections 2023-10-30T14:14:18.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b6e74640 1 -- 172.21.15.57:0/2893551017 wait complete. 2023-10-30T14:14:18.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b6e74640 1 Processor -- start 2023-10-30T14:14:18.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b6e74640 1 -- start start 2023-10-30T14:14:18.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b6e74640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37b006aeb0 0x7f37b0150ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:18.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b6e74640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b00c8eb0 0x7f37b0154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:18.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b6e74640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37b015ac70 0x7f37b014e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:18.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b6e74640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b014eff0 con 0x7f37b00c8eb0 2023-10-30T14:14:18.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b6e74640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b0150850 con 0x7f37b006aeb0 2023-10-30T14:14:18.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b6e74640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b0158b80 con 0x7f37b015ac70 2023-10-30T14:14:18.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b5671640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b00c8eb0 0x7f37b0154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:18.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b5671640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b00c8eb0 0x7f37b0154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40080/0 (socket says 172.21.15.57:40080) 2023-10-30T14:14:18.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b5671640 1 -- 172.21.15.57:0/2233299222 learned_addr learned my addr 172.21.15.57:0/2233299222 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:18.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b5e72640 1 --2- 172.21.15.57:0/2233299222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37b006aeb0 0x7f37b0150ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:18.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b6673640 1 --2- 172.21.15.57:0/2233299222 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37b015ac70 0x7f37b014e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:18.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b5671640 1 -- 172.21.15.57:0/2233299222 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37b015ac70 msgr2=0x7f37b014e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:18.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b5671640 1 --2- 172.21.15.57:0/2233299222 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37b015ac70 0x7f37b014e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b5671640 1 -- 172.21.15.57:0/2233299222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37b006aeb0 msgr2=0x7f37b0150ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:18.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b5671640 1 --2- 172.21.15.57:0/2233299222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37b006aeb0 0x7f37b0150ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b5671640 1 -- 172.21.15.57:0/2233299222 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a00094d0 con 0x7f37b00c8eb0 2023-10-30T14:14:18.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.032+0000 7f37b5e72640 1 --2- 172.21.15.57:0/2233299222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37b006aeb0 0x7f37b0150ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:18.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.036+0000 7f37b5671640 1 --2- 172.21.15.57:0/2233299222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b00c8eb0 0x7f37b0154170 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f37a00b3040 tx=0x7f37a001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:18.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.036+0000 7f37a67fc640 1 -- 172.21.15.57:0/2233299222 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a00ba600 con 0x7f37b00c8eb0 2023-10-30T14:14:18.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.036+0000 7f37a67fc640 1 -- 172.21.15.57:0/2233299222 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a00ba780 con 0x7f37b00c8eb0 2023-10-30T14:14:18.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.036+0000 7f37a67fc640 1 -- 172.21.15.57:0/2233299222 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a00bab30 con 0x7f37b00c8eb0 2023-10-30T14:14:18.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.036+0000 7f37b6e74640 1 -- 172.21.15.57:0/2233299222 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b0158e60 con 0x7f37b00c8eb0 2023-10-30T14:14:18.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.036+0000 7f37b6e74640 1 -- 172.21.15.57:0/2233299222 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b0150200 con 0x7f37b00c8eb0 2023-10-30T14:14:18.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.036+0000 7f37b6e74640 1 -- 172.21.15.57:0/2233299222 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b0150a00 con 0x7f37b00c8eb0 2023-10-30T14:14:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.036+0000 7f37a67fc640 1 -- 172.21.15.57:0/2233299222 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f37a00bac90 con 0x7f37b00c8eb0 2023-10-30T14:14:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.036+0000 7f37a67fc640 1 --2- 172.21.15.57:0/2233299222 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f37940664a0 0x7f3794068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.036+0000 7f37a67fc640 1 -- 172.21.15.57:0/2233299222 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37a01224c0 con 0x7f37b00c8eb0 2023-10-30T14:14:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.036+0000 7f37b5e72640 1 --2- 172.21.15.57:0/2233299222 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f37940664a0 0x7f3794068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.040+0000 7f37b5e72640 1 --2- 172.21.15.57:0/2233299222 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f37940664a0 0x7f3794068960 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f37ac007a70 tx=0x7f37ac008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:18.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.040+0000 7f37a67fc640 1 -- 172.21.15.57:0/2233299222 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f37b0150a00 con 0x7f37b00c8eb0 2023-10-30T14:14:18.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.228+0000 7f37b6e74640 1 -- 172.21.15.57:0/2233299222 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37b0150a00 con 0x7f37b00c8eb0 2023-10-30T14:14:18.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.228+0000 7f37a67fc640 1 -- 172.21.15.57:0/2233299222 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+6518 (secure 0 0 0) 0x7f37b0150a00 con 0x7f37b00c8eb0 2023-10-30T14:14:18.235 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:18.236 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:14:15.802770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"e","rank":2,"incarnation":11,"state":"up:creating","state_seq":1,"addr":"172.21.15.57:6837/901968115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":901968115},{"type":"v1","addr":"172.21.15.57:6837","nonce":901968115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:18.236 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 11 2023-10-30T14:14:18.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.232+0000 7f37b6e74640 1 -- 172.21.15.57:0/2233299222 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f37940664a0 msgr2=0x7f3794068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:18.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.232+0000 7f37b6e74640 1 --2- 172.21.15.57:0/2233299222 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f37940664a0 0x7f3794068960 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f37ac007a70 tx=0x7f37ac008040 comp rx=0 tx=0).stop 2023-10-30T14:14:18.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.232+0000 7f37b6e74640 1 -- 172.21.15.57:0/2233299222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b00c8eb0 msgr2=0x7f37b0154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:18.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.232+0000 7f37b6e74640 1 --2- 172.21.15.57:0/2233299222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b00c8eb0 0x7f37b0154170 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f37a00b3040 tx=0x7f37a001bad0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.232+0000 7f37b6e74640 1 -- 172.21.15.57:0/2233299222 shutdown_connections 2023-10-30T14:14:18.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.232+0000 7f37b6e74640 1 --2- 172.21.15.57:0/2233299222 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f37940664a0 0x7f3794068960 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.232+0000 7f37b6e74640 1 --2- 172.21.15.57:0/2233299222 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37b015ac70 0x7f37b014e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.232+0000 7f37b6e74640 1 --2- 172.21.15.57:0/2233299222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37b00c8eb0 0x7f37b0154170 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.232+0000 7f37b6e74640 1 --2- 172.21.15.57:0/2233299222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37b006aeb0 0x7f37b0150ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.232+0000 7f37b6e74640 1 -- 172.21.15.57:0/2233299222 >> 172.21.15.57:0/2233299222 conn(0x7f37b00cef00 msgr2=0x7f37b00c65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:18.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.232+0000 7f37b6e74640 1 -- 172.21.15.57:0/2233299222 shutdown_connections 2023-10-30T14:14:18.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.232+0000 7f37b6e74640 1 -- 172.21.15.57:0/2233299222 wait complete. 2023-10-30T14:14:18.256 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:14:18.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.392+0000 7fa131e46640 1 Processor -- start 2023-10-30T14:14:18.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.392+0000 7fa131e46640 1 -- start start 2023-10-30T14:14:18.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.396+0000 7fa131e46640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa12c1537f0 0x7fa12c0701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:18.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.396+0000 7fa131e46640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa12c0c2aa0 con 0x7fa12c06b2f0 2023-10-30T14:14:18.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.396+0000 7fa131e46640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa12c0708e0 con 0x7fa12c15a2e0 2023-10-30T14:14:18.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.396+0000 7fa131e46640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa12c0c3ab0 con 0x7fa12c1537f0 2023-10-30T14:14:18.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.396+0000 7fa12bfff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa12c06b2f0 0x7fa12c06b700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:37762/0 (socket says 172.21.15.57:37762) 2023-10-30T14:14:18.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.396+0000 7fa12b7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa12c1537f0 0x7fa12c0701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:18.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.396+0000 7fa12bfff640 1 -- 172.21.15.57:0/580892081 learned_addr learned my addr 172.21.15.57:0/580892081 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:18.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.396+0000 7fa12b7fe640 1 -- 172.21.15.57:0/580892081 >> v1:172.21.15.88:6789/0 conn(0x7fa12c15a2e0 legacy=0x7fa12c0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:18.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.396+0000 7fa12b7fe640 1 -- 172.21.15.57:0/580892081 >> v1:172.21.15.57:6789/0 conn(0x7fa12c06b2f0 legacy=0x7fa12c06b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:18.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.396+0000 7fa12b7fe640 1 -- 172.21.15.57:0/580892081 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa12c0c1eb0 con 0x7fa12c1537f0 2023-10-30T14:14:18.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.396+0000 7fa12a7fc640 1 -- 172.21.15.57:0/580892081 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504238446 0 0) 0x7fa12c0c2aa0 con 0x7fa12c06b2f0 2023-10-30T14:14:18.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.400+0000 7fa12b7fe640 1 --2- 172.21.15.57:0/580892081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa12c1537f0 0x7fa12c0701a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0097c0 tx=0x7fa11c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a81860e710a8888 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:18.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.400+0000 7fa12a7fc640 1 -- 172.21.15.57:0/580892081 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11c0a8970 con 0x7fa12c1537f0 2023-10-30T14:14:18.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.400+0000 7fa12a7fc640 1 -- 172.21.15.57:0/580892081 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11c0a8ad0 con 0x7fa12c1537f0 2023-10-30T14:14:18.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.400+0000 7fa12a7fc640 1 -- 172.21.15.57:0/580892081 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11c0a8e60 con 0x7fa12c1537f0 2023-10-30T14:14:18.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.400+0000 7fa131e46640 1 -- 172.21.15.57:0/580892081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa12c1537f0 msgr2=0x7fa12c0701a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:18.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.400+0000 7fa131e46640 1 --2- 172.21.15.57:0/580892081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa12c1537f0 0x7fa12c0701a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0097c0 tx=0x7fa11c01ec70 comp rx=0 tx=0).stop 2023-10-30T14:14:18.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.400+0000 7fa131e46640 1 -- 172.21.15.57:0/580892081 shutdown_connections 2023-10-30T14:14:18.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.400+0000 7fa131e46640 1 --2- 172.21.15.57:0/580892081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa12c1537f0 0x7fa12c0701a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.400+0000 7fa131e46640 1 -- 172.21.15.57:0/580892081 >> 172.21.15.57:0/580892081 conn(0x7fa12c071fb0 msgr2=0x7fa12c0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:18.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.400+0000 7fa131e46640 1 -- 172.21.15.57:0/580892081 shutdown_connections 2023-10-30T14:14:18.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.400+0000 7fa131e46640 1 -- 172.21.15.57:0/580892081 wait complete. 2023-10-30T14:14:18.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.400+0000 7fa131e46640 1 Processor -- start 2023-10-30T14:14:18.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.400+0000 7fa131e46640 1 -- start start 2023-10-30T14:14:18.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa131e46640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa12c06b2f0 0x7fa12c0cc480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:18.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa131e46640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa12c1537f0 0x7fa12c0c5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:18.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa131e46640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa12c15a2e0 0x7fa12c0c6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:18.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa131e46640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12c0ca4f0 con 0x7fa12c15a2e0 2023-10-30T14:14:18.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa131e46640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12c0ca210 con 0x7fa12c1537f0 2023-10-30T14:14:18.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa131e46640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa12c0c8c90 con 0x7fa12c06b2f0 2023-10-30T14:14:18.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa12bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa12c15a2e0 0x7fa12c0c6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:18.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa12bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa12c15a2e0 0x7fa12c0c6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40084/0 (socket says 172.21.15.57:40084) 2023-10-30T14:14:18.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa12bfff640 1 -- 172.21.15.57:0/2580586327 learned_addr learned my addr 172.21.15.57:0/2580586327 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:18.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa12affd640 1 --2- 172.21.15.57:0/2580586327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa12c1537f0 0x7fa12c0c5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:18.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa12b7fe640 1 --2- 172.21.15.57:0/2580586327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa12c06b2f0 0x7fa12c0cc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:18.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa12bfff640 1 -- 172.21.15.57:0/2580586327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa12c06b2f0 msgr2=0x7fa12c0cc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:18.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa12bfff640 1 --2- 172.21.15.57:0/2580586327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa12c06b2f0 0x7fa12c0cc480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa12bfff640 1 -- 172.21.15.57:0/2580586327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa12c1537f0 msgr2=0x7fa12c0c5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:18.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa12bfff640 1 --2- 172.21.15.57:0/2580586327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa12c1537f0 0x7fa12c0c5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa12bfff640 1 -- 172.21.15.57:0/2580586327 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c009420 con 0x7fa12c15a2e0 2023-10-30T14:14:18.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa12b7fe640 1 --2- 172.21.15.57:0/2580586327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa12c06b2f0 0x7fa12c0cc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:18.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa12bfff640 1 --2- 172.21.15.57:0/2580586327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa12c15a2e0 0x7fa12c0c6250 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa124001570 tx=0x7fa12400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:18.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa113fff640 1 -- 172.21.15.57:0/2580586327 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12400a640 con 0x7fa12c15a2e0 2023-10-30T14:14:18.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa113fff640 1 -- 172.21.15.57:0/2580586327 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12400a7c0 con 0x7fa12c15a2e0 2023-10-30T14:14:18.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa113fff640 1 -- 172.21.15.57:0/2580586327 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12400ab70 con 0x7fa12c15a2e0 2023-10-30T14:14:18.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa131e46640 1 -- 172.21.15.57:0/2580586327 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa12c0c8f70 con 0x7fa12c15a2e0 2023-10-30T14:14:18.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa131e46640 1 -- 172.21.15.57:0/2580586327 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa12c0c9f20 con 0x7fa12c15a2e0 2023-10-30T14:14:18.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.404+0000 7fa131e46640 1 -- 172.21.15.57:0/2580586327 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa12c0c5a30 con 0x7fa12c15a2e0 2023-10-30T14:14:18.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.408+0000 7fa113fff640 1 -- 172.21.15.57:0/2580586327 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa12400acd0 con 0x7fa12c15a2e0 2023-10-30T14:14:18.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.408+0000 7fa113fff640 1 --2- 172.21.15.57:0/2580586327 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa1180664d0 0x7fa118068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:18.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.408+0000 7fa113fff640 1 -- 172.21.15.57:0/2580586327 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1240833a0 con 0x7fa12c15a2e0 2023-10-30T14:14:18.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.408+0000 7fa12b7fe640 1 --2- 172.21.15.57:0/2580586327 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa1180664d0 0x7fa118068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:18.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.408+0000 7fa12b7fe640 1 --2- 172.21.15.57:0/2580586327 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa1180664d0 0x7fa118068990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0b2000 tx=0x7fa11c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:18.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.412+0000 7fa113fff640 1 -- 172.21.15.57:0/2580586327 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa12c0c5a30 con 0x7fa12c15a2e0 2023-10-30T14:14:18.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.572+0000 7fa131e46640 1 -- 172.21.15.57:0/2580586327 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa12c0c5a30 con 0x7fa12c15a2e0 2023-10-30T14:14:18.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.572+0000 7fa113fff640 1 -- 172.21.15.57:0/2580586327 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16433 (secure 0 0 0) 0x7fa12c0c5a30 con 0x7fa12c15a2e0 2023-10-30T14:14:18.577 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:18.577 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:14:05.576818+0000","last_up_change":"2023-10-30T14:13:51.171287+0000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T14:13:52.574639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T14:14:02.600122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T14:14:03.069429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6801","nonce":3287367848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6803","nonce":3287367848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6807","nonce":3287367848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6805","nonce":3287367848}]},"public_addr":"172.21.15.57:6801/3287367848","cluster_addr":"172.21.15.57:6803/3287367848","heartbeat_back_addr":"172.21.15.57:6807/3287367848","heartbeat_front_addr":"172.21.15.57:6805/3287367848","state":["exists","up"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6809","nonce":3340797836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6811","nonce":3340797836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6815","nonce":3340797836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6813","nonce":3340797836}]},"public_addr":"172.21.15.57:6809/3340797836","cluster_addr":"172.21.15.57:6811/3340797836","heartbeat_back_addr":"172.21.15.57:6815/3340797836","heartbeat_front_addr":"172.21.15.57:6813/3340797836","state":["exists","up"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6825","nonce":1700292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6827","nonce":1700292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6831","nonce":1700292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6829","nonce":1700292}]},"public_addr":"172.21.15.57:6825/1700292","cluster_addr":"172.21.15.57:6827/1700292","heartbeat_back_addr":"172.21.15.57:6831/1700292","heartbeat_front_addr":"172.21.15.57:6829/1700292","state":["exists","up"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6817","nonce":2893420631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6819","nonce":2893420631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6823","nonce":2893420631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6821","nonce":2893420631}]},"public_addr":"172.21.15.57:6817/2893420631","cluster_addr":"172.21.15.57:6819/2893420631","heartbeat_back_addr":"172.21.15.57:6823/2893420631","heartbeat_front_addr":"172.21.15.57:6821/2893420631","state":["exists","up"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6807","nonce":2784816485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6811","nonce":2784816485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6815","nonce":2784816485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6813","nonce":2784816485}]},"public_addr":"172.21.15.88:6807/2784816485","cluster_addr":"172.21.15.88:6811/2784816485","heartbeat_back_addr":"172.21.15.88:6815/2784816485","heartbeat_front_addr":"172.21.15.88:6813/2784816485","state":["exists","up"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6801","nonce":2682449812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6803","nonce":2682449812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6810","nonce":2682449812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6806","nonce":2682449812}]},"public_addr":"172.21.15.88:6801/2682449812","cluster_addr":"172.21.15.88:6803/2682449812","heartbeat_back_addr":"172.21.15.88:6810/2682449812","heartbeat_front_addr":"172.21.15.88:6806/2682449812","state":["exists","up"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6822","nonce":3340693897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6827","nonce":3340693897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6831","nonce":3340693897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6829","nonce":3340693897}]},"public_addr":"172.21.15.88:6822/3340693897","cluster_addr":"172.21.15.88:6827/3340693897","heartbeat_back_addr":"172.21.15.88:6831/3340693897","heartbeat_front_addr":"172.21.15.88:6829/3340693897","state":["exists","up"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6817","nonce":2862111199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6820","nonce":2862111199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6825","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6826","nonce":2862111199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6821","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6823","nonce":2862111199}]},"public_addr":"172.21.15.88:6817/2862111199","cluster_addr":"172.21.15.88:6820/2862111199","heartbeat_back_addr":"172.21.15.88:6826/2862111199","heartbeat_front_addr":"172.21.15.88:6823/2862111199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.634920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.773993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.725825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.761452+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.694274+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.650718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:14:18.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.572+0000 7fa131e46640 1 -- 172.21.15.57:0/2580586327 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa1180664d0 msgr2=0x7fa118068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:18.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.572+0000 7fa131e46640 1 --2- 172.21.15.57:0/2580586327 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa1180664d0 0x7fa118068990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0b2000 tx=0x7fa11c008040 comp rx=0 tx=0).stop 2023-10-30T14:14:18.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.572+0000 7fa131e46640 1 -- 172.21.15.57:0/2580586327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa12c15a2e0 msgr2=0x7fa12c0c6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:18.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.572+0000 7fa131e46640 1 --2- 172.21.15.57:0/2580586327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa12c15a2e0 0x7fa12c0c6250 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa124001570 tx=0x7fa12400cca0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.572+0000 7fa131e46640 1 -- 172.21.15.57:0/2580586327 shutdown_connections 2023-10-30T14:14:18.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.572+0000 7fa131e46640 1 --2- 172.21.15.57:0/2580586327 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa1180664d0 0x7fa118068990 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.572+0000 7fa131e46640 1 --2- 172.21.15.57:0/2580586327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa12c15a2e0 0x7fa12c0c6250 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.572+0000 7fa131e46640 1 --2- 172.21.15.57:0/2580586327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa12c1537f0 0x7fa12c0c5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.572+0000 7fa131e46640 1 --2- 172.21.15.57:0/2580586327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa12c06b2f0 0x7fa12c0cc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.572+0000 7fa131e46640 1 -- 172.21.15.57:0/2580586327 >> 172.21.15.57:0/2580586327 conn(0x7fa12c071fb0 msgr2=0x7fa12c06be50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:18.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.572+0000 7fa131e46640 1 -- 172.21.15.57:0/2580586327 shutdown_connections 2023-10-30T14:14:18.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.572+0000 7fa131e46640 1 -- 172.21.15.57:0/2580586327 wait complete. 2023-10-30T14:14:18.599 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T14:13:52.574639+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T14:14:02.600122+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T14:14:03.069429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T14:14:18.600 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T14:14:18.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.740+0000 7f097cbf2640 1 Processor -- start 2023-10-30T14:14:18.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.740+0000 7f097cbf2640 1 -- start start 2023-10-30T14:14:18.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.744+0000 7f097cbf2640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0978071cb0 0x7f09780c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:18.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.744+0000 7f097cbf2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0978153300 0x7f0978153770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:18.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.744+0000 7f097cbf2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0978153cb0 con 0x7f09780718d0 2023-10-30T14:14:18.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.744+0000 7f097cbf2640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09780cca10 con 0x7f0978153300 2023-10-30T14:14:18.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.744+0000 7f097cbf2640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f097815d0a0 con 0x7f0978071cb0 2023-10-30T14:14:18.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.744+0000 7f0977fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0978153300 0x7f0978153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:18.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.744+0000 7f0977fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0978153300 0x7f0978153770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40092/0 (socket says 172.21.15.57:40092) 2023-10-30T14:14:18.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.744+0000 7f0977fff640 1 -- 172.21.15.57:0/2520773041 learned_addr learned my addr 172.21.15.57:0/2520773041 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:18.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.744+0000 7f0976ffd640 1 --2- 172.21.15.57:0/2520773041 >> v2:172.21.15.88:3300/0 conn(0x7f0978071cb0 0x7f09780c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:18.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.744+0000 7f0977fff640 1 -- 172.21.15.57:0/2520773041 >> v2:172.21.15.88:3300/0 conn(0x7f0978071cb0 msgr2=0x7f09780c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:18.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.744+0000 7f0977fff640 1 --2- 172.21.15.57:0/2520773041 >> v2:172.21.15.88:3300/0 conn(0x7f0978071cb0 0x7f09780c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.744+0000 7f0977fff640 1 -- 172.21.15.57:0/2520773041 >> v1:172.21.15.88:6789/0 conn(0x7f09780718d0 legacy=0x7f097815a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:18.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.744+0000 7f0977fff640 1 -- 172.21.15.57:0/2520773041 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097815d380 con 0x7f0978153300 2023-10-30T14:14:18.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.744+0000 7f0977fff640 1 --2- 172.21.15.57:0/2520773041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0978153300 0x7f0978153770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0978072470 tx=0x7f097001f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77626d55e0685f90 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:18.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f09767fc640 1 -- 172.21.15.57:0/2520773041 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09700af400 con 0x7f0978153300 2023-10-30T14:14:18.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f09767fc640 1 -- 172.21.15.57:0/2520773041 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09700ab050 con 0x7f0978153300 2023-10-30T14:14:18.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f09767fc640 1 -- 172.21.15.57:0/2520773041 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09700af780 con 0x7f0978153300 2023-10-30T14:14:18.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f097cbf2640 1 -- 172.21.15.57:0/2520773041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0978153300 msgr2=0x7f0978153770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:18.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f097cbf2640 1 --2- 172.21.15.57:0/2520773041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0978153300 0x7f0978153770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0978072470 tx=0x7f097001f180 comp rx=0 tx=0).stop 2023-10-30T14:14:18.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f097cbf2640 1 -- 172.21.15.57:0/2520773041 shutdown_connections 2023-10-30T14:14:18.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f097cbf2640 1 --2- 172.21.15.57:0/2520773041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0978153300 0x7f0978153770 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f097cbf2640 1 --2- 172.21.15.57:0/2520773041 >> v2:172.21.15.88:3300/0 conn(0x7f0978071cb0 0x7f09780c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f097cbf2640 1 -- 172.21.15.57:0/2520773041 >> 172.21.15.57:0/2520773041 conn(0x7f097806ab00 msgr2=0x7f0978065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:18.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f097cbf2640 1 -- 172.21.15.57:0/2520773041 shutdown_connections 2023-10-30T14:14:18.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f097cbf2640 1 -- 172.21.15.57:0/2520773041 wait complete. 2023-10-30T14:14:18.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f097cbf2640 1 Processor -- start 2023-10-30T14:14:18.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f097cbf2640 1 -- start start 2023-10-30T14:14:18.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f097cbf2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09780718d0 0x7f097814e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:18.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f097cbf2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0978071cb0 0x7f0978146370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:18.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f097cbf2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0978153300 0x7f097814ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:18.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f097cbf2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097814e760 con 0x7f0978153300 2023-10-30T14:14:18.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f097cbf2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09781468b0 con 0x7f09780718d0 2023-10-30T14:14:18.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f097cbf2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09780b3640 con 0x7f0978071cb0 2023-10-30T14:14:18.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.748+0000 7f0977fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0978153300 0x7f097814ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:18.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.752+0000 7f0977fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0978153300 0x7f097814ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40106/0 (socket says 172.21.15.57:40106) 2023-10-30T14:14:18.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.752+0000 7f0977fff640 1 -- 172.21.15.57:0/1724294978 learned_addr learned my addr 172.21.15.57:0/1724294978 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:18.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.752+0000 7f09777fe640 1 --2- 172.21.15.57:0/1724294978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09780718d0 0x7f097814e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:18.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.752+0000 7f0976ffd640 1 --2- 172.21.15.57:0/1724294978 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0978071cb0 0x7f0978146370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:18.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.752+0000 7f0977fff640 1 -- 172.21.15.57:0/1724294978 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0978071cb0 msgr2=0x7f0978146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:18.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.752+0000 7f0977fff640 1 --2- 172.21.15.57:0/1724294978 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0978071cb0 0x7f0978146370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.752+0000 7f0977fff640 1 -- 172.21.15.57:0/1724294978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09780718d0 msgr2=0x7f097814e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:18.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.752+0000 7f0977fff640 1 --2- 172.21.15.57:0/1724294978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09780718d0 0x7f097814e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.752+0000 7f0977fff640 1 -- 172.21.15.57:0/1724294978 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0970009b30 con 0x7f0978153300 2023-10-30T14:14:18.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.752+0000 7f09777fe640 1 --2- 172.21.15.57:0/1724294978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09780718d0 0x7f097814e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:18.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.752+0000 7f0977fff640 1 --2- 172.21.15.57:0/1724294978 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0978153300 0x7f097814ee20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f097001cdb0 tx=0x7f097001bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:18.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.752+0000 7f095ffff640 1 -- 172.21.15.57:0/1724294978 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09700afde0 con 0x7f0978153300 2023-10-30T14:14:18.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.752+0000 7f095ffff640 1 -- 172.21.15.57:0/1724294978 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0970009510 con 0x7f0978153300 2023-10-30T14:14:18.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.752+0000 7f095ffff640 1 -- 172.21.15.57:0/1724294978 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09700ba660 con 0x7f0978153300 2023-10-30T14:14:18.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.752+0000 7f097cbf2640 1 -- 172.21.15.57:0/1724294978 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09780c7360 con 0x7f0978153300 2023-10-30T14:14:18.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.752+0000 7f097cbf2640 1 -- 172.21.15.57:0/1724294978 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09780b3b90 con 0x7f0978153300 2023-10-30T14:14:18.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.752+0000 7f097cbf2640 1 -- 172.21.15.57:0/1724294978 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09780c74f0 con 0x7f0978153300 2023-10-30T14:14:18.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.752+0000 7f095ffff640 1 -- 172.21.15.57:0/1724294978 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f09700b6040 con 0x7f0978153300 2023-10-30T14:14:18.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.756+0000 7f095ffff640 1 --2- 172.21.15.57:0/1724294978 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09640664a0 0x7f0964068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:18.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.756+0000 7f095ffff640 1 -- 172.21.15.57:0/1724294978 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09701220d0 con 0x7f0978153300 2023-10-30T14:14:18.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.756+0000 7f09777fe640 1 --2- 172.21.15.57:0/1724294978 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09640664a0 0x7f0964068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:18.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.756+0000 7f09777fe640 1 --2- 172.21.15.57:0/1724294978 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09640664a0 0x7f0964068960 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0968003ec0 tx=0x7f0968009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:18.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.760+0000 7f095ffff640 1 -- 172.21.15.57:0/1724294978 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f09700ef420 con 0x7f0978153300 2023-10-30T14:14:18.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.920+0000 7f097cbf2640 1 -- 172.21.15.57:0/1724294978 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f097804e170 con 0x7f0978153300 2023-10-30T14:14:18.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.920+0000 7f095ffff640 1 -- 172.21.15.57:0/1724294978 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f09700f2a40 con 0x7f0978153300 2023-10-30T14:14:18.926 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-30T14:14:18.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.920+0000 7f097cbf2640 1 -- 172.21.15.57:0/1724294978 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09640664a0 msgr2=0x7f0964068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:18.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.920+0000 7f097cbf2640 1 --2- 172.21.15.57:0/1724294978 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09640664a0 0x7f0964068960 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0968003ec0 tx=0x7f0968009040 comp rx=0 tx=0).stop 2023-10-30T14:14:18.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.920+0000 7f097cbf2640 1 -- 172.21.15.57:0/1724294978 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0978153300 msgr2=0x7f097814ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:18.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.920+0000 7f097cbf2640 1 --2- 172.21.15.57:0/1724294978 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0978153300 0x7f097814ee20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f097001cdb0 tx=0x7f097001bf20 comp rx=0 tx=0).stop 2023-10-30T14:14:18.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.920+0000 7f097cbf2640 1 -- 172.21.15.57:0/1724294978 shutdown_connections 2023-10-30T14:14:18.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.920+0000 7f097cbf2640 1 --2- 172.21.15.57:0/1724294978 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09640664a0 0x7f0964068960 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.920+0000 7f097cbf2640 1 --2- 172.21.15.57:0/1724294978 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0978153300 0x7f097814ee20 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.920+0000 7f097cbf2640 1 --2- 172.21.15.57:0/1724294978 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0978071cb0 0x7f0978146370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.920+0000 7f097cbf2640 1 --2- 172.21.15.57:0/1724294978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09780718d0 0x7f097814e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:18.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.920+0000 7f097cbf2640 1 -- 172.21.15.57:0/1724294978 >> 172.21.15.57:0/1724294978 conn(0x7f097806ab00 msgr2=0x7f0978158620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:18.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.920+0000 7f097cbf2640 1 -- 172.21.15.57:0/1724294978 shutdown_connections 2023-10-30T14:14:18.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:18.920+0000 7f097cbf2640 1 -- 172.21.15.57:0/1724294978 wait complete. 2023-10-30T14:14:18.947 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T14:14:19.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2d331d640 1 Processor -- start 2023-10-30T14:14:19.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2d331d640 1 -- start start 2023-10-30T14:14:19.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2d331d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe2cc0c4410 0x7fe2cc0cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2d331d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2cc0c4810 0x7fe2cc15a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2d331d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2cc153f80 con 0x7fe2cc153ba0 2023-10-30T14:14:19.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2d331d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2cc159df0 con 0x7fe2cc0c4410 2023-10-30T14:14:19.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2d331d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc15a7c0 con 0x7fe2cc0c4810 2023-10-30T14:14:19.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2d10b9640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fe2cc153ba0 0x7fe2cc15cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:37778/0 (socket says 172.21.15.57:37778) 2023-10-30T14:14:19.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2d10b9640 1 -- 172.21.15.57:0/1661345830 learned_addr learned my addr 172.21.15.57:0/1661345830 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:19.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2d18ba640 1 --2- 172.21.15.57:0/1661345830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2cc0c4810 0x7fe2cc15a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2d08b8640 1 --2- 172.21.15.57:0/1661345830 >> v2:172.21.15.88:3300/0 conn(0x7fe2cc0c4410 0x7fe2cc0cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2c3fff640 1 -- 172.21.15.57:0/1661345830 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208888633 0 0) 0x7fe2cc153f80 con 0x7fe2cc153ba0 2023-10-30T14:14:19.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2c3fff640 1 -- 172.21.15.57:0/1661345830 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b8003530 con 0x7fe2cc153ba0 2023-10-30T14:14:19.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2c3fff640 1 -- 172.21.15.57:0/1661345830 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 372485563 0 0) 0x7fe2b8003530 con 0x7fe2cc153ba0 2023-10-30T14:14:19.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2c3fff640 1 -- 172.21.15.57:0/1661345830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2cc0c4810 msgr2=0x7fe2cc15a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:19.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2c3fff640 1 --2- 172.21.15.57:0/1661345830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2cc0c4810 0x7fe2cc15a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2c3fff640 1 -- 172.21.15.57:0/1661345830 >> v2:172.21.15.88:3300/0 conn(0x7fe2cc0c4410 msgr2=0x7fe2cc0cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:19.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2c3fff640 1 --2- 172.21.15.57:0/1661345830 >> v2:172.21.15.88:3300/0 conn(0x7fe2cc0c4410 0x7fe2cc0cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2c3fff640 1 -- 172.21.15.57:0/1661345830 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc1597c0 con 0x7fe2cc153ba0 2023-10-30T14:14:19.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2c3fff640 1 -- 172.21.15.57:0/1661345830 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fe2c8001680 con 0x7fe2cc153ba0 2023-10-30T14:14:19.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2c3fff640 1 -- 172.21.15.57:0/1661345830 >> v1:172.21.15.57:6789/0 conn(0x7fe2cc153ba0 legacy=0x7fe2cc15cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:19.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2c3fff640 1 --2- 172.21.15.57:0/1661345830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2b8003980 0x7fe2b8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2c3fff640 1 --2- 172.21.15.57:0/1661345830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2b80047d0 0x7fe2b8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2c3fff640 1 --2- 172.21.15.57:0/1661345830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2cc0c4810 0x7fe2b8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2c3fff640 1 -- 172.21.15.57:0/1661345830 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc159df0 con 0x7fe2b8003980 2023-10-30T14:14:19.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.088+0000 7fe2c3fff640 1 -- 172.21.15.57:0/1661345830 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc15a7c0 con 0x7fe2b80047d0 2023-10-30T14:14:19.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.092+0000 7fe2c3fff640 1 -- 172.21.15.57:0/1661345830 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b800b990 con 0x7fe2cc0c4810 2023-10-30T14:14:19.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.092+0000 7fe2d10b9640 1 --2- 172.21.15.57:0/1661345830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2b8003980 0x7fe2b8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.092+0000 7fe2d08b8640 1 --2- 172.21.15.57:0/1661345830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2b80047d0 0x7fe2b8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.092+0000 7fe2d18ba640 1 --2- 172.21.15.57:0/1661345830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2cc0c4810 0x7fe2b8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.092+0000 7fe2d10b9640 1 -- 172.21.15.57:0/1661345830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2cc0c4810 msgr2=0x7fe2b8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:19.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.092+0000 7fe2d10b9640 1 --2- 172.21.15.57:0/1661345830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2cc0c4810 0x7fe2b8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.092+0000 7fe2d10b9640 1 -- 172.21.15.57:0/1661345830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2b80047d0 msgr2=0x7fe2b8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:19.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.092+0000 7fe2d10b9640 1 --2- 172.21.15.57:0/1661345830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2b80047d0 0x7fe2b8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.092+0000 7fe2d10b9640 1 -- 172.21.15.57:0/1661345830 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b8003530 con 0x7fe2b8003980 2023-10-30T14:14:19.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.092+0000 7fe2d18ba640 1 --2- 172.21.15.57:0/1661345830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2cc0c4810 0x7fe2b8005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:19.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.092+0000 7fe2d10b9640 1 --2- 172.21.15.57:0/1661345830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2b8003980 0x7fe2b8003da0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800c270 tx=0x7fe2c801cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d79898b26842e8bc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:19.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.092+0000 7fe2c3fff640 1 -- 172.21.15.57:0/1661345830 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c80058e0 con 0x7fe2b8003980 2023-10-30T14:14:19.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.092+0000 7fe2c3fff640 1 -- 172.21.15.57:0/1661345830 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c8005a40 con 0x7fe2b8003980 2023-10-30T14:14:19.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.092+0000 7fe2c3fff640 1 -- 172.21.15.57:0/1661345830 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c8005e10 con 0x7fe2b8003980 2023-10-30T14:14:19.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d331d640 1 -- 172.21.15.57:0/1661345830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2b8003980 msgr2=0x7fe2b8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:19.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d331d640 1 --2- 172.21.15.57:0/1661345830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2b8003980 0x7fe2b8003da0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800c270 tx=0x7fe2c801cc30 comp rx=0 tx=0).stop 2023-10-30T14:14:19.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d18ba640 1 -- 172.21.15.57:0/1661345830 reap_dead start 2023-10-30T14:14:19.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d331d640 1 -- 172.21.15.57:0/1661345830 shutdown_connections 2023-10-30T14:14:19.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d331d640 1 -- 172.21.15.57:0/1661345830 >> 172.21.15.57:0/1661345830 conn(0x7fe2cc071790 msgr2=0x7fe2cc071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:19.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d331d640 1 -- 172.21.15.57:0/1661345830 shutdown_connections 2023-10-30T14:14:19.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d331d640 1 -- 172.21.15.57:0/1661345830 wait complete. 2023-10-30T14:14:19.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d331d640 1 Processor -- start 2023-10-30T14:14:19.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d331d640 1 -- start start 2023-10-30T14:14:19.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d331d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2b8003980 0x7fe2cc15be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d331d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2cc150800 0x7fe2cc1499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d331d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2cc150c60 0x7fe2cc149ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d331d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc0c6530 con 0x7fe2cc150c60 2023-10-30T14:14:19.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d331d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc0c67f0 con 0x7fe2b8003980 2023-10-30T14:14:19.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d331d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc0c8210 con 0x7fe2cc150800 2023-10-30T14:14:19.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d18ba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2cc150c60 0x7fe2cc149ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d18ba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2cc150c60 0x7fe2cc149ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40112/0 (socket says 172.21.15.57:40112) 2023-10-30T14:14:19.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d18ba640 1 -- 172.21.15.57:0/3432691794 learned_addr learned my addr 172.21.15.57:0/3432691794 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:19.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d10b9640 1 --2- 172.21.15.57:0/3432691794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2b8003980 0x7fe2cc15be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d08b8640 1 --2- 172.21.15.57:0/3432691794 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2cc150800 0x7fe2cc1499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d18ba640 1 -- 172.21.15.57:0/3432691794 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2cc150800 msgr2=0x7fe2cc1499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:19.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d18ba640 1 --2- 172.21.15.57:0/3432691794 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2cc150800 0x7fe2cc1499a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d18ba640 1 -- 172.21.15.57:0/3432691794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2b8003980 msgr2=0x7fe2cc15be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:19.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d18ba640 1 --2- 172.21.15.57:0/3432691794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2b8003980 0x7fe2cc15be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d18ba640 1 -- 172.21.15.57:0/3432691794 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc0c84f0 con 0x7fe2cc150c60 2023-10-30T14:14:19.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.096+0000 7fe2d18ba640 1 --2- 172.21.15.57:0/3432691794 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2cc150c60 0x7fe2cc149ee0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc06a450 tx=0x7fe2c40115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:19.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.100+0000 7fe2c1ffb640 1 -- 172.21.15.57:0/3432691794 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c400e030 con 0x7fe2cc150c60 2023-10-30T14:14:19.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.100+0000 7fe2c1ffb640 1 -- 172.21.15.57:0/3432691794 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c4011ed0 con 0x7fe2cc150c60 2023-10-30T14:14:19.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.100+0000 7fe2c1ffb640 1 -- 172.21.15.57:0/3432691794 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c40166b0 con 0x7fe2cc150c60 2023-10-30T14:14:19.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.100+0000 7fe2d331d640 1 -- 172.21.15.57:0/3432691794 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2cc0c87d0 con 0x7fe2cc150c60 2023-10-30T14:14:19.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.100+0000 7fe2d331d640 1 -- 172.21.15.57:0/3432691794 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2cc15c380 con 0x7fe2cc150c60 2023-10-30T14:14:19.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.100+0000 7fe2d331d640 1 -- 172.21.15.57:0/3432691794 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2cc04e170 con 0x7fe2cc150c60 2023-10-30T14:14:19.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.100+0000 7fe2c1ffb640 1 -- 172.21.15.57:0/3432691794 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe2c4016810 con 0x7fe2cc150c60 2023-10-30T14:14:19.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.100+0000 7fe2c1ffb640 1 --2- 172.21.15.57:0/3432691794 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe2b00664a0 0x7fe2b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.100+0000 7fe2c1ffb640 1 -- 172.21.15.57:0/3432691794 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2c400f070 con 0x7fe2cc150c60 2023-10-30T14:14:19.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.100+0000 7fe2d10b9640 1 --2- 172.21.15.57:0/3432691794 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe2b00664a0 0x7fe2b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.104+0000 7fe2d10b9640 1 --2- 172.21.15.57:0/3432691794 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe2b00664a0 0x7fe2b0068960 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe2c80aa000 tx=0x7fe2c8001a80 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:19.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.104+0000 7fe2c1ffb640 1 -- 172.21.15.57:0/3432691794 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe2c4051cd0 con 0x7fe2cc150c60 2023-10-30T14:14:19.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.264+0000 7fe2d331d640 1 -- 172.21.15.57:0/3432691794 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe2cc0cabb0 con 0x7fe2cc150c60 2023-10-30T14:14:19.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.264+0000 7fe2c1ffb640 1 -- 172.21.15.57:0/3432691794 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe2c40552f0 con 0x7fe2cc150c60 2023-10-30T14:14:19.269 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-30T14:14:19.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.264+0000 7fe2d331d640 1 -- 172.21.15.57:0/3432691794 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe2b00664a0 msgr2=0x7fe2b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:19.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.264+0000 7fe2d331d640 1 --2- 172.21.15.57:0/3432691794 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe2b00664a0 0x7fe2b0068960 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe2c80aa000 tx=0x7fe2c8001a80 comp rx=0 tx=0).stop 2023-10-30T14:14:19.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.264+0000 7fe2d331d640 1 -- 172.21.15.57:0/3432691794 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2cc150c60 msgr2=0x7fe2cc149ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:19.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.264+0000 7fe2d331d640 1 --2- 172.21.15.57:0/3432691794 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2cc150c60 0x7fe2cc149ee0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc06a450 tx=0x7fe2c40115a0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.264+0000 7fe2d331d640 1 -- 172.21.15.57:0/3432691794 shutdown_connections 2023-10-30T14:14:19.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.264+0000 7fe2d331d640 1 --2- 172.21.15.57:0/3432691794 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe2b00664a0 0x7fe2b0068960 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.264+0000 7fe2d331d640 1 --2- 172.21.15.57:0/3432691794 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2cc150c60 0x7fe2cc149ee0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.264+0000 7fe2d331d640 1 --2- 172.21.15.57:0/3432691794 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2cc150800 0x7fe2cc1499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.264+0000 7fe2d331d640 1 --2- 172.21.15.57:0/3432691794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2b8003980 0x7fe2cc15be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.264+0000 7fe2d331d640 1 -- 172.21.15.57:0/3432691794 >> 172.21.15.57:0/3432691794 conn(0x7fe2cc071790 msgr2=0x7fe2cc06bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:19.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.264+0000 7fe2d331d640 1 -- 172.21.15.57:0/3432691794 shutdown_connections 2023-10-30T14:14:19.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.264+0000 7fe2d331d640 1 -- 172.21.15.57:0/3432691794 wait complete. 2023-10-30T14:14:19.290 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T14:14:19.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.432+0000 7fed79a46640 1 Processor -- start 2023-10-30T14:14:19.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.432+0000 7fed79a46640 1 -- start start 2023-10-30T14:14:19.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.432+0000 7fed79a46640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fed740c4410 0x7fed740cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.432+0000 7fed79a46640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fed740c4810 0x7fed7415a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.432+0000 7fed79a46640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed74153f80 con 0x7fed74153ba0 2023-10-30T14:14:19.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.432+0000 7fed79a46640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed74159df0 con 0x7fed740c4410 2023-10-30T14:14:19.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.432+0000 7fed79a46640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed7415a7c0 con 0x7fed740c4810 2023-10-30T14:14:19.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.432+0000 7fed727fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fed740c4410 0x7fed740cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.432+0000 7fed72ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fed74153ba0 0x7fed7415cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:46138/0 (socket says 172.21.15.57:46138) 2023-10-30T14:14:19.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.432+0000 7fed727fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fed740c4410 0x7fed740cb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40128/0 (socket says 172.21.15.57:40128) 2023-10-30T14:14:19.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.432+0000 7fed72ffd640 1 -- 172.21.15.57:0/2540841399 learned_addr learned my addr 172.21.15.57:0/2540841399 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:19.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.432+0000 7fed737fe640 1 --2- 172.21.15.57:0/2540841399 >> v2:172.21.15.88:3300/0 conn(0x7fed740c4810 0x7fed7415a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.432+0000 7fed727fc640 1 -- 172.21.15.57:0/2540841399 >> v2:172.21.15.88:3300/0 conn(0x7fed740c4810 msgr2=0x7fed7415a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:19.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.432+0000 7fed727fc640 1 --2- 172.21.15.57:0/2540841399 >> v2:172.21.15.88:3300/0 conn(0x7fed740c4810 0x7fed7415a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.432+0000 7fed727fc640 1 -- 172.21.15.57:0/2540841399 >> v1:172.21.15.88:6789/0 conn(0x7fed74153ba0 legacy=0x7fed7415cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:19.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.432+0000 7fed727fc640 1 -- 172.21.15.57:0/2540841399 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed741597c0 con 0x7fed740c4410 2023-10-30T14:14:19.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.436+0000 7fed727fc640 1 --2- 172.21.15.57:0/2540841399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed740c4410 0x7fed740cb4e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fed740c4250 tx=0x7fed6801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a4901ec352904c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:19.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.436+0000 7fed71ffb640 1 -- 172.21.15.57:0/2540841399 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed680ac600 con 0x7fed740c4410 2023-10-30T14:14:19.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.436+0000 7fed71ffb640 1 -- 172.21.15.57:0/2540841399 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed680a8050 con 0x7fed740c4410 2023-10-30T14:14:19.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.436+0000 7fed71ffb640 1 -- 172.21.15.57:0/2540841399 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed680ac980 con 0x7fed740c4410 2023-10-30T14:14:19.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.436+0000 7fed79a46640 1 -- 172.21.15.57:0/2540841399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed740c4410 msgr2=0x7fed740cb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:19.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.436+0000 7fed79a46640 1 --2- 172.21.15.57:0/2540841399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed740c4410 0x7fed740cb4e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fed740c4250 tx=0x7fed6801c3a0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.436+0000 7fed79a46640 1 -- 172.21.15.57:0/2540841399 shutdown_connections 2023-10-30T14:14:19.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.436+0000 7fed79a46640 1 --2- 172.21.15.57:0/2540841399 >> v2:172.21.15.88:3300/0 conn(0x7fed740c4810 0x7fed7415a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.436+0000 7fed79a46640 1 --2- 172.21.15.57:0/2540841399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed740c4410 0x7fed740cb4e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.436+0000 7fed79a46640 1 -- 172.21.15.57:0/2540841399 >> 172.21.15.57:0/2540841399 conn(0x7fed74071790 msgr2=0x7fed74071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:19.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.436+0000 7fed79a46640 1 -- 172.21.15.57:0/2540841399 shutdown_connections 2023-10-30T14:14:19.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.436+0000 7fed79a46640 1 -- 172.21.15.57:0/2540841399 wait complete. 2023-10-30T14:14:19.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.436+0000 7fed79a46640 1 Processor -- start 2023-10-30T14:14:19.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed79a46640 1 -- start start 2023-10-30T14:14:19.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed79a46640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed740c4410 0x7fed741f6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed79a46640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed740c4810 0x7fed7415a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed79a46640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed74153ba0 0x7fed740c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed79a46640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed740c8980 con 0x7fed74153ba0 2023-10-30T14:14:19.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed79a46640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7415c5e0 con 0x7fed740c4810 2023-10-30T14:14:19.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed79a46640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed741f73f0 con 0x7fed740c4410 2023-10-30T14:14:19.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed737fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed74153ba0 0x7fed740c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed737fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed74153ba0 0x7fed740c8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40140/0 (socket says 172.21.15.57:40140) 2023-10-30T14:14:19.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed737fe640 1 -- 172.21.15.57:0/3540420404 learned_addr learned my addr 172.21.15.57:0/3540420404 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:19.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed72ffd640 1 --2- 172.21.15.57:0/3540420404 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed740c4410 0x7fed741f6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed727fc640 1 --2- 172.21.15.57:0/3540420404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed740c4810 0x7fed7415a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed737fe640 1 -- 172.21.15.57:0/3540420404 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed740c4410 msgr2=0x7fed741f6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:19.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed737fe640 1 --2- 172.21.15.57:0/3540420404 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed740c4410 0x7fed741f6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed737fe640 1 -- 172.21.15.57:0/3540420404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed740c4810 msgr2=0x7fed7415a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:19.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed737fe640 1 --2- 172.21.15.57:0/3540420404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed740c4810 0x7fed7415a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed737fe640 1 -- 172.21.15.57:0/3540420404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed68009420 con 0x7fed74153ba0 2023-10-30T14:14:19.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed72ffd640 1 --2- 172.21.15.57:0/3540420404 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed740c4410 0x7fed741f6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:19.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed737fe640 1 --2- 172.21.15.57:0/3540420404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed74153ba0 0x7fed740c8d50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fed6c007e40 tx=0x7fed6c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:19.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed5bfff640 1 -- 172.21.15.57:0/3540420404 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c014620 con 0x7fed74153ba0 2023-10-30T14:14:19.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed5bfff640 1 -- 172.21.15.57:0/3540420404 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6c00ee90 con 0x7fed74153ba0 2023-10-30T14:14:19.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed5bfff640 1 -- 172.21.15.57:0/3540420404 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c0149e0 con 0x7fed74153ba0 2023-10-30T14:14:19.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed79a46640 1 -- 172.21.15.57:0/3540420404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed7415b260 con 0x7fed74153ba0 2023-10-30T14:14:19.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed79a46640 1 -- 172.21.15.57:0/3540420404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed7415b820 con 0x7fed74153ba0 2023-10-30T14:14:19.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.440+0000 7fed79a46640 1 -- 172.21.15.57:0/3540420404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed7404e170 con 0x7fed74153ba0 2023-10-30T14:14:19.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.444+0000 7fed5bfff640 1 -- 172.21.15.57:0/3540420404 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fed6c014b40 con 0x7fed74153ba0 2023-10-30T14:14:19.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.444+0000 7fed5bfff640 1 --2- 172.21.15.57:0/3540420404 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fed600664a0 0x7fed60068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.444+0000 7fed5bfff640 1 -- 172.21.15.57:0/3540420404 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed6c0848d0 con 0x7fed74153ba0 2023-10-30T14:14:19.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.444+0000 7fed72ffd640 1 --2- 172.21.15.57:0/3540420404 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fed600664a0 0x7fed60068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.444+0000 7fed72ffd640 1 --2- 172.21.15.57:0/3540420404 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fed600664a0 0x7fed60068960 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fed64005890 tx=0x7fed64009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:19.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.448+0000 7fed5bfff640 1 -- 172.21.15.57:0/3540420404 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fed6c051c20 con 0x7fed74153ba0 2023-10-30T14:14:19.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.608+0000 7fed79a46640 1 -- 172.21.15.57:0/3540420404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fed740cabb0 con 0x7fed74153ba0 2023-10-30T14:14:19.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.608+0000 7fed5bfff640 1 -- 172.21.15.57:0/3540420404 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fed6c055240 con 0x7fed74153ba0 2023-10-30T14:14:19.614 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-30T14:14:19.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.608+0000 7fed79a46640 1 -- 172.21.15.57:0/3540420404 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fed600664a0 msgr2=0x7fed60068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:19.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.608+0000 7fed79a46640 1 --2- 172.21.15.57:0/3540420404 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fed600664a0 0x7fed60068960 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fed64005890 tx=0x7fed64009040 comp rx=0 tx=0).stop 2023-10-30T14:14:19.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.608+0000 7fed79a46640 1 -- 172.21.15.57:0/3540420404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed74153ba0 msgr2=0x7fed740c8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:19.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.608+0000 7fed79a46640 1 --2- 172.21.15.57:0/3540420404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed74153ba0 0x7fed740c8d50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fed6c007e40 tx=0x7fed6c00e4b0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.608+0000 7fed79a46640 1 -- 172.21.15.57:0/3540420404 shutdown_connections 2023-10-30T14:14:19.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.608+0000 7fed79a46640 1 --2- 172.21.15.57:0/3540420404 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fed600664a0 0x7fed60068960 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.608+0000 7fed79a46640 1 --2- 172.21.15.57:0/3540420404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed74153ba0 0x7fed740c8d50 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.608+0000 7fed79a46640 1 --2- 172.21.15.57:0/3540420404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed740c4810 0x7fed7415a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.608+0000 7fed79a46640 1 --2- 172.21.15.57:0/3540420404 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed740c4410 0x7fed741f6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.608+0000 7fed79a46640 1 -- 172.21.15.57:0/3540420404 >> 172.21.15.57:0/3540420404 conn(0x7fed74071790 msgr2=0x7fed741474e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:19.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.608+0000 7fed79a46640 1 -- 172.21.15.57:0/3540420404 shutdown_connections 2023-10-30T14:14:19.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.612+0000 7fed79a46640 1 -- 172.21.15.57:0/3540420404 wait complete. 2023-10-30T14:14:19.635 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:19.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.772+0000 7f598133e640 1 Processor -- start 2023-10-30T14:14:19.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.772+0000 7f598133e640 1 -- start start 2023-10-30T14:14:19.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f598133e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f597c0c59b0 0x7f597c151480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f598133e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f597c1578e0 con 0x7f597c0c55d0 2023-10-30T14:14:19.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f598133e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f597c06ed40 con 0x7f597c0c90c0 2023-10-30T14:14:19.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f598133e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f597c158720 con 0x7f597c0c59b0 2023-10-30T14:14:19.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597affd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f597c0c55d0 0x7f597c06e630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:37782/0 (socket says 172.21.15.57:37782) 2023-10-30T14:14:19.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597affd640 1 -- 172.21.15.57:0/3118022731 learned_addr learned my addr 172.21.15.57:0/3118022731 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:19.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f5973fff640 1 --2- 172.21.15.57:0/3118022731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f597c0c59b0 0x7f597c151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597a7fc640 1 -- 172.21.15.57:0/3118022731 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585014147 0 0) 0x7f597c1578e0 con 0x7f597c0c55d0 2023-10-30T14:14:19.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597a7fc640 1 -- 172.21.15.57:0/3118022731 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5968003530 con 0x7f597c0c55d0 2023-10-30T14:14:19.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597a7fc640 1 -- 172.21.15.57:0/3118022731 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 619613467 0 0) 0x7f5968003530 con 0x7f597c0c55d0 2023-10-30T14:14:19.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597a7fc640 1 -- 172.21.15.57:0/3118022731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f597c0c59b0 msgr2=0x7f597c151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:19.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597a7fc640 1 --2- 172.21.15.57:0/3118022731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f597c0c59b0 0x7f597c151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597a7fc640 1 -- 172.21.15.57:0/3118022731 >> v1:172.21.15.88:6789/0 conn(0x7f597c0c90c0 legacy=0x7f597c15b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:19.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597a7fc640 1 -- 172.21.15.57:0/3118022731 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597c157ba0 con 0x7f597c0c55d0 2023-10-30T14:14:19.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597a7fc640 1 -- 172.21.15.57:0/3118022731 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f596c001680 con 0x7f597c0c55d0 2023-10-30T14:14:19.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597a7fc640 1 -- 172.21.15.57:0/3118022731 >> v1:172.21.15.57:6789/0 conn(0x7f597c0c55d0 legacy=0x7f597c06e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:19.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597a7fc640 1 --2- 172.21.15.57:0/3118022731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59680039b0 0x7f5968003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597a7fc640 1 --2- 172.21.15.57:0/3118022731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5968004360 0x7f5968004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597a7fc640 1 --2- 172.21.15.57:0/3118022731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f597c0c59b0 0x7f5968005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597a7fc640 1 -- 172.21.15.57:0/3118022731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597c158720 con 0x7f59680039b0 2023-10-30T14:14:19.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597a7fc640 1 -- 172.21.15.57:0/3118022731 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596800b9c0 con 0x7f5968004360 2023-10-30T14:14:19.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597a7fc640 1 -- 172.21.15.57:0/3118022731 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596800bb80 con 0x7f597c0c59b0 2023-10-30T14:14:19.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597affd640 1 --2- 172.21.15.57:0/3118022731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59680039b0 0x7f5968003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f5973fff640 1 --2- 172.21.15.57:0/3118022731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5968004360 0x7f5968004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597b7fe640 1 --2- 172.21.15.57:0/3118022731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f597c0c59b0 0x7f5968005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597affd640 1 -- 172.21.15.57:0/3118022731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f597c0c59b0 msgr2=0x7f5968005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:19.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597affd640 1 --2- 172.21.15.57:0/3118022731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f597c0c59b0 0x7f5968005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597affd640 1 -- 172.21.15.57:0/3118022731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5968004360 msgr2=0x7f5968004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:19.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597affd640 1 --2- 172.21.15.57:0/3118022731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5968004360 0x7f5968004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.776+0000 7f597affd640 1 -- 172.21.15.57:0/3118022731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5968003530 con 0x7f59680039b0 2023-10-30T14:14:19.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.780+0000 7f597affd640 1 --2- 172.21.15.57:0/3118022731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59680039b0 0x7f5968003d90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f596c00c210 tx=0x7f596c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2543371e2aa2f231 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:19.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.780+0000 7f597a7fc640 1 -- 172.21.15.57:0/3118022731 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c0a5b60 con 0x7f59680039b0 2023-10-30T14:14:19.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.780+0000 7f597a7fc640 1 -- 172.21.15.57:0/3118022731 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596c0a5ce0 con 0x7f59680039b0 2023-10-30T14:14:19.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.780+0000 7f597a7fc640 1 -- 172.21.15.57:0/3118022731 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c005b20 con 0x7f59680039b0 2023-10-30T14:14:19.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.780+0000 7f598133e640 1 -- 172.21.15.57:0/3118022731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59680039b0 msgr2=0x7f5968003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:19.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.780+0000 7f598133e640 1 --2- 172.21.15.57:0/3118022731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59680039b0 0x7f5968003d90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f596c00c210 tx=0x7f596c01cc60 comp rx=0 tx=0).stop 2023-10-30T14:14:19.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.780+0000 7f597b7fe640 1 -- 172.21.15.57:0/3118022731 reap_dead start 2023-10-30T14:14:19.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.780+0000 7f598133e640 1 -- 172.21.15.57:0/3118022731 shutdown_connections 2023-10-30T14:14:19.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.780+0000 7f598133e640 1 -- 172.21.15.57:0/3118022731 >> 172.21.15.57:0/3118022731 conn(0x7f597c06afa0 msgr2=0x7f597c06b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:19.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.780+0000 7f598133e640 1 -- 172.21.15.57:0/3118022731 shutdown_connections 2023-10-30T14:14:19.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.780+0000 7f598133e640 1 -- 172.21.15.57:0/3118022731 wait complete. 2023-10-30T14:14:19.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f598133e640 1 Processor -- start 2023-10-30T14:14:19.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f598133e640 1 -- start start 2023-10-30T14:14:19.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f598133e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59680039b0 0x7f597c0c4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f598133e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f597c155b50 0x7f597c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f598133e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f597c14cf50 0x7f597c14d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f598133e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597c157320 con 0x7f59680039b0 2023-10-30T14:14:19.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f598133e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597c145e80 con 0x7f597c14cf50 2023-10-30T14:14:19.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f598133e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f597c1564f0 con 0x7f597c155b50 2023-10-30T14:14:19.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f597affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59680039b0 0x7f597c0c4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f597affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59680039b0 0x7f597c0c4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40166/0 (socket says 172.21.15.57:40166) 2023-10-30T14:14:19.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f597affd640 1 -- 172.21.15.57:0/432564522 learned_addr learned my addr 172.21.15.57:0/432564522 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:19.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f597b7fe640 1 --2- 172.21.15.57:0/432564522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f597c14cf50 0x7f597c14d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f5973fff640 1 --2- 172.21.15.57:0/432564522 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f597c155b50 0x7f597c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f597affd640 1 -- 172.21.15.57:0/432564522 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f597c155b50 msgr2=0x7f597c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:19.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f597affd640 1 --2- 172.21.15.57:0/432564522 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f597c155b50 0x7f597c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f597affd640 1 -- 172.21.15.57:0/432564522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f597c14cf50 msgr2=0x7f597c14d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:19.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f597affd640 1 --2- 172.21.15.57:0/432564522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f597c14cf50 0x7f597c14d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f597affd640 1 -- 172.21.15.57:0/432564522 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597c14b7e0 con 0x7f59680039b0 2023-10-30T14:14:19.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f597b7fe640 1 --2- 172.21.15.57:0/432564522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f597c14cf50 0x7f597c14d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:19.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f597affd640 1 --2- 172.21.15.57:0/432564522 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59680039b0 0x7f597c0c4b00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f596c01ca90 tx=0x7f5968005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:19.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f59737fe640 1 -- 172.21.15.57:0/432564522 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c0b4070 con 0x7f59680039b0 2023-10-30T14:14:19.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f59737fe640 1 -- 172.21.15.57:0/432564522 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596c0a5b70 con 0x7f59680039b0 2023-10-30T14:14:19.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f59737fe640 1 -- 172.21.15.57:0/432564522 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c001640 con 0x7f59680039b0 2023-10-30T14:14:19.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f598133e640 1 -- 172.21.15.57:0/432564522 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f597c14bac0 con 0x7f59680039b0 2023-10-30T14:14:19.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f598133e640 1 -- 172.21.15.57:0/432564522 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f597c14c080 con 0x7f59680039b0 2023-10-30T14:14:19.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.784+0000 7f598133e640 1 -- 172.21.15.57:0/432564522 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f597c04e170 con 0x7f59680039b0 2023-10-30T14:14:19.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.788+0000 7f59737fe640 1 -- 172.21.15.57:0/432564522 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f596c007070 con 0x7f59680039b0 2023-10-30T14:14:19.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.788+0000 7f59737fe640 1 --2- 172.21.15.57:0/432564522 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f595c0664d0 0x7f595c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:19.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.788+0000 7f59737fe640 1 -- 172.21.15.57:0/432564522 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f596c11d8c0 con 0x7f59680039b0 2023-10-30T14:14:19.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.788+0000 7f5973fff640 1 --2- 172.21.15.57:0/432564522 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f595c0664d0 0x7f595c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:19.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.788+0000 7f5973fff640 1 --2- 172.21.15.57:0/432564522 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f595c0664d0 0x7f595c068990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f597c155140 tx=0x7f5964008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:19.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.792+0000 7f59737fe640 1 -- 172.21.15.57:0/432564522 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f596c0eac10 con 0x7f59680039b0 2023-10-30T14:14:19.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.984+0000 7f598133e640 1 -- 172.21.15.57:0/432564522 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f597c158130 con 0x7f59680039b0 2023-10-30T14:14:19.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.984+0000 7f59737fe640 1 -- 172.21.15.57:0/432564522 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+6518 (secure 0 0 0) 0x7f596c0bf090 con 0x7f59680039b0 2023-10-30T14:14:19.992 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:19.992 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:14:15.802770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"e","rank":2,"incarnation":11,"state":"up:creating","state_seq":1,"addr":"172.21.15.57:6837/901968115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":901968115},{"type":"v1","addr":"172.21.15.57:6837","nonce":901968115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:19.993 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 11 2023-10-30T14:14:19.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.988+0000 7f598133e640 1 -- 172.21.15.57:0/432564522 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f595c0664d0 msgr2=0x7f595c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:19.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.988+0000 7f598133e640 1 --2- 172.21.15.57:0/432564522 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f595c0664d0 0x7f595c068990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f597c155140 tx=0x7f5964008040 comp rx=0 tx=0).stop 2023-10-30T14:14:19.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.988+0000 7f598133e640 1 -- 172.21.15.57:0/432564522 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59680039b0 msgr2=0x7f597c0c4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:19.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.988+0000 7f598133e640 1 --2- 172.21.15.57:0/432564522 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59680039b0 0x7f597c0c4b00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f596c01ca90 tx=0x7f5968005800 comp rx=0 tx=0).stop 2023-10-30T14:14:19.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.988+0000 7f598133e640 1 -- 172.21.15.57:0/432564522 shutdown_connections 2023-10-30T14:14:19.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.988+0000 7f598133e640 1 --2- 172.21.15.57:0/432564522 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f595c0664d0 0x7f595c068990 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.988+0000 7f598133e640 1 --2- 172.21.15.57:0/432564522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f597c14cf50 0x7f597c14d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.988+0000 7f598133e640 1 --2- 172.21.15.57:0/432564522 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f597c155b50 0x7f597c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.988+0000 7f598133e640 1 --2- 172.21.15.57:0/432564522 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59680039b0 0x7f597c0c4b00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:19.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.988+0000 7f598133e640 1 -- 172.21.15.57:0/432564522 >> 172.21.15.57:0/432564522 conn(0x7f597c06afa0 msgr2=0x7f597c14f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:19.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.988+0000 7f598133e640 1 -- 172.21.15.57:0/432564522 shutdown_connections 2023-10-30T14:14:19.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:19.988+0000 7f598133e640 1 -- 172.21.15.57:0/432564522 wait complete. 2023-10-30T14:14:20.012 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:14:20.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.148+0000 7fd0fc46b640 1 Processor -- start 2023-10-30T14:14:20.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.148+0000 7fd0fc46b640 1 -- start start 2023-10-30T14:14:20.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.152+0000 7fd0fc46b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0f4153c80 0x7fd0f415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:20.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.152+0000 7fd0fc46b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd0f40c4a10 0x7fd0f40c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:20.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.152+0000 7fd0fc46b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0f4159f20 con 0x7fd0f415a3b0 2023-10-30T14:14:20.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.152+0000 7fd0fc46b640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0f415dec0 con 0x7fd0f40c4a10 2023-10-30T14:14:20.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.152+0000 7fd0fc46b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f41599a0 con 0x7fd0f4153c80 2023-10-30T14:14:20.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.152+0000 7fd0f9a06640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd0f40c4a10 0x7fd0f40c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:20.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.152+0000 7fd0faa08640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fd0f415a3b0 0x7fd0f40ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:37792/0 (socket says 172.21.15.57:37792) 2023-10-30T14:14:20.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.152+0000 7fd0faa08640 1 -- 172.21.15.57:0/79406166 learned_addr learned my addr 172.21.15.57:0/79406166 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:20.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.152+0000 7fd0fa207640 1 --2- 172.21.15.57:0/79406166 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0f4153c80 0x7fd0f415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:20.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.152+0000 7fd0f9a06640 1 -- 172.21.15.57:0/79406166 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0f4153c80 msgr2=0x7fd0f415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:20.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.152+0000 7fd0f9a06640 1 --2- 172.21.15.57:0/79406166 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0f4153c80 0x7fd0f415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:20.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.152+0000 7fd0f9a06640 1 -- 172.21.15.57:0/79406166 >> v1:172.21.15.57:6789/0 conn(0x7fd0f415a3b0 legacy=0x7fd0f40ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:20.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.152+0000 7fd0f9a06640 1 -- 172.21.15.57:0/79406166 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f40c47e0 con 0x7fd0f40c4a10 2023-10-30T14:14:20.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.152+0000 7fd0f9205640 1 -- 172.21.15.57:0/79406166 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307698863 0 0) 0x7fd0f4159f20 con 0x7fd0f415a3b0 2023-10-30T14:14:20.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.156+0000 7fd0f9a06640 1 --2- 172.21.15.57:0/79406166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f40c4a10 0x7fd0f40c42a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40097c0 tx=0x7fd0e401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94323875896ad605 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:20.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.156+0000 7fd0f9205640 1 -- 172.21.15.57:0/79406166 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e40a8970 con 0x7fd0f40c4a10 2023-10-30T14:14:20.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.156+0000 7fd0f9205640 1 -- 172.21.15.57:0/79406166 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e40a8ad0 con 0x7fd0f40c4a10 2023-10-30T14:14:20.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.156+0000 7fd0f9205640 1 -- 172.21.15.57:0/79406166 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e40a8e60 con 0x7fd0f40c4a10 2023-10-30T14:14:20.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.156+0000 7fd0fc46b640 1 -- 172.21.15.57:0/79406166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f40c4a10 msgr2=0x7fd0f40c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:20.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.156+0000 7fd0fc46b640 1 --2- 172.21.15.57:0/79406166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f40c4a10 0x7fd0f40c42a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40097c0 tx=0x7fd0e401ec70 comp rx=0 tx=0).stop 2023-10-30T14:14:20.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.156+0000 7fd0fc46b640 1 -- 172.21.15.57:0/79406166 shutdown_connections 2023-10-30T14:14:20.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.156+0000 7fd0fc46b640 1 --2- 172.21.15.57:0/79406166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f40c4a10 0x7fd0f40c42a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:20.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.156+0000 7fd0fc46b640 1 --2- 172.21.15.57:0/79406166 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0f4153c80 0x7fd0f415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:20.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.156+0000 7fd0fc46b640 1 -- 172.21.15.57:0/79406166 >> 172.21.15.57:0/79406166 conn(0x7fd0f40715c0 msgr2=0x7fd0f40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:20.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.156+0000 7fd0fc46b640 1 -- 172.21.15.57:0/79406166 shutdown_connections 2023-10-30T14:14:20.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.156+0000 7fd0fc46b640 1 -- 172.21.15.57:0/79406166 wait complete. 2023-10-30T14:14:20.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.156+0000 7fd0fc46b640 1 Processor -- start 2023-10-30T14:14:20.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.156+0000 7fd0fc46b640 1 -- start start 2023-10-30T14:14:20.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0fc46b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0f40c4a10 0x7fd0f40ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:20.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0fc46b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0f4153c80 0x7fd0f40b4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:20.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0fc46b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f415a3b0 0x7fd0f40b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:20.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0fc46b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f40b4070 con 0x7fd0f415a3b0 2023-10-30T14:14:20.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0fc46b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f40b3180 con 0x7fd0f40c4a10 2023-10-30T14:14:20.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0fc46b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f40b3460 con 0x7fd0f4153c80 2023-10-30T14:14:20.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0faa08640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f415a3b0 0x7fd0f40b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:20.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0faa08640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f415a3b0 0x7fd0f40b4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40180/0 (socket says 172.21.15.57:40180) 2023-10-30T14:14:20.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0fa207640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0f40c4a10 0x7fd0f40ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:20.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0faa08640 1 -- 172.21.15.57:0/142223092 learned_addr learned my addr 172.21.15.57:0/142223092 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:20.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0f9a06640 1 --2- 172.21.15.57:0/142223092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0f4153c80 0x7fd0f40b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:20.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0faa08640 1 -- 172.21.15.57:0/142223092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0f4153c80 msgr2=0x7fd0f40b4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:20.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0faa08640 1 --2- 172.21.15.57:0/142223092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0f4153c80 0x7fd0f40b4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:20.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0faa08640 1 -- 172.21.15.57:0/142223092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0f40c4a10 msgr2=0x7fd0f40ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:20.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0faa08640 1 --2- 172.21.15.57:0/142223092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0f40c4a10 0x7fd0f40ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:20.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0faa08640 1 -- 172.21.15.57:0/142223092 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e4009420 con 0x7fd0f415a3b0 2023-10-30T14:14:20.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0faa08640 1 --2- 172.21.15.57:0/142223092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f415a3b0 0x7fd0f40b4ec0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec00c780 tx=0x7fd0ec00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:20.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0eaffd640 1 -- 172.21.15.57:0/142223092 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec00ce90 con 0x7fd0f415a3b0 2023-10-30T14:14:20.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0eaffd640 1 -- 172.21.15.57:0/142223092 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0ec00a640 con 0x7fd0f415a3b0 2023-10-30T14:14:20.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0eaffd640 1 -- 172.21.15.57:0/142223092 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec00a9d0 con 0x7fd0f415a3b0 2023-10-30T14:14:20.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0fc46b640 1 -- 172.21.15.57:0/142223092 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f40b3740 con 0x7fd0f415a3b0 2023-10-30T14:14:20.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0fc46b640 1 -- 172.21.15.57:0/142223092 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f40b3d00 con 0x7fd0f415a3b0 2023-10-30T14:14:20.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0fc46b640 1 -- 172.21.15.57:0/142223092 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f40b4220 con 0x7fd0f415a3b0 2023-10-30T14:14:20.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.160+0000 7fd0eaffd640 1 -- 172.21.15.57:0/142223092 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd0ec00ab30 con 0x7fd0f415a3b0 2023-10-30T14:14:20.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.164+0000 7fd0eaffd640 1 --2- 172.21.15.57:0/142223092 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0d80664a0 0x7fd0d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:20.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.164+0000 7fd0eaffd640 1 -- 172.21.15.57:0/142223092 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0ec083450 con 0x7fd0f415a3b0 2023-10-30T14:14:20.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.164+0000 7fd0fa207640 1 --2- 172.21.15.57:0/142223092 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0d80664a0 0x7fd0d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:20.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.164+0000 7fd0fa207640 1 --2- 172.21.15.57:0/142223092 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0d80664a0 0x7fd0d8068960 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0007a60 tx=0x7fd0f0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:20.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.168+0000 7fd0eaffd640 1 -- 172.21.15.57:0/142223092 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd0f40b4220 con 0x7fd0f415a3b0 2023-10-30T14:14:20.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.324+0000 7fd0fc46b640 1 -- 172.21.15.57:0/142223092 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0f40b4220 con 0x7fd0f415a3b0 2023-10-30T14:14:20.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.324+0000 7fd0eaffd640 1 -- 172.21.15.57:0/142223092 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16433 (secure 0 0 0) 0x7fd0f40b4220 con 0x7fd0f415a3b0 2023-10-30T14:14:20.331 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:20.331 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:14:05.576818+0000","last_up_change":"2023-10-30T14:13:51.171287+0000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T14:13:52.574639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T14:14:02.600122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T14:14:03.069429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6801","nonce":3287367848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6803","nonce":3287367848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6807","nonce":3287367848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6805","nonce":3287367848}]},"public_addr":"172.21.15.57:6801/3287367848","cluster_addr":"172.21.15.57:6803/3287367848","heartbeat_back_addr":"172.21.15.57:6807/3287367848","heartbeat_front_addr":"172.21.15.57:6805/3287367848","state":["exists","up"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6809","nonce":3340797836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6811","nonce":3340797836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6815","nonce":3340797836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6813","nonce":3340797836}]},"public_addr":"172.21.15.57:6809/3340797836","cluster_addr":"172.21.15.57:6811/3340797836","heartbeat_back_addr":"172.21.15.57:6815/3340797836","heartbeat_front_addr":"172.21.15.57:6813/3340797836","state":["exists","up"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6825","nonce":1700292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6827","nonce":1700292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6831","nonce":1700292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6829","nonce":1700292}]},"public_addr":"172.21.15.57:6825/1700292","cluster_addr":"172.21.15.57:6827/1700292","heartbeat_back_addr":"172.21.15.57:6831/1700292","heartbeat_front_addr":"172.21.15.57:6829/1700292","state":["exists","up"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6817","nonce":2893420631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6819","nonce":2893420631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6823","nonce":2893420631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6821","nonce":2893420631}]},"public_addr":"172.21.15.57:6817/2893420631","cluster_addr":"172.21.15.57:6819/2893420631","heartbeat_back_addr":"172.21.15.57:6823/2893420631","heartbeat_front_addr":"172.21.15.57:6821/2893420631","state":["exists","up"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6807","nonce":2784816485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6811","nonce":2784816485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6815","nonce":2784816485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6813","nonce":2784816485}]},"public_addr":"172.21.15.88:6807/2784816485","cluster_addr":"172.21.15.88:6811/2784816485","heartbeat_back_addr":"172.21.15.88:6815/2784816485","heartbeat_front_addr":"172.21.15.88:6813/2784816485","state":["exists","up"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6801","nonce":2682449812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6803","nonce":2682449812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6810","nonce":2682449812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6806","nonce":2682449812}]},"public_addr":"172.21.15.88:6801/2682449812","cluster_addr":"172.21.15.88:6803/2682449812","heartbeat_back_addr":"172.21.15.88:6810/2682449812","heartbeat_front_addr":"172.21.15.88:6806/2682449812","state":["exists","up"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6822","nonce":3340693897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6827","nonce":3340693897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6831","nonce":3340693897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6829","nonce":3340693897}]},"public_addr":"172.21.15.88:6822/3340693897","cluster_addr":"172.21.15.88:6827/3340693897","heartbeat_back_addr":"172.21.15.88:6831/3340693897","heartbeat_front_addr":"172.21.15.88:6829/3340693897","state":["exists","up"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6817","nonce":2862111199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6820","nonce":2862111199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6825","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6826","nonce":2862111199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6821","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6823","nonce":2862111199}]},"public_addr":"172.21.15.88:6817/2862111199","cluster_addr":"172.21.15.88:6820/2862111199","heartbeat_back_addr":"172.21.15.88:6826/2862111199","heartbeat_front_addr":"172.21.15.88:6823/2862111199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.634920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.773993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.725825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.761452+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.694274+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.650718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:14:20.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.328+0000 7fd0fc46b640 1 -- 172.21.15.57:0/142223092 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0d80664a0 msgr2=0x7fd0d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:20.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.328+0000 7fd0fc46b640 1 --2- 172.21.15.57:0/142223092 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0d80664a0 0x7fd0d8068960 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0007a60 tx=0x7fd0f0008040 comp rx=0 tx=0).stop 2023-10-30T14:14:20.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.328+0000 7fd0fc46b640 1 -- 172.21.15.57:0/142223092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f415a3b0 msgr2=0x7fd0f40b4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:20.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.328+0000 7fd0fc46b640 1 --2- 172.21.15.57:0/142223092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f415a3b0 0x7fd0f40b4ec0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec00c780 tx=0x7fd0ec00cc40 comp rx=0 tx=0).stop 2023-10-30T14:14:20.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.328+0000 7fd0fc46b640 1 -- 172.21.15.57:0/142223092 shutdown_connections 2023-10-30T14:14:20.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.328+0000 7fd0fc46b640 1 --2- 172.21.15.57:0/142223092 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0d80664a0 0x7fd0d8068960 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:20.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.328+0000 7fd0fc46b640 1 --2- 172.21.15.57:0/142223092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f415a3b0 0x7fd0f40b4ec0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:20.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.328+0000 7fd0fc46b640 1 --2- 172.21.15.57:0/142223092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0f4153c80 0x7fd0f40b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:20.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.328+0000 7fd0fc46b640 1 --2- 172.21.15.57:0/142223092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0f40c4a10 0x7fd0f40ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:20.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.328+0000 7fd0fc46b640 1 -- 172.21.15.57:0/142223092 >> 172.21.15.57:0/142223092 conn(0x7fd0f40715c0 msgr2=0x7fd0f4157a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:20.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.328+0000 7fd0fc46b640 1 -- 172.21.15.57:0/142223092 shutdown_connections 2023-10-30T14:14:20.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.328+0000 7fd0fc46b640 1 -- 172.21.15.57:0/142223092 wait complete. 2023-10-30T14:14:20.352 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:20.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.492+0000 7fbe8dfc4640 1 Processor -- start 2023-10-30T14:14:20.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.492+0000 7fbe8dfc4640 1 -- start start 2023-10-30T14:14:20.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.492+0000 7fbe8dfc4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbe88153c80 0x7fbe8815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:20.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.492+0000 7fbe8dfc4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe88159f20 con 0x7fbe880c4a10 2023-10-30T14:14:20.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.492+0000 7fbe8dfc4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe8815d620 con 0x7fbe880c42a0 2023-10-30T14:14:20.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.492+0000 7fbe8dfc4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe8815dec0 con 0x7fbe88153c80 2023-10-30T14:14:20.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.492+0000 7fbe86ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fbe880c4a10 0x7fbe8815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:37800/0 (socket says 172.21.15.57:37800) 2023-10-30T14:14:20.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.492+0000 7fbe86ffd640 1 -- 172.21.15.57:0/3029383763 learned_addr learned my addr 172.21.15.57:0/3029383763 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:20.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.492+0000 7fbe877fe640 1 --2- 172.21.15.57:0/3029383763 >> v2:172.21.15.88:3300/0 conn(0x7fbe88153c80 0x7fbe8815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:20.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.492+0000 7fbe867fc640 1 -- 172.21.15.57:0/3029383763 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384682313 0 0) 0x7fbe88159f20 con 0x7fbe880c4a10 2023-10-30T14:14:20.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.492+0000 7fbe867fc640 1 -- 172.21.15.57:0/3029383763 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe70003530 con 0x7fbe880c4a10 2023-10-30T14:14:20.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.492+0000 7fbe877fe640 1 -- 172.21.15.57:0/3029383763 >> v1:172.21.15.88:6789/0 conn(0x7fbe880c42a0 legacy=0x7fbe880ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:20.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.492+0000 7fbe867fc640 1 -- 172.21.15.57:0/3029383763 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3735043970 0 0) 0x7fbe70003530 con 0x7fbe880c4a10 2023-10-30T14:14:20.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.492+0000 7fbe877fe640 1 -- 172.21.15.57:0/3029383763 >> v1:172.21.15.57:6789/0 conn(0x7fbe880c4a10 legacy=0x7fbe8815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:20.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.492+0000 7fbe877fe640 1 -- 172.21.15.57:0/3029383763 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8815d900 con 0x7fbe88153c80 2023-10-30T14:14:20.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.496+0000 7fbe877fe640 1 --2- 172.21.15.57:0/3029383763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe88153c80 0x7fbe8815cf90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbe780097c0 tx=0x7fbe7801c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a00c865aac61d705 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:20.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.496+0000 7fbe867fc640 1 -- 172.21.15.57:0/3029383763 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c001680 con 0x7fbe88153c80 2023-10-30T14:14:20.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.496+0000 7fbe867fc640 1 -- 172.21.15.57:0/3029383763 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe780a7050 con 0x7fbe88153c80 2023-10-30T14:14:20.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.496+0000 7fbe867fc640 1 -- 172.21.15.57:0/3029383763 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe780ab620 con 0x7fbe88153c80 2023-10-30T14:14:20.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.496+0000 7fbe8dfc4640 1 -- 172.21.15.57:0/3029383763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe88153c80 msgr2=0x7fbe8815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:20.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.496+0000 7fbe8dfc4640 1 --2- 172.21.15.57:0/3029383763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe88153c80 0x7fbe8815cf90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbe780097c0 tx=0x7fbe7801c2e0 comp rx=0 tx=0).stop 2023-10-30T14:14:20.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.496+0000 7fbe8dfc4640 1 -- 172.21.15.57:0/3029383763 shutdown_connections 2023-10-30T14:14:20.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.496+0000 7fbe8dfc4640 1 --2- 172.21.15.57:0/3029383763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe88153c80 0x7fbe8815cf90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:20.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.496+0000 7fbe8dfc4640 1 -- 172.21.15.57:0/3029383763 >> 172.21.15.57:0/3029383763 conn(0x7fbe880715c0 msgr2=0x7fbe880719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:20.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.496+0000 7fbe8dfc4640 1 -- 172.21.15.57:0/3029383763 shutdown_connections 2023-10-30T14:14:20.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.496+0000 7fbe8dfc4640 1 -- 172.21.15.57:0/3029383763 wait complete. 2023-10-30T14:14:20.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe8dfc4640 1 Processor -- start 2023-10-30T14:14:20.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe8dfc4640 1 -- start start 2023-10-30T14:14:20.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe8dfc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe880c42a0 0x7fbe8815ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:20.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe8dfc4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe880c4a10 0x7fbe8815b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:20.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe8dfc4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe88153c80 0x7fbe8814f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:20.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe8dfc4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8814e570 con 0x7fbe88153c80 2023-10-30T14:14:20.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe8dfc4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8814ad80 con 0x7fbe880c42a0 2023-10-30T14:14:20.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe8dfc4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8814aa50 con 0x7fbe880c4a10 2023-10-30T14:14:20.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe87fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe88153c80 0x7fbe8814f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:20.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe877fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe880c42a0 0x7fbe8815ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:20.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe87fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe88153c80 0x7fbe8814f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40194/0 (socket says 172.21.15.57:40194) 2023-10-30T14:14:20.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe87fff640 1 -- 172.21.15.57:0/2109599116 learned_addr learned my addr 172.21.15.57:0/2109599116 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:20.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe86ffd640 1 --2- 172.21.15.57:0/2109599116 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe880c4a10 0x7fbe8815b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:20.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe87fff640 1 -- 172.21.15.57:0/2109599116 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe880c4a10 msgr2=0x7fbe8815b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:20.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe87fff640 1 --2- 172.21.15.57:0/2109599116 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe880c4a10 0x7fbe8815b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:20.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe87fff640 1 -- 172.21.15.57:0/2109599116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe880c42a0 msgr2=0x7fbe8815ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:20.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe87fff640 1 --2- 172.21.15.57:0/2109599116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe880c42a0 0x7fbe8815ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:20.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe87fff640 1 -- 172.21.15.57:0/2109599116 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe78009420 con 0x7fbe88153c80 2023-10-30T14:14:20.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe87fff640 1 --2- 172.21.15.57:0/2109599116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe88153c80 0x7fbe8814f950 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbe8000c160 tx=0x7fbe8000c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:20.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe6ffff640 1 -- 172.21.15.57:0/2109599116 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe80009050 con 0x7fbe88153c80 2023-10-30T14:14:20.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe6ffff640 1 -- 172.21.15.57:0/2109599116 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe800091b0 con 0x7fbe88153c80 2023-10-30T14:14:20.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe6ffff640 1 -- 172.21.15.57:0/2109599116 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe80009560 con 0x7fbe88153c80 2023-10-30T14:14:20.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe8dfc4640 1 -- 172.21.15.57:0/2109599116 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe8815d090 con 0x7fbe88153c80 2023-10-30T14:14:20.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.500+0000 7fbe8dfc4640 1 -- 172.21.15.57:0/2109599116 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe881f0050 con 0x7fbe88153c80 2023-10-30T14:14:20.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.504+0000 7fbe8dfc4640 1 -- 172.21.15.57:0/2109599116 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe8804e170 con 0x7fbe88153c80 2023-10-30T14:14:20.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.504+0000 7fbe6ffff640 1 -- 172.21.15.57:0/2109599116 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbe800096c0 con 0x7fbe88153c80 2023-10-30T14:14:20.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.504+0000 7fbe6ffff640 1 --2- 172.21.15.57:0/2109599116 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbe740664a0 0x7fbe74068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:20.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.504+0000 7fbe6ffff640 1 -- 172.21.15.57:0/2109599116 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe80085dd0 con 0x7fbe88153c80 2023-10-30T14:14:20.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.504+0000 7fbe877fe640 1 --2- 172.21.15.57:0/2109599116 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbe740664a0 0x7fbe74068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:20.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.504+0000 7fbe877fe640 1 --2- 172.21.15.57:0/2109599116 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbe740664a0 0x7fbe74068960 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbe780ad000 tx=0x7fbe780a6000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:20.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.508+0000 7fbe6ffff640 1 -- 172.21.15.57:0/2109599116 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbe80053270 con 0x7fbe88153c80 2023-10-30T14:14:20.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.700+0000 7fbe8dfc4640 1 -- 172.21.15.57:0/2109599116 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe8814aec0 con 0x7fbe88153c80 2023-10-30T14:14:20.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.704+0000 7fbe6ffff640 1 -- 172.21.15.57:0/2109599116 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+6518 (secure 0 0 0) 0x7fbe80087020 con 0x7fbe88153c80 2023-10-30T14:14:20.708 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:20.708 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:14:15.802770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"e","rank":2,"incarnation":11,"state":"up:creating","state_seq":1,"addr":"172.21.15.57:6837/901968115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":901968115},{"type":"v1","addr":"172.21.15.57:6837","nonce":901968115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:20.709 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 11 2023-10-30T14:14:20.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.704+0000 7fbe8dfc4640 1 -- 172.21.15.57:0/2109599116 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbe740664a0 msgr2=0x7fbe74068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:20.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.704+0000 7fbe8dfc4640 1 --2- 172.21.15.57:0/2109599116 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbe740664a0 0x7fbe74068960 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbe780ad000 tx=0x7fbe780a6000 comp rx=0 tx=0).stop 2023-10-30T14:14:20.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.704+0000 7fbe8dfc4640 1 -- 172.21.15.57:0/2109599116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe88153c80 msgr2=0x7fbe8814f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:20.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.704+0000 7fbe8dfc4640 1 --2- 172.21.15.57:0/2109599116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe88153c80 0x7fbe8814f950 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbe8000c160 tx=0x7fbe8000c620 comp rx=0 tx=0).stop 2023-10-30T14:14:20.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.704+0000 7fbe8dfc4640 1 -- 172.21.15.57:0/2109599116 shutdown_connections 2023-10-30T14:14:20.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.704+0000 7fbe8dfc4640 1 --2- 172.21.15.57:0/2109599116 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbe740664a0 0x7fbe74068960 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:20.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.704+0000 7fbe8dfc4640 1 --2- 172.21.15.57:0/2109599116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe88153c80 0x7fbe8814f950 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:20.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.704+0000 7fbe8dfc4640 1 --2- 172.21.15.57:0/2109599116 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe880c4a10 0x7fbe8815b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:20.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.704+0000 7fbe8dfc4640 1 --2- 172.21.15.57:0/2109599116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe880c42a0 0x7fbe8815ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:20.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.704+0000 7fbe8dfc4640 1 -- 172.21.15.57:0/2109599116 >> 172.21.15.57:0/2109599116 conn(0x7fbe880715c0 msgr2=0x7fbe88151d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:20.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.704+0000 7fbe8dfc4640 1 -- 172.21.15.57:0/2109599116 shutdown_connections 2023-10-30T14:14:20.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:20.704+0000 7fbe8dfc4640 1 -- 172.21.15.57:0/2109599116 wait complete. 2023-10-30T14:14:21.332 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 11, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-30T14:14:04.087067+0000', 'modified': '2023-10-30T14:14:15.802770+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 4282, 'mds_1': 4266, 'mds_2': 4265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 2, 'incarnation': 11, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.57:6837/901968115', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6833', 'nonce': 901968115}, {'type': 'v1', 'addr': '172.21.15.57:6837', 'nonce': 901968115}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4266': {'gid': 4266, 'name': 'c', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6836/2693548867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6835', 'nonce': 2693548867}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 2693548867}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4282': {'gid': 4282, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.88:6839/2945854794', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6838', 'nonce': 2945854794}, {'type': 'v1', 'addr': '172.21.15.88:6839', 'nonce': 2945854794}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-30T14:14:21.333 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4265:up:creating 2023-10-30T14:14:22.334 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:22.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6afc479640 1 Processor -- start 2023-10-30T14:14:22.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6afc479640 1 -- start start 2023-10-30T14:14:22.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6afc479640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6af40c4a10 0x7f6af415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:22.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6afc479640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6af4159f20 con 0x7f6af40c42a0 2023-10-30T14:14:22.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6afc479640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6af415d620 con 0x7f6af4153c80 2023-10-30T14:14:22.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6afc479640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af415dec0 con 0x7f6af40c4a10 2023-10-30T14:14:22.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6afaa16640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f6af40c42a0 0x7f6af40ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:37816/0 (socket says 172.21.15.57:37816) 2023-10-30T14:14:22.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6afaa16640 1 -- 172.21.15.57:0/3821580803 learned_addr learned my addr 172.21.15.57:0/3821580803 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:22.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6af9a14640 1 --2- 172.21.15.57:0/3821580803 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6af40c4a10 0x7f6af415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:22.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6af9213640 1 -- 172.21.15.57:0/3821580803 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120189788 0 0) 0x7f6af4159f20 con 0x7f6af40c42a0 2023-10-30T14:14:22.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6af9213640 1 -- 172.21.15.57:0/3821580803 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae0003530 con 0x7f6af40c42a0 2023-10-30T14:14:22.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6af9213640 1 -- 172.21.15.57:0/3821580803 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831859470 0 0) 0x7f6af415d620 con 0x7f6af4153c80 2023-10-30T14:14:22.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6af9213640 1 -- 172.21.15.57:0/3821580803 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6af4159f20 con 0x7f6af4153c80 2023-10-30T14:14:22.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6af9213640 1 -- 172.21.15.57:0/3821580803 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2185303765 0 0) 0x7f6ae0003530 con 0x7f6af40c42a0 2023-10-30T14:14:22.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6af9213640 1 -- 172.21.15.57:0/3821580803 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6af40c4a10 msgr2=0x7f6af415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:22.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6af9213640 1 --2- 172.21.15.57:0/3821580803 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6af40c4a10 0x7f6af415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:22.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6af9213640 1 -- 172.21.15.57:0/3821580803 >> v1:172.21.15.88:6789/0 conn(0x7f6af4153c80 legacy=0x7f6af415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:22.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6af9213640 1 -- 172.21.15.57:0/3821580803 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af415d900 con 0x7f6af40c42a0 2023-10-30T14:14:22.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6af9213640 1 -- 172.21.15.57:0/3821580803 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f6aec001a00 con 0x7f6af40c42a0 2023-10-30T14:14:22.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6af9213640 1 -- 172.21.15.57:0/3821580803 >> v1:172.21.15.57:6789/0 conn(0x7f6af40c42a0 legacy=0x7f6af40ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:22.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6af9213640 1 --2- 172.21.15.57:0/3821580803 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ae0003b40 0x7f6ae0003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:22.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6af9213640 1 --2- 172.21.15.57:0/3821580803 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6af40c4a10 0x7f6ae00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:22.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6af9213640 1 --2- 172.21.15.57:0/3821580803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ae0004d10 0x7f6ae0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:22.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6af9213640 1 -- 172.21.15.57:0/3821580803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af415dec0 con 0x7f6ae0004d10 2023-10-30T14:14:22.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6af9213640 1 -- 172.21.15.57:0/3821580803 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae000ba40 con 0x7f6ae0003b40 2023-10-30T14:14:22.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6af9213640 1 -- 172.21.15.57:0/3821580803 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae000bbb0 con 0x7f6af40c4a10 2023-10-30T14:14:22.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6afaa16640 1 --2- 172.21.15.57:0/3821580803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ae0004d10 0x7f6ae0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:22.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6af9a14640 1 --2- 172.21.15.57:0/3821580803 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6af40c4a10 0x7f6ae00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:22.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6afa215640 1 --2- 172.21.15.57:0/3821580803 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ae0003b40 0x7f6ae0003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6afaa16640 1 -- 172.21.15.57:0/3821580803 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6af40c4a10 msgr2=0x7f6ae00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6afaa16640 1 --2- 172.21.15.57:0/3821580803 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6af40c4a10 0x7f6ae00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6afaa16640 1 -- 172.21.15.57:0/3821580803 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ae0003b40 msgr2=0x7f6ae0003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6afaa16640 1 --2- 172.21.15.57:0/3821580803 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ae0003b40 0x7f6ae0003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.484+0000 7f6afaa16640 1 -- 172.21.15.57:0/3821580803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae0003530 con 0x7f6ae0004d10 2023-10-30T14:14:22.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.488+0000 7f6afaa16640 1 --2- 172.21.15.57:0/3821580803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ae0004d10 0x7f6ae0005190 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6aec00a7c0 tx=0x7f6aec01d2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20efdf89c4ba9f54 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:22.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.488+0000 7f6af9213640 1 -- 172.21.15.57:0/3821580803 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec0ad630 con 0x7f6ae0004d10 2023-10-30T14:14:22.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.488+0000 7f6af9213640 1 -- 172.21.15.57:0/3821580803 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aec0a8070 con 0x7f6ae0004d10 2023-10-30T14:14:22.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.488+0000 7f6af9213640 1 -- 172.21.15.57:0/3821580803 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec0ad9f0 con 0x7f6ae0004d10 2023-10-30T14:14:22.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.488+0000 7f6afc479640 1 -- 172.21.15.57:0/3821580803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ae0004d10 msgr2=0x7f6ae0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:22.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.488+0000 7f6afc479640 1 --2- 172.21.15.57:0/3821580803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ae0004d10 0x7f6ae0005190 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6aec00a7c0 tx=0x7f6aec01d2a0 comp rx=0 tx=0).stop 2023-10-30T14:14:22.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.488+0000 7f6afaa16640 1 -- 172.21.15.57:0/3821580803 reap_dead start 2023-10-30T14:14:22.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.488+0000 7f6afc479640 1 -- 172.21.15.57:0/3821580803 shutdown_connections 2023-10-30T14:14:22.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.488+0000 7f6afc479640 1 -- 172.21.15.57:0/3821580803 >> 172.21.15.57:0/3821580803 conn(0x7f6af40715c0 msgr2=0x7f6af40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:22.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.488+0000 7f6afc479640 1 -- 172.21.15.57:0/3821580803 shutdown_connections 2023-10-30T14:14:22.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.488+0000 7f6afc479640 1 -- 172.21.15.57:0/3821580803 wait complete. 2023-10-30T14:14:22.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afc479640 1 Processor -- start 2023-10-30T14:14:22.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afc479640 1 -- start start 2023-10-30T14:14:22.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afc479640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ae0004d10 0x7f6af415c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:22.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afc479640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6af41f1cd0 0x7f6af41f11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:22.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afc479640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6af41f1710 0x7f6af41edc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:22.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afc479640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af4152480 con 0x7f6af41f1710 2023-10-30T14:14:22.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afc479640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af415cfa0 con 0x7f6af41f1cd0 2023-10-30T14:14:22.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afc479640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af41efbd0 con 0x7f6ae0004d10 2023-10-30T14:14:22.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afaa16640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6af41f1710 0x7f6af41edc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:22.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afaa16640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6af41f1710 0x7f6af41edc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40206/0 (socket says 172.21.15.57:40206) 2023-10-30T14:14:22.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afaa16640 1 -- 172.21.15.57:0/1456751626 learned_addr learned my addr 172.21.15.57:0/1456751626 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:22.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6af9a14640 1 --2- 172.21.15.57:0/1456751626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6af41f1cd0 0x7f6af41f11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:22.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afa215640 1 --2- 172.21.15.57:0/1456751626 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ae0004d10 0x7f6af415c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:22.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afaa16640 1 -- 172.21.15.57:0/1456751626 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ae0004d10 msgr2=0x7f6af415c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:22.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afaa16640 1 --2- 172.21.15.57:0/1456751626 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ae0004d10 0x7f6af415c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:22.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afaa16640 1 -- 172.21.15.57:0/1456751626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6af41f1cd0 msgr2=0x7f6af41f11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:22.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afaa16640 1 --2- 172.21.15.57:0/1456751626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6af41f1cd0 0x7f6af41f11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:22.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afaa16640 1 -- 172.21.15.57:0/1456751626 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af41efe70 con 0x7f6af41f1710 2023-10-30T14:14:22.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afa215640 1 --2- 172.21.15.57:0/1456751626 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ae0004d10 0x7f6af415c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:22.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afaa16640 1 --2- 172.21.15.57:0/1456751626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6af41f1710 0x7f6af41edc10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6aec01d0d0 tx=0x7f6ae0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:22.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6aeaffd640 1 -- 172.21.15.57:0/1456751626 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec001f60 con 0x7f6af41f1710 2023-10-30T14:14:22.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6aeaffd640 1 -- 172.21.15.57:0/1456751626 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aec001800 con 0x7f6af41f1710 2023-10-30T14:14:22.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afc479640 1 -- 172.21.15.57:0/1456751626 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af41f0130 con 0x7f6af41f1710 2023-10-30T14:14:22.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6aeaffd640 1 -- 172.21.15.57:0/1456751626 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec004d70 con 0x7f6af41f1710 2023-10-30T14:14:22.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.492+0000 7f6afc479640 1 -- 172.21.15.57:0/1456751626 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af41f06f0 con 0x7f6af41f1710 2023-10-30T14:14:22.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.496+0000 7f6afc479640 1 -- 172.21.15.57:0/1456751626 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af404e170 con 0x7f6af41f1710 2023-10-30T14:14:22.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.496+0000 7f6aeaffd640 1 -- 172.21.15.57:0/1456751626 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6aec0aa030 con 0x7f6af41f1710 2023-10-30T14:14:22.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.496+0000 7f6aeaffd640 1 --2- 172.21.15.57:0/1456751626 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ad80664a0 0x7f6ad8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.496+0000 7f6aeaffd640 1 -- 172.21.15.57:0/1456751626 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6aec11db80 con 0x7f6af41f1710 2023-10-30T14:14:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.496+0000 7f6afa215640 1 --2- 172.21.15.57:0/1456751626 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ad80664a0 0x7f6ad8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.496+0000 7f6afa215640 1 --2- 172.21.15.57:0/1456751626 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ad80664a0 0x7f6ad8068960 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6af0007fd0 tx=0x7f6af0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.500+0000 7f6aeaffd640 1 -- 172.21.15.57:0/1456751626 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6aec0eaf60 con 0x7f6af41f1710 2023-10-30T14:14:22.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.692+0000 7f6afc479640 1 -- 172.21.15.57:0/1456751626 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6af40cad40 con 0x7f6af41f1710 2023-10-30T14:14:22.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.692+0000 7f6aeaffd640 1 -- 172.21.15.57:0/1456751626 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7f6aec0bc090 con 0x7f6af41f1710 2023-10-30T14:14:22.698 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:22.698 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:14:20.837447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6837/901968115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":901968115},{"type":"v1","addr":"172.21.15.57:6837","nonce":901968115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:22.699 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 12 2023-10-30T14:14:22.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.692+0000 7f6afc479640 1 -- 172.21.15.57:0/1456751626 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ad80664a0 msgr2=0x7f6ad8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:22.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.692+0000 7f6afc479640 1 --2- 172.21.15.57:0/1456751626 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ad80664a0 0x7f6ad8068960 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6af0007fd0 tx=0x7f6af0009040 comp rx=0 tx=0).stop 2023-10-30T14:14:22.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.692+0000 7f6afc479640 1 -- 172.21.15.57:0/1456751626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6af41f1710 msgr2=0x7f6af41edc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:22.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.692+0000 7f6afc479640 1 --2- 172.21.15.57:0/1456751626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6af41f1710 0x7f6af41edc10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6aec01d0d0 tx=0x7f6ae0005800 comp rx=0 tx=0).stop 2023-10-30T14:14:22.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.692+0000 7f6afc479640 1 -- 172.21.15.57:0/1456751626 shutdown_connections 2023-10-30T14:14:22.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.692+0000 7f6afc479640 1 --2- 172.21.15.57:0/1456751626 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ad80664a0 0x7f6ad8068960 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:22.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.692+0000 7f6afc479640 1 --2- 172.21.15.57:0/1456751626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6af41f1710 0x7f6af41edc10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:22.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.692+0000 7f6afc479640 1 --2- 172.21.15.57:0/1456751626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6af41f1cd0 0x7f6af41f11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:22.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.692+0000 7f6afc479640 1 --2- 172.21.15.57:0/1456751626 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ae0004d10 0x7f6af415c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:22.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.692+0000 7f6afc479640 1 -- 172.21.15.57:0/1456751626 >> 172.21.15.57:0/1456751626 conn(0x7f6af40715c0 msgr2=0x7f6af4151410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:22.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.692+0000 7f6afc479640 1 -- 172.21.15.57:0/1456751626 shutdown_connections 2023-10-30T14:14:22.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:22.696+0000 7f6afc479640 1 -- 172.21.15.57:0/1456751626 wait complete. 2023-10-30T14:14:22.865 INFO:tasks.ceph.mds.f.smithi088.stderr:2023-10-30T14:14:22.857+0000 7f1418be9640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T14:14:23.322 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 12, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-30T14:14:04.087067+0000', 'modified': '2023-10-30T14:14:20.837447+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 4282, 'mds_1': 4266, 'mds_2': 4265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 2, 'incarnation': 11, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6837/901968115', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6833', 'nonce': 901968115}, {'type': 'v1', 'addr': '172.21.15.57:6837', 'nonce': 901968115}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4266': {'gid': 4266, 'name': 'c', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6836/2693548867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6835', 'nonce': 2693548867}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 2693548867}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4282': {'gid': 4282, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.88:6839/2945854794', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6838', 'nonce': 2945854794}, {'type': 'v1', 'addr': '172.21.15.88:6839', 'nonce': 2945854794}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-30T14:14:23.323 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-10-30T14:14:23.323 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2023-10-30T14:14:23.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.468+0000 7f8914dc5640 1 Processor -- start 2023-10-30T14:14:23.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.468+0000 7f8914dc5640 1 -- start start 2023-10-30T14:14:23.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f8914dc5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8910153c80 0x7f891015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:23.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f8914dc5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f89100c42a0 0x7f89100ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:23.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f8914dc5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8910159f20 con 0x7f89100c4a10 2023-10-30T14:14:23.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f8914dc5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f891015dec0 con 0x7f8910153c80 2023-10-30T14:14:23.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f8914dc5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89101599a0 con 0x7f89100c42a0 2023-10-30T14:14:23.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890dd9b640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f89100c4a10 0x7f891015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:37818/0 (socket says 172.21.15.57:37818) 2023-10-30T14:14:23.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890dd9b640 1 -- 172.21.15.57:0/1111776950 learned_addr learned my addr 172.21.15.57:0/1111776950 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:23.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890ed9d640 1 --2- 172.21.15.57:0/1111776950 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f89100c42a0 0x7f89100ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:23.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890e59c640 1 --2- 172.21.15.57:0/1111776950 >> v2:172.21.15.88:3300/0 conn(0x7f8910153c80 0x7f891015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:23.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890d59a640 1 -- 172.21.15.57:0/1111776950 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123999946 0 0) 0x7f8910159f20 con 0x7f89100c4a10 2023-10-30T14:14:23.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890d59a640 1 -- 172.21.15.57:0/1111776950 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88f8003530 con 0x7f89100c4a10 2023-10-30T14:14:23.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890d59a640 1 -- 172.21.15.57:0/1111776950 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1934315186 0 0) 0x7f88f8003530 con 0x7f89100c4a10 2023-10-30T14:14:23.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890d59a640 1 -- 172.21.15.57:0/1111776950 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f89100c42a0 msgr2=0x7f89100ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:23.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890d59a640 1 --2- 172.21.15.57:0/1111776950 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f89100c42a0 0x7f89100ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:23.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890d59a640 1 -- 172.21.15.57:0/1111776950 >> v2:172.21.15.88:3300/0 conn(0x7f8910153c80 msgr2=0x7f891015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:23.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890d59a640 1 --2- 172.21.15.57:0/1111776950 >> v2:172.21.15.88:3300/0 conn(0x7f8910153c80 0x7f891015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:23.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890d59a640 1 -- 172.21.15.57:0/1111776950 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891015d620 con 0x7f89100c4a10 2023-10-30T14:14:23.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890d59a640 1 -- 172.21.15.57:0/1111776950 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f8904001680 con 0x7f89100c4a10 2023-10-30T14:14:23.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890d59a640 1 -- 172.21.15.57:0/1111776950 >> v1:172.21.15.57:6789/0 conn(0x7f89100c4a10 legacy=0x7f891015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:23.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890d59a640 1 --2- 172.21.15.57:0/1111776950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88f8003980 0x7f88f8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:23.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890d59a640 1 --2- 172.21.15.57:0/1111776950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88f80047d0 0x7f88f8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:23.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890d59a640 1 --2- 172.21.15.57:0/1111776950 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f89100c42a0 0x7f88f8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:23.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890d59a640 1 -- 172.21.15.57:0/1111776950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891015dec0 con 0x7f88f80047d0 2023-10-30T14:14:23.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890d59a640 1 -- 172.21.15.57:0/1111776950 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89101599a0 con 0x7f88f8003980 2023-10-30T14:14:23.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890d59a640 1 -- 172.21.15.57:0/1111776950 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f800b9c0 con 0x7f89100c42a0 2023-10-30T14:14:23.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890dd9b640 1 --2- 172.21.15.57:0/1111776950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88f80047d0 0x7f88f8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:23.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890e59c640 1 --2- 172.21.15.57:0/1111776950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88f8003980 0x7f88f8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:23.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890ed9d640 1 --2- 172.21.15.57:0/1111776950 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f89100c42a0 0x7f88f8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:23.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890dd9b640 1 -- 172.21.15.57:0/1111776950 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f89100c42a0 msgr2=0x7f88f8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:23.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890dd9b640 1 --2- 172.21.15.57:0/1111776950 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f89100c42a0 0x7f88f8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:23.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890dd9b640 1 -- 172.21.15.57:0/1111776950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88f8003980 msgr2=0x7f88f8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:23.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890dd9b640 1 --2- 172.21.15.57:0/1111776950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88f8003980 0x7f88f8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:23.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890dd9b640 1 -- 172.21.15.57:0/1111776950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f8003530 con 0x7f88f80047d0 2023-10-30T14:14:23.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.472+0000 7f890e59c640 1 --2- 172.21.15.57:0/1111776950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88f8003980 0x7f88f8003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:23.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.476+0000 7f890dd9b640 1 --2- 172.21.15.57:0/1111776950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88f80047d0 0x7f88f8004c50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f890400c210 tx=0x7f890401cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae113d84a379558a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:23.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.476+0000 7f890d59a640 1 -- 172.21.15.57:0/1111776950 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89040058e0 con 0x7f88f80047d0 2023-10-30T14:14:23.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.476+0000 7f890d59a640 1 -- 172.21.15.57:0/1111776950 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8904005a40 con 0x7f88f80047d0 2023-10-30T14:14:23.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.476+0000 7f890d59a640 1 -- 172.21.15.57:0/1111776950 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8904005dd0 con 0x7f88f80047d0 2023-10-30T14:14:23.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.476+0000 7f8914dc5640 1 -- 172.21.15.57:0/1111776950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88f80047d0 msgr2=0x7f88f8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:23.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.476+0000 7f8914dc5640 1 --2- 172.21.15.57:0/1111776950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88f80047d0 0x7f88f8004c50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f890400c210 tx=0x7f890401cc40 comp rx=0 tx=0).stop 2023-10-30T14:14:23.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.476+0000 7f890ed9d640 1 -- 172.21.15.57:0/1111776950 reap_dead start 2023-10-30T14:14:23.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.476+0000 7f8914dc5640 1 -- 172.21.15.57:0/1111776950 shutdown_connections 2023-10-30T14:14:23.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.476+0000 7f8914dc5640 1 -- 172.21.15.57:0/1111776950 >> 172.21.15.57:0/1111776950 conn(0x7f89100715c0 msgr2=0x7f89100719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:23.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.476+0000 7f8914dc5640 1 -- 172.21.15.57:0/1111776950 shutdown_connections 2023-10-30T14:14:23.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.476+0000 7f8914dc5640 1 -- 172.21.15.57:0/1111776950 wait complete. 2023-10-30T14:14:23.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.476+0000 7f8914dc5640 1 Processor -- start 2023-10-30T14:14:23.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.476+0000 7f8914dc5640 1 -- start start 2023-10-30T14:14:23.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f8914dc5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88f80047d0 0x7f89101556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:23.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f8914dc5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f891015c460 0x7f891015b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:23.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f8914dc5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f891015cea0 0x7f89100c8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:23.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f8914dc5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891015ab60 con 0x7f891015c460 2023-10-30T14:14:23.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f8914dc5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89101f26a0 con 0x7f88f80047d0 2023-10-30T14:14:23.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f8914dc5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89101f2980 con 0x7f891015cea0 2023-10-30T14:14:23.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f890dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f891015c460 0x7f891015b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:23.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f890dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f891015c460 0x7f891015b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40220/0 (socket says 172.21.15.57:40220) 2023-10-30T14:14:23.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f890dd9b640 1 -- 172.21.15.57:0/5477596 learned_addr learned my addr 172.21.15.57:0/5477596 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:23.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f890e59c640 1 --2- 172.21.15.57:0/5477596 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88f80047d0 0x7f89101556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:23.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f890ed9d640 1 --2- 172.21.15.57:0/5477596 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f891015cea0 0x7f89100c8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:23.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f890dd9b640 1 -- 172.21.15.57:0/5477596 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f891015cea0 msgr2=0x7f89100c8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:23.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f890dd9b640 1 --2- 172.21.15.57:0/5477596 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f891015cea0 0x7f89100c8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:23.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f890dd9b640 1 -- 172.21.15.57:0/5477596 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88f80047d0 msgr2=0x7f89101556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:23.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f890dd9b640 1 --2- 172.21.15.57:0/5477596 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88f80047d0 0x7f89101556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:23.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f890dd9b640 1 -- 172.21.15.57:0/5477596 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891015adb0 con 0x7f891015c460 2023-10-30T14:14:23.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f890dd9b640 1 --2- 172.21.15.57:0/5477596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f891015c460 0x7f891015b890 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f890400c210 tx=0x7f88f8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:23.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f88feffd640 1 -- 172.21.15.57:0/5477596 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89040a5c70 con 0x7f891015c460 2023-10-30T14:14:23.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f88feffd640 1 -- 172.21.15.57:0/5477596 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89040a5dd0 con 0x7f891015c460 2023-10-30T14:14:23.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f88feffd640 1 -- 172.21.15.57:0/5477596 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89040049f0 con 0x7f891015c460 2023-10-30T14:14:23.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f8914dc5640 1 -- 172.21.15.57:0/5477596 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f891015b090 con 0x7f891015c460 2023-10-30T14:14:23.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.480+0000 7f8914dc5640 1 -- 172.21.15.57:0/5477596 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89100c9200 con 0x7f891015c460 2023-10-30T14:14:23.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f88feffd640 1 -- 172.21.15.57:0/5477596 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8904004b50 con 0x7f891015c460 2023-10-30T14:14:23.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f88feffd640 1 --2- 172.21.15.57:0/5477596 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f88f00664a0 0x7f88f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:23.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f88feffd640 1 -- 172.21.15.57:0/5477596 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f890411d620 con 0x7f891015c460 2023-10-30T14:14:23.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f890e59c640 1 --2- 172.21.15.57:0/5477596 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f88f00664a0 0x7f88f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:23.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f8918152000 1 Processor -- start 2023-10-30T14:14:23.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f8918152000 1 -- start start 2023-10-30T14:14:23.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f890e59c640 1 --2- 172.21.15.57:0/5477596 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f88f00664a0 0x7f88f0068960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8900009a30 tx=0x7f8900008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:23.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f8918152000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbc810 0x5626ffbbcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:23.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f8918152000 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbd130 0x5626ffbbd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:23.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f8918152000 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 0x5626ffbc7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:23.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f8918152000 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626ff970cc0 con 0x5626ffbbda50 2023-10-30T14:14:23.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f8918152000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626ff9d3920 con 0x5626ffbbc810 2023-10-30T14:14:23.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f8918152000 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626ff9d0e50 con 0x5626ffbbd130 2023-10-30T14:14:23.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f890ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 0x5626ffbc7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:23.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f890ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 0x5626ffbc7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40224/0 (socket says 172.21.15.57:40224) 2023-10-30T14:14:23.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f890ed9d640 1 -- 172.21.15.57:0/1280611578 learned_addr learned my addr 172.21.15.57:0/1280611578 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:23.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f890e59c640 1 --2- 172.21.15.57:0/1280611578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbc810 0x5626ffbbcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:23.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f890dd9b640 1 --2- 172.21.15.57:0/1280611578 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbd130 0x5626ffbbd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:23.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f890ed9d640 1 -- 172.21.15.57:0/1280611578 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbd130 msgr2=0x5626ffbbd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:23.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f890ed9d640 1 --2- 172.21.15.57:0/1280611578 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbd130 0x5626ffbbd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:23.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f890ed9d640 1 -- 172.21.15.57:0/1280611578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbc810 msgr2=0x5626ffbbcbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:23.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f890ed9d640 1 --2- 172.21.15.57:0/1280611578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbc810 0x5626ffbbcbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:23.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f890ed9d640 1 -- 172.21.15.57:0/1280611578 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626ff6c5890 con 0x5626ffbbda50 2023-10-30T14:14:23.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f890dd9b640 1 --2- 172.21.15.57:0/1280611578 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbd130 0x5626ffbbd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:23.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f890ed9d640 1 --2- 172.21.15.57:0/1280611578 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 0x5626ffbc7aa0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f89100c97d0 tx=0x7f8908009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9164657a672cc9b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:23.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f88effff640 1 -- 172.21.15.57:0/1280611578 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890800e030 con 0x5626ffbbda50 2023-10-30T14:14:23.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f88effff640 1 -- 172.21.15.57:0/1280611578 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8908016460 con 0x5626ffbbda50 2023-10-30T14:14:23.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.484+0000 7f88effff640 1 -- 172.21.15.57:0/1280611578 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89080167d0 con 0x5626ffbbda50 2023-10-30T14:14:23.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f8918152000 1 -- 172.21.15.57:0/1280611578 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 msgr2=0x5626ffbc7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:23.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f8918152000 1 --2- 172.21.15.57:0/1280611578 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 0x5626ffbc7aa0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f89100c97d0 tx=0x7f8908009750 comp rx=0 tx=0).stop 2023-10-30T14:14:23.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f8918152000 1 -- 172.21.15.57:0/1280611578 shutdown_connections 2023-10-30T14:14:23.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f8918152000 1 --2- 172.21.15.57:0/1280611578 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 0x5626ffbc7aa0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:23.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f8918152000 1 --2- 172.21.15.57:0/1280611578 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbd130 0x5626ffbbd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:23.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f8918152000 1 --2- 172.21.15.57:0/1280611578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbc810 0x5626ffbbcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:23.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f8918152000 1 -- 172.21.15.57:0/1280611578 >> 172.21.15.57:0/1280611578 conn(0x5626ffb72fc0 msgr2=0x5626ffbba5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:23.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f8918152000 1 -- 172.21.15.57:0/1280611578 shutdown_connections 2023-10-30T14:14:23.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f8918152000 1 -- 172.21.15.57:0/1280611578 wait complete. 2023-10-30T14:14:23.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f8918152000 1 -- start start 2023-10-30T14:14:23.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f8918152000 1 Processor -- start 2023-10-30T14:14:23.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f8918152000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbc810 0x5626ffbb6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:23.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f8918152000 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbd130 0x5626ffbb70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:23.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f8918152000 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 0x5626ffbb75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:23.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f8918152000 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626ff9cc740 con 0x5626ffbbda50 2023-10-30T14:14:23.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f8918152000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626ff98bd10 con 0x5626ffbbc810 2023-10-30T14:14:23.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f8918152000 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626ff98b9b0 con 0x5626ffbbd130 2023-10-30T14:14:23.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f8918152000 10 client.0 _ll_register_callbacks cb 0x5626ffb67390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T14:14:23.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f890ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 0x5626ffbb75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:23.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f890ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 0x5626ffbb75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40240/0 (socket says 172.21.15.57:40240) 2023-10-30T14:14:23.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f890ed9d640 1 -- 172.21.15.57:0/1004481166 learned_addr learned my addr 172.21.15.57:0/1004481166 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:23.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f890e59c640 1 --2- 172.21.15.57:0/1004481166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbc810 0x5626ffbb6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:23.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f890dd9b640 1 --2- 172.21.15.57:0/1004481166 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbd130 0x5626ffbb70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:23.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f890ed9d640 1 -- 172.21.15.57:0/1004481166 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbd130 msgr2=0x5626ffbb70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:23.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f890ed9d640 1 --2- 172.21.15.57:0/1004481166 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbd130 0x5626ffbb70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:23.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f890ed9d640 1 -- 172.21.15.57:0/1004481166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbc810 msgr2=0x5626ffbb6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:23.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f890ed9d640 1 --2- 172.21.15.57:0/1004481166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbc810 0x5626ffbb6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:23.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f890ed9d640 1 -- 172.21.15.57:0/1004481166 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8908009d80 con 0x5626ffbbda50 2023-10-30T14:14:23.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f890e59c640 1 --2- 172.21.15.57:0/1004481166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbc810 0x5626ffbb6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:23.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f890ed9d640 1 --2- 172.21.15.57:0/1004481166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 0x5626ffbb75e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8908019040 tx=0x7f8908004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1308aa246f92313a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:23.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.488+0000 7f88ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-30T14:14:23.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f88ed7fa640 1 -- 172.21.15.57:0/1004481166 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890800e040 con 0x5626ffbbda50 2023-10-30T14:14:23.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f8918152000 1 -- 172.21.15.57:0/1004481166 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5626ff980370 con 0x5626ffbbda50 2023-10-30T14:14:23.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f88ed7fa640 1 -- 172.21.15.57:0/1004481166 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8908026070 con 0x5626ffbbda50 2023-10-30T14:14:23.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f88ed7fa640 1 -- 172.21.15.57:0/1004481166 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8908016e10 con 0x5626ffbbda50 2023-10-30T14:14:23.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f8918152000 10 client.4653 fetch_fsmap learned FSMap version 12 2023-10-30T14:14:23.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f8918152000 1 -- 172.21.15.57:0/1004481166 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x5626ff98e460 con 0x5626ffbbda50 2023-10-30T14:14:23.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f88ed7fa640 1 -- 172.21.15.57:0/1004481166 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5626ff980370 con 0x5626ffbbda50 2023-10-30T14:14:23.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f88ed7fa640 1 -- 172.21.15.57:0/1004481166 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f8908014040 con 0x5626ffbbda50 2023-10-30T14:14:23.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f8918152000 10 client.4653 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T14:14:23.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f8918152000 10 client.4653 resolve_mds: resolved name 'e' to daemon mds.e 2023-10-30T14:14:23.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f8918152000 20 client.4653 populate_metadata read hostname 'smithi057' 2023-10-30T14:14:23.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f8918152000 1 --2- 172.21.15.57:0/1004481166 >> [v2:172.21.15.57:6833/901968115,v1:172.21.15.57:6837/901968115] conn(0x5626ffbd0260 0x5626ffbd2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:23.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f8918152000 4 client.4653 mds_command: new command op to 4265 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-30T14:14:23.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f8918152000 1 -- 172.21.15.57:0/1004481166 --> [v2:172.21.15.57:6833/901968115,v1:172.21.15.57:6837/901968115] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5626ff9662d0 con 0x5626ffbd0260 2023-10-30T14:14:23.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f890e59c640 1 --2- 172.21.15.57:0/1004481166 >> [v2:172.21.15.57:6833/901968115,v1:172.21.15.57:6837/901968115] conn(0x5626ffbd0260 0x5626ffbd2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:23.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f890e59c640 1 --2- 172.21.15.57:0/1004481166 >> [v2:172.21.15.57:6833/901968115,v1:172.21.15.57:6837/901968115] conn(0x5626ffbd0260 0x5626ffbd2650 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e3b1473d08b68a53 server_cookie=4aaab4bc6fd6072d in_seq=0 out_seq=0 2023-10-30T14:14:23.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f88ed7fa640 10 client.4653 ms_handle_connect on v2:172.21.15.57:6833/901968115 2023-10-30T14:14:23.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f890e59c640 1 -- 172.21.15.57:0/1004481166 >> [v2:172.21.15.57:6833/901968115,v1:172.21.15.57:6837/901968115] conn(0x5626ffbd0260 msgr2=0x5626ffbd2650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2023-10-30T14:14:23.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f890e59c640 1 -- 172.21.15.57:0/1004481166 >> [v2:172.21.15.57:6833/901968115,v1:172.21.15.57:6837/901968115] conn(0x5626ffbd0260 msgr2=0x5626ffbd2650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T14:14:23.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f890e59c640 1 --2- 172.21.15.57:0/1004481166 >> [v2:172.21.15.57:6833/901968115,v1:172.21.15.57:6837/901968115] conn(0x5626ffbd0260 0x5626ffbd2650 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T14:14:23.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f890e59c640 1 --2- 172.21.15.57:0/1004481166 >> [v2:172.21.15.57:6833/901968115,v1:172.21.15.57:6837/901968115] conn(0x5626ffbd0260 0x5626ffbd2650 unknown :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2023-10-30T14:14:23.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f890e59c640 1 --2- 172.21.15.57:0/1004481166 >> [v2:172.21.15.57:6833/901968115,v1:172.21.15.57:6837/901968115] conn(0x5626ffbd0260 0x5626ffbd2650 unknown :-1 s=BANNER_CONNECTING pgs=13 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:23.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f890e59c640 1 --2- 172.21.15.57:0/1004481166 >> [v2:172.21.15.57:6833/901968115,v1:172.21.15.57:6837/901968115] conn(0x5626ffbd0260 0x5626ffbd2650 crc :-1 s=READY pgs=13 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e3b1473d08b68a53 server_cookie=4aaab4bc6fd6072d in_seq=0 out_seq=1 2023-10-30T14:14:23.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:23.492+0000 7f88ed7fa640 10 client.4653 ms_handle_connect on v2:172.21.15.57:6833/901968115 2023-10-30T14:14:26.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.836+0000 7f88ed7fa640 1 -- 172.21.15.57:0/1004481166 <== mds.2 v2:172.21.15.57:6833/901968115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5626ff9662d0 con 0x5626ffbd0260 2023-10-30T14:14:26.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.836+0000 7f88ed7fa640 10 client.4653 handle_command_reply: tid=0 2023-10-30T14:14:26.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.836+0000 7f8918152000 1 client.4653 shutdown 2023-10-30T14:14:26.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.836+0000 7f8918152000 1 -- 172.21.15.57:0/1004481166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 msgr2=0x5626ffbb75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:26.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.836+0000 7f8918152000 1 --2- 172.21.15.57:0/1004481166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 0x5626ffbb75e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8908019040 tx=0x7f8908004a50 comp rx=0 tx=0).stop 2023-10-30T14:14:26.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.836+0000 7f8918152000 1 -- 172.21.15.57:0/1004481166 shutdown_connections 2023-10-30T14:14:26.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.836+0000 7f8918152000 1 --2- 172.21.15.57:0/1004481166 >> [v2:172.21.15.57:6833/901968115,v1:172.21.15.57:6837/901968115] conn(0x5626ffbd0260 0x5626ffbd2650 crc :-1 s=READY pgs=13 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:26.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.836+0000 7f8918152000 1 --2- 172.21.15.57:0/1004481166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 0x5626ffbb75e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:26.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.836+0000 7f8918152000 1 --2- 172.21.15.57:0/1004481166 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbd130 0x5626ffbb70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:26.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.836+0000 7f8918152000 1 --2- 172.21.15.57:0/1004481166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbc810 0x5626ffbb6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:26.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.836+0000 7f88ed7fa640 0 client.4653 ms_handle_reset on v2:172.21.15.57:6833/901968115 2023-10-30T14:14:26.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.836+0000 7f8918152000 1 -- 172.21.15.57:0/1004481166 >> 172.21.15.57:0/1004481166 conn(0x5626ffb72fc0 msgr2=0x5626ffbbc090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:26.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.836+0000 7f8918152000 1 -- 172.21.15.57:0/1004481166 shutdown_connections 2023-10-30T14:14:26.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.836+0000 7f8918152000 1 -- 172.21.15.57:0/1004481166 wait complete. 2023-10-30T14:14:26.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.836+0000 7f8918152000 20 client.4653 trim_cache size 0 max 16384 2023-10-30T14:14:26.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.852+0000 7f8918152000 1 Processor -- start 2023-10-30T14:14:26.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.852+0000 7f8918152000 1 -- start start 2023-10-30T14:14:26.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.852+0000 7f8918152000 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbc810 0x5626ffc14fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:26.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.852+0000 7f8918152000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbd130 0x5626ffc154e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:26.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.852+0000 7f8918152000 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbda50 0x5626ffc15a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:26.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.852+0000 7f8918152000 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626ff686940 con 0x5626ffbbc810 2023-10-30T14:14:26.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.852+0000 7f8918152000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626ff9cf2a0 con 0x5626ffbbd130 2023-10-30T14:14:26.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.852+0000 7f8918152000 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626ff9a2a10 con 0x5626ffbbda50 2023-10-30T14:14:26.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.852+0000 7f890e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbc810 0x5626ffc14fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:26.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.852+0000 7f890e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbc810 0x5626ffc14fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40814/0 (socket says 172.21.15.57:40814) 2023-10-30T14:14:26.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.852+0000 7f890e59c640 1 -- 172.21.15.57:0/3881483685 learned_addr learned my addr 172.21.15.57:0/3881483685 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:26.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.852+0000 7f890dd9b640 1 --2- 172.21.15.57:0/3881483685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbd130 0x5626ffc154e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:26.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.852+0000 7f890ed9d640 1 --2- 172.21.15.57:0/3881483685 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbda50 0x5626ffc15a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:26.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f890e59c640 1 -- 172.21.15.57:0/3881483685 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbda50 msgr2=0x5626ffc15a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:26.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f890e59c640 1 --2- 172.21.15.57:0/3881483685 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbda50 0x5626ffc15a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:26.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f890e59c640 1 -- 172.21.15.57:0/3881483685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbd130 msgr2=0x5626ffc154e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:26.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f890e59c640 1 --2- 172.21.15.57:0/3881483685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbd130 0x5626ffc154e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:26.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f890e59c640 1 -- 172.21.15.57:0/3881483685 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626ff974760 con 0x5626ffbbc810 2023-10-30T14:14:26.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f890ed9d640 1 --2- 172.21.15.57:0/3881483685 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbda50 0x5626ffc15a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:26.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f890e59c640 1 --2- 172.21.15.57:0/3881483685 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbc810 0x5626ffc14fa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x5626ff9d0130 tx=0x7f8900016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc9846026fda52dd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:26.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f88fcff9640 1 -- 172.21.15.57:0/3881483685 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8900013ad0 con 0x5626ffbbc810 2023-10-30T14:14:26.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f88fcff9640 1 -- 172.21.15.57:0/3881483685 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8900016a70 con 0x5626ffbbc810 2023-10-30T14:14:26.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f88fcff9640 1 -- 172.21.15.57:0/3881483685 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8900016de0 con 0x5626ffbbc810 2023-10-30T14:14:26.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f8918152000 1 -- 172.21.15.57:0/3881483685 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbc810 msgr2=0x5626ffc14fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:26.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f8918152000 1 --2- 172.21.15.57:0/3881483685 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbc810 0x5626ffc14fa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x5626ff9d0130 tx=0x7f8900016000 comp rx=0 tx=0).stop 2023-10-30T14:14:26.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f8918152000 1 -- 172.21.15.57:0/3881483685 shutdown_connections 2023-10-30T14:14:26.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f8918152000 1 --2- 172.21.15.57:0/3881483685 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbda50 0x5626ffc15a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:26.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f8918152000 1 --2- 172.21.15.57:0/3881483685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbd130 0x5626ffc154e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:26.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f8918152000 1 --2- 172.21.15.57:0/3881483685 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbc810 0x5626ffc14fa0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:26.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f8918152000 1 -- 172.21.15.57:0/3881483685 >> 172.21.15.57:0/3881483685 conn(0x5626ffb72fc0 msgr2=0x5626ffbbde30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:26.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f8918152000 1 -- 172.21.15.57:0/3881483685 shutdown_connections 2023-10-30T14:14:26.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f8918152000 1 -- 172.21.15.57:0/3881483685 wait complete. 2023-10-30T14:14:26.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f8918152000 1 -- start start 2023-10-30T14:14:26.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f8918152000 1 Processor -- start 2023-10-30T14:14:26.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f8918152000 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbc810 0x5626ffc14170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:26.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f8918152000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbd130 0x5626ffc146b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:26.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f8918152000 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 0x5626ffbbee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:26.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f8918152000 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626ff97a880 con 0x5626ffbbda50 2023-10-30T14:14:26.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f8918152000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626ff6c61b0 con 0x5626ffbbd130 2023-10-30T14:14:26.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.856+0000 7f8918152000 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626ff98b4d0 con 0x5626ffbbc810 2023-10-30T14:14:26.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f8918152000 10 client.0 _ll_register_callbacks cb 0x7f8908009bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T14:14:26.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f890ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 0x5626ffbbee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:26.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f890ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 0x5626ffbbee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40830/0 (socket says 172.21.15.57:40830) 2023-10-30T14:14:26.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f890ed9d640 1 -- 172.21.15.57:0/2496407896 learned_addr learned my addr 172.21.15.57:0/2496407896 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:26.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f890dd9b640 1 --2- 172.21.15.57:0/2496407896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbd130 0x5626ffc146b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:26.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f890e59c640 1 --2- 172.21.15.57:0/2496407896 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbc810 0x5626ffc14170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:26.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f890ed9d640 1 -- 172.21.15.57:0/2496407896 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbc810 msgr2=0x5626ffc14170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:26.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f890ed9d640 1 --2- 172.21.15.57:0/2496407896 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbc810 0x5626ffc14170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:26.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f890ed9d640 1 -- 172.21.15.57:0/2496407896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbd130 msgr2=0x5626ffc146b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:26.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f890ed9d640 1 --2- 172.21.15.57:0/2496407896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbd130 0x5626ffc146b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:26.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f890ed9d640 1 -- 172.21.15.57:0/2496407896 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8900009760 con 0x5626ffbbda50 2023-10-30T14:14:26.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f890ed9d640 1 --2- 172.21.15.57:0/2496407896 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 0x5626ffbbee80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f890801f690 tx=0x7f890800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1590febd73d4c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:26.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f88ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-30T14:14:26.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f88ed7fa640 1 -- 172.21.15.57:0/2496407896 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890800e040 con 0x5626ffbbda50 2023-10-30T14:14:26.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f88ed7fa640 1 -- 172.21.15.57:0/2496407896 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8908026070 con 0x5626ffbbda50 2023-10-30T14:14:26.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f8918152000 1 -- 172.21.15.57:0/2496407896 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5626ff9646f0 con 0x5626ffbbda50 2023-10-30T14:14:26.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f88ed7fa640 1 -- 172.21.15.57:0/2496407896 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f890802b830 con 0x5626ffbbda50 2023-10-30T14:14:26.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f8918152000 10 client.4659 fetch_fsmap learned FSMap version 12 2023-10-30T14:14:26.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f8918152000 1 -- 172.21.15.57:0/2496407896 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x5626ff6c5cc0 con 0x5626ffbbda50 2023-10-30T14:14:26.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f88ed7fa640 1 -- 172.21.15.57:0/2496407896 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5626ff9646f0 con 0x5626ffbbda50 2023-10-30T14:14:26.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f88ed7fa640 1 -- 172.21.15.57:0/2496407896 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f890802b990 con 0x5626ffbbda50 2023-10-30T14:14:26.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f8918152000 10 client.4659 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T14:14:26.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f8918152000 10 client.4659 resolve_mds: resolved name 'e' to daemon mds.e 2023-10-30T14:14:26.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f8918152000 20 client.4659 populate_metadata read hostname 'smithi057' 2023-10-30T14:14:26.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f8918152000 1 --2- 172.21.15.57:0/2496407896 >> [v2:172.21.15.57:6833/901968115,v1:172.21.15.57:6837/901968115] conn(0x5626ffbbf810 0x5626ffbbfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:26.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f8918152000 4 client.4659 mds_command: new command op to 4265 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-30T14:14:26.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f8918152000 1 -- 172.21.15.57:0/2496407896 --> [v2:172.21.15.57:6833/901968115,v1:172.21.15.57:6837/901968115] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5626ff9c5720 con 0x5626ffbbf810 2023-10-30T14:14:26.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f890e59c640 1 --2- 172.21.15.57:0/2496407896 >> [v2:172.21.15.57:6833/901968115,v1:172.21.15.57:6837/901968115] conn(0x5626ffbbf810 0x5626ffbbfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:26.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f88ed7fa640 10 client.4659 ms_handle_connect on v2:172.21.15.57:6833/901968115 2023-10-30T14:14:26.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f890e59c640 1 --2- 172.21.15.57:0/2496407896 >> [v2:172.21.15.57:6833/901968115,v1:172.21.15.57:6837/901968115] conn(0x5626ffbbf810 0x5626ffbbfbf0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=860aa6deb27a6cda server_cookie=767c5442df106626 in_seq=0 out_seq=0 2023-10-30T14:14:26.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f88ed7fa640 1 -- 172.21.15.57:0/2496407896 <== mds.2 v2:172.21.15.57:6833/901968115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5626ff9c5720 con 0x5626ffbbf810 2023-10-30T14:14:26.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f88ed7fa640 10 client.4659 handle_command_reply: tid=0 2023-10-30T14:14:26.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f8918152000 1 client.4659 shutdown 2023-10-30T14:14:26.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f8918152000 1 -- 172.21.15.57:0/2496407896 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 msgr2=0x5626ffbbee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:26.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f8918152000 1 --2- 172.21.15.57:0/2496407896 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 0x5626ffbbee80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f890801f690 tx=0x7f890800b040 comp rx=0 tx=0).stop 2023-10-30T14:14:26.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f8918152000 1 -- 172.21.15.57:0/2496407896 shutdown_connections 2023-10-30T14:14:26.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f8918152000 1 --2- 172.21.15.57:0/2496407896 >> [v2:172.21.15.57:6833/901968115,v1:172.21.15.57:6837/901968115] conn(0x5626ffbbf810 0x5626ffbbfbf0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:26.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f8918152000 1 --2- 172.21.15.57:0/2496407896 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626ffbbda50 0x5626ffbbee80 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:26.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f8918152000 1 --2- 172.21.15.57:0/2496407896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626ffbbd130 0x5626ffc146b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:26.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f88ed7fa640 0 client.4659 ms_handle_reset on v2:172.21.15.57:6833/901968115 2023-10-30T14:14:26.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f8918152000 1 --2- 172.21.15.57:0/2496407896 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626ffbbc810 0x5626ffc14170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:26.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.860+0000 7f8918152000 1 -- 172.21.15.57:0/2496407896 >> 172.21.15.57:0/2496407896 conn(0x5626ffb72fc0 msgr2=0x5626ffb77750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:26.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.864+0000 7f8918152000 1 -- 172.21.15.57:0/2496407896 shutdown_connections 2023-10-30T14:14:26.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.864+0000 7f8918152000 1 -- 172.21.15.57:0/2496407896 wait complete. 2023-10-30T14:14:26.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.864+0000 7f8918152000 20 client.4659 trim_cache size 0 max 16384 2023-10-30T14:14:26.878 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-30T14:14:26.879 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "e4ade530-0fca-4330-9824-3adb64478f80", 2023-10-30T14:14:26.879 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 2, 2023-10-30T14:14:26.879 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 4265, 2023-10-30T14:14:26.879 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-30T14:14:26.879 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-30T14:14:26.880 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-10-30T14:14:26.880 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 11.056485950000001, 2023-10-30T14:14:26.880 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 12, 2023-10-30T14:14:26.880 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 21, 2023-10-30T14:14:26.880 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 21, 2023-10-30T14:14:26.881 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 29.016905008999998 2023-10-30T14:14:26.881 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-30T14:14:26.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.864+0000 7f8914dc5640 1 -- 172.21.15.57:0/5477596 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f88f00664a0 msgr2=0x7f88f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:26.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.864+0000 7f8914dc5640 1 --2- 172.21.15.57:0/5477596 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f88f00664a0 0x7f88f0068960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8900009a30 tx=0x7f8900008040 comp rx=0 tx=0).stop 2023-10-30T14:14:26.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.864+0000 7f8914dc5640 1 -- 172.21.15.57:0/5477596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f891015c460 msgr2=0x7f891015b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:26.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.864+0000 7f8914dc5640 1 --2- 172.21.15.57:0/5477596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f891015c460 0x7f891015b890 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f890400c210 tx=0x7f88f8005800 comp rx=0 tx=0).stop 2023-10-30T14:14:26.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.864+0000 7f8914dc5640 1 -- 172.21.15.57:0/5477596 shutdown_connections 2023-10-30T14:14:26.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.864+0000 7f8914dc5640 1 --2- 172.21.15.57:0/5477596 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f88f00664a0 0x7f88f0068960 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:26.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.864+0000 7f8914dc5640 1 --2- 172.21.15.57:0/5477596 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f891015cea0 0x7f89100c8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:26.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.864+0000 7f8914dc5640 1 --2- 172.21.15.57:0/5477596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f891015c460 0x7f891015b890 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:26.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.864+0000 7f8914dc5640 1 --2- 172.21.15.57:0/5477596 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88f80047d0 0x7f89101556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:26.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.864+0000 7f8914dc5640 1 -- 172.21.15.57:0/5477596 >> 172.21.15.57:0/5477596 conn(0x7f89100715c0 msgr2=0x7f8910157670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:26.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.864+0000 7f8914dc5640 1 -- 172.21.15.57:0/5477596 shutdown_connections 2023-10-30T14:14:26.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:26.864+0000 7f8914dc5640 1 -- 172.21.15.57:0/5477596 wait complete. 2023-10-30T14:14:26.891 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-30T14:14:27.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.032+0000 7f14e4c32640 1 Processor -- start 2023-10-30T14:14:27.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.036+0000 7f14e4c32640 1 -- start start 2023-10-30T14:14:27.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.036+0000 7f14e4c32640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f14e00701a0 0x7f14e015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.036+0000 7f14e4c32640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f14e006b2f0 0x7f14e006b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.036+0000 7f14e4c32640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e00c2aa0 con 0x7f14e01537f0 2023-10-30T14:14:27.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.036+0000 7f14e4c32640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e00c3ab0 con 0x7f14e00701a0 2023-10-30T14:14:27.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.036+0000 7f14e4c32640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e015a820 con 0x7f14e006b2f0 2023-10-30T14:14:27.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.036+0000 7f14de59c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f14e01537f0 0x7f14e00ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41358/0 (socket says 172.21.15.57:41358) 2023-10-30T14:14:27.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.036+0000 7f14de59c640 1 -- 172.21.15.57:0/3993393213 learned_addr learned my addr 172.21.15.57:0/3993393213 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:27.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.036+0000 7f14ddd9b640 1 --2- 172.21.15.57:0/3993393213 >> v2:172.21.15.57:3300/0 conn(0x7f14e00701a0 0x7f14e015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.036+0000 7f14ded9d640 1 --2- 172.21.15.57:0/3993393213 >> v2:172.21.15.88:3300/0 conn(0x7f14e006b2f0 0x7f14e006b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.036+0000 7f14ddd9b640 1 -- 172.21.15.57:0/3993393213 >> v2:172.21.15.88:3300/0 conn(0x7f14e006b2f0 msgr2=0x7f14e006b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:27.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.036+0000 7f14ddd9b640 1 --2- 172.21.15.57:0/3993393213 >> v2:172.21.15.88:3300/0 conn(0x7f14e006b2f0 0x7f14e006b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.036+0000 7f14ddd9b640 1 -- 172.21.15.57:0/3993393213 >> v1:172.21.15.57:6789/0 conn(0x7f14e01537f0 legacy=0x7f14e00ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.036+0000 7f14ddd9b640 1 -- 172.21.15.57:0/3993393213 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e00708e0 con 0x7f14e00701a0 2023-10-30T14:14:27.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.036+0000 7f14dd59a640 1 -- 172.21.15.57:0/3993393213 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494884945 0 0) 0x7f14e00c2aa0 con 0x7f14e01537f0 2023-10-30T14:14:27.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.036+0000 7f14ded9d640 1 --2- 172.21.15.57:0/3993393213 >> v2:172.21.15.88:3300/0 conn(0x7f14e006b2f0 0x7f14e006b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:27.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.040+0000 7f14ddd9b640 1 --2- 172.21.15.57:0/3993393213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14e00701a0 0x7f14e015a2e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f14e006f4d0 tx=0x7f14d401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f0529d61be94a2a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:27.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.040+0000 7f14dd59a640 1 -- 172.21.15.57:0/3993393213 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d40a8950 con 0x7f14e00701a0 2023-10-30T14:14:27.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.040+0000 7f14dd59a640 1 -- 172.21.15.57:0/3993393213 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d40a8ab0 con 0x7f14e00701a0 2023-10-30T14:14:27.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.040+0000 7f14dd59a640 1 -- 172.21.15.57:0/3993393213 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d40a8e40 con 0x7f14e00701a0 2023-10-30T14:14:27.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.040+0000 7f14e4c32640 1 -- 172.21.15.57:0/3993393213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14e00701a0 msgr2=0x7f14e015a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.040+0000 7f14e4c32640 1 --2- 172.21.15.57:0/3993393213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14e00701a0 0x7f14e015a2e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f14e006f4d0 tx=0x7f14d401ec50 comp rx=0 tx=0).stop 2023-10-30T14:14:27.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.040+0000 7f14e4c32640 1 -- 172.21.15.57:0/3993393213 shutdown_connections 2023-10-30T14:14:27.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.040+0000 7f14e4c32640 1 --2- 172.21.15.57:0/3993393213 >> v2:172.21.15.88:3300/0 conn(0x7f14e006b2f0 0x7f14e006b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.040+0000 7f14e4c32640 1 --2- 172.21.15.57:0/3993393213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14e00701a0 0x7f14e015a2e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.040+0000 7f14e4c32640 1 -- 172.21.15.57:0/3993393213 >> 172.21.15.57:0/3993393213 conn(0x7f14e0071fb0 msgr2=0x7f14e00723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:27.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.040+0000 7f14e4c32640 1 -- 172.21.15.57:0/3993393213 shutdown_connections 2023-10-30T14:14:27.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.040+0000 7f14e4c32640 1 -- 172.21.15.57:0/3993393213 wait complete. 2023-10-30T14:14:27.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14e4c32640 1 Processor -- start 2023-10-30T14:14:27.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14e4c32640 1 -- start start 2023-10-30T14:14:27.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14e4c32640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14e006b2f0 0x7f14e015bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14e4c32640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14e00701a0 0x7f14e01f23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14e4c32640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14e01537f0 0x7f14e014bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14e4c32640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e00cc260 con 0x7f14e01537f0 2023-10-30T14:14:27.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14e4c32640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e0152460 con 0x7f14e006b2f0 2023-10-30T14:14:27.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14e4c32640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e00cc010 con 0x7f14e00701a0 2023-10-30T14:14:27.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14ded9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14e01537f0 0x7f14e014bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14ded9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14e01537f0 0x7f14e014bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40848/0 (socket says 172.21.15.57:40848) 2023-10-30T14:14:27.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14ded9d640 1 -- 172.21.15.57:0/56358818 learned_addr learned my addr 172.21.15.57:0/56358818 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:27.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14de59c640 1 --2- 172.21.15.57:0/56358818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14e006b2f0 0x7f14e015bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14ddd9b640 1 --2- 172.21.15.57:0/56358818 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14e00701a0 0x7f14e01f23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14ded9d640 1 -- 172.21.15.57:0/56358818 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14e00701a0 msgr2=0x7f14e01f23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14ded9d640 1 --2- 172.21.15.57:0/56358818 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14e00701a0 0x7f14e01f23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14ded9d640 1 -- 172.21.15.57:0/56358818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14e006b2f0 msgr2=0x7f14e015bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14ded9d640 1 --2- 172.21.15.57:0/56358818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14e006b2f0 0x7f14e015bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14ded9d640 1 -- 172.21.15.57:0/56358818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d4009420 con 0x7f14e01537f0 2023-10-30T14:14:27.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14ded9d640 1 --2- 172.21.15.57:0/56358818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14e01537f0 0x7f14e014bad0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f14d8007df0 tx=0x7f14d800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:27.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14ceffd640 1 -- 172.21.15.57:0/56358818 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d80039a0 con 0x7f14e01537f0 2023-10-30T14:14:27.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14ceffd640 1 -- 172.21.15.57:0/56358818 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d800f040 con 0x7f14e01537f0 2023-10-30T14:14:27.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14ceffd640 1 -- 172.21.15.57:0/56358818 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d8003d40 con 0x7f14e01537f0 2023-10-30T14:14:27.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14e4c32640 1 -- 172.21.15.57:0/56358818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e015aa90 con 0x7f14e01537f0 2023-10-30T14:14:27.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.044+0000 7f14e4c32640 1 -- 172.21.15.57:0/56358818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e01f2dc0 con 0x7f14e01537f0 2023-10-30T14:14:27.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.048+0000 7f14ceffd640 1 -- 172.21.15.57:0/56358818 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f14d8004030 con 0x7f14e01537f0 2023-10-30T14:14:27.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.048+0000 7f14ceffd640 1 --2- 172.21.15.57:0/56358818 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f14c00664a0 0x7f14c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.048+0000 7f14ceffd640 1 -- 172.21.15.57:0/56358818 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14d8084130 con 0x7f14e01537f0 2023-10-30T14:14:27.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.048+0000 7f14de59c640 1 --2- 172.21.15.57:0/56358818 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f14c00664a0 0x7f14c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.048+0000 7f14e7fb4000 1 Processor -- start 2023-10-30T14:14:27.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.048+0000 7f14e7fb4000 1 -- start start 2023-10-30T14:14:27.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.048+0000 7f14de59c640 1 --2- 172.21.15.57:0/56358818 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f14c00664a0 0x7f14c0068960 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f14d0007fd0 tx=0x7f14d0007d60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:27.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.048+0000 7f14e7fb4000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e45810 0x5654a8e45bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.048+0000 7f14e7fb4000 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e46130 0x5654a8e46510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.048+0000 7f14e7fb4000 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46a50 0x5654a8e50aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.048+0000 7f14e7fb4000 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a8bf9cc0 con 0x5654a8e46130 2023-10-30T14:14:27.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.048+0000 7f14e7fb4000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a8c5c920 con 0x5654a8e45810 2023-10-30T14:14:27.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.048+0000 7f14e7fb4000 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x5654a8c59e50 con 0x5654a8e46a50 2023-10-30T14:14:27.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.048+0000 7f14ddd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e46130 0x5654a8e46510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.048+0000 7f14ddd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e46130 0x5654a8e46510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40860/0 (socket says 172.21.15.57:40860) 2023-10-30T14:14:27.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.048+0000 7f14ddd9b640 1 -- 172.21.15.57:0/2199499729 learned_addr learned my addr 172.21.15.57:0/2199499729 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:27.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.048+0000 7f14de59c640 1 --2- 172.21.15.57:0/2199499729 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e45810 0x5654a8e45bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.048+0000 7f14ded9d640 1 --2- 172.21.15.57:0/2199499729 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46a50 0x5654a8e50aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14ddd9b640 1 -- 172.21.15.57:0/2199499729 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46a50 msgr2=0x5654a8e50aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:27.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14ddd9b640 1 --2- 172.21.15.57:0/2199499729 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46a50 0x5654a8e50aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14ddd9b640 1 -- 172.21.15.57:0/2199499729 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e45810 msgr2=0x5654a8e45bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:27.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14ddd9b640 1 --2- 172.21.15.57:0/2199499729 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e45810 0x5654a8e45bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14ddd9b640 1 -- 172.21.15.57:0/2199499729 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654a894e890 con 0x5654a8e46130 2023-10-30T14:14:27.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14ddd9b640 1 --2- 172.21.15.57:0/2199499729 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e46130 0x5654a8e46510 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f14d40b3040 tx=0x7f14d40aa070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72fc90ae7b0cdc22 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:27.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14bffff640 1 -- 172.21.15.57:0/2199499729 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d40b0440 con 0x5654a8e46130 2023-10-30T14:14:27.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14bffff640 1 -- 172.21.15.57:0/2199499729 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d40b05a0 con 0x5654a8e46130 2023-10-30T14:14:27.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14bffff640 1 -- 172.21.15.57:0/2199499729 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d40b0910 con 0x5654a8e46130 2023-10-30T14:14:27.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14e7fb4000 1 -- 172.21.15.57:0/2199499729 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e46130 msgr2=0x5654a8e46510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/2199499729 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e46130 0x5654a8e46510 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f14d40b3040 tx=0x7f14d40aa070 comp rx=0 tx=0).stop 2023-10-30T14:14:27.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14e7fb4000 1 -- 172.21.15.57:0/2199499729 shutdown_connections 2023-10-30T14:14:27.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/2199499729 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46a50 0x5654a8e50aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/2199499729 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e46130 0x5654a8e46510 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/2199499729 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e45810 0x5654a8e45bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14e7fb4000 1 -- 172.21.15.57:0/2199499729 >> 172.21.15.57:0/2199499729 conn(0x5654a8dfbfc0 msgr2=0x5654a8e435d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:27.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14e7fb4000 1 -- 172.21.15.57:0/2199499729 shutdown_connections 2023-10-30T14:14:27.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14e7fb4000 1 -- 172.21.15.57:0/2199499729 wait complete. 2023-10-30T14:14:27.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14e7fb4000 1 -- start start 2023-10-30T14:14:27.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14e7fb4000 1 Processor -- start 2023-10-30T14:14:27.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14e7fb4000 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 0x5654a8e3ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14e7fb4000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e46130 0x5654a8e40490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14e7fb4000 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46a50 0x5654a8e4bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14e7fb4000 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a8c55740 con 0x5654a8e45810 2023-10-30T14:14:27.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14e7fb4000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a8c14d10 con 0x5654a8e46130 2023-10-30T14:14:27.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14e7fb4000 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x5654a8c149b0 con 0x5654a8e46a50 2023-10-30T14:14:27.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14e7fb4000 10 client.0 _ll_register_callbacks cb 0x5654a8df0390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T14:14:27.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14de59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 0x5654a8e3ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14de59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 0x5654a8e3ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40868/0 (socket says 172.21.15.57:40868) 2023-10-30T14:14:27.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.052+0000 7f14de59c640 1 -- 172.21.15.57:0/1524376566 learned_addr learned my addr 172.21.15.57:0/1524376566 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:27.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14ded9d640 1 --2- 172.21.15.57:0/1524376566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46a50 0x5654a8e4bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14ddd9b640 1 --2- 172.21.15.57:0/1524376566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e46130 0x5654a8e40490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14de59c640 1 -- 172.21.15.57:0/1524376566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46a50 msgr2=0x5654a8e4bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:27.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14de59c640 1 --2- 172.21.15.57:0/1524376566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46a50 0x5654a8e4bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14de59c640 1 -- 172.21.15.57:0/1524376566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e46130 msgr2=0x5654a8e40490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:27.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14de59c640 1 --2- 172.21.15.57:0/1524376566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e46130 0x5654a8e40490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14de59c640 1 -- 172.21.15.57:0/1524376566 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e00708e0 con 0x5654a8e45810 2023-10-30T14:14:27.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14de59c640 1 --2- 172.21.15.57:0/1524376566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 0x5654a8e3ff50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f14d0013670 tx=0x7f14d0000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d9616489acb952b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:27.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-30T14:14:27.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14bd7fa640 1 -- 172.21.15.57:0/1524376566 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d0012560 con 0x5654a8e45810 2023-10-30T14:14:27.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14bd7fa640 1 -- 172.21.15.57:0/1524376566 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14d001b070 con 0x5654a8e45810 2023-10-30T14:14:27.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 1 -- 172.21.15.57:0/1524376566 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5654a8c09370 con 0x5654a8e45810 2023-10-30T14:14:27.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14bd7fa640 1 -- 172.21.15.57:0/1524376566 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14d00126c0 con 0x5654a8e45810 2023-10-30T14:14:27.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 10 client.4674 fetch_fsmap learned FSMap version 12 2023-10-30T14:14:27.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 1 -- 172.21.15.57:0/1524376566 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x5654a8c17460 con 0x5654a8e45810 2023-10-30T14:14:27.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14bd7fa640 1 -- 172.21.15.57:0/1524376566 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5654a8c09370 con 0x5654a8e45810 2023-10-30T14:14:27.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14bd7fa640 1 -- 172.21.15.57:0/1524376566 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f14d0012070 con 0x5654a8e45810 2023-10-30T14:14:27.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 10 client.4674 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T14:14:27.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 10 client.4674 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-30T14:14:27.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 20 client.4674 populate_metadata read hostname 'smithi057' 2023-10-30T14:14:27.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/1524376566 >> [v2:172.21.15.57:6835/2693548867,v1:172.21.15.57:6836/2693548867] conn(0x5654a8e4c9a0 0x5654a8e4cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 4 client.4674 mds_command: new command op to 4266 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-30T14:14:27.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 1 -- 172.21.15.57:0/1524376566 --> [v2:172.21.15.57:6835/2693548867,v1:172.21.15.57:6836/2693548867] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5654a8bef2d0 con 0x5654a8e4c9a0 2023-10-30T14:14:27.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14ddd9b640 1 --2- 172.21.15.57:0/1524376566 >> [v2:172.21.15.57:6835/2693548867,v1:172.21.15.57:6836/2693548867] conn(0x5654a8e4c9a0 0x5654a8e4cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14ddd9b640 1 --2- 172.21.15.57:0/1524376566 >> [v2:172.21.15.57:6835/2693548867,v1:172.21.15.57:6836/2693548867] conn(0x5654a8e4c9a0 0x5654a8e4cd80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=81db2c4b31077f6c server_cookie=c2dc7ff7293bbac4 in_seq=0 out_seq=0 2023-10-30T14:14:27.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14bd7fa640 10 client.4674 ms_handle_connect on v2:172.21.15.57:6835/2693548867 2023-10-30T14:14:27.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14bd7fa640 1 -- 172.21.15.57:0/1524376566 <== mds.1 v2:172.21.15.57:6835/2693548867 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5654a8bef2d0 con 0x5654a8e4c9a0 2023-10-30T14:14:27.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14bd7fa640 10 client.4674 handle_command_reply: tid=0 2023-10-30T14:14:27.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 1 client.4674 shutdown 2023-10-30T14:14:27.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 1 -- 172.21.15.57:0/1524376566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 msgr2=0x5654a8e3ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/1524376566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 0x5654a8e3ff50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f14d0013670 tx=0x7f14d0000ee0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 1 -- 172.21.15.57:0/1524376566 shutdown_connections 2023-10-30T14:14:27.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/1524376566 >> [v2:172.21.15.57:6835/2693548867,v1:172.21.15.57:6836/2693548867] conn(0x5654a8e4c9a0 0x5654a8e4cd80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/1524376566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46a50 0x5654a8e4bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/1524376566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e46130 0x5654a8e40490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/1524376566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 0x5654a8e3ff50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 1 -- 172.21.15.57:0/1524376566 >> 172.21.15.57:0/1524376566 conn(0x5654a8dfbfc0 msgr2=0x5654a8e45090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:27.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14bd7fa640 0 client.4674 ms_handle_reset on v2:172.21.15.57:6835/2693548867 2023-10-30T14:14:27.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 1 -- 172.21.15.57:0/1524376566 shutdown_connections 2023-10-30T14:14:27.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 1 -- 172.21.15.57:0/1524376566 wait complete. 2023-10-30T14:14:27.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.056+0000 7f14e7fb4000 20 client.4674 trim_cache size 0 max 16384 2023-10-30T14:14:27.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.072+0000 7f14e7fb4000 1 Processor -- start 2023-10-30T14:14:27.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.072+0000 7f14e7fb4000 1 -- start start 2023-10-30T14:14:27.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14e7fb4000 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 0x5654a8e4dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14e7fb4000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e46130 0x5654a8e4e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14e7fb4000 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46a50 0x5654a8e49650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14e7fb4000 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a8e456d0 con 0x5654a8e45810 2023-10-30T14:14:27.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14e7fb4000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a890f940 con 0x5654a8e46130 2023-10-30T14:14:27.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14e7fb4000 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x5654a8c545d0 con 0x5654a8e46a50 2023-10-30T14:14:27.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14de59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 0x5654a8e4dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14de59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 0x5654a8e4dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40870/0 (socket says 172.21.15.57:40870) 2023-10-30T14:14:27.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14de59c640 1 -- 172.21.15.57:0/3255578496 learned_addr learned my addr 172.21.15.57:0/3255578496 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:27.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14ddd9b640 1 --2- 172.21.15.57:0/3255578496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e46130 0x5654a8e4e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14ded9d640 1 --2- 172.21.15.57:0/3255578496 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46a50 0x5654a8e49650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14de59c640 1 -- 172.21.15.57:0/3255578496 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46a50 msgr2=0x5654a8e49650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:27.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14de59c640 1 --2- 172.21.15.57:0/3255578496 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46a50 0x5654a8e49650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14de59c640 1 -- 172.21.15.57:0/3255578496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e46130 msgr2=0x5654a8e4e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:27.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14de59c640 1 --2- 172.21.15.57:0/3255578496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e46130 0x5654a8e4e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14de59c640 1 -- 172.21.15.57:0/3255578496 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654a8bfd760 con 0x5654a8e45810 2023-10-30T14:14:27.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14de59c640 1 --2- 172.21.15.57:0/3255578496 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 0x5654a8e4dd90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f14d0007830 tx=0x7f14d0020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f82448924bc4959 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:27.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14ccff9640 1 -- 172.21.15.57:0/3255578496 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d0001ca0 con 0x5654a8e45810 2023-10-30T14:14:27.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14ccff9640 1 -- 172.21.15.57:0/3255578496 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d000b040 con 0x5654a8e45810 2023-10-30T14:14:27.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14ccff9640 1 -- 172.21.15.57:0/3255578496 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d002c3b0 con 0x5654a8e45810 2023-10-30T14:14:27.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14e7fb4000 1 -- 172.21.15.57:0/3255578496 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 msgr2=0x5654a8e4dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/3255578496 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 0x5654a8e4dd90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f14d0007830 tx=0x7f14d0020000 comp rx=0 tx=0).stop 2023-10-30T14:14:27.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14e7fb4000 1 -- 172.21.15.57:0/3255578496 shutdown_connections 2023-10-30T14:14:27.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/3255578496 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46a50 0x5654a8e49650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/3255578496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e46130 0x5654a8e4e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/3255578496 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 0x5654a8e4dd90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14e7fb4000 1 -- 172.21.15.57:0/3255578496 >> 172.21.15.57:0/3255578496 conn(0x5654a8dfbfc0 msgr2=0x5654a8e09c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:27.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14e7fb4000 1 -- 172.21.15.57:0/3255578496 shutdown_connections 2023-10-30T14:14:27.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14e7fb4000 1 -- 172.21.15.57:0/3255578496 wait complete. 2023-10-30T14:14:27.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14e7fb4000 1 -- start start 2023-10-30T14:14:27.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14e7fb4000 1 Processor -- start 2023-10-30T14:14:27.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.076+0000 7f14e7fb4000 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 0x5654a8ea5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46130 0x5654a8ea6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e46a50 0x5654a8ea6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a8c59740 con 0x5654a8e45810 2023-10-30T14:14:27.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a8c4fc20 con 0x5654a8e46a50 2023-10-30T14:14:27.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x5654a8c5f200 con 0x5654a8e46130 2023-10-30T14:14:27.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 10 client.0 _ll_register_callbacks cb 0x7f14d4009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T14:14:27.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14de59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 0x5654a8ea5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14de59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 0x5654a8ea5880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40884/0 (socket says 172.21.15.57:40884) 2023-10-30T14:14:27.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14de59c640 1 -- 172.21.15.57:0/822407169 learned_addr learned my addr 172.21.15.57:0/822407169 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:27.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14ded9d640 1 --2- 172.21.15.57:0/822407169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e46a50 0x5654a8ea6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14ddd9b640 1 --2- 172.21.15.57:0/822407169 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46130 0x5654a8ea6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14de59c640 1 -- 172.21.15.57:0/822407169 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46130 msgr2=0x5654a8ea6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:27.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14de59c640 1 --2- 172.21.15.57:0/822407169 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46130 0x5654a8ea6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14de59c640 1 -- 172.21.15.57:0/822407169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e46a50 msgr2=0x5654a8ea6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:27.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14de59c640 1 --2- 172.21.15.57:0/822407169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e46a50 0x5654a8ea6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14de59c640 1 -- 172.21.15.57:0/822407169 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5654a8c17460 con 0x5654a8e45810 2023-10-30T14:14:27.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14de59c640 1 --2- 172.21.15.57:0/822407169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 0x5654a8ea5880 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f14d0013670 tx=0x7f14d000a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc3c437c7ffc05f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:27.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-30T14:14:27.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14bd7fa640 1 -- 172.21.15.57:0/822407169 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d00079a0 con 0x5654a8e45810 2023-10-30T14:14:27.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14bd7fa640 1 -- 172.21.15.57:0/822407169 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14d0036470 con 0x5654a8e45810 2023-10-30T14:14:27.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 -- 172.21.15.57:0/822407169 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5654a8bed6f0 con 0x5654a8e45810 2023-10-30T14:14:27.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14bd7fa640 1 -- 172.21.15.57:0/822407169 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14d0036670 con 0x5654a8e45810 2023-10-30T14:14:27.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 10 client.4680 fetch_fsmap learned FSMap version 12 2023-10-30T14:14:27.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 -- 172.21.15.57:0/822407169 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x5654a894ecc0 con 0x5654a8e45810 2023-10-30T14:14:27.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14bd7fa640 1 -- 172.21.15.57:0/822407169 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5654a8bed6f0 con 0x5654a8e45810 2023-10-30T14:14:27.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14bd7fa640 1 -- 172.21.15.57:0/822407169 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f14d003e070 con 0x5654a8e45810 2023-10-30T14:14:27.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 10 client.4680 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T14:14:27.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 10 client.4680 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-30T14:14:27.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 20 client.4680 populate_metadata read hostname 'smithi057' 2023-10-30T14:14:27.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/822407169 >> [v2:172.21.15.57:6835/2693548867,v1:172.21.15.57:6836/2693548867] conn(0x5654a8ea6dc0 0x5654a8ea71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 4 client.4680 mds_command: new command op to 4266 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-30T14:14:27.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 -- 172.21.15.57:0/822407169 --> [v2:172.21.15.57:6835/2693548867,v1:172.21.15.57:6836/2693548867] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5654a8c4e720 con 0x5654a8ea6dc0 2023-10-30T14:14:27.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14ddd9b640 1 --2- 172.21.15.57:0/822407169 >> [v2:172.21.15.57:6835/2693548867,v1:172.21.15.57:6836/2693548867] conn(0x5654a8ea6dc0 0x5654a8ea71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14ddd9b640 1 --2- 172.21.15.57:0/822407169 >> [v2:172.21.15.57:6835/2693548867,v1:172.21.15.57:6836/2693548867] conn(0x5654a8ea6dc0 0x5654a8ea71a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a564d663e8dd9a1a server_cookie=a54c6ccd63c71100 in_seq=0 out_seq=0 2023-10-30T14:14:27.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14bd7fa640 10 client.4680 ms_handle_connect on v2:172.21.15.57:6835/2693548867 2023-10-30T14:14:27.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14bd7fa640 1 -- 172.21.15.57:0/822407169 <== mds.1 v2:172.21.15.57:6835/2693548867 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5654a8c4e720 con 0x5654a8ea6dc0 2023-10-30T14:14:27.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14bd7fa640 10 client.4680 handle_command_reply: tid=0 2023-10-30T14:14:27.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 client.4680 shutdown 2023-10-30T14:14:27.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 -- 172.21.15.57:0/822407169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 msgr2=0x5654a8ea5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/822407169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 0x5654a8ea5880 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f14d0013670 tx=0x7f14d000a000 comp rx=0 tx=0).stop 2023-10-30T14:14:27.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 -- 172.21.15.57:0/822407169 shutdown_connections 2023-10-30T14:14:27.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/822407169 >> [v2:172.21.15.57:6835/2693548867,v1:172.21.15.57:6836/2693548867] conn(0x5654a8ea6dc0 0x5654a8ea71a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/822407169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5654a8e46a50 0x5654a8ea6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/822407169 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5654a8e46130 0x5654a8ea6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 --2- 172.21.15.57:0/822407169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a8e45810 0x5654a8ea5880 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14bd7fa640 0 client.4680 ms_handle_reset on v2:172.21.15.57:6835/2693548867 2023-10-30T14:14:27.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 -- 172.21.15.57:0/822407169 >> 172.21.15.57:0/822407169 conn(0x5654a8dfbfc0 msgr2=0x5654a8e09c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:27.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 -- 172.21.15.57:0/822407169 shutdown_connections 2023-10-30T14:14:27.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 1 -- 172.21.15.57:0/822407169 wait complete. 2023-10-30T14:14:27.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.080+0000 7f14e7fb4000 20 client.4680 trim_cache size 0 max 16384 2023-10-30T14:14:27.100 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-30T14:14:27.100 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "e4ade530-0fca-4330-9824-3adb64478f80", 2023-10-30T14:14:27.100 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 1, 2023-10-30T14:14:27.100 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 4266, 2023-10-30T14:14:27.100 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-30T14:14:27.101 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-30T14:14:27.101 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-10-30T14:14:27.101 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 17.336477519999999, 2023-10-30T14:14:27.101 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 12, 2023-10-30T14:14:27.101 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 21, 2023-10-30T14:14:27.101 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 21, 2023-10-30T14:14:27.102 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 29.236915953 2023-10-30T14:14:27.102 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-30T14:14:27.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.084+0000 7f14e4c32640 1 -- 172.21.15.57:0/56358818 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f14c00664a0 msgr2=0x7f14c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.084+0000 7f14e4c32640 1 --2- 172.21.15.57:0/56358818 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f14c00664a0 0x7f14c0068960 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f14d0007fd0 tx=0x7f14d0007d60 comp rx=0 tx=0).stop 2023-10-30T14:14:27.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.084+0000 7f14e4c32640 1 -- 172.21.15.57:0/56358818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14e01537f0 msgr2=0x7f14e014bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.084+0000 7f14e4c32640 1 --2- 172.21.15.57:0/56358818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14e01537f0 0x7f14e014bad0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f14d8007df0 tx=0x7f14d800d4a0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.084+0000 7f14e4c32640 1 -- 172.21.15.57:0/56358818 shutdown_connections 2023-10-30T14:14:27.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.084+0000 7f14e4c32640 1 --2- 172.21.15.57:0/56358818 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f14c00664a0 0x7f14c0068960 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.084+0000 7f14e4c32640 1 --2- 172.21.15.57:0/56358818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14e01537f0 0x7f14e014bad0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.084+0000 7f14e4c32640 1 --2- 172.21.15.57:0/56358818 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14e00701a0 0x7f14e01f23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.084+0000 7f14e4c32640 1 --2- 172.21.15.57:0/56358818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14e006b2f0 0x7f14e015bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.084+0000 7f14e4c32640 1 -- 172.21.15.57:0/56358818 >> 172.21.15.57:0/56358818 conn(0x7f14e0071fb0 msgr2=0x7f14e00ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:27.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.084+0000 7f14e4c32640 1 -- 172.21.15.57:0/56358818 shutdown_connections 2023-10-30T14:14:27.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.084+0000 7f14e4c32640 1 -- 172.21.15.57:0/56358818 wait complete. 2023-10-30T14:14:27.107 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-10-30T14:14:27.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.248+0000 7f8542479640 1 Processor -- start 2023-10-30T14:14:27.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.248+0000 7f8542479640 1 -- start start 2023-10-30T14:14:27.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.248+0000 7f8542479640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f853c153c80 0x7f853c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.248+0000 7f8542479640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f853c0c4a10 0x7f853c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.248+0000 7f8542479640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c159f20 con 0x7f853c15a3b0 2023-10-30T14:14:27.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.248+0000 7f8542479640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f853c15dec0 con 0x7f853c153c80 2023-10-30T14:14:27.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.248+0000 7f8542479640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f853c1599a0 con 0x7f853c0c4a10 2023-10-30T14:14:27.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.248+0000 7f8540a16640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f853c15a3b0 0x7f853c0ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41374/0 (socket says 172.21.15.57:41374) 2023-10-30T14:14:27.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.248+0000 7f8540a16640 1 -- 172.21.15.57:0/989362927 learned_addr learned my addr 172.21.15.57:0/989362927 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:27.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.248+0000 7f853bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f853c153c80 0x7f853c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.248+0000 7f853b7fe640 1 --2- 172.21.15.57:0/989362927 >> v2:172.21.15.88:3300/0 conn(0x7f853c0c4a10 0x7f853c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.248+0000 7f853bfff640 1 -- 172.21.15.57:0/989362927 >> v2:172.21.15.88:3300/0 conn(0x7f853c0c4a10 msgr2=0x7f853c0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:27.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.248+0000 7f853bfff640 1 --2- 172.21.15.57:0/989362927 >> v2:172.21.15.88:3300/0 conn(0x7f853c0c4a10 0x7f853c0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.248+0000 7f853bfff640 1 -- 172.21.15.57:0/989362927 >> v1:172.21.15.57:6789/0 conn(0x7f853c15a3b0 legacy=0x7f853c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.248+0000 7f853bfff640 1 -- 172.21.15.57:0/989362927 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c0c47e0 con 0x7f853c153c80 2023-10-30T14:14:27.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.248+0000 7f853affd640 1 -- 172.21.15.57:0/989362927 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085893363 0 0) 0x7f853c159f20 con 0x7f853c15a3b0 2023-10-30T14:14:27.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.252+0000 7f853bfff640 1 --2- 172.21.15.57:0/989362927 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f853c153c80 0x7f853c15cf90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f853c06be50 tx=0x7f852c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82accccc32c1e88d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:27.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.252+0000 7f853affd640 1 -- 172.21.15.57:0/989362927 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c0a8950 con 0x7f853c153c80 2023-10-30T14:14:27.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.252+0000 7f853affd640 1 -- 172.21.15.57:0/989362927 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c0a8ab0 con 0x7f853c153c80 2023-10-30T14:14:27.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.252+0000 7f853affd640 1 -- 172.21.15.57:0/989362927 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c0a8e40 con 0x7f853c153c80 2023-10-30T14:14:27.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.252+0000 7f8542479640 1 -- 172.21.15.57:0/989362927 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f853c153c80 msgr2=0x7f853c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.252+0000 7f8542479640 1 --2- 172.21.15.57:0/989362927 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f853c153c80 0x7f853c15cf90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f853c06be50 tx=0x7f852c01ec50 comp rx=0 tx=0).stop 2023-10-30T14:14:27.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.252+0000 7f8542479640 1 -- 172.21.15.57:0/989362927 shutdown_connections 2023-10-30T14:14:27.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.252+0000 7f8542479640 1 --2- 172.21.15.57:0/989362927 >> v2:172.21.15.88:3300/0 conn(0x7f853c0c4a10 0x7f853c0c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.252+0000 7f8542479640 1 --2- 172.21.15.57:0/989362927 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f853c153c80 0x7f853c15cf90 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.252+0000 7f8542479640 1 -- 172.21.15.57:0/989362927 >> 172.21.15.57:0/989362927 conn(0x7f853c0715c0 msgr2=0x7f853c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:27.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.252+0000 7f8542479640 1 -- 172.21.15.57:0/989362927 shutdown_connections 2023-10-30T14:14:27.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.252+0000 7f8542479640 1 -- 172.21.15.57:0/989362927 wait complete. 2023-10-30T14:14:27.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f8542479640 1 Processor -- start 2023-10-30T14:14:27.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f8542479640 1 -- start start 2023-10-30T14:14:27.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f8542479640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f853c0c4a10 0x7f853c1529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f8542479640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f853c153c80 0x7f853c154160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f8542479640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f853c15a3b0 0x7f853c148e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f8542479640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853c15d280 con 0x7f853c0c4a10 2023-10-30T14:14:27.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f8542479640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853c154720 con 0x7f853c153c80 2023-10-30T14:14:27.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f8542479640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853c154a00 con 0x7f853c15a3b0 2023-10-30T14:14:27.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f853bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f853c0c4a10 0x7f853c1529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f853bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f853c0c4a10 0x7f853c1529e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40908/0 (socket says 172.21.15.57:40908) 2023-10-30T14:14:27.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f853bfff640 1 -- 172.21.15.57:0/3677448710 learned_addr learned my addr 172.21.15.57:0/3677448710 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:27.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f853b7fe640 1 --2- 172.21.15.57:0/3677448710 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f853c153c80 0x7f853c154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f8540a16640 1 --2- 172.21.15.57:0/3677448710 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f853c15a3b0 0x7f853c148e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f853bfff640 1 -- 172.21.15.57:0/3677448710 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f853c15a3b0 msgr2=0x7f853c148e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f853bfff640 1 --2- 172.21.15.57:0/3677448710 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f853c15a3b0 0x7f853c148e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f853bfff640 1 -- 172.21.15.57:0/3677448710 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f853c153c80 msgr2=0x7f853c154160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f853bfff640 1 --2- 172.21.15.57:0/3677448710 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f853c153c80 0x7f853c154160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f853bfff640 1 -- 172.21.15.57:0/3677448710 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c009420 con 0x7f853c0c4a10 2023-10-30T14:14:27.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f853bfff640 1 --2- 172.21.15.57:0/3677448710 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f853c0c4a10 0x7f853c1529e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f852c0b3040 tx=0x7f852c01b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:27.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f8538ff9640 1 -- 172.21.15.57:0/3677448710 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c0b0df0 con 0x7f853c0c4a10 2023-10-30T14:14:27.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f8538ff9640 1 -- 172.21.15.57:0/3677448710 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c0b9410 con 0x7f853c0c4a10 2023-10-30T14:14:27.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f8542479640 1 -- 172.21.15.57:0/3677448710 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853c156ff0 con 0x7f853c0c4a10 2023-10-30T14:14:27.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f8538ff9640 1 -- 172.21.15.57:0/3677448710 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c0b97c0 con 0x7f853c0c4a10 2023-10-30T14:14:27.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.256+0000 7f8542479640 1 -- 172.21.15.57:0/3677448710 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853c1ef670 con 0x7f853c0c4a10 2023-10-30T14:14:27.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f8538ff9640 1 -- 172.21.15.57:0/3677448710 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f852c0b7040 con 0x7f853c0c4a10 2023-10-30T14:14:27.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f8538ff9640 1 --2- 172.21.15.57:0/3677448710 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85280664d0 0x7f8528068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f8538ff9640 1 -- 172.21.15.57:0/3677448710 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f852c121a10 con 0x7f853c0c4a10 2023-10-30T14:14:27.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f8545802000 1 Processor -- start 2023-10-30T14:14:27.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f853b7fe640 1 --2- 172.21.15.57:0/3677448710 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85280664d0 0x7f8528068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f8545802000 1 -- start start 2023-10-30T14:14:27.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f8545802000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 0x55e43741bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f8545802000 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741c130 0x55e43741c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f8545802000 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741ca50 0x55e437426aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f853b7fe640 1 --2- 172.21.15.57:0/3677448710 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85280664d0 0x7f8528068990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f853c15d470 tx=0x7f8530008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:27.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f8545802000 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4371cfcc0 con 0x55e43741c130 2023-10-30T14:14:27.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f8545802000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e437232920 con 0x55e43741b810 2023-10-30T14:14:27.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f8545802000 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e43722fe50 con 0x55e43741ca50 2023-10-30T14:14:27.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f853b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741c130 0x55e43741c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f853b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741c130 0x55e43741c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40922/0 (socket says 172.21.15.57:40922) 2023-10-30T14:14:27.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f853b7fe640 1 -- 172.21.15.57:0/494054130 learned_addr learned my addr 172.21.15.57:0/494054130 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:27.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f8540a16640 1 --2- 172.21.15.57:0/494054130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741ca50 0x55e437426aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f853bfff640 1 --2- 172.21.15.57:0/494054130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 0x55e43741bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f853b7fe640 1 -- 172.21.15.57:0/494054130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741ca50 msgr2=0x55e437426aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:27.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f853b7fe640 1 --2- 172.21.15.57:0/494054130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741ca50 0x55e437426aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f853b7fe640 1 -- 172.21.15.57:0/494054130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 msgr2=0x55e43741bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:27.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f853b7fe640 1 --2- 172.21.15.57:0/494054130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 0x55e43741bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f853b7fe640 1 -- 172.21.15.57:0/494054130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e436f24890 con 0x55e43741c130 2023-10-30T14:14:27.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f853b7fe640 1 --2- 172.21.15.57:0/494054130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741c130 0x55e43741c510 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8530011d20 tx=0x7f8530014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170a1f6ad7012f37 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:27.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f8521ffb640 1 -- 172.21.15.57:0/494054130 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8530011d50 con 0x55e43741c130 2023-10-30T14:14:27.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f8521ffb640 1 -- 172.21.15.57:0/494054130 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8530011eb0 con 0x55e43741c130 2023-10-30T14:14:27.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.260+0000 7f8521ffb640 1 -- 172.21.15.57:0/494054130 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8530021640 con 0x55e43741c130 2023-10-30T14:14:27.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8545802000 1 -- 172.21.15.57:0/494054130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741c130 msgr2=0x55e43741c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8545802000 1 --2- 172.21.15.57:0/494054130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741c130 0x55e43741c510 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8530011d20 tx=0x7f8530014000 comp rx=0 tx=0).stop 2023-10-30T14:14:27.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8545802000 1 -- 172.21.15.57:0/494054130 shutdown_connections 2023-10-30T14:14:27.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8545802000 1 --2- 172.21.15.57:0/494054130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741ca50 0x55e437426aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8545802000 1 --2- 172.21.15.57:0/494054130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741c130 0x55e43741c510 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8545802000 1 --2- 172.21.15.57:0/494054130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 0x55e43741bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8545802000 1 -- 172.21.15.57:0/494054130 >> 172.21.15.57:0/494054130 conn(0x55e4373d1fc0 msgr2=0x55e4374195d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:27.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8545802000 1 -- 172.21.15.57:0/494054130 shutdown_connections 2023-10-30T14:14:27.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8545802000 1 -- 172.21.15.57:0/494054130 wait complete. 2023-10-30T14:14:27.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8545802000 1 -- start start 2023-10-30T14:14:27.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8545802000 1 Processor -- start 2023-10-30T14:14:27.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8545802000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 0x55e437415f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8545802000 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741c130 0x55e437416490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8545802000 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 0x55e437421a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8545802000 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e43722b740 con 0x55e43741ca50 2023-10-30T14:14:27.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8545802000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4371ead10 con 0x55e43741b810 2023-10-30T14:14:27.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8545802000 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e4371ea9b0 con 0x55e43741c130 2023-10-30T14:14:27.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8545802000 10 client.0 _ll_register_callbacks cb 0x55e4373c6390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T14:14:27.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8540a16640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 0x55e437421a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8540a16640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 0x55e437421a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40934/0 (socket says 172.21.15.57:40934) 2023-10-30T14:14:27.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8540a16640 1 -- 172.21.15.57:0/1126105232 learned_addr learned my addr 172.21.15.57:0/1126105232 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:27.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f853bfff640 1 --2- 172.21.15.57:0/1126105232 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 0x55e437415f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f853b7fe640 1 --2- 172.21.15.57:0/1126105232 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741c130 0x55e437416490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8540a16640 1 -- 172.21.15.57:0/1126105232 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741c130 msgr2=0x55e437416490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:27.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8540a16640 1 --2- 172.21.15.57:0/1126105232 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741c130 0x55e437416490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8540a16640 1 -- 172.21.15.57:0/1126105232 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 msgr2=0x55e437415f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:27.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8540a16640 1 --2- 172.21.15.57:0/1126105232 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 0x55e437415f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8540a16640 1 -- 172.21.15.57:0/1126105232 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85300119d0 con 0x55e43741ca50 2023-10-30T14:14:27.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f8540a16640 1 --2- 172.21.15.57:0/1126105232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 0x55e437421a20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8534007ae0 tx=0x7f85340018b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b82d62da7a97da server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:27.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f851affd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-30T14:14:27.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.264+0000 7f851affd640 1 -- 172.21.15.57:0/1126105232 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534002080 con 0x55e43741ca50 2023-10-30T14:14:27.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f851affd640 1 -- 172.21.15.57:0/1126105232 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8534013570 con 0x55e43741ca50 2023-10-30T14:14:27.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f8545802000 1 -- 172.21.15.57:0/1126105232 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e4371df370 con 0x55e43741ca50 2023-10-30T14:14:27.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f851affd640 1 -- 172.21.15.57:0/1126105232 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8534013730 con 0x55e43741ca50 2023-10-30T14:14:27.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f8545802000 10 client.4695 fetch_fsmap learned FSMap version 12 2023-10-30T14:14:27.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f8545802000 1 -- 172.21.15.57:0/1126105232 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x55e4371ed460 con 0x55e43741ca50 2023-10-30T14:14:27.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f851affd640 1 -- 172.21.15.57:0/1126105232 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e4371df370 con 0x55e43741ca50 2023-10-30T14:14:27.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f851affd640 1 -- 172.21.15.57:0/1126105232 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f8534013890 con 0x55e43741ca50 2023-10-30T14:14:27.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f8545802000 10 client.4695 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T14:14:27.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f8545802000 10 client.4695 resolve_mds: resolved name 'f' to daemon mds.f 2023-10-30T14:14:27.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f8545802000 20 client.4695 populate_metadata read hostname 'smithi057' 2023-10-30T14:14:27.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f8545802000 1 --2- 172.21.15.57:0/1126105232 >> [v2:172.21.15.88:6838/2945854794,v1:172.21.15.88:6839/2945854794] conn(0x55e437422790 0x55e437422b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f8545802000 4 client.4695 mds_command: new command op to 4282 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-30T14:14:27.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f8545802000 1 -- 172.21.15.57:0/1126105232 --> [v2:172.21.15.88:6838/2945854794,v1:172.21.15.88:6839/2945854794] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e4371c52d0 con 0x55e437422790 2023-10-30T14:14:27.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f853bfff640 1 --2- 172.21.15.57:0/1126105232 >> [v2:172.21.15.88:6838/2945854794,v1:172.21.15.88:6839/2945854794] conn(0x55e437422790 0x55e437422b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f853bfff640 1 --2- 172.21.15.57:0/1126105232 >> [v2:172.21.15.88:6838/2945854794,v1:172.21.15.88:6839/2945854794] conn(0x55e437422790 0x55e437422b70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bde368dc428da79 server_cookie=1f7b3c5d2984b42 in_seq=0 out_seq=0 2023-10-30T14:14:27.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f851affd640 10 client.4695 ms_handle_connect on v2:172.21.15.88:6838/2945854794 2023-10-30T14:14:27.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f851affd640 1 -- 172.21.15.57:0/1126105232 <== mds.0 v2:172.21.15.88:6838/2945854794 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e4371c52d0 con 0x55e437422790 2023-10-30T14:14:27.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f851affd640 10 client.4695 handle_command_reply: tid=0 2023-10-30T14:14:27.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f8545802000 1 client.4695 shutdown 2023-10-30T14:14:27.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f8545802000 1 -- 172.21.15.57:0/1126105232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 msgr2=0x55e437421a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.268+0000 7f8545802000 1 --2- 172.21.15.57:0/1126105232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 0x55e437421a20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8534007ae0 tx=0x7f85340018b0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.272+0000 7f8545802000 1 -- 172.21.15.57:0/1126105232 shutdown_connections 2023-10-30T14:14:27.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.272+0000 7f8545802000 1 --2- 172.21.15.57:0/1126105232 >> [v2:172.21.15.88:6838/2945854794,v1:172.21.15.88:6839/2945854794] conn(0x55e437422790 0x55e437422b70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.272+0000 7f8545802000 1 --2- 172.21.15.57:0/1126105232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 0x55e437421a20 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.272+0000 7f8545802000 1 --2- 172.21.15.57:0/1126105232 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741c130 0x55e437416490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.272+0000 7f8545802000 1 --2- 172.21.15.57:0/1126105232 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 0x55e437415f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.272+0000 7f851affd640 0 client.4695 ms_handle_reset on v2:172.21.15.88:6838/2945854794 2023-10-30T14:14:27.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.272+0000 7f8545802000 1 -- 172.21.15.57:0/1126105232 >> 172.21.15.57:0/1126105232 conn(0x55e4373d1fc0 msgr2=0x55e43741b090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:27.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.272+0000 7f8545802000 1 -- 172.21.15.57:0/1126105232 shutdown_connections 2023-10-30T14:14:27.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.272+0000 7f8545802000 1 -- 172.21.15.57:0/1126105232 wait complete. 2023-10-30T14:14:27.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.272+0000 7f8545802000 20 client.4695 trim_cache size 0 max 16384 2023-10-30T14:14:27.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.284+0000 7f8545802000 1 Processor -- start 2023-10-30T14:14:27.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.284+0000 7f8545802000 1 -- start start 2023-10-30T14:14:27.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 0x55e4373e03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741c130 0x55e4373e08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 0x55e43741f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e436ee5940 con 0x55e43741ca50 2023-10-30T14:14:27.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e43742cdf0 con 0x55e43741b810 2023-10-30T14:14:27.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e4371ea4d0 con 0x55e43741c130 2023-10-30T14:14:27.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8540a16640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 0x55e43741f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8540a16640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 0x55e43741f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40950/0 (socket says 172.21.15.57:40950) 2023-10-30T14:14:27.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8540a16640 1 -- 172.21.15.57:0/625833595 learned_addr learned my addr 172.21.15.57:0/625833595 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:27.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f853bfff640 1 --2- 172.21.15.57:0/625833595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 0x55e4373e03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f853b7fe640 1 --2- 172.21.15.57:0/625833595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741c130 0x55e4373e08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8540a16640 1 -- 172.21.15.57:0/625833595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741c130 msgr2=0x55e4373e08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:27.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8540a16640 1 --2- 172.21.15.57:0/625833595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741c130 0x55e4373e08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8540a16640 1 -- 172.21.15.57:0/625833595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 msgr2=0x55e4373e03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:27.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8540a16640 1 --2- 172.21.15.57:0/625833595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 0x55e4373e03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8540a16640 1 -- 172.21.15.57:0/625833595 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4371d3760 con 0x55e43741ca50 2023-10-30T14:14:27.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8540a16640 1 --2- 172.21.15.57:0/625833595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 0x55e43741f5e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8534007ab0 tx=0x7f853400e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33f04974f944378 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:27.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f85227fc640 1 -- 172.21.15.57:0/625833595 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534001db0 con 0x55e43741ca50 2023-10-30T14:14:27.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f85227fc640 1 -- 172.21.15.57:0/625833595 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8534029070 con 0x55e43741ca50 2023-10-30T14:14:27.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f85227fc640 1 -- 172.21.15.57:0/625833595 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534007650 con 0x55e43741ca50 2023-10-30T14:14:27.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 -- 172.21.15.57:0/625833595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 msgr2=0x55e43741f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 --2- 172.21.15.57:0/625833595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 0x55e43741f5e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8534007ab0 tx=0x7f853400e000 comp rx=0 tx=0).stop 2023-10-30T14:14:27.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 -- 172.21.15.57:0/625833595 shutdown_connections 2023-10-30T14:14:27.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 --2- 172.21.15.57:0/625833595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 0x55e43741f5e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 --2- 172.21.15.57:0/625833595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741c130 0x55e4373e08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 --2- 172.21.15.57:0/625833595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 0x55e4373e03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 -- 172.21.15.57:0/625833595 >> 172.21.15.57:0/625833595 conn(0x55e4373d1fc0 msgr2=0x55e4373d7970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:27.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 -- 172.21.15.57:0/625833595 shutdown_connections 2023-10-30T14:14:27.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 -- 172.21.15.57:0/625833595 wait complete. 2023-10-30T14:14:27.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 -- start start 2023-10-30T14:14:27.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 Processor -- start 2023-10-30T14:14:27.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 0x55e43743ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741c130 0x55e43743f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 0x55e43743f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4372308b0 con 0x55e43741ca50 2023-10-30T14:14:27.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e43722a5d0 con 0x55e43741b810 2023-10-30T14:14:27.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e43722e2a0 con 0x55e43741c130 2023-10-30T14:14:27.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.288+0000 7f8545802000 10 client.0 _ll_register_callbacks cb 0x7f8530009860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T14:14:27.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8540a16640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 0x55e43743f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8540a16640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 0x55e43743f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40958/0 (socket says 172.21.15.57:40958) 2023-10-30T14:14:27.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8540a16640 1 -- 172.21.15.57:0/885674502 learned_addr learned my addr 172.21.15.57:0/885674502 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:27.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f853bfff640 1 --2- 172.21.15.57:0/885674502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 0x55e43743ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f853b7fe640 1 --2- 172.21.15.57:0/885674502 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741c130 0x55e43743f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8540a16640 1 -- 172.21.15.57:0/885674502 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741c130 msgr2=0x55e43743f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:27.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8540a16640 1 --2- 172.21.15.57:0/885674502 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741c130 0x55e43743f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8540a16640 1 -- 172.21.15.57:0/885674502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 msgr2=0x55e43743ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:27.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8540a16640 1 --2- 172.21.15.57:0/885674502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 0x55e43743ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8540a16640 1 -- 172.21.15.57:0/885674502 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4371ed460 con 0x55e43741ca50 2023-10-30T14:14:27.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f853bfff640 1 --2- 172.21.15.57:0/885674502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 0x55e43743ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:27.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8540a16640 1 --2- 172.21.15.57:0/885674502 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 0x55e43743f650 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f85340018f0 tx=0x7f8534001410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf3c1b2e894bb9f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:27.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f851affd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-30T14:14:27.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f851affd640 1 -- 172.21.15.57:0/885674502 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534013570 con 0x55e43741ca50 2023-10-30T14:14:27.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f851affd640 1 -- 172.21.15.57:0/885674502 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85340136d0 con 0x55e43741ca50 2023-10-30T14:14:27.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 1 -- 172.21.15.57:0/885674502 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e4371c36f0 con 0x55e43741ca50 2023-10-30T14:14:27.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f851affd640 1 -- 172.21.15.57:0/885674502 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8534013890 con 0x55e43741ca50 2023-10-30T14:14:27.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 10 client.4701 fetch_fsmap learned FSMap version 12 2023-10-30T14:14:27.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 1 -- 172.21.15.57:0/885674502 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x55e436f24cc0 con 0x55e43741ca50 2023-10-30T14:14:27.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f851affd640 1 -- 172.21.15.57:0/885674502 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e4371c36f0 con 0x55e43741ca50 2023-10-30T14:14:27.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f851affd640 1 -- 172.21.15.57:0/885674502 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f853402e480 con 0x55e43741ca50 2023-10-30T14:14:27.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 10 client.4701 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T14:14:27.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 10 client.4701 resolve_mds: resolved name 'f' to daemon mds.f 2023-10-30T14:14:27.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 20 client.4701 populate_metadata read hostname 'smithi057' 2023-10-30T14:14:27.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 1 --2- 172.21.15.57:0/885674502 >> [v2:172.21.15.88:6838/2945854794,v1:172.21.15.88:6839/2945854794] conn(0x55e4374848d0 0x55e437486cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 4 client.4701 mds_command: new command op to 4282 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-30T14:14:27.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 1 -- 172.21.15.57:0/885674502 --> [v2:172.21.15.88:6838/2945854794,v1:172.21.15.88:6839/2945854794] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e437224720 con 0x55e4374848d0 2023-10-30T14:14:27.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f853bfff640 1 --2- 172.21.15.57:0/885674502 >> [v2:172.21.15.88:6838/2945854794,v1:172.21.15.88:6839/2945854794] conn(0x55e4374848d0 0x55e437486cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f853bfff640 1 --2- 172.21.15.57:0/885674502 >> [v2:172.21.15.88:6838/2945854794,v1:172.21.15.88:6839/2945854794] conn(0x55e4374848d0 0x55e437486cc0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3ced66ad185818f server_cookie=9bc70470decf86ff in_seq=0 out_seq=0 2023-10-30T14:14:27.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f851affd640 10 client.4701 ms_handle_connect on v2:172.21.15.88:6838/2945854794 2023-10-30T14:14:27.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f851affd640 1 -- 172.21.15.57:0/885674502 <== mds.0 v2:172.21.15.88:6838/2945854794 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55e437224720 con 0x55e4374848d0 2023-10-30T14:14:27.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f851affd640 10 client.4701 handle_command_reply: tid=0 2023-10-30T14:14:27.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 1 client.4701 shutdown 2023-10-30T14:14:27.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 1 -- 172.21.15.57:0/885674502 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 msgr2=0x55e43743f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 1 --2- 172.21.15.57:0/885674502 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 0x55e43743f650 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f85340018f0 tx=0x7f8534001410 comp rx=0 tx=0).stop 2023-10-30T14:14:27.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 1 -- 172.21.15.57:0/885674502 shutdown_connections 2023-10-30T14:14:27.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 1 --2- 172.21.15.57:0/885674502 >> [v2:172.21.15.88:6838/2945854794,v1:172.21.15.88:6839/2945854794] conn(0x55e4374848d0 0x55e437486cc0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 1 --2- 172.21.15.57:0/885674502 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e43741ca50 0x55e43743f650 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 1 --2- 172.21.15.57:0/885674502 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e43741c130 0x55e43743f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 1 --2- 172.21.15.57:0/885674502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e43741b810 0x55e43743ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f851affd640 0 client.4701 ms_handle_reset on v2:172.21.15.88:6838/2945854794 2023-10-30T14:14:27.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 1 -- 172.21.15.57:0/885674502 >> 172.21.15.57:0/885674502 conn(0x55e4373d1fc0 msgr2=0x55e4373d6750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:27.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 1 -- 172.21.15.57:0/885674502 shutdown_connections 2023-10-30T14:14:27.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 1 -- 172.21.15.57:0/885674502 wait complete. 2023-10-30T14:14:27.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.292+0000 7f8545802000 20 client.4701 trim_cache size 0 max 16384 2023-10-30T14:14:27.314 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-30T14:14:27.314 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "e4ade530-0fca-4330-9824-3adb64478f80", 2023-10-30T14:14:27.314 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-10-30T14:14:27.315 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 4282, 2023-10-30T14:14:27.315 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-30T14:14:27.315 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-30T14:14:27.315 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-10-30T14:14:27.315 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 22.596112668, 2023-10-30T14:14:27.315 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 12, 2023-10-30T14:14:27.316 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 21, 2023-10-30T14:14:27.316 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 21, 2023-10-30T14:14:27.316 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 29.451363878999999 2023-10-30T14:14:27.316 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-30T14:14:27.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.296+0000 7f8542479640 1 -- 172.21.15.57:0/3677448710 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85280664d0 msgr2=0x7f8528068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.296+0000 7f8542479640 1 --2- 172.21.15.57:0/3677448710 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85280664d0 0x7f8528068990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f853c15d470 tx=0x7f8530008040 comp rx=0 tx=0).stop 2023-10-30T14:14:27.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.296+0000 7f8542479640 1 -- 172.21.15.57:0/3677448710 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f853c0c4a10 msgr2=0x7f853c1529e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.296+0000 7f8542479640 1 --2- 172.21.15.57:0/3677448710 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f853c0c4a10 0x7f853c1529e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f852c0b3040 tx=0x7f852c01b9f0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.296+0000 7f8542479640 1 -- 172.21.15.57:0/3677448710 shutdown_connections 2023-10-30T14:14:27.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.296+0000 7f8542479640 1 --2- 172.21.15.57:0/3677448710 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85280664d0 0x7f8528068990 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.296+0000 7f8542479640 1 --2- 172.21.15.57:0/3677448710 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f853c15a3b0 0x7f853c148e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.296+0000 7f8542479640 1 --2- 172.21.15.57:0/3677448710 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f853c153c80 0x7f853c154160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.296+0000 7f8542479640 1 --2- 172.21.15.57:0/3677448710 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f853c0c4a10 0x7f853c1529e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.296+0000 7f8542479640 1 -- 172.21.15.57:0/3677448710 >> 172.21.15.57:0/3677448710 conn(0x7f853c0715c0 msgr2=0x7f853c149e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:27.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.296+0000 7f8542479640 1 -- 172.21.15.57:0/3677448710 shutdown_connections 2023-10-30T14:14:27.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.296+0000 7f8542479640 1 -- 172.21.15.57:0/3677448710 wait complete. 2023-10-30T14:14:27.320 INFO:teuthology.run_tasks:Running task kclient... 2023-10-30T14:14:27.342 INFO:tasks.kclient:Mounting kernel clients... 2023-10-30T14:14:27.343 INFO:tasks.kclient:config is {'client.0': {}} 2023-10-30T14:14:27.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:14:27.343 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-30T14:14:27.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:14:27.354 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link delete ceph-brx 2023-10-30T14:14:27.418 INFO:teuthology.orchestra.run.smithi057.stderr:Cannot find device "ceph-brx" 2023-10-30T14:14:27.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T14:14:27.421 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-10-30T14:14:27.421 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-30T14:14:27.421 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-30T14:14:27.422 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-30T14:14:27.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.600+0000 7fb61ce6b640 1 Processor -- start 2023-10-30T14:14:27.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.600+0000 7fb61ce6b640 1 -- start start 2023-10-30T14:14:27.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.600+0000 7fb61ce6b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb618159980 0x7fb618159dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.600+0000 7fb61ce6b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6180c5f80 con 0x7fb618153bb0 2023-10-30T14:14:27.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.600+0000 7fb61ce6b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb61815a310 con 0x7fb61806c420 2023-10-30T14:14:27.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.600+0000 7fb61ce6b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6180c5690 con 0x7fb618159980 2023-10-30T14:14:27.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.600+0000 7fb61659c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb618153bb0 0x7fb6180c6410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41386/0 (socket says 172.21.15.57:41386) 2023-10-30T14:14:27.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.600+0000 7fb61659c640 1 -- 172.21.15.57:0/370555238 learned_addr learned my addr 172.21.15.57:0/370555238 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:27.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.600+0000 7fb616d9d640 1 --2- 172.21.15.57:0/370555238 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb618159980 0x7fb618159dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.600+0000 7fb615d9b640 1 -- 172.21.15.57:0/370555238 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887762067 0 0) 0x7fb6180c5f80 con 0x7fb618153bb0 2023-10-30T14:14:27.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.600+0000 7fb615d9b640 1 -- 172.21.15.57:0/370555238 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb604003530 con 0x7fb618153bb0 2023-10-30T14:14:27.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.604+0000 7fb615d9b640 1 -- 172.21.15.57:0/370555238 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810933496 0 0) 0x7fb61815a310 con 0x7fb61806c420 2023-10-30T14:14:27.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.604+0000 7fb615d9b640 1 -- 172.21.15.57:0/370555238 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6180c5f80 con 0x7fb61806c420 2023-10-30T14:14:27.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.604+0000 7fb616d9d640 1 -- 172.21.15.57:0/370555238 >> v1:172.21.15.88:6789/0 conn(0x7fb61806c420 legacy=0x7fb6180c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.604+0000 7fb616d9d640 1 -- 172.21.15.57:0/370555238 >> v1:172.21.15.57:6789/0 conn(0x7fb618153bb0 legacy=0x7fb6180c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.604+0000 7fb615d9b640 1 -- 172.21.15.57:0/370555238 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1409562834 0 0) 0x7fb604003530 con 0x7fb618153bb0 2023-10-30T14:14:27.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.604+0000 7fb616d9d640 1 -- 172.21.15.57:0/370555238 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb61815a580 con 0x7fb618159980 2023-10-30T14:14:27.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.604+0000 7fb616d9d640 1 --2- 172.21.15.57:0/370555238 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb618159980 0x7fb618159dd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb610009dc0 tx=0x7fb61001f080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc07bbb3cf4c2c18 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:27.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.604+0000 7fb615d9b640 1 -- 172.21.15.57:0/370555238 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb608001680 con 0x7fb618159980 2023-10-30T14:14:27.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.604+0000 7fb615d9b640 1 -- 172.21.15.57:0/370555238 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6100a8c10 con 0x7fb618159980 2023-10-30T14:14:27.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.604+0000 7fb615d9b640 1 -- 172.21.15.57:0/370555238 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6100b0440 con 0x7fb618159980 2023-10-30T14:14:27.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.604+0000 7fb61ce6b640 1 -- 172.21.15.57:0/370555238 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb618159980 msgr2=0x7fb618159dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.604+0000 7fb61ce6b640 1 --2- 172.21.15.57:0/370555238 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb618159980 0x7fb618159dd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb610009dc0 tx=0x7fb61001f080 comp rx=0 tx=0).stop 2023-10-30T14:14:27.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61ce6b640 1 -- 172.21.15.57:0/370555238 shutdown_connections 2023-10-30T14:14:27.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61ce6b640 1 --2- 172.21.15.57:0/370555238 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb618159980 0x7fb618159dd0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61ce6b640 1 -- 172.21.15.57:0/370555238 >> 172.21.15.57:0/370555238 conn(0x7fb6180cac70 msgr2=0x7fb6180cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:27.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61ce6b640 1 -- 172.21.15.57:0/370555238 shutdown_connections 2023-10-30T14:14:27.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61ce6b640 1 -- 172.21.15.57:0/370555238 wait complete. 2023-10-30T14:14:27.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61ce6b640 1 Processor -- start 2023-10-30T14:14:27.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61ce6b640 1 -- start start 2023-10-30T14:14:27.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61ce6b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb61806c420 0x7fb618156230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61ce6b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb618153bb0 0x7fb61814e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61ce6b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb618159980 0x7fb6180c4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61ce6b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6181f30b0 con 0x7fb61806c420 2023-10-30T14:14:27.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61ce6b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6180c2770 con 0x7fb618153bb0 2023-10-30T14:14:27.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61ce6b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6180c40d0 con 0x7fb618159980 2023-10-30T14:14:27.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb61806c420 0x7fb618156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb61806c420 0x7fb618156230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40964/0 (socket says 172.21.15.57:40964) 2023-10-30T14:14:27.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61659c640 1 -- 172.21.15.57:0/2576119938 learned_addr learned my addr 172.21.15.57:0/2576119938 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:27.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb60ffff640 1 --2- 172.21.15.57:0/2576119938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb618153bb0 0x7fb61814e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb616d9d640 1 --2- 172.21.15.57:0/2576119938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb618159980 0x7fb6180c4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61659c640 1 -- 172.21.15.57:0/2576119938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb618159980 msgr2=0x7fb6180c4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61659c640 1 --2- 172.21.15.57:0/2576119938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb618159980 0x7fb6180c4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61659c640 1 -- 172.21.15.57:0/2576119938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb618153bb0 msgr2=0x7fb61814e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61659c640 1 --2- 172.21.15.57:0/2576119938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb618153bb0 0x7fb61814e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb61659c640 1 -- 172.21.15.57:0/2576119938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb610009a70 con 0x7fb61806c420 2023-10-30T14:14:27.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.608+0000 7fb616d9d640 1 --2- 172.21.15.57:0/2576119938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb618159980 0x7fb6180c4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:27.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.612+0000 7fb61659c640 1 --2- 172.21.15.57:0/2576119938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb61806c420 0x7fb618156230 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb60800c7b0 tx=0x7fb60800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:27.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.612+0000 7fb60effd640 1 -- 172.21.15.57:0/2576119938 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb608005ae0 con 0x7fb61806c420 2023-10-30T14:14:27.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.612+0000 7fb60effd640 1 -- 172.21.15.57:0/2576119938 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb608005c40 con 0x7fb61806c420 2023-10-30T14:14:27.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.612+0000 7fb60effd640 1 -- 172.21.15.57:0/2576119938 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb608007070 con 0x7fb61806c420 2023-10-30T14:14:27.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.612+0000 7fb61ce6b640 1 -- 172.21.15.57:0/2576119938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6181f2b00 con 0x7fb61806c420 2023-10-30T14:14:27.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.612+0000 7fb61ce6b640 1 -- 172.21.15.57:0/2576119938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6181f2dd0 con 0x7fb61806c420 2023-10-30T14:14:27.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.612+0000 7fb61ce6b640 1 -- 172.21.15.57:0/2576119938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb618158cf0 con 0x7fb61806c420 2023-10-30T14:14:27.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.612+0000 7fb60effd640 1 -- 172.21.15.57:0/2576119938 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb6080071f0 con 0x7fb61806c420 2023-10-30T14:14:27.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.612+0000 7fb60effd640 1 --2- 172.21.15.57:0/2576119938 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb5f80664a0 0x7fb5f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:27.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.612+0000 7fb60effd640 1 -- 172.21.15.57:0/2576119938 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6080834d0 con 0x7fb61806c420 2023-10-30T14:14:27.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.612+0000 7fb60ffff640 1 --2- 172.21.15.57:0/2576119938 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb5f80664a0 0x7fb5f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:27.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.616+0000 7fb60ffff640 1 --2- 172.21.15.57:0/2576119938 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb5f80664a0 0x7fb5f8068960 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb61806d260 tx=0x7fb600009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:27.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.616+0000 7fb60effd640 1 -- 172.21.15.57:0/2576119938 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb608054490 con 0x7fb61806c420 2023-10-30T14:14:27.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.812+0000 7fb61ce6b640 1 -- 172.21.15.57:0/2576119938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb6180c3ea0 con 0x7fb61806c420 2023-10-30T14:14:27.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.812+0000 7fb60effd640 1 -- 172.21.15.57:0/2576119938 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v12) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb608017060 con 0x7fb61806c420 2023-10-30T14:14:27.816 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-30T14:14:27.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.812+0000 7fb61ce6b640 1 -- 172.21.15.57:0/2576119938 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb5f80664a0 msgr2=0x7fb5f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.812+0000 7fb61ce6b640 1 --2- 172.21.15.57:0/2576119938 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb5f80664a0 0x7fb5f8068960 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb61806d260 tx=0x7fb600009040 comp rx=0 tx=0).stop 2023-10-30T14:14:27.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.812+0000 7fb61ce6b640 1 -- 172.21.15.57:0/2576119938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb61806c420 msgr2=0x7fb618156230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:27.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.812+0000 7fb61ce6b640 1 --2- 172.21.15.57:0/2576119938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb61806c420 0x7fb618156230 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb60800c7b0 tx=0x7fb60800cc70 comp rx=0 tx=0).stop 2023-10-30T14:14:27.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.812+0000 7fb61ce6b640 1 -- 172.21.15.57:0/2576119938 shutdown_connections 2023-10-30T14:14:27.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.812+0000 7fb61ce6b640 1 --2- 172.21.15.57:0/2576119938 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb5f80664a0 0x7fb5f8068960 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.812+0000 7fb61ce6b640 1 --2- 172.21.15.57:0/2576119938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb618159980 0x7fb6180c4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.812+0000 7fb61ce6b640 1 --2- 172.21.15.57:0/2576119938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb618153bb0 0x7fb61814e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.812+0000 7fb61ce6b640 1 --2- 172.21.15.57:0/2576119938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb61806c420 0x7fb618156230 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:27.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.812+0000 7fb61ce6b640 1 -- 172.21.15.57:0/2576119938 >> 172.21.15.57:0/2576119938 conn(0x7fb6180cac70 msgr2=0x7fb61806b590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:27.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.812+0000 7fb61ce6b640 1 -- 172.21.15.57:0/2576119938 shutdown_connections 2023-10-30T14:14:27.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:27.812+0000 7fb61ce6b640 1 -- 172.21.15.57:0/2576119938 wait complete. 2023-10-30T14:14:27.837 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-30T14:14:27.837 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-30T14:14:27.837 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-10-30T14:14:27.838 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-30T14:14:27.838 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-30T14:14:27.838 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-30T14:14:27.838 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-30T14:14:27.839 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-30T14:14:27.839 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-30T14:14:27.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:14:27.840 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-10-30T14:14:27.849 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-30T14:14:27.850 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-30T14:14:27.850 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-30T14:14:27.850 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-30T14:14:27.850 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-30T14:14:27.851 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-30T14:14:27.851 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-30T14:14:27.851 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-10-30T14:14:27.851 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-10-30T14:14:27.851 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-30T14:14:27.852 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-10-30T14:14:27.852 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-10-30T14:14:27.852 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-30T14:14:27.852 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-10-30T14:14:27.852 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-10-30T14:14:27.852 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-10-30T14:14:27.853 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 42391sec preferred_lft 42391sec 2023-10-30T14:14:27.853 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link 2023-10-30T14:14:27.853 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-30T14:14:27.853 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-30T14:14:27.853 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-10-30T14:14:27.854 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-10-30T14:14:27.854 INFO:teuthology.orchestra.run.smithi057.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-10-30T14:14:27.854 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 02:42:8d:87:46:21 brd ff:ff:ff:ff:ff:ff 2023-10-30T14:14:27.854 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-10-30T14:14:27.854 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-30T14:14:27.855 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-30T14:14:27.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:14:27.856 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-30T14:14:27.856 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-30T14:14:27.856 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link add name ceph-brx type bridge 2023-10-30T14:14:27.856 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip addr flush dev ceph-brx 2023-10-30T14:14:27.857 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set ceph-brx up 2023-10-30T14:14:27.857 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-30T14:14:27.857 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-30T14:14:27.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:14:27.967 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-30T14:14:27.986 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-10-30T14:14:27.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:14:27.989 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-10-30T14:14:28.061 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-10-30T14:14:28.062 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-30T14:14:28.062 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-10-30T14:14:28.062 INFO:teuthology.orchestra.run.smithi057.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-10-30T14:14:28.063 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-10-30T14:14:28.063 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-30T14:14:28.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:14:28.064 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-30T14:14:28.064 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-30T14:14:28.064 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-30T14:14:28.064 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-30T14:14:28.065 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-30T14:14:28.065 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-30T14:14:28.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:14:28.155 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-30T14:14:28.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:14:28.166 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list-id 2023-10-30T14:14:28.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:14:28.217 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-30T14:14:28.217 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-30T14:14:28.217 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-30T14:14:28.218 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-30T14:14:28.218 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-30T14:14:28.304 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-30T14:14:28.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:14:28.305 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-30T14:14:28.305 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-30T14:14:28.305 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-30T14:14:28.305 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-30T14:14:28.306 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-30T14:14:28.306 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-30T14:14:28.306 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-30T14:14:28.307 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-30T14:14:28.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:14:28.473 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-30T14:14:28.473 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-30T14:14:28.474 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2023-10-30T14:14:28.474 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set dev brx.0 master ceph-brx 2023-10-30T14:14:28.474 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-30T14:14:28.516 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-30T14:14:28.516 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-30T14:14:28.568 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-30T14:14:28.568 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-30T14:14:28.568 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-30T14:14:28.619 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-30T14:14:28.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:14:28.620 DEBUG:teuthology.orchestra.run.smithi057:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T14:14:28.726 INFO:teuthology.orchestra.run.smithi057.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T14:14:28.726 INFO:teuthology.orchestra.run.smithi057.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-30T14:14:28.726 INFO:teuthology.orchestra.run.smithi057.stdout:invalid new device string format 2023-10-30T14:14:28.840 INFO:teuthology.orchestra.run.smithi057.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T14:14:28.841 INFO:teuthology.orchestra.run.smithi057.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-30T14:14:28.841 INFO:teuthology.orchestra.run.smithi057.stdout:invalid new device string format 2023-10-30T14:14:28.841 INFO:teuthology.orchestra.run.smithi057.stdout:mount.ceph: resolved to: "172.21.15.57:3300,172.21.15.88:3300,172.21.15.88:3301" 2023-10-30T14:14:28.841 INFO:teuthology.orchestra.run.smithi057.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-30T14:14:28.854 INFO:teuthology.orchestra.run.smithi057.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T14:14:28.854 INFO:teuthology.orchestra.run.smithi057.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-30T14:14:28.854 INFO:teuthology.orchestra.run.smithi057.stdout:invalid new device string format 2023-10-30T14:14:28.854 INFO:teuthology.orchestra.run.smithi057.stdout:mount.ceph: resolved to: "172.21.15.57:3300,172.21.15.88:3300,172.21.15.88:3301" 2023-10-30T14:14:28.855 INFO:teuthology.orchestra.run.smithi057.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-30T14:14:28.855 INFO:teuthology.orchestra.run.smithi057.stdout:mount.ceph: resolved to: "172.21.15.57:6789,172.21.15.88:6789,172.21.15.88:6790" 2023-10-30T14:14:28.855 INFO:teuthology.orchestra.run.smithi057.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.57:6789,172.21.15.88:6789,172.21.15.88:6790:/ 2023-10-30T14:14:28.855 INFO:teuthology.orchestra.run.smithi057.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e4ade530-0fca-4330-9824-3adb64478f80" will pass to kernel 2023-10-30T14:14:28.858 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-30T14:14:28.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:14:28.859 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-30T14:14:28.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:14:28.887 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-30T14:14:28.976 INFO:teuthology.orchestra.run.smithi057.stdout:client4714 2023-10-30T14:14:28.976 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:14:28.976 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-30T14:14:29.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:14:29.029 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-30T14:14:29.100 INFO:teuthology.orchestra.run.smithi057.stdout:client4714 2023-10-30T14:14:29.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:14:29.101 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-30T14:14:29.438 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:14:29.442 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:14:29.938 INFO:teuthology.orchestra.run.smithi057.stdout:client4714 2023-10-30T14:14:29.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:14:29.939 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e4ade530-0fca-4330-9824-3adb64478f80.client4714/status') 2023-10-30T14:14:29.963 INFO:teuthology.orchestra.run.smithi057.stdout:instance: client.4714 (3)172.21.15.57:0/1210550538 2023-10-30T14:14:29.963 INFO:teuthology.orchestra.run.smithi057.stdout:blocklisted: false 2023-10-30T14:14:29.963 INFO:teuthology.orchestra.run.smithi057.stderr:0+1 records in 2023-10-30T14:14:29.964 INFO:teuthology.orchestra.run.smithi057.stderr:0+1 records out 2023-10-30T14:14:29.964 INFO:teuthology.orchestra.run.smithi057.stderr:70 bytes copied, 5.7876e-05 s, 1.2 MB/s 2023-10-30T14:14:29.965 INFO:teuthology.run_tasks:Running task mds_thrash... 2023-10-30T14:14:29.977 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:14:30.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.152+0000 7fcd79e79640 1 Processor -- start 2023-10-30T14:14:30.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.152+0000 7fcd79e79640 1 -- start start 2023-10-30T14:14:30.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.156+0000 7fcd79e79640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcd74150830 0x7fcd7406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:30.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.156+0000 7fcd79e79640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd74070520 0x7fcd740709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:30.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.156+0000 7fcd79e79640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd740c2310 con 0x7fcd740cb7d0 2023-10-30T14:14:30.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.156+0000 7fcd79e79640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd74156710 con 0x7fcd74150830 2023-10-30T14:14:30.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.156+0000 7fcd79e79640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd740c4360 con 0x7fcd74070520 2023-10-30T14:14:30.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.156+0000 7fcd737fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcd74150830 0x7fcd7406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:30.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.156+0000 7fcd72ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fcd740cb7d0 0x7fcd740c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41400/0 (socket says 172.21.15.57:41400) 2023-10-30T14:14:30.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.156+0000 7fcd72ffd640 1 -- 172.21.15.57:0/2365778397 learned_addr learned my addr 172.21.15.57:0/2365778397 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:30.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.156+0000 7fcd73fff640 1 --2- 172.21.15.57:0/2365778397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd74070520 0x7fcd740709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:30.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.156+0000 7fcd737fe640 1 -- 172.21.15.57:0/2365778397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd74070520 msgr2=0x7fcd740709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:30.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.156+0000 7fcd737fe640 1 --2- 172.21.15.57:0/2365778397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd74070520 0x7fcd740709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.156+0000 7fcd737fe640 1 -- 172.21.15.57:0/2365778397 >> v1:172.21.15.57:6789/0 conn(0x7fcd740cb7d0 legacy=0x7fcd740c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:30.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.156+0000 7fcd737fe640 1 -- 172.21.15.57:0/2365778397 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd741569b0 con 0x7fcd74150830 2023-10-30T14:14:30.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.156+0000 7fcd727fc640 1 -- 172.21.15.57:0/2365778397 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111742091 0 0) 0x7fcd740c2310 con 0x7fcd740cb7d0 2023-10-30T14:14:30.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.156+0000 7fcd737fe640 1 --2- 172.21.15.57:0/2365778397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd74150830 0x7fcd7406b500 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcd640097c0 tx=0x7fcd6401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b0ee3a4f0f3180 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:30.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.156+0000 7fcd727fc640 1 -- 172.21.15.57:0/2365778397 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd640a8970 con 0x7fcd74150830 2023-10-30T14:14:30.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd727fc640 1 -- 172.21.15.57:0/2365778397 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd640a8ad0 con 0x7fcd74150830 2023-10-30T14:14:30.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd727fc640 1 -- 172.21.15.57:0/2365778397 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd640a8e60 con 0x7fcd74150830 2023-10-30T14:14:30.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd79e79640 1 -- 172.21.15.57:0/2365778397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd74150830 msgr2=0x7fcd7406b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:30.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd79e79640 1 --2- 172.21.15.57:0/2365778397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd74150830 0x7fcd7406b500 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcd640097c0 tx=0x7fcd6401ec70 comp rx=0 tx=0).stop 2023-10-30T14:14:30.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd79e79640 1 -- 172.21.15.57:0/2365778397 shutdown_connections 2023-10-30T14:14:30.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd79e79640 1 --2- 172.21.15.57:0/2365778397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd74070520 0x7fcd740709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd79e79640 1 --2- 172.21.15.57:0/2365778397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd74150830 0x7fcd7406b500 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd79e79640 1 -- 172.21.15.57:0/2365778397 >> 172.21.15.57:0/2365778397 conn(0x7fcd740cac70 msgr2=0x7fcd740cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:30.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd79e79640 1 -- 172.21.15.57:0/2365778397 shutdown_connections 2023-10-30T14:14:30.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd79e79640 1 -- 172.21.15.57:0/2365778397 wait complete. 2023-10-30T14:14:30.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd79e79640 1 Processor -- start 2023-10-30T14:14:30.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd79e79640 1 -- start start 2023-10-30T14:14:30.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd79e79640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd74070520 0x7fcd7406bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:30.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd79e79640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd740cb7d0 0x7fcd741eeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:30.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd79e79640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd74150830 0x7fcd741ef060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:30.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd79e79640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7414a220 con 0x7fcd74070520 2023-10-30T14:14:30.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd79e79640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd74147540 con 0x7fcd740cb7d0 2023-10-30T14:14:30.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd79e79640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd74153800 con 0x7fcd74150830 2023-10-30T14:14:30.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd737fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd74070520 0x7fcd7406bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:30.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd737fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd74070520 0x7fcd7406bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40992/0 (socket says 172.21.15.57:40992) 2023-10-30T14:14:30.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.160+0000 7fcd737fe640 1 -- 172.21.15.57:0/1230839704 learned_addr learned my addr 172.21.15.57:0/1230839704 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:30.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.164+0000 7fcd72ffd640 1 --2- 172.21.15.57:0/1230839704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd740cb7d0 0x7fcd741eeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:30.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.164+0000 7fcd73fff640 1 --2- 172.21.15.57:0/1230839704 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd74150830 0x7fcd741ef060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:30.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.164+0000 7fcd737fe640 1 -- 172.21.15.57:0/1230839704 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd74150830 msgr2=0x7fcd741ef060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:30.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.164+0000 7fcd737fe640 1 --2- 172.21.15.57:0/1230839704 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd74150830 0x7fcd741ef060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.164+0000 7fcd737fe640 1 -- 172.21.15.57:0/1230839704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd740cb7d0 msgr2=0x7fcd741eeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:30.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.164+0000 7fcd737fe640 1 --2- 172.21.15.57:0/1230839704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd740cb7d0 0x7fcd741eeb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.164+0000 7fcd737fe640 1 -- 172.21.15.57:0/1230839704 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd64009420 con 0x7fcd74070520 2023-10-30T14:14:30.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.164+0000 7fcd737fe640 1 --2- 172.21.15.57:0/1230839704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd74070520 0x7fcd7406bf60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcd640b3040 tx=0x7fcd6401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:30.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.164+0000 7fcd5bfff640 1 -- 172.21.15.57:0/1230839704 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd640b9630 con 0x7fcd74070520 2023-10-30T14:14:30.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.164+0000 7fcd5bfff640 1 -- 172.21.15.57:0/1230839704 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd640b97b0 con 0x7fcd74070520 2023-10-30T14:14:30.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.164+0000 7fcd79e79640 1 -- 172.21.15.57:0/1230839704 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd741f2ff0 con 0x7fcd74070520 2023-10-30T14:14:30.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.164+0000 7fcd5bfff640 1 -- 172.21.15.57:0/1230839704 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd640b9b60 con 0x7fcd74070520 2023-10-30T14:14:30.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.164+0000 7fcd79e79640 1 -- 172.21.15.57:0/1230839704 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd741f2230 con 0x7fcd74070520 2023-10-30T14:14:30.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.164+0000 7fcd79e79640 1 -- 172.21.15.57:0/1230839704 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd74066510 con 0x7fcd74070520 2023-10-30T14:14:30.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.164+0000 7fcd5bfff640 1 -- 172.21.15.57:0/1230839704 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcd640b7040 con 0x7fcd74070520 2023-10-30T14:14:30.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.164+0000 7fcd5bfff640 1 --2- 172.21.15.57:0/1230839704 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcd600664a0 0x7fcd60068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:30.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.164+0000 7fcd5bfff640 1 -- 172.21.15.57:0/1230839704 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd64121f20 con 0x7fcd74070520 2023-10-30T14:14:30.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.168+0000 7fcd72ffd640 1 --2- 172.21.15.57:0/1230839704 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcd600664a0 0x7fcd60068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:30.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.168+0000 7fcd72ffd640 1 --2- 172.21.15.57:0/1230839704 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcd600664a0 0x7fcd60068960 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcd741554f0 tx=0x7fcd68007b60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:30.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.168+0000 7fcd5bfff640 1 -- 172.21.15.57:0/1230839704 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcd640ef420 con 0x7fcd74070520 2023-10-30T14:14:30.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.328+0000 7fcd79e79640 1 -- 172.21.15.57:0/1230839704 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd7404e170 con 0x7fcd74070520 2023-10-30T14:14:30.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.328+0000 7fcd5bfff640 1 -- 172.21.15.57:0/1230839704 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16433 (secure 0 0 0) 0x7fcd640b5020 con 0x7fcd74070520 2023-10-30T14:14:30.334 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:30.334 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:14:05.576818+0000","last_up_change":"2023-10-30T14:13:51.171287+0000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T14:13:52.574639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T14:14:02.600122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T14:14:03.069429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6801","nonce":3287367848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6803","nonce":3287367848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6807","nonce":3287367848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6805","nonce":3287367848}]},"public_addr":"172.21.15.57:6801/3287367848","cluster_addr":"172.21.15.57:6803/3287367848","heartbeat_back_addr":"172.21.15.57:6807/3287367848","heartbeat_front_addr":"172.21.15.57:6805/3287367848","state":["exists","up"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6809","nonce":3340797836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6811","nonce":3340797836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6815","nonce":3340797836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6813","nonce":3340797836}]},"public_addr":"172.21.15.57:6809/3340797836","cluster_addr":"172.21.15.57:6811/3340797836","heartbeat_back_addr":"172.21.15.57:6815/3340797836","heartbeat_front_addr":"172.21.15.57:6813/3340797836","state":["exists","up"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6825","nonce":1700292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6827","nonce":1700292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6831","nonce":1700292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6829","nonce":1700292}]},"public_addr":"172.21.15.57:6825/1700292","cluster_addr":"172.21.15.57:6827/1700292","heartbeat_back_addr":"172.21.15.57:6831/1700292","heartbeat_front_addr":"172.21.15.57:6829/1700292","state":["exists","up"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6817","nonce":2893420631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6819","nonce":2893420631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6823","nonce":2893420631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6821","nonce":2893420631}]},"public_addr":"172.21.15.57:6817/2893420631","cluster_addr":"172.21.15.57:6819/2893420631","heartbeat_back_addr":"172.21.15.57:6823/2893420631","heartbeat_front_addr":"172.21.15.57:6821/2893420631","state":["exists","up"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6807","nonce":2784816485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6811","nonce":2784816485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6815","nonce":2784816485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6813","nonce":2784816485}]},"public_addr":"172.21.15.88:6807/2784816485","cluster_addr":"172.21.15.88:6811/2784816485","heartbeat_back_addr":"172.21.15.88:6815/2784816485","heartbeat_front_addr":"172.21.15.88:6813/2784816485","state":["exists","up"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6801","nonce":2682449812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6803","nonce":2682449812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6810","nonce":2682449812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6806","nonce":2682449812}]},"public_addr":"172.21.15.88:6801/2682449812","cluster_addr":"172.21.15.88:6803/2682449812","heartbeat_back_addr":"172.21.15.88:6810/2682449812","heartbeat_front_addr":"172.21.15.88:6806/2682449812","state":["exists","up"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6822","nonce":3340693897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6827","nonce":3340693897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6831","nonce":3340693897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6829","nonce":3340693897}]},"public_addr":"172.21.15.88:6822/3340693897","cluster_addr":"172.21.15.88:6827/3340693897","heartbeat_back_addr":"172.21.15.88:6831/3340693897","heartbeat_front_addr":"172.21.15.88:6829/3340693897","state":["exists","up"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6817","nonce":2862111199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6820","nonce":2862111199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6825","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6826","nonce":2862111199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6821","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6823","nonce":2862111199}]},"public_addr":"172.21.15.88:6817/2862111199","cluster_addr":"172.21.15.88:6820/2862111199","heartbeat_back_addr":"172.21.15.88:6826/2862111199","heartbeat_front_addr":"172.21.15.88:6823/2862111199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.634920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.773993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.725825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.761452+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.694274+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.650718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:14:30.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.332+0000 7fcd79e79640 1 -- 172.21.15.57:0/1230839704 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcd600664a0 msgr2=0x7fcd60068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:30.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.332+0000 7fcd79e79640 1 --2- 172.21.15.57:0/1230839704 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcd600664a0 0x7fcd60068960 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcd741554f0 tx=0x7fcd68007b60 comp rx=0 tx=0).stop 2023-10-30T14:14:30.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.332+0000 7fcd79e79640 1 -- 172.21.15.57:0/1230839704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd74070520 msgr2=0x7fcd7406bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:30.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.332+0000 7fcd79e79640 1 --2- 172.21.15.57:0/1230839704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd74070520 0x7fcd7406bf60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcd640b3040 tx=0x7fcd6401ba10 comp rx=0 tx=0).stop 2023-10-30T14:14:30.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.332+0000 7fcd79e79640 1 -- 172.21.15.57:0/1230839704 shutdown_connections 2023-10-30T14:14:30.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.332+0000 7fcd79e79640 1 --2- 172.21.15.57:0/1230839704 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcd600664a0 0x7fcd60068960 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.332+0000 7fcd79e79640 1 --2- 172.21.15.57:0/1230839704 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd74150830 0x7fcd741ef060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.332+0000 7fcd79e79640 1 --2- 172.21.15.57:0/1230839704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd740cb7d0 0x7fcd741eeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.332+0000 7fcd79e79640 1 --2- 172.21.15.57:0/1230839704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd74070520 0x7fcd7406bf60 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.332+0000 7fcd79e79640 1 -- 172.21.15.57:0/1230839704 >> 172.21.15.57:0/1230839704 conn(0x7fcd740cac70 msgr2=0x7fcd740c3770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:30.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.332+0000 7fcd79e79640 1 -- 172.21.15.57:0/1230839704 shutdown_connections 2023-10-30T14:14:30.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.332+0000 7fcd79e79640 1 -- 172.21.15.57:0/1230839704 wait complete. 2023-10-30T14:14:30.355 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T14:13:52.574639+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T14:14:02.600122+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T14:14:03.069429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T14:14:30.356 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T14:14:30.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.496+0000 7f09d7faa640 1 Processor -- start 2023-10-30T14:14:30.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.496+0000 7f09d7faa640 1 -- start start 2023-10-30T14:14:30.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.496+0000 7f09d7faa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f09d0153ba0 0x7f09d00cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:30.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.496+0000 7f09d7faa640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f09d00c4410 0x7f09d015a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:30.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.496+0000 7f09d7faa640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d0153f80 con 0x7f09d00c4810 2023-10-30T14:14:30.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.496+0000 7f09d7faa640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d0159df0 con 0x7f09d00c4410 2023-10-30T14:14:30.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.496+0000 7f09d7faa640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d015a7c0 con 0x7f09d0153ba0 2023-10-30T14:14:30.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.496+0000 7f09d6547640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f09d00c4810 0x7f09d015cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41412/0 (socket says 172.21.15.57:41412) 2023-10-30T14:14:30.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.496+0000 7f09d5545640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f09d00c4410 0x7f09d015a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:30.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.496+0000 7f09d6547640 1 -- 172.21.15.57:0/3272469313 learned_addr learned my addr 172.21.15.57:0/3272469313 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:30.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.500+0000 7f09d5d46640 1 --2- 172.21.15.57:0/3272469313 >> v2:172.21.15.88:3300/0 conn(0x7f09d0153ba0 0x7f09d00cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:30.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.500+0000 7f09d5545640 1 -- 172.21.15.57:0/3272469313 >> v2:172.21.15.88:3300/0 conn(0x7f09d0153ba0 msgr2=0x7f09d00cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:30.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.500+0000 7f09d5545640 1 --2- 172.21.15.57:0/3272469313 >> v2:172.21.15.88:3300/0 conn(0x7f09d0153ba0 0x7f09d00cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.500+0000 7f09d5545640 1 -- 172.21.15.57:0/3272469313 >> v1:172.21.15.57:6789/0 conn(0x7f09d00c4810 legacy=0x7f09d015cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:30.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.500+0000 7f09d5545640 1 -- 172.21.15.57:0/3272469313 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d01597c0 con 0x7f09d00c4410 2023-10-30T14:14:30.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.500+0000 7f09d4d44640 1 -- 172.21.15.57:0/3272469313 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887923945 0 0) 0x7f09d0153f80 con 0x7f09d00c4810 2023-10-30T14:14:30.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.500+0000 7f09d5d46640 1 --2- 172.21.15.57:0/3272469313 >> v2:172.21.15.88:3300/0 conn(0x7f09d0153ba0 0x7f09d00cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:30.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.500+0000 7f09d5545640 1 --2- 172.21.15.57:0/3272469313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09d00c4410 0x7f09d015a280 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f09d00d0170 tx=0x7f09c001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4bc7c33120cb5e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:30.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.500+0000 7f09d4d44640 1 -- 172.21.15.57:0/3272469313 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c00ac600 con 0x7f09d00c4410 2023-10-30T14:14:30.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.500+0000 7f09d4d44640 1 -- 172.21.15.57:0/3272469313 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c00a8050 con 0x7f09d00c4410 2023-10-30T14:14:30.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.500+0000 7f09d4d44640 1 -- 172.21.15.57:0/3272469313 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c00ac980 con 0x7f09d00c4410 2023-10-30T14:14:30.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d7faa640 1 -- 172.21.15.57:0/3272469313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09d00c4410 msgr2=0x7f09d015a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:30.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d7faa640 1 --2- 172.21.15.57:0/3272469313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09d00c4410 0x7f09d015a280 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f09d00d0170 tx=0x7f09c001c3a0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d7faa640 1 -- 172.21.15.57:0/3272469313 shutdown_connections 2023-10-30T14:14:30.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d7faa640 1 --2- 172.21.15.57:0/3272469313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09d00c4410 0x7f09d015a280 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d7faa640 1 --2- 172.21.15.57:0/3272469313 >> v2:172.21.15.88:3300/0 conn(0x7f09d0153ba0 0x7f09d00cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d7faa640 1 -- 172.21.15.57:0/3272469313 >> 172.21.15.57:0/3272469313 conn(0x7f09d0071790 msgr2=0x7f09d0071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:30.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d7faa640 1 -- 172.21.15.57:0/3272469313 shutdown_connections 2023-10-30T14:14:30.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d7faa640 1 -- 172.21.15.57:0/3272469313 wait complete. 2023-10-30T14:14:30.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d7faa640 1 Processor -- start 2023-10-30T14:14:30.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d7faa640 1 -- start start 2023-10-30T14:14:30.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d7faa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09d00c4410 0x7f09d01509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:30.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d7faa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09d00c4810 0x7f09d01f5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:30.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d7faa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09d0153ba0 0x7f09d00c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:30.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d7faa640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d00c6c50 con 0x7f09d00c4410 2023-10-30T14:14:30.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d7faa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d0151670 con 0x7f09d0153ba0 2023-10-30T14:14:30.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d7faa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d01510b0 con 0x7f09d00c4810 2023-10-30T14:14:30.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d5d46640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09d00c4410 0x7f09d01509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:30.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d5d46640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09d00c4410 0x7f09d01509b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41012/0 (socket says 172.21.15.57:41012) 2023-10-30T14:14:30.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d5d46640 1 -- 172.21.15.57:0/4034764242 learned_addr learned my addr 172.21.15.57:0/4034764242 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:30.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d5545640 1 --2- 172.21.15.57:0/4034764242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09d00c4810 0x7f09d01f5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:30.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d6547640 1 --2- 172.21.15.57:0/4034764242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09d0153ba0 0x7f09d00c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:30.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d5d46640 1 -- 172.21.15.57:0/4034764242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09d00c4810 msgr2=0x7f09d01f5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:30.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d5d46640 1 --2- 172.21.15.57:0/4034764242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09d00c4810 0x7f09d01f5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d5d46640 1 -- 172.21.15.57:0/4034764242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09d0153ba0 msgr2=0x7f09d00c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:30.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d5d46640 1 --2- 172.21.15.57:0/4034764242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09d0153ba0 0x7f09d00c9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d5d46640 1 -- 172.21.15.57:0/4034764242 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c0009420 con 0x7f09d00c4410 2023-10-30T14:14:30.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.504+0000 7f09d5d46640 1 --2- 172.21.15.57:0/4034764242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09d00c4410 0x7f09d01509b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00d800 tx=0x7f09cc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:30.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.508+0000 7f09c67fc640 1 -- 172.21.15.57:0/4034764242 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc009660 con 0x7f09d00c4410 2023-10-30T14:14:30.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.508+0000 7f09c67fc640 1 -- 172.21.15.57:0/4034764242 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09cc0097c0 con 0x7f09d00c4410 2023-10-30T14:14:30.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.508+0000 7f09c67fc640 1 -- 172.21.15.57:0/4034764242 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc009b70 con 0x7f09d00c4410 2023-10-30T14:14:30.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.508+0000 7f09d7faa640 1 -- 172.21.15.57:0/4034764242 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d0151390 con 0x7f09d00c4410 2023-10-30T14:14:30.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.508+0000 7f09d7faa640 1 -- 172.21.15.57:0/4034764242 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d00c6870 con 0x7f09d00c4410 2023-10-30T14:14:30.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.508+0000 7f09d7faa640 1 -- 172.21.15.57:0/4034764242 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d00c6ec0 con 0x7f09d00c4410 2023-10-30T14:14:30.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.508+0000 7f09c67fc640 1 -- 172.21.15.57:0/4034764242 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f09cc009cd0 con 0x7f09d00c4410 2023-10-30T14:14:30.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.508+0000 7f09c67fc640 1 --2- 172.21.15.57:0/4034764242 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09b40664a0 0x7f09b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:30.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.508+0000 7f09c67fc640 1 -- 172.21.15.57:0/4034764242 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09cc084600 con 0x7f09d00c4410 2023-10-30T14:14:30.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.508+0000 7f09d5545640 1 --2- 172.21.15.57:0/4034764242 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09b40664a0 0x7f09b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:30.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.512+0000 7f09d5545640 1 --2- 172.21.15.57:0/4034764242 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09b40664a0 0x7f09b4068960 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f09c00ae000 tx=0x7f09c0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:30.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.512+0000 7f09c67fc640 1 -- 172.21.15.57:0/4034764242 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f09cc0519e0 con 0x7f09d00c4410 2023-10-30T14:14:30.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.668+0000 7f09d7faa640 1 -- 172.21.15.57:0/4034764242 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f09d00c63b0 con 0x7f09d00c4410 2023-10-30T14:14:30.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.668+0000 7f09c67fc640 1 -- 172.21.15.57:0/4034764242 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f09cc055000 con 0x7f09d00c4410 2023-10-30T14:14:30.674 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-30T14:14:30.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.672+0000 7f09d7faa640 1 -- 172.21.15.57:0/4034764242 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09b40664a0 msgr2=0x7f09b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:30.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.672+0000 7f09d7faa640 1 --2- 172.21.15.57:0/4034764242 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09b40664a0 0x7f09b4068960 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f09c00ae000 tx=0x7f09c0008040 comp rx=0 tx=0).stop 2023-10-30T14:14:30.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.672+0000 7f09d7faa640 1 -- 172.21.15.57:0/4034764242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09d00c4410 msgr2=0x7f09d01509b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:30.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.672+0000 7f09d7faa640 1 --2- 172.21.15.57:0/4034764242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09d00c4410 0x7f09d01509b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00d800 tx=0x7f09cc00dcc0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.672+0000 7f09d7faa640 1 -- 172.21.15.57:0/4034764242 shutdown_connections 2023-10-30T14:14:30.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.672+0000 7f09d7faa640 1 --2- 172.21.15.57:0/4034764242 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09b40664a0 0x7f09b4068960 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.672+0000 7f09d7faa640 1 --2- 172.21.15.57:0/4034764242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09d0153ba0 0x7f09d00c9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.672+0000 7f09d7faa640 1 --2- 172.21.15.57:0/4034764242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09d00c4810 0x7f09d01f5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.672+0000 7f09d7faa640 1 --2- 172.21.15.57:0/4034764242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09d00c4410 0x7f09d01509b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.672+0000 7f09d7faa640 1 -- 172.21.15.57:0/4034764242 >> 172.21.15.57:0/4034764242 conn(0x7f09d0071790 msgr2=0x7f09d00c51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:30.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.672+0000 7f09d7faa640 1 -- 172.21.15.57:0/4034764242 shutdown_connections 2023-10-30T14:14:30.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.672+0000 7f09d7faa640 1 -- 172.21.15.57:0/4034764242 wait complete. 2023-10-30T14:14:30.695 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T14:14:30.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.840+0000 7f8aa0054640 1 Processor -- start 2023-10-30T14:14:30.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.840+0000 7f8aa0054640 1 -- start start 2023-10-30T14:14:30.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.840+0000 7f8aa0054640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8a98153b20 0x7f8a9815a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:30.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.840+0000 7f8aa0054640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8a980c79f0 0x7f8a98070590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:30.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.840+0000 7f8aa0054640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9806b0f0 con 0x7f8a980cd2d0 2023-10-30T14:14:30.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.840+0000 7f8aa0054640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a98070ad0 con 0x7f8a980c79f0 2023-10-30T14:14:30.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.840+0000 7f8aa0054640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a9806bf50 con 0x7f8a98153b20 2023-10-30T14:14:30.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.840+0000 7f8a9d5ef640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8a980c79f0 0x7f8a98070590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:30.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.840+0000 7f8a9ddf0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8a98153b20 0x7f8a9815a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:30.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.840+0000 7f8a9ddf0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8a98153b20 0x7f8a9815a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:60938/0 (socket says 172.21.15.57:60938) 2023-10-30T14:14:30.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.840+0000 7f8a9e5f1640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8a980cd2d0 0x7f8a980c25c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:48382/0 (socket says 172.21.15.57:48382) 2023-10-30T14:14:30.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.840+0000 7f8a9ddf0640 1 -- 172.21.15.57:0/1786431725 learned_addr learned my addr 172.21.15.57:0/1786431725 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:30.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.840+0000 7f8a9d5ef640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8a980c79f0 0x7f8a98070590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41018/0 (socket says 172.21.15.57:41018) 2023-10-30T14:14:30.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.840+0000 7f8a9d5ef640 1 -- 172.21.15.57:0/1786431725 >> v2:172.21.15.88:3300/0 conn(0x7f8a98153b20 msgr2=0x7f8a9815a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:30.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.840+0000 7f8a9d5ef640 1 --2- 172.21.15.57:0/1786431725 >> v2:172.21.15.88:3300/0 conn(0x7f8a98153b20 0x7f8a9815a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.840+0000 7f8a9d5ef640 1 -- 172.21.15.57:0/1786431725 >> v1:172.21.15.88:6789/0 conn(0x7f8a980cd2d0 legacy=0x7f8a980c25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:30.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.840+0000 7f8a9d5ef640 1 -- 172.21.15.57:0/1786431725 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a98153f00 con 0x7f8a980c79f0 2023-10-30T14:14:30.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.844+0000 7f8a9d5ef640 1 --2- 172.21.15.57:0/1786431725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a980c79f0 0x7f8a98070590 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8a9806b0c0 tx=0x7f8a8801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2adfdec0f3dde47 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:30.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.844+0000 7f8a9cdee640 1 -- 172.21.15.57:0/1786431725 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a880a8950 con 0x7f8a980c79f0 2023-10-30T14:14:30.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.844+0000 7f8a9cdee640 1 -- 172.21.15.57:0/1786431725 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a880a8ab0 con 0x7f8a980c79f0 2023-10-30T14:14:30.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.844+0000 7f8a9cdee640 1 -- 172.21.15.57:0/1786431725 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a880a8e40 con 0x7f8a980c79f0 2023-10-30T14:14:30.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.844+0000 7f8aa0054640 1 -- 172.21.15.57:0/1786431725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a980c79f0 msgr2=0x7f8a98070590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:30.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.844+0000 7f8aa0054640 1 --2- 172.21.15.57:0/1786431725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a980c79f0 0x7f8a98070590 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8a9806b0c0 tx=0x7f8a8801ec50 comp rx=0 tx=0).stop 2023-10-30T14:14:30.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.844+0000 7f8aa0054640 1 -- 172.21.15.57:0/1786431725 shutdown_connections 2023-10-30T14:14:30.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.844+0000 7f8aa0054640 1 --2- 172.21.15.57:0/1786431725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a980c79f0 0x7f8a98070590 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.844+0000 7f8aa0054640 1 --2- 172.21.15.57:0/1786431725 >> v2:172.21.15.88:3300/0 conn(0x7f8a98153b20 0x7f8a9815a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.844+0000 7f8aa0054640 1 -- 172.21.15.57:0/1786431725 >> 172.21.15.57:0/1786431725 conn(0x7f8a98072010 msgr2=0x7f8a98072420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:30.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.844+0000 7f8aa0054640 1 -- 172.21.15.57:0/1786431725 shutdown_connections 2023-10-30T14:14:30.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.844+0000 7f8aa0054640 1 -- 172.21.15.57:0/1786431725 wait complete. 2023-10-30T14:14:30.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8aa0054640 1 Processor -- start 2023-10-30T14:14:30.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8aa0054640 1 -- start start 2023-10-30T14:14:30.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8aa0054640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a980c79f0 0x7f8a981f2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:30.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8aa0054640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a980cd2d0 0x7f8a980c8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:30.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8aa0054640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a98153b20 0x7f8a980c90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:30.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8aa0054640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a981f2400 con 0x7f8a980c79f0 2023-10-30T14:14:30.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8aa0054640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a98154990 con 0x7f8a980cd2d0 2023-10-30T14:14:30.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8aa0054640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a98155f70 con 0x7f8a98153b20 2023-10-30T14:14:30.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8a9ddf0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a980c79f0 0x7f8a981f2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:30.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8a9ddf0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a980c79f0 0x7f8a981f2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41022/0 (socket says 172.21.15.57:41022) 2023-10-30T14:14:30.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8a9ddf0640 1 -- 172.21.15.57:0/341022259 learned_addr learned my addr 172.21.15.57:0/341022259 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:30.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8a9d5ef640 1 --2- 172.21.15.57:0/341022259 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a980cd2d0 0x7f8a980c8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:30.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8a9e5f1640 1 --2- 172.21.15.57:0/341022259 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a98153b20 0x7f8a980c90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:30.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8a9ddf0640 1 -- 172.21.15.57:0/341022259 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a98153b20 msgr2=0x7f8a980c90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:30.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8a9ddf0640 1 --2- 172.21.15.57:0/341022259 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a98153b20 0x7f8a980c90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8a9ddf0640 1 -- 172.21.15.57:0/341022259 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a980cd2d0 msgr2=0x7f8a980c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:30.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8a9ddf0640 1 --2- 172.21.15.57:0/341022259 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a980cd2d0 0x7f8a980c8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:30.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8a9ddf0640 1 -- 172.21.15.57:0/341022259 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a88009420 con 0x7f8a980c79f0 2023-10-30T14:14:30.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8a9ddf0640 1 --2- 172.21.15.57:0/341022259 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a980c79f0 0x7f8a981f2b30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8a9400e8c0 tx=0x7f8a9400ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:30.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8a8e7fc640 1 -- 172.21.15.57:0/341022259 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a94009620 con 0x7f8a980c79f0 2023-10-30T14:14:30.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8a8e7fc640 1 -- 172.21.15.57:0/341022259 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9400c940 con 0x7f8a980c79f0 2023-10-30T14:14:30.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8a8e7fc640 1 -- 172.21.15.57:0/341022259 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9400ccf0 con 0x7f8a980c79f0 2023-10-30T14:14:30.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8aa0054640 1 -- 172.21.15.57:0/341022259 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a98154be0 con 0x7f8a980c79f0 2023-10-30T14:14:30.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8aa0054640 1 -- 172.21.15.57:0/341022259 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a981551a0 con 0x7f8a980c79f0 2023-10-30T14:14:30.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.848+0000 7f8aa0054640 1 -- 172.21.15.57:0/341022259 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a98067540 con 0x7f8a980c79f0 2023-10-30T14:14:30.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.852+0000 7f8a8e7fc640 1 -- 172.21.15.57:0/341022259 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8a94013470 con 0x7f8a980c79f0 2023-10-30T14:14:30.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.852+0000 7f8a8e7fc640 1 --2- 172.21.15.57:0/341022259 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8a7c0664d0 0x7f8a7c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:30.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.852+0000 7f8a8e7fc640 1 -- 172.21.15.57:0/341022259 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a940840b0 con 0x7f8a980c79f0 2023-10-30T14:14:30.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.852+0000 7f8a9d5ef640 1 --2- 172.21.15.57:0/341022259 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8a7c0664d0 0x7f8a7c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:30.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.852+0000 7f8a9d5ef640 1 --2- 172.21.15.57:0/341022259 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8a7c0664d0 0x7f8a7c068990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8a880b2000 tx=0x7f8a88008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:30.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:30.856+0000 7f8a8e7fc640 1 -- 172.21.15.57:0/341022259 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8a94051400 con 0x7f8a980c79f0 2023-10-30T14:14:31.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.012+0000 7f8aa0054640 1 -- 172.21.15.57:0/341022259 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8a980c7e50 con 0x7f8a980c79f0 2023-10-30T14:14:31.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.012+0000 7f8a8e7fc640 1 -- 172.21.15.57:0/341022259 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8a94017020 con 0x7f8a980c79f0 2023-10-30T14:14:31.019 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-30T14:14:31.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.016+0000 7f8aa0054640 1 -- 172.21.15.57:0/341022259 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8a7c0664d0 msgr2=0x7f8a7c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:31.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.016+0000 7f8aa0054640 1 --2- 172.21.15.57:0/341022259 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8a7c0664d0 0x7f8a7c068990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8a880b2000 tx=0x7f8a88008040 comp rx=0 tx=0).stop 2023-10-30T14:14:31.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.016+0000 7f8aa0054640 1 -- 172.21.15.57:0/341022259 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a980c79f0 msgr2=0x7f8a981f2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:31.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.016+0000 7f8aa0054640 1 --2- 172.21.15.57:0/341022259 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a980c79f0 0x7f8a981f2b30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8a9400e8c0 tx=0x7f8a9400ed80 comp rx=0 tx=0).stop 2023-10-30T14:14:31.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.016+0000 7f8aa0054640 1 -- 172.21.15.57:0/341022259 shutdown_connections 2023-10-30T14:14:31.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.016+0000 7f8aa0054640 1 --2- 172.21.15.57:0/341022259 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8a7c0664d0 0x7f8a7c068990 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.016+0000 7f8aa0054640 1 --2- 172.21.15.57:0/341022259 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a98153b20 0x7f8a980c90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.016+0000 7f8aa0054640 1 --2- 172.21.15.57:0/341022259 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a980cd2d0 0x7f8a980c8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.016+0000 7f8aa0054640 1 --2- 172.21.15.57:0/341022259 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a980c79f0 0x7f8a981f2b30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.016+0000 7f8aa0054640 1 -- 172.21.15.57:0/341022259 >> 172.21.15.57:0/341022259 conn(0x7f8a98072010 msgr2=0x7f8a981578b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:31.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.016+0000 7f8aa0054640 1 -- 172.21.15.57:0/341022259 shutdown_connections 2023-10-30T14:14:31.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.016+0000 7f8aa0054640 1 -- 172.21.15.57:0/341022259 wait complete. 2023-10-30T14:14:31.039 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T14:14:31.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.180+0000 7f1844b0f640 1 Processor -- start 2023-10-30T14:14:31.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.180+0000 7f1844b0f640 1 -- start start 2023-10-30T14:14:31.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.180+0000 7f1844b0f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1840153ba0 0x7f18400cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.180+0000 7f1844b0f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f18400c4810 0x7f184015a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.180+0000 7f1844b0f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1840153f80 con 0x7f18400c4410 2023-10-30T14:14:31.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.180+0000 7f1844b0f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1840159df0 con 0x7f18400c4810 2023-10-30T14:14:31.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.180+0000 7f1844b0f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f184015a7c0 con 0x7f1840153ba0 2023-10-30T14:14:31.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.180+0000 7f183ed9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f18400c4810 0x7f184015a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.180+0000 7f1837fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f18400c4410 0x7f184015cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41426/0 (socket says 172.21.15.57:41426) 2023-10-30T14:14:31.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.180+0000 7f1837fff640 1 -- 172.21.15.57:0/3353666509 learned_addr learned my addr 172.21.15.57:0/3353666509 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:31.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.180+0000 7f183e59c640 1 --2- 172.21.15.57:0/3353666509 >> v2:172.21.15.88:3300/0 conn(0x7f1840153ba0 0x7f18400cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.180+0000 7f183ed9d640 1 -- 172.21.15.57:0/3353666509 >> v2:172.21.15.88:3300/0 conn(0x7f1840153ba0 msgr2=0x7f18400cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:31.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.180+0000 7f183ed9d640 1 --2- 172.21.15.57:0/3353666509 >> v2:172.21.15.88:3300/0 conn(0x7f1840153ba0 0x7f18400cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.180+0000 7f183ed9d640 1 -- 172.21.15.57:0/3353666509 >> v1:172.21.15.57:6789/0 conn(0x7f18400c4410 legacy=0x7f184015cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:31.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.180+0000 7f183ed9d640 1 -- 172.21.15.57:0/3353666509 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18401597c0 con 0x7f18400c4810 2023-10-30T14:14:31.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.180+0000 7f183dd9b640 1 -- 172.21.15.57:0/3353666509 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731815034 0 0) 0x7f1840153f80 con 0x7f18400c4410 2023-10-30T14:14:31.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.184+0000 7f183ed9d640 1 --2- 172.21.15.57:0/3353666509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18400c4810 0x7f184015a280 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f18400d0170 tx=0x7f183801c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1bd0674faf46fea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:31.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.184+0000 7f183dd9b640 1 -- 172.21.15.57:0/3353666509 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18380a5910 con 0x7f18400c4810 2023-10-30T14:14:31.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.184+0000 7f183dd9b640 1 -- 172.21.15.57:0/3353666509 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18380a5a70 con 0x7f18400c4810 2023-10-30T14:14:31.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.184+0000 7f183dd9b640 1 -- 172.21.15.57:0/3353666509 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18380a5e00 con 0x7f18400c4810 2023-10-30T14:14:31.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.184+0000 7f1844b0f640 1 -- 172.21.15.57:0/3353666509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18400c4810 msgr2=0x7f184015a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:31.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.184+0000 7f1844b0f640 1 --2- 172.21.15.57:0/3353666509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18400c4810 0x7f184015a280 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f18400d0170 tx=0x7f183801c850 comp rx=0 tx=0).stop 2023-10-30T14:14:31.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.184+0000 7f1844b0f640 1 -- 172.21.15.57:0/3353666509 shutdown_connections 2023-10-30T14:14:31.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.184+0000 7f1844b0f640 1 --2- 172.21.15.57:0/3353666509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18400c4810 0x7f184015a280 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.184+0000 7f1844b0f640 1 --2- 172.21.15.57:0/3353666509 >> v2:172.21.15.88:3300/0 conn(0x7f1840153ba0 0x7f18400cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.184+0000 7f1844b0f640 1 -- 172.21.15.57:0/3353666509 >> 172.21.15.57:0/3353666509 conn(0x7f1840071790 msgr2=0x7f1840071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:31.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.184+0000 7f1844b0f640 1 -- 172.21.15.57:0/3353666509 shutdown_connections 2023-10-30T14:14:31.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.184+0000 7f1844b0f640 1 -- 172.21.15.57:0/3353666509 wait complete. 2023-10-30T14:14:31.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f1844b0f640 1 Processor -- start 2023-10-30T14:14:31.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f1844b0f640 1 -- start start 2023-10-30T14:14:31.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f1844b0f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18400c4410 0x7f1840158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f1844b0f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18400c4810 0x7f184015be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f1844b0f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1840153ba0 0x7f18401f1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f1844b0f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18401590d0 con 0x7f1840153ba0 2023-10-30T14:14:31.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f1844b0f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18401f5af0 con 0x7f18400c4410 2023-10-30T14:14:31.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f1844b0f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184015cdf0 con 0x7f18400c4810 2023-10-30T14:14:31.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f183ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1840153ba0 0x7f18401f1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f183ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1840153ba0 0x7f18401f1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41038/0 (socket says 172.21.15.57:41038) 2023-10-30T14:14:31.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f183ed9d640 1 -- 172.21.15.57:0/10824010 learned_addr learned my addr 172.21.15.57:0/10824010 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:31.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f183e59c640 1 --2- 172.21.15.57:0/10824010 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18400c4410 0x7f1840158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f1837fff640 1 --2- 172.21.15.57:0/10824010 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18400c4810 0x7f184015be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f183ed9d640 1 -- 172.21.15.57:0/10824010 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18400c4810 msgr2=0x7f184015be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:31.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f183ed9d640 1 --2- 172.21.15.57:0/10824010 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18400c4810 0x7f184015be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f183ed9d640 1 -- 172.21.15.57:0/10824010 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18400c4410 msgr2=0x7f1840158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:31.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f183ed9d640 1 --2- 172.21.15.57:0/10824010 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18400c4410 0x7f1840158b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f183ed9d640 1 -- 172.21.15.57:0/10824010 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1838009a70 con 0x7f1840153ba0 2023-10-30T14:14:31.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f183ed9d640 1 --2- 172.21.15.57:0/10824010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1840153ba0 0x7f18401f1970 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1838019460 tx=0x7f18380195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:31.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f1836ffd640 1 -- 172.21.15.57:0/10824010 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18380bd070 con 0x7f1840153ba0 2023-10-30T14:14:31.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f1836ffd640 1 -- 172.21.15.57:0/10824010 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18380ad650 con 0x7f1840153ba0 2023-10-30T14:14:31.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f1836ffd640 1 -- 172.21.15.57:0/10824010 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18380ada00 con 0x7f1840153ba0 2023-10-30T14:14:31.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f1844b0f640 1 -- 172.21.15.57:0/10824010 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18401f4ee0 con 0x7f1840153ba0 2023-10-30T14:14:31.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.188+0000 7f1844b0f640 1 -- 172.21.15.57:0/10824010 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18401f54a0 con 0x7f1840153ba0 2023-10-30T14:14:31.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.192+0000 7f1844b0f640 1 -- 172.21.15.57:0/10824010 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f184004e170 con 0x7f1840153ba0 2023-10-30T14:14:31.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.192+0000 7f1836ffd640 1 -- 172.21.15.57:0/10824010 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f18380b4070 con 0x7f1840153ba0 2023-10-30T14:14:31.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.192+0000 7f1836ffd640 1 --2- 172.21.15.57:0/10824010 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f18200664a0 0x7f1820068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.192+0000 7f1836ffd640 1 -- 172.21.15.57:0/10824010 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f183811e8f0 con 0x7f1840153ba0 2023-10-30T14:14:31.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.192+0000 7f183e59c640 1 --2- 172.21.15.57:0/10824010 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f18200664a0 0x7f1820068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.192+0000 7f183e59c640 1 --2- 172.21.15.57:0/10824010 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f18200664a0 0x7f1820068960 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f183000b9b0 tx=0x7f1830008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:31.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.196+0000 7f1836ffd640 1 -- 172.21.15.57:0/10824010 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f18380ef7f0 con 0x7f1840153ba0 2023-10-30T14:14:31.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.352+0000 7f1844b0f640 1 -- 172.21.15.57:0/10824010 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f18400cabb0 con 0x7f1840153ba0 2023-10-30T14:14:31.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.352+0000 7f1836ffd640 1 -- 172.21.15.57:0/10824010 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f18380b2070 con 0x7f1840153ba0 2023-10-30T14:14:31.357 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-30T14:14:31.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.352+0000 7f1844b0f640 1 -- 172.21.15.57:0/10824010 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f18200664a0 msgr2=0x7f1820068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:31.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.352+0000 7f1844b0f640 1 --2- 172.21.15.57:0/10824010 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f18200664a0 0x7f1820068960 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f183000b9b0 tx=0x7f1830008040 comp rx=0 tx=0).stop 2023-10-30T14:14:31.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.352+0000 7f1844b0f640 1 -- 172.21.15.57:0/10824010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1840153ba0 msgr2=0x7f18401f1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:31.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.352+0000 7f1844b0f640 1 --2- 172.21.15.57:0/10824010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1840153ba0 0x7f18401f1970 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1838019460 tx=0x7f18380195f0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.352+0000 7f1844b0f640 1 -- 172.21.15.57:0/10824010 shutdown_connections 2023-10-30T14:14:31.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.352+0000 7f1844b0f640 1 --2- 172.21.15.57:0/10824010 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f18200664a0 0x7f1820068960 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.352+0000 7f1844b0f640 1 --2- 172.21.15.57:0/10824010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1840153ba0 0x7f18401f1970 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.352+0000 7f1844b0f640 1 --2- 172.21.15.57:0/10824010 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18400c4810 0x7f184015be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.352+0000 7f1844b0f640 1 --2- 172.21.15.57:0/10824010 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18400c4410 0x7f1840158b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.352+0000 7f1844b0f640 1 -- 172.21.15.57:0/10824010 >> 172.21.15.57:0/10824010 conn(0x7f1840071790 msgr2=0x7f18400c9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:31.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.352+0000 7f1844b0f640 1 -- 172.21.15.57:0/10824010 shutdown_connections 2023-10-30T14:14:31.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.352+0000 7f1844b0f640 1 -- 172.21.15.57:0/10824010 wait complete. 2023-10-30T14:14:31.378 INFO:tasks.mds_thrash:mds thrasher using random seed: 1698675271 2023-10-30T14:14:31.378 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:14:31.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.516+0000 7ffb6f0c9640 1 Processor -- start 2023-10-30T14:14:31.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.516+0000 7ffb6f0c9640 1 -- start start 2023-10-30T14:14:31.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.516+0000 7ffb6f0c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ffb68153c80 0x7ffb6815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.516+0000 7ffb6f0c9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ffb680c4a10 0x7ffb680c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.516+0000 7ffb6f0c9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb68159f20 con 0x7ffb6815a3b0 2023-10-30T14:14:31.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.516+0000 7ffb6f0c9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb6815dec0 con 0x7ffb680c4a10 2023-10-30T14:14:31.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.516+0000 7ffb6f0c9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb681599a0 con 0x7ffb68153c80 2023-10-30T14:14:31.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.516+0000 7ffb67fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ffb680c4a10 0x7ffb680c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.516+0000 7ffb6d666640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ffb6815a3b0 0x7ffb680ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41434/0 (socket says 172.21.15.57:41434) 2023-10-30T14:14:31.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.516+0000 7ffb6d666640 1 -- 172.21.15.57:0/2972430361 learned_addr learned my addr 172.21.15.57:0/2972430361 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:31.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.516+0000 7ffb6ce65640 1 --2- 172.21.15.57:0/2972430361 >> v2:172.21.15.88:3300/0 conn(0x7ffb68153c80 0x7ffb6815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.516+0000 7ffb67fff640 1 -- 172.21.15.57:0/2972430361 >> v2:172.21.15.88:3300/0 conn(0x7ffb68153c80 msgr2=0x7ffb6815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:31.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.516+0000 7ffb67fff640 1 --2- 172.21.15.57:0/2972430361 >> v2:172.21.15.88:3300/0 conn(0x7ffb68153c80 0x7ffb6815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.516+0000 7ffb67fff640 1 -- 172.21.15.57:0/2972430361 >> v1:172.21.15.57:6789/0 conn(0x7ffb6815a3b0 legacy=0x7ffb680ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:31.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.516+0000 7ffb67fff640 1 -- 172.21.15.57:0/2972430361 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb680c47e0 con 0x7ffb680c4a10 2023-10-30T14:14:31.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.516+0000 7ffb677fe640 1 -- 172.21.15.57:0/2972430361 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895478481 0 0) 0x7ffb68159f20 con 0x7ffb6815a3b0 2023-10-30T14:14:31.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.520+0000 7ffb67fff640 1 --2- 172.21.15.57:0/2972430361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb680c4a10 0x7ffb680c42a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ffb680d00f0 tx=0x7ffb5401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4041e5b84bd54f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:31.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.520+0000 7ffb677fe640 1 -- 172.21.15.57:0/2972430361 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb540a8950 con 0x7ffb680c4a10 2023-10-30T14:14:31.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.520+0000 7ffb677fe640 1 -- 172.21.15.57:0/2972430361 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb540a8ab0 con 0x7ffb680c4a10 2023-10-30T14:14:31.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.520+0000 7ffb677fe640 1 -- 172.21.15.57:0/2972430361 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb540a8e40 con 0x7ffb680c4a10 2023-10-30T14:14:31.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.520+0000 7ffb6f0c9640 1 -- 172.21.15.57:0/2972430361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb680c4a10 msgr2=0x7ffb680c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:31.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.520+0000 7ffb6f0c9640 1 --2- 172.21.15.57:0/2972430361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb680c4a10 0x7ffb680c42a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ffb680d00f0 tx=0x7ffb5401ec50 comp rx=0 tx=0).stop 2023-10-30T14:14:31.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.520+0000 7ffb6f0c9640 1 -- 172.21.15.57:0/2972430361 shutdown_connections 2023-10-30T14:14:31.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.520+0000 7ffb6f0c9640 1 --2- 172.21.15.57:0/2972430361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb680c4a10 0x7ffb680c42a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.520+0000 7ffb6f0c9640 1 --2- 172.21.15.57:0/2972430361 >> v2:172.21.15.88:3300/0 conn(0x7ffb68153c80 0x7ffb6815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.520+0000 7ffb6f0c9640 1 -- 172.21.15.57:0/2972430361 >> 172.21.15.57:0/2972430361 conn(0x7ffb680715c0 msgr2=0x7ffb680719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:31.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.520+0000 7ffb6f0c9640 1 -- 172.21.15.57:0/2972430361 shutdown_connections 2023-10-30T14:14:31.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.520+0000 7ffb6f0c9640 1 -- 172.21.15.57:0/2972430361 wait complete. 2023-10-30T14:14:31.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb6f0c9640 1 Processor -- start 2023-10-30T14:14:31.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb6f0c9640 1 -- start start 2023-10-30T14:14:31.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb6f0c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb680c4a10 0x7ffb6814e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb6f0c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb68153c80 0x7ffb68152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb6f0c9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6815a3b0 0x7ffb681eea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb6f0c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb68150ab0 con 0x7ffb68153c80 2023-10-30T14:14:31.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb6f0c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb68150d70 con 0x7ffb680c4a10 2023-10-30T14:14:31.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb6f0c9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681f1fe0 con 0x7ffb6815a3b0 2023-10-30T14:14:31.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb67fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb68153c80 0x7ffb68152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb67fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb68153c80 0x7ffb68152e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41054/0 (socket says 172.21.15.57:41054) 2023-10-30T14:14:31.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb67fff640 1 -- 172.21.15.57:0/2870040332 learned_addr learned my addr 172.21.15.57:0/2870040332 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:31.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb6ce65640 1 --2- 172.21.15.57:0/2870040332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb680c4a10 0x7ffb6814e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb6d666640 1 --2- 172.21.15.57:0/2870040332 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6815a3b0 0x7ffb681eea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb67fff640 1 -- 172.21.15.57:0/2870040332 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6815a3b0 msgr2=0x7ffb681eea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:31.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb67fff640 1 --2- 172.21.15.57:0/2870040332 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6815a3b0 0x7ffb681eea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb67fff640 1 -- 172.21.15.57:0/2870040332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb680c4a10 msgr2=0x7ffb6814e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:31.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb67fff640 1 --2- 172.21.15.57:0/2870040332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb680c4a10 0x7ffb6814e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb67fff640 1 -- 172.21.15.57:0/2870040332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb54009420 con 0x7ffb68153c80 2023-10-30T14:14:31.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb67fff640 1 --2- 172.21.15.57:0/2870040332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb68153c80 0x7ffb68152e90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ffb540b3040 tx=0x7ffb5401b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:31.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb657fa640 1 -- 172.21.15.57:0/2870040332 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb540b0df0 con 0x7ffb68153c80 2023-10-30T14:14:31.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb657fa640 1 -- 172.21.15.57:0/2870040332 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb540b9410 con 0x7ffb68153c80 2023-10-30T14:14:31.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb657fa640 1 -- 172.21.15.57:0/2870040332 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb540b97c0 con 0x7ffb68153c80 2023-10-30T14:14:31.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb6f0c9640 1 -- 172.21.15.57:0/2870040332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb681f22c0 con 0x7ffb68153c80 2023-10-30T14:14:31.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb6f0c9640 1 -- 172.21.15.57:0/2870040332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb681f2880 con 0x7ffb68153c80 2023-10-30T14:14:31.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.524+0000 7ffb6f0c9640 1 -- 172.21.15.57:0/2870040332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6804e170 con 0x7ffb68153c80 2023-10-30T14:14:31.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.528+0000 7ffb657fa640 1 -- 172.21.15.57:0/2870040332 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffb540b7040 con 0x7ffb68153c80 2023-10-30T14:14:31.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.528+0000 7ffb657fa640 1 --2- 172.21.15.57:0/2870040332 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffb4c0664a0 0x7ffb4c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.528+0000 7ffb657fa640 1 -- 172.21.15.57:0/2870040332 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb54121b30 con 0x7ffb68153c80 2023-10-30T14:14:31.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.528+0000 7ffb6ce65640 1 --2- 172.21.15.57:0/2870040332 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffb4c0664a0 0x7ffb4c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.528+0000 7ffb6ce65640 1 --2- 172.21.15.57:0/2870040332 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffb4c0664a0 0x7ffb4c068960 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c009af0 tx=0x7ffb5c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:31.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.532+0000 7ffb657fa640 1 -- 172.21.15.57:0/2870040332 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffb540eef10 con 0x7ffb68153c80 2023-10-30T14:14:31.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.688+0000 7ffb6f0c9640 1 -- 172.21.15.57:0/2870040332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb681f2a70 con 0x7ffb68153c80 2023-10-30T14:14:31.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.692+0000 7ffb657fa640 1 -- 172.21.15.57:0/2870040332 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16433 (secure 0 0 0) 0x7ffb540b5070 con 0x7ffb68153c80 2023-10-30T14:14:31.697 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:31.697 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:14:05.576818+0000","last_up_change":"2023-10-30T14:13:51.171287+0000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T14:13:52.574639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T14:14:02.600122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T14:14:03.069429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6801","nonce":3287367848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6803","nonce":3287367848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6807","nonce":3287367848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6805","nonce":3287367848}]},"public_addr":"172.21.15.57:6801/3287367848","cluster_addr":"172.21.15.57:6803/3287367848","heartbeat_back_addr":"172.21.15.57:6807/3287367848","heartbeat_front_addr":"172.21.15.57:6805/3287367848","state":["exists","up"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6809","nonce":3340797836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6811","nonce":3340797836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6815","nonce":3340797836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6813","nonce":3340797836}]},"public_addr":"172.21.15.57:6809/3340797836","cluster_addr":"172.21.15.57:6811/3340797836","heartbeat_back_addr":"172.21.15.57:6815/3340797836","heartbeat_front_addr":"172.21.15.57:6813/3340797836","state":["exists","up"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6825","nonce":1700292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6827","nonce":1700292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6831","nonce":1700292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6829","nonce":1700292}]},"public_addr":"172.21.15.57:6825/1700292","cluster_addr":"172.21.15.57:6827/1700292","heartbeat_back_addr":"172.21.15.57:6831/1700292","heartbeat_front_addr":"172.21.15.57:6829/1700292","state":["exists","up"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6817","nonce":2893420631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6819","nonce":2893420631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6823","nonce":2893420631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6821","nonce":2893420631}]},"public_addr":"172.21.15.57:6817/2893420631","cluster_addr":"172.21.15.57:6819/2893420631","heartbeat_back_addr":"172.21.15.57:6823/2893420631","heartbeat_front_addr":"172.21.15.57:6821/2893420631","state":["exists","up"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6807","nonce":2784816485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6811","nonce":2784816485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6815","nonce":2784816485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6813","nonce":2784816485}]},"public_addr":"172.21.15.88:6807/2784816485","cluster_addr":"172.21.15.88:6811/2784816485","heartbeat_back_addr":"172.21.15.88:6815/2784816485","heartbeat_front_addr":"172.21.15.88:6813/2784816485","state":["exists","up"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6801","nonce":2682449812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6803","nonce":2682449812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6810","nonce":2682449812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6806","nonce":2682449812}]},"public_addr":"172.21.15.88:6801/2682449812","cluster_addr":"172.21.15.88:6803/2682449812","heartbeat_back_addr":"172.21.15.88:6810/2682449812","heartbeat_front_addr":"172.21.15.88:6806/2682449812","state":["exists","up"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6822","nonce":3340693897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6827","nonce":3340693897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6831","nonce":3340693897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6829","nonce":3340693897}]},"public_addr":"172.21.15.88:6822/3340693897","cluster_addr":"172.21.15.88:6827/3340693897","heartbeat_back_addr":"172.21.15.88:6831/3340693897","heartbeat_front_addr":"172.21.15.88:6829/3340693897","state":["exists","up"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6817","nonce":2862111199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6820","nonce":2862111199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6825","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6826","nonce":2862111199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6821","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6823","nonce":2862111199}]},"public_addr":"172.21.15.88:6817/2862111199","cluster_addr":"172.21.15.88:6820/2862111199","heartbeat_back_addr":"172.21.15.88:6826/2862111199","heartbeat_front_addr":"172.21.15.88:6823/2862111199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.634920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.773993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.725825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.761452+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.694274+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.650718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:14:31.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.692+0000 7ffb6f0c9640 1 -- 172.21.15.57:0/2870040332 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffb4c0664a0 msgr2=0x7ffb4c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:31.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.692+0000 7ffb6f0c9640 1 --2- 172.21.15.57:0/2870040332 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffb4c0664a0 0x7ffb4c068960 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c009af0 tx=0x7ffb5c008040 comp rx=0 tx=0).stop 2023-10-30T14:14:31.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.692+0000 7ffb6f0c9640 1 -- 172.21.15.57:0/2870040332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb68153c80 msgr2=0x7ffb68152e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:31.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.692+0000 7ffb6f0c9640 1 --2- 172.21.15.57:0/2870040332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb68153c80 0x7ffb68152e90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ffb540b3040 tx=0x7ffb5401b9f0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.692+0000 7ffb6f0c9640 1 -- 172.21.15.57:0/2870040332 shutdown_connections 2023-10-30T14:14:31.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.692+0000 7ffb6f0c9640 1 --2- 172.21.15.57:0/2870040332 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffb4c0664a0 0x7ffb4c068960 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.692+0000 7ffb6f0c9640 1 --2- 172.21.15.57:0/2870040332 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6815a3b0 0x7ffb681eea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.692+0000 7ffb6f0c9640 1 --2- 172.21.15.57:0/2870040332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb68153c80 0x7ffb68152e90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.692+0000 7ffb6f0c9640 1 --2- 172.21.15.57:0/2870040332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb680c4a10 0x7ffb6814e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.692+0000 7ffb6f0c9640 1 -- 172.21.15.57:0/2870040332 >> 172.21.15.57:0/2870040332 conn(0x7ffb680715c0 msgr2=0x7ffb680cb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:31.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.692+0000 7ffb6f0c9640 1 -- 172.21.15.57:0/2870040332 shutdown_connections 2023-10-30T14:14:31.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.692+0000 7ffb6f0c9640 1 -- 172.21.15.57:0/2870040332 wait complete. 2023-10-30T14:14:31.719 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T14:13:52.574639+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T14:14:02.600122+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T14:14:03.069429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T14:14:31.720 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T14:14:31.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.860+0000 7fe322409640 1 Processor -- start 2023-10-30T14:14:31.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.860+0000 7fe322409640 1 -- start start 2023-10-30T14:14:31.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe322409640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe31c0c74a0 0x7fe31c0c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe322409640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31c1581f0 con 0x7fe31c151a20 2023-10-30T14:14:31.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe322409640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31c151e00 con 0x7fe31c0c1020 2023-10-30T14:14:31.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe322409640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe31c1576d0 con 0x7fe31c0c74a0 2023-10-30T14:14:31.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31bfff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fe31c151a20 0x7fe31c0cb4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41442/0 (socket says 172.21.15.57:41442) 2023-10-30T14:14:31.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31bfff640 1 -- 172.21.15.57:0/3717115915 learned_addr learned my addr 172.21.15.57:0/3717115915 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:31.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe3209a6640 1 --2- 172.21.15.57:0/3717115915 >> v2:172.21.15.88:3300/0 conn(0x7fe31c0c74a0 0x7fe31c0c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31affd640 1 -- 172.21.15.57:0/3717115915 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148708417 0 0) 0x7fe31c1581f0 con 0x7fe31c151a20 2023-10-30T14:14:31.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31affd640 1 -- 172.21.15.57:0/3717115915 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe304003530 con 0x7fe31c151a20 2023-10-30T14:14:31.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31affd640 1 -- 172.21.15.57:0/3717115915 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882081266 0 0) 0x7fe31c151e00 con 0x7fe31c0c1020 2023-10-30T14:14:31.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31affd640 1 -- 172.21.15.57:0/3717115915 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe31c1581f0 con 0x7fe31c0c1020 2023-10-30T14:14:31.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31affd640 1 -- 172.21.15.57:0/3717115915 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1155370246 0 0) 0x7fe304003530 con 0x7fe31c151a20 2023-10-30T14:14:31.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31affd640 1 -- 172.21.15.57:0/3717115915 >> v2:172.21.15.88:3300/0 conn(0x7fe31c0c74a0 msgr2=0x7fe31c0c78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:31.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31affd640 1 --2- 172.21.15.57:0/3717115915 >> v2:172.21.15.88:3300/0 conn(0x7fe31c0c74a0 0x7fe31c0c78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31affd640 1 -- 172.21.15.57:0/3717115915 >> v1:172.21.15.88:6789/0 conn(0x7fe31c0c1020 legacy=0x7fe31c072080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:31.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31affd640 1 -- 172.21.15.57:0/3717115915 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31c0c19d0 con 0x7fe31c151a20 2023-10-30T14:14:31.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31affd640 1 -- 172.21.15.57:0/3717115915 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fe30c001680 con 0x7fe31c151a20 2023-10-30T14:14:31.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31affd640 1 -- 172.21.15.57:0/3717115915 >> v1:172.21.15.57:6789/0 conn(0x7fe31c151a20 legacy=0x7fe31c0cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:31.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31affd640 1 --2- 172.21.15.57:0/3717115915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe304003b60 0x7fe304003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31affd640 1 --2- 172.21.15.57:0/3717115915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3040047d0 0x7fe304004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31affd640 1 --2- 172.21.15.57:0/3717115915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe304005190 0x7fe30400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31affd640 1 -- 172.21.15.57:0/3717115915 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30400bf80 con 0x7fe304003b60 2023-10-30T14:14:31.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31affd640 1 -- 172.21.15.57:0/3717115915 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30400c0f0 con 0x7fe3040047d0 2023-10-30T14:14:31.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31affd640 1 -- 172.21.15.57:0/3717115915 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe30400c2b0 con 0x7fe304005190 2023-10-30T14:14:31.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31bfff640 1 --2- 172.21.15.57:0/3717115915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe304003b60 0x7fe304003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31b7fe640 1 --2- 172.21.15.57:0/3717115915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3040047d0 0x7fe304004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe3209a6640 1 --2- 172.21.15.57:0/3717115915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe304005190 0x7fe30400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31bfff640 1 -- 172.21.15.57:0/3717115915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe304005190 msgr2=0x7fe30400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:31.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31bfff640 1 --2- 172.21.15.57:0/3717115915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe304005190 0x7fe30400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31bfff640 1 -- 172.21.15.57:0/3717115915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3040047d0 msgr2=0x7fe304004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:31.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31bfff640 1 --2- 172.21.15.57:0/3717115915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3040047d0 0x7fe304004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31bfff640 1 -- 172.21.15.57:0/3717115915 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe304003530 con 0x7fe304003b60 2023-10-30T14:14:31.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe31b7fe640 1 --2- 172.21.15.57:0/3717115915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3040047d0 0x7fe304004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:31.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.864+0000 7fe3209a6640 1 -- 172.21.15.57:0/3717115915 reap_dead start 2023-10-30T14:14:31.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.868+0000 7fe31bfff640 1 --2- 172.21.15.57:0/3717115915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe304003b60 0x7fe304003f60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe30c007f80 tx=0x7fe30c01cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa29dd97bc82a8ae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:31.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.868+0000 7fe31affd640 1 -- 172.21.15.57:0/3717115915 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c005ae0 con 0x7fe304003b60 2023-10-30T14:14:31.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.868+0000 7fe31affd640 1 -- 172.21.15.57:0/3717115915 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30c005c40 con 0x7fe304003b60 2023-10-30T14:14:31.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.868+0000 7fe31affd640 1 -- 172.21.15.57:0/3717115915 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c0a5820 con 0x7fe304003b60 2023-10-30T14:14:31.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.868+0000 7fe322409640 1 -- 172.21.15.57:0/3717115915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe304003b60 msgr2=0x7fe304003f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:31.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.868+0000 7fe322409640 1 --2- 172.21.15.57:0/3717115915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe304003b60 0x7fe304003f60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe30c007f80 tx=0x7fe30c01cb20 comp rx=0 tx=0).stop 2023-10-30T14:14:31.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.868+0000 7fe322409640 1 -- 172.21.15.57:0/3717115915 shutdown_connections 2023-10-30T14:14:31.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.868+0000 7fe322409640 1 --2- 172.21.15.57:0/3717115915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe304003b60 0x7fe304003f60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.868+0000 7fe322409640 1 -- 172.21.15.57:0/3717115915 >> 172.21.15.57:0/3717115915 conn(0x7fe31c06da40 msgr2=0x7fe31c06de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:31.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.868+0000 7fe322409640 1 -- 172.21.15.57:0/3717115915 shutdown_connections 2023-10-30T14:14:31.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.868+0000 7fe322409640 1 -- 172.21.15.57:0/3717115915 wait complete. 2023-10-30T14:14:31.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe322409640 1 Processor -- start 2023-10-30T14:14:31.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe322409640 1 -- start start 2023-10-30T14:14:31.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe322409640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe304003b60 0x7fe31c1f0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe322409640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe31c1f0590 0x7fe31c0c8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe322409640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe31c0c9380 0x7fe31c1f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe322409640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c157340 con 0x7fe31c0c9380 2023-10-30T14:14:31.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe322409640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c157760 con 0x7fe304003b60 2023-10-30T14:14:31.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe322409640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c0c3ee0 con 0x7fe31c1f0590 2023-10-30T14:14:31.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe3209a6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe31c0c9380 0x7fe31c1f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe3209a6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe31c0c9380 0x7fe31c1f2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41066/0 (socket says 172.21.15.57:41066) 2023-10-30T14:14:31.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe3209a6640 1 -- 172.21.15.57:0/1679656010 learned_addr learned my addr 172.21.15.57:0/1679656010 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:31.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe31b7fe640 1 --2- 172.21.15.57:0/1679656010 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe31c1f0590 0x7fe31c0c8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe31bfff640 1 --2- 172.21.15.57:0/1679656010 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe304003b60 0x7fe31c1f0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe3209a6640 1 -- 172.21.15.57:0/1679656010 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe31c1f0590 msgr2=0x7fe31c0c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:31.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe3209a6640 1 --2- 172.21.15.57:0/1679656010 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe31c1f0590 0x7fe31c0c8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe3209a6640 1 -- 172.21.15.57:0/1679656010 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe304003b60 msgr2=0x7fe31c1f0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:31.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe3209a6640 1 --2- 172.21.15.57:0/1679656010 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe304003b60 0x7fe31c1f0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:31.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe3209a6640 1 -- 172.21.15.57:0/1679656010 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31c0c46b0 con 0x7fe31c0c9380 2023-10-30T14:14:31.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe3209a6640 1 --2- 172.21.15.57:0/1679656010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe31c0c9380 0x7fe31c1f2150 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe31c06e980 tx=0x7fe3140115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:31.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe318ff9640 1 -- 172.21.15.57:0/1679656010 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31400e070 con 0x7fe31c0c9380 2023-10-30T14:14:31.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe318ff9640 1 -- 172.21.15.57:0/1679656010 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe314011e10 con 0x7fe31c0c9380 2023-10-30T14:14:31.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe318ff9640 1 -- 172.21.15.57:0/1679656010 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3140039a0 con 0x7fe31c0c9380 2023-10-30T14:14:31.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe322409640 1 -- 172.21.15.57:0/1679656010 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe31c0ca940 con 0x7fe31c0c9380 2023-10-30T14:14:31.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe322409640 1 -- 172.21.15.57:0/1679656010 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31c0c40f0 con 0x7fe31c0c9380 2023-10-30T14:14:31.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.872+0000 7fe322409640 1 -- 172.21.15.57:0/1679656010 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe31c1578a0 con 0x7fe31c0c9380 2023-10-30T14:14:31.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.876+0000 7fe318ff9640 1 -- 172.21.15.57:0/1679656010 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe314003b00 con 0x7fe31c0c9380 2023-10-30T14:14:31.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.876+0000 7fe318ff9640 1 --2- 172.21.15.57:0/1679656010 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3080664a0 0x7fe308068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:31.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.876+0000 7fe318ff9640 1 -- 172.21.15.57:0/1679656010 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe314084780 con 0x7fe31c0c9380 2023-10-30T14:14:31.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.876+0000 7fe31bfff640 1 --2- 172.21.15.57:0/1679656010 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3080664a0 0x7fe308068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:31.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.876+0000 7fe31bfff640 1 --2- 172.21.15.57:0/1679656010 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3080664a0 0x7fe308068960 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0ab000 tx=0x7fe30c0b1040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:31.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:31.880+0000 7fe318ff9640 1 -- 172.21.15.57:0/1679656010 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe31c1578a0 con 0x7fe31c0c9380 2023-10-30T14:14:32.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.040+0000 7fe322409640 1 -- 172.21.15.57:0/1679656010 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe31c157480 con 0x7fe31c0c9380 2023-10-30T14:14:32.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.040+0000 7fe318ff9640 1 -- 172.21.15.57:0/1679656010 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe31c157480 con 0x7fe31c0c9380 2023-10-30T14:14:32.045 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-30T14:14:32.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.040+0000 7fe322409640 1 -- 172.21.15.57:0/1679656010 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3080664a0 msgr2=0x7fe308068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:32.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.040+0000 7fe322409640 1 --2- 172.21.15.57:0/1679656010 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3080664a0 0x7fe308068960 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0ab000 tx=0x7fe30c0b1040 comp rx=0 tx=0).stop 2023-10-30T14:14:32.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.040+0000 7fe322409640 1 -- 172.21.15.57:0/1679656010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe31c0c9380 msgr2=0x7fe31c1f2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:32.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.040+0000 7fe322409640 1 --2- 172.21.15.57:0/1679656010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe31c0c9380 0x7fe31c1f2150 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe31c06e980 tx=0x7fe3140115a0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.040+0000 7fe322409640 1 -- 172.21.15.57:0/1679656010 shutdown_connections 2023-10-30T14:14:32.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.040+0000 7fe322409640 1 --2- 172.21.15.57:0/1679656010 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3080664a0 0x7fe308068960 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.040+0000 7fe322409640 1 --2- 172.21.15.57:0/1679656010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe31c0c9380 0x7fe31c1f2150 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.040+0000 7fe322409640 1 --2- 172.21.15.57:0/1679656010 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe31c1f0590 0x7fe31c0c8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.040+0000 7fe322409640 1 --2- 172.21.15.57:0/1679656010 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe304003b60 0x7fe31c1f0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.040+0000 7fe322409640 1 -- 172.21.15.57:0/1679656010 >> 172.21.15.57:0/1679656010 conn(0x7fe31c06da40 msgr2=0x7fe31c071270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:32.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.040+0000 7fe322409640 1 -- 172.21.15.57:0/1679656010 shutdown_connections 2023-10-30T14:14:32.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.040+0000 7fe322409640 1 -- 172.21.15.57:0/1679656010 wait complete. 2023-10-30T14:14:32.066 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T14:14:32.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f973869a640 1 Processor -- start 2023-10-30T14:14:32.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f973869a640 1 -- start start 2023-10-30T14:14:32.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f973869a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f97300c1020 0x7f9730072080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f973869a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97300725c0 0x7f97300c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f973869a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97301581f0 con 0x7f9730151a20 2023-10-30T14:14:32.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f973869a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97301576d0 con 0x7f97300c1020 2023-10-30T14:14:32.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f973869a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9730158680 con 0x7f97300725c0 2023-10-30T14:14:32.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9736436640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f9730151a20 0x7f97300cb4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41452/0 (socket says 172.21.15.57:41452) 2023-10-30T14:14:32.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9736436640 1 -- 172.21.15.57:0/4248253326 learned_addr learned my addr 172.21.15.57:0/4248253326 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:32.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9736c37640 1 --2- 172.21.15.57:0/4248253326 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97300725c0 0x7f97300c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735c35640 1 --2- 172.21.15.57:0/4248253326 >> v2:172.21.15.88:3300/0 conn(0x7f97300c1020 0x7f9730072080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735434640 1 -- 172.21.15.57:0/4248253326 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477697146 0 0) 0x7f97301581f0 con 0x7f9730151a20 2023-10-30T14:14:32.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735434640 1 -- 172.21.15.57:0/4248253326 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f971c003530 con 0x7f9730151a20 2023-10-30T14:14:32.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735434640 1 -- 172.21.15.57:0/4248253326 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 975650256 0 0) 0x7f971c003530 con 0x7f9730151a20 2023-10-30T14:14:32.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735434640 1 -- 172.21.15.57:0/4248253326 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97300725c0 msgr2=0x7f97300c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:32.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735434640 1 --2- 172.21.15.57:0/4248253326 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97300725c0 0x7f97300c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735434640 1 -- 172.21.15.57:0/4248253326 >> v2:172.21.15.88:3300/0 conn(0x7f97300c1020 msgr2=0x7f9730072080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:32.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735434640 1 --2- 172.21.15.57:0/4248253326 >> v2:172.21.15.88:3300/0 conn(0x7f97300c1020 0x7f9730072080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735434640 1 -- 172.21.15.57:0/4248253326 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9730151e00 con 0x7f9730151a20 2023-10-30T14:14:32.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735434640 1 -- 172.21.15.57:0/4248253326 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f972c001680 con 0x7f9730151a20 2023-10-30T14:14:32.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735434640 1 -- 172.21.15.57:0/4248253326 >> v1:172.21.15.57:6789/0 conn(0x7f9730151a20 legacy=0x7f97300cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:32.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735434640 1 --2- 172.21.15.57:0/4248253326 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97300725c0 0x7f971c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735434640 1 --2- 172.21.15.57:0/4248253326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f971c003f30 0x7f971c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735434640 1 --2- 172.21.15.57:0/4248253326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f971c004d10 0x7f971c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735434640 1 -- 172.21.15.57:0/4248253326 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9730158680 con 0x7f971c003f30 2023-10-30T14:14:32.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735434640 1 -- 172.21.15.57:0/4248253326 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c00b990 con 0x7f971c004d10 2023-10-30T14:14:32.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735434640 1 -- 172.21.15.57:0/4248253326 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f971c00bb50 con 0x7f97300725c0 2023-10-30T14:14:32.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735c35640 1 --2- 172.21.15.57:0/4248253326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f971c003f30 0x7f971c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9736436640 1 --2- 172.21.15.57:0/4248253326 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97300725c0 0x7f971c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9736c37640 1 --2- 172.21.15.57:0/4248253326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f971c004d10 0x7f971c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735c35640 1 -- 172.21.15.57:0/4248253326 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97300725c0 msgr2=0x7f971c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:32.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735c35640 1 --2- 172.21.15.57:0/4248253326 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97300725c0 0x7f971c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735c35640 1 -- 172.21.15.57:0/4248253326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f971c004d10 msgr2=0x7f971c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:32.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735c35640 1 --2- 172.21.15.57:0/4248253326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f971c004d10 0x7f971c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9735c35640 1 -- 172.21.15.57:0/4248253326 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971c003530 con 0x7f971c003f30 2023-10-30T14:14:32.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.208+0000 7f9736436640 1 --2- 172.21.15.57:0/4248253326 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97300725c0 0x7f971c0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:32.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.212+0000 7f9735c35640 1 --2- 172.21.15.57:0/4248253326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f971c003f30 0x7f971c0047d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9720009ae0 tx=0x7f972001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b48cf56a5319da server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:32.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.212+0000 7f9735434640 1 -- 172.21.15.57:0/4248253326 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97200a5680 con 0x7f971c003f30 2023-10-30T14:14:32.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.212+0000 7f9735434640 1 -- 172.21.15.57:0/4248253326 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97200a57e0 con 0x7f971c003f30 2023-10-30T14:14:32.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.212+0000 7f9735434640 1 -- 172.21.15.57:0/4248253326 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97200a5b90 con 0x7f971c003f30 2023-10-30T14:14:32.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.212+0000 7f973869a640 1 -- 172.21.15.57:0/4248253326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f971c003f30 msgr2=0x7f971c0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:32.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.212+0000 7f973869a640 1 --2- 172.21.15.57:0/4248253326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f971c003f30 0x7f971c0047d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9720009ae0 tx=0x7f972001ca90 comp rx=0 tx=0).stop 2023-10-30T14:14:32.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.212+0000 7f9736c37640 1 -- 172.21.15.57:0/4248253326 reap_dead start 2023-10-30T14:14:32.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.212+0000 7f973869a640 1 -- 172.21.15.57:0/4248253326 shutdown_connections 2023-10-30T14:14:32.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.212+0000 7f973869a640 1 -- 172.21.15.57:0/4248253326 >> 172.21.15.57:0/4248253326 conn(0x7f973006da40 msgr2=0x7f973006de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:32.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.212+0000 7f973869a640 1 -- 172.21.15.57:0/4248253326 shutdown_connections 2023-10-30T14:14:32.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.212+0000 7f973869a640 1 -- 172.21.15.57:0/4248253326 wait complete. 2023-10-30T14:14:32.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f973869a640 1 Processor -- start 2023-10-30T14:14:32.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f973869a640 1 -- start start 2023-10-30T14:14:32.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f973869a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f971c003f30 0x7f973014d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f973869a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9730155840 0x7f973014ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f973869a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9730153f50 0x7f97301543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f973869a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97301ec630 con 0x7f9730153f50 2023-10-30T14:14:32.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f973869a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97301ed5c0 con 0x7f9730155840 2023-10-30T14:14:32.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f973869a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f973014d6d0 con 0x7f971c003f30 2023-10-30T14:14:32.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f9736436640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f971c003f30 0x7f973014d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f9736c37640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9730153f50 0x7f97301543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f9736436640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f971c003f30 0x7f973014d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:51798/0 (socket says 172.21.15.57:51798) 2023-10-30T14:14:32.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f9736436640 1 -- 172.21.15.57:0/195609270 learned_addr learned my addr 172.21.15.57:0/195609270 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:32.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f9736c37640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9730153f50 0x7f97301543d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41090/0 (socket says 172.21.15.57:41090) 2023-10-30T14:14:32.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f9735c35640 1 --2- 172.21.15.57:0/195609270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9730155840 0x7f973014ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f9736c37640 1 -- 172.21.15.57:0/195609270 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f971c003f30 msgr2=0x7f973014d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:32.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f9736c37640 1 --2- 172.21.15.57:0/195609270 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f971c003f30 0x7f973014d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f9736c37640 1 -- 172.21.15.57:0/195609270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9730155840 msgr2=0x7f973014ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:32.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f9736c37640 1 --2- 172.21.15.57:0/195609270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9730155840 0x7f973014ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f9736c37640 1 -- 172.21.15.57:0/195609270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97301501f0 con 0x7f9730153f50 2023-10-30T14:14:32.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f9736c37640 1 --2- 172.21.15.57:0/195609270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9730153f50 0x7f97301543d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f97300c8860 tx=0x7f97280115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:32.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f9726ffd640 1 -- 172.21.15.57:0/195609270 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972800e070 con 0x7f9730153f50 2023-10-30T14:14:32.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f9726ffd640 1 -- 172.21.15.57:0/195609270 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9728011e10 con 0x7f9730153f50 2023-10-30T14:14:32.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f9726ffd640 1 -- 172.21.15.57:0/195609270 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97280039a0 con 0x7f9730153f50 2023-10-30T14:14:32.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f973869a640 1 -- 172.21.15.57:0/195609270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f973014fcb0 con 0x7f9730153f50 2023-10-30T14:14:32.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.216+0000 7f973869a640 1 -- 172.21.15.57:0/195609270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f973014ff10 con 0x7f9730153f50 2023-10-30T14:14:32.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.220+0000 7f973869a640 1 -- 172.21.15.57:0/195609270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f973004e170 con 0x7f9730153f50 2023-10-30T14:14:32.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.220+0000 7f9726ffd640 1 -- 172.21.15.57:0/195609270 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9728003b00 con 0x7f9730153f50 2023-10-30T14:14:32.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.220+0000 7f9726ffd640 1 --2- 172.21.15.57:0/195609270 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f97140664a0 0x7f9714068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.220+0000 7f9726ffd640 1 -- 172.21.15.57:0/195609270 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9728084140 con 0x7f9730153f50 2023-10-30T14:14:32.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.220+0000 7f9736436640 1 --2- 172.21.15.57:0/195609270 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f97140664a0 0x7f9714068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.220+0000 7f9736436640 1 --2- 172.21.15.57:0/195609270 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f97140664a0 0x7f9714068960 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f972c001bc0 tx=0x7f972c007040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:32.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.224+0000 7f9726ffd640 1 -- 172.21.15.57:0/195609270 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f97280515d0 con 0x7f9730153f50 2023-10-30T14:14:32.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.380+0000 7f973869a640 1 -- 172.21.15.57:0/195609270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f97301501f0 con 0x7f9730153f50 2023-10-30T14:14:32.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.380+0000 7f9726ffd640 1 -- 172.21.15.57:0/195609270 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f97301501f0 con 0x7f9730153f50 2023-10-30T14:14:32.386 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-30T14:14:32.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.380+0000 7f973869a640 1 -- 172.21.15.57:0/195609270 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f97140664a0 msgr2=0x7f9714068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:32.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.380+0000 7f973869a640 1 --2- 172.21.15.57:0/195609270 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f97140664a0 0x7f9714068960 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f972c001bc0 tx=0x7f972c007040 comp rx=0 tx=0).stop 2023-10-30T14:14:32.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.380+0000 7f973869a640 1 -- 172.21.15.57:0/195609270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9730153f50 msgr2=0x7f97301543d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:32.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.380+0000 7f973869a640 1 --2- 172.21.15.57:0/195609270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9730153f50 0x7f97301543d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f97300c8860 tx=0x7f97280115a0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.380+0000 7f973869a640 1 -- 172.21.15.57:0/195609270 shutdown_connections 2023-10-30T14:14:32.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.380+0000 7f973869a640 1 --2- 172.21.15.57:0/195609270 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f97140664a0 0x7f9714068960 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.380+0000 7f973869a640 1 --2- 172.21.15.57:0/195609270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9730153f50 0x7f97301543d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.380+0000 7f973869a640 1 --2- 172.21.15.57:0/195609270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9730155840 0x7f973014ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.380+0000 7f973869a640 1 --2- 172.21.15.57:0/195609270 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f971c003f30 0x7f973014d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.380+0000 7f973869a640 1 -- 172.21.15.57:0/195609270 >> 172.21.15.57:0/195609270 conn(0x7f973006da40 msgr2=0x7f97300c7380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:32.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.380+0000 7f973869a640 1 -- 172.21.15.57:0/195609270 shutdown_connections 2023-10-30T14:14:32.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.380+0000 7f973869a640 1 -- 172.21.15.57:0/195609270 wait complete. 2023-10-30T14:14:32.406 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T14:14:32.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.544+0000 7f4c37dab640 1 Processor -- start 2023-10-30T14:14:32.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.544+0000 7f4c37dab640 1 -- start start 2023-10-30T14:14:32.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.548+0000 7f4c37dab640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4c300b2000 0x7f4c3006cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.548+0000 7f4c37dab640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4c3015dcc0 0x7f4c30159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.548+0000 7f4c37dab640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3006b040 con 0x7f4c3006efb0 2023-10-30T14:14:32.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.548+0000 7f4c37dab640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c3006b600 con 0x7f4c300b2000 2023-10-30T14:14:32.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.548+0000 7f4c37dab640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c3006c520 con 0x7f4c3015dcc0 2023-10-30T14:14:32.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.548+0000 7f4c35b47640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4c300b2000 0x7f4c3006cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.548+0000 7f4c35b47640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4c300b2000 0x7f4c3006cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41098/0 (socket says 172.21.15.57:41098) 2023-10-30T14:14:32.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.548+0000 7f4c35b47640 1 -- 172.21.15.57:0/2928081123 learned_addr learned my addr 172.21.15.57:0/2928081123 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:32.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.548+0000 7f4c35346640 1 --2- 172.21.15.57:0/2928081123 >> v2:172.21.15.88:3300/0 conn(0x7f4c3015dcc0 0x7f4c30159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.548+0000 7f4c35b47640 1 -- 172.21.15.57:0/2928081123 >> v2:172.21.15.88:3300/0 conn(0x7f4c3015dcc0 msgr2=0x7f4c30159ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:32.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.548+0000 7f4c35b47640 1 --2- 172.21.15.57:0/2928081123 >> v2:172.21.15.88:3300/0 conn(0x7f4c3015dcc0 0x7f4c30159ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.548+0000 7f4c35b47640 1 -- 172.21.15.57:0/2928081123 >> v1:172.21.15.88:6789/0 conn(0x7f4c3006efb0 legacy=0x7f4c3006e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:32.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.548+0000 7f4c35b47640 1 -- 172.21.15.57:0/2928081123 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3006b320 con 0x7f4c300b2000 2023-10-30T14:14:32.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.552+0000 7f4c35b47640 1 --2- 172.21.15.57:0/2928081123 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c300b2000 0x7f4c3006cb70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4c300c7630 tx=0x7f4c2c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd0b1133ef1a05a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:32.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.552+0000 7f4c34b45640 1 -- 172.21.15.57:0/2928081123 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c0a89f0 con 0x7f4c300b2000 2023-10-30T14:14:32.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.552+0000 7f4c34b45640 1 -- 172.21.15.57:0/2928081123 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2c0a8b50 con 0x7f4c300b2000 2023-10-30T14:14:32.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.552+0000 7f4c34b45640 1 -- 172.21.15.57:0/2928081123 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c0b0440 con 0x7f4c300b2000 2023-10-30T14:14:32.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.552+0000 7f4c37dab640 1 -- 172.21.15.57:0/2928081123 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c300b2000 msgr2=0x7f4c3006cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:32.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.552+0000 7f4c37dab640 1 --2- 172.21.15.57:0/2928081123 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c300b2000 0x7f4c3006cb70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4c300c7630 tx=0x7f4c2c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.552+0000 7f4c37dab640 1 -- 172.21.15.57:0/2928081123 shutdown_connections 2023-10-30T14:14:32.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.552+0000 7f4c37dab640 1 --2- 172.21.15.57:0/2928081123 >> v2:172.21.15.88:3300/0 conn(0x7f4c3015dcc0 0x7f4c30159ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.552+0000 7f4c37dab640 1 --2- 172.21.15.57:0/2928081123 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c300b2000 0x7f4c3006cb70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.552+0000 7f4c37dab640 1 -- 172.21.15.57:0/2928081123 >> 172.21.15.57:0/2928081123 conn(0x7f4c300723f0 msgr2=0x7f4c30072800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:32.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.552+0000 7f4c37dab640 1 -- 172.21.15.57:0/2928081123 shutdown_connections 2023-10-30T14:14:32.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.552+0000 7f4c37dab640 1 -- 172.21.15.57:0/2928081123 wait complete. 2023-10-30T14:14:32.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.552+0000 7f4c37dab640 1 Processor -- start 2023-10-30T14:14:32.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.552+0000 7f4c37dab640 1 -- start start 2023-10-30T14:14:32.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c37dab640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c3006efb0 0x7f4c301f3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c37dab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c300b2000 0x7f4c301ef9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c37dab640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c3015dcc0 0x7f4c30152990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c37dab640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3015bdc0 con 0x7f4c300b2000 2023-10-30T14:14:32.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c37dab640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c301ef540 con 0x7f4c3015dcc0 2023-10-30T14:14:32.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c37dab640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c301f3550 con 0x7f4c3006efb0 2023-10-30T14:14:32.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c35346640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c300b2000 0x7f4c301ef9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c35346640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c300b2000 0x7f4c301ef9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41112/0 (socket says 172.21.15.57:41112) 2023-10-30T14:14:32.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c35346640 1 -- 172.21.15.57:0/2246163053 learned_addr learned my addr 172.21.15.57:0/2246163053 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:32.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c35b47640 1 --2- 172.21.15.57:0/2246163053 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c3006efb0 0x7f4c301f3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c36348640 1 --2- 172.21.15.57:0/2246163053 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c3015dcc0 0x7f4c30152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c35346640 1 -- 172.21.15.57:0/2246163053 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c3006efb0 msgr2=0x7f4c301f3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:32.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c35346640 1 --2- 172.21.15.57:0/2246163053 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c3006efb0 0x7f4c301f3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c35346640 1 -- 172.21.15.57:0/2246163053 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c3015dcc0 msgr2=0x7f4c30152990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:32.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c35346640 1 --2- 172.21.15.57:0/2246163053 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c3015dcc0 0x7f4c30152990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c35346640 1 -- 172.21.15.57:0/2246163053 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c0094d0 con 0x7f4c300b2000 2023-10-30T14:14:32.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c35346640 1 --2- 172.21.15.57:0/2246163053 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c300b2000 0x7f4c301ef9d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000d5f0 tx=0x7f4c2000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:32.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c267fc640 1 -- 172.21.15.57:0/2246163053 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c200125f0 con 0x7f4c300b2000 2023-10-30T14:14:32.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c267fc640 1 -- 172.21.15.57:0/2246163053 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c20012750 con 0x7f4c300b2000 2023-10-30T14:14:32.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c267fc640 1 -- 172.21.15.57:0/2246163053 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c20012ae0 con 0x7f4c300b2000 2023-10-30T14:14:32.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c37dab640 1 -- 172.21.15.57:0/2246163053 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c3015c010 con 0x7f4c300b2000 2023-10-30T14:14:32.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c37dab640 1 -- 172.21.15.57:0/2246163053 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c3015c5d0 con 0x7f4c300b2000 2023-10-30T14:14:32.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c37dab640 1 -- 172.21.15.57:0/2246163053 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c301ef6b0 con 0x7f4c300b2000 2023-10-30T14:14:32.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.556+0000 7f4c267fc640 1 -- 172.21.15.57:0/2246163053 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4c20012c40 con 0x7f4c300b2000 2023-10-30T14:14:32.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.560+0000 7f4c267fc640 1 --2- 172.21.15.57:0/2246163053 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4c140664d0 0x7f4c14068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.560+0000 7f4c267fc640 1 -- 172.21.15.57:0/2246163053 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c20084370 con 0x7f4c300b2000 2023-10-30T14:14:32.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.560+0000 7f4c35b47640 1 --2- 172.21.15.57:0/2246163053 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4c140664d0 0x7f4c14068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.560+0000 7f4c35b47640 1 --2- 172.21.15.57:0/2246163053 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4c140664d0 0x7f4c14068990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c0b2000 tx=0x7f4c2c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:32.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.564+0000 7f4c267fc640 1 -- 172.21.15.57:0/2246163053 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4c200516c0 con 0x7f4c300b2000 2023-10-30T14:14:32.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.720+0000 7f4c37dab640 1 -- 172.21.15.57:0/2246163053 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4c3004e170 con 0x7f4c300b2000 2023-10-30T14:14:32.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.720+0000 7f4c267fc640 1 -- 172.21.15.57:0/2246163053 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4c20004650 con 0x7f4c300b2000 2023-10-30T14:14:32.727 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-30T14:14:32.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.724+0000 7f4c37dab640 1 -- 172.21.15.57:0/2246163053 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4c140664d0 msgr2=0x7f4c14068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:32.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.724+0000 7f4c37dab640 1 --2- 172.21.15.57:0/2246163053 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4c140664d0 0x7f4c14068990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c0b2000 tx=0x7f4c2c009040 comp rx=0 tx=0).stop 2023-10-30T14:14:32.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.724+0000 7f4c37dab640 1 -- 172.21.15.57:0/2246163053 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c300b2000 msgr2=0x7f4c301ef9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:32.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.724+0000 7f4c37dab640 1 --2- 172.21.15.57:0/2246163053 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c300b2000 0x7f4c301ef9d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000d5f0 tx=0x7f4c2000dab0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.724+0000 7f4c37dab640 1 -- 172.21.15.57:0/2246163053 shutdown_connections 2023-10-30T14:14:32.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.724+0000 7f4c37dab640 1 --2- 172.21.15.57:0/2246163053 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4c140664d0 0x7f4c14068990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.724+0000 7f4c37dab640 1 --2- 172.21.15.57:0/2246163053 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c3015dcc0 0x7f4c30152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.724+0000 7f4c37dab640 1 --2- 172.21.15.57:0/2246163053 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c300b2000 0x7f4c301ef9d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.724+0000 7f4c37dab640 1 --2- 172.21.15.57:0/2246163053 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c3006efb0 0x7f4c301f3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.724+0000 7f4c37dab640 1 -- 172.21.15.57:0/2246163053 >> 172.21.15.57:0/2246163053 conn(0x7f4c300723f0 msgr2=0x7f4c300660e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:32.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.724+0000 7f4c37dab640 1 -- 172.21.15.57:0/2246163053 shutdown_connections 2023-10-30T14:14:32.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.724+0000 7f4c37dab640 1 -- 172.21.15.57:0/2246163053 wait complete. 2023-10-30T14:14:32.748 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2023-10-30T14:14:32.748 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:32.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.888+0000 7fc6ce171640 1 Processor -- start 2023-10-30T14:14:32.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.888+0000 7fc6ce171640 1 -- start start 2023-10-30T14:14:32.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.888+0000 7fc6ce171640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc6c8153b50 0x7fc6c81597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.888+0000 7fc6ce171640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6c80c62b0 0x7fc6c80ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.888+0000 7fc6ce171640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c815a770 con 0x7fc6c80cb180 2023-10-30T14:14:32.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.888+0000 7fc6ce171640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6c8159cf0 con 0x7fc6c8153b50 2023-10-30T14:14:32.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.888+0000 7fc6ce171640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c806aa30 con 0x7fc6c80c62b0 2023-10-30T14:14:32.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.888+0000 7fc6cd16f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc6c8153b50 0x7fc6c81597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.888+0000 7fc6cd970640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc6c80cb180 0x7fc6c806ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41460/0 (socket says 172.21.15.57:41460) 2023-10-30T14:14:32.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.888+0000 7fc6cd970640 1 -- 172.21.15.57:0/746268856 learned_addr learned my addr 172.21.15.57:0/746268856 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:32.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.888+0000 7fc6cc96e640 1 --2- 172.21.15.57:0/746268856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6c80c62b0 0x7fc6c80ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.888+0000 7fc6cd16f640 1 -- 172.21.15.57:0/746268856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6c80c62b0 msgr2=0x7fc6c80ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:32.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.888+0000 7fc6cd16f640 1 --2- 172.21.15.57:0/746268856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6c80c62b0 0x7fc6c80ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.888+0000 7fc6cd16f640 1 -- 172.21.15.57:0/746268856 >> v1:172.21.15.57:6789/0 conn(0x7fc6c80cb180 legacy=0x7fc6c806ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:32.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.888+0000 7fc6cd16f640 1 -- 172.21.15.57:0/746268856 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c80c9d40 con 0x7fc6c8153b50 2023-10-30T14:14:32.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.888+0000 7fc6cc96e640 1 --2- 172.21.15.57:0/746268856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6c80c62b0 0x7fc6c80ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:32.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.892+0000 7fc6cd16f640 1 --2- 172.21.15.57:0/746268856 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6c8153b50 0x7fc6c81597b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc6c40097c0 tx=0x7fc6c401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=848117badc779ede server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:32.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.892+0000 7fc6bffff640 1 -- 172.21.15.57:0/746268856 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c40ab600 con 0x7fc6c8153b50 2023-10-30T14:14:32.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.892+0000 7fc6bffff640 1 -- 172.21.15.57:0/746268856 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c40a7050 con 0x7fc6c8153b50 2023-10-30T14:14:32.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.892+0000 7fc6bffff640 1 -- 172.21.15.57:0/746268856 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c40ab980 con 0x7fc6c8153b50 2023-10-30T14:14:32.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.892+0000 7fc6ce171640 1 -- 172.21.15.57:0/746268856 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6c8153b50 msgr2=0x7fc6c81597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:32.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.892+0000 7fc6ce171640 1 --2- 172.21.15.57:0/746268856 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6c8153b50 0x7fc6c81597b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc6c40097c0 tx=0x7fc6c401c480 comp rx=0 tx=0).stop 2023-10-30T14:14:32.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.892+0000 7fc6ce171640 1 -- 172.21.15.57:0/746268856 shutdown_connections 2023-10-30T14:14:32.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.892+0000 7fc6ce171640 1 --2- 172.21.15.57:0/746268856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6c80c62b0 0x7fc6c80ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.892+0000 7fc6ce171640 1 --2- 172.21.15.57:0/746268856 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6c8153b50 0x7fc6c81597b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.892+0000 7fc6ce171640 1 -- 172.21.15.57:0/746268856 >> 172.21.15.57:0/746268856 conn(0x7fc6c8065a80 msgr2=0x7fc6c8065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:32.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.892+0000 7fc6ce171640 1 -- 172.21.15.57:0/746268856 shutdown_connections 2023-10-30T14:14:32.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.892+0000 7fc6ce171640 1 -- 172.21.15.57:0/746268856 wait complete. 2023-10-30T14:14:32.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.892+0000 7fc6ce171640 1 Processor -- start 2023-10-30T14:14:32.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.892+0000 7fc6ce171640 1 -- start start 2023-10-30T14:14:32.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6ce171640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6c80c62b0 0x7fc6c81f4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6ce171640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6c80cb180 0x7fc6c8159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6ce171640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6c8153b50 0x7fc6c8147370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6ce171640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c80c24d0 con 0x7fc6c80cb180 2023-10-30T14:14:32.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6ce171640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c814ae40 con 0x7fc6c8153b50 2023-10-30T14:14:32.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6ce171640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c8152910 con 0x7fc6c80c62b0 2023-10-30T14:14:32.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6cc96e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6c80cb180 0x7fc6c8159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6cc96e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6c80cb180 0x7fc6c8159230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41116/0 (socket says 172.21.15.57:41116) 2023-10-30T14:14:32.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6cc96e640 1 -- 172.21.15.57:0/4017487097 learned_addr learned my addr 172.21.15.57:0/4017487097 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:32.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6cd970640 1 --2- 172.21.15.57:0/4017487097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6c8153b50 0x7fc6c8147370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6cd16f640 1 --2- 172.21.15.57:0/4017487097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6c80c62b0 0x7fc6c81f4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6cc96e640 1 -- 172.21.15.57:0/4017487097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6c80c62b0 msgr2=0x7fc6c81f4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:32.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6cc96e640 1 --2- 172.21.15.57:0/4017487097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6c80c62b0 0x7fc6c81f4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6cc96e640 1 -- 172.21.15.57:0/4017487097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6c8153b50 msgr2=0x7fc6c8147370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:32.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6cc96e640 1 --2- 172.21.15.57:0/4017487097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6c8153b50 0x7fc6c8147370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:32.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6cc96e640 1 -- 172.21.15.57:0/4017487097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c4009420 con 0x7fc6c80cb180 2023-10-30T14:14:32.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6cd970640 1 --2- 172.21.15.57:0/4017487097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6c8153b50 0x7fc6c8147370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:32.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6cc96e640 1 --2- 172.21.15.57:0/4017487097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6c80cb180 0x7fc6c8159230 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc6b800e7e0 tx=0x7fc6b800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:32.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6bdffb640 1 -- 172.21.15.57:0/4017487097 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b800c6d0 con 0x7fc6c80cb180 2023-10-30T14:14:32.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6bdffb640 1 -- 172.21.15.57:0/4017487097 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b800c830 con 0x7fc6c80cb180 2023-10-30T14:14:32.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6bdffb640 1 -- 172.21.15.57:0/4017487097 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b800cbe0 con 0x7fc6c80cb180 2023-10-30T14:14:32.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6ce171640 1 -- 172.21.15.57:0/4017487097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6c80c7e50 con 0x7fc6c80cb180 2023-10-30T14:14:32.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6ce171640 1 -- 172.21.15.57:0/4017487097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c806a650 con 0x7fc6c80cb180 2023-10-30T14:14:32.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.896+0000 7fc6ce171640 1 -- 172.21.15.57:0/4017487097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6c80c5dc0 con 0x7fc6c80cb180 2023-10-30T14:14:32.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.900+0000 7fc6bdffb640 1 -- 172.21.15.57:0/4017487097 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc6b800cd40 con 0x7fc6c80cb180 2023-10-30T14:14:32.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.900+0000 7fc6bdffb640 1 --2- 172.21.15.57:0/4017487097 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc6ac0664d0 0x7fc6ac068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:32.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.900+0000 7fc6bdffb640 1 -- 172.21.15.57:0/4017487097 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6b8083d00 con 0x7fc6c80cb180 2023-10-30T14:14:32.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.900+0000 7fc6cd16f640 1 --2- 172.21.15.57:0/4017487097 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc6ac0664d0 0x7fc6ac068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:32.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.900+0000 7fc6cd16f640 1 --2- 172.21.15.57:0/4017487097 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc6ac0664d0 0x7fc6ac068990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc6c40ad000 tx=0x7fc6c4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:32.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:32.904+0000 7fc6bdffb640 1 -- 172.21.15.57:0/4017487097 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc6b8051190 con 0x7fc6c80cb180 2023-10-30T14:14:33.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.096+0000 7fc6ce171640 1 -- 172.21.15.57:0/4017487097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6c8067540 con 0x7fc6c80cb180 2023-10-30T14:14:33.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.096+0000 7fc6bdffb640 1 -- 172.21.15.57:0/4017487097 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7fc6b80547b0 con 0x7fc6c80cb180 2023-10-30T14:14:33.101 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:33.102 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:14:20.837447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6837/901968115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":901968115},{"type":"v1","addr":"172.21.15.57:6837","nonce":901968115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:33.102 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 12 2023-10-30T14:14:33.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.096+0000 7fc6ce171640 1 -- 172.21.15.57:0/4017487097 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc6ac0664d0 msgr2=0x7fc6ac068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:33.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.096+0000 7fc6ce171640 1 --2- 172.21.15.57:0/4017487097 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc6ac0664d0 0x7fc6ac068990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc6c40ad000 tx=0x7fc6c4008040 comp rx=0 tx=0).stop 2023-10-30T14:14:33.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.096+0000 7fc6ce171640 1 -- 172.21.15.57:0/4017487097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6c80cb180 msgr2=0x7fc6c8159230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:33.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.096+0000 7fc6ce171640 1 --2- 172.21.15.57:0/4017487097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6c80cb180 0x7fc6c8159230 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc6b800e7e0 tx=0x7fc6b800eca0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.100+0000 7fc6ce171640 1 -- 172.21.15.57:0/4017487097 shutdown_connections 2023-10-30T14:14:33.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.100+0000 7fc6ce171640 1 --2- 172.21.15.57:0/4017487097 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc6ac0664d0 0x7fc6ac068990 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.100+0000 7fc6ce171640 1 --2- 172.21.15.57:0/4017487097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6c8153b50 0x7fc6c8147370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.100+0000 7fc6ce171640 1 --2- 172.21.15.57:0/4017487097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6c80cb180 0x7fc6c8159230 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.100+0000 7fc6ce171640 1 --2- 172.21.15.57:0/4017487097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6c80c62b0 0x7fc6c81f4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.100+0000 7fc6ce171640 1 -- 172.21.15.57:0/4017487097 >> 172.21.15.57:0/4017487097 conn(0x7fc6c8065a80 msgr2=0x7fc6c814f020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:33.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.100+0000 7fc6ce171640 1 -- 172.21.15.57:0/4017487097 shutdown_connections 2023-10-30T14:14:33.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.100+0000 7fc6ce171640 1 -- 172.21.15.57:0/4017487097 wait complete. 2023-10-30T14:14:33.123 INFO:tasks.mds_thrash:Ready to start thrashing 2023-10-30T14:14:33.123 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2023-10-30T14:14:33.123 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:14:33.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.260+0000 7fbdb3868640 1 Processor -- start 2023-10-30T14:14:33.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.260+0000 7fbdb3868640 1 -- start start 2023-10-30T14:14:33.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.260+0000 7fbdb3868640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbdac153c80 0x7fbdac15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:33.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.260+0000 7fbdb3868640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbdac0c42a0 0x7fbdac0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:33.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.260+0000 7fbdb3868640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdac159f20 con 0x7fbdac0c4a10 2023-10-30T14:14:33.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.260+0000 7fbdb3868640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdac15dec0 con 0x7fbdac153c80 2023-10-30T14:14:33.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.260+0000 7fbdb3868640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdac1599a0 con 0x7fbdac0c42a0 2023-10-30T14:14:33.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.264+0000 7fbdb1604640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbdac153c80 0x7fbdac15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:33.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.264+0000 7fbdb1604640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbdac153c80 0x7fbdac15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41130/0 (socket says 172.21.15.57:41130) 2023-10-30T14:14:33.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.264+0000 7fbdb1604640 1 -- 172.21.15.57:0/3393445311 learned_addr learned my addr 172.21.15.57:0/3393445311 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:33.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.264+0000 7fbdb1e05640 1 --2- 172.21.15.57:0/3393445311 >> v2:172.21.15.88:3300/0 conn(0x7fbdac0c42a0 0x7fbdac0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:33.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.264+0000 7fbdb1604640 1 -- 172.21.15.57:0/3393445311 >> v2:172.21.15.88:3300/0 conn(0x7fbdac0c42a0 msgr2=0x7fbdac0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:33.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.264+0000 7fbdb1604640 1 --2- 172.21.15.57:0/3393445311 >> v2:172.21.15.88:3300/0 conn(0x7fbdac0c42a0 0x7fbdac0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.264+0000 7fbdb1604640 1 -- 172.21.15.57:0/3393445311 >> v1:172.21.15.88:6789/0 conn(0x7fbdac0c4a10 legacy=0x7fbdac15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:33.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.264+0000 7fbdb1604640 1 -- 172.21.15.57:0/3393445311 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdac15d620 con 0x7fbdac153c80 2023-10-30T14:14:33.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.264+0000 7fbdb1604640 1 --2- 172.21.15.57:0/3393445311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdac153c80 0x7fbdac15cf90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbdac0b1d40 tx=0x7fbda801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d49a64f8764fc3e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:33.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.264+0000 7fbda3fff640 1 -- 172.21.15.57:0/3393445311 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda80a89f0 con 0x7fbdac153c80 2023-10-30T14:14:33.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.264+0000 7fbda3fff640 1 -- 172.21.15.57:0/3393445311 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda80a8b50 con 0x7fbdac153c80 2023-10-30T14:14:33.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.264+0000 7fbda3fff640 1 -- 172.21.15.57:0/3393445311 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda80b0440 con 0x7fbdac153c80 2023-10-30T14:14:33.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb3868640 1 -- 172.21.15.57:0/3393445311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdac153c80 msgr2=0x7fbdac15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:33.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb3868640 1 --2- 172.21.15.57:0/3393445311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdac153c80 0x7fbdac15cf90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbdac0b1d40 tx=0x7fbda801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb3868640 1 -- 172.21.15.57:0/3393445311 shutdown_connections 2023-10-30T14:14:33.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb3868640 1 --2- 172.21.15.57:0/3393445311 >> v2:172.21.15.88:3300/0 conn(0x7fbdac0c42a0 0x7fbdac0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb3868640 1 --2- 172.21.15.57:0/3393445311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdac153c80 0x7fbdac15cf90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb3868640 1 -- 172.21.15.57:0/3393445311 >> 172.21.15.57:0/3393445311 conn(0x7fbdac0715c0 msgr2=0x7fbdac0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:33.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb3868640 1 -- 172.21.15.57:0/3393445311 shutdown_connections 2023-10-30T14:14:33.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb3868640 1 -- 172.21.15.57:0/3393445311 wait complete. 2023-10-30T14:14:33.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb3868640 1 Processor -- start 2023-10-30T14:14:33.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb3868640 1 -- start start 2023-10-30T14:14:33.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb3868640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdac0c42a0 0x7fbdac1ee760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:33.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb3868640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdac0c4a10 0x7fbdac15c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:33.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb3868640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbdac153c80 0x7fbdac149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:33.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb3868640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac0c95f0 con 0x7fbdac0c4a10 2023-10-30T14:14:33.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb3868640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac15c080 con 0x7fbdac0c42a0 2023-10-30T14:14:33.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb3868640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac15ad00 con 0x7fbdac153c80 2023-10-30T14:14:33.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb0e03640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdac0c4a10 0x7fbdac15c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:33.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb0e03640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdac0c4a10 0x7fbdac15c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41144/0 (socket says 172.21.15.57:41144) 2023-10-30T14:14:33.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb0e03640 1 -- 172.21.15.57:0/2700935695 learned_addr learned my addr 172.21.15.57:0/2700935695 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:33.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb1604640 1 --2- 172.21.15.57:0/2700935695 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdac0c42a0 0x7fbdac1ee760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:33.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.268+0000 7fbdb1e05640 1 --2- 172.21.15.57:0/2700935695 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbdac153c80 0x7fbdac149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:33.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.272+0000 7fbdb0e03640 1 -- 172.21.15.57:0/2700935695 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbdac153c80 msgr2=0x7fbdac149f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:33.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.272+0000 7fbdb0e03640 1 --2- 172.21.15.57:0/2700935695 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbdac153c80 0x7fbdac149f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.272+0000 7fbdb0e03640 1 -- 172.21.15.57:0/2700935695 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdac0c42a0 msgr2=0x7fbdac1ee760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:33.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.272+0000 7fbdb0e03640 1 --2- 172.21.15.57:0/2700935695 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdac0c42a0 0x7fbdac1ee760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.272+0000 7fbdb0e03640 1 -- 172.21.15.57:0/2700935695 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda80094d0 con 0x7fbdac0c4a10 2023-10-30T14:14:33.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.272+0000 7fbdb1604640 1 --2- 172.21.15.57:0/2700935695 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdac0c42a0 0x7fbdac1ee760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:33.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.272+0000 7fbdb0e03640 1 --2- 172.21.15.57:0/2700935695 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdac0c4a10 0x7fbdac15c320 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c00c4d0 tx=0x7fbd9c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:33.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.272+0000 7fbda1ffb640 1 -- 172.21.15.57:0/2700935695 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c0052d0 con 0x7fbdac0c4a10 2023-10-30T14:14:33.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.272+0000 7fbda1ffb640 1 -- 172.21.15.57:0/2700935695 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9c005430 con 0x7fbdac0c4a10 2023-10-30T14:14:33.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.272+0000 7fbda1ffb640 1 -- 172.21.15.57:0/2700935695 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c012830 con 0x7fbdac0c4a10 2023-10-30T14:14:33.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.272+0000 7fbdb3868640 1 -- 172.21.15.57:0/2700935695 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdac15afe0 con 0x7fbdac0c4a10 2023-10-30T14:14:33.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.272+0000 7fbdb3868640 1 -- 172.21.15.57:0/2700935695 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdac15b5a0 con 0x7fbdac0c4a10 2023-10-30T14:14:33.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.272+0000 7fbdb3868640 1 -- 172.21.15.57:0/2700935695 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdac04e170 con 0x7fbdac0c4a10 2023-10-30T14:14:33.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.272+0000 7fbda1ffb640 1 -- 172.21.15.57:0/2700935695 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbd9c012990 con 0x7fbdac0c4a10 2023-10-30T14:14:33.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.272+0000 7fbda1ffb640 1 --2- 172.21.15.57:0/2700935695 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd900664d0 0x7fbd90068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:33.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.272+0000 7fbda1ffb640 1 -- 172.21.15.57:0/2700935695 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd9c083a70 con 0x7fbdac0c4a10 2023-10-30T14:14:33.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.276+0000 7fbdb1604640 1 --2- 172.21.15.57:0/2700935695 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd900664d0 0x7fbd90068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:33.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.276+0000 7fbdb1604640 1 --2- 172.21.15.57:0/2700935695 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd900664d0 0x7fbd90068990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbda80b2000 tx=0x7fbda8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:33.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.276+0000 7fbda1ffb640 1 -- 172.21.15.57:0/2700935695 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbd9c050dc0 con 0x7fbdac0c4a10 2023-10-30T14:14:33.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.432+0000 7fbdb3868640 1 -- 172.21.15.57:0/2700935695 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbdac15dbe0 con 0x7fbd900664d0 2023-10-30T14:14:33.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.452+0000 7fbda1ffb640 1 -- 172.21.15.57:0/2700935695 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264675 (secure 0 0 0) 0x7fbda80bf090 con 0x7fbd900664d0 2023-10-30T14:14:33.461 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:33.463 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-30T14:14:33.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.456+0000 7fbdb3868640 1 -- 172.21.15.57:0/2700935695 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd900664d0 msgr2=0x7fbd90068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:33.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.456+0000 7fbdb3868640 1 --2- 172.21.15.57:0/2700935695 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd900664d0 0x7fbd90068990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbda80b2000 tx=0x7fbda8008040 comp rx=0 tx=0).stop 2023-10-30T14:14:33.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.456+0000 7fbdb3868640 1 -- 172.21.15.57:0/2700935695 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdac0c4a10 msgr2=0x7fbdac15c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:33.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.456+0000 7fbdb3868640 1 --2- 172.21.15.57:0/2700935695 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdac0c4a10 0x7fbdac15c320 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c00c4d0 tx=0x7fbd9c00c990 comp rx=0 tx=0).stop 2023-10-30T14:14:33.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.456+0000 7fbdb3868640 1 -- 172.21.15.57:0/2700935695 shutdown_connections 2023-10-30T14:14:33.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.456+0000 7fbdb3868640 1 --2- 172.21.15.57:0/2700935695 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd900664d0 0x7fbd90068990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.456+0000 7fbdb3868640 1 --2- 172.21.15.57:0/2700935695 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbdac153c80 0x7fbdac149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.456+0000 7fbdb3868640 1 --2- 172.21.15.57:0/2700935695 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdac0c4a10 0x7fbdac15c320 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.456+0000 7fbdb3868640 1 --2- 172.21.15.57:0/2700935695 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdac0c42a0 0x7fbdac1ee760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.456+0000 7fbdb3868640 1 -- 172.21.15.57:0/2700935695 >> 172.21.15.57:0/2700935695 conn(0x7fbdac0715c0 msgr2=0x7fbdac158640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:33.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.456+0000 7fbdb3868640 1 -- 172.21.15.57:0/2700935695 shutdown_connections 2023-10-30T14:14:33.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.456+0000 7fbdb3868640 1 -- 172.21.15.57:0/2700935695 wait complete. 2023-10-30T14:14:33.481 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-10-30T14:14:32.504236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596921,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":288,"num_write_kb":2458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227464,"kb_used_data":3016,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754747256,"statfs":{"total":773094113280,"available":772861190144,"internally_reserved":0,"allocated":3088384,"data_stored":1798394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":229756473},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2737,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004216"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757043+0000","last_change":"2023-10-30T14:14:04.651413+0000","last_active":"2023-10-30T14:14:05.757043+0000","last_peered":"2023-10-30T14:14:05.757043+0000","last_clean":"2023-10-30T14:14:05.757043+0000","last_became_active":"2023-10-30T14:14:04.650882+0000","last_became_peered":"2023-10-30T14:14:04.650882+0000","last_unstale":"2023-10-30T14:14:05.757043+0000","last_undegraded":"2023-10-30T14:14:05.757043+0000","last_fullsized":"2023-10-30T14:14:05.757043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:30:08.122326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585036+0000","last_change":"2023-10-30T14:14:04.588523+0000","last_active":"2023-10-30T14:14:05.585036+0000","last_peered":"2023-10-30T14:14:05.585036+0000","last_clean":"2023-10-30T14:14:05.585036+0000","last_became_active":"2023-10-30T14:14:04.587942+0000","last_became_peered":"2023-10-30T14:14:04.587942+0000","last_unstale":"2023-10-30T14:14:05.585036+0000","last_undegraded":"2023-10-30T14:14:05.585036+0000","last_fullsized":"2023-10-30T14:14:05.585036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:16.028564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586543+0000","last_change":"2023-10-30T14:14:04.590976+0000","last_active":"2023-10-30T14:14:05.586543+0000","last_peered":"2023-10-30T14:14:05.586543+0000","last_clean":"2023-10-30T14:14:05.586543+0000","last_became_active":"2023-10-30T14:14:04.590700+0000","last_became_peered":"2023-10-30T14:14:04.590700+0000","last_unstale":"2023-10-30T14:14:05.586543+0000","last_undegraded":"2023-10-30T14:14:05.586543+0000","last_fullsized":"2023-10-30T14:14:05.586543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:49.169325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586525+0000","last_change":"2023-10-30T14:14:04.618603+0000","last_active":"2023-10-30T14:14:05.586525+0000","last_peered":"2023-10-30T14:14:05.586525+0000","last_clean":"2023-10-30T14:14:05.586525+0000","last_became_active":"2023-10-30T14:14:04.617768+0000","last_became_peered":"2023-10-30T14:14:04.617768+0000","last_unstale":"2023-10-30T14:14:05.586525+0000","last_undegraded":"2023-10-30T14:14:05.586525+0000","last_fullsized":"2023-10-30T14:14:05.586525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:16.443567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756348+0000","last_change":"2023-10-30T14:14:04.647842+0000","last_active":"2023-10-30T14:14:05.756348+0000","last_peered":"2023-10-30T14:14:05.756348+0000","last_clean":"2023-10-30T14:14:05.756348+0000","last_became_active":"2023-10-30T14:14:04.644683+0000","last_became_peered":"2023-10-30T14:14:04.644683+0000","last_unstale":"2023-10-30T14:14:05.756348+0000","last_undegraded":"2023-10-30T14:14:05.756348+0000","last_fullsized":"2023-10-30T14:14:05.756348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:51:01.230722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585902+0000","last_change":"2023-10-30T14:14:04.620587+0000","last_active":"2023-10-30T14:14:05.585902+0000","last_peered":"2023-10-30T14:14:05.585902+0000","last_clean":"2023-10-30T14:14:05.585902+0000","last_became_active":"2023-10-30T14:14:04.619335+0000","last_became_peered":"2023-10-30T14:14:04.619335+0000","last_unstale":"2023-10-30T14:14:05.585902+0000","last_undegraded":"2023-10-30T14:14:05.585902+0000","last_fullsized":"2023-10-30T14:14:05.585902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:01.735027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755914+0000","last_change":"2023-10-30T14:14:04.641688+0000","last_active":"2023-10-30T14:14:05.755914+0000","last_peered":"2023-10-30T14:14:05.755914+0000","last_clean":"2023-10-30T14:14:05.755914+0000","last_became_active":"2023-10-30T14:14:04.641323+0000","last_became_peered":"2023-10-30T14:14:04.641323+0000","last_unstale":"2023-10-30T14:14:05.755914+0000","last_undegraded":"2023-10-30T14:14:05.755914+0000","last_fullsized":"2023-10-30T14:14:05.755914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:43:35.285019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805751+0000","last_change":"2023-10-30T14:14:04.618530+0000","last_active":"2023-10-30T14:14:05.805751+0000","last_peered":"2023-10-30T14:14:05.805751+0000","last_clean":"2023-10-30T14:14:05.805751+0000","last_became_active":"2023-10-30T14:14:04.618312+0000","last_became_peered":"2023-10-30T14:14:04.618312+0000","last_unstale":"2023-10-30T14:14:05.805751+0000","last_undegraded":"2023-10-30T14:14:05.805751+0000","last_fullsized":"2023-10-30T14:14:05.805751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:25.089106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757146+0000","last_change":"2023-10-30T14:14:04.650330+0000","last_active":"2023-10-30T14:14:05.757146+0000","last_peered":"2023-10-30T14:14:05.757146+0000","last_clean":"2023-10-30T14:14:05.757146+0000","last_became_active":"2023-10-30T14:14:04.647361+0000","last_became_peered":"2023-10-30T14:14:04.647361+0000","last_unstale":"2023-10-30T14:14:05.757146+0000","last_undegraded":"2023-10-30T14:14:05.757146+0000","last_fullsized":"2023-10-30T14:14:05.757146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:49:57.370980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808090+0000","last_change":"2023-10-30T14:14:04.631540+0000","last_active":"2023-10-30T14:14:05.808090+0000","last_peered":"2023-10-30T14:14:05.808090+0000","last_clean":"2023-10-30T14:14:05.808090+0000","last_became_active":"2023-10-30T14:14:04.630959+0000","last_became_peered":"2023-10-30T14:14:04.630959+0000","last_unstale":"2023-10-30T14:14:05.808090+0000","last_undegraded":"2023-10-30T14:14:05.808090+0000","last_fullsized":"2023-10-30T14:14:05.808090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:49.468429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807762+0000","last_change":"2023-10-30T14:14:04.613800+0000","last_active":"2023-10-30T14:14:05.807762+0000","last_peered":"2023-10-30T14:14:05.807762+0000","last_clean":"2023-10-30T14:14:05.807762+0000","last_became_active":"2023-10-30T14:14:04.612653+0000","last_became_peered":"2023-10-30T14:14:04.612653+0000","last_unstale":"2023-10-30T14:14:05.807762+0000","last_undegraded":"2023-10-30T14:14:05.807762+0000","last_fullsized":"2023-10-30T14:14:05.807762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:44.665588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809700+0000","last_change":"2023-10-30T14:14:04.622483+0000","last_active":"2023-10-30T14:14:05.809700+0000","last_peered":"2023-10-30T14:14:05.809700+0000","last_clean":"2023-10-30T14:14:05.809700+0000","last_became_active":"2023-10-30T14:14:04.621135+0000","last_became_peered":"2023-10-30T14:14:04.621135+0000","last_unstale":"2023-10-30T14:14:05.809700+0000","last_undegraded":"2023-10-30T14:14:05.809700+0000","last_fullsized":"2023-10-30T14:14:05.809700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:52.299412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806054+0000","last_change":"2023-10-30T14:14:04.616357+0000","last_active":"2023-10-30T14:14:05.806054+0000","last_peered":"2023-10-30T14:14:05.806054+0000","last_clean":"2023-10-30T14:14:05.806054+0000","last_became_active":"2023-10-30T14:14:04.616142+0000","last_became_peered":"2023-10-30T14:14:04.616142+0000","last_unstale":"2023-10-30T14:14:05.806054+0000","last_undegraded":"2023-10-30T14:14:05.806054+0000","last_fullsized":"2023-10-30T14:14:05.806054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:32:26.432551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586106+0000","last_change":"2023-10-30T14:14:04.588759+0000","last_active":"2023-10-30T14:14:05.586106+0000","last_peered":"2023-10-30T14:14:05.586106+0000","last_clean":"2023-10-30T14:14:05.586106+0000","last_became_active":"2023-10-30T14:14:04.588452+0000","last_became_peered":"2023-10-30T14:14:04.588452+0000","last_unstale":"2023-10-30T14:14:05.586106+0000","last_undegraded":"2023-10-30T14:14:05.586106+0000","last_fullsized":"2023-10-30T14:14:05.586106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:38:21.206284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584572+0000","last_change":"2023-10-30T14:14:04.621073+0000","last_active":"2023-10-30T14:14:05.584572+0000","last_peered":"2023-10-30T14:14:05.584572+0000","last_clean":"2023-10-30T14:14:05.584572+0000","last_became_active":"2023-10-30T14:14:04.620781+0000","last_became_peered":"2023-10-30T14:14:04.620781+0000","last_unstale":"2023-10-30T14:14:05.584572+0000","last_undegraded":"2023-10-30T14:14:05.584572+0000","last_fullsized":"2023-10-30T14:14:05.584572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:17:10.938195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585712+0000","last_change":"2023-10-30T14:14:04.608942+0000","last_active":"2023-10-30T14:14:05.585712+0000","last_peered":"2023-10-30T14:14:05.585712+0000","last_clean":"2023-10-30T14:14:05.585712+0000","last_became_active":"2023-10-30T14:14:04.608190+0000","last_became_peered":"2023-10-30T14:14:04.608190+0000","last_unstale":"2023-10-30T14:14:05.585712+0000","last_undegraded":"2023-10-30T14:14:05.585712+0000","last_fullsized":"2023-10-30T14:14:05.585712+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:18:39.846841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587056+0000","last_change":"2023-10-30T14:14:04.641876+0000","last_active":"2023-10-30T14:14:05.587056+0000","last_peered":"2023-10-30T14:14:05.587056+0000","last_clean":"2023-10-30T14:14:05.587056+0000","last_became_active":"2023-10-30T14:14:04.641306+0000","last_became_peered":"2023-10-30T14:14:04.641306+0000","last_unstale":"2023-10-30T14:14:05.587056+0000","last_undegraded":"2023-10-30T14:14:05.587056+0000","last_fullsized":"2023-10-30T14:14:05.587056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:10:08.169786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805744+0000","last_change":"2023-10-30T14:14:04.613999+0000","last_active":"2023-10-30T14:14:05.805744+0000","last_peered":"2023-10-30T14:14:05.805744+0000","last_clean":"2023-10-30T14:14:05.805744+0000","last_became_active":"2023-10-30T14:14:04.613644+0000","last_became_peered":"2023-10-30T14:14:04.613644+0000","last_unstale":"2023-10-30T14:14:05.805744+0000","last_undegraded":"2023-10-30T14:14:05.805744+0000","last_fullsized":"2023-10-30T14:14:05.805744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:14:50.232028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756929+0000","last_change":"2023-10-30T14:14:04.650393+0000","last_active":"2023-10-30T14:14:05.756929+0000","last_peered":"2023-10-30T14:14:05.756929+0000","last_clean":"2023-10-30T14:14:05.756929+0000","last_became_active":"2023-10-30T14:14:04.649850+0000","last_became_peered":"2023-10-30T14:14:04.649850+0000","last_unstale":"2023-10-30T14:14:05.756929+0000","last_undegraded":"2023-10-30T14:14:05.756929+0000","last_fullsized":"2023-10-30T14:14:05.756929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:25.174023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808107+0000","last_change":"2023-10-30T14:14:04.625744+0000","last_active":"2023-10-30T14:14:05.808107+0000","last_peered":"2023-10-30T14:14:05.808107+0000","last_clean":"2023-10-30T14:14:05.808107+0000","last_became_active":"2023-10-30T14:14:04.625042+0000","last_became_peered":"2023-10-30T14:14:04.625042+0000","last_unstale":"2023-10-30T14:14:05.808107+0000","last_undegraded":"2023-10-30T14:14:05.808107+0000","last_fullsized":"2023-10-30T14:14:05.808107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:30:36.063455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586236+0000","last_change":"2023-10-30T14:14:04.633942+0000","last_active":"2023-10-30T14:14:05.586236+0000","last_peered":"2023-10-30T14:14:05.586236+0000","last_clean":"2023-10-30T14:14:05.586236+0000","last_became_active":"2023-10-30T14:14:04.633586+0000","last_became_peered":"2023-10-30T14:14:04.633586+0000","last_unstale":"2023-10-30T14:14:05.586236+0000","last_undegraded":"2023-10-30T14:14:05.586236+0000","last_fullsized":"2023-10-30T14:14:05.586236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:42:58.067892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586050+0000","last_change":"2023-10-30T14:14:04.642847+0000","last_active":"2023-10-30T14:14:05.586050+0000","last_peered":"2023-10-30T14:14:05.586050+0000","last_clean":"2023-10-30T14:14:05.586050+0000","last_became_active":"2023-10-30T14:14:04.642244+0000","last_became_peered":"2023-10-30T14:14:04.642244+0000","last_unstale":"2023-10-30T14:14:05.586050+0000","last_undegraded":"2023-10-30T14:14:05.586050+0000","last_fullsized":"2023-10-30T14:14:05.586050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:28.780866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805762+0000","last_change":"2023-10-30T14:14:04.620651+0000","last_active":"2023-10-30T14:14:05.805762+0000","last_peered":"2023-10-30T14:14:05.805762+0000","last_clean":"2023-10-30T14:14:05.805762+0000","last_became_active":"2023-10-30T14:14:04.620343+0000","last_became_peered":"2023-10-30T14:14:04.620343+0000","last_unstale":"2023-10-30T14:14:05.805762+0000","last_undegraded":"2023-10-30T14:14:05.805762+0000","last_fullsized":"2023-10-30T14:14:05.805762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:16:00.557118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587224+0000","last_change":"2023-10-30T14:14:04.610472+0000","last_active":"2023-10-30T14:14:05.587224+0000","last_peered":"2023-10-30T14:14:05.587224+0000","last_clean":"2023-10-30T14:14:05.587224+0000","last_became_active":"2023-10-30T14:14:04.609720+0000","last_became_peered":"2023-10-30T14:14:04.609720+0000","last_unstale":"2023-10-30T14:14:05.587224+0000","last_undegraded":"2023-10-30T14:14:05.587224+0000","last_fullsized":"2023-10-30T14:14:05.587224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:44:28.391589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805906+0000","last_change":"2023-10-30T14:14:04.620548+0000","last_active":"2023-10-30T14:14:05.805906+0000","last_peered":"2023-10-30T14:14:05.805906+0000","last_clean":"2023-10-30T14:14:05.805906+0000","last_became_active":"2023-10-30T14:14:04.620217+0000","last_became_peered":"2023-10-30T14:14:04.620217+0000","last_unstale":"2023-10-30T14:14:05.805906+0000","last_undegraded":"2023-10-30T14:14:05.805906+0000","last_fullsized":"2023-10-30T14:14:05.805906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:40:22.912696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808191+0000","last_change":"2023-10-30T14:14:04.644315+0000","last_active":"2023-10-30T14:14:05.808191+0000","last_peered":"2023-10-30T14:14:05.808191+0000","last_clean":"2023-10-30T14:14:05.808191+0000","last_became_active":"2023-10-30T14:14:04.643707+0000","last_became_peered":"2023-10-30T14:14:04.643707+0000","last_unstale":"2023-10-30T14:14:05.808191+0000","last_undegraded":"2023-10-30T14:14:05.808191+0000","last_fullsized":"2023-10-30T14:14:05.808191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:20:33.544167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584874+0000","last_change":"2023-10-30T14:14:04.597855+0000","last_active":"2023-10-30T14:14:05.584874+0000","last_peered":"2023-10-30T14:14:05.584874+0000","last_clean":"2023-10-30T14:14:05.584874+0000","last_became_active":"2023-10-30T14:14:04.597590+0000","last_became_peered":"2023-10-30T14:14:04.597590+0000","last_unstale":"2023-10-30T14:14:05.584874+0000","last_undegraded":"2023-10-30T14:14:05.584874+0000","last_fullsized":"2023-10-30T14:14:05.584874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:05.905027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806303+0000","last_change":"2023-10-30T14:14:04.624133+0000","last_active":"2023-10-30T14:14:05.806303+0000","last_peered":"2023-10-30T14:14:05.806303+0000","last_clean":"2023-10-30T14:14:05.806303+0000","last_became_active":"2023-10-30T14:14:04.623902+0000","last_became_peered":"2023-10-30T14:14:04.623902+0000","last_unstale":"2023-10-30T14:14:05.806303+0000","last_undegraded":"2023-10-30T14:14:05.806303+0000","last_fullsized":"2023-10-30T14:14:05.806303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:33:10.476461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586328+0000","last_change":"2023-10-30T14:14:04.597875+0000","last_active":"2023-10-30T14:14:05.586328+0000","last_peered":"2023-10-30T14:14:05.586328+0000","last_clean":"2023-10-30T14:14:05.586328+0000","last_became_active":"2023-10-30T14:14:04.597556+0000","last_became_peered":"2023-10-30T14:14:04.597556+0000","last_unstale":"2023-10-30T14:14:05.586328+0000","last_undegraded":"2023-10-30T14:14:05.586328+0000","last_fullsized":"2023-10-30T14:14:05.586328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:08:46.407610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586842+0000","last_change":"2023-10-30T14:14:04.620110+0000","last_active":"2023-10-30T14:14:05.586842+0000","last_peered":"2023-10-30T14:14:05.586842+0000","last_clean":"2023-10-30T14:14:05.586842+0000","last_became_active":"2023-10-30T14:14:04.618430+0000","last_became_peered":"2023-10-30T14:14:04.618430+0000","last_unstale":"2023-10-30T14:14:05.586842+0000","last_undegraded":"2023-10-30T14:14:05.586842+0000","last_fullsized":"2023-10-30T14:14:05.586842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:36.966564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805526+0000","last_change":"2023-10-30T14:14:04.606692+0000","last_active":"2023-10-30T14:14:05.805526+0000","last_peered":"2023-10-30T14:14:05.805526+0000","last_clean":"2023-10-30T14:14:05.805526+0000","last_became_active":"2023-10-30T14:14:04.606340+0000","last_became_peered":"2023-10-30T14:14:04.606340+0000","last_unstale":"2023-10-30T14:14:05.805526+0000","last_undegraded":"2023-10-30T14:14:05.805526+0000","last_fullsized":"2023-10-30T14:14:05.805526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:02:24.518205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807314+0000","last_change":"2023-10-30T14:14:04.613535+0000","last_active":"2023-10-30T14:14:05.807314+0000","last_peered":"2023-10-30T14:14:05.807314+0000","last_clean":"2023-10-30T14:14:05.807314+0000","last_became_active":"2023-10-30T14:14:04.612588+0000","last_became_peered":"2023-10-30T14:14:04.612588+0000","last_unstale":"2023-10-30T14:14:05.807314+0000","last_undegraded":"2023-10-30T14:14:05.807314+0000","last_fullsized":"2023-10-30T14:14:05.807314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:58:23.057988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809168+0000","last_change":"2023-10-30T14:14:04.608434+0000","last_active":"2023-10-30T14:14:05.809168+0000","last_peered":"2023-10-30T14:14:05.809168+0000","last_clean":"2023-10-30T14:14:05.809168+0000","last_became_active":"2023-10-30T14:14:04.607683+0000","last_became_peered":"2023-10-30T14:14:04.607683+0000","last_unstale":"2023-10-30T14:14:05.809168+0000","last_undegraded":"2023-10-30T14:14:05.809168+0000","last_fullsized":"2023-10-30T14:14:05.809168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:17:27.467330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.795624+0000","last_change":"2023-10-30T14:14:03.732084+0000","last_active":"2023-10-30T14:14:14.795624+0000","last_peered":"2023-10-30T14:14:14.795624+0000","last_clean":"2023-10-30T14:14:14.795624+0000","last_became_active":"2023-10-30T14:14:03.731740+0000","last_became_peered":"2023-10-30T14:14:03.731740+0000","last_unstale":"2023-10-30T14:14:14.795624+0000","last_undegraded":"2023-10-30T14:14:14.795624+0000","last_fullsized":"2023-10-30T14:14:14.795624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:34:27.462477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586305+0000","last_change":"2023-10-30T14:14:04.621869+0000","last_active":"2023-10-30T14:14:05.586305+0000","last_peered":"2023-10-30T14:14:05.586305+0000","last_clean":"2023-10-30T14:14:05.586305+0000","last_became_active":"2023-10-30T14:14:04.621526+0000","last_became_peered":"2023-10-30T14:14:04.621526+0000","last_unstale":"2023-10-30T14:14:05.586305+0000","last_undegraded":"2023-10-30T14:14:05.586305+0000","last_fullsized":"2023-10-30T14:14:05.586305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:08:21.260692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.820804+0000","last_change":"2023-10-30T14:14:03.730789+0000","last_active":"2023-10-30T14:14:20.820804+0000","last_peered":"2023-10-30T14:14:20.820804+0000","last_clean":"2023-10-30T14:14:20.820804+0000","last_became_active":"2023-10-30T14:14:03.729386+0000","last_became_peered":"2023-10-30T14:14:03.729386+0000","last_unstale":"2023-10-30T14:14:20.820804+0000","last_undegraded":"2023-10-30T14:14:20.820804+0000","last_fullsized":"2023-10-30T14:14:20.820804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.822932+0000","last_change":"2023-10-30T14:14:03.724379+0000","last_active":"2023-10-30T14:14:20.822932+0000","last_peered":"2023-10-30T14:14:20.822932+0000","last_clean":"2023-10-30T14:14:20.822932+0000","last_became_active":"2023-10-30T14:14:03.724078+0000","last_became_peered":"2023-10-30T14:14:03.724078+0000","last_unstale":"2023-10-30T14:14:20.822932+0000","last_undegraded":"2023-10-30T14:14:20.822932+0000","last_fullsized":"2023-10-30T14:14:20.822932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:10:17.829899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805886+0000","last_change":"2023-10-30T14:14:04.597423+0000","last_active":"2023-10-30T14:14:05.805886+0000","last_peered":"2023-10-30T14:14:05.805886+0000","last_clean":"2023-10-30T14:14:05.805886+0000","last_became_active":"2023-10-30T14:14:04.597099+0000","last_became_peered":"2023-10-30T14:14:04.597099+0000","last_unstale":"2023-10-30T14:14:05.805886+0000","last_undegraded":"2023-10-30T14:14:05.805886+0000","last_fullsized":"2023-10-30T14:14:05.805886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:54:52.346295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.965041+0000","last_change":"2023-10-30T14:14:03.721030+0000","last_active":"2023-10-30T14:14:06.965041+0000","last_peered":"2023-10-30T14:14:06.965041+0000","last_clean":"2023-10-30T14:14:06.965041+0000","last_became_active":"2023-10-30T14:14:03.719816+0000","last_became_peered":"2023-10-30T14:14:03.719816+0000","last_unstale":"2023-10-30T14:14:06.965041+0000","last_undegraded":"2023-10-30T14:14:06.965041+0000","last_fullsized":"2023-10-30T14:14:06.965041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:43:50.150156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584729+0000","last_change":"2023-10-30T14:14:04.599863+0000","last_active":"2023-10-30T14:14:05.584729+0000","last_peered":"2023-10-30T14:14:05.584729+0000","last_clean":"2023-10-30T14:14:05.584729+0000","last_became_active":"2023-10-30T14:14:04.599323+0000","last_became_peered":"2023-10-30T14:14:04.599323+0000","last_unstale":"2023-10-30T14:14:05.584729+0000","last_undegraded":"2023-10-30T14:14:05.584729+0000","last_fullsized":"2023-10-30T14:14:05.584729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:42:36.939959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.773337+0000","last_change":"2023-10-30T14:14:03.738694+0000","last_active":"2023-10-30T14:14:14.773337+0000","last_peered":"2023-10-30T14:14:14.773337+0000","last_clean":"2023-10-30T14:14:14.773337+0000","last_became_active":"2023-10-30T14:14:03.738132+0000","last_became_peered":"2023-10-30T14:14:03.738132+0000","last_unstale":"2023-10-30T14:14:14.773337+0000","last_undegraded":"2023-10-30T14:14:14.773337+0000","last_fullsized":"2023-10-30T14:14:14.773337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:46:59.692900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586432+0000","last_change":"2023-10-30T14:14:04.649502+0000","last_active":"2023-10-30T14:14:05.586432+0000","last_peered":"2023-10-30T14:14:05.586432+0000","last_clean":"2023-10-30T14:14:05.586432+0000","last_became_active":"2023-10-30T14:14:04.648816+0000","last_became_peered":"2023-10-30T14:14:04.648816+0000","last_unstale":"2023-10-30T14:14:05.586432+0000","last_undegraded":"2023-10-30T14:14:05.586432+0000","last_fullsized":"2023-10-30T14:14:05.586432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:23:07.149253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.769420+0000","last_change":"2023-10-30T14:14:03.726112+0000","last_active":"2023-10-30T14:14:14.769420+0000","last_peered":"2023-10-30T14:14:14.769420+0000","last_clean":"2023-10-30T14:14:14.769420+0000","last_became_active":"2023-10-30T14:14:03.725272+0000","last_became_peered":"2023-10-30T14:14:03.725272+0000","last_unstale":"2023-10-30T14:14:14.769420+0000","last_undegraded":"2023-10-30T14:14:14.769420+0000","last_fullsized":"2023-10-30T14:14:14.769420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:37:20.764294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586677+0000","last_change":"2023-10-30T14:14:04.610043+0000","last_active":"2023-10-30T14:14:05.586677+0000","last_peered":"2023-10-30T14:14:05.586677+0000","last_clean":"2023-10-30T14:14:05.586677+0000","last_became_active":"2023-10-30T14:14:04.609697+0000","last_became_peered":"2023-10-30T14:14:04.609697+0000","last_unstale":"2023-10-30T14:14:05.586677+0000","last_undegraded":"2023-10-30T14:14:05.586677+0000","last_fullsized":"2023-10-30T14:14:05.586677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:28.543547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.721037+0000","last_change":"2023-10-30T14:14:03.727871+0000","last_active":"2023-10-30T14:14:09.721037+0000","last_peered":"2023-10-30T14:14:09.721037+0000","last_clean":"2023-10-30T14:14:09.721037+0000","last_became_active":"2023-10-30T14:14:03.727422+0000","last_became_peered":"2023-10-30T14:14:03.727422+0000","last_unstale":"2023-10-30T14:14:09.721037+0000","last_undegraded":"2023-10-30T14:14:09.721037+0000","last_fullsized":"2023-10-30T14:14:09.721037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:54.782533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809764+0000","last_change":"2023-10-30T14:14:04.645734+0000","last_active":"2023-10-30T14:14:05.809764+0000","last_peered":"2023-10-30T14:14:05.809764+0000","last_clean":"2023-10-30T14:14:05.809764+0000","last_became_active":"2023-10-30T14:14:04.645333+0000","last_became_peered":"2023-10-30T14:14:04.645333+0000","last_unstale":"2023-10-30T14:14:05.809764+0000","last_undegraded":"2023-10-30T14:14:05.809764+0000","last_fullsized":"2023-10-30T14:14:05.809764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:36:11.322798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'11","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:28.884184+0000","last_change":"2023-10-30T14:14:03.725554+0000","last_active":"2023-10-30T14:14:28.884184+0000","last_peered":"2023-10-30T14:14:28.884184+0000","last_clean":"2023-10-30T14:14:28.884184+0000","last_became_active":"2023-10-30T14:14:03.724806+0000","last_became_peered":"2023-10-30T14:14:03.724806+0000","last_unstale":"2023-10-30T14:14:28.884184+0000","last_undegraded":"2023-10-30T14:14:28.884184+0000","last_fullsized":"2023-10-30T14:14:28.884184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:37.823561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2685,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756505+0000","last_change":"2023-10-30T14:14:04.647122+0000","last_active":"2023-10-30T14:14:05.756505+0000","last_peered":"2023-10-30T14:14:05.756505+0000","last_clean":"2023-10-30T14:14:05.756505+0000","last_became_active":"2023-10-30T14:14:04.646543+0000","last_became_peered":"2023-10-30T14:14:04.646543+0000","last_unstale":"2023-10-30T14:14:05.756505+0000","last_undegraded":"2023-10-30T14:14:05.756505+0000","last_fullsized":"2023-10-30T14:14:05.756505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:16.158442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.822260+0000","last_change":"2023-10-30T14:14:03.717903+0000","last_active":"2023-10-30T14:14:20.822260+0000","last_peered":"2023-10-30T14:14:20.822260+0000","last_clean":"2023-10-30T14:14:20.822260+0000","last_became_active":"2023-10-30T14:14:03.716941+0000","last_became_peered":"2023-10-30T14:14:03.716941+0000","last_unstale":"2023-10-30T14:14:20.822260+0000","last_undegraded":"2023-10-30T14:14:20.822260+0000","last_fullsized":"2023-10-30T14:14:20.822260+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:58.178166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586173+0000","last_change":"2023-10-30T14:14:04.641757+0000","last_active":"2023-10-30T14:14:05.586173+0000","last_peered":"2023-10-30T14:14:05.586173+0000","last_clean":"2023-10-30T14:14:05.586173+0000","last_became_active":"2023-10-30T14:14:04.641494+0000","last_became_peered":"2023-10-30T14:14:04.641494+0000","last_unstale":"2023-10-30T14:14:05.586173+0000","last_undegraded":"2023-10-30T14:14:05.586173+0000","last_fullsized":"2023-10-30T14:14:05.586173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:00:18.795083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.773009+0000","last_change":"2023-10-30T14:14:03.729626+0000","last_active":"2023-10-30T14:14:14.773009+0000","last_peered":"2023-10-30T14:14:14.773009+0000","last_clean":"2023-10-30T14:14:14.773009+0000","last_became_active":"2023-10-30T14:14:03.728053+0000","last_became_peered":"2023-10-30T14:14:03.728053+0000","last_unstale":"2023-10-30T14:14:14.773009+0000","last_undegraded":"2023-10-30T14:14:14.773009+0000","last_fullsized":"2023-10-30T14:14:14.773009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:44:53.362066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756773+0000","last_change":"2023-10-30T14:14:04.650513+0000","last_active":"2023-10-30T14:14:05.756773+0000","last_peered":"2023-10-30T14:14:05.756773+0000","last_clean":"2023-10-30T14:14:05.756773+0000","last_became_active":"2023-10-30T14:14:04.649896+0000","last_became_peered":"2023-10-30T14:14:04.649896+0000","last_unstale":"2023-10-30T14:14:05.756773+0000","last_undegraded":"2023-10-30T14:14:05.756773+0000","last_fullsized":"2023-10-30T14:14:05.756773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:23.100490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.832690+0000","last_change":"2023-10-30T14:14:03.736824+0000","last_active":"2023-10-30T14:14:20.832690+0000","last_peered":"2023-10-30T14:14:20.832690+0000","last_clean":"2023-10-30T14:14:20.832690+0000","last_became_active":"2023-10-30T14:14:03.736155+0000","last_became_peered":"2023-10-30T14:14:03.736155+0000","last_unstale":"2023-10-30T14:14:20.832690+0000","last_undegraded":"2023-10-30T14:14:20.832690+0000","last_fullsized":"2023-10-30T14:14:20.832690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:35.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":846,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807372+0000","last_change":"2023-10-30T14:14:04.615159+0000","last_active":"2023-10-30T14:14:05.807372+0000","last_peered":"2023-10-30T14:14:05.807372+0000","last_clean":"2023-10-30T14:14:05.807372+0000","last_became_active":"2023-10-30T14:14:04.614328+0000","last_became_peered":"2023-10-30T14:14:04.614328+0000","last_unstale":"2023-10-30T14:14:05.807372+0000","last_undegraded":"2023-10-30T14:14:05.807372+0000","last_fullsized":"2023-10-30T14:14:05.807372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:16:52.999786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'15","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.823603+0000","last_change":"2023-10-30T14:14:03.700690+0000","last_active":"2023-10-30T14:14:20.823603+0000","last_peered":"2023-10-30T14:14:20.823603+0000","last_clean":"2023-10-30T14:14:20.823603+0000","last_became_active":"2023-10-30T14:14:03.700352+0000","last_became_peered":"2023-10-30T14:14:03.700352+0000","last_unstale":"2023-10-30T14:14:20.823603+0000","last_undegraded":"2023-10-30T14:14:20.823603+0000","last_fullsized":"2023-10-30T14:14:20.823603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:03:01.103359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755948+0000","last_change":"2023-10-30T14:14:04.647253+0000","last_active":"2023-10-30T14:14:05.755948+0000","last_peered":"2023-10-30T14:14:05.755948+0000","last_clean":"2023-10-30T14:14:05.755948+0000","last_became_active":"2023-10-30T14:14:04.646639+0000","last_became_peered":"2023-10-30T14:14:04.646639+0000","last_unstale":"2023-10-30T14:14:05.755948+0000","last_undegraded":"2023-10-30T14:14:05.755948+0000","last_fullsized":"2023-10-30T14:14:05.755948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:22:50.893734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.722823+0000","last_change":"2023-10-30T14:14:03.710717+0000","last_active":"2023-10-30T14:14:09.722823+0000","last_peered":"2023-10-30T14:14:09.722823+0000","last_clean":"2023-10-30T14:14:09.722823+0000","last_became_active":"2023-10-30T14:14:03.707950+0000","last_became_peered":"2023-10-30T14:14:03.707950+0000","last_unstale":"2023-10-30T14:14:09.722823+0000","last_undegraded":"2023-10-30T14:14:09.722823+0000","last_fullsized":"2023-10-30T14:14:09.722823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:29:41.232902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757066+0000","last_change":"2023-10-30T14:14:04.647688+0000","last_active":"2023-10-30T14:14:05.757066+0000","last_peered":"2023-10-30T14:14:05.757066+0000","last_clean":"2023-10-30T14:14:05.757066+0000","last_became_active":"2023-10-30T14:14:04.644761+0000","last_became_peered":"2023-10-30T14:14:04.644761+0000","last_unstale":"2023-10-30T14:14:05.757066+0000","last_undegraded":"2023-10-30T14:14:05.757066+0000","last_fullsized":"2023-10-30T14:14:05.757066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:00:38.482522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.716374+0000","last_change":"2023-10-30T14:14:03.705567+0000","last_active":"2023-10-30T14:14:09.716374+0000","last_peered":"2023-10-30T14:14:09.716374+0000","last_clean":"2023-10-30T14:14:09.716374+0000","last_became_active":"2023-10-30T14:14:03.705186+0000","last_became_peered":"2023-10-30T14:14:03.705186+0000","last_unstale":"2023-10-30T14:14:09.716374+0000","last_undegraded":"2023-10-30T14:14:09.716374+0000","last_fullsized":"2023-10-30T14:14:09.716374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:14.609243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757235+0000","last_change":"2023-10-30T14:14:04.647335+0000","last_active":"2023-10-30T14:14:05.757235+0000","last_peered":"2023-10-30T14:14:05.757235+0000","last_clean":"2023-10-30T14:14:05.757235+0000","last_became_active":"2023-10-30T14:14:04.646785+0000","last_became_peered":"2023-10-30T14:14:04.646785+0000","last_unstale":"2023-10-30T14:14:05.757235+0000","last_undegraded":"2023-10-30T14:14:05.757235+0000","last_fullsized":"2023-10-30T14:14:05.757235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:58:41.600761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.935812+0000","last_change":"2023-10-30T14:14:03.693349+0000","last_active":"2023-10-30T14:14:06.935812+0000","last_peered":"2023-10-30T14:14:06.935812+0000","last_clean":"2023-10-30T14:14:06.935812+0000","last_became_active":"2023-10-30T14:14:03.692966+0000","last_became_peered":"2023-10-30T14:14:03.692966+0000","last_unstale":"2023-10-30T14:14:06.935812+0000","last_undegraded":"2023-10-30T14:14:06.935812+0000","last_fullsized":"2023-10-30T14:14:06.935812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:30:28.800214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587213+0000","last_change":"2023-10-30T14:14:04.617023+0000","last_active":"2023-10-30T14:14:05.587213+0000","last_peered":"2023-10-30T14:14:05.587213+0000","last_clean":"2023-10-30T14:14:05.587213+0000","last_became_active":"2023-10-30T14:14:04.615925+0000","last_became_peered":"2023-10-30T14:14:04.615925+0000","last_unstale":"2023-10-30T14:14:05.587213+0000","last_undegraded":"2023-10-30T14:14:05.587213+0000","last_fullsized":"2023-10-30T14:14:05.587213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:32:02.289324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.823287+0000","last_change":"2023-10-30T14:14:03.727659+0000","last_active":"2023-10-30T14:14:20.823287+0000","last_peered":"2023-10-30T14:14:20.823287+0000","last_clean":"2023-10-30T14:14:20.823287+0000","last_became_active":"2023-10-30T14:14:03.727409+0000","last_became_peered":"2023-10-30T14:14:03.727409+0000","last_unstale":"2023-10-30T14:14:20.823287+0000","last_undegraded":"2023-10-30T14:14:20.823287+0000","last_fullsized":"2023-10-30T14:14:20.823287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:24.210635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756468+0000","last_change":"2023-10-30T14:14:04.650083+0000","last_active":"2023-10-30T14:14:05.756468+0000","last_peered":"2023-10-30T14:14:05.756468+0000","last_clean":"2023-10-30T14:14:05.756468+0000","last_became_active":"2023-10-30T14:14:04.647562+0000","last_became_peered":"2023-10-30T14:14:04.647562+0000","last_unstale":"2023-10-30T14:14:05.756468+0000","last_undegraded":"2023-10-30T14:14:05.756468+0000","last_fullsized":"2023-10-30T14:14:05.756468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:11:41.743850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T14:14:04.632968+0000","last_change":"2023-10-30T14:13:54.251572+0000","last_active":"2023-10-30T14:14:04.632968+0000","last_peered":"2023-10-30T14:14:04.632968+0000","last_clean":"2023-10-30T14:14:04.632968+0000","last_became_active":"2023-10-30T14:13:54.250907+0000","last_became_peered":"2023-10-30T14:13:54.250907+0000","last_unstale":"2023-10-30T14:14:04.632968+0000","last_undegraded":"2023-10-30T14:14:04.632968+0000","last_fullsized":"2023-10-30T14:14:04.632968+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_clean_scrub_stamp":"2023-10-30T14:13:53.220267+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:44.590852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.779821+0000","last_change":"2023-10-30T14:14:03.729034+0000","last_active":"2023-10-30T14:14:14.779821+0000","last_peered":"2023-10-30T14:14:14.779821+0000","last_clean":"2023-10-30T14:14:14.779821+0000","last_became_active":"2023-10-30T14:14:03.728293+0000","last_became_peered":"2023-10-30T14:14:03.728293+0000","last_unstale":"2023-10-30T14:14:14.779821+0000","last_undegraded":"2023-10-30T14:14:14.779821+0000","last_fullsized":"2023-10-30T14:14:14.779821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806225+0000","last_change":"2023-10-30T14:14:04.630325+0000","last_active":"2023-10-30T14:14:05.806225+0000","last_peered":"2023-10-30T14:14:05.806225+0000","last_clean":"2023-10-30T14:14:05.806225+0000","last_became_active":"2023-10-30T14:14:04.630121+0000","last_became_peered":"2023-10-30T14:14:04.630121+0000","last_unstale":"2023-10-30T14:14:05.806225+0000","last_undegraded":"2023-10-30T14:14:05.806225+0000","last_fullsized":"2023-10-30T14:14:05.806225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:02:28.792879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.822027+0000","last_change":"2023-10-30T14:14:03.732909+0000","last_active":"2023-10-30T14:14:20.822027+0000","last_peered":"2023-10-30T14:14:20.822027+0000","last_clean":"2023-10-30T14:14:20.822027+0000","last_became_active":"2023-10-30T14:14:03.732199+0000","last_became_peered":"2023-10-30T14:14:03.732199+0000","last_unstale":"2023-10-30T14:14:20.822027+0000","last_undegraded":"2023-10-30T14:14:20.822027+0000","last_fullsized":"2023-10-30T14:14:20.822027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:38.117750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586172+0000","last_change":"2023-10-30T14:14:04.620314+0000","last_active":"2023-10-30T14:14:05.586172+0000","last_peered":"2023-10-30T14:14:05.586172+0000","last_clean":"2023-10-30T14:14:05.586172+0000","last_became_active":"2023-10-30T14:14:04.619990+0000","last_became_peered":"2023-10-30T14:14:04.619990+0000","last_unstale":"2023-10-30T14:14:05.586172+0000","last_undegraded":"2023-10-30T14:14:05.586172+0000","last_fullsized":"2023-10-30T14:14:05.586172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:38:25.657586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.822214+0000","last_change":"2023-10-30T14:14:03.702862+0000","last_active":"2023-10-30T14:14:20.822214+0000","last_peered":"2023-10-30T14:14:20.822214+0000","last_clean":"2023-10-30T14:14:20.822214+0000","last_became_active":"2023-10-30T14:14:03.702513+0000","last_became_peered":"2023-10-30T14:14:03.702513+0000","last_unstale":"2023-10-30T14:14:20.822214+0000","last_undegraded":"2023-10-30T14:14:20.822214+0000","last_fullsized":"2023-10-30T14:14:20.822214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:25.499390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584623+0000","last_change":"2023-10-30T14:14:04.589353+0000","last_active":"2023-10-30T14:14:05.584623+0000","last_peered":"2023-10-30T14:14:05.584623+0000","last_clean":"2023-10-30T14:14:05.584623+0000","last_became_active":"2023-10-30T14:14:04.589087+0000","last_became_peered":"2023-10-30T14:14:04.589087+0000","last_unstale":"2023-10-30T14:14:05.584623+0000","last_undegraded":"2023-10-30T14:14:05.584623+0000","last_fullsized":"2023-10-30T14:14:05.584623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:20:12.808641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.965154+0000","last_change":"2023-10-30T14:14:03.730048+0000","last_active":"2023-10-30T14:14:06.965154+0000","last_peered":"2023-10-30T14:14:06.965154+0000","last_clean":"2023-10-30T14:14:06.965154+0000","last_became_active":"2023-10-30T14:14:03.728925+0000","last_became_peered":"2023-10-30T14:14:03.728925+0000","last_unstale":"2023-10-30T14:14:06.965154+0000","last_undegraded":"2023-10-30T14:14:06.965154+0000","last_fullsized":"2023-10-30T14:14:06.965154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:32:05.056039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809437+0000","last_change":"2023-10-30T14:14:04.608289+0000","last_active":"2023-10-30T14:14:05.809437+0000","last_peered":"2023-10-30T14:14:05.809437+0000","last_clean":"2023-10-30T14:14:05.809437+0000","last_became_active":"2023-10-30T14:14:04.607377+0000","last_became_peered":"2023-10-30T14:14:04.607377+0000","last_unstale":"2023-10-30T14:14:05.809437+0000","last_undegraded":"2023-10-30T14:14:05.809437+0000","last_fullsized":"2023-10-30T14:14:05.809437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:43:40.869579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584795+0000","last_change":"2023-10-30T14:14:03.693753+0000","last_active":"2023-10-30T14:14:05.584795+0000","last_peered":"2023-10-30T14:14:05.584795+0000","last_clean":"2023-10-30T14:14:05.584795+0000","last_became_active":"2023-10-30T14:14:03.692621+0000","last_became_peered":"2023-10-30T14:14:03.692621+0000","last_unstale":"2023-10-30T14:14:05.584795+0000","last_undegraded":"2023-10-30T14:14:05.584795+0000","last_fullsized":"2023-10-30T14:14:05.584795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:32:28.577584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587225+0000","last_change":"2023-10-30T14:14:04.614649+0000","last_active":"2023-10-30T14:14:05.587225+0000","last_peered":"2023-10-30T14:14:05.587225+0000","last_clean":"2023-10-30T14:14:05.587225+0000","last_became_active":"2023-10-30T14:14:04.613780+0000","last_became_peered":"2023-10-30T14:14:04.613780+0000","last_unstale":"2023-10-30T14:14:05.587225+0000","last_undegraded":"2023-10-30T14:14:05.587225+0000","last_fullsized":"2023-10-30T14:14:05.587225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:47:40.354022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.937485+0000","last_change":"2023-10-30T14:14:03.740213+0000","last_active":"2023-10-30T14:14:06.937485+0000","last_peered":"2023-10-30T14:14:06.937485+0000","last_clean":"2023-10-30T14:14:06.937485+0000","last_became_active":"2023-10-30T14:14:03.739650+0000","last_became_peered":"2023-10-30T14:14:03.739650+0000","last_unstale":"2023-10-30T14:14:06.937485+0000","last_undegraded":"2023-10-30T14:14:06.937485+0000","last_fullsized":"2023-10-30T14:14:06.937485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:54.120757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584691+0000","last_change":"2023-10-30T14:14:04.590569+0000","last_active":"2023-10-30T14:14:05.584691+0000","last_peered":"2023-10-30T14:14:05.584691+0000","last_clean":"2023-10-30T14:14:05.584691+0000","last_became_active":"2023-10-30T14:14:04.590138+0000","last_became_peered":"2023-10-30T14:14:04.590138+0000","last_unstale":"2023-10-30T14:14:05.584691+0000","last_undegraded":"2023-10-30T14:14:05.584691+0000","last_fullsized":"2023-10-30T14:14:05.584691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:41:30.333668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.824452+0000","last_change":"2023-10-30T14:14:03.721450+0000","last_active":"2023-10-30T14:14:20.824452+0000","last_peered":"2023-10-30T14:14:20.824452+0000","last_clean":"2023-10-30T14:14:20.824452+0000","last_became_active":"2023-10-30T14:14:03.721234+0000","last_became_peered":"2023-10-30T14:14:03.721234+0000","last_unstale":"2023-10-30T14:14:20.824452+0000","last_undegraded":"2023-10-30T14:14:20.824452+0000","last_fullsized":"2023-10-30T14:14:20.824452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:26.959676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806005+0000","last_change":"2023-10-30T14:14:04.634004+0000","last_active":"2023-10-30T14:14:05.806005+0000","last_peered":"2023-10-30T14:14:05.806005+0000","last_clean":"2023-10-30T14:14:05.806005+0000","last_became_active":"2023-10-30T14:14:04.633643+0000","last_became_peered":"2023-10-30T14:14:04.633643+0000","last_unstale":"2023-10-30T14:14:05.806005+0000","last_undegraded":"2023-10-30T14:14:05.806005+0000","last_fullsized":"2023-10-30T14:14:05.806005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:05:14.572477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.769216+0000","last_change":"2023-10-30T14:14:03.738557+0000","last_active":"2023-10-30T14:14:14.769216+0000","last_peered":"2023-10-30T14:14:14.769216+0000","last_clean":"2023-10-30T14:14:14.769216+0000","last_became_active":"2023-10-30T14:14:03.737801+0000","last_became_peered":"2023-10-30T14:14:03.737801+0000","last_unstale":"2023-10-30T14:14:14.769216+0000","last_undegraded":"2023-10-30T14:14:14.769216+0000","last_fullsized":"2023-10-30T14:14:14.769216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:26.938343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586581+0000","last_change":"2023-10-30T14:14:04.589273+0000","last_active":"2023-10-30T14:14:05.586581+0000","last_peered":"2023-10-30T14:14:05.586581+0000","last_clean":"2023-10-30T14:14:05.586581+0000","last_became_active":"2023-10-30T14:14:04.588826+0000","last_became_peered":"2023-10-30T14:14:04.588826+0000","last_unstale":"2023-10-30T14:14:05.586581+0000","last_undegraded":"2023-10-30T14:14:05.586581+0000","last_fullsized":"2023-10-30T14:14:05.586581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:33:52.452394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.787158+0000","last_change":"2023-10-30T14:14:03.733836+0000","last_active":"2023-10-30T14:14:14.787158+0000","last_peered":"2023-10-30T14:14:14.787158+0000","last_clean":"2023-10-30T14:14:14.787158+0000","last_became_active":"2023-10-30T14:14:03.732912+0000","last_became_peered":"2023-10-30T14:14:03.732912+0000","last_unstale":"2023-10-30T14:14:14.787158+0000","last_undegraded":"2023-10-30T14:14:14.787158+0000","last_fullsized":"2023-10-30T14:14:14.787158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:50.332738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809314+0000","last_change":"2023-10-30T14:14:04.610696+0000","last_active":"2023-10-30T14:14:05.809314+0000","last_peered":"2023-10-30T14:14:05.809314+0000","last_clean":"2023-10-30T14:14:05.809314+0000","last_became_active":"2023-10-30T14:14:04.609835+0000","last_became_peered":"2023-10-30T14:14:04.609835+0000","last_unstale":"2023-10-30T14:14:05.809314+0000","last_undegraded":"2023-10-30T14:14:05.809314+0000","last_fullsized":"2023-10-30T14:14:05.809314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:04:31.270262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.795220+0000","last_change":"2023-10-30T14:14:03.732539+0000","last_active":"2023-10-30T14:14:14.795220+0000","last_peered":"2023-10-30T14:14:14.795220+0000","last_clean":"2023-10-30T14:14:14.795220+0000","last_became_active":"2023-10-30T14:14:03.732265+0000","last_became_peered":"2023-10-30T14:14:03.732265+0000","last_unstale":"2023-10-30T14:14:14.795220+0000","last_undegraded":"2023-10-30T14:14:14.795220+0000","last_fullsized":"2023-10-30T14:14:14.795220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:02:00.929612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806211+0000","last_change":"2023-10-30T14:14:04.595357+0000","last_active":"2023-10-30T14:14:05.806211+0000","last_peered":"2023-10-30T14:14:05.806211+0000","last_clean":"2023-10-30T14:14:05.806211+0000","last_became_active":"2023-10-30T14:14:04.594998+0000","last_became_peered":"2023-10-30T14:14:04.594998+0000","last_unstale":"2023-10-30T14:14:05.806211+0000","last_undegraded":"2023-10-30T14:14:05.806211+0000","last_fullsized":"2023-10-30T14:14:05.806211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:44.365673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.935738+0000","last_change":"2023-10-30T14:14:03.703744+0000","last_active":"2023-10-30T14:14:06.935738+0000","last_peered":"2023-10-30T14:14:06.935738+0000","last_clean":"2023-10-30T14:14:06.935738+0000","last_became_active":"2023-10-30T14:14:03.701759+0000","last_became_peered":"2023-10-30T14:14:03.701759+0000","last_unstale":"2023-10-30T14:14:06.935738+0000","last_undegraded":"2023-10-30T14:14:06.935738+0000","last_fullsized":"2023-10-30T14:14:06.935738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:22:27.948007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806186+0000","last_change":"2023-10-30T14:14:04.600759+0000","last_active":"2023-10-30T14:14:05.806186+0000","last_peered":"2023-10-30T14:14:05.806186+0000","last_clean":"2023-10-30T14:14:05.806186+0000","last_became_active":"2023-10-30T14:14:04.600100+0000","last_became_peered":"2023-10-30T14:14:04.600100+0000","last_unstale":"2023-10-30T14:14:05.806186+0000","last_undegraded":"2023-10-30T14:14:05.806186+0000","last_fullsized":"2023-10-30T14:14:05.806186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:02.831207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.773849+0000","last_change":"2023-10-30T14:14:03.741268+0000","last_active":"2023-10-30T14:14:14.773849+0000","last_peered":"2023-10-30T14:14:14.773849+0000","last_clean":"2023-10-30T14:14:14.773849+0000","last_became_active":"2023-10-30T14:14:03.740648+0000","last_became_peered":"2023-10-30T14:14:03.740648+0000","last_unstale":"2023-10-30T14:14:14.773849+0000","last_undegraded":"2023-10-30T14:14:14.773849+0000","last_fullsized":"2023-10-30T14:14:14.773849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:30.799148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807541+0000","last_change":"2023-10-30T14:14:04.623644+0000","last_active":"2023-10-30T14:14:05.807541+0000","last_peered":"2023-10-30T14:14:05.807541+0000","last_clean":"2023-10-30T14:14:05.807541+0000","last_became_active":"2023-10-30T14:14:04.622921+0000","last_became_peered":"2023-10-30T14:14:04.622921+0000","last_unstale":"2023-10-30T14:14:05.807541+0000","last_undegraded":"2023-10-30T14:14:05.807541+0000","last_fullsized":"2023-10-30T14:14:05.807541+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:11:41.810610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587103+0000","last_change":"2023-10-30T14:14:04.624799+0000","last_active":"2023-10-30T14:14:05.587103+0000","last_peered":"2023-10-30T14:14:05.587103+0000","last_clean":"2023-10-30T14:14:05.587103+0000","last_became_active":"2023-10-30T14:14:04.621885+0000","last_became_peered":"2023-10-30T14:14:04.621885+0000","last_unstale":"2023-10-30T14:14:05.587103+0000","last_undegraded":"2023-10-30T14:14:05.587103+0000","last_fullsized":"2023-10-30T14:14:05.587103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:12.055247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.721291+0000","last_change":"2023-10-30T14:14:03.740897+0000","last_active":"2023-10-30T14:14:09.721291+0000","last_peered":"2023-10-30T14:14:09.721291+0000","last_clean":"2023-10-30T14:14:09.721291+0000","last_became_active":"2023-10-30T14:14:03.740247+0000","last_became_peered":"2023-10-30T14:14:03.740247+0000","last_unstale":"2023-10-30T14:14:09.721291+0000","last_undegraded":"2023-10-30T14:14:09.721291+0000","last_fullsized":"2023-10-30T14:14:09.721291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:29.601978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586861+0000","last_change":"2023-10-30T14:14:04.625498+0000","last_active":"2023-10-30T14:14:05.586861+0000","last_peered":"2023-10-30T14:14:05.586861+0000","last_clean":"2023-10-30T14:14:05.586861+0000","last_became_active":"2023-10-30T14:14:04.623638+0000","last_became_peered":"2023-10-30T14:14:04.623638+0000","last_unstale":"2023-10-30T14:14:05.586861+0000","last_undegraded":"2023-10-30T14:14:05.586861+0000","last_fullsized":"2023-10-30T14:14:05.586861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:19.717965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.825113+0000","last_change":"2023-10-30T14:14:03.733233+0000","last_active":"2023-10-30T14:14:20.825113+0000","last_peered":"2023-10-30T14:14:20.825113+0000","last_clean":"2023-10-30T14:14:20.825113+0000","last_became_active":"2023-10-30T14:14:03.732538+0000","last_became_peered":"2023-10-30T14:14:03.732538+0000","last_unstale":"2023-10-30T14:14:20.825113+0000","last_undegraded":"2023-10-30T14:14:20.825113+0000","last_fullsized":"2023-10-30T14:14:20.825113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:25.696173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586619+0000","last_change":"2023-10-30T14:14:04.612104+0000","last_active":"2023-10-30T14:14:05.586619+0000","last_peered":"2023-10-30T14:14:05.586619+0000","last_clean":"2023-10-30T14:14:05.586619+0000","last_became_active":"2023-10-30T14:14:04.611773+0000","last_became_peered":"2023-10-30T14:14:04.611773+0000","last_unstale":"2023-10-30T14:14:05.586619+0000","last_undegraded":"2023-10-30T14:14:05.586619+0000","last_fullsized":"2023-10-30T14:14:05.586619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:44:21.952103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.721542+0000","last_change":"2023-10-30T14:14:03.703988+0000","last_active":"2023-10-30T14:14:09.721542+0000","last_peered":"2023-10-30T14:14:09.721542+0000","last_clean":"2023-10-30T14:14:09.721542+0000","last_became_active":"2023-10-30T14:14:03.703691+0000","last_became_peered":"2023-10-30T14:14:03.703691+0000","last_unstale":"2023-10-30T14:14:09.721542+0000","last_undegraded":"2023-10-30T14:14:09.721542+0000","last_fullsized":"2023-10-30T14:14:09.721542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:34.538541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806119+0000","last_change":"2023-10-30T14:14:04.613738+0000","last_active":"2023-10-30T14:14:05.806119+0000","last_peered":"2023-10-30T14:14:05.806119+0000","last_clean":"2023-10-30T14:14:05.806119+0000","last_became_active":"2023-10-30T14:14:04.613420+0000","last_became_peered":"2023-10-30T14:14:04.613420+0000","last_unstale":"2023-10-30T14:14:05.806119+0000","last_undegraded":"2023-10-30T14:14:05.806119+0000","last_fullsized":"2023-10-30T14:14:05.806119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:01:52.695195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'9","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.825364+0000","last_change":"2023-10-30T14:14:03.737319+0000","last_active":"2023-10-30T14:14:20.825364+0000","last_peered":"2023-10-30T14:14:20.825364+0000","last_clean":"2023-10-30T14:14:20.825364+0000","last_became_active":"2023-10-30T14:14:03.736964+0000","last_became_peered":"2023-10-30T14:14:03.736964+0000","last_unstale":"2023-10-30T14:14:20.825364+0000","last_undegraded":"2023-10-30T14:14:20.825364+0000","last_fullsized":"2023-10-30T14:14:20.825364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:41.469809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6553,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":13106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113,"ondisk_log_size":113,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":831488,"data_stored":667253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2849999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.462}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3580000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":79190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1160000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]}]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":76947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.018}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":76541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.45}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6719999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]}]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":241664,"data_stored":78966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":76387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":76513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574861,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":827392,"data_stored":666597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:14:33.486 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:14:33.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.660+0000 7f493c054640 1 Processor -- start 2023-10-30T14:14:33.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.660+0000 7f493c054640 1 -- start start 2023-10-30T14:14:33.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.660+0000 7f493c054640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f493406b760 0x7f493415e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:33.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.660+0000 7f493c054640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4934071e00 con 0x7f493406c420 2023-10-30T14:14:33.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.660+0000 7f493c054640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4934070020 con 0x7f493406bb40 2023-10-30T14:14:33.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.660+0000 7f493c054640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f493415eb90 con 0x7f493406b760 2023-10-30T14:14:33.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.660+0000 7f493a5f1640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f493406c420 0x7f49340c57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41464/0 (socket says 172.21.15.57:41464) 2023-10-30T14:14:33.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.660+0000 7f493a5f1640 1 -- 172.21.15.57:0/1113088547 learned_addr learned my addr 172.21.15.57:0/1113088547 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:33.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.660+0000 7f4939df0640 1 --2- 172.21.15.57:0/1113088547 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f493406b760 0x7f493415e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:33.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.664+0000 7f4938dee640 1 -- 172.21.15.57:0/1113088547 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246102227 0 0) 0x7f4934071e00 con 0x7f493406c420 2023-10-30T14:14:33.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.664+0000 7f4938dee640 1 -- 172.21.15.57:0/1113088547 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4920003530 con 0x7f493406c420 2023-10-30T14:14:33.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.664+0000 7f4938dee640 1 -- 172.21.15.57:0/1113088547 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896101981 0 0) 0x7f4934070020 con 0x7f493406bb40 2023-10-30T14:14:33.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.664+0000 7f4938dee640 1 -- 172.21.15.57:0/1113088547 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4934071e00 con 0x7f493406bb40 2023-10-30T14:14:33.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.664+0000 7f4939df0640 1 -- 172.21.15.57:0/1113088547 >> v1:172.21.15.88:6789/0 conn(0x7f493406bb40 legacy=0x7f4934157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:33.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.664+0000 7f4939df0640 1 -- 172.21.15.57:0/1113088547 >> v1:172.21.15.57:6789/0 conn(0x7f493406c420 legacy=0x7f49340c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:33.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.664+0000 7f4938dee640 1 -- 172.21.15.57:0/1113088547 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2986239302 0 0) 0x7f4920003530 con 0x7f493406c420 2023-10-30T14:14:33.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.664+0000 7f4939df0640 1 -- 172.21.15.57:0/1113088547 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4934070f10 con 0x7f493406b760 2023-10-30T14:14:33.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.664+0000 7f4939df0640 1 --2- 172.21.15.57:0/1113088547 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f493406b760 0x7f493415e650 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f49300097f0 tx=0x7f493001c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cdd8dea17ab1915 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:33.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.664+0000 7f4938dee640 1 -- 172.21.15.57:0/1113088547 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c001a00 con 0x7f493406b760 2023-10-30T14:14:33.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f4938dee640 1 -- 172.21.15.57:0/1113088547 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49300a7050 con 0x7f493406b760 2023-10-30T14:14:33.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f4938dee640 1 -- 172.21.15.57:0/1113088547 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49300ab620 con 0x7f493406b760 2023-10-30T14:14:33.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f493c054640 1 -- 172.21.15.57:0/1113088547 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f493406b760 msgr2=0x7f493415e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:33.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f493c054640 1 --2- 172.21.15.57:0/1113088547 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f493406b760 0x7f493415e650 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f49300097f0 tx=0x7f493001c3b0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f493c054640 1 -- 172.21.15.57:0/1113088547 shutdown_connections 2023-10-30T14:14:33.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f493c054640 1 --2- 172.21.15.57:0/1113088547 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f493406b760 0x7f493415e650 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f493c054640 1 -- 172.21.15.57:0/1113088547 >> 172.21.15.57:0/1113088547 conn(0x7f49340cac70 msgr2=0x7f49340cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:33.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f493c054640 1 -- 172.21.15.57:0/1113088547 shutdown_connections 2023-10-30T14:14:33.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f493c054640 1 -- 172.21.15.57:0/1113088547 wait complete. 2023-10-30T14:14:33.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f493c054640 1 Processor -- start 2023-10-30T14:14:33.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f493c054640 1 -- start start 2023-10-30T14:14:33.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f493c054640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f493406b760 0x7f4934148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:33.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f493c054640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f493406bb40 0x7f49340c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:33.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f493c054640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f493406c420 0x7f49340c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:33.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f493c054640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493415d5e0 con 0x7f493406c420 2023-10-30T14:14:33.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f493c054640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4934150910 con 0x7f493406bb40 2023-10-30T14:14:33.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f493c054640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49340c36a0 con 0x7f493406b760 2023-10-30T14:14:33.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f493a5f1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f493406c420 0x7f49340c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:33.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f493a5f1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f493406c420 0x7f49340c9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41160/0 (socket says 172.21.15.57:41160) 2023-10-30T14:14:33.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.668+0000 7f493a5f1640 1 -- 172.21.15.57:0/783116855 learned_addr learned my addr 172.21.15.57:0/783116855 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:33.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.672+0000 7f4939df0640 1 --2- 172.21.15.57:0/783116855 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f493406b760 0x7f4934148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:33.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.672+0000 7f49395ef640 1 --2- 172.21.15.57:0/783116855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f493406bb40 0x7f49340c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:33.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.672+0000 7f493a5f1640 1 -- 172.21.15.57:0/783116855 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f493406b760 msgr2=0x7f4934148080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:33.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.672+0000 7f493a5f1640 1 --2- 172.21.15.57:0/783116855 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f493406b760 0x7f4934148080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.672+0000 7f493a5f1640 1 -- 172.21.15.57:0/783116855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f493406bb40 msgr2=0x7f49340c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:33.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.672+0000 7f493a5f1640 1 --2- 172.21.15.57:0/783116855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f493406bb40 0x7f49340c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.672+0000 7f493a5f1640 1 -- 172.21.15.57:0/783116855 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4930009420 con 0x7f493406c420 2023-10-30T14:14:33.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.672+0000 7f493a5f1640 1 --2- 172.21.15.57:0/783116855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f493406c420 0x7f49340c9fe0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f492c002240 tx=0x7f492c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:33.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.672+0000 7f492a7fc640 1 -- 172.21.15.57:0/783116855 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c00ce50 con 0x7f493406c420 2023-10-30T14:14:33.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.672+0000 7f492a7fc640 1 -- 172.21.15.57:0/783116855 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492c012400 con 0x7f493406c420 2023-10-30T14:14:33.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.672+0000 7f492a7fc640 1 -- 172.21.15.57:0/783116855 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c0127b0 con 0x7f493406c420 2023-10-30T14:14:33.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.672+0000 7f493c054640 1 -- 172.21.15.57:0/783116855 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49340c3980 con 0x7f493406c420 2023-10-30T14:14:33.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.672+0000 7f493c054640 1 -- 172.21.15.57:0/783116855 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49340c4050 con 0x7f493406c420 2023-10-30T14:14:33.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.672+0000 7f493c054640 1 -- 172.21.15.57:0/783116855 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f493404e170 con 0x7f493406c420 2023-10-30T14:14:33.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.672+0000 7f492a7fc640 1 -- 172.21.15.57:0/783116855 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f492c012910 con 0x7f493406c420 2023-10-30T14:14:33.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.672+0000 7f492a7fc640 1 --2- 172.21.15.57:0/783116855 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f49180664a0 0x7f4918068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:33.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.672+0000 7f492a7fc640 1 -- 172.21.15.57:0/783116855 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f492c0845a0 con 0x7f493406c420 2023-10-30T14:14:33.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.676+0000 7f4939df0640 1 --2- 172.21.15.57:0/783116855 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f49180664a0 0x7f4918068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:33.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.676+0000 7f4939df0640 1 --2- 172.21.15.57:0/783116855 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f49180664a0 0x7f4918068960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f49300ad000 tx=0x7f49300a6000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:33.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.676+0000 7f492a7fc640 1 -- 172.21.15.57:0/783116855 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f492c0519a0 con 0x7f493406c420 2023-10-30T14:14:33.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.832+0000 7f493c054640 1 -- 172.21.15.57:0/783116855 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4934071a90 con 0x7f49180664a0 2023-10-30T14:14:33.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.852+0000 7f492a7fc640 1 -- 172.21.15.57:0/783116855 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264675 (secure 0 0 0) 0x7f49300ab5e0 con 0x7f49180664a0 2023-10-30T14:14:33.861 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:33.862 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-30T14:14:33.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.856+0000 7f493c054640 1 -- 172.21.15.57:0/783116855 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f49180664a0 msgr2=0x7f4918068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:33.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.856+0000 7f493c054640 1 --2- 172.21.15.57:0/783116855 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f49180664a0 0x7f4918068960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f49300ad000 tx=0x7f49300a6000 comp rx=0 tx=0).stop 2023-10-30T14:14:33.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.856+0000 7f493c054640 1 -- 172.21.15.57:0/783116855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f493406c420 msgr2=0x7f49340c9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:33.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.856+0000 7f493c054640 1 --2- 172.21.15.57:0/783116855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f493406c420 0x7f49340c9fe0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f492c002240 tx=0x7f492c00a430 comp rx=0 tx=0).stop 2023-10-30T14:14:33.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.856+0000 7f493c054640 1 -- 172.21.15.57:0/783116855 shutdown_connections 2023-10-30T14:14:33.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.856+0000 7f493c054640 1 --2- 172.21.15.57:0/783116855 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f49180664a0 0x7f4918068960 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.856+0000 7f493c054640 1 --2- 172.21.15.57:0/783116855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f493406c420 0x7f49340c9fe0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.856+0000 7f493c054640 1 --2- 172.21.15.57:0/783116855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f493406bb40 0x7f49340c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.856+0000 7f493c054640 1 --2- 172.21.15.57:0/783116855 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f493406b760 0x7f4934148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:33.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.856+0000 7f493c054640 1 -- 172.21.15.57:0/783116855 >> 172.21.15.57:0/783116855 conn(0x7f49340cac70 msgr2=0x7f49341567e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:33.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.856+0000 7f493c054640 1 -- 172.21.15.57:0/783116855 shutdown_connections 2023-10-30T14:14:33.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:33.856+0000 7f493c054640 1 -- 172.21.15.57:0/783116855 wait complete. 2023-10-30T14:14:33.881 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-10-30T14:14:32.504236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596921,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":288,"num_write_kb":2458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227464,"kb_used_data":3016,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754747256,"statfs":{"total":773094113280,"available":772861190144,"internally_reserved":0,"allocated":3088384,"data_stored":1798394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":229756473},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2737,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004216"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757043+0000","last_change":"2023-10-30T14:14:04.651413+0000","last_active":"2023-10-30T14:14:05.757043+0000","last_peered":"2023-10-30T14:14:05.757043+0000","last_clean":"2023-10-30T14:14:05.757043+0000","last_became_active":"2023-10-30T14:14:04.650882+0000","last_became_peered":"2023-10-30T14:14:04.650882+0000","last_unstale":"2023-10-30T14:14:05.757043+0000","last_undegraded":"2023-10-30T14:14:05.757043+0000","last_fullsized":"2023-10-30T14:14:05.757043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:30:08.122326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585036+0000","last_change":"2023-10-30T14:14:04.588523+0000","last_active":"2023-10-30T14:14:05.585036+0000","last_peered":"2023-10-30T14:14:05.585036+0000","last_clean":"2023-10-30T14:14:05.585036+0000","last_became_active":"2023-10-30T14:14:04.587942+0000","last_became_peered":"2023-10-30T14:14:04.587942+0000","last_unstale":"2023-10-30T14:14:05.585036+0000","last_undegraded":"2023-10-30T14:14:05.585036+0000","last_fullsized":"2023-10-30T14:14:05.585036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:16.028564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586543+0000","last_change":"2023-10-30T14:14:04.590976+0000","last_active":"2023-10-30T14:14:05.586543+0000","last_peered":"2023-10-30T14:14:05.586543+0000","last_clean":"2023-10-30T14:14:05.586543+0000","last_became_active":"2023-10-30T14:14:04.590700+0000","last_became_peered":"2023-10-30T14:14:04.590700+0000","last_unstale":"2023-10-30T14:14:05.586543+0000","last_undegraded":"2023-10-30T14:14:05.586543+0000","last_fullsized":"2023-10-30T14:14:05.586543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:49.169325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586525+0000","last_change":"2023-10-30T14:14:04.618603+0000","last_active":"2023-10-30T14:14:05.586525+0000","last_peered":"2023-10-30T14:14:05.586525+0000","last_clean":"2023-10-30T14:14:05.586525+0000","last_became_active":"2023-10-30T14:14:04.617768+0000","last_became_peered":"2023-10-30T14:14:04.617768+0000","last_unstale":"2023-10-30T14:14:05.586525+0000","last_undegraded":"2023-10-30T14:14:05.586525+0000","last_fullsized":"2023-10-30T14:14:05.586525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:16.443567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756348+0000","last_change":"2023-10-30T14:14:04.647842+0000","last_active":"2023-10-30T14:14:05.756348+0000","last_peered":"2023-10-30T14:14:05.756348+0000","last_clean":"2023-10-30T14:14:05.756348+0000","last_became_active":"2023-10-30T14:14:04.644683+0000","last_became_peered":"2023-10-30T14:14:04.644683+0000","last_unstale":"2023-10-30T14:14:05.756348+0000","last_undegraded":"2023-10-30T14:14:05.756348+0000","last_fullsized":"2023-10-30T14:14:05.756348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:51:01.230722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585902+0000","last_change":"2023-10-30T14:14:04.620587+0000","last_active":"2023-10-30T14:14:05.585902+0000","last_peered":"2023-10-30T14:14:05.585902+0000","last_clean":"2023-10-30T14:14:05.585902+0000","last_became_active":"2023-10-30T14:14:04.619335+0000","last_became_peered":"2023-10-30T14:14:04.619335+0000","last_unstale":"2023-10-30T14:14:05.585902+0000","last_undegraded":"2023-10-30T14:14:05.585902+0000","last_fullsized":"2023-10-30T14:14:05.585902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:01.735027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755914+0000","last_change":"2023-10-30T14:14:04.641688+0000","last_active":"2023-10-30T14:14:05.755914+0000","last_peered":"2023-10-30T14:14:05.755914+0000","last_clean":"2023-10-30T14:14:05.755914+0000","last_became_active":"2023-10-30T14:14:04.641323+0000","last_became_peered":"2023-10-30T14:14:04.641323+0000","last_unstale":"2023-10-30T14:14:05.755914+0000","last_undegraded":"2023-10-30T14:14:05.755914+0000","last_fullsized":"2023-10-30T14:14:05.755914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:43:35.285019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805751+0000","last_change":"2023-10-30T14:14:04.618530+0000","last_active":"2023-10-30T14:14:05.805751+0000","last_peered":"2023-10-30T14:14:05.805751+0000","last_clean":"2023-10-30T14:14:05.805751+0000","last_became_active":"2023-10-30T14:14:04.618312+0000","last_became_peered":"2023-10-30T14:14:04.618312+0000","last_unstale":"2023-10-30T14:14:05.805751+0000","last_undegraded":"2023-10-30T14:14:05.805751+0000","last_fullsized":"2023-10-30T14:14:05.805751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:25.089106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757146+0000","last_change":"2023-10-30T14:14:04.650330+0000","last_active":"2023-10-30T14:14:05.757146+0000","last_peered":"2023-10-30T14:14:05.757146+0000","last_clean":"2023-10-30T14:14:05.757146+0000","last_became_active":"2023-10-30T14:14:04.647361+0000","last_became_peered":"2023-10-30T14:14:04.647361+0000","last_unstale":"2023-10-30T14:14:05.757146+0000","last_undegraded":"2023-10-30T14:14:05.757146+0000","last_fullsized":"2023-10-30T14:14:05.757146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:49:57.370980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808090+0000","last_change":"2023-10-30T14:14:04.631540+0000","last_active":"2023-10-30T14:14:05.808090+0000","last_peered":"2023-10-30T14:14:05.808090+0000","last_clean":"2023-10-30T14:14:05.808090+0000","last_became_active":"2023-10-30T14:14:04.630959+0000","last_became_peered":"2023-10-30T14:14:04.630959+0000","last_unstale":"2023-10-30T14:14:05.808090+0000","last_undegraded":"2023-10-30T14:14:05.808090+0000","last_fullsized":"2023-10-30T14:14:05.808090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:49.468429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807762+0000","last_change":"2023-10-30T14:14:04.613800+0000","last_active":"2023-10-30T14:14:05.807762+0000","last_peered":"2023-10-30T14:14:05.807762+0000","last_clean":"2023-10-30T14:14:05.807762+0000","last_became_active":"2023-10-30T14:14:04.612653+0000","last_became_peered":"2023-10-30T14:14:04.612653+0000","last_unstale":"2023-10-30T14:14:05.807762+0000","last_undegraded":"2023-10-30T14:14:05.807762+0000","last_fullsized":"2023-10-30T14:14:05.807762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:44.665588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809700+0000","last_change":"2023-10-30T14:14:04.622483+0000","last_active":"2023-10-30T14:14:05.809700+0000","last_peered":"2023-10-30T14:14:05.809700+0000","last_clean":"2023-10-30T14:14:05.809700+0000","last_became_active":"2023-10-30T14:14:04.621135+0000","last_became_peered":"2023-10-30T14:14:04.621135+0000","last_unstale":"2023-10-30T14:14:05.809700+0000","last_undegraded":"2023-10-30T14:14:05.809700+0000","last_fullsized":"2023-10-30T14:14:05.809700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:52.299412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806054+0000","last_change":"2023-10-30T14:14:04.616357+0000","last_active":"2023-10-30T14:14:05.806054+0000","last_peered":"2023-10-30T14:14:05.806054+0000","last_clean":"2023-10-30T14:14:05.806054+0000","last_became_active":"2023-10-30T14:14:04.616142+0000","last_became_peered":"2023-10-30T14:14:04.616142+0000","last_unstale":"2023-10-30T14:14:05.806054+0000","last_undegraded":"2023-10-30T14:14:05.806054+0000","last_fullsized":"2023-10-30T14:14:05.806054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:32:26.432551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586106+0000","last_change":"2023-10-30T14:14:04.588759+0000","last_active":"2023-10-30T14:14:05.586106+0000","last_peered":"2023-10-30T14:14:05.586106+0000","last_clean":"2023-10-30T14:14:05.586106+0000","last_became_active":"2023-10-30T14:14:04.588452+0000","last_became_peered":"2023-10-30T14:14:04.588452+0000","last_unstale":"2023-10-30T14:14:05.586106+0000","last_undegraded":"2023-10-30T14:14:05.586106+0000","last_fullsized":"2023-10-30T14:14:05.586106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:38:21.206284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584572+0000","last_change":"2023-10-30T14:14:04.621073+0000","last_active":"2023-10-30T14:14:05.584572+0000","last_peered":"2023-10-30T14:14:05.584572+0000","last_clean":"2023-10-30T14:14:05.584572+0000","last_became_active":"2023-10-30T14:14:04.620781+0000","last_became_peered":"2023-10-30T14:14:04.620781+0000","last_unstale":"2023-10-30T14:14:05.584572+0000","last_undegraded":"2023-10-30T14:14:05.584572+0000","last_fullsized":"2023-10-30T14:14:05.584572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:17:10.938195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585712+0000","last_change":"2023-10-30T14:14:04.608942+0000","last_active":"2023-10-30T14:14:05.585712+0000","last_peered":"2023-10-30T14:14:05.585712+0000","last_clean":"2023-10-30T14:14:05.585712+0000","last_became_active":"2023-10-30T14:14:04.608190+0000","last_became_peered":"2023-10-30T14:14:04.608190+0000","last_unstale":"2023-10-30T14:14:05.585712+0000","last_undegraded":"2023-10-30T14:14:05.585712+0000","last_fullsized":"2023-10-30T14:14:05.585712+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:18:39.846841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587056+0000","last_change":"2023-10-30T14:14:04.641876+0000","last_active":"2023-10-30T14:14:05.587056+0000","last_peered":"2023-10-30T14:14:05.587056+0000","last_clean":"2023-10-30T14:14:05.587056+0000","last_became_active":"2023-10-30T14:14:04.641306+0000","last_became_peered":"2023-10-30T14:14:04.641306+0000","last_unstale":"2023-10-30T14:14:05.587056+0000","last_undegraded":"2023-10-30T14:14:05.587056+0000","last_fullsized":"2023-10-30T14:14:05.587056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:10:08.169786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805744+0000","last_change":"2023-10-30T14:14:04.613999+0000","last_active":"2023-10-30T14:14:05.805744+0000","last_peered":"2023-10-30T14:14:05.805744+0000","last_clean":"2023-10-30T14:14:05.805744+0000","last_became_active":"2023-10-30T14:14:04.613644+0000","last_became_peered":"2023-10-30T14:14:04.613644+0000","last_unstale":"2023-10-30T14:14:05.805744+0000","last_undegraded":"2023-10-30T14:14:05.805744+0000","last_fullsized":"2023-10-30T14:14:05.805744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:14:50.232028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756929+0000","last_change":"2023-10-30T14:14:04.650393+0000","last_active":"2023-10-30T14:14:05.756929+0000","last_peered":"2023-10-30T14:14:05.756929+0000","last_clean":"2023-10-30T14:14:05.756929+0000","last_became_active":"2023-10-30T14:14:04.649850+0000","last_became_peered":"2023-10-30T14:14:04.649850+0000","last_unstale":"2023-10-30T14:14:05.756929+0000","last_undegraded":"2023-10-30T14:14:05.756929+0000","last_fullsized":"2023-10-30T14:14:05.756929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:25.174023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808107+0000","last_change":"2023-10-30T14:14:04.625744+0000","last_active":"2023-10-30T14:14:05.808107+0000","last_peered":"2023-10-30T14:14:05.808107+0000","last_clean":"2023-10-30T14:14:05.808107+0000","last_became_active":"2023-10-30T14:14:04.625042+0000","last_became_peered":"2023-10-30T14:14:04.625042+0000","last_unstale":"2023-10-30T14:14:05.808107+0000","last_undegraded":"2023-10-30T14:14:05.808107+0000","last_fullsized":"2023-10-30T14:14:05.808107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:30:36.063455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586236+0000","last_change":"2023-10-30T14:14:04.633942+0000","last_active":"2023-10-30T14:14:05.586236+0000","last_peered":"2023-10-30T14:14:05.586236+0000","last_clean":"2023-10-30T14:14:05.586236+0000","last_became_active":"2023-10-30T14:14:04.633586+0000","last_became_peered":"2023-10-30T14:14:04.633586+0000","last_unstale":"2023-10-30T14:14:05.586236+0000","last_undegraded":"2023-10-30T14:14:05.586236+0000","last_fullsized":"2023-10-30T14:14:05.586236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:42:58.067892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586050+0000","last_change":"2023-10-30T14:14:04.642847+0000","last_active":"2023-10-30T14:14:05.586050+0000","last_peered":"2023-10-30T14:14:05.586050+0000","last_clean":"2023-10-30T14:14:05.586050+0000","last_became_active":"2023-10-30T14:14:04.642244+0000","last_became_peered":"2023-10-30T14:14:04.642244+0000","last_unstale":"2023-10-30T14:14:05.586050+0000","last_undegraded":"2023-10-30T14:14:05.586050+0000","last_fullsized":"2023-10-30T14:14:05.586050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:28.780866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805762+0000","last_change":"2023-10-30T14:14:04.620651+0000","last_active":"2023-10-30T14:14:05.805762+0000","last_peered":"2023-10-30T14:14:05.805762+0000","last_clean":"2023-10-30T14:14:05.805762+0000","last_became_active":"2023-10-30T14:14:04.620343+0000","last_became_peered":"2023-10-30T14:14:04.620343+0000","last_unstale":"2023-10-30T14:14:05.805762+0000","last_undegraded":"2023-10-30T14:14:05.805762+0000","last_fullsized":"2023-10-30T14:14:05.805762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:16:00.557118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587224+0000","last_change":"2023-10-30T14:14:04.610472+0000","last_active":"2023-10-30T14:14:05.587224+0000","last_peered":"2023-10-30T14:14:05.587224+0000","last_clean":"2023-10-30T14:14:05.587224+0000","last_became_active":"2023-10-30T14:14:04.609720+0000","last_became_peered":"2023-10-30T14:14:04.609720+0000","last_unstale":"2023-10-30T14:14:05.587224+0000","last_undegraded":"2023-10-30T14:14:05.587224+0000","last_fullsized":"2023-10-30T14:14:05.587224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:44:28.391589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805906+0000","last_change":"2023-10-30T14:14:04.620548+0000","last_active":"2023-10-30T14:14:05.805906+0000","last_peered":"2023-10-30T14:14:05.805906+0000","last_clean":"2023-10-30T14:14:05.805906+0000","last_became_active":"2023-10-30T14:14:04.620217+0000","last_became_peered":"2023-10-30T14:14:04.620217+0000","last_unstale":"2023-10-30T14:14:05.805906+0000","last_undegraded":"2023-10-30T14:14:05.805906+0000","last_fullsized":"2023-10-30T14:14:05.805906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:40:22.912696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808191+0000","last_change":"2023-10-30T14:14:04.644315+0000","last_active":"2023-10-30T14:14:05.808191+0000","last_peered":"2023-10-30T14:14:05.808191+0000","last_clean":"2023-10-30T14:14:05.808191+0000","last_became_active":"2023-10-30T14:14:04.643707+0000","last_became_peered":"2023-10-30T14:14:04.643707+0000","last_unstale":"2023-10-30T14:14:05.808191+0000","last_undegraded":"2023-10-30T14:14:05.808191+0000","last_fullsized":"2023-10-30T14:14:05.808191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:20:33.544167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584874+0000","last_change":"2023-10-30T14:14:04.597855+0000","last_active":"2023-10-30T14:14:05.584874+0000","last_peered":"2023-10-30T14:14:05.584874+0000","last_clean":"2023-10-30T14:14:05.584874+0000","last_became_active":"2023-10-30T14:14:04.597590+0000","last_became_peered":"2023-10-30T14:14:04.597590+0000","last_unstale":"2023-10-30T14:14:05.584874+0000","last_undegraded":"2023-10-30T14:14:05.584874+0000","last_fullsized":"2023-10-30T14:14:05.584874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:05.905027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806303+0000","last_change":"2023-10-30T14:14:04.624133+0000","last_active":"2023-10-30T14:14:05.806303+0000","last_peered":"2023-10-30T14:14:05.806303+0000","last_clean":"2023-10-30T14:14:05.806303+0000","last_became_active":"2023-10-30T14:14:04.623902+0000","last_became_peered":"2023-10-30T14:14:04.623902+0000","last_unstale":"2023-10-30T14:14:05.806303+0000","last_undegraded":"2023-10-30T14:14:05.806303+0000","last_fullsized":"2023-10-30T14:14:05.806303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:33:10.476461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586328+0000","last_change":"2023-10-30T14:14:04.597875+0000","last_active":"2023-10-30T14:14:05.586328+0000","last_peered":"2023-10-30T14:14:05.586328+0000","last_clean":"2023-10-30T14:14:05.586328+0000","last_became_active":"2023-10-30T14:14:04.597556+0000","last_became_peered":"2023-10-30T14:14:04.597556+0000","last_unstale":"2023-10-30T14:14:05.586328+0000","last_undegraded":"2023-10-30T14:14:05.586328+0000","last_fullsized":"2023-10-30T14:14:05.586328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:08:46.407610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586842+0000","last_change":"2023-10-30T14:14:04.620110+0000","last_active":"2023-10-30T14:14:05.586842+0000","last_peered":"2023-10-30T14:14:05.586842+0000","last_clean":"2023-10-30T14:14:05.586842+0000","last_became_active":"2023-10-30T14:14:04.618430+0000","last_became_peered":"2023-10-30T14:14:04.618430+0000","last_unstale":"2023-10-30T14:14:05.586842+0000","last_undegraded":"2023-10-30T14:14:05.586842+0000","last_fullsized":"2023-10-30T14:14:05.586842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:36.966564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805526+0000","last_change":"2023-10-30T14:14:04.606692+0000","last_active":"2023-10-30T14:14:05.805526+0000","last_peered":"2023-10-30T14:14:05.805526+0000","last_clean":"2023-10-30T14:14:05.805526+0000","last_became_active":"2023-10-30T14:14:04.606340+0000","last_became_peered":"2023-10-30T14:14:04.606340+0000","last_unstale":"2023-10-30T14:14:05.805526+0000","last_undegraded":"2023-10-30T14:14:05.805526+0000","last_fullsized":"2023-10-30T14:14:05.805526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:02:24.518205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807314+0000","last_change":"2023-10-30T14:14:04.613535+0000","last_active":"2023-10-30T14:14:05.807314+0000","last_peered":"2023-10-30T14:14:05.807314+0000","last_clean":"2023-10-30T14:14:05.807314+0000","last_became_active":"2023-10-30T14:14:04.612588+0000","last_became_peered":"2023-10-30T14:14:04.612588+0000","last_unstale":"2023-10-30T14:14:05.807314+0000","last_undegraded":"2023-10-30T14:14:05.807314+0000","last_fullsized":"2023-10-30T14:14:05.807314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:58:23.057988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809168+0000","last_change":"2023-10-30T14:14:04.608434+0000","last_active":"2023-10-30T14:14:05.809168+0000","last_peered":"2023-10-30T14:14:05.809168+0000","last_clean":"2023-10-30T14:14:05.809168+0000","last_became_active":"2023-10-30T14:14:04.607683+0000","last_became_peered":"2023-10-30T14:14:04.607683+0000","last_unstale":"2023-10-30T14:14:05.809168+0000","last_undegraded":"2023-10-30T14:14:05.809168+0000","last_fullsized":"2023-10-30T14:14:05.809168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:17:27.467330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.795624+0000","last_change":"2023-10-30T14:14:03.732084+0000","last_active":"2023-10-30T14:14:14.795624+0000","last_peered":"2023-10-30T14:14:14.795624+0000","last_clean":"2023-10-30T14:14:14.795624+0000","last_became_active":"2023-10-30T14:14:03.731740+0000","last_became_peered":"2023-10-30T14:14:03.731740+0000","last_unstale":"2023-10-30T14:14:14.795624+0000","last_undegraded":"2023-10-30T14:14:14.795624+0000","last_fullsized":"2023-10-30T14:14:14.795624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:34:27.462477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586305+0000","last_change":"2023-10-30T14:14:04.621869+0000","last_active":"2023-10-30T14:14:05.586305+0000","last_peered":"2023-10-30T14:14:05.586305+0000","last_clean":"2023-10-30T14:14:05.586305+0000","last_became_active":"2023-10-30T14:14:04.621526+0000","last_became_peered":"2023-10-30T14:14:04.621526+0000","last_unstale":"2023-10-30T14:14:05.586305+0000","last_undegraded":"2023-10-30T14:14:05.586305+0000","last_fullsized":"2023-10-30T14:14:05.586305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:08:21.260692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.820804+0000","last_change":"2023-10-30T14:14:03.730789+0000","last_active":"2023-10-30T14:14:20.820804+0000","last_peered":"2023-10-30T14:14:20.820804+0000","last_clean":"2023-10-30T14:14:20.820804+0000","last_became_active":"2023-10-30T14:14:03.729386+0000","last_became_peered":"2023-10-30T14:14:03.729386+0000","last_unstale":"2023-10-30T14:14:20.820804+0000","last_undegraded":"2023-10-30T14:14:20.820804+0000","last_fullsized":"2023-10-30T14:14:20.820804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.822932+0000","last_change":"2023-10-30T14:14:03.724379+0000","last_active":"2023-10-30T14:14:20.822932+0000","last_peered":"2023-10-30T14:14:20.822932+0000","last_clean":"2023-10-30T14:14:20.822932+0000","last_became_active":"2023-10-30T14:14:03.724078+0000","last_became_peered":"2023-10-30T14:14:03.724078+0000","last_unstale":"2023-10-30T14:14:20.822932+0000","last_undegraded":"2023-10-30T14:14:20.822932+0000","last_fullsized":"2023-10-30T14:14:20.822932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:10:17.829899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805886+0000","last_change":"2023-10-30T14:14:04.597423+0000","last_active":"2023-10-30T14:14:05.805886+0000","last_peered":"2023-10-30T14:14:05.805886+0000","last_clean":"2023-10-30T14:14:05.805886+0000","last_became_active":"2023-10-30T14:14:04.597099+0000","last_became_peered":"2023-10-30T14:14:04.597099+0000","last_unstale":"2023-10-30T14:14:05.805886+0000","last_undegraded":"2023-10-30T14:14:05.805886+0000","last_fullsized":"2023-10-30T14:14:05.805886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:54:52.346295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.965041+0000","last_change":"2023-10-30T14:14:03.721030+0000","last_active":"2023-10-30T14:14:06.965041+0000","last_peered":"2023-10-30T14:14:06.965041+0000","last_clean":"2023-10-30T14:14:06.965041+0000","last_became_active":"2023-10-30T14:14:03.719816+0000","last_became_peered":"2023-10-30T14:14:03.719816+0000","last_unstale":"2023-10-30T14:14:06.965041+0000","last_undegraded":"2023-10-30T14:14:06.965041+0000","last_fullsized":"2023-10-30T14:14:06.965041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:43:50.150156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584729+0000","last_change":"2023-10-30T14:14:04.599863+0000","last_active":"2023-10-30T14:14:05.584729+0000","last_peered":"2023-10-30T14:14:05.584729+0000","last_clean":"2023-10-30T14:14:05.584729+0000","last_became_active":"2023-10-30T14:14:04.599323+0000","last_became_peered":"2023-10-30T14:14:04.599323+0000","last_unstale":"2023-10-30T14:14:05.584729+0000","last_undegraded":"2023-10-30T14:14:05.584729+0000","last_fullsized":"2023-10-30T14:14:05.584729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:42:36.939959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.773337+0000","last_change":"2023-10-30T14:14:03.738694+0000","last_active":"2023-10-30T14:14:14.773337+0000","last_peered":"2023-10-30T14:14:14.773337+0000","last_clean":"2023-10-30T14:14:14.773337+0000","last_became_active":"2023-10-30T14:14:03.738132+0000","last_became_peered":"2023-10-30T14:14:03.738132+0000","last_unstale":"2023-10-30T14:14:14.773337+0000","last_undegraded":"2023-10-30T14:14:14.773337+0000","last_fullsized":"2023-10-30T14:14:14.773337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:46:59.692900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586432+0000","last_change":"2023-10-30T14:14:04.649502+0000","last_active":"2023-10-30T14:14:05.586432+0000","last_peered":"2023-10-30T14:14:05.586432+0000","last_clean":"2023-10-30T14:14:05.586432+0000","last_became_active":"2023-10-30T14:14:04.648816+0000","last_became_peered":"2023-10-30T14:14:04.648816+0000","last_unstale":"2023-10-30T14:14:05.586432+0000","last_undegraded":"2023-10-30T14:14:05.586432+0000","last_fullsized":"2023-10-30T14:14:05.586432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:23:07.149253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.769420+0000","last_change":"2023-10-30T14:14:03.726112+0000","last_active":"2023-10-30T14:14:14.769420+0000","last_peered":"2023-10-30T14:14:14.769420+0000","last_clean":"2023-10-30T14:14:14.769420+0000","last_became_active":"2023-10-30T14:14:03.725272+0000","last_became_peered":"2023-10-30T14:14:03.725272+0000","last_unstale":"2023-10-30T14:14:14.769420+0000","last_undegraded":"2023-10-30T14:14:14.769420+0000","last_fullsized":"2023-10-30T14:14:14.769420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:37:20.764294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586677+0000","last_change":"2023-10-30T14:14:04.610043+0000","last_active":"2023-10-30T14:14:05.586677+0000","last_peered":"2023-10-30T14:14:05.586677+0000","last_clean":"2023-10-30T14:14:05.586677+0000","last_became_active":"2023-10-30T14:14:04.609697+0000","last_became_peered":"2023-10-30T14:14:04.609697+0000","last_unstale":"2023-10-30T14:14:05.586677+0000","last_undegraded":"2023-10-30T14:14:05.586677+0000","last_fullsized":"2023-10-30T14:14:05.586677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:28.543547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.721037+0000","last_change":"2023-10-30T14:14:03.727871+0000","last_active":"2023-10-30T14:14:09.721037+0000","last_peered":"2023-10-30T14:14:09.721037+0000","last_clean":"2023-10-30T14:14:09.721037+0000","last_became_active":"2023-10-30T14:14:03.727422+0000","last_became_peered":"2023-10-30T14:14:03.727422+0000","last_unstale":"2023-10-30T14:14:09.721037+0000","last_undegraded":"2023-10-30T14:14:09.721037+0000","last_fullsized":"2023-10-30T14:14:09.721037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:54.782533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809764+0000","last_change":"2023-10-30T14:14:04.645734+0000","last_active":"2023-10-30T14:14:05.809764+0000","last_peered":"2023-10-30T14:14:05.809764+0000","last_clean":"2023-10-30T14:14:05.809764+0000","last_became_active":"2023-10-30T14:14:04.645333+0000","last_became_peered":"2023-10-30T14:14:04.645333+0000","last_unstale":"2023-10-30T14:14:05.809764+0000","last_undegraded":"2023-10-30T14:14:05.809764+0000","last_fullsized":"2023-10-30T14:14:05.809764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:36:11.322798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'11","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:28.884184+0000","last_change":"2023-10-30T14:14:03.725554+0000","last_active":"2023-10-30T14:14:28.884184+0000","last_peered":"2023-10-30T14:14:28.884184+0000","last_clean":"2023-10-30T14:14:28.884184+0000","last_became_active":"2023-10-30T14:14:03.724806+0000","last_became_peered":"2023-10-30T14:14:03.724806+0000","last_unstale":"2023-10-30T14:14:28.884184+0000","last_undegraded":"2023-10-30T14:14:28.884184+0000","last_fullsized":"2023-10-30T14:14:28.884184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:37.823561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2685,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756505+0000","last_change":"2023-10-30T14:14:04.647122+0000","last_active":"2023-10-30T14:14:05.756505+0000","last_peered":"2023-10-30T14:14:05.756505+0000","last_clean":"2023-10-30T14:14:05.756505+0000","last_became_active":"2023-10-30T14:14:04.646543+0000","last_became_peered":"2023-10-30T14:14:04.646543+0000","last_unstale":"2023-10-30T14:14:05.756505+0000","last_undegraded":"2023-10-30T14:14:05.756505+0000","last_fullsized":"2023-10-30T14:14:05.756505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:16.158442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.822260+0000","last_change":"2023-10-30T14:14:03.717903+0000","last_active":"2023-10-30T14:14:20.822260+0000","last_peered":"2023-10-30T14:14:20.822260+0000","last_clean":"2023-10-30T14:14:20.822260+0000","last_became_active":"2023-10-30T14:14:03.716941+0000","last_became_peered":"2023-10-30T14:14:03.716941+0000","last_unstale":"2023-10-30T14:14:20.822260+0000","last_undegraded":"2023-10-30T14:14:20.822260+0000","last_fullsized":"2023-10-30T14:14:20.822260+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:58.178166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586173+0000","last_change":"2023-10-30T14:14:04.641757+0000","last_active":"2023-10-30T14:14:05.586173+0000","last_peered":"2023-10-30T14:14:05.586173+0000","last_clean":"2023-10-30T14:14:05.586173+0000","last_became_active":"2023-10-30T14:14:04.641494+0000","last_became_peered":"2023-10-30T14:14:04.641494+0000","last_unstale":"2023-10-30T14:14:05.586173+0000","last_undegraded":"2023-10-30T14:14:05.586173+0000","last_fullsized":"2023-10-30T14:14:05.586173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:00:18.795083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.773009+0000","last_change":"2023-10-30T14:14:03.729626+0000","last_active":"2023-10-30T14:14:14.773009+0000","last_peered":"2023-10-30T14:14:14.773009+0000","last_clean":"2023-10-30T14:14:14.773009+0000","last_became_active":"2023-10-30T14:14:03.728053+0000","last_became_peered":"2023-10-30T14:14:03.728053+0000","last_unstale":"2023-10-30T14:14:14.773009+0000","last_undegraded":"2023-10-30T14:14:14.773009+0000","last_fullsized":"2023-10-30T14:14:14.773009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:44:53.362066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756773+0000","last_change":"2023-10-30T14:14:04.650513+0000","last_active":"2023-10-30T14:14:05.756773+0000","last_peered":"2023-10-30T14:14:05.756773+0000","last_clean":"2023-10-30T14:14:05.756773+0000","last_became_active":"2023-10-30T14:14:04.649896+0000","last_became_peered":"2023-10-30T14:14:04.649896+0000","last_unstale":"2023-10-30T14:14:05.756773+0000","last_undegraded":"2023-10-30T14:14:05.756773+0000","last_fullsized":"2023-10-30T14:14:05.756773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:23.100490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.832690+0000","last_change":"2023-10-30T14:14:03.736824+0000","last_active":"2023-10-30T14:14:20.832690+0000","last_peered":"2023-10-30T14:14:20.832690+0000","last_clean":"2023-10-30T14:14:20.832690+0000","last_became_active":"2023-10-30T14:14:03.736155+0000","last_became_peered":"2023-10-30T14:14:03.736155+0000","last_unstale":"2023-10-30T14:14:20.832690+0000","last_undegraded":"2023-10-30T14:14:20.832690+0000","last_fullsized":"2023-10-30T14:14:20.832690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:35.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":846,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807372+0000","last_change":"2023-10-30T14:14:04.615159+0000","last_active":"2023-10-30T14:14:05.807372+0000","last_peered":"2023-10-30T14:14:05.807372+0000","last_clean":"2023-10-30T14:14:05.807372+0000","last_became_active":"2023-10-30T14:14:04.614328+0000","last_became_peered":"2023-10-30T14:14:04.614328+0000","last_unstale":"2023-10-30T14:14:05.807372+0000","last_undegraded":"2023-10-30T14:14:05.807372+0000","last_fullsized":"2023-10-30T14:14:05.807372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:16:52.999786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'15","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.823603+0000","last_change":"2023-10-30T14:14:03.700690+0000","last_active":"2023-10-30T14:14:20.823603+0000","last_peered":"2023-10-30T14:14:20.823603+0000","last_clean":"2023-10-30T14:14:20.823603+0000","last_became_active":"2023-10-30T14:14:03.700352+0000","last_became_peered":"2023-10-30T14:14:03.700352+0000","last_unstale":"2023-10-30T14:14:20.823603+0000","last_undegraded":"2023-10-30T14:14:20.823603+0000","last_fullsized":"2023-10-30T14:14:20.823603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:03:01.103359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755948+0000","last_change":"2023-10-30T14:14:04.647253+0000","last_active":"2023-10-30T14:14:05.755948+0000","last_peered":"2023-10-30T14:14:05.755948+0000","last_clean":"2023-10-30T14:14:05.755948+0000","last_became_active":"2023-10-30T14:14:04.646639+0000","last_became_peered":"2023-10-30T14:14:04.646639+0000","last_unstale":"2023-10-30T14:14:05.755948+0000","last_undegraded":"2023-10-30T14:14:05.755948+0000","last_fullsized":"2023-10-30T14:14:05.755948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:22:50.893734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.722823+0000","last_change":"2023-10-30T14:14:03.710717+0000","last_active":"2023-10-30T14:14:09.722823+0000","last_peered":"2023-10-30T14:14:09.722823+0000","last_clean":"2023-10-30T14:14:09.722823+0000","last_became_active":"2023-10-30T14:14:03.707950+0000","last_became_peered":"2023-10-30T14:14:03.707950+0000","last_unstale":"2023-10-30T14:14:09.722823+0000","last_undegraded":"2023-10-30T14:14:09.722823+0000","last_fullsized":"2023-10-30T14:14:09.722823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:29:41.232902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757066+0000","last_change":"2023-10-30T14:14:04.647688+0000","last_active":"2023-10-30T14:14:05.757066+0000","last_peered":"2023-10-30T14:14:05.757066+0000","last_clean":"2023-10-30T14:14:05.757066+0000","last_became_active":"2023-10-30T14:14:04.644761+0000","last_became_peered":"2023-10-30T14:14:04.644761+0000","last_unstale":"2023-10-30T14:14:05.757066+0000","last_undegraded":"2023-10-30T14:14:05.757066+0000","last_fullsized":"2023-10-30T14:14:05.757066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:00:38.482522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.716374+0000","last_change":"2023-10-30T14:14:03.705567+0000","last_active":"2023-10-30T14:14:09.716374+0000","last_peered":"2023-10-30T14:14:09.716374+0000","last_clean":"2023-10-30T14:14:09.716374+0000","last_became_active":"2023-10-30T14:14:03.705186+0000","last_became_peered":"2023-10-30T14:14:03.705186+0000","last_unstale":"2023-10-30T14:14:09.716374+0000","last_undegraded":"2023-10-30T14:14:09.716374+0000","last_fullsized":"2023-10-30T14:14:09.716374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:14.609243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757235+0000","last_change":"2023-10-30T14:14:04.647335+0000","last_active":"2023-10-30T14:14:05.757235+0000","last_peered":"2023-10-30T14:14:05.757235+0000","last_clean":"2023-10-30T14:14:05.757235+0000","last_became_active":"2023-10-30T14:14:04.646785+0000","last_became_peered":"2023-10-30T14:14:04.646785+0000","last_unstale":"2023-10-30T14:14:05.757235+0000","last_undegraded":"2023-10-30T14:14:05.757235+0000","last_fullsized":"2023-10-30T14:14:05.757235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:58:41.600761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.935812+0000","last_change":"2023-10-30T14:14:03.693349+0000","last_active":"2023-10-30T14:14:06.935812+0000","last_peered":"2023-10-30T14:14:06.935812+0000","last_clean":"2023-10-30T14:14:06.935812+0000","last_became_active":"2023-10-30T14:14:03.692966+0000","last_became_peered":"2023-10-30T14:14:03.692966+0000","last_unstale":"2023-10-30T14:14:06.935812+0000","last_undegraded":"2023-10-30T14:14:06.935812+0000","last_fullsized":"2023-10-30T14:14:06.935812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:30:28.800214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587213+0000","last_change":"2023-10-30T14:14:04.617023+0000","last_active":"2023-10-30T14:14:05.587213+0000","last_peered":"2023-10-30T14:14:05.587213+0000","last_clean":"2023-10-30T14:14:05.587213+0000","last_became_active":"2023-10-30T14:14:04.615925+0000","last_became_peered":"2023-10-30T14:14:04.615925+0000","last_unstale":"2023-10-30T14:14:05.587213+0000","last_undegraded":"2023-10-30T14:14:05.587213+0000","last_fullsized":"2023-10-30T14:14:05.587213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:32:02.289324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.823287+0000","last_change":"2023-10-30T14:14:03.727659+0000","last_active":"2023-10-30T14:14:20.823287+0000","last_peered":"2023-10-30T14:14:20.823287+0000","last_clean":"2023-10-30T14:14:20.823287+0000","last_became_active":"2023-10-30T14:14:03.727409+0000","last_became_peered":"2023-10-30T14:14:03.727409+0000","last_unstale":"2023-10-30T14:14:20.823287+0000","last_undegraded":"2023-10-30T14:14:20.823287+0000","last_fullsized":"2023-10-30T14:14:20.823287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:24.210635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756468+0000","last_change":"2023-10-30T14:14:04.650083+0000","last_active":"2023-10-30T14:14:05.756468+0000","last_peered":"2023-10-30T14:14:05.756468+0000","last_clean":"2023-10-30T14:14:05.756468+0000","last_became_active":"2023-10-30T14:14:04.647562+0000","last_became_peered":"2023-10-30T14:14:04.647562+0000","last_unstale":"2023-10-30T14:14:05.756468+0000","last_undegraded":"2023-10-30T14:14:05.756468+0000","last_fullsized":"2023-10-30T14:14:05.756468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:11:41.743850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T14:14:04.632968+0000","last_change":"2023-10-30T14:13:54.251572+0000","last_active":"2023-10-30T14:14:04.632968+0000","last_peered":"2023-10-30T14:14:04.632968+0000","last_clean":"2023-10-30T14:14:04.632968+0000","last_became_active":"2023-10-30T14:13:54.250907+0000","last_became_peered":"2023-10-30T14:13:54.250907+0000","last_unstale":"2023-10-30T14:14:04.632968+0000","last_undegraded":"2023-10-30T14:14:04.632968+0000","last_fullsized":"2023-10-30T14:14:04.632968+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_clean_scrub_stamp":"2023-10-30T14:13:53.220267+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:44.590852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.779821+0000","last_change":"2023-10-30T14:14:03.729034+0000","last_active":"2023-10-30T14:14:14.779821+0000","last_peered":"2023-10-30T14:14:14.779821+0000","last_clean":"2023-10-30T14:14:14.779821+0000","last_became_active":"2023-10-30T14:14:03.728293+0000","last_became_peered":"2023-10-30T14:14:03.728293+0000","last_unstale":"2023-10-30T14:14:14.779821+0000","last_undegraded":"2023-10-30T14:14:14.779821+0000","last_fullsized":"2023-10-30T14:14:14.779821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806225+0000","last_change":"2023-10-30T14:14:04.630325+0000","last_active":"2023-10-30T14:14:05.806225+0000","last_peered":"2023-10-30T14:14:05.806225+0000","last_clean":"2023-10-30T14:14:05.806225+0000","last_became_active":"2023-10-30T14:14:04.630121+0000","last_became_peered":"2023-10-30T14:14:04.630121+0000","last_unstale":"2023-10-30T14:14:05.806225+0000","last_undegraded":"2023-10-30T14:14:05.806225+0000","last_fullsized":"2023-10-30T14:14:05.806225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:02:28.792879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.822027+0000","last_change":"2023-10-30T14:14:03.732909+0000","last_active":"2023-10-30T14:14:20.822027+0000","last_peered":"2023-10-30T14:14:20.822027+0000","last_clean":"2023-10-30T14:14:20.822027+0000","last_became_active":"2023-10-30T14:14:03.732199+0000","last_became_peered":"2023-10-30T14:14:03.732199+0000","last_unstale":"2023-10-30T14:14:20.822027+0000","last_undegraded":"2023-10-30T14:14:20.822027+0000","last_fullsized":"2023-10-30T14:14:20.822027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:38.117750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586172+0000","last_change":"2023-10-30T14:14:04.620314+0000","last_active":"2023-10-30T14:14:05.586172+0000","last_peered":"2023-10-30T14:14:05.586172+0000","last_clean":"2023-10-30T14:14:05.586172+0000","last_became_active":"2023-10-30T14:14:04.619990+0000","last_became_peered":"2023-10-30T14:14:04.619990+0000","last_unstale":"2023-10-30T14:14:05.586172+0000","last_undegraded":"2023-10-30T14:14:05.586172+0000","last_fullsized":"2023-10-30T14:14:05.586172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:38:25.657586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.822214+0000","last_change":"2023-10-30T14:14:03.702862+0000","last_active":"2023-10-30T14:14:20.822214+0000","last_peered":"2023-10-30T14:14:20.822214+0000","last_clean":"2023-10-30T14:14:20.822214+0000","last_became_active":"2023-10-30T14:14:03.702513+0000","last_became_peered":"2023-10-30T14:14:03.702513+0000","last_unstale":"2023-10-30T14:14:20.822214+0000","last_undegraded":"2023-10-30T14:14:20.822214+0000","last_fullsized":"2023-10-30T14:14:20.822214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:25.499390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584623+0000","last_change":"2023-10-30T14:14:04.589353+0000","last_active":"2023-10-30T14:14:05.584623+0000","last_peered":"2023-10-30T14:14:05.584623+0000","last_clean":"2023-10-30T14:14:05.584623+0000","last_became_active":"2023-10-30T14:14:04.589087+0000","last_became_peered":"2023-10-30T14:14:04.589087+0000","last_unstale":"2023-10-30T14:14:05.584623+0000","last_undegraded":"2023-10-30T14:14:05.584623+0000","last_fullsized":"2023-10-30T14:14:05.584623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:20:12.808641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.965154+0000","last_change":"2023-10-30T14:14:03.730048+0000","last_active":"2023-10-30T14:14:06.965154+0000","last_peered":"2023-10-30T14:14:06.965154+0000","last_clean":"2023-10-30T14:14:06.965154+0000","last_became_active":"2023-10-30T14:14:03.728925+0000","last_became_peered":"2023-10-30T14:14:03.728925+0000","last_unstale":"2023-10-30T14:14:06.965154+0000","last_undegraded":"2023-10-30T14:14:06.965154+0000","last_fullsized":"2023-10-30T14:14:06.965154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:32:05.056039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809437+0000","last_change":"2023-10-30T14:14:04.608289+0000","last_active":"2023-10-30T14:14:05.809437+0000","last_peered":"2023-10-30T14:14:05.809437+0000","last_clean":"2023-10-30T14:14:05.809437+0000","last_became_active":"2023-10-30T14:14:04.607377+0000","last_became_peered":"2023-10-30T14:14:04.607377+0000","last_unstale":"2023-10-30T14:14:05.809437+0000","last_undegraded":"2023-10-30T14:14:05.809437+0000","last_fullsized":"2023-10-30T14:14:05.809437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:43:40.869579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584795+0000","last_change":"2023-10-30T14:14:03.693753+0000","last_active":"2023-10-30T14:14:05.584795+0000","last_peered":"2023-10-30T14:14:05.584795+0000","last_clean":"2023-10-30T14:14:05.584795+0000","last_became_active":"2023-10-30T14:14:03.692621+0000","last_became_peered":"2023-10-30T14:14:03.692621+0000","last_unstale":"2023-10-30T14:14:05.584795+0000","last_undegraded":"2023-10-30T14:14:05.584795+0000","last_fullsized":"2023-10-30T14:14:05.584795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:32:28.577584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587225+0000","last_change":"2023-10-30T14:14:04.614649+0000","last_active":"2023-10-30T14:14:05.587225+0000","last_peered":"2023-10-30T14:14:05.587225+0000","last_clean":"2023-10-30T14:14:05.587225+0000","last_became_active":"2023-10-30T14:14:04.613780+0000","last_became_peered":"2023-10-30T14:14:04.613780+0000","last_unstale":"2023-10-30T14:14:05.587225+0000","last_undegraded":"2023-10-30T14:14:05.587225+0000","last_fullsized":"2023-10-30T14:14:05.587225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:47:40.354022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.937485+0000","last_change":"2023-10-30T14:14:03.740213+0000","last_active":"2023-10-30T14:14:06.937485+0000","last_peered":"2023-10-30T14:14:06.937485+0000","last_clean":"2023-10-30T14:14:06.937485+0000","last_became_active":"2023-10-30T14:14:03.739650+0000","last_became_peered":"2023-10-30T14:14:03.739650+0000","last_unstale":"2023-10-30T14:14:06.937485+0000","last_undegraded":"2023-10-30T14:14:06.937485+0000","last_fullsized":"2023-10-30T14:14:06.937485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:54.120757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584691+0000","last_change":"2023-10-30T14:14:04.590569+0000","last_active":"2023-10-30T14:14:05.584691+0000","last_peered":"2023-10-30T14:14:05.584691+0000","last_clean":"2023-10-30T14:14:05.584691+0000","last_became_active":"2023-10-30T14:14:04.590138+0000","last_became_peered":"2023-10-30T14:14:04.590138+0000","last_unstale":"2023-10-30T14:14:05.584691+0000","last_undegraded":"2023-10-30T14:14:05.584691+0000","last_fullsized":"2023-10-30T14:14:05.584691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:41:30.333668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.824452+0000","last_change":"2023-10-30T14:14:03.721450+0000","last_active":"2023-10-30T14:14:20.824452+0000","last_peered":"2023-10-30T14:14:20.824452+0000","last_clean":"2023-10-30T14:14:20.824452+0000","last_became_active":"2023-10-30T14:14:03.721234+0000","last_became_peered":"2023-10-30T14:14:03.721234+0000","last_unstale":"2023-10-30T14:14:20.824452+0000","last_undegraded":"2023-10-30T14:14:20.824452+0000","last_fullsized":"2023-10-30T14:14:20.824452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:26.959676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806005+0000","last_change":"2023-10-30T14:14:04.634004+0000","last_active":"2023-10-30T14:14:05.806005+0000","last_peered":"2023-10-30T14:14:05.806005+0000","last_clean":"2023-10-30T14:14:05.806005+0000","last_became_active":"2023-10-30T14:14:04.633643+0000","last_became_peered":"2023-10-30T14:14:04.633643+0000","last_unstale":"2023-10-30T14:14:05.806005+0000","last_undegraded":"2023-10-30T14:14:05.806005+0000","last_fullsized":"2023-10-30T14:14:05.806005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:05:14.572477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.769216+0000","last_change":"2023-10-30T14:14:03.738557+0000","last_active":"2023-10-30T14:14:14.769216+0000","last_peered":"2023-10-30T14:14:14.769216+0000","last_clean":"2023-10-30T14:14:14.769216+0000","last_became_active":"2023-10-30T14:14:03.737801+0000","last_became_peered":"2023-10-30T14:14:03.737801+0000","last_unstale":"2023-10-30T14:14:14.769216+0000","last_undegraded":"2023-10-30T14:14:14.769216+0000","last_fullsized":"2023-10-30T14:14:14.769216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:26.938343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586581+0000","last_change":"2023-10-30T14:14:04.589273+0000","last_active":"2023-10-30T14:14:05.586581+0000","last_peered":"2023-10-30T14:14:05.586581+0000","last_clean":"2023-10-30T14:14:05.586581+0000","last_became_active":"2023-10-30T14:14:04.588826+0000","last_became_peered":"2023-10-30T14:14:04.588826+0000","last_unstale":"2023-10-30T14:14:05.586581+0000","last_undegraded":"2023-10-30T14:14:05.586581+0000","last_fullsized":"2023-10-30T14:14:05.586581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:33:52.452394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.787158+0000","last_change":"2023-10-30T14:14:03.733836+0000","last_active":"2023-10-30T14:14:14.787158+0000","last_peered":"2023-10-30T14:14:14.787158+0000","last_clean":"2023-10-30T14:14:14.787158+0000","last_became_active":"2023-10-30T14:14:03.732912+0000","last_became_peered":"2023-10-30T14:14:03.732912+0000","last_unstale":"2023-10-30T14:14:14.787158+0000","last_undegraded":"2023-10-30T14:14:14.787158+0000","last_fullsized":"2023-10-30T14:14:14.787158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:50.332738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809314+0000","last_change":"2023-10-30T14:14:04.610696+0000","last_active":"2023-10-30T14:14:05.809314+0000","last_peered":"2023-10-30T14:14:05.809314+0000","last_clean":"2023-10-30T14:14:05.809314+0000","last_became_active":"2023-10-30T14:14:04.609835+0000","last_became_peered":"2023-10-30T14:14:04.609835+0000","last_unstale":"2023-10-30T14:14:05.809314+0000","last_undegraded":"2023-10-30T14:14:05.809314+0000","last_fullsized":"2023-10-30T14:14:05.809314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:04:31.270262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.795220+0000","last_change":"2023-10-30T14:14:03.732539+0000","last_active":"2023-10-30T14:14:14.795220+0000","last_peered":"2023-10-30T14:14:14.795220+0000","last_clean":"2023-10-30T14:14:14.795220+0000","last_became_active":"2023-10-30T14:14:03.732265+0000","last_became_peered":"2023-10-30T14:14:03.732265+0000","last_unstale":"2023-10-30T14:14:14.795220+0000","last_undegraded":"2023-10-30T14:14:14.795220+0000","last_fullsized":"2023-10-30T14:14:14.795220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:02:00.929612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806211+0000","last_change":"2023-10-30T14:14:04.595357+0000","last_active":"2023-10-30T14:14:05.806211+0000","last_peered":"2023-10-30T14:14:05.806211+0000","last_clean":"2023-10-30T14:14:05.806211+0000","last_became_active":"2023-10-30T14:14:04.594998+0000","last_became_peered":"2023-10-30T14:14:04.594998+0000","last_unstale":"2023-10-30T14:14:05.806211+0000","last_undegraded":"2023-10-30T14:14:05.806211+0000","last_fullsized":"2023-10-30T14:14:05.806211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:44.365673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.935738+0000","last_change":"2023-10-30T14:14:03.703744+0000","last_active":"2023-10-30T14:14:06.935738+0000","last_peered":"2023-10-30T14:14:06.935738+0000","last_clean":"2023-10-30T14:14:06.935738+0000","last_became_active":"2023-10-30T14:14:03.701759+0000","last_became_peered":"2023-10-30T14:14:03.701759+0000","last_unstale":"2023-10-30T14:14:06.935738+0000","last_undegraded":"2023-10-30T14:14:06.935738+0000","last_fullsized":"2023-10-30T14:14:06.935738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:22:27.948007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806186+0000","last_change":"2023-10-30T14:14:04.600759+0000","last_active":"2023-10-30T14:14:05.806186+0000","last_peered":"2023-10-30T14:14:05.806186+0000","last_clean":"2023-10-30T14:14:05.806186+0000","last_became_active":"2023-10-30T14:14:04.600100+0000","last_became_peered":"2023-10-30T14:14:04.600100+0000","last_unstale":"2023-10-30T14:14:05.806186+0000","last_undegraded":"2023-10-30T14:14:05.806186+0000","last_fullsized":"2023-10-30T14:14:05.806186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:02.831207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.773849+0000","last_change":"2023-10-30T14:14:03.741268+0000","last_active":"2023-10-30T14:14:14.773849+0000","last_peered":"2023-10-30T14:14:14.773849+0000","last_clean":"2023-10-30T14:14:14.773849+0000","last_became_active":"2023-10-30T14:14:03.740648+0000","last_became_peered":"2023-10-30T14:14:03.740648+0000","last_unstale":"2023-10-30T14:14:14.773849+0000","last_undegraded":"2023-10-30T14:14:14.773849+0000","last_fullsized":"2023-10-30T14:14:14.773849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:30.799148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807541+0000","last_change":"2023-10-30T14:14:04.623644+0000","last_active":"2023-10-30T14:14:05.807541+0000","last_peered":"2023-10-30T14:14:05.807541+0000","last_clean":"2023-10-30T14:14:05.807541+0000","last_became_active":"2023-10-30T14:14:04.622921+0000","last_became_peered":"2023-10-30T14:14:04.622921+0000","last_unstale":"2023-10-30T14:14:05.807541+0000","last_undegraded":"2023-10-30T14:14:05.807541+0000","last_fullsized":"2023-10-30T14:14:05.807541+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:11:41.810610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587103+0000","last_change":"2023-10-30T14:14:04.624799+0000","last_active":"2023-10-30T14:14:05.587103+0000","last_peered":"2023-10-30T14:14:05.587103+0000","last_clean":"2023-10-30T14:14:05.587103+0000","last_became_active":"2023-10-30T14:14:04.621885+0000","last_became_peered":"2023-10-30T14:14:04.621885+0000","last_unstale":"2023-10-30T14:14:05.587103+0000","last_undegraded":"2023-10-30T14:14:05.587103+0000","last_fullsized":"2023-10-30T14:14:05.587103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:12.055247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.721291+0000","last_change":"2023-10-30T14:14:03.740897+0000","last_active":"2023-10-30T14:14:09.721291+0000","last_peered":"2023-10-30T14:14:09.721291+0000","last_clean":"2023-10-30T14:14:09.721291+0000","last_became_active":"2023-10-30T14:14:03.740247+0000","last_became_peered":"2023-10-30T14:14:03.740247+0000","last_unstale":"2023-10-30T14:14:09.721291+0000","last_undegraded":"2023-10-30T14:14:09.721291+0000","last_fullsized":"2023-10-30T14:14:09.721291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:29.601978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586861+0000","last_change":"2023-10-30T14:14:04.625498+0000","last_active":"2023-10-30T14:14:05.586861+0000","last_peered":"2023-10-30T14:14:05.586861+0000","last_clean":"2023-10-30T14:14:05.586861+0000","last_became_active":"2023-10-30T14:14:04.623638+0000","last_became_peered":"2023-10-30T14:14:04.623638+0000","last_unstale":"2023-10-30T14:14:05.586861+0000","last_undegraded":"2023-10-30T14:14:05.586861+0000","last_fullsized":"2023-10-30T14:14:05.586861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:19.717965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.825113+0000","last_change":"2023-10-30T14:14:03.733233+0000","last_active":"2023-10-30T14:14:20.825113+0000","last_peered":"2023-10-30T14:14:20.825113+0000","last_clean":"2023-10-30T14:14:20.825113+0000","last_became_active":"2023-10-30T14:14:03.732538+0000","last_became_peered":"2023-10-30T14:14:03.732538+0000","last_unstale":"2023-10-30T14:14:20.825113+0000","last_undegraded":"2023-10-30T14:14:20.825113+0000","last_fullsized":"2023-10-30T14:14:20.825113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:25.696173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586619+0000","last_change":"2023-10-30T14:14:04.612104+0000","last_active":"2023-10-30T14:14:05.586619+0000","last_peered":"2023-10-30T14:14:05.586619+0000","last_clean":"2023-10-30T14:14:05.586619+0000","last_became_active":"2023-10-30T14:14:04.611773+0000","last_became_peered":"2023-10-30T14:14:04.611773+0000","last_unstale":"2023-10-30T14:14:05.586619+0000","last_undegraded":"2023-10-30T14:14:05.586619+0000","last_fullsized":"2023-10-30T14:14:05.586619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:44:21.952103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.721542+0000","last_change":"2023-10-30T14:14:03.703988+0000","last_active":"2023-10-30T14:14:09.721542+0000","last_peered":"2023-10-30T14:14:09.721542+0000","last_clean":"2023-10-30T14:14:09.721542+0000","last_became_active":"2023-10-30T14:14:03.703691+0000","last_became_peered":"2023-10-30T14:14:03.703691+0000","last_unstale":"2023-10-30T14:14:09.721542+0000","last_undegraded":"2023-10-30T14:14:09.721542+0000","last_fullsized":"2023-10-30T14:14:09.721542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:34.538541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806119+0000","last_change":"2023-10-30T14:14:04.613738+0000","last_active":"2023-10-30T14:14:05.806119+0000","last_peered":"2023-10-30T14:14:05.806119+0000","last_clean":"2023-10-30T14:14:05.806119+0000","last_became_active":"2023-10-30T14:14:04.613420+0000","last_became_peered":"2023-10-30T14:14:04.613420+0000","last_unstale":"2023-10-30T14:14:05.806119+0000","last_undegraded":"2023-10-30T14:14:05.806119+0000","last_fullsized":"2023-10-30T14:14:05.806119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:01:52.695195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'9","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.825364+0000","last_change":"2023-10-30T14:14:03.737319+0000","last_active":"2023-10-30T14:14:20.825364+0000","last_peered":"2023-10-30T14:14:20.825364+0000","last_clean":"2023-10-30T14:14:20.825364+0000","last_became_active":"2023-10-30T14:14:03.736964+0000","last_became_peered":"2023-10-30T14:14:03.736964+0000","last_unstale":"2023-10-30T14:14:20.825364+0000","last_undegraded":"2023-10-30T14:14:20.825364+0000","last_fullsized":"2023-10-30T14:14:20.825364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:41.469809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6553,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":13106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113,"ondisk_log_size":113,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":831488,"data_stored":667253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2849999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.462}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3580000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":79190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1160000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]}]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":76947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.018}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":76541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.45}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6719999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]}]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":241664,"data_stored":78966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":76387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":76513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574861,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":827392,"data_stored":666597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:14:33.886 INFO:tasks.mds_thrash.ceph_manager:clean! 2023-10-30T14:14:33.887 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:14:34.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.064+0000 7f537289c640 1 Processor -- start 2023-10-30T14:14:34.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.064+0000 7f537289c640 1 -- start start 2023-10-30T14:14:34.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.064+0000 7f537289c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f536c0701a0 0x7f536c15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:34.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.064+0000 7f537289c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f536c06b2f0 0x7f536c06b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:34.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.064+0000 7f537289c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c0c2aa0 con 0x7f536c1537f0 2023-10-30T14:14:34.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.064+0000 7f537289c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f536c0c3ab0 con 0x7f536c06b2f0 2023-10-30T14:14:34.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.064+0000 7f537289c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f536c15a820 con 0x7f536c0701a0 2023-10-30T14:14:34.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.064+0000 7f5370e39640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f536c06b2f0 0x7f536c06b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:34.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.064+0000 7f5370e39640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f536c06b2f0 0x7f536c06b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39766/0 (socket says 172.21.15.57:39766) 2023-10-30T14:14:34.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.064+0000 7f5370e39640 1 -- 172.21.15.57:0/2579977422 learned_addr learned my addr 172.21.15.57:0/2579977422 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:34.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.064+0000 7f536b7fe640 1 --2- 172.21.15.57:0/2579977422 >> v2:172.21.15.88:3300/0 conn(0x7f536c0701a0 0x7f536c15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:34.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.068+0000 7f5370e39640 1 -- 172.21.15.57:0/2579977422 >> v2:172.21.15.88:3300/0 conn(0x7f536c0701a0 msgr2=0x7f536c15a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:34.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.068+0000 7f5370e39640 1 --2- 172.21.15.57:0/2579977422 >> v2:172.21.15.88:3300/0 conn(0x7f536c0701a0 0x7f536c15a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.068+0000 7f5370e39640 1 -- 172.21.15.57:0/2579977422 >> v1:172.21.15.88:6789/0 conn(0x7f536c1537f0 legacy=0x7f536c0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:34.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.068+0000 7f5370e39640 1 -- 172.21.15.57:0/2579977422 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536c0708e0 con 0x7f536c06b2f0 2023-10-30T14:14:34.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.068+0000 7f5370e39640 1 --2- 172.21.15.57:0/2579977422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f536c06b2f0 0x7f536c06b700 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f536c06de40 tx=0x7f536401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74850c7bc411f21 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:34.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.068+0000 7f536affd640 1 -- 172.21.15.57:0/2579977422 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53640af400 con 0x7f536c06b2f0 2023-10-30T14:14:34.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.068+0000 7f536affd640 1 -- 172.21.15.57:0/2579977422 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53640ab050 con 0x7f536c06b2f0 2023-10-30T14:14:34.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.068+0000 7f536affd640 1 -- 172.21.15.57:0/2579977422 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53640af780 con 0x7f536c06b2f0 2023-10-30T14:14:34.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.068+0000 7f537289c640 1 -- 172.21.15.57:0/2579977422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f536c06b2f0 msgr2=0x7f536c06b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:34.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.068+0000 7f537289c640 1 --2- 172.21.15.57:0/2579977422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f536c06b2f0 0x7f536c06b700 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f536c06de40 tx=0x7f536401f1c0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f537289c640 1 -- 172.21.15.57:0/2579977422 shutdown_connections 2023-10-30T14:14:34.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f537289c640 1 --2- 172.21.15.57:0/2579977422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f536c06b2f0 0x7f536c06b700 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f537289c640 1 --2- 172.21.15.57:0/2579977422 >> v2:172.21.15.88:3300/0 conn(0x7f536c0701a0 0x7f536c15a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f537289c640 1 -- 172.21.15.57:0/2579977422 >> 172.21.15.57:0/2579977422 conn(0x7f536c071fb0 msgr2=0x7f536c0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:34.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f537289c640 1 -- 172.21.15.57:0/2579977422 shutdown_connections 2023-10-30T14:14:34.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f537289c640 1 -- 172.21.15.57:0/2579977422 wait complete. 2023-10-30T14:14:34.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f537289c640 1 Processor -- start 2023-10-30T14:14:34.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f537289c640 1 -- start start 2023-10-30T14:14:34.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f537289c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f536c06b2f0 0x7f536c14e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:34.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f537289c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f536c0701a0 0x7f536c0c6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:34.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f537289c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f536c1537f0 0x7f536c0c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:34.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f537289c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c1f4350 con 0x7f536c1537f0 2023-10-30T14:14:34.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f537289c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c0b2c40 con 0x7f536c0701a0 2023-10-30T14:14:34.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f537289c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f536c0c8d40 con 0x7f536c06b2f0 2023-10-30T14:14:34.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f5370e39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f536c1537f0 0x7f536c0c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:34.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f5370e39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f536c1537f0 0x7f536c0c6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39772/0 (socket says 172.21.15.57:39772) 2023-10-30T14:14:34.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f5370e39640 1 -- 172.21.15.57:0/4116691699 learned_addr learned my addr 172.21.15.57:0/4116691699 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:34.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f536b7fe640 1 --2- 172.21.15.57:0/4116691699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f536c0701a0 0x7f536c0c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:34.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f536bfff640 1 --2- 172.21.15.57:0/4116691699 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f536c06b2f0 0x7f536c14e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:34.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f5370e39640 1 -- 172.21.15.57:0/4116691699 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f536c06b2f0 msgr2=0x7f536c14e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f5370e39640 1 --2- 172.21.15.57:0/4116691699 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f536c06b2f0 0x7f536c14e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f5370e39640 1 -- 172.21.15.57:0/4116691699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f536c0701a0 msgr2=0x7f536c0c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f5370e39640 1 --2- 172.21.15.57:0/4116691699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f536c0701a0 0x7f536c0c6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f5370e39640 1 -- 172.21.15.57:0/4116691699 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5364009b30 con 0x7f536c1537f0 2023-10-30T14:14:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f536b7fe640 1 --2- 172.21.15.57:0/4116691699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f536c0701a0 0x7f536c0c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:34.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f536bfff640 1 --2- 172.21.15.57:0/4116691699 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f536c06b2f0 0x7f536c14e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:34.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.072+0000 7f5370e39640 1 --2- 172.21.15.57:0/4116691699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f536c1537f0 0x7f536c0c6ea0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f53640b2040 tx=0x7f536401bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:34.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.076+0000 7f5368ff9640 1 -- 172.21.15.57:0/4116691699 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53640afdc0 con 0x7f536c1537f0 2023-10-30T14:14:34.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.076+0000 7f5368ff9640 1 -- 172.21.15.57:0/4116691699 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5364009510 con 0x7f536c1537f0 2023-10-30T14:14:34.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.076+0000 7f5368ff9640 1 -- 172.21.15.57:0/4116691699 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53640ba680 con 0x7f536c1537f0 2023-10-30T14:14:34.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.076+0000 7f537289c640 1 -- 172.21.15.57:0/4116691699 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536c0b2960 con 0x7f536c1537f0 2023-10-30T14:14:34.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.076+0000 7f537289c640 1 -- 172.21.15.57:0/4116691699 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536c0ca5b0 con 0x7f536c1537f0 2023-10-30T14:14:34.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.076+0000 7f537289c640 1 -- 172.21.15.57:0/4116691699 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f536c04e170 con 0x7f536c1537f0 2023-10-30T14:14:34.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.076+0000 7f5368ff9640 1 -- 172.21.15.57:0/4116691699 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f53640b6040 con 0x7f536c1537f0 2023-10-30T14:14:34.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.076+0000 7f5368ff9640 1 --2- 172.21.15.57:0/4116691699 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f53580664a0 0x7f5358068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:34.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.076+0000 7f5368ff9640 1 -- 172.21.15.57:0/4116691699 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5364122090 con 0x7f536c1537f0 2023-10-30T14:14:34.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.076+0000 7f536bfff640 1 --2- 172.21.15.57:0/4116691699 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f53580664a0 0x7f5358068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:34.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.080+0000 7f536bfff640 1 --2- 172.21.15.57:0/4116691699 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f53580664a0 0x7f5358068960 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f535c000f70 tx=0x7f535c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:34.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.080+0000 7f5368ff9640 1 -- 172.21.15.57:0/4116691699 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f53640ef5c0 con 0x7f536c1537f0 2023-10-30T14:14:34.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.236+0000 7f537289c640 1 -- 172.21.15.57:0/4116691699 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f536c0c1eb0 con 0x7f53580664a0 2023-10-30T14:14:34.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.256+0000 7f5368ff9640 1 -- 172.21.15.57:0/4116691699 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264675 (secure 0 0 0) 0x7f535c0013f0 con 0x7f53580664a0 2023-10-30T14:14:34.262 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:34.264 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-30T14:14:34.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.256+0000 7f537289c640 1 -- 172.21.15.57:0/4116691699 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f53580664a0 msgr2=0x7f5358068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:34.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.256+0000 7f537289c640 1 --2- 172.21.15.57:0/4116691699 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f53580664a0 0x7f5358068960 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f535c000f70 tx=0x7f535c009040 comp rx=0 tx=0).stop 2023-10-30T14:14:34.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.256+0000 7f537289c640 1 -- 172.21.15.57:0/4116691699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f536c1537f0 msgr2=0x7f536c0c6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:34.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.256+0000 7f537289c640 1 --2- 172.21.15.57:0/4116691699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f536c1537f0 0x7f536c0c6ea0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f53640b2040 tx=0x7f536401bf60 comp rx=0 tx=0).stop 2023-10-30T14:14:34.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.256+0000 7f537289c640 1 -- 172.21.15.57:0/4116691699 shutdown_connections 2023-10-30T14:14:34.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.256+0000 7f537289c640 1 --2- 172.21.15.57:0/4116691699 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f53580664a0 0x7f5358068960 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.256+0000 7f537289c640 1 --2- 172.21.15.57:0/4116691699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f536c1537f0 0x7f536c0c6ea0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.256+0000 7f537289c640 1 --2- 172.21.15.57:0/4116691699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f536c0701a0 0x7f536c0c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.256+0000 7f537289c640 1 --2- 172.21.15.57:0/4116691699 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f536c06b2f0 0x7f536c14e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.256+0000 7f537289c640 1 -- 172.21.15.57:0/4116691699 >> 172.21.15.57:0/4116691699 conn(0x7f536c071fb0 msgr2=0x7f536c1591c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:34.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.260+0000 7f537289c640 1 -- 172.21.15.57:0/4116691699 shutdown_connections 2023-10-30T14:14:34.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.260+0000 7f537289c640 1 -- 172.21.15.57:0/4116691699 wait complete. 2023-10-30T14:14:34.283 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-10-30T14:14:32.504236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596921,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":288,"num_write_kb":2458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227464,"kb_used_data":3016,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754747256,"statfs":{"total":773094113280,"available":772861190144,"internally_reserved":0,"allocated":3088384,"data_stored":1798394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":229756473},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2737,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004216"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757043+0000","last_change":"2023-10-30T14:14:04.651413+0000","last_active":"2023-10-30T14:14:05.757043+0000","last_peered":"2023-10-30T14:14:05.757043+0000","last_clean":"2023-10-30T14:14:05.757043+0000","last_became_active":"2023-10-30T14:14:04.650882+0000","last_became_peered":"2023-10-30T14:14:04.650882+0000","last_unstale":"2023-10-30T14:14:05.757043+0000","last_undegraded":"2023-10-30T14:14:05.757043+0000","last_fullsized":"2023-10-30T14:14:05.757043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:30:08.122326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585036+0000","last_change":"2023-10-30T14:14:04.588523+0000","last_active":"2023-10-30T14:14:05.585036+0000","last_peered":"2023-10-30T14:14:05.585036+0000","last_clean":"2023-10-30T14:14:05.585036+0000","last_became_active":"2023-10-30T14:14:04.587942+0000","last_became_peered":"2023-10-30T14:14:04.587942+0000","last_unstale":"2023-10-30T14:14:05.585036+0000","last_undegraded":"2023-10-30T14:14:05.585036+0000","last_fullsized":"2023-10-30T14:14:05.585036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:16.028564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586543+0000","last_change":"2023-10-30T14:14:04.590976+0000","last_active":"2023-10-30T14:14:05.586543+0000","last_peered":"2023-10-30T14:14:05.586543+0000","last_clean":"2023-10-30T14:14:05.586543+0000","last_became_active":"2023-10-30T14:14:04.590700+0000","last_became_peered":"2023-10-30T14:14:04.590700+0000","last_unstale":"2023-10-30T14:14:05.586543+0000","last_undegraded":"2023-10-30T14:14:05.586543+0000","last_fullsized":"2023-10-30T14:14:05.586543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:49.169325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586525+0000","last_change":"2023-10-30T14:14:04.618603+0000","last_active":"2023-10-30T14:14:05.586525+0000","last_peered":"2023-10-30T14:14:05.586525+0000","last_clean":"2023-10-30T14:14:05.586525+0000","last_became_active":"2023-10-30T14:14:04.617768+0000","last_became_peered":"2023-10-30T14:14:04.617768+0000","last_unstale":"2023-10-30T14:14:05.586525+0000","last_undegraded":"2023-10-30T14:14:05.586525+0000","last_fullsized":"2023-10-30T14:14:05.586525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:16.443567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756348+0000","last_change":"2023-10-30T14:14:04.647842+0000","last_active":"2023-10-30T14:14:05.756348+0000","last_peered":"2023-10-30T14:14:05.756348+0000","last_clean":"2023-10-30T14:14:05.756348+0000","last_became_active":"2023-10-30T14:14:04.644683+0000","last_became_peered":"2023-10-30T14:14:04.644683+0000","last_unstale":"2023-10-30T14:14:05.756348+0000","last_undegraded":"2023-10-30T14:14:05.756348+0000","last_fullsized":"2023-10-30T14:14:05.756348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:51:01.230722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585902+0000","last_change":"2023-10-30T14:14:04.620587+0000","last_active":"2023-10-30T14:14:05.585902+0000","last_peered":"2023-10-30T14:14:05.585902+0000","last_clean":"2023-10-30T14:14:05.585902+0000","last_became_active":"2023-10-30T14:14:04.619335+0000","last_became_peered":"2023-10-30T14:14:04.619335+0000","last_unstale":"2023-10-30T14:14:05.585902+0000","last_undegraded":"2023-10-30T14:14:05.585902+0000","last_fullsized":"2023-10-30T14:14:05.585902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:41:01.735027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755914+0000","last_change":"2023-10-30T14:14:04.641688+0000","last_active":"2023-10-30T14:14:05.755914+0000","last_peered":"2023-10-30T14:14:05.755914+0000","last_clean":"2023-10-30T14:14:05.755914+0000","last_became_active":"2023-10-30T14:14:04.641323+0000","last_became_peered":"2023-10-30T14:14:04.641323+0000","last_unstale":"2023-10-30T14:14:05.755914+0000","last_undegraded":"2023-10-30T14:14:05.755914+0000","last_fullsized":"2023-10-30T14:14:05.755914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:43:35.285019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805751+0000","last_change":"2023-10-30T14:14:04.618530+0000","last_active":"2023-10-30T14:14:05.805751+0000","last_peered":"2023-10-30T14:14:05.805751+0000","last_clean":"2023-10-30T14:14:05.805751+0000","last_became_active":"2023-10-30T14:14:04.618312+0000","last_became_peered":"2023-10-30T14:14:04.618312+0000","last_unstale":"2023-10-30T14:14:05.805751+0000","last_undegraded":"2023-10-30T14:14:05.805751+0000","last_fullsized":"2023-10-30T14:14:05.805751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:25.089106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757146+0000","last_change":"2023-10-30T14:14:04.650330+0000","last_active":"2023-10-30T14:14:05.757146+0000","last_peered":"2023-10-30T14:14:05.757146+0000","last_clean":"2023-10-30T14:14:05.757146+0000","last_became_active":"2023-10-30T14:14:04.647361+0000","last_became_peered":"2023-10-30T14:14:04.647361+0000","last_unstale":"2023-10-30T14:14:05.757146+0000","last_undegraded":"2023-10-30T14:14:05.757146+0000","last_fullsized":"2023-10-30T14:14:05.757146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:49:57.370980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808090+0000","last_change":"2023-10-30T14:14:04.631540+0000","last_active":"2023-10-30T14:14:05.808090+0000","last_peered":"2023-10-30T14:14:05.808090+0000","last_clean":"2023-10-30T14:14:05.808090+0000","last_became_active":"2023-10-30T14:14:04.630959+0000","last_became_peered":"2023-10-30T14:14:04.630959+0000","last_unstale":"2023-10-30T14:14:05.808090+0000","last_undegraded":"2023-10-30T14:14:05.808090+0000","last_fullsized":"2023-10-30T14:14:05.808090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:49.468429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807762+0000","last_change":"2023-10-30T14:14:04.613800+0000","last_active":"2023-10-30T14:14:05.807762+0000","last_peered":"2023-10-30T14:14:05.807762+0000","last_clean":"2023-10-30T14:14:05.807762+0000","last_became_active":"2023-10-30T14:14:04.612653+0000","last_became_peered":"2023-10-30T14:14:04.612653+0000","last_unstale":"2023-10-30T14:14:05.807762+0000","last_undegraded":"2023-10-30T14:14:05.807762+0000","last_fullsized":"2023-10-30T14:14:05.807762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:44.665588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809700+0000","last_change":"2023-10-30T14:14:04.622483+0000","last_active":"2023-10-30T14:14:05.809700+0000","last_peered":"2023-10-30T14:14:05.809700+0000","last_clean":"2023-10-30T14:14:05.809700+0000","last_became_active":"2023-10-30T14:14:04.621135+0000","last_became_peered":"2023-10-30T14:14:04.621135+0000","last_unstale":"2023-10-30T14:14:05.809700+0000","last_undegraded":"2023-10-30T14:14:05.809700+0000","last_fullsized":"2023-10-30T14:14:05.809700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:52.299412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806054+0000","last_change":"2023-10-30T14:14:04.616357+0000","last_active":"2023-10-30T14:14:05.806054+0000","last_peered":"2023-10-30T14:14:05.806054+0000","last_clean":"2023-10-30T14:14:05.806054+0000","last_became_active":"2023-10-30T14:14:04.616142+0000","last_became_peered":"2023-10-30T14:14:04.616142+0000","last_unstale":"2023-10-30T14:14:05.806054+0000","last_undegraded":"2023-10-30T14:14:05.806054+0000","last_fullsized":"2023-10-30T14:14:05.806054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:32:26.432551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586106+0000","last_change":"2023-10-30T14:14:04.588759+0000","last_active":"2023-10-30T14:14:05.586106+0000","last_peered":"2023-10-30T14:14:05.586106+0000","last_clean":"2023-10-30T14:14:05.586106+0000","last_became_active":"2023-10-30T14:14:04.588452+0000","last_became_peered":"2023-10-30T14:14:04.588452+0000","last_unstale":"2023-10-30T14:14:05.586106+0000","last_undegraded":"2023-10-30T14:14:05.586106+0000","last_fullsized":"2023-10-30T14:14:05.586106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:38:21.206284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584572+0000","last_change":"2023-10-30T14:14:04.621073+0000","last_active":"2023-10-30T14:14:05.584572+0000","last_peered":"2023-10-30T14:14:05.584572+0000","last_clean":"2023-10-30T14:14:05.584572+0000","last_became_active":"2023-10-30T14:14:04.620781+0000","last_became_peered":"2023-10-30T14:14:04.620781+0000","last_unstale":"2023-10-30T14:14:05.584572+0000","last_undegraded":"2023-10-30T14:14:05.584572+0000","last_fullsized":"2023-10-30T14:14:05.584572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:17:10.938195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.585712+0000","last_change":"2023-10-30T14:14:04.608942+0000","last_active":"2023-10-30T14:14:05.585712+0000","last_peered":"2023-10-30T14:14:05.585712+0000","last_clean":"2023-10-30T14:14:05.585712+0000","last_became_active":"2023-10-30T14:14:04.608190+0000","last_became_peered":"2023-10-30T14:14:04.608190+0000","last_unstale":"2023-10-30T14:14:05.585712+0000","last_undegraded":"2023-10-30T14:14:05.585712+0000","last_fullsized":"2023-10-30T14:14:05.585712+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:18:39.846841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587056+0000","last_change":"2023-10-30T14:14:04.641876+0000","last_active":"2023-10-30T14:14:05.587056+0000","last_peered":"2023-10-30T14:14:05.587056+0000","last_clean":"2023-10-30T14:14:05.587056+0000","last_became_active":"2023-10-30T14:14:04.641306+0000","last_became_peered":"2023-10-30T14:14:04.641306+0000","last_unstale":"2023-10-30T14:14:05.587056+0000","last_undegraded":"2023-10-30T14:14:05.587056+0000","last_fullsized":"2023-10-30T14:14:05.587056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:10:08.169786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805744+0000","last_change":"2023-10-30T14:14:04.613999+0000","last_active":"2023-10-30T14:14:05.805744+0000","last_peered":"2023-10-30T14:14:05.805744+0000","last_clean":"2023-10-30T14:14:05.805744+0000","last_became_active":"2023-10-30T14:14:04.613644+0000","last_became_peered":"2023-10-30T14:14:04.613644+0000","last_unstale":"2023-10-30T14:14:05.805744+0000","last_undegraded":"2023-10-30T14:14:05.805744+0000","last_fullsized":"2023-10-30T14:14:05.805744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:14:50.232028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756929+0000","last_change":"2023-10-30T14:14:04.650393+0000","last_active":"2023-10-30T14:14:05.756929+0000","last_peered":"2023-10-30T14:14:05.756929+0000","last_clean":"2023-10-30T14:14:05.756929+0000","last_became_active":"2023-10-30T14:14:04.649850+0000","last_became_peered":"2023-10-30T14:14:04.649850+0000","last_unstale":"2023-10-30T14:14:05.756929+0000","last_undegraded":"2023-10-30T14:14:05.756929+0000","last_fullsized":"2023-10-30T14:14:05.756929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:25.174023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808107+0000","last_change":"2023-10-30T14:14:04.625744+0000","last_active":"2023-10-30T14:14:05.808107+0000","last_peered":"2023-10-30T14:14:05.808107+0000","last_clean":"2023-10-30T14:14:05.808107+0000","last_became_active":"2023-10-30T14:14:04.625042+0000","last_became_peered":"2023-10-30T14:14:04.625042+0000","last_unstale":"2023-10-30T14:14:05.808107+0000","last_undegraded":"2023-10-30T14:14:05.808107+0000","last_fullsized":"2023-10-30T14:14:05.808107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:30:36.063455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586236+0000","last_change":"2023-10-30T14:14:04.633942+0000","last_active":"2023-10-30T14:14:05.586236+0000","last_peered":"2023-10-30T14:14:05.586236+0000","last_clean":"2023-10-30T14:14:05.586236+0000","last_became_active":"2023-10-30T14:14:04.633586+0000","last_became_peered":"2023-10-30T14:14:04.633586+0000","last_unstale":"2023-10-30T14:14:05.586236+0000","last_undegraded":"2023-10-30T14:14:05.586236+0000","last_fullsized":"2023-10-30T14:14:05.586236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:42:58.067892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586050+0000","last_change":"2023-10-30T14:14:04.642847+0000","last_active":"2023-10-30T14:14:05.586050+0000","last_peered":"2023-10-30T14:14:05.586050+0000","last_clean":"2023-10-30T14:14:05.586050+0000","last_became_active":"2023-10-30T14:14:04.642244+0000","last_became_peered":"2023-10-30T14:14:04.642244+0000","last_unstale":"2023-10-30T14:14:05.586050+0000","last_undegraded":"2023-10-30T14:14:05.586050+0000","last_fullsized":"2023-10-30T14:14:05.586050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:28.780866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805762+0000","last_change":"2023-10-30T14:14:04.620651+0000","last_active":"2023-10-30T14:14:05.805762+0000","last_peered":"2023-10-30T14:14:05.805762+0000","last_clean":"2023-10-30T14:14:05.805762+0000","last_became_active":"2023-10-30T14:14:04.620343+0000","last_became_peered":"2023-10-30T14:14:04.620343+0000","last_unstale":"2023-10-30T14:14:05.805762+0000","last_undegraded":"2023-10-30T14:14:05.805762+0000","last_fullsized":"2023-10-30T14:14:05.805762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:16:00.557118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587224+0000","last_change":"2023-10-30T14:14:04.610472+0000","last_active":"2023-10-30T14:14:05.587224+0000","last_peered":"2023-10-30T14:14:05.587224+0000","last_clean":"2023-10-30T14:14:05.587224+0000","last_became_active":"2023-10-30T14:14:04.609720+0000","last_became_peered":"2023-10-30T14:14:04.609720+0000","last_unstale":"2023-10-30T14:14:05.587224+0000","last_undegraded":"2023-10-30T14:14:05.587224+0000","last_fullsized":"2023-10-30T14:14:05.587224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:44:28.391589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805906+0000","last_change":"2023-10-30T14:14:04.620548+0000","last_active":"2023-10-30T14:14:05.805906+0000","last_peered":"2023-10-30T14:14:05.805906+0000","last_clean":"2023-10-30T14:14:05.805906+0000","last_became_active":"2023-10-30T14:14:04.620217+0000","last_became_peered":"2023-10-30T14:14:04.620217+0000","last_unstale":"2023-10-30T14:14:05.805906+0000","last_undegraded":"2023-10-30T14:14:05.805906+0000","last_fullsized":"2023-10-30T14:14:05.805906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:40:22.912696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.808191+0000","last_change":"2023-10-30T14:14:04.644315+0000","last_active":"2023-10-30T14:14:05.808191+0000","last_peered":"2023-10-30T14:14:05.808191+0000","last_clean":"2023-10-30T14:14:05.808191+0000","last_became_active":"2023-10-30T14:14:04.643707+0000","last_became_peered":"2023-10-30T14:14:04.643707+0000","last_unstale":"2023-10-30T14:14:05.808191+0000","last_undegraded":"2023-10-30T14:14:05.808191+0000","last_fullsized":"2023-10-30T14:14:05.808191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:20:33.544167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584874+0000","last_change":"2023-10-30T14:14:04.597855+0000","last_active":"2023-10-30T14:14:05.584874+0000","last_peered":"2023-10-30T14:14:05.584874+0000","last_clean":"2023-10-30T14:14:05.584874+0000","last_became_active":"2023-10-30T14:14:04.597590+0000","last_became_peered":"2023-10-30T14:14:04.597590+0000","last_unstale":"2023-10-30T14:14:05.584874+0000","last_undegraded":"2023-10-30T14:14:05.584874+0000","last_fullsized":"2023-10-30T14:14:05.584874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:05.905027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806303+0000","last_change":"2023-10-30T14:14:04.624133+0000","last_active":"2023-10-30T14:14:05.806303+0000","last_peered":"2023-10-30T14:14:05.806303+0000","last_clean":"2023-10-30T14:14:05.806303+0000","last_became_active":"2023-10-30T14:14:04.623902+0000","last_became_peered":"2023-10-30T14:14:04.623902+0000","last_unstale":"2023-10-30T14:14:05.806303+0000","last_undegraded":"2023-10-30T14:14:05.806303+0000","last_fullsized":"2023-10-30T14:14:05.806303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:33:10.476461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586328+0000","last_change":"2023-10-30T14:14:04.597875+0000","last_active":"2023-10-30T14:14:05.586328+0000","last_peered":"2023-10-30T14:14:05.586328+0000","last_clean":"2023-10-30T14:14:05.586328+0000","last_became_active":"2023-10-30T14:14:04.597556+0000","last_became_peered":"2023-10-30T14:14:04.597556+0000","last_unstale":"2023-10-30T14:14:05.586328+0000","last_undegraded":"2023-10-30T14:14:05.586328+0000","last_fullsized":"2023-10-30T14:14:05.586328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:08:46.407610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586842+0000","last_change":"2023-10-30T14:14:04.620110+0000","last_active":"2023-10-30T14:14:05.586842+0000","last_peered":"2023-10-30T14:14:05.586842+0000","last_clean":"2023-10-30T14:14:05.586842+0000","last_became_active":"2023-10-30T14:14:04.618430+0000","last_became_peered":"2023-10-30T14:14:04.618430+0000","last_unstale":"2023-10-30T14:14:05.586842+0000","last_undegraded":"2023-10-30T14:14:05.586842+0000","last_fullsized":"2023-10-30T14:14:05.586842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:36.966564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805526+0000","last_change":"2023-10-30T14:14:04.606692+0000","last_active":"2023-10-30T14:14:05.805526+0000","last_peered":"2023-10-30T14:14:05.805526+0000","last_clean":"2023-10-30T14:14:05.805526+0000","last_became_active":"2023-10-30T14:14:04.606340+0000","last_became_peered":"2023-10-30T14:14:04.606340+0000","last_unstale":"2023-10-30T14:14:05.805526+0000","last_undegraded":"2023-10-30T14:14:05.805526+0000","last_fullsized":"2023-10-30T14:14:05.805526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:02:24.518205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807314+0000","last_change":"2023-10-30T14:14:04.613535+0000","last_active":"2023-10-30T14:14:05.807314+0000","last_peered":"2023-10-30T14:14:05.807314+0000","last_clean":"2023-10-30T14:14:05.807314+0000","last_became_active":"2023-10-30T14:14:04.612588+0000","last_became_peered":"2023-10-30T14:14:04.612588+0000","last_unstale":"2023-10-30T14:14:05.807314+0000","last_undegraded":"2023-10-30T14:14:05.807314+0000","last_fullsized":"2023-10-30T14:14:05.807314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:58:23.057988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809168+0000","last_change":"2023-10-30T14:14:04.608434+0000","last_active":"2023-10-30T14:14:05.809168+0000","last_peered":"2023-10-30T14:14:05.809168+0000","last_clean":"2023-10-30T14:14:05.809168+0000","last_became_active":"2023-10-30T14:14:04.607683+0000","last_became_peered":"2023-10-30T14:14:04.607683+0000","last_unstale":"2023-10-30T14:14:05.809168+0000","last_undegraded":"2023-10-30T14:14:05.809168+0000","last_fullsized":"2023-10-30T14:14:05.809168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:17:27.467330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.795624+0000","last_change":"2023-10-30T14:14:03.732084+0000","last_active":"2023-10-30T14:14:14.795624+0000","last_peered":"2023-10-30T14:14:14.795624+0000","last_clean":"2023-10-30T14:14:14.795624+0000","last_became_active":"2023-10-30T14:14:03.731740+0000","last_became_peered":"2023-10-30T14:14:03.731740+0000","last_unstale":"2023-10-30T14:14:14.795624+0000","last_undegraded":"2023-10-30T14:14:14.795624+0000","last_fullsized":"2023-10-30T14:14:14.795624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:34:27.462477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586305+0000","last_change":"2023-10-30T14:14:04.621869+0000","last_active":"2023-10-30T14:14:05.586305+0000","last_peered":"2023-10-30T14:14:05.586305+0000","last_clean":"2023-10-30T14:14:05.586305+0000","last_became_active":"2023-10-30T14:14:04.621526+0000","last_became_peered":"2023-10-30T14:14:04.621526+0000","last_unstale":"2023-10-30T14:14:05.586305+0000","last_undegraded":"2023-10-30T14:14:05.586305+0000","last_fullsized":"2023-10-30T14:14:05.586305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:08:21.260692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.820804+0000","last_change":"2023-10-30T14:14:03.730789+0000","last_active":"2023-10-30T14:14:20.820804+0000","last_peered":"2023-10-30T14:14:20.820804+0000","last_clean":"2023-10-30T14:14:20.820804+0000","last_became_active":"2023-10-30T14:14:03.729386+0000","last_became_peered":"2023-10-30T14:14:03.729386+0000","last_unstale":"2023-10-30T14:14:20.820804+0000","last_undegraded":"2023-10-30T14:14:20.820804+0000","last_fullsized":"2023-10-30T14:14:20.820804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.822932+0000","last_change":"2023-10-30T14:14:03.724379+0000","last_active":"2023-10-30T14:14:20.822932+0000","last_peered":"2023-10-30T14:14:20.822932+0000","last_clean":"2023-10-30T14:14:20.822932+0000","last_became_active":"2023-10-30T14:14:03.724078+0000","last_became_peered":"2023-10-30T14:14:03.724078+0000","last_unstale":"2023-10-30T14:14:20.822932+0000","last_undegraded":"2023-10-30T14:14:20.822932+0000","last_fullsized":"2023-10-30T14:14:20.822932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:10:17.829899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.805886+0000","last_change":"2023-10-30T14:14:04.597423+0000","last_active":"2023-10-30T14:14:05.805886+0000","last_peered":"2023-10-30T14:14:05.805886+0000","last_clean":"2023-10-30T14:14:05.805886+0000","last_became_active":"2023-10-30T14:14:04.597099+0000","last_became_peered":"2023-10-30T14:14:04.597099+0000","last_unstale":"2023-10-30T14:14:05.805886+0000","last_undegraded":"2023-10-30T14:14:05.805886+0000","last_fullsized":"2023-10-30T14:14:05.805886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:54:52.346295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.965041+0000","last_change":"2023-10-30T14:14:03.721030+0000","last_active":"2023-10-30T14:14:06.965041+0000","last_peered":"2023-10-30T14:14:06.965041+0000","last_clean":"2023-10-30T14:14:06.965041+0000","last_became_active":"2023-10-30T14:14:03.719816+0000","last_became_peered":"2023-10-30T14:14:03.719816+0000","last_unstale":"2023-10-30T14:14:06.965041+0000","last_undegraded":"2023-10-30T14:14:06.965041+0000","last_fullsized":"2023-10-30T14:14:06.965041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:43:50.150156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584729+0000","last_change":"2023-10-30T14:14:04.599863+0000","last_active":"2023-10-30T14:14:05.584729+0000","last_peered":"2023-10-30T14:14:05.584729+0000","last_clean":"2023-10-30T14:14:05.584729+0000","last_became_active":"2023-10-30T14:14:04.599323+0000","last_became_peered":"2023-10-30T14:14:04.599323+0000","last_unstale":"2023-10-30T14:14:05.584729+0000","last_undegraded":"2023-10-30T14:14:05.584729+0000","last_fullsized":"2023-10-30T14:14:05.584729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:42:36.939959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.773337+0000","last_change":"2023-10-30T14:14:03.738694+0000","last_active":"2023-10-30T14:14:14.773337+0000","last_peered":"2023-10-30T14:14:14.773337+0000","last_clean":"2023-10-30T14:14:14.773337+0000","last_became_active":"2023-10-30T14:14:03.738132+0000","last_became_peered":"2023-10-30T14:14:03.738132+0000","last_unstale":"2023-10-30T14:14:14.773337+0000","last_undegraded":"2023-10-30T14:14:14.773337+0000","last_fullsized":"2023-10-30T14:14:14.773337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:46:59.692900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586432+0000","last_change":"2023-10-30T14:14:04.649502+0000","last_active":"2023-10-30T14:14:05.586432+0000","last_peered":"2023-10-30T14:14:05.586432+0000","last_clean":"2023-10-30T14:14:05.586432+0000","last_became_active":"2023-10-30T14:14:04.648816+0000","last_became_peered":"2023-10-30T14:14:04.648816+0000","last_unstale":"2023-10-30T14:14:05.586432+0000","last_undegraded":"2023-10-30T14:14:05.586432+0000","last_fullsized":"2023-10-30T14:14:05.586432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:23:07.149253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.769420+0000","last_change":"2023-10-30T14:14:03.726112+0000","last_active":"2023-10-30T14:14:14.769420+0000","last_peered":"2023-10-30T14:14:14.769420+0000","last_clean":"2023-10-30T14:14:14.769420+0000","last_became_active":"2023-10-30T14:14:03.725272+0000","last_became_peered":"2023-10-30T14:14:03.725272+0000","last_unstale":"2023-10-30T14:14:14.769420+0000","last_undegraded":"2023-10-30T14:14:14.769420+0000","last_fullsized":"2023-10-30T14:14:14.769420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:37:20.764294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586677+0000","last_change":"2023-10-30T14:14:04.610043+0000","last_active":"2023-10-30T14:14:05.586677+0000","last_peered":"2023-10-30T14:14:05.586677+0000","last_clean":"2023-10-30T14:14:05.586677+0000","last_became_active":"2023-10-30T14:14:04.609697+0000","last_became_peered":"2023-10-30T14:14:04.609697+0000","last_unstale":"2023-10-30T14:14:05.586677+0000","last_undegraded":"2023-10-30T14:14:05.586677+0000","last_fullsized":"2023-10-30T14:14:05.586677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:28.543547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.721037+0000","last_change":"2023-10-30T14:14:03.727871+0000","last_active":"2023-10-30T14:14:09.721037+0000","last_peered":"2023-10-30T14:14:09.721037+0000","last_clean":"2023-10-30T14:14:09.721037+0000","last_became_active":"2023-10-30T14:14:03.727422+0000","last_became_peered":"2023-10-30T14:14:03.727422+0000","last_unstale":"2023-10-30T14:14:09.721037+0000","last_undegraded":"2023-10-30T14:14:09.721037+0000","last_fullsized":"2023-10-30T14:14:09.721037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:54.782533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809764+0000","last_change":"2023-10-30T14:14:04.645734+0000","last_active":"2023-10-30T14:14:05.809764+0000","last_peered":"2023-10-30T14:14:05.809764+0000","last_clean":"2023-10-30T14:14:05.809764+0000","last_became_active":"2023-10-30T14:14:04.645333+0000","last_became_peered":"2023-10-30T14:14:04.645333+0000","last_unstale":"2023-10-30T14:14:05.809764+0000","last_undegraded":"2023-10-30T14:14:05.809764+0000","last_fullsized":"2023-10-30T14:14:05.809764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:36:11.322798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'11","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:28.884184+0000","last_change":"2023-10-30T14:14:03.725554+0000","last_active":"2023-10-30T14:14:28.884184+0000","last_peered":"2023-10-30T14:14:28.884184+0000","last_clean":"2023-10-30T14:14:28.884184+0000","last_became_active":"2023-10-30T14:14:03.724806+0000","last_became_peered":"2023-10-30T14:14:03.724806+0000","last_unstale":"2023-10-30T14:14:28.884184+0000","last_undegraded":"2023-10-30T14:14:28.884184+0000","last_fullsized":"2023-10-30T14:14:28.884184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:37.823561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2685,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756505+0000","last_change":"2023-10-30T14:14:04.647122+0000","last_active":"2023-10-30T14:14:05.756505+0000","last_peered":"2023-10-30T14:14:05.756505+0000","last_clean":"2023-10-30T14:14:05.756505+0000","last_became_active":"2023-10-30T14:14:04.646543+0000","last_became_peered":"2023-10-30T14:14:04.646543+0000","last_unstale":"2023-10-30T14:14:05.756505+0000","last_undegraded":"2023-10-30T14:14:05.756505+0000","last_fullsized":"2023-10-30T14:14:05.756505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:16.158442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.822260+0000","last_change":"2023-10-30T14:14:03.717903+0000","last_active":"2023-10-30T14:14:20.822260+0000","last_peered":"2023-10-30T14:14:20.822260+0000","last_clean":"2023-10-30T14:14:20.822260+0000","last_became_active":"2023-10-30T14:14:03.716941+0000","last_became_peered":"2023-10-30T14:14:03.716941+0000","last_unstale":"2023-10-30T14:14:20.822260+0000","last_undegraded":"2023-10-30T14:14:20.822260+0000","last_fullsized":"2023-10-30T14:14:20.822260+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:58.178166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586173+0000","last_change":"2023-10-30T14:14:04.641757+0000","last_active":"2023-10-30T14:14:05.586173+0000","last_peered":"2023-10-30T14:14:05.586173+0000","last_clean":"2023-10-30T14:14:05.586173+0000","last_became_active":"2023-10-30T14:14:04.641494+0000","last_became_peered":"2023-10-30T14:14:04.641494+0000","last_unstale":"2023-10-30T14:14:05.586173+0000","last_undegraded":"2023-10-30T14:14:05.586173+0000","last_fullsized":"2023-10-30T14:14:05.586173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:00:18.795083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.773009+0000","last_change":"2023-10-30T14:14:03.729626+0000","last_active":"2023-10-30T14:14:14.773009+0000","last_peered":"2023-10-30T14:14:14.773009+0000","last_clean":"2023-10-30T14:14:14.773009+0000","last_became_active":"2023-10-30T14:14:03.728053+0000","last_became_peered":"2023-10-30T14:14:03.728053+0000","last_unstale":"2023-10-30T14:14:14.773009+0000","last_undegraded":"2023-10-30T14:14:14.773009+0000","last_fullsized":"2023-10-30T14:14:14.773009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:44:53.362066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756773+0000","last_change":"2023-10-30T14:14:04.650513+0000","last_active":"2023-10-30T14:14:05.756773+0000","last_peered":"2023-10-30T14:14:05.756773+0000","last_clean":"2023-10-30T14:14:05.756773+0000","last_became_active":"2023-10-30T14:14:04.649896+0000","last_became_peered":"2023-10-30T14:14:04.649896+0000","last_unstale":"2023-10-30T14:14:05.756773+0000","last_undegraded":"2023-10-30T14:14:05.756773+0000","last_fullsized":"2023-10-30T14:14:05.756773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:23.100490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.832690+0000","last_change":"2023-10-30T14:14:03.736824+0000","last_active":"2023-10-30T14:14:20.832690+0000","last_peered":"2023-10-30T14:14:20.832690+0000","last_clean":"2023-10-30T14:14:20.832690+0000","last_became_active":"2023-10-30T14:14:03.736155+0000","last_became_peered":"2023-10-30T14:14:03.736155+0000","last_unstale":"2023-10-30T14:14:20.832690+0000","last_undegraded":"2023-10-30T14:14:20.832690+0000","last_fullsized":"2023-10-30T14:14:20.832690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:35.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":846,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807372+0000","last_change":"2023-10-30T14:14:04.615159+0000","last_active":"2023-10-30T14:14:05.807372+0000","last_peered":"2023-10-30T14:14:05.807372+0000","last_clean":"2023-10-30T14:14:05.807372+0000","last_became_active":"2023-10-30T14:14:04.614328+0000","last_became_peered":"2023-10-30T14:14:04.614328+0000","last_unstale":"2023-10-30T14:14:05.807372+0000","last_undegraded":"2023-10-30T14:14:05.807372+0000","last_fullsized":"2023-10-30T14:14:05.807372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:16:52.999786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'15","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.823603+0000","last_change":"2023-10-30T14:14:03.700690+0000","last_active":"2023-10-30T14:14:20.823603+0000","last_peered":"2023-10-30T14:14:20.823603+0000","last_clean":"2023-10-30T14:14:20.823603+0000","last_became_active":"2023-10-30T14:14:03.700352+0000","last_became_peered":"2023-10-30T14:14:03.700352+0000","last_unstale":"2023-10-30T14:14:20.823603+0000","last_undegraded":"2023-10-30T14:14:20.823603+0000","last_fullsized":"2023-10-30T14:14:20.823603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:03:01.103359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.755948+0000","last_change":"2023-10-30T14:14:04.647253+0000","last_active":"2023-10-30T14:14:05.755948+0000","last_peered":"2023-10-30T14:14:05.755948+0000","last_clean":"2023-10-30T14:14:05.755948+0000","last_became_active":"2023-10-30T14:14:04.646639+0000","last_became_peered":"2023-10-30T14:14:04.646639+0000","last_unstale":"2023-10-30T14:14:05.755948+0000","last_undegraded":"2023-10-30T14:14:05.755948+0000","last_fullsized":"2023-10-30T14:14:05.755948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:22:50.893734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.722823+0000","last_change":"2023-10-30T14:14:03.710717+0000","last_active":"2023-10-30T14:14:09.722823+0000","last_peered":"2023-10-30T14:14:09.722823+0000","last_clean":"2023-10-30T14:14:09.722823+0000","last_became_active":"2023-10-30T14:14:03.707950+0000","last_became_peered":"2023-10-30T14:14:03.707950+0000","last_unstale":"2023-10-30T14:14:09.722823+0000","last_undegraded":"2023-10-30T14:14:09.722823+0000","last_fullsized":"2023-10-30T14:14:09.722823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:29:41.232902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757066+0000","last_change":"2023-10-30T14:14:04.647688+0000","last_active":"2023-10-30T14:14:05.757066+0000","last_peered":"2023-10-30T14:14:05.757066+0000","last_clean":"2023-10-30T14:14:05.757066+0000","last_became_active":"2023-10-30T14:14:04.644761+0000","last_became_peered":"2023-10-30T14:14:04.644761+0000","last_unstale":"2023-10-30T14:14:05.757066+0000","last_undegraded":"2023-10-30T14:14:05.757066+0000","last_fullsized":"2023-10-30T14:14:05.757066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:00:38.482522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.716374+0000","last_change":"2023-10-30T14:14:03.705567+0000","last_active":"2023-10-30T14:14:09.716374+0000","last_peered":"2023-10-30T14:14:09.716374+0000","last_clean":"2023-10-30T14:14:09.716374+0000","last_became_active":"2023-10-30T14:14:03.705186+0000","last_became_peered":"2023-10-30T14:14:03.705186+0000","last_unstale":"2023-10-30T14:14:09.716374+0000","last_undegraded":"2023-10-30T14:14:09.716374+0000","last_fullsized":"2023-10-30T14:14:09.716374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:14.609243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.757235+0000","last_change":"2023-10-30T14:14:04.647335+0000","last_active":"2023-10-30T14:14:05.757235+0000","last_peered":"2023-10-30T14:14:05.757235+0000","last_clean":"2023-10-30T14:14:05.757235+0000","last_became_active":"2023-10-30T14:14:04.646785+0000","last_became_peered":"2023-10-30T14:14:04.646785+0000","last_unstale":"2023-10-30T14:14:05.757235+0000","last_undegraded":"2023-10-30T14:14:05.757235+0000","last_fullsized":"2023-10-30T14:14:05.757235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:58:41.600761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.935812+0000","last_change":"2023-10-30T14:14:03.693349+0000","last_active":"2023-10-30T14:14:06.935812+0000","last_peered":"2023-10-30T14:14:06.935812+0000","last_clean":"2023-10-30T14:14:06.935812+0000","last_became_active":"2023-10-30T14:14:03.692966+0000","last_became_peered":"2023-10-30T14:14:03.692966+0000","last_unstale":"2023-10-30T14:14:06.935812+0000","last_undegraded":"2023-10-30T14:14:06.935812+0000","last_fullsized":"2023-10-30T14:14:06.935812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:30:28.800214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587213+0000","last_change":"2023-10-30T14:14:04.617023+0000","last_active":"2023-10-30T14:14:05.587213+0000","last_peered":"2023-10-30T14:14:05.587213+0000","last_clean":"2023-10-30T14:14:05.587213+0000","last_became_active":"2023-10-30T14:14:04.615925+0000","last_became_peered":"2023-10-30T14:14:04.615925+0000","last_unstale":"2023-10-30T14:14:05.587213+0000","last_undegraded":"2023-10-30T14:14:05.587213+0000","last_fullsized":"2023-10-30T14:14:05.587213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:32:02.289324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.823287+0000","last_change":"2023-10-30T14:14:03.727659+0000","last_active":"2023-10-30T14:14:20.823287+0000","last_peered":"2023-10-30T14:14:20.823287+0000","last_clean":"2023-10-30T14:14:20.823287+0000","last_became_active":"2023-10-30T14:14:03.727409+0000","last_became_peered":"2023-10-30T14:14:03.727409+0000","last_unstale":"2023-10-30T14:14:20.823287+0000","last_undegraded":"2023-10-30T14:14:20.823287+0000","last_fullsized":"2023-10-30T14:14:20.823287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:24.210635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.756468+0000","last_change":"2023-10-30T14:14:04.650083+0000","last_active":"2023-10-30T14:14:05.756468+0000","last_peered":"2023-10-30T14:14:05.756468+0000","last_clean":"2023-10-30T14:14:05.756468+0000","last_became_active":"2023-10-30T14:14:04.647562+0000","last_became_peered":"2023-10-30T14:14:04.647562+0000","last_unstale":"2023-10-30T14:14:05.756468+0000","last_undegraded":"2023-10-30T14:14:05.756468+0000","last_fullsized":"2023-10-30T14:14:05.756468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:11:41.743850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T14:14:04.632968+0000","last_change":"2023-10-30T14:13:54.251572+0000","last_active":"2023-10-30T14:14:04.632968+0000","last_peered":"2023-10-30T14:14:04.632968+0000","last_clean":"2023-10-30T14:14:04.632968+0000","last_became_active":"2023-10-30T14:13:54.250907+0000","last_became_peered":"2023-10-30T14:13:54.250907+0000","last_unstale":"2023-10-30T14:14:04.632968+0000","last_undegraded":"2023-10-30T14:14:04.632968+0000","last_fullsized":"2023-10-30T14:14:04.632968+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_clean_scrub_stamp":"2023-10-30T14:13:53.220267+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:44.590852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.779821+0000","last_change":"2023-10-30T14:14:03.729034+0000","last_active":"2023-10-30T14:14:14.779821+0000","last_peered":"2023-10-30T14:14:14.779821+0000","last_clean":"2023-10-30T14:14:14.779821+0000","last_became_active":"2023-10-30T14:14:03.728293+0000","last_became_peered":"2023-10-30T14:14:03.728293+0000","last_unstale":"2023-10-30T14:14:14.779821+0000","last_undegraded":"2023-10-30T14:14:14.779821+0000","last_fullsized":"2023-10-30T14:14:14.779821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806225+0000","last_change":"2023-10-30T14:14:04.630325+0000","last_active":"2023-10-30T14:14:05.806225+0000","last_peered":"2023-10-30T14:14:05.806225+0000","last_clean":"2023-10-30T14:14:05.806225+0000","last_became_active":"2023-10-30T14:14:04.630121+0000","last_became_peered":"2023-10-30T14:14:04.630121+0000","last_unstale":"2023-10-30T14:14:05.806225+0000","last_undegraded":"2023-10-30T14:14:05.806225+0000","last_fullsized":"2023-10-30T14:14:05.806225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:02:28.792879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.822027+0000","last_change":"2023-10-30T14:14:03.732909+0000","last_active":"2023-10-30T14:14:20.822027+0000","last_peered":"2023-10-30T14:14:20.822027+0000","last_clean":"2023-10-30T14:14:20.822027+0000","last_became_active":"2023-10-30T14:14:03.732199+0000","last_became_peered":"2023-10-30T14:14:03.732199+0000","last_unstale":"2023-10-30T14:14:20.822027+0000","last_undegraded":"2023-10-30T14:14:20.822027+0000","last_fullsized":"2023-10-30T14:14:20.822027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:38.117750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586172+0000","last_change":"2023-10-30T14:14:04.620314+0000","last_active":"2023-10-30T14:14:05.586172+0000","last_peered":"2023-10-30T14:14:05.586172+0000","last_clean":"2023-10-30T14:14:05.586172+0000","last_became_active":"2023-10-30T14:14:04.619990+0000","last_became_peered":"2023-10-30T14:14:04.619990+0000","last_unstale":"2023-10-30T14:14:05.586172+0000","last_undegraded":"2023-10-30T14:14:05.586172+0000","last_fullsized":"2023-10-30T14:14:05.586172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:38:25.657586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.822214+0000","last_change":"2023-10-30T14:14:03.702862+0000","last_active":"2023-10-30T14:14:20.822214+0000","last_peered":"2023-10-30T14:14:20.822214+0000","last_clean":"2023-10-30T14:14:20.822214+0000","last_became_active":"2023-10-30T14:14:03.702513+0000","last_became_peered":"2023-10-30T14:14:03.702513+0000","last_unstale":"2023-10-30T14:14:20.822214+0000","last_undegraded":"2023-10-30T14:14:20.822214+0000","last_fullsized":"2023-10-30T14:14:20.822214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:25.499390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584623+0000","last_change":"2023-10-30T14:14:04.589353+0000","last_active":"2023-10-30T14:14:05.584623+0000","last_peered":"2023-10-30T14:14:05.584623+0000","last_clean":"2023-10-30T14:14:05.584623+0000","last_became_active":"2023-10-30T14:14:04.589087+0000","last_became_peered":"2023-10-30T14:14:04.589087+0000","last_unstale":"2023-10-30T14:14:05.584623+0000","last_undegraded":"2023-10-30T14:14:05.584623+0000","last_fullsized":"2023-10-30T14:14:05.584623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:20:12.808641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.965154+0000","last_change":"2023-10-30T14:14:03.730048+0000","last_active":"2023-10-30T14:14:06.965154+0000","last_peered":"2023-10-30T14:14:06.965154+0000","last_clean":"2023-10-30T14:14:06.965154+0000","last_became_active":"2023-10-30T14:14:03.728925+0000","last_became_peered":"2023-10-30T14:14:03.728925+0000","last_unstale":"2023-10-30T14:14:06.965154+0000","last_undegraded":"2023-10-30T14:14:06.965154+0000","last_fullsized":"2023-10-30T14:14:06.965154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:32:05.056039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809437+0000","last_change":"2023-10-30T14:14:04.608289+0000","last_active":"2023-10-30T14:14:05.809437+0000","last_peered":"2023-10-30T14:14:05.809437+0000","last_clean":"2023-10-30T14:14:05.809437+0000","last_became_active":"2023-10-30T14:14:04.607377+0000","last_became_peered":"2023-10-30T14:14:04.607377+0000","last_unstale":"2023-10-30T14:14:05.809437+0000","last_undegraded":"2023-10-30T14:14:05.809437+0000","last_fullsized":"2023-10-30T14:14:05.809437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:43:40.869579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584795+0000","last_change":"2023-10-30T14:14:03.693753+0000","last_active":"2023-10-30T14:14:05.584795+0000","last_peered":"2023-10-30T14:14:05.584795+0000","last_clean":"2023-10-30T14:14:05.584795+0000","last_became_active":"2023-10-30T14:14:03.692621+0000","last_became_peered":"2023-10-30T14:14:03.692621+0000","last_unstale":"2023-10-30T14:14:05.584795+0000","last_undegraded":"2023-10-30T14:14:05.584795+0000","last_fullsized":"2023-10-30T14:14:05.584795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:32:28.577584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587225+0000","last_change":"2023-10-30T14:14:04.614649+0000","last_active":"2023-10-30T14:14:05.587225+0000","last_peered":"2023-10-30T14:14:05.587225+0000","last_clean":"2023-10-30T14:14:05.587225+0000","last_became_active":"2023-10-30T14:14:04.613780+0000","last_became_peered":"2023-10-30T14:14:04.613780+0000","last_unstale":"2023-10-30T14:14:05.587225+0000","last_undegraded":"2023-10-30T14:14:05.587225+0000","last_fullsized":"2023-10-30T14:14:05.587225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:47:40.354022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.937485+0000","last_change":"2023-10-30T14:14:03.740213+0000","last_active":"2023-10-30T14:14:06.937485+0000","last_peered":"2023-10-30T14:14:06.937485+0000","last_clean":"2023-10-30T14:14:06.937485+0000","last_became_active":"2023-10-30T14:14:03.739650+0000","last_became_peered":"2023-10-30T14:14:03.739650+0000","last_unstale":"2023-10-30T14:14:06.937485+0000","last_undegraded":"2023-10-30T14:14:06.937485+0000","last_fullsized":"2023-10-30T14:14:06.937485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:54.120757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.584691+0000","last_change":"2023-10-30T14:14:04.590569+0000","last_active":"2023-10-30T14:14:05.584691+0000","last_peered":"2023-10-30T14:14:05.584691+0000","last_clean":"2023-10-30T14:14:05.584691+0000","last_became_active":"2023-10-30T14:14:04.590138+0000","last_became_peered":"2023-10-30T14:14:04.590138+0000","last_unstale":"2023-10-30T14:14:05.584691+0000","last_undegraded":"2023-10-30T14:14:05.584691+0000","last_fullsized":"2023-10-30T14:14:05.584691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:41:30.333668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.824452+0000","last_change":"2023-10-30T14:14:03.721450+0000","last_active":"2023-10-30T14:14:20.824452+0000","last_peered":"2023-10-30T14:14:20.824452+0000","last_clean":"2023-10-30T14:14:20.824452+0000","last_became_active":"2023-10-30T14:14:03.721234+0000","last_became_peered":"2023-10-30T14:14:03.721234+0000","last_unstale":"2023-10-30T14:14:20.824452+0000","last_undegraded":"2023-10-30T14:14:20.824452+0000","last_fullsized":"2023-10-30T14:14:20.824452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:26.959676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806005+0000","last_change":"2023-10-30T14:14:04.634004+0000","last_active":"2023-10-30T14:14:05.806005+0000","last_peered":"2023-10-30T14:14:05.806005+0000","last_clean":"2023-10-30T14:14:05.806005+0000","last_became_active":"2023-10-30T14:14:04.633643+0000","last_became_peered":"2023-10-30T14:14:04.633643+0000","last_unstale":"2023-10-30T14:14:05.806005+0000","last_undegraded":"2023-10-30T14:14:05.806005+0000","last_fullsized":"2023-10-30T14:14:05.806005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:05:14.572477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.769216+0000","last_change":"2023-10-30T14:14:03.738557+0000","last_active":"2023-10-30T14:14:14.769216+0000","last_peered":"2023-10-30T14:14:14.769216+0000","last_clean":"2023-10-30T14:14:14.769216+0000","last_became_active":"2023-10-30T14:14:03.737801+0000","last_became_peered":"2023-10-30T14:14:03.737801+0000","last_unstale":"2023-10-30T14:14:14.769216+0000","last_undegraded":"2023-10-30T14:14:14.769216+0000","last_fullsized":"2023-10-30T14:14:14.769216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:26.938343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586581+0000","last_change":"2023-10-30T14:14:04.589273+0000","last_active":"2023-10-30T14:14:05.586581+0000","last_peered":"2023-10-30T14:14:05.586581+0000","last_clean":"2023-10-30T14:14:05.586581+0000","last_became_active":"2023-10-30T14:14:04.588826+0000","last_became_peered":"2023-10-30T14:14:04.588826+0000","last_unstale":"2023-10-30T14:14:05.586581+0000","last_undegraded":"2023-10-30T14:14:05.586581+0000","last_fullsized":"2023-10-30T14:14:05.586581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:33:52.452394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.787158+0000","last_change":"2023-10-30T14:14:03.733836+0000","last_active":"2023-10-30T14:14:14.787158+0000","last_peered":"2023-10-30T14:14:14.787158+0000","last_clean":"2023-10-30T14:14:14.787158+0000","last_became_active":"2023-10-30T14:14:03.732912+0000","last_became_peered":"2023-10-30T14:14:03.732912+0000","last_unstale":"2023-10-30T14:14:14.787158+0000","last_undegraded":"2023-10-30T14:14:14.787158+0000","last_fullsized":"2023-10-30T14:14:14.787158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:50.332738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.809314+0000","last_change":"2023-10-30T14:14:04.610696+0000","last_active":"2023-10-30T14:14:05.809314+0000","last_peered":"2023-10-30T14:14:05.809314+0000","last_clean":"2023-10-30T14:14:05.809314+0000","last_became_active":"2023-10-30T14:14:04.609835+0000","last_became_peered":"2023-10-30T14:14:04.609835+0000","last_unstale":"2023-10-30T14:14:05.809314+0000","last_undegraded":"2023-10-30T14:14:05.809314+0000","last_fullsized":"2023-10-30T14:14:05.809314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:04:31.270262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.795220+0000","last_change":"2023-10-30T14:14:03.732539+0000","last_active":"2023-10-30T14:14:14.795220+0000","last_peered":"2023-10-30T14:14:14.795220+0000","last_clean":"2023-10-30T14:14:14.795220+0000","last_became_active":"2023-10-30T14:14:03.732265+0000","last_became_peered":"2023-10-30T14:14:03.732265+0000","last_unstale":"2023-10-30T14:14:14.795220+0000","last_undegraded":"2023-10-30T14:14:14.795220+0000","last_fullsized":"2023-10-30T14:14:14.795220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:02:00.929612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806211+0000","last_change":"2023-10-30T14:14:04.595357+0000","last_active":"2023-10-30T14:14:05.806211+0000","last_peered":"2023-10-30T14:14:05.806211+0000","last_clean":"2023-10-30T14:14:05.806211+0000","last_became_active":"2023-10-30T14:14:04.594998+0000","last_became_peered":"2023-10-30T14:14:04.594998+0000","last_unstale":"2023-10-30T14:14:05.806211+0000","last_undegraded":"2023-10-30T14:14:05.806211+0000","last_fullsized":"2023-10-30T14:14:05.806211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:44.365673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:06.935738+0000","last_change":"2023-10-30T14:14:03.703744+0000","last_active":"2023-10-30T14:14:06.935738+0000","last_peered":"2023-10-30T14:14:06.935738+0000","last_clean":"2023-10-30T14:14:06.935738+0000","last_became_active":"2023-10-30T14:14:03.701759+0000","last_became_peered":"2023-10-30T14:14:03.701759+0000","last_unstale":"2023-10-30T14:14:06.935738+0000","last_undegraded":"2023-10-30T14:14:06.935738+0000","last_fullsized":"2023-10-30T14:14:06.935738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:22:27.948007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806186+0000","last_change":"2023-10-30T14:14:04.600759+0000","last_active":"2023-10-30T14:14:05.806186+0000","last_peered":"2023-10-30T14:14:05.806186+0000","last_clean":"2023-10-30T14:14:05.806186+0000","last_became_active":"2023-10-30T14:14:04.600100+0000","last_became_peered":"2023-10-30T14:14:04.600100+0000","last_unstale":"2023-10-30T14:14:05.806186+0000","last_undegraded":"2023-10-30T14:14:05.806186+0000","last_fullsized":"2023-10-30T14:14:05.806186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:02.831207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:14.773849+0000","last_change":"2023-10-30T14:14:03.741268+0000","last_active":"2023-10-30T14:14:14.773849+0000","last_peered":"2023-10-30T14:14:14.773849+0000","last_clean":"2023-10-30T14:14:14.773849+0000","last_became_active":"2023-10-30T14:14:03.740648+0000","last_became_peered":"2023-10-30T14:14:03.740648+0000","last_unstale":"2023-10-30T14:14:14.773849+0000","last_undegraded":"2023-10-30T14:14:14.773849+0000","last_fullsized":"2023-10-30T14:14:14.773849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:30.799148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.807541+0000","last_change":"2023-10-30T14:14:04.623644+0000","last_active":"2023-10-30T14:14:05.807541+0000","last_peered":"2023-10-30T14:14:05.807541+0000","last_clean":"2023-10-30T14:14:05.807541+0000","last_became_active":"2023-10-30T14:14:04.622921+0000","last_became_peered":"2023-10-30T14:14:04.622921+0000","last_unstale":"2023-10-30T14:14:05.807541+0000","last_undegraded":"2023-10-30T14:14:05.807541+0000","last_fullsized":"2023-10-30T14:14:05.807541+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:11:41.810610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.587103+0000","last_change":"2023-10-30T14:14:04.624799+0000","last_active":"2023-10-30T14:14:05.587103+0000","last_peered":"2023-10-30T14:14:05.587103+0000","last_clean":"2023-10-30T14:14:05.587103+0000","last_became_active":"2023-10-30T14:14:04.621885+0000","last_became_peered":"2023-10-30T14:14:04.621885+0000","last_unstale":"2023-10-30T14:14:05.587103+0000","last_undegraded":"2023-10-30T14:14:05.587103+0000","last_fullsized":"2023-10-30T14:14:05.587103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:12.055247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.721291+0000","last_change":"2023-10-30T14:14:03.740897+0000","last_active":"2023-10-30T14:14:09.721291+0000","last_peered":"2023-10-30T14:14:09.721291+0000","last_clean":"2023-10-30T14:14:09.721291+0000","last_became_active":"2023-10-30T14:14:03.740247+0000","last_became_peered":"2023-10-30T14:14:03.740247+0000","last_unstale":"2023-10-30T14:14:09.721291+0000","last_undegraded":"2023-10-30T14:14:09.721291+0000","last_fullsized":"2023-10-30T14:14:09.721291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:29.601978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586861+0000","last_change":"2023-10-30T14:14:04.625498+0000","last_active":"2023-10-30T14:14:05.586861+0000","last_peered":"2023-10-30T14:14:05.586861+0000","last_clean":"2023-10-30T14:14:05.586861+0000","last_became_active":"2023-10-30T14:14:04.623638+0000","last_became_peered":"2023-10-30T14:14:04.623638+0000","last_unstale":"2023-10-30T14:14:05.586861+0000","last_undegraded":"2023-10-30T14:14:05.586861+0000","last_fullsized":"2023-10-30T14:14:05.586861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:19.717965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.825113+0000","last_change":"2023-10-30T14:14:03.733233+0000","last_active":"2023-10-30T14:14:20.825113+0000","last_peered":"2023-10-30T14:14:20.825113+0000","last_clean":"2023-10-30T14:14:20.825113+0000","last_became_active":"2023-10-30T14:14:03.732538+0000","last_became_peered":"2023-10-30T14:14:03.732538+0000","last_unstale":"2023-10-30T14:14:20.825113+0000","last_undegraded":"2023-10-30T14:14:20.825113+0000","last_fullsized":"2023-10-30T14:14:20.825113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:25.696173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.586619+0000","last_change":"2023-10-30T14:14:04.612104+0000","last_active":"2023-10-30T14:14:05.586619+0000","last_peered":"2023-10-30T14:14:05.586619+0000","last_clean":"2023-10-30T14:14:05.586619+0000","last_became_active":"2023-10-30T14:14:04.611773+0000","last_became_peered":"2023-10-30T14:14:04.611773+0000","last_unstale":"2023-10-30T14:14:05.586619+0000","last_undegraded":"2023-10-30T14:14:05.586619+0000","last_fullsized":"2023-10-30T14:14:05.586619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:44:21.952103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:09.721542+0000","last_change":"2023-10-30T14:14:03.703988+0000","last_active":"2023-10-30T14:14:09.721542+0000","last_peered":"2023-10-30T14:14:09.721542+0000","last_clean":"2023-10-30T14:14:09.721542+0000","last_became_active":"2023-10-30T14:14:03.703691+0000","last_became_peered":"2023-10-30T14:14:03.703691+0000","last_unstale":"2023-10-30T14:14:09.721542+0000","last_undegraded":"2023-10-30T14:14:09.721542+0000","last_fullsized":"2023-10-30T14:14:09.721542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:34.538541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:05.806119+0000","last_change":"2023-10-30T14:14:04.613738+0000","last_active":"2023-10-30T14:14:05.806119+0000","last_peered":"2023-10-30T14:14:05.806119+0000","last_clean":"2023-10-30T14:14:05.806119+0000","last_became_active":"2023-10-30T14:14:04.613420+0000","last_became_peered":"2023-10-30T14:14:04.613420+0000","last_unstale":"2023-10-30T14:14:05.806119+0000","last_undegraded":"2023-10-30T14:14:05.806119+0000","last_fullsized":"2023-10-30T14:14:05.806119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:01:52.695195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'9","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T14:14:20.825364+0000","last_change":"2023-10-30T14:14:03.737319+0000","last_active":"2023-10-30T14:14:20.825364+0000","last_peered":"2023-10-30T14:14:20.825364+0000","last_clean":"2023-10-30T14:14:20.825364+0000","last_became_active":"2023-10-30T14:14:03.736964+0000","last_became_peered":"2023-10-30T14:14:03.736964+0000","last_unstale":"2023-10-30T14:14:20.825364+0000","last_undegraded":"2023-10-30T14:14:20.825364+0000","last_fullsized":"2023-10-30T14:14:20.825364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:41.469809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6553,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":13106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113,"ondisk_log_size":113,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":831488,"data_stored":667253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2849999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.462}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3580000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":79190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1160000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]}]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":76947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.018}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":76541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.45}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6719999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]}]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":241664,"data_stored":78966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":76387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":76513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574861,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":827392,"data_stored":666597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:14:34.288 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:14:34.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.463+0000 7f3db83a1640 1 Processor -- start 2023-10-30T14:14:34.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.463+0000 7f3db83a1640 1 -- start start 2023-10-30T14:14:34.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.463+0000 7f3db83a1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3db0153c80 0x7f3db015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:34.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.463+0000 7f3db83a1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3db00c4a10 0x7f3db00c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:34.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.463+0000 7f3db83a1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3db015a3b0 0x7f3db00ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:34.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.463+0000 7f3db83a1640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3db015dec0 con 0x7f3db015a3b0 2023-10-30T14:14:34.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.463+0000 7f3db83a1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3db01599a0 con 0x7f3db00c4a10 2023-10-30T14:14:34.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.463+0000 7f3db83a1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db0159f20 con 0x7f3db0153c80 2023-10-30T14:14:34.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.463+0000 7f3db693e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3db015a3b0 0x7f3db00ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:34.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.463+0000 7f3db693e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3db015a3b0 0x7f3db00ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39782/0 (socket says 172.21.15.57:39782) 2023-10-30T14:14:34.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.463+0000 7f3db693e640 1 -- 172.21.15.57:0/478977783 learned_addr learned my addr 172.21.15.57:0/478977783 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:34.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.463+0000 7f3db593c640 1 --2- 172.21.15.57:0/478977783 >> v2:172.21.15.88:3300/0 conn(0x7f3db00c4a10 0x7f3db00c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:34.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.463+0000 7f3db613d640 1 --2- 172.21.15.57:0/478977783 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3db0153c80 0x7f3db015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:34.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.467+0000 7f3db693e640 1 -- 172.21.15.57:0/478977783 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3db0153c80 msgr2=0x7f3db015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:34.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.467+0000 7f3db693e640 1 --2- 172.21.15.57:0/478977783 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3db0153c80 0x7f3db015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.467+0000 7f3db693e640 1 -- 172.21.15.57:0/478977783 >> v2:172.21.15.88:3300/0 conn(0x7f3db00c4a10 msgr2=0x7f3db00c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:34.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.467+0000 7f3db693e640 1 --2- 172.21.15.57:0/478977783 >> v2:172.21.15.88:3300/0 conn(0x7f3db00c4a10 0x7f3db00c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.467+0000 7f3db693e640 1 -- 172.21.15.57:0/478977783 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db00c47e0 con 0x7f3db015a3b0 2023-10-30T14:14:34.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.467+0000 7f3db693e640 1 --2- 172.21.15.57:0/478977783 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3db015a3b0 0x7f3db00ca1e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3db008cc20 tx=0x7f3da801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff6113947f52f66 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:34.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.467+0000 7f3db513b640 1 -- 172.21.15.57:0/478977783 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da80af400 con 0x7f3db015a3b0 2023-10-30T14:14:34.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.467+0000 7f3db513b640 1 -- 172.21.15.57:0/478977783 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da80ab050 con 0x7f3db015a3b0 2023-10-30T14:14:34.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.467+0000 7f3db513b640 1 -- 172.21.15.57:0/478977783 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da80af780 con 0x7f3db015a3b0 2023-10-30T14:14:34.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.467+0000 7f3db83a1640 1 -- 172.21.15.57:0/478977783 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3db015a3b0 msgr2=0x7f3db00ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:34.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.467+0000 7f3db83a1640 1 --2- 172.21.15.57:0/478977783 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3db015a3b0 0x7f3db00ca1e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3db008cc20 tx=0x7f3da801f1c0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db83a1640 1 -- 172.21.15.57:0/478977783 shutdown_connections 2023-10-30T14:14:34.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db83a1640 1 --2- 172.21.15.57:0/478977783 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3db015a3b0 0x7f3db00ca1e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db83a1640 1 --2- 172.21.15.57:0/478977783 >> v2:172.21.15.88:3300/0 conn(0x7f3db00c4a10 0x7f3db00c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db83a1640 1 --2- 172.21.15.57:0/478977783 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3db0153c80 0x7f3db015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db83a1640 1 -- 172.21.15.57:0/478977783 >> 172.21.15.57:0/478977783 conn(0x7f3db00715c0 msgr2=0x7f3db00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:34.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db83a1640 1 -- 172.21.15.57:0/478977783 shutdown_connections 2023-10-30T14:14:34.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db83a1640 1 -- 172.21.15.57:0/478977783 wait complete. 2023-10-30T14:14:34.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db83a1640 1 Processor -- start 2023-10-30T14:14:34.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db83a1640 1 -- start start 2023-10-30T14:14:34.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db83a1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3db00c4a10 0x7f3db01f2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:34.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db83a1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3db0153c80 0x7f3db01ef420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:34.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db83a1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3db015a3b0 0x7f3db014f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:34.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db83a1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db014ad60 con 0x7f3db00c4a10 2023-10-30T14:14:34.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db83a1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db01ef000 con 0x7f3db015a3b0 2023-10-30T14:14:34.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db83a1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db00ce3b0 con 0x7f3db0153c80 2023-10-30T14:14:34.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db613d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3db00c4a10 0x7f3db01f2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:34.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db613d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3db00c4a10 0x7f3db01f2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39788/0 (socket says 172.21.15.57:39788) 2023-10-30T14:14:34.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db613d640 1 -- 172.21.15.57:0/1008250454 learned_addr learned my addr 172.21.15.57:0/1008250454 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:34.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db613d640 1 -- 172.21.15.57:0/1008250454 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3db0153c80 msgr2=0x7f3db01ef420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:34.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db593c640 1 --2- 172.21.15.57:0/1008250454 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3db0153c80 0x7f3db01ef420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:34.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db693e640 1 --2- 172.21.15.57:0/1008250454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3db015a3b0 0x7f3db014f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:34.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db613d640 1 --2- 172.21.15.57:0/1008250454 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3db0153c80 0x7f3db01ef420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db613d640 1 -- 172.21.15.57:0/1008250454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3db015a3b0 msgr2=0x7f3db014f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:34.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db613d640 1 --2- 172.21.15.57:0/1008250454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3db015a3b0 0x7f3db014f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db613d640 1 -- 172.21.15.57:0/1008250454 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da8009b30 con 0x7f3db00c4a10 2023-10-30T14:14:34.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db693e640 1 --2- 172.21.15.57:0/1008250454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3db015a3b0 0x7f3db014f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:14:34.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db593c640 1 --2- 172.21.15.57:0/1008250454 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3db0153c80 0x7f3db01ef420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:14:34.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.471+0000 7f3db613d640 1 --2- 172.21.15.57:0/1008250454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3db00c4a10 0x7f3db01f2030 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3dac00d800 tx=0x7f3dac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:34.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.475+0000 7f3da6ffd640 1 -- 172.21.15.57:0/1008250454 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac00b6c0 con 0x7f3db00c4a10 2023-10-30T14:14:34.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.475+0000 7f3da6ffd640 1 -- 172.21.15.57:0/1008250454 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dac00b820 con 0x7f3db00c4a10 2023-10-30T14:14:34.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.475+0000 7f3da6ffd640 1 -- 172.21.15.57:0/1008250454 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac00bb90 con 0x7f3db00c4a10 2023-10-30T14:14:34.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.475+0000 7f3db83a1640 1 -- 172.21.15.57:0/1008250454 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db01f2f20 con 0x7f3db00c4a10 2023-10-30T14:14:34.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.475+0000 7f3db83a1640 1 -- 172.21.15.57:0/1008250454 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db01f2960 con 0x7f3db00c4a10 2023-10-30T14:14:34.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.475+0000 7f3db83a1640 1 -- 172.21.15.57:0/1008250454 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db004e170 con 0x7f3db00c4a10 2023-10-30T14:14:34.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.475+0000 7f3da6ffd640 1 -- 172.21.15.57:0/1008250454 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3dac00bcf0 con 0x7f3db00c4a10 2023-10-30T14:14:34.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.475+0000 7f3da6ffd640 1 --2- 172.21.15.57:0/1008250454 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3d940664a0 0x7f3d94068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:34.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.475+0000 7f3da6ffd640 1 -- 172.21.15.57:0/1008250454 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3dac0844e0 con 0x7f3db00c4a10 2023-10-30T14:14:34.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.475+0000 7f3db593c640 1 --2- 172.21.15.57:0/1008250454 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3d940664a0 0x7f3d94068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:34.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.479+0000 7f3db593c640 1 --2- 172.21.15.57:0/1008250454 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3d940664a0 0x7f3d94068960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3db0158b60 tx=0x7f3da0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:34.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.479+0000 7f3da6ffd640 1 -- 172.21.15.57:0/1008250454 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3dac0518d0 con 0x7f3db00c4a10 2023-10-30T14:14:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.639+0000 7f3db83a1640 1 -- 172.21.15.57:0/1008250454 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3db00cad40 con 0x7f3db00c4a10 2023-10-30T14:14:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.639+0000 7f3da6ffd640 1 -- 172.21.15.57:0/1008250454 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16433 (secure 0 0 0) 0x7f3dac054ef0 con 0x7f3db00c4a10 2023-10-30T14:14:34.646 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:34.646 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:14:05.576818+0000","last_up_change":"2023-10-30T14:13:51.171287+0000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T14:13:52.574639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T14:14:02.600122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T14:14:03.069429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6801","nonce":3287367848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6803","nonce":3287367848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6807","nonce":3287367848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6805","nonce":3287367848}]},"public_addr":"172.21.15.57:6801/3287367848","cluster_addr":"172.21.15.57:6803/3287367848","heartbeat_back_addr":"172.21.15.57:6807/3287367848","heartbeat_front_addr":"172.21.15.57:6805/3287367848","state":["exists","up"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6809","nonce":3340797836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6811","nonce":3340797836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6815","nonce":3340797836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6813","nonce":3340797836}]},"public_addr":"172.21.15.57:6809/3340797836","cluster_addr":"172.21.15.57:6811/3340797836","heartbeat_back_addr":"172.21.15.57:6815/3340797836","heartbeat_front_addr":"172.21.15.57:6813/3340797836","state":["exists","up"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6825","nonce":1700292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6827","nonce":1700292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6831","nonce":1700292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6829","nonce":1700292}]},"public_addr":"172.21.15.57:6825/1700292","cluster_addr":"172.21.15.57:6827/1700292","heartbeat_back_addr":"172.21.15.57:6831/1700292","heartbeat_front_addr":"172.21.15.57:6829/1700292","state":["exists","up"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6817","nonce":2893420631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6819","nonce":2893420631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6823","nonce":2893420631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6821","nonce":2893420631}]},"public_addr":"172.21.15.57:6817/2893420631","cluster_addr":"172.21.15.57:6819/2893420631","heartbeat_back_addr":"172.21.15.57:6823/2893420631","heartbeat_front_addr":"172.21.15.57:6821/2893420631","state":["exists","up"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6807","nonce":2784816485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6811","nonce":2784816485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6815","nonce":2784816485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6813","nonce":2784816485}]},"public_addr":"172.21.15.88:6807/2784816485","cluster_addr":"172.21.15.88:6811/2784816485","heartbeat_back_addr":"172.21.15.88:6815/2784816485","heartbeat_front_addr":"172.21.15.88:6813/2784816485","state":["exists","up"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6801","nonce":2682449812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6803","nonce":2682449812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6810","nonce":2682449812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6806","nonce":2682449812}]},"public_addr":"172.21.15.88:6801/2682449812","cluster_addr":"172.21.15.88:6803/2682449812","heartbeat_back_addr":"172.21.15.88:6810/2682449812","heartbeat_front_addr":"172.21.15.88:6806/2682449812","state":["exists","up"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6822","nonce":3340693897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6827","nonce":3340693897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6831","nonce":3340693897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6829","nonce":3340693897}]},"public_addr":"172.21.15.88:6822/3340693897","cluster_addr":"172.21.15.88:6827/3340693897","heartbeat_back_addr":"172.21.15.88:6831/3340693897","heartbeat_front_addr":"172.21.15.88:6829/3340693897","state":["exists","up"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6817","nonce":2862111199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6820","nonce":2862111199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6825","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6826","nonce":2862111199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6821","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6823","nonce":2862111199}]},"public_addr":"172.21.15.88:6817/2862111199","cluster_addr":"172.21.15.88:6820/2862111199","heartbeat_back_addr":"172.21.15.88:6826/2862111199","heartbeat_front_addr":"172.21.15.88:6823/2862111199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.634920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.773993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.725825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.761452+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.694274+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.650718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:14:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.643+0000 7f3db83a1640 1 -- 172.21.15.57:0/1008250454 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3d940664a0 msgr2=0x7f3d94068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.643+0000 7f3db83a1640 1 --2- 172.21.15.57:0/1008250454 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3d940664a0 0x7f3d94068960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3db0158b60 tx=0x7f3da0008040 comp rx=0 tx=0).stop 2023-10-30T14:14:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.643+0000 7f3db83a1640 1 -- 172.21.15.57:0/1008250454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3db00c4a10 msgr2=0x7f3db01f2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.643+0000 7f3db83a1640 1 --2- 172.21.15.57:0/1008250454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3db00c4a10 0x7f3db01f2030 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3dac00d800 tx=0x7f3dac00dcc0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.643+0000 7f3db83a1640 1 -- 172.21.15.57:0/1008250454 shutdown_connections 2023-10-30T14:14:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.643+0000 7f3db83a1640 1 --2- 172.21.15.57:0/1008250454 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3d940664a0 0x7f3d94068960 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.643+0000 7f3db83a1640 1 --2- 172.21.15.57:0/1008250454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3db015a3b0 0x7f3db014f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.643+0000 7f3db83a1640 1 --2- 172.21.15.57:0/1008250454 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3db0153c80 0x7f3db01ef420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.643+0000 7f3db83a1640 1 --2- 172.21.15.57:0/1008250454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3db00c4a10 0x7f3db01f2030 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.643+0000 7f3db83a1640 1 -- 172.21.15.57:0/1008250454 >> 172.21.15.57:0/1008250454 conn(0x7f3db00715c0 msgr2=0x7f3db00c4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.643+0000 7f3db83a1640 1 -- 172.21.15.57:0/1008250454 shutdown_connections 2023-10-30T14:14:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.643+0000 7f3db83a1640 1 -- 172.21.15.57:0/1008250454 wait complete. 2023-10-30T14:14:34.667 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T14:13:52.574639+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T14:14:02.600122+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T14:14:03.069429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T14:14:34.668 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T14:14:34.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.807+0000 7f8df6e9c640 1 Processor -- start 2023-10-30T14:14:34.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.807+0000 7f8df6e9c640 1 -- start start 2023-10-30T14:14:34.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.807+0000 7f8df6e9c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8df0151a20 0x7f8df00cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:34.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.807+0000 7f8df6e9c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8df00c74a0 0x7f8df00c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:34.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.807+0000 7f8df6e9c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df01581f0 con 0x7f8df00c1020 2023-10-30T14:14:34.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.807+0000 7f8df6e9c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df01576d0 con 0x7f8df0151a20 2023-10-30T14:14:34.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.807+0000 7f8df6e9c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df00c7e10 con 0x7f8df00c74a0 2023-10-30T14:14:34.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.807+0000 7f8deffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f8df00c1020 0x7f8df0072080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46562/0 (socket says 172.21.15.57:46562) 2023-10-30T14:14:34.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.807+0000 7f8deffff640 1 -- 172.21.15.57:0/361547009 learned_addr learned my addr 172.21.15.57:0/361547009 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:34.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.807+0000 7f8df4c38640 1 --2- 172.21.15.57:0/361547009 >> v2:172.21.15.57:3300/0 conn(0x7f8df0151a20 0x7f8df00cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:34.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.807+0000 7f8df5439640 1 --2- 172.21.15.57:0/361547009 >> v2:172.21.15.88:3300/0 conn(0x7f8df00c74a0 0x7f8df00c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:34.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.807+0000 7f8df4c38640 1 -- 172.21.15.57:0/361547009 >> v2:172.21.15.88:3300/0 conn(0x7f8df00c74a0 msgr2=0x7f8df00c78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:34.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.807+0000 7f8df4c38640 1 --2- 172.21.15.57:0/361547009 >> v2:172.21.15.88:3300/0 conn(0x7f8df00c74a0 0x7f8df00c78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.807+0000 7f8df4c38640 1 -- 172.21.15.57:0/361547009 >> v1:172.21.15.57:6789/0 conn(0x7f8df00c1020 legacy=0x7f8df0072080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.807+0000 7f8df4c38640 1 -- 172.21.15.57:0/361547009 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df00cb9f0 con 0x7f8df0151a20 2023-10-30T14:14:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.807+0000 7f8def7fe640 1 -- 172.21.15.57:0/361547009 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785371418 0 0) 0x7f8df01581f0 con 0x7f8df00c1020 2023-10-30T14:14:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.811+0000 7f8df4c38640 1 --2- 172.21.15.57:0/361547009 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8df0151a20 0x7f8df00cb4b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8df00c8bb0 tx=0x7f8de401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb5c42ba3efc7f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.811+0000 7f8def7fe640 1 -- 172.21.15.57:0/361547009 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de40a8950 con 0x7f8df0151a20 2023-10-30T14:14:34.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.811+0000 7f8def7fe640 1 -- 172.21.15.57:0/361547009 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de40a8ab0 con 0x7f8df0151a20 2023-10-30T14:14:34.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.811+0000 7f8def7fe640 1 -- 172.21.15.57:0/361547009 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de40a8e40 con 0x7f8df0151a20 2023-10-30T14:14:34.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.811+0000 7f8df6e9c640 1 -- 172.21.15.57:0/361547009 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8df0151a20 msgr2=0x7f8df00cb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:34.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.811+0000 7f8df6e9c640 1 --2- 172.21.15.57:0/361547009 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8df0151a20 0x7f8df00cb4b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8df00c8bb0 tx=0x7f8de401ec50 comp rx=0 tx=0).stop 2023-10-30T14:14:34.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.811+0000 7f8df6e9c640 1 -- 172.21.15.57:0/361547009 shutdown_connections 2023-10-30T14:14:34.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.811+0000 7f8df6e9c640 1 --2- 172.21.15.57:0/361547009 >> v2:172.21.15.88:3300/0 conn(0x7f8df00c74a0 0x7f8df00c78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.811+0000 7f8df6e9c640 1 --2- 172.21.15.57:0/361547009 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8df0151a20 0x7f8df00cb4b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.811+0000 7f8df6e9c640 1 -- 172.21.15.57:0/361547009 >> 172.21.15.57:0/361547009 conn(0x7f8df006da40 msgr2=0x7f8df006de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:34.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.811+0000 7f8df6e9c640 1 -- 172.21.15.57:0/361547009 shutdown_connections 2023-10-30T14:14:34.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.811+0000 7f8df6e9c640 1 -- 172.21.15.57:0/361547009 wait complete. 2023-10-30T14:14:34.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df6e9c640 1 Processor -- start 2023-10-30T14:14:34.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df6e9c640 1 -- start start 2023-10-30T14:14:34.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df6e9c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8df00c1020 0x7f8df0070780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:34.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df6e9c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8df00c74a0 0x7f8df0143130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:34.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df6e9c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8df0151a20 0x7f8df01eed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:34.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df6e9c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df01e28e0 con 0x7f8df00c1020 2023-10-30T14:14:34.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df6e9c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df01f22f0 con 0x7f8df00c74a0 2023-10-30T14:14:34.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df6e9c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df01e26b0 con 0x7f8df0151a20 2023-10-30T14:14:34.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df4c38640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8df00c1020 0x7f8df0070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:34.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df4c38640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8df00c1020 0x7f8df0070780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39806/0 (socket says 172.21.15.57:39806) 2023-10-30T14:14:34.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df4c38640 1 -- 172.21.15.57:0/1482759388 learned_addr learned my addr 172.21.15.57:0/1482759388 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:34.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8deffff640 1 --2- 172.21.15.57:0/1482759388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8df00c74a0 0x7f8df0143130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:34.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df5439640 1 --2- 172.21.15.57:0/1482759388 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8df0151a20 0x7f8df01eed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:34.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df4c38640 1 -- 172.21.15.57:0/1482759388 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8df0151a20 msgr2=0x7f8df01eed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:34.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df4c38640 1 --2- 172.21.15.57:0/1482759388 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8df0151a20 0x7f8df01eed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df4c38640 1 -- 172.21.15.57:0/1482759388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8df00c74a0 msgr2=0x7f8df0143130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:34.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df4c38640 1 --2- 172.21.15.57:0/1482759388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8df00c74a0 0x7f8df0143130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df4c38640 1 -- 172.21.15.57:0/1482759388 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de4009420 con 0x7f8df00c1020 2023-10-30T14:14:34.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df5439640 1 --2- 172.21.15.57:0/1482759388 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8df0151a20 0x7f8df01eed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:14:34.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df4c38640 1 --2- 172.21.15.57:0/1482759388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8df00c1020 0x7f8df0070780 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8de40b3040 tx=0x7f8de401b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:34.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8ded7fa640 1 -- 172.21.15.57:0/1482759388 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de40b0dd0 con 0x7f8df00c1020 2023-10-30T14:14:34.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8ded7fa640 1 -- 172.21.15.57:0/1482759388 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de40b9430 con 0x7f8df00c1020 2023-10-30T14:14:34.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df6e9c640 1 -- 172.21.15.57:0/1482759388 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df0070cc0 con 0x7f8df00c1020 2023-10-30T14:14:34.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8ded7fa640 1 -- 172.21.15.57:0/1482759388 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de40b97e0 con 0x7f8df00c1020 2023-10-30T14:14:34.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.815+0000 7f8df6e9c640 1 -- 172.21.15.57:0/1482759388 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df01f2500 con 0x7f8df00c1020 2023-10-30T14:14:34.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.819+0000 7f8df6e9c640 1 -- 172.21.15.57:0/1482759388 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df004e170 con 0x7f8df00c1020 2023-10-30T14:14:34.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.819+0000 7f8ded7fa640 1 -- 172.21.15.57:0/1482759388 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8de40b7040 con 0x7f8df00c1020 2023-10-30T14:14:34.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.819+0000 7f8ded7fa640 1 --2- 172.21.15.57:0/1482759388 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dd40664a0 0x7f8dd4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:34.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.819+0000 7f8ded7fa640 1 -- 172.21.15.57:0/1482759388 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8de4121a10 con 0x7f8df00c1020 2023-10-30T14:14:34.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.819+0000 7f8deffff640 1 --2- 172.21.15.57:0/1482759388 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dd40664a0 0x7f8dd4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:34.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.819+0000 7f8deffff640 1 --2- 172.21.15.57:0/1482759388 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dd40664a0 0x7f8dd4068960 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc000f70 tx=0x7f8ddc007c10 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:34.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.823+0000 7f8ded7fa640 1 -- 172.21.15.57:0/1482759388 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8de40eee80 con 0x7f8df00c1020 2023-10-30T14:14:34.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.979+0000 7f8df6e9c640 1 -- 172.21.15.57:0/1482759388 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8df01e2a50 con 0x7f8df00c1020 2023-10-30T14:14:34.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.983+0000 7f8ded7fa640 1 -- 172.21.15.57:0/1482759388 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8de40b5020 con 0x7f8df00c1020 2023-10-30T14:14:34.988 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-30T14:14:34.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.983+0000 7f8df6e9c640 1 -- 172.21.15.57:0/1482759388 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dd40664a0 msgr2=0x7f8dd4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:34.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.983+0000 7f8df6e9c640 1 --2- 172.21.15.57:0/1482759388 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dd40664a0 0x7f8dd4068960 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc000f70 tx=0x7f8ddc007c10 comp rx=0 tx=0).stop 2023-10-30T14:14:34.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.983+0000 7f8df6e9c640 1 -- 172.21.15.57:0/1482759388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8df00c1020 msgr2=0x7f8df0070780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:34.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.983+0000 7f8df6e9c640 1 --2- 172.21.15.57:0/1482759388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8df00c1020 0x7f8df0070780 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8de40b3040 tx=0x7f8de401b9f0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.983+0000 7f8df6e9c640 1 -- 172.21.15.57:0/1482759388 shutdown_connections 2023-10-30T14:14:34.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.983+0000 7f8df6e9c640 1 --2- 172.21.15.57:0/1482759388 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dd40664a0 0x7f8dd4068960 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.983+0000 7f8df6e9c640 1 --2- 172.21.15.57:0/1482759388 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8df0151a20 0x7f8df01eed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.983+0000 7f8df6e9c640 1 --2- 172.21.15.57:0/1482759388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8df00c74a0 0x7f8df0143130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.983+0000 7f8df6e9c640 1 --2- 172.21.15.57:0/1482759388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8df00c1020 0x7f8df0070780 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:34.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.983+0000 7f8df6e9c640 1 -- 172.21.15.57:0/1482759388 >> 172.21.15.57:0/1482759388 conn(0x7f8df006da40 msgr2=0x7f8df0072080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:34.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.983+0000 7f8df6e9c640 1 -- 172.21.15.57:0/1482759388 shutdown_connections 2023-10-30T14:14:34.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:34.983+0000 7f8df6e9c640 1 -- 172.21.15.57:0/1482759388 wait complete. 2023-10-30T14:14:35.009 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T14:14:35.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.147+0000 7f389333e640 1 Processor -- start 2023-10-30T14:14:35.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.151+0000 7f389333e640 1 -- start start 2023-10-30T14:14:35.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.151+0000 7f389333e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f388c0c1430 0x7f388c159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:35.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.151+0000 7f389333e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f388c15a040 0x7f388c0cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:35.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.151+0000 7f389333e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388c157100 con 0x7f388c150630 2023-10-30T14:14:35.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.151+0000 7f389333e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f388c150a10 con 0x7f388c0c1430 2023-10-30T14:14:35.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.151+0000 7f389333e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f388c072000 con 0x7f388c15a040 2023-10-30T14:14:35.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.151+0000 7f38910da640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f388c150630 0x7f388c0c56a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46566/0 (socket says 172.21.15.57:46566) 2023-10-30T14:14:35.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.151+0000 7f38910da640 1 -- 172.21.15.57:0/1114732587 learned_addr learned my addr 172.21.15.57:0/1114732587 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:35.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.151+0000 7f38908d9640 1 --2- 172.21.15.57:0/1114732587 >> v2:172.21.15.57:3300/0 conn(0x7f388c0c1430 0x7f388c159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:35.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.151+0000 7f3883fff640 1 -- 172.21.15.57:0/1114732587 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465449189 0 0) 0x7f388c157100 con 0x7f388c150630 2023-10-30T14:14:35.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.151+0000 7f38918db640 1 --2- 172.21.15.57:0/1114732587 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f388c15a040 0x7f388c0cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:35.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.151+0000 7f3883fff640 1 -- 172.21.15.57:0/1114732587 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3878003530 con 0x7f388c150630 2023-10-30T14:14:35.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.151+0000 7f38908d9640 1 -- 172.21.15.57:0/1114732587 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f388c15a040 msgr2=0x7f388c0cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:35.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.151+0000 7f38908d9640 1 --2- 172.21.15.57:0/1114732587 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f388c15a040 0x7f388c0cacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.151+0000 7f38908d9640 1 -- 172.21.15.57:0/1114732587 >> v1:172.21.15.57:6789/0 conn(0x7f388c150630 legacy=0x7f388c0c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:35.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.151+0000 7f38908d9640 1 -- 172.21.15.57:0/1114732587 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388c156510 con 0x7f388c0c1430 2023-10-30T14:14:35.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.151+0000 7f38918db640 1 --2- 172.21.15.57:0/1114732587 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f388c15a040 0x7f388c0cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:14:35.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.155+0000 7f38908d9640 1 --2- 172.21.15.57:0/1114732587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f388c0c1430 0x7f388c159b00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f387c0097c0 tx=0x7f387c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d535f1a0bc40a6bb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:35.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.155+0000 7f3883fff640 1 -- 172.21.15.57:0/1114732587 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c0ab600 con 0x7f388c0c1430 2023-10-30T14:14:35.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.155+0000 7f3883fff640 1 -- 172.21.15.57:0/1114732587 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387c0a7050 con 0x7f388c0c1430 2023-10-30T14:14:35.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.155+0000 7f3883fff640 1 -- 172.21.15.57:0/1114732587 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c0ab980 con 0x7f388c0c1430 2023-10-30T14:14:35.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.155+0000 7f389333e640 1 -- 172.21.15.57:0/1114732587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f388c0c1430 msgr2=0x7f388c159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:35.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.155+0000 7f389333e640 1 --2- 172.21.15.57:0/1114732587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f388c0c1430 0x7f388c159b00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f387c0097c0 tx=0x7f387c01c480 comp rx=0 tx=0).stop 2023-10-30T14:14:35.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.155+0000 7f389333e640 1 -- 172.21.15.57:0/1114732587 shutdown_connections 2023-10-30T14:14:35.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.155+0000 7f389333e640 1 --2- 172.21.15.57:0/1114732587 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f388c15a040 0x7f388c0cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.155+0000 7f389333e640 1 --2- 172.21.15.57:0/1114732587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f388c0c1430 0x7f388c159b00 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.155+0000 7f389333e640 1 -- 172.21.15.57:0/1114732587 >> 172.21.15.57:0/1114732587 conn(0x7f388c06da40 msgr2=0x7f388c06de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:35.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.155+0000 7f389333e640 1 -- 172.21.15.57:0/1114732587 shutdown_connections 2023-10-30T14:14:35.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.155+0000 7f389333e640 1 -- 172.21.15.57:0/1114732587 wait complete. 2023-10-30T14:14:35.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.155+0000 7f389333e640 1 Processor -- start 2023-10-30T14:14:35.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.155+0000 7f389333e640 1 -- start start 2023-10-30T14:14:35.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f389333e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f388c0c1430 0x7f388c160020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:35.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f389333e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f388c150630 0x7f388c0c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:35.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f389333e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f388c15a040 0x7f388c155580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:35.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f389333e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388c14e730 con 0x7f388c150630 2023-10-30T14:14:35.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f389333e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388c14e960 con 0x7f388c15a040 2023-10-30T14:14:35.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f389333e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f388c15ef80 con 0x7f388c0c1430 2023-10-30T14:14:35.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f38908d9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f388c150630 0x7f388c0c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:35.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f38908d9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f388c150630 0x7f388c0c9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39832/0 (socket says 172.21.15.57:39832) 2023-10-30T14:14:35.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f38908d9640 1 -- 172.21.15.57:0/1730512311 learned_addr learned my addr 172.21.15.57:0/1730512311 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:35.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f38918db640 1 --2- 172.21.15.57:0/1730512311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f388c15a040 0x7f388c155580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:35.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f38910da640 1 --2- 172.21.15.57:0/1730512311 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f388c0c1430 0x7f388c160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:35.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f38908d9640 1 -- 172.21.15.57:0/1730512311 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f388c0c1430 msgr2=0x7f388c160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:35.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f38908d9640 1 --2- 172.21.15.57:0/1730512311 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f388c0c1430 0x7f388c160020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f38908d9640 1 -- 172.21.15.57:0/1730512311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f388c15a040 msgr2=0x7f388c155580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:35.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f38908d9640 1 --2- 172.21.15.57:0/1730512311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f388c15a040 0x7f388c155580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f38908d9640 1 -- 172.21.15.57:0/1730512311 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387c009420 con 0x7f388c150630 2023-10-30T14:14:35.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f38908d9640 1 --2- 172.21.15.57:0/1730512311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f388c150630 0x7f388c0c9620 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f387c0ae040 tx=0x7f387c019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:35.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f3881ffb640 1 -- 172.21.15.57:0/1730512311 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c0b4430 con 0x7f388c150630 2023-10-30T14:14:35.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f3881ffb640 1 -- 172.21.15.57:0/1730512311 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387c0b4590 con 0x7f388c150630 2023-10-30T14:14:35.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f3881ffb640 1 -- 172.21.15.57:0/1730512311 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c0b4940 con 0x7f388c150630 2023-10-30T14:14:35.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f389333e640 1 -- 172.21.15.57:0/1730512311 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f388c15f260 con 0x7f388c150630 2023-10-30T14:14:35.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f389333e640 1 -- 172.21.15.57:0/1730512311 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f388c15f820 con 0x7f388c150630 2023-10-30T14:14:35.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.159+0000 7f389333e640 1 -- 172.21.15.57:0/1730512311 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f388c04e170 con 0x7f388c150630 2023-10-30T14:14:35.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.163+0000 7f3881ffb640 1 -- 172.21.15.57:0/1730512311 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f387c0b2040 con 0x7f388c150630 2023-10-30T14:14:35.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.163+0000 7f3881ffb640 1 --2- 172.21.15.57:0/1730512311 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f38700664a0 0x7f3870068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:35.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.163+0000 7f3881ffb640 1 -- 172.21.15.57:0/1730512311 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f387c11deb0 con 0x7f388c150630 2023-10-30T14:14:35.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.163+0000 7f38910da640 1 --2- 172.21.15.57:0/1730512311 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f38700664a0 0x7f3870068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:35.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.163+0000 7f38910da640 1 --2- 172.21.15.57:0/1730512311 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f38700664a0 0x7f3870068960 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f38880011f0 tx=0x7f3888009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:35.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.167+0000 7f3881ffb640 1 -- 172.21.15.57:0/1730512311 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f387c0eb2a0 con 0x7f388c150630 2023-10-30T14:14:35.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.319+0000 7f389333e640 1 -- 172.21.15.57:0/1730512311 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f388c156ab0 con 0x7f388c150630 2023-10-30T14:14:35.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.323+0000 7f3881ffb640 1 -- 172.21.15.57:0/1730512311 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f387c0ee8c0 con 0x7f388c150630 2023-10-30T14:14:35.328 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-30T14:14:35.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.323+0000 7f389333e640 1 -- 172.21.15.57:0/1730512311 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f38700664a0 msgr2=0x7f3870068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:35.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.323+0000 7f389333e640 1 --2- 172.21.15.57:0/1730512311 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f38700664a0 0x7f3870068960 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f38880011f0 tx=0x7f3888009040 comp rx=0 tx=0).stop 2023-10-30T14:14:35.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.323+0000 7f389333e640 1 -- 172.21.15.57:0/1730512311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f388c150630 msgr2=0x7f388c0c9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:35.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.323+0000 7f389333e640 1 --2- 172.21.15.57:0/1730512311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f388c150630 0x7f388c0c9620 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f387c0ae040 tx=0x7f387c019220 comp rx=0 tx=0).stop 2023-10-30T14:14:35.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.323+0000 7f389333e640 1 -- 172.21.15.57:0/1730512311 shutdown_connections 2023-10-30T14:14:35.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.323+0000 7f389333e640 1 --2- 172.21.15.57:0/1730512311 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f38700664a0 0x7f3870068960 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.323+0000 7f389333e640 1 --2- 172.21.15.57:0/1730512311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f388c15a040 0x7f388c155580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.323+0000 7f389333e640 1 --2- 172.21.15.57:0/1730512311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f388c150630 0x7f388c0c9620 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.323+0000 7f389333e640 1 --2- 172.21.15.57:0/1730512311 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f388c0c1430 0x7f388c160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.323+0000 7f389333e640 1 -- 172.21.15.57:0/1730512311 >> 172.21.15.57:0/1730512311 conn(0x7f388c06da40 msgr2=0x7f388c0c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:35.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.323+0000 7f389333e640 1 -- 172.21.15.57:0/1730512311 shutdown_connections 2023-10-30T14:14:35.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.323+0000 7f389333e640 1 -- 172.21.15.57:0/1730512311 wait complete. 2023-10-30T14:14:35.348 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T14:14:35.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.487+0000 7f8d876d6640 1 Processor -- start 2023-10-30T14:14:35.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.487+0000 7f8d876d6640 1 -- start start 2023-10-30T14:14:35.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.491+0000 7f8d876d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d80153ba0 0x7f8d800cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:35.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.491+0000 7f8d876d6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8d800c4410 0x7f8d8015a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:35.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.491+0000 7f8d876d6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d80153f80 con 0x7f8d800c4810 2023-10-30T14:14:35.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.491+0000 7f8d876d6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d80159df0 con 0x7f8d800c4410 2023-10-30T14:14:35.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.491+0000 7f8d876d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8015a7c0 con 0x7f8d80153ba0 2023-10-30T14:14:35.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.491+0000 7f8d85c73640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f8d800c4810 0x7f8d8015cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46576/0 (socket says 172.21.15.57:46576) 2023-10-30T14:14:35.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.491+0000 7f8d85c73640 1 -- 172.21.15.57:0/2008604017 learned_addr learned my addr 172.21.15.57:0/2008604017 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:35.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.491+0000 7f8d84c71640 1 --2- 172.21.15.57:0/2008604017 >> v2:172.21.15.57:3300/0 conn(0x7f8d800c4410 0x7f8d8015a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:35.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.491+0000 7f8d85472640 1 --2- 172.21.15.57:0/2008604017 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d80153ba0 0x7f8d800cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:35.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.491+0000 7f8d77fff640 1 -- 172.21.15.57:0/2008604017 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990472128 0 0) 0x7f8d80153f80 con 0x7f8d800c4810 2023-10-30T14:14:35.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.491+0000 7f8d77fff640 1 -- 172.21.15.57:0/2008604017 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d6c003530 con 0x7f8d800c4810 2023-10-30T14:14:35.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.491+0000 7f8d84c71640 1 -- 172.21.15.57:0/2008604017 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d80153ba0 msgr2=0x7f8d800cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:35.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.491+0000 7f8d84c71640 1 --2- 172.21.15.57:0/2008604017 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d80153ba0 0x7f8d800cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.491+0000 7f8d84c71640 1 -- 172.21.15.57:0/2008604017 >> v1:172.21.15.57:6789/0 conn(0x7f8d800c4810 legacy=0x7f8d8015cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:35.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.491+0000 7f8d84c71640 1 -- 172.21.15.57:0/2008604017 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d801597c0 con 0x7f8d800c4410 2023-10-30T14:14:35.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.495+0000 7f8d84c71640 1 --2- 172.21.15.57:0/2008604017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d800c4410 0x7f8d8015a280 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8d700097c0 tx=0x7f8d7001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1fa9e8751de1b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:35.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.495+0000 7f8d77fff640 1 -- 172.21.15.57:0/2008604017 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d700ac620 con 0x7f8d800c4410 2023-10-30T14:14:35.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.495+0000 7f8d77fff640 1 -- 172.21.15.57:0/2008604017 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d700ac780 con 0x7f8d800c4410 2023-10-30T14:14:35.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.495+0000 7f8d77fff640 1 -- 172.21.15.57:0/2008604017 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d700acb10 con 0x7f8d800c4410 2023-10-30T14:14:35.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.495+0000 7f8d876d6640 1 -- 172.21.15.57:0/2008604017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d800c4410 msgr2=0x7f8d8015a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:35.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.495+0000 7f8d876d6640 1 --2- 172.21.15.57:0/2008604017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d800c4410 0x7f8d8015a280 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8d700097c0 tx=0x7f8d7001c3c0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.495+0000 7f8d876d6640 1 -- 172.21.15.57:0/2008604017 shutdown_connections 2023-10-30T14:14:35.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.495+0000 7f8d876d6640 1 --2- 172.21.15.57:0/2008604017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d800c4410 0x7f8d8015a280 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.495+0000 7f8d876d6640 1 --2- 172.21.15.57:0/2008604017 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d80153ba0 0x7f8d800cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.495+0000 7f8d876d6640 1 -- 172.21.15.57:0/2008604017 >> 172.21.15.57:0/2008604017 conn(0x7f8d80071790 msgr2=0x7f8d80071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:35.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.495+0000 7f8d876d6640 1 -- 172.21.15.57:0/2008604017 shutdown_connections 2023-10-30T14:14:35.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.495+0000 7f8d876d6640 1 -- 172.21.15.57:0/2008604017 wait complete. 2023-10-30T14:14:35.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.495+0000 7f8d876d6640 1 Processor -- start 2023-10-30T14:14:35.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.495+0000 7f8d876d6640 1 -- start start 2023-10-30T14:14:35.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d876d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d800c4410 0x7f8d80144750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:35.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d876d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d800c4810 0x7f8d801f2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:35.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d876d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d80153ba0 0x7f8d80143480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:35.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d876d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8014f490 con 0x7f8d800c4810 2023-10-30T14:14:35.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d876d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d801533a0 con 0x7f8d80153ba0 2023-10-30T14:14:35.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d876d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d801f2890 con 0x7f8d800c4410 2023-10-30T14:14:35.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d84c71640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d800c4810 0x7f8d801f2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:35.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d84c71640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d800c4810 0x7f8d801f2d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39852/0 (socket says 172.21.15.57:39852) 2023-10-30T14:14:35.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d84c71640 1 -- 172.21.15.57:0/4067100092 learned_addr learned my addr 172.21.15.57:0/4067100092 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:35.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d85472640 1 --2- 172.21.15.57:0/4067100092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d800c4410 0x7f8d80144750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:35.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d85c73640 1 --2- 172.21.15.57:0/4067100092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d80153ba0 0x7f8d80143480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:35.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d84c71640 1 -- 172.21.15.57:0/4067100092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d800c4410 msgr2=0x7f8d80144750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:35.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d84c71640 1 --2- 172.21.15.57:0/4067100092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d800c4410 0x7f8d80144750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d84c71640 1 -- 172.21.15.57:0/4067100092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d80153ba0 msgr2=0x7f8d80143480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:35.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d84c71640 1 --2- 172.21.15.57:0/4067100092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d80153ba0 0x7f8d80143480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d84c71640 1 -- 172.21.15.57:0/4067100092 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d70009420 con 0x7f8d800c4810 2023-10-30T14:14:35.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d84c71640 1 --2- 172.21.15.57:0/4067100092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d800c4810 0x7f8d801f2d20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8d70019ff0 tx=0x7f8d70019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:35.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d75ffb640 1 -- 172.21.15.57:0/4067100092 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d700ac620 con 0x7f8d800c4810 2023-10-30T14:14:35.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d75ffb640 1 -- 172.21.15.57:0/4067100092 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d700b5410 con 0x7f8d800c4810 2023-10-30T14:14:35.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d75ffb640 1 -- 172.21.15.57:0/4067100092 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d700b57a0 con 0x7f8d800c4810 2023-10-30T14:14:35.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d876d6640 1 -- 172.21.15.57:0/4067100092 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d801f2260 con 0x7f8d800c4810 2023-10-30T14:14:35.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d876d6640 1 -- 172.21.15.57:0/4067100092 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d80146530 con 0x7f8d800c4810 2023-10-30T14:14:35.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d876d6640 1 -- 172.21.15.57:0/4067100092 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d8004e170 con 0x7f8d800c4810 2023-10-30T14:14:35.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.499+0000 7f8d75ffb640 1 -- 172.21.15.57:0/4067100092 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8d700b3040 con 0x7f8d800c4810 2023-10-30T14:14:35.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.503+0000 7f8d75ffb640 1 --2- 172.21.15.57:0/4067100092 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8d640664a0 0x7f8d64068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:35.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.503+0000 7f8d75ffb640 1 -- 172.21.15.57:0/4067100092 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d7011de20 con 0x7f8d800c4810 2023-10-30T14:14:35.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.503+0000 7f8d85472640 1 --2- 172.21.15.57:0/4067100092 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8d640664a0 0x7f8d64068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:35.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.503+0000 7f8d85472640 1 --2- 172.21.15.57:0/4067100092 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8d640664a0 0x7f8d64068960 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c009a50 tx=0x7f8d7c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:35.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.507+0000 7f8d75ffb640 1 -- 172.21.15.57:0/4067100092 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8d700eb2d0 con 0x7f8d800c4810 2023-10-30T14:14:35.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.663+0000 7f8d876d6640 1 -- 172.21.15.57:0/4067100092 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8d800b1dc0 con 0x7f8d800c4810 2023-10-30T14:14:35.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.663+0000 7f8d75ffb640 1 -- 172.21.15.57:0/4067100092 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8d700ee8f0 con 0x7f8d800c4810 2023-10-30T14:14:35.672 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-30T14:14:35.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.667+0000 7f8d876d6640 1 -- 172.21.15.57:0/4067100092 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8d640664a0 msgr2=0x7f8d64068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:35.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.667+0000 7f8d876d6640 1 --2- 172.21.15.57:0/4067100092 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8d640664a0 0x7f8d64068960 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c009a50 tx=0x7f8d7c008040 comp rx=0 tx=0).stop 2023-10-30T14:14:35.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.667+0000 7f8d876d6640 1 -- 172.21.15.57:0/4067100092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d800c4810 msgr2=0x7f8d801f2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:35.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.667+0000 7f8d876d6640 1 --2- 172.21.15.57:0/4067100092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d800c4810 0x7f8d801f2d20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8d70019ff0 tx=0x7f8d70019160 comp rx=0 tx=0).stop 2023-10-30T14:14:35.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.667+0000 7f8d876d6640 1 -- 172.21.15.57:0/4067100092 shutdown_connections 2023-10-30T14:14:35.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.667+0000 7f8d876d6640 1 --2- 172.21.15.57:0/4067100092 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8d640664a0 0x7f8d64068960 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.667+0000 7f8d876d6640 1 --2- 172.21.15.57:0/4067100092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d80153ba0 0x7f8d80143480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.667+0000 7f8d876d6640 1 --2- 172.21.15.57:0/4067100092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d800c4810 0x7f8d801f2d20 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.667+0000 7f8d876d6640 1 --2- 172.21.15.57:0/4067100092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d800c4410 0x7f8d80144750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.667+0000 7f8d876d6640 1 -- 172.21.15.57:0/4067100092 >> 172.21.15.57:0/4067100092 conn(0x7f8d80071790 msgr2=0x7f8d80147090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:35.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.667+0000 7f8d876d6640 1 -- 172.21.15.57:0/4067100092 shutdown_connections 2023-10-30T14:14:35.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.667+0000 7f8d876d6640 1 -- 172.21.15.57:0/4067100092 wait complete. 2023-10-30T14:14:35.693 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:35.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.831+0000 7f310d7d3640 1 Processor -- start 2023-10-30T14:14:35.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.831+0000 7f310d7d3640 1 -- start start 2023-10-30T14:14:35.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.831+0000 7f310d7d3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f310815cf90 0x7f31080c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:35.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.831+0000 7f310d7d3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3108159f20 con 0x7f31080c4a10 2023-10-30T14:14:35.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.831+0000 7f310d7d3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31080c47e0 con 0x7f3108153c80 2023-10-30T14:14:35.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.831+0000 7f310d7d3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f310815dec0 con 0x7f310815cf90 2023-10-30T14:14:35.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.831+0000 7f30fffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f31080c4a10 0x7f31080ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46582/0 (socket says 172.21.15.57:46582) 2023-10-30T14:14:35.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.831+0000 7f30fffff640 1 -- 172.21.15.57:0/729811159 learned_addr learned my addr 172.21.15.57:0/729811159 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:35.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.831+0000 7f31077fe640 1 --2- 172.21.15.57:0/729811159 >> v2:172.21.15.57:3300/0 conn(0x7f310815cf90 0x7f31080c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:35.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.831+0000 7f31067fc640 1 -- 172.21.15.57:0/729811159 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064611335 0 0) 0x7f3108159f20 con 0x7f31080c4a10 2023-10-30T14:14:35.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.831+0000 7f31067fc640 1 -- 172.21.15.57:0/729811159 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f4003530 con 0x7f31080c4a10 2023-10-30T14:14:35.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.831+0000 7f31077fe640 1 -- 172.21.15.57:0/729811159 >> v1:172.21.15.88:6789/0 conn(0x7f3108153c80 legacy=0x7f310815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:35.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.831+0000 7f31077fe640 1 -- 172.21.15.57:0/729811159 >> v1:172.21.15.57:6789/0 conn(0x7f31080c4a10 legacy=0x7f31080ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:35.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.831+0000 7f31077fe640 1 -- 172.21.15.57:0/729811159 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310815d620 con 0x7f310815cf90 2023-10-30T14:14:35.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.835+0000 7f31077fe640 1 --2- 172.21.15.57:0/729811159 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f310815cf90 0x7f31080c42a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3100009d90 tx=0x7f310001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfadf4530c527e82 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:35.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.835+0000 7f31067fc640 1 -- 172.21.15.57:0/729811159 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31000af600 con 0x7f310815cf90 2023-10-30T14:14:35.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.835+0000 7f31067fc640 1 -- 172.21.15.57:0/729811159 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31000ab050 con 0x7f310815cf90 2023-10-30T14:14:35.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.835+0000 7f31067fc640 1 -- 172.21.15.57:0/729811159 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31000af980 con 0x7f310815cf90 2023-10-30T14:14:35.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.835+0000 7f310d7d3640 1 -- 172.21.15.57:0/729811159 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f310815cf90 msgr2=0x7f31080c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:35.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.835+0000 7f310d7d3640 1 --2- 172.21.15.57:0/729811159 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f310815cf90 0x7f31080c42a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3100009d90 tx=0x7f310001f130 comp rx=0 tx=0).stop 2023-10-30T14:14:35.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.835+0000 7f310d7d3640 1 -- 172.21.15.57:0/729811159 shutdown_connections 2023-10-30T14:14:35.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.835+0000 7f310d7d3640 1 --2- 172.21.15.57:0/729811159 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f310815cf90 0x7f31080c42a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.835+0000 7f310d7d3640 1 -- 172.21.15.57:0/729811159 >> 172.21.15.57:0/729811159 conn(0x7f31080715c0 msgr2=0x7f31080719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:35.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.835+0000 7f310d7d3640 1 -- 172.21.15.57:0/729811159 shutdown_connections 2023-10-30T14:14:35.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.835+0000 7f310d7d3640 1 -- 172.21.15.57:0/729811159 wait complete. 2023-10-30T14:14:35.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f310d7d3640 1 Processor -- start 2023-10-30T14:14:35.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f310d7d3640 1 -- start start 2023-10-30T14:14:35.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f310d7d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f31080c4a10 0x7f31081ef620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:35.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f310d7d3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3108153c80 0x7f310814ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:35.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f310d7d3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f310815cf90 0x7f310814ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:35.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f310d7d3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31081599c0 con 0x7f310815cf90 2023-10-30T14:14:35.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f310d7d3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31081f05c0 con 0x7f3108153c80 2023-10-30T14:14:35.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f310d7d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31081efd20 con 0x7f31080c4a10 2023-10-30T14:14:35.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f31077fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f310815cf90 0x7f310814ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:35.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f31077fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f310815cf90 0x7f310814ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39864/0 (socket says 172.21.15.57:39864) 2023-10-30T14:14:35.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f31077fe640 1 -- 172.21.15.57:0/819635251 learned_addr learned my addr 172.21.15.57:0/819635251 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:35.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f30fffff640 1 --2- 172.21.15.57:0/819635251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3108153c80 0x7f310814ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:35.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f3106ffd640 1 --2- 172.21.15.57:0/819635251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f31080c4a10 0x7f31081ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:35.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f31077fe640 1 -- 172.21.15.57:0/819635251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f31080c4a10 msgr2=0x7f31081ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:35.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f31077fe640 1 --2- 172.21.15.57:0/819635251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f31080c4a10 0x7f31081ef620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f31077fe640 1 -- 172.21.15.57:0/819635251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3108153c80 msgr2=0x7f310814ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:35.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f31077fe640 1 --2- 172.21.15.57:0/819635251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3108153c80 0x7f310814ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:35.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f31077fe640 1 -- 172.21.15.57:0/819635251 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3100009a70 con 0x7f310815cf90 2023-10-30T14:14:35.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f3106ffd640 1 --2- 172.21.15.57:0/819635251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f31080c4a10 0x7f31081ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:35.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f30fffff640 1 --2- 172.21.15.57:0/819635251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3108153c80 0x7f310814ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:35.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f31077fe640 1 --2- 172.21.15.57:0/819635251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f310815cf90 0x7f310814ce40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f31000b2040 tx=0x7f310001bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:35.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f30ff7fe640 1 -- 172.21.15.57:0/819635251 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3100009450 con 0x7f310815cf90 2023-10-30T14:14:35.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f30ff7fe640 1 -- 172.21.15.57:0/819635251 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31000ba480 con 0x7f310815cf90 2023-10-30T14:14:35.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f30ff7fe640 1 -- 172.21.15.57:0/819635251 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31000ba830 con 0x7f310815cf90 2023-10-30T14:14:35.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.839+0000 7f310d7d3640 1 -- 172.21.15.57:0/819635251 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31081f0000 con 0x7f310815cf90 2023-10-30T14:14:35.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.843+0000 7f310d7d3640 1 -- 172.21.15.57:0/819635251 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f310814c640 con 0x7f310815cf90 2023-10-30T14:14:35.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.843+0000 7f310d7d3640 1 -- 172.21.15.57:0/819635251 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f310804e170 con 0x7f310815cf90 2023-10-30T14:14:35.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.843+0000 7f30ff7fe640 1 -- 172.21.15.57:0/819635251 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f31000b6040 con 0x7f310815cf90 2023-10-30T14:14:35.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.843+0000 7f30ff7fe640 1 --2- 172.21.15.57:0/819635251 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f30e80664d0 0x7f30e8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:35.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.843+0000 7f30ff7fe640 1 -- 172.21.15.57:0/819635251 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31001227d0 con 0x7f310815cf90 2023-10-30T14:14:35.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.843+0000 7f3106ffd640 1 --2- 172.21.15.57:0/819635251 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f30e80664d0 0x7f30e8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:35.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.847+0000 7f3106ffd640 1 --2- 172.21.15.57:0/819635251 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f30e80664d0 0x7f30e8068990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f30f8000f70 tx=0x7f30f8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:35.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:35.847+0000 7f30ff7fe640 1 -- 172.21.15.57:0/819635251 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f31000efbb0 con 0x7f310815cf90 2023-10-30T14:14:36.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.035+0000 7f310d7d3640 1 -- 172.21.15.57:0/819635251 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3108153290 con 0x7f310815cf90 2023-10-30T14:14:36.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.035+0000 7f30ff7fe640 1 -- 172.21.15.57:0/819635251 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7f31000b4090 con 0x7f310815cf90 2023-10-30T14:14:36.043 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:36.043 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:14:20.837447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6837/901968115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":901968115},{"type":"v1","addr":"172.21.15.57:6837","nonce":901968115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:36.044 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 12 2023-10-30T14:14:36.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.039+0000 7f310d7d3640 1 -- 172.21.15.57:0/819635251 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f30e80664d0 msgr2=0x7f30e8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:36.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.039+0000 7f310d7d3640 1 --2- 172.21.15.57:0/819635251 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f30e80664d0 0x7f30e8068990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f30f8000f70 tx=0x7f30f8009040 comp rx=0 tx=0).stop 2023-10-30T14:14:36.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.039+0000 7f310d7d3640 1 -- 172.21.15.57:0/819635251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f310815cf90 msgr2=0x7f310814ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:36.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.039+0000 7f310d7d3640 1 --2- 172.21.15.57:0/819635251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f310815cf90 0x7f310814ce40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f31000b2040 tx=0x7f310001bed0 comp rx=0 tx=0).stop 2023-10-30T14:14:36.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.039+0000 7f310d7d3640 1 -- 172.21.15.57:0/819635251 shutdown_connections 2023-10-30T14:14:36.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.039+0000 7f310d7d3640 1 --2- 172.21.15.57:0/819635251 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f30e80664d0 0x7f30e8068990 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:36.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.039+0000 7f310d7d3640 1 --2- 172.21.15.57:0/819635251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f310815cf90 0x7f310814ce40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:36.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.039+0000 7f310d7d3640 1 --2- 172.21.15.57:0/819635251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3108153c80 0x7f310814ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:36.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.039+0000 7f310d7d3640 1 --2- 172.21.15.57:0/819635251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f31080c4a10 0x7f31081ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:36.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.039+0000 7f310d7d3640 1 -- 172.21.15.57:0/819635251 >> 172.21.15.57:0/819635251 conn(0x7f31080715c0 msgr2=0x7f31080c9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:36.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.039+0000 7f310d7d3640 1 -- 172.21.15.57:0/819635251 shutdown_connections 2023-10-30T14:14:36.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.039+0000 7f310d7d3640 1 -- 172.21.15.57:0/819635251 wait complete. 2023-10-30T14:14:36.065 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:14:36.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.199+0000 7f17a59d3640 1 Processor -- start 2023-10-30T14:14:36.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.203+0000 7f17a59d3640 1 -- start start 2023-10-30T14:14:36.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.203+0000 7f17a59d3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f17a015cf90 0x7f17a00c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:36.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.203+0000 7f17a59d3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a0159f20 con 0x7f17a00c4a10 2023-10-30T14:14:36.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.203+0000 7f17a59d3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a00c47e0 con 0x7f17a0153c80 2023-10-30T14:14:36.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.203+0000 7f17a59d3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a015dec0 con 0x7f17a015cf90 2023-10-30T14:14:36.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.203+0000 7f179e7fc640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f17a00c4a10 0x7f17a00ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46588/0 (socket says 172.21.15.57:46588) 2023-10-30T14:14:36.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.203+0000 7f179e7fc640 1 -- 172.21.15.57:0/2720789848 learned_addr learned my addr 172.21.15.57:0/2720789848 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:36.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.203+0000 7f179f7fe640 1 --2- 172.21.15.57:0/2720789848 >> v2:172.21.15.57:3300/0 conn(0x7f17a015cf90 0x7f17a00c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:36.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.203+0000 7f179dffb640 1 -- 172.21.15.57:0/2720789848 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719540275 0 0) 0x7f17a0159f20 con 0x7f17a00c4a10 2023-10-30T14:14:36.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.203+0000 7f179dffb640 1 -- 172.21.15.57:0/2720789848 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1788003530 con 0x7f17a00c4a10 2023-10-30T14:14:36.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.203+0000 7f179f7fe640 1 -- 172.21.15.57:0/2720789848 >> v1:172.21.15.88:6789/0 conn(0x7f17a0153c80 legacy=0x7f17a015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:36.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.203+0000 7f179f7fe640 1 -- 172.21.15.57:0/2720789848 >> v1:172.21.15.57:6789/0 conn(0x7f17a00c4a10 legacy=0x7f17a00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:36.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.203+0000 7f179f7fe640 1 -- 172.21.15.57:0/2720789848 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a015d620 con 0x7f17a015cf90 2023-10-30T14:14:36.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.207+0000 7f179f7fe640 1 --2- 172.21.15.57:0/2720789848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17a015cf90 0x7f17a00c42a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1798009d90 tx=0x7f179801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19867038ac49f750 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:36.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.207+0000 7f179dffb640 1 -- 172.21.15.57:0/2720789848 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17980af600 con 0x7f17a015cf90 2023-10-30T14:14:36.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.207+0000 7f179dffb640 1 -- 172.21.15.57:0/2720789848 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17980ab050 con 0x7f17a015cf90 2023-10-30T14:14:36.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.207+0000 7f179dffb640 1 -- 172.21.15.57:0/2720789848 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17980af980 con 0x7f17a015cf90 2023-10-30T14:14:36.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.207+0000 7f17a59d3640 1 -- 172.21.15.57:0/2720789848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17a015cf90 msgr2=0x7f17a00c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:36.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.207+0000 7f17a59d3640 1 --2- 172.21.15.57:0/2720789848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17a015cf90 0x7f17a00c42a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1798009d90 tx=0x7f179801f130 comp rx=0 tx=0).stop 2023-10-30T14:14:36.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.207+0000 7f17a59d3640 1 -- 172.21.15.57:0/2720789848 shutdown_connections 2023-10-30T14:14:36.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.207+0000 7f17a59d3640 1 --2- 172.21.15.57:0/2720789848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17a015cf90 0x7f17a00c42a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:36.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.207+0000 7f17a59d3640 1 -- 172.21.15.57:0/2720789848 >> 172.21.15.57:0/2720789848 conn(0x7f17a00715c0 msgr2=0x7f17a00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:36.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.207+0000 7f17a59d3640 1 -- 172.21.15.57:0/2720789848 shutdown_connections 2023-10-30T14:14:36.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.207+0000 7f17a59d3640 1 -- 172.21.15.57:0/2720789848 wait complete. 2023-10-30T14:14:36.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.207+0000 7f17a59d3640 1 Processor -- start 2023-10-30T14:14:36.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.207+0000 7f17a59d3640 1 -- start start 2023-10-30T14:14:36.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f17a59d3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17a00c4a10 0x7f17a014ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:36.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f17a59d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17a0153c80 0x7f17a00b3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:36.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f17a59d3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17a015cf90 0x7f17a00b3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:36.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f17a59d3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a00b4210 con 0x7f17a00c4a10 2023-10-30T14:14:36.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f17a59d3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a00b3340 con 0x7f17a015cf90 2023-10-30T14:14:36.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f17a59d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a014d400 con 0x7f17a0153c80 2023-10-30T14:14:36.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f179effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17a00c4a10 0x7f17a014ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:36.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f179effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17a00c4a10 0x7f17a014ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39882/0 (socket says 172.21.15.57:39882) 2023-10-30T14:14:36.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f179effd640 1 -- 172.21.15.57:0/643674081 learned_addr learned my addr 172.21.15.57:0/643674081 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:36.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f179f7fe640 1 --2- 172.21.15.57:0/643674081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17a015cf90 0x7f17a00b3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:36.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f179e7fc640 1 --2- 172.21.15.57:0/643674081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17a0153c80 0x7f17a00b3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:36.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f179effd640 1 -- 172.21.15.57:0/643674081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17a0153c80 msgr2=0x7f17a00b3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:36.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f179effd640 1 --2- 172.21.15.57:0/643674081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17a0153c80 0x7f17a00b3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:36.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f179effd640 1 -- 172.21.15.57:0/643674081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17a015cf90 msgr2=0x7f17a00b3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:36.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f179effd640 1 --2- 172.21.15.57:0/643674081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17a015cf90 0x7f17a00b3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:36.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f179effd640 1 -- 172.21.15.57:0/643674081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1798009a70 con 0x7f17a00c4a10 2023-10-30T14:14:36.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f179effd640 1 --2- 172.21.15.57:0/643674081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17a00c4a10 0x7f17a014ac10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f179000c500 tx=0x7f179000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:36.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f1787fff640 1 -- 172.21.15.57:0/643674081 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1790005220 con 0x7f17a00c4a10 2023-10-30T14:14:36.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f1787fff640 1 -- 172.21.15.57:0/643674081 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1790005380 con 0x7f17a00c4a10 2023-10-30T14:14:36.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f17a59d3640 1 -- 172.21.15.57:0/643674081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a014d6a0 con 0x7f17a00c4a10 2023-10-30T14:14:36.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f1787fff640 1 -- 172.21.15.57:0/643674081 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1790007270 con 0x7f17a00c4a10 2023-10-30T14:14:36.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f17a59d3640 1 -- 172.21.15.57:0/643674081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a00b54d0 con 0x7f17a00c4a10 2023-10-30T14:14:36.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f17a59d3640 1 -- 172.21.15.57:0/643674081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a004e170 con 0x7f17a00c4a10 2023-10-30T14:14:36.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.211+0000 7f1787fff640 1 -- 172.21.15.57:0/643674081 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f17900073d0 con 0x7f17a00c4a10 2023-10-30T14:14:36.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.215+0000 7f1787fff640 1 --2- 172.21.15.57:0/643674081 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f178c0664a0 0x7f178c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:36.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.215+0000 7f1787fff640 1 -- 172.21.15.57:0/643674081 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1790083850 con 0x7f17a00c4a10 2023-10-30T14:14:36.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.215+0000 7f179e7fc640 1 --2- 172.21.15.57:0/643674081 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f178c0664a0 0x7f178c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:36.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.215+0000 7f179e7fc640 1 --2- 172.21.15.57:0/643674081 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f178c0664a0 0x7f178c068960 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f17a0152aa0 tx=0x7f1794009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:36.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.219+0000 7f1787fff640 1 -- 172.21.15.57:0/643674081 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1790050c40 con 0x7f17a00c4a10 2023-10-30T14:14:36.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.375+0000 7f17a59d3640 1 -- 172.21.15.57:0/643674081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17a00713a0 con 0x7f17a00c4a10 2023-10-30T14:14:36.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.375+0000 7f1787fff640 1 -- 172.21.15.57:0/643674081 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16433 (secure 0 0 0) 0x7f1790054260 con 0x7f17a00c4a10 2023-10-30T14:14:36.382 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:36.382 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:14:05.576818+0000","last_up_change":"2023-10-30T14:13:51.171287+0000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T14:13:52.574639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T14:14:02.600122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T14:14:03.069429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6801","nonce":3287367848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6803","nonce":3287367848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6807","nonce":3287367848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6805","nonce":3287367848}]},"public_addr":"172.21.15.57:6801/3287367848","cluster_addr":"172.21.15.57:6803/3287367848","heartbeat_back_addr":"172.21.15.57:6807/3287367848","heartbeat_front_addr":"172.21.15.57:6805/3287367848","state":["exists","up"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6809","nonce":3340797836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6811","nonce":3340797836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6815","nonce":3340797836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6813","nonce":3340797836}]},"public_addr":"172.21.15.57:6809/3340797836","cluster_addr":"172.21.15.57:6811/3340797836","heartbeat_back_addr":"172.21.15.57:6815/3340797836","heartbeat_front_addr":"172.21.15.57:6813/3340797836","state":["exists","up"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6825","nonce":1700292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6827","nonce":1700292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6831","nonce":1700292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6829","nonce":1700292}]},"public_addr":"172.21.15.57:6825/1700292","cluster_addr":"172.21.15.57:6827/1700292","heartbeat_back_addr":"172.21.15.57:6831/1700292","heartbeat_front_addr":"172.21.15.57:6829/1700292","state":["exists","up"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6817","nonce":2893420631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6819","nonce":2893420631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6823","nonce":2893420631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6821","nonce":2893420631}]},"public_addr":"172.21.15.57:6817/2893420631","cluster_addr":"172.21.15.57:6819/2893420631","heartbeat_back_addr":"172.21.15.57:6823/2893420631","heartbeat_front_addr":"172.21.15.57:6821/2893420631","state":["exists","up"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6807","nonce":2784816485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6811","nonce":2784816485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6815","nonce":2784816485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6813","nonce":2784816485}]},"public_addr":"172.21.15.88:6807/2784816485","cluster_addr":"172.21.15.88:6811/2784816485","heartbeat_back_addr":"172.21.15.88:6815/2784816485","heartbeat_front_addr":"172.21.15.88:6813/2784816485","state":["exists","up"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6801","nonce":2682449812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6803","nonce":2682449812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6810","nonce":2682449812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6806","nonce":2682449812}]},"public_addr":"172.21.15.88:6801/2682449812","cluster_addr":"172.21.15.88:6803/2682449812","heartbeat_back_addr":"172.21.15.88:6810/2682449812","heartbeat_front_addr":"172.21.15.88:6806/2682449812","state":["exists","up"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6822","nonce":3340693897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6827","nonce":3340693897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6831","nonce":3340693897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6829","nonce":3340693897}]},"public_addr":"172.21.15.88:6822/3340693897","cluster_addr":"172.21.15.88:6827/3340693897","heartbeat_back_addr":"172.21.15.88:6831/3340693897","heartbeat_front_addr":"172.21.15.88:6829/3340693897","state":["exists","up"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6817","nonce":2862111199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6820","nonce":2862111199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6825","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6826","nonce":2862111199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6821","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6823","nonce":2862111199}]},"public_addr":"172.21.15.88:6817/2862111199","cluster_addr":"172.21.15.88:6820/2862111199","heartbeat_back_addr":"172.21.15.88:6826/2862111199","heartbeat_front_addr":"172.21.15.88:6823/2862111199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.634920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.773993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.725825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.761452+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.694274+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.650718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:14:36.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.379+0000 7f17a59d3640 1 -- 172.21.15.57:0/643674081 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f178c0664a0 msgr2=0x7f178c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:36.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.379+0000 7f17a59d3640 1 --2- 172.21.15.57:0/643674081 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f178c0664a0 0x7f178c068960 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f17a0152aa0 tx=0x7f1794009040 comp rx=0 tx=0).stop 2023-10-30T14:14:36.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.379+0000 7f17a59d3640 1 -- 172.21.15.57:0/643674081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17a00c4a10 msgr2=0x7f17a014ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:36.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.379+0000 7f17a59d3640 1 --2- 172.21.15.57:0/643674081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17a00c4a10 0x7f17a014ac10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f179000c500 tx=0x7f179000c9c0 comp rx=0 tx=0).stop 2023-10-30T14:14:36.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.379+0000 7f17a59d3640 1 -- 172.21.15.57:0/643674081 shutdown_connections 2023-10-30T14:14:36.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.379+0000 7f17a59d3640 1 --2- 172.21.15.57:0/643674081 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f178c0664a0 0x7f178c068960 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:36.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.379+0000 7f17a59d3640 1 --2- 172.21.15.57:0/643674081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17a015cf90 0x7f17a00b3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:36.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.379+0000 7f17a59d3640 1 --2- 172.21.15.57:0/643674081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17a0153c80 0x7f17a00b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:36.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.379+0000 7f17a59d3640 1 --2- 172.21.15.57:0/643674081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17a00c4a10 0x7f17a014ac10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:36.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.379+0000 7f17a59d3640 1 -- 172.21.15.57:0/643674081 >> 172.21.15.57:0/643674081 conn(0x7f17a00715c0 msgr2=0x7f17a00c9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:36.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.379+0000 7f17a59d3640 1 -- 172.21.15.57:0/643674081 shutdown_connections 2023-10-30T14:14:36.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:36.379+0000 7f17a59d3640 1 -- 172.21.15.57:0/643674081 wait complete. 2023-10-30T14:14:36.404 DEBUG:tasks.mds_thrash:Yielding 2023-10-30T14:14:36.404 INFO:teuthology.run_tasks:Running task workunit... 2023-10-30T14:14:36.416 INFO:tasks.workunit:Pulling workunits from ref 00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T14:14:36.417 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-30T14:14:36.417 INFO:tasks.workunit:timeout=6h 2023-10-30T14:14:36.418 INFO:tasks.workunit:cleanup=True 2023-10-30T14:14:36.418 DEBUG:teuthology.orchestra.run.smithi057:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-30T14:14:36.418 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2023-10-30T14:14:36.419 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before thrashing 2023-10-30T14:14:36.429 INFO:teuthology.orchestra.run.smithi057.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-10-30T14:14:36.430 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-10-30T14:14:36.430 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 2bh/43d Inode: 1 Links: 2 2023-10-30T14:14:36.430 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T14:14:36.430 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-10-30T14:14:36.431 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-10-30 14:14:09.704534605 +0000 2023-10-30T14:14:36.431 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-10-30 14:14:28.872113162 +0000 2023-10-30T14:14:36.431 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2023-10-30 14:14:09.704534605 +0000 2023-10-30T14:14:36.431 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-10-30T14:14:36.432 DEBUG:teuthology.orchestra.run.smithi057:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-10-30T14:14:36.505 DEBUG:teuthology.orchestra.run.smithi057:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T14:14:36.557 INFO:tasks.workunit.client.0.smithi057.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-30T14:14:41.420 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:41.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.567+0000 7f14fab80640 1 Processor -- start 2023-10-30T14:14:41.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.567+0000 7f14fab80640 1 -- start start 2023-10-30T14:14:41.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.571+0000 7f14fab80640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f14f41516c0 0x7f14f40c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:41.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.571+0000 7f14fab80640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14f40c82d0 0x7f14f40ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:41.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.571+0000 7f14fab80640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f14f40cd4a0 0x7f14f40c2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:41.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.571+0000 7f14fab80640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14f40c74a0 con 0x7f14f41516c0 2023-10-30T14:14:41.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.571+0000 7f14fab80640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14f4151aa0 con 0x7f14f40cd4a0 2023-10-30T14:14:41.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.571+0000 7f14fab80640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f4158af0 con 0x7f14f40c82d0 2023-10-30T14:14:41.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.571+0000 7f14f891c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f14f41516c0 0x7f14f40c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:41.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.571+0000 7f14f891c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f14f41516c0 0x7f14f40c7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39890/0 (socket says 172.21.15.57:39890) 2023-10-30T14:14:41.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.571+0000 7f14f891c640 1 -- 172.21.15.57:0/4155111618 learned_addr learned my addr 172.21.15.57:0/4155111618 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:41.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.571+0000 7f14f3fff640 1 --2- 172.21.15.57:0/4155111618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14f40c82d0 0x7f14f40ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:41.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.571+0000 7f14f911d640 1 --2- 172.21.15.57:0/4155111618 >> v2:172.21.15.88:3300/0 conn(0x7f14f40cd4a0 0x7f14f40c2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:41.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.571+0000 7f14f891c640 1 -- 172.21.15.57:0/4155111618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14f40c82d0 msgr2=0x7f14f40ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:41.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.571+0000 7f14f891c640 1 --2- 172.21.15.57:0/4155111618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14f40c82d0 0x7f14f40ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.571+0000 7f14f891c640 1 -- 172.21.15.57:0/4155111618 >> v2:172.21.15.88:3300/0 conn(0x7f14f40cd4a0 msgr2=0x7f14f40c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:41.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.571+0000 7f14f891c640 1 --2- 172.21.15.57:0/4155111618 >> v2:172.21.15.88:3300/0 conn(0x7f14f40cd4a0 0x7f14f40c2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.571+0000 7f14f891c640 1 -- 172.21.15.57:0/4155111618 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f4158510 con 0x7f14f41516c0 2023-10-30T14:14:41.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14f891c640 1 --2- 172.21.15.57:0/4155111618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14f41516c0 0x7f14f40c7d90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f14f4071f60 tx=0x7f14e801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=549903b1c12084d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:41.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14f37fe640 1 -- 172.21.15.57:0/4155111618 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e80ab600 con 0x7f14f41516c0 2023-10-30T14:14:41.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14f37fe640 1 -- 172.21.15.57:0/4155111618 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e80a7050 con 0x7f14f41516c0 2023-10-30T14:14:41.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14f37fe640 1 -- 172.21.15.57:0/4155111618 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e80ab980 con 0x7f14f41516c0 2023-10-30T14:14:41.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14fab80640 1 -- 172.21.15.57:0/4155111618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14f41516c0 msgr2=0x7f14f40c7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:41.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14fab80640 1 --2- 172.21.15.57:0/4155111618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14f41516c0 0x7f14f40c7d90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f14f4071f60 tx=0x7f14e801c500 comp rx=0 tx=0).stop 2023-10-30T14:14:41.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14fab80640 1 -- 172.21.15.57:0/4155111618 shutdown_connections 2023-10-30T14:14:41.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14fab80640 1 --2- 172.21.15.57:0/4155111618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14f40c82d0 0x7f14f40ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14fab80640 1 --2- 172.21.15.57:0/4155111618 >> v2:172.21.15.88:3300/0 conn(0x7f14f40cd4a0 0x7f14f40c2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14fab80640 1 --2- 172.21.15.57:0/4155111618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14f41516c0 0x7f14f40c7d90 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14fab80640 1 -- 172.21.15.57:0/4155111618 >> 172.21.15.57:0/4155111618 conn(0x7f14f4071fb0 msgr2=0x7f14f40723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:41.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14fab80640 1 -- 172.21.15.57:0/4155111618 shutdown_connections 2023-10-30T14:14:41.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14fab80640 1 -- 172.21.15.57:0/4155111618 wait complete. 2023-10-30T14:14:41.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14fab80640 1 Processor -- start 2023-10-30T14:14:41.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14fab80640 1 -- start start 2023-10-30T14:14:41.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14fab80640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14f40c82d0 0x7f14f41467d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:41.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14fab80640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14f40cd4a0 0x7f14f4149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:41.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14fab80640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14f41516c0 0x7f14f4144ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:41.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14fab80640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f41f2d60 con 0x7f14f40c82d0 2023-10-30T14:14:41.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14fab80640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f4144290 con 0x7f14f40cd4a0 2023-10-30T14:14:41.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.575+0000 7f14fab80640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f41f20e0 con 0x7f14f41516c0 2023-10-30T14:14:41.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.579+0000 7f14f891c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14f40c82d0 0x7f14f41467d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:41.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.579+0000 7f14f891c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14f40c82d0 0x7f14f41467d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39900/0 (socket says 172.21.15.57:39900) 2023-10-30T14:14:41.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.579+0000 7f14f891c640 1 -- 172.21.15.57:0/3454012403 learned_addr learned my addr 172.21.15.57:0/3454012403 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:41.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.579+0000 7f14f3fff640 1 --2- 172.21.15.57:0/3454012403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14f40cd4a0 0x7f14f4149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:41.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.579+0000 7f14f911d640 1 --2- 172.21.15.57:0/3454012403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14f41516c0 0x7f14f4144ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:41.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.579+0000 7f14f891c640 1 -- 172.21.15.57:0/3454012403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14f41516c0 msgr2=0x7f14f4144ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:41.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.579+0000 7f14f891c640 1 --2- 172.21.15.57:0/3454012403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14f41516c0 0x7f14f4144ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.579+0000 7f14f891c640 1 -- 172.21.15.57:0/3454012403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14f40cd4a0 msgr2=0x7f14f4149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:41.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.579+0000 7f14f891c640 1 --2- 172.21.15.57:0/3454012403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14f40cd4a0 0x7f14f4149b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.579+0000 7f14f891c640 1 -- 172.21.15.57:0/3454012403 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e80094d0 con 0x7f14f40c82d0 2023-10-30T14:14:41.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.579+0000 7f14f891c640 1 --2- 172.21.15.57:0/3454012403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14f40c82d0 0x7f14f41467d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f14e80ae040 tx=0x7f14e801ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:41.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.579+0000 7f14f17fa640 1 -- 172.21.15.57:0/3454012403 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e80b6620 con 0x7f14f40c82d0 2023-10-30T14:14:41.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.579+0000 7f14f17fa640 1 -- 172.21.15.57:0/3454012403 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e80b67a0 con 0x7f14f40c82d0 2023-10-30T14:14:41.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.579+0000 7f14f17fa640 1 -- 172.21.15.57:0/3454012403 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e80b6b50 con 0x7f14f40c82d0 2023-10-30T14:14:41.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.579+0000 7f14fab80640 1 -- 172.21.15.57:0/3454012403 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f41f23c0 con 0x7f14f40c82d0 2023-10-30T14:14:41.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.579+0000 7f14fab80640 1 -- 172.21.15.57:0/3454012403 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f41f2980 con 0x7f14f40c82d0 2023-10-30T14:14:41.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.579+0000 7f14fab80640 1 -- 172.21.15.57:0/3454012403 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f404e170 con 0x7f14f40c82d0 2023-10-30T14:14:41.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.579+0000 7f14f17fa640 1 -- 172.21.15.57:0/3454012403 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f14e80b2040 con 0x7f14f40c82d0 2023-10-30T14:14:41.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.579+0000 7f14f17fa640 1 --2- 172.21.15.57:0/3454012403 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f14d80664a0 0x7f14d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:41.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.583+0000 7f14f17fa640 1 -- 172.21.15.57:0/3454012403 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14e811e500 con 0x7f14f40c82d0 2023-10-30T14:14:41.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.583+0000 7f14f3fff640 1 --2- 172.21.15.57:0/3454012403 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f14d80664a0 0x7f14d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:41.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.583+0000 7f14f3fff640 1 --2- 172.21.15.57:0/3454012403 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f14d80664a0 0x7f14d8068960 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f14f415a5e0 tx=0x7f14e0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:41.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.587+0000 7f14f17fa640 1 -- 172.21.15.57:0/3454012403 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f14e80eb970 con 0x7f14f40c82d0 2023-10-30T14:14:41.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.775+0000 7f14fab80640 1 -- 172.21.15.57:0/3454012403 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14f40c7b70 con 0x7f14f40c82d0 2023-10-30T14:14:41.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.775+0000 7f14f17fa640 1 -- 172.21.15.57:0/3454012403 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7f14e80eef90 con 0x7f14f40c82d0 2023-10-30T14:14:41.784 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:41.784 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:14:20.837447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6837/901968115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":901968115},{"type":"v1","addr":"172.21.15.57:6837","nonce":901968115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:41.784 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 12 2023-10-30T14:14:41.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.779+0000 7f14fab80640 1 -- 172.21.15.57:0/3454012403 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f14d80664a0 msgr2=0x7f14d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:41.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.779+0000 7f14fab80640 1 --2- 172.21.15.57:0/3454012403 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f14d80664a0 0x7f14d8068960 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f14f415a5e0 tx=0x7f14e0008040 comp rx=0 tx=0).stop 2023-10-30T14:14:41.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.779+0000 7f14fab80640 1 -- 172.21.15.57:0/3454012403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14f40c82d0 msgr2=0x7f14f41467d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:41.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.779+0000 7f14fab80640 1 --2- 172.21.15.57:0/3454012403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14f40c82d0 0x7f14f41467d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f14e80ae040 tx=0x7f14e801ad40 comp rx=0 tx=0).stop 2023-10-30T14:14:41.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.779+0000 7f14fab80640 1 -- 172.21.15.57:0/3454012403 shutdown_connections 2023-10-30T14:14:41.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.779+0000 7f14fab80640 1 --2- 172.21.15.57:0/3454012403 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f14d80664a0 0x7f14d8068960 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.779+0000 7f14fab80640 1 --2- 172.21.15.57:0/3454012403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14f41516c0 0x7f14f4144ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.779+0000 7f14fab80640 1 --2- 172.21.15.57:0/3454012403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14f40cd4a0 0x7f14f4149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.779+0000 7f14fab80640 1 --2- 172.21.15.57:0/3454012403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14f40c82d0 0x7f14f41467d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.779+0000 7f14fab80640 1 -- 172.21.15.57:0/3454012403 >> 172.21.15.57:0/3454012403 conn(0x7f14f4071fb0 msgr2=0x7f14f4156250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:41.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.779+0000 7f14fab80640 1 -- 172.21.15.57:0/3454012403 shutdown_connections 2023-10-30T14:14:41.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:41.779+0000 7f14fab80640 1 -- 172.21.15.57:0/3454012403 wait complete. 2023-10-30T14:14:41.806 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=2) 2023-10-30T14:14:41.806 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T14:14:41.806 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:14:41.821 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T14:14:41.821 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:42.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.003+0000 7f4535f65640 1 Processor -- start 2023-10-30T14:14:42.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.003+0000 7f4535f65640 1 -- start start 2023-10-30T14:14:42.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.003+0000 7f4535f65640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4530151840 0x7f45300c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:42.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.003+0000 7f4535f65640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4530151c20 0x7f4530072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:42.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.003+0000 7f4535f65640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45300c5030 con 0x7f45300cb8b0 2023-10-30T14:14:42.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.003+0000 7f4535f65640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4530157830 con 0x7f4530151840 2023-10-30T14:14:42.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.003+0000 7f4535f65640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4530157b10 con 0x7f4530151c20 2023-10-30T14:14:42.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.003+0000 7f452ffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f45300cb8b0 0x7f453006b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46604/0 (socket says 172.21.15.57:46604) 2023-10-30T14:14:42.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.003+0000 7f452f7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4530151840 0x7f45300c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:42.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.003+0000 7f452ffff640 1 -- 172.21.15.57:0/2382625210 learned_addr learned my addr 172.21.15.57:0/2382625210 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:42.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.003+0000 7f452effd640 1 --2- 172.21.15.57:0/2382625210 >> v2:172.21.15.88:3300/0 conn(0x7f4530151c20 0x7f4530072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:42.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.003+0000 7f452f7fe640 1 -- 172.21.15.57:0/2382625210 >> v2:172.21.15.88:3300/0 conn(0x7f4530151c20 msgr2=0x7f4530072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:42.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.007+0000 7f452f7fe640 1 --2- 172.21.15.57:0/2382625210 >> v2:172.21.15.88:3300/0 conn(0x7f4530151c20 0x7f4530072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:42.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.007+0000 7f452f7fe640 1 -- 172.21.15.57:0/2382625210 >> v1:172.21.15.57:6789/0 conn(0x7f45300cb8b0 legacy=0x7f453006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:42.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.007+0000 7f452f7fe640 1 -- 172.21.15.57:0/2382625210 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4530157df0 con 0x7f4530151840 2023-10-30T14:14:42.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.007+0000 7f452e7fc640 1 -- 172.21.15.57:0/2382625210 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214923829 0 0) 0x7f45300c5030 con 0x7f45300cb8b0 2023-10-30T14:14:42.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.007+0000 7f452f7fe640 1 --2- 172.21.15.57:0/2382625210 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4530151840 0x7f45300c49a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f453006d4d0 tx=0x7f452001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33feca82e9484b80 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:42.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.007+0000 7f452e7fc640 1 -- 172.21.15.57:0/2382625210 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45200a8950 con 0x7f4530151840 2023-10-30T14:14:42.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.007+0000 7f452e7fc640 1 -- 172.21.15.57:0/2382625210 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45200a8ab0 con 0x7f4530151840 2023-10-30T14:14:42.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.007+0000 7f452e7fc640 1 -- 172.21.15.57:0/2382625210 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45200a8e40 con 0x7f4530151840 2023-10-30T14:14:42.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.007+0000 7f4535f65640 1 -- 172.21.15.57:0/2382625210 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4530151840 msgr2=0x7f45300c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:42.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.007+0000 7f4535f65640 1 --2- 172.21.15.57:0/2382625210 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4530151840 0x7f45300c49a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f453006d4d0 tx=0x7f452001ec50 comp rx=0 tx=0).stop 2023-10-30T14:14:42.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.007+0000 7f4535f65640 1 -- 172.21.15.57:0/2382625210 shutdown_connections 2023-10-30T14:14:42.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.007+0000 7f4535f65640 1 --2- 172.21.15.57:0/2382625210 >> v2:172.21.15.88:3300/0 conn(0x7f4530151c20 0x7f4530072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:42.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.007+0000 7f4535f65640 1 --2- 172.21.15.57:0/2382625210 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4530151840 0x7f45300c49a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:42.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.007+0000 7f4535f65640 1 -- 172.21.15.57:0/2382625210 >> 172.21.15.57:0/2382625210 conn(0x7f45300cac70 msgr2=0x7f45300cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:42.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.007+0000 7f4535f65640 1 -- 172.21.15.57:0/2382625210 shutdown_connections 2023-10-30T14:14:42.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.007+0000 7f4535f65640 1 -- 172.21.15.57:0/2382625210 wait complete. 2023-10-30T14:14:42.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f4535f65640 1 Processor -- start 2023-10-30T14:14:42.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f4535f65640 1 -- start start 2023-10-30T14:14:42.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f4535f65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45300cb8b0 0x7f4530156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:42.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f4535f65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4530151840 0x7f45301e1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:42.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f4535f65640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4530151c20 0x7f45300c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:42.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f4535f65640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45301f1e80 con 0x7f4530151c20 2023-10-30T14:14:42.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f4535f65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45301f2120 con 0x7f45300cb8b0 2023-10-30T14:14:42.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f4535f65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45301f2400 con 0x7f4530151840 2023-10-30T14:14:42.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f452ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4530151c20 0x7f45300c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:42.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f452ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4530151c20 0x7f45300c9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39918/0 (socket says 172.21.15.57:39918) 2023-10-30T14:14:42.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f452ffff640 1 -- 172.21.15.57:0/917439656 learned_addr learned my addr 172.21.15.57:0/917439656 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:42.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f452f7fe640 1 --2- 172.21.15.57:0/917439656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45300cb8b0 0x7f4530156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:42.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f452effd640 1 --2- 172.21.15.57:0/917439656 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4530151840 0x7f45301e1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:42.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f452ffff640 1 -- 172.21.15.57:0/917439656 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4530151840 msgr2=0x7f45301e1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:42.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f452ffff640 1 --2- 172.21.15.57:0/917439656 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4530151840 0x7f45301e1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:42.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f452ffff640 1 -- 172.21.15.57:0/917439656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45300cb8b0 msgr2=0x7f4530156980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:42.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f452ffff640 1 --2- 172.21.15.57:0/917439656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45300cb8b0 0x7f4530156980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:42.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f452ffff640 1 -- 172.21.15.57:0/917439656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4520009420 con 0x7f4530151c20 2023-10-30T14:14:42.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f452ffff640 1 --2- 172.21.15.57:0/917439656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4530151c20 0x7f45300c9bd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4528001570 tx=0x7f452800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:42.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f4517fff640 1 -- 172.21.15.57:0/917439656 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452800a640 con 0x7f4530151c20 2023-10-30T14:14:42.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f4517fff640 1 -- 172.21.15.57:0/917439656 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f452800a7c0 con 0x7f4530151c20 2023-10-30T14:14:42.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f4535f65640 1 -- 172.21.15.57:0/917439656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45301f26e0 con 0x7f4530151c20 2023-10-30T14:14:42.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.011+0000 7f4517fff640 1 -- 172.21.15.57:0/917439656 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452800ab70 con 0x7f4530151c20 2023-10-30T14:14:42.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.015+0000 7f4535f65640 1 -- 172.21.15.57:0/917439656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45301f2ca0 con 0x7f4530151c20 2023-10-30T14:14:42.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.015+0000 7f4535f65640 1 -- 172.21.15.57:0/917439656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f453004e170 con 0x7f4530151c20 2023-10-30T14:14:42.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.015+0000 7f4517fff640 1 -- 172.21.15.57:0/917439656 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f452800acd0 con 0x7f4530151c20 2023-10-30T14:14:42.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.015+0000 7f4517fff640 1 --2- 172.21.15.57:0/917439656 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f451c0664a0 0x7f451c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:42.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.015+0000 7f4517fff640 1 -- 172.21.15.57:0/917439656 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4528083660 con 0x7f4530151c20 2023-10-30T14:14:42.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.015+0000 7f452f7fe640 1 --2- 172.21.15.57:0/917439656 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f451c0664a0 0x7f451c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:42.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.019+0000 7f452f7fe640 1 --2- 172.21.15.57:0/917439656 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f451c0664a0 0x7f451c068960 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f45200b2000 tx=0x7f4520008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:42.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.019+0000 7f4517fff640 1 -- 172.21.15.57:0/917439656 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4528050b00 con 0x7f4530151c20 2023-10-30T14:14:42.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.215+0000 7f4535f65640 1 -- 172.21.15.57:0/917439656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4530155060 con 0x7f4530151c20 2023-10-30T14:14:42.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.215+0000 7f4517fff640 1 -- 172.21.15.57:0/917439656 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7f4528018090 con 0x7f4530151c20 2023-10-30T14:14:42.223 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:42.223 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:14:20.837447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6837/901968115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":901968115},{"type":"v1","addr":"172.21.15.57:6837","nonce":901968115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:42.223 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 12 2023-10-30T14:14:42.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.219+0000 7f4535f65640 1 -- 172.21.15.57:0/917439656 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f451c0664a0 msgr2=0x7f451c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:42.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.219+0000 7f4535f65640 1 --2- 172.21.15.57:0/917439656 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f451c0664a0 0x7f451c068960 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f45200b2000 tx=0x7f4520008040 comp rx=0 tx=0).stop 2023-10-30T14:14:42.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.219+0000 7f4535f65640 1 -- 172.21.15.57:0/917439656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4530151c20 msgr2=0x7f45300c9bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:42.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.219+0000 7f4535f65640 1 --2- 172.21.15.57:0/917439656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4530151c20 0x7f45300c9bd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4528001570 tx=0x7f452800cca0 comp rx=0 tx=0).stop 2023-10-30T14:14:42.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.219+0000 7f4535f65640 1 -- 172.21.15.57:0/917439656 shutdown_connections 2023-10-30T14:14:42.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.219+0000 7f4535f65640 1 --2- 172.21.15.57:0/917439656 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f451c0664a0 0x7f451c068960 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:42.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.219+0000 7f4535f65640 1 --2- 172.21.15.57:0/917439656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4530151c20 0x7f45300c9bd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:42.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.219+0000 7f4535f65640 1 --2- 172.21.15.57:0/917439656 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4530151840 0x7f45301e1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:42.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.219+0000 7f4535f65640 1 --2- 172.21.15.57:0/917439656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45300cb8b0 0x7f4530156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:42.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.219+0000 7f4535f65640 1 -- 172.21.15.57:0/917439656 >> 172.21.15.57:0/917439656 conn(0x7f45300cac70 msgr2=0x7f453014f390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:42.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.219+0000 7f4535f65640 1 -- 172.21.15.57:0/917439656 shutdown_connections 2023-10-30T14:14:42.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:42.219+0000 7f4535f65640 1 -- 172.21.15.57:0/917439656 wait complete. 2023-10-30T14:14:42.244 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:14:44.246 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:44.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.395+0000 7fa7c32ea640 1 Processor -- start 2023-10-30T14:14:44.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.395+0000 7fa7c32ea640 1 -- start start 2023-10-30T14:14:44.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.395+0000 7fa7c32ea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa7bc153c80 0x7fa7bc15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:44.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.395+0000 7fa7c32ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7bc0c42a0 0x7fa7bc0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:44.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.395+0000 7fa7c32ea640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7bc159f20 con 0x7fa7bc0c4a10 2023-10-30T14:14:44.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.395+0000 7fa7c32ea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7bc15dec0 con 0x7fa7bc153c80 2023-10-30T14:14:44.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.395+0000 7fa7c32ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc1599a0 con 0x7fa7bc0c42a0 2023-10-30T14:14:44.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.395+0000 7fa7c0885640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa7bc0c4a10 0x7fa7bc15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46274/0 (socket says 172.21.15.57:46274) 2023-10-30T14:14:44.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.395+0000 7fa7c0885640 1 -- 172.21.15.57:0/3226474853 learned_addr learned my addr 172.21.15.57:0/3226474853 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:44.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.395+0000 7fa7c1086640 1 --2- 172.21.15.57:0/3226474853 >> v2:172.21.15.88:3300/0 conn(0x7fa7bc153c80 0x7fa7bc15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:44.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.395+0000 7fa7c1887640 1 --2- 172.21.15.57:0/3226474853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7bc0c42a0 0x7fa7bc0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:44.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.395+0000 7fa7b3fff640 1 -- 172.21.15.57:0/3226474853 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610115104 0 0) 0x7fa7bc159f20 con 0x7fa7bc0c4a10 2023-10-30T14:14:44.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.395+0000 7fa7b3fff640 1 -- 172.21.15.57:0/3226474853 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a8003530 con 0x7fa7bc0c4a10 2023-10-30T14:14:44.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.395+0000 7fa7c1086640 1 -- 172.21.15.57:0/3226474853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7bc0c42a0 msgr2=0x7fa7bc0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:44.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.395+0000 7fa7c1086640 1 --2- 172.21.15.57:0/3226474853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7bc0c42a0 0x7fa7bc0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.395+0000 7fa7c1086640 1 -- 172.21.15.57:0/3226474853 >> v1:172.21.15.57:6789/0 conn(0x7fa7bc0c4a10 legacy=0x7fa7bc15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:44.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.395+0000 7fa7c1086640 1 -- 172.21.15.57:0/3226474853 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7bc15d620 con 0x7fa7bc153c80 2023-10-30T14:14:44.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.399+0000 7fa7c1086640 1 --2- 172.21.15.57:0/3226474853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7bc153c80 0x7fa7bc15cf90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa7b80097c0 tx=0x7fa7b801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=537a4acfb84b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:44.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.399+0000 7fa7b3fff640 1 -- 172.21.15.57:0/3226474853 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b80a8970 con 0x7fa7bc153c80 2023-10-30T14:14:44.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.399+0000 7fa7b3fff640 1 -- 172.21.15.57:0/3226474853 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b80a8ad0 con 0x7fa7bc153c80 2023-10-30T14:14:44.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.399+0000 7fa7b3fff640 1 -- 172.21.15.57:0/3226474853 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b80a8e40 con 0x7fa7bc153c80 2023-10-30T14:14:44.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.399+0000 7fa7c32ea640 1 -- 172.21.15.57:0/3226474853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7bc153c80 msgr2=0x7fa7bc15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:44.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.399+0000 7fa7c32ea640 1 --2- 172.21.15.57:0/3226474853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7bc153c80 0x7fa7bc15cf90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa7b80097c0 tx=0x7fa7b801ec70 comp rx=0 tx=0).stop 2023-10-30T14:14:44.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.399+0000 7fa7c32ea640 1 -- 172.21.15.57:0/3226474853 shutdown_connections 2023-10-30T14:14:44.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.399+0000 7fa7c32ea640 1 --2- 172.21.15.57:0/3226474853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7bc0c42a0 0x7fa7bc0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.399+0000 7fa7c32ea640 1 --2- 172.21.15.57:0/3226474853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7bc153c80 0x7fa7bc15cf90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.399+0000 7fa7c32ea640 1 -- 172.21.15.57:0/3226474853 >> 172.21.15.57:0/3226474853 conn(0x7fa7bc0715c0 msgr2=0x7fa7bc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:44.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.399+0000 7fa7c32ea640 1 -- 172.21.15.57:0/3226474853 shutdown_connections 2023-10-30T14:14:44.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.399+0000 7fa7c32ea640 1 -- 172.21.15.57:0/3226474853 wait complete. 2023-10-30T14:14:44.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c32ea640 1 Processor -- start 2023-10-30T14:14:44.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c32ea640 1 -- start start 2023-10-30T14:14:44.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c32ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7bc0c42a0 0x7fa7bc0b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:44.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c32ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7bc0c4a10 0x7fa7bc0c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:44.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c32ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7bc153c80 0x7fa7bc145b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:44.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c32ea640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc0b4ac0 con 0x7fa7bc0c42a0 2023-10-30T14:14:44.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c32ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc0b5150 con 0x7fa7bc0c4a10 2023-10-30T14:14:44.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c32ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc15c2f0 con 0x7fa7bc153c80 2023-10-30T14:14:44.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c1086640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7bc0c42a0 0x7fa7bc0b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:44.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c1086640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7bc0c42a0 0x7fa7bc0b2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54568/0 (socket says 172.21.15.57:54568) 2023-10-30T14:14:44.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c1086640 1 -- 172.21.15.57:0/3314810507 learned_addr learned my addr 172.21.15.57:0/3314810507 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:44.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c0885640 1 --2- 172.21.15.57:0/3314810507 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7bc0c4a10 0x7fa7bc0c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:44.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c1887640 1 --2- 172.21.15.57:0/3314810507 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7bc153c80 0x7fa7bc145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:44.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c1086640 1 -- 172.21.15.57:0/3314810507 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7bc153c80 msgr2=0x7fa7bc145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:44.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c1086640 1 --2- 172.21.15.57:0/3314810507 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7bc153c80 0x7fa7bc145b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c1086640 1 -- 172.21.15.57:0/3314810507 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7bc0c4a10 msgr2=0x7fa7bc0c8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:44.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c1086640 1 --2- 172.21.15.57:0/3314810507 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7bc0c4a10 0x7fa7bc0c8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c1086640 1 -- 172.21.15.57:0/3314810507 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b8009420 con 0x7fa7bc0c42a0 2023-10-30T14:14:44.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c1086640 1 --2- 172.21.15.57:0/3314810507 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7bc0c42a0 0x7fa7bc0b2020 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa7b801d320 tx=0x7fa7b801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:44.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7b1ffb640 1 -- 172.21.15.57:0/3314810507 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b80bf070 con 0x7fa7bc0c42a0 2023-10-30T14:14:44.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7b1ffb640 1 -- 172.21.15.57:0/3314810507 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b80b1b50 con 0x7fa7bc0c42a0 2023-10-30T14:14:44.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7b1ffb640 1 -- 172.21.15.57:0/3314810507 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b80b9410 con 0x7fa7bc0c42a0 2023-10-30T14:14:44.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.403+0000 7fa7c32ea640 1 -- 172.21.15.57:0/3314810507 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7bc1f2220 con 0x7fa7bc0c42a0 2023-10-30T14:14:44.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.407+0000 7fa7c32ea640 1 -- 172.21.15.57:0/3314810507 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7bc15c010 con 0x7fa7bc0c42a0 2023-10-30T14:14:44.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.407+0000 7fa7c32ea640 1 -- 172.21.15.57:0/3314810507 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7bc04e170 con 0x7fa7bc0c42a0 2023-10-30T14:14:44.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.407+0000 7fa7b1ffb640 1 -- 172.21.15.57:0/3314810507 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa7b80b7040 con 0x7fa7bc0c42a0 2023-10-30T14:14:44.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.407+0000 7fa7b1ffb640 1 --2- 172.21.15.57:0/3314810507 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa7a00664a0 0x7fa7a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:44.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.407+0000 7fa7b1ffb640 1 -- 172.21.15.57:0/3314810507 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7b8121340 con 0x7fa7bc0c42a0 2023-10-30T14:14:44.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.407+0000 7fa7c0885640 1 --2- 172.21.15.57:0/3314810507 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa7a00664a0 0x7fa7a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:44.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.407+0000 7fa7c0885640 1 --2- 172.21.15.57:0/3314810507 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa7a00664a0 0x7fa7a0068960 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc14dcc0 tx=0x7fa7ac009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:44.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.411+0000 7fa7b1ffb640 1 -- 172.21.15.57:0/3314810507 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa7b80ee7b0 con 0x7fa7bc0c42a0 2023-10-30T14:14:44.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.603+0000 7fa7c32ea640 1 -- 172.21.15.57:0/3314810507 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7bc0cad40 con 0x7fa7bc0c42a0 2023-10-30T14:14:44.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.603+0000 7fa7b1ffb640 1 -- 172.21.15.57:0/3314810507 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7fa7b80f1dd0 con 0x7fa7bc0c42a0 2023-10-30T14:14:44.612 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:44.612 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:14:20.837447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6837/901968115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":901968115},{"type":"v1","addr":"172.21.15.57:6837","nonce":901968115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:44.613 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 12 2023-10-30T14:14:44.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.607+0000 7fa7c32ea640 1 -- 172.21.15.57:0/3314810507 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa7a00664a0 msgr2=0x7fa7a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:44.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.607+0000 7fa7c32ea640 1 --2- 172.21.15.57:0/3314810507 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa7a00664a0 0x7fa7a0068960 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc14dcc0 tx=0x7fa7ac009040 comp rx=0 tx=0).stop 2023-10-30T14:14:44.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.607+0000 7fa7c32ea640 1 -- 172.21.15.57:0/3314810507 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7bc0c42a0 msgr2=0x7fa7bc0b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:44.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.607+0000 7fa7c32ea640 1 --2- 172.21.15.57:0/3314810507 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7bc0c42a0 0x7fa7bc0b2020 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa7b801d320 tx=0x7fa7b801ba10 comp rx=0 tx=0).stop 2023-10-30T14:14:44.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.607+0000 7fa7c32ea640 1 -- 172.21.15.57:0/3314810507 shutdown_connections 2023-10-30T14:14:44.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.607+0000 7fa7c32ea640 1 --2- 172.21.15.57:0/3314810507 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa7a00664a0 0x7fa7a0068960 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.607+0000 7fa7c32ea640 1 --2- 172.21.15.57:0/3314810507 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7bc153c80 0x7fa7bc145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.607+0000 7fa7c32ea640 1 --2- 172.21.15.57:0/3314810507 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7bc0c4a10 0x7fa7bc0c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.607+0000 7fa7c32ea640 1 --2- 172.21.15.57:0/3314810507 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7bc0c42a0 0x7fa7bc0b2020 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.607+0000 7fa7c32ea640 1 -- 172.21.15.57:0/3314810507 >> 172.21.15.57:0/3314810507 conn(0x7fa7bc0715c0 msgr2=0x7fa7bc158c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:44.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.607+0000 7fa7c32ea640 1 -- 172.21.15.57:0/3314810507 shutdown_connections 2023-10-30T14:14:44.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:44.607+0000 7fa7c32ea640 1 -- 172.21.15.57:0/3314810507 wait complete. 2023-10-30T14:14:44.634 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:14:46.635 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:46.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.783+0000 7ff397580640 1 Processor -- start 2023-10-30T14:14:46.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.783+0000 7ff397580640 1 -- start start 2023-10-30T14:14:46.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff397580640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff390153bb0 0x7ff3900c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:46.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff397580640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff39006c420 0x7ff3900c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:46.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff397580640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3900c5f80 con 0x7ff3900c7170 2023-10-30T14:14:46.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff397580640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3900c5690 con 0x7ff39006c420 2023-10-30T14:14:46.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff397580640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff39015a580 con 0x7ff390153bb0 2023-10-30T14:14:46.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff395b1d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff3900c7170 0x7ff390159980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46286/0 (socket says 172.21.15.57:46286) 2023-10-30T14:14:46.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff395b1d640 1 -- 172.21.15.57:0/480344826 learned_addr learned my addr 172.21.15.57:0/480344826 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:46.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff394b1b640 1 --2- 172.21.15.57:0/480344826 >> v2:172.21.15.88:3300/0 conn(0x7ff39006c420 0x7ff3900c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:46.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff39531c640 1 --2- 172.21.15.57:0/480344826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff390153bb0 0x7ff3900c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:46.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff387fff640 1 -- 172.21.15.57:0/480344826 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657890900 0 0) 0x7ff3900c5f80 con 0x7ff3900c7170 2023-10-30T14:14:46.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff387fff640 1 -- 172.21.15.57:0/480344826 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff37c003530 con 0x7ff3900c7170 2023-10-30T14:14:46.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff387fff640 1 -- 172.21.15.57:0/480344826 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2155569199 0 0) 0x7ff37c003530 con 0x7ff3900c7170 2023-10-30T14:14:46.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff387fff640 1 -- 172.21.15.57:0/480344826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff390153bb0 msgr2=0x7ff3900c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:46.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff387fff640 1 --2- 172.21.15.57:0/480344826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff390153bb0 0x7ff3900c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff387fff640 1 -- 172.21.15.57:0/480344826 >> v2:172.21.15.88:3300/0 conn(0x7ff39006c420 msgr2=0x7ff3900c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:46.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff387fff640 1 --2- 172.21.15.57:0/480344826 >> v2:172.21.15.88:3300/0 conn(0x7ff39006c420 0x7ff3900c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff387fff640 1 -- 172.21.15.57:0/480344826 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3900c5930 con 0x7ff3900c7170 2023-10-30T14:14:46.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff387fff640 1 -- 172.21.15.57:0/480344826 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7ff388001a00 con 0x7ff3900c7170 2023-10-30T14:14:46.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff387fff640 1 -- 172.21.15.57:0/480344826 >> v1:172.21.15.57:6789/0 conn(0x7ff3900c7170 legacy=0x7ff390159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:46.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff387fff640 1 --2- 172.21.15.57:0/480344826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff390153bb0 0x7ff37c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:46.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff387fff640 1 --2- 172.21.15.57:0/480344826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff37c003f00 0x7ff37c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:46.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff387fff640 1 --2- 172.21.15.57:0/480344826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff37c004d10 0x7ff37c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:46.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff387fff640 1 -- 172.21.15.57:0/480344826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3900c5690 con 0x7ff37c004d10 2023-10-30T14:14:46.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff387fff640 1 -- 172.21.15.57:0/480344826 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39015a580 con 0x7ff37c003f00 2023-10-30T14:14:46.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff387fff640 1 -- 172.21.15.57:0/480344826 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff37c00b9c0 con 0x7ff390153bb0 2023-10-30T14:14:46.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff395b1d640 1 --2- 172.21.15.57:0/480344826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff37c004d10 0x7ff37c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:46.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff39531c640 1 --2- 172.21.15.57:0/480344826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff390153bb0 0x7ff37c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:46.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff394b1b640 1 --2- 172.21.15.57:0/480344826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff37c003f00 0x7ff37c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:46.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff395b1d640 1 -- 172.21.15.57:0/480344826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff390153bb0 msgr2=0x7ff37c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:46.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff395b1d640 1 --2- 172.21.15.57:0/480344826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff390153bb0 0x7ff37c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff395b1d640 1 -- 172.21.15.57:0/480344826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff37c003f00 msgr2=0x7ff37c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:46.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff395b1d640 1 --2- 172.21.15.57:0/480344826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff37c003f00 0x7ff37c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.787+0000 7ff395b1d640 1 -- 172.21.15.57:0/480344826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff37c003530 con 0x7ff37c004d10 2023-10-30T14:14:46.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.791+0000 7ff395b1d640 1 --2- 172.21.15.57:0/480344826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff37c004d10 0x7ff37c005190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff38800a8d0 tx=0x7ff38801d3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed7edd0f91ce9aba server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:46.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.791+0000 7ff387fff640 1 -- 172.21.15.57:0/480344826 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3880ad630 con 0x7ff37c004d10 2023-10-30T14:14:46.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.791+0000 7ff387fff640 1 -- 172.21.15.57:0/480344826 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3880a8070 con 0x7ff37c004d10 2023-10-30T14:14:46.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.791+0000 7ff387fff640 1 -- 172.21.15.57:0/480344826 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3880ad9d0 con 0x7ff37c004d10 2023-10-30T14:14:46.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.791+0000 7ff397580640 1 -- 172.21.15.57:0/480344826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff37c004d10 msgr2=0x7ff37c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:46.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.791+0000 7ff397580640 1 --2- 172.21.15.57:0/480344826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff37c004d10 0x7ff37c005190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff38800a8d0 tx=0x7ff38801d3e0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.791+0000 7ff395b1d640 1 -- 172.21.15.57:0/480344826 reap_dead start 2023-10-30T14:14:46.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.791+0000 7ff397580640 1 -- 172.21.15.57:0/480344826 shutdown_connections 2023-10-30T14:14:46.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.791+0000 7ff397580640 1 -- 172.21.15.57:0/480344826 >> 172.21.15.57:0/480344826 conn(0x7ff3900cac70 msgr2=0x7ff3900cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:46.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.791+0000 7ff397580640 1 -- 172.21.15.57:0/480344826 shutdown_connections 2023-10-30T14:14:46.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.791+0000 7ff397580640 1 -- 172.21.15.57:0/480344826 wait complete. 2023-10-30T14:14:46.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.791+0000 7ff397580640 1 Processor -- start 2023-10-30T14:14:46.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.791+0000 7ff397580640 1 -- start start 2023-10-30T14:14:46.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff397580640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff37c004d10 0x7ff3900c3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:46.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff397580640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3900c4560 0x7ff3900c3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:46.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff397580640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3900c0ce0 0x7ff39014f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:46.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff397580640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3900c1370 con 0x7ff37c004d10 2023-10-30T14:14:46.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff397580640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff390156f80 con 0x7ff3900c4560 2023-10-30T14:14:46.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff397580640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff390157500 con 0x7ff3900c0ce0 2023-10-30T14:14:46.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff39531c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff37c004d10 0x7ff3900c3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:46.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff39531c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff37c004d10 0x7ff3900c3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54592/0 (socket says 172.21.15.57:54592) 2023-10-30T14:14:46.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff39531c640 1 -- 172.21.15.57:0/2607043594 learned_addr learned my addr 172.21.15.57:0/2607043594 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:46.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff394b1b640 1 --2- 172.21.15.57:0/2607043594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3900c4560 0x7ff3900c3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:46.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff395b1d640 1 --2- 172.21.15.57:0/2607043594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3900c0ce0 0x7ff39014f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:46.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff39531c640 1 -- 172.21.15.57:0/2607043594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3900c0ce0 msgr2=0x7ff39014f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:46.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff39531c640 1 --2- 172.21.15.57:0/2607043594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3900c0ce0 0x7ff39014f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff39531c640 1 -- 172.21.15.57:0/2607043594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3900c4560 msgr2=0x7ff3900c3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:46.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff39531c640 1 --2- 172.21.15.57:0/2607043594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3900c4560 0x7ff3900c3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff39531c640 1 -- 172.21.15.57:0/2607043594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3901ef210 con 0x7ff37c004d10 2023-10-30T14:14:46.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff394b1b640 1 --2- 172.21.15.57:0/2607043594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3900c4560 0x7ff3900c3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:46.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff39531c640 1 --2- 172.21.15.57:0/2607043594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff37c004d10 0x7ff3900c3a50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff38c0119d0 tx=0x7ff38c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:46.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff385ffb640 1 -- 172.21.15.57:0/2607043594 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c00fb50 con 0x7ff37c004d10 2023-10-30T14:14:46.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff385ffb640 1 -- 172.21.15.57:0/2607043594 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38c00fcb0 con 0x7ff37c004d10 2023-10-30T14:14:46.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff385ffb640 1 -- 172.21.15.57:0/2607043594 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c014440 con 0x7ff37c004d10 2023-10-30T14:14:46.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff397580640 1 -- 172.21.15.57:0/2607043594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff390157220 con 0x7ff37c004d10 2023-10-30T14:14:46.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff397580640 1 -- 172.21.15.57:0/2607043594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3901ef4f0 con 0x7ff37c004d10 2023-10-30T14:14:46.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.795+0000 7ff397580640 1 -- 172.21.15.57:0/2607043594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff39004e170 con 0x7ff37c004d10 2023-10-30T14:14:46.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.799+0000 7ff385ffb640 1 -- 172.21.15.57:0/2607043594 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff38c0145a0 con 0x7ff37c004d10 2023-10-30T14:14:46.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.799+0000 7ff385ffb640 1 --2- 172.21.15.57:0/2607043594 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff3740664d0 0x7ff374068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:46.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.799+0000 7ff385ffb640 1 -- 172.21.15.57:0/2607043594 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff38c0848b0 con 0x7ff37c004d10 2023-10-30T14:14:46.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.799+0000 7ff394b1b640 1 --2- 172.21.15.57:0/2607043594 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff3740664d0 0x7ff374068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:46.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.799+0000 7ff394b1b640 1 --2- 172.21.15.57:0/2607043594 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff3740664d0 0x7ff374068990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff39006baa0 tx=0x7ff380008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:46.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.803+0000 7ff385ffb640 1 -- 172.21.15.57:0/2607043594 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff38c051c40 con 0x7ff37c004d10 2023-10-30T14:14:46.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.991+0000 7ff397580640 1 -- 172.21.15.57:0/2607043594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff39006b730 con 0x7ff37c004d10 2023-10-30T14:14:47.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.995+0000 7ff385ffb640 1 -- 172.21.15.57:0/2607043594 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7ff38c022070 con 0x7ff37c004d10 2023-10-30T14:14:47.000 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:47.001 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:14:20.837447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6837/901968115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":901968115},{"type":"v1","addr":"172.21.15.57:6837","nonce":901968115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:47.001 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 12 2023-10-30T14:14:47.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.995+0000 7ff397580640 1 -- 172.21.15.57:0/2607043594 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff3740664d0 msgr2=0x7ff374068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:47.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.995+0000 7ff397580640 1 --2- 172.21.15.57:0/2607043594 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff3740664d0 0x7ff374068990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff39006baa0 tx=0x7ff380008040 comp rx=0 tx=0).stop 2023-10-30T14:14:47.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.995+0000 7ff397580640 1 -- 172.21.15.57:0/2607043594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff37c004d10 msgr2=0x7ff3900c3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:47.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.995+0000 7ff397580640 1 --2- 172.21.15.57:0/2607043594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff37c004d10 0x7ff3900c3a50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff38c0119d0 tx=0x7ff38c011e90 comp rx=0 tx=0).stop 2023-10-30T14:14:47.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.995+0000 7ff397580640 1 -- 172.21.15.57:0/2607043594 shutdown_connections 2023-10-30T14:14:47.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.995+0000 7ff397580640 1 --2- 172.21.15.57:0/2607043594 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff3740664d0 0x7ff374068990 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:47.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.995+0000 7ff397580640 1 --2- 172.21.15.57:0/2607043594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3900c0ce0 0x7ff39014f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:47.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.995+0000 7ff397580640 1 --2- 172.21.15.57:0/2607043594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3900c4560 0x7ff3900c3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:47.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.995+0000 7ff397580640 1 --2- 172.21.15.57:0/2607043594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff37c004d10 0x7ff3900c3a50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:47.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.995+0000 7ff397580640 1 -- 172.21.15.57:0/2607043594 >> 172.21.15.57:0/2607043594 conn(0x7ff3900cac70 msgr2=0x7ff390152d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:47.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.995+0000 7ff397580640 1 -- 172.21.15.57:0/2607043594 shutdown_connections 2023-10-30T14:14:47.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:46.995+0000 7ff397580640 1 -- 172.21.15.57:0/2607043594 wait complete. 2023-10-30T14:14:47.021 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:14:49.022 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:49.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a4e5c5640 1 Processor -- start 2023-10-30T14:14:49.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a4e5c5640 1 -- start start 2023-10-30T14:14:49.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a4e5c5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4a4815cf90 0x7f4a480c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:49.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a4e5c5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a48159f20 con 0x7f4a480c4a10 2023-10-30T14:14:49.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a4e5c5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a480c47e0 con 0x7f4a48153c80 2023-10-30T14:14:49.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a4e5c5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4815dec0 con 0x7f4a4815cf90 2023-10-30T14:14:49.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a477fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f4a480c4a10 0x7f4a480ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46296/0 (socket says 172.21.15.57:46296) 2023-10-30T14:14:49.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a477fe640 1 -- 172.21.15.57:0/2582762436 learned_addr learned my addr 172.21.15.57:0/2582762436 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:49.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a4cb62640 1 --2- 172.21.15.57:0/2582762436 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4a4815cf90 0x7f4a480c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:49.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a46ffd640 1 -- 172.21.15.57:0/2582762436 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3275211517 0 0) 0x7f4a48159f20 con 0x7f4a480c4a10 2023-10-30T14:14:49.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a46ffd640 1 -- 172.21.15.57:0/2582762436 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a30003530 con 0x7f4a480c4a10 2023-10-30T14:14:49.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a46ffd640 1 -- 172.21.15.57:0/2582762436 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3826139797 0 0) 0x7f4a30003530 con 0x7f4a480c4a10 2023-10-30T14:14:49.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a46ffd640 1 -- 172.21.15.57:0/2582762436 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4a4815cf90 msgr2=0x7f4a480c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:49.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a46ffd640 1 --2- 172.21.15.57:0/2582762436 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4a4815cf90 0x7f4a480c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:49.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a46ffd640 1 -- 172.21.15.57:0/2582762436 >> v1:172.21.15.88:6789/0 conn(0x7f4a48153c80 legacy=0x7f4a4815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:49.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a46ffd640 1 -- 172.21.15.57:0/2582762436 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4815d620 con 0x7f4a480c4a10 2023-10-30T14:14:49.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a46ffd640 1 -- 172.21.15.57:0/2582762436 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f4a3c001680 con 0x7f4a480c4a10 2023-10-30T14:14:49.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a46ffd640 1 -- 172.21.15.57:0/2582762436 >> v1:172.21.15.57:6789/0 conn(0x7f4a480c4a10 legacy=0x7f4a480ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:49.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a46ffd640 1 --2- 172.21.15.57:0/2582762436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a300039b0 0x7f4a30003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:49.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a46ffd640 1 --2- 172.21.15.57:0/2582762436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a300047d0 0x7f4a30004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:49.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a46ffd640 1 --2- 172.21.15.57:0/2582762436 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4a4815cf90 0x7f4a30005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:49.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a46ffd640 1 -- 172.21.15.57:0/2582762436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4815dec0 con 0x7f4a300039b0 2023-10-30T14:14:49.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a46ffd640 1 -- 172.21.15.57:0/2582762436 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3000bb00 con 0x7f4a300047d0 2023-10-30T14:14:49.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a46ffd640 1 -- 172.21.15.57:0/2582762436 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3000bcc0 con 0x7f4a4815cf90 2023-10-30T14:14:49.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.171+0000 7f4a47fff640 1 --2- 172.21.15.57:0/2582762436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a300039b0 0x7f4a30003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:49.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.175+0000 7f4a4cb62640 1 --2- 172.21.15.57:0/2582762436 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4a4815cf90 0x7f4a30005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:49.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.175+0000 7f4a477fe640 1 --2- 172.21.15.57:0/2582762436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a300047d0 0x7f4a30004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:49.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.175+0000 7f4a47fff640 1 -- 172.21.15.57:0/2582762436 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4a4815cf90 msgr2=0x7f4a30005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:49.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.175+0000 7f4a47fff640 1 --2- 172.21.15.57:0/2582762436 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4a4815cf90 0x7f4a30005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:49.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.175+0000 7f4a47fff640 1 -- 172.21.15.57:0/2582762436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a300047d0 msgr2=0x7f4a30004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:49.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.175+0000 7f4a47fff640 1 --2- 172.21.15.57:0/2582762436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a300047d0 0x7f4a30004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:49.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.175+0000 7f4a47fff640 1 -- 172.21.15.57:0/2582762436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a30003530 con 0x7f4a300039b0 2023-10-30T14:14:49.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.175+0000 7f4a4cb62640 1 --2- 172.21.15.57:0/2582762436 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4a4815cf90 0x7f4a30005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:49.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.175+0000 7f4a47fff640 1 --2- 172.21.15.57:0/2582762436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a300039b0 0x7f4a30003dd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4a38001750 tx=0x7f4a3801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ac2ded9f479019 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:49.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.175+0000 7f4a46ffd640 1 -- 172.21.15.57:0/2582762436 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a380052d0 con 0x7f4a300039b0 2023-10-30T14:14:49.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.175+0000 7f4a46ffd640 1 -- 172.21.15.57:0/2582762436 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a380a7050 con 0x7f4a300039b0 2023-10-30T14:14:49.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.175+0000 7f4a46ffd640 1 -- 172.21.15.57:0/2582762436 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a38003600 con 0x7f4a300039b0 2023-10-30T14:14:49.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a4e5c5640 1 -- 172.21.15.57:0/2582762436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a300039b0 msgr2=0x7f4a30003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:49.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a4e5c5640 1 --2- 172.21.15.57:0/2582762436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a300039b0 0x7f4a30003dd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4a38001750 tx=0x7f4a3801c3a0 comp rx=0 tx=0).stop 2023-10-30T14:14:49.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a4cb62640 1 -- 172.21.15.57:0/2582762436 reap_dead start 2023-10-30T14:14:49.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a4e5c5640 1 -- 172.21.15.57:0/2582762436 shutdown_connections 2023-10-30T14:14:49.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a4e5c5640 1 -- 172.21.15.57:0/2582762436 >> 172.21.15.57:0/2582762436 conn(0x7f4a480715c0 msgr2=0x7f4a480719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:49.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a4e5c5640 1 -- 172.21.15.57:0/2582762436 shutdown_connections 2023-10-30T14:14:49.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a4e5c5640 1 -- 172.21.15.57:0/2582762436 wait complete. 2023-10-30T14:14:49.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a4e5c5640 1 Processor -- start 2023-10-30T14:14:49.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a4e5c5640 1 -- start start 2023-10-30T14:14:49.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a4e5c5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a300039b0 0x7f4a481f20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:49.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a4e5c5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4a48157660 0x7f4a481480a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:49.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a4e5c5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a481496e0 0x7f4a481485e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:49.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a4e5c5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a48147c10 con 0x7f4a300039b0 2023-10-30T14:14:49.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a4e5c5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a481f2bc0 con 0x7f4a481496e0 2023-10-30T14:14:49.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a4e5c5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a48158de0 con 0x7f4a48157660 2023-10-30T14:14:49.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a47fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a300039b0 0x7f4a481f20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:49.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a47fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a300039b0 0x7f4a481f20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54616/0 (socket says 172.21.15.57:54616) 2023-10-30T14:14:49.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a47fff640 1 -- 172.21.15.57:0/989782066 learned_addr learned my addr 172.21.15.57:0/989782066 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:49.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a477fe640 1 --2- 172.21.15.57:0/989782066 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4a48157660 0x7f4a481480a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:49.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.179+0000 7f4a4cb62640 1 --2- 172.21.15.57:0/989782066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a481496e0 0x7f4a481485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:49.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.183+0000 7f4a47fff640 1 -- 172.21.15.57:0/989782066 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4a48157660 msgr2=0x7f4a481480a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:49.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.183+0000 7f4a47fff640 1 --2- 172.21.15.57:0/989782066 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4a48157660 0x7f4a481480a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:49.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.183+0000 7f4a47fff640 1 -- 172.21.15.57:0/989782066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a481496e0 msgr2=0x7f4a481485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:49.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.183+0000 7f4a47fff640 1 --2- 172.21.15.57:0/989782066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a481496e0 0x7f4a481485e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:49.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.183+0000 7f4a47fff640 1 -- 172.21.15.57:0/989782066 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a481590c0 con 0x7f4a300039b0 2023-10-30T14:14:49.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.183+0000 7f4a47fff640 1 --2- 172.21.15.57:0/989782066 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a300039b0 0x7f4a481f20d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4a38005040 tx=0x7f4a30005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:49.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.183+0000 7f4a44ff9640 1 -- 172.21.15.57:0/989782066 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a38007a40 con 0x7f4a300039b0 2023-10-30T14:14:49.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.183+0000 7f4a44ff9640 1 -- 172.21.15.57:0/989782066 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a38007680 con 0x7f4a300039b0 2023-10-30T14:14:49.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.183+0000 7f4a4e5c5640 1 -- 172.21.15.57:0/989782066 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a481478a0 con 0x7f4a300039b0 2023-10-30T14:14:49.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.183+0000 7f4a44ff9640 1 -- 172.21.15.57:0/989782066 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a38006a20 con 0x7f4a300039b0 2023-10-30T14:14:49.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.183+0000 7f4a4e5c5640 1 -- 172.21.15.57:0/989782066 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a481593a0 con 0x7f4a300039b0 2023-10-30T14:14:49.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.183+0000 7f4a4e5c5640 1 -- 172.21.15.57:0/989782066 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a4804e170 con 0x7f4a300039b0 2023-10-30T14:14:49.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.183+0000 7f4a44ff9640 1 -- 172.21.15.57:0/989782066 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4a38006b80 con 0x7f4a300039b0 2023-10-30T14:14:49.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.183+0000 7f4a44ff9640 1 --2- 172.21.15.57:0/989782066 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4a340664d0 0x7f4a34068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:49.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.183+0000 7f4a44ff9640 1 -- 172.21.15.57:0/989782066 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4a3811e530 con 0x7f4a300039b0 2023-10-30T14:14:49.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.187+0000 7f4a477fe640 1 --2- 172.21.15.57:0/989782066 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4a340664d0 0x7f4a34068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:49.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.187+0000 7f4a477fe640 1 --2- 172.21.15.57:0/989782066 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4a340664d0 0x7f4a34068990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4a480cbb20 tx=0x7f4a3c000f60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:49.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.187+0000 7f4a44ff9640 1 -- 172.21.15.57:0/989782066 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4a380eb8b0 con 0x7f4a300039b0 2023-10-30T14:14:49.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.383+0000 7f4a4e5c5640 1 -- 172.21.15.57:0/989782066 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a48072b90 con 0x7f4a300039b0 2023-10-30T14:14:49.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.383+0000 7f4a44ff9640 1 -- 172.21.15.57:0/989782066 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7f4a380eeed0 con 0x7f4a300039b0 2023-10-30T14:14:49.391 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:49.392 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:14:20.837447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6837/901968115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":901968115},{"type":"v1","addr":"172.21.15.57:6837","nonce":901968115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:49.392 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 12 2023-10-30T14:14:49.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.387+0000 7f4a4e5c5640 1 -- 172.21.15.57:0/989782066 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4a340664d0 msgr2=0x7f4a34068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:49.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.387+0000 7f4a4e5c5640 1 --2- 172.21.15.57:0/989782066 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4a340664d0 0x7f4a34068990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4a480cbb20 tx=0x7f4a3c000f60 comp rx=0 tx=0).stop 2023-10-30T14:14:49.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.387+0000 7f4a4e5c5640 1 -- 172.21.15.57:0/989782066 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a300039b0 msgr2=0x7f4a481f20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:49.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.387+0000 7f4a4e5c5640 1 --2- 172.21.15.57:0/989782066 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a300039b0 0x7f4a481f20d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4a38005040 tx=0x7f4a30005800 comp rx=0 tx=0).stop 2023-10-30T14:14:49.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.387+0000 7f4a4e5c5640 1 -- 172.21.15.57:0/989782066 shutdown_connections 2023-10-30T14:14:49.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.387+0000 7f4a4e5c5640 1 --2- 172.21.15.57:0/989782066 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4a340664d0 0x7f4a34068990 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:49.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.387+0000 7f4a4e5c5640 1 --2- 172.21.15.57:0/989782066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a481496e0 0x7f4a481485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:49.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.387+0000 7f4a4e5c5640 1 --2- 172.21.15.57:0/989782066 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4a48157660 0x7f4a481480a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:49.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.387+0000 7f4a4e5c5640 1 --2- 172.21.15.57:0/989782066 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a300039b0 0x7f4a481f20d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:49.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.387+0000 7f4a4e5c5640 1 -- 172.21.15.57:0/989782066 >> 172.21.15.57:0/989782066 conn(0x7f4a480715c0 msgr2=0x7f4a480c4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:49.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.387+0000 7f4a4e5c5640 1 -- 172.21.15.57:0/989782066 shutdown_connections 2023-10-30T14:14:49.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:49.387+0000 7f4a4e5c5640 1 -- 172.21.15.57:0/989782066 wait complete. 2023-10-30T14:14:49.413 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:14:51.413 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:51.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.563+0000 7f80ca901640 1 Processor -- start 2023-10-30T14:14:51.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.563+0000 7f80ca901640 1 -- start start 2023-10-30T14:14:51.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.563+0000 7f80ca901640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f80c40c4a10 0x7f80c415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:51.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.563+0000 7f80ca901640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80c40c42a0 0x7f80c40ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:51.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.563+0000 7f80ca901640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80c4159f20 con 0x7f80c4153c80 2023-10-30T14:14:51.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.563+0000 7f80ca901640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80c415dec0 con 0x7f80c40c4a10 2023-10-30T14:14:51.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.563+0000 7f80ca901640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c41599a0 con 0x7f80c40c42a0 2023-10-30T14:14:51.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.563+0000 7f80bbfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f80c40c4a10 0x7f80c415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:51.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.563+0000 7f80bbfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f80c40c4a10 0x7f80c415cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54618/0 (socket says 172.21.15.57:54618) 2023-10-30T14:14:51.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.563+0000 7f80bbfff640 1 -- 172.21.15.57:0/630741023 learned_addr learned my addr 172.21.15.57:0/630741023 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:51.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.563+0000 7f80c8e9e640 1 --2- 172.21.15.57:0/630741023 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80c40c42a0 0x7f80c40ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:51.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.563+0000 7f80bbfff640 1 -- 172.21.15.57:0/630741023 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80c40c42a0 msgr2=0x7f80c40ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:51.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.563+0000 7f80bbfff640 1 --2- 172.21.15.57:0/630741023 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80c40c42a0 0x7f80c40ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:51.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.563+0000 7f80bbfff640 1 -- 172.21.15.57:0/630741023 >> v1:172.21.15.88:6789/0 conn(0x7f80c4153c80 legacy=0x7f80c415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:51.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.563+0000 7f80bbfff640 1 -- 172.21.15.57:0/630741023 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c415d620 con 0x7f80c40c4a10 2023-10-30T14:14:51.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.567+0000 7f80bbfff640 1 --2- 172.21.15.57:0/630741023 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80c40c4a10 0x7f80c415cf90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f80ac009870 tx=0x7f80ac01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef55b4aa0a74a48 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:51.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.567+0000 7f80c37fe640 1 -- 172.21.15.57:0/630741023 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac0a8a30 con 0x7f80c40c4a10 2023-10-30T14:14:51.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.567+0000 7f80c37fe640 1 -- 172.21.15.57:0/630741023 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac0a8b90 con 0x7f80c40c4a10 2023-10-30T14:14:51.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.567+0000 7f80c37fe640 1 -- 172.21.15.57:0/630741023 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac0b0440 con 0x7f80c40c4a10 2023-10-30T14:14:51.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.567+0000 7f80ca901640 1 -- 172.21.15.57:0/630741023 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80c40c4a10 msgr2=0x7f80c415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:51.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.567+0000 7f80ca901640 1 --2- 172.21.15.57:0/630741023 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80c40c4a10 0x7f80c415cf90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f80ac009870 tx=0x7f80ac01ed30 comp rx=0 tx=0).stop 2023-10-30T14:14:51.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.567+0000 7f80ca901640 1 -- 172.21.15.57:0/630741023 shutdown_connections 2023-10-30T14:14:51.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.567+0000 7f80ca901640 1 --2- 172.21.15.57:0/630741023 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80c40c42a0 0x7f80c40ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:51.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.567+0000 7f80ca901640 1 --2- 172.21.15.57:0/630741023 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80c40c4a10 0x7f80c415cf90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:51.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.567+0000 7f80ca901640 1 -- 172.21.15.57:0/630741023 >> 172.21.15.57:0/630741023 conn(0x7f80c40715c0 msgr2=0x7f80c40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:51.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.567+0000 7f80ca901640 1 -- 172.21.15.57:0/630741023 shutdown_connections 2023-10-30T14:14:51.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.567+0000 7f80ca901640 1 -- 172.21.15.57:0/630741023 wait complete. 2023-10-30T14:14:51.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80ca901640 1 Processor -- start 2023-10-30T14:14:51.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80ca901640 1 -- start start 2023-10-30T14:14:51.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80ca901640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80c40c42a0 0x7f80c40c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:51.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80ca901640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80c40c4a10 0x7f80c4149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:51.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80ca901640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80c4153c80 0x7f80c414a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:51.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80ca901640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c41f1e30 con 0x7f80c40c4a10 2023-10-30T14:14:51.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80ca901640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c41f2110 con 0x7f80c40c42a0 2023-10-30T14:14:51.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80ca901640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c41f23f0 con 0x7f80c4153c80 2023-10-30T14:14:51.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80bbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80c40c4a10 0x7f80c4149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:51.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80bbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80c40c4a10 0x7f80c4149b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54624/0 (socket says 172.21.15.57:54624) 2023-10-30T14:14:51.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80bbfff640 1 -- 172.21.15.57:0/177533936 learned_addr learned my addr 172.21.15.57:0/177533936 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:51.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80c3fff640 1 --2- 172.21.15.57:0/177533936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80c40c42a0 0x7f80c40c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:51.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80c8e9e640 1 --2- 172.21.15.57:0/177533936 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80c4153c80 0x7f80c414a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:51.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80bbfff640 1 -- 172.21.15.57:0/177533936 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80c4153c80 msgr2=0x7f80c414a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:51.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80bbfff640 1 --2- 172.21.15.57:0/177533936 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80c4153c80 0x7f80c414a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:51.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80bbfff640 1 -- 172.21.15.57:0/177533936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80c40c42a0 msgr2=0x7f80c40c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:51.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80bbfff640 1 --2- 172.21.15.57:0/177533936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80c40c42a0 0x7f80c40c89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:51.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80bbfff640 1 -- 172.21.15.57:0/177533936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac0094d0 con 0x7f80c40c4a10 2023-10-30T14:14:51.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80bbfff640 1 --2- 172.21.15.57:0/177533936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80c40c4a10 0x7f80c4149b30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f80ac0b3040 tx=0x7f80ac01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:51.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80c17fa640 1 -- 172.21.15.57:0/177533936 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac0be070 con 0x7f80c40c4a10 2023-10-30T14:14:51.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80c17fa640 1 -- 172.21.15.57:0/177533936 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac0ba400 con 0x7f80c40c4a10 2023-10-30T14:14:51.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80c17fa640 1 -- 172.21.15.57:0/177533936 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac0ba7d0 con 0x7f80c40c4a10 2023-10-30T14:14:51.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80ca901640 1 -- 172.21.15.57:0/177533936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c41f26d0 con 0x7f80c40c4a10 2023-10-30T14:14:51.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80ca901640 1 -- 172.21.15.57:0/177533936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c41592b0 con 0x7f80c40c4a10 2023-10-30T14:14:51.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.571+0000 7f80ca901640 1 -- 172.21.15.57:0/177533936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c404e170 con 0x7f80c40c4a10 2023-10-30T14:14:51.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.575+0000 7f80c17fa640 1 -- 172.21.15.57:0/177533936 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f80ac0c3410 con 0x7f80c40c4a10 2023-10-30T14:14:51.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.575+0000 7f80c17fa640 1 --2- 172.21.15.57:0/177533936 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80a40664a0 0x7f80a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:51.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.575+0000 7f80c17fa640 1 -- 172.21.15.57:0/177533936 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f80ac121b10 con 0x7f80c40c4a10 2023-10-30T14:14:51.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.575+0000 7f80c3fff640 1 --2- 172.21.15.57:0/177533936 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80a40664a0 0x7f80a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:51.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.575+0000 7f80c3fff640 1 --2- 172.21.15.57:0/177533936 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80a40664a0 0x7f80a4068960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f80b4000f70 tx=0x7f80b4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:51.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.579+0000 7f80c17fa640 1 -- 172.21.15.57:0/177533936 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f80ac0eef00 con 0x7f80c40c4a10 2023-10-30T14:14:51.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.771+0000 7f80ca901640 1 -- 172.21.15.57:0/177533936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80c404e170 con 0x7f80c40c4a10 2023-10-30T14:14:51.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.771+0000 7f80c17fa640 1 -- 172.21.15.57:0/177533936 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7f80ac0f2520 con 0x7f80c40c4a10 2023-10-30T14:14:51.778 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:51.778 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:14:20.837447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6837/901968115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":901968115},{"type":"v1","addr":"172.21.15.57:6837","nonce":901968115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:51.779 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 12 2023-10-30T14:14:51.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.771+0000 7f80ca901640 1 -- 172.21.15.57:0/177533936 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80a40664a0 msgr2=0x7f80a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:51.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.771+0000 7f80ca901640 1 --2- 172.21.15.57:0/177533936 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80a40664a0 0x7f80a4068960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f80b4000f70 tx=0x7f80b4009040 comp rx=0 tx=0).stop 2023-10-30T14:14:51.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.771+0000 7f80ca901640 1 -- 172.21.15.57:0/177533936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80c40c4a10 msgr2=0x7f80c4149b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:51.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.771+0000 7f80ca901640 1 --2- 172.21.15.57:0/177533936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80c40c4a10 0x7f80c4149b30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f80ac0b3040 tx=0x7f80ac01bad0 comp rx=0 tx=0).stop 2023-10-30T14:14:51.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.771+0000 7f80ca901640 1 -- 172.21.15.57:0/177533936 shutdown_connections 2023-10-30T14:14:51.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.771+0000 7f80ca901640 1 --2- 172.21.15.57:0/177533936 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80a40664a0 0x7f80a4068960 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:51.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.771+0000 7f80ca901640 1 --2- 172.21.15.57:0/177533936 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80c4153c80 0x7f80c414a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:51.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.771+0000 7f80ca901640 1 --2- 172.21.15.57:0/177533936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80c40c4a10 0x7f80c4149b30 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:51.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.771+0000 7f80ca901640 1 --2- 172.21.15.57:0/177533936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80c40c42a0 0x7f80c40c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:51.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.771+0000 7f80ca901640 1 -- 172.21.15.57:0/177533936 >> 172.21.15.57:0/177533936 conn(0x7f80c40715c0 msgr2=0x7f80c41582a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:51.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.771+0000 7f80ca901640 1 -- 172.21.15.57:0/177533936 shutdown_connections 2023-10-30T14:14:51.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:51.771+0000 7f80ca901640 1 -- 172.21.15.57:0/177533936 wait complete. 2023-10-30T14:14:51.799 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:14:53.800 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:53.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.947+0000 7f098829c640 1 Processor -- start 2023-10-30T14:14:53.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.947+0000 7f098829c640 1 -- start start 2023-10-30T14:14:53.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.951+0000 7f098829c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09800c4a10 0x7f098015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:53.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.951+0000 7f098829c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f09800c42a0 0x7f09800ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:53.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.951+0000 7f098829c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0980159f20 con 0x7f0980153c80 2023-10-30T14:14:53.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.951+0000 7f098829c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f098015dec0 con 0x7f09800c42a0 2023-10-30T14:14:53.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.951+0000 7f098829c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09801599a0 con 0x7f09800c4a10 2023-10-30T14:14:53.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.951+0000 7f0986839640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f09800c42a0 0x7f09800ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:53.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.951+0000 7f0986839640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f09800c42a0 0x7f09800ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40388/0 (socket says 172.21.15.57:40388) 2023-10-30T14:14:53.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.951+0000 7f0986839640 1 -- 172.21.15.57:0/4145134497 learned_addr learned my addr 172.21.15.57:0/4145134497 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:53.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.951+0000 7f0985837640 1 --2- 172.21.15.57:0/4145134497 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09800c4a10 0x7f098015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:53.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.951+0000 7f0986839640 1 -- 172.21.15.57:0/4145134497 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09800c4a10 msgr2=0x7f098015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:53.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.951+0000 7f0986839640 1 --2- 172.21.15.57:0/4145134497 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09800c4a10 0x7f098015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:53.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.951+0000 7f0986839640 1 -- 172.21.15.57:0/4145134497 >> v1:172.21.15.88:6789/0 conn(0x7f0980153c80 legacy=0x7f098015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:53.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.951+0000 7f0986839640 1 -- 172.21.15.57:0/4145134497 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f098015d620 con 0x7f09800c42a0 2023-10-30T14:14:53.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.955+0000 7f0986839640 1 --2- 172.21.15.57:0/4145134497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09800c42a0 0x7f09800ca1e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0978009e50 tx=0x7f097801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a64b409c13d338dc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:53.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.955+0000 7f0985036640 1 -- 172.21.15.57:0/4145134497 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09780af400 con 0x7f09800c42a0 2023-10-30T14:14:53.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.955+0000 7f0985036640 1 -- 172.21.15.57:0/4145134497 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09780ab050 con 0x7f09800c42a0 2023-10-30T14:14:53.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.955+0000 7f0985036640 1 -- 172.21.15.57:0/4145134497 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09780af780 con 0x7f09800c42a0 2023-10-30T14:14:53.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.955+0000 7f098829c640 1 -- 172.21.15.57:0/4145134497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09800c42a0 msgr2=0x7f09800ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:53.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.955+0000 7f098829c640 1 --2- 172.21.15.57:0/4145134497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09800c42a0 0x7f09800ca1e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0978009e50 tx=0x7f097801f1f0 comp rx=0 tx=0).stop 2023-10-30T14:14:53.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.955+0000 7f098829c640 1 -- 172.21.15.57:0/4145134497 shutdown_connections 2023-10-30T14:14:53.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.955+0000 7f098829c640 1 --2- 172.21.15.57:0/4145134497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09800c42a0 0x7f09800ca1e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:53.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.955+0000 7f098829c640 1 --2- 172.21.15.57:0/4145134497 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09800c4a10 0x7f098015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:53.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.955+0000 7f098829c640 1 -- 172.21.15.57:0/4145134497 >> 172.21.15.57:0/4145134497 conn(0x7f09800715c0 msgr2=0x7f09800719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:53.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.955+0000 7f098829c640 1 -- 172.21.15.57:0/4145134497 shutdown_connections 2023-10-30T14:14:53.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.955+0000 7f098829c640 1 -- 172.21.15.57:0/4145134497 wait complete. 2023-10-30T14:14:53.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.955+0000 7f098829c640 1 Processor -- start 2023-10-30T14:14:53.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.955+0000 7f098829c640 1 -- start start 2023-10-30T14:14:53.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f098829c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09800c42a0 0x7f0980149fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:53.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f098829c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09800c4a10 0x7f0980154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:53.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f098829c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0980153c80 0x7f098015bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:53.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f098829c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09800b3da0 con 0x7f09800c42a0 2023-10-30T14:14:53.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f098829c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09800b42d0 con 0x7f09800c4a10 2023-10-30T14:14:53.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f098829c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098014a4e0 con 0x7f0980153c80 2023-10-30T14:14:53.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f0986038640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09800c42a0 0x7f0980149fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:53.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f0986038640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09800c42a0 0x7f0980149fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40392/0 (socket says 172.21.15.57:40392) 2023-10-30T14:14:53.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f0986038640 1 -- 172.21.15.57:0/1842959178 learned_addr learned my addr 172.21.15.57:0/1842959178 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:53.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f0985837640 1 --2- 172.21.15.57:0/1842959178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09800c4a10 0x7f0980154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:53.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f0986839640 1 --2- 172.21.15.57:0/1842959178 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0980153c80 0x7f098015bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:53.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f0986038640 1 -- 172.21.15.57:0/1842959178 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0980153c80 msgr2=0x7f098015bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:53.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f0986038640 1 --2- 172.21.15.57:0/1842959178 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0980153c80 0x7f098015bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:53.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f0986038640 1 -- 172.21.15.57:0/1842959178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09800c4a10 msgr2=0x7f0980154b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:53.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f0986038640 1 --2- 172.21.15.57:0/1842959178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09800c4a10 0x7f0980154b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:53.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f0986038640 1 -- 172.21.15.57:0/1842959178 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0978009b30 con 0x7f09800c42a0 2023-10-30T14:14:53.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f0986038640 1 --2- 172.21.15.57:0/1842959178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09800c42a0 0x7f0980149fa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f097c00c420 tx=0x7f097c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:53.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f0976ffd640 1 -- 172.21.15.57:0/1842959178 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097c0052d0 con 0x7f09800c42a0 2023-10-30T14:14:53.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f0976ffd640 1 -- 172.21.15.57:0/1842959178 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f097c005430 con 0x7f09800c42a0 2023-10-30T14:14:53.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f0976ffd640 1 -- 172.21.15.57:0/1842959178 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097c007260 con 0x7f09800c42a0 2023-10-30T14:14:53.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f098829c640 1 -- 172.21.15.57:0/1842959178 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09801efd20 con 0x7f09800c42a0 2023-10-30T14:14:53.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f098829c640 1 -- 172.21.15.57:0/1842959178 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0980154390 con 0x7f09800c42a0 2023-10-30T14:14:53.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f098829c640 1 -- 172.21.15.57:0/1842959178 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09801537f0 con 0x7f09800c42a0 2023-10-30T14:14:53.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.959+0000 7f0976ffd640 1 -- 172.21.15.57:0/1842959178 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f097c0073c0 con 0x7f09800c42a0 2023-10-30T14:14:53.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.963+0000 7f0976ffd640 1 --2- 172.21.15.57:0/1842959178 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09640664a0 0x7f0964068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:53.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.963+0000 7f0976ffd640 1 -- 172.21.15.57:0/1842959178 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f097c083310 con 0x7f09800c42a0 2023-10-30T14:14:53.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.963+0000 7f0985837640 1 --2- 172.21.15.57:0/1842959178 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09640664a0 0x7f0964068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:53.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.963+0000 7f0985837640 1 --2- 172.21.15.57:0/1842959178 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09640664a0 0x7f0964068960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0980158b00 tx=0x7f0970008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:53.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:53.967+0000 7f0976ffd640 1 -- 172.21.15.57:0/1842959178 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f097c050710 con 0x7f09800c42a0 2023-10-30T14:14:54.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:54.151+0000 7f098829c640 1 -- 172.21.15.57:0/1842959178 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f098004e170 con 0x7f09800c42a0 2023-10-30T14:14:54.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:54.155+0000 7f0976ffd640 1 -- 172.21.15.57:0/1842959178 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7f097c053d30 con 0x7f09800c42a0 2023-10-30T14:14:54.160 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:54.160 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:14:20.837447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6837/901968115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":901968115},{"type":"v1","addr":"172.21.15.57:6837","nonce":901968115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:54.160 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 12 2023-10-30T14:14:54.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:54.155+0000 7f098829c640 1 -- 172.21.15.57:0/1842959178 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09640664a0 msgr2=0x7f0964068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:54.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:54.155+0000 7f098829c640 1 --2- 172.21.15.57:0/1842959178 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09640664a0 0x7f0964068960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0980158b00 tx=0x7f0970008040 comp rx=0 tx=0).stop 2023-10-30T14:14:54.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:54.155+0000 7f098829c640 1 -- 172.21.15.57:0/1842959178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09800c42a0 msgr2=0x7f0980149fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:54.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:54.155+0000 7f098829c640 1 --2- 172.21.15.57:0/1842959178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09800c42a0 0x7f0980149fa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f097c00c420 tx=0x7f097c00c8e0 comp rx=0 tx=0).stop 2023-10-30T14:14:54.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:54.155+0000 7f098829c640 1 -- 172.21.15.57:0/1842959178 shutdown_connections 2023-10-30T14:14:54.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:54.155+0000 7f098829c640 1 --2- 172.21.15.57:0/1842959178 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09640664a0 0x7f0964068960 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:54.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:54.155+0000 7f098829c640 1 --2- 172.21.15.57:0/1842959178 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0980153c80 0x7f098015bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:54.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:54.155+0000 7f098829c640 1 --2- 172.21.15.57:0/1842959178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09800c4a10 0x7f0980154b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:54.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:54.155+0000 7f098829c640 1 --2- 172.21.15.57:0/1842959178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09800c42a0 0x7f0980149fa0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:54.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:54.155+0000 7f098829c640 1 -- 172.21.15.57:0/1842959178 >> 172.21.15.57:0/1842959178 conn(0x7f09800715c0 msgr2=0x7f098006bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:54.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:54.155+0000 7f098829c640 1 -- 172.21.15.57:0/1842959178 shutdown_connections 2023-10-30T14:14:54.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:54.155+0000 7f098829c640 1 -- 172.21.15.57:0/1842959178 wait complete. 2023-10-30T14:14:54.181 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:14:56.182 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:56.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.331+0000 7fe311d3e640 1 Processor -- start 2023-10-30T14:14:56.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.331+0000 7fe311d3e640 1 -- start start 2023-10-30T14:14:56.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.331+0000 7fe311d3e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe30c153bb0 0x7fe30c0c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:56.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.331+0000 7fe311d3e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe30c159980 0x7fe30c159d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:56.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.331+0000 7fe311d3e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe30c0c5f80 con 0x7fe30c06c420 2023-10-30T14:14:56.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.331+0000 7fe311d3e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe30c0c5690 con 0x7fe30c159980 2023-10-30T14:14:56.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.331+0000 7fe311d3e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe30c15a2d0 con 0x7fe30c153bb0 2023-10-30T14:14:56.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.331+0000 7fe30bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe30c159980 0x7fe30c159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:56.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.331+0000 7fe30affd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fe30c06c420 0x7fe30c0c6c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44640/0 (socket says 172.21.15.57:44640) 2023-10-30T14:14:56.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.331+0000 7fe30affd640 1 -- 172.21.15.57:0/2384551679 learned_addr learned my addr 172.21.15.57:0/2384551679 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:56.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.331+0000 7fe30b7fe640 1 --2- 172.21.15.57:0/2384551679 >> v2:172.21.15.88:3300/0 conn(0x7fe30c153bb0 0x7fe30c0c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:56.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.331+0000 7fe30bfff640 1 -- 172.21.15.57:0/2384551679 >> v2:172.21.15.88:3300/0 conn(0x7fe30c153bb0 msgr2=0x7fe30c0c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:56.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.331+0000 7fe30bfff640 1 --2- 172.21.15.57:0/2384551679 >> v2:172.21.15.88:3300/0 conn(0x7fe30c153bb0 0x7fe30c0c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.331+0000 7fe30bfff640 1 -- 172.21.15.57:0/2384551679 >> v1:172.21.15.57:6789/0 conn(0x7fe30c06c420 legacy=0x7fe30c0c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:56.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.331+0000 7fe30bfff640 1 -- 172.21.15.57:0/2384551679 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30c15a580 con 0x7fe30c159980 2023-10-30T14:14:56.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.331+0000 7fe30a7fc640 1 -- 172.21.15.57:0/2384551679 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013399645 0 0) 0x7fe30c0c5f80 con 0x7fe30c06c420 2023-10-30T14:14:56.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.335+0000 7fe30bfff640 1 --2- 172.21.15.57:0/2384551679 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe30c159980 0x7fe30c159d90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0cb840 tx=0x7fe30401f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff4351c03c4efb1e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:56.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.335+0000 7fe30a7fc640 1 -- 172.21.15.57:0/2384551679 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3040b0440 con 0x7fe30c159980 2023-10-30T14:14:56.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.335+0000 7fe30a7fc640 1 -- 172.21.15.57:0/2384551679 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3040b05a0 con 0x7fe30c159980 2023-10-30T14:14:56.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.335+0000 7fe30a7fc640 1 -- 172.21.15.57:0/2384551679 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3040b0930 con 0x7fe30c159980 2023-10-30T14:14:56.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.335+0000 7fe311d3e640 1 -- 172.21.15.57:0/2384551679 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe30c159980 msgr2=0x7fe30c159d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:56.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.335+0000 7fe311d3e640 1 --2- 172.21.15.57:0/2384551679 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe30c159980 0x7fe30c159d90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0cb840 tx=0x7fe30401f0f0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.335+0000 7fe311d3e640 1 -- 172.21.15.57:0/2384551679 shutdown_connections 2023-10-30T14:14:56.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.335+0000 7fe311d3e640 1 --2- 172.21.15.57:0/2384551679 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe30c159980 0x7fe30c159d90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.335+0000 7fe311d3e640 1 --2- 172.21.15.57:0/2384551679 >> v2:172.21.15.88:3300/0 conn(0x7fe30c153bb0 0x7fe30c0c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.335+0000 7fe311d3e640 1 -- 172.21.15.57:0/2384551679 >> 172.21.15.57:0/2384551679 conn(0x7fe30c0cac70 msgr2=0x7fe30c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:56.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.335+0000 7fe311d3e640 1 -- 172.21.15.57:0/2384551679 shutdown_connections 2023-10-30T14:14:56.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.335+0000 7fe311d3e640 1 -- 172.21.15.57:0/2384551679 wait complete. 2023-10-30T14:14:56.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe311d3e640 1 Processor -- start 2023-10-30T14:14:56.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe311d3e640 1 -- start start 2023-10-30T14:14:56.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe311d3e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe30c06c420 0x7fe30c156ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:56.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe311d3e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe30c153bb0 0x7fe30c157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:56.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe311d3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe30c159980 0x7fe30c14e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:56.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe311d3e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30c0c4940 con 0x7fe30c153bb0 2023-10-30T14:14:56.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe311d3e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30c0bd200 con 0x7fe30c159980 2023-10-30T14:14:56.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe311d3e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe30c1561b0 con 0x7fe30c06c420 2023-10-30T14:14:56.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe30affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe30c153bb0 0x7fe30c157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:56.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe30affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe30c153bb0 0x7fe30c157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40408/0 (socket says 172.21.15.57:40408) 2023-10-30T14:14:56.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe30affd640 1 -- 172.21.15.57:0/3053389450 learned_addr learned my addr 172.21.15.57:0/3053389450 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:56.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe30bfff640 1 --2- 172.21.15.57:0/3053389450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe30c159980 0x7fe30c14e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:56.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe30b7fe640 1 --2- 172.21.15.57:0/3053389450 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe30c06c420 0x7fe30c156ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:56.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe30affd640 1 -- 172.21.15.57:0/3053389450 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe30c06c420 msgr2=0x7fe30c156ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:56.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe30affd640 1 --2- 172.21.15.57:0/3053389450 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe30c06c420 0x7fe30c156ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe30affd640 1 -- 172.21.15.57:0/3053389450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe30c159980 msgr2=0x7fe30c14e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:56.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe30affd640 1 --2- 172.21.15.57:0/3053389450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe30c159980 0x7fe30c14e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe30affd640 1 -- 172.21.15.57:0/3053389450 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe304009a70 con 0x7fe30c153bb0 2023-10-30T14:14:56.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe30affd640 1 --2- 172.21.15.57:0/3053389450 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe30c153bb0 0x7fe30c157420 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe30000c4b0 tx=0x7fe30000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:56.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe2f3fff640 1 -- 172.21.15.57:0/3053389450 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3000074e0 con 0x7fe30c153bb0 2023-10-30T14:14:56.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe2f3fff640 1 -- 172.21.15.57:0/3053389450 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe300007660 con 0x7fe30c153bb0 2023-10-30T14:14:56.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe2f3fff640 1 -- 172.21.15.57:0/3053389450 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe300004a30 con 0x7fe30c153bb0 2023-10-30T14:14:56.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe311d3e640 1 -- 172.21.15.57:0/3053389450 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe30c156a50 con 0x7fe30c153bb0 2023-10-30T14:14:56.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.339+0000 7fe311d3e640 1 -- 172.21.15.57:0/3053389450 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe30c0c4b70 con 0x7fe30c153bb0 2023-10-30T14:14:56.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.343+0000 7fe311d3e640 1 -- 172.21.15.57:0/3053389450 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe30c156be0 con 0x7fe30c153bb0 2023-10-30T14:14:56.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.343+0000 7fe2f3fff640 1 -- 172.21.15.57:0/3053389450 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe300004b90 con 0x7fe30c153bb0 2023-10-30T14:14:56.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.343+0000 7fe2f3fff640 1 --2- 172.21.15.57:0/3053389450 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe2f80664a0 0x7fe2f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:56.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.343+0000 7fe2f3fff640 1 -- 172.21.15.57:0/3053389450 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe300082ff0 con 0x7fe30c153bb0 2023-10-30T14:14:56.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.343+0000 7fe30b7fe640 1 --2- 172.21.15.57:0/3053389450 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe2f80664a0 0x7fe2f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:56.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.343+0000 7fe30b7fe640 1 --2- 172.21.15.57:0/3053389450 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe2f80664a0 0x7fe2f8068960 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0079c0 tx=0x7fe2fc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:56.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.347+0000 7fe2f3fff640 1 -- 172.21.15.57:0/3053389450 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe3000b4090 con 0x7fe30c153bb0 2023-10-30T14:14:56.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.539+0000 7fe311d3e640 1 -- 172.21.15.57:0/3053389450 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe30c0c5be0 con 0x7fe30c153bb0 2023-10-30T14:14:56.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.539+0000 7fe2f3fff640 1 -- 172.21.15.57:0/3053389450 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7fe300018090 con 0x7fe30c153bb0 2023-10-30T14:14:56.548 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:56.548 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:14:20.837447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6837/901968115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":901968115},{"type":"v1","addr":"172.21.15.57:6837","nonce":901968115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:56.550 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 12 2023-10-30T14:14:56.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.543+0000 7fe311d3e640 1 -- 172.21.15.57:0/3053389450 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe2f80664a0 msgr2=0x7fe2f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:56.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.543+0000 7fe311d3e640 1 --2- 172.21.15.57:0/3053389450 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe2f80664a0 0x7fe2f8068960 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0079c0 tx=0x7fe2fc008040 comp rx=0 tx=0).stop 2023-10-30T14:14:56.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.543+0000 7fe311d3e640 1 -- 172.21.15.57:0/3053389450 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe30c153bb0 msgr2=0x7fe30c157420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:56.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.543+0000 7fe311d3e640 1 --2- 172.21.15.57:0/3053389450 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe30c153bb0 0x7fe30c157420 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe30000c4b0 tx=0x7fe30000c970 comp rx=0 tx=0).stop 2023-10-30T14:14:56.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.543+0000 7fe311d3e640 1 -- 172.21.15.57:0/3053389450 shutdown_connections 2023-10-30T14:14:56.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.543+0000 7fe311d3e640 1 --2- 172.21.15.57:0/3053389450 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe2f80664a0 0x7fe2f8068960 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.543+0000 7fe311d3e640 1 --2- 172.21.15.57:0/3053389450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe30c159980 0x7fe30c14e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.543+0000 7fe311d3e640 1 --2- 172.21.15.57:0/3053389450 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe30c153bb0 0x7fe30c157420 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.543+0000 7fe311d3e640 1 --2- 172.21.15.57:0/3053389450 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe30c06c420 0x7fe30c156ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.543+0000 7fe311d3e640 1 -- 172.21.15.57:0/3053389450 >> 172.21.15.57:0/3053389450 conn(0x7fe30c0cac70 msgr2=0x7fe30c1503b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:56.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.543+0000 7fe311d3e640 1 -- 172.21.15.57:0/3053389450 shutdown_connections 2023-10-30T14:14:56.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:56.543+0000 7fe311d3e640 1 -- 172.21.15.57:0/3053389450 wait complete. 2023-10-30T14:14:56.569 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:14:58.570 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:58.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.719+0000 7eff397f5640 1 Processor -- start 2023-10-30T14:14:58.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.719+0000 7eff397f5640 1 -- start start 2023-10-30T14:14:58.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.719+0000 7eff397f5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff34153c80 0x7eff3415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:58.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.719+0000 7eff397f5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7eff340c4a10 0x7eff340c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:58.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.719+0000 7eff397f5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff34159f20 con 0x7eff3415a3b0 2023-10-30T14:14:58.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.719+0000 7eff397f5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff3415dec0 con 0x7eff340c4a10 2023-10-30T14:14:58.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.719+0000 7eff397f5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff341599a0 con 0x7eff34153c80 2023-10-30T14:14:58.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.719+0000 7eff337fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7eff3415a3b0 0x7eff340ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44656/0 (socket says 172.21.15.57:44656) 2023-10-30T14:14:58.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.719+0000 7eff2a7fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7eff340c4a10 0x7eff340c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:58.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.719+0000 7eff337fe640 1 -- 172.21.15.57:0/2842249426 learned_addr learned my addr 172.21.15.57:0/2842249426 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:14:58.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.719+0000 7eff32ffd640 1 --2- 172.21.15.57:0/2842249426 >> v2:172.21.15.88:3300/0 conn(0x7eff34153c80 0x7eff3415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:58.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.719+0000 7eff2a7fc640 1 -- 172.21.15.57:0/2842249426 >> v2:172.21.15.88:3300/0 conn(0x7eff34153c80 msgr2=0x7eff3415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:58.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.719+0000 7eff2a7fc640 1 --2- 172.21.15.57:0/2842249426 >> v2:172.21.15.88:3300/0 conn(0x7eff34153c80 0x7eff3415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.719+0000 7eff2a7fc640 1 -- 172.21.15.57:0/2842249426 >> v1:172.21.15.57:6789/0 conn(0x7eff3415a3b0 legacy=0x7eff340ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:58.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.719+0000 7eff2a7fc640 1 -- 172.21.15.57:0/2842249426 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff340c47e0 con 0x7eff340c4a10 2023-10-30T14:14:58.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.719+0000 7eff327fc640 1 -- 172.21.15.57:0/2842249426 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688670252 0 0) 0x7eff34159f20 con 0x7eff3415a3b0 2023-10-30T14:14:58.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.723+0000 7eff2a7fc640 1 --2- 172.21.15.57:0/2842249426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff340c4a10 0x7eff340c42a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7eff340d00f0 tx=0x7eff1c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9565a0c030d9ae4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:58.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.723+0000 7eff327fc640 1 -- 172.21.15.57:0/2842249426 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c0a8950 con 0x7eff340c4a10 2023-10-30T14:14:58.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.723+0000 7eff327fc640 1 -- 172.21.15.57:0/2842249426 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1c0a8ab0 con 0x7eff340c4a10 2023-10-30T14:14:58.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.723+0000 7eff327fc640 1 -- 172.21.15.57:0/2842249426 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c0a8e40 con 0x7eff340c4a10 2023-10-30T14:14:58.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.723+0000 7eff397f5640 1 -- 172.21.15.57:0/2842249426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff340c4a10 msgr2=0x7eff340c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:58.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.723+0000 7eff397f5640 1 --2- 172.21.15.57:0/2842249426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff340c4a10 0x7eff340c42a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7eff340d00f0 tx=0x7eff1c01ec50 comp rx=0 tx=0).stop 2023-10-30T14:14:58.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.723+0000 7eff397f5640 1 -- 172.21.15.57:0/2842249426 shutdown_connections 2023-10-30T14:14:58.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.723+0000 7eff397f5640 1 --2- 172.21.15.57:0/2842249426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff340c4a10 0x7eff340c42a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.723+0000 7eff397f5640 1 --2- 172.21.15.57:0/2842249426 >> v2:172.21.15.88:3300/0 conn(0x7eff34153c80 0x7eff3415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.723+0000 7eff397f5640 1 -- 172.21.15.57:0/2842249426 >> 172.21.15.57:0/2842249426 conn(0x7eff340715c0 msgr2=0x7eff340719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:58.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.723+0000 7eff397f5640 1 -- 172.21.15.57:0/2842249426 shutdown_connections 2023-10-30T14:14:58.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.723+0000 7eff397f5640 1 -- 172.21.15.57:0/2842249426 wait complete. 2023-10-30T14:14:58.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff397f5640 1 Processor -- start 2023-10-30T14:14:58.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff397f5640 1 -- start start 2023-10-30T14:14:58.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff397f5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff340c4a10 0x7eff3414e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:58.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff397f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff34153c80 0x7eff34152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:58.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff397f5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff3415a3b0 0x7eff341eea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:58.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff397f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff34150ab0 con 0x7eff34153c80 2023-10-30T14:14:58.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff397f5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff34150d70 con 0x7eff3415a3b0 2023-10-30T14:14:58.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff397f5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff341f1fe0 con 0x7eff340c4a10 2023-10-30T14:14:58.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff2a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff34153c80 0x7eff34152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:58.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff32ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff340c4a10 0x7eff3414e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:58.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff2a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff34153c80 0x7eff34152e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40416/0 (socket says 172.21.15.57:40416) 2023-10-30T14:14:58.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff32ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff340c4a10 0x7eff3414e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:42430/0 (socket says 172.21.15.57:42430) 2023-10-30T14:14:58.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff2a7fc640 1 -- 172.21.15.57:0/1627247826 learned_addr learned my addr 172.21.15.57:0/1627247826 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:14:58.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff337fe640 1 --2- 172.21.15.57:0/1627247826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff3415a3b0 0x7eff341eea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:58.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff2a7fc640 1 -- 172.21.15.57:0/1627247826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff340c4a10 msgr2=0x7eff3414e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:58.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff2a7fc640 1 --2- 172.21.15.57:0/1627247826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff340c4a10 0x7eff3414e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff2a7fc640 1 -- 172.21.15.57:0/1627247826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff3415a3b0 msgr2=0x7eff341eea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:58.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff2a7fc640 1 --2- 172.21.15.57:0/1627247826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff3415a3b0 0x7eff341eea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff2a7fc640 1 -- 172.21.15.57:0/1627247826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1c009420 con 0x7eff34153c80 2023-10-30T14:14:58.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.727+0000 7eff2a7fc640 1 --2- 172.21.15.57:0/1627247826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff34153c80 0x7eff34152e90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7eff1c0b3040 tx=0x7eff1c01b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:58.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.731+0000 7eff2bfff640 1 -- 172.21.15.57:0/1627247826 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c0b0df0 con 0x7eff34153c80 2023-10-30T14:14:58.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.731+0000 7eff2bfff640 1 -- 172.21.15.57:0/1627247826 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1c0b9410 con 0x7eff34153c80 2023-10-30T14:14:58.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.731+0000 7eff2bfff640 1 -- 172.21.15.57:0/1627247826 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c0b97c0 con 0x7eff34153c80 2023-10-30T14:14:58.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.731+0000 7eff397f5640 1 -- 172.21.15.57:0/1627247826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff341f22c0 con 0x7eff34153c80 2023-10-30T14:14:58.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.731+0000 7eff397f5640 1 -- 172.21.15.57:0/1627247826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff341f2880 con 0x7eff34153c80 2023-10-30T14:14:58.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.731+0000 7eff397f5640 1 -- 172.21.15.57:0/1627247826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff3404e170 con 0x7eff34153c80 2023-10-30T14:14:58.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.731+0000 7eff2bfff640 1 -- 172.21.15.57:0/1627247826 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7eff1c0b7040 con 0x7eff34153c80 2023-10-30T14:14:58.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.735+0000 7eff2bfff640 1 --2- 172.21.15.57:0/1627247826 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7eff140664a0 0x7eff14068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:58.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.735+0000 7eff2bfff640 1 -- 172.21.15.57:0/1627247826 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7eff1c121a10 con 0x7eff34153c80 2023-10-30T14:14:58.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.735+0000 7eff32ffd640 1 --2- 172.21.15.57:0/1627247826 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7eff140664a0 0x7eff14068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:58.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.735+0000 7eff32ffd640 1 --2- 172.21.15.57:0/1627247826 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7eff140664a0 0x7eff14068960 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7eff24009b60 tx=0x7eff24009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:58.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.739+0000 7eff2bfff640 1 -- 172.21.15.57:0/1627247826 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7eff1c0eee20 con 0x7eff34153c80 2023-10-30T14:14:58.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.927+0000 7eff397f5640 1 -- 172.21.15.57:0/1627247826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff341f2a70 con 0x7eff34153c80 2023-10-30T14:14:58.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.931+0000 7eff2bfff640 1 -- 172.21.15.57:0/1627247826 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7eff1c0b5090 con 0x7eff34153c80 2023-10-30T14:14:58.937 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:14:58.937 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:14:20.837447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6837/901968115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":901968115},{"type":"v1","addr":"172.21.15.57:6837","nonce":901968115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:58.937 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 12 2023-10-30T14:14:58.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.931+0000 7eff397f5640 1 -- 172.21.15.57:0/1627247826 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7eff140664a0 msgr2=0x7eff14068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:58.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.931+0000 7eff397f5640 1 --2- 172.21.15.57:0/1627247826 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7eff140664a0 0x7eff14068960 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7eff24009b60 tx=0x7eff24009190 comp rx=0 tx=0).stop 2023-10-30T14:14:58.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.931+0000 7eff397f5640 1 -- 172.21.15.57:0/1627247826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff34153c80 msgr2=0x7eff34152e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:58.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.931+0000 7eff397f5640 1 --2- 172.21.15.57:0/1627247826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff34153c80 0x7eff34152e90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7eff1c0b3040 tx=0x7eff1c01b9f0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.931+0000 7eff397f5640 1 -- 172.21.15.57:0/1627247826 shutdown_connections 2023-10-30T14:14:58.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.931+0000 7eff397f5640 1 --2- 172.21.15.57:0/1627247826 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7eff140664a0 0x7eff14068960 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.931+0000 7eff397f5640 1 --2- 172.21.15.57:0/1627247826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff3415a3b0 0x7eff341eea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.931+0000 7eff397f5640 1 --2- 172.21.15.57:0/1627247826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff34153c80 0x7eff34152e90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.931+0000 7eff397f5640 1 --2- 172.21.15.57:0/1627247826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff340c4a10 0x7eff3414e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.931+0000 7eff397f5640 1 -- 172.21.15.57:0/1627247826 >> 172.21.15.57:0/1627247826 conn(0x7eff340715c0 msgr2=0x7eff340cb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:58.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.931+0000 7eff397f5640 1 -- 172.21.15.57:0/1627247826 shutdown_connections 2023-10-30T14:14:58.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:14:58.931+0000 7eff397f5640 1 -- 172.21.15.57:0/1627247826 wait complete. 2023-10-30T14:14:58.958 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:14:59.487 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:14:59.492 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:15:00.959 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:01.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.111+0000 7f881a5d3640 1 Processor -- start 2023-10-30T14:15:01.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.111+0000 7f881a5d3640 1 -- start start 2023-10-30T14:15:01.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.111+0000 7f881a5d3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f88140701a0 0x7f881415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:01.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.115+0000 7f881a5d3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88140c2aa0 con 0x7f881406b2f0 2023-10-30T14:15:01.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.115+0000 7f881a5d3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88140708e0 con 0x7f88141537f0 2023-10-30T14:15:01.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.115+0000 7f881a5d3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88140c3ab0 con 0x7f88140701a0 2023-10-30T14:15:01.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.115+0000 7f8818b70640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f881406b2f0 0x7f881406b700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44668/0 (socket says 172.21.15.57:44668) 2023-10-30T14:15:01.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.115+0000 7f8818b70640 1 -- 172.21.15.57:0/2487469504 learned_addr learned my addr 172.21.15.57:0/2487469504 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:15:01.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.115+0000 7f88137fe640 1 --2- 172.21.15.57:0/2487469504 >> v2:172.21.15.88:3300/0 conn(0x7f88140701a0 0x7f881415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:01.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.115+0000 7f8812ffd640 1 -- 172.21.15.57:0/2487469504 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439225000 0 0) 0x7f88140c2aa0 con 0x7f881406b2f0 2023-10-30T14:15:01.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.115+0000 7f8812ffd640 1 -- 172.21.15.57:0/2487469504 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87fc003530 con 0x7f881406b2f0 2023-10-30T14:15:01.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.115+0000 7f8812ffd640 1 -- 172.21.15.57:0/2487469504 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2552690117 0 0) 0x7f87fc003530 con 0x7f881406b2f0 2023-10-30T14:15:01.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.115+0000 7f88137fe640 1 -- 172.21.15.57:0/2487469504 >> v1:172.21.15.88:6789/0 conn(0x7f88141537f0 legacy=0x7f88140ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.115+0000 7f88137fe640 1 -- 172.21.15.57:0/2487469504 >> v1:172.21.15.57:6789/0 conn(0x7f881406b2f0 legacy=0x7f881406b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.115+0000 7f88137fe640 1 -- 172.21.15.57:0/2487469504 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88140c1eb0 con 0x7f88140701a0 2023-10-30T14:15:01.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.119+0000 7f88137fe640 1 --2- 172.21.15.57:0/2487469504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88140701a0 0x7f881415a2e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f88080097c0 tx=0x7f880801eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d0aef089d7c2a0d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:01.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.119+0000 7f8812ffd640 1 -- 172.21.15.57:0/2487469504 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880c001a00 con 0x7f88140701a0 2023-10-30T14:15:01.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.119+0000 7f8812ffd640 1 -- 172.21.15.57:0/2487469504 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88080a8610 con 0x7f88140701a0 2023-10-30T14:15:01.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.119+0000 7f8812ffd640 1 -- 172.21.15.57:0/2487469504 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88080a8980 con 0x7f88140701a0 2023-10-30T14:15:01.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.119+0000 7f881a5d3640 1 -- 172.21.15.57:0/2487469504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88140701a0 msgr2=0x7f881415a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.119+0000 7f881a5d3640 1 --2- 172.21.15.57:0/2487469504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88140701a0 0x7f881415a2e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f88080097c0 tx=0x7f880801eb90 comp rx=0 tx=0).stop 2023-10-30T14:15:01.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.119+0000 7f881a5d3640 1 -- 172.21.15.57:0/2487469504 shutdown_connections 2023-10-30T14:15:01.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.119+0000 7f881a5d3640 1 --2- 172.21.15.57:0/2487469504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88140701a0 0x7f881415a2e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.119+0000 7f881a5d3640 1 -- 172.21.15.57:0/2487469504 >> 172.21.15.57:0/2487469504 conn(0x7f8814071fb0 msgr2=0x7f88140723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:01.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.119+0000 7f881a5d3640 1 -- 172.21.15.57:0/2487469504 shutdown_connections 2023-10-30T14:15:01.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.119+0000 7f881a5d3640 1 -- 172.21.15.57:0/2487469504 wait complete. 2023-10-30T14:15:01.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.119+0000 7f881a5d3640 1 Processor -- start 2023-10-30T14:15:01.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.119+0000 7f881a5d3640 1 -- start start 2023-10-30T14:15:01.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f881a5d3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f881406b2f0 0x7f8814156e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:01.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f881a5d3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88140701a0 0x7f881415d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:01.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f881a5d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88141537f0 0x7f88141f2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:01.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f881a5d3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88141f6e90 con 0x7f88140701a0 2023-10-30T14:15:01.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f881a5d3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8814154ad0 con 0x7f881406b2f0 2023-10-30T14:15:01.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f881a5d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8814154d70 con 0x7f88141537f0 2023-10-30T14:15:01.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f88137fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88140701a0 0x7f881415d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:01.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f88137fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88140701a0 0x7f881415d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40430/0 (socket says 172.21.15.57:40430) 2023-10-30T14:15:01.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f88137fe640 1 -- 172.21.15.57:0/1521666002 learned_addr learned my addr 172.21.15.57:0/1521666002 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:15:01.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f8813fff640 1 --2- 172.21.15.57:0/1521666002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f881406b2f0 0x7f8814156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:01.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f8818b70640 1 --2- 172.21.15.57:0/1521666002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88141537f0 0x7f88141f2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:01.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f88137fe640 1 -- 172.21.15.57:0/1521666002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88141537f0 msgr2=0x7f88141f2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f88137fe640 1 --2- 172.21.15.57:0/1521666002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88141537f0 0x7f88141f2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f88137fe640 1 -- 172.21.15.57:0/1521666002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f881406b2f0 msgr2=0x7f8814156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f88137fe640 1 --2- 172.21.15.57:0/1521666002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f881406b2f0 0x7f8814156e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f88137fe640 1 -- 172.21.15.57:0/1521666002 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8808009420 con 0x7f88140701a0 2023-10-30T14:15:01.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f88137fe640 1 --2- 172.21.15.57:0/1521666002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88140701a0 0x7f881415d0f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8808000c10 tx=0x7f880801b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:01.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f8810ff9640 1 -- 172.21.15.57:0/1521666002 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88080a9ca0 con 0x7f88140701a0 2023-10-30T14:15:01.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f8810ff9640 1 -- 172.21.15.57:0/1521666002 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88080a9e00 con 0x7f88140701a0 2023-10-30T14:15:01.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f8810ff9640 1 -- 172.21.15.57:0/1521666002 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808003820 con 0x7f88140701a0 2023-10-30T14:15:01.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f881a5d3640 1 -- 172.21.15.57:0/1521666002 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8814155030 con 0x7f88140701a0 2023-10-30T14:15:01.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f881a5d3640 1 -- 172.21.15.57:0/1521666002 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88141f6840 con 0x7f88140701a0 2023-10-30T14:15:01.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f881a5d3640 1 -- 172.21.15.57:0/1521666002 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88141f7040 con 0x7f88140701a0 2023-10-30T14:15:01.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.123+0000 7f8810ff9640 1 -- 172.21.15.57:0/1521666002 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f88080b7040 con 0x7f88140701a0 2023-10-30T14:15:01.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.127+0000 7f8810ff9640 1 --2- 172.21.15.57:0/1521666002 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f88000664a0 0x7f8800068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:01.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.127+0000 7f8810ff9640 1 -- 172.21.15.57:0/1521666002 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f8808121620 con 0x7f88140701a0 2023-10-30T14:15:01.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.127+0000 7f8813fff640 1 --2- 172.21.15.57:0/1521666002 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f88000664a0 0x7f8800068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:01.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.127+0000 7f8813fff640 1 --2- 172.21.15.57:0/1521666002 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f88000664a0 0x7f8800068960 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8804007fd0 tx=0x7f8804009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:01.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.131+0000 7f8810ff9640 1 -- 172.21.15.57:0/1521666002 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f88141f7040 con 0x7f88140701a0 2023-10-30T14:15:01.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.319+0000 7f881a5d3640 1 -- 172.21.15.57:0/1521666002 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88141f7040 con 0x7f88140701a0 2023-10-30T14:15:01.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.319+0000 7f8810ff9640 1 -- 172.21.15.57:0/1521666002 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+5730 (secure 0 0 0) 0x7f88141f7040 con 0x7f88140701a0 2023-10-30T14:15:01.322 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:15:01.322 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:14:59.588369+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"a","rank":2,"incarnation":13,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:01.322 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 13 2023-10-30T14:15:01.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.319+0000 7f881a5d3640 1 -- 172.21.15.57:0/1521666002 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f88000664a0 msgr2=0x7f8800068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.319+0000 7f881a5d3640 1 --2- 172.21.15.57:0/1521666002 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f88000664a0 0x7f8800068960 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8804007fd0 tx=0x7f8804009040 comp rx=0 tx=0).stop 2023-10-30T14:15:01.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.319+0000 7f881a5d3640 1 -- 172.21.15.57:0/1521666002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88140701a0 msgr2=0x7f881415d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.319+0000 7f881a5d3640 1 --2- 172.21.15.57:0/1521666002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88140701a0 0x7f881415d0f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8808000c10 tx=0x7f880801b930 comp rx=0 tx=0).stop 2023-10-30T14:15:01.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.319+0000 7f881a5d3640 1 -- 172.21.15.57:0/1521666002 shutdown_connections 2023-10-30T14:15:01.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.319+0000 7f881a5d3640 1 --2- 172.21.15.57:0/1521666002 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f88000664a0 0x7f8800068960 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.319+0000 7f881a5d3640 1 --2- 172.21.15.57:0/1521666002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88141537f0 0x7f88141f2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.319+0000 7f881a5d3640 1 --2- 172.21.15.57:0/1521666002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88140701a0 0x7f881415d0f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.319+0000 7f881a5d3640 1 --2- 172.21.15.57:0/1521666002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f881406b2f0 0x7f8814156e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.319+0000 7f881a5d3640 1 -- 172.21.15.57:0/1521666002 >> 172.21.15.57:0/1521666002 conn(0x7f8814071fb0 msgr2=0x7f881406be30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:01.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.319+0000 7f881a5d3640 1 -- 172.21.15.57:0/1521666002 shutdown_connections 2023-10-30T14:15:01.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.319+0000 7f881a5d3640 1 -- 172.21.15.57:0/1521666002 wait complete. 2023-10-30T14:15:01.343 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T14:15:01.344 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:15:01.344 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:01.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.483+0000 7f25af53e640 1 Processor -- start 2023-10-30T14:15:01.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.483+0000 7f25af53e640 1 -- start start 2023-10-30T14:15:01.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.483+0000 7f25af53e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f25a80c4a10 0x7f25a815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:01.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.483+0000 7f25af53e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25a80c42a0 0x7f25a80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:01.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.483+0000 7f25af53e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a8159f20 con 0x7f25a8153c80 2023-10-30T14:15:01.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.483+0000 7f25af53e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a815dec0 con 0x7f25a80c4a10 2023-10-30T14:15:01.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.483+0000 7f25af53e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a81599a0 con 0x7f25a80c42a0 2023-10-30T14:15:01.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.487+0000 7f25ad2da640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f25a8153c80 0x7f25a815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44676/0 (socket says 172.21.15.57:44676) 2023-10-30T14:15:01.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.487+0000 7f25ad2da640 1 -- 172.21.15.57:0/1960941504 learned_addr learned my addr 172.21.15.57:0/1960941504 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:15:01.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.487+0000 7f25acad9640 1 --2- 172.21.15.57:0/1960941504 >> v2:172.21.15.57:3300/0 conn(0x7f25a80c4a10 0x7f25a815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:01.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.487+0000 7f25adadb640 1 --2- 172.21.15.57:0/1960941504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25a80c42a0 0x7f25a80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:01.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.487+0000 7f25acad9640 1 -- 172.21.15.57:0/1960941504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25a80c42a0 msgr2=0x7f25a80ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:01.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.487+0000 7f25acad9640 1 --2- 172.21.15.57:0/1960941504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25a80c42a0 0x7f25a80ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.487+0000 7f25acad9640 1 -- 172.21.15.57:0/1960941504 >> v1:172.21.15.57:6789/0 conn(0x7f25a8153c80 legacy=0x7f25a815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.487+0000 7f25acad9640 1 -- 172.21.15.57:0/1960941504 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a815d620 con 0x7f25a80c4a10 2023-10-30T14:15:01.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.487+0000 7f259ffff640 1 -- 172.21.15.57:0/1960941504 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2410154403 0 0) 0x7f25a8159f20 con 0x7f25a8153c80 2023-10-30T14:15:01.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.487+0000 7f25acad9640 1 --2- 172.21.15.57:0/1960941504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a80c4a10 0x7f25a815cf90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f25980097c0 tx=0x7f259801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bcc9a4dfc1c4bc7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:01.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.487+0000 7f259ffff640 1 -- 172.21.15.57:0/1960941504 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25980a8970 con 0x7f25a80c4a10 2023-10-30T14:15:01.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.487+0000 7f259ffff640 1 -- 172.21.15.57:0/1960941504 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25980a8ad0 con 0x7f25a80c4a10 2023-10-30T14:15:01.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.487+0000 7f259ffff640 1 -- 172.21.15.57:0/1960941504 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25980a8e60 con 0x7f25a80c4a10 2023-10-30T14:15:01.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25af53e640 1 -- 172.21.15.57:0/1960941504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a80c4a10 msgr2=0x7f25a815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25af53e640 1 --2- 172.21.15.57:0/1960941504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a80c4a10 0x7f25a815cf90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f25980097c0 tx=0x7f259801ec70 comp rx=0 tx=0).stop 2023-10-30T14:15:01.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25af53e640 1 -- 172.21.15.57:0/1960941504 shutdown_connections 2023-10-30T14:15:01.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25af53e640 1 --2- 172.21.15.57:0/1960941504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25a80c42a0 0x7f25a80ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25af53e640 1 --2- 172.21.15.57:0/1960941504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a80c4a10 0x7f25a815cf90 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25af53e640 1 -- 172.21.15.57:0/1960941504 >> 172.21.15.57:0/1960941504 conn(0x7f25a80715c0 msgr2=0x7f25a80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:01.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25af53e640 1 -- 172.21.15.57:0/1960941504 shutdown_connections 2023-10-30T14:15:01.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25af53e640 1 -- 172.21.15.57:0/1960941504 wait complete. 2023-10-30T14:15:01.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25af53e640 1 Processor -- start 2023-10-30T14:15:01.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25af53e640 1 -- start start 2023-10-30T14:15:01.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25af53e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a80c42a0 0x7f25a814a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:01.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25af53e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25a80c4a10 0x7f25a80ce910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:01.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25af53e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25a8153c80 0x7f25a815c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:01.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25af53e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a814b4a0 con 0x7f25a80c42a0 2023-10-30T14:15:01.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25af53e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a814b780 con 0x7f25a80c4a10 2023-10-30T14:15:01.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25af53e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a814ba60 con 0x7f25a8153c80 2023-10-30T14:15:01.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25ad2da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a80c42a0 0x7f25a814a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:01.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25ad2da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a80c42a0 0x7f25a814a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40452/0 (socket says 172.21.15.57:40452) 2023-10-30T14:15:01.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25ad2da640 1 -- 172.21.15.57:0/1436058980 learned_addr learned my addr 172.21.15.57:0/1436058980 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:15:01.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25adadb640 1 --2- 172.21.15.57:0/1436058980 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25a8153c80 0x7f25a815c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:01.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25acad9640 1 --2- 172.21.15.57:0/1436058980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25a80c4a10 0x7f25a80ce910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:01.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25ad2da640 1 -- 172.21.15.57:0/1436058980 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25a8153c80 msgr2=0x7f25a815c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25ad2da640 1 --2- 172.21.15.57:0/1436058980 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25a8153c80 0x7f25a815c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25ad2da640 1 -- 172.21.15.57:0/1436058980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25a80c4a10 msgr2=0x7f25a80ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25ad2da640 1 --2- 172.21.15.57:0/1436058980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25a80c4a10 0x7f25a80ce910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25ad2da640 1 -- 172.21.15.57:0/1436058980 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2598009420 con 0x7f25a80c42a0 2023-10-30T14:15:01.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.491+0000 7f25ad2da640 1 --2- 172.21.15.57:0/1436058980 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a80c42a0 0x7f25a814a9b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f25a400c480 tx=0x7f25a400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:01.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.495+0000 7f259dffb640 1 -- 172.21.15.57:0/1436058980 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a40074e0 con 0x7f25a80c42a0 2023-10-30T14:15:01.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.495+0000 7f259dffb640 1 -- 172.21.15.57:0/1436058980 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a4007660 con 0x7f25a80c42a0 2023-10-30T14:15:01.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.495+0000 7f259dffb640 1 -- 172.21.15.57:0/1436058980 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a4004a30 con 0x7f25a80c42a0 2023-10-30T14:15:01.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.495+0000 7f25af53e640 1 -- 172.21.15.57:0/1436058980 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a814bd40 con 0x7f25a80c42a0 2023-10-30T14:15:01.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.495+0000 7f25af53e640 1 -- 172.21.15.57:0/1436058980 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a80ce410 con 0x7f25a80c42a0 2023-10-30T14:15:01.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.495+0000 7f25af53e640 1 -- 172.21.15.57:0/1436058980 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a804e170 con 0x7f25a80c42a0 2023-10-30T14:15:01.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.495+0000 7f259dffb640 1 -- 172.21.15.57:0/1436058980 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f25a4004b90 con 0x7f25a80c42a0 2023-10-30T14:15:01.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.495+0000 7f259dffb640 1 --2- 172.21.15.57:0/1436058980 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f258c0664a0 0x7f258c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:01.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.495+0000 7f259dffb640 1 -- 172.21.15.57:0/1436058980 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f25a4083060 con 0x7f25a80c42a0 2023-10-30T14:15:01.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.495+0000 7f25acad9640 1 --2- 172.21.15.57:0/1436058980 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f258c0664a0 0x7f258c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:01.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.499+0000 7f25acad9640 1 --2- 172.21.15.57:0/1436058980 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f258c0664a0 0x7f258c068960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f25980b2000 tx=0x7f2598008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:01.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.499+0000 7f259dffb640 1 -- 172.21.15.57:0/1436058980 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f25a40b4090 con 0x7f25a80c42a0 2023-10-30T14:15:01.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.691+0000 7f25af53e640 1 -- 172.21.15.57:0/1436058980 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25a80cad40 con 0x7f25a80c42a0 2023-10-30T14:15:01.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.691+0000 7f259dffb640 1 -- 172.21.15.57:0/1436058980 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+5730 (secure 0 0 0) 0x7f25a4053f40 con 0x7f25a80c42a0 2023-10-30T14:15:01.695 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:15:01.695 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:14:59.588369+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"a","rank":2,"incarnation":13,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:01.695 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 13 2023-10-30T14:15:01.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.695+0000 7f25af53e640 1 -- 172.21.15.57:0/1436058980 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f258c0664a0 msgr2=0x7f258c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.695+0000 7f25af53e640 1 --2- 172.21.15.57:0/1436058980 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f258c0664a0 0x7f258c068960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f25980b2000 tx=0x7f2598008040 comp rx=0 tx=0).stop 2023-10-30T14:15:01.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.695+0000 7f25af53e640 1 -- 172.21.15.57:0/1436058980 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a80c42a0 msgr2=0x7f25a814a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.695+0000 7f25af53e640 1 --2- 172.21.15.57:0/1436058980 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a80c42a0 0x7f25a814a9b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f25a400c480 tx=0x7f25a400c940 comp rx=0 tx=0).stop 2023-10-30T14:15:01.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.695+0000 7f25af53e640 1 -- 172.21.15.57:0/1436058980 shutdown_connections 2023-10-30T14:15:01.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.695+0000 7f25af53e640 1 --2- 172.21.15.57:0/1436058980 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f258c0664a0 0x7f258c068960 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.695+0000 7f25af53e640 1 --2- 172.21.15.57:0/1436058980 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25a8153c80 0x7f25a815c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.695+0000 7f25af53e640 1 --2- 172.21.15.57:0/1436058980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25a80c4a10 0x7f25a80ce910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.695+0000 7f25af53e640 1 --2- 172.21.15.57:0/1436058980 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a80c42a0 0x7f25a814a9b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.695+0000 7f25af53e640 1 -- 172.21.15.57:0/1436058980 >> 172.21.15.57:0/1436058980 conn(0x7f25a80715c0 msgr2=0x7f25a80cb460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:01.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.695+0000 7f25af53e640 1 -- 172.21.15.57:0/1436058980 shutdown_connections 2023-10-30T14:15:01.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:01.695+0000 7f25af53e640 1 -- 172.21.15.57:0/1436058980 wait complete. 2023-10-30T14:15:01.717 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 13, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 13, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4262": { "addr": "172.21.15.57:6834/3701655528", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 3701655528, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 3701655528, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 13, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:replay", "state_seq": 1 }, "gid_4266": { "addr": "172.21.15.57:6836/2693548867", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2693548867, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2693548867, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4266, "incarnation": 9, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4282": { "addr": "172.21.15.88:6839/2945854794", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 2945854794, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2945854794, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 24, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:14:59.588369+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 4266, "mds_2": 4262 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:15:03.718 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:03.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.875+0000 7fec16c17640 1 Processor -- start 2023-10-30T14:15:03.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.875+0000 7fec16c17640 1 -- start start 2023-10-30T14:15:03.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.875+0000 7fec16c17640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fec10151840 0x7fec100c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:03.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.875+0000 7fec16c17640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec10151c20 0x7fec10072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:03.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.875+0000 7fec16c17640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec100c5030 con 0x7fec100cb8b0 2023-10-30T14:15:03.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.875+0000 7fec16c17640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec10157830 con 0x7fec10151840 2023-10-30T14:15:03.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.875+0000 7fec16c17640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec10157b10 con 0x7fec10151c20 2023-10-30T14:15:03.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.875+0000 7fec151b4640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fec100cb8b0 0x7fec1006b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:57632/0 (socket says 172.21.15.57:57632) 2023-10-30T14:15:03.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.875+0000 7fec151b4640 1 -- 172.21.15.57:0/869586802 learned_addr learned my addr 172.21.15.57:0/869586802 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:15:03.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.875+0000 7fec149b3640 1 --2- 172.21.15.57:0/869586802 >> v2:172.21.15.57:3300/0 conn(0x7fec10151840 0x7fec100c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:03.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.875+0000 7fec0ffff640 1 --2- 172.21.15.57:0/869586802 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec10151c20 0x7fec10072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:03.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.875+0000 7fec149b3640 1 -- 172.21.15.57:0/869586802 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec10151c20 msgr2=0x7fec10072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:03.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.875+0000 7fec149b3640 1 --2- 172.21.15.57:0/869586802 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec10151c20 0x7fec10072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:03.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.875+0000 7fec149b3640 1 -- 172.21.15.57:0/869586802 >> v1:172.21.15.57:6789/0 conn(0x7fec100cb8b0 legacy=0x7fec1006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:03.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.875+0000 7fec149b3640 1 -- 172.21.15.57:0/869586802 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec10157df0 con 0x7fec10151840 2023-10-30T14:15:03.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.875+0000 7fec0f7fe640 1 -- 172.21.15.57:0/869586802 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050447482 0 0) 0x7fec100c5030 con 0x7fec100cb8b0 2023-10-30T14:15:03.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.879+0000 7fec149b3640 1 --2- 172.21.15.57:0/869586802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec10151840 0x7fec100c49a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fec040097c0 tx=0x7fec0401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f15e6c071747cc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:03.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.879+0000 7fec0f7fe640 1 -- 172.21.15.57:0/869586802 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec040a8970 con 0x7fec10151840 2023-10-30T14:15:03.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.879+0000 7fec0f7fe640 1 -- 172.21.15.57:0/869586802 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec040a8ad0 con 0x7fec10151840 2023-10-30T14:15:03.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.879+0000 7fec0f7fe640 1 -- 172.21.15.57:0/869586802 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec040a8e60 con 0x7fec10151840 2023-10-30T14:15:03.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.879+0000 7fec16c17640 1 -- 172.21.15.57:0/869586802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec10151840 msgr2=0x7fec100c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:03.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.879+0000 7fec16c17640 1 --2- 172.21.15.57:0/869586802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec10151840 0x7fec100c49a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fec040097c0 tx=0x7fec0401ec70 comp rx=0 tx=0).stop 2023-10-30T14:15:03.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.879+0000 7fec16c17640 1 -- 172.21.15.57:0/869586802 shutdown_connections 2023-10-30T14:15:03.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.879+0000 7fec16c17640 1 --2- 172.21.15.57:0/869586802 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec10151c20 0x7fec10072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:03.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.879+0000 7fec16c17640 1 --2- 172.21.15.57:0/869586802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec10151840 0x7fec100c49a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:03.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.879+0000 7fec16c17640 1 -- 172.21.15.57:0/869586802 >> 172.21.15.57:0/869586802 conn(0x7fec100cac70 msgr2=0x7fec100cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:03.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.879+0000 7fec16c17640 1 -- 172.21.15.57:0/869586802 shutdown_connections 2023-10-30T14:15:03.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.879+0000 7fec16c17640 1 -- 172.21.15.57:0/869586802 wait complete. 2023-10-30T14:15:03.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.883+0000 7fec16c17640 1 Processor -- start 2023-10-30T14:15:03.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.883+0000 7fec16c17640 1 -- start start 2023-10-30T14:15:03.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.883+0000 7fec16c17640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec100cb8b0 0x7fec101dfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:03.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.883+0000 7fec16c17640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec10151840 0x7fec101f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:03.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.883+0000 7fec16c17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec10151c20 0x7fec100c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:03.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.883+0000 7fec16c17640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec101df490 con 0x7fec100cb8b0 2023-10-30T14:15:03.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.883+0000 7fec16c17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec101557d0 con 0x7fec10151c20 2023-10-30T14:15:03.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.883+0000 7fec16c17640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec101e0130 con 0x7fec10151840 2023-10-30T14:15:03.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.883+0000 7fec149b3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec100cb8b0 0x7fec101dfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:03.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.883+0000 7fec149b3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec100cb8b0 0x7fec101dfbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53248/0 (socket says 172.21.15.57:53248) 2023-10-30T14:15:03.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.883+0000 7fec149b3640 1 -- 172.21.15.57:0/2031848621 learned_addr learned my addr 172.21.15.57:0/2031848621 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:15:03.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.883+0000 7fec151b4640 1 --2- 172.21.15.57:0/2031848621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec10151c20 0x7fec100c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:03.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.883+0000 7fec0ffff640 1 --2- 172.21.15.57:0/2031848621 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec10151840 0x7fec101f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:03.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.883+0000 7fec149b3640 1 -- 172.21.15.57:0/2031848621 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec10151840 msgr2=0x7fec101f2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:03.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.883+0000 7fec149b3640 1 --2- 172.21.15.57:0/2031848621 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec10151840 0x7fec101f2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:03.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.883+0000 7fec149b3640 1 -- 172.21.15.57:0/2031848621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec10151c20 msgr2=0x7fec100c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:03.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.883+0000 7fec149b3640 1 --2- 172.21.15.57:0/2031848621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec10151c20 0x7fec100c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:03.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.883+0000 7fec149b3640 1 -- 172.21.15.57:0/2031848621 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec04009420 con 0x7fec100cb8b0 2023-10-30T14:15:03.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.887+0000 7fec149b3640 1 --2- 172.21.15.57:0/2031848621 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec100cb8b0 0x7fec101dfbf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fec0401c8a0 tx=0x7fec0401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:03.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.887+0000 7fec0d7fa640 1 -- 172.21.15.57:0/2031848621 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec040a8970 con 0x7fec100cb8b0 2023-10-30T14:15:03.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.887+0000 7fec0d7fa640 1 -- 172.21.15.57:0/2031848621 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec040b9410 con 0x7fec100cb8b0 2023-10-30T14:15:03.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.887+0000 7fec16c17640 1 -- 172.21.15.57:0/2031848621 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec101f2a70 con 0x7fec100cb8b0 2023-10-30T14:15:03.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.887+0000 7fec0d7fa640 1 -- 172.21.15.57:0/2031848621 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec040b97a0 con 0x7fec100cb8b0 2023-10-30T14:15:03.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.887+0000 7fec16c17640 1 -- 172.21.15.57:0/2031848621 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec101def10 con 0x7fec100cb8b0 2023-10-30T14:15:03.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.887+0000 7fec16c17640 1 -- 172.21.15.57:0/2031848621 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec1004e170 con 0x7fec100cb8b0 2023-10-30T14:15:03.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.887+0000 7fec0d7fa640 1 -- 172.21.15.57:0/2031848621 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fec040b7040 con 0x7fec100cb8b0 2023-10-30T14:15:03.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.887+0000 7fec0d7fa640 1 --2- 172.21.15.57:0/2031848621 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7febf40664d0 0x7febf4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:03.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.887+0000 7fec0d7fa640 1 -- 172.21.15.57:0/2031848621 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fec04121f90 con 0x7fec100cb8b0 2023-10-30T14:15:03.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.891+0000 7fec0ffff640 1 --2- 172.21.15.57:0/2031848621 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7febf40664d0 0x7febf4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:03.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.891+0000 7fec0ffff640 1 --2- 172.21.15.57:0/2031848621 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7febf40664d0 0x7febf4068990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fec101ef730 tx=0x7febfc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:03.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:03.891+0000 7fec0d7fa640 1 -- 172.21.15.57:0/2031848621 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fec040ef450 con 0x7fec100cb8b0 2023-10-30T14:15:04.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:04.095+0000 7fec16c17640 1 -- 172.21.15.57:0/2031848621 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec10155060 con 0x7fec100cb8b0 2023-10-30T14:15:04.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:04.099+0000 7fec0d7fa640 1 -- 172.21.15.57:0/2031848621 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+5730 (secure 0 0 0) 0x7fec040c7090 con 0x7fec100cb8b0 2023-10-30T14:15:04.100 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:15:04.100 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:14:59.588369+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"a","rank":2,"incarnation":13,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:04.101 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 13 2023-10-30T14:15:04.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:04.099+0000 7fec16c17640 1 -- 172.21.15.57:0/2031848621 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7febf40664d0 msgr2=0x7febf4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:04.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:04.099+0000 7fec16c17640 1 --2- 172.21.15.57:0/2031848621 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7febf40664d0 0x7febf4068990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fec101ef730 tx=0x7febfc008040 comp rx=0 tx=0).stop 2023-10-30T14:15:04.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:04.099+0000 7fec16c17640 1 -- 172.21.15.57:0/2031848621 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec100cb8b0 msgr2=0x7fec101dfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:04.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:04.099+0000 7fec16c17640 1 --2- 172.21.15.57:0/2031848621 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec100cb8b0 0x7fec101dfbf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fec0401c8a0 tx=0x7fec0401ba10 comp rx=0 tx=0).stop 2023-10-30T14:15:04.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:04.099+0000 7fec16c17640 1 -- 172.21.15.57:0/2031848621 shutdown_connections 2023-10-30T14:15:04.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:04.099+0000 7fec16c17640 1 --2- 172.21.15.57:0/2031848621 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7febf40664d0 0x7febf4068990 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:04.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:04.099+0000 7fec16c17640 1 --2- 172.21.15.57:0/2031848621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec10151c20 0x7fec100c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:04.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:04.099+0000 7fec16c17640 1 --2- 172.21.15.57:0/2031848621 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec10151840 0x7fec101f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:04.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:04.099+0000 7fec16c17640 1 --2- 172.21.15.57:0/2031848621 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec100cb8b0 0x7fec101dfbf0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:04.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:04.099+0000 7fec16c17640 1 -- 172.21.15.57:0/2031848621 >> 172.21.15.57:0/2031848621 conn(0x7fec100cac70 msgr2=0x7fec10150220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:04.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:04.099+0000 7fec16c17640 1 -- 172.21.15.57:0/2031848621 shutdown_connections 2023-10-30T14:15:04.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:04.099+0000 7fec16c17640 1 -- 172.21.15.57:0/2031848621 wait complete. 2023-10-30T14:15:04.122 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:15:06.123 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:06.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.283+0000 7f76d6edc640 1 Processor -- start 2023-10-30T14:15:06.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.283+0000 7f76d6edc640 1 -- start start 2023-10-30T14:15:06.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.287+0000 7f76d6edc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76d015ac70 0x7f76d015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:06.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.287+0000 7f76d6edc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f76d006aeb0 0x7f76d0069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:06.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.287+0000 7f76d6edc640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d015dcc0 con 0x7f76d00c8eb0 2023-10-30T14:15:06.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.287+0000 7f76d6edc640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d0159800 con 0x7f76d006aeb0 2023-10-30T14:15:06.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.287+0000 7f76d6edc640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d0159d80 con 0x7f76d015ac70 2023-10-30T14:15:06.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.287+0000 7f76d56d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76d015ac70 0x7f76d015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:06.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.287+0000 7f76d66db640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f76d006aeb0 0x7f76d0069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:06.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.287+0000 7f76d66db640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f76d006aeb0 0x7f76d0069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:44182/0 (socket says 172.21.15.57:44182) 2023-10-30T14:15:06.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.287+0000 7f76d66db640 1 -- 172.21.15.57:0/944484795 learned_addr learned my addr 172.21.15.57:0/944484795 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:15:06.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.287+0000 7f76d56d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76d015ac70 0x7f76d015cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:44300/0 (socket says 172.21.15.57:44300) 2023-10-30T14:15:06.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.287+0000 7f76d5eda640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f76d00c8eb0 0x7f76d015a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:47982/0 (socket says 172.21.15.57:47982) 2023-10-30T14:15:06.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.287+0000 7f76d66db640 1 -- 172.21.15.57:0/944484795 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76d015ac70 msgr2=0x7f76d015cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:06.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.287+0000 7f76d66db640 1 --2- 172.21.15.57:0/944484795 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76d015ac70 0x7f76d015cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.287+0000 7f76d66db640 1 -- 172.21.15.57:0/944484795 >> v1:172.21.15.88:6789/0 conn(0x7f76d00c8eb0 legacy=0x7f76d015a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:06.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.287+0000 7f76d66db640 1 -- 172.21.15.57:0/944484795 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d015d420 con 0x7f76d006aeb0 2023-10-30T14:15:06.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.291+0000 7f76d66db640 1 --2- 172.21.15.57:0/944484795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76d006aeb0 0x7f76d0069e90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009e50 tx=0x7f76c801f1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b61598abaf60edaf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:06.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.291+0000 7f76d4ed8640 1 -- 172.21.15.57:0/944484795 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c80af400 con 0x7f76d006aeb0 2023-10-30T14:15:06.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.291+0000 7f76d4ed8640 1 -- 172.21.15.57:0/944484795 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c80ab050 con 0x7f76d006aeb0 2023-10-30T14:15:06.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.291+0000 7f76d4ed8640 1 -- 172.21.15.57:0/944484795 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c80af780 con 0x7f76d006aeb0 2023-10-30T14:15:06.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.291+0000 7f76d6edc640 1 -- 172.21.15.57:0/944484795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76d006aeb0 msgr2=0x7f76d0069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:06.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.291+0000 7f76d6edc640 1 --2- 172.21.15.57:0/944484795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76d006aeb0 0x7f76d0069e90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009e50 tx=0x7f76c801f1f0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.291+0000 7f76d6edc640 1 -- 172.21.15.57:0/944484795 shutdown_connections 2023-10-30T14:15:06.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.291+0000 7f76d6edc640 1 --2- 172.21.15.57:0/944484795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76d006aeb0 0x7f76d0069e90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.291+0000 7f76d6edc640 1 --2- 172.21.15.57:0/944484795 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76d015ac70 0x7f76d015cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.291+0000 7f76d6edc640 1 -- 172.21.15.57:0/944484795 >> 172.21.15.57:0/944484795 conn(0x7f76d00cef00 msgr2=0x7f76d00cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:06.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.291+0000 7f76d6edc640 1 -- 172.21.15.57:0/944484795 shutdown_connections 2023-10-30T14:15:06.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.291+0000 7f76d6edc640 1 -- 172.21.15.57:0/944484795 wait complete. 2023-10-30T14:15:06.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d6edc640 1 Processor -- start 2023-10-30T14:15:06.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d6edc640 1 -- start start 2023-10-30T14:15:06.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d6edc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76d006aeb0 0x7f76d00c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:06.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d6edc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76d00c8eb0 0x7f76d01f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:06.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d6edc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76d015ac70 0x7f76d01f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:06.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d6edc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d01e3230 con 0x7f76d015ac70 2023-10-30T14:15:06.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d6edc640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d00ca0e0 con 0x7f76d00c8eb0 2023-10-30T14:15:06.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d6edc640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d00ca330 con 0x7f76d006aeb0 2023-10-30T14:15:06.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d66db640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76d015ac70 0x7f76d01f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:06.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d5eda640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76d006aeb0 0x7f76d00c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:06.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d56d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76d00c8eb0 0x7f76d01f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:06.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d66db640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76d015ac70 0x7f76d01f2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53256/0 (socket says 172.21.15.57:53256) 2023-10-30T14:15:06.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d66db640 1 -- 172.21.15.57:0/2028114706 learned_addr learned my addr 172.21.15.57:0/2028114706 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:15:06.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d66db640 1 -- 172.21.15.57:0/2028114706 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76d006aeb0 msgr2=0x7f76d00c5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:06.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d66db640 1 --2- 172.21.15.57:0/2028114706 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76d006aeb0 0x7f76d00c5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d66db640 1 -- 172.21.15.57:0/2028114706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76d00c8eb0 msgr2=0x7f76d01f24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:06.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d66db640 1 --2- 172.21.15.57:0/2028114706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76d00c8eb0 0x7f76d01f24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d66db640 1 -- 172.21.15.57:0/2028114706 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c8009b30 con 0x7f76d015ac70 2023-10-30T14:15:06.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d66db640 1 --2- 172.21.15.57:0/2028114706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76d015ac70 0x7f76d01f2a10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f76c801ce20 tx=0x7f76c801bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:06.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76c67fc640 1 -- 172.21.15.57:0/2028114706 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c80be070 con 0x7f76d015ac70 2023-10-30T14:15:06.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76c67fc640 1 -- 172.21.15.57:0/2028114706 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c8009510 con 0x7f76d015ac70 2023-10-30T14:15:06.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76c67fc640 1 -- 172.21.15.57:0/2028114706 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c80afd10 con 0x7f76d015ac70 2023-10-30T14:15:06.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d6edc640 1 -- 172.21.15.57:0/2028114706 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d00c79c0 con 0x7f76d015ac70 2023-10-30T14:15:06.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.295+0000 7f76d6edc640 1 -- 172.21.15.57:0/2028114706 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d00c37f0 con 0x7f76d015ac70 2023-10-30T14:15:06.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.299+0000 7f76d6edc640 1 -- 172.21.15.57:0/2028114706 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d01f21f0 con 0x7f76d015ac70 2023-10-30T14:15:06.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.299+0000 7f76c67fc640 1 -- 172.21.15.57:0/2028114706 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f76c80b6040 con 0x7f76d015ac70 2023-10-30T14:15:06.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.299+0000 7f76c67fc640 1 --2- 172.21.15.57:0/2028114706 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76b40664a0 0x7f76b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:06.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.299+0000 7f76c67fc640 1 -- 172.21.15.57:0/2028114706 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f76c8121670 con 0x7f76d015ac70 2023-10-30T14:15:06.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.299+0000 7f76d5eda640 1 --2- 172.21.15.57:0/2028114706 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76b40664a0 0x7f76b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:06.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.299+0000 7f76d5eda640 1 --2- 172.21.15.57:0/2028114706 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76b40664a0 0x7f76b4068960 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f76cc000f70 tx=0x7f76cc007e50 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:06.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.403+0000 7f76c67fc640 1 -- 172.21.15.57:0/2028114706 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f76d01f21f0 con 0x7f76d015ac70 2023-10-30T14:15:06.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.603+0000 7f76d6edc640 1 -- 172.21.15.57:0/2028114706 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76d01f21f0 con 0x7f76d015ac70 2023-10-30T14:15:06.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.603+0000 7f76c67fc640 1 -- 172.21.15.57:0/2028114706 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5732 (secure 0 0 0) 0x7f76d01f21f0 con 0x7f76d015ac70 2023-10-30T14:15:06.608 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:15:06.608 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:15:04.606551+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"a","rank":2,"incarnation":13,"state":"up:resolve","state_seq":18,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:06.609 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 14 2023-10-30T14:15:06.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.607+0000 7f76d6edc640 1 -- 172.21.15.57:0/2028114706 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76b40664a0 msgr2=0x7f76b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:06.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.607+0000 7f76d6edc640 1 --2- 172.21.15.57:0/2028114706 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76b40664a0 0x7f76b4068960 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f76cc000f70 tx=0x7f76cc007e50 comp rx=0 tx=0).stop 2023-10-30T14:15:06.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.607+0000 7f76d6edc640 1 -- 172.21.15.57:0/2028114706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76d015ac70 msgr2=0x7f76d01f2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:06.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.607+0000 7f76d6edc640 1 --2- 172.21.15.57:0/2028114706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76d015ac70 0x7f76d01f2a10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f76c801ce20 tx=0x7f76c801bf90 comp rx=0 tx=0).stop 2023-10-30T14:15:06.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.607+0000 7f76d6edc640 1 -- 172.21.15.57:0/2028114706 shutdown_connections 2023-10-30T14:15:06.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.607+0000 7f76d6edc640 1 --2- 172.21.15.57:0/2028114706 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76b40664a0 0x7f76b4068960 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.607+0000 7f76d6edc640 1 --2- 172.21.15.57:0/2028114706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76d015ac70 0x7f76d01f2a10 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.607+0000 7f76d6edc640 1 --2- 172.21.15.57:0/2028114706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76d00c8eb0 0x7f76d01f24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.607+0000 7f76d6edc640 1 --2- 172.21.15.57:0/2028114706 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76d006aeb0 0x7f76d00c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.607+0000 7f76d6edc640 1 -- 172.21.15.57:0/2028114706 >> 172.21.15.57:0/2028114706 conn(0x7f76d00cef00 msgr2=0x7f76d0069f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:06.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.607+0000 7f76d6edc640 1 -- 172.21.15.57:0/2028114706 shutdown_connections 2023-10-30T14:15:06.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:06.607+0000 7f76d6edc640 1 -- 172.21.15.57:0/2028114706 wait complete. 2023-10-30T14:15:06.630 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:15:08.631 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:08.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.783+0000 7f85ad4c9640 1 Processor -- start 2023-10-30T14:15:08.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.783+0000 7f85ad4c9640 1 -- start start 2023-10-30T14:15:08.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.783+0000 7f85ad4c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f85a8153d50 0x7f85a80cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:08.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.783+0000 7f85ad4c9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f85a815acd0 0x7f85a80cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:08.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.783+0000 7f85ad4c9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a80cc900 con 0x7f85a8153970 2023-10-30T14:15:08.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.783+0000 7f85ad4c9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85a80cbfa0 con 0x7f85a815acd0 2023-10-30T14:15:08.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.783+0000 7f85ad4c9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85a80cb040 con 0x7f85a8153d50 2023-10-30T14:15:08.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.783+0000 7f85a6ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f85a8153970 0x7f85a815d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:57642/0 (socket says 172.21.15.57:57642) 2023-10-30T14:15:08.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.783+0000 7f85a6ffd640 1 -- 172.21.15.57:0/3901703372 learned_addr learned my addr 172.21.15.57:0/3901703372 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:15:08.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.783+0000 7f85a77fe640 1 --2- 172.21.15.57:0/3901703372 >> v2:172.21.15.57:3300/0 conn(0x7f85a815acd0 0x7f85a80cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:08.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.783+0000 7f85a67fc640 1 --2- 172.21.15.57:0/3901703372 >> v2:172.21.15.88:3300/0 conn(0x7f85a8153d50 0x7f85a80cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:08.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.783+0000 7f85a77fe640 1 -- 172.21.15.57:0/3901703372 >> v2:172.21.15.88:3300/0 conn(0x7f85a8153d50 msgr2=0x7f85a80cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:08.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.783+0000 7f85a77fe640 1 --2- 172.21.15.57:0/3901703372 >> v2:172.21.15.88:3300/0 conn(0x7f85a8153d50 0x7f85a80cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:08.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.783+0000 7f85a77fe640 1 -- 172.21.15.57:0/3901703372 >> v1:172.21.15.57:6789/0 conn(0x7f85a8153970 legacy=0x7f85a815d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:08.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.783+0000 7f85a77fe640 1 -- 172.21.15.57:0/3901703372 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a80cb320 con 0x7f85a815acd0 2023-10-30T14:15:08.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.783+0000 7f85a5ffb640 1 -- 172.21.15.57:0/3901703372 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246655516 0 0) 0x7f85a80cc900 con 0x7f85a8153970 2023-10-30T14:15:08.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.787+0000 7f85a77fe640 1 --2- 172.21.15.57:0/3901703372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85a815acd0 0x7f85a80cefa0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f85a806cb40 tx=0x7f85a001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=445c1c7d65357fd9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:08.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.787+0000 7f85a5ffb640 1 -- 172.21.15.57:0/3901703372 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a00a59d0 con 0x7f85a815acd0 2023-10-30T14:15:08.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.787+0000 7f85a5ffb640 1 -- 172.21.15.57:0/3901703372 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a00a5b30 con 0x7f85a815acd0 2023-10-30T14:15:08.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.787+0000 7f85a5ffb640 1 -- 172.21.15.57:0/3901703372 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a00ad440 con 0x7f85a815acd0 2023-10-30T14:15:08.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.787+0000 7f85ad4c9640 1 -- 172.21.15.57:0/3901703372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85a815acd0 msgr2=0x7f85a80cefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:08.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.787+0000 7f85ad4c9640 1 --2- 172.21.15.57:0/3901703372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85a815acd0 0x7f85a80cefa0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f85a806cb40 tx=0x7f85a001c910 comp rx=0 tx=0).stop 2023-10-30T14:15:08.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.787+0000 7f85ad4c9640 1 -- 172.21.15.57:0/3901703372 shutdown_connections 2023-10-30T14:15:08.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.787+0000 7f85ad4c9640 1 --2- 172.21.15.57:0/3901703372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85a815acd0 0x7f85a80cefa0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:08.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.787+0000 7f85ad4c9640 1 --2- 172.21.15.57:0/3901703372 >> v2:172.21.15.88:3300/0 conn(0x7f85a8153d50 0x7f85a80cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:08.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.787+0000 7f85ad4c9640 1 -- 172.21.15.57:0/3901703372 >> 172.21.15.57:0/3901703372 conn(0x7f85a806bf60 msgr2=0x7f85a806c370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:08.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.787+0000 7f85ad4c9640 1 -- 172.21.15.57:0/3901703372 shutdown_connections 2023-10-30T14:15:08.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.787+0000 7f85ad4c9640 1 -- 172.21.15.57:0/3901703372 wait complete. 2023-10-30T14:15:08.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85ad4c9640 1 Processor -- start 2023-10-30T14:15:08.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85ad4c9640 1 -- start start 2023-10-30T14:15:08.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85ad4c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85a8153970 0x7f85a814d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:08.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85ad4c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85a8153d50 0x7f85a81f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:08.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85ad4c9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f85a815acd0 0x7f85a81f3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:08.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85ad4c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a814a8e0 con 0x7f85a8153970 2023-10-30T14:15:08.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85ad4c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a80c58f0 con 0x7f85a8153d50 2023-10-30T14:15:08.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85ad4c9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a81453d0 con 0x7f85a815acd0 2023-10-30T14:15:08.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85a6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85a8153970 0x7f85a814d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:08.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85a6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85a8153970 0x7f85a814d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53268/0 (socket says 172.21.15.57:53268) 2023-10-30T14:15:08.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85a6ffd640 1 -- 172.21.15.57:0/3097194814 learned_addr learned my addr 172.21.15.57:0/3097194814 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:15:08.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85a67fc640 1 --2- 172.21.15.57:0/3097194814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85a8153d50 0x7f85a81f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:08.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85a77fe640 1 --2- 172.21.15.57:0/3097194814 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f85a815acd0 0x7f85a81f3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:08.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85a6ffd640 1 -- 172.21.15.57:0/3097194814 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f85a815acd0 msgr2=0x7f85a81f3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:08.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85a6ffd640 1 --2- 172.21.15.57:0/3097194814 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f85a815acd0 0x7f85a81f3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:08.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85a6ffd640 1 -- 172.21.15.57:0/3097194814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85a8153d50 msgr2=0x7f85a81f3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:08.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85a6ffd640 1 --2- 172.21.15.57:0/3097194814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85a8153d50 0x7f85a81f3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:08.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85a6ffd640 1 -- 172.21.15.57:0/3097194814 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a0009a70 con 0x7f85a8153970 2023-10-30T14:15:08.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85a77fe640 1 --2- 172.21.15.57:0/3097194814 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f85a815acd0 0x7f85a81f3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:15:08.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85a6ffd640 1 --2- 172.21.15.57:0/3097194814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85a8153970 0x7f85a814d150 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f85980011f0 tx=0x7f859800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:08.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f858f7fe640 1 -- 172.21.15.57:0/3097194814 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85980074e0 con 0x7f85a8153970 2023-10-30T14:15:08.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f858f7fe640 1 -- 172.21.15.57:0/3097194814 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8598007640 con 0x7f85a8153970 2023-10-30T14:15:08.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f858f7fe640 1 -- 172.21.15.57:0/3097194814 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8598004a10 con 0x7f85a8153970 2023-10-30T14:15:08.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.791+0000 7f85ad4c9640 1 -- 172.21.15.57:0/3097194814 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a8144b30 con 0x7f85a8153970 2023-10-30T14:15:08.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.795+0000 7f85ad4c9640 1 -- 172.21.15.57:0/3097194814 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a81450f0 con 0x7f85a8153970 2023-10-30T14:15:08.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.795+0000 7f85ad4c9640 1 -- 172.21.15.57:0/3097194814 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a8067cb0 con 0x7f85a8153970 2023-10-30T14:15:08.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.795+0000 7f858f7fe640 1 -- 172.21.15.57:0/3097194814 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8598004b70 con 0x7f85a8153970 2023-10-30T14:15:08.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.795+0000 7f858f7fe640 1 --2- 172.21.15.57:0/3097194814 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85940664a0 0x7f8594068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:08.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.795+0000 7f858f7fe640 1 -- 172.21.15.57:0/3097194814 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f8598083250 con 0x7f85a8153970 2023-10-30T14:15:08.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.795+0000 7f85a67fc640 1 --2- 172.21.15.57:0/3097194814 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85940664a0 0x7f8594068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:08.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.799+0000 7f85a67fc640 1 --2- 172.21.15.57:0/3097194814 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85940664a0 0x7f8594068960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f85a80b3ca0 tx=0x7f859c009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:08.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:08.803+0000 7f858f7fe640 1 -- 172.21.15.57:0/3097194814 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8598050600 con 0x7f85a8153970 2023-10-30T14:15:09.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:09.023+0000 7f85ad4c9640 1 -- 172.21.15.57:0/3097194814 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85a8067540 con 0x7f85a8153970 2023-10-30T14:15:09.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:09.023+0000 7f858f7fe640 1 -- 172.21.15.57:0/3097194814 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5732 (secure 0 0 0) 0x7f8598018020 con 0x7f85a8153970 2023-10-30T14:15:09.028 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:15:09.028 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:15:04.606551+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"a","rank":2,"incarnation":13,"state":"up:resolve","state_seq":18,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:09.029 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 14 2023-10-30T14:15:09.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:09.027+0000 7f85ad4c9640 1 -- 172.21.15.57:0/3097194814 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85940664a0 msgr2=0x7f8594068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:09.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:09.027+0000 7f85ad4c9640 1 --2- 172.21.15.57:0/3097194814 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85940664a0 0x7f8594068960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f85a80b3ca0 tx=0x7f859c009190 comp rx=0 tx=0).stop 2023-10-30T14:15:09.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:09.027+0000 7f85ad4c9640 1 -- 172.21.15.57:0/3097194814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85a8153970 msgr2=0x7f85a814d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:09.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:09.027+0000 7f85ad4c9640 1 --2- 172.21.15.57:0/3097194814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85a8153970 0x7f85a814d150 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f85980011f0 tx=0x7f859800c9b0 comp rx=0 tx=0).stop 2023-10-30T14:15:09.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:09.027+0000 7f85ad4c9640 1 -- 172.21.15.57:0/3097194814 shutdown_connections 2023-10-30T14:15:09.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:09.027+0000 7f85ad4c9640 1 --2- 172.21.15.57:0/3097194814 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85940664a0 0x7f8594068960 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:09.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:09.027+0000 7f85ad4c9640 1 --2- 172.21.15.57:0/3097194814 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f85a815acd0 0x7f85a81f3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:09.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:09.027+0000 7f85ad4c9640 1 --2- 172.21.15.57:0/3097194814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85a8153d50 0x7f85a81f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:09.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:09.027+0000 7f85ad4c9640 1 --2- 172.21.15.57:0/3097194814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85a8153970 0x7f85a814d150 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:09.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:09.027+0000 7f85ad4c9640 1 -- 172.21.15.57:0/3097194814 >> 172.21.15.57:0/3097194814 conn(0x7f85a806bf60 msgr2=0x7f85a80ccf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:09.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:09.027+0000 7f85ad4c9640 1 -- 172.21.15.57:0/3097194814 shutdown_connections 2023-10-30T14:15:09.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:09.027+0000 7f85ad4c9640 1 -- 172.21.15.57:0/3097194814 wait complete. 2023-10-30T14:15:09.057 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:15:11.059 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:11.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.223+0000 7fc661c2f640 1 Processor -- start 2023-10-30T14:15:11.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.223+0000 7fc661c2f640 1 -- start start 2023-10-30T14:15:11.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.227+0000 7fc661c2f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc65c0c8eb0 0x7fc65c15cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:11.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.227+0000 7fc661c2f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc65c15ac70 0x7fc65c15a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:11.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.227+0000 7fc661c2f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc65c06aeb0 0x7fc65c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:11.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.227+0000 7fc661c2f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc65c159800 con 0x7fc65c0c8eb0 2023-10-30T14:15:11.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.227+0000 7fc661c2f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc65c15dcc0 con 0x7fc65c15ac70 2023-10-30T14:15:11.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.227+0000 7fc661c2f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c159d80 con 0x7fc65c06aeb0 2023-10-30T14:15:11.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.227+0000 7fc660c2d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc65c0c8eb0 0x7fc65c15cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:11.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.227+0000 7fc660c2d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc65c0c8eb0 0x7fc65c15cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53274/0 (socket says 172.21.15.57:53274) 2023-10-30T14:15:11.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.227+0000 7fc660c2d640 1 -- 172.21.15.57:0/2225198153 learned_addr learned my addr 172.21.15.57:0/2225198153 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:15:11.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.227+0000 7fc65bfff640 1 --2- 172.21.15.57:0/2225198153 >> v2:172.21.15.88:3300/0 conn(0x7fc65c15ac70 0x7fc65c15a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:11.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.227+0000 7fc66142e640 1 --2- 172.21.15.57:0/2225198153 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc65c06aeb0 0x7fc65c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:11.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.227+0000 7fc660c2d640 1 -- 172.21.15.57:0/2225198153 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc65c06aeb0 msgr2=0x7fc65c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:11.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.227+0000 7fc660c2d640 1 --2- 172.21.15.57:0/2225198153 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc65c06aeb0 0x7fc65c069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:11.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.227+0000 7fc660c2d640 1 -- 172.21.15.57:0/2225198153 >> v2:172.21.15.88:3300/0 conn(0x7fc65c15ac70 msgr2=0x7fc65c15a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:11.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.227+0000 7fc660c2d640 1 --2- 172.21.15.57:0/2225198153 >> v2:172.21.15.88:3300/0 conn(0x7fc65c15ac70 0x7fc65c15a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:11.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.227+0000 7fc660c2d640 1 -- 172.21.15.57:0/2225198153 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c15a750 con 0x7fc65c0c8eb0 2023-10-30T14:15:11.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.231+0000 7fc660c2d640 1 --2- 172.21.15.57:0/2225198153 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c0c8eb0 0x7fc65c15cd70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc65c0b4130 tx=0x7fc65001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e8b3c59f422d72e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:11.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.231+0000 7fc65b7fe640 1 -- 172.21.15.57:0/2225198153 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6500a89b0 con 0x7fc65c0c8eb0 2023-10-30T14:15:11.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.231+0000 7fc65b7fe640 1 -- 172.21.15.57:0/2225198153 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6500a8b10 con 0x7fc65c0c8eb0 2023-10-30T14:15:11.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.231+0000 7fc65b7fe640 1 -- 172.21.15.57:0/2225198153 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6500a8ea0 con 0x7fc65c0c8eb0 2023-10-30T14:15:11.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.231+0000 7fc661c2f640 1 -- 172.21.15.57:0/2225198153 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c0c8eb0 msgr2=0x7fc65c15cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:11.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.231+0000 7fc661c2f640 1 --2- 172.21.15.57:0/2225198153 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c0c8eb0 0x7fc65c15cd70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc65c0b4130 tx=0x7fc65001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:15:11.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.231+0000 7fc661c2f640 1 -- 172.21.15.57:0/2225198153 shutdown_connections 2023-10-30T14:15:11.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.231+0000 7fc661c2f640 1 --2- 172.21.15.57:0/2225198153 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc65c06aeb0 0x7fc65c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:11.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.231+0000 7fc661c2f640 1 --2- 172.21.15.57:0/2225198153 >> v2:172.21.15.88:3300/0 conn(0x7fc65c15ac70 0x7fc65c15a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:11.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.231+0000 7fc661c2f640 1 --2- 172.21.15.57:0/2225198153 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c0c8eb0 0x7fc65c15cd70 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:11.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.231+0000 7fc661c2f640 1 -- 172.21.15.57:0/2225198153 >> 172.21.15.57:0/2225198153 conn(0x7fc65c0cef00 msgr2=0x7fc65c0cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:11.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.231+0000 7fc661c2f640 1 -- 172.21.15.57:0/2225198153 shutdown_connections 2023-10-30T14:15:11.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.231+0000 7fc661c2f640 1 -- 172.21.15.57:0/2225198153 wait complete. 2023-10-30T14:15:11.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc661c2f640 1 Processor -- start 2023-10-30T14:15:11.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc661c2f640 1 -- start start 2023-10-30T14:15:11.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc661c2f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc65c06aeb0 0x7fc65c1eea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:11.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc661c2f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc65c0c8eb0 0x7fc65c14f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:11.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc661c2f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c15ac70 0x7fc65c0b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:11.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc661c2f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c0c8120 con 0x7fc65c15ac70 2023-10-30T14:15:11.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc661c2f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c154170 con 0x7fc65c0c8eb0 2023-10-30T14:15:11.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc661c2f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c154730 con 0x7fc65c06aeb0 2023-10-30T14:15:11.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc66142e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c15ac70 0x7fc65c0b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:11.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc66142e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c15ac70 0x7fc65c0b2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53286/0 (socket says 172.21.15.57:53286) 2023-10-30T14:15:11.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc66142e640 1 -- 172.21.15.57:0/1646869417 learned_addr learned my addr 172.21.15.57:0/1646869417 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:15:11.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc660c2d640 1 --2- 172.21.15.57:0/1646869417 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc65c06aeb0 0x7fc65c1eea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:11.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc65bfff640 1 --2- 172.21.15.57:0/1646869417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc65c0c8eb0 0x7fc65c14f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:11.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc66142e640 1 -- 172.21.15.57:0/1646869417 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc65c06aeb0 msgr2=0x7fc65c1eea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:11.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc66142e640 1 --2- 172.21.15.57:0/1646869417 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc65c06aeb0 0x7fc65c1eea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:11.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc66142e640 1 -- 172.21.15.57:0/1646869417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc65c0c8eb0 msgr2=0x7fc65c14f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:11.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc66142e640 1 --2- 172.21.15.57:0/1646869417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc65c0c8eb0 0x7fc65c14f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:11.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc66142e640 1 -- 172.21.15.57:0/1646869417 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6500094d0 con 0x7fc65c15ac70 2023-10-30T14:15:11.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc65bfff640 1 --2- 172.21.15.57:0/1646869417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc65c0c8eb0 0x7fc65c14f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:15:11.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc66142e640 1 --2- 172.21.15.57:0/1646869417 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c15ac70 0x7fc65c0b2e50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc654007d20 tx=0x7fc65400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:11.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc6597fa640 1 -- 172.21.15.57:0/1646869417 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc654014420 con 0x7fc65c15ac70 2023-10-30T14:15:11.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc6597fa640 1 -- 172.21.15.57:0/1646869417 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6540145a0 con 0x7fc65c15ac70 2023-10-30T14:15:11.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc661c2f640 1 -- 172.21.15.57:0/1646869417 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65c1eef60 con 0x7fc65c15ac70 2023-10-30T14:15:11.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc6597fa640 1 -- 172.21.15.57:0/1646869417 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc654014950 con 0x7fc65c15ac70 2023-10-30T14:15:11.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc661c2f640 1 -- 172.21.15.57:0/1646869417 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65c1ef4d0 con 0x7fc65c15ac70 2023-10-30T14:15:11.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.235+0000 7fc661c2f640 1 -- 172.21.15.57:0/1646869417 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc65c1548a0 con 0x7fc65c15ac70 2023-10-30T14:15:11.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.239+0000 7fc6597fa640 1 -- 172.21.15.57:0/1646869417 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc654014ab0 con 0x7fc65c15ac70 2023-10-30T14:15:11.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.239+0000 7fc6597fa640 1 --2- 172.21.15.57:0/1646869417 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc6400664a0 0x7fc640068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:11.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.239+0000 7fc6597fa640 1 -- 172.21.15.57:0/1646869417 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fc6540847f0 con 0x7fc65c15ac70 2023-10-30T14:15:11.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.239+0000 7fc660c2d640 1 --2- 172.21.15.57:0/1646869417 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc6400664a0 0x7fc640068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:11.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.239+0000 7fc660c2d640 1 --2- 172.21.15.57:0/1646869417 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc6400664a0 0x7fc640068960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc6500b2000 tx=0x7fc650009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:11.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.243+0000 7fc6597fa640 1 -- 172.21.15.57:0/1646869417 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc654051b00 con 0x7fc65c15ac70 2023-10-30T14:15:11.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.443+0000 7fc661c2f640 1 -- 172.21.15.57:0/1646869417 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc65c04e170 con 0x7fc65c15ac70 2023-10-30T14:15:11.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.443+0000 7fc6597fa640 1 -- 172.21.15.57:0/1646869417 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5734 (secure 0 0 0) 0x7fc654055120 con 0x7fc65c15ac70 2023-10-30T14:15:11.445 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:15:11.446 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:15:10.601489+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"a","rank":2,"incarnation":13,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:11.446 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 15 2023-10-30T14:15:11.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.443+0000 7fc661c2f640 1 -- 172.21.15.57:0/1646869417 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc6400664a0 msgr2=0x7fc640068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:11.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.443+0000 7fc661c2f640 1 --2- 172.21.15.57:0/1646869417 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc6400664a0 0x7fc640068960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc6500b2000 tx=0x7fc650009040 comp rx=0 tx=0).stop 2023-10-30T14:15:11.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.443+0000 7fc661c2f640 1 -- 172.21.15.57:0/1646869417 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c15ac70 msgr2=0x7fc65c0b2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:11.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.443+0000 7fc661c2f640 1 --2- 172.21.15.57:0/1646869417 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c15ac70 0x7fc65c0b2e50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc654007d20 tx=0x7fc65400e4b0 comp rx=0 tx=0).stop 2023-10-30T14:15:11.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.443+0000 7fc661c2f640 1 -- 172.21.15.57:0/1646869417 shutdown_connections 2023-10-30T14:15:11.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.443+0000 7fc661c2f640 1 --2- 172.21.15.57:0/1646869417 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc6400664a0 0x7fc640068960 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:11.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.443+0000 7fc661c2f640 1 --2- 172.21.15.57:0/1646869417 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c15ac70 0x7fc65c0b2e50 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:11.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.443+0000 7fc661c2f640 1 --2- 172.21.15.57:0/1646869417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc65c0c8eb0 0x7fc65c14f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:11.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.443+0000 7fc661c2f640 1 --2- 172.21.15.57:0/1646869417 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc65c06aeb0 0x7fc65c1eea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:11.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.443+0000 7fc661c2f640 1 -- 172.21.15.57:0/1646869417 >> 172.21.15.57:0/1646869417 conn(0x7fc65c0cef00 msgr2=0x7fc65c0c9530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:11.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.443+0000 7fc661c2f640 1 -- 172.21.15.57:0/1646869417 shutdown_connections 2023-10-30T14:15:11.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:11.443+0000 7fc661c2f640 1 -- 172.21.15.57:0/1646869417 wait complete. 2023-10-30T14:15:11.467 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:15:13.468 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:13.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.610+0000 7f5b0671d640 1 Processor -- start 2023-10-30T14:15:13.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.610+0000 7f5b0671d640 1 -- start start 2023-10-30T14:15:13.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.610+0000 7f5b0671d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b000c4a10 0x7f5b0015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:13.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.610+0000 7f5b0671d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5b000c42a0 0x7f5b000ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:13.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.610+0000 7f5b0671d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b00159f20 con 0x7f5b00153c80 2023-10-30T14:15:13.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.610+0000 7f5b0671d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b0015dec0 con 0x7f5b000c42a0 2023-10-30T14:15:13.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.610+0000 7f5b0671d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b001599a0 con 0x7f5b000c4a10 2023-10-30T14:15:13.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.610+0000 7f5b04cba640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5b000c42a0 0x7f5b000ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:13.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.610+0000 7f5b04cba640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5b000c42a0 0x7f5b000ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53294/0 (socket says 172.21.15.57:53294) 2023-10-30T14:15:13.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.610+0000 7f5aff7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b000c4a10 0x7f5b0015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:13.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.610+0000 7f5b04cba640 1 -- 172.21.15.57:0/3824999769 learned_addr learned my addr 172.21.15.57:0/3824999769 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:15:13.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.610+0000 7f5b04cba640 1 -- 172.21.15.57:0/3824999769 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b000c4a10 msgr2=0x7f5b0015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:13.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.610+0000 7f5b04cba640 1 --2- 172.21.15.57:0/3824999769 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b000c4a10 0x7f5b0015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:13.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.610+0000 7f5b04cba640 1 -- 172.21.15.57:0/3824999769 >> v1:172.21.15.88:6789/0 conn(0x7f5b00153c80 legacy=0x7f5b0015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:13.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.610+0000 7f5b04cba640 1 -- 172.21.15.57:0/3824999769 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0015d620 con 0x7f5b000c42a0 2023-10-30T14:15:13.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.614+0000 7f5b04cba640 1 --2- 172.21.15.57:0/3824999769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b000c42a0 0x7f5b000ca1e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009e50 tx=0x7f5af801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67be6b986e1bd63e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:13.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.614+0000 7f5afeffd640 1 -- 172.21.15.57:0/3824999769 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af80af400 con 0x7f5b000c42a0 2023-10-30T14:15:13.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.614+0000 7f5afeffd640 1 -- 172.21.15.57:0/3824999769 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af80ab050 con 0x7f5b000c42a0 2023-10-30T14:15:13.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.614+0000 7f5afeffd640 1 -- 172.21.15.57:0/3824999769 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af80af780 con 0x7f5b000c42a0 2023-10-30T14:15:13.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.614+0000 7f5b0671d640 1 -- 172.21.15.57:0/3824999769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b000c42a0 msgr2=0x7f5b000ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:13.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.614+0000 7f5b0671d640 1 --2- 172.21.15.57:0/3824999769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b000c42a0 0x7f5b000ca1e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009e50 tx=0x7f5af801f1f0 comp rx=0 tx=0).stop 2023-10-30T14:15:13.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.614+0000 7f5b0671d640 1 -- 172.21.15.57:0/3824999769 shutdown_connections 2023-10-30T14:15:13.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.614+0000 7f5b0671d640 1 --2- 172.21.15.57:0/3824999769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b000c42a0 0x7f5b000ca1e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:13.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.614+0000 7f5b0671d640 1 --2- 172.21.15.57:0/3824999769 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b000c4a10 0x7f5b0015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:13.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.614+0000 7f5b0671d640 1 -- 172.21.15.57:0/3824999769 >> 172.21.15.57:0/3824999769 conn(0x7f5b000715c0 msgr2=0x7f5b000719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:13.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.614+0000 7f5b0671d640 1 -- 172.21.15.57:0/3824999769 shutdown_connections 2023-10-30T14:15:13.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.614+0000 7f5b0671d640 1 -- 172.21.15.57:0/3824999769 wait complete. 2023-10-30T14:15:13.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5b0671d640 1 Processor -- start 2023-10-30T14:15:13.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5b0671d640 1 -- start start 2023-10-30T14:15:13.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5b0671d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b000c42a0 0x7f5b0014ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:13.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5b0671d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b000c4a10 0x7f5b001f2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:13.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5b0671d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b00153c80 0x7f5b00148b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:13.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5b0671d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0014ef90 con 0x7f5b000c42a0 2023-10-30T14:15:13.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5b0671d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b000cf080 con 0x7f5b00153c80 2023-10-30T14:15:13.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5b0671d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b001f2fc0 con 0x7f5b000c4a10 2023-10-30T14:15:13.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5afffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b000c42a0 0x7f5b0014ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:13.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5afffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b000c42a0 0x7f5b0014ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53310/0 (socket says 172.21.15.57:53310) 2023-10-30T14:15:13.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5afffff640 1 -- 172.21.15.57:0/3257932059 learned_addr learned my addr 172.21.15.57:0/3257932059 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:15:13.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5aff7fe640 1 --2- 172.21.15.57:0/3257932059 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b000c4a10 0x7f5b001f2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:13.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5b04cba640 1 --2- 172.21.15.57:0/3257932059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b00153c80 0x7f5b00148b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:13.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5afffff640 1 -- 172.21.15.57:0/3257932059 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b000c4a10 msgr2=0x7f5b001f2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:13.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5afffff640 1 --2- 172.21.15.57:0/3257932059 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b000c4a10 0x7f5b001f2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:13.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5afffff640 1 -- 172.21.15.57:0/3257932059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b00153c80 msgr2=0x7f5b00148b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:13.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5afffff640 1 --2- 172.21.15.57:0/3257932059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b00153c80 0x7f5b00148b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:13.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5afffff640 1 -- 172.21.15.57:0/3257932059 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af8009b30 con 0x7f5b000c42a0 2023-10-30T14:15:13.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5b04cba640 1 --2- 172.21.15.57:0/3257932059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b00153c80 0x7f5b00148b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:15:13.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5afffff640 1 --2- 172.21.15.57:0/3257932059 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b000c42a0 0x7f5b0014ea50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5af000c3f0 tx=0x7f5af000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:13.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5afcff9640 1 -- 172.21.15.57:0/3257932059 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af00052d0 con 0x7f5b000c42a0 2023-10-30T14:15:13.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.618+0000 7f5afcff9640 1 -- 172.21.15.57:0/3257932059 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af0005430 con 0x7f5b000c42a0 2023-10-30T14:15:13.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.622+0000 7f5b0671d640 1 -- 172.21.15.57:0/3257932059 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b001f2d20 con 0x7f5b000c42a0 2023-10-30T14:15:13.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.622+0000 7f5afcff9640 1 -- 172.21.15.57:0/3257932059 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af0003840 con 0x7f5b000c42a0 2023-10-30T14:15:13.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.622+0000 7f5b0671d640 1 -- 172.21.15.57:0/3257932059 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b00159690 con 0x7f5b000c42a0 2023-10-30T14:15:13.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.622+0000 7f5b0671d640 1 -- 172.21.15.57:0/3257932059 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b0004e170 con 0x7f5b000c42a0 2023-10-30T14:15:13.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.622+0000 7f5afcff9640 1 -- 172.21.15.57:0/3257932059 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5af00039a0 con 0x7f5b000c42a0 2023-10-30T14:15:13.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.622+0000 7f5afcff9640 1 --2- 172.21.15.57:0/3257932059 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5aec0664a0 0x7f5aec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:13.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.622+0000 7f5afcff9640 1 -- 172.21.15.57:0/3257932059 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f5af0083380 con 0x7f5b000c42a0 2023-10-30T14:15:13.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.626+0000 7f5aff7fe640 1 --2- 172.21.15.57:0/3257932059 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5aec0664a0 0x7f5aec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:13.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.626+0000 7f5aff7fe640 1 --2- 172.21.15.57:0/3257932059 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5aec0664a0 0x7f5aec068960 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5b00149b70 tx=0x7f5af4009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:13.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.626+0000 7f5afcff9640 1 -- 172.21.15.57:0/3257932059 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5af0050870 con 0x7f5b000c42a0 2023-10-30T14:15:13.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.818+0000 7f5b0671d640 1 -- 172.21.15.57:0/3257932059 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b00159b80 con 0x7f5b000c42a0 2023-10-30T14:15:13.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.822+0000 7f5afcff9640 1 -- 172.21.15.57:0/3257932059 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f5af0053e90 con 0x7f5b000c42a0 2023-10-30T14:15:13.826 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:15:13.826 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:15:10.601489+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"a","rank":2,"incarnation":13,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:13.826 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 15 2023-10-30T14:15:13.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.822+0000 7f5b0671d640 1 -- 172.21.15.57:0/3257932059 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5aec0664a0 msgr2=0x7f5aec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:13.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.822+0000 7f5b0671d640 1 --2- 172.21.15.57:0/3257932059 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5aec0664a0 0x7f5aec068960 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5b00149b70 tx=0x7f5af4009190 comp rx=0 tx=0).stop 2023-10-30T14:15:13.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.822+0000 7f5b0671d640 1 -- 172.21.15.57:0/3257932059 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b000c42a0 msgr2=0x7f5b0014ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:13.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.822+0000 7f5b0671d640 1 --2- 172.21.15.57:0/3257932059 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b000c42a0 0x7f5b0014ea50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5af000c3f0 tx=0x7f5af000c8b0 comp rx=0 tx=0).stop 2023-10-30T14:15:13.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.822+0000 7f5b0671d640 1 -- 172.21.15.57:0/3257932059 shutdown_connections 2023-10-30T14:15:13.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.822+0000 7f5b0671d640 1 --2- 172.21.15.57:0/3257932059 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5aec0664a0 0x7f5aec068960 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:13.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.822+0000 7f5b0671d640 1 --2- 172.21.15.57:0/3257932059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b00153c80 0x7f5b00148b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:13.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.822+0000 7f5b0671d640 1 --2- 172.21.15.57:0/3257932059 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b000c4a10 0x7f5b001f2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:13.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.822+0000 7f5b0671d640 1 --2- 172.21.15.57:0/3257932059 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b000c42a0 0x7f5b0014ea50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:13.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.822+0000 7f5b0671d640 1 -- 172.21.15.57:0/3257932059 >> 172.21.15.57:0/3257932059 conn(0x7f5b000715c0 msgr2=0x7f5b001588e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:13.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.822+0000 7f5b0671d640 1 -- 172.21.15.57:0/3257932059 shutdown_connections 2023-10-30T14:15:13.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:13.822+0000 7f5b0671d640 1 -- 172.21.15.57:0/3257932059 wait complete. 2023-10-30T14:15:13.852 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 15, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 15, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4262": { "addr": "172.21.15.57:6834/3701655528", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 3701655528, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 3701655528, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 13, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:reconnect", "state_seq": 20 }, "gid_4266": { "addr": "172.21.15.57:6836/2693548867", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2693548867, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2693548867, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4266, "incarnation": 9, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4282": { "addr": "172.21.15.88:6839/2945854794", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 2945854794, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2945854794, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 24, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:15:10.601489+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 4266, "mds_2": 4262 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:15:15.852 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:16.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:15.998+0000 7fb441d63640 1 Processor -- start 2023-10-30T14:15:16.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:15.998+0000 7fb441d63640 1 -- start start 2023-10-30T14:15:16.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:15.998+0000 7fb441d63640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb43c1515c0 0x7fb43c0caf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:16.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:15.998+0000 7fb441d63640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb43c0c62d0 0x7fb43c070d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:16.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:15.998+0000 7fb441d63640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb43c157590 con 0x7fb43c0cb490 2023-10-30T14:15:16.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:15.998+0000 7fb441d63640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb43c157380 con 0x7fb43c0c62d0 2023-10-30T14:15:16.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:15.998+0000 7fb441d63640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c157870 con 0x7fb43c1515c0 2023-10-30T14:15:16.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:15.998+0000 7fb441562640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb43c0c62d0 0x7fb43c070d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:16.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:15.998+0000 7fb440d61640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb43c1515c0 0x7fb43c0caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:16.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:15.998+0000 7fb43bfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb43c0cb490 0x7fb43c0c5bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:44370/0 (socket says 172.21.15.57:44370) 2023-10-30T14:15:16.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:15.998+0000 7fb43bfff640 1 -- 172.21.15.57:0/3603684564 learned_addr learned my addr 172.21.15.57:0/3603684564 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:15:16.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:15.998+0000 7fb441562640 1 -- 172.21.15.57:0/3603684564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb43c1515c0 msgr2=0x7fb43c0caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:16.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:15.998+0000 7fb441562640 1 --2- 172.21.15.57:0/3603684564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb43c1515c0 0x7fb43c0caf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:16.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:15.998+0000 7fb441562640 1 -- 172.21.15.57:0/3603684564 >> v1:172.21.15.88:6789/0 conn(0x7fb43c0cb490 legacy=0x7fb43c0c5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:16.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:15.998+0000 7fb43b7fe640 1 -- 172.21.15.57:0/3603684564 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117164042 0 0) 0x7fb43c157590 con 0x7fb43c0cb490 2023-10-30T14:15:16.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:15.998+0000 7fb441562640 1 -- 172.21.15.57:0/3603684564 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43c157b50 con 0x7fb43c0c62d0 2023-10-30T14:15:16.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.006+0000 7fb441562640 1 --2- 172.21.15.57:0/3603684564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb43c0c62d0 0x7fb43c070d00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb434009d90 tx=0x7fb43401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=828e780ca49f6aa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:16.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.006+0000 7fb43b7fe640 1 -- 172.21.15.57:0/3603684564 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4340af600 con 0x7fb43c0c62d0 2023-10-30T14:15:16.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.006+0000 7fb43b7fe640 1 -- 172.21.15.57:0/3603684564 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4340ab050 con 0x7fb43c0c62d0 2023-10-30T14:15:16.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.006+0000 7fb43b7fe640 1 -- 172.21.15.57:0/3603684564 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4340af980 con 0x7fb43c0c62d0 2023-10-30T14:15:16.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.006+0000 7fb441d63640 1 -- 172.21.15.57:0/3603684564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb43c0c62d0 msgr2=0x7fb43c070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:16.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.006+0000 7fb441d63640 1 --2- 172.21.15.57:0/3603684564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb43c0c62d0 0x7fb43c070d00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb434009d90 tx=0x7fb43401f130 comp rx=0 tx=0).stop 2023-10-30T14:15:16.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.006+0000 7fb441d63640 1 -- 172.21.15.57:0/3603684564 shutdown_connections 2023-10-30T14:15:16.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.006+0000 7fb441d63640 1 --2- 172.21.15.57:0/3603684564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb43c0c62d0 0x7fb43c070d00 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:16.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.006+0000 7fb441d63640 1 --2- 172.21.15.57:0/3603684564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb43c1515c0 0x7fb43c0caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:16.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.006+0000 7fb441d63640 1 -- 172.21.15.57:0/3603684564 >> 172.21.15.57:0/3603684564 conn(0x7fb43c069e90 msgr2=0x7fb43c06a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:16.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.006+0000 7fb441d63640 1 -- 172.21.15.57:0/3603684564 shutdown_connections 2023-10-30T14:15:16.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.006+0000 7fb441d63640 1 -- 172.21.15.57:0/3603684564 wait complete. 2023-10-30T14:15:16.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.006+0000 7fb441d63640 1 Processor -- start 2023-10-30T14:15:16.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.006+0000 7fb441d63640 1 -- start start 2023-10-30T14:15:16.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb441d63640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb43c0c62d0 0x7fb43c1587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:16.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb441d63640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb43c0cb490 0x7fb43c1544f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:16.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb441d63640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb43c1515c0 0x7fb43c154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:16.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb441d63640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c152a80 con 0x7fb43c0c62d0 2023-10-30T14:15:16.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb441d63640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c1eef50 con 0x7fb43c1515c0 2023-10-30T14:15:16.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb441d63640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c153490 con 0x7fb43c0cb490 2023-10-30T14:15:16.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb440d61640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb43c0c62d0 0x7fb43c1587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:16.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb440d61640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb43c0c62d0 0x7fb43c1587a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35188/0 (socket says 172.21.15.57:35188) 2023-10-30T14:15:16.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb440d61640 1 -- 172.21.15.57:0/3825206505 learned_addr learned my addr 172.21.15.57:0/3825206505 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:15:16.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb441562640 1 --2- 172.21.15.57:0/3825206505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb43c1515c0 0x7fb43c154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:16.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb43bfff640 1 --2- 172.21.15.57:0/3825206505 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb43c0cb490 0x7fb43c1544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:16.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb440d61640 1 -- 172.21.15.57:0/3825206505 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb43c0cb490 msgr2=0x7fb43c1544f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:16.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb440d61640 1 --2- 172.21.15.57:0/3825206505 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb43c0cb490 0x7fb43c1544f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:16.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb440d61640 1 -- 172.21.15.57:0/3825206505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb43c1515c0 msgr2=0x7fb43c154a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:16.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb440d61640 1 --2- 172.21.15.57:0/3825206505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb43c1515c0 0x7fb43c154a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:16.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb440d61640 1 -- 172.21.15.57:0/3825206505 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb434009a70 con 0x7fb43c0c62d0 2023-10-30T14:15:16.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb441562640 1 --2- 172.21.15.57:0/3825206505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb43c1515c0 0x7fb43c154a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:15:16.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb440d61640 1 --2- 172.21.15.57:0/3825206505 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb43c0c62d0 0x7fb43c1587a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb43000e930 tx=0x7fb43000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:16.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb4397fa640 1 -- 172.21.15.57:0/3825206505 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb430009690 con 0x7fb43c0c62d0 2023-10-30T14:15:16.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb4397fa640 1 -- 172.21.15.57:0/3825206505 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43000ca00 con 0x7fb43c0c62d0 2023-10-30T14:15:16.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb4397fa640 1 -- 172.21.15.57:0/3825206505 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43000cd70 con 0x7fb43c0c62d0 2023-10-30T14:15:16.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb441d63640 1 -- 172.21.15.57:0/3825206505 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb43c153730 con 0x7fb43c0c62d0 2023-10-30T14:15:16.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb441d63640 1 -- 172.21.15.57:0/3825206505 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb43c153cf0 con 0x7fb43c0c62d0 2023-10-30T14:15:16.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.010+0000 7fb441d63640 1 -- 172.21.15.57:0/3825206505 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb43c067540 con 0x7fb43c0c62d0 2023-10-30T14:15:16.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.014+0000 7fb4397fa640 1 -- 172.21.15.57:0/3825206505 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb430013470 con 0x7fb43c0c62d0 2023-10-30T14:15:16.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.014+0000 7fb4397fa640 1 --2- 172.21.15.57:0/3825206505 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4200664f0 0x7fb4200689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:16.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.014+0000 7fb4397fa640 1 -- 172.21.15.57:0/3825206505 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fb430084190 con 0x7fb43c0c62d0 2023-10-30T14:15:16.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.014+0000 7fb43bfff640 1 --2- 172.21.15.57:0/3825206505 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4200664f0 0x7fb4200689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:16.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.014+0000 7fb43bfff640 1 --2- 172.21.15.57:0/3825206505 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4200664f0 0x7fb4200689b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb43c070350 tx=0x7fb428007b60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:16.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.018+0000 7fb4397fa640 1 -- 172.21.15.57:0/3825206505 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb4300515e0 con 0x7fb43c0c62d0 2023-10-30T14:15:16.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.226+0000 7fb441d63640 1 -- 172.21.15.57:0/3825206505 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb43c0c7350 con 0x7fb43c0c62d0 2023-10-30T14:15:16.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.226+0000 7fb4397fa640 1 -- 172.21.15.57:0/3825206505 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5731 (secure 0 0 0) 0x7fb430054c00 con 0x7fb43c0c62d0 2023-10-30T14:15:16.230 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:15:16.231 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:15:15.808396+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"a","rank":2,"incarnation":13,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:16.231 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 16 2023-10-30T14:15:16.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.230+0000 7fb441d63640 1 -- 172.21.15.57:0/3825206505 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4200664f0 msgr2=0x7fb4200689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:16.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.230+0000 7fb441d63640 1 --2- 172.21.15.57:0/3825206505 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4200664f0 0x7fb4200689b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb43c070350 tx=0x7fb428007b60 comp rx=0 tx=0).stop 2023-10-30T14:15:16.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.230+0000 7fb441d63640 1 -- 172.21.15.57:0/3825206505 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb43c0c62d0 msgr2=0x7fb43c1587a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:16.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.230+0000 7fb441d63640 1 --2- 172.21.15.57:0/3825206505 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb43c0c62d0 0x7fb43c1587a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb43000e930 tx=0x7fb43000edf0 comp rx=0 tx=0).stop 2023-10-30T14:15:16.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.230+0000 7fb441d63640 1 -- 172.21.15.57:0/3825206505 shutdown_connections 2023-10-30T14:15:16.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.230+0000 7fb441d63640 1 --2- 172.21.15.57:0/3825206505 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4200664f0 0x7fb4200689b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:16.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.230+0000 7fb441d63640 1 --2- 172.21.15.57:0/3825206505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb43c1515c0 0x7fb43c154a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:16.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.230+0000 7fb441d63640 1 --2- 172.21.15.57:0/3825206505 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb43c0cb490 0x7fb43c1544f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:16.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.230+0000 7fb441d63640 1 --2- 172.21.15.57:0/3825206505 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb43c0c62d0 0x7fb43c1587a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:16.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.230+0000 7fb441d63640 1 -- 172.21.15.57:0/3825206505 >> 172.21.15.57:0/3825206505 conn(0x7fb43c069e90 msgr2=0x7fb43c070c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:16.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.230+0000 7fb441d63640 1 -- 172.21.15.57:0/3825206505 shutdown_connections 2023-10-30T14:15:16.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:16.230+0000 7fb441d63640 1 -- 172.21.15.57:0/3825206505 wait complete. 2023-10-30T14:15:16.260 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:15:18.261 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:18.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.406+0000 7fde0d73e640 1 Processor -- start 2023-10-30T14:15:18.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.406+0000 7fde0d73e640 1 -- start start 2023-10-30T14:15:18.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.406+0000 7fde0d73e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde080718d0 0x7fde080c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:18.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.406+0000 7fde0d73e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0815d0a0 con 0x7fde08153300 2023-10-30T14:15:18.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.406+0000 7fde0d73e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0815d380 con 0x7fde08071cb0 2023-10-30T14:15:18.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.406+0000 7fde0d73e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde080cca10 con 0x7fde080718d0 2023-10-30T14:15:18.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.406+0000 7fde06ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde080718d0 0x7fde080c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:18.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.406+0000 7fde077fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fde08153300 0x7fde080cade0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:42408/0 (socket says 172.21.15.57:42408) 2023-10-30T14:15:18.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.406+0000 7fde067fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fde08071cb0 0x7fde0815a230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:44374/0 (socket says 172.21.15.57:44374) 2023-10-30T14:15:18.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.406+0000 7fde077fe640 1 -- 172.21.15.57:0/3036246541 learned_addr learned my addr 172.21.15.57:0/3036246541 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:15:18.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.406+0000 7fde05ffb640 1 -- 172.21.15.57:0/3036246541 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197279301 0 0) 0x7fde0815d0a0 con 0x7fde08153300 2023-10-30T14:15:18.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.406+0000 7fde05ffb640 1 -- 172.21.15.57:0/3036246541 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddf0003530 con 0x7fde08153300 2023-10-30T14:15:18.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.406+0000 7fde05ffb640 1 -- 172.21.15.57:0/3036246541 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498915233 0 0) 0x7fde0815d380 con 0x7fde08071cb0 2023-10-30T14:15:18.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.406+0000 7fde06ffd640 1 -- 172.21.15.57:0/3036246541 >> v1:172.21.15.88:6789/0 conn(0x7fde08071cb0 legacy=0x7fde0815a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:18.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.406+0000 7fde06ffd640 1 -- 172.21.15.57:0/3036246541 >> v1:172.21.15.57:6789/0 conn(0x7fde08153300 legacy=0x7fde080cade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:18.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.406+0000 7fde06ffd640 1 -- 172.21.15.57:0/3036246541 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0815d660 con 0x7fde080718d0 2023-10-30T14:15:18.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.414+0000 7fde06ffd640 1 --2- 172.21.15.57:0/3036246541 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde080718d0 0x7fde080c9a10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fddf80097f0 tx=0x7fddf801ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1371df68e43da4f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:18.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.414+0000 7fde05ffb640 1 -- 172.21.15.57:0/3036246541 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf80a89b0 con 0x7fde080718d0 2023-10-30T14:15:18.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.414+0000 7fde05ffb640 1 -- 172.21.15.57:0/3036246541 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf80a8b10 con 0x7fde080718d0 2023-10-30T14:15:18.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.414+0000 7fde05ffb640 1 -- 172.21.15.57:0/3036246541 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf80a8ea0 con 0x7fde080718d0 2023-10-30T14:15:18.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.414+0000 7fde0d73e640 1 -- 172.21.15.57:0/3036246541 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde080718d0 msgr2=0x7fde080c9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:18.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.414+0000 7fde0d73e640 1 --2- 172.21.15.57:0/3036246541 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde080718d0 0x7fde080c9a10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fddf80097f0 tx=0x7fddf801ecd0 comp rx=0 tx=0).stop 2023-10-30T14:15:18.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.414+0000 7fde0d73e640 1 -- 172.21.15.57:0/3036246541 shutdown_connections 2023-10-30T14:15:18.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.414+0000 7fde0d73e640 1 --2- 172.21.15.57:0/3036246541 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde080718d0 0x7fde080c9a10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:18.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.414+0000 7fde0d73e640 1 -- 172.21.15.57:0/3036246541 >> 172.21.15.57:0/3036246541 conn(0x7fde0806ab00 msgr2=0x7fde08065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:18.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.414+0000 7fde0d73e640 1 -- 172.21.15.57:0/3036246541 shutdown_connections 2023-10-30T14:15:18.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.414+0000 7fde0d73e640 1 -- 172.21.15.57:0/3036246541 wait complete. 2023-10-30T14:15:18.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.418+0000 7fde0d73e640 1 Processor -- start 2023-10-30T14:15:18.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.418+0000 7fde0d73e640 1 -- start start 2023-10-30T14:15:18.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.418+0000 7fde0d73e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde080718d0 0x7fde080b3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:18.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.418+0000 7fde0d73e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde08071cb0 0x7fde0815bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:18.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.418+0000 7fde0d73e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde08153300 0x7fde0815c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:18.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.418+0000 7fde0d73e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde081f2e30 con 0x7fde080718d0 2023-10-30T14:15:18.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.418+0000 7fde0d73e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde081f30f0 con 0x7fde08071cb0 2023-10-30T14:15:18.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.418+0000 7fde0d73e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde08150230 con 0x7fde08153300 2023-10-30T14:15:18.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.418+0000 7fde06ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde080718d0 0x7fde080b3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:18.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.418+0000 7fde06ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde080718d0 0x7fde080b3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35200/0 (socket says 172.21.15.57:35200) 2023-10-30T14:15:18.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.418+0000 7fde06ffd640 1 -- 172.21.15.57:0/327540381 learned_addr learned my addr 172.21.15.57:0/327540381 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:15:18.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.418+0000 7fde067fc640 1 --2- 172.21.15.57:0/327540381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde08071cb0 0x7fde0815bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:18.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.418+0000 7fde077fe640 1 --2- 172.21.15.57:0/327540381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde08153300 0x7fde0815c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:18.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.418+0000 7fde06ffd640 1 -- 172.21.15.57:0/327540381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde08153300 msgr2=0x7fde0815c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:18.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.418+0000 7fde06ffd640 1 --2- 172.21.15.57:0/327540381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde08153300 0x7fde0815c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:18.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.418+0000 7fde06ffd640 1 -- 172.21.15.57:0/327540381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde08071cb0 msgr2=0x7fde0815bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:18.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.418+0000 7fde06ffd640 1 --2- 172.21.15.57:0/327540381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde08071cb0 0x7fde0815bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:18.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.418+0000 7fde06ffd640 1 -- 172.21.15.57:0/327540381 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf8009420 con 0x7fde080718d0 2023-10-30T14:15:18.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.418+0000 7fde06ffd640 1 --2- 172.21.15.57:0/327540381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde080718d0 0x7fde080b3e40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fddf80097f0 tx=0x7fddf801ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:18.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.422+0000 7fddef7fe640 1 -- 172.21.15.57:0/327540381 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf80bf070 con 0x7fde080718d0 2023-10-30T14:15:18.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.422+0000 7fddef7fe640 1 -- 172.21.15.57:0/327540381 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf80b0bf0 con 0x7fde080718d0 2023-10-30T14:15:18.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.422+0000 7fddef7fe640 1 -- 172.21.15.57:0/327540381 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf80b9410 con 0x7fde080718d0 2023-10-30T14:15:18.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.422+0000 7fde0d73e640 1 -- 172.21.15.57:0/327540381 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde08150510 con 0x7fde080718d0 2023-10-30T14:15:18.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.422+0000 7fde0d73e640 1 -- 172.21.15.57:0/327540381 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde081472b0 con 0x7fde080718d0 2023-10-30T14:15:18.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.422+0000 7fde0d73e640 1 -- 172.21.15.57:0/327540381 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0804e170 con 0x7fde080718d0 2023-10-30T14:15:18.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.422+0000 7fddef7fe640 1 -- 172.21.15.57:0/327540381 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fddf80b7040 con 0x7fde080718d0 2023-10-30T14:15:18.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.426+0000 7fddef7fe640 1 --2- 172.21.15.57:0/327540381 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fddf40664a0 0x7fddf4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:18.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.426+0000 7fddef7fe640 1 -- 172.21.15.57:0/327540381 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fddf8121260 con 0x7fde080718d0 2023-10-30T14:15:18.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.426+0000 7fde067fc640 1 --2- 172.21.15.57:0/327540381 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fddf40664a0 0x7fddf4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:18.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.426+0000 7fde067fc640 1 --2- 172.21.15.57:0/327540381 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fddf40664a0 0x7fddf4068960 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fde08066010 tx=0x7fddfc007d60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:18.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.426+0000 7fddef7fe640 1 -- 172.21.15.57:0/327540381 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fddf80ee590 con 0x7fde080718d0 2023-10-30T14:15:18.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.686+0000 7fde0d73e640 1 -- 172.21.15.57:0/327540381 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde08065a80 con 0x7fde080718d0 2023-10-30T14:15:18.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.686+0000 7fddef7fe640 1 -- 172.21.15.57:0/327540381 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5731 (secure 0 0 0) 0x7fddf80b5090 con 0x7fde080718d0 2023-10-30T14:15:18.690 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:15:18.690 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:15:15.808396+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"a","rank":2,"incarnation":13,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:18.690 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 16 2023-10-30T14:15:18.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.690+0000 7fde0d73e640 1 -- 172.21.15.57:0/327540381 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fddf40664a0 msgr2=0x7fddf4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:18.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.690+0000 7fde0d73e640 1 --2- 172.21.15.57:0/327540381 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fddf40664a0 0x7fddf4068960 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fde08066010 tx=0x7fddfc007d60 comp rx=0 tx=0).stop 2023-10-30T14:15:18.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.690+0000 7fde0d73e640 1 -- 172.21.15.57:0/327540381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde080718d0 msgr2=0x7fde080b3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:18.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.690+0000 7fde0d73e640 1 --2- 172.21.15.57:0/327540381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde080718d0 0x7fde080b3e40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fddf80097f0 tx=0x7fddf801ba70 comp rx=0 tx=0).stop 2023-10-30T14:15:18.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.690+0000 7fde0d73e640 1 -- 172.21.15.57:0/327540381 shutdown_connections 2023-10-30T14:15:18.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.690+0000 7fde0d73e640 1 --2- 172.21.15.57:0/327540381 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fddf40664a0 0x7fddf4068960 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:18.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.690+0000 7fde0d73e640 1 --2- 172.21.15.57:0/327540381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde08153300 0x7fde0815c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:18.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.690+0000 7fde0d73e640 1 --2- 172.21.15.57:0/327540381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde08071cb0 0x7fde0815bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:18.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.690+0000 7fde0d73e640 1 --2- 172.21.15.57:0/327540381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde080718d0 0x7fde080b3e40 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:18.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.690+0000 7fde0d73e640 1 -- 172.21.15.57:0/327540381 >> 172.21.15.57:0/327540381 conn(0x7fde0806ab00 msgr2=0x7fde080cade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:18.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.690+0000 7fde0d73e640 1 -- 172.21.15.57:0/327540381 shutdown_connections 2023-10-30T14:15:18.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:18.690+0000 7fde0d73e640 1 -- 172.21.15.57:0/327540381 wait complete. 2023-10-30T14:15:18.712 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:15:20.713 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:20.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.874+0000 7f1b32c09640 1 Processor -- start 2023-10-30T14:15:20.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.878+0000 7f1b32c09640 1 -- start start 2023-10-30T14:15:20.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.878+0000 7f1b32c09640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1b2c153b50 0x7f1b2c1597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:20.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.878+0000 7f1b32c09640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b2c0c62b0 0x7f1b2c0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:20.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.878+0000 7f1b32c09640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b2c15a770 con 0x7f1b2c0cb180 2023-10-30T14:15:20.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.878+0000 7f1b32c09640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b2c159cf0 con 0x7f1b2c153b50 2023-10-30T14:15:20.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.878+0000 7f1b32c09640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c06aa30 con 0x7f1b2c0c62b0 2023-10-30T14:15:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.878+0000 7f1b32408640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1b2c0cb180 0x7f1b2c06ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:42420/0 (socket says 172.21.15.57:42420) 2023-10-30T14:15:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.878+0000 7f1b32408640 1 -- 172.21.15.57:0/3040791675 learned_addr learned my addr 172.21.15.57:0/3040791675 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:15:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.878+0000 7f1b31c07640 1 --2- 172.21.15.57:0/3040791675 >> v2:172.21.15.57:3300/0 conn(0x7f1b2c153b50 0x7f1b2c1597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.878+0000 7f1b31406640 1 --2- 172.21.15.57:0/3040791675 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b2c0c62b0 0x7f1b2c0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.878+0000 7f1b30c05640 1 -- 172.21.15.57:0/3040791675 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894823757 0 0) 0x7f1b240014a0 con 0x7f1b2c0cb180 2023-10-30T14:15:20.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.878+0000 7f1b31c07640 1 -- 172.21.15.57:0/3040791675 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b2c0c62b0 msgr2=0x7f1b2c0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:20.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.878+0000 7f1b31c07640 1 --2- 172.21.15.57:0/3040791675 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b2c0c62b0 0x7f1b2c0ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:20.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.878+0000 7f1b31c07640 1 -- 172.21.15.57:0/3040791675 >> v1:172.21.15.57:6789/0 conn(0x7f1b2c0cb180 legacy=0x7f1b2c06ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:20.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.878+0000 7f1b31c07640 1 -- 172.21.15.57:0/3040791675 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2c0c9d40 con 0x7f1b2c153b50 2023-10-30T14:15:20.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.882+0000 7f1b31c07640 1 --2- 172.21.15.57:0/3040791675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b2c153b50 0x7f1b2c1597b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1b280097c0 tx=0x7f1b2801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f292048ed8d6b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:20.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.882+0000 7f1b30c05640 1 -- 172.21.15.57:0/3040791675 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b280ab600 con 0x7f1b2c153b50 2023-10-30T14:15:20.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.882+0000 7f1b30c05640 1 -- 172.21.15.57:0/3040791675 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b280a7050 con 0x7f1b2c153b50 2023-10-30T14:15:20.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.882+0000 7f1b30c05640 1 -- 172.21.15.57:0/3040791675 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b280ab980 con 0x7f1b2c153b50 2023-10-30T14:15:20.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.882+0000 7f1b32c09640 1 -- 172.21.15.57:0/3040791675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b2c153b50 msgr2=0x7f1b2c1597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:20.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.882+0000 7f1b32c09640 1 --2- 172.21.15.57:0/3040791675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b2c153b50 0x7f1b2c1597b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1b280097c0 tx=0x7f1b2801c480 comp rx=0 tx=0).stop 2023-10-30T14:15:20.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.882+0000 7f1b32c09640 1 -- 172.21.15.57:0/3040791675 shutdown_connections 2023-10-30T14:15:20.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.882+0000 7f1b32c09640 1 --2- 172.21.15.57:0/3040791675 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b2c0c62b0 0x7f1b2c0ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:20.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.882+0000 7f1b32c09640 1 --2- 172.21.15.57:0/3040791675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b2c153b50 0x7f1b2c1597b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:20.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.882+0000 7f1b32c09640 1 -- 172.21.15.57:0/3040791675 >> 172.21.15.57:0/3040791675 conn(0x7f1b2c065a80 msgr2=0x7f1b2c065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:20.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.886+0000 7f1b32c09640 1 -- 172.21.15.57:0/3040791675 shutdown_connections 2023-10-30T14:15:20.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.886+0000 7f1b32c09640 1 -- 172.21.15.57:0/3040791675 wait complete. 2023-10-30T14:15:20.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.886+0000 7f1b32c09640 1 Processor -- start 2023-10-30T14:15:20.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.886+0000 7f1b32c09640 1 -- start start 2023-10-30T14:15:20.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b32c09640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b2c0c62b0 0x7f1b2c159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:20.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b32c09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b2c0cb180 0x7f1b2c1f5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:20.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b32c09640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b2c153b50 0x7f1b2c157620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:20.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b32c09640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c1f5770 con 0x7f1b2c153b50 2023-10-30T14:15:20.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b32c09640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c158730 con 0x7f1b2c0cb180 2023-10-30T14:15:20.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b32c09640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c1f53d0 con 0x7f1b2c0c62b0 2023-10-30T14:15:20.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b32408640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b2c153b50 0x7f1b2c157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:20.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b32408640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b2c153b50 0x7f1b2c157620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35216/0 (socket says 172.21.15.57:35216) 2023-10-30T14:15:20.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b32408640 1 -- 172.21.15.57:0/2682391549 learned_addr learned my addr 172.21.15.57:0/2682391549 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:15:20.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b31406640 1 --2- 172.21.15.57:0/2682391549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b2c0cb180 0x7f1b2c1f5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:20.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b31c07640 1 --2- 172.21.15.57:0/2682391549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b2c0c62b0 0x7f1b2c159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:20.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b32408640 1 -- 172.21.15.57:0/2682391549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b2c0c62b0 msgr2=0x7f1b2c159430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:20.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b32408640 1 --2- 172.21.15.57:0/2682391549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b2c0c62b0 0x7f1b2c159430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:20.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b32408640 1 -- 172.21.15.57:0/2682391549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b2c0cb180 msgr2=0x7f1b2c1f5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:20.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b32408640 1 --2- 172.21.15.57:0/2682391549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b2c0cb180 0x7f1b2c1f5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:20.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b32408640 1 -- 172.21.15.57:0/2682391549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b28009420 con 0x7f1b2c153b50 2023-10-30T14:15:20.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b32408640 1 --2- 172.21.15.57:0/2682391549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b2c153b50 0x7f1b2c157620 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1b2400c7e0 tx=0x7f1b2400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:20.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b227fc640 1 -- 172.21.15.57:0/2682391549 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2400a640 con 0x7f1b2c153b50 2023-10-30T14:15:20.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b227fc640 1 -- 172.21.15.57:0/2682391549 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2400a7c0 con 0x7f1b2c153b50 2023-10-30T14:15:20.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b227fc640 1 -- 172.21.15.57:0/2682391549 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2400abb0 con 0x7f1b2c153b50 2023-10-30T14:15:20.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b32c09640 1 -- 172.21.15.57:0/2682391549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b2c0c2c40 con 0x7f1b2c153b50 2023-10-30T14:15:20.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.890+0000 7f1b32c09640 1 -- 172.21.15.57:0/2682391549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b2c0c2f00 con 0x7f1b2c153b50 2023-10-30T14:15:20.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.894+0000 7f1b32c09640 1 -- 172.21.15.57:0/2682391549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b14005260 con 0x7f1b2c153b50 2023-10-30T14:15:20.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.894+0000 7f1b227fc640 1 -- 172.21.15.57:0/2682391549 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1b2400ad10 con 0x7f1b2c153b50 2023-10-30T14:15:20.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.894+0000 7f1b227fc640 1 --2- 172.21.15.57:0/2682391549 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1b100664f0 0x7f1b100689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:20.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.894+0000 7f1b227fc640 1 -- 172.21.15.57:0/2682391549 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f1b24083740 con 0x7f1b2c153b50 2023-10-30T14:15:20.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.894+0000 7f1b31c07640 1 --2- 172.21.15.57:0/2682391549 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1b100664f0 0x7f1b100689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:20.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.894+0000 7f1b31c07640 1 --2- 172.21.15.57:0/2682391549 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1b100664f0 0x7f1b100689b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1b280ad000 tx=0x7f1b28008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:20.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:20.902+0000 7f1b227fc640 1 -- 172.21.15.57:0/2682391549 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1b24050af0 con 0x7f1b2c153b50 2023-10-30T14:15:21.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:21.230+0000 7f1b32c09640 1 -- 172.21.15.57:0/2682391549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b14005a90 con 0x7f1b2c153b50 2023-10-30T14:15:21.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:21.230+0000 7f1b227fc640 1 -- 172.21.15.57:0/2682391549 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5731 (secure 0 0 0) 0x7f1b24018090 con 0x7f1b2c153b50 2023-10-30T14:15:21.235 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:15:21.235 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:15:20.844622+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:21.236 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 17 2023-10-30T14:15:21.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:21.234+0000 7f1b32c09640 1 -- 172.21.15.57:0/2682391549 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1b100664f0 msgr2=0x7f1b100689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:21.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:21.234+0000 7f1b32c09640 1 --2- 172.21.15.57:0/2682391549 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1b100664f0 0x7f1b100689b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1b280ad000 tx=0x7f1b28008040 comp rx=0 tx=0).stop 2023-10-30T14:15:21.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:21.234+0000 7f1b32c09640 1 -- 172.21.15.57:0/2682391549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b2c153b50 msgr2=0x7f1b2c157620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:21.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:21.234+0000 7f1b32c09640 1 --2- 172.21.15.57:0/2682391549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b2c153b50 0x7f1b2c157620 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1b2400c7e0 tx=0x7f1b2400cca0 comp rx=0 tx=0).stop 2023-10-30T14:15:21.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:21.234+0000 7f1b32c09640 1 -- 172.21.15.57:0/2682391549 shutdown_connections 2023-10-30T14:15:21.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:21.234+0000 7f1b32c09640 1 --2- 172.21.15.57:0/2682391549 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1b100664f0 0x7f1b100689b0 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1b280ad000 tx=0x7f1b28008040 comp rx=0 tx=0).stop 2023-10-30T14:15:21.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:21.234+0000 7f1b32c09640 1 --2- 172.21.15.57:0/2682391549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b2c153b50 0x7f1b2c157620 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:21.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:21.234+0000 7f1b32c09640 1 --2- 172.21.15.57:0/2682391549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b2c0cb180 0x7f1b2c1f5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:21.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:21.234+0000 7f1b32c09640 1 --2- 172.21.15.57:0/2682391549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b2c0c62b0 0x7f1b2c159430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:21.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:21.234+0000 7f1b32c09640 1 -- 172.21.15.57:0/2682391549 >> 172.21.15.57:0/2682391549 conn(0x7f1b2c065a80 msgr2=0x7f1b2c1520c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:21.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:21.234+0000 7f1b32c09640 1 -- 172.21.15.57:0/2682391549 shutdown_connections 2023-10-30T14:15:21.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:21.234+0000 7f1b32c09640 1 -- 172.21.15.57:0/2682391549 wait complete. 2023-10-30T14:15:21.259 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=2, replacing gid=4265 2023-10-30T14:15:21.260 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.e 2023-10-30T14:15:22.870 INFO:tasks.ceph.mds.f.smithi088.stderr:2023-10-30T14:15:22.864+0000 7f1418be9640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T14:15:28.260 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T14:15:28.260 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T14:15:28.261 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T14:15:28.266 INFO:tasks.ceph.mds.e:Started 2023-10-30T14:15:28.267 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:28.413 INFO:tasks.ceph.mds.e.smithi057.stdout:starting mds.e at 2023-10-30T14:15:28.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.502+0000 7f0e639d3640 1 Processor -- start 2023-10-30T14:15:28.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.502+0000 7f0e639d3640 1 -- start start 2023-10-30T14:15:28.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.502+0000 7f0e639d3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0e5c0c4a10 0x7f0e5c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:28.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.502+0000 7f0e639d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e5c0c42a0 0x7f0e5c0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:28.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.502+0000 7f0e639d3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5c159f20 con 0x7f0e5c153c80 2023-10-30T14:15:28.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.502+0000 7f0e639d3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e5c15dec0 con 0x7f0e5c0c4a10 2023-10-30T14:15:28.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.502+0000 7f0e639d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c1599a0 con 0x7f0e5c0c42a0 2023-10-30T14:15:28.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.502+0000 7f0e60f6e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0e5c0c4a10 0x7f0e5c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:28.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.502+0000 7f0e60f6e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0e5c0c4a10 0x7f0e5c15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:53396/0 (socket says 172.21.15.57:53396) 2023-10-30T14:15:28.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.502+0000 7f0e60f6e640 1 -- 172.21.15.57:0/2400519810 learned_addr learned my addr 172.21.15.57:0/2400519810 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:15:28.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.506+0000 7f0e61f70640 1 --2- 172.21.15.57:0/2400519810 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e5c0c42a0 0x7f0e5c0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:28.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.506+0000 7f0e60f6e640 1 -- 172.21.15.57:0/2400519810 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e5c0c42a0 msgr2=0x7f0e5c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:28.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.506+0000 7f0e60f6e640 1 --2- 172.21.15.57:0/2400519810 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e5c0c42a0 0x7f0e5c0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:28.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.506+0000 7f0e60f6e640 1 -- 172.21.15.57:0/2400519810 >> v1:172.21.15.88:6789/0 conn(0x7f0e5c153c80 legacy=0x7f0e5c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:28.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.506+0000 7f0e60f6e640 1 -- 172.21.15.57:0/2400519810 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c15d620 con 0x7f0e5c0c4a10 2023-10-30T14:15:28.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.510+0000 7f0e60f6e640 1 --2- 172.21.15.57:0/2400519810 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e5c0c4a10 0x7f0e5c15cf90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c009870 tx=0x7f0e4c01ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45c3d936e2c4fb69 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:28.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.510+0000 7f0e53fff640 1 -- 172.21.15.57:0/2400519810 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0a8a30 con 0x7f0e5c0c4a10 2023-10-30T14:15:28.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.510+0000 7f0e53fff640 1 -- 172.21.15.57:0/2400519810 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4c0a8b90 con 0x7f0e5c0c4a10 2023-10-30T14:15:28.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.510+0000 7f0e53fff640 1 -- 172.21.15.57:0/2400519810 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0b0440 con 0x7f0e5c0c4a10 2023-10-30T14:15:28.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.510+0000 7f0e639d3640 1 -- 172.21.15.57:0/2400519810 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e5c0c4a10 msgr2=0x7f0e5c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:28.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.510+0000 7f0e639d3640 1 --2- 172.21.15.57:0/2400519810 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e5c0c4a10 0x7f0e5c15cf90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c009870 tx=0x7f0e4c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:15:28.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.510+0000 7f0e639d3640 1 -- 172.21.15.57:0/2400519810 shutdown_connections 2023-10-30T14:15:28.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.510+0000 7f0e639d3640 1 --2- 172.21.15.57:0/2400519810 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e5c0c42a0 0x7f0e5c0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:28.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.510+0000 7f0e639d3640 1 --2- 172.21.15.57:0/2400519810 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e5c0c4a10 0x7f0e5c15cf90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:28.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.510+0000 7f0e639d3640 1 -- 172.21.15.57:0/2400519810 >> 172.21.15.57:0/2400519810 conn(0x7f0e5c0715c0 msgr2=0x7f0e5c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:28.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.510+0000 7f0e639d3640 1 -- 172.21.15.57:0/2400519810 shutdown_connections 2023-10-30T14:15:28.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.510+0000 7f0e639d3640 1 -- 172.21.15.57:0/2400519810 wait complete. 2023-10-30T14:15:28.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.514+0000 7f0e639d3640 1 Processor -- start 2023-10-30T14:15:28.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.514+0000 7f0e639d3640 1 -- start start 2023-10-30T14:15:28.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.514+0000 7f0e639d3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e5c0c42a0 0x7f0e5c0c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:28.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.514+0000 7f0e639d3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e5c0c4a10 0x7f0e5c149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:28.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.514+0000 7f0e639d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e5c153c80 0x7f0e5c14a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:28.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.514+0000 7f0e639d3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c1f1e30 con 0x7f0e5c0c4a10 2023-10-30T14:15:28.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.514+0000 7f0e639d3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c1f2110 con 0x7f0e5c0c42a0 2023-10-30T14:15:28.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.514+0000 7f0e639d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c1f23f0 con 0x7f0e5c153c80 2023-10-30T14:15:28.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.518+0000 7f0e60f6e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e5c0c4a10 0x7f0e5c149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:28.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.518+0000 7f0e60f6e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e5c0c4a10 0x7f0e5c149b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49226/0 (socket says 172.21.15.57:49226) 2023-10-30T14:15:28.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.518+0000 7f0e60f6e640 1 -- 172.21.15.57:0/1963995619 learned_addr learned my addr 172.21.15.57:0/1963995619 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:15:28.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.518+0000 7f0e61f70640 1 --2- 172.21.15.57:0/1963995619 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e5c153c80 0x7f0e5c14a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:28.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.518+0000 7f0e6176f640 1 --2- 172.21.15.57:0/1963995619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e5c0c42a0 0x7f0e5c0c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:28.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.518+0000 7f0e60f6e640 1 -- 172.21.15.57:0/1963995619 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e5c153c80 msgr2=0x7f0e5c14a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:28.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.518+0000 7f0e60f6e640 1 --2- 172.21.15.57:0/1963995619 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e5c153c80 0x7f0e5c14a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:28.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.518+0000 7f0e60f6e640 1 -- 172.21.15.57:0/1963995619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e5c0c42a0 msgr2=0x7f0e5c0c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:28.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.518+0000 7f0e60f6e640 1 --2- 172.21.15.57:0/1963995619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e5c0c42a0 0x7f0e5c0c89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:28.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.518+0000 7f0e60f6e640 1 -- 172.21.15.57:0/1963995619 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4c0094d0 con 0x7f0e5c0c4a10 2023-10-30T14:15:28.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.518+0000 7f0e6176f640 1 --2- 172.21.15.57:0/1963995619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e5c0c42a0 0x7f0e5c0c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:15:28.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.518+0000 7f0e60f6e640 1 --2- 172.21.15.57:0/1963995619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e5c0c4a10 0x7f0e5c149b30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c0b3040 tx=0x7f0e4c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:28.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.518+0000 7f0e51ffb640 1 -- 172.21.15.57:0/1963995619 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0be070 con 0x7f0e5c0c4a10 2023-10-30T14:15:28.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.518+0000 7f0e51ffb640 1 -- 172.21.15.57:0/1963995619 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4c0ba400 con 0x7f0e5c0c4a10 2023-10-30T14:15:28.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.518+0000 7f0e639d3640 1 -- 172.21.15.57:0/1963995619 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e5c1f26d0 con 0x7f0e5c0c4a10 2023-10-30T14:15:28.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.518+0000 7f0e51ffb640 1 -- 172.21.15.57:0/1963995619 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0ba7d0 con 0x7f0e5c0c4a10 2023-10-30T14:15:28.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.518+0000 7f0e639d3640 1 -- 172.21.15.57:0/1963995619 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5c1592b0 con 0x7f0e5c0c4a10 2023-10-30T14:15:28.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.518+0000 7f0e639d3640 1 -- 172.21.15.57:0/1963995619 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e5c04e170 con 0x7f0e5c0c4a10 2023-10-30T14:15:28.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.522+0000 7f0e51ffb640 1 -- 172.21.15.57:0/1963995619 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0e4c0c3410 con 0x7f0e5c0c4a10 2023-10-30T14:15:28.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.526+0000 7f0e51ffb640 1 --2- 172.21.15.57:0/1963995619 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0e400664f0 0x7f0e400689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:28.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.526+0000 7f0e51ffb640 1 -- 172.21.15.57:0/1963995619 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f0e4c122350 con 0x7f0e5c0c4a10 2023-10-30T14:15:28.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.526+0000 7f0e6176f640 1 --2- 172.21.15.57:0/1963995619 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0e400664f0 0x7f0e400689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:28.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.526+0000 7f0e6176f640 1 --2- 172.21.15.57:0/1963995619 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0e400664f0 0x7f0e400689b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0e58000f70 tx=0x7f0e58009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:28.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.530+0000 7f0e51ffb640 1 -- 172.21.15.57:0/1963995619 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0e4c0ef890 con 0x7f0e5c0c4a10 2023-10-30T14:15:28.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.918+0000 7f0e639d3640 1 -- 172.21.15.57:0/1963995619 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e5c14dad0 con 0x7f0e5c0c4a10 2023-10-30T14:15:28.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.918+0000 7f0e51ffb640 1 -- 172.21.15.57:0/1963995619 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+6522 (secure 0 0 0) 0x7f0e4c0f2eb0 con 0x7f0e5c0c4a10 2023-10-30T14:15:28.921 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:15:28.921 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4936,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:15:20.844622+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:28.922 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 18 2023-10-30T14:15:28.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.922+0000 7f0e639d3640 1 -- 172.21.15.57:0/1963995619 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0e400664f0 msgr2=0x7f0e400689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:28.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.922+0000 7f0e639d3640 1 --2- 172.21.15.57:0/1963995619 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0e400664f0 0x7f0e400689b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0e58000f70 tx=0x7f0e58009040 comp rx=0 tx=0).stop 2023-10-30T14:15:28.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.922+0000 7f0e639d3640 1 -- 172.21.15.57:0/1963995619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e5c0c4a10 msgr2=0x7f0e5c149b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:28.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.922+0000 7f0e639d3640 1 --2- 172.21.15.57:0/1963995619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e5c0c4a10 0x7f0e5c149b30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c0b3040 tx=0x7f0e4c01bad0 comp rx=0 tx=0).stop 2023-10-30T14:15:28.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.922+0000 7f0e639d3640 1 -- 172.21.15.57:0/1963995619 shutdown_connections 2023-10-30T14:15:28.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.922+0000 7f0e639d3640 1 --2- 172.21.15.57:0/1963995619 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0e400664f0 0x7f0e400689b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:28.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.922+0000 7f0e639d3640 1 --2- 172.21.15.57:0/1963995619 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e5c153c80 0x7f0e5c14a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:28.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.922+0000 7f0e639d3640 1 --2- 172.21.15.57:0/1963995619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e5c0c4a10 0x7f0e5c149b30 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:28.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.922+0000 7f0e639d3640 1 --2- 172.21.15.57:0/1963995619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e5c0c42a0 0x7f0e5c0c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:28.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.922+0000 7f0e639d3640 1 -- 172.21.15.57:0/1963995619 >> 172.21.15.57:0/1963995619 conn(0x7f0e5c0715c0 msgr2=0x7f0e5c1582a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:28.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.922+0000 7f0e639d3640 1 -- 172.21.15.57:0/1963995619 shutdown_connections 2023-10-30T14:15:28.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:15:28.922+0000 7f0e639d3640 1 -- 172.21.15.57:0/1963995619 wait complete. 2023-10-30T14:15:28.955 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T14:15:28.955 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 67 secs before thrashing 2023-10-30T14:15:29.566 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:15:29.572 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:15:44.715 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 65% (7885/12071) Updating files: 66% (7967/12071) Updating files: 67% (8088/12071) Updating files: 68% (8209/12071) Updating files: 69% (8329/12071) Updating files: 70% (8450/12071) Updating files: 71% (8571/12071) Updating files: 72% (8692/12071) Updating files: 73% (8812/12071) Updating files: 74% (8933/12071) Updating files: 75% (9054/12071) Updating files: 76% (9174/12071) Updating files: 77% (9295/12071) Updating files: 78% (9416/12071) Updating files: 79% (9537/12071) Updating files: 80% (9657/12071) Updating files: 81% (9778/12071) Updating files: 82% (9899/12071) Updating files: 83% (10019/12071) Updating files: 84% (10140/12071) Updating files: 85% (10261/12071) Updating files: 86% (10382/12071) Updating files: 87% (10502/12071) Updating files: 88% (10623/12071) Updating files: 89% (10744/12071) Updating files: 90% (10864/12071) Updating files: 91% (10985/12071) Updating files: 92% (11106/12071) Updating files: 93% (11227/12071) Updating files: 94% (11347/12071) Updating files: 95% (11468/12071) Updating files: 96% (11589/12071) Updating files: 97% (11709/12071) Updating files: 98% (11830/12071) Updating files: 99% (11951/12071) Updating files: 100% (12071/12071) Updating files: 100% (12071/12071), done. 2023-10-30T14:15:45.269 INFO:tasks.workunit.client.0.smithi057.stderr:Note: switching to '00c1c13cea21a686b046584596fbeac5315a7d71'. 2023-10-30T14:15:45.269 INFO:tasks.workunit.client.0.smithi057.stderr: 2023-10-30T14:15:45.269 INFO:tasks.workunit.client.0.smithi057.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-30T14:15:45.269 INFO:tasks.workunit.client.0.smithi057.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-30T14:15:45.269 INFO:tasks.workunit.client.0.smithi057.stderr:state without impacting any branches by switching back to a branch. 2023-10-30T14:15:45.270 INFO:tasks.workunit.client.0.smithi057.stderr: 2023-10-30T14:15:45.270 INFO:tasks.workunit.client.0.smithi057.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-30T14:15:45.270 INFO:tasks.workunit.client.0.smithi057.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-30T14:15:45.270 INFO:tasks.workunit.client.0.smithi057.stderr: 2023-10-30T14:15:45.270 INFO:tasks.workunit.client.0.smithi057.stderr: git switch -c 2023-10-30T14:15:45.271 INFO:tasks.workunit.client.0.smithi057.stderr: 2023-10-30T14:15:45.271 INFO:tasks.workunit.client.0.smithi057.stderr:Or undo this operation with: 2023-10-30T14:15:45.271 INFO:tasks.workunit.client.0.smithi057.stderr: 2023-10-30T14:15:45.271 INFO:tasks.workunit.client.0.smithi057.stderr: git switch - 2023-10-30T14:15:45.271 INFO:tasks.workunit.client.0.smithi057.stderr: 2023-10-30T14:15:45.271 INFO:tasks.workunit.client.0.smithi057.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-30T14:15:45.272 INFO:tasks.workunit.client.0.smithi057.stderr: 2023-10-30T14:15:45.272 INFO:tasks.workunit.client.0.smithi057.stderr:HEAD is now at 00c1c13cea2 mds/MDBalancer: ignore queued callbacks if MDS is not active 2023-10-30T14:15:45.274 DEBUG:teuthology.orchestra.run.smithi057:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-30T14:15:45.284 INFO:tasks.workunit.client.0.smithi057.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-30T14:15:45.287 INFO:tasks.workunit.client.0.smithi057.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T14:15:45.287 INFO:tasks.workunit.client.0.smithi057.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-30T14:15:46.052 INFO:tasks.workunit.client.0.smithi057.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-30T14:15:46.148 INFO:tasks.workunit.client.0.smithi057.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-30T14:15:46.206 INFO:tasks.workunit.client.0.smithi057.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T14:15:46.209 INFO:tasks.workunit.client.0.smithi057.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T14:15:46.209 INFO:tasks.workunit.client.0.smithi057.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-30T14:15:46.264 INFO:tasks.workunit.client.0.smithi057.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T14:15:46.270 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:15:46.270 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-30T14:15:46.320 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2023-10-30T14:15:46.321 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2023-10-30T14:15:46.322 DEBUG:teuthology.orchestra.run.smithi057:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2023-10-30T14:15:46.384 INFO:tasks.workunit.client.0.smithi057.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2023-10-30T14:15:48.935 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir foo 2023-10-30T14:15:48.951 INFO:tasks.workunit.client.0.smithi057.stderr:+ cp linux-2.6.33.tar.bz2 foo 2023-10-30T14:15:49.057 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir foo/.snap/barsnap 2023-10-30T14:15:49.072 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm foo/linux-2.6.33.tar.bz2 2023-10-30T14:15:49.086 INFO:tasks.workunit.client.0.smithi057.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2023-10-30T14:15:49.578 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo passed: files are identical 2023-10-30T14:15:49.578 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir foo/.snap/barsnap 2023-10-30T14:15:49.578 INFO:tasks.workunit.client.0.smithi057.stdout:passed: files are identical 2023-10-30T14:15:49.615 INFO:tasks.workunit.client.0.smithi057.stdout:OK 2023-10-30T14:15:49.616 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo OK 2023-10-30T14:15:49.616 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T14:15:49.616 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T14:15:49.707 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2023-10-30T14:15:49.707 DEBUG:teuthology.orchestra.run.smithi057:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2023-10-30T14:15:49.765 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo 1 2023-10-30T14:15:49.768 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo 2 2023-10-30T14:15:49.768 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo 3 2023-10-30T14:15:49.769 INFO:tasks.workunit.client.0.smithi057.stderr:+ '[' -e file4 ']' 2023-10-30T14:15:49.769 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/snap1 2023-10-30T14:15:49.792 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo 4 2023-10-30T14:15:49.794 INFO:tasks.workunit.client.0.smithi057.stderr:++ ls 2023-10-30T14:15:49.801 INFO:tasks.workunit.client.0.smithi057.stderr:+ now='file1 2023-10-30T14:15:49.802 INFO:tasks.workunit.client.0.smithi057.stderr:file2 2023-10-30T14:15:49.802 INFO:tasks.workunit.client.0.smithi057.stderr:file3 2023-10-30T14:15:49.802 INFO:tasks.workunit.client.0.smithi057.stderr:file4' 2023-10-30T14:15:49.803 INFO:tasks.workunit.client.0.smithi057.stderr:++ ls .snap/snap1 2023-10-30T14:15:49.815 INFO:tasks.workunit.client.0.smithi057.stderr:+ then='file1 2023-10-30T14:15:49.815 INFO:tasks.workunit.client.0.smithi057.stderr:file2 2023-10-30T14:15:49.815 INFO:tasks.workunit.client.0.smithi057.stderr:file3' 2023-10-30T14:15:49.816 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir .snap/snap1 2023-10-30T14:15:49.825 INFO:tasks.workunit.client.0.smithi057.stderr:+ '[' 'file1 2023-10-30T14:15:49.826 INFO:tasks.workunit.client.0.smithi057.stderr:file2 2023-10-30T14:15:49.826 INFO:tasks.workunit.client.0.smithi057.stderr:file3 2023-10-30T14:15:49.826 INFO:tasks.workunit.client.0.smithi057.stderr:file4' = 'file1 2023-10-30T14:15:49.826 INFO:tasks.workunit.client.0.smithi057.stderr:file2 2023-10-30T14:15:49.826 INFO:tasks.workunit.client.0.smithi057.stderr:file3' ']' 2023-10-30T14:15:49.827 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo 1 2023-10-30T14:15:49.857 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo 2 2023-10-30T14:15:49.876 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo 3 2023-10-30T14:15:49.887 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/snap1 2023-10-30T14:15:49.899 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo 4 2023-10-30T14:15:49.911 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir .snap/snap1 2023-10-30T14:15:49.925 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm file1 file2 file3 file4 2023-10-30T14:15:49.937 INFO:tasks.workunit.client.0.smithi057.stdout:OK 2023-10-30T14:15:49.938 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo OK 2023-10-30T14:15:49.938 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T14:15:49.939 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T14:15:49.974 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2023-10-30T14:15:49.975 DEBUG:teuthology.orchestra.run.smithi057:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2023-10-30T14:15:50.036 INFO:tasks.workunit.client.0.smithi057.stdout:Create dir 100 to 199 ... 2023-10-30T14:15:50.038 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 100 2023-10-30T14:15:50.048 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 101 2023-10-30T14:15:50.711 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 102 2023-10-30T14:15:50.882 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 103 2023-10-30T14:15:50.903 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 104 2023-10-30T14:15:51.702 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 105 2023-10-30T14:15:51.841 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 106 2023-10-30T14:15:52.107 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 107 2023-10-30T14:15:52.372 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 108 2023-10-30T14:15:52.393 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 109 2023-10-30T14:15:52.415 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 110 2023-10-30T14:15:52.436 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 111 2023-10-30T14:15:52.457 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 112 2023-10-30T14:15:52.474 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 113 2023-10-30T14:15:52.492 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 114 2023-10-30T14:15:52.507 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 115 2023-10-30T14:15:52.695 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 116 2023-10-30T14:15:53.033 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 117 2023-10-30T14:15:53.055 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 118 2023-10-30T14:15:53.077 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 119 2023-10-30T14:15:53.121 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 120 2023-10-30T14:15:53.142 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 121 2023-10-30T14:15:53.164 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 122 2023-10-30T14:15:53.186 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 123 2023-10-30T14:15:53.595 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 124 2023-10-30T14:15:53.617 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 125 2023-10-30T14:15:53.640 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 126 2023-10-30T14:15:53.906 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 127 2023-10-30T14:15:53.927 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 128 2023-10-30T14:15:53.971 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 129 2023-10-30T14:15:54.643 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 130 2023-10-30T14:15:54.666 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 131 2023-10-30T14:15:54.689 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 132 2023-10-30T14:15:54.705 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 133 2023-10-30T14:15:54.718 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 134 2023-10-30T14:15:54.733 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 135 2023-10-30T14:15:54.747 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 136 2023-10-30T14:15:54.768 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 137 2023-10-30T14:15:54.788 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 138 2023-10-30T14:15:54.814 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 139 2023-10-30T14:15:54.843 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 140 2023-10-30T14:15:54.873 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 141 2023-10-30T14:15:55.098 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 142 2023-10-30T14:15:55.108 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 143 2023-10-30T14:15:55.122 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 144 2023-10-30T14:15:55.137 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 145 2023-10-30T14:15:55.153 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 146 2023-10-30T14:15:55.169 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 147 2023-10-30T14:15:55.185 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 148 2023-10-30T14:15:55.199 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 149 2023-10-30T14:15:55.213 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 150 2023-10-30T14:15:55.474 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 151 2023-10-30T14:15:55.490 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 152 2023-10-30T14:15:55.511 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 153 2023-10-30T14:15:55.532 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 154 2023-10-30T14:15:55.794 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 155 2023-10-30T14:15:55.816 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 156 2023-10-30T14:15:56.081 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 157 2023-10-30T14:15:56.102 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 158 2023-10-30T14:15:56.124 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 159 2023-10-30T14:15:56.695 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 160 2023-10-30T14:15:56.885 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 161 2023-10-30T14:15:56.907 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 162 2023-10-30T14:15:56.929 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 163 2023-10-30T14:15:56.951 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 164 2023-10-30T14:15:56.972 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 165 2023-10-30T14:15:56.991 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 166 2023-10-30T14:15:57.007 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 167 2023-10-30T14:15:57.271 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 168 2023-10-30T14:15:57.294 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 169 2023-10-30T14:15:57.316 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 170 2023-10-30T14:15:57.338 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 171 2023-10-30T14:15:57.361 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 172 2023-10-30T14:15:57.378 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 173 2023-10-30T14:15:57.396 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 174 2023-10-30T14:15:57.413 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 175 2023-10-30T14:15:57.430 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 176 2023-10-30T14:15:57.446 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 177 2023-10-30T14:15:57.461 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 178 2023-10-30T14:15:57.475 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 179 2023-10-30T14:15:57.490 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 180 2023-10-30T14:15:57.503 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 181 2023-10-30T14:15:57.516 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 182 2023-10-30T14:15:57.528 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 183 2023-10-30T14:15:57.540 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 184 2023-10-30T14:15:57.552 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 185 2023-10-30T14:15:57.564 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 186 2023-10-30T14:15:57.575 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 187 2023-10-30T14:15:57.586 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 188 2023-10-30T14:15:57.597 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 189 2023-10-30T14:15:57.607 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 190 2023-10-30T14:15:57.617 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 191 2023-10-30T14:15:57.884 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 192 2023-10-30T14:15:57.904 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 193 2023-10-30T14:15:57.924 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 194 2023-10-30T14:15:57.940 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 195 2023-10-30T14:15:58.220 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 196 2023-10-30T14:15:58.244 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 197 2023-10-30T14:15:58.267 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 198 2023-10-30T14:15:58.291 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 199 2023-10-30T14:15:58.312 INFO:tasks.workunit.client.0.smithi057.stdout:Take first snapshot .snap/test1 2023-10-30T14:15:58.742 INFO:tasks.workunit.client.0.smithi057.stdout:Create dir 200 to 299 ... 2023-10-30T14:15:58.744 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 200 2023-10-30T14:15:58.864 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 201 2023-10-30T14:15:58.886 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 202 2023-10-30T14:15:58.905 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 203 2023-10-30T14:15:58.925 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 204 2023-10-30T14:15:59.141 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 205 2023-10-30T14:15:59.162 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 206 2023-10-30T14:15:59.183 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 207 2023-10-30T14:15:59.204 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 208 2023-10-30T14:15:59.224 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 209 2023-10-30T14:15:59.240 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 210 2023-10-30T14:15:59.679 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:15:59.683 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:15:59.711 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 211 2023-10-30T14:15:59.740 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 212 2023-10-30T14:16:00.009 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 213 2023-10-30T14:16:00.049 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 214 2023-10-30T14:16:00.070 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 215 2023-10-30T14:16:00.090 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 216 2023-10-30T14:16:00.109 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 217 2023-10-30T14:16:00.127 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 218 2023-10-30T14:16:00.143 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 219 2023-10-30T14:16:00.159 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 220 2023-10-30T14:16:00.174 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 221 2023-10-30T14:16:00.188 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 222 2023-10-30T14:16:00.203 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 223 2023-10-30T14:16:00.217 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 224 2023-10-30T14:16:00.230 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 225 2023-10-30T14:16:00.243 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 226 2023-10-30T14:16:00.255 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 227 2023-10-30T14:16:00.268 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 228 2023-10-30T14:16:00.279 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 229 2023-10-30T14:16:00.290 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 230 2023-10-30T14:16:00.695 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 231 2023-10-30T14:16:00.785 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 232 2023-10-30T14:16:00.806 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 233 2023-10-30T14:16:00.826 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 234 2023-10-30T14:16:00.848 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 235 2023-10-30T14:16:00.869 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 236 2023-10-30T14:16:00.888 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 237 2023-10-30T14:16:00.905 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 238 2023-10-30T14:16:00.920 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 239 2023-10-30T14:16:00.935 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 240 2023-10-30T14:16:00.949 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 241 2023-10-30T14:16:00.963 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 242 2023-10-30T14:16:00.976 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 243 2023-10-30T14:16:00.989 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 244 2023-10-30T14:16:01.002 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 245 2023-10-30T14:16:01.014 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 246 2023-10-30T14:16:01.026 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 247 2023-10-30T14:16:01.038 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 248 2023-10-30T14:16:01.050 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 249 2023-10-30T14:16:01.062 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 250 2023-10-30T14:16:04.784 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 251 2023-10-30T14:16:04.795 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 252 2023-10-30T14:16:04.814 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 253 2023-10-30T14:16:04.833 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 254 2023-10-30T14:16:04.852 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 255 2023-10-30T14:16:04.869 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 256 2023-10-30T14:16:04.885 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 257 2023-10-30T14:16:04.900 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 258 2023-10-30T14:16:04.914 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 259 2023-10-30T14:16:04.928 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 260 2023-10-30T14:16:04.942 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 261 2023-10-30T14:16:04.956 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 262 2023-10-30T14:16:04.969 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 263 2023-10-30T14:16:04.981 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 264 2023-10-30T14:16:04.992 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 265 2023-10-30T14:16:05.252 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 266 2023-10-30T14:16:05.273 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 267 2023-10-30T14:16:05.293 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 268 2023-10-30T14:16:05.314 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 269 2023-10-30T14:16:05.332 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 270 2023-10-30T14:16:05.349 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 271 2023-10-30T14:16:05.696 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 272 2023-10-30T14:16:05.716 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 273 2023-10-30T14:16:05.737 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 274 2023-10-30T14:16:05.755 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 275 2023-10-30T14:16:05.774 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 276 2023-10-30T14:16:05.793 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 277 2023-10-30T14:16:05.810 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 278 2023-10-30T14:16:05.825 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 279 2023-10-30T14:16:05.840 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 280 2023-10-30T14:16:06.178 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 281 2023-10-30T14:16:06.195 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 282 2023-10-30T14:16:06.215 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 283 2023-10-30T14:16:06.237 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 284 2023-10-30T14:16:06.257 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 285 2023-10-30T14:16:06.276 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 286 2023-10-30T14:16:06.294 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 287 2023-10-30T14:16:06.312 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 288 2023-10-30T14:16:06.328 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 289 2023-10-30T14:16:06.343 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 290 2023-10-30T14:16:06.358 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 291 2023-10-30T14:16:06.372 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 292 2023-10-30T14:16:06.631 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 293 2023-10-30T14:16:06.652 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 294 2023-10-30T14:16:06.674 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 295 2023-10-30T14:16:06.702 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 296 2023-10-30T14:16:06.738 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 297 2023-10-30T14:16:06.949 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 298 2023-10-30T14:16:06.978 INFO:tasks.workunit.client.0.smithi057.stdout: create dir 299 2023-10-30T14:16:07.011 INFO:tasks.workunit.client.0.smithi057.stdout:Create a snapshot in every first level dir ... 2023-10-30T14:16:07.169 INFO:tasks.workunit.client.0.smithi057.stdout: create 100/.snap/snap-subdir-test 2023-10-30T14:16:07.833 INFO:tasks.workunit.client.0.smithi057.stdout: create 100/30 file after the snapshot 2023-10-30T14:16:07.836 INFO:tasks.workunit.client.0.smithi057.stdout: create 100/31 file after the snapshot 2023-10-30T14:16:07.836 INFO:tasks.workunit.client.0.smithi057.stdout: create 100/32 file after the snapshot 2023-10-30T14:16:07.836 INFO:tasks.workunit.client.0.smithi057.stdout: create 100/33 file after the snapshot 2023-10-30T14:16:07.837 INFO:tasks.workunit.client.0.smithi057.stdout: create 100/34 file after the snapshot 2023-10-30T14:16:07.837 INFO:tasks.workunit.client.0.smithi057.stdout: create 100/35 file after the snapshot 2023-10-30T14:16:07.837 INFO:tasks.workunit.client.0.smithi057.stdout: create 100/36 file after the snapshot 2023-10-30T14:16:07.837 INFO:tasks.workunit.client.0.smithi057.stdout: create 100/37 file after the snapshot 2023-10-30T14:16:07.837 INFO:tasks.workunit.client.0.smithi057.stdout: create 100/38 file after the snapshot 2023-10-30T14:16:07.838 INFO:tasks.workunit.client.0.smithi057.stdout: create 100/39 file after the snapshot 2023-10-30T14:16:07.838 INFO:tasks.workunit.client.0.smithi057.stdout: create 101/.snap/snap-subdir-test 2023-10-30T14:16:07.921 INFO:tasks.workunit.client.0.smithi057.stdout: create 101/30 file after the snapshot 2023-10-30T14:16:07.926 INFO:tasks.workunit.client.0.smithi057.stdout: create 101/31 file after the snapshot 2023-10-30T14:16:07.927 INFO:tasks.workunit.client.0.smithi057.stdout: create 101/32 file after the snapshot 2023-10-30T14:16:07.927 INFO:tasks.workunit.client.0.smithi057.stdout: create 101/33 file after the snapshot 2023-10-30T14:16:07.927 INFO:tasks.workunit.client.0.smithi057.stdout: create 101/34 file after the snapshot 2023-10-30T14:16:07.927 INFO:tasks.workunit.client.0.smithi057.stdout: create 101/35 file after the snapshot 2023-10-30T14:16:07.928 INFO:tasks.workunit.client.0.smithi057.stdout: create 101/36 file after the snapshot 2023-10-30T14:16:07.928 INFO:tasks.workunit.client.0.smithi057.stdout: create 101/37 file after the snapshot 2023-10-30T14:16:07.928 INFO:tasks.workunit.client.0.smithi057.stdout: create 101/38 file after the snapshot 2023-10-30T14:16:07.928 INFO:tasks.workunit.client.0.smithi057.stdout: create 101/39 file after the snapshot 2023-10-30T14:16:07.928 INFO:tasks.workunit.client.0.smithi057.stdout: create 102/.snap/snap-subdir-test 2023-10-30T14:16:07.980 INFO:tasks.workunit.client.0.smithi057.stdout: create 102/30 file after the snapshot 2023-10-30T14:16:07.984 INFO:tasks.workunit.client.0.smithi057.stdout: create 102/31 file after the snapshot 2023-10-30T14:16:07.984 INFO:tasks.workunit.client.0.smithi057.stdout: create 102/32 file after the snapshot 2023-10-30T14:16:07.984 INFO:tasks.workunit.client.0.smithi057.stdout: create 102/33 file after the snapshot 2023-10-30T14:16:07.984 INFO:tasks.workunit.client.0.smithi057.stdout: create 102/34 file after the snapshot 2023-10-30T14:16:07.984 INFO:tasks.workunit.client.0.smithi057.stdout: create 102/35 file after the snapshot 2023-10-30T14:16:07.985 INFO:tasks.workunit.client.0.smithi057.stdout: create 102/36 file after the snapshot 2023-10-30T14:16:07.985 INFO:tasks.workunit.client.0.smithi057.stdout: create 102/37 file after the snapshot 2023-10-30T14:16:07.985 INFO:tasks.workunit.client.0.smithi057.stdout: create 102/38 file after the snapshot 2023-10-30T14:16:07.985 INFO:tasks.workunit.client.0.smithi057.stdout: create 102/39 file after the snapshot 2023-10-30T14:16:07.985 INFO:tasks.workunit.client.0.smithi057.stdout: create 103/.snap/snap-subdir-test 2023-10-30T14:16:08.050 INFO:tasks.workunit.client.0.smithi057.stdout: create 103/30 file after the snapshot 2023-10-30T14:16:08.053 INFO:tasks.workunit.client.0.smithi057.stdout: create 103/31 file after the snapshot 2023-10-30T14:16:08.053 INFO:tasks.workunit.client.0.smithi057.stdout: create 103/32 file after the snapshot 2023-10-30T14:16:08.053 INFO:tasks.workunit.client.0.smithi057.stdout: create 103/33 file after the snapshot 2023-10-30T14:16:08.053 INFO:tasks.workunit.client.0.smithi057.stdout: create 103/34 file after the snapshot 2023-10-30T14:16:08.054 INFO:tasks.workunit.client.0.smithi057.stdout: create 103/35 file after the snapshot 2023-10-30T14:16:08.054 INFO:tasks.workunit.client.0.smithi057.stdout: create 103/36 file after the snapshot 2023-10-30T14:16:08.054 INFO:tasks.workunit.client.0.smithi057.stdout: create 103/37 file after the snapshot 2023-10-30T14:16:08.054 INFO:tasks.workunit.client.0.smithi057.stdout: create 103/38 file after the snapshot 2023-10-30T14:16:08.054 INFO:tasks.workunit.client.0.smithi057.stdout: create 103/39 file after the snapshot 2023-10-30T14:16:08.055 INFO:tasks.workunit.client.0.smithi057.stdout: create 104/.snap/snap-subdir-test 2023-10-30T14:16:08.123 INFO:tasks.workunit.client.0.smithi057.stdout: create 104/30 file after the snapshot 2023-10-30T14:16:08.128 INFO:tasks.workunit.client.0.smithi057.stdout: create 104/31 file after the snapshot 2023-10-30T14:16:08.129 INFO:tasks.workunit.client.0.smithi057.stdout: create 104/32 file after the snapshot 2023-10-30T14:16:08.129 INFO:tasks.workunit.client.0.smithi057.stdout: create 104/33 file after the snapshot 2023-10-30T14:16:08.129 INFO:tasks.workunit.client.0.smithi057.stdout: create 104/34 file after the snapshot 2023-10-30T14:16:08.129 INFO:tasks.workunit.client.0.smithi057.stdout: create 104/35 file after the snapshot 2023-10-30T14:16:08.129 INFO:tasks.workunit.client.0.smithi057.stdout: create 104/36 file after the snapshot 2023-10-30T14:16:08.130 INFO:tasks.workunit.client.0.smithi057.stdout: create 104/37 file after the snapshot 2023-10-30T14:16:08.130 INFO:tasks.workunit.client.0.smithi057.stdout: create 104/38 file after the snapshot 2023-10-30T14:16:08.130 INFO:tasks.workunit.client.0.smithi057.stdout: create 104/39 file after the snapshot 2023-10-30T14:16:08.130 INFO:tasks.workunit.client.0.smithi057.stdout: create 105/.snap/snap-subdir-test 2023-10-30T14:16:08.196 INFO:tasks.workunit.client.0.smithi057.stdout: create 105/30 file after the snapshot 2023-10-30T14:16:08.199 INFO:tasks.workunit.client.0.smithi057.stdout: create 105/31 file after the snapshot 2023-10-30T14:16:08.199 INFO:tasks.workunit.client.0.smithi057.stdout: create 105/32 file after the snapshot 2023-10-30T14:16:08.199 INFO:tasks.workunit.client.0.smithi057.stdout: create 105/33 file after the snapshot 2023-10-30T14:16:08.199 INFO:tasks.workunit.client.0.smithi057.stdout: create 105/34 file after the snapshot 2023-10-30T14:16:08.200 INFO:tasks.workunit.client.0.smithi057.stdout: create 105/35 file after the snapshot 2023-10-30T14:16:08.200 INFO:tasks.workunit.client.0.smithi057.stdout: create 105/36 file after the snapshot 2023-10-30T14:16:08.200 INFO:tasks.workunit.client.0.smithi057.stdout: create 105/37 file after the snapshot 2023-10-30T14:16:08.200 INFO:tasks.workunit.client.0.smithi057.stdout: create 105/38 file after the snapshot 2023-10-30T14:16:08.200 INFO:tasks.workunit.client.0.smithi057.stdout: create 105/39 file after the snapshot 2023-10-30T14:16:08.200 INFO:tasks.workunit.client.0.smithi057.stdout: create 106/.snap/snap-subdir-test 2023-10-30T14:16:08.267 INFO:tasks.workunit.client.0.smithi057.stdout: create 106/30 file after the snapshot 2023-10-30T14:16:08.271 INFO:tasks.workunit.client.0.smithi057.stdout: create 106/31 file after the snapshot 2023-10-30T14:16:08.271 INFO:tasks.workunit.client.0.smithi057.stdout: create 106/32 file after the snapshot 2023-10-30T14:16:08.271 INFO:tasks.workunit.client.0.smithi057.stdout: create 106/33 file after the snapshot 2023-10-30T14:16:08.272 INFO:tasks.workunit.client.0.smithi057.stdout: create 106/34 file after the snapshot 2023-10-30T14:16:08.272 INFO:tasks.workunit.client.0.smithi057.stdout: create 106/35 file after the snapshot 2023-10-30T14:16:08.272 INFO:tasks.workunit.client.0.smithi057.stdout: create 106/36 file after the snapshot 2023-10-30T14:16:08.272 INFO:tasks.workunit.client.0.smithi057.stdout: create 106/37 file after the snapshot 2023-10-30T14:16:08.273 INFO:tasks.workunit.client.0.smithi057.stdout: create 106/38 file after the snapshot 2023-10-30T14:16:08.273 INFO:tasks.workunit.client.0.smithi057.stdout: create 106/39 file after the snapshot 2023-10-30T14:16:08.273 INFO:tasks.workunit.client.0.smithi057.stdout: create 107/.snap/snap-subdir-test 2023-10-30T14:16:08.338 INFO:tasks.workunit.client.0.smithi057.stdout: create 107/30 file after the snapshot 2023-10-30T14:16:08.343 INFO:tasks.workunit.client.0.smithi057.stdout: create 107/31 file after the snapshot 2023-10-30T14:16:08.343 INFO:tasks.workunit.client.0.smithi057.stdout: create 107/32 file after the snapshot 2023-10-30T14:16:08.344 INFO:tasks.workunit.client.0.smithi057.stdout: create 107/33 file after the snapshot 2023-10-30T14:16:08.344 INFO:tasks.workunit.client.0.smithi057.stdout: create 107/34 file after the snapshot 2023-10-30T14:16:08.344 INFO:tasks.workunit.client.0.smithi057.stdout: create 107/35 file after the snapshot 2023-10-30T14:16:08.345 INFO:tasks.workunit.client.0.smithi057.stdout: create 107/36 file after the snapshot 2023-10-30T14:16:08.345 INFO:tasks.workunit.client.0.smithi057.stdout: create 107/37 file after the snapshot 2023-10-30T14:16:08.345 INFO:tasks.workunit.client.0.smithi057.stdout: create 107/38 file after the snapshot 2023-10-30T14:16:08.345 INFO:tasks.workunit.client.0.smithi057.stdout: create 107/39 file after the snapshot 2023-10-30T14:16:08.346 INFO:tasks.workunit.client.0.smithi057.stdout: create 108/.snap/snap-subdir-test 2023-10-30T14:16:08.409 INFO:tasks.workunit.client.0.smithi057.stdout: create 108/30 file after the snapshot 2023-10-30T14:16:08.413 INFO:tasks.workunit.client.0.smithi057.stdout: create 108/31 file after the snapshot 2023-10-30T14:16:08.413 INFO:tasks.workunit.client.0.smithi057.stdout: create 108/32 file after the snapshot 2023-10-30T14:16:08.413 INFO:tasks.workunit.client.0.smithi057.stdout: create 108/33 file after the snapshot 2023-10-30T14:16:08.414 INFO:tasks.workunit.client.0.smithi057.stdout: create 108/34 file after the snapshot 2023-10-30T14:16:08.414 INFO:tasks.workunit.client.0.smithi057.stdout: create 108/35 file after the snapshot 2023-10-30T14:16:08.414 INFO:tasks.workunit.client.0.smithi057.stdout: create 108/36 file after the snapshot 2023-10-30T14:16:08.414 INFO:tasks.workunit.client.0.smithi057.stdout: create 108/37 file after the snapshot 2023-10-30T14:16:08.415 INFO:tasks.workunit.client.0.smithi057.stdout: create 108/38 file after the snapshot 2023-10-30T14:16:08.415 INFO:tasks.workunit.client.0.smithi057.stdout: create 108/39 file after the snapshot 2023-10-30T14:16:08.415 INFO:tasks.workunit.client.0.smithi057.stdout: create 109/.snap/snap-subdir-test 2023-10-30T14:16:09.213 INFO:tasks.workunit.client.0.smithi057.stdout: create 109/30 file after the snapshot 2023-10-30T14:16:09.215 INFO:tasks.workunit.client.0.smithi057.stdout: create 109/31 file after the snapshot 2023-10-30T14:16:09.215 INFO:tasks.workunit.client.0.smithi057.stdout: create 109/32 file after the snapshot 2023-10-30T14:16:09.215 INFO:tasks.workunit.client.0.smithi057.stdout: create 109/33 file after the snapshot 2023-10-30T14:16:09.215 INFO:tasks.workunit.client.0.smithi057.stdout: create 109/34 file after the snapshot 2023-10-30T14:16:09.216 INFO:tasks.workunit.client.0.smithi057.stdout: create 109/35 file after the snapshot 2023-10-30T14:16:09.216 INFO:tasks.workunit.client.0.smithi057.stdout: create 109/36 file after the snapshot 2023-10-30T14:16:09.216 INFO:tasks.workunit.client.0.smithi057.stdout: create 109/37 file after the snapshot 2023-10-30T14:16:09.216 INFO:tasks.workunit.client.0.smithi057.stdout: create 109/38 file after the snapshot 2023-10-30T14:16:09.216 INFO:tasks.workunit.client.0.smithi057.stdout: create 109/39 file after the snapshot 2023-10-30T14:16:09.217 INFO:tasks.workunit.client.0.smithi057.stdout: create 110/.snap/snap-subdir-test 2023-10-30T14:16:09.980 INFO:tasks.workunit.client.0.smithi057.stdout: create 110/30 file after the snapshot 2023-10-30T14:16:09.985 INFO:tasks.workunit.client.0.smithi057.stdout: create 110/31 file after the snapshot 2023-10-30T14:16:09.985 INFO:tasks.workunit.client.0.smithi057.stdout: create 110/32 file after the snapshot 2023-10-30T14:16:09.985 INFO:tasks.workunit.client.0.smithi057.stdout: create 110/33 file after the snapshot 2023-10-30T14:16:09.986 INFO:tasks.workunit.client.0.smithi057.stdout: create 110/34 file after the snapshot 2023-10-30T14:16:09.986 INFO:tasks.workunit.client.0.smithi057.stdout: create 110/35 file after the snapshot 2023-10-30T14:16:09.986 INFO:tasks.workunit.client.0.smithi057.stdout: create 110/36 file after the snapshot 2023-10-30T14:16:09.986 INFO:tasks.workunit.client.0.smithi057.stdout: create 110/37 file after the snapshot 2023-10-30T14:16:09.987 INFO:tasks.workunit.client.0.smithi057.stdout: create 110/38 file after the snapshot 2023-10-30T14:16:09.987 INFO:tasks.workunit.client.0.smithi057.stdout: create 110/39 file after the snapshot 2023-10-30T14:16:09.987 INFO:tasks.workunit.client.0.smithi057.stdout: create 111/.snap/snap-subdir-test 2023-10-30T14:16:10.070 INFO:tasks.workunit.client.0.smithi057.stdout: create 111/30 file after the snapshot 2023-10-30T14:16:10.075 INFO:tasks.workunit.client.0.smithi057.stdout: create 111/31 file after the snapshot 2023-10-30T14:16:10.075 INFO:tasks.workunit.client.0.smithi057.stdout: create 111/32 file after the snapshot 2023-10-30T14:16:10.075 INFO:tasks.workunit.client.0.smithi057.stdout: create 111/33 file after the snapshot 2023-10-30T14:16:10.076 INFO:tasks.workunit.client.0.smithi057.stdout: create 111/34 file after the snapshot 2023-10-30T14:16:10.076 INFO:tasks.workunit.client.0.smithi057.stdout: create 111/35 file after the snapshot 2023-10-30T14:16:10.076 INFO:tasks.workunit.client.0.smithi057.stdout: create 111/36 file after the snapshot 2023-10-30T14:16:10.076 INFO:tasks.workunit.client.0.smithi057.stdout: create 111/37 file after the snapshot 2023-10-30T14:16:10.077 INFO:tasks.workunit.client.0.smithi057.stdout: create 111/38 file after the snapshot 2023-10-30T14:16:10.077 INFO:tasks.workunit.client.0.smithi057.stdout: create 111/39 file after the snapshot 2023-10-30T14:16:10.077 INFO:tasks.workunit.client.0.smithi057.stdout: create 112/.snap/snap-subdir-test 2023-10-30T14:16:10.616 INFO:tasks.workunit.client.0.smithi057.stdout: create 112/30 file after the snapshot 2023-10-30T14:16:10.621 INFO:tasks.workunit.client.0.smithi057.stdout: create 112/31 file after the snapshot 2023-10-30T14:16:10.621 INFO:tasks.workunit.client.0.smithi057.stdout: create 112/32 file after the snapshot 2023-10-30T14:16:10.621 INFO:tasks.workunit.client.0.smithi057.stdout: create 112/33 file after the snapshot 2023-10-30T14:16:10.622 INFO:tasks.workunit.client.0.smithi057.stdout: create 112/34 file after the snapshot 2023-10-30T14:16:10.622 INFO:tasks.workunit.client.0.smithi057.stdout: create 112/35 file after the snapshot 2023-10-30T14:16:10.622 INFO:tasks.workunit.client.0.smithi057.stdout: create 112/36 file after the snapshot 2023-10-30T14:16:10.622 INFO:tasks.workunit.client.0.smithi057.stdout: create 112/37 file after the snapshot 2023-10-30T14:16:10.623 INFO:tasks.workunit.client.0.smithi057.stdout: create 112/38 file after the snapshot 2023-10-30T14:16:10.623 INFO:tasks.workunit.client.0.smithi057.stdout: create 112/39 file after the snapshot 2023-10-30T14:16:10.623 INFO:tasks.workunit.client.0.smithi057.stdout: create 113/.snap/snap-subdir-test 2023-10-30T14:16:10.695 INFO:tasks.workunit.client.0.smithi057.stdout: create 113/30 file after the snapshot 2023-10-30T14:16:10.700 INFO:tasks.workunit.client.0.smithi057.stdout: create 113/31 file after the snapshot 2023-10-30T14:16:10.700 INFO:tasks.workunit.client.0.smithi057.stdout: create 113/32 file after the snapshot 2023-10-30T14:16:10.701 INFO:tasks.workunit.client.0.smithi057.stdout: create 113/33 file after the snapshot 2023-10-30T14:16:10.701 INFO:tasks.workunit.client.0.smithi057.stdout: create 113/34 file after the snapshot 2023-10-30T14:16:10.701 INFO:tasks.workunit.client.0.smithi057.stdout: create 113/35 file after the snapshot 2023-10-30T14:16:10.701 INFO:tasks.workunit.client.0.smithi057.stdout: create 113/36 file after the snapshot 2023-10-30T14:16:10.701 INFO:tasks.workunit.client.0.smithi057.stdout: create 113/37 file after the snapshot 2023-10-30T14:16:10.702 INFO:tasks.workunit.client.0.smithi057.stdout: create 113/38 file after the snapshot 2023-10-30T14:16:10.702 INFO:tasks.workunit.client.0.smithi057.stdout: create 113/39 file after the snapshot 2023-10-30T14:16:10.702 INFO:tasks.workunit.client.0.smithi057.stdout: create 114/.snap/snap-subdir-test 2023-10-30T14:16:10.757 INFO:tasks.workunit.client.0.smithi057.stdout: create 114/30 file after the snapshot 2023-10-30T14:16:10.760 INFO:tasks.workunit.client.0.smithi057.stdout: create 114/31 file after the snapshot 2023-10-30T14:16:10.761 INFO:tasks.workunit.client.0.smithi057.stdout: create 114/32 file after the snapshot 2023-10-30T14:16:10.761 INFO:tasks.workunit.client.0.smithi057.stdout: create 114/33 file after the snapshot 2023-10-30T14:16:10.761 INFO:tasks.workunit.client.0.smithi057.stdout: create 114/34 file after the snapshot 2023-10-30T14:16:10.761 INFO:tasks.workunit.client.0.smithi057.stdout: create 114/35 file after the snapshot 2023-10-30T14:16:10.761 INFO:tasks.workunit.client.0.smithi057.stdout: create 114/36 file after the snapshot 2023-10-30T14:16:10.762 INFO:tasks.workunit.client.0.smithi057.stdout: create 114/37 file after the snapshot 2023-10-30T14:16:10.762 INFO:tasks.workunit.client.0.smithi057.stdout: create 114/38 file after the snapshot 2023-10-30T14:16:10.762 INFO:tasks.workunit.client.0.smithi057.stdout: create 114/39 file after the snapshot 2023-10-30T14:16:10.762 INFO:tasks.workunit.client.0.smithi057.stdout: create 115/.snap/snap-subdir-test 2023-10-30T14:16:10.819 INFO:tasks.workunit.client.0.smithi057.stdout: create 115/30 file after the snapshot 2023-10-30T14:16:10.822 INFO:tasks.workunit.client.0.smithi057.stdout: create 115/31 file after the snapshot 2023-10-30T14:16:10.822 INFO:tasks.workunit.client.0.smithi057.stdout: create 115/32 file after the snapshot 2023-10-30T14:16:10.823 INFO:tasks.workunit.client.0.smithi057.stdout: create 115/33 file after the snapshot 2023-10-30T14:16:10.823 INFO:tasks.workunit.client.0.smithi057.stdout: create 115/34 file after the snapshot 2023-10-30T14:16:10.823 INFO:tasks.workunit.client.0.smithi057.stdout: create 115/35 file after the snapshot 2023-10-30T14:16:10.823 INFO:tasks.workunit.client.0.smithi057.stdout: create 115/36 file after the snapshot 2023-10-30T14:16:10.823 INFO:tasks.workunit.client.0.smithi057.stdout: create 115/37 file after the snapshot 2023-10-30T14:16:10.823 INFO:tasks.workunit.client.0.smithi057.stdout: create 115/38 file after the snapshot 2023-10-30T14:16:10.824 INFO:tasks.workunit.client.0.smithi057.stdout: create 115/39 file after the snapshot 2023-10-30T14:16:10.824 INFO:tasks.workunit.client.0.smithi057.stdout: create 116/.snap/snap-subdir-test 2023-10-30T14:16:11.134 INFO:tasks.workunit.client.0.smithi057.stdout: create 116/30 file after the snapshot 2023-10-30T14:16:11.139 INFO:tasks.workunit.client.0.smithi057.stdout: create 116/31 file after the snapshot 2023-10-30T14:16:11.139 INFO:tasks.workunit.client.0.smithi057.stdout: create 116/32 file after the snapshot 2023-10-30T14:16:11.139 INFO:tasks.workunit.client.0.smithi057.stdout: create 116/33 file after the snapshot 2023-10-30T14:16:11.139 INFO:tasks.workunit.client.0.smithi057.stdout: create 116/34 file after the snapshot 2023-10-30T14:16:11.140 INFO:tasks.workunit.client.0.smithi057.stdout: create 116/35 file after the snapshot 2023-10-30T14:16:11.141 INFO:tasks.workunit.client.0.smithi057.stdout: create 116/36 file after the snapshot 2023-10-30T14:16:11.141 INFO:tasks.workunit.client.0.smithi057.stdout: create 116/37 file after the snapshot 2023-10-30T14:16:11.141 INFO:tasks.workunit.client.0.smithi057.stdout: create 116/38 file after the snapshot 2023-10-30T14:16:11.141 INFO:tasks.workunit.client.0.smithi057.stdout: create 116/39 file after the snapshot 2023-10-30T14:16:11.142 INFO:tasks.workunit.client.0.smithi057.stdout: create 117/.snap/snap-subdir-test 2023-10-30T14:16:11.628 INFO:tasks.workunit.client.0.smithi057.stdout: create 117/30 file after the snapshot 2023-10-30T14:16:11.632 INFO:tasks.workunit.client.0.smithi057.stdout: create 117/31 file after the snapshot 2023-10-30T14:16:11.633 INFO:tasks.workunit.client.0.smithi057.stdout: create 117/32 file after the snapshot 2023-10-30T14:16:11.633 INFO:tasks.workunit.client.0.smithi057.stdout: create 117/33 file after the snapshot 2023-10-30T14:16:11.633 INFO:tasks.workunit.client.0.smithi057.stdout: create 117/34 file after the snapshot 2023-10-30T14:16:11.634 INFO:tasks.workunit.client.0.smithi057.stdout: create 117/35 file after the snapshot 2023-10-30T14:16:11.634 INFO:tasks.workunit.client.0.smithi057.stdout: create 117/36 file after the snapshot 2023-10-30T14:16:11.635 INFO:tasks.workunit.client.0.smithi057.stdout: create 117/37 file after the snapshot 2023-10-30T14:16:11.635 INFO:tasks.workunit.client.0.smithi057.stdout: create 117/38 file after the snapshot 2023-10-30T14:16:11.635 INFO:tasks.workunit.client.0.smithi057.stdout: create 117/39 file after the snapshot 2023-10-30T14:16:11.635 INFO:tasks.workunit.client.0.smithi057.stdout: create 118/.snap/snap-subdir-test 2023-10-30T14:16:11.803 INFO:tasks.workunit.client.0.smithi057.stdout: create 118/30 file after the snapshot 2023-10-30T14:16:12.187 INFO:tasks.workunit.client.0.smithi057.stdout: create 118/31 file after the snapshot 2023-10-30T14:16:12.188 INFO:tasks.workunit.client.0.smithi057.stdout: create 118/32 file after the snapshot 2023-10-30T14:16:12.188 INFO:tasks.workunit.client.0.smithi057.stdout: create 118/33 file after the snapshot 2023-10-30T14:16:12.188 INFO:tasks.workunit.client.0.smithi057.stdout: create 118/34 file after the snapshot 2023-10-30T14:16:12.188 INFO:tasks.workunit.client.0.smithi057.stdout: create 118/35 file after the snapshot 2023-10-30T14:16:12.188 INFO:tasks.workunit.client.0.smithi057.stdout: create 118/36 file after the snapshot 2023-10-30T14:16:12.189 INFO:tasks.workunit.client.0.smithi057.stdout: create 118/37 file after the snapshot 2023-10-30T14:16:12.189 INFO:tasks.workunit.client.0.smithi057.stdout: create 118/38 file after the snapshot 2023-10-30T14:16:12.189 INFO:tasks.workunit.client.0.smithi057.stdout: create 118/39 file after the snapshot 2023-10-30T14:16:12.189 INFO:tasks.workunit.client.0.smithi057.stdout: create 119/.snap/snap-subdir-test 2023-10-30T14:16:12.255 INFO:tasks.workunit.client.0.smithi057.stdout: create 119/30 file after the snapshot 2023-10-30T14:16:12.258 INFO:tasks.workunit.client.0.smithi057.stdout: create 119/31 file after the snapshot 2023-10-30T14:16:12.259 INFO:tasks.workunit.client.0.smithi057.stdout: create 119/32 file after the snapshot 2023-10-30T14:16:12.259 INFO:tasks.workunit.client.0.smithi057.stdout: create 119/33 file after the snapshot 2023-10-30T14:16:12.259 INFO:tasks.workunit.client.0.smithi057.stdout: create 119/34 file after the snapshot 2023-10-30T14:16:12.259 INFO:tasks.workunit.client.0.smithi057.stdout: create 119/35 file after the snapshot 2023-10-30T14:16:12.259 INFO:tasks.workunit.client.0.smithi057.stdout: create 119/36 file after the snapshot 2023-10-30T14:16:12.259 INFO:tasks.workunit.client.0.smithi057.stdout: create 119/37 file after the snapshot 2023-10-30T14:16:12.260 INFO:tasks.workunit.client.0.smithi057.stdout: create 119/38 file after the snapshot 2023-10-30T14:16:12.260 INFO:tasks.workunit.client.0.smithi057.stdout: create 119/39 file after the snapshot 2023-10-30T14:16:12.260 INFO:tasks.workunit.client.0.smithi057.stdout: create 120/.snap/snap-subdir-test 2023-10-30T14:16:12.326 INFO:tasks.workunit.client.0.smithi057.stdout: create 120/30 file after the snapshot 2023-10-30T14:16:12.329 INFO:tasks.workunit.client.0.smithi057.stdout: create 120/31 file after the snapshot 2023-10-30T14:16:12.329 INFO:tasks.workunit.client.0.smithi057.stdout: create 120/32 file after the snapshot 2023-10-30T14:16:12.329 INFO:tasks.workunit.client.0.smithi057.stdout: create 120/33 file after the snapshot 2023-10-30T14:16:12.330 INFO:tasks.workunit.client.0.smithi057.stdout: create 120/34 file after the snapshot 2023-10-30T14:16:12.330 INFO:tasks.workunit.client.0.smithi057.stdout: create 120/35 file after the snapshot 2023-10-30T14:16:12.330 INFO:tasks.workunit.client.0.smithi057.stdout: create 120/36 file after the snapshot 2023-10-30T14:16:12.330 INFO:tasks.workunit.client.0.smithi057.stdout: create 120/37 file after the snapshot 2023-10-30T14:16:12.330 INFO:tasks.workunit.client.0.smithi057.stdout: create 120/38 file after the snapshot 2023-10-30T14:16:12.331 INFO:tasks.workunit.client.0.smithi057.stdout: create 120/39 file after the snapshot 2023-10-30T14:16:12.331 INFO:tasks.workunit.client.0.smithi057.stdout: create 121/.snap/snap-subdir-test 2023-10-30T14:16:13.206 INFO:tasks.workunit.client.0.smithi057.stdout: create 121/30 file after the snapshot 2023-10-30T14:16:13.214 INFO:tasks.workunit.client.0.smithi057.stdout: create 121/31 file after the snapshot 2023-10-30T14:16:13.214 INFO:tasks.workunit.client.0.smithi057.stdout: create 121/32 file after the snapshot 2023-10-30T14:16:13.214 INFO:tasks.workunit.client.0.smithi057.stdout: create 121/33 file after the snapshot 2023-10-30T14:16:13.214 INFO:tasks.workunit.client.0.smithi057.stdout: create 121/34 file after the snapshot 2023-10-30T14:16:13.215 INFO:tasks.workunit.client.0.smithi057.stdout: create 121/35 file after the snapshot 2023-10-30T14:16:13.215 INFO:tasks.workunit.client.0.smithi057.stdout: create 121/36 file after the snapshot 2023-10-30T14:16:13.215 INFO:tasks.workunit.client.0.smithi057.stdout: create 121/37 file after the snapshot 2023-10-30T14:16:13.215 INFO:tasks.workunit.client.0.smithi057.stdout: create 121/38 file after the snapshot 2023-10-30T14:16:13.215 INFO:tasks.workunit.client.0.smithi057.stdout: create 121/39 file after the snapshot 2023-10-30T14:16:13.216 INFO:tasks.workunit.client.0.smithi057.stdout: create 122/.snap/snap-subdir-test 2023-10-30T14:16:13.302 INFO:tasks.workunit.client.0.smithi057.stdout: create 122/30 file after the snapshot 2023-10-30T14:16:13.307 INFO:tasks.workunit.client.0.smithi057.stdout: create 122/31 file after the snapshot 2023-10-30T14:16:13.307 INFO:tasks.workunit.client.0.smithi057.stdout: create 122/32 file after the snapshot 2023-10-30T14:16:13.307 INFO:tasks.workunit.client.0.smithi057.stdout: create 122/33 file after the snapshot 2023-10-30T14:16:13.307 INFO:tasks.workunit.client.0.smithi057.stdout: create 122/34 file after the snapshot 2023-10-30T14:16:13.308 INFO:tasks.workunit.client.0.smithi057.stdout: create 122/35 file after the snapshot 2023-10-30T14:16:13.308 INFO:tasks.workunit.client.0.smithi057.stdout: create 122/36 file after the snapshot 2023-10-30T14:16:13.308 INFO:tasks.workunit.client.0.smithi057.stdout: create 122/37 file after the snapshot 2023-10-30T14:16:13.308 INFO:tasks.workunit.client.0.smithi057.stdout: create 122/38 file after the snapshot 2023-10-30T14:16:13.308 INFO:tasks.workunit.client.0.smithi057.stdout: create 122/39 file after the snapshot 2023-10-30T14:16:13.309 INFO:tasks.workunit.client.0.smithi057.stdout: create 123/.snap/snap-subdir-test 2023-10-30T14:16:13.396 INFO:tasks.workunit.client.0.smithi057.stdout: create 123/30 file after the snapshot 2023-10-30T14:16:13.398 INFO:tasks.workunit.client.0.smithi057.stdout: create 123/31 file after the snapshot 2023-10-30T14:16:13.399 INFO:tasks.workunit.client.0.smithi057.stdout: create 123/32 file after the snapshot 2023-10-30T14:16:13.399 INFO:tasks.workunit.client.0.smithi057.stdout: create 123/33 file after the snapshot 2023-10-30T14:16:13.399 INFO:tasks.workunit.client.0.smithi057.stdout: create 123/34 file after the snapshot 2023-10-30T14:16:13.399 INFO:tasks.workunit.client.0.smithi057.stdout: create 123/35 file after the snapshot 2023-10-30T14:16:13.399 INFO:tasks.workunit.client.0.smithi057.stdout: create 123/36 file after the snapshot 2023-10-30T14:16:13.400 INFO:tasks.workunit.client.0.smithi057.stdout: create 123/37 file after the snapshot 2023-10-30T14:16:13.400 INFO:tasks.workunit.client.0.smithi057.stdout: create 123/38 file after the snapshot 2023-10-30T14:16:13.400 INFO:tasks.workunit.client.0.smithi057.stdout: create 123/39 file after the snapshot 2023-10-30T14:16:13.400 INFO:tasks.workunit.client.0.smithi057.stdout: create 124/.snap/snap-subdir-test 2023-10-30T14:16:13.457 INFO:tasks.workunit.client.0.smithi057.stdout: create 124/30 file after the snapshot 2023-10-30T14:16:13.459 INFO:tasks.workunit.client.0.smithi057.stdout: create 124/31 file after the snapshot 2023-10-30T14:16:13.460 INFO:tasks.workunit.client.0.smithi057.stdout: create 124/32 file after the snapshot 2023-10-30T14:16:13.460 INFO:tasks.workunit.client.0.smithi057.stdout: create 124/33 file after the snapshot 2023-10-30T14:16:13.460 INFO:tasks.workunit.client.0.smithi057.stdout: create 124/34 file after the snapshot 2023-10-30T14:16:13.460 INFO:tasks.workunit.client.0.smithi057.stdout: create 124/35 file after the snapshot 2023-10-30T14:16:13.460 INFO:tasks.workunit.client.0.smithi057.stdout: create 124/36 file after the snapshot 2023-10-30T14:16:13.461 INFO:tasks.workunit.client.0.smithi057.stdout: create 124/37 file after the snapshot 2023-10-30T14:16:13.461 INFO:tasks.workunit.client.0.smithi057.stdout: create 124/38 file after the snapshot 2023-10-30T14:16:13.461 INFO:tasks.workunit.client.0.smithi057.stdout: create 124/39 file after the snapshot 2023-10-30T14:16:13.461 INFO:tasks.workunit.client.0.smithi057.stdout: create 125/.snap/snap-subdir-test 2023-10-30T14:16:14.238 INFO:tasks.workunit.client.0.smithi057.stdout: create 125/30 file after the snapshot 2023-10-30T14:16:14.241 INFO:tasks.workunit.client.0.smithi057.stdout: create 125/31 file after the snapshot 2023-10-30T14:16:14.241 INFO:tasks.workunit.client.0.smithi057.stdout: create 125/32 file after the snapshot 2023-10-30T14:16:14.242 INFO:tasks.workunit.client.0.smithi057.stdout: create 125/33 file after the snapshot 2023-10-30T14:16:14.242 INFO:tasks.workunit.client.0.smithi057.stdout: create 125/34 file after the snapshot 2023-10-30T14:16:14.242 INFO:tasks.workunit.client.0.smithi057.stdout: create 125/35 file after the snapshot 2023-10-30T14:16:14.242 INFO:tasks.workunit.client.0.smithi057.stdout: create 125/36 file after the snapshot 2023-10-30T14:16:14.243 INFO:tasks.workunit.client.0.smithi057.stdout: create 125/37 file after the snapshot 2023-10-30T14:16:14.243 INFO:tasks.workunit.client.0.smithi057.stdout: create 125/38 file after the snapshot 2023-10-30T14:16:14.243 INFO:tasks.workunit.client.0.smithi057.stdout: create 125/39 file after the snapshot 2023-10-30T14:16:14.243 INFO:tasks.workunit.client.0.smithi057.stdout: create 126/.snap/snap-subdir-test 2023-10-30T14:16:14.308 INFO:tasks.workunit.client.0.smithi057.stdout: create 126/30 file after the snapshot 2023-10-30T14:16:14.310 INFO:tasks.workunit.client.0.smithi057.stdout: create 126/31 file after the snapshot 2023-10-30T14:16:14.310 INFO:tasks.workunit.client.0.smithi057.stdout: create 126/32 file after the snapshot 2023-10-30T14:16:14.311 INFO:tasks.workunit.client.0.smithi057.stdout: create 126/33 file after the snapshot 2023-10-30T14:16:14.312 INFO:tasks.workunit.client.0.smithi057.stdout: create 126/34 file after the snapshot 2023-10-30T14:16:14.312 INFO:tasks.workunit.client.0.smithi057.stdout: create 126/35 file after the snapshot 2023-10-30T14:16:14.312 INFO:tasks.workunit.client.0.smithi057.stdout: create 126/36 file after the snapshot 2023-10-30T14:16:14.312 INFO:tasks.workunit.client.0.smithi057.stdout: create 126/37 file after the snapshot 2023-10-30T14:16:14.312 INFO:tasks.workunit.client.0.smithi057.stdout: create 126/38 file after the snapshot 2023-10-30T14:16:14.312 INFO:tasks.workunit.client.0.smithi057.stdout: create 126/39 file after the snapshot 2023-10-30T14:16:14.313 INFO:tasks.workunit.client.0.smithi057.stdout: create 127/.snap/snap-subdir-test 2023-10-30T14:16:14.377 INFO:tasks.workunit.client.0.smithi057.stdout: create 127/30 file after the snapshot 2023-10-30T14:16:14.380 INFO:tasks.workunit.client.0.smithi057.stdout: create 127/31 file after the snapshot 2023-10-30T14:16:14.380 INFO:tasks.workunit.client.0.smithi057.stdout: create 127/32 file after the snapshot 2023-10-30T14:16:14.380 INFO:tasks.workunit.client.0.smithi057.stdout: create 127/33 file after the snapshot 2023-10-30T14:16:14.380 INFO:tasks.workunit.client.0.smithi057.stdout: create 127/34 file after the snapshot 2023-10-30T14:16:14.380 INFO:tasks.workunit.client.0.smithi057.stdout: create 127/35 file after the snapshot 2023-10-30T14:16:14.381 INFO:tasks.workunit.client.0.smithi057.stdout: create 127/36 file after the snapshot 2023-10-30T14:16:14.381 INFO:tasks.workunit.client.0.smithi057.stdout: create 127/37 file after the snapshot 2023-10-30T14:16:14.381 INFO:tasks.workunit.client.0.smithi057.stdout: create 127/38 file after the snapshot 2023-10-30T14:16:14.381 INFO:tasks.workunit.client.0.smithi057.stdout: create 127/39 file after the snapshot 2023-10-30T14:16:14.381 INFO:tasks.workunit.client.0.smithi057.stdout: create 128/.snap/snap-subdir-test 2023-10-30T14:16:14.449 INFO:tasks.workunit.client.0.smithi057.stdout: create 128/30 file after the snapshot 2023-10-30T14:16:14.453 INFO:tasks.workunit.client.0.smithi057.stdout: create 128/31 file after the snapshot 2023-10-30T14:16:14.453 INFO:tasks.workunit.client.0.smithi057.stdout: create 128/32 file after the snapshot 2023-10-30T14:16:14.453 INFO:tasks.workunit.client.0.smithi057.stdout: create 128/33 file after the snapshot 2023-10-30T14:16:14.453 INFO:tasks.workunit.client.0.smithi057.stdout: create 128/34 file after the snapshot 2023-10-30T14:16:14.454 INFO:tasks.workunit.client.0.smithi057.stdout: create 128/35 file after the snapshot 2023-10-30T14:16:14.454 INFO:tasks.workunit.client.0.smithi057.stdout: create 128/36 file after the snapshot 2023-10-30T14:16:14.454 INFO:tasks.workunit.client.0.smithi057.stdout: create 128/37 file after the snapshot 2023-10-30T14:16:14.454 INFO:tasks.workunit.client.0.smithi057.stdout: create 128/38 file after the snapshot 2023-10-30T14:16:14.454 INFO:tasks.workunit.client.0.smithi057.stdout: create 128/39 file after the snapshot 2023-10-30T14:16:14.455 INFO:tasks.workunit.client.0.smithi057.stdout: create 129/.snap/snap-subdir-test 2023-10-30T14:16:14.518 INFO:tasks.workunit.client.0.smithi057.stdout: create 129/30 file after the snapshot 2023-10-30T14:16:14.523 INFO:tasks.workunit.client.0.smithi057.stdout: create 129/31 file after the snapshot 2023-10-30T14:16:14.523 INFO:tasks.workunit.client.0.smithi057.stdout: create 129/32 file after the snapshot 2023-10-30T14:16:14.523 INFO:tasks.workunit.client.0.smithi057.stdout: create 129/33 file after the snapshot 2023-10-30T14:16:14.523 INFO:tasks.workunit.client.0.smithi057.stdout: create 129/34 file after the snapshot 2023-10-30T14:16:14.524 INFO:tasks.workunit.client.0.smithi057.stdout: create 129/35 file after the snapshot 2023-10-30T14:16:14.524 INFO:tasks.workunit.client.0.smithi057.stdout: create 129/36 file after the snapshot 2023-10-30T14:16:14.524 INFO:tasks.workunit.client.0.smithi057.stdout: create 129/37 file after the snapshot 2023-10-30T14:16:14.524 INFO:tasks.workunit.client.0.smithi057.stdout: create 129/38 file after the snapshot 2023-10-30T14:16:14.524 INFO:tasks.workunit.client.0.smithi057.stdout: create 129/39 file after the snapshot 2023-10-30T14:16:14.524 INFO:tasks.workunit.client.0.smithi057.stdout: create 130/.snap/snap-subdir-test 2023-10-30T14:16:14.590 INFO:tasks.workunit.client.0.smithi057.stdout: create 130/30 file after the snapshot 2023-10-30T14:16:14.595 INFO:tasks.workunit.client.0.smithi057.stdout: create 130/31 file after the snapshot 2023-10-30T14:16:14.595 INFO:tasks.workunit.client.0.smithi057.stdout: create 130/32 file after the snapshot 2023-10-30T14:16:14.596 INFO:tasks.workunit.client.0.smithi057.stdout: create 130/33 file after the snapshot 2023-10-30T14:16:14.596 INFO:tasks.workunit.client.0.smithi057.stdout: create 130/34 file after the snapshot 2023-10-30T14:16:14.596 INFO:tasks.workunit.client.0.smithi057.stdout: create 130/35 file after the snapshot 2023-10-30T14:16:14.596 INFO:tasks.workunit.client.0.smithi057.stdout: create 130/36 file after the snapshot 2023-10-30T14:16:14.596 INFO:tasks.workunit.client.0.smithi057.stdout: create 130/37 file after the snapshot 2023-10-30T14:16:14.597 INFO:tasks.workunit.client.0.smithi057.stdout: create 130/38 file after the snapshot 2023-10-30T14:16:14.597 INFO:tasks.workunit.client.0.smithi057.stdout: create 130/39 file after the snapshot 2023-10-30T14:16:14.597 INFO:tasks.workunit.client.0.smithi057.stdout: create 131/.snap/snap-subdir-test 2023-10-30T14:16:14.664 INFO:tasks.workunit.client.0.smithi057.stdout: create 131/30 file after the snapshot 2023-10-30T14:16:14.670 INFO:tasks.workunit.client.0.smithi057.stdout: create 131/31 file after the snapshot 2023-10-30T14:16:14.670 INFO:tasks.workunit.client.0.smithi057.stdout: create 131/32 file after the snapshot 2023-10-30T14:16:14.670 INFO:tasks.workunit.client.0.smithi057.stdout: create 131/33 file after the snapshot 2023-10-30T14:16:14.670 INFO:tasks.workunit.client.0.smithi057.stdout: create 131/34 file after the snapshot 2023-10-30T14:16:14.671 INFO:tasks.workunit.client.0.smithi057.stdout: create 131/35 file after the snapshot 2023-10-30T14:16:14.671 INFO:tasks.workunit.client.0.smithi057.stdout: create 131/36 file after the snapshot 2023-10-30T14:16:14.671 INFO:tasks.workunit.client.0.smithi057.stdout: create 131/37 file after the snapshot 2023-10-30T14:16:14.671 INFO:tasks.workunit.client.0.smithi057.stdout: create 131/38 file after the snapshot 2023-10-30T14:16:14.671 INFO:tasks.workunit.client.0.smithi057.stdout: create 131/39 file after the snapshot 2023-10-30T14:16:14.672 INFO:tasks.workunit.client.0.smithi057.stdout: create 132/.snap/snap-subdir-test 2023-10-30T14:16:14.737 INFO:tasks.workunit.client.0.smithi057.stdout: create 132/30 file after the snapshot 2023-10-30T14:16:14.741 INFO:tasks.workunit.client.0.smithi057.stdout: create 132/31 file after the snapshot 2023-10-30T14:16:14.741 INFO:tasks.workunit.client.0.smithi057.stdout: create 132/32 file after the snapshot 2023-10-30T14:16:14.742 INFO:tasks.workunit.client.0.smithi057.stdout: create 132/33 file after the snapshot 2023-10-30T14:16:14.742 INFO:tasks.workunit.client.0.smithi057.stdout: create 132/34 file after the snapshot 2023-10-30T14:16:14.742 INFO:tasks.workunit.client.0.smithi057.stdout: create 132/35 file after the snapshot 2023-10-30T14:16:14.742 INFO:tasks.workunit.client.0.smithi057.stdout: create 132/36 file after the snapshot 2023-10-30T14:16:14.743 INFO:tasks.workunit.client.0.smithi057.stdout: create 132/37 file after the snapshot 2023-10-30T14:16:14.743 INFO:tasks.workunit.client.0.smithi057.stdout: create 132/38 file after the snapshot 2023-10-30T14:16:14.743 INFO:tasks.workunit.client.0.smithi057.stdout: create 132/39 file after the snapshot 2023-10-30T14:16:14.743 INFO:tasks.workunit.client.0.smithi057.stdout: create 133/.snap/snap-subdir-test 2023-10-30T14:16:15.617 INFO:tasks.workunit.client.0.smithi057.stdout: create 133/30 file after the snapshot 2023-10-30T14:16:15.622 INFO:tasks.workunit.client.0.smithi057.stdout: create 133/31 file after the snapshot 2023-10-30T14:16:15.623 INFO:tasks.workunit.client.0.smithi057.stdout: create 133/32 file after the snapshot 2023-10-30T14:16:15.623 INFO:tasks.workunit.client.0.smithi057.stdout: create 133/33 file after the snapshot 2023-10-30T14:16:15.623 INFO:tasks.workunit.client.0.smithi057.stdout: create 133/34 file after the snapshot 2023-10-30T14:16:15.623 INFO:tasks.workunit.client.0.smithi057.stdout: create 133/35 file after the snapshot 2023-10-30T14:16:15.623 INFO:tasks.workunit.client.0.smithi057.stdout: create 133/36 file after the snapshot 2023-10-30T14:16:15.623 INFO:tasks.workunit.client.0.smithi057.stdout: create 133/37 file after the snapshot 2023-10-30T14:16:15.624 INFO:tasks.workunit.client.0.smithi057.stdout: create 133/38 file after the snapshot 2023-10-30T14:16:15.624 INFO:tasks.workunit.client.0.smithi057.stdout: create 133/39 file after the snapshot 2023-10-30T14:16:15.624 INFO:tasks.workunit.client.0.smithi057.stdout: create 134/.snap/snap-subdir-test 2023-10-30T14:16:15.701 INFO:tasks.workunit.client.0.smithi057.stdout: create 134/30 file after the snapshot 2023-10-30T14:16:15.722 INFO:tasks.workunit.client.0.smithi057.stdout: create 134/31 file after the snapshot 2023-10-30T14:16:15.722 INFO:tasks.workunit.client.0.smithi057.stdout: create 134/32 file after the snapshot 2023-10-30T14:16:15.722 INFO:tasks.workunit.client.0.smithi057.stdout: create 134/33 file after the snapshot 2023-10-30T14:16:15.722 INFO:tasks.workunit.client.0.smithi057.stdout: create 134/34 file after the snapshot 2023-10-30T14:16:15.723 INFO:tasks.workunit.client.0.smithi057.stdout: create 134/35 file after the snapshot 2023-10-30T14:16:15.723 INFO:tasks.workunit.client.0.smithi057.stdout: create 134/36 file after the snapshot 2023-10-30T14:16:15.723 INFO:tasks.workunit.client.0.smithi057.stdout: create 134/37 file after the snapshot 2023-10-30T14:16:15.723 INFO:tasks.workunit.client.0.smithi057.stdout: create 134/38 file after the snapshot 2023-10-30T14:16:15.723 INFO:tasks.workunit.client.0.smithi057.stdout: create 134/39 file after the snapshot 2023-10-30T14:16:15.723 INFO:tasks.workunit.client.0.smithi057.stdout: create 135/.snap/snap-subdir-test 2023-10-30T14:16:15.794 INFO:tasks.workunit.client.0.smithi057.stdout: create 135/30 file after the snapshot 2023-10-30T14:16:15.798 INFO:tasks.workunit.client.0.smithi057.stdout: create 135/31 file after the snapshot 2023-10-30T14:16:15.798 INFO:tasks.workunit.client.0.smithi057.stdout: create 135/32 file after the snapshot 2023-10-30T14:16:15.799 INFO:tasks.workunit.client.0.smithi057.stdout: create 135/33 file after the snapshot 2023-10-30T14:16:15.799 INFO:tasks.workunit.client.0.smithi057.stdout: create 135/34 file after the snapshot 2023-10-30T14:16:15.799 INFO:tasks.workunit.client.0.smithi057.stdout: create 135/35 file after the snapshot 2023-10-30T14:16:15.799 INFO:tasks.workunit.client.0.smithi057.stdout: create 135/36 file after the snapshot 2023-10-30T14:16:15.799 INFO:tasks.workunit.client.0.smithi057.stdout: create 135/37 file after the snapshot 2023-10-30T14:16:15.800 INFO:tasks.workunit.client.0.smithi057.stdout: create 135/38 file after the snapshot 2023-10-30T14:16:15.800 INFO:tasks.workunit.client.0.smithi057.stdout: create 135/39 file after the snapshot 2023-10-30T14:16:15.800 INFO:tasks.workunit.client.0.smithi057.stdout: create 136/.snap/snap-subdir-test 2023-10-30T14:16:15.865 INFO:tasks.workunit.client.0.smithi057.stdout: create 136/30 file after the snapshot 2023-10-30T14:16:15.867 INFO:tasks.workunit.client.0.smithi057.stdout: create 136/31 file after the snapshot 2023-10-30T14:16:15.867 INFO:tasks.workunit.client.0.smithi057.stdout: create 136/32 file after the snapshot 2023-10-30T14:16:15.868 INFO:tasks.workunit.client.0.smithi057.stdout: create 136/33 file after the snapshot 2023-10-30T14:16:15.868 INFO:tasks.workunit.client.0.smithi057.stdout: create 136/34 file after the snapshot 2023-10-30T14:16:15.868 INFO:tasks.workunit.client.0.smithi057.stdout: create 136/35 file after the snapshot 2023-10-30T14:16:15.868 INFO:tasks.workunit.client.0.smithi057.stdout: create 136/36 file after the snapshot 2023-10-30T14:16:15.868 INFO:tasks.workunit.client.0.smithi057.stdout: create 136/37 file after the snapshot 2023-10-30T14:16:15.869 INFO:tasks.workunit.client.0.smithi057.stdout: create 136/38 file after the snapshot 2023-10-30T14:16:15.869 INFO:tasks.workunit.client.0.smithi057.stdout: create 136/39 file after the snapshot 2023-10-30T14:16:15.869 INFO:tasks.workunit.client.0.smithi057.stdout: create 137/.snap/snap-subdir-test 2023-10-30T14:16:15.918 INFO:tasks.workunit.client.0.smithi057.stdout: create 137/30 file after the snapshot 2023-10-30T14:16:15.921 INFO:tasks.workunit.client.0.smithi057.stdout: create 137/31 file after the snapshot 2023-10-30T14:16:15.922 INFO:tasks.workunit.client.0.smithi057.stdout: create 137/32 file after the snapshot 2023-10-30T14:16:15.922 INFO:tasks.workunit.client.0.smithi057.stdout: create 137/33 file after the snapshot 2023-10-30T14:16:15.922 INFO:tasks.workunit.client.0.smithi057.stdout: create 137/34 file after the snapshot 2023-10-30T14:16:15.922 INFO:tasks.workunit.client.0.smithi057.stdout: create 137/35 file after the snapshot 2023-10-30T14:16:15.922 INFO:tasks.workunit.client.0.smithi057.stdout: create 137/36 file after the snapshot 2023-10-30T14:16:15.923 INFO:tasks.workunit.client.0.smithi057.stdout: create 137/37 file after the snapshot 2023-10-30T14:16:15.923 INFO:tasks.workunit.client.0.smithi057.stdout: create 137/38 file after the snapshot 2023-10-30T14:16:15.923 INFO:tasks.workunit.client.0.smithi057.stdout: create 137/39 file after the snapshot 2023-10-30T14:16:15.923 INFO:tasks.workunit.client.0.smithi057.stdout: create 138/.snap/snap-subdir-test 2023-10-30T14:16:15.989 INFO:tasks.workunit.client.0.smithi057.stdout: create 138/30 file after the snapshot 2023-10-30T14:16:15.993 INFO:tasks.workunit.client.0.smithi057.stdout: create 138/31 file after the snapshot 2023-10-30T14:16:15.993 INFO:tasks.workunit.client.0.smithi057.stdout: create 138/32 file after the snapshot 2023-10-30T14:16:15.993 INFO:tasks.workunit.client.0.smithi057.stdout: create 138/33 file after the snapshot 2023-10-30T14:16:15.994 INFO:tasks.workunit.client.0.smithi057.stdout: create 138/34 file after the snapshot 2023-10-30T14:16:15.994 INFO:tasks.workunit.client.0.smithi057.stdout: create 138/35 file after the snapshot 2023-10-30T14:16:15.994 INFO:tasks.workunit.client.0.smithi057.stdout: create 138/36 file after the snapshot 2023-10-30T14:16:15.994 INFO:tasks.workunit.client.0.smithi057.stdout: create 138/37 file after the snapshot 2023-10-30T14:16:15.995 INFO:tasks.workunit.client.0.smithi057.stdout: create 138/38 file after the snapshot 2023-10-30T14:16:15.995 INFO:tasks.workunit.client.0.smithi057.stdout: create 138/39 file after the snapshot 2023-10-30T14:16:15.995 INFO:tasks.workunit.client.0.smithi057.stdout: create 139/.snap/snap-subdir-test 2023-10-30T14:16:16.047 INFO:tasks.workunit.client.0.smithi057.stdout: create 139/30 file after the snapshot 2023-10-30T14:16:16.050 INFO:tasks.workunit.client.0.smithi057.stdout: create 139/31 file after the snapshot 2023-10-30T14:16:16.050 INFO:tasks.workunit.client.0.smithi057.stdout: create 139/32 file after the snapshot 2023-10-30T14:16:16.050 INFO:tasks.workunit.client.0.smithi057.stdout: create 139/33 file after the snapshot 2023-10-30T14:16:16.050 INFO:tasks.workunit.client.0.smithi057.stdout: create 139/34 file after the snapshot 2023-10-30T14:16:16.051 INFO:tasks.workunit.client.0.smithi057.stdout: create 139/35 file after the snapshot 2023-10-30T14:16:16.051 INFO:tasks.workunit.client.0.smithi057.stdout: create 139/36 file after the snapshot 2023-10-30T14:16:16.051 INFO:tasks.workunit.client.0.smithi057.stdout: create 139/37 file after the snapshot 2023-10-30T14:16:16.051 INFO:tasks.workunit.client.0.smithi057.stdout: create 139/38 file after the snapshot 2023-10-30T14:16:16.051 INFO:tasks.workunit.client.0.smithi057.stdout: create 139/39 file after the snapshot 2023-10-30T14:16:16.052 INFO:tasks.workunit.client.0.smithi057.stdout: create 140/.snap/snap-subdir-test 2023-10-30T14:16:16.109 INFO:tasks.workunit.client.0.smithi057.stdout: create 140/30 file after the snapshot 2023-10-30T14:16:16.112 INFO:tasks.workunit.client.0.smithi057.stdout: create 140/31 file after the snapshot 2023-10-30T14:16:16.112 INFO:tasks.workunit.client.0.smithi057.stdout: create 140/32 file after the snapshot 2023-10-30T14:16:16.112 INFO:tasks.workunit.client.0.smithi057.stdout: create 140/33 file after the snapshot 2023-10-30T14:16:16.112 INFO:tasks.workunit.client.0.smithi057.stdout: create 140/34 file after the snapshot 2023-10-30T14:16:16.112 INFO:tasks.workunit.client.0.smithi057.stdout: create 140/35 file after the snapshot 2023-10-30T14:16:16.113 INFO:tasks.workunit.client.0.smithi057.stdout: create 140/36 file after the snapshot 2023-10-30T14:16:16.113 INFO:tasks.workunit.client.0.smithi057.stdout: create 140/37 file after the snapshot 2023-10-30T14:16:16.113 INFO:tasks.workunit.client.0.smithi057.stdout: create 140/38 file after the snapshot 2023-10-30T14:16:16.113 INFO:tasks.workunit.client.0.smithi057.stdout: create 140/39 file after the snapshot 2023-10-30T14:16:16.113 INFO:tasks.workunit.client.0.smithi057.stdout: create 141/.snap/snap-subdir-test 2023-10-30T14:16:16.170 INFO:tasks.workunit.client.0.smithi057.stdout: create 141/30 file after the snapshot 2023-10-30T14:16:16.173 INFO:tasks.workunit.client.0.smithi057.stdout: create 141/31 file after the snapshot 2023-10-30T14:16:16.173 INFO:tasks.workunit.client.0.smithi057.stdout: create 141/32 file after the snapshot 2023-10-30T14:16:16.173 INFO:tasks.workunit.client.0.smithi057.stdout: create 141/33 file after the snapshot 2023-10-30T14:16:16.173 INFO:tasks.workunit.client.0.smithi057.stdout: create 141/34 file after the snapshot 2023-10-30T14:16:16.173 INFO:tasks.workunit.client.0.smithi057.stdout: create 141/35 file after the snapshot 2023-10-30T14:16:16.174 INFO:tasks.workunit.client.0.smithi057.stdout: create 141/36 file after the snapshot 2023-10-30T14:16:16.174 INFO:tasks.workunit.client.0.smithi057.stdout: create 141/37 file after the snapshot 2023-10-30T14:16:16.174 INFO:tasks.workunit.client.0.smithi057.stdout: create 141/38 file after the snapshot 2023-10-30T14:16:16.174 INFO:tasks.workunit.client.0.smithi057.stdout: create 141/39 file after the snapshot 2023-10-30T14:16:16.174 INFO:tasks.workunit.client.0.smithi057.stdout: create 142/.snap/snap-subdir-test 2023-10-30T14:16:16.680 INFO:tasks.workunit.client.0.smithi057.stdout: create 142/30 file after the snapshot 2023-10-30T14:16:16.685 INFO:tasks.workunit.client.0.smithi057.stdout: create 142/31 file after the snapshot 2023-10-30T14:16:16.686 INFO:tasks.workunit.client.0.smithi057.stdout: create 142/32 file after the snapshot 2023-10-30T14:16:16.686 INFO:tasks.workunit.client.0.smithi057.stdout: create 142/33 file after the snapshot 2023-10-30T14:16:16.686 INFO:tasks.workunit.client.0.smithi057.stdout: create 142/34 file after the snapshot 2023-10-30T14:16:16.686 INFO:tasks.workunit.client.0.smithi057.stdout: create 142/35 file after the snapshot 2023-10-30T14:16:16.687 INFO:tasks.workunit.client.0.smithi057.stdout: create 142/36 file after the snapshot 2023-10-30T14:16:16.687 INFO:tasks.workunit.client.0.smithi057.stdout: create 142/37 file after the snapshot 2023-10-30T14:16:16.687 INFO:tasks.workunit.client.0.smithi057.stdout: create 142/38 file after the snapshot 2023-10-30T14:16:16.687 INFO:tasks.workunit.client.0.smithi057.stdout: create 142/39 file after the snapshot 2023-10-30T14:16:16.687 INFO:tasks.workunit.client.0.smithi057.stdout: create 143/.snap/snap-subdir-test 2023-10-30T14:16:17.570 INFO:tasks.workunit.client.0.smithi057.stdout: create 143/30 file after the snapshot 2023-10-30T14:16:17.572 INFO:tasks.workunit.client.0.smithi057.stdout: create 143/31 file after the snapshot 2023-10-30T14:16:17.572 INFO:tasks.workunit.client.0.smithi057.stdout: create 143/32 file after the snapshot 2023-10-30T14:16:17.573 INFO:tasks.workunit.client.0.smithi057.stdout: create 143/33 file after the snapshot 2023-10-30T14:16:17.573 INFO:tasks.workunit.client.0.smithi057.stdout: create 143/34 file after the snapshot 2023-10-30T14:16:17.573 INFO:tasks.workunit.client.0.smithi057.stdout: create 143/35 file after the snapshot 2023-10-30T14:16:17.573 INFO:tasks.workunit.client.0.smithi057.stdout: create 143/36 file after the snapshot 2023-10-30T14:16:17.573 INFO:tasks.workunit.client.0.smithi057.stdout: create 143/37 file after the snapshot 2023-10-30T14:16:17.574 INFO:tasks.workunit.client.0.smithi057.stdout: create 143/38 file after the snapshot 2023-10-30T14:16:17.574 INFO:tasks.workunit.client.0.smithi057.stdout: create 143/39 file after the snapshot 2023-10-30T14:16:17.574 INFO:tasks.workunit.client.0.smithi057.stdout: create 144/.snap/snap-subdir-test 2023-10-30T14:16:17.652 INFO:tasks.workunit.client.0.smithi057.stdout: create 144/30 file after the snapshot 2023-10-30T14:16:17.657 INFO:tasks.workunit.client.0.smithi057.stdout: create 144/31 file after the snapshot 2023-10-30T14:16:17.657 INFO:tasks.workunit.client.0.smithi057.stdout: create 144/32 file after the snapshot 2023-10-30T14:16:17.657 INFO:tasks.workunit.client.0.smithi057.stdout: create 144/33 file after the snapshot 2023-10-30T14:16:17.657 INFO:tasks.workunit.client.0.smithi057.stdout: create 144/34 file after the snapshot 2023-10-30T14:16:17.657 INFO:tasks.workunit.client.0.smithi057.stdout: create 144/35 file after the snapshot 2023-10-30T14:16:17.658 INFO:tasks.workunit.client.0.smithi057.stdout: create 144/36 file after the snapshot 2023-10-30T14:16:17.658 INFO:tasks.workunit.client.0.smithi057.stdout: create 144/37 file after the snapshot 2023-10-30T14:16:17.658 INFO:tasks.workunit.client.0.smithi057.stdout: create 144/38 file after the snapshot 2023-10-30T14:16:17.658 INFO:tasks.workunit.client.0.smithi057.stdout: create 144/39 file after the snapshot 2023-10-30T14:16:17.659 INFO:tasks.workunit.client.0.smithi057.stdout: create 145/.snap/snap-subdir-test 2023-10-30T14:16:17.723 INFO:tasks.workunit.client.0.smithi057.stdout: create 145/30 file after the snapshot 2023-10-30T14:16:17.727 INFO:tasks.workunit.client.0.smithi057.stdout: create 145/31 file after the snapshot 2023-10-30T14:16:17.727 INFO:tasks.workunit.client.0.smithi057.stdout: create 145/32 file after the snapshot 2023-10-30T14:16:17.727 INFO:tasks.workunit.client.0.smithi057.stdout: create 145/33 file after the snapshot 2023-10-30T14:16:17.728 INFO:tasks.workunit.client.0.smithi057.stdout: create 145/34 file after the snapshot 2023-10-30T14:16:17.728 INFO:tasks.workunit.client.0.smithi057.stdout: create 145/35 file after the snapshot 2023-10-30T14:16:17.728 INFO:tasks.workunit.client.0.smithi057.stdout: create 145/36 file after the snapshot 2023-10-30T14:16:17.728 INFO:tasks.workunit.client.0.smithi057.stdout: create 145/37 file after the snapshot 2023-10-30T14:16:17.729 INFO:tasks.workunit.client.0.smithi057.stdout: create 145/38 file after the snapshot 2023-10-30T14:16:17.729 INFO:tasks.workunit.client.0.smithi057.stdout: create 145/39 file after the snapshot 2023-10-30T14:16:17.729 INFO:tasks.workunit.client.0.smithi057.stdout: create 146/.snap/snap-subdir-test 2023-10-30T14:16:17.798 INFO:tasks.workunit.client.0.smithi057.stdout: create 146/30 file after the snapshot 2023-10-30T14:16:17.804 INFO:tasks.workunit.client.0.smithi057.stdout: create 146/31 file after the snapshot 2023-10-30T14:16:17.804 INFO:tasks.workunit.client.0.smithi057.stdout: create 146/32 file after the snapshot 2023-10-30T14:16:17.804 INFO:tasks.workunit.client.0.smithi057.stdout: create 146/33 file after the snapshot 2023-10-30T14:16:17.804 INFO:tasks.workunit.client.0.smithi057.stdout: create 146/34 file after the snapshot 2023-10-30T14:16:17.804 INFO:tasks.workunit.client.0.smithi057.stdout: create 146/35 file after the snapshot 2023-10-30T14:16:17.805 INFO:tasks.workunit.client.0.smithi057.stdout: create 146/36 file after the snapshot 2023-10-30T14:16:17.805 INFO:tasks.workunit.client.0.smithi057.stdout: create 146/37 file after the snapshot 2023-10-30T14:16:17.805 INFO:tasks.workunit.client.0.smithi057.stdout: create 146/38 file after the snapshot 2023-10-30T14:16:17.805 INFO:tasks.workunit.client.0.smithi057.stdout: create 146/39 file after the snapshot 2023-10-30T14:16:17.805 INFO:tasks.workunit.client.0.smithi057.stdout: create 147/.snap/snap-subdir-test 2023-10-30T14:16:17.872 INFO:tasks.workunit.client.0.smithi057.stdout: create 147/30 file after the snapshot 2023-10-30T14:16:17.879 INFO:tasks.workunit.client.0.smithi057.stdout: create 147/31 file after the snapshot 2023-10-30T14:16:17.879 INFO:tasks.workunit.client.0.smithi057.stdout: create 147/32 file after the snapshot 2023-10-30T14:16:17.879 INFO:tasks.workunit.client.0.smithi057.stdout: create 147/33 file after the snapshot 2023-10-30T14:16:17.879 INFO:tasks.workunit.client.0.smithi057.stdout: create 147/34 file after the snapshot 2023-10-30T14:16:17.880 INFO:tasks.workunit.client.0.smithi057.stdout: create 147/35 file after the snapshot 2023-10-30T14:16:17.880 INFO:tasks.workunit.client.0.smithi057.stdout: create 147/36 file after the snapshot 2023-10-30T14:16:17.880 INFO:tasks.workunit.client.0.smithi057.stdout: create 147/37 file after the snapshot 2023-10-30T14:16:17.880 INFO:tasks.workunit.client.0.smithi057.stdout: create 147/38 file after the snapshot 2023-10-30T14:16:17.880 INFO:tasks.workunit.client.0.smithi057.stdout: create 147/39 file after the snapshot 2023-10-30T14:16:17.881 INFO:tasks.workunit.client.0.smithi057.stdout: create 148/.snap/snap-subdir-test 2023-10-30T14:16:18.875 INFO:tasks.workunit.client.0.smithi057.stdout: create 148/30 file after the snapshot 2023-10-30T14:16:18.880 INFO:tasks.workunit.client.0.smithi057.stdout: create 148/31 file after the snapshot 2023-10-30T14:16:18.880 INFO:tasks.workunit.client.0.smithi057.stdout: create 148/32 file after the snapshot 2023-10-30T14:16:18.880 INFO:tasks.workunit.client.0.smithi057.stdout: create 148/33 file after the snapshot 2023-10-30T14:16:18.880 INFO:tasks.workunit.client.0.smithi057.stdout: create 148/34 file after the snapshot 2023-10-30T14:16:18.881 INFO:tasks.workunit.client.0.smithi057.stdout: create 148/35 file after the snapshot 2023-10-30T14:16:18.881 INFO:tasks.workunit.client.0.smithi057.stdout: create 148/36 file after the snapshot 2023-10-30T14:16:18.881 INFO:tasks.workunit.client.0.smithi057.stdout: create 148/37 file after the snapshot 2023-10-30T14:16:18.881 INFO:tasks.workunit.client.0.smithi057.stdout: create 148/38 file after the snapshot 2023-10-30T14:16:18.882 INFO:tasks.workunit.client.0.smithi057.stdout: create 148/39 file after the snapshot 2023-10-30T14:16:18.882 INFO:tasks.workunit.client.0.smithi057.stdout: create 149/.snap/snap-subdir-test 2023-10-30T14:16:18.945 INFO:tasks.workunit.client.0.smithi057.stdout: create 149/30 file after the snapshot 2023-10-30T14:16:18.950 INFO:tasks.workunit.client.0.smithi057.stdout: create 149/31 file after the snapshot 2023-10-30T14:16:18.951 INFO:tasks.workunit.client.0.smithi057.stdout: create 149/32 file after the snapshot 2023-10-30T14:16:18.951 INFO:tasks.workunit.client.0.smithi057.stdout: create 149/33 file after the snapshot 2023-10-30T14:16:18.951 INFO:tasks.workunit.client.0.smithi057.stdout: create 149/34 file after the snapshot 2023-10-30T14:16:18.951 INFO:tasks.workunit.client.0.smithi057.stdout: create 149/35 file after the snapshot 2023-10-30T14:16:18.952 INFO:tasks.workunit.client.0.smithi057.stdout: create 149/36 file after the snapshot 2023-10-30T14:16:18.952 INFO:tasks.workunit.client.0.smithi057.stdout: create 149/37 file after the snapshot 2023-10-30T14:16:18.952 INFO:tasks.workunit.client.0.smithi057.stdout: create 149/38 file after the snapshot 2023-10-30T14:16:18.952 INFO:tasks.workunit.client.0.smithi057.stdout: create 149/39 file after the snapshot 2023-10-30T14:16:18.952 INFO:tasks.workunit.client.0.smithi057.stdout: create 150/.snap/snap-subdir-test 2023-10-30T14:16:19.015 INFO:tasks.workunit.client.0.smithi057.stdout: create 150/30 file after the snapshot 2023-10-30T14:16:19.018 INFO:tasks.workunit.client.0.smithi057.stdout: create 150/31 file after the snapshot 2023-10-30T14:16:19.018 INFO:tasks.workunit.client.0.smithi057.stdout: create 150/32 file after the snapshot 2023-10-30T14:16:19.018 INFO:tasks.workunit.client.0.smithi057.stdout: create 150/33 file after the snapshot 2023-10-30T14:16:19.019 INFO:tasks.workunit.client.0.smithi057.stdout: create 150/34 file after the snapshot 2023-10-30T14:16:19.019 INFO:tasks.workunit.client.0.smithi057.stdout: create 150/35 file after the snapshot 2023-10-30T14:16:19.019 INFO:tasks.workunit.client.0.smithi057.stdout: create 150/36 file after the snapshot 2023-10-30T14:16:19.021 INFO:tasks.workunit.client.0.smithi057.stdout: create 150/37 file after the snapshot 2023-10-30T14:16:19.021 INFO:tasks.workunit.client.0.smithi057.stdout: create 150/38 file after the snapshot 2023-10-30T14:16:19.021 INFO:tasks.workunit.client.0.smithi057.stdout: create 150/39 file after the snapshot 2023-10-30T14:16:19.021 INFO:tasks.workunit.client.0.smithi057.stdout: create 151/.snap/snap-subdir-test 2023-10-30T14:16:19.084 INFO:tasks.workunit.client.0.smithi057.stdout: create 151/30 file after the snapshot 2023-10-30T14:16:19.087 INFO:tasks.workunit.client.0.smithi057.stdout: create 151/31 file after the snapshot 2023-10-30T14:16:19.087 INFO:tasks.workunit.client.0.smithi057.stdout: create 151/32 file after the snapshot 2023-10-30T14:16:19.087 INFO:tasks.workunit.client.0.smithi057.stdout: create 151/33 file after the snapshot 2023-10-30T14:16:19.087 INFO:tasks.workunit.client.0.smithi057.stdout: create 151/34 file after the snapshot 2023-10-30T14:16:19.088 INFO:tasks.workunit.client.0.smithi057.stdout: create 151/35 file after the snapshot 2023-10-30T14:16:19.088 INFO:tasks.workunit.client.0.smithi057.stdout: create 151/36 file after the snapshot 2023-10-30T14:16:19.088 INFO:tasks.workunit.client.0.smithi057.stdout: create 151/37 file after the snapshot 2023-10-30T14:16:19.088 INFO:tasks.workunit.client.0.smithi057.stdout: create 151/38 file after the snapshot 2023-10-30T14:16:19.088 INFO:tasks.workunit.client.0.smithi057.stdout: create 151/39 file after the snapshot 2023-10-30T14:16:19.088 INFO:tasks.workunit.client.0.smithi057.stdout: create 152/.snap/snap-subdir-test 2023-10-30T14:16:19.153 INFO:tasks.workunit.client.0.smithi057.stdout: create 152/30 file after the snapshot 2023-10-30T14:16:19.158 INFO:tasks.workunit.client.0.smithi057.stdout: create 152/31 file after the snapshot 2023-10-30T14:16:19.158 INFO:tasks.workunit.client.0.smithi057.stdout: create 152/32 file after the snapshot 2023-10-30T14:16:19.158 INFO:tasks.workunit.client.0.smithi057.stdout: create 152/33 file after the snapshot 2023-10-30T14:16:19.159 INFO:tasks.workunit.client.0.smithi057.stdout: create 152/34 file after the snapshot 2023-10-30T14:16:19.159 INFO:tasks.workunit.client.0.smithi057.stdout: create 152/35 file after the snapshot 2023-10-30T14:16:19.159 INFO:tasks.workunit.client.0.smithi057.stdout: create 152/36 file after the snapshot 2023-10-30T14:16:19.159 INFO:tasks.workunit.client.0.smithi057.stdout: create 152/37 file after the snapshot 2023-10-30T14:16:19.159 INFO:tasks.workunit.client.0.smithi057.stdout: create 152/38 file after the snapshot 2023-10-30T14:16:19.159 INFO:tasks.workunit.client.0.smithi057.stdout: create 152/39 file after the snapshot 2023-10-30T14:16:19.160 INFO:tasks.workunit.client.0.smithi057.stdout: create 153/.snap/snap-subdir-test 2023-10-30T14:16:19.222 INFO:tasks.workunit.client.0.smithi057.stdout: create 153/30 file after the snapshot 2023-10-30T14:16:19.225 INFO:tasks.workunit.client.0.smithi057.stdout: create 153/31 file after the snapshot 2023-10-30T14:16:19.225 INFO:tasks.workunit.client.0.smithi057.stdout: create 153/32 file after the snapshot 2023-10-30T14:16:19.225 INFO:tasks.workunit.client.0.smithi057.stdout: create 153/33 file after the snapshot 2023-10-30T14:16:19.225 INFO:tasks.workunit.client.0.smithi057.stdout: create 153/34 file after the snapshot 2023-10-30T14:16:19.226 INFO:tasks.workunit.client.0.smithi057.stdout: create 153/35 file after the snapshot 2023-10-30T14:16:19.226 INFO:tasks.workunit.client.0.smithi057.stdout: create 153/36 file after the snapshot 2023-10-30T14:16:19.226 INFO:tasks.workunit.client.0.smithi057.stdout: create 153/37 file after the snapshot 2023-10-30T14:16:19.226 INFO:tasks.workunit.client.0.smithi057.stdout: create 153/38 file after the snapshot 2023-10-30T14:16:19.226 INFO:tasks.workunit.client.0.smithi057.stdout: create 153/39 file after the snapshot 2023-10-30T14:16:19.227 INFO:tasks.workunit.client.0.smithi057.stdout: create 154/.snap/snap-subdir-test 2023-10-30T14:16:19.608 INFO:tasks.workunit.client.0.smithi057.stdout: create 154/30 file after the snapshot 2023-10-30T14:16:19.611 INFO:tasks.workunit.client.0.smithi057.stdout: create 154/31 file after the snapshot 2023-10-30T14:16:19.611 INFO:tasks.workunit.client.0.smithi057.stdout: create 154/32 file after the snapshot 2023-10-30T14:16:19.611 INFO:tasks.workunit.client.0.smithi057.stdout: create 154/33 file after the snapshot 2023-10-30T14:16:19.611 INFO:tasks.workunit.client.0.smithi057.stdout: create 154/34 file after the snapshot 2023-10-30T14:16:19.611 INFO:tasks.workunit.client.0.smithi057.stdout: create 154/35 file after the snapshot 2023-10-30T14:16:19.612 INFO:tasks.workunit.client.0.smithi057.stdout: create 154/36 file after the snapshot 2023-10-30T14:16:19.612 INFO:tasks.workunit.client.0.smithi057.stdout: create 154/37 file after the snapshot 2023-10-30T14:16:19.612 INFO:tasks.workunit.client.0.smithi057.stdout: create 154/38 file after the snapshot 2023-10-30T14:16:19.612 INFO:tasks.workunit.client.0.smithi057.stdout: create 154/39 file after the snapshot 2023-10-30T14:16:19.612 INFO:tasks.workunit.client.0.smithi057.stdout: create 155/.snap/snap-subdir-test 2023-10-30T14:16:19.676 INFO:tasks.workunit.client.0.smithi057.stdout: create 155/30 file after the snapshot 2023-10-30T14:16:19.678 INFO:tasks.workunit.client.0.smithi057.stdout: create 155/31 file after the snapshot 2023-10-30T14:16:19.679 INFO:tasks.workunit.client.0.smithi057.stdout: create 155/32 file after the snapshot 2023-10-30T14:16:19.679 INFO:tasks.workunit.client.0.smithi057.stdout: create 155/33 file after the snapshot 2023-10-30T14:16:19.679 INFO:tasks.workunit.client.0.smithi057.stdout: create 155/34 file after the snapshot 2023-10-30T14:16:19.679 INFO:tasks.workunit.client.0.smithi057.stdout: create 155/35 file after the snapshot 2023-10-30T14:16:19.680 INFO:tasks.workunit.client.0.smithi057.stdout: create 155/36 file after the snapshot 2023-10-30T14:16:19.680 INFO:tasks.workunit.client.0.smithi057.stdout: create 155/37 file after the snapshot 2023-10-30T14:16:19.680 INFO:tasks.workunit.client.0.smithi057.stdout: create 155/38 file after the snapshot 2023-10-30T14:16:19.680 INFO:tasks.workunit.client.0.smithi057.stdout: create 155/39 file after the snapshot 2023-10-30T14:16:19.680 INFO:tasks.workunit.client.0.smithi057.stdout: create 156/.snap/snap-subdir-test 2023-10-30T14:16:19.746 INFO:tasks.workunit.client.0.smithi057.stdout: create 156/30 file after the snapshot 2023-10-30T14:16:19.750 INFO:tasks.workunit.client.0.smithi057.stdout: create 156/31 file after the snapshot 2023-10-30T14:16:19.751 INFO:tasks.workunit.client.0.smithi057.stdout: create 156/32 file after the snapshot 2023-10-30T14:16:19.751 INFO:tasks.workunit.client.0.smithi057.stdout: create 156/33 file after the snapshot 2023-10-30T14:16:19.751 INFO:tasks.workunit.client.0.smithi057.stdout: create 156/34 file after the snapshot 2023-10-30T14:16:19.751 INFO:tasks.workunit.client.0.smithi057.stdout: create 156/35 file after the snapshot 2023-10-30T14:16:19.751 INFO:tasks.workunit.client.0.smithi057.stdout: create 156/36 file after the snapshot 2023-10-30T14:16:19.752 INFO:tasks.workunit.client.0.smithi057.stdout: create 156/37 file after the snapshot 2023-10-30T14:16:19.752 INFO:tasks.workunit.client.0.smithi057.stdout: create 156/38 file after the snapshot 2023-10-30T14:16:19.752 INFO:tasks.workunit.client.0.smithi057.stdout: create 156/39 file after the snapshot 2023-10-30T14:16:19.752 INFO:tasks.workunit.client.0.smithi057.stdout: create 157/.snap/snap-subdir-test 2023-10-30T14:16:20.697 INFO:tasks.workunit.client.0.smithi057.stdout: create 157/30 file after the snapshot 2023-10-30T14:16:20.700 INFO:tasks.workunit.client.0.smithi057.stdout: create 157/31 file after the snapshot 2023-10-30T14:16:20.700 INFO:tasks.workunit.client.0.smithi057.stdout: create 157/32 file after the snapshot 2023-10-30T14:16:20.700 INFO:tasks.workunit.client.0.smithi057.stdout: create 157/33 file after the snapshot 2023-10-30T14:16:20.700 INFO:tasks.workunit.client.0.smithi057.stdout: create 157/34 file after the snapshot 2023-10-30T14:16:20.701 INFO:tasks.workunit.client.0.smithi057.stdout: create 157/35 file after the snapshot 2023-10-30T14:16:20.701 INFO:tasks.workunit.client.0.smithi057.stdout: create 157/36 file after the snapshot 2023-10-30T14:16:20.701 INFO:tasks.workunit.client.0.smithi057.stdout: create 157/37 file after the snapshot 2023-10-30T14:16:20.701 INFO:tasks.workunit.client.0.smithi057.stdout: create 157/38 file after the snapshot 2023-10-30T14:16:20.702 INFO:tasks.workunit.client.0.smithi057.stdout: create 157/39 file after the snapshot 2023-10-30T14:16:20.702 INFO:tasks.workunit.client.0.smithi057.stdout: create 158/.snap/snap-subdir-test 2023-10-30T14:16:21.978 INFO:tasks.workunit.client.0.smithi057.stdout: create 158/30 file after the snapshot 2023-10-30T14:16:21.993 INFO:tasks.workunit.client.0.smithi057.stdout: create 158/31 file after the snapshot 2023-10-30T14:16:21.993 INFO:tasks.workunit.client.0.smithi057.stdout: create 158/32 file after the snapshot 2023-10-30T14:16:21.993 INFO:tasks.workunit.client.0.smithi057.stdout: create 158/33 file after the snapshot 2023-10-30T14:16:21.994 INFO:tasks.workunit.client.0.smithi057.stdout: create 158/34 file after the snapshot 2023-10-30T14:16:21.995 INFO:tasks.workunit.client.0.smithi057.stdout: create 158/35 file after the snapshot 2023-10-30T14:16:21.995 INFO:tasks.workunit.client.0.smithi057.stdout: create 158/36 file after the snapshot 2023-10-30T14:16:21.995 INFO:tasks.workunit.client.0.smithi057.stdout: create 158/37 file after the snapshot 2023-10-30T14:16:21.995 INFO:tasks.workunit.client.0.smithi057.stdout: create 158/38 file after the snapshot 2023-10-30T14:16:21.995 INFO:tasks.workunit.client.0.smithi057.stdout: create 158/39 file after the snapshot 2023-10-30T14:16:21.996 INFO:tasks.workunit.client.0.smithi057.stdout: create 159/.snap/snap-subdir-test 2023-10-30T14:16:22.758 INFO:tasks.workunit.client.0.smithi057.stdout: create 159/30 file after the snapshot 2023-10-30T14:16:22.774 INFO:tasks.workunit.client.0.smithi057.stdout: create 159/31 file after the snapshot 2023-10-30T14:16:22.775 INFO:tasks.workunit.client.0.smithi057.stdout: create 159/32 file after the snapshot 2023-10-30T14:16:22.776 INFO:tasks.workunit.client.0.smithi057.stdout: create 159/33 file after the snapshot 2023-10-30T14:16:22.777 INFO:tasks.workunit.client.0.smithi057.stdout: create 159/34 file after the snapshot 2023-10-30T14:16:22.778 INFO:tasks.workunit.client.0.smithi057.stdout: create 159/35 file after the snapshot 2023-10-30T14:16:22.779 INFO:tasks.workunit.client.0.smithi057.stdout: create 159/36 file after the snapshot 2023-10-30T14:16:22.780 INFO:tasks.workunit.client.0.smithi057.stdout: create 159/37 file after the snapshot 2023-10-30T14:16:22.780 INFO:tasks.workunit.client.0.smithi057.stdout: create 159/38 file after the snapshot 2023-10-30T14:16:22.781 INFO:tasks.workunit.client.0.smithi057.stdout: create 159/39 file after the snapshot 2023-10-30T14:16:22.782 INFO:tasks.workunit.client.0.smithi057.stdout: create 160/.snap/snap-subdir-test 2023-10-30T14:16:22.831 INFO:tasks.workunit.client.0.smithi057.stdout: create 160/30 file after the snapshot 2023-10-30T14:16:22.835 INFO:tasks.workunit.client.0.smithi057.stdout: create 160/31 file after the snapshot 2023-10-30T14:16:22.835 INFO:tasks.workunit.client.0.smithi057.stdout: create 160/32 file after the snapshot 2023-10-30T14:16:22.836 INFO:tasks.workunit.client.0.smithi057.stdout: create 160/33 file after the snapshot 2023-10-30T14:16:22.836 INFO:tasks.workunit.client.0.smithi057.stdout: create 160/34 file after the snapshot 2023-10-30T14:16:22.836 INFO:tasks.workunit.client.0.smithi057.stdout: create 160/35 file after the snapshot 2023-10-30T14:16:22.836 INFO:tasks.workunit.client.0.smithi057.stdout: create 160/36 file after the snapshot 2023-10-30T14:16:22.837 INFO:tasks.workunit.client.0.smithi057.stdout: create 160/37 file after the snapshot 2023-10-30T14:16:22.837 INFO:tasks.workunit.client.0.smithi057.stdout: create 160/38 file after the snapshot 2023-10-30T14:16:22.837 INFO:tasks.workunit.client.0.smithi057.stdout: create 160/39 file after the snapshot 2023-10-30T14:16:22.837 INFO:tasks.workunit.client.0.smithi057.stdout: create 161/.snap/snap-subdir-test 2023-10-30T14:16:22.885 INFO:tasks.workunit.client.0.smithi057.stdout: create 161/30 file after the snapshot 2023-10-30T14:16:22.888 INFO:tasks.workunit.client.0.smithi057.stdout: create 161/31 file after the snapshot 2023-10-30T14:16:22.888 INFO:tasks.workunit.client.0.smithi057.stdout: create 161/32 file after the snapshot 2023-10-30T14:16:22.888 INFO:tasks.workunit.client.0.smithi057.stdout: create 161/33 file after the snapshot 2023-10-30T14:16:22.889 INFO:tasks.workunit.client.0.smithi057.stdout: create 161/34 file after the snapshot 2023-10-30T14:16:22.889 INFO:tasks.workunit.client.0.smithi057.stdout: create 161/35 file after the snapshot 2023-10-30T14:16:22.889 INFO:tasks.workunit.client.0.smithi057.stdout: create 161/36 file after the snapshot 2023-10-30T14:16:22.889 INFO:tasks.workunit.client.0.smithi057.stdout: create 161/37 file after the snapshot 2023-10-30T14:16:22.890 INFO:tasks.workunit.client.0.smithi057.stdout: create 161/38 file after the snapshot 2023-10-30T14:16:22.890 INFO:tasks.workunit.client.0.smithi057.stdout: create 161/39 file after the snapshot 2023-10-30T14:16:22.890 INFO:tasks.workunit.client.0.smithi057.stdout: create 162/.snap/snap-subdir-test 2023-10-30T14:16:22.947 INFO:tasks.workunit.client.0.smithi057.stdout: create 162/30 file after the snapshot 2023-10-30T14:16:22.950 INFO:tasks.workunit.client.0.smithi057.stdout: create 162/31 file after the snapshot 2023-10-30T14:16:22.950 INFO:tasks.workunit.client.0.smithi057.stdout: create 162/32 file after the snapshot 2023-10-30T14:16:22.950 INFO:tasks.workunit.client.0.smithi057.stdout: create 162/33 file after the snapshot 2023-10-30T14:16:22.951 INFO:tasks.workunit.client.0.smithi057.stdout: create 162/34 file after the snapshot 2023-10-30T14:16:22.951 INFO:tasks.workunit.client.0.smithi057.stdout: create 162/35 file after the snapshot 2023-10-30T14:16:22.951 INFO:tasks.workunit.client.0.smithi057.stdout: create 162/36 file after the snapshot 2023-10-30T14:16:22.951 INFO:tasks.workunit.client.0.smithi057.stdout: create 162/37 file after the snapshot 2023-10-30T14:16:22.951 INFO:tasks.workunit.client.0.smithi057.stdout: create 162/38 file after the snapshot 2023-10-30T14:16:22.952 INFO:tasks.workunit.client.0.smithi057.stdout: create 162/39 file after the snapshot 2023-10-30T14:16:22.952 INFO:tasks.workunit.client.0.smithi057.stdout: create 163/.snap/snap-subdir-test 2023-10-30T14:16:23.007 INFO:tasks.workunit.client.0.smithi057.stdout: create 163/30 file after the snapshot 2023-10-30T14:16:23.011 INFO:tasks.workunit.client.0.smithi057.stdout: create 163/31 file after the snapshot 2023-10-30T14:16:23.012 INFO:tasks.workunit.client.0.smithi057.stdout: create 163/32 file after the snapshot 2023-10-30T14:16:23.012 INFO:tasks.workunit.client.0.smithi057.stdout: create 163/33 file after the snapshot 2023-10-30T14:16:23.012 INFO:tasks.workunit.client.0.smithi057.stdout: create 163/34 file after the snapshot 2023-10-30T14:16:23.012 INFO:tasks.workunit.client.0.smithi057.stdout: create 163/35 file after the snapshot 2023-10-30T14:16:23.012 INFO:tasks.workunit.client.0.smithi057.stdout: create 163/36 file after the snapshot 2023-10-30T14:16:23.012 INFO:tasks.workunit.client.0.smithi057.stdout: create 163/37 file after the snapshot 2023-10-30T14:16:23.013 INFO:tasks.workunit.client.0.smithi057.stdout: create 163/38 file after the snapshot 2023-10-30T14:16:23.013 INFO:tasks.workunit.client.0.smithi057.stdout: create 163/39 file after the snapshot 2023-10-30T14:16:23.013 INFO:tasks.workunit.client.0.smithi057.stdout: create 164/.snap/snap-subdir-test 2023-10-30T14:16:23.048 INFO:tasks.workunit.client.0.smithi057.stdout: create 164/30 file after the snapshot 2023-10-30T14:16:23.050 INFO:tasks.workunit.client.0.smithi057.stdout: create 164/31 file after the snapshot 2023-10-30T14:16:23.050 INFO:tasks.workunit.client.0.smithi057.stdout: create 164/32 file after the snapshot 2023-10-30T14:16:23.050 INFO:tasks.workunit.client.0.smithi057.stdout: create 164/33 file after the snapshot 2023-10-30T14:16:23.051 INFO:tasks.workunit.client.0.smithi057.stdout: create 164/34 file after the snapshot 2023-10-30T14:16:23.051 INFO:tasks.workunit.client.0.smithi057.stdout: create 164/35 file after the snapshot 2023-10-30T14:16:23.051 INFO:tasks.workunit.client.0.smithi057.stdout: create 164/36 file after the snapshot 2023-10-30T14:16:23.051 INFO:tasks.workunit.client.0.smithi057.stdout: create 164/37 file after the snapshot 2023-10-30T14:16:23.051 INFO:tasks.workunit.client.0.smithi057.stdout: create 164/38 file after the snapshot 2023-10-30T14:16:23.051 INFO:tasks.workunit.client.0.smithi057.stdout: create 164/39 file after the snapshot 2023-10-30T14:16:23.052 INFO:tasks.workunit.client.0.smithi057.stdout: create 165/.snap/snap-subdir-test 2023-10-30T14:16:23.086 INFO:tasks.workunit.client.0.smithi057.stdout: create 165/30 file after the snapshot 2023-10-30T14:16:23.091 INFO:tasks.workunit.client.0.smithi057.stdout: create 165/31 file after the snapshot 2023-10-30T14:16:23.091 INFO:tasks.workunit.client.0.smithi057.stdout: create 165/32 file after the snapshot 2023-10-30T14:16:23.091 INFO:tasks.workunit.client.0.smithi057.stdout: create 165/33 file after the snapshot 2023-10-30T14:16:23.092 INFO:tasks.workunit.client.0.smithi057.stdout: create 165/34 file after the snapshot 2023-10-30T14:16:23.092 INFO:tasks.workunit.client.0.smithi057.stdout: create 165/35 file after the snapshot 2023-10-30T14:16:23.092 INFO:tasks.workunit.client.0.smithi057.stdout: create 165/36 file after the snapshot 2023-10-30T14:16:23.092 INFO:tasks.workunit.client.0.smithi057.stdout: create 165/37 file after the snapshot 2023-10-30T14:16:23.092 INFO:tasks.workunit.client.0.smithi057.stdout: create 165/38 file after the snapshot 2023-10-30T14:16:23.093 INFO:tasks.workunit.client.0.smithi057.stdout: create 165/39 file after the snapshot 2023-10-30T14:16:23.093 INFO:tasks.workunit.client.0.smithi057.stdout: create 166/.snap/snap-subdir-test 2023-10-30T14:16:23.148 INFO:tasks.workunit.client.0.smithi057.stdout: create 166/30 file after the snapshot 2023-10-30T14:16:23.151 INFO:tasks.workunit.client.0.smithi057.stdout: create 166/31 file after the snapshot 2023-10-30T14:16:23.152 INFO:tasks.workunit.client.0.smithi057.stdout: create 166/32 file after the snapshot 2023-10-30T14:16:23.152 INFO:tasks.workunit.client.0.smithi057.stdout: create 166/33 file after the snapshot 2023-10-30T14:16:23.152 INFO:tasks.workunit.client.0.smithi057.stdout: create 166/34 file after the snapshot 2023-10-30T14:16:23.152 INFO:tasks.workunit.client.0.smithi057.stdout: create 166/35 file after the snapshot 2023-10-30T14:16:23.152 INFO:tasks.workunit.client.0.smithi057.stdout: create 166/36 file after the snapshot 2023-10-30T14:16:23.153 INFO:tasks.workunit.client.0.smithi057.stdout: create 166/37 file after the snapshot 2023-10-30T14:16:23.153 INFO:tasks.workunit.client.0.smithi057.stdout: create 166/38 file after the snapshot 2023-10-30T14:16:23.153 INFO:tasks.workunit.client.0.smithi057.stdout: create 166/39 file after the snapshot 2023-10-30T14:16:23.153 INFO:tasks.workunit.client.0.smithi057.stdout: create 167/.snap/snap-subdir-test 2023-10-30T14:16:23.207 INFO:tasks.workunit.client.0.smithi057.stdout: create 167/30 file after the snapshot 2023-10-30T14:16:23.210 INFO:tasks.workunit.client.0.smithi057.stdout: create 167/31 file after the snapshot 2023-10-30T14:16:23.211 INFO:tasks.workunit.client.0.smithi057.stdout: create 167/32 file after the snapshot 2023-10-30T14:16:23.211 INFO:tasks.workunit.client.0.smithi057.stdout: create 167/33 file after the snapshot 2023-10-30T14:16:23.211 INFO:tasks.workunit.client.0.smithi057.stdout: create 167/34 file after the snapshot 2023-10-30T14:16:23.211 INFO:tasks.workunit.client.0.smithi057.stdout: create 167/35 file after the snapshot 2023-10-30T14:16:23.211 INFO:tasks.workunit.client.0.smithi057.stdout: create 167/36 file after the snapshot 2023-10-30T14:16:23.211 INFO:tasks.workunit.client.0.smithi057.stdout: create 167/37 file after the snapshot 2023-10-30T14:16:23.212 INFO:tasks.workunit.client.0.smithi057.stdout: create 167/38 file after the snapshot 2023-10-30T14:16:23.212 INFO:tasks.workunit.client.0.smithi057.stdout: create 167/39 file after the snapshot 2023-10-30T14:16:23.212 INFO:tasks.workunit.client.0.smithi057.stdout: create 168/.snap/snap-subdir-test 2023-10-30T14:16:23.279 INFO:tasks.workunit.client.0.smithi057.stdout: create 168/30 file after the snapshot 2023-10-30T14:16:23.284 INFO:tasks.workunit.client.0.smithi057.stdout: create 168/31 file after the snapshot 2023-10-30T14:16:23.284 INFO:tasks.workunit.client.0.smithi057.stdout: create 168/32 file after the snapshot 2023-10-30T14:16:23.284 INFO:tasks.workunit.client.0.smithi057.stdout: create 168/33 file after the snapshot 2023-10-30T14:16:23.285 INFO:tasks.workunit.client.0.smithi057.stdout: create 168/34 file after the snapshot 2023-10-30T14:16:23.285 INFO:tasks.workunit.client.0.smithi057.stdout: create 168/35 file after the snapshot 2023-10-30T14:16:23.285 INFO:tasks.workunit.client.0.smithi057.stdout: create 168/36 file after the snapshot 2023-10-30T14:16:23.285 INFO:tasks.workunit.client.0.smithi057.stdout: create 168/37 file after the snapshot 2023-10-30T14:16:23.286 INFO:tasks.workunit.client.0.smithi057.stdout: create 168/38 file after the snapshot 2023-10-30T14:16:23.286 INFO:tasks.workunit.client.0.smithi057.stdout: create 168/39 file after the snapshot 2023-10-30T14:16:23.286 INFO:tasks.workunit.client.0.smithi057.stdout: create 169/.snap/snap-subdir-test 2023-10-30T14:16:23.323 INFO:tasks.workunit.client.0.smithi057.stdout: create 169/30 file after the snapshot 2023-10-30T14:16:23.325 INFO:tasks.workunit.client.0.smithi057.stdout: create 169/31 file after the snapshot 2023-10-30T14:16:23.325 INFO:tasks.workunit.client.0.smithi057.stdout: create 169/32 file after the snapshot 2023-10-30T14:16:23.325 INFO:tasks.workunit.client.0.smithi057.stdout: create 169/33 file after the snapshot 2023-10-30T14:16:23.325 INFO:tasks.workunit.client.0.smithi057.stdout: create 169/34 file after the snapshot 2023-10-30T14:16:23.326 INFO:tasks.workunit.client.0.smithi057.stdout: create 169/35 file after the snapshot 2023-10-30T14:16:23.326 INFO:tasks.workunit.client.0.smithi057.stdout: create 169/36 file after the snapshot 2023-10-30T14:16:23.326 INFO:tasks.workunit.client.0.smithi057.stdout: create 169/37 file after the snapshot 2023-10-30T14:16:23.326 INFO:tasks.workunit.client.0.smithi057.stdout: create 169/38 file after the snapshot 2023-10-30T14:16:23.326 INFO:tasks.workunit.client.0.smithi057.stdout: create 169/39 file after the snapshot 2023-10-30T14:16:23.327 INFO:tasks.workunit.client.0.smithi057.stdout: create 170/.snap/snap-subdir-test 2023-10-30T14:16:23.741 INFO:tasks.workunit.client.0.smithi057.stdout: create 170/30 file after the snapshot 2023-10-30T14:16:23.747 INFO:tasks.workunit.client.0.smithi057.stdout: create 170/31 file after the snapshot 2023-10-30T14:16:23.747 INFO:tasks.workunit.client.0.smithi057.stdout: create 170/32 file after the snapshot 2023-10-30T14:16:23.747 INFO:tasks.workunit.client.0.smithi057.stdout: create 170/33 file after the snapshot 2023-10-30T14:16:23.748 INFO:tasks.workunit.client.0.smithi057.stdout: create 170/34 file after the snapshot 2023-10-30T14:16:23.748 INFO:tasks.workunit.client.0.smithi057.stdout: create 170/35 file after the snapshot 2023-10-30T14:16:23.748 INFO:tasks.workunit.client.0.smithi057.stdout: create 170/36 file after the snapshot 2023-10-30T14:16:23.748 INFO:tasks.workunit.client.0.smithi057.stdout: create 170/37 file after the snapshot 2023-10-30T14:16:23.749 INFO:tasks.workunit.client.0.smithi057.stdout: create 170/38 file after the snapshot 2023-10-30T14:16:23.749 INFO:tasks.workunit.client.0.smithi057.stdout: create 170/39 file after the snapshot 2023-10-30T14:16:23.749 INFO:tasks.workunit.client.0.smithi057.stdout: create 171/.snap/snap-subdir-test 2023-10-30T14:16:23.786 INFO:tasks.workunit.client.0.smithi057.stdout: create 171/30 file after the snapshot 2023-10-30T14:16:23.788 INFO:tasks.workunit.client.0.smithi057.stdout: create 171/31 file after the snapshot 2023-10-30T14:16:23.788 INFO:tasks.workunit.client.0.smithi057.stdout: create 171/32 file after the snapshot 2023-10-30T14:16:23.789 INFO:tasks.workunit.client.0.smithi057.stdout: create 171/33 file after the snapshot 2023-10-30T14:16:23.789 INFO:tasks.workunit.client.0.smithi057.stdout: create 171/34 file after the snapshot 2023-10-30T14:16:23.789 INFO:tasks.workunit.client.0.smithi057.stdout: create 171/35 file after the snapshot 2023-10-30T14:16:23.789 INFO:tasks.workunit.client.0.smithi057.stdout: create 171/36 file after the snapshot 2023-10-30T14:16:23.790 INFO:tasks.workunit.client.0.smithi057.stdout: create 171/37 file after the snapshot 2023-10-30T14:16:23.790 INFO:tasks.workunit.client.0.smithi057.stdout: create 171/38 file after the snapshot 2023-10-30T14:16:23.790 INFO:tasks.workunit.client.0.smithi057.stdout: create 171/39 file after the snapshot 2023-10-30T14:16:23.790 INFO:tasks.workunit.client.0.smithi057.stdout: create 172/.snap/snap-subdir-test 2023-10-30T14:16:24.079 INFO:tasks.workunit.client.0.smithi057.stdout: create 172/30 file after the snapshot 2023-10-30T14:16:24.080 INFO:tasks.workunit.client.0.smithi057.stdout: create 172/31 file after the snapshot 2023-10-30T14:16:24.080 INFO:tasks.workunit.client.0.smithi057.stdout: create 172/32 file after the snapshot 2023-10-30T14:16:24.081 INFO:tasks.workunit.client.0.smithi057.stdout: create 172/33 file after the snapshot 2023-10-30T14:16:24.081 INFO:tasks.workunit.client.0.smithi057.stdout: create 172/34 file after the snapshot 2023-10-30T14:16:24.081 INFO:tasks.workunit.client.0.smithi057.stdout: create 172/35 file after the snapshot 2023-10-30T14:16:24.082 INFO:tasks.workunit.client.0.smithi057.stdout: create 172/36 file after the snapshot 2023-10-30T14:16:24.082 INFO:tasks.workunit.client.0.smithi057.stdout: create 172/37 file after the snapshot 2023-10-30T14:16:24.082 INFO:tasks.workunit.client.0.smithi057.stdout: create 172/38 file after the snapshot 2023-10-30T14:16:24.082 INFO:tasks.workunit.client.0.smithi057.stdout: create 172/39 file after the snapshot 2023-10-30T14:16:24.083 INFO:tasks.workunit.client.0.smithi057.stdout: create 173/.snap/snap-subdir-test 2023-10-30T14:16:24.114 INFO:tasks.workunit.client.0.smithi057.stdout: create 173/30 file after the snapshot 2023-10-30T14:16:24.117 INFO:tasks.workunit.client.0.smithi057.stdout: create 173/31 file after the snapshot 2023-10-30T14:16:24.117 INFO:tasks.workunit.client.0.smithi057.stdout: create 173/32 file after the snapshot 2023-10-30T14:16:24.117 INFO:tasks.workunit.client.0.smithi057.stdout: create 173/33 file after the snapshot 2023-10-30T14:16:24.117 INFO:tasks.workunit.client.0.smithi057.stdout: create 173/34 file after the snapshot 2023-10-30T14:16:24.118 INFO:tasks.workunit.client.0.smithi057.stdout: create 173/35 file after the snapshot 2023-10-30T14:16:24.118 INFO:tasks.workunit.client.0.smithi057.stdout: create 173/36 file after the snapshot 2023-10-30T14:16:24.118 INFO:tasks.workunit.client.0.smithi057.stdout: create 173/37 file after the snapshot 2023-10-30T14:16:24.118 INFO:tasks.workunit.client.0.smithi057.stdout: create 173/38 file after the snapshot 2023-10-30T14:16:24.119 INFO:tasks.workunit.client.0.smithi057.stdout: create 173/39 file after the snapshot 2023-10-30T14:16:24.119 INFO:tasks.workunit.client.0.smithi057.stdout: create 174/.snap/snap-subdir-test 2023-10-30T14:16:24.176 INFO:tasks.workunit.client.0.smithi057.stdout: create 174/30 file after the snapshot 2023-10-30T14:16:24.180 INFO:tasks.workunit.client.0.smithi057.stdout: create 174/31 file after the snapshot 2023-10-30T14:16:24.180 INFO:tasks.workunit.client.0.smithi057.stdout: create 174/32 file after the snapshot 2023-10-30T14:16:24.180 INFO:tasks.workunit.client.0.smithi057.stdout: create 174/33 file after the snapshot 2023-10-30T14:16:24.180 INFO:tasks.workunit.client.0.smithi057.stdout: create 174/34 file after the snapshot 2023-10-30T14:16:24.180 INFO:tasks.workunit.client.0.smithi057.stdout: create 174/35 file after the snapshot 2023-10-30T14:16:24.181 INFO:tasks.workunit.client.0.smithi057.stdout: create 174/36 file after the snapshot 2023-10-30T14:16:24.181 INFO:tasks.workunit.client.0.smithi057.stdout: create 174/37 file after the snapshot 2023-10-30T14:16:24.181 INFO:tasks.workunit.client.0.smithi057.stdout: create 174/38 file after the snapshot 2023-10-30T14:16:24.181 INFO:tasks.workunit.client.0.smithi057.stdout: create 174/39 file after the snapshot 2023-10-30T14:16:24.182 INFO:tasks.workunit.client.0.smithi057.stdout: create 175/.snap/snap-subdir-test 2023-10-30T14:16:24.222 INFO:tasks.workunit.client.0.smithi057.stdout: create 175/30 file after the snapshot 2023-10-30T14:16:24.224 INFO:tasks.workunit.client.0.smithi057.stdout: create 175/31 file after the snapshot 2023-10-30T14:16:24.224 INFO:tasks.workunit.client.0.smithi057.stdout: create 175/32 file after the snapshot 2023-10-30T14:16:24.224 INFO:tasks.workunit.client.0.smithi057.stdout: create 175/33 file after the snapshot 2023-10-30T14:16:24.225 INFO:tasks.workunit.client.0.smithi057.stdout: create 175/34 file after the snapshot 2023-10-30T14:16:24.225 INFO:tasks.workunit.client.0.smithi057.stdout: create 175/35 file after the snapshot 2023-10-30T14:16:24.226 INFO:tasks.workunit.client.0.smithi057.stdout: create 175/36 file after the snapshot 2023-10-30T14:16:24.226 INFO:tasks.workunit.client.0.smithi057.stdout: create 175/37 file after the snapshot 2023-10-30T14:16:24.226 INFO:tasks.workunit.client.0.smithi057.stdout: create 175/38 file after the snapshot 2023-10-30T14:16:24.227 INFO:tasks.workunit.client.0.smithi057.stdout: create 175/39 file after the snapshot 2023-10-30T14:16:24.227 INFO:tasks.workunit.client.0.smithi057.stdout: create 176/.snap/snap-subdir-test 2023-10-30T14:16:24.264 INFO:tasks.workunit.client.0.smithi057.stdout: create 176/30 file after the snapshot 2023-10-30T14:16:24.270 INFO:tasks.workunit.client.0.smithi057.stdout: create 176/31 file after the snapshot 2023-10-30T14:16:24.270 INFO:tasks.workunit.client.0.smithi057.stdout: create 176/32 file after the snapshot 2023-10-30T14:16:24.270 INFO:tasks.workunit.client.0.smithi057.stdout: create 176/33 file after the snapshot 2023-10-30T14:16:24.271 INFO:tasks.workunit.client.0.smithi057.stdout: create 176/34 file after the snapshot 2023-10-30T14:16:24.271 INFO:tasks.workunit.client.0.smithi057.stdout: create 176/35 file after the snapshot 2023-10-30T14:16:24.271 INFO:tasks.workunit.client.0.smithi057.stdout: create 176/36 file after the snapshot 2023-10-30T14:16:24.271 INFO:tasks.workunit.client.0.smithi057.stdout: create 176/37 file after the snapshot 2023-10-30T14:16:24.271 INFO:tasks.workunit.client.0.smithi057.stdout: create 176/38 file after the snapshot 2023-10-30T14:16:24.272 INFO:tasks.workunit.client.0.smithi057.stdout: create 176/39 file after the snapshot 2023-10-30T14:16:24.272 INFO:tasks.workunit.client.0.smithi057.stdout: create 177/.snap/snap-subdir-test 2023-10-30T14:16:24.308 INFO:tasks.workunit.client.0.smithi057.stdout: create 177/30 file after the snapshot 2023-10-30T14:16:24.310 INFO:tasks.workunit.client.0.smithi057.stdout: create 177/31 file after the snapshot 2023-10-30T14:16:24.310 INFO:tasks.workunit.client.0.smithi057.stdout: create 177/32 file after the snapshot 2023-10-30T14:16:24.311 INFO:tasks.workunit.client.0.smithi057.stdout: create 177/33 file after the snapshot 2023-10-30T14:16:24.311 INFO:tasks.workunit.client.0.smithi057.stdout: create 177/34 file after the snapshot 2023-10-30T14:16:24.311 INFO:tasks.workunit.client.0.smithi057.stdout: create 177/35 file after the snapshot 2023-10-30T14:16:24.311 INFO:tasks.workunit.client.0.smithi057.stdout: create 177/36 file after the snapshot 2023-10-30T14:16:24.311 INFO:tasks.workunit.client.0.smithi057.stdout: create 177/37 file after the snapshot 2023-10-30T14:16:24.312 INFO:tasks.workunit.client.0.smithi057.stdout: create 177/38 file after the snapshot 2023-10-30T14:16:24.312 INFO:tasks.workunit.client.0.smithi057.stdout: create 177/39 file after the snapshot 2023-10-30T14:16:24.313 INFO:tasks.workunit.client.0.smithi057.stdout: create 178/.snap/snap-subdir-test 2023-10-30T14:16:24.345 INFO:tasks.workunit.client.0.smithi057.stdout: create 178/30 file after the snapshot 2023-10-30T14:16:24.348 INFO:tasks.workunit.client.0.smithi057.stdout: create 178/31 file after the snapshot 2023-10-30T14:16:24.349 INFO:tasks.workunit.client.0.smithi057.stdout: create 178/32 file after the snapshot 2023-10-30T14:16:24.349 INFO:tasks.workunit.client.0.smithi057.stdout: create 178/33 file after the snapshot 2023-10-30T14:16:24.349 INFO:tasks.workunit.client.0.smithi057.stdout: create 178/34 file after the snapshot 2023-10-30T14:16:24.349 INFO:tasks.workunit.client.0.smithi057.stdout: create 178/35 file after the snapshot 2023-10-30T14:16:24.350 INFO:tasks.workunit.client.0.smithi057.stdout: create 178/36 file after the snapshot 2023-10-30T14:16:24.350 INFO:tasks.workunit.client.0.smithi057.stdout: create 178/37 file after the snapshot 2023-10-30T14:16:24.350 INFO:tasks.workunit.client.0.smithi057.stdout: create 178/38 file after the snapshot 2023-10-30T14:16:24.350 INFO:tasks.workunit.client.0.smithi057.stdout: create 178/39 file after the snapshot 2023-10-30T14:16:24.351 INFO:tasks.workunit.client.0.smithi057.stdout: create 179/.snap/snap-subdir-test 2023-10-30T14:16:24.398 INFO:tasks.workunit.client.0.smithi057.stdout: create 179/30 file after the snapshot 2023-10-30T14:16:24.401 INFO:tasks.workunit.client.0.smithi057.stdout: create 179/31 file after the snapshot 2023-10-30T14:16:24.401 INFO:tasks.workunit.client.0.smithi057.stdout: create 179/32 file after the snapshot 2023-10-30T14:16:24.401 INFO:tasks.workunit.client.0.smithi057.stdout: create 179/33 file after the snapshot 2023-10-30T14:16:24.401 INFO:tasks.workunit.client.0.smithi057.stdout: create 179/34 file after the snapshot 2023-10-30T14:16:24.402 INFO:tasks.workunit.client.0.smithi057.stdout: create 179/35 file after the snapshot 2023-10-30T14:16:24.402 INFO:tasks.workunit.client.0.smithi057.stdout: create 179/36 file after the snapshot 2023-10-30T14:16:24.402 INFO:tasks.workunit.client.0.smithi057.stdout: create 179/37 file after the snapshot 2023-10-30T14:16:24.402 INFO:tasks.workunit.client.0.smithi057.stdout: create 179/38 file after the snapshot 2023-10-30T14:16:24.402 INFO:tasks.workunit.client.0.smithi057.stdout: create 179/39 file after the snapshot 2023-10-30T14:16:24.402 INFO:tasks.workunit.client.0.smithi057.stdout: create 180/.snap/snap-subdir-test 2023-10-30T14:16:24.475 INFO:tasks.workunit.client.0.smithi057.stdout: create 180/30 file after the snapshot 2023-10-30T14:16:24.479 INFO:tasks.workunit.client.0.smithi057.stdout: create 180/31 file after the snapshot 2023-10-30T14:16:24.479 INFO:tasks.workunit.client.0.smithi057.stdout: create 180/32 file after the snapshot 2023-10-30T14:16:24.479 INFO:tasks.workunit.client.0.smithi057.stdout: create 180/33 file after the snapshot 2023-10-30T14:16:24.479 INFO:tasks.workunit.client.0.smithi057.stdout: create 180/34 file after the snapshot 2023-10-30T14:16:24.479 INFO:tasks.workunit.client.0.smithi057.stdout: create 180/35 file after the snapshot 2023-10-30T14:16:24.480 INFO:tasks.workunit.client.0.smithi057.stdout: create 180/36 file after the snapshot 2023-10-30T14:16:24.480 INFO:tasks.workunit.client.0.smithi057.stdout: create 180/37 file after the snapshot 2023-10-30T14:16:24.480 INFO:tasks.workunit.client.0.smithi057.stdout: create 180/38 file after the snapshot 2023-10-30T14:16:24.480 INFO:tasks.workunit.client.0.smithi057.stdout: create 180/39 file after the snapshot 2023-10-30T14:16:24.481 INFO:tasks.workunit.client.0.smithi057.stdout: create 181/.snap/snap-subdir-test 2023-10-30T14:16:24.522 INFO:tasks.workunit.client.0.smithi057.stdout: create 181/30 file after the snapshot 2023-10-30T14:16:24.524 INFO:tasks.workunit.client.0.smithi057.stdout: create 181/31 file after the snapshot 2023-10-30T14:16:24.524 INFO:tasks.workunit.client.0.smithi057.stdout: create 181/32 file after the snapshot 2023-10-30T14:16:24.525 INFO:tasks.workunit.client.0.smithi057.stdout: create 181/33 file after the snapshot 2023-10-30T14:16:24.525 INFO:tasks.workunit.client.0.smithi057.stdout: create 181/34 file after the snapshot 2023-10-30T14:16:24.525 INFO:tasks.workunit.client.0.smithi057.stdout: create 181/35 file after the snapshot 2023-10-30T14:16:24.525 INFO:tasks.workunit.client.0.smithi057.stdout: create 181/36 file after the snapshot 2023-10-30T14:16:24.525 INFO:tasks.workunit.client.0.smithi057.stdout: create 181/37 file after the snapshot 2023-10-30T14:16:24.526 INFO:tasks.workunit.client.0.smithi057.stdout: create 181/38 file after the snapshot 2023-10-30T14:16:24.526 INFO:tasks.workunit.client.0.smithi057.stdout: create 181/39 file after the snapshot 2023-10-30T14:16:24.527 INFO:tasks.workunit.client.0.smithi057.stdout: create 182/.snap/snap-subdir-test 2023-10-30T14:16:24.567 INFO:tasks.workunit.client.0.smithi057.stdout: create 182/30 file after the snapshot 2023-10-30T14:16:24.570 INFO:tasks.workunit.client.0.smithi057.stdout: create 182/31 file after the snapshot 2023-10-30T14:16:24.570 INFO:tasks.workunit.client.0.smithi057.stdout: create 182/32 file after the snapshot 2023-10-30T14:16:24.570 INFO:tasks.workunit.client.0.smithi057.stdout: create 182/33 file after the snapshot 2023-10-30T14:16:24.571 INFO:tasks.workunit.client.0.smithi057.stdout: create 182/34 file after the snapshot 2023-10-30T14:16:24.571 INFO:tasks.workunit.client.0.smithi057.stdout: create 182/35 file after the snapshot 2023-10-30T14:16:24.571 INFO:tasks.workunit.client.0.smithi057.stdout: create 182/36 file after the snapshot 2023-10-30T14:16:24.571 INFO:tasks.workunit.client.0.smithi057.stdout: create 182/37 file after the snapshot 2023-10-30T14:16:24.571 INFO:tasks.workunit.client.0.smithi057.stdout: create 182/38 file after the snapshot 2023-10-30T14:16:24.572 INFO:tasks.workunit.client.0.smithi057.stdout: create 182/39 file after the snapshot 2023-10-30T14:16:24.572 INFO:tasks.workunit.client.0.smithi057.stdout: create 183/.snap/snap-subdir-test 2023-10-30T14:16:24.610 INFO:tasks.workunit.client.0.smithi057.stdout: create 183/30 file after the snapshot 2023-10-30T14:16:24.611 INFO:tasks.workunit.client.0.smithi057.stdout: create 183/31 file after the snapshot 2023-10-30T14:16:24.612 INFO:tasks.workunit.client.0.smithi057.stdout: create 183/32 file after the snapshot 2023-10-30T14:16:24.612 INFO:tasks.workunit.client.0.smithi057.stdout: create 183/33 file after the snapshot 2023-10-30T14:16:24.612 INFO:tasks.workunit.client.0.smithi057.stdout: create 183/34 file after the snapshot 2023-10-30T14:16:24.612 INFO:tasks.workunit.client.0.smithi057.stdout: create 183/35 file after the snapshot 2023-10-30T14:16:24.612 INFO:tasks.workunit.client.0.smithi057.stdout: create 183/36 file after the snapshot 2023-10-30T14:16:24.613 INFO:tasks.workunit.client.0.smithi057.stdout: create 183/37 file after the snapshot 2023-10-30T14:16:24.613 INFO:tasks.workunit.client.0.smithi057.stdout: create 183/38 file after the snapshot 2023-10-30T14:16:24.613 INFO:tasks.workunit.client.0.smithi057.stdout: create 183/39 file after the snapshot 2023-10-30T14:16:24.614 INFO:tasks.workunit.client.0.smithi057.stdout: create 184/.snap/snap-subdir-test 2023-10-30T14:16:24.653 INFO:tasks.workunit.client.0.smithi057.stdout: create 184/30 file after the snapshot 2023-10-30T14:16:24.657 INFO:tasks.workunit.client.0.smithi057.stdout: create 184/31 file after the snapshot 2023-10-30T14:16:24.657 INFO:tasks.workunit.client.0.smithi057.stdout: create 184/32 file after the snapshot 2023-10-30T14:16:24.657 INFO:tasks.workunit.client.0.smithi057.stdout: create 184/33 file after the snapshot 2023-10-30T14:16:24.658 INFO:tasks.workunit.client.0.smithi057.stdout: create 184/34 file after the snapshot 2023-10-30T14:16:24.658 INFO:tasks.workunit.client.0.smithi057.stdout: create 184/35 file after the snapshot 2023-10-30T14:16:24.658 INFO:tasks.workunit.client.0.smithi057.stdout: create 184/36 file after the snapshot 2023-10-30T14:16:24.658 INFO:tasks.workunit.client.0.smithi057.stdout: create 184/37 file after the snapshot 2023-10-30T14:16:24.658 INFO:tasks.workunit.client.0.smithi057.stdout: create 184/38 file after the snapshot 2023-10-30T14:16:24.659 INFO:tasks.workunit.client.0.smithi057.stdout: create 184/39 file after the snapshot 2023-10-30T14:16:24.659 INFO:tasks.workunit.client.0.smithi057.stdout: create 185/.snap/snap-subdir-test 2023-10-30T14:16:24.695 INFO:tasks.workunit.client.0.smithi057.stdout: create 185/30 file after the snapshot 2023-10-30T14:16:24.697 INFO:tasks.workunit.client.0.smithi057.stdout: create 185/31 file after the snapshot 2023-10-30T14:16:24.697 INFO:tasks.workunit.client.0.smithi057.stdout: create 185/32 file after the snapshot 2023-10-30T14:16:24.697 INFO:tasks.workunit.client.0.smithi057.stdout: create 185/33 file after the snapshot 2023-10-30T14:16:24.697 INFO:tasks.workunit.client.0.smithi057.stdout: create 185/34 file after the snapshot 2023-10-30T14:16:24.697 INFO:tasks.workunit.client.0.smithi057.stdout: create 185/35 file after the snapshot 2023-10-30T14:16:24.697 INFO:tasks.workunit.client.0.smithi057.stdout: create 185/36 file after the snapshot 2023-10-30T14:16:24.698 INFO:tasks.workunit.client.0.smithi057.stdout: create 185/37 file after the snapshot 2023-10-30T14:16:24.698 INFO:tasks.workunit.client.0.smithi057.stdout: create 185/38 file after the snapshot 2023-10-30T14:16:24.698 INFO:tasks.workunit.client.0.smithi057.stdout: create 185/39 file after the snapshot 2023-10-30T14:16:24.698 INFO:tasks.workunit.client.0.smithi057.stdout: create 186/.snap/snap-subdir-test 2023-10-30T14:16:24.730 INFO:tasks.workunit.client.0.smithi057.stdout: create 186/30 file after the snapshot 2023-10-30T14:16:24.731 INFO:tasks.workunit.client.0.smithi057.stdout: create 186/31 file after the snapshot 2023-10-30T14:16:24.731 INFO:tasks.workunit.client.0.smithi057.stdout: create 186/32 file after the snapshot 2023-10-30T14:16:24.732 INFO:tasks.workunit.client.0.smithi057.stdout: create 186/33 file after the snapshot 2023-10-30T14:16:24.732 INFO:tasks.workunit.client.0.smithi057.stdout: create 186/34 file after the snapshot 2023-10-30T14:16:24.732 INFO:tasks.workunit.client.0.smithi057.stdout: create 186/35 file after the snapshot 2023-10-30T14:16:24.732 INFO:tasks.workunit.client.0.smithi057.stdout: create 186/36 file after the snapshot 2023-10-30T14:16:24.732 INFO:tasks.workunit.client.0.smithi057.stdout: create 186/37 file after the snapshot 2023-10-30T14:16:24.732 INFO:tasks.workunit.client.0.smithi057.stdout: create 186/38 file after the snapshot 2023-10-30T14:16:24.733 INFO:tasks.workunit.client.0.smithi057.stdout: create 186/39 file after the snapshot 2023-10-30T14:16:24.733 INFO:tasks.workunit.client.0.smithi057.stdout: create 187/.snap/snap-subdir-test 2023-10-30T14:16:24.769 INFO:tasks.workunit.client.0.smithi057.stdout: create 187/30 file after the snapshot 2023-10-30T14:16:24.771 INFO:tasks.workunit.client.0.smithi057.stdout: create 187/31 file after the snapshot 2023-10-30T14:16:24.771 INFO:tasks.workunit.client.0.smithi057.stdout: create 187/32 file after the snapshot 2023-10-30T14:16:24.771 INFO:tasks.workunit.client.0.smithi057.stdout: create 187/33 file after the snapshot 2023-10-30T14:16:24.772 INFO:tasks.workunit.client.0.smithi057.stdout: create 187/34 file after the snapshot 2023-10-30T14:16:24.772 INFO:tasks.workunit.client.0.smithi057.stdout: create 187/35 file after the snapshot 2023-10-30T14:16:24.772 INFO:tasks.workunit.client.0.smithi057.stdout: create 187/36 file after the snapshot 2023-10-30T14:16:24.772 INFO:tasks.workunit.client.0.smithi057.stdout: create 187/37 file after the snapshot 2023-10-30T14:16:24.772 INFO:tasks.workunit.client.0.smithi057.stdout: create 187/38 file after the snapshot 2023-10-30T14:16:24.773 INFO:tasks.workunit.client.0.smithi057.stdout: create 187/39 file after the snapshot 2023-10-30T14:16:24.773 INFO:tasks.workunit.client.0.smithi057.stdout: create 188/.snap/snap-subdir-test 2023-10-30T14:16:25.515 INFO:tasks.workunit.client.0.smithi057.stdout: create 188/30 file after the snapshot 2023-10-30T14:16:25.519 INFO:tasks.workunit.client.0.smithi057.stdout: create 188/31 file after the snapshot 2023-10-30T14:16:25.519 INFO:tasks.workunit.client.0.smithi057.stdout: create 188/32 file after the snapshot 2023-10-30T14:16:25.519 INFO:tasks.workunit.client.0.smithi057.stdout: create 188/33 file after the snapshot 2023-10-30T14:16:25.520 INFO:tasks.workunit.client.0.smithi057.stdout: create 188/34 file after the snapshot 2023-10-30T14:16:25.520 INFO:tasks.workunit.client.0.smithi057.stdout: create 188/35 file after the snapshot 2023-10-30T14:16:25.520 INFO:tasks.workunit.client.0.smithi057.stdout: create 188/36 file after the snapshot 2023-10-30T14:16:25.520 INFO:tasks.workunit.client.0.smithi057.stdout: create 188/37 file after the snapshot 2023-10-30T14:16:25.521 INFO:tasks.workunit.client.0.smithi057.stdout: create 188/38 file after the snapshot 2023-10-30T14:16:25.521 INFO:tasks.workunit.client.0.smithi057.stdout: create 188/39 file after the snapshot 2023-10-30T14:16:25.521 INFO:tasks.workunit.client.0.smithi057.stdout: create 189/.snap/snap-subdir-test 2023-10-30T14:16:25.596 INFO:tasks.workunit.client.0.smithi057.stdout: create 189/30 file after the snapshot 2023-10-30T14:16:25.599 INFO:tasks.workunit.client.0.smithi057.stdout: create 189/31 file after the snapshot 2023-10-30T14:16:25.599 INFO:tasks.workunit.client.0.smithi057.stdout: create 189/32 file after the snapshot 2023-10-30T14:16:25.599 INFO:tasks.workunit.client.0.smithi057.stdout: create 189/33 file after the snapshot 2023-10-30T14:16:25.599 INFO:tasks.workunit.client.0.smithi057.stdout: create 189/34 file after the snapshot 2023-10-30T14:16:25.599 INFO:tasks.workunit.client.0.smithi057.stdout: create 189/35 file after the snapshot 2023-10-30T14:16:25.600 INFO:tasks.workunit.client.0.smithi057.stdout: create 189/36 file after the snapshot 2023-10-30T14:16:25.600 INFO:tasks.workunit.client.0.smithi057.stdout: create 189/37 file after the snapshot 2023-10-30T14:16:25.600 INFO:tasks.workunit.client.0.smithi057.stdout: create 189/38 file after the snapshot 2023-10-30T14:16:25.600 INFO:tasks.workunit.client.0.smithi057.stdout: create 189/39 file after the snapshot 2023-10-30T14:16:25.600 INFO:tasks.workunit.client.0.smithi057.stdout: create 190/.snap/snap-subdir-test 2023-10-30T14:16:25.641 INFO:tasks.workunit.client.0.smithi057.stdout: create 190/30 file after the snapshot 2023-10-30T14:16:25.643 INFO:tasks.workunit.client.0.smithi057.stdout: create 190/31 file after the snapshot 2023-10-30T14:16:25.644 INFO:tasks.workunit.client.0.smithi057.stdout: create 190/32 file after the snapshot 2023-10-30T14:16:25.644 INFO:tasks.workunit.client.0.smithi057.stdout: create 190/33 file after the snapshot 2023-10-30T14:16:25.644 INFO:tasks.workunit.client.0.smithi057.stdout: create 190/34 file after the snapshot 2023-10-30T14:16:25.644 INFO:tasks.workunit.client.0.smithi057.stdout: create 190/35 file after the snapshot 2023-10-30T14:16:25.644 INFO:tasks.workunit.client.0.smithi057.stdout: create 190/36 file after the snapshot 2023-10-30T14:16:25.644 INFO:tasks.workunit.client.0.smithi057.stdout: create 190/37 file after the snapshot 2023-10-30T14:16:25.645 INFO:tasks.workunit.client.0.smithi057.stdout: create 190/38 file after the snapshot 2023-10-30T14:16:25.645 INFO:tasks.workunit.client.0.smithi057.stdout: create 190/39 file after the snapshot 2023-10-30T14:16:25.645 INFO:tasks.workunit.client.0.smithi057.stdout: create 191/.snap/snap-subdir-test 2023-10-30T14:16:25.695 INFO:tasks.workunit.client.0.smithi057.stdout: create 191/30 file after the snapshot 2023-10-30T14:16:25.698 INFO:tasks.workunit.client.0.smithi057.stdout: create 191/31 file after the snapshot 2023-10-30T14:16:25.698 INFO:tasks.workunit.client.0.smithi057.stdout: create 191/32 file after the snapshot 2023-10-30T14:16:25.699 INFO:tasks.workunit.client.0.smithi057.stdout: create 191/33 file after the snapshot 2023-10-30T14:16:25.699 INFO:tasks.workunit.client.0.smithi057.stdout: create 191/34 file after the snapshot 2023-10-30T14:16:25.699 INFO:tasks.workunit.client.0.smithi057.stdout: create 191/35 file after the snapshot 2023-10-30T14:16:25.699 INFO:tasks.workunit.client.0.smithi057.stdout: create 191/36 file after the snapshot 2023-10-30T14:16:25.699 INFO:tasks.workunit.client.0.smithi057.stdout: create 191/37 file after the snapshot 2023-10-30T14:16:25.700 INFO:tasks.workunit.client.0.smithi057.stdout: create 191/38 file after the snapshot 2023-10-30T14:16:25.700 INFO:tasks.workunit.client.0.smithi057.stdout: create 191/39 file after the snapshot 2023-10-30T14:16:25.700 INFO:tasks.workunit.client.0.smithi057.stdout: create 192/.snap/snap-subdir-test 2023-10-30T14:16:26.721 INFO:tasks.workunit.client.0.smithi057.stdout: create 192/30 file after the snapshot 2023-10-30T14:16:26.727 INFO:tasks.workunit.client.0.smithi057.stdout: create 192/31 file after the snapshot 2023-10-30T14:16:26.727 INFO:tasks.workunit.client.0.smithi057.stdout: create 192/32 file after the snapshot 2023-10-30T14:16:26.727 INFO:tasks.workunit.client.0.smithi057.stdout: create 192/33 file after the snapshot 2023-10-30T14:16:26.727 INFO:tasks.workunit.client.0.smithi057.stdout: create 192/34 file after the snapshot 2023-10-30T14:16:26.728 INFO:tasks.workunit.client.0.smithi057.stdout: create 192/35 file after the snapshot 2023-10-30T14:16:26.728 INFO:tasks.workunit.client.0.smithi057.stdout: create 192/36 file after the snapshot 2023-10-30T14:16:26.728 INFO:tasks.workunit.client.0.smithi057.stdout: create 192/37 file after the snapshot 2023-10-30T14:16:26.728 INFO:tasks.workunit.client.0.smithi057.stdout: create 192/38 file after the snapshot 2023-10-30T14:16:26.729 INFO:tasks.workunit.client.0.smithi057.stdout: create 192/39 file after the snapshot 2023-10-30T14:16:26.729 INFO:tasks.workunit.client.0.smithi057.stdout: create 193/.snap/snap-subdir-test 2023-10-30T14:16:26.767 INFO:tasks.workunit.client.0.smithi057.stdout: create 193/30 file after the snapshot 2023-10-30T14:16:26.770 INFO:tasks.workunit.client.0.smithi057.stdout: create 193/31 file after the snapshot 2023-10-30T14:16:26.770 INFO:tasks.workunit.client.0.smithi057.stdout: create 193/32 file after the snapshot 2023-10-30T14:16:26.770 INFO:tasks.workunit.client.0.smithi057.stdout: create 193/33 file after the snapshot 2023-10-30T14:16:26.770 INFO:tasks.workunit.client.0.smithi057.stdout: create 193/34 file after the snapshot 2023-10-30T14:16:26.771 INFO:tasks.workunit.client.0.smithi057.stdout: create 193/35 file after the snapshot 2023-10-30T14:16:26.771 INFO:tasks.workunit.client.0.smithi057.stdout: create 193/36 file after the snapshot 2023-10-30T14:16:26.771 INFO:tasks.workunit.client.0.smithi057.stdout: create 193/37 file after the snapshot 2023-10-30T14:16:26.771 INFO:tasks.workunit.client.0.smithi057.stdout: create 193/38 file after the snapshot 2023-10-30T14:16:26.771 INFO:tasks.workunit.client.0.smithi057.stdout: create 193/39 file after the snapshot 2023-10-30T14:16:26.772 INFO:tasks.workunit.client.0.smithi057.stdout: create 194/.snap/snap-subdir-test 2023-10-30T14:16:26.812 INFO:tasks.workunit.client.0.smithi057.stdout: create 194/30 file after the snapshot 2023-10-30T14:16:26.814 INFO:tasks.workunit.client.0.smithi057.stdout: create 194/31 file after the snapshot 2023-10-30T14:16:26.815 INFO:tasks.workunit.client.0.smithi057.stdout: create 194/32 file after the snapshot 2023-10-30T14:16:26.815 INFO:tasks.workunit.client.0.smithi057.stdout: create 194/33 file after the snapshot 2023-10-30T14:16:26.815 INFO:tasks.workunit.client.0.smithi057.stdout: create 194/34 file after the snapshot 2023-10-30T14:16:26.815 INFO:tasks.workunit.client.0.smithi057.stdout: create 194/35 file after the snapshot 2023-10-30T14:16:26.816 INFO:tasks.workunit.client.0.smithi057.stdout: create 194/36 file after the snapshot 2023-10-30T14:16:26.816 INFO:tasks.workunit.client.0.smithi057.stdout: create 194/37 file after the snapshot 2023-10-30T14:16:26.816 INFO:tasks.workunit.client.0.smithi057.stdout: create 194/38 file after the snapshot 2023-10-30T14:16:26.816 INFO:tasks.workunit.client.0.smithi057.stdout: create 194/39 file after the snapshot 2023-10-30T14:16:26.816 INFO:tasks.workunit.client.0.smithi057.stdout: create 195/.snap/snap-subdir-test 2023-10-30T14:16:26.843 INFO:tasks.workunit.client.0.smithi057.stdout: create 195/30 file after the snapshot 2023-10-30T14:16:26.844 INFO:tasks.workunit.client.0.smithi057.stdout: create 195/31 file after the snapshot 2023-10-30T14:16:26.845 INFO:tasks.workunit.client.0.smithi057.stdout: create 195/32 file after the snapshot 2023-10-30T14:16:26.845 INFO:tasks.workunit.client.0.smithi057.stdout: create 195/33 file after the snapshot 2023-10-30T14:16:26.845 INFO:tasks.workunit.client.0.smithi057.stdout: create 195/34 file after the snapshot 2023-10-30T14:16:26.845 INFO:tasks.workunit.client.0.smithi057.stdout: create 195/35 file after the snapshot 2023-10-30T14:16:26.846 INFO:tasks.workunit.client.0.smithi057.stdout: create 195/36 file after the snapshot 2023-10-30T14:16:26.846 INFO:tasks.workunit.client.0.smithi057.stdout: create 195/37 file after the snapshot 2023-10-30T14:16:26.846 INFO:tasks.workunit.client.0.smithi057.stdout: create 195/38 file after the snapshot 2023-10-30T14:16:26.846 INFO:tasks.workunit.client.0.smithi057.stdout: create 195/39 file after the snapshot 2023-10-30T14:16:26.847 INFO:tasks.workunit.client.0.smithi057.stdout: create 196/.snap/snap-subdir-test 2023-10-30T14:16:26.873 INFO:tasks.workunit.client.0.smithi057.stdout: create 196/30 file after the snapshot 2023-10-30T14:16:26.878 INFO:tasks.workunit.client.0.smithi057.stdout: create 196/31 file after the snapshot 2023-10-30T14:16:26.879 INFO:tasks.workunit.client.0.smithi057.stdout: create 196/32 file after the snapshot 2023-10-30T14:16:26.879 INFO:tasks.workunit.client.0.smithi057.stdout: create 196/33 file after the snapshot 2023-10-30T14:16:26.879 INFO:tasks.workunit.client.0.smithi057.stdout: create 196/34 file after the snapshot 2023-10-30T14:16:26.879 INFO:tasks.workunit.client.0.smithi057.stdout: create 196/35 file after the snapshot 2023-10-30T14:16:26.880 INFO:tasks.workunit.client.0.smithi057.stdout: create 196/36 file after the snapshot 2023-10-30T14:16:26.880 INFO:tasks.workunit.client.0.smithi057.stdout: create 196/37 file after the snapshot 2023-10-30T14:16:26.880 INFO:tasks.workunit.client.0.smithi057.stdout: create 196/38 file after the snapshot 2023-10-30T14:16:26.880 INFO:tasks.workunit.client.0.smithi057.stdout: create 196/39 file after the snapshot 2023-10-30T14:16:26.880 INFO:tasks.workunit.client.0.smithi057.stdout: create 197/.snap/snap-subdir-test 2023-10-30T14:16:27.175 INFO:tasks.workunit.client.0.smithi057.stdout: create 197/30 file after the snapshot 2023-10-30T14:16:27.178 INFO:tasks.workunit.client.0.smithi057.stdout: create 197/31 file after the snapshot 2023-10-30T14:16:27.178 INFO:tasks.workunit.client.0.smithi057.stdout: create 197/32 file after the snapshot 2023-10-30T14:16:27.179 INFO:tasks.workunit.client.0.smithi057.stdout: create 197/33 file after the snapshot 2023-10-30T14:16:27.179 INFO:tasks.workunit.client.0.smithi057.stdout: create 197/34 file after the snapshot 2023-10-30T14:16:27.179 INFO:tasks.workunit.client.0.smithi057.stdout: create 197/35 file after the snapshot 2023-10-30T14:16:27.179 INFO:tasks.workunit.client.0.smithi057.stdout: create 197/36 file after the snapshot 2023-10-30T14:16:27.179 INFO:tasks.workunit.client.0.smithi057.stdout: create 197/37 file after the snapshot 2023-10-30T14:16:27.180 INFO:tasks.workunit.client.0.smithi057.stdout: create 197/38 file after the snapshot 2023-10-30T14:16:27.180 INFO:tasks.workunit.client.0.smithi057.stdout: create 197/39 file after the snapshot 2023-10-30T14:16:27.180 INFO:tasks.workunit.client.0.smithi057.stdout: create 198/.snap/snap-subdir-test 2023-10-30T14:16:27.221 INFO:tasks.workunit.client.0.smithi057.stdout: create 198/30 file after the snapshot 2023-10-30T14:16:27.223 INFO:tasks.workunit.client.0.smithi057.stdout: create 198/31 file after the snapshot 2023-10-30T14:16:27.223 INFO:tasks.workunit.client.0.smithi057.stdout: create 198/32 file after the snapshot 2023-10-30T14:16:27.223 INFO:tasks.workunit.client.0.smithi057.stdout: create 198/33 file after the snapshot 2023-10-30T14:16:27.223 INFO:tasks.workunit.client.0.smithi057.stdout: create 198/34 file after the snapshot 2023-10-30T14:16:27.224 INFO:tasks.workunit.client.0.smithi057.stdout: create 198/35 file after the snapshot 2023-10-30T14:16:27.224 INFO:tasks.workunit.client.0.smithi057.stdout: create 198/36 file after the snapshot 2023-10-30T14:16:27.224 INFO:tasks.workunit.client.0.smithi057.stdout: create 198/37 file after the snapshot 2023-10-30T14:16:27.224 INFO:tasks.workunit.client.0.smithi057.stdout: create 198/38 file after the snapshot 2023-10-30T14:16:27.224 INFO:tasks.workunit.client.0.smithi057.stdout: create 198/39 file after the snapshot 2023-10-30T14:16:27.225 INFO:tasks.workunit.client.0.smithi057.stdout: create 199/.snap/snap-subdir-test 2023-10-30T14:16:27.276 INFO:tasks.workunit.client.0.smithi057.stdout: create 199/30 file after the snapshot 2023-10-30T14:16:27.282 INFO:tasks.workunit.client.0.smithi057.stdout: create 199/31 file after the snapshot 2023-10-30T14:16:27.282 INFO:tasks.workunit.client.0.smithi057.stdout: create 199/32 file after the snapshot 2023-10-30T14:16:27.282 INFO:tasks.workunit.client.0.smithi057.stdout: create 199/33 file after the snapshot 2023-10-30T14:16:27.282 INFO:tasks.workunit.client.0.smithi057.stdout: create 199/34 file after the snapshot 2023-10-30T14:16:27.283 INFO:tasks.workunit.client.0.smithi057.stdout: create 199/35 file after the snapshot 2023-10-30T14:16:27.283 INFO:tasks.workunit.client.0.smithi057.stdout: create 199/36 file after the snapshot 2023-10-30T14:16:27.283 INFO:tasks.workunit.client.0.smithi057.stdout: create 199/37 file after the snapshot 2023-10-30T14:16:27.283 INFO:tasks.workunit.client.0.smithi057.stdout: create 199/38 file after the snapshot 2023-10-30T14:16:27.283 INFO:tasks.workunit.client.0.smithi057.stdout: create 199/39 file after the snapshot 2023-10-30T14:16:27.283 INFO:tasks.workunit.client.0.smithi057.stdout: create 200/.snap/snap-subdir-test 2023-10-30T14:16:27.325 INFO:tasks.workunit.client.0.smithi057.stdout: create 200/30 file after the snapshot 2023-10-30T14:16:27.326 INFO:tasks.workunit.client.0.smithi057.stdout: create 200/31 file after the snapshot 2023-10-30T14:16:27.327 INFO:tasks.workunit.client.0.smithi057.stdout: create 200/32 file after the snapshot 2023-10-30T14:16:27.327 INFO:tasks.workunit.client.0.smithi057.stdout: create 200/33 file after the snapshot 2023-10-30T14:16:27.327 INFO:tasks.workunit.client.0.smithi057.stdout: create 200/34 file after the snapshot 2023-10-30T14:16:27.327 INFO:tasks.workunit.client.0.smithi057.stdout: create 200/35 file after the snapshot 2023-10-30T14:16:27.327 INFO:tasks.workunit.client.0.smithi057.stdout: create 200/36 file after the snapshot 2023-10-30T14:16:27.328 INFO:tasks.workunit.client.0.smithi057.stdout: create 200/37 file after the snapshot 2023-10-30T14:16:27.328 INFO:tasks.workunit.client.0.smithi057.stdout: create 200/38 file after the snapshot 2023-10-30T14:16:27.328 INFO:tasks.workunit.client.0.smithi057.stdout: create 200/39 file after the snapshot 2023-10-30T14:16:27.328 INFO:tasks.workunit.client.0.smithi057.stdout: create 201/.snap/snap-subdir-test 2023-10-30T14:16:28.116 INFO:tasks.workunit.client.0.smithi057.stdout: create 201/30 file after the snapshot 2023-10-30T14:16:28.227 INFO:tasks.workunit.client.0.smithi057.stdout: create 201/31 file after the snapshot 2023-10-30T14:16:28.227 INFO:tasks.workunit.client.0.smithi057.stdout: create 201/32 file after the snapshot 2023-10-30T14:16:28.227 INFO:tasks.workunit.client.0.smithi057.stdout: create 201/33 file after the snapshot 2023-10-30T14:16:28.228 INFO:tasks.workunit.client.0.smithi057.stdout: create 201/34 file after the snapshot 2023-10-30T14:16:28.228 INFO:tasks.workunit.client.0.smithi057.stdout: create 201/35 file after the snapshot 2023-10-30T14:16:28.228 INFO:tasks.workunit.client.0.smithi057.stdout: create 201/36 file after the snapshot 2023-10-30T14:16:28.228 INFO:tasks.workunit.client.0.smithi057.stdout: create 201/37 file after the snapshot 2023-10-30T14:16:28.228 INFO:tasks.workunit.client.0.smithi057.stdout: create 201/38 file after the snapshot 2023-10-30T14:16:28.229 INFO:tasks.workunit.client.0.smithi057.stdout: create 201/39 file after the snapshot 2023-10-30T14:16:28.229 INFO:tasks.workunit.client.0.smithi057.stdout: create 202/.snap/snap-subdir-test 2023-10-30T14:16:29.029 INFO:tasks.workunit.client.0.smithi057.stdout: create 202/30 file after the snapshot 2023-10-30T14:16:29.031 INFO:tasks.workunit.client.0.smithi057.stdout: create 202/31 file after the snapshot 2023-10-30T14:16:29.031 INFO:tasks.workunit.client.0.smithi057.stdout: create 202/32 file after the snapshot 2023-10-30T14:16:29.031 INFO:tasks.workunit.client.0.smithi057.stdout: create 202/33 file after the snapshot 2023-10-30T14:16:29.032 INFO:tasks.workunit.client.0.smithi057.stdout: create 202/34 file after the snapshot 2023-10-30T14:16:29.032 INFO:tasks.workunit.client.0.smithi057.stdout: create 202/35 file after the snapshot 2023-10-30T14:16:29.032 INFO:tasks.workunit.client.0.smithi057.stdout: create 202/36 file after the snapshot 2023-10-30T14:16:29.032 INFO:tasks.workunit.client.0.smithi057.stdout: create 202/37 file after the snapshot 2023-10-30T14:16:29.032 INFO:tasks.workunit.client.0.smithi057.stdout: create 202/38 file after the snapshot 2023-10-30T14:16:29.032 INFO:tasks.workunit.client.0.smithi057.stdout: create 202/39 file after the snapshot 2023-10-30T14:16:29.033 INFO:tasks.workunit.client.0.smithi057.stdout: create 203/.snap/snap-subdir-test 2023-10-30T14:16:29.059 INFO:tasks.workunit.client.0.smithi057.stdout: create 203/30 file after the snapshot 2023-10-30T14:16:29.061 INFO:tasks.workunit.client.0.smithi057.stdout: create 203/31 file after the snapshot 2023-10-30T14:16:29.061 INFO:tasks.workunit.client.0.smithi057.stdout: create 203/32 file after the snapshot 2023-10-30T14:16:29.061 INFO:tasks.workunit.client.0.smithi057.stdout: create 203/33 file after the snapshot 2023-10-30T14:16:29.061 INFO:tasks.workunit.client.0.smithi057.stdout: create 203/34 file after the snapshot 2023-10-30T14:16:29.061 INFO:tasks.workunit.client.0.smithi057.stdout: create 203/35 file after the snapshot 2023-10-30T14:16:29.062 INFO:tasks.workunit.client.0.smithi057.stdout: create 203/36 file after the snapshot 2023-10-30T14:16:29.062 INFO:tasks.workunit.client.0.smithi057.stdout: create 203/37 file after the snapshot 2023-10-30T14:16:29.062 INFO:tasks.workunit.client.0.smithi057.stdout: create 203/38 file after the snapshot 2023-10-30T14:16:29.062 INFO:tasks.workunit.client.0.smithi057.stdout: create 203/39 file after the snapshot 2023-10-30T14:16:29.063 INFO:tasks.workunit.client.0.smithi057.stdout: create 204/.snap/snap-subdir-test 2023-10-30T14:16:29.086 INFO:tasks.workunit.client.0.smithi057.stdout: create 204/30 file after the snapshot 2023-10-30T14:16:29.089 INFO:tasks.workunit.client.0.smithi057.stdout: create 204/31 file after the snapshot 2023-10-30T14:16:29.089 INFO:tasks.workunit.client.0.smithi057.stdout: create 204/32 file after the snapshot 2023-10-30T14:16:29.089 INFO:tasks.workunit.client.0.smithi057.stdout: create 204/33 file after the snapshot 2023-10-30T14:16:29.089 INFO:tasks.workunit.client.0.smithi057.stdout: create 204/34 file after the snapshot 2023-10-30T14:16:29.090 INFO:tasks.workunit.client.0.smithi057.stdout: create 204/35 file after the snapshot 2023-10-30T14:16:29.090 INFO:tasks.workunit.client.0.smithi057.stdout: create 204/36 file after the snapshot 2023-10-30T14:16:29.090 INFO:tasks.workunit.client.0.smithi057.stdout: create 204/37 file after the snapshot 2023-10-30T14:16:29.090 INFO:tasks.workunit.client.0.smithi057.stdout: create 204/38 file after the snapshot 2023-10-30T14:16:29.090 INFO:tasks.workunit.client.0.smithi057.stdout: create 204/39 file after the snapshot 2023-10-30T14:16:29.091 INFO:tasks.workunit.client.0.smithi057.stdout: create 205/.snap/snap-subdir-test 2023-10-30T14:16:29.142 INFO:tasks.workunit.client.0.smithi057.stdout: create 205/30 file after the snapshot 2023-10-30T14:16:29.145 INFO:tasks.workunit.client.0.smithi057.stdout: create 205/31 file after the snapshot 2023-10-30T14:16:29.145 INFO:tasks.workunit.client.0.smithi057.stdout: create 205/32 file after the snapshot 2023-10-30T14:16:29.145 INFO:tasks.workunit.client.0.smithi057.stdout: create 205/33 file after the snapshot 2023-10-30T14:16:29.145 INFO:tasks.workunit.client.0.smithi057.stdout: create 205/34 file after the snapshot 2023-10-30T14:16:29.145 INFO:tasks.workunit.client.0.smithi057.stdout: create 205/35 file after the snapshot 2023-10-30T14:16:29.146 INFO:tasks.workunit.client.0.smithi057.stdout: create 205/36 file after the snapshot 2023-10-30T14:16:29.146 INFO:tasks.workunit.client.0.smithi057.stdout: create 205/37 file after the snapshot 2023-10-30T14:16:29.146 INFO:tasks.workunit.client.0.smithi057.stdout: create 205/38 file after the snapshot 2023-10-30T14:16:29.146 INFO:tasks.workunit.client.0.smithi057.stdout: create 205/39 file after the snapshot 2023-10-30T14:16:29.146 INFO:tasks.workunit.client.0.smithi057.stdout: create 206/.snap/snap-subdir-test 2023-10-30T14:16:29.647 INFO:tasks.workunit.client.0.smithi057.stdout: create 206/30 file after the snapshot 2023-10-30T14:16:29.650 INFO:tasks.workunit.client.0.smithi057.stdout: create 206/31 file after the snapshot 2023-10-30T14:16:29.650 INFO:tasks.workunit.client.0.smithi057.stdout: create 206/32 file after the snapshot 2023-10-30T14:16:29.651 INFO:tasks.workunit.client.0.smithi057.stdout: create 206/33 file after the snapshot 2023-10-30T14:16:29.651 INFO:tasks.workunit.client.0.smithi057.stdout: create 206/34 file after the snapshot 2023-10-30T14:16:29.651 INFO:tasks.workunit.client.0.smithi057.stdout: create 206/35 file after the snapshot 2023-10-30T14:16:29.651 INFO:tasks.workunit.client.0.smithi057.stdout: create 206/36 file after the snapshot 2023-10-30T14:16:29.651 INFO:tasks.workunit.client.0.smithi057.stdout: create 206/37 file after the snapshot 2023-10-30T14:16:29.651 INFO:tasks.workunit.client.0.smithi057.stdout: create 206/38 file after the snapshot 2023-10-30T14:16:29.652 INFO:tasks.workunit.client.0.smithi057.stdout: create 206/39 file after the snapshot 2023-10-30T14:16:29.652 INFO:tasks.workunit.client.0.smithi057.stdout: create 207/.snap/snap-subdir-test 2023-10-30T14:16:29.690 INFO:tasks.workunit.client.0.smithi057.stdout: create 207/30 file after the snapshot 2023-10-30T14:16:29.691 INFO:tasks.workunit.client.0.smithi057.stdout: create 207/31 file after the snapshot 2023-10-30T14:16:29.692 INFO:tasks.workunit.client.0.smithi057.stdout: create 207/32 file after the snapshot 2023-10-30T14:16:29.692 INFO:tasks.workunit.client.0.smithi057.stdout: create 207/33 file after the snapshot 2023-10-30T14:16:29.692 INFO:tasks.workunit.client.0.smithi057.stdout: create 207/34 file after the snapshot 2023-10-30T14:16:29.692 INFO:tasks.workunit.client.0.smithi057.stdout: create 207/35 file after the snapshot 2023-10-30T14:16:29.692 INFO:tasks.workunit.client.0.smithi057.stdout: create 207/36 file after the snapshot 2023-10-30T14:16:29.693 INFO:tasks.workunit.client.0.smithi057.stdout: create 207/37 file after the snapshot 2023-10-30T14:16:29.693 INFO:tasks.workunit.client.0.smithi057.stdout: create 207/38 file after the snapshot 2023-10-30T14:16:29.693 INFO:tasks.workunit.client.0.smithi057.stdout: create 207/39 file after the snapshot 2023-10-30T14:16:29.693 INFO:tasks.workunit.client.0.smithi057.stdout: create 208/.snap/snap-subdir-test 2023-10-30T14:16:29.728 INFO:tasks.workunit.client.0.smithi057.stdout: create 208/30 file after the snapshot 2023-10-30T14:16:29.731 INFO:tasks.workunit.client.0.smithi057.stdout: create 208/31 file after the snapshot 2023-10-30T14:16:29.731 INFO:tasks.workunit.client.0.smithi057.stdout: create 208/32 file after the snapshot 2023-10-30T14:16:29.732 INFO:tasks.workunit.client.0.smithi057.stdout: create 208/33 file after the snapshot 2023-10-30T14:16:29.732 INFO:tasks.workunit.client.0.smithi057.stdout: create 208/34 file after the snapshot 2023-10-30T14:16:29.732 INFO:tasks.workunit.client.0.smithi057.stdout: create 208/35 file after the snapshot 2023-10-30T14:16:29.732 INFO:tasks.workunit.client.0.smithi057.stdout: create 208/36 file after the snapshot 2023-10-30T14:16:29.732 INFO:tasks.workunit.client.0.smithi057.stdout: create 208/37 file after the snapshot 2023-10-30T14:16:29.733 INFO:tasks.workunit.client.0.smithi057.stdout: create 208/38 file after the snapshot 2023-10-30T14:16:29.733 INFO:tasks.workunit.client.0.smithi057.stdout: create 208/39 file after the snapshot 2023-10-30T14:16:29.733 INFO:tasks.workunit.client.0.smithi057.stdout: create 209/.snap/snap-subdir-test 2023-10-30T14:16:29.781 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:16:29.785 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:16:30.017 INFO:tasks.workunit.client.0.smithi057.stdout: create 209/30 file after the snapshot 2023-10-30T14:16:30.019 INFO:tasks.workunit.client.0.smithi057.stdout: create 209/31 file after the snapshot 2023-10-30T14:16:30.020 INFO:tasks.workunit.client.0.smithi057.stdout: create 209/32 file after the snapshot 2023-10-30T14:16:30.020 INFO:tasks.workunit.client.0.smithi057.stdout: create 209/33 file after the snapshot 2023-10-30T14:16:30.020 INFO:tasks.workunit.client.0.smithi057.stdout: create 209/34 file after the snapshot 2023-10-30T14:16:30.020 INFO:tasks.workunit.client.0.smithi057.stdout: create 209/35 file after the snapshot 2023-10-30T14:16:30.021 INFO:tasks.workunit.client.0.smithi057.stdout: create 209/36 file after the snapshot 2023-10-30T14:16:30.021 INFO:tasks.workunit.client.0.smithi057.stdout: create 209/37 file after the snapshot 2023-10-30T14:16:30.021 INFO:tasks.workunit.client.0.smithi057.stdout: create 209/38 file after the snapshot 2023-10-30T14:16:30.021 INFO:tasks.workunit.client.0.smithi057.stdout: create 209/39 file after the snapshot 2023-10-30T14:16:30.021 INFO:tasks.workunit.client.0.smithi057.stdout: create 210/.snap/snap-subdir-test 2023-10-30T14:16:30.074 INFO:tasks.workunit.client.0.smithi057.stdout: create 210/30 file after the snapshot 2023-10-30T14:16:30.076 INFO:tasks.workunit.client.0.smithi057.stdout: create 210/31 file after the snapshot 2023-10-30T14:16:30.077 INFO:tasks.workunit.client.0.smithi057.stdout: create 210/32 file after the snapshot 2023-10-30T14:16:30.077 INFO:tasks.workunit.client.0.smithi057.stdout: create 210/33 file after the snapshot 2023-10-30T14:16:30.077 INFO:tasks.workunit.client.0.smithi057.stdout: create 210/34 file after the snapshot 2023-10-30T14:16:30.077 INFO:tasks.workunit.client.0.smithi057.stdout: create 210/35 file after the snapshot 2023-10-30T14:16:30.077 INFO:tasks.workunit.client.0.smithi057.stdout: create 210/36 file after the snapshot 2023-10-30T14:16:30.078 INFO:tasks.workunit.client.0.smithi057.stdout: create 210/37 file after the snapshot 2023-10-30T14:16:30.078 INFO:tasks.workunit.client.0.smithi057.stdout: create 210/38 file after the snapshot 2023-10-30T14:16:30.078 INFO:tasks.workunit.client.0.smithi057.stdout: create 210/39 file after the snapshot 2023-10-30T14:16:30.078 INFO:tasks.workunit.client.0.smithi057.stdout: create 211/.snap/snap-subdir-test 2023-10-30T14:16:30.902 INFO:tasks.workunit.client.0.smithi057.stdout: create 211/30 file after the snapshot 2023-10-30T14:16:30.905 INFO:tasks.workunit.client.0.smithi057.stdout: create 211/31 file after the snapshot 2023-10-30T14:16:30.905 INFO:tasks.workunit.client.0.smithi057.stdout: create 211/32 file after the snapshot 2023-10-30T14:16:30.906 INFO:tasks.workunit.client.0.smithi057.stdout: create 211/33 file after the snapshot 2023-10-30T14:16:30.906 INFO:tasks.workunit.client.0.smithi057.stdout: create 211/34 file after the snapshot 2023-10-30T14:16:30.906 INFO:tasks.workunit.client.0.smithi057.stdout: create 211/35 file after the snapshot 2023-10-30T14:16:30.906 INFO:tasks.workunit.client.0.smithi057.stdout: create 211/36 file after the snapshot 2023-10-30T14:16:30.907 INFO:tasks.workunit.client.0.smithi057.stdout: create 211/37 file after the snapshot 2023-10-30T14:16:30.907 INFO:tasks.workunit.client.0.smithi057.stdout: create 211/38 file after the snapshot 2023-10-30T14:16:30.907 INFO:tasks.workunit.client.0.smithi057.stdout: create 211/39 file after the snapshot 2023-10-30T14:16:30.907 INFO:tasks.workunit.client.0.smithi057.stdout: create 212/.snap/snap-subdir-test 2023-10-30T14:16:30.940 INFO:tasks.workunit.client.0.smithi057.stdout: create 212/30 file after the snapshot 2023-10-30T14:16:30.942 INFO:tasks.workunit.client.0.smithi057.stdout: create 212/31 file after the snapshot 2023-10-30T14:16:30.942 INFO:tasks.workunit.client.0.smithi057.stdout: create 212/32 file after the snapshot 2023-10-30T14:16:30.942 INFO:tasks.workunit.client.0.smithi057.stdout: create 212/33 file after the snapshot 2023-10-30T14:16:30.942 INFO:tasks.workunit.client.0.smithi057.stdout: create 212/34 file after the snapshot 2023-10-30T14:16:30.943 INFO:tasks.workunit.client.0.smithi057.stdout: create 212/35 file after the snapshot 2023-10-30T14:16:30.943 INFO:tasks.workunit.client.0.smithi057.stdout: create 212/36 file after the snapshot 2023-10-30T14:16:30.943 INFO:tasks.workunit.client.0.smithi057.stdout: create 212/37 file after the snapshot 2023-10-30T14:16:30.943 INFO:tasks.workunit.client.0.smithi057.stdout: create 212/38 file after the snapshot 2023-10-30T14:16:30.943 INFO:tasks.workunit.client.0.smithi057.stdout: create 212/39 file after the snapshot 2023-10-30T14:16:30.944 INFO:tasks.workunit.client.0.smithi057.stdout: create 213/.snap/snap-subdir-test 2023-10-30T14:16:30.979 INFO:tasks.workunit.client.0.smithi057.stdout: create 213/30 file after the snapshot 2023-10-30T14:16:30.981 INFO:tasks.workunit.client.0.smithi057.stdout: create 213/31 file after the snapshot 2023-10-30T14:16:30.981 INFO:tasks.workunit.client.0.smithi057.stdout: create 213/32 file after the snapshot 2023-10-30T14:16:30.981 INFO:tasks.workunit.client.0.smithi057.stdout: create 213/33 file after the snapshot 2023-10-30T14:16:30.981 INFO:tasks.workunit.client.0.smithi057.stdout: create 213/34 file after the snapshot 2023-10-30T14:16:30.982 INFO:tasks.workunit.client.0.smithi057.stdout: create 213/35 file after the snapshot 2023-10-30T14:16:30.982 INFO:tasks.workunit.client.0.smithi057.stdout: create 213/36 file after the snapshot 2023-10-30T14:16:30.982 INFO:tasks.workunit.client.0.smithi057.stdout: create 213/37 file after the snapshot 2023-10-30T14:16:30.982 INFO:tasks.workunit.client.0.smithi057.stdout: create 213/38 file after the snapshot 2023-10-30T14:16:30.982 INFO:tasks.workunit.client.0.smithi057.stdout: create 213/39 file after the snapshot 2023-10-30T14:16:30.983 INFO:tasks.workunit.client.0.smithi057.stdout: create 214/.snap/snap-subdir-test 2023-10-30T14:16:31.009 INFO:tasks.workunit.client.0.smithi057.stdout: create 214/30 file after the snapshot 2023-10-30T14:16:31.012 INFO:tasks.workunit.client.0.smithi057.stdout: create 214/31 file after the snapshot 2023-10-30T14:16:31.012 INFO:tasks.workunit.client.0.smithi057.stdout: create 214/32 file after the snapshot 2023-10-30T14:16:31.013 INFO:tasks.workunit.client.0.smithi057.stdout: create 214/33 file after the snapshot 2023-10-30T14:16:31.013 INFO:tasks.workunit.client.0.smithi057.stdout: create 214/34 file after the snapshot 2023-10-30T14:16:31.013 INFO:tasks.workunit.client.0.smithi057.stdout: create 214/35 file after the snapshot 2023-10-30T14:16:31.013 INFO:tasks.workunit.client.0.smithi057.stdout: create 214/36 file after the snapshot 2023-10-30T14:16:31.013 INFO:tasks.workunit.client.0.smithi057.stdout: create 214/37 file after the snapshot 2023-10-30T14:16:31.014 INFO:tasks.workunit.client.0.smithi057.stdout: create 214/38 file after the snapshot 2023-10-30T14:16:31.014 INFO:tasks.workunit.client.0.smithi057.stdout: create 214/39 file after the snapshot 2023-10-30T14:16:31.014 INFO:tasks.workunit.client.0.smithi057.stdout: create 215/.snap/snap-subdir-test 2023-10-30T14:16:31.067 INFO:tasks.workunit.client.0.smithi057.stdout: create 215/30 file after the snapshot 2023-10-30T14:16:31.071 INFO:tasks.workunit.client.0.smithi057.stdout: create 215/31 file after the snapshot 2023-10-30T14:16:31.071 INFO:tasks.workunit.client.0.smithi057.stdout: create 215/32 file after the snapshot 2023-10-30T14:16:31.071 INFO:tasks.workunit.client.0.smithi057.stdout: create 215/33 file after the snapshot 2023-10-30T14:16:31.071 INFO:tasks.workunit.client.0.smithi057.stdout: create 215/34 file after the snapshot 2023-10-30T14:16:31.071 INFO:tasks.workunit.client.0.smithi057.stdout: create 215/35 file after the snapshot 2023-10-30T14:16:31.072 INFO:tasks.workunit.client.0.smithi057.stdout: create 215/36 file after the snapshot 2023-10-30T14:16:31.072 INFO:tasks.workunit.client.0.smithi057.stdout: create 215/37 file after the snapshot 2023-10-30T14:16:31.072 INFO:tasks.workunit.client.0.smithi057.stdout: create 215/38 file after the snapshot 2023-10-30T14:16:31.072 INFO:tasks.workunit.client.0.smithi057.stdout: create 215/39 file after the snapshot 2023-10-30T14:16:31.072 INFO:tasks.workunit.client.0.smithi057.stdout: create 216/.snap/snap-subdir-test 2023-10-30T14:16:31.125 INFO:tasks.workunit.client.0.smithi057.stdout: create 216/30 file after the snapshot 2023-10-30T14:16:31.128 INFO:tasks.workunit.client.0.smithi057.stdout: create 216/31 file after the snapshot 2023-10-30T14:16:31.128 INFO:tasks.workunit.client.0.smithi057.stdout: create 216/32 file after the snapshot 2023-10-30T14:16:31.128 INFO:tasks.workunit.client.0.smithi057.stdout: create 216/33 file after the snapshot 2023-10-30T14:16:31.129 INFO:tasks.workunit.client.0.smithi057.stdout: create 216/34 file after the snapshot 2023-10-30T14:16:31.129 INFO:tasks.workunit.client.0.smithi057.stdout: create 216/35 file after the snapshot 2023-10-30T14:16:31.129 INFO:tasks.workunit.client.0.smithi057.stdout: create 216/36 file after the snapshot 2023-10-30T14:16:31.129 INFO:tasks.workunit.client.0.smithi057.stdout: create 216/37 file after the snapshot 2023-10-30T14:16:31.129 INFO:tasks.workunit.client.0.smithi057.stdout: create 216/38 file after the snapshot 2023-10-30T14:16:31.129 INFO:tasks.workunit.client.0.smithi057.stdout: create 216/39 file after the snapshot 2023-10-30T14:16:31.130 INFO:tasks.workunit.client.0.smithi057.stdout: create 217/.snap/snap-subdir-test 2023-10-30T14:16:31.454 INFO:tasks.workunit.client.0.smithi057.stdout: create 217/30 file after the snapshot 2023-10-30T14:16:31.458 INFO:tasks.workunit.client.0.smithi057.stdout: create 217/31 file after the snapshot 2023-10-30T14:16:31.458 INFO:tasks.workunit.client.0.smithi057.stdout: create 217/32 file after the snapshot 2023-10-30T14:16:31.458 INFO:tasks.workunit.client.0.smithi057.stdout: create 217/33 file after the snapshot 2023-10-30T14:16:31.458 INFO:tasks.workunit.client.0.smithi057.stdout: create 217/34 file after the snapshot 2023-10-30T14:16:31.459 INFO:tasks.workunit.client.0.smithi057.stdout: create 217/35 file after the snapshot 2023-10-30T14:16:31.459 INFO:tasks.workunit.client.0.smithi057.stdout: create 217/36 file after the snapshot 2023-10-30T14:16:31.459 INFO:tasks.workunit.client.0.smithi057.stdout: create 217/37 file after the snapshot 2023-10-30T14:16:31.459 INFO:tasks.workunit.client.0.smithi057.stdout: create 217/38 file after the snapshot 2023-10-30T14:16:31.459 INFO:tasks.workunit.client.0.smithi057.stdout: create 217/39 file after the snapshot 2023-10-30T14:16:31.460 INFO:tasks.workunit.client.0.smithi057.stdout: create 218/.snap/snap-subdir-test 2023-10-30T14:16:31.511 INFO:tasks.workunit.client.0.smithi057.stdout: create 218/30 file after the snapshot 2023-10-30T14:16:31.514 INFO:tasks.workunit.client.0.smithi057.stdout: create 218/31 file after the snapshot 2023-10-30T14:16:31.514 INFO:tasks.workunit.client.0.smithi057.stdout: create 218/32 file after the snapshot 2023-10-30T14:16:31.515 INFO:tasks.workunit.client.0.smithi057.stdout: create 218/33 file after the snapshot 2023-10-30T14:16:31.515 INFO:tasks.workunit.client.0.smithi057.stdout: create 218/34 file after the snapshot 2023-10-30T14:16:31.515 INFO:tasks.workunit.client.0.smithi057.stdout: create 218/35 file after the snapshot 2023-10-30T14:16:31.515 INFO:tasks.workunit.client.0.smithi057.stdout: create 218/36 file after the snapshot 2023-10-30T14:16:31.515 INFO:tasks.workunit.client.0.smithi057.stdout: create 218/37 file after the snapshot 2023-10-30T14:16:31.516 INFO:tasks.workunit.client.0.smithi057.stdout: create 218/38 file after the snapshot 2023-10-30T14:16:31.516 INFO:tasks.workunit.client.0.smithi057.stdout: create 218/39 file after the snapshot 2023-10-30T14:16:31.516 INFO:tasks.workunit.client.0.smithi057.stdout: create 219/.snap/snap-subdir-test 2023-10-30T14:16:31.553 INFO:tasks.workunit.client.0.smithi057.stdout: create 219/30 file after the snapshot 2023-10-30T14:16:31.555 INFO:tasks.workunit.client.0.smithi057.stdout: create 219/31 file after the snapshot 2023-10-30T14:16:31.555 INFO:tasks.workunit.client.0.smithi057.stdout: create 219/32 file after the snapshot 2023-10-30T14:16:31.555 INFO:tasks.workunit.client.0.smithi057.stdout: create 219/33 file after the snapshot 2023-10-30T14:16:31.556 INFO:tasks.workunit.client.0.smithi057.stdout: create 219/34 file after the snapshot 2023-10-30T14:16:31.556 INFO:tasks.workunit.client.0.smithi057.stdout: create 219/35 file after the snapshot 2023-10-30T14:16:31.556 INFO:tasks.workunit.client.0.smithi057.stdout: create 219/36 file after the snapshot 2023-10-30T14:16:31.556 INFO:tasks.workunit.client.0.smithi057.stdout: create 219/37 file after the snapshot 2023-10-30T14:16:31.556 INFO:tasks.workunit.client.0.smithi057.stdout: create 219/38 file after the snapshot 2023-10-30T14:16:31.557 INFO:tasks.workunit.client.0.smithi057.stdout: create 219/39 file after the snapshot 2023-10-30T14:16:31.557 INFO:tasks.workunit.client.0.smithi057.stdout: create 220/.snap/snap-subdir-test 2023-10-30T14:16:31.955 INFO:tasks.workunit.client.0.smithi057.stdout: create 220/30 file after the snapshot 2023-10-30T14:16:31.957 INFO:tasks.workunit.client.0.smithi057.stdout: create 220/31 file after the snapshot 2023-10-30T14:16:31.957 INFO:tasks.workunit.client.0.smithi057.stdout: create 220/32 file after the snapshot 2023-10-30T14:16:31.957 INFO:tasks.workunit.client.0.smithi057.stdout: create 220/33 file after the snapshot 2023-10-30T14:16:31.958 INFO:tasks.workunit.client.0.smithi057.stdout: create 220/34 file after the snapshot 2023-10-30T14:16:31.958 INFO:tasks.workunit.client.0.smithi057.stdout: create 220/35 file after the snapshot 2023-10-30T14:16:31.958 INFO:tasks.workunit.client.0.smithi057.stdout: create 220/36 file after the snapshot 2023-10-30T14:16:31.958 INFO:tasks.workunit.client.0.smithi057.stdout: create 220/37 file after the snapshot 2023-10-30T14:16:31.958 INFO:tasks.workunit.client.0.smithi057.stdout: create 220/38 file after the snapshot 2023-10-30T14:16:31.959 INFO:tasks.workunit.client.0.smithi057.stdout: create 220/39 file after the snapshot 2023-10-30T14:16:31.959 INFO:tasks.workunit.client.0.smithi057.stdout: create 221/.snap/snap-subdir-test 2023-10-30T14:16:31.985 INFO:tasks.workunit.client.0.smithi057.stdout: create 221/30 file after the snapshot 2023-10-30T14:16:31.988 INFO:tasks.workunit.client.0.smithi057.stdout: create 221/31 file after the snapshot 2023-10-30T14:16:31.989 INFO:tasks.workunit.client.0.smithi057.stdout: create 221/32 file after the snapshot 2023-10-30T14:16:31.989 INFO:tasks.workunit.client.0.smithi057.stdout: create 221/33 file after the snapshot 2023-10-30T14:16:31.989 INFO:tasks.workunit.client.0.smithi057.stdout: create 221/34 file after the snapshot 2023-10-30T14:16:31.989 INFO:tasks.workunit.client.0.smithi057.stdout: create 221/35 file after the snapshot 2023-10-30T14:16:31.989 INFO:tasks.workunit.client.0.smithi057.stdout: create 221/36 file after the snapshot 2023-10-30T14:16:31.989 INFO:tasks.workunit.client.0.smithi057.stdout: create 221/37 file after the snapshot 2023-10-30T14:16:31.990 INFO:tasks.workunit.client.0.smithi057.stdout: create 221/38 file after the snapshot 2023-10-30T14:16:31.990 INFO:tasks.workunit.client.0.smithi057.stdout: create 221/39 file after the snapshot 2023-10-30T14:16:31.990 INFO:tasks.workunit.client.0.smithi057.stdout: create 222/.snap/snap-subdir-test 2023-10-30T14:16:32.024 INFO:tasks.workunit.client.0.smithi057.stdout: create 222/30 file after the snapshot 2023-10-30T14:16:32.026 INFO:tasks.workunit.client.0.smithi057.stdout: create 222/31 file after the snapshot 2023-10-30T14:16:32.026 INFO:tasks.workunit.client.0.smithi057.stdout: create 222/32 file after the snapshot 2023-10-30T14:16:32.026 INFO:tasks.workunit.client.0.smithi057.stdout: create 222/33 file after the snapshot 2023-10-30T14:16:32.026 INFO:tasks.workunit.client.0.smithi057.stdout: create 222/34 file after the snapshot 2023-10-30T14:16:32.027 INFO:tasks.workunit.client.0.smithi057.stdout: create 222/35 file after the snapshot 2023-10-30T14:16:32.027 INFO:tasks.workunit.client.0.smithi057.stdout: create 222/36 file after the snapshot 2023-10-30T14:16:32.027 INFO:tasks.workunit.client.0.smithi057.stdout: create 222/37 file after the snapshot 2023-10-30T14:16:32.027 INFO:tasks.workunit.client.0.smithi057.stdout: create 222/38 file after the snapshot 2023-10-30T14:16:32.027 INFO:tasks.workunit.client.0.smithi057.stdout: create 222/39 file after the snapshot 2023-10-30T14:16:32.027 INFO:tasks.workunit.client.0.smithi057.stdout: create 223/.snap/snap-subdir-test 2023-10-30T14:16:32.058 INFO:tasks.workunit.client.0.smithi057.stdout: create 223/30 file after the snapshot 2023-10-30T14:16:32.060 INFO:tasks.workunit.client.0.smithi057.stdout: create 223/31 file after the snapshot 2023-10-30T14:16:32.060 INFO:tasks.workunit.client.0.smithi057.stdout: create 223/32 file after the snapshot 2023-10-30T14:16:32.061 INFO:tasks.workunit.client.0.smithi057.stdout: create 223/33 file after the snapshot 2023-10-30T14:16:32.061 INFO:tasks.workunit.client.0.smithi057.stdout: create 223/34 file after the snapshot 2023-10-30T14:16:32.061 INFO:tasks.workunit.client.0.smithi057.stdout: create 223/35 file after the snapshot 2023-10-30T14:16:32.061 INFO:tasks.workunit.client.0.smithi057.stdout: create 223/36 file after the snapshot 2023-10-30T14:16:32.061 INFO:tasks.workunit.client.0.smithi057.stdout: create 223/37 file after the snapshot 2023-10-30T14:16:32.061 INFO:tasks.workunit.client.0.smithi057.stdout: create 223/38 file after the snapshot 2023-10-30T14:16:32.062 INFO:tasks.workunit.client.0.smithi057.stdout: create 223/39 file after the snapshot 2023-10-30T14:16:32.062 INFO:tasks.workunit.client.0.smithi057.stdout: create 224/.snap/snap-subdir-test 2023-10-30T14:16:32.089 INFO:tasks.workunit.client.0.smithi057.stdout: create 224/30 file after the snapshot 2023-10-30T14:16:32.092 INFO:tasks.workunit.client.0.smithi057.stdout: create 224/31 file after the snapshot 2023-10-30T14:16:32.092 INFO:tasks.workunit.client.0.smithi057.stdout: create 224/32 file after the snapshot 2023-10-30T14:16:32.092 INFO:tasks.workunit.client.0.smithi057.stdout: create 224/33 file after the snapshot 2023-10-30T14:16:32.093 INFO:tasks.workunit.client.0.smithi057.stdout: create 224/34 file after the snapshot 2023-10-30T14:16:32.093 INFO:tasks.workunit.client.0.smithi057.stdout: create 224/35 file after the snapshot 2023-10-30T14:16:32.093 INFO:tasks.workunit.client.0.smithi057.stdout: create 224/36 file after the snapshot 2023-10-30T14:16:32.093 INFO:tasks.workunit.client.0.smithi057.stdout: create 224/37 file after the snapshot 2023-10-30T14:16:32.093 INFO:tasks.workunit.client.0.smithi057.stdout: create 224/38 file after the snapshot 2023-10-30T14:16:32.094 INFO:tasks.workunit.client.0.smithi057.stdout: create 224/39 file after the snapshot 2023-10-30T14:16:32.094 INFO:tasks.workunit.client.0.smithi057.stdout: create 225/.snap/snap-subdir-test 2023-10-30T14:16:32.143 INFO:tasks.workunit.client.0.smithi057.stdout: create 225/30 file after the snapshot 2023-10-30T14:16:32.146 INFO:tasks.workunit.client.0.smithi057.stdout: create 225/31 file after the snapshot 2023-10-30T14:16:32.146 INFO:tasks.workunit.client.0.smithi057.stdout: create 225/32 file after the snapshot 2023-10-30T14:16:32.146 INFO:tasks.workunit.client.0.smithi057.stdout: create 225/33 file after the snapshot 2023-10-30T14:16:32.147 INFO:tasks.workunit.client.0.smithi057.stdout: create 225/34 file after the snapshot 2023-10-30T14:16:32.147 INFO:tasks.workunit.client.0.smithi057.stdout: create 225/35 file after the snapshot 2023-10-30T14:16:32.147 INFO:tasks.workunit.client.0.smithi057.stdout: create 225/36 file after the snapshot 2023-10-30T14:16:32.147 INFO:tasks.workunit.client.0.smithi057.stdout: create 225/37 file after the snapshot 2023-10-30T14:16:32.147 INFO:tasks.workunit.client.0.smithi057.stdout: create 225/38 file after the snapshot 2023-10-30T14:16:32.148 INFO:tasks.workunit.client.0.smithi057.stdout: create 225/39 file after the snapshot 2023-10-30T14:16:32.148 INFO:tasks.workunit.client.0.smithi057.stdout: create 226/.snap/snap-subdir-test 2023-10-30T14:16:32.197 INFO:tasks.workunit.client.0.smithi057.stdout: create 226/30 file after the snapshot 2023-10-30T14:16:32.200 INFO:tasks.workunit.client.0.smithi057.stdout: create 226/31 file after the snapshot 2023-10-30T14:16:32.200 INFO:tasks.workunit.client.0.smithi057.stdout: create 226/32 file after the snapshot 2023-10-30T14:16:32.201 INFO:tasks.workunit.client.0.smithi057.stdout: create 226/33 file after the snapshot 2023-10-30T14:16:32.201 INFO:tasks.workunit.client.0.smithi057.stdout: create 226/34 file after the snapshot 2023-10-30T14:16:32.201 INFO:tasks.workunit.client.0.smithi057.stdout: create 226/35 file after the snapshot 2023-10-30T14:16:32.201 INFO:tasks.workunit.client.0.smithi057.stdout: create 226/36 file after the snapshot 2023-10-30T14:16:32.201 INFO:tasks.workunit.client.0.smithi057.stdout: create 226/37 file after the snapshot 2023-10-30T14:16:32.201 INFO:tasks.workunit.client.0.smithi057.stdout: create 226/38 file after the snapshot 2023-10-30T14:16:32.202 INFO:tasks.workunit.client.0.smithi057.stdout: create 226/39 file after the snapshot 2023-10-30T14:16:32.202 INFO:tasks.workunit.client.0.smithi057.stdout: create 227/.snap/snap-subdir-test 2023-10-30T14:16:32.495 INFO:tasks.workunit.client.0.smithi057.stdout: create 227/30 file after the snapshot 2023-10-30T14:16:32.498 INFO:tasks.workunit.client.0.smithi057.stdout: create 227/31 file after the snapshot 2023-10-30T14:16:32.498 INFO:tasks.workunit.client.0.smithi057.stdout: create 227/32 file after the snapshot 2023-10-30T14:16:32.498 INFO:tasks.workunit.client.0.smithi057.stdout: create 227/33 file after the snapshot 2023-10-30T14:16:32.499 INFO:tasks.workunit.client.0.smithi057.stdout: create 227/34 file after the snapshot 2023-10-30T14:16:32.499 INFO:tasks.workunit.client.0.smithi057.stdout: create 227/35 file after the snapshot 2023-10-30T14:16:32.499 INFO:tasks.workunit.client.0.smithi057.stdout: create 227/36 file after the snapshot 2023-10-30T14:16:32.499 INFO:tasks.workunit.client.0.smithi057.stdout: create 227/37 file after the snapshot 2023-10-30T14:16:32.500 INFO:tasks.workunit.client.0.smithi057.stdout: create 227/38 file after the snapshot 2023-10-30T14:16:32.500 INFO:tasks.workunit.client.0.smithi057.stdout: create 227/39 file after the snapshot 2023-10-30T14:16:32.500 INFO:tasks.workunit.client.0.smithi057.stdout: create 228/.snap/snap-subdir-test 2023-10-30T14:16:32.537 INFO:tasks.workunit.client.0.smithi057.stdout: create 228/30 file after the snapshot 2023-10-30T14:16:32.539 INFO:tasks.workunit.client.0.smithi057.stdout: create 228/31 file after the snapshot 2023-10-30T14:16:32.539 INFO:tasks.workunit.client.0.smithi057.stdout: create 228/32 file after the snapshot 2023-10-30T14:16:32.539 INFO:tasks.workunit.client.0.smithi057.stdout: create 228/33 file after the snapshot 2023-10-30T14:16:32.539 INFO:tasks.workunit.client.0.smithi057.stdout: create 228/34 file after the snapshot 2023-10-30T14:16:32.540 INFO:tasks.workunit.client.0.smithi057.stdout: create 228/35 file after the snapshot 2023-10-30T14:16:32.540 INFO:tasks.workunit.client.0.smithi057.stdout: create 228/36 file after the snapshot 2023-10-30T14:16:32.540 INFO:tasks.workunit.client.0.smithi057.stdout: create 228/37 file after the snapshot 2023-10-30T14:16:32.540 INFO:tasks.workunit.client.0.smithi057.stdout: create 228/38 file after the snapshot 2023-10-30T14:16:32.540 INFO:tasks.workunit.client.0.smithi057.stdout: create 228/39 file after the snapshot 2023-10-30T14:16:32.541 INFO:tasks.workunit.client.0.smithi057.stdout: create 229/.snap/snap-subdir-test 2023-10-30T14:16:32.573 INFO:tasks.workunit.client.0.smithi057.stdout: create 229/30 file after the snapshot 2023-10-30T14:16:32.578 INFO:tasks.workunit.client.0.smithi057.stdout: create 229/31 file after the snapshot 2023-10-30T14:16:32.578 INFO:tasks.workunit.client.0.smithi057.stdout: create 229/32 file after the snapshot 2023-10-30T14:16:32.578 INFO:tasks.workunit.client.0.smithi057.stdout: create 229/33 file after the snapshot 2023-10-30T14:16:32.578 INFO:tasks.workunit.client.0.smithi057.stdout: create 229/34 file after the snapshot 2023-10-30T14:16:32.579 INFO:tasks.workunit.client.0.smithi057.stdout: create 229/35 file after the snapshot 2023-10-30T14:16:32.579 INFO:tasks.workunit.client.0.smithi057.stdout: create 229/36 file after the snapshot 2023-10-30T14:16:32.579 INFO:tasks.workunit.client.0.smithi057.stdout: create 229/37 file after the snapshot 2023-10-30T14:16:32.579 INFO:tasks.workunit.client.0.smithi057.stdout: create 229/38 file after the snapshot 2023-10-30T14:16:32.579 INFO:tasks.workunit.client.0.smithi057.stdout: create 229/39 file after the snapshot 2023-10-30T14:16:32.580 INFO:tasks.workunit.client.0.smithi057.stdout: create 230/.snap/snap-subdir-test 2023-10-30T14:16:32.621 INFO:tasks.workunit.client.0.smithi057.stdout: create 230/30 file after the snapshot 2023-10-30T14:16:32.623 INFO:tasks.workunit.client.0.smithi057.stdout: create 230/31 file after the snapshot 2023-10-30T14:16:32.623 INFO:tasks.workunit.client.0.smithi057.stdout: create 230/32 file after the snapshot 2023-10-30T14:16:32.623 INFO:tasks.workunit.client.0.smithi057.stdout: create 230/33 file after the snapshot 2023-10-30T14:16:32.623 INFO:tasks.workunit.client.0.smithi057.stdout: create 230/34 file after the snapshot 2023-10-30T14:16:32.624 INFO:tasks.workunit.client.0.smithi057.stdout: create 230/35 file after the snapshot 2023-10-30T14:16:32.624 INFO:tasks.workunit.client.0.smithi057.stdout: create 230/36 file after the snapshot 2023-10-30T14:16:32.624 INFO:tasks.workunit.client.0.smithi057.stdout: create 230/37 file after the snapshot 2023-10-30T14:16:32.624 INFO:tasks.workunit.client.0.smithi057.stdout: create 230/38 file after the snapshot 2023-10-30T14:16:32.624 INFO:tasks.workunit.client.0.smithi057.stdout: create 230/39 file after the snapshot 2023-10-30T14:16:32.625 INFO:tasks.workunit.client.0.smithi057.stdout: create 231/.snap/snap-subdir-test 2023-10-30T14:16:32.656 INFO:tasks.workunit.client.0.smithi057.stdout: create 231/30 file after the snapshot 2023-10-30T14:16:32.659 INFO:tasks.workunit.client.0.smithi057.stdout: create 231/31 file after the snapshot 2023-10-30T14:16:32.659 INFO:tasks.workunit.client.0.smithi057.stdout: create 231/32 file after the snapshot 2023-10-30T14:16:32.659 INFO:tasks.workunit.client.0.smithi057.stdout: create 231/33 file after the snapshot 2023-10-30T14:16:32.659 INFO:tasks.workunit.client.0.smithi057.stdout: create 231/34 file after the snapshot 2023-10-30T14:16:32.660 INFO:tasks.workunit.client.0.smithi057.stdout: create 231/35 file after the snapshot 2023-10-30T14:16:32.660 INFO:tasks.workunit.client.0.smithi057.stdout: create 231/36 file after the snapshot 2023-10-30T14:16:32.660 INFO:tasks.workunit.client.0.smithi057.stdout: create 231/37 file after the snapshot 2023-10-30T14:16:32.660 INFO:tasks.workunit.client.0.smithi057.stdout: create 231/38 file after the snapshot 2023-10-30T14:16:32.660 INFO:tasks.workunit.client.0.smithi057.stdout: create 231/39 file after the snapshot 2023-10-30T14:16:32.661 INFO:tasks.workunit.client.0.smithi057.stdout: create 232/.snap/snap-subdir-test 2023-10-30T14:16:32.696 INFO:tasks.workunit.client.0.smithi057.stdout: create 232/30 file after the snapshot 2023-10-30T14:16:32.698 INFO:tasks.workunit.client.0.smithi057.stdout: create 232/31 file after the snapshot 2023-10-30T14:16:32.698 INFO:tasks.workunit.client.0.smithi057.stdout: create 232/32 file after the snapshot 2023-10-30T14:16:32.698 INFO:tasks.workunit.client.0.smithi057.stdout: create 232/33 file after the snapshot 2023-10-30T14:16:32.699 INFO:tasks.workunit.client.0.smithi057.stdout: create 232/34 file after the snapshot 2023-10-30T14:16:32.699 INFO:tasks.workunit.client.0.smithi057.stdout: create 232/35 file after the snapshot 2023-10-30T14:16:32.699 INFO:tasks.workunit.client.0.smithi057.stdout: create 232/36 file after the snapshot 2023-10-30T14:16:32.699 INFO:tasks.workunit.client.0.smithi057.stdout: create 232/37 file after the snapshot 2023-10-30T14:16:32.699 INFO:tasks.workunit.client.0.smithi057.stdout: create 232/38 file after the snapshot 2023-10-30T14:16:32.700 INFO:tasks.workunit.client.0.smithi057.stdout: create 232/39 file after the snapshot 2023-10-30T14:16:32.700 INFO:tasks.workunit.client.0.smithi057.stdout: create 233/.snap/snap-subdir-test 2023-10-30T14:16:33.705 INFO:tasks.workunit.client.0.smithi057.stdout: create 233/30 file after the snapshot 2023-10-30T14:16:33.707 INFO:tasks.workunit.client.0.smithi057.stdout: create 233/31 file after the snapshot 2023-10-30T14:16:33.707 INFO:tasks.workunit.client.0.smithi057.stdout: create 233/32 file after the snapshot 2023-10-30T14:16:33.707 INFO:tasks.workunit.client.0.smithi057.stdout: create 233/33 file after the snapshot 2023-10-30T14:16:33.707 INFO:tasks.workunit.client.0.smithi057.stdout: create 233/34 file after the snapshot 2023-10-30T14:16:33.708 INFO:tasks.workunit.client.0.smithi057.stdout: create 233/35 file after the snapshot 2023-10-30T14:16:33.708 INFO:tasks.workunit.client.0.smithi057.stdout: create 233/36 file after the snapshot 2023-10-30T14:16:33.708 INFO:tasks.workunit.client.0.smithi057.stdout: create 233/37 file after the snapshot 2023-10-30T14:16:33.708 INFO:tasks.workunit.client.0.smithi057.stdout: create 233/38 file after the snapshot 2023-10-30T14:16:33.708 INFO:tasks.workunit.client.0.smithi057.stdout: create 233/39 file after the snapshot 2023-10-30T14:16:33.709 INFO:tasks.workunit.client.0.smithi057.stdout: create 234/.snap/snap-subdir-test 2023-10-30T14:16:33.731 INFO:tasks.workunit.client.0.smithi057.stdout: create 234/30 file after the snapshot 2023-10-30T14:16:33.735 INFO:tasks.workunit.client.0.smithi057.stdout: create 234/31 file after the snapshot 2023-10-30T14:16:33.735 INFO:tasks.workunit.client.0.smithi057.stdout: create 234/32 file after the snapshot 2023-10-30T14:16:33.736 INFO:tasks.workunit.client.0.smithi057.stdout: create 234/33 file after the snapshot 2023-10-30T14:16:33.736 INFO:tasks.workunit.client.0.smithi057.stdout: create 234/34 file after the snapshot 2023-10-30T14:16:33.736 INFO:tasks.workunit.client.0.smithi057.stdout: create 234/35 file after the snapshot 2023-10-30T14:16:33.736 INFO:tasks.workunit.client.0.smithi057.stdout: create 234/36 file after the snapshot 2023-10-30T14:16:33.736 INFO:tasks.workunit.client.0.smithi057.stdout: create 234/37 file after the snapshot 2023-10-30T14:16:33.737 INFO:tasks.workunit.client.0.smithi057.stdout: create 234/38 file after the snapshot 2023-10-30T14:16:33.737 INFO:tasks.workunit.client.0.smithi057.stdout: create 234/39 file after the snapshot 2023-10-30T14:16:33.737 INFO:tasks.workunit.client.0.smithi057.stdout: create 235/.snap/snap-subdir-test 2023-10-30T14:16:33.775 INFO:tasks.workunit.client.0.smithi057.stdout: create 235/30 file after the snapshot 2023-10-30T14:16:33.776 INFO:tasks.workunit.client.0.smithi057.stdout: create 235/31 file after the snapshot 2023-10-30T14:16:33.777 INFO:tasks.workunit.client.0.smithi057.stdout: create 235/32 file after the snapshot 2023-10-30T14:16:33.777 INFO:tasks.workunit.client.0.smithi057.stdout: create 235/33 file after the snapshot 2023-10-30T14:16:33.777 INFO:tasks.workunit.client.0.smithi057.stdout: create 235/34 file after the snapshot 2023-10-30T14:16:33.777 INFO:tasks.workunit.client.0.smithi057.stdout: create 235/35 file after the snapshot 2023-10-30T14:16:33.778 INFO:tasks.workunit.client.0.smithi057.stdout: create 235/36 file after the snapshot 2023-10-30T14:16:33.778 INFO:tasks.workunit.client.0.smithi057.stdout: create 235/37 file after the snapshot 2023-10-30T14:16:33.778 INFO:tasks.workunit.client.0.smithi057.stdout: create 235/38 file after the snapshot 2023-10-30T14:16:33.778 INFO:tasks.workunit.client.0.smithi057.stdout: create 235/39 file after the snapshot 2023-10-30T14:16:33.778 INFO:tasks.workunit.client.0.smithi057.stdout: create 236/.snap/snap-subdir-test 2023-10-30T14:16:33.810 INFO:tasks.workunit.client.0.smithi057.stdout: create 236/30 file after the snapshot 2023-10-30T14:16:33.814 INFO:tasks.workunit.client.0.smithi057.stdout: create 236/31 file after the snapshot 2023-10-30T14:16:33.814 INFO:tasks.workunit.client.0.smithi057.stdout: create 236/32 file after the snapshot 2023-10-30T14:16:33.814 INFO:tasks.workunit.client.0.smithi057.stdout: create 236/33 file after the snapshot 2023-10-30T14:16:33.814 INFO:tasks.workunit.client.0.smithi057.stdout: create 236/34 file after the snapshot 2023-10-30T14:16:33.814 INFO:tasks.workunit.client.0.smithi057.stdout: create 236/35 file after the snapshot 2023-10-30T14:16:33.815 INFO:tasks.workunit.client.0.smithi057.stdout: create 236/36 file after the snapshot 2023-10-30T14:16:33.815 INFO:tasks.workunit.client.0.smithi057.stdout: create 236/37 file after the snapshot 2023-10-30T14:16:33.815 INFO:tasks.workunit.client.0.smithi057.stdout: create 236/38 file after the snapshot 2023-10-30T14:16:33.815 INFO:tasks.workunit.client.0.smithi057.stdout: create 236/39 file after the snapshot 2023-10-30T14:16:33.815 INFO:tasks.workunit.client.0.smithi057.stdout: create 237/.snap/snap-subdir-test 2023-10-30T14:16:33.842 INFO:tasks.workunit.client.0.smithi057.stdout: create 237/30 file after the snapshot 2023-10-30T14:16:33.844 INFO:tasks.workunit.client.0.smithi057.stdout: create 237/31 file after the snapshot 2023-10-30T14:16:33.844 INFO:tasks.workunit.client.0.smithi057.stdout: create 237/32 file after the snapshot 2023-10-30T14:16:33.844 INFO:tasks.workunit.client.0.smithi057.stdout: create 237/33 file after the snapshot 2023-10-30T14:16:33.844 INFO:tasks.workunit.client.0.smithi057.stdout: create 237/34 file after the snapshot 2023-10-30T14:16:33.845 INFO:tasks.workunit.client.0.smithi057.stdout: create 237/35 file after the snapshot 2023-10-30T14:16:33.845 INFO:tasks.workunit.client.0.smithi057.stdout: create 237/36 file after the snapshot 2023-10-30T14:16:33.845 INFO:tasks.workunit.client.0.smithi057.stdout: create 237/37 file after the snapshot 2023-10-30T14:16:33.845 INFO:tasks.workunit.client.0.smithi057.stdout: create 237/38 file after the snapshot 2023-10-30T14:16:33.845 INFO:tasks.workunit.client.0.smithi057.stdout: create 237/39 file after the snapshot 2023-10-30T14:16:33.846 INFO:tasks.workunit.client.0.smithi057.stdout: create 238/.snap/snap-subdir-test 2023-10-30T14:16:33.872 INFO:tasks.workunit.client.0.smithi057.stdout: create 238/30 file after the snapshot 2023-10-30T14:16:33.875 INFO:tasks.workunit.client.0.smithi057.stdout: create 238/31 file after the snapshot 2023-10-30T14:16:33.875 INFO:tasks.workunit.client.0.smithi057.stdout: create 238/32 file after the snapshot 2023-10-30T14:16:33.875 INFO:tasks.workunit.client.0.smithi057.stdout: create 238/33 file after the snapshot 2023-10-30T14:16:33.876 INFO:tasks.workunit.client.0.smithi057.stdout: create 238/34 file after the snapshot 2023-10-30T14:16:33.876 INFO:tasks.workunit.client.0.smithi057.stdout: create 238/35 file after the snapshot 2023-10-30T14:16:33.876 INFO:tasks.workunit.client.0.smithi057.stdout: create 238/36 file after the snapshot 2023-10-30T14:16:33.876 INFO:tasks.workunit.client.0.smithi057.stdout: create 238/37 file after the snapshot 2023-10-30T14:16:33.876 INFO:tasks.workunit.client.0.smithi057.stdout: create 238/38 file after the snapshot 2023-10-30T14:16:33.877 INFO:tasks.workunit.client.0.smithi057.stdout: create 238/39 file after the snapshot 2023-10-30T14:16:33.877 INFO:tasks.workunit.client.0.smithi057.stdout: create 239/.snap/snap-subdir-test 2023-10-30T14:16:33.926 INFO:tasks.workunit.client.0.smithi057.stdout: create 239/30 file after the snapshot 2023-10-30T14:16:33.929 INFO:tasks.workunit.client.0.smithi057.stdout: create 239/31 file after the snapshot 2023-10-30T14:16:33.929 INFO:tasks.workunit.client.0.smithi057.stdout: create 239/32 file after the snapshot 2023-10-30T14:16:33.929 INFO:tasks.workunit.client.0.smithi057.stdout: create 239/33 file after the snapshot 2023-10-30T14:16:33.930 INFO:tasks.workunit.client.0.smithi057.stdout: create 239/34 file after the snapshot 2023-10-30T14:16:33.930 INFO:tasks.workunit.client.0.smithi057.stdout: create 239/35 file after the snapshot 2023-10-30T14:16:33.930 INFO:tasks.workunit.client.0.smithi057.stdout: create 239/36 file after the snapshot 2023-10-30T14:16:33.930 INFO:tasks.workunit.client.0.smithi057.stdout: create 239/37 file after the snapshot 2023-10-30T14:16:33.930 INFO:tasks.workunit.client.0.smithi057.stdout: create 239/38 file after the snapshot 2023-10-30T14:16:33.931 INFO:tasks.workunit.client.0.smithi057.stdout: create 239/39 file after the snapshot 2023-10-30T14:16:33.931 INFO:tasks.workunit.client.0.smithi057.stdout: create 240/.snap/snap-subdir-test 2023-10-30T14:16:33.977 INFO:tasks.workunit.client.0.smithi057.stdout: create 240/30 file after the snapshot 2023-10-30T14:16:33.980 INFO:tasks.workunit.client.0.smithi057.stdout: create 240/31 file after the snapshot 2023-10-30T14:16:33.980 INFO:tasks.workunit.client.0.smithi057.stdout: create 240/32 file after the snapshot 2023-10-30T14:16:33.980 INFO:tasks.workunit.client.0.smithi057.stdout: create 240/33 file after the snapshot 2023-10-30T14:16:33.981 INFO:tasks.workunit.client.0.smithi057.stdout: create 240/34 file after the snapshot 2023-10-30T14:16:33.981 INFO:tasks.workunit.client.0.smithi057.stdout: create 240/35 file after the snapshot 2023-10-30T14:16:33.981 INFO:tasks.workunit.client.0.smithi057.stdout: create 240/36 file after the snapshot 2023-10-30T14:16:33.981 INFO:tasks.workunit.client.0.smithi057.stdout: create 240/37 file after the snapshot 2023-10-30T14:16:33.981 INFO:tasks.workunit.client.0.smithi057.stdout: create 240/38 file after the snapshot 2023-10-30T14:16:33.982 INFO:tasks.workunit.client.0.smithi057.stdout: create 240/39 file after the snapshot 2023-10-30T14:16:33.982 INFO:tasks.workunit.client.0.smithi057.stdout: create 241/.snap/snap-subdir-test 2023-10-30T14:16:34.028 INFO:tasks.workunit.client.0.smithi057.stdout: create 241/30 file after the snapshot 2023-10-30T14:16:34.031 INFO:tasks.workunit.client.0.smithi057.stdout: create 241/31 file after the snapshot 2023-10-30T14:16:34.031 INFO:tasks.workunit.client.0.smithi057.stdout: create 241/32 file after the snapshot 2023-10-30T14:16:34.032 INFO:tasks.workunit.client.0.smithi057.stdout: create 241/33 file after the snapshot 2023-10-30T14:16:34.032 INFO:tasks.workunit.client.0.smithi057.stdout: create 241/34 file after the snapshot 2023-10-30T14:16:34.032 INFO:tasks.workunit.client.0.smithi057.stdout: create 241/35 file after the snapshot 2023-10-30T14:16:34.032 INFO:tasks.workunit.client.0.smithi057.stdout: create 241/36 file after the snapshot 2023-10-30T14:16:34.033 INFO:tasks.workunit.client.0.smithi057.stdout: create 241/37 file after the snapshot 2023-10-30T14:16:34.033 INFO:tasks.workunit.client.0.smithi057.stdout: create 241/38 file after the snapshot 2023-10-30T14:16:34.033 INFO:tasks.workunit.client.0.smithi057.stdout: create 241/39 file after the snapshot 2023-10-30T14:16:34.033 INFO:tasks.workunit.client.0.smithi057.stdout: create 242/.snap/snap-subdir-test 2023-10-30T14:16:34.714 INFO:tasks.workunit.client.0.smithi057.stdout: create 242/30 file after the snapshot 2023-10-30T14:16:34.717 INFO:tasks.workunit.client.0.smithi057.stdout: create 242/31 file after the snapshot 2023-10-30T14:16:34.717 INFO:tasks.workunit.client.0.smithi057.stdout: create 242/32 file after the snapshot 2023-10-30T14:16:34.717 INFO:tasks.workunit.client.0.smithi057.stdout: create 242/33 file after the snapshot 2023-10-30T14:16:34.717 INFO:tasks.workunit.client.0.smithi057.stdout: create 242/34 file after the snapshot 2023-10-30T14:16:34.717 INFO:tasks.workunit.client.0.smithi057.stdout: create 242/35 file after the snapshot 2023-10-30T14:16:34.718 INFO:tasks.workunit.client.0.smithi057.stdout: create 242/36 file after the snapshot 2023-10-30T14:16:34.718 INFO:tasks.workunit.client.0.smithi057.stdout: create 242/37 file after the snapshot 2023-10-30T14:16:34.718 INFO:tasks.workunit.client.0.smithi057.stdout: create 242/38 file after the snapshot 2023-10-30T14:16:34.718 INFO:tasks.workunit.client.0.smithi057.stdout: create 242/39 file after the snapshot 2023-10-30T14:16:34.718 INFO:tasks.workunit.client.0.smithi057.stdout: create 243/.snap/snap-subdir-test 2023-10-30T14:16:34.773 INFO:tasks.workunit.client.0.smithi057.stdout: create 243/30 file after the snapshot 2023-10-30T14:16:34.776 INFO:tasks.workunit.client.0.smithi057.stdout: create 243/31 file after the snapshot 2023-10-30T14:16:34.776 INFO:tasks.workunit.client.0.smithi057.stdout: create 243/32 file after the snapshot 2023-10-30T14:16:34.777 INFO:tasks.workunit.client.0.smithi057.stdout: create 243/33 file after the snapshot 2023-10-30T14:16:34.777 INFO:tasks.workunit.client.0.smithi057.stdout: create 243/34 file after the snapshot 2023-10-30T14:16:34.777 INFO:tasks.workunit.client.0.smithi057.stdout: create 243/35 file after the snapshot 2023-10-30T14:16:34.777 INFO:tasks.workunit.client.0.smithi057.stdout: create 243/36 file after the snapshot 2023-10-30T14:16:34.777 INFO:tasks.workunit.client.0.smithi057.stdout: create 243/37 file after the snapshot 2023-10-30T14:16:34.778 INFO:tasks.workunit.client.0.smithi057.stdout: create 243/38 file after the snapshot 2023-10-30T14:16:34.778 INFO:tasks.workunit.client.0.smithi057.stdout: create 243/39 file after the snapshot 2023-10-30T14:16:34.778 INFO:tasks.workunit.client.0.smithi057.stdout: create 244/.snap/snap-subdir-test 2023-10-30T14:16:34.825 INFO:tasks.workunit.client.0.smithi057.stdout: create 244/30 file after the snapshot 2023-10-30T14:16:34.828 INFO:tasks.workunit.client.0.smithi057.stdout: create 244/31 file after the snapshot 2023-10-30T14:16:34.828 INFO:tasks.workunit.client.0.smithi057.stdout: create 244/32 file after the snapshot 2023-10-30T14:16:34.828 INFO:tasks.workunit.client.0.smithi057.stdout: create 244/33 file after the snapshot 2023-10-30T14:16:34.829 INFO:tasks.workunit.client.0.smithi057.stdout: create 244/34 file after the snapshot 2023-10-30T14:16:34.829 INFO:tasks.workunit.client.0.smithi057.stdout: create 244/35 file after the snapshot 2023-10-30T14:16:34.829 INFO:tasks.workunit.client.0.smithi057.stdout: create 244/36 file after the snapshot 2023-10-30T14:16:34.829 INFO:tasks.workunit.client.0.smithi057.stdout: create 244/37 file after the snapshot 2023-10-30T14:16:34.829 INFO:tasks.workunit.client.0.smithi057.stdout: create 244/38 file after the snapshot 2023-10-30T14:16:34.829 INFO:tasks.workunit.client.0.smithi057.stdout: create 244/39 file after the snapshot 2023-10-30T14:16:34.830 INFO:tasks.workunit.client.0.smithi057.stdout: create 245/.snap/snap-subdir-test 2023-10-30T14:16:34.861 INFO:tasks.workunit.client.0.smithi057.stdout: create 245/30 file after the snapshot 2023-10-30T14:16:34.862 INFO:tasks.workunit.client.0.smithi057.stdout: create 245/31 file after the snapshot 2023-10-30T14:16:34.863 INFO:tasks.workunit.client.0.smithi057.stdout: create 245/32 file after the snapshot 2023-10-30T14:16:34.863 INFO:tasks.workunit.client.0.smithi057.stdout: create 245/33 file after the snapshot 2023-10-30T14:16:34.863 INFO:tasks.workunit.client.0.smithi057.stdout: create 245/34 file after the snapshot 2023-10-30T14:16:34.863 INFO:tasks.workunit.client.0.smithi057.stdout: create 245/35 file after the snapshot 2023-10-30T14:16:34.864 INFO:tasks.workunit.client.0.smithi057.stdout: create 245/36 file after the snapshot 2023-10-30T14:16:34.864 INFO:tasks.workunit.client.0.smithi057.stdout: create 245/37 file after the snapshot 2023-10-30T14:16:34.864 INFO:tasks.workunit.client.0.smithi057.stdout: create 245/38 file after the snapshot 2023-10-30T14:16:34.864 INFO:tasks.workunit.client.0.smithi057.stdout: create 245/39 file after the snapshot 2023-10-30T14:16:34.864 INFO:tasks.workunit.client.0.smithi057.stdout: create 246/.snap/snap-subdir-test 2023-10-30T14:16:35.145 INFO:tasks.workunit.client.0.smithi057.stdout: create 246/30 file after the snapshot 2023-10-30T14:16:35.147 INFO:tasks.workunit.client.0.smithi057.stdout: create 246/31 file after the snapshot 2023-10-30T14:16:35.147 INFO:tasks.workunit.client.0.smithi057.stdout: create 246/32 file after the snapshot 2023-10-30T14:16:35.147 INFO:tasks.workunit.client.0.smithi057.stdout: create 246/33 file after the snapshot 2023-10-30T14:16:35.147 INFO:tasks.workunit.client.0.smithi057.stdout: create 246/34 file after the snapshot 2023-10-30T14:16:35.147 INFO:tasks.workunit.client.0.smithi057.stdout: create 246/35 file after the snapshot 2023-10-30T14:16:35.148 INFO:tasks.workunit.client.0.smithi057.stdout: create 246/36 file after the snapshot 2023-10-30T14:16:35.148 INFO:tasks.workunit.client.0.smithi057.stdout: create 246/37 file after the snapshot 2023-10-30T14:16:35.148 INFO:tasks.workunit.client.0.smithi057.stdout: create 246/38 file after the snapshot 2023-10-30T14:16:35.148 INFO:tasks.workunit.client.0.smithi057.stdout: create 246/39 file after the snapshot 2023-10-30T14:16:35.148 INFO:tasks.workunit.client.0.smithi057.stdout: create 247/.snap/snap-subdir-test 2023-10-30T14:16:35.170 INFO:tasks.workunit.client.0.smithi057.stdout: create 247/30 file after the snapshot 2023-10-30T14:16:35.173 INFO:tasks.workunit.client.0.smithi057.stdout: create 247/31 file after the snapshot 2023-10-30T14:16:35.174 INFO:tasks.workunit.client.0.smithi057.stdout: create 247/32 file after the snapshot 2023-10-30T14:16:35.174 INFO:tasks.workunit.client.0.smithi057.stdout: create 247/33 file after the snapshot 2023-10-30T14:16:35.174 INFO:tasks.workunit.client.0.smithi057.stdout: create 247/34 file after the snapshot 2023-10-30T14:16:35.174 INFO:tasks.workunit.client.0.smithi057.stdout: create 247/35 file after the snapshot 2023-10-30T14:16:35.174 INFO:tasks.workunit.client.0.smithi057.stdout: create 247/36 file after the snapshot 2023-10-30T14:16:35.175 INFO:tasks.workunit.client.0.smithi057.stdout: create 247/37 file after the snapshot 2023-10-30T14:16:35.175 INFO:tasks.workunit.client.0.smithi057.stdout: create 247/38 file after the snapshot 2023-10-30T14:16:35.175 INFO:tasks.workunit.client.0.smithi057.stdout: create 247/39 file after the snapshot 2023-10-30T14:16:35.175 INFO:tasks.workunit.client.0.smithi057.stdout: create 248/.snap/snap-subdir-test 2023-10-30T14:16:35.798 INFO:tasks.workunit.client.0.smithi057.stdout: create 248/30 file after the snapshot 2023-10-30T14:16:35.801 INFO:tasks.workunit.client.0.smithi057.stdout: create 248/31 file after the snapshot 2023-10-30T14:16:35.801 INFO:tasks.workunit.client.0.smithi057.stdout: create 248/32 file after the snapshot 2023-10-30T14:16:35.801 INFO:tasks.workunit.client.0.smithi057.stdout: create 248/33 file after the snapshot 2023-10-30T14:16:35.802 INFO:tasks.workunit.client.0.smithi057.stdout: create 248/34 file after the snapshot 2023-10-30T14:16:35.802 INFO:tasks.workunit.client.0.smithi057.stdout: create 248/35 file after the snapshot 2023-10-30T14:16:35.802 INFO:tasks.workunit.client.0.smithi057.stdout: create 248/36 file after the snapshot 2023-10-30T14:16:35.802 INFO:tasks.workunit.client.0.smithi057.stdout: create 248/37 file after the snapshot 2023-10-30T14:16:35.802 INFO:tasks.workunit.client.0.smithi057.stdout: create 248/38 file after the snapshot 2023-10-30T14:16:35.803 INFO:tasks.workunit.client.0.smithi057.stdout: create 248/39 file after the snapshot 2023-10-30T14:16:35.803 INFO:tasks.workunit.client.0.smithi057.stdout: create 249/.snap/snap-subdir-test 2023-10-30T14:16:35.854 INFO:tasks.workunit.client.0.smithi057.stdout: create 249/30 file after the snapshot 2023-10-30T14:16:35.857 INFO:tasks.workunit.client.0.smithi057.stdout: create 249/31 file after the snapshot 2023-10-30T14:16:35.858 INFO:tasks.workunit.client.0.smithi057.stdout: create 249/32 file after the snapshot 2023-10-30T14:16:35.858 INFO:tasks.workunit.client.0.smithi057.stdout: create 249/33 file after the snapshot 2023-10-30T14:16:35.858 INFO:tasks.workunit.client.0.smithi057.stdout: create 249/34 file after the snapshot 2023-10-30T14:16:35.858 INFO:tasks.workunit.client.0.smithi057.stdout: create 249/35 file after the snapshot 2023-10-30T14:16:35.859 INFO:tasks.workunit.client.0.smithi057.stdout: create 249/36 file after the snapshot 2023-10-30T14:16:35.859 INFO:tasks.workunit.client.0.smithi057.stdout: create 249/37 file after the snapshot 2023-10-30T14:16:35.859 INFO:tasks.workunit.client.0.smithi057.stdout: create 249/38 file after the snapshot 2023-10-30T14:16:35.859 INFO:tasks.workunit.client.0.smithi057.stdout: create 249/39 file after the snapshot 2023-10-30T14:16:35.859 INFO:tasks.workunit.client.0.smithi057.stdout: create 250/.snap/snap-subdir-test 2023-10-30T14:16:35.927 INFO:tasks.workunit.client.0.smithi057.stdout: create 250/30 file after the snapshot 2023-10-30T14:16:35.931 INFO:tasks.workunit.client.0.smithi057.stdout: create 250/31 file after the snapshot 2023-10-30T14:16:35.931 INFO:tasks.workunit.client.0.smithi057.stdout: create 250/32 file after the snapshot 2023-10-30T14:16:35.931 INFO:tasks.workunit.client.0.smithi057.stdout: create 250/33 file after the snapshot 2023-10-30T14:16:35.931 INFO:tasks.workunit.client.0.smithi057.stdout: create 250/34 file after the snapshot 2023-10-30T14:16:35.932 INFO:tasks.workunit.client.0.smithi057.stdout: create 250/35 file after the snapshot 2023-10-30T14:16:35.932 INFO:tasks.workunit.client.0.smithi057.stdout: create 250/36 file after the snapshot 2023-10-30T14:16:35.932 INFO:tasks.workunit.client.0.smithi057.stdout: create 250/37 file after the snapshot 2023-10-30T14:16:35.932 INFO:tasks.workunit.client.0.smithi057.stdout: create 250/38 file after the snapshot 2023-10-30T14:16:35.932 INFO:tasks.workunit.client.0.smithi057.stdout: create 250/39 file after the snapshot 2023-10-30T14:16:35.932 INFO:tasks.workunit.client.0.smithi057.stdout: create 251/.snap/snap-subdir-test 2023-10-30T14:16:35.956 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:35.961 INFO:tasks.workunit.client.0.smithi057.stdout: create 251/30 file after the snapshot 2023-10-30T14:16:35.961 INFO:tasks.workunit.client.0.smithi057.stdout: create 251/31 file after the snapshot 2023-10-30T14:16:35.961 INFO:tasks.workunit.client.0.smithi057.stdout: create 251/32 file after the snapshot 2023-10-30T14:16:35.961 INFO:tasks.workunit.client.0.smithi057.stdout: create 251/33 file after the snapshot 2023-10-30T14:16:35.962 INFO:tasks.workunit.client.0.smithi057.stdout: create 251/34 file after the snapshot 2023-10-30T14:16:35.962 INFO:tasks.workunit.client.0.smithi057.stdout: create 251/35 file after the snapshot 2023-10-30T14:16:35.962 INFO:tasks.workunit.client.0.smithi057.stdout: create 251/36 file after the snapshot 2023-10-30T14:16:35.962 INFO:tasks.workunit.client.0.smithi057.stdout: create 251/37 file after the snapshot 2023-10-30T14:16:35.963 INFO:tasks.workunit.client.0.smithi057.stdout: create 251/38 file after the snapshot 2023-10-30T14:16:35.963 INFO:tasks.workunit.client.0.smithi057.stdout: create 251/39 file after the snapshot 2023-10-30T14:16:35.963 INFO:tasks.workunit.client.0.smithi057.stdout: create 252/.snap/snap-subdir-test 2023-10-30T14:16:35.999 INFO:tasks.workunit.client.0.smithi057.stdout: create 252/30 file after the snapshot 2023-10-30T14:16:36.003 INFO:tasks.workunit.client.0.smithi057.stdout: create 252/31 file after the snapshot 2023-10-30T14:16:36.003 INFO:tasks.workunit.client.0.smithi057.stdout: create 252/32 file after the snapshot 2023-10-30T14:16:36.004 INFO:tasks.workunit.client.0.smithi057.stdout: create 252/33 file after the snapshot 2023-10-30T14:16:36.004 INFO:tasks.workunit.client.0.smithi057.stdout: create 252/34 file after the snapshot 2023-10-30T14:16:36.004 INFO:tasks.workunit.client.0.smithi057.stdout: create 252/35 file after the snapshot 2023-10-30T14:16:36.004 INFO:tasks.workunit.client.0.smithi057.stdout: create 252/36 file after the snapshot 2023-10-30T14:16:36.005 INFO:tasks.workunit.client.0.smithi057.stdout: create 252/37 file after the snapshot 2023-10-30T14:16:36.005 INFO:tasks.workunit.client.0.smithi057.stdout: create 252/38 file after the snapshot 2023-10-30T14:16:36.005 INFO:tasks.workunit.client.0.smithi057.stdout: create 252/39 file after the snapshot 2023-10-30T14:16:36.005 INFO:tasks.workunit.client.0.smithi057.stdout: create 253/.snap/snap-subdir-test 2023-10-30T14:16:36.054 INFO:tasks.workunit.client.0.smithi057.stdout: create 253/30 file after the snapshot 2023-10-30T14:16:36.058 INFO:tasks.workunit.client.0.smithi057.stdout: create 253/31 file after the snapshot 2023-10-30T14:16:36.058 INFO:tasks.workunit.client.0.smithi057.stdout: create 253/32 file after the snapshot 2023-10-30T14:16:36.059 INFO:tasks.workunit.client.0.smithi057.stdout: create 253/33 file after the snapshot 2023-10-30T14:16:36.059 INFO:tasks.workunit.client.0.smithi057.stdout: create 253/34 file after the snapshot 2023-10-30T14:16:36.059 INFO:tasks.workunit.client.0.smithi057.stdout: create 253/35 file after the snapshot 2023-10-30T14:16:36.059 INFO:tasks.workunit.client.0.smithi057.stdout: create 253/36 file after the snapshot 2023-10-30T14:16:36.059 INFO:tasks.workunit.client.0.smithi057.stdout: create 253/37 file after the snapshot 2023-10-30T14:16:36.060 INFO:tasks.workunit.client.0.smithi057.stdout: create 253/38 file after the snapshot 2023-10-30T14:16:36.060 INFO:tasks.workunit.client.0.smithi057.stdout: create 253/39 file after the snapshot 2023-10-30T14:16:36.060 INFO:tasks.workunit.client.0.smithi057.stdout: create 254/.snap/snap-subdir-test 2023-10-30T14:16:36.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.105+0000 7fb4d8572640 1 Processor -- start 2023-10-30T14:16:36.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.105+0000 7fb4d8572640 1 -- start start 2023-10-30T14:16:36.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.105+0000 7fb4d8572640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb4d0153c80 0x7fb4d015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:36.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.105+0000 7fb4d8572640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb4d00c4a10 0x7fb4d00c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:36.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.105+0000 7fb4d8572640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4d015a3b0 0x7fb4d00ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:36.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.105+0000 7fb4d8572640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d015dec0 con 0x7fb4d0153c80 2023-10-30T14:16:36.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.105+0000 7fb4d8572640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d01599a0 con 0x7fb4d00c4a10 2023-10-30T14:16:36.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.105+0000 7fb4d8572640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d0159f20 con 0x7fb4d015a3b0 2023-10-30T14:16:36.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.105+0000 7fb4d630e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb4d0153c80 0x7fb4d015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:36.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.105+0000 7fb4d630e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb4d0153c80 0x7fb4d015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43818/0 (socket says 172.21.15.57:43818) 2023-10-30T14:16:36.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.105+0000 7fb4d630e640 1 -- 172.21.15.57:0/2846626672 learned_addr learned my addr 172.21.15.57:0/2846626672 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:16:36.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.105+0000 7fb4d5b0d640 1 --2- 172.21.15.57:0/2846626672 >> v2:172.21.15.88:3300/0 conn(0x7fb4d00c4a10 0x7fb4d00c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:36.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.105+0000 7fb4d6b0f640 1 --2- 172.21.15.57:0/2846626672 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4d015a3b0 0x7fb4d00ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:36.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.105+0000 7fb4d630e640 1 -- 172.21.15.57:0/2846626672 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4d015a3b0 msgr2=0x7fb4d00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:36.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.105+0000 7fb4d630e640 1 --2- 172.21.15.57:0/2846626672 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4d015a3b0 0x7fb4d00ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.105+0000 7fb4d630e640 1 -- 172.21.15.57:0/2846626672 >> v2:172.21.15.88:3300/0 conn(0x7fb4d00c4a10 msgr2=0x7fb4d00c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:36.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.105+0000 7fb4d630e640 1 --2- 172.21.15.57:0/2846626672 >> v2:172.21.15.88:3300/0 conn(0x7fb4d00c4a10 0x7fb4d00c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.105+0000 7fb4d630e640 1 -- 172.21.15.57:0/2846626672 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d00c47e0 con 0x7fb4d0153c80 2023-10-30T14:16:36.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.105+0000 7fb4d5b0d640 1 --2- 172.21.15.57:0/2846626672 >> v2:172.21.15.88:3300/0 conn(0x7fb4d00c4a10 0x7fb4d00c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:36.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.109+0000 7fb4d630e640 1 --2- 172.21.15.57:0/2846626672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4d0153c80 0x7fb4d015cf90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb4d006be50 tx=0x7fb4cc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b76df765cb657ed4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:36.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.109+0000 7fb4d530c640 1 -- 172.21.15.57:0/2846626672 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc0a89b0 con 0x7fb4d0153c80 2023-10-30T14:16:36.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.109+0000 7fb4d530c640 1 -- 172.21.15.57:0/2846626672 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4cc0a8b10 con 0x7fb4d0153c80 2023-10-30T14:16:36.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.109+0000 7fb4d530c640 1 -- 172.21.15.57:0/2846626672 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc0a8ea0 con 0x7fb4d0153c80 2023-10-30T14:16:36.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.109+0000 7fb4d8572640 1 -- 172.21.15.57:0/2846626672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4d0153c80 msgr2=0x7fb4d015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:36.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.109+0000 7fb4d8572640 1 --2- 172.21.15.57:0/2846626672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4d0153c80 0x7fb4d015cf90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb4d006be50 tx=0x7fb4cc01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.109+0000 7fb4d8572640 1 -- 172.21.15.57:0/2846626672 shutdown_connections 2023-10-30T14:16:36.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.109+0000 7fb4d8572640 1 --2- 172.21.15.57:0/2846626672 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4d015a3b0 0x7fb4d00ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.109+0000 7fb4d8572640 1 --2- 172.21.15.57:0/2846626672 >> v2:172.21.15.88:3300/0 conn(0x7fb4d00c4a10 0x7fb4d00c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.109+0000 7fb4d8572640 1 --2- 172.21.15.57:0/2846626672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4d0153c80 0x7fb4d015cf90 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.109+0000 7fb4d8572640 1 -- 172.21.15.57:0/2846626672 >> 172.21.15.57:0/2846626672 conn(0x7fb4d00715c0 msgr2=0x7fb4d00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:36.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.109+0000 7fb4d8572640 1 -- 172.21.15.57:0/2846626672 shutdown_connections 2023-10-30T14:16:36.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.109+0000 7fb4d8572640 1 -- 172.21.15.57:0/2846626672 wait complete. 2023-10-30T14:16:36.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d8572640 1 Processor -- start 2023-10-30T14:16:36.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d8572640 1 -- start start 2023-10-30T14:16:36.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d8572640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4d00c4a10 0x7fb4d0156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:36.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d8572640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4d0153c80 0x7fb4d00ce280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:36.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d8572640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4d015a3b0 0x7fb4d00ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:36.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d8572640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d00cfe30 con 0x7fb4d00c4a10 2023-10-30T14:16:36.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d8572640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d0150d40 con 0x7fb4d0153c80 2023-10-30T14:16:36.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d8572640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d00cefd0 con 0x7fb4d015a3b0 2023-10-30T14:16:36.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d630e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4d00c4a10 0x7fb4d0156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:36.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d630e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4d00c4a10 0x7fb4d0156d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43822/0 (socket says 172.21.15.57:43822) 2023-10-30T14:16:36.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d630e640 1 -- 172.21.15.57:0/2978421202 learned_addr learned my addr 172.21.15.57:0/2978421202 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:16:36.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d5b0d640 1 --2- 172.21.15.57:0/2978421202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4d0153c80 0x7fb4d00ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:36.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d6b0f640 1 --2- 172.21.15.57:0/2978421202 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4d015a3b0 0x7fb4d00ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:36.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d630e640 1 -- 172.21.15.57:0/2978421202 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4d015a3b0 msgr2=0x7fb4d00ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:36.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d630e640 1 --2- 172.21.15.57:0/2978421202 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4d015a3b0 0x7fb4d00ce7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d630e640 1 -- 172.21.15.57:0/2978421202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4d0153c80 msgr2=0x7fb4d00ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:36.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d630e640 1 --2- 172.21.15.57:0/2978421202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4d0153c80 0x7fb4d00ce280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d630e640 1 -- 172.21.15.57:0/2978421202 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc0094d0 con 0x7fb4d00c4a10 2023-10-30T14:16:36.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d5b0d640 1 --2- 172.21.15.57:0/2978421202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4d0153c80 0x7fb4d00ce280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:36.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d6b0f640 1 --2- 172.21.15.57:0/2978421202 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4d015a3b0 0x7fb4d00ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:36.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d630e640 1 --2- 172.21.15.57:0/2978421202 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4d00c4a10 0x7fb4d0156d50 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc01d3a0 tx=0x7fb4cc01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:36.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4c6ffd640 1 -- 172.21.15.57:0/2978421202 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc0ba600 con 0x7fb4d00c4a10 2023-10-30T14:16:36.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4c6ffd640 1 -- 172.21.15.57:0/2978421202 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4cc0ba760 con 0x7fb4d00c4a10 2023-10-30T14:16:36.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4c6ffd640 1 -- 172.21.15.57:0/2978421202 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc0bab10 con 0x7fb4d00c4a10 2023-10-30T14:16:36.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d8572640 1 -- 172.21.15.57:0/2978421202 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d00cf2b0 con 0x7fb4d00c4a10 2023-10-30T14:16:36.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.113+0000 7fb4d8572640 1 -- 172.21.15.57:0/2978421202 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d00cf870 con 0x7fb4d00c4a10 2023-10-30T14:16:36.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.117+0000 7fb4d8572640 1 -- 172.21.15.57:0/2978421202 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d004e170 con 0x7fb4d00c4a10 2023-10-30T14:16:36.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.117+0000 7fb4c6ffd640 1 -- 172.21.15.57:0/2978421202 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb4cc0bac70 con 0x7fb4d00c4a10 2023-10-30T14:16:36.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.117+0000 7fb4c6ffd640 1 --2- 172.21.15.57:0/2978421202 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4b40664a0 0x7fb4b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:36.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.117+0000 7fb4c6ffd640 1 -- 172.21.15.57:0/2978421202 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fb4cc1226a0 con 0x7fb4d00c4a10 2023-10-30T14:16:36.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.117+0000 7fb4d5b0d640 1 --2- 172.21.15.57:0/2978421202 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4b40664a0 0x7fb4b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:36.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.117+0000 7fb4d5b0d640 1 --2- 172.21.15.57:0/2978421202 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4b40664a0 0x7fb4b4068960 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0073dd0 tx=0x7fb4c0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:36.126 INFO:tasks.workunit.client.0.smithi057.stdout: create 254/30 file after the snapshot 2023-10-30T14:16:36.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.121+0000 7fb4c6ffd640 1 -- 172.21.15.57:0/2978421202 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb4cc0ef900 con 0x7fb4d00c4a10 2023-10-30T14:16:36.128 INFO:tasks.workunit.client.0.smithi057.stdout: create 254/31 file after the snapshot 2023-10-30T14:16:36.128 INFO:tasks.workunit.client.0.smithi057.stdout: create 254/32 file after the snapshot 2023-10-30T14:16:36.129 INFO:tasks.workunit.client.0.smithi057.stdout: create 254/33 file after the snapshot 2023-10-30T14:16:36.129 INFO:tasks.workunit.client.0.smithi057.stdout: create 254/34 file after the snapshot 2023-10-30T14:16:36.129 INFO:tasks.workunit.client.0.smithi057.stdout: create 254/35 file after the snapshot 2023-10-30T14:16:36.129 INFO:tasks.workunit.client.0.smithi057.stdout: create 254/36 file after the snapshot 2023-10-30T14:16:36.130 INFO:tasks.workunit.client.0.smithi057.stdout: create 254/37 file after the snapshot 2023-10-30T14:16:36.130 INFO:tasks.workunit.client.0.smithi057.stdout: create 254/38 file after the snapshot 2023-10-30T14:16:36.130 INFO:tasks.workunit.client.0.smithi057.stdout: create 254/39 file after the snapshot 2023-10-30T14:16:36.130 INFO:tasks.workunit.client.0.smithi057.stdout: create 255/.snap/snap-subdir-test 2023-10-30T14:16:36.174 INFO:tasks.workunit.client.0.smithi057.stdout: create 255/30 file after the snapshot 2023-10-30T14:16:36.177 INFO:tasks.workunit.client.0.smithi057.stdout: create 255/31 file after the snapshot 2023-10-30T14:16:36.177 INFO:tasks.workunit.client.0.smithi057.stdout: create 255/32 file after the snapshot 2023-10-30T14:16:36.177 INFO:tasks.workunit.client.0.smithi057.stdout: create 255/33 file after the snapshot 2023-10-30T14:16:36.177 INFO:tasks.workunit.client.0.smithi057.stdout: create 255/34 file after the snapshot 2023-10-30T14:16:36.177 INFO:tasks.workunit.client.0.smithi057.stdout: create 255/35 file after the snapshot 2023-10-30T14:16:36.178 INFO:tasks.workunit.client.0.smithi057.stdout: create 255/36 file after the snapshot 2023-10-30T14:16:36.178 INFO:tasks.workunit.client.0.smithi057.stdout: create 255/37 file after the snapshot 2023-10-30T14:16:36.178 INFO:tasks.workunit.client.0.smithi057.stdout: create 255/38 file after the snapshot 2023-10-30T14:16:36.178 INFO:tasks.workunit.client.0.smithi057.stdout: create 255/39 file after the snapshot 2023-10-30T14:16:36.178 INFO:tasks.workunit.client.0.smithi057.stdout: create 256/.snap/snap-subdir-test 2023-10-30T14:16:36.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.325+0000 7fb4d8572640 1 -- 172.21.15.57:0/2978421202 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4d00cad40 con 0x7fb4d00c4a10 2023-10-30T14:16:36.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.325+0000 7fb4c6ffd640 1 -- 172.21.15.57:0/2978421202 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6525 (secure 0 0 0) 0x7fb4cc120e60 con 0x7fb4d00c4a10 2023-10-30T14:16:36.329 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:16:36.330 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4936,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:16:19.752531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:36.330 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 20 2023-10-30T14:16:36.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.325+0000 7fb4d8572640 1 -- 172.21.15.57:0/2978421202 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4b40664a0 msgr2=0x7fb4b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:36.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.325+0000 7fb4d8572640 1 --2- 172.21.15.57:0/2978421202 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4b40664a0 0x7fb4b4068960 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0073dd0 tx=0x7fb4c0008040 comp rx=0 tx=0).stop 2023-10-30T14:16:36.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.325+0000 7fb4d8572640 1 -- 172.21.15.57:0/2978421202 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4d00c4a10 msgr2=0x7fb4d0156d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:36.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.325+0000 7fb4d8572640 1 --2- 172.21.15.57:0/2978421202 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4d00c4a10 0x7fb4d0156d50 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc01d3a0 tx=0x7fb4cc01ba90 comp rx=0 tx=0).stop 2023-10-30T14:16:36.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.325+0000 7fb4d8572640 1 -- 172.21.15.57:0/2978421202 shutdown_connections 2023-10-30T14:16:36.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.325+0000 7fb4d8572640 1 --2- 172.21.15.57:0/2978421202 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4b40664a0 0x7fb4b4068960 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.325+0000 7fb4d8572640 1 --2- 172.21.15.57:0/2978421202 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4d015a3b0 0x7fb4d00ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.325+0000 7fb4d8572640 1 --2- 172.21.15.57:0/2978421202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4d0153c80 0x7fb4d00ce280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.325+0000 7fb4d8572640 1 --2- 172.21.15.57:0/2978421202 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4d00c4a10 0x7fb4d0156d50 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.325+0000 7fb4d8572640 1 -- 172.21.15.57:0/2978421202 >> 172.21.15.57:0/2978421202 conn(0x7fb4d00715c0 msgr2=0x7fb4d01513e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:36.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.325+0000 7fb4d8572640 1 -- 172.21.15.57:0/2978421202 shutdown_connections 2023-10-30T14:16:36.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.325+0000 7fb4d8572640 1 -- 172.21.15.57:0/2978421202 wait complete. 2023-10-30T14:16:36.350 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=2) 2023-10-30T14:16:36.351 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T14:16:36.351 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:16:36.407 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T14:16:36.408 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:36.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.589+0000 7fd08baea640 1 Processor -- start 2023-10-30T14:16:36.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.589+0000 7fd08baea640 1 -- start start 2023-10-30T14:16:36.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.589+0000 7fd08baea640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd0840c4a10 0x7fd08415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:36.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.589+0000 7fd08baea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd0840c42a0 0x7fd0840ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:36.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.589+0000 7fd08baea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd084159f20 con 0x7fd084153c80 2023-10-30T14:16:36.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.589+0000 7fd08baea640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd08415dec0 con 0x7fd0840c4a10 2023-10-30T14:16:36.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.589+0000 7fd08baea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0841599a0 con 0x7fd0840c42a0 2023-10-30T14:16:36.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.589+0000 7fd089085640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd0840c4a10 0x7fd08415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:36.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.589+0000 7fd089085640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd0840c4a10 0x7fd08415cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43824/0 (socket says 172.21.15.57:43824) 2023-10-30T14:16:36.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.589+0000 7fd089085640 1 -- 172.21.15.57:0/2812178224 learned_addr learned my addr 172.21.15.57:0/2812178224 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:16:36.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.589+0000 7fd08a087640 1 --2- 172.21.15.57:0/2812178224 >> v2:172.21.15.88:3300/0 conn(0x7fd0840c42a0 0x7fd0840ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:36.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.589+0000 7fd089085640 1 -- 172.21.15.57:0/2812178224 >> v2:172.21.15.88:3300/0 conn(0x7fd0840c42a0 msgr2=0x7fd0840ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:36.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.589+0000 7fd089085640 1 --2- 172.21.15.57:0/2812178224 >> v2:172.21.15.88:3300/0 conn(0x7fd0840c42a0 0x7fd0840ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.589+0000 7fd089085640 1 -- 172.21.15.57:0/2812178224 >> v1:172.21.15.88:6789/0 conn(0x7fd084153c80 legacy=0x7fd08415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:36.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.589+0000 7fd089085640 1 -- 172.21.15.57:0/2812178224 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08415d620 con 0x7fd0840c4a10 2023-10-30T14:16:36.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.589+0000 7fd08a087640 1 --2- 172.21.15.57:0/2812178224 >> v2:172.21.15.88:3300/0 conn(0x7fd0840c42a0 0x7fd0840ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:36.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.593+0000 7fd089085640 1 --2- 172.21.15.57:0/2812178224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0840c4a10 0x7fd08415cf90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd0840b1d40 tx=0x7fd07401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c84de139c2f5ae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:36.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.593+0000 7fd088884640 1 -- 172.21.15.57:0/2812178224 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0740a89f0 con 0x7fd0840c4a10 2023-10-30T14:16:36.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.593+0000 7fd088884640 1 -- 172.21.15.57:0/2812178224 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0740a8b50 con 0x7fd0840c4a10 2023-10-30T14:16:36.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.593+0000 7fd088884640 1 -- 172.21.15.57:0/2812178224 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0740b0440 con 0x7fd0840c4a10 2023-10-30T14:16:36.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.593+0000 7fd08baea640 1 -- 172.21.15.57:0/2812178224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0840c4a10 msgr2=0x7fd08415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:36.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.593+0000 7fd08baea640 1 --2- 172.21.15.57:0/2812178224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0840c4a10 0x7fd08415cf90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd0840b1d40 tx=0x7fd07401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.593+0000 7fd08baea640 1 -- 172.21.15.57:0/2812178224 shutdown_connections 2023-10-30T14:16:36.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.593+0000 7fd08baea640 1 --2- 172.21.15.57:0/2812178224 >> v2:172.21.15.88:3300/0 conn(0x7fd0840c42a0 0x7fd0840ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.593+0000 7fd08baea640 1 --2- 172.21.15.57:0/2812178224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0840c4a10 0x7fd08415cf90 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.593+0000 7fd08baea640 1 -- 172.21.15.57:0/2812178224 >> 172.21.15.57:0/2812178224 conn(0x7fd0840715c0 msgr2=0x7fd0840719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:36.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.593+0000 7fd08baea640 1 -- 172.21.15.57:0/2812178224 shutdown_connections 2023-10-30T14:16:36.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.593+0000 7fd08baea640 1 -- 172.21.15.57:0/2812178224 wait complete. 2023-10-30T14:16:36.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.597+0000 7fd08baea640 1 Processor -- start 2023-10-30T14:16:36.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.597+0000 7fd08baea640 1 -- start start 2023-10-30T14:16:36.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.597+0000 7fd08baea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0840c42a0 0x7fd084154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:36.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.597+0000 7fd08baea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0840c4a10 0x7fd0840cf620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:36.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.597+0000 7fd08baea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd084153c80 0x7fd0840b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:36.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.597+0000 7fd08baea640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08414fd00 con 0x7fd084153c80 2023-10-30T14:16:36.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.597+0000 7fd08baea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08414ffe0 con 0x7fd0840c42a0 2023-10-30T14:16:36.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.597+0000 7fd08baea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0841502c0 con 0x7fd0840c4a10 2023-10-30T14:16:36.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.597+0000 7fd08a087640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd084153c80 0x7fd0840b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:36.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.597+0000 7fd08a087640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd084153c80 0x7fd0840b42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43826/0 (socket says 172.21.15.57:43826) 2023-10-30T14:16:36.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.597+0000 7fd08a087640 1 -- 172.21.15.57:0/1987906654 learned_addr learned my addr 172.21.15.57:0/1987906654 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:16:36.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.597+0000 7fd089085640 1 --2- 172.21.15.57:0/1987906654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0840c4a10 0x7fd0840cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:36.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.597+0000 7fd089886640 1 --2- 172.21.15.57:0/1987906654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0840c42a0 0x7fd084154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:36.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.597+0000 7fd08a087640 1 -- 172.21.15.57:0/1987906654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0840c4a10 msgr2=0x7fd0840cf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:36.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.597+0000 7fd08a087640 1 --2- 172.21.15.57:0/1987906654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0840c4a10 0x7fd0840cf620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.597+0000 7fd08a087640 1 -- 172.21.15.57:0/1987906654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0840c42a0 msgr2=0x7fd084154320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:36.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.597+0000 7fd08a087640 1 --2- 172.21.15.57:0/1987906654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0840c42a0 0x7fd084154320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.597+0000 7fd08a087640 1 -- 172.21.15.57:0/1987906654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0740094d0 con 0x7fd084153c80 2023-10-30T14:16:36.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.597+0000 7fd08a087640 1 --2- 172.21.15.57:0/1987906654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd084153c80 0x7fd0840b42c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd07c007ef0 tx=0x7fd07c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:36.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.601+0000 7fd07a7fc640 1 -- 172.21.15.57:0/1987906654 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c003ba0 con 0x7fd084153c80 2023-10-30T14:16:36.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.601+0000 7fd07a7fc640 1 -- 172.21.15.57:0/1987906654 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07c00f040 con 0x7fd084153c80 2023-10-30T14:16:36.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.601+0000 7fd07a7fc640 1 -- 172.21.15.57:0/1987906654 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c00de80 con 0x7fd084153c80 2023-10-30T14:16:36.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.601+0000 7fd08baea640 1 -- 172.21.15.57:0/1987906654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0840b37e0 con 0x7fd084153c80 2023-10-30T14:16:36.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.601+0000 7fd08baea640 1 -- 172.21.15.57:0/1987906654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0840b3da0 con 0x7fd084153c80 2023-10-30T14:16:36.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.601+0000 7fd08baea640 1 -- 172.21.15.57:0/1987906654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd08404e170 con 0x7fd084153c80 2023-10-30T14:16:36.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.601+0000 7fd07a7fc640 1 -- 172.21.15.57:0/1987906654 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd07c004030 con 0x7fd084153c80 2023-10-30T14:16:36.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.601+0000 7fd07a7fc640 1 --2- 172.21.15.57:0/1987906654 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0680664a0 0x7fd068068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:36.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.601+0000 7fd07a7fc640 1 -- 172.21.15.57:0/1987906654 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fd07c084380 con 0x7fd084153c80 2023-10-30T14:16:36.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.601+0000 7fd089886640 1 --2- 172.21.15.57:0/1987906654 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0680664a0 0x7fd068068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:36.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.605+0000 7fd089886640 1 --2- 172.21.15.57:0/1987906654 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0680664a0 0x7fd068068960 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd080007fd0 tx=0x7fd080009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:36.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.605+0000 7fd07a7fc640 1 -- 172.21.15.57:0/1987906654 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd07c051690 con 0x7fd084153c80 2023-10-30T14:16:36.696 INFO:tasks.workunit.client.0.smithi057.stdout: create 256/30 file after the snapshot 2023-10-30T14:16:36.699 INFO:tasks.workunit.client.0.smithi057.stdout: create 256/31 file after the snapshot 2023-10-30T14:16:36.699 INFO:tasks.workunit.client.0.smithi057.stdout: create 256/32 file after the snapshot 2023-10-30T14:16:36.699 INFO:tasks.workunit.client.0.smithi057.stdout: create 256/33 file after the snapshot 2023-10-30T14:16:36.699 INFO:tasks.workunit.client.0.smithi057.stdout: create 256/34 file after the snapshot 2023-10-30T14:16:36.699 INFO:tasks.workunit.client.0.smithi057.stdout: create 256/35 file after the snapshot 2023-10-30T14:16:36.700 INFO:tasks.workunit.client.0.smithi057.stdout: create 256/36 file after the snapshot 2023-10-30T14:16:36.700 INFO:tasks.workunit.client.0.smithi057.stdout: create 256/37 file after the snapshot 2023-10-30T14:16:36.700 INFO:tasks.workunit.client.0.smithi057.stdout: create 256/38 file after the snapshot 2023-10-30T14:16:36.701 INFO:tasks.workunit.client.0.smithi057.stdout: create 256/39 file after the snapshot 2023-10-30T14:16:36.701 INFO:tasks.workunit.client.0.smithi057.stdout: create 257/.snap/snap-subdir-test 2023-10-30T14:16:36.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.801+0000 7fd08baea640 1 -- 172.21.15.57:0/1987906654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0840cad40 con 0x7fd084153c80 2023-10-30T14:16:36.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.805+0000 7fd07a7fc640 1 -- 172.21.15.57:0/1987906654 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6525 (secure 0 0 0) 0x7fd07c054cb0 con 0x7fd084153c80 2023-10-30T14:16:36.809 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:16:36.809 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4936,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:16:19.752531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:36.810 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 20 2023-10-30T14:16:36.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.805+0000 7fd08baea640 1 -- 172.21.15.57:0/1987906654 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0680664a0 msgr2=0x7fd068068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:36.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.805+0000 7fd08baea640 1 --2- 172.21.15.57:0/1987906654 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0680664a0 0x7fd068068960 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd080007fd0 tx=0x7fd080009040 comp rx=0 tx=0).stop 2023-10-30T14:16:36.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.805+0000 7fd08baea640 1 -- 172.21.15.57:0/1987906654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd084153c80 msgr2=0x7fd0840b42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:36.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.805+0000 7fd08baea640 1 --2- 172.21.15.57:0/1987906654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd084153c80 0x7fd0840b42c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd07c007ef0 tx=0x7fd07c00d4a0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.805+0000 7fd08baea640 1 -- 172.21.15.57:0/1987906654 shutdown_connections 2023-10-30T14:16:36.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.805+0000 7fd08baea640 1 --2- 172.21.15.57:0/1987906654 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0680664a0 0x7fd068068960 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.805+0000 7fd08baea640 1 --2- 172.21.15.57:0/1987906654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd084153c80 0x7fd0840b42c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.805+0000 7fd08baea640 1 --2- 172.21.15.57:0/1987906654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0840c4a10 0x7fd0840cf620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.805+0000 7fd08baea640 1 --2- 172.21.15.57:0/1987906654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0840c42a0 0x7fd084154320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:36.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.805+0000 7fd08baea640 1 -- 172.21.15.57:0/1987906654 >> 172.21.15.57:0/1987906654 conn(0x7fd0840715c0 msgr2=0x7fd084159140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:36.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.805+0000 7fd08baea640 1 -- 172.21.15.57:0/1987906654 shutdown_connections 2023-10-30T14:16:36.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:36.805+0000 7fd08baea640 1 -- 172.21.15.57:0/1987906654 wait complete. 2023-10-30T14:16:36.829 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:16:38.831 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:38.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.985+0000 7f109e1e4640 1 Processor -- start 2023-10-30T14:16:38.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.985+0000 7f109e1e4640 1 -- start start 2023-10-30T14:16:38.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.985+0000 7f109e1e4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f109815ac70 0x7f109815cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:38.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.985+0000 7f109e1e4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f109806aeb0 0x7f109806b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:38.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.985+0000 7f109e1e4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f109815dcc0 con 0x7f10980c8eb0 2023-10-30T14:16:38.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.985+0000 7f109e1e4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1098159800 con 0x7f109815ac70 2023-10-30T14:16:38.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.985+0000 7f109e1e4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1098159d80 con 0x7f109806aeb0 2023-10-30T14:16:38.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.985+0000 7f109c9e1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f109815ac70 0x7f109815cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:38.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.985+0000 7f109d1e2640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f10980c8eb0 0x7f109815a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48358/0 (socket says 172.21.15.57:48358) 2023-10-30T14:16:38.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.985+0000 7f109d1e2640 1 -- 172.21.15.57:0/3920505868 learned_addr learned my addr 172.21.15.57:0/3920505868 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:16:38.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.985+0000 7f109d9e3640 1 --2- 172.21.15.57:0/3920505868 >> v2:172.21.15.88:3300/0 conn(0x7f109806aeb0 0x7f109806b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:38.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.985+0000 7f108ffff640 1 -- 172.21.15.57:0/3920505868 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699655282 0 0) 0x7f1094001120 con 0x7f10980c8eb0 2023-10-30T14:16:38.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.985+0000 7f108ffff640 1 -- 172.21.15.57:0/3920505868 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1084003530 con 0x7f10980c8eb0 2023-10-30T14:16:38.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.985+0000 7f109c9e1640 1 -- 172.21.15.57:0/3920505868 >> v2:172.21.15.88:3300/0 conn(0x7f109806aeb0 msgr2=0x7f109806b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:38.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.985+0000 7f109c9e1640 1 --2- 172.21.15.57:0/3920505868 >> v2:172.21.15.88:3300/0 conn(0x7f109806aeb0 0x7f109806b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:38.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.985+0000 7f109c9e1640 1 -- 172.21.15.57:0/3920505868 >> v1:172.21.15.57:6789/0 conn(0x7f10980c8eb0 legacy=0x7f109815a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:38.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.985+0000 7f109c9e1640 1 -- 172.21.15.57:0/3920505868 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f109815d420 con 0x7f109815ac70 2023-10-30T14:16:38.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.985+0000 7f109d9e3640 1 --2- 172.21.15.57:0/3920505868 >> v2:172.21.15.88:3300/0 conn(0x7f109806aeb0 0x7f109806b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:38.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.989+0000 7f109c9e1640 1 --2- 172.21.15.57:0/3920505868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f109815ac70 0x7f109815cd70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f109808cce0 tx=0x7f108801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd8d150c9c953a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:38.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.989+0000 7f108ffff640 1 -- 172.21.15.57:0/3920505868 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10880a8950 con 0x7f109815ac70 2023-10-30T14:16:38.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.989+0000 7f108ffff640 1 -- 172.21.15.57:0/3920505868 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10880a8ab0 con 0x7f109815ac70 2023-10-30T14:16:38.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.989+0000 7f108ffff640 1 -- 172.21.15.57:0/3920505868 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10880a8e40 con 0x7f109815ac70 2023-10-30T14:16:38.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.989+0000 7f109e1e4640 1 -- 172.21.15.57:0/3920505868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f109815ac70 msgr2=0x7f109815cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:38.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.989+0000 7f109e1e4640 1 --2- 172.21.15.57:0/3920505868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f109815ac70 0x7f109815cd70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f109808cce0 tx=0x7f108801ec50 comp rx=0 tx=0).stop 2023-10-30T14:16:38.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.989+0000 7f109e1e4640 1 -- 172.21.15.57:0/3920505868 shutdown_connections 2023-10-30T14:16:38.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.989+0000 7f109e1e4640 1 --2- 172.21.15.57:0/3920505868 >> v2:172.21.15.88:3300/0 conn(0x7f109806aeb0 0x7f109806b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:38.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.989+0000 7f109e1e4640 1 --2- 172.21.15.57:0/3920505868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f109815ac70 0x7f109815cd70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:38.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.989+0000 7f109e1e4640 1 -- 172.21.15.57:0/3920505868 >> 172.21.15.57:0/3920505868 conn(0x7f10980cef00 msgr2=0x7f10980cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:38.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.989+0000 7f109e1e4640 1 -- 172.21.15.57:0/3920505868 shutdown_connections 2023-10-30T14:16:38.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.989+0000 7f109e1e4640 1 -- 172.21.15.57:0/3920505868 wait complete. 2023-10-30T14:16:38.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109e1e4640 1 Processor -- start 2023-10-30T14:16:38.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109e1e4640 1 -- start start 2023-10-30T14:16:39.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109e1e4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f109806aeb0 0x7f1098154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:39.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109e1e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10980c8eb0 0x7f1098158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:39.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109e1e4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f109815ac70 0x7f10980ce400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:39.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109e1e4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10981ee860 con 0x7f109806aeb0 2023-10-30T14:16:39.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109e1e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f109814e070 con 0x7f10980c8eb0 2023-10-30T14:16:39.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109e1e4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10981eea70 con 0x7f109815ac70 2023-10-30T14:16:39.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109d1e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f109806aeb0 0x7f1098154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:39.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109d1e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f109806aeb0 0x7f1098154710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43836/0 (socket says 172.21.15.57:43836) 2023-10-30T14:16:39.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109d1e2640 1 -- 172.21.15.57:0/2723913693 learned_addr learned my addr 172.21.15.57:0/2723913693 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:16:39.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109c9e1640 1 --2- 172.21.15.57:0/2723913693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10980c8eb0 0x7f1098158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:39.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109d9e3640 1 --2- 172.21.15.57:0/2723913693 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f109815ac70 0x7f10980ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:39.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109d1e2640 1 -- 172.21.15.57:0/2723913693 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f109815ac70 msgr2=0x7f10980ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:39.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109d1e2640 1 --2- 172.21.15.57:0/2723913693 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f109815ac70 0x7f10980ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:39.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109d1e2640 1 -- 172.21.15.57:0/2723913693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10980c8eb0 msgr2=0x7f1098158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:39.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109d1e2640 1 --2- 172.21.15.57:0/2723913693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10980c8eb0 0x7f1098158ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:39.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109d1e2640 1 -- 172.21.15.57:0/2723913693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1088009420 con 0x7f109806aeb0 2023-10-30T14:16:39.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109c9e1640 1 --2- 172.21.15.57:0/2723913693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10980c8eb0 0x7f1098158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:39.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109d1e2640 1 --2- 172.21.15.57:0/2723913693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f109806aeb0 0x7f1098154710 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f109400c4d0 tx=0x7f109400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:39.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f108dffb640 1 -- 172.21.15.57:0/2723913693 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1094005cf0 con 0x7f109806aeb0 2023-10-30T14:16:39.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f108dffb640 1 -- 172.21.15.57:0/2723913693 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1094005e50 con 0x7f109806aeb0 2023-10-30T14:16:39.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f108dffb640 1 -- 172.21.15.57:0/2723913693 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1094007290 con 0x7f109806aeb0 2023-10-30T14:16:39.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109e1e4640 1 -- 172.21.15.57:0/2723913693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10981eed50 con 0x7f109806aeb0 2023-10-30T14:16:39.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.993+0000 7f109e1e4640 1 -- 172.21.15.57:0/2723913693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10981ef310 con 0x7f109806aeb0 2023-10-30T14:16:39.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.997+0000 7f109e1e4640 1 -- 172.21.15.57:0/2723913693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f109804e170 con 0x7f109806aeb0 2023-10-30T14:16:39.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.997+0000 7f108dffb640 1 -- 172.21.15.57:0/2723913693 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f10940073f0 con 0x7f109806aeb0 2023-10-30T14:16:39.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.997+0000 7f108dffb640 1 --2- 172.21.15.57:0/2723913693 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f107c0664a0 0x7f107c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:39.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.997+0000 7f108dffb640 1 -- 172.21.15.57:0/2723913693 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f1094083430 con 0x7f109806aeb0 2023-10-30T14:16:39.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.997+0000 7f109c9e1640 1 --2- 172.21.15.57:0/2723913693 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f107c0664a0 0x7f107c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:39.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:38.997+0000 7f109c9e1640 1 --2- 172.21.15.57:0/2723913693 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f107c0664a0 0x7f107c068960 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f10880b2000 tx=0x7f1088008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:39.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:39.001+0000 7f108dffb640 1 -- 172.21.15.57:0/2723913693 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f10940507b0 con 0x7f109806aeb0 2023-10-30T14:16:39.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:39.193+0000 7f109e1e4640 1 -- 172.21.15.57:0/2723913693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f109815dcc0 con 0x7f109806aeb0 2023-10-30T14:16:39.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:39.193+0000 7f108dffb640 1 -- 172.21.15.57:0/2723913693 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f109815dcc0 con 0x7f109806aeb0 2023-10-30T14:16:39.197 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:16:39.197 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4936,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:16:19.752531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:39.197 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 20 2023-10-30T14:16:39.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:39.193+0000 7f109e1e4640 1 -- 172.21.15.57:0/2723913693 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f107c0664a0 msgr2=0x7f107c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:39.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:39.193+0000 7f109e1e4640 1 --2- 172.21.15.57:0/2723913693 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f107c0664a0 0x7f107c068960 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f10880b2000 tx=0x7f1088008040 comp rx=0 tx=0).stop 2023-10-30T14:16:39.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:39.193+0000 7f109e1e4640 1 -- 172.21.15.57:0/2723913693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f109806aeb0 msgr2=0x7f1098154710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:39.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:39.193+0000 7f109e1e4640 1 --2- 172.21.15.57:0/2723913693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f109806aeb0 0x7f1098154710 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f109400c4d0 tx=0x7f109400c990 comp rx=0 tx=0).stop 2023-10-30T14:16:39.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:39.193+0000 7f109e1e4640 1 -- 172.21.15.57:0/2723913693 shutdown_connections 2023-10-30T14:16:39.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:39.193+0000 7f109e1e4640 1 --2- 172.21.15.57:0/2723913693 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f107c0664a0 0x7f107c068960 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:39.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:39.193+0000 7f109e1e4640 1 --2- 172.21.15.57:0/2723913693 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f109815ac70 0x7f10980ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:39.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:39.193+0000 7f109e1e4640 1 --2- 172.21.15.57:0/2723913693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10980c8eb0 0x7f1098158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:39.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:39.193+0000 7f109e1e4640 1 --2- 172.21.15.57:0/2723913693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f109806aeb0 0x7f1098154710 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:39.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:39.193+0000 7f109e1e4640 1 -- 172.21.15.57:0/2723913693 >> 172.21.15.57:0/2723913693 conn(0x7f10980cef00 msgr2=0x7f10980c7420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:39.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:39.193+0000 7f109e1e4640 1 -- 172.21.15.57:0/2723913693 shutdown_connections 2023-10-30T14:16:39.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:39.193+0000 7f109e1e4640 1 -- 172.21.15.57:0/2723913693 wait complete. 2023-10-30T14:16:39.218 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:16:41.218 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:41.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.369+0000 7feb6326b640 1 Processor -- start 2023-10-30T14:16:41.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.369+0000 7feb6326b640 1 -- start start 2023-10-30T14:16:41.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.369+0000 7feb6326b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7feb5c153c80 0x7feb5c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:41.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.369+0000 7feb6326b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7feb5c0c42a0 0x7feb5c0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:41.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.369+0000 7feb6326b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5c159f20 con 0x7feb5c0c4a10 2023-10-30T14:16:41.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.369+0000 7feb6326b640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb5c15dec0 con 0x7feb5c0c42a0 2023-10-30T14:16:41.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.369+0000 7feb6326b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb5c1599a0 con 0x7feb5c153c80 2023-10-30T14:16:41.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.369+0000 7feb61808640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7feb5c0c42a0 0x7feb5c0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:41.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.369+0000 7feb60806640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7feb5c0c4a10 0x7feb5c15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48368/0 (socket says 172.21.15.57:48368) 2023-10-30T14:16:41.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.369+0000 7feb60806640 1 -- 172.21.15.57:0/520948323 learned_addr learned my addr 172.21.15.57:0/520948323 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:16:41.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.369+0000 7feb61007640 1 --2- 172.21.15.57:0/520948323 >> v2:172.21.15.88:3300/0 conn(0x7feb5c153c80 0x7feb5c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:41.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.369+0000 7feb61808640 1 -- 172.21.15.57:0/520948323 >> v2:172.21.15.88:3300/0 conn(0x7feb5c153c80 msgr2=0x7feb5c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:41.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.369+0000 7feb61808640 1 --2- 172.21.15.57:0/520948323 >> v2:172.21.15.88:3300/0 conn(0x7feb5c153c80 0x7feb5c15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:41.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.369+0000 7feb61808640 1 -- 172.21.15.57:0/520948323 >> v1:172.21.15.57:6789/0 conn(0x7feb5c0c4a10 legacy=0x7feb5c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:41.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.369+0000 7feb61808640 1 -- 172.21.15.57:0/520948323 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5c15d620 con 0x7feb5c0c42a0 2023-10-30T14:16:41.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.369+0000 7feb53fff640 1 -- 172.21.15.57:0/520948323 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279763357 0 0) 0x7feb5c159f20 con 0x7feb5c0c4a10 2023-10-30T14:16:41.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.373+0000 7feb61808640 1 --2- 172.21.15.57:0/520948323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb5c0c42a0 0x7feb5c0ca1e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7feb5c0d00f0 tx=0x7feb5401f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad833b9bcf72ce44 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:41.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.373+0000 7feb53fff640 1 -- 172.21.15.57:0/520948323 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb540b0440 con 0x7feb5c0c42a0 2023-10-30T14:16:41.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.373+0000 7feb53fff640 1 -- 172.21.15.57:0/520948323 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb540b05a0 con 0x7feb5c0c42a0 2023-10-30T14:16:41.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.373+0000 7feb53fff640 1 -- 172.21.15.57:0/520948323 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb540b0930 con 0x7feb5c0c42a0 2023-10-30T14:16:41.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.373+0000 7feb6326b640 1 -- 172.21.15.57:0/520948323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb5c0c42a0 msgr2=0x7feb5c0ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:41.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.373+0000 7feb6326b640 1 --2- 172.21.15.57:0/520948323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb5c0c42a0 0x7feb5c0ca1e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7feb5c0d00f0 tx=0x7feb5401f0f0 comp rx=0 tx=0).stop 2023-10-30T14:16:41.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.373+0000 7feb6326b640 1 -- 172.21.15.57:0/520948323 shutdown_connections 2023-10-30T14:16:41.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.373+0000 7feb6326b640 1 --2- 172.21.15.57:0/520948323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb5c0c42a0 0x7feb5c0ca1e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:41.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.373+0000 7feb6326b640 1 --2- 172.21.15.57:0/520948323 >> v2:172.21.15.88:3300/0 conn(0x7feb5c153c80 0x7feb5c15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:41.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.373+0000 7feb6326b640 1 -- 172.21.15.57:0/520948323 >> 172.21.15.57:0/520948323 conn(0x7feb5c0715c0 msgr2=0x7feb5c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:41.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.373+0000 7feb6326b640 1 -- 172.21.15.57:0/520948323 shutdown_connections 2023-10-30T14:16:41.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.373+0000 7feb6326b640 1 -- 172.21.15.57:0/520948323 wait complete. 2023-10-30T14:16:41.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb6326b640 1 Processor -- start 2023-10-30T14:16:41.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb6326b640 1 -- start start 2023-10-30T14:16:41.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb6326b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb5c0c42a0 0x7feb5c0cf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:41.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb6326b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb5c0c4a10 0x7feb5c0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:41.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb6326b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb5c153c80 0x7feb5c0cd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:41.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb6326b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c0ce360 con 0x7feb5c153c80 2023-10-30T14:16:41.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb6326b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c15d210 con 0x7feb5c0c42a0 2023-10-30T14:16:41.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb6326b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c1efd70 con 0x7feb5c0c4a10 2023-10-30T14:16:41.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb61808640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb5c153c80 0x7feb5c0cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:41.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb61808640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb5c153c80 0x7feb5c0cd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43858/0 (socket says 172.21.15.57:43858) 2023-10-30T14:16:41.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb61808640 1 -- 172.21.15.57:0/3050438447 learned_addr learned my addr 172.21.15.57:0/3050438447 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:16:41.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb61007640 1 --2- 172.21.15.57:0/3050438447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb5c0c42a0 0x7feb5c0cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:41.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb60806640 1 --2- 172.21.15.57:0/3050438447 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb5c0c4a10 0x7feb5c0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:41.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb61808640 1 -- 172.21.15.57:0/3050438447 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb5c0c4a10 msgr2=0x7feb5c0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:41.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb61808640 1 --2- 172.21.15.57:0/3050438447 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb5c0c4a10 0x7feb5c0cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:41.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb61808640 1 -- 172.21.15.57:0/3050438447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb5c0c42a0 msgr2=0x7feb5c0cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:41.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb61808640 1 --2- 172.21.15.57:0/3050438447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb5c0c42a0 0x7feb5c0cf610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:41.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb61808640 1 -- 172.21.15.57:0/3050438447 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb54009a70 con 0x7feb5c153c80 2023-10-30T14:16:41.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb60806640 1 --2- 172.21.15.57:0/3050438447 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb5c0c4a10 0x7feb5c0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:41.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb61808640 1 --2- 172.21.15.57:0/3050438447 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb5c153c80 0x7feb5c0cd680 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7feb540b3040 tx=0x7feb5401be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:41.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb51ffb640 1 -- 172.21.15.57:0/3050438447 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb540b0440 con 0x7feb5c153c80 2023-10-30T14:16:41.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb51ffb640 1 -- 172.21.15.57:0/3050438447 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb540ba820 con 0x7feb5c153c80 2023-10-30T14:16:41.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb51ffb640 1 -- 172.21.15.57:0/3050438447 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb540bac10 con 0x7feb5c153c80 2023-10-30T14:16:41.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb6326b640 1 -- 172.21.15.57:0/3050438447 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb5c15cc50 con 0x7feb5c153c80 2023-10-30T14:16:41.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.377+0000 7feb6326b640 1 -- 172.21.15.57:0/3050438447 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb5c0ce540 con 0x7feb5c153c80 2023-10-30T14:16:41.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.381+0000 7feb6326b640 1 -- 172.21.15.57:0/3050438447 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb5c04e170 con 0x7feb5c153c80 2023-10-30T14:16:41.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.381+0000 7feb51ffb640 1 -- 172.21.15.57:0/3050438447 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feb540b7050 con 0x7feb5c153c80 2023-10-30T14:16:41.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.381+0000 7feb51ffb640 1 --2- 172.21.15.57:0/3050438447 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7feb400664d0 0x7feb40068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:41.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.381+0000 7feb51ffb640 1 -- 172.21.15.57:0/3050438447 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5939+0+0 (secure 0 0 0) 0x7feb541220f0 con 0x7feb5c153c80 2023-10-30T14:16:41.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.381+0000 7feb61007640 1 --2- 172.21.15.57:0/3050438447 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7feb400664d0 0x7feb40068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:41.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.381+0000 7feb61007640 1 --2- 172.21.15.57:0/3050438447 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7feb400664d0 0x7feb40068990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7feb580079c0 tx=0x7feb58008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:41.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.385+0000 7feb51ffb640 1 -- 172.21.15.57:0/3050438447 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feb540ef3d0 con 0x7feb5c153c80 2023-10-30T14:16:41.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.577+0000 7feb6326b640 1 -- 172.21.15.57:0/3050438447 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb5c0cad40 con 0x7feb5c153c80 2023-10-30T14:16:41.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.577+0000 7feb51ffb640 1 -- 172.21.15.57:0/3050438447 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6525 (secure 0 0 0) 0x7feb540b5090 con 0x7feb5c153c80 2023-10-30T14:16:41.581 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:16:41.581 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4936,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:16:19.752531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:41.581 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 20 2023-10-30T14:16:41.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.577+0000 7feb6326b640 1 -- 172.21.15.57:0/3050438447 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7feb400664d0 msgr2=0x7feb40068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:41.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.577+0000 7feb6326b640 1 --2- 172.21.15.57:0/3050438447 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7feb400664d0 0x7feb40068990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7feb580079c0 tx=0x7feb58008040 comp rx=0 tx=0).stop 2023-10-30T14:16:41.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.577+0000 7feb6326b640 1 -- 172.21.15.57:0/3050438447 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb5c153c80 msgr2=0x7feb5c0cd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:41.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.577+0000 7feb6326b640 1 --2- 172.21.15.57:0/3050438447 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb5c153c80 0x7feb5c0cd680 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7feb540b3040 tx=0x7feb5401be90 comp rx=0 tx=0).stop 2023-10-30T14:16:41.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.577+0000 7feb6326b640 1 -- 172.21.15.57:0/3050438447 shutdown_connections 2023-10-30T14:16:41.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.577+0000 7feb6326b640 1 --2- 172.21.15.57:0/3050438447 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7feb400664d0 0x7feb40068990 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:41.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.577+0000 7feb6326b640 1 --2- 172.21.15.57:0/3050438447 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb5c153c80 0x7feb5c0cd680 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:41.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.577+0000 7feb6326b640 1 --2- 172.21.15.57:0/3050438447 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb5c0c4a10 0x7feb5c0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:41.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.577+0000 7feb6326b640 1 --2- 172.21.15.57:0/3050438447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb5c0c42a0 0x7feb5c0cf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:41.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.577+0000 7feb6326b640 1 -- 172.21.15.57:0/3050438447 >> 172.21.15.57:0/3050438447 conn(0x7feb5c0715c0 msgr2=0x7feb5c06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:41.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.577+0000 7feb6326b640 1 -- 172.21.15.57:0/3050438447 shutdown_connections 2023-10-30T14:16:41.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:41.577+0000 7feb6326b640 1 -- 172.21.15.57:0/3050438447 wait complete. 2023-10-30T14:16:41.602 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:16:43.603 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:43.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaf6780640 1 Processor -- start 2023-10-30T14:16:43.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaf6780640 1 -- start start 2023-10-30T14:16:43.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaf6780640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbaf00c4a10 0x7fbaf015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:43.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaf6780640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf0159f20 con 0x7fbaf0153c80 2023-10-30T14:16:43.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaf6780640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf015d620 con 0x7fbaf00c42a0 2023-10-30T14:16:43.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaf6780640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf015dec0 con 0x7fbaf00c4a10 2023-10-30T14:16:43.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fbaf0153c80 0x7fbaf015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:38900/0 (socket says 172.21.15.57:38900) 2023-10-30T14:16:43.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeffff640 1 -- 172.21.15.57:0/3512766609 learned_addr learned my addr 172.21.15.57:0/3512766609 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:16:43.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaef7fe640 1 --2- 172.21.15.57:0/3512766609 >> v2:172.21.15.88:3300/0 conn(0x7fbaf00c4a10 0x7fbaf015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:43.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeeffd640 1 -- 172.21.15.57:0/3512766609 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70675461 0 0) 0x7fbaf0159f20 con 0x7fbaf0153c80 2023-10-30T14:16:43.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeeffd640 1 -- 172.21.15.57:0/3512766609 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad8003530 con 0x7fbaf0153c80 2023-10-30T14:16:43.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeeffd640 1 -- 172.21.15.57:0/3512766609 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 480803231 0 0) 0x7fbad8003530 con 0x7fbaf0153c80 2023-10-30T14:16:43.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeeffd640 1 -- 172.21.15.57:0/3512766609 >> v2:172.21.15.88:3300/0 conn(0x7fbaf00c4a10 msgr2=0x7fbaf015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:43.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeeffd640 1 --2- 172.21.15.57:0/3512766609 >> v2:172.21.15.88:3300/0 conn(0x7fbaf00c4a10 0x7fbaf015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:43.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeeffd640 1 -- 172.21.15.57:0/3512766609 >> v1:172.21.15.88:6789/0 conn(0x7fbaf00c42a0 legacy=0x7fbaf00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:43.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeeffd640 1 -- 172.21.15.57:0/3512766609 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf015d900 con 0x7fbaf0153c80 2023-10-30T14:16:43.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeeffd640 1 -- 172.21.15.57:0/3512766609 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fbae0001680 con 0x7fbaf0153c80 2023-10-30T14:16:43.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeeffd640 1 -- 172.21.15.57:0/3512766609 >> v1:172.21.15.57:6789/0 conn(0x7fbaf0153c80 legacy=0x7fbaf015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:43.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeeffd640 1 --2- 172.21.15.57:0/3512766609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbad8003980 0x7fbad8003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:43.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeeffd640 1 --2- 172.21.15.57:0/3512766609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbad8004380 0x7fbad8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:43.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeeffd640 1 --2- 172.21.15.57:0/3512766609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbad8005050 0x7fbad800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:43.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeeffd640 1 -- 172.21.15.57:0/3512766609 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf015dec0 con 0x7fbad8003980 2023-10-30T14:16:43.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeeffd640 1 -- 172.21.15.57:0/3512766609 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad800bf00 con 0x7fbad8005050 2023-10-30T14:16:43.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeeffd640 1 -- 172.21.15.57:0/3512766609 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad800c0c0 con 0x7fbad8004380 2023-10-30T14:16:43.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeffff640 1 --2- 172.21.15.57:0/3512766609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbad8003980 0x7fbad8003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:43.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaf4d1d640 1 --2- 172.21.15.57:0/3512766609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbad8005050 0x7fbad800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:43.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaef7fe640 1 --2- 172.21.15.57:0/3512766609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbad8004380 0x7fbad8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:43.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeffff640 1 -- 172.21.15.57:0/3512766609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbad8004380 msgr2=0x7fbad8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:43.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeffff640 1 --2- 172.21.15.57:0/3512766609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbad8004380 0x7fbad8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:43.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeffff640 1 -- 172.21.15.57:0/3512766609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbad8005050 msgr2=0x7fbad800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:43.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeffff640 1 --2- 172.21.15.57:0/3512766609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbad8005050 0x7fbad800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:43.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaeffff640 1 -- 172.21.15.57:0/3512766609 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad8003530 con 0x7fbad8003980 2023-10-30T14:16:43.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaf4d1d640 1 -- 172.21.15.57:0/3512766609 reap_dead start 2023-10-30T14:16:43.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.757+0000 7fbaef7fe640 1 --2- 172.21.15.57:0/3512766609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbad8004380 0x7fbad8004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:16:43.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.761+0000 7fbaeffff640 1 --2- 172.21.15.57:0/3512766609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbad8003980 0x7fbad8003d60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbae000c190 tx=0x7fbae001cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec3636ad05aa1970 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:43.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.761+0000 7fbaeeffd640 1 -- 172.21.15.57:0/3512766609 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae0005ae0 con 0x7fbad8003980 2023-10-30T14:16:43.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.761+0000 7fbaeeffd640 1 -- 172.21.15.57:0/3512766609 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae0005c60 con 0x7fbad8003980 2023-10-30T14:16:43.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.761+0000 7fbaeeffd640 1 -- 172.21.15.57:0/3512766609 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae00a5b20 con 0x7fbad8003980 2023-10-30T14:16:43.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.761+0000 7fbaf6780640 1 -- 172.21.15.57:0/3512766609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbad8003980 msgr2=0x7fbad8003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:43.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.761+0000 7fbaf6780640 1 --2- 172.21.15.57:0/3512766609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbad8003980 0x7fbad8003d60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbae000c190 tx=0x7fbae001cbe0 comp rx=0 tx=0).stop 2023-10-30T14:16:43.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.761+0000 7fbaf6780640 1 -- 172.21.15.57:0/3512766609 shutdown_connections 2023-10-30T14:16:43.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.761+0000 7fbaf6780640 1 --2- 172.21.15.57:0/3512766609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbad8003980 0x7fbad8003d60 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:43.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.761+0000 7fbaf6780640 1 -- 172.21.15.57:0/3512766609 >> 172.21.15.57:0/3512766609 conn(0x7fbaf00715c0 msgr2=0x7fbaf00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:43.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.761+0000 7fbaf6780640 1 -- 172.21.15.57:0/3512766609 shutdown_connections 2023-10-30T14:16:43.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.761+0000 7fbaf6780640 1 -- 172.21.15.57:0/3512766609 wait complete. 2023-10-30T14:16:43.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaf6780640 1 Processor -- start 2023-10-30T14:16:43.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaf6780640 1 -- start start 2023-10-30T14:16:43.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaf6780640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbad8003980 0x7fbaf00c9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:43.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaf6780640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaf01f64f0 0x7fbaf01f37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:43.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaf6780640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf00c7740 0x7fbaf00ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:43.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaf6780640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf01586f0 con 0x7fbaf00c7740 2023-10-30T14:16:43.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaf6780640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf01f3360 con 0x7fbaf01f64f0 2023-10-30T14:16:43.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaf6780640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf01f6a30 con 0x7fbad8003980 2023-10-30T14:16:43.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaf4d1d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf00c7740 0x7fbaf00ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:43.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaf4d1d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf00c7740 0x7fbaf00ce1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51060/0 (socket says 172.21.15.57:51060) 2023-10-30T14:16:43.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaf4d1d640 1 -- 172.21.15.57:0/1741447747 learned_addr learned my addr 172.21.15.57:0/1741447747 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:16:43.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaeffff640 1 --2- 172.21.15.57:0/1741447747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbad8003980 0x7fbaf00c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:43.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaef7fe640 1 --2- 172.21.15.57:0/1741447747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaf01f64f0 0x7fbaf01f37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:43.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaf4d1d640 1 -- 172.21.15.57:0/1741447747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbad8003980 msgr2=0x7fbaf00c9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:43.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaf4d1d640 1 --2- 172.21.15.57:0/1741447747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbad8003980 0x7fbaf00c9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:43.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaf4d1d640 1 -- 172.21.15.57:0/1741447747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaf01f64f0 msgr2=0x7fbaf01f37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:43.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaf4d1d640 1 --2- 172.21.15.57:0/1741447747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaf01f64f0 0x7fbaf01f37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:43.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaf4d1d640 1 -- 172.21.15.57:0/1741447747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf01f6d10 con 0x7fbaf00c7740 2023-10-30T14:16:43.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaf4d1d640 1 --2- 172.21.15.57:0/1741447747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf00c7740 0x7fbaf00ce1c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbaf00c97d0 tx=0x7fbae800f540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:43.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaecff9640 1 -- 172.21.15.57:0/1741447747 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae800fe20 con 0x7fbaf00c7740 2023-10-30T14:16:43.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaecff9640 1 -- 172.21.15.57:0/1741447747 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae8015410 con 0x7fbaf00c7740 2023-10-30T14:16:43.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaecff9640 1 -- 172.21.15.57:0/1741447747 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae80157c0 con 0x7fbaf00c7740 2023-10-30T14:16:43.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.765+0000 7fbaf6780640 1 -- 172.21.15.57:0/1741447747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf01f6ff0 con 0x7fbaf00c7740 2023-10-30T14:16:43.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.769+0000 7fbaf6780640 1 -- 172.21.15.57:0/1741447747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf0151700 con 0x7fbaf00c7740 2023-10-30T14:16:43.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.769+0000 7fbaf6780640 1 -- 172.21.15.57:0/1741447747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf004e170 con 0x7fbaf00c7740 2023-10-30T14:16:43.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.769+0000 7fbaecff9640 1 -- 172.21.15.57:0/1741447747 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbae8015920 con 0x7fbaf00c7740 2023-10-30T14:16:43.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.769+0000 7fbaecff9640 1 --2- 172.21.15.57:0/1741447747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbadc0664a0 0x7fbadc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:43.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.769+0000 7fbaecff9640 1 -- 172.21.15.57:0/1741447747 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fbae80872e0 con 0x7fbaf00c7740 2023-10-30T14:16:43.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.769+0000 7fbaeffff640 1 --2- 172.21.15.57:0/1741447747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbadc0664a0 0x7fbadc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:43.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.773+0000 7fbaeffff640 1 --2- 172.21.15.57:0/1741447747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbadc0664a0 0x7fbadc068960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbae00ab000 tx=0x7fbae00b1040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:43.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.773+0000 7fbaecff9640 1 -- 172.21.15.57:0/1741447747 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbae8054710 con 0x7fbaf00c7740 2023-10-30T14:16:43.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.961+0000 7fbaf6780640 1 -- 172.21.15.57:0/1741447747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbaf00713a0 con 0x7fbaf00c7740 2023-10-30T14:16:43.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.965+0000 7fbaecff9640 1 -- 172.21.15.57:0/1741447747 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6525 (secure 0 0 0) 0x7fbae8057d30 con 0x7fbaf00c7740 2023-10-30T14:16:43.969 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:16:43.969 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4936,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:16:19.752531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:43.970 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 20 2023-10-30T14:16:43.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.965+0000 7fbaf6780640 1 -- 172.21.15.57:0/1741447747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbadc0664a0 msgr2=0x7fbadc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:43.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.965+0000 7fbaf6780640 1 --2- 172.21.15.57:0/1741447747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbadc0664a0 0x7fbadc068960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbae00ab000 tx=0x7fbae00b1040 comp rx=0 tx=0).stop 2023-10-30T14:16:43.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.965+0000 7fbaf6780640 1 -- 172.21.15.57:0/1741447747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf00c7740 msgr2=0x7fbaf00ce1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:43.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.965+0000 7fbaf6780640 1 --2- 172.21.15.57:0/1741447747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf00c7740 0x7fbaf00ce1c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbaf00c97d0 tx=0x7fbae800f540 comp rx=0 tx=0).stop 2023-10-30T14:16:43.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.965+0000 7fbaf6780640 1 -- 172.21.15.57:0/1741447747 shutdown_connections 2023-10-30T14:16:43.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.965+0000 7fbaf6780640 1 --2- 172.21.15.57:0/1741447747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbadc0664a0 0x7fbadc068960 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:43.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.965+0000 7fbaf6780640 1 --2- 172.21.15.57:0/1741447747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf00c7740 0x7fbaf00ce1c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:43.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.965+0000 7fbaf6780640 1 --2- 172.21.15.57:0/1741447747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaf01f64f0 0x7fbaf01f37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:43.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.965+0000 7fbaf6780640 1 --2- 172.21.15.57:0/1741447747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbad8003980 0x7fbaf00c9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:43.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.965+0000 7fbaf6780640 1 -- 172.21.15.57:0/1741447747 >> 172.21.15.57:0/1741447747 conn(0x7fbaf00715c0 msgr2=0x7fbaf00b4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:43.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.965+0000 7fbaf6780640 1 -- 172.21.15.57:0/1741447747 shutdown_connections 2023-10-30T14:16:43.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:43.965+0000 7fbaf6780640 1 -- 172.21.15.57:0/1741447747 wait complete. 2023-10-30T14:16:43.990 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:16:45.991 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:46.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.141+0000 7fca3752a640 1 Processor -- start 2023-10-30T14:16:46.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.141+0000 7fca3752a640 1 -- start start 2023-10-30T14:16:46.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.141+0000 7fca3752a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca300c4a10 0x7fca3015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:46.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.141+0000 7fca3752a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca30159f20 con 0x7fca30153c80 2023-10-30T14:16:46.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.141+0000 7fca3752a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3015d620 con 0x7fca300c42a0 2023-10-30T14:16:46.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.141+0000 7fca3752a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3015dec0 con 0x7fca300c4a10 2023-10-30T14:16:46.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.141+0000 7fca352c6640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fca30153c80 0x7fca3015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:38912/0 (socket says 172.21.15.57:38912) 2023-10-30T14:16:46.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.141+0000 7fca352c6640 1 -- 172.21.15.57:0/2439529807 learned_addr learned my addr 172.21.15.57:0/2439529807 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:16:46.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.141+0000 7fca34ac5640 1 --2- 172.21.15.57:0/2439529807 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca300c4a10 0x7fca3015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:46.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.141+0000 7fca27fff640 1 -- 172.21.15.57:0/2439529807 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839667178 0 0) 0x7fca30159f20 con 0x7fca30153c80 2023-10-30T14:16:46.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.141+0000 7fca27fff640 1 -- 172.21.15.57:0/2439529807 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca1c003530 con 0x7fca30153c80 2023-10-30T14:16:46.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca27fff640 1 -- 172.21.15.57:0/2439529807 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888501320 0 0) 0x7fca3015d620 con 0x7fca300c42a0 2023-10-30T14:16:46.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca27fff640 1 -- 172.21.15.57:0/2439529807 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca30159f20 con 0x7fca300c42a0 2023-10-30T14:16:46.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca27fff640 1 -- 172.21.15.57:0/2439529807 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1936844865 0 0) 0x7fca1c003530 con 0x7fca30153c80 2023-10-30T14:16:46.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca27fff640 1 -- 172.21.15.57:0/2439529807 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca300c4a10 msgr2=0x7fca3015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:46.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca27fff640 1 --2- 172.21.15.57:0/2439529807 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca300c4a10 0x7fca3015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:46.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca27fff640 1 -- 172.21.15.57:0/2439529807 >> v1:172.21.15.88:6789/0 conn(0x7fca300c42a0 legacy=0x7fca300ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:46.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca27fff640 1 -- 172.21.15.57:0/2439529807 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3015d900 con 0x7fca30153c80 2023-10-30T14:16:46.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca27fff640 1 -- 172.21.15.57:0/2439529807 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fca2c001680 con 0x7fca30153c80 2023-10-30T14:16:46.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca34ac5640 1 --2- 172.21.15.57:0/2439529807 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca300c4a10 0x7fca3015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:16:46.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca27fff640 1 -- 172.21.15.57:0/2439529807 >> v1:172.21.15.57:6789/0 conn(0x7fca30153c80 legacy=0x7fca3015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:46.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca27fff640 1 --2- 172.21.15.57:0/2439529807 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca300c4a10 0x7fca1c003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:46.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca27fff640 1 --2- 172.21.15.57:0/2439529807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca1c004080 0x7fca1c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:46.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca27fff640 1 --2- 172.21.15.57:0/2439529807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca1c004d10 0x7fca1c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:46.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca27fff640 1 -- 172.21.15.57:0/2439529807 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3015dec0 con 0x7fca1c004d10 2023-10-30T14:16:46.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca27fff640 1 -- 172.21.15.57:0/2439529807 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c00ba40 con 0x7fca1c004080 2023-10-30T14:16:46.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca27fff640 1 -- 172.21.15.57:0/2439529807 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c00bc00 con 0x7fca300c4a10 2023-10-30T14:16:46.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca35ac7640 1 --2- 172.21.15.57:0/2439529807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca1c004d10 0x7fca1c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:46.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca34ac5640 1 --2- 172.21.15.57:0/2439529807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca1c004080 0x7fca1c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:46.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca352c6640 1 --2- 172.21.15.57:0/2439529807 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca300c4a10 0x7fca1c003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:46.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca35ac7640 1 -- 172.21.15.57:0/2439529807 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca300c4a10 msgr2=0x7fca1c003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:46.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca35ac7640 1 --2- 172.21.15.57:0/2439529807 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca300c4a10 0x7fca1c003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:46.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca35ac7640 1 -- 172.21.15.57:0/2439529807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca1c004080 msgr2=0x7fca1c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:46.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca35ac7640 1 --2- 172.21.15.57:0/2439529807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca1c004080 0x7fca1c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:46.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca35ac7640 1 -- 172.21.15.57:0/2439529807 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca20009420 con 0x7fca1c004d10 2023-10-30T14:16:46.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca352c6640 1 --2- 172.21.15.57:0/2439529807 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca300c4a10 0x7fca1c003b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:46.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca35ac7640 1 --2- 172.21.15.57:0/2439529807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca1c004d10 0x7fca1c005190 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fca28001fd0 tx=0x7fca2801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6407b76edd501a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:46.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca27fff640 1 -- 172.21.15.57:0/2439529807 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca28003b80 con 0x7fca1c004d10 2023-10-30T14:16:46.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca27fff640 1 -- 172.21.15.57:0/2439529807 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca28003d00 con 0x7fca1c004d10 2023-10-30T14:16:46.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.145+0000 7fca27fff640 1 -- 172.21.15.57:0/2439529807 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca28005310 con 0x7fca1c004d10 2023-10-30T14:16:46.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca3752a640 1 -- 172.21.15.57:0/2439529807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca1c004d10 msgr2=0x7fca1c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:46.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca3752a640 1 --2- 172.21.15.57:0/2439529807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca1c004d10 0x7fca1c005190 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fca28001fd0 tx=0x7fca2801c8e0 comp rx=0 tx=0).stop 2023-10-30T14:16:46.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca35ac7640 1 -- 172.21.15.57:0/2439529807 reap_dead start 2023-10-30T14:16:46.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca3752a640 1 -- 172.21.15.57:0/2439529807 shutdown_connections 2023-10-30T14:16:46.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca3752a640 1 -- 172.21.15.57:0/2439529807 >> 172.21.15.57:0/2439529807 conn(0x7fca300715c0 msgr2=0x7fca300719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:46.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca3752a640 1 -- 172.21.15.57:0/2439529807 shutdown_connections 2023-10-30T14:16:46.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca3752a640 1 -- 172.21.15.57:0/2439529807 wait complete. 2023-10-30T14:16:46.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca3752a640 1 Processor -- start 2023-10-30T14:16:46.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca3752a640 1 -- start start 2023-10-30T14:16:46.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca3752a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca1c004d10 0x7fca301f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:46.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca3752a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca3015ab60 0x7fca301ef1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:46.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca3752a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca300c8d40 0x7fca300c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:46.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca3752a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca300b1b80 con 0x7fca3015ab60 2023-10-30T14:16:46.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca3752a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca301542f0 con 0x7fca300c8d40 2023-10-30T14:16:46.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca3752a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca301545d0 con 0x7fca1c004d10 2023-10-30T14:16:46.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca34ac5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca3015ab60 0x7fca301ef1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:46.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca34ac5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca3015ab60 0x7fca301ef1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51078/0 (socket says 172.21.15.57:51078) 2023-10-30T14:16:46.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca34ac5640 1 -- 172.21.15.57:0/2768085583 learned_addr learned my addr 172.21.15.57:0/2768085583 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:16:46.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca352c6640 1 --2- 172.21.15.57:0/2768085583 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca1c004d10 0x7fca301f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:46.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.149+0000 7fca35ac7640 1 --2- 172.21.15.57:0/2768085583 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca300c8d40 0x7fca300c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:46.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.153+0000 7fca34ac5640 1 -- 172.21.15.57:0/2768085583 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca1c004d10 msgr2=0x7fca301f2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:46.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.153+0000 7fca34ac5640 1 --2- 172.21.15.57:0/2768085583 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca1c004d10 0x7fca301f2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:46.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.153+0000 7fca34ac5640 1 -- 172.21.15.57:0/2768085583 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca300c8d40 msgr2=0x7fca300c91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:46.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.153+0000 7fca34ac5640 1 --2- 172.21.15.57:0/2768085583 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca300c8d40 0x7fca300c91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:46.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.153+0000 7fca34ac5640 1 -- 172.21.15.57:0/2768085583 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca301548b0 con 0x7fca3015ab60 2023-10-30T14:16:46.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.153+0000 7fca34ac5640 1 --2- 172.21.15.57:0/2768085583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca3015ab60 0x7fca301ef1c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fca20011a50 tx=0x7fca20011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:46.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.153+0000 7fca25ffb640 1 -- 172.21.15.57:0/2768085583 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2000fc50 con 0x7fca3015ab60 2023-10-30T14:16:46.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.153+0000 7fca25ffb640 1 -- 172.21.15.57:0/2768085583 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2000fdb0 con 0x7fca3015ab60 2023-10-30T14:16:46.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.153+0000 7fca3752a640 1 -- 172.21.15.57:0/2768085583 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca30154b90 con 0x7fca3015ab60 2023-10-30T14:16:46.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.153+0000 7fca25ffb640 1 -- 172.21.15.57:0/2768085583 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca20014680 con 0x7fca3015ab60 2023-10-30T14:16:46.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.153+0000 7fca3752a640 1 -- 172.21.15.57:0/2768085583 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca301ee970 con 0x7fca3015ab60 2023-10-30T14:16:46.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.153+0000 7fca3752a640 1 -- 172.21.15.57:0/2768085583 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca3004e170 con 0x7fca3015ab60 2023-10-30T14:16:46.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.153+0000 7fca25ffb640 1 -- 172.21.15.57:0/2768085583 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fca200147e0 con 0x7fca3015ab60 2023-10-30T14:16:46.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.153+0000 7fca25ffb640 1 --2- 172.21.15.57:0/2768085583 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fca140664d0 0x7fca14068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:46.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.153+0000 7fca25ffb640 1 -- 172.21.15.57:0/2768085583 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fca200849c0 con 0x7fca3015ab60 2023-10-30T14:16:46.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.157+0000 7fca352c6640 1 --2- 172.21.15.57:0/2768085583 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fca140664d0 0x7fca14068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:46.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.157+0000 7fca352c6640 1 --2- 172.21.15.57:0/2768085583 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fca140664d0 0x7fca14068990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fca2c000fe0 tx=0x7fca2c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:46.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.157+0000 7fca25ffb640 1 -- 172.21.15.57:0/2768085583 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fca20051cb0 con 0x7fca3015ab60 2023-10-30T14:16:46.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.353+0000 7fca3752a640 1 -- 172.21.15.57:0/2768085583 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca300cad40 con 0x7fca3015ab60 2023-10-30T14:16:46.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.353+0000 7fca25ffb640 1 -- 172.21.15.57:0/2768085583 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6525 (secure 0 0 0) 0x7fca200552d0 con 0x7fca3015ab60 2023-10-30T14:16:46.357 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:16:46.358 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4936,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:16:19.752531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:46.358 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 20 2023-10-30T14:16:46.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.353+0000 7fca3752a640 1 -- 172.21.15.57:0/2768085583 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fca140664d0 msgr2=0x7fca14068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:46.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.353+0000 7fca3752a640 1 --2- 172.21.15.57:0/2768085583 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fca140664d0 0x7fca14068990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fca2c000fe0 tx=0x7fca2c009040 comp rx=0 tx=0).stop 2023-10-30T14:16:46.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.353+0000 7fca3752a640 1 -- 172.21.15.57:0/2768085583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca3015ab60 msgr2=0x7fca301ef1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:46.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.353+0000 7fca3752a640 1 --2- 172.21.15.57:0/2768085583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca3015ab60 0x7fca301ef1c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fca20011a50 tx=0x7fca20011f10 comp rx=0 tx=0).stop 2023-10-30T14:16:46.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.353+0000 7fca3752a640 1 -- 172.21.15.57:0/2768085583 shutdown_connections 2023-10-30T14:16:46.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.353+0000 7fca3752a640 1 --2- 172.21.15.57:0/2768085583 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fca140664d0 0x7fca14068990 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:46.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.353+0000 7fca3752a640 1 --2- 172.21.15.57:0/2768085583 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca300c8d40 0x7fca300c91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:46.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.353+0000 7fca3752a640 1 --2- 172.21.15.57:0/2768085583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca3015ab60 0x7fca301ef1c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:46.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.353+0000 7fca3752a640 1 --2- 172.21.15.57:0/2768085583 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca1c004d10 0x7fca301f2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:46.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.353+0000 7fca3752a640 1 -- 172.21.15.57:0/2768085583 >> 172.21.15.57:0/2768085583 conn(0x7fca300715c0 msgr2=0x7fca300719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:46.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.353+0000 7fca3752a640 1 -- 172.21.15.57:0/2768085583 shutdown_connections 2023-10-30T14:16:46.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:46.353+0000 7fca3752a640 1 -- 172.21.15.57:0/2768085583 wait complete. 2023-10-30T14:16:46.379 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:16:48.380 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:48.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.525+0000 7f96c8965640 1 Processor -- start 2023-10-30T14:16:48.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.525+0000 7f96c8965640 1 -- start start 2023-10-30T14:16:48.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.525+0000 7f96c8965640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f96c4153c80 0x7f96c415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:48.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.525+0000 7f96c8965640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96c40c42a0 0x7f96c40ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:48.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.525+0000 7f96c8965640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c4159f20 con 0x7f96c40c4a10 2023-10-30T14:16:48.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.525+0000 7f96c8965640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96c415dec0 con 0x7f96c4153c80 2023-10-30T14:16:48.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.525+0000 7f96c8965640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c41599a0 con 0x7f96c40c42a0 2023-10-30T14:16:48.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.525+0000 7f96c259c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f96c4153c80 0x7f96c415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:48.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.525+0000 7f96bbfff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f96c40c4a10 0x7f96c415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:38920/0 (socket says 172.21.15.57:38920) 2023-10-30T14:16:48.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.525+0000 7f96bbfff640 1 -- 172.21.15.57:0/429859200 learned_addr learned my addr 172.21.15.57:0/429859200 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:16:48.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.525+0000 7f96c2d9d640 1 --2- 172.21.15.57:0/429859200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96c40c42a0 0x7f96c40ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:48.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.525+0000 7f96c259c640 1 -- 172.21.15.57:0/429859200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96c40c42a0 msgr2=0x7f96c40ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:48.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.525+0000 7f96c259c640 1 --2- 172.21.15.57:0/429859200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96c40c42a0 0x7f96c40ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:48.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.525+0000 7f96c259c640 1 -- 172.21.15.57:0/429859200 >> v1:172.21.15.57:6789/0 conn(0x7f96c40c4a10 legacy=0x7f96c415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:48.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.525+0000 7f96c259c640 1 -- 172.21.15.57:0/429859200 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c415d620 con 0x7f96c4153c80 2023-10-30T14:16:48.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.525+0000 7f96c1d9b640 1 -- 172.21.15.57:0/429859200 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279552397 0 0) 0x7f96c4159f20 con 0x7f96c40c4a10 2023-10-30T14:16:48.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.529+0000 7f96c259c640 1 --2- 172.21.15.57:0/429859200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96c4153c80 0x7f96c415cf90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f96b40097c0 tx=0x7f96b401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c05ac1ac0b75844 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:48.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.529+0000 7f96c1d9b640 1 -- 172.21.15.57:0/429859200 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b40a8970 con 0x7f96c4153c80 2023-10-30T14:16:48.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.529+0000 7f96c1d9b640 1 -- 172.21.15.57:0/429859200 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b40a8ad0 con 0x7f96c4153c80 2023-10-30T14:16:48.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.529+0000 7f96c1d9b640 1 -- 172.21.15.57:0/429859200 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b40a8e60 con 0x7f96c4153c80 2023-10-30T14:16:48.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.529+0000 7f96c8965640 1 -- 172.21.15.57:0/429859200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96c4153c80 msgr2=0x7f96c415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:48.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.529+0000 7f96c8965640 1 --2- 172.21.15.57:0/429859200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96c4153c80 0x7f96c415cf90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f96b40097c0 tx=0x7f96b401ec70 comp rx=0 tx=0).stop 2023-10-30T14:16:48.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.529+0000 7f96c8965640 1 -- 172.21.15.57:0/429859200 shutdown_connections 2023-10-30T14:16:48.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.529+0000 7f96c8965640 1 --2- 172.21.15.57:0/429859200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96c40c42a0 0x7f96c40ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:48.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.529+0000 7f96c8965640 1 --2- 172.21.15.57:0/429859200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96c4153c80 0x7f96c415cf90 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:48.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.529+0000 7f96c8965640 1 -- 172.21.15.57:0/429859200 >> 172.21.15.57:0/429859200 conn(0x7f96c40715c0 msgr2=0x7f96c40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:48.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.529+0000 7f96c8965640 1 -- 172.21.15.57:0/429859200 shutdown_connections 2023-10-30T14:16:48.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.529+0000 7f96c8965640 1 -- 172.21.15.57:0/429859200 wait complete. 2023-10-30T14:16:48.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96c8965640 1 Processor -- start 2023-10-30T14:16:48.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96c8965640 1 -- start start 2023-10-30T14:16:48.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96c8965640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96c40c42a0 0x7f96c4154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:48.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96c8965640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96c40c4a10 0x7f96c4154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:48.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96c8965640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96c4153c80 0x7f96c4155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:48.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96c8965640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c4156620 con 0x7f96c40c4a10 2023-10-30T14:16:48.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96c8965640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c40ceac0 con 0x7f96c4153c80 2023-10-30T14:16:48.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96c8965640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c4154850 con 0x7f96c40c42a0 2023-10-30T14:16:48.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96bbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96c40c4a10 0x7f96c4154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:48.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96bbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96c40c4a10 0x7f96c4154ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51088/0 (socket says 172.21.15.57:51088) 2023-10-30T14:16:48.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96bbfff640 1 -- 172.21.15.57:0/1725049097 learned_addr learned my addr 172.21.15.57:0/1725049097 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:16:48.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96c259c640 1 --2- 172.21.15.57:0/1725049097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96c40c42a0 0x7f96c4154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:48.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96c2d9d640 1 --2- 172.21.15.57:0/1725049097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96c4153c80 0x7f96c4155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:48.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96bbfff640 1 -- 172.21.15.57:0/1725049097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96c40c42a0 msgr2=0x7f96c4154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:48.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96bbfff640 1 --2- 172.21.15.57:0/1725049097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96c40c42a0 0x7f96c4154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:48.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96bbfff640 1 -- 172.21.15.57:0/1725049097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96c4153c80 msgr2=0x7f96c4155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:48.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96bbfff640 1 --2- 172.21.15.57:0/1725049097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96c4153c80 0x7f96c4155220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:48.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96bbfff640 1 -- 172.21.15.57:0/1725049097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b4009420 con 0x7f96c40c4a10 2023-10-30T14:16:48.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96bbfff640 1 --2- 172.21.15.57:0/1725049097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96c40c4a10 0x7f96c4154ce0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f96ac00c4e0 tx=0x7f96ac00c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:48.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96baffd640 1 -- 172.21.15.57:0/1725049097 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac0074e0 con 0x7f96c40c4a10 2023-10-30T14:16:48.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96baffd640 1 -- 172.21.15.57:0/1725049097 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ac007660 con 0x7f96c40c4a10 2023-10-30T14:16:48.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96baffd640 1 -- 172.21.15.57:0/1725049097 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac004a30 con 0x7f96c40c4a10 2023-10-30T14:16:48.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96c8965640 1 -- 172.21.15.57:0/1725049097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c41f29f0 con 0x7f96c40c4a10 2023-10-30T14:16:48.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96c8965640 1 -- 172.21.15.57:0/1725049097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c41f2fa0 con 0x7f96c40c4a10 2023-10-30T14:16:48.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.533+0000 7f96c8965640 1 -- 172.21.15.57:0/1725049097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c404e170 con 0x7f96c40c4a10 2023-10-30T14:16:48.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.537+0000 7f96baffd640 1 -- 172.21.15.57:0/1725049097 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f96ac004b90 con 0x7f96c40c4a10 2023-10-30T14:16:48.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.537+0000 7f96baffd640 1 --2- 172.21.15.57:0/1725049097 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f96a40664a0 0x7f96a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:48.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.537+0000 7f96baffd640 1 -- 172.21.15.57:0/1725049097 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f96ac083260 con 0x7f96c40c4a10 2023-10-30T14:16:48.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.537+0000 7f96c259c640 1 --2- 172.21.15.57:0/1725049097 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f96a40664a0 0x7f96a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:48.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.537+0000 7f96c259c640 1 --2- 172.21.15.57:0/1725049097 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f96a40664a0 0x7f96a4068960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f96b40b2000 tx=0x7f96b4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:48.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.541+0000 7f96baffd640 1 -- 172.21.15.57:0/1725049097 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f96ac050630 con 0x7f96c40c4a10 2023-10-30T14:16:48.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.733+0000 7f96c8965640 1 -- 172.21.15.57:0/1725049097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96c414dae0 con 0x7f96c40c4a10 2023-10-30T14:16:48.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.733+0000 7f96baffd640 1 -- 172.21.15.57:0/1725049097 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f96ac018020 con 0x7f96c40c4a10 2023-10-30T14:16:48.737 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:16:48.738 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4936,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:16:19.752531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6834/3701655528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3701655528},{"type":"v1","addr":"172.21.15.57:6834","nonce":3701655528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:48.738 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 20 2023-10-30T14:16:48.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.733+0000 7f96c8965640 1 -- 172.21.15.57:0/1725049097 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f96a40664a0 msgr2=0x7f96a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:48.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.733+0000 7f96c8965640 1 --2- 172.21.15.57:0/1725049097 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f96a40664a0 0x7f96a4068960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f96b40b2000 tx=0x7f96b4008040 comp rx=0 tx=0).stop 2023-10-30T14:16:48.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.733+0000 7f96c8965640 1 -- 172.21.15.57:0/1725049097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96c40c4a10 msgr2=0x7f96c4154ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:48.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.733+0000 7f96c8965640 1 --2- 172.21.15.57:0/1725049097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96c40c4a10 0x7f96c4154ce0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f96ac00c4e0 tx=0x7f96ac00c9a0 comp rx=0 tx=0).stop 2023-10-30T14:16:48.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.733+0000 7f96c8965640 1 -- 172.21.15.57:0/1725049097 shutdown_connections 2023-10-30T14:16:48.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.733+0000 7f96c8965640 1 --2- 172.21.15.57:0/1725049097 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f96a40664a0 0x7f96a4068960 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:48.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.733+0000 7f96c8965640 1 --2- 172.21.15.57:0/1725049097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96c4153c80 0x7f96c4155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:48.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.733+0000 7f96c8965640 1 --2- 172.21.15.57:0/1725049097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96c40c4a10 0x7f96c4154ce0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:48.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.733+0000 7f96c8965640 1 --2- 172.21.15.57:0/1725049097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96c40c42a0 0x7f96c4154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:48.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.733+0000 7f96c8965640 1 -- 172.21.15.57:0/1725049097 >> 172.21.15.57:0/1725049097 conn(0x7f96c40715c0 msgr2=0x7f96c4158450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:48.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.733+0000 7f96c8965640 1 -- 172.21.15.57:0/1725049097 shutdown_connections 2023-10-30T14:16:48.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:48.733+0000 7f96c8965640 1 -- 172.21.15.57:0/1725049097 wait complete. 2023-10-30T14:16:48.759 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:16:49.715 INFO:tasks.workunit.client.0.smithi057.stdout: create 257/30 file after the snapshot 2023-10-30T14:16:49.719 INFO:tasks.workunit.client.0.smithi057.stdout: create 257/31 file after the snapshot 2023-10-30T14:16:49.719 INFO:tasks.workunit.client.0.smithi057.stdout: create 257/32 file after the snapshot 2023-10-30T14:16:49.719 INFO:tasks.workunit.client.0.smithi057.stdout: create 257/33 file after the snapshot 2023-10-30T14:16:49.719 INFO:tasks.workunit.client.0.smithi057.stdout: create 257/34 file after the snapshot 2023-10-30T14:16:49.720 INFO:tasks.workunit.client.0.smithi057.stdout: create 257/35 file after the snapshot 2023-10-30T14:16:49.720 INFO:tasks.workunit.client.0.smithi057.stdout: create 257/36 file after the snapshot 2023-10-30T14:16:49.720 INFO:tasks.workunit.client.0.smithi057.stdout: create 257/37 file after the snapshot 2023-10-30T14:16:49.720 INFO:tasks.workunit.client.0.smithi057.stdout: create 257/38 file after the snapshot 2023-10-30T14:16:49.720 INFO:tasks.workunit.client.0.smithi057.stdout: create 257/39 file after the snapshot 2023-10-30T14:16:49.720 INFO:tasks.workunit.client.0.smithi057.stdout: create 258/.snap/snap-subdir-test 2023-10-30T14:16:49.772 INFO:tasks.workunit.client.0.smithi057.stdout: create 258/30 file after the snapshot 2023-10-30T14:16:49.775 INFO:tasks.workunit.client.0.smithi057.stdout: create 258/31 file after the snapshot 2023-10-30T14:16:49.775 INFO:tasks.workunit.client.0.smithi057.stdout: create 258/32 file after the snapshot 2023-10-30T14:16:49.775 INFO:tasks.workunit.client.0.smithi057.stdout: create 258/33 file after the snapshot 2023-10-30T14:16:49.775 INFO:tasks.workunit.client.0.smithi057.stdout: create 258/34 file after the snapshot 2023-10-30T14:16:49.776 INFO:tasks.workunit.client.0.smithi057.stdout: create 258/35 file after the snapshot 2023-10-30T14:16:49.776 INFO:tasks.workunit.client.0.smithi057.stdout: create 258/36 file after the snapshot 2023-10-30T14:16:49.777 INFO:tasks.workunit.client.0.smithi057.stdout: create 258/37 file after the snapshot 2023-10-30T14:16:49.777 INFO:tasks.workunit.client.0.smithi057.stdout: create 258/38 file after the snapshot 2023-10-30T14:16:49.777 INFO:tasks.workunit.client.0.smithi057.stdout: create 258/39 file after the snapshot 2023-10-30T14:16:49.777 INFO:tasks.workunit.client.0.smithi057.stdout: create 259/.snap/snap-subdir-test 2023-10-30T14:16:49.838 INFO:tasks.workunit.client.0.smithi057.stdout: create 259/30 file after the snapshot 2023-10-30T14:16:49.841 INFO:tasks.workunit.client.0.smithi057.stdout: create 259/31 file after the snapshot 2023-10-30T14:16:49.841 INFO:tasks.workunit.client.0.smithi057.stdout: create 259/32 file after the snapshot 2023-10-30T14:16:49.841 INFO:tasks.workunit.client.0.smithi057.stdout: create 259/33 file after the snapshot 2023-10-30T14:16:49.842 INFO:tasks.workunit.client.0.smithi057.stdout: create 259/34 file after the snapshot 2023-10-30T14:16:49.842 INFO:tasks.workunit.client.0.smithi057.stdout: create 259/35 file after the snapshot 2023-10-30T14:16:49.842 INFO:tasks.workunit.client.0.smithi057.stdout: create 259/36 file after the snapshot 2023-10-30T14:16:49.842 INFO:tasks.workunit.client.0.smithi057.stdout: create 259/37 file after the snapshot 2023-10-30T14:16:49.843 INFO:tasks.workunit.client.0.smithi057.stdout: create 259/38 file after the snapshot 2023-10-30T14:16:49.843 INFO:tasks.workunit.client.0.smithi057.stdout: create 259/39 file after the snapshot 2023-10-30T14:16:49.843 INFO:tasks.workunit.client.0.smithi057.stdout: create 260/.snap/snap-subdir-test 2023-10-30T14:16:50.759 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:50.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.901+0000 7fa8f6980640 1 Processor -- start 2023-10-30T14:16:50.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.901+0000 7fa8f6980640 1 -- start start 2023-10-30T14:16:50.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.901+0000 7fa8f6980640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f014fc50 0x7fa8f01616d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:50.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.901+0000 7fa8f6980640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f00c55f0 con 0x7fa8f006fdf0 2023-10-30T14:16:50.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.901+0000 7fa8f6980640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f01567b0 con 0x7fa8f00c6320 2023-10-30T14:16:50.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.901+0000 7fa8f6980640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f00c5f00 con 0x7fa8f014fc50 2023-10-30T14:16:50.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.901+0000 7fa8ef7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa8f006fdf0 0x7fa8f0155c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:38932/0 (socket says 172.21.15.57:38932) 2023-10-30T14:16:50.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.901+0000 7fa8ef7fe640 1 -- 172.21.15.57:0/3002199841 learned_addr learned my addr 172.21.15.57:0/3002199841 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:16:50.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8f4f1d640 1 --2- 172.21.15.57:0/3002199841 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f014fc50 0x7fa8f01616d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:50.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8eeffd640 1 -- 172.21.15.57:0/3002199841 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876428789 0 0) 0x7fa8f00c55f0 con 0x7fa8f006fdf0 2023-10-30T14:16:50.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8eeffd640 1 -- 172.21.15.57:0/3002199841 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8d8005740 con 0x7fa8f006fdf0 2023-10-30T14:16:50.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8eeffd640 1 -- 172.21.15.57:0/3002199841 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4072741024 0 0) 0x7fa8d8005740 con 0x7fa8f006fdf0 2023-10-30T14:16:50.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8eeffd640 1 -- 172.21.15.57:0/3002199841 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f014fc50 msgr2=0x7fa8f01616d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:50.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8eeffd640 1 --2- 172.21.15.57:0/3002199841 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f014fc50 0x7fa8f01616d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:50.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8eeffd640 1 -- 172.21.15.57:0/3002199841 >> v1:172.21.15.88:6789/0 conn(0x7fa8f00c6320 legacy=0x7fa8f0070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:50.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8eeffd640 1 -- 172.21.15.57:0/3002199841 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f01500e0 con 0x7fa8f006fdf0 2023-10-30T14:16:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8eeffd640 1 -- 172.21.15.57:0/3002199841 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fa8e4001680 con 0x7fa8f006fdf0 2023-10-30T14:16:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8eeffd640 1 -- 172.21.15.57:0/3002199841 >> v1:172.21.15.57:6789/0 conn(0x7fa8f006fdf0 legacy=0x7fa8f0155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8eeffd640 1 --2- 172.21.15.57:0/3002199841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8d8005bc0 0x7fa8d8005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8eeffd640 1 --2- 172.21.15.57:0/3002199841 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f014fc50 0x7fa8d80069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8eeffd640 1 --2- 172.21.15.57:0/3002199841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8d8006f20 0x7fa8d80073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8eeffd640 1 -- 172.21.15.57:0/3002199841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f00c5f00 con 0x7fa8d8005bc0 2023-10-30T14:16:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8eeffd640 1 -- 172.21.15.57:0/3002199841 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d800dc50 con 0x7fa8d8006f20 2023-10-30T14:16:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8eeffd640 1 -- 172.21.15.57:0/3002199841 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d800de10 con 0x7fa8f014fc50 2023-10-30T14:16:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8effff640 1 --2- 172.21.15.57:0/3002199841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8d8005bc0 0x7fa8d8005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8ef7fe640 1 --2- 172.21.15.57:0/3002199841 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f014fc50 0x7fa8d80069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8f4f1d640 1 --2- 172.21.15.57:0/3002199841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8d8006f20 0x7fa8d80073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8effff640 1 -- 172.21.15.57:0/3002199841 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f014fc50 msgr2=0x7fa8d80069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8effff640 1 --2- 172.21.15.57:0/3002199841 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f014fc50 0x7fa8d80069e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8effff640 1 -- 172.21.15.57:0/3002199841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8d8006f20 msgr2=0x7fa8d80073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8effff640 1 --2- 172.21.15.57:0/3002199841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8d8006f20 0x7fa8d80073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8effff640 1 -- 172.21.15.57:0/3002199841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d8005740 con 0x7fa8d8005bc0 2023-10-30T14:16:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.905+0000 7fa8ef7fe640 1 --2- 172.21.15.57:0/3002199841 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f014fc50 0x7fa8d80069e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:50.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8effff640 1 --2- 172.21.15.57:0/3002199841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8d8005bc0 0x7fa8d8005fa0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0001910 tx=0x7fa8e001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=990598339882c1e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:50.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8eeffd640 1 -- 172.21.15.57:0/3002199841 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e00ab600 con 0x7fa8d8005bc0 2023-10-30T14:16:50.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8eeffd640 1 -- 172.21.15.57:0/3002199841 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e00a7050 con 0x7fa8d8005bc0 2023-10-30T14:16:50.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8eeffd640 1 -- 172.21.15.57:0/3002199841 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e00aba00 con 0x7fa8d8005bc0 2023-10-30T14:16:50.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8f6980640 1 -- 172.21.15.57:0/3002199841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8d8005bc0 msgr2=0x7fa8d8005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:50.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8f6980640 1 --2- 172.21.15.57:0/3002199841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8d8005bc0 0x7fa8d8005fa0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0001910 tx=0x7fa8e001c560 comp rx=0 tx=0).stop 2023-10-30T14:16:50.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8f4f1d640 1 -- 172.21.15.57:0/3002199841 reap_dead start 2023-10-30T14:16:50.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8f6980640 1 -- 172.21.15.57:0/3002199841 shutdown_connections 2023-10-30T14:16:50.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8f6980640 1 -- 172.21.15.57:0/3002199841 >> 172.21.15.57:0/3002199841 conn(0x7fa8f00cac70 msgr2=0x7fa8f00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:50.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8f6980640 1 -- 172.21.15.57:0/3002199841 shutdown_connections 2023-10-30T14:16:50.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8f6980640 1 -- 172.21.15.57:0/3002199841 wait complete. 2023-10-30T14:16:50.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8f6980640 1 Processor -- start 2023-10-30T14:16:50.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8f6980640 1 -- start start 2023-10-30T14:16:50.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8f6980640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8d8005bc0 0x7fa8f0144260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:50.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8f6980640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8f015e620 0x7fa8f00c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:50.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8f6980640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f00c3fd0 0x7fa8f00c4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:50.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8f6980640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f015d950 con 0x7fa8d8005bc0 2023-10-30T14:16:50.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8f6980640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f00c49c0 con 0x7fa8f015e620 2023-10-30T14:16:50.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8f6980640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f00c25c0 con 0x7fa8f00c3fd0 2023-10-30T14:16:50.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.909+0000 7fa8effff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8d8005bc0 0x7fa8f0144260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:50.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8effff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8d8005bc0 0x7fa8f0144260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51110/0 (socket says 172.21.15.57:51110) 2023-10-30T14:16:50.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8effff640 1 -- 172.21.15.57:0/1757935907 learned_addr learned my addr 172.21.15.57:0/1757935907 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:16:50.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8ef7fe640 1 --2- 172.21.15.57:0/1757935907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8f015e620 0x7fa8f00c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:50.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8f4f1d640 1 --2- 172.21.15.57:0/1757935907 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f00c3fd0 0x7fa8f00c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:50.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8effff640 1 -- 172.21.15.57:0/1757935907 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f00c3fd0 msgr2=0x7fa8f00c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:50.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8effff640 1 --2- 172.21.15.57:0/1757935907 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f00c3fd0 0x7fa8f00c4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:50.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8effff640 1 -- 172.21.15.57:0/1757935907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8f015e620 msgr2=0x7fa8f00c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:50.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8effff640 1 --2- 172.21.15.57:0/1757935907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8f015e620 0x7fa8f00c96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:50.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8effff640 1 -- 172.21.15.57:0/1757935907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f006c790 con 0x7fa8d8005bc0 2023-10-30T14:16:50.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8ef7fe640 1 --2- 172.21.15.57:0/1757935907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8f015e620 0x7fa8f00c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:50.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8effff640 1 --2- 172.21.15.57:0/1757935907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8d8005bc0 0x7fa8f0144260 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0001910 tx=0x7fa8d8007a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:50.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8ecff9640 1 -- 172.21.15.57:0/1757935907 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e00ab600 con 0x7fa8d8005bc0 2023-10-30T14:16:50.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8ecff9640 1 -- 172.21.15.57:0/1757935907 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e00ab9e0 con 0x7fa8d8005bc0 2023-10-30T14:16:50.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8ecff9640 1 -- 172.21.15.57:0/1757935907 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e00abd50 con 0x7fa8d8005bc0 2023-10-30T14:16:50.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8f6980640 1 -- 172.21.15.57:0/1757935907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f01447a0 con 0x7fa8d8005bc0 2023-10-30T14:16:50.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8f6980640 1 -- 172.21.15.57:0/1757935907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f015bb00 con 0x7fa8d8005bc0 2023-10-30T14:16:50.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8f6980640 1 -- 172.21.15.57:0/1757935907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f0070c80 con 0x7fa8d8005bc0 2023-10-30T14:16:50.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8ecff9640 1 -- 172.21.15.57:0/1757935907 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa8e00b2040 con 0x7fa8d8005bc0 2023-10-30T14:16:50.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8ecff9640 1 --2- 172.21.15.57:0/1757935907 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa8dc0664a0 0x7fa8dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:50.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.913+0000 7fa8ecff9640 1 -- 172.21.15.57:0/1757935907 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7fa8e011c340 con 0x7fa8d8005bc0 2023-10-30T14:16:50.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.917+0000 7fa8ef7fe640 1 --2- 172.21.15.57:0/1757935907 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa8dc0664a0 0x7fa8dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:50.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.917+0000 7fa8ef7fe640 1 --2- 172.21.15.57:0/1757935907 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa8dc0664a0 0x7fa8dc068960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4000f70 tx=0x7fa8e4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:50.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:50.917+0000 7fa8ecff9640 1 -- 172.21.15.57:0/1757935907 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa8e00e9600 con 0x7fa8d8005bc0 2023-10-30T14:16:51.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.113+0000 7fa8f6980640 1 -- 172.21.15.57:0/1757935907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8f015c710 con 0x7fa8d8005bc0 2023-10-30T14:16:51.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.113+0000 7fa8ecff9640 1 -- 172.21.15.57:0/1757935907 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5733 (secure 0 0 0) 0x7fa8f015c710 con 0x7fa8d8005bc0 2023-10-30T14:16:51.117 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:16:51.117 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:16:49.618118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:51.117 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2023-10-30T14:16:51.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.113+0000 7fa8f6980640 1 -- 172.21.15.57:0/1757935907 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa8dc0664a0 msgr2=0x7fa8dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:51.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.113+0000 7fa8f6980640 1 --2- 172.21.15.57:0/1757935907 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa8dc0664a0 0x7fa8dc068960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4000f70 tx=0x7fa8e4009040 comp rx=0 tx=0).stop 2023-10-30T14:16:51.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.113+0000 7fa8f6980640 1 -- 172.21.15.57:0/1757935907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8d8005bc0 msgr2=0x7fa8f0144260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:51.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.113+0000 7fa8f6980640 1 --2- 172.21.15.57:0/1757935907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8d8005bc0 0x7fa8f0144260 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0001910 tx=0x7fa8d8007a10 comp rx=0 tx=0).stop 2023-10-30T14:16:51.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.113+0000 7fa8f6980640 1 -- 172.21.15.57:0/1757935907 shutdown_connections 2023-10-30T14:16:51.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.113+0000 7fa8f6980640 1 --2- 172.21.15.57:0/1757935907 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa8dc0664a0 0x7fa8dc068960 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:51.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.113+0000 7fa8f6980640 1 --2- 172.21.15.57:0/1757935907 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f00c3fd0 0x7fa8f00c4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:51.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.113+0000 7fa8f6980640 1 --2- 172.21.15.57:0/1757935907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8f015e620 0x7fa8f00c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:51.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.113+0000 7fa8f6980640 1 --2- 172.21.15.57:0/1757935907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8d8005bc0 0x7fa8f0144260 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:51.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.113+0000 7fa8f6980640 1 -- 172.21.15.57:0/1757935907 >> 172.21.15.57:0/1757935907 conn(0x7fa8f00cac70 msgr2=0x7fa8f014e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:51.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.113+0000 7fa8f6980640 1 -- 172.21.15.57:0/1757935907 shutdown_connections 2023-10-30T14:16:51.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.113+0000 7fa8f6980640 1 -- 172.21.15.57:0/1757935907 wait complete. 2023-10-30T14:16:51.138 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T14:16:51.138 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:16:51.138 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:51.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.277+0000 7f72181d3640 1 Processor -- start 2023-10-30T14:16:51.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.277+0000 7f72181d3640 1 -- start start 2023-10-30T14:16:51.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.277+0000 7f72181d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7210153c80 0x7f721015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:51.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.277+0000 7f72181d3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f72100c4a10 0x7f72100c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:51.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.277+0000 7f72181d3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f721015a3b0 0x7f72100ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:51.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.277+0000 7f72181d3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f721015dec0 con 0x7f72100c4a10 2023-10-30T14:16:51.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.277+0000 7f72181d3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72101599a0 con 0x7f721015a3b0 2023-10-30T14:16:51.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.277+0000 7f72181d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7210159f20 con 0x7f7210153c80 2023-10-30T14:16:51.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.277+0000 7f721576e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f72100c4a10 0x7f72100c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:51.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.277+0000 7f721576e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f72100c4a10 0x7f72100c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51112/0 (socket says 172.21.15.57:51112) 2023-10-30T14:16:51.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.277+0000 7f721576e640 1 -- 172.21.15.57:0/1894762498 learned_addr learned my addr 172.21.15.57:0/1894762498 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:16:51.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.277+0000 7f7215f6f640 1 --2- 172.21.15.57:0/1894762498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7210153c80 0x7f721015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:51.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.277+0000 7f7216770640 1 --2- 172.21.15.57:0/1894762498 >> v2:172.21.15.88:3300/0 conn(0x7f721015a3b0 0x7f72100ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:51.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.277+0000 7f721576e640 1 -- 172.21.15.57:0/1894762498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7210153c80 msgr2=0x7f721015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:51.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.277+0000 7f721576e640 1 --2- 172.21.15.57:0/1894762498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7210153c80 0x7f721015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:51.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.277+0000 7f721576e640 1 -- 172.21.15.57:0/1894762498 >> v2:172.21.15.88:3300/0 conn(0x7f721015a3b0 msgr2=0x7f72100ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:51.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.277+0000 7f721576e640 1 --2- 172.21.15.57:0/1894762498 >> v2:172.21.15.88:3300/0 conn(0x7f721015a3b0 0x7f72100ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:51.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.277+0000 7f721576e640 1 -- 172.21.15.57:0/1894762498 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72100c47e0 con 0x7f72100c4a10 2023-10-30T14:16:51.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.277+0000 7f7216770640 1 --2- 172.21.15.57:0/1894762498 >> v2:172.21.15.88:3300/0 conn(0x7f721015a3b0 0x7f72100ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:51.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.281+0000 7f721576e640 1 --2- 172.21.15.57:0/1894762498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72100c4a10 0x7f72100c42a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f72100cbbc0 tx=0x7f720001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c0fe94e2124a9c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:51.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.281+0000 7f7214f6d640 1 -- 172.21.15.57:0/1894762498 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72000a89b0 con 0x7f72100c4a10 2023-10-30T14:16:51.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.281+0000 7f7214f6d640 1 -- 172.21.15.57:0/1894762498 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72000a8b10 con 0x7f72100c4a10 2023-10-30T14:16:51.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.281+0000 7f7214f6d640 1 -- 172.21.15.57:0/1894762498 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72000a8ea0 con 0x7f72100c4a10 2023-10-30T14:16:51.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.281+0000 7f72181d3640 1 -- 172.21.15.57:0/1894762498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72100c4a10 msgr2=0x7f72100c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:51.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.281+0000 7f72181d3640 1 --2- 172.21.15.57:0/1894762498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72100c4a10 0x7f72100c42a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f72100cbbc0 tx=0x7f720001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:16:51.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.281+0000 7f72181d3640 1 -- 172.21.15.57:0/1894762498 shutdown_connections 2023-10-30T14:16:51.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.281+0000 7f72181d3640 1 --2- 172.21.15.57:0/1894762498 >> v2:172.21.15.88:3300/0 conn(0x7f721015a3b0 0x7f72100ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:51.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.281+0000 7f72181d3640 1 --2- 172.21.15.57:0/1894762498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72100c4a10 0x7f72100c42a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:51.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.281+0000 7f72181d3640 1 --2- 172.21.15.57:0/1894762498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7210153c80 0x7f721015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:51.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.281+0000 7f72181d3640 1 -- 172.21.15.57:0/1894762498 >> 172.21.15.57:0/1894762498 conn(0x7f72100715c0 msgr2=0x7f72100719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:51.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.281+0000 7f72181d3640 1 -- 172.21.15.57:0/1894762498 shutdown_connections 2023-10-30T14:16:51.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.281+0000 7f72181d3640 1 -- 172.21.15.57:0/1894762498 wait complete. 2023-10-30T14:16:51.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f72181d3640 1 Processor -- start 2023-10-30T14:16:51.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f72181d3640 1 -- start start 2023-10-30T14:16:51.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f72181d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72100c4a10 0x7f72100ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:51.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f72181d3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7210153c80 0x7f721015ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:51.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f72181d3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f721015a3b0 0x7f721015b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:51.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f72181d3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72100cece0 con 0x7f7210153c80 2023-10-30T14:16:51.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f72181d3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72100b4100 con 0x7f721015a3b0 2023-10-30T14:16:51.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f72181d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72100b3210 con 0x7f72100c4a10 2023-10-30T14:16:51.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f721576e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7210153c80 0x7f721015ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:51.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f721576e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7210153c80 0x7f721015ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51120/0 (socket says 172.21.15.57:51120) 2023-10-30T14:16:51.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f721576e640 1 -- 172.21.15.57:0/28645492 learned_addr learned my addr 172.21.15.57:0/28645492 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:16:51.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f7215f6f640 1 --2- 172.21.15.57:0/28645492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72100c4a10 0x7f72100ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:51.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f7216770640 1 --2- 172.21.15.57:0/28645492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f721015a3b0 0x7f721015b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:51.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f721576e640 1 -- 172.21.15.57:0/28645492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72100c4a10 msgr2=0x7f72100ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:51.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f721576e640 1 --2- 172.21.15.57:0/28645492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72100c4a10 0x7f72100ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:51.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f721576e640 1 -- 172.21.15.57:0/28645492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f721015a3b0 msgr2=0x7f721015b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:51.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f721576e640 1 --2- 172.21.15.57:0/28645492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f721015a3b0 0x7f721015b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:51.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f721576e640 1 -- 172.21.15.57:0/28645492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72000094d0 con 0x7f7210153c80 2023-10-30T14:16:51.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f7216770640 1 --2- 172.21.15.57:0/28645492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f721015a3b0 0x7f721015b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:51.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f721576e640 1 --2- 172.21.15.57:0/28645492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7210153c80 0x7f721015ab60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f720001d3a0 tx=0x7f720001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:51.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f72067fc640 1 -- 172.21.15.57:0/28645492 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72000ba600 con 0x7f7210153c80 2023-10-30T14:16:51.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f72067fc640 1 -- 172.21.15.57:0/28645492 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72000ba760 con 0x7f7210153c80 2023-10-30T14:16:51.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f72181d3640 1 -- 172.21.15.57:0/28645492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72100b34f0 con 0x7f7210153c80 2023-10-30T14:16:51.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f72067fc640 1 -- 172.21.15.57:0/28645492 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72000bab10 con 0x7f7210153c80 2023-10-30T14:16:51.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.285+0000 7f72181d3640 1 -- 172.21.15.57:0/28645492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72100b3ab0 con 0x7f7210153c80 2023-10-30T14:16:51.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.289+0000 7f72181d3640 1 -- 172.21.15.57:0/28645492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f721004e170 con 0x7f7210153c80 2023-10-30T14:16:51.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.289+0000 7f72067fc640 1 -- 172.21.15.57:0/28645492 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f72000bac70 con 0x7f7210153c80 2023-10-30T14:16:51.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.289+0000 7f72067fc640 1 --2- 172.21.15.57:0/28645492 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f71f40664a0 0x7f71f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:51.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.289+0000 7f72067fc640 1 -- 172.21.15.57:0/28645492 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f7200122670 con 0x7f7210153c80 2023-10-30T14:16:51.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.289+0000 7f7215f6f640 1 --2- 172.21.15.57:0/28645492 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f71f40664a0 0x7f71f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:51.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.289+0000 7f7215f6f640 1 --2- 172.21.15.57:0/28645492 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f71f40664a0 0x7f71f4068960 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f720c009ae0 tx=0x7f720c009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:51.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.293+0000 7f72067fc640 1 -- 172.21.15.57:0/28645492 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f72000ef8a0 con 0x7f7210153c80 2023-10-30T14:16:51.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.489+0000 7f72181d3640 1 -- 172.21.15.57:0/28645492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72100b4240 con 0x7f7210153c80 2023-10-30T14:16:51.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.489+0000 7f72067fc640 1 -- 172.21.15.57:0/28645492 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5733 (secure 0 0 0) 0x7f72000b5090 con 0x7f7210153c80 2023-10-30T14:16:51.495 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:16:51.496 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:16:49.618118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:51.496 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2023-10-30T14:16:51.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.493+0000 7f72181d3640 1 -- 172.21.15.57:0/28645492 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f71f40664a0 msgr2=0x7f71f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:51.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.493+0000 7f72181d3640 1 --2- 172.21.15.57:0/28645492 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f71f40664a0 0x7f71f4068960 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f720c009ae0 tx=0x7f720c009190 comp rx=0 tx=0).stop 2023-10-30T14:16:51.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.493+0000 7f72181d3640 1 -- 172.21.15.57:0/28645492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7210153c80 msgr2=0x7f721015ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:51.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.493+0000 7f72181d3640 1 --2- 172.21.15.57:0/28645492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7210153c80 0x7f721015ab60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f720001d3a0 tx=0x7f720001ba90 comp rx=0 tx=0).stop 2023-10-30T14:16:51.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.493+0000 7f72181d3640 1 -- 172.21.15.57:0/28645492 shutdown_connections 2023-10-30T14:16:51.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.493+0000 7f72181d3640 1 --2- 172.21.15.57:0/28645492 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f71f40664a0 0x7f71f4068960 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:51.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.493+0000 7f72181d3640 1 --2- 172.21.15.57:0/28645492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f721015a3b0 0x7f721015b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:51.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.493+0000 7f72181d3640 1 --2- 172.21.15.57:0/28645492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7210153c80 0x7f721015ab60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:51.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.493+0000 7f72181d3640 1 --2- 172.21.15.57:0/28645492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72100c4a10 0x7f72100ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:51.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.493+0000 7f72181d3640 1 -- 172.21.15.57:0/28645492 >> 172.21.15.57:0/28645492 conn(0x7f72100715c0 msgr2=0x7f72100c9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:51.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.493+0000 7f72181d3640 1 -- 172.21.15.57:0/28645492 shutdown_connections 2023-10-30T14:16:51.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:51.493+0000 7f72181d3640 1 -- 172.21.15.57:0/28645492 wait complete. 2023-10-30T14:16:51.517 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 21, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 21, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4266": { "addr": "172.21.15.57:6836/2693548867", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2693548867, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2693548867, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4266, "incarnation": 9, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4282": { "addr": "172.21.15.88:6839/2945854794", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 2945854794, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2945854794, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4936": { "addr": "172.21.15.57:6837/1988100930", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 1988100930, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 1988100930, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4936, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:16:49.618118+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 4266, "mds_2": 4936 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:16:53.518 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:53.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.661+0000 7f7bbd091640 1 Processor -- start 2023-10-30T14:16:53.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.661+0000 7f7bbd091640 1 -- start start 2023-10-30T14:16:53.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.661+0000 7f7bbd091640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7bb80cb7d0 0x7f7bb80c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:53.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.661+0000 7f7bbd091640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7bb80c3e40 0x7f7bb8070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:53.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.661+0000 7f7bbd091640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb80c2310 con 0x7f7bb8150830 2023-10-30T14:16:53.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.661+0000 7f7bbd091640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bb806bc10 con 0x7f7bb80cb7d0 2023-10-30T14:16:53.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.661+0000 7f7bbd091640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb8156710 con 0x7f7bb80c3e40 2023-10-30T14:16:53.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.661+0000 7f7bb659c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7bb80cb7d0 0x7f7bb80c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:53.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.661+0000 7f7bb6d9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f7bb8150830 0x7f7bb806b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:38938/0 (socket says 172.21.15.57:38938) 2023-10-30T14:16:53.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.661+0000 7f7bb6d9d640 1 -- 172.21.15.57:0/2714037993 learned_addr learned my addr 172.21.15.57:0/2714037993 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:16:53.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.665+0000 7f7bb759e640 1 --2- 172.21.15.57:0/2714037993 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7bb80c3e40 0x7f7bb8070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:53.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.665+0000 7f7bb659c640 1 -- 172.21.15.57:0/2714037993 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7bb80c3e40 msgr2=0x7f7bb8070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:53.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.665+0000 7f7bb659c640 1 --2- 172.21.15.57:0/2714037993 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7bb80c3e40 0x7f7bb8070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:53.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.665+0000 7f7bb659c640 1 -- 172.21.15.57:0/2714037993 >> v1:172.21.15.57:6789/0 conn(0x7f7bb8150830 legacy=0x7f7bb806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:53.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.665+0000 7f7bb659c640 1 -- 172.21.15.57:0/2714037993 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb80c4360 con 0x7f7bb80cb7d0 2023-10-30T14:16:53.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.665+0000 7f7bb5d9b640 1 -- 172.21.15.57:0/2714037993 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632790255 0 0) 0x7f7bb80c2310 con 0x7f7bb8150830 2023-10-30T14:16:53.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.665+0000 7f7bb659c640 1 --2- 172.21.15.57:0/2714037993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bb80cb7d0 0x7f7bb80c3900 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0097c0 tx=0x7f7bac01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=800a38613dd03deb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:53.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.665+0000 7f7bb5d9b640 1 -- 172.21.15.57:0/2714037993 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bac0a8970 con 0x7f7bb80cb7d0 2023-10-30T14:16:53.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.665+0000 7f7bb5d9b640 1 -- 172.21.15.57:0/2714037993 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bac0a8ad0 con 0x7f7bb80cb7d0 2023-10-30T14:16:53.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.665+0000 7f7bb5d9b640 1 -- 172.21.15.57:0/2714037993 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bac0a8e60 con 0x7f7bb80cb7d0 2023-10-30T14:16:53.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bbd091640 1 -- 172.21.15.57:0/2714037993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bb80cb7d0 msgr2=0x7f7bb80c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:53.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bbd091640 1 --2- 172.21.15.57:0/2714037993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bb80cb7d0 0x7f7bb80c3900 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0097c0 tx=0x7f7bac01ec70 comp rx=0 tx=0).stop 2023-10-30T14:16:53.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bbd091640 1 -- 172.21.15.57:0/2714037993 shutdown_connections 2023-10-30T14:16:53.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bbd091640 1 --2- 172.21.15.57:0/2714037993 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7bb80c3e40 0x7f7bb8070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:53.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bbd091640 1 --2- 172.21.15.57:0/2714037993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bb80cb7d0 0x7f7bb80c3900 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:53.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bbd091640 1 -- 172.21.15.57:0/2714037993 >> 172.21.15.57:0/2714037993 conn(0x7f7bb80cac70 msgr2=0x7f7bb80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:53.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bbd091640 1 -- 172.21.15.57:0/2714037993 shutdown_connections 2023-10-30T14:16:53.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bbd091640 1 -- 172.21.15.57:0/2714037993 wait complete. 2023-10-30T14:16:53.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bbd091640 1 Processor -- start 2023-10-30T14:16:53.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bbd091640 1 -- start start 2023-10-30T14:16:53.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bbd091640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bb80c3e40 0x7f7bb81397f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:53.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bbd091640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7bb80cb7d0 0x7f7bb8158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:53.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bbd091640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7bb8150830 0x7f7bb80c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:53.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bbd091640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb81f29e0 con 0x7f7bb80c3e40 2023-10-30T14:16:53.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bbd091640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb8147260 con 0x7f7bb80cb7d0 2023-10-30T14:16:53.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bbd091640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb80c92f0 con 0x7f7bb8150830 2023-10-30T14:16:53.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bb6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bb80c3e40 0x7f7bb81397f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:53.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bb6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bb80c3e40 0x7f7bb81397f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51136/0 (socket says 172.21.15.57:51136) 2023-10-30T14:16:53.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bb6d9d640 1 -- 172.21.15.57:0/3996658378 learned_addr learned my addr 172.21.15.57:0/3996658378 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:16:53.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bb759e640 1 --2- 172.21.15.57:0/3996658378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7bb8150830 0x7f7bb80c9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:53.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bb659c640 1 --2- 172.21.15.57:0/3996658378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7bb80cb7d0 0x7f7bb8158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:53.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bb6d9d640 1 -- 172.21.15.57:0/3996658378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7bb8150830 msgr2=0x7f7bb80c9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:53.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bb6d9d640 1 --2- 172.21.15.57:0/3996658378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7bb8150830 0x7f7bb80c9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:53.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bb6d9d640 1 -- 172.21.15.57:0/3996658378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7bb80cb7d0 msgr2=0x7f7bb8158810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:53.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bb6d9d640 1 --2- 172.21.15.57:0/3996658378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7bb80cb7d0 0x7f7bb8158810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:53.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bb6d9d640 1 -- 172.21.15.57:0/3996658378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bac009420 con 0x7f7bb80c3e40 2023-10-30T14:16:53.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bb759e640 1 --2- 172.21.15.57:0/3996658378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7bb8150830 0x7f7bb80c9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:53.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.669+0000 7f7bb659c640 1 --2- 172.21.15.57:0/3996658378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7bb80cb7d0 0x7f7bb8158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:53.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.673+0000 7f7bb6d9d640 1 --2- 172.21.15.57:0/3996658378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bb80c3e40 0x7f7bb81397f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7ba800c480 tx=0x7f7ba800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:53.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.673+0000 7f7b9f7fe640 1 -- 172.21.15.57:0/3996658378 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba80074e0 con 0x7f7bb80c3e40 2023-10-30T14:16:53.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.673+0000 7f7b9f7fe640 1 -- 172.21.15.57:0/3996658378 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba8007660 con 0x7f7bb80c3e40 2023-10-30T14:16:53.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.673+0000 7f7b9f7fe640 1 -- 172.21.15.57:0/3996658378 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba8004a30 con 0x7f7bb80c3e40 2023-10-30T14:16:53.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.673+0000 7f7bbd091640 1 -- 172.21.15.57:0/3996658378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb80c95d0 con 0x7f7bb80c3e40 2023-10-30T14:16:53.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.673+0000 7f7bbd091640 1 -- 172.21.15.57:0/3996658378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb81f2390 con 0x7f7bb80c3e40 2023-10-30T14:16:53.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.673+0000 7f7bbd091640 1 -- 172.21.15.57:0/3996658378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb804e170 con 0x7f7bb80c3e40 2023-10-30T14:16:53.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.673+0000 7f7b9f7fe640 1 -- 172.21.15.57:0/3996658378 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7ba8004b90 con 0x7f7bb80c3e40 2023-10-30T14:16:53.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.673+0000 7f7b9f7fe640 1 --2- 172.21.15.57:0/3996658378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7ba40664d0 0x7f7ba4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:53.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.673+0000 7f7b9f7fe640 1 -- 172.21.15.57:0/3996658378 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f7ba8083130 con 0x7f7bb80c3e40 2023-10-30T14:16:53.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.673+0000 7f7bb659c640 1 --2- 172.21.15.57:0/3996658378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7ba40664d0 0x7f7ba4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:53.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.677+0000 7f7bb659c640 1 --2- 172.21.15.57:0/3996658378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7ba40664d0 0x7f7ba4068990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0b2000 tx=0x7f7bac008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:53.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.677+0000 7f7b9f7fe640 1 -- 172.21.15.57:0/3996658378 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7ba8053f10 con 0x7f7bb80c3e40 2023-10-30T14:16:53.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.869+0000 7f7bbd091640 1 -- 172.21.15.57:0/3996658378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bb8073560 con 0x7f7bb80c3e40 2023-10-30T14:16:53.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.869+0000 7f7b9f7fe640 1 -- 172.21.15.57:0/3996658378 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5733 (secure 0 0 0) 0x7f7ba8053aa0 con 0x7f7bb80c3e40 2023-10-30T14:16:53.875 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:16:53.876 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:16:49.618118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:53.876 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2023-10-30T14:16:53.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.873+0000 7f7bbd091640 1 -- 172.21.15.57:0/3996658378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7ba40664d0 msgr2=0x7f7ba4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:53.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.873+0000 7f7bbd091640 1 --2- 172.21.15.57:0/3996658378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7ba40664d0 0x7f7ba4068990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0b2000 tx=0x7f7bac008040 comp rx=0 tx=0).stop 2023-10-30T14:16:53.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.873+0000 7f7bbd091640 1 -- 172.21.15.57:0/3996658378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bb80c3e40 msgr2=0x7f7bb81397f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:53.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.873+0000 7f7bbd091640 1 --2- 172.21.15.57:0/3996658378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bb80c3e40 0x7f7bb81397f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7ba800c480 tx=0x7f7ba800c940 comp rx=0 tx=0).stop 2023-10-30T14:16:53.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.873+0000 7f7bbd091640 1 -- 172.21.15.57:0/3996658378 shutdown_connections 2023-10-30T14:16:53.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.873+0000 7f7bbd091640 1 --2- 172.21.15.57:0/3996658378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7ba40664d0 0x7f7ba4068990 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:53.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.873+0000 7f7bbd091640 1 --2- 172.21.15.57:0/3996658378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7bb8150830 0x7f7bb80c9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:53.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.873+0000 7f7bbd091640 1 --2- 172.21.15.57:0/3996658378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7bb80cb7d0 0x7f7bb8158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:53.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.873+0000 7f7bbd091640 1 --2- 172.21.15.57:0/3996658378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bb80c3e40 0x7f7bb81397f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:53.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.873+0000 7f7bbd091640 1 -- 172.21.15.57:0/3996658378 >> 172.21.15.57:0/3996658378 conn(0x7f7bb80cac70 msgr2=0x7f7bb80c54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:53.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.873+0000 7f7bbd091640 1 -- 172.21.15.57:0/3996658378 shutdown_connections 2023-10-30T14:16:53.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:53.873+0000 7f7bbd091640 1 -- 172.21.15.57:0/3996658378 wait complete. 2023-10-30T14:16:53.896 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:16:55.897 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:56.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.049+0000 7fcce930f640 1 Processor -- start 2023-10-30T14:16:56.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.049+0000 7fcce930f640 1 -- start start 2023-10-30T14:16:56.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.049+0000 7fcce930f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcce4159760 0x7fcce415d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:56.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.049+0000 7fcce930f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcce415da00 0x7fcce4069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:56.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.049+0000 7fcce930f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce415cd50 con 0x7fcce40c5c10 2023-10-30T14:16:56.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.049+0000 7fcce930f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce415d0a0 con 0x7fcce4159760 2023-10-30T14:16:56.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.049+0000 7fcce930f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce406f600 con 0x7fcce415da00 2023-10-30T14:16:56.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.049+0000 7fccda7fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcce4159760 0x7fcce415d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:56.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.049+0000 7fccda7fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcce4159760 0x7fcce415d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46680/0 (socket says 172.21.15.57:46680) 2023-10-30T14:16:56.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.049+0000 7fccda7fc640 1 -- 172.21.15.57:0/1012757848 learned_addr learned my addr 172.21.15.57:0/1012757848 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:16:56.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.049+0000 7fcce37fe640 1 --2- 172.21.15.57:0/1012757848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcce415da00 0x7fcce4069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:56.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.049+0000 7fccda7fc640 1 -- 172.21.15.57:0/1012757848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcce415da00 msgr2=0x7fcce4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:56.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.049+0000 7fccda7fc640 1 --2- 172.21.15.57:0/1012757848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcce415da00 0x7fcce4069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:56.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.049+0000 7fccda7fc640 1 -- 172.21.15.57:0/1012757848 >> v1:172.21.15.88:6789/0 conn(0x7fcce40c5c10 legacy=0x7fcce406eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:56.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.049+0000 7fccda7fc640 1 -- 172.21.15.57:0/1012757848 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce40ca000 con 0x7fcce4159760 2023-10-30T14:16:56.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.053+0000 7fccda7fc640 1 --2- 172.21.15.57:0/1012757848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcce4159760 0x7fcce415d4c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fcccc009870 tx=0x7fcccc01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b585167492f8e28f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:56.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.053+0000 7fcce27fc640 1 -- 172.21.15.57:0/1012757848 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc0ab620 con 0x7fcce4159760 2023-10-30T14:16:56.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.053+0000 7fcce27fc640 1 -- 172.21.15.57:0/1012757848 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcccc0ab780 con 0x7fcce4159760 2023-10-30T14:16:56.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.053+0000 7fcce27fc640 1 -- 172.21.15.57:0/1012757848 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc0abb10 con 0x7fcce4159760 2023-10-30T14:16:56.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.053+0000 7fcce930f640 1 -- 172.21.15.57:0/1012757848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcce4159760 msgr2=0x7fcce415d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:56.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.053+0000 7fcce930f640 1 --2- 172.21.15.57:0/1012757848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcce4159760 0x7fcce415d4c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fcccc009870 tx=0x7fcccc01c510 comp rx=0 tx=0).stop 2023-10-30T14:16:56.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.053+0000 7fcce930f640 1 -- 172.21.15.57:0/1012757848 shutdown_connections 2023-10-30T14:16:56.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.053+0000 7fcce930f640 1 --2- 172.21.15.57:0/1012757848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcce415da00 0x7fcce4069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:56.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.053+0000 7fcce930f640 1 --2- 172.21.15.57:0/1012757848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcce4159760 0x7fcce415d4c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:56.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.053+0000 7fcce930f640 1 -- 172.21.15.57:0/1012757848 >> 172.21.15.57:0/1012757848 conn(0x7fcce406afa0 msgr2=0x7fcce406b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:56.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.053+0000 7fcce930f640 1 -- 172.21.15.57:0/1012757848 shutdown_connections 2023-10-30T14:16:56.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.053+0000 7fcce930f640 1 -- 172.21.15.57:0/1012757848 wait complete. 2023-10-30T14:16:56.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.053+0000 7fcce930f640 1 Processor -- start 2023-10-30T14:16:56.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce930f640 1 -- start start 2023-10-30T14:16:56.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce930f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcce40c5c10 0x7fcce41f5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:56.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce930f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcce4159760 0x7fcce4154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:56.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce930f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcce415da00 0x7fcce4155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:56.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce930f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce4151f40 con 0x7fcce40c5c10 2023-10-30T14:16:56.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce930f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce414f4e0 con 0x7fcce415da00 2023-10-30T14:16:56.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce930f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce4151c60 con 0x7fcce4159760 2023-10-30T14:16:56.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcce40c5c10 0x7fcce41f5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:56.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcce40c5c10 0x7fcce41f5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46692/0 (socket says 172.21.15.57:46692) 2023-10-30T14:16:56.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce2ffd640 1 -- 172.21.15.57:0/3194457672 learned_addr learned my addr 172.21.15.57:0/3194457672 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:16:56.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fccda7fc640 1 --2- 172.21.15.57:0/3194457672 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcce4159760 0x7fcce4154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:56.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce37fe640 1 --2- 172.21.15.57:0/3194457672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcce415da00 0x7fcce4155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:56.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce2ffd640 1 -- 172.21.15.57:0/3194457672 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcce4159760 msgr2=0x7fcce4154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:56.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce2ffd640 1 --2- 172.21.15.57:0/3194457672 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcce4159760 0x7fcce4154150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:56.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce2ffd640 1 -- 172.21.15.57:0/3194457672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcce415da00 msgr2=0x7fcce4155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:56.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce2ffd640 1 --2- 172.21.15.57:0/3194457672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcce415da00 0x7fcce4155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:56.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce2ffd640 1 -- 172.21.15.57:0/3194457672 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc0094d0 con 0x7fcce40c5c10 2023-10-30T14:16:56.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fccda7fc640 1 --2- 172.21.15.57:0/3194457672 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcce4159760 0x7fcce4154150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:56.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce2ffd640 1 --2- 172.21.15.57:0/3194457672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcce40c5c10 0x7fcce41f5600 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fccd400c3c0 tx=0x7fccd400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:56.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fccdbfff640 1 -- 172.21.15.57:0/3194457672 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd40052d0 con 0x7fcce40c5c10 2023-10-30T14:16:56.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fccdbfff640 1 -- 172.21.15.57:0/3194457672 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd4005430 con 0x7fcce40c5c10 2023-10-30T14:16:56.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fccdbfff640 1 -- 172.21.15.57:0/3194457672 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd4003840 con 0x7fcce40c5c10 2023-10-30T14:16:56.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce930f640 1 -- 172.21.15.57:0/3194457672 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce41534f0 con 0x7fcce40c5c10 2023-10-30T14:16:56.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce930f640 1 -- 172.21.15.57:0/3194457672 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce41f5b40 con 0x7fcce40c5c10 2023-10-30T14:16:56.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.057+0000 7fcce930f640 1 -- 172.21.15.57:0/3194457672 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce404e170 con 0x7fcce40c5c10 2023-10-30T14:16:56.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.061+0000 7fccdbfff640 1 -- 172.21.15.57:0/3194457672 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fccd40039a0 con 0x7fcce40c5c10 2023-10-30T14:16:56.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.061+0000 7fccdbfff640 1 --2- 172.21.15.57:0/3194457672 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fccc40664a0 0x7fccc4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:56.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.061+0000 7fccdbfff640 1 -- 172.21.15.57:0/3194457672 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7fccd4083500 con 0x7fcce40c5c10 2023-10-30T14:16:56.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.061+0000 7fccda7fc640 1 --2- 172.21.15.57:0/3194457672 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fccc40664a0 0x7fccc4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:56.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.061+0000 7fccda7fc640 1 --2- 172.21.15.57:0/3194457672 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fccc40664a0 0x7fccc4068960 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcccc0af000 tx=0x7fcccc0a7000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:56.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.065+0000 7fccdbfff640 1 -- 172.21.15.57:0/3194457672 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fccd4050850 con 0x7fcce40c5c10 2023-10-30T14:16:56.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.253+0000 7fcce930f640 1 -- 172.21.15.57:0/3194457672 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcce41539a0 con 0x7fcce40c5c10 2023-10-30T14:16:56.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.253+0000 7fccdbfff640 1 -- 172.21.15.57:0/3194457672 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5735 (secure 0 0 0) 0x7fccd4053e70 con 0x7fcce40c5c10 2023-10-30T14:16:56.257 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:16:56.257 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:16:54.775384+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:resolve","state_seq":23,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:56.257 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 22 2023-10-30T14:16:56.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.253+0000 7fcce930f640 1 -- 172.21.15.57:0/3194457672 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fccc40664a0 msgr2=0x7fccc4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:56.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.253+0000 7fcce930f640 1 --2- 172.21.15.57:0/3194457672 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fccc40664a0 0x7fccc4068960 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcccc0af000 tx=0x7fcccc0a7000 comp rx=0 tx=0).stop 2023-10-30T14:16:56.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.253+0000 7fcce930f640 1 -- 172.21.15.57:0/3194457672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcce40c5c10 msgr2=0x7fcce41f5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:56.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.253+0000 7fcce930f640 1 --2- 172.21.15.57:0/3194457672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcce40c5c10 0x7fcce41f5600 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fccd400c3c0 tx=0x7fccd400c880 comp rx=0 tx=0).stop 2023-10-30T14:16:56.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.253+0000 7fcce930f640 1 -- 172.21.15.57:0/3194457672 shutdown_connections 2023-10-30T14:16:56.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.253+0000 7fcce930f640 1 --2- 172.21.15.57:0/3194457672 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fccc40664a0 0x7fccc4068960 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:56.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.253+0000 7fcce930f640 1 --2- 172.21.15.57:0/3194457672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcce415da00 0x7fcce4155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:56.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.253+0000 7fcce930f640 1 --2- 172.21.15.57:0/3194457672 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcce4159760 0x7fcce4154150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:56.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.253+0000 7fcce930f640 1 --2- 172.21.15.57:0/3194457672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcce40c5c10 0x7fcce41f5600 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:56.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.253+0000 7fcce930f640 1 -- 172.21.15.57:0/3194457672 >> 172.21.15.57:0/3194457672 conn(0x7fcce406afa0 msgr2=0x7fcce4072650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:56.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.253+0000 7fcce930f640 1 -- 172.21.15.57:0/3194457672 shutdown_connections 2023-10-30T14:16:56.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:56.253+0000 7fcce930f640 1 -- 172.21.15.57:0/3194457672 wait complete. 2023-10-30T14:16:56.278 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:16:58.279 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:58.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.429+0000 7f6372654640 1 Processor -- start 2023-10-30T14:16:58.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.429+0000 7f6372654640 1 -- start start 2023-10-30T14:16:58.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.433+0000 7f6372654640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f636c1537f0 0x7f636c0701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:58.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.433+0000 7f6372654640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636c0c2aa0 con 0x7f636c15a2e0 2023-10-30T14:16:58.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.433+0000 7f6372654640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636c0708e0 con 0x7f636c06b2f0 2023-10-30T14:16:58.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.433+0000 7f6372654640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f636c0c3ab0 con 0x7f636c1537f0 2023-10-30T14:16:58.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.433+0000 7f636bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f636c1537f0 0x7f636c0701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:58.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.433+0000 7f636b7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f636c15a2e0 0x7f636c0ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:52906/0 (socket says 172.21.15.57:52906) 2023-10-30T14:16:58.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.433+0000 7f636b7fe640 1 -- 172.21.15.57:0/802064697 learned_addr learned my addr 172.21.15.57:0/802064697 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:16:58.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.433+0000 7f636bfff640 1 -- 172.21.15.57:0/802064697 >> v1:172.21.15.88:6789/0 conn(0x7f636c06b2f0 legacy=0x7f636c06b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:58.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.433+0000 7f636bfff640 1 -- 172.21.15.57:0/802064697 >> v1:172.21.15.57:6789/0 conn(0x7f636c15a2e0 legacy=0x7f636c0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:58.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.433+0000 7f636bfff640 1 -- 172.21.15.57:0/802064697 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636c0c1eb0 con 0x7f636c1537f0 2023-10-30T14:16:58.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.433+0000 7f636bfff640 1 --2- 172.21.15.57:0/802064697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f636c1537f0 0x7f636c0701a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f635c0097c0 tx=0x7f635c01ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acd5216b05b8abc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:58.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.437+0000 7f636affd640 1 -- 172.21.15.57:0/802064697 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c0a8930 con 0x7f636c1537f0 2023-10-30T14:16:58.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.437+0000 7f636affd640 1 -- 172.21.15.57:0/802064697 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635c0a8a90 con 0x7f636c1537f0 2023-10-30T14:16:58.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.437+0000 7f636affd640 1 -- 172.21.15.57:0/802064697 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c0a8e20 con 0x7f636c1537f0 2023-10-30T14:16:58.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.437+0000 7f6372654640 1 -- 172.21.15.57:0/802064697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f636c1537f0 msgr2=0x7f636c0701a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:58.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.437+0000 7f6372654640 1 --2- 172.21.15.57:0/802064697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f636c1537f0 0x7f636c0701a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f635c0097c0 tx=0x7f635c01ec30 comp rx=0 tx=0).stop 2023-10-30T14:16:58.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.437+0000 7f6372654640 1 -- 172.21.15.57:0/802064697 shutdown_connections 2023-10-30T14:16:58.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.437+0000 7f6372654640 1 --2- 172.21.15.57:0/802064697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f636c1537f0 0x7f636c0701a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:58.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.437+0000 7f6372654640 1 -- 172.21.15.57:0/802064697 >> 172.21.15.57:0/802064697 conn(0x7f636c071fb0 msgr2=0x7f636c0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:58.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.437+0000 7f6372654640 1 -- 172.21.15.57:0/802064697 shutdown_connections 2023-10-30T14:16:58.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.437+0000 7f6372654640 1 -- 172.21.15.57:0/802064697 wait complete. 2023-10-30T14:16:58.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.437+0000 7f6372654640 1 Processor -- start 2023-10-30T14:16:58.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.437+0000 7f6372654640 1 -- start start 2023-10-30T14:16:58.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.437+0000 7f6372654640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f636c06b2f0 0x7f636c1545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:58.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.437+0000 7f6372654640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f636c1537f0 0x7f636c0c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:58.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.437+0000 7f6372654640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f636c15a2e0 0x7f636c155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:58.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.437+0000 7f6372654640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c0ca330 con 0x7f636c06b2f0 2023-10-30T14:16:58.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.437+0000 7f6372654640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c1efd10 con 0x7f636c15a2e0 2023-10-30T14:16:58.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.437+0000 7f6372654640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f636c159420 con 0x7f636c1537f0 2023-10-30T14:16:58.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.437+0000 7f636bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f636c06b2f0 0x7f636c1545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:58.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.441+0000 7f636bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f636c06b2f0 0x7f636c1545d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46720/0 (socket says 172.21.15.57:46720) 2023-10-30T14:16:58.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.441+0000 7f636bfff640 1 -- 172.21.15.57:0/2763244649 learned_addr learned my addr 172.21.15.57:0/2763244649 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:16:58.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.441+0000 7f636b7fe640 1 --2- 172.21.15.57:0/2763244649 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f636c1537f0 0x7f636c0c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:58.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.441+0000 7f6370bf1640 1 --2- 172.21.15.57:0/2763244649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f636c15a2e0 0x7f636c155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:58.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.441+0000 7f636bfff640 1 -- 172.21.15.57:0/2763244649 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f636c1537f0 msgr2=0x7f636c0c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:58.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.441+0000 7f636bfff640 1 --2- 172.21.15.57:0/2763244649 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f636c1537f0 0x7f636c0c9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:58.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.441+0000 7f636bfff640 1 -- 172.21.15.57:0/2763244649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f636c15a2e0 msgr2=0x7f636c155300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:58.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.441+0000 7f636bfff640 1 --2- 172.21.15.57:0/2763244649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f636c15a2e0 0x7f636c155300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:58.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.441+0000 7f636bfff640 1 -- 172.21.15.57:0/2763244649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635c009420 con 0x7f636c06b2f0 2023-10-30T14:16:58.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.441+0000 7f636b7fe640 1 --2- 172.21.15.57:0/2763244649 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f636c1537f0 0x7f636c0c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:58.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.441+0000 7f6370bf1640 1 --2- 172.21.15.57:0/2763244649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f636c15a2e0 0x7f636c155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:58.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.441+0000 7f636bfff640 1 --2- 172.21.15.57:0/2763244649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f636c06b2f0 0x7f636c1545d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f635c01b840 tx=0x7f635c01b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:58.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.441+0000 7f6368ff9640 1 -- 172.21.15.57:0/2763244649 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c0a8930 con 0x7f636c06b2f0 2023-10-30T14:16:58.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.441+0000 7f6368ff9640 1 -- 172.21.15.57:0/2763244649 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635c0b9410 con 0x7f636c06b2f0 2023-10-30T14:16:58.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.441+0000 7f6368ff9640 1 -- 172.21.15.57:0/2763244649 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c0b97a0 con 0x7f636c06b2f0 2023-10-30T14:16:58.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.441+0000 7f6372654640 1 -- 172.21.15.57:0/2763244649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636c1f0200 con 0x7f636c06b2f0 2023-10-30T14:16:58.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.441+0000 7f6372654640 1 -- 172.21.15.57:0/2763244649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f636c159640 con 0x7f636c06b2f0 2023-10-30T14:16:58.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.441+0000 7f6372654640 1 -- 172.21.15.57:0/2763244649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f636c04e170 con 0x7f636c06b2f0 2023-10-30T14:16:58.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.441+0000 7f6368ff9640 1 -- 172.21.15.57:0/2763244649 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f635c0b7040 con 0x7f636c06b2f0 2023-10-30T14:16:58.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.445+0000 7f6368ff9640 1 --2- 172.21.15.57:0/2763244649 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f63580664d0 0x7f6358068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:58.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.445+0000 7f6368ff9640 1 -- 172.21.15.57:0/2763244649 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f635c121ca0 con 0x7f636c06b2f0 2023-10-30T14:16:58.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.445+0000 7f636b7fe640 1 --2- 172.21.15.57:0/2763244649 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f63580664d0 0x7f6358068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:58.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.445+0000 7f636b7fe640 1 --2- 172.21.15.57:0/2763244649 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f63580664d0 0x7f6358068990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f636c0b31c0 tx=0x7f6360009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:58.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.449+0000 7f6368ff9640 1 -- 172.21.15.57:0/2763244649 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f635c0eeff0 con 0x7f636c06b2f0 2023-10-30T14:16:58.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.677+0000 7f6372654640 1 -- 172.21.15.57:0/2763244649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f636c04e170 con 0x7f636c06b2f0 2023-10-30T14:16:58.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.681+0000 7f6368ff9640 1 -- 172.21.15.57:0/2763244649 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5735 (secure 0 0 0) 0x7f635c0b5020 con 0x7f636c06b2f0 2023-10-30T14:16:58.684 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:16:58.684 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:16:54.775384+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:resolve","state_seq":23,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:58.685 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 22 2023-10-30T14:16:58.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.681+0000 7f6372654640 1 -- 172.21.15.57:0/2763244649 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f63580664d0 msgr2=0x7f6358068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:58.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.681+0000 7f6372654640 1 --2- 172.21.15.57:0/2763244649 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f63580664d0 0x7f6358068990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f636c0b31c0 tx=0x7f6360009040 comp rx=0 tx=0).stop 2023-10-30T14:16:58.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.681+0000 7f6372654640 1 -- 172.21.15.57:0/2763244649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f636c06b2f0 msgr2=0x7f636c1545d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:58.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.681+0000 7f6372654640 1 --2- 172.21.15.57:0/2763244649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f636c06b2f0 0x7f636c1545d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f635c01b840 tx=0x7f635c01b9d0 comp rx=0 tx=0).stop 2023-10-30T14:16:58.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.681+0000 7f6372654640 1 -- 172.21.15.57:0/2763244649 shutdown_connections 2023-10-30T14:16:58.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.681+0000 7f6372654640 1 --2- 172.21.15.57:0/2763244649 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f63580664d0 0x7f6358068990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:58.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.681+0000 7f6372654640 1 --2- 172.21.15.57:0/2763244649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f636c15a2e0 0x7f636c155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:58.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.681+0000 7f6372654640 1 --2- 172.21.15.57:0/2763244649 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f636c1537f0 0x7f636c0c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:58.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.681+0000 7f6372654640 1 --2- 172.21.15.57:0/2763244649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f636c06b2f0 0x7f636c1545d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:58.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.681+0000 7f6372654640 1 -- 172.21.15.57:0/2763244649 >> 172.21.15.57:0/2763244649 conn(0x7f636c071fb0 msgr2=0x7f636c152810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:58.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.681+0000 7f6372654640 1 -- 172.21.15.57:0/2763244649 shutdown_connections 2023-10-30T14:16:58.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:16:58.681+0000 7f6372654640 1 -- 172.21.15.57:0/2763244649 wait complete. 2023-10-30T14:16:58.705 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:16:59.857 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:16:59.861 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:17:00.706 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:00.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.857+0000 7f5194a17640 1 Processor -- start 2023-10-30T14:17:00.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.857+0000 7f5194a17640 1 -- start start 2023-10-30T14:17:00.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.857+0000 7f5194a17640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51900c4a10 0x7f519015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:00.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.857+0000 7f5194a17640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f51900c42a0 0x7f51900ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:00.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.857+0000 7f5194a17640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5190159f20 con 0x7f5190153c80 2023-10-30T14:17:00.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.857+0000 7f5194a17640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f519015dec0 con 0x7f51900c42a0 2023-10-30T14:17:00.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.857+0000 7f5194a17640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51901599a0 con 0x7f51900c4a10 2023-10-30T14:17:00.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.857+0000 7f518ed9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f51900c42a0 0x7f51900ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:00.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.857+0000 7f518ed9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f51900c42a0 0x7f51900ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46728/0 (socket says 172.21.15.57:46728) 2023-10-30T14:17:00.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.857+0000 7f518ed9d640 1 -- 172.21.15.57:0/3884134636 learned_addr learned my addr 172.21.15.57:0/3884134636 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:00.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.857+0000 7f518dd9b640 1 --2- 172.21.15.57:0/3884134636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51900c4a10 0x7f519015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:00.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.857+0000 7f518ed9d640 1 -- 172.21.15.57:0/3884134636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51900c4a10 msgr2=0x7f519015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:00.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.857+0000 7f518ed9d640 1 --2- 172.21.15.57:0/3884134636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51900c4a10 0x7f519015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:00.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.857+0000 7f518ed9d640 1 -- 172.21.15.57:0/3884134636 >> v1:172.21.15.88:6789/0 conn(0x7f5190153c80 legacy=0x7f519015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:00.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.857+0000 7f518ed9d640 1 -- 172.21.15.57:0/3884134636 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519015d620 con 0x7f51900c42a0 2023-10-30T14:17:00.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.861+0000 7f518ed9d640 1 --2- 172.21.15.57:0/3884134636 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51900c42a0 0x7f51900ca1e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5188009e50 tx=0x7f518801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=310e70ec91f65f7b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:00.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.861+0000 7f518d59a640 1 -- 172.21.15.57:0/3884134636 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51880af400 con 0x7f51900c42a0 2023-10-30T14:17:00.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.861+0000 7f518d59a640 1 -- 172.21.15.57:0/3884134636 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51880ab050 con 0x7f51900c42a0 2023-10-30T14:17:00.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.861+0000 7f518d59a640 1 -- 172.21.15.57:0/3884134636 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51880af780 con 0x7f51900c42a0 2023-10-30T14:17:00.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.861+0000 7f5194a17640 1 -- 172.21.15.57:0/3884134636 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51900c42a0 msgr2=0x7f51900ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:00.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.861+0000 7f5194a17640 1 --2- 172.21.15.57:0/3884134636 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51900c42a0 0x7f51900ca1e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5188009e50 tx=0x7f518801f1f0 comp rx=0 tx=0).stop 2023-10-30T14:17:00.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.861+0000 7f5194a17640 1 -- 172.21.15.57:0/3884134636 shutdown_connections 2023-10-30T14:17:00.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.861+0000 7f5194a17640 1 --2- 172.21.15.57:0/3884134636 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51900c42a0 0x7f51900ca1e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:00.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.861+0000 7f5194a17640 1 --2- 172.21.15.57:0/3884134636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51900c4a10 0x7f519015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:00.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.861+0000 7f5194a17640 1 -- 172.21.15.57:0/3884134636 >> 172.21.15.57:0/3884134636 conn(0x7f51900715c0 msgr2=0x7f51900719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:00.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.861+0000 7f5194a17640 1 -- 172.21.15.57:0/3884134636 shutdown_connections 2023-10-30T14:17:00.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.861+0000 7f5194a17640 1 -- 172.21.15.57:0/3884134636 wait complete. 2023-10-30T14:17:00.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f5194a17640 1 Processor -- start 2023-10-30T14:17:00.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f5194a17640 1 -- start start 2023-10-30T14:17:00.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f5194a17640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51900c42a0 0x7f51901f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:00.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f5194a17640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51900c4a10 0x7f51900b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:00.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f5194a17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5190153c80 0x7f519014f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:00.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f5194a17640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51900b2610 con 0x7f51900c4a10 2023-10-30T14:17:00.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f5194a17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51900b3a60 con 0x7f5190153c80 2023-10-30T14:17:00.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f5194a17640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51901f0630 con 0x7f51900c42a0 2023-10-30T14:17:00.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f518dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51900c4a10 0x7f51900b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:00.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f518dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51900c4a10 0x7f51900b3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46744/0 (socket says 172.21.15.57:46744) 2023-10-30T14:17:00.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f518dd9b640 1 -- 172.21.15.57:0/83377198 learned_addr learned my addr 172.21.15.57:0/83377198 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:00.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f518e59c640 1 --2- 172.21.15.57:0/83377198 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51900c42a0 0x7f51901f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:00.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f518ed9d640 1 --2- 172.21.15.57:0/83377198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5190153c80 0x7f519014f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:00.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f518dd9b640 1 -- 172.21.15.57:0/83377198 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51900c42a0 msgr2=0x7f51901f00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:00.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f518dd9b640 1 --2- 172.21.15.57:0/83377198 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51900c42a0 0x7f51901f00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:00.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f518dd9b640 1 -- 172.21.15.57:0/83377198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5190153c80 msgr2=0x7f519014f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:00.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f518dd9b640 1 --2- 172.21.15.57:0/83377198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5190153c80 0x7f519014f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:00.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f518dd9b640 1 -- 172.21.15.57:0/83377198 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5188009b30 con 0x7f51900c4a10 2023-10-30T14:17:00.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f518ed9d640 1 --2- 172.21.15.57:0/83377198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5190153c80 0x7f519014f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:17:00.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f518dd9b640 1 --2- 172.21.15.57:0/83377198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51900c4a10 0x7f51900b3080 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f518400d830 tx=0x7f518400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:00.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f517effd640 1 -- 172.21.15.57:0/83377198 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518400b6c0 con 0x7f51900c4a10 2023-10-30T14:17:00.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f517effd640 1 -- 172.21.15.57:0/83377198 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f518400b820 con 0x7f51900c4a10 2023-10-30T14:17:00.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f517effd640 1 -- 172.21.15.57:0/83377198 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518400bbd0 con 0x7f51900c4a10 2023-10-30T14:17:00.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f5194a17640 1 -- 172.21.15.57:0/83377198 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51900b3cd0 con 0x7f51900c4a10 2023-10-30T14:17:00.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.865+0000 7f5194a17640 1 -- 172.21.15.57:0/83377198 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51900b4570 con 0x7f51900c4a10 2023-10-30T14:17:00.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.869+0000 7f5194a17640 1 -- 172.21.15.57:0/83377198 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f519004e170 con 0x7f51900c4a10 2023-10-30T14:17:00.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.869+0000 7f517effd640 1 -- 172.21.15.57:0/83377198 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f518401e460 con 0x7f51900c4a10 2023-10-30T14:17:00.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.869+0000 7f517effd640 1 --2- 172.21.15.57:0/83377198 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51700664a0 0x7f5170068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:00.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.869+0000 7f517effd640 1 -- 172.21.15.57:0/83377198 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f5184084660 con 0x7f51900c4a10 2023-10-30T14:17:00.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.869+0000 7f518e59c640 1 --2- 172.21.15.57:0/83377198 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51700664a0 0x7f5170068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:00.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.869+0000 7f518e59c640 1 --2- 172.21.15.57:0/83377198 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51700664a0 0x7f5170068960 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5180001770 tx=0x7f5180009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:00.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:00.873+0000 7f517effd640 1 -- 172.21.15.57:0/83377198 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f51840519b0 con 0x7f51900c4a10 2023-10-30T14:17:01.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:01.061+0000 7f5194a17640 1 -- 172.21.15.57:0/83377198 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f519014e460 con 0x7f51900c4a10 2023-10-30T14:17:01.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:01.065+0000 7f517effd640 1 -- 172.21.15.57:0/83377198 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f5184054fd0 con 0x7f51900c4a10 2023-10-30T14:17:01.068 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:01.068 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:00.637245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:01.069 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 23 2023-10-30T14:17:01.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:01.065+0000 7f5194a17640 1 -- 172.21.15.57:0/83377198 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51700664a0 msgr2=0x7f5170068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:01.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:01.065+0000 7f5194a17640 1 --2- 172.21.15.57:0/83377198 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51700664a0 0x7f5170068960 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5180001770 tx=0x7f5180009040 comp rx=0 tx=0).stop 2023-10-30T14:17:01.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:01.065+0000 7f5194a17640 1 -- 172.21.15.57:0/83377198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51900c4a10 msgr2=0x7f51900b3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:01.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:01.065+0000 7f5194a17640 1 --2- 172.21.15.57:0/83377198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51900c4a10 0x7f51900b3080 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f518400d830 tx=0x7f518400dcf0 comp rx=0 tx=0).stop 2023-10-30T14:17:01.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:01.065+0000 7f5194a17640 1 -- 172.21.15.57:0/83377198 shutdown_connections 2023-10-30T14:17:01.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:01.065+0000 7f5194a17640 1 --2- 172.21.15.57:0/83377198 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51700664a0 0x7f5170068960 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:01.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:01.065+0000 7f5194a17640 1 --2- 172.21.15.57:0/83377198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5190153c80 0x7f519014f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:01.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:01.065+0000 7f5194a17640 1 --2- 172.21.15.57:0/83377198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51900c4a10 0x7f51900b3080 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:01.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:01.065+0000 7f5194a17640 1 --2- 172.21.15.57:0/83377198 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51900c42a0 0x7f51901f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:01.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:01.065+0000 7f5194a17640 1 -- 172.21.15.57:0/83377198 >> 172.21.15.57:0/83377198 conn(0x7f51900715c0 msgr2=0x7f51900719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:01.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:01.065+0000 7f5194a17640 1 -- 172.21.15.57:0/83377198 shutdown_connections 2023-10-30T14:17:01.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:01.065+0000 7f5194a17640 1 -- 172.21.15.57:0/83377198 wait complete. 2023-10-30T14:17:01.089 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:17:03.090 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:03.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.241+0000 7ffacaa68640 1 Processor -- start 2023-10-30T14:17:03.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.241+0000 7ffacaa68640 1 -- start start 2023-10-30T14:17:03.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.241+0000 7ffacaa68640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ffac40701a0 0x7ffac415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:03.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.241+0000 7ffacaa68640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ffac406b2f0 0x7ffac406b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:03.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.241+0000 7ffacaa68640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac40c2aa0 con 0x7ffac41537f0 2023-10-30T14:17:03.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.241+0000 7ffacaa68640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac40c3ab0 con 0x7ffac40701a0 2023-10-30T14:17:03.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.241+0000 7ffacaa68640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac415a820 con 0x7ffac406b2f0 2023-10-30T14:17:03.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.241+0000 7ffac3fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ffac40701a0 0x7ffac415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:03.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.241+0000 7ffac3fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ffac40701a0 0x7ffac415a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46760/0 (socket says 172.21.15.57:46760) 2023-10-30T14:17:03.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.241+0000 7ffac3fff640 1 -- 172.21.15.57:0/276849941 learned_addr learned my addr 172.21.15.57:0/276849941 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:03.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.241+0000 7ffac9005640 1 --2- 172.21.15.57:0/276849941 >> v2:172.21.15.88:3300/0 conn(0x7ffac406b2f0 0x7ffac406b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:03.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.241+0000 7ffac3fff640 1 -- 172.21.15.57:0/276849941 >> v2:172.21.15.88:3300/0 conn(0x7ffac406b2f0 msgr2=0x7ffac406b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:03.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.241+0000 7ffac3fff640 1 --2- 172.21.15.57:0/276849941 >> v2:172.21.15.88:3300/0 conn(0x7ffac406b2f0 0x7ffac406b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:03.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.241+0000 7ffac3fff640 1 -- 172.21.15.57:0/276849941 >> v1:172.21.15.88:6789/0 conn(0x7ffac41537f0 legacy=0x7ffac40ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:03.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.241+0000 7ffac3fff640 1 -- 172.21.15.57:0/276849941 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac40708e0 con 0x7ffac40701a0 2023-10-30T14:17:03.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.245+0000 7ffac3fff640 1 --2- 172.21.15.57:0/276849941 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffac40701a0 0x7ffac415a2e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ffac406f4d0 tx=0x7ffab001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e8a89743864c758 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:03.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.245+0000 7ffac37fe640 1 -- 172.21.15.57:0/276849941 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab00a89f0 con 0x7ffac40701a0 2023-10-30T14:17:03.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.245+0000 7ffac37fe640 1 -- 172.21.15.57:0/276849941 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab00a8b50 con 0x7ffac40701a0 2023-10-30T14:17:03.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.245+0000 7ffac37fe640 1 -- 172.21.15.57:0/276849941 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab00b0440 con 0x7ffac40701a0 2023-10-30T14:17:03.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.245+0000 7ffacaa68640 1 -- 172.21.15.57:0/276849941 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffac40701a0 msgr2=0x7ffac415a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:03.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.245+0000 7ffacaa68640 1 --2- 172.21.15.57:0/276849941 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffac40701a0 0x7ffac415a2e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ffac406f4d0 tx=0x7ffab001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:17:03.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.245+0000 7ffacaa68640 1 -- 172.21.15.57:0/276849941 shutdown_connections 2023-10-30T14:17:03.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.245+0000 7ffacaa68640 1 --2- 172.21.15.57:0/276849941 >> v2:172.21.15.88:3300/0 conn(0x7ffac406b2f0 0x7ffac406b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:03.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.245+0000 7ffacaa68640 1 --2- 172.21.15.57:0/276849941 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffac40701a0 0x7ffac415a2e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:03.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.245+0000 7ffacaa68640 1 -- 172.21.15.57:0/276849941 >> 172.21.15.57:0/276849941 conn(0x7ffac4071fb0 msgr2=0x7ffac40723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:03.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.245+0000 7ffacaa68640 1 -- 172.21.15.57:0/276849941 shutdown_connections 2023-10-30T14:17:03.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.245+0000 7ffacaa68640 1 -- 172.21.15.57:0/276849941 wait complete. 2023-10-30T14:17:03.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffacaa68640 1 Processor -- start 2023-10-30T14:17:03.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffacaa68640 1 -- start start 2023-10-30T14:17:03.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffacaa68640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffac406b2f0 0x7ffac414f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:03.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffacaa68640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffac40701a0 0x7ffac40c5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:03.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffacaa68640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffac41537f0 0x7ffac40c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:03.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffacaa68640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac414ede0 con 0x7ffac406b2f0 2023-10-30T14:17:03.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffacaa68640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac414fd30 con 0x7ffac41537f0 2023-10-30T14:17:03.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffacaa68640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac414e4b0 con 0x7ffac40701a0 2023-10-30T14:17:03.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffac8804640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffac406b2f0 0x7ffac414f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:03.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffac8804640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffac406b2f0 0x7ffac414f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46774/0 (socket says 172.21.15.57:46774) 2023-10-30T14:17:03.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffac8804640 1 -- 172.21.15.57:0/2390955945 learned_addr learned my addr 172.21.15.57:0/2390955945 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:03.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffac3fff640 1 --2- 172.21.15.57:0/2390955945 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffac40701a0 0x7ffac40c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:03.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffac9005640 1 --2- 172.21.15.57:0/2390955945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffac41537f0 0x7ffac40c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:03.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffac8804640 1 -- 172.21.15.57:0/2390955945 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffac40701a0 msgr2=0x7ffac40c5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:03.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffac8804640 1 --2- 172.21.15.57:0/2390955945 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffac40701a0 0x7ffac40c5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:03.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffac8804640 1 -- 172.21.15.57:0/2390955945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffac41537f0 msgr2=0x7ffac40c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:03.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffac8804640 1 --2- 172.21.15.57:0/2390955945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffac41537f0 0x7ffac40c5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:03.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffac8804640 1 -- 172.21.15.57:0/2390955945 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab00094d0 con 0x7ffac406b2f0 2023-10-30T14:17:03.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffac8804640 1 --2- 172.21.15.57:0/2390955945 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffac406b2f0 0x7ffac414f680 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ffab800c500 tx=0x7ffab800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:03.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffac17fa640 1 -- 172.21.15.57:0/2390955945 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab8005220 con 0x7ffac406b2f0 2023-10-30T14:17:03.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffac17fa640 1 -- 172.21.15.57:0/2390955945 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab8005380 con 0x7ffac406b2f0 2023-10-30T14:17:03.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffacaa68640 1 -- 172.21.15.57:0/2390955945 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac414e790 con 0x7ffac406b2f0 2023-10-30T14:17:03.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffac17fa640 1 -- 172.21.15.57:0/2390955945 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab8007270 con 0x7ffac406b2f0 2023-10-30T14:17:03.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.249+0000 7ffacaa68640 1 -- 172.21.15.57:0/2390955945 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac414e1a0 con 0x7ffac406b2f0 2023-10-30T14:17:03.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.253+0000 7ffacaa68640 1 -- 172.21.15.57:0/2390955945 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac4158f80 con 0x7ffac406b2f0 2023-10-30T14:17:03.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.253+0000 7ffac17fa640 1 -- 172.21.15.57:0/2390955945 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffab80073d0 con 0x7ffac406b2f0 2023-10-30T14:17:03.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.253+0000 7ffac17fa640 1 --2- 172.21.15.57:0/2390955945 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffaa80664a0 0x7ffaa8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:03.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.253+0000 7ffac17fa640 1 -- 172.21.15.57:0/2390955945 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7ffab80838e0 con 0x7ffac406b2f0 2023-10-30T14:17:03.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.253+0000 7ffac3fff640 1 --2- 172.21.15.57:0/2390955945 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffaa80664a0 0x7ffaa8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:03.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.257+0000 7ffac3fff640 1 --2- 172.21.15.57:0/2390955945 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffaa80664a0 0x7ffaa8068960 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ffab00b2000 tx=0x7ffab0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:03.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.257+0000 7ffac17fa640 1 -- 172.21.15.57:0/2390955945 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffab8050bb0 con 0x7ffac406b2f0 2023-10-30T14:17:03.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.449+0000 7ffacaa68640 1 -- 172.21.15.57:0/2390955945 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffac404e170 con 0x7ffac406b2f0 2023-10-30T14:17:03.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.449+0000 7ffac17fa640 1 -- 172.21.15.57:0/2390955945 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5737 (secure 0 0 0) 0x7ffab80541d0 con 0x7ffac406b2f0 2023-10-30T14:17:03.453 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:03.453 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:00.637245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:03.453 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 23 2023-10-30T14:17:03.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.449+0000 7ffacaa68640 1 -- 172.21.15.57:0/2390955945 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffaa80664a0 msgr2=0x7ffaa8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:03.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.449+0000 7ffacaa68640 1 --2- 172.21.15.57:0/2390955945 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffaa80664a0 0x7ffaa8068960 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ffab00b2000 tx=0x7ffab0009040 comp rx=0 tx=0).stop 2023-10-30T14:17:03.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.449+0000 7ffacaa68640 1 -- 172.21.15.57:0/2390955945 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffac406b2f0 msgr2=0x7ffac414f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:03.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.449+0000 7ffacaa68640 1 --2- 172.21.15.57:0/2390955945 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffac406b2f0 0x7ffac414f680 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ffab800c500 tx=0x7ffab800c9c0 comp rx=0 tx=0).stop 2023-10-30T14:17:03.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.449+0000 7ffacaa68640 1 -- 172.21.15.57:0/2390955945 shutdown_connections 2023-10-30T14:17:03.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.449+0000 7ffacaa68640 1 --2- 172.21.15.57:0/2390955945 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffaa80664a0 0x7ffaa8068960 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:03.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.449+0000 7ffacaa68640 1 --2- 172.21.15.57:0/2390955945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffac41537f0 0x7ffac40c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:03.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.449+0000 7ffacaa68640 1 --2- 172.21.15.57:0/2390955945 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffac40701a0 0x7ffac40c5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:03.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.449+0000 7ffacaa68640 1 --2- 172.21.15.57:0/2390955945 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffac406b2f0 0x7ffac414f680 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:03.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.449+0000 7ffacaa68640 1 -- 172.21.15.57:0/2390955945 >> 172.21.15.57:0/2390955945 conn(0x7ffac4071fb0 msgr2=0x7ffac406be50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:03.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.449+0000 7ffacaa68640 1 -- 172.21.15.57:0/2390955945 shutdown_connections 2023-10-30T14:17:03.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:03.449+0000 7ffacaa68640 1 -- 172.21.15.57:0/2390955945 wait complete. 2023-10-30T14:17:03.475 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 23, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 23, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4266": { "addr": "172.21.15.57:6836/2693548867", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2693548867, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2693548867, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4266, "incarnation": 9, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4282": { "addr": "172.21.15.88:6839/2945854794", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 2945854794, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2945854794, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4936": { "addr": "172.21.15.57:6837/1988100930", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 1988100930, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 1988100930, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4936, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:reconnect", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:17:00.637245+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 4266, "mds_2": 4936 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:17:05.476 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:05.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a96246640 1 Processor -- start 2023-10-30T14:17:05.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a96246640 1 -- start start 2023-10-30T14:17:05.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a96246640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8a90153c80 0x7f8a9015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:05.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a96246640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8a900c4a10 0x7f8a900c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:05.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a96246640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a9015a3b0 0x7f8a900ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:05.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a96246640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a9015dec0 con 0x7f8a900c4a10 2023-10-30T14:17:05.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a96246640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a901599a0 con 0x7f8a90153c80 2023-10-30T14:17:05.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a96246640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a90159f20 con 0x7f8a9015a3b0 2023-10-30T14:17:05.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a8effd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8a900c4a10 0x7f8a900c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:05.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a8effd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8a900c4a10 0x7f8a900c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44060/0 (socket says 172.21.15.57:44060) 2023-10-30T14:17:05.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a8effd640 1 -- 172.21.15.57:0/3297894448 learned_addr learned my addr 172.21.15.57:0/3297894448 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:05.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a8f7fe640 1 --2- 172.21.15.57:0/3297894448 >> v2:172.21.15.88:3300/0 conn(0x7f8a90153c80 0x7f8a9015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:05.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a8ffff640 1 --2- 172.21.15.57:0/3297894448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a9015a3b0 0x7f8a900ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:05.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a8effd640 1 -- 172.21.15.57:0/3297894448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a9015a3b0 msgr2=0x7f8a900ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:05.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a8effd640 1 --2- 172.21.15.57:0/3297894448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a9015a3b0 0x7f8a900ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:05.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a8effd640 1 -- 172.21.15.57:0/3297894448 >> v2:172.21.15.88:3300/0 conn(0x7f8a90153c80 msgr2=0x7f8a9015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:05.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a8effd640 1 --2- 172.21.15.57:0/3297894448 >> v2:172.21.15.88:3300/0 conn(0x7f8a90153c80 0x7f8a9015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:05.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a8effd640 1 -- 172.21.15.57:0/3297894448 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a900c47e0 con 0x7f8a900c4a10 2023-10-30T14:17:05.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a8f7fe640 1 --2- 172.21.15.57:0/3297894448 >> v2:172.21.15.88:3300/0 conn(0x7f8a90153c80 0x7f8a9015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:17:05.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.629+0000 7f8a8ffff640 1 --2- 172.21.15.57:0/3297894448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a9015a3b0 0x7f8a900ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:17:05.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.633+0000 7f8a8effd640 1 --2- 172.21.15.57:0/3297894448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a900c4a10 0x7f8a900c42a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8a900d00f0 tx=0x7f8a8401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3e74349385cb88 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:05.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.633+0000 7f8a8e7fc640 1 -- 172.21.15.57:0/3297894448 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a840a89b0 con 0x7f8a900c4a10 2023-10-30T14:17:05.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.633+0000 7f8a8e7fc640 1 -- 172.21.15.57:0/3297894448 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a840a8b10 con 0x7f8a900c4a10 2023-10-30T14:17:05.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.633+0000 7f8a8e7fc640 1 -- 172.21.15.57:0/3297894448 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a840a8ea0 con 0x7f8a900c4a10 2023-10-30T14:17:05.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.633+0000 7f8a96246640 1 -- 172.21.15.57:0/3297894448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a900c4a10 msgr2=0x7f8a900c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:05.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.633+0000 7f8a96246640 1 --2- 172.21.15.57:0/3297894448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a900c4a10 0x7f8a900c42a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8a900d00f0 tx=0x7f8a8401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:17:05.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.633+0000 7f8a96246640 1 -- 172.21.15.57:0/3297894448 shutdown_connections 2023-10-30T14:17:05.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.633+0000 7f8a96246640 1 --2- 172.21.15.57:0/3297894448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a9015a3b0 0x7f8a900ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:05.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.633+0000 7f8a96246640 1 --2- 172.21.15.57:0/3297894448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a900c4a10 0x7f8a900c42a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:05.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.633+0000 7f8a96246640 1 --2- 172.21.15.57:0/3297894448 >> v2:172.21.15.88:3300/0 conn(0x7f8a90153c80 0x7f8a9015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:05.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.633+0000 7f8a96246640 1 -- 172.21.15.57:0/3297894448 >> 172.21.15.57:0/3297894448 conn(0x7f8a900715c0 msgr2=0x7f8a900719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:05.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.633+0000 7f8a96246640 1 -- 172.21.15.57:0/3297894448 shutdown_connections 2023-10-30T14:17:05.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.633+0000 7f8a96246640 1 -- 172.21.15.57:0/3297894448 wait complete. 2023-10-30T14:17:05.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a96246640 1 Processor -- start 2023-10-30T14:17:05.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a96246640 1 -- start start 2023-10-30T14:17:05.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a96246640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a900c4a10 0x7f8a901586d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:05.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a96246640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a90153c80 0x7f8a9015ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:05.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a96246640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a9015a3b0 0x7f8a9015b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:05.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a96246640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a901ee6f0 con 0x7f8a9015a3b0 2023-10-30T14:17:05.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a96246640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a901ee8e0 con 0x7f8a900c4a10 2023-10-30T14:17:05.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a96246640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a901eebc0 con 0x7f8a90153c80 2023-10-30T14:17:05.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a8ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a9015a3b0 0x7f8a9015b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:05.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a8ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a9015a3b0 0x7f8a9015b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44074/0 (socket says 172.21.15.57:44074) 2023-10-30T14:17:05.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a8f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a900c4a10 0x7f8a901586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:05.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a8ffff640 1 -- 172.21.15.57:0/1983909752 learned_addr learned my addr 172.21.15.57:0/1983909752 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:05.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a8effd640 1 --2- 172.21.15.57:0/1983909752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a90153c80 0x7f8a9015ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:05.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a8ffff640 1 -- 172.21.15.57:0/1983909752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a90153c80 msgr2=0x7f8a9015ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:05.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a8ffff640 1 --2- 172.21.15.57:0/1983909752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a90153c80 0x7f8a9015ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:05.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a8ffff640 1 -- 172.21.15.57:0/1983909752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a900c4a10 msgr2=0x7f8a901586d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:05.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a8ffff640 1 --2- 172.21.15.57:0/1983909752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a900c4a10 0x7f8a901586d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:05.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a8ffff640 1 -- 172.21.15.57:0/1983909752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a840094d0 con 0x7f8a9015a3b0 2023-10-30T14:17:05.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a8ffff640 1 --2- 172.21.15.57:0/1983909752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a9015a3b0 0x7f8a9015b0a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8a88007df0 tx=0x7f8a8800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:05.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a77fff640 1 -- 172.21.15.57:0/1983909752 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a880039a0 con 0x7f8a9015a3b0 2023-10-30T14:17:05.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a77fff640 1 -- 172.21.15.57:0/1983909752 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8800f040 con 0x7f8a9015a3b0 2023-10-30T14:17:05.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a77fff640 1 -- 172.21.15.57:0/1983909752 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a88003d40 con 0x7f8a9015a3b0 2023-10-30T14:17:05.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a96246640 1 -- 172.21.15.57:0/1983909752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a901eeea0 con 0x7f8a9015a3b0 2023-10-30T14:17:05.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.637+0000 7f8a96246640 1 -- 172.21.15.57:0/1983909752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a901ef460 con 0x7f8a9015a3b0 2023-10-30T14:17:05.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.641+0000 7f8a96246640 1 -- 172.21.15.57:0/1983909752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a901ef650 con 0x7f8a9015a3b0 2023-10-30T14:17:05.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.641+0000 7f8a77fff640 1 -- 172.21.15.57:0/1983909752 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8a88004030 con 0x7f8a9015a3b0 2023-10-30T14:17:05.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.641+0000 7f8a77fff640 1 --2- 172.21.15.57:0/1983909752 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8a7c0664a0 0x7f8a7c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:05.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.641+0000 7f8a77fff640 1 -- 172.21.15.57:0/1983909752 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f8a88084310 con 0x7f8a9015a3b0 2023-10-30T14:17:05.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.641+0000 7f8a8f7fe640 1 --2- 172.21.15.57:0/1983909752 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8a7c0664a0 0x7f8a7c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:05.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.641+0000 7f8a8f7fe640 1 --2- 172.21.15.57:0/1983909752 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8a7c0664a0 0x7f8a7c068960 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8a800098d0 tx=0x7f8a80008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:05.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.645+0000 7f8a77fff640 1 -- 172.21.15.57:0/1983909752 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8a880515f0 con 0x7f8a9015a3b0 2023-10-30T14:17:05.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.837+0000 7f8a96246640 1 -- 172.21.15.57:0/1983909752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a9004e170 con 0x7f8a9015a3b0 2023-10-30T14:17:05.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.837+0000 7f8a77fff640 1 -- 172.21.15.57:0/1983909752 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f8a88054c10 con 0x7f8a9015a3b0 2023-10-30T14:17:05.841 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:05.841 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:00.637245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:05.841 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 23 2023-10-30T14:17:05.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.837+0000 7f8a96246640 1 -- 172.21.15.57:0/1983909752 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8a7c0664a0 msgr2=0x7f8a7c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:05.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.837+0000 7f8a96246640 1 --2- 172.21.15.57:0/1983909752 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8a7c0664a0 0x7f8a7c068960 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8a800098d0 tx=0x7f8a80008040 comp rx=0 tx=0).stop 2023-10-30T14:17:05.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.837+0000 7f8a96246640 1 -- 172.21.15.57:0/1983909752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a9015a3b0 msgr2=0x7f8a9015b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:05.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.837+0000 7f8a96246640 1 --2- 172.21.15.57:0/1983909752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a9015a3b0 0x7f8a9015b0a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8a88007df0 tx=0x7f8a8800d4a0 comp rx=0 tx=0).stop 2023-10-30T14:17:05.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.837+0000 7f8a96246640 1 -- 172.21.15.57:0/1983909752 shutdown_connections 2023-10-30T14:17:05.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.837+0000 7f8a96246640 1 --2- 172.21.15.57:0/1983909752 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8a7c0664a0 0x7f8a7c068960 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:05.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.837+0000 7f8a96246640 1 --2- 172.21.15.57:0/1983909752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a9015a3b0 0x7f8a9015b0a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:05.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.837+0000 7f8a96246640 1 --2- 172.21.15.57:0/1983909752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a90153c80 0x7f8a9015ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:05.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.837+0000 7f8a96246640 1 --2- 172.21.15.57:0/1983909752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a900c4a10 0x7f8a901586d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:05.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.837+0000 7f8a96246640 1 -- 172.21.15.57:0/1983909752 >> 172.21.15.57:0/1983909752 conn(0x7f8a900715c0 msgr2=0x7f8a900c5680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:05.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.837+0000 7f8a96246640 1 -- 172.21.15.57:0/1983909752 shutdown_connections 2023-10-30T14:17:05.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:05.837+0000 7f8a96246640 1 -- 172.21.15.57:0/1983909752 wait complete. 2023-10-30T14:17:05.862 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:17:07.863 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:08.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.013+0000 7f71b4e09640 1 Processor -- start 2023-10-30T14:17:08.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.013+0000 7f71b4e09640 1 -- start start 2023-10-30T14:17:08.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.017+0000 7f71b4e09640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f71b00c5ba0 0x7f71b00c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:08.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.017+0000 7f71b4e09640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71b00c5f80 0x7f71b0069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:08.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.017+0000 7f71b4e09640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b015cda0 con 0x7f71b00c9b40 2023-10-30T14:17:08.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.017+0000 7f71b4e09640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71b015d990 con 0x7f71b00c5ba0 2023-10-30T14:17:08.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.017+0000 7f71b4e09640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b0153f60 con 0x7f71b00c5f80 2023-10-30T14:17:08.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.017+0000 7f71aed9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f71b00c9b40 0x7f71b01597b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50130/0 (socket says 172.21.15.57:50130) 2023-10-30T14:17:08.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.017+0000 7f71aed9d640 1 -- 172.21.15.57:0/3245405894 learned_addr learned my addr 172.21.15.57:0/3245405894 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:17:08.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.017+0000 7f71a7fff640 1 --2- 172.21.15.57:0/3245405894 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71b00c5f80 0x7f71b0069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:08.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.017+0000 7f71ae59c640 1 --2- 172.21.15.57:0/3245405894 >> v2:172.21.15.88:3300/0 conn(0x7f71b00c5ba0 0x7f71b00c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:08.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.017+0000 7f71add9b640 1 -- 172.21.15.57:0/3245405894 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991494283 0 0) 0x7f71b015cda0 con 0x7f71b00c9b40 2023-10-30T14:17:08.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.017+0000 7f71add9b640 1 -- 172.21.15.57:0/3245405894 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f719c003530 con 0x7f71b00c9b40 2023-10-30T14:17:08.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.017+0000 7f71ae59c640 1 -- 172.21.15.57:0/3245405894 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71b00c5f80 msgr2=0x7f71b0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:08.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.017+0000 7f71ae59c640 1 --2- 172.21.15.57:0/3245405894 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71b00c5f80 0x7f71b0069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:08.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.017+0000 7f71ae59c640 1 -- 172.21.15.57:0/3245405894 >> v1:172.21.15.57:6789/0 conn(0x7f71b00c9b40 legacy=0x7f71b01597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:08.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.017+0000 7f71ae59c640 1 -- 172.21.15.57:0/3245405894 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b006f490 con 0x7f71b00c5ba0 2023-10-30T14:17:08.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.021+0000 7f71ae59c640 1 --2- 172.21.15.57:0/3245405894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71b00c5ba0 0x7f71b00c70f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f71a00097f0 tx=0x7f71a001c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d815742370ba73c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:08.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.021+0000 7f71add9b640 1 -- 172.21.15.57:0/3245405894 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a00ab600 con 0x7f71b00c5ba0 2023-10-30T14:17:08.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.021+0000 7f71add9b640 1 -- 172.21.15.57:0/3245405894 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a00a7050 con 0x7f71b00c5ba0 2023-10-30T14:17:08.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.021+0000 7f71add9b640 1 -- 172.21.15.57:0/3245405894 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a00ab960 con 0x7f71b00c5ba0 2023-10-30T14:17:08.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.021+0000 7f71b4e09640 1 -- 172.21.15.57:0/3245405894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71b00c5ba0 msgr2=0x7f71b00c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:08.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.021+0000 7f71b4e09640 1 --2- 172.21.15.57:0/3245405894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71b00c5ba0 0x7f71b00c70f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f71a00097f0 tx=0x7f71a001c3f0 comp rx=0 tx=0).stop 2023-10-30T14:17:08.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.021+0000 7f71b4e09640 1 -- 172.21.15.57:0/3245405894 shutdown_connections 2023-10-30T14:17:08.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.021+0000 7f71b4e09640 1 --2- 172.21.15.57:0/3245405894 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71b00c5f80 0x7f71b0069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:08.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.021+0000 7f71b4e09640 1 --2- 172.21.15.57:0/3245405894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71b00c5ba0 0x7f71b00c70f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:08.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.021+0000 7f71b4e09640 1 -- 172.21.15.57:0/3245405894 >> 172.21.15.57:0/3245405894 conn(0x7f71b006afa0 msgr2=0x7f71b006b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:08.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.021+0000 7f71b4e09640 1 -- 172.21.15.57:0/3245405894 shutdown_connections 2023-10-30T14:17:08.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.021+0000 7f71b4e09640 1 -- 172.21.15.57:0/3245405894 wait complete. 2023-10-30T14:17:08.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.021+0000 7f71b4e09640 1 Processor -- start 2023-10-30T14:17:08.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.021+0000 7f71b4e09640 1 -- start start 2023-10-30T14:17:08.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71b4e09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71b00c5ba0 0x7f71b01536d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:08.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71b4e09640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71b00c5f80 0x7f71b01f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:08.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71b4e09640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71b00c9b40 0x7f71b0156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:08.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71b4e09640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b01560e0 con 0x7f71b00c5f80 2023-10-30T14:17:08.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71b4e09640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b0157900 con 0x7f71b00c5ba0 2023-10-30T14:17:08.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71b4e09640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b01f5d90 con 0x7f71b00c9b40 2023-10-30T14:17:08.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71a7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71b00c5f80 0x7f71b01f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:08.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71a7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71b00c5f80 0x7f71b01f2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44076/0 (socket says 172.21.15.57:44076) 2023-10-30T14:17:08.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71a7fff640 1 -- 172.21.15.57:0/1312231365 learned_addr learned my addr 172.21.15.57:0/1312231365 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:08.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71ae59c640 1 --2- 172.21.15.57:0/1312231365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71b00c5ba0 0x7f71b01536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:08.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71aed9d640 1 --2- 172.21.15.57:0/1312231365 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71b00c9b40 0x7f71b0156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:08.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71a7fff640 1 -- 172.21.15.57:0/1312231365 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71b00c9b40 msgr2=0x7f71b0156b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:08.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71a7fff640 1 --2- 172.21.15.57:0/1312231365 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71b00c9b40 0x7f71b0156b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:08.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71a7fff640 1 -- 172.21.15.57:0/1312231365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71b00c5ba0 msgr2=0x7f71b01536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:08.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71a7fff640 1 --2- 172.21.15.57:0/1312231365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71b00c5ba0 0x7f71b01536d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:08.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71a7fff640 1 -- 172.21.15.57:0/1312231365 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a0009420 con 0x7f71b00c5f80 2023-10-30T14:17:08.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71a7fff640 1 --2- 172.21.15.57:0/1312231365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71b00c5f80 0x7f71b01f2f20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f719800e870 tx=0x7f719800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:08.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71a6ffd640 1 -- 172.21.15.57:0/1312231365 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7198009510 con 0x7f71b00c5f80 2023-10-30T14:17:08.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71a6ffd640 1 -- 172.21.15.57:0/1312231365 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719800eec0 con 0x7f71b00c5f80 2023-10-30T14:17:08.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71b4e09640 1 -- 172.21.15.57:0/1312231365 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b0156330 con 0x7f71b00c5f80 2023-10-30T14:17:08.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71a6ffd640 1 -- 172.21.15.57:0/1312231365 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719800cb40 con 0x7f71b00c5f80 2023-10-30T14:17:08.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71b4e09640 1 -- 172.21.15.57:0/1312231365 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b01575c0 con 0x7f71b00c5f80 2023-10-30T14:17:08.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71b4e09640 1 -- 172.21.15.57:0/1312231365 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b00ca2f0 con 0x7f71b00c5f80 2023-10-30T14:17:08.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.025+0000 7f71a6ffd640 1 -- 172.21.15.57:0/1312231365 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f719800cca0 con 0x7f71b00c5f80 2023-10-30T14:17:08.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.029+0000 7f71a6ffd640 1 --2- 172.21.15.57:0/1312231365 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f71900664d0 0x7f7190068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:08.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.029+0000 7f71a6ffd640 1 -- 172.21.15.57:0/1312231365 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f7198084150 con 0x7f71b00c5f80 2023-10-30T14:17:08.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.029+0000 7f71ae59c640 1 --2- 172.21.15.57:0/1312231365 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f71900664d0 0x7f7190068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:08.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.029+0000 7f71ae59c640 1 --2- 172.21.15.57:0/1312231365 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f71900664d0 0x7f7190068990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f71a00ad000 tx=0x7f71a00a6000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:08.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.029+0000 7f71a6ffd640 1 -- 172.21.15.57:0/1312231365 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7198051380 con 0x7f71b00c5f80 2023-10-30T14:17:08.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.221+0000 7f71b4e09640 1 -- 172.21.15.57:0/1312231365 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71b015d5b0 con 0x7f71b00c5f80 2023-10-30T14:17:08.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.221+0000 7f71a6ffd640 1 -- 172.21.15.57:0/1312231365 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f71980549a0 con 0x7f71b00c5f80 2023-10-30T14:17:08.227 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:08.228 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:05.905427+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:08.229 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 24 2023-10-30T14:17:08.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.225+0000 7f71b4e09640 1 -- 172.21.15.57:0/1312231365 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f71900664d0 msgr2=0x7f7190068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:08.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.225+0000 7f71b4e09640 1 --2- 172.21.15.57:0/1312231365 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f71900664d0 0x7f7190068990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f71a00ad000 tx=0x7f71a00a6000 comp rx=0 tx=0).stop 2023-10-30T14:17:08.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.225+0000 7f71b4e09640 1 -- 172.21.15.57:0/1312231365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71b00c5f80 msgr2=0x7f71b01f2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:08.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.225+0000 7f71b4e09640 1 --2- 172.21.15.57:0/1312231365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71b00c5f80 0x7f71b01f2f20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f719800e870 tx=0x7f719800ec70 comp rx=0 tx=0).stop 2023-10-30T14:17:08.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.225+0000 7f71b4e09640 1 -- 172.21.15.57:0/1312231365 shutdown_connections 2023-10-30T14:17:08.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.225+0000 7f71b4e09640 1 --2- 172.21.15.57:0/1312231365 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f71900664d0 0x7f7190068990 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:08.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.225+0000 7f71b4e09640 1 --2- 172.21.15.57:0/1312231365 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71b00c9b40 0x7f71b0156b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:08.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.225+0000 7f71b4e09640 1 --2- 172.21.15.57:0/1312231365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71b00c5f80 0x7f71b01f2f20 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:08.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.225+0000 7f71b4e09640 1 --2- 172.21.15.57:0/1312231365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71b00c5ba0 0x7f71b01536d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:08.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.225+0000 7f71b4e09640 1 -- 172.21.15.57:0/1312231365 >> 172.21.15.57:0/1312231365 conn(0x7f71b006afa0 msgr2=0x7f71b00c8700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:08.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.225+0000 7f71b4e09640 1 -- 172.21.15.57:0/1312231365 shutdown_connections 2023-10-30T14:17:08.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:08.225+0000 7f71b4e09640 1 -- 172.21.15.57:0/1312231365 wait complete. 2023-10-30T14:17:08.248 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:17:10.250 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:10.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.401+0000 7f3fe5abb640 1 Processor -- start 2023-10-30T14:17:10.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.401+0000 7f3fe5abb640 1 -- start start 2023-10-30T14:17:10.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.401+0000 7f3fe5abb640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3fe015ac70 0x7f3fe015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:10.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.401+0000 7f3fe5abb640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe015dcc0 con 0x7f3fe00c8eb0 2023-10-30T14:17:10.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.401+0000 7f3fe5abb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe0159d80 con 0x7f3fe006aeb0 2023-10-30T14:17:10.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.401+0000 7f3fe5abb640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe0159800 con 0x7f3fe015ac70 2023-10-30T14:17:10.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.401+0000 7f3fdffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3fe015ac70 0x7f3fe015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:10.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.401+0000 7f3fe4ab9640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f3fe00c8eb0 0x7f3fe015a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50146/0 (socket says 172.21.15.57:50146) 2023-10-30T14:17:10.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.401+0000 7f3fe4ab9640 1 -- 172.21.15.57:0/4210433126 learned_addr learned my addr 172.21.15.57:0/4210433126 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:17:10.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.401+0000 7f3fdffff640 1 -- 172.21.15.57:0/4210433126 >> v1:172.21.15.88:6789/0 conn(0x7f3fe006aeb0 legacy=0x7f3fe0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:10.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.401+0000 7f3fdffff640 1 -- 172.21.15.57:0/4210433126 >> v1:172.21.15.57:6789/0 conn(0x7f3fe00c8eb0 legacy=0x7f3fe015a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:10.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.401+0000 7f3fdffff640 1 -- 172.21.15.57:0/4210433126 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe015d420 con 0x7f3fe015ac70 2023-10-30T14:17:10.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.401+0000 7f3fdf7fe640 1 -- 172.21.15.57:0/4210433126 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753495756 0 0) 0x7f3fd4001120 con 0x7f3fe00c8eb0 2023-10-30T14:17:10.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.405+0000 7f3fdffff640 1 --2- 172.21.15.57:0/4210433126 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fe015ac70 0x7f3fe015cd70 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc0097c0 tx=0x7f3fcc01ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4556c2b970ef15d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:10.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.405+0000 7f3fdf7fe640 1 -- 172.21.15.57:0/4210433126 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc0a8930 con 0x7f3fe015ac70 2023-10-30T14:17:10.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.405+0000 7f3fdf7fe640 1 -- 172.21.15.57:0/4210433126 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fcc0a8a90 con 0x7f3fe015ac70 2023-10-30T14:17:10.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.405+0000 7f3fdf7fe640 1 -- 172.21.15.57:0/4210433126 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc0a8e20 con 0x7f3fe015ac70 2023-10-30T14:17:10.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.405+0000 7f3fe5abb640 1 -- 172.21.15.57:0/4210433126 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fe015ac70 msgr2=0x7f3fe015cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:10.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.405+0000 7f3fe5abb640 1 --2- 172.21.15.57:0/4210433126 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fe015ac70 0x7f3fe015cd70 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc0097c0 tx=0x7f3fcc01ec30 comp rx=0 tx=0).stop 2023-10-30T14:17:10.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.405+0000 7f3fe5abb640 1 -- 172.21.15.57:0/4210433126 shutdown_connections 2023-10-30T14:17:10.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.405+0000 7f3fe5abb640 1 --2- 172.21.15.57:0/4210433126 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fe015ac70 0x7f3fe015cd70 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:10.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.405+0000 7f3fe5abb640 1 -- 172.21.15.57:0/4210433126 >> 172.21.15.57:0/4210433126 conn(0x7f3fe00cef00 msgr2=0x7f3fe00cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:10.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.405+0000 7f3fe5abb640 1 -- 172.21.15.57:0/4210433126 shutdown_connections 2023-10-30T14:17:10.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.405+0000 7f3fe5abb640 1 -- 172.21.15.57:0/4210433126 wait complete. 2023-10-30T14:17:10.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.405+0000 7f3fe5abb640 1 Processor -- start 2023-10-30T14:17:10.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe5abb640 1 -- start start 2023-10-30T14:17:10.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe5abb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fe006aeb0 0x7f3fe00c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:10.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe5abb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fe00c8eb0 0x7f3fe01f0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:10.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe5abb640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fe015ac70 0x7f3fe01f0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:10.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe5abb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe00cdda0 con 0x7f3fe006aeb0 2023-10-30T14:17:10.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe5abb640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe00ce080 con 0x7f3fe00c8eb0 2023-10-30T14:17:10.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe5abb640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe00ce360 con 0x7f3fe015ac70 2023-10-30T14:17:10.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe4ab9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fe006aeb0 0x7f3fe00c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:10.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe4ab9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fe006aeb0 0x7f3fe00c9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44102/0 (socket says 172.21.15.57:44102) 2023-10-30T14:17:10.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe4ab9640 1 -- 172.21.15.57:0/2683070566 learned_addr learned my addr 172.21.15.57:0/2683070566 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:10.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fdffff640 1 --2- 172.21.15.57:0/2683070566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fe00c8eb0 0x7f3fe01f0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:10.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe52ba640 1 --2- 172.21.15.57:0/2683070566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fe015ac70 0x7f3fe01f0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:10.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe4ab9640 1 -- 172.21.15.57:0/2683070566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fe015ac70 msgr2=0x7f3fe01f0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:10.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe4ab9640 1 --2- 172.21.15.57:0/2683070566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fe015ac70 0x7f3fe01f0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:10.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe4ab9640 1 -- 172.21.15.57:0/2683070566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fe00c8eb0 msgr2=0x7f3fe01f0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:10.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe4ab9640 1 --2- 172.21.15.57:0/2683070566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fe00c8eb0 0x7f3fe01f0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:10.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe4ab9640 1 -- 172.21.15.57:0/2683070566 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fcc009420 con 0x7f3fe006aeb0 2023-10-30T14:17:10.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe52ba640 1 --2- 172.21.15.57:0/2683070566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fe015ac70 0x7f3fe01f0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:17:10.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe4ab9640 1 --2- 172.21.15.57:0/2683070566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fe006aeb0 0x7f3fe00c9e10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3fd400c780 tx=0x7f3fd400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:10.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fdd7fa640 1 -- 172.21.15.57:0/2683070566 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd40072d0 con 0x7f3fe006aeb0 2023-10-30T14:17:10.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fdd7fa640 1 -- 172.21.15.57:0/2683070566 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd4007450 con 0x7f3fe006aeb0 2023-10-30T14:17:10.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fdd7fa640 1 -- 172.21.15.57:0/2683070566 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd40047f0 con 0x7f3fe006aeb0 2023-10-30T14:17:10.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe5abb640 1 -- 172.21.15.57:0/2683070566 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe00ce640 con 0x7f3fe006aeb0 2023-10-30T14:17:10.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe5abb640 1 -- 172.21.15.57:0/2683070566 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe00c7480 con 0x7f3fe006aeb0 2023-10-30T14:17:10.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.409+0000 7f3fe5abb640 1 -- 172.21.15.57:0/2683070566 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe00ce880 con 0x7f3fe006aeb0 2023-10-30T14:17:10.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.413+0000 7f3fdd7fa640 1 -- 172.21.15.57:0/2683070566 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3fd4004950 con 0x7f3fe006aeb0 2023-10-30T14:17:10.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.413+0000 7f3fdd7fa640 1 --2- 172.21.15.57:0/2683070566 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3fc40664a0 0x7f3fc4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:10.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.413+0000 7f3fdd7fa640 1 -- 172.21.15.57:0/2683070566 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f3fd40830d0 con 0x7f3fe006aeb0 2023-10-30T14:17:10.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.413+0000 7f3fdffff640 1 --2- 172.21.15.57:0/2683070566 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3fc40664a0 0x7f3fc4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:10.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.413+0000 7f3fdffff640 1 --2- 172.21.15.57:0/2683070566 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3fc40664a0 0x7f3fc4068960 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc0b2000 tx=0x7f3fcc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:10.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.417+0000 7f3fdd7fa640 1 -- 172.21.15.57:0/2683070566 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3fe00ce880 con 0x7f3fe006aeb0 2023-10-30T14:17:10.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.605+0000 7f3fe5abb640 1 -- 172.21.15.57:0/2683070566 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fe01f00a0 con 0x7f3fe006aeb0 2023-10-30T14:17:10.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.605+0000 7f3fdd7fa640 1 -- 172.21.15.57:0/2683070566 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f3fe01f00a0 con 0x7f3fe006aeb0 2023-10-30T14:17:10.611 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:10.612 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:05.905427+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:10.612 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 24 2023-10-30T14:17:10.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.609+0000 7f3fe5abb640 1 -- 172.21.15.57:0/2683070566 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3fc40664a0 msgr2=0x7f3fc4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:10.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.609+0000 7f3fe5abb640 1 --2- 172.21.15.57:0/2683070566 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3fc40664a0 0x7f3fc4068960 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc0b2000 tx=0x7f3fcc008040 comp rx=0 tx=0).stop 2023-10-30T14:17:10.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.609+0000 7f3fe5abb640 1 -- 172.21.15.57:0/2683070566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fe006aeb0 msgr2=0x7f3fe00c9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:10.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.609+0000 7f3fe5abb640 1 --2- 172.21.15.57:0/2683070566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fe006aeb0 0x7f3fe00c9e10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3fd400c780 tx=0x7f3fd400cc40 comp rx=0 tx=0).stop 2023-10-30T14:17:10.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.609+0000 7f3fe5abb640 1 -- 172.21.15.57:0/2683070566 shutdown_connections 2023-10-30T14:17:10.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.609+0000 7f3fe5abb640 1 --2- 172.21.15.57:0/2683070566 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3fc40664a0 0x7f3fc4068960 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:10.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.609+0000 7f3fe5abb640 1 --2- 172.21.15.57:0/2683070566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fe015ac70 0x7f3fe01f0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:10.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.609+0000 7f3fe5abb640 1 --2- 172.21.15.57:0/2683070566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fe00c8eb0 0x7f3fe01f0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:10.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.609+0000 7f3fe5abb640 1 --2- 172.21.15.57:0/2683070566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fe006aeb0 0x7f3fe00c9e10 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:10.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.609+0000 7f3fe5abb640 1 -- 172.21.15.57:0/2683070566 >> 172.21.15.57:0/2683070566 conn(0x7f3fe00cef00 msgr2=0x7f3fe006a230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:10.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.609+0000 7f3fe5abb640 1 -- 172.21.15.57:0/2683070566 shutdown_connections 2023-10-30T14:17:10.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:10.609+0000 7f3fe5abb640 1 -- 172.21.15.57:0/2683070566 wait complete. 2023-10-30T14:17:10.632 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:17:12.633 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:12.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.781+0000 7f02e546b640 1 Processor -- start 2023-10-30T14:17:12.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.785+0000 7f02e546b640 1 -- start start 2023-10-30T14:17:12.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.785+0000 7f02e546b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02e0153c80 0x7f02e015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:12.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.785+0000 7f02e546b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f02e00c4a10 0x7f02e00c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:12.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.785+0000 7f02e546b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e0159f20 con 0x7f02e015a3b0 2023-10-30T14:17:12.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.785+0000 7f02e546b640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e015dec0 con 0x7f02e00c4a10 2023-10-30T14:17:12.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.785+0000 7f02e546b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e01599a0 con 0x7f02e0153c80 2023-10-30T14:17:12.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.785+0000 7f02d7fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f02e00c4a10 0x7f02e00c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:12.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.785+0000 7f02df7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f02e015a3b0 0x7f02e00ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50158/0 (socket says 172.21.15.57:50158) 2023-10-30T14:17:12.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.785+0000 7f02df7fe640 1 -- 172.21.15.57:0/2644983362 learned_addr learned my addr 172.21.15.57:0/2644983362 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:17:12.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.785+0000 7f02deffd640 1 --2- 172.21.15.57:0/2644983362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02e0153c80 0x7f02e015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:12.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.785+0000 7f02d7fff640 1 -- 172.21.15.57:0/2644983362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02e0153c80 msgr2=0x7f02e015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:12.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.785+0000 7f02d7fff640 1 --2- 172.21.15.57:0/2644983362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02e0153c80 0x7f02e015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:12.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.785+0000 7f02d7fff640 1 -- 172.21.15.57:0/2644983362 >> v1:172.21.15.57:6789/0 conn(0x7f02e015a3b0 legacy=0x7f02e00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:12.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.785+0000 7f02d7fff640 1 -- 172.21.15.57:0/2644983362 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e00c47e0 con 0x7f02e00c4a10 2023-10-30T14:17:12.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.785+0000 7f02de7fc640 1 -- 172.21.15.57:0/2644983362 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854025184 0 0) 0x7f02e0159f20 con 0x7f02e015a3b0 2023-10-30T14:17:12.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.789+0000 7f02d7fff640 1 --2- 172.21.15.57:0/2644983362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e00c4a10 0x7f02e00c42a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f02c80097c0 tx=0x7f02c801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c777a91be433797 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:12.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.789+0000 7f02de7fc640 1 -- 172.21.15.57:0/2644983362 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c80a8970 con 0x7f02e00c4a10 2023-10-30T14:17:12.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.789+0000 7f02de7fc640 1 -- 172.21.15.57:0/2644983362 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c80a8ad0 con 0x7f02e00c4a10 2023-10-30T14:17:12.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.789+0000 7f02de7fc640 1 -- 172.21.15.57:0/2644983362 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c80a8e60 con 0x7f02e00c4a10 2023-10-30T14:17:12.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.789+0000 7f02e546b640 1 -- 172.21.15.57:0/2644983362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e00c4a10 msgr2=0x7f02e00c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:12.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.789+0000 7f02e546b640 1 --2- 172.21.15.57:0/2644983362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e00c4a10 0x7f02e00c42a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f02c80097c0 tx=0x7f02c801ec70 comp rx=0 tx=0).stop 2023-10-30T14:17:12.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.789+0000 7f02e546b640 1 -- 172.21.15.57:0/2644983362 shutdown_connections 2023-10-30T14:17:12.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.789+0000 7f02e546b640 1 --2- 172.21.15.57:0/2644983362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e00c4a10 0x7f02e00c42a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:12.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.789+0000 7f02e546b640 1 --2- 172.21.15.57:0/2644983362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02e0153c80 0x7f02e015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:12.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.789+0000 7f02e546b640 1 -- 172.21.15.57:0/2644983362 >> 172.21.15.57:0/2644983362 conn(0x7f02e00715c0 msgr2=0x7f02e00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:12.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.789+0000 7f02e546b640 1 -- 172.21.15.57:0/2644983362 shutdown_connections 2023-10-30T14:17:12.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.789+0000 7f02e546b640 1 -- 172.21.15.57:0/2644983362 wait complete. 2023-10-30T14:17:12.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.789+0000 7f02e546b640 1 Processor -- start 2023-10-30T14:17:12.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.789+0000 7f02e546b640 1 -- start start 2023-10-30T14:17:12.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02e546b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e00c4a10 0x7f02e00ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:12.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02e546b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02e0153c80 0x7f02e00b4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:12.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02e546b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02e015a3b0 0x7f02e00b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:12.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02e546b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e00b4070 con 0x7f02e00c4a10 2023-10-30T14:17:12.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02e546b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e00b3180 con 0x7f02e015a3b0 2023-10-30T14:17:12.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02e546b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e00b3460 con 0x7f02e0153c80 2023-10-30T14:17:12.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02deffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e00c4a10 0x7f02e00ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:12.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02deffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e00c4a10 0x7f02e00ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44128/0 (socket says 172.21.15.57:44128) 2023-10-30T14:17:12.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02deffd640 1 -- 172.21.15.57:0/2480934072 learned_addr learned my addr 172.21.15.57:0/2480934072 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:12.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02df7fe640 1 --2- 172.21.15.57:0/2480934072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02e015a3b0 0x7f02e00b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:12.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02d7fff640 1 --2- 172.21.15.57:0/2480934072 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02e0153c80 0x7f02e00b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:12.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02deffd640 1 -- 172.21.15.57:0/2480934072 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02e0153c80 msgr2=0x7f02e00b4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:12.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02deffd640 1 --2- 172.21.15.57:0/2480934072 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02e0153c80 0x7f02e00b4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:12.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02deffd640 1 -- 172.21.15.57:0/2480934072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02e015a3b0 msgr2=0x7f02e00b4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:12.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02deffd640 1 --2- 172.21.15.57:0/2480934072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02e015a3b0 0x7f02e00b4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:12.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02deffd640 1 -- 172.21.15.57:0/2480934072 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c8009420 con 0x7f02e00c4a10 2023-10-30T14:17:12.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02deffd640 1 --2- 172.21.15.57:0/2480934072 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e00c4a10 0x7f02e00ce2f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f02d000d800 tx=0x7f02d000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:12.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02d77fe640 1 -- 172.21.15.57:0/2480934072 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d000b6c0 con 0x7f02e00c4a10 2023-10-30T14:17:12.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02d77fe640 1 -- 172.21.15.57:0/2480934072 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d000b820 con 0x7f02e00c4a10 2023-10-30T14:17:12.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02e546b640 1 -- 172.21.15.57:0/2480934072 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e00b3740 con 0x7f02e00c4a10 2023-10-30T14:17:12.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02d77fe640 1 -- 172.21.15.57:0/2480934072 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d000bb90 con 0x7f02e00c4a10 2023-10-30T14:17:12.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02e546b640 1 -- 172.21.15.57:0/2480934072 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e00b3d00 con 0x7f02e00c4a10 2023-10-30T14:17:12.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.793+0000 7f02e546b640 1 -- 172.21.15.57:0/2480934072 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e004e170 con 0x7f02e00c4a10 2023-10-30T14:17:12.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.797+0000 7f02d77fe640 1 -- 172.21.15.57:0/2480934072 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f02d000bcf0 con 0x7f02e00c4a10 2023-10-30T14:17:12.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.797+0000 7f02d77fe640 1 --2- 172.21.15.57:0/2480934072 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f02c00664a0 0x7f02c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:12.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.797+0000 7f02d77fe640 1 -- 172.21.15.57:0/2480934072 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f02d00846a0 con 0x7f02e00c4a10 2023-10-30T14:17:12.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.797+0000 7f02d7fff640 1 --2- 172.21.15.57:0/2480934072 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f02c00664a0 0x7f02c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:12.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.797+0000 7f02d7fff640 1 --2- 172.21.15.57:0/2480934072 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f02c00664a0 0x7f02c0068960 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f02c80b2000 tx=0x7f02c80aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:12.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.801+0000 7f02d77fe640 1 -- 172.21.15.57:0/2480934072 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f02d00518d0 con 0x7f02e00c4a10 2023-10-30T14:17:12.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.997+0000 7f02e546b640 1 -- 172.21.15.57:0/2480934072 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02e00b1b20 con 0x7f02e00c4a10 2023-10-30T14:17:13.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.997+0000 7f02d77fe640 1 -- 172.21.15.57:0/2480934072 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f02d0054ef0 con 0x7f02e00c4a10 2023-10-30T14:17:13.001 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:13.001 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:10.944689+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:clientreplay","state_seq":29,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:13.001 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 25 2023-10-30T14:17:13.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.997+0000 7f02e546b640 1 -- 172.21.15.57:0/2480934072 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f02c00664a0 msgr2=0x7f02c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:13.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.997+0000 7f02e546b640 1 --2- 172.21.15.57:0/2480934072 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f02c00664a0 0x7f02c0068960 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f02c80b2000 tx=0x7f02c80aa070 comp rx=0 tx=0).stop 2023-10-30T14:17:13.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.997+0000 7f02e546b640 1 -- 172.21.15.57:0/2480934072 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e00c4a10 msgr2=0x7f02e00ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:13.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.997+0000 7f02e546b640 1 --2- 172.21.15.57:0/2480934072 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e00c4a10 0x7f02e00ce2f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f02d000d800 tx=0x7f02d000dcc0 comp rx=0 tx=0).stop 2023-10-30T14:17:13.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.997+0000 7f02e546b640 1 -- 172.21.15.57:0/2480934072 shutdown_connections 2023-10-30T14:17:13.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.997+0000 7f02e546b640 1 --2- 172.21.15.57:0/2480934072 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f02c00664a0 0x7f02c0068960 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:13.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.997+0000 7f02e546b640 1 --2- 172.21.15.57:0/2480934072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02e015a3b0 0x7f02e00b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:13.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.997+0000 7f02e546b640 1 --2- 172.21.15.57:0/2480934072 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02e0153c80 0x7f02e00b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:13.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.997+0000 7f02e546b640 1 --2- 172.21.15.57:0/2480934072 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e00c4a10 0x7f02e00ce2f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:13.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.997+0000 7f02e546b640 1 -- 172.21.15.57:0/2480934072 >> 172.21.15.57:0/2480934072 conn(0x7f02e00715c0 msgr2=0x7f02e0157a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:13.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.997+0000 7f02e546b640 1 -- 172.21.15.57:0/2480934072 shutdown_connections 2023-10-30T14:17:13.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:12.997+0000 7f02e546b640 1 -- 172.21.15.57:0/2480934072 wait complete. 2023-10-30T14:17:13.021 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:17:15.023 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:15.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.173+0000 7f38bb73e640 1 Processor -- start 2023-10-30T14:17:15.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.173+0000 7f38bb73e640 1 -- start start 2023-10-30T14:17:15.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.177+0000 7f38bb73e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f38b4071f30 0x7f38b406e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:15.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.177+0000 7f38bb73e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38b4156880 0x7f38b4157270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:15.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.177+0000 7f38bb73e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f38b406dbe0 0x7f38b4065a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:15.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.177+0000 7f38bb73e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38b406ea70 con 0x7f38b406dbe0 2023-10-30T14:17:15.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.177+0000 7f38bb73e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38b4070c40 con 0x7f38b4071f30 2023-10-30T14:17:15.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.177+0000 7f38bb73e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b41577b0 con 0x7f38b4156880 2023-10-30T14:17:15.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.177+0000 7f38b9cdb640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f38b406dbe0 0x7f38b4065a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:15.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.177+0000 7f38b9cdb640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f38b406dbe0 0x7f38b4065a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36398/0 (socket says 172.21.15.57:36398) 2023-10-30T14:17:15.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.177+0000 7f38b9cdb640 1 -- 172.21.15.57:0/1953674439 learned_addr learned my addr 172.21.15.57:0/1953674439 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:15.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.177+0000 7f38b94da640 1 --2- 172.21.15.57:0/1953674439 >> v2:172.21.15.88:3300/0 conn(0x7f38b4071f30 0x7f38b406e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:15.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.177+0000 7f38b8cd9640 1 --2- 172.21.15.57:0/1953674439 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38b4156880 0x7f38b4157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:15.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.177+0000 7f38b9cdb640 1 -- 172.21.15.57:0/1953674439 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38b4156880 msgr2=0x7f38b4157270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:15.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.177+0000 7f38b9cdb640 1 --2- 172.21.15.57:0/1953674439 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38b4156880 0x7f38b4157270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:15.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.177+0000 7f38b9cdb640 1 -- 172.21.15.57:0/1953674439 >> v2:172.21.15.88:3300/0 conn(0x7f38b4071f30 msgr2=0x7f38b406e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:15.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.177+0000 7f38b9cdb640 1 --2- 172.21.15.57:0/1953674439 >> v2:172.21.15.88:3300/0 conn(0x7f38b4071f30 0x7f38b406e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:15.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.177+0000 7f38b9cdb640 1 -- 172.21.15.57:0/1953674439 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b4070e90 con 0x7f38b406dbe0 2023-10-30T14:17:15.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.177+0000 7f38b9cdb640 1 --2- 172.21.15.57:0/1953674439 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38b406dbe0 0x7f38b4065a80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f38b406d650 tx=0x7f38ac01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd1012a56032e376 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:15.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38abfff640 1 -- 172.21.15.57:0/1953674439 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac0a5820 con 0x7f38b406dbe0 2023-10-30T14:17:15.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38abfff640 1 -- 172.21.15.57:0/1953674439 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ac0a5980 con 0x7f38b406dbe0 2023-10-30T14:17:15.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38abfff640 1 -- 172.21.15.57:0/1953674439 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac0a5d10 con 0x7f38b406dbe0 2023-10-30T14:17:15.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38bb73e640 1 -- 172.21.15.57:0/1953674439 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38b406dbe0 msgr2=0x7f38b4065a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:15.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38bb73e640 1 --2- 172.21.15.57:0/1953674439 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38b406dbe0 0x7f38b4065a80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f38b406d650 tx=0x7f38ac01c9a0 comp rx=0 tx=0).stop 2023-10-30T14:17:15.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38bb73e640 1 -- 172.21.15.57:0/1953674439 shutdown_connections 2023-10-30T14:17:15.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38bb73e640 1 --2- 172.21.15.57:0/1953674439 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38b4156880 0x7f38b4157270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:15.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38bb73e640 1 --2- 172.21.15.57:0/1953674439 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38b406dbe0 0x7f38b4065a80 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:15.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38bb73e640 1 --2- 172.21.15.57:0/1953674439 >> v2:172.21.15.88:3300/0 conn(0x7f38b4071f30 0x7f38b406e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:15.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38bb73e640 1 -- 172.21.15.57:0/1953674439 >> 172.21.15.57:0/1953674439 conn(0x7f38b40c7f60 msgr2=0x7f38b40c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:15.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38bb73e640 1 -- 172.21.15.57:0/1953674439 shutdown_connections 2023-10-30T14:17:15.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38bb73e640 1 -- 172.21.15.57:0/1953674439 wait complete. 2023-10-30T14:17:15.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38bb73e640 1 Processor -- start 2023-10-30T14:17:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38bb73e640 1 -- start start 2023-10-30T14:17:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38bb73e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38b406dbe0 0x7f38b4144900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38bb73e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38b4071f30 0x7f38b41f1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38bb73e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38b4156880 0x7f38b41f2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38bb73e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b41f5ca0 con 0x7f38b406dbe0 2023-10-30T14:17:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38bb73e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b40c6610 con 0x7f38b4071f30 2023-10-30T14:17:15.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.181+0000 7f38bb73e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b413fc30 con 0x7f38b4156880 2023-10-30T14:17:15.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.185+0000 7f38b94da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38b406dbe0 0x7f38b4144900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:15.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.185+0000 7f38b94da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38b406dbe0 0x7f38b4144900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36404/0 (socket says 172.21.15.57:36404) 2023-10-30T14:17:15.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.185+0000 7f38b94da640 1 -- 172.21.15.57:0/882967661 learned_addr learned my addr 172.21.15.57:0/882967661 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:15.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.185+0000 7f38b9cdb640 1 --2- 172.21.15.57:0/882967661 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38b4156880 0x7f38b41f2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:15.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.185+0000 7f38b8cd9640 1 --2- 172.21.15.57:0/882967661 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38b4071f30 0x7f38b41f1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:15.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.185+0000 7f38b94da640 1 -- 172.21.15.57:0/882967661 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38b4156880 msgr2=0x7f38b41f2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:15.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.185+0000 7f38b94da640 1 --2- 172.21.15.57:0/882967661 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38b4156880 0x7f38b41f2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:15.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.185+0000 7f38b94da640 1 -- 172.21.15.57:0/882967661 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38b4071f30 msgr2=0x7f38b41f1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:15.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.185+0000 7f38b94da640 1 --2- 172.21.15.57:0/882967661 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38b4071f30 0x7f38b41f1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:15.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.185+0000 7f38b94da640 1 -- 172.21.15.57:0/882967661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ac009b30 con 0x7f38b406dbe0 2023-10-30T14:17:15.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.185+0000 7f38b94da640 1 --2- 172.21.15.57:0/882967661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38b406dbe0 0x7f38b4144900 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f38b000d830 tx=0x7f38b000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:15.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.185+0000 7f38a9ffb640 1 -- 172.21.15.57:0/882967661 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b000b6c0 con 0x7f38b406dbe0 2023-10-30T14:17:15.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.185+0000 7f38a9ffb640 1 -- 172.21.15.57:0/882967661 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b000b820 con 0x7f38b406dbe0 2023-10-30T14:17:15.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.185+0000 7f38bb73e640 1 -- 172.21.15.57:0/882967661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b413fe60 con 0x7f38b406dbe0 2023-10-30T14:17:15.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.185+0000 7f38a9ffb640 1 -- 172.21.15.57:0/882967661 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b000bb90 con 0x7f38b406dbe0 2023-10-30T14:17:15.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.185+0000 7f38bb73e640 1 -- 172.21.15.57:0/882967661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b41f5710 con 0x7f38b406dbe0 2023-10-30T14:17:15.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.185+0000 7f38bb73e640 1 -- 172.21.15.57:0/882967661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b413f400 con 0x7f38b406dbe0 2023-10-30T14:17:15.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.185+0000 7f38a9ffb640 1 -- 172.21.15.57:0/882967661 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f38b001e460 con 0x7f38b406dbe0 2023-10-30T14:17:15.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.189+0000 7f38a9ffb640 1 --2- 172.21.15.57:0/882967661 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f38980664d0 0x7f3898068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:15.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.189+0000 7f38a9ffb640 1 -- 172.21.15.57:0/882967661 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f38b00846c0 con 0x7f38b406dbe0 2023-10-30T14:17:15.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.189+0000 7f38b8cd9640 1 --2- 172.21.15.57:0/882967661 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f38980664d0 0x7f3898068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:15.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.189+0000 7f38b8cd9640 1 --2- 172.21.15.57:0/882967661 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f38980664d0 0x7f3898068990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f38b4072a40 tx=0x7f38a4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:15.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.193+0000 7f38a9ffb640 1 -- 172.21.15.57:0/882967661 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f38b413f400 con 0x7f38b406dbe0 2023-10-30T14:17:15.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.381+0000 7f38bb73e640 1 -- 172.21.15.57:0/882967661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38b413f400 con 0x7f38b406dbe0 2023-10-30T14:17:15.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.385+0000 7f38a9ffb640 1 -- 172.21.15.57:0/882967661 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f38b413f400 con 0x7f38b406dbe0 2023-10-30T14:17:15.388 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:15.388 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:10.944689+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:clientreplay","state_seq":29,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:15.388 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 25 2023-10-30T14:17:15.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.385+0000 7f38bb73e640 1 -- 172.21.15.57:0/882967661 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f38980664d0 msgr2=0x7f3898068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:15.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.385+0000 7f38bb73e640 1 --2- 172.21.15.57:0/882967661 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f38980664d0 0x7f3898068990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f38b4072a40 tx=0x7f38a4008040 comp rx=0 tx=0).stop 2023-10-30T14:17:15.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.385+0000 7f38bb73e640 1 -- 172.21.15.57:0/882967661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38b406dbe0 msgr2=0x7f38b4144900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:15.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.385+0000 7f38bb73e640 1 --2- 172.21.15.57:0/882967661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38b406dbe0 0x7f38b4144900 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f38b000d830 tx=0x7f38b000dcf0 comp rx=0 tx=0).stop 2023-10-30T14:17:15.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.385+0000 7f38bb73e640 1 -- 172.21.15.57:0/882967661 shutdown_connections 2023-10-30T14:17:15.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.385+0000 7f38bb73e640 1 --2- 172.21.15.57:0/882967661 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f38980664d0 0x7f3898068990 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:15.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.385+0000 7f38bb73e640 1 --2- 172.21.15.57:0/882967661 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38b4156880 0x7f38b41f2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:15.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.385+0000 7f38bb73e640 1 --2- 172.21.15.57:0/882967661 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38b4071f30 0x7f38b41f1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:15.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.385+0000 7f38bb73e640 1 --2- 172.21.15.57:0/882967661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38b406dbe0 0x7f38b4144900 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:15.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.385+0000 7f38bb73e640 1 -- 172.21.15.57:0/882967661 >> 172.21.15.57:0/882967661 conn(0x7f38b40c7f60 msgr2=0x7f38b414ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:15.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.385+0000 7f38bb73e640 1 -- 172.21.15.57:0/882967661 shutdown_connections 2023-10-30T14:17:15.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:15.385+0000 7f38bb73e640 1 -- 172.21.15.57:0/882967661 wait complete. 2023-10-30T14:17:15.409 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 25, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 25, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4266": { "addr": "172.21.15.57:6836/2693548867", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2693548867, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2693548867, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4266, "incarnation": 9, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4282": { "addr": "172.21.15.88:6839/2945854794", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 2945854794, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2945854794, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4936": { "addr": "172.21.15.57:6837/1988100930", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 1988100930, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 1988100930, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4936, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:clientreplay", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:17:10.944689+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 4266, "mds_2": 4936 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:17:17.410 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:17.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.557+0000 7f27bb994640 1 Processor -- start 2023-10-30T14:17:17.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.557+0000 7f27bb994640 1 -- start start 2023-10-30T14:17:17.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.557+0000 7f27bb994640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f27b406dfc0 0x7f27b406e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:17.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.557+0000 7f27bb994640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f27b4071f30 0x7f27b4156880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:17.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.557+0000 7f27bb994640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b4156dc0 con 0x7f27b406dbe0 2023-10-30T14:17:17.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.557+0000 7f27bb994640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27b406ea70 con 0x7f27b4071f30 2023-10-30T14:17:17.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.557+0000 7f27bb994640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27b4070c40 con 0x7f27b406dfc0 2023-10-30T14:17:17.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.557+0000 7f27b9f31640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f27b4071f30 0x7f27b4156880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:17.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.557+0000 7f27b9f31640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f27b4071f30 0x7f27b4156880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36418/0 (socket says 172.21.15.57:36418) 2023-10-30T14:17:17.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.557+0000 7f27b9f31640 1 -- 172.21.15.57:0/3402257235 learned_addr learned my addr 172.21.15.57:0/3402257235 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:17.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.557+0000 7f27b8f2f640 1 --2- 172.21.15.57:0/3402257235 >> v2:172.21.15.88:3300/0 conn(0x7f27b406dfc0 0x7f27b406e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:17.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.561+0000 7f27b9f31640 1 -- 172.21.15.57:0/3402257235 >> v2:172.21.15.88:3300/0 conn(0x7f27b406dfc0 msgr2=0x7f27b406e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:17.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.561+0000 7f27b9f31640 1 --2- 172.21.15.57:0/3402257235 >> v2:172.21.15.88:3300/0 conn(0x7f27b406dfc0 0x7f27b406e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:17.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.561+0000 7f27b9f31640 1 -- 172.21.15.57:0/3402257235 >> v1:172.21.15.88:6789/0 conn(0x7f27b406dbe0 legacy=0x7f27b4157270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:17.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.561+0000 7f27b9f31640 1 -- 172.21.15.57:0/3402257235 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b4070e90 con 0x7f27b4071f30 2023-10-30T14:17:17.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.561+0000 7f27b9f31640 1 --2- 172.21.15.57:0/3402257235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b4071f30 0x7f27b4156880 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f27b4073de0 tx=0x7f27ac01eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98ea7753aed8e2ae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:17.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.561+0000 7f27abfff640 1 -- 172.21.15.57:0/3402257235 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac00b2d0 con 0x7f27b4071f30 2023-10-30T14:17:17.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.561+0000 7f27abfff640 1 -- 172.21.15.57:0/3402257235 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ac00b430 con 0x7f27b4071f30 2023-10-30T14:17:17.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.561+0000 7f27abfff640 1 -- 172.21.15.57:0/3402257235 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac0a7c10 con 0x7f27b4071f30 2023-10-30T14:17:17.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.561+0000 7f27bb994640 1 -- 172.21.15.57:0/3402257235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b4071f30 msgr2=0x7f27b4156880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:17.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.561+0000 7f27bb994640 1 --2- 172.21.15.57:0/3402257235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b4071f30 0x7f27b4156880 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f27b4073de0 tx=0x7f27ac01eba0 comp rx=0 tx=0).stop 2023-10-30T14:17:17.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.561+0000 7f27bb994640 1 -- 172.21.15.57:0/3402257235 shutdown_connections 2023-10-30T14:17:17.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.561+0000 7f27bb994640 1 --2- 172.21.15.57:0/3402257235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b4071f30 0x7f27b4156880 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:17.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.561+0000 7f27bb994640 1 --2- 172.21.15.57:0/3402257235 >> v2:172.21.15.88:3300/0 conn(0x7f27b406dfc0 0x7f27b406e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:17.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.561+0000 7f27bb994640 1 -- 172.21.15.57:0/3402257235 >> 172.21.15.57:0/3402257235 conn(0x7f27b40c7f60 msgr2=0x7f27b40c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:17.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27bb994640 1 -- 172.21.15.57:0/3402257235 shutdown_connections 2023-10-30T14:17:17.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27bb994640 1 -- 172.21.15.57:0/3402257235 wait complete. 2023-10-30T14:17:17.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27bb994640 1 Processor -- start 2023-10-30T14:17:17.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27bb994640 1 -- start start 2023-10-30T14:17:17.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27bb994640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b406dbe0 0x7f27b4146220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:17.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27bb994640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27b406dfc0 0x7f27b41e8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:17.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27bb994640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27b4071f30 0x7f27b41e8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:17.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27bb994640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b41580c0 con 0x7f27b406dbe0 2023-10-30T14:17:17.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27bb994640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b41f1a30 con 0x7f27b406dfc0 2023-10-30T14:17:17.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27bb994640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27b4146760 con 0x7f27b4071f30 2023-10-30T14:17:17.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27b9730640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b406dbe0 0x7f27b4146220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:17.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27b9730640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b406dbe0 0x7f27b4146220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36432/0 (socket says 172.21.15.57:36432) 2023-10-30T14:17:17.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27b9730640 1 -- 172.21.15.57:0/1522406877 learned_addr learned my addr 172.21.15.57:0/1522406877 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:17.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27b9f31640 1 --2- 172.21.15.57:0/1522406877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27b4071f30 0x7f27b41e8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:17.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27b8f2f640 1 --2- 172.21.15.57:0/1522406877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27b406dfc0 0x7f27b41e8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:17.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27b9730640 1 -- 172.21.15.57:0/1522406877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27b4071f30 msgr2=0x7f27b41e8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:17.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27b9730640 1 --2- 172.21.15.57:0/1522406877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27b4071f30 0x7f27b41e8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:17.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27b9730640 1 -- 172.21.15.57:0/1522406877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27b406dfc0 msgr2=0x7f27b41e8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:17.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27b9730640 1 --2- 172.21.15.57:0/1522406877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27b406dfc0 0x7f27b41e8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:17.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27b9730640 1 -- 172.21.15.57:0/1522406877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ac00bd40 con 0x7f27b406dbe0 2023-10-30T14:17:17.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27b9730640 1 --2- 172.21.15.57:0/1522406877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b406dbe0 0x7f27b4146220 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f27b000c500 tx=0x7f27b000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:17.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27a9ffb640 1 -- 172.21.15.57:0/1522406877 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b0005220 con 0x7f27b406dbe0 2023-10-30T14:17:17.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27a9ffb640 1 -- 172.21.15.57:0/1522406877 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b0005380 con 0x7f27b406dbe0 2023-10-30T14:17:17.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27bb994640 1 -- 172.21.15.57:0/1522406877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b4160070 con 0x7f27b406dbe0 2023-10-30T14:17:17.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.565+0000 7f27a9ffb640 1 -- 172.21.15.57:0/1522406877 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b0007270 con 0x7f27b406dbe0 2023-10-30T14:17:17.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.569+0000 7f27bb994640 1 -- 172.21.15.57:0/1522406877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b41f1c20 con 0x7f27b406dbe0 2023-10-30T14:17:17.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.569+0000 7f27bb994640 1 -- 172.21.15.57:0/1522406877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b404e170 con 0x7f27b406dbe0 2023-10-30T14:17:17.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.569+0000 7f27a9ffb640 1 -- 172.21.15.57:0/1522406877 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f27b00073d0 con 0x7f27b406dbe0 2023-10-30T14:17:17.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.569+0000 7f27a9ffb640 1 --2- 172.21.15.57:0/1522406877 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2798066470 0x7f2798068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:17.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.569+0000 7f27a9ffb640 1 -- 172.21.15.57:0/1522406877 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f27b0083710 con 0x7f27b406dbe0 2023-10-30T14:17:17.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.569+0000 7f27b8f2f640 1 --2- 172.21.15.57:0/1522406877 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2798066470 0x7f2798068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:17.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.569+0000 7f27b8f2f640 1 --2- 172.21.15.57:0/1522406877 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2798066470 0x7f2798068930 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f27b406e560 tx=0x7f27a4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:17.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.573+0000 7f27a9ffb640 1 -- 172.21.15.57:0/1522406877 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f27b0050a80 con 0x7f27b406dbe0 2023-10-30T14:17:17.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.765+0000 7f27bb994640 1 -- 172.21.15.57:0/1522406877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27b4156dc0 con 0x7f27b406dbe0 2023-10-30T14:17:17.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.765+0000 7f27a9ffb640 1 -- 172.21.15.57:0/1522406877 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f27b4156dc0 con 0x7f27b406dbe0 2023-10-30T14:17:17.771 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:17.771 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:16.215482+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:17.771 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 26 2023-10-30T14:17:17.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.769+0000 7f27bb994640 1 -- 172.21.15.57:0/1522406877 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2798066470 msgr2=0x7f2798068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:17.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.769+0000 7f27bb994640 1 --2- 172.21.15.57:0/1522406877 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2798066470 0x7f2798068930 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f27b406e560 tx=0x7f27a4008040 comp rx=0 tx=0).stop 2023-10-30T14:17:17.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.769+0000 7f27bb994640 1 -- 172.21.15.57:0/1522406877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b406dbe0 msgr2=0x7f27b4146220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:17.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.769+0000 7f27bb994640 1 --2- 172.21.15.57:0/1522406877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b406dbe0 0x7f27b4146220 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f27b000c500 tx=0x7f27b000c9c0 comp rx=0 tx=0).stop 2023-10-30T14:17:17.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.769+0000 7f27bb994640 1 -- 172.21.15.57:0/1522406877 shutdown_connections 2023-10-30T14:17:17.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.769+0000 7f27bb994640 1 --2- 172.21.15.57:0/1522406877 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2798066470 0x7f2798068930 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:17.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.769+0000 7f27bb994640 1 --2- 172.21.15.57:0/1522406877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27b4071f30 0x7f27b41e8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:17.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.769+0000 7f27bb994640 1 --2- 172.21.15.57:0/1522406877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27b406dfc0 0x7f27b41e8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:17.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.769+0000 7f27bb994640 1 --2- 172.21.15.57:0/1522406877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27b406dbe0 0x7f27b4146220 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:17.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.769+0000 7f27bb994640 1 -- 172.21.15.57:0/1522406877 >> 172.21.15.57:0/1522406877 conn(0x7f27b40c7f60 msgr2=0x7f27b415e850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:17.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.769+0000 7f27bb994640 1 -- 172.21.15.57:0/1522406877 shutdown_connections 2023-10-30T14:17:17.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:17.769+0000 7f27bb994640 1 -- 172.21.15.57:0/1522406877 wait complete. 2023-10-30T14:17:17.791 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=2, replacing gid=4262 2023-10-30T14:17:17.791 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.a 2023-10-30T14:17:17.880 INFO:tasks.ceph.mds.f.smithi088.stderr:2023-10-30T14:17:17.878+0000 7f1418be9640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T14:17:19.792 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T14:17:19.793 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T14:17:19.793 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T14:17:19.797 INFO:tasks.ceph.mds.a:Started 2023-10-30T14:17:19.797 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:19.895 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-30T14:17:19.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.949+0000 7f2f7329a640 1 Processor -- start 2023-10-30T14:17:19.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.949+0000 7f2f7329a640 1 -- start start 2023-10-30T14:17:19.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.949+0000 7f2f7329a640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2f6c0c4a10 0x7f2f6c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:19.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.949+0000 7f2f7329a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6c159f20 con 0x7f2f6c0c42a0 2023-10-30T14:17:19.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.949+0000 7f2f7329a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6c15d620 con 0x7f2f6c153c80 2023-10-30T14:17:19.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.949+0000 7f2f7329a640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f6c15dec0 con 0x7f2f6c0c4a10 2023-10-30T14:17:19.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.949+0000 7f2f70835640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2f6c0c4a10 0x7f2f6c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:19.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.949+0000 7f2f70835640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2f6c0c4a10 0x7f2f6c15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36466/0 (socket says 172.21.15.57:36466) 2023-10-30T14:17:19.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.949+0000 7f2f70835640 1 -- 172.21.15.57:0/3874020804 learned_addr learned my addr 172.21.15.57:0/3874020804 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:19.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.949+0000 7f2f71837640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f2f6c0c42a0 0x7f2f6c0ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:35380/0 (socket says 172.21.15.57:35380) 2023-10-30T14:17:19.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.949+0000 7f2f70835640 1 -- 172.21.15.57:0/3874020804 >> v1:172.21.15.88:6789/0 conn(0x7f2f6c153c80 legacy=0x7f2f6c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:19.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.949+0000 7f2f70835640 1 -- 172.21.15.57:0/3874020804 >> v1:172.21.15.57:6789/0 conn(0x7f2f6c0c42a0 legacy=0x7f2f6c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:19.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.949+0000 7f2f70835640 1 -- 172.21.15.57:0/3874020804 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c15d900 con 0x7f2f6c0c4a10 2023-10-30T14:17:19.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.949+0000 7f2f63fff640 1 -- 172.21.15.57:0/3874020804 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098615358 0 0) 0x7f2f6c159f20 con 0x7f2f6c0c42a0 2023-10-30T14:17:19.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.953+0000 7f2f70835640 1 --2- 172.21.15.57:0/3874020804 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f6c0c4a10 0x7f2f6c15cf90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c009870 tx=0x7f2f5c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61ff4b1b773c3391 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:19.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.953+0000 7f2f63fff640 1 -- 172.21.15.57:0/3874020804 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c0a8a30 con 0x7f2f6c0c4a10 2023-10-30T14:17:19.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.953+0000 7f2f63fff640 1 -- 172.21.15.57:0/3874020804 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5c0a8b90 con 0x7f2f6c0c4a10 2023-10-30T14:17:19.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.953+0000 7f2f63fff640 1 -- 172.21.15.57:0/3874020804 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c0b0440 con 0x7f2f6c0c4a10 2023-10-30T14:17:19.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.953+0000 7f2f7329a640 1 -- 172.21.15.57:0/3874020804 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f6c0c4a10 msgr2=0x7f2f6c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:19.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.953+0000 7f2f7329a640 1 --2- 172.21.15.57:0/3874020804 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f6c0c4a10 0x7f2f6c15cf90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c009870 tx=0x7f2f5c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:17:19.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.953+0000 7f2f7329a640 1 -- 172.21.15.57:0/3874020804 shutdown_connections 2023-10-30T14:17:19.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.953+0000 7f2f7329a640 1 --2- 172.21.15.57:0/3874020804 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f6c0c4a10 0x7f2f6c15cf90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:19.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.953+0000 7f2f7329a640 1 -- 172.21.15.57:0/3874020804 >> 172.21.15.57:0/3874020804 conn(0x7f2f6c0715c0 msgr2=0x7f2f6c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:19.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.953+0000 7f2f7329a640 1 -- 172.21.15.57:0/3874020804 shutdown_connections 2023-10-30T14:17:19.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.953+0000 7f2f7329a640 1 -- 172.21.15.57:0/3874020804 wait complete. 2023-10-30T14:17:19.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f7329a640 1 Processor -- start 2023-10-30T14:17:19.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f7329a640 1 -- start start 2023-10-30T14:17:19.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f7329a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f6c0c42a0 0x7f2f6c158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:19.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f7329a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f6c0c4a10 0x7f2f6c1ef150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:19.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f7329a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f6c153c80 0x7f2f6c159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:19.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f7329a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c1ef690 con 0x7f2f6c0c42a0 2023-10-30T14:17:19.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f7329a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c0cec30 con 0x7f2f6c0c4a10 2023-10-30T14:17:19.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f7329a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c1579f0 con 0x7f2f6c153c80 2023-10-30T14:17:19.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f71036640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f6c0c42a0 0x7f2f6c158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:19.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f71036640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f6c0c42a0 0x7f2f6c158440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36478/0 (socket says 172.21.15.57:36478) 2023-10-30T14:17:19.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f71036640 1 -- 172.21.15.57:0/2934707312 learned_addr learned my addr 172.21.15.57:0/2934707312 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:19.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f70835640 1 --2- 172.21.15.57:0/2934707312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f6c0c4a10 0x7f2f6c1ef150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:19.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f71837640 1 --2- 172.21.15.57:0/2934707312 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f6c153c80 0x7f2f6c159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:19.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f71036640 1 -- 172.21.15.57:0/2934707312 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f6c153c80 msgr2=0x7f2f6c159380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:19.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f71036640 1 --2- 172.21.15.57:0/2934707312 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f6c153c80 0x7f2f6c159380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:19.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f71036640 1 -- 172.21.15.57:0/2934707312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f6c0c4a10 msgr2=0x7f2f6c1ef150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:19.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f71036640 1 --2- 172.21.15.57:0/2934707312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f6c0c4a10 0x7f2f6c1ef150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:19.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f71036640 1 -- 172.21.15.57:0/2934707312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c0094d0 con 0x7f2f6c0c42a0 2023-10-30T14:17:19.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f71036640 1 --2- 172.21.15.57:0/2934707312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f6c0c42a0 0x7f2f6c158440 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800c3f0 tx=0x7f2f6800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:19.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f61ffb640 1 -- 172.21.15.57:0/2934707312 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f680052d0 con 0x7f2f6c0c42a0 2023-10-30T14:17:19.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f61ffb640 1 -- 172.21.15.57:0/2934707312 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f68005430 con 0x7f2f6c0c42a0 2023-10-30T14:17:19.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f61ffb640 1 -- 172.21.15.57:0/2934707312 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68003840 con 0x7f2f6c0c42a0 2023-10-30T14:17:19.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f7329a640 1 -- 172.21.15.57:0/2934707312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6c14f930 con 0x7f2f6c0c42a0 2023-10-30T14:17:19.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.957+0000 7f2f7329a640 1 -- 172.21.15.57:0/2934707312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6c14fef0 con 0x7f2f6c0c42a0 2023-10-30T14:17:19.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.961+0000 7f2f7329a640 1 -- 172.21.15.57:0/2934707312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6c04e170 con 0x7f2f6c0c42a0 2023-10-30T14:17:19.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.961+0000 7f2f61ffb640 1 -- 172.21.15.57:0/2934707312 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2f680039a0 con 0x7f2f6c0c42a0 2023-10-30T14:17:19.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.961+0000 7f2f61ffb640 1 --2- 172.21.15.57:0/2934707312 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f500664d0 0x7f2f50068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:19.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.961+0000 7f2f61ffb640 1 -- 172.21.15.57:0/2934707312 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f2f68083470 con 0x7f2f6c0c42a0 2023-10-30T14:17:19.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.961+0000 7f2f70835640 1 --2- 172.21.15.57:0/2934707312 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f500664d0 0x7f2f50068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:19.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.961+0000 7f2f70835640 1 --2- 172.21.15.57:0/2934707312 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f500664d0 0x7f2f50068990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c0b2000 tx=0x7f2f5c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:19.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:19.965+0000 7f2f61ffb640 1 -- 172.21.15.57:0/2934707312 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2f68050870 con 0x7f2f6c0c42a0 2023-10-30T14:17:20.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:20.157+0000 7f2f7329a640 1 -- 172.21.15.57:0/2934707312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f6c0b5510 con 0x7f2f6c0c42a0 2023-10-30T14:17:20.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:20.157+0000 7f2f61ffb640 1 -- 172.21.15.57:0/2934707312 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f2f68053e90 con 0x7f2f6c0c42a0 2023-10-30T14:17:20.162 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:20.162 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:16.215482+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:20.163 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 26 2023-10-30T14:17:20.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:20.161+0000 7f2f7329a640 1 -- 172.21.15.57:0/2934707312 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f500664d0 msgr2=0x7f2f50068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:20.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:20.161+0000 7f2f7329a640 1 --2- 172.21.15.57:0/2934707312 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f500664d0 0x7f2f50068990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c0b2000 tx=0x7f2f5c008040 comp rx=0 tx=0).stop 2023-10-30T14:17:20.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:20.161+0000 7f2f7329a640 1 -- 172.21.15.57:0/2934707312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f6c0c42a0 msgr2=0x7f2f6c158440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:20.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:20.161+0000 7f2f7329a640 1 --2- 172.21.15.57:0/2934707312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f6c0c42a0 0x7f2f6c158440 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800c3f0 tx=0x7f2f6800c8b0 comp rx=0 tx=0).stop 2023-10-30T14:17:20.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:20.161+0000 7f2f7329a640 1 -- 172.21.15.57:0/2934707312 shutdown_connections 2023-10-30T14:17:20.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:20.161+0000 7f2f7329a640 1 --2- 172.21.15.57:0/2934707312 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f500664d0 0x7f2f50068990 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:20.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:20.161+0000 7f2f7329a640 1 --2- 172.21.15.57:0/2934707312 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f6c153c80 0x7f2f6c159380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:20.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:20.161+0000 7f2f7329a640 1 --2- 172.21.15.57:0/2934707312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f6c0c4a10 0x7f2f6c1ef150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:20.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:20.161+0000 7f2f7329a640 1 --2- 172.21.15.57:0/2934707312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f6c0c42a0 0x7f2f6c158440 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:20.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:20.161+0000 7f2f7329a640 1 -- 172.21.15.57:0/2934707312 >> 172.21.15.57:0/2934707312 conn(0x7f2f6c0715c0 msgr2=0x7f2f6c1518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:20.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:20.161+0000 7f2f7329a640 1 -- 172.21.15.57:0/2934707312 shutdown_connections 2023-10-30T14:17:20.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:20.161+0000 7f2f7329a640 1 -- 172.21.15.57:0/2934707312 wait complete. 2023-10-30T14:17:20.183 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-30T14:17:22.184 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:22.228 INFO:tasks.workunit.client.0.smithi057.stdout: create 260/30 file after the snapshot 2023-10-30T14:17:22.230 INFO:tasks.workunit.client.0.smithi057.stdout: create 260/31 file after the snapshot 2023-10-30T14:17:22.231 INFO:tasks.workunit.client.0.smithi057.stdout: create 260/32 file after the snapshot 2023-10-30T14:17:22.232 INFO:tasks.workunit.client.0.smithi057.stdout: create 260/33 file after the snapshot 2023-10-30T14:17:22.233 INFO:tasks.workunit.client.0.smithi057.stdout: create 260/34 file after the snapshot 2023-10-30T14:17:22.234 INFO:tasks.workunit.client.0.smithi057.stdout: create 260/35 file after the snapshot 2023-10-30T14:17:22.235 INFO:tasks.workunit.client.0.smithi057.stdout: create 260/36 file after the snapshot 2023-10-30T14:17:22.236 INFO:tasks.workunit.client.0.smithi057.stdout: create 260/37 file after the snapshot 2023-10-30T14:17:22.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.337+0000 7f1923854640 1 Processor -- start 2023-10-30T14:17:22.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.337+0000 7f1923854640 1 -- start start 2023-10-30T14:17:22.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.337+0000 7f1923854640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f191c1537f0 0x7f191c0701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:22.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.337+0000 7f1923854640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f191c15a2e0 0x7f191c0ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:22.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.337+0000 7f1923854640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f191c0cd4a0 0x7f191c06b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:22.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.337+0000 7f1923854640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f191c0c3ab0 con 0x7f191c15a2e0 2023-10-30T14:17:22.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.337+0000 7f1923854640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f191c0706e0 con 0x7f191c1537f0 2023-10-30T14:17:22.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.337+0000 7f1923854640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f191c0c2aa0 con 0x7f191c0cd4a0 2023-10-30T14:17:22.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.337+0000 7f1920def640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f191c15a2e0 0x7f191c0ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:22.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.337+0000 7f1920def640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f191c15a2e0 0x7f191c0ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36492/0 (socket says 172.21.15.57:36492) 2023-10-30T14:17:22.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.337+0000 7f1920def640 1 -- 172.21.15.57:0/2123129589 learned_addr learned my addr 172.21.15.57:0/2123129589 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:22.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.337+0000 7f1921df1640 1 --2- 172.21.15.57:0/2123129589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f191c0cd4a0 0x7f191c06b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:22.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.337+0000 7f19215f0640 1 --2- 172.21.15.57:0/2123129589 >> v2:172.21.15.88:3300/0 conn(0x7f191c1537f0 0x7f191c0701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:22.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.337+0000 7f1920def640 1 -- 172.21.15.57:0/2123129589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f191c0cd4a0 msgr2=0x7f191c06b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:22.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.337+0000 7f1920def640 1 --2- 172.21.15.57:0/2123129589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f191c0cd4a0 0x7f191c06b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:22.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.337+0000 7f1920def640 1 -- 172.21.15.57:0/2123129589 >> v2:172.21.15.88:3300/0 conn(0x7f191c1537f0 msgr2=0x7f191c0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:22.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.337+0000 7f1920def640 1 --2- 172.21.15.57:0/2123129589 >> v2:172.21.15.88:3300/0 conn(0x7f191c1537f0 0x7f191c0701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:22.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.337+0000 7f1920def640 1 -- 172.21.15.57:0/2123129589 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191c0708e0 con 0x7f191c15a2e0 2023-10-30T14:17:22.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.341+0000 7f1920def640 1 --2- 172.21.15.57:0/2123129589 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f191c15a2e0 0x7f191c0ccf60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f191c0cdb70 tx=0x7f190c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f336da7e5fadafae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:22.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.341+0000 7f1913fff640 1 -- 172.21.15.57:0/2123129589 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c0a89b0 con 0x7f191c15a2e0 2023-10-30T14:17:22.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.341+0000 7f1913fff640 1 -- 172.21.15.57:0/2123129589 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190c0a8b10 con 0x7f191c15a2e0 2023-10-30T14:17:22.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.341+0000 7f1913fff640 1 -- 172.21.15.57:0/2123129589 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c0a8ea0 con 0x7f191c15a2e0 2023-10-30T14:17:22.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.341+0000 7f1923854640 1 -- 172.21.15.57:0/2123129589 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f191c15a2e0 msgr2=0x7f191c0ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:22.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.341+0000 7f1923854640 1 --2- 172.21.15.57:0/2123129589 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f191c15a2e0 0x7f191c0ccf60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f191c0cdb70 tx=0x7f190c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:17:22.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.341+0000 7f1923854640 1 -- 172.21.15.57:0/2123129589 shutdown_connections 2023-10-30T14:17:22.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.341+0000 7f1923854640 1 --2- 172.21.15.57:0/2123129589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f191c0cd4a0 0x7f191c06b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:22.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.341+0000 7f1923854640 1 --2- 172.21.15.57:0/2123129589 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f191c15a2e0 0x7f191c0ccf60 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:22.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.341+0000 7f1923854640 1 --2- 172.21.15.57:0/2123129589 >> v2:172.21.15.88:3300/0 conn(0x7f191c1537f0 0x7f191c0701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:22.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.341+0000 7f1923854640 1 -- 172.21.15.57:0/2123129589 >> 172.21.15.57:0/2123129589 conn(0x7f191c071fb0 msgr2=0x7f191c0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:22.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.341+0000 7f1923854640 1 -- 172.21.15.57:0/2123129589 shutdown_connections 2023-10-30T14:17:22.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.341+0000 7f1923854640 1 -- 172.21.15.57:0/2123129589 wait complete. 2023-10-30T14:17:22.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1923854640 1 Processor -- start 2023-10-30T14:17:22.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1923854640 1 -- start start 2023-10-30T14:17:22.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1923854640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f191c0cd4a0 0x7f191c157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:22.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1923854640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f191c1537f0 0x7f191c151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:22.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1923854640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f191c15a2e0 0x7f191c1eea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:22.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1923854640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191c15d3a0 con 0x7f191c1537f0 2023-10-30T14:17:22.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1923854640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191c0c4a00 con 0x7f191c15a2e0 2023-10-30T14:17:22.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1923854640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f191c1ef590 con 0x7f191c0cd4a0 2023-10-30T14:17:22.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1920def640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f191c1537f0 0x7f191c151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:22.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1920def640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f191c1537f0 0x7f191c151e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36500/0 (socket says 172.21.15.57:36500) 2023-10-30T14:17:22.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1920def640 1 -- 172.21.15.57:0/3917168329 learned_addr learned my addr 172.21.15.57:0/3917168329 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:22.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1921df1640 1 --2- 172.21.15.57:0/3917168329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f191c15a2e0 0x7f191c1eea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:22.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f19215f0640 1 --2- 172.21.15.57:0/3917168329 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f191c0cd4a0 0x7f191c157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:22.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1920def640 1 -- 172.21.15.57:0/3917168329 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f191c0cd4a0 msgr2=0x7f191c157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:22.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1920def640 1 --2- 172.21.15.57:0/3917168329 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f191c0cd4a0 0x7f191c157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:22.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1920def640 1 -- 172.21.15.57:0/3917168329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f191c15a2e0 msgr2=0x7f191c1eea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:22.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1920def640 1 --2- 172.21.15.57:0/3917168329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f191c15a2e0 0x7f191c1eea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:22.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1920def640 1 -- 172.21.15.57:0/3917168329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190c0094d0 con 0x7f191c1537f0 2023-10-30T14:17:22.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1920def640 1 --2- 172.21.15.57:0/3917168329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f191c1537f0 0x7f191c151e20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f190c01d3a0 tx=0x7f190c01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:22.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1911ffb640 1 -- 172.21.15.57:0/3917168329 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c0b9620 con 0x7f191c1537f0 2023-10-30T14:17:22.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1911ffb640 1 -- 172.21.15.57:0/3917168329 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190c0b9780 con 0x7f191c1537f0 2023-10-30T14:17:22.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1923854640 1 -- 172.21.15.57:0/3917168329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f191c1ef870 con 0x7f191c1537f0 2023-10-30T14:17:22.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.345+0000 7f1911ffb640 1 -- 172.21.15.57:0/3917168329 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c0b9b30 con 0x7f191c1537f0 2023-10-30T14:17:22.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.349+0000 7f1923854640 1 -- 172.21.15.57:0/3917168329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f191c15cce0 con 0x7f191c1537f0 2023-10-30T14:17:22.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.349+0000 7f1923854640 1 -- 172.21.15.57:0/3917168329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f191c15d4e0 con 0x7f191c1537f0 2023-10-30T14:17:22.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.349+0000 7f1911ffb640 1 -- 172.21.15.57:0/3917168329 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f190c0b7070 con 0x7f191c1537f0 2023-10-30T14:17:22.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.349+0000 7f1911ffb640 1 --2- 172.21.15.57:0/3917168329 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f19000664a0 0x7f1900068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:22.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.349+0000 7f1911ffb640 1 -- 172.21.15.57:0/3917168329 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f190c122010 con 0x7f191c1537f0 2023-10-30T14:17:22.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.349+0000 7f19215f0640 1 --2- 172.21.15.57:0/3917168329 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f19000664a0 0x7f1900068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:22.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.353+0000 7f19215f0640 1 --2- 172.21.15.57:0/3917168329 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f19000664a0 0x7f1900068960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f191800b9d0 tx=0x7f1918008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:22.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.353+0000 7f1911ffb640 1 -- 172.21.15.57:0/3917168329 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f191c15d4e0 con 0x7f191c1537f0 2023-10-30T14:17:22.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.541+0000 7f1923854640 1 -- 172.21.15.57:0/3917168329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f191c15d4e0 con 0x7f191c1537f0 2023-10-30T14:17:22.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.541+0000 7f1911ffb640 1 -- 172.21.15.57:0/3917168329 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f191c15d4e0 con 0x7f191c1537f0 2023-10-30T14:17:22.547 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:22.548 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5133,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:16.215482+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:22.548 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 27 2023-10-30T14:17:22.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.545+0000 7f1923854640 1 -- 172.21.15.57:0/3917168329 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f19000664a0 msgr2=0x7f1900068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:22.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.545+0000 7f1923854640 1 --2- 172.21.15.57:0/3917168329 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f19000664a0 0x7f1900068960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f191800b9d0 tx=0x7f1918008040 comp rx=0 tx=0).stop 2023-10-30T14:17:22.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.545+0000 7f1923854640 1 -- 172.21.15.57:0/3917168329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f191c1537f0 msgr2=0x7f191c151e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:22.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.545+0000 7f1923854640 1 --2- 172.21.15.57:0/3917168329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f191c1537f0 0x7f191c151e20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f190c01d3a0 tx=0x7f190c01ba90 comp rx=0 tx=0).stop 2023-10-30T14:17:22.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.545+0000 7f1923854640 1 -- 172.21.15.57:0/3917168329 shutdown_connections 2023-10-30T14:17:22.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.545+0000 7f1923854640 1 --2- 172.21.15.57:0/3917168329 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f19000664a0 0x7f1900068960 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:22.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.545+0000 7f1923854640 1 --2- 172.21.15.57:0/3917168329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f191c15a2e0 0x7f191c1eea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:22.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.545+0000 7f1923854640 1 --2- 172.21.15.57:0/3917168329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f191c1537f0 0x7f191c151e20 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:22.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.545+0000 7f1923854640 1 --2- 172.21.15.57:0/3917168329 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f191c0cd4a0 0x7f191c157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:22.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.545+0000 7f1923854640 1 -- 172.21.15.57:0/3917168329 >> 172.21.15.57:0/3917168329 conn(0x7f191c071fb0 msgr2=0x7f191c06b210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:22.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.545+0000 7f1923854640 1 -- 172.21.15.57:0/3917168329 shutdown_connections 2023-10-30T14:17:22.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:22.545+0000 7f1923854640 1 -- 172.21.15.57:0/3917168329 wait complete. 2023-10-30T14:17:22.568 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T14:17:22.568 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before thrashing 2023-10-30T14:17:22.668 INFO:tasks.workunit.client.0.smithi057.stdout: create 260/38 file after the snapshot 2023-10-30T14:17:22.669 INFO:tasks.workunit.client.0.smithi057.stdout: create 260/39 file after the snapshot 2023-10-30T14:17:22.670 INFO:tasks.workunit.client.0.smithi057.stdout: create 261/.snap/snap-subdir-test 2023-10-30T14:17:22.693 INFO:tasks.workunit.client.0.smithi057.stdout: create 261/30 file after the snapshot 2023-10-30T14:17:22.695 INFO:tasks.workunit.client.0.smithi057.stdout: create 261/31 file after the snapshot 2023-10-30T14:17:22.697 INFO:tasks.workunit.client.0.smithi057.stdout: create 261/32 file after the snapshot 2023-10-30T14:17:22.699 INFO:tasks.workunit.client.0.smithi057.stdout: create 261/33 file after the snapshot 2023-10-30T14:17:22.701 INFO:tasks.workunit.client.0.smithi057.stdout: create 261/34 file after the snapshot 2023-10-30T14:17:22.702 INFO:tasks.workunit.client.0.smithi057.stdout: create 261/35 file after the snapshot 2023-10-30T14:17:22.704 INFO:tasks.workunit.client.0.smithi057.stdout: create 261/36 file after the snapshot 2023-10-30T14:17:22.706 INFO:tasks.workunit.client.0.smithi057.stdout: create 261/37 file after the snapshot 2023-10-30T14:17:22.707 INFO:tasks.workunit.client.0.smithi057.stdout: create 261/38 file after the snapshot 2023-10-30T14:17:22.709 INFO:tasks.workunit.client.0.smithi057.stdout: create 261/39 file after the snapshot 2023-10-30T14:17:22.711 INFO:tasks.workunit.client.0.smithi057.stdout: create 262/.snap/snap-subdir-test 2023-10-30T14:17:22.740 INFO:tasks.workunit.client.0.smithi057.stdout: create 262/30 file after the snapshot 2023-10-30T14:17:22.745 INFO:tasks.workunit.client.0.smithi057.stdout: create 262/31 file after the snapshot 2023-10-30T14:17:22.747 INFO:tasks.workunit.client.0.smithi057.stdout: create 262/32 file after the snapshot 2023-10-30T14:17:23.244 INFO:tasks.workunit.client.0.smithi057.stdout: create 262/33 file after the snapshot 2023-10-30T14:17:23.246 INFO:tasks.workunit.client.0.smithi057.stdout: create 262/34 file after the snapshot 2023-10-30T14:17:23.248 INFO:tasks.workunit.client.0.smithi057.stdout: create 262/35 file after the snapshot 2023-10-30T14:17:23.250 INFO:tasks.workunit.client.0.smithi057.stdout: create 262/36 file after the snapshot 2023-10-30T14:17:23.252 INFO:tasks.workunit.client.0.smithi057.stdout: create 262/37 file after the snapshot 2023-10-30T14:17:23.254 INFO:tasks.workunit.client.0.smithi057.stdout: create 262/38 file after the snapshot 2023-10-30T14:17:23.256 INFO:tasks.workunit.client.0.smithi057.stdout: create 262/39 file after the snapshot 2023-10-30T14:17:23.258 INFO:tasks.workunit.client.0.smithi057.stdout: create 263/.snap/snap-subdir-test 2023-10-30T14:17:23.283 INFO:tasks.workunit.client.0.smithi057.stdout: create 263/30 file after the snapshot 2023-10-30T14:17:23.284 INFO:tasks.workunit.client.0.smithi057.stdout: create 263/31 file after the snapshot 2023-10-30T14:17:23.285 INFO:tasks.workunit.client.0.smithi057.stdout: create 263/32 file after the snapshot 2023-10-30T14:17:23.286 INFO:tasks.workunit.client.0.smithi057.stdout: create 263/33 file after the snapshot 2023-10-30T14:17:23.287 INFO:tasks.workunit.client.0.smithi057.stdout: create 263/34 file after the snapshot 2023-10-30T14:17:23.288 INFO:tasks.workunit.client.0.smithi057.stdout: create 263/35 file after the snapshot 2023-10-30T14:17:23.288 INFO:tasks.workunit.client.0.smithi057.stdout: create 263/36 file after the snapshot 2023-10-30T14:17:23.289 INFO:tasks.workunit.client.0.smithi057.stdout: create 263/37 file after the snapshot 2023-10-30T14:17:23.290 INFO:tasks.workunit.client.0.smithi057.stdout: create 263/38 file after the snapshot 2023-10-30T14:17:23.291 INFO:tasks.workunit.client.0.smithi057.stdout: create 263/39 file after the snapshot 2023-10-30T14:17:23.292 INFO:tasks.workunit.client.0.smithi057.stdout: create 264/.snap/snap-subdir-test 2023-10-30T14:17:23.315 INFO:tasks.workunit.client.0.smithi057.stdout: create 264/30 file after the snapshot 2023-10-30T14:17:23.318 INFO:tasks.workunit.client.0.smithi057.stdout: create 264/31 file after the snapshot 2023-10-30T14:17:23.320 INFO:tasks.workunit.client.0.smithi057.stdout: create 264/32 file after the snapshot 2023-10-30T14:17:23.321 INFO:tasks.workunit.client.0.smithi057.stdout: create 264/33 file after the snapshot 2023-10-30T14:17:23.323 INFO:tasks.workunit.client.0.smithi057.stdout: create 264/34 file after the snapshot 2023-10-30T14:17:23.325 INFO:tasks.workunit.client.0.smithi057.stdout: create 264/35 file after the snapshot 2023-10-30T14:17:23.327 INFO:tasks.workunit.client.0.smithi057.stdout: create 264/36 file after the snapshot 2023-10-30T14:17:23.329 INFO:tasks.workunit.client.0.smithi057.stdout: create 264/37 file after the snapshot 2023-10-30T14:17:23.331 INFO:tasks.workunit.client.0.smithi057.stdout: create 264/38 file after the snapshot 2023-10-30T14:17:23.333 INFO:tasks.workunit.client.0.smithi057.stdout: create 264/39 file after the snapshot 2023-10-30T14:17:23.335 INFO:tasks.workunit.client.0.smithi057.stdout: create 265/.snap/snap-subdir-test 2023-10-30T14:17:23.364 INFO:tasks.workunit.client.0.smithi057.stdout: create 265/30 file after the snapshot 2023-10-30T14:17:23.366 INFO:tasks.workunit.client.0.smithi057.stdout: create 265/31 file after the snapshot 2023-10-30T14:17:23.368 INFO:tasks.workunit.client.0.smithi057.stdout: create 265/32 file after the snapshot 2023-10-30T14:17:23.370 INFO:tasks.workunit.client.0.smithi057.stdout: create 265/33 file after the snapshot 2023-10-30T14:17:23.371 INFO:tasks.workunit.client.0.smithi057.stdout: create 265/34 file after the snapshot 2023-10-30T14:17:23.373 INFO:tasks.workunit.client.0.smithi057.stdout: create 265/35 file after the snapshot 2023-10-30T14:17:23.375 INFO:tasks.workunit.client.0.smithi057.stdout: create 265/36 file after the snapshot 2023-10-30T14:17:23.377 INFO:tasks.workunit.client.0.smithi057.stdout: create 265/37 file after the snapshot 2023-10-30T14:17:23.379 INFO:tasks.workunit.client.0.smithi057.stdout: create 265/38 file after the snapshot 2023-10-30T14:17:23.381 INFO:tasks.workunit.client.0.smithi057.stdout: create 265/39 file after the snapshot 2023-10-30T14:17:23.383 INFO:tasks.workunit.client.0.smithi057.stdout: create 266/.snap/snap-subdir-test 2023-10-30T14:17:23.412 INFO:tasks.workunit.client.0.smithi057.stdout: create 266/30 file after the snapshot 2023-10-30T14:17:23.415 INFO:tasks.workunit.client.0.smithi057.stdout: create 266/31 file after the snapshot 2023-10-30T14:17:23.417 INFO:tasks.workunit.client.0.smithi057.stdout: create 266/32 file after the snapshot 2023-10-30T14:17:23.418 INFO:tasks.workunit.client.0.smithi057.stdout: create 266/33 file after the snapshot 2023-10-30T14:17:23.420 INFO:tasks.workunit.client.0.smithi057.stdout: create 266/34 file after the snapshot 2023-10-30T14:17:23.422 INFO:tasks.workunit.client.0.smithi057.stdout: create 266/35 file after the snapshot 2023-10-30T14:17:23.424 INFO:tasks.workunit.client.0.smithi057.stdout: create 266/36 file after the snapshot 2023-10-30T14:17:23.426 INFO:tasks.workunit.client.0.smithi057.stdout: create 266/37 file after the snapshot 2023-10-30T14:17:23.428 INFO:tasks.workunit.client.0.smithi057.stdout: create 266/38 file after the snapshot 2023-10-30T14:17:23.429 INFO:tasks.workunit.client.0.smithi057.stdout: create 266/39 file after the snapshot 2023-10-30T14:17:23.431 INFO:tasks.workunit.client.0.smithi057.stdout: create 267/.snap/snap-subdir-test 2023-10-30T14:17:23.459 INFO:tasks.workunit.client.0.smithi057.stdout: create 267/30 file after the snapshot 2023-10-30T14:17:23.462 INFO:tasks.workunit.client.0.smithi057.stdout: create 267/31 file after the snapshot 2023-10-30T14:17:23.463 INFO:tasks.workunit.client.0.smithi057.stdout: create 267/32 file after the snapshot 2023-10-30T14:17:23.464 INFO:tasks.workunit.client.0.smithi057.stdout: create 267/33 file after the snapshot 2023-10-30T14:17:23.465 INFO:tasks.workunit.client.0.smithi057.stdout: create 267/34 file after the snapshot 2023-10-30T14:17:23.467 INFO:tasks.workunit.client.0.smithi057.stdout: create 267/35 file after the snapshot 2023-10-30T14:17:23.468 INFO:tasks.workunit.client.0.smithi057.stdout: create 267/36 file after the snapshot 2023-10-30T14:17:23.469 INFO:tasks.workunit.client.0.smithi057.stdout: create 267/37 file after the snapshot 2023-10-30T14:17:23.470 INFO:tasks.workunit.client.0.smithi057.stdout: create 267/38 file after the snapshot 2023-10-30T14:17:23.471 INFO:tasks.workunit.client.0.smithi057.stdout: create 267/39 file after the snapshot 2023-10-30T14:17:23.472 INFO:tasks.workunit.client.0.smithi057.stdout: create 268/.snap/snap-subdir-test 2023-10-30T14:17:23.495 INFO:tasks.workunit.client.0.smithi057.stdout: create 268/30 file after the snapshot 2023-10-30T14:17:23.498 INFO:tasks.workunit.client.0.smithi057.stdout: create 268/31 file after the snapshot 2023-10-30T14:17:23.500 INFO:tasks.workunit.client.0.smithi057.stdout: create 268/32 file after the snapshot 2023-10-30T14:17:23.501 INFO:tasks.workunit.client.0.smithi057.stdout: create 268/33 file after the snapshot 2023-10-30T14:17:23.503 INFO:tasks.workunit.client.0.smithi057.stdout: create 268/34 file after the snapshot 2023-10-30T14:17:23.505 INFO:tasks.workunit.client.0.smithi057.stdout: create 268/35 file after the snapshot 2023-10-30T14:17:23.507 INFO:tasks.workunit.client.0.smithi057.stdout: create 268/36 file after the snapshot 2023-10-30T14:17:23.509 INFO:tasks.workunit.client.0.smithi057.stdout: create 268/37 file after the snapshot 2023-10-30T14:17:23.511 INFO:tasks.workunit.client.0.smithi057.stdout: create 268/38 file after the snapshot 2023-10-30T14:17:23.512 INFO:tasks.workunit.client.0.smithi057.stdout: create 268/39 file after the snapshot 2023-10-30T14:17:23.514 INFO:tasks.workunit.client.0.smithi057.stdout: create 269/.snap/snap-subdir-test 2023-10-30T14:17:23.544 INFO:tasks.workunit.client.0.smithi057.stdout: create 269/30 file after the snapshot 2023-10-30T14:17:23.547 INFO:tasks.workunit.client.0.smithi057.stdout: create 269/31 file after the snapshot 2023-10-30T14:17:23.549 INFO:tasks.workunit.client.0.smithi057.stdout: create 269/32 file after the snapshot 2023-10-30T14:17:23.550 INFO:tasks.workunit.client.0.smithi057.stdout: create 269/33 file after the snapshot 2023-10-30T14:17:23.552 INFO:tasks.workunit.client.0.smithi057.stdout: create 269/34 file after the snapshot 2023-10-30T14:17:23.554 INFO:tasks.workunit.client.0.smithi057.stdout: create 269/35 file after the snapshot 2023-10-30T14:17:23.556 INFO:tasks.workunit.client.0.smithi057.stdout: create 269/36 file after the snapshot 2023-10-30T14:17:23.557 INFO:tasks.workunit.client.0.smithi057.stdout: create 269/37 file after the snapshot 2023-10-30T14:17:23.559 INFO:tasks.workunit.client.0.smithi057.stdout: create 269/38 file after the snapshot 2023-10-30T14:17:23.561 INFO:tasks.workunit.client.0.smithi057.stdout: create 269/39 file after the snapshot 2023-10-30T14:17:23.563 INFO:tasks.workunit.client.0.smithi057.stdout: create 270/.snap/snap-subdir-test 2023-10-30T14:17:23.588 INFO:tasks.workunit.client.0.smithi057.stdout: create 270/30 file after the snapshot 2023-10-30T14:17:23.590 INFO:tasks.workunit.client.0.smithi057.stdout: create 270/31 file after the snapshot 2023-10-30T14:17:23.590 INFO:tasks.workunit.client.0.smithi057.stdout: create 270/32 file after the snapshot 2023-10-30T14:17:23.591 INFO:tasks.workunit.client.0.smithi057.stdout: create 270/33 file after the snapshot 2023-10-30T14:17:23.592 INFO:tasks.workunit.client.0.smithi057.stdout: create 270/34 file after the snapshot 2023-10-30T14:17:23.593 INFO:tasks.workunit.client.0.smithi057.stdout: create 270/35 file after the snapshot 2023-10-30T14:17:23.594 INFO:tasks.workunit.client.0.smithi057.stdout: create 270/36 file after the snapshot 2023-10-30T14:17:23.594 INFO:tasks.workunit.client.0.smithi057.stdout: create 270/37 file after the snapshot 2023-10-30T14:17:23.595 INFO:tasks.workunit.client.0.smithi057.stdout: create 270/38 file after the snapshot 2023-10-30T14:17:23.596 INFO:tasks.workunit.client.0.smithi057.stdout: create 270/39 file after the snapshot 2023-10-30T14:17:23.597 INFO:tasks.workunit.client.0.smithi057.stdout: create 271/.snap/snap-subdir-test 2023-10-30T14:17:23.614 INFO:tasks.workunit.client.0.smithi057.stdout: create 271/30 file after the snapshot 2023-10-30T14:17:23.615 INFO:tasks.workunit.client.0.smithi057.stdout: create 271/31 file after the snapshot 2023-10-30T14:17:23.616 INFO:tasks.workunit.client.0.smithi057.stdout: create 271/32 file after the snapshot 2023-10-30T14:17:23.616 INFO:tasks.workunit.client.0.smithi057.stdout: create 271/33 file after the snapshot 2023-10-30T14:17:23.617 INFO:tasks.workunit.client.0.smithi057.stdout: create 271/34 file after the snapshot 2023-10-30T14:17:23.618 INFO:tasks.workunit.client.0.smithi057.stdout: create 271/35 file after the snapshot 2023-10-30T14:17:23.618 INFO:tasks.workunit.client.0.smithi057.stdout: create 271/36 file after the snapshot 2023-10-30T14:17:23.619 INFO:tasks.workunit.client.0.smithi057.stdout: create 271/37 file after the snapshot 2023-10-30T14:17:23.620 INFO:tasks.workunit.client.0.smithi057.stdout: create 271/38 file after the snapshot 2023-10-30T14:17:23.621 INFO:tasks.workunit.client.0.smithi057.stdout: create 271/39 file after the snapshot 2023-10-30T14:17:23.622 INFO:tasks.workunit.client.0.smithi057.stdout: create 272/.snap/snap-subdir-test 2023-10-30T14:17:23.640 INFO:tasks.workunit.client.0.smithi057.stdout: create 272/30 file after the snapshot 2023-10-30T14:17:23.893 INFO:tasks.workunit.client.0.smithi057.stdout: create 272/31 file after the snapshot 2023-10-30T14:17:23.893 INFO:tasks.workunit.client.0.smithi057.stdout: create 272/32 file after the snapshot 2023-10-30T14:17:23.894 INFO:tasks.workunit.client.0.smithi057.stdout: create 272/33 file after the snapshot 2023-10-30T14:17:23.895 INFO:tasks.workunit.client.0.smithi057.stdout: create 272/34 file after the snapshot 2023-10-30T14:17:23.896 INFO:tasks.workunit.client.0.smithi057.stdout: create 272/35 file after the snapshot 2023-10-30T14:17:23.896 INFO:tasks.workunit.client.0.smithi057.stdout: create 272/36 file after the snapshot 2023-10-30T14:17:23.897 INFO:tasks.workunit.client.0.smithi057.stdout: create 272/37 file after the snapshot 2023-10-30T14:17:23.898 INFO:tasks.workunit.client.0.smithi057.stdout: create 272/38 file after the snapshot 2023-10-30T14:17:23.899 INFO:tasks.workunit.client.0.smithi057.stdout: create 272/39 file after the snapshot 2023-10-30T14:17:23.900 INFO:tasks.workunit.client.0.smithi057.stdout: create 273/.snap/snap-subdir-test 2023-10-30T14:17:24.758 INFO:tasks.workunit.client.0.smithi057.stdout: create 273/30 file after the snapshot 2023-10-30T14:17:24.760 INFO:tasks.workunit.client.0.smithi057.stdout: create 273/31 file after the snapshot 2023-10-30T14:17:24.763 INFO:tasks.workunit.client.0.smithi057.stdout: create 273/32 file after the snapshot 2023-10-30T14:17:24.766 INFO:tasks.workunit.client.0.smithi057.stdout: create 273/33 file after the snapshot 2023-10-30T14:17:24.769 INFO:tasks.workunit.client.0.smithi057.stdout: create 273/34 file after the snapshot 2023-10-30T14:17:24.772 INFO:tasks.workunit.client.0.smithi057.stdout: create 273/35 file after the snapshot 2023-10-30T14:17:24.775 INFO:tasks.workunit.client.0.smithi057.stdout: create 273/36 file after the snapshot 2023-10-30T14:17:24.778 INFO:tasks.workunit.client.0.smithi057.stdout: create 273/37 file after the snapshot 2023-10-30T14:17:24.781 INFO:tasks.workunit.client.0.smithi057.stdout: create 273/38 file after the snapshot 2023-10-30T14:17:24.784 INFO:tasks.workunit.client.0.smithi057.stdout: create 273/39 file after the snapshot 2023-10-30T14:17:24.787 INFO:tasks.workunit.client.0.smithi057.stdout: create 274/.snap/snap-subdir-test 2023-10-30T14:17:24.843 INFO:tasks.workunit.client.0.smithi057.stdout: create 274/30 file after the snapshot 2023-10-30T14:17:24.845 INFO:tasks.workunit.client.0.smithi057.stdout: create 274/31 file after the snapshot 2023-10-30T14:17:24.847 INFO:tasks.workunit.client.0.smithi057.stdout: create 274/32 file after the snapshot 2023-10-30T14:17:24.849 INFO:tasks.workunit.client.0.smithi057.stdout: create 274/33 file after the snapshot 2023-10-30T14:17:24.851 INFO:tasks.workunit.client.0.smithi057.stdout: create 274/34 file after the snapshot 2023-10-30T14:17:24.853 INFO:tasks.workunit.client.0.smithi057.stdout: create 274/35 file after the snapshot 2023-10-30T14:17:24.855 INFO:tasks.workunit.client.0.smithi057.stdout: create 274/36 file after the snapshot 2023-10-30T14:17:24.857 INFO:tasks.workunit.client.0.smithi057.stdout: create 274/37 file after the snapshot 2023-10-30T14:17:24.859 INFO:tasks.workunit.client.0.smithi057.stdout: create 274/38 file after the snapshot 2023-10-30T14:17:24.861 INFO:tasks.workunit.client.0.smithi057.stdout: create 274/39 file after the snapshot 2023-10-30T14:17:24.863 INFO:tasks.workunit.client.0.smithi057.stdout: create 275/.snap/snap-subdir-test 2023-10-30T14:17:24.891 INFO:tasks.workunit.client.0.smithi057.stdout: create 275/30 file after the snapshot 2023-10-30T14:17:24.894 INFO:tasks.workunit.client.0.smithi057.stdout: create 275/31 file after the snapshot 2023-10-30T14:17:24.897 INFO:tasks.workunit.client.0.smithi057.stdout: create 275/32 file after the snapshot 2023-10-30T14:17:24.899 INFO:tasks.workunit.client.0.smithi057.stdout: create 275/33 file after the snapshot 2023-10-30T14:17:24.902 INFO:tasks.workunit.client.0.smithi057.stdout: create 275/34 file after the snapshot 2023-10-30T14:17:24.904 INFO:tasks.workunit.client.0.smithi057.stdout: create 275/35 file after the snapshot 2023-10-30T14:17:24.907 INFO:tasks.workunit.client.0.smithi057.stdout: create 275/36 file after the snapshot 2023-10-30T14:17:24.909 INFO:tasks.workunit.client.0.smithi057.stdout: create 275/37 file after the snapshot 2023-10-30T14:17:24.912 INFO:tasks.workunit.client.0.smithi057.stdout: create 275/38 file after the snapshot 2023-10-30T14:17:24.914 INFO:tasks.workunit.client.0.smithi057.stdout: create 275/39 file after the snapshot 2023-10-30T14:17:24.917 INFO:tasks.workunit.client.0.smithi057.stdout: create 276/.snap/snap-subdir-test 2023-10-30T14:17:24.938 INFO:tasks.workunit.client.0.smithi057.stdout: create 276/30 file after the snapshot 2023-10-30T14:17:24.939 INFO:tasks.workunit.client.0.smithi057.stdout: create 276/31 file after the snapshot 2023-10-30T14:17:24.940 INFO:tasks.workunit.client.0.smithi057.stdout: create 276/32 file after the snapshot 2023-10-30T14:17:24.940 INFO:tasks.workunit.client.0.smithi057.stdout: create 276/33 file after the snapshot 2023-10-30T14:17:24.941 INFO:tasks.workunit.client.0.smithi057.stdout: create 276/34 file after the snapshot 2023-10-30T14:17:24.942 INFO:tasks.workunit.client.0.smithi057.stdout: create 276/35 file after the snapshot 2023-10-30T14:17:24.943 INFO:tasks.workunit.client.0.smithi057.stdout: create 276/36 file after the snapshot 2023-10-30T14:17:24.944 INFO:tasks.workunit.client.0.smithi057.stdout: create 276/37 file after the snapshot 2023-10-30T14:17:24.945 INFO:tasks.workunit.client.0.smithi057.stdout: create 276/38 file after the snapshot 2023-10-30T14:17:24.946 INFO:tasks.workunit.client.0.smithi057.stdout: create 276/39 file after the snapshot 2023-10-30T14:17:24.947 INFO:tasks.workunit.client.0.smithi057.stdout: create 277/.snap/snap-subdir-test 2023-10-30T14:17:24.965 INFO:tasks.workunit.client.0.smithi057.stdout: create 277/30 file after the snapshot 2023-10-30T14:17:24.968 INFO:tasks.workunit.client.0.smithi057.stdout: create 277/31 file after the snapshot 2023-10-30T14:17:24.969 INFO:tasks.workunit.client.0.smithi057.stdout: create 277/32 file after the snapshot 2023-10-30T14:17:24.970 INFO:tasks.workunit.client.0.smithi057.stdout: create 277/33 file after the snapshot 2023-10-30T14:17:24.970 INFO:tasks.workunit.client.0.smithi057.stdout: create 277/34 file after the snapshot 2023-10-30T14:17:24.971 INFO:tasks.workunit.client.0.smithi057.stdout: create 277/35 file after the snapshot 2023-10-30T14:17:24.972 INFO:tasks.workunit.client.0.smithi057.stdout: create 277/36 file after the snapshot 2023-10-30T14:17:24.973 INFO:tasks.workunit.client.0.smithi057.stdout: create 277/37 file after the snapshot 2023-10-30T14:17:24.974 INFO:tasks.workunit.client.0.smithi057.stdout: create 277/38 file after the snapshot 2023-10-30T14:17:24.975 INFO:tasks.workunit.client.0.smithi057.stdout: create 277/39 file after the snapshot 2023-10-30T14:17:24.975 INFO:tasks.workunit.client.0.smithi057.stdout: create 278/.snap/snap-subdir-test 2023-10-30T14:17:25.001 INFO:tasks.workunit.client.0.smithi057.stdout: create 278/30 file after the snapshot 2023-10-30T14:17:25.004 INFO:tasks.workunit.client.0.smithi057.stdout: create 278/31 file after the snapshot 2023-10-30T14:17:25.005 INFO:tasks.workunit.client.0.smithi057.stdout: create 278/32 file after the snapshot 2023-10-30T14:17:25.007 INFO:tasks.workunit.client.0.smithi057.stdout: create 278/33 file after the snapshot 2023-10-30T14:17:25.009 INFO:tasks.workunit.client.0.smithi057.stdout: create 278/34 file after the snapshot 2023-10-30T14:17:25.011 INFO:tasks.workunit.client.0.smithi057.stdout: create 278/35 file after the snapshot 2023-10-30T14:17:25.013 INFO:tasks.workunit.client.0.smithi057.stdout: create 278/36 file after the snapshot 2023-10-30T14:17:25.015 INFO:tasks.workunit.client.0.smithi057.stdout: create 278/37 file after the snapshot 2023-10-30T14:17:25.017 INFO:tasks.workunit.client.0.smithi057.stdout: create 278/38 file after the snapshot 2023-10-30T14:17:25.019 INFO:tasks.workunit.client.0.smithi057.stdout: create 278/39 file after the snapshot 2023-10-30T14:17:25.021 INFO:tasks.workunit.client.0.smithi057.stdout: create 279/.snap/snap-subdir-test 2023-10-30T14:17:25.724 INFO:tasks.workunit.client.0.smithi057.stdout: create 279/30 file after the snapshot 2023-10-30T14:17:25.727 INFO:tasks.workunit.client.0.smithi057.stdout: create 279/31 file after the snapshot 2023-10-30T14:17:25.729 INFO:tasks.workunit.client.0.smithi057.stdout: create 279/32 file after the snapshot 2023-10-30T14:17:25.731 INFO:tasks.workunit.client.0.smithi057.stdout: create 279/33 file after the snapshot 2023-10-30T14:17:25.733 INFO:tasks.workunit.client.0.smithi057.stdout: create 279/34 file after the snapshot 2023-10-30T14:17:25.735 INFO:tasks.workunit.client.0.smithi057.stdout: create 279/35 file after the snapshot 2023-10-30T14:17:25.737 INFO:tasks.workunit.client.0.smithi057.stdout: create 279/36 file after the snapshot 2023-10-30T14:17:25.739 INFO:tasks.workunit.client.0.smithi057.stdout: create 279/37 file after the snapshot 2023-10-30T14:17:25.741 INFO:tasks.workunit.client.0.smithi057.stdout: create 279/38 file after the snapshot 2023-10-30T14:17:25.743 INFO:tasks.workunit.client.0.smithi057.stdout: create 279/39 file after the snapshot 2023-10-30T14:17:25.745 INFO:tasks.workunit.client.0.smithi057.stdout: create 280/.snap/snap-subdir-test 2023-10-30T14:17:25.770 INFO:tasks.workunit.client.0.smithi057.stdout: create 280/30 file after the snapshot 2023-10-30T14:17:25.771 INFO:tasks.workunit.client.0.smithi057.stdout: create 280/31 file after the snapshot 2023-10-30T14:17:25.772 INFO:tasks.workunit.client.0.smithi057.stdout: create 280/32 file after the snapshot 2023-10-30T14:17:25.773 INFO:tasks.workunit.client.0.smithi057.stdout: create 280/33 file after the snapshot 2023-10-30T14:17:25.774 INFO:tasks.workunit.client.0.smithi057.stdout: create 280/34 file after the snapshot 2023-10-30T14:17:25.775 INFO:tasks.workunit.client.0.smithi057.stdout: create 280/35 file after the snapshot 2023-10-30T14:17:25.776 INFO:tasks.workunit.client.0.smithi057.stdout: create 280/36 file after the snapshot 2023-10-30T14:17:25.777 INFO:tasks.workunit.client.0.smithi057.stdout: create 280/37 file after the snapshot 2023-10-30T14:17:25.778 INFO:tasks.workunit.client.0.smithi057.stdout: create 280/38 file after the snapshot 2023-10-30T14:17:25.778 INFO:tasks.workunit.client.0.smithi057.stdout: create 280/39 file after the snapshot 2023-10-30T14:17:25.779 INFO:tasks.workunit.client.0.smithi057.stdout: create 281/.snap/snap-subdir-test 2023-10-30T14:17:25.802 INFO:tasks.workunit.client.0.smithi057.stdout: create 281/30 file after the snapshot 2023-10-30T14:17:25.804 INFO:tasks.workunit.client.0.smithi057.stdout: create 281/31 file after the snapshot 2023-10-30T14:17:25.806 INFO:tasks.workunit.client.0.smithi057.stdout: create 281/32 file after the snapshot 2023-10-30T14:17:25.807 INFO:tasks.workunit.client.0.smithi057.stdout: create 281/33 file after the snapshot 2023-10-30T14:17:25.810 INFO:tasks.workunit.client.0.smithi057.stdout: create 281/34 file after the snapshot 2023-10-30T14:17:25.814 INFO:tasks.workunit.client.0.smithi057.stdout: create 281/35 file after the snapshot 2023-10-30T14:17:25.814 INFO:tasks.workunit.client.0.smithi057.stdout: create 281/36 file after the snapshot 2023-10-30T14:17:25.815 INFO:tasks.workunit.client.0.smithi057.stdout: create 281/37 file after the snapshot 2023-10-30T14:17:25.817 INFO:tasks.workunit.client.0.smithi057.stdout: create 281/38 file after the snapshot 2023-10-30T14:17:25.819 INFO:tasks.workunit.client.0.smithi057.stdout: create 281/39 file after the snapshot 2023-10-30T14:17:25.821 INFO:tasks.workunit.client.0.smithi057.stdout: create 282/.snap/snap-subdir-test 2023-10-30T14:17:25.852 INFO:tasks.workunit.client.0.smithi057.stdout: create 282/30 file after the snapshot 2023-10-30T14:17:25.855 INFO:tasks.workunit.client.0.smithi057.stdout: create 282/31 file after the snapshot 2023-10-30T14:17:25.856 INFO:tasks.workunit.client.0.smithi057.stdout: create 282/32 file after the snapshot 2023-10-30T14:17:25.858 INFO:tasks.workunit.client.0.smithi057.stdout: create 282/33 file after the snapshot 2023-10-30T14:17:25.861 INFO:tasks.workunit.client.0.smithi057.stdout: create 282/34 file after the snapshot 2023-10-30T14:17:25.862 INFO:tasks.workunit.client.0.smithi057.stdout: create 282/35 file after the snapshot 2023-10-30T14:17:25.865 INFO:tasks.workunit.client.0.smithi057.stdout: create 282/36 file after the snapshot 2023-10-30T14:17:25.866 INFO:tasks.workunit.client.0.smithi057.stdout: create 282/37 file after the snapshot 2023-10-30T14:17:25.869 INFO:tasks.workunit.client.0.smithi057.stdout: create 282/38 file after the snapshot 2023-10-30T14:17:25.871 INFO:tasks.workunit.client.0.smithi057.stdout: create 282/39 file after the snapshot 2023-10-30T14:17:25.873 INFO:tasks.workunit.client.0.smithi057.stdout: create 283/.snap/snap-subdir-test 2023-10-30T14:17:25.899 INFO:tasks.workunit.client.0.smithi057.stdout: create 283/30 file after the snapshot 2023-10-30T14:17:25.913 INFO:tasks.workunit.client.0.smithi057.stdout: create 283/31 file after the snapshot 2023-10-30T14:17:25.914 INFO:tasks.workunit.client.0.smithi057.stdout: create 283/32 file after the snapshot 2023-10-30T14:17:25.915 INFO:tasks.workunit.client.0.smithi057.stdout: create 283/33 file after the snapshot 2023-10-30T14:17:25.916 INFO:tasks.workunit.client.0.smithi057.stdout: create 283/34 file after the snapshot 2023-10-30T14:17:25.916 INFO:tasks.workunit.client.0.smithi057.stdout: create 283/35 file after the snapshot 2023-10-30T14:17:25.917 INFO:tasks.workunit.client.0.smithi057.stdout: create 283/36 file after the snapshot 2023-10-30T14:17:25.918 INFO:tasks.workunit.client.0.smithi057.stdout: create 283/37 file after the snapshot 2023-10-30T14:17:25.919 INFO:tasks.workunit.client.0.smithi057.stdout: create 283/38 file after the snapshot 2023-10-30T14:17:25.920 INFO:tasks.workunit.client.0.smithi057.stdout: create 283/39 file after the snapshot 2023-10-30T14:17:25.921 INFO:tasks.workunit.client.0.smithi057.stdout: create 284/.snap/snap-subdir-test 2023-10-30T14:17:25.937 INFO:tasks.workunit.client.0.smithi057.stdout: create 284/30 file after the snapshot 2023-10-30T14:17:25.939 INFO:tasks.workunit.client.0.smithi057.stdout: create 284/31 file after the snapshot 2023-10-30T14:17:25.940 INFO:tasks.workunit.client.0.smithi057.stdout: create 284/32 file after the snapshot 2023-10-30T14:17:25.941 INFO:tasks.workunit.client.0.smithi057.stdout: create 284/33 file after the snapshot 2023-10-30T14:17:25.941 INFO:tasks.workunit.client.0.smithi057.stdout: create 284/34 file after the snapshot 2023-10-30T14:17:25.942 INFO:tasks.workunit.client.0.smithi057.stdout: create 284/35 file after the snapshot 2023-10-30T14:17:25.943 INFO:tasks.workunit.client.0.smithi057.stdout: create 284/36 file after the snapshot 2023-10-30T14:17:25.944 INFO:tasks.workunit.client.0.smithi057.stdout: create 284/37 file after the snapshot 2023-10-30T14:17:25.945 INFO:tasks.workunit.client.0.smithi057.stdout: create 284/38 file after the snapshot 2023-10-30T14:17:25.945 INFO:tasks.workunit.client.0.smithi057.stdout: create 284/39 file after the snapshot 2023-10-30T14:17:25.946 INFO:tasks.workunit.client.0.smithi057.stdout: create 285/.snap/snap-subdir-test 2023-10-30T14:17:25.969 INFO:tasks.workunit.client.0.smithi057.stdout: create 285/30 file after the snapshot 2023-10-30T14:17:25.971 INFO:tasks.workunit.client.0.smithi057.stdout: create 285/31 file after the snapshot 2023-10-30T14:17:25.973 INFO:tasks.workunit.client.0.smithi057.stdout: create 285/32 file after the snapshot 2023-10-30T14:17:25.975 INFO:tasks.workunit.client.0.smithi057.stdout: create 285/33 file after the snapshot 2023-10-30T14:17:25.977 INFO:tasks.workunit.client.0.smithi057.stdout: create 285/34 file after the snapshot 2023-10-30T14:17:25.979 INFO:tasks.workunit.client.0.smithi057.stdout: create 285/35 file after the snapshot 2023-10-30T14:17:25.981 INFO:tasks.workunit.client.0.smithi057.stdout: create 285/36 file after the snapshot 2023-10-30T14:17:25.983 INFO:tasks.workunit.client.0.smithi057.stdout: create 285/37 file after the snapshot 2023-10-30T14:17:25.985 INFO:tasks.workunit.client.0.smithi057.stdout: create 285/38 file after the snapshot 2023-10-30T14:17:25.987 INFO:tasks.workunit.client.0.smithi057.stdout: create 285/39 file after the snapshot 2023-10-30T14:17:25.989 INFO:tasks.workunit.client.0.smithi057.stdout: create 286/.snap/snap-subdir-test 2023-10-30T14:17:26.014 INFO:tasks.workunit.client.0.smithi057.stdout: create 286/30 file after the snapshot 2023-10-30T14:17:26.015 INFO:tasks.workunit.client.0.smithi057.stdout: create 286/31 file after the snapshot 2023-10-30T14:17:26.017 INFO:tasks.workunit.client.0.smithi057.stdout: create 286/32 file after the snapshot 2023-10-30T14:17:26.018 INFO:tasks.workunit.client.0.smithi057.stdout: create 286/33 file after the snapshot 2023-10-30T14:17:26.018 INFO:tasks.workunit.client.0.smithi057.stdout: create 286/34 file after the snapshot 2023-10-30T14:17:26.019 INFO:tasks.workunit.client.0.smithi057.stdout: create 286/35 file after the snapshot 2023-10-30T14:17:26.019 INFO:tasks.workunit.client.0.smithi057.stdout: create 286/36 file after the snapshot 2023-10-30T14:17:26.020 INFO:tasks.workunit.client.0.smithi057.stdout: create 286/37 file after the snapshot 2023-10-30T14:17:26.021 INFO:tasks.workunit.client.0.smithi057.stdout: create 286/38 file after the snapshot 2023-10-30T14:17:26.022 INFO:tasks.workunit.client.0.smithi057.stdout: create 286/39 file after the snapshot 2023-10-30T14:17:26.022 INFO:tasks.workunit.client.0.smithi057.stdout: create 287/.snap/snap-subdir-test 2023-10-30T14:17:26.046 INFO:tasks.workunit.client.0.smithi057.stdout: create 287/30 file after the snapshot 2023-10-30T14:17:26.048 INFO:tasks.workunit.client.0.smithi057.stdout: create 287/31 file after the snapshot 2023-10-30T14:17:26.050 INFO:tasks.workunit.client.0.smithi057.stdout: create 287/32 file after the snapshot 2023-10-30T14:17:26.053 INFO:tasks.workunit.client.0.smithi057.stdout: create 287/33 file after the snapshot 2023-10-30T14:17:26.054 INFO:tasks.workunit.client.0.smithi057.stdout: create 287/34 file after the snapshot 2023-10-30T14:17:26.057 INFO:tasks.workunit.client.0.smithi057.stdout: create 287/35 file after the snapshot 2023-10-30T14:17:26.058 INFO:tasks.workunit.client.0.smithi057.stdout: create 287/36 file after the snapshot 2023-10-30T14:17:26.060 INFO:tasks.workunit.client.0.smithi057.stdout: create 287/37 file after the snapshot 2023-10-30T14:17:26.062 INFO:tasks.workunit.client.0.smithi057.stdout: create 287/38 file after the snapshot 2023-10-30T14:17:26.064 INFO:tasks.workunit.client.0.smithi057.stdout: create 287/39 file after the snapshot 2023-10-30T14:17:26.066 INFO:tasks.workunit.client.0.smithi057.stdout: create 288/.snap/snap-subdir-test 2023-10-30T14:17:26.717 INFO:tasks.workunit.client.0.smithi057.stdout: create 288/30 file after the snapshot 2023-10-30T14:17:26.725 INFO:tasks.workunit.client.0.smithi057.stdout: create 288/31 file after the snapshot 2023-10-30T14:17:26.727 INFO:tasks.workunit.client.0.smithi057.stdout: create 288/32 file after the snapshot 2023-10-30T14:17:26.729 INFO:tasks.workunit.client.0.smithi057.stdout: create 288/33 file after the snapshot 2023-10-30T14:17:26.731 INFO:tasks.workunit.client.0.smithi057.stdout: create 288/34 file after the snapshot 2023-10-30T14:17:26.733 INFO:tasks.workunit.client.0.smithi057.stdout: create 288/35 file after the snapshot 2023-10-30T14:17:26.736 INFO:tasks.workunit.client.0.smithi057.stdout: create 288/36 file after the snapshot 2023-10-30T14:17:26.738 INFO:tasks.workunit.client.0.smithi057.stdout: create 288/37 file after the snapshot 2023-10-30T14:17:26.740 INFO:tasks.workunit.client.0.smithi057.stdout: create 288/38 file after the snapshot 2023-10-30T14:17:26.743 INFO:tasks.workunit.client.0.smithi057.stdout: create 288/39 file after the snapshot 2023-10-30T14:17:26.745 INFO:tasks.workunit.client.0.smithi057.stdout: create 289/.snap/snap-subdir-test 2023-10-30T14:17:27.287 INFO:tasks.workunit.client.0.smithi057.stdout: create 289/30 file after the snapshot 2023-10-30T14:17:27.289 INFO:tasks.workunit.client.0.smithi057.stdout: create 289/31 file after the snapshot 2023-10-30T14:17:27.291 INFO:tasks.workunit.client.0.smithi057.stdout: create 289/32 file after the snapshot 2023-10-30T14:17:27.293 INFO:tasks.workunit.client.0.smithi057.stdout: create 289/33 file after the snapshot 2023-10-30T14:17:27.295 INFO:tasks.workunit.client.0.smithi057.stdout: create 289/34 file after the snapshot 2023-10-30T14:17:27.298 INFO:tasks.workunit.client.0.smithi057.stdout: create 289/35 file after the snapshot 2023-10-30T14:17:27.299 INFO:tasks.workunit.client.0.smithi057.stdout: create 289/36 file after the snapshot 2023-10-30T14:17:27.301 INFO:tasks.workunit.client.0.smithi057.stdout: create 289/37 file after the snapshot 2023-10-30T14:17:27.303 INFO:tasks.workunit.client.0.smithi057.stdout: create 289/38 file after the snapshot 2023-10-30T14:17:27.305 INFO:tasks.workunit.client.0.smithi057.stdout: create 289/39 file after the snapshot 2023-10-30T14:17:27.307 INFO:tasks.workunit.client.0.smithi057.stdout: create 290/.snap/snap-subdir-test 2023-10-30T14:17:27.336 INFO:tasks.workunit.client.0.smithi057.stdout: create 290/30 file after the snapshot 2023-10-30T14:17:27.338 INFO:tasks.workunit.client.0.smithi057.stdout: create 290/31 file after the snapshot 2023-10-30T14:17:27.340 INFO:tasks.workunit.client.0.smithi057.stdout: create 290/32 file after the snapshot 2023-10-30T14:17:27.342 INFO:tasks.workunit.client.0.smithi057.stdout: create 290/33 file after the snapshot 2023-10-30T14:17:27.344 INFO:tasks.workunit.client.0.smithi057.stdout: create 290/34 file after the snapshot 2023-10-30T14:17:27.346 INFO:tasks.workunit.client.0.smithi057.stdout: create 290/35 file after the snapshot 2023-10-30T14:17:27.348 INFO:tasks.workunit.client.0.smithi057.stdout: create 290/36 file after the snapshot 2023-10-30T14:17:27.350 INFO:tasks.workunit.client.0.smithi057.stdout: create 290/37 file after the snapshot 2023-10-30T14:17:27.351 INFO:tasks.workunit.client.0.smithi057.stdout: create 290/38 file after the snapshot 2023-10-30T14:17:27.353 INFO:tasks.workunit.client.0.smithi057.stdout: create 290/39 file after the snapshot 2023-10-30T14:17:27.355 INFO:tasks.workunit.client.0.smithi057.stdout: create 291/.snap/snap-subdir-test 2023-10-30T14:17:27.570 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:27.704 INFO:tasks.workunit.client.0.smithi057.stdout: create 291/30 file after the snapshot 2023-10-30T14:17:27.706 INFO:tasks.workunit.client.0.smithi057.stdout: create 291/31 file after the snapshot 2023-10-30T14:17:27.708 INFO:tasks.workunit.client.0.smithi057.stdout: create 291/32 file after the snapshot 2023-10-30T14:17:27.709 INFO:tasks.workunit.client.0.smithi057.stdout: create 291/33 file after the snapshot 2023-10-30T14:17:27.711 INFO:tasks.workunit.client.0.smithi057.stdout: create 291/34 file after the snapshot 2023-10-30T14:17:27.713 INFO:tasks.workunit.client.0.smithi057.stdout: create 291/35 file after the snapshot 2023-10-30T14:17:27.719 INFO:tasks.workunit.client.0.smithi057.stdout: create 291/36 file after the snapshot 2023-10-30T14:17:27.723 INFO:tasks.workunit.client.0.smithi057.stdout: create 291/37 file after the snapshot 2023-10-30T14:17:27.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.721+0000 7f5dbc617640 1 Processor -- start 2023-10-30T14:17:27.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.721+0000 7f5dbc617640 1 -- start start 2023-10-30T14:17:27.725 INFO:tasks.workunit.client.0.smithi057.stdout: create 291/38 file after the snapshot 2023-10-30T14:17:27.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.721+0000 7f5dbc617640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5db4153c80 0x7f5db415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:27.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.721+0000 7f5dbc617640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5db40c4a10 0x7f5db40c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:27.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.721+0000 7f5dbc617640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db4159f20 con 0x7f5db415a3b0 2023-10-30T14:17:27.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.721+0000 7f5dbc617640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5db415dec0 con 0x7f5db4153c80 2023-10-30T14:17:27.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.721+0000 7f5dbc617640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db41599a0 con 0x7f5db40c4a10 2023-10-30T14:17:27.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.721+0000 7f5dba3b3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5db4153c80 0x7f5db415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:27.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.721+0000 7f5dbabb4640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5db415a3b0 0x7f5db40ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:41944/0 (socket says 172.21.15.57:41944) 2023-10-30T14:17:27.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.721+0000 7f5dbabb4640 1 -- 172.21.15.57:0/3342933766 learned_addr learned my addr 172.21.15.57:0/3342933766 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:17:27.728 INFO:tasks.workunit.client.0.smithi057.stdout: create 291/39 file after the snapshot 2023-10-30T14:17:27.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.721+0000 7f5db9bb2640 1 --2- 172.21.15.57:0/3342933766 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5db40c4a10 0x7f5db40c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:27.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.721+0000 7f5dba3b3640 1 -- 172.21.15.57:0/3342933766 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5db40c4a10 msgr2=0x7f5db40c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:27.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.721+0000 7f5dba3b3640 1 --2- 172.21.15.57:0/3342933766 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5db40c4a10 0x7f5db40c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:27.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.725+0000 7f5dba3b3640 1 -- 172.21.15.57:0/3342933766 >> v1:172.21.15.88:6789/0 conn(0x7f5db415a3b0 legacy=0x7f5db40ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:27.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.725+0000 7f5dba3b3640 1 -- 172.21.15.57:0/3342933766 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db40c47e0 con 0x7f5db4153c80 2023-10-30T14:17:27.730 INFO:tasks.workunit.client.0.smithi057.stdout: create 292/.snap/snap-subdir-test 2023-10-30T14:17:27.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.725+0000 7f5dba3b3640 1 --2- 172.21.15.57:0/3342933766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5db4153c80 0x7f5db415cf90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5db00097c0 tx=0x7f5db001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa01f957c72dee05 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:27.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.725+0000 7f5db93b1640 1 -- 172.21.15.57:0/3342933766 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db00a8970 con 0x7f5db4153c80 2023-10-30T14:17:27.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.725+0000 7f5db93b1640 1 -- 172.21.15.57:0/3342933766 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db00a8ad0 con 0x7f5db4153c80 2023-10-30T14:17:27.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.725+0000 7f5db93b1640 1 -- 172.21.15.57:0/3342933766 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db00a8e60 con 0x7f5db4153c80 2023-10-30T14:17:27.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dbc617640 1 -- 172.21.15.57:0/3342933766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5db4153c80 msgr2=0x7f5db415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:27.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dbc617640 1 --2- 172.21.15.57:0/3342933766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5db4153c80 0x7f5db415cf90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5db00097c0 tx=0x7f5db001ec70 comp rx=0 tx=0).stop 2023-10-30T14:17:27.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dbc617640 1 -- 172.21.15.57:0/3342933766 shutdown_connections 2023-10-30T14:17:27.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dbc617640 1 --2- 172.21.15.57:0/3342933766 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5db40c4a10 0x7f5db40c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:27.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dbc617640 1 --2- 172.21.15.57:0/3342933766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5db4153c80 0x7f5db415cf90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:27.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dbc617640 1 -- 172.21.15.57:0/3342933766 >> 172.21.15.57:0/3342933766 conn(0x7f5db40715c0 msgr2=0x7f5db40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:27.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dbc617640 1 -- 172.21.15.57:0/3342933766 shutdown_connections 2023-10-30T14:17:27.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dbc617640 1 -- 172.21.15.57:0/3342933766 wait complete. 2023-10-30T14:17:27.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dbc617640 1 Processor -- start 2023-10-30T14:17:27.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dbc617640 1 -- start start 2023-10-30T14:17:27.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dbc617640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5db40c4a10 0x7f5db40cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:27.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dbc617640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5db4153c80 0x7f5db40cf200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:27.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dbc617640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5db415a3b0 0x7f5db40cf740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:27.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dbc617640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db4155390 con 0x7f5db40c4a10 2023-10-30T14:17:27.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dbc617640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db40cea70 con 0x7f5db4153c80 2023-10-30T14:17:27.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dbc617640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db41ef910 con 0x7f5db415a3b0 2023-10-30T14:17:27.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dba3b3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5db40c4a10 0x7f5db40cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:27.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dba3b3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5db40c4a10 0x7f5db40cdd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53624/0 (socket says 172.21.15.57:53624) 2023-10-30T14:17:27.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dba3b3640 1 -- 172.21.15.57:0/1769258454 learned_addr learned my addr 172.21.15.57:0/1769258454 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:27.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5db9bb2640 1 --2- 172.21.15.57:0/1769258454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5db4153c80 0x7f5db40cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:27.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dbabb4640 1 --2- 172.21.15.57:0/1769258454 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5db415a3b0 0x7f5db40cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:27.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dba3b3640 1 -- 172.21.15.57:0/1769258454 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5db415a3b0 msgr2=0x7f5db40cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:27.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.729+0000 7f5dba3b3640 1 --2- 172.21.15.57:0/1769258454 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5db415a3b0 0x7f5db40cf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:27.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.733+0000 7f5dba3b3640 1 -- 172.21.15.57:0/1769258454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5db4153c80 msgr2=0x7f5db40cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:27.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.733+0000 7f5dba3b3640 1 --2- 172.21.15.57:0/1769258454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5db4153c80 0x7f5db40cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:27.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.733+0000 7f5dba3b3640 1 -- 172.21.15.57:0/1769258454 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db0009420 con 0x7f5db40c4a10 2023-10-30T14:17:27.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.733+0000 7f5dbabb4640 1 --2- 172.21.15.57:0/1769258454 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5db415a3b0 0x7f5db40cf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:17:27.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.733+0000 7f5dba3b3640 1 --2- 172.21.15.57:0/1769258454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5db40c4a10 0x7f5db40cdd40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5db001c8a0 tx=0x7f5db001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:27.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.733+0000 7f5daaffd640 1 -- 172.21.15.57:0/1769258454 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db00b9630 con 0x7f5db40c4a10 2023-10-30T14:17:27.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.733+0000 7f5daaffd640 1 -- 172.21.15.57:0/1769258454 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db00b97b0 con 0x7f5db40c4a10 2023-10-30T14:17:27.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.733+0000 7f5dbc617640 1 -- 172.21.15.57:0/1769258454 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db40cd250 con 0x7f5db40c4a10 2023-10-30T14:17:27.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.733+0000 7f5daaffd640 1 -- 172.21.15.57:0/1769258454 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db00b9b60 con 0x7f5db40c4a10 2023-10-30T14:17:27.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.733+0000 7f5dbc617640 1 -- 172.21.15.57:0/1769258454 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db40cece0 con 0x7f5db40c4a10 2023-10-30T14:17:27.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.733+0000 7f5dbc617640 1 -- 172.21.15.57:0/1769258454 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db40ceed0 con 0x7f5db40c4a10 2023-10-30T14:17:27.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.733+0000 7f5daaffd640 1 -- 172.21.15.57:0/1769258454 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5db00b7040 con 0x7f5db40c4a10 2023-10-30T14:17:27.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.733+0000 7f5daaffd640 1 --2- 172.21.15.57:0/1769258454 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5d980664d0 0x7f5d98068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:27.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.733+0000 7f5daaffd640 1 -- 172.21.15.57:0/1769258454 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f5db0121e70 con 0x7f5db40c4a10 2023-10-30T14:17:27.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.737+0000 7f5db9bb2640 1 --2- 172.21.15.57:0/1769258454 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5d980664d0 0x7f5d98068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:27.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.737+0000 7f5db9bb2640 1 --2- 172.21.15.57:0/1769258454 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5d980664d0 0x7f5d98068990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5db41581e0 tx=0x7f5da4009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:27.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.741+0000 7f5daaffd640 1 -- 172.21.15.57:0/1769258454 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5db00ef1d0 con 0x7f5db40c4a10 2023-10-30T14:17:27.759 INFO:tasks.workunit.client.0.smithi057.stdout: create 292/30 file after the snapshot 2023-10-30T14:17:27.761 INFO:tasks.workunit.client.0.smithi057.stdout: create 292/31 file after the snapshot 2023-10-30T14:17:27.763 INFO:tasks.workunit.client.0.smithi057.stdout: create 292/32 file after the snapshot 2023-10-30T14:17:27.764 INFO:tasks.workunit.client.0.smithi057.stdout: create 292/33 file after the snapshot 2023-10-30T14:17:27.766 INFO:tasks.workunit.client.0.smithi057.stdout: create 292/34 file after the snapshot 2023-10-30T14:17:27.767 INFO:tasks.workunit.client.0.smithi057.stdout: create 292/35 file after the snapshot 2023-10-30T14:17:27.769 INFO:tasks.workunit.client.0.smithi057.stdout: create 292/36 file after the snapshot 2023-10-30T14:17:27.771 INFO:tasks.workunit.client.0.smithi057.stdout: create 292/37 file after the snapshot 2023-10-30T14:17:27.772 INFO:tasks.workunit.client.0.smithi057.stdout: create 292/38 file after the snapshot 2023-10-30T14:17:27.774 INFO:tasks.workunit.client.0.smithi057.stdout: create 292/39 file after the snapshot 2023-10-30T14:17:27.775 INFO:tasks.workunit.client.0.smithi057.stdout: create 293/.snap/snap-subdir-test 2023-10-30T14:17:27.801 INFO:tasks.workunit.client.0.smithi057.stdout: create 293/30 file after the snapshot 2023-10-30T14:17:27.803 INFO:tasks.workunit.client.0.smithi057.stdout: create 293/31 file after the snapshot 2023-10-30T14:17:27.804 INFO:tasks.workunit.client.0.smithi057.stdout: create 293/32 file after the snapshot 2023-10-30T14:17:27.805 INFO:tasks.workunit.client.0.smithi057.stdout: create 293/33 file after the snapshot 2023-10-30T14:17:27.806 INFO:tasks.workunit.client.0.smithi057.stdout: create 293/34 file after the snapshot 2023-10-30T14:17:27.807 INFO:tasks.workunit.client.0.smithi057.stdout: create 293/35 file after the snapshot 2023-10-30T14:17:27.808 INFO:tasks.workunit.client.0.smithi057.stdout: create 293/36 file after the snapshot 2023-10-30T14:17:27.809 INFO:tasks.workunit.client.0.smithi057.stdout: create 293/37 file after the snapshot 2023-10-30T14:17:27.811 INFO:tasks.workunit.client.0.smithi057.stdout: create 293/38 file after the snapshot 2023-10-30T14:17:27.812 INFO:tasks.workunit.client.0.smithi057.stdout: create 293/39 file after the snapshot 2023-10-30T14:17:27.813 INFO:tasks.workunit.client.0.smithi057.stdout: create 294/.snap/snap-subdir-test 2023-10-30T14:17:27.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.937+0000 7f5dbc617640 1 -- 172.21.15.57:0/1769258454 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5db404e170 con 0x7f5db40c4a10 2023-10-30T14:17:27.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.937+0000 7f5daaffd640 1 -- 172.21.15.57:0/1769258454 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6528 (secure 0 0 0) 0x7f5db00f27f0 con 0x7f5db40c4a10 2023-10-30T14:17:27.943 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:27.943 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5133,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:26.649359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:27.944 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 28 2023-10-30T14:17:27.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.941+0000 7f5dbc617640 1 -- 172.21.15.57:0/1769258454 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5d980664d0 msgr2=0x7f5d98068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:27.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.941+0000 7f5dbc617640 1 --2- 172.21.15.57:0/1769258454 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5d980664d0 0x7f5d98068990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5db41581e0 tx=0x7f5da4009190 comp rx=0 tx=0).stop 2023-10-30T14:17:27.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.941+0000 7f5dbc617640 1 -- 172.21.15.57:0/1769258454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5db40c4a10 msgr2=0x7f5db40cdd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:27.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.941+0000 7f5dbc617640 1 --2- 172.21.15.57:0/1769258454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5db40c4a10 0x7f5db40cdd40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5db001c8a0 tx=0x7f5db001ba10 comp rx=0 tx=0).stop 2023-10-30T14:17:27.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.941+0000 7f5dbc617640 1 -- 172.21.15.57:0/1769258454 shutdown_connections 2023-10-30T14:17:27.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.941+0000 7f5dbc617640 1 --2- 172.21.15.57:0/1769258454 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5d980664d0 0x7f5d98068990 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:27.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.941+0000 7f5dbc617640 1 --2- 172.21.15.57:0/1769258454 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5db415a3b0 0x7f5db40cf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:27.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.941+0000 7f5dbc617640 1 --2- 172.21.15.57:0/1769258454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5db4153c80 0x7f5db40cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:27.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.941+0000 7f5dbc617640 1 --2- 172.21.15.57:0/1769258454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5db40c4a10 0x7f5db40cdd40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:27.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.941+0000 7f5dbc617640 1 -- 172.21.15.57:0/1769258454 >> 172.21.15.57:0/1769258454 conn(0x7f5db40715c0 msgr2=0x7f5db40cb460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:27.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.941+0000 7f5dbc617640 1 -- 172.21.15.57:0/1769258454 shutdown_connections 2023-10-30T14:17:27.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:27.941+0000 7f5dbc617640 1 -- 172.21.15.57:0/1769258454 wait complete. 2023-10-30T14:17:27.963 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=1) 2023-10-30T14:17:27.964 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T14:17:27.964 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:17:27.978 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T14:17:27.978 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:28.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.161+0000 7f8e7b90f640 1 Processor -- start 2023-10-30T14:17:28.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.161+0000 7f8e7b90f640 1 -- start start 2023-10-30T14:17:28.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.161+0000 7f8e7b90f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8e74159980 0x7f8e74159d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:28.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.161+0000 7f8e7b90f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e740c5f80 con 0x7f8e74153bb0 2023-10-30T14:17:28.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.161+0000 7f8e7b90f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e740c5690 con 0x7f8e7406c420 2023-10-30T14:17:28.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.161+0000 7f8e7b90f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e7415a2d0 con 0x7f8e74159980 2023-10-30T14:17:28.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.165+0000 7f8e796ab640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f8e74153bb0 0x7f8e740c6410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:58474/0 (socket says 172.21.15.57:58474) 2023-10-30T14:17:28.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.165+0000 7f8e796ab640 1 -- 172.21.15.57:0/1019915198 learned_addr learned my addr 172.21.15.57:0/1019915198 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:17:28.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.165+0000 7f8e79eac640 1 --2- 172.21.15.57:0/1019915198 >> v2:172.21.15.57:3300/0 conn(0x7f8e74159980 0x7f8e74159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:28.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.165+0000 7f8e6bfff640 1 -- 172.21.15.57:0/1019915198 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938002635 0 0) 0x7f8e740c5f80 con 0x7f8e74153bb0 2023-10-30T14:17:28.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.165+0000 7f8e79eac640 1 -- 172.21.15.57:0/1019915198 >> v1:172.21.15.88:6789/0 conn(0x7f8e7406c420 legacy=0x7f8e740c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:28.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.165+0000 7f8e79eac640 1 -- 172.21.15.57:0/1019915198 >> v1:172.21.15.57:6789/0 conn(0x7f8e74153bb0 legacy=0x7f8e740c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:28.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.165+0000 7f8e79eac640 1 -- 172.21.15.57:0/1019915198 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7415a580 con 0x7f8e74159980 2023-10-30T14:17:28.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.165+0000 7f8e79eac640 1 --2- 172.21.15.57:0/1019915198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e74159980 0x7f8e74159d90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c009d90 tx=0x7f8e6c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=987546fd953ca5b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:28.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.165+0000 7f8e6bfff640 1 -- 172.21.15.57:0/1019915198 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c0af600 con 0x7f8e74159980 2023-10-30T14:17:28.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.165+0000 7f8e6bfff640 1 -- 172.21.15.57:0/1019915198 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6c0ab050 con 0x7f8e74159980 2023-10-30T14:17:28.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.165+0000 7f8e6bfff640 1 -- 172.21.15.57:0/1019915198 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c0af980 con 0x7f8e74159980 2023-10-30T14:17:28.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e7b90f640 1 -- 172.21.15.57:0/1019915198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e74159980 msgr2=0x7f8e74159d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:28.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e7b90f640 1 --2- 172.21.15.57:0/1019915198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e74159980 0x7f8e74159d90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c009d90 tx=0x7f8e6c01f130 comp rx=0 tx=0).stop 2023-10-30T14:17:28.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e7b90f640 1 -- 172.21.15.57:0/1019915198 shutdown_connections 2023-10-30T14:17:28.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e7b90f640 1 --2- 172.21.15.57:0/1019915198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e74159980 0x7f8e74159d90 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:28.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e7b90f640 1 -- 172.21.15.57:0/1019915198 >> 172.21.15.57:0/1019915198 conn(0x7f8e740cac70 msgr2=0x7f8e740cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:28.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e7b90f640 1 -- 172.21.15.57:0/1019915198 shutdown_connections 2023-10-30T14:17:28.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e7b90f640 1 -- 172.21.15.57:0/1019915198 wait complete. 2023-10-30T14:17:28.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e7b90f640 1 Processor -- start 2023-10-30T14:17:28.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e7b90f640 1 -- start start 2023-10-30T14:17:28.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e7b90f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e7406c420 0x7f8e74153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:28.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e7b90f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e74153bb0 0x7f8e7414f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:28.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e7b90f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e74159980 0x7f8e74150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:28.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e7b90f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e741509d0 con 0x7f8e74153bb0 2023-10-30T14:17:28.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e7b90f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e741503c0 con 0x7f8e74159980 2023-10-30T14:17:28.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e7b90f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e740c7fa0 con 0x7f8e7406c420 2023-10-30T14:17:28.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e78eaa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e74153bb0 0x7f8e7414f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:28.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e78eaa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e74153bb0 0x7f8e7414f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53650/0 (socket says 172.21.15.57:53650) 2023-10-30T14:17:28.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e78eaa640 1 -- 172.21.15.57:0/3236609731 learned_addr learned my addr 172.21.15.57:0/3236609731 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:28.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e796ab640 1 --2- 172.21.15.57:0/3236609731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e7406c420 0x7f8e74153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:28.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e79eac640 1 --2- 172.21.15.57:0/3236609731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e74159980 0x7f8e74150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:28.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e78eaa640 1 -- 172.21.15.57:0/3236609731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e7406c420 msgr2=0x7f8e74153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:28.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e78eaa640 1 --2- 172.21.15.57:0/3236609731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e7406c420 0x7f8e74153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:28.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e78eaa640 1 -- 172.21.15.57:0/3236609731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e74159980 msgr2=0x7f8e74150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:28.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e78eaa640 1 --2- 172.21.15.57:0/3236609731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e74159980 0x7f8e74150e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:28.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e78eaa640 1 -- 172.21.15.57:0/3236609731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6c009a70 con 0x7f8e74153bb0 2023-10-30T14:17:28.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e79eac640 1 --2- 172.21.15.57:0/3236609731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e74159980 0x7f8e74150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:17:28.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.169+0000 7f8e78eaa640 1 --2- 172.21.15.57:0/3236609731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e74153bb0 0x7f8e7414f970 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8e6400c3c0 tx=0x7f8e6400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:28.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.173+0000 7f8e69ffb640 1 -- 172.21.15.57:0/3236609731 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e640052d0 con 0x7f8e74153bb0 2023-10-30T14:17:28.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.173+0000 7f8e69ffb640 1 -- 172.21.15.57:0/3236609731 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e64005430 con 0x7f8e74153bb0 2023-10-30T14:17:28.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.173+0000 7f8e7b90f640 1 -- 172.21.15.57:0/3236609731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e740c8280 con 0x7f8e74153bb0 2023-10-30T14:17:28.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.173+0000 7f8e69ffb640 1 -- 172.21.15.57:0/3236609731 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e64003840 con 0x7f8e74153bb0 2023-10-30T14:17:28.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.173+0000 7f8e7b90f640 1 -- 172.21.15.57:0/3236609731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e740c8560 con 0x7f8e74153bb0 2023-10-30T14:17:28.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.173+0000 7f8e7b90f640 1 -- 172.21.15.57:0/3236609731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e7404e170 con 0x7f8e74153bb0 2023-10-30T14:17:28.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.173+0000 7f8e69ffb640 1 -- 172.21.15.57:0/3236609731 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8e640039a0 con 0x7f8e74153bb0 2023-10-30T14:17:28.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.173+0000 7f8e69ffb640 1 --2- 172.21.15.57:0/3236609731 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8e580664a0 0x7f8e58068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:28.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.173+0000 7f8e69ffb640 1 -- 172.21.15.57:0/3236609731 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f8e64083430 con 0x7f8e74153bb0 2023-10-30T14:17:28.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.173+0000 7f8e796ab640 1 --2- 172.21.15.57:0/3236609731 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8e580664a0 0x7f8e58068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:28.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.177+0000 7f8e796ab640 1 --2- 172.21.15.57:0/3236609731 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8e580664a0 0x7f8e58068960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8e70000f70 tx=0x7f8e70007240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:28.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.177+0000 7f8e69ffb640 1 -- 172.21.15.57:0/3236609731 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8e64050840 con 0x7f8e74153bb0 2023-10-30T14:17:28.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.365+0000 7f8e7b90f640 1 -- 172.21.15.57:0/3236609731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e7406bc80 con 0x7f8e74153bb0 2023-10-30T14:17:28.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.365+0000 7f8e69ffb640 1 -- 172.21.15.57:0/3236609731 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6528 (secure 0 0 0) 0x7f8e64053e60 con 0x7f8e74153bb0 2023-10-30T14:17:28.372 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:28.372 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5133,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:26.649359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:28.372 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 28 2023-10-30T14:17:28.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.369+0000 7f8e7b90f640 1 -- 172.21.15.57:0/3236609731 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8e580664a0 msgr2=0x7f8e58068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:28.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.369+0000 7f8e7b90f640 1 --2- 172.21.15.57:0/3236609731 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8e580664a0 0x7f8e58068960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8e70000f70 tx=0x7f8e70007240 comp rx=0 tx=0).stop 2023-10-30T14:17:28.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.369+0000 7f8e7b90f640 1 -- 172.21.15.57:0/3236609731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e74153bb0 msgr2=0x7f8e7414f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:28.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.369+0000 7f8e7b90f640 1 --2- 172.21.15.57:0/3236609731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e74153bb0 0x7f8e7414f970 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8e6400c3c0 tx=0x7f8e6400c880 comp rx=0 tx=0).stop 2023-10-30T14:17:28.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.369+0000 7f8e7b90f640 1 -- 172.21.15.57:0/3236609731 shutdown_connections 2023-10-30T14:17:28.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.369+0000 7f8e7b90f640 1 --2- 172.21.15.57:0/3236609731 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8e580664a0 0x7f8e58068960 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:28.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.369+0000 7f8e7b90f640 1 --2- 172.21.15.57:0/3236609731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e74159980 0x7f8e74150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:28.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.369+0000 7f8e7b90f640 1 --2- 172.21.15.57:0/3236609731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e74153bb0 0x7f8e7414f970 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:28.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.369+0000 7f8e7b90f640 1 --2- 172.21.15.57:0/3236609731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e7406c420 0x7f8e74153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:28.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.369+0000 7f8e7b90f640 1 -- 172.21.15.57:0/3236609731 >> 172.21.15.57:0/3236609731 conn(0x7f8e740cac70 msgr2=0x7f8e74071230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:28.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.369+0000 7f8e7b90f640 1 -- 172.21.15.57:0/3236609731 shutdown_connections 2023-10-30T14:17:28.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:28.369+0000 7f8e7b90f640 1 -- 172.21.15.57:0/3236609731 wait complete. 2023-10-30T14:17:28.393 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:17:29.929 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:17:29.933 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:17:30.395 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:30.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.545+0000 7ffb7331d640 1 Processor -- start 2023-10-30T14:17:30.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.545+0000 7ffb7331d640 1 -- start start 2023-10-30T14:17:30.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.545+0000 7ffb7331d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ffb6c0c4a10 0x7ffb6c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:30.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.545+0000 7ffb7331d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ffb6c0c42a0 0x7ffb6c0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:30.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.545+0000 7ffb7331d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6c159f20 con 0x7ffb6c153c80 2023-10-30T14:17:30.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.545+0000 7ffb7331d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb6c15dec0 con 0x7ffb6c0c42a0 2023-10-30T14:17:30.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.545+0000 7ffb7331d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb6c1599a0 con 0x7ffb6c0c4a10 2023-10-30T14:17:30.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.545+0000 7ffb718ba640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ffb6c0c42a0 0x7ffb6c0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:30.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.545+0000 7ffb718ba640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ffb6c0c42a0 0x7ffb6c0ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53654/0 (socket says 172.21.15.57:53654) 2023-10-30T14:17:30.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.545+0000 7ffb718ba640 1 -- 172.21.15.57:0/1685545258 learned_addr learned my addr 172.21.15.57:0/1685545258 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:30.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.545+0000 7ffb708b8640 1 --2- 172.21.15.57:0/1685545258 >> v2:172.21.15.88:3300/0 conn(0x7ffb6c0c4a10 0x7ffb6c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:30.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.545+0000 7ffb718ba640 1 -- 172.21.15.57:0/1685545258 >> v2:172.21.15.88:3300/0 conn(0x7ffb6c0c4a10 msgr2=0x7ffb6c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:30.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.545+0000 7ffb718ba640 1 --2- 172.21.15.57:0/1685545258 >> v2:172.21.15.88:3300/0 conn(0x7ffb6c0c4a10 0x7ffb6c15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:30.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.545+0000 7ffb718ba640 1 -- 172.21.15.57:0/1685545258 >> v1:172.21.15.88:6789/0 conn(0x7ffb6c153c80 legacy=0x7ffb6c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:30.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.545+0000 7ffb718ba640 1 -- 172.21.15.57:0/1685545258 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6c15d620 con 0x7ffb6c0c42a0 2023-10-30T14:17:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.549+0000 7ffb718ba640 1 --2- 172.21.15.57:0/1685545258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c0c42a0 0x7ffb6c0ca1e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c06be50 tx=0x7ffb6401f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b94c8d79bcd0ed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.549+0000 7ffb63fff640 1 -- 172.21.15.57:0/1685545258 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb640af400 con 0x7ffb6c0c42a0 2023-10-30T14:17:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.549+0000 7ffb63fff640 1 -- 172.21.15.57:0/1685545258 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb640ab050 con 0x7ffb6c0c42a0 2023-10-30T14:17:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.549+0000 7ffb63fff640 1 -- 172.21.15.57:0/1685545258 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb640af780 con 0x7ffb6c0c42a0 2023-10-30T14:17:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.549+0000 7ffb7331d640 1 -- 172.21.15.57:0/1685545258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c0c42a0 msgr2=0x7ffb6c0ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.549+0000 7ffb7331d640 1 --2- 172.21.15.57:0/1685545258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c0c42a0 0x7ffb6c0ca1e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c06be50 tx=0x7ffb6401f180 comp rx=0 tx=0).stop 2023-10-30T14:17:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.549+0000 7ffb7331d640 1 -- 172.21.15.57:0/1685545258 shutdown_connections 2023-10-30T14:17:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.549+0000 7ffb7331d640 1 --2- 172.21.15.57:0/1685545258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c0c42a0 0x7ffb6c0ca1e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.549+0000 7ffb7331d640 1 --2- 172.21.15.57:0/1685545258 >> v2:172.21.15.88:3300/0 conn(0x7ffb6c0c4a10 0x7ffb6c15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.549+0000 7ffb7331d640 1 -- 172.21.15.57:0/1685545258 >> 172.21.15.57:0/1685545258 conn(0x7ffb6c0715c0 msgr2=0x7ffb6c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.549+0000 7ffb7331d640 1 -- 172.21.15.57:0/1685545258 shutdown_connections 2023-10-30T14:17:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.549+0000 7ffb7331d640 1 -- 172.21.15.57:0/1685545258 wait complete. 2023-10-30T14:17:30.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb7331d640 1 Processor -- start 2023-10-30T14:17:30.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb7331d640 1 -- start start 2023-10-30T14:17:30.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb7331d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c0c42a0 0x7ffb6c1ef480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:30.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb7331d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb6c0c4a10 0x7ffb6c15c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:30.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb7331d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6c153c80 0x7ffb6c0cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:30.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb7331d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c0cf170 con 0x7ffb6c0c42a0 2023-10-30T14:17:30.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb7331d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c1eeff0 con 0x7ffb6c0c4a10 2023-10-30T14:17:30.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb7331d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c1f2940 con 0x7ffb6c153c80 2023-10-30T14:17:30.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb710b9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c0c42a0 0x7ffb6c1ef480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:30.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb710b9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c0c42a0 0x7ffb6c1ef480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53666/0 (socket says 172.21.15.57:53666) 2023-10-30T14:17:30.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb710b9640 1 -- 172.21.15.57:0/2029270068 learned_addr learned my addr 172.21.15.57:0/2029270068 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:30.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb718ba640 1 --2- 172.21.15.57:0/2029270068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6c153c80 0x7ffb6c0cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:30.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb708b8640 1 --2- 172.21.15.57:0/2029270068 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb6c0c4a10 0x7ffb6c15c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:30.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb710b9640 1 -- 172.21.15.57:0/2029270068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6c153c80 msgr2=0x7ffb6c0cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:30.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb710b9640 1 --2- 172.21.15.57:0/2029270068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6c153c80 0x7ffb6c0cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:30.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb710b9640 1 -- 172.21.15.57:0/2029270068 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb6c0c4a10 msgr2=0x7ffb6c15c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:30.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb710b9640 1 --2- 172.21.15.57:0/2029270068 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb6c0c4a10 0x7ffb6c15c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:30.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb710b9640 1 -- 172.21.15.57:0/2029270068 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb64009b30 con 0x7ffb6c0c42a0 2023-10-30T14:17:30.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb710b9640 1 --2- 172.21.15.57:0/2029270068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c0c42a0 0x7ffb6c1ef480 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ffb6800b410 tx=0x7ffb6800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:30.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb61ffb640 1 -- 172.21.15.57:0/2029270068 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb68004020 con 0x7ffb6c0c42a0 2023-10-30T14:17:30.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb61ffb640 1 -- 172.21.15.57:0/2029270068 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb68004180 con 0x7ffb6c0c42a0 2023-10-30T14:17:30.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb61ffb640 1 -- 172.21.15.57:0/2029270068 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb68011630 con 0x7ffb6c0c42a0 2023-10-30T14:17:30.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb7331d640 1 -- 172.21.15.57:0/2029270068 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb6c1ee9a0 con 0x7ffb6c0c42a0 2023-10-30T14:17:30.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb7331d640 1 -- 172.21.15.57:0/2029270068 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb6c071e30 con 0x7ffb6c0c42a0 2023-10-30T14:17:30.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.553+0000 7ffb7331d640 1 -- 172.21.15.57:0/2029270068 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6c04e170 con 0x7ffb6c0c42a0 2023-10-30T14:17:30.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.557+0000 7ffb61ffb640 1 -- 172.21.15.57:0/2029270068 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffb68011790 con 0x7ffb6c0c42a0 2023-10-30T14:17:30.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.557+0000 7ffb61ffb640 1 --2- 172.21.15.57:0/2029270068 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffb500664a0 0x7ffb50068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:30.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.557+0000 7ffb61ffb640 1 -- 172.21.15.57:0/2029270068 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7ffb68083e00 con 0x7ffb6c0c42a0 2023-10-30T14:17:30.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.557+0000 7ffb708b8640 1 --2- 172.21.15.57:0/2029270068 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffb500664a0 0x7ffb50068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:30.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.557+0000 7ffb708b8640 1 --2- 172.21.15.57:0/2029270068 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffb500664a0 0x7ffb50068960 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c0c4e30 tx=0x7ffb5c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:30.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.561+0000 7ffb61ffb640 1 -- 172.21.15.57:0/2029270068 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffb680510c0 con 0x7ffb6c0c42a0 2023-10-30T14:17:30.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.753+0000 7ffb7331d640 1 -- 172.21.15.57:0/2029270068 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb6c0cb9d0 con 0x7ffb6c0c42a0 2023-10-30T14:17:30.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.753+0000 7ffb61ffb640 1 -- 172.21.15.57:0/2029270068 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6528 (secure 0 0 0) 0x7ffb680546e0 con 0x7ffb6c0c42a0 2023-10-30T14:17:30.756 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:30.757 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5133,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:26.649359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:30.757 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 28 2023-10-30T14:17:30.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.753+0000 7ffb7331d640 1 -- 172.21.15.57:0/2029270068 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffb500664a0 msgr2=0x7ffb50068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:30.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.753+0000 7ffb7331d640 1 --2- 172.21.15.57:0/2029270068 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffb500664a0 0x7ffb50068960 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c0c4e30 tx=0x7ffb5c008040 comp rx=0 tx=0).stop 2023-10-30T14:17:30.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.753+0000 7ffb7331d640 1 -- 172.21.15.57:0/2029270068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c0c42a0 msgr2=0x7ffb6c1ef480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:30.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.753+0000 7ffb7331d640 1 --2- 172.21.15.57:0/2029270068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c0c42a0 0x7ffb6c1ef480 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ffb6800b410 tx=0x7ffb6800b8d0 comp rx=0 tx=0).stop 2023-10-30T14:17:30.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.753+0000 7ffb7331d640 1 -- 172.21.15.57:0/2029270068 shutdown_connections 2023-10-30T14:17:30.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.753+0000 7ffb7331d640 1 --2- 172.21.15.57:0/2029270068 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffb500664a0 0x7ffb50068960 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:30.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.753+0000 7ffb7331d640 1 --2- 172.21.15.57:0/2029270068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6c153c80 0x7ffb6c0cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:30.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.753+0000 7ffb7331d640 1 --2- 172.21.15.57:0/2029270068 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb6c0c4a10 0x7ffb6c15c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:30.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.753+0000 7ffb7331d640 1 --2- 172.21.15.57:0/2029270068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c0c42a0 0x7ffb6c1ef480 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:30.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.753+0000 7ffb7331d640 1 -- 172.21.15.57:0/2029270068 >> 172.21.15.57:0/2029270068 conn(0x7ffb6c0715c0 msgr2=0x7ffb6c06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:30.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.753+0000 7ffb7331d640 1 -- 172.21.15.57:0/2029270068 shutdown_connections 2023-10-30T14:17:30.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:30.753+0000 7ffb7331d640 1 -- 172.21.15.57:0/2029270068 wait complete. 2023-10-30T14:17:30.779 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:17:32.779 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:32.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.929+0000 7fbd5c70f640 1 Processor -- start 2023-10-30T14:17:32.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.929+0000 7fbd5c70f640 1 -- start start 2023-10-30T14:17:32.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.929+0000 7fbd5c70f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd541537f0 0x7fbd540701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:32.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.929+0000 7fbd5c70f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbd5406b2f0 0x7fbd5406b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:32.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.929+0000 7fbd5c70f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd540c2aa0 con 0x7fbd5415a2e0 2023-10-30T14:17:32.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.929+0000 7fbd5c70f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd540c3ab0 con 0x7fbd5406b2f0 2023-10-30T14:17:32.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.929+0000 7fbd5c70f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd540706e0 con 0x7fbd541537f0 2023-10-30T14:17:32.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.933+0000 7fbd59caa640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fbd5415a2e0 0x7fbd540ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:58490/0 (socket says 172.21.15.57:58490) 2023-10-30T14:17:32.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.933+0000 7fbd59caa640 1 -- 172.21.15.57:0/2477033704 learned_addr learned my addr 172.21.15.57:0/2477033704 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:17:32.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.933+0000 7fbd5a4ab640 1 --2- 172.21.15.57:0/2477033704 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd541537f0 0x7fbd540701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:32.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.933+0000 7fbd5acac640 1 --2- 172.21.15.57:0/2477033704 >> v2:172.21.15.88:3300/0 conn(0x7fbd5406b2f0 0x7fbd5406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:32.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.933+0000 7fbd594a9640 1 -- 172.21.15.57:0/2477033704 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280852640 0 0) 0x7fbd540c2aa0 con 0x7fbd5415a2e0 2023-10-30T14:17:32.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.933+0000 7fbd594a9640 1 -- 172.21.15.57:0/2477033704 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd40003530 con 0x7fbd5415a2e0 2023-10-30T14:17:32.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.933+0000 7fbd5a4ab640 1 -- 172.21.15.57:0/2477033704 >> v2:172.21.15.88:3300/0 conn(0x7fbd5406b2f0 msgr2=0x7fbd5406b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:32.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.933+0000 7fbd5a4ab640 1 --2- 172.21.15.57:0/2477033704 >> v2:172.21.15.88:3300/0 conn(0x7fbd5406b2f0 0x7fbd5406b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:32.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.933+0000 7fbd594a9640 1 -- 172.21.15.57:0/2477033704 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3120962272 0 0) 0x7fbd40003530 con 0x7fbd5415a2e0 2023-10-30T14:17:32.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.933+0000 7fbd5a4ab640 1 -- 172.21.15.57:0/2477033704 >> v1:172.21.15.57:6789/0 conn(0x7fbd5415a2e0 legacy=0x7fbd540ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:32.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.933+0000 7fbd5a4ab640 1 -- 172.21.15.57:0/2477033704 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5406bca0 con 0x7fbd541537f0 2023-10-30T14:17:32.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.933+0000 7fbd5a4ab640 1 --2- 172.21.15.57:0/2477033704 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd541537f0 0x7fbd540701a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbd500097c0 tx=0x7fbd5001eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b1998664e6499b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:32.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd594a9640 1 -- 172.21.15.57:0/2477033704 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd44001680 con 0x7fbd541537f0 2023-10-30T14:17:32.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd594a9640 1 -- 172.21.15.57:0/2477033704 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd500a85d0 con 0x7fbd541537f0 2023-10-30T14:17:32.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd594a9640 1 -- 172.21.15.57:0/2477033704 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd500a8940 con 0x7fbd541537f0 2023-10-30T14:17:32.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd5c70f640 1 -- 172.21.15.57:0/2477033704 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd541537f0 msgr2=0x7fbd540701a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:32.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd5c70f640 1 --2- 172.21.15.57:0/2477033704 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd541537f0 0x7fbd540701a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbd500097c0 tx=0x7fbd5001eb90 comp rx=0 tx=0).stop 2023-10-30T14:17:32.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd5c70f640 1 -- 172.21.15.57:0/2477033704 shutdown_connections 2023-10-30T14:17:32.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd5c70f640 1 --2- 172.21.15.57:0/2477033704 >> v2:172.21.15.88:3300/0 conn(0x7fbd5406b2f0 0x7fbd5406b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:32.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd5c70f640 1 --2- 172.21.15.57:0/2477033704 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd541537f0 0x7fbd540701a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:32.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd5c70f640 1 -- 172.21.15.57:0/2477033704 >> 172.21.15.57:0/2477033704 conn(0x7fbd54071fb0 msgr2=0x7fbd540723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:32.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd5c70f640 1 -- 172.21.15.57:0/2477033704 shutdown_connections 2023-10-30T14:17:32.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd5c70f640 1 -- 172.21.15.57:0/2477033704 wait complete. 2023-10-30T14:17:32.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd5c70f640 1 Processor -- start 2023-10-30T14:17:32.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd5c70f640 1 -- start start 2023-10-30T14:17:32.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd5c70f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd5406b2f0 0x7fbd541f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:32.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd5c70f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd541537f0 0x7fbd5414ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:32.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd5c70f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd5415a2e0 0x7fbd5415cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:32.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd5c70f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5415be00 con 0x7fbd541537f0 2023-10-30T14:17:32.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd5c70f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd541f2fb0 con 0x7fbd5406b2f0 2023-10-30T14:17:32.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd5c70f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd54158fe0 con 0x7fbd5415a2e0 2023-10-30T14:17:32.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd59caa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd541537f0 0x7fbd5414ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:32.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd59caa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd541537f0 0x7fbd5414ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53682/0 (socket says 172.21.15.57:53682) 2023-10-30T14:17:32.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.937+0000 7fbd59caa640 1 -- 172.21.15.57:0/3428400293 learned_addr learned my addr 172.21.15.57:0/3428400293 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:32.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.941+0000 7fbd5a4ab640 1 --2- 172.21.15.57:0/3428400293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd5406b2f0 0x7fbd541f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:32.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.941+0000 7fbd5acac640 1 --2- 172.21.15.57:0/3428400293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd5415a2e0 0x7fbd5415cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:32.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.941+0000 7fbd59caa640 1 -- 172.21.15.57:0/3428400293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd5415a2e0 msgr2=0x7fbd5415cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:32.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.941+0000 7fbd59caa640 1 --2- 172.21.15.57:0/3428400293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd5415a2e0 0x7fbd5415cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:32.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.941+0000 7fbd59caa640 1 -- 172.21.15.57:0/3428400293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd5406b2f0 msgr2=0x7fbd541f2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:32.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.941+0000 7fbd59caa640 1 --2- 172.21.15.57:0/3428400293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd5406b2f0 0x7fbd541f2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:32.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.941+0000 7fbd59caa640 1 -- 172.21.15.57:0/3428400293 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd50009420 con 0x7fbd541537f0 2023-10-30T14:17:32.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.941+0000 7fbd59caa640 1 --2- 172.21.15.57:0/3428400293 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd541537f0 0x7fbd5414ee60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbd4400c750 tx=0x7fbd4400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:32.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.941+0000 7fbd4affd640 1 -- 172.21.15.57:0/3428400293 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd44005ae0 con 0x7fbd541537f0 2023-10-30T14:17:32.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.941+0000 7fbd4affd640 1 -- 172.21.15.57:0/3428400293 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd44005c40 con 0x7fbd541537f0 2023-10-30T14:17:32.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.941+0000 7fbd4affd640 1 -- 172.21.15.57:0/3428400293 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd44007070 con 0x7fbd541537f0 2023-10-30T14:17:32.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.941+0000 7fbd5c70f640 1 -- 172.21.15.57:0/3428400293 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd5415cab0 con 0x7fbd541537f0 2023-10-30T14:17:32.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.941+0000 7fbd5c70f640 1 -- 172.21.15.57:0/3428400293 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd54159230 con 0x7fbd541537f0 2023-10-30T14:17:32.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.941+0000 7fbd5c70f640 1 -- 172.21.15.57:0/3428400293 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5415cc40 con 0x7fbd541537f0 2023-10-30T14:17:32.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.941+0000 7fbd4affd640 1 -- 172.21.15.57:0/3428400293 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbd440071d0 con 0x7fbd541537f0 2023-10-30T14:17:32.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.941+0000 7fbd4affd640 1 --2- 172.21.15.57:0/3428400293 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd380664a0 0x7fbd38068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:32.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.941+0000 7fbd4affd640 1 -- 172.21.15.57:0/3428400293 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7fbd44083d80 con 0x7fbd541537f0 2023-10-30T14:17:32.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.945+0000 7fbd5a4ab640 1 --2- 172.21.15.57:0/3428400293 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd380664a0 0x7fbd38068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:32.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.945+0000 7fbd5a4ab640 1 --2- 172.21.15.57:0/3428400293 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd380664a0 0x7fbd38068960 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbd500b2000 tx=0x7fbd500aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:32.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:32.945+0000 7fbd4affd640 1 -- 172.21.15.57:0/3428400293 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbd44051060 con 0x7fbd541537f0 2023-10-30T14:17:33.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:33.137+0000 7fbd5c70f640 1 -- 172.21.15.57:0/3428400293 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd5404e170 con 0x7fbd541537f0 2023-10-30T14:17:33.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:33.137+0000 7fbd4affd640 1 -- 172.21.15.57:0/3428400293 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6528 (secure 0 0 0) 0x7fbd44054680 con 0x7fbd541537f0 2023-10-30T14:17:33.141 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:33.141 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5133,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:26.649359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:33.141 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 28 2023-10-30T14:17:33.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:33.137+0000 7fbd5c70f640 1 -- 172.21.15.57:0/3428400293 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd380664a0 msgr2=0x7fbd38068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:33.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:33.137+0000 7fbd5c70f640 1 --2- 172.21.15.57:0/3428400293 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd380664a0 0x7fbd38068960 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbd500b2000 tx=0x7fbd500aa070 comp rx=0 tx=0).stop 2023-10-30T14:17:33.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:33.137+0000 7fbd5c70f640 1 -- 172.21.15.57:0/3428400293 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd541537f0 msgr2=0x7fbd5414ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:33.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:33.137+0000 7fbd5c70f640 1 --2- 172.21.15.57:0/3428400293 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd541537f0 0x7fbd5414ee60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbd4400c750 tx=0x7fbd4400cc10 comp rx=0 tx=0).stop 2023-10-30T14:17:33.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:33.137+0000 7fbd5c70f640 1 -- 172.21.15.57:0/3428400293 shutdown_connections 2023-10-30T14:17:33.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:33.137+0000 7fbd5c70f640 1 --2- 172.21.15.57:0/3428400293 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd380664a0 0x7fbd38068960 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:33.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:33.137+0000 7fbd5c70f640 1 --2- 172.21.15.57:0/3428400293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd5415a2e0 0x7fbd5415cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:33.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:33.137+0000 7fbd5c70f640 1 --2- 172.21.15.57:0/3428400293 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd541537f0 0x7fbd5414ee60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:33.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:33.137+0000 7fbd5c70f640 1 --2- 172.21.15.57:0/3428400293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd5406b2f0 0x7fbd541f2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:33.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:33.137+0000 7fbd5c70f640 1 -- 172.21.15.57:0/3428400293 >> 172.21.15.57:0/3428400293 conn(0x7fbd54071fb0 msgr2=0x7fbd54150af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:33.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:33.137+0000 7fbd5c70f640 1 -- 172.21.15.57:0/3428400293 shutdown_connections 2023-10-30T14:17:33.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:33.137+0000 7fbd5c70f640 1 -- 172.21.15.57:0/3428400293 wait complete. 2023-10-30T14:17:33.161 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:17:35.162 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:35.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.313+0000 7fb06269e640 1 Processor -- start 2023-10-30T14:17:35.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.313+0000 7fb06269e640 1 -- start start 2023-10-30T14:17:35.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.313+0000 7fb06269e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb05c153c10 0x7fb05c065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:35.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.313+0000 7fb06269e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb05c070c30 0x7fb05c071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:35.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.313+0000 7fb06269e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05c06e390 con 0x7fb05c159970 2023-10-30T14:17:35.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.317+0000 7fb06269e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb05c072390 con 0x7fb05c153c10 2023-10-30T14:17:35.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.317+0000 7fb06269e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb05c0660e0 con 0x7fb05c070c30 2023-10-30T14:17:35.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.317+0000 7fb05bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb05c153c10 0x7fb05c065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:35.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.317+0000 7fb05bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb05c153c10 0x7fb05c065ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59606/0 (socket says 172.21.15.57:59606) 2023-10-30T14:17:35.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.317+0000 7fb05bfff640 1 -- 172.21.15.57:0/3778572421 learned_addr learned my addr 172.21.15.57:0/3778572421 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:35.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.317+0000 7fb05b7fe640 1 --2- 172.21.15.57:0/3778572421 >> v2:172.21.15.88:3300/0 conn(0x7fb05c070c30 0x7fb05c071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:35.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.317+0000 7fb05bfff640 1 -- 172.21.15.57:0/3778572421 >> v2:172.21.15.88:3300/0 conn(0x7fb05c070c30 msgr2=0x7fb05c071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:35.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.317+0000 7fb05bfff640 1 --2- 172.21.15.57:0/3778572421 >> v2:172.21.15.88:3300/0 conn(0x7fb05c070c30 0x7fb05c071070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:35.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.317+0000 7fb05bfff640 1 -- 172.21.15.57:0/3778572421 >> v1:172.21.15.88:6789/0 conn(0x7fb05c159970 legacy=0x7fb05c159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:35.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.317+0000 7fb05bfff640 1 -- 172.21.15.57:0/3778572421 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05c0715b0 con 0x7fb05c153c10 2023-10-30T14:17:35.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.317+0000 7fb05bfff640 1 --2- 172.21.15.57:0/3778572421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb05c153c10 0x7fb05c065ba0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb05c06f200 tx=0x7fb04c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=788bc91be05339fd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:35.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.317+0000 7fb05affd640 1 -- 172.21.15.57:0/3778572421 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c0ab600 con 0x7fb05c153c10 2023-10-30T14:17:35.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.317+0000 7fb05affd640 1 -- 172.21.15.57:0/3778572421 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb04c0a7050 con 0x7fb05c153c10 2023-10-30T14:17:35.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.317+0000 7fb05affd640 1 -- 172.21.15.57:0/3778572421 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c0ab980 con 0x7fb05c153c10 2023-10-30T14:17:35.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb06269e640 1 -- 172.21.15.57:0/3778572421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb05c153c10 msgr2=0x7fb05c065ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:35.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb06269e640 1 --2- 172.21.15.57:0/3778572421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb05c153c10 0x7fb05c065ba0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb05c06f200 tx=0x7fb04c01c430 comp rx=0 tx=0).stop 2023-10-30T14:17:35.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb06269e640 1 -- 172.21.15.57:0/3778572421 shutdown_connections 2023-10-30T14:17:35.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb06269e640 1 --2- 172.21.15.57:0/3778572421 >> v2:172.21.15.88:3300/0 conn(0x7fb05c070c30 0x7fb05c071070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:35.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb06269e640 1 --2- 172.21.15.57:0/3778572421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb05c153c10 0x7fb05c065ba0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:35.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb06269e640 1 -- 172.21.15.57:0/3778572421 >> 172.21.15.57:0/3778572421 conn(0x7fb05c0c7f60 msgr2=0x7fb05c0c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:35.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb06269e640 1 -- 172.21.15.57:0/3778572421 shutdown_connections 2023-10-30T14:17:35.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb06269e640 1 -- 172.21.15.57:0/3778572421 wait complete. 2023-10-30T14:17:35.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb06269e640 1 Processor -- start 2023-10-30T14:17:35.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb06269e640 1 -- start start 2023-10-30T14:17:35.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb06269e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb05c070c30 0x7fb05c0c2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:35.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb06269e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb05c153c10 0x7fb05c06d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:35.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb06269e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb05c159970 0x7fb05c0ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:35.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb06269e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05c0c3050 con 0x7fb05c159970 2023-10-30T14:17:35.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb06269e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05c0c33f0 con 0x7fb05c070c30 2023-10-30T14:17:35.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb06269e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb05c0c93e0 con 0x7fb05c153c10 2023-10-30T14:17:35.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb060c3b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb05c159970 0x7fb05c0ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:35.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb060c3b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb05c159970 0x7fb05c0ca110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59616/0 (socket says 172.21.15.57:59616) 2023-10-30T14:17:35.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb060c3b640 1 -- 172.21.15.57:0/1255312532 learned_addr learned my addr 172.21.15.57:0/1255312532 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:35.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb05bfff640 1 --2- 172.21.15.57:0/1255312532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb05c070c30 0x7fb05c0c2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:35.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.321+0000 7fb05b7fe640 1 --2- 172.21.15.57:0/1255312532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb05c153c10 0x7fb05c06d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:35.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.325+0000 7fb060c3b640 1 -- 172.21.15.57:0/1255312532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb05c153c10 msgr2=0x7fb05c06d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:35.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.325+0000 7fb060c3b640 1 --2- 172.21.15.57:0/1255312532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb05c153c10 0x7fb05c06d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:35.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.325+0000 7fb060c3b640 1 -- 172.21.15.57:0/1255312532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb05c070c30 msgr2=0x7fb05c0c2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:35.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.325+0000 7fb060c3b640 1 --2- 172.21.15.57:0/1255312532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb05c070c30 0x7fb05c0c2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:35.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.325+0000 7fb060c3b640 1 -- 172.21.15.57:0/1255312532 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c0094d0 con 0x7fb05c159970 2023-10-30T14:17:35.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.325+0000 7fb05b7fe640 1 --2- 172.21.15.57:0/1255312532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb05c153c10 0x7fb05c06d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:17:35.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.325+0000 7fb060c3b640 1 --2- 172.21.15.57:0/1255312532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb05c159970 0x7fb05c0ca110 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb05400b970 tx=0x7fb05400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:35.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.325+0000 7fb058ff9640 1 -- 172.21.15.57:0/1255312532 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05400cb40 con 0x7fb05c159970 2023-10-30T14:17:35.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.325+0000 7fb058ff9640 1 -- 172.21.15.57:0/1255312532 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05400cca0 con 0x7fb05c159970 2023-10-30T14:17:35.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.325+0000 7fb058ff9640 1 -- 172.21.15.57:0/1255312532 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb054014680 con 0x7fb05c159970 2023-10-30T14:17:35.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.325+0000 7fb06269e640 1 -- 172.21.15.57:0/1255312532 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb05c0c9c80 con 0x7fb05c159970 2023-10-30T14:17:35.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.325+0000 7fb06269e640 1 -- 172.21.15.57:0/1255312532 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb05c1f6090 con 0x7fb05c159970 2023-10-30T14:17:35.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.325+0000 7fb06269e640 1 -- 172.21.15.57:0/1255312532 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb05c0c9e10 con 0x7fb05c159970 2023-10-30T14:17:35.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.325+0000 7fb058ff9640 1 -- 172.21.15.57:0/1255312532 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb05401d3b0 con 0x7fb05c159970 2023-10-30T14:17:35.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.325+0000 7fb058ff9640 1 --2- 172.21.15.57:0/1255312532 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb0480664a0 0x7fb048068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:35.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.325+0000 7fb058ff9640 1 -- 172.21.15.57:0/1255312532 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7fb054083e00 con 0x7fb05c159970 2023-10-30T14:17:35.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.325+0000 7fb05bfff640 1 --2- 172.21.15.57:0/1255312532 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb0480664a0 0x7fb048068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:35.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.329+0000 7fb05bfff640 1 --2- 172.21.15.57:0/1255312532 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb0480664a0 0x7fb048068960 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb04c0ad000 tx=0x7fb04c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:35.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.329+0000 7fb058ff9640 1 -- 172.21.15.57:0/1255312532 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb054051170 con 0x7fb05c159970 2023-10-30T14:17:35.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.521+0000 7fb06269e640 1 -- 172.21.15.57:0/1255312532 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb05c04e170 con 0x7fb05c159970 2023-10-30T14:17:35.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.521+0000 7fb058ff9640 1 -- 172.21.15.57:0/1255312532 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6528 (secure 0 0 0) 0x7fb054054790 con 0x7fb05c159970 2023-10-30T14:17:35.525 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:35.525 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5133,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:26.649359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:35.526 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 28 2023-10-30T14:17:35.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.521+0000 7fb06269e640 1 -- 172.21.15.57:0/1255312532 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb0480664a0 msgr2=0x7fb048068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:35.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.521+0000 7fb06269e640 1 --2- 172.21.15.57:0/1255312532 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb0480664a0 0x7fb048068960 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb04c0ad000 tx=0x7fb04c008040 comp rx=0 tx=0).stop 2023-10-30T14:17:35.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.521+0000 7fb06269e640 1 -- 172.21.15.57:0/1255312532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb05c159970 msgr2=0x7fb05c0ca110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:35.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.521+0000 7fb06269e640 1 --2- 172.21.15.57:0/1255312532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb05c159970 0x7fb05c0ca110 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb05400b970 tx=0x7fb05400be30 comp rx=0 tx=0).stop 2023-10-30T14:17:35.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.521+0000 7fb06269e640 1 -- 172.21.15.57:0/1255312532 shutdown_connections 2023-10-30T14:17:35.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.521+0000 7fb06269e640 1 --2- 172.21.15.57:0/1255312532 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb0480664a0 0x7fb048068960 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:35.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.521+0000 7fb06269e640 1 --2- 172.21.15.57:0/1255312532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb05c159970 0x7fb05c0ca110 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:35.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.521+0000 7fb06269e640 1 --2- 172.21.15.57:0/1255312532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb05c153c10 0x7fb05c06d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:35.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.521+0000 7fb06269e640 1 --2- 172.21.15.57:0/1255312532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb05c070c30 0x7fb05c0c2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:35.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.521+0000 7fb06269e640 1 -- 172.21.15.57:0/1255312532 >> 172.21.15.57:0/1255312532 conn(0x7fb05c0c7f60 msgr2=0x7fb05c14f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:35.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.521+0000 7fb06269e640 1 -- 172.21.15.57:0/1255312532 shutdown_connections 2023-10-30T14:17:35.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:35.521+0000 7fb06269e640 1 -- 172.21.15.57:0/1255312532 wait complete. 2023-10-30T14:17:35.547 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:17:37.548 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:37.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.697+0000 7f414613e640 1 Processor -- start 2023-10-30T14:17:37.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.697+0000 7f414613e640 1 -- start start 2023-10-30T14:17:37.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.697+0000 7f414613e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f41400701a0 0x7f414015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:37.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.697+0000 7f414613e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f414006b2f0 0x7f414006b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:37.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.697+0000 7f414613e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41400c2aa0 con 0x7f41401537f0 2023-10-30T14:17:37.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.697+0000 7f414613e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41400c3ab0 con 0x7f41400701a0 2023-10-30T14:17:37.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.697+0000 7f414613e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f414015a820 con 0x7f414006b2f0 2023-10-30T14:17:37.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.697+0000 7f413effd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f41400701a0 0x7f414015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:37.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.697+0000 7f413effd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f41400701a0 0x7f414015a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59630/0 (socket says 172.21.15.57:59630) 2023-10-30T14:17:37.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.697+0000 7f413effd640 1 -- 172.21.15.57:0/3520967320 learned_addr learned my addr 172.21.15.57:0/3520967320 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:37.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.697+0000 7f413f7fe640 1 --1- 172.21.15.57:0/3520967320 >> v1:172.21.15.57:6789/0 conn(0x7f41401537f0 0x7f41400ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:39218/0 (socket says 172.21.15.57:39218) 2023-10-30T14:17:37.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.701+0000 7f413ffff640 1 --2- 172.21.15.57:0/3520967320 >> v2:172.21.15.88:3300/0 conn(0x7f414006b2f0 0x7f414006b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:37.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.701+0000 7f413effd640 1 -- 172.21.15.57:0/3520967320 >> v2:172.21.15.88:3300/0 conn(0x7f414006b2f0 msgr2=0x7f414006b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:37.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.701+0000 7f413effd640 1 --2- 172.21.15.57:0/3520967320 >> v2:172.21.15.88:3300/0 conn(0x7f414006b2f0 0x7f414006b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:37.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.701+0000 7f413effd640 1 -- 172.21.15.57:0/3520967320 >> v1:172.21.15.57:6789/0 conn(0x7f41401537f0 legacy=0x7f41400ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:37.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.701+0000 7f413effd640 1 -- 172.21.15.57:0/3520967320 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41400708e0 con 0x7f41400701a0 2023-10-30T14:17:37.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.701+0000 7f413effd640 1 --2- 172.21.15.57:0/3520967320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41400701a0 0x7f414015a2e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f414006f4d0 tx=0x7f413401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cffd6a8f4ccba1d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:37.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.701+0000 7f413e7fc640 1 -- 172.21.15.57:0/3520967320 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41340a89f0 con 0x7f41400701a0 2023-10-30T14:17:37.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.701+0000 7f413e7fc640 1 -- 172.21.15.57:0/3520967320 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41340a8b50 con 0x7f41400701a0 2023-10-30T14:17:37.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.701+0000 7f413e7fc640 1 -- 172.21.15.57:0/3520967320 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41340b0440 con 0x7f41400701a0 2023-10-30T14:17:37.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f414613e640 1 -- 172.21.15.57:0/3520967320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41400701a0 msgr2=0x7f414015a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:37.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f414613e640 1 --2- 172.21.15.57:0/3520967320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41400701a0 0x7f414015a2e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f414006f4d0 tx=0x7f413401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:17:37.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f414613e640 1 -- 172.21.15.57:0/3520967320 shutdown_connections 2023-10-30T14:17:37.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f414613e640 1 --2- 172.21.15.57:0/3520967320 >> v2:172.21.15.88:3300/0 conn(0x7f414006b2f0 0x7f414006b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:37.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f414613e640 1 --2- 172.21.15.57:0/3520967320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41400701a0 0x7f414015a2e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:37.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f414613e640 1 -- 172.21.15.57:0/3520967320 >> 172.21.15.57:0/3520967320 conn(0x7f4140071fb0 msgr2=0x7f41400723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:37.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f414613e640 1 -- 172.21.15.57:0/3520967320 shutdown_connections 2023-10-30T14:17:37.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f414613e640 1 -- 172.21.15.57:0/3520967320 wait complete. 2023-10-30T14:17:37.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f414613e640 1 Processor -- start 2023-10-30T14:17:37.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f414613e640 1 -- start start 2023-10-30T14:17:37.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f414613e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f414006b2f0 0x7f41400c46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:37.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f414613e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41400701a0 0x7f41401f2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:37.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f414613e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41401537f0 0x7f41401f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:37.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f414613e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414014c710 con 0x7f414006b2f0 2023-10-30T14:17:37.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f414613e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4140154eb0 con 0x7f41401537f0 2023-10-30T14:17:37.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f414613e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41401f2f60 con 0x7f41400701a0 2023-10-30T14:17:37.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f413f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f414006b2f0 0x7f41400c46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:37.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f413f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f414006b2f0 0x7f41400c46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59636/0 (socket says 172.21.15.57:59636) 2023-10-30T14:17:37.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f413f7fe640 1 -- 172.21.15.57:0/1574429200 learned_addr learned my addr 172.21.15.57:0/1574429200 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:37.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f413effd640 1 --2- 172.21.15.57:0/1574429200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41400701a0 0x7f41401f2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:37.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f413ffff640 1 --2- 172.21.15.57:0/1574429200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41401537f0 0x7f41401f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:37.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f413f7fe640 1 -- 172.21.15.57:0/1574429200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41400701a0 msgr2=0x7f41401f2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:37.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f413f7fe640 1 --2- 172.21.15.57:0/1574429200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41400701a0 0x7f41401f2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:37.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f413f7fe640 1 -- 172.21.15.57:0/1574429200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41401537f0 msgr2=0x7f41401f0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:37.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f413f7fe640 1 --2- 172.21.15.57:0/1574429200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41401537f0 0x7f41401f0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:37.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f413f7fe640 1 -- 172.21.15.57:0/1574429200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41340094d0 con 0x7f414006b2f0 2023-10-30T14:17:37.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f413ffff640 1 --2- 172.21.15.57:0/1574429200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41401537f0 0x7f41401f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:17:37.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.705+0000 7f413f7fe640 1 --2- 172.21.15.57:0/1574429200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f414006b2f0 0x7f41400c46c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f413000c490 tx=0x7f413000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:37.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.709+0000 7f4127fff640 1 -- 172.21.15.57:0/1574429200 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130007480 con 0x7f414006b2f0 2023-10-30T14:17:37.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.709+0000 7f4127fff640 1 -- 172.21.15.57:0/1574429200 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41300075e0 con 0x7f414006b2f0 2023-10-30T14:17:37.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.709+0000 7f4127fff640 1 -- 172.21.15.57:0/1574429200 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130004a10 con 0x7f414006b2f0 2023-10-30T14:17:37.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.709+0000 7f414613e640 1 -- 172.21.15.57:0/1574429200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4140155100 con 0x7f414006b2f0 2023-10-30T14:17:37.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.709+0000 7f414613e640 1 -- 172.21.15.57:0/1574429200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41401556c0 con 0x7f414006b2f0 2023-10-30T14:17:37.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.709+0000 7f414613e640 1 -- 172.21.15.57:0/1574429200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f414004e170 con 0x7f414006b2f0 2023-10-30T14:17:37.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.709+0000 7f4127fff640 1 -- 172.21.15.57:0/1574429200 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4130004b70 con 0x7f414006b2f0 2023-10-30T14:17:37.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.709+0000 7f4127fff640 1 --2- 172.21.15.57:0/1574429200 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f412c0664d0 0x7f412c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:37.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.709+0000 7f4127fff640 1 -- 172.21.15.57:0/1574429200 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f4130083070 con 0x7f414006b2f0 2023-10-30T14:17:37.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.713+0000 7f413effd640 1 --2- 172.21.15.57:0/1574429200 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f412c0664d0 0x7f412c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:37.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.713+0000 7f413effd640 1 --2- 172.21.15.57:0/1574429200 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f412c0664d0 0x7f412c068990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f41340b2000 tx=0x7f4134009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:37.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.713+0000 7f4127fff640 1 -- 172.21.15.57:0/1574429200 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f41300b4090 con 0x7f414006b2f0 2023-10-30T14:17:37.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.909+0000 7f414613e640 1 -- 172.21.15.57:0/1574429200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f414006b870 con 0x7f414006b2f0 2023-10-30T14:17:37.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.909+0000 7f4127fff640 1 -- 172.21.15.57:0/1574429200 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6528 (secure 0 0 0) 0x7f41300503e0 con 0x7f414006b2f0 2023-10-30T14:17:37.914 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:37.914 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5133,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:26.649359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:37.914 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 28 2023-10-30T14:17:37.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.909+0000 7f414613e640 1 -- 172.21.15.57:0/1574429200 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f412c0664d0 msgr2=0x7f412c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:37.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.909+0000 7f414613e640 1 --2- 172.21.15.57:0/1574429200 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f412c0664d0 0x7f412c068990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f41340b2000 tx=0x7f4134009040 comp rx=0 tx=0).stop 2023-10-30T14:17:37.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.909+0000 7f414613e640 1 -- 172.21.15.57:0/1574429200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f414006b2f0 msgr2=0x7f41400c46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:37.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.909+0000 7f414613e640 1 --2- 172.21.15.57:0/1574429200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f414006b2f0 0x7f41400c46c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f413000c490 tx=0x7f413000c950 comp rx=0 tx=0).stop 2023-10-30T14:17:37.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.909+0000 7f414613e640 1 -- 172.21.15.57:0/1574429200 shutdown_connections 2023-10-30T14:17:37.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.909+0000 7f414613e640 1 --2- 172.21.15.57:0/1574429200 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f412c0664d0 0x7f412c068990 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:37.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.909+0000 7f414613e640 1 --2- 172.21.15.57:0/1574429200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41401537f0 0x7f41401f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:37.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.909+0000 7f414613e640 1 --2- 172.21.15.57:0/1574429200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41400701a0 0x7f41401f2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:37.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.909+0000 7f414613e640 1 --2- 172.21.15.57:0/1574429200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f414006b2f0 0x7f41400c46c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:37.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.909+0000 7f414613e640 1 -- 172.21.15.57:0/1574429200 >> 172.21.15.57:0/1574429200 conn(0x7f4140071fb0 msgr2=0x7f4140158860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:37.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.913+0000 7f414613e640 1 -- 172.21.15.57:0/1574429200 shutdown_connections 2023-10-30T14:17:37.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:37.913+0000 7f414613e640 1 -- 172.21.15.57:0/1574429200 wait complete. 2023-10-30T14:17:37.935 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:17:39.936 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:40.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.085+0000 7f0df2dd3640 1 Processor -- start 2023-10-30T14:17:40.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.085+0000 7f0df2dd3640 1 -- start start 2023-10-30T14:17:40.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.089+0000 7f0df2dd3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dec06c420 0x7f0dec0c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:40.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.089+0000 7f0df2dd3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dec071e00 con 0x7f0dec06bb40 2023-10-30T14:17:40.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.089+0000 7f0df2dd3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dec070020 con 0x7f0dec06b760 2023-10-30T14:17:40.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.089+0000 7f0df2dd3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dec070f10 con 0x7f0dec06c420 2023-10-30T14:17:40.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.089+0000 7f0debfff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f0dec06bb40 0x7f0dec157a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:39234/0 (socket says 172.21.15.57:39234) 2023-10-30T14:17:40.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.089+0000 7f0debfff640 1 -- 172.21.15.57:0/783175842 learned_addr learned my addr 172.21.15.57:0/783175842 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:17:40.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.089+0000 7f0df1370640 1 --2- 172.21.15.57:0/783175842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dec06c420 0x7f0dec0c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:40.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.089+0000 7f0deb7fe640 1 -- 172.21.15.57:0/783175842 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998982011 0 0) 0x7f0dec071e00 con 0x7f0dec06bb40 2023-10-30T14:17:40.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.089+0000 7f0deb7fe640 1 -- 172.21.15.57:0/783175842 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ddc003530 con 0x7f0dec06bb40 2023-10-30T14:17:40.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.089+0000 7f0deb7fe640 1 -- 172.21.15.57:0/783175842 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048058680 0 0) 0x7f0dec070020 con 0x7f0dec06b760 2023-10-30T14:17:40.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.089+0000 7f0deb7fe640 1 -- 172.21.15.57:0/783175842 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dec071e00 con 0x7f0dec06b760 2023-10-30T14:17:40.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.089+0000 7f0df1370640 1 -- 172.21.15.57:0/783175842 >> v1:172.21.15.88:6789/0 conn(0x7f0dec06b760 legacy=0x7f0dec15e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:40.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.089+0000 7f0df1370640 1 -- 172.21.15.57:0/783175842 >> v1:172.21.15.57:6789/0 conn(0x7f0dec06bb40 legacy=0x7f0dec157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:40.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.089+0000 7f0df1370640 1 -- 172.21.15.57:0/783175842 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dec0711f0 con 0x7f0dec06c420 2023-10-30T14:17:40.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.093+0000 7f0df1370640 1 --2- 172.21.15.57:0/783175842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dec06c420 0x7f0dec0c57a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0de4009dc0 tx=0x7f0de401c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5953704b049638e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:40.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.093+0000 7f0deb7fe640 1 -- 172.21.15.57:0/783175842 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de40a5a00 con 0x7f0dec06c420 2023-10-30T14:17:40.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.093+0000 7f0deb7fe640 1 -- 172.21.15.57:0/783175842 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de40a5b60 con 0x7f0dec06c420 2023-10-30T14:17:40.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.093+0000 7f0deb7fe640 1 -- 172.21.15.57:0/783175842 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de40ad440 con 0x7f0dec06c420 2023-10-30T14:17:40.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.093+0000 7f0df2dd3640 1 -- 172.21.15.57:0/783175842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dec06c420 msgr2=0x7f0dec0c57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:40.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.093+0000 7f0df2dd3640 1 --2- 172.21.15.57:0/783175842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dec06c420 0x7f0dec0c57a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0de4009dc0 tx=0x7f0de401c940 comp rx=0 tx=0).stop 2023-10-30T14:17:40.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.093+0000 7f0df2dd3640 1 -- 172.21.15.57:0/783175842 shutdown_connections 2023-10-30T14:17:40.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.093+0000 7f0df2dd3640 1 --2- 172.21.15.57:0/783175842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dec06c420 0x7f0dec0c57a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:40.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.093+0000 7f0df2dd3640 1 -- 172.21.15.57:0/783175842 >> 172.21.15.57:0/783175842 conn(0x7f0dec0cac70 msgr2=0x7f0dec0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:40.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.093+0000 7f0df2dd3640 1 -- 172.21.15.57:0/783175842 shutdown_connections 2023-10-30T14:17:40.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.093+0000 7f0df2dd3640 1 -- 172.21.15.57:0/783175842 wait complete. 2023-10-30T14:17:40.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.093+0000 7f0df2dd3640 1 Processor -- start 2023-10-30T14:17:40.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.093+0000 7f0df2dd3640 1 -- start start 2023-10-30T14:17:40.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0df2dd3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dec06b760 0x7f0dec14c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:40.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0df2dd3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dec06bb40 0x7f0dec0c3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:40.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0df2dd3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dec06c420 0x7f0dec15f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:40.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0df2dd3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dec1f6660 con 0x7f0dec06c420 2023-10-30T14:17:40.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0df2dd3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dec15ed40 con 0x7f0dec06b760 2023-10-30T14:17:40.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0df2dd3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dec1f6b70 con 0x7f0dec06bb40 2023-10-30T14:17:40.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0df1370640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dec06c420 0x7f0dec15f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:40.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0df1370640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dec06c420 0x7f0dec15f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59638/0 (socket says 172.21.15.57:59638) 2023-10-30T14:17:40.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0df1370640 1 -- 172.21.15.57:0/4058421811 learned_addr learned my addr 172.21.15.57:0/4058421811 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:40.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0debfff640 1 --2- 172.21.15.57:0/4058421811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dec06bb40 0x7f0dec0c3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:40.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0df0b6f640 1 --2- 172.21.15.57:0/4058421811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dec06b760 0x7f0dec14c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:40.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0df1370640 1 -- 172.21.15.57:0/4058421811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dec06bb40 msgr2=0x7f0dec0c3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:40.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0df1370640 1 --2- 172.21.15.57:0/4058421811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dec06bb40 0x7f0dec0c3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:40.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0df1370640 1 -- 172.21.15.57:0/4058421811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dec06b760 msgr2=0x7f0dec14c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:40.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0df1370640 1 --2- 172.21.15.57:0/4058421811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dec06b760 0x7f0dec14c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:40.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0df1370640 1 -- 172.21.15.57:0/4058421811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de4009a70 con 0x7f0dec06c420 2023-10-30T14:17:40.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0df1370640 1 --2- 172.21.15.57:0/4058421811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dec06c420 0x7f0dec15f490 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0de4019550 tx=0x7f0de40196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:40.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0de97fa640 1 -- 172.21.15.57:0/4058421811 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de40aed10 con 0x7f0dec06c420 2023-10-30T14:17:40.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0de97fa640 1 -- 172.21.15.57:0/4058421811 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de40aee70 con 0x7f0dec06c420 2023-10-30T14:17:40.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0de97fa640 1 -- 172.21.15.57:0/4058421811 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de40b7630 con 0x7f0dec06c420 2023-10-30T14:17:40.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0df2dd3640 1 -- 172.21.15.57:0/4058421811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dec1f6890 con 0x7f0dec06c420 2023-10-30T14:17:40.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0df2dd3640 1 -- 172.21.15.57:0/4058421811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dec161e20 con 0x7f0dec06c420 2023-10-30T14:17:40.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0df2dd3640 1 -- 172.21.15.57:0/4058421811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dec04e170 con 0x7f0dec06c420 2023-10-30T14:17:40.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.097+0000 7f0de97fa640 1 -- 172.21.15.57:0/4058421811 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0de40b4050 con 0x7f0dec06c420 2023-10-30T14:17:40.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.101+0000 7f0de97fa640 1 --2- 172.21.15.57:0/4058421811 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0dd00664a0 0x7f0dd0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:40.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.101+0000 7f0de97fa640 1 -- 172.21.15.57:0/4058421811 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7f0de411e9c0 con 0x7f0dec06c420 2023-10-30T14:17:40.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.101+0000 7f0df0b6f640 1 --2- 172.21.15.57:0/4058421811 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0dd00664a0 0x7f0dd0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:40.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.101+0000 7f0df0b6f640 1 --2- 172.21.15.57:0/4058421811 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0dd00664a0 0x7f0dd0068960 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0de00011f0 tx=0x7f0de0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:40.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.105+0000 7f0de97fa640 1 -- 172.21.15.57:0/4058421811 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0de40ebe40 con 0x7f0dec06c420 2023-10-30T14:17:40.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.293+0000 7f0df2dd3640 1 -- 172.21.15.57:0/4058421811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0dec066510 con 0x7f0dec06c420 2023-10-30T14:17:40.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.293+0000 7f0de97fa640 1 -- 172.21.15.57:0/4058421811 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6528 (secure 0 0 0) 0x7f0de40b2090 con 0x7f0dec06c420 2023-10-30T14:17:40.366 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:40.366 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5133,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:26.649359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:40.369 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 28 2023-10-30T14:17:40.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.297+0000 7f0df2dd3640 1 -- 172.21.15.57:0/4058421811 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0dd00664a0 msgr2=0x7f0dd0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:40.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.297+0000 7f0df2dd3640 1 --2- 172.21.15.57:0/4058421811 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0dd00664a0 0x7f0dd0068960 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0de00011f0 tx=0x7f0de0009040 comp rx=0 tx=0).stop 2023-10-30T14:17:40.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.297+0000 7f0df2dd3640 1 -- 172.21.15.57:0/4058421811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dec06c420 msgr2=0x7f0dec15f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:40.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.297+0000 7f0df2dd3640 1 --2- 172.21.15.57:0/4058421811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dec06c420 0x7f0dec15f490 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0de4019550 tx=0x7f0de40196e0 comp rx=0 tx=0).stop 2023-10-30T14:17:40.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.297+0000 7f0df2dd3640 1 -- 172.21.15.57:0/4058421811 shutdown_connections 2023-10-30T14:17:40.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.297+0000 7f0df2dd3640 1 --2- 172.21.15.57:0/4058421811 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0dd00664a0 0x7f0dd0068960 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:40.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.297+0000 7f0df2dd3640 1 --2- 172.21.15.57:0/4058421811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dec06c420 0x7f0dec15f490 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:40.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.297+0000 7f0df2dd3640 1 --2- 172.21.15.57:0/4058421811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dec06bb40 0x7f0dec0c3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:40.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.297+0000 7f0df2dd3640 1 --2- 172.21.15.57:0/4058421811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dec06b760 0x7f0dec14c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:40.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.297+0000 7f0df2dd3640 1 -- 172.21.15.57:0/4058421811 >> 172.21.15.57:0/4058421811 conn(0x7f0dec0cac70 msgr2=0x7f0dec152df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:40.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.297+0000 7f0df2dd3640 1 -- 172.21.15.57:0/4058421811 shutdown_connections 2023-10-30T14:17:40.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:40.297+0000 7f0df2dd3640 1 -- 172.21.15.57:0/4058421811 wait complete. 2023-10-30T14:17:40.372 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:17:42.373 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:42.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.521+0000 7fc7694d6640 1 Processor -- start 2023-10-30T14:17:42.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.521+0000 7fc7694d6640 1 -- start start 2023-10-30T14:17:42.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.521+0000 7fc7694d6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc764150830 0x7fc76406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:42.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.521+0000 7fc7694d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7640cb7d0 0x7fc7640c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:42.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.521+0000 7fc7694d6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7640c2310 con 0x7fc7640c3e40 2023-10-30T14:17:42.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.521+0000 7fc7694d6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc764156710 con 0x7fc764150830 2023-10-30T14:17:42.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.521+0000 7fc7694d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7640c4360 con 0x7fc7640cb7d0 2023-10-30T14:17:42.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.521+0000 7fc762ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc764150830 0x7fc76406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:42.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.521+0000 7fc7637fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc7640c3e40 0x7fc764070520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:39236/0 (socket says 172.21.15.57:39236) 2023-10-30T14:17:42.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.521+0000 7fc762ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc764150830 0x7fc76406b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59642/0 (socket says 172.21.15.57:59642) 2023-10-30T14:17:42.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.521+0000 7fc7637fe640 1 -- 172.21.15.57:0/3682181880 learned_addr learned my addr 172.21.15.57:0/3682181880 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:17:42.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.521+0000 7fc7627fc640 1 --2- 172.21.15.57:0/3682181880 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7640cb7d0 0x7fc7640c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:42.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.521+0000 7fc762ffd640 1 -- 172.21.15.57:0/3682181880 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7640cb7d0 msgr2=0x7fc7640c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:42.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.521+0000 7fc762ffd640 1 --2- 172.21.15.57:0/3682181880 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7640cb7d0 0x7fc7640c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:42.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.521+0000 7fc762ffd640 1 -- 172.21.15.57:0/3682181880 >> v1:172.21.15.57:6789/0 conn(0x7fc7640c3e40 legacy=0x7fc764070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:42.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.521+0000 7fc762ffd640 1 -- 172.21.15.57:0/3682181880 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7641569b0 con 0x7fc764150830 2023-10-30T14:17:42.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.521+0000 7fc761ffb640 1 -- 172.21.15.57:0/3682181880 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536655106 0 0) 0x7fc7640c2310 con 0x7fc7640c3e40 2023-10-30T14:17:42.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.525+0000 7fc762ffd640 1 --2- 172.21.15.57:0/3682181880 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc764150830 0x7fc76406b500 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc7540097c0 tx=0x7fc75401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2cf6f7cf6068973 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:42.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.525+0000 7fc761ffb640 1 -- 172.21.15.57:0/3682181880 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7540a8970 con 0x7fc764150830 2023-10-30T14:17:42.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.525+0000 7fc761ffb640 1 -- 172.21.15.57:0/3682181880 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7540a8ad0 con 0x7fc764150830 2023-10-30T14:17:42.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.525+0000 7fc761ffb640 1 -- 172.21.15.57:0/3682181880 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7540a8e60 con 0x7fc764150830 2023-10-30T14:17:42.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.525+0000 7fc7694d6640 1 -- 172.21.15.57:0/3682181880 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc764150830 msgr2=0x7fc76406b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:42.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.525+0000 7fc7694d6640 1 --2- 172.21.15.57:0/3682181880 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc764150830 0x7fc76406b500 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc7540097c0 tx=0x7fc75401ec70 comp rx=0 tx=0).stop 2023-10-30T14:17:42.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.525+0000 7fc7694d6640 1 -- 172.21.15.57:0/3682181880 shutdown_connections 2023-10-30T14:17:42.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.525+0000 7fc7694d6640 1 --2- 172.21.15.57:0/3682181880 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7640cb7d0 0x7fc7640c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:42.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.525+0000 7fc7694d6640 1 --2- 172.21.15.57:0/3682181880 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc764150830 0x7fc76406b500 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:42.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.525+0000 7fc7694d6640 1 -- 172.21.15.57:0/3682181880 >> 172.21.15.57:0/3682181880 conn(0x7fc7640cac70 msgr2=0x7fc7640cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:42.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.525+0000 7fc7694d6640 1 -- 172.21.15.57:0/3682181880 shutdown_connections 2023-10-30T14:17:42.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.525+0000 7fc7694d6640 1 -- 172.21.15.57:0/3682181880 wait complete. 2023-10-30T14:17:42.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc7694d6640 1 Processor -- start 2023-10-30T14:17:42.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc7694d6640 1 -- start start 2023-10-30T14:17:42.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc7694d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7640c3e40 0x7fc764155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:42.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc7694d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7640cb7d0 0x7fc7640c56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:42.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc7694d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc764150830 0x7fc764142a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:42.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc7694d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc764155d80 con 0x7fc7640c3e40 2023-10-30T14:17:42.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc7694d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7641425c0 con 0x7fc764150830 2023-10-30T14:17:42.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc7694d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc764155540 con 0x7fc7640cb7d0 2023-10-30T14:17:42.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc762ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7640c3e40 0x7fc764155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:42.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc762ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7640c3e40 0x7fc764155840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59644/0 (socket says 172.21.15.57:59644) 2023-10-30T14:17:42.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc762ffd640 1 -- 172.21.15.57:0/3711770923 learned_addr learned my addr 172.21.15.57:0/3711770923 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:42.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc7627fc640 1 --2- 172.21.15.57:0/3711770923 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7640cb7d0 0x7fc7640c56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:42.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc7637fe640 1 --2- 172.21.15.57:0/3711770923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc764150830 0x7fc764142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:42.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc762ffd640 1 -- 172.21.15.57:0/3711770923 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7640cb7d0 msgr2=0x7fc7640c56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:42.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc762ffd640 1 --2- 172.21.15.57:0/3711770923 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7640cb7d0 0x7fc7640c56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:42.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc762ffd640 1 -- 172.21.15.57:0/3711770923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc764150830 msgr2=0x7fc764142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:42.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc762ffd640 1 --2- 172.21.15.57:0/3711770923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc764150830 0x7fc764142a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:42.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc762ffd640 1 -- 172.21.15.57:0/3711770923 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc754009420 con 0x7fc7640c3e40 2023-10-30T14:17:42.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc762ffd640 1 --2- 172.21.15.57:0/3711770923 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7640c3e40 0x7fc764155840 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc75401c8a0 tx=0x7fc75401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:42.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc74b7fe640 1 -- 172.21.15.57:0/3711770923 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7540a8970 con 0x7fc7640c3e40 2023-10-30T14:17:42.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc74b7fe640 1 -- 172.21.15.57:0/3711770923 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7540b9410 con 0x7fc7640c3e40 2023-10-30T14:17:42.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc74b7fe640 1 -- 172.21.15.57:0/3711770923 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7540b97a0 con 0x7fc7640c3e40 2023-10-30T14:17:42.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc7694d6640 1 -- 172.21.15.57:0/3711770923 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc764156520 con 0x7fc7640c3e40 2023-10-30T14:17:42.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc7694d6640 1 -- 172.21.15.57:0/3711770923 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc764155f60 con 0x7fc7640c3e40 2023-10-30T14:17:42.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.529+0000 7fc7694d6640 1 -- 172.21.15.57:0/3711770923 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc76404e170 con 0x7fc7640c3e40 2023-10-30T14:17:42.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.533+0000 7fc74b7fe640 1 -- 172.21.15.57:0/3711770923 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc7540b7040 con 0x7fc7640c3e40 2023-10-30T14:17:42.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.533+0000 7fc74b7fe640 1 --2- 172.21.15.57:0/3711770923 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc7500664d0 0x7fc750068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:42.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.533+0000 7fc74b7fe640 1 -- 172.21.15.57:0/3711770923 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5997+0+0 (secure 0 0 0) 0x7fc754121b50 con 0x7fc7640c3e40 2023-10-30T14:17:42.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.533+0000 7fc7627fc640 1 --2- 172.21.15.57:0/3711770923 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc7500664d0 0x7fc750068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:42.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.533+0000 7fc7627fc640 1 --2- 172.21.15.57:0/3711770923 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc7500664d0 0x7fc750068990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc7641576f0 tx=0x7fc758008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:42.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.537+0000 7fc74b7fe640 1 -- 172.21.15.57:0/3711770923 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc7540eeea0 con 0x7fc7640c3e40 2023-10-30T14:17:42.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.725+0000 7fc7694d6640 1 -- 172.21.15.57:0/3711770923 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc764073a30 con 0x7fc7640c3e40 2023-10-30T14:17:42.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.729+0000 7fc74b7fe640 1 -- 172.21.15.57:0/3711770923 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6528 (secure 0 0 0) 0x7fc7540b5070 con 0x7fc7640c3e40 2023-10-30T14:17:42.732 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:42.732 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5133,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:26.649359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":4266,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4266":{"gid":4266,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6836/2693548867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2693548867},{"type":"v1","addr":"172.21.15.57:6836","nonce":2693548867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:42.733 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 28 2023-10-30T14:17:42.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.729+0000 7fc7694d6640 1 -- 172.21.15.57:0/3711770923 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc7500664d0 msgr2=0x7fc750068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:42.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.729+0000 7fc7694d6640 1 --2- 172.21.15.57:0/3711770923 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc7500664d0 0x7fc750068990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc7641576f0 tx=0x7fc758008040 comp rx=0 tx=0).stop 2023-10-30T14:17:42.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.729+0000 7fc7694d6640 1 -- 172.21.15.57:0/3711770923 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7640c3e40 msgr2=0x7fc764155840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:42.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.729+0000 7fc7694d6640 1 --2- 172.21.15.57:0/3711770923 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7640c3e40 0x7fc764155840 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc75401c8a0 tx=0x7fc75401ba10 comp rx=0 tx=0).stop 2023-10-30T14:17:42.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.729+0000 7fc7694d6640 1 -- 172.21.15.57:0/3711770923 shutdown_connections 2023-10-30T14:17:42.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.729+0000 7fc7694d6640 1 --2- 172.21.15.57:0/3711770923 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc7500664d0 0x7fc750068990 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:42.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.729+0000 7fc7694d6640 1 --2- 172.21.15.57:0/3711770923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc764150830 0x7fc764142a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:42.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.729+0000 7fc7694d6640 1 --2- 172.21.15.57:0/3711770923 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7640cb7d0 0x7fc7640c56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:42.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.729+0000 7fc7694d6640 1 --2- 172.21.15.57:0/3711770923 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7640c3e40 0x7fc764155840 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:42.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.729+0000 7fc7694d6640 1 -- 172.21.15.57:0/3711770923 >> 172.21.15.57:0/3711770923 conn(0x7fc7640cac70 msgr2=0x7fc76414d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:42.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.729+0000 7fc7694d6640 1 -- 172.21.15.57:0/3711770923 shutdown_connections 2023-10-30T14:17:42.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:42.729+0000 7fc7694d6640 1 -- 172.21.15.57:0/3711770923 wait complete. 2023-10-30T14:17:42.753 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:17:44.655 INFO:tasks.workunit.client.0.smithi057.stdout: create 294/30 file after the snapshot 2023-10-30T14:17:44.656 INFO:tasks.workunit.client.0.smithi057.stdout: create 294/31 file after the snapshot 2023-10-30T14:17:44.657 INFO:tasks.workunit.client.0.smithi057.stdout: create 294/32 file after the snapshot 2023-10-30T14:17:44.658 INFO:tasks.workunit.client.0.smithi057.stdout: create 294/33 file after the snapshot 2023-10-30T14:17:44.659 INFO:tasks.workunit.client.0.smithi057.stdout: create 294/34 file after the snapshot 2023-10-30T14:17:44.660 INFO:tasks.workunit.client.0.smithi057.stdout: create 294/35 file after the snapshot 2023-10-30T14:17:44.661 INFO:tasks.workunit.client.0.smithi057.stdout: create 294/36 file after the snapshot 2023-10-30T14:17:44.662 INFO:tasks.workunit.client.0.smithi057.stdout: create 294/37 file after the snapshot 2023-10-30T14:17:44.663 INFO:tasks.workunit.client.0.smithi057.stdout: create 294/38 file after the snapshot 2023-10-30T14:17:44.664 INFO:tasks.workunit.client.0.smithi057.stdout: create 294/39 file after the snapshot 2023-10-30T14:17:44.665 INFO:tasks.workunit.client.0.smithi057.stdout: create 295/.snap/snap-subdir-test 2023-10-30T14:17:44.694 INFO:tasks.workunit.client.0.smithi057.stdout: create 295/30 file after the snapshot 2023-10-30T14:17:44.696 INFO:tasks.workunit.client.0.smithi057.stdout: create 295/31 file after the snapshot 2023-10-30T14:17:44.697 INFO:tasks.workunit.client.0.smithi057.stdout: create 295/32 file after the snapshot 2023-10-30T14:17:44.697 INFO:tasks.workunit.client.0.smithi057.stdout: create 295/33 file after the snapshot 2023-10-30T14:17:44.698 INFO:tasks.workunit.client.0.smithi057.stdout: create 295/34 file after the snapshot 2023-10-30T14:17:44.699 INFO:tasks.workunit.client.0.smithi057.stdout: create 295/35 file after the snapshot 2023-10-30T14:17:44.700 INFO:tasks.workunit.client.0.smithi057.stdout: create 295/36 file after the snapshot 2023-10-30T14:17:44.700 INFO:tasks.workunit.client.0.smithi057.stdout: create 295/37 file after the snapshot 2023-10-30T14:17:44.701 INFO:tasks.workunit.client.0.smithi057.stdout: create 295/38 file after the snapshot 2023-10-30T14:17:44.702 INFO:tasks.workunit.client.0.smithi057.stdout: create 295/39 file after the snapshot 2023-10-30T14:17:44.702 INFO:tasks.workunit.client.0.smithi057.stdout: create 296/.snap/snap-subdir-test 2023-10-30T14:17:44.722 INFO:tasks.workunit.client.0.smithi057.stdout: create 296/30 file after the snapshot 2023-10-30T14:17:44.724 INFO:tasks.workunit.client.0.smithi057.stdout: create 296/31 file after the snapshot 2023-10-30T14:17:44.726 INFO:tasks.workunit.client.0.smithi057.stdout: create 296/32 file after the snapshot 2023-10-30T14:17:44.728 INFO:tasks.workunit.client.0.smithi057.stdout: create 296/33 file after the snapshot 2023-10-30T14:17:44.730 INFO:tasks.workunit.client.0.smithi057.stdout: create 296/34 file after the snapshot 2023-10-30T14:17:44.732 INFO:tasks.workunit.client.0.smithi057.stdout: create 296/35 file after the snapshot 2023-10-30T14:17:44.733 INFO:tasks.workunit.client.0.smithi057.stdout: create 296/36 file after the snapshot 2023-10-30T14:17:44.735 INFO:tasks.workunit.client.0.smithi057.stdout: create 296/37 file after the snapshot 2023-10-30T14:17:44.737 INFO:tasks.workunit.client.0.smithi057.stdout: create 296/38 file after the snapshot 2023-10-30T14:17:44.739 INFO:tasks.workunit.client.0.smithi057.stdout: create 296/39 file after the snapshot 2023-10-30T14:17:44.741 INFO:tasks.workunit.client.0.smithi057.stdout: create 297/.snap/snap-subdir-test 2023-10-30T14:17:44.754 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:44.769 INFO:tasks.workunit.client.0.smithi057.stdout: create 297/30 file after the snapshot 2023-10-30T14:17:44.772 INFO:tasks.workunit.client.0.smithi057.stdout: create 297/31 file after the snapshot 2023-10-30T14:17:44.772 INFO:tasks.workunit.client.0.smithi057.stdout: create 297/32 file after the snapshot 2023-10-30T14:17:44.773 INFO:tasks.workunit.client.0.smithi057.stdout: create 297/33 file after the snapshot 2023-10-30T14:17:44.774 INFO:tasks.workunit.client.0.smithi057.stdout: create 297/34 file after the snapshot 2023-10-30T14:17:44.775 INFO:tasks.workunit.client.0.smithi057.stdout: create 297/35 file after the snapshot 2023-10-30T14:17:44.776 INFO:tasks.workunit.client.0.smithi057.stdout: create 297/36 file after the snapshot 2023-10-30T14:17:44.776 INFO:tasks.workunit.client.0.smithi057.stdout: create 297/37 file after the snapshot 2023-10-30T14:17:44.778 INFO:tasks.workunit.client.0.smithi057.stdout: create 297/38 file after the snapshot 2023-10-30T14:17:44.779 INFO:tasks.workunit.client.0.smithi057.stdout: create 297/39 file after the snapshot 2023-10-30T14:17:44.780 INFO:tasks.workunit.client.0.smithi057.stdout: create 298/.snap/snap-subdir-test 2023-10-30T14:17:44.808 INFO:tasks.workunit.client.0.smithi057.stdout: create 298/30 file after the snapshot 2023-10-30T14:17:44.810 INFO:tasks.workunit.client.0.smithi057.stdout: create 298/31 file after the snapshot 2023-10-30T14:17:44.810 INFO:tasks.workunit.client.0.smithi057.stdout: create 298/32 file after the snapshot 2023-10-30T14:17:44.811 INFO:tasks.workunit.client.0.smithi057.stdout: create 298/33 file after the snapshot 2023-10-30T14:17:44.812 INFO:tasks.workunit.client.0.smithi057.stdout: create 298/34 file after the snapshot 2023-10-30T14:17:44.813 INFO:tasks.workunit.client.0.smithi057.stdout: create 298/35 file after the snapshot 2023-10-30T14:17:44.813 INFO:tasks.workunit.client.0.smithi057.stdout: create 298/36 file after the snapshot 2023-10-30T14:17:44.814 INFO:tasks.workunit.client.0.smithi057.stdout: create 298/37 file after the snapshot 2023-10-30T14:17:44.815 INFO:tasks.workunit.client.0.smithi057.stdout: create 298/38 file after the snapshot 2023-10-30T14:17:44.816 INFO:tasks.workunit.client.0.smithi057.stdout: create 298/39 file after the snapshot 2023-10-30T14:17:44.816 INFO:tasks.workunit.client.0.smithi057.stdout: create 299/.snap/snap-subdir-test 2023-10-30T14:17:44.836 INFO:tasks.workunit.client.0.smithi057.stdout: create 299/30 file after the snapshot 2023-10-30T14:17:44.839 INFO:tasks.workunit.client.0.smithi057.stdout: create 299/31 file after the snapshot 2023-10-30T14:17:44.840 INFO:tasks.workunit.client.0.smithi057.stdout: create 299/32 file after the snapshot 2023-10-30T14:17:44.842 INFO:tasks.workunit.client.0.smithi057.stdout: create 299/33 file after the snapshot 2023-10-30T14:17:44.844 INFO:tasks.workunit.client.0.smithi057.stdout: create 299/34 file after the snapshot 2023-10-30T14:17:44.846 INFO:tasks.workunit.client.0.smithi057.stdout: create 299/35 file after the snapshot 2023-10-30T14:17:44.847 INFO:tasks.workunit.client.0.smithi057.stdout: create 299/36 file after the snapshot 2023-10-30T14:17:44.849 INFO:tasks.workunit.client.0.smithi057.stdout: create 299/37 file after the snapshot 2023-10-30T14:17:44.851 INFO:tasks.workunit.client.0.smithi057.stdout: create 299/38 file after the snapshot 2023-10-30T14:17:44.852 INFO:tasks.workunit.client.0.smithi057.stdout: create 299/39 file after the snapshot 2023-10-30T14:17:44.854 INFO:tasks.workunit.client.0.smithi057.stdout:Take second snapshot .snap/test2 2023-10-30T14:17:44.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.905+0000 7f185653e640 1 Processor -- start 2023-10-30T14:17:44.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.905+0000 7f185653e640 1 -- start start 2023-10-30T14:17:44.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.905+0000 7f185653e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f18500c4a10 0x7f185015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:44.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.905+0000 7f185653e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f18500c42a0 0x7f18500ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:44.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.905+0000 7f185653e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1850159f20 con 0x7f1850153c80 2023-10-30T14:17:44.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.905+0000 7f185653e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f185015dec0 con 0x7f18500c4a10 2023-10-30T14:17:44.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.905+0000 7f185653e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18501599a0 con 0x7f18500c42a0 2023-10-30T14:17:44.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.905+0000 7f184f7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f18500c4a10 0x7f185015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:44.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.905+0000 7f184f7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f18500c4a10 0x7f185015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41740/0 (socket says 172.21.15.57:41740) 2023-10-30T14:17:44.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.905+0000 7f184f7fe640 1 -- 172.21.15.57:0/2027296336 learned_addr learned my addr 172.21.15.57:0/2027296336 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:44.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.905+0000 7f1854adb640 1 --2- 172.21.15.57:0/2027296336 >> v2:172.21.15.88:3300/0 conn(0x7f18500c42a0 0x7f18500ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:44.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.905+0000 7f184f7fe640 1 -- 172.21.15.57:0/2027296336 >> v2:172.21.15.88:3300/0 conn(0x7f18500c42a0 msgr2=0x7f18500ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:44.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.905+0000 7f184f7fe640 1 --2- 172.21.15.57:0/2027296336 >> v2:172.21.15.88:3300/0 conn(0x7f18500c42a0 0x7f18500ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:44.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.905+0000 7f184f7fe640 1 -- 172.21.15.57:0/2027296336 >> v1:172.21.15.88:6789/0 conn(0x7f1850153c80 legacy=0x7f185015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:44.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.905+0000 7f184f7fe640 1 -- 172.21.15.57:0/2027296336 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185015d620 con 0x7f18500c4a10 2023-10-30T14:17:44.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.909+0000 7f184f7fe640 1 --2- 172.21.15.57:0/2027296336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18500c4a10 0x7f185015cf90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f18500b1d40 tx=0x7f184401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b960d0aa20c65c21 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:44.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.909+0000 7f184effd640 1 -- 172.21.15.57:0/2027296336 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18440a89f0 con 0x7f18500c4a10 2023-10-30T14:17:44.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.909+0000 7f184effd640 1 -- 172.21.15.57:0/2027296336 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18440a8b50 con 0x7f18500c4a10 2023-10-30T14:17:44.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.909+0000 7f184effd640 1 -- 172.21.15.57:0/2027296336 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18440b0440 con 0x7f18500c4a10 2023-10-30T14:17:44.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.909+0000 7f185653e640 1 -- 172.21.15.57:0/2027296336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18500c4a10 msgr2=0x7f185015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:44.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.909+0000 7f185653e640 1 --2- 172.21.15.57:0/2027296336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18500c4a10 0x7f185015cf90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f18500b1d40 tx=0x7f184401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:17:44.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.909+0000 7f185653e640 1 -- 172.21.15.57:0/2027296336 shutdown_connections 2023-10-30T14:17:44.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.909+0000 7f185653e640 1 --2- 172.21.15.57:0/2027296336 >> v2:172.21.15.88:3300/0 conn(0x7f18500c42a0 0x7f18500ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:44.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.909+0000 7f185653e640 1 --2- 172.21.15.57:0/2027296336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18500c4a10 0x7f185015cf90 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:44.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.909+0000 7f185653e640 1 -- 172.21.15.57:0/2027296336 >> 172.21.15.57:0/2027296336 conn(0x7f18500715c0 msgr2=0x7f18500719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:44.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.909+0000 7f185653e640 1 -- 172.21.15.57:0/2027296336 shutdown_connections 2023-10-30T14:17:44.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.909+0000 7f185653e640 1 -- 172.21.15.57:0/2027296336 wait complete. 2023-10-30T14:17:44.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f185653e640 1 Processor -- start 2023-10-30T14:17:44.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f185653e640 1 -- start start 2023-10-30T14:17:44.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f185653e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18500c42a0 0x7f1850150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:44.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f185653e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18500c4a10 0x7f18500cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:44.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f185653e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1850153c80 0x7f1850149e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:44.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f185653e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1850151940 con 0x7f1850153c80 2023-10-30T14:17:44.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f185653e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185014fd40 con 0x7f18500c42a0 2023-10-30T14:17:44.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f185653e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1850150020 con 0x7f18500c4a10 2023-10-30T14:17:44.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f1854adb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1850153c80 0x7f1850149e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:44.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f1854adb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1850153c80 0x7f1850149e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41742/0 (socket says 172.21.15.57:41742) 2023-10-30T14:17:44.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f1854adb640 1 -- 172.21.15.57:0/4040005954 learned_addr learned my addr 172.21.15.57:0/4040005954 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:44.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f184f7fe640 1 --2- 172.21.15.57:0/4040005954 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18500c4a10 0x7f18500cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:44.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f184ffff640 1 --2- 172.21.15.57:0/4040005954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18500c42a0 0x7f1850150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:44.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f1854adb640 1 -- 172.21.15.57:0/4040005954 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18500c4a10 msgr2=0x7f18500cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:44.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f1854adb640 1 --2- 172.21.15.57:0/4040005954 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18500c4a10 0x7f18500cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:44.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f1854adb640 1 -- 172.21.15.57:0/4040005954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18500c42a0 msgr2=0x7f1850150ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:44.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f1854adb640 1 --2- 172.21.15.57:0/4040005954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18500c42a0 0x7f1850150ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:44.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f1854adb640 1 -- 172.21.15.57:0/4040005954 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18440094d0 con 0x7f1850153c80 2023-10-30T14:17:44.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f1854adb640 1 --2- 172.21.15.57:0/4040005954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1850153c80 0x7f1850149e60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f184800efc0 tx=0x7f184800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:44.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f184cff9640 1 -- 172.21.15.57:0/4040005954 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1848014480 con 0x7f1850153c80 2023-10-30T14:17:44.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f184cff9640 1 -- 172.21.15.57:0/4040005954 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184800f040 con 0x7f1850153c80 2023-10-30T14:17:44.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f184cff9640 1 -- 172.21.15.57:0/4040005954 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1848014820 con 0x7f1850153c80 2023-10-30T14:17:44.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f185653e640 1 -- 172.21.15.57:0/4040005954 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18501f1f10 con 0x7f1850153c80 2023-10-30T14:17:44.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.913+0000 7f185653e640 1 -- 172.21.15.57:0/4040005954 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18501f24d0 con 0x7f1850153c80 2023-10-30T14:17:44.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.917+0000 7f185653e640 1 -- 172.21.15.57:0/4040005954 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f185004e170 con 0x7f1850153c80 2023-10-30T14:17:44.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.917+0000 7f184cff9640 1 -- 172.21.15.57:0/4040005954 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1848014980 con 0x7f1850153c80 2023-10-30T14:17:44.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.917+0000 7f184cff9640 1 --2- 172.21.15.57:0/4040005954 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f183c0664a0 0x7f183c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:44.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.917+0000 7f184cff9640 1 -- 172.21.15.57:0/4040005954 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f1848084b60 con 0x7f1850153c80 2023-10-30T14:17:44.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.917+0000 7f184ffff640 1 --2- 172.21.15.57:0/4040005954 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f183c0664a0 0x7f183c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:44.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.917+0000 7f184ffff640 1 --2- 172.21.15.57:0/4040005954 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f183c0664a0 0x7f183c068960 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1840007fd0 tx=0x7f1840009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:44.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:44.921+0000 7f184cff9640 1 -- 172.21.15.57:0/4040005954 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1848051de0 con 0x7f1850153c80 2023-10-30T14:17:45.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.109+0000 7f185653e640 1 -- 172.21.15.57:0/4040005954 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18500cad40 con 0x7f1850153c80 2023-10-30T14:17:45.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.109+0000 7f184cff9640 1 -- 172.21.15.57:0/4040005954 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f1848018070 con 0x7f1850153c80 2023-10-30T14:17:45.116 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:45.116 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:44.635131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:45.116 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 29 2023-10-30T14:17:45.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.113+0000 7f185653e640 1 -- 172.21.15.57:0/4040005954 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f183c0664a0 msgr2=0x7f183c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:45.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.113+0000 7f185653e640 1 --2- 172.21.15.57:0/4040005954 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f183c0664a0 0x7f183c068960 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1840007fd0 tx=0x7f1840009040 comp rx=0 tx=0).stop 2023-10-30T14:17:45.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.113+0000 7f185653e640 1 -- 172.21.15.57:0/4040005954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1850153c80 msgr2=0x7f1850149e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:45.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.113+0000 7f185653e640 1 --2- 172.21.15.57:0/4040005954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1850153c80 0x7f1850149e60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f184800efc0 tx=0x7f184800c490 comp rx=0 tx=0).stop 2023-10-30T14:17:45.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.113+0000 7f185653e640 1 -- 172.21.15.57:0/4040005954 shutdown_connections 2023-10-30T14:17:45.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.113+0000 7f185653e640 1 --2- 172.21.15.57:0/4040005954 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f183c0664a0 0x7f183c068960 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:45.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.113+0000 7f185653e640 1 --2- 172.21.15.57:0/4040005954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1850153c80 0x7f1850149e60 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:45.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.113+0000 7f185653e640 1 --2- 172.21.15.57:0/4040005954 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18500c4a10 0x7f18500cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:45.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.113+0000 7f185653e640 1 --2- 172.21.15.57:0/4040005954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18500c42a0 0x7f1850150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:45.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.113+0000 7f185653e640 1 -- 172.21.15.57:0/4040005954 >> 172.21.15.57:0/4040005954 conn(0x7f18500715c0 msgr2=0x7f18500c4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:45.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.113+0000 7f185653e640 1 -- 172.21.15.57:0/4040005954 shutdown_connections 2023-10-30T14:17:45.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.113+0000 7f185653e640 1 -- 172.21.15.57:0/4040005954 wait complete. 2023-10-30T14:17:45.137 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T14:17:45.137 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:17:45.138 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:45.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd8ca6b640 1 Processor -- start 2023-10-30T14:17:45.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd8ca6b640 1 -- start start 2023-10-30T14:17:45.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd8ca6b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcd88153c80 0x7fcd8815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:45.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd8ca6b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd880c4a10 0x7fcd880c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:45.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd8ca6b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcd8815a3b0 0x7fcd880ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:45.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd8ca6b640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd8815dec0 con 0x7fcd8815a3b0 2023-10-30T14:17:45.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd8ca6b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd881599a0 con 0x7fcd88153c80 2023-10-30T14:17:45.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd8ca6b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd88159f20 con 0x7fcd880c4a10 2023-10-30T14:17:45.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd86d9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcd8815a3b0 0x7fcd880ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:45.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd86d9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcd8815a3b0 0x7fcd880ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41756/0 (socket says 172.21.15.57:41756) 2023-10-30T14:17:45.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd86d9d640 1 -- 172.21.15.57:0/2491112917 learned_addr learned my addr 172.21.15.57:0/2491112917 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:45.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd85d9b640 1 --2- 172.21.15.57:0/2491112917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd880c4a10 0x7fcd880c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:45.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd8659c640 1 --2- 172.21.15.57:0/2491112917 >> v2:172.21.15.88:3300/0 conn(0x7fcd88153c80 0x7fcd8815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:45.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd86d9d640 1 -- 172.21.15.57:0/2491112917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd880c4a10 msgr2=0x7fcd880c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:45.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd86d9d640 1 --2- 172.21.15.57:0/2491112917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd880c4a10 0x7fcd880c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:45.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd86d9d640 1 -- 172.21.15.57:0/2491112917 >> v2:172.21.15.88:3300/0 conn(0x7fcd88153c80 msgr2=0x7fcd8815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:45.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd86d9d640 1 --2- 172.21.15.57:0/2491112917 >> v2:172.21.15.88:3300/0 conn(0x7fcd88153c80 0x7fcd8815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:45.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd86d9d640 1 -- 172.21.15.57:0/2491112917 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd880c47e0 con 0x7fcd8815a3b0 2023-10-30T14:17:45.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd8659c640 1 --2- 172.21.15.57:0/2491112917 >> v2:172.21.15.88:3300/0 conn(0x7fcd88153c80 0x7fcd8815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:17:45.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.321+0000 7fcd85d9b640 1 --2- 172.21.15.57:0/2491112917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd880c4a10 0x7fcd880c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:17:45.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.325+0000 7fcd86d9d640 1 --2- 172.21.15.57:0/2491112917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd8815a3b0 0x7fcd880ca1e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fcd880d00f0 tx=0x7fcd8001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bbcb4a454fe2c38 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:45.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.325+0000 7fcd8559a640 1 -- 172.21.15.57:0/2491112917 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd800af400 con 0x7fcd8815a3b0 2023-10-30T14:17:45.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.325+0000 7fcd8559a640 1 -- 172.21.15.57:0/2491112917 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd800ab050 con 0x7fcd8815a3b0 2023-10-30T14:17:45.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.325+0000 7fcd8559a640 1 -- 172.21.15.57:0/2491112917 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd800af780 con 0x7fcd8815a3b0 2023-10-30T14:17:45.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.325+0000 7fcd8ca6b640 1 -- 172.21.15.57:0/2491112917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd8815a3b0 msgr2=0x7fcd880ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:45.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.325+0000 7fcd8ca6b640 1 --2- 172.21.15.57:0/2491112917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd8815a3b0 0x7fcd880ca1e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fcd880d00f0 tx=0x7fcd8001f1c0 comp rx=0 tx=0).stop 2023-10-30T14:17:45.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.325+0000 7fcd8ca6b640 1 -- 172.21.15.57:0/2491112917 shutdown_connections 2023-10-30T14:17:45.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.325+0000 7fcd8ca6b640 1 --2- 172.21.15.57:0/2491112917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd880c4a10 0x7fcd880c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:45.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.325+0000 7fcd8ca6b640 1 --2- 172.21.15.57:0/2491112917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd8815a3b0 0x7fcd880ca1e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:45.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.325+0000 7fcd8ca6b640 1 --2- 172.21.15.57:0/2491112917 >> v2:172.21.15.88:3300/0 conn(0x7fcd88153c80 0x7fcd8815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:45.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.325+0000 7fcd8ca6b640 1 -- 172.21.15.57:0/2491112917 >> 172.21.15.57:0/2491112917 conn(0x7fcd880715c0 msgr2=0x7fcd880719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:45.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.325+0000 7fcd8ca6b640 1 -- 172.21.15.57:0/2491112917 shutdown_connections 2023-10-30T14:17:45.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.325+0000 7fcd8ca6b640 1 -- 172.21.15.57:0/2491112917 wait complete. 2023-10-30T14:17:45.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd8ca6b640 1 Processor -- start 2023-10-30T14:17:45.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd8ca6b640 1 -- start start 2023-10-30T14:17:45.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd8ca6b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd880c4a10 0x7fcd88155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:45.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd8ca6b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd88153c80 0x7fcd881ef120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:45.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd8ca6b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd8815a3b0 0x7fcd881ef660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:45.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd8ca6b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd881501e0 con 0x7fcd88153c80 2023-10-30T14:17:45.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd8ca6b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd881f32c0 con 0x7fcd8815a3b0 2023-10-30T14:17:45.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd8ca6b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd88156410 con 0x7fcd880c4a10 2023-10-30T14:17:45.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd85d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd88153c80 0x7fcd881ef120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:45.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd85d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd88153c80 0x7fcd881ef120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41770/0 (socket says 172.21.15.57:41770) 2023-10-30T14:17:45.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd85d9b640 1 -- 172.21.15.57:0/2162898508 learned_addr learned my addr 172.21.15.57:0/2162898508 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:45.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd8659c640 1 --2- 172.21.15.57:0/2162898508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd880c4a10 0x7fcd88155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:45.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd86d9d640 1 --2- 172.21.15.57:0/2162898508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd8815a3b0 0x7fcd881ef660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:45.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd85d9b640 1 -- 172.21.15.57:0/2162898508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd880c4a10 msgr2=0x7fcd88155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:45.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd85d9b640 1 --2- 172.21.15.57:0/2162898508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd880c4a10 0x7fcd88155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:45.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd85d9b640 1 -- 172.21.15.57:0/2162898508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd8815a3b0 msgr2=0x7fcd881ef660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:45.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd85d9b640 1 --2- 172.21.15.57:0/2162898508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd8815a3b0 0x7fcd881ef660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:45.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd85d9b640 1 -- 172.21.15.57:0/2162898508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd80009b30 con 0x7fcd88153c80 2023-10-30T14:17:45.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd85d9b640 1 --2- 172.21.15.57:0/2162898508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd88153c80 0x7fcd881ef120 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c00e7e0 tx=0x7fcd7c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:45.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd76ffd640 1 -- 172.21.15.57:0/2162898508 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c004030 con 0x7fcd88153c80 2023-10-30T14:17:45.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd76ffd640 1 -- 172.21.15.57:0/2162898508 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7c004190 con 0x7fcd88153c80 2023-10-30T14:17:45.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd8ca6b640 1 -- 172.21.15.57:0/2162898508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd881566f0 con 0x7fcd88153c80 2023-10-30T14:17:45.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd76ffd640 1 -- 172.21.15.57:0/2162898508 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c004500 con 0x7fcd88153c80 2023-10-30T14:17:45.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.329+0000 7fcd8ca6b640 1 -- 172.21.15.57:0/2162898508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd88155430 con 0x7fcd88153c80 2023-10-30T14:17:45.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.333+0000 7fcd8ca6b640 1 -- 172.21.15.57:0/2162898508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd8814fcd0 con 0x7fcd88153c80 2023-10-30T14:17:45.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.333+0000 7fcd76ffd640 1 -- 172.21.15.57:0/2162898508 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcd7c00c6d0 con 0x7fcd88153c80 2023-10-30T14:17:45.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.333+0000 7fcd76ffd640 1 --2- 172.21.15.57:0/2162898508 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcd680664d0 0x7fcd68068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:45.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.333+0000 7fcd76ffd640 1 -- 172.21.15.57:0/2162898508 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7fcd7c083b60 con 0x7fcd88153c80 2023-10-30T14:17:45.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.333+0000 7fcd8659c640 1 --2- 172.21.15.57:0/2162898508 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcd680664d0 0x7fcd68068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:45.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.333+0000 7fcd8659c640 1 --2- 172.21.15.57:0/2162898508 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcd680664d0 0x7fcd68068990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcd7800c900 tx=0x7fcd78008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:45.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.337+0000 7fcd76ffd640 1 -- 172.21.15.57:0/2162898508 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcd8814fcd0 con 0x7fcd88153c80 2023-10-30T14:17:45.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.525+0000 7fcd8ca6b640 1 -- 172.21.15.57:0/2162898508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd8814fcd0 con 0x7fcd88153c80 2023-10-30T14:17:45.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.525+0000 7fcd76ffd640 1 -- 172.21.15.57:0/2162898508 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5738 (secure 0 0 0) 0x7fcd8814fcd0 con 0x7fcd88153c80 2023-10-30T14:17:45.531 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:45.531 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:44.635131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:45.531 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 29 2023-10-30T14:17:45.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.529+0000 7fcd8ca6b640 1 -- 172.21.15.57:0/2162898508 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcd680664d0 msgr2=0x7fcd68068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:45.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.529+0000 7fcd8ca6b640 1 --2- 172.21.15.57:0/2162898508 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcd680664d0 0x7fcd68068990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcd7800c900 tx=0x7fcd78008040 comp rx=0 tx=0).stop 2023-10-30T14:17:45.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.529+0000 7fcd8ca6b640 1 -- 172.21.15.57:0/2162898508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd88153c80 msgr2=0x7fcd881ef120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:45.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.529+0000 7fcd8ca6b640 1 --2- 172.21.15.57:0/2162898508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd88153c80 0x7fcd881ef120 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c00e7e0 tx=0x7fcd7c00eca0 comp rx=0 tx=0).stop 2023-10-30T14:17:45.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.529+0000 7fcd8ca6b640 1 -- 172.21.15.57:0/2162898508 shutdown_connections 2023-10-30T14:17:45.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.529+0000 7fcd8ca6b640 1 --2- 172.21.15.57:0/2162898508 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcd680664d0 0x7fcd68068990 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:45.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.529+0000 7fcd8ca6b640 1 --2- 172.21.15.57:0/2162898508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd8815a3b0 0x7fcd881ef660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:45.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.529+0000 7fcd8ca6b640 1 --2- 172.21.15.57:0/2162898508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd88153c80 0x7fcd881ef120 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:45.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.529+0000 7fcd8ca6b640 1 --2- 172.21.15.57:0/2162898508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd880c4a10 0x7fcd88155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:45.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.529+0000 7fcd8ca6b640 1 -- 172.21.15.57:0/2162898508 >> 172.21.15.57:0/2162898508 conn(0x7fcd880715c0 msgr2=0x7fcd88158640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:45.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.529+0000 7fcd8ca6b640 1 -- 172.21.15.57:0/2162898508 shutdown_connections 2023-10-30T14:17:45.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:45.529+0000 7fcd8ca6b640 1 -- 172.21.15.57:0/2162898508 wait complete. 2023-10-30T14:17:45.552 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 29, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 29, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4282": { "addr": "172.21.15.88:6839/2945854794", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 2945854794, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2945854794, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4936": { "addr": "172.21.15.57:6837/1988100930", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 1988100930, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 1988100930, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4936, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 31 }, "gid_5133": { "addr": "172.21.15.57:6834/2912050965", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2912050965, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2912050965, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5133, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:17:44.635131+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 5133, "mds_2": 4936 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:17:47.553 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:47.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.701+0000 7f0533479640 1 Processor -- start 2023-10-30T14:17:47.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.705+0000 7f0533479640 1 -- start start 2023-10-30T14:17:47.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.705+0000 7f0533479640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f052c069e90 0x7f052c06f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:47.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.705+0000 7f0533479640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f052c0c8440 con 0x7f052c06a270 2023-10-30T14:17:47.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.705+0000 7f0533479640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f052c0c89f0 con 0x7f052c0c9c40 2023-10-30T14:17:47.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.705+0000 7f0533479640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f052c153f60 con 0x7f052c069e90 2023-10-30T14:17:47.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.705+0000 7f0531a16640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f052c06a270 0x7f052c0c8e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44124/0 (socket says 172.21.15.57:44124) 2023-10-30T14:17:47.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.705+0000 7f0531a16640 1 -- 172.21.15.57:0/1931567997 learned_addr learned my addr 172.21.15.57:0/1931567997 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:17:47.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.705+0000 7f0531215640 1 --2- 172.21.15.57:0/1931567997 >> v2:172.21.15.88:3300/0 conn(0x7f052c069e90 0x7f052c06f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:47.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.705+0000 7f0523fff640 1 -- 172.21.15.57:0/1931567997 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073205510 0 0) 0x7f05240014a0 con 0x7f052c06a270 2023-10-30T14:17:47.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.705+0000 7f0523fff640 1 -- 172.21.15.57:0/1931567997 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0518003530 con 0x7f052c06a270 2023-10-30T14:17:47.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.705+0000 7f0531215640 1 -- 172.21.15.57:0/1931567997 >> v1:172.21.15.88:6789/0 conn(0x7f052c0c9c40 legacy=0x7f052c1597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:47.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.705+0000 7f0531215640 1 -- 172.21.15.57:0/1931567997 >> v1:172.21.15.57:6789/0 conn(0x7f052c06a270 legacy=0x7f052c0c8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:47.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.705+0000 7f0531215640 1 -- 172.21.15.57:0/1931567997 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c0ca290 con 0x7f052c069e90 2023-10-30T14:17:47.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.709+0000 7f0531215640 1 --2- 172.21.15.57:0/1931567997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f052c069e90 0x7f052c06f1d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f05280097c0 tx=0x7f052801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce6f8cba41454001 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:47.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.709+0000 7f0523fff640 1 -- 172.21.15.57:0/1931567997 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05280ab600 con 0x7f052c069e90 2023-10-30T14:17:47.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.709+0000 7f0523fff640 1 -- 172.21.15.57:0/1931567997 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05280a7050 con 0x7f052c069e90 2023-10-30T14:17:47.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.709+0000 7f0523fff640 1 -- 172.21.15.57:0/1931567997 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05280ab980 con 0x7f052c069e90 2023-10-30T14:17:47.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.709+0000 7f0533479640 1 -- 172.21.15.57:0/1931567997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f052c069e90 msgr2=0x7f052c06f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:47.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.709+0000 7f0533479640 1 --2- 172.21.15.57:0/1931567997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f052c069e90 0x7f052c06f1d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f05280097c0 tx=0x7f052801c480 comp rx=0 tx=0).stop 2023-10-30T14:17:47.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.709+0000 7f0533479640 1 -- 172.21.15.57:0/1931567997 shutdown_connections 2023-10-30T14:17:47.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.709+0000 7f0533479640 1 --2- 172.21.15.57:0/1931567997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f052c069e90 0x7f052c06f1d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:47.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.709+0000 7f0533479640 1 -- 172.21.15.57:0/1931567997 >> 172.21.15.57:0/1931567997 conn(0x7f052c06afa0 msgr2=0x7f052c06b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:47.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.709+0000 7f0533479640 1 -- 172.21.15.57:0/1931567997 shutdown_connections 2023-10-30T14:17:47.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.709+0000 7f0533479640 1 -- 172.21.15.57:0/1931567997 wait complete. 2023-10-30T14:17:47.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0533479640 1 Processor -- start 2023-10-30T14:17:47.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0533479640 1 -- start start 2023-10-30T14:17:47.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0533479640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f052c069e90 0x7f052c14c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:47.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0533479640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f052c06a270 0x7f052c152070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:47.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0533479640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f052c0c9c40 0x7f052c1525b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:47.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0533479640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052c1532e0 con 0x7f052c069e90 2023-10-30T14:17:47.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0533479640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052c153580 con 0x7f052c06a270 2023-10-30T14:17:47.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0533479640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f052c153860 con 0x7f052c0c9c40 2023-10-30T14:17:47.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0531215640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f052c069e90 0x7f052c14c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:47.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0531215640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f052c069e90 0x7f052c14c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41782/0 (socket says 172.21.15.57:41782) 2023-10-30T14:17:47.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0531215640 1 -- 172.21.15.57:0/2536539067 learned_addr learned my addr 172.21.15.57:0/2536539067 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:47.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0530a14640 1 --2- 172.21.15.57:0/2536539067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f052c06a270 0x7f052c152070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:47.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0531a16640 1 --2- 172.21.15.57:0/2536539067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f052c0c9c40 0x7f052c1525b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:47.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0531215640 1 -- 172.21.15.57:0/2536539067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f052c0c9c40 msgr2=0x7f052c1525b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:47.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0531215640 1 --2- 172.21.15.57:0/2536539067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f052c0c9c40 0x7f052c1525b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:47.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0531215640 1 -- 172.21.15.57:0/2536539067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f052c06a270 msgr2=0x7f052c152070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:47.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0531215640 1 --2- 172.21.15.57:0/2536539067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f052c06a270 0x7f052c152070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:47.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0531215640 1 -- 172.21.15.57:0/2536539067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0528009420 con 0x7f052c069e90 2023-10-30T14:17:47.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0531215640 1 --2- 172.21.15.57:0/2536539067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f052c069e90 0x7f052c14c370 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0528019090 tx=0x7f0528019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:47.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0521ffb640 1 -- 172.21.15.57:0/2536539067 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05280b4410 con 0x7f052c069e90 2023-10-30T14:17:47.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0521ffb640 1 -- 172.21.15.57:0/2536539067 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05280b4570 con 0x7f052c069e90 2023-10-30T14:17:47.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0521ffb640 1 -- 172.21.15.57:0/2536539067 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05280b4920 con 0x7f052c069e90 2023-10-30T14:17:47.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0533479640 1 -- 172.21.15.57:0/2536539067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f052c14ce80 con 0x7f052c069e90 2023-10-30T14:17:47.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.713+0000 7f0533479640 1 -- 172.21.15.57:0/2536539067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f052c151b00 con 0x7f052c069e90 2023-10-30T14:17:47.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.717+0000 7f0533479640 1 -- 172.21.15.57:0/2536539067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f052c151d50 con 0x7f052c069e90 2023-10-30T14:17:47.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.717+0000 7f0521ffb640 1 -- 172.21.15.57:0/2536539067 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f05280b2040 con 0x7f052c069e90 2023-10-30T14:17:47.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.717+0000 7f0521ffb640 1 --2- 172.21.15.57:0/2536539067 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05100664a0 0x7f0510068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:47.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.717+0000 7f0521ffb640 1 -- 172.21.15.57:0/2536539067 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f052811e0a0 con 0x7f052c069e90 2023-10-30T14:17:47.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.717+0000 7f0530a14640 1 --2- 172.21.15.57:0/2536539067 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05100664a0 0x7f0510068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:47.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.717+0000 7f0530a14640 1 --2- 172.21.15.57:0/2536539067 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05100664a0 0x7f0510068960 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f052c146e60 tx=0x7f051c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:47.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.721+0000 7f0521ffb640 1 -- 172.21.15.57:0/2536539067 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f05280eb390 con 0x7f052c069e90 2023-10-30T14:17:47.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.909+0000 7f0533479640 1 -- 172.21.15.57:0/2536539067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f052c04e170 con 0x7f052c069e90 2023-10-30T14:17:47.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.909+0000 7f0521ffb640 1 -- 172.21.15.57:0/2536539067 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f05280b0070 con 0x7f052c069e90 2023-10-30T14:17:47.915 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:47.916 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:44.635131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:47.916 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 29 2023-10-30T14:17:47.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.913+0000 7f0533479640 1 -- 172.21.15.57:0/2536539067 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05100664a0 msgr2=0x7f0510068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:47.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.913+0000 7f0533479640 1 --2- 172.21.15.57:0/2536539067 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05100664a0 0x7f0510068960 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f052c146e60 tx=0x7f051c009040 comp rx=0 tx=0).stop 2023-10-30T14:17:47.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.913+0000 7f0533479640 1 -- 172.21.15.57:0/2536539067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f052c069e90 msgr2=0x7f052c14c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:47.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.913+0000 7f0533479640 1 --2- 172.21.15.57:0/2536539067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f052c069e90 0x7f052c14c370 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0528019090 tx=0x7f0528019220 comp rx=0 tx=0).stop 2023-10-30T14:17:47.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.913+0000 7f0533479640 1 -- 172.21.15.57:0/2536539067 shutdown_connections 2023-10-30T14:17:47.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.913+0000 7f0533479640 1 --2- 172.21.15.57:0/2536539067 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05100664a0 0x7f0510068960 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:47.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.913+0000 7f0533479640 1 --2- 172.21.15.57:0/2536539067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f052c0c9c40 0x7f052c1525b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:47.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.913+0000 7f0533479640 1 --2- 172.21.15.57:0/2536539067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f052c06a270 0x7f052c152070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:47.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.913+0000 7f0533479640 1 --2- 172.21.15.57:0/2536539067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f052c069e90 0x7f052c14c370 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:47.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.913+0000 7f0533479640 1 -- 172.21.15.57:0/2536539067 >> 172.21.15.57:0/2536539067 conn(0x7f052c06afa0 msgr2=0x7f052c0c4580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:47.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.913+0000 7f0533479640 1 -- 172.21.15.57:0/2536539067 shutdown_connections 2023-10-30T14:17:47.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:47.913+0000 7f0533479640 1 -- 172.21.15.57:0/2536539067 wait complete. 2023-10-30T14:17:47.936 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:17:49.938 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:50.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.085+0000 7f9859417640 1 Processor -- start 2023-10-30T14:17:50.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.085+0000 7f9859417640 1 -- start start 2023-10-30T14:17:50.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.085+0000 7f9859417640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f985414fc50 0x7f985406fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:50.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.085+0000 7f9859417640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98540c55f0 con 0x7f98540c6320 2023-10-30T14:17:50.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.085+0000 7f9859417640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98541567b0 con 0x7f985406be90 2023-10-30T14:17:50.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.085+0000 7f9859417640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98540c5f00 con 0x7f985414fc50 2023-10-30T14:17:50.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.085+0000 7f9852ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f985414fc50 0x7f985406fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:50.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.085+0000 7f984bfff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f98540c6320 0x7f9854070e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44132/0 (socket says 172.21.15.57:44132) 2023-10-30T14:17:50.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.085+0000 7f984bfff640 1 -- 172.21.15.57:0/4147576789 learned_addr learned my addr 172.21.15.57:0/4147576789 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:17:50.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.085+0000 7f9852ffd640 1 -- 172.21.15.57:0/4147576789 >> v1:172.21.15.88:6789/0 conn(0x7f985406be90 legacy=0x7f9854155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:50.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.085+0000 7f9852ffd640 1 -- 172.21.15.57:0/4147576789 >> v1:172.21.15.57:6789/0 conn(0x7f98540c6320 legacy=0x7f9854070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:50.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.085+0000 7f9852ffd640 1 -- 172.21.15.57:0/4147576789 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9854156a70 con 0x7f985414fc50 2023-10-30T14:17:50.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.089+0000 7f9852ffd640 1 --2- 172.21.15.57:0/4147576789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f985414fc50 0x7f985406fdf0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f98440097c0 tx=0x7f984401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3059e65a7add9a8e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:50.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.089+0000 7f98527fc640 1 -- 172.21.15.57:0/4147576789 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98440ab600 con 0x7f985414fc50 2023-10-30T14:17:50.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.089+0000 7f98527fc640 1 -- 172.21.15.57:0/4147576789 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98440a7050 con 0x7f985414fc50 2023-10-30T14:17:50.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.089+0000 7f98527fc640 1 -- 172.21.15.57:0/4147576789 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98440ab980 con 0x7f985414fc50 2023-10-30T14:17:50.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.089+0000 7f9859417640 1 -- 172.21.15.57:0/4147576789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f985414fc50 msgr2=0x7f985406fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:50.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.089+0000 7f9859417640 1 --2- 172.21.15.57:0/4147576789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f985414fc50 0x7f985406fdf0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f98440097c0 tx=0x7f984401c3f0 comp rx=0 tx=0).stop 2023-10-30T14:17:50.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.089+0000 7f9859417640 1 -- 172.21.15.57:0/4147576789 shutdown_connections 2023-10-30T14:17:50.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.089+0000 7f9859417640 1 --2- 172.21.15.57:0/4147576789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f985414fc50 0x7f985406fdf0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:50.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.089+0000 7f9859417640 1 -- 172.21.15.57:0/4147576789 >> 172.21.15.57:0/4147576789 conn(0x7f98540cac70 msgr2=0x7f98540cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:50.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.089+0000 7f9859417640 1 -- 172.21.15.57:0/4147576789 shutdown_connections 2023-10-30T14:17:50.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.089+0000 7f9859417640 1 -- 172.21.15.57:0/4147576789 wait complete. 2023-10-30T14:17:50.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f9859417640 1 Processor -- start 2023-10-30T14:17:50.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f9859417640 1 -- start start 2023-10-30T14:17:50.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f9859417640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f985406be90 0x7f9854160e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:50.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f9859417640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98540c6320 0x7f98540c3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:50.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f9859417640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f985414fc50 0x7f9854140950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:50.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f9859417640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985415f9d0 con 0x7f98540c6320 2023-10-30T14:17:50.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f9859417640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985413ed30 con 0x7f985406be90 2023-10-30T14:17:50.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f9859417640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98540c37b0 con 0x7f985414fc50 2023-10-30T14:17:50.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f984bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98540c6320 0x7f98540c3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:50.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f984bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98540c6320 0x7f98540c3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41802/0 (socket says 172.21.15.57:41802) 2023-10-30T14:17:50.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f984bfff640 1 -- 172.21.15.57:0/3681152861 learned_addr learned my addr 172.21.15.57:0/3681152861 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:50.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f9852ffd640 1 --2- 172.21.15.57:0/3681152861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f985406be90 0x7f9854160e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:50.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f98537fe640 1 --2- 172.21.15.57:0/3681152861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f985414fc50 0x7f9854140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:50.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f984bfff640 1 -- 172.21.15.57:0/3681152861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f985414fc50 msgr2=0x7f9854140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:50.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f984bfff640 1 --2- 172.21.15.57:0/3681152861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f985414fc50 0x7f9854140950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:50.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f984bfff640 1 -- 172.21.15.57:0/3681152861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f985406be90 msgr2=0x7f9854160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:50.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f984bfff640 1 --2- 172.21.15.57:0/3681152861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f985406be90 0x7f9854160e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:50.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f984bfff640 1 -- 172.21.15.57:0/3681152861 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9844009420 con 0x7f98540c6320 2023-10-30T14:17:50.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f984bfff640 1 --2- 172.21.15.57:0/3681152861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98540c6320 0x7f98540c3c40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f983c00c530 tx=0x7f983c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:50.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f984b7fe640 1 -- 172.21.15.57:0/3681152861 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983c005cf0 con 0x7f98540c6320 2023-10-30T14:17:50.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f984b7fe640 1 -- 172.21.15.57:0/3681152861 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983c005e50 con 0x7f98540c6320 2023-10-30T14:17:50.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f984b7fe640 1 -- 172.21.15.57:0/3681152861 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983c007270 con 0x7f98540c6320 2023-10-30T14:17:50.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f9859417640 1 -- 172.21.15.57:0/3681152861 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98541613d0 con 0x7f98540c6320 2023-10-30T14:17:50.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.093+0000 7f9859417640 1 -- 172.21.15.57:0/3681152861 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f985413e770 con 0x7f98540c6320 2023-10-30T14:17:50.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.097+0000 7f9859417640 1 -- 172.21.15.57:0/3681152861 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f985404e170 con 0x7f98540c6320 2023-10-30T14:17:50.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.097+0000 7f984b7fe640 1 -- 172.21.15.57:0/3681152861 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f983c0073d0 con 0x7f98540c6320 2023-10-30T14:17:50.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.097+0000 7f984b7fe640 1 --2- 172.21.15.57:0/3681152861 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f98340664a0 0x7f9834068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:50.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.097+0000 7f984b7fe640 1 -- 172.21.15.57:0/3681152861 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f983c0833b0 con 0x7f98540c6320 2023-10-30T14:17:50.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.097+0000 7f9852ffd640 1 --2- 172.21.15.57:0/3681152861 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f98340664a0 0x7f9834068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:50.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.097+0000 7f9852ffd640 1 --2- 172.21.15.57:0/3681152861 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f98340664a0 0x7f9834068960 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f98440ad000 tx=0x7f9844008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:50.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.101+0000 7f984b7fe640 1 -- 172.21.15.57:0/3681152861 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f983c0507d0 con 0x7f98540c6320 2023-10-30T14:17:50.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.293+0000 7f9859417640 1 -- 172.21.15.57:0/3681152861 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98540c38f0 con 0x7f98540c6320 2023-10-30T14:17:50.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.293+0000 7f984b7fe640 1 -- 172.21.15.57:0/3681152861 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f983c053df0 con 0x7f98540c6320 2023-10-30T14:17:50.299 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:50.300 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:44.635131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:50.300 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 29 2023-10-30T14:17:50.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.297+0000 7f9859417640 1 -- 172.21.15.57:0/3681152861 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f98340664a0 msgr2=0x7f9834068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:50.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.297+0000 7f9859417640 1 --2- 172.21.15.57:0/3681152861 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f98340664a0 0x7f9834068960 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f98440ad000 tx=0x7f9844008040 comp rx=0 tx=0).stop 2023-10-30T14:17:50.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.297+0000 7f9859417640 1 -- 172.21.15.57:0/3681152861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98540c6320 msgr2=0x7f98540c3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:50.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.297+0000 7f9859417640 1 --2- 172.21.15.57:0/3681152861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98540c6320 0x7f98540c3c40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f983c00c530 tx=0x7f983c00c9f0 comp rx=0 tx=0).stop 2023-10-30T14:17:50.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.297+0000 7f9859417640 1 -- 172.21.15.57:0/3681152861 shutdown_connections 2023-10-30T14:17:50.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.297+0000 7f9859417640 1 --2- 172.21.15.57:0/3681152861 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f98340664a0 0x7f9834068960 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:50.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.297+0000 7f9859417640 1 --2- 172.21.15.57:0/3681152861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f985414fc50 0x7f9854140950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:50.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.297+0000 7f9859417640 1 --2- 172.21.15.57:0/3681152861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98540c6320 0x7f98540c3c40 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:50.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.297+0000 7f9859417640 1 --2- 172.21.15.57:0/3681152861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f985406be90 0x7f9854160e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:50.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.297+0000 7f9859417640 1 -- 172.21.15.57:0/3681152861 >> 172.21.15.57:0/3681152861 conn(0x7f98540cac70 msgr2=0x7f98540ca490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:50.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.297+0000 7f9859417640 1 -- 172.21.15.57:0/3681152861 shutdown_connections 2023-10-30T14:17:50.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:50.297+0000 7f9859417640 1 -- 172.21.15.57:0/3681152861 wait complete. 2023-10-30T14:17:50.321 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:17:52.322 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:52.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.473+0000 7f36f2d2a640 1 Processor -- start 2023-10-30T14:17:52.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.473+0000 7f36f2d2a640 1 -- start start 2023-10-30T14:17:52.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.473+0000 7f36f2d2a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36ec151c20 0x7f36ec0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:52.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.473+0000 7f36f2d2a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ec0c5030 con 0x7f36ec151840 2023-10-30T14:17:52.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.473+0000 7f36f2d2a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ec157830 con 0x7f36ec0cb8b0 2023-10-30T14:17:52.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.473+0000 7f36f2d2a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec157b10 con 0x7f36ec151c20 2023-10-30T14:17:52.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.473+0000 7f36f0ac6640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f36ec151840 0x7f36ec072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44134/0 (socket says 172.21.15.57:44134) 2023-10-30T14:17:52.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.473+0000 7f36f0ac6640 1 -- 172.21.15.57:0/1096621970 learned_addr learned my addr 172.21.15.57:0/1096621970 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:17:52.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.473+0000 7f36ebfff640 1 --2- 172.21.15.57:0/1096621970 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36ec151c20 0x7f36ec0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:52.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.473+0000 7f36eb7fe640 1 -- 172.21.15.57:0/1096621970 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312467843 0 0) 0x7f36ec0c5030 con 0x7f36ec151840 2023-10-30T14:17:52.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.473+0000 7f36eb7fe640 1 -- 172.21.15.57:0/1096621970 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36dc003530 con 0x7f36ec151840 2023-10-30T14:17:52.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.473+0000 7f36eb7fe640 1 -- 172.21.15.57:0/1096621970 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1055131529 0 0) 0x7f36dc003530 con 0x7f36ec151840 2023-10-30T14:17:52.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.473+0000 7f36ebfff640 1 -- 172.21.15.57:0/1096621970 >> v1:172.21.15.88:6789/0 conn(0x7f36ec0cb8b0 legacy=0x7f36ec06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:52.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.473+0000 7f36ebfff640 1 -- 172.21.15.57:0/1096621970 >> v1:172.21.15.57:6789/0 conn(0x7f36ec151840 legacy=0x7f36ec072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:52.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.473+0000 7f36ebfff640 1 -- 172.21.15.57:0/1096621970 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ec157df0 con 0x7f36ec151c20 2023-10-30T14:17:52.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.477+0000 7f36ebfff640 1 --2- 172.21.15.57:0/1096621970 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36ec151c20 0x7f36ec0c49a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f36d80097f0 tx=0x7f36d801c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ff54f65c48b8e83 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:52.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.477+0000 7f36eb7fe640 1 -- 172.21.15.57:0/1096621970 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e0001680 con 0x7f36ec151c20 2023-10-30T14:17:52.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.477+0000 7f36eb7fe640 1 -- 172.21.15.57:0/1096621970 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d80a7050 con 0x7f36ec151c20 2023-10-30T14:17:52.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.477+0000 7f36eb7fe640 1 -- 172.21.15.57:0/1096621970 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d80ab600 con 0x7f36ec151c20 2023-10-30T14:17:52.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.477+0000 7f36f2d2a640 1 -- 172.21.15.57:0/1096621970 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36ec151c20 msgr2=0x7f36ec0c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:52.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.477+0000 7f36f2d2a640 1 --2- 172.21.15.57:0/1096621970 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36ec151c20 0x7f36ec0c49a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f36d80097f0 tx=0x7f36d801c320 comp rx=0 tx=0).stop 2023-10-30T14:17:52.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.477+0000 7f36f2d2a640 1 -- 172.21.15.57:0/1096621970 shutdown_connections 2023-10-30T14:17:52.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.477+0000 7f36f2d2a640 1 --2- 172.21.15.57:0/1096621970 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36ec151c20 0x7f36ec0c49a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:52.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.477+0000 7f36f2d2a640 1 -- 172.21.15.57:0/1096621970 >> 172.21.15.57:0/1096621970 conn(0x7f36ec0cac70 msgr2=0x7f36ec0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:52.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.477+0000 7f36f2d2a640 1 -- 172.21.15.57:0/1096621970 shutdown_connections 2023-10-30T14:17:52.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.477+0000 7f36f2d2a640 1 -- 172.21.15.57:0/1096621970 wait complete. 2023-10-30T14:17:52.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36f2d2a640 1 Processor -- start 2023-10-30T14:17:52.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36f2d2a640 1 -- start start 2023-10-30T14:17:52.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36f2d2a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36ec0cb8b0 0x7f36ec14f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:52.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36f2d2a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36ec151840 0x7f36ec143490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:52.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36f2d2a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36ec151c20 0x7f36ec1439d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:52.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36f2d2a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec143f10 con 0x7f36ec151c20 2023-10-30T14:17:52.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36f2d2a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec073d10 con 0x7f36ec151840 2023-10-30T14:17:52.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36f2d2a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec073820 con 0x7f36ec0cb8b0 2023-10-30T14:17:52.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36f12c7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36ec151c20 0x7f36ec1439d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:52.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36f12c7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36ec151c20 0x7f36ec1439d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41816/0 (socket says 172.21.15.57:41816) 2023-10-30T14:17:52.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36f12c7640 1 -- 172.21.15.57:0/1505785687 learned_addr learned my addr 172.21.15.57:0/1505785687 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:52.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36f0ac6640 1 --2- 172.21.15.57:0/1505785687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36ec0cb8b0 0x7f36ec14f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:52.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36ebfff640 1 --2- 172.21.15.57:0/1505785687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36ec151840 0x7f36ec143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:52.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36f12c7640 1 -- 172.21.15.57:0/1505785687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36ec0cb8b0 msgr2=0x7f36ec14f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:52.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36f12c7640 1 --2- 172.21.15.57:0/1505785687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36ec0cb8b0 0x7f36ec14f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:52.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36f12c7640 1 -- 172.21.15.57:0/1505785687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36ec151840 msgr2=0x7f36ec143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:52.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36f12c7640 1 --2- 172.21.15.57:0/1505785687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36ec151840 0x7f36ec143490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:52.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36f12c7640 1 -- 172.21.15.57:0/1505785687 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d8009420 con 0x7f36ec151c20 2023-10-30T14:17:52.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36f12c7640 1 --2- 172.21.15.57:0/1505785687 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36ec151c20 0x7f36ec1439d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f36e400c360 tx=0x7f36e400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:52.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36e97fa640 1 -- 172.21.15.57:0/1505785687 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e4018070 con 0x7f36ec151c20 2023-10-30T14:17:52.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36f2d2a640 1 -- 172.21.15.57:0/1505785687 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36ec154730 con 0x7f36ec151c20 2023-10-30T14:17:52.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36e97fa640 1 -- 172.21.15.57:0/1505785687 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e4003980 con 0x7f36ec151c20 2023-10-30T14:17:52.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.481+0000 7f36e97fa640 1 -- 172.21.15.57:0/1505785687 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e4003d50 con 0x7f36ec151c20 2023-10-30T14:17:52.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.485+0000 7f36f2d2a640 1 -- 172.21.15.57:0/1505785687 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36ec073a30 con 0x7f36ec151c20 2023-10-30T14:17:52.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.485+0000 7f36f2d2a640 1 -- 172.21.15.57:0/1505785687 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36ec04e170 con 0x7f36ec151c20 2023-10-30T14:17:52.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.485+0000 7f36e97fa640 1 -- 172.21.15.57:0/1505785687 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f36e40135b0 con 0x7f36ec151c20 2023-10-30T14:17:52.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.485+0000 7f36e97fa640 1 --2- 172.21.15.57:0/1505785687 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f36d00664a0 0x7f36d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:52.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.485+0000 7f36e97fa640 1 -- 172.21.15.57:0/1505785687 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f36e4086cf0 con 0x7f36ec151c20 2023-10-30T14:17:52.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.485+0000 7f36f0ac6640 1 --2- 172.21.15.57:0/1505785687 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f36d00664a0 0x7f36d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:52.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.489+0000 7f36f0ac6640 1 --2- 172.21.15.57:0/1505785687 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f36d00664a0 0x7f36d0068960 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f36e00011f0 tx=0x7f36e0000ee0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:52.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.489+0000 7f36e97fa640 1 -- 172.21.15.57:0/1505785687 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f36e4054090 con 0x7f36ec151c20 2023-10-30T14:17:52.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.681+0000 7f36f2d2a640 1 -- 172.21.15.57:0/1505785687 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36ec0c3f00 con 0x7f36ec151c20 2023-10-30T14:17:52.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.681+0000 7f36e97fa640 1 -- 172.21.15.57:0/1505785687 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f36e40576b0 con 0x7f36ec151c20 2023-10-30T14:17:52.686 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:52.686 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:49.655748+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:resolve","state_seq":9,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:52.686 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 30 2023-10-30T14:17:52.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.681+0000 7f36f2d2a640 1 -- 172.21.15.57:0/1505785687 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f36d00664a0 msgr2=0x7f36d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:52.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.681+0000 7f36f2d2a640 1 --2- 172.21.15.57:0/1505785687 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f36d00664a0 0x7f36d0068960 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f36e00011f0 tx=0x7f36e0000ee0 comp rx=0 tx=0).stop 2023-10-30T14:17:52.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.681+0000 7f36f2d2a640 1 -- 172.21.15.57:0/1505785687 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36ec151c20 msgr2=0x7f36ec1439d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:52.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.681+0000 7f36f2d2a640 1 --2- 172.21.15.57:0/1505785687 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36ec151c20 0x7f36ec1439d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f36e400c360 tx=0x7f36e400c820 comp rx=0 tx=0).stop 2023-10-30T14:17:52.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.681+0000 7f36f2d2a640 1 -- 172.21.15.57:0/1505785687 shutdown_connections 2023-10-30T14:17:52.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.681+0000 7f36f2d2a640 1 --2- 172.21.15.57:0/1505785687 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f36d00664a0 0x7f36d0068960 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:52.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.681+0000 7f36f2d2a640 1 --2- 172.21.15.57:0/1505785687 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36ec151c20 0x7f36ec1439d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:52.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.681+0000 7f36f2d2a640 1 --2- 172.21.15.57:0/1505785687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36ec151840 0x7f36ec143490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:52.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.681+0000 7f36f2d2a640 1 --2- 172.21.15.57:0/1505785687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36ec0cb8b0 0x7f36ec14f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:52.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.685+0000 7f36f2d2a640 1 -- 172.21.15.57:0/1505785687 >> 172.21.15.57:0/1505785687 conn(0x7f36ec0cac70 msgr2=0x7f36ec0c63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:52.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.685+0000 7f36f2d2a640 1 -- 172.21.15.57:0/1505785687 shutdown_connections 2023-10-30T14:17:52.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:52.685+0000 7f36f2d2a640 1 -- 172.21.15.57:0/1505785687 wait complete. 2023-10-30T14:17:52.707 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:17:54.708 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:54.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3266e3a640 1 Processor -- start 2023-10-30T14:17:54.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3266e3a640 1 -- start start 2023-10-30T14:17:54.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3266e3a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f32600cb180 0x7f32601597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:54.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3266e3a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326015a770 con 0x7f3260153b50 2023-10-30T14:17:54.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3266e3a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32600c9d40 con 0x7f32600c62b0 2023-10-30T14:17:54.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3266e3a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3260159cf0 con 0x7f32600cb180 2023-10-30T14:17:54.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3265e38640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f3260153b50 0x7f32600ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:58078/0 (socket says 172.21.15.57:58078) 2023-10-30T14:17:54.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3265e38640 1 -- 172.21.15.57:0/2051655833 learned_addr learned my addr 172.21.15.57:0/2051655833 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:17:54.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3266639640 1 --2- 172.21.15.57:0/2051655833 >> v2:172.21.15.88:3300/0 conn(0x7f32600cb180 0x7f32601597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:54.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3264e36640 1 -- 172.21.15.57:0/2051655833 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692835024 0 0) 0x7f325c001120 con 0x7f3260153b50 2023-10-30T14:17:54.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3264e36640 1 -- 172.21.15.57:0/2051655833 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f324c003530 con 0x7f3260153b50 2023-10-30T14:17:54.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3264e36640 1 -- 172.21.15.57:0/2051655833 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239149813 0 0) 0x7f32600c9d40 con 0x7f32600c62b0 2023-10-30T14:17:54.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3264e36640 1 -- 172.21.15.57:0/2051655833 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f325c001120 con 0x7f32600c62b0 2023-10-30T14:17:54.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3264e36640 1 -- 172.21.15.57:0/2051655833 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3573764119 0 0) 0x7f324c003530 con 0x7f3260153b50 2023-10-30T14:17:54.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3264e36640 1 -- 172.21.15.57:0/2051655833 >> v2:172.21.15.88:3300/0 conn(0x7f32600cb180 msgr2=0x7f32601597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:54.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3264e36640 1 --2- 172.21.15.57:0/2051655833 >> v2:172.21.15.88:3300/0 conn(0x7f32600cb180 0x7f32601597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:54.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3264e36640 1 -- 172.21.15.57:0/2051655833 >> v1:172.21.15.88:6789/0 conn(0x7f32600c62b0 legacy=0x7f326006ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:54.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3264e36640 1 -- 172.21.15.57:0/2051655833 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3260153f30 con 0x7f3260153b50 2023-10-30T14:17:54.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3264e36640 1 -- 172.21.15.57:0/2051655833 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f325c001810 con 0x7f3260153b50 2023-10-30T14:17:54.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3264e36640 1 -- 172.21.15.57:0/2051655833 >> v1:172.21.15.57:6789/0 conn(0x7f3260153b50 legacy=0x7f32600ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:54.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3264e36640 1 --2- 172.21.15.57:0/2051655833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f324c003b10 0x7f324c003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:54.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3264e36640 1 --2- 172.21.15.57:0/2051655833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f324c0047d0 0x7f324c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:54.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3264e36640 1 --2- 172.21.15.57:0/2051655833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f324c005190 0x7f324c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:54.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3264e36640 1 -- 172.21.15.57:0/2051655833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3260159cf0 con 0x7f324c0047d0 2023-10-30T14:17:54.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3264e36640 1 -- 172.21.15.57:0/2051655833 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324c00bf80 con 0x7f324c003b10 2023-10-30T14:17:54.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3264e36640 1 -- 172.21.15.57:0/2051655833 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f324c00c140 con 0x7f324c005190 2023-10-30T14:17:54.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3265637640 1 --2- 172.21.15.57:0/2051655833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f324c0047d0 0x7f324c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:54.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3265e38640 1 --2- 172.21.15.57:0/2051655833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f324c003b10 0x7f324c003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:54.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3266639640 1 --2- 172.21.15.57:0/2051655833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f324c005190 0x7f324c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:54.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3265637640 1 -- 172.21.15.57:0/2051655833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f324c005190 msgr2=0x7f324c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:54.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3265637640 1 --2- 172.21.15.57:0/2051655833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f324c005190 0x7f324c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:54.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3265637640 1 -- 172.21.15.57:0/2051655833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f324c003b10 msgr2=0x7f324c003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:54.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3265637640 1 --2- 172.21.15.57:0/2051655833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f324c003b10 0x7f324c003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:54.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.861+0000 7f3265637640 1 -- 172.21.15.57:0/2051655833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f324c003530 con 0x7f324c0047d0 2023-10-30T14:17:54.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.865+0000 7f3266639640 1 -- 172.21.15.57:0/2051655833 reap_dead start 2023-10-30T14:17:54.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.865+0000 7f3265637640 1 --2- 172.21.15.57:0/2051655833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f324c0047d0 0x7f324c004c50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3250007fd0 tx=0x7f325001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c3cf702dc075514 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:54.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.865+0000 7f3264e36640 1 -- 172.21.15.57:0/2051655833 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32500052d0 con 0x7f324c0047d0 2023-10-30T14:17:54.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.865+0000 7f3264e36640 1 -- 172.21.15.57:0/2051655833 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32500a7050 con 0x7f324c0047d0 2023-10-30T14:17:54.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.865+0000 7f3264e36640 1 -- 172.21.15.57:0/2051655833 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250003600 con 0x7f324c0047d0 2023-10-30T14:17:54.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.865+0000 7f3266e3a640 1 -- 172.21.15.57:0/2051655833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f324c0047d0 msgr2=0x7f324c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:54.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.865+0000 7f3266e3a640 1 --2- 172.21.15.57:0/2051655833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f324c0047d0 0x7f324c004c50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3250007fd0 tx=0x7f325001c480 comp rx=0 tx=0).stop 2023-10-30T14:17:54.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.865+0000 7f3266e3a640 1 -- 172.21.15.57:0/2051655833 shutdown_connections 2023-10-30T14:17:54.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.865+0000 7f3266e3a640 1 --2- 172.21.15.57:0/2051655833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f324c0047d0 0x7f324c004c50 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:54.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.865+0000 7f3266e3a640 1 -- 172.21.15.57:0/2051655833 >> 172.21.15.57:0/2051655833 conn(0x7f3260065a80 msgr2=0x7f3260065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:54.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.865+0000 7f3266e3a640 1 -- 172.21.15.57:0/2051655833 shutdown_connections 2023-10-30T14:17:54.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.865+0000 7f3266e3a640 1 -- 172.21.15.57:0/2051655833 wait complete. 2023-10-30T14:17:54.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3266e3a640 1 Processor -- start 2023-10-30T14:17:54.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3266e3a640 1 -- start start 2023-10-30T14:17:54.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3266e3a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f324c0047d0 0x7f32600c40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:54.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3266e3a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3260150a50 0x7f3260152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:54.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3266e3a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3260151d40 0x7f32601521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:54.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3266e3a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32600c32a0 con 0x7f324c0047d0 2023-10-30T14:17:54.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3266e3a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3260151920 con 0x7f3260150a50 2023-10-30T14:17:54.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3266e3a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32600c2ce0 con 0x7f3260151d40 2023-10-30T14:17:54.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3265e38640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f324c0047d0 0x7f32600c40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:54.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3265e38640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f324c0047d0 0x7f32600c40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40962/0 (socket says 172.21.15.57:40962) 2023-10-30T14:17:54.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3265e38640 1 -- 172.21.15.57:0/319566565 learned_addr learned my addr 172.21.15.57:0/319566565 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:54.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3265637640 1 --2- 172.21.15.57:0/319566565 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3260150a50 0x7f3260152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:54.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3266639640 1 --2- 172.21.15.57:0/319566565 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3260151d40 0x7f32601521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:54.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3265e38640 1 -- 172.21.15.57:0/319566565 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3260151d40 msgr2=0x7f32601521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:54.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3265e38640 1 --2- 172.21.15.57:0/319566565 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3260151d40 0x7f32601521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:54.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3265e38640 1 -- 172.21.15.57:0/319566565 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3260150a50 msgr2=0x7f3260152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:54.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3265e38640 1 --2- 172.21.15.57:0/319566565 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3260150a50 0x7f3260152a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:54.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3265e38640 1 -- 172.21.15.57:0/319566565 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32600c2fc0 con 0x7f324c0047d0 2023-10-30T14:17:54.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3266639640 1 --2- 172.21.15.57:0/319566565 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3260151d40 0x7f32601521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:17:54.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3265e38640 1 --2- 172.21.15.57:0/319566565 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f324c0047d0 0x7f32600c40a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f325c001380 tx=0x7f325c0129a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:54.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f32567fc640 1 -- 172.21.15.57:0/319566565 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325c014810 con 0x7f324c0047d0 2023-10-30T14:17:54.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f32567fc640 1 -- 172.21.15.57:0/319566565 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325c014970 con 0x7f324c0047d0 2023-10-30T14:17:54.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3266e3a640 1 -- 172.21.15.57:0/319566565 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3260151260 con 0x7f324c0047d0 2023-10-30T14:17:54.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f32567fc640 1 -- 172.21.15.57:0/319566565 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325c014d00 con 0x7f324c0047d0 2023-10-30T14:17:54.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.869+0000 7f3266e3a640 1 -- 172.21.15.57:0/319566565 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32600c5ab0 con 0x7f324c0047d0 2023-10-30T14:17:54.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.873+0000 7f3266e3a640 1 -- 172.21.15.57:0/319566565 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32601510c0 con 0x7f324c0047d0 2023-10-30T14:17:54.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.873+0000 7f32567fc640 1 -- 172.21.15.57:0/319566565 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f325c0153e0 con 0x7f324c0047d0 2023-10-30T14:17:54.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.873+0000 7f32567fc640 1 --2- 172.21.15.57:0/319566565 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32440664d0 0x7f3244068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:54.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.873+0000 7f32567fc640 1 -- 172.21.15.57:0/319566565 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f325c086a50 con 0x7f324c0047d0 2023-10-30T14:17:54.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.873+0000 7f3265637640 1 --2- 172.21.15.57:0/319566565 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32440664d0 0x7f3244068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:54.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.873+0000 7f3265637640 1 --2- 172.21.15.57:0/319566565 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32440664d0 0x7f3244068990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3250007fd0 tx=0x7f32500a6000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:54.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:54.877+0000 7f32567fc640 1 -- 172.21.15.57:0/319566565 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f32601510c0 con 0x7f324c0047d0 2023-10-30T14:17:55.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:55.069+0000 7f3266e3a640 1 -- 172.21.15.57:0/319566565 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3260151a60 con 0x7f324c0047d0 2023-10-30T14:17:55.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:55.069+0000 7f32567fc640 1 -- 172.21.15.57:0/319566565 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f3260151a60 con 0x7f324c0047d0 2023-10-30T14:17:55.074 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:55.075 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:49.655748+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:resolve","state_seq":9,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:55.075 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 30 2023-10-30T14:17:55.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:55.073+0000 7f3266e3a640 1 -- 172.21.15.57:0/319566565 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32440664d0 msgr2=0x7f3244068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:55.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:55.073+0000 7f3266e3a640 1 --2- 172.21.15.57:0/319566565 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32440664d0 0x7f3244068990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3250007fd0 tx=0x7f32500a6000 comp rx=0 tx=0).stop 2023-10-30T14:17:55.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:55.073+0000 7f3266e3a640 1 -- 172.21.15.57:0/319566565 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f324c0047d0 msgr2=0x7f32600c40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:55.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:55.073+0000 7f3266e3a640 1 --2- 172.21.15.57:0/319566565 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f324c0047d0 0x7f32600c40a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f325c001380 tx=0x7f325c0129a0 comp rx=0 tx=0).stop 2023-10-30T14:17:55.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:55.073+0000 7f3266e3a640 1 -- 172.21.15.57:0/319566565 shutdown_connections 2023-10-30T14:17:55.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:55.073+0000 7f3266e3a640 1 --2- 172.21.15.57:0/319566565 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32440664d0 0x7f3244068990 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:55.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:55.073+0000 7f3266e3a640 1 --2- 172.21.15.57:0/319566565 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3260151d40 0x7f32601521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:55.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:55.073+0000 7f3266e3a640 1 --2- 172.21.15.57:0/319566565 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3260150a50 0x7f3260152a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:55.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:55.073+0000 7f3266e3a640 1 --2- 172.21.15.57:0/319566565 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f324c0047d0 0x7f32600c40a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:55.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:55.073+0000 7f3266e3a640 1 -- 172.21.15.57:0/319566565 >> 172.21.15.57:0/319566565 conn(0x7f3260065a80 msgr2=0x7f32600c6700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:55.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:55.073+0000 7f3266e3a640 1 -- 172.21.15.57:0/319566565 shutdown_connections 2023-10-30T14:17:55.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:55.073+0000 7f3266e3a640 1 -- 172.21.15.57:0/319566565 wait complete. 2023-10-30T14:17:55.095 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:17:57.097 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:57.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.249+0000 7fcc6893b640 1 Processor -- start 2023-10-30T14:17:57.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.249+0000 7fcc6893b640 1 -- start start 2023-10-30T14:17:57.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.249+0000 7fcc6893b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcc640c8eb0 0x7fcc6415cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:57.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.249+0000 7fcc6893b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc6415dcc0 con 0x7fcc6406aeb0 2023-10-30T14:17:57.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.249+0000 7fcc6893b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc64159d80 con 0x7fcc6415ac70 2023-10-30T14:17:57.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.249+0000 7fcc6893b640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc64159800 con 0x7fcc640c8eb0 2023-10-30T14:17:57.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.249+0000 7fcc637fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcc640c8eb0 0x7fcc6415cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:57.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.249+0000 7fcc637fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcc640c8eb0 0x7fcc6415cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40970/0 (socket says 172.21.15.57:40970) 2023-10-30T14:17:57.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.249+0000 7fcc637fe640 1 -- 172.21.15.57:0/1504637057 learned_addr learned my addr 172.21.15.57:0/1504637057 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:57.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.249+0000 7fcc63fff640 1 --1- 172.21.15.57:0/1504637057 >> v1:172.21.15.57:6789/0 conn(0x7fcc6406aeb0 0x7fcc64069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:58088/0 (socket says 172.21.15.57:58088) 2023-10-30T14:17:57.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.249+0000 7fcc637fe640 1 -- 172.21.15.57:0/1504637057 >> v1:172.21.15.88:6789/0 conn(0x7fcc6415ac70 legacy=0x7fcc6415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:57.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.249+0000 7fcc637fe640 1 -- 172.21.15.57:0/1504637057 >> v1:172.21.15.57:6789/0 conn(0x7fcc6406aeb0 legacy=0x7fcc64069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:57.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.253+0000 7fcc627fc640 1 -- 172.21.15.57:0/1504637057 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050370747 0 0) 0x7fcc5c0014a0 con 0x7fcc6406aeb0 2023-10-30T14:17:57.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.253+0000 7fcc637fe640 1 -- 172.21.15.57:0/1504637057 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6415d420 con 0x7fcc640c8eb0 2023-10-30T14:17:57.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.253+0000 7fcc637fe640 1 --2- 172.21.15.57:0/1504637057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc640c8eb0 0x7fcc6415cd70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcc54009870 tx=0x7fcc5401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812d6f37970ddc97 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:57.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.253+0000 7fcc627fc640 1 -- 172.21.15.57:0/1504637057 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc540a8a30 con 0x7fcc640c8eb0 2023-10-30T14:17:57.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.253+0000 7fcc627fc640 1 -- 172.21.15.57:0/1504637057 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc540a8b90 con 0x7fcc640c8eb0 2023-10-30T14:17:57.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.253+0000 7fcc627fc640 1 -- 172.21.15.57:0/1504637057 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc540b0440 con 0x7fcc640c8eb0 2023-10-30T14:17:57.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.253+0000 7fcc6893b640 1 -- 172.21.15.57:0/1504637057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc640c8eb0 msgr2=0x7fcc6415cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:57.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.253+0000 7fcc6893b640 1 --2- 172.21.15.57:0/1504637057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc640c8eb0 0x7fcc6415cd70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcc54009870 tx=0x7fcc5401ed30 comp rx=0 tx=0).stop 2023-10-30T14:17:57.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.253+0000 7fcc6893b640 1 -- 172.21.15.57:0/1504637057 shutdown_connections 2023-10-30T14:17:57.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.253+0000 7fcc6893b640 1 --2- 172.21.15.57:0/1504637057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc640c8eb0 0x7fcc6415cd70 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:57.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.253+0000 7fcc6893b640 1 -- 172.21.15.57:0/1504637057 >> 172.21.15.57:0/1504637057 conn(0x7fcc640cef00 msgr2=0x7fcc640cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:57.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.253+0000 7fcc6893b640 1 -- 172.21.15.57:0/1504637057 shutdown_connections 2023-10-30T14:17:57.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.253+0000 7fcc6893b640 1 -- 172.21.15.57:0/1504637057 wait complete. 2023-10-30T14:17:57.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc6893b640 1 Processor -- start 2023-10-30T14:17:57.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc6893b640 1 -- start start 2023-10-30T14:17:57.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc6893b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcc6406aeb0 0x7fcc6414f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:57.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc6893b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc640c8eb0 0x7fcc641f09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:57.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc6893b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc6415ac70 0x7fcc641f0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:57.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc6893b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc641e51c0 con 0x7fcc640c8eb0 2023-10-30T14:17:57.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc6893b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc641e5830 con 0x7fcc6415ac70 2023-10-30T14:17:57.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc6893b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc64158b70 con 0x7fcc6406aeb0 2023-10-30T14:17:57.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc62ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc640c8eb0 0x7fcc641f09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:57.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc62ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc640c8eb0 0x7fcc641f09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40974/0 (socket says 172.21.15.57:40974) 2023-10-30T14:17:57.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc62ffd640 1 -- 172.21.15.57:0/1800008378 learned_addr learned my addr 172.21.15.57:0/1800008378 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:57.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc637fe640 1 --2- 172.21.15.57:0/1800008378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcc6406aeb0 0x7fcc6414f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:57.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc63fff640 1 --2- 172.21.15.57:0/1800008378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc6415ac70 0x7fcc641f0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:57.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc62ffd640 1 -- 172.21.15.57:0/1800008378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcc6406aeb0 msgr2=0x7fcc6414f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:57.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc62ffd640 1 --2- 172.21.15.57:0/1800008378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcc6406aeb0 0x7fcc6414f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:57.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc62ffd640 1 -- 172.21.15.57:0/1800008378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc6415ac70 msgr2=0x7fcc641f0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:57.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc62ffd640 1 --2- 172.21.15.57:0/1800008378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc6415ac70 0x7fcc641f0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:57.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc62ffd640 1 -- 172.21.15.57:0/1800008378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc540094d0 con 0x7fcc640c8eb0 2023-10-30T14:17:57.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc62ffd640 1 --2- 172.21.15.57:0/1800008378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc640c8eb0 0x7fcc641f09f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fcc5800c3c0 tx=0x7fcc5800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:57.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.257+0000 7fcc4bfff640 1 -- 172.21.15.57:0/1800008378 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc580052d0 con 0x7fcc640c8eb0 2023-10-30T14:17:57.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.261+0000 7fcc4bfff640 1 -- 172.21.15.57:0/1800008378 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc58005430 con 0x7fcc640c8eb0 2023-10-30T14:17:57.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.261+0000 7fcc4bfff640 1 -- 172.21.15.57:0/1800008378 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc58003840 con 0x7fcc640c8eb0 2023-10-30T14:17:57.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.261+0000 7fcc6893b640 1 -- 172.21.15.57:0/1800008378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc6414fb70 con 0x7fcc640c8eb0 2023-10-30T14:17:57.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.261+0000 7fcc6893b640 1 -- 172.21.15.57:0/1800008378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc641e3900 con 0x7fcc640c8eb0 2023-10-30T14:17:57.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.261+0000 7fcc6893b640 1 -- 172.21.15.57:0/1800008378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc641f0710 con 0x7fcc640c8eb0 2023-10-30T14:17:57.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.261+0000 7fcc4bfff640 1 -- 172.21.15.57:0/1800008378 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcc580039a0 con 0x7fcc640c8eb0 2023-10-30T14:17:57.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.261+0000 7fcc4bfff640 1 --2- 172.21.15.57:0/1800008378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcc500664a0 0x7fcc50068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:57.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.261+0000 7fcc4bfff640 1 -- 172.21.15.57:0/1800008378 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7fcc58083680 con 0x7fcc640c8eb0 2023-10-30T14:17:57.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.261+0000 7fcc637fe640 1 --2- 172.21.15.57:0/1800008378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcc500664a0 0x7fcc50068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:57.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.265+0000 7fcc637fe640 1 --2- 172.21.15.57:0/1800008378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcc500664a0 0x7fcc50068960 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcc540b2000 tx=0x7fcc54008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:57.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.265+0000 7fcc4bfff640 1 -- 172.21.15.57:0/1800008378 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcc641f0710 con 0x7fcc640c8eb0 2023-10-30T14:17:57.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.457+0000 7fcc6893b640 1 -- 172.21.15.57:0/1800008378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc641f0710 con 0x7fcc640c8eb0 2023-10-30T14:17:57.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.457+0000 7fcc4bfff640 1 -- 172.21.15.57:0/1800008378 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fcc641f0710 con 0x7fcc640c8eb0 2023-10-30T14:17:57.461 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:57.461 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:55.647291+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:57.462 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 31 2023-10-30T14:17:57.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.457+0000 7fcc6893b640 1 -- 172.21.15.57:0/1800008378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcc500664a0 msgr2=0x7fcc50068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:57.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.457+0000 7fcc6893b640 1 --2- 172.21.15.57:0/1800008378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcc500664a0 0x7fcc50068960 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcc540b2000 tx=0x7fcc54008040 comp rx=0 tx=0).stop 2023-10-30T14:17:57.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.457+0000 7fcc6893b640 1 -- 172.21.15.57:0/1800008378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc640c8eb0 msgr2=0x7fcc641f09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:57.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.457+0000 7fcc6893b640 1 --2- 172.21.15.57:0/1800008378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc640c8eb0 0x7fcc641f09f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fcc5800c3c0 tx=0x7fcc5800c880 comp rx=0 tx=0).stop 2023-10-30T14:17:57.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.457+0000 7fcc6893b640 1 -- 172.21.15.57:0/1800008378 shutdown_connections 2023-10-30T14:17:57.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.457+0000 7fcc6893b640 1 --2- 172.21.15.57:0/1800008378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcc500664a0 0x7fcc50068960 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:57.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.457+0000 7fcc6893b640 1 --2- 172.21.15.57:0/1800008378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc6415ac70 0x7fcc641f0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:57.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.457+0000 7fcc6893b640 1 --2- 172.21.15.57:0/1800008378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc640c8eb0 0x7fcc641f09f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:57.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.457+0000 7fcc6893b640 1 --2- 172.21.15.57:0/1800008378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcc6406aeb0 0x7fcc6414f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:57.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.457+0000 7fcc6893b640 1 -- 172.21.15.57:0/1800008378 >> 172.21.15.57:0/1800008378 conn(0x7fcc640cef00 msgr2=0x7fcc640c65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:57.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.457+0000 7fcc6893b640 1 -- 172.21.15.57:0/1800008378 shutdown_connections 2023-10-30T14:17:57.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:57.457+0000 7fcc6893b640 1 -- 172.21.15.57:0/1800008378 wait complete. 2023-10-30T14:17:57.482 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 31, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 31, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4282": { "addr": "172.21.15.88:6839/2945854794", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 2945854794, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2945854794, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4282, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4936": { "addr": "172.21.15.57:6837/1988100930", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 1988100930, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 1988100930, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4936, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 31 }, "gid_5133": { "addr": "172.21.15.57:6834/2912050965", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2912050965, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2912050965, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5133, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:reconnect", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:17:55.647291+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4282, "mds_1": 5133, "mds_2": 4936 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:17:59.483 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:17:59.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.633+0000 7f53b1bb8640 1 Processor -- start 2023-10-30T14:17:59.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.633+0000 7f53b1bb8640 1 -- start start 2023-10-30T14:17:59.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.633+0000 7f53b1bb8640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f53ac0cb7d0 0x7f53ac0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:59.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.633+0000 7f53b1bb8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f53ac0c3e40 0x7f53ac070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:59.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.633+0000 7f53b1bb8640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53ac0c2310 con 0x7f53ac150830 2023-10-30T14:17:59.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.633+0000 7f53b1bb8640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53ac06bc10 con 0x7f53ac0cb7d0 2023-10-30T14:17:59.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.633+0000 7f53b1bb8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53ac156710 con 0x7f53ac0c3e40 2023-10-30T14:17:59.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.633+0000 7f53aaffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f53ac0cb7d0 0x7f53ac0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:59.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.633+0000 7f53aaffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f53ac0cb7d0 0x7f53ac0c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40976/0 (socket says 172.21.15.57:40976) 2023-10-30T14:17:59.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.633+0000 7f53aaffd640 1 -- 172.21.15.57:0/4073672232 learned_addr learned my addr 172.21.15.57:0/4073672232 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:59.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.637+0000 7f53abfff640 1 --2- 172.21.15.57:0/4073672232 >> v2:172.21.15.88:3300/0 conn(0x7f53ac0c3e40 0x7f53ac070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:59.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.637+0000 7f53aaffd640 1 -- 172.21.15.57:0/4073672232 >> v2:172.21.15.88:3300/0 conn(0x7f53ac0c3e40 msgr2=0x7f53ac070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:17:59.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.637+0000 7f53aaffd640 1 --2- 172.21.15.57:0/4073672232 >> v2:172.21.15.88:3300/0 conn(0x7f53ac0c3e40 0x7f53ac070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:59.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.637+0000 7f53aaffd640 1 -- 172.21.15.57:0/4073672232 >> v1:172.21.15.57:6789/0 conn(0x7f53ac150830 legacy=0x7f53ac06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:59.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.637+0000 7f53aaffd640 1 -- 172.21.15.57:0/4073672232 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ac0c4360 con 0x7f53ac0cb7d0 2023-10-30T14:17:59.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.637+0000 7f53aaffd640 1 --2- 172.21.15.57:0/4073672232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53ac0cb7d0 0x7f53ac0c3900 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f53ac066e10 tx=0x7f53a0020f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68a6bdf8136a8547 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:59.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.641+0000 7f53aa7fc640 1 -- 172.21.15.57:0/4073672232 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a00aac00 con 0x7f53ac0cb7d0 2023-10-30T14:17:59.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.641+0000 7f53aa7fc640 1 -- 172.21.15.57:0/4073672232 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a00aad60 con 0x7f53ac0cb7d0 2023-10-30T14:17:59.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.641+0000 7f53aa7fc640 1 -- 172.21.15.57:0/4073672232 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a00b2660 con 0x7f53ac0cb7d0 2023-10-30T14:17:59.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.641+0000 7f53b1bb8640 1 -- 172.21.15.57:0/4073672232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53ac0cb7d0 msgr2=0x7f53ac0c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:59.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.641+0000 7f53b1bb8640 1 --2- 172.21.15.57:0/4073672232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53ac0cb7d0 0x7f53ac0c3900 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f53ac066e10 tx=0x7f53a0020f00 comp rx=0 tx=0).stop 2023-10-30T14:17:59.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.641+0000 7f53b1bb8640 1 -- 172.21.15.57:0/4073672232 shutdown_connections 2023-10-30T14:17:59.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.641+0000 7f53b1bb8640 1 --2- 172.21.15.57:0/4073672232 >> v2:172.21.15.88:3300/0 conn(0x7f53ac0c3e40 0x7f53ac070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:59.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.641+0000 7f53b1bb8640 1 --2- 172.21.15.57:0/4073672232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53ac0cb7d0 0x7f53ac0c3900 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:59.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.641+0000 7f53b1bb8640 1 -- 172.21.15.57:0/4073672232 >> 172.21.15.57:0/4073672232 conn(0x7f53ac0cac70 msgr2=0x7f53ac0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:59.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.641+0000 7f53b1bb8640 1 -- 172.21.15.57:0/4073672232 shutdown_connections 2023-10-30T14:17:59.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.641+0000 7f53b1bb8640 1 -- 172.21.15.57:0/4073672232 wait complete. 2023-10-30T14:17:59.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53b1bb8640 1 Processor -- start 2023-10-30T14:17:59.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53b1bb8640 1 -- start start 2023-10-30T14:17:59.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53b1bb8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53ac0c3e40 0x7f53ac14b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:59.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53b1bb8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53ac0cb7d0 0x7f53ac1577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:59.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53b1bb8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53ac150830 0x7f53ac1f2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:59.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53b1bb8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac1f6c80 con 0x7f53ac150830 2023-10-30T14:17:59.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53b1bb8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac148100 con 0x7f53ac0c3e40 2023-10-30T14:17:59.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53b1bb8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac1f70a0 con 0x7f53ac0cb7d0 2023-10-30T14:17:59.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53abfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53ac150830 0x7f53ac1f2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:59.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53abfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53ac150830 0x7f53ac1f2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40978/0 (socket says 172.21.15.57:40978) 2023-10-30T14:17:59.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53abfff640 1 -- 172.21.15.57:0/713990147 learned_addr learned my addr 172.21.15.57:0/713990147 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:17:59.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53ab7fe640 1 --2- 172.21.15.57:0/713990147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53ac0c3e40 0x7f53ac14b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:59.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53aaffd640 1 --2- 172.21.15.57:0/713990147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53ac0cb7d0 0x7f53ac1577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:59.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53abfff640 1 -- 172.21.15.57:0/713990147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53ac0cb7d0 msgr2=0x7f53ac1577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:59.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53abfff640 1 --2- 172.21.15.57:0/713990147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53ac0cb7d0 0x7f53ac1577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:59.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53abfff640 1 -- 172.21.15.57:0/713990147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53ac0c3e40 msgr2=0x7f53ac14b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:59.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53abfff640 1 --2- 172.21.15.57:0/713990147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53ac0c3e40 0x7f53ac14b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:59.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53abfff640 1 -- 172.21.15.57:0/713990147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a00094d0 con 0x7f53ac150830 2023-10-30T14:17:59.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53abfff640 1 --2- 172.21.15.57:0/713990147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53ac150830 0x7f53ac1f2c60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f53a4007d80 tx=0x7f53a400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:59.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f5393fff640 1 -- 172.21.15.57:0/713990147 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a4014420 con 0x7f53ac150830 2023-10-30T14:17:59.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f5393fff640 1 -- 172.21.15.57:0/713990147 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a4014580 con 0x7f53ac150830 2023-10-30T14:17:59.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f5393fff640 1 -- 172.21.15.57:0/713990147 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a4014930 con 0x7f53ac150830 2023-10-30T14:17:59.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53b1bb8640 1 -- 172.21.15.57:0/713990147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53ac147e60 con 0x7f53ac150830 2023-10-30T14:17:59.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.645+0000 7f53b1bb8640 1 -- 172.21.15.57:0/713990147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53ac1f65c0 con 0x7f53ac150830 2023-10-30T14:17:59.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.649+0000 7f53b1bb8640 1 -- 172.21.15.57:0/713990147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53ac1f6dc0 con 0x7f53ac150830 2023-10-30T14:17:59.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.649+0000 7f5393fff640 1 -- 172.21.15.57:0/713990147 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f53a4014a90 con 0x7f53ac150830 2023-10-30T14:17:59.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.649+0000 7f5393fff640 1 --2- 172.21.15.57:0/713990147 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f53980664d0 0x7f5398068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:17:59.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.649+0000 7f5393fff640 1 -- 172.21.15.57:0/713990147 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f53a4084790 con 0x7f53ac150830 2023-10-30T14:17:59.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.649+0000 7f53ab7fe640 1 --2- 172.21.15.57:0/713990147 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f53980664d0 0x7f5398068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:17:59.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.649+0000 7f53ab7fe640 1 --2- 172.21.15.57:0/713990147 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f53980664d0 0x7f5398068990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f539c007fd0 tx=0x7f539c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:17:59.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.653+0000 7f5393fff640 1 -- 172.21.15.57:0/713990147 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f53ac1f6dc0 con 0x7f53ac150830 2023-10-30T14:17:59.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.849+0000 7f53b1bb8640 1 -- 172.21.15.57:0/713990147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53ac1f6dc0 con 0x7f53ac150830 2023-10-30T14:17:59.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.849+0000 7f5393fff640 1 -- 172.21.15.57:0/713990147 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+5742 (secure 0 0 0) 0x7f53ac1f6dc0 con 0x7f53ac150830 2023-10-30T14:17:59.853 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:17:59.853 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:17:55.647291+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:17:59.853 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 31 2023-10-30T14:17:59.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.849+0000 7f53b1bb8640 1 -- 172.21.15.57:0/713990147 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f53980664d0 msgr2=0x7f5398068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:59.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.849+0000 7f53b1bb8640 1 --2- 172.21.15.57:0/713990147 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f53980664d0 0x7f5398068990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f539c007fd0 tx=0x7f539c009040 comp rx=0 tx=0).stop 2023-10-30T14:17:59.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.849+0000 7f53b1bb8640 1 -- 172.21.15.57:0/713990147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53ac150830 msgr2=0x7f53ac1f2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:17:59.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.849+0000 7f53b1bb8640 1 --2- 172.21.15.57:0/713990147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53ac150830 0x7f53ac1f2c60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f53a4007d80 tx=0x7f53a400e4b0 comp rx=0 tx=0).stop 2023-10-30T14:17:59.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.849+0000 7f53b1bb8640 1 -- 172.21.15.57:0/713990147 shutdown_connections 2023-10-30T14:17:59.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.849+0000 7f53b1bb8640 1 --2- 172.21.15.57:0/713990147 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f53980664d0 0x7f5398068990 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:59.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.849+0000 7f53b1bb8640 1 --2- 172.21.15.57:0/713990147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53ac150830 0x7f53ac1f2c60 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:59.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.849+0000 7f53b1bb8640 1 --2- 172.21.15.57:0/713990147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53ac0cb7d0 0x7f53ac1577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:59.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.849+0000 7f53b1bb8640 1 --2- 172.21.15.57:0/713990147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53ac0c3e40 0x7f53ac14b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:17:59.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.849+0000 7f53b1bb8640 1 -- 172.21.15.57:0/713990147 >> 172.21.15.57:0/713990147 conn(0x7f53ac0cac70 msgr2=0x7f53ac06bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:17:59.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.849+0000 7f53b1bb8640 1 -- 172.21.15.57:0/713990147 shutdown_connections 2023-10-30T14:17:59.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:17:59.849+0000 7f53b1bb8640 1 -- 172.21.15.57:0/713990147 wait complete. 2023-10-30T14:17:59.873 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:18:00.101 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:18:00.105 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:18:01.875 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:02.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.025+0000 7f2f59e46640 1 Processor -- start 2023-10-30T14:18:02.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.025+0000 7f2f59e46640 1 -- start start 2023-10-30T14:18:02.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.029+0000 7f2f59e46640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2f54070520 0x7f2f54070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:02.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.029+0000 7f2f59e46640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540c2310 con 0x7f2f54150830 2023-10-30T14:18:02.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.029+0000 7f2f59e46640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f54156710 con 0x7f2f540cb7d0 2023-10-30T14:18:02.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.029+0000 7f2f59e46640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f5406bc10 con 0x7f2f54070520 2023-10-30T14:18:02.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.029+0000 7f2f537fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f2f54150830 0x7f2f5406b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:58104/0 (socket says 172.21.15.57:58104) 2023-10-30T14:18:02.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.029+0000 7f2f537fe640 1 -- 172.21.15.57:0/936282604 learned_addr learned my addr 172.21.15.57:0/936282604 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:18:02.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.029+0000 7f2f53fff640 1 --2- 172.21.15.57:0/936282604 >> v2:172.21.15.57:3300/0 conn(0x7f2f54070520 0x7f2f54070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:02.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.029+0000 7f2f527fc640 1 -- 172.21.15.57:0/936282604 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763487166 0 0) 0x7f2f540c2310 con 0x7f2f54150830 2023-10-30T14:18:02.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.029+0000 7f2f527fc640 1 -- 172.21.15.57:0/936282604 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f3c003530 con 0x7f2f54150830 2023-10-30T14:18:02.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.029+0000 7f2f53fff640 1 -- 172.21.15.57:0/936282604 >> v1:172.21.15.88:6789/0 conn(0x7f2f540cb7d0 legacy=0x7f2f540c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:02.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.029+0000 7f2f53fff640 1 -- 172.21.15.57:0/936282604 >> v1:172.21.15.57:6789/0 conn(0x7f2f54150830 legacy=0x7f2f5406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:02.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.029+0000 7f2f53fff640 1 -- 172.21.15.57:0/936282604 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f540c4360 con 0x7f2f54070520 2023-10-30T14:18:02.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f53fff640 1 --2- 172.21.15.57:0/936282604 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f54070520 0x7f2f54070990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c009d90 tx=0x7f2f4c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c62c45d070893b89 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:02.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f527fc640 1 -- 172.21.15.57:0/936282604 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c0af600 con 0x7f2f54070520 2023-10-30T14:18:02.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f527fc640 1 -- 172.21.15.57:0/936282604 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4c0ab050 con 0x7f2f54070520 2023-10-30T14:18:02.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f527fc640 1 -- 172.21.15.57:0/936282604 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c0af980 con 0x7f2f54070520 2023-10-30T14:18:02.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f59e46640 1 -- 172.21.15.57:0/936282604 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f54070520 msgr2=0x7f2f54070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:02.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f59e46640 1 --2- 172.21.15.57:0/936282604 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f54070520 0x7f2f54070990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c009d90 tx=0x7f2f4c01f130 comp rx=0 tx=0).stop 2023-10-30T14:18:02.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f59e46640 1 -- 172.21.15.57:0/936282604 shutdown_connections 2023-10-30T14:18:02.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f59e46640 1 --2- 172.21.15.57:0/936282604 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f54070520 0x7f2f54070990 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:02.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f59e46640 1 -- 172.21.15.57:0/936282604 >> 172.21.15.57:0/936282604 conn(0x7f2f540cac70 msgr2=0x7f2f540cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:02.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f59e46640 1 -- 172.21.15.57:0/936282604 shutdown_connections 2023-10-30T14:18:02.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f59e46640 1 -- 172.21.15.57:0/936282604 wait complete. 2023-10-30T14:18:02.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f59e46640 1 Processor -- start 2023-10-30T14:18:02.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f59e46640 1 -- start start 2023-10-30T14:18:02.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f59e46640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f54070520 0x7f2f540ca490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:02.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f59e46640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f540cb7d0 0x7f2f540c5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:02.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f59e46640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f54150830 0x7f2f54143730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:02.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f59e46640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f54148d90 con 0x7f2f54150830 2023-10-30T14:18:02.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f59e46640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f54148760 con 0x7f2f54070520 2023-10-30T14:18:02.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f59e46640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f541491b0 con 0x7f2f540cb7d0 2023-10-30T14:18:02.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f53fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f54150830 0x7f2f54143730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:02.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.033+0000 7f2f53fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f54150830 0x7f2f54143730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40998/0 (socket says 172.21.15.57:40998) 2023-10-30T14:18:02.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.037+0000 7f2f53fff640 1 -- 172.21.15.57:0/2994040471 learned_addr learned my addr 172.21.15.57:0/2994040471 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:02.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.037+0000 7f2f52ffd640 1 --2- 172.21.15.57:0/2994040471 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f540cb7d0 0x7f2f540c5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:02.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.037+0000 7f2f537fe640 1 --2- 172.21.15.57:0/2994040471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f54070520 0x7f2f540ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:02.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.037+0000 7f2f53fff640 1 -- 172.21.15.57:0/2994040471 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f540cb7d0 msgr2=0x7f2f540c5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:02.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.037+0000 7f2f53fff640 1 --2- 172.21.15.57:0/2994040471 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f540cb7d0 0x7f2f540c5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:02.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.037+0000 7f2f53fff640 1 -- 172.21.15.57:0/2994040471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f54070520 msgr2=0x7f2f540ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:02.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.037+0000 7f2f53fff640 1 --2- 172.21.15.57:0/2994040471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f54070520 0x7f2f540ca490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:02.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.037+0000 7f2f53fff640 1 -- 172.21.15.57:0/2994040471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f4c009a70 con 0x7f2f54150830 2023-10-30T14:18:02.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.037+0000 7f2f537fe640 1 --2- 172.21.15.57:0/2994040471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f54070520 0x7f2f540ca490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:18:02.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.037+0000 7f2f53fff640 1 --2- 172.21.15.57:0/2994040471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f54150830 0x7f2f54143730 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c0b2040 tx=0x7f2f4c01bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:02.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.037+0000 7f2f3bfff640 1 -- 172.21.15.57:0/2994040471 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c009450 con 0x7f2f54150830 2023-10-30T14:18:02.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.037+0000 7f2f3bfff640 1 -- 172.21.15.57:0/2994040471 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4c0b9410 con 0x7f2f54150830 2023-10-30T14:18:02.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.037+0000 7f2f3bfff640 1 -- 172.21.15.57:0/2994040471 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c0b97c0 con 0x7f2f54150830 2023-10-30T14:18:02.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.037+0000 7f2f59e46640 1 -- 172.21.15.57:0/2994040471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f541412d0 con 0x7f2f54150830 2023-10-30T14:18:02.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.037+0000 7f2f59e46640 1 -- 172.21.15.57:0/2994040471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f541477a0 con 0x7f2f54150830 2023-10-30T14:18:02.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.037+0000 7f2f59e46640 1 -- 172.21.15.57:0/2994040471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f54156990 con 0x7f2f54150830 2023-10-30T14:18:02.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.037+0000 7f2f3bfff640 1 -- 172.21.15.57:0/2994040471 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2f4c0b6050 con 0x7f2f54150830 2023-10-30T14:18:02.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.037+0000 7f2f3bfff640 1 --2- 172.21.15.57:0/2994040471 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f400664a0 0x7f2f40068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:02.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.041+0000 7f2f3bfff640 1 -- 172.21.15.57:0/2994040471 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f2f4c1224e0 con 0x7f2f54150830 2023-10-30T14:18:02.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.041+0000 7f2f537fe640 1 --2- 172.21.15.57:0/2994040471 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f400664a0 0x7f2f40068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:02.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.041+0000 7f2f537fe640 1 --2- 172.21.15.57:0/2994040471 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f400664a0 0x7f2f40068960 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2f44005890 tx=0x7f2f44009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:02.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.041+0000 7f2f3bfff640 1 -- 172.21.15.57:0/2994040471 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2f54156990 con 0x7f2f54150830 2023-10-30T14:18:02.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.233+0000 7f2f59e46640 1 -- 172.21.15.57:0/2994040471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f54148ed0 con 0x7f2f54150830 2023-10-30T14:18:02.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.233+0000 7f2f3bfff640 1 -- 172.21.15.57:0/2994040471 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f2f54148ed0 con 0x7f2f54150830 2023-10-30T14:18:02.238 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:18:02.238 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:18:00.864216+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:02.239 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 32 2023-10-30T14:18:02.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.233+0000 7f2f59e46640 1 -- 172.21.15.57:0/2994040471 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f400664a0 msgr2=0x7f2f40068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:02.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.233+0000 7f2f59e46640 1 --2- 172.21.15.57:0/2994040471 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f400664a0 0x7f2f40068960 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2f44005890 tx=0x7f2f44009040 comp rx=0 tx=0).stop 2023-10-30T14:18:02.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.233+0000 7f2f59e46640 1 -- 172.21.15.57:0/2994040471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f54150830 msgr2=0x7f2f54143730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:02.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.233+0000 7f2f59e46640 1 --2- 172.21.15.57:0/2994040471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f54150830 0x7f2f54143730 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c0b2040 tx=0x7f2f4c01bed0 comp rx=0 tx=0).stop 2023-10-30T14:18:02.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.233+0000 7f2f59e46640 1 -- 172.21.15.57:0/2994040471 shutdown_connections 2023-10-30T14:18:02.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.233+0000 7f2f59e46640 1 --2- 172.21.15.57:0/2994040471 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f400664a0 0x7f2f40068960 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:02.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.233+0000 7f2f59e46640 1 --2- 172.21.15.57:0/2994040471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f54150830 0x7f2f54143730 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:02.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.233+0000 7f2f59e46640 1 --2- 172.21.15.57:0/2994040471 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f540cb7d0 0x7f2f540c5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:02.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.233+0000 7f2f59e46640 1 --2- 172.21.15.57:0/2994040471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f54070520 0x7f2f540ca490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:02.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.233+0000 7f2f59e46640 1 -- 172.21.15.57:0/2994040471 >> 172.21.15.57:0/2994040471 conn(0x7f2f540cac70 msgr2=0x7f2f540733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:02.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.233+0000 7f2f59e46640 1 -- 172.21.15.57:0/2994040471 shutdown_connections 2023-10-30T14:18:02.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:02.233+0000 7f2f59e46640 1 -- 172.21.15.57:0/2994040471 wait complete. 2023-10-30T14:18:02.259 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:18:04.260 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:04.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.413+0000 7fd5eb5c4640 1 Processor -- start 2023-10-30T14:18:04.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.413+0000 7fd5eb5c4640 1 -- start start 2023-10-30T14:18:04.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.413+0000 7fd5eb5c4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5e406b760 0x7fd5e415e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:04.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.413+0000 7fd5eb5c4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd5e406c420 0x7fd5e40c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:04.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.413+0000 7fd5eb5c4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e4071e00 con 0x7fd5e406bb40 2023-10-30T14:18:04.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.413+0000 7fd5eb5c4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5e4070020 con 0x7fd5e406c420 2023-10-30T14:18:04.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.413+0000 7fd5eb5c4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e415eb90 con 0x7fd5e406b760 2023-10-30T14:18:04.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.413+0000 7fd5e9b61640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd5e406c420 0x7fd5e40c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:04.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.413+0000 7fd5e9360640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5e406b760 0x7fd5e415e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:04.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.413+0000 7fd5e8b5f640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd5e406bb40 0x7fd5e4157a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:34830/0 (socket says 172.21.15.57:34830) 2023-10-30T14:18:04.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.413+0000 7fd5e9360640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5e406b760 0x7fd5e415e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:60824/0 (socket says 172.21.15.57:60824) 2023-10-30T14:18:04.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.413+0000 7fd5e9360640 1 -- 172.21.15.57:0/566010662 learned_addr learned my addr 172.21.15.57:0/566010662 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:04.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.413+0000 7fd5dbfff640 1 -- 172.21.15.57:0/566010662 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540491065 0 0) 0x7fd5e4071e00 con 0x7fd5e406bb40 2023-10-30T14:18:04.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.413+0000 7fd5dbfff640 1 -- 172.21.15.57:0/566010662 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5d0003530 con 0x7fd5e406bb40 2023-10-30T14:18:04.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.413+0000 7fd5e9360640 1 -- 172.21.15.57:0/566010662 >> v2:172.21.15.88:3300/0 conn(0x7fd5e406c420 msgr2=0x7fd5e40c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:04.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.413+0000 7fd5e9360640 1 --2- 172.21.15.57:0/566010662 >> v2:172.21.15.88:3300/0 conn(0x7fd5e406c420 0x7fd5e40c57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:04.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.413+0000 7fd5e9360640 1 -- 172.21.15.57:0/566010662 >> v1:172.21.15.88:6789/0 conn(0x7fd5e406bb40 legacy=0x7fd5e4157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:04.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.413+0000 7fd5e9360640 1 -- 172.21.15.57:0/566010662 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e4070f10 con 0x7fd5e406b760 2023-10-30T14:18:04.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.413+0000 7fd5e9b61640 1 --2- 172.21.15.57:0/566010662 >> v2:172.21.15.88:3300/0 conn(0x7fd5e406c420 0x7fd5e40c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:18:04.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.417+0000 7fd5e9360640 1 --2- 172.21.15.57:0/566010662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5e406b760 0x7fd5e415e650 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0009870 tx=0x7fd5e001c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68f1aacd0043fa26 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:04.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.417+0000 7fd5dbfff640 1 -- 172.21.15.57:0/566010662 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e00aa670 con 0x7fd5e406b760 2023-10-30T14:18:04.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.417+0000 7fd5dbfff640 1 -- 172.21.15.57:0/566010662 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e00aa7d0 con 0x7fd5e406b760 2023-10-30T14:18:04.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.417+0000 7fd5dbfff640 1 -- 172.21.15.57:0/566010662 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e00aab60 con 0x7fd5e406b760 2023-10-30T14:18:04.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.417+0000 7fd5eb5c4640 1 -- 172.21.15.57:0/566010662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5e406b760 msgr2=0x7fd5e415e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:04.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.417+0000 7fd5eb5c4640 1 --2- 172.21.15.57:0/566010662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5e406b760 0x7fd5e415e650 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0009870 tx=0x7fd5e001c510 comp rx=0 tx=0).stop 2023-10-30T14:18:04.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.417+0000 7fd5eb5c4640 1 -- 172.21.15.57:0/566010662 shutdown_connections 2023-10-30T14:18:04.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.417+0000 7fd5eb5c4640 1 --2- 172.21.15.57:0/566010662 >> v2:172.21.15.88:3300/0 conn(0x7fd5e406c420 0x7fd5e40c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:04.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.417+0000 7fd5eb5c4640 1 --2- 172.21.15.57:0/566010662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5e406b760 0x7fd5e415e650 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:04.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.417+0000 7fd5eb5c4640 1 -- 172.21.15.57:0/566010662 >> 172.21.15.57:0/566010662 conn(0x7fd5e40cac70 msgr2=0x7fd5e40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:04.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.417+0000 7fd5eb5c4640 1 -- 172.21.15.57:0/566010662 shutdown_connections 2023-10-30T14:18:04.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.417+0000 7fd5eb5c4640 1 -- 172.21.15.57:0/566010662 wait complete. 2023-10-30T14:18:04.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5eb5c4640 1 Processor -- start 2023-10-30T14:18:04.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5eb5c4640 1 -- start start 2023-10-30T14:18:04.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5eb5c4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5e406b760 0x7fd5e4161360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:04.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5eb5c4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e406bb40 0x7fd5e415ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:04.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5eb5c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5e406c420 0x7fd5e415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:04.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5eb5c4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e415ba40 con 0x7fd5e406bb40 2023-10-30T14:18:04.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5eb5c4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e41f6980 con 0x7fd5e406c420 2023-10-30T14:18:04.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5eb5c4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e415bc90 con 0x7fd5e406b760 2023-10-30T14:18:04.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5e8b5f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e406bb40 0x7fd5e415ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:04.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5e9360640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5e406b760 0x7fd5e4161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:04.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5e9360640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5e406b760 0x7fd5e4161360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:60826/0 (socket says 172.21.15.57:60826) 2023-10-30T14:18:04.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5e9360640 1 -- 172.21.15.57:0/3872911310 learned_addr learned my addr 172.21.15.57:0/3872911310 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:04.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5e8b5f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e406bb40 0x7fd5e415ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43266/0 (socket says 172.21.15.57:43266) 2023-10-30T14:18:04.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5e9b61640 1 --2- 172.21.15.57:0/3872911310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5e406c420 0x7fd5e415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:04.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5e8b5f640 1 -- 172.21.15.57:0/3872911310 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5e406b760 msgr2=0x7fd5e4161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:04.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5e8b5f640 1 --2- 172.21.15.57:0/3872911310 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5e406b760 0x7fd5e4161360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:04.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5e8b5f640 1 -- 172.21.15.57:0/3872911310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5e406c420 msgr2=0x7fd5e415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:04.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5e8b5f640 1 --2- 172.21.15.57:0/3872911310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5e406c420 0x7fd5e415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:04.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5e8b5f640 1 -- 172.21.15.57:0/3872911310 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5dc009a70 con 0x7fd5e406bb40 2023-10-30T14:18:04.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5e8b5f640 1 --2- 172.21.15.57:0/3872911310 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e406bb40 0x7fd5e415ca50 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd5e406bf60 tx=0x7fd5d400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:04.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5d9ffb640 1 -- 172.21.15.57:0/3872911310 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d40074e0 con 0x7fd5e406bb40 2023-10-30T14:18:04.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5d9ffb640 1 -- 172.21.15.57:0/3872911310 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d4007660 con 0x7fd5e406bb40 2023-10-30T14:18:04.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5d9ffb640 1 -- 172.21.15.57:0/3872911310 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4004a30 con 0x7fd5e406bb40 2023-10-30T14:18:04.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5eb5c4640 1 -- 172.21.15.57:0/3872911310 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5e00094d0 con 0x7fd5e406bb40 2023-10-30T14:18:04.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.421+0000 7fd5eb5c4640 1 -- 172.21.15.57:0/3872911310 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e415c250 con 0x7fd5e406bb40 2023-10-30T14:18:04.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.425+0000 7fd5eb5c4640 1 -- 172.21.15.57:0/3872911310 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e404e170 con 0x7fd5e406bb40 2023-10-30T14:18:04.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.425+0000 7fd5d9ffb640 1 -- 172.21.15.57:0/3872911310 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd5d4004b90 con 0x7fd5e406bb40 2023-10-30T14:18:04.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.425+0000 7fd5d9ffb640 1 --2- 172.21.15.57:0/3872911310 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd5c80664d0 0x7fd5c8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:04.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.425+0000 7fd5d9ffb640 1 -- 172.21.15.57:0/3872911310 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7fd5d40839b0 con 0x7fd5e406bb40 2023-10-30T14:18:04.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.425+0000 7fd5e9360640 1 --2- 172.21.15.57:0/3872911310 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd5c80664d0 0x7fd5c8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:04.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.425+0000 7fd5e9360640 1 --2- 172.21.15.57:0/3872911310 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd5c80664d0 0x7fd5c8068990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd5e001c550 tx=0x7fd5e00b5040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:04.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.429+0000 7fd5d9ffb640 1 -- 172.21.15.57:0/3872911310 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd5d4050cc0 con 0x7fd5e406bb40 2023-10-30T14:18:04.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.621+0000 7fd5eb5c4640 1 -- 172.21.15.57:0/3872911310 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5e4155e40 con 0x7fd5e406bb40 2023-10-30T14:18:04.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.621+0000 7fd5d9ffb640 1 -- 172.21.15.57:0/3872911310 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5739 (secure 0 0 0) 0x7fd5d40542e0 con 0x7fd5e406bb40 2023-10-30T14:18:04.627 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:18:04.627 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:18:00.864216+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:04.627 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 32 2023-10-30T14:18:04.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.625+0000 7fd5eb5c4640 1 -- 172.21.15.57:0/3872911310 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd5c80664d0 msgr2=0x7fd5c8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:04.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.625+0000 7fd5eb5c4640 1 --2- 172.21.15.57:0/3872911310 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd5c80664d0 0x7fd5c8068990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd5e001c550 tx=0x7fd5e00b5040 comp rx=0 tx=0).stop 2023-10-30T14:18:04.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.625+0000 7fd5eb5c4640 1 -- 172.21.15.57:0/3872911310 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e406bb40 msgr2=0x7fd5e415ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:04.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.625+0000 7fd5eb5c4640 1 --2- 172.21.15.57:0/3872911310 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e406bb40 0x7fd5e415ca50 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd5e406bf60 tx=0x7fd5d400c890 comp rx=0 tx=0).stop 2023-10-30T14:18:04.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.625+0000 7fd5eb5c4640 1 -- 172.21.15.57:0/3872911310 shutdown_connections 2023-10-30T14:18:04.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.625+0000 7fd5eb5c4640 1 --2- 172.21.15.57:0/3872911310 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd5c80664d0 0x7fd5c8068990 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:04.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.625+0000 7fd5eb5c4640 1 --2- 172.21.15.57:0/3872911310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5e406c420 0x7fd5e415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:04.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.625+0000 7fd5eb5c4640 1 --2- 172.21.15.57:0/3872911310 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e406bb40 0x7fd5e415ca50 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:04.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.625+0000 7fd5eb5c4640 1 --2- 172.21.15.57:0/3872911310 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5e406b760 0x7fd5e4161360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:04.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.625+0000 7fd5eb5c4640 1 -- 172.21.15.57:0/3872911310 >> 172.21.15.57:0/3872911310 conn(0x7fd5e40cac70 msgr2=0x7fd5e4156640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:04.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.625+0000 7fd5eb5c4640 1 -- 172.21.15.57:0/3872911310 shutdown_connections 2023-10-30T14:18:04.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:04.625+0000 7fd5eb5c4640 1 -- 172.21.15.57:0/3872911310 wait complete. 2023-10-30T14:18:04.649 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:18:06.650 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:06.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.797+0000 7f6b89202640 1 Processor -- start 2023-10-30T14:18:06.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.801+0000 7f6b89202640 1 -- start start 2023-10-30T14:18:06.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.801+0000 7f6b89202640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6b840724c0 0x7f6b84158a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:06.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.801+0000 7f6b89202640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b84158110 con 0x7f6b84157c90 2023-10-30T14:18:06.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.801+0000 7f6b89202640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b841583f0 con 0x7f6b840720e0 2023-10-30T14:18:06.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.801+0000 7f6b89202640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b840c1f80 con 0x7f6b840724c0 2023-10-30T14:18:06.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.801+0000 7f6b8359e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6b840724c0 0x7f6b84158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:06.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.801+0000 7f6b7bfff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f6b84157c90 0x7f6b84150f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:55896/0 (socket says 172.21.15.57:55896) 2023-10-30T14:18:06.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.801+0000 7f6b7bfff640 1 -- 172.21.15.57:0/895773948 learned_addr learned my addr 172.21.15.57:0/895773948 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:18:06.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.801+0000 7f6b8259c640 1 -- 172.21.15.57:0/895773948 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493839251 0 0) 0x7f6b84158110 con 0x7f6b84157c90 2023-10-30T14:18:06.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.801+0000 7f6b8259c640 1 -- 172.21.15.57:0/895773948 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b70003530 con 0x7f6b84157c90 2023-10-30T14:18:06.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.801+0000 7f6b8359e640 1 -- 172.21.15.57:0/895773948 >> v1:172.21.15.88:6789/0 conn(0x7f6b840720e0 legacy=0x7f6b84157580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:06.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.801+0000 7f6b8359e640 1 -- 172.21.15.57:0/895773948 >> v1:172.21.15.57:6789/0 conn(0x7f6b84157c90 legacy=0x7f6b84150f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:06.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.801+0000 7f6b8359e640 1 -- 172.21.15.57:0/895773948 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b840c2260 con 0x7f6b840724c0 2023-10-30T14:18:06.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.805+0000 7f6b8359e640 1 --2- 172.21.15.57:0/895773948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b840724c0 0x7f6b84158a70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c009d90 tx=0x7f6b7c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25c695cf8a463bd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:06.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.805+0000 7f6b8259c640 1 -- 172.21.15.57:0/895773948 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c0af600 con 0x7f6b840724c0 2023-10-30T14:18:06.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.805+0000 7f6b8259c640 1 -- 172.21.15.57:0/895773948 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7c0ab050 con 0x7f6b840724c0 2023-10-30T14:18:06.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.805+0000 7f6b8259c640 1 -- 172.21.15.57:0/895773948 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c0af960 con 0x7f6b840724c0 2023-10-30T14:18:06.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.805+0000 7f6b89202640 1 -- 172.21.15.57:0/895773948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b840724c0 msgr2=0x7f6b84158a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:06.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.805+0000 7f6b89202640 1 --2- 172.21.15.57:0/895773948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b840724c0 0x7f6b84158a70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c009d90 tx=0x7f6b7c01f130 comp rx=0 tx=0).stop 2023-10-30T14:18:06.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.805+0000 7f6b89202640 1 -- 172.21.15.57:0/895773948 shutdown_connections 2023-10-30T14:18:06.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.805+0000 7f6b89202640 1 --2- 172.21.15.57:0/895773948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b840724c0 0x7f6b84158a70 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:06.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.805+0000 7f6b89202640 1 -- 172.21.15.57:0/895773948 >> 172.21.15.57:0/895773948 conn(0x7f6b8406d810 msgr2=0x7f6b8406dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:06.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.805+0000 7f6b89202640 1 -- 172.21.15.57:0/895773948 shutdown_connections 2023-10-30T14:18:06.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.805+0000 7f6b89202640 1 -- 172.21.15.57:0/895773948 wait complete. 2023-10-30T14:18:06.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.805+0000 7f6b89202640 1 Processor -- start 2023-10-30T14:18:06.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b89202640 1 -- start start 2023-10-30T14:18:06.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b89202640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6b840720e0 0x7f6b84155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:06.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b89202640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b840724c0 0x7f6b840c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:06.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b89202640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b84157c90 0x7f6b841ef0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:06.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b89202640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b840c5400 con 0x7f6b84157c90 2023-10-30T14:18:06.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b89202640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b841eebe0 con 0x7f6b840724c0 2023-10-30T14:18:06.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b89202640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b840c56a0 con 0x7f6b840720e0 2023-10-30T14:18:06.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b8359e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b84157c90 0x7f6b841ef0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:06.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b8359e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b84157c90 0x7f6b841ef0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43284/0 (socket says 172.21.15.57:43284) 2023-10-30T14:18:06.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b8359e640 1 -- 172.21.15.57:0/651159320 learned_addr learned my addr 172.21.15.57:0/651159320 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:06.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b7bfff640 1 --2- 172.21.15.57:0/651159320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b840724c0 0x7f6b840c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:06.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b82d9d640 1 --2- 172.21.15.57:0/651159320 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6b840720e0 0x7f6b84155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:06.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b8359e640 1 -- 172.21.15.57:0/651159320 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6b840720e0 msgr2=0x7f6b84155880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:06.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b8359e640 1 --2- 172.21.15.57:0/651159320 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6b840720e0 0x7f6b84155880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:06.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b8359e640 1 -- 172.21.15.57:0/651159320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b840724c0 msgr2=0x7f6b840c69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:06.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b8359e640 1 --2- 172.21.15.57:0/651159320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b840724c0 0x7f6b840c69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:06.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b8359e640 1 -- 172.21.15.57:0/651159320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7c009a70 con 0x7f6b84157c90 2023-10-30T14:18:06.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b8359e640 1 --2- 172.21.15.57:0/651159320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b84157c90 0x7f6b841ef0e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c01cd60 tx=0x7f6b7c01bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:06.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b7b7fe640 1 -- 172.21.15.57:0/651159320 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c009450 con 0x7f6b84157c90 2023-10-30T14:18:06.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b7b7fe640 1 -- 172.21.15.57:0/651159320 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7c0039a0 con 0x7f6b84157c90 2023-10-30T14:18:06.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b7b7fe640 1 -- 172.21.15.57:0/651159320 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c003d10 con 0x7f6b84157c90 2023-10-30T14:18:06.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b89202640 1 -- 172.21.15.57:0/651159320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b840c5980 con 0x7f6b84157c90 2023-10-30T14:18:06.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b89202640 1 -- 172.21.15.57:0/651159320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b840c5f40 con 0x7f6b84157c90 2023-10-30T14:18:06.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.809+0000 7f6b89202640 1 -- 172.21.15.57:0/651159320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b841eee00 con 0x7f6b84157c90 2023-10-30T14:18:06.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.813+0000 7f6b7b7fe640 1 -- 172.21.15.57:0/651159320 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6b7c004030 con 0x7f6b84157c90 2023-10-30T14:18:06.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.813+0000 7f6b7b7fe640 1 --2- 172.21.15.57:0/651159320 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6b640664d0 0x7f6b64068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:06.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.813+0000 7f6b7b7fe640 1 -- 172.21.15.57:0/651159320 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f6b7c122680 con 0x7f6b84157c90 2023-10-30T14:18:06.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.813+0000 7f6b82d9d640 1 --2- 172.21.15.57:0/651159320 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6b640664d0 0x7f6b64068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:06.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.813+0000 7f6b82d9d640 1 --2- 172.21.15.57:0/651159320 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6b640664d0 0x7f6b64068990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6b74007fd0 tx=0x7f6b74009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:06.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:06.817+0000 7f6b7b7fe640 1 -- 172.21.15.57:0/651159320 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6b841eee00 con 0x7f6b84157c90 2023-10-30T14:18:07.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:07.013+0000 7f6b89202640 1 -- 172.21.15.57:0/651159320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b841eee00 con 0x7f6b84157c90 2023-10-30T14:18:07.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:07.013+0000 7f6b7b7fe640 1 -- 172.21.15.57:0/651159320 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f6b841eee00 con 0x7f6b84157c90 2023-10-30T14:18:07.020 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:18:07.020 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:18:05.907040+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:07.020 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 33 2023-10-30T14:18:07.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:07.017+0000 7f6b89202640 1 -- 172.21.15.57:0/651159320 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6b640664d0 msgr2=0x7f6b64068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:07.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:07.017+0000 7f6b89202640 1 --2- 172.21.15.57:0/651159320 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6b640664d0 0x7f6b64068990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6b74007fd0 tx=0x7f6b74009040 comp rx=0 tx=0).stop 2023-10-30T14:18:07.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:07.017+0000 7f6b89202640 1 -- 172.21.15.57:0/651159320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b84157c90 msgr2=0x7f6b841ef0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:07.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:07.017+0000 7f6b89202640 1 --2- 172.21.15.57:0/651159320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b84157c90 0x7f6b841ef0e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c01cd60 tx=0x7f6b7c01bed0 comp rx=0 tx=0).stop 2023-10-30T14:18:07.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:07.017+0000 7f6b89202640 1 -- 172.21.15.57:0/651159320 shutdown_connections 2023-10-30T14:18:07.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:07.017+0000 7f6b89202640 1 --2- 172.21.15.57:0/651159320 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6b640664d0 0x7f6b64068990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:07.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:07.017+0000 7f6b89202640 1 --2- 172.21.15.57:0/651159320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b84157c90 0x7f6b841ef0e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:07.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:07.017+0000 7f6b89202640 1 --2- 172.21.15.57:0/651159320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b840724c0 0x7f6b840c69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:07.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:07.017+0000 7f6b89202640 1 --2- 172.21.15.57:0/651159320 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6b840720e0 0x7f6b84155880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:07.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:07.017+0000 7f6b89202640 1 -- 172.21.15.57:0/651159320 >> 172.21.15.57:0/651159320 conn(0x7f6b8406d810 msgr2=0x7f6b84150ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:07.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:07.017+0000 7f6b89202640 1 -- 172.21.15.57:0/651159320 shutdown_connections 2023-10-30T14:18:07.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:07.017+0000 7f6b89202640 1 -- 172.21.15.57:0/651159320 wait complete. 2023-10-30T14:18:07.041 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=1, replacing gid=4266 2023-10-30T14:18:07.041 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.c 2023-10-30T14:18:07.884 INFO:tasks.ceph.mds.f.smithi088.stderr:2023-10-30T14:18:07.881+0000 7f1418be9640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:18:08.042 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T14:18:08.042 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T14:18:08.043 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T14:18:08.047 INFO:tasks.ceph.mds.c:Started 2023-10-30T14:18:08.047 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:08.143 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-30T14:18:08.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.197+0000 7f5a79ed6640 1 Processor -- start 2023-10-30T14:18:08.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.197+0000 7f5a79ed6640 1 -- start start 2023-10-30T14:18:08.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a79ed6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a7415cf90 0x7f5a740c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:08.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a79ed6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a74159f20 con 0x7f5a740c4a10 2023-10-30T14:18:08.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a79ed6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a740c47e0 con 0x7f5a74153c80 2023-10-30T14:18:08.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a79ed6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7415dec0 con 0x7f5a7415cf90 2023-10-30T14:18:08.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a72ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f5a740c4a10 0x7f5a740ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:55922/0 (socket says 172.21.15.57:55922) 2023-10-30T14:18:08.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a72ffd640 1 -- 172.21.15.57:0/3627718666 learned_addr learned my addr 172.21.15.57:0/3627718666 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:18:08.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a73fff640 1 --2- 172.21.15.57:0/3627718666 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a7415cf90 0x7f5a740c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:08.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a727fc640 1 -- 172.21.15.57:0/3627718666 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117911535 0 0) 0x7f5a74159f20 con 0x7f5a740c4a10 2023-10-30T14:18:08.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a727fc640 1 -- 172.21.15.57:0/3627718666 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a5c003530 con 0x7f5a740c4a10 2023-10-30T14:18:08.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a727fc640 1 -- 172.21.15.57:0/3627718666 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136702228 0 0) 0x7f5a740c47e0 con 0x7f5a74153c80 2023-10-30T14:18:08.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a727fc640 1 -- 172.21.15.57:0/3627718666 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a74159f20 con 0x7f5a74153c80 2023-10-30T14:18:08.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a727fc640 1 -- 172.21.15.57:0/3627718666 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1787953040 0 0) 0x7f5a5c003530 con 0x7f5a740c4a10 2023-10-30T14:18:08.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a727fc640 1 -- 172.21.15.57:0/3627718666 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a7415cf90 msgr2=0x7f5a740c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:08.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a727fc640 1 --2- 172.21.15.57:0/3627718666 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a7415cf90 0x7f5a740c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:08.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a727fc640 1 -- 172.21.15.57:0/3627718666 >> v1:172.21.15.88:6789/0 conn(0x7f5a74153c80 legacy=0x7f5a7415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:08.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a727fc640 1 -- 172.21.15.57:0/3627718666 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7415d620 con 0x7f5a740c4a10 2023-10-30T14:18:08.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a727fc640 1 -- 172.21.15.57:0/3627718666 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f5a68001680 con 0x7f5a740c4a10 2023-10-30T14:18:08.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a727fc640 1 -- 172.21.15.57:0/3627718666 >> v1:172.21.15.57:6789/0 conn(0x7f5a740c4a10 legacy=0x7f5a740ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:08.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a727fc640 1 --2- 172.21.15.57:0/3627718666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a5c003b40 0x7f5a5c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:08.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a727fc640 1 --2- 172.21.15.57:0/3627718666 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a7415cf90 0x7f5a5c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:08.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a727fc640 1 --2- 172.21.15.57:0/3627718666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a5c004d10 0x7f5a5c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:08.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a727fc640 1 -- 172.21.15.57:0/3627718666 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7415dec0 con 0x7f5a5c003b40 2023-10-30T14:18:08.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a727fc640 1 -- 172.21.15.57:0/3627718666 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c00ba40 con 0x7f5a5c004d10 2023-10-30T14:18:08.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a727fc640 1 -- 172.21.15.57:0/3627718666 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c00bc00 con 0x7f5a7415cf90 2023-10-30T14:18:08.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a737fe640 1 --2- 172.21.15.57:0/3627718666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a5c003b40 0x7f5a5c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:08.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a72ffd640 1 --2- 172.21.15.57:0/3627718666 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a7415cf90 0x7f5a5c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:08.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a73fff640 1 --2- 172.21.15.57:0/3627718666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a5c004d10 0x7f5a5c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:08.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a737fe640 1 -- 172.21.15.57:0/3627718666 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a7415cf90 msgr2=0x7f5a5c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:08.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a737fe640 1 --2- 172.21.15.57:0/3627718666 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a7415cf90 0x7f5a5c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:08.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a737fe640 1 -- 172.21.15.57:0/3627718666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a5c004d10 msgr2=0x7f5a5c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:08.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a737fe640 1 --2- 172.21.15.57:0/3627718666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a5c004d10 0x7f5a5c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:08.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.201+0000 7f5a737fe640 1 -- 172.21.15.57:0/3627718666 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c003530 con 0x7f5a5c003b40 2023-10-30T14:18:08.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.205+0000 7f5a737fe640 1 --2- 172.21.15.57:0/3627718666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a5c003b40 0x7f5a5c003f40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5a640018d0 tx=0x7f5a6401ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4062be5023ef3bc4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:08.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.205+0000 7f5a727fc640 1 -- 172.21.15.57:0/3627718666 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a640052d0 con 0x7f5a5c003b40 2023-10-30T14:18:08.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.205+0000 7f5a727fc640 1 -- 172.21.15.57:0/3627718666 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a64005450 con 0x7f5a5c003b40 2023-10-30T14:18:08.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.205+0000 7f5a727fc640 1 -- 172.21.15.57:0/3627718666 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a640a8930 con 0x7f5a5c003b40 2023-10-30T14:18:08.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.205+0000 7f5a79ed6640 1 -- 172.21.15.57:0/3627718666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a5c003b40 msgr2=0x7f5a5c003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:08.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.205+0000 7f5a79ed6640 1 --2- 172.21.15.57:0/3627718666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a5c003b40 0x7f5a5c003f40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5a640018d0 tx=0x7f5a6401ecd0 comp rx=0 tx=0).stop 2023-10-30T14:18:08.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.205+0000 7f5a73fff640 1 -- 172.21.15.57:0/3627718666 reap_dead start 2023-10-30T14:18:08.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.205+0000 7f5a79ed6640 1 -- 172.21.15.57:0/3627718666 shutdown_connections 2023-10-30T14:18:08.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.205+0000 7f5a79ed6640 1 -- 172.21.15.57:0/3627718666 >> 172.21.15.57:0/3627718666 conn(0x7f5a740715c0 msgr2=0x7f5a740719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:08.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.205+0000 7f5a79ed6640 1 -- 172.21.15.57:0/3627718666 shutdown_connections 2023-10-30T14:18:08.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.205+0000 7f5a79ed6640 1 -- 172.21.15.57:0/3627718666 wait complete. 2023-10-30T14:18:08.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.205+0000 7f5a79ed6640 1 Processor -- start 2023-10-30T14:18:08.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a79ed6640 1 -- start start 2023-10-30T14:18:08.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a79ed6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a5c003b40 0x7f5a74157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:08.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a79ed6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a741552e0 0x7f5a74156920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:08.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a79ed6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a74156e60 0x7f5a741455f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:08.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a79ed6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a74154ec0 con 0x7f5a74156e60 2023-10-30T14:18:08.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a79ed6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a74147000 con 0x7f5a5c003b40 2023-10-30T14:18:08.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a79ed6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a74155760 con 0x7f5a741552e0 2023-10-30T14:18:08.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a73fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a74156e60 0x7f5a741455f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:08.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a737fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a5c003b40 0x7f5a74157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:08.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a72ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a741552e0 0x7f5a74156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:08.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a73fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a74156e60 0x7f5a741455f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43314/0 (socket says 172.21.15.57:43314) 2023-10-30T14:18:08.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a73fff640 1 -- 172.21.15.57:0/882666361 learned_addr learned my addr 172.21.15.57:0/882666361 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:08.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a73fff640 1 -- 172.21.15.57:0/882666361 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a741552e0 msgr2=0x7f5a74156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:08.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a73fff640 1 --2- 172.21.15.57:0/882666361 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a741552e0 0x7f5a74156920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:08.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a73fff640 1 -- 172.21.15.57:0/882666361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a5c003b40 msgr2=0x7f5a74157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:08.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a73fff640 1 --2- 172.21.15.57:0/882666361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a5c003b40 0x7f5a74157810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:08.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a73fff640 1 -- 172.21.15.57:0/882666361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a74149580 con 0x7f5a74156e60 2023-10-30T14:18:08.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a73fff640 1 --2- 172.21.15.57:0/882666361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a74156e60 0x7f5a741455f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5a740c97d0 tx=0x7f5a6c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:08.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a5bfff640 1 -- 172.21.15.57:0/882666361 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c00e040 con 0x7f5a74156e60 2023-10-30T14:18:08.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a5bfff640 1 -- 172.21.15.57:0/882666361 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6c00f040 con 0x7f5a74156e60 2023-10-30T14:18:08.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a79ed6640 1 -- 172.21.15.57:0/882666361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a741572f0 con 0x7f5a74156e60 2023-10-30T14:18:08.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a5bfff640 1 -- 172.21.15.57:0/882666361 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c015430 con 0x7f5a74156e60 2023-10-30T14:18:08.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a79ed6640 1 -- 172.21.15.57:0/882666361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a741492a0 con 0x7f5a74156e60 2023-10-30T14:18:08.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.209+0000 7f5a79ed6640 1 -- 172.21.15.57:0/882666361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a74155000 con 0x7f5a74156e60 2023-10-30T14:18:08.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.213+0000 7f5a5bfff640 1 -- 172.21.15.57:0/882666361 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5a6c015590 con 0x7f5a74156e60 2023-10-30T14:18:08.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.213+0000 7f5a5bfff640 1 --2- 172.21.15.57:0/882666361 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5a600664a0 0x7f5a60068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:08.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.213+0000 7f5a5bfff640 1 -- 172.21.15.57:0/882666361 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f5a6c0845e0 con 0x7f5a74156e60 2023-10-30T14:18:08.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.213+0000 7f5a737fe640 1 --2- 172.21.15.57:0/882666361 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5a600664a0 0x7f5a60068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:08.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.213+0000 7f5a737fe640 1 --2- 172.21.15.57:0/882666361 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5a600664a0 0x7f5a60068960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5a640b0000 tx=0x7f5a640aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:08.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.217+0000 7f5a5bfff640 1 -- 172.21.15.57:0/882666361 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5a74155000 con 0x7f5a74156e60 2023-10-30T14:18:08.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.405+0000 7f5a79ed6640 1 -- 172.21.15.57:0/882666361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a74155000 con 0x7f5a74156e60 2023-10-30T14:18:08.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.405+0000 7f5a5bfff640 1 -- 172.21.15.57:0/882666361 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f5a74155000 con 0x7f5a74156e60 2023-10-30T14:18:08.412 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:18:08.412 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:18:05.907040+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:08.412 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 33 2023-10-30T14:18:08.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.409+0000 7f5a79ed6640 1 -- 172.21.15.57:0/882666361 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5a600664a0 msgr2=0x7f5a60068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:08.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.409+0000 7f5a79ed6640 1 --2- 172.21.15.57:0/882666361 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5a600664a0 0x7f5a60068960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5a640b0000 tx=0x7f5a640aa070 comp rx=0 tx=0).stop 2023-10-30T14:18:08.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.409+0000 7f5a79ed6640 1 -- 172.21.15.57:0/882666361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a74156e60 msgr2=0x7f5a741455f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:08.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.409+0000 7f5a79ed6640 1 --2- 172.21.15.57:0/882666361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a74156e60 0x7f5a741455f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5a740c97d0 tx=0x7f5a6c00f580 comp rx=0 tx=0).stop 2023-10-30T14:18:08.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.409+0000 7f5a79ed6640 1 -- 172.21.15.57:0/882666361 shutdown_connections 2023-10-30T14:18:08.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.409+0000 7f5a79ed6640 1 --2- 172.21.15.57:0/882666361 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5a600664a0 0x7f5a60068960 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:08.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.409+0000 7f5a79ed6640 1 --2- 172.21.15.57:0/882666361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a74156e60 0x7f5a741455f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:08.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.409+0000 7f5a79ed6640 1 --2- 172.21.15.57:0/882666361 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a741552e0 0x7f5a74156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:08.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.409+0000 7f5a79ed6640 1 --2- 172.21.15.57:0/882666361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a5c003b40 0x7f5a74157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:08.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.409+0000 7f5a79ed6640 1 -- 172.21.15.57:0/882666361 >> 172.21.15.57:0/882666361 conn(0x7f5a740715c0 msgr2=0x7f5a741528d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:08.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.409+0000 7f5a79ed6640 1 -- 172.21.15.57:0/882666361 shutdown_connections 2023-10-30T14:18:08.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:08.409+0000 7f5a79ed6640 1 -- 172.21.15.57:0/882666361 wait complete. 2023-10-30T14:18:08.432 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T14:18:10.433 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:10.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c528ea640 1 Processor -- start 2023-10-30T14:18:10.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c528ea640 1 -- start start 2023-10-30T14:18:10.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c528ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c4c151c20 0x7f5c4c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:10.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c528ea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5c4c0cb8b0 0x7f5c4c06b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:10.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c528ea640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c4c0c5030 con 0x7f5c4c151840 2023-10-30T14:18:10.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c528ea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c4c157830 con 0x7f5c4c0cb8b0 2023-10-30T14:18:10.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c528ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c157b10 con 0x7f5c4c151c20 2023-10-30T14:18:10.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4bfff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f5c4c151840 0x7f5c4c072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:55924/0 (socket says 172.21.15.57:55924) 2023-10-30T14:18:10.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4bfff640 1 -- 172.21.15.57:0/3808113098 learned_addr learned my addr 172.21.15.57:0/3808113098 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:18:10.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4b7fe640 1 --2- 172.21.15.57:0/3808113098 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c4c151c20 0x7f5c4c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:10.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c50e87640 1 --2- 172.21.15.57:0/3808113098 >> v2:172.21.15.88:3300/0 conn(0x7f5c4c0cb8b0 0x7f5c4c06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:10.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4affd640 1 -- 172.21.15.57:0/3808113098 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925928942 0 0) 0x7f5c4c0c5030 con 0x7f5c4c151840 2023-10-30T14:18:10.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4affd640 1 -- 172.21.15.57:0/3808113098 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c34003530 con 0x7f5c4c151840 2023-10-30T14:18:10.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4affd640 1 -- 172.21.15.57:0/3808113098 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2102206859 0 0) 0x7f5c34003530 con 0x7f5c4c151840 2023-10-30T14:18:10.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4affd640 1 -- 172.21.15.57:0/3808113098 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c4c151c20 msgr2=0x7f5c4c0c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:10.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4affd640 1 --2- 172.21.15.57:0/3808113098 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c4c151c20 0x7f5c4c0c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:10.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4affd640 1 -- 172.21.15.57:0/3808113098 >> v2:172.21.15.88:3300/0 conn(0x7f5c4c0cb8b0 msgr2=0x7f5c4c06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:10.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4affd640 1 --2- 172.21.15.57:0/3808113098 >> v2:172.21.15.88:3300/0 conn(0x7f5c4c0cb8b0 0x7f5c4c06b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:10.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4affd640 1 -- 172.21.15.57:0/3808113098 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c157df0 con 0x7f5c4c151840 2023-10-30T14:18:10.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4affd640 1 -- 172.21.15.57:0/3808113098 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f5c3c001680 con 0x7f5c4c151840 2023-10-30T14:18:10.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4affd640 1 -- 172.21.15.57:0/3808113098 >> v1:172.21.15.57:6789/0 conn(0x7f5c4c151840 legacy=0x7f5c4c072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:10.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4affd640 1 --2- 172.21.15.57:0/3808113098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c34003980 0x7f5c34003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:10.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4affd640 1 --2- 172.21.15.57:0/3808113098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c340047d0 0x7f5c34004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:10.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4affd640 1 --2- 172.21.15.57:0/3808113098 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c4c151c20 0x7f5c34005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:10.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4affd640 1 -- 172.21.15.57:0/3808113098 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c157830 con 0x7f5c340047d0 2023-10-30T14:18:10.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4affd640 1 -- 172.21.15.57:0/3808113098 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c157b10 con 0x7f5c34003980 2023-10-30T14:18:10.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4affd640 1 -- 172.21.15.57:0/3808113098 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3400b9c0 con 0x7f5c4c151c20 2023-10-30T14:18:10.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c50e87640 1 --2- 172.21.15.57:0/3808113098 >> v2:172.21.15.88:3300/0 conn(0x7f5c4c0cb8b0 0x7f5c4c06b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:18:10.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4b7fe640 1 --2- 172.21.15.57:0/3808113098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c340047d0 0x7f5c34004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:10.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.585+0000 7f5c4bfff640 1 --2- 172.21.15.57:0/3808113098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c34003980 0x7f5c34003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:10.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.589+0000 7f5c50e87640 1 --2- 172.21.15.57:0/3808113098 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c4c151c20 0x7f5c34005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:10.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.589+0000 7f5c4b7fe640 1 -- 172.21.15.57:0/3808113098 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c4c151c20 msgr2=0x7f5c34005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:10.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.589+0000 7f5c4b7fe640 1 --2- 172.21.15.57:0/3808113098 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c4c151c20 0x7f5c34005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:10.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.589+0000 7f5c4b7fe640 1 -- 172.21.15.57:0/3808113098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c34003980 msgr2=0x7f5c34003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:10.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.589+0000 7f5c4b7fe640 1 --2- 172.21.15.57:0/3808113098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c34003980 0x7f5c34003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:10.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.589+0000 7f5c4b7fe640 1 -- 172.21.15.57:0/3808113098 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c34003530 con 0x7f5c340047d0 2023-10-30T14:18:10.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.589+0000 7f5c4b7fe640 1 --2- 172.21.15.57:0/3808113098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c340047d0 0x7f5c34004c50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5c40009ab0 tx=0x7f5c4001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55cf2f3bd310b9f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:10.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.589+0000 7f5c4affd640 1 -- 172.21.15.57:0/3808113098 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c400a5540 con 0x7f5c340047d0 2023-10-30T14:18:10.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.589+0000 7f5c4affd640 1 -- 172.21.15.57:0/3808113098 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c400a56a0 con 0x7f5c340047d0 2023-10-30T14:18:10.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.589+0000 7f5c4affd640 1 -- 172.21.15.57:0/3808113098 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c400a5a50 con 0x7f5c340047d0 2023-10-30T14:18:10.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.589+0000 7f5c528ea640 1 -- 172.21.15.57:0/3808113098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c340047d0 msgr2=0x7f5c34004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:10.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.589+0000 7f5c528ea640 1 --2- 172.21.15.57:0/3808113098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c340047d0 0x7f5c34004c50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5c40009ab0 tx=0x7f5c4001ca30 comp rx=0 tx=0).stop 2023-10-30T14:18:10.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.589+0000 7f5c50e87640 1 -- 172.21.15.57:0/3808113098 reap_dead start 2023-10-30T14:18:10.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c528ea640 1 -- 172.21.15.57:0/3808113098 shutdown_connections 2023-10-30T14:18:10.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c528ea640 1 -- 172.21.15.57:0/3808113098 >> 172.21.15.57:0/3808113098 conn(0x7f5c4c0cac70 msgr2=0x7f5c4c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:10.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c528ea640 1 -- 172.21.15.57:0/3808113098 shutdown_connections 2023-10-30T14:18:10.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c528ea640 1 -- 172.21.15.57:0/3808113098 wait complete. 2023-10-30T14:18:10.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c528ea640 1 Processor -- start 2023-10-30T14:18:10.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c528ea640 1 -- start start 2023-10-30T14:18:10.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c528ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c340047d0 0x7f5c4c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:10.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c528ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c4c1e0da0 0x7f5c4c1dfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:10.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c528ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c4c1dffe0 0x7f5c4c1443a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:10.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c528ea640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c0ca970 con 0x7f5c4c1dffe0 2023-10-30T14:18:10.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c528ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c1e1400 con 0x7f5c4c1e0da0 2023-10-30T14:18:10.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c528ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c1e16c0 con 0x7f5c340047d0 2023-10-30T14:18:10.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c50e87640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c4c1dffe0 0x7f5c4c1443a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:10.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c50e87640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c4c1dffe0 0x7f5c4c1443a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43326/0 (socket says 172.21.15.57:43326) 2023-10-30T14:18:10.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c50e87640 1 -- 172.21.15.57:0/697016438 learned_addr learned my addr 172.21.15.57:0/697016438 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:10.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c4b7fe640 1 --2- 172.21.15.57:0/697016438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c4c1e0da0 0x7f5c4c1dfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:10.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c4bfff640 1 --2- 172.21.15.57:0/697016438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c340047d0 0x7f5c4c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:10.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c50e87640 1 -- 172.21.15.57:0/697016438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c340047d0 msgr2=0x7f5c4c06b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:10.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c50e87640 1 --2- 172.21.15.57:0/697016438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c340047d0 0x7f5c4c06b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:10.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c50e87640 1 -- 172.21.15.57:0/697016438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c4c1e0da0 msgr2=0x7f5c4c1dfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:10.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c50e87640 1 --2- 172.21.15.57:0/697016438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c4c1e0da0 0x7f5c4c1dfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:10.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.593+0000 7f5c50e87640 1 -- 172.21.15.57:0/697016438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c1e19a0 con 0x7f5c4c1dffe0 2023-10-30T14:18:10.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.597+0000 7f5c50e87640 1 --2- 172.21.15.57:0/697016438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c4c1dffe0 0x7f5c4c1443a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c06fa40 tx=0x7f5c440117d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:10.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.597+0000 7f5c48ff9640 1 -- 172.21.15.57:0/697016438 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4400e070 con 0x7f5c4c1dffe0 2023-10-30T14:18:10.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.597+0000 7f5c48ff9640 1 -- 172.21.15.57:0/697016438 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c440039a0 con 0x7f5c4c1dffe0 2023-10-30T14:18:10.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.597+0000 7f5c48ff9640 1 -- 172.21.15.57:0/697016438 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c44003b50 con 0x7f5c4c1dffe0 2023-10-30T14:18:10.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.597+0000 7f5c528ea640 1 -- 172.21.15.57:0/697016438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c4c0c9e90 con 0x7f5c4c1dffe0 2023-10-30T14:18:10.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.597+0000 7f5c528ea640 1 -- 172.21.15.57:0/697016438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c4c0ca450 con 0x7f5c4c1dffe0 2023-10-30T14:18:10.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.597+0000 7f5c528ea640 1 -- 172.21.15.57:0/697016438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c4c04e170 con 0x7f5c4c1dffe0 2023-10-30T14:18:10.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.597+0000 7f5c48ff9640 1 -- 172.21.15.57:0/697016438 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5c44003cb0 con 0x7f5c4c1dffe0 2023-10-30T14:18:10.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.597+0000 7f5c48ff9640 1 --2- 172.21.15.57:0/697016438 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5c380664a0 0x7f5c38068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:10.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.597+0000 7f5c48ff9640 1 -- 172.21.15.57:0/697016438 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f5c44084820 con 0x7f5c4c1dffe0 2023-10-30T14:18:10.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.597+0000 7f5c4bfff640 1 --2- 172.21.15.57:0/697016438 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5c380664a0 0x7f5c38068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:10.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.601+0000 7f5c4bfff640 1 --2- 172.21.15.57:0/697016438 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5c380664a0 0x7f5c38068960 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c0015e0 tx=0x7f5c3c007900 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:10.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.601+0000 7f5c48ff9640 1 -- 172.21.15.57:0/697016438 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5c44051a80 con 0x7f5c4c1dffe0 2023-10-30T14:18:10.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.797+0000 7f5c528ea640 1 -- 172.21.15.57:0/697016438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c4c0c4d30 con 0x7f5c4c1dffe0 2023-10-30T14:18:10.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.797+0000 7f5c48ff9640 1 -- 172.21.15.57:0/697016438 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f5c440550a0 con 0x7f5c4c1dffe0 2023-10-30T14:18:10.801 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:18:10.802 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5289,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:18:05.907040+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:10.802 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 34 2023-10-30T14:18:10.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.797+0000 7f5c528ea640 1 -- 172.21.15.57:0/697016438 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5c380664a0 msgr2=0x7f5c38068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:10.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.797+0000 7f5c528ea640 1 --2- 172.21.15.57:0/697016438 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5c380664a0 0x7f5c38068960 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c0015e0 tx=0x7f5c3c007900 comp rx=0 tx=0).stop 2023-10-30T14:18:10.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.797+0000 7f5c528ea640 1 -- 172.21.15.57:0/697016438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c4c1dffe0 msgr2=0x7f5c4c1443a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:10.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.797+0000 7f5c528ea640 1 --2- 172.21.15.57:0/697016438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c4c1dffe0 0x7f5c4c1443a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c06fa40 tx=0x7f5c440117d0 comp rx=0 tx=0).stop 2023-10-30T14:18:10.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.797+0000 7f5c528ea640 1 -- 172.21.15.57:0/697016438 shutdown_connections 2023-10-30T14:18:10.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.797+0000 7f5c528ea640 1 --2- 172.21.15.57:0/697016438 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5c380664a0 0x7f5c38068960 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:10.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.797+0000 7f5c528ea640 1 --2- 172.21.15.57:0/697016438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c4c1dffe0 0x7f5c4c1443a0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:10.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.797+0000 7f5c528ea640 1 --2- 172.21.15.57:0/697016438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c4c1e0da0 0x7f5c4c1dfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:10.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.797+0000 7f5c528ea640 1 --2- 172.21.15.57:0/697016438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c340047d0 0x7f5c4c06b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:10.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.797+0000 7f5c528ea640 1 -- 172.21.15.57:0/697016438 >> 172.21.15.57:0/697016438 conn(0x7f5c4c0cac70 msgr2=0x7f5c4c0c60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:10.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.797+0000 7f5c528ea640 1 -- 172.21.15.57:0/697016438 shutdown_connections 2023-10-30T14:18:10.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:10.797+0000 7f5c528ea640 1 -- 172.21.15.57:0/697016438 wait complete. 2023-10-30T14:18:10.822 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T14:18:10.823 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 28 secs before thrashing 2023-10-30T14:18:10.981 INFO:tasks.workunit.client.0.smithi057.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2023-10-30T14:18:12.100 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2023-10-30T14:18:12.101 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2023-10-30T14:18:12.101 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2023-10-30T14:18:12.101 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2023-10-30T14:18:12.101 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2023-10-30T14:18:12.101 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2023-10-30T14:18:12.102 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2023-10-30T14:18:12.102 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2023-10-30T14:18:12.102 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2023-10-30T14:18:12.102 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2023-10-30T14:18:12.102 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2023-10-30T14:18:12.103 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2023-10-30T14:18:12.103 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2023-10-30T14:18:12.103 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2023-10-30T14:18:12.103 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2023-10-30T14:18:12.103 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2023-10-30T14:18:12.104 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2023-10-30T14:18:12.104 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2023-10-30T14:18:12.104 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2023-10-30T14:18:12.104 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2023-10-30T14:18:12.104 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2023-10-30T14:18:12.104 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2023-10-30T14:18:12.105 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2023-10-30T14:18:12.105 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2023-10-30T14:18:12.105 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2023-10-30T14:18:12.105 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2023-10-30T14:18:12.105 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2023-10-30T14:18:12.106 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2023-10-30T14:18:12.106 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2023-10-30T14:18:12.106 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2023-10-30T14:18:12.106 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2023-10-30T14:18:12.106 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2023-10-30T14:18:12.106 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2023-10-30T14:18:12.107 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2023-10-30T14:18:12.107 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2023-10-30T14:18:12.107 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2023-10-30T14:18:12.107 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2023-10-30T14:18:12.107 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2023-10-30T14:18:12.108 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2023-10-30T14:18:12.108 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2023-10-30T14:18:12.108 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2023-10-30T14:18:12.108 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2023-10-30T14:18:12.108 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2023-10-30T14:18:12.108 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2023-10-30T14:18:12.109 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2023-10-30T14:18:12.109 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2023-10-30T14:18:12.109 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2023-10-30T14:18:12.109 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2023-10-30T14:18:12.109 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2023-10-30T14:18:12.110 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2023-10-30T14:18:12.110 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2023-10-30T14:18:12.110 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2023-10-30T14:18:12.110 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2023-10-30T14:18:12.110 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2023-10-30T14:18:12.110 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2023-10-30T14:18:12.111 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2023-10-30T14:18:12.111 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2023-10-30T14:18:12.111 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2023-10-30T14:18:12.111 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2023-10-30T14:18:12.111 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2023-10-30T14:18:12.111 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2023-10-30T14:18:12.112 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2023-10-30T14:18:12.112 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2023-10-30T14:18:12.112 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2023-10-30T14:18:12.112 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2023-10-30T14:18:12.112 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2023-10-30T14:18:12.113 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2023-10-30T14:18:12.113 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2023-10-30T14:18:12.113 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2023-10-30T14:18:12.113 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2023-10-30T14:18:12.113 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2023-10-30T14:18:12.113 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2023-10-30T14:18:12.114 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2023-10-30T14:18:12.114 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2023-10-30T14:18:12.114 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2023-10-30T14:18:12.114 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2023-10-30T14:18:12.114 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2023-10-30T14:18:12.115 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2023-10-30T14:18:12.115 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2023-10-30T14:18:12.115 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2023-10-30T14:18:12.115 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2023-10-30T14:18:12.384 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2023-10-30T14:18:12.385 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2023-10-30T14:18:12.385 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2023-10-30T14:18:12.385 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2023-10-30T14:18:12.385 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2023-10-30T14:18:12.385 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2023-10-30T14:18:12.386 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2023-10-30T14:18:12.386 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2023-10-30T14:18:12.386 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2023-10-30T14:18:12.386 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2023-10-30T14:18:12.386 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2023-10-30T14:18:12.387 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2023-10-30T14:18:12.387 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2023-10-30T14:18:12.387 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2023-10-30T14:18:12.387 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2023-10-30T14:18:12.387 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2023-10-30T14:18:12.388 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2023-10-30T14:18:12.388 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2023-10-30T14:18:12.388 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2023-10-30T14:18:12.388 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2023-10-30T14:18:12.388 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2023-10-30T14:18:12.388 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2023-10-30T14:18:12.389 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2023-10-30T14:18:12.389 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2023-10-30T14:18:12.389 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2023-10-30T14:18:12.389 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2023-10-30T14:18:12.389 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2023-10-30T14:18:12.390 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2023-10-30T14:18:12.390 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2023-10-30T14:18:12.390 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2023-10-30T14:18:12.390 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2023-10-30T14:18:12.390 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2023-10-30T14:18:12.391 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2023-10-30T14:18:12.391 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2023-10-30T14:18:12.391 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2023-10-30T14:18:12.391 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2023-10-30T14:18:12.391 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2023-10-30T14:18:12.392 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2023-10-30T14:18:12.392 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2023-10-30T14:18:12.392 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2023-10-30T14:18:12.392 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2023-10-30T14:18:12.392 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2023-10-30T14:18:12.393 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2023-10-30T14:18:12.393 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2023-10-30T14:18:12.393 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2023-10-30T14:18:12.393 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2023-10-30T14:18:12.393 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2023-10-30T14:18:12.393 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2023-10-30T14:18:12.394 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2023-10-30T14:18:12.394 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2023-10-30T14:18:12.394 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2023-10-30T14:18:12.394 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2023-10-30T14:18:12.394 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2023-10-30T14:18:12.395 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2023-10-30T14:18:12.395 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2023-10-30T14:18:12.395 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2023-10-30T14:18:12.395 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2023-10-30T14:18:12.395 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2023-10-30T14:18:12.396 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2023-10-30T14:18:12.396 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2023-10-30T14:18:12.396 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2023-10-30T14:18:12.396 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2023-10-30T14:18:12.396 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2023-10-30T14:18:12.397 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2023-10-30T14:18:12.397 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2023-10-30T14:18:12.397 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2023-10-30T14:18:12.397 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2023-10-30T14:18:12.397 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2023-10-30T14:18:12.398 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2023-10-30T14:18:12.398 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2023-10-30T14:18:12.398 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2023-10-30T14:18:12.398 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2023-10-30T14:18:12.398 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2023-10-30T14:18:12.398 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2023-10-30T14:18:12.399 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2023-10-30T14:18:12.399 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2023-10-30T14:18:12.399 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2023-10-30T14:18:12.399 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2023-10-30T14:18:12.399 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2023-10-30T14:18:12.400 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2023-10-30T14:18:12.400 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2023-10-30T14:18:13.898 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2023-10-30T14:18:13.899 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2023-10-30T14:18:13.899 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2023-10-30T14:18:13.899 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2023-10-30T14:18:13.899 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2023-10-30T14:18:13.900 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2023-10-30T14:18:13.900 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2023-10-30T14:18:13.900 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2023-10-30T14:18:13.900 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2023-10-30T14:18:13.900 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2023-10-30T14:18:13.901 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2023-10-30T14:18:13.901 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2023-10-30T14:18:13.901 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2023-10-30T14:18:13.901 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2023-10-30T14:18:13.901 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2023-10-30T14:18:13.902 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2023-10-30T14:18:13.902 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2023-10-30T14:18:13.902 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2023-10-30T14:18:13.902 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2023-10-30T14:18:13.902 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2023-10-30T14:18:13.903 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2023-10-30T14:18:13.903 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2023-10-30T14:18:13.903 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2023-10-30T14:18:13.903 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2023-10-30T14:18:13.903 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2023-10-30T14:18:13.903 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2023-10-30T14:18:13.904 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2023-10-30T14:18:13.904 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2023-10-30T14:18:13.904 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2023-10-30T14:18:13.904 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2023-10-30T14:18:13.904 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2023-10-30T14:18:13.905 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2023-10-30T14:18:13.905 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2023-10-30T14:18:13.905 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2023-10-30T14:18:13.905 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2023-10-30T14:18:13.905 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2023-10-30T14:18:13.906 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2023-10-30T14:18:13.906 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2023-10-30T14:18:13.906 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2023-10-30T14:18:13.906 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2023-10-30T14:18:13.906 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2023-10-30T14:18:13.907 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2023-10-30T14:18:13.907 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2023-10-30T14:18:13.907 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2023-10-30T14:18:13.907 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2023-10-30T14:18:13.907 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2023-10-30T14:18:13.907 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2023-10-30T14:18:13.908 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2023-10-30T14:18:13.908 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2023-10-30T14:18:13.908 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2023-10-30T14:18:13.908 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2023-10-30T14:18:13.908 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2023-10-30T14:18:13.909 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2023-10-30T14:18:13.909 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2023-10-30T14:18:13.909 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2023-10-30T14:18:13.909 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2023-10-30T14:18:13.909 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2023-10-30T14:18:13.910 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2023-10-30T14:18:13.910 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2023-10-30T14:18:13.910 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2023-10-30T14:18:13.910 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2023-10-30T14:18:13.910 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2023-10-30T14:18:13.911 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2023-10-30T14:18:13.911 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2023-10-30T14:18:13.911 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2023-10-30T14:18:13.911 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2023-10-30T14:18:13.911 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2023-10-30T14:18:13.912 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2023-10-30T14:18:13.912 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2023-10-30T14:18:13.912 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2023-10-30T14:18:13.912 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2023-10-30T14:18:13.912 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2023-10-30T14:18:13.913 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2023-10-30T14:18:13.913 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2023-10-30T14:18:13.913 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2023-10-30T14:18:13.913 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2023-10-30T14:18:13.913 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2023-10-30T14:18:13.914 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2023-10-30T14:18:13.914 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2023-10-30T14:18:13.914 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2023-10-30T14:18:13.914 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2023-10-30T14:18:14.151 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2023-10-30T14:18:14.152 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2023-10-30T14:18:14.152 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2023-10-30T14:18:14.152 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2023-10-30T14:18:14.152 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2023-10-30T14:18:14.153 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2023-10-30T14:18:14.153 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2023-10-30T14:18:14.153 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2023-10-30T14:18:14.153 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2023-10-30T14:18:14.153 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2023-10-30T14:18:14.154 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2023-10-30T14:18:14.154 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2023-10-30T14:18:14.154 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2023-10-30T14:18:14.154 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2023-10-30T14:18:14.154 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2023-10-30T14:18:14.155 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2023-10-30T14:18:14.155 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2023-10-30T14:18:14.155 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2023-10-30T14:18:14.155 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2023-10-30T14:18:14.155 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2023-10-30T14:18:14.156 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2023-10-30T14:18:14.156 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2023-10-30T14:18:14.156 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2023-10-30T14:18:14.156 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2023-10-30T14:18:14.156 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2023-10-30T14:18:14.157 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2023-10-30T14:18:14.157 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2023-10-30T14:18:14.157 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2023-10-30T14:18:14.157 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2023-10-30T14:18:14.157 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2023-10-30T14:18:14.157 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2023-10-30T14:18:14.158 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2023-10-30T14:18:14.158 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2023-10-30T14:18:14.158 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2023-10-30T14:18:14.158 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2023-10-30T14:18:14.158 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2023-10-30T14:18:14.159 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2023-10-30T14:18:14.159 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2023-10-30T14:18:14.159 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2023-10-30T14:18:14.159 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2023-10-30T14:18:14.159 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2023-10-30T14:18:14.160 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2023-10-30T14:18:14.160 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2023-10-30T14:18:14.160 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2023-10-30T14:18:14.160 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2023-10-30T14:18:14.160 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2023-10-30T14:18:14.160 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2023-10-30T14:18:14.161 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2023-10-30T14:18:14.161 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2023-10-30T14:18:14.161 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2023-10-30T14:18:14.161 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2023-10-30T14:18:14.161 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2023-10-30T14:18:14.161 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2023-10-30T14:18:14.162 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2023-10-30T14:18:14.162 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2023-10-30T14:18:14.162 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2023-10-30T14:18:14.162 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2023-10-30T14:18:14.162 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2023-10-30T14:18:14.163 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2023-10-30T14:18:14.163 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2023-10-30T14:18:14.163 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2023-10-30T14:18:14.163 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2023-10-30T14:18:14.163 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2023-10-30T14:18:14.164 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2023-10-30T14:18:14.164 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2023-10-30T14:18:14.164 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2023-10-30T14:18:14.164 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2023-10-30T14:18:14.164 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2023-10-30T14:18:14.165 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2023-10-30T14:18:14.165 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2023-10-30T14:18:14.165 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2023-10-30T14:18:14.165 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2023-10-30T14:18:14.165 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2023-10-30T14:18:14.166 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2023-10-30T14:18:14.166 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2023-10-30T14:18:14.166 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2023-10-30T14:18:14.166 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2023-10-30T14:18:14.166 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2023-10-30T14:18:14.166 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2023-10-30T14:18:14.167 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2023-10-30T14:18:14.167 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2023-10-30T14:18:14.700 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2023-10-30T14:18:14.700 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2023-10-30T14:18:14.700 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2023-10-30T14:18:14.700 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2023-10-30T14:18:14.701 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2023-10-30T14:18:14.701 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2023-10-30T14:18:14.701 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2023-10-30T14:18:14.701 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2023-10-30T14:18:14.701 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2023-10-30T14:18:14.702 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2023-10-30T14:18:14.702 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2023-10-30T14:18:14.702 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2023-10-30T14:18:14.702 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2023-10-30T14:18:14.702 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2023-10-30T14:18:14.703 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2023-10-30T14:18:14.703 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2023-10-30T14:18:14.703 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2023-10-30T14:18:14.703 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2023-10-30T14:18:14.703 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2023-10-30T14:18:14.704 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2023-10-30T14:18:14.704 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2023-10-30T14:18:14.704 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2023-10-30T14:18:14.704 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2023-10-30T14:18:14.705 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2023-10-30T14:18:14.705 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2023-10-30T14:18:14.705 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2023-10-30T14:18:14.705 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2023-10-30T14:18:14.705 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2023-10-30T14:18:14.706 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2023-10-30T14:18:14.706 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2023-10-30T14:18:14.706 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2023-10-30T14:18:14.706 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2023-10-30T14:18:14.707 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2023-10-30T14:18:14.707 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2023-10-30T14:18:14.707 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2023-10-30T14:18:14.707 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2023-10-30T14:18:14.707 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2023-10-30T14:18:14.708 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2023-10-30T14:18:14.708 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2023-10-30T14:18:14.708 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2023-10-30T14:18:14.708 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2023-10-30T14:18:14.709 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2023-10-30T14:18:14.709 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2023-10-30T14:18:14.709 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2023-10-30T14:18:14.709 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2023-10-30T14:18:14.710 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2023-10-30T14:18:14.710 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2023-10-30T14:18:14.710 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2023-10-30T14:18:14.710 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2023-10-30T14:18:14.711 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2023-10-30T14:18:14.711 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2023-10-30T14:18:14.711 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2023-10-30T14:18:14.711 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2023-10-30T14:18:14.712 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2023-10-30T14:18:14.712 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2023-10-30T14:18:14.712 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2023-10-30T14:18:14.712 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2023-10-30T14:18:14.713 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2023-10-30T14:18:14.713 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2023-10-30T14:18:14.713 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2023-10-30T14:18:14.713 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2023-10-30T14:18:14.713 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2023-10-30T14:18:14.714 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2023-10-30T14:18:14.714 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2023-10-30T14:18:14.714 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2023-10-30T14:18:14.714 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2023-10-30T14:18:14.715 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2023-10-30T14:18:14.715 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2023-10-30T14:18:14.715 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2023-10-30T14:18:14.715 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2023-10-30T14:18:14.716 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2023-10-30T14:18:14.716 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2023-10-30T14:18:14.716 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2023-10-30T14:18:14.717 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2023-10-30T14:18:14.717 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2023-10-30T14:18:14.717 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2023-10-30T14:18:14.717 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2023-10-30T14:18:14.718 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2023-10-30T14:18:14.718 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2023-10-30T14:18:14.718 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2023-10-30T14:18:14.719 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2023-10-30T14:18:14.979 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2023-10-30T14:18:14.979 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2023-10-30T14:18:14.979 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2023-10-30T14:18:14.979 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2023-10-30T14:18:14.980 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2023-10-30T14:18:14.980 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2023-10-30T14:18:14.980 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2023-10-30T14:18:14.980 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2023-10-30T14:18:14.980 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2023-10-30T14:18:14.980 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2023-10-30T14:18:14.981 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2023-10-30T14:18:14.981 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2023-10-30T14:18:14.981 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2023-10-30T14:18:14.981 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2023-10-30T14:18:14.981 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2023-10-30T14:18:14.981 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2023-10-30T14:18:14.982 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2023-10-30T14:18:14.982 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2023-10-30T14:18:14.982 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2023-10-30T14:18:14.982 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2023-10-30T14:18:14.982 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2023-10-30T14:18:14.983 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2023-10-30T14:18:14.983 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2023-10-30T14:18:14.983 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2023-10-30T14:18:14.983 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2023-10-30T14:18:14.983 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2023-10-30T14:18:14.984 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2023-10-30T14:18:14.984 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2023-10-30T14:18:14.984 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2023-10-30T14:18:14.984 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2023-10-30T14:18:14.984 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2023-10-30T14:18:14.984 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2023-10-30T14:18:14.985 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2023-10-30T14:18:14.985 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2023-10-30T14:18:14.985 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2023-10-30T14:18:14.985 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2023-10-30T14:18:14.985 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2023-10-30T14:18:14.985 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2023-10-30T14:18:14.986 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2023-10-30T14:18:14.986 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2023-10-30T14:18:14.986 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2023-10-30T14:18:14.986 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2023-10-30T14:18:14.986 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2023-10-30T14:18:14.986 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2023-10-30T14:18:14.987 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2023-10-30T14:18:14.987 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2023-10-30T14:18:14.987 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2023-10-30T14:18:14.987 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2023-10-30T14:18:14.987 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2023-10-30T14:18:14.987 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2023-10-30T14:18:14.988 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2023-10-30T14:18:14.988 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2023-10-30T14:18:14.988 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2023-10-30T14:18:14.988 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2023-10-30T14:18:14.988 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2023-10-30T14:18:14.989 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2023-10-30T14:18:14.989 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2023-10-30T14:18:14.989 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2023-10-30T14:18:14.989 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2023-10-30T14:18:14.989 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2023-10-30T14:18:14.989 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2023-10-30T14:18:14.990 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2023-10-30T14:18:14.990 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2023-10-30T14:18:14.990 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2023-10-30T14:18:14.990 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2023-10-30T14:18:14.990 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2023-10-30T14:18:14.990 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2023-10-30T14:18:14.991 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2023-10-30T14:18:14.991 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2023-10-30T14:18:14.991 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2023-10-30T14:18:14.991 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2023-10-30T14:18:14.991 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2023-10-30T14:18:14.992 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2023-10-30T14:18:14.992 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2023-10-30T14:18:14.992 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2023-10-30T14:18:14.992 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2023-10-30T14:18:14.992 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2023-10-30T14:18:14.992 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2023-10-30T14:18:14.993 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2023-10-30T14:18:14.993 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2023-10-30T14:18:14.993 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2023-10-30T14:18:17.082 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2023-10-30T14:18:17.082 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2023-10-30T14:18:17.082 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2023-10-30T14:18:17.083 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2023-10-30T14:18:17.083 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2023-10-30T14:18:17.083 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2023-10-30T14:18:17.083 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2023-10-30T14:18:17.083 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2023-10-30T14:18:17.084 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2023-10-30T14:18:17.084 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2023-10-30T14:18:17.084 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2023-10-30T14:18:17.084 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2023-10-30T14:18:17.085 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2023-10-30T14:18:17.085 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2023-10-30T14:18:17.085 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2023-10-30T14:18:17.085 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2023-10-30T14:18:17.085 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2023-10-30T14:18:17.086 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2023-10-30T14:18:17.086 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2023-10-30T14:18:17.086 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2023-10-30T14:18:17.086 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2023-10-30T14:18:17.086 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2023-10-30T14:18:17.087 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2023-10-30T14:18:17.087 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2023-10-30T14:18:17.087 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2023-10-30T14:18:17.087 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2023-10-30T14:18:17.087 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2023-10-30T14:18:17.088 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2023-10-30T14:18:17.088 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2023-10-30T14:18:17.088 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2023-10-30T14:18:17.088 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2023-10-30T14:18:17.089 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2023-10-30T14:18:17.089 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2023-10-30T14:18:17.089 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2023-10-30T14:18:17.089 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2023-10-30T14:18:17.089 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2023-10-30T14:18:17.090 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2023-10-30T14:18:17.090 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2023-10-30T14:18:17.090 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2023-10-30T14:18:17.090 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2023-10-30T14:18:17.090 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2023-10-30T14:18:17.091 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2023-10-30T14:18:17.091 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2023-10-30T14:18:17.091 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2023-10-30T14:18:17.091 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2023-10-30T14:18:17.091 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2023-10-30T14:18:17.092 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2023-10-30T14:18:17.092 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2023-10-30T14:18:17.092 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2023-10-30T14:18:17.092 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2023-10-30T14:18:17.092 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2023-10-30T14:18:17.093 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2023-10-30T14:18:17.093 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2023-10-30T14:18:17.093 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2023-10-30T14:18:17.093 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2023-10-30T14:18:17.094 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2023-10-30T14:18:17.094 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2023-10-30T14:18:17.094 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2023-10-30T14:18:17.094 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2023-10-30T14:18:17.094 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2023-10-30T14:18:17.095 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2023-10-30T14:18:17.095 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2023-10-30T14:18:17.095 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2023-10-30T14:18:17.095 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2023-10-30T14:18:17.095 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2023-10-30T14:18:17.096 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2023-10-30T14:18:17.096 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2023-10-30T14:18:17.096 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2023-10-30T14:18:17.096 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2023-10-30T14:18:17.096 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2023-10-30T14:18:17.097 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2023-10-30T14:18:17.097 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2023-10-30T14:18:17.097 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2023-10-30T14:18:17.098 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2023-10-30T14:18:17.098 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2023-10-30T14:18:17.098 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2023-10-30T14:18:17.099 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2023-10-30T14:18:17.099 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2023-10-30T14:18:17.100 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2023-10-30T14:18:17.100 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2023-10-30T14:18:17.100 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2023-10-30T14:18:17.101 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2023-10-30T14:18:18.772 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2023-10-30T14:18:18.773 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2023-10-30T14:18:18.773 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2023-10-30T14:18:18.773 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2023-10-30T14:18:18.773 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2023-10-30T14:18:18.773 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2023-10-30T14:18:18.774 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2023-10-30T14:18:18.774 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2023-10-30T14:18:18.774 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2023-10-30T14:18:18.774 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2023-10-30T14:18:18.775 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2023-10-30T14:18:18.775 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2023-10-30T14:18:18.775 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2023-10-30T14:18:18.775 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2023-10-30T14:18:18.775 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2023-10-30T14:18:18.776 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2023-10-30T14:18:18.776 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2023-10-30T14:18:18.776 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2023-10-30T14:18:18.776 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2023-10-30T14:18:18.776 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2023-10-30T14:18:18.777 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2023-10-30T14:18:18.777 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2023-10-30T14:18:18.777 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2023-10-30T14:18:18.777 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2023-10-30T14:18:18.777 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2023-10-30T14:18:18.778 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2023-10-30T14:18:18.778 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2023-10-30T14:18:18.778 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2023-10-30T14:18:18.778 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2023-10-30T14:18:18.778 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2023-10-30T14:18:18.779 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2023-10-30T14:18:18.779 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2023-10-30T14:18:18.779 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2023-10-30T14:18:18.779 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2023-10-30T14:18:18.779 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2023-10-30T14:18:18.780 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2023-10-30T14:18:18.780 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2023-10-30T14:18:18.780 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2023-10-30T14:18:18.780 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2023-10-30T14:18:18.780 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2023-10-30T14:18:18.781 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2023-10-30T14:18:18.781 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2023-10-30T14:18:18.781 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2023-10-30T14:18:18.781 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2023-10-30T14:18:18.781 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2023-10-30T14:18:18.782 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2023-10-30T14:18:18.782 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2023-10-30T14:18:18.782 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2023-10-30T14:18:18.782 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2023-10-30T14:18:18.782 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2023-10-30T14:18:18.783 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2023-10-30T14:18:18.783 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2023-10-30T14:18:18.783 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2023-10-30T14:18:18.783 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2023-10-30T14:18:18.784 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2023-10-30T14:18:18.784 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2023-10-30T14:18:18.784 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2023-10-30T14:18:18.784 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2023-10-30T14:18:18.784 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2023-10-30T14:18:18.784 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2023-10-30T14:18:18.785 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2023-10-30T14:18:18.785 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2023-10-30T14:18:18.785 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2023-10-30T14:18:18.785 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2023-10-30T14:18:18.786 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2023-10-30T14:18:18.786 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2023-10-30T14:18:18.786 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2023-10-30T14:18:18.786 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2023-10-30T14:18:18.786 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2023-10-30T14:18:18.787 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2023-10-30T14:18:18.787 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2023-10-30T14:18:18.787 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2023-10-30T14:18:18.787 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2023-10-30T14:18:18.787 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2023-10-30T14:18:18.787 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2023-10-30T14:18:18.788 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2023-10-30T14:18:18.788 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2023-10-30T14:18:18.788 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2023-10-30T14:18:18.788 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2023-10-30T14:18:18.788 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2023-10-30T14:18:18.789 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2023-10-30T14:18:20.036 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2023-10-30T14:18:20.036 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2023-10-30T14:18:20.036 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2023-10-30T14:18:20.037 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2023-10-30T14:18:20.037 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2023-10-30T14:18:20.037 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2023-10-30T14:18:20.037 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2023-10-30T14:18:20.037 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2023-10-30T14:18:20.038 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2023-10-30T14:18:20.038 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2023-10-30T14:18:20.038 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2023-10-30T14:18:20.038 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2023-10-30T14:18:20.038 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2023-10-30T14:18:20.039 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2023-10-30T14:18:20.039 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2023-10-30T14:18:20.039 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2023-10-30T14:18:20.039 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2023-10-30T14:18:20.039 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2023-10-30T14:18:20.040 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2023-10-30T14:18:20.040 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2023-10-30T14:18:20.040 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2023-10-30T14:18:20.040 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2023-10-30T14:18:20.040 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2023-10-30T14:18:20.041 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2023-10-30T14:18:20.041 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2023-10-30T14:18:20.041 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2023-10-30T14:18:20.041 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2023-10-30T14:18:20.041 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2023-10-30T14:18:20.041 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2023-10-30T14:18:20.042 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2023-10-30T14:18:20.042 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2023-10-30T14:18:20.042 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2023-10-30T14:18:20.042 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2023-10-30T14:18:20.043 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2023-10-30T14:18:20.043 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2023-10-30T14:18:20.043 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2023-10-30T14:18:20.043 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2023-10-30T14:18:20.043 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2023-10-30T14:18:20.043 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2023-10-30T14:18:20.044 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2023-10-30T14:18:20.044 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2023-10-30T14:18:20.044 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2023-10-30T14:18:20.044 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2023-10-30T14:18:20.044 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2023-10-30T14:18:20.045 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2023-10-30T14:18:20.045 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2023-10-30T14:18:20.045 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2023-10-30T14:18:20.045 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2023-10-30T14:18:20.045 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2023-10-30T14:18:20.046 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2023-10-30T14:18:20.046 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2023-10-30T14:18:20.046 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2023-10-30T14:18:20.046 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2023-10-30T14:18:20.046 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2023-10-30T14:18:20.046 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2023-10-30T14:18:20.047 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2023-10-30T14:18:20.047 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2023-10-30T14:18:20.047 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2023-10-30T14:18:20.047 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2023-10-30T14:18:20.047 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2023-10-30T14:18:20.048 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2023-10-30T14:18:20.048 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2023-10-30T14:18:20.048 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2023-10-30T14:18:20.048 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2023-10-30T14:18:20.048 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2023-10-30T14:18:20.049 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2023-10-30T14:18:20.049 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2023-10-30T14:18:20.049 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2023-10-30T14:18:20.049 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2023-10-30T14:18:20.049 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2023-10-30T14:18:20.049 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2023-10-30T14:18:20.050 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2023-10-30T14:18:20.050 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2023-10-30T14:18:20.050 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2023-10-30T14:18:20.050 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2023-10-30T14:18:20.050 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2023-10-30T14:18:20.051 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2023-10-30T14:18:20.051 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2023-10-30T14:18:20.051 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2023-10-30T14:18:20.051 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2023-10-30T14:18:20.051 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2023-10-30T14:18:22.566 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2023-10-30T14:18:22.566 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2023-10-30T14:18:22.566 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2023-10-30T14:18:22.566 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2023-10-30T14:18:22.567 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2023-10-30T14:18:22.567 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2023-10-30T14:18:22.567 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2023-10-30T14:18:22.567 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2023-10-30T14:18:22.567 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2023-10-30T14:18:22.568 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2023-10-30T14:18:22.568 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2023-10-30T14:18:22.568 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2023-10-30T14:18:22.568 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2023-10-30T14:18:22.568 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2023-10-30T14:18:22.569 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2023-10-30T14:18:22.569 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2023-10-30T14:18:22.569 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2023-10-30T14:18:22.569 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2023-10-30T14:18:22.569 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2023-10-30T14:18:22.569 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2023-10-30T14:18:22.570 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2023-10-30T14:18:22.570 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2023-10-30T14:18:22.570 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2023-10-30T14:18:22.570 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2023-10-30T14:18:22.570 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2023-10-30T14:18:22.571 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2023-10-30T14:18:22.571 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2023-10-30T14:18:22.571 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2023-10-30T14:18:22.571 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2023-10-30T14:18:22.571 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2023-10-30T14:18:22.572 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2023-10-30T14:18:22.572 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2023-10-30T14:18:22.572 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2023-10-30T14:18:22.572 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2023-10-30T14:18:22.572 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2023-10-30T14:18:22.573 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2023-10-30T14:18:22.573 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2023-10-30T14:18:22.573 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2023-10-30T14:18:22.573 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2023-10-30T14:18:22.573 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2023-10-30T14:18:22.573 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2023-10-30T14:18:22.574 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2023-10-30T14:18:22.574 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2023-10-30T14:18:22.574 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2023-10-30T14:18:22.574 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2023-10-30T14:18:22.574 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2023-10-30T14:18:22.575 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2023-10-30T14:18:22.575 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2023-10-30T14:18:22.575 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2023-10-30T14:18:22.575 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2023-10-30T14:18:22.575 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2023-10-30T14:18:22.576 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2023-10-30T14:18:22.576 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2023-10-30T14:18:22.576 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2023-10-30T14:18:22.576 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2023-10-30T14:18:22.576 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2023-10-30T14:18:22.576 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2023-10-30T14:18:22.577 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2023-10-30T14:18:22.577 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2023-10-30T14:18:22.577 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2023-10-30T14:18:22.577 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2023-10-30T14:18:22.577 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2023-10-30T14:18:22.578 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2023-10-30T14:18:22.578 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2023-10-30T14:18:22.578 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2023-10-30T14:18:22.578 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2023-10-30T14:18:22.578 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2023-10-30T14:18:22.578 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2023-10-30T14:18:22.579 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2023-10-30T14:18:22.579 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2023-10-30T14:18:22.579 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2023-10-30T14:18:22.579 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2023-10-30T14:18:22.579 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2023-10-30T14:18:22.580 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2023-10-30T14:18:22.580 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2023-10-30T14:18:22.580 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2023-10-30T14:18:22.580 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2023-10-30T14:18:22.580 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2023-10-30T14:18:22.581 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2023-10-30T14:18:22.581 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2023-10-30T14:18:22.581 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2023-10-30T14:18:24.430 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2023-10-30T14:18:24.430 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2023-10-30T14:18:24.430 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2023-10-30T14:18:24.430 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2023-10-30T14:18:24.431 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2023-10-30T14:18:24.431 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2023-10-30T14:18:24.431 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2023-10-30T14:18:24.431 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2023-10-30T14:18:24.431 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2023-10-30T14:18:24.432 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2023-10-30T14:18:24.432 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2023-10-30T14:18:24.432 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2023-10-30T14:18:24.432 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2023-10-30T14:18:24.432 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2023-10-30T14:18:24.433 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2023-10-30T14:18:24.433 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2023-10-30T14:18:24.433 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2023-10-30T14:18:24.433 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2023-10-30T14:18:24.433 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2023-10-30T14:18:24.434 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2023-10-30T14:18:24.434 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2023-10-30T14:18:24.434 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2023-10-30T14:18:24.434 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2023-10-30T14:18:24.434 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2023-10-30T14:18:24.434 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2023-10-30T14:18:24.435 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2023-10-30T14:18:24.435 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2023-10-30T14:18:24.435 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2023-10-30T14:18:24.435 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2023-10-30T14:18:24.435 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2023-10-30T14:18:24.436 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2023-10-30T14:18:24.436 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2023-10-30T14:18:24.436 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2023-10-30T14:18:24.436 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2023-10-30T14:18:24.436 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2023-10-30T14:18:24.436 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2023-10-30T14:18:24.437 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2023-10-30T14:18:24.437 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2023-10-30T14:18:24.437 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2023-10-30T14:18:24.437 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2023-10-30T14:18:24.437 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2023-10-30T14:18:24.438 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2023-10-30T14:18:24.438 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2023-10-30T14:18:24.438 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2023-10-30T14:18:24.438 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2023-10-30T14:18:24.438 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2023-10-30T14:18:24.439 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2023-10-30T14:18:24.439 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2023-10-30T14:18:24.439 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2023-10-30T14:18:24.439 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2023-10-30T14:18:24.439 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2023-10-30T14:18:24.439 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2023-10-30T14:18:24.440 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2023-10-30T14:18:24.440 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2023-10-30T14:18:24.440 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2023-10-30T14:18:24.440 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2023-10-30T14:18:24.440 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2023-10-30T14:18:24.441 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2023-10-30T14:18:24.441 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2023-10-30T14:18:24.441 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2023-10-30T14:18:24.441 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2023-10-30T14:18:24.441 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2023-10-30T14:18:24.442 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2023-10-30T14:18:24.442 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2023-10-30T14:18:24.442 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2023-10-30T14:18:24.442 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2023-10-30T14:18:24.442 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2023-10-30T14:18:24.442 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2023-10-30T14:18:24.443 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2023-10-30T14:18:24.443 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2023-10-30T14:18:24.443 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2023-10-30T14:18:24.443 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2023-10-30T14:18:24.443 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2023-10-30T14:18:24.444 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2023-10-30T14:18:24.444 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2023-10-30T14:18:24.444 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2023-10-30T14:18:24.444 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2023-10-30T14:18:24.444 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2023-10-30T14:18:24.445 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2023-10-30T14:18:24.445 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2023-10-30T14:18:24.445 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2023-10-30T14:18:25.929 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2023-10-30T14:18:25.930 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2023-10-30T14:18:25.930 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2023-10-30T14:18:25.930 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2023-10-30T14:18:25.930 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2023-10-30T14:18:25.931 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2023-10-30T14:18:25.931 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2023-10-30T14:18:25.931 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2023-10-30T14:18:25.931 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2023-10-30T14:18:25.931 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2023-10-30T14:18:25.932 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2023-10-30T14:18:25.932 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2023-10-30T14:18:25.932 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2023-10-30T14:18:25.932 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2023-10-30T14:18:25.932 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2023-10-30T14:18:25.932 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2023-10-30T14:18:25.933 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2023-10-30T14:18:25.933 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2023-10-30T14:18:25.933 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2023-10-30T14:18:25.933 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2023-10-30T14:18:25.933 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2023-10-30T14:18:25.934 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2023-10-30T14:18:25.934 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2023-10-30T14:18:25.934 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2023-10-30T14:18:25.934 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2023-10-30T14:18:25.934 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2023-10-30T14:18:25.935 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2023-10-30T14:18:25.935 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2023-10-30T14:18:25.935 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2023-10-30T14:18:25.935 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2023-10-30T14:18:25.935 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2023-10-30T14:18:25.935 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2023-10-30T14:18:25.936 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2023-10-30T14:18:25.936 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2023-10-30T14:18:25.936 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2023-10-30T14:18:25.936 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2023-10-30T14:18:25.936 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2023-10-30T14:18:25.937 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2023-10-30T14:18:25.937 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2023-10-30T14:18:25.937 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2023-10-30T14:18:25.937 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2023-10-30T14:18:25.937 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2023-10-30T14:18:25.937 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2023-10-30T14:18:25.938 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2023-10-30T14:18:25.938 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2023-10-30T14:18:25.938 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2023-10-30T14:18:25.938 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2023-10-30T14:18:25.938 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2023-10-30T14:18:25.939 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2023-10-30T14:18:25.939 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2023-10-30T14:18:25.939 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2023-10-30T14:18:25.939 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2023-10-30T14:18:25.939 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2023-10-30T14:18:25.939 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2023-10-30T14:18:25.940 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2023-10-30T14:18:25.940 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2023-10-30T14:18:25.940 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2023-10-30T14:18:25.940 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2023-10-30T14:18:25.940 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2023-10-30T14:18:25.941 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2023-10-30T14:18:25.941 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2023-10-30T14:18:25.941 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2023-10-30T14:18:25.941 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2023-10-30T14:18:25.941 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2023-10-30T14:18:25.942 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2023-10-30T14:18:25.942 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2023-10-30T14:18:25.942 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2023-10-30T14:18:25.942 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2023-10-30T14:18:25.942 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2023-10-30T14:18:25.942 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2023-10-30T14:18:25.943 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2023-10-30T14:18:25.943 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2023-10-30T14:18:25.943 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2023-10-30T14:18:25.943 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2023-10-30T14:18:25.943 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2023-10-30T14:18:25.944 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2023-10-30T14:18:25.944 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2023-10-30T14:18:25.944 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2023-10-30T14:18:25.944 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2023-10-30T14:18:25.944 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2023-10-30T14:18:25.944 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2023-10-30T14:18:27.849 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2023-10-30T14:18:27.850 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2023-10-30T14:18:27.850 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2023-10-30T14:18:27.850 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2023-10-30T14:18:27.850 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2023-10-30T14:18:27.851 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2023-10-30T14:18:27.851 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2023-10-30T14:18:27.851 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2023-10-30T14:18:27.851 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2023-10-30T14:18:27.852 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2023-10-30T14:18:27.852 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2023-10-30T14:18:27.852 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2023-10-30T14:18:27.852 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2023-10-30T14:18:27.852 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2023-10-30T14:18:27.853 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2023-10-30T14:18:27.853 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2023-10-30T14:18:27.853 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2023-10-30T14:18:27.853 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2023-10-30T14:18:27.853 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2023-10-30T14:18:27.854 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2023-10-30T14:18:27.854 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2023-10-30T14:18:27.854 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2023-10-30T14:18:27.854 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2023-10-30T14:18:27.855 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2023-10-30T14:18:27.855 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2023-10-30T14:18:27.855 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2023-10-30T14:18:27.855 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2023-10-30T14:18:27.856 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2023-10-30T14:18:27.856 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2023-10-30T14:18:27.856 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2023-10-30T14:18:27.856 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2023-10-30T14:18:27.856 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2023-10-30T14:18:27.857 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2023-10-30T14:18:27.857 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2023-10-30T14:18:27.857 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2023-10-30T14:18:27.857 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2023-10-30T14:18:27.858 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2023-10-30T14:18:27.858 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2023-10-30T14:18:27.858 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2023-10-30T14:18:27.858 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2023-10-30T14:18:27.859 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2023-10-30T14:18:27.859 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2023-10-30T14:18:27.859 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2023-10-30T14:18:27.859 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2023-10-30T14:18:27.860 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2023-10-30T14:18:27.860 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2023-10-30T14:18:27.860 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2023-10-30T14:18:27.860 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2023-10-30T14:18:27.860 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2023-10-30T14:18:27.861 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2023-10-30T14:18:27.861 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2023-10-30T14:18:27.861 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2023-10-30T14:18:27.861 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2023-10-30T14:18:27.861 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2023-10-30T14:18:27.861 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2023-10-30T14:18:27.862 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2023-10-30T14:18:27.862 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2023-10-30T14:18:27.862 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2023-10-30T14:18:27.862 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2023-10-30T14:18:27.862 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2023-10-30T14:18:27.863 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2023-10-30T14:18:27.863 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2023-10-30T14:18:27.863 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2023-10-30T14:18:27.863 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2023-10-30T14:18:27.863 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2023-10-30T14:18:27.864 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2023-10-30T14:18:27.864 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2023-10-30T14:18:27.864 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2023-10-30T14:18:27.864 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2023-10-30T14:18:27.864 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2023-10-30T14:18:27.864 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2023-10-30T14:18:27.865 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2023-10-30T14:18:27.865 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2023-10-30T14:18:27.865 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2023-10-30T14:18:27.865 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2023-10-30T14:18:27.865 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2023-10-30T14:18:27.866 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2023-10-30T14:18:27.866 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2023-10-30T14:18:27.866 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2023-10-30T14:18:27.866 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2023-10-30T14:18:27.866 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2023-10-30T14:18:28.561 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2023-10-30T14:18:28.562 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2023-10-30T14:18:28.562 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2023-10-30T14:18:28.562 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2023-10-30T14:18:28.563 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2023-10-30T14:18:28.563 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2023-10-30T14:18:28.563 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2023-10-30T14:18:28.563 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2023-10-30T14:18:28.563 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2023-10-30T14:18:28.564 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2023-10-30T14:18:28.564 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2023-10-30T14:18:28.564 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2023-10-30T14:18:28.564 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2023-10-30T14:18:28.564 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2023-10-30T14:18:28.565 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2023-10-30T14:18:28.565 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2023-10-30T14:18:28.565 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2023-10-30T14:18:28.565 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2023-10-30T14:18:28.565 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2023-10-30T14:18:28.566 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2023-10-30T14:18:28.566 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2023-10-30T14:18:28.566 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2023-10-30T14:18:28.566 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2023-10-30T14:18:28.566 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2023-10-30T14:18:28.567 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2023-10-30T14:18:28.567 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2023-10-30T14:18:28.567 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2023-10-30T14:18:28.567 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2023-10-30T14:18:28.567 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2023-10-30T14:18:28.568 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2023-10-30T14:18:28.568 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2023-10-30T14:18:28.568 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2023-10-30T14:18:28.568 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2023-10-30T14:18:28.569 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2023-10-30T14:18:28.569 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2023-10-30T14:18:28.569 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2023-10-30T14:18:28.569 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2023-10-30T14:18:28.569 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2023-10-30T14:18:28.570 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2023-10-30T14:18:28.570 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2023-10-30T14:18:28.570 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2023-10-30T14:18:28.570 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2023-10-30T14:18:28.570 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2023-10-30T14:18:28.571 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2023-10-30T14:18:28.571 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2023-10-30T14:18:28.571 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2023-10-30T14:18:28.571 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2023-10-30T14:18:28.571 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2023-10-30T14:18:28.572 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2023-10-30T14:18:28.572 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2023-10-30T14:18:28.572 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2023-10-30T14:18:28.572 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2023-10-30T14:18:28.572 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2023-10-30T14:18:28.573 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2023-10-30T14:18:28.573 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2023-10-30T14:18:28.573 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2023-10-30T14:18:28.573 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2023-10-30T14:18:28.573 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2023-10-30T14:18:28.574 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2023-10-30T14:18:28.574 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2023-10-30T14:18:28.574 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2023-10-30T14:18:28.574 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2023-10-30T14:18:28.574 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2023-10-30T14:18:28.575 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2023-10-30T14:18:28.575 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2023-10-30T14:18:28.575 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2023-10-30T14:18:28.575 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2023-10-30T14:18:28.575 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2023-10-30T14:18:28.576 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2023-10-30T14:18:28.576 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2023-10-30T14:18:28.576 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2023-10-30T14:18:28.576 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2023-10-30T14:18:28.576 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2023-10-30T14:18:28.577 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2023-10-30T14:18:28.577 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2023-10-30T14:18:28.577 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2023-10-30T14:18:28.577 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2023-10-30T14:18:28.577 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2023-10-30T14:18:28.578 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2023-10-30T14:18:28.578 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2023-10-30T14:18:28.578 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2023-10-30T14:18:29.447 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2023-10-30T14:18:29.448 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2023-10-30T14:18:29.448 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2023-10-30T14:18:29.448 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2023-10-30T14:18:29.448 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2023-10-30T14:18:29.449 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2023-10-30T14:18:29.449 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2023-10-30T14:18:29.449 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2023-10-30T14:18:29.449 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2023-10-30T14:18:29.449 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2023-10-30T14:18:29.450 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2023-10-30T14:18:29.450 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2023-10-30T14:18:29.450 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2023-10-30T14:18:29.450 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2023-10-30T14:18:29.450 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2023-10-30T14:18:29.451 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2023-10-30T14:18:29.451 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2023-10-30T14:18:29.451 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2023-10-30T14:18:29.451 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2023-10-30T14:18:29.451 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2023-10-30T14:18:29.452 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2023-10-30T14:18:29.452 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2023-10-30T14:18:29.452 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2023-10-30T14:18:29.452 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2023-10-30T14:18:29.452 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2023-10-30T14:18:29.453 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2023-10-30T14:18:29.453 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2023-10-30T14:18:29.453 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2023-10-30T14:18:29.453 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2023-10-30T14:18:29.453 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2023-10-30T14:18:29.454 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2023-10-30T14:18:29.454 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2023-10-30T14:18:29.454 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2023-10-30T14:18:29.454 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2023-10-30T14:18:29.455 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2023-10-30T14:18:29.455 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2023-10-30T14:18:29.455 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2023-10-30T14:18:29.455 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2023-10-30T14:18:29.455 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2023-10-30T14:18:29.456 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2023-10-30T14:18:29.456 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2023-10-30T14:18:29.456 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2023-10-30T14:18:29.456 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2023-10-30T14:18:29.456 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2023-10-30T14:18:29.457 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2023-10-30T14:18:29.457 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2023-10-30T14:18:29.457 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2023-10-30T14:18:29.457 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2023-10-30T14:18:29.457 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2023-10-30T14:18:29.458 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2023-10-30T14:18:29.458 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2023-10-30T14:18:29.458 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2023-10-30T14:18:29.458 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2023-10-30T14:18:29.458 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2023-10-30T14:18:29.459 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2023-10-30T14:18:29.459 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2023-10-30T14:18:29.459 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2023-10-30T14:18:29.459 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2023-10-30T14:18:29.459 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2023-10-30T14:18:29.460 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2023-10-30T14:18:29.460 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2023-10-30T14:18:29.460 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2023-10-30T14:18:29.460 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2023-10-30T14:18:29.460 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2023-10-30T14:18:29.461 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2023-10-30T14:18:29.461 INFO:tasks.workunit.client.0.smithi057.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2023-10-30T14:18:29.461 INFO:tasks.workunit.client.0.smithi057.stdout:Take third snapshot .snap/test3 2023-10-30T14:18:29.470 INFO:tasks.workunit.client.0.smithi057.stdout:Delete the snapshots... 2023-10-30T14:18:30.154 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:18:30.158 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:18:34.636 INFO:tasks.workunit.client.0.smithi057.stdout:.//.snap/test1 2023-10-30T14:18:34.636 INFO:tasks.workunit.client.0.smithi057.stdout:.//.snap/test2 2023-10-30T14:18:34.636 INFO:tasks.workunit.client.0.smithi057.stdout:.//.snap/test3 2023-10-30T14:18:34.646 INFO:tasks.workunit.client.0.smithi057.stdout:./274/.snap/snap-subdir-test 2023-10-30T14:18:34.909 INFO:tasks.workunit.client.0.smithi057.stdout:./166/.snap/snap-subdir-test 2023-10-30T14:18:34.919 INFO:tasks.workunit.client.0.smithi057.stdout:./256/.snap/snap-subdir-test 2023-10-30T14:18:34.928 INFO:tasks.workunit.client.0.smithi057.stdout:./247/.snap/snap-subdir-test 2023-10-30T14:18:34.939 INFO:tasks.workunit.client.0.smithi057.stdout:./248/.snap/snap-subdir-test 2023-10-30T14:18:34.947 INFO:tasks.workunit.client.0.smithi057.stdout:./147/.snap/snap-subdir-test 2023-10-30T14:18:34.957 INFO:tasks.workunit.client.0.smithi057.stdout:./151/.snap/snap-subdir-test 2023-10-30T14:18:34.964 INFO:tasks.workunit.client.0.smithi057.stdout:./170/.snap/snap-subdir-test 2023-10-30T14:18:34.972 INFO:tasks.workunit.client.0.smithi057.stdout:./167/.snap/snap-subdir-test 2023-10-30T14:18:34.980 INFO:tasks.workunit.client.0.smithi057.stdout:./140/.snap/snap-subdir-test 2023-10-30T14:18:34.989 INFO:tasks.workunit.client.0.smithi057.stdout:./142/.snap/snap-subdir-test 2023-10-30T14:18:34.996 INFO:tasks.workunit.client.0.smithi057.stdout:./150/.snap/snap-subdir-test 2023-10-30T14:18:35.005 INFO:tasks.workunit.client.0.smithi057.stdout:./273/.snap/snap-subdir-test 2023-10-30T14:18:35.531 INFO:tasks.workunit.client.0.smithi057.stdout:./269/.snap/snap-subdir-test 2023-10-30T14:18:35.537 INFO:tasks.workunit.client.0.smithi057.stdout:./105/.snap/snap-subdir-test 2023-10-30T14:18:35.550 INFO:tasks.workunit.client.0.smithi057.stdout:./210/.snap/snap-subdir-test 2023-10-30T14:18:35.558 INFO:tasks.workunit.client.0.smithi057.stdout:./265/.snap/snap-subdir-test 2023-10-30T14:18:35.569 INFO:tasks.workunit.client.0.smithi057.stdout:./218/.snap/snap-subdir-test 2023-10-30T14:18:35.580 INFO:tasks.workunit.client.0.smithi057.stdout:./290/.snap/snap-subdir-test 2023-10-30T14:18:35.591 INFO:tasks.workunit.client.0.smithi057.stdout:./149/.snap/snap-subdir-test 2023-10-30T14:18:35.602 INFO:tasks.workunit.client.0.smithi057.stdout:./109/.snap/snap-subdir-test 2023-10-30T14:18:35.609 INFO:tasks.workunit.client.0.smithi057.stdout:./292/.snap/snap-subdir-test 2023-10-30T14:18:35.617 INFO:tasks.workunit.client.0.smithi057.stdout:./242/.snap/snap-subdir-test 2023-10-30T14:18:35.625 INFO:tasks.workunit.client.0.smithi057.stdout:./158/.snap/snap-subdir-test 2023-10-30T14:18:35.633 INFO:tasks.workunit.client.0.smithi057.stdout:./243/.snap/snap-subdir-test 2023-10-30T14:18:35.642 INFO:tasks.workunit.client.0.smithi057.stdout:./106/.snap/snap-subdir-test 2023-10-30T14:18:35.653 INFO:tasks.workunit.client.0.smithi057.stdout:./250/.snap/snap-subdir-test 2023-10-30T14:18:35.663 INFO:tasks.workunit.client.0.smithi057.stdout:./221/.snap/snap-subdir-test 2023-10-30T14:18:35.675 INFO:tasks.workunit.client.0.smithi057.stdout:./244/.snap/snap-subdir-test 2023-10-30T14:18:35.688 INFO:tasks.workunit.client.0.smithi057.stdout:./258/.snap/snap-subdir-test 2023-10-30T14:18:35.699 INFO:tasks.workunit.client.0.smithi057.stdout:./268/.snap/snap-subdir-test 2023-10-30T14:18:35.704 INFO:tasks.workunit.client.0.smithi057.stdout:./189/.snap/snap-subdir-test 2023-10-30T14:18:35.748 INFO:tasks.workunit.client.0.smithi057.stdout:./285/.snap/snap-subdir-test 2023-10-30T14:18:35.757 INFO:tasks.workunit.client.0.smithi057.stdout:./208/.snap/snap-subdir-test 2023-10-30T14:18:35.768 INFO:tasks.workunit.client.0.smithi057.stdout:./288/.snap/snap-subdir-test 2023-10-30T14:18:35.780 INFO:tasks.workunit.client.0.smithi057.stdout:./209/.snap/snap-subdir-test 2023-10-30T14:18:36.414 INFO:tasks.workunit.client.0.smithi057.stdout:./253/.snap/snap-subdir-test 2023-10-30T14:18:36.426 INFO:tasks.workunit.client.0.smithi057.stdout:./249/.snap/snap-subdir-test 2023-10-30T14:18:36.432 INFO:tasks.workunit.client.0.smithi057.stdout:./133/.snap/snap-subdir-test 2023-10-30T14:18:36.445 INFO:tasks.workunit.client.0.smithi057.stdout:./291/.snap/snap-subdir-test 2023-10-30T14:18:36.457 INFO:tasks.workunit.client.0.smithi057.stdout:./226/.snap/snap-subdir-test 2023-10-30T14:18:36.472 INFO:tasks.workunit.client.0.smithi057.stdout:./173/.snap/snap-subdir-test 2023-10-30T14:18:36.486 INFO:tasks.workunit.client.0.smithi057.stdout:./174/.snap/snap-subdir-test 2023-10-30T14:18:36.500 INFO:tasks.workunit.client.0.smithi057.stdout:./299/.snap/snap-subdir-test 2023-10-30T14:18:36.514 INFO:tasks.workunit.client.0.smithi057.stdout:./262/.snap/snap-subdir-test 2023-10-30T14:18:36.526 INFO:tasks.workunit.client.0.smithi057.stdout:./231/.snap/snap-subdir-test 2023-10-30T14:18:36.539 INFO:tasks.workunit.client.0.smithi057.stdout:./214/.snap/snap-subdir-test 2023-10-30T14:18:36.551 INFO:tasks.workunit.client.0.smithi057.stdout:./281/.snap/snap-subdir-test 2023-10-30T14:18:36.562 INFO:tasks.workunit.client.0.smithi057.stdout:./289/.snap/snap-subdir-test 2023-10-30T14:18:36.569 INFO:tasks.workunit.client.0.smithi057.stdout:./283/.snap/snap-subdir-test 2023-10-30T14:18:36.578 INFO:tasks.workunit.client.0.smithi057.stdout:./232/.snap/snap-subdir-test 2023-10-30T14:18:36.587 INFO:tasks.workunit.client.0.smithi057.stdout:./298/.snap/snap-subdir-test 2023-10-30T14:18:36.596 INFO:tasks.workunit.client.0.smithi057.stdout:./233/.snap/snap-subdir-test 2023-10-30T14:18:36.608 INFO:tasks.workunit.client.0.smithi057.stdout:./155/.snap/snap-subdir-test 2023-10-30T14:18:36.613 INFO:tasks.workunit.client.0.smithi057.stdout:./114/.snap/snap-subdir-test 2023-10-30T14:18:36.617 INFO:tasks.workunit.client.0.smithi057.stdout:./159/.snap/snap-subdir-test 2023-10-30T14:18:36.627 INFO:tasks.workunit.client.0.smithi057.stdout:./187/.snap/snap-subdir-test 2023-10-30T14:18:36.634 INFO:tasks.workunit.client.0.smithi057.stdout:./185/.snap/snap-subdir-test 2023-10-30T14:18:36.643 INFO:tasks.workunit.client.0.smithi057.stdout:./293/.snap/snap-subdir-test 2023-10-30T14:18:36.651 INFO:tasks.workunit.client.0.smithi057.stdout:./228/.snap/snap-subdir-test 2023-10-30T14:18:36.660 INFO:tasks.workunit.client.0.smithi057.stdout:./120/.snap/snap-subdir-test 2023-10-30T14:18:36.667 INFO:tasks.workunit.client.0.smithi057.stdout:./110/.snap/snap-subdir-test 2023-10-30T14:18:36.671 INFO:tasks.workunit.client.0.smithi057.stdout:./267/.snap/snap-subdir-test 2023-10-30T14:18:36.679 INFO:tasks.workunit.client.0.smithi057.stdout:./202/.snap/snap-subdir-test 2023-10-30T14:18:36.688 INFO:tasks.workunit.client.0.smithi057.stdout:./137/.snap/snap-subdir-test 2023-10-30T14:18:36.695 INFO:tasks.workunit.client.0.smithi057.stdout:./246/.snap/snap-subdir-test 2023-10-30T14:18:36.703 INFO:tasks.workunit.client.0.smithi057.stdout:./219/.snap/snap-subdir-test 2023-10-30T14:18:36.711 INFO:tasks.workunit.client.0.smithi057.stdout:./198/.snap/snap-subdir-test 2023-10-30T14:18:36.724 INFO:tasks.workunit.client.0.smithi057.stdout:./136/.snap/snap-subdir-test 2023-10-30T14:18:36.941 INFO:tasks.workunit.client.0.smithi057.stdout:./270/.snap/snap-subdir-test 2023-10-30T14:18:37.640 INFO:tasks.workunit.client.0.smithi057.stdout:./125/.snap/snap-subdir-test 2023-10-30T14:18:37.652 INFO:tasks.workunit.client.0.smithi057.stdout:./194/.snap/snap-subdir-test 2023-10-30T14:18:37.661 INFO:tasks.workunit.client.0.smithi057.stdout:./119/.snap/snap-subdir-test 2023-10-30T14:18:37.670 INFO:tasks.workunit.client.0.smithi057.stdout:./297/.snap/snap-subdir-test 2023-10-30T14:18:37.963 INFO:tasks.workunit.client.0.smithi057.stdout:./230/.snap/snap-subdir-test 2023-10-30T14:18:37.974 INFO:tasks.workunit.client.0.smithi057.stdout:./260/.snap/snap-subdir-test 2023-10-30T14:18:37.983 INFO:tasks.workunit.client.0.smithi057.stdout:./157/.snap/snap-subdir-test 2023-10-30T14:18:37.995 INFO:tasks.workunit.client.0.smithi057.stdout:./172/.snap/snap-subdir-test 2023-10-30T14:18:38.004 INFO:tasks.workunit.client.0.smithi057.stdout:./277/.snap/snap-subdir-test 2023-10-30T14:18:38.012 INFO:tasks.workunit.client.0.smithi057.stdout:./186/.snap/snap-subdir-test 2023-10-30T14:18:38.023 INFO:tasks.workunit.client.0.smithi057.stdout:./237/.snap/snap-subdir-test 2023-10-30T14:18:38.033 INFO:tasks.workunit.client.0.smithi057.stdout:./190/.snap/snap-subdir-test 2023-10-30T14:18:38.824 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:38.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb2d90e640 1 Processor -- start 2023-10-30T14:18:38.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb2d90e640 1 -- start start 2023-10-30T14:18:38.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb2d90e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb28070520 0x7fdb280709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:38.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb2d90e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb280c2310 con 0x7fdb28150830 2023-10-30T14:18:38.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb2d90e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb28156710 con 0x7fdb280cb7d0 2023-10-30T14:18:38.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb2d90e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2806bc10 con 0x7fdb28070520 2023-10-30T14:18:38.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb26ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fdb28150830 0x7fdb2806b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:55132/0 (socket says 172.21.15.57:55132) 2023-10-30T14:18:38.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb26ffd640 1 -- 172.21.15.57:0/2689025141 learned_addr learned my addr 172.21.15.57:0/2689025141 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:18:38.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb277fe640 1 --2- 172.21.15.57:0/2689025141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb28070520 0x7fdb280709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:38.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb267fc640 1 -- 172.21.15.57:0/2689025141 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205938356 0 0) 0x7fdb280c2310 con 0x7fdb28150830 2023-10-30T14:18:38.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb267fc640 1 -- 172.21.15.57:0/2689025141 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb14005740 con 0x7fdb28150830 2023-10-30T14:18:38.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb267fc640 1 -- 172.21.15.57:0/2689025141 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3251000902 0 0) 0x7fdb14005740 con 0x7fdb28150830 2023-10-30T14:18:38.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb267fc640 1 -- 172.21.15.57:0/2689025141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb28070520 msgr2=0x7fdb280709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:38.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb267fc640 1 --2- 172.21.15.57:0/2689025141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb28070520 0x7fdb280709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:38.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb267fc640 1 -- 172.21.15.57:0/2689025141 >> v1:172.21.15.88:6789/0 conn(0x7fdb280cb7d0 legacy=0x7fdb280c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:38.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb267fc640 1 -- 172.21.15.57:0/2689025141 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb280c4360 con 0x7fdb28150830 2023-10-30T14:18:38.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb267fc640 1 -- 172.21.15.57:0/2689025141 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fdb18001680 con 0x7fdb28150830 2023-10-30T14:18:38.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb267fc640 1 -- 172.21.15.57:0/2689025141 >> v1:172.21.15.57:6789/0 conn(0x7fdb28150830 legacy=0x7fdb2806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:38.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb267fc640 1 --2- 172.21.15.57:0/2689025141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb28070520 0x7fdb14005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:38.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb267fc640 1 --2- 172.21.15.57:0/2689025141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb14006140 0x7fdb140069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:38.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb267fc640 1 --2- 172.21.15.57:0/2689025141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb14006f20 0x7fdb140073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:38.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb267fc640 1 -- 172.21.15.57:0/2689025141 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2806bc10 con 0x7fdb14006140 2023-10-30T14:18:38.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb267fc640 1 -- 172.21.15.57:0/2689025141 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1400dd10 con 0x7fdb14006f20 2023-10-30T14:18:38.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb267fc640 1 -- 172.21.15.57:0/2689025141 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1400ded0 con 0x7fdb28070520 2023-10-30T14:18:38.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb1ffff640 1 --2- 172.21.15.57:0/2689025141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb14006140 0x7fdb140069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:38.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb26ffd640 1 --2- 172.21.15.57:0/2689025141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb28070520 0x7fdb14005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:38.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb277fe640 1 --2- 172.21.15.57:0/2689025141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb14006f20 0x7fdb140073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:38.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb1ffff640 1 -- 172.21.15.57:0/2689025141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb28070520 msgr2=0x7fdb14005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:38.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb1ffff640 1 --2- 172.21.15.57:0/2689025141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb28070520 0x7fdb14005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:38.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb1ffff640 1 -- 172.21.15.57:0/2689025141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb14006f20 msgr2=0x7fdb140073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:38.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb1ffff640 1 --2- 172.21.15.57:0/2689025141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb14006f20 0x7fdb140073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:38.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.976+0000 7fdb1ffff640 1 -- 172.21.15.57:0/2689025141 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb14005740 con 0x7fdb14006140 2023-10-30T14:18:38.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.980+0000 7fdb1ffff640 1 --2- 172.21.15.57:0/2689025141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb14006140 0x7fdb140069e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fdb10001780 tx=0x7fdb1001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9381a21bfd164c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:38.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.980+0000 7fdb267fc640 1 -- 172.21.15.57:0/2689025141 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb100050d0 con 0x7fdb14006140 2023-10-30T14:18:38.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.980+0000 7fdb267fc640 1 -- 172.21.15.57:0/2689025141 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb100a7050 con 0x7fdb14006140 2023-10-30T14:18:38.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.980+0000 7fdb267fc640 1 -- 172.21.15.57:0/2689025141 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb10005490 con 0x7fdb14006140 2023-10-30T14:18:38.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.980+0000 7fdb2d90e640 1 -- 172.21.15.57:0/2689025141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb14006140 msgr2=0x7fdb140069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:38.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.980+0000 7fdb2d90e640 1 --2- 172.21.15.57:0/2689025141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb14006140 0x7fdb140069e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fdb10001780 tx=0x7fdb1001c400 comp rx=0 tx=0).stop 2023-10-30T14:18:38.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.980+0000 7fdb277fe640 1 -- 172.21.15.57:0/2689025141 reap_dead start 2023-10-30T14:18:38.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.980+0000 7fdb2d90e640 1 -- 172.21.15.57:0/2689025141 shutdown_connections 2023-10-30T14:18:38.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.980+0000 7fdb2d90e640 1 -- 172.21.15.57:0/2689025141 >> 172.21.15.57:0/2689025141 conn(0x7fdb280cac70 msgr2=0x7fdb280cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:38.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.980+0000 7fdb2d90e640 1 -- 172.21.15.57:0/2689025141 shutdown_connections 2023-10-30T14:18:38.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.980+0000 7fdb2d90e640 1 -- 172.21.15.57:0/2689025141 wait complete. 2023-10-30T14:18:38.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb2d90e640 1 Processor -- start 2023-10-30T14:18:38.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb2d90e640 1 -- start start 2023-10-30T14:18:38.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb2d90e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb14006140 0x7fdb280c5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:38.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb2d90e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb280c66f0 0x7fdb2814a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:38.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb2d90e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb2814af20 0x7fdb281473d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:38.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb2d90e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2814a2e0 con 0x7fdb2814af20 2023-10-30T14:18:38.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb2d90e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2814a5c0 con 0x7fdb14006140 2023-10-30T14:18:38.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb2d90e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb28152090 con 0x7fdb280c66f0 2023-10-30T14:18:38.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb277fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb2814af20 0x7fdb281473d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:38.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb277fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb2814af20 0x7fdb281473d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50818/0 (socket says 172.21.15.57:50818) 2023-10-30T14:18:38.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb277fe640 1 -- 172.21.15.57:0/1025155317 learned_addr learned my addr 172.21.15.57:0/1025155317 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:38.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb26ffd640 1 --2- 172.21.15.57:0/1025155317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb14006140 0x7fdb280c5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:38.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb1ffff640 1 --2- 172.21.15.57:0/1025155317 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb280c66f0 0x7fdb2814a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:38.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb277fe640 1 -- 172.21.15.57:0/1025155317 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb280c66f0 msgr2=0x7fdb2814a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:38.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb277fe640 1 --2- 172.21.15.57:0/1025155317 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb280c66f0 0x7fdb2814a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:38.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb277fe640 1 -- 172.21.15.57:0/1025155317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb14006140 msgr2=0x7fdb280c5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:38.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb277fe640 1 --2- 172.21.15.57:0/1025155317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb14006140 0x7fdb280c5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:38.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb277fe640 1 -- 172.21.15.57:0/1025155317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb280ca210 con 0x7fdb2814af20 2023-10-30T14:18:38.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb277fe640 1 --2- 172.21.15.57:0/1025155317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb2814af20 0x7fdb281473d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fdb281541f0 tx=0x7fdb200115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:38.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb24ff9640 1 -- 172.21.15.57:0/1025155317 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2000e070 con 0x7fdb2814af20 2023-10-30T14:18:38.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb24ff9640 1 -- 172.21.15.57:0/1025155317 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb20011e40 con 0x7fdb2814af20 2023-10-30T14:18:38.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb24ff9640 1 -- 172.21.15.57:0/1025155317 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb200039a0 con 0x7fdb2814af20 2023-10-30T14:18:38.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb2d90e640 1 -- 172.21.15.57:0/1025155317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb280ca480 con 0x7fdb2814af20 2023-10-30T14:18:38.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.984+0000 7fdb2d90e640 1 -- 172.21.15.57:0/1025155317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb281ea470 con 0x7fdb2814af20 2023-10-30T14:18:38.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.988+0000 7fdb2d90e640 1 -- 172.21.15.57:0/1025155317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb2814a700 con 0x7fdb2814af20 2023-10-30T14:18:39.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.988+0000 7fdb24ff9640 1 -- 172.21.15.57:0/1025155317 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdb20003b00 con 0x7fdb2814af20 2023-10-30T14:18:39.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.988+0000 7fdb24ff9640 1 --2- 172.21.15.57:0/1025155317 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdb080664a0 0x7fdb08068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:39.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.988+0000 7fdb24ff9640 1 -- 172.21.15.57:0/1025155317 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7fdb20084480 con 0x7fdb2814af20 2023-10-30T14:18:39.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.988+0000 7fdb26ffd640 1 --2- 172.21.15.57:0/1025155317 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdb080664a0 0x7fdb08068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:39.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.988+0000 7fdb26ffd640 1 --2- 172.21.15.57:0/1025155317 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdb080664a0 0x7fdb08068960 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdb18001ca0 tx=0x7fdb18007040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:39.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:38.992+0000 7fdb24ff9640 1 -- 172.21.15.57:0/1025155317 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdb2814a700 con 0x7fdb2814af20 2023-10-30T14:18:39.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.180+0000 7fdb2d90e640 1 -- 172.21.15.57:0/1025155317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb2814a700 con 0x7fdb2814af20 2023-10-30T14:18:39.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.180+0000 7fdb24ff9640 1 -- 172.21.15.57:0/1025155317 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fdb2814a700 con 0x7fdb2814af20 2023-10-30T14:18:39.187 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:18:39.187 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5289,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:18:05.907040+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:39.188 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 34 2023-10-30T14:18:39.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.184+0000 7fdb2d90e640 1 -- 172.21.15.57:0/1025155317 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdb080664a0 msgr2=0x7fdb08068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:39.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.184+0000 7fdb2d90e640 1 --2- 172.21.15.57:0/1025155317 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdb080664a0 0x7fdb08068960 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdb18001ca0 tx=0x7fdb18007040 comp rx=0 tx=0).stop 2023-10-30T14:18:39.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.184+0000 7fdb2d90e640 1 -- 172.21.15.57:0/1025155317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb2814af20 msgr2=0x7fdb281473d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:39.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.184+0000 7fdb2d90e640 1 --2- 172.21.15.57:0/1025155317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb2814af20 0x7fdb281473d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fdb281541f0 tx=0x7fdb200115a0 comp rx=0 tx=0).stop 2023-10-30T14:18:39.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.184+0000 7fdb2d90e640 1 -- 172.21.15.57:0/1025155317 shutdown_connections 2023-10-30T14:18:39.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.184+0000 7fdb2d90e640 1 --2- 172.21.15.57:0/1025155317 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdb080664a0 0x7fdb08068960 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:39.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.184+0000 7fdb2d90e640 1 --2- 172.21.15.57:0/1025155317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb2814af20 0x7fdb281473d0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:39.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.184+0000 7fdb2d90e640 1 --2- 172.21.15.57:0/1025155317 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb280c66f0 0x7fdb2814a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:39.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.184+0000 7fdb2d90e640 1 --2- 172.21.15.57:0/1025155317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb14006140 0x7fdb280c5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:39.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.184+0000 7fdb2d90e640 1 -- 172.21.15.57:0/1025155317 >> 172.21.15.57:0/1025155317 conn(0x7fdb280cac70 msgr2=0x7fdb280c36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:39.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.184+0000 7fdb2d90e640 1 -- 172.21.15.57:0/1025155317 shutdown_connections 2023-10-30T14:18:39.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.184+0000 7fdb2d90e640 1 -- 172.21.15.57:0/1025155317 wait complete. 2023-10-30T14:18:39.207 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2023-10-30T14:18:39.208 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T14:18:39.208 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:18:39.225 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T14:18:39.225 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:39.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.364+0000 7f79783d3640 1 Processor -- start 2023-10-30T14:18:39.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.364+0000 7f79783d3640 1 -- start start 2023-10-30T14:18:39.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.364+0000 7f79783d3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7970153c80 0x7f797015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:39.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.364+0000 7f79783d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f79700c4a10 0x7f79700c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:39.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.364+0000 7f79783d3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7970159f20 con 0x7f797015a3b0 2023-10-30T14:18:39.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.364+0000 7f79783d3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f797015dec0 con 0x7f7970153c80 2023-10-30T14:18:39.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.364+0000 7f79783d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79701599a0 con 0x7f79700c4a10 2023-10-30T14:18:39.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.364+0000 7f797616f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7970153c80 0x7f797015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:39.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.364+0000 7f797596e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f79700c4a10 0x7f79700c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:39.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.364+0000 7f797616f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7970153c80 0x7f797015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:47312/0 (socket says 172.21.15.57:47312) 2023-10-30T14:18:39.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.364+0000 7f797616f640 1 -- 172.21.15.57:0/3298027434 learned_addr learned my addr 172.21.15.57:0/3298027434 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:39.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.364+0000 7f7976970640 1 --1- 172.21.15.57:0/3298027434 >> v1:172.21.15.88:6789/0 conn(0x7f797015a3b0 0x7f79700ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:58346/0 (socket says 172.21.15.57:58346) 2023-10-30T14:18:39.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.364+0000 7f797616f640 1 -- 172.21.15.57:0/3298027434 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f79700c4a10 msgr2=0x7f79700c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:39.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.364+0000 7f797616f640 1 --2- 172.21.15.57:0/3298027434 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f79700c4a10 0x7f79700c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:39.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.364+0000 7f797616f640 1 -- 172.21.15.57:0/3298027434 >> v1:172.21.15.88:6789/0 conn(0x7f797015a3b0 legacy=0x7f79700ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:39.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.364+0000 7f797616f640 1 -- 172.21.15.57:0/3298027434 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79700c47e0 con 0x7f7970153c80 2023-10-30T14:18:39.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.368+0000 7f797616f640 1 --2- 172.21.15.57:0/3298027434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7970153c80 0x7f797015cf90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f796c009870 tx=0x7f796c01ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0b3f1ffc8dd0882 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:39.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.368+0000 7f797516d640 1 -- 172.21.15.57:0/3298027434 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796c0a8a30 con 0x7f7970153c80 2023-10-30T14:18:39.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.368+0000 7f797516d640 1 -- 172.21.15.57:0/3298027434 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796c0a8b90 con 0x7f7970153c80 2023-10-30T14:18:39.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.368+0000 7f797516d640 1 -- 172.21.15.57:0/3298027434 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796c0b0440 con 0x7f7970153c80 2023-10-30T14:18:39.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.368+0000 7f79783d3640 1 -- 172.21.15.57:0/3298027434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7970153c80 msgr2=0x7f797015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:39.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.368+0000 7f79783d3640 1 --2- 172.21.15.57:0/3298027434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7970153c80 0x7f797015cf90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f796c009870 tx=0x7f796c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:18:39.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.368+0000 7f79783d3640 1 -- 172.21.15.57:0/3298027434 shutdown_connections 2023-10-30T14:18:39.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.368+0000 7f79783d3640 1 --2- 172.21.15.57:0/3298027434 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f79700c4a10 0x7f79700c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:39.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.368+0000 7f79783d3640 1 --2- 172.21.15.57:0/3298027434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7970153c80 0x7f797015cf90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:39.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.368+0000 7f79783d3640 1 -- 172.21.15.57:0/3298027434 >> 172.21.15.57:0/3298027434 conn(0x7f79700715c0 msgr2=0x7f79700719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:39.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.368+0000 7f79783d3640 1 -- 172.21.15.57:0/3298027434 shutdown_connections 2023-10-30T14:18:39.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.368+0000 7f79783d3640 1 -- 172.21.15.57:0/3298027434 wait complete. 2023-10-30T14:18:39.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f79783d3640 1 Processor -- start 2023-10-30T14:18:39.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f79783d3640 1 -- start start 2023-10-30T14:18:39.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f79783d3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79700c4a10 0x7f79701ef8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:39.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f79783d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7970153c80 0x7f7970155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:39.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f79783d3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f797015a3b0 0x7f79700c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:39.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f79783d3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79701efe00 con 0x7f79700c4a10 2023-10-30T14:18:39.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f79783d3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797014f8b0 con 0x7f797015a3b0 2023-10-30T14:18:39.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f79783d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7970158f20 con 0x7f7970153c80 2023-10-30T14:18:39.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f797616f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79700c4a10 0x7f79701ef8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:39.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f797616f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79700c4a10 0x7f79701ef8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50826/0 (socket says 172.21.15.57:50826) 2023-10-30T14:18:39.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f797616f640 1 -- 172.21.15.57:0/3282468289 learned_addr learned my addr 172.21.15.57:0/3282468289 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:39.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f797596e640 1 --2- 172.21.15.57:0/3282468289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7970153c80 0x7f7970155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:39.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f7976970640 1 --2- 172.21.15.57:0/3282468289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f797015a3b0 0x7f79700c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:39.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f797616f640 1 -- 172.21.15.57:0/3282468289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7970153c80 msgr2=0x7f7970155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:39.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f797616f640 1 --2- 172.21.15.57:0/3282468289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7970153c80 0x7f7970155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:39.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f797616f640 1 -- 172.21.15.57:0/3282468289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f797015a3b0 msgr2=0x7f79700c8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:39.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f797616f640 1 --2- 172.21.15.57:0/3282468289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f797015a3b0 0x7f79700c8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:39.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f797616f640 1 -- 172.21.15.57:0/3282468289 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796c0094d0 con 0x7f79700c4a10 2023-10-30T14:18:39.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f7976970640 1 --2- 172.21.15.57:0/3282468289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f797015a3b0 0x7f79700c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:18:39.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f797616f640 1 --2- 172.21.15.57:0/3282468289 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79700c4a10 0x7f79701ef8c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f796c01d3e0 tx=0x7f796c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:39.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f7966ffd640 1 -- 172.21.15.57:0/3282468289 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796c0b1de0 con 0x7f79700c4a10 2023-10-30T14:18:39.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f7966ffd640 1 -- 172.21.15.57:0/3282468289 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796c0b9410 con 0x7f79700c4a10 2023-10-30T14:18:39.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f7966ffd640 1 -- 172.21.15.57:0/3282468289 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796c0b97c0 con 0x7f79700c4a10 2023-10-30T14:18:39.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f79783d3640 1 -- 172.21.15.57:0/3282468289 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7970157990 con 0x7f79700c4a10 2023-10-30T14:18:39.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.372+0000 7f79783d3640 1 -- 172.21.15.57:0/3282468289 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f797014ef80 con 0x7f79700c4a10 2023-10-30T14:18:39.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.376+0000 7f79783d3640 1 -- 172.21.15.57:0/3282468289 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7970157700 con 0x7f79700c4a10 2023-10-30T14:18:39.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.376+0000 7f7966ffd640 1 -- 172.21.15.57:0/3282468289 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f796c0b7040 con 0x7f79700c4a10 2023-10-30T14:18:39.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.376+0000 7f7966ffd640 1 --2- 172.21.15.57:0/3282468289 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f79540664a0 0x7f7954068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:39.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.376+0000 7f7966ffd640 1 -- 172.21.15.57:0/3282468289 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f796c121ca0 con 0x7f79700c4a10 2023-10-30T14:18:39.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.376+0000 7f797596e640 1 --2- 172.21.15.57:0/3282468289 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f79540664a0 0x7f7954068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:39.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.376+0000 7f797596e640 1 --2- 172.21.15.57:0/3282468289 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f79540664a0 0x7f7954068960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f79701e9780 tx=0x7f7960009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:39.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.380+0000 7f7966ffd640 1 -- 172.21.15.57:0/3282468289 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f796c0eedf0 con 0x7f79700c4a10 2023-10-30T14:18:39.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.572+0000 7f79783d3640 1 -- 172.21.15.57:0/3282468289 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f797004e170 con 0x7f79700c4a10 2023-10-30T14:18:39.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.572+0000 7f7966ffd640 1 -- 172.21.15.57:0/3282468289 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f796c0b5090 con 0x7f79700c4a10 2023-10-30T14:18:39.578 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:18:39.579 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5289,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:18:05.907040+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:39.579 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 34 2023-10-30T14:18:39.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.576+0000 7f79783d3640 1 -- 172.21.15.57:0/3282468289 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f79540664a0 msgr2=0x7f7954068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:39.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.576+0000 7f79783d3640 1 --2- 172.21.15.57:0/3282468289 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f79540664a0 0x7f7954068960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f79701e9780 tx=0x7f7960009190 comp rx=0 tx=0).stop 2023-10-30T14:18:39.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.576+0000 7f79783d3640 1 -- 172.21.15.57:0/3282468289 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79700c4a10 msgr2=0x7f79701ef8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:39.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.576+0000 7f79783d3640 1 --2- 172.21.15.57:0/3282468289 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79700c4a10 0x7f79701ef8c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f796c01d3e0 tx=0x7f796c01bad0 comp rx=0 tx=0).stop 2023-10-30T14:18:39.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.576+0000 7f79783d3640 1 -- 172.21.15.57:0/3282468289 shutdown_connections 2023-10-30T14:18:39.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.576+0000 7f79783d3640 1 --2- 172.21.15.57:0/3282468289 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f79540664a0 0x7f7954068960 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:39.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.576+0000 7f79783d3640 1 --2- 172.21.15.57:0/3282468289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f797015a3b0 0x7f79700c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:39.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.576+0000 7f79783d3640 1 --2- 172.21.15.57:0/3282468289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7970153c80 0x7f7970155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:39.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.576+0000 7f79783d3640 1 --2- 172.21.15.57:0/3282468289 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79700c4a10 0x7f79701ef8c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:39.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.576+0000 7f79783d3640 1 -- 172.21.15.57:0/3282468289 >> 172.21.15.57:0/3282468289 conn(0x7f79700715c0 msgr2=0x7f79700ca5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:39.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.576+0000 7f79783d3640 1 -- 172.21.15.57:0/3282468289 shutdown_connections 2023-10-30T14:18:39.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:39.576+0000 7f79783d3640 1 -- 172.21.15.57:0/3282468289 wait complete. 2023-10-30T14:18:39.599 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:18:41.600 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:41.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.792+0000 7f07b93b8640 1 Processor -- start 2023-10-30T14:18:41.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.792+0000 7f07b93b8640 1 -- start start 2023-10-30T14:18:41.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.792+0000 7f07b93b8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f07b4153c10 0x7f07b4065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.792+0000 7f07b93b8640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f07b4159970 0x7f07b4159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.792+0000 7f07b93b8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b406e390 con 0x7f07b4070c30 2023-10-30T14:18:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.792+0000 7f07b93b8640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b4072390 con 0x7f07b4159970 2023-10-30T14:18:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.792+0000 7f07b93b8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b40660e0 con 0x7f07b4153c10 2023-10-30T14:18:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.792+0000 7f07b37fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f07b4159970 0x7f07b4159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.792+0000 7f07b37fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f07b4159970 0x7f07b4159dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50830/0 (socket says 172.21.15.57:50830) 2023-10-30T14:18:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.792+0000 7f07b37fe640 1 -- 172.21.15.57:0/396939809 learned_addr learned my addr 172.21.15.57:0/396939809 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.792+0000 7f07b2ffd640 1 --2- 172.21.15.57:0/396939809 >> v2:172.21.15.88:3300/0 conn(0x7f07b4153c10 0x7f07b4065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:41.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.792+0000 7f07b37fe640 1 -- 172.21.15.57:0/396939809 >> v2:172.21.15.88:3300/0 conn(0x7f07b4153c10 msgr2=0x7f07b4065ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:41.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.792+0000 7f07b37fe640 1 --2- 172.21.15.57:0/396939809 >> v2:172.21.15.88:3300/0 conn(0x7f07b4153c10 0x7f07b4065ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:41.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.792+0000 7f07b37fe640 1 -- 172.21.15.57:0/396939809 >> v1:172.21.15.88:6789/0 conn(0x7f07b4070c30 legacy=0x7f07b4071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:41.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.792+0000 7f07b37fe640 1 -- 172.21.15.57:0/396939809 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b40bec10 con 0x7f07b4159970 2023-10-30T14:18:41.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.796+0000 7f07b37fe640 1 --2- 172.21.15.57:0/396939809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b4159970 0x7f07b4159dc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f07b40c8e10 tx=0x7f07ac01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cba775b58fc0ab9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.796+0000 7f07b27fc640 1 -- 172.21.15.57:0/396939809 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac0a64e0 con 0x7f07b4159970 2023-10-30T14:18:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.796+0000 7f07b27fc640 1 -- 172.21.15.57:0/396939809 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07ac0a6640 con 0x7f07b4159970 2023-10-30T14:18:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.796+0000 7f07b27fc640 1 -- 172.21.15.57:0/396939809 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac0a69d0 con 0x7f07b4159970 2023-10-30T14:18:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.796+0000 7f07b93b8640 1 -- 172.21.15.57:0/396939809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b4159970 msgr2=0x7f07b4159dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.796+0000 7f07b93b8640 1 --2- 172.21.15.57:0/396939809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b4159970 0x7f07b4159dc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f07b40c8e10 tx=0x7f07ac01c940 comp rx=0 tx=0).stop 2023-10-30T14:18:41.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.796+0000 7f07b93b8640 1 -- 172.21.15.57:0/396939809 shutdown_connections 2023-10-30T14:18:41.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.796+0000 7f07b93b8640 1 --2- 172.21.15.57:0/396939809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b4159970 0x7f07b4159dc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:41.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.796+0000 7f07b93b8640 1 --2- 172.21.15.57:0/396939809 >> v2:172.21.15.88:3300/0 conn(0x7f07b4153c10 0x7f07b4065ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:41.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.796+0000 7f07b93b8640 1 -- 172.21.15.57:0/396939809 >> 172.21.15.57:0/396939809 conn(0x7f07b40c7f60 msgr2=0x7f07b40c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:41.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.796+0000 7f07b93b8640 1 -- 172.21.15.57:0/396939809 shutdown_connections 2023-10-30T14:18:41.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.796+0000 7f07b93b8640 1 -- 172.21.15.57:0/396939809 wait complete. 2023-10-30T14:18:41.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.796+0000 7f07b93b8640 1 Processor -- start 2023-10-30T14:18:41.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.796+0000 7f07b93b8640 1 -- start start 2023-10-30T14:18:41.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b93b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07b4070c30 0x7f07b40c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:41.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b93b8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07b4153c10 0x7f07b414f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:41.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b93b8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b4159970 0x7f07b414fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:41.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b93b8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b4143260 con 0x7f07b4159970 2023-10-30T14:18:41.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b93b8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b4141350 con 0x7f07b4153c10 2023-10-30T14:18:41.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b93b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b4141630 con 0x7f07b4070c30 2023-10-30T14:18:41.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b4159970 0x7f07b414fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:41.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b4159970 0x7f07b414fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50834/0 (socket says 172.21.15.57:50834) 2023-10-30T14:18:41.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b37fe640 1 -- 172.21.15.57:0/1488217861 learned_addr learned my addr 172.21.15.57:0/1488217861 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:41.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07abfff640 1 --2- 172.21.15.57:0/1488217861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07b4153c10 0x7f07b414f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:41.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b2ffd640 1 --2- 172.21.15.57:0/1488217861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07b4070c30 0x7f07b40c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:41.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b37fe640 1 -- 172.21.15.57:0/1488217861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07b4070c30 msgr2=0x7f07b40c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:41.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b37fe640 1 --2- 172.21.15.57:0/1488217861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07b4070c30 0x7f07b40c9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:41.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b37fe640 1 -- 172.21.15.57:0/1488217861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07b4153c10 msgr2=0x7f07b414f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:41.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b37fe640 1 --2- 172.21.15.57:0/1488217861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07b4153c10 0x7f07b414f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:41.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b37fe640 1 -- 172.21.15.57:0/1488217861 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac009b30 con 0x7f07b4159970 2023-10-30T14:18:41.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07abfff640 1 --2- 172.21.15.57:0/1488217861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07b4153c10 0x7f07b414f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:18:41.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b2ffd640 1 --2- 172.21.15.57:0/1488217861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07b4070c30 0x7f07b40c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:18:41.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b37fe640 1 --2- 172.21.15.57:0/1488217861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b4159970 0x7f07b414fb40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f07ac009e20 tx=0x7f07ac0196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:41.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07ab7fe640 1 -- 172.21.15.57:0/1488217861 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac0b4070 con 0x7f07b4159970 2023-10-30T14:18:41.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07ab7fe640 1 -- 172.21.15.57:0/1488217861 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07ac0091d0 con 0x7f07b4159970 2023-10-30T14:18:41.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07ab7fe640 1 -- 172.21.15.57:0/1488217861 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac0a69f0 con 0x7f07b4159970 2023-10-30T14:18:41.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b93b8640 1 -- 172.21.15.57:0/1488217861 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b4142ba0 con 0x7f07b4159970 2023-10-30T14:18:41.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b93b8640 1 -- 172.21.15.57:0/1488217861 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b4140d80 con 0x7f07b4159970 2023-10-30T14:18:41.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.800+0000 7f07b93b8640 1 -- 172.21.15.57:0/1488217861 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b41433a0 con 0x7f07b4159970 2023-10-30T14:18:41.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.804+0000 7f07ab7fe640 1 -- 172.21.15.57:0/1488217861 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f07ac0b8070 con 0x7f07b4159970 2023-10-30T14:18:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.804+0000 7f07ab7fe640 1 --2- 172.21.15.57:0/1488217861 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07940664a0 0x7f0794068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.804+0000 7f07ab7fe640 1 -- 172.21.15.57:0/1488217861 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f07ac0aa070 con 0x7f07b4159970 2023-10-30T14:18:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.804+0000 7f07b2ffd640 1 --2- 172.21.15.57:0/1488217861 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07940664a0 0x7f0794068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.804+0000 7f07b2ffd640 1 --2- 172.21.15.57:0/1488217861 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07940664a0 0x7f0794068960 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f07a4007a70 tx=0x7f07a4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:41.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:41.808+0000 7f07ab7fe640 1 -- 172.21.15.57:0/1488217861 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f07b41433a0 con 0x7f07b4159970 2023-10-30T14:18:42.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:42.000+0000 7f07b93b8640 1 -- 172.21.15.57:0/1488217861 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07b41433a0 con 0x7f07b4159970 2023-10-30T14:18:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:42.000+0000 7f07ab7fe640 1 -- 172.21.15.57:0/1488217861 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f07b41433a0 con 0x7f07b4159970 2023-10-30T14:18:42.004 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:18:42.004 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5289,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:18:05.907040+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 34 2023-10-30T14:18:42.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:42.000+0000 7f07b93b8640 1 -- 172.21.15.57:0/1488217861 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07940664a0 msgr2=0x7f0794068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:42.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:42.000+0000 7f07b93b8640 1 --2- 172.21.15.57:0/1488217861 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07940664a0 0x7f0794068960 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f07a4007a70 tx=0x7f07a4008040 comp rx=0 tx=0).stop 2023-10-30T14:18:42.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:42.000+0000 7f07b93b8640 1 -- 172.21.15.57:0/1488217861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b4159970 msgr2=0x7f07b414fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:42.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:42.000+0000 7f07b93b8640 1 --2- 172.21.15.57:0/1488217861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b4159970 0x7f07b414fb40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f07ac009e20 tx=0x7f07ac0196e0 comp rx=0 tx=0).stop 2023-10-30T14:18:42.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:42.000+0000 7f07b93b8640 1 -- 172.21.15.57:0/1488217861 shutdown_connections 2023-10-30T14:18:42.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:42.000+0000 7f07b93b8640 1 --2- 172.21.15.57:0/1488217861 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07940664a0 0x7f0794068960 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:42.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:42.000+0000 7f07b93b8640 1 --2- 172.21.15.57:0/1488217861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b4159970 0x7f07b414fb40 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:42.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:42.000+0000 7f07b93b8640 1 --2- 172.21.15.57:0/1488217861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07b4153c10 0x7f07b414f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:42.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:42.000+0000 7f07b93b8640 1 --2- 172.21.15.57:0/1488217861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07b4070c30 0x7f07b40c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:42.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:42.000+0000 7f07b93b8640 1 -- 172.21.15.57:0/1488217861 >> 172.21.15.57:0/1488217861 conn(0x7f07b40c7f60 msgr2=0x7f07b40c6920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:42.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:42.000+0000 7f07b93b8640 1 -- 172.21.15.57:0/1488217861 shutdown_connections 2023-10-30T14:18:42.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:42.000+0000 7f07b93b8640 1 -- 172.21.15.57:0/1488217861 wait complete. 2023-10-30T14:18:42.025 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:18:44.026 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:44.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.176+0000 7fa45b279640 1 Processor -- start 2023-10-30T14:18:44.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.176+0000 7fa45b279640 1 -- start start 2023-10-30T14:18:44.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.176+0000 7fa45b279640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4540c4a10 0x7fa45415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:44.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.176+0000 7fa45b279640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa4540c42a0 0x7fa4540ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:44.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.176+0000 7fa45b279640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa454159f20 con 0x7fa454153c80 2023-10-30T14:18:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.176+0000 7fa45b279640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa45415dec0 con 0x7fa4540c42a0 2023-10-30T14:18:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.176+0000 7fa45b279640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4541599a0 con 0x7fa4540c4a10 2023-10-30T14:18:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.176+0000 7fa459816640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa4540c42a0 0x7fa4540ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.176+0000 7fa459015640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa454153c80 0x7fa45415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:33434/0 (socket says 172.21.15.57:33434) 2023-10-30T14:18:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.176+0000 7fa459015640 1 -- 172.21.15.57:0/2219836111 learned_addr learned my addr 172.21.15.57:0/2219836111 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:18:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.176+0000 7fa458814640 1 --2- 172.21.15.57:0/2219836111 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4540c4a10 0x7fa45415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:44.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.176+0000 7fa44bfff640 1 -- 172.21.15.57:0/2219836111 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005347245 0 0) 0x7fa454159f20 con 0x7fa454153c80 2023-10-30T14:18:44.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.176+0000 7fa44bfff640 1 -- 172.21.15.57:0/2219836111 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa440003530 con 0x7fa454153c80 2023-10-30T14:18:44.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.176+0000 7fa459816640 1 -- 172.21.15.57:0/2219836111 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4540c4a10 msgr2=0x7fa45415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:44.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.176+0000 7fa459816640 1 --2- 172.21.15.57:0/2219836111 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4540c4a10 0x7fa45415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:44.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.176+0000 7fa459816640 1 -- 172.21.15.57:0/2219836111 >> v1:172.21.15.57:6789/0 conn(0x7fa454153c80 legacy=0x7fa45415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:44.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.176+0000 7fa459816640 1 -- 172.21.15.57:0/2219836111 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45415d620 con 0x7fa4540c42a0 2023-10-30T14:18:44.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.180+0000 7fa459816640 1 --2- 172.21.15.57:0/2219836111 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4540c42a0 0x7fa4540ca1e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa44c009d90 tx=0x7fa44c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3b2d2b1139c6b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:44.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.180+0000 7fa44bfff640 1 -- 172.21.15.57:0/2219836111 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44c0af600 con 0x7fa4540c42a0 2023-10-30T14:18:44.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.180+0000 7fa44bfff640 1 -- 172.21.15.57:0/2219836111 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44c0ab050 con 0x7fa4540c42a0 2023-10-30T14:18:44.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.180+0000 7fa44bfff640 1 -- 172.21.15.57:0/2219836111 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44c0af960 con 0x7fa4540c42a0 2023-10-30T14:18:44.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.180+0000 7fa45b279640 1 -- 172.21.15.57:0/2219836111 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4540c42a0 msgr2=0x7fa4540ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:44.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.180+0000 7fa45b279640 1 --2- 172.21.15.57:0/2219836111 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4540c42a0 0x7fa4540ca1e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa44c009d90 tx=0x7fa44c01f130 comp rx=0 tx=0).stop 2023-10-30T14:18:44.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.180+0000 7fa45b279640 1 -- 172.21.15.57:0/2219836111 shutdown_connections 2023-10-30T14:18:44.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.180+0000 7fa45b279640 1 --2- 172.21.15.57:0/2219836111 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4540c4a10 0x7fa45415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:44.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.180+0000 7fa45b279640 1 --2- 172.21.15.57:0/2219836111 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4540c42a0 0x7fa4540ca1e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:44.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.180+0000 7fa45b279640 1 -- 172.21.15.57:0/2219836111 >> 172.21.15.57:0/2219836111 conn(0x7fa4540715c0 msgr2=0x7fa4540719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:44.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.180+0000 7fa45b279640 1 -- 172.21.15.57:0/2219836111 shutdown_connections 2023-10-30T14:18:44.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.180+0000 7fa45b279640 1 -- 172.21.15.57:0/2219836111 wait complete. 2023-10-30T14:18:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa45b279640 1 Processor -- start 2023-10-30T14:18:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa45b279640 1 -- start start 2023-10-30T14:18:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa45b279640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4540c42a0 0x7fa45414d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa45b279640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4540c4a10 0x7fa454149f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa45b279640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa454153c80 0x7fa45414e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:44.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa45b279640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa454157630 con 0x7fa4540c4a10 2023-10-30T14:18:44.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa45b279640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45414a4c0 con 0x7fa4540c42a0 2023-10-30T14:18:44.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa45b279640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45414c820 con 0x7fa454153c80 2023-10-30T14:18:44.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa458814640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4540c4a10 0x7fa454149f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:44.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa458814640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4540c4a10 0x7fa454149f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59566/0 (socket says 172.21.15.57:59566) 2023-10-30T14:18:44.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa458814640 1 -- 172.21.15.57:0/1252170428 learned_addr learned my addr 172.21.15.57:0/1252170428 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:44.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa459015640 1 --2- 172.21.15.57:0/1252170428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4540c42a0 0x7fa45414d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:44.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa459816640 1 --2- 172.21.15.57:0/1252170428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa454153c80 0x7fa45414e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:44.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa458814640 1 -- 172.21.15.57:0/1252170428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa454153c80 msgr2=0x7fa45414e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:44.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa458814640 1 --2- 172.21.15.57:0/1252170428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa454153c80 0x7fa45414e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:44.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa458814640 1 -- 172.21.15.57:0/1252170428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4540c42a0 msgr2=0x7fa45414d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:44.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa458814640 1 --2- 172.21.15.57:0/1252170428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4540c42a0 0x7fa45414d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:44.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa458814640 1 -- 172.21.15.57:0/1252170428 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44c009a70 con 0x7fa4540c4a10 2023-10-30T14:18:44.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa459015640 1 --2- 172.21.15.57:0/1252170428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4540c42a0 0x7fa45414d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:18:44.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa458814640 1 --2- 172.21.15.57:0/1252170428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4540c4a10 0x7fa454149f80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa44400e930 tx=0x7fa44400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:44.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa449ffb640 1 -- 172.21.15.57:0/1252170428 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa444009690 con 0x7fa4540c4a10 2023-10-30T14:18:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa449ffb640 1 -- 172.21.15.57:0/1252170428 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44400c9a0 con 0x7fa4540c4a10 2023-10-30T14:18:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa449ffb640 1 -- 172.21.15.57:0/1252170428 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44400cd50 con 0x7fa4540c4a10 2023-10-30T14:18:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa45b279640 1 -- 172.21.15.57:0/1252170428 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa454150ac0 con 0x7fa4540c4a10 2023-10-30T14:18:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.184+0000 7fa45b279640 1 -- 172.21.15.57:0/1252170428 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45414ca90 con 0x7fa4540c4a10 2023-10-30T14:18:44.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.188+0000 7fa45b279640 1 -- 172.21.15.57:0/1252170428 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa45404e170 con 0x7fa4540c4a10 2023-10-30T14:18:44.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.188+0000 7fa449ffb640 1 -- 172.21.15.57:0/1252170428 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa444013470 con 0x7fa4540c4a10 2023-10-30T14:18:44.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.188+0000 7fa449ffb640 1 --2- 172.21.15.57:0/1252170428 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa4380664a0 0x7fa438068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:44.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.188+0000 7fa449ffb640 1 -- 172.21.15.57:0/1252170428 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7fa4440841d0 con 0x7fa4540c4a10 2023-10-30T14:18:44.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.188+0000 7fa459015640 1 --2- 172.21.15.57:0/1252170428 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa4380664a0 0x7fa438068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:44.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.188+0000 7fa459015640 1 --2- 172.21.15.57:0/1252170428 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa4380664a0 0x7fa438068960 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa450005890 tx=0x7fa450009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:44.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.192+0000 7fa449ffb640 1 -- 172.21.15.57:0/1252170428 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa444051580 con 0x7fa4540c4a10 2023-10-30T14:18:44.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.380+0000 7fa45b279640 1 -- 172.21.15.57:0/1252170428 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4540cad40 con 0x7fa4540c4a10 2023-10-30T14:18:44.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.380+0000 7fa449ffb640 1 -- 172.21.15.57:0/1252170428 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fa444054ba0 con 0x7fa4540c4a10 2023-10-30T14:18:44.386 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:18:44.386 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5289,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:18:05.907040+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:44.386 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 34 2023-10-30T14:18:44.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.384+0000 7fa45b279640 1 -- 172.21.15.57:0/1252170428 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa4380664a0 msgr2=0x7fa438068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:44.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.384+0000 7fa45b279640 1 --2- 172.21.15.57:0/1252170428 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa4380664a0 0x7fa438068960 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa450005890 tx=0x7fa450009040 comp rx=0 tx=0).stop 2023-10-30T14:18:44.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.384+0000 7fa45b279640 1 -- 172.21.15.57:0/1252170428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4540c4a10 msgr2=0x7fa454149f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:44.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.384+0000 7fa45b279640 1 --2- 172.21.15.57:0/1252170428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4540c4a10 0x7fa454149f80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa44400e930 tx=0x7fa44400edf0 comp rx=0 tx=0).stop 2023-10-30T14:18:44.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.384+0000 7fa45b279640 1 -- 172.21.15.57:0/1252170428 shutdown_connections 2023-10-30T14:18:44.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.384+0000 7fa45b279640 1 --2- 172.21.15.57:0/1252170428 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa4380664a0 0x7fa438068960 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:44.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.384+0000 7fa45b279640 1 --2- 172.21.15.57:0/1252170428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa454153c80 0x7fa45414e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:44.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.384+0000 7fa45b279640 1 --2- 172.21.15.57:0/1252170428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4540c4a10 0x7fa454149f80 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:44.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.384+0000 7fa45b279640 1 --2- 172.21.15.57:0/1252170428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4540c42a0 0x7fa45414d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:44.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.384+0000 7fa45b279640 1 -- 172.21.15.57:0/1252170428 >> 172.21.15.57:0/1252170428 conn(0x7fa4540715c0 msgr2=0x7fa454158800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:44.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.384+0000 7fa45b279640 1 -- 172.21.15.57:0/1252170428 shutdown_connections 2023-10-30T14:18:44.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:44.384+0000 7fa45b279640 1 -- 172.21.15.57:0/1252170428 wait complete. 2023-10-30T14:18:44.407 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:18:46.408 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:46.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.560+0000 7fca4a468640 1 Processor -- start 2023-10-30T14:18:46.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.560+0000 7fca4a468640 1 -- start start 2023-10-30T14:18:46.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.560+0000 7fca4a468640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fca440cb7d0 0x7fca440c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:46.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.560+0000 7fca4a468640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fca440c3e40 0x7fca44070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:46.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.560+0000 7fca4a468640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca440c2310 con 0x7fca44150830 2023-10-30T14:18:46.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.560+0000 7fca4a468640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca4406bc10 con 0x7fca440cb7d0 2023-10-30T14:18:46.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.560+0000 7fca4a468640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca44156710 con 0x7fca440c3e40 2023-10-30T14:18:46.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.560+0000 7fca437fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fca440cb7d0 0x7fca440c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:46.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.560+0000 7fca437fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fca440cb7d0 0x7fca440c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59576/0 (socket says 172.21.15.57:59576) 2023-10-30T14:18:46.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.560+0000 7fca437fe640 1 -- 172.21.15.57:0/1352541920 learned_addr learned my addr 172.21.15.57:0/1352541920 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:46.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.560+0000 7fca48a05640 1 --2- 172.21.15.57:0/1352541920 >> v2:172.21.15.88:3300/0 conn(0x7fca440c3e40 0x7fca44070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:46.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.560+0000 7fca437fe640 1 -- 172.21.15.57:0/1352541920 >> v2:172.21.15.88:3300/0 conn(0x7fca440c3e40 msgr2=0x7fca44070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:46.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.560+0000 7fca437fe640 1 --2- 172.21.15.57:0/1352541920 >> v2:172.21.15.88:3300/0 conn(0x7fca440c3e40 0x7fca44070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:46.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.560+0000 7fca437fe640 1 -- 172.21.15.57:0/1352541920 >> v1:172.21.15.88:6789/0 conn(0x7fca44150830 legacy=0x7fca4406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:46.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.560+0000 7fca437fe640 1 -- 172.21.15.57:0/1352541920 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca440c4360 con 0x7fca440cb7d0 2023-10-30T14:18:46.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.564+0000 7fca437fe640 1 --2- 172.21.15.57:0/1352541920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca440cb7d0 0x7fca440c3900 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fca44066e10 tx=0x7fca3801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efc4c40dff379e20 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:46.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.564+0000 7fca42ffd640 1 -- 172.21.15.57:0/1352541920 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca380a89f0 con 0x7fca440cb7d0 2023-10-30T14:18:46.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.564+0000 7fca42ffd640 1 -- 172.21.15.57:0/1352541920 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca380a8b50 con 0x7fca440cb7d0 2023-10-30T14:18:46.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.564+0000 7fca42ffd640 1 -- 172.21.15.57:0/1352541920 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca380b0440 con 0x7fca440cb7d0 2023-10-30T14:18:46.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.564+0000 7fca4a468640 1 -- 172.21.15.57:0/1352541920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca440cb7d0 msgr2=0x7fca440c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:46.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.564+0000 7fca4a468640 1 --2- 172.21.15.57:0/1352541920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca440cb7d0 0x7fca440c3900 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fca44066e10 tx=0x7fca3801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:18:46.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.564+0000 7fca4a468640 1 -- 172.21.15.57:0/1352541920 shutdown_connections 2023-10-30T14:18:46.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.564+0000 7fca4a468640 1 --2- 172.21.15.57:0/1352541920 >> v2:172.21.15.88:3300/0 conn(0x7fca440c3e40 0x7fca44070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:46.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.564+0000 7fca4a468640 1 --2- 172.21.15.57:0/1352541920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca440cb7d0 0x7fca440c3900 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:46.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.564+0000 7fca4a468640 1 -- 172.21.15.57:0/1352541920 >> 172.21.15.57:0/1352541920 conn(0x7fca440cac70 msgr2=0x7fca440cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:46.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.564+0000 7fca4a468640 1 -- 172.21.15.57:0/1352541920 shutdown_connections 2023-10-30T14:18:46.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.564+0000 7fca4a468640 1 -- 172.21.15.57:0/1352541920 wait complete. 2023-10-30T14:18:46.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca4a468640 1 Processor -- start 2023-10-30T14:18:46.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca4a468640 1 -- start start 2023-10-30T14:18:46.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca4a468640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca440c3e40 0x7fca440c5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:46.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca4a468640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca440cb7d0 0x7fca441ee280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:46.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca4a468640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca44150830 0x7fca441eaab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:46.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca4a468640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca441eddf0 con 0x7fca44150830 2023-10-30T14:18:46.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca4a468640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4415a710 con 0x7fca440c3e40 2023-10-30T14:18:46.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca4a468640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca441edaf0 con 0x7fca440cb7d0 2023-10-30T14:18:46.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca48a05640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca44150830 0x7fca441eaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:46.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca48a05640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca44150830 0x7fca441eaab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59582/0 (socket says 172.21.15.57:59582) 2023-10-30T14:18:46.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca48a05640 1 -- 172.21.15.57:0/1477115782 learned_addr learned my addr 172.21.15.57:0/1477115782 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:46.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca437fe640 1 --2- 172.21.15.57:0/1477115782 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca440cb7d0 0x7fca441ee280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:46.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca43fff640 1 --2- 172.21.15.57:0/1477115782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca440c3e40 0x7fca440c5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:46.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca48a05640 1 -- 172.21.15.57:0/1477115782 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca440cb7d0 msgr2=0x7fca441ee280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:46.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca48a05640 1 --2- 172.21.15.57:0/1477115782 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca440cb7d0 0x7fca441ee280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:46.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca48a05640 1 -- 172.21.15.57:0/1477115782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca440c3e40 msgr2=0x7fca440c5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:46.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca48a05640 1 --2- 172.21.15.57:0/1477115782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca440c3e40 0x7fca440c5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:46.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca48a05640 1 -- 172.21.15.57:0/1477115782 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca380094d0 con 0x7fca44150830 2023-10-30T14:18:46.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca48a05640 1 --2- 172.21.15.57:0/1477115782 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca44150830 0x7fca441eaab0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fca3c007db0 tx=0x7fca3c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:46.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca40ff9640 1 -- 172.21.15.57:0/1477115782 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c014420 con 0x7fca44150830 2023-10-30T14:18:46.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca40ff9640 1 -- 172.21.15.57:0/1477115782 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3c014580 con 0x7fca44150830 2023-10-30T14:18:46.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca4a468640 1 -- 172.21.15.57:0/1477115782 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca4415a980 con 0x7fca44150830 2023-10-30T14:18:46.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca40ff9640 1 -- 172.21.15.57:0/1477115782 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c014930 con 0x7fca44150830 2023-10-30T14:18:46.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.568+0000 7fca4a468640 1 -- 172.21.15.57:0/1477115782 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca440ca9c0 con 0x7fca44150830 2023-10-30T14:18:46.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.572+0000 7fca4a468640 1 -- 172.21.15.57:0/1477115782 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca441edfa0 con 0x7fca44150830 2023-10-30T14:18:46.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.572+0000 7fca40ff9640 1 -- 172.21.15.57:0/1477115782 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fca3c014a90 con 0x7fca44150830 2023-10-30T14:18:46.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.572+0000 7fca40ff9640 1 --2- 172.21.15.57:0/1477115782 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fca300664d0 0x7fca30068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:46.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.572+0000 7fca40ff9640 1 -- 172.21.15.57:0/1477115782 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7fca3c084980 con 0x7fca44150830 2023-10-30T14:18:46.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.572+0000 7fca43fff640 1 --2- 172.21.15.57:0/1477115782 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fca300664d0 0x7fca30068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:46.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.572+0000 7fca43fff640 1 --2- 172.21.15.57:0/1477115782 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fca300664d0 0x7fca30068990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fca34000f70 tx=0x7fca34009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:46.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.576+0000 7fca40ff9640 1 -- 172.21.15.57:0/1477115782 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fca441edfa0 con 0x7fca44150830 2023-10-30T14:18:46.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.768+0000 7fca4a468640 1 -- 172.21.15.57:0/1477115782 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca441edc30 con 0x7fca44150830 2023-10-30T14:18:46.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.768+0000 7fca40ff9640 1 -- 172.21.15.57:0/1477115782 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fca441edc30 con 0x7fca44150830 2023-10-30T14:18:46.772 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:18:46.772 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5289,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:18:05.907040+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:46.773 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 34 2023-10-30T14:18:46.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.768+0000 7fca4a468640 1 -- 172.21.15.57:0/1477115782 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fca300664d0 msgr2=0x7fca30068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:46.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.768+0000 7fca4a468640 1 --2- 172.21.15.57:0/1477115782 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fca300664d0 0x7fca30068990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fca34000f70 tx=0x7fca34009040 comp rx=0 tx=0).stop 2023-10-30T14:18:46.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.768+0000 7fca4a468640 1 -- 172.21.15.57:0/1477115782 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca44150830 msgr2=0x7fca441eaab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:46.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.768+0000 7fca4a468640 1 --2- 172.21.15.57:0/1477115782 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca44150830 0x7fca441eaab0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fca3c007db0 tx=0x7fca3c00e4b0 comp rx=0 tx=0).stop 2023-10-30T14:18:46.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.768+0000 7fca4a468640 1 -- 172.21.15.57:0/1477115782 shutdown_connections 2023-10-30T14:18:46.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.768+0000 7fca4a468640 1 --2- 172.21.15.57:0/1477115782 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fca300664d0 0x7fca30068990 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:46.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.768+0000 7fca4a468640 1 --2- 172.21.15.57:0/1477115782 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca44150830 0x7fca441eaab0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:46.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.768+0000 7fca4a468640 1 --2- 172.21.15.57:0/1477115782 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca440cb7d0 0x7fca441ee280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:46.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.768+0000 7fca4a468640 1 --2- 172.21.15.57:0/1477115782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca440c3e40 0x7fca440c5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:46.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.768+0000 7fca4a468640 1 -- 172.21.15.57:0/1477115782 >> 172.21.15.57:0/1477115782 conn(0x7fca440cac70 msgr2=0x7fca44070520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:46.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.768+0000 7fca4a468640 1 -- 172.21.15.57:0/1477115782 shutdown_connections 2023-10-30T14:18:46.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:46.768+0000 7fca4a468640 1 -- 172.21.15.57:0/1477115782 wait complete. 2023-10-30T14:18:46.793 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:18:48.794 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:48.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.944+0000 7f30822d6640 1 Processor -- start 2023-10-30T14:18:48.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.944+0000 7f30822d6640 1 -- start start 2023-10-30T14:18:48.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.944+0000 7f30822d6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f307c151c20 0x7f307c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:48.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.944+0000 7f30822d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f307c0cb8b0 0x7f307c06b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:48.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.944+0000 7f30822d6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307c0c5030 con 0x7f307c151840 2023-10-30T14:18:48.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.944+0000 7f30822d6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f307c157830 con 0x7f307c151c20 2023-10-30T14:18:48.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.944+0000 7f30822d6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f307c157b10 con 0x7f307c0cb8b0 2023-10-30T14:18:48.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.944+0000 7f307bfff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f307c151840 0x7f307c072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:33440/0 (socket says 172.21.15.57:33440) 2023-10-30T14:18:48.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.944+0000 7f307bfff640 1 -- 172.21.15.57:0/3367700176 learned_addr learned my addr 172.21.15.57:0/3367700176 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:18:48.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.944+0000 7f307b7fe640 1 --2- 172.21.15.57:0/3367700176 >> v2:172.21.15.57:3300/0 conn(0x7f307c151c20 0x7f307c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:48.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.944+0000 7f3080873640 1 --2- 172.21.15.57:0/3367700176 >> v2:172.21.15.88:3300/0 conn(0x7f307c0cb8b0 0x7f307c06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:48.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.948+0000 7f307affd640 1 -- 172.21.15.57:0/3367700176 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977978456 0 0) 0x7f307c0c5030 con 0x7f307c151840 2023-10-30T14:18:48.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.948+0000 7f307affd640 1 -- 172.21.15.57:0/3367700176 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3064003530 con 0x7f307c151840 2023-10-30T14:18:48.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.948+0000 7f307b7fe640 1 -- 172.21.15.57:0/3367700176 >> v2:172.21.15.88:3300/0 conn(0x7f307c0cb8b0 msgr2=0x7f307c06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:48.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.948+0000 7f307b7fe640 1 --2- 172.21.15.57:0/3367700176 >> v2:172.21.15.88:3300/0 conn(0x7f307c0cb8b0 0x7f307c06b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:48.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.948+0000 7f307b7fe640 1 -- 172.21.15.57:0/3367700176 >> v1:172.21.15.57:6789/0 conn(0x7f307c151840 legacy=0x7f307c072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:48.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.948+0000 7f307b7fe640 1 -- 172.21.15.57:0/3367700176 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c157df0 con 0x7f307c151c20 2023-10-30T14:18:48.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.948+0000 7f307b7fe640 1 --2- 172.21.15.57:0/3367700176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307c151c20 0x7f307c0c49a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f307c0c6cb0 tx=0x7f307001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0385fd06972ae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:48.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.948+0000 7f307affd640 1 -- 172.21.15.57:0/3367700176 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30700a8950 con 0x7f307c151c20 2023-10-30T14:18:48.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.948+0000 7f307affd640 1 -- 172.21.15.57:0/3367700176 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30700a8ab0 con 0x7f307c151c20 2023-10-30T14:18:48.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.948+0000 7f307affd640 1 -- 172.21.15.57:0/3367700176 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30700a8e40 con 0x7f307c151c20 2023-10-30T14:18:48.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.948+0000 7f30822d6640 1 -- 172.21.15.57:0/3367700176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307c151c20 msgr2=0x7f307c0c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:48.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.948+0000 7f30822d6640 1 --2- 172.21.15.57:0/3367700176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307c151c20 0x7f307c0c49a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f307c0c6cb0 tx=0x7f307001ec50 comp rx=0 tx=0).stop 2023-10-30T14:18:48.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.948+0000 7f30822d6640 1 -- 172.21.15.57:0/3367700176 shutdown_connections 2023-10-30T14:18:48.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.948+0000 7f30822d6640 1 --2- 172.21.15.57:0/3367700176 >> v2:172.21.15.88:3300/0 conn(0x7f307c0cb8b0 0x7f307c06b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:48.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.948+0000 7f30822d6640 1 --2- 172.21.15.57:0/3367700176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307c151c20 0x7f307c0c49a0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:48.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.948+0000 7f30822d6640 1 -- 172.21.15.57:0/3367700176 >> 172.21.15.57:0/3367700176 conn(0x7f307c0cac70 msgr2=0x7f307c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:48.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.948+0000 7f30822d6640 1 -- 172.21.15.57:0/3367700176 shutdown_connections 2023-10-30T14:18:48.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f30822d6640 1 -- 172.21.15.57:0/3367700176 wait complete. 2023-10-30T14:18:48.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f30822d6640 1 Processor -- start 2023-10-30T14:18:48.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f30822d6640 1 -- start start 2023-10-30T14:18:48.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f30822d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f307c0cb8b0 0x7f307c14baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:48.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f30822d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307c151840 0x7f307c14c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:48.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f30822d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f307c151c20 0x7f307c1476a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:48.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f30822d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c155b60 con 0x7f307c151840 2023-10-30T14:18:48.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f30822d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c14f710 con 0x7f307c0cb8b0 2023-10-30T14:18:48.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f30822d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307c1ec750 con 0x7f307c151c20 2023-10-30T14:18:48.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f307b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307c151840 0x7f307c14c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f307b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307c151840 0x7f307c14c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59612/0 (socket says 172.21.15.57:59612) 2023-10-30T14:18:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f307b7fe640 1 -- 172.21.15.57:0/496265359 learned_addr learned my addr 172.21.15.57:0/496265359 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f307bfff640 1 --2- 172.21.15.57:0/496265359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f307c0cb8b0 0x7f307c14baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f3080873640 1 --2- 172.21.15.57:0/496265359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f307c151c20 0x7f307c1476a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f307b7fe640 1 -- 172.21.15.57:0/496265359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f307c151c20 msgr2=0x7f307c1476a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:48.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f307b7fe640 1 --2- 172.21.15.57:0/496265359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f307c151c20 0x7f307c1476a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:48.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f307b7fe640 1 -- 172.21.15.57:0/496265359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f307c0cb8b0 msgr2=0x7f307c14baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:48.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f307b7fe640 1 --2- 172.21.15.57:0/496265359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f307c0cb8b0 0x7f307c14baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:48.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f307b7fe640 1 -- 172.21.15.57:0/496265359 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3070009420 con 0x7f307c151840 2023-10-30T14:18:48.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f3080873640 1 --2- 172.21.15.57:0/496265359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f307c151c20 0x7f307c1476a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:18:48.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f307b7fe640 1 --2- 172.21.15.57:0/496265359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307c151840 0x7f307c14c410 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f30700b3040 tx=0x7f307001b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:48.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.952+0000 7f3078ff9640 1 -- 172.21.15.57:0/496265359 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30700b0dd0 con 0x7f307c151840 2023-10-30T14:18:48.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.956+0000 7f3078ff9640 1 -- 172.21.15.57:0/496265359 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30700b9430 con 0x7f307c151840 2023-10-30T14:18:48.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.956+0000 7f3078ff9640 1 -- 172.21.15.57:0/496265359 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30700b97e0 con 0x7f307c151840 2023-10-30T14:18:48.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.956+0000 7f30822d6640 1 -- 172.21.15.57:0/496265359 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f307c14c130 con 0x7f307c151840 2023-10-30T14:18:48.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.956+0000 7f30822d6640 1 -- 172.21.15.57:0/496265359 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307c1eccb0 con 0x7f307c151840 2023-10-30T14:18:48.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.956+0000 7f30822d6640 1 -- 172.21.15.57:0/496265359 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307c0726f0 con 0x7f307c151840 2023-10-30T14:18:48.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.956+0000 7f3078ff9640 1 -- 172.21.15.57:0/496265359 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f30700b7040 con 0x7f307c151840 2023-10-30T14:18:48.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.956+0000 7f3078ff9640 1 --2- 172.21.15.57:0/496265359 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f30680664a0 0x7f3068068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:48.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.956+0000 7f3078ff9640 1 -- 172.21.15.57:0/496265359 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f3070121ad0 con 0x7f307c151840 2023-10-30T14:18:48.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.956+0000 7f307bfff640 1 --2- 172.21.15.57:0/496265359 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f30680664a0 0x7f3068068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:48.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.960+0000 7f307bfff640 1 --2- 172.21.15.57:0/496265359 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f30680664a0 0x7f3068068960 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f306c0011f0 tx=0x7f306c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:48.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:48.960+0000 7f3078ff9640 1 -- 172.21.15.57:0/496265359 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f307c0726f0 con 0x7f307c151840 2023-10-30T14:18:49.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:49.156+0000 7f30822d6640 1 -- 172.21.15.57:0/496265359 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f307c0726f0 con 0x7f307c151840 2023-10-30T14:18:49.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:49.156+0000 7f3078ff9640 1 -- 172.21.15.57:0/496265359 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f307c0726f0 con 0x7f307c151840 2023-10-30T14:18:49.162 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:18:49.163 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5289,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:18:05.907040+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:49.164 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 34 2023-10-30T14:18:49.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:49.160+0000 7f30822d6640 1 -- 172.21.15.57:0/496265359 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f30680664a0 msgr2=0x7f3068068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:49.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:49.160+0000 7f30822d6640 1 --2- 172.21.15.57:0/496265359 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f30680664a0 0x7f3068068960 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f306c0011f0 tx=0x7f306c009040 comp rx=0 tx=0).stop 2023-10-30T14:18:49.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:49.160+0000 7f30822d6640 1 -- 172.21.15.57:0/496265359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307c151840 msgr2=0x7f307c14c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:49.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:49.160+0000 7f30822d6640 1 --2- 172.21.15.57:0/496265359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307c151840 0x7f307c14c410 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f30700b3040 tx=0x7f307001b9f0 comp rx=0 tx=0).stop 2023-10-30T14:18:49.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:49.160+0000 7f30822d6640 1 -- 172.21.15.57:0/496265359 shutdown_connections 2023-10-30T14:18:49.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:49.160+0000 7f30822d6640 1 --2- 172.21.15.57:0/496265359 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f30680664a0 0x7f3068068960 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:49.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:49.160+0000 7f30822d6640 1 --2- 172.21.15.57:0/496265359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f307c151c20 0x7f307c1476a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:49.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:49.160+0000 7f30822d6640 1 --2- 172.21.15.57:0/496265359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307c151840 0x7f307c14c410 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:49.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:49.160+0000 7f30822d6640 1 --2- 172.21.15.57:0/496265359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f307c0cb8b0 0x7f307c14baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:49.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:49.160+0000 7f30822d6640 1 -- 172.21.15.57:0/496265359 >> 172.21.15.57:0/496265359 conn(0x7f307c0cac70 msgr2=0x7f307c0733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:49.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:49.160+0000 7f30822d6640 1 -- 172.21.15.57:0/496265359 shutdown_connections 2023-10-30T14:18:49.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:49.160+0000 7f30822d6640 1 -- 172.21.15.57:0/496265359 wait complete. 2023-10-30T14:18:49.182 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:18:51.184 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:51.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.332+0000 7fe2551c5640 1 Processor -- start 2023-10-30T14:18:51.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.332+0000 7fe2551c5640 1 -- start start 2023-10-30T14:18:51.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.332+0000 7fe2551c5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe250150830 0x7fe25006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:51.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.332+0000 7fe2551c5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe250070520 0x7fe2500709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:51.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.332+0000 7fe2551c5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2500c2310 con 0x7fe2500cb7d0 2023-10-30T14:18:51.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.332+0000 7fe2551c5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe250156710 con 0x7fe250070520 2023-10-30T14:18:51.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.332+0000 7fe2551c5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2500c4360 con 0x7fe250150830 2023-10-30T14:18:51.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.332+0000 7fe24ed9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe250150830 0x7fe25006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:51.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.332+0000 7fe24f59e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe250070520 0x7fe2500709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:51.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.336+0000 7fe24ed9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe250150830 0x7fe25006b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:58862/0 (socket says 172.21.15.57:58862) 2023-10-30T14:18:51.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.336+0000 7fe24ed9d640 1 -- 172.21.15.57:0/2369814353 learned_addr learned my addr 172.21.15.57:0/2369814353 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:51.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.336+0000 7fe24e59c640 1 --1- 172.21.15.57:0/2369814353 >> v1:172.21.15.88:6789/0 conn(0x7fe2500cb7d0 0x7fe2500c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:39146/0 (socket says 172.21.15.57:39146) 2023-10-30T14:18:51.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.336+0000 7fe24ed9d640 1 -- 172.21.15.57:0/2369814353 >> v2:172.21.15.88:3300/0 conn(0x7fe250070520 msgr2=0x7fe2500709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:51.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.336+0000 7fe24ed9d640 1 --2- 172.21.15.57:0/2369814353 >> v2:172.21.15.88:3300/0 conn(0x7fe250070520 0x7fe2500709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:51.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.336+0000 7fe24ed9d640 1 -- 172.21.15.57:0/2369814353 >> v1:172.21.15.88:6789/0 conn(0x7fe2500cb7d0 legacy=0x7fe2500c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:51.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.336+0000 7fe24ed9d640 1 -- 172.21.15.57:0/2369814353 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2501569b0 con 0x7fe250150830 2023-10-30T14:18:51.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.336+0000 7fe24f59e640 1 --2- 172.21.15.57:0/2369814353 >> v2:172.21.15.88:3300/0 conn(0x7fe250070520 0x7fe2500709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:18:51.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.336+0000 7fe24ed9d640 1 --2- 172.21.15.57:0/2369814353 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe250150830 0x7fe25006b500 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe240009870 tx=0x7fe24001ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67ce8a15b25e9d70 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:51.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.336+0000 7fe24dd9b640 1 -- 172.21.15.57:0/2369814353 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2400a8a30 con 0x7fe250150830 2023-10-30T14:18:51.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.340+0000 7fe24dd9b640 1 -- 172.21.15.57:0/2369814353 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2400a8b90 con 0x7fe250150830 2023-10-30T14:18:51.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.340+0000 7fe24dd9b640 1 -- 172.21.15.57:0/2369814353 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2400b0440 con 0x7fe250150830 2023-10-30T14:18:51.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.340+0000 7fe2551c5640 1 -- 172.21.15.57:0/2369814353 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe250150830 msgr2=0x7fe25006b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:51.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.340+0000 7fe2551c5640 1 --2- 172.21.15.57:0/2369814353 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe250150830 0x7fe25006b500 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe240009870 tx=0x7fe24001ed30 comp rx=0 tx=0).stop 2023-10-30T14:18:51.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.340+0000 7fe2551c5640 1 -- 172.21.15.57:0/2369814353 shutdown_connections 2023-10-30T14:18:51.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.340+0000 7fe2551c5640 1 --2- 172.21.15.57:0/2369814353 >> v2:172.21.15.88:3300/0 conn(0x7fe250070520 0x7fe2500709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:51.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.340+0000 7fe2551c5640 1 --2- 172.21.15.57:0/2369814353 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe250150830 0x7fe25006b500 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:51.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.340+0000 7fe2551c5640 1 -- 172.21.15.57:0/2369814353 >> 172.21.15.57:0/2369814353 conn(0x7fe2500cac70 msgr2=0x7fe2500cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:51.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.340+0000 7fe2551c5640 1 -- 172.21.15.57:0/2369814353 shutdown_connections 2023-10-30T14:18:51.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.340+0000 7fe2551c5640 1 -- 172.21.15.57:0/2369814353 wait complete. 2023-10-30T14:18:51.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.340+0000 7fe2551c5640 1 Processor -- start 2023-10-30T14:18:51.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.340+0000 7fe2551c5640 1 -- start start 2023-10-30T14:18:51.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.340+0000 7fe2551c5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe250070520 0x7fe2500ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:51.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.340+0000 7fe2551c5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2500cb7d0 0x7fe2501577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:51.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.340+0000 7fe2551c5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe250150830 0x7fe250158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:51.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.340+0000 7fe2551c5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe250159c00 con 0x7fe250070520 2023-10-30T14:18:51.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.340+0000 7fe2551c5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2501ee7a0 con 0x7fe2500cb7d0 2023-10-30T14:18:51.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.340+0000 7fe2551c5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2501ef770 con 0x7fe250150830 2023-10-30T14:18:51.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.344+0000 7fe24ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe250070520 0x7fe2500ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:51.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.344+0000 7fe24ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe250070520 0x7fe2500ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59614/0 (socket says 172.21.15.57:59614) 2023-10-30T14:18:51.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.344+0000 7fe24ed9d640 1 -- 172.21.15.57:0/873809754 learned_addr learned my addr 172.21.15.57:0/873809754 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:51.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.344+0000 7fe24e59c640 1 --2- 172.21.15.57:0/873809754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2500cb7d0 0x7fe2501577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:51.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.344+0000 7fe24f59e640 1 --2- 172.21.15.57:0/873809754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe250150830 0x7fe250158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:51.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.344+0000 7fe24ed9d640 1 -- 172.21.15.57:0/873809754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe250150830 msgr2=0x7fe250158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:51.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.344+0000 7fe24ed9d640 1 --2- 172.21.15.57:0/873809754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe250150830 0x7fe250158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:51.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.344+0000 7fe24ed9d640 1 -- 172.21.15.57:0/873809754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2500cb7d0 msgr2=0x7fe2501577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:51.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.344+0000 7fe24ed9d640 1 --2- 172.21.15.57:0/873809754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2500cb7d0 0x7fe2501577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:51.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.344+0000 7fe24ed9d640 1 -- 172.21.15.57:0/873809754 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe248009a70 con 0x7fe250070520 2023-10-30T14:18:51.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.344+0000 7fe24f59e640 1 --2- 172.21.15.57:0/873809754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe250150830 0x7fe250158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:18:51.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.344+0000 7fe24ed9d640 1 --2- 172.21.15.57:0/873809754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe250070520 0x7fe2500ca2a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe240000c10 tx=0x7fe24001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:51.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.344+0000 7fe2377fe640 1 -- 172.21.15.57:0/873809754 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2400bf070 con 0x7fe250070520 2023-10-30T14:18:51.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.344+0000 7fe2377fe640 1 -- 172.21.15.57:0/873809754 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2400b1be0 con 0x7fe250070520 2023-10-30T14:18:51.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.344+0000 7fe2551c5640 1 -- 172.21.15.57:0/873809754 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2400094d0 con 0x7fe250070520 2023-10-30T14:18:51.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.344+0000 7fe2377fe640 1 -- 172.21.15.57:0/873809754 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2400b9410 con 0x7fe250070520 2023-10-30T14:18:51.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.344+0000 7fe2551c5640 1 -- 172.21.15.57:0/873809754 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2501ef2e0 con 0x7fe250070520 2023-10-30T14:18:51.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.344+0000 7fe2551c5640 1 -- 172.21.15.57:0/873809754 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe250066510 con 0x7fe250070520 2023-10-30T14:18:51.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.344+0000 7fe2377fe640 1 -- 172.21.15.57:0/873809754 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe2400b7040 con 0x7fe250070520 2023-10-30T14:18:51.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.348+0000 7fe2377fe640 1 --2- 172.21.15.57:0/873809754 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe23c0664d0 0x7fe23c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:51.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.348+0000 7fe2377fe640 1 -- 172.21.15.57:0/873809754 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7fe240121260 con 0x7fe250070520 2023-10-30T14:18:51.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.348+0000 7fe24e59c640 1 --2- 172.21.15.57:0/873809754 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe23c0664d0 0x7fe23c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:51.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.348+0000 7fe24e59c640 1 --2- 172.21.15.57:0/873809754 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe23c0664d0 0x7fe23c068990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe2500cabe0 tx=0x7fe244007e90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:51.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.352+0000 7fe2377fe640 1 -- 172.21.15.57:0/873809754 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe2400ee430 con 0x7fe250070520 2023-10-30T14:18:51.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.544+0000 7fe2551c5640 1 -- 172.21.15.57:0/873809754 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe25004e170 con 0x7fe250070520 2023-10-30T14:18:51.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.544+0000 7fe2377fe640 1 -- 172.21.15.57:0/873809754 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fe2400b5090 con 0x7fe250070520 2023-10-30T14:18:51.548 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:18:51.549 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5289,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:18:05.907040+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:51.550 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 34 2023-10-30T14:18:51.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.544+0000 7fe2551c5640 1 -- 172.21.15.57:0/873809754 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe23c0664d0 msgr2=0x7fe23c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:51.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.544+0000 7fe2551c5640 1 --2- 172.21.15.57:0/873809754 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe23c0664d0 0x7fe23c068990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe2500cabe0 tx=0x7fe244007e90 comp rx=0 tx=0).stop 2023-10-30T14:18:51.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.544+0000 7fe2551c5640 1 -- 172.21.15.57:0/873809754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe250070520 msgr2=0x7fe2500ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:51.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.544+0000 7fe2551c5640 1 --2- 172.21.15.57:0/873809754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe250070520 0x7fe2500ca2a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe240000c10 tx=0x7fe24001bad0 comp rx=0 tx=0).stop 2023-10-30T14:18:51.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.544+0000 7fe2551c5640 1 -- 172.21.15.57:0/873809754 shutdown_connections 2023-10-30T14:18:51.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.544+0000 7fe2551c5640 1 --2- 172.21.15.57:0/873809754 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe23c0664d0 0x7fe23c068990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:51.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.544+0000 7fe2551c5640 1 --2- 172.21.15.57:0/873809754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe250150830 0x7fe250158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:51.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.544+0000 7fe2551c5640 1 --2- 172.21.15.57:0/873809754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2500cb7d0 0x7fe2501577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:51.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.544+0000 7fe2551c5640 1 --2- 172.21.15.57:0/873809754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe250070520 0x7fe2500ca2a0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:51.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.544+0000 7fe2551c5640 1 -- 172.21.15.57:0/873809754 >> 172.21.15.57:0/873809754 conn(0x7fe2500cac70 msgr2=0x7fe2500c5820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:51.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.544+0000 7fe2551c5640 1 -- 172.21.15.57:0/873809754 shutdown_connections 2023-10-30T14:18:51.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:51.544+0000 7fe2551c5640 1 -- 172.21.15.57:0/873809754 wait complete. 2023-10-30T14:18:51.569 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:18:53.571 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:53.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.720+0000 7f00f8565640 1 Processor -- start 2023-10-30T14:18:53.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.720+0000 7f00f8565640 1 -- start start 2023-10-30T14:18:53.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.724+0000 7f00f8565640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f00f0151840 0x7f00f00c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:53.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.724+0000 7f00f8565640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00f00cb8b0 0x7f00f006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:53.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.724+0000 7f00f8565640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f00c5030 con 0x7f00f0151c20 2023-10-30T14:18:53.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.724+0000 7f00f8565640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00f0157830 con 0x7f00f0151840 2023-10-30T14:18:53.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.724+0000 7f00f8565640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f0157b10 con 0x7f00f00cb8b0 2023-10-30T14:18:53.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.724+0000 7f00f6301640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f00f0151840 0x7f00f00c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:53.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.724+0000 7f00f5b00640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f00f0151c20 0x7f00f0072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50106/0 (socket says 172.21.15.57:50106) 2023-10-30T14:18:53.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.724+0000 7f00f5b00640 1 -- 172.21.15.57:0/2198800106 learned_addr learned my addr 172.21.15.57:0/2198800106 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:18:53.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.724+0000 7f00f6b02640 1 --2- 172.21.15.57:0/2198800106 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00f00cb8b0 0x7f00f006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:53.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.724+0000 7f00f6301640 1 -- 172.21.15.57:0/2198800106 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00f00cb8b0 msgr2=0x7f00f006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:53.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.724+0000 7f00f6301640 1 --2- 172.21.15.57:0/2198800106 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00f00cb8b0 0x7f00f006b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:53.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.724+0000 7f00f6301640 1 -- 172.21.15.57:0/2198800106 >> v1:172.21.15.57:6789/0 conn(0x7f00f0151c20 legacy=0x7f00f0072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:53.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.724+0000 7f00f6301640 1 -- 172.21.15.57:0/2198800106 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f0157df0 con 0x7f00f0151840 2023-10-30T14:18:53.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.724+0000 7f00f52ff640 1 -- 172.21.15.57:0/2198800106 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3538295643 0 0) 0x7f00f00c5030 con 0x7f00f0151c20 2023-10-30T14:18:53.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f6301640 1 --2- 172.21.15.57:0/2198800106 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00f0151840 0x7f00f00c49a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0097c0 tx=0x7f00ec01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df1cea2bca33aa11 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:53.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f52ff640 1 -- 172.21.15.57:0/2198800106 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec0a8970 con 0x7f00f0151840 2023-10-30T14:18:53.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f52ff640 1 -- 172.21.15.57:0/2198800106 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00ec0a8ad0 con 0x7f00f0151840 2023-10-30T14:18:53.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f52ff640 1 -- 172.21.15.57:0/2198800106 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec0a8e60 con 0x7f00f0151840 2023-10-30T14:18:53.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f8565640 1 -- 172.21.15.57:0/2198800106 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00f0151840 msgr2=0x7f00f00c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:53.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f8565640 1 --2- 172.21.15.57:0/2198800106 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00f0151840 0x7f00f00c49a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0097c0 tx=0x7f00ec01ec70 comp rx=0 tx=0).stop 2023-10-30T14:18:53.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f8565640 1 -- 172.21.15.57:0/2198800106 shutdown_connections 2023-10-30T14:18:53.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f8565640 1 --2- 172.21.15.57:0/2198800106 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00f00cb8b0 0x7f00f006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:53.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f8565640 1 --2- 172.21.15.57:0/2198800106 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00f0151840 0x7f00f00c49a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:53.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f8565640 1 -- 172.21.15.57:0/2198800106 >> 172.21.15.57:0/2198800106 conn(0x7f00f00cac70 msgr2=0x7f00f00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:53.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f8565640 1 -- 172.21.15.57:0/2198800106 shutdown_connections 2023-10-30T14:18:53.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f8565640 1 -- 172.21.15.57:0/2198800106 wait complete. 2023-10-30T14:18:53.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f8565640 1 Processor -- start 2023-10-30T14:18:53.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f8565640 1 -- start start 2023-10-30T14:18:53.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f8565640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00f00cb8b0 0x7f00f01571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:53.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f8565640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00f0151840 0x7f00f006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:53.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f8565640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00f0151c20 0x7f00f0155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:53.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f8565640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f0072280 con 0x7f00f00cb8b0 2023-10-30T14:18:53.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f8565640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f0156600 con 0x7f00f0151840 2023-10-30T14:18:53.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f8565640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f006c670 con 0x7f00f0151c20 2023-10-30T14:18:53.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.728+0000 7f00f6301640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00f00cb8b0 0x7f00f01571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:53.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.732+0000 7f00f6301640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00f00cb8b0 0x7f00f01571d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46238/0 (socket says 172.21.15.57:46238) 2023-10-30T14:18:53.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.732+0000 7f00f6301640 1 -- 172.21.15.57:0/2105715080 learned_addr learned my addr 172.21.15.57:0/2105715080 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:53.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.732+0000 7f00f6b02640 1 --2- 172.21.15.57:0/2105715080 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00f0151c20 0x7f00f0155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:53.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.732+0000 7f00f5b00640 1 --2- 172.21.15.57:0/2105715080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00f0151840 0x7f00f006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:53.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.732+0000 7f00f6301640 1 -- 172.21.15.57:0/2105715080 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00f0151c20 msgr2=0x7f00f0155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:53.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.732+0000 7f00f6301640 1 --2- 172.21.15.57:0/2105715080 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00f0151c20 0x7f00f0155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:53.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.732+0000 7f00f6301640 1 -- 172.21.15.57:0/2105715080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00f0151840 msgr2=0x7f00f006b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:53.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.732+0000 7f00f6301640 1 --2- 172.21.15.57:0/2105715080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00f0151840 0x7f00f006b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:53.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.732+0000 7f00f6301640 1 -- 172.21.15.57:0/2105715080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec009420 con 0x7f00f00cb8b0 2023-10-30T14:18:53.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.732+0000 7f00f6b02640 1 --2- 172.21.15.57:0/2105715080 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00f0151c20 0x7f00f0155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:18:53.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.732+0000 7f00f5b00640 1 --2- 172.21.15.57:0/2105715080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00f0151840 0x7f00f006b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:18:53.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.732+0000 7f00f6301640 1 --2- 172.21.15.57:0/2105715080 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00f00cb8b0 0x7f00f01571d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0b3040 tx=0x7f00ec01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:53.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.732+0000 7f00e6ffd640 1 -- 172.21.15.57:0/2105715080 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec0b9630 con 0x7f00f00cb8b0 2023-10-30T14:18:53.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.732+0000 7f00e6ffd640 1 -- 172.21.15.57:0/2105715080 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00ec0b97b0 con 0x7f00f00cb8b0 2023-10-30T14:18:53.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.732+0000 7f00e6ffd640 1 -- 172.21.15.57:0/2105715080 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec0b9b60 con 0x7f00f00cb8b0 2023-10-30T14:18:53.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.732+0000 7f00f8565640 1 -- 172.21.15.57:0/2105715080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f01f23f0 con 0x7f00f00cb8b0 2023-10-30T14:18:53.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.732+0000 7f00f8565640 1 -- 172.21.15.57:0/2105715080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f01f29b0 con 0x7f00f00cb8b0 2023-10-30T14:18:53.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.732+0000 7f00f8565640 1 -- 172.21.15.57:0/2105715080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f0066510 con 0x7f00f00cb8b0 2023-10-30T14:18:53.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.732+0000 7f00e6ffd640 1 -- 172.21.15.57:0/2105715080 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f00ec0b7040 con 0x7f00f00cb8b0 2023-10-30T14:18:53.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.736+0000 7f00e6ffd640 1 --2- 172.21.15.57:0/2105715080 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f00d40664f0 0x7f00d40689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:53.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.736+0000 7f00e6ffd640 1 -- 172.21.15.57:0/2105715080 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f00ec121f10 con 0x7f00f00cb8b0 2023-10-30T14:18:53.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.736+0000 7f00f5b00640 1 --2- 172.21.15.57:0/2105715080 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f00d40664f0 0x7f00d40689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:53.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.736+0000 7f00f5b00640 1 --2- 172.21.15.57:0/2105715080 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f00d40664f0 0x7f00d40689b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f00f01e1b00 tx=0x7f00e0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:53.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.736+0000 7f00e6ffd640 1 -- 172.21.15.57:0/2105715080 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f00ec0ef380 con 0x7f00f00cb8b0 2023-10-30T14:18:53.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.936+0000 7f00f8565640 1 -- 172.21.15.57:0/2105715080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00f01580a0 con 0x7f00f00cb8b0 2023-10-30T14:18:53.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.936+0000 7f00e6ffd640 1 -- 172.21.15.57:0/2105715080 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f00ec0b5030 con 0x7f00f00cb8b0 2023-10-30T14:18:53.940 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:18:53.941 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5289,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:18:05.907040+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4282,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6839/2945854794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2945854794},{"type":"v1","addr":"172.21.15.88:6839","nonce":2945854794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:53.942 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 34 2023-10-30T14:18:53.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.936+0000 7f00f8565640 1 -- 172.21.15.57:0/2105715080 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f00d40664f0 msgr2=0x7f00d40689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:53.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.936+0000 7f00f8565640 1 --2- 172.21.15.57:0/2105715080 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f00d40664f0 0x7f00d40689b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f00f01e1b00 tx=0x7f00e0009040 comp rx=0 tx=0).stop 2023-10-30T14:18:53.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.936+0000 7f00f8565640 1 -- 172.21.15.57:0/2105715080 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00f00cb8b0 msgr2=0x7f00f01571d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:53.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.936+0000 7f00f8565640 1 --2- 172.21.15.57:0/2105715080 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00f00cb8b0 0x7f00f01571d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0b3040 tx=0x7f00ec01ba10 comp rx=0 tx=0).stop 2023-10-30T14:18:53.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.936+0000 7f00f8565640 1 -- 172.21.15.57:0/2105715080 shutdown_connections 2023-10-30T14:18:53.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.936+0000 7f00f8565640 1 --2- 172.21.15.57:0/2105715080 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f00d40664f0 0x7f00d40689b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:53.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.936+0000 7f00f8565640 1 --2- 172.21.15.57:0/2105715080 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00f0151c20 0x7f00f0155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:53.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.936+0000 7f00f8565640 1 --2- 172.21.15.57:0/2105715080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00f0151840 0x7f00f006b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:53.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.936+0000 7f00f8565640 1 --2- 172.21.15.57:0/2105715080 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00f00cb8b0 0x7f00f01571d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:53.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.936+0000 7f00f8565640 1 -- 172.21.15.57:0/2105715080 >> 172.21.15.57:0/2105715080 conn(0x7f00f00cac70 msgr2=0x7f00f014f720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:53.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.936+0000 7f00f8565640 1 -- 172.21.15.57:0/2105715080 shutdown_connections 2023-10-30T14:18:53.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:53.936+0000 7f00f8565640 1 -- 172.21.15.57:0/2105715080 wait complete. 2023-10-30T14:18:53.961 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:18:54.663 INFO:tasks.workunit.client.0.smithi057.stdout:./213/.snap/snap-subdir-test 2023-10-30T14:18:55.962 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:56.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.112+0000 7f06c933e640 1 Processor -- start 2023-10-30T14:18:56.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.112+0000 7f06c933e640 1 -- start start 2023-10-30T14:18:56.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.112+0000 7f06c933e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f06c40701a0 0x7f06c415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:56.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.112+0000 7f06c933e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c40c2aa0 con 0x7f06c406b2f0 2023-10-30T14:18:56.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.112+0000 7f06c933e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c40708e0 con 0x7f06c41537f0 2023-10-30T14:18:56.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.112+0000 7f06c933e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c40c3ab0 con 0x7f06c40701a0 2023-10-30T14:18:56.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.112+0000 7f06bbfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f06c40701a0 0x7f06c415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:56.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.112+0000 7f06c37fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f06c406b2f0 0x7f06c406b700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50108/0 (socket says 172.21.15.57:50108) 2023-10-30T14:18:56.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.112+0000 7f06bbfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f06c40701a0 0x7f06c415a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46244/0 (socket says 172.21.15.57:46244) 2023-10-30T14:18:56.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.112+0000 7f06c37fe640 1 -- 172.21.15.57:0/2754474297 learned_addr learned my addr 172.21.15.57:0/2754474297 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:18:56.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.116+0000 7f06bbfff640 1 -- 172.21.15.57:0/2754474297 >> v1:172.21.15.88:6789/0 conn(0x7f06c41537f0 legacy=0x7f06c40ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:56.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.116+0000 7f06bbfff640 1 -- 172.21.15.57:0/2754474297 >> v1:172.21.15.57:6789/0 conn(0x7f06c406b2f0 legacy=0x7f06c406b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:56.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.116+0000 7f06c27fc640 1 -- 172.21.15.57:0/2754474297 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37345875 0 0) 0x7f06c40c2aa0 con 0x7f06c406b2f0 2023-10-30T14:18:56.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.116+0000 7f06bbfff640 1 -- 172.21.15.57:0/2754474297 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c40c1eb0 con 0x7f06c40701a0 2023-10-30T14:18:56.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.116+0000 7f06bbfff640 1 --2- 172.21.15.57:0/2754474297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c40701a0 0x7f06c415a2e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f06ac0097c0 tx=0x7f06ac01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ff94d3730afbca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:56.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.116+0000 7f06c27fc640 1 -- 172.21.15.57:0/2754474297 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ac0a8970 con 0x7f06c40701a0 2023-10-30T14:18:56.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.116+0000 7f06c27fc640 1 -- 172.21.15.57:0/2754474297 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ac0a8ad0 con 0x7f06c40701a0 2023-10-30T14:18:56.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.116+0000 7f06c27fc640 1 -- 172.21.15.57:0/2754474297 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ac0a8e60 con 0x7f06c40701a0 2023-10-30T14:18:56.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.116+0000 7f06c933e640 1 -- 172.21.15.57:0/2754474297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c40701a0 msgr2=0x7f06c415a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:56.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.116+0000 7f06c933e640 1 --2- 172.21.15.57:0/2754474297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c40701a0 0x7f06c415a2e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f06ac0097c0 tx=0x7f06ac01ec70 comp rx=0 tx=0).stop 2023-10-30T14:18:56.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.116+0000 7f06c933e640 1 -- 172.21.15.57:0/2754474297 shutdown_connections 2023-10-30T14:18:56.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.116+0000 7f06c933e640 1 --2- 172.21.15.57:0/2754474297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c40701a0 0x7f06c415a2e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:56.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.116+0000 7f06c933e640 1 -- 172.21.15.57:0/2754474297 >> 172.21.15.57:0/2754474297 conn(0x7f06c4071fb0 msgr2=0x7f06c40723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:56.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.116+0000 7f06c933e640 1 -- 172.21.15.57:0/2754474297 shutdown_connections 2023-10-30T14:18:56.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c933e640 1 -- 172.21.15.57:0/2754474297 wait complete. 2023-10-30T14:18:56.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c933e640 1 Processor -- start 2023-10-30T14:18:56.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c933e640 1 -- start start 2023-10-30T14:18:56.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c933e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c406b2f0 0x7f06c40c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:56.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c933e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c40701a0 0x7f06c4154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:56.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c933e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c41537f0 0x7f06c4154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:56.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c933e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c41594a0 con 0x7f06c406b2f0 2023-10-30T14:18:56.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c933e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c4156b90 con 0x7f06c41537f0 2023-10-30T14:18:56.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c933e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c4156de0 con 0x7f06c40701a0 2023-10-30T14:18:56.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c406b2f0 0x7f06c40c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:56.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c406b2f0 0x7f06c40c9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46246/0 (socket says 172.21.15.57:46246) 2023-10-30T14:18:56.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c2ffd640 1 -- 172.21.15.57:0/2853023975 learned_addr learned my addr 172.21.15.57:0/2853023975 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:56.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06bbfff640 1 --2- 172.21.15.57:0/2853023975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c40701a0 0x7f06c4154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:56.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c37fe640 1 --2- 172.21.15.57:0/2853023975 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c41537f0 0x7f06c4154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:56.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c2ffd640 1 -- 172.21.15.57:0/2853023975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c40701a0 msgr2=0x7f06c4154250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:56.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c2ffd640 1 --2- 172.21.15.57:0/2853023975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c40701a0 0x7f06c4154250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:56.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c2ffd640 1 -- 172.21.15.57:0/2853023975 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c41537f0 msgr2=0x7f06c4154db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:56.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c2ffd640 1 --2- 172.21.15.57:0/2853023975 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c41537f0 0x7f06c4154db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:56.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c2ffd640 1 -- 172.21.15.57:0/2853023975 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ac009420 con 0x7f06c406b2f0 2023-10-30T14:18:56.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c2ffd640 1 --2- 172.21.15.57:0/2853023975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c406b2f0 0x7f06c40c9a60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f06b400c3c0 tx=0x7f06b400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:56.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06bb7fe640 1 -- 172.21.15.57:0/2853023975 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b40052d0 con 0x7f06c406b2f0 2023-10-30T14:18:56.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06bb7fe640 1 -- 172.21.15.57:0/2853023975 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b4005430 con 0x7f06c406b2f0 2023-10-30T14:18:56.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06bb7fe640 1 -- 172.21.15.57:0/2853023975 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b4003840 con 0x7f06c406b2f0 2023-10-30T14:18:56.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.120+0000 7f06c933e640 1 -- 172.21.15.57:0/2853023975 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c41570c0 con 0x7f06c406b2f0 2023-10-30T14:18:56.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.124+0000 7f06c933e640 1 -- 172.21.15.57:0/2853023975 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c4159130 con 0x7f06c406b2f0 2023-10-30T14:18:56.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.124+0000 7f06c933e640 1 -- 172.21.15.57:0/2853023975 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c404e170 con 0x7f06c406b2f0 2023-10-30T14:18:56.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.124+0000 7f06bb7fe640 1 -- 172.21.15.57:0/2853023975 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f06b40039a0 con 0x7f06c406b2f0 2023-10-30T14:18:56.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.124+0000 7f06bb7fe640 1 --2- 172.21.15.57:0/2853023975 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06a40664a0 0x7f06a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:56.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.124+0000 7f06bb7fe640 1 -- 172.21.15.57:0/2853023975 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f06b40835a0 con 0x7f06c406b2f0 2023-10-30T14:18:56.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.124+0000 7f06bbfff640 1 --2- 172.21.15.57:0/2853023975 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06a40664a0 0x7f06a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:56.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.128+0000 7f06bbfff640 1 --2- 172.21.15.57:0/2853023975 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06a40664a0 0x7f06a4068960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f06ac0b2000 tx=0x7f06ac008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:56.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.128+0000 7f06bb7fe640 1 -- 172.21.15.57:0/2853023975 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f06b4050840 con 0x7f06c406b2f0 2023-10-30T14:18:56.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.320+0000 7f06c933e640 1 -- 172.21.15.57:0/2853023975 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06c40cc9e0 con 0x7f06c406b2f0 2023-10-30T14:18:56.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.320+0000 7f06bb7fe640 1 -- 172.21.15.57:0/2853023975 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f06b4053e60 con 0x7f06c406b2f0 2023-10-30T14:18:56.325 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:18:56.326 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:18:54.654903+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:56.326 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 35 2023-10-30T14:18:56.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.320+0000 7f06c933e640 1 -- 172.21.15.57:0/2853023975 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06a40664a0 msgr2=0x7f06a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:56.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.320+0000 7f06c933e640 1 --2- 172.21.15.57:0/2853023975 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06a40664a0 0x7f06a4068960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f06ac0b2000 tx=0x7f06ac008040 comp rx=0 tx=0).stop 2023-10-30T14:18:56.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.320+0000 7f06c933e640 1 -- 172.21.15.57:0/2853023975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c406b2f0 msgr2=0x7f06c40c9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:56.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.320+0000 7f06c933e640 1 --2- 172.21.15.57:0/2853023975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c406b2f0 0x7f06c40c9a60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f06b400c3c0 tx=0x7f06b400c880 comp rx=0 tx=0).stop 2023-10-30T14:18:56.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.320+0000 7f06c933e640 1 -- 172.21.15.57:0/2853023975 shutdown_connections 2023-10-30T14:18:56.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.324+0000 7f06c933e640 1 --2- 172.21.15.57:0/2853023975 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06a40664a0 0x7f06a4068960 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:56.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.324+0000 7f06c933e640 1 --2- 172.21.15.57:0/2853023975 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c41537f0 0x7f06c4154db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:56.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.324+0000 7f06c933e640 1 --2- 172.21.15.57:0/2853023975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c40701a0 0x7f06c4154250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:56.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.324+0000 7f06c933e640 1 --2- 172.21.15.57:0/2853023975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c406b2f0 0x7f06c40c9a60 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:56.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.324+0000 7f06c933e640 1 -- 172.21.15.57:0/2853023975 >> 172.21.15.57:0/2853023975 conn(0x7f06c4071fb0 msgr2=0x7f06c40cadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:56.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.324+0000 7f06c933e640 1 -- 172.21.15.57:0/2853023975 shutdown_connections 2023-10-30T14:18:56.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.324+0000 7f06c933e640 1 -- 172.21.15.57:0/2853023975 wait complete. 2023-10-30T14:18:56.346 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-30T14:18:56.347 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:18:56.347 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:56.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.484+0000 7fe73a274640 1 Processor -- start 2023-10-30T14:18:56.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.484+0000 7fe73a274640 1 -- start start 2023-10-30T14:18:56.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.484+0000 7fe73a274640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe73415ac70 0x7fe73415cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:56.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.484+0000 7fe73a274640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73415dcc0 con 0x7fe73406aeb0 2023-10-30T14:18:56.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.484+0000 7fe73a274640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe734159d80 con 0x7fe7340c8eb0 2023-10-30T14:18:56.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.484+0000 7fe73a274640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe734159800 con 0x7fe73415ac70 2023-10-30T14:18:56.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.488+0000 7fe739a73640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fe73406aeb0 0x7fe734069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50124/0 (socket says 172.21.15.57:50124) 2023-10-30T14:18:56.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.488+0000 7fe739a73640 1 -- 172.21.15.57:0/396322589 learned_addr learned my addr 172.21.15.57:0/396322589 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:18:56.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.488+0000 7fe738a71640 1 --2- 172.21.15.57:0/396322589 >> v2:172.21.15.88:3300/0 conn(0x7fe73415ac70 0x7fe73415cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:56.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.488+0000 7fe72bfff640 1 -- 172.21.15.57:0/396322589 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1458023659 0 0) 0x7fe72c0014a0 con 0x7fe73406aeb0 2023-10-30T14:18:56.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.488+0000 7fe72bfff640 1 -- 172.21.15.57:0/396322589 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe720003530 con 0x7fe73406aeb0 2023-10-30T14:18:56.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.488+0000 7fe738a71640 1 -- 172.21.15.57:0/396322589 >> v1:172.21.15.88:6789/0 conn(0x7fe7340c8eb0 legacy=0x7fe73415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:56.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.488+0000 7fe738a71640 1 -- 172.21.15.57:0/396322589 >> v1:172.21.15.57:6789/0 conn(0x7fe73406aeb0 legacy=0x7fe734069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:56.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.488+0000 7fe738a71640 1 -- 172.21.15.57:0/396322589 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73415d420 con 0x7fe73415ac70 2023-10-30T14:18:56.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.488+0000 7fe738a71640 1 --2- 172.21.15.57:0/396322589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe73415ac70 0x7fe73415cd70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe7240097c0 tx=0x7fe72401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92fd8ae23b5f75f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:56.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe72bfff640 1 -- 172.21.15.57:0/396322589 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7240a8970 con 0x7fe73415ac70 2023-10-30T14:18:56.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe72bfff640 1 -- 172.21.15.57:0/396322589 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7240a8ad0 con 0x7fe73415ac70 2023-10-30T14:18:56.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe72bfff640 1 -- 172.21.15.57:0/396322589 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7240a8e40 con 0x7fe73415ac70 2023-10-30T14:18:56.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe73a274640 1 -- 172.21.15.57:0/396322589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe73415ac70 msgr2=0x7fe73415cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:56.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe73a274640 1 --2- 172.21.15.57:0/396322589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe73415ac70 0x7fe73415cd70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe7240097c0 tx=0x7fe72401ec70 comp rx=0 tx=0).stop 2023-10-30T14:18:56.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe73a274640 1 -- 172.21.15.57:0/396322589 shutdown_connections 2023-10-30T14:18:56.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe73a274640 1 --2- 172.21.15.57:0/396322589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe73415ac70 0x7fe73415cd70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:56.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe73a274640 1 -- 172.21.15.57:0/396322589 >> 172.21.15.57:0/396322589 conn(0x7fe7340cef00 msgr2=0x7fe7340cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:56.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe73a274640 1 -- 172.21.15.57:0/396322589 shutdown_connections 2023-10-30T14:18:56.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe73a274640 1 -- 172.21.15.57:0/396322589 wait complete. 2023-10-30T14:18:56.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe73a274640 1 Processor -- start 2023-10-30T14:18:56.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe73a274640 1 -- start start 2023-10-30T14:18:56.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe73a274640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe73406aeb0 0x7fe734156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:56.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe73a274640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7340c8eb0 0x7fe7340c4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:56.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe73a274640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe73415ac70 0x7fe7340c4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:56.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe73a274640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7341556d0 con 0x7fe73415ac70 2023-10-30T14:18:56.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe73a274640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe734154210 con 0x7fe73406aeb0 2023-10-30T14:18:56.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe73a274640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe734155070 con 0x7fe7340c8eb0 2023-10-30T14:18:56.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe739a73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe73415ac70 0x7fe7340c4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:56.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe739272640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe73406aeb0 0x7fe734156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:56.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe739a73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe73415ac70 0x7fe7340c4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46262/0 (socket says 172.21.15.57:46262) 2023-10-30T14:18:56.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.492+0000 7fe739a73640 1 -- 172.21.15.57:0/4291663415 learned_addr learned my addr 172.21.15.57:0/4291663415 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:56.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.496+0000 7fe738a71640 1 --2- 172.21.15.57:0/4291663415 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7340c8eb0 0x7fe7340c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:56.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.496+0000 7fe739a73640 1 -- 172.21.15.57:0/4291663415 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7340c8eb0 msgr2=0x7fe7340c4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:56.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.496+0000 7fe739a73640 1 --2- 172.21.15.57:0/4291663415 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7340c8eb0 0x7fe7340c4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:56.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.496+0000 7fe739a73640 1 -- 172.21.15.57:0/4291663415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe73406aeb0 msgr2=0x7fe734156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:56.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.496+0000 7fe739a73640 1 --2- 172.21.15.57:0/4291663415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe73406aeb0 0x7fe734156a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:56.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.496+0000 7fe739a73640 1 -- 172.21.15.57:0/4291663415 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe724009420 con 0x7fe73415ac70 2023-10-30T14:18:56.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.496+0000 7fe739a73640 1 --2- 172.21.15.57:0/4291663415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe73415ac70 0x7fe7340c4dc0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe72c00cb40 tx=0x7fe72c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:56.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.496+0000 7fe729ffb640 1 -- 172.21.15.57:0/4291663415 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c00aca0 con 0x7fe73415ac70 2023-10-30T14:18:56.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.496+0000 7fe729ffb640 1 -- 172.21.15.57:0/4291663415 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72c00ae00 con 0x7fe73415ac70 2023-10-30T14:18:56.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.496+0000 7fe729ffb640 1 -- 172.21.15.57:0/4291663415 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c012620 con 0x7fe73415ac70 2023-10-30T14:18:56.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.496+0000 7fe73a274640 1 -- 172.21.15.57:0/4291663415 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe734154460 con 0x7fe73415ac70 2023-10-30T14:18:56.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.496+0000 7fe73a274640 1 -- 172.21.15.57:0/4291663415 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe734154a20 con 0x7fe73415ac70 2023-10-30T14:18:56.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.496+0000 7fe73a274640 1 -- 172.21.15.57:0/4291663415 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe73404e170 con 0x7fe73415ac70 2023-10-30T14:18:56.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.496+0000 7fe729ffb640 1 -- 172.21.15.57:0/4291663415 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe72c007070 con 0x7fe73415ac70 2023-10-30T14:18:56.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.500+0000 7fe729ffb640 1 --2- 172.21.15.57:0/4291663415 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe7180664a0 0x7fe718068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:56.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.500+0000 7fe729ffb640 1 -- 172.21.15.57:0/4291663415 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fe72c083f40 con 0x7fe73415ac70 2023-10-30T14:18:56.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.500+0000 7fe739272640 1 --2- 172.21.15.57:0/4291663415 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe7180664a0 0x7fe718068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:56.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.500+0000 7fe739272640 1 --2- 172.21.15.57:0/4291663415 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe7180664a0 0x7fe718068960 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe730001810 tx=0x7fe730009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:56.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.500+0000 7fe729ffb640 1 -- 172.21.15.57:0/4291663415 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe72c051200 con 0x7fe73415ac70 2023-10-30T14:18:56.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.696+0000 7fe73a274640 1 -- 172.21.15.57:0/4291663415 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe73415dcc0 con 0x7fe73415ac70 2023-10-30T14:18:56.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.696+0000 7fe729ffb640 1 -- 172.21.15.57:0/4291663415 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5737 (secure 0 0 0) 0x7fe73415dcc0 con 0x7fe73415ac70 2023-10-30T14:18:56.700 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:18:56.701 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:18:54.654903+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:56.701 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 35 2023-10-30T14:18:56.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.696+0000 7fe73a274640 1 -- 172.21.15.57:0/4291663415 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe7180664a0 msgr2=0x7fe718068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:56.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.696+0000 7fe73a274640 1 --2- 172.21.15.57:0/4291663415 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe7180664a0 0x7fe718068960 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe730001810 tx=0x7fe730009040 comp rx=0 tx=0).stop 2023-10-30T14:18:56.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.696+0000 7fe73a274640 1 -- 172.21.15.57:0/4291663415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe73415ac70 msgr2=0x7fe7340c4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:56.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.696+0000 7fe73a274640 1 --2- 172.21.15.57:0/4291663415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe73415ac70 0x7fe7340c4dc0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe72c00cb40 tx=0x7fe72c00a060 comp rx=0 tx=0).stop 2023-10-30T14:18:56.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.696+0000 7fe73a274640 1 -- 172.21.15.57:0/4291663415 shutdown_connections 2023-10-30T14:18:56.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.696+0000 7fe73a274640 1 --2- 172.21.15.57:0/4291663415 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe7180664a0 0x7fe718068960 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:56.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.696+0000 7fe73a274640 1 --2- 172.21.15.57:0/4291663415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe73415ac70 0x7fe7340c4dc0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:56.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.696+0000 7fe73a274640 1 --2- 172.21.15.57:0/4291663415 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7340c8eb0 0x7fe7340c4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:56.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.696+0000 7fe73a274640 1 --2- 172.21.15.57:0/4291663415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe73406aeb0 0x7fe734156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:56.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.696+0000 7fe73a274640 1 -- 172.21.15.57:0/4291663415 >> 172.21.15.57:0/4291663415 conn(0x7fe7340cef00 msgr2=0x7fe7340c9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:56.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.696+0000 7fe73a274640 1 -- 172.21.15.57:0/4291663415 shutdown_connections 2023-10-30T14:18:56.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:56.696+0000 7fe73a274640 1 -- 172.21.15.57:0/4291663415 wait complete. 2023-10-30T14:18:56.721 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 35, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 35, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4936": { "addr": "172.21.15.57:6837/1988100930", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 1988100930, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 1988100930, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4936, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 31 }, "gid_5133": { "addr": "172.21.15.57:6834/2912050965", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2912050965, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2912050965, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5133, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 15 }, "gid_5289": { "addr": "172.21.15.57:6836/2427081813", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2427081813, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2427081813, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5289, "incarnation": 35, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:18:54.654903+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5289, "mds_1": 5133, "mds_2": 4936 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:18:58.723 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:58.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.872+0000 7ff5607f5640 1 Processor -- start 2023-10-30T14:18:58.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.872+0000 7ff5607f5640 1 -- start start 2023-10-30T14:18:58.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.872+0000 7ff5607f5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5580cb7d0 0x7ff5580c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:58.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.872+0000 7ff5607f5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5580c2310 con 0x7ff558150830 2023-10-30T14:18:58.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.876+0000 7ff5607f5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff558156710 con 0x7ff5580c3e40 2023-10-30T14:18:58.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.876+0000 7ff5607f5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff55806bc10 con 0x7ff5580cb7d0 2023-10-30T14:18:58.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.876+0000 7ff55e591640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff558150830 0x7ff55806b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50134/0 (socket says 172.21.15.57:50134) 2023-10-30T14:18:58.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.876+0000 7ff55e591640 1 -- 172.21.15.57:0/2155226104 learned_addr learned my addr 172.21.15.57:0/2155226104 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:18:58.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.876+0000 7ff55dd90640 1 --2- 172.21.15.57:0/2155226104 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5580cb7d0 0x7ff5580c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:58.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.876+0000 7ff55d58f640 1 -- 172.21.15.57:0/2155226104 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996729447 0 0) 0x7ff5580c2310 con 0x7ff558150830 2023-10-30T14:18:58.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.876+0000 7ff55d58f640 1 -- 172.21.15.57:0/2155226104 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff544003530 con 0x7ff558150830 2023-10-30T14:18:58.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.876+0000 7ff55d58f640 1 -- 172.21.15.57:0/2155226104 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295702174 0 0) 0x7ff558156710 con 0x7ff5580c3e40 2023-10-30T14:18:58.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.876+0000 7ff55d58f640 1 -- 172.21.15.57:0/2155226104 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5580c2310 con 0x7ff5580c3e40 2023-10-30T14:18:58.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.876+0000 7ff55d58f640 1 -- 172.21.15.57:0/2155226104 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 280330425 0 0) 0x7ff544003530 con 0x7ff558150830 2023-10-30T14:18:58.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.876+0000 7ff55dd90640 1 -- 172.21.15.57:0/2155226104 >> v1:172.21.15.88:6789/0 conn(0x7ff5580c3e40 legacy=0x7ff558070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:58.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.876+0000 7ff55dd90640 1 -- 172.21.15.57:0/2155226104 >> v1:172.21.15.57:6789/0 conn(0x7ff558150830 legacy=0x7ff55806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:58.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.876+0000 7ff55dd90640 1 -- 172.21.15.57:0/2155226104 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5580c4360 con 0x7ff5580cb7d0 2023-10-30T14:18:58.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.880+0000 7ff55dd90640 1 --2- 172.21.15.57:0/2155226104 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5580cb7d0 0x7ff5580c3900 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff5480097f0 tx=0x7ff54801ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3d371db2e65c68b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:58.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.880+0000 7ff55d58f640 1 -- 172.21.15.57:0/2155226104 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff554001680 con 0x7ff5580cb7d0 2023-10-30T14:18:58.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.880+0000 7ff55d58f640 1 -- 172.21.15.57:0/2155226104 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5480a8650 con 0x7ff5580cb7d0 2023-10-30T14:18:58.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.880+0000 7ff55d58f640 1 -- 172.21.15.57:0/2155226104 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5480a89e0 con 0x7ff5580cb7d0 2023-10-30T14:18:58.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.880+0000 7ff5607f5640 1 -- 172.21.15.57:0/2155226104 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5580cb7d0 msgr2=0x7ff5580c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:58.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.880+0000 7ff5607f5640 1 --2- 172.21.15.57:0/2155226104 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5580cb7d0 0x7ff5580c3900 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff5480097f0 tx=0x7ff54801ebd0 comp rx=0 tx=0).stop 2023-10-30T14:18:58.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.880+0000 7ff5607f5640 1 -- 172.21.15.57:0/2155226104 shutdown_connections 2023-10-30T14:18:58.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.880+0000 7ff5607f5640 1 --2- 172.21.15.57:0/2155226104 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5580cb7d0 0x7ff5580c3900 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:58.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.880+0000 7ff5607f5640 1 -- 172.21.15.57:0/2155226104 >> 172.21.15.57:0/2155226104 conn(0x7ff5580cac70 msgr2=0x7ff5580cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:58.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.880+0000 7ff5607f5640 1 -- 172.21.15.57:0/2155226104 shutdown_connections 2023-10-30T14:18:58.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.880+0000 7ff5607f5640 1 -- 172.21.15.57:0/2155226104 wait complete. 2023-10-30T14:18:58.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.880+0000 7ff5607f5640 1 Processor -- start 2023-10-30T14:18:58.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.880+0000 7ff5607f5640 1 -- start start 2023-10-30T14:18:58.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.880+0000 7ff5607f5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5580c3e40 0x7ff558144ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:58.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.880+0000 7ff5607f5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5580cb7d0 0x7ff5581ee490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:58.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.880+0000 7ff5607f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff558150830 0x7ff55814eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:58.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.880+0000 7ff5607f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff558154b30 con 0x7ff558150830 2023-10-30T14:18:58.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.880+0000 7ff5607f5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5581543e0 con 0x7ff5580cb7d0 2023-10-30T14:18:58.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.880+0000 7ff5607f5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5581ee000 con 0x7ff5580c3e40 2023-10-30T14:18:58.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.884+0000 7ff55ed92640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff558150830 0x7ff55814eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:58.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.884+0000 7ff55ed92640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff558150830 0x7ff55814eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46274/0 (socket says 172.21.15.57:46274) 2023-10-30T14:18:58.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.884+0000 7ff55ed92640 1 -- 172.21.15.57:0/3112241166 learned_addr learned my addr 172.21.15.57:0/3112241166 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:18:58.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.884+0000 7ff55e591640 1 --2- 172.21.15.57:0/3112241166 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5580c3e40 0x7ff558144ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:58.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.884+0000 7ff55dd90640 1 --2- 172.21.15.57:0/3112241166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5580cb7d0 0x7ff5581ee490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:58.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.884+0000 7ff55ed92640 1 -- 172.21.15.57:0/3112241166 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5580c3e40 msgr2=0x7ff558144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:58.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.884+0000 7ff55ed92640 1 --2- 172.21.15.57:0/3112241166 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5580c3e40 0x7ff558144ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:58.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.884+0000 7ff55ed92640 1 -- 172.21.15.57:0/3112241166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5580cb7d0 msgr2=0x7ff5581ee490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:58.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.884+0000 7ff55ed92640 1 --2- 172.21.15.57:0/3112241166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5580cb7d0 0x7ff5581ee490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:58.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.884+0000 7ff55ed92640 1 -- 172.21.15.57:0/3112241166 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff548009420 con 0x7ff558150830 2023-10-30T14:18:58.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.884+0000 7ff55ed92640 1 --2- 172.21.15.57:0/3112241166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff558150830 0x7ff55814eb00 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff55000c7a0 tx=0x7ff55000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:58.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.884+0000 7ff54effd640 1 -- 172.21.15.57:0/3112241166 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff550003b80 con 0x7ff558150830 2023-10-30T14:18:58.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.884+0000 7ff54effd640 1 -- 172.21.15.57:0/3112241166 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff550003ce0 con 0x7ff558150830 2023-10-30T14:18:58.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.884+0000 7ff54effd640 1 -- 172.21.15.57:0/3112241166 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55000aa90 con 0x7ff558150830 2023-10-30T14:18:58.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.884+0000 7ff5607f5640 1 -- 172.21.15.57:0/3112241166 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff558146f90 con 0x7ff558150830 2023-10-30T14:18:58.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.884+0000 7ff5607f5640 1 -- 172.21.15.57:0/3112241166 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff558145920 con 0x7ff558150830 2023-10-30T14:18:58.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.884+0000 7ff5607f5640 1 -- 172.21.15.57:0/3112241166 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5581eddd0 con 0x7ff558150830 2023-10-30T14:18:58.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.884+0000 7ff54effd640 1 -- 172.21.15.57:0/3112241166 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff55000abf0 con 0x7ff558150830 2023-10-30T14:18:58.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.884+0000 7ff54effd640 1 --2- 172.21.15.57:0/3112241166 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff53c0664a0 0x7ff53c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:58.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.888+0000 7ff54effd640 1 -- 172.21.15.57:0/3112241166 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6169+0+0 (secure 0 0 0) 0x7ff550083d20 con 0x7ff558150830 2023-10-30T14:18:58.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.888+0000 7ff55e591640 1 --2- 172.21.15.57:0/3112241166 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff53c0664a0 0x7ff53c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:58.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.888+0000 7ff55e591640 1 --2- 172.21.15.57:0/3112241166 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff53c0664a0 0x7ff53c068960 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff554000f70 tx=0x7ff5540016c0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:58.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:58.892+0000 7ff54effd640 1 -- 172.21.15.57:0/3112241166 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff5581eddd0 con 0x7ff558150830 2023-10-30T14:18:59.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:59.080+0000 7ff5607f5640 1 -- 172.21.15.57:0/3112241166 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5581eddd0 con 0x7ff558150830 2023-10-30T14:18:59.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:59.080+0000 7ff54effd640 1 -- 172.21.15.57:0/3112241166 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5737 (secure 0 0 0) 0x7ff5581eddd0 con 0x7ff558150830 2023-10-30T14:18:59.086 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:18:59.087 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:18:54.654903+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:59.088 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 35 2023-10-30T14:18:59.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:59.084+0000 7ff5607f5640 1 -- 172.21.15.57:0/3112241166 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff53c0664a0 msgr2=0x7ff53c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:59.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:59.084+0000 7ff5607f5640 1 --2- 172.21.15.57:0/3112241166 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff53c0664a0 0x7ff53c068960 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff554000f70 tx=0x7ff5540016c0 comp rx=0 tx=0).stop 2023-10-30T14:18:59.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:59.084+0000 7ff5607f5640 1 -- 172.21.15.57:0/3112241166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff558150830 msgr2=0x7ff55814eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:59.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:59.084+0000 7ff5607f5640 1 --2- 172.21.15.57:0/3112241166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff558150830 0x7ff55814eb00 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff55000c7a0 tx=0x7ff55000cc60 comp rx=0 tx=0).stop 2023-10-30T14:18:59.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:59.084+0000 7ff5607f5640 1 -- 172.21.15.57:0/3112241166 shutdown_connections 2023-10-30T14:18:59.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:59.084+0000 7ff5607f5640 1 --2- 172.21.15.57:0/3112241166 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff53c0664a0 0x7ff53c068960 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:59.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:59.084+0000 7ff5607f5640 1 --2- 172.21.15.57:0/3112241166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff558150830 0x7ff55814eb00 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:59.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:59.084+0000 7ff5607f5640 1 --2- 172.21.15.57:0/3112241166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5580cb7d0 0x7ff5581ee490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:59.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:59.084+0000 7ff5607f5640 1 --2- 172.21.15.57:0/3112241166 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5580c3e40 0x7ff558144ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:59.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:59.084+0000 7ff5607f5640 1 -- 172.21.15.57:0/3112241166 >> 172.21.15.57:0/3112241166 conn(0x7ff5580cac70 msgr2=0x7ff558070520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:59.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:59.084+0000 7ff5607f5640 1 -- 172.21.15.57:0/3112241166 shutdown_connections 2023-10-30T14:18:59.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:18:59.084+0000 7ff5607f5640 1 -- 172.21.15.57:0/3112241166 wait complete. 2023-10-30T14:18:59.106 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:19:00.349 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:19:00.353 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:19:01.107 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:19:01.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.252+0000 7f271de6b640 1 Processor -- start 2023-10-30T14:19:01.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.252+0000 7f271de6b640 1 -- start start 2023-10-30T14:19:01.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.252+0000 7f271de6b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2718153c80 0x7f271815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:01.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.252+0000 7f271de6b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f27180c4a10 0x7f27180c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:01.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.252+0000 7f271de6b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2718159f20 con 0x7f271815a3b0 2023-10-30T14:19:01.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.252+0000 7f271de6b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f271815dec0 con 0x7f27180c4a10 2023-10-30T14:19:01.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.252+0000 7f271de6b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27181599a0 con 0x7f2718153c80 2023-10-30T14:19:01.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.252+0000 7f2716ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f27180c4a10 0x7f27180c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:01.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.252+0000 7f27177fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2718153c80 0x7f271815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:01.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.252+0000 7f2717fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f271815a3b0 0x7f27180ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:54436/0 (socket says 172.21.15.57:54436) 2023-10-30T14:19:01.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.252+0000 7f2716ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f27180c4a10 0x7f27180c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:40936/0 (socket says 172.21.15.57:40936) 2023-10-30T14:19:01.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.252+0000 7f2716ffd640 1 -- 172.21.15.57:0/3160281649 learned_addr learned my addr 172.21.15.57:0/3160281649 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:19:01.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.252+0000 7f27177fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2718153c80 0x7f271815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:43134/0 (socket says 172.21.15.57:43134) 2023-10-30T14:19:01.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.252+0000 7f27177fe640 1 -- 172.21.15.57:0/3160281649 >> v2:172.21.15.88:3300/0 conn(0x7f27180c4a10 msgr2=0x7f27180c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:01.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.252+0000 7f27177fe640 1 --2- 172.21.15.57:0/3160281649 >> v2:172.21.15.88:3300/0 conn(0x7f27180c4a10 0x7f27180c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:01.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.252+0000 7f27177fe640 1 -- 172.21.15.57:0/3160281649 >> v1:172.21.15.88:6789/0 conn(0x7f271815a3b0 legacy=0x7f27180ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:01.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.252+0000 7f27177fe640 1 -- 172.21.15.57:0/3160281649 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27180c47e0 con 0x7f2718153c80 2023-10-30T14:19:01.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.256+0000 7f27177fe640 1 --2- 172.21.15.57:0/3160281649 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2718153c80 0x7f271815cf90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f27080097c0 tx=0x7f270801ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f52259bfefdeff15 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:01.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.256+0000 7f27167fc640 1 -- 172.21.15.57:0/3160281649 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27080a8970 con 0x7f2718153c80 2023-10-30T14:19:01.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.256+0000 7f27167fc640 1 -- 172.21.15.57:0/3160281649 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27080a8ad0 con 0x7f2718153c80 2023-10-30T14:19:01.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.256+0000 7f27167fc640 1 -- 172.21.15.57:0/3160281649 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27080a8e60 con 0x7f2718153c80 2023-10-30T14:19:01.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.256+0000 7f271de6b640 1 -- 172.21.15.57:0/3160281649 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2718153c80 msgr2=0x7f271815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:01.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.256+0000 7f271de6b640 1 --2- 172.21.15.57:0/3160281649 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2718153c80 0x7f271815cf90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f27080097c0 tx=0x7f270801ec70 comp rx=0 tx=0).stop 2023-10-30T14:19:01.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.256+0000 7f271de6b640 1 -- 172.21.15.57:0/3160281649 shutdown_connections 2023-10-30T14:19:01.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.256+0000 7f271de6b640 1 --2- 172.21.15.57:0/3160281649 >> v2:172.21.15.88:3300/0 conn(0x7f27180c4a10 0x7f27180c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:01.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.256+0000 7f271de6b640 1 --2- 172.21.15.57:0/3160281649 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2718153c80 0x7f271815cf90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:01.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.256+0000 7f271de6b640 1 -- 172.21.15.57:0/3160281649 >> 172.21.15.57:0/3160281649 conn(0x7f27180715c0 msgr2=0x7f27180719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:01.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.256+0000 7f271de6b640 1 -- 172.21.15.57:0/3160281649 shutdown_connections 2023-10-30T14:19:01.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.256+0000 7f271de6b640 1 -- 172.21.15.57:0/3160281649 wait complete. 2023-10-30T14:19:01.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f271de6b640 1 Processor -- start 2023-10-30T14:19:01.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f271de6b640 1 -- start start 2023-10-30T14:19:01.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f271de6b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27180c4a10 0x7f271814f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:01.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f271de6b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2718153c80 0x7f27180c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:01.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f271de6b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f271815a3b0 0x7f27180c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:01.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f271de6b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27181f2260 con 0x7f2718153c80 2023-10-30T14:19:01.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f271de6b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27181f2540 con 0x7f271815a3b0 2023-10-30T14:19:01.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f271de6b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27181f2820 con 0x7f27180c4a10 2023-10-30T14:19:01.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f2716ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2718153c80 0x7f27180c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:01.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f2716ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2718153c80 0x7f27180c88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46278/0 (socket says 172.21.15.57:46278) 2023-10-30T14:19:01.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f2716ffd640 1 -- 172.21.15.57:0/3430121130 learned_addr learned my addr 172.21.15.57:0/3430121130 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:19:01.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f27177fe640 1 --2- 172.21.15.57:0/3430121130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27180c4a10 0x7f271814f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:01.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f2717fff640 1 --2- 172.21.15.57:0/3430121130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f271815a3b0 0x7f27180c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:01.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f2716ffd640 1 -- 172.21.15.57:0/3430121130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27180c4a10 msgr2=0x7f271814f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:01.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f2716ffd640 1 --2- 172.21.15.57:0/3430121130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27180c4a10 0x7f271814f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:01.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f2716ffd640 1 -- 172.21.15.57:0/3430121130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f271815a3b0 msgr2=0x7f27180c8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:01.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f2716ffd640 1 --2- 172.21.15.57:0/3430121130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f271815a3b0 0x7f27180c8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:01.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f2716ffd640 1 -- 172.21.15.57:0/3430121130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2708009420 con 0x7f2718153c80 2023-10-30T14:19:01.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f2717fff640 1 --2- 172.21.15.57:0/3430121130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f271815a3b0 0x7f27180c8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:19:01.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f2716ffd640 1 --2- 172.21.15.57:0/3430121130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2718153c80 0x7f27180c88a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f270c00e900 tx=0x7f270c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:01.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f26fffff640 1 -- 172.21.15.57:0/3430121130 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270c009660 con 0x7f2718153c80 2023-10-30T14:19:01.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f26fffff640 1 -- 172.21.15.57:0/3430121130 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f270c00ca00 con 0x7f2718153c80 2023-10-30T14:19:01.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f26fffff640 1 -- 172.21.15.57:0/3430121130 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270c00cd70 con 0x7f2718153c80 2023-10-30T14:19:01.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f271de6b640 1 -- 172.21.15.57:0/3430121130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27181f2b00 con 0x7f2718153c80 2023-10-30T14:19:01.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.260+0000 7f271de6b640 1 -- 172.21.15.57:0/3430121130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f271815d150 con 0x7f2718153c80 2023-10-30T14:19:01.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.264+0000 7f271de6b640 1 -- 172.21.15.57:0/3430121130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f271804e170 con 0x7f2718153c80 2023-10-30T14:19:01.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.264+0000 7f26fffff640 1 -- 172.21.15.57:0/3430121130 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f270c013470 con 0x7f2718153c80 2023-10-30T14:19:01.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.264+0000 7f26fffff640 1 --2- 172.21.15.57:0/3430121130 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f27040664a0 0x7f2704068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:01.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.264+0000 7f26fffff640 1 -- 172.21.15.57:0/3430121130 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f270c0841c0 con 0x7f2718153c80 2023-10-30T14:19:01.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.264+0000 7f27177fe640 1 --2- 172.21.15.57:0/3430121130 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f27040664a0 0x7f2704068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:01.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.268+0000 7f27177fe640 1 --2- 172.21.15.57:0/3430121130 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f27040664a0 0x7f2704068960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f27080b2000 tx=0x7f2708008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:01.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.268+0000 7f26fffff640 1 -- 172.21.15.57:0/3430121130 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f270c0515b0 con 0x7f2718153c80 2023-10-30T14:19:01.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.460+0000 7f271de6b640 1 -- 172.21.15.57:0/3430121130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27180cad40 con 0x7f2718153c80 2023-10-30T14:19:01.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.460+0000 7f26fffff640 1 -- 172.21.15.57:0/3430121130 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f270c054bd0 con 0x7f2718153c80 2023-10-30T14:19:01.465 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:19:01.465 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:19:00.507867+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:resolve","state_seq":15,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:19:01.467 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 36 2023-10-30T14:19:01.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.460+0000 7f271de6b640 1 -- 172.21.15.57:0/3430121130 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f27040664a0 msgr2=0x7f2704068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:01.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.460+0000 7f271de6b640 1 --2- 172.21.15.57:0/3430121130 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f27040664a0 0x7f2704068960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f27080b2000 tx=0x7f2708008040 comp rx=0 tx=0).stop 2023-10-30T14:19:01.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.460+0000 7f271de6b640 1 -- 172.21.15.57:0/3430121130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2718153c80 msgr2=0x7f27180c88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:01.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.460+0000 7f271de6b640 1 --2- 172.21.15.57:0/3430121130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2718153c80 0x7f27180c88a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f270c00e900 tx=0x7f270c00edc0 comp rx=0 tx=0).stop 2023-10-30T14:19:01.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.460+0000 7f271de6b640 1 -- 172.21.15.57:0/3430121130 shutdown_connections 2023-10-30T14:19:01.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.460+0000 7f271de6b640 1 --2- 172.21.15.57:0/3430121130 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f27040664a0 0x7f2704068960 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:01.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.460+0000 7f271de6b640 1 --2- 172.21.15.57:0/3430121130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f271815a3b0 0x7f27180c8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:01.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.460+0000 7f271de6b640 1 --2- 172.21.15.57:0/3430121130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2718153c80 0x7f27180c88a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:01.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.460+0000 7f271de6b640 1 --2- 172.21.15.57:0/3430121130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27180c4a10 0x7f271814f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:01.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.460+0000 7f271de6b640 1 -- 172.21.15.57:0/3430121130 >> 172.21.15.57:0/3430121130 conn(0x7f27180715c0 msgr2=0x7f2718157cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:01.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.460+0000 7f271de6b640 1 -- 172.21.15.57:0/3430121130 shutdown_connections 2023-10-30T14:19:01.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:01.460+0000 7f271de6b640 1 -- 172.21.15.57:0/3430121130 wait complete. 2023-10-30T14:19:01.485 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:19:03.486 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:19:03.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.636+0000 7fb28c794640 1 Processor -- start 2023-10-30T14:19:03.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.636+0000 7fb28c794640 1 -- start start 2023-10-30T14:19:03.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.640+0000 7fb28c794640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb28406c420 0x7fb2840c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:03.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.640+0000 7fb28c794640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb2840c7170 0x7fb284159980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:03.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.640+0000 7fb28c794640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2840c5f80 con 0x7fb284153bb0 2023-10-30T14:19:03.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.640+0000 7fb28c794640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2840c7580 con 0x7fb28406c420 2023-10-30T14:19:03.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.640+0000 7fb28c794640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2840c5690 con 0x7fb2840c7170 2023-10-30T14:19:03.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.640+0000 7fb289d2f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb28406c420 0x7fb2840c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:03.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.640+0000 7fb289d2f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb28406c420 0x7fb2840c6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46284/0 (socket says 172.21.15.57:46284) 2023-10-30T14:19:03.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.640+0000 7fb289d2f640 1 -- 172.21.15.57:0/1971873514 learned_addr learned my addr 172.21.15.57:0/1971873514 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:19:03.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.640+0000 7fb28ad31640 1 --2- 172.21.15.57:0/1971873514 >> v2:172.21.15.88:3300/0 conn(0x7fb2840c7170 0x7fb284159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:03.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.640+0000 7fb289d2f640 1 -- 172.21.15.57:0/1971873514 >> v2:172.21.15.88:3300/0 conn(0x7fb2840c7170 msgr2=0x7fb284159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:03.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.640+0000 7fb289d2f640 1 --2- 172.21.15.57:0/1971873514 >> v2:172.21.15.88:3300/0 conn(0x7fb2840c7170 0x7fb284159980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:03.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.640+0000 7fb289d2f640 1 -- 172.21.15.57:0/1971873514 >> v1:172.21.15.88:6789/0 conn(0x7fb284153bb0 legacy=0x7fb2840c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:03.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.640+0000 7fb289d2f640 1 -- 172.21.15.57:0/1971873514 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb28415a580 con 0x7fb28406c420 2023-10-30T14:19:03.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb289d2f640 1 --2- 172.21.15.57:0/1971873514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb28406c420 0x7fb2840c6c30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb2840c8b20 tx=0x7fb27401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fda35605e663ee8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:03.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb28952e640 1 -- 172.21.15.57:0/1971873514 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2740a89f0 con 0x7fb28406c420 2023-10-30T14:19:03.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb28952e640 1 -- 172.21.15.57:0/1971873514 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2740a8b50 con 0x7fb28406c420 2023-10-30T14:19:03.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb28952e640 1 -- 172.21.15.57:0/1971873514 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2740b0440 con 0x7fb28406c420 2023-10-30T14:19:03.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb28c794640 1 -- 172.21.15.57:0/1971873514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb28406c420 msgr2=0x7fb2840c6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:03.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb28c794640 1 --2- 172.21.15.57:0/1971873514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb28406c420 0x7fb2840c6c30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb2840c8b20 tx=0x7fb27401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:19:03.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb28c794640 1 -- 172.21.15.57:0/1971873514 shutdown_connections 2023-10-30T14:19:03.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb28c794640 1 --2- 172.21.15.57:0/1971873514 >> v2:172.21.15.88:3300/0 conn(0x7fb2840c7170 0x7fb284159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:03.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb28c794640 1 --2- 172.21.15.57:0/1971873514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb28406c420 0x7fb2840c6c30 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:03.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb28c794640 1 -- 172.21.15.57:0/1971873514 >> 172.21.15.57:0/1971873514 conn(0x7fb2840cac70 msgr2=0x7fb2840cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:03.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb28c794640 1 -- 172.21.15.57:0/1971873514 shutdown_connections 2023-10-30T14:19:03.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb28c794640 1 -- 172.21.15.57:0/1971873514 wait complete. 2023-10-30T14:19:03.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb28c794640 1 Processor -- start 2023-10-30T14:19:03.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb28c794640 1 -- start start 2023-10-30T14:19:03.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb28c794640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb28406c420 0x7fb284152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:03.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb28c794640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2840c7170 0x7fb284155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:03.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb28c794640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb284153bb0 0x7fb28414fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:03.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb28c794640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2841ef320 con 0x7fb2840c7170 2023-10-30T14:19:03.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb28c794640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb284155c50 con 0x7fb284153bb0 2023-10-30T14:19:03.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.644+0000 7fb28c794640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2841ee9f0 con 0x7fb28406c420 2023-10-30T14:19:03.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb289d2f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2840c7170 0x7fb284155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:03.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb289d2f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2840c7170 0x7fb284155210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46296/0 (socket says 172.21.15.57:46296) 2023-10-30T14:19:03.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb289d2f640 1 -- 172.21.15.57:0/4255678398 learned_addr learned my addr 172.21.15.57:0/4255678398 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:19:03.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb28ad31640 1 --2- 172.21.15.57:0/4255678398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb284153bb0 0x7fb28414fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:03.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb28a530640 1 --2- 172.21.15.57:0/4255678398 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb28406c420 0x7fb284152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:03.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb289d2f640 1 -- 172.21.15.57:0/4255678398 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb28406c420 msgr2=0x7fb284152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:03.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb289d2f640 1 --2- 172.21.15.57:0/4255678398 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb28406c420 0x7fb284152150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:03.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb289d2f640 1 -- 172.21.15.57:0/4255678398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb284153bb0 msgr2=0x7fb28414fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:03.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb289d2f640 1 --2- 172.21.15.57:0/4255678398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb284153bb0 0x7fb28414fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:03.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb289d2f640 1 -- 172.21.15.57:0/4255678398 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2740094d0 con 0x7fb2840c7170 2023-10-30T14:19:03.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb28ad31640 1 --2- 172.21.15.57:0/4255678398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb284153bb0 0x7fb28414fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:19:03.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb289d2f640 1 --2- 172.21.15.57:0/4255678398 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2840c7170 0x7fb284155210 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb27401b900 tx=0x7fb27401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:03.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb27affd640 1 -- 172.21.15.57:0/4255678398 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2740ba600 con 0x7fb2840c7170 2023-10-30T14:19:03.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb27affd640 1 -- 172.21.15.57:0/4255678398 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2740ba760 con 0x7fb2840c7170 2023-10-30T14:19:03.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb27affd640 1 -- 172.21.15.57:0/4255678398 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2740bab10 con 0x7fb2840c7170 2023-10-30T14:19:03.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb28c794640 1 -- 172.21.15.57:0/4255678398 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2841eecd0 con 0x7fb2840c7170 2023-10-30T14:19:03.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb28c794640 1 -- 172.21.15.57:0/4255678398 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2840c4ac0 con 0x7fb2840c7170 2023-10-30T14:19:03.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb28c794640 1 -- 172.21.15.57:0/4255678398 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb284066510 con 0x7fb2840c7170 2023-10-30T14:19:03.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb27affd640 1 -- 172.21.15.57:0/4255678398 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb2740bac70 con 0x7fb2840c7170 2023-10-30T14:19:03.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.648+0000 7fb27affd640 1 --2- 172.21.15.57:0/4255678398 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb2680664a0 0x7fb268068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:03.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.652+0000 7fb27affd640 1 -- 172.21.15.57:0/4255678398 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fb2741225f0 con 0x7fb2840c7170 2023-10-30T14:19:03.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.652+0000 7fb28a530640 1 --2- 172.21.15.57:0/4255678398 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb2680664a0 0x7fb268068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:03.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.652+0000 7fb28a530640 1 --2- 172.21.15.57:0/4255678398 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb2680664a0 0x7fb268068960 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb280000f70 tx=0x7fb280009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:03.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.652+0000 7fb27affd640 1 -- 172.21.15.57:0/4255678398 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb2740ef920 con 0x7fb2840c7170 2023-10-30T14:19:03.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.844+0000 7fb28c794640 1 -- 172.21.15.57:0/4255678398 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb28404e170 con 0x7fb2840c7170 2023-10-30T14:19:03.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.844+0000 7fb27affd640 1 -- 172.21.15.57:0/4255678398 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5739 (secure 0 0 0) 0x7fb2740f2f40 con 0x7fb2840c7170 2023-10-30T14:19:03.849 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:19:03.850 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:19:00.507867+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:resolve","state_seq":15,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:19:03.850 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 36 2023-10-30T14:19:03.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.844+0000 7fb28c794640 1 -- 172.21.15.57:0/4255678398 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb2680664a0 msgr2=0x7fb268068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:03.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.844+0000 7fb28c794640 1 --2- 172.21.15.57:0/4255678398 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb2680664a0 0x7fb268068960 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb280000f70 tx=0x7fb280009040 comp rx=0 tx=0).stop 2023-10-30T14:19:03.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.844+0000 7fb28c794640 1 -- 172.21.15.57:0/4255678398 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2840c7170 msgr2=0x7fb284155210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:03.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.844+0000 7fb28c794640 1 --2- 172.21.15.57:0/4255678398 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2840c7170 0x7fb284155210 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb27401b900 tx=0x7fb27401ba90 comp rx=0 tx=0).stop 2023-10-30T14:19:03.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.844+0000 7fb28c794640 1 -- 172.21.15.57:0/4255678398 shutdown_connections 2023-10-30T14:19:03.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.844+0000 7fb28c794640 1 --2- 172.21.15.57:0/4255678398 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb2680664a0 0x7fb268068960 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:03.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.844+0000 7fb28c794640 1 --2- 172.21.15.57:0/4255678398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb284153bb0 0x7fb28414fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:03.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.844+0000 7fb28c794640 1 --2- 172.21.15.57:0/4255678398 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2840c7170 0x7fb284155210 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:03.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.844+0000 7fb28c794640 1 --2- 172.21.15.57:0/4255678398 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb28406c420 0x7fb284152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:03.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.844+0000 7fb28c794640 1 -- 172.21.15.57:0/4255678398 >> 172.21.15.57:0/4255678398 conn(0x7fb2840cac70 msgr2=0x7fb2840c96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:03.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.848+0000 7fb28c794640 1 -- 172.21.15.57:0/4255678398 shutdown_connections 2023-10-30T14:19:03.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:03.848+0000 7fb28c794640 1 -- 172.21.15.57:0/4255678398 wait complete. 2023-10-30T14:19:03.871 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:19:05.872 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:19:06.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.020+0000 7f660b868640 1 Processor -- start 2023-10-30T14:19:06.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.020+0000 7f660b868640 1 -- start start 2023-10-30T14:19:06.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.024+0000 7f660b868640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f66040cb7d0 0x7f66040c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:06.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.024+0000 7f660b868640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66040c3e40 0x7f6604070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:06.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.024+0000 7f660b868640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66040c2310 con 0x7f6604150830 2023-10-30T14:19:06.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.024+0000 7f660b868640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f660406bc10 con 0x7f66040cb7d0 2023-10-30T14:19:06.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.024+0000 7f660b868640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6604156710 con 0x7f66040c3e40 2023-10-30T14:19:06.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.024+0000 7f6609e05640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66040c3e40 0x7f6604070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:06.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.024+0000 7f6608e03640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f66040cb7d0 0x7f66040c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:06.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.024+0000 7f6609e05640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66040c3e40 0x7f6604070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:39734/0 (socket says 172.21.15.57:39734) 2023-10-30T14:19:06.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.024+0000 7f6609e05640 1 -- 172.21.15.57:0/1384750982 learned_addr learned my addr 172.21.15.57:0/1384750982 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:19:06.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.024+0000 7f6609604640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6604150830 0x7f660406b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:50222/0 (socket says 172.21.15.57:50222) 2023-10-30T14:19:06.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.024+0000 7f6608e03640 1 -- 172.21.15.57:0/1384750982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66040c3e40 msgr2=0x7f6604070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:06.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.024+0000 7f6608e03640 1 --2- 172.21.15.57:0/1384750982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66040c3e40 0x7f6604070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:06.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.024+0000 7f6608e03640 1 -- 172.21.15.57:0/1384750982 >> v1:172.21.15.88:6789/0 conn(0x7f6604150830 legacy=0x7f660406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:06.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.024+0000 7f6608e03640 1 -- 172.21.15.57:0/1384750982 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66040c4360 con 0x7f66040cb7d0 2023-10-30T14:19:06.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.024+0000 7f6609e05640 1 --2- 172.21.15.57:0/1384750982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66040c3e40 0x7f6604070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:19:06.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.028+0000 7f6608e03640 1 --2- 172.21.15.57:0/1384750982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66040cb7d0 0x7f66040c3900 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f65f40097c0 tx=0x7f65f401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b382268efd5d17a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:06.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.028+0000 7f65fbfff640 1 -- 172.21.15.57:0/1384750982 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f40a8970 con 0x7f66040cb7d0 2023-10-30T14:19:06.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.028+0000 7f65fbfff640 1 -- 172.21.15.57:0/1384750982 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f40a8ad0 con 0x7f66040cb7d0 2023-10-30T14:19:06.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.028+0000 7f65fbfff640 1 -- 172.21.15.57:0/1384750982 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f40a8e60 con 0x7f66040cb7d0 2023-10-30T14:19:06.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.028+0000 7f660b868640 1 -- 172.21.15.57:0/1384750982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66040cb7d0 msgr2=0x7f66040c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:06.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.028+0000 7f660b868640 1 --2- 172.21.15.57:0/1384750982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66040cb7d0 0x7f66040c3900 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f65f40097c0 tx=0x7f65f401ec70 comp rx=0 tx=0).stop 2023-10-30T14:19:06.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.028+0000 7f660b868640 1 -- 172.21.15.57:0/1384750982 shutdown_connections 2023-10-30T14:19:06.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.028+0000 7f660b868640 1 --2- 172.21.15.57:0/1384750982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66040c3e40 0x7f6604070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:06.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.028+0000 7f660b868640 1 --2- 172.21.15.57:0/1384750982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66040cb7d0 0x7f66040c3900 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:06.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.028+0000 7f660b868640 1 -- 172.21.15.57:0/1384750982 >> 172.21.15.57:0/1384750982 conn(0x7f66040cac70 msgr2=0x7f66040cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:06.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.028+0000 7f660b868640 1 -- 172.21.15.57:0/1384750982 shutdown_connections 2023-10-30T14:19:06.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.028+0000 7f660b868640 1 -- 172.21.15.57:0/1384750982 wait complete. 2023-10-30T14:19:06.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.028+0000 7f660b868640 1 Processor -- start 2023-10-30T14:19:06.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.028+0000 7f660b868640 1 -- start start 2023-10-30T14:19:06.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f660b868640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66040c3e40 0x7f6604148aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:06.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f660b868640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66040cb7d0 0x7f6604148fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:06.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f660b868640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6604150830 0x7f6604157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:06.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f660b868640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6604158910 con 0x7f66040cb7d0 2023-10-30T14:19:06.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f660b868640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66041485c0 con 0x7f66040c3e40 2023-10-30T14:19:06.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f660b868640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66041eac50 con 0x7f6604150830 2023-10-30T14:19:06.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f6608e03640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66040cb7d0 0x7f6604148fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:06.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f6608e03640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66040cb7d0 0x7f6604148fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33532/0 (socket says 172.21.15.57:33532) 2023-10-30T14:19:06.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f6608e03640 1 -- 172.21.15.57:0/768681974 learned_addr learned my addr 172.21.15.57:0/768681974 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:19:06.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f6609604640 1 --2- 172.21.15.57:0/768681974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66040c3e40 0x7f6604148aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:06.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f6609e05640 1 --2- 172.21.15.57:0/768681974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6604150830 0x7f6604157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:06.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f6608e03640 1 -- 172.21.15.57:0/768681974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6604150830 msgr2=0x7f6604157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:06.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f6608e03640 1 --2- 172.21.15.57:0/768681974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6604150830 0x7f6604157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:06.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f6608e03640 1 -- 172.21.15.57:0/768681974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66040c3e40 msgr2=0x7f6604148aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:06.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f6608e03640 1 --2- 172.21.15.57:0/768681974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66040c3e40 0x7f6604148aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:06.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f6608e03640 1 -- 172.21.15.57:0/768681974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65fc009b30 con 0x7f66040cb7d0 2023-10-30T14:19:06.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f6608e03640 1 --2- 172.21.15.57:0/768681974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66040cb7d0 0x7f6604148fe0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f65f40b3040 tx=0x7f65f401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:06.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f65f9ffb640 1 -- 172.21.15.57:0/768681974 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f40b9630 con 0x7f66040cb7d0 2023-10-30T14:19:06.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f65f9ffb640 1 -- 172.21.15.57:0/768681974 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f40b97b0 con 0x7f66040cb7d0 2023-10-30T14:19:06.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f65f9ffb640 1 -- 172.21.15.57:0/768681974 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f40b9b60 con 0x7f66040cb7d0 2023-10-30T14:19:06.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f660b868640 1 -- 172.21.15.57:0/768681974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f4009420 con 0x7f66040cb7d0 2023-10-30T14:19:06.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f660b868640 1 -- 172.21.15.57:0/768681974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66041eb1f0 con 0x7f66040cb7d0 2023-10-30T14:19:06.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.032+0000 7f660b868640 1 -- 172.21.15.57:0/768681974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6604066510 con 0x7f66040cb7d0 2023-10-30T14:19:06.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.036+0000 7f65f9ffb640 1 -- 172.21.15.57:0/768681974 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f65f40b7040 con 0x7f66040cb7d0 2023-10-30T14:19:06.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.036+0000 7f65f9ffb640 1 --2- 172.21.15.57:0/768681974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65e80664a0 0x7f65e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:06.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.036+0000 7f65f9ffb640 1 -- 172.21.15.57:0/768681974 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f65f4121f30 con 0x7f66040cb7d0 2023-10-30T14:19:06.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.036+0000 7f6609604640 1 --2- 172.21.15.57:0/768681974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65e80664a0 0x7f65e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:06.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.036+0000 7f6609604640 1 --2- 172.21.15.57:0/768681974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65e80664a0 0x7f65e8068960 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6600000f70 tx=0x7f6600007e50 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:06.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.040+0000 7f65f9ffb640 1 -- 172.21.15.57:0/768681974 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f65f40ef1f0 con 0x7f66040cb7d0 2023-10-30T14:19:06.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.232+0000 7f660b868640 1 -- 172.21.15.57:0/768681974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f660404e170 con 0x7f66040cb7d0 2023-10-30T14:19:06.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.232+0000 7f65f9ffb640 1 -- 172.21.15.57:0/768681974 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f65f40b5020 con 0x7f66040cb7d0 2023-10-30T14:19:06.238 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:19:06.239 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:19:00.507867+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:resolve","state_seq":15,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:19:06.239 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 36 2023-10-30T14:19:06.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.236+0000 7f660b868640 1 -- 172.21.15.57:0/768681974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65e80664a0 msgr2=0x7f65e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:06.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.236+0000 7f660b868640 1 --2- 172.21.15.57:0/768681974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65e80664a0 0x7f65e8068960 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6600000f70 tx=0x7f6600007e50 comp rx=0 tx=0).stop 2023-10-30T14:19:06.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.236+0000 7f660b868640 1 -- 172.21.15.57:0/768681974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66040cb7d0 msgr2=0x7f6604148fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:06.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.236+0000 7f660b868640 1 --2- 172.21.15.57:0/768681974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66040cb7d0 0x7f6604148fe0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f65f40b3040 tx=0x7f65f401ba10 comp rx=0 tx=0).stop 2023-10-30T14:19:06.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.236+0000 7f660b868640 1 -- 172.21.15.57:0/768681974 shutdown_connections 2023-10-30T14:19:06.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.236+0000 7f660b868640 1 --2- 172.21.15.57:0/768681974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65e80664a0 0x7f65e8068960 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:06.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.236+0000 7f660b868640 1 --2- 172.21.15.57:0/768681974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6604150830 0x7f6604157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:06.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.236+0000 7f660b868640 1 --2- 172.21.15.57:0/768681974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66040cb7d0 0x7f6604148fe0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:06.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.236+0000 7f660b868640 1 --2- 172.21.15.57:0/768681974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66040c3e40 0x7f6604148aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:06.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.236+0000 7f660b868640 1 -- 172.21.15.57:0/768681974 >> 172.21.15.57:0/768681974 conn(0x7f66040cac70 msgr2=0x7f6604070520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:06.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.236+0000 7f660b868640 1 -- 172.21.15.57:0/768681974 shutdown_connections 2023-10-30T14:19:06.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:06.236+0000 7f660b868640 1 -- 172.21.15.57:0/768681974 wait complete. 2023-10-30T14:19:06.259 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:19:06.953 INFO:tasks.workunit.client.0.smithi057.stdout:./104/.snap/snap-subdir-test 2023-10-30T14:19:06.960 INFO:tasks.workunit.client.0.smithi057.stdout:./164/.snap/snap-subdir-test 2023-10-30T14:19:08.260 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:19:08.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.412+0000 7fd70b1c5640 1 Processor -- start 2023-10-30T14:19:08.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.412+0000 7fd70b1c5640 1 -- start start 2023-10-30T14:19:08.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.412+0000 7fd70b1c5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd704153bb0 0x7fd7040c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:08.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.412+0000 7fd70b1c5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd70406c420 0x7fd7040c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:08.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.412+0000 7fd70b1c5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd7040c7170 0x7fd704159980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:08.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.412+0000 7fd70b1c5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7040c5f80 con 0x7fd7040c7170 2023-10-30T14:19:08.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.412+0000 7fd70b1c5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7040c5690 con 0x7fd704153bb0 2023-10-30T14:19:08.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.412+0000 7fd70b1c5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd70415a580 con 0x7fd70406c420 2023-10-30T14:19:08.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.412+0000 7fd709762640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd7040c7170 0x7fd704159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:08.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.412+0000 7fd709762640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd7040c7170 0x7fd704159980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33544/0 (socket says 172.21.15.57:33544) 2023-10-30T14:19:08.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.412+0000 7fd709762640 1 -- 172.21.15.57:0/1012257024 learned_addr learned my addr 172.21.15.57:0/1012257024 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:19:08.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.412+0000 7fd708f61640 1 --2- 172.21.15.57:0/1012257024 >> v2:172.21.15.88:3300/0 conn(0x7fd704153bb0 0x7fd7040c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:08.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.412+0000 7fd703fff640 1 --2- 172.21.15.57:0/1012257024 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd70406c420 0x7fd7040c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:08.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.412+0000 7fd709762640 1 -- 172.21.15.57:0/1012257024 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd70406c420 msgr2=0x7fd7040c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:08.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.412+0000 7fd709762640 1 --2- 172.21.15.57:0/1012257024 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd70406c420 0x7fd7040c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:08.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.412+0000 7fd709762640 1 -- 172.21.15.57:0/1012257024 >> v2:172.21.15.88:3300/0 conn(0x7fd704153bb0 msgr2=0x7fd7040c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:08.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.412+0000 7fd709762640 1 --2- 172.21.15.57:0/1012257024 >> v2:172.21.15.88:3300/0 conn(0x7fd704153bb0 0x7fd7040c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:08.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.412+0000 7fd709762640 1 -- 172.21.15.57:0/1012257024 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7040c5930 con 0x7fd7040c7170 2023-10-30T14:19:08.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.412+0000 7fd703fff640 1 --2- 172.21.15.57:0/1012257024 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd70406c420 0x7fd7040c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:19:08.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.416+0000 7fd709762640 1 --2- 172.21.15.57:0/1012257024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7040c7170 0x7fd704159980 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd7040cb840 tx=0x7fd6fc01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=174588a74f07908 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:08.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.416+0000 7fd7037fe640 1 -- 172.21.15.57:0/1012257024 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc0af400 con 0x7fd7040c7170 2023-10-30T14:19:08.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.416+0000 7fd7037fe640 1 -- 172.21.15.57:0/1012257024 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6fc0ab050 con 0x7fd7040c7170 2023-10-30T14:19:08.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.416+0000 7fd7037fe640 1 -- 172.21.15.57:0/1012257024 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc0af780 con 0x7fd7040c7170 2023-10-30T14:19:08.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.416+0000 7fd70b1c5640 1 -- 172.21.15.57:0/1012257024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7040c7170 msgr2=0x7fd704159980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:08.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.416+0000 7fd70b1c5640 1 --2- 172.21.15.57:0/1012257024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7040c7170 0x7fd704159980 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd7040cb840 tx=0x7fd6fc01f1c0 comp rx=0 tx=0).stop 2023-10-30T14:19:08.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.416+0000 7fd70b1c5640 1 -- 172.21.15.57:0/1012257024 shutdown_connections 2023-10-30T14:19:08.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.416+0000 7fd70b1c5640 1 --2- 172.21.15.57:0/1012257024 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd70406c420 0x7fd7040c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:08.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.416+0000 7fd70b1c5640 1 --2- 172.21.15.57:0/1012257024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7040c7170 0x7fd704159980 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:08.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.416+0000 7fd70b1c5640 1 --2- 172.21.15.57:0/1012257024 >> v2:172.21.15.88:3300/0 conn(0x7fd704153bb0 0x7fd7040c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:08.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.416+0000 7fd70b1c5640 1 -- 172.21.15.57:0/1012257024 >> 172.21.15.57:0/1012257024 conn(0x7fd7040cac70 msgr2=0x7fd7040cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:08.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.416+0000 7fd70b1c5640 1 -- 172.21.15.57:0/1012257024 shutdown_connections 2023-10-30T14:19:08.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.416+0000 7fd70b1c5640 1 -- 172.21.15.57:0/1012257024 wait complete. 2023-10-30T14:19:08.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd70b1c5640 1 Processor -- start 2023-10-30T14:19:08.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd70b1c5640 1 -- start start 2023-10-30T14:19:08.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd70b1c5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd70406c420 0x7fd7041f0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:08.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd70b1c5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7040c7170 0x7fd7041f2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:08.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd70b1c5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd704153bb0 0x7fd70414c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:08.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd70b1c5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70415a330 con 0x7fd704153bb0 2023-10-30T14:19:08.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd70b1c5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7041f0f70 con 0x7fd7040c7170 2023-10-30T14:19:08.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd70b1c5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7041594e0 con 0x7fd70406c420 2023-10-30T14:19:08.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd709762640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd704153bb0 0x7fd70414c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:08.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd709762640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd704153bb0 0x7fd70414c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33546/0 (socket says 172.21.15.57:33546) 2023-10-30T14:19:08.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd709762640 1 -- 172.21.15.57:0/2116729259 learned_addr learned my addr 172.21.15.57:0/2116729259 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:19:08.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd708f61640 1 --2- 172.21.15.57:0/2116729259 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd70406c420 0x7fd7041f0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:08.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd703fff640 1 --2- 172.21.15.57:0/2116729259 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7040c7170 0x7fd7041f2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:08.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd709762640 1 -- 172.21.15.57:0/2116729259 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd70406c420 msgr2=0x7fd7041f0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:08.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd709762640 1 --2- 172.21.15.57:0/2116729259 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd70406c420 0x7fd7041f0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:08.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd709762640 1 -- 172.21.15.57:0/2116729259 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7040c7170 msgr2=0x7fd7041f2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:08.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd709762640 1 --2- 172.21.15.57:0/2116729259 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7040c7170 0x7fd7041f2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:08.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd709762640 1 -- 172.21.15.57:0/2116729259 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc009b30 con 0x7fd704153bb0 2023-10-30T14:19:08.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd708f61640 1 --2- 172.21.15.57:0/2116729259 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd70406c420 0x7fd7041f0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:19:08.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd709762640 1 --2- 172.21.15.57:0/2116729259 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd704153bb0 0x7fd70414c720 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc01fa60 tx=0x7fd6fc01bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:08.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd7017fa640 1 -- 172.21.15.57:0/2116729259 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc0afd60 con 0x7fd704153bb0 2023-10-30T14:19:08.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd7017fa640 1 -- 172.21.15.57:0/2116729259 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6fc009510 con 0x7fd704153bb0 2023-10-30T14:19:08.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd7017fa640 1 -- 172.21.15.57:0/2116729259 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc0ba680 con 0x7fd704153bb0 2023-10-30T14:19:08.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd70b1c5640 1 -- 172.21.15.57:0/2116729259 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7041597b0 con 0x7fd704153bb0 2023-10-30T14:19:08.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.420+0000 7fd70b1c5640 1 -- 172.21.15.57:0/2116729259 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7041f2ec0 con 0x7fd704153bb0 2023-10-30T14:19:08.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.424+0000 7fd70b1c5640 1 -- 172.21.15.57:0/2116729259 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd70404e170 con 0x7fd704153bb0 2023-10-30T14:19:08.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.424+0000 7fd7017fa640 1 -- 172.21.15.57:0/2116729259 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd6fc0b6040 con 0x7fd704153bb0 2023-10-30T14:19:08.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.424+0000 7fd7017fa640 1 --2- 172.21.15.57:0/2116729259 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd6e80664a0 0x7fd6e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:08.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.424+0000 7fd7017fa640 1 -- 172.21.15.57:0/2116729259 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fd6fc122280 con 0x7fd704153bb0 2023-10-30T14:19:08.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.424+0000 7fd708f61640 1 --2- 172.21.15.57:0/2116729259 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd6e80664a0 0x7fd6e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:08.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.424+0000 7fd708f61640 1 --2- 172.21.15.57:0/2116729259 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd6e80664a0 0x7fd6e8068960 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8007a70 tx=0x7fd6f8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:08.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.428+0000 7fd7017fa640 1 -- 172.21.15.57:0/2116729259 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd6fc0ef560 con 0x7fd704153bb0 2023-10-30T14:19:08.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.620+0000 7fd70b1c5640 1 -- 172.21.15.57:0/2116729259 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd704066510 con 0x7fd704153bb0 2023-10-30T14:19:08.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.620+0000 7fd7017fa640 1 -- 172.21.15.57:0/2116729259 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd6fc0f2b80 con 0x7fd704153bb0 2023-10-30T14:19:08.625 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:19:08.625 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:19:06.064908+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:19:08.625 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 37 2023-10-30T14:19:08.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.620+0000 7fd70b1c5640 1 -- 172.21.15.57:0/2116729259 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd6e80664a0 msgr2=0x7fd6e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:08.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.620+0000 7fd70b1c5640 1 --2- 172.21.15.57:0/2116729259 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd6e80664a0 0x7fd6e8068960 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8007a70 tx=0x7fd6f8008040 comp rx=0 tx=0).stop 2023-10-30T14:19:08.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.620+0000 7fd70b1c5640 1 -- 172.21.15.57:0/2116729259 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd704153bb0 msgr2=0x7fd70414c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:08.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.620+0000 7fd70b1c5640 1 --2- 172.21.15.57:0/2116729259 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd704153bb0 0x7fd70414c720 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc01fa60 tx=0x7fd6fc01bf60 comp rx=0 tx=0).stop 2023-10-30T14:19:08.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.620+0000 7fd70b1c5640 1 -- 172.21.15.57:0/2116729259 shutdown_connections 2023-10-30T14:19:08.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.620+0000 7fd70b1c5640 1 --2- 172.21.15.57:0/2116729259 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd6e80664a0 0x7fd6e8068960 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:08.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.620+0000 7fd70b1c5640 1 --2- 172.21.15.57:0/2116729259 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd704153bb0 0x7fd70414c720 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:08.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.620+0000 7fd70b1c5640 1 --2- 172.21.15.57:0/2116729259 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7040c7170 0x7fd7041f2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:08.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.620+0000 7fd70b1c5640 1 --2- 172.21.15.57:0/2116729259 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd70406c420 0x7fd7041f0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:08.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.620+0000 7fd70b1c5640 1 -- 172.21.15.57:0/2116729259 >> 172.21.15.57:0/2116729259 conn(0x7fd7040cac70 msgr2=0x7fd704152780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:08.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.620+0000 7fd70b1c5640 1 -- 172.21.15.57:0/2116729259 shutdown_connections 2023-10-30T14:19:08.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:08.620+0000 7fd70b1c5640 1 -- 172.21.15.57:0/2116729259 wait complete. 2023-10-30T14:19:08.647 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 37, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 37, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4936": { "addr": "172.21.15.57:6837/1988100930", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 1988100930, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 1988100930, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4936, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 31 }, "gid_5133": { "addr": "172.21.15.57:6834/2912050965", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2912050965, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2912050965, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5133, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 15 }, "gid_5289": { "addr": "172.21.15.57:6836/2427081813", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2427081813, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2427081813, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5289, "incarnation": 35, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:19:06.064908+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5289, "mds_1": 5133, "mds_2": 4936 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:19:10.648 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:19:10.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.800+0000 7f3b6c609640 1 Processor -- start 2023-10-30T14:19:10.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.800+0000 7f3b6c609640 1 -- start start 2023-10-30T14:19:10.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.800+0000 7f3b6c609640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3b64153c80 0x7f3b6415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:10.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.800+0000 7f3b6c609640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b640c42a0 0x7f3b640ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:10.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.800+0000 7f3b6c609640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b64159f20 con 0x7f3b640c4a10 2023-10-30T14:19:10.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.800+0000 7f3b6c609640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6415dec0 con 0x7f3b64153c80 2023-10-30T14:19:10.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.800+0000 7f3b6c609640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b641599a0 con 0x7f3b640c42a0 2023-10-30T14:19:10.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.800+0000 7f3b6a3a5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3b64153c80 0x7f3b6415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:10.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.800+0000 7f3b6aba6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b640c42a0 0x7f3b640ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:10.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.800+0000 7f3b6a3a5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3b64153c80 0x7f3b6415cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:55118/0 (socket says 172.21.15.57:55118) 2023-10-30T14:19:10.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.800+0000 7f3b6aba6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b640c42a0 0x7f3b640ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:39766/0 (socket says 172.21.15.57:39766) 2023-10-30T14:19:10.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.800+0000 7f3b6a3a5640 1 -- 172.21.15.57:0/3961603543 learned_addr learned my addr 172.21.15.57:0/3961603543 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:19:10.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.800+0000 7f3b69ba4640 1 --1- 172.21.15.57:0/3961603543 >> v1:172.21.15.88:6789/0 conn(0x7f3b640c4a10 0x7f3b6415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:50228/0 (socket says 172.21.15.57:50228) 2023-10-30T14:19:10.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.800+0000 7f3b6a3a5640 1 -- 172.21.15.57:0/3961603543 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b640c42a0 msgr2=0x7f3b640ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:10.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.800+0000 7f3b6a3a5640 1 --2- 172.21.15.57:0/3961603543 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b640c42a0 0x7f3b640ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:10.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.800+0000 7f3b6a3a5640 1 -- 172.21.15.57:0/3961603543 >> v1:172.21.15.88:6789/0 conn(0x7f3b640c4a10 legacy=0x7f3b6415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:10.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.800+0000 7f3b6a3a5640 1 -- 172.21.15.57:0/3961603543 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6415d620 con 0x7f3b64153c80 2023-10-30T14:19:10.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.800+0000 7f3b6aba6640 1 --2- 172.21.15.57:0/3961603543 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b640c42a0 0x7f3b640ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:19:10.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.804+0000 7f3b6a3a5640 1 --2- 172.21.15.57:0/3961603543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b64153c80 0x7f3b6415cf90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3b60009870 tx=0x7f3b6001ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1121810318e8b91 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:10.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.804+0000 7f3b693a3640 1 -- 172.21.15.57:0/3961603543 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b600a8a30 con 0x7f3b64153c80 2023-10-30T14:19:10.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.804+0000 7f3b693a3640 1 -- 172.21.15.57:0/3961603543 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b600a8b90 con 0x7f3b64153c80 2023-10-30T14:19:10.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.804+0000 7f3b693a3640 1 -- 172.21.15.57:0/3961603543 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b600b0440 con 0x7f3b64153c80 2023-10-30T14:19:10.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.804+0000 7f3b6c609640 1 -- 172.21.15.57:0/3961603543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b64153c80 msgr2=0x7f3b6415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:10.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.804+0000 7f3b6c609640 1 --2- 172.21.15.57:0/3961603543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b64153c80 0x7f3b6415cf90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3b60009870 tx=0x7f3b6001ed30 comp rx=0 tx=0).stop 2023-10-30T14:19:10.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.804+0000 7f3b6c609640 1 -- 172.21.15.57:0/3961603543 shutdown_connections 2023-10-30T14:19:10.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.804+0000 7f3b6c609640 1 --2- 172.21.15.57:0/3961603543 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b640c42a0 0x7f3b640ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:10.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.804+0000 7f3b6c609640 1 --2- 172.21.15.57:0/3961603543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b64153c80 0x7f3b6415cf90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:10.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.804+0000 7f3b6c609640 1 -- 172.21.15.57:0/3961603543 >> 172.21.15.57:0/3961603543 conn(0x7f3b640715c0 msgr2=0x7f3b640719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:10.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.804+0000 7f3b6c609640 1 -- 172.21.15.57:0/3961603543 shutdown_connections 2023-10-30T14:19:10.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.804+0000 7f3b6c609640 1 -- 172.21.15.57:0/3961603543 wait complete. 2023-10-30T14:19:10.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6c609640 1 Processor -- start 2023-10-30T14:19:10.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6c609640 1 -- start start 2023-10-30T14:19:10.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6c609640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b640c42a0 0x7f3b64149400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:10.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6c609640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b640c4a10 0x7f3b6415cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:10.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6c609640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b64153c80 0x7f3b641560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:10.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6c609640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b64156f50 con 0x7f3b64153c80 2023-10-30T14:19:10.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6c609640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b64149940 con 0x7f3b640c4a10 2023-10-30T14:19:10.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6c609640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b64154d10 con 0x7f3b640c42a0 2023-10-30T14:19:10.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6aba6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b64153c80 0x7f3b641560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:10.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6aba6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b64153c80 0x7f3b641560b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33550/0 (socket says 172.21.15.57:33550) 2023-10-30T14:19:10.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6aba6640 1 -- 172.21.15.57:0/560143753 learned_addr learned my addr 172.21.15.57:0/560143753 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:19:10.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b69ba4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b640c4a10 0x7f3b6415cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:10.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6a3a5640 1 --2- 172.21.15.57:0/560143753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b640c42a0 0x7f3b64149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:10.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6aba6640 1 -- 172.21.15.57:0/560143753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b640c42a0 msgr2=0x7f3b64149400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:10.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6aba6640 1 --2- 172.21.15.57:0/560143753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b640c42a0 0x7f3b64149400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:10.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6aba6640 1 -- 172.21.15.57:0/560143753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b640c4a10 msgr2=0x7f3b6415cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:10.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6aba6640 1 --2- 172.21.15.57:0/560143753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b640c4a10 0x7f3b6415cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:10.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6aba6640 1 -- 172.21.15.57:0/560143753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c009a70 con 0x7f3b64153c80 2023-10-30T14:19:10.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6aba6640 1 --2- 172.21.15.57:0/560143753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b64153c80 0x7f3b641560b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00efc0 tx=0x7f3b5c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:10.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b5affd640 1 -- 172.21.15.57:0/560143753 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c009d70 con 0x7f3b64153c80 2023-10-30T14:19:10.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b5affd640 1 -- 172.21.15.57:0/560143753 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5c00f040 con 0x7f3b64153c80 2023-10-30T14:19:10.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b5affd640 1 -- 172.21.15.57:0/560143753 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c0039a0 con 0x7f3b64153c80 2023-10-30T14:19:10.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6c609640 1 -- 172.21.15.57:0/560143753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b600094d0 con 0x7f3b64153c80 2023-10-30T14:19:10.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.808+0000 7f3b6c609640 1 -- 172.21.15.57:0/560143753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b641552d0 con 0x7f3b64153c80 2023-10-30T14:19:10.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.812+0000 7f3b6c609640 1 -- 172.21.15.57:0/560143753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b641554f0 con 0x7f3b64153c80 2023-10-30T14:19:10.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.812+0000 7f3b5affd640 1 -- 172.21.15.57:0/560143753 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3b5c0144a0 con 0x7f3b64153c80 2023-10-30T14:19:10.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.812+0000 7f3b5affd640 1 --2- 172.21.15.57:0/560143753 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3b480664a0 0x7f3b48068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:10.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.812+0000 7f3b5affd640 1 -- 172.21.15.57:0/560143753 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f3b5c084ca0 con 0x7f3b64153c80 2023-10-30T14:19:10.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.812+0000 7f3b6a3a5640 1 --2- 172.21.15.57:0/560143753 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3b480664a0 0x7f3b48068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:10.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.812+0000 7f3b6a3a5640 1 --2- 172.21.15.57:0/560143753 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3b480664a0 0x7f3b48068960 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3b600b2000 tx=0x7f3b60009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:10.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:10.816+0000 7f3b5affd640 1 -- 172.21.15.57:0/560143753 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3b5c051ed0 con 0x7f3b64153c80 2023-10-30T14:19:11.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:11.020+0000 7f3b6c609640 1 -- 172.21.15.57:0/560143753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b6404e170 con 0x7f3b64153c80 2023-10-30T14:19:11.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:11.020+0000 7f3b5affd640 1 -- 172.21.15.57:0/560143753 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3b5c018020 con 0x7f3b64153c80 2023-10-30T14:19:11.025 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:19:11.025 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:19:06.064908+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:19:11.026 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 37 2023-10-30T14:19:11.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:11.020+0000 7f3b6c609640 1 -- 172.21.15.57:0/560143753 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3b480664a0 msgr2=0x7f3b48068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:11.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:11.020+0000 7f3b6c609640 1 --2- 172.21.15.57:0/560143753 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3b480664a0 0x7f3b48068960 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3b600b2000 tx=0x7f3b60009040 comp rx=0 tx=0).stop 2023-10-30T14:19:11.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:11.020+0000 7f3b6c609640 1 -- 172.21.15.57:0/560143753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b64153c80 msgr2=0x7f3b641560b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:11.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:11.020+0000 7f3b6c609640 1 --2- 172.21.15.57:0/560143753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b64153c80 0x7f3b641560b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00efc0 tx=0x7f3b5c009f90 comp rx=0 tx=0).stop 2023-10-30T14:19:11.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:11.020+0000 7f3b6c609640 1 -- 172.21.15.57:0/560143753 shutdown_connections 2023-10-30T14:19:11.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:11.020+0000 7f3b6c609640 1 --2- 172.21.15.57:0/560143753 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3b480664a0 0x7f3b48068960 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:11.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:11.020+0000 7f3b6c609640 1 --2- 172.21.15.57:0/560143753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b64153c80 0x7f3b641560b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:11.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:11.020+0000 7f3b6c609640 1 --2- 172.21.15.57:0/560143753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b640c4a10 0x7f3b6415cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:11.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:11.020+0000 7f3b6c609640 1 --2- 172.21.15.57:0/560143753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b640c42a0 0x7f3b64149400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:11.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:11.020+0000 7f3b6c609640 1 -- 172.21.15.57:0/560143753 >> 172.21.15.57:0/560143753 conn(0x7f3b640715c0 msgr2=0x7f3b64157cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:11.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:11.020+0000 7f3b6c609640 1 -- 172.21.15.57:0/560143753 shutdown_connections 2023-10-30T14:19:11.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:11.024+0000 7f3b6c609640 1 -- 172.21.15.57:0/560143753 wait complete. 2023-10-30T14:19:11.046 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:19:11.930 INFO:tasks.workunit.client.0.smithi057.stdout:./112/.snap/snap-subdir-test 2023-10-30T14:19:11.935 INFO:tasks.workunit.client.0.smithi057.stdout:./295/.snap/snap-subdir-test 2023-10-30T14:19:13.047 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:19:13.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.196+0000 7f918c5d3640 1 Processor -- start 2023-10-30T14:19:13.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.196+0000 7f918c5d3640 1 -- start start 2023-10-30T14:19:13.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.196+0000 7f918c5d3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f91840c8060 0x7f918406f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:13.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.196+0000 7f918c5d3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f91840cb8b0 0x7f918415a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:13.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.196+0000 7f918c5d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91840ca9b0 0x7f91840c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:13.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.200+0000 7f918c5d3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9184150d00 con 0x7f91840c8060 2023-10-30T14:19:13.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.200+0000 7f918c5d3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91840ca740 con 0x7f91840cb8b0 2023-10-30T14:19:13.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.200+0000 7f918c5d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91840ca180 con 0x7f91840ca9b0 2023-10-30T14:19:13.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.200+0000 7f918a36f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f91840c8060 0x7f918406f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:13.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.200+0000 7f918a36f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f91840c8060 0x7f918406f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33566/0 (socket says 172.21.15.57:33566) 2023-10-30T14:19:13.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.200+0000 7f918a36f640 1 -- 172.21.15.57:0/1459454809 learned_addr learned my addr 172.21.15.57:0/1459454809 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:19:13.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.200+0000 7f918ab70640 1 --2- 172.21.15.57:0/1459454809 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91840ca9b0 0x7f91840c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:13.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.200+0000 7f9189b6e640 1 --2- 172.21.15.57:0/1459454809 >> v2:172.21.15.88:3300/0 conn(0x7f91840cb8b0 0x7f918415a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:13.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.200+0000 7f918a36f640 1 -- 172.21.15.57:0/1459454809 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91840ca9b0 msgr2=0x7f91840c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:13.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.200+0000 7f918a36f640 1 --2- 172.21.15.57:0/1459454809 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91840ca9b0 0x7f91840c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:13.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.200+0000 7f918a36f640 1 -- 172.21.15.57:0/1459454809 >> v2:172.21.15.88:3300/0 conn(0x7f91840cb8b0 msgr2=0x7f918415a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:13.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.200+0000 7f918a36f640 1 --2- 172.21.15.57:0/1459454809 >> v2:172.21.15.88:3300/0 conn(0x7f91840cb8b0 0x7f918415a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:13.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.200+0000 7f918a36f640 1 -- 172.21.15.57:0/1459454809 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91840ca460 con 0x7f91840c8060 2023-10-30T14:19:13.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.200+0000 7f918a36f640 1 --2- 172.21.15.57:0/1459454809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91840c8060 0x7f918406f290 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f918406e290 tx=0x7f918001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b666eee6e6040c10 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:13.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.200+0000 7f918936d640 1 -- 172.21.15.57:0/1459454809 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91800a89b0 con 0x7f91840c8060 2023-10-30T14:19:13.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.200+0000 7f918936d640 1 -- 172.21.15.57:0/1459454809 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91800a8b10 con 0x7f91840c8060 2023-10-30T14:19:13.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.200+0000 7f918936d640 1 -- 172.21.15.57:0/1459454809 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91800a8ea0 con 0x7f91840c8060 2023-10-30T14:19:13.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918c5d3640 1 -- 172.21.15.57:0/1459454809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91840c8060 msgr2=0x7f918406f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:13.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918c5d3640 1 --2- 172.21.15.57:0/1459454809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91840c8060 0x7f918406f290 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f918406e290 tx=0x7f918001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:19:13.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918c5d3640 1 -- 172.21.15.57:0/1459454809 shutdown_connections 2023-10-30T14:19:13.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918c5d3640 1 --2- 172.21.15.57:0/1459454809 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91840ca9b0 0x7f91840c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:13.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918c5d3640 1 --2- 172.21.15.57:0/1459454809 >> v2:172.21.15.88:3300/0 conn(0x7f91840cb8b0 0x7f918415a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:13.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918c5d3640 1 --2- 172.21.15.57:0/1459454809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91840c8060 0x7f918406f290 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:13.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918c5d3640 1 -- 172.21.15.57:0/1459454809 >> 172.21.15.57:0/1459454809 conn(0x7f918406afa0 msgr2=0x7f918406b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:13.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918c5d3640 1 -- 172.21.15.57:0/1459454809 shutdown_connections 2023-10-30T14:19:13.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918c5d3640 1 -- 172.21.15.57:0/1459454809 wait complete. 2023-10-30T14:19:13.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918c5d3640 1 Processor -- start 2023-10-30T14:19:13.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918c5d3640 1 -- start start 2023-10-30T14:19:13.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918c5d3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91840c8060 0x7f91841f1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:13.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918c5d3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91840ca9b0 0x7f91840c51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:13.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918c5d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91840cb8b0 0x7f91840c56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:13.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918c5d3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91841e8ec0 con 0x7f91840c8060 2023-10-30T14:19:13.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918c5d3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91840c5c30 con 0x7f91840ca9b0 2023-10-30T14:19:13.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918c5d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9184157700 con 0x7f91840cb8b0 2023-10-30T14:19:13.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918a36f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91840c8060 0x7f91841f1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:13.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918a36f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91840c8060 0x7f91841f1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33578/0 (socket says 172.21.15.57:33578) 2023-10-30T14:19:13.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918a36f640 1 -- 172.21.15.57:0/1443807020 learned_addr learned my addr 172.21.15.57:0/1443807020 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:19:13.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f9189b6e640 1 --2- 172.21.15.57:0/1443807020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91840ca9b0 0x7f91840c51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:13.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918ab70640 1 --2- 172.21.15.57:0/1443807020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91840cb8b0 0x7f91840c56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:13.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918a36f640 1 -- 172.21.15.57:0/1443807020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91840cb8b0 msgr2=0x7f91840c56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:13.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918a36f640 1 --2- 172.21.15.57:0/1443807020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91840cb8b0 0x7f91840c56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:13.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918a36f640 1 -- 172.21.15.57:0/1443807020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91840ca9b0 msgr2=0x7f91840c51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:13.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918a36f640 1 --2- 172.21.15.57:0/1443807020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91840ca9b0 0x7f91840c51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:13.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.204+0000 7f918a36f640 1 -- 172.21.15.57:0/1443807020 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91800094d0 con 0x7f91840c8060 2023-10-30T14:19:13.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.208+0000 7f918a36f640 1 --2- 172.21.15.57:0/1443807020 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91840c8060 0x7f91841f1e60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f91800b3040 tx=0x7f918001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:13.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.208+0000 7f917affd640 1 -- 172.21.15.57:0/1443807020 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91800ba600 con 0x7f91840c8060 2023-10-30T14:19:13.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.208+0000 7f917affd640 1 -- 172.21.15.57:0/1443807020 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91800ba760 con 0x7f91840c8060 2023-10-30T14:19:13.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.208+0000 7f917affd640 1 -- 172.21.15.57:0/1443807020 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91800bab10 con 0x7f91840c8060 2023-10-30T14:19:13.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.208+0000 7f918c5d3640 1 -- 172.21.15.57:0/1443807020 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91841f2930 con 0x7f91840c8060 2023-10-30T14:19:13.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.208+0000 7f918c5d3640 1 -- 172.21.15.57:0/1443807020 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91841e8be0 con 0x7f91840c8060 2023-10-30T14:19:13.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.208+0000 7f918c5d3640 1 -- 172.21.15.57:0/1443807020 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918404e170 con 0x7f91840c8060 2023-10-30T14:19:13.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.208+0000 7f917affd640 1 -- 172.21.15.57:0/1443807020 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f91800bac70 con 0x7f91840c8060 2023-10-30T14:19:13.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.208+0000 7f917affd640 1 --2- 172.21.15.57:0/1443807020 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9168066520 0x7f91680689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:13.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.208+0000 7f917affd640 1 -- 172.21.15.57:0/1443807020 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f91801224d0 con 0x7f91840c8060 2023-10-30T14:19:13.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.212+0000 7f9189b6e640 1 --2- 172.21.15.57:0/1443807020 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9168066520 0x7f91680689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:13.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.212+0000 7f9189b6e640 1 --2- 172.21.15.57:0/1443807020 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9168066520 0x7f91680689e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f917400b930 tx=0x7f9174008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:13.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.212+0000 7f917affd640 1 -- 172.21.15.57:0/1443807020 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f91800ef860 con 0x7f91840c8060 2023-10-30T14:19:13.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.400+0000 7f918c5d3640 1 -- 172.21.15.57:0/1443807020 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91840c77f0 con 0x7f91840c8060 2023-10-30T14:19:13.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.400+0000 7f917affd640 1 -- 172.21.15.57:0/1443807020 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f91800f2e80 con 0x7f91840c8060 2023-10-30T14:19:13.406 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:19:13.407 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:19:11.882149+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:19:13.407 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 38 2023-10-30T14:19:13.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.404+0000 7f918c5d3640 1 -- 172.21.15.57:0/1443807020 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9168066520 msgr2=0x7f91680689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:13.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.404+0000 7f918c5d3640 1 --2- 172.21.15.57:0/1443807020 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9168066520 0x7f91680689e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f917400b930 tx=0x7f9174008040 comp rx=0 tx=0).stop 2023-10-30T14:19:13.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.404+0000 7f918c5d3640 1 -- 172.21.15.57:0/1443807020 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91840c8060 msgr2=0x7f91841f1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:13.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.404+0000 7f918c5d3640 1 --2- 172.21.15.57:0/1443807020 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91840c8060 0x7f91841f1e60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f91800b3040 tx=0x7f918001ba90 comp rx=0 tx=0).stop 2023-10-30T14:19:13.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.404+0000 7f918c5d3640 1 -- 172.21.15.57:0/1443807020 shutdown_connections 2023-10-30T14:19:13.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.404+0000 7f918c5d3640 1 --2- 172.21.15.57:0/1443807020 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9168066520 0x7f91680689e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:13.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.404+0000 7f918c5d3640 1 --2- 172.21.15.57:0/1443807020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91840cb8b0 0x7f91840c56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:13.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.404+0000 7f918c5d3640 1 --2- 172.21.15.57:0/1443807020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91840ca9b0 0x7f91840c51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:13.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.404+0000 7f918c5d3640 1 --2- 172.21.15.57:0/1443807020 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91840c8060 0x7f91841f1e60 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:13.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.404+0000 7f918c5d3640 1 -- 172.21.15.57:0/1443807020 >> 172.21.15.57:0/1443807020 conn(0x7f918406afa0 msgr2=0x7f91840716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:13.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.404+0000 7f918c5d3640 1 -- 172.21.15.57:0/1443807020 shutdown_connections 2023-10-30T14:19:13.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:13.404+0000 7f918c5d3640 1 -- 172.21.15.57:0/1443807020 wait complete. 2023-10-30T14:19:13.427 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:19:15.428 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:19:15.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.580+0000 7fc295c9a640 1 Processor -- start 2023-10-30T14:19:15.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.580+0000 7fc295c9a640 1 -- start start 2023-10-30T14:19:15.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.580+0000 7fc295c9a640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc290151b20 0x7fc2900c28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:15.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.580+0000 7fc295c9a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc2900c2e20 0x7fc2900c4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:15.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.580+0000 7fc295c9a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc290066160 con 0x7fc290151740 2023-10-30T14:19:15.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.580+0000 7fc295c9a640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc29015b640 con 0x7fc290151b20 2023-10-30T14:19:15.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.580+0000 7fc295c9a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc29015bad0 con 0x7fc2900c2e20 2023-10-30T14:19:15.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.580+0000 7fc28f7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc290151740 0x7fc290158040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:43830/0 (socket says 172.21.15.57:43830) 2023-10-30T14:19:15.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.580+0000 7fc28f7fe640 1 -- 172.21.15.57:0/1908437801 learned_addr learned my addr 172.21.15.57:0/1908437801 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:19:15.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.580+0000 7fc28effd640 1 --2- 172.21.15.57:0/1908437801 >> v2:172.21.15.57:3300/0 conn(0x7fc290151b20 0x7fc2900c28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:15.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.580+0000 7fc28ffff640 1 --2- 172.21.15.57:0/1908437801 >> v2:172.21.15.88:3300/0 conn(0x7fc2900c2e20 0x7fc2900c4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:15.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.580+0000 7fc28e7fc640 1 -- 172.21.15.57:0/1908437801 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3362101338 0 0) 0x7fc290066160 con 0x7fc290151740 2023-10-30T14:19:15.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.580+0000 7fc28e7fc640 1 -- 172.21.15.57:0/1908437801 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc278003530 con 0x7fc290151740 2023-10-30T14:19:15.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.580+0000 7fc28effd640 1 -- 172.21.15.57:0/1908437801 >> v2:172.21.15.88:3300/0 conn(0x7fc2900c2e20 msgr2=0x7fc2900c4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:15.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.580+0000 7fc28effd640 1 --2- 172.21.15.57:0/1908437801 >> v2:172.21.15.88:3300/0 conn(0x7fc2900c2e20 0x7fc2900c4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:15.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.580+0000 7fc28effd640 1 -- 172.21.15.57:0/1908437801 >> v1:172.21.15.57:6789/0 conn(0x7fc290151740 legacy=0x7fc290158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:15.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.580+0000 7fc28effd640 1 -- 172.21.15.57:0/1908437801 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc290065e80 con 0x7fc290151b20 2023-10-30T14:19:15.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.580+0000 7fc28ffff640 1 --2- 172.21.15.57:0/1908437801 >> v2:172.21.15.88:3300/0 conn(0x7fc2900c2e20 0x7fc2900c4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:19:15.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.584+0000 7fc28effd640 1 --2- 172.21.15.57:0/1908437801 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc290151b20 0x7fc2900c28e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc290070600 tx=0x7fc28401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6410aa4573f4777a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:15.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.584+0000 7fc28e7fc640 1 -- 172.21.15.57:0/1908437801 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2840a7070 con 0x7fc290151b20 2023-10-30T14:19:15.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.584+0000 7fc28e7fc640 1 -- 172.21.15.57:0/1908437801 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2840ab430 con 0x7fc290151b20 2023-10-30T14:19:15.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.584+0000 7fc28e7fc640 1 -- 172.21.15.57:0/1908437801 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2840ab7c0 con 0x7fc290151b20 2023-10-30T14:19:15.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.584+0000 7fc295c9a640 1 -- 172.21.15.57:0/1908437801 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc290151b20 msgr2=0x7fc2900c28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:15.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.584+0000 7fc295c9a640 1 --2- 172.21.15.57:0/1908437801 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc290151b20 0x7fc2900c28e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc290070600 tx=0x7fc28401c430 comp rx=0 tx=0).stop 2023-10-30T14:19:15.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.584+0000 7fc295c9a640 1 -- 172.21.15.57:0/1908437801 shutdown_connections 2023-10-30T14:19:15.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.584+0000 7fc295c9a640 1 --2- 172.21.15.57:0/1908437801 >> v2:172.21.15.88:3300/0 conn(0x7fc2900c2e20 0x7fc2900c4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:15.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.584+0000 7fc295c9a640 1 --2- 172.21.15.57:0/1908437801 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc290151b20 0x7fc2900c28e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:15.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.584+0000 7fc295c9a640 1 -- 172.21.15.57:0/1908437801 >> 172.21.15.57:0/1908437801 conn(0x7fc2900c7f60 msgr2=0x7fc2900c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:15.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.584+0000 7fc295c9a640 1 -- 172.21.15.57:0/1908437801 shutdown_connections 2023-10-30T14:19:15.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.584+0000 7fc295c9a640 1 -- 172.21.15.57:0/1908437801 wait complete. 2023-10-30T14:19:15.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc295c9a640 1 Processor -- start 2023-10-30T14:19:15.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc295c9a640 1 -- start start 2023-10-30T14:19:15.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc295c9a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2900c2e20 0x7fc290157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:15.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc295c9a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc290151740 0x7fc2901f2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:15.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc295c9a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc290151b20 0x7fc290071530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:15.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc295c9a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2901593a0 con 0x7fc290151b20 2023-10-30T14:19:15.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc295c9a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2901f2da0 con 0x7fc2900c2e20 2023-10-30T14:19:15.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc295c9a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc290071100 con 0x7fc290151740 2023-10-30T14:19:15.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc28ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc290151b20 0x7fc290071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:15.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc28ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc290151b20 0x7fc290071530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36568/0 (socket says 172.21.15.57:36568) 2023-10-30T14:19:15.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc28ffff640 1 -- 172.21.15.57:0/2702905005 learned_addr learned my addr 172.21.15.57:0/2702905005 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:19:15.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc28f7fe640 1 --2- 172.21.15.57:0/2702905005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2900c2e20 0x7fc290157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:15.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc28effd640 1 --2- 172.21.15.57:0/2702905005 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc290151740 0x7fc2901f2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:15.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc28ffff640 1 -- 172.21.15.57:0/2702905005 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc290151740 msgr2=0x7fc2901f2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:15.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc28ffff640 1 --2- 172.21.15.57:0/2702905005 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc290151740 0x7fc2901f2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:15.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc28ffff640 1 -- 172.21.15.57:0/2702905005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2900c2e20 msgr2=0x7fc290157500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:15.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc28ffff640 1 --2- 172.21.15.57:0/2702905005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2900c2e20 0x7fc290157500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:15.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc28ffff640 1 -- 172.21.15.57:0/2702905005 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc284009420 con 0x7fc290151b20 2023-10-30T14:19:15.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc28ffff640 1 --2- 172.21.15.57:0/2702905005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc290151b20 0x7fc290071530 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc288007f20 tx=0x7fc28800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:15.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc277fff640 1 -- 172.21.15.57:0/2702905005 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28800dee0 con 0x7fc290151b20 2023-10-30T14:19:15.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc277fff640 1 -- 172.21.15.57:0/2702905005 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28800f040 con 0x7fc290151b20 2023-10-30T14:19:15.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc295c9a640 1 -- 172.21.15.57:0/2702905005 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2901590c0 con 0x7fc290151b20 2023-10-30T14:19:15.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc277fff640 1 -- 172.21.15.57:0/2702905005 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288003de0 con 0x7fc290151b20 2023-10-30T14:19:15.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.588+0000 7fc295c9a640 1 -- 172.21.15.57:0/2702905005 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc290071e90 con 0x7fc290151b20 2023-10-30T14:19:15.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.592+0000 7fc295c9a640 1 -- 172.21.15.57:0/2702905005 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2901594e0 con 0x7fc290151b20 2023-10-30T14:19:15.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.592+0000 7fc277fff640 1 -- 172.21.15.57:0/2702905005 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc288004030 con 0x7fc290151b20 2023-10-30T14:19:15.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.592+0000 7fc277fff640 1 --2- 172.21.15.57:0/2702905005 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc27c0664a0 0x7fc27c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:15.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.592+0000 7fc277fff640 1 -- 172.21.15.57:0/2702905005 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fc2880844a0 con 0x7fc290151b20 2023-10-30T14:19:15.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.592+0000 7fc28f7fe640 1 --2- 172.21.15.57:0/2702905005 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc27c0664a0 0x7fc27c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:15.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.592+0000 7fc28f7fe640 1 --2- 172.21.15.57:0/2702905005 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc27c0664a0 0x7fc27c068960 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc2800011f0 tx=0x7fc280009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:15.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.596+0000 7fc277fff640 1 -- 172.21.15.57:0/2702905005 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc2901594e0 con 0x7fc290151b20 2023-10-30T14:19:15.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.788+0000 7fc295c9a640 1 -- 172.21.15.57:0/2702905005 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2901594e0 con 0x7fc290151b20 2023-10-30T14:19:15.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.788+0000 7fc277fff640 1 -- 172.21.15.57:0/2702905005 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5738 (secure 0 0 0) 0x7fc2901594e0 con 0x7fc290151b20 2023-10-30T14:19:15.795 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:19:15.795 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:19:11.882149+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:19:15.796 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 38 2023-10-30T14:19:15.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.792+0000 7fc295c9a640 1 -- 172.21.15.57:0/2702905005 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc27c0664a0 msgr2=0x7fc27c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:15.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.792+0000 7fc295c9a640 1 --2- 172.21.15.57:0/2702905005 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc27c0664a0 0x7fc27c068960 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc2800011f0 tx=0x7fc280009040 comp rx=0 tx=0).stop 2023-10-30T14:19:15.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.792+0000 7fc295c9a640 1 -- 172.21.15.57:0/2702905005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc290151b20 msgr2=0x7fc290071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:15.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.792+0000 7fc295c9a640 1 --2- 172.21.15.57:0/2702905005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc290151b20 0x7fc290071530 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc288007f20 tx=0x7fc28800d560 comp rx=0 tx=0).stop 2023-10-30T14:19:15.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.792+0000 7fc295c9a640 1 -- 172.21.15.57:0/2702905005 shutdown_connections 2023-10-30T14:19:15.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.792+0000 7fc295c9a640 1 --2- 172.21.15.57:0/2702905005 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc27c0664a0 0x7fc27c068960 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:15.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.792+0000 7fc295c9a640 1 --2- 172.21.15.57:0/2702905005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc290151b20 0x7fc290071530 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:15.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.792+0000 7fc295c9a640 1 --2- 172.21.15.57:0/2702905005 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc290151740 0x7fc2901f2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:15.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.792+0000 7fc295c9a640 1 --2- 172.21.15.57:0/2702905005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2900c2e20 0x7fc290157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:15.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.792+0000 7fc295c9a640 1 -- 172.21.15.57:0/2702905005 >> 172.21.15.57:0/2702905005 conn(0x7fc2900c7f60 msgr2=0x7fc29014ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:15.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.792+0000 7fc295c9a640 1 -- 172.21.15.57:0/2702905005 shutdown_connections 2023-10-30T14:19:15.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:15.792+0000 7fc295c9a640 1 -- 172.21.15.57:0/2702905005 wait complete. 2023-10-30T14:19:15.816 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:19:17.817 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:19:17.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.968+0000 7f88b5e9c640 1 Processor -- start 2023-10-30T14:19:17.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.968+0000 7f88b5e9c640 1 -- start start 2023-10-30T14:19:17.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.968+0000 7f88b5e9c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88b0153c80 0x7f88b015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:17.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.968+0000 7f88b5e9c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f88b00c4a10 0x7f88b00c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:17.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.968+0000 7f88b5e9c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b0159f20 con 0x7f88b015a3b0 2023-10-30T14:19:17.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.968+0000 7f88b5e9c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b015dec0 con 0x7f88b00c4a10 2023-10-30T14:19:17.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.968+0000 7f88b5e9c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b01599a0 con 0x7f88b0153c80 2023-10-30T14:19:17.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.968+0000 7f88affff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f88b015a3b0 0x7f88b00ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:43844/0 (socket says 172.21.15.57:43844) 2023-10-30T14:19:17.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.968+0000 7f88affff640 1 -- 172.21.15.57:0/297512101 learned_addr learned my addr 172.21.15.57:0/297512101 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:19:17.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.972+0000 7f88aeffd640 1 --2- 172.21.15.57:0/297512101 >> v2:172.21.15.88:3300/0 conn(0x7f88b00c4a10 0x7f88b00c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:17.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.972+0000 7f88af7fe640 1 --2- 172.21.15.57:0/297512101 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88b0153c80 0x7f88b015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:17.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.972+0000 7f88ae7fc640 1 -- 172.21.15.57:0/297512101 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33050817 0 0) 0x7f88b0159f20 con 0x7f88b015a3b0 2023-10-30T14:19:17.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.972+0000 7f88ae7fc640 1 -- 172.21.15.57:0/297512101 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8898003530 con 0x7f88b015a3b0 2023-10-30T14:19:17.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.972+0000 7f88aeffd640 1 -- 172.21.15.57:0/297512101 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88b0153c80 msgr2=0x7f88b015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:17.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.972+0000 7f88aeffd640 1 --2- 172.21.15.57:0/297512101 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88b0153c80 0x7f88b015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:17.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.972+0000 7f88aeffd640 1 -- 172.21.15.57:0/297512101 >> v1:172.21.15.57:6789/0 conn(0x7f88b015a3b0 legacy=0x7f88b00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:17.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.972+0000 7f88aeffd640 1 -- 172.21.15.57:0/297512101 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b00c47e0 con 0x7f88b00c4a10 2023-10-30T14:19:17.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.972+0000 7f88aeffd640 1 --2- 172.21.15.57:0/297512101 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88b00c4a10 0x7f88b00c42a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f88a40097c0 tx=0x7f88a401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd4ce634520adc2a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:17.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.976+0000 7f88ae7fc640 1 -- 172.21.15.57:0/297512101 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a40a8970 con 0x7f88b00c4a10 2023-10-30T14:19:17.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.976+0000 7f88ae7fc640 1 -- 172.21.15.57:0/297512101 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a40a8ad0 con 0x7f88b00c4a10 2023-10-30T14:19:17.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.976+0000 7f88ae7fc640 1 -- 172.21.15.57:0/297512101 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a40a8e40 con 0x7f88b00c4a10 2023-10-30T14:19:17.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.976+0000 7f88b5e9c640 1 -- 172.21.15.57:0/297512101 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88b00c4a10 msgr2=0x7f88b00c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:17.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.976+0000 7f88b5e9c640 1 --2- 172.21.15.57:0/297512101 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88b00c4a10 0x7f88b00c42a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f88a40097c0 tx=0x7f88a401ec70 comp rx=0 tx=0).stop 2023-10-30T14:19:17.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.976+0000 7f88b5e9c640 1 -- 172.21.15.57:0/297512101 shutdown_connections 2023-10-30T14:19:17.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.976+0000 7f88b5e9c640 1 --2- 172.21.15.57:0/297512101 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88b00c4a10 0x7f88b00c42a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:17.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.976+0000 7f88b5e9c640 1 --2- 172.21.15.57:0/297512101 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88b0153c80 0x7f88b015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:17.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.976+0000 7f88b5e9c640 1 -- 172.21.15.57:0/297512101 >> 172.21.15.57:0/297512101 conn(0x7f88b00715c0 msgr2=0x7f88b00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:17.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.976+0000 7f88b5e9c640 1 -- 172.21.15.57:0/297512101 shutdown_connections 2023-10-30T14:19:17.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.976+0000 7f88b5e9c640 1 -- 172.21.15.57:0/297512101 wait complete. 2023-10-30T14:19:17.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.976+0000 7f88b5e9c640 1 Processor -- start 2023-10-30T14:19:17.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.976+0000 7f88b5e9c640 1 -- start start 2023-10-30T14:19:17.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.976+0000 7f88b5e9c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88b00c4a10 0x7f88b0144f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:17.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.976+0000 7f88b5e9c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88b0153c80 0x7f88b015b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:17.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.976+0000 7f88b5e9c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88b015a3b0 0x7f88b0142cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:17.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.976+0000 7f88b5e9c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b015bc70 con 0x7f88b015a3b0 2023-10-30T14:19:17.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.976+0000 7f88b5e9c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b015ac00 con 0x7f88b00c4a10 2023-10-30T14:19:17.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.976+0000 7f88b5e9c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b015d230 con 0x7f88b0153c80 2023-10-30T14:19:17.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.980+0000 7f88affff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88b015a3b0 0x7f88b0142cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:17.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.980+0000 7f88affff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88b015a3b0 0x7f88b0142cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36578/0 (socket says 172.21.15.57:36578) 2023-10-30T14:19:17.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.980+0000 7f88affff640 1 -- 172.21.15.57:0/1036427323 learned_addr learned my addr 172.21.15.57:0/1036427323 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:19:17.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.980+0000 7f88af7fe640 1 --2- 172.21.15.57:0/1036427323 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88b00c4a10 0x7f88b0144f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:17.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.980+0000 7f88aeffd640 1 --2- 172.21.15.57:0/1036427323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88b0153c80 0x7f88b015b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:17.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.980+0000 7f88affff640 1 -- 172.21.15.57:0/1036427323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88b0153c80 msgr2=0x7f88b015b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:17.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.980+0000 7f88affff640 1 --2- 172.21.15.57:0/1036427323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88b0153c80 0x7f88b015b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:17.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.980+0000 7f88affff640 1 -- 172.21.15.57:0/1036427323 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88b00c4a10 msgr2=0x7f88b0144f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:17.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.980+0000 7f88affff640 1 --2- 172.21.15.57:0/1036427323 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88b00c4a10 0x7f88b0144f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:17.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.980+0000 7f88affff640 1 -- 172.21.15.57:0/1036427323 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a4009420 con 0x7f88b015a3b0 2023-10-30T14:19:17.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.980+0000 7f88affff640 1 --2- 172.21.15.57:0/1036427323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88b015a3b0 0x7f88b0142cf0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f88a800caa0 tx=0x7f88a800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:17.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.980+0000 7f8897fff640 1 -- 172.21.15.57:0/1036427323 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a800ac00 con 0x7f88b015a3b0 2023-10-30T14:19:17.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.980+0000 7f8897fff640 1 -- 172.21.15.57:0/1036427323 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a800ad60 con 0x7f88b015a3b0 2023-10-30T14:19:17.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.980+0000 7f8897fff640 1 -- 172.21.15.57:0/1036427323 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a8012620 con 0x7f88b015a3b0 2023-10-30T14:19:17.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.980+0000 7f88b5e9c640 1 -- 172.21.15.57:0/1036427323 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b00c7e40 con 0x7f88b015a3b0 2023-10-30T14:19:17.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.980+0000 7f88b5e9c640 1 -- 172.21.15.57:0/1036427323 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b00c8400 con 0x7f88b015a3b0 2023-10-30T14:19:17.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.980+0000 7f88b5e9c640 1 -- 172.21.15.57:0/1036427323 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b015ad70 con 0x7f88b015a3b0 2023-10-30T14:19:17.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.980+0000 7f8897fff640 1 -- 172.21.15.57:0/1036427323 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f88a8007070 con 0x7f88b015a3b0 2023-10-30T14:19:17.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.980+0000 7f8897fff640 1 --2- 172.21.15.57:0/1036427323 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f889c0664a0 0x7f889c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:17.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.984+0000 7f8897fff640 1 -- 172.21.15.57:0/1036427323 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f88a8083fa0 con 0x7f88b015a3b0 2023-10-30T14:19:17.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.984+0000 7f88af7fe640 1 --2- 172.21.15.57:0/1036427323 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f889c0664a0 0x7f889c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:17.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.984+0000 7f88af7fe640 1 --2- 172.21.15.57:0/1036427323 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f889c0664a0 0x7f889c068960 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f88a0009b60 tx=0x7f88a0009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:17.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:17.984+0000 7f8897fff640 1 -- 172.21.15.57:0/1036427323 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f88a80512f0 con 0x7f88b015a3b0 2023-10-30T14:19:18.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:18.176+0000 7f88b5e9c640 1 -- 172.21.15.57:0/1036427323 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88b004e170 con 0x7f88b015a3b0 2023-10-30T14:19:18.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:18.176+0000 7f8897fff640 1 -- 172.21.15.57:0/1036427323 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f88a8017090 con 0x7f88b015a3b0 2023-10-30T14:19:18.182 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:19:18.182 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:19:16.970293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:19:18.184 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 39 2023-10-30T14:19:18.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:18.180+0000 7f88b5e9c640 1 -- 172.21.15.57:0/1036427323 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f889c0664a0 msgr2=0x7f889c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:18.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:18.180+0000 7f88b5e9c640 1 --2- 172.21.15.57:0/1036427323 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f889c0664a0 0x7f889c068960 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f88a0009b60 tx=0x7f88a0009190 comp rx=0 tx=0).stop 2023-10-30T14:19:18.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:18.180+0000 7f88b5e9c640 1 -- 172.21.15.57:0/1036427323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88b015a3b0 msgr2=0x7f88b0142cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:18.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:18.180+0000 7f88b5e9c640 1 --2- 172.21.15.57:0/1036427323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88b015a3b0 0x7f88b0142cf0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f88a800caa0 tx=0x7f88a800cf60 comp rx=0 tx=0).stop 2023-10-30T14:19:18.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:18.180+0000 7f88b5e9c640 1 -- 172.21.15.57:0/1036427323 shutdown_connections 2023-10-30T14:19:18.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:18.180+0000 7f88b5e9c640 1 --2- 172.21.15.57:0/1036427323 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f889c0664a0 0x7f889c068960 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:18.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:18.180+0000 7f88b5e9c640 1 --2- 172.21.15.57:0/1036427323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88b015a3b0 0x7f88b0142cf0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:18.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:18.180+0000 7f88b5e9c640 1 --2- 172.21.15.57:0/1036427323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88b0153c80 0x7f88b015b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:18.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:18.180+0000 7f88b5e9c640 1 --2- 172.21.15.57:0/1036427323 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88b00c4a10 0x7f88b0144f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:18.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:18.180+0000 7f88b5e9c640 1 -- 172.21.15.57:0/1036427323 >> 172.21.15.57:0/1036427323 conn(0x7f88b00715c0 msgr2=0x7f88b00cb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:18.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:18.180+0000 7f88b5e9c640 1 -- 172.21.15.57:0/1036427323 shutdown_connections 2023-10-30T14:19:18.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:18.180+0000 7f88b5e9c640 1 -- 172.21.15.57:0/1036427323 wait complete. 2023-10-30T14:19:18.203 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=4282 2023-10-30T14:19:18.203 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.f 2023-10-30T14:19:23.098 INFO:tasks.workunit.client.0.smithi057.stdout:./272/.snap/snap-subdir-test 2023-10-30T14:19:23.578 INFO:tasks.workunit.client.0.smithi057.stdout:./100/.snap/snap-subdir-test 2023-10-30T14:19:23.588 INFO:tasks.workunit.client.0.smithi057.stdout:./134/.snap/snap-subdir-test 2023-10-30T14:19:23.595 INFO:tasks.workunit.client.0.smithi057.stdout:./200/.snap/snap-subdir-test 2023-10-30T14:19:23.604 INFO:tasks.workunit.client.0.smithi057.stdout:./115/.snap/snap-subdir-test 2023-10-30T14:19:23.613 INFO:tasks.workunit.client.0.smithi057.stdout:./284/.snap/snap-subdir-test 2023-10-30T14:19:23.622 INFO:tasks.workunit.client.0.smithi057.stdout:./280/.snap/snap-subdir-test 2023-10-30T14:19:23.630 INFO:tasks.workunit.client.0.smithi057.stdout:./116/.snap/snap-subdir-test 2023-10-30T14:19:23.637 INFO:tasks.workunit.client.0.smithi057.stdout:./245/.snap/snap-subdir-test 2023-10-30T14:19:23.645 INFO:tasks.workunit.client.0.smithi057.stdout:./195/.snap/snap-subdir-test 2023-10-30T14:19:23.907 INFO:tasks.workunit.client.0.smithi057.stdout:./138/.snap/snap-subdir-test 2023-10-30T14:19:23.914 INFO:tasks.workunit.client.0.smithi057.stdout:./135/.snap/snap-subdir-test 2023-10-30T14:19:23.923 INFO:tasks.workunit.client.0.smithi057.stdout:./117/.snap/snap-subdir-test 2023-10-30T14:19:23.930 INFO:tasks.workunit.client.0.smithi057.stdout:./241/.snap/snap-subdir-test 2023-10-30T14:19:23.939 INFO:tasks.workunit.client.0.smithi057.stdout:./143/.snap/snap-subdir-test 2023-10-30T14:19:23.948 INFO:tasks.workunit.client.0.smithi057.stdout:./180/.snap/snap-subdir-test 2023-10-30T14:19:23.952 INFO:tasks.workunit.client.0.smithi057.stdout:./126/.snap/snap-subdir-test 2023-10-30T14:19:23.961 INFO:tasks.workunit.client.0.smithi057.stdout:./161/.snap/snap-subdir-test 2023-10-30T14:19:23.968 INFO:tasks.workunit.client.0.smithi057.stdout:./204/.snap/snap-subdir-test 2023-10-30T14:19:23.970 INFO:tasks.ceph.mds.c.smithi057.stderr:2023-10-30T14:19:23.964+0000 7f03bf478640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T14:19:23.970 INFO:tasks.ceph.mds.c.smithi057.stderr:2023-10-30T14:19:23.964+0000 7f03bf478640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:19:23.970 INFO:tasks.ceph.mds.c.smithi057.stderr:2023-10-30T14:19:23.964+0000 7f03bf478640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T14:19:23.978 INFO:tasks.workunit.client.0.smithi057.stdout:./176/.snap/snap-subdir-test 2023-10-30T14:19:23.986 INFO:tasks.workunit.client.0.smithi057.stdout:./255/.snap/snap-subdir-test 2023-10-30T14:19:23.994 INFO:tasks.workunit.client.0.smithi057.stdout:./215/.snap/snap-subdir-test 2023-10-30T14:19:24.765 INFO:tasks.workunit.client.0.smithi057.stdout:./111/.snap/snap-subdir-test 2023-10-30T14:19:24.774 INFO:tasks.workunit.client.0.smithi057.stdout:./238/.snap/snap-subdir-test 2023-10-30T14:19:24.784 INFO:tasks.workunit.client.0.smithi057.stdout:./191/.snap/snap-subdir-test 2023-10-30T14:19:25.128 INFO:tasks.workunit.client.0.smithi057.stdout:./184/.snap/snap-subdir-test 2023-10-30T14:19:25.138 INFO:tasks.workunit.client.0.smithi057.stdout:./192/.snap/snap-subdir-test 2023-10-30T14:19:25.148 INFO:tasks.workunit.client.0.smithi057.stdout:./103/.snap/snap-subdir-test 2023-10-30T14:19:25.158 INFO:tasks.workunit.client.0.smithi057.stdout:./128/.snap/snap-subdir-test 2023-10-30T14:19:25.167 INFO:tasks.workunit.client.0.smithi057.stdout:./229/.snap/snap-subdir-test 2023-10-30T14:19:25.204 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-30T14:19:25.204 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T14:19:25.205 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T14:19:25.208 INFO:tasks.ceph.mds.f:Started 2023-10-30T14:19:25.208 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:19:25.285 INFO:tasks.ceph.mds.f.smithi088.stdout:starting mds.f at 2023-10-30T14:19:25.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.356+0000 7f60aaff2640 1 Processor -- start 2023-10-30T14:19:25.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.356+0000 7f60aaff2640 1 -- start start 2023-10-30T14:19:25.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.356+0000 7f60aaff2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60a40c8eb0 0x7f60a415cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:25.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.356+0000 7f60aaff2640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a415dcc0 con 0x7f60a415ac70 2023-10-30T14:19:25.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.356+0000 7f60aaff2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a4159d80 con 0x7f60a406aeb0 2023-10-30T14:19:25.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.356+0000 7f60aaff2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a4159800 con 0x7f60a40c8eb0 2023-10-30T14:19:25.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.356+0000 7f60a97ef640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f60a415ac70 0x7f60a415a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:56810/0 (socket says 172.21.15.57:56810) 2023-10-30T14:19:25.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.356+0000 7f60a97ef640 1 -- 172.21.15.57:0/278133739 learned_addr learned my addr 172.21.15.57:0/278133739 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:19:25.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.356+0000 7f60a9ff0640 1 --2- 172.21.15.57:0/278133739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60a40c8eb0 0x7f60a415cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:25.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.356+0000 7f60a8fee640 1 -- 172.21.15.57:0/278133739 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105944559 0 0) 0x7f6094001120 con 0x7f60a415ac70 2023-10-30T14:19:25.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.356+0000 7f60a8fee640 1 -- 172.21.15.57:0/278133739 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6090003530 con 0x7f60a415ac70 2023-10-30T14:19:25.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.356+0000 7f60a8fee640 1 -- 172.21.15.57:0/278133739 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88542874 0 0) 0x7f60a4159d80 con 0x7f60a406aeb0 2023-10-30T14:19:25.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.356+0000 7f60a8fee640 1 -- 172.21.15.57:0/278133739 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6094001120 con 0x7f60a406aeb0 2023-10-30T14:19:25.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.356+0000 7f60a9ff0640 1 -- 172.21.15.57:0/278133739 >> v1:172.21.15.88:6789/0 conn(0x7f60a406aeb0 legacy=0x7f60a4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:25.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.356+0000 7f60a9ff0640 1 -- 172.21.15.57:0/278133739 >> v1:172.21.15.57:6789/0 conn(0x7f60a415ac70 legacy=0x7f60a415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:25.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.356+0000 7f60a8fee640 1 -- 172.21.15.57:0/278133739 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4091947436 0 0) 0x7f6090003530 con 0x7f60a415ac70 2023-10-30T14:19:25.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.356+0000 7f60a9ff0640 1 -- 172.21.15.57:0/278133739 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a415d420 con 0x7f60a40c8eb0 2023-10-30T14:19:25.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.360+0000 7f60a9ff0640 1 --2- 172.21.15.57:0/278133739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60a40c8eb0 0x7f60a415cd70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f60a00097f0 tx=0x7f60a001ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1cd0529ceec080 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:25.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.360+0000 7f60a8fee640 1 -- 172.21.15.57:0/278133739 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6094001810 con 0x7f60a40c8eb0 2023-10-30T14:19:25.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.360+0000 7f60a8fee640 1 -- 172.21.15.57:0/278133739 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a00a8650 con 0x7f60a40c8eb0 2023-10-30T14:19:25.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.360+0000 7f60a8fee640 1 -- 172.21.15.57:0/278133739 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a00a89e0 con 0x7f60a40c8eb0 2023-10-30T14:19:25.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.360+0000 7f60aaff2640 1 -- 172.21.15.57:0/278133739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60a40c8eb0 msgr2=0x7f60a415cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:25.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.360+0000 7f60aaff2640 1 --2- 172.21.15.57:0/278133739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60a40c8eb0 0x7f60a415cd70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f60a00097f0 tx=0x7f60a001ebd0 comp rx=0 tx=0).stop 2023-10-30T14:19:25.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.360+0000 7f60aaff2640 1 -- 172.21.15.57:0/278133739 shutdown_connections 2023-10-30T14:19:25.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.360+0000 7f60aaff2640 1 --2- 172.21.15.57:0/278133739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60a40c8eb0 0x7f60a415cd70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:25.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.360+0000 7f60aaff2640 1 -- 172.21.15.57:0/278133739 >> 172.21.15.57:0/278133739 conn(0x7f60a40cef00 msgr2=0x7f60a40cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:25.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.360+0000 7f60aaff2640 1 -- 172.21.15.57:0/278133739 shutdown_connections 2023-10-30T14:19:25.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.360+0000 7f60aaff2640 1 -- 172.21.15.57:0/278133739 wait complete. 2023-10-30T14:19:25.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60aaff2640 1 Processor -- start 2023-10-30T14:19:25.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60aaff2640 1 -- start start 2023-10-30T14:19:25.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60aaff2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60a406aeb0 0x7f60a415c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:25.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60aaff2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60a40c8eb0 0x7f60a414dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:25.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60aaff2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60a415ac70 0x7f60a4154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:25.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60aaff2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a41f2a20 con 0x7f60a406aeb0 2023-10-30T14:19:25.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60aaff2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a414f920 con 0x7f60a40c8eb0 2023-10-30T14:19:25.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60aaff2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a41f2eb0 con 0x7f60a415ac70 2023-10-30T14:19:25.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60a9ff0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60a406aeb0 0x7f60a415c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:25.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60a9ff0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60a406aeb0 0x7f60a415c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55736/0 (socket says 172.21.15.57:55736) 2023-10-30T14:19:25.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60a9ff0640 1 -- 172.21.15.57:0/3586204027 learned_addr learned my addr 172.21.15.57:0/3586204027 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:19:25.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60a97ef640 1 --2- 172.21.15.57:0/3586204027 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60a40c8eb0 0x7f60a414dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:25.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60aa7f1640 1 --2- 172.21.15.57:0/3586204027 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60a415ac70 0x7f60a4154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:25.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60a9ff0640 1 -- 172.21.15.57:0/3586204027 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60a415ac70 msgr2=0x7f60a4154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:25.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60a9ff0640 1 --2- 172.21.15.57:0/3586204027 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60a415ac70 0x7f60a4154af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:25.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60a9ff0640 1 -- 172.21.15.57:0/3586204027 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60a40c8eb0 msgr2=0x7f60a414dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:25.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60a9ff0640 1 --2- 172.21.15.57:0/3586204027 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60a40c8eb0 0x7f60a414dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:25.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60a9ff0640 1 -- 172.21.15.57:0/3586204027 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a0009420 con 0x7f60a406aeb0 2023-10-30T14:19:25.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60a97ef640 1 --2- 172.21.15.57:0/3586204027 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60a40c8eb0 0x7f60a414dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:19:25.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60a9ff0640 1 --2- 172.21.15.57:0/3586204027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60a406aeb0 0x7f60a415c880 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f60a00097f0 tx=0x7f60a001b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:25.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f609a7fc640 1 -- 172.21.15.57:0/3586204027 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a00bf070 con 0x7f60a406aeb0 2023-10-30T14:19:25.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f609a7fc640 1 -- 172.21.15.57:0/3586204027 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a0006e20 con 0x7f60a406aeb0 2023-10-30T14:19:25.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f609a7fc640 1 -- 172.21.15.57:0/3586204027 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a00b0870 con 0x7f60a406aeb0 2023-10-30T14:19:25.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60aaff2640 1 -- 172.21.15.57:0/3586204027 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60a4156830 con 0x7f60a406aeb0 2023-10-30T14:19:25.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60aaff2640 1 -- 172.21.15.57:0/3586204027 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a41f20f0 con 0x7f60a406aeb0 2023-10-30T14:19:25.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.364+0000 7f60aaff2640 1 -- 172.21.15.57:0/3586204027 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60a404e170 con 0x7f60a406aeb0 2023-10-30T14:19:25.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.368+0000 7f609a7fc640 1 -- 172.21.15.57:0/3586204027 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f60a00b7040 con 0x7f60a406aeb0 2023-10-30T14:19:25.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.368+0000 7f609a7fc640 1 --2- 172.21.15.57:0/3586204027 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f60880664a0 0x7f6088068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:25.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.368+0000 7f609a7fc640 1 -- 172.21.15.57:0/3586204027 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f60a0120df0 con 0x7f60a406aeb0 2023-10-30T14:19:25.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.368+0000 7f60a97ef640 1 --2- 172.21.15.57:0/3586204027 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f60880664a0 0x7f6088068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:25.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.368+0000 7f60a97ef640 1 --2- 172.21.15.57:0/3586204027 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f60880664a0 0x7f6088068960 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f60a414f740 tx=0x7f6094009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:25.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.372+0000 7f609a7fc640 1 -- 172.21.15.57:0/3586204027 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f60a00ee000 con 0x7f60a406aeb0 2023-10-30T14:19:25.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.560+0000 7f60aaff2640 1 -- 172.21.15.57:0/3586204027 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60a40ca960 con 0x7f60a406aeb0 2023-10-30T14:19:25.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.564+0000 7f609a7fc640 1 -- 172.21.15.57:0/3586204027 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f60a00b5020 con 0x7f60a406aeb0 2023-10-30T14:19:25.568 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:19:25.568 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:19:16.970293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:19:25.568 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 39 2023-10-30T14:19:25.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.564+0000 7f60aaff2640 1 -- 172.21.15.57:0/3586204027 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f60880664a0 msgr2=0x7f6088068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:25.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.564+0000 7f60aaff2640 1 --2- 172.21.15.57:0/3586204027 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f60880664a0 0x7f6088068960 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f60a414f740 tx=0x7f6094009040 comp rx=0 tx=0).stop 2023-10-30T14:19:25.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.564+0000 7f60aaff2640 1 -- 172.21.15.57:0/3586204027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60a406aeb0 msgr2=0x7f60a415c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:25.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.564+0000 7f60aaff2640 1 --2- 172.21.15.57:0/3586204027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60a406aeb0 0x7f60a415c880 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f60a00097f0 tx=0x7f60a001b970 comp rx=0 tx=0).stop 2023-10-30T14:19:25.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.564+0000 7f60aaff2640 1 -- 172.21.15.57:0/3586204027 shutdown_connections 2023-10-30T14:19:25.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.564+0000 7f60aaff2640 1 --2- 172.21.15.57:0/3586204027 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f60880664a0 0x7f6088068960 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:25.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.564+0000 7f60aaff2640 1 --2- 172.21.15.57:0/3586204027 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60a415ac70 0x7f60a4154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:25.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.564+0000 7f60aaff2640 1 --2- 172.21.15.57:0/3586204027 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60a40c8eb0 0x7f60a414dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:25.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.564+0000 7f60aaff2640 1 --2- 172.21.15.57:0/3586204027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60a406aeb0 0x7f60a415c880 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:25.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.564+0000 7f60aaff2640 1 -- 172.21.15.57:0/3586204027 >> 172.21.15.57:0/3586204027 conn(0x7f60a40cef00 msgr2=0x7f60a4158770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:25.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.564+0000 7f60aaff2640 1 -- 172.21.15.57:0/3586204027 shutdown_connections 2023-10-30T14:19:25.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:25.564+0000 7f60aaff2640 1 -- 172.21.15.57:0/3586204027 wait complete. 2023-10-30T14:19:25.588 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-30T14:19:26.707 INFO:tasks.workunit.client.0.smithi057.stdout:./144/.snap/snap-subdir-test 2023-10-30T14:19:27.014 INFO:tasks.workunit.client.0.smithi057.stdout:./146/.snap/snap-subdir-test 2023-10-30T14:19:27.022 INFO:tasks.workunit.client.0.smithi057.stdout:./227/.snap/snap-subdir-test 2023-10-30T14:19:27.031 INFO:tasks.workunit.client.0.smithi057.stdout:./278/.snap/snap-subdir-test 2023-10-30T14:19:27.293 INFO:tasks.workunit.client.0.smithi057.stdout:./141/.snap/snap-subdir-test 2023-10-30T14:19:27.304 INFO:tasks.workunit.client.0.smithi057.stdout:./234/.snap/snap-subdir-test 2023-10-30T14:19:27.314 INFO:tasks.workunit.client.0.smithi057.stdout:./225/.snap/snap-subdir-test 2023-10-30T14:19:27.589 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:19:27.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.740+0000 7f25b9017640 1 Processor -- start 2023-10-30T14:19:27.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.740+0000 7f25b9017640 1 -- start start 2023-10-30T14:19:27.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.740+0000 7f25b9017640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f25b40c4a10 0x7f25b415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:27.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.740+0000 7f25b9017640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25b40c42a0 0x7f25b40ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:27.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.740+0000 7f25b9017640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b4159f20 con 0x7f25b4153c80 2023-10-30T14:19:27.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.740+0000 7f25b9017640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b415dec0 con 0x7f25b40c4a10 2023-10-30T14:19:27.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.740+0000 7f25b9017640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b41599a0 con 0x7f25b40c42a0 2023-10-30T14:19:27.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.740+0000 7f25abfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f25b40c4a10 0x7f25b415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:27.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.740+0000 7f25b359e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25b40c42a0 0x7f25b40ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:27.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.740+0000 7f25abfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f25b40c4a10 0x7f25b415cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:49986/0 (socket says 172.21.15.57:49986) 2023-10-30T14:19:27.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.740+0000 7f25abfff640 1 -- 172.21.15.57:0/1618608239 learned_addr learned my addr 172.21.15.57:0/1618608239 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:19:27.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.740+0000 7f25b359e640 1 -- 172.21.15.57:0/1618608239 >> v2:172.21.15.88:3300/0 conn(0x7f25b40c4a10 msgr2=0x7f25b415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:27.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.740+0000 7f25b359e640 1 --2- 172.21.15.57:0/1618608239 >> v2:172.21.15.88:3300/0 conn(0x7f25b40c4a10 0x7f25b415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:27.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.740+0000 7f25b359e640 1 -- 172.21.15.57:0/1618608239 >> v1:172.21.15.88:6789/0 conn(0x7f25b4153c80 legacy=0x7f25b415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:27.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.740+0000 7f25b359e640 1 -- 172.21.15.57:0/1618608239 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b415d620 con 0x7f25b40c42a0 2023-10-30T14:19:27.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.744+0000 7f25b359e640 1 --2- 172.21.15.57:0/1618608239 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25b40c42a0 0x7f25b40ca1e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f25ac009d90 tx=0x7f25ac01f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bafc9074caf06dd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:27.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.744+0000 7f25b259c640 1 -- 172.21.15.57:0/1618608239 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac0af600 con 0x7f25b40c42a0 2023-10-30T14:19:27.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.744+0000 7f25b259c640 1 -- 172.21.15.57:0/1618608239 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ac0ab050 con 0x7f25b40c42a0 2023-10-30T14:19:27.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.744+0000 7f25b259c640 1 -- 172.21.15.57:0/1618608239 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac0af980 con 0x7f25b40c42a0 2023-10-30T14:19:27.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.744+0000 7f25b9017640 1 -- 172.21.15.57:0/1618608239 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25b40c42a0 msgr2=0x7f25b40ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:27.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.744+0000 7f25b9017640 1 --2- 172.21.15.57:0/1618608239 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25b40c42a0 0x7f25b40ca1e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f25ac009d90 tx=0x7f25ac01f130 comp rx=0 tx=0).stop 2023-10-30T14:19:27.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25b9017640 1 -- 172.21.15.57:0/1618608239 shutdown_connections 2023-10-30T14:19:27.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25b9017640 1 --2- 172.21.15.57:0/1618608239 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25b40c42a0 0x7f25b40ca1e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:27.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25b9017640 1 --2- 172.21.15.57:0/1618608239 >> v2:172.21.15.88:3300/0 conn(0x7f25b40c4a10 0x7f25b415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:27.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25b9017640 1 -- 172.21.15.57:0/1618608239 >> 172.21.15.57:0/1618608239 conn(0x7f25b40715c0 msgr2=0x7f25b40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:27.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25b9017640 1 -- 172.21.15.57:0/1618608239 shutdown_connections 2023-10-30T14:19:27.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25b9017640 1 -- 172.21.15.57:0/1618608239 wait complete. 2023-10-30T14:19:27.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25b9017640 1 Processor -- start 2023-10-30T14:19:27.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25b9017640 1 -- start start 2023-10-30T14:19:27.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25b9017640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25b40c42a0 0x7f25b415ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:27.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25b9017640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25b40c4a10 0x7f25b40c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:27.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25b9017640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25b4153c80 0x7f25b415c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:27.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25b9017640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b415c210 con 0x7f25b40c4a10 2023-10-30T14:19:27.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25b9017640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b41ef280 con 0x7f25b4153c80 2023-10-30T14:19:27.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25b9017640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b41ef4f0 con 0x7f25b40c42a0 2023-10-30T14:19:27.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25abfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25b40c4a10 0x7f25b40c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:27.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25abfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25b40c4a10 0x7f25b40c8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55752/0 (socket says 172.21.15.57:55752) 2023-10-30T14:19:27.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25abfff640 1 -- 172.21.15.57:0/1295041129 learned_addr learned my addr 172.21.15.57:0/1295041129 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:19:27.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25b2d9d640 1 --2- 172.21.15.57:0/1295041129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25b40c42a0 0x7f25b415ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:27.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25b359e640 1 --2- 172.21.15.57:0/1295041129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25b4153c80 0x7f25b415c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:27.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25abfff640 1 -- 172.21.15.57:0/1295041129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25b40c42a0 msgr2=0x7f25b415ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:27.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25abfff640 1 --2- 172.21.15.57:0/1295041129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25b40c42a0 0x7f25b415ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:27.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25abfff640 1 -- 172.21.15.57:0/1295041129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25b4153c80 msgr2=0x7f25b415c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:27.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25abfff640 1 --2- 172.21.15.57:0/1295041129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25b4153c80 0x7f25b415c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:27.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.748+0000 7f25abfff640 1 -- 172.21.15.57:0/1295041129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac009a70 con 0x7f25b40c4a10 2023-10-30T14:19:27.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.752+0000 7f25abfff640 1 --2- 172.21.15.57:0/1295041129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25b40c4a10 0x7f25b40c8ee0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f259c00e900 tx=0x7f259c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:27.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.752+0000 7f25ab7fe640 1 -- 172.21.15.57:0/1295041129 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c009660 con 0x7f25b40c4a10 2023-10-30T14:19:27.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.752+0000 7f25ab7fe640 1 -- 172.21.15.57:0/1295041129 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259c00ca00 con 0x7f25b40c4a10 2023-10-30T14:19:27.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.752+0000 7f25ab7fe640 1 -- 172.21.15.57:0/1295041129 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c00cd70 con 0x7f25b40c4a10 2023-10-30T14:19:27.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.752+0000 7f25b9017640 1 -- 172.21.15.57:0/1295041129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b415bea0 con 0x7f25b40c4a10 2023-10-30T14:19:27.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.752+0000 7f25b9017640 1 -- 172.21.15.57:0/1295041129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b415b1c0 con 0x7f25b40c4a10 2023-10-30T14:19:27.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.752+0000 7f25b9017640 1 -- 172.21.15.57:0/1295041129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b415bd00 con 0x7f25b40c4a10 2023-10-30T14:19:27.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.752+0000 7f25ab7fe640 1 -- 172.21.15.57:0/1295041129 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f259c013470 con 0x7f25b40c4a10 2023-10-30T14:19:27.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.752+0000 7f25ab7fe640 1 --2- 172.21.15.57:0/1295041129 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f25940664a0 0x7f2594068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:27.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.752+0000 7f25ab7fe640 1 -- 172.21.15.57:0/1295041129 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f259c084230 con 0x7f25b40c4a10 2023-10-30T14:19:27.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.756+0000 7f25b2d9d640 1 --2- 172.21.15.57:0/1295041129 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f25940664a0 0x7f2594068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:27.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.756+0000 7f25b2d9d640 1 --2- 172.21.15.57:0/1295041129 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f25940664a0 0x7f2594068960 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f25a4007fd0 tx=0x7f25a4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:27.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.756+0000 7f25ab7fe640 1 -- 172.21.15.57:0/1295041129 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f25b415bd00 con 0x7f25b40c4a10 2023-10-30T14:19:27.948 INFO:tasks.workunit.client.0.smithi057.stdout:./182/.snap/snap-subdir-test 2023-10-30T14:19:27.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.948+0000 7f25b9017640 1 -- 172.21.15.57:0/1295041129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25b415bd00 con 0x7f25b40c4a10 2023-10-30T14:19:27.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.952+0000 7f25ab7fe640 1 -- 172.21.15.57:0/1295041129 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f25b415bd00 con 0x7f25b40c4a10 2023-10-30T14:19:27.955 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:19:27.956 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5078,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:19:16.970293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:19:27.956 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 40 2023-10-30T14:19:27.957 INFO:tasks.workunit.client.0.smithi057.stdout:./107/.snap/snap-subdir-test 2023-10-30T14:19:27.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.952+0000 7f25b9017640 1 -- 172.21.15.57:0/1295041129 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f25940664a0 msgr2=0x7f2594068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:27.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.952+0000 7f25b9017640 1 --2- 172.21.15.57:0/1295041129 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f25940664a0 0x7f2594068960 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f25a4007fd0 tx=0x7f25a4009040 comp rx=0 tx=0).stop 2023-10-30T14:19:27.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.952+0000 7f25b9017640 1 -- 172.21.15.57:0/1295041129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25b40c4a10 msgr2=0x7f25b40c8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:27.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.952+0000 7f25b9017640 1 --2- 172.21.15.57:0/1295041129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25b40c4a10 0x7f25b40c8ee0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f259c00e900 tx=0x7f259c00edc0 comp rx=0 tx=0).stop 2023-10-30T14:19:27.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.952+0000 7f25b9017640 1 -- 172.21.15.57:0/1295041129 shutdown_connections 2023-10-30T14:19:27.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.952+0000 7f25b9017640 1 --2- 172.21.15.57:0/1295041129 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f25940664a0 0x7f2594068960 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:27.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.952+0000 7f25b9017640 1 --2- 172.21.15.57:0/1295041129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25b4153c80 0x7f25b415c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:27.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.952+0000 7f25b9017640 1 --2- 172.21.15.57:0/1295041129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25b40c4a10 0x7f25b40c8ee0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:27.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.952+0000 7f25b9017640 1 --2- 172.21.15.57:0/1295041129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25b40c42a0 0x7f25b415ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:27.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.952+0000 7f25b9017640 1 -- 172.21.15.57:0/1295041129 >> 172.21.15.57:0/1295041129 conn(0x7f25b40715c0 msgr2=0x7f25b41587d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:27.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.952+0000 7f25b9017640 1 -- 172.21.15.57:0/1295041129 shutdown_connections 2023-10-30T14:19:27.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:19:27.952+0000 7f25b9017640 1 -- 172.21.15.57:0/1295041129 wait complete. 2023-10-30T14:19:27.978 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-30T14:19:27.979 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 50 secs before thrashing 2023-10-30T14:19:28.438 INFO:tasks.workunit.client.0.smithi057.stdout:./178/.snap/snap-subdir-test 2023-10-30T14:19:28.446 INFO:tasks.workunit.client.0.smithi057.stdout:./266/.snap/snap-subdir-test 2023-10-30T14:19:28.452 INFO:tasks.workunit.client.0.smithi057.stdout:./240/.snap/snap-subdir-test 2023-10-30T14:19:28.461 INFO:tasks.workunit.client.0.smithi057.stdout:./156/.snap/snap-subdir-test 2023-10-30T14:19:28.470 INFO:tasks.workunit.client.0.smithi057.stdout:./131/.snap/snap-subdir-test 2023-10-30T14:19:28.474 INFO:tasks.workunit.client.0.smithi057.stdout:./123/.snap/snap-subdir-test 2023-10-30T14:19:28.483 INFO:tasks.workunit.client.0.smithi057.stdout:./160/.snap/snap-subdir-test 2023-10-30T14:19:28.922 INFO:tasks.workunit.client.0.smithi057.stdout:./165/.snap/snap-subdir-test 2023-10-30T14:19:28.933 INFO:tasks.workunit.client.0.smithi057.stdout:./101/.snap/snap-subdir-test 2023-10-30T14:19:28.944 INFO:tasks.workunit.client.0.smithi057.stdout:./148/.snap/snap-subdir-test 2023-10-30T14:19:28.954 INFO:tasks.workunit.client.0.smithi057.stdout:./168/.snap/snap-subdir-test 2023-10-30T14:19:28.963 INFO:tasks.workunit.client.0.smithi057.stdout:./179/.snap/snap-subdir-test 2023-10-30T14:19:28.969 INFO:tasks.workunit.client.0.smithi057.stdout:./257/.snap/snap-subdir-test 2023-10-30T14:19:28.977 INFO:tasks.workunit.client.0.smithi057.stdout:./252/.snap/snap-subdir-test 2023-10-30T14:19:29.402 INFO:tasks.workunit.client.0.smithi057.stdout:./132/.snap/snap-subdir-test 2023-10-30T14:19:29.910 INFO:tasks.workunit.client.0.smithi057.stdout:./236/.snap/snap-subdir-test 2023-10-30T14:19:29.919 INFO:tasks.workunit.client.0.smithi057.stdout:./211/.snap/snap-subdir-test 2023-10-30T14:19:29.929 INFO:tasks.workunit.client.0.smithi057.stdout:./199/.snap/snap-subdir-test 2023-10-30T14:19:29.937 INFO:tasks.workunit.client.0.smithi057.stdout:./205/.snap/snap-subdir-test 2023-10-30T14:19:29.947 INFO:tasks.workunit.client.0.smithi057.stdout:./163/.snap/snap-subdir-test 2023-10-30T14:19:29.956 INFO:tasks.workunit.client.0.smithi057.stdout:./261/.snap/snap-subdir-test 2023-10-30T14:19:29.964 INFO:tasks.workunit.client.0.smithi057.stdout:./201/.snap/snap-subdir-test 2023-10-30T14:19:29.973 INFO:tasks.workunit.client.0.smithi057.stdout:./282/.snap/snap-subdir-test 2023-10-30T14:19:30.478 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:19:30.482 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:19:30.635 INFO:tasks.workunit.client.0.smithi057.stdout:./296/.snap/snap-subdir-test 2023-10-30T14:19:30.639 INFO:tasks.workunit.client.0.smithi057.stdout:./217/.snap/snap-subdir-test 2023-10-30T14:19:30.644 INFO:tasks.workunit.client.0.smithi057.stdout:./153/.snap/snap-subdir-test 2023-10-30T14:19:30.649 INFO:tasks.workunit.client.0.smithi057.stdout:./206/.snap/snap-subdir-test 2023-10-30T14:19:30.654 INFO:tasks.workunit.client.0.smithi057.stdout:./122/.snap/snap-subdir-test 2023-10-30T14:19:30.659 INFO:tasks.workunit.client.0.smithi057.stdout:./162/.snap/snap-subdir-test 2023-10-30T14:19:30.663 INFO:tasks.workunit.client.0.smithi057.stdout:./197/.snap/snap-subdir-test 2023-10-30T14:19:30.668 INFO:tasks.workunit.client.0.smithi057.stdout:./216/.snap/snap-subdir-test 2023-10-30T14:19:32.015 INFO:tasks.workunit.client.0.smithi057.stdout:./254/.snap/snap-subdir-test 2023-10-30T14:19:32.251 INFO:tasks.workunit.client.0.smithi057.stdout:./113/.snap/snap-subdir-test 2023-10-30T14:19:32.256 INFO:tasks.workunit.client.0.smithi057.stdout:./152/.snap/snap-subdir-test 2023-10-30T14:19:32.260 INFO:tasks.workunit.client.0.smithi057.stdout:./121/.snap/snap-subdir-test 2023-10-30T14:19:32.264 INFO:tasks.workunit.client.0.smithi057.stdout:./118/.snap/snap-subdir-test 2023-10-30T14:19:32.269 INFO:tasks.workunit.client.0.smithi057.stdout:./196/.snap/snap-subdir-test 2023-10-30T14:19:32.273 INFO:tasks.workunit.client.0.smithi057.stdout:./124/.snap/snap-subdir-test 2023-10-30T14:19:32.277 INFO:tasks.workunit.client.0.smithi057.stdout:./239/.snap/snap-subdir-test 2023-10-30T14:19:32.281 INFO:tasks.workunit.client.0.smithi057.stdout:./188/.snap/snap-subdir-test 2023-10-30T14:19:32.285 INFO:tasks.workunit.client.0.smithi057.stdout:./287/.snap/snap-subdir-test 2023-10-30T14:19:32.289 INFO:tasks.workunit.client.0.smithi057.stdout:./275/.snap/snap-subdir-test 2023-10-30T14:19:32.293 INFO:tasks.workunit.client.0.smithi057.stdout:./259/.snap/snap-subdir-test 2023-10-30T14:19:32.297 INFO:tasks.workunit.client.0.smithi057.stdout:./224/.snap/snap-subdir-test 2023-10-30T14:19:32.301 INFO:tasks.workunit.client.0.smithi057.stdout:./264/.snap/snap-subdir-test 2023-10-30T14:19:32.305 INFO:tasks.workunit.client.0.smithi057.stdout:./145/.snap/snap-subdir-test 2023-10-30T14:19:32.309 INFO:tasks.workunit.client.0.smithi057.stdout:./279/.snap/snap-subdir-test 2023-10-30T14:19:32.314 INFO:tasks.workunit.client.0.smithi057.stdout:./263/.snap/snap-subdir-test 2023-10-30T14:19:32.323 INFO:tasks.workunit.client.0.smithi057.stdout:./130/.snap/snap-subdir-test 2023-10-30T14:19:32.333 INFO:tasks.workunit.client.0.smithi057.stdout:./193/.snap/snap-subdir-test 2023-10-30T14:19:32.889 INFO:tasks.workunit.client.0.smithi057.stdout:./223/.snap/snap-subdir-test 2023-10-30T14:19:32.898 INFO:tasks.workunit.client.0.smithi057.stdout:./207/.snap/snap-subdir-test 2023-10-30T14:19:33.028 INFO:tasks.workunit.client.0.smithi057.stdout:./276/.snap/snap-subdir-test 2023-10-30T14:19:33.355 INFO:tasks.workunit.client.0.smithi057.stdout:./271/.snap/snap-subdir-test 2023-10-30T14:19:33.364 INFO:tasks.workunit.client.0.smithi057.stdout:./251/.snap/snap-subdir-test 2023-10-30T14:19:33.373 INFO:tasks.workunit.client.0.smithi057.stdout:./175/.snap/snap-subdir-test 2023-10-30T14:19:33.383 INFO:tasks.workunit.client.0.smithi057.stdout:./222/.snap/snap-subdir-test 2023-10-30T14:19:33.393 INFO:tasks.workunit.client.0.smithi057.stdout:./220/.snap/snap-subdir-test 2023-10-30T14:19:33.401 INFO:tasks.workunit.client.0.smithi057.stdout:./203/.snap/snap-subdir-test 2023-10-30T14:19:33.410 INFO:tasks.workunit.client.0.smithi057.stdout:./181/.snap/snap-subdir-test 2023-10-30T14:19:33.419 INFO:tasks.workunit.client.0.smithi057.stdout:./286/.snap/snap-subdir-test 2023-10-30T14:19:33.428 INFO:tasks.workunit.client.0.smithi057.stdout:./127/.snap/snap-subdir-test 2023-10-30T14:19:33.434 INFO:tasks.workunit.client.0.smithi057.stdout:./108/.snap/snap-subdir-test 2023-10-30T14:19:33.695 INFO:tasks.workunit.client.0.smithi057.stdout:./183/.snap/snap-subdir-test 2023-10-30T14:19:33.704 INFO:tasks.workunit.client.0.smithi057.stdout:./294/.snap/snap-subdir-test 2023-10-30T14:19:33.714 INFO:tasks.workunit.client.0.smithi057.stdout:./235/.snap/snap-subdir-test 2023-10-30T14:19:33.723 INFO:tasks.workunit.client.0.smithi057.stdout:./177/.snap/snap-subdir-test 2023-10-30T14:19:33.746 INFO:tasks.workunit.client.0.smithi057.stdout:./169/.snap/snap-subdir-test 2023-10-30T14:19:33.754 INFO:tasks.workunit.client.0.smithi057.stdout:./212/.snap/snap-subdir-test 2023-10-30T14:19:33.762 INFO:tasks.workunit.client.0.smithi057.stdout:./154/.snap/snap-subdir-test 2023-10-30T14:19:33.770 INFO:tasks.workunit.client.0.smithi057.stdout:./171/.snap/snap-subdir-test 2023-10-30T14:19:33.779 INFO:tasks.workunit.client.0.smithi057.stdout:./139/.snap/snap-subdir-test 2023-10-30T14:19:33.788 INFO:tasks.workunit.client.0.smithi057.stdout:./102/.snap/snap-subdir-test 2023-10-30T14:19:33.798 INFO:tasks.workunit.client.0.smithi057.stdout:./129/.snap/snap-subdir-test 2023-10-30T14:19:47.262 INFO:tasks.workunit.client.0.smithi057.stdout:Delete all the files and directories ... 2023-10-30T14:19:48.518 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/19' 2023-10-30T14:19:48.518 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/15' 2023-10-30T14:19:48.518 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/31' 2023-10-30T14:19:48.519 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/36' 2023-10-30T14:19:48.519 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/20' 2023-10-30T14:19:48.519 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/12' 2023-10-30T14:19:48.519 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/39' 2023-10-30T14:19:48.519 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/10' 2023-10-30T14:19:48.520 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/34' 2023-10-30T14:19:48.520 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/38' 2023-10-30T14:19:48.520 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/11' 2023-10-30T14:19:48.520 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/16' 2023-10-30T14:19:48.520 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/32' 2023-10-30T14:19:48.521 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/30' 2023-10-30T14:19:48.521 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/14' 2023-10-30T14:19:48.521 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/18' 2023-10-30T14:19:48.521 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/13' 2023-10-30T14:19:48.521 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/33' 2023-10-30T14:19:48.522 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/17' 2023-10-30T14:19:48.522 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/35' 2023-10-30T14:19:48.522 INFO:tasks.workunit.client.0.smithi057.stdout:removed './100/37' 2023-10-30T14:19:48.522 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './100' 2023-10-30T14:19:48.522 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/19' 2023-10-30T14:19:48.523 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/15' 2023-10-30T14:19:48.523 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/31' 2023-10-30T14:19:48.523 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/36' 2023-10-30T14:19:48.523 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/20' 2023-10-30T14:19:48.523 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/12' 2023-10-30T14:19:48.523 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/39' 2023-10-30T14:19:48.524 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/10' 2023-10-30T14:19:48.524 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/34' 2023-10-30T14:19:48.524 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/38' 2023-10-30T14:19:48.524 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/11' 2023-10-30T14:19:48.524 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/16' 2023-10-30T14:19:48.525 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/32' 2023-10-30T14:19:48.525 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/30' 2023-10-30T14:19:48.525 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/14' 2023-10-30T14:19:48.525 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/18' 2023-10-30T14:19:48.525 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/13' 2023-10-30T14:19:48.526 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/33' 2023-10-30T14:19:48.526 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/17' 2023-10-30T14:19:48.526 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/35' 2023-10-30T14:19:48.526 INFO:tasks.workunit.client.0.smithi057.stdout:removed './101/37' 2023-10-30T14:19:48.526 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './101' 2023-10-30T14:19:48.527 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/19' 2023-10-30T14:19:48.527 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/15' 2023-10-30T14:19:48.527 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/31' 2023-10-30T14:19:48.527 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/36' 2023-10-30T14:19:48.527 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/20' 2023-10-30T14:19:48.528 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/12' 2023-10-30T14:19:48.528 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/39' 2023-10-30T14:19:48.528 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/10' 2023-10-30T14:19:48.528 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/34' 2023-10-30T14:19:48.528 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/38' 2023-10-30T14:19:48.529 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/11' 2023-10-30T14:19:48.529 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/16' 2023-10-30T14:19:48.529 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/32' 2023-10-30T14:19:48.529 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/30' 2023-10-30T14:19:48.529 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/14' 2023-10-30T14:19:48.529 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/18' 2023-10-30T14:19:48.530 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/13' 2023-10-30T14:19:48.530 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/33' 2023-10-30T14:19:48.530 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/17' 2023-10-30T14:19:48.530 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/35' 2023-10-30T14:19:48.531 INFO:tasks.workunit.client.0.smithi057.stdout:removed './102/37' 2023-10-30T14:19:48.531 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './102' 2023-10-30T14:19:48.531 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/19' 2023-10-30T14:19:48.531 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/15' 2023-10-30T14:19:48.531 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/31' 2023-10-30T14:19:48.531 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/36' 2023-10-30T14:19:48.532 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/20' 2023-10-30T14:19:48.532 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/12' 2023-10-30T14:19:48.532 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/39' 2023-10-30T14:19:48.532 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/10' 2023-10-30T14:19:48.532 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/34' 2023-10-30T14:19:48.533 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/38' 2023-10-30T14:19:48.533 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/11' 2023-10-30T14:19:48.533 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/16' 2023-10-30T14:19:48.533 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/32' 2023-10-30T14:19:48.534 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/30' 2023-10-30T14:19:48.534 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/14' 2023-10-30T14:19:48.534 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/18' 2023-10-30T14:19:48.534 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/13' 2023-10-30T14:19:48.534 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/33' 2023-10-30T14:19:48.535 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/17' 2023-10-30T14:19:48.535 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/35' 2023-10-30T14:19:48.535 INFO:tasks.workunit.client.0.smithi057.stdout:removed './103/37' 2023-10-30T14:19:48.535 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './103' 2023-10-30T14:19:48.535 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/19' 2023-10-30T14:19:48.536 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/15' 2023-10-30T14:19:48.536 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/31' 2023-10-30T14:19:48.536 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/36' 2023-10-30T14:19:48.536 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/20' 2023-10-30T14:19:48.536 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/12' 2023-10-30T14:19:48.537 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/39' 2023-10-30T14:19:48.537 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/10' 2023-10-30T14:19:48.537 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/34' 2023-10-30T14:19:48.537 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/38' 2023-10-30T14:19:48.537 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/11' 2023-10-30T14:19:48.538 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/16' 2023-10-30T14:19:48.538 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/32' 2023-10-30T14:19:48.538 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/30' 2023-10-30T14:19:48.538 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/14' 2023-10-30T14:19:48.538 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/18' 2023-10-30T14:19:48.539 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/13' 2023-10-30T14:19:48.539 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/33' 2023-10-30T14:19:48.539 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/17' 2023-10-30T14:19:48.539 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/35' 2023-10-30T14:19:48.539 INFO:tasks.workunit.client.0.smithi057.stdout:removed './104/37' 2023-10-30T14:19:48.540 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './104' 2023-10-30T14:19:48.540 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/19' 2023-10-30T14:19:48.540 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/15' 2023-10-30T14:19:48.540 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/31' 2023-10-30T14:19:48.540 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/36' 2023-10-30T14:19:48.540 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/20' 2023-10-30T14:19:48.541 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/12' 2023-10-30T14:19:48.541 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/39' 2023-10-30T14:19:48.541 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/10' 2023-10-30T14:19:48.541 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/34' 2023-10-30T14:19:48.541 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/38' 2023-10-30T14:19:48.542 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/11' 2023-10-30T14:19:48.542 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/16' 2023-10-30T14:19:48.542 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/32' 2023-10-30T14:19:48.542 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/30' 2023-10-30T14:19:48.542 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/14' 2023-10-30T14:19:48.543 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/18' 2023-10-30T14:19:48.543 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/13' 2023-10-30T14:19:48.543 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/33' 2023-10-30T14:19:48.543 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/17' 2023-10-30T14:19:48.543 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/35' 2023-10-30T14:19:48.544 INFO:tasks.workunit.client.0.smithi057.stdout:removed './105/37' 2023-10-30T14:19:48.544 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './105' 2023-10-30T14:19:48.544 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/19' 2023-10-30T14:19:48.544 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/15' 2023-10-30T14:19:48.544 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/31' 2023-10-30T14:19:48.544 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/36' 2023-10-30T14:19:48.545 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/20' 2023-10-30T14:19:48.545 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/12' 2023-10-30T14:19:48.545 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/39' 2023-10-30T14:19:48.545 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/10' 2023-10-30T14:19:48.545 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/34' 2023-10-30T14:19:48.546 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/38' 2023-10-30T14:19:48.546 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/11' 2023-10-30T14:19:48.546 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/16' 2023-10-30T14:19:48.546 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/32' 2023-10-30T14:19:48.546 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/30' 2023-10-30T14:19:48.547 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/14' 2023-10-30T14:19:48.547 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/18' 2023-10-30T14:19:48.547 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/13' 2023-10-30T14:19:48.547 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/33' 2023-10-30T14:19:48.547 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/17' 2023-10-30T14:19:48.548 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/35' 2023-10-30T14:19:48.548 INFO:tasks.workunit.client.0.smithi057.stdout:removed './106/37' 2023-10-30T14:19:48.548 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './106' 2023-10-30T14:19:48.548 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/19' 2023-10-30T14:19:48.548 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/15' 2023-10-30T14:19:48.549 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/31' 2023-10-30T14:19:48.549 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/36' 2023-10-30T14:19:48.549 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/20' 2023-10-30T14:19:48.549 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/12' 2023-10-30T14:19:48.549 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/39' 2023-10-30T14:19:48.549 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/10' 2023-10-30T14:19:48.550 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/34' 2023-10-30T14:19:48.550 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/38' 2023-10-30T14:19:48.550 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/11' 2023-10-30T14:19:48.550 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/16' 2023-10-30T14:19:48.550 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/32' 2023-10-30T14:19:48.551 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/30' 2023-10-30T14:19:48.551 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/14' 2023-10-30T14:19:48.551 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/18' 2023-10-30T14:19:48.551 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/13' 2023-10-30T14:19:48.551 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/33' 2023-10-30T14:19:48.552 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/17' 2023-10-30T14:19:48.552 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/35' 2023-10-30T14:19:48.552 INFO:tasks.workunit.client.0.smithi057.stdout:removed './107/37' 2023-10-30T14:19:48.552 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './107' 2023-10-30T14:19:48.552 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/19' 2023-10-30T14:19:48.552 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/15' 2023-10-30T14:19:48.553 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/31' 2023-10-30T14:19:48.553 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/36' 2023-10-30T14:19:48.553 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/20' 2023-10-30T14:19:48.553 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/12' 2023-10-30T14:19:48.553 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/39' 2023-10-30T14:19:48.554 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/10' 2023-10-30T14:19:48.554 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/34' 2023-10-30T14:19:48.554 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/38' 2023-10-30T14:19:48.554 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/11' 2023-10-30T14:19:48.554 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/16' 2023-10-30T14:19:48.555 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/32' 2023-10-30T14:19:48.555 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/30' 2023-10-30T14:19:48.555 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/14' 2023-10-30T14:19:48.555 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/18' 2023-10-30T14:19:48.555 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/13' 2023-10-30T14:19:48.555 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/33' 2023-10-30T14:19:48.556 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/17' 2023-10-30T14:19:48.556 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/35' 2023-10-30T14:19:48.556 INFO:tasks.workunit.client.0.smithi057.stdout:removed './108/37' 2023-10-30T14:19:48.556 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './108' 2023-10-30T14:19:48.556 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/19' 2023-10-30T14:19:48.557 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/15' 2023-10-30T14:19:48.557 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/31' 2023-10-30T14:19:48.557 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/36' 2023-10-30T14:19:48.557 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/20' 2023-10-30T14:19:48.557 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/12' 2023-10-30T14:19:48.557 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/39' 2023-10-30T14:19:48.558 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/10' 2023-10-30T14:19:48.558 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/34' 2023-10-30T14:19:48.558 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/38' 2023-10-30T14:19:48.558 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/11' 2023-10-30T14:19:48.558 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/16' 2023-10-30T14:19:48.559 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/32' 2023-10-30T14:19:48.559 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/30' 2023-10-30T14:19:49.995 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/14' 2023-10-30T14:19:49.996 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/18' 2023-10-30T14:19:49.996 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/13' 2023-10-30T14:19:49.996 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/33' 2023-10-30T14:19:49.996 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/17' 2023-10-30T14:19:49.996 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/35' 2023-10-30T14:19:49.997 INFO:tasks.workunit.client.0.smithi057.stdout:removed './109/37' 2023-10-30T14:19:49.997 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './109' 2023-10-30T14:19:49.997 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/19' 2023-10-30T14:19:49.997 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/15' 2023-10-30T14:19:49.997 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/31' 2023-10-30T14:19:49.998 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/36' 2023-10-30T14:19:49.998 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/20' 2023-10-30T14:19:49.998 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/12' 2023-10-30T14:19:49.998 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/39' 2023-10-30T14:19:49.998 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/10' 2023-10-30T14:19:50.001 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/34' 2023-10-30T14:19:50.001 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/38' 2023-10-30T14:19:50.001 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/11' 2023-10-30T14:19:50.001 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/16' 2023-10-30T14:19:50.001 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/32' 2023-10-30T14:19:50.002 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/30' 2023-10-30T14:19:50.002 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/14' 2023-10-30T14:19:50.002 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/18' 2023-10-30T14:19:50.002 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/13' 2023-10-30T14:19:50.002 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/33' 2023-10-30T14:19:50.003 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/17' 2023-10-30T14:19:50.003 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/35' 2023-10-30T14:19:50.003 INFO:tasks.workunit.client.0.smithi057.stdout:removed './110/37' 2023-10-30T14:19:50.003 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './110' 2023-10-30T14:19:50.003 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/19' 2023-10-30T14:19:50.004 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/15' 2023-10-30T14:19:50.004 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/31' 2023-10-30T14:19:50.004 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/36' 2023-10-30T14:19:50.004 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/20' 2023-10-30T14:19:50.004 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/12' 2023-10-30T14:19:50.005 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/39' 2023-10-30T14:19:50.005 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/10' 2023-10-30T14:19:50.005 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/34' 2023-10-30T14:19:50.005 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/38' 2023-10-30T14:19:50.005 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/11' 2023-10-30T14:19:50.006 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/16' 2023-10-30T14:19:50.006 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/32' 2023-10-30T14:19:50.006 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/30' 2023-10-30T14:19:50.006 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/14' 2023-10-30T14:19:50.006 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/18' 2023-10-30T14:19:50.007 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/13' 2023-10-30T14:19:50.007 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/33' 2023-10-30T14:19:50.007 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/17' 2023-10-30T14:19:50.007 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/35' 2023-10-30T14:19:50.007 INFO:tasks.workunit.client.0.smithi057.stdout:removed './111/37' 2023-10-30T14:19:50.008 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './111' 2023-10-30T14:19:50.008 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/19' 2023-10-30T14:19:50.008 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/15' 2023-10-30T14:19:50.008 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/31' 2023-10-30T14:19:50.008 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/36' 2023-10-30T14:19:50.008 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/20' 2023-10-30T14:19:50.009 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/12' 2023-10-30T14:19:50.009 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/39' 2023-10-30T14:19:50.009 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/10' 2023-10-30T14:19:50.009 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/34' 2023-10-30T14:19:50.009 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/38' 2023-10-30T14:19:50.010 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/11' 2023-10-30T14:19:50.010 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/16' 2023-10-30T14:19:50.010 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/32' 2023-10-30T14:19:50.010 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/30' 2023-10-30T14:19:50.010 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/14' 2023-10-30T14:19:50.011 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/18' 2023-10-30T14:19:50.011 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/13' 2023-10-30T14:19:50.011 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/33' 2023-10-30T14:19:50.011 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/17' 2023-10-30T14:19:50.011 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/35' 2023-10-30T14:19:50.012 INFO:tasks.workunit.client.0.smithi057.stdout:removed './112/37' 2023-10-30T14:19:50.012 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './112' 2023-10-30T14:19:50.012 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/19' 2023-10-30T14:19:50.012 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/15' 2023-10-30T14:19:50.012 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/31' 2023-10-30T14:19:50.012 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/36' 2023-10-30T14:19:50.013 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/20' 2023-10-30T14:19:50.013 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/12' 2023-10-30T14:19:50.013 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/39' 2023-10-30T14:19:50.013 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/10' 2023-10-30T14:19:50.013 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/34' 2023-10-30T14:19:50.014 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/38' 2023-10-30T14:19:50.014 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/11' 2023-10-30T14:19:50.014 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/16' 2023-10-30T14:19:50.014 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/32' 2023-10-30T14:19:50.014 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/30' 2023-10-30T14:19:50.015 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/14' 2023-10-30T14:19:50.015 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/18' 2023-10-30T14:19:50.015 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/13' 2023-10-30T14:19:50.015 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/33' 2023-10-30T14:19:50.015 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/17' 2023-10-30T14:19:50.015 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/35' 2023-10-30T14:19:50.016 INFO:tasks.workunit.client.0.smithi057.stdout:removed './113/37' 2023-10-30T14:19:50.016 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './113' 2023-10-30T14:19:50.016 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/19' 2023-10-30T14:19:50.016 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/15' 2023-10-30T14:19:50.016 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/31' 2023-10-30T14:19:50.016 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/36' 2023-10-30T14:19:50.017 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/20' 2023-10-30T14:19:50.017 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/12' 2023-10-30T14:19:50.017 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/39' 2023-10-30T14:19:50.017 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/10' 2023-10-30T14:19:50.017 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/34' 2023-10-30T14:19:50.018 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/38' 2023-10-30T14:19:50.018 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/11' 2023-10-30T14:19:50.018 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/16' 2023-10-30T14:19:50.018 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/32' 2023-10-30T14:19:50.018 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/30' 2023-10-30T14:19:50.019 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/14' 2023-10-30T14:19:50.019 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/18' 2023-10-30T14:19:50.019 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/13' 2023-10-30T14:19:50.019 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/33' 2023-10-30T14:19:50.019 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/17' 2023-10-30T14:19:50.020 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/35' 2023-10-30T14:19:50.020 INFO:tasks.workunit.client.0.smithi057.stdout:removed './114/37' 2023-10-30T14:19:50.020 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './114' 2023-10-30T14:19:50.020 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/19' 2023-10-30T14:19:50.020 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/15' 2023-10-30T14:19:50.021 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/31' 2023-10-30T14:19:50.021 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/36' 2023-10-30T14:19:50.021 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/20' 2023-10-30T14:19:50.021 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/12' 2023-10-30T14:19:50.021 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/39' 2023-10-30T14:19:50.022 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/10' 2023-10-30T14:19:50.022 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/34' 2023-10-30T14:19:50.022 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/38' 2023-10-30T14:19:50.022 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/11' 2023-10-30T14:19:50.022 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/16' 2023-10-30T14:19:50.023 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/32' 2023-10-30T14:19:50.023 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/30' 2023-10-30T14:19:50.023 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/14' 2023-10-30T14:19:50.023 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/18' 2023-10-30T14:19:50.023 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/13' 2023-10-30T14:19:50.024 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/33' 2023-10-30T14:19:50.024 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/17' 2023-10-30T14:19:50.024 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/35' 2023-10-30T14:19:50.024 INFO:tasks.workunit.client.0.smithi057.stdout:removed './115/37' 2023-10-30T14:19:50.024 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './115' 2023-10-30T14:19:50.025 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/19' 2023-10-30T14:19:50.025 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/15' 2023-10-30T14:19:50.025 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/31' 2023-10-30T14:19:50.025 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/36' 2023-10-30T14:19:50.025 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/20' 2023-10-30T14:19:50.026 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/12' 2023-10-30T14:19:50.026 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/39' 2023-10-30T14:19:50.026 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/10' 2023-10-30T14:19:50.026 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/34' 2023-10-30T14:19:50.026 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/38' 2023-10-30T14:19:50.027 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/11' 2023-10-30T14:19:50.027 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/16' 2023-10-30T14:19:50.027 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/32' 2023-10-30T14:19:50.027 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/30' 2023-10-30T14:19:50.027 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/14' 2023-10-30T14:19:50.028 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/18' 2023-10-30T14:19:50.028 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/13' 2023-10-30T14:19:50.028 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/33' 2023-10-30T14:19:50.028 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/17' 2023-10-30T14:19:50.028 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/35' 2023-10-30T14:19:50.029 INFO:tasks.workunit.client.0.smithi057.stdout:removed './116/37' 2023-10-30T14:19:50.029 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './116' 2023-10-30T14:19:50.029 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/19' 2023-10-30T14:19:50.029 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/15' 2023-10-30T14:19:50.029 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/31' 2023-10-30T14:19:50.029 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/36' 2023-10-30T14:19:50.030 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/20' 2023-10-30T14:19:50.030 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/12' 2023-10-30T14:19:50.030 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/39' 2023-10-30T14:19:50.030 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/10' 2023-10-30T14:19:50.030 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/34' 2023-10-30T14:19:50.031 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/38' 2023-10-30T14:19:50.031 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/11' 2023-10-30T14:19:50.031 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/16' 2023-10-30T14:19:50.031 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/32' 2023-10-30T14:19:50.031 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/30' 2023-10-30T14:19:50.032 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/14' 2023-10-30T14:19:50.032 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/18' 2023-10-30T14:19:50.032 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/13' 2023-10-30T14:19:50.032 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/33' 2023-10-30T14:19:50.032 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/17' 2023-10-30T14:19:50.033 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/35' 2023-10-30T14:19:50.033 INFO:tasks.workunit.client.0.smithi057.stdout:removed './117/37' 2023-10-30T14:19:50.033 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './117' 2023-10-30T14:19:50.033 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/19' 2023-10-30T14:19:50.033 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/15' 2023-10-30T14:19:50.033 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/31' 2023-10-30T14:19:50.034 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/36' 2023-10-30T14:19:50.034 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/20' 2023-10-30T14:19:50.034 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/12' 2023-10-30T14:19:50.034 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/39' 2023-10-30T14:19:50.034 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/10' 2023-10-30T14:19:50.035 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/34' 2023-10-30T14:19:50.035 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/38' 2023-10-30T14:19:50.035 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/11' 2023-10-30T14:19:50.035 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/16' 2023-10-30T14:19:50.035 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/32' 2023-10-30T14:19:50.036 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/30' 2023-10-30T14:19:50.036 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/14' 2023-10-30T14:19:50.036 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/18' 2023-10-30T14:19:50.036 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/13' 2023-10-30T14:19:50.036 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/33' 2023-10-30T14:19:50.037 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/17' 2023-10-30T14:19:50.037 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/35' 2023-10-30T14:19:50.037 INFO:tasks.workunit.client.0.smithi057.stdout:removed './118/37' 2023-10-30T14:19:50.037 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './118' 2023-10-30T14:19:50.037 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/19' 2023-10-30T14:19:50.038 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/15' 2023-10-30T14:19:50.038 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/31' 2023-10-30T14:19:50.038 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/36' 2023-10-30T14:19:50.038 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/20' 2023-10-30T14:19:50.038 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/12' 2023-10-30T14:19:54.316 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/39' 2023-10-30T14:19:54.317 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/10' 2023-10-30T14:19:54.317 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/34' 2023-10-30T14:19:54.317 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/38' 2023-10-30T14:19:54.317 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/11' 2023-10-30T14:19:54.317 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/16' 2023-10-30T14:19:54.318 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/32' 2023-10-30T14:19:54.318 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/30' 2023-10-30T14:19:54.318 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/14' 2023-10-30T14:19:54.318 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/18' 2023-10-30T14:19:54.318 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/13' 2023-10-30T14:19:54.319 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/33' 2023-10-30T14:19:54.319 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/17' 2023-10-30T14:19:54.319 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/35' 2023-10-30T14:19:54.319 INFO:tasks.workunit.client.0.smithi057.stdout:removed './119/37' 2023-10-30T14:19:54.319 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './119' 2023-10-30T14:19:54.319 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/19' 2023-10-30T14:19:54.320 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/15' 2023-10-30T14:19:54.320 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/31' 2023-10-30T14:19:54.320 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/36' 2023-10-30T14:19:54.320 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/20' 2023-10-30T14:19:54.320 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/12' 2023-10-30T14:19:54.320 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/39' 2023-10-30T14:19:54.320 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/10' 2023-10-30T14:19:54.321 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/34' 2023-10-30T14:19:54.321 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/38' 2023-10-30T14:19:54.321 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/11' 2023-10-30T14:19:54.321 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/16' 2023-10-30T14:19:54.321 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/32' 2023-10-30T14:19:54.322 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/30' 2023-10-30T14:19:54.322 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/14' 2023-10-30T14:19:54.322 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/18' 2023-10-30T14:19:54.322 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/13' 2023-10-30T14:19:54.322 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/33' 2023-10-30T14:19:54.323 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/17' 2023-10-30T14:19:54.323 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/35' 2023-10-30T14:19:54.323 INFO:tasks.workunit.client.0.smithi057.stdout:removed './120/37' 2023-10-30T14:19:54.323 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './120' 2023-10-30T14:19:54.323 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/19' 2023-10-30T14:19:54.323 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/15' 2023-10-30T14:19:54.324 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/31' 2023-10-30T14:19:54.324 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/36' 2023-10-30T14:19:54.324 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/20' 2023-10-30T14:19:54.324 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/12' 2023-10-30T14:19:54.324 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/39' 2023-10-30T14:19:54.324 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/10' 2023-10-30T14:19:54.325 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/34' 2023-10-30T14:19:54.325 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/38' 2023-10-30T14:19:54.325 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/11' 2023-10-30T14:19:54.325 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/16' 2023-10-30T14:19:54.325 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/32' 2023-10-30T14:19:54.325 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/30' 2023-10-30T14:19:54.326 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/14' 2023-10-30T14:19:54.326 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/18' 2023-10-30T14:19:54.326 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/13' 2023-10-30T14:19:54.326 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/33' 2023-10-30T14:19:54.326 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/17' 2023-10-30T14:19:54.327 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/35' 2023-10-30T14:19:54.327 INFO:tasks.workunit.client.0.smithi057.stdout:removed './121/37' 2023-10-30T14:19:54.327 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './121' 2023-10-30T14:19:54.327 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/19' 2023-10-30T14:19:54.327 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/15' 2023-10-30T14:19:54.328 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/31' 2023-10-30T14:19:54.328 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/36' 2023-10-30T14:19:54.328 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/20' 2023-10-30T14:19:54.328 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/12' 2023-10-30T14:19:54.328 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/39' 2023-10-30T14:19:54.328 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/10' 2023-10-30T14:19:54.329 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/34' 2023-10-30T14:19:54.329 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/38' 2023-10-30T14:19:54.329 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/11' 2023-10-30T14:19:54.329 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/16' 2023-10-30T14:19:54.329 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/32' 2023-10-30T14:19:54.329 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/30' 2023-10-30T14:19:54.330 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/14' 2023-10-30T14:19:54.330 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/18' 2023-10-30T14:19:54.330 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/13' 2023-10-30T14:19:54.330 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/33' 2023-10-30T14:19:54.330 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/17' 2023-10-30T14:19:54.331 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/35' 2023-10-30T14:19:54.331 INFO:tasks.workunit.client.0.smithi057.stdout:removed './122/37' 2023-10-30T14:19:54.331 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './122' 2023-10-30T14:19:54.331 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/19' 2023-10-30T14:19:54.331 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/15' 2023-10-30T14:19:54.331 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/31' 2023-10-30T14:19:54.332 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/36' 2023-10-30T14:19:54.332 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/20' 2023-10-30T14:19:54.332 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/12' 2023-10-30T14:19:54.332 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/39' 2023-10-30T14:19:54.332 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/10' 2023-10-30T14:19:54.332 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/34' 2023-10-30T14:19:54.333 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/38' 2023-10-30T14:19:54.333 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/11' 2023-10-30T14:19:54.333 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/16' 2023-10-30T14:19:54.333 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/32' 2023-10-30T14:19:54.333 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/30' 2023-10-30T14:19:54.333 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/14' 2023-10-30T14:19:54.334 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/18' 2023-10-30T14:19:54.334 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/13' 2023-10-30T14:19:54.334 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/33' 2023-10-30T14:19:54.334 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/17' 2023-10-30T14:19:54.334 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/35' 2023-10-30T14:19:54.335 INFO:tasks.workunit.client.0.smithi057.stdout:removed './123/37' 2023-10-30T14:19:54.335 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './123' 2023-10-30T14:19:54.335 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/19' 2023-10-30T14:19:54.335 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/15' 2023-10-30T14:19:54.335 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/31' 2023-10-30T14:19:54.336 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/36' 2023-10-30T14:19:54.336 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/20' 2023-10-30T14:19:54.336 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/12' 2023-10-30T14:19:54.336 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/39' 2023-10-30T14:19:54.336 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/10' 2023-10-30T14:19:54.336 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/34' 2023-10-30T14:19:54.337 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/38' 2023-10-30T14:19:54.337 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/11' 2023-10-30T14:19:54.337 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/16' 2023-10-30T14:19:54.337 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/32' 2023-10-30T14:19:54.337 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/30' 2023-10-30T14:19:54.337 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/14' 2023-10-30T14:19:54.338 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/18' 2023-10-30T14:19:54.338 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/13' 2023-10-30T14:19:54.338 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/33' 2023-10-30T14:19:54.338 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/17' 2023-10-30T14:19:54.338 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/35' 2023-10-30T14:19:54.339 INFO:tasks.workunit.client.0.smithi057.stdout:removed './124/37' 2023-10-30T14:19:54.339 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './124' 2023-10-30T14:19:54.339 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/19' 2023-10-30T14:19:54.339 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/15' 2023-10-30T14:19:54.339 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/31' 2023-10-30T14:19:54.340 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/36' 2023-10-30T14:19:54.340 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/20' 2023-10-30T14:19:54.340 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/12' 2023-10-30T14:19:54.340 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/39' 2023-10-30T14:19:54.340 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/10' 2023-10-30T14:19:54.341 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/34' 2023-10-30T14:19:54.341 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/38' 2023-10-30T14:19:54.341 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/11' 2023-10-30T14:19:54.341 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/16' 2023-10-30T14:19:54.341 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/32' 2023-10-30T14:19:54.341 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/30' 2023-10-30T14:19:54.342 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/14' 2023-10-30T14:19:54.342 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/18' 2023-10-30T14:19:54.342 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/13' 2023-10-30T14:19:54.342 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/33' 2023-10-30T14:19:54.342 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/17' 2023-10-30T14:19:54.343 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/35' 2023-10-30T14:19:54.343 INFO:tasks.workunit.client.0.smithi057.stdout:removed './125/37' 2023-10-30T14:19:54.343 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './125' 2023-10-30T14:19:54.343 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/19' 2023-10-30T14:19:54.343 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/15' 2023-10-30T14:19:54.343 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/31' 2023-10-30T14:19:54.344 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/36' 2023-10-30T14:19:54.344 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/20' 2023-10-30T14:19:54.344 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/12' 2023-10-30T14:19:54.344 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/39' 2023-10-30T14:19:54.344 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/10' 2023-10-30T14:19:54.345 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/34' 2023-10-30T14:19:54.345 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/38' 2023-10-30T14:19:54.345 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/11' 2023-10-30T14:19:54.345 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/16' 2023-10-30T14:19:54.345 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/32' 2023-10-30T14:19:54.345 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/30' 2023-10-30T14:19:54.346 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/14' 2023-10-30T14:19:54.346 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/18' 2023-10-30T14:19:54.346 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/13' 2023-10-30T14:19:54.346 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/33' 2023-10-30T14:19:54.346 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/17' 2023-10-30T14:19:54.346 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/35' 2023-10-30T14:19:54.347 INFO:tasks.workunit.client.0.smithi057.stdout:removed './126/37' 2023-10-30T14:19:54.347 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './126' 2023-10-30T14:19:54.347 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/19' 2023-10-30T14:19:54.347 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/15' 2023-10-30T14:19:54.347 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/31' 2023-10-30T14:19:54.348 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/36' 2023-10-30T14:19:54.348 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/20' 2023-10-30T14:19:54.348 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/12' 2023-10-30T14:19:54.348 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/39' 2023-10-30T14:19:54.348 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/10' 2023-10-30T14:19:54.349 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/34' 2023-10-30T14:19:54.349 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/38' 2023-10-30T14:19:54.349 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/11' 2023-10-30T14:19:54.349 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/16' 2023-10-30T14:19:54.349 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/32' 2023-10-30T14:19:54.349 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/30' 2023-10-30T14:19:54.350 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/14' 2023-10-30T14:19:54.350 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/18' 2023-10-30T14:19:54.350 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/13' 2023-10-30T14:19:54.350 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/33' 2023-10-30T14:19:54.350 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/17' 2023-10-30T14:19:54.351 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/35' 2023-10-30T14:19:54.351 INFO:tasks.workunit.client.0.smithi057.stdout:removed './127/37' 2023-10-30T14:19:54.351 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './127' 2023-10-30T14:19:54.351 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/19' 2023-10-30T14:19:54.351 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/15' 2023-10-30T14:19:54.351 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/31' 2023-10-30T14:19:54.352 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/36' 2023-10-30T14:19:54.352 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/20' 2023-10-30T14:19:54.352 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/12' 2023-10-30T14:19:54.352 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/39' 2023-10-30T14:19:54.352 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/10' 2023-10-30T14:19:54.352 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/34' 2023-10-30T14:19:54.353 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/38' 2023-10-30T14:19:54.353 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/11' 2023-10-30T14:19:54.353 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/16' 2023-10-30T14:19:54.353 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/32' 2023-10-30T14:19:54.353 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/30' 2023-10-30T14:19:54.354 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/14' 2023-10-30T14:19:54.354 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/18' 2023-10-30T14:19:54.354 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/13' 2023-10-30T14:19:54.354 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/33' 2023-10-30T14:19:54.354 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/17' 2023-10-30T14:19:54.355 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/35' 2023-10-30T14:20:00.680 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:20:00.685 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:20:02.425 INFO:tasks.workunit.client.0.smithi057.stdout:removed './128/37' 2023-10-30T14:20:02.426 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './128' 2023-10-30T14:20:02.426 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/19' 2023-10-30T14:20:02.426 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/15' 2023-10-30T14:20:02.426 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/31' 2023-10-30T14:20:02.427 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/36' 2023-10-30T14:20:02.427 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/20' 2023-10-30T14:20:02.427 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/12' 2023-10-30T14:20:02.427 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/39' 2023-10-30T14:20:02.427 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/10' 2023-10-30T14:20:02.427 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/34' 2023-10-30T14:20:02.428 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/38' 2023-10-30T14:20:02.428 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/11' 2023-10-30T14:20:02.428 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/16' 2023-10-30T14:20:02.428 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/32' 2023-10-30T14:20:02.428 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/30' 2023-10-30T14:20:02.429 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/14' 2023-10-30T14:20:02.429 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/18' 2023-10-30T14:20:02.429 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/13' 2023-10-30T14:20:02.429 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/33' 2023-10-30T14:20:02.429 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/17' 2023-10-30T14:20:02.429 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/35' 2023-10-30T14:20:02.430 INFO:tasks.workunit.client.0.smithi057.stdout:removed './129/37' 2023-10-30T14:20:02.430 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './129' 2023-10-30T14:20:02.430 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/19' 2023-10-30T14:20:02.430 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/15' 2023-10-30T14:20:02.430 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/31' 2023-10-30T14:20:02.430 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/36' 2023-10-30T14:20:02.431 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/20' 2023-10-30T14:20:02.431 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/12' 2023-10-30T14:20:02.431 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/39' 2023-10-30T14:20:02.431 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/10' 2023-10-30T14:20:02.431 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/34' 2023-10-30T14:20:02.431 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/38' 2023-10-30T14:20:02.432 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/11' 2023-10-30T14:20:02.432 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/16' 2023-10-30T14:20:02.432 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/32' 2023-10-30T14:20:02.432 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/30' 2023-10-30T14:20:02.432 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/14' 2023-10-30T14:20:02.432 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/18' 2023-10-30T14:20:02.433 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/13' 2023-10-30T14:20:02.433 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/33' 2023-10-30T14:20:02.433 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/17' 2023-10-30T14:20:02.433 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/35' 2023-10-30T14:20:02.433 INFO:tasks.workunit.client.0.smithi057.stdout:removed './130/37' 2023-10-30T14:20:02.433 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './130' 2023-10-30T14:20:02.434 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/19' 2023-10-30T14:20:02.434 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/15' 2023-10-30T14:20:02.434 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/31' 2023-10-30T14:20:02.434 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/36' 2023-10-30T14:20:02.434 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/20' 2023-10-30T14:20:02.435 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/12' 2023-10-30T14:20:02.435 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/39' 2023-10-30T14:20:02.435 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/10' 2023-10-30T14:20:02.435 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/34' 2023-10-30T14:20:02.435 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/38' 2023-10-30T14:20:02.435 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/11' 2023-10-30T14:20:02.436 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/16' 2023-10-30T14:20:02.436 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/32' 2023-10-30T14:20:02.436 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/30' 2023-10-30T14:20:02.436 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/14' 2023-10-30T14:20:02.436 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/18' 2023-10-30T14:20:02.436 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/13' 2023-10-30T14:20:02.437 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/33' 2023-10-30T14:20:02.437 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/17' 2023-10-30T14:20:02.437 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/35' 2023-10-30T14:20:02.437 INFO:tasks.workunit.client.0.smithi057.stdout:removed './131/37' 2023-10-30T14:20:02.437 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './131' 2023-10-30T14:20:02.437 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/19' 2023-10-30T14:20:02.438 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/15' 2023-10-30T14:20:02.438 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/31' 2023-10-30T14:20:02.438 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/36' 2023-10-30T14:20:02.438 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/20' 2023-10-30T14:20:02.438 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/12' 2023-10-30T14:20:02.439 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/39' 2023-10-30T14:20:02.439 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/10' 2023-10-30T14:20:02.439 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/34' 2023-10-30T14:20:02.439 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/38' 2023-10-30T14:20:02.439 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/11' 2023-10-30T14:20:02.439 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/16' 2023-10-30T14:20:02.440 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/32' 2023-10-30T14:20:02.440 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/30' 2023-10-30T14:20:02.440 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/14' 2023-10-30T14:20:02.440 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/18' 2023-10-30T14:20:02.440 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/13' 2023-10-30T14:20:02.440 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/33' 2023-10-30T14:20:02.441 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/17' 2023-10-30T14:20:02.441 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/35' 2023-10-30T14:20:02.441 INFO:tasks.workunit.client.0.smithi057.stdout:removed './132/37' 2023-10-30T14:20:02.441 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './132' 2023-10-30T14:20:02.441 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/19' 2023-10-30T14:20:02.442 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/15' 2023-10-30T14:20:02.442 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/31' 2023-10-30T14:20:02.442 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/36' 2023-10-30T14:20:02.442 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/20' 2023-10-30T14:20:02.442 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/12' 2023-10-30T14:20:02.443 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/39' 2023-10-30T14:20:02.443 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/10' 2023-10-30T14:20:02.443 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/34' 2023-10-30T14:20:02.443 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/38' 2023-10-30T14:20:02.443 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/11' 2023-10-30T14:20:02.443 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/16' 2023-10-30T14:20:02.444 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/32' 2023-10-30T14:20:02.444 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/30' 2023-10-30T14:20:02.444 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/14' 2023-10-30T14:20:02.444 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/18' 2023-10-30T14:20:02.444 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/13' 2023-10-30T14:20:02.445 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/33' 2023-10-30T14:20:02.445 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/17' 2023-10-30T14:20:02.445 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/35' 2023-10-30T14:20:02.445 INFO:tasks.workunit.client.0.smithi057.stdout:removed './133/37' 2023-10-30T14:20:02.445 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './133' 2023-10-30T14:20:02.445 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/19' 2023-10-30T14:20:02.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/15' 2023-10-30T14:20:02.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/31' 2023-10-30T14:20:02.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/36' 2023-10-30T14:20:02.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/20' 2023-10-30T14:20:02.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/12' 2023-10-30T14:20:02.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/39' 2023-10-30T14:20:02.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/10' 2023-10-30T14:20:02.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/34' 2023-10-30T14:20:02.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/38' 2023-10-30T14:20:02.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/11' 2023-10-30T14:20:02.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/16' 2023-10-30T14:20:02.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/32' 2023-10-30T14:20:02.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/30' 2023-10-30T14:20:02.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/14' 2023-10-30T14:20:02.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/18' 2023-10-30T14:20:02.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/13' 2023-10-30T14:20:02.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/33' 2023-10-30T14:20:02.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/17' 2023-10-30T14:20:02.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/35' 2023-10-30T14:20:02.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed './134/37' 2023-10-30T14:20:02.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './134' 2023-10-30T14:20:02.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/19' 2023-10-30T14:20:02.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/15' 2023-10-30T14:20:02.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/31' 2023-10-30T14:20:02.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/36' 2023-10-30T14:20:02.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/20' 2023-10-30T14:20:02.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/12' 2023-10-30T14:20:02.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/39' 2023-10-30T14:20:02.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/10' 2023-10-30T14:20:02.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/34' 2023-10-30T14:20:02.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/38' 2023-10-30T14:20:02.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/11' 2023-10-30T14:20:02.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/16' 2023-10-30T14:20:02.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/32' 2023-10-30T14:20:02.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/30' 2023-10-30T14:20:02.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/14' 2023-10-30T14:20:02.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/18' 2023-10-30T14:20:02.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/13' 2023-10-30T14:20:02.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/33' 2023-10-30T14:20:02.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/17' 2023-10-30T14:20:02.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/35' 2023-10-30T14:20:02.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed './135/37' 2023-10-30T14:20:02.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './135' 2023-10-30T14:20:02.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/19' 2023-10-30T14:20:02.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/15' 2023-10-30T14:20:02.454 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/31' 2023-10-30T14:20:02.454 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/36' 2023-10-30T14:20:02.454 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/20' 2023-10-30T14:20:02.454 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/12' 2023-10-30T14:20:02.454 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/39' 2023-10-30T14:20:02.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/10' 2023-10-30T14:20:02.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/34' 2023-10-30T14:20:02.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/38' 2023-10-30T14:20:02.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/11' 2023-10-30T14:20:02.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/16' 2023-10-30T14:20:02.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/32' 2023-10-30T14:20:02.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/30' 2023-10-30T14:20:02.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/14' 2023-10-30T14:20:02.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/18' 2023-10-30T14:20:02.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/13' 2023-10-30T14:20:02.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/33' 2023-10-30T14:20:02.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/17' 2023-10-30T14:20:02.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/35' 2023-10-30T14:20:02.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './136/37' 2023-10-30T14:20:02.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './136' 2023-10-30T14:20:02.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/19' 2023-10-30T14:20:02.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/15' 2023-10-30T14:20:02.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/31' 2023-10-30T14:20:02.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/36' 2023-10-30T14:20:02.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/20' 2023-10-30T14:20:02.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/12' 2023-10-30T14:20:02.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/39' 2023-10-30T14:20:02.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/10' 2023-10-30T14:20:02.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/34' 2023-10-30T14:20:02.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/38' 2023-10-30T14:20:02.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/11' 2023-10-30T14:20:02.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/16' 2023-10-30T14:20:02.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/32' 2023-10-30T14:20:02.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/30' 2023-10-30T14:20:02.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/14' 2023-10-30T14:20:02.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/18' 2023-10-30T14:20:02.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/13' 2023-10-30T14:20:02.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/33' 2023-10-30T14:20:02.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/17' 2023-10-30T14:20:02.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/35' 2023-10-30T14:20:02.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './137/37' 2023-10-30T14:20:02.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './137' 2023-10-30T14:20:02.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/19' 2023-10-30T14:20:02.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/15' 2023-10-30T14:20:02.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/31' 2023-10-30T14:20:02.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/36' 2023-10-30T14:20:02.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/20' 2023-10-30T14:20:02.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/12' 2023-10-30T14:20:02.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/39' 2023-10-30T14:20:02.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/10' 2023-10-30T14:20:02.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/34' 2023-10-30T14:20:02.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/38' 2023-10-30T14:20:02.463 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/11' 2023-10-30T14:20:02.463 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/16' 2023-10-30T14:20:06.486 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/32' 2023-10-30T14:20:06.487 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/30' 2023-10-30T14:20:06.487 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/14' 2023-10-30T14:20:06.487 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/18' 2023-10-30T14:20:06.487 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/13' 2023-10-30T14:20:06.487 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/33' 2023-10-30T14:20:06.488 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/17' 2023-10-30T14:20:06.488 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/35' 2023-10-30T14:20:06.488 INFO:tasks.workunit.client.0.smithi057.stdout:removed './138/37' 2023-10-30T14:20:06.488 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './138' 2023-10-30T14:20:06.488 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/19' 2023-10-30T14:20:06.489 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/15' 2023-10-30T14:20:06.489 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/31' 2023-10-30T14:20:06.489 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/36' 2023-10-30T14:20:06.489 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/20' 2023-10-30T14:20:06.489 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/12' 2023-10-30T14:20:06.489 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/39' 2023-10-30T14:20:06.490 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/10' 2023-10-30T14:20:06.490 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/34' 2023-10-30T14:20:06.490 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/38' 2023-10-30T14:20:06.490 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/11' 2023-10-30T14:20:06.490 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/16' 2023-10-30T14:20:06.490 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/32' 2023-10-30T14:20:06.491 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/30' 2023-10-30T14:20:06.491 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/14' 2023-10-30T14:20:06.491 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/18' 2023-10-30T14:20:06.491 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/13' 2023-10-30T14:20:06.491 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/33' 2023-10-30T14:20:06.491 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/17' 2023-10-30T14:20:06.492 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/35' 2023-10-30T14:20:06.492 INFO:tasks.workunit.client.0.smithi057.stdout:removed './139/37' 2023-10-30T14:20:06.492 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './139' 2023-10-30T14:20:06.492 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/19' 2023-10-30T14:20:06.492 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/15' 2023-10-30T14:20:06.493 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/31' 2023-10-30T14:20:06.493 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/36' 2023-10-30T14:20:06.493 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/20' 2023-10-30T14:20:06.493 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/12' 2023-10-30T14:20:06.493 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/39' 2023-10-30T14:20:06.493 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/10' 2023-10-30T14:20:06.494 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/34' 2023-10-30T14:20:06.494 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/38' 2023-10-30T14:20:06.494 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/11' 2023-10-30T14:20:06.494 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/16' 2023-10-30T14:20:06.494 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/32' 2023-10-30T14:20:06.495 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/30' 2023-10-30T14:20:06.495 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/14' 2023-10-30T14:20:06.495 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/18' 2023-10-30T14:20:06.495 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/13' 2023-10-30T14:20:06.495 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/33' 2023-10-30T14:20:06.495 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/17' 2023-10-30T14:20:06.496 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/35' 2023-10-30T14:20:06.496 INFO:tasks.workunit.client.0.smithi057.stdout:removed './140/37' 2023-10-30T14:20:06.496 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './140' 2023-10-30T14:20:06.496 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/19' 2023-10-30T14:20:06.496 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/15' 2023-10-30T14:20:06.496 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/31' 2023-10-30T14:20:06.497 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/36' 2023-10-30T14:20:06.497 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/20' 2023-10-30T14:20:06.497 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/12' 2023-10-30T14:20:06.497 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/39' 2023-10-30T14:20:06.497 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/10' 2023-10-30T14:20:06.497 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/34' 2023-10-30T14:20:06.498 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/38' 2023-10-30T14:20:06.498 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/11' 2023-10-30T14:20:06.498 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/16' 2023-10-30T14:20:06.498 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/32' 2023-10-30T14:20:06.498 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/30' 2023-10-30T14:20:06.499 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/14' 2023-10-30T14:20:06.499 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/18' 2023-10-30T14:20:06.499 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/13' 2023-10-30T14:20:06.499 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/33' 2023-10-30T14:20:06.499 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/17' 2023-10-30T14:20:06.500 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/35' 2023-10-30T14:20:06.500 INFO:tasks.workunit.client.0.smithi057.stdout:removed './141/37' 2023-10-30T14:20:06.500 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './141' 2023-10-30T14:20:06.500 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/19' 2023-10-30T14:20:06.500 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/15' 2023-10-30T14:20:06.501 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/31' 2023-10-30T14:20:06.501 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/36' 2023-10-30T14:20:06.501 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/20' 2023-10-30T14:20:06.501 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/12' 2023-10-30T14:20:06.501 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/39' 2023-10-30T14:20:06.501 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/10' 2023-10-30T14:20:06.502 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/34' 2023-10-30T14:20:06.502 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/38' 2023-10-30T14:20:06.502 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/11' 2023-10-30T14:20:06.502 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/16' 2023-10-30T14:20:06.502 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/32' 2023-10-30T14:20:06.502 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/30' 2023-10-30T14:20:06.503 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/14' 2023-10-30T14:20:06.503 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/18' 2023-10-30T14:20:06.503 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/13' 2023-10-30T14:20:06.503 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/33' 2023-10-30T14:20:06.503 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/17' 2023-10-30T14:20:06.503 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/35' 2023-10-30T14:20:06.504 INFO:tasks.workunit.client.0.smithi057.stdout:removed './142/37' 2023-10-30T14:20:06.504 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './142' 2023-10-30T14:20:06.504 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/19' 2023-10-30T14:20:06.504 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/15' 2023-10-30T14:20:06.504 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/31' 2023-10-30T14:20:06.504 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/36' 2023-10-30T14:20:06.505 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/20' 2023-10-30T14:20:06.505 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/12' 2023-10-30T14:20:06.505 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/39' 2023-10-30T14:20:06.505 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/10' 2023-10-30T14:20:06.505 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/34' 2023-10-30T14:20:06.506 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/38' 2023-10-30T14:20:06.506 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/11' 2023-10-30T14:20:06.506 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/16' 2023-10-30T14:20:06.506 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/32' 2023-10-30T14:20:06.506 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/30' 2023-10-30T14:20:06.507 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/14' 2023-10-30T14:20:06.507 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/18' 2023-10-30T14:20:06.507 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/13' 2023-10-30T14:20:06.507 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/33' 2023-10-30T14:20:06.507 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/17' 2023-10-30T14:20:06.507 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/35' 2023-10-30T14:20:06.508 INFO:tasks.workunit.client.0.smithi057.stdout:removed './143/37' 2023-10-30T14:20:06.508 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './143' 2023-10-30T14:20:06.508 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/19' 2023-10-30T14:20:06.508 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/15' 2023-10-30T14:20:06.508 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/31' 2023-10-30T14:20:06.508 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/36' 2023-10-30T14:20:06.509 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/20' 2023-10-30T14:20:06.509 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/12' 2023-10-30T14:20:06.509 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/39' 2023-10-30T14:20:06.509 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/10' 2023-10-30T14:20:06.509 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/34' 2023-10-30T14:20:06.510 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/38' 2023-10-30T14:20:06.510 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/11' 2023-10-30T14:20:06.510 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/16' 2023-10-30T14:20:06.510 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/32' 2023-10-30T14:20:06.510 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/30' 2023-10-30T14:20:06.511 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/14' 2023-10-30T14:20:06.511 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/18' 2023-10-30T14:20:06.511 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/13' 2023-10-30T14:20:06.511 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/33' 2023-10-30T14:20:06.511 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/17' 2023-10-30T14:20:06.511 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/35' 2023-10-30T14:20:06.512 INFO:tasks.workunit.client.0.smithi057.stdout:removed './144/37' 2023-10-30T14:20:06.512 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './144' 2023-10-30T14:20:06.512 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/19' 2023-10-30T14:20:06.512 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/15' 2023-10-30T14:20:06.512 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/31' 2023-10-30T14:20:06.512 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/36' 2023-10-30T14:20:06.513 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/20' 2023-10-30T14:20:06.513 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/12' 2023-10-30T14:20:06.513 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/39' 2023-10-30T14:20:06.513 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/10' 2023-10-30T14:20:06.513 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/34' 2023-10-30T14:20:06.514 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/38' 2023-10-30T14:20:06.514 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/11' 2023-10-30T14:20:06.514 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/16' 2023-10-30T14:20:06.514 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/32' 2023-10-30T14:20:06.514 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/30' 2023-10-30T14:20:06.515 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/14' 2023-10-30T14:20:06.515 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/18' 2023-10-30T14:20:06.515 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/13' 2023-10-30T14:20:06.515 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/33' 2023-10-30T14:20:06.515 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/17' 2023-10-30T14:20:06.515 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/35' 2023-10-30T14:20:06.516 INFO:tasks.workunit.client.0.smithi057.stdout:removed './145/37' 2023-10-30T14:20:06.516 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './145' 2023-10-30T14:20:06.516 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/19' 2023-10-30T14:20:06.516 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/15' 2023-10-30T14:20:06.517 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/31' 2023-10-30T14:20:06.517 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/36' 2023-10-30T14:20:06.517 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/20' 2023-10-30T14:20:06.517 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/12' 2023-10-30T14:20:06.517 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/39' 2023-10-30T14:20:06.517 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/10' 2023-10-30T14:20:06.518 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/34' 2023-10-30T14:20:06.518 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/38' 2023-10-30T14:20:06.518 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/11' 2023-10-30T14:20:06.518 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/16' 2023-10-30T14:20:06.518 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/32' 2023-10-30T14:20:06.519 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/30' 2023-10-30T14:20:06.519 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/14' 2023-10-30T14:20:06.519 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/18' 2023-10-30T14:20:06.519 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/13' 2023-10-30T14:20:06.519 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/33' 2023-10-30T14:20:06.520 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/17' 2023-10-30T14:20:06.520 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/35' 2023-10-30T14:20:06.520 INFO:tasks.workunit.client.0.smithi057.stdout:removed './146/37' 2023-10-30T14:20:06.520 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './146' 2023-10-30T14:20:06.520 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/19' 2023-10-30T14:20:06.521 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/15' 2023-10-30T14:20:06.521 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/31' 2023-10-30T14:20:06.521 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/36' 2023-10-30T14:20:06.521 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/20' 2023-10-30T14:20:06.521 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/12' 2023-10-30T14:20:06.521 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/39' 2023-10-30T14:20:06.522 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/10' 2023-10-30T14:20:06.522 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/34' 2023-10-30T14:20:06.522 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/38' 2023-10-30T14:20:06.522 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/11' 2023-10-30T14:20:06.522 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/16' 2023-10-30T14:20:06.523 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/32' 2023-10-30T14:20:06.523 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/30' 2023-10-30T14:20:06.523 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/14' 2023-10-30T14:20:06.523 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/18' 2023-10-30T14:20:06.523 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/13' 2023-10-30T14:20:06.524 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/33' 2023-10-30T14:20:06.524 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/17' 2023-10-30T14:20:06.524 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/35' 2023-10-30T14:20:06.524 INFO:tasks.workunit.client.0.smithi057.stdout:removed './147/37' 2023-10-30T14:20:06.524 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './147' 2023-10-30T14:20:06.525 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/19' 2023-10-30T14:20:06.525 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/15' 2023-10-30T14:20:06.525 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/31' 2023-10-30T14:20:06.525 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/36' 2023-10-30T14:20:12.592 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/20' 2023-10-30T14:20:12.592 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/12' 2023-10-30T14:20:12.593 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/39' 2023-10-30T14:20:12.593 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/10' 2023-10-30T14:20:12.593 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/34' 2023-10-30T14:20:12.593 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/38' 2023-10-30T14:20:12.594 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/11' 2023-10-30T14:20:12.594 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/16' 2023-10-30T14:20:12.594 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/32' 2023-10-30T14:20:12.594 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/30' 2023-10-30T14:20:12.594 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/14' 2023-10-30T14:20:12.595 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/18' 2023-10-30T14:20:12.595 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/13' 2023-10-30T14:20:12.595 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/33' 2023-10-30T14:20:12.595 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/17' 2023-10-30T14:20:12.595 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/35' 2023-10-30T14:20:12.596 INFO:tasks.workunit.client.0.smithi057.stdout:removed './148/37' 2023-10-30T14:20:12.596 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './148' 2023-10-30T14:20:12.596 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/19' 2023-10-30T14:20:12.596 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/15' 2023-10-30T14:20:12.596 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/31' 2023-10-30T14:20:12.597 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/36' 2023-10-30T14:20:12.597 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/20' 2023-10-30T14:20:12.597 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/12' 2023-10-30T14:20:12.597 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/39' 2023-10-30T14:20:12.598 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/10' 2023-10-30T14:20:12.598 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/34' 2023-10-30T14:20:12.598 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/38' 2023-10-30T14:20:12.598 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/11' 2023-10-30T14:20:12.598 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/16' 2023-10-30T14:20:12.599 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/32' 2023-10-30T14:20:12.599 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/30' 2023-10-30T14:20:12.599 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/14' 2023-10-30T14:20:12.599 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/18' 2023-10-30T14:20:12.599 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/13' 2023-10-30T14:20:12.599 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/33' 2023-10-30T14:20:12.600 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/17' 2023-10-30T14:20:12.600 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/35' 2023-10-30T14:20:12.600 INFO:tasks.workunit.client.0.smithi057.stdout:removed './149/37' 2023-10-30T14:20:12.600 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './149' 2023-10-30T14:20:12.600 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/19' 2023-10-30T14:20:12.601 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/15' 2023-10-30T14:20:12.601 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/31' 2023-10-30T14:20:12.601 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/36' 2023-10-30T14:20:12.601 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/20' 2023-10-30T14:20:12.601 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/12' 2023-10-30T14:20:12.602 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/39' 2023-10-30T14:20:12.602 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/10' 2023-10-30T14:20:12.602 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/34' 2023-10-30T14:20:12.602 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/38' 2023-10-30T14:20:12.602 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/11' 2023-10-30T14:20:12.603 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/16' 2023-10-30T14:20:12.603 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/32' 2023-10-30T14:20:12.603 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/30' 2023-10-30T14:20:12.603 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/14' 2023-10-30T14:20:12.603 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/18' 2023-10-30T14:20:12.604 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/13' 2023-10-30T14:20:12.604 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/33' 2023-10-30T14:20:12.604 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/17' 2023-10-30T14:20:12.604 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/35' 2023-10-30T14:20:12.605 INFO:tasks.workunit.client.0.smithi057.stdout:removed './150/37' 2023-10-30T14:20:12.605 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './150' 2023-10-30T14:20:12.605 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/19' 2023-10-30T14:20:12.605 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/15' 2023-10-30T14:20:12.605 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/31' 2023-10-30T14:20:12.606 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/36' 2023-10-30T14:20:12.606 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/20' 2023-10-30T14:20:12.606 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/12' 2023-10-30T14:20:12.606 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/39' 2023-10-30T14:20:12.606 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/10' 2023-10-30T14:20:12.607 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/34' 2023-10-30T14:20:12.607 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/38' 2023-10-30T14:20:12.607 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/11' 2023-10-30T14:20:12.607 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/16' 2023-10-30T14:20:12.607 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/32' 2023-10-30T14:20:12.608 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/30' 2023-10-30T14:20:12.608 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/14' 2023-10-30T14:20:12.608 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/18' 2023-10-30T14:20:12.608 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/13' 2023-10-30T14:20:12.608 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/33' 2023-10-30T14:20:12.609 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/17' 2023-10-30T14:20:12.609 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/35' 2023-10-30T14:20:12.609 INFO:tasks.workunit.client.0.smithi057.stdout:removed './151/37' 2023-10-30T14:20:12.609 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './151' 2023-10-30T14:20:12.610 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/19' 2023-10-30T14:20:12.610 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/15' 2023-10-30T14:20:12.610 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/31' 2023-10-30T14:20:12.610 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/36' 2023-10-30T14:20:12.610 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/20' 2023-10-30T14:20:12.611 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/12' 2023-10-30T14:20:12.611 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/39' 2023-10-30T14:20:12.611 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/10' 2023-10-30T14:20:12.611 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/34' 2023-10-30T14:20:12.611 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/38' 2023-10-30T14:20:12.612 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/11' 2023-10-30T14:20:12.612 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/16' 2023-10-30T14:20:12.612 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/32' 2023-10-30T14:20:12.612 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/30' 2023-10-30T14:20:12.613 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/14' 2023-10-30T14:20:12.613 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/18' 2023-10-30T14:20:12.613 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/13' 2023-10-30T14:20:12.613 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/33' 2023-10-30T14:20:12.613 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/17' 2023-10-30T14:20:12.613 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/35' 2023-10-30T14:20:12.614 INFO:tasks.workunit.client.0.smithi057.stdout:removed './152/37' 2023-10-30T14:20:12.614 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './152' 2023-10-30T14:20:12.614 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/19' 2023-10-30T14:20:12.614 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/15' 2023-10-30T14:20:12.614 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/31' 2023-10-30T14:20:12.615 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/36' 2023-10-30T14:20:12.615 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/20' 2023-10-30T14:20:12.615 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/12' 2023-10-30T14:20:12.615 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/39' 2023-10-30T14:20:12.615 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/10' 2023-10-30T14:20:12.616 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/34' 2023-10-30T14:20:12.616 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/38' 2023-10-30T14:20:12.616 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/11' 2023-10-30T14:20:12.616 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/16' 2023-10-30T14:20:12.616 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/32' 2023-10-30T14:20:12.616 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/30' 2023-10-30T14:20:12.617 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/14' 2023-10-30T14:20:12.617 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/18' 2023-10-30T14:20:12.617 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/13' 2023-10-30T14:20:12.617 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/33' 2023-10-30T14:20:12.617 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/17' 2023-10-30T14:20:12.618 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/35' 2023-10-30T14:20:12.618 INFO:tasks.workunit.client.0.smithi057.stdout:removed './153/37' 2023-10-30T14:20:12.618 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './153' 2023-10-30T14:20:12.618 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/19' 2023-10-30T14:20:12.618 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/15' 2023-10-30T14:20:12.619 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/31' 2023-10-30T14:20:12.619 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/36' 2023-10-30T14:20:12.619 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/20' 2023-10-30T14:20:12.619 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/12' 2023-10-30T14:20:12.619 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/39' 2023-10-30T14:20:12.620 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/10' 2023-10-30T14:20:12.620 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/34' 2023-10-30T14:20:12.620 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/38' 2023-10-30T14:20:12.620 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/11' 2023-10-30T14:20:12.620 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/16' 2023-10-30T14:20:12.621 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/32' 2023-10-30T14:20:12.621 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/30' 2023-10-30T14:20:12.621 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/14' 2023-10-30T14:20:12.621 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/18' 2023-10-30T14:20:12.621 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/13' 2023-10-30T14:20:12.622 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/33' 2023-10-30T14:20:12.622 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/17' 2023-10-30T14:20:12.622 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/35' 2023-10-30T14:20:12.622 INFO:tasks.workunit.client.0.smithi057.stdout:removed './154/37' 2023-10-30T14:20:12.622 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './154' 2023-10-30T14:20:12.623 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/19' 2023-10-30T14:20:12.623 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/15' 2023-10-30T14:20:12.623 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/31' 2023-10-30T14:20:12.623 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/36' 2023-10-30T14:20:12.624 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/20' 2023-10-30T14:20:12.624 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/12' 2023-10-30T14:20:12.624 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/39' 2023-10-30T14:20:12.624 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/10' 2023-10-30T14:20:12.624 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/34' 2023-10-30T14:20:12.624 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/38' 2023-10-30T14:20:12.625 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/11' 2023-10-30T14:20:12.625 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/16' 2023-10-30T14:20:12.625 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/32' 2023-10-30T14:20:12.625 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/30' 2023-10-30T14:20:12.625 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/14' 2023-10-30T14:20:12.626 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/18' 2023-10-30T14:20:12.626 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/13' 2023-10-30T14:20:12.627 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/33' 2023-10-30T14:20:12.627 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/17' 2023-10-30T14:20:12.627 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/35' 2023-10-30T14:20:12.627 INFO:tasks.workunit.client.0.smithi057.stdout:removed './155/37' 2023-10-30T14:20:12.627 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './155' 2023-10-30T14:20:12.627 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/19' 2023-10-30T14:20:12.628 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/15' 2023-10-30T14:20:12.628 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/31' 2023-10-30T14:20:12.628 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/36' 2023-10-30T14:20:12.628 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/20' 2023-10-30T14:20:12.628 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/12' 2023-10-30T14:20:12.629 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/39' 2023-10-30T14:20:12.629 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/10' 2023-10-30T14:20:12.629 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/34' 2023-10-30T14:20:12.629 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/38' 2023-10-30T14:20:12.629 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/11' 2023-10-30T14:20:12.629 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/16' 2023-10-30T14:20:12.630 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/32' 2023-10-30T14:20:12.630 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/30' 2023-10-30T14:20:12.630 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/14' 2023-10-30T14:20:12.630 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/18' 2023-10-30T14:20:12.630 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/13' 2023-10-30T14:20:12.631 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/33' 2023-10-30T14:20:12.631 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/17' 2023-10-30T14:20:12.631 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/35' 2023-10-30T14:20:12.631 INFO:tasks.workunit.client.0.smithi057.stdout:removed './156/37' 2023-10-30T14:20:12.631 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './156' 2023-10-30T14:20:12.632 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/19' 2023-10-30T14:20:12.632 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/15' 2023-10-30T14:20:12.632 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/31' 2023-10-30T14:20:12.632 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/36' 2023-10-30T14:20:12.632 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/20' 2023-10-30T14:20:12.632 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/12' 2023-10-30T14:20:12.633 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/39' 2023-10-30T14:20:12.633 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/10' 2023-10-30T14:20:12.633 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/34' 2023-10-30T14:20:12.633 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/38' 2023-10-30T14:20:12.633 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/11' 2023-10-30T14:20:12.633 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/16' 2023-10-30T14:20:12.634 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/32' 2023-10-30T14:20:12.634 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/30' 2023-10-30T14:20:12.634 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/14' 2023-10-30T14:20:12.634 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/18' 2023-10-30T14:20:12.634 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/13' 2023-10-30T14:20:12.634 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/33' 2023-10-30T14:20:15.109 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/17' 2023-10-30T14:20:15.110 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/35' 2023-10-30T14:20:15.110 INFO:tasks.workunit.client.0.smithi057.stdout:removed './157/37' 2023-10-30T14:20:15.110 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './157' 2023-10-30T14:20:15.110 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/19' 2023-10-30T14:20:15.110 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/15' 2023-10-30T14:20:15.111 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/31' 2023-10-30T14:20:15.111 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/36' 2023-10-30T14:20:15.111 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/20' 2023-10-30T14:20:15.111 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/12' 2023-10-30T14:20:15.111 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/39' 2023-10-30T14:20:15.112 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/10' 2023-10-30T14:20:15.112 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/34' 2023-10-30T14:20:15.112 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/38' 2023-10-30T14:20:15.112 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/11' 2023-10-30T14:20:15.112 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/16' 2023-10-30T14:20:15.113 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/32' 2023-10-30T14:20:15.113 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/30' 2023-10-30T14:20:15.113 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/14' 2023-10-30T14:20:15.113 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/18' 2023-10-30T14:20:15.113 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/13' 2023-10-30T14:20:15.114 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/33' 2023-10-30T14:20:15.114 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/17' 2023-10-30T14:20:15.114 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/35' 2023-10-30T14:20:15.114 INFO:tasks.workunit.client.0.smithi057.stdout:removed './158/37' 2023-10-30T14:20:15.114 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './158' 2023-10-30T14:20:15.115 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/19' 2023-10-30T14:20:15.115 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/15' 2023-10-30T14:20:15.115 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/31' 2023-10-30T14:20:15.115 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/36' 2023-10-30T14:20:15.115 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/20' 2023-10-30T14:20:15.116 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/12' 2023-10-30T14:20:15.116 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/39' 2023-10-30T14:20:15.116 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/10' 2023-10-30T14:20:15.116 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/34' 2023-10-30T14:20:15.116 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/38' 2023-10-30T14:20:15.117 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/11' 2023-10-30T14:20:15.117 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/16' 2023-10-30T14:20:15.117 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/32' 2023-10-30T14:20:15.117 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/30' 2023-10-30T14:20:15.117 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/14' 2023-10-30T14:20:15.117 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/18' 2023-10-30T14:20:15.118 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/13' 2023-10-30T14:20:15.118 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/33' 2023-10-30T14:20:15.118 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/17' 2023-10-30T14:20:15.118 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/35' 2023-10-30T14:20:15.119 INFO:tasks.workunit.client.0.smithi057.stdout:removed './159/37' 2023-10-30T14:20:15.119 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './159' 2023-10-30T14:20:15.119 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/19' 2023-10-30T14:20:15.119 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/15' 2023-10-30T14:20:15.119 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/31' 2023-10-30T14:20:15.120 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/36' 2023-10-30T14:20:15.120 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/20' 2023-10-30T14:20:15.120 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/12' 2023-10-30T14:20:15.120 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/39' 2023-10-30T14:20:15.120 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/10' 2023-10-30T14:20:15.121 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/34' 2023-10-30T14:20:15.121 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/38' 2023-10-30T14:20:15.121 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/11' 2023-10-30T14:20:15.121 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/16' 2023-10-30T14:20:15.121 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/32' 2023-10-30T14:20:15.122 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/30' 2023-10-30T14:20:15.122 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/14' 2023-10-30T14:20:15.122 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/18' 2023-10-30T14:20:15.122 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/13' 2023-10-30T14:20:15.122 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/33' 2023-10-30T14:20:15.123 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/17' 2023-10-30T14:20:15.123 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/35' 2023-10-30T14:20:15.123 INFO:tasks.workunit.client.0.smithi057.stdout:removed './160/37' 2023-10-30T14:20:15.123 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './160' 2023-10-30T14:20:15.123 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/19' 2023-10-30T14:20:15.124 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/15' 2023-10-30T14:20:15.124 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/31' 2023-10-30T14:20:15.124 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/36' 2023-10-30T14:20:15.124 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/20' 2023-10-30T14:20:15.124 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/12' 2023-10-30T14:20:15.124 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/39' 2023-10-30T14:20:15.125 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/10' 2023-10-30T14:20:15.125 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/34' 2023-10-30T14:20:15.125 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/38' 2023-10-30T14:20:15.125 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/11' 2023-10-30T14:20:15.125 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/16' 2023-10-30T14:20:15.125 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/32' 2023-10-30T14:20:15.126 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/30' 2023-10-30T14:20:15.126 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/14' 2023-10-30T14:20:15.126 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/18' 2023-10-30T14:20:15.126 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/13' 2023-10-30T14:20:15.127 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/33' 2023-10-30T14:20:15.127 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/17' 2023-10-30T14:20:15.127 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/35' 2023-10-30T14:20:15.127 INFO:tasks.workunit.client.0.smithi057.stdout:removed './161/37' 2023-10-30T14:20:15.127 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './161' 2023-10-30T14:20:15.127 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/19' 2023-10-30T14:20:15.128 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/15' 2023-10-30T14:20:15.128 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/31' 2023-10-30T14:20:15.128 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/36' 2023-10-30T14:20:15.128 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/20' 2023-10-30T14:20:15.129 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/12' 2023-10-30T14:20:15.129 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/39' 2023-10-30T14:20:15.129 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/10' 2023-10-30T14:20:15.129 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/34' 2023-10-30T14:20:15.129 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/38' 2023-10-30T14:20:15.129 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/11' 2023-10-30T14:20:15.130 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/16' 2023-10-30T14:20:15.130 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/32' 2023-10-30T14:20:15.130 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/30' 2023-10-30T14:20:15.130 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/14' 2023-10-30T14:20:15.130 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/18' 2023-10-30T14:20:15.131 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/13' 2023-10-30T14:20:15.131 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/33' 2023-10-30T14:20:15.131 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/17' 2023-10-30T14:20:15.131 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/35' 2023-10-30T14:20:15.131 INFO:tasks.workunit.client.0.smithi057.stdout:removed './162/37' 2023-10-30T14:20:15.132 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './162' 2023-10-30T14:20:15.132 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/19' 2023-10-30T14:20:15.132 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/15' 2023-10-30T14:20:15.132 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/31' 2023-10-30T14:20:15.132 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/36' 2023-10-30T14:20:15.133 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/20' 2023-10-30T14:20:15.133 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/12' 2023-10-30T14:20:15.133 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/39' 2023-10-30T14:20:15.133 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/10' 2023-10-30T14:20:15.133 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/34' 2023-10-30T14:20:15.134 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/38' 2023-10-30T14:20:15.134 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/11' 2023-10-30T14:20:15.134 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/16' 2023-10-30T14:20:15.134 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/32' 2023-10-30T14:20:15.134 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/30' 2023-10-30T14:20:15.135 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/14' 2023-10-30T14:20:15.135 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/18' 2023-10-30T14:20:15.135 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/13' 2023-10-30T14:20:15.135 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/33' 2023-10-30T14:20:15.135 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/17' 2023-10-30T14:20:15.136 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/35' 2023-10-30T14:20:15.136 INFO:tasks.workunit.client.0.smithi057.stdout:removed './163/37' 2023-10-30T14:20:15.136 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './163' 2023-10-30T14:20:15.136 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/19' 2023-10-30T14:20:15.136 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/15' 2023-10-30T14:20:15.137 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/31' 2023-10-30T14:20:15.137 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/36' 2023-10-30T14:20:15.137 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/20' 2023-10-30T14:20:15.137 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/12' 2023-10-30T14:20:15.137 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/39' 2023-10-30T14:20:15.138 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/10' 2023-10-30T14:20:15.138 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/34' 2023-10-30T14:20:15.138 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/38' 2023-10-30T14:20:15.138 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/11' 2023-10-30T14:20:15.138 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/16' 2023-10-30T14:20:15.139 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/32' 2023-10-30T14:20:15.139 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/30' 2023-10-30T14:20:15.139 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/14' 2023-10-30T14:20:15.139 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/18' 2023-10-30T14:20:15.139 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/13' 2023-10-30T14:20:15.140 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/33' 2023-10-30T14:20:15.140 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/17' 2023-10-30T14:20:15.140 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/35' 2023-10-30T14:20:15.140 INFO:tasks.workunit.client.0.smithi057.stdout:removed './164/37' 2023-10-30T14:20:15.140 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './164' 2023-10-30T14:20:15.140 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/19' 2023-10-30T14:20:15.141 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/15' 2023-10-30T14:20:15.141 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/31' 2023-10-30T14:20:15.141 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/36' 2023-10-30T14:20:15.141 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/20' 2023-10-30T14:20:15.141 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/12' 2023-10-30T14:20:15.141 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/39' 2023-10-30T14:20:15.142 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/10' 2023-10-30T14:20:15.142 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/34' 2023-10-30T14:20:15.142 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/38' 2023-10-30T14:20:15.142 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/11' 2023-10-30T14:20:15.142 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/16' 2023-10-30T14:20:15.143 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/32' 2023-10-30T14:20:15.143 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/30' 2023-10-30T14:20:15.143 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/14' 2023-10-30T14:20:15.143 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/18' 2023-10-30T14:20:15.143 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/13' 2023-10-30T14:20:15.143 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/33' 2023-10-30T14:20:15.144 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/17' 2023-10-30T14:20:15.144 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/35' 2023-10-30T14:20:15.144 INFO:tasks.workunit.client.0.smithi057.stdout:removed './165/37' 2023-10-30T14:20:15.144 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './165' 2023-10-30T14:20:15.144 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/19' 2023-10-30T14:20:15.145 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/15' 2023-10-30T14:20:15.145 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/31' 2023-10-30T14:20:15.145 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/36' 2023-10-30T14:20:15.145 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/20' 2023-10-30T14:20:15.145 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/12' 2023-10-30T14:20:15.145 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/39' 2023-10-30T14:20:15.146 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/10' 2023-10-30T14:20:15.146 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/34' 2023-10-30T14:20:15.146 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/38' 2023-10-30T14:20:15.146 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/11' 2023-10-30T14:20:15.146 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/16' 2023-10-30T14:20:15.147 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/32' 2023-10-30T14:20:15.147 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/30' 2023-10-30T14:20:15.147 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/14' 2023-10-30T14:20:15.147 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/18' 2023-10-30T14:20:15.147 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/13' 2023-10-30T14:20:15.147 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/33' 2023-10-30T14:20:15.148 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/17' 2023-10-30T14:20:15.148 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/35' 2023-10-30T14:20:15.148 INFO:tasks.workunit.client.0.smithi057.stdout:removed './166/37' 2023-10-30T14:20:15.148 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './166' 2023-10-30T14:20:15.148 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/19' 2023-10-30T14:20:15.148 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/15' 2023-10-30T14:20:15.149 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/31' 2023-10-30T14:20:15.149 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/36' 2023-10-30T14:20:15.149 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/20' 2023-10-30T14:20:15.149 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/12' 2023-10-30T14:20:15.149 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/39' 2023-10-30T14:20:15.150 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/10' 2023-10-30T14:20:15.150 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/34' 2023-10-30T14:20:15.150 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/38' 2023-10-30T14:20:17.980 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:18.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.119+0000 7f7e54d3e640 1 Processor -- start 2023-10-30T14:20:18.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.119+0000 7f7e54d3e640 1 -- start start 2023-10-30T14:20:18.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.119+0000 7f7e54d3e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7e500cb7d0 0x7f7e500c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:18.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.119+0000 7f7e54d3e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e500c3e40 0x7f7e50070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:18.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.119+0000 7f7e54d3e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e500c2310 con 0x7f7e50150830 2023-10-30T14:20:18.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.119+0000 7f7e54d3e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5006bc10 con 0x7f7e500cb7d0 2023-10-30T14:20:18.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.119+0000 7f7e54d3e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e50156710 con 0x7f7e500c3e40 2023-10-30T14:20:18.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.119+0000 7f7e4dd9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7e500cb7d0 0x7f7e500c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:18.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.119+0000 7f7e4dd9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7e500cb7d0 0x7f7e500c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44932/0 (socket says 172.21.15.57:44932) 2023-10-30T14:20:18.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.119+0000 7f7e4dd9b640 1 -- 172.21.15.57:0/2922348068 learned_addr learned my addr 172.21.15.57:0/2922348068 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:18.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.123+0000 7f7e4ed9d640 1 --2- 172.21.15.57:0/2922348068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e500c3e40 0x7f7e50070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:18.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.123+0000 7f7e4dd9b640 1 -- 172.21.15.57:0/2922348068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e500c3e40 msgr2=0x7f7e50070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:18.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.123+0000 7f7e4dd9b640 1 --2- 172.21.15.57:0/2922348068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e500c3e40 0x7f7e50070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.123+0000 7f7e4dd9b640 1 -- 172.21.15.57:0/2922348068 >> v1:172.21.15.88:6789/0 conn(0x7f7e50150830 legacy=0x7f7e5006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:18.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.123+0000 7f7e4dd9b640 1 -- 172.21.15.57:0/2922348068 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e500c4360 con 0x7f7e500cb7d0 2023-10-30T14:20:18.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.123+0000 7f7e4dd9b640 1 --2- 172.21.15.57:0/2922348068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e500cb7d0 0x7f7e500c3900 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7e44009870 tx=0x7f7e4401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa50d2ad372f818d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:18.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.123+0000 7f7e4d59a640 1 -- 172.21.15.57:0/2922348068 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e440a8a30 con 0x7f7e500cb7d0 2023-10-30T14:20:18.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.123+0000 7f7e4d59a640 1 -- 172.21.15.57:0/2922348068 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e440a8b90 con 0x7f7e500cb7d0 2023-10-30T14:20:18.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.123+0000 7f7e4d59a640 1 -- 172.21.15.57:0/2922348068 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e440b0440 con 0x7f7e500cb7d0 2023-10-30T14:20:18.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.123+0000 7f7e54d3e640 1 -- 172.21.15.57:0/2922348068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e500cb7d0 msgr2=0x7f7e500c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:18.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.123+0000 7f7e54d3e640 1 --2- 172.21.15.57:0/2922348068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e500cb7d0 0x7f7e500c3900 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7e44009870 tx=0x7f7e4401ed30 comp rx=0 tx=0).stop 2023-10-30T14:20:18.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e54d3e640 1 -- 172.21.15.57:0/2922348068 shutdown_connections 2023-10-30T14:20:18.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e54d3e640 1 --2- 172.21.15.57:0/2922348068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e500c3e40 0x7f7e50070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e54d3e640 1 --2- 172.21.15.57:0/2922348068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e500cb7d0 0x7f7e500c3900 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e54d3e640 1 -- 172.21.15.57:0/2922348068 >> 172.21.15.57:0/2922348068 conn(0x7f7e500cac70 msgr2=0x7f7e500cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:18.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e54d3e640 1 -- 172.21.15.57:0/2922348068 shutdown_connections 2023-10-30T14:20:18.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e54d3e640 1 -- 172.21.15.57:0/2922348068 wait complete. 2023-10-30T14:20:18.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e54d3e640 1 Processor -- start 2023-10-30T14:20:18.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e54d3e640 1 -- start start 2023-10-30T14:20:18.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e54d3e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e500c3e40 0x7f7e50151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:18.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e54d3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e500cb7d0 0x7f7e50159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:18.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e54d3e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e50150830 0x7f7e5014abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:18.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e54d3e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e501475d0 con 0x7f7e50150830 2023-10-30T14:20:18.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e54d3e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e50147440 con 0x7f7e500cb7d0 2023-10-30T14:20:18.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e54d3e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e501ede30 con 0x7f7e500c3e40 2023-10-30T14:20:18.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e4ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e50150830 0x7f7e5014abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:18.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e4ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e50150830 0x7f7e5014abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44938/0 (socket says 172.21.15.57:44938) 2023-10-30T14:20:18.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e4ed9d640 1 -- 172.21.15.57:0/1951414875 learned_addr learned my addr 172.21.15.57:0/1951414875 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:18.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e4dd9b640 1 --2- 172.21.15.57:0/1951414875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e500cb7d0 0x7f7e50159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:18.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e4e59c640 1 --2- 172.21.15.57:0/1951414875 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e500c3e40 0x7f7e50151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:18.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e4ed9d640 1 -- 172.21.15.57:0/1951414875 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e500c3e40 msgr2=0x7f7e50151690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:18.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e4ed9d640 1 --2- 172.21.15.57:0/1951414875 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e500c3e40 0x7f7e50151690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e4ed9d640 1 -- 172.21.15.57:0/1951414875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e500cb7d0 msgr2=0x7f7e50159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:18.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e4ed9d640 1 --2- 172.21.15.57:0/1951414875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e500cb7d0 0x7f7e50159bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.127+0000 7f7e4ed9d640 1 -- 172.21.15.57:0/1951414875 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e440094d0 con 0x7f7e50150830 2023-10-30T14:20:18.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.131+0000 7f7e4ed9d640 1 --2- 172.21.15.57:0/1951414875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e50150830 0x7f7e5014abb0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7e48007d80 tx=0x7f7e4800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:18.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.131+0000 7f7e3effd640 1 -- 172.21.15.57:0/1951414875 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e48014420 con 0x7f7e50150830 2023-10-30T14:20:18.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.131+0000 7f7e3effd640 1 -- 172.21.15.57:0/1951414875 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e48014580 con 0x7f7e50150830 2023-10-30T14:20:18.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.131+0000 7f7e3effd640 1 -- 172.21.15.57:0/1951414875 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e48014930 con 0x7f7e50150830 2023-10-30T14:20:18.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.131+0000 7f7e54d3e640 1 -- 172.21.15.57:0/1951414875 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e501ed8b0 con 0x7f7e50150830 2023-10-30T14:20:18.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.131+0000 7f7e54d3e640 1 -- 172.21.15.57:0/1951414875 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e501edb50 con 0x7f7e50150830 2023-10-30T14:20:18.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.131+0000 7f7e54d3e640 1 -- 172.21.15.57:0/1951414875 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e5004e170 con 0x7f7e50150830 2023-10-30T14:20:18.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.131+0000 7f7e3effd640 1 -- 172.21.15.57:0/1951414875 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7e48014a90 con 0x7f7e50150830 2023-10-30T14:20:18.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.131+0000 7f7e3effd640 1 --2- 172.21.15.57:0/1951414875 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7e300664a0 0x7f7e30068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:18.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.135+0000 7f7e3effd640 1 -- 172.21.15.57:0/1951414875 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6613+0+0 (secure 0 0 0) 0x7f7e48084960 con 0x7f7e50150830 2023-10-30T14:20:18.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.135+0000 7f7e4e59c640 1 --2- 172.21.15.57:0/1951414875 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7e300664a0 0x7f7e30068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:18.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.135+0000 7f7e4e59c640 1 --2- 172.21.15.57:0/1951414875 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7e300664a0 0x7f7e30068960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7e40007fd0 tx=0x7f7e40009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:18.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.135+0000 7f7e3effd640 1 -- 172.21.15.57:0/1951414875 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7e48051b90 con 0x7f7e50150830 2023-10-30T14:20:18.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.327+0000 7f7e54d3e640 1 -- 172.21.15.57:0/1951414875 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e500c6c30 con 0x7f7e50150830 2023-10-30T14:20:18.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.327+0000 7f7e3effd640 1 -- 172.21.15.57:0/1951414875 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6533 (secure 0 0 0) 0x7f7e480551b0 con 0x7f7e50150830 2023-10-30T14:20:18.333 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:18.333 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5078,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:15.987759+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:18.333 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 47 2023-10-30T14:20:18.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.327+0000 7f7e54d3e640 1 -- 172.21.15.57:0/1951414875 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7e300664a0 msgr2=0x7f7e30068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:18.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.327+0000 7f7e54d3e640 1 --2- 172.21.15.57:0/1951414875 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7e300664a0 0x7f7e30068960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7e40007fd0 tx=0x7f7e40009040 comp rx=0 tx=0).stop 2023-10-30T14:20:18.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.327+0000 7f7e54d3e640 1 -- 172.21.15.57:0/1951414875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e50150830 msgr2=0x7f7e5014abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:18.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.327+0000 7f7e54d3e640 1 --2- 172.21.15.57:0/1951414875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e50150830 0x7f7e5014abb0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7e48007d80 tx=0x7f7e4800e4b0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.327+0000 7f7e54d3e640 1 -- 172.21.15.57:0/1951414875 shutdown_connections 2023-10-30T14:20:18.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.327+0000 7f7e54d3e640 1 --2- 172.21.15.57:0/1951414875 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7e300664a0 0x7f7e30068960 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.327+0000 7f7e54d3e640 1 --2- 172.21.15.57:0/1951414875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e50150830 0x7f7e5014abb0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.327+0000 7f7e54d3e640 1 --2- 172.21.15.57:0/1951414875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e500cb7d0 0x7f7e50159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.327+0000 7f7e54d3e640 1 --2- 172.21.15.57:0/1951414875 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e500c3e40 0x7f7e50151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.327+0000 7f7e54d3e640 1 -- 172.21.15.57:0/1951414875 >> 172.21.15.57:0/1951414875 conn(0x7f7e500cac70 msgr2=0x7f7e5014ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:18.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.331+0000 7f7e54d3e640 1 -- 172.21.15.57:0/1951414875 shutdown_connections 2023-10-30T14:20:18.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.331+0000 7f7e54d3e640 1 -- 172.21.15.57:0/1951414875 wait complete. 2023-10-30T14:20:18.354 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=2) 2023-10-30T14:20:18.354 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T14:20:18.354 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:20:18.372 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T14:20:18.372 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:18.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.551+0000 7f8e910f1640 1 Processor -- start 2023-10-30T14:20:18.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.551+0000 7f8e910f1640 1 -- start start 2023-10-30T14:20:18.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.551+0000 7f8e910f1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8e8c070520 0x7f8e8c070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:18.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.551+0000 7f8e910f1640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8c0c2310 con 0x7f8e8c150830 2023-10-30T14:20:18.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.551+0000 7f8e910f1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8c156710 con 0x7f8e8c0cb7d0 2023-10-30T14:20:18.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.551+0000 7f8e910f1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e8c06bc10 con 0x7f8e8c070520 2023-10-30T14:20:18.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.551+0000 7f8e8ad9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f8e8c150830 0x7f8e8c06b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:57830/0 (socket says 172.21.15.57:57830) 2023-10-30T14:20:18.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.551+0000 7f8e8ad9d640 1 -- 172.21.15.57:0/1498258622 learned_addr learned my addr 172.21.15.57:0/1498258622 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:20:18.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.551+0000 7f8e8b59e640 1 --2- 172.21.15.57:0/1498258622 >> v2:172.21.15.88:3300/0 conn(0x7f8e8c070520 0x7f8e8c070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:18.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.551+0000 7f8e89d9b640 1 -- 172.21.15.57:0/1498258622 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2317044325 0 0) 0x7f8e8c0c2310 con 0x7f8e8c150830 2023-10-30T14:20:18.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e89d9b640 1 -- 172.21.15.57:0/1498258622 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e74005740 con 0x7f8e8c150830 2023-10-30T14:20:18.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e89d9b640 1 -- 172.21.15.57:0/1498258622 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1689183258 0 0) 0x7f8e74005740 con 0x7f8e8c150830 2023-10-30T14:20:18.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e89d9b640 1 -- 172.21.15.57:0/1498258622 >> v2:172.21.15.88:3300/0 conn(0x7f8e8c070520 msgr2=0x7f8e8c070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:18.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e89d9b640 1 --2- 172.21.15.57:0/1498258622 >> v2:172.21.15.88:3300/0 conn(0x7f8e8c070520 0x7f8e8c070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e89d9b640 1 -- 172.21.15.57:0/1498258622 >> v1:172.21.15.88:6789/0 conn(0x7f8e8c0cb7d0 legacy=0x7f8e8c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:18.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e89d9b640 1 -- 172.21.15.57:0/1498258622 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8c0c4360 con 0x7f8e8c150830 2023-10-30T14:20:18.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e89d9b640 1 -- 172.21.15.57:0/1498258622 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f8e7c001a20 con 0x7f8e8c150830 2023-10-30T14:20:18.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e89d9b640 1 -- 172.21.15.57:0/1498258622 >> v1:172.21.15.57:6789/0 conn(0x7f8e8c150830 legacy=0x7f8e8c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:18.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e89d9b640 1 --2- 172.21.15.57:0/1498258622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e74005b90 0x7f8e74005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:18.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e89d9b640 1 --2- 172.21.15.57:0/1498258622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e740069e0 0x7f8e74006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:18.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e89d9b640 1 --2- 172.21.15.57:0/1498258622 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e740073a0 0x7f8e7400dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:18.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e89d9b640 1 -- 172.21.15.57:0/1498258622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c06bc10 con 0x7f8e74005b90 2023-10-30T14:20:18.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e89d9b640 1 -- 172.21.15.57:0/1498258622 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7400e250 con 0x7f8e740069e0 2023-10-30T14:20:18.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e89d9b640 1 -- 172.21.15.57:0/1498258622 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7400e410 con 0x7f8e740073a0 2023-10-30T14:20:18.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e8ad9d640 1 --2- 172.21.15.57:0/1498258622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e74005b90 0x7f8e74005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:18.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e8a59c640 1 --2- 172.21.15.57:0/1498258622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e740069e0 0x7f8e74006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:18.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e8b59e640 1 --2- 172.21.15.57:0/1498258622 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e740073a0 0x7f8e7400dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:18.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e8ad9d640 1 -- 172.21.15.57:0/1498258622 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e740073a0 msgr2=0x7f8e7400dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:18.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e8ad9d640 1 --2- 172.21.15.57:0/1498258622 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e740073a0 0x7f8e7400dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e8ad9d640 1 -- 172.21.15.57:0/1498258622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e740069e0 msgr2=0x7f8e74006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:18.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e8ad9d640 1 --2- 172.21.15.57:0/1498258622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e740069e0 0x7f8e74006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e8ad9d640 1 -- 172.21.15.57:0/1498258622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e74005740 con 0x7f8e74005b90 2023-10-30T14:20:18.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e8a59c640 1 --2- 172.21.15.57:0/1498258622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e740069e0 0x7f8e74006e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:18.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e8b59e640 1 -- 172.21.15.57:0/1498258622 reap_dead start 2023-10-30T14:20:18.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.555+0000 7f8e8ad9d640 1 --2- 172.21.15.57:0/1498258622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e74005b90 0x7f8e74005fb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c0c24a0 tx=0x7f8e7c01f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31b3dc0967c0fa4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:18.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e89d9b640 1 -- 172.21.15.57:0/1498258622 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c0aa610 con 0x7f8e74005b90 2023-10-30T14:20:18.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e89d9b640 1 -- 172.21.15.57:0/1498258622 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7c0aa770 con 0x7f8e74005b90 2023-10-30T14:20:18.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e89d9b640 1 -- 172.21.15.57:0/1498258622 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c0aab20 con 0x7f8e74005b90 2023-10-30T14:20:18.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e910f1640 1 -- 172.21.15.57:0/1498258622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e74005b90 msgr2=0x7f8e74005fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:18.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e910f1640 1 --2- 172.21.15.57:0/1498258622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e74005b90 0x7f8e74005fb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c0c24a0 tx=0x7f8e7c01f9e0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e910f1640 1 -- 172.21.15.57:0/1498258622 shutdown_connections 2023-10-30T14:20:18.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e910f1640 1 --2- 172.21.15.57:0/1498258622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e74005b90 0x7f8e74005fb0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e910f1640 1 -- 172.21.15.57:0/1498258622 >> 172.21.15.57:0/1498258622 conn(0x7f8e8c0cac70 msgr2=0x7f8e8c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:18.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e910f1640 1 -- 172.21.15.57:0/1498258622 shutdown_connections 2023-10-30T14:20:18.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e910f1640 1 -- 172.21.15.57:0/1498258622 wait complete. 2023-10-30T14:20:18.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e910f1640 1 Processor -- start 2023-10-30T14:20:18.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e910f1640 1 -- start start 2023-10-30T14:20:18.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e910f1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e74005b90 0x7f8e8c0c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:18.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e910f1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e8c159360 0x7f8e8c1eac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:18.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e910f1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e8c1577c0 0x7f8e8c14daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:18.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e910f1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c1597c0 con 0x7f8e8c159360 2023-10-30T14:20:18.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e910f1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c1584e0 con 0x7f8e8c1577c0 2023-10-30T14:20:18.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e910f1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c1ea1b0 con 0x7f8e74005b90 2023-10-30T14:20:18.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e8a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e8c159360 0x7f8e8c1eac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:18.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e8a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e8c159360 0x7f8e8c1eac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44956/0 (socket says 172.21.15.57:44956) 2023-10-30T14:20:18.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.559+0000 7f8e8a59c640 1 -- 172.21.15.57:0/2736247575 learned_addr learned my addr 172.21.15.57:0/2736247575 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:18.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.563+0000 7f8e8ad9d640 1 --2- 172.21.15.57:0/2736247575 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e74005b90 0x7f8e8c0c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:18.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.563+0000 7f8e8b59e640 1 --2- 172.21.15.57:0/2736247575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e8c1577c0 0x7f8e8c14daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:18.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.563+0000 7f8e8a59c640 1 -- 172.21.15.57:0/2736247575 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e74005b90 msgr2=0x7f8e8c0c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:18.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.563+0000 7f8e8a59c640 1 --2- 172.21.15.57:0/2736247575 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e74005b90 0x7f8e8c0c8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.563+0000 7f8e8a59c640 1 -- 172.21.15.57:0/2736247575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e8c1577c0 msgr2=0x7f8e8c14daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:18.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.563+0000 7f8e8a59c640 1 --2- 172.21.15.57:0/2736247575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e8c1577c0 0x7f8e8c14daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.563+0000 7f8e8a59c640 1 -- 172.21.15.57:0/2736247575 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8c0c95a0 con 0x7f8e8c159360 2023-10-30T14:20:18.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.563+0000 7f8e8ad9d640 1 --2- 172.21.15.57:0/2736247575 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e74005b90 0x7f8e8c0c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:18.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.563+0000 7f8e8a59c640 1 --2- 172.21.15.57:0/2736247575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e8c159360 0x7f8e8c1eac70 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000e500 tx=0x7f8e8000e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:18.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.563+0000 7f8e737fe640 1 -- 172.21.15.57:0/2736247575 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e800052e0 con 0x7f8e8c159360 2023-10-30T14:20:18.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.563+0000 7f8e737fe640 1 -- 172.21.15.57:0/2736247575 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e80005440 con 0x7f8e8c159360 2023-10-30T14:20:18.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.563+0000 7f8e910f1640 1 -- 172.21.15.57:0/2736247575 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e8c0c9880 con 0x7f8e8c159360 2023-10-30T14:20:18.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.563+0000 7f8e737fe640 1 -- 172.21.15.57:0/2736247575 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e80003820 con 0x7f8e8c159360 2023-10-30T14:20:18.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.563+0000 7f8e910f1640 1 -- 172.21.15.57:0/2736247575 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e8c1ea470 con 0x7f8e8c159360 2023-10-30T14:20:18.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.563+0000 7f8e910f1640 1 -- 172.21.15.57:0/2736247575 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e8c156990 con 0x7f8e8c159360 2023-10-30T14:20:18.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.563+0000 7f8e737fe640 1 -- 172.21.15.57:0/2736247575 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8e80003980 con 0x7f8e8c159360 2023-10-30T14:20:18.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.567+0000 7f8e737fe640 1 --2- 172.21.15.57:0/2736247575 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8e780664d0 0x7f8e78068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:18.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.567+0000 7f8e737fe640 1 -- 172.21.15.57:0/2736247575 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6613+0+0 (secure 0 0 0) 0x7f8e80083be0 con 0x7f8e8c159360 2023-10-30T14:20:18.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.567+0000 7f8e8ad9d640 1 --2- 172.21.15.57:0/2736247575 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8e780664d0 0x7f8e78068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:18.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.567+0000 7f8e8ad9d640 1 --2- 172.21.15.57:0/2736247575 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8e780664d0 0x7f8e78068990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c0b0000 tx=0x7f8e7c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:18.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.571+0000 7f8e737fe640 1 -- 172.21.15.57:0/2736247575 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8e8c156990 con 0x7f8e8c159360 2023-10-30T14:20:18.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.759+0000 7f8e910f1640 1 -- 172.21.15.57:0/2736247575 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e8c156990 con 0x7f8e8c159360 2023-10-30T14:20:18.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.763+0000 7f8e737fe640 1 -- 172.21.15.57:0/2736247575 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6533 (secure 0 0 0) 0x7f8e8c156990 con 0x7f8e8c159360 2023-10-30T14:20:18.767 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:18.767 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5078,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:15.987759+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:18.768 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 47 2023-10-30T14:20:18.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.763+0000 7f8e910f1640 1 -- 172.21.15.57:0/2736247575 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8e780664d0 msgr2=0x7f8e78068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:18.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.763+0000 7f8e910f1640 1 --2- 172.21.15.57:0/2736247575 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8e780664d0 0x7f8e78068990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c0b0000 tx=0x7f8e7c009040 comp rx=0 tx=0).stop 2023-10-30T14:20:18.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.763+0000 7f8e910f1640 1 -- 172.21.15.57:0/2736247575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e8c159360 msgr2=0x7f8e8c1eac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:18.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.763+0000 7f8e910f1640 1 --2- 172.21.15.57:0/2736247575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e8c159360 0x7f8e8c1eac70 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000e500 tx=0x7f8e8000e9c0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.763+0000 7f8e910f1640 1 -- 172.21.15.57:0/2736247575 shutdown_connections 2023-10-30T14:20:18.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.763+0000 7f8e910f1640 1 --2- 172.21.15.57:0/2736247575 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8e780664d0 0x7f8e78068990 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.763+0000 7f8e910f1640 1 --2- 172.21.15.57:0/2736247575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e8c1577c0 0x7f8e8c14daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.763+0000 7f8e910f1640 1 --2- 172.21.15.57:0/2736247575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e8c159360 0x7f8e8c1eac70 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.763+0000 7f8e910f1640 1 --2- 172.21.15.57:0/2736247575 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e74005b90 0x7f8e8c0c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:18.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.763+0000 7f8e910f1640 1 -- 172.21.15.57:0/2736247575 >> 172.21.15.57:0/2736247575 conn(0x7f8e8c0cac70 msgr2=0x7f8e8c14e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:18.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.763+0000 7f8e910f1640 1 -- 172.21.15.57:0/2736247575 shutdown_connections 2023-10-30T14:20:18.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:18.763+0000 7f8e910f1640 1 -- 172.21.15.57:0/2736247575 wait complete. 2023-10-30T14:20:18.788 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:20:20.790 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:20.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.939+0000 7f58e6165640 1 Processor -- start 2023-10-30T14:20:20.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.939+0000 7f58e6165640 1 -- start start 2023-10-30T14:20:20.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.943+0000 7f58e6165640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f58e0151840 0x7f58e00c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:20.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.943+0000 7f58e6165640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f58e0151c20 0x7f58e0072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:20.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.943+0000 7f58e6165640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e00c5030 con 0x7f58e00cb8b0 2023-10-30T14:20:20.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.943+0000 7f58e6165640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58e0157830 con 0x7f58e0151840 2023-10-30T14:20:20.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.943+0000 7f58e6165640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58e0157b10 con 0x7f58e0151c20 2023-10-30T14:20:20.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.943+0000 7f58df7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f58e0151840 0x7f58e00c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:20.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.943+0000 7f58df7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f58e0151840 0x7f58e00c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44964/0 (socket says 172.21.15.57:44964) 2023-10-30T14:20:20.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.943+0000 7f58df7fe640 1 -- 172.21.15.57:0/1371371353 learned_addr learned my addr 172.21.15.57:0/1371371353 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:20.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.943+0000 7f58deffd640 1 --2- 172.21.15.57:0/1371371353 >> v2:172.21.15.88:3300/0 conn(0x7f58e0151c20 0x7f58e0072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:20.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.943+0000 7f58df7fe640 1 -- 172.21.15.57:0/1371371353 >> v2:172.21.15.88:3300/0 conn(0x7f58e0151c20 msgr2=0x7f58e0072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:20.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.943+0000 7f58df7fe640 1 --2- 172.21.15.57:0/1371371353 >> v2:172.21.15.88:3300/0 conn(0x7f58e0151c20 0x7f58e0072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:20.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.943+0000 7f58df7fe640 1 -- 172.21.15.57:0/1371371353 >> v1:172.21.15.88:6789/0 conn(0x7f58e00cb8b0 legacy=0x7f58e006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:20.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.943+0000 7f58df7fe640 1 -- 172.21.15.57:0/1371371353 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e0157df0 con 0x7f58e0151840 2023-10-30T14:20:20.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.947+0000 7f58df7fe640 1 --2- 172.21.15.57:0/1371371353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e0151840 0x7f58e00c49a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f58e006d4d0 tx=0x7f58d001ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e164c4c4b7d527a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:20.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.947+0000 7f58de7fc640 1 -- 172.21.15.57:0/1371371353 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d00a89b0 con 0x7f58e0151840 2023-10-30T14:20:20.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.947+0000 7f58de7fc640 1 -- 172.21.15.57:0/1371371353 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d00a8b10 con 0x7f58e0151840 2023-10-30T14:20:20.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.947+0000 7f58de7fc640 1 -- 172.21.15.57:0/1371371353 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d00a8ea0 con 0x7f58e0151840 2023-10-30T14:20:20.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.947+0000 7f58e6165640 1 -- 172.21.15.57:0/1371371353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e0151840 msgr2=0x7f58e00c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:20.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.947+0000 7f58e6165640 1 --2- 172.21.15.57:0/1371371353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e0151840 0x7f58e00c49a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f58e006d4d0 tx=0x7f58d001ecb0 comp rx=0 tx=0).stop 2023-10-30T14:20:20.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.947+0000 7f58e6165640 1 -- 172.21.15.57:0/1371371353 shutdown_connections 2023-10-30T14:20:20.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.947+0000 7f58e6165640 1 --2- 172.21.15.57:0/1371371353 >> v2:172.21.15.88:3300/0 conn(0x7f58e0151c20 0x7f58e0072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:20.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.947+0000 7f58e6165640 1 --2- 172.21.15.57:0/1371371353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e0151840 0x7f58e00c49a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:20.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.947+0000 7f58e6165640 1 -- 172.21.15.57:0/1371371353 >> 172.21.15.57:0/1371371353 conn(0x7f58e00cac70 msgr2=0x7f58e00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:20.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.947+0000 7f58e6165640 1 -- 172.21.15.57:0/1371371353 shutdown_connections 2023-10-30T14:20:20.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.947+0000 7f58e6165640 1 -- 172.21.15.57:0/1371371353 wait complete. 2023-10-30T14:20:20.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.947+0000 7f58e6165640 1 Processor -- start 2023-10-30T14:20:20.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.947+0000 7f58e6165640 1 -- start start 2023-10-30T14:20:20.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58e6165640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e00cb8b0 0x7f58e00c9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:20.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58e6165640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58e0151840 0x7f58e01538e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:20.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58e6165640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58e0151c20 0x7f58e0143370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:20.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58e6165640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e0153120 con 0x7f58e00cb8b0 2023-10-30T14:20:20.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58e6165640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e01451d0 con 0x7f58e0151c20 2023-10-30T14:20:20.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58e6165640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e01f0ad0 con 0x7f58e0151840 2023-10-30T14:20:20.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58df7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e00cb8b0 0x7f58e00c9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:20.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58df7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e00cb8b0 0x7f58e00c9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44972/0 (socket says 172.21.15.57:44972) 2023-10-30T14:20:20.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58df7fe640 1 -- 172.21.15.57:0/1548217828 learned_addr learned my addr 172.21.15.57:0/1548217828 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:20.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58dffff640 1 --2- 172.21.15.57:0/1548217828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58e0151c20 0x7f58e0143370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:20.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58deffd640 1 --2- 172.21.15.57:0/1548217828 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58e0151840 0x7f58e01538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:20.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58df7fe640 1 -- 172.21.15.57:0/1548217828 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58e0151840 msgr2=0x7f58e01538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:20.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58df7fe640 1 --2- 172.21.15.57:0/1548217828 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58e0151840 0x7f58e01538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:20.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58df7fe640 1 -- 172.21.15.57:0/1548217828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58e0151c20 msgr2=0x7f58e0143370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:20.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58df7fe640 1 --2- 172.21.15.57:0/1548217828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58e0151c20 0x7f58e0143370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:20.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58df7fe640 1 -- 172.21.15.57:0/1548217828 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d00094d0 con 0x7f58e00cb8b0 2023-10-30T14:20:20.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58dffff640 1 --2- 172.21.15.57:0/1548217828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58e0151c20 0x7f58e0143370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:20:20.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58df7fe640 1 --2- 172.21.15.57:0/1548217828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e00cb8b0 0x7f58e00c9da0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f58d00b3040 tx=0x7f58d001ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:20.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58c7fff640 1 -- 172.21.15.57:0/1548217828 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d00b0e80 con 0x7f58e00cb8b0 2023-10-30T14:20:20.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58c7fff640 1 -- 172.21.15.57:0/1548217828 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d00bb480 con 0x7f58e00cb8b0 2023-10-30T14:20:20.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58c7fff640 1 -- 172.21.15.57:0/1548217828 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d00bb810 con 0x7f58e00cb8b0 2023-10-30T14:20:20.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58e6165640 1 -- 172.21.15.57:0/1548217828 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e01547e0 con 0x7f58e00cb8b0 2023-10-30T14:20:20.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58e6165640 1 -- 172.21.15.57:0/1548217828 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e01533c0 con 0x7f58e00cb8b0 2023-10-30T14:20:20.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58e6165640 1 -- 172.21.15.57:0/1548217828 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e01535b0 con 0x7f58e00cb8b0 2023-10-30T14:20:20.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.951+0000 7f58c7fff640 1 -- 172.21.15.57:0/1548217828 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f58d00bb970 con 0x7f58e00cb8b0 2023-10-30T14:20:20.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.955+0000 7f58c7fff640 1 --2- 172.21.15.57:0/1548217828 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f58cc0664a0 0x7f58cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:20.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.955+0000 7f58c7fff640 1 -- 172.21.15.57:0/1548217828 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6613+0+0 (secure 0 0 0) 0x7f58d01229d0 con 0x7f58e00cb8b0 2023-10-30T14:20:20.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.955+0000 7f58deffd640 1 --2- 172.21.15.57:0/1548217828 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f58cc0664a0 0x7f58cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:20.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.955+0000 7f58deffd640 1 --2- 172.21.15.57:0/1548217828 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f58cc0664a0 0x7f58cc068960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f58e0148ac0 tx=0x7f58d4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:20.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:20.959+0000 7f58c7fff640 1 -- 172.21.15.57:0/1548217828 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f58d00ef990 con 0x7f58e00cb8b0 2023-10-30T14:20:21.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:21.171+0000 7f58e6165640 1 -- 172.21.15.57:0/1548217828 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58e01535b0 con 0x7f58e00cb8b0 2023-10-30T14:20:21.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:21.171+0000 7f58c7fff640 1 -- 172.21.15.57:0/1548217828 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6533 (secure 0 0 0) 0x7f58d00b5020 con 0x7f58e00cb8b0 2023-10-30T14:20:21.176 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:21.176 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5078,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:15.987759+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:21.176 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 47 2023-10-30T14:20:21.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:21.171+0000 7f58e6165640 1 -- 172.21.15.57:0/1548217828 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f58cc0664a0 msgr2=0x7f58cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:21.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:21.171+0000 7f58e6165640 1 --2- 172.21.15.57:0/1548217828 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f58cc0664a0 0x7f58cc068960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f58e0148ac0 tx=0x7f58d4008040 comp rx=0 tx=0).stop 2023-10-30T14:20:21.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:21.171+0000 7f58e6165640 1 -- 172.21.15.57:0/1548217828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e00cb8b0 msgr2=0x7f58e00c9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:21.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:21.171+0000 7f58e6165640 1 --2- 172.21.15.57:0/1548217828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e00cb8b0 0x7f58e00c9da0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f58d00b3040 tx=0x7f58d001ba50 comp rx=0 tx=0).stop 2023-10-30T14:20:21.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:21.171+0000 7f58e6165640 1 -- 172.21.15.57:0/1548217828 shutdown_connections 2023-10-30T14:20:21.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:21.171+0000 7f58e6165640 1 --2- 172.21.15.57:0/1548217828 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f58cc0664a0 0x7f58cc068960 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:21.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:21.171+0000 7f58e6165640 1 --2- 172.21.15.57:0/1548217828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58e0151c20 0x7f58e0143370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:21.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:21.171+0000 7f58e6165640 1 --2- 172.21.15.57:0/1548217828 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58e0151840 0x7f58e01538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:21.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:21.171+0000 7f58e6165640 1 --2- 172.21.15.57:0/1548217828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e00cb8b0 0x7f58e00c9da0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:21.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:21.171+0000 7f58e6165640 1 -- 172.21.15.57:0/1548217828 >> 172.21.15.57:0/1548217828 conn(0x7f58e00cac70 msgr2=0x7f58e00c6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:21.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:21.171+0000 7f58e6165640 1 -- 172.21.15.57:0/1548217828 shutdown_connections 2023-10-30T14:20:21.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:21.171+0000 7f58e6165640 1 -- 172.21.15.57:0/1548217828 wait complete. 2023-10-30T14:20:21.197 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:20:23.198 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:23.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.347+0000 7fb322a17640 1 Processor -- start 2023-10-30T14:20:23.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.347+0000 7fb322a17640 1 -- start start 2023-10-30T14:20:23.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.351+0000 7fb322a17640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb31c153aa0 0x7fb31c15d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:23.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.351+0000 7fb322a17640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb31c0701e0 0x7fb31c0ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:23.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.351+0000 7fb322a17640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31c0c7b00 con 0x7fb31c0cd420 2023-10-30T14:20:23.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.351+0000 7fb322a17640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb31c070a50 con 0x7fb31c0701e0 2023-10-30T14:20:23.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.351+0000 7fb322a17640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb31c15dd80 con 0x7fb31c153aa0 2023-10-30T14:20:23.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.351+0000 7fb3137fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb31c0701e0 0x7fb31c0ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:23.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.351+0000 7fb3137fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb31c0701e0 0x7fb31c0ccee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44984/0 (socket says 172.21.15.57:44984) 2023-10-30T14:20:23.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.351+0000 7fb3137fe640 1 -- 172.21.15.57:0/2377032377 learned_addr learned my addr 172.21.15.57:0/2377032377 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:23.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.351+0000 7fb31bfff640 1 --2- 172.21.15.57:0/2377032377 >> v2:172.21.15.88:3300/0 conn(0x7fb31c153aa0 0x7fb31c15d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:23.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.351+0000 7fb3137fe640 1 -- 172.21.15.57:0/2377032377 >> v2:172.21.15.88:3300/0 conn(0x7fb31c153aa0 msgr2=0x7fb31c15d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:23.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.351+0000 7fb3137fe640 1 --2- 172.21.15.57:0/2377032377 >> v2:172.21.15.88:3300/0 conn(0x7fb31c153aa0 0x7fb31c15d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:23.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.351+0000 7fb3137fe640 1 -- 172.21.15.57:0/2377032377 >> v1:172.21.15.88:6789/0 conn(0x7fb31c0cd420 legacy=0x7fb31c06b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:23.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.351+0000 7fb3137fe640 1 -- 172.21.15.57:0/2377032377 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c0c2050 con 0x7fb31c0701e0 2023-10-30T14:20:23.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.351+0000 7fb3137fe640 1 --2- 172.21.15.57:0/2377032377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb31c0701e0 0x7fb31c0ccee0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb31c0caae0 tx=0x7fb30401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee97595e6a7ef7b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:23.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.351+0000 7fb31b7fe640 1 -- 172.21.15.57:0/2377032377 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3040a89f0 con 0x7fb31c0701e0 2023-10-30T14:20:23.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb31b7fe640 1 -- 172.21.15.57:0/2377032377 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3040a8b50 con 0x7fb31c0701e0 2023-10-30T14:20:23.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb31b7fe640 1 -- 172.21.15.57:0/2377032377 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3040b0440 con 0x7fb31c0701e0 2023-10-30T14:20:23.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb322a17640 1 -- 172.21.15.57:0/2377032377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb31c0701e0 msgr2=0x7fb31c0ccee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:23.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb322a17640 1 --2- 172.21.15.57:0/2377032377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb31c0701e0 0x7fb31c0ccee0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb31c0caae0 tx=0x7fb30401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:20:23.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb322a17640 1 -- 172.21.15.57:0/2377032377 shutdown_connections 2023-10-30T14:20:23.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb322a17640 1 --2- 172.21.15.57:0/2377032377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb31c0701e0 0x7fb31c0ccee0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:23.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb322a17640 1 --2- 172.21.15.57:0/2377032377 >> v2:172.21.15.88:3300/0 conn(0x7fb31c153aa0 0x7fb31c15d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:23.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb322a17640 1 -- 172.21.15.57:0/2377032377 >> 172.21.15.57:0/2377032377 conn(0x7fb31c071fb0 msgr2=0x7fb31c0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:23.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb322a17640 1 -- 172.21.15.57:0/2377032377 shutdown_connections 2023-10-30T14:20:23.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb322a17640 1 -- 172.21.15.57:0/2377032377 wait complete. 2023-10-30T14:20:23.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb322a17640 1 Processor -- start 2023-10-30T14:20:23.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb322a17640 1 -- start start 2023-10-30T14:20:23.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb322a17640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb31c0701e0 0x7fb31c0c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:23.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb322a17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb31c0cd420 0x7fb31c0c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:23.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb322a17640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb31c153aa0 0x7fb31c158a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:23.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb322a17640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c157850 con 0x7fb31c153aa0 2023-10-30T14:20:23.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb322a17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c157b10 con 0x7fb31c0cd420 2023-10-30T14:20:23.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb322a17640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c0c65f0 con 0x7fb31c0701e0 2023-10-30T14:20:23.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb320fb4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb31c153aa0 0x7fb31c158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:23.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb320fb4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb31c153aa0 0x7fb31c158a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44992/0 (socket says 172.21.15.57:44992) 2023-10-30T14:20:23.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb320fb4640 1 -- 172.21.15.57:0/3917643089 learned_addr learned my addr 172.21.15.57:0/3917643089 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:23.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.355+0000 7fb31bfff640 1 --2- 172.21.15.57:0/3917643089 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb31c0701e0 0x7fb31c0c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:23.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.359+0000 7fb3137fe640 1 --2- 172.21.15.57:0/3917643089 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb31c0cd420 0x7fb31c0c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:23.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.359+0000 7fb320fb4640 1 -- 172.21.15.57:0/3917643089 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb31c0701e0 msgr2=0x7fb31c0c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:23.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.359+0000 7fb320fb4640 1 --2- 172.21.15.57:0/3917643089 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb31c0701e0 0x7fb31c0c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:23.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.359+0000 7fb320fb4640 1 -- 172.21.15.57:0/3917643089 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb31c0cd420 msgr2=0x7fb31c0c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:23.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.359+0000 7fb320fb4640 1 --2- 172.21.15.57:0/3917643089 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb31c0cd420 0x7fb31c0c5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:23.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.359+0000 7fb320fb4640 1 -- 172.21.15.57:0/3917643089 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3040094d0 con 0x7fb31c153aa0 2023-10-30T14:20:23.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.359+0000 7fb320fb4640 1 --2- 172.21.15.57:0/3917643089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb31c153aa0 0x7fb31c158a50 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb31400c770 tx=0x7fb31400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:23.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.359+0000 7fb3197fa640 1 -- 172.21.15.57:0/3917643089 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb314003b80 con 0x7fb31c153aa0 2023-10-30T14:20:23.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.359+0000 7fb3197fa640 1 -- 172.21.15.57:0/3917643089 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb314003ce0 con 0x7fb31c153aa0 2023-10-30T14:20:23.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.359+0000 7fb322a17640 1 -- 172.21.15.57:0/3917643089 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31c0c68d0 con 0x7fb31c153aa0 2023-10-30T14:20:23.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.359+0000 7fb3197fa640 1 -- 172.21.15.57:0/3917643089 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31400aa90 con 0x7fb31c153aa0 2023-10-30T14:20:23.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.359+0000 7fb322a17640 1 -- 172.21.15.57:0/3917643089 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb31c0c5460 con 0x7fb31c153aa0 2023-10-30T14:20:23.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.359+0000 7fb322a17640 1 -- 172.21.15.57:0/3917643089 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31c0c6df0 con 0x7fb31c153aa0 2023-10-30T14:20:23.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.359+0000 7fb3197fa640 1 -- 172.21.15.57:0/3917643089 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb31400abf0 con 0x7fb31c153aa0 2023-10-30T14:20:23.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.359+0000 7fb3197fa640 1 --2- 172.21.15.57:0/3917643089 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb2fc0664a0 0x7fb2fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:23.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.359+0000 7fb3197fa640 1 -- 172.21.15.57:0/3917643089 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6613+0+0 (secure 0 0 0) 0x7fb314083fc0 con 0x7fb31c153aa0 2023-10-30T14:20:23.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.363+0000 7fb31bfff640 1 --2- 172.21.15.57:0/3917643089 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb2fc0664a0 0x7fb2fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:23.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.363+0000 7fb31bfff640 1 --2- 172.21.15.57:0/3917643089 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb2fc0664a0 0x7fb2fc068960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0079c0 tx=0x7fb30c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:23.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.363+0000 7fb3197fa640 1 -- 172.21.15.57:0/3917643089 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb31c0c6df0 con 0x7fb31c153aa0 2023-10-30T14:20:23.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.559+0000 7fb322a17640 1 -- 172.21.15.57:0/3917643089 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb31c0c6df0 con 0x7fb31c153aa0 2023-10-30T14:20:23.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.559+0000 7fb3197fa640 1 -- 172.21.15.57:0/3917643089 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6533 (secure 0 0 0) 0x7fb31c0c6df0 con 0x7fb31c153aa0 2023-10-30T14:20:23.566 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:23.566 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5078,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:15.987759+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:23.567 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 47 2023-10-30T14:20:23.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.563+0000 7fb322a17640 1 -- 172.21.15.57:0/3917643089 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb2fc0664a0 msgr2=0x7fb2fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:23.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.563+0000 7fb322a17640 1 --2- 172.21.15.57:0/3917643089 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb2fc0664a0 0x7fb2fc068960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0079c0 tx=0x7fb30c008040 comp rx=0 tx=0).stop 2023-10-30T14:20:23.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.563+0000 7fb322a17640 1 -- 172.21.15.57:0/3917643089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb31c153aa0 msgr2=0x7fb31c158a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:23.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.563+0000 7fb322a17640 1 --2- 172.21.15.57:0/3917643089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb31c153aa0 0x7fb31c158a50 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb31400c770 tx=0x7fb31400cc30 comp rx=0 tx=0).stop 2023-10-30T14:20:23.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.563+0000 7fb322a17640 1 -- 172.21.15.57:0/3917643089 shutdown_connections 2023-10-30T14:20:23.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.563+0000 7fb322a17640 1 --2- 172.21.15.57:0/3917643089 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb2fc0664a0 0x7fb2fc068960 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:23.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.563+0000 7fb322a17640 1 --2- 172.21.15.57:0/3917643089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb31c153aa0 0x7fb31c158a50 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:23.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.563+0000 7fb322a17640 1 --2- 172.21.15.57:0/3917643089 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb31c0cd420 0x7fb31c0c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:23.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.563+0000 7fb322a17640 1 --2- 172.21.15.57:0/3917643089 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb31c0701e0 0x7fb31c0c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:23.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.563+0000 7fb322a17640 1 -- 172.21.15.57:0/3917643089 >> 172.21.15.57:0/3917643089 conn(0x7fb31c071fb0 msgr2=0x7fb31c1523a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:23.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.563+0000 7fb322a17640 1 -- 172.21.15.57:0/3917643089 shutdown_connections 2023-10-30T14:20:23.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:23.563+0000 7fb322a17640 1 -- 172.21.15.57:0/3917643089 wait complete. 2023-10-30T14:20:23.587 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:20:25.588 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:25.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.739+0000 7ff050cbb640 1 Processor -- start 2023-10-30T14:20:25.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.739+0000 7ff050cbb640 1 -- start start 2023-10-30T14:20:25.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.739+0000 7ff050cbb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff04c153b50 0x7ff04c1597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:25.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.739+0000 7ff050cbb640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff04c0cb180 0x7ff04c06ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:25.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.739+0000 7ff050cbb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04c15a770 con 0x7ff04c0c62b0 2023-10-30T14:20:25.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.739+0000 7ff050cbb640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff04c159cf0 con 0x7ff04c153b50 2023-10-30T14:20:25.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.739+0000 7ff050cbb640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c06aa30 con 0x7ff04c0cb180 2023-10-30T14:20:25.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.739+0000 7ff04b7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff04c153b50 0x7ff04c1597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:25.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.739+0000 7ff04bfff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff04c0cb180 0x7ff04c06ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:25.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.739+0000 7ff04affd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff04c0c62b0 0x7ff04c0ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:48016/0 (socket says 172.21.15.57:48016) 2023-10-30T14:20:25.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.739+0000 7ff04b7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff04c153b50 0x7ff04c1597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:55176/0 (socket says 172.21.15.57:55176) 2023-10-30T14:20:25.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.739+0000 7ff04affd640 1 -- 172.21.15.57:0/1636311295 learned_addr learned my addr 172.21.15.57:0/1636311295 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:20:25.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.739+0000 7ff04b7fe640 1 -- 172.21.15.57:0/1636311295 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff04c0cb180 msgr2=0x7ff04c06ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:25.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.739+0000 7ff04b7fe640 1 --2- 172.21.15.57:0/1636311295 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff04c0cb180 0x7ff04c06ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:25.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.739+0000 7ff04b7fe640 1 -- 172.21.15.57:0/1636311295 >> v1:172.21.15.88:6789/0 conn(0x7ff04c0c62b0 legacy=0x7ff04c0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:25.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.739+0000 7ff04b7fe640 1 -- 172.21.15.57:0/1636311295 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04c0c9d40 con 0x7ff04c153b50 2023-10-30T14:20:25.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.739+0000 7ff04a7fc640 1 -- 172.21.15.57:0/1636311295 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267999017 0 0) 0x7ff040001120 con 0x7ff04c0c62b0 2023-10-30T14:20:25.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.739+0000 7ff04bfff640 1 --2- 172.21.15.57:0/1636311295 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff04c0cb180 0x7ff04c06ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:20:25.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.743+0000 7ff04b7fe640 1 --2- 172.21.15.57:0/1636311295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff04c153b50 0x7ff04c1597b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0097c0 tx=0x7ff03c01ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3279607cb3adf65 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:25.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.743+0000 7ff04a7fc640 1 -- 172.21.15.57:0/1636311295 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0a8970 con 0x7ff04c153b50 2023-10-30T14:20:25.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.743+0000 7ff04a7fc640 1 -- 172.21.15.57:0/1636311295 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03c0a8ad0 con 0x7ff04c153b50 2023-10-30T14:20:25.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.743+0000 7ff04a7fc640 1 -- 172.21.15.57:0/1636311295 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0a8e60 con 0x7ff04c153b50 2023-10-30T14:20:25.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.743+0000 7ff050cbb640 1 -- 172.21.15.57:0/1636311295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff04c153b50 msgr2=0x7ff04c1597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:25.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.743+0000 7ff050cbb640 1 --2- 172.21.15.57:0/1636311295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff04c153b50 0x7ff04c1597b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0097c0 tx=0x7ff03c01ec70 comp rx=0 tx=0).stop 2023-10-30T14:20:25.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.743+0000 7ff050cbb640 1 -- 172.21.15.57:0/1636311295 shutdown_connections 2023-10-30T14:20:25.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.743+0000 7ff050cbb640 1 --2- 172.21.15.57:0/1636311295 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff04c0cb180 0x7ff04c06ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:25.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.743+0000 7ff050cbb640 1 --2- 172.21.15.57:0/1636311295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff04c153b50 0x7ff04c1597b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:25.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.743+0000 7ff050cbb640 1 -- 172.21.15.57:0/1636311295 >> 172.21.15.57:0/1636311295 conn(0x7ff04c065a80 msgr2=0x7ff04c065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:25.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.743+0000 7ff050cbb640 1 -- 172.21.15.57:0/1636311295 shutdown_connections 2023-10-30T14:20:25.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.743+0000 7ff050cbb640 1 -- 172.21.15.57:0/1636311295 wait complete. 2023-10-30T14:20:25.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff050cbb640 1 Processor -- start 2023-10-30T14:20:25.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff050cbb640 1 -- start start 2023-10-30T14:20:25.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff050cbb640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff04c0c62b0 0x7ff04c1f2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:25.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff050cbb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff04c0cb180 0x7ff04c06add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:25.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff050cbb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff04c153b50 0x7ff04c14a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:25.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff050cbb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c1f3200 con 0x7ff04c153b50 2023-10-30T14:20:25.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff050cbb640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c0c31e0 con 0x7ff04c0cb180 2023-10-30T14:20:25.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff050cbb640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c0c34c0 con 0x7ff04c0c62b0 2023-10-30T14:20:25.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff04bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff04c153b50 0x7ff04c14a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:25.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff04bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff04c153b50 0x7ff04c14a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57230/0 (socket says 172.21.15.57:57230) 2023-10-30T14:20:25.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff04bfff640 1 -- 172.21.15.57:0/6509238 learned_addr learned my addr 172.21.15.57:0/6509238 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:25.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff04b7fe640 1 --2- 172.21.15.57:0/6509238 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff04c0c62b0 0x7ff04c1f2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:25.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff04affd640 1 --2- 172.21.15.57:0/6509238 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff04c0cb180 0x7ff04c06add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:25.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff04bfff640 1 -- 172.21.15.57:0/6509238 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff04c0c62b0 msgr2=0x7ff04c1f2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:25.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff04bfff640 1 --2- 172.21.15.57:0/6509238 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff04c0c62b0 0x7ff04c1f2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:25.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff04bfff640 1 -- 172.21.15.57:0/6509238 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff04c0cb180 msgr2=0x7ff04c06add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:25.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff04bfff640 1 --2- 172.21.15.57:0/6509238 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff04c0cb180 0x7ff04c06add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:25.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff04bfff640 1 -- 172.21.15.57:0/6509238 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff044009a70 con 0x7ff04c153b50 2023-10-30T14:20:25.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff04affd640 1 --2- 172.21.15.57:0/6509238 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff04c0cb180 0x7ff04c06add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:25.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff04bfff640 1 --2- 172.21.15.57:0/6509238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff04c153b50 0x7ff04c14a230 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff04400efc0 tx=0x7ff044009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:25.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff033fff640 1 -- 172.21.15.57:0/6509238 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff044009d70 con 0x7ff04c153b50 2023-10-30T14:20:25.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff033fff640 1 -- 172.21.15.57:0/6509238 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff04400f040 con 0x7ff04c153b50 2023-10-30T14:20:25.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff050cbb640 1 -- 172.21.15.57:0/6509238 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff03c009420 con 0x7ff04c153b50 2023-10-30T14:20:25.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff033fff640 1 -- 172.21.15.57:0/6509238 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0440039a0 con 0x7ff04c153b50 2023-10-30T14:20:25.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.747+0000 7ff050cbb640 1 -- 172.21.15.57:0/6509238 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04c1f24c0 con 0x7ff04c153b50 2023-10-30T14:20:25.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.751+0000 7ff050cbb640 1 -- 172.21.15.57:0/6509238 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff04c1f2320 con 0x7ff04c153b50 2023-10-30T14:20:25.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.751+0000 7ff033fff640 1 -- 172.21.15.57:0/6509238 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff0440144a0 con 0x7ff04c153b50 2023-10-30T14:20:25.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.751+0000 7ff033fff640 1 --2- 172.21.15.57:0/6509238 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0380664a0 0x7ff038068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:25.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.751+0000 7ff033fff640 1 -- 172.21.15.57:0/6509238 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6613+0+0 (secure 0 0 0) 0x7ff044084f40 con 0x7ff04c153b50 2023-10-30T14:20:25.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.751+0000 7ff04b7fe640 1 --2- 172.21.15.57:0/6509238 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0380664a0 0x7ff038068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:25.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.755+0000 7ff04b7fe640 1 --2- 172.21.15.57:0/6509238 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0380664a0 0x7ff038068960 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0b2000 tx=0x7ff03c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:25.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.755+0000 7ff033fff640 1 -- 172.21.15.57:0/6509238 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff04c1f2320 con 0x7ff04c153b50 2023-10-30T14:20:25.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.947+0000 7ff050cbb640 1 -- 172.21.15.57:0/6509238 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff04c1f2320 con 0x7ff04c153b50 2023-10-30T14:20:25.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.947+0000 7ff033fff640 1 -- 172.21.15.57:0/6509238 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6533 (secure 0 0 0) 0x7ff04c1f2320 con 0x7ff04c153b50 2023-10-30T14:20:25.953 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:25.954 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5078,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:15.987759+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:25.954 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 47 2023-10-30T14:20:25.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.951+0000 7ff050cbb640 1 -- 172.21.15.57:0/6509238 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0380664a0 msgr2=0x7ff038068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:25.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.951+0000 7ff050cbb640 1 --2- 172.21.15.57:0/6509238 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0380664a0 0x7ff038068960 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0b2000 tx=0x7ff03c008040 comp rx=0 tx=0).stop 2023-10-30T14:20:25.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.951+0000 7ff050cbb640 1 -- 172.21.15.57:0/6509238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff04c153b50 msgr2=0x7ff04c14a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:25.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.951+0000 7ff050cbb640 1 --2- 172.21.15.57:0/6509238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff04c153b50 0x7ff04c14a230 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff04400efc0 tx=0x7ff044009f90 comp rx=0 tx=0).stop 2023-10-30T14:20:25.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.951+0000 7ff050cbb640 1 -- 172.21.15.57:0/6509238 shutdown_connections 2023-10-30T14:20:25.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.951+0000 7ff050cbb640 1 --2- 172.21.15.57:0/6509238 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0380664a0 0x7ff038068960 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:25.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.951+0000 7ff050cbb640 1 --2- 172.21.15.57:0/6509238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff04c153b50 0x7ff04c14a230 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:25.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.951+0000 7ff050cbb640 1 --2- 172.21.15.57:0/6509238 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff04c0cb180 0x7ff04c06add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:25.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.951+0000 7ff050cbb640 1 --2- 172.21.15.57:0/6509238 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff04c0c62b0 0x7ff04c1f2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:25.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.951+0000 7ff050cbb640 1 -- 172.21.15.57:0/6509238 >> 172.21.15.57:0/6509238 conn(0x7ff04c065a80 msgr2=0x7ff04c14e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:25.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.951+0000 7ff050cbb640 1 -- 172.21.15.57:0/6509238 shutdown_connections 2023-10-30T14:20:25.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:25.951+0000 7ff050cbb640 1 -- 172.21.15.57:0/6509238 wait complete. 2023-10-30T14:20:25.975 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:20:27.976 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:28.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.123+0000 7fc8d5cc9640 1 Processor -- start 2023-10-30T14:20:28.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.123+0000 7fc8d5cc9640 1 -- start start 2023-10-30T14:20:28.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.123+0000 7fc8d5cc9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc8d0153c80 0x7fc8d015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:28.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.123+0000 7fc8d5cc9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d0159f20 con 0x7fc8d00c42a0 2023-10-30T14:20:28.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.123+0000 7fc8d5cc9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d015d620 con 0x7fc8d00c4a10 2023-10-30T14:20:28.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.123+0000 7fc8d5cc9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8d015dec0 con 0x7fc8d0153c80 2023-10-30T14:20:28.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.123+0000 7fc8cf7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc8d0153c80 0x7fc8d015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:28.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.123+0000 7fc8cf7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc8d0153c80 0x7fc8d015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57242/0 (socket says 172.21.15.57:57242) 2023-10-30T14:20:28.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.123+0000 7fc8cf7fe640 1 -- 172.21.15.57:0/3285839255 learned_addr learned my addr 172.21.15.57:0/3285839255 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:28.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.123+0000 7fc8cffff640 1 --1- 172.21.15.57:0/3285839255 >> v1:172.21.15.57:6789/0 conn(0x7fc8d00c42a0 0x7fc8d00ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50324/0 (socket says 172.21.15.57:50324) 2023-10-30T14:20:28.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.123+0000 7fc8cf7fe640 1 -- 172.21.15.57:0/3285839255 >> v1:172.21.15.88:6789/0 conn(0x7fc8d00c4a10 legacy=0x7fc8d015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:28.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.123+0000 7fc8cf7fe640 1 -- 172.21.15.57:0/3285839255 >> v1:172.21.15.57:6789/0 conn(0x7fc8d00c42a0 legacy=0x7fc8d00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:28.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.123+0000 7fc8cf7fe640 1 -- 172.21.15.57:0/3285839255 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d015d900 con 0x7fc8d0153c80 2023-10-30T14:20:28.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.127+0000 7fc8cf7fe640 1 --2- 172.21.15.57:0/3285839255 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8d0153c80 0x7fc8d015cf90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0009870 tx=0x7fc8c001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd123640f909c0eb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:28.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.127+0000 7fc8ce7fc640 1 -- 172.21.15.57:0/3285839255 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c00a8a30 con 0x7fc8d0153c80 2023-10-30T14:20:28.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.127+0000 7fc8ce7fc640 1 -- 172.21.15.57:0/3285839255 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c00a8b90 con 0x7fc8d0153c80 2023-10-30T14:20:28.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.127+0000 7fc8ce7fc640 1 -- 172.21.15.57:0/3285839255 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c00b0440 con 0x7fc8d0153c80 2023-10-30T14:20:28.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.127+0000 7fc8d5cc9640 1 -- 172.21.15.57:0/3285839255 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8d0153c80 msgr2=0x7fc8d015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:28.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.127+0000 7fc8d5cc9640 1 --2- 172.21.15.57:0/3285839255 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8d0153c80 0x7fc8d015cf90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0009870 tx=0x7fc8c001ed30 comp rx=0 tx=0).stop 2023-10-30T14:20:28.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.127+0000 7fc8d5cc9640 1 -- 172.21.15.57:0/3285839255 shutdown_connections 2023-10-30T14:20:28.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.127+0000 7fc8d5cc9640 1 --2- 172.21.15.57:0/3285839255 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8d0153c80 0x7fc8d015cf90 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:28.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.127+0000 7fc8d5cc9640 1 -- 172.21.15.57:0/3285839255 >> 172.21.15.57:0/3285839255 conn(0x7fc8d00715c0 msgr2=0x7fc8d00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:28.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.127+0000 7fc8d5cc9640 1 -- 172.21.15.57:0/3285839255 shutdown_connections 2023-10-30T14:20:28.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.127+0000 7fc8d5cc9640 1 -- 172.21.15.57:0/3285839255 wait complete. 2023-10-30T14:20:28.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8d5cc9640 1 Processor -- start 2023-10-30T14:20:28.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8d5cc9640 1 -- start start 2023-10-30T14:20:28.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8d5cc9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8d00c42a0 0x7fc8d0155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:28.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8d5cc9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8d00c4a10 0x7fc8d00c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:28.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8d5cc9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8d0153c80 0x7fc8d0158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:28.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8d5cc9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d0155190 con 0x7fc8d0153c80 2023-10-30T14:20:28.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8d5cc9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d01f2d80 con 0x7fc8d00c42a0 2023-10-30T14:20:28.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8d5cc9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d0154380 con 0x7fc8d00c4a10 2023-10-30T14:20:28.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8cffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8d0153c80 0x7fc8d0158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:28.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8cffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8d0153c80 0x7fc8d0158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57246/0 (socket says 172.21.15.57:57246) 2023-10-30T14:20:28.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8cffff640 1 -- 172.21.15.57:0/1334195327 learned_addr learned my addr 172.21.15.57:0/1334195327 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:28.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8ceffd640 1 --2- 172.21.15.57:0/1334195327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8d00c4a10 0x7fc8d00c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:28.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8cf7fe640 1 --2- 172.21.15.57:0/1334195327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8d00c42a0 0x7fc8d0155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:28.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8cffff640 1 -- 172.21.15.57:0/1334195327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8d00c4a10 msgr2=0x7fc8d00c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:28.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8cffff640 1 --2- 172.21.15.57:0/1334195327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8d00c4a10 0x7fc8d00c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:28.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8cffff640 1 -- 172.21.15.57:0/1334195327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8d00c42a0 msgr2=0x7fc8d0155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:28.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8cffff640 1 --2- 172.21.15.57:0/1334195327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8d00c42a0 0x7fc8d0155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:28.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8cffff640 1 -- 172.21.15.57:0/1334195327 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c00094d0 con 0x7fc8d0153c80 2023-10-30T14:20:28.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8ceffd640 1 --2- 172.21.15.57:0/1334195327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8d00c4a10 0x7fc8d00c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:28.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8cffff640 1 --2- 172.21.15.57:0/1334195327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8d0153c80 0x7fc8d0158860 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc8c800c7a0 tx=0x7fc8c800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:28.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8b7fff640 1 -- 172.21.15.57:0/1334195327 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c800a640 con 0x7fc8d0153c80 2023-10-30T14:20:28.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8b7fff640 1 -- 172.21.15.57:0/1334195327 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c800a7a0 con 0x7fc8d0153c80 2023-10-30T14:20:28.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8b7fff640 1 -- 172.21.15.57:0/1334195327 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c800ab10 con 0x7fc8d0153c80 2023-10-30T14:20:28.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8d5cc9640 1 -- 172.21.15.57:0/1334195327 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d0154860 con 0x7fc8d0153c80 2023-10-30T14:20:28.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.131+0000 7fc8d5cc9640 1 -- 172.21.15.57:0/1334195327 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d0154e20 con 0x7fc8d0153c80 2023-10-30T14:20:28.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.135+0000 7fc8d5cc9640 1 -- 172.21.15.57:0/1334195327 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d0155340 con 0x7fc8d0153c80 2023-10-30T14:20:28.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.135+0000 7fc8b7fff640 1 -- 172.21.15.57:0/1334195327 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc8c800ac70 con 0x7fc8d0153c80 2023-10-30T14:20:28.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.135+0000 7fc8b7fff640 1 --2- 172.21.15.57:0/1334195327 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8bc0664a0 0x7fc8bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:28.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.135+0000 7fc8b7fff640 1 -- 172.21.15.57:0/1334195327 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6613+0+0 (secure 0 0 0) 0x7fc8c80839e0 con 0x7fc8d0153c80 2023-10-30T14:20:28.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.135+0000 7fc8cf7fe640 1 --2- 172.21.15.57:0/1334195327 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8bc0664a0 0x7fc8bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:28.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.135+0000 7fc8cf7fe640 1 --2- 172.21.15.57:0/1334195327 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8bc0664a0 0x7fc8bc068960 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc8c00b2000 tx=0x7fc8c00aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:28.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.139+0000 7fc8b7fff640 1 -- 172.21.15.57:0/1334195327 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc8d0155340 con 0x7fc8d0153c80 2023-10-30T14:20:28.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.327+0000 7fc8d5cc9640 1 -- 172.21.15.57:0/1334195327 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8d0155340 con 0x7fc8d0153c80 2023-10-30T14:20:28.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.327+0000 7fc8b7fff640 1 -- 172.21.15.57:0/1334195327 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6533 (secure 0 0 0) 0x7fc8d0155340 con 0x7fc8d0153c80 2023-10-30T14:20:28.334 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:28.334 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5078,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:15.987759+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:28.335 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 47 2023-10-30T14:20:28.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.331+0000 7fc8d5cc9640 1 -- 172.21.15.57:0/1334195327 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8bc0664a0 msgr2=0x7fc8bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:28.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.331+0000 7fc8d5cc9640 1 --2- 172.21.15.57:0/1334195327 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8bc0664a0 0x7fc8bc068960 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc8c00b2000 tx=0x7fc8c00aa070 comp rx=0 tx=0).stop 2023-10-30T14:20:28.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.331+0000 7fc8d5cc9640 1 -- 172.21.15.57:0/1334195327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8d0153c80 msgr2=0x7fc8d0158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:28.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.331+0000 7fc8d5cc9640 1 --2- 172.21.15.57:0/1334195327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8d0153c80 0x7fc8d0158860 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc8c800c7a0 tx=0x7fc8c800cc60 comp rx=0 tx=0).stop 2023-10-30T14:20:28.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.331+0000 7fc8d5cc9640 1 -- 172.21.15.57:0/1334195327 shutdown_connections 2023-10-30T14:20:28.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.331+0000 7fc8d5cc9640 1 --2- 172.21.15.57:0/1334195327 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8bc0664a0 0x7fc8bc068960 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:28.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.331+0000 7fc8d5cc9640 1 --2- 172.21.15.57:0/1334195327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8d0153c80 0x7fc8d0158860 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:28.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.331+0000 7fc8d5cc9640 1 --2- 172.21.15.57:0/1334195327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8d00c4a10 0x7fc8d00c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:28.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.331+0000 7fc8d5cc9640 1 --2- 172.21.15.57:0/1334195327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8d00c42a0 0x7fc8d0155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:28.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.331+0000 7fc8d5cc9640 1 -- 172.21.15.57:0/1334195327 >> 172.21.15.57:0/1334195327 conn(0x7fc8d00715c0 msgr2=0x7fc8d00719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:28.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.331+0000 7fc8d5cc9640 1 -- 172.21.15.57:0/1334195327 shutdown_connections 2023-10-30T14:20:28.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:28.331+0000 7fc8d5cc9640 1 -- 172.21.15.57:0/1334195327 wait complete. 2023-10-30T14:20:28.355 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:20:30.356 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:30.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.507+0000 7fe411006640 1 Processor -- start 2023-10-30T14:20:30.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.507+0000 7fe411006640 1 -- start start 2023-10-30T14:20:30.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.507+0000 7fe411006640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe40c0c8eb0 0x7fe40c15cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:30.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.507+0000 7fe411006640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40c15dcc0 con 0x7fe40c15ac70 2023-10-30T14:20:30.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.507+0000 7fe411006640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40c159d80 con 0x7fe40c06aeb0 2023-10-30T14:20:30.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.507+0000 7fe411006640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe40c159800 con 0x7fe40c0c8eb0 2023-10-30T14:20:30.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.507+0000 7fe40bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe40c0c8eb0 0x7fe40c15cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:30.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.507+0000 7fe40bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe40c0c8eb0 0x7fe40c15cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57248/0 (socket says 172.21.15.57:57248) 2023-10-30T14:20:30.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.507+0000 7fe40bfff640 1 -- 172.21.15.57:0/3724018337 learned_addr learned my addr 172.21.15.57:0/3724018337 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:30.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.507+0000 7fe40b7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fe40c15ac70 0x7fe40c15a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50328/0 (socket says 172.21.15.57:50328) 2023-10-30T14:20:30.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.507+0000 7fe40bfff640 1 -- 172.21.15.57:0/3724018337 >> v1:172.21.15.88:6789/0 conn(0x7fe40c06aeb0 legacy=0x7fe40c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:30.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.507+0000 7fe40bfff640 1 -- 172.21.15.57:0/3724018337 >> v1:172.21.15.57:6789/0 conn(0x7fe40c15ac70 legacy=0x7fe40c15a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:30.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.507+0000 7fe40bfff640 1 -- 172.21.15.57:0/3724018337 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40c15d420 con 0x7fe40c0c8eb0 2023-10-30T14:20:30.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.507+0000 7fe40affd640 1 -- 172.21.15.57:0/3724018337 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947758013 0 0) 0x7fe400001120 con 0x7fe40c15ac70 2023-10-30T14:20:30.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.511+0000 7fe40bfff640 1 --2- 172.21.15.57:0/3724018337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe40c0c8eb0 0x7fe40c15cd70 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc009870 tx=0x7fe3fc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=517306e567c90ab server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:30.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.511+0000 7fe40affd640 1 -- 172.21.15.57:0/3724018337 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc0a89f0 con 0x7fe40c0c8eb0 2023-10-30T14:20:30.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.511+0000 7fe40affd640 1 -- 172.21.15.57:0/3724018337 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc0a8b50 con 0x7fe40c0c8eb0 2023-10-30T14:20:30.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.511+0000 7fe40affd640 1 -- 172.21.15.57:0/3724018337 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc0b0440 con 0x7fe40c0c8eb0 2023-10-30T14:20:30.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.511+0000 7fe411006640 1 -- 172.21.15.57:0/3724018337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe40c0c8eb0 msgr2=0x7fe40c15cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:30.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.511+0000 7fe411006640 1 --2- 172.21.15.57:0/3724018337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe40c0c8eb0 0x7fe40c15cd70 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc009870 tx=0x7fe3fc01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:20:30.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.511+0000 7fe411006640 1 -- 172.21.15.57:0/3724018337 shutdown_connections 2023-10-30T14:20:30.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.511+0000 7fe411006640 1 --2- 172.21.15.57:0/3724018337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe40c0c8eb0 0x7fe40c15cd70 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:30.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.511+0000 7fe411006640 1 -- 172.21.15.57:0/3724018337 >> 172.21.15.57:0/3724018337 conn(0x7fe40c0cef00 msgr2=0x7fe40c0cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:30.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.511+0000 7fe411006640 1 -- 172.21.15.57:0/3724018337 shutdown_connections 2023-10-30T14:20:30.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.511+0000 7fe411006640 1 -- 172.21.15.57:0/3724018337 wait complete. 2023-10-30T14:20:30.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.511+0000 7fe411006640 1 Processor -- start 2023-10-30T14:20:30.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe411006640 1 -- start start 2023-10-30T14:20:30.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe411006640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe40c06aeb0 0x7fe40c1f2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:30.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe411006640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe40c0c8eb0 0x7fe40c0c3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:30.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe411006640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe40c15ac70 0x7fe40c0c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:30.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe411006640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c0cecb0 con 0x7fe40c06aeb0 2023-10-30T14:20:30.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe411006640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c1f30b0 con 0x7fe40c0c8eb0 2023-10-30T14:20:30.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe411006640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c0c4220 con 0x7fe40c15ac70 2023-10-30T14:20:30.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe40bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe40c06aeb0 0x7fe40c1f2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:30.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe40bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe40c06aeb0 0x7fe40c1f2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57264/0 (socket says 172.21.15.57:57264) 2023-10-30T14:20:30.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe40bfff640 1 -- 172.21.15.57:0/2265631752 learned_addr learned my addr 172.21.15.57:0/2265631752 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:30.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe40b7fe640 1 --2- 172.21.15.57:0/2265631752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe40c0c8eb0 0x7fe40c0c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:30.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe410805640 1 --2- 172.21.15.57:0/2265631752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe40c15ac70 0x7fe40c0c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:30.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe40bfff640 1 -- 172.21.15.57:0/2265631752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe40c15ac70 msgr2=0x7fe40c0c3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:30.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe40bfff640 1 --2- 172.21.15.57:0/2265631752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe40c15ac70 0x7fe40c0c3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:30.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe40bfff640 1 -- 172.21.15.57:0/2265631752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe40c0c8eb0 msgr2=0x7fe40c0c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:30.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe40bfff640 1 --2- 172.21.15.57:0/2265631752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe40c0c8eb0 0x7fe40c0c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:30.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe40bfff640 1 -- 172.21.15.57:0/2265631752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3fc0094d0 con 0x7fe40c06aeb0 2023-10-30T14:20:30.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe40b7fe640 1 --2- 172.21.15.57:0/2265631752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe40c0c8eb0 0x7fe40c0c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:30.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe40bfff640 1 --2- 172.21.15.57:0/2265631752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe40c06aeb0 0x7fe40c1f2b70 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0b3040 tx=0x7fe3fc01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:30.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe408ff9640 1 -- 172.21.15.57:0/2265631752 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc0b0e40 con 0x7fe40c06aeb0 2023-10-30T14:20:30.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe408ff9640 1 -- 172.21.15.57:0/2265631752 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc0bb470 con 0x7fe40c06aeb0 2023-10-30T14:20:30.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe408ff9640 1 -- 172.21.15.57:0/2265631752 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc0bb820 con 0x7fe40c06aeb0 2023-10-30T14:20:30.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe411006640 1 -- 172.21.15.57:0/2265631752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe40c0ce6f0 con 0x7fe40c06aeb0 2023-10-30T14:20:30.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe411006640 1 -- 172.21.15.57:0/2265631752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40c0c47f0 con 0x7fe40c06aeb0 2023-10-30T14:20:30.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.515+0000 7fe411006640 1 -- 172.21.15.57:0/2265631752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe40c0c4630 con 0x7fe40c06aeb0 2023-10-30T14:20:30.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.519+0000 7fe408ff9640 1 -- 172.21.15.57:0/2265631752 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe3fc0b7070 con 0x7fe40c06aeb0 2023-10-30T14:20:30.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.519+0000 7fe408ff9640 1 --2- 172.21.15.57:0/2265631752 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3f80664d0 0x7fe3f8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:30.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.519+0000 7fe408ff9640 1 -- 172.21.15.57:0/2265631752 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6613+0+0 (secure 0 0 0) 0x7fe3fc122780 con 0x7fe40c06aeb0 2023-10-30T14:20:30.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.519+0000 7fe40b7fe640 1 --2- 172.21.15.57:0/2265631752 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3f80664d0 0x7fe3f8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:30.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.519+0000 7fe40b7fe640 1 --2- 172.21.15.57:0/2265631752 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3f80664d0 0x7fe3f8068990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe40c0c83b0 tx=0x7fe400007cb0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:30.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.523+0000 7fe408ff9640 1 -- 172.21.15.57:0/2265631752 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe3fc0ef740 con 0x7fe40c06aeb0 2023-10-30T14:20:30.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.715+0000 7fe411006640 1 -- 172.21.15.57:0/2265631752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe40c0c4630 con 0x7fe40c06aeb0 2023-10-30T14:20:30.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.715+0000 7fe408ff9640 1 -- 172.21.15.57:0/2265631752 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6533 (secure 0 0 0) 0x7fe3fc0b5020 con 0x7fe40c06aeb0 2023-10-30T14:20:30.721 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:30.721 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5078,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:15.987759+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:30.722 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 47 2023-10-30T14:20:30.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.715+0000 7fe411006640 1 -- 172.21.15.57:0/2265631752 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3f80664d0 msgr2=0x7fe3f8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:30.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.715+0000 7fe411006640 1 --2- 172.21.15.57:0/2265631752 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3f80664d0 0x7fe3f8068990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe40c0c83b0 tx=0x7fe400007cb0 comp rx=0 tx=0).stop 2023-10-30T14:20:30.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.715+0000 7fe411006640 1 -- 172.21.15.57:0/2265631752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe40c06aeb0 msgr2=0x7fe40c1f2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:30.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.715+0000 7fe411006640 1 --2- 172.21.15.57:0/2265631752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe40c06aeb0 0x7fe40c1f2b70 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0b3040 tx=0x7fe3fc01ba90 comp rx=0 tx=0).stop 2023-10-30T14:20:30.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.715+0000 7fe411006640 1 -- 172.21.15.57:0/2265631752 shutdown_connections 2023-10-30T14:20:30.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.715+0000 7fe411006640 1 --2- 172.21.15.57:0/2265631752 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3f80664d0 0x7fe3f8068990 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:30.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.715+0000 7fe411006640 1 --2- 172.21.15.57:0/2265631752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe40c15ac70 0x7fe40c0c3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:30.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.715+0000 7fe411006640 1 --2- 172.21.15.57:0/2265631752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe40c0c8eb0 0x7fe40c0c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:30.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.715+0000 7fe411006640 1 --2- 172.21.15.57:0/2265631752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe40c06aeb0 0x7fe40c1f2b70 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:30.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.715+0000 7fe411006640 1 -- 172.21.15.57:0/2265631752 >> 172.21.15.57:0/2265631752 conn(0x7fe40c0cef00 msgr2=0x7fe40c0c71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:30.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.719+0000 7fe411006640 1 -- 172.21.15.57:0/2265631752 shutdown_connections 2023-10-30T14:20:30.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:30.719+0000 7fe411006640 1 -- 172.21.15.57:0/2265631752 wait complete. 2023-10-30T14:20:30.727 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:20:30.731 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:20:30.742 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:20:32.744 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:32.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.891+0000 7fb1fbd1d640 1 Processor -- start 2023-10-30T14:20:32.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.891+0000 7fb1fbd1d640 1 -- start start 2023-10-30T14:20:32.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.895+0000 7fb1fbd1d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb1f4153c10 0x7fb1f4065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:32.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.895+0000 7fb1fbd1d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb1f4070c30 0x7fb1f4071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:32.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.895+0000 7fb1fbd1d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f406e390 con 0x7fb1f4159970 2023-10-30T14:20:32.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.895+0000 7fb1fbd1d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f4072390 con 0x7fb1f4153c10 2023-10-30T14:20:32.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.895+0000 7fb1fbd1d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f40660e0 con 0x7fb1f4070c30 2023-10-30T14:20:32.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.895+0000 7fb1f9ab9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb1f4153c10 0x7fb1f4065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:32.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.895+0000 7fb1f9ab9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb1f4153c10 0x7fb1f4065ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57272/0 (socket says 172.21.15.57:57272) 2023-10-30T14:20:32.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.895+0000 7fb1f92b8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb1f4070c30 0x7fb1f4071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:32.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.895+0000 7fb1fa2ba640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb1f4159970 0x7fb1f4159dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50336/0 (socket says 172.21.15.57:50336) 2023-10-30T14:20:32.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.895+0000 7fb1f9ab9640 1 -- 172.21.15.57:0/323142281 learned_addr learned my addr 172.21.15.57:0/323142281 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:32.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.895+0000 7fb1f9ab9640 1 -- 172.21.15.57:0/323142281 >> v2:172.21.15.88:3300/0 conn(0x7fb1f4070c30 msgr2=0x7fb1f4071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:32.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.895+0000 7fb1f9ab9640 1 --2- 172.21.15.57:0/323142281 >> v2:172.21.15.88:3300/0 conn(0x7fb1f4070c30 0x7fb1f4071070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:32.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.895+0000 7fb1f9ab9640 1 -- 172.21.15.57:0/323142281 >> v1:172.21.15.57:6789/0 conn(0x7fb1f4159970 legacy=0x7fb1f4159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:32.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.895+0000 7fb1f9ab9640 1 -- 172.21.15.57:0/323142281 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f40715b0 con 0x7fb1f4153c10 2023-10-30T14:20:32.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.895+0000 7fb1f8ab7640 1 -- 172.21.15.57:0/323142281 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343600407 0 0) 0x7fb1f406e390 con 0x7fb1f4159970 2023-10-30T14:20:32.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.899+0000 7fb1f9ab9640 1 --2- 172.21.15.57:0/323142281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1f4153c10 0x7fb1f4065ba0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fb1f406f200 tx=0x7fb1f001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da662c836a539906 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:32.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.899+0000 7fb1f8ab7640 1 -- 172.21.15.57:0/323142281 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f00ab600 con 0x7fb1f4153c10 2023-10-30T14:20:32.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.899+0000 7fb1f8ab7640 1 -- 172.21.15.57:0/323142281 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f00a7050 con 0x7fb1f4153c10 2023-10-30T14:20:32.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.899+0000 7fb1f8ab7640 1 -- 172.21.15.57:0/323142281 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f00ab980 con 0x7fb1f4153c10 2023-10-30T14:20:32.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.899+0000 7fb1fbd1d640 1 -- 172.21.15.57:0/323142281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1f4153c10 msgr2=0x7fb1f4065ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:32.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.899+0000 7fb1fbd1d640 1 --2- 172.21.15.57:0/323142281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1f4153c10 0x7fb1f4065ba0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fb1f406f200 tx=0x7fb1f001c470 comp rx=0 tx=0).stop 2023-10-30T14:20:32.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.899+0000 7fb1fbd1d640 1 -- 172.21.15.57:0/323142281 shutdown_connections 2023-10-30T14:20:32.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.899+0000 7fb1fbd1d640 1 --2- 172.21.15.57:0/323142281 >> v2:172.21.15.88:3300/0 conn(0x7fb1f4070c30 0x7fb1f4071070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:32.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.899+0000 7fb1fbd1d640 1 --2- 172.21.15.57:0/323142281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1f4153c10 0x7fb1f4065ba0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:32.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.899+0000 7fb1fbd1d640 1 -- 172.21.15.57:0/323142281 >> 172.21.15.57:0/323142281 conn(0x7fb1f40c7f60 msgr2=0x7fb1f40c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:32.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.899+0000 7fb1fbd1d640 1 -- 172.21.15.57:0/323142281 shutdown_connections 2023-10-30T14:20:32.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.899+0000 7fb1fbd1d640 1 -- 172.21.15.57:0/323142281 wait complete. 2023-10-30T14:20:32.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.899+0000 7fb1fbd1d640 1 Processor -- start 2023-10-30T14:20:32.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.899+0000 7fb1fbd1d640 1 -- start start 2023-10-30T14:20:32.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1fbd1d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1f4070c30 0x7fb1f40ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:32.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1fbd1d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1f4153c10 0x7fb1f40caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:32.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1fbd1d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f4159970 0x7fb1f414da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:32.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1fbd1d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f41f2040 con 0x7fb1f4153c10 2023-10-30T14:20:32.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1fbd1d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f41f2e60 con 0x7fb1f4159970 2023-10-30T14:20:32.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1fbd1d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f41f22e0 con 0x7fb1f4070c30 2023-10-30T14:20:32.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1f92b8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1f4153c10 0x7fb1f40caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:32.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1f92b8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1f4153c10 0x7fb1f40caaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57282/0 (socket says 172.21.15.57:57282) 2023-10-30T14:20:32.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1f92b8640 1 -- 172.21.15.57:0/2229484067 learned_addr learned my addr 172.21.15.57:0/2229484067 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:32.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1f9ab9640 1 --2- 172.21.15.57:0/2229484067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1f4070c30 0x7fb1f40ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:32.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1fa2ba640 1 --2- 172.21.15.57:0/2229484067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f4159970 0x7fb1f414da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:32.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1f92b8640 1 -- 172.21.15.57:0/2229484067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1f4070c30 msgr2=0x7fb1f40ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:32.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1f92b8640 1 --2- 172.21.15.57:0/2229484067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1f4070c30 0x7fb1f40ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:32.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1f92b8640 1 -- 172.21.15.57:0/2229484067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f4159970 msgr2=0x7fb1f414da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:32.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1f92b8640 1 --2- 172.21.15.57:0/2229484067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f4159970 0x7fb1f414da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:32.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1f92b8640 1 -- 172.21.15.57:0/2229484067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f00094d0 con 0x7fb1f4153c10 2023-10-30T14:20:32.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1f9ab9640 1 --2- 172.21.15.57:0/2229484067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1f4070c30 0x7fb1f40ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:32.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1f92b8640 1 --2- 172.21.15.57:0/2229484067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1f4153c10 0x7fb1f40caaa0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb1e400e930 tx=0x7fb1e400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:32.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1ea7fc640 1 -- 172.21.15.57:0/2229484067 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4009690 con 0x7fb1f4153c10 2023-10-30T14:20:32.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1ea7fc640 1 -- 172.21.15.57:0/2229484067 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e400ca00 con 0x7fb1f4153c10 2023-10-30T14:20:32.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1ea7fc640 1 -- 172.21.15.57:0/2229484067 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e400cd70 con 0x7fb1f4153c10 2023-10-30T14:20:32.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1fbd1d640 1 -- 172.21.15.57:0/2229484067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f41f25c0 con 0x7fb1f4153c10 2023-10-30T14:20:32.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1fbd1d640 1 -- 172.21.15.57:0/2229484067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f41f2b80 con 0x7fb1f4153c10 2023-10-30T14:20:32.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1fbd1d640 1 -- 172.21.15.57:0/2229484067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f404e170 con 0x7fb1f4153c10 2023-10-30T14:20:32.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.903+0000 7fb1ea7fc640 1 -- 172.21.15.57:0/2229484067 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb1e4013470 con 0x7fb1f4153c10 2023-10-30T14:20:32.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.907+0000 7fb1ea7fc640 1 --2- 172.21.15.57:0/2229484067 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb1d80664d0 0x7fb1d8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:32.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.907+0000 7fb1ea7fc640 1 -- 172.21.15.57:0/2229484067 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6613+0+0 (secure 0 0 0) 0x7fb1e4084560 con 0x7fb1f4153c10 2023-10-30T14:20:32.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.907+0000 7fb1f9ab9640 1 --2- 172.21.15.57:0/2229484067 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb1d80664d0 0x7fb1d8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:32.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.907+0000 7fb1f9ab9640 1 --2- 172.21.15.57:0/2229484067 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb1d80664d0 0x7fb1d8068990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb1f00ad000 tx=0x7fb1f00a6000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:32.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:32.911+0000 7fb1ea7fc640 1 -- 172.21.15.57:0/2229484067 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb1e4051660 con 0x7fb1f4153c10 2023-10-30T14:20:33.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:33.099+0000 7fb1fbd1d640 1 -- 172.21.15.57:0/2229484067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1f4070150 con 0x7fb1f4153c10 2023-10-30T14:20:33.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:33.099+0000 7fb1ea7fc640 1 -- 172.21.15.57:0/2229484067 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6533 (secure 0 0 0) 0x7fb1e4054c80 con 0x7fb1f4153c10 2023-10-30T14:20:33.106 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:33.106 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5078,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:15.987759+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":4936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4936":{"gid":4936,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":31,"addr":"172.21.15.57:6837/1988100930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1988100930},{"type":"v1","addr":"172.21.15.57:6837","nonce":1988100930}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:33.107 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 47 2023-10-30T14:20:33.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:33.103+0000 7fb1fbd1d640 1 -- 172.21.15.57:0/2229484067 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb1d80664d0 msgr2=0x7fb1d8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:33.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:33.103+0000 7fb1fbd1d640 1 --2- 172.21.15.57:0/2229484067 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb1d80664d0 0x7fb1d8068990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb1f00ad000 tx=0x7fb1f00a6000 comp rx=0 tx=0).stop 2023-10-30T14:20:33.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:33.103+0000 7fb1fbd1d640 1 -- 172.21.15.57:0/2229484067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1f4153c10 msgr2=0x7fb1f40caaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:33.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:33.103+0000 7fb1fbd1d640 1 --2- 172.21.15.57:0/2229484067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1f4153c10 0x7fb1f40caaa0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb1e400e930 tx=0x7fb1e400edf0 comp rx=0 tx=0).stop 2023-10-30T14:20:33.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:33.103+0000 7fb1fbd1d640 1 -- 172.21.15.57:0/2229484067 shutdown_connections 2023-10-30T14:20:33.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:33.103+0000 7fb1fbd1d640 1 --2- 172.21.15.57:0/2229484067 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb1d80664d0 0x7fb1d8068990 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:33.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:33.103+0000 7fb1fbd1d640 1 --2- 172.21.15.57:0/2229484067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f4159970 0x7fb1f414da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:33.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:33.103+0000 7fb1fbd1d640 1 --2- 172.21.15.57:0/2229484067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1f4153c10 0x7fb1f40caaa0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:33.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:33.103+0000 7fb1fbd1d640 1 --2- 172.21.15.57:0/2229484067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1f4070c30 0x7fb1f40ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:33.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:33.103+0000 7fb1fbd1d640 1 -- 172.21.15.57:0/2229484067 >> 172.21.15.57:0/2229484067 conn(0x7fb1f40c7f60 msgr2=0x7fb1f41532a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:33.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:33.103+0000 7fb1fbd1d640 1 -- 172.21.15.57:0/2229484067 shutdown_connections 2023-10-30T14:20:33.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:33.103+0000 7fb1fbd1d640 1 -- 172.21.15.57:0/2229484067 wait complete. 2023-10-30T14:20:33.127 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:20:35.128 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:35.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.279+0000 7f147ec54640 1 Processor -- start 2023-10-30T14:20:35.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.279+0000 7f147ec54640 1 -- start start 2023-10-30T14:20:35.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.279+0000 7f147ec54640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f14780c4a10 0x7f147815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:35.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.279+0000 7f147ec54640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14780c42a0 0x7f14780ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:35.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.279+0000 7f147ec54640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1478159f20 con 0x7f1478153c80 2023-10-30T14:20:35.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.279+0000 7f147ec54640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f147815dec0 con 0x7f14780c4a10 2023-10-30T14:20:35.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.279+0000 7f147ec54640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14781599a0 con 0x7f14780c42a0 2023-10-30T14:20:35.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.279+0000 7f1477fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f14780c4a10 0x7f147815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:35.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.279+0000 7f147c9f0640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1478153c80 0x7f147815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:47710/0 (socket says 172.21.15.57:47710) 2023-10-30T14:20:35.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.279+0000 7f147c9f0640 1 -- 172.21.15.57:0/2080159372 learned_addr learned my addr 172.21.15.57:0/2080159372 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:20:35.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.279+0000 7f147d1f1640 1 --2- 172.21.15.57:0/2080159372 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14780c42a0 0x7f14780ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:35.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.279+0000 7f1477fff640 1 -- 172.21.15.57:0/2080159372 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14780c42a0 msgr2=0x7f14780ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:35.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.279+0000 7f1477fff640 1 --2- 172.21.15.57:0/2080159372 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14780c42a0 0x7f14780ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:35.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.279+0000 7f1477fff640 1 -- 172.21.15.57:0/2080159372 >> v1:172.21.15.57:6789/0 conn(0x7f1478153c80 legacy=0x7f147815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:35.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.279+0000 7f1477fff640 1 -- 172.21.15.57:0/2080159372 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147815d620 con 0x7f14780c4a10 2023-10-30T14:20:35.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.283+0000 7f1477fff640 1 --2- 172.21.15.57:0/2080159372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14780c4a10 0x7f147815cf90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f14640097c0 tx=0x7f146401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f18a0ac6756e7ac6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:35.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.283+0000 7f14777fe640 1 -- 172.21.15.57:0/2080159372 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14640a8970 con 0x7f14780c4a10 2023-10-30T14:20:35.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.283+0000 7f14777fe640 1 -- 172.21.15.57:0/2080159372 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14640a8ad0 con 0x7f14780c4a10 2023-10-30T14:20:35.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.283+0000 7f14777fe640 1 -- 172.21.15.57:0/2080159372 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14640a8e60 con 0x7f14780c4a10 2023-10-30T14:20:35.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.283+0000 7f147ec54640 1 -- 172.21.15.57:0/2080159372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14780c4a10 msgr2=0x7f147815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:35.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.283+0000 7f147ec54640 1 --2- 172.21.15.57:0/2080159372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14780c4a10 0x7f147815cf90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f14640097c0 tx=0x7f146401ec70 comp rx=0 tx=0).stop 2023-10-30T14:20:35.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.283+0000 7f147ec54640 1 -- 172.21.15.57:0/2080159372 shutdown_connections 2023-10-30T14:20:35.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.283+0000 7f147ec54640 1 --2- 172.21.15.57:0/2080159372 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14780c42a0 0x7f14780ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:35.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.283+0000 7f147ec54640 1 --2- 172.21.15.57:0/2080159372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14780c4a10 0x7f147815cf90 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:35.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.283+0000 7f147ec54640 1 -- 172.21.15.57:0/2080159372 >> 172.21.15.57:0/2080159372 conn(0x7f14780715c0 msgr2=0x7f14780719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:35.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.283+0000 7f147ec54640 1 -- 172.21.15.57:0/2080159372 shutdown_connections 2023-10-30T14:20:35.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.283+0000 7f147ec54640 1 -- 172.21.15.57:0/2080159372 wait complete. 2023-10-30T14:20:35.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f147ec54640 1 Processor -- start 2023-10-30T14:20:35.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f147ec54640 1 -- start start 2023-10-30T14:20:35.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f147ec54640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14780c42a0 0x7f147814b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:35.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f147ec54640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14780c4a10 0x7f1478149ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:35.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f147ec54640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1478153c80 0x7f147815c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:35.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f147ec54640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14781550d0 con 0x7f14780c4a10 2023-10-30T14:20:35.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f147ec54640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147814ba50 con 0x7f14780c42a0 2023-10-30T14:20:35.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f147ec54640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1478152670 con 0x7f1478153c80 2023-10-30T14:20:35.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f1477fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14780c4a10 0x7f1478149ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:35.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f1477fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14780c4a10 0x7f1478149ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55080/0 (socket says 172.21.15.57:55080) 2023-10-30T14:20:35.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f1477fff640 1 -- 172.21.15.57:0/446334561 learned_addr learned my addr 172.21.15.57:0/446334561 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:35.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f147c9f0640 1 --2- 172.21.15.57:0/446334561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14780c42a0 0x7f147814b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:35.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f147d1f1640 1 --2- 172.21.15.57:0/446334561 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1478153c80 0x7f147815c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:35.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f1477fff640 1 -- 172.21.15.57:0/446334561 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1478153c80 msgr2=0x7f147815c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:35.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f1477fff640 1 --2- 172.21.15.57:0/446334561 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1478153c80 0x7f147815c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:35.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f1477fff640 1 -- 172.21.15.57:0/446334561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14780c42a0 msgr2=0x7f147814b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:35.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f1477fff640 1 --2- 172.21.15.57:0/446334561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14780c42a0 0x7f147814b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:35.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f1477fff640 1 -- 172.21.15.57:0/446334561 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1464009420 con 0x7f14780c4a10 2023-10-30T14:20:35.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f1477fff640 1 --2- 172.21.15.57:0/446334561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14780c4a10 0x7f1478149ea0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f14640b3040 tx=0x7f146401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:35.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f14757fa640 1 -- 172.21.15.57:0/446334561 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14640b9630 con 0x7f14780c4a10 2023-10-30T14:20:35.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f14757fa640 1 -- 172.21.15.57:0/446334561 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14640b97b0 con 0x7f14780c4a10 2023-10-30T14:20:35.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f147ec54640 1 -- 172.21.15.57:0/446334561 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14781520b0 con 0x7f14780c4a10 2023-10-30T14:20:35.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f14757fa640 1 -- 172.21.15.57:0/446334561 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14640b9b60 con 0x7f14780c4a10 2023-10-30T14:20:35.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.287+0000 7f147ec54640 1 -- 172.21.15.57:0/446334561 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147814aa30 con 0x7f14780c4a10 2023-10-30T14:20:35.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.291+0000 7f147ec54640 1 -- 172.21.15.57:0/446334561 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f147804e170 con 0x7f14780c4a10 2023-10-30T14:20:35.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.291+0000 7f14757fa640 1 -- 172.21.15.57:0/446334561 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f14640b7040 con 0x7f14780c4a10 2023-10-30T14:20:35.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.291+0000 7f14757fa640 1 --2- 172.21.15.57:0/446334561 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f145c0664a0 0x7f145c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:35.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.291+0000 7f14757fa640 1 -- 172.21.15.57:0/446334561 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f1464122190 con 0x7f14780c4a10 2023-10-30T14:20:35.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.291+0000 7f147c9f0640 1 --2- 172.21.15.57:0/446334561 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f145c0664a0 0x7f145c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:35.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.291+0000 7f147c9f0640 1 --2- 172.21.15.57:0/446334561 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f145c0664a0 0x7f145c068960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f146c005890 tx=0x7f146c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:35.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.295+0000 7f14757fa640 1 -- 172.21.15.57:0/446334561 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f14640ef2c0 con 0x7f14780c4a10 2023-10-30T14:20:35.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.487+0000 7f147ec54640 1 -- 172.21.15.57:0/446334561 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14780cad40 con 0x7f14780c4a10 2023-10-30T14:20:35.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.487+0000 7f14757fa640 1 -- 172.21.15.57:0/446334561 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f14640b5020 con 0x7f14780c4a10 2023-10-30T14:20:35.492 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:35.492 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:34.693643+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:35.493 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 48 2023-10-30T14:20:35.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.487+0000 7f147ec54640 1 -- 172.21.15.57:0/446334561 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f145c0664a0 msgr2=0x7f145c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:35.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.487+0000 7f147ec54640 1 --2- 172.21.15.57:0/446334561 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f145c0664a0 0x7f145c068960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f146c005890 tx=0x7f146c009040 comp rx=0 tx=0).stop 2023-10-30T14:20:35.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.487+0000 7f147ec54640 1 -- 172.21.15.57:0/446334561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14780c4a10 msgr2=0x7f1478149ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:35.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.487+0000 7f147ec54640 1 --2- 172.21.15.57:0/446334561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14780c4a10 0x7f1478149ea0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f14640b3040 tx=0x7f146401ba10 comp rx=0 tx=0).stop 2023-10-30T14:20:35.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.487+0000 7f147ec54640 1 -- 172.21.15.57:0/446334561 shutdown_connections 2023-10-30T14:20:35.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.487+0000 7f147ec54640 1 --2- 172.21.15.57:0/446334561 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f145c0664a0 0x7f145c068960 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:35.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.487+0000 7f147ec54640 1 --2- 172.21.15.57:0/446334561 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1478153c80 0x7f147815c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:35.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.487+0000 7f147ec54640 1 --2- 172.21.15.57:0/446334561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14780c4a10 0x7f1478149ea0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:35.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.487+0000 7f147ec54640 1 --2- 172.21.15.57:0/446334561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14780c42a0 0x7f147814b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:35.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.487+0000 7f147ec54640 1 -- 172.21.15.57:0/446334561 >> 172.21.15.57:0/446334561 conn(0x7f14780715c0 msgr2=0x7f14780c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:35.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.487+0000 7f147ec54640 1 -- 172.21.15.57:0/446334561 shutdown_connections 2023-10-30T14:20:35.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.487+0000 7f147ec54640 1 -- 172.21.15.57:0/446334561 wait complete. 2023-10-30T14:20:35.513 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T14:20:35.513 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:20:35.514 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:35.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.651+0000 7f83acbc5640 1 Processor -- start 2023-10-30T14:20:35.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.651+0000 7f83acbc5640 1 -- start start 2023-10-30T14:20:35.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.651+0000 7f83acbc5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f83a80c4a10 0x7f83a815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:35.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.651+0000 7f83acbc5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83a80c42a0 0x7f83a80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:35.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.651+0000 7f83acbc5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83a8159f20 con 0x7f83a8153c80 2023-10-30T14:20:35.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.651+0000 7f83acbc5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83a815dec0 con 0x7f83a80c4a10 2023-10-30T14:20:35.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.651+0000 7f83acbc5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a81599a0 con 0x7f83a80c42a0 2023-10-30T14:20:35.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.651+0000 7f839ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f83a80c4a10 0x7f83a815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:35.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.651+0000 7f839ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f83a80c4a10 0x7f83a815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55092/0 (socket says 172.21.15.57:55092) 2023-10-30T14:20:35.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.651+0000 7f839ffff640 1 -- 172.21.15.57:0/1613676924 learned_addr learned my addr 172.21.15.57:0/1613676924 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:35.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.651+0000 7f83a6d9d640 1 --2- 172.21.15.57:0/1613676924 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83a80c42a0 0x7f83a80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:35.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.651+0000 7f839ffff640 1 -- 172.21.15.57:0/1613676924 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83a80c42a0 msgr2=0x7f83a80ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:35.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.651+0000 7f839ffff640 1 --2- 172.21.15.57:0/1613676924 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83a80c42a0 0x7f83a80ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:35.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.651+0000 7f839ffff640 1 -- 172.21.15.57:0/1613676924 >> v1:172.21.15.88:6789/0 conn(0x7f83a8153c80 legacy=0x7f83a815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:35.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.655+0000 7f839ffff640 1 -- 172.21.15.57:0/1613676924 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a815d620 con 0x7f83a80c4a10 2023-10-30T14:20:35.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.655+0000 7f83a6d9d640 1 --2- 172.21.15.57:0/1613676924 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83a80c42a0 0x7f83a80ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:35.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.655+0000 7f839ffff640 1 --2- 172.21.15.57:0/1613676924 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83a80c4a10 0x7f83a815cf90 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8390009870 tx=0x7f839001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a8c53321b2457fd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:35.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.655+0000 7f83a5d9b640 1 -- 172.21.15.57:0/1613676924 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83900a8a30 con 0x7f83a80c4a10 2023-10-30T14:20:35.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.655+0000 7f83a5d9b640 1 -- 172.21.15.57:0/1613676924 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83900a8b90 con 0x7f83a80c4a10 2023-10-30T14:20:35.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.655+0000 7f83a5d9b640 1 -- 172.21.15.57:0/1613676924 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83900b0440 con 0x7f83a80c4a10 2023-10-30T14:20:35.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.655+0000 7f83acbc5640 1 -- 172.21.15.57:0/1613676924 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83a80c4a10 msgr2=0x7f83a815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:35.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.655+0000 7f83acbc5640 1 --2- 172.21.15.57:0/1613676924 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83a80c4a10 0x7f83a815cf90 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8390009870 tx=0x7f839001ed30 comp rx=0 tx=0).stop 2023-10-30T14:20:35.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.655+0000 7f83acbc5640 1 -- 172.21.15.57:0/1613676924 shutdown_connections 2023-10-30T14:20:35.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.655+0000 7f83acbc5640 1 --2- 172.21.15.57:0/1613676924 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83a80c42a0 0x7f83a80ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:35.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.655+0000 7f83acbc5640 1 --2- 172.21.15.57:0/1613676924 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83a80c4a10 0x7f83a815cf90 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:35.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.655+0000 7f83acbc5640 1 -- 172.21.15.57:0/1613676924 >> 172.21.15.57:0/1613676924 conn(0x7f83a80715c0 msgr2=0x7f83a80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:35.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.655+0000 7f83acbc5640 1 -- 172.21.15.57:0/1613676924 shutdown_connections 2023-10-30T14:20:35.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.655+0000 7f83acbc5640 1 -- 172.21.15.57:0/1613676924 wait complete. 2023-10-30T14:20:35.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f83acbc5640 1 Processor -- start 2023-10-30T14:20:35.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f83acbc5640 1 -- start start 2023-10-30T14:20:35.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f83acbc5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83a80c42a0 0x7f83a814f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:35.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f83acbc5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83a80c4a10 0x7f83a814e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:35.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f83acbc5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83a8153c80 0x7f83a81f1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:35.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f83acbc5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a8154c40 con 0x7f83a80c42a0 2023-10-30T14:20:35.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f83acbc5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a814fc30 con 0x7f83a80c4a10 2023-10-30T14:20:35.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f83acbc5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a814ec10 con 0x7f83a8153c80 2023-10-30T14:20:35.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f83a659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83a80c42a0 0x7f83a814f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:35.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f83a659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83a80c42a0 0x7f83a814f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55098/0 (socket says 172.21.15.57:55098) 2023-10-30T14:20:35.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f83a659c640 1 -- 172.21.15.57:0/1120249723 learned_addr learned my addr 172.21.15.57:0/1120249723 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:35.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f83a6d9d640 1 --2- 172.21.15.57:0/1120249723 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83a8153c80 0x7f83a81f1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:35.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f839ffff640 1 --2- 172.21.15.57:0/1120249723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83a80c4a10 0x7f83a814e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:35.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f83a659c640 1 -- 172.21.15.57:0/1120249723 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83a8153c80 msgr2=0x7f83a81f1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:35.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f83a659c640 1 --2- 172.21.15.57:0/1120249723 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83a8153c80 0x7f83a81f1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:35.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f83a659c640 1 -- 172.21.15.57:0/1120249723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83a80c4a10 msgr2=0x7f83a814e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:35.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f83a659c640 1 --2- 172.21.15.57:0/1120249723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83a80c4a10 0x7f83a814e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:35.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f83a659c640 1 -- 172.21.15.57:0/1120249723 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83900094d0 con 0x7f83a80c42a0 2023-10-30T14:20:35.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f83a659c640 1 --2- 172.21.15.57:0/1120249723 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83a80c42a0 0x7f83a814f6f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f839800c3c0 tx=0x7f839800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:35.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f839effd640 1 -- 172.21.15.57:0/1120249723 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83980052d0 con 0x7f83a80c42a0 2023-10-30T14:20:35.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f839effd640 1 -- 172.21.15.57:0/1120249723 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8398005430 con 0x7f83a80c42a0 2023-10-30T14:20:35.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f83acbc5640 1 -- 172.21.15.57:0/1120249723 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a81f25c0 con 0x7f83a80c42a0 2023-10-30T14:20:35.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.659+0000 7f839effd640 1 -- 172.21.15.57:0/1120249723 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8398003840 con 0x7f83a80c42a0 2023-10-30T14:20:35.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.663+0000 7f83acbc5640 1 -- 172.21.15.57:0/1120249723 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a815bbd0 con 0x7f83a80c42a0 2023-10-30T14:20:35.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.663+0000 7f83acbc5640 1 -- 172.21.15.57:0/1120249723 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a804e170 con 0x7f83a80c42a0 2023-10-30T14:20:35.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.663+0000 7f839effd640 1 -- 172.21.15.57:0/1120249723 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f83980039a0 con 0x7f83a80c42a0 2023-10-30T14:20:35.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.663+0000 7f839effd640 1 --2- 172.21.15.57:0/1120249723 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f83880664a0 0x7f8388068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:35.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.663+0000 7f839effd640 1 -- 172.21.15.57:0/1120249723 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f8398083720 con 0x7f83a80c42a0 2023-10-30T14:20:35.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.663+0000 7f839ffff640 1 --2- 172.21.15.57:0/1120249723 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f83880664a0 0x7f8388068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:35.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.663+0000 7f839ffff640 1 --2- 172.21.15.57:0/1120249723 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f83880664a0 0x7f8388068960 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f83900b2000 tx=0x7f8390008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:35.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.667+0000 7f839effd640 1 -- 172.21.15.57:0/1120249723 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8398050850 con 0x7f83a80c42a0 2023-10-30T14:20:35.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.863+0000 7f83acbc5640 1 -- 172.21.15.57:0/1120249723 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83a80cad40 con 0x7f83a80c42a0 2023-10-30T14:20:35.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.863+0000 7f839effd640 1 -- 172.21.15.57:0/1120249723 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f8398053e70 con 0x7f83a80c42a0 2023-10-30T14:20:35.869 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:35.869 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:34.693643+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:35.869 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 48 2023-10-30T14:20:35.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.863+0000 7f83acbc5640 1 -- 172.21.15.57:0/1120249723 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f83880664a0 msgr2=0x7f8388068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:35.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.863+0000 7f83acbc5640 1 --2- 172.21.15.57:0/1120249723 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f83880664a0 0x7f8388068960 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f83900b2000 tx=0x7f8390008040 comp rx=0 tx=0).stop 2023-10-30T14:20:35.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.863+0000 7f83acbc5640 1 -- 172.21.15.57:0/1120249723 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83a80c42a0 msgr2=0x7f83a814f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:35.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.863+0000 7f83acbc5640 1 --2- 172.21.15.57:0/1120249723 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83a80c42a0 0x7f83a814f6f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f839800c3c0 tx=0x7f839800c880 comp rx=0 tx=0).stop 2023-10-30T14:20:35.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.863+0000 7f83acbc5640 1 -- 172.21.15.57:0/1120249723 shutdown_connections 2023-10-30T14:20:35.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.863+0000 7f83acbc5640 1 --2- 172.21.15.57:0/1120249723 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f83880664a0 0x7f8388068960 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:35.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.863+0000 7f83acbc5640 1 --2- 172.21.15.57:0/1120249723 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83a8153c80 0x7f83a81f1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:35.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.863+0000 7f83acbc5640 1 --2- 172.21.15.57:0/1120249723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83a80c4a10 0x7f83a814e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:35.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.863+0000 7f83acbc5640 1 --2- 172.21.15.57:0/1120249723 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83a80c42a0 0x7f83a814f6f0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:35.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.863+0000 7f83acbc5640 1 -- 172.21.15.57:0/1120249723 >> 172.21.15.57:0/1120249723 conn(0x7f83a80715c0 msgr2=0x7f83a81591b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:35.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.867+0000 7f83acbc5640 1 -- 172.21.15.57:0/1120249723 shutdown_connections 2023-10-30T14:20:35.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:35.867+0000 7f83acbc5640 1 -- 172.21.15.57:0/1120249723 wait complete. 2023-10-30T14:20:35.891 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 48, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 48, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5078": { "addr": "172.21.15.88:6839/1677924344", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1677924344, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1677924344, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5078, "incarnation": 48, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:replay", "state_seq": 1 }, "gid_5133": { "addr": "172.21.15.57:6834/2912050965", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2912050965, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2912050965, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5133, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 15 }, "gid_5289": { "addr": "172.21.15.57:6836/2427081813", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2427081813, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2427081813, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5289, "incarnation": 35, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:20:34.693643+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5289, "mds_1": 5133, "mds_2": 5078 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:20:37.892 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:38.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.043+0000 7f2b68bb8640 1 Processor -- start 2023-10-30T14:20:38.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.043+0000 7f2b68bb8640 1 -- start start 2023-10-30T14:20:38.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.047+0000 7f2b68bb8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2b640cb7d0 0x7f2b640c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:38.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.047+0000 7f2b68bb8640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2b640c3e40 0x7f2b64070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:38.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.047+0000 7f2b68bb8640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b640c2310 con 0x7f2b64150830 2023-10-30T14:20:38.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.047+0000 7f2b68bb8640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b6406bc10 con 0x7f2b640c3e40 2023-10-30T14:20:38.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.047+0000 7f2b68bb8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b64156710 con 0x7f2b640cb7d0 2023-10-30T14:20:38.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.047+0000 7f2b62d9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2b640c3e40 0x7f2b64070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:38.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.047+0000 7f2b62d9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2b640c3e40 0x7f2b64070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55106/0 (socket says 172.21.15.57:55106) 2023-10-30T14:20:38.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.047+0000 7f2b62d9d640 1 -- 172.21.15.57:0/2141265700 learned_addr learned my addr 172.21.15.57:0/2141265700 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:38.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.047+0000 7f2b6259c640 1 --1- 172.21.15.57:0/2141265700 >> v1:172.21.15.57:6789/0 conn(0x7f2b64150830 0x7f2b6406b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:47714/0 (socket says 172.21.15.57:47714) 2023-10-30T14:20:38.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.047+0000 7f2b5bfff640 1 --2- 172.21.15.57:0/2141265700 >> v2:172.21.15.88:3300/0 conn(0x7f2b640cb7d0 0x7f2b640c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:38.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.047+0000 7f2b62d9d640 1 -- 172.21.15.57:0/2141265700 >> v2:172.21.15.88:3300/0 conn(0x7f2b640cb7d0 msgr2=0x7f2b640c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:38.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.047+0000 7f2b62d9d640 1 --2- 172.21.15.57:0/2141265700 >> v2:172.21.15.88:3300/0 conn(0x7f2b640cb7d0 0x7f2b640c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.047+0000 7f2b62d9d640 1 -- 172.21.15.57:0/2141265700 >> v1:172.21.15.57:6789/0 conn(0x7f2b64150830 legacy=0x7f2b6406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:38.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.047+0000 7f2b62d9d640 1 -- 172.21.15.57:0/2141265700 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b640c4360 con 0x7f2b640c3e40 2023-10-30T14:20:38.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.047+0000 7f2b61d9b640 1 -- 172.21.15.57:0/2141265700 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218552288 0 0) 0x7f2b640c2310 con 0x7f2b64150830 2023-10-30T14:20:38.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.047+0000 7f2b62d9d640 1 --2- 172.21.15.57:0/2141265700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b640c3e40 0x7f2b64070520 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2b6406f9f0 tx=0x7f2b5c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=559f396f3a6c0728 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:38.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.051+0000 7f2b61d9b640 1 -- 172.21.15.57:0/2141265700 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c0af400 con 0x7f2b640c3e40 2023-10-30T14:20:38.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.051+0000 7f2b61d9b640 1 -- 172.21.15.57:0/2141265700 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5c0ab050 con 0x7f2b640c3e40 2023-10-30T14:20:38.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.051+0000 7f2b61d9b640 1 -- 172.21.15.57:0/2141265700 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c0af780 con 0x7f2b640c3e40 2023-10-30T14:20:38.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.051+0000 7f2b68bb8640 1 -- 172.21.15.57:0/2141265700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b640c3e40 msgr2=0x7f2b64070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:38.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.051+0000 7f2b68bb8640 1 --2- 172.21.15.57:0/2141265700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b640c3e40 0x7f2b64070520 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2b6406f9f0 tx=0x7f2b5c01f1c0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.051+0000 7f2b68bb8640 1 -- 172.21.15.57:0/2141265700 shutdown_connections 2023-10-30T14:20:38.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.051+0000 7f2b68bb8640 1 --2- 172.21.15.57:0/2141265700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b640c3e40 0x7f2b64070520 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.051+0000 7f2b68bb8640 1 --2- 172.21.15.57:0/2141265700 >> v2:172.21.15.88:3300/0 conn(0x7f2b640cb7d0 0x7f2b640c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.051+0000 7f2b68bb8640 1 -- 172.21.15.57:0/2141265700 >> 172.21.15.57:0/2141265700 conn(0x7f2b640cac70 msgr2=0x7f2b640cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:38.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.051+0000 7f2b68bb8640 1 -- 172.21.15.57:0/2141265700 shutdown_connections 2023-10-30T14:20:38.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.051+0000 7f2b68bb8640 1 -- 172.21.15.57:0/2141265700 wait complete. 2023-10-30T14:20:38.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.051+0000 7f2b68bb8640 1 Processor -- start 2023-10-30T14:20:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.051+0000 7f2b68bb8640 1 -- start start 2023-10-30T14:20:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.051+0000 7f2b68bb8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b640c3e40 0x7f2b6414a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.051+0000 7f2b68bb8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b640cb7d0 0x7f2b64149d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.051+0000 7f2b68bb8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b64150830 0x7f2b64159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.051+0000 7f2b68bb8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b641ed120 con 0x7f2b640c3e40 2023-10-30T14:20:38.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.051+0000 7f2b68bb8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6414a240 con 0x7f2b640cb7d0 2023-10-30T14:20:38.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.051+0000 7f2b68bb8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b641ec7a0 con 0x7f2b64150830 2023-10-30T14:20:38.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.055+0000 7f2b6259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b640c3e40 0x7f2b6414a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:38.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.055+0000 7f2b6259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b640c3e40 0x7f2b6414a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55118/0 (socket says 172.21.15.57:55118) 2023-10-30T14:20:38.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.055+0000 7f2b6259c640 1 -- 172.21.15.57:0/3527161824 learned_addr learned my addr 172.21.15.57:0/3527161824 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:38.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.055+0000 7f2b62d9d640 1 --2- 172.21.15.57:0/3527161824 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b64150830 0x7f2b64159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:38.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.055+0000 7f2b5bfff640 1 --2- 172.21.15.57:0/3527161824 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b640cb7d0 0x7f2b64149d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:38.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.055+0000 7f2b6259c640 1 -- 172.21.15.57:0/3527161824 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b64150830 msgr2=0x7f2b64159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:38.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.055+0000 7f2b6259c640 1 --2- 172.21.15.57:0/3527161824 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b64150830 0x7f2b64159540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.055+0000 7f2b6259c640 1 -- 172.21.15.57:0/3527161824 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b640cb7d0 msgr2=0x7f2b64149d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:38.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.055+0000 7f2b6259c640 1 --2- 172.21.15.57:0/3527161824 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b640cb7d0 0x7f2b64149d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.055+0000 7f2b6259c640 1 -- 172.21.15.57:0/3527161824 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c009b30 con 0x7f2b640c3e40 2023-10-30T14:20:38.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.055+0000 7f2b6259c640 1 --2- 172.21.15.57:0/3527161824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b640c3e40 0x7f2b6414a900 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2b5400c4b0 tx=0x7f2b5400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:38.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.055+0000 7f2b5affd640 1 -- 172.21.15.57:0/3527161824 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b54007480 con 0x7f2b640c3e40 2023-10-30T14:20:38.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.055+0000 7f2b5affd640 1 -- 172.21.15.57:0/3527161824 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b540075e0 con 0x7f2b640c3e40 2023-10-30T14:20:38.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.055+0000 7f2b5affd640 1 -- 172.21.15.57:0/3527161824 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b54004a30 con 0x7f2b640c3e40 2023-10-30T14:20:38.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.055+0000 7f2b68bb8640 1 -- 172.21.15.57:0/3527161824 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b6414aee0 con 0x7f2b640c3e40 2023-10-30T14:20:38.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.055+0000 7f2b68bb8640 1 -- 172.21.15.57:0/3527161824 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b641eca60 con 0x7f2b640c3e40 2023-10-30T14:20:38.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.055+0000 7f2b68bb8640 1 -- 172.21.15.57:0/3527161824 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b6404e170 con 0x7f2b640c3e40 2023-10-30T14:20:38.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.055+0000 7f2b5affd640 1 -- 172.21.15.57:0/3527161824 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2b54004b90 con 0x7f2b640c3e40 2023-10-30T14:20:38.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.059+0000 7f2b5affd640 1 --2- 172.21.15.57:0/3527161824 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b440664a0 0x7f2b44068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:38.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.059+0000 7f2b5affd640 1 -- 172.21.15.57:0/3527161824 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f2b54083260 con 0x7f2b640c3e40 2023-10-30T14:20:38.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.059+0000 7f2b5bfff640 1 --2- 172.21.15.57:0/3527161824 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b440664a0 0x7f2b44068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:38.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.059+0000 7f2b5bfff640 1 --2- 172.21.15.57:0/3527161824 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b440664a0 0x7f2b44068960 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2b64070f80 tx=0x7f2b4c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:38.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.063+0000 7f2b5affd640 1 -- 172.21.15.57:0/3527161824 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2b54050620 con 0x7f2b640c3e40 2023-10-30T14:20:38.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.251+0000 7f2b68bb8640 1 -- 172.21.15.57:0/3527161824 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b6414cfc0 con 0x7f2b640c3e40 2023-10-30T14:20:38.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.251+0000 7f2b5affd640 1 -- 172.21.15.57:0/3527161824 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f2b54053c40 con 0x7f2b640c3e40 2023-10-30T14:20:38.258 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:38.258 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:34.693643+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:38.259 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 48 2023-10-30T14:20:38.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.255+0000 7f2b68bb8640 1 -- 172.21.15.57:0/3527161824 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b440664a0 msgr2=0x7f2b44068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:38.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.255+0000 7f2b68bb8640 1 --2- 172.21.15.57:0/3527161824 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b440664a0 0x7f2b44068960 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2b64070f80 tx=0x7f2b4c008040 comp rx=0 tx=0).stop 2023-10-30T14:20:38.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.255+0000 7f2b68bb8640 1 -- 172.21.15.57:0/3527161824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b640c3e40 msgr2=0x7f2b6414a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:38.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.255+0000 7f2b68bb8640 1 --2- 172.21.15.57:0/3527161824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b640c3e40 0x7f2b6414a900 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2b5400c4b0 tx=0x7f2b5400c970 comp rx=0 tx=0).stop 2023-10-30T14:20:38.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.255+0000 7f2b68bb8640 1 -- 172.21.15.57:0/3527161824 shutdown_connections 2023-10-30T14:20:38.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.255+0000 7f2b68bb8640 1 --2- 172.21.15.57:0/3527161824 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b440664a0 0x7f2b44068960 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.255+0000 7f2b68bb8640 1 --2- 172.21.15.57:0/3527161824 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b64150830 0x7f2b64159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.255+0000 7f2b68bb8640 1 --2- 172.21.15.57:0/3527161824 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b640cb7d0 0x7f2b64149d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.255+0000 7f2b68bb8640 1 --2- 172.21.15.57:0/3527161824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b640c3e40 0x7f2b6414a900 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.255+0000 7f2b68bb8640 1 -- 172.21.15.57:0/3527161824 >> 172.21.15.57:0/3527161824 conn(0x7f2b640cac70 msgr2=0x7f2b640c65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:38.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.255+0000 7f2b68bb8640 1 -- 172.21.15.57:0/3527161824 shutdown_connections 2023-10-30T14:20:38.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:38.255+0000 7f2b68bb8640 1 -- 172.21.15.57:0/3527161824 wait complete. 2023-10-30T14:20:38.278 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:20:40.279 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:40.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.427+0000 7fbc9b594640 1 Processor -- start 2023-10-30T14:20:40.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.427+0000 7fbc9b594640 1 -- start start 2023-10-30T14:20:40.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.431+0000 7fbc9b594640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbc940701a0 0x7fbc9415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:40.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.431+0000 7fbc9b594640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc9406b2f0 0x7fbc9406b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:40.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.431+0000 7fbc9b594640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc940c2aa0 con 0x7fbc941537f0 2023-10-30T14:20:40.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.431+0000 7fbc9b594640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc940c3ab0 con 0x7fbc940701a0 2023-10-30T14:20:40.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.431+0000 7fbc9b594640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9415a820 con 0x7fbc9406b2f0 2023-10-30T14:20:40.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.431+0000 7fbc98b2f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbc940701a0 0x7fbc9415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:40.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.431+0000 7fbc98b2f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbc940701a0 0x7fbc9415a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55134/0 (socket says 172.21.15.57:55134) 2023-10-30T14:20:40.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.431+0000 7fbc98b2f640 1 -- 172.21.15.57:0/569091562 learned_addr learned my addr 172.21.15.57:0/569091562 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:40.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.431+0000 7fbc99b31640 1 --2- 172.21.15.57:0/569091562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc9406b2f0 0x7fbc9406b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:40.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.431+0000 7fbc98b2f640 1 -- 172.21.15.57:0/569091562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc9406b2f0 msgr2=0x7fbc9406b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:40.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.431+0000 7fbc98b2f640 1 --2- 172.21.15.57:0/569091562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc9406b2f0 0x7fbc9406b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:40.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.431+0000 7fbc98b2f640 1 -- 172.21.15.57:0/569091562 >> v1:172.21.15.88:6789/0 conn(0x7fbc941537f0 legacy=0x7fbc940ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:40.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.431+0000 7fbc98b2f640 1 -- 172.21.15.57:0/569091562 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9406bc80 con 0x7fbc940701a0 2023-10-30T14:20:40.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.431+0000 7fbc98b2f640 1 --2- 172.21.15.57:0/569091562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc940701a0 0x7fbc9415a2e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fbc84009870 tx=0x7fbc8401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be3acf91155117a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:40.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc8bfff640 1 -- 172.21.15.57:0/569091562 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc840a8a30 con 0x7fbc940701a0 2023-10-30T14:20:40.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc8bfff640 1 -- 172.21.15.57:0/569091562 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc840a8b90 con 0x7fbc940701a0 2023-10-30T14:20:40.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc8bfff640 1 -- 172.21.15.57:0/569091562 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc840b0440 con 0x7fbc940701a0 2023-10-30T14:20:40.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc9b594640 1 -- 172.21.15.57:0/569091562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc940701a0 msgr2=0x7fbc9415a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:40.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc9b594640 1 --2- 172.21.15.57:0/569091562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc940701a0 0x7fbc9415a2e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fbc84009870 tx=0x7fbc8401ed30 comp rx=0 tx=0).stop 2023-10-30T14:20:40.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc9b594640 1 -- 172.21.15.57:0/569091562 shutdown_connections 2023-10-30T14:20:40.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc9b594640 1 --2- 172.21.15.57:0/569091562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc9406b2f0 0x7fbc9406b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:40.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc9b594640 1 --2- 172.21.15.57:0/569091562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc940701a0 0x7fbc9415a2e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:40.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc9b594640 1 -- 172.21.15.57:0/569091562 >> 172.21.15.57:0/569091562 conn(0x7fbc94071fb0 msgr2=0x7fbc940723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:40.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc9b594640 1 -- 172.21.15.57:0/569091562 shutdown_connections 2023-10-30T14:20:40.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc9b594640 1 -- 172.21.15.57:0/569091562 wait complete. 2023-10-30T14:20:40.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc9b594640 1 Processor -- start 2023-10-30T14:20:40.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc9b594640 1 -- start start 2023-10-30T14:20:40.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc9b594640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc9406b2f0 0x7fbc94159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:40.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc9b594640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc940701a0 0x7fbc940c9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:40.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc9b594640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc941537f0 0x7fbc9414bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:40.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc9b594640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc94155820 con 0x7fbc9406b2f0 2023-10-30T14:20:40.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc9b594640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc941563a0 con 0x7fbc940701a0 2023-10-30T14:20:40.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc9b594640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc94152640 con 0x7fbc941537f0 2023-10-30T14:20:40.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc99330640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc9406b2f0 0x7fbc94159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:40.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc99330640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc9406b2f0 0x7fbc94159000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55140/0 (socket says 172.21.15.57:55140) 2023-10-30T14:20:40.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc99330640 1 -- 172.21.15.57:0/2664201594 learned_addr learned my addr 172.21.15.57:0/2664201594 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:40.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.435+0000 7fbc99b31640 1 --2- 172.21.15.57:0/2664201594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc941537f0 0x7fbc9414bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:40.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.439+0000 7fbc98b2f640 1 --2- 172.21.15.57:0/2664201594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc940701a0 0x7fbc940c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:40.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.439+0000 7fbc99330640 1 -- 172.21.15.57:0/2664201594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc941537f0 msgr2=0x7fbc9414bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:40.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.439+0000 7fbc99330640 1 --2- 172.21.15.57:0/2664201594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc941537f0 0x7fbc9414bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:40.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.439+0000 7fbc99330640 1 -- 172.21.15.57:0/2664201594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc940701a0 msgr2=0x7fbc940c9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:40.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.439+0000 7fbc99330640 1 --2- 172.21.15.57:0/2664201594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc940701a0 0x7fbc940c9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:40.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.439+0000 7fbc99330640 1 -- 172.21.15.57:0/2664201594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc840094d0 con 0x7fbc9406b2f0 2023-10-30T14:20:40.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.439+0000 7fbc99330640 1 --2- 172.21.15.57:0/2664201594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc9406b2f0 0x7fbc94159000 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fbc9000c420 tx=0x7fbc9000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:40.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.439+0000 7fbc89ffb640 1 -- 172.21.15.57:0/2664201594 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc900052d0 con 0x7fbc9406b2f0 2023-10-30T14:20:40.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.439+0000 7fbc89ffb640 1 -- 172.21.15.57:0/2664201594 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc90005430 con 0x7fbc9406b2f0 2023-10-30T14:20:40.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.439+0000 7fbc89ffb640 1 -- 172.21.15.57:0/2664201594 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc90007260 con 0x7fbc9406b2f0 2023-10-30T14:20:40.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.439+0000 7fbc9b594640 1 -- 172.21.15.57:0/2664201594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc94155a70 con 0x7fbc9406b2f0 2023-10-30T14:20:40.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.439+0000 7fbc9b594640 1 -- 172.21.15.57:0/2664201594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc94156030 con 0x7fbc9406b2f0 2023-10-30T14:20:40.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.439+0000 7fbc9b594640 1 -- 172.21.15.57:0/2664201594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9404e170 con 0x7fbc9406b2f0 2023-10-30T14:20:40.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.439+0000 7fbc89ffb640 1 -- 172.21.15.57:0/2664201594 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbc900073c0 con 0x7fbc9406b2f0 2023-10-30T14:20:40.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.443+0000 7fbc89ffb640 1 --2- 172.21.15.57:0/2664201594 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbc780664d0 0x7fbc78068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:40.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.443+0000 7fbc89ffb640 1 -- 172.21.15.57:0/2664201594 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fbc90083470 con 0x7fbc9406b2f0 2023-10-30T14:20:40.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.443+0000 7fbc98b2f640 1 --2- 172.21.15.57:0/2664201594 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbc780664d0 0x7fbc78068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:40.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.443+0000 7fbc98b2f640 1 --2- 172.21.15.57:0/2664201594 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbc780664d0 0x7fbc78068990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbc840b2000 tx=0x7fbc840aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:40.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.447+0000 7fbc89ffb640 1 -- 172.21.15.57:0/2664201594 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbc900506c0 con 0x7fbc9406b2f0 2023-10-30T14:20:40.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.635+0000 7fbc9b594640 1 -- 172.21.15.57:0/2664201594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc940ccb50 con 0x7fbc9406b2f0 2023-10-30T14:20:40.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.639+0000 7fbc89ffb640 1 -- 172.21.15.57:0/2664201594 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5738 (secure 0 0 0) 0x7fbc90053ce0 con 0x7fbc9406b2f0 2023-10-30T14:20:40.643 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:40.643 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:34.693643+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:40.644 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 48 2023-10-30T14:20:40.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.639+0000 7fbc9b594640 1 -- 172.21.15.57:0/2664201594 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbc780664d0 msgr2=0x7fbc78068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:40.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.639+0000 7fbc9b594640 1 --2- 172.21.15.57:0/2664201594 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbc780664d0 0x7fbc78068990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbc840b2000 tx=0x7fbc840aa070 comp rx=0 tx=0).stop 2023-10-30T14:20:40.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.639+0000 7fbc9b594640 1 -- 172.21.15.57:0/2664201594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc9406b2f0 msgr2=0x7fbc94159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:40.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.639+0000 7fbc9b594640 1 --2- 172.21.15.57:0/2664201594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc9406b2f0 0x7fbc94159000 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fbc9000c420 tx=0x7fbc9000c8e0 comp rx=0 tx=0).stop 2023-10-30T14:20:40.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.639+0000 7fbc9b594640 1 -- 172.21.15.57:0/2664201594 shutdown_connections 2023-10-30T14:20:40.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.639+0000 7fbc9b594640 1 --2- 172.21.15.57:0/2664201594 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbc780664d0 0x7fbc78068990 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:40.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.639+0000 7fbc9b594640 1 --2- 172.21.15.57:0/2664201594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbc941537f0 0x7fbc9414bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:40.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.639+0000 7fbc9b594640 1 --2- 172.21.15.57:0/2664201594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc940701a0 0x7fbc940c9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:40.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.639+0000 7fbc9b594640 1 --2- 172.21.15.57:0/2664201594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc9406b2f0 0x7fbc94159000 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:40.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.639+0000 7fbc9b594640 1 -- 172.21.15.57:0/2664201594 >> 172.21.15.57:0/2664201594 conn(0x7fbc94071fb0 msgr2=0x7fbc9406c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:40.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.639+0000 7fbc9b594640 1 -- 172.21.15.57:0/2664201594 shutdown_connections 2023-10-30T14:20:40.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:40.639+0000 7fbc9b594640 1 -- 172.21.15.57:0/2664201594 wait complete. 2023-10-30T14:20:40.664 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:20:42.666 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:42.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0f73d3640 1 Processor -- start 2023-10-30T14:20:42.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0f73d3640 1 -- start start 2023-10-30T14:20:42.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0f73d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f00c4a10 0x7fc0f015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:42.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0f73d3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc0f00c42a0 0x7fc0f00ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:42.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0f73d3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f0159f20 con 0x7fc0f0153c80 2023-10-30T14:20:42.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0f73d3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f015dec0 con 0x7fc0f00c42a0 2023-10-30T14:20:42.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0f73d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f01599a0 con 0x7fc0f00c4a10 2023-10-30T14:20:42.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0f516f640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc0f0153c80 0x7fc0f015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:47716/0 (socket says 172.21.15.57:47716) 2023-10-30T14:20:42.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0f516f640 1 -- 172.21.15.57:0/3592745403 learned_addr learned my addr 172.21.15.57:0/3592745403 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:20:42.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0f496e640 1 --2- 172.21.15.57:0/3592745403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f00c4a10 0x7fc0f015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:42.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0f5970640 1 --2- 172.21.15.57:0/3592745403 >> v2:172.21.15.88:3300/0 conn(0x7fc0f00c42a0 0x7fc0f00ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:42.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0e7fff640 1 -- 172.21.15.57:0/3592745403 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816904795 0 0) 0x7fc0f0159f20 con 0x7fc0f0153c80 2023-10-30T14:20:42.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0e7fff640 1 -- 172.21.15.57:0/3592745403 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0dc003530 con 0x7fc0f0153c80 2023-10-30T14:20:42.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0e7fff640 1 -- 172.21.15.57:0/3592745403 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2741633801 0 0) 0x7fc0dc003530 con 0x7fc0f0153c80 2023-10-30T14:20:42.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0e7fff640 1 -- 172.21.15.57:0/3592745403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f00c4a10 msgr2=0x7fc0f015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:42.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0e7fff640 1 --2- 172.21.15.57:0/3592745403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f00c4a10 0x7fc0f015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:42.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0e7fff640 1 -- 172.21.15.57:0/3592745403 >> v2:172.21.15.88:3300/0 conn(0x7fc0f00c42a0 msgr2=0x7fc0f00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:42.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0e7fff640 1 --2- 172.21.15.57:0/3592745403 >> v2:172.21.15.88:3300/0 conn(0x7fc0f00c42a0 0x7fc0f00ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:42.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0e7fff640 1 -- 172.21.15.57:0/3592745403 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f015d620 con 0x7fc0f0153c80 2023-10-30T14:20:42.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0e7fff640 1 -- 172.21.15.57:0/3592745403 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fc0ec001680 con 0x7fc0f0153c80 2023-10-30T14:20:42.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0e7fff640 1 -- 172.21.15.57:0/3592745403 >> v1:172.21.15.57:6789/0 conn(0x7fc0f0153c80 legacy=0x7fc0f015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:42.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0e7fff640 1 --2- 172.21.15.57:0/3592745403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0dc003980 0x7fc0dc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:42.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0e7fff640 1 --2- 172.21.15.57:0/3592745403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f00c4a10 0x7fc0dc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:42.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0e7fff640 1 --2- 172.21.15.57:0/3592745403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0dc004d10 0x7fc0dc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:42.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0e7fff640 1 -- 172.21.15.57:0/3592745403 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f015dec0 con 0x7fc0dc003980 2023-10-30T14:20:42.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0e7fff640 1 -- 172.21.15.57:0/3592745403 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f01599a0 con 0x7fc0dc004d10 2023-10-30T14:20:42.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.815+0000 7fc0e7fff640 1 -- 172.21.15.57:0/3592745403 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc00b990 con 0x7fc0f00c4a10 2023-10-30T14:20:42.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.819+0000 7fc0f516f640 1 --2- 172.21.15.57:0/3592745403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0dc003980 0x7fc0dc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:42.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.819+0000 7fc0f496e640 1 --2- 172.21.15.57:0/3592745403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f00c4a10 0x7fc0dc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:42.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.819+0000 7fc0f5970640 1 --2- 172.21.15.57:0/3592745403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0dc004d10 0x7fc0dc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:42.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.819+0000 7fc0f516f640 1 -- 172.21.15.57:0/3592745403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f00c4a10 msgr2=0x7fc0dc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:42.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.819+0000 7fc0f516f640 1 --2- 172.21.15.57:0/3592745403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f00c4a10 0x7fc0dc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:42.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.819+0000 7fc0f516f640 1 -- 172.21.15.57:0/3592745403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0dc004d10 msgr2=0x7fc0dc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:42.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.819+0000 7fc0f516f640 1 --2- 172.21.15.57:0/3592745403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0dc004d10 0x7fc0dc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:42.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.819+0000 7fc0f516f640 1 -- 172.21.15.57:0/3592745403 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0dc003530 con 0x7fc0dc003980 2023-10-30T14:20:42.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.819+0000 7fc0f496e640 1 --2- 172.21.15.57:0/3592745403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f00c4a10 0x7fc0dc0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:42.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.819+0000 7fc0f516f640 1 --2- 172.21.15.57:0/3592745403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0dc003980 0x7fc0dc003da0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec00c240 tx=0x7fc0ec01f3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5cfb683387388bb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:42.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.819+0000 7fc0e7fff640 1 -- 172.21.15.57:0/3592745403 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec005ae0 con 0x7fc0dc003980 2023-10-30T14:20:42.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.819+0000 7fc0e7fff640 1 -- 172.21.15.57:0/3592745403 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ec0ab070 con 0x7fc0dc003980 2023-10-30T14:20:42.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.819+0000 7fc0e7fff640 1 -- 172.21.15.57:0/3592745403 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec005ea0 con 0x7fc0dc003980 2023-10-30T14:20:42.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f73d3640 1 -- 172.21.15.57:0/3592745403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0dc003980 msgr2=0x7fc0dc003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:42.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f73d3640 1 --2- 172.21.15.57:0/3592745403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0dc003980 0x7fc0dc003da0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec00c240 tx=0x7fc0ec01f3d0 comp rx=0 tx=0).stop 2023-10-30T14:20:42.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f5970640 1 -- 172.21.15.57:0/3592745403 reap_dead start 2023-10-30T14:20:42.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f73d3640 1 -- 172.21.15.57:0/3592745403 shutdown_connections 2023-10-30T14:20:42.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f73d3640 1 -- 172.21.15.57:0/3592745403 >> 172.21.15.57:0/3592745403 conn(0x7fc0f00715c0 msgr2=0x7fc0f00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:42.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f73d3640 1 -- 172.21.15.57:0/3592745403 shutdown_connections 2023-10-30T14:20:42.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f73d3640 1 -- 172.21.15.57:0/3592745403 wait complete. 2023-10-30T14:20:42.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f73d3640 1 Processor -- start 2023-10-30T14:20:42.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f73d3640 1 -- start start 2023-10-30T14:20:42.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f73d3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0dc003980 0x7fc0f0147c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:42.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f73d3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f00b4fa0 0x7fc0f0145b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:42.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f73d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f01460a0 0x7fc0f015bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:42.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f73d3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f00b3e40 con 0x7fc0f00b4fa0 2023-10-30T14:20:42.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f73d3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f01534d0 con 0x7fc0dc003980 2023-10-30T14:20:42.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f73d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f015b710 con 0x7fc0f01460a0 2023-10-30T14:20:42.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f496e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f00b4fa0 0x7fc0f0145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:42.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f496e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f00b4fa0 0x7fc0f0145b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55154/0 (socket says 172.21.15.57:55154) 2023-10-30T14:20:42.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f496e640 1 -- 172.21.15.57:0/1337537139 learned_addr learned my addr 172.21.15.57:0/1337537139 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:42.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f5970640 1 --2- 172.21.15.57:0/1337537139 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f01460a0 0x7fc0f015bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:42.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f516f640 1 --2- 172.21.15.57:0/1337537139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0dc003980 0x7fc0f0147c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:42.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f496e640 1 -- 172.21.15.57:0/1337537139 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f01460a0 msgr2=0x7fc0f015bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:42.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f496e640 1 --2- 172.21.15.57:0/1337537139 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f01460a0 0x7fc0f015bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:42.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f496e640 1 -- 172.21.15.57:0/1337537139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0dc003980 msgr2=0x7fc0f0147c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:42.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f496e640 1 --2- 172.21.15.57:0/1337537139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0dc003980 0x7fc0f0147c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:42.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f496e640 1 -- 172.21.15.57:0/1337537139 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f0153230 con 0x7fc0f00b4fa0 2023-10-30T14:20:42.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.823+0000 7fc0f5970640 1 --2- 172.21.15.57:0/1337537139 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f01460a0 0x7fc0f015bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:42.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.827+0000 7fc0f496e640 1 --2- 172.21.15.57:0/1337537139 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f00b4fa0 0x7fc0f0145b60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00115d0 tx=0x7fc0e0011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:42.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.827+0000 7fc0e5ffb640 1 -- 172.21.15.57:0/1337537139 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0004220 con 0x7fc0f00b4fa0 2023-10-30T14:20:42.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.827+0000 7fc0e5ffb640 1 -- 172.21.15.57:0/1337537139 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e0004380 con 0x7fc0f00b4fa0 2023-10-30T14:20:42.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.827+0000 7fc0e5ffb640 1 -- 172.21.15.57:0/1337537139 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e000f8f0 con 0x7fc0f00b4fa0 2023-10-30T14:20:42.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.827+0000 7fc0f73d3640 1 -- 172.21.15.57:0/1337537139 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f015ab60 con 0x7fc0f00b4fa0 2023-10-30T14:20:42.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.827+0000 7fc0f73d3640 1 -- 172.21.15.57:0/1337537139 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f01f2fa0 con 0x7fc0f00b4fa0 2023-10-30T14:20:42.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.827+0000 7fc0f73d3640 1 -- 172.21.15.57:0/1337537139 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f0146480 con 0x7fc0f00b4fa0 2023-10-30T14:20:42.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.827+0000 7fc0e5ffb640 1 -- 172.21.15.57:0/1337537139 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc0e000fa50 con 0x7fc0f00b4fa0 2023-10-30T14:20:42.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.827+0000 7fc0e5ffb640 1 --2- 172.21.15.57:0/1337537139 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0d40664d0 0x7fc0d4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:42.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.827+0000 7fc0e5ffb640 1 -- 172.21.15.57:0/1337537139 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fc0e00845b0 con 0x7fc0f00b4fa0 2023-10-30T14:20:42.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.831+0000 7fc0f516f640 1 --2- 172.21.15.57:0/1337537139 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0d40664d0 0x7fc0d4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:42.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.831+0000 7fc0f516f640 1 --2- 172.21.15.57:0/1337537139 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0d40664d0 0x7fc0d4068990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0ae000 tx=0x7fc0ec001a80 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:42.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:42.831+0000 7fc0e5ffb640 1 -- 172.21.15.57:0/1337537139 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc0f0146480 con 0x7fc0f00b4fa0 2023-10-30T14:20:43.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:43.023+0000 7fc0f73d3640 1 -- 172.21.15.57:0/1337537139 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0f0146480 con 0x7fc0f00b4fa0 2023-10-30T14:20:43.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:43.023+0000 7fc0e5ffb640 1 -- 172.21.15.57:0/1337537139 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5740 (secure 0 0 0) 0x7fc0f0146480 con 0x7fc0f00b4fa0 2023-10-30T14:20:43.030 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:43.031 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:40.044663+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:resolve","state_seq":20,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:43.031 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 49 2023-10-30T14:20:43.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:43.027+0000 7fc0f73d3640 1 -- 172.21.15.57:0/1337537139 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0d40664d0 msgr2=0x7fc0d4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:43.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:43.027+0000 7fc0f73d3640 1 --2- 172.21.15.57:0/1337537139 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0d40664d0 0x7fc0d4068990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0ae000 tx=0x7fc0ec001a80 comp rx=0 tx=0).stop 2023-10-30T14:20:43.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:43.027+0000 7fc0f73d3640 1 -- 172.21.15.57:0/1337537139 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f00b4fa0 msgr2=0x7fc0f0145b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:43.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:43.027+0000 7fc0f73d3640 1 --2- 172.21.15.57:0/1337537139 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f00b4fa0 0x7fc0f0145b60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00115d0 tx=0x7fc0e0011a90 comp rx=0 tx=0).stop 2023-10-30T14:20:43.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:43.027+0000 7fc0f73d3640 1 -- 172.21.15.57:0/1337537139 shutdown_connections 2023-10-30T14:20:43.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:43.027+0000 7fc0f73d3640 1 --2- 172.21.15.57:0/1337537139 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0d40664d0 0x7fc0d4068990 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:43.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:43.027+0000 7fc0f73d3640 1 --2- 172.21.15.57:0/1337537139 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f01460a0 0x7fc0f015bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:43.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:43.027+0000 7fc0f73d3640 1 --2- 172.21.15.57:0/1337537139 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f00b4fa0 0x7fc0f0145b60 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:43.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:43.027+0000 7fc0f73d3640 1 --2- 172.21.15.57:0/1337537139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0dc003980 0x7fc0f0147c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:43.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:43.027+0000 7fc0f73d3640 1 -- 172.21.15.57:0/1337537139 >> 172.21.15.57:0/1337537139 conn(0x7fc0f00715c0 msgr2=0x7fc0f0148970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:43.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:43.027+0000 7fc0f73d3640 1 -- 172.21.15.57:0/1337537139 shutdown_connections 2023-10-30T14:20:43.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:43.027+0000 7fc0f73d3640 1 -- 172.21.15.57:0/1337537139 wait complete. 2023-10-30T14:20:43.051 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:20:45.052 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:45.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbfcdf5640 1 Processor -- start 2023-10-30T14:20:45.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbfcdf5640 1 -- start start 2023-10-30T14:20:45.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbfcdf5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbf8153c80 0x7fdbf815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:45.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbfcdf5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf8159f20 con 0x7fdbf80c42a0 2023-10-30T14:20:45.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbfcdf5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf815d620 con 0x7fdbf80c4a10 2023-10-30T14:20:45.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbfcdf5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf815dec0 con 0x7fdbf8153c80 2023-10-30T14:20:45.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf6d9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fdbf80c42a0 0x7fdbf80ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:39676/0 (socket says 172.21.15.57:39676) 2023-10-30T14:20:45.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf6d9d640 1 -- 172.21.15.57:0/825995892 learned_addr learned my addr 172.21.15.57:0/825995892 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:20:45.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf659c640 1 --2- 172.21.15.57:0/825995892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbf8153c80 0x7fdbf815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:45.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf5d9b640 1 -- 172.21.15.57:0/825995892 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193637354 0 0) 0x7fdbf8159f20 con 0x7fdbf80c42a0 2023-10-30T14:20:45.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf5d9b640 1 -- 172.21.15.57:0/825995892 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbe4003530 con 0x7fdbf80c42a0 2023-10-30T14:20:45.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf5d9b640 1 -- 172.21.15.57:0/825995892 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 370546720 0 0) 0x7fdbe4003530 con 0x7fdbf80c42a0 2023-10-30T14:20:45.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf5d9b640 1 -- 172.21.15.57:0/825995892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbf8153c80 msgr2=0x7fdbf815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:45.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf5d9b640 1 --2- 172.21.15.57:0/825995892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbf8153c80 0x7fdbf815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:45.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf5d9b640 1 -- 172.21.15.57:0/825995892 >> v1:172.21.15.88:6789/0 conn(0x7fdbf80c4a10 legacy=0x7fdbf815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:45.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf5d9b640 1 -- 172.21.15.57:0/825995892 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf815d900 con 0x7fdbf80c42a0 2023-10-30T14:20:45.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf5d9b640 1 -- 172.21.15.57:0/825995892 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fdbf0001a00 con 0x7fdbf80c42a0 2023-10-30T14:20:45.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf5d9b640 1 -- 172.21.15.57:0/825995892 >> v1:172.21.15.57:6789/0 conn(0x7fdbf80c42a0 legacy=0x7fdbf80ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:45.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf5d9b640 1 --2- 172.21.15.57:0/825995892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbe40039b0 0x7fdbe4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:45.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf5d9b640 1 --2- 172.21.15.57:0/825995892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbf8153c80 0x7fdbe40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:45.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf5d9b640 1 --2- 172.21.15.57:0/825995892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbe4004d10 0x7fdbe4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:45.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf5d9b640 1 -- 172.21.15.57:0/825995892 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf815dec0 con 0x7fdbe4004d10 2023-10-30T14:20:45.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf5d9b640 1 -- 172.21.15.57:0/825995892 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe400b990 con 0x7fdbe40039b0 2023-10-30T14:20:45.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf5d9b640 1 -- 172.21.15.57:0/825995892 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe400bb50 con 0x7fdbf8153c80 2023-10-30T14:20:45.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf6d9d640 1 --2- 172.21.15.57:0/825995892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbe4004d10 0x7fdbe4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:45.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbf659c640 1 --2- 172.21.15.57:0/825995892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbe40039b0 0x7fdbe4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:45.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.203+0000 7fdbeffff640 1 --2- 172.21.15.57:0/825995892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbf8153c80 0x7fdbe40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:45.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.207+0000 7fdbf6d9d640 1 -- 172.21.15.57:0/825995892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbf8153c80 msgr2=0x7fdbe40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:45.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.207+0000 7fdbf6d9d640 1 --2- 172.21.15.57:0/825995892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbf8153c80 0x7fdbe40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:45.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.207+0000 7fdbf6d9d640 1 -- 172.21.15.57:0/825995892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbe40039b0 msgr2=0x7fdbe4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:45.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.207+0000 7fdbf6d9d640 1 --2- 172.21.15.57:0/825995892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbe40039b0 0x7fdbe4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:45.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.207+0000 7fdbf6d9d640 1 -- 172.21.15.57:0/825995892 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe4003530 con 0x7fdbe4004d10 2023-10-30T14:20:45.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.207+0000 7fdbf6d9d640 1 --2- 172.21.15.57:0/825995892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbe4004d10 0x7fdbe4005190 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000a7d0 tx=0x7fdbf001d2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570baf702428a7e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:45.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.207+0000 7fdbf5d9b640 1 -- 172.21.15.57:0/825995892 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf00a5ea0 con 0x7fdbe4004d10 2023-10-30T14:20:45.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.207+0000 7fdbf5d9b640 1 -- 172.21.15.57:0/825995892 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf00a8070 con 0x7fdbe4004d10 2023-10-30T14:20:45.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.207+0000 7fdbf5d9b640 1 -- 172.21.15.57:0/825995892 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf00ad890 con 0x7fdbe4004d10 2023-10-30T14:20:45.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.207+0000 7fdbfcdf5640 1 -- 172.21.15.57:0/825995892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbe4004d10 msgr2=0x7fdbe4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:45.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.207+0000 7fdbfcdf5640 1 --2- 172.21.15.57:0/825995892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbe4004d10 0x7fdbe4005190 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000a7d0 tx=0x7fdbf001d2c0 comp rx=0 tx=0).stop 2023-10-30T14:20:45.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.207+0000 7fdbf6d9d640 1 -- 172.21.15.57:0/825995892 reap_dead start 2023-10-30T14:20:45.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.207+0000 7fdbfcdf5640 1 -- 172.21.15.57:0/825995892 shutdown_connections 2023-10-30T14:20:45.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.207+0000 7fdbfcdf5640 1 -- 172.21.15.57:0/825995892 >> 172.21.15.57:0/825995892 conn(0x7fdbf80715c0 msgr2=0x7fdbf80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:45.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.207+0000 7fdbfcdf5640 1 -- 172.21.15.57:0/825995892 shutdown_connections 2023-10-30T14:20:45.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.207+0000 7fdbfcdf5640 1 -- 172.21.15.57:0/825995892 wait complete. 2023-10-30T14:20:45.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbfcdf5640 1 Processor -- start 2023-10-30T14:20:45.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbfcdf5640 1 -- start start 2023-10-30T14:20:45.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbfcdf5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbe4004d10 0x7fdbf80b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:45.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbfcdf5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbf814d840 0x7fdbf80b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:45.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbfcdf5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbf80b4b90 0x7fdbf80cf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:45.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbfcdf5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf80b2c60 con 0x7fdbf80b4b90 2023-10-30T14:20:45.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbfcdf5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf80b4820 con 0x7fdbf814d840 2023-10-30T14:20:45.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbfcdf5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf80b35c0 con 0x7fdbe4004d10 2023-10-30T14:20:45.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbf6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbf80b4b90 0x7fdbf80cf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:45.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbf6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbf80b4b90 0x7fdbf80cf4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47878/0 (socket says 172.21.15.57:47878) 2023-10-30T14:20:45.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbf6d9d640 1 -- 172.21.15.57:0/608950523 learned_addr learned my addr 172.21.15.57:0/608950523 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:45.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbf659c640 1 --2- 172.21.15.57:0/608950523 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbe4004d10 0x7fdbf80b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:45.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbeffff640 1 --2- 172.21.15.57:0/608950523 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbf814d840 0x7fdbf80b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:45.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbf6d9d640 1 -- 172.21.15.57:0/608950523 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbe4004d10 msgr2=0x7fdbf80b42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:45.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbf6d9d640 1 --2- 172.21.15.57:0/608950523 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbe4004d10 0x7fdbf80b42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:45.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbf6d9d640 1 -- 172.21.15.57:0/608950523 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbf814d840 msgr2=0x7fdbf80b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:45.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbf6d9d640 1 --2- 172.21.15.57:0/608950523 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbf814d840 0x7fdbf80b3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:45.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbf6d9d640 1 -- 172.21.15.57:0/608950523 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf81eea70 con 0x7fdbf80b4b90 2023-10-30T14:20:45.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbeffff640 1 --2- 172.21.15.57:0/608950523 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbf814d840 0x7fdbf80b3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:45.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbf6d9d640 1 --2- 172.21.15.57:0/608950523 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbf80b4b90 0x7fdbf80cf4f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fdbf001d0f0 tx=0x7fdbe4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:45.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.211+0000 7fdbeeffd640 1 -- 172.21.15.57:0/608950523 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf00bb070 con 0x7fdbf80b4b90 2023-10-30T14:20:45.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.215+0000 7fdbeeffd640 1 -- 172.21.15.57:0/608950523 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf00a5ea0 con 0x7fdbf80b4b90 2023-10-30T14:20:45.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.215+0000 7fdbfcdf5640 1 -- 172.21.15.57:0/608950523 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf81eed10 con 0x7fdbf80b4b90 2023-10-30T14:20:45.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.215+0000 7fdbeeffd640 1 -- 172.21.15.57:0/608950523 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf00019c0 con 0x7fdbf80b4b90 2023-10-30T14:20:45.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.215+0000 7fdbfcdf5640 1 -- 172.21.15.57:0/608950523 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf80b22c0 con 0x7fdbf80b4b90 2023-10-30T14:20:45.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.215+0000 7fdbfcdf5640 1 -- 172.21.15.57:0/608950523 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf80b2da0 con 0x7fdbf80b4b90 2023-10-30T14:20:45.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.215+0000 7fdbeeffd640 1 -- 172.21.15.57:0/608950523 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdbf00aa030 con 0x7fdbf80b4b90 2023-10-30T14:20:45.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.215+0000 7fdbeeffd640 1 --2- 172.21.15.57:0/608950523 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdbd80664d0 0x7fdbd8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:45.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.215+0000 7fdbeeffd640 1 -- 172.21.15.57:0/608950523 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fdbf011e170 con 0x7fdbf80b4b90 2023-10-30T14:20:45.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.215+0000 7fdbf659c640 1 --2- 172.21.15.57:0/608950523 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdbd80664d0 0x7fdbd8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:45.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.219+0000 7fdbf659c640 1 --2- 172.21.15.57:0/608950523 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdbd80664d0 0x7fdbd8068990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8009be0 tx=0x7fdbe8009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:45.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.219+0000 7fdbeeffd640 1 -- 172.21.15.57:0/608950523 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdbf80b2da0 con 0x7fdbf80b4b90 2023-10-30T14:20:45.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.411+0000 7fdbfcdf5640 1 -- 172.21.15.57:0/608950523 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbf80b2da0 con 0x7fdbf80b4b90 2023-10-30T14:20:45.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.411+0000 7fdbeeffd640 1 -- 172.21.15.57:0/608950523 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5740 (secure 0 0 0) 0x7fdbf80b2da0 con 0x7fdbf80b4b90 2023-10-30T14:20:45.416 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:45.417 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:40.044663+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:resolve","state_seq":20,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:45.417 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 49 2023-10-30T14:20:45.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.411+0000 7fdbfcdf5640 1 -- 172.21.15.57:0/608950523 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdbd80664d0 msgr2=0x7fdbd8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:45.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.411+0000 7fdbfcdf5640 1 --2- 172.21.15.57:0/608950523 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdbd80664d0 0x7fdbd8068990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8009be0 tx=0x7fdbe8009190 comp rx=0 tx=0).stop 2023-10-30T14:20:45.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.411+0000 7fdbfcdf5640 1 -- 172.21.15.57:0/608950523 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbf80b4b90 msgr2=0x7fdbf80cf4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:45.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.411+0000 7fdbfcdf5640 1 --2- 172.21.15.57:0/608950523 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbf80b4b90 0x7fdbf80cf4f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fdbf001d0f0 tx=0x7fdbe4005800 comp rx=0 tx=0).stop 2023-10-30T14:20:45.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.411+0000 7fdbfcdf5640 1 -- 172.21.15.57:0/608950523 shutdown_connections 2023-10-30T14:20:45.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.411+0000 7fdbfcdf5640 1 --2- 172.21.15.57:0/608950523 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdbd80664d0 0x7fdbd8068990 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:45.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.411+0000 7fdbfcdf5640 1 --2- 172.21.15.57:0/608950523 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbf80b4b90 0x7fdbf80cf4f0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:45.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.411+0000 7fdbfcdf5640 1 --2- 172.21.15.57:0/608950523 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbf814d840 0x7fdbf80b3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:45.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.411+0000 7fdbfcdf5640 1 --2- 172.21.15.57:0/608950523 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbe4004d10 0x7fdbf80b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:45.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.411+0000 7fdbfcdf5640 1 -- 172.21.15.57:0/608950523 >> 172.21.15.57:0/608950523 conn(0x7fdbf80715c0 msgr2=0x7fdbf8151c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:45.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.411+0000 7fdbfcdf5640 1 -- 172.21.15.57:0/608950523 shutdown_connections 2023-10-30T14:20:45.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:45.411+0000 7fdbfcdf5640 1 -- 172.21.15.57:0/608950523 wait complete. 2023-10-30T14:20:45.438 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:20:47.439 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:47.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.587+0000 7fdd4ad1d640 1 Processor -- start 2023-10-30T14:20:47.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.587+0000 7fdd4ad1d640 1 -- start start 2023-10-30T14:20:47.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.587+0000 7fdd4ad1d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd4406bb40 0x7fdd44157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:47.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.587+0000 7fdd4ad1d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdd4406c420 0x7fdd44157f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:47.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.587+0000 7fdd4ad1d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd44071e00 con 0x7fdd4406b760 2023-10-30T14:20:47.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.587+0000 7fdd4ad1d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd44070020 con 0x7fdd4406c420 2023-10-30T14:20:47.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.587+0000 7fdd4ad1d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd44070f10 con 0x7fdd4406bb40 2023-10-30T14:20:47.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.587+0000 7fdd48ab9640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fdd4406b760 0x7fdd4415e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:39690/0 (socket says 172.21.15.57:39690) 2023-10-30T14:20:47.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.587+0000 7fdd48ab9640 1 -- 172.21.15.57:0/2524904241 learned_addr learned my addr 172.21.15.57:0/2524904241 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:20:47.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.587+0000 7fdd43fff640 1 --2- 172.21.15.57:0/2524904241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd4406bb40 0x7fdd44157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:47.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.587+0000 7fdd492ba640 1 --2- 172.21.15.57:0/2524904241 >> v2:172.21.15.88:3300/0 conn(0x7fdd4406c420 0x7fdd44157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:47.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd437fe640 1 -- 172.21.15.57:0/2524904241 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242758252 0 0) 0x7fdd44071e00 con 0x7fdd4406b760 2023-10-30T14:20:47.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd437fe640 1 -- 172.21.15.57:0/2524904241 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd34003530 con 0x7fdd4406b760 2023-10-30T14:20:47.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd437fe640 1 -- 172.21.15.57:0/2524904241 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3528622315 0 0) 0x7fdd34003530 con 0x7fdd4406b760 2023-10-30T14:20:47.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd437fe640 1 -- 172.21.15.57:0/2524904241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd4406bb40 msgr2=0x7fdd44157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:47.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd437fe640 1 --2- 172.21.15.57:0/2524904241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd4406bb40 0x7fdd44157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:47.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd437fe640 1 -- 172.21.15.57:0/2524904241 >> v2:172.21.15.88:3300/0 conn(0x7fdd4406c420 msgr2=0x7fdd44157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:47.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd437fe640 1 --2- 172.21.15.57:0/2524904241 >> v2:172.21.15.88:3300/0 conn(0x7fdd4406c420 0x7fdd44157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:47.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd437fe640 1 -- 172.21.15.57:0/2524904241 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd440711f0 con 0x7fdd4406b760 2023-10-30T14:20:47.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd437fe640 1 -- 172.21.15.57:0/2524904241 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fdd38001680 con 0x7fdd4406b760 2023-10-30T14:20:47.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd437fe640 1 -- 172.21.15.57:0/2524904241 >> v1:172.21.15.57:6789/0 conn(0x7fdd4406b760 legacy=0x7fdd4415e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:47.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd437fe640 1 --2- 172.21.15.57:0/2524904241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd4406bb40 0x7fdd34003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:47.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd437fe640 1 --2- 172.21.15.57:0/2524904241 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd34003f00 0x7fdd340047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:47.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd437fe640 1 --2- 172.21.15.57:0/2524904241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd34004d10 0x7fdd34005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:47.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd437fe640 1 -- 172.21.15.57:0/2524904241 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd44070020 con 0x7fdd34003f00 2023-10-30T14:20:47.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd437fe640 1 -- 172.21.15.57:0/2524904241 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd44070f10 con 0x7fdd34004d10 2023-10-30T14:20:47.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd437fe640 1 -- 172.21.15.57:0/2524904241 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3400b990 con 0x7fdd4406bb40 2023-10-30T14:20:47.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd43fff640 1 --2- 172.21.15.57:0/2524904241 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd34003f00 0x7fdd340047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:47.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd48ab9640 1 --2- 172.21.15.57:0/2524904241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd4406bb40 0x7fdd34003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:47.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd492ba640 1 --2- 172.21.15.57:0/2524904241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd34004d10 0x7fdd34005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:47.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd43fff640 1 -- 172.21.15.57:0/2524904241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd4406bb40 msgr2=0x7fdd34003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:47.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd43fff640 1 --2- 172.21.15.57:0/2524904241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd4406bb40 0x7fdd34003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:47.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd43fff640 1 -- 172.21.15.57:0/2524904241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd34004d10 msgr2=0x7fdd34005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:47.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd43fff640 1 --2- 172.21.15.57:0/2524904241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd34004d10 0x7fdd34005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:47.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd43fff640 1 -- 172.21.15.57:0/2524904241 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd34003530 con 0x7fdd34003f00 2023-10-30T14:20:47.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.591+0000 7fdd43fff640 1 --2- 172.21.15.57:0/2524904241 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd34003f00 0x7fdd340047d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009ab0 tx=0x7fdd3001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a86c4cf71ab35e59 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:47.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.595+0000 7fdd437fe640 1 -- 172.21.15.57:0/2524904241 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd300a5510 con 0x7fdd34003f00 2023-10-30T14:20:47.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.595+0000 7fdd437fe640 1 -- 172.21.15.57:0/2524904241 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd300a5670 con 0x7fdd34003f00 2023-10-30T14:20:47.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.595+0000 7fdd437fe640 1 -- 172.21.15.57:0/2524904241 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd300a5a20 con 0x7fdd34003f00 2023-10-30T14:20:47.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.595+0000 7fdd4ad1d640 1 -- 172.21.15.57:0/2524904241 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd34003f00 msgr2=0x7fdd340047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:47.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.595+0000 7fdd4ad1d640 1 --2- 172.21.15.57:0/2524904241 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd34003f00 0x7fdd340047d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009ab0 tx=0x7fdd3001c9d0 comp rx=0 tx=0).stop 2023-10-30T14:20:47.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.595+0000 7fdd492ba640 1 -- 172.21.15.57:0/2524904241 reap_dead start 2023-10-30T14:20:47.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.595+0000 7fdd4ad1d640 1 -- 172.21.15.57:0/2524904241 shutdown_connections 2023-10-30T14:20:47.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.595+0000 7fdd4ad1d640 1 -- 172.21.15.57:0/2524904241 >> 172.21.15.57:0/2524904241 conn(0x7fdd440cac70 msgr2=0x7fdd440cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:47.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.595+0000 7fdd4ad1d640 1 -- 172.21.15.57:0/2524904241 shutdown_connections 2023-10-30T14:20:47.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.595+0000 7fdd4ad1d640 1 -- 172.21.15.57:0/2524904241 wait complete. 2023-10-30T14:20:47.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.595+0000 7fdd4ad1d640 1 Processor -- start 2023-10-30T14:20:47.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.595+0000 7fdd4ad1d640 1 -- start start 2023-10-30T14:20:47.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.595+0000 7fdd4ad1d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd34003f00 0x7fdd441f3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:47.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.595+0000 7fdd4ad1d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd441f35f0 0x7fdd441557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:47.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.595+0000 7fdd4ad1d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd4414fe50 0x7fdd441f5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:47.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.595+0000 7fdd4ad1d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd441f6dc0 con 0x7fdd441f35f0 2023-10-30T14:20:47.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.595+0000 7fdd4ad1d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd441f4320 con 0x7fdd4414fe50 2023-10-30T14:20:47.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.595+0000 7fdd4ad1d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd441f6800 con 0x7fdd34003f00 2023-10-30T14:20:47.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.599+0000 7fdd43fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd441f35f0 0x7fdd441557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:47.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.599+0000 7fdd43fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd441f35f0 0x7fdd441557e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47896/0 (socket says 172.21.15.57:47896) 2023-10-30T14:20:47.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.599+0000 7fdd43fff640 1 -- 172.21.15.57:0/2144861343 learned_addr learned my addr 172.21.15.57:0/2144861343 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:47.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.599+0000 7fdd48ab9640 1 --2- 172.21.15.57:0/2144861343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd34003f00 0x7fdd441f3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:47.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.599+0000 7fdd492ba640 1 --2- 172.21.15.57:0/2144861343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd4414fe50 0x7fdd441f5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:47.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.599+0000 7fdd43fff640 1 -- 172.21.15.57:0/2144861343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd34003f00 msgr2=0x7fdd441f3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:47.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.599+0000 7fdd43fff640 1 --2- 172.21.15.57:0/2144861343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd34003f00 0x7fdd441f3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:47.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.599+0000 7fdd43fff640 1 -- 172.21.15.57:0/2144861343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd4414fe50 msgr2=0x7fdd441f5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:47.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.599+0000 7fdd43fff640 1 --2- 172.21.15.57:0/2144861343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd4414fe50 0x7fdd441f5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:47.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.599+0000 7fdd43fff640 1 -- 172.21.15.57:0/2144861343 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd441f6ae0 con 0x7fdd441f35f0 2023-10-30T14:20:47.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.599+0000 7fdd492ba640 1 --2- 172.21.15.57:0/2144861343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd4414fe50 0x7fdd441f5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:47.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.599+0000 7fdd43fff640 1 --2- 172.21.15.57:0/2144861343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd441f35f0 0x7fdd441557e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fdd3001d270 tx=0x7fdd34005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:47.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.599+0000 7fdd417fa640 1 -- 172.21.15.57:0/2144861343 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd300bc070 con 0x7fdd441f35f0 2023-10-30T14:20:47.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.599+0000 7fdd417fa640 1 -- 172.21.15.57:0/2144861343 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd300b8440 con 0x7fdd441f35f0 2023-10-30T14:20:47.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.599+0000 7fdd417fa640 1 -- 172.21.15.57:0/2144861343 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd300b87b0 con 0x7fdd441f35f0 2023-10-30T14:20:47.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.599+0000 7fdd4ad1d640 1 -- 172.21.15.57:0/2144861343 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd441f2b10 con 0x7fdd441f35f0 2023-10-30T14:20:47.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.599+0000 7fdd4ad1d640 1 -- 172.21.15.57:0/2144861343 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd441f30d0 con 0x7fdd441f35f0 2023-10-30T14:20:47.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.599+0000 7fdd4ad1d640 1 -- 172.21.15.57:0/2144861343 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd441f2970 con 0x7fdd441f35f0 2023-10-30T14:20:47.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.599+0000 7fdd417fa640 1 -- 172.21.15.57:0/2144861343 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdd300b4040 con 0x7fdd441f35f0 2023-10-30T14:20:47.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.603+0000 7fdd417fa640 1 --2- 172.21.15.57:0/2144861343 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdd280664d0 0x7fdd28068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:47.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.603+0000 7fdd417fa640 1 -- 172.21.15.57:0/2144861343 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fdd3011df40 con 0x7fdd441f35f0 2023-10-30T14:20:47.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.603+0000 7fdd48ab9640 1 --2- 172.21.15.57:0/2144861343 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdd280664d0 0x7fdd28068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:47.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.603+0000 7fdd48ab9640 1 --2- 172.21.15.57:0/2144861343 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdd280664d0 0x7fdd28068990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdd380054a0 tx=0x7fdd380014c0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:47.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.603+0000 7fdd417fa640 1 -- 172.21.15.57:0/2144861343 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdd441f2970 con 0x7fdd441f35f0 2023-10-30T14:20:47.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.795+0000 7fdd4ad1d640 1 -- 172.21.15.57:0/2144861343 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd441f2970 con 0x7fdd441f35f0 2023-10-30T14:20:47.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.795+0000 7fdd417fa640 1 -- 172.21.15.57:0/2144861343 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fdd441f2970 con 0x7fdd441f35f0 2023-10-30T14:20:47.802 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:47.803 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:45.990498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:47.803 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 50 2023-10-30T14:20:47.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.799+0000 7fdd4ad1d640 1 -- 172.21.15.57:0/2144861343 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdd280664d0 msgr2=0x7fdd28068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:47.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.799+0000 7fdd4ad1d640 1 --2- 172.21.15.57:0/2144861343 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdd280664d0 0x7fdd28068990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdd380054a0 tx=0x7fdd380014c0 comp rx=0 tx=0).stop 2023-10-30T14:20:47.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.799+0000 7fdd4ad1d640 1 -- 172.21.15.57:0/2144861343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd441f35f0 msgr2=0x7fdd441557e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:47.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.799+0000 7fdd4ad1d640 1 --2- 172.21.15.57:0/2144861343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd441f35f0 0x7fdd441557e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fdd3001d270 tx=0x7fdd34005800 comp rx=0 tx=0).stop 2023-10-30T14:20:47.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.799+0000 7fdd4ad1d640 1 -- 172.21.15.57:0/2144861343 shutdown_connections 2023-10-30T14:20:47.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.799+0000 7fdd4ad1d640 1 --2- 172.21.15.57:0/2144861343 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdd280664d0 0x7fdd28068990 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:47.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.799+0000 7fdd4ad1d640 1 --2- 172.21.15.57:0/2144861343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd4414fe50 0x7fdd441f5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:47.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.799+0000 7fdd4ad1d640 1 --2- 172.21.15.57:0/2144861343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd441f35f0 0x7fdd441557e0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:47.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.799+0000 7fdd4ad1d640 1 --2- 172.21.15.57:0/2144861343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd34003f00 0x7fdd441f3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:47.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.799+0000 7fdd4ad1d640 1 -- 172.21.15.57:0/2144861343 >> 172.21.15.57:0/2144861343 conn(0x7fdd440cac70 msgr2=0x7fdd440c6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:47.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.799+0000 7fdd4ad1d640 1 -- 172.21.15.57:0/2144861343 shutdown_connections 2023-10-30T14:20:47.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:47.799+0000 7fdd4ad1d640 1 -- 172.21.15.57:0/2144861343 wait complete. 2023-10-30T14:20:47.824 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5078": { "addr": "172.21.15.88:6839/1677924344", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1677924344, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1677924344, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5078, "incarnation": 48, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:reconnect", "state_seq": 22 }, "gid_5133": { "addr": "172.21.15.57:6834/2912050965", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2912050965, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2912050965, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5133, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 15 }, "gid_5289": { "addr": "172.21.15.57:6836/2427081813", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2427081813, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2427081813, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5289, "incarnation": 35, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:20:45.990498+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5289, "mds_1": 5133, "mds_2": 5078 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:20:49.826 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:49.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.975+0000 7f06b750f640 1 Processor -- start 2023-10-30T14:20:49.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.975+0000 7f06b750f640 1 -- start start 2023-10-30T14:20:49.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.975+0000 7f06b750f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f06b00ccf60 0x7f06b00cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:49.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.975+0000 7f06b750f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06b00c2500 0x7f06b00c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:49.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.975+0000 7f06b750f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b0151aa0 con 0x7f06b01516c0 2023-10-30T14:20:49.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.975+0000 7f06b750f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b00c74a0 con 0x7f06b00ccf60 2023-10-30T14:20:49.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.975+0000 7f06b750f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b00c2e70 con 0x7f06b00c2500 2023-10-30T14:20:49.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.975+0000 7f06b4aaa640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f06b00ccf60 0x7f06b00cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:49.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.975+0000 7f06b4aaa640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f06b00ccf60 0x7f06b00cd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47904/0 (socket says 172.21.15.57:47904) 2023-10-30T14:20:49.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.975+0000 7f06b4aaa640 1 -- 172.21.15.57:0/3644127636 learned_addr learned my addr 172.21.15.57:0/3644127636 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:49.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.975+0000 7f06b5aac640 1 --2- 172.21.15.57:0/3644127636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06b00c2500 0x7f06b00c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:49.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.975+0000 7f06b4aaa640 1 -- 172.21.15.57:0/3644127636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06b00c2500 msgr2=0x7f06b00c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:49.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.975+0000 7f06b4aaa640 1 --2- 172.21.15.57:0/3644127636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06b00c2500 0x7f06b00c2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:49.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.975+0000 7f06b4aaa640 1 -- 172.21.15.57:0/3644127636 >> v1:172.21.15.88:6789/0 conn(0x7f06b01516c0 legacy=0x7f06b00c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:49.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.975+0000 7f06b4aaa640 1 -- 172.21.15.57:0/3644127636 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b0158af0 con 0x7f06b00ccf60 2023-10-30T14:20:49.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.975+0000 7f06b5aac640 1 --2- 172.21.15.57:0/3644127636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06b00c2500 0x7f06b00c2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:49.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.979+0000 7f06b4aaa640 1 --2- 172.21.15.57:0/3644127636 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06b00ccf60 0x7f06b00cd3a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009870 tx=0x7f06a001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ffb9a575eba539 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:49.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.979+0000 7f06a7fff640 1 -- 172.21.15.57:0/3644127636 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a00ab600 con 0x7f06b00ccf60 2023-10-30T14:20:49.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.979+0000 7f06a7fff640 1 -- 172.21.15.57:0/3644127636 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a00a7050 con 0x7f06b00ccf60 2023-10-30T14:20:49.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.979+0000 7f06a7fff640 1 -- 172.21.15.57:0/3644127636 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a00ab980 con 0x7f06b00ccf60 2023-10-30T14:20:49.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.979+0000 7f06b750f640 1 -- 172.21.15.57:0/3644127636 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06b00ccf60 msgr2=0x7f06b00cd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:49.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.979+0000 7f06b750f640 1 --2- 172.21.15.57:0/3644127636 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06b00ccf60 0x7f06b00cd3a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009870 tx=0x7f06a001c540 comp rx=0 tx=0).stop 2023-10-30T14:20:49.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.979+0000 7f06b750f640 1 -- 172.21.15.57:0/3644127636 shutdown_connections 2023-10-30T14:20:49.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.979+0000 7f06b750f640 1 --2- 172.21.15.57:0/3644127636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06b00c2500 0x7f06b00c2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:49.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.979+0000 7f06b750f640 1 --2- 172.21.15.57:0/3644127636 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06b00ccf60 0x7f06b00cd3a0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:49.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.979+0000 7f06b750f640 1 -- 172.21.15.57:0/3644127636 >> 172.21.15.57:0/3644127636 conn(0x7f06b0071fb0 msgr2=0x7f06b00723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:49.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.979+0000 7f06b750f640 1 -- 172.21.15.57:0/3644127636 shutdown_connections 2023-10-30T14:20:49.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.979+0000 7f06b750f640 1 -- 172.21.15.57:0/3644127636 wait complete. 2023-10-30T14:20:49.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06b750f640 1 Processor -- start 2023-10-30T14:20:49.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06b750f640 1 -- start start 2023-10-30T14:20:49.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06b750f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06b00c2500 0x7f06b01508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:49.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06b750f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06b00ccf60 0x7f06b014df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:49.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06b750f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06b01516c0 0x7f06b014e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:49.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06b750f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b006b970 con 0x7f06b00ccf60 2023-10-30T14:20:49.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06b750f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b01f2910 con 0x7f06b00c2500 2023-10-30T14:20:49.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06b750f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b006caf0 con 0x7f06b01516c0 2023-10-30T14:20:49.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06b4aaa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06b00ccf60 0x7f06b014df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:49.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06b4aaa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06b00ccf60 0x7f06b014df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47916/0 (socket says 172.21.15.57:47916) 2023-10-30T14:20:49.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06b4aaa640 1 -- 172.21.15.57:0/2690343468 learned_addr learned my addr 172.21.15.57:0/2690343468 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:49.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06b52ab640 1 --2- 172.21.15.57:0/2690343468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06b00c2500 0x7f06b01508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:49.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06b5aac640 1 --2- 172.21.15.57:0/2690343468 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06b01516c0 0x7f06b014e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:49.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06b4aaa640 1 -- 172.21.15.57:0/2690343468 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06b01516c0 msgr2=0x7f06b014e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:49.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06b4aaa640 1 --2- 172.21.15.57:0/2690343468 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06b01516c0 0x7f06b014e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:49.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06b4aaa640 1 -- 172.21.15.57:0/2690343468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06b00c2500 msgr2=0x7f06b01508c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:49.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06b4aaa640 1 --2- 172.21.15.57:0/2690343468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06b00c2500 0x7f06b01508c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:49.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06b4aaa640 1 -- 172.21.15.57:0/2690343468 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a00094d0 con 0x7f06b00ccf60 2023-10-30T14:20:49.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06b4aaa640 1 --2- 172.21.15.57:0/2690343468 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06b00ccf60 0x7f06b014df80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f06a001a170 tx=0x7f06a00192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:49.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06a5ffb640 1 -- 172.21.15.57:0/2690343468 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a00aa5b0 con 0x7f06b00ccf60 2023-10-30T14:20:49.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06a5ffb640 1 -- 172.21.15.57:0/2690343468 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a00aa710 con 0x7f06b00ccf60 2023-10-30T14:20:49.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.983+0000 7f06a5ffb640 1 -- 172.21.15.57:0/2690343468 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a00aaa80 con 0x7f06b00ccf60 2023-10-30T14:20:49.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.987+0000 7f06b750f640 1 -- 172.21.15.57:0/2690343468 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b006cdd0 con 0x7f06b00ccf60 2023-10-30T14:20:49.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.987+0000 7f06b750f640 1 -- 172.21.15.57:0/2690343468 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b006c7f0 con 0x7f06b00ccf60 2023-10-30T14:20:49.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.987+0000 7f06b750f640 1 -- 172.21.15.57:0/2690343468 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b014dca0 con 0x7f06b00ccf60 2023-10-30T14:20:49.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.987+0000 7f06a5ffb640 1 -- 172.21.15.57:0/2690343468 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f06a00b2070 con 0x7f06b00ccf60 2023-10-30T14:20:49.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.987+0000 7f06a5ffb640 1 --2- 172.21.15.57:0/2690343468 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06940664a0 0x7f0694068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:49.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.987+0000 7f06a5ffb640 1 -- 172.21.15.57:0/2690343468 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f06a011d4b0 con 0x7f06b00ccf60 2023-10-30T14:20:49.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.987+0000 7f06b52ab640 1 --2- 172.21.15.57:0/2690343468 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06940664a0 0x7f0694068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:49.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.991+0000 7f06b52ab640 1 --2- 172.21.15.57:0/2690343468 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06940664a0 0x7f0694068960 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f06ac007fd0 tx=0x7f06ac009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:49.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:49.991+0000 7f06a5ffb640 1 -- 172.21.15.57:0/2690343468 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f06b014dca0 con 0x7f06b00ccf60 2023-10-30T14:20:50.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:50.183+0000 7f06b750f640 1 -- 172.21.15.57:0/2690343468 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06b014dca0 con 0x7f06b00ccf60 2023-10-30T14:20:50.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:50.183+0000 7f06a5ffb640 1 -- 172.21.15.57:0/2690343468 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5742 (secure 0 0 0) 0x7f06b014dca0 con 0x7f06b00ccf60 2023-10-30T14:20:50.188 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:50.188 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:45.990498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:50.188 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 50 2023-10-30T14:20:50.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:50.183+0000 7f06b750f640 1 -- 172.21.15.57:0/2690343468 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06940664a0 msgr2=0x7f0694068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:50.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:50.183+0000 7f06b750f640 1 --2- 172.21.15.57:0/2690343468 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06940664a0 0x7f0694068960 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f06ac007fd0 tx=0x7f06ac009040 comp rx=0 tx=0).stop 2023-10-30T14:20:50.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:50.183+0000 7f06b750f640 1 -- 172.21.15.57:0/2690343468 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06b00ccf60 msgr2=0x7f06b014df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:50.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:50.183+0000 7f06b750f640 1 --2- 172.21.15.57:0/2690343468 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06b00ccf60 0x7f06b014df80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f06a001a170 tx=0x7f06a00192e0 comp rx=0 tx=0).stop 2023-10-30T14:20:50.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:50.183+0000 7f06b750f640 1 -- 172.21.15.57:0/2690343468 shutdown_connections 2023-10-30T14:20:50.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:50.183+0000 7f06b750f640 1 --2- 172.21.15.57:0/2690343468 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06940664a0 0x7f0694068960 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:50.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:50.183+0000 7f06b750f640 1 --2- 172.21.15.57:0/2690343468 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06b01516c0 0x7f06b014e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:50.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:50.183+0000 7f06b750f640 1 --2- 172.21.15.57:0/2690343468 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06b00ccf60 0x7f06b014df80 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:50.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:50.183+0000 7f06b750f640 1 --2- 172.21.15.57:0/2690343468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06b00c2500 0x7f06b01508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:50.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:50.183+0000 7f06b750f640 1 -- 172.21.15.57:0/2690343468 >> 172.21.15.57:0/2690343468 conn(0x7f06b0071fb0 msgr2=0x7f06b0161860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:50.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:50.183+0000 7f06b750f640 1 -- 172.21.15.57:0/2690343468 shutdown_connections 2023-10-30T14:20:50.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:50.183+0000 7f06b750f640 1 -- 172.21.15.57:0/2690343468 wait complete. 2023-10-30T14:20:50.209 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:20:52.210 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:52.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.359+0000 7fd917a17640 1 Processor -- start 2023-10-30T14:20:52.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.359+0000 7fd917a17640 1 -- start start 2023-10-30T14:20:52.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.363+0000 7fd917a17640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd910065ba0 0x7fd91015a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:52.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.363+0000 7fd917a17640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91006e390 con 0x7fd910153c10 2023-10-30T14:20:52.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.363+0000 7fd917a17640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9100bec10 con 0x7fd910071340 2023-10-30T14:20:52.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.363+0000 7fd917a17640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd910072390 con 0x7fd910065ba0 2023-10-30T14:20:52.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.363+0000 7fd9157b3640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fd910153c10 0x7fd910070c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:39704/0 (socket says 172.21.15.57:39704) 2023-10-30T14:20:52.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.363+0000 7fd9157b3640 1 -- 172.21.15.57:0/2943023049 learned_addr learned my addr 172.21.15.57:0/2943023049 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:20:52.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.363+0000 7fd915fb4640 1 --2- 172.21.15.57:0/2943023049 >> v2:172.21.15.88:3300/0 conn(0x7fd910065ba0 0x7fd91015a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:52.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.363+0000 7fd907fff640 1 -- 172.21.15.57:0/2943023049 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257735629 0 0) 0x7fd91006e390 con 0x7fd910153c10 2023-10-30T14:20:52.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.363+0000 7fd907fff640 1 -- 172.21.15.57:0/2943023049 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8fc003530 con 0x7fd910153c10 2023-10-30T14:20:52.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.363+0000 7fd915fb4640 1 -- 172.21.15.57:0/2943023049 >> v1:172.21.15.88:6789/0 conn(0x7fd910071340 legacy=0x7fd910159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:52.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.363+0000 7fd915fb4640 1 -- 172.21.15.57:0/2943023049 >> v1:172.21.15.57:6789/0 conn(0x7fd910153c10 legacy=0x7fd910070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:52.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.363+0000 7fd907fff640 1 -- 172.21.15.57:0/2943023049 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2876802861 0 0) 0x7fd8fc003530 con 0x7fd910153c10 2023-10-30T14:20:52.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.363+0000 7fd915fb4640 1 -- 172.21.15.57:0/2943023049 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91006e5e0 con 0x7fd910065ba0 2023-10-30T14:20:52.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.367+0000 7fd915fb4640 1 --2- 172.21.15.57:0/2943023049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd910065ba0 0x7fd91015a080 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd908009d90 tx=0x7fd90801c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98b18289791e806a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:52.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.367+0000 7fd907fff640 1 -- 172.21.15.57:0/2943023049 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c001680 con 0x7fd910065ba0 2023-10-30T14:20:52.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.367+0000 7fd907fff640 1 -- 172.21.15.57:0/2943023049 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9080a9400 con 0x7fd910065ba0 2023-10-30T14:20:52.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.367+0000 7fd907fff640 1 -- 172.21.15.57:0/2943023049 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9080a9570 con 0x7fd910065ba0 2023-10-30T14:20:52.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.367+0000 7fd917a17640 1 -- 172.21.15.57:0/2943023049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd910065ba0 msgr2=0x7fd91015a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:52.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.367+0000 7fd917a17640 1 --2- 172.21.15.57:0/2943023049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd910065ba0 0x7fd91015a080 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd908009d90 tx=0x7fd90801c7e0 comp rx=0 tx=0).stop 2023-10-30T14:20:52.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.367+0000 7fd917a17640 1 -- 172.21.15.57:0/2943023049 shutdown_connections 2023-10-30T14:20:52.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.367+0000 7fd917a17640 1 --2- 172.21.15.57:0/2943023049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd910065ba0 0x7fd91015a080 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:52.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.367+0000 7fd917a17640 1 -- 172.21.15.57:0/2943023049 >> 172.21.15.57:0/2943023049 conn(0x7fd9100c7f60 msgr2=0x7fd9100c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:52.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.367+0000 7fd917a17640 1 -- 172.21.15.57:0/2943023049 shutdown_connections 2023-10-30T14:20:52.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.367+0000 7fd917a17640 1 -- 172.21.15.57:0/2943023049 wait complete. 2023-10-30T14:20:52.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.367+0000 7fd917a17640 1 Processor -- start 2023-10-30T14:20:52.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.367+0000 7fd917a17640 1 -- start start 2023-10-30T14:20:52.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd917a17640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd910065ba0 0x7fd9100c31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:52.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd917a17640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd910071340 0x7fd910147e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:52.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd917a17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd910153c10 0x7fd9100c3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:52.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd917a17640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91015dbf0 con 0x7fd910065ba0 2023-10-30T14:20:52.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd917a17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9100c5580 con 0x7fd910153c10 2023-10-30T14:20:52.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd917a17640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9100c3730 con 0x7fd910071340 2023-10-30T14:20:52.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd9157b3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd910065ba0 0x7fd9100c31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:52.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd9157b3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd910065ba0 0x7fd9100c31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47922/0 (socket says 172.21.15.57:47922) 2023-10-30T14:20:52.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd9157b3640 1 -- 172.21.15.57:0/997636224 learned_addr learned my addr 172.21.15.57:0/997636224 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:52.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd915fb4640 1 --2- 172.21.15.57:0/997636224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd910153c10 0x7fd9100c3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:52.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd914fb2640 1 --2- 172.21.15.57:0/997636224 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd910071340 0x7fd910147e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:52.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd9157b3640 1 -- 172.21.15.57:0/997636224 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd910071340 msgr2=0x7fd910147e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:52.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd9157b3640 1 --2- 172.21.15.57:0/997636224 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd910071340 0x7fd910147e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:52.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd9157b3640 1 -- 172.21.15.57:0/997636224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd910153c10 msgr2=0x7fd9100c3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:52.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd9157b3640 1 --2- 172.21.15.57:0/997636224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd910153c10 0x7fd9100c3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:52.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd9157b3640 1 -- 172.21.15.57:0/997636224 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd908009a70 con 0x7fd910065ba0 2023-10-30T14:20:52.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd915fb4640 1 --2- 172.21.15.57:0/997636224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd910153c10 0x7fd9100c3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:52.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd9157b3640 1 --2- 172.21.15.57:0/997636224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd910065ba0 0x7fd9100c31f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0011f0 tx=0x7fd90c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:52.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd905ffb640 1 -- 172.21.15.57:0/997636224 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c005ae0 con 0x7fd910065ba0 2023-10-30T14:20:52.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd905ffb640 1 -- 172.21.15.57:0/997636224 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90c005c60 con 0x7fd910065ba0 2023-10-30T14:20:52.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd905ffb640 1 -- 172.21.15.57:0/997636224 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c0072b0 con 0x7fd910065ba0 2023-10-30T14:20:52.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd917a17640 1 -- 172.21.15.57:0/997636224 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9101f28b0 con 0x7fd910065ba0 2023-10-30T14:20:52.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd917a17640 1 -- 172.21.15.57:0/997636224 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9101f1af0 con 0x7fd910065ba0 2023-10-30T14:20:52.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.371+0000 7fd917a17640 1 -- 172.21.15.57:0/997636224 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd910071860 con 0x7fd910065ba0 2023-10-30T14:20:52.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.375+0000 7fd905ffb640 1 -- 172.21.15.57:0/997636224 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd90c007410 con 0x7fd910065ba0 2023-10-30T14:20:52.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.375+0000 7fd905ffb640 1 --2- 172.21.15.57:0/997636224 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd8f40664a0 0x7fd8f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:52.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.375+0000 7fd905ffb640 1 -- 172.21.15.57:0/997636224 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fd90c083880 con 0x7fd910065ba0 2023-10-30T14:20:52.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.375+0000 7fd914fb2640 1 --2- 172.21.15.57:0/997636224 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd8f40664a0 0x7fd8f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:52.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.375+0000 7fd914fb2640 1 --2- 172.21.15.57:0/997636224 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd8f40664a0 0x7fd8f4068960 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd91006df30 tx=0x7fd900009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:52.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.379+0000 7fd905ffb640 1 -- 172.21.15.57:0/997636224 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd90c0509b0 con 0x7fd910065ba0 2023-10-30T14:20:52.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.567+0000 7fd917a17640 1 -- 172.21.15.57:0/997636224 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd91004e170 con 0x7fd910065ba0 2023-10-30T14:20:52.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.567+0000 7fd905ffb640 1 -- 172.21.15.57:0/997636224 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fd90c017090 con 0x7fd910065ba0 2023-10-30T14:20:52.573 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:52.573 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:45.990498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:52.573 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 50 2023-10-30T14:20:52.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.571+0000 7fd917a17640 1 -- 172.21.15.57:0/997636224 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd8f40664a0 msgr2=0x7fd8f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:52.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.571+0000 7fd917a17640 1 --2- 172.21.15.57:0/997636224 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd8f40664a0 0x7fd8f4068960 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd91006df30 tx=0x7fd900009040 comp rx=0 tx=0).stop 2023-10-30T14:20:52.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.571+0000 7fd917a17640 1 -- 172.21.15.57:0/997636224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd910065ba0 msgr2=0x7fd9100c31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:52.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.571+0000 7fd917a17640 1 --2- 172.21.15.57:0/997636224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd910065ba0 0x7fd9100c31f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0011f0 tx=0x7fd90c00cb90 comp rx=0 tx=0).stop 2023-10-30T14:20:52.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.571+0000 7fd917a17640 1 -- 172.21.15.57:0/997636224 shutdown_connections 2023-10-30T14:20:52.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.571+0000 7fd917a17640 1 --2- 172.21.15.57:0/997636224 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd8f40664a0 0x7fd8f4068960 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:52.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.571+0000 7fd917a17640 1 --2- 172.21.15.57:0/997636224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd910153c10 0x7fd9100c3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:52.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.571+0000 7fd917a17640 1 --2- 172.21.15.57:0/997636224 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd910071340 0x7fd910147e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:52.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.571+0000 7fd917a17640 1 --2- 172.21.15.57:0/997636224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd910065ba0 0x7fd9100c31f0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:52.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.571+0000 7fd917a17640 1 -- 172.21.15.57:0/997636224 >> 172.21.15.57:0/997636224 conn(0x7fd9100c7f60 msgr2=0x7fd9100c5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:52.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.571+0000 7fd917a17640 1 -- 172.21.15.57:0/997636224 shutdown_connections 2023-10-30T14:20:52.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:52.571+0000 7fd917a17640 1 -- 172.21.15.57:0/997636224 wait complete. 2023-10-30T14:20:52.595 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:20:54.596 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:54.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.747+0000 7f06ce6ea640 1 Processor -- start 2023-10-30T14:20:54.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.747+0000 7f06ce6ea640 1 -- start start 2023-10-30T14:20:54.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.747+0000 7f06ce6ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c8153c80 0x7f06c815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:54.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.747+0000 7f06ce6ea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f06c80c42a0 0x7f06c80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:54.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.747+0000 7f06ce6ea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c8159f20 con 0x7f06c80c4a10 2023-10-30T14:20:54.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.747+0000 7f06ce6ea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c815dec0 con 0x7f06c80c42a0 2023-10-30T14:20:54.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.747+0000 7f06ce6ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c81599a0 con 0x7f06c8153c80 2023-10-30T14:20:54.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.747+0000 7f06c7fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c8153c80 0x7f06c815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:54.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.747+0000 7f06ccc87640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f06c80c42a0 0x7f06c80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:54.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.747+0000 7f06c7fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c8153c80 0x7f06c815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:43620/0 (socket says 172.21.15.57:43620) 2023-10-30T14:20:54.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.747+0000 7f06c7fff640 1 -- 172.21.15.57:0/2277754641 learned_addr learned my addr 172.21.15.57:0/2277754641 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:54.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.747+0000 7f06c7fff640 1 -- 172.21.15.57:0/2277754641 >> v2:172.21.15.88:3300/0 conn(0x7f06c80c42a0 msgr2=0x7f06c80ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:54.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.747+0000 7f06c7fff640 1 --2- 172.21.15.57:0/2277754641 >> v2:172.21.15.88:3300/0 conn(0x7f06c80c42a0 0x7f06c80ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:54.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.747+0000 7f06c7fff640 1 -- 172.21.15.57:0/2277754641 >> v1:172.21.15.88:6789/0 conn(0x7f06c80c4a10 legacy=0x7f06c815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:54.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.747+0000 7f06c7fff640 1 -- 172.21.15.57:0/2277754641 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c815d620 con 0x7f06c8153c80 2023-10-30T14:20:54.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.751+0000 7f06c7fff640 1 --2- 172.21.15.57:0/2277754641 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c8153c80 0x7f06c815cf90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f06b8009870 tx=0x7f06b801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2787c0fc64ffaa56 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:54.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.751+0000 7f06c6ffd640 1 -- 172.21.15.57:0/2277754641 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b80a8a30 con 0x7f06c8153c80 2023-10-30T14:20:54.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.751+0000 7f06c6ffd640 1 -- 172.21.15.57:0/2277754641 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b80a8b90 con 0x7f06c8153c80 2023-10-30T14:20:54.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.751+0000 7f06c6ffd640 1 -- 172.21.15.57:0/2277754641 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b80b0440 con 0x7f06c8153c80 2023-10-30T14:20:54.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.751+0000 7f06ce6ea640 1 -- 172.21.15.57:0/2277754641 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c8153c80 msgr2=0x7f06c815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:54.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.751+0000 7f06ce6ea640 1 --2- 172.21.15.57:0/2277754641 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c8153c80 0x7f06c815cf90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f06b8009870 tx=0x7f06b801ed30 comp rx=0 tx=0).stop 2023-10-30T14:20:54.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.751+0000 7f06ce6ea640 1 -- 172.21.15.57:0/2277754641 shutdown_connections 2023-10-30T14:20:54.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.751+0000 7f06ce6ea640 1 --2- 172.21.15.57:0/2277754641 >> v2:172.21.15.88:3300/0 conn(0x7f06c80c42a0 0x7f06c80ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:54.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.751+0000 7f06ce6ea640 1 --2- 172.21.15.57:0/2277754641 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c8153c80 0x7f06c815cf90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:54.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.751+0000 7f06ce6ea640 1 -- 172.21.15.57:0/2277754641 >> 172.21.15.57:0/2277754641 conn(0x7f06c80715c0 msgr2=0x7f06c80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:54.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.751+0000 7f06ce6ea640 1 -- 172.21.15.57:0/2277754641 shutdown_connections 2023-10-30T14:20:54.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.751+0000 7f06ce6ea640 1 -- 172.21.15.57:0/2277754641 wait complete. 2023-10-30T14:20:54.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06ce6ea640 1 Processor -- start 2023-10-30T14:20:54.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06ce6ea640 1 -- start start 2023-10-30T14:20:54.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06ce6ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c80c42a0 0x7f06c815ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:54.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06ce6ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c80c4a10 0x7f06c8151150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:54.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06ce6ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c8153c80 0x7f06c8151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:54.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06ce6ea640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c80b50b0 con 0x7f06c80c42a0 2023-10-30T14:20:54.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06ce6ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c8152ea0 con 0x7f06c80c4a10 2023-10-30T14:20:54.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06ce6ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c8151fb0 con 0x7f06c8153c80 2023-10-30T14:20:54.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06c7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c80c42a0 0x7f06c815ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:54.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06c7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c80c42a0 0x7f06c815ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45744/0 (socket says 172.21.15.57:45744) 2023-10-30T14:20:54.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06c7fff640 1 -- 172.21.15.57:0/1563105383 learned_addr learned my addr 172.21.15.57:0/1563105383 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:54.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06c77fe640 1 --2- 172.21.15.57:0/1563105383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c80c4a10 0x7f06c8151150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:54.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06ccc87640 1 --2- 172.21.15.57:0/1563105383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c8153c80 0x7f06c8151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:54.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06c7fff640 1 -- 172.21.15.57:0/1563105383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c8153c80 msgr2=0x7f06c8151690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:54.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06c7fff640 1 --2- 172.21.15.57:0/1563105383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c8153c80 0x7f06c8151690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:54.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06c7fff640 1 -- 172.21.15.57:0/1563105383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c80c4a10 msgr2=0x7f06c8151150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:54.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06c7fff640 1 --2- 172.21.15.57:0/1563105383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c80c4a10 0x7f06c8151150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:54.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06c7fff640 1 -- 172.21.15.57:0/1563105383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b80094d0 con 0x7f06c80c42a0 2023-10-30T14:20:54.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06c7fff640 1 --2- 172.21.15.57:0/1563105383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c80c42a0 0x7f06c815ab60 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f06b801c960 tx=0x7f06b801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:54.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06c4ff9640 1 -- 172.21.15.57:0/1563105383 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b80c0070 con 0x7f06c80c42a0 2023-10-30T14:20:54.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06c4ff9640 1 -- 172.21.15.57:0/1563105383 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b80b1bc0 con 0x7f06c80c42a0 2023-10-30T14:20:54.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06ce6ea640 1 -- 172.21.15.57:0/1563105383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c8152290 con 0x7f06c80c42a0 2023-10-30T14:20:54.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06c4ff9640 1 -- 172.21.15.57:0/1563105383 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b80bb440 con 0x7f06c80c42a0 2023-10-30T14:20:54.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06ce6ea640 1 -- 172.21.15.57:0/1563105383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c8152850 con 0x7f06c80c42a0 2023-10-30T14:20:54.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.755+0000 7f06ce6ea640 1 -- 172.21.15.57:0/1563105383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c804e170 con 0x7f06c80c42a0 2023-10-30T14:20:54.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.759+0000 7f06c4ff9640 1 -- 172.21.15.57:0/1563105383 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f06b80b7040 con 0x7f06c80c42a0 2023-10-30T14:20:54.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.759+0000 7f06c4ff9640 1 --2- 172.21.15.57:0/1563105383 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06b40664a0 0x7f06b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:54.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.759+0000 7f06c4ff9640 1 -- 172.21.15.57:0/1563105383 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f06b81216b0 con 0x7f06c80c42a0 2023-10-30T14:20:54.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.759+0000 7f06c77fe640 1 --2- 172.21.15.57:0/1563105383 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06b40664a0 0x7f06b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:54.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.759+0000 7f06c77fe640 1 --2- 172.21.15.57:0/1563105383 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06b40664a0 0x7f06b4068960 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f06c8159780 tx=0x7f06bc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:54.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.763+0000 7f06c4ff9640 1 -- 172.21.15.57:0/1563105383 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f06b80ee890 con 0x7f06c80c42a0 2023-10-30T14:20:54.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.955+0000 7f06ce6ea640 1 -- 172.21.15.57:0/1563105383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06c8150300 con 0x7f06c80c42a0 2023-10-30T14:20:54.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.955+0000 7f06c4ff9640 1 -- 172.21.15.57:0/1563105383 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f06b80f1eb0 con 0x7f06c80c42a0 2023-10-30T14:20:54.962 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:54.962 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:52.001657+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:54.962 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 51 2023-10-30T14:20:54.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.959+0000 7f06ce6ea640 1 -- 172.21.15.57:0/1563105383 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06b40664a0 msgr2=0x7f06b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:54.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.959+0000 7f06ce6ea640 1 --2- 172.21.15.57:0/1563105383 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06b40664a0 0x7f06b4068960 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f06c8159780 tx=0x7f06bc009040 comp rx=0 tx=0).stop 2023-10-30T14:20:54.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.959+0000 7f06ce6ea640 1 -- 172.21.15.57:0/1563105383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c80c42a0 msgr2=0x7f06c815ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:54.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.959+0000 7f06ce6ea640 1 --2- 172.21.15.57:0/1563105383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c80c42a0 0x7f06c815ab60 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f06b801c960 tx=0x7f06b801bad0 comp rx=0 tx=0).stop 2023-10-30T14:20:54.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.959+0000 7f06ce6ea640 1 -- 172.21.15.57:0/1563105383 shutdown_connections 2023-10-30T14:20:54.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.959+0000 7f06ce6ea640 1 --2- 172.21.15.57:0/1563105383 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06b40664a0 0x7f06b4068960 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:54.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.959+0000 7f06ce6ea640 1 --2- 172.21.15.57:0/1563105383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c8153c80 0x7f06c8151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:54.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.959+0000 7f06ce6ea640 1 --2- 172.21.15.57:0/1563105383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c80c4a10 0x7f06c8151150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:54.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.959+0000 7f06ce6ea640 1 --2- 172.21.15.57:0/1563105383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c80c42a0 0x7f06c815ab60 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:54.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.959+0000 7f06ce6ea640 1 -- 172.21.15.57:0/1563105383 >> 172.21.15.57:0/1563105383 conn(0x7f06c80715c0 msgr2=0x7f06c81586d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:54.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.959+0000 7f06ce6ea640 1 -- 172.21.15.57:0/1563105383 shutdown_connections 2023-10-30T14:20:54.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:54.959+0000 7f06ce6ea640 1 -- 172.21.15.57:0/1563105383 wait complete. 2023-10-30T14:20:54.983 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:20:56.984 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:57.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.131+0000 7fcbd233e640 1 Processor -- start 2023-10-30T14:20:57.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.131+0000 7fcbd233e640 1 -- start start 2023-10-30T14:20:57.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbd233e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbcc1537f0 0x7fcbcc0701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:57.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbd233e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbcc0c2aa0 con 0x7fcbcc06b2f0 2023-10-30T14:20:57.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbd233e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbcc0708e0 con 0x7fcbcc15a2e0 2023-10-30T14:20:57.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbd233e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc0c3ab0 con 0x7fcbcc1537f0 2023-10-30T14:20:57.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbd08db640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fcbcc06b2f0 0x7fcbcc1641e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:36472/0 (socket says 172.21.15.57:36472) 2023-10-30T14:20:57.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbd08db640 1 -- 172.21.15.57:0/1008178249 learned_addr learned my addr 172.21.15.57:0/1008178249 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:20:57.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcbfff640 1 --2- 172.21.15.57:0/1008178249 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbcc1537f0 0x7fcbcc0701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:57.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcaffd640 1 -- 172.21.15.57:0/1008178249 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 354982985 0 0) 0x7fcbcc0c2aa0 con 0x7fcbcc06b2f0 2023-10-30T14:20:57.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcaffd640 1 -- 172.21.15.57:0/1008178249 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbb4003530 con 0x7fcbcc06b2f0 2023-10-30T14:20:57.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcaffd640 1 -- 172.21.15.57:0/1008178249 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16430699 0 0) 0x7fcbcc0708e0 con 0x7fcbcc15a2e0 2023-10-30T14:20:57.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcaffd640 1 -- 172.21.15.57:0/1008178249 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbcc0c2aa0 con 0x7fcbcc15a2e0 2023-10-30T14:20:57.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcaffd640 1 -- 172.21.15.57:0/1008178249 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 290523141 0 0) 0x7fcbb4003530 con 0x7fcbcc06b2f0 2023-10-30T14:20:57.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcaffd640 1 -- 172.21.15.57:0/1008178249 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbcc1537f0 msgr2=0x7fcbcc0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:57.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcaffd640 1 --2- 172.21.15.57:0/1008178249 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbcc1537f0 0x7fcbcc0701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:57.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcaffd640 1 -- 172.21.15.57:0/1008178249 >> v1:172.21.15.88:6789/0 conn(0x7fcbcc15a2e0 legacy=0x7fcbcc0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:57.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcaffd640 1 -- 172.21.15.57:0/1008178249 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbcc0c1eb0 con 0x7fcbcc06b2f0 2023-10-30T14:20:57.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcaffd640 1 -- 172.21.15.57:0/1008178249 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fcbc4001a00 con 0x7fcbcc06b2f0 2023-10-30T14:20:57.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcaffd640 1 -- 172.21.15.57:0/1008178249 >> v1:172.21.15.57:6789/0 conn(0x7fcbcc06b2f0 legacy=0x7fcbcc1641e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:57.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcaffd640 1 --2- 172.21.15.57:0/1008178249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbb4003b40 0x7fcbb4003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:57.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcaffd640 1 --2- 172.21.15.57:0/1008178249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbb40047d0 0x7fcbb4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:57.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcaffd640 1 --2- 172.21.15.57:0/1008178249 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbcc1537f0 0x7fcbb4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:57.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcaffd640 1 -- 172.21.15.57:0/1008178249 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb400ba40 con 0x7fcbb40047d0 2023-10-30T14:20:57.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcaffd640 1 -- 172.21.15.57:0/1008178249 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb400bb80 con 0x7fcbb4003b40 2023-10-30T14:20:57.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcaffd640 1 -- 172.21.15.57:0/1008178249 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb400bd10 con 0x7fcbcc1537f0 2023-10-30T14:20:57.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcb7fe640 1 --2- 172.21.15.57:0/1008178249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbb40047d0 0x7fcbb4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:57.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcbfff640 1 --2- 172.21.15.57:0/1008178249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbb4003b40 0x7fcbb4003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:57.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbd08db640 1 --2- 172.21.15.57:0/1008178249 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbcc1537f0 0x7fcbb4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:57.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcb7fe640 1 -- 172.21.15.57:0/1008178249 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbcc1537f0 msgr2=0x7fcbb4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:57.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcb7fe640 1 --2- 172.21.15.57:0/1008178249 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbcc1537f0 0x7fcbb4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:57.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcb7fe640 1 -- 172.21.15.57:0/1008178249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbb4003b40 msgr2=0x7fcbb4003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:57.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcb7fe640 1 --2- 172.21.15.57:0/1008178249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbb4003b40 0x7fcbb4003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:57.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbcb7fe640 1 -- 172.21.15.57:0/1008178249 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb4003530 con 0x7fcbb40047d0 2023-10-30T14:20:57.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.135+0000 7fcbd08db640 1 --2- 172.21.15.57:0/1008178249 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbcc1537f0 0x7fcbb4005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:57.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.139+0000 7fcbcb7fe640 1 --2- 172.21.15.57:0/1008178249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbb40047d0 0x7fcbb4004c50 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00018d0 tx=0x7fcbc001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bf02458b605d41a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:57.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.139+0000 7fcbcaffd640 1 -- 172.21.15.57:0/1008178249 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc00052d0 con 0x7fcbb40047d0 2023-10-30T14:20:57.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.139+0000 7fcbcaffd640 1 -- 172.21.15.57:0/1008178249 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc00a7050 con 0x7fcbb40047d0 2023-10-30T14:20:57.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.139+0000 7fcbcaffd640 1 -- 172.21.15.57:0/1008178249 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0003600 con 0x7fcbb40047d0 2023-10-30T14:20:57.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.139+0000 7fcbd233e640 1 -- 172.21.15.57:0/1008178249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbb40047d0 msgr2=0x7fcbb4004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:57.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.139+0000 7fcbd233e640 1 --2- 172.21.15.57:0/1008178249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbb40047d0 0x7fcbb4004c50 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00018d0 tx=0x7fcbc001c500 comp rx=0 tx=0).stop 2023-10-30T14:20:57.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.139+0000 7fcbd08db640 1 -- 172.21.15.57:0/1008178249 reap_dead start 2023-10-30T14:20:57.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.139+0000 7fcbd233e640 1 -- 172.21.15.57:0/1008178249 shutdown_connections 2023-10-30T14:20:57.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.139+0000 7fcbd233e640 1 -- 172.21.15.57:0/1008178249 >> 172.21.15.57:0/1008178249 conn(0x7fcbcc071fb0 msgr2=0x7fcbcc0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:57.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.139+0000 7fcbd233e640 1 -- 172.21.15.57:0/1008178249 shutdown_connections 2023-10-30T14:20:57.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.139+0000 7fcbd233e640 1 -- 172.21.15.57:0/1008178249 wait complete. 2023-10-30T14:20:57.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.139+0000 7fcbd233e640 1 Processor -- start 2023-10-30T14:20:57.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.139+0000 7fcbd233e640 1 -- start start 2023-10-30T14:20:57.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbd233e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbb40047d0 0x7fcbcc1eed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:57.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbd233e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbcc1f2400 0x7fcbcc0c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:57.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbd233e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbcc0ca0f0 0x7fcbcc14b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:57.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbd233e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc15a100 con 0x7fcbcc0ca0f0 2023-10-30T14:20:57.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbd233e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc1f2910 con 0x7fcbcc1f2400 2023-10-30T14:20:57.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbd233e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc14f130 con 0x7fcbb40047d0 2023-10-30T14:20:57.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbd08db640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbcc0ca0f0 0x7fcbcc14b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:57.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbd08db640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbcc0ca0f0 0x7fcbcc14b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45760/0 (socket says 172.21.15.57:45760) 2023-10-30T14:20:57.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbd08db640 1 -- 172.21.15.57:0/283548378 learned_addr learned my addr 172.21.15.57:0/283548378 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:57.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbcbfff640 1 --2- 172.21.15.57:0/283548378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbb40047d0 0x7fcbcc1eed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:57.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbcb7fe640 1 --2- 172.21.15.57:0/283548378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbcc1f2400 0x7fcbcc0c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:57.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbd08db640 1 -- 172.21.15.57:0/283548378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbb40047d0 msgr2=0x7fcbcc1eed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:57.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbd08db640 1 --2- 172.21.15.57:0/283548378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbb40047d0 0x7fcbcc1eed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:57.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbd08db640 1 -- 172.21.15.57:0/283548378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbcc1f2400 msgr2=0x7fcbcc0c9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:57.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbd08db640 1 --2- 172.21.15.57:0/283548378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbcc1f2400 0x7fcbcc0c9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:57.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbd08db640 1 -- 172.21.15.57:0/283548378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbcc14f3d0 con 0x7fcbcc0ca0f0 2023-10-30T14:20:57.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbcbfff640 1 --2- 172.21.15.57:0/283548378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbb40047d0 0x7fcbcc1eed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:57.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbd08db640 1 --2- 172.21.15.57:0/283548378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbcc0ca0f0 0x7fcbcc14b400 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc06d180 tx=0x7fcbc4011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:57.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbc8ff9640 1 -- 172.21.15.57:0/283548378 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc400f920 con 0x7fcbcc0ca0f0 2023-10-30T14:20:57.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbc8ff9640 1 -- 172.21.15.57:0/283548378 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc400faa0 con 0x7fcbcc0ca0f0 2023-10-30T14:20:57.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbc8ff9640 1 -- 172.21.15.57:0/283548378 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc400fe50 con 0x7fcbcc0ca0f0 2023-10-30T14:20:57.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbd233e640 1 -- 172.21.15.57:0/283548378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbcc14f6b0 con 0x7fcbcc0ca0f0 2023-10-30T14:20:57.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbd233e640 1 -- 172.21.15.57:0/283548378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbcc1ee590 con 0x7fcbcc0ca0f0 2023-10-30T14:20:57.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbd233e640 1 -- 172.21.15.57:0/283548378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbcc04e170 con 0x7fcbcc0ca0f0 2023-10-30T14:20:57.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.143+0000 7fcbc8ff9640 1 -- 172.21.15.57:0/283548378 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcbc401f460 con 0x7fcbcc0ca0f0 2023-10-30T14:20:57.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.147+0000 7fcbc8ff9640 1 --2- 172.21.15.57:0/283548378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcbb80664d0 0x7fcbb8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:57.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.147+0000 7fcbc8ff9640 1 -- 172.21.15.57:0/283548378 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fcbc4087140 con 0x7fcbcc0ca0f0 2023-10-30T14:20:57.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.147+0000 7fcbcbfff640 1 --2- 172.21.15.57:0/283548378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcbb80664d0 0x7fcbb8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:57.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.147+0000 7fcbcbfff640 1 --2- 172.21.15.57:0/283548378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcbb80664d0 0x7fcbb8068990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc009b00 tx=0x7fcbbc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:57.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.151+0000 7fcbc8ff9640 1 -- 172.21.15.57:0/283548378 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcbc40543a0 con 0x7fcbcc0ca0f0 2023-10-30T14:20:57.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.343+0000 7fcbd233e640 1 -- 172.21.15.57:0/283548378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbcc073de0 con 0x7fcbcc0ca0f0 2023-10-30T14:20:57.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.343+0000 7fcbc8ff9640 1 -- 172.21.15.57:0/283548378 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+5739 (secure 0 0 0) 0x7fcbc40579c0 con 0x7fcbcc0ca0f0 2023-10-30T14:20:57.350 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:57.351 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:52.001657+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:57.351 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 51 2023-10-30T14:20:57.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.347+0000 7fcbd233e640 1 -- 172.21.15.57:0/283548378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcbb80664d0 msgr2=0x7fcbb8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:57.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.347+0000 7fcbd233e640 1 --2- 172.21.15.57:0/283548378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcbb80664d0 0x7fcbb8068990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc009b00 tx=0x7fcbbc008040 comp rx=0 tx=0).stop 2023-10-30T14:20:57.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.347+0000 7fcbd233e640 1 -- 172.21.15.57:0/283548378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbcc0ca0f0 msgr2=0x7fcbcc14b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:57.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.347+0000 7fcbd233e640 1 --2- 172.21.15.57:0/283548378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbcc0ca0f0 0x7fcbcc14b400 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc06d180 tx=0x7fcbc4011c20 comp rx=0 tx=0).stop 2023-10-30T14:20:57.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.347+0000 7fcbd233e640 1 -- 172.21.15.57:0/283548378 shutdown_connections 2023-10-30T14:20:57.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.347+0000 7fcbd233e640 1 --2- 172.21.15.57:0/283548378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcbb80664d0 0x7fcbb8068990 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:57.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.347+0000 7fcbd233e640 1 --2- 172.21.15.57:0/283548378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbcc0ca0f0 0x7fcbcc14b400 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:57.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.347+0000 7fcbd233e640 1 --2- 172.21.15.57:0/283548378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbcc1f2400 0x7fcbcc0c9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:57.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.347+0000 7fcbd233e640 1 --2- 172.21.15.57:0/283548378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbb40047d0 0x7fcbcc1eed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:57.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.347+0000 7fcbd233e640 1 -- 172.21.15.57:0/283548378 >> 172.21.15.57:0/283548378 conn(0x7fcbcc071fb0 msgr2=0x7fcbcc152670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:57.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.347+0000 7fcbd233e640 1 -- 172.21.15.57:0/283548378 shutdown_connections 2023-10-30T14:20:57.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:57.347+0000 7fcbd233e640 1 -- 172.21.15.57:0/283548378 wait complete. 2023-10-30T14:20:57.372 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:20:59.373 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:59.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.515+0000 7f4e5f594640 1 Processor -- start 2023-10-30T14:20:59.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.515+0000 7f4e5f594640 1 -- start start 2023-10-30T14:20:59.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.515+0000 7f4e5f594640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4e58153c80 0x7f4e5815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:59.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.515+0000 7f4e5f594640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e580c42a0 0x7f4e580ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:59.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.515+0000 7f4e5f594640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e58159f20 con 0x7f4e580c4a10 2023-10-30T14:20:59.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.515+0000 7f4e5f594640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e5815dec0 con 0x7f4e58153c80 2023-10-30T14:20:59.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.515+0000 7f4e5f594640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e581599a0 con 0x7f4e580c42a0 2023-10-30T14:20:59.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.515+0000 7f4e5cb2f640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f4e580c4a10 0x7f4e5815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:36476/0 (socket says 172.21.15.57:36476) 2023-10-30T14:20:59.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.515+0000 7f4e5cb2f640 1 -- 172.21.15.57:0/2960206621 learned_addr learned my addr 172.21.15.57:0/2960206621 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:20:59.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.515+0000 7f4e5d330640 1 --2- 172.21.15.57:0/2960206621 >> v2:172.21.15.88:3300/0 conn(0x7f4e58153c80 0x7f4e5815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:59.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.515+0000 7f4e5db31640 1 --2- 172.21.15.57:0/2960206621 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e580c42a0 0x7f4e580ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:59.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.515+0000 7f4e4ffff640 1 -- 172.21.15.57:0/2960206621 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827469572 0 0) 0x7f4e58159f20 con 0x7f4e580c4a10 2023-10-30T14:20:59.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.515+0000 7f4e4ffff640 1 -- 172.21.15.57:0/2960206621 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e44003530 con 0x7f4e580c4a10 2023-10-30T14:20:59.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e4ffff640 1 -- 172.21.15.57:0/2960206621 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3087972020 0 0) 0x7f4e44003530 con 0x7f4e580c4a10 2023-10-30T14:20:59.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e4ffff640 1 -- 172.21.15.57:0/2960206621 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e580c42a0 msgr2=0x7f4e580ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:59.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e4ffff640 1 --2- 172.21.15.57:0/2960206621 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e580c42a0 0x7f4e580ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:59.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e4ffff640 1 -- 172.21.15.57:0/2960206621 >> v2:172.21.15.88:3300/0 conn(0x7f4e58153c80 msgr2=0x7f4e5815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:59.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e4ffff640 1 --2- 172.21.15.57:0/2960206621 >> v2:172.21.15.88:3300/0 conn(0x7f4e58153c80 0x7f4e5815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:59.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e4ffff640 1 -- 172.21.15.57:0/2960206621 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5815d620 con 0x7f4e580c4a10 2023-10-30T14:20:59.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e4ffff640 1 -- 172.21.15.57:0/2960206621 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f4e48001680 con 0x7f4e580c4a10 2023-10-30T14:20:59.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e4ffff640 1 -- 172.21.15.57:0/2960206621 >> v1:172.21.15.57:6789/0 conn(0x7f4e580c4a10 legacy=0x7f4e5815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:59.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e4ffff640 1 --2- 172.21.15.57:0/2960206621 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e44003980 0x7f4e44003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:59.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e4ffff640 1 --2- 172.21.15.57:0/2960206621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e440047d0 0x7f4e44004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:59.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e4ffff640 1 --2- 172.21.15.57:0/2960206621 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e580c42a0 0x7f4e44005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:59.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e4ffff640 1 -- 172.21.15.57:0/2960206621 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5815dec0 con 0x7f4e44003980 2023-10-30T14:20:59.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e4ffff640 1 -- 172.21.15.57:0/2960206621 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e581599a0 con 0x7f4e440047d0 2023-10-30T14:20:59.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e4ffff640 1 -- 172.21.15.57:0/2960206621 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4400b9c0 con 0x7f4e580c42a0 2023-10-30T14:20:59.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e5d330640 1 --2- 172.21.15.57:0/2960206621 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e44003980 0x7f4e44003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:59.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e5db31640 1 --2- 172.21.15.57:0/2960206621 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e580c42a0 0x7f4e44005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:59.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e5cb2f640 1 --2- 172.21.15.57:0/2960206621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e440047d0 0x7f4e44004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:59.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e5d330640 1 -- 172.21.15.57:0/2960206621 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e580c42a0 msgr2=0x7f4e44005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:59.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e5d330640 1 --2- 172.21.15.57:0/2960206621 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e580c42a0 0x7f4e44005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:59.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e5d330640 1 -- 172.21.15.57:0/2960206621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e440047d0 msgr2=0x7f4e44004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:59.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e5d330640 1 --2- 172.21.15.57:0/2960206621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e440047d0 0x7f4e44004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:59.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e5d330640 1 -- 172.21.15.57:0/2960206621 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e44003530 con 0x7f4e44003980 2023-10-30T14:20:59.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.519+0000 7f4e5d330640 1 --2- 172.21.15.57:0/2960206621 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e44003980 0x7f4e44003da0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4e54009ae0 tx=0x7f4e5401ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fce48f640f9aa38 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:59.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.523+0000 7f4e4ffff640 1 -- 172.21.15.57:0/2960206621 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e540a64a0 con 0x7f4e44003980 2023-10-30T14:20:59.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.523+0000 7f4e4ffff640 1 -- 172.21.15.57:0/2960206621 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e540a6600 con 0x7f4e44003980 2023-10-30T14:20:59.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.523+0000 7f4e4ffff640 1 -- 172.21.15.57:0/2960206621 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e540a69b0 con 0x7f4e44003980 2023-10-30T14:20:59.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.523+0000 7f4e5f594640 1 -- 172.21.15.57:0/2960206621 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e44003980 msgr2=0x7f4e44003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:59.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.523+0000 7f4e5f594640 1 --2- 172.21.15.57:0/2960206621 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e44003980 0x7f4e44003da0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4e54009ae0 tx=0x7f4e5401ca90 comp rx=0 tx=0).stop 2023-10-30T14:20:59.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.523+0000 7f4e5db31640 1 -- 172.21.15.57:0/2960206621 reap_dead start 2023-10-30T14:20:59.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.523+0000 7f4e5f594640 1 -- 172.21.15.57:0/2960206621 shutdown_connections 2023-10-30T14:20:59.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.523+0000 7f4e5f594640 1 -- 172.21.15.57:0/2960206621 >> 172.21.15.57:0/2960206621 conn(0x7f4e580715c0 msgr2=0x7f4e580719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:59.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.523+0000 7f4e5f594640 1 -- 172.21.15.57:0/2960206621 shutdown_connections 2023-10-30T14:20:59.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.523+0000 7f4e5f594640 1 -- 172.21.15.57:0/2960206621 wait complete. 2023-10-30T14:20:59.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.523+0000 7f4e5f594640 1 Processor -- start 2023-10-30T14:20:59.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.523+0000 7f4e5f594640 1 -- start start 2023-10-30T14:20:59.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.523+0000 7f4e5f594640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e580c9ef0 0x7f4e580b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:59.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.523+0000 7f4e5f594640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e580ceb00 0x7f4e58154160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:59.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.523+0000 7f4e5f594640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e580ca2d0 0x7f4e580b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:59.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.523+0000 7f4e5f594640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e580b3d30 con 0x7f4e580ca2d0 2023-10-30T14:20:59.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.523+0000 7f4e5f594640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e580cf350 con 0x7f4e580ceb00 2023-10-30T14:20:59.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.523+0000 7f4e5f594640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e580cf630 con 0x7f4e580c9ef0 2023-10-30T14:20:59.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.523+0000 7f4e5db31640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e580ca2d0 0x7f4e580b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:59.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.527+0000 7f4e5db31640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e580ca2d0 0x7f4e580b20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45770/0 (socket says 172.21.15.57:45770) 2023-10-30T14:20:59.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.527+0000 7f4e5db31640 1 -- 172.21.15.57:0/720133643 learned_addr learned my addr 172.21.15.57:0/720133643 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:20:59.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.527+0000 7f4e5d330640 1 --2- 172.21.15.57:0/720133643 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e580c9ef0 0x7f4e580b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:59.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.527+0000 7f4e5cb2f640 1 --2- 172.21.15.57:0/720133643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e580ceb00 0x7f4e58154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:59.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.527+0000 7f4e5db31640 1 -- 172.21.15.57:0/720133643 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e580c9ef0 msgr2=0x7f4e580b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:59.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.527+0000 7f4e5db31640 1 --2- 172.21.15.57:0/720133643 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e580c9ef0 0x7f4e580b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:59.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.527+0000 7f4e5db31640 1 -- 172.21.15.57:0/720133643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e580ceb00 msgr2=0x7f4e58154160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:59.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.527+0000 7f4e5db31640 1 --2- 172.21.15.57:0/720133643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e580ceb00 0x7f4e58154160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:59.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.527+0000 7f4e5db31640 1 -- 172.21.15.57:0/720133643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e580b3400 con 0x7f4e580ca2d0 2023-10-30T14:20:59.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.527+0000 7f4e5db31640 1 --2- 172.21.15.57:0/720133643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e580ca2d0 0x7f4e580b20b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4e580c97d0 tx=0x7f4e500115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:59.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.527+0000 7f4e4dffb640 1 -- 172.21.15.57:0/720133643 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5000e070 con 0x7f4e580ca2d0 2023-10-30T14:20:59.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.527+0000 7f4e4dffb640 1 -- 172.21.15.57:0/720133643 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e50011e10 con 0x7f4e580ca2d0 2023-10-30T14:20:59.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.527+0000 7f4e4dffb640 1 -- 172.21.15.57:0/720133643 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e500039a0 con 0x7f4e580ca2d0 2023-10-30T14:20:59.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.527+0000 7f4e5f594640 1 -- 172.21.15.57:0/720133643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e580b36e0 con 0x7f4e580ca2d0 2023-10-30T14:20:59.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.527+0000 7f4e5f594640 1 -- 172.21.15.57:0/720133643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e58152620 con 0x7f4e580ca2d0 2023-10-30T14:20:59.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.527+0000 7f4e5f594640 1 -- 172.21.15.57:0/720133643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e5804e170 con 0x7f4e580ca2d0 2023-10-30T14:20:59.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.527+0000 7f4e4dffb640 1 -- 172.21.15.57:0/720133643 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4e50003b00 con 0x7f4e580ca2d0 2023-10-30T14:20:59.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.527+0000 7f4e4dffb640 1 --2- 172.21.15.57:0/720133643 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4e3c0664a0 0x7f4e3c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:59.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.531+0000 7f4e4dffb640 1 -- 172.21.15.57:0/720133643 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f4e50084860 con 0x7f4e580ca2d0 2023-10-30T14:20:59.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.531+0000 7f4e5d330640 1 --2- 172.21.15.57:0/720133643 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4e3c0664a0 0x7f4e3c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:59.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.531+0000 7f4e5d330640 1 --2- 172.21.15.57:0/720133643 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4e3c0664a0 0x7f4e3c068960 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4e540ad000 tx=0x7f4e540b3040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:59.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.531+0000 7f4e4dffb640 1 -- 172.21.15.57:0/720133643 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4e50051ad0 con 0x7f4e580ca2d0 2023-10-30T14:20:59.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.723+0000 7f4e5f594640 1 -- 172.21.15.57:0/720133643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e580b3e70 con 0x7f4e580ca2d0 2023-10-30T14:20:59.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.723+0000 7f4e4dffb640 1 -- 172.21.15.57:0/720133643 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+5745 (secure 0 0 0) 0x7f4e500550f0 con 0x7f4e580ca2d0 2023-10-30T14:20:59.730 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:20:59.730 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:57.788102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:clientreplay","state_seq":26,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:59.730 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 52 2023-10-30T14:20:59.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.727+0000 7f4e5f594640 1 -- 172.21.15.57:0/720133643 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4e3c0664a0 msgr2=0x7f4e3c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:59.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.727+0000 7f4e5f594640 1 --2- 172.21.15.57:0/720133643 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4e3c0664a0 0x7f4e3c068960 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4e540ad000 tx=0x7f4e540b3040 comp rx=0 tx=0).stop 2023-10-30T14:20:59.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.727+0000 7f4e5f594640 1 -- 172.21.15.57:0/720133643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e580ca2d0 msgr2=0x7f4e580b20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:59.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.727+0000 7f4e5f594640 1 --2- 172.21.15.57:0/720133643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e580ca2d0 0x7f4e580b20b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4e580c97d0 tx=0x7f4e500115a0 comp rx=0 tx=0).stop 2023-10-30T14:20:59.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.727+0000 7f4e5f594640 1 -- 172.21.15.57:0/720133643 shutdown_connections 2023-10-30T14:20:59.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.727+0000 7f4e5f594640 1 --2- 172.21.15.57:0/720133643 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4e3c0664a0 0x7f4e3c068960 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:59.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.727+0000 7f4e5f594640 1 --2- 172.21.15.57:0/720133643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e580ca2d0 0x7f4e580b20b0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:59.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.727+0000 7f4e5f594640 1 --2- 172.21.15.57:0/720133643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e580ceb00 0x7f4e58154160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:59.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.727+0000 7f4e5f594640 1 --2- 172.21.15.57:0/720133643 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e580c9ef0 0x7f4e580b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:59.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.727+0000 7f4e5f594640 1 -- 172.21.15.57:0/720133643 >> 172.21.15.57:0/720133643 conn(0x7f4e580715c0 msgr2=0x7f4e580cb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:59.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.727+0000 7f4e5f594640 1 -- 172.21.15.57:0/720133643 shutdown_connections 2023-10-30T14:20:59.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:20:59.727+0000 7f4e5f594640 1 -- 172.21.15.57:0/720133643 wait complete. 2023-10-30T14:20:59.752 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 52, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 52, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5078": { "addr": "172.21.15.88:6839/1677924344", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1677924344, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1677924344, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5078, "incarnation": 48, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:clientreplay", "state_seq": 26 }, "gid_5133": { "addr": "172.21.15.57:6834/2912050965", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2912050965, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2912050965, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5133, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 15 }, "gid_5289": { "addr": "172.21.15.57:6836/2427081813", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2427081813, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2427081813, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5289, "incarnation": 35, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:20:57.788102+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5289, "mds_1": 5133, "mds_2": 5078 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:21:00.282 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/11' 2023-10-30T14:21:00.282 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/16' 2023-10-30T14:21:00.282 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/32' 2023-10-30T14:21:00.283 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/30' 2023-10-30T14:21:00.283 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/14' 2023-10-30T14:21:00.283 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/18' 2023-10-30T14:21:00.283 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/13' 2023-10-30T14:21:00.284 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/33' 2023-10-30T14:21:00.284 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/17' 2023-10-30T14:21:00.284 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/35' 2023-10-30T14:21:00.284 INFO:tasks.workunit.client.0.smithi057.stdout:removed './167/37' 2023-10-30T14:21:00.284 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './167' 2023-10-30T14:21:00.284 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/19' 2023-10-30T14:21:00.285 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/15' 2023-10-30T14:21:00.285 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/31' 2023-10-30T14:21:00.285 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/36' 2023-10-30T14:21:00.285 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/20' 2023-10-30T14:21:00.285 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/12' 2023-10-30T14:21:00.286 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/39' 2023-10-30T14:21:00.286 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/10' 2023-10-30T14:21:00.286 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/34' 2023-10-30T14:21:00.286 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/38' 2023-10-30T14:21:00.286 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/11' 2023-10-30T14:21:00.287 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/16' 2023-10-30T14:21:00.287 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/32' 2023-10-30T14:21:00.287 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/30' 2023-10-30T14:21:00.287 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/14' 2023-10-30T14:21:00.287 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/18' 2023-10-30T14:21:00.287 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/13' 2023-10-30T14:21:00.288 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/33' 2023-10-30T14:21:00.288 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/17' 2023-10-30T14:21:00.288 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/35' 2023-10-30T14:21:00.288 INFO:tasks.workunit.client.0.smithi057.stdout:removed './168/37' 2023-10-30T14:21:00.288 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './168' 2023-10-30T14:21:00.289 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/19' 2023-10-30T14:21:00.289 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/15' 2023-10-30T14:21:00.289 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/31' 2023-10-30T14:21:00.289 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/36' 2023-10-30T14:21:00.289 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/20' 2023-10-30T14:21:00.289 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/12' 2023-10-30T14:21:00.290 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/39' 2023-10-30T14:21:00.290 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/10' 2023-10-30T14:21:00.290 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/34' 2023-10-30T14:21:00.290 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/38' 2023-10-30T14:21:00.290 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/11' 2023-10-30T14:21:00.291 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/16' 2023-10-30T14:21:00.291 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/32' 2023-10-30T14:21:00.291 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/30' 2023-10-30T14:21:00.291 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/14' 2023-10-30T14:21:00.291 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/18' 2023-10-30T14:21:00.291 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/13' 2023-10-30T14:21:00.292 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/33' 2023-10-30T14:21:00.292 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/17' 2023-10-30T14:21:00.292 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/35' 2023-10-30T14:21:00.292 INFO:tasks.workunit.client.0.smithi057.stdout:removed './169/37' 2023-10-30T14:21:00.292 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './169' 2023-10-30T14:21:00.293 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/19' 2023-10-30T14:21:00.293 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/15' 2023-10-30T14:21:00.293 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/31' 2023-10-30T14:21:00.293 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/36' 2023-10-30T14:21:00.293 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/20' 2023-10-30T14:21:00.293 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/12' 2023-10-30T14:21:00.294 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/39' 2023-10-30T14:21:00.294 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/10' 2023-10-30T14:21:00.294 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/34' 2023-10-30T14:21:00.294 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/38' 2023-10-30T14:21:00.294 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/11' 2023-10-30T14:21:00.295 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/16' 2023-10-30T14:21:00.295 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/32' 2023-10-30T14:21:00.295 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/30' 2023-10-30T14:21:00.295 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/14' 2023-10-30T14:21:00.295 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/18' 2023-10-30T14:21:00.295 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/13' 2023-10-30T14:21:00.296 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/33' 2023-10-30T14:21:00.296 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/17' 2023-10-30T14:21:00.296 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/35' 2023-10-30T14:21:00.296 INFO:tasks.workunit.client.0.smithi057.stdout:removed './170/37' 2023-10-30T14:21:00.296 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './170' 2023-10-30T14:21:00.296 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/19' 2023-10-30T14:21:00.297 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/15' 2023-10-30T14:21:00.297 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/31' 2023-10-30T14:21:00.297 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/36' 2023-10-30T14:21:00.297 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/20' 2023-10-30T14:21:00.297 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/12' 2023-10-30T14:21:00.297 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/39' 2023-10-30T14:21:00.298 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/10' 2023-10-30T14:21:00.298 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/34' 2023-10-30T14:21:00.298 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/38' 2023-10-30T14:21:00.298 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/11' 2023-10-30T14:21:00.298 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/16' 2023-10-30T14:21:00.299 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/32' 2023-10-30T14:21:00.299 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/30' 2023-10-30T14:21:00.299 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/14' 2023-10-30T14:21:00.299 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/18' 2023-10-30T14:21:00.299 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/13' 2023-10-30T14:21:00.300 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/33' 2023-10-30T14:21:00.300 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/17' 2023-10-30T14:21:00.300 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/35' 2023-10-30T14:21:00.300 INFO:tasks.workunit.client.0.smithi057.stdout:removed './171/37' 2023-10-30T14:21:00.300 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './171' 2023-10-30T14:21:00.300 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/19' 2023-10-30T14:21:00.301 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/15' 2023-10-30T14:21:00.301 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/31' 2023-10-30T14:21:00.301 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/36' 2023-10-30T14:21:00.301 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/20' 2023-10-30T14:21:00.301 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/12' 2023-10-30T14:21:00.302 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/39' 2023-10-30T14:21:00.302 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/10' 2023-10-30T14:21:00.302 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/34' 2023-10-30T14:21:00.302 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/38' 2023-10-30T14:21:00.302 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/11' 2023-10-30T14:21:00.303 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/16' 2023-10-30T14:21:00.303 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/32' 2023-10-30T14:21:00.303 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/30' 2023-10-30T14:21:00.303 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/14' 2023-10-30T14:21:00.303 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/18' 2023-10-30T14:21:00.303 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/13' 2023-10-30T14:21:00.304 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/33' 2023-10-30T14:21:00.304 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/17' 2023-10-30T14:21:00.304 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/35' 2023-10-30T14:21:00.304 INFO:tasks.workunit.client.0.smithi057.stdout:removed './172/37' 2023-10-30T14:21:00.304 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './172' 2023-10-30T14:21:00.305 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/19' 2023-10-30T14:21:00.305 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/15' 2023-10-30T14:21:00.305 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/31' 2023-10-30T14:21:00.305 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/36' 2023-10-30T14:21:00.305 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/20' 2023-10-30T14:21:00.306 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/12' 2023-10-30T14:21:00.306 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/39' 2023-10-30T14:21:00.306 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/10' 2023-10-30T14:21:00.306 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/34' 2023-10-30T14:21:00.306 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/38' 2023-10-30T14:21:00.307 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/11' 2023-10-30T14:21:00.307 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/16' 2023-10-30T14:21:00.307 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/32' 2023-10-30T14:21:00.307 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/30' 2023-10-30T14:21:00.307 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/14' 2023-10-30T14:21:00.308 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/18' 2023-10-30T14:21:00.308 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/13' 2023-10-30T14:21:00.308 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/33' 2023-10-30T14:21:00.308 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/17' 2023-10-30T14:21:00.308 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/35' 2023-10-30T14:21:00.309 INFO:tasks.workunit.client.0.smithi057.stdout:removed './173/37' 2023-10-30T14:21:00.309 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './173' 2023-10-30T14:21:00.309 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/19' 2023-10-30T14:21:00.309 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/15' 2023-10-30T14:21:00.309 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/31' 2023-10-30T14:21:00.310 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/36' 2023-10-30T14:21:00.310 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/20' 2023-10-30T14:21:00.310 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/12' 2023-10-30T14:21:00.310 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/39' 2023-10-30T14:21:00.310 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/10' 2023-10-30T14:21:00.311 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/34' 2023-10-30T14:21:00.311 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/38' 2023-10-30T14:21:00.311 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/11' 2023-10-30T14:21:00.311 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/16' 2023-10-30T14:21:00.311 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/32' 2023-10-30T14:21:00.312 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/30' 2023-10-30T14:21:00.312 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/14' 2023-10-30T14:21:00.312 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/18' 2023-10-30T14:21:00.312 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/13' 2023-10-30T14:21:00.312 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/33' 2023-10-30T14:21:00.313 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/17' 2023-10-30T14:21:00.313 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/35' 2023-10-30T14:21:00.313 INFO:tasks.workunit.client.0.smithi057.stdout:removed './174/37' 2023-10-30T14:21:00.313 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './174' 2023-10-30T14:21:00.313 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/19' 2023-10-30T14:21:00.314 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/15' 2023-10-30T14:21:00.314 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/31' 2023-10-30T14:21:00.314 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/36' 2023-10-30T14:21:00.314 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/20' 2023-10-30T14:21:00.314 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/12' 2023-10-30T14:21:00.315 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/39' 2023-10-30T14:21:00.315 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/10' 2023-10-30T14:21:00.315 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/34' 2023-10-30T14:21:00.315 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/38' 2023-10-30T14:21:00.315 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/11' 2023-10-30T14:21:00.316 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/16' 2023-10-30T14:21:00.316 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/32' 2023-10-30T14:21:00.316 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/30' 2023-10-30T14:21:00.316 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/14' 2023-10-30T14:21:00.316 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/18' 2023-10-30T14:21:00.317 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/13' 2023-10-30T14:21:00.317 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/33' 2023-10-30T14:21:00.317 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/17' 2023-10-30T14:21:00.317 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/35' 2023-10-30T14:21:00.317 INFO:tasks.workunit.client.0.smithi057.stdout:removed './175/37' 2023-10-30T14:21:00.317 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './175' 2023-10-30T14:21:00.318 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/19' 2023-10-30T14:21:00.318 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/15' 2023-10-30T14:21:00.318 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/31' 2023-10-30T14:21:00.318 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/36' 2023-10-30T14:21:00.319 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/20' 2023-10-30T14:21:00.319 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/12' 2023-10-30T14:21:00.319 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/39' 2023-10-30T14:21:00.319 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/10' 2023-10-30T14:21:00.319 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/34' 2023-10-30T14:21:00.320 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/38' 2023-10-30T14:21:00.320 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/11' 2023-10-30T14:21:00.320 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/16' 2023-10-30T14:21:00.320 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/32' 2023-10-30T14:21:00.320 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/30' 2023-10-30T14:21:00.320 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/14' 2023-10-30T14:21:00.321 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/18' 2023-10-30T14:21:00.321 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/13' 2023-10-30T14:21:00.321 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/33' 2023-10-30T14:21:00.321 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/17' 2023-10-30T14:21:00.321 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/35' 2023-10-30T14:21:00.322 INFO:tasks.workunit.client.0.smithi057.stdout:removed './176/37' 2023-10-30T14:21:00.322 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './176' 2023-10-30T14:21:00.322 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/19' 2023-10-30T14:21:00.322 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/15' 2023-10-30T14:21:00.862 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:21:00.867 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:21:01.752 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:21:01.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.907+0000 7f59ef5ab640 1 Processor -- start 2023-10-30T14:21:01.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.907+0000 7f59ef5ab640 1 -- start start 2023-10-30T14:21:01.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.911+0000 7f59ef5ab640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59e80c5ba0 0x7f59e80c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:01.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.911+0000 7f59ef5ab640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f59e80c5f80 0x7f59e8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:01.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.911+0000 7f59ef5ab640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e815cda0 con 0x7f59e80c9b40 2023-10-30T14:21:01.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.911+0000 7f59ef5ab640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e815d990 con 0x7f59e80c5f80 2023-10-30T14:21:01.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.911+0000 7f59ef5ab640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e8153f60 con 0x7f59e80c5ba0 2023-10-30T14:21:01.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.911+0000 7f59ed347640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59e80c5ba0 0x7f59e80c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:01.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.911+0000 7f59ed347640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59e80c5ba0 0x7f59e80c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:43698/0 (socket says 172.21.15.57:43698) 2023-10-30T14:21:01.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.911+0000 7f59ed347640 1 -- 172.21.15.57:0/2895862014 learned_addr learned my addr 172.21.15.57:0/2895862014 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:21:01.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.911+0000 7f59ecb46640 1 --2- 172.21.15.57:0/2895862014 >> v2:172.21.15.88:3300/0 conn(0x7f59e80c5f80 0x7f59e8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:01.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.911+0000 7f59ed347640 1 -- 172.21.15.57:0/2895862014 >> v2:172.21.15.88:3300/0 conn(0x7f59e80c5f80 msgr2=0x7f59e8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:01.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.911+0000 7f59ed347640 1 --2- 172.21.15.57:0/2895862014 >> v2:172.21.15.88:3300/0 conn(0x7f59e80c5f80 0x7f59e8069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:01.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.911+0000 7f59ed347640 1 -- 172.21.15.57:0/2895862014 >> v1:172.21.15.88:6789/0 conn(0x7f59e80c9b40 legacy=0x7f59e81597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:01.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.911+0000 7f59ed347640 1 -- 172.21.15.57:0/2895862014 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e806f490 con 0x7f59e80c5ba0 2023-10-30T14:21:01.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59ed347640 1 --2- 172.21.15.57:0/2895862014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59e80c5ba0 0x7f59e80c70f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f59e4009870 tx=0x7f59e401c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d581e2ab2ea3030 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:01.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59dffff640 1 -- 172.21.15.57:0/2895862014 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e40ab600 con 0x7f59e80c5ba0 2023-10-30T14:21:01.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59dffff640 1 -- 172.21.15.57:0/2895862014 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e40a7050 con 0x7f59e80c5ba0 2023-10-30T14:21:01.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59dffff640 1 -- 172.21.15.57:0/2895862014 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e40ab980 con 0x7f59e80c5ba0 2023-10-30T14:21:01.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59ef5ab640 1 -- 172.21.15.57:0/2895862014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59e80c5ba0 msgr2=0x7f59e80c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:01.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59ef5ab640 1 --2- 172.21.15.57:0/2895862014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59e80c5ba0 0x7f59e80c70f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f59e4009870 tx=0x7f59e401c540 comp rx=0 tx=0).stop 2023-10-30T14:21:01.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59ef5ab640 1 -- 172.21.15.57:0/2895862014 shutdown_connections 2023-10-30T14:21:01.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59ef5ab640 1 --2- 172.21.15.57:0/2895862014 >> v2:172.21.15.88:3300/0 conn(0x7f59e80c5f80 0x7f59e8069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:01.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59ef5ab640 1 --2- 172.21.15.57:0/2895862014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59e80c5ba0 0x7f59e80c70f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:01.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59ef5ab640 1 -- 172.21.15.57:0/2895862014 >> 172.21.15.57:0/2895862014 conn(0x7f59e806afa0 msgr2=0x7f59e806b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:01.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59ef5ab640 1 -- 172.21.15.57:0/2895862014 shutdown_connections 2023-10-30T14:21:01.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59ef5ab640 1 -- 172.21.15.57:0/2895862014 wait complete. 2023-10-30T14:21:01.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59ef5ab640 1 Processor -- start 2023-10-30T14:21:01.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59ef5ab640 1 -- start start 2023-10-30T14:21:01.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59ef5ab640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59e80c5ba0 0x7f59e81f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:01.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59ef5ab640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59e80c5f80 0x7f59e8071060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:01.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59ef5ab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59e80c9b40 0x7f59e814cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:01.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59ef5ab640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e814e700 con 0x7f59e80c9b40 2023-10-30T14:21:01.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59ef5ab640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e814d6d0 con 0x7f59e80c5f80 2023-10-30T14:21:01.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.915+0000 7f59ef5ab640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e81f2a70 con 0x7f59e80c5ba0 2023-10-30T14:21:01.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.919+0000 7f59edb48640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59e80c9b40 0x7f59e814cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:01.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.919+0000 7f59edb48640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59e80c9b40 0x7f59e814cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45780/0 (socket says 172.21.15.57:45780) 2023-10-30T14:21:01.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.919+0000 7f59edb48640 1 -- 172.21.15.57:0/2310660724 learned_addr learned my addr 172.21.15.57:0/2310660724 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:21:01.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.919+0000 7f59ed347640 1 --2- 172.21.15.57:0/2310660724 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59e80c5ba0 0x7f59e81f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:01.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.919+0000 7f59ecb46640 1 --2- 172.21.15.57:0/2310660724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59e80c5f80 0x7f59e8071060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:01.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.919+0000 7f59edb48640 1 -- 172.21.15.57:0/2310660724 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59e80c5ba0 msgr2=0x7f59e81f2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:01.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.919+0000 7f59edb48640 1 --2- 172.21.15.57:0/2310660724 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59e80c5ba0 0x7f59e81f2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:01.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.919+0000 7f59edb48640 1 -- 172.21.15.57:0/2310660724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59e80c5f80 msgr2=0x7f59e8071060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:01.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.919+0000 7f59edb48640 1 --2- 172.21.15.57:0/2310660724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59e80c5f80 0x7f59e8071060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:01.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.919+0000 7f59edb48640 1 -- 172.21.15.57:0/2310660724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e40094d0 con 0x7f59e80c9b40 2023-10-30T14:21:01.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.919+0000 7f59edb48640 1 --2- 172.21.15.57:0/2310660724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59e80c9b40 0x7f59e814cb40 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f59e000c730 tx=0x7f59e000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:01.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.919+0000 7f59ddffb640 1 -- 172.21.15.57:0/2310660724 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e0003b80 con 0x7f59e80c9b40 2023-10-30T14:21:01.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.919+0000 7f59ddffb640 1 -- 172.21.15.57:0/2310660724 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e0003ce0 con 0x7f59e80c9b40 2023-10-30T14:21:01.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.919+0000 7f59ddffb640 1 -- 172.21.15.57:0/2310660724 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e0005310 con 0x7f59e80c9b40 2023-10-30T14:21:01.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.919+0000 7f59ef5ab640 1 -- 172.21.15.57:0/2310660724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e8147b50 con 0x7f59e80c9b40 2023-10-30T14:21:01.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.919+0000 7f59ef5ab640 1 -- 172.21.15.57:0/2310660724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e814db80 con 0x7f59e80c9b40 2023-10-30T14:21:01.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.919+0000 7f59ef5ab640 1 -- 172.21.15.57:0/2310660724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e804e170 con 0x7f59e80c9b40 2023-10-30T14:21:01.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.919+0000 7f59ddffb640 1 -- 172.21.15.57:0/2310660724 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f59e0005470 con 0x7f59e80c9b40 2023-10-30T14:21:01.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.923+0000 7f59ddffb640 1 --2- 172.21.15.57:0/2310660724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f59cc0664a0 0x7f59cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:01.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.923+0000 7f59ddffb640 1 -- 172.21.15.57:0/2310660724 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f59e0083da0 con 0x7f59e80c9b40 2023-10-30T14:21:01.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.923+0000 7f59ed347640 1 --2- 172.21.15.57:0/2310660724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f59cc0664a0 0x7f59cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:01.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.923+0000 7f59ed347640 1 --2- 172.21.15.57:0/2310660724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f59cc0664a0 0x7f59cc068960 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f59e40ad000 tx=0x7f59e4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:01.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:01.927+0000 7f59ddffb640 1 -- 172.21.15.57:0/2310660724 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f59e0050f00 con 0x7f59e80c9b40 2023-10-30T14:21:02.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:02.151+0000 7f59ef5ab640 1 -- 172.21.15.57:0/2310660724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59e80c3440 con 0x7f59e80c9b40 2023-10-30T14:21:02.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:02.151+0000 7f59ddffb640 1 -- 172.21.15.57:0/2310660724 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+5745 (secure 0 0 0) 0x7f59e0054520 con 0x7f59e80c9b40 2023-10-30T14:21:02.156 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:21:02.156 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:57.788102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:clientreplay","state_seq":26,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:21:02.157 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 52 2023-10-30T14:21:02.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:02.155+0000 7f59ef5ab640 1 -- 172.21.15.57:0/2310660724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f59cc0664a0 msgr2=0x7f59cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:02.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:02.155+0000 7f59ef5ab640 1 --2- 172.21.15.57:0/2310660724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f59cc0664a0 0x7f59cc068960 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f59e40ad000 tx=0x7f59e4008040 comp rx=0 tx=0).stop 2023-10-30T14:21:02.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:02.155+0000 7f59ef5ab640 1 -- 172.21.15.57:0/2310660724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59e80c9b40 msgr2=0x7f59e814cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:02.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:02.155+0000 7f59ef5ab640 1 --2- 172.21.15.57:0/2310660724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59e80c9b40 0x7f59e814cb40 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f59e000c730 tx=0x7f59e000cbf0 comp rx=0 tx=0).stop 2023-10-30T14:21:02.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:02.155+0000 7f59ef5ab640 1 -- 172.21.15.57:0/2310660724 shutdown_connections 2023-10-30T14:21:02.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:02.155+0000 7f59ef5ab640 1 --2- 172.21.15.57:0/2310660724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f59cc0664a0 0x7f59cc068960 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:02.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:02.155+0000 7f59ef5ab640 1 --2- 172.21.15.57:0/2310660724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59e80c9b40 0x7f59e814cb40 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:02.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:02.155+0000 7f59ef5ab640 1 --2- 172.21.15.57:0/2310660724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59e80c5f80 0x7f59e8071060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:02.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:02.155+0000 7f59ef5ab640 1 --2- 172.21.15.57:0/2310660724 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59e80c5ba0 0x7f59e81f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:02.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:02.155+0000 7f59ef5ab640 1 -- 172.21.15.57:0/2310660724 >> 172.21.15.57:0/2310660724 conn(0x7f59e806afa0 msgr2=0x7f59e8071d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:02.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:02.155+0000 7f59ef5ab640 1 -- 172.21.15.57:0/2310660724 shutdown_connections 2023-10-30T14:21:02.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:02.155+0000 7f59ef5ab640 1 -- 172.21.15.57:0/2310660724 wait complete. 2023-10-30T14:21:02.182 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:21:04.182 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:21:04.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.327+0000 7fc5e43f8640 1 Processor -- start 2023-10-30T14:21:04.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.327+0000 7fc5e43f8640 1 -- start start 2023-10-30T14:21:04.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.331+0000 7fc5e43f8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc5dc06b760 0x7fc5dc15e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:04.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.331+0000 7fc5e43f8640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc5dc06c420 0x7fc5dc0c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:04.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.331+0000 7fc5e43f8640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5dc071e00 con 0x7fc5dc06bb40 2023-10-30T14:21:04.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.331+0000 7fc5e43f8640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5dc070020 con 0x7fc5dc06c420 2023-10-30T14:21:04.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.331+0000 7fc5e43f8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5dc15eb90 con 0x7fc5dc06b760 2023-10-30T14:21:04.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.331+0000 7fc5e1993640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc5dc06bb40 0x7fc5dc157a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:54814/0 (socket says 172.21.15.57:54814) 2023-10-30T14:21:04.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.331+0000 7fc5e1993640 1 -- 172.21.15.57:0/4201774011 learned_addr learned my addr 172.21.15.57:0/4201774011 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:21:04.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.331+0000 7fc5e2995640 1 --2- 172.21.15.57:0/4201774011 >> v2:172.21.15.57:3300/0 conn(0x7fc5dc06c420 0x7fc5dc0c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:04.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.331+0000 7fc5e2194640 1 --2- 172.21.15.57:0/4201774011 >> v2:172.21.15.88:3300/0 conn(0x7fc5dc06b760 0x7fc5dc15e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:04.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.331+0000 7fc5e2995640 1 -- 172.21.15.57:0/4201774011 >> v2:172.21.15.88:3300/0 conn(0x7fc5dc06b760 msgr2=0x7fc5dc15e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:04.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.331+0000 7fc5e2995640 1 --2- 172.21.15.57:0/4201774011 >> v2:172.21.15.88:3300/0 conn(0x7fc5dc06b760 0x7fc5dc15e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:04.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.331+0000 7fc5e2995640 1 -- 172.21.15.57:0/4201774011 >> v1:172.21.15.57:6789/0 conn(0x7fc5dc06bb40 legacy=0x7fc5dc157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:04.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.331+0000 7fc5e2995640 1 -- 172.21.15.57:0/4201774011 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5dc070f10 con 0x7fc5dc06c420 2023-10-30T14:21:04.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.331+0000 7fc5e2194640 1 --2- 172.21.15.57:0/4201774011 >> v2:172.21.15.88:3300/0 conn(0x7fc5dc06b760 0x7fc5dc15e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:21:04.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e2995640 1 --2- 172.21.15.57:0/4201774011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5dc06c420 0x7fc5dc0c57a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0cb840 tx=0x7fc5d401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51e349d1fa7b274c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:04.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e1192640 1 -- 172.21.15.57:0/4201774011 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d40a59c0 con 0x7fc5dc06c420 2023-10-30T14:21:04.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e1192640 1 -- 172.21.15.57:0/4201774011 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d40a5b20 con 0x7fc5dc06c420 2023-10-30T14:21:04.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e1192640 1 -- 172.21.15.57:0/4201774011 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d40ad440 con 0x7fc5dc06c420 2023-10-30T14:21:04.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e43f8640 1 -- 172.21.15.57:0/4201774011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5dc06c420 msgr2=0x7fc5dc0c57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:04.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e43f8640 1 --2- 172.21.15.57:0/4201774011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5dc06c420 0x7fc5dc0c57a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0cb840 tx=0x7fc5d401c8e0 comp rx=0 tx=0).stop 2023-10-30T14:21:04.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e43f8640 1 -- 172.21.15.57:0/4201774011 shutdown_connections 2023-10-30T14:21:04.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e43f8640 1 --2- 172.21.15.57:0/4201774011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5dc06c420 0x7fc5dc0c57a0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:04.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e43f8640 1 --2- 172.21.15.57:0/4201774011 >> v2:172.21.15.88:3300/0 conn(0x7fc5dc06b760 0x7fc5dc15e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:04.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e43f8640 1 -- 172.21.15.57:0/4201774011 >> 172.21.15.57:0/4201774011 conn(0x7fc5dc0cac70 msgr2=0x7fc5dc0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:04.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e43f8640 1 -- 172.21.15.57:0/4201774011 shutdown_connections 2023-10-30T14:21:04.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e43f8640 1 -- 172.21.15.57:0/4201774011 wait complete. 2023-10-30T14:21:04.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e43f8640 1 Processor -- start 2023-10-30T14:21:04.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e43f8640 1 -- start start 2023-10-30T14:21:04.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e43f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5dc06b760 0x7fc5dc14cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:04.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e43f8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5dc06bb40 0x7fc5dc1fa360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:04.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e43f8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5dc06c420 0x7fc5dc1fa8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:04.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e43f8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc1f9f40 con 0x7fc5dc06bb40 2023-10-30T14:21:04.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e43f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc160cd0 con 0x7fc5dc06b760 2023-10-30T14:21:04.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e43f8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc15ed90 con 0x7fc5dc06c420 2023-10-30T14:21:04.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.335+0000 7fc5e1993640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5dc06bb40 0x7fc5dc1fa360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:04.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.339+0000 7fc5e1993640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5dc06bb40 0x7fc5dc1fa360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40350/0 (socket says 172.21.15.57:40350) 2023-10-30T14:21:04.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.339+0000 7fc5e1993640 1 -- 172.21.15.57:0/2277864990 learned_addr learned my addr 172.21.15.57:0/2277864990 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:21:04.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.339+0000 7fc5e2995640 1 --2- 172.21.15.57:0/2277864990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5dc06c420 0x7fc5dc1fa8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:04.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.339+0000 7fc5e2194640 1 --2- 172.21.15.57:0/2277864990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5dc06b760 0x7fc5dc14cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:04.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.339+0000 7fc5e1993640 1 -- 172.21.15.57:0/2277864990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5dc06c420 msgr2=0x7fc5dc1fa8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:04.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.339+0000 7fc5e1993640 1 --2- 172.21.15.57:0/2277864990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5dc06c420 0x7fc5dc1fa8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:04.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.339+0000 7fc5e1993640 1 -- 172.21.15.57:0/2277864990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5dc06b760 msgr2=0x7fc5dc14cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:04.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.339+0000 7fc5e1993640 1 --2- 172.21.15.57:0/2277864990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5dc06b760 0x7fc5dc14cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:04.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.339+0000 7fc5e1993640 1 -- 172.21.15.57:0/2277864990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d4009a70 con 0x7fc5dc06bb40 2023-10-30T14:21:04.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.339+0000 7fc5e2995640 1 --2- 172.21.15.57:0/2277864990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5dc06c420 0x7fc5dc1fa8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:21:04.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.339+0000 7fc5e1993640 1 --2- 172.21.15.57:0/2277864990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5dc06bb40 0x7fc5dc1fa360 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00c4d0 tx=0x7fc5cc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:04.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.339+0000 7fc5d2ffd640 1 -- 172.21.15.57:0/2277864990 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc005cf0 con 0x7fc5dc06bb40 2023-10-30T14:21:04.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.339+0000 7fc5d2ffd640 1 -- 172.21.15.57:0/2277864990 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5cc005e50 con 0x7fc5dc06bb40 2023-10-30T14:21:04.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.339+0000 7fc5d2ffd640 1 -- 172.21.15.57:0/2277864990 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc007270 con 0x7fc5dc06bb40 2023-10-30T14:21:04.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.339+0000 7fc5e43f8640 1 -- 172.21.15.57:0/2277864990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5dc15f050 con 0x7fc5dc06bb40 2023-10-30T14:21:04.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.339+0000 7fc5e43f8640 1 -- 172.21.15.57:0/2277864990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5dc15f610 con 0x7fc5dc06bb40 2023-10-30T14:21:04.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.339+0000 7fc5e43f8640 1 -- 172.21.15.57:0/2277864990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5dc04e170 con 0x7fc5dc06bb40 2023-10-30T14:21:04.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.339+0000 7fc5d2ffd640 1 -- 172.21.15.57:0/2277864990 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc5cc0073d0 con 0x7fc5dc06bb40 2023-10-30T14:21:04.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.343+0000 7fc5d2ffd640 1 --2- 172.21.15.57:0/2277864990 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5c00664a0 0x7fc5c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:04.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.343+0000 7fc5d2ffd640 1 -- 172.21.15.57:0/2277864990 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fc5cc0834a0 con 0x7fc5dc06bb40 2023-10-30T14:21:04.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.343+0000 7fc5e2194640 1 --2- 172.21.15.57:0/2277864990 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5c00664a0 0x7fc5c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:04.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.343+0000 7fc5e2194640 1 --2- 172.21.15.57:0/2277864990 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5c00664a0 0x7fc5c0068960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8009950 tx=0x7fc5d8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:04.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.347+0000 7fc5d2ffd640 1 -- 172.21.15.57:0/2277864990 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc5cc0507b0 con 0x7fc5dc06bb40 2023-10-30T14:21:04.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.539+0000 7fc5e43f8640 1 -- 172.21.15.57:0/2277864990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5dc153e80 con 0x7fc5dc06bb40 2023-10-30T14:21:04.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.539+0000 7fc5d2ffd640 1 -- 172.21.15.57:0/2277864990 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+5745 (secure 0 0 0) 0x7fc5cc053dd0 con 0x7fc5dc06bb40 2023-10-30T14:21:04.544 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:21:04.544 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:20:57.788102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:clientreplay","state_seq":26,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:21:04.544 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 52 2023-10-30T14:21:04.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.539+0000 7fc5e43f8640 1 -- 172.21.15.57:0/2277864990 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5c00664a0 msgr2=0x7fc5c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:04.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.539+0000 7fc5e43f8640 1 --2- 172.21.15.57:0/2277864990 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5c00664a0 0x7fc5c0068960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8009950 tx=0x7fc5d8008040 comp rx=0 tx=0).stop 2023-10-30T14:21:04.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.539+0000 7fc5e43f8640 1 -- 172.21.15.57:0/2277864990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5dc06bb40 msgr2=0x7fc5dc1fa360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:04.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.539+0000 7fc5e43f8640 1 --2- 172.21.15.57:0/2277864990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5dc06bb40 0x7fc5dc1fa360 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00c4d0 tx=0x7fc5cc00c990 comp rx=0 tx=0).stop 2023-10-30T14:21:04.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.539+0000 7fc5e43f8640 1 -- 172.21.15.57:0/2277864990 shutdown_connections 2023-10-30T14:21:04.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.539+0000 7fc5e43f8640 1 --2- 172.21.15.57:0/2277864990 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5c00664a0 0x7fc5c0068960 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:04.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.539+0000 7fc5e43f8640 1 --2- 172.21.15.57:0/2277864990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5dc06c420 0x7fc5dc1fa8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:04.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.539+0000 7fc5e43f8640 1 --2- 172.21.15.57:0/2277864990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5dc06bb40 0x7fc5dc1fa360 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:04.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.539+0000 7fc5e43f8640 1 --2- 172.21.15.57:0/2277864990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5dc06b760 0x7fc5dc14cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:04.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.539+0000 7fc5e43f8640 1 -- 172.21.15.57:0/2277864990 >> 172.21.15.57:0/2277864990 conn(0x7fc5dc0cac70 msgr2=0x7fc5dc072550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:04.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.539+0000 7fc5e43f8640 1 -- 172.21.15.57:0/2277864990 shutdown_connections 2023-10-30T14:21:04.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:04.539+0000 7fc5e43f8640 1 -- 172.21.15.57:0/2277864990 wait complete. 2023-10-30T14:21:04.565 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:21:06.566 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:21:06.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.715+0000 7f4d94017640 1 Processor -- start 2023-10-30T14:21:06.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.715+0000 7f4d94017640 1 -- start start 2023-10-30T14:21:06.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.715+0000 7f4d94017640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4d8c0c4a10 0x7f4d8c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:06.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.715+0000 7f4d94017640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4d8c0c42a0 0x7f4d8c0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:06.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.715+0000 7f4d94017640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8c159f20 con 0x7f4d8c153c80 2023-10-30T14:21:06.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.715+0000 7f4d94017640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d8c15dec0 con 0x7f4d8c0c4a10 2023-10-30T14:21:06.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.715+0000 7f4d94017640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d8c1599a0 con 0x7f4d8c0c42a0 2023-10-30T14:21:06.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.715+0000 7f4d915b2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4d8c0c4a10 0x7f4d8c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:06.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.715+0000 7f4d91db3640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f4d8c153c80 0x7f4d8c15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:54822/0 (socket says 172.21.15.57:54822) 2023-10-30T14:21:06.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.715+0000 7f4d91db3640 1 -- 172.21.15.57:0/2935438437 learned_addr learned my addr 172.21.15.57:0/2935438437 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:21:06.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.715+0000 7f4d915b2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4d8c0c4a10 0x7f4d8c15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40352/0 (socket says 172.21.15.57:40352) 2023-10-30T14:21:06.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.715+0000 7f4d925b4640 1 --2- 172.21.15.57:0/2935438437 >> v2:172.21.15.88:3300/0 conn(0x7f4d8c0c42a0 0x7f4d8c0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:06.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.715+0000 7f4d915b2640 1 -- 172.21.15.57:0/2935438437 >> v2:172.21.15.88:3300/0 conn(0x7f4d8c0c42a0 msgr2=0x7f4d8c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:06.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.715+0000 7f4d915b2640 1 --2- 172.21.15.57:0/2935438437 >> v2:172.21.15.88:3300/0 conn(0x7f4d8c0c42a0 0x7f4d8c0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:06.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.715+0000 7f4d915b2640 1 -- 172.21.15.57:0/2935438437 >> v1:172.21.15.57:6789/0 conn(0x7f4d8c153c80 legacy=0x7f4d8c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:06.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.715+0000 7f4d915b2640 1 -- 172.21.15.57:0/2935438437 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8c15d620 con 0x7f4d8c0c4a10 2023-10-30T14:21:06.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.715+0000 7f4d90db1640 1 -- 172.21.15.57:0/2935438437 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359334887 0 0) 0x7f4d8c159f20 con 0x7f4d8c153c80 2023-10-30T14:21:06.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.719+0000 7f4d915b2640 1 --2- 172.21.15.57:0/2935438437 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d8c0c4a10 0x7f4d8c15cf90 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c0b1d40 tx=0x7f4d7c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e3c16f59d1ef630 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:06.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.719+0000 7f4d90db1640 1 -- 172.21.15.57:0/2935438437 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c0a8950 con 0x7f4d8c0c4a10 2023-10-30T14:21:06.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.719+0000 7f4d90db1640 1 -- 172.21.15.57:0/2935438437 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d7c0a8ab0 con 0x7f4d8c0c4a10 2023-10-30T14:21:06.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.719+0000 7f4d90db1640 1 -- 172.21.15.57:0/2935438437 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c0a8e40 con 0x7f4d8c0c4a10 2023-10-30T14:21:06.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.719+0000 7f4d94017640 1 -- 172.21.15.57:0/2935438437 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d8c0c4a10 msgr2=0x7f4d8c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:06.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.719+0000 7f4d94017640 1 --2- 172.21.15.57:0/2935438437 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d8c0c4a10 0x7f4d8c15cf90 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c0b1d40 tx=0x7f4d7c01ec50 comp rx=0 tx=0).stop 2023-10-30T14:21:06.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.719+0000 7f4d94017640 1 -- 172.21.15.57:0/2935438437 shutdown_connections 2023-10-30T14:21:06.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.719+0000 7f4d94017640 1 --2- 172.21.15.57:0/2935438437 >> v2:172.21.15.88:3300/0 conn(0x7f4d8c0c42a0 0x7f4d8c0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:06.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.719+0000 7f4d94017640 1 --2- 172.21.15.57:0/2935438437 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d8c0c4a10 0x7f4d8c15cf90 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:06.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.719+0000 7f4d94017640 1 -- 172.21.15.57:0/2935438437 >> 172.21.15.57:0/2935438437 conn(0x7f4d8c0715c0 msgr2=0x7f4d8c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:06.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.719+0000 7f4d94017640 1 -- 172.21.15.57:0/2935438437 shutdown_connections 2023-10-30T14:21:06.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.719+0000 7f4d94017640 1 -- 172.21.15.57:0/2935438437 wait complete. 2023-10-30T14:21:06.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d94017640 1 Processor -- start 2023-10-30T14:21:06.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d94017640 1 -- start start 2023-10-30T14:21:06.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d94017640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d8c0c42a0 0x7f4d8c155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:06.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d94017640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d8c0c4a10 0x7f4d8c1f2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:06.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d94017640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d8c153c80 0x7f4d8c1f2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:06.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d94017640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c14ea10 con 0x7f4d8c0c42a0 2023-10-30T14:21:06.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d94017640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c14ded0 con 0x7f4d8c0c4a10 2023-10-30T14:21:06.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d94017640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c1541e0 con 0x7f4d8c153c80 2023-10-30T14:21:06.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d91db3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d8c0c42a0 0x7f4d8c155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:06.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d91db3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d8c0c42a0 0x7f4d8c155cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40362/0 (socket says 172.21.15.57:40362) 2023-10-30T14:21:06.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d91db3640 1 -- 172.21.15.57:0/1897053964 learned_addr learned my addr 172.21.15.57:0/1897053964 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:21:06.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d925b4640 1 --2- 172.21.15.57:0/1897053964 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d8c153c80 0x7f4d8c1f2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:06.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d915b2640 1 --2- 172.21.15.57:0/1897053964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d8c0c4a10 0x7f4d8c1f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:06.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d91db3640 1 -- 172.21.15.57:0/1897053964 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d8c153c80 msgr2=0x7f4d8c1f2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:06.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d91db3640 1 --2- 172.21.15.57:0/1897053964 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d8c153c80 0x7f4d8c1f2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:06.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d91db3640 1 -- 172.21.15.57:0/1897053964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d8c0c4a10 msgr2=0x7f4d8c1f2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:06.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d91db3640 1 --2- 172.21.15.57:0/1897053964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d8c0c4a10 0x7f4d8c1f2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:06.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d91db3640 1 -- 172.21.15.57:0/1897053964 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d7c009420 con 0x7f4d8c0c42a0 2023-10-30T14:21:06.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d915b2640 1 --2- 172.21.15.57:0/1897053964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d8c0c4a10 0x7f4d8c1f2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:21:06.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d91db3640 1 --2- 172.21.15.57:0/1897053964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d8c0c42a0 0x7f4d8c155cb0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4d8800c480 tx=0x7f4d8800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:06.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d827fc640 1 -- 172.21.15.57:0/1897053964 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d880074e0 con 0x7f4d8c0c42a0 2023-10-30T14:21:06.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d827fc640 1 -- 172.21.15.57:0/1897053964 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d88007660 con 0x7f4d8c0c42a0 2023-10-30T14:21:06.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d827fc640 1 -- 172.21.15.57:0/1897053964 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d88004a30 con 0x7f4d8c0c42a0 2023-10-30T14:21:06.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d94017640 1 -- 172.21.15.57:0/1897053964 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8c14e0e0 con 0x7f4d8c0c42a0 2023-10-30T14:21:06.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.723+0000 7f4d94017640 1 -- 172.21.15.57:0/1897053964 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8c14e6a0 con 0x7f4d8c0c42a0 2023-10-30T14:21:06.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.727+0000 7f4d94017640 1 -- 172.21.15.57:0/1897053964 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d8c04e170 con 0x7f4d8c0c42a0 2023-10-30T14:21:06.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.727+0000 7f4d827fc640 1 -- 172.21.15.57:0/1897053964 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4d88004b90 con 0x7f4d8c0c42a0 2023-10-30T14:21:06.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.727+0000 7f4d827fc640 1 --2- 172.21.15.57:0/1897053964 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4d700664d0 0x7f4d70068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:06.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.727+0000 7f4d827fc640 1 -- 172.21.15.57:0/1897053964 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f4d88083280 con 0x7f4d8c0c42a0 2023-10-30T14:21:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.727+0000 7f4d915b2640 1 --2- 172.21.15.57:0/1897053964 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4d700664d0 0x7f4d70068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.727+0000 7f4d915b2640 1 --2- 172.21.15.57:0/1897053964 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4d700664d0 0x7f4d70068990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c0b2000 tx=0x7f4d7c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.731+0000 7f4d827fc640 1 -- 172.21.15.57:0/1897053964 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4d88053f60 con 0x7f4d8c0c42a0 2023-10-30T14:21:06.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.923+0000 7f4d94017640 1 -- 172.21.15.57:0/1897053964 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d8c1573b0 con 0x7f4d8c0c42a0 2023-10-30T14:21:06.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.923+0000 7f4d827fc640 1 -- 172.21.15.57:0/1897053964 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f4d88053af0 con 0x7f4d8c0c42a0 2023-10-30T14:21:06.928 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:21:06.928 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:21:03.820033+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:active","state_seq":28,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:21:06.929 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 53 2023-10-30T14:21:06.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.923+0000 7f4d94017640 1 -- 172.21.15.57:0/1897053964 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4d700664d0 msgr2=0x7f4d70068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:06.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.923+0000 7f4d94017640 1 --2- 172.21.15.57:0/1897053964 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4d700664d0 0x7f4d70068990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c0b2000 tx=0x7f4d7c008040 comp rx=0 tx=0).stop 2023-10-30T14:21:06.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.923+0000 7f4d94017640 1 -- 172.21.15.57:0/1897053964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d8c0c42a0 msgr2=0x7f4d8c155cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:06.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.923+0000 7f4d94017640 1 --2- 172.21.15.57:0/1897053964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d8c0c42a0 0x7f4d8c155cb0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4d8800c480 tx=0x7f4d8800c940 comp rx=0 tx=0).stop 2023-10-30T14:21:06.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.923+0000 7f4d94017640 1 -- 172.21.15.57:0/1897053964 shutdown_connections 2023-10-30T14:21:06.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.923+0000 7f4d94017640 1 --2- 172.21.15.57:0/1897053964 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4d700664d0 0x7f4d70068990 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:06.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.923+0000 7f4d94017640 1 --2- 172.21.15.57:0/1897053964 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d8c153c80 0x7f4d8c1f2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:06.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.923+0000 7f4d94017640 1 --2- 172.21.15.57:0/1897053964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d8c0c4a10 0x7f4d8c1f2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:06.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.923+0000 7f4d94017640 1 --2- 172.21.15.57:0/1897053964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d8c0c42a0 0x7f4d8c155cb0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:06.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.923+0000 7f4d94017640 1 -- 172.21.15.57:0/1897053964 >> 172.21.15.57:0/1897053964 conn(0x7f4d8c0715c0 msgr2=0x7f4d8c1587b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:06.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.923+0000 7f4d94017640 1 -- 172.21.15.57:0/1897053964 shutdown_connections 2023-10-30T14:21:06.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:06.923+0000 7f4d94017640 1 -- 172.21.15.57:0/1897053964 wait complete. 2023-10-30T14:21:06.949 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=2, replacing gid=4936 2023-10-30T14:21:06.949 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.e 2023-10-30T14:21:07.950 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T14:21:07.950 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T14:21:07.951 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T14:21:07.955 INFO:tasks.ceph.mds.e:Started 2023-10-30T14:21:07.955 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:21:08.053 INFO:tasks.ceph.mds.e.smithi057.stdout:starting mds.e at 2023-10-30T14:21:08.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7bef1d640 1 Processor -- start 2023-10-30T14:21:08.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7bef1d640 1 -- start start 2023-10-30T14:21:08.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7bef1d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7b80c2c10 0x7fa7b80cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:08.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7bef1d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b8151aa0 con 0x7fa7b80ccf60 2023-10-30T14:21:08.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7bef1d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b8158af0 con 0x7fa7b81516c0 2023-10-30T14:21:08.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7bef1d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b80c74a0 con 0x7fa7b80c2c10 2023-10-30T14:21:08.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b7fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa7b80ccf60 0x7fa7b80c2500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:54838/0 (socket says 172.21.15.57:54838) 2023-10-30T14:21:08.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b7fff640 1 -- 172.21.15.57:0/2702782509 learned_addr learned my addr 172.21.15.57:0/2702782509 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:21:08.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7bd4ba640 1 --2- 172.21.15.57:0/2702782509 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7b80c2c10 0x7fa7b80cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:08.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b77fe640 1 -- 172.21.15.57:0/2702782509 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564057661 0 0) 0x7fa7b8151aa0 con 0x7fa7b80ccf60 2023-10-30T14:21:08.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b77fe640 1 -- 172.21.15.57:0/2702782509 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a8003530 con 0x7fa7b80ccf60 2023-10-30T14:21:08.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b77fe640 1 -- 172.21.15.57:0/2702782509 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2838537410 0 0) 0x7fa7a8003530 con 0x7fa7b80ccf60 2023-10-30T14:21:08.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b77fe640 1 -- 172.21.15.57:0/2702782509 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7b80c2c10 msgr2=0x7fa7b80cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:08.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b77fe640 1 --2- 172.21.15.57:0/2702782509 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7b80c2c10 0x7fa7b80cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b77fe640 1 -- 172.21.15.57:0/2702782509 >> v1:172.21.15.88:6789/0 conn(0x7fa7b81516c0 legacy=0x7fa7b80c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:08.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b77fe640 1 -- 172.21.15.57:0/2702782509 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b8158510 con 0x7fa7b80ccf60 2023-10-30T14:21:08.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b77fe640 1 -- 172.21.15.57:0/2702782509 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fa7a4001680 con 0x7fa7b80ccf60 2023-10-30T14:21:08.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b77fe640 1 -- 172.21.15.57:0/2702782509 >> v1:172.21.15.57:6789/0 conn(0x7fa7b80ccf60 legacy=0x7fa7b80c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:08.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b77fe640 1 --2- 172.21.15.57:0/2702782509 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7b80c2c10 0x7fa7a80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:08.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b77fe640 1 --2- 172.21.15.57:0/2702782509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7a8003ef0 0x7fa7a80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:08.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b77fe640 1 --2- 172.21.15.57:0/2702782509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7a8004d10 0x7fa7a8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:08.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b77fe640 1 -- 172.21.15.57:0/2702782509 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b80c74a0 con 0x7fa7a8003ef0 2023-10-30T14:21:08.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b77fe640 1 -- 172.21.15.57:0/2702782509 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a800b9f0 con 0x7fa7a8004d10 2023-10-30T14:21:08.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b77fe640 1 -- 172.21.15.57:0/2702782509 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a800bbb0 con 0x7fa7b80c2c10 2023-10-30T14:21:08.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b7fff640 1 --2- 172.21.15.57:0/2702782509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7a8003ef0 0x7fa7a80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:08.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7bd4ba640 1 --2- 172.21.15.57:0/2702782509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7a8004d10 0x7fa7a8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:08.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7bccb9640 1 --2- 172.21.15.57:0/2702782509 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7b80c2c10 0x7fa7a80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:08.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b7fff640 1 -- 172.21.15.57:0/2702782509 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7b80c2c10 msgr2=0x7fa7a80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:08.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b7fff640 1 --2- 172.21.15.57:0/2702782509 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7b80c2c10 0x7fa7a80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b7fff640 1 -- 172.21.15.57:0/2702782509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7a8004d10 msgr2=0x7fa7a8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:08.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b7fff640 1 --2- 172.21.15.57:0/2702782509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7a8004d10 0x7fa7a8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.107+0000 7fa7b7fff640 1 -- 172.21.15.57:0/2702782509 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a8003530 con 0x7fa7a8003ef0 2023-10-30T14:21:08.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.111+0000 7fa7b7fff640 1 --2- 172.21.15.57:0/2702782509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7a8003ef0 0x7fa7a80047d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400c190 tx=0x7fa7a401cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b354ca11b9ec6458 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:08.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.111+0000 7fa7b77fe640 1 -- 172.21.15.57:0/2702782509 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a40058e0 con 0x7fa7a8003ef0 2023-10-30T14:21:08.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.111+0000 7fa7b77fe640 1 -- 172.21.15.57:0/2702782509 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a4005a60 con 0x7fa7a8003ef0 2023-10-30T14:21:08.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.111+0000 7fa7b77fe640 1 -- 172.21.15.57:0/2702782509 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4005e90 con 0x7fa7a8003ef0 2023-10-30T14:21:08.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.111+0000 7fa7bef1d640 1 -- 172.21.15.57:0/2702782509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7a8003ef0 msgr2=0x7fa7a80047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:08.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.111+0000 7fa7bef1d640 1 --2- 172.21.15.57:0/2702782509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7a8003ef0 0x7fa7a80047d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400c190 tx=0x7fa7a401cae0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.111+0000 7fa7bd4ba640 1 -- 172.21.15.57:0/2702782509 reap_dead start 2023-10-30T14:21:08.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.111+0000 7fa7bef1d640 1 -- 172.21.15.57:0/2702782509 shutdown_connections 2023-10-30T14:21:08.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.111+0000 7fa7bef1d640 1 -- 172.21.15.57:0/2702782509 >> 172.21.15.57:0/2702782509 conn(0x7fa7b8071fb0 msgr2=0x7fa7b80723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:08.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.111+0000 7fa7bef1d640 1 -- 172.21.15.57:0/2702782509 shutdown_connections 2023-10-30T14:21:08.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.111+0000 7fa7bef1d640 1 -- 172.21.15.57:0/2702782509 wait complete. 2023-10-30T14:21:08.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.115+0000 7fa7bef1d640 1 Processor -- start 2023-10-30T14:21:08.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.115+0000 7fa7bef1d640 1 -- start start 2023-10-30T14:21:08.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.115+0000 7fa7bef1d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7a8003ef0 0x7fa7b8157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:08.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.115+0000 7fa7bef1d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7b8155170 0x7fa7b806c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:08.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.115+0000 7fa7bef1d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7b81eec60 0x7fa7b81ef0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:08.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.115+0000 7fa7bef1d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b8153e20 con 0x7fa7b8155170 2023-10-30T14:21:08.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.115+0000 7fa7bef1d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b81576c0 con 0x7fa7b81eec60 2023-10-30T14:21:08.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.115+0000 7fa7bef1d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b80c6280 con 0x7fa7a8003ef0 2023-10-30T14:21:08.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.115+0000 7fa7b7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7b8155170 0x7fa7b806c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:08.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.115+0000 7fa7b7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7b8155170 0x7fa7b806c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40372/0 (socket says 172.21.15.57:40372) 2023-10-30T14:21:08.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.115+0000 7fa7b7fff640 1 -- 172.21.15.57:0/2581201180 learned_addr learned my addr 172.21.15.57:0/2581201180 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:21:08.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.115+0000 7fa7bccb9640 1 --2- 172.21.15.57:0/2581201180 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7a8003ef0 0x7fa7b8157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:08.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.115+0000 7fa7bd4ba640 1 --2- 172.21.15.57:0/2581201180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7b81eec60 0x7fa7b81ef0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:08.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.115+0000 7fa7b7fff640 1 -- 172.21.15.57:0/2581201180 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7a8003ef0 msgr2=0x7fa7b8157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:08.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.115+0000 7fa7b7fff640 1 --2- 172.21.15.57:0/2581201180 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7a8003ef0 0x7fa7b8157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.115+0000 7fa7b7fff640 1 -- 172.21.15.57:0/2581201180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7b81eec60 msgr2=0x7fa7b81ef0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:08.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.115+0000 7fa7b7fff640 1 --2- 172.21.15.57:0/2581201180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7b81eec60 0x7fa7b81ef0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.115+0000 7fa7b7fff640 1 -- 172.21.15.57:0/2581201180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b80c6560 con 0x7fa7b8155170 2023-10-30T14:21:08.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.115+0000 7fa7b7fff640 1 --2- 172.21.15.57:0/2581201180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7b8155170 0x7fa7b806c250 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400c190 tx=0x7fa7a8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:08.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.119+0000 7fa7b57fa640 1 -- 172.21.15.57:0/2581201180 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4004a00 con 0x7fa7b8155170 2023-10-30T14:21:08.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.119+0000 7fa7b57fa640 1 -- 172.21.15.57:0/2581201180 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a4004b60 con 0x7fa7b8155170 2023-10-30T14:21:08.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.119+0000 7fa7b57fa640 1 -- 172.21.15.57:0/2581201180 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a40a57e0 con 0x7fa7b8155170 2023-10-30T14:21:08.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.119+0000 7fa7bef1d640 1 -- 172.21.15.57:0/2581201180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b8154090 con 0x7fa7b8155170 2023-10-30T14:21:08.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.119+0000 7fa7bef1d640 1 -- 172.21.15.57:0/2581201180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b8154650 con 0x7fa7b8155170 2023-10-30T14:21:08.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.119+0000 7fa7bef1d640 1 -- 172.21.15.57:0/2581201180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b804e170 con 0x7fa7b8155170 2023-10-30T14:21:08.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.119+0000 7fa7b57fa640 1 -- 172.21.15.57:0/2581201180 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa7a40a5940 con 0x7fa7b8155170 2023-10-30T14:21:08.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.119+0000 7fa7b57fa640 1 --2- 172.21.15.57:0/2581201180 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa79c066470 0x7fa79c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:08.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.119+0000 7fa7b57fa640 1 -- 172.21.15.57:0/2581201180 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fa7a411ddc0 con 0x7fa7b8155170 2023-10-30T14:21:08.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.119+0000 7fa7bccb9640 1 --2- 172.21.15.57:0/2581201180 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa79c066470 0x7fa79c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:08.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.123+0000 7fa7bccb9640 1 --2- 172.21.15.57:0/2581201180 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa79c066470 0x7fa79c068930 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac007fd0 tx=0x7fa7ac007cd0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:08.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.123+0000 7fa7b57fa640 1 -- 172.21.15.57:0/2581201180 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa7a40eaf90 con 0x7fa7b8155170 2023-10-30T14:21:08.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.311+0000 7fa7bef1d640 1 -- 172.21.15.57:0/2581201180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7b8154f20 con 0x7fa7b8155170 2023-10-30T14:21:08.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.315+0000 7fa7b57fa640 1 -- 172.21.15.57:0/2581201180 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+5739 (secure 0 0 0) 0x7fa7a40ee5b0 con 0x7fa7b8155170 2023-10-30T14:21:08.319 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:21:08.319 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:21:03.820033+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:active","state_seq":28,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:21:08.320 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 53 2023-10-30T14:21:08.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.315+0000 7fa7bef1d640 1 -- 172.21.15.57:0/2581201180 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa79c066470 msgr2=0x7fa79c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:08.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.315+0000 7fa7bef1d640 1 --2- 172.21.15.57:0/2581201180 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa79c066470 0x7fa79c068930 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac007fd0 tx=0x7fa7ac007cd0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.315+0000 7fa7bef1d640 1 -- 172.21.15.57:0/2581201180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7b8155170 msgr2=0x7fa7b806c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:08.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.315+0000 7fa7bef1d640 1 --2- 172.21.15.57:0/2581201180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7b8155170 0x7fa7b806c250 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400c190 tx=0x7fa7a8005800 comp rx=0 tx=0).stop 2023-10-30T14:21:08.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.315+0000 7fa7bef1d640 1 -- 172.21.15.57:0/2581201180 shutdown_connections 2023-10-30T14:21:08.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.315+0000 7fa7bef1d640 1 --2- 172.21.15.57:0/2581201180 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa79c066470 0x7fa79c068930 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.315+0000 7fa7bef1d640 1 --2- 172.21.15.57:0/2581201180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7b81eec60 0x7fa7b81ef0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.315+0000 7fa7bef1d640 1 --2- 172.21.15.57:0/2581201180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7b8155170 0x7fa7b806c250 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.315+0000 7fa7bef1d640 1 --2- 172.21.15.57:0/2581201180 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7a8003ef0 0x7fa7b8157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.315+0000 7fa7bef1d640 1 -- 172.21.15.57:0/2581201180 >> 172.21.15.57:0/2581201180 conn(0x7fa7b8071fb0 msgr2=0x7fa7b80cb230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:08.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.315+0000 7fa7bef1d640 1 -- 172.21.15.57:0/2581201180 shutdown_connections 2023-10-30T14:21:08.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:08.315+0000 7fa7bef1d640 1 -- 172.21.15.57:0/2581201180 wait complete. 2023-10-30T14:21:08.340 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-30T14:21:10.341 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:21:10.445 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/31' 2023-10-30T14:21:10.445 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/36' 2023-10-30T14:21:10.445 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/20' 2023-10-30T14:21:10.445 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/12' 2023-10-30T14:21:10.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/39' 2023-10-30T14:21:10.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/10' 2023-10-30T14:21:10.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/34' 2023-10-30T14:21:10.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/38' 2023-10-30T14:21:10.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/11' 2023-10-30T14:21:10.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/16' 2023-10-30T14:21:10.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/32' 2023-10-30T14:21:10.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/30' 2023-10-30T14:21:10.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/14' 2023-10-30T14:21:10.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/18' 2023-10-30T14:21:10.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/13' 2023-10-30T14:21:10.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/33' 2023-10-30T14:21:10.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/17' 2023-10-30T14:21:10.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/35' 2023-10-30T14:21:10.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed './177/37' 2023-10-30T14:21:10.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './177' 2023-10-30T14:21:10.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/19' 2023-10-30T14:21:10.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/15' 2023-10-30T14:21:10.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/31' 2023-10-30T14:21:10.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/36' 2023-10-30T14:21:10.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/20' 2023-10-30T14:21:10.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/12' 2023-10-30T14:21:10.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/39' 2023-10-30T14:21:10.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/10' 2023-10-30T14:21:10.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/34' 2023-10-30T14:21:10.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/38' 2023-10-30T14:21:10.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/11' 2023-10-30T14:21:10.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/16' 2023-10-30T14:21:10.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/32' 2023-10-30T14:21:10.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/30' 2023-10-30T14:21:10.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/14' 2023-10-30T14:21:10.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/18' 2023-10-30T14:21:10.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/13' 2023-10-30T14:21:10.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/33' 2023-10-30T14:21:10.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/17' 2023-10-30T14:21:10.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/35' 2023-10-30T14:21:10.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed './178/37' 2023-10-30T14:21:10.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './178' 2023-10-30T14:21:10.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/19' 2023-10-30T14:21:10.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/15' 2023-10-30T14:21:10.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/31' 2023-10-30T14:21:10.454 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/36' 2023-10-30T14:21:10.454 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/20' 2023-10-30T14:21:10.454 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/12' 2023-10-30T14:21:10.454 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/39' 2023-10-30T14:21:10.454 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/10' 2023-10-30T14:21:10.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/34' 2023-10-30T14:21:10.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/38' 2023-10-30T14:21:10.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/11' 2023-10-30T14:21:10.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/16' 2023-10-30T14:21:10.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/32' 2023-10-30T14:21:10.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/30' 2023-10-30T14:21:10.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/14' 2023-10-30T14:21:10.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/18' 2023-10-30T14:21:10.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/13' 2023-10-30T14:21:10.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/33' 2023-10-30T14:21:10.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/17' 2023-10-30T14:21:10.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/35' 2023-10-30T14:21:10.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './179/37' 2023-10-30T14:21:10.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './179' 2023-10-30T14:21:10.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/19' 2023-10-30T14:21:10.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/15' 2023-10-30T14:21:10.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/31' 2023-10-30T14:21:10.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/36' 2023-10-30T14:21:10.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/20' 2023-10-30T14:21:10.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/12' 2023-10-30T14:21:10.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/39' 2023-10-30T14:21:10.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/10' 2023-10-30T14:21:10.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/34' 2023-10-30T14:21:10.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/38' 2023-10-30T14:21:10.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/11' 2023-10-30T14:21:10.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/16' 2023-10-30T14:21:10.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/32' 2023-10-30T14:21:10.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/30' 2023-10-30T14:21:10.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/14' 2023-10-30T14:21:10.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/18' 2023-10-30T14:21:10.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/13' 2023-10-30T14:21:10.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/33' 2023-10-30T14:21:10.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/17' 2023-10-30T14:21:10.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/35' 2023-10-30T14:21:10.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './180/37' 2023-10-30T14:21:10.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './180' 2023-10-30T14:21:10.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/19' 2023-10-30T14:21:10.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/15' 2023-10-30T14:21:10.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/31' 2023-10-30T14:21:10.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/36' 2023-10-30T14:21:10.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/20' 2023-10-30T14:21:10.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/12' 2023-10-30T14:21:10.463 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/39' 2023-10-30T14:21:10.463 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/10' 2023-10-30T14:21:10.463 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/34' 2023-10-30T14:21:10.463 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/38' 2023-10-30T14:21:10.463 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/11' 2023-10-30T14:21:10.464 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/16' 2023-10-30T14:21:10.464 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/32' 2023-10-30T14:21:10.464 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/30' 2023-10-30T14:21:10.464 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/14' 2023-10-30T14:21:10.464 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/18' 2023-10-30T14:21:10.464 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/13' 2023-10-30T14:21:10.465 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/33' 2023-10-30T14:21:10.465 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/17' 2023-10-30T14:21:10.465 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/35' 2023-10-30T14:21:10.465 INFO:tasks.workunit.client.0.smithi057.stdout:removed './181/37' 2023-10-30T14:21:10.465 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './181' 2023-10-30T14:21:10.465 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/19' 2023-10-30T14:21:10.466 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/15' 2023-10-30T14:21:10.466 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/31' 2023-10-30T14:21:10.466 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/36' 2023-10-30T14:21:10.466 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/20' 2023-10-30T14:21:10.466 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/12' 2023-10-30T14:21:10.467 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/39' 2023-10-30T14:21:10.467 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/10' 2023-10-30T14:21:10.467 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/34' 2023-10-30T14:21:10.467 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/38' 2023-10-30T14:21:10.467 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/11' 2023-10-30T14:21:10.468 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/16' 2023-10-30T14:21:10.468 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/32' 2023-10-30T14:21:10.468 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/30' 2023-10-30T14:21:10.468 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/14' 2023-10-30T14:21:10.468 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/18' 2023-10-30T14:21:10.468 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/13' 2023-10-30T14:21:10.469 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/33' 2023-10-30T14:21:10.469 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/17' 2023-10-30T14:21:10.469 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/35' 2023-10-30T14:21:10.469 INFO:tasks.workunit.client.0.smithi057.stdout:removed './182/37' 2023-10-30T14:21:10.469 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './182' 2023-10-30T14:21:10.469 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/19' 2023-10-30T14:21:10.470 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/15' 2023-10-30T14:21:10.470 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/31' 2023-10-30T14:21:10.470 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/36' 2023-10-30T14:21:10.470 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/20' 2023-10-30T14:21:10.470 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/12' 2023-10-30T14:21:10.470 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/39' 2023-10-30T14:21:10.471 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/10' 2023-10-30T14:21:10.471 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/34' 2023-10-30T14:21:10.471 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/38' 2023-10-30T14:21:10.471 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/11' 2023-10-30T14:21:10.471 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/16' 2023-10-30T14:21:10.472 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/32' 2023-10-30T14:21:10.472 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/30' 2023-10-30T14:21:10.472 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/14' 2023-10-30T14:21:10.472 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/18' 2023-10-30T14:21:10.472 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/13' 2023-10-30T14:21:10.472 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/33' 2023-10-30T14:21:10.473 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/17' 2023-10-30T14:21:10.473 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/35' 2023-10-30T14:21:10.473 INFO:tasks.workunit.client.0.smithi057.stdout:removed './183/37' 2023-10-30T14:21:10.473 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './183' 2023-10-30T14:21:10.473 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/19' 2023-10-30T14:21:10.473 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/15' 2023-10-30T14:21:10.474 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/31' 2023-10-30T14:21:10.474 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/36' 2023-10-30T14:21:10.474 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/20' 2023-10-30T14:21:10.474 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/12' 2023-10-30T14:21:10.474 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/39' 2023-10-30T14:21:10.474 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/10' 2023-10-30T14:21:10.475 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/34' 2023-10-30T14:21:10.475 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/38' 2023-10-30T14:21:10.475 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/11' 2023-10-30T14:21:10.475 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/16' 2023-10-30T14:21:10.475 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/32' 2023-10-30T14:21:10.475 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/30' 2023-10-30T14:21:10.476 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/14' 2023-10-30T14:21:10.476 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/18' 2023-10-30T14:21:10.476 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/13' 2023-10-30T14:21:10.476 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/33' 2023-10-30T14:21:10.476 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/17' 2023-10-30T14:21:10.476 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/35' 2023-10-30T14:21:10.477 INFO:tasks.workunit.client.0.smithi057.stdout:removed './184/37' 2023-10-30T14:21:10.477 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './184' 2023-10-30T14:21:10.477 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/19' 2023-10-30T14:21:10.477 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/15' 2023-10-30T14:21:10.477 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/31' 2023-10-30T14:21:10.477 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/36' 2023-10-30T14:21:10.478 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/20' 2023-10-30T14:21:10.478 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/12' 2023-10-30T14:21:10.478 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/39' 2023-10-30T14:21:10.478 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/10' 2023-10-30T14:21:10.478 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/34' 2023-10-30T14:21:10.478 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/38' 2023-10-30T14:21:10.479 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/11' 2023-10-30T14:21:10.479 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/16' 2023-10-30T14:21:10.479 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/32' 2023-10-30T14:21:10.479 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/30' 2023-10-30T14:21:10.479 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/14' 2023-10-30T14:21:10.480 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/18' 2023-10-30T14:21:10.480 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/13' 2023-10-30T14:21:10.480 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/33' 2023-10-30T14:21:10.480 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/17' 2023-10-30T14:21:10.480 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/35' 2023-10-30T14:21:10.480 INFO:tasks.workunit.client.0.smithi057.stdout:removed './185/37' 2023-10-30T14:21:10.481 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './185' 2023-10-30T14:21:10.481 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/19' 2023-10-30T14:21:10.481 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/15' 2023-10-30T14:21:10.481 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/31' 2023-10-30T14:21:10.481 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/36' 2023-10-30T14:21:10.481 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/20' 2023-10-30T14:21:10.482 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/12' 2023-10-30T14:21:10.482 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/39' 2023-10-30T14:21:10.482 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/10' 2023-10-30T14:21:10.482 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/34' 2023-10-30T14:21:10.482 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/38' 2023-10-30T14:21:10.482 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/11' 2023-10-30T14:21:10.483 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/16' 2023-10-30T14:21:10.483 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/32' 2023-10-30T14:21:10.483 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/30' 2023-10-30T14:21:10.483 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/14' 2023-10-30T14:21:10.483 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/18' 2023-10-30T14:21:10.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.495+0000 7f7394de4640 1 Processor -- start 2023-10-30T14:21:10.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.495+0000 7f7394de4640 1 -- start start 2023-10-30T14:21:10.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.495+0000 7f7394de4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f73900c5460 0x7f739006d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:10.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.495+0000 7f7394de4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f739006fc30 con 0x7f73900706a0 2023-10-30T14:21:10.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.495+0000 7f7394de4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f739015b360 con 0x7f739006dfc0 2023-10-30T14:21:10.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.495+0000 7f7394de4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f739015b170 con 0x7f73900c5460 2023-10-30T14:21:10.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.499+0000 7f738ffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f73900706a0 0x7f73900c40f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:54846/0 (socket says 172.21.15.57:54846) 2023-10-30T14:21:10.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.499+0000 7f738ffff640 1 -- 172.21.15.57:0/1837892171 learned_addr learned my addr 172.21.15.57:0/1837892171 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:21:10.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.499+0000 7f738effd640 1 --2- 172.21.15.57:0/1837892171 >> v2:172.21.15.88:3300/0 conn(0x7f73900c5460 0x7f739006d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:10.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.499+0000 7f738e7fc640 1 -- 172.21.15.57:0/1837892171 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366817792 0 0) 0x7f739006fc30 con 0x7f73900706a0 2023-10-30T14:21:10.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.499+0000 7f738e7fc640 1 -- 172.21.15.57:0/1837892171 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7378003530 con 0x7f73900706a0 2023-10-30T14:21:10.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.499+0000 7f738effd640 1 -- 172.21.15.57:0/1837892171 >> v1:172.21.15.88:6789/0 conn(0x7f739006dfc0 legacy=0x7f739015aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:10.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.499+0000 7f738effd640 1 -- 172.21.15.57:0/1837892171 >> v1:172.21.15.57:6789/0 conn(0x7f73900706a0 legacy=0x7f73900c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:10.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.499+0000 7f738effd640 1 -- 172.21.15.57:0/1837892171 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739015b620 con 0x7f73900c5460 2023-10-30T14:21:10.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.499+0000 7f738effd640 1 --2- 172.21.15.57:0/1837892171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73900c5460 0x7f739006d4f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f73840097c0 tx=0x7f738401c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32e87e572fdbc1aa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:10.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.503+0000 7f738e7fc640 1 -- 172.21.15.57:0/1837892171 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73840ab650 con 0x7f73900c5460 2023-10-30T14:21:10.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.503+0000 7f738e7fc640 1 -- 172.21.15.57:0/1837892171 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73840ab7b0 con 0x7f73900c5460 2023-10-30T14:21:10.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.503+0000 7f738e7fc640 1 -- 172.21.15.57:0/1837892171 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73840abb40 con 0x7f73900c5460 2023-10-30T14:21:10.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.503+0000 7f7394de4640 1 -- 172.21.15.57:0/1837892171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73900c5460 msgr2=0x7f739006d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:10.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.503+0000 7f7394de4640 1 --2- 172.21.15.57:0/1837892171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73900c5460 0x7f739006d4f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f73840097c0 tx=0x7f738401c450 comp rx=0 tx=0).stop 2023-10-30T14:21:10.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.503+0000 7f7394de4640 1 -- 172.21.15.57:0/1837892171 shutdown_connections 2023-10-30T14:21:10.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.503+0000 7f7394de4640 1 --2- 172.21.15.57:0/1837892171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73900c5460 0x7f739006d4f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:10.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.503+0000 7f7394de4640 1 -- 172.21.15.57:0/1837892171 >> 172.21.15.57:0/1837892171 conn(0x7f73900caae0 msgr2=0x7f73900caec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:10.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.503+0000 7f7394de4640 1 -- 172.21.15.57:0/1837892171 shutdown_connections 2023-10-30T14:21:10.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.503+0000 7f7394de4640 1 -- 172.21.15.57:0/1837892171 wait complete. 2023-10-30T14:21:10.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.503+0000 7f7394de4640 1 Processor -- start 2023-10-30T14:21:10.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.503+0000 7f7394de4640 1 -- start start 2023-10-30T14:21:10.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.503+0000 7f7394de4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f739006dfc0 0x7f73901556b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:10.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.503+0000 7f7394de4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73900706a0 0x7f7390145340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:10.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f7394de4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73900c5460 0x7f73901563e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:10.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f7394de4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7390158ea0 con 0x7f73900c5460 2023-10-30T14:21:10.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f7394de4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7390156120 con 0x7f73900706a0 2023-10-30T14:21:10.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f7394de4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f739014b0c0 con 0x7f739006dfc0 2023-10-30T14:21:10.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f738ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73900c5460 0x7f73901563e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:10.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f738ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73900c5460 0x7f73901563e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40382/0 (socket says 172.21.15.57:40382) 2023-10-30T14:21:10.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f738ffff640 1 -- 172.21.15.57:0/2302868851 learned_addr learned my addr 172.21.15.57:0/2302868851 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:21:10.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f738f7fe640 1 --2- 172.21.15.57:0/2302868851 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f739006dfc0 0x7f73901556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:10.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f738effd640 1 --2- 172.21.15.57:0/2302868851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73900706a0 0x7f7390145340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:10.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f738ffff640 1 -- 172.21.15.57:0/2302868851 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f739006dfc0 msgr2=0x7f73901556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:10.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f738ffff640 1 --2- 172.21.15.57:0/2302868851 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f739006dfc0 0x7f73901556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:10.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f738ffff640 1 -- 172.21.15.57:0/2302868851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73900706a0 msgr2=0x7f7390145340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:10.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f738ffff640 1 --2- 172.21.15.57:0/2302868851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73900706a0 0x7f7390145340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:10.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f738ffff640 1 -- 172.21.15.57:0/2302868851 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7384009420 con 0x7f73900c5460 2023-10-30T14:21:10.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f738effd640 1 --2- 172.21.15.57:0/2302868851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73900706a0 0x7f7390145340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:21:10.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f738ffff640 1 --2- 172.21.15.57:0/2302868851 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73900c5460 0x7f73901563e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f738800cb40 tx=0x7f738800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:10.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f7377fff640 1 -- 172.21.15.57:0/2302868851 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738800aca0 con 0x7f73900c5460 2023-10-30T14:21:10.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f7377fff640 1 -- 172.21.15.57:0/2302868851 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f738800ae00 con 0x7f73900c5460 2023-10-30T14:21:10.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f7377fff640 1 -- 172.21.15.57:0/2302868851 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7388012620 con 0x7f73900c5460 2023-10-30T14:21:10.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f7394de4640 1 -- 172.21.15.57:0/2302868851 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739014aab0 con 0x7f73900c5460 2023-10-30T14:21:10.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f7394de4640 1 -- 172.21.15.57:0/2302868851 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739014ad50 con 0x7f73900c5460 2023-10-30T14:21:10.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f7394de4640 1 -- 172.21.15.57:0/2302868851 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f739004e170 con 0x7f73900c5460 2023-10-30T14:21:10.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.507+0000 7f7377fff640 1 -- 172.21.15.57:0/2302868851 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7388007070 con 0x7f73900c5460 2023-10-30T14:21:10.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.511+0000 7f7377fff640 1 --2- 172.21.15.57:0/2302868851 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f737c0664a0 0x7f737c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:10.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.511+0000 7f7377fff640 1 -- 172.21.15.57:0/2302868851 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f7388084000 con 0x7f73900c5460 2023-10-30T14:21:10.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.511+0000 7f738f7fe640 1 --2- 172.21.15.57:0/2302868851 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f737c0664a0 0x7f737c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:10.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.511+0000 7f738f7fe640 1 --2- 172.21.15.57:0/2302868851 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f737c0664a0 0x7f737c068960 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7380007fd0 tx=0x7f7380009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:10.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.515+0000 7f7377fff640 1 -- 172.21.15.57:0/2302868851 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f73880513a0 con 0x7f73900c5460 2023-10-30T14:21:10.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.703+0000 7f7394de4640 1 -- 172.21.15.57:0/2302868851 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f739014b200 con 0x7f73900c5460 2023-10-30T14:21:10.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.703+0000 7f7377fff640 1 -- 172.21.15.57:0/2302868851 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6527 (secure 0 0 0) 0x7f73880549c0 con 0x7f73900c5460 2023-10-30T14:21:10.709 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:21:10.709 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:21:03.820033+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:active","state_seq":28,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:21:10.709 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 54 2023-10-30T14:21:10.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.703+0000 7f7394de4640 1 -- 172.21.15.57:0/2302868851 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f737c0664a0 msgr2=0x7f737c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:10.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.703+0000 7f7394de4640 1 --2- 172.21.15.57:0/2302868851 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f737c0664a0 0x7f737c068960 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7380007fd0 tx=0x7f7380009040 comp rx=0 tx=0).stop 2023-10-30T14:21:10.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.703+0000 7f7394de4640 1 -- 172.21.15.57:0/2302868851 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73900c5460 msgr2=0x7f73901563e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:10.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.703+0000 7f7394de4640 1 --2- 172.21.15.57:0/2302868851 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73900c5460 0x7f73901563e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f738800cb40 tx=0x7f738800a060 comp rx=0 tx=0).stop 2023-10-30T14:21:10.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.707+0000 7f7394de4640 1 -- 172.21.15.57:0/2302868851 shutdown_connections 2023-10-30T14:21:10.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.707+0000 7f7394de4640 1 --2- 172.21.15.57:0/2302868851 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f737c0664a0 0x7f737c068960 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:10.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.707+0000 7f7394de4640 1 --2- 172.21.15.57:0/2302868851 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73900c5460 0x7f73901563e0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:10.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.707+0000 7f7394de4640 1 --2- 172.21.15.57:0/2302868851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73900706a0 0x7f7390145340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:10.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.707+0000 7f7394de4640 1 --2- 172.21.15.57:0/2302868851 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f739006dfc0 0x7f73901556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:10.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.707+0000 7f7394de4640 1 -- 172.21.15.57:0/2302868851 >> 172.21.15.57:0/2302868851 conn(0x7f73900caae0 msgr2=0x7f73900cafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:10.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.707+0000 7f7394de4640 1 -- 172.21.15.57:0/2302868851 shutdown_connections 2023-10-30T14:21:10.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:10.707+0000 7f7394de4640 1 -- 172.21.15.57:0/2302868851 wait complete. 2023-10-30T14:21:10.730 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T14:21:10.730 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 42 secs before thrashing 2023-10-30T14:21:14.318 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/13' 2023-10-30T14:21:14.318 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/33' 2023-10-30T14:21:14.319 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/17' 2023-10-30T14:21:14.319 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/35' 2023-10-30T14:21:14.319 INFO:tasks.workunit.client.0.smithi057.stdout:removed './186/37' 2023-10-30T14:21:14.319 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './186' 2023-10-30T14:21:14.319 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/19' 2023-10-30T14:21:14.320 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/15' 2023-10-30T14:21:14.320 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/31' 2023-10-30T14:21:14.320 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/36' 2023-10-30T14:21:14.320 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/20' 2023-10-30T14:21:14.320 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/12' 2023-10-30T14:21:14.321 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/39' 2023-10-30T14:21:14.321 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/10' 2023-10-30T14:21:14.321 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/34' 2023-10-30T14:21:14.321 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/38' 2023-10-30T14:21:14.321 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/11' 2023-10-30T14:21:14.322 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/16' 2023-10-30T14:21:14.322 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/32' 2023-10-30T14:21:14.322 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/30' 2023-10-30T14:21:14.322 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/14' 2023-10-30T14:21:14.322 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/18' 2023-10-30T14:21:14.323 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/13' 2023-10-30T14:21:14.323 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/33' 2023-10-30T14:21:14.323 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/17' 2023-10-30T14:21:14.323 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/35' 2023-10-30T14:21:14.323 INFO:tasks.workunit.client.0.smithi057.stdout:removed './187/37' 2023-10-30T14:21:14.324 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './187' 2023-10-30T14:21:14.324 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/19' 2023-10-30T14:21:14.324 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/15' 2023-10-30T14:21:14.324 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/31' 2023-10-30T14:21:14.324 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/36' 2023-10-30T14:21:14.325 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/20' 2023-10-30T14:21:14.325 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/12' 2023-10-30T14:21:14.325 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/39' 2023-10-30T14:21:14.325 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/10' 2023-10-30T14:21:14.325 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/34' 2023-10-30T14:21:14.326 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/38' 2023-10-30T14:21:14.326 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/11' 2023-10-30T14:21:14.326 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/16' 2023-10-30T14:21:14.326 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/32' 2023-10-30T14:21:14.326 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/30' 2023-10-30T14:21:14.327 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/14' 2023-10-30T14:21:14.327 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/18' 2023-10-30T14:21:14.327 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/13' 2023-10-30T14:21:14.327 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/33' 2023-10-30T14:21:14.327 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/17' 2023-10-30T14:21:14.328 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/35' 2023-10-30T14:21:14.328 INFO:tasks.workunit.client.0.smithi057.stdout:removed './188/37' 2023-10-30T14:21:14.328 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './188' 2023-10-30T14:21:14.328 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/19' 2023-10-30T14:21:14.328 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/15' 2023-10-30T14:21:14.329 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/31' 2023-10-30T14:21:14.329 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/36' 2023-10-30T14:21:14.329 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/20' 2023-10-30T14:21:14.329 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/12' 2023-10-30T14:21:14.329 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/39' 2023-10-30T14:21:14.330 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/10' 2023-10-30T14:21:14.330 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/34' 2023-10-30T14:21:14.330 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/38' 2023-10-30T14:21:14.330 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/11' 2023-10-30T14:21:14.330 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/16' 2023-10-30T14:21:14.331 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/32' 2023-10-30T14:21:14.331 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/30' 2023-10-30T14:21:14.331 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/14' 2023-10-30T14:21:14.331 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/18' 2023-10-30T14:21:14.331 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/13' 2023-10-30T14:21:14.332 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/33' 2023-10-30T14:21:14.332 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/17' 2023-10-30T14:21:14.332 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/35' 2023-10-30T14:21:14.332 INFO:tasks.workunit.client.0.smithi057.stdout:removed './189/37' 2023-10-30T14:21:14.332 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './189' 2023-10-30T14:21:14.332 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/19' 2023-10-30T14:21:14.333 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/15' 2023-10-30T14:21:14.333 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/31' 2023-10-30T14:21:14.333 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/36' 2023-10-30T14:21:14.333 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/20' 2023-10-30T14:21:14.334 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/12' 2023-10-30T14:21:14.334 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/39' 2023-10-30T14:21:14.334 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/10' 2023-10-30T14:21:14.334 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/34' 2023-10-30T14:21:14.334 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/38' 2023-10-30T14:21:14.335 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/11' 2023-10-30T14:21:14.335 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/16' 2023-10-30T14:21:14.335 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/32' 2023-10-30T14:21:14.335 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/30' 2023-10-30T14:21:14.336 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/14' 2023-10-30T14:21:14.336 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/18' 2023-10-30T14:21:14.336 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/13' 2023-10-30T14:21:14.336 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/33' 2023-10-30T14:21:14.336 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/17' 2023-10-30T14:21:14.337 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/35' 2023-10-30T14:21:14.337 INFO:tasks.workunit.client.0.smithi057.stdout:removed './190/37' 2023-10-30T14:21:14.337 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './190' 2023-10-30T14:21:14.337 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/19' 2023-10-30T14:21:14.337 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/15' 2023-10-30T14:21:14.338 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/31' 2023-10-30T14:21:14.338 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/36' 2023-10-30T14:21:14.338 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/20' 2023-10-30T14:21:14.338 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/12' 2023-10-30T14:21:14.339 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/39' 2023-10-30T14:21:14.339 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/10' 2023-10-30T14:21:14.339 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/34' 2023-10-30T14:21:14.339 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/38' 2023-10-30T14:21:14.339 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/11' 2023-10-30T14:21:14.339 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/16' 2023-10-30T14:21:14.340 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/32' 2023-10-30T14:21:14.340 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/30' 2023-10-30T14:21:14.340 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/14' 2023-10-30T14:21:14.340 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/18' 2023-10-30T14:21:14.340 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/13' 2023-10-30T14:21:14.341 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/33' 2023-10-30T14:21:14.341 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/17' 2023-10-30T14:21:14.341 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/35' 2023-10-30T14:21:14.341 INFO:tasks.workunit.client.0.smithi057.stdout:removed './191/37' 2023-10-30T14:21:14.341 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './191' 2023-10-30T14:21:14.341 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/19' 2023-10-30T14:21:14.342 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/15' 2023-10-30T14:21:14.342 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/31' 2023-10-30T14:21:14.342 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/36' 2023-10-30T14:21:14.342 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/20' 2023-10-30T14:21:14.342 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/12' 2023-10-30T14:21:14.343 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/39' 2023-10-30T14:21:14.343 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/10' 2023-10-30T14:21:14.343 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/34' 2023-10-30T14:21:14.343 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/38' 2023-10-30T14:21:14.343 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/11' 2023-10-30T14:21:14.344 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/16' 2023-10-30T14:21:14.344 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/32' 2023-10-30T14:21:14.344 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/30' 2023-10-30T14:21:14.344 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/14' 2023-10-30T14:21:14.344 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/18' 2023-10-30T14:21:14.344 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/13' 2023-10-30T14:21:14.345 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/33' 2023-10-30T14:21:14.345 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/17' 2023-10-30T14:21:14.345 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/35' 2023-10-30T14:21:14.345 INFO:tasks.workunit.client.0.smithi057.stdout:removed './192/37' 2023-10-30T14:21:14.345 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './192' 2023-10-30T14:21:14.346 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/19' 2023-10-30T14:21:14.346 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/15' 2023-10-30T14:21:14.346 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/31' 2023-10-30T14:21:14.346 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/36' 2023-10-30T14:21:14.346 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/20' 2023-10-30T14:21:14.347 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/12' 2023-10-30T14:21:14.347 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/39' 2023-10-30T14:21:14.347 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/10' 2023-10-30T14:21:14.347 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/34' 2023-10-30T14:21:14.347 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/38' 2023-10-30T14:21:14.347 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/11' 2023-10-30T14:21:14.348 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/16' 2023-10-30T14:21:14.348 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/32' 2023-10-30T14:21:14.348 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/30' 2023-10-30T14:21:14.348 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/14' 2023-10-30T14:21:14.348 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/18' 2023-10-30T14:21:14.349 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/13' 2023-10-30T14:21:14.349 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/33' 2023-10-30T14:21:14.349 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/17' 2023-10-30T14:21:14.349 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/35' 2023-10-30T14:21:14.349 INFO:tasks.workunit.client.0.smithi057.stdout:removed './193/37' 2023-10-30T14:21:14.350 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './193' 2023-10-30T14:21:14.350 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/19' 2023-10-30T14:21:14.350 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/15' 2023-10-30T14:21:14.350 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/31' 2023-10-30T14:21:14.350 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/36' 2023-10-30T14:21:14.351 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/20' 2023-10-30T14:21:14.351 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/12' 2023-10-30T14:21:14.351 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/39' 2023-10-30T14:21:14.351 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/10' 2023-10-30T14:21:14.351 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/34' 2023-10-30T14:21:14.351 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/38' 2023-10-30T14:21:14.352 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/11' 2023-10-30T14:21:14.352 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/16' 2023-10-30T14:21:14.352 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/32' 2023-10-30T14:21:14.352 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/30' 2023-10-30T14:21:14.352 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/14' 2023-10-30T14:21:14.353 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/18' 2023-10-30T14:21:14.353 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/13' 2023-10-30T14:21:14.353 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/33' 2023-10-30T14:21:14.353 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/17' 2023-10-30T14:21:14.353 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/35' 2023-10-30T14:21:14.353 INFO:tasks.workunit.client.0.smithi057.stdout:removed './194/37' 2023-10-30T14:21:14.354 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './194' 2023-10-30T14:21:14.354 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/19' 2023-10-30T14:21:14.354 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/15' 2023-10-30T14:21:14.354 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/31' 2023-10-30T14:21:14.354 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/36' 2023-10-30T14:21:14.355 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/20' 2023-10-30T14:21:14.355 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/12' 2023-10-30T14:21:14.355 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/39' 2023-10-30T14:21:14.355 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/10' 2023-10-30T14:21:14.355 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/34' 2023-10-30T14:21:14.355 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/38' 2023-10-30T14:21:14.356 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/11' 2023-10-30T14:21:14.356 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/16' 2023-10-30T14:21:14.356 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/32' 2023-10-30T14:21:14.356 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/30' 2023-10-30T14:21:14.356 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/14' 2023-10-30T14:21:14.356 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/18' 2023-10-30T14:21:14.357 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/13' 2023-10-30T14:21:14.357 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/33' 2023-10-30T14:21:14.357 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/17' 2023-10-30T14:21:14.357 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/35' 2023-10-30T14:21:14.358 INFO:tasks.workunit.client.0.smithi057.stdout:removed './195/37' 2023-10-30T14:21:14.358 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './195' 2023-10-30T14:21:14.358 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/19' 2023-10-30T14:21:14.358 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/15' 2023-10-30T14:21:14.358 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/31' 2023-10-30T14:21:14.359 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/36' 2023-10-30T14:21:14.359 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/20' 2023-10-30T14:21:14.359 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/12' 2023-10-30T14:21:14.359 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/39' 2023-10-30T14:21:14.359 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/10' 2023-10-30T14:21:19.609 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/34' 2023-10-30T14:21:19.609 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/38' 2023-10-30T14:21:19.609 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/11' 2023-10-30T14:21:19.610 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/16' 2023-10-30T14:21:19.610 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/32' 2023-10-30T14:21:19.610 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/30' 2023-10-30T14:21:19.610 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/14' 2023-10-30T14:21:19.610 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/18' 2023-10-30T14:21:19.611 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/13' 2023-10-30T14:21:19.611 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/33' 2023-10-30T14:21:19.611 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/17' 2023-10-30T14:21:19.611 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/35' 2023-10-30T14:21:19.611 INFO:tasks.workunit.client.0.smithi057.stdout:removed './196/37' 2023-10-30T14:21:19.611 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './196' 2023-10-30T14:21:19.612 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/19' 2023-10-30T14:21:19.612 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/15' 2023-10-30T14:21:19.612 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/31' 2023-10-30T14:21:19.612 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/36' 2023-10-30T14:21:19.612 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/20' 2023-10-30T14:21:19.613 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/12' 2023-10-30T14:21:19.613 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/39' 2023-10-30T14:21:19.613 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/10' 2023-10-30T14:21:19.613 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/34' 2023-10-30T14:21:19.613 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/38' 2023-10-30T14:21:19.614 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/11' 2023-10-30T14:21:19.614 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/16' 2023-10-30T14:21:19.614 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/32' 2023-10-30T14:21:19.614 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/30' 2023-10-30T14:21:19.614 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/14' 2023-10-30T14:21:19.615 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/18' 2023-10-30T14:21:19.615 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/13' 2023-10-30T14:21:19.615 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/33' 2023-10-30T14:21:19.615 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/17' 2023-10-30T14:21:19.615 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/35' 2023-10-30T14:21:19.616 INFO:tasks.workunit.client.0.smithi057.stdout:removed './197/37' 2023-10-30T14:21:19.616 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './197' 2023-10-30T14:21:19.616 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/19' 2023-10-30T14:21:19.616 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/15' 2023-10-30T14:21:19.616 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/31' 2023-10-30T14:21:19.617 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/36' 2023-10-30T14:21:19.617 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/20' 2023-10-30T14:21:19.617 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/12' 2023-10-30T14:21:19.617 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/39' 2023-10-30T14:21:19.617 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/10' 2023-10-30T14:21:19.618 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/34' 2023-10-30T14:21:19.618 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/38' 2023-10-30T14:21:19.618 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/11' 2023-10-30T14:21:19.618 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/16' 2023-10-30T14:21:19.618 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/32' 2023-10-30T14:21:19.619 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/30' 2023-10-30T14:21:19.619 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/14' 2023-10-30T14:21:19.619 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/18' 2023-10-30T14:21:19.619 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/13' 2023-10-30T14:21:19.619 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/33' 2023-10-30T14:21:19.620 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/17' 2023-10-30T14:21:19.620 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/35' 2023-10-30T14:21:19.620 INFO:tasks.workunit.client.0.smithi057.stdout:removed './198/37' 2023-10-30T14:21:19.620 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './198' 2023-10-30T14:21:19.620 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/19' 2023-10-30T14:21:19.621 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/15' 2023-10-30T14:21:19.621 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/31' 2023-10-30T14:21:19.621 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/36' 2023-10-30T14:21:19.621 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/20' 2023-10-30T14:21:19.621 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/12' 2023-10-30T14:21:19.622 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/39' 2023-10-30T14:21:19.622 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/10' 2023-10-30T14:21:19.622 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/34' 2023-10-30T14:21:19.622 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/38' 2023-10-30T14:21:19.622 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/11' 2023-10-30T14:21:19.623 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/16' 2023-10-30T14:21:19.623 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/32' 2023-10-30T14:21:19.623 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/30' 2023-10-30T14:21:19.623 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/14' 2023-10-30T14:21:19.623 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/18' 2023-10-30T14:21:19.624 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/13' 2023-10-30T14:21:19.624 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/33' 2023-10-30T14:21:19.624 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/17' 2023-10-30T14:21:19.624 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/35' 2023-10-30T14:21:19.624 INFO:tasks.workunit.client.0.smithi057.stdout:removed './199/37' 2023-10-30T14:21:19.625 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './199' 2023-10-30T14:21:19.625 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/21' 2023-10-30T14:21:19.625 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/27' 2023-10-30T14:21:19.625 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/31' 2023-10-30T14:21:19.625 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/36' 2023-10-30T14:21:19.625 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/20' 2023-10-30T14:21:19.626 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/39' 2023-10-30T14:21:19.626 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/34' 2023-10-30T14:21:19.626 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/38' 2023-10-30T14:21:19.626 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/28' 2023-10-30T14:21:19.626 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/24' 2023-10-30T14:21:19.627 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/29' 2023-10-30T14:21:19.627 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/32' 2023-10-30T14:21:19.627 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/22' 2023-10-30T14:21:19.627 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/25' 2023-10-30T14:21:19.627 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/30' 2023-10-30T14:21:19.628 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/33' 2023-10-30T14:21:19.628 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/23' 2023-10-30T14:21:19.628 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/35' 2023-10-30T14:21:19.628 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/26' 2023-10-30T14:21:19.628 INFO:tasks.workunit.client.0.smithi057.stdout:removed './200/37' 2023-10-30T14:21:19.628 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './200' 2023-10-30T14:21:19.629 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/21' 2023-10-30T14:21:19.629 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/27' 2023-10-30T14:21:19.629 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/31' 2023-10-30T14:21:19.629 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/36' 2023-10-30T14:21:19.629 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/20' 2023-10-30T14:21:19.630 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/39' 2023-10-30T14:21:19.630 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/34' 2023-10-30T14:21:19.630 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/38' 2023-10-30T14:21:19.630 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/28' 2023-10-30T14:21:19.630 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/24' 2023-10-30T14:21:19.631 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/29' 2023-10-30T14:21:19.631 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/32' 2023-10-30T14:21:19.631 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/22' 2023-10-30T14:21:19.631 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/25' 2023-10-30T14:21:19.631 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/30' 2023-10-30T14:21:19.631 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/33' 2023-10-30T14:21:19.632 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/23' 2023-10-30T14:21:19.632 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/35' 2023-10-30T14:21:19.632 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/26' 2023-10-30T14:21:19.632 INFO:tasks.workunit.client.0.smithi057.stdout:removed './201/37' 2023-10-30T14:21:19.632 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './201' 2023-10-30T14:21:19.632 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/21' 2023-10-30T14:21:19.633 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/27' 2023-10-30T14:21:19.633 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/31' 2023-10-30T14:21:19.633 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/36' 2023-10-30T14:21:19.633 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/20' 2023-10-30T14:21:19.633 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/39' 2023-10-30T14:21:19.633 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/34' 2023-10-30T14:21:19.634 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/38' 2023-10-30T14:21:19.634 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/28' 2023-10-30T14:21:19.634 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/24' 2023-10-30T14:21:19.634 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/29' 2023-10-30T14:21:19.634 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/32' 2023-10-30T14:21:19.634 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/22' 2023-10-30T14:21:19.635 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/25' 2023-10-30T14:21:19.635 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/30' 2023-10-30T14:21:19.635 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/33' 2023-10-30T14:21:19.635 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/23' 2023-10-30T14:21:19.635 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/35' 2023-10-30T14:21:19.636 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/26' 2023-10-30T14:21:19.636 INFO:tasks.workunit.client.0.smithi057.stdout:removed './202/37' 2023-10-30T14:21:19.636 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './202' 2023-10-30T14:21:19.636 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/21' 2023-10-30T14:21:19.636 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/27' 2023-10-30T14:21:19.637 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/31' 2023-10-30T14:21:19.637 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/36' 2023-10-30T14:21:19.637 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/20' 2023-10-30T14:21:19.637 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/39' 2023-10-30T14:21:19.637 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/34' 2023-10-30T14:21:19.637 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/38' 2023-10-30T14:21:19.638 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/28' 2023-10-30T14:21:19.638 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/24' 2023-10-30T14:21:19.638 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/29' 2023-10-30T14:21:19.638 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/32' 2023-10-30T14:21:19.638 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/22' 2023-10-30T14:21:19.639 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/25' 2023-10-30T14:21:19.639 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/30' 2023-10-30T14:21:19.639 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/33' 2023-10-30T14:21:19.639 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/23' 2023-10-30T14:21:19.640 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/35' 2023-10-30T14:21:19.640 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/26' 2023-10-30T14:21:19.640 INFO:tasks.workunit.client.0.smithi057.stdout:removed './203/37' 2023-10-30T14:21:19.640 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './203' 2023-10-30T14:21:19.640 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/21' 2023-10-30T14:21:19.640 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/27' 2023-10-30T14:21:19.641 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/31' 2023-10-30T14:21:19.641 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/36' 2023-10-30T14:21:19.641 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/20' 2023-10-30T14:21:19.641 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/39' 2023-10-30T14:21:19.641 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/34' 2023-10-30T14:21:19.642 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/38' 2023-10-30T14:21:19.642 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/28' 2023-10-30T14:21:19.642 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/24' 2023-10-30T14:21:19.642 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/29' 2023-10-30T14:21:19.642 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/32' 2023-10-30T14:21:19.643 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/22' 2023-10-30T14:21:19.643 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/25' 2023-10-30T14:21:19.643 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/30' 2023-10-30T14:21:19.643 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/33' 2023-10-30T14:21:19.644 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/23' 2023-10-30T14:21:19.644 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/35' 2023-10-30T14:21:19.644 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/26' 2023-10-30T14:21:19.644 INFO:tasks.workunit.client.0.smithi057.stdout:removed './204/37' 2023-10-30T14:21:19.644 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './204' 2023-10-30T14:21:19.645 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/21' 2023-10-30T14:21:19.645 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/27' 2023-10-30T14:21:19.645 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/31' 2023-10-30T14:21:19.645 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/36' 2023-10-30T14:21:19.645 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/20' 2023-10-30T14:21:19.645 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/39' 2023-10-30T14:21:19.646 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/34' 2023-10-30T14:21:19.646 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/38' 2023-10-30T14:21:19.646 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/28' 2023-10-30T14:21:19.646 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/24' 2023-10-30T14:21:19.646 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/29' 2023-10-30T14:21:19.647 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/32' 2023-10-30T14:21:19.647 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/22' 2023-10-30T14:21:19.647 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/25' 2023-10-30T14:21:19.647 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/30' 2023-10-30T14:21:19.647 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/33' 2023-10-30T14:21:19.648 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/23' 2023-10-30T14:21:19.648 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/35' 2023-10-30T14:21:19.648 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/26' 2023-10-30T14:21:19.648 INFO:tasks.workunit.client.0.smithi057.stdout:removed './205/37' 2023-10-30T14:21:19.648 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './205' 2023-10-30T14:21:19.649 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/21' 2023-10-30T14:21:19.649 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/27' 2023-10-30T14:21:19.649 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/31' 2023-10-30T14:21:19.649 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/36' 2023-10-30T14:21:19.649 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/20' 2023-10-30T14:21:19.650 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/39' 2023-10-30T14:21:25.020 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/34' 2023-10-30T14:21:25.020 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/38' 2023-10-30T14:21:25.020 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/28' 2023-10-30T14:21:25.020 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/24' 2023-10-30T14:21:25.021 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/29' 2023-10-30T14:21:25.021 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/32' 2023-10-30T14:21:25.021 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/22' 2023-10-30T14:21:25.021 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/25' 2023-10-30T14:21:25.022 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/30' 2023-10-30T14:21:25.022 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/33' 2023-10-30T14:21:25.022 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/23' 2023-10-30T14:21:25.022 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/35' 2023-10-30T14:21:25.022 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/26' 2023-10-30T14:21:25.023 INFO:tasks.workunit.client.0.smithi057.stdout:removed './206/37' 2023-10-30T14:21:25.023 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './206' 2023-10-30T14:21:25.023 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/21' 2023-10-30T14:21:25.023 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/27' 2023-10-30T14:21:25.023 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/31' 2023-10-30T14:21:25.023 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/36' 2023-10-30T14:21:25.024 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/20' 2023-10-30T14:21:25.024 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/39' 2023-10-30T14:21:25.024 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/34' 2023-10-30T14:21:25.024 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/38' 2023-10-30T14:21:25.024 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/28' 2023-10-30T14:21:25.025 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/24' 2023-10-30T14:21:25.025 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/29' 2023-10-30T14:21:25.025 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/32' 2023-10-30T14:21:25.025 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/22' 2023-10-30T14:21:25.025 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/25' 2023-10-30T14:21:25.026 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/30' 2023-10-30T14:21:25.026 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/33' 2023-10-30T14:21:25.026 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/23' 2023-10-30T14:21:25.026 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/35' 2023-10-30T14:21:25.026 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/26' 2023-10-30T14:21:25.026 INFO:tasks.workunit.client.0.smithi057.stdout:removed './207/37' 2023-10-30T14:21:25.027 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './207' 2023-10-30T14:21:25.027 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/21' 2023-10-30T14:21:25.027 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/27' 2023-10-30T14:21:25.027 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/31' 2023-10-30T14:21:25.027 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/36' 2023-10-30T14:21:25.028 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/20' 2023-10-30T14:21:25.028 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/39' 2023-10-30T14:21:25.028 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/34' 2023-10-30T14:21:25.028 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/38' 2023-10-30T14:21:25.028 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/28' 2023-10-30T14:21:25.029 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/24' 2023-10-30T14:21:25.029 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/29' 2023-10-30T14:21:25.029 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/32' 2023-10-30T14:21:25.029 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/22' 2023-10-30T14:21:25.029 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/25' 2023-10-30T14:21:25.030 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/30' 2023-10-30T14:21:25.030 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/33' 2023-10-30T14:21:25.030 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/23' 2023-10-30T14:21:25.030 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/35' 2023-10-30T14:21:25.030 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/26' 2023-10-30T14:21:25.031 INFO:tasks.workunit.client.0.smithi057.stdout:removed './208/37' 2023-10-30T14:21:25.031 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './208' 2023-10-30T14:21:25.031 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/21' 2023-10-30T14:21:25.031 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/27' 2023-10-30T14:21:25.031 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/31' 2023-10-30T14:21:25.032 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/36' 2023-10-30T14:21:25.032 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/20' 2023-10-30T14:21:25.032 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/39' 2023-10-30T14:21:25.032 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/34' 2023-10-30T14:21:25.032 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/38' 2023-10-30T14:21:25.032 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/28' 2023-10-30T14:21:25.033 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/24' 2023-10-30T14:21:25.033 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/29' 2023-10-30T14:21:25.033 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/32' 2023-10-30T14:21:25.033 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/22' 2023-10-30T14:21:25.033 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/25' 2023-10-30T14:21:25.034 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/30' 2023-10-30T14:21:25.034 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/33' 2023-10-30T14:21:25.034 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/23' 2023-10-30T14:21:25.034 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/35' 2023-10-30T14:21:25.034 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/26' 2023-10-30T14:21:25.035 INFO:tasks.workunit.client.0.smithi057.stdout:removed './209/37' 2023-10-30T14:21:25.035 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './209' 2023-10-30T14:21:25.035 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/21' 2023-10-30T14:21:25.035 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/27' 2023-10-30T14:21:25.035 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/31' 2023-10-30T14:21:25.035 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/36' 2023-10-30T14:21:25.036 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/20' 2023-10-30T14:21:25.036 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/39' 2023-10-30T14:21:25.036 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/34' 2023-10-30T14:21:25.036 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/38' 2023-10-30T14:21:25.036 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/28' 2023-10-30T14:21:25.037 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/24' 2023-10-30T14:21:25.037 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/29' 2023-10-30T14:21:25.037 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/32' 2023-10-30T14:21:25.037 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/22' 2023-10-30T14:21:25.037 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/25' 2023-10-30T14:21:25.038 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/30' 2023-10-30T14:21:25.038 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/33' 2023-10-30T14:21:25.038 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/23' 2023-10-30T14:21:25.038 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/35' 2023-10-30T14:21:25.038 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/26' 2023-10-30T14:21:25.039 INFO:tasks.workunit.client.0.smithi057.stdout:removed './210/37' 2023-10-30T14:21:25.039 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './210' 2023-10-30T14:21:25.039 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/21' 2023-10-30T14:21:25.039 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/27' 2023-10-30T14:21:25.039 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/31' 2023-10-30T14:21:25.040 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/36' 2023-10-30T14:21:25.040 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/20' 2023-10-30T14:21:25.040 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/39' 2023-10-30T14:21:25.040 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/34' 2023-10-30T14:21:25.040 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/38' 2023-10-30T14:21:25.040 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/28' 2023-10-30T14:21:25.041 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/24' 2023-10-30T14:21:25.041 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/29' 2023-10-30T14:21:25.041 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/32' 2023-10-30T14:21:25.041 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/22' 2023-10-30T14:21:25.041 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/25' 2023-10-30T14:21:25.042 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/30' 2023-10-30T14:21:25.042 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/33' 2023-10-30T14:21:25.042 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/23' 2023-10-30T14:21:25.042 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/35' 2023-10-30T14:21:25.042 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/26' 2023-10-30T14:21:25.043 INFO:tasks.workunit.client.0.smithi057.stdout:removed './211/37' 2023-10-30T14:21:25.043 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './211' 2023-10-30T14:21:25.043 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/21' 2023-10-30T14:21:25.043 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/27' 2023-10-30T14:21:25.043 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/31' 2023-10-30T14:21:25.044 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/36' 2023-10-30T14:21:25.044 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/20' 2023-10-30T14:21:25.044 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/39' 2023-10-30T14:21:25.044 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/34' 2023-10-30T14:21:25.044 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/38' 2023-10-30T14:21:25.044 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/28' 2023-10-30T14:21:25.045 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/24' 2023-10-30T14:21:25.045 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/29' 2023-10-30T14:21:25.045 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/32' 2023-10-30T14:21:25.045 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/22' 2023-10-30T14:21:25.046 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/25' 2023-10-30T14:21:25.046 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/30' 2023-10-30T14:21:25.046 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/33' 2023-10-30T14:21:25.046 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/23' 2023-10-30T14:21:25.047 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/35' 2023-10-30T14:21:25.047 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/26' 2023-10-30T14:21:25.047 INFO:tasks.workunit.client.0.smithi057.stdout:removed './212/37' 2023-10-30T14:21:25.047 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './212' 2023-10-30T14:21:25.047 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/21' 2023-10-30T14:21:25.048 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/27' 2023-10-30T14:21:25.048 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/31' 2023-10-30T14:21:25.048 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/36' 2023-10-30T14:21:25.048 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/20' 2023-10-30T14:21:25.048 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/39' 2023-10-30T14:21:25.049 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/34' 2023-10-30T14:21:25.049 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/38' 2023-10-30T14:21:25.049 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/28' 2023-10-30T14:21:25.049 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/24' 2023-10-30T14:21:25.049 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/29' 2023-10-30T14:21:25.050 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/32' 2023-10-30T14:21:25.050 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/22' 2023-10-30T14:21:25.050 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/25' 2023-10-30T14:21:25.050 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/30' 2023-10-30T14:21:25.051 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/33' 2023-10-30T14:21:25.051 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/23' 2023-10-30T14:21:25.051 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/35' 2023-10-30T14:21:25.051 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/26' 2023-10-30T14:21:25.051 INFO:tasks.workunit.client.0.smithi057.stdout:removed './213/37' 2023-10-30T14:21:25.052 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './213' 2023-10-30T14:21:25.052 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/21' 2023-10-30T14:21:25.052 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/27' 2023-10-30T14:21:25.052 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/31' 2023-10-30T14:21:25.052 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/36' 2023-10-30T14:21:25.053 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/20' 2023-10-30T14:21:25.053 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/39' 2023-10-30T14:21:25.053 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/34' 2023-10-30T14:21:25.053 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/38' 2023-10-30T14:21:25.053 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/28' 2023-10-30T14:21:25.054 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/24' 2023-10-30T14:21:25.054 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/29' 2023-10-30T14:21:25.054 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/32' 2023-10-30T14:21:25.054 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/22' 2023-10-30T14:21:25.054 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/25' 2023-10-30T14:21:25.055 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/30' 2023-10-30T14:21:25.055 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/33' 2023-10-30T14:21:25.055 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/23' 2023-10-30T14:21:25.055 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/35' 2023-10-30T14:21:25.055 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/26' 2023-10-30T14:21:25.056 INFO:tasks.workunit.client.0.smithi057.stdout:removed './214/37' 2023-10-30T14:21:25.056 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './214' 2023-10-30T14:21:25.056 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/21' 2023-10-30T14:21:25.056 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/27' 2023-10-30T14:21:25.056 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/31' 2023-10-30T14:21:25.057 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/36' 2023-10-30T14:21:25.057 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/20' 2023-10-30T14:21:25.057 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/39' 2023-10-30T14:21:25.057 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/34' 2023-10-30T14:21:25.058 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/38' 2023-10-30T14:21:25.058 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/28' 2023-10-30T14:21:25.058 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/24' 2023-10-30T14:21:25.058 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/29' 2023-10-30T14:21:25.058 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/32' 2023-10-30T14:21:25.059 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/22' 2023-10-30T14:21:25.059 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/25' 2023-10-30T14:21:25.059 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/30' 2023-10-30T14:21:25.059 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/33' 2023-10-30T14:21:25.059 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/23' 2023-10-30T14:21:25.060 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/35' 2023-10-30T14:21:25.060 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/26' 2023-10-30T14:21:25.060 INFO:tasks.workunit.client.0.smithi057.stdout:removed './215/37' 2023-10-30T14:21:25.060 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './215' 2023-10-30T14:21:25.060 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/21' 2023-10-30T14:21:25.061 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/27' 2023-10-30T14:21:25.061 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/31' 2023-10-30T14:21:25.061 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/36' 2023-10-30T14:21:25.061 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/20' 2023-10-30T14:21:25.061 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/39' 2023-10-30T14:21:25.062 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/34' 2023-10-30T14:21:25.062 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/38' 2023-10-30T14:21:30.982 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:21:30.987 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:21:36.896 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/28' 2023-10-30T14:21:36.896 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/24' 2023-10-30T14:21:36.896 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/29' 2023-10-30T14:21:36.896 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/32' 2023-10-30T14:21:36.896 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/22' 2023-10-30T14:21:36.897 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/25' 2023-10-30T14:21:36.897 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/30' 2023-10-30T14:21:36.897 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/33' 2023-10-30T14:21:36.897 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/23' 2023-10-30T14:21:36.897 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/35' 2023-10-30T14:21:36.897 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/26' 2023-10-30T14:21:36.898 INFO:tasks.workunit.client.0.smithi057.stdout:removed './216/37' 2023-10-30T14:21:36.898 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './216' 2023-10-30T14:21:36.898 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/21' 2023-10-30T14:21:36.898 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/27' 2023-10-30T14:21:36.898 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/31' 2023-10-30T14:21:36.899 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/36' 2023-10-30T14:21:36.899 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/20' 2023-10-30T14:21:36.899 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/39' 2023-10-30T14:21:36.899 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/34' 2023-10-30T14:21:36.899 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/38' 2023-10-30T14:21:36.899 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/28' 2023-10-30T14:21:36.900 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/24' 2023-10-30T14:21:36.900 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/29' 2023-10-30T14:21:36.900 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/32' 2023-10-30T14:21:36.900 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/22' 2023-10-30T14:21:36.900 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/25' 2023-10-30T14:21:36.900 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/30' 2023-10-30T14:21:36.901 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/33' 2023-10-30T14:21:36.901 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/23' 2023-10-30T14:21:36.901 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/35' 2023-10-30T14:21:36.901 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/26' 2023-10-30T14:21:36.901 INFO:tasks.workunit.client.0.smithi057.stdout:removed './217/37' 2023-10-30T14:21:36.901 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './217' 2023-10-30T14:21:36.902 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/21' 2023-10-30T14:21:36.902 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/27' 2023-10-30T14:21:36.902 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/31' 2023-10-30T14:21:36.902 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/36' 2023-10-30T14:21:36.902 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/20' 2023-10-30T14:21:36.903 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/39' 2023-10-30T14:21:36.903 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/34' 2023-10-30T14:21:36.903 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/38' 2023-10-30T14:21:36.903 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/28' 2023-10-30T14:21:36.903 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/24' 2023-10-30T14:21:36.903 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/29' 2023-10-30T14:21:36.904 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/32' 2023-10-30T14:21:36.904 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/22' 2023-10-30T14:21:36.904 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/25' 2023-10-30T14:21:36.904 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/30' 2023-10-30T14:21:36.904 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/33' 2023-10-30T14:21:36.904 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/23' 2023-10-30T14:21:36.905 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/35' 2023-10-30T14:21:36.905 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/26' 2023-10-30T14:21:36.905 INFO:tasks.workunit.client.0.smithi057.stdout:removed './218/37' 2023-10-30T14:21:36.905 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './218' 2023-10-30T14:21:36.905 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/21' 2023-10-30T14:21:36.905 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/27' 2023-10-30T14:21:36.906 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/31' 2023-10-30T14:21:36.906 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/36' 2023-10-30T14:21:36.906 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/20' 2023-10-30T14:21:36.906 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/39' 2023-10-30T14:21:36.906 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/34' 2023-10-30T14:21:36.906 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/38' 2023-10-30T14:21:36.907 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/28' 2023-10-30T14:21:36.907 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/24' 2023-10-30T14:21:36.907 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/29' 2023-10-30T14:21:36.907 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/32' 2023-10-30T14:21:36.907 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/22' 2023-10-30T14:21:36.908 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/25' 2023-10-30T14:21:36.908 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/30' 2023-10-30T14:21:36.908 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/33' 2023-10-30T14:21:36.908 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/23' 2023-10-30T14:21:36.908 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/35' 2023-10-30T14:21:36.908 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/26' 2023-10-30T14:21:36.909 INFO:tasks.workunit.client.0.smithi057.stdout:removed './219/37' 2023-10-30T14:21:36.909 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './219' 2023-10-30T14:21:36.909 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/21' 2023-10-30T14:21:36.909 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/27' 2023-10-30T14:21:36.909 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/31' 2023-10-30T14:21:36.909 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/36' 2023-10-30T14:21:36.909 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/20' 2023-10-30T14:21:36.910 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/39' 2023-10-30T14:21:36.910 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/34' 2023-10-30T14:21:36.910 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/38' 2023-10-30T14:21:36.910 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/28' 2023-10-30T14:21:36.910 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/24' 2023-10-30T14:21:36.911 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/29' 2023-10-30T14:21:36.911 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/32' 2023-10-30T14:21:36.911 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/22' 2023-10-30T14:21:36.911 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/25' 2023-10-30T14:21:36.911 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/30' 2023-10-30T14:21:36.911 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/33' 2023-10-30T14:21:36.912 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/23' 2023-10-30T14:21:36.912 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/35' 2023-10-30T14:21:36.912 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/26' 2023-10-30T14:21:36.912 INFO:tasks.workunit.client.0.smithi057.stdout:removed './220/37' 2023-10-30T14:21:36.912 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './220' 2023-10-30T14:21:36.912 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/21' 2023-10-30T14:21:36.913 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/27' 2023-10-30T14:21:36.913 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/31' 2023-10-30T14:21:36.913 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/36' 2023-10-30T14:21:36.913 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/20' 2023-10-30T14:21:36.913 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/39' 2023-10-30T14:21:36.913 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/34' 2023-10-30T14:21:36.914 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/38' 2023-10-30T14:21:36.914 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/28' 2023-10-30T14:21:36.914 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/24' 2023-10-30T14:21:36.914 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/29' 2023-10-30T14:21:36.914 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/32' 2023-10-30T14:21:36.914 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/22' 2023-10-30T14:21:36.915 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/25' 2023-10-30T14:21:36.915 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/30' 2023-10-30T14:21:36.915 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/33' 2023-10-30T14:21:36.915 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/23' 2023-10-30T14:21:36.915 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/35' 2023-10-30T14:21:36.915 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/26' 2023-10-30T14:21:36.916 INFO:tasks.workunit.client.0.smithi057.stdout:removed './221/37' 2023-10-30T14:21:36.916 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './221' 2023-10-30T14:21:36.916 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/21' 2023-10-30T14:21:36.916 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/27' 2023-10-30T14:21:36.916 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/31' 2023-10-30T14:21:36.916 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/36' 2023-10-30T14:21:36.917 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/20' 2023-10-30T14:21:36.917 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/39' 2023-10-30T14:21:36.917 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/34' 2023-10-30T14:21:36.917 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/38' 2023-10-30T14:21:36.917 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/28' 2023-10-30T14:21:36.917 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/24' 2023-10-30T14:21:36.918 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/29' 2023-10-30T14:21:36.918 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/32' 2023-10-30T14:21:36.918 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/22' 2023-10-30T14:21:36.918 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/25' 2023-10-30T14:21:36.918 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/30' 2023-10-30T14:21:36.918 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/33' 2023-10-30T14:21:36.919 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/23' 2023-10-30T14:21:36.919 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/35' 2023-10-30T14:21:36.919 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/26' 2023-10-30T14:21:36.919 INFO:tasks.workunit.client.0.smithi057.stdout:removed './222/37' 2023-10-30T14:21:36.919 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './222' 2023-10-30T14:21:36.919 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/21' 2023-10-30T14:21:36.920 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/27' 2023-10-30T14:21:36.920 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/31' 2023-10-30T14:21:36.920 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/36' 2023-10-30T14:21:36.920 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/20' 2023-10-30T14:21:36.920 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/39' 2023-10-30T14:21:36.920 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/34' 2023-10-30T14:21:36.921 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/38' 2023-10-30T14:21:36.921 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/28' 2023-10-30T14:21:36.921 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/24' 2023-10-30T14:21:36.921 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/29' 2023-10-30T14:21:36.921 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/32' 2023-10-30T14:21:36.922 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/22' 2023-10-30T14:21:36.922 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/25' 2023-10-30T14:21:36.922 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/30' 2023-10-30T14:21:36.922 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/33' 2023-10-30T14:21:36.922 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/23' 2023-10-30T14:21:36.922 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/35' 2023-10-30T14:21:36.923 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/26' 2023-10-30T14:21:36.923 INFO:tasks.workunit.client.0.smithi057.stdout:removed './223/37' 2023-10-30T14:21:36.923 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './223' 2023-10-30T14:21:36.923 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/21' 2023-10-30T14:21:36.923 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/27' 2023-10-30T14:21:36.923 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/31' 2023-10-30T14:21:36.924 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/36' 2023-10-30T14:21:36.924 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/20' 2023-10-30T14:21:36.924 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/39' 2023-10-30T14:21:36.924 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/34' 2023-10-30T14:21:36.924 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/38' 2023-10-30T14:21:36.925 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/28' 2023-10-30T14:21:36.925 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/24' 2023-10-30T14:21:36.925 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/29' 2023-10-30T14:21:36.925 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/32' 2023-10-30T14:21:36.925 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/22' 2023-10-30T14:21:36.925 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/25' 2023-10-30T14:21:36.925 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/30' 2023-10-30T14:21:36.926 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/33' 2023-10-30T14:21:36.926 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/23' 2023-10-30T14:21:36.926 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/35' 2023-10-30T14:21:36.926 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/26' 2023-10-30T14:21:36.926 INFO:tasks.workunit.client.0.smithi057.stdout:removed './224/37' 2023-10-30T14:21:36.927 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './224' 2023-10-30T14:21:36.927 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/21' 2023-10-30T14:21:36.927 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/27' 2023-10-30T14:21:36.927 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/31' 2023-10-30T14:21:36.927 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/36' 2023-10-30T14:21:36.927 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/20' 2023-10-30T14:21:36.928 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/39' 2023-10-30T14:21:36.928 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/34' 2023-10-30T14:21:36.928 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/38' 2023-10-30T14:21:36.928 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/28' 2023-10-30T14:21:36.928 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/24' 2023-10-30T14:21:36.929 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/29' 2023-10-30T14:21:36.929 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/32' 2023-10-30T14:21:36.929 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/22' 2023-10-30T14:21:36.929 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/25' 2023-10-30T14:21:36.929 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/30' 2023-10-30T14:21:36.929 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/33' 2023-10-30T14:21:36.930 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/23' 2023-10-30T14:21:36.930 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/35' 2023-10-30T14:21:36.930 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/26' 2023-10-30T14:21:36.930 INFO:tasks.workunit.client.0.smithi057.stdout:removed './225/37' 2023-10-30T14:21:36.930 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './225' 2023-10-30T14:21:36.930 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/21' 2023-10-30T14:21:36.931 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/27' 2023-10-30T14:21:36.931 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/31' 2023-10-30T14:21:36.931 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/36' 2023-10-30T14:21:36.931 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/20' 2023-10-30T14:21:36.931 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/39' 2023-10-30T14:21:36.931 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/34' 2023-10-30T14:21:36.932 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/38' 2023-10-30T14:21:36.932 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/28' 2023-10-30T14:21:36.932 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/24' 2023-10-30T14:21:42.240 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/29' 2023-10-30T14:21:42.240 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/32' 2023-10-30T14:21:42.240 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/22' 2023-10-30T14:21:42.241 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/25' 2023-10-30T14:21:42.241 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/30' 2023-10-30T14:21:42.241 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/33' 2023-10-30T14:21:42.241 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/23' 2023-10-30T14:21:42.241 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/35' 2023-10-30T14:21:42.241 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/26' 2023-10-30T14:21:42.242 INFO:tasks.workunit.client.0.smithi057.stdout:removed './226/37' 2023-10-30T14:21:42.242 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './226' 2023-10-30T14:21:42.242 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/21' 2023-10-30T14:21:42.242 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/27' 2023-10-30T14:21:42.242 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/31' 2023-10-30T14:21:42.243 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/36' 2023-10-30T14:21:42.243 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/20' 2023-10-30T14:21:42.243 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/39' 2023-10-30T14:21:42.243 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/34' 2023-10-30T14:21:42.243 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/38' 2023-10-30T14:21:42.244 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/28' 2023-10-30T14:21:42.244 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/24' 2023-10-30T14:21:42.244 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/29' 2023-10-30T14:21:42.244 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/32' 2023-10-30T14:21:42.244 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/22' 2023-10-30T14:21:42.245 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/25' 2023-10-30T14:21:42.245 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/30' 2023-10-30T14:21:42.245 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/33' 2023-10-30T14:21:42.245 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/23' 2023-10-30T14:21:42.245 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/35' 2023-10-30T14:21:42.246 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/26' 2023-10-30T14:21:42.246 INFO:tasks.workunit.client.0.smithi057.stdout:removed './227/37' 2023-10-30T14:21:42.246 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './227' 2023-10-30T14:21:42.246 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/21' 2023-10-30T14:21:42.246 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/27' 2023-10-30T14:21:42.247 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/31' 2023-10-30T14:21:42.247 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/36' 2023-10-30T14:21:42.247 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/20' 2023-10-30T14:21:42.247 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/39' 2023-10-30T14:21:42.247 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/34' 2023-10-30T14:21:42.247 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/38' 2023-10-30T14:21:42.248 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/28' 2023-10-30T14:21:42.248 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/24' 2023-10-30T14:21:42.248 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/29' 2023-10-30T14:21:42.248 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/32' 2023-10-30T14:21:42.248 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/22' 2023-10-30T14:21:42.249 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/25' 2023-10-30T14:21:42.249 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/30' 2023-10-30T14:21:42.249 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/33' 2023-10-30T14:21:42.249 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/23' 2023-10-30T14:21:42.249 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/35' 2023-10-30T14:21:42.249 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/26' 2023-10-30T14:21:42.250 INFO:tasks.workunit.client.0.smithi057.stdout:removed './228/37' 2023-10-30T14:21:42.250 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './228' 2023-10-30T14:21:42.250 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/21' 2023-10-30T14:21:42.250 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/27' 2023-10-30T14:21:42.250 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/31' 2023-10-30T14:21:42.251 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/36' 2023-10-30T14:21:42.251 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/20' 2023-10-30T14:21:42.251 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/39' 2023-10-30T14:21:42.251 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/34' 2023-10-30T14:21:42.251 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/38' 2023-10-30T14:21:42.251 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/28' 2023-10-30T14:21:42.252 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/24' 2023-10-30T14:21:42.252 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/29' 2023-10-30T14:21:42.252 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/32' 2023-10-30T14:21:42.252 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/22' 2023-10-30T14:21:42.252 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/25' 2023-10-30T14:21:42.252 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/30' 2023-10-30T14:21:42.253 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/33' 2023-10-30T14:21:42.253 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/23' 2023-10-30T14:21:42.253 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/35' 2023-10-30T14:21:42.253 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/26' 2023-10-30T14:21:42.253 INFO:tasks.workunit.client.0.smithi057.stdout:removed './229/37' 2023-10-30T14:21:42.254 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './229' 2023-10-30T14:21:42.254 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/21' 2023-10-30T14:21:42.254 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/27' 2023-10-30T14:21:42.254 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/31' 2023-10-30T14:21:42.254 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/36' 2023-10-30T14:21:42.255 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/20' 2023-10-30T14:21:42.255 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/39' 2023-10-30T14:21:42.255 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/34' 2023-10-30T14:21:42.255 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/38' 2023-10-30T14:21:42.255 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/28' 2023-10-30T14:21:42.256 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/24' 2023-10-30T14:21:42.256 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/29' 2023-10-30T14:21:42.256 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/32' 2023-10-30T14:21:42.256 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/22' 2023-10-30T14:21:42.256 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/25' 2023-10-30T14:21:42.257 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/30' 2023-10-30T14:21:42.257 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/33' 2023-10-30T14:21:42.257 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/23' 2023-10-30T14:21:42.257 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/35' 2023-10-30T14:21:42.258 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/26' 2023-10-30T14:21:42.258 INFO:tasks.workunit.client.0.smithi057.stdout:removed './230/37' 2023-10-30T14:21:42.258 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './230' 2023-10-30T14:21:42.258 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/21' 2023-10-30T14:21:42.259 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/27' 2023-10-30T14:21:42.259 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/31' 2023-10-30T14:21:42.259 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/36' 2023-10-30T14:21:42.259 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/20' 2023-10-30T14:21:42.259 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/39' 2023-10-30T14:21:42.260 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/34' 2023-10-30T14:21:42.260 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/38' 2023-10-30T14:21:42.260 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/28' 2023-10-30T14:21:42.260 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/24' 2023-10-30T14:21:42.260 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/29' 2023-10-30T14:21:42.260 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/32' 2023-10-30T14:21:42.261 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/22' 2023-10-30T14:21:42.261 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/25' 2023-10-30T14:21:42.261 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/30' 2023-10-30T14:21:42.261 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/33' 2023-10-30T14:21:42.261 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/23' 2023-10-30T14:21:42.262 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/35' 2023-10-30T14:21:42.262 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/26' 2023-10-30T14:21:42.262 INFO:tasks.workunit.client.0.smithi057.stdout:removed './231/37' 2023-10-30T14:21:42.262 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './231' 2023-10-30T14:21:42.262 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/21' 2023-10-30T14:21:42.262 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/27' 2023-10-30T14:21:42.263 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/31' 2023-10-30T14:21:42.263 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/36' 2023-10-30T14:21:42.263 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/20' 2023-10-30T14:21:42.263 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/39' 2023-10-30T14:21:42.263 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/34' 2023-10-30T14:21:42.264 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/38' 2023-10-30T14:21:42.264 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/28' 2023-10-30T14:21:42.264 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/24' 2023-10-30T14:21:42.264 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/29' 2023-10-30T14:21:42.264 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/32' 2023-10-30T14:21:42.264 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/22' 2023-10-30T14:21:42.265 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/25' 2023-10-30T14:21:42.265 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/30' 2023-10-30T14:21:42.265 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/33' 2023-10-30T14:21:42.265 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/23' 2023-10-30T14:21:42.265 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/35' 2023-10-30T14:21:42.265 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/26' 2023-10-30T14:21:42.266 INFO:tasks.workunit.client.0.smithi057.stdout:removed './232/37' 2023-10-30T14:21:42.266 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './232' 2023-10-30T14:21:42.266 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/21' 2023-10-30T14:21:42.266 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/27' 2023-10-30T14:21:42.266 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/31' 2023-10-30T14:21:42.267 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/36' 2023-10-30T14:21:42.267 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/20' 2023-10-30T14:21:42.267 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/39' 2023-10-30T14:21:42.267 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/34' 2023-10-30T14:21:42.267 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/38' 2023-10-30T14:21:42.268 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/28' 2023-10-30T14:21:42.268 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/24' 2023-10-30T14:21:42.268 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/29' 2023-10-30T14:21:42.268 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/32' 2023-10-30T14:21:42.268 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/22' 2023-10-30T14:21:42.268 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/25' 2023-10-30T14:21:42.269 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/30' 2023-10-30T14:21:42.269 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/33' 2023-10-30T14:21:42.269 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/23' 2023-10-30T14:21:42.269 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/35' 2023-10-30T14:21:42.269 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/26' 2023-10-30T14:21:42.270 INFO:tasks.workunit.client.0.smithi057.stdout:removed './233/37' 2023-10-30T14:21:42.270 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './233' 2023-10-30T14:21:42.270 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/21' 2023-10-30T14:21:42.270 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/27' 2023-10-30T14:21:42.270 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/31' 2023-10-30T14:21:42.271 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/36' 2023-10-30T14:21:42.271 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/20' 2023-10-30T14:21:42.271 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/39' 2023-10-30T14:21:42.271 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/34' 2023-10-30T14:21:42.271 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/38' 2023-10-30T14:21:42.272 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/28' 2023-10-30T14:21:42.272 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/24' 2023-10-30T14:21:42.272 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/29' 2023-10-30T14:21:42.272 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/32' 2023-10-30T14:21:42.272 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/22' 2023-10-30T14:21:42.272 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/25' 2023-10-30T14:21:42.273 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/30' 2023-10-30T14:21:42.273 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/33' 2023-10-30T14:21:42.273 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/23' 2023-10-30T14:21:42.273 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/35' 2023-10-30T14:21:42.273 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/26' 2023-10-30T14:21:42.273 INFO:tasks.workunit.client.0.smithi057.stdout:removed './234/37' 2023-10-30T14:21:42.274 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './234' 2023-10-30T14:21:42.274 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/21' 2023-10-30T14:21:42.274 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/27' 2023-10-30T14:21:42.274 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/31' 2023-10-30T14:21:42.274 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/36' 2023-10-30T14:21:42.275 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/20' 2023-10-30T14:21:42.275 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/39' 2023-10-30T14:21:42.275 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/34' 2023-10-30T14:21:42.275 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/38' 2023-10-30T14:21:42.275 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/28' 2023-10-30T14:21:42.275 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/24' 2023-10-30T14:21:42.276 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/29' 2023-10-30T14:21:42.276 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/32' 2023-10-30T14:21:42.276 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/22' 2023-10-30T14:21:42.276 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/25' 2023-10-30T14:21:42.276 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/30' 2023-10-30T14:21:42.276 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/33' 2023-10-30T14:21:42.277 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/23' 2023-10-30T14:21:42.277 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/35' 2023-10-30T14:21:42.277 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/26' 2023-10-30T14:21:42.277 INFO:tasks.workunit.client.0.smithi057.stdout:removed './235/37' 2023-10-30T14:21:42.277 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './235' 2023-10-30T14:21:42.278 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/21' 2023-10-30T14:21:42.278 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/27' 2023-10-30T14:21:42.278 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/31' 2023-10-30T14:21:42.278 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/36' 2023-10-30T14:21:42.278 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/20' 2023-10-30T14:21:42.279 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/39' 2023-10-30T14:21:42.279 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/34' 2023-10-30T14:21:42.279 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/38' 2023-10-30T14:21:42.279 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/28' 2023-10-30T14:21:42.279 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/24' 2023-10-30T14:21:42.279 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/29' 2023-10-30T14:21:42.280 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/32' 2023-10-30T14:21:44.694 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/22' 2023-10-30T14:21:44.694 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/25' 2023-10-30T14:21:44.695 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/30' 2023-10-30T14:21:44.695 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/33' 2023-10-30T14:21:44.695 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/23' 2023-10-30T14:21:44.695 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/35' 2023-10-30T14:21:44.695 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/26' 2023-10-30T14:21:44.695 INFO:tasks.workunit.client.0.smithi057.stdout:removed './236/37' 2023-10-30T14:21:44.696 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './236' 2023-10-30T14:21:44.696 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/21' 2023-10-30T14:21:44.696 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/27' 2023-10-30T14:21:44.696 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/31' 2023-10-30T14:21:44.696 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/36' 2023-10-30T14:21:44.696 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/20' 2023-10-30T14:21:44.697 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/39' 2023-10-30T14:21:44.697 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/34' 2023-10-30T14:21:44.697 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/38' 2023-10-30T14:21:44.697 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/28' 2023-10-30T14:21:44.697 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/24' 2023-10-30T14:21:44.698 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/29' 2023-10-30T14:21:44.698 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/32' 2023-10-30T14:21:44.698 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/22' 2023-10-30T14:21:44.698 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/25' 2023-10-30T14:21:44.698 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/30' 2023-10-30T14:21:44.699 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/33' 2023-10-30T14:21:44.699 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/23' 2023-10-30T14:21:44.699 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/35' 2023-10-30T14:21:44.699 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/26' 2023-10-30T14:21:44.699 INFO:tasks.workunit.client.0.smithi057.stdout:removed './237/37' 2023-10-30T14:21:44.699 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './237' 2023-10-30T14:21:44.700 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/21' 2023-10-30T14:21:44.700 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/27' 2023-10-30T14:21:44.700 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/31' 2023-10-30T14:21:44.700 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/36' 2023-10-30T14:21:44.700 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/20' 2023-10-30T14:21:44.701 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/39' 2023-10-30T14:21:44.701 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/34' 2023-10-30T14:21:44.701 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/38' 2023-10-30T14:21:44.701 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/28' 2023-10-30T14:21:44.701 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/24' 2023-10-30T14:21:44.701 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/29' 2023-10-30T14:21:44.702 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/32' 2023-10-30T14:21:44.702 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/22' 2023-10-30T14:21:44.702 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/25' 2023-10-30T14:21:44.702 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/30' 2023-10-30T14:21:44.702 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/33' 2023-10-30T14:21:44.703 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/23' 2023-10-30T14:21:44.703 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/35' 2023-10-30T14:21:44.703 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/26' 2023-10-30T14:21:44.703 INFO:tasks.workunit.client.0.smithi057.stdout:removed './238/37' 2023-10-30T14:21:44.703 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './238' 2023-10-30T14:21:44.704 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/21' 2023-10-30T14:21:44.704 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/27' 2023-10-30T14:21:44.704 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/31' 2023-10-30T14:21:44.704 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/36' 2023-10-30T14:21:44.704 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/20' 2023-10-30T14:21:44.704 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/39' 2023-10-30T14:21:44.705 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/34' 2023-10-30T14:21:44.705 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/38' 2023-10-30T14:21:44.705 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/28' 2023-10-30T14:21:44.705 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/24' 2023-10-30T14:21:44.705 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/29' 2023-10-30T14:21:44.706 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/32' 2023-10-30T14:21:44.706 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/22' 2023-10-30T14:21:44.706 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/25' 2023-10-30T14:21:44.706 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/30' 2023-10-30T14:21:44.706 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/33' 2023-10-30T14:21:44.706 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/23' 2023-10-30T14:21:44.707 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/35' 2023-10-30T14:21:44.707 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/26' 2023-10-30T14:21:44.707 INFO:tasks.workunit.client.0.smithi057.stdout:removed './239/37' 2023-10-30T14:21:44.707 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './239' 2023-10-30T14:21:44.707 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/21' 2023-10-30T14:21:44.707 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/27' 2023-10-30T14:21:44.708 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/31' 2023-10-30T14:21:44.708 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/36' 2023-10-30T14:21:44.708 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/20' 2023-10-30T14:21:44.708 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/39' 2023-10-30T14:21:44.708 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/34' 2023-10-30T14:21:44.709 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/38' 2023-10-30T14:21:44.709 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/28' 2023-10-30T14:21:44.709 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/24' 2023-10-30T14:21:44.709 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/29' 2023-10-30T14:21:44.709 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/32' 2023-10-30T14:21:44.709 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/22' 2023-10-30T14:21:44.710 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/25' 2023-10-30T14:21:44.710 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/30' 2023-10-30T14:21:44.710 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/33' 2023-10-30T14:21:44.710 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/23' 2023-10-30T14:21:44.710 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/35' 2023-10-30T14:21:44.711 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/26' 2023-10-30T14:21:44.711 INFO:tasks.workunit.client.0.smithi057.stdout:removed './240/37' 2023-10-30T14:21:44.711 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './240' 2023-10-30T14:21:44.711 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/21' 2023-10-30T14:21:44.711 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/27' 2023-10-30T14:21:44.711 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/31' 2023-10-30T14:21:44.711 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/36' 2023-10-30T14:21:44.712 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/20' 2023-10-30T14:21:44.712 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/39' 2023-10-30T14:21:44.712 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/34' 2023-10-30T14:21:44.712 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/38' 2023-10-30T14:21:44.712 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/28' 2023-10-30T14:21:44.713 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/24' 2023-10-30T14:21:44.713 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/29' 2023-10-30T14:21:44.713 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/32' 2023-10-30T14:21:44.713 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/22' 2023-10-30T14:21:44.713 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/25' 2023-10-30T14:21:44.713 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/30' 2023-10-30T14:21:44.714 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/33' 2023-10-30T14:21:44.714 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/23' 2023-10-30T14:21:44.714 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/35' 2023-10-30T14:21:44.714 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/26' 2023-10-30T14:21:44.714 INFO:tasks.workunit.client.0.smithi057.stdout:removed './241/37' 2023-10-30T14:21:44.714 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './241' 2023-10-30T14:21:44.715 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/21' 2023-10-30T14:21:44.715 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/27' 2023-10-30T14:21:44.715 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/31' 2023-10-30T14:21:44.715 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/36' 2023-10-30T14:21:44.715 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/20' 2023-10-30T14:21:44.715 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/39' 2023-10-30T14:21:44.716 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/34' 2023-10-30T14:21:44.716 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/38' 2023-10-30T14:21:44.716 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/28' 2023-10-30T14:21:44.716 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/24' 2023-10-30T14:21:44.716 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/29' 2023-10-30T14:21:44.716 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/32' 2023-10-30T14:21:44.717 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/22' 2023-10-30T14:21:44.717 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/25' 2023-10-30T14:21:44.717 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/30' 2023-10-30T14:21:44.717 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/33' 2023-10-30T14:21:44.717 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/23' 2023-10-30T14:21:44.717 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/35' 2023-10-30T14:21:44.718 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/26' 2023-10-30T14:21:44.718 INFO:tasks.workunit.client.0.smithi057.stdout:removed './242/37' 2023-10-30T14:21:44.718 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './242' 2023-10-30T14:21:44.718 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/21' 2023-10-30T14:21:44.718 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/27' 2023-10-30T14:21:44.719 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/31' 2023-10-30T14:21:44.719 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/36' 2023-10-30T14:21:44.719 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/20' 2023-10-30T14:21:44.719 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/39' 2023-10-30T14:21:44.719 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/34' 2023-10-30T14:21:44.719 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/38' 2023-10-30T14:21:44.720 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/28' 2023-10-30T14:21:44.720 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/24' 2023-10-30T14:21:44.720 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/29' 2023-10-30T14:21:44.720 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/32' 2023-10-30T14:21:44.720 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/22' 2023-10-30T14:21:44.720 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/25' 2023-10-30T14:21:44.721 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/30' 2023-10-30T14:21:44.721 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/33' 2023-10-30T14:21:44.721 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/23' 2023-10-30T14:21:44.721 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/35' 2023-10-30T14:21:44.721 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/26' 2023-10-30T14:21:44.721 INFO:tasks.workunit.client.0.smithi057.stdout:removed './243/37' 2023-10-30T14:21:44.722 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './243' 2023-10-30T14:21:44.722 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/21' 2023-10-30T14:21:44.722 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/27' 2023-10-30T14:21:44.722 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/31' 2023-10-30T14:21:44.722 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/36' 2023-10-30T14:21:44.722 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/20' 2023-10-30T14:21:44.723 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/39' 2023-10-30T14:21:44.723 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/34' 2023-10-30T14:21:44.723 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/38' 2023-10-30T14:21:44.723 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/28' 2023-10-30T14:21:44.723 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/24' 2023-10-30T14:21:44.723 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/29' 2023-10-30T14:21:44.724 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/32' 2023-10-30T14:21:44.724 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/22' 2023-10-30T14:21:44.724 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/25' 2023-10-30T14:21:44.724 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/30' 2023-10-30T14:21:44.724 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/33' 2023-10-30T14:21:44.724 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/23' 2023-10-30T14:21:44.725 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/35' 2023-10-30T14:21:44.725 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/26' 2023-10-30T14:21:44.725 INFO:tasks.workunit.client.0.smithi057.stdout:removed './244/37' 2023-10-30T14:21:44.725 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './244' 2023-10-30T14:21:44.725 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/21' 2023-10-30T14:21:44.725 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/27' 2023-10-30T14:21:44.726 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/31' 2023-10-30T14:21:44.726 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/36' 2023-10-30T14:21:44.726 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/20' 2023-10-30T14:21:44.726 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/39' 2023-10-30T14:21:44.726 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/34' 2023-10-30T14:21:44.726 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/38' 2023-10-30T14:21:44.727 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/28' 2023-10-30T14:21:44.727 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/24' 2023-10-30T14:21:44.727 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/29' 2023-10-30T14:21:44.727 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/32' 2023-10-30T14:21:44.727 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/22' 2023-10-30T14:21:44.727 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/25' 2023-10-30T14:21:44.728 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/30' 2023-10-30T14:21:44.728 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/33' 2023-10-30T14:21:44.728 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/23' 2023-10-30T14:21:44.728 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/35' 2023-10-30T14:21:44.728 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/26' 2023-10-30T14:21:44.728 INFO:tasks.workunit.client.0.smithi057.stdout:removed './245/37' 2023-10-30T14:21:44.729 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './245' 2023-10-30T14:21:44.729 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/21' 2023-10-30T14:21:44.729 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/27' 2023-10-30T14:21:44.729 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/31' 2023-10-30T14:21:44.729 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/36' 2023-10-30T14:21:44.729 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/20' 2023-10-30T14:21:44.730 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/39' 2023-10-30T14:21:44.730 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/34' 2023-10-30T14:21:44.730 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/38' 2023-10-30T14:21:44.730 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/28' 2023-10-30T14:21:44.730 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/24' 2023-10-30T14:21:44.731 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/29' 2023-10-30T14:21:44.731 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/32' 2023-10-30T14:21:44.731 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/22' 2023-10-30T14:21:45.981 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/25' 2023-10-30T14:21:45.981 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/30' 2023-10-30T14:21:45.982 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/33' 2023-10-30T14:21:45.982 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/23' 2023-10-30T14:21:45.982 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/35' 2023-10-30T14:21:45.982 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/26' 2023-10-30T14:21:45.982 INFO:tasks.workunit.client.0.smithi057.stdout:removed './246/37' 2023-10-30T14:21:45.983 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './246' 2023-10-30T14:21:45.983 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/21' 2023-10-30T14:21:45.983 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/27' 2023-10-30T14:21:45.983 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/31' 2023-10-30T14:21:45.983 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/36' 2023-10-30T14:21:45.983 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/20' 2023-10-30T14:21:45.984 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/39' 2023-10-30T14:21:45.984 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/34' 2023-10-30T14:21:45.984 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/38' 2023-10-30T14:21:45.984 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/28' 2023-10-30T14:21:45.984 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/24' 2023-10-30T14:21:45.984 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/29' 2023-10-30T14:21:45.985 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/32' 2023-10-30T14:21:45.985 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/22' 2023-10-30T14:21:45.985 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/25' 2023-10-30T14:21:45.985 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/30' 2023-10-30T14:21:45.985 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/33' 2023-10-30T14:21:45.985 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/23' 2023-10-30T14:21:45.986 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/35' 2023-10-30T14:21:45.986 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/26' 2023-10-30T14:21:45.986 INFO:tasks.workunit.client.0.smithi057.stdout:removed './247/37' 2023-10-30T14:21:45.986 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './247' 2023-10-30T14:21:45.986 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/21' 2023-10-30T14:21:45.987 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/27' 2023-10-30T14:21:45.987 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/31' 2023-10-30T14:21:45.987 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/36' 2023-10-30T14:21:45.987 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/20' 2023-10-30T14:21:45.987 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/39' 2023-10-30T14:21:45.987 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/34' 2023-10-30T14:21:45.988 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/38' 2023-10-30T14:21:45.988 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/28' 2023-10-30T14:21:45.988 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/24' 2023-10-30T14:21:45.988 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/29' 2023-10-30T14:21:45.988 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/32' 2023-10-30T14:21:45.988 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/22' 2023-10-30T14:21:45.989 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/25' 2023-10-30T14:21:45.989 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/30' 2023-10-30T14:21:45.989 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/33' 2023-10-30T14:21:45.989 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/23' 2023-10-30T14:21:45.989 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/35' 2023-10-30T14:21:45.989 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/26' 2023-10-30T14:21:45.990 INFO:tasks.workunit.client.0.smithi057.stdout:removed './248/37' 2023-10-30T14:21:45.990 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './248' 2023-10-30T14:21:45.990 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/21' 2023-10-30T14:21:45.990 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/27' 2023-10-30T14:21:45.990 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/31' 2023-10-30T14:21:45.990 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/36' 2023-10-30T14:21:45.991 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/20' 2023-10-30T14:21:45.991 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/39' 2023-10-30T14:21:45.991 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/34' 2023-10-30T14:21:45.991 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/38' 2023-10-30T14:21:45.991 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/28' 2023-10-30T14:21:45.991 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/24' 2023-10-30T14:21:45.992 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/29' 2023-10-30T14:21:45.992 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/32' 2023-10-30T14:21:45.992 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/22' 2023-10-30T14:21:45.992 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/25' 2023-10-30T14:21:45.992 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/30' 2023-10-30T14:21:45.993 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/33' 2023-10-30T14:21:45.993 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/23' 2023-10-30T14:21:45.993 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/35' 2023-10-30T14:21:45.993 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/26' 2023-10-30T14:21:45.993 INFO:tasks.workunit.client.0.smithi057.stdout:removed './249/37' 2023-10-30T14:21:45.993 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './249' 2023-10-30T14:21:45.994 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/21' 2023-10-30T14:21:45.994 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/27' 2023-10-30T14:21:45.994 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/31' 2023-10-30T14:21:45.994 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/36' 2023-10-30T14:21:45.994 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/20' 2023-10-30T14:21:45.995 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/39' 2023-10-30T14:21:45.995 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/34' 2023-10-30T14:21:45.995 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/38' 2023-10-30T14:21:45.995 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/28' 2023-10-30T14:21:45.995 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/24' 2023-10-30T14:21:45.996 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/29' 2023-10-30T14:21:45.996 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/32' 2023-10-30T14:21:45.996 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/22' 2023-10-30T14:21:45.996 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/25' 2023-10-30T14:21:45.996 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/30' 2023-10-30T14:21:45.996 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/33' 2023-10-30T14:21:45.997 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/23' 2023-10-30T14:21:45.997 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/35' 2023-10-30T14:21:45.997 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/26' 2023-10-30T14:21:45.997 INFO:tasks.workunit.client.0.smithi057.stdout:removed './250/37' 2023-10-30T14:21:45.997 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './250' 2023-10-30T14:21:45.997 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/21' 2023-10-30T14:21:45.998 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/27' 2023-10-30T14:21:45.998 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/31' 2023-10-30T14:21:45.998 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/36' 2023-10-30T14:21:45.998 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/20' 2023-10-30T14:21:45.998 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/39' 2023-10-30T14:21:45.999 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/34' 2023-10-30T14:21:45.999 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/38' 2023-10-30T14:21:45.999 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/28' 2023-10-30T14:21:45.999 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/24' 2023-10-30T14:21:45.999 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/29' 2023-10-30T14:21:45.999 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/32' 2023-10-30T14:21:46.000 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/22' 2023-10-30T14:21:46.000 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/25' 2023-10-30T14:21:46.000 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/30' 2023-10-30T14:21:46.000 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/33' 2023-10-30T14:21:46.000 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/23' 2023-10-30T14:21:46.000 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/35' 2023-10-30T14:21:46.001 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/26' 2023-10-30T14:21:46.001 INFO:tasks.workunit.client.0.smithi057.stdout:removed './251/37' 2023-10-30T14:21:46.001 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './251' 2023-10-30T14:21:46.001 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/21' 2023-10-30T14:21:46.001 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/27' 2023-10-30T14:21:46.001 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/31' 2023-10-30T14:21:46.002 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/36' 2023-10-30T14:21:46.002 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/20' 2023-10-30T14:21:46.002 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/39' 2023-10-30T14:21:46.002 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/34' 2023-10-30T14:21:46.002 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/38' 2023-10-30T14:21:46.003 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/28' 2023-10-30T14:21:46.003 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/24' 2023-10-30T14:21:46.003 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/29' 2023-10-30T14:21:46.003 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/32' 2023-10-30T14:21:46.003 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/22' 2023-10-30T14:21:46.003 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/25' 2023-10-30T14:21:46.004 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/30' 2023-10-30T14:21:46.004 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/33' 2023-10-30T14:21:46.004 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/23' 2023-10-30T14:21:46.004 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/35' 2023-10-30T14:21:46.004 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/26' 2023-10-30T14:21:46.004 INFO:tasks.workunit.client.0.smithi057.stdout:removed './252/37' 2023-10-30T14:21:46.005 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './252' 2023-10-30T14:21:46.005 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/21' 2023-10-30T14:21:46.005 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/27' 2023-10-30T14:21:46.005 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/31' 2023-10-30T14:21:46.005 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/36' 2023-10-30T14:21:46.005 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/20' 2023-10-30T14:21:46.006 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/39' 2023-10-30T14:21:46.006 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/34' 2023-10-30T14:21:46.006 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/38' 2023-10-30T14:21:46.006 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/28' 2023-10-30T14:21:46.006 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/24' 2023-10-30T14:21:46.007 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/29' 2023-10-30T14:21:46.007 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/32' 2023-10-30T14:21:46.007 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/22' 2023-10-30T14:21:46.007 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/25' 2023-10-30T14:21:46.007 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/30' 2023-10-30T14:21:46.007 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/33' 2023-10-30T14:21:46.008 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/23' 2023-10-30T14:21:46.008 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/35' 2023-10-30T14:21:46.008 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/26' 2023-10-30T14:21:46.008 INFO:tasks.workunit.client.0.smithi057.stdout:removed './253/37' 2023-10-30T14:21:46.008 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './253' 2023-10-30T14:21:46.008 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/21' 2023-10-30T14:21:46.009 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/27' 2023-10-30T14:21:46.009 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/31' 2023-10-30T14:21:46.009 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/36' 2023-10-30T14:21:46.009 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/20' 2023-10-30T14:21:46.009 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/39' 2023-10-30T14:21:46.009 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/34' 2023-10-30T14:21:46.009 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/38' 2023-10-30T14:21:46.010 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/28' 2023-10-30T14:21:46.010 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/24' 2023-10-30T14:21:46.010 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/29' 2023-10-30T14:21:46.010 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/32' 2023-10-30T14:21:46.010 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/22' 2023-10-30T14:21:46.011 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/25' 2023-10-30T14:21:46.011 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/30' 2023-10-30T14:21:46.011 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/33' 2023-10-30T14:21:46.011 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/23' 2023-10-30T14:21:46.011 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/35' 2023-10-30T14:21:46.011 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/26' 2023-10-30T14:21:46.012 INFO:tasks.workunit.client.0.smithi057.stdout:removed './254/37' 2023-10-30T14:21:46.012 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './254' 2023-10-30T14:21:46.012 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/21' 2023-10-30T14:21:46.012 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/27' 2023-10-30T14:21:46.012 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/31' 2023-10-30T14:21:46.012 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/36' 2023-10-30T14:21:46.012 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/20' 2023-10-30T14:21:46.013 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/39' 2023-10-30T14:21:46.013 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/34' 2023-10-30T14:21:46.013 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/38' 2023-10-30T14:21:46.013 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/28' 2023-10-30T14:21:46.013 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/24' 2023-10-30T14:21:46.013 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/29' 2023-10-30T14:21:46.014 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/32' 2023-10-30T14:21:46.014 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/22' 2023-10-30T14:21:46.014 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/25' 2023-10-30T14:21:46.014 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/30' 2023-10-30T14:21:46.014 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/33' 2023-10-30T14:21:46.015 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/23' 2023-10-30T14:21:46.015 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/35' 2023-10-30T14:21:46.015 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/26' 2023-10-30T14:21:46.015 INFO:tasks.workunit.client.0.smithi057.stdout:removed './255/37' 2023-10-30T14:21:46.015 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './255' 2023-10-30T14:21:46.015 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/21' 2023-10-30T14:21:46.015 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/27' 2023-10-30T14:21:46.016 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/31' 2023-10-30T14:21:46.016 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/36' 2023-10-30T14:21:46.016 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/20' 2023-10-30T14:21:46.016 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/39' 2023-10-30T14:21:46.016 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/34' 2023-10-30T14:21:46.016 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/38' 2023-10-30T14:21:46.017 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/28' 2023-10-30T14:21:46.017 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/24' 2023-10-30T14:21:46.017 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/29' 2023-10-30T14:21:46.017 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/32' 2023-10-30T14:21:46.017 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/22' 2023-10-30T14:21:46.017 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/25' 2023-10-30T14:21:46.018 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/30' 2023-10-30T14:21:52.589 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/33' 2023-10-30T14:21:52.589 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/23' 2023-10-30T14:21:52.589 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/35' 2023-10-30T14:21:52.590 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/26' 2023-10-30T14:21:52.590 INFO:tasks.workunit.client.0.smithi057.stdout:removed './256/37' 2023-10-30T14:21:52.590 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './256' 2023-10-30T14:21:52.590 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/21' 2023-10-30T14:21:52.590 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/27' 2023-10-30T14:21:52.591 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/31' 2023-10-30T14:21:52.591 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/36' 2023-10-30T14:21:52.591 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/20' 2023-10-30T14:21:52.591 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/39' 2023-10-30T14:21:52.591 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/34' 2023-10-30T14:21:52.592 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/38' 2023-10-30T14:21:52.592 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/28' 2023-10-30T14:21:52.592 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/24' 2023-10-30T14:21:52.592 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/29' 2023-10-30T14:21:52.592 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/32' 2023-10-30T14:21:52.592 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/22' 2023-10-30T14:21:52.593 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/25' 2023-10-30T14:21:52.593 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/30' 2023-10-30T14:21:52.593 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/33' 2023-10-30T14:21:52.593 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/23' 2023-10-30T14:21:52.593 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/35' 2023-10-30T14:21:52.593 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/26' 2023-10-30T14:21:52.594 INFO:tasks.workunit.client.0.smithi057.stdout:removed './257/37' 2023-10-30T14:21:52.594 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './257' 2023-10-30T14:21:52.594 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/21' 2023-10-30T14:21:52.594 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/27' 2023-10-30T14:21:52.594 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/31' 2023-10-30T14:21:52.595 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/36' 2023-10-30T14:21:52.595 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/20' 2023-10-30T14:21:52.595 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/39' 2023-10-30T14:21:52.595 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/34' 2023-10-30T14:21:52.595 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/38' 2023-10-30T14:21:52.596 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/28' 2023-10-30T14:21:52.596 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/24' 2023-10-30T14:21:52.596 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/29' 2023-10-30T14:21:52.596 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/32' 2023-10-30T14:21:52.596 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/22' 2023-10-30T14:21:52.596 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/25' 2023-10-30T14:21:52.597 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/30' 2023-10-30T14:21:52.597 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/33' 2023-10-30T14:21:52.597 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/23' 2023-10-30T14:21:52.597 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/35' 2023-10-30T14:21:52.597 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/26' 2023-10-30T14:21:52.598 INFO:tasks.workunit.client.0.smithi057.stdout:removed './258/37' 2023-10-30T14:21:52.598 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './258' 2023-10-30T14:21:52.598 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/21' 2023-10-30T14:21:52.598 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/27' 2023-10-30T14:21:52.598 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/31' 2023-10-30T14:21:52.599 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/36' 2023-10-30T14:21:52.599 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/20' 2023-10-30T14:21:52.599 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/39' 2023-10-30T14:21:52.599 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/34' 2023-10-30T14:21:52.599 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/38' 2023-10-30T14:21:52.600 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/28' 2023-10-30T14:21:52.600 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/24' 2023-10-30T14:21:52.600 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/29' 2023-10-30T14:21:52.600 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/32' 2023-10-30T14:21:52.600 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/22' 2023-10-30T14:21:52.600 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/25' 2023-10-30T14:21:52.601 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/30' 2023-10-30T14:21:52.601 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/33' 2023-10-30T14:21:52.601 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/23' 2023-10-30T14:21:52.601 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/35' 2023-10-30T14:21:52.601 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/26' 2023-10-30T14:21:52.602 INFO:tasks.workunit.client.0.smithi057.stdout:removed './259/37' 2023-10-30T14:21:52.602 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './259' 2023-10-30T14:21:52.602 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/21' 2023-10-30T14:21:52.602 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/27' 2023-10-30T14:21:52.602 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/31' 2023-10-30T14:21:52.603 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/36' 2023-10-30T14:21:52.603 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/20' 2023-10-30T14:21:52.603 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/39' 2023-10-30T14:21:52.603 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/34' 2023-10-30T14:21:52.603 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/38' 2023-10-30T14:21:52.604 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/28' 2023-10-30T14:21:52.604 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/24' 2023-10-30T14:21:52.604 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/29' 2023-10-30T14:21:52.604 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/32' 2023-10-30T14:21:52.604 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/22' 2023-10-30T14:21:52.604 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/25' 2023-10-30T14:21:52.605 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/30' 2023-10-30T14:21:52.605 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/33' 2023-10-30T14:21:52.605 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/23' 2023-10-30T14:21:52.605 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/35' 2023-10-30T14:21:52.605 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/26' 2023-10-30T14:21:52.606 INFO:tasks.workunit.client.0.smithi057.stdout:removed './260/37' 2023-10-30T14:21:52.606 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './260' 2023-10-30T14:21:52.606 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/21' 2023-10-30T14:21:52.606 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/27' 2023-10-30T14:21:52.606 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/31' 2023-10-30T14:21:52.606 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/36' 2023-10-30T14:21:52.607 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/20' 2023-10-30T14:21:52.607 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/39' 2023-10-30T14:21:52.607 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/34' 2023-10-30T14:21:52.607 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/38' 2023-10-30T14:21:52.607 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/28' 2023-10-30T14:21:52.608 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/24' 2023-10-30T14:21:52.608 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/29' 2023-10-30T14:21:52.608 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/32' 2023-10-30T14:21:52.608 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/22' 2023-10-30T14:21:52.608 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/25' 2023-10-30T14:21:52.608 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/30' 2023-10-30T14:21:52.609 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/33' 2023-10-30T14:21:52.609 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/23' 2023-10-30T14:21:52.609 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/35' 2023-10-30T14:21:52.609 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/26' 2023-10-30T14:21:52.609 INFO:tasks.workunit.client.0.smithi057.stdout:removed './261/37' 2023-10-30T14:21:52.610 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './261' 2023-10-30T14:21:52.610 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/21' 2023-10-30T14:21:52.610 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/27' 2023-10-30T14:21:52.610 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/31' 2023-10-30T14:21:52.610 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/36' 2023-10-30T14:21:52.611 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/20' 2023-10-30T14:21:52.611 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/39' 2023-10-30T14:21:52.611 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/34' 2023-10-30T14:21:52.611 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/38' 2023-10-30T14:21:52.611 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/28' 2023-10-30T14:21:52.612 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/24' 2023-10-30T14:21:52.612 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/29' 2023-10-30T14:21:52.612 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/32' 2023-10-30T14:21:52.612 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/22' 2023-10-30T14:21:52.612 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/25' 2023-10-30T14:21:52.613 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/30' 2023-10-30T14:21:52.613 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/33' 2023-10-30T14:21:52.613 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/23' 2023-10-30T14:21:52.613 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/35' 2023-10-30T14:21:52.613 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/26' 2023-10-30T14:21:52.613 INFO:tasks.workunit.client.0.smithi057.stdout:removed './262/37' 2023-10-30T14:21:52.614 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './262' 2023-10-30T14:21:52.614 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/21' 2023-10-30T14:21:52.614 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/27' 2023-10-30T14:21:52.614 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/31' 2023-10-30T14:21:52.614 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/36' 2023-10-30T14:21:52.615 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/20' 2023-10-30T14:21:52.615 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/39' 2023-10-30T14:21:52.615 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/34' 2023-10-30T14:21:52.615 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/38' 2023-10-30T14:21:52.615 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/28' 2023-10-30T14:21:52.615 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/24' 2023-10-30T14:21:52.616 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/29' 2023-10-30T14:21:52.616 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/32' 2023-10-30T14:21:52.616 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/22' 2023-10-30T14:21:52.616 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/25' 2023-10-30T14:21:52.616 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/30' 2023-10-30T14:21:52.616 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/33' 2023-10-30T14:21:52.617 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/23' 2023-10-30T14:21:52.617 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/35' 2023-10-30T14:21:52.617 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/26' 2023-10-30T14:21:52.617 INFO:tasks.workunit.client.0.smithi057.stdout:removed './263/37' 2023-10-30T14:21:52.617 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './263' 2023-10-30T14:21:52.617 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/21' 2023-10-30T14:21:52.618 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/27' 2023-10-30T14:21:52.618 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/31' 2023-10-30T14:21:52.618 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/36' 2023-10-30T14:21:52.618 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/20' 2023-10-30T14:21:52.618 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/39' 2023-10-30T14:21:52.619 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/34' 2023-10-30T14:21:52.619 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/38' 2023-10-30T14:21:52.619 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/28' 2023-10-30T14:21:52.619 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/24' 2023-10-30T14:21:52.619 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/29' 2023-10-30T14:21:52.620 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/32' 2023-10-30T14:21:52.620 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/22' 2023-10-30T14:21:52.620 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/25' 2023-10-30T14:21:52.620 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/30' 2023-10-30T14:21:52.620 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/33' 2023-10-30T14:21:52.620 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/23' 2023-10-30T14:21:52.621 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/35' 2023-10-30T14:21:52.621 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/26' 2023-10-30T14:21:52.621 INFO:tasks.workunit.client.0.smithi057.stdout:removed './264/37' 2023-10-30T14:21:52.621 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './264' 2023-10-30T14:21:52.621 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/21' 2023-10-30T14:21:52.622 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/27' 2023-10-30T14:21:52.622 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/31' 2023-10-30T14:21:52.622 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/36' 2023-10-30T14:21:52.622 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/20' 2023-10-30T14:21:52.622 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/39' 2023-10-30T14:21:52.623 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/34' 2023-10-30T14:21:52.623 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/38' 2023-10-30T14:21:52.623 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/28' 2023-10-30T14:21:52.623 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/24' 2023-10-30T14:21:52.623 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/29' 2023-10-30T14:21:52.624 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/32' 2023-10-30T14:21:52.624 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/22' 2023-10-30T14:21:52.624 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/25' 2023-10-30T14:21:52.624 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/30' 2023-10-30T14:21:52.624 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/33' 2023-10-30T14:21:52.625 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/23' 2023-10-30T14:21:52.625 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/35' 2023-10-30T14:21:52.625 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/26' 2023-10-30T14:21:52.625 INFO:tasks.workunit.client.0.smithi057.stdout:removed './265/37' 2023-10-30T14:21:52.625 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './265' 2023-10-30T14:21:52.625 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/21' 2023-10-30T14:21:52.626 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/27' 2023-10-30T14:21:52.626 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/31' 2023-10-30T14:21:52.626 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/36' 2023-10-30T14:21:52.626 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/20' 2023-10-30T14:21:52.626 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/39' 2023-10-30T14:21:52.627 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/34' 2023-10-30T14:21:52.627 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/38' 2023-10-30T14:21:52.627 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/28' 2023-10-30T14:21:52.627 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/24' 2023-10-30T14:21:52.627 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/29' 2023-10-30T14:21:52.628 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/32' 2023-10-30T14:21:52.628 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/22' 2023-10-30T14:21:52.628 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/25' 2023-10-30T14:21:52.628 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/30' 2023-10-30T14:21:52.628 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/33' 2023-10-30T14:21:52.628 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/23' 2023-10-30T14:21:52.732 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:21:52.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.874+0000 7fd782e09640 1 Processor -- start 2023-10-30T14:21:52.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.874+0000 7fd782e09640 1 -- start start 2023-10-30T14:21:52.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.874+0000 7fd782e09640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd77c153c80 0x7fd77c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:52.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.874+0000 7fd782e09640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd77c0c4a10 0x7fd77c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:52.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.874+0000 7fd782e09640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd77c15a3b0 0x7fd77c0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:52.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.874+0000 7fd782e09640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd77c15dec0 con 0x7fd77c0c4a10 2023-10-30T14:21:52.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.874+0000 7fd782e09640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd77c1599a0 con 0x7fd77c15a3b0 2023-10-30T14:21:52.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.874+0000 7fd782e09640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c159f20 con 0x7fd77c153c80 2023-10-30T14:21:52.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.878+0000 7fd77bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd77c0c4a10 0x7fd77c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:52.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.878+0000 7fd77bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd77c0c4a10 0x7fd77c0c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33392/0 (socket says 172.21.15.57:33392) 2023-10-30T14:21:52.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.878+0000 7fd77bfff640 1 -- 172.21.15.57:0/2371972606 learned_addr learned my addr 172.21.15.57:0/2371972606 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:21:52.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.878+0000 7fd780ba5640 1 --2- 172.21.15.57:0/2371972606 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd77c153c80 0x7fd77c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:52.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.878+0000 7fd7813a6640 1 --2- 172.21.15.57:0/2371972606 >> v2:172.21.15.88:3300/0 conn(0x7fd77c15a3b0 0x7fd77c0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:52.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.878+0000 7fd77bfff640 1 -- 172.21.15.57:0/2371972606 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd77c153c80 msgr2=0x7fd77c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:52.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.878+0000 7fd77bfff640 1 --2- 172.21.15.57:0/2371972606 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd77c153c80 0x7fd77c15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:52.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.878+0000 7fd77bfff640 1 -- 172.21.15.57:0/2371972606 >> v2:172.21.15.88:3300/0 conn(0x7fd77c15a3b0 msgr2=0x7fd77c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:52.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.878+0000 7fd77bfff640 1 --2- 172.21.15.57:0/2371972606 >> v2:172.21.15.88:3300/0 conn(0x7fd77c15a3b0 0x7fd77c0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:52.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.878+0000 7fd77bfff640 1 -- 172.21.15.57:0/2371972606 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77c0c47e0 con 0x7fd77c0c4a10 2023-10-30T14:21:52.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.878+0000 7fd7813a6640 1 --2- 172.21.15.57:0/2371972606 >> v2:172.21.15.88:3300/0 conn(0x7fd77c15a3b0 0x7fd77c0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:21:52.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.878+0000 7fd780ba5640 1 --2- 172.21.15.57:0/2371972606 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd77c153c80 0x7fd77c15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:21:52.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.878+0000 7fd77bfff640 1 --2- 172.21.15.57:0/2371972606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd77c0c4a10 0x7fd77c0c42a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd77c0cbbc0 tx=0x7fd76801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f22bd0046fe88a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:52.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.878+0000 7fd77b7fe640 1 -- 172.21.15.57:0/2371972606 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7680a89b0 con 0x7fd77c0c4a10 2023-10-30T14:21:52.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.878+0000 7fd77b7fe640 1 -- 172.21.15.57:0/2371972606 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7680a8b10 con 0x7fd77c0c4a10 2023-10-30T14:21:52.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.878+0000 7fd77b7fe640 1 -- 172.21.15.57:0/2371972606 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7680a8ea0 con 0x7fd77c0c4a10 2023-10-30T14:21:52.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd782e09640 1 -- 172.21.15.57:0/2371972606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd77c0c4a10 msgr2=0x7fd77c0c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:52.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd782e09640 1 --2- 172.21.15.57:0/2371972606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd77c0c4a10 0x7fd77c0c42a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd77c0cbbc0 tx=0x7fd76801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:21:52.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd782e09640 1 -- 172.21.15.57:0/2371972606 shutdown_connections 2023-10-30T14:21:52.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd782e09640 1 --2- 172.21.15.57:0/2371972606 >> v2:172.21.15.88:3300/0 conn(0x7fd77c15a3b0 0x7fd77c0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:52.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd782e09640 1 --2- 172.21.15.57:0/2371972606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd77c0c4a10 0x7fd77c0c42a0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:52.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd782e09640 1 --2- 172.21.15.57:0/2371972606 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd77c153c80 0x7fd77c15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:52.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd782e09640 1 -- 172.21.15.57:0/2371972606 >> 172.21.15.57:0/2371972606 conn(0x7fd77c0715c0 msgr2=0x7fd77c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:52.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd782e09640 1 -- 172.21.15.57:0/2371972606 shutdown_connections 2023-10-30T14:21:52.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd782e09640 1 -- 172.21.15.57:0/2371972606 wait complete. 2023-10-30T14:21:52.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd782e09640 1 Processor -- start 2023-10-30T14:21:52.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd782e09640 1 -- start start 2023-10-30T14:21:52.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd782e09640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd77c0c4a10 0x7fd77c14f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:52.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd782e09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd77c153c80 0x7fd77c0ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:52.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd782e09640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd77c15a3b0 0x7fd77c1f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:52.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd782e09640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c0b3010 con 0x7fd77c15a3b0 2023-10-30T14:21:52.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd782e09640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c0b4810 con 0x7fd77c153c80 2023-10-30T14:21:52.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd782e09640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c0b3f50 con 0x7fd77c0c4a10 2023-10-30T14:21:52.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd7813a6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd77c15a3b0 0x7fd77c1f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:52.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd7813a6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd77c15a3b0 0x7fd77c1f2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33400/0 (socket says 172.21.15.57:33400) 2023-10-30T14:21:52.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd7813a6640 1 -- 172.21.15.57:0/3995055682 learned_addr learned my addr 172.21.15.57:0/3995055682 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:21:52.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd780ba5640 1 --2- 172.21.15.57:0/3995055682 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd77c0c4a10 0x7fd77c14f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:52.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd77bfff640 1 --2- 172.21.15.57:0/3995055682 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd77c153c80 0x7fd77c0ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:52.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd7813a6640 1 -- 172.21.15.57:0/3995055682 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd77c0c4a10 msgr2=0x7fd77c14f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:52.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd7813a6640 1 --2- 172.21.15.57:0/3995055682 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd77c0c4a10 0x7fd77c14f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:52.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd7813a6640 1 -- 172.21.15.57:0/3995055682 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd77c153c80 msgr2=0x7fd77c0ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:52.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd7813a6640 1 --2- 172.21.15.57:0/3995055682 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd77c153c80 0x7fd77c0ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:52.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.882+0000 7fd7813a6640 1 -- 172.21.15.57:0/3995055682 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7680094d0 con 0x7fd77c15a3b0 2023-10-30T14:21:52.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.886+0000 7fd7813a6640 1 --2- 172.21.15.57:0/3995055682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd77c15a3b0 0x7fd77c1f2a10 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fd774007e20 tx=0x7fd77400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:52.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.886+0000 7fd7797fa640 1 -- 172.21.15.57:0/3995055682 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd774003ba0 con 0x7fd77c15a3b0 2023-10-30T14:21:52.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.886+0000 7fd7797fa640 1 -- 172.21.15.57:0/3995055682 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd77400f040 con 0x7fd77c15a3b0 2023-10-30T14:21:52.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.886+0000 7fd782e09640 1 -- 172.21.15.57:0/3995055682 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd77c14fef0 con 0x7fd77c15a3b0 2023-10-30T14:21:52.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.886+0000 7fd7797fa640 1 -- 172.21.15.57:0/3995055682 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77400de80 con 0x7fd77c15a3b0 2023-10-30T14:21:52.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.886+0000 7fd782e09640 1 -- 172.21.15.57:0/3995055682 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd77c0b41c0 con 0x7fd77c15a3b0 2023-10-30T14:21:52.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.886+0000 7fd782e09640 1 -- 172.21.15.57:0/3995055682 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd77c1f2730 con 0x7fd77c15a3b0 2023-10-30T14:21:52.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.886+0000 7fd7797fa640 1 -- 172.21.15.57:0/3995055682 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd774004030 con 0x7fd77c15a3b0 2023-10-30T14:21:52.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.886+0000 7fd7797fa640 1 --2- 172.21.15.57:0/3995055682 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd7600664a0 0x7fd760068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:52.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.886+0000 7fd7797fa640 1 -- 172.21.15.57:0/3995055682 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fd774084320 con 0x7fd77c15a3b0 2023-10-30T14:21:52.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.890+0000 7fd780ba5640 1 --2- 172.21.15.57:0/3995055682 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd7600664a0 0x7fd760068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:52.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.890+0000 7fd780ba5640 1 --2- 172.21.15.57:0/3995055682 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd7600664a0 0x7fd760068960 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd7700098d0 tx=0x7fd770008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:52.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:52.894+0000 7fd7797fa640 1 -- 172.21.15.57:0/3995055682 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd77c1f2730 con 0x7fd77c15a3b0 2023-10-30T14:21:53.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.082+0000 7fd782e09640 1 -- 172.21.15.57:0/3995055682 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd77c1f2730 con 0x7fd77c15a3b0 2023-10-30T14:21:53.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.086+0000 7fd7797fa640 1 -- 172.21.15.57:0/3995055682 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fd77c1f2730 con 0x7fd77c15a3b0 2023-10-30T14:21:53.086 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:21:53.087 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:21:29.808921+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:active","state_seq":28,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:21:53.087 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 56 2023-10-30T14:21:53.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.086+0000 7fd782e09640 1 -- 172.21.15.57:0/3995055682 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd7600664a0 msgr2=0x7fd760068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:53.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.086+0000 7fd782e09640 1 --2- 172.21.15.57:0/3995055682 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd7600664a0 0x7fd760068960 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd7700098d0 tx=0x7fd770008040 comp rx=0 tx=0).stop 2023-10-30T14:21:53.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.086+0000 7fd782e09640 1 -- 172.21.15.57:0/3995055682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd77c15a3b0 msgr2=0x7fd77c1f2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:53.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.086+0000 7fd782e09640 1 --2- 172.21.15.57:0/3995055682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd77c15a3b0 0x7fd77c1f2a10 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fd774007e20 tx=0x7fd77400d4a0 comp rx=0 tx=0).stop 2023-10-30T14:21:53.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.086+0000 7fd782e09640 1 -- 172.21.15.57:0/3995055682 shutdown_connections 2023-10-30T14:21:53.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.086+0000 7fd782e09640 1 --2- 172.21.15.57:0/3995055682 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd7600664a0 0x7fd760068960 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:53.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.086+0000 7fd782e09640 1 --2- 172.21.15.57:0/3995055682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd77c15a3b0 0x7fd77c1f2a10 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:53.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.086+0000 7fd782e09640 1 --2- 172.21.15.57:0/3995055682 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd77c153c80 0x7fd77c0ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:53.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.086+0000 7fd782e09640 1 --2- 172.21.15.57:0/3995055682 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd77c0c4a10 0x7fd77c14f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:53.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.086+0000 7fd782e09640 1 -- 172.21.15.57:0/3995055682 >> 172.21.15.57:0/3995055682 conn(0x7fd77c0715c0 msgr2=0x7fd77c0c9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:53.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.086+0000 7fd782e09640 1 -- 172.21.15.57:0/3995055682 shutdown_connections 2023-10-30T14:21:53.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.086+0000 7fd782e09640 1 -- 172.21.15.57:0/3995055682 wait complete. 2023-10-30T14:21:53.108 INFO:tasks.mds_thrash.fs.[cephfs]:thrashing max_mds: 3 -> 2 2023-10-30T14:21:53.108 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2023-10-30T14:21:53.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.254+0000 7f75145c4640 1 Processor -- start 2023-10-30T14:21:53.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.254+0000 7f75145c4640 1 -- start start 2023-10-30T14:21:53.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.254+0000 7f75145c4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f750c0c56a0 0x7f750c1500e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:53.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.254+0000 7f75145c4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f750c0c64d0 0x7f750c0b0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:53.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.254+0000 7f75145c4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750c0c5e90 con 0x7f750c072520 2023-10-30T14:21:53.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.254+0000 7f75145c4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f750c0c5ca0 con 0x7f750c0c56a0 2023-10-30T14:21:53.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.254+0000 7f75145c4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750c0cb430 con 0x7f750c0c64d0 2023-10-30T14:21:53.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.258+0000 7f7511b5f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f750c0c56a0 0x7f750c1500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:53.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.258+0000 7f7511b5f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f750c0c56a0 0x7f750c1500e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33414/0 (socket says 172.21.15.57:33414) 2023-10-30T14:21:53.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.258+0000 7f7511b5f640 1 -- 172.21.15.57:0/2030689812 learned_addr learned my addr 172.21.15.57:0/2030689812 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:21:53.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.258+0000 7f7512b61640 1 --2- 172.21.15.57:0/2030689812 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f750c0c64d0 0x7f750c0b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:53.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.258+0000 7f7511b5f640 1 -- 172.21.15.57:0/2030689812 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f750c0c64d0 msgr2=0x7f750c0b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:53.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.258+0000 7f7511b5f640 1 --2- 172.21.15.57:0/2030689812 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f750c0c64d0 0x7f750c0b0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:53.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.258+0000 7f7511b5f640 1 -- 172.21.15.57:0/2030689812 >> v1:172.21.15.88:6789/0 conn(0x7f750c072520 legacy=0x7f750c0cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:53.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.258+0000 7f7511b5f640 1 -- 172.21.15.57:0/2030689812 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750c0c6170 con 0x7f750c0c56a0 2023-10-30T14:21:53.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.258+0000 7f7511b5f640 1 --2- 172.21.15.57:0/2030689812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f750c0c56a0 0x7f750c1500e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f74fc0098a0 tx=0x7f74fc01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea15c9c95091f159 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:53.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.258+0000 7f751135e640 1 -- 172.21.15.57:0/2030689812 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74fc0a8a10 con 0x7f750c0c56a0 2023-10-30T14:21:53.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.258+0000 7f751135e640 1 -- 172.21.15.57:0/2030689812 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74fc0a8b70 con 0x7f750c0c56a0 2023-10-30T14:21:53.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.258+0000 7f751135e640 1 -- 172.21.15.57:0/2030689812 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74fc0b0440 con 0x7f750c0c56a0 2023-10-30T14:21:53.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f75145c4640 1 -- 172.21.15.57:0/2030689812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f750c0c56a0 msgr2=0x7f750c1500e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:53.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f75145c4640 1 --2- 172.21.15.57:0/2030689812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f750c0c56a0 0x7f750c1500e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f74fc0098a0 tx=0x7f74fc01ed10 comp rx=0 tx=0).stop 2023-10-30T14:21:53.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f75145c4640 1 -- 172.21.15.57:0/2030689812 shutdown_connections 2023-10-30T14:21:53.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f75145c4640 1 --2- 172.21.15.57:0/2030689812 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f750c0c64d0 0x7f750c0b0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:53.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f75145c4640 1 --2- 172.21.15.57:0/2030689812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f750c0c56a0 0x7f750c1500e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:53.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f75145c4640 1 -- 172.21.15.57:0/2030689812 >> 172.21.15.57:0/2030689812 conn(0x7f750c06da40 msgr2=0x7f750c06de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:53.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f75145c4640 1 -- 172.21.15.57:0/2030689812 shutdown_connections 2023-10-30T14:21:53.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f75145c4640 1 -- 172.21.15.57:0/2030689812 wait complete. 2023-10-30T14:21:53.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f75145c4640 1 Processor -- start 2023-10-30T14:21:53.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f75145c4640 1 -- start start 2023-10-30T14:21:53.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f75145c4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f750c072520 0x7f750c1f0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:53.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f75145c4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f750c0c56a0 0x7f750c0c0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:53.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f75145c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f750c0c64d0 0x7f750c1ed070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:53.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f75145c4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750c1f1030 con 0x7f750c0c56a0 2023-10-30T14:21:53.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f75145c4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750c0c1b30 con 0x7f750c0c64d0 2023-10-30T14:21:53.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f75145c4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750c1ecb70 con 0x7f750c072520 2023-10-30T14:21:53.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f7511b5f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f750c0c56a0 0x7f750c0c0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:53.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f7511b5f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f750c0c56a0 0x7f750c0c0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33430/0 (socket says 172.21.15.57:33430) 2023-10-30T14:21:53.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f7511b5f640 1 -- 172.21.15.57:0/2497997673 learned_addr learned my addr 172.21.15.57:0/2497997673 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:21:53.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f7512360640 1 --2- 172.21.15.57:0/2497997673 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f750c072520 0x7f750c1f0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:53.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f7512b61640 1 --2- 172.21.15.57:0/2497997673 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f750c0c64d0 0x7f750c1ed070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:53.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f7511b5f640 1 -- 172.21.15.57:0/2497997673 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f750c072520 msgr2=0x7f750c1f0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:53.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f7511b5f640 1 --2- 172.21.15.57:0/2497997673 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f750c072520 0x7f750c1f0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:53.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f7511b5f640 1 -- 172.21.15.57:0/2497997673 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f750c0c64d0 msgr2=0x7f750c1ed070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:53.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f7511b5f640 1 --2- 172.21.15.57:0/2497997673 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f750c0c64d0 0x7f750c1ed070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:53.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f7511b5f640 1 -- 172.21.15.57:0/2497997673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74fc0094d0 con 0x7f750c0c56a0 2023-10-30T14:21:53.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f7512b61640 1 --2- 172.21.15.57:0/2497997673 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f750c0c64d0 0x7f750c1ed070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:21:53.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.262+0000 7f7512360640 1 --2- 172.21.15.57:0/2497997673 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f750c072520 0x7f750c1f0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:21:53.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.266+0000 7f7511b5f640 1 --2- 172.21.15.57:0/2497997673 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f750c0c56a0 0x7f750c0c0060 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f74fc0b3040 tx=0x7f74fc01bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:53.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.266+0000 7f7502ffd640 1 -- 172.21.15.57:0/2497997673 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74fc0be070 con 0x7f750c0c56a0 2023-10-30T14:21:53.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.266+0000 7f7502ffd640 1 -- 172.21.15.57:0/2497997673 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74fc0ba400 con 0x7f750c0c56a0 2023-10-30T14:21:53.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.266+0000 7f7502ffd640 1 -- 172.21.15.57:0/2497997673 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74fc0ba7d0 con 0x7f750c0c56a0 2023-10-30T14:21:53.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.266+0000 7f75145c4640 1 -- 172.21.15.57:0/2497997673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f750c159940 con 0x7f750c0c56a0 2023-10-30T14:21:53.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.266+0000 7f75145c4640 1 -- 172.21.15.57:0/2497997673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f750c159ef0 con 0x7f750c0c56a0 2023-10-30T14:21:53.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.266+0000 7f75145c4640 1 -- 172.21.15.57:0/2497997673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f750c04e170 con 0x7f750c0c56a0 2023-10-30T14:21:53.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.266+0000 7f7502ffd640 1 -- 172.21.15.57:0/2497997673 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f74fc0c3410 con 0x7f750c0c56a0 2023-10-30T14:21:53.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.266+0000 7f7502ffd640 1 --2- 172.21.15.57:0/2497997673 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f74f00664d0 0x7f74f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:53.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.266+0000 7f7502ffd640 1 -- 172.21.15.57:0/2497997673 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f74fc121c60 con 0x7f750c0c56a0 2023-10-30T14:21:53.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.266+0000 7f7512360640 1 --2- 172.21.15.57:0/2497997673 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f74f00664d0 0x7f74f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:53.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.270+0000 7f7512360640 1 --2- 172.21.15.57:0/2497997673 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f74f00664d0 0x7f74f0068990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7508000f70 tx=0x7f7508009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:53.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.270+0000 7f7502ffd640 1 -- 172.21.15.57:0/2497997673 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f74fc0eeeb0 con 0x7f750c0c56a0 2023-10-30T14:21:53.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.466+0000 7f75145c4640 1 -- 172.21.15.57:0/2497997673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f750c070120 con 0x7f750c0c56a0 2023-10-30T14:21:53.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.822+0000 7f7502ffd640 1 -- 172.21.15.57:0/2497997673 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v57) v1 ==== 105+0+0 (secure 0 0 0) 0x7f74fc0f24d0 con 0x7f750c0c56a0 2023-10-30T14:21:53.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.822+0000 7f75145c4640 1 -- 172.21.15.57:0/2497997673 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f74f00664d0 msgr2=0x7f74f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:53.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.822+0000 7f75145c4640 1 --2- 172.21.15.57:0/2497997673 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f74f00664d0 0x7f74f0068990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7508000f70 tx=0x7f7508009040 comp rx=0 tx=0).stop 2023-10-30T14:21:53.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.822+0000 7f75145c4640 1 -- 172.21.15.57:0/2497997673 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f750c0c56a0 msgr2=0x7f750c0c0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:53.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.822+0000 7f75145c4640 1 --2- 172.21.15.57:0/2497997673 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f750c0c56a0 0x7f750c0c0060 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f74fc0b3040 tx=0x7f74fc01bab0 comp rx=0 tx=0).stop 2023-10-30T14:21:53.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.822+0000 7f75145c4640 1 -- 172.21.15.57:0/2497997673 shutdown_connections 2023-10-30T14:21:53.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.822+0000 7f75145c4640 1 --2- 172.21.15.57:0/2497997673 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f74f00664d0 0x7f74f0068990 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:53.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.822+0000 7f75145c4640 1 --2- 172.21.15.57:0/2497997673 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f750c0c64d0 0x7f750c1ed070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:53.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.822+0000 7f75145c4640 1 --2- 172.21.15.57:0/2497997673 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f750c0c56a0 0x7f750c0c0060 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:53.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.822+0000 7f75145c4640 1 --2- 172.21.15.57:0/2497997673 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f750c072520 0x7f750c1f0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:53.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.822+0000 7f75145c4640 1 -- 172.21.15.57:0/2497997673 >> 172.21.15.57:0/2497997673 conn(0x7f750c06da40 msgr2=0x7f750c14e180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:53.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.822+0000 7f75145c4640 1 -- 172.21.15.57:0/2497997673 shutdown_connections 2023-10-30T14:21:53.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.822+0000 7f75145c4640 1 -- 172.21.15.57:0/2497997673 wait complete. 2023-10-30T14:21:53.846 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:21:53.847 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:21:53.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.990+0000 7f5a20dab640 1 Processor -- start 2023-10-30T14:21:53.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.990+0000 7f5a20dab640 1 -- start start 2023-10-30T14:21:53.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.990+0000 7f5a20dab640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5a1c153c80 0x7f5a1c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:53.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.990+0000 7f5a20dab640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a1c159f20 con 0x7f5a1c0c4a10 2023-10-30T14:21:53.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.990+0000 7f5a20dab640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a1c15d620 con 0x7f5a1c0c42a0 2023-10-30T14:21:53.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.990+0000 7f5a20dab640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a1c15dec0 con 0x7f5a1c153c80 2023-10-30T14:21:53.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.990+0000 7f5a1a59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5a1c153c80 0x7f5a1c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:53.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.990+0000 7f5a19d9b640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f5a1c0c4a10 0x7f5a1c15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:52482/0 (socket says 172.21.15.57:52482) 2023-10-30T14:21:53.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.990+0000 7f5a19d9b640 1 -- 172.21.15.57:0/2725564797 learned_addr learned my addr 172.21.15.57:0/2725564797 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:21:53.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.990+0000 7f5a1a59c640 1 -- 172.21.15.57:0/2725564797 >> v1:172.21.15.88:6789/0 conn(0x7f5a1c0c42a0 legacy=0x7f5a1c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:53.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.990+0000 7f5a1a59c640 1 -- 172.21.15.57:0/2725564797 >> v1:172.21.15.57:6789/0 conn(0x7f5a1c0c4a10 legacy=0x7f5a1c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:53.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.990+0000 7f5a1a59c640 1 -- 172.21.15.57:0/2725564797 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1c15d900 con 0x7f5a1c153c80 2023-10-30T14:21:53.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.990+0000 7f5a1959a640 1 -- 172.21.15.57:0/2725564797 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217876069 0 0) 0x7f5a1c159f20 con 0x7f5a1c0c4a10 2023-10-30T14:21:53.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.994+0000 7f5a1a59c640 1 --2- 172.21.15.57:0/2725564797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a1c153c80 0x7f5a1c15cf90 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0097c0 tx=0x7f5a0c01ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee42a696fa902ce server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:53.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.994+0000 7f5a1959a640 1 -- 172.21.15.57:0/2725564797 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c0a8930 con 0x7f5a1c153c80 2023-10-30T14:21:53.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.994+0000 7f5a1959a640 1 -- 172.21.15.57:0/2725564797 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0c0a8a90 con 0x7f5a1c153c80 2023-10-30T14:21:53.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.994+0000 7f5a1959a640 1 -- 172.21.15.57:0/2725564797 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c0a8e20 con 0x7f5a1c153c80 2023-10-30T14:21:53.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.994+0000 7f5a20dab640 1 -- 172.21.15.57:0/2725564797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a1c153c80 msgr2=0x7f5a1c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:53.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.994+0000 7f5a20dab640 1 --2- 172.21.15.57:0/2725564797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a1c153c80 0x7f5a1c15cf90 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0097c0 tx=0x7f5a0c01ec30 comp rx=0 tx=0).stop 2023-10-30T14:21:53.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.994+0000 7f5a20dab640 1 -- 172.21.15.57:0/2725564797 shutdown_connections 2023-10-30T14:21:53.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.994+0000 7f5a20dab640 1 --2- 172.21.15.57:0/2725564797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a1c153c80 0x7f5a1c15cf90 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:53.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.994+0000 7f5a20dab640 1 -- 172.21.15.57:0/2725564797 >> 172.21.15.57:0/2725564797 conn(0x7f5a1c0715c0 msgr2=0x7f5a1c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:53.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.994+0000 7f5a20dab640 1 -- 172.21.15.57:0/2725564797 shutdown_connections 2023-10-30T14:21:53.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.994+0000 7f5a20dab640 1 -- 172.21.15.57:0/2725564797 wait complete. 2023-10-30T14:21:54.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a20dab640 1 Processor -- start 2023-10-30T14:21:54.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a20dab640 1 -- start start 2023-10-30T14:21:54.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a20dab640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a1c0c42a0 0x7f5a1c159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:54.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a20dab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a1c0c4a10 0x7f5a1c15ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:54.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a20dab640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a1c153c80 0x7f5a1c15b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:54.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a20dab640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c1ef900 con 0x7f5a1c0c4a10 2023-10-30T14:21:54.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a20dab640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c15cad0 con 0x7f5a1c153c80 2023-10-30T14:21:54.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a20dab640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c1eeca0 con 0x7f5a1c0c42a0 2023-10-30T14:21:54.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a19d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a1c0c4a10 0x7f5a1c15ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:54.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a19d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a1c0c4a10 0x7f5a1c15ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38648/0 (socket says 172.21.15.57:38648) 2023-10-30T14:21:54.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a19d9b640 1 -- 172.21.15.57:0/2291350886 learned_addr learned my addr 172.21.15.57:0/2291350886 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:21:54.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a1a59c640 1 --2- 172.21.15.57:0/2291350886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a1c0c42a0 0x7f5a1c159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:54.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a1ad9d640 1 --2- 172.21.15.57:0/2291350886 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a1c153c80 0x7f5a1c15b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:54.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a19d9b640 1 -- 172.21.15.57:0/2291350886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a1c0c42a0 msgr2=0x7f5a1c159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:54.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a19d9b640 1 --2- 172.21.15.57:0/2291350886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a1c0c42a0 0x7f5a1c159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:54.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a19d9b640 1 -- 172.21.15.57:0/2291350886 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a1c153c80 msgr2=0x7f5a1c15b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:54.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a19d9b640 1 --2- 172.21.15.57:0/2291350886 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a1c153c80 0x7f5a1c15b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:54.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a19d9b640 1 -- 172.21.15.57:0/2291350886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c009420 con 0x7f5a1c0c4a10 2023-10-30T14:21:54.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a1ad9d640 1 --2- 172.21.15.57:0/2291350886 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a1c153c80 0x7f5a1c15b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:21:54.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a19d9b640 1 --2- 172.21.15.57:0/2291350886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a1c0c4a10 0x7f5a1c15ab60 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5a1000c4b0 tx=0x7f5a1000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:54.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a0affd640 1 -- 172.21.15.57:0/2291350886 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a100074e0 con 0x7f5a1c0c4a10 2023-10-30T14:21:54.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a0affd640 1 -- 172.21.15.57:0/2291350886 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a10007660 con 0x7f5a1c0c4a10 2023-10-30T14:21:54.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a0affd640 1 -- 172.21.15.57:0/2291350886 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a10004a30 con 0x7f5a1c0c4a10 2023-10-30T14:21:54.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a20dab640 1 -- 172.21.15.57:0/2291350886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a1c157150 con 0x7f5a1c0c4a10 2023-10-30T14:21:54.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a20dab640 1 -- 172.21.15.57:0/2291350886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a1c1eef60 con 0x7f5a1c0c4a10 2023-10-30T14:21:54.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:53.998+0000 7f5a20dab640 1 -- 172.21.15.57:0/2291350886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a1c04e170 con 0x7f5a1c0c4a10 2023-10-30T14:21:54.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.002+0000 7f5a0affd640 1 -- 172.21.15.57:0/2291350886 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5a10004b90 con 0x7f5a1c0c4a10 2023-10-30T14:21:54.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.002+0000 7f5a0affd640 1 --2- 172.21.15.57:0/2291350886 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f59fc0664a0 0x7f59fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:54.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.002+0000 7f5a0affd640 1 -- 172.21.15.57:0/2291350886 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f5a10083320 con 0x7f5a1c0c4a10 2023-10-30T14:21:54.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.002+0000 7f5a1a59c640 1 --2- 172.21.15.57:0/2291350886 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f59fc0664a0 0x7f59fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:54.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.002+0000 7f5a1a59c640 1 --2- 172.21.15.57:0/2291350886 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f59fc0664a0 0x7f59fc068960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0b2000 tx=0x7f5a0c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:54.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.006+0000 7f5a0affd640 1 -- 172.21.15.57:0/2291350886 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5a10050650 con 0x7f5a1c0c4a10 2023-10-30T14:21:54.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.194+0000 7f5a20dab640 1 -- 172.21.15.57:0/2291350886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a1c14dae0 con 0x7f5a1c0c4a10 2023-10-30T14:21:54.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.194+0000 7f5a0affd640 1 -- 172.21.15.57:0/2291350886 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f5a10018090 con 0x7f5a1c0c4a10 2023-10-30T14:21:54.196 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:21:54.196 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:21:53.826787+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:stopping","state_seq":28,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:21:54.197 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 58 2023-10-30T14:21:54.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.194+0000 7f5a20dab640 1 -- 172.21.15.57:0/2291350886 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f59fc0664a0 msgr2=0x7f59fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:54.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.194+0000 7f5a20dab640 1 --2- 172.21.15.57:0/2291350886 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f59fc0664a0 0x7f59fc068960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0b2000 tx=0x7f5a0c008040 comp rx=0 tx=0).stop 2023-10-30T14:21:54.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.194+0000 7f5a20dab640 1 -- 172.21.15.57:0/2291350886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a1c0c4a10 msgr2=0x7f5a1c15ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:54.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.194+0000 7f5a20dab640 1 --2- 172.21.15.57:0/2291350886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a1c0c4a10 0x7f5a1c15ab60 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5a1000c4b0 tx=0x7f5a1000c970 comp rx=0 tx=0).stop 2023-10-30T14:21:54.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.194+0000 7f5a20dab640 1 -- 172.21.15.57:0/2291350886 shutdown_connections 2023-10-30T14:21:54.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.194+0000 7f5a20dab640 1 --2- 172.21.15.57:0/2291350886 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f59fc0664a0 0x7f59fc068960 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:54.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.194+0000 7f5a20dab640 1 --2- 172.21.15.57:0/2291350886 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a1c153c80 0x7f5a1c15b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:54.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.194+0000 7f5a20dab640 1 --2- 172.21.15.57:0/2291350886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a1c0c4a10 0x7f5a1c15ab60 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:54.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.194+0000 7f5a20dab640 1 --2- 172.21.15.57:0/2291350886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a1c0c42a0 0x7f5a1c159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:54.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.194+0000 7f5a20dab640 1 -- 172.21.15.57:0/2291350886 >> 172.21.15.57:0/2291350886 conn(0x7f5a1c0715c0 msgr2=0x7f5a1c06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:54.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.194+0000 7f5a20dab640 1 -- 172.21.15.57:0/2291350886 shutdown_connections 2023-10-30T14:21:54.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:54.194+0000 7f5a20dab640 1 -- 172.21.15.57:0/2291350886 wait complete. 2023-10-30T14:21:54.217 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T14:21:54.218 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 58, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 58, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5078": { "addr": "172.21.15.88:6839/1677924344", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1677924344, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1677924344, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5078, "incarnation": 48, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:stopping", "state_seq": 28 }, "gid_5133": { "addr": "172.21.15.57:6834/2912050965", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2912050965, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2912050965, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5133, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 15 }, "gid_5289": { "addr": "172.21.15.57:6836/2427081813", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2427081813, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2427081813, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5289, "incarnation": 35, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:21:53.826787+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5289, "mds_1": 5133, "mds_2": 5078 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.57:6837/862088802", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 862088802, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 862088802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 54, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5586, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:21:56.219 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:21:56.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.398+0000 7fc572e09640 1 Processor -- start 2023-10-30T14:21:56.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.398+0000 7fc572e09640 1 -- start start 2023-10-30T14:21:56.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.398+0000 7fc572e09640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc56c153c80 0x7fc56c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:56.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.398+0000 7fc572e09640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc56c0c4a10 0x7fc56c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:56.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.398+0000 7fc572e09640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc56c15a3b0 0x7fc56c0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:56.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.398+0000 7fc572e09640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc56c15dec0 con 0x7fc56c0c4a10 2023-10-30T14:21:56.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.398+0000 7fc572e09640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc56c1599a0 con 0x7fc56c15a3b0 2023-10-30T14:21:56.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.398+0000 7fc572e09640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c159f20 con 0x7fc56c153c80 2023-10-30T14:21:56.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.398+0000 7fc56bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc56c0c4a10 0x7fc56c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:56.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.398+0000 7fc56bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc56c0c4a10 0x7fc56c0c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38654/0 (socket says 172.21.15.57:38654) 2023-10-30T14:21:56.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.398+0000 7fc56bfff640 1 -- 172.21.15.57:0/1466998691 learned_addr learned my addr 172.21.15.57:0/1466998691 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:21:56.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.398+0000 7fc570ba5640 1 --2- 172.21.15.57:0/1466998691 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc56c153c80 0x7fc56c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:56.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.398+0000 7fc5713a6640 1 --2- 172.21.15.57:0/1466998691 >> v2:172.21.15.88:3300/0 conn(0x7fc56c15a3b0 0x7fc56c0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:56.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.398+0000 7fc56bfff640 1 -- 172.21.15.57:0/1466998691 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc56c153c80 msgr2=0x7fc56c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:56.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.398+0000 7fc56bfff640 1 --2- 172.21.15.57:0/1466998691 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc56c153c80 0x7fc56c15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:56.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.398+0000 7fc56bfff640 1 -- 172.21.15.57:0/1466998691 >> v2:172.21.15.88:3300/0 conn(0x7fc56c15a3b0 msgr2=0x7fc56c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:56.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.398+0000 7fc56bfff640 1 --2- 172.21.15.57:0/1466998691 >> v2:172.21.15.88:3300/0 conn(0x7fc56c15a3b0 0x7fc56c0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:56.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.398+0000 7fc56bfff640 1 -- 172.21.15.57:0/1466998691 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56c0c47e0 con 0x7fc56c0c4a10 2023-10-30T14:21:56.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.402+0000 7fc56bfff640 1 --2- 172.21.15.57:0/1466998691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc56c0c4a10 0x7fc56c0c42a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc56c0cbbc0 tx=0x7fc55801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b58647afaf829044 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:56.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.402+0000 7fc56b7fe640 1 -- 172.21.15.57:0/1466998691 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5580a89b0 con 0x7fc56c0c4a10 2023-10-30T14:21:56.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.402+0000 7fc56b7fe640 1 -- 172.21.15.57:0/1466998691 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5580a8b10 con 0x7fc56c0c4a10 2023-10-30T14:21:56.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.402+0000 7fc56b7fe640 1 -- 172.21.15.57:0/1466998691 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5580a8ea0 con 0x7fc56c0c4a10 2023-10-30T14:21:56.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.402+0000 7fc572e09640 1 -- 172.21.15.57:0/1466998691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc56c0c4a10 msgr2=0x7fc56c0c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:56.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.402+0000 7fc572e09640 1 --2- 172.21.15.57:0/1466998691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc56c0c4a10 0x7fc56c0c42a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc56c0cbbc0 tx=0x7fc55801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:21:56.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.402+0000 7fc572e09640 1 -- 172.21.15.57:0/1466998691 shutdown_connections 2023-10-30T14:21:56.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.402+0000 7fc572e09640 1 --2- 172.21.15.57:0/1466998691 >> v2:172.21.15.88:3300/0 conn(0x7fc56c15a3b0 0x7fc56c0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:56.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.402+0000 7fc572e09640 1 --2- 172.21.15.57:0/1466998691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc56c0c4a10 0x7fc56c0c42a0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:56.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.402+0000 7fc572e09640 1 --2- 172.21.15.57:0/1466998691 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc56c153c80 0x7fc56c15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:56.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.402+0000 7fc572e09640 1 -- 172.21.15.57:0/1466998691 >> 172.21.15.57:0/1466998691 conn(0x7fc56c0715c0 msgr2=0x7fc56c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:56.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.402+0000 7fc572e09640 1 -- 172.21.15.57:0/1466998691 shutdown_connections 2023-10-30T14:21:56.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.402+0000 7fc572e09640 1 -- 172.21.15.57:0/1466998691 wait complete. 2023-10-30T14:21:56.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc572e09640 1 Processor -- start 2023-10-30T14:21:56.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc572e09640 1 -- start start 2023-10-30T14:21:56.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc572e09640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc56c0c4a10 0x7fc56c14e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:56.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc572e09640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc56c153c80 0x7fc56c0ce890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:56.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc572e09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc56c15a3b0 0x7fc56c0b2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:56.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc572e09640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c14df30 con 0x7fc56c153c80 2023-10-30T14:21:56.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc572e09640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c14e480 con 0x7fc56c15a3b0 2023-10-30T14:21:56.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc572e09640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c14f820 con 0x7fc56c0c4a10 2023-10-30T14:21:56.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc56bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc56c153c80 0x7fc56c0ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:56.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc56bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc56c153c80 0x7fc56c0ce890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38666/0 (socket says 172.21.15.57:38666) 2023-10-30T14:21:56.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc56bfff640 1 -- 172.21.15.57:0/3854132675 learned_addr learned my addr 172.21.15.57:0/3854132675 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:21:56.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc570ba5640 1 --2- 172.21.15.57:0/3854132675 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc56c0c4a10 0x7fc56c14e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:56.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc5713a6640 1 --2- 172.21.15.57:0/3854132675 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc56c15a3b0 0x7fc56c0b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:56.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc56bfff640 1 -- 172.21.15.57:0/3854132675 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc56c0c4a10 msgr2=0x7fc56c14e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:56.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc56bfff640 1 --2- 172.21.15.57:0/3854132675 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc56c0c4a10 0x7fc56c14e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:56.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc56bfff640 1 -- 172.21.15.57:0/3854132675 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc56c15a3b0 msgr2=0x7fc56c0b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:56.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc56bfff640 1 --2- 172.21.15.57:0/3854132675 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc56c15a3b0 0x7fc56c0b2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:56.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc56bfff640 1 -- 172.21.15.57:0/3854132675 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5580094d0 con 0x7fc56c153c80 2023-10-30T14:21:56.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc56bfff640 1 --2- 172.21.15.57:0/3854132675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc56c153c80 0x7fc56c0ce890 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc55801d3a0 tx=0x7fc55801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:56.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc5697fa640 1 -- 172.21.15.57:0/3854132675 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5580ba600 con 0x7fc56c153c80 2023-10-30T14:21:56.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc5697fa640 1 -- 172.21.15.57:0/3854132675 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5580ba760 con 0x7fc56c153c80 2023-10-30T14:21:56.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc572e09640 1 -- 172.21.15.57:0/3854132675 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc56c14ee50 con 0x7fc56c153c80 2023-10-30T14:21:56.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc5697fa640 1 -- 172.21.15.57:0/3854132675 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5580bab10 con 0x7fc56c153c80 2023-10-30T14:21:56.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.406+0000 7fc572e09640 1 -- 172.21.15.57:0/3854132675 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc56c14fd40 con 0x7fc56c153c80 2023-10-30T14:21:56.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.410+0000 7fc572e09640 1 -- 172.21.15.57:0/3854132675 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc56c04e170 con 0x7fc56c153c80 2023-10-30T14:21:56.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.410+0000 7fc5697fa640 1 -- 172.21.15.57:0/3854132675 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc5580bac70 con 0x7fc56c153c80 2023-10-30T14:21:56.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.410+0000 7fc5697fa640 1 --2- 172.21.15.57:0/3854132675 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5500664d0 0x7fc550068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:56.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.410+0000 7fc5697fa640 1 -- 172.21.15.57:0/3854132675 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fc5581225c0 con 0x7fc56c153c80 2023-10-30T14:21:56.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.410+0000 7fc570ba5640 1 --2- 172.21.15.57:0/3854132675 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5500664d0 0x7fc550068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:56.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.410+0000 7fc570ba5640 1 --2- 172.21.15.57:0/3854132675 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5500664d0 0x7fc550068990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc5600045f0 tx=0x7fc560008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:56.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.414+0000 7fc5697fa640 1 -- 172.21.15.57:0/3854132675 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc5580ef830 con 0x7fc56c153c80 2023-10-30T14:21:56.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.606+0000 7fc572e09640 1 -- 172.21.15.57:0/3854132675 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc56c0cb9d0 con 0x7fc56c153c80 2023-10-30T14:21:56.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.606+0000 7fc5697fa640 1 -- 172.21.15.57:0/3854132675 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+6532 (secure 0 0 0) 0x7fc5580f2e50 con 0x7fc56c153c80 2023-10-30T14:21:56.609 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:21:56.609 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:21:53.826787+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:stopping","state_seq":28,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:21:56.610 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 58 2023-10-30T14:21:56.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.610+0000 7fc572e09640 1 -- 172.21.15.57:0/3854132675 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5500664d0 msgr2=0x7fc550068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:56.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.610+0000 7fc572e09640 1 --2- 172.21.15.57:0/3854132675 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5500664d0 0x7fc550068990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc5600045f0 tx=0x7fc560008040 comp rx=0 tx=0).stop 2023-10-30T14:21:56.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.610+0000 7fc572e09640 1 -- 172.21.15.57:0/3854132675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc56c153c80 msgr2=0x7fc56c0ce890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:56.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.610+0000 7fc572e09640 1 --2- 172.21.15.57:0/3854132675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc56c153c80 0x7fc56c0ce890 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc55801d3a0 tx=0x7fc55801ba90 comp rx=0 tx=0).stop 2023-10-30T14:21:56.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.610+0000 7fc572e09640 1 -- 172.21.15.57:0/3854132675 shutdown_connections 2023-10-30T14:21:56.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.610+0000 7fc572e09640 1 --2- 172.21.15.57:0/3854132675 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5500664d0 0x7fc550068990 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:56.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.610+0000 7fc572e09640 1 --2- 172.21.15.57:0/3854132675 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc56c15a3b0 0x7fc56c0b2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:56.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.610+0000 7fc572e09640 1 --2- 172.21.15.57:0/3854132675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc56c153c80 0x7fc56c0ce890 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:56.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.610+0000 7fc572e09640 1 --2- 172.21.15.57:0/3854132675 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc56c0c4a10 0x7fc56c14e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:56.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.610+0000 7fc572e09640 1 -- 172.21.15.57:0/3854132675 >> 172.21.15.57:0/3854132675 conn(0x7fc56c0715c0 msgr2=0x7fc56c158470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:56.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.610+0000 7fc572e09640 1 -- 172.21.15.57:0/3854132675 shutdown_connections 2023-10-30T14:21:56.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:56.610+0000 7fc572e09640 1 -- 172.21.15.57:0/3854132675 wait complete. 2023-10-30T14:21:56.631 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:21:58.632 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:21:58.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.786+0000 7fdbf5a09640 1 Processor -- start 2023-10-30T14:21:58.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.786+0000 7fdbf5a09640 1 -- start start 2023-10-30T14:21:58.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.786+0000 7fdbf5a09640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbf00c4a10 0x7fdbf015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:58.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.786+0000 7fdbf5a09640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fdbf00c42a0 0x7fdbf00ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:58.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.786+0000 7fdbf5a09640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf0159f20 con 0x7fdbf0153c80 2023-10-30T14:21:58.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.786+0000 7fdbf5a09640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbf015dec0 con 0x7fdbf00c42a0 2023-10-30T14:21:58.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.786+0000 7fdbf5a09640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf01599a0 con 0x7fdbf00c4a10 2023-10-30T14:21:58.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.786+0000 7fdbeeffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fdbf0153c80 0x7fdbf015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:52492/0 (socket says 172.21.15.57:52492) 2023-10-30T14:21:58.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.786+0000 7fdbeeffd640 1 -- 172.21.15.57:0/3211497574 learned_addr learned my addr 172.21.15.57:0/3211497574 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:21:58.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.786+0000 7fdbef7fe640 1 --2- 172.21.15.57:0/3211497574 >> v2:172.21.15.57:3300/0 conn(0x7fdbf00c42a0 0x7fdbf00ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:58.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.786+0000 7fdbe7fff640 1 --2- 172.21.15.57:0/3211497574 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbf00c4a10 0x7fdbf015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:58.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.790+0000 7fdbef7fe640 1 -- 172.21.15.57:0/3211497574 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbf00c4a10 msgr2=0x7fdbf015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:58.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.790+0000 7fdbef7fe640 1 --2- 172.21.15.57:0/3211497574 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbf00c4a10 0x7fdbf015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:58.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.790+0000 7fdbef7fe640 1 -- 172.21.15.57:0/3211497574 >> v1:172.21.15.57:6789/0 conn(0x7fdbf0153c80 legacy=0x7fdbf015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:58.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.790+0000 7fdbef7fe640 1 -- 172.21.15.57:0/3211497574 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf015d620 con 0x7fdbf00c42a0 2023-10-30T14:21:58.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.790+0000 7fdbee7fc640 1 -- 172.21.15.57:0/3211497574 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90963074 0 0) 0x7fdbf0159f20 con 0x7fdbf0153c80 2023-10-30T14:21:58.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.790+0000 7fdbef7fe640 1 --2- 172.21.15.57:0/3211497574 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbf00c42a0 0x7fdbf00ca1e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8009d90 tx=0x7fdbe801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0db2197418c057d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:58.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.790+0000 7fdbee7fc640 1 -- 172.21.15.57:0/3211497574 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe80af600 con 0x7fdbf00c42a0 2023-10-30T14:21:58.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.790+0000 7fdbee7fc640 1 -- 172.21.15.57:0/3211497574 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe80ab050 con 0x7fdbf00c42a0 2023-10-30T14:21:58.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.790+0000 7fdbee7fc640 1 -- 172.21.15.57:0/3211497574 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe80af980 con 0x7fdbf00c42a0 2023-10-30T14:21:58.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.790+0000 7fdbf5a09640 1 -- 172.21.15.57:0/3211497574 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbf00c42a0 msgr2=0x7fdbf00ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:58.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.790+0000 7fdbf5a09640 1 --2- 172.21.15.57:0/3211497574 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbf00c42a0 0x7fdbf00ca1e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8009d90 tx=0x7fdbe801f130 comp rx=0 tx=0).stop 2023-10-30T14:21:58.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.790+0000 7fdbf5a09640 1 -- 172.21.15.57:0/3211497574 shutdown_connections 2023-10-30T14:21:58.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.790+0000 7fdbf5a09640 1 --2- 172.21.15.57:0/3211497574 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbf00c4a10 0x7fdbf015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:58.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.790+0000 7fdbf5a09640 1 --2- 172.21.15.57:0/3211497574 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbf00c42a0 0x7fdbf00ca1e0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:58.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.790+0000 7fdbf5a09640 1 -- 172.21.15.57:0/3211497574 >> 172.21.15.57:0/3211497574 conn(0x7fdbf00715c0 msgr2=0x7fdbf00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:58.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.790+0000 7fdbf5a09640 1 -- 172.21.15.57:0/3211497574 shutdown_connections 2023-10-30T14:21:58.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.790+0000 7fdbf5a09640 1 -- 172.21.15.57:0/3211497574 wait complete. 2023-10-30T14:21:58.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbf5a09640 1 Processor -- start 2023-10-30T14:21:58.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbf5a09640 1 -- start start 2023-10-30T14:21:58.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbf5a09640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbf00c42a0 0x7fdbf00c5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:58.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbf5a09640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbf00c4a10 0x7fdbf014f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:58.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbf5a09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbf0153c80 0x7fdbf014fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:58.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbf5a09640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf0149870 con 0x7fdbf00c4a10 2023-10-30T14:21:58.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbf5a09640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf01f2c50 con 0x7fdbf0153c80 2023-10-30T14:21:58.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbf5a09640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf014ef70 con 0x7fdbf00c42a0 2023-10-30T14:21:58.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbe7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbf00c4a10 0x7fdbf014f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:58.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbe7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbf00c4a10 0x7fdbf014f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38672/0 (socket says 172.21.15.57:38672) 2023-10-30T14:21:58.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbe7fff640 1 -- 172.21.15.57:0/838225178 learned_addr learned my addr 172.21.15.57:0/838225178 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:21:58.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbeeffd640 1 --2- 172.21.15.57:0/838225178 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbf00c42a0 0x7fdbf00c5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:58.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbef7fe640 1 --2- 172.21.15.57:0/838225178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbf0153c80 0x7fdbf014fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:58.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbe7fff640 1 -- 172.21.15.57:0/838225178 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbf00c42a0 msgr2=0x7fdbf00c5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:58.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbe7fff640 1 --2- 172.21.15.57:0/838225178 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbf00c42a0 0x7fdbf00c5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:58.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbe7fff640 1 -- 172.21.15.57:0/838225178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbf0153c80 msgr2=0x7fdbf014fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:58.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbe7fff640 1 --2- 172.21.15.57:0/838225178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbf0153c80 0x7fdbf014fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:58.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbe7fff640 1 -- 172.21.15.57:0/838225178 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe8009a70 con 0x7fdbf00c4a10 2023-10-30T14:21:58.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbeeffd640 1 --2- 172.21.15.57:0/838225178 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbf00c42a0 0x7fdbf00c5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:21:58.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbe7fff640 1 --2- 172.21.15.57:0/838225178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbf00c4a10 0x7fdbf014f870 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fdbd800d7d0 tx=0x7fdbd800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:58.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbecff9640 1 -- 172.21.15.57:0/838225178 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd800b660 con 0x7fdbf00c4a10 2023-10-30T14:21:58.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbecff9640 1 -- 172.21.15.57:0/838225178 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd800b7c0 con 0x7fdbf00c4a10 2023-10-30T14:21:58.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbecff9640 1 -- 172.21.15.57:0/838225178 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd800bb70 con 0x7fdbf00c4a10 2023-10-30T14:21:58.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbf5a09640 1 -- 172.21.15.57:0/838225178 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf0157de0 con 0x7fdbf00c4a10 2023-10-30T14:21:58.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.794+0000 7fdbf5a09640 1 -- 172.21.15.57:0/838225178 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf01f2ef0 con 0x7fdbf00c4a10 2023-10-30T14:21:58.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.798+0000 7fdbf5a09640 1 -- 172.21.15.57:0/838225178 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf004e170 con 0x7fdbf00c4a10 2023-10-30T14:21:58.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.798+0000 7fdbecff9640 1 -- 172.21.15.57:0/838225178 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdbd800bcd0 con 0x7fdbf00c4a10 2023-10-30T14:21:58.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.798+0000 7fdbecff9640 1 --2- 172.21.15.57:0/838225178 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdbd00664a0 0x7fdbd0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:58.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.798+0000 7fdbecff9640 1 -- 172.21.15.57:0/838225178 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fdbd80844a0 con 0x7fdbf00c4a10 2023-10-30T14:21:58.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.798+0000 7fdbeeffd640 1 --2- 172.21.15.57:0/838225178 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdbd00664a0 0x7fdbd0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:58.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.802+0000 7fdbeeffd640 1 --2- 172.21.15.57:0/838225178 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdbd00664a0 0x7fdbd0068960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0000f70 tx=0x7fdbe0007cb0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:58.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:58.802+0000 7fdbecff9640 1 -- 172.21.15.57:0/838225178 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdbd8051660 con 0x7fdbf00c4a10 2023-10-30T14:21:59.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:59.030+0000 7fdbf5a09640 1 -- 172.21.15.57:0/838225178 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbf01f2ef0 con 0x7fdbf00c4a10 2023-10-30T14:21:59.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:59.030+0000 7fdbecff9640 1 -- 172.21.15.57:0/838225178 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6534 (secure 0 0 0) 0x7fdbf01f2ef0 con 0x7fdbf00c4a10 2023-10-30T14:21:59.034 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:21:59.035 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:21:55.990144+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:stopping","state_seq":28,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:21:59.035 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 59 2023-10-30T14:21:59.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:59.034+0000 7fdbf5a09640 1 -- 172.21.15.57:0/838225178 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdbd00664a0 msgr2=0x7fdbd0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:59.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:59.034+0000 7fdbf5a09640 1 --2- 172.21.15.57:0/838225178 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdbd00664a0 0x7fdbd0068960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0000f70 tx=0x7fdbe0007cb0 comp rx=0 tx=0).stop 2023-10-30T14:21:59.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:59.034+0000 7fdbf5a09640 1 -- 172.21.15.57:0/838225178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbf00c4a10 msgr2=0x7fdbf014f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:59.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:59.034+0000 7fdbf5a09640 1 --2- 172.21.15.57:0/838225178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbf00c4a10 0x7fdbf014f870 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fdbd800d7d0 tx=0x7fdbd800dc90 comp rx=0 tx=0).stop 2023-10-30T14:21:59.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:59.034+0000 7fdbf5a09640 1 -- 172.21.15.57:0/838225178 shutdown_connections 2023-10-30T14:21:59.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:59.034+0000 7fdbf5a09640 1 --2- 172.21.15.57:0/838225178 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdbd00664a0 0x7fdbd0068960 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:59.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:59.034+0000 7fdbf5a09640 1 --2- 172.21.15.57:0/838225178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbf0153c80 0x7fdbf014fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:59.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:59.034+0000 7fdbf5a09640 1 --2- 172.21.15.57:0/838225178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbf00c4a10 0x7fdbf014f870 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:59.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:59.034+0000 7fdbf5a09640 1 --2- 172.21.15.57:0/838225178 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbf00c42a0 0x7fdbf00c5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:59.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:59.034+0000 7fdbf5a09640 1 -- 172.21.15.57:0/838225178 >> 172.21.15.57:0/838225178 conn(0x7fdbf00715c0 msgr2=0x7fdbf015a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:59.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:59.034+0000 7fdbf5a09640 1 -- 172.21.15.57:0/838225178 shutdown_connections 2023-10-30T14:21:59.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:21:59.034+0000 7fdbf5a09640 1 -- 172.21.15.57:0/838225178 wait complete. 2023-10-30T14:21:59.060 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:22:00.441 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/35' 2023-10-30T14:22:00.441 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/26' 2023-10-30T14:22:00.441 INFO:tasks.workunit.client.0.smithi057.stdout:removed './266/37' 2023-10-30T14:22:00.441 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './266' 2023-10-30T14:22:00.442 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/21' 2023-10-30T14:22:00.442 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/27' 2023-10-30T14:22:00.442 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/31' 2023-10-30T14:22:00.442 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/36' 2023-10-30T14:22:00.443 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/20' 2023-10-30T14:22:00.443 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/39' 2023-10-30T14:22:00.443 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/34' 2023-10-30T14:22:00.443 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/38' 2023-10-30T14:22:00.443 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/28' 2023-10-30T14:22:00.444 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/24' 2023-10-30T14:22:00.444 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/29' 2023-10-30T14:22:00.444 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/32' 2023-10-30T14:22:00.444 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/22' 2023-10-30T14:22:00.444 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/25' 2023-10-30T14:22:00.444 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/30' 2023-10-30T14:22:00.445 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/33' 2023-10-30T14:22:00.445 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/23' 2023-10-30T14:22:00.445 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/35' 2023-10-30T14:22:00.445 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/26' 2023-10-30T14:22:00.445 INFO:tasks.workunit.client.0.smithi057.stdout:removed './267/37' 2023-10-30T14:22:00.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './267' 2023-10-30T14:22:00.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/21' 2023-10-30T14:22:00.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/27' 2023-10-30T14:22:00.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/31' 2023-10-30T14:22:00.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/36' 2023-10-30T14:22:00.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/20' 2023-10-30T14:22:00.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/39' 2023-10-30T14:22:00.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/34' 2023-10-30T14:22:00.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/38' 2023-10-30T14:22:00.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/28' 2023-10-30T14:22:00.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/24' 2023-10-30T14:22:00.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/29' 2023-10-30T14:22:00.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/32' 2023-10-30T14:22:00.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/22' 2023-10-30T14:22:00.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/25' 2023-10-30T14:22:00.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/30' 2023-10-30T14:22:00.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/33' 2023-10-30T14:22:00.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/23' 2023-10-30T14:22:00.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/35' 2023-10-30T14:22:00.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/26' 2023-10-30T14:22:00.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed './268/37' 2023-10-30T14:22:00.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './268' 2023-10-30T14:22:00.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/21' 2023-10-30T14:22:00.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/27' 2023-10-30T14:22:00.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/31' 2023-10-30T14:22:00.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/36' 2023-10-30T14:22:00.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/20' 2023-10-30T14:22:00.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/39' 2023-10-30T14:22:00.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/34' 2023-10-30T14:22:00.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/38' 2023-10-30T14:22:00.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/28' 2023-10-30T14:22:00.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/24' 2023-10-30T14:22:00.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/29' 2023-10-30T14:22:00.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/32' 2023-10-30T14:22:00.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/22' 2023-10-30T14:22:00.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/25' 2023-10-30T14:22:00.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/30' 2023-10-30T14:22:00.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/33' 2023-10-30T14:22:00.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/23' 2023-10-30T14:22:00.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/35' 2023-10-30T14:22:00.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/26' 2023-10-30T14:22:00.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed './269/37' 2023-10-30T14:22:00.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './269' 2023-10-30T14:22:00.454 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/21' 2023-10-30T14:22:00.454 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/27' 2023-10-30T14:22:00.454 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/31' 2023-10-30T14:22:00.454 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/36' 2023-10-30T14:22:00.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/20' 2023-10-30T14:22:00.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/39' 2023-10-30T14:22:00.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/34' 2023-10-30T14:22:00.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/38' 2023-10-30T14:22:00.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/28' 2023-10-30T14:22:00.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/24' 2023-10-30T14:22:00.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/29' 2023-10-30T14:22:00.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/32' 2023-10-30T14:22:00.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/22' 2023-10-30T14:22:00.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/25' 2023-10-30T14:22:00.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/30' 2023-10-30T14:22:00.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/33' 2023-10-30T14:22:00.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/23' 2023-10-30T14:22:00.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/35' 2023-10-30T14:22:00.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/26' 2023-10-30T14:22:00.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed './270/37' 2023-10-30T14:22:00.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './270' 2023-10-30T14:22:00.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/21' 2023-10-30T14:22:00.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/27' 2023-10-30T14:22:00.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/31' 2023-10-30T14:22:00.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/36' 2023-10-30T14:22:00.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/20' 2023-10-30T14:22:00.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/39' 2023-10-30T14:22:00.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/34' 2023-10-30T14:22:00.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/38' 2023-10-30T14:22:00.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/28' 2023-10-30T14:22:00.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/24' 2023-10-30T14:22:00.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/29' 2023-10-30T14:22:00.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/32' 2023-10-30T14:22:00.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/22' 2023-10-30T14:22:00.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/25' 2023-10-30T14:22:00.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/30' 2023-10-30T14:22:00.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/33' 2023-10-30T14:22:00.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/23' 2023-10-30T14:22:00.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/35' 2023-10-30T14:22:00.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/26' 2023-10-30T14:22:00.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed './271/37' 2023-10-30T14:22:00.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './271' 2023-10-30T14:22:00.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/21' 2023-10-30T14:22:00.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/27' 2023-10-30T14:22:00.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/31' 2023-10-30T14:22:00.463 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/36' 2023-10-30T14:22:00.463 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/20' 2023-10-30T14:22:00.463 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/39' 2023-10-30T14:22:00.463 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/34' 2023-10-30T14:22:00.463 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/38' 2023-10-30T14:22:00.463 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/28' 2023-10-30T14:22:00.464 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/24' 2023-10-30T14:22:00.464 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/29' 2023-10-30T14:22:00.464 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/32' 2023-10-30T14:22:00.464 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/22' 2023-10-30T14:22:00.464 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/25' 2023-10-30T14:22:00.465 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/30' 2023-10-30T14:22:00.465 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/33' 2023-10-30T14:22:00.465 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/23' 2023-10-30T14:22:00.465 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/35' 2023-10-30T14:22:00.465 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/26' 2023-10-30T14:22:00.465 INFO:tasks.workunit.client.0.smithi057.stdout:removed './272/37' 2023-10-30T14:22:00.466 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './272' 2023-10-30T14:22:00.466 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/21' 2023-10-30T14:22:00.466 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/27' 2023-10-30T14:22:00.466 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/31' 2023-10-30T14:22:00.466 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/36' 2023-10-30T14:22:00.467 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/20' 2023-10-30T14:22:00.467 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/39' 2023-10-30T14:22:00.467 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/34' 2023-10-30T14:22:00.467 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/38' 2023-10-30T14:22:00.467 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/28' 2023-10-30T14:22:00.467 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/24' 2023-10-30T14:22:00.468 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/29' 2023-10-30T14:22:00.468 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/32' 2023-10-30T14:22:00.468 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/22' 2023-10-30T14:22:00.468 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/25' 2023-10-30T14:22:00.468 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/30' 2023-10-30T14:22:00.469 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/33' 2023-10-30T14:22:00.469 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/23' 2023-10-30T14:22:00.469 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/35' 2023-10-30T14:22:00.469 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/26' 2023-10-30T14:22:00.469 INFO:tasks.workunit.client.0.smithi057.stdout:removed './273/37' 2023-10-30T14:22:00.470 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './273' 2023-10-30T14:22:00.470 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/21' 2023-10-30T14:22:00.470 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/27' 2023-10-30T14:22:00.470 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/31' 2023-10-30T14:22:00.470 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/36' 2023-10-30T14:22:00.470 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/20' 2023-10-30T14:22:00.471 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/39' 2023-10-30T14:22:00.471 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/34' 2023-10-30T14:22:00.471 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/38' 2023-10-30T14:22:00.471 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/28' 2023-10-30T14:22:00.471 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/24' 2023-10-30T14:22:00.471 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/29' 2023-10-30T14:22:00.472 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/32' 2023-10-30T14:22:00.472 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/22' 2023-10-30T14:22:00.472 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/25' 2023-10-30T14:22:00.472 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/30' 2023-10-30T14:22:00.472 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/33' 2023-10-30T14:22:00.473 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/23' 2023-10-30T14:22:00.473 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/35' 2023-10-30T14:22:00.473 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/26' 2023-10-30T14:22:00.473 INFO:tasks.workunit.client.0.smithi057.stdout:removed './274/37' 2023-10-30T14:22:00.473 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './274' 2023-10-30T14:22:00.473 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/21' 2023-10-30T14:22:00.474 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/27' 2023-10-30T14:22:00.474 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/31' 2023-10-30T14:22:00.474 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/36' 2023-10-30T14:22:00.474 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/20' 2023-10-30T14:22:00.474 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/39' 2023-10-30T14:22:00.475 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/34' 2023-10-30T14:22:00.475 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/38' 2023-10-30T14:22:00.475 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/28' 2023-10-30T14:22:00.475 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/24' 2023-10-30T14:22:00.475 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/29' 2023-10-30T14:22:00.475 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/32' 2023-10-30T14:22:00.476 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/22' 2023-10-30T14:22:00.476 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/25' 2023-10-30T14:22:00.476 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/30' 2023-10-30T14:22:00.476 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/33' 2023-10-30T14:22:00.476 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/23' 2023-10-30T14:22:00.476 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/35' 2023-10-30T14:22:00.477 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/26' 2023-10-30T14:22:00.477 INFO:tasks.workunit.client.0.smithi057.stdout:removed './275/37' 2023-10-30T14:22:00.477 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './275' 2023-10-30T14:22:00.477 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/21' 2023-10-30T14:22:00.477 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/27' 2023-10-30T14:22:00.478 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/31' 2023-10-30T14:22:00.478 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/36' 2023-10-30T14:22:00.478 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/20' 2023-10-30T14:22:00.478 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/39' 2023-10-30T14:22:00.478 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/34' 2023-10-30T14:22:00.478 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/38' 2023-10-30T14:22:00.479 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/28' 2023-10-30T14:22:00.479 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/24' 2023-10-30T14:22:00.479 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/29' 2023-10-30T14:22:00.479 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/32' 2023-10-30T14:22:00.479 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/22' 2023-10-30T14:22:00.480 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/25' 2023-10-30T14:22:00.480 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/30' 2023-10-30T14:22:00.480 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/33' 2023-10-30T14:22:00.480 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/23' 2023-10-30T14:22:00.480 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/35' 2023-10-30T14:22:00.480 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/26' 2023-10-30T14:22:01.052 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:22:01.056 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:22:01.061 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:01.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.290+0000 7f2943e9c640 1 Processor -- start 2023-10-30T14:22:01.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.290+0000 7f2943e9c640 1 -- start start 2023-10-30T14:22:01.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.294+0000 7f2943e9c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f293c0c5f80 0x7f293c0c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:01.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.294+0000 7f2943e9c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f293c0c9b40 0x7f293c1597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:01.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.294+0000 7f2943e9c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f293c15cda0 con 0x7f293c0c5ba0 2023-10-30T14:22:01.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.294+0000 7f2943e9c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f293c15d990 con 0x7f293c0c9b40 2023-10-30T14:22:01.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.294+0000 7f2943e9c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f293c153f60 con 0x7f293c0c5f80 2023-10-30T14:22:01.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.294+0000 7f2941c38640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f293c0c5ba0 0x7f293c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:52502/0 (socket says 172.21.15.57:52502) 2023-10-30T14:22:01.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.294+0000 7f2942439640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f293c0c9b40 0x7f293c1597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:01.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.294+0000 7f2941c38640 1 -- 172.21.15.57:0/493352773 learned_addr learned my addr 172.21.15.57:0/493352773 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:22:01.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.294+0000 7f2941437640 1 --2- 172.21.15.57:0/493352773 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f293c0c5f80 0x7f293c0c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:01.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.294+0000 7f2942439640 1 -- 172.21.15.57:0/493352773 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f293c0c5f80 msgr2=0x7f293c0c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:01.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.294+0000 7f2942439640 1 --2- 172.21.15.57:0/493352773 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f293c0c5f80 0x7f293c0c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:01.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.294+0000 7f2942439640 1 -- 172.21.15.57:0/493352773 >> v1:172.21.15.57:6789/0 conn(0x7f293c0c5ba0 legacy=0x7f293c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:01.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.294+0000 7f2942439640 1 -- 172.21.15.57:0/493352773 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f293c06f490 con 0x7f293c0c9b40 2023-10-30T14:22:01.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.294+0000 7f2940c36640 1 -- 172.21.15.57:0/493352773 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280120172 0 0) 0x7f293c15cda0 con 0x7f293c0c5ba0 2023-10-30T14:22:01.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.294+0000 7f2942439640 1 --2- 172.21.15.57:0/493352773 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f293c0c9b40 0x7f293c1597b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2934009dc0 tx=0x7f293401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4359591630a2b22 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:01.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.298+0000 7f2940c36640 1 -- 172.21.15.57:0/493352773 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29340a5760 con 0x7f293c0c9b40 2023-10-30T14:22:01.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.298+0000 7f2940c36640 1 -- 172.21.15.57:0/493352773 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29340a58c0 con 0x7f293c0c9b40 2023-10-30T14:22:01.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.298+0000 7f2940c36640 1 -- 172.21.15.57:0/493352773 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29340a5c50 con 0x7f293c0c9b40 2023-10-30T14:22:01.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.298+0000 7f2943e9c640 1 -- 172.21.15.57:0/493352773 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f293c0c9b40 msgr2=0x7f293c1597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:01.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.298+0000 7f2943e9c640 1 --2- 172.21.15.57:0/493352773 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f293c0c9b40 0x7f293c1597b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2934009dc0 tx=0x7f293401c8a0 comp rx=0 tx=0).stop 2023-10-30T14:22:01.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.298+0000 7f2943e9c640 1 -- 172.21.15.57:0/493352773 shutdown_connections 2023-10-30T14:22:01.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.298+0000 7f2943e9c640 1 --2- 172.21.15.57:0/493352773 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f293c0c5f80 0x7f293c0c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:01.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.298+0000 7f2943e9c640 1 --2- 172.21.15.57:0/493352773 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f293c0c9b40 0x7f293c1597b0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:01.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.298+0000 7f2943e9c640 1 -- 172.21.15.57:0/493352773 >> 172.21.15.57:0/493352773 conn(0x7f293c06afa0 msgr2=0x7f293c06b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:01.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.298+0000 7f2943e9c640 1 -- 172.21.15.57:0/493352773 shutdown_connections 2023-10-30T14:22:01.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.298+0000 7f2943e9c640 1 -- 172.21.15.57:0/493352773 wait complete. 2023-10-30T14:22:01.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.298+0000 7f2943e9c640 1 Processor -- start 2023-10-30T14:22:01.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.298+0000 7f2943e9c640 1 -- start start 2023-10-30T14:22:01.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.298+0000 7f2943e9c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f293c0c5ba0 0x7f293c156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:01.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.298+0000 7f2943e9c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f293c0c5f80 0x7f293c155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:01.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.298+0000 7f2943e9c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f293c0c9b40 0x7f293c0c3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:01.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.298+0000 7f2943e9c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c157790 con 0x7f293c0c9b40 2023-10-30T14:22:01.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.298+0000 7f2943e9c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c157440 con 0x7f293c0c5f80 2023-10-30T14:22:01.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.298+0000 7f2943e9c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f293c0c8200 con 0x7f293c0c5ba0 2023-10-30T14:22:01.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.302+0000 7f2942439640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f293c0c9b40 0x7f293c0c3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:01.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.302+0000 7f2942439640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f293c0c9b40 0x7f293c0c3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38700/0 (socket says 172.21.15.57:38700) 2023-10-30T14:22:01.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.302+0000 7f2941c38640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f293c0c5ba0 0x7f293c156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:01.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.302+0000 7f2942439640 1 -- 172.21.15.57:0/3574268049 learned_addr learned my addr 172.21.15.57:0/3574268049 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:01.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.302+0000 7f2941437640 1 --2- 172.21.15.57:0/3574268049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f293c0c5f80 0x7f293c155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:01.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.302+0000 7f2942439640 1 -- 172.21.15.57:0/3574268049 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f293c0c5ba0 msgr2=0x7f293c156620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:01.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.302+0000 7f2942439640 1 --2- 172.21.15.57:0/3574268049 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f293c0c5ba0 0x7f293c156620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:01.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.302+0000 7f2942439640 1 -- 172.21.15.57:0/3574268049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f293c0c5f80 msgr2=0x7f293c155740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:01.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.302+0000 7f2942439640 1 --2- 172.21.15.57:0/3574268049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f293c0c5f80 0x7f293c155740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:01.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.302+0000 7f2942439640 1 -- 172.21.15.57:0/3574268049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2934009a70 con 0x7f293c0c9b40 2023-10-30T14:22:01.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.302+0000 7f2942439640 1 --2- 172.21.15.57:0/3574268049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f293c0c9b40 0x7f293c0c3a90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f293401cd80 tx=0x7f2934019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:01.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.302+0000 7f29327fc640 1 -- 172.21.15.57:0/3574268049 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29340ad640 con 0x7f293c0c9b40 2023-10-30T14:22:01.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.302+0000 7f29327fc640 1 -- 172.21.15.57:0/3574268049 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29340ad7a0 con 0x7f293c0c9b40 2023-10-30T14:22:01.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.302+0000 7f29327fc640 1 -- 172.21.15.57:0/3574268049 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29340adb10 con 0x7f293c0c9b40 2023-10-30T14:22:01.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.302+0000 7f2943e9c640 1 -- 172.21.15.57:0/3574268049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f293c159360 con 0x7f293c0c9b40 2023-10-30T14:22:01.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.302+0000 7f2943e9c640 1 -- 172.21.15.57:0/3574268049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f293c159920 con 0x7f293c0c9b40 2023-10-30T14:22:01.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.302+0000 7f2943e9c640 1 -- 172.21.15.57:0/3574268049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f293c1578d0 con 0x7f293c0c9b40 2023-10-30T14:22:01.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.302+0000 7f29327fc640 1 -- 172.21.15.57:0/3574268049 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f29340b4070 con 0x7f293c0c9b40 2023-10-30T14:22:01.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.306+0000 7f29327fc640 1 --2- 172.21.15.57:0/3574268049 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29200664a0 0x7f2920068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:01.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.306+0000 7f29327fc640 1 -- 172.21.15.57:0/3574268049 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f293411e8e0 con 0x7f293c0c9b40 2023-10-30T14:22:01.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.306+0000 7f2941c38640 1 --2- 172.21.15.57:0/3574268049 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29200664a0 0x7f2920068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:01.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.306+0000 7f2941c38640 1 --2- 172.21.15.57:0/3574268049 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29200664a0 0x7f2920068960 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2938000f70 tx=0x7f2938007df0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:01.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.310+0000 7f29327fc640 1 -- 172.21.15.57:0/3574268049 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f293c1578d0 con 0x7f293c0c9b40 2023-10-30T14:22:01.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.514+0000 7f2943e9c640 1 -- 172.21.15.57:0/3574268049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f293c1578d0 con 0x7f293c0c9b40 2023-10-30T14:22:01.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.514+0000 7f29327fc640 1 -- 172.21.15.57:0/3574268049 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6534 (secure 0 0 0) 0x7f293c1578d0 con 0x7f293c0c9b40 2023-10-30T14:22:01.517 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:22:01.517 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:21:55.990144+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:stopping","state_seq":28,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:01.518 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 59 2023-10-30T14:22:01.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.518+0000 7f2943e9c640 1 -- 172.21.15.57:0/3574268049 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29200664a0 msgr2=0x7f2920068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:01.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.518+0000 7f2943e9c640 1 --2- 172.21.15.57:0/3574268049 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29200664a0 0x7f2920068960 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2938000f70 tx=0x7f2938007df0 comp rx=0 tx=0).stop 2023-10-30T14:22:01.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.518+0000 7f2943e9c640 1 -- 172.21.15.57:0/3574268049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f293c0c9b40 msgr2=0x7f293c0c3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:01.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.518+0000 7f2943e9c640 1 --2- 172.21.15.57:0/3574268049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f293c0c9b40 0x7f293c0c3a90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f293401cd80 tx=0x7f2934019640 comp rx=0 tx=0).stop 2023-10-30T14:22:01.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.518+0000 7f2943e9c640 1 -- 172.21.15.57:0/3574268049 shutdown_connections 2023-10-30T14:22:01.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.518+0000 7f2943e9c640 1 --2- 172.21.15.57:0/3574268049 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29200664a0 0x7f2920068960 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:01.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.518+0000 7f2943e9c640 1 --2- 172.21.15.57:0/3574268049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f293c0c9b40 0x7f293c0c3a90 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:01.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.518+0000 7f2943e9c640 1 --2- 172.21.15.57:0/3574268049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f293c0c5f80 0x7f293c155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:01.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.518+0000 7f2943e9c640 1 --2- 172.21.15.57:0/3574268049 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f293c0c5ba0 0x7f293c156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:01.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.518+0000 7f2943e9c640 1 -- 172.21.15.57:0/3574268049 >> 172.21.15.57:0/3574268049 conn(0x7f293c06afa0 msgr2=0x7f293c071c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:01.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.518+0000 7f2943e9c640 1 -- 172.21.15.57:0/3574268049 shutdown_connections 2023-10-30T14:22:01.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:01.518+0000 7f2943e9c640 1 -- 172.21.15.57:0/3574268049 wait complete. 2023-10-30T14:22:01.538 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:22:02.791 INFO:tasks.workunit.client.0.smithi057.stdout:removed './276/37' 2023-10-30T14:22:02.791 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './276' 2023-10-30T14:22:02.791 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/21' 2023-10-30T14:22:02.791 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/27' 2023-10-30T14:22:02.792 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/31' 2023-10-30T14:22:02.792 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/36' 2023-10-30T14:22:02.792 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/20' 2023-10-30T14:22:02.792 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/39' 2023-10-30T14:22:02.793 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/34' 2023-10-30T14:22:02.793 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/38' 2023-10-30T14:22:02.793 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/28' 2023-10-30T14:22:02.793 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/24' 2023-10-30T14:22:02.794 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/29' 2023-10-30T14:22:02.794 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/32' 2023-10-30T14:22:02.794 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/22' 2023-10-30T14:22:02.794 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/25' 2023-10-30T14:22:02.795 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/30' 2023-10-30T14:22:02.795 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/33' 2023-10-30T14:22:02.795 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/23' 2023-10-30T14:22:02.795 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/35' 2023-10-30T14:22:02.795 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/26' 2023-10-30T14:22:02.796 INFO:tasks.workunit.client.0.smithi057.stdout:removed './277/37' 2023-10-30T14:22:02.796 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './277' 2023-10-30T14:22:02.796 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/21' 2023-10-30T14:22:02.796 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/27' 2023-10-30T14:22:02.796 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/31' 2023-10-30T14:22:02.796 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/36' 2023-10-30T14:22:02.797 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/20' 2023-10-30T14:22:02.797 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/39' 2023-10-30T14:22:02.797 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/34' 2023-10-30T14:22:02.797 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/38' 2023-10-30T14:22:02.797 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/28' 2023-10-30T14:22:02.798 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/24' 2023-10-30T14:22:02.798 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/29' 2023-10-30T14:22:02.798 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/32' 2023-10-30T14:22:02.798 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/22' 2023-10-30T14:22:02.798 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/25' 2023-10-30T14:22:02.799 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/30' 2023-10-30T14:22:02.799 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/33' 2023-10-30T14:22:02.799 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/23' 2023-10-30T14:22:02.799 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/35' 2023-10-30T14:22:02.799 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/26' 2023-10-30T14:22:02.799 INFO:tasks.workunit.client.0.smithi057.stdout:removed './278/37' 2023-10-30T14:22:02.800 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './278' 2023-10-30T14:22:02.800 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/21' 2023-10-30T14:22:02.800 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/27' 2023-10-30T14:22:02.800 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/31' 2023-10-30T14:22:02.800 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/36' 2023-10-30T14:22:02.800 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/20' 2023-10-30T14:22:02.801 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/39' 2023-10-30T14:22:02.801 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/34' 2023-10-30T14:22:02.801 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/38' 2023-10-30T14:22:02.801 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/28' 2023-10-30T14:22:02.801 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/24' 2023-10-30T14:22:02.801 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/29' 2023-10-30T14:22:02.802 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/32' 2023-10-30T14:22:02.802 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/22' 2023-10-30T14:22:02.802 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/25' 2023-10-30T14:22:02.802 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/30' 2023-10-30T14:22:02.802 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/33' 2023-10-30T14:22:02.803 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/23' 2023-10-30T14:22:02.803 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/35' 2023-10-30T14:22:02.803 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/26' 2023-10-30T14:22:02.803 INFO:tasks.workunit.client.0.smithi057.stdout:removed './279/37' 2023-10-30T14:22:02.803 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './279' 2023-10-30T14:22:02.803 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/21' 2023-10-30T14:22:02.804 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/27' 2023-10-30T14:22:02.804 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/31' 2023-10-30T14:22:02.804 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/36' 2023-10-30T14:22:02.804 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/20' 2023-10-30T14:22:02.804 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/39' 2023-10-30T14:22:02.804 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/34' 2023-10-30T14:22:02.805 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/38' 2023-10-30T14:22:02.805 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/28' 2023-10-30T14:22:02.805 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/24' 2023-10-30T14:22:02.806 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/29' 2023-10-30T14:22:02.806 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/32' 2023-10-30T14:22:02.806 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/22' 2023-10-30T14:22:02.806 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/25' 2023-10-30T14:22:02.806 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/30' 2023-10-30T14:22:02.806 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/33' 2023-10-30T14:22:02.807 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/23' 2023-10-30T14:22:02.807 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/35' 2023-10-30T14:22:02.807 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/26' 2023-10-30T14:22:02.807 INFO:tasks.workunit.client.0.smithi057.stdout:removed './280/37' 2023-10-30T14:22:02.807 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './280' 2023-10-30T14:22:02.808 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/21' 2023-10-30T14:22:02.808 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/27' 2023-10-30T14:22:02.808 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/31' 2023-10-30T14:22:02.808 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/36' 2023-10-30T14:22:02.808 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/20' 2023-10-30T14:22:02.808 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/39' 2023-10-30T14:22:02.809 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/34' 2023-10-30T14:22:02.809 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/38' 2023-10-30T14:22:02.809 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/28' 2023-10-30T14:22:02.809 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/24' 2023-10-30T14:22:02.809 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/29' 2023-10-30T14:22:02.809 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/32' 2023-10-30T14:22:02.810 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/22' 2023-10-30T14:22:02.810 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/25' 2023-10-30T14:22:02.810 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/30' 2023-10-30T14:22:02.810 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/33' 2023-10-30T14:22:02.810 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/23' 2023-10-30T14:22:02.810 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/35' 2023-10-30T14:22:02.811 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/26' 2023-10-30T14:22:02.811 INFO:tasks.workunit.client.0.smithi057.stdout:removed './281/37' 2023-10-30T14:22:02.811 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './281' 2023-10-30T14:22:02.811 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/21' 2023-10-30T14:22:02.811 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/27' 2023-10-30T14:22:02.812 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/31' 2023-10-30T14:22:02.812 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/36' 2023-10-30T14:22:02.812 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/20' 2023-10-30T14:22:02.812 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/39' 2023-10-30T14:22:02.812 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/34' 2023-10-30T14:22:02.812 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/38' 2023-10-30T14:22:02.813 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/28' 2023-10-30T14:22:02.813 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/24' 2023-10-30T14:22:02.813 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/29' 2023-10-30T14:22:02.813 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/32' 2023-10-30T14:22:02.813 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/22' 2023-10-30T14:22:02.814 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/25' 2023-10-30T14:22:02.814 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/30' 2023-10-30T14:22:02.814 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/33' 2023-10-30T14:22:02.814 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/23' 2023-10-30T14:22:02.814 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/35' 2023-10-30T14:22:02.815 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/26' 2023-10-30T14:22:02.815 INFO:tasks.workunit.client.0.smithi057.stdout:removed './282/37' 2023-10-30T14:22:02.815 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './282' 2023-10-30T14:22:02.815 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/21' 2023-10-30T14:22:02.815 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/27' 2023-10-30T14:22:02.815 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/31' 2023-10-30T14:22:02.816 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/36' 2023-10-30T14:22:02.816 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/20' 2023-10-30T14:22:02.816 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/39' 2023-10-30T14:22:02.816 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/34' 2023-10-30T14:22:02.817 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/38' 2023-10-30T14:22:02.817 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/28' 2023-10-30T14:22:02.817 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/24' 2023-10-30T14:22:02.817 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/29' 2023-10-30T14:22:02.817 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/32' 2023-10-30T14:22:02.818 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/22' 2023-10-30T14:22:02.818 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/25' 2023-10-30T14:22:02.818 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/30' 2023-10-30T14:22:02.818 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/33' 2023-10-30T14:22:02.818 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/23' 2023-10-30T14:22:02.819 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/35' 2023-10-30T14:22:02.819 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/26' 2023-10-30T14:22:02.819 INFO:tasks.workunit.client.0.smithi057.stdout:removed './283/37' 2023-10-30T14:22:02.819 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './283' 2023-10-30T14:22:02.820 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/21' 2023-10-30T14:22:02.820 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/27' 2023-10-30T14:22:02.820 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/31' 2023-10-30T14:22:02.820 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/36' 2023-10-30T14:22:02.820 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/20' 2023-10-30T14:22:02.821 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/39' 2023-10-30T14:22:02.821 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/34' 2023-10-30T14:22:02.821 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/38' 2023-10-30T14:22:02.821 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/28' 2023-10-30T14:22:02.821 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/24' 2023-10-30T14:22:02.822 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/29' 2023-10-30T14:22:02.822 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/32' 2023-10-30T14:22:02.822 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/22' 2023-10-30T14:22:02.822 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/25' 2023-10-30T14:22:02.823 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/30' 2023-10-30T14:22:02.823 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/33' 2023-10-30T14:22:02.823 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/23' 2023-10-30T14:22:02.823 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/35' 2023-10-30T14:22:02.823 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/26' 2023-10-30T14:22:02.823 INFO:tasks.workunit.client.0.smithi057.stdout:removed './284/37' 2023-10-30T14:22:02.824 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './284' 2023-10-30T14:22:02.824 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/21' 2023-10-30T14:22:02.824 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/27' 2023-10-30T14:22:02.824 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/31' 2023-10-30T14:22:02.824 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/36' 2023-10-30T14:22:02.825 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/20' 2023-10-30T14:22:02.825 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/39' 2023-10-30T14:22:02.825 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/34' 2023-10-30T14:22:02.825 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/38' 2023-10-30T14:22:02.825 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/28' 2023-10-30T14:22:02.826 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/24' 2023-10-30T14:22:02.826 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/29' 2023-10-30T14:22:02.826 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/32' 2023-10-30T14:22:02.826 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/22' 2023-10-30T14:22:02.827 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/25' 2023-10-30T14:22:02.827 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/30' 2023-10-30T14:22:02.827 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/33' 2023-10-30T14:22:02.827 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/23' 2023-10-30T14:22:02.827 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/35' 2023-10-30T14:22:02.828 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/26' 2023-10-30T14:22:02.828 INFO:tasks.workunit.client.0.smithi057.stdout:removed './285/37' 2023-10-30T14:22:02.828 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './285' 2023-10-30T14:22:02.828 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/21' 2023-10-30T14:22:02.829 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/27' 2023-10-30T14:22:02.829 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/31' 2023-10-30T14:22:02.829 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/36' 2023-10-30T14:22:02.829 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/20' 2023-10-30T14:22:02.830 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/39' 2023-10-30T14:22:02.830 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/34' 2023-10-30T14:22:02.830 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/38' 2023-10-30T14:22:02.830 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/28' 2023-10-30T14:22:02.830 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/24' 2023-10-30T14:22:02.830 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/29' 2023-10-30T14:22:02.831 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/32' 2023-10-30T14:22:02.831 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/22' 2023-10-30T14:22:02.831 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/25' 2023-10-30T14:22:02.831 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/30' 2023-10-30T14:22:02.831 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/33' 2023-10-30T14:22:02.832 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/23' 2023-10-30T14:22:02.832 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/35' 2023-10-30T14:22:02.832 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/26' 2023-10-30T14:22:02.832 INFO:tasks.workunit.client.0.smithi057.stdout:removed './286/37' 2023-10-30T14:22:02.832 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './286' 2023-10-30T14:22:03.539 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:03.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.682+0000 7f95bd66b640 1 Processor -- start 2023-10-30T14:22:03.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.682+0000 7f95bd66b640 1 -- start start 2023-10-30T14:22:03.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.686+0000 7f95bd66b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f95b8153c80 0x7f95b815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:03.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.686+0000 7f95bd66b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f95b80c4a10 0x7f95b80c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:03.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.686+0000 7f95bd66b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95b815a3b0 0x7f95b80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:03.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.686+0000 7f95bd66b640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95b815dec0 con 0x7f95b8153c80 2023-10-30T14:22:03.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.686+0000 7f95bd66b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95b81599a0 con 0x7f95b80c4a10 2023-10-30T14:22:03.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.686+0000 7f95bd66b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95b8159f20 con 0x7f95b815a3b0 2023-10-30T14:22:03.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.686+0000 7f95b6ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f95b8153c80 0x7f95b815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:03.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.686+0000 7f95b6ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f95b8153c80 0x7f95b815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38714/0 (socket says 172.21.15.57:38714) 2023-10-30T14:22:03.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.686+0000 7f95b6ffd640 1 -- 172.21.15.57:0/3065438532 learned_addr learned my addr 172.21.15.57:0/3065438532 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:03.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.686+0000 7f95b77fe640 1 --2- 172.21.15.57:0/3065438532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95b815a3b0 0x7f95b80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:03.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.686+0000 7f95affff640 1 --2- 172.21.15.57:0/3065438532 >> v2:172.21.15.88:3300/0 conn(0x7f95b80c4a10 0x7f95b80c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:03.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.686+0000 7f95b6ffd640 1 -- 172.21.15.57:0/3065438532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95b815a3b0 msgr2=0x7f95b80ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:03.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.686+0000 7f95b6ffd640 1 --2- 172.21.15.57:0/3065438532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95b815a3b0 0x7f95b80ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:03.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.686+0000 7f95b6ffd640 1 -- 172.21.15.57:0/3065438532 >> v2:172.21.15.88:3300/0 conn(0x7f95b80c4a10 msgr2=0x7f95b80c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:03.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.686+0000 7f95b6ffd640 1 --2- 172.21.15.57:0/3065438532 >> v2:172.21.15.88:3300/0 conn(0x7f95b80c4a10 0x7f95b80c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:03.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.686+0000 7f95b6ffd640 1 -- 172.21.15.57:0/3065438532 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b80c47e0 con 0x7f95b8153c80 2023-10-30T14:22:03.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.690+0000 7f95b6ffd640 1 --2- 172.21.15.57:0/3065438532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95b8153c80 0x7f95b815cf90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f95b806be50 tx=0x7f95a801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b02c352f8c017021 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:03.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.690+0000 7f95b67fc640 1 -- 172.21.15.57:0/3065438532 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a80a89b0 con 0x7f95b8153c80 2023-10-30T14:22:03.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.690+0000 7f95b67fc640 1 -- 172.21.15.57:0/3065438532 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a80a8b10 con 0x7f95b8153c80 2023-10-30T14:22:03.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.690+0000 7f95b67fc640 1 -- 172.21.15.57:0/3065438532 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a80a8ea0 con 0x7f95b8153c80 2023-10-30T14:22:03.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.690+0000 7f95bd66b640 1 -- 172.21.15.57:0/3065438532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95b8153c80 msgr2=0x7f95b815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:03.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.690+0000 7f95bd66b640 1 --2- 172.21.15.57:0/3065438532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95b8153c80 0x7f95b815cf90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f95b806be50 tx=0x7f95a801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:22:03.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.690+0000 7f95bd66b640 1 -- 172.21.15.57:0/3065438532 shutdown_connections 2023-10-30T14:22:03.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.690+0000 7f95bd66b640 1 --2- 172.21.15.57:0/3065438532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95b815a3b0 0x7f95b80ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:03.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.690+0000 7f95bd66b640 1 --2- 172.21.15.57:0/3065438532 >> v2:172.21.15.88:3300/0 conn(0x7f95b80c4a10 0x7f95b80c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:03.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.690+0000 7f95bd66b640 1 --2- 172.21.15.57:0/3065438532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95b8153c80 0x7f95b815cf90 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:03.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.690+0000 7f95bd66b640 1 -- 172.21.15.57:0/3065438532 >> 172.21.15.57:0/3065438532 conn(0x7f95b80715c0 msgr2=0x7f95b80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:03.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.690+0000 7f95bd66b640 1 -- 172.21.15.57:0/3065438532 shutdown_connections 2023-10-30T14:22:03.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.690+0000 7f95bd66b640 1 -- 172.21.15.57:0/3065438532 wait complete. 2023-10-30T14:22:03.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.690+0000 7f95bd66b640 1 Processor -- start 2023-10-30T14:22:03.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.690+0000 7f95bd66b640 1 -- start start 2023-10-30T14:22:03.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.690+0000 7f95bd66b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95b80c4a10 0x7f95b8156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:03.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.690+0000 7f95bd66b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95b8153c80 0x7f95b80ce280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:03.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95bd66b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95b815a3b0 0x7f95b80ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:03.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95bd66b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b80cfe30 con 0x7f95b8153c80 2023-10-30T14:22:03.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95bd66b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b8150d40 con 0x7f95b80c4a10 2023-10-30T14:22:03.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95bd66b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95b80cefd0 con 0x7f95b815a3b0 2023-10-30T14:22:03.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95affff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95b8153c80 0x7f95b80ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:03.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95affff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95b8153c80 0x7f95b80ce280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38724/0 (socket says 172.21.15.57:38724) 2023-10-30T14:22:03.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95affff640 1 -- 172.21.15.57:0/2950886726 learned_addr learned my addr 172.21.15.57:0/2950886726 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:03.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95b6ffd640 1 --2- 172.21.15.57:0/2950886726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95b80c4a10 0x7f95b8156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:03.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95b77fe640 1 --2- 172.21.15.57:0/2950886726 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95b815a3b0 0x7f95b80ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:03.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95affff640 1 -- 172.21.15.57:0/2950886726 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95b815a3b0 msgr2=0x7f95b80ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:03.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95affff640 1 --2- 172.21.15.57:0/2950886726 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95b815a3b0 0x7f95b80ce7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:03.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95affff640 1 -- 172.21.15.57:0/2950886726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95b80c4a10 msgr2=0x7f95b8156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:03.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95affff640 1 --2- 172.21.15.57:0/2950886726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95b80c4a10 0x7f95b8156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:03.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95affff640 1 -- 172.21.15.57:0/2950886726 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a80094d0 con 0x7f95b8153c80 2023-10-30T14:22:03.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95b77fe640 1 --2- 172.21.15.57:0/2950886726 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95b815a3b0 0x7f95b80ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:22:03.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95affff640 1 --2- 172.21.15.57:0/2950886726 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95b8153c80 0x7f95b80ce280 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f95a000d7d0 tx=0x7f95a000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:03.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95af7fe640 1 -- 172.21.15.57:0/2950886726 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a00096a0 con 0x7f95b8153c80 2023-10-30T14:22:03.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95af7fe640 1 -- 172.21.15.57:0/2950886726 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a0009800 con 0x7f95b8153c80 2023-10-30T14:22:03.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95af7fe640 1 -- 172.21.15.57:0/2950886726 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a0009bb0 con 0x7f95b8153c80 2023-10-30T14:22:03.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95bd66b640 1 -- 172.21.15.57:0/2950886726 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95b80cf2b0 con 0x7f95b8153c80 2023-10-30T14:22:03.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95bd66b640 1 -- 172.21.15.57:0/2950886726 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95b80cf870 con 0x7f95b8153c80 2023-10-30T14:22:03.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95bd66b640 1 -- 172.21.15.57:0/2950886726 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95b804e170 con 0x7f95b8153c80 2023-10-30T14:22:03.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.694+0000 7f95af7fe640 1 -- 172.21.15.57:0/2950886726 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f95a0009d10 con 0x7f95b8153c80 2023-10-30T14:22:03.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.698+0000 7f95af7fe640 1 --2- 172.21.15.57:0/2950886726 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95980664a0 0x7f9598068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:03.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.698+0000 7f95af7fe640 1 -- 172.21.15.57:0/2950886726 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f95a0084670 con 0x7f95b8153c80 2023-10-30T14:22:03.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.698+0000 7f95b6ffd640 1 --2- 172.21.15.57:0/2950886726 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95980664a0 0x7f9598068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:03.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.698+0000 7f95b6ffd640 1 --2- 172.21.15.57:0/2950886726 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95980664a0 0x7f9598068960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f95a80b2000 tx=0x7f95a8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:03.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.698+0000 7f95af7fe640 1 -- 172.21.15.57:0/2950886726 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f95a00517a0 con 0x7f95b8153c80 2023-10-30T14:22:03.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.894+0000 7f95bd66b640 1 -- 172.21.15.57:0/2950886726 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95b80b36d0 con 0x7f95b8153c80 2023-10-30T14:22:03.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.894+0000 7f95af7fe640 1 -- 172.21.15.57:0/2950886726 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6534 (secure 0 0 0) 0x7f95a0054dc0 con 0x7f95b8153c80 2023-10-30T14:22:03.897 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:22:03.897 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:21:55.990144+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:stopping","state_seq":28,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:03.898 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 59 2023-10-30T14:22:03.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.898+0000 7f95bd66b640 1 -- 172.21.15.57:0/2950886726 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95980664a0 msgr2=0x7f9598068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:03.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.898+0000 7f95bd66b640 1 --2- 172.21.15.57:0/2950886726 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95980664a0 0x7f9598068960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f95a80b2000 tx=0x7f95a8009040 comp rx=0 tx=0).stop 2023-10-30T14:22:03.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.898+0000 7f95bd66b640 1 -- 172.21.15.57:0/2950886726 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95b8153c80 msgr2=0x7f95b80ce280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:03.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.898+0000 7f95bd66b640 1 --2- 172.21.15.57:0/2950886726 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95b8153c80 0x7f95b80ce280 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f95a000d7d0 tx=0x7f95a000dc90 comp rx=0 tx=0).stop 2023-10-30T14:22:03.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.898+0000 7f95bd66b640 1 -- 172.21.15.57:0/2950886726 shutdown_connections 2023-10-30T14:22:03.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.898+0000 7f95bd66b640 1 --2- 172.21.15.57:0/2950886726 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95980664a0 0x7f9598068960 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:03.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.898+0000 7f95bd66b640 1 --2- 172.21.15.57:0/2950886726 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95b815a3b0 0x7f95b80ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:03.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.898+0000 7f95bd66b640 1 --2- 172.21.15.57:0/2950886726 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95b8153c80 0x7f95b80ce280 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:03.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.898+0000 7f95bd66b640 1 --2- 172.21.15.57:0/2950886726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95b80c4a10 0x7f95b8156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:03.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.898+0000 7f95bd66b640 1 -- 172.21.15.57:0/2950886726 >> 172.21.15.57:0/2950886726 conn(0x7f95b80715c0 msgr2=0x7f95b81513e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:03.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.898+0000 7f95bd66b640 1 -- 172.21.15.57:0/2950886726 shutdown_connections 2023-10-30T14:22:03.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:03.898+0000 7f95bd66b640 1 -- 172.21.15.57:0/2950886726 wait complete. 2023-10-30T14:22:03.918 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:22:05.919 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:06.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.074+0000 7fb832c46640 1 Processor -- start 2023-10-30T14:22:06.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.074+0000 7fb832c46640 1 -- start start 2023-10-30T14:22:06.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb832c46640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb82c1537f0 0x7fb82c0701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:06.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb832c46640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c0c2aa0 con 0x7fb82c06b2f0 2023-10-30T14:22:06.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb832c46640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c0708e0 con 0x7fb82c15a2e0 2023-10-30T14:22:06.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb832c46640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb82c0c3ab0 con 0x7fb82c1537f0 2023-10-30T14:22:06.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb8311e3640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb82c06b2f0 0x7fb82c06b700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:38436/0 (socket says 172.21.15.57:38436) 2023-10-30T14:22:06.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb8311e3640 1 -- 172.21.15.57:0/1462630189 learned_addr learned my addr 172.21.15.57:0/1462630189 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:22:06.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb8309e2640 1 --2- 172.21.15.57:0/1462630189 >> v2:172.21.15.88:3300/0 conn(0x7fb82c1537f0 0x7fb82c0701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:06.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb82b7fe640 1 -- 172.21.15.57:0/1462630189 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579623526 0 0) 0x7fb82c0c2aa0 con 0x7fb82c06b2f0 2023-10-30T14:22:06.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb82b7fe640 1 -- 172.21.15.57:0/1462630189 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb81c003530 con 0x7fb82c06b2f0 2023-10-30T14:22:06.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb82b7fe640 1 -- 172.21.15.57:0/1462630189 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3999430514 0 0) 0x7fb81c003530 con 0x7fb82c06b2f0 2023-10-30T14:22:06.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb82b7fe640 1 -- 172.21.15.57:0/1462630189 >> v2:172.21.15.88:3300/0 conn(0x7fb82c1537f0 msgr2=0x7fb82c0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:06.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb82b7fe640 1 --2- 172.21.15.57:0/1462630189 >> v2:172.21.15.88:3300/0 conn(0x7fb82c1537f0 0x7fb82c0701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:06.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb82b7fe640 1 -- 172.21.15.57:0/1462630189 >> v1:172.21.15.88:6789/0 conn(0x7fb82c15a2e0 legacy=0x7fb82c0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:06.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb82b7fe640 1 -- 172.21.15.57:0/1462630189 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c0c1eb0 con 0x7fb82c06b2f0 2023-10-30T14:22:06.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb82b7fe640 1 -- 172.21.15.57:0/1462630189 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fb824001a00 con 0x7fb82c06b2f0 2023-10-30T14:22:06.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb82b7fe640 1 -- 172.21.15.57:0/1462630189 >> v1:172.21.15.57:6789/0 conn(0x7fb82c06b2f0 legacy=0x7fb82c06b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:06.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb82b7fe640 1 --2- 172.21.15.57:0/1462630189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb81c003980 0x7fb81c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:06.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb82b7fe640 1 --2- 172.21.15.57:0/1462630189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb81c0047d0 0x7fb81c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:06.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb82b7fe640 1 --2- 172.21.15.57:0/1462630189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb81c005190 0x7fb81c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:06.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb82b7fe640 1 -- 172.21.15.57:0/1462630189 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c00c010 con 0x7fb81c003980 2023-10-30T14:22:06.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb82b7fe640 1 -- 172.21.15.57:0/1462630189 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c00c180 con 0x7fb81c005190 2023-10-30T14:22:06.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb82b7fe640 1 -- 172.21.15.57:0/1462630189 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c00c340 con 0x7fb81c0047d0 2023-10-30T14:22:06.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb8309e2640 1 --2- 172.21.15.57:0/1462630189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb81c003980 0x7fb81c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:06.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb82bfff640 1 --2- 172.21.15.57:0/1462630189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb81c0047d0 0x7fb81c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:06.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb8311e3640 1 --2- 172.21.15.57:0/1462630189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb81c005190 0x7fb81c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:06.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb8309e2640 1 -- 172.21.15.57:0/1462630189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb81c0047d0 msgr2=0x7fb81c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:06.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb8309e2640 1 --2- 172.21.15.57:0/1462630189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb81c0047d0 0x7fb81c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:06.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb8309e2640 1 -- 172.21.15.57:0/1462630189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb81c005190 msgr2=0x7fb81c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:06.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb8309e2640 1 --2- 172.21.15.57:0/1462630189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb81c005190 0x7fb81c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:06.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb8309e2640 1 -- 172.21.15.57:0/1462630189 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81c003530 con 0x7fb81c003980 2023-10-30T14:22:06.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.078+0000 7fb8311e3640 1 -- 172.21.15.57:0/1462630189 reap_dead start 2023-10-30T14:22:06.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.082+0000 7fb8309e2640 1 --2- 172.21.15.57:0/1462630189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb81c003980 0x7fb81c003da0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fb820009ae0 tx=0x7fb82001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36fc4c8a8019bb3c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:06.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.082+0000 7fb82b7fe640 1 -- 172.21.15.57:0/1462630189 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8200a5530 con 0x7fb81c003980 2023-10-30T14:22:06.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.082+0000 7fb82b7fe640 1 -- 172.21.15.57:0/1462630189 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8200a5690 con 0x7fb81c003980 2023-10-30T14:22:06.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.082+0000 7fb82b7fe640 1 -- 172.21.15.57:0/1462630189 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8200a5a40 con 0x7fb81c003980 2023-10-30T14:22:06.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.082+0000 7fb832c46640 1 -- 172.21.15.57:0/1462630189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb81c003980 msgr2=0x7fb81c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:06.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.082+0000 7fb832c46640 1 --2- 172.21.15.57:0/1462630189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb81c003980 0x7fb81c003da0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fb820009ae0 tx=0x7fb82001ca50 comp rx=0 tx=0).stop 2023-10-30T14:22:06.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.082+0000 7fb832c46640 1 -- 172.21.15.57:0/1462630189 shutdown_connections 2023-10-30T14:22:06.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.082+0000 7fb832c46640 1 --2- 172.21.15.57:0/1462630189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb81c003980 0x7fb81c003da0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:06.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.082+0000 7fb832c46640 1 -- 172.21.15.57:0/1462630189 >> 172.21.15.57:0/1462630189 conn(0x7fb82c071fb0 msgr2=0x7fb82c0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:06.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.082+0000 7fb832c46640 1 -- 172.21.15.57:0/1462630189 shutdown_connections 2023-10-30T14:22:06.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.082+0000 7fb832c46640 1 -- 172.21.15.57:0/1462630189 wait complete. 2023-10-30T14:22:06.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb832c46640 1 Processor -- start 2023-10-30T14:22:06.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb832c46640 1 -- start start 2023-10-30T14:22:06.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb832c46640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb81c003980 0x7fb82c158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:06.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb832c46640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb82c0c9df0 0x7fb82c0c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:06.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb832c46640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb82c0ca3f0 0x7fb82c1458f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:06.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb832c46640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c1ef4a0 con 0x7fb82c0ca3f0 2023-10-30T14:22:06.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb832c46640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c0c9100 con 0x7fb82c0c9df0 2023-10-30T14:22:06.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb832c46640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c14c870 con 0x7fb81c003980 2023-10-30T14:22:06.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb8311e3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb82c0ca3f0 0x7fb82c1458f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:06.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb8311e3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb82c0ca3f0 0x7fb82c1458f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39162/0 (socket says 172.21.15.57:39162) 2023-10-30T14:22:06.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb8311e3640 1 -- 172.21.15.57:0/979085099 learned_addr learned my addr 172.21.15.57:0/979085099 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:06.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb8309e2640 1 --2- 172.21.15.57:0/979085099 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb81c003980 0x7fb82c158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:06.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb82bfff640 1 --2- 172.21.15.57:0/979085099 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb82c0c9df0 0x7fb82c0c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:06.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb8311e3640 1 -- 172.21.15.57:0/979085099 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb81c003980 msgr2=0x7fb82c158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:06.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb8311e3640 1 --2- 172.21.15.57:0/979085099 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb81c003980 0x7fb82c158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:06.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb8311e3640 1 -- 172.21.15.57:0/979085099 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb82c0c9df0 msgr2=0x7fb82c0c9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:06.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb8311e3640 1 --2- 172.21.15.57:0/979085099 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb82c0c9df0 0x7fb82c0c9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:06.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb8311e3640 1 -- 172.21.15.57:0/979085099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c1ef1c0 con 0x7fb82c0ca3f0 2023-10-30T14:22:06.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb8311e3640 1 --2- 172.21.15.57:0/979085099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb82c0ca3f0 0x7fb82c1458f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0caae0 tx=0x7fb824011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:06.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb8297fa640 1 -- 172.21.15.57:0/979085099 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82400f920 con 0x7fb82c0ca3f0 2023-10-30T14:22:06.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb8297fa640 1 -- 172.21.15.57:0/979085099 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82400faa0 con 0x7fb82c0ca3f0 2023-10-30T14:22:06.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb832c46640 1 -- 172.21.15.57:0/979085099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb82c0c87d0 con 0x7fb82c0ca3f0 2023-10-30T14:22:06.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb8297fa640 1 -- 172.21.15.57:0/979085099 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82400fe50 con 0x7fb82c0ca3f0 2023-10-30T14:22:06.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb832c46640 1 -- 172.21.15.57:0/979085099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb82c0c8d90 con 0x7fb82c0ca3f0 2023-10-30T14:22:06.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.086+0000 7fb832c46640 1 -- 172.21.15.57:0/979085099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb82c0c9270 con 0x7fb82c0ca3f0 2023-10-30T14:22:06.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.090+0000 7fb8297fa640 1 -- 172.21.15.57:0/979085099 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb82401f460 con 0x7fb82c0ca3f0 2023-10-30T14:22:06.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.090+0000 7fb8297fa640 1 --2- 172.21.15.57:0/979085099 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8100664a0 0x7fb810068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:06.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.090+0000 7fb8297fa640 1 -- 172.21.15.57:0/979085099 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fb824087100 con 0x7fb82c0ca3f0 2023-10-30T14:22:06.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.090+0000 7fb8309e2640 1 --2- 172.21.15.57:0/979085099 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8100664a0 0x7fb810068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:06.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.090+0000 7fb8309e2640 1 --2- 172.21.15.57:0/979085099 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8100664a0 0x7fb810068960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb8200af000 tx=0x7fb8200046d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:06.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.094+0000 7fb8297fa640 1 -- 172.21.15.57:0/979085099 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb8240542c0 con 0x7fb82c0ca3f0 2023-10-30T14:22:06.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.338+0000 7fb832c46640 1 -- 172.21.15.57:0/979085099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb82c04e170 con 0x7fb82c0ca3f0 2023-10-30T14:22:06.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.342+0000 7fb8297fa640 1 -- 172.21.15.57:0/979085099 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6534 (secure 0 0 0) 0x7fb8240578e0 con 0x7fb82c0ca3f0 2023-10-30T14:22:06.343 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:22:06.343 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:21:55.990144+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:stopping","state_seq":28,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:06.345 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 59 2023-10-30T14:22:06.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.342+0000 7fb832c46640 1 -- 172.21.15.57:0/979085099 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8100664a0 msgr2=0x7fb810068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:06.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.342+0000 7fb832c46640 1 --2- 172.21.15.57:0/979085099 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8100664a0 0x7fb810068960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb8200af000 tx=0x7fb8200046d0 comp rx=0 tx=0).stop 2023-10-30T14:22:06.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.342+0000 7fb832c46640 1 -- 172.21.15.57:0/979085099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb82c0ca3f0 msgr2=0x7fb82c1458f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:06.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.342+0000 7fb832c46640 1 --2- 172.21.15.57:0/979085099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb82c0ca3f0 0x7fb82c1458f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0caae0 tx=0x7fb824011c20 comp rx=0 tx=0).stop 2023-10-30T14:22:06.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.342+0000 7fb832c46640 1 -- 172.21.15.57:0/979085099 shutdown_connections 2023-10-30T14:22:06.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.342+0000 7fb832c46640 1 --2- 172.21.15.57:0/979085099 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8100664a0 0x7fb810068960 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:06.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.342+0000 7fb832c46640 1 --2- 172.21.15.57:0/979085099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb82c0ca3f0 0x7fb82c1458f0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:06.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.342+0000 7fb832c46640 1 --2- 172.21.15.57:0/979085099 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb82c0c9df0 0x7fb82c0c9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:06.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.342+0000 7fb832c46640 1 --2- 172.21.15.57:0/979085099 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb81c003980 0x7fb82c158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:06.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.342+0000 7fb832c46640 1 -- 172.21.15.57:0/979085099 >> 172.21.15.57:0/979085099 conn(0x7fb82c071fb0 msgr2=0x7fb82c150810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:06.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.342+0000 7fb832c46640 1 -- 172.21.15.57:0/979085099 shutdown_connections 2023-10-30T14:22:06.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:06.342+0000 7fb832c46640 1 -- 172.21.15.57:0/979085099 wait complete. 2023-10-30T14:22:06.365 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T14:22:06.365 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 59, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 59, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5078": { "addr": "172.21.15.88:6839/1677924344", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1677924344, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1677924344, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5078, "incarnation": 48, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:stopping", "state_seq": 28 }, "gid_5133": { "addr": "172.21.15.57:6834/2912050965", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2912050965, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2912050965, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5133, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 15 }, "gid_5289": { "addr": "172.21.15.57:6836/2427081813", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2427081813, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2427081813, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5289, "incarnation": 35, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:21:55.990144+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5289, "mds_1": 5133, "mds_2": 5078 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.57:6837/862088802", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 862088802, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 862088802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 54, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5586, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:22:08.367 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:08.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.514+0000 7f0448e09640 1 Processor -- start 2023-10-30T14:22:08.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.514+0000 7f0448e09640 1 -- start start 2023-10-30T14:22:08.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.514+0000 7f0448e09640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0444153c80 0x7f044415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:08.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.514+0000 7f0448e09640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f04440c4a10 0x7f04440c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:08.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.514+0000 7f0448e09640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0444159f20 con 0x7f044415a3b0 2023-10-30T14:22:08.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.514+0000 7f0448e09640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044415dec0 con 0x7f0444153c80 2023-10-30T14:22:08.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.514+0000 7f0448e09640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04441599a0 con 0x7f04440c4a10 2023-10-30T14:22:08.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.514+0000 7f043bfff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f04440c4a10 0x7f04440c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:08.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.514+0000 7f043bfff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f04440c4a10 0x7f04440c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:33234/0 (socket says 172.21.15.57:33234) 2023-10-30T14:22:08.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.514+0000 7f044259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0444153c80 0x7f044415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:08.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.514+0000 7f043bfff640 1 -- 172.21.15.57:0/1151647064 learned_addr learned my addr 172.21.15.57:0/1151647064 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:08.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.518+0000 7f043bfff640 1 -- 172.21.15.57:0/1151647064 >> v2:172.21.15.88:3300/0 conn(0x7f0444153c80 msgr2=0x7f044415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:08.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.518+0000 7f043bfff640 1 --2- 172.21.15.57:0/1151647064 >> v2:172.21.15.88:3300/0 conn(0x7f0444153c80 0x7f044415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:08.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.518+0000 7f043bfff640 1 -- 172.21.15.57:0/1151647064 >> v1:172.21.15.88:6789/0 conn(0x7f044415a3b0 legacy=0x7f04440ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:08.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.518+0000 7f043bfff640 1 -- 172.21.15.57:0/1151647064 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04440c47e0 con 0x7f04440c4a10 2023-10-30T14:22:08.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.518+0000 7f044259c640 1 --2- 172.21.15.57:0/1151647064 >> v2:172.21.15.88:3300/0 conn(0x7f0444153c80 0x7f044415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:22:08.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.518+0000 7f043bfff640 1 --2- 172.21.15.57:0/1151647064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f04440c4a10 0x7f04440c42a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f042c009870 tx=0x7f042c01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebb4311857397e7e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:08.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.518+0000 7f0441d9b640 1 -- 172.21.15.57:0/1151647064 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c0a8a30 con 0x7f04440c4a10 2023-10-30T14:22:08.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.518+0000 7f0441d9b640 1 -- 172.21.15.57:0/1151647064 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042c0a8b90 con 0x7f04440c4a10 2023-10-30T14:22:08.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.518+0000 7f0441d9b640 1 -- 172.21.15.57:0/1151647064 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c0b0440 con 0x7f04440c4a10 2023-10-30T14:22:08.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.518+0000 7f0448e09640 1 -- 172.21.15.57:0/1151647064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f04440c4a10 msgr2=0x7f04440c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:08.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.518+0000 7f0448e09640 1 --2- 172.21.15.57:0/1151647064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f04440c4a10 0x7f04440c42a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f042c009870 tx=0x7f042c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:22:08.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f0448e09640 1 -- 172.21.15.57:0/1151647064 shutdown_connections 2023-10-30T14:22:08.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f0448e09640 1 --2- 172.21.15.57:0/1151647064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f04440c4a10 0x7f04440c42a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:08.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f0448e09640 1 --2- 172.21.15.57:0/1151647064 >> v2:172.21.15.88:3300/0 conn(0x7f0444153c80 0x7f044415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:08.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f0448e09640 1 -- 172.21.15.57:0/1151647064 >> 172.21.15.57:0/1151647064 conn(0x7f04440715c0 msgr2=0x7f04440719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:08.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f0448e09640 1 -- 172.21.15.57:0/1151647064 shutdown_connections 2023-10-30T14:22:08.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f0448e09640 1 -- 172.21.15.57:0/1151647064 wait complete. 2023-10-30T14:22:08.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f0448e09640 1 Processor -- start 2023-10-30T14:22:08.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f0448e09640 1 -- start start 2023-10-30T14:22:08.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f0448e09640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04440c4a10 0x7f044414bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:08.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f0448e09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0444153c80 0x7f044414a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:08.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f0448e09640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f044415a3b0 0x7f0444158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:08.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f0448e09640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044414c1c0 con 0x7f04440c4a10 2023-10-30T14:22:08.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f0448e09640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04441f30f0 con 0x7f0444153c80 2023-10-30T14:22:08.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f0448e09640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04441f2290 con 0x7f044415a3b0 2023-10-30T14:22:08.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f044259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04440c4a10 0x7f044414bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:08.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f044259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04440c4a10 0x7f044414bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39174/0 (socket says 172.21.15.57:39174) 2023-10-30T14:22:08.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f044259c640 1 -- 172.21.15.57:0/3252587802 learned_addr learned my addr 172.21.15.57:0/3252587802 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:08.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f0442d9d640 1 --2- 172.21.15.57:0/3252587802 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f044415a3b0 0x7f0444158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:08.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f043bfff640 1 --2- 172.21.15.57:0/3252587802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0444153c80 0x7f044414a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:08.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f044259c640 1 -- 172.21.15.57:0/3252587802 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f044415a3b0 msgr2=0x7f0444158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:08.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f044259c640 1 --2- 172.21.15.57:0/3252587802 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f044415a3b0 0x7f0444158850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:08.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f044259c640 1 -- 172.21.15.57:0/3252587802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0444153c80 msgr2=0x7f044414a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:08.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f044259c640 1 --2- 172.21.15.57:0/3252587802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0444153c80 0x7f044414a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:08.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f044259c640 1 -- 172.21.15.57:0/3252587802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0434009420 con 0x7f04440c4a10 2023-10-30T14:22:08.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.522+0000 7f044259c640 1 --2- 172.21.15.57:0/3252587802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04440c4a10 0x7f044414bc80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f043400ebe0 tx=0x7f043400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:08.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.526+0000 7f043affd640 1 -- 172.21.15.57:0/3252587802 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043400eee0 con 0x7f04440c4a10 2023-10-30T14:22:08.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.526+0000 7f043affd640 1 -- 172.21.15.57:0/3252587802 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043400cc30 con 0x7f04440c4a10 2023-10-30T14:22:08.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.526+0000 7f0448e09640 1 -- 172.21.15.57:0/3252587802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f042c0094d0 con 0x7f04440c4a10 2023-10-30T14:22:08.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.526+0000 7f043affd640 1 -- 172.21.15.57:0/3252587802 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0434013470 con 0x7f04440c4a10 2023-10-30T14:22:08.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.526+0000 7f0448e09640 1 -- 172.21.15.57:0/3252587802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04441f2850 con 0x7f04440c4a10 2023-10-30T14:22:08.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.526+0000 7f0448e09640 1 -- 172.21.15.57:0/3252587802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044404e170 con 0x7f04440c4a10 2023-10-30T14:22:08.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.526+0000 7f043affd640 1 -- 172.21.15.57:0/3252587802 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f04340135d0 con 0x7f04440c4a10 2023-10-30T14:22:08.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.526+0000 7f043affd640 1 --2- 172.21.15.57:0/3252587802 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f04240664a0 0x7f0424068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:08.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.526+0000 7f043affd640 1 -- 172.21.15.57:0/3252587802 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f0434084410 con 0x7f04440c4a10 2023-10-30T14:22:08.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.526+0000 7f043bfff640 1 --2- 172.21.15.57:0/3252587802 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f04240664a0 0x7f0424068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:08.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.530+0000 7f043bfff640 1 --2- 172.21.15.57:0/3252587802 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f04240664a0 0x7f0424068960 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f042c0b2000 tx=0x7f042c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:08.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.530+0000 7f043affd640 1 -- 172.21.15.57:0/3252587802 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f04340515e0 con 0x7f04440c4a10 2023-10-30T14:22:08.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.718+0000 7f0448e09640 1 -- 172.21.15.57:0/3252587802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0444157550 con 0x7f04440c4a10 2023-10-30T14:22:08.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.722+0000 7f043affd640 1 -- 172.21.15.57:0/3252587802 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6534 (secure 0 0 0) 0x7f0434054c00 con 0x7f04440c4a10 2023-10-30T14:22:08.723 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:22:08.723 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:21:55.990144+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:stopping","state_seq":28,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:08.723 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 59 2023-10-30T14:22:08.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.722+0000 7f0448e09640 1 -- 172.21.15.57:0/3252587802 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f04240664a0 msgr2=0x7f0424068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:08.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.722+0000 7f0448e09640 1 --2- 172.21.15.57:0/3252587802 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f04240664a0 0x7f0424068960 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f042c0b2000 tx=0x7f042c008040 comp rx=0 tx=0).stop 2023-10-30T14:22:08.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.722+0000 7f0448e09640 1 -- 172.21.15.57:0/3252587802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04440c4a10 msgr2=0x7f044414bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:08.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.722+0000 7f0448e09640 1 --2- 172.21.15.57:0/3252587802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04440c4a10 0x7f044414bc80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f043400ebe0 tx=0x7f043400c520 comp rx=0 tx=0).stop 2023-10-30T14:22:08.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.722+0000 7f0448e09640 1 -- 172.21.15.57:0/3252587802 shutdown_connections 2023-10-30T14:22:08.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.722+0000 7f0448e09640 1 --2- 172.21.15.57:0/3252587802 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f04240664a0 0x7f0424068960 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:08.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.722+0000 7f0448e09640 1 --2- 172.21.15.57:0/3252587802 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f044415a3b0 0x7f0444158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:08.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.722+0000 7f0448e09640 1 --2- 172.21.15.57:0/3252587802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0444153c80 0x7f044414a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:08.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.722+0000 7f0448e09640 1 --2- 172.21.15.57:0/3252587802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04440c4a10 0x7f044414bc80 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:08.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.722+0000 7f0448e09640 1 -- 172.21.15.57:0/3252587802 >> 172.21.15.57:0/3252587802 conn(0x7f04440715c0 msgr2=0x7f04440719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:08.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.722+0000 7f0448e09640 1 -- 172.21.15.57:0/3252587802 shutdown_connections 2023-10-30T14:22:08.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:08.722+0000 7f0448e09640 1 -- 172.21.15.57:0/3252587802 wait complete. 2023-10-30T14:22:08.743 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:22:10.744 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:10.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.898+0000 7f82f599d640 1 Processor -- start 2023-10-30T14:22:10.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.898+0000 7f82f599d640 1 -- start start 2023-10-30T14:22:10.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.898+0000 7f82f599d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f82f015d840 0x7f82f00ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:10.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.898+0000 7f82f599d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f00c7b00 con 0x7f82f00cd420 2023-10-30T14:22:10.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.898+0000 7f82f599d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f00c2050 con 0x7f82f0153aa0 2023-10-30T14:22:10.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.898+0000 7f82f599d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f0070a50 con 0x7f82f015d840 2023-10-30T14:22:10.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.898+0000 7f82e7fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f82f015d840 0x7f82f00ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:10.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.898+0000 7f82ef7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f82f00cd420 0x7f82f006b3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:38440/0 (socket says 172.21.15.57:38440) 2023-10-30T14:22:10.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.898+0000 7f82e7fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f82f015d840 0x7f82f00ccee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39184/0 (socket says 172.21.15.57:39184) 2023-10-30T14:22:10.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.898+0000 7f82ef7fe640 1 -- 172.21.15.57:0/3707576446 learned_addr learned my addr 172.21.15.57:0/3707576446 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:22:10.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.898+0000 7f82e7fff640 1 -- 172.21.15.57:0/3707576446 >> v1:172.21.15.88:6789/0 conn(0x7f82f0153aa0 legacy=0x7f82f00701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:10.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.898+0000 7f82e7fff640 1 -- 172.21.15.57:0/3707576446 >> v1:172.21.15.57:6789/0 conn(0x7f82f00cd420 legacy=0x7f82f006b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:10.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.898+0000 7f82e7fff640 1 -- 172.21.15.57:0/3707576446 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f00c2330 con 0x7f82f015d840 2023-10-30T14:22:10.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.902+0000 7f82e7fff640 1 --2- 172.21.15.57:0/3707576446 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f015d840 0x7f82f00ccee0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f82d80097c0 tx=0x7f82d801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65de6ab942e93536 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:10.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.902+0000 7f82ee7fc640 1 -- 172.21.15.57:0/3707576446 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d80a8970 con 0x7f82f015d840 2023-10-30T14:22:10.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.902+0000 7f82ee7fc640 1 -- 172.21.15.57:0/3707576446 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d80a8ad0 con 0x7f82f015d840 2023-10-30T14:22:10.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.902+0000 7f82ee7fc640 1 -- 172.21.15.57:0/3707576446 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d80a8e60 con 0x7f82f015d840 2023-10-30T14:22:10.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.902+0000 7f82f599d640 1 -- 172.21.15.57:0/3707576446 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f015d840 msgr2=0x7f82f00ccee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:10.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.902+0000 7f82f599d640 1 --2- 172.21.15.57:0/3707576446 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f015d840 0x7f82f00ccee0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f82d80097c0 tx=0x7f82d801ec70 comp rx=0 tx=0).stop 2023-10-30T14:22:10.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.902+0000 7f82f599d640 1 -- 172.21.15.57:0/3707576446 shutdown_connections 2023-10-30T14:22:10.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.902+0000 7f82f599d640 1 --2- 172.21.15.57:0/3707576446 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f015d840 0x7f82f00ccee0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:10.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.902+0000 7f82f599d640 1 -- 172.21.15.57:0/3707576446 >> 172.21.15.57:0/3707576446 conn(0x7f82f0071fb0 msgr2=0x7f82f00723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:10.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.902+0000 7f82f599d640 1 -- 172.21.15.57:0/3707576446 shutdown_connections 2023-10-30T14:22:10.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.902+0000 7f82f599d640 1 -- 172.21.15.57:0/3707576446 wait complete. 2023-10-30T14:22:10.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.906+0000 7f82f599d640 1 Processor -- start 2023-10-30T14:22:10.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.906+0000 7f82f599d640 1 -- start start 2023-10-30T14:22:10.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.906+0000 7f82f599d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82f00cd420 0x7f82f0158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:10.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.906+0000 7f82f599d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f0153aa0 0x7f82f015c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:10.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.906+0000 7f82f599d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f015d840 0x7f82f01f2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:10.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.906+0000 7f82f599d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f01ef680 con 0x7f82f0153aa0 2023-10-30T14:22:10.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.906+0000 7f82f599d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f01eea90 con 0x7f82f015d840 2023-10-30T14:22:10.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.906+0000 7f82f599d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f01587f0 con 0x7f82f00cd420 2023-10-30T14:22:10.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.906+0000 7f82e7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f0153aa0 0x7f82f015c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:10.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.906+0000 7f82e7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f0153aa0 0x7f82f015c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39190/0 (socket says 172.21.15.57:39190) 2023-10-30T14:22:10.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.906+0000 7f82e7fff640 1 -- 172.21.15.57:0/3813351677 learned_addr learned my addr 172.21.15.57:0/3813351677 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:10.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.906+0000 7f82eeffd640 1 --2- 172.21.15.57:0/3813351677 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82f00cd420 0x7f82f0158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:10.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.906+0000 7f82ef7fe640 1 --2- 172.21.15.57:0/3813351677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f015d840 0x7f82f01f2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:10.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.906+0000 7f82e7fff640 1 -- 172.21.15.57:0/3813351677 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82f00cd420 msgr2=0x7f82f0158160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:10.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.906+0000 7f82e7fff640 1 --2- 172.21.15.57:0/3813351677 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82f00cd420 0x7f82f0158160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:10.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.906+0000 7f82e7fff640 1 -- 172.21.15.57:0/3813351677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f015d840 msgr2=0x7f82f01f2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:10.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.906+0000 7f82e7fff640 1 --2- 172.21.15.57:0/3813351677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f015d840 0x7f82f01f2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:10.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.906+0000 7f82e7fff640 1 -- 172.21.15.57:0/3813351677 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d8009420 con 0x7f82f0153aa0 2023-10-30T14:22:10.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.906+0000 7f82e7fff640 1 --2- 172.21.15.57:0/3813351677 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f0153aa0 0x7f82f015c3d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f82d80b3040 tx=0x7f82d801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:10.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.910+0000 7f82ecff9640 1 -- 172.21.15.57:0/3813351677 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d80b9630 con 0x7f82f0153aa0 2023-10-30T14:22:10.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.910+0000 7f82ecff9640 1 -- 172.21.15.57:0/3813351677 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d80b9790 con 0x7f82f0153aa0 2023-10-30T14:22:10.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.910+0000 7f82ecff9640 1 -- 172.21.15.57:0/3813351677 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d80b9b40 con 0x7f82f0153aa0 2023-10-30T14:22:10.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.910+0000 7f82f599d640 1 -- 172.21.15.57:0/3813351677 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f01eed50 con 0x7f82f0153aa0 2023-10-30T14:22:10.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.910+0000 7f82f599d640 1 -- 172.21.15.57:0/3813351677 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f01ef310 con 0x7f82f0153aa0 2023-10-30T14:22:10.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.910+0000 7f82f599d640 1 -- 172.21.15.57:0/3813351677 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f004e170 con 0x7f82f0153aa0 2023-10-30T14:22:10.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.910+0000 7f82ecff9640 1 -- 172.21.15.57:0/3813351677 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f82d80b7040 con 0x7f82f0153aa0 2023-10-30T14:22:10.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.910+0000 7f82ecff9640 1 --2- 172.21.15.57:0/3813351677 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f82d00664a0 0x7f82d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:10.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.910+0000 7f82ecff9640 1 -- 172.21.15.57:0/3813351677 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f82d8122110 con 0x7f82f0153aa0 2023-10-30T14:22:10.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.910+0000 7f82eeffd640 1 --2- 172.21.15.57:0/3813351677 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f82d00664a0 0x7f82d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:10.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.914+0000 7f82eeffd640 1 --2- 172.21.15.57:0/3813351677 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f82d00664a0 0x7f82d0068960 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f82e0007fd0 tx=0x7f82e0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:10.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:10.914+0000 7f82ecff9640 1 -- 172.21.15.57:0/3813351677 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f82d80ef360 con 0x7f82f0153aa0 2023-10-30T14:22:11.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:11.110+0000 7f82f599d640 1 -- 172.21.15.57:0/3813351677 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82f00c25e0 con 0x7f82f0153aa0 2023-10-30T14:22:11.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:11.114+0000 7f82ecff9640 1 -- 172.21.15.57:0/3813351677 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6534 (secure 0 0 0) 0x7f82d80b5020 con 0x7f82f0153aa0 2023-10-30T14:22:11.115 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:22:11.115 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:21:55.990144+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:stopping","state_seq":28,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:11.115 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 59 2023-10-30T14:22:11.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:11.114+0000 7f82f599d640 1 -- 172.21.15.57:0/3813351677 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f82d00664a0 msgr2=0x7f82d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:11.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:11.114+0000 7f82f599d640 1 --2- 172.21.15.57:0/3813351677 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f82d00664a0 0x7f82d0068960 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f82e0007fd0 tx=0x7f82e0009040 comp rx=0 tx=0).stop 2023-10-30T14:22:11.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:11.114+0000 7f82f599d640 1 -- 172.21.15.57:0/3813351677 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f0153aa0 msgr2=0x7f82f015c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:11.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:11.114+0000 7f82f599d640 1 --2- 172.21.15.57:0/3813351677 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f0153aa0 0x7f82f015c3d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f82d80b3040 tx=0x7f82d801ba10 comp rx=0 tx=0).stop 2023-10-30T14:22:11.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:11.114+0000 7f82f599d640 1 -- 172.21.15.57:0/3813351677 shutdown_connections 2023-10-30T14:22:11.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:11.114+0000 7f82f599d640 1 --2- 172.21.15.57:0/3813351677 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f82d00664a0 0x7f82d0068960 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:11.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:11.114+0000 7f82f599d640 1 --2- 172.21.15.57:0/3813351677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f015d840 0x7f82f01f2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:11.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:11.114+0000 7f82f599d640 1 --2- 172.21.15.57:0/3813351677 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f0153aa0 0x7f82f015c3d0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:11.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:11.114+0000 7f82f599d640 1 --2- 172.21.15.57:0/3813351677 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82f00cd420 0x7f82f0158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:11.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:11.114+0000 7f82f599d640 1 -- 172.21.15.57:0/3813351677 >> 172.21.15.57:0/3813351677 conn(0x7f82f0071fb0 msgr2=0x7f82f0151b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:11.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:11.114+0000 7f82f599d640 1 -- 172.21.15.57:0/3813351677 shutdown_connections 2023-10-30T14:22:11.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:11.114+0000 7f82f599d640 1 -- 172.21.15.57:0/3813351677 wait complete. 2023-10-30T14:22:11.135 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:22:13.137 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:13.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.290+0000 7f329cd0f640 1 Processor -- start 2023-10-30T14:22:13.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.290+0000 7f329cd0f640 1 -- start start 2023-10-30T14:22:13.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.290+0000 7f329cd0f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3298151840 0x7f32980c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:13.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.290+0000 7f329cd0f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f32980cb8b0 0x7f329806b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:13.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.290+0000 7f329cd0f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32980c5030 con 0x7f3298151c20 2023-10-30T14:22:13.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.290+0000 7f329cd0f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3298157830 con 0x7f32980cb8b0 2023-10-30T14:22:13.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.290+0000 7f329cd0f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3298157b10 con 0x7f3298151840 2023-10-30T14:22:13.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.290+0000 7f3296d9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f32980cb8b0 0x7f329806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:13.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.290+0000 7f3296d9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f32980cb8b0 0x7f329806b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39198/0 (socket says 172.21.15.57:39198) 2023-10-30T14:22:13.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.290+0000 7f3296d9d640 1 -- 172.21.15.57:0/1241522920 learned_addr learned my addr 172.21.15.57:0/1241522920 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:13.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.290+0000 7f329659c640 1 --2- 172.21.15.57:0/1241522920 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3298151840 0x7f32980c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:13.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.294+0000 7f3296d9d640 1 -- 172.21.15.57:0/1241522920 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3298151840 msgr2=0x7f32980c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:13.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.294+0000 7f3296d9d640 1 --2- 172.21.15.57:0/1241522920 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3298151840 0x7f32980c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:13.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.294+0000 7f3296d9d640 1 -- 172.21.15.57:0/1241522920 >> v1:172.21.15.57:6789/0 conn(0x7f3298151c20 legacy=0x7f3298072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:13.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.294+0000 7f3296d9d640 1 -- 172.21.15.57:0/1241522920 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3298157df0 con 0x7f32980cb8b0 2023-10-30T14:22:13.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.294+0000 7f3295d9b640 1 -- 172.21.15.57:0/1241522920 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584262675 0 0) 0x7f32980c5030 con 0x7f3298151c20 2023-10-30T14:22:13.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.294+0000 7f3296d9d640 1 --2- 172.21.15.57:0/1241522920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32980cb8b0 0x7f329806b600 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3290009e50 tx=0x7f329001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=621c83ce73b2306d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:13.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.294+0000 7f3295d9b640 1 -- 172.21.15.57:0/1241522920 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32900af400 con 0x7f32980cb8b0 2023-10-30T14:22:13.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.294+0000 7f3295d9b640 1 -- 172.21.15.57:0/1241522920 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32900ab050 con 0x7f32980cb8b0 2023-10-30T14:22:13.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.294+0000 7f3295d9b640 1 -- 172.21.15.57:0/1241522920 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32900af780 con 0x7f32980cb8b0 2023-10-30T14:22:13.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.294+0000 7f329cd0f640 1 -- 172.21.15.57:0/1241522920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32980cb8b0 msgr2=0x7f329806b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:13.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.294+0000 7f329cd0f640 1 --2- 172.21.15.57:0/1241522920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32980cb8b0 0x7f329806b600 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3290009e50 tx=0x7f329001f1f0 comp rx=0 tx=0).stop 2023-10-30T14:22:13.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.294+0000 7f329cd0f640 1 -- 172.21.15.57:0/1241522920 shutdown_connections 2023-10-30T14:22:13.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.294+0000 7f329cd0f640 1 --2- 172.21.15.57:0/1241522920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32980cb8b0 0x7f329806b600 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:13.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.294+0000 7f329cd0f640 1 --2- 172.21.15.57:0/1241522920 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3298151840 0x7f32980c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:13.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.294+0000 7f329cd0f640 1 -- 172.21.15.57:0/1241522920 >> 172.21.15.57:0/1241522920 conn(0x7f32980cac70 msgr2=0x7f32980cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:13.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.294+0000 7f329cd0f640 1 -- 172.21.15.57:0/1241522920 shutdown_connections 2023-10-30T14:22:13.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.294+0000 7f329cd0f640 1 -- 172.21.15.57:0/1241522920 wait complete. 2023-10-30T14:22:13.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f329cd0f640 1 Processor -- start 2023-10-30T14:22:13.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f329cd0f640 1 -- start start 2023-10-30T14:22:13.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f329cd0f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32980cb8b0 0x7f3298155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:13.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f329cd0f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3298151840 0x7f32981f1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:13.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f329cd0f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3298151c20 0x7f3298073620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:13.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f329cd0f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32981564e0 con 0x7f3298151840 2023-10-30T14:22:13.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f329cd0f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32981e02c0 con 0x7f32980cb8b0 2023-10-30T14:22:13.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f329cd0f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32981ef100 con 0x7f3298151c20 2023-10-30T14:22:13.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f328ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3298151840 0x7f32981f1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:13.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f328ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3298151840 0x7f32981f1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39204/0 (socket says 172.21.15.57:39204) 2023-10-30T14:22:13.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f328ffff640 1 -- 172.21.15.57:0/2271318474 learned_addr learned my addr 172.21.15.57:0/2271318474 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:13.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f3296d9d640 1 --2- 172.21.15.57:0/2271318474 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3298151c20 0x7f3298073620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:13.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f329659c640 1 --2- 172.21.15.57:0/2271318474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32980cb8b0 0x7f3298155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:13.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f328ffff640 1 -- 172.21.15.57:0/2271318474 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3298151c20 msgr2=0x7f3298073620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:13.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f328ffff640 1 --2- 172.21.15.57:0/2271318474 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3298151c20 0x7f3298073620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:13.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f328ffff640 1 -- 172.21.15.57:0/2271318474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32980cb8b0 msgr2=0x7f3298155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:13.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f328ffff640 1 --2- 172.21.15.57:0/2271318474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32980cb8b0 0x7f3298155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:13.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f328ffff640 1 -- 172.21.15.57:0/2271318474 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3290009b30 con 0x7f3298151840 2023-10-30T14:22:13.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f328ffff640 1 --2- 172.21.15.57:0/2271318474 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3298151840 0x7f32981f1e80 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f328000c420 tx=0x7f328000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:13.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f328effd640 1 -- 172.21.15.57:0/2271318474 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32800054e0 con 0x7f3298151840 2023-10-30T14:22:13.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f328effd640 1 -- 172.21.15.57:0/2271318474 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3280003600 con 0x7f3298151840 2023-10-30T14:22:13.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f328effd640 1 -- 172.21.15.57:0/2271318474 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3280003990 con 0x7f3298151840 2023-10-30T14:22:13.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.298+0000 7f329cd0f640 1 -- 172.21.15.57:0/2271318474 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32981f23c0 con 0x7f3298151840 2023-10-30T14:22:13.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.302+0000 7f329cd0f640 1 -- 172.21.15.57:0/2271318474 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32981dffe0 con 0x7f3298151840 2023-10-30T14:22:13.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.302+0000 7f329cd0f640 1 -- 172.21.15.57:0/2271318474 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f329804e170 con 0x7f3298151840 2023-10-30T14:22:13.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.302+0000 7f328effd640 1 -- 172.21.15.57:0/2271318474 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3280003af0 con 0x7f3298151840 2023-10-30T14:22:13.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.302+0000 7f328effd640 1 --2- 172.21.15.57:0/2271318474 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32780664a0 0x7f3278068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:13.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.302+0000 7f328effd640 1 -- 172.21.15.57:0/2271318474 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f3280083660 con 0x7f3298151840 2023-10-30T14:22:13.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.302+0000 7f329659c640 1 --2- 172.21.15.57:0/2271318474 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32780664a0 0x7f3278068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:13.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.306+0000 7f329659c640 1 --2- 172.21.15.57:0/2271318474 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32780664a0 0x7f3278068960 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3288009af0 tx=0x7f3288008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:13.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.306+0000 7f328effd640 1 -- 172.21.15.57:0/2271318474 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f32800508b0 con 0x7f3298151840 2023-10-30T14:22:13.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.494+0000 7f329cd0f640 1 -- 172.21.15.57:0/2271318474 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f329814fcb0 con 0x7f3298151840 2023-10-30T14:22:13.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.494+0000 7f328effd640 1 -- 172.21.15.57:0/2271318474 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6534 (secure 0 0 0) 0x7f3280017090 con 0x7f3298151840 2023-10-30T14:22:13.497 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:22:13.497 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:21:55.990144+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:stopping","state_seq":28,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:13.498 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 59 2023-10-30T14:22:13.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.498+0000 7f329cd0f640 1 -- 172.21.15.57:0/2271318474 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32780664a0 msgr2=0x7f3278068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:13.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.498+0000 7f329cd0f640 1 --2- 172.21.15.57:0/2271318474 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32780664a0 0x7f3278068960 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3288009af0 tx=0x7f3288008040 comp rx=0 tx=0).stop 2023-10-30T14:22:13.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.498+0000 7f329cd0f640 1 -- 172.21.15.57:0/2271318474 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3298151840 msgr2=0x7f32981f1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:13.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.498+0000 7f329cd0f640 1 --2- 172.21.15.57:0/2271318474 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3298151840 0x7f32981f1e80 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f328000c420 tx=0x7f328000c8e0 comp rx=0 tx=0).stop 2023-10-30T14:22:13.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.498+0000 7f329cd0f640 1 -- 172.21.15.57:0/2271318474 shutdown_connections 2023-10-30T14:22:13.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.498+0000 7f329cd0f640 1 --2- 172.21.15.57:0/2271318474 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32780664a0 0x7f3278068960 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:13.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.498+0000 7f329cd0f640 1 --2- 172.21.15.57:0/2271318474 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3298151c20 0x7f3298073620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:13.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.498+0000 7f329cd0f640 1 --2- 172.21.15.57:0/2271318474 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3298151840 0x7f32981f1e80 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:13.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.498+0000 7f329cd0f640 1 --2- 172.21.15.57:0/2271318474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32980cb8b0 0x7f3298155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:13.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.498+0000 7f329cd0f640 1 -- 172.21.15.57:0/2271318474 >> 172.21.15.57:0/2271318474 conn(0x7f32980cac70 msgr2=0x7f32981e2670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:13.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.498+0000 7f329cd0f640 1 -- 172.21.15.57:0/2271318474 shutdown_connections 2023-10-30T14:22:13.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:13.498+0000 7f329cd0f640 1 -- 172.21.15.57:0/2271318474 wait complete. 2023-10-30T14:22:13.521 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:22:15.522 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:15.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.674+0000 7f8f3750f640 1 Processor -- start 2023-10-30T14:22:15.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.674+0000 7f8f3750f640 1 -- start start 2023-10-30T14:22:15.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.674+0000 7f8f3750f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f30153c80 0x7f8f3015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:15.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.674+0000 7f8f3750f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f30159f20 con 0x7f8f300c4a10 2023-10-30T14:22:15.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.674+0000 7f8f3750f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f3015d620 con 0x7f8f300c42a0 2023-10-30T14:22:15.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.674+0000 7f8f3750f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3015dec0 con 0x7f8f30153c80 2023-10-30T14:22:15.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.674+0000 7f8f34aaa640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f8f300c4a10 0x7f8f3015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:56534/0 (socket says 172.21.15.57:56534) 2023-10-30T14:22:15.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.674+0000 7f8f34aaa640 1 -- 172.21.15.57:0/264982780 learned_addr learned my addr 172.21.15.57:0/264982780 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:22:15.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.674+0000 7f8f352ab640 1 --2- 172.21.15.57:0/264982780 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f30153c80 0x7f8f3015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:15.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.674+0000 7f8f27fff640 1 -- 172.21.15.57:0/264982780 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625959897 0 0) 0x7f8f30159f20 con 0x7f8f300c4a10 2023-10-30T14:22:15.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.674+0000 7f8f27fff640 1 -- 172.21.15.57:0/264982780 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f1c003530 con 0x7f8f300c4a10 2023-10-30T14:22:15.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.674+0000 7f8f27fff640 1 -- 172.21.15.57:0/264982780 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 142251295 0 0) 0x7f8f1c003530 con 0x7f8f300c4a10 2023-10-30T14:22:15.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.674+0000 7f8f27fff640 1 -- 172.21.15.57:0/264982780 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f30153c80 msgr2=0x7f8f3015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:15.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.674+0000 7f8f27fff640 1 --2- 172.21.15.57:0/264982780 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f30153c80 0x7f8f3015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:15.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.674+0000 7f8f27fff640 1 -- 172.21.15.57:0/264982780 >> v1:172.21.15.88:6789/0 conn(0x7f8f300c42a0 legacy=0x7f8f300ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:15.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.674+0000 7f8f27fff640 1 -- 172.21.15.57:0/264982780 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3015d900 con 0x7f8f300c4a10 2023-10-30T14:22:15.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.674+0000 7f8f27fff640 1 -- 172.21.15.57:0/264982780 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f8f20001680 con 0x7f8f300c4a10 2023-10-30T14:22:15.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.678+0000 7f8f27fff640 1 -- 172.21.15.57:0/264982780 >> v1:172.21.15.57:6789/0 conn(0x7f8f300c4a10 legacy=0x7f8f3015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:15.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.678+0000 7f8f27fff640 1 --2- 172.21.15.57:0/264982780 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f1c0039b0 0x7f8f1c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:15.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.678+0000 7f8f27fff640 1 --2- 172.21.15.57:0/264982780 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f30153c80 0x7f8f1c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:15.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.678+0000 7f8f27fff640 1 --2- 172.21.15.57:0/264982780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f1c004d10 0x7f8f1c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:15.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.678+0000 7f8f27fff640 1 -- 172.21.15.57:0/264982780 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3015dec0 con 0x7f8f1c004d10 2023-10-30T14:22:15.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.678+0000 7f8f27fff640 1 -- 172.21.15.57:0/264982780 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c00b990 con 0x7f8f1c0039b0 2023-10-30T14:22:15.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.678+0000 7f8f27fff640 1 -- 172.21.15.57:0/264982780 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c00bb50 con 0x7f8f30153c80 2023-10-30T14:22:15.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.678+0000 7f8f35aac640 1 --2- 172.21.15.57:0/264982780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f1c004d10 0x7f8f1c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:15.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.678+0000 7f8f352ab640 1 --2- 172.21.15.57:0/264982780 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f1c0039b0 0x7f8f1c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:15.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.678+0000 7f8f34aaa640 1 --2- 172.21.15.57:0/264982780 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f30153c80 0x7f8f1c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:15.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.678+0000 7f8f35aac640 1 -- 172.21.15.57:0/264982780 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f30153c80 msgr2=0x7f8f1c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:15.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.678+0000 7f8f35aac640 1 --2- 172.21.15.57:0/264982780 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f30153c80 0x7f8f1c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:15.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.678+0000 7f8f35aac640 1 -- 172.21.15.57:0/264982780 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f1c0039b0 msgr2=0x7f8f1c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:15.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.678+0000 7f8f35aac640 1 --2- 172.21.15.57:0/264982780 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f1c0039b0 0x7f8f1c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:15.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.678+0000 7f8f35aac640 1 -- 172.21.15.57:0/264982780 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c003530 con 0x7f8f1c004d10 2023-10-30T14:22:15.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.678+0000 7f8f35aac640 1 --2- 172.21.15.57:0/264982780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f1c004d10 0x7f8f1c005190 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f8f28001f60 tx=0x7f8f2801c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d6a47c7c38f12b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:15.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.678+0000 7f8f27fff640 1 -- 172.21.15.57:0/264982780 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f280a5750 con 0x7f8f1c004d10 2023-10-30T14:22:15.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.678+0000 7f8f27fff640 1 -- 172.21.15.57:0/264982780 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f280a58b0 con 0x7f8f1c004d10 2023-10-30T14:22:15.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.678+0000 7f8f27fff640 1 -- 172.21.15.57:0/264982780 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f280a5c60 con 0x7f8f1c004d10 2023-10-30T14:22:15.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f3750f640 1 -- 172.21.15.57:0/264982780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f1c004d10 msgr2=0x7f8f1c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:15.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f3750f640 1 --2- 172.21.15.57:0/264982780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f1c004d10 0x7f8f1c005190 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f8f28001f60 tx=0x7f8f2801c860 comp rx=0 tx=0).stop 2023-10-30T14:22:15.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f35aac640 1 -- 172.21.15.57:0/264982780 reap_dead start 2023-10-30T14:22:15.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f3750f640 1 -- 172.21.15.57:0/264982780 shutdown_connections 2023-10-30T14:22:15.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f3750f640 1 -- 172.21.15.57:0/264982780 >> 172.21.15.57:0/264982780 conn(0x7f8f300715c0 msgr2=0x7f8f300719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:15.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f3750f640 1 -- 172.21.15.57:0/264982780 shutdown_connections 2023-10-30T14:22:15.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f3750f640 1 -- 172.21.15.57:0/264982780 wait complete. 2023-10-30T14:22:15.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f3750f640 1 Processor -- start 2023-10-30T14:22:15.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f3750f640 1 -- start start 2023-10-30T14:22:15.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f3750f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f3015c5c0 0x7f8f300b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:15.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f3750f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f3015c9a0 0x7f8f3014ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:15.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f3750f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f300b2670 0x7f8f3014d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:15.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f3750f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f300b3990 con 0x7f8f300b2670 2023-10-30T14:22:15.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f3750f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f30153480 con 0x7f8f3015c5c0 2023-10-30T14:22:15.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f3750f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f301ef930 con 0x7f8f3015c9a0 2023-10-30T14:22:15.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f35aac640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f300b2670 0x7f8f3014d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:15.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f35aac640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f300b2670 0x7f8f3014d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45378/0 (socket says 172.21.15.57:45378) 2023-10-30T14:22:15.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f35aac640 1 -- 172.21.15.57:0/2367451014 learned_addr learned my addr 172.21.15.57:0/2367451014 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:15.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f34aaa640 1 --2- 172.21.15.57:0/2367451014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f3015c9a0 0x7f8f3014ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:15.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.682+0000 7f8f352ab640 1 --2- 172.21.15.57:0/2367451014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f3015c5c0 0x7f8f300b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:15.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.686+0000 7f8f35aac640 1 -- 172.21.15.57:0/2367451014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f3015c9a0 msgr2=0x7f8f3014ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:15.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.686+0000 7f8f35aac640 1 --2- 172.21.15.57:0/2367451014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f3015c9a0 0x7f8f3014ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:15.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.686+0000 7f8f35aac640 1 -- 172.21.15.57:0/2367451014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f3015c5c0 msgr2=0x7f8f300b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:15.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.686+0000 7f8f35aac640 1 --2- 172.21.15.57:0/2367451014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f3015c5c0 0x7f8f300b2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:15.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.686+0000 7f8f35aac640 1 -- 172.21.15.57:0/2367451014 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f300b2d10 con 0x7f8f300b2670 2023-10-30T14:22:15.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.686+0000 7f8f35aac640 1 --2- 172.21.15.57:0/2367451014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f300b2670 0x7f8f3014d3f0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8f28001f60 tx=0x7f8f1c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:15.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.686+0000 7f8f25ffb640 1 -- 172.21.15.57:0/2367451014 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f28003d90 con 0x7f8f300b2670 2023-10-30T14:22:15.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.686+0000 7f8f25ffb640 1 -- 172.21.15.57:0/2367451014 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f280a5e50 con 0x7f8f300b2670 2023-10-30T14:22:15.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.686+0000 7f8f25ffb640 1 -- 172.21.15.57:0/2367451014 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f28005270 con 0x7f8f300b2670 2023-10-30T14:22:15.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.686+0000 7f8f3750f640 1 -- 172.21.15.57:0/2367451014 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f300b2ff0 con 0x7f8f300b2670 2023-10-30T14:22:15.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.686+0000 7f8f3750f640 1 -- 172.21.15.57:0/2367451014 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f300b35b0 con 0x7f8f300b2670 2023-10-30T14:22:15.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.686+0000 7f8f3750f640 1 -- 172.21.15.57:0/2367451014 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f300b3ad0 con 0x7f8f300b2670 2023-10-30T14:22:15.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.686+0000 7f8f25ffb640 1 -- 172.21.15.57:0/2367451014 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8f280b4040 con 0x7f8f300b2670 2023-10-30T14:22:15.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.686+0000 7f8f25ffb640 1 --2- 172.21.15.57:0/2367451014 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f140664a0 0x7f8f14068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:15.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.686+0000 7f8f25ffb640 1 -- 172.21.15.57:0/2367451014 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f8f2811d690 con 0x7f8f300b2670 2023-10-30T14:22:15.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.690+0000 7f8f352ab640 1 --2- 172.21.15.57:0/2367451014 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f140664a0 0x7f8f14068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:15.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.690+0000 7f8f352ab640 1 --2- 172.21.15.57:0/2367451014 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f140664a0 0x7f8f14068960 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8f301503a0 tx=0x7f8f2c009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:15.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.690+0000 7f8f25ffb640 1 -- 172.21.15.57:0/2367451014 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8f300b3ad0 con 0x7f8f300b2670 2023-10-30T14:22:15.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.886+0000 7f8f3750f640 1 -- 172.21.15.57:0/2367451014 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f300b3ad0 con 0x7f8f300b2670 2023-10-30T14:22:15.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.886+0000 7f8f25ffb640 1 -- 172.21.15.57:0/2367451014 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f8f300b3ad0 con 0x7f8f300b2670 2023-10-30T14:22:15.888 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:22:15.889 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:22:12.973521+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:stopping","state_seq":28,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:15.890 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 60 2023-10-30T14:22:15.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.886+0000 7f8f3750f640 1 -- 172.21.15.57:0/2367451014 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f140664a0 msgr2=0x7f8f14068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:15.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.886+0000 7f8f3750f640 1 --2- 172.21.15.57:0/2367451014 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f140664a0 0x7f8f14068960 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8f301503a0 tx=0x7f8f2c009190 comp rx=0 tx=0).stop 2023-10-30T14:22:15.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.886+0000 7f8f3750f640 1 -- 172.21.15.57:0/2367451014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f300b2670 msgr2=0x7f8f3014d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:15.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.886+0000 7f8f3750f640 1 --2- 172.21.15.57:0/2367451014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f300b2670 0x7f8f3014d3f0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8f28001f60 tx=0x7f8f1c005800 comp rx=0 tx=0).stop 2023-10-30T14:22:15.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.886+0000 7f8f3750f640 1 -- 172.21.15.57:0/2367451014 shutdown_connections 2023-10-30T14:22:15.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.886+0000 7f8f3750f640 1 --2- 172.21.15.57:0/2367451014 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f140664a0 0x7f8f14068960 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:15.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.886+0000 7f8f3750f640 1 --2- 172.21.15.57:0/2367451014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f300b2670 0x7f8f3014d3f0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:15.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.886+0000 7f8f3750f640 1 --2- 172.21.15.57:0/2367451014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f3015c9a0 0x7f8f3014ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:15.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.886+0000 7f8f3750f640 1 --2- 172.21.15.57:0/2367451014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f3015c5c0 0x7f8f300b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:15.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.886+0000 7f8f3750f640 1 -- 172.21.15.57:0/2367451014 >> 172.21.15.57:0/2367451014 conn(0x7f8f300715c0 msgr2=0x7f8f300c5710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:15.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.886+0000 7f8f3750f640 1 -- 172.21.15.57:0/2367451014 shutdown_connections 2023-10-30T14:22:15.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:15.886+0000 7f8f3750f640 1 -- 172.21.15.57:0/2367451014 wait complete. 2023-10-30T14:22:15.909 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:22:17.910 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:18.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.062+0000 7f84edd3e640 1 Processor -- start 2023-10-30T14:22:18.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.062+0000 7f84edd3e640 1 -- start start 2023-10-30T14:22:18.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.066+0000 7f84edd3e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f84e8153c80 0x7f84e815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:18.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.066+0000 7f84edd3e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f84e80c42a0 0x7f84e80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:18.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.066+0000 7f84edd3e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e8159f20 con 0x7f84e80c4a10 2023-10-30T14:22:18.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.066+0000 7f84edd3e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e815dec0 con 0x7f84e80c42a0 2023-10-30T14:22:18.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.066+0000 7f84edd3e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e81599a0 con 0x7f84e8153c80 2023-10-30T14:22:18.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.066+0000 7f84e7fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f84e80c42a0 0x7f84e80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:18.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.066+0000 7f84e7fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f84e80c42a0 0x7f84e80ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45394/0 (socket says 172.21.15.57:45394) 2023-10-30T14:22:18.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.066+0000 7f84e7fff640 1 -- 172.21.15.57:0/2932081286 learned_addr learned my addr 172.21.15.57:0/2932081286 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:18.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.066+0000 7f84e77fe640 1 --2- 172.21.15.57:0/2932081286 >> v2:172.21.15.88:3300/0 conn(0x7f84e8153c80 0x7f84e815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:18.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.066+0000 7f84e7fff640 1 -- 172.21.15.57:0/2932081286 >> v2:172.21.15.88:3300/0 conn(0x7f84e8153c80 msgr2=0x7f84e815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:18.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.066+0000 7f84e7fff640 1 --2- 172.21.15.57:0/2932081286 >> v2:172.21.15.88:3300/0 conn(0x7f84e8153c80 0x7f84e815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:18.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.066+0000 7f84e7fff640 1 -- 172.21.15.57:0/2932081286 >> v1:172.21.15.88:6789/0 conn(0x7f84e80c4a10 legacy=0x7f84e815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:18.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.066+0000 7f84e7fff640 1 -- 172.21.15.57:0/2932081286 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e815d620 con 0x7f84e80c42a0 2023-10-30T14:22:18.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84e7fff640 1 --2- 172.21.15.57:0/2932081286 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84e80c42a0 0x7f84e80ca1e0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f84e80d00f0 tx=0x7f84e001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9308d0f3037b219b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:18.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84e67fc640 1 -- 172.21.15.57:0/2932081286 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e00af400 con 0x7f84e80c42a0 2023-10-30T14:22:18.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84e67fc640 1 -- 172.21.15.57:0/2932081286 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e00ab050 con 0x7f84e80c42a0 2023-10-30T14:22:18.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84e67fc640 1 -- 172.21.15.57:0/2932081286 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e00af780 con 0x7f84e80c42a0 2023-10-30T14:22:18.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84edd3e640 1 -- 172.21.15.57:0/2932081286 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84e80c42a0 msgr2=0x7f84e80ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:18.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84edd3e640 1 --2- 172.21.15.57:0/2932081286 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84e80c42a0 0x7f84e80ca1e0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f84e80d00f0 tx=0x7f84e001f1c0 comp rx=0 tx=0).stop 2023-10-30T14:22:18.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84edd3e640 1 -- 172.21.15.57:0/2932081286 shutdown_connections 2023-10-30T14:22:18.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84edd3e640 1 --2- 172.21.15.57:0/2932081286 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84e80c42a0 0x7f84e80ca1e0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:18.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84edd3e640 1 --2- 172.21.15.57:0/2932081286 >> v2:172.21.15.88:3300/0 conn(0x7f84e8153c80 0x7f84e815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:18.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84edd3e640 1 -- 172.21.15.57:0/2932081286 >> 172.21.15.57:0/2932081286 conn(0x7f84e80715c0 msgr2=0x7f84e80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:18.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84edd3e640 1 -- 172.21.15.57:0/2932081286 shutdown_connections 2023-10-30T14:22:18.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84edd3e640 1 -- 172.21.15.57:0/2932081286 wait complete. 2023-10-30T14:22:18.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84edd3e640 1 Processor -- start 2023-10-30T14:22:18.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84edd3e640 1 -- start start 2023-10-30T14:22:18.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84edd3e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84e80c42a0 0x7f84e8152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:18.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84edd3e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84e80c4a10 0x7f84e8154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:18.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84edd3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e8153c80 0x7f84e81f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:18.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84edd3e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e814a4d0 con 0x7f84e80c42a0 2023-10-30T14:22:18.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84edd3e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e80cef00 con 0x7f84e8153c80 2023-10-30T14:22:18.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.070+0000 7f84edd3e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e8150270 con 0x7f84e80c4a10 2023-10-30T14:22:18.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.074+0000 7f84e77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84e80c42a0 0x7f84e8152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:18.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.074+0000 7f84e77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84e80c42a0 0x7f84e8152950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45406/0 (socket says 172.21.15.57:45406) 2023-10-30T14:22:18.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.074+0000 7f84e77fe640 1 -- 172.21.15.57:0/3314085688 learned_addr learned my addr 172.21.15.57:0/3314085688 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:18.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.074+0000 7f84e6ffd640 1 --2- 172.21.15.57:0/3314085688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84e80c4a10 0x7f84e8154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:18.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.074+0000 7f84e7fff640 1 --2- 172.21.15.57:0/3314085688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e8153c80 0x7f84e81f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:18.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.074+0000 7f84e77fe640 1 -- 172.21.15.57:0/3314085688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84e80c4a10 msgr2=0x7f84e8154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:18.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.074+0000 7f84e77fe640 1 --2- 172.21.15.57:0/3314085688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84e80c4a10 0x7f84e8154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:18.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.074+0000 7f84e77fe640 1 -- 172.21.15.57:0/3314085688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e8153c80 msgr2=0x7f84e81f21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:18.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.074+0000 7f84e77fe640 1 --2- 172.21.15.57:0/3314085688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e8153c80 0x7f84e81f21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:18.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.074+0000 7f84e77fe640 1 -- 172.21.15.57:0/3314085688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e0009b30 con 0x7f84e80c42a0 2023-10-30T14:22:18.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.074+0000 7f84e77fe640 1 --2- 172.21.15.57:0/3314085688 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84e80c42a0 0x7f84e8152950 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f84d800d5f0 tx=0x7f84d800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:18.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.074+0000 7f84cffff640 1 -- 172.21.15.57:0/3314085688 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d80125f0 con 0x7f84e80c42a0 2023-10-30T14:22:18.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.074+0000 7f84cffff640 1 -- 172.21.15.57:0/3314085688 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d8012750 con 0x7f84e80c42a0 2023-10-30T14:22:18.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.074+0000 7f84edd3e640 1 -- 172.21.15.57:0/3314085688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e80cebd0 con 0x7f84e80c42a0 2023-10-30T14:22:18.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.074+0000 7f84cffff640 1 -- 172.21.15.57:0/3314085688 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d8012ae0 con 0x7f84e80c42a0 2023-10-30T14:22:18.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.074+0000 7f84edd3e640 1 -- 172.21.15.57:0/3314085688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e80cf790 con 0x7f84e80c42a0 2023-10-30T14:22:18.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.074+0000 7f84edd3e640 1 -- 172.21.15.57:0/3314085688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e80cb660 con 0x7f84e80c42a0 2023-10-30T14:22:18.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.074+0000 7f84cffff640 1 -- 172.21.15.57:0/3314085688 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f84d8012c40 con 0x7f84e80c42a0 2023-10-30T14:22:18.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.074+0000 7f84cffff640 1 --2- 172.21.15.57:0/3314085688 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f84d40664a0 0x7f84d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:18.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.078+0000 7f84cffff640 1 -- 172.21.15.57:0/3314085688 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f84d8084630 con 0x7f84e80c42a0 2023-10-30T14:22:18.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.078+0000 7f84e6ffd640 1 --2- 172.21.15.57:0/3314085688 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f84d40664a0 0x7f84d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:18.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.078+0000 7f84e6ffd640 1 --2- 172.21.15.57:0/3314085688 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f84d40664a0 0x7f84d4068960 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f84e80b19f0 tx=0x7f84dc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:18.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.078+0000 7f84cffff640 1 -- 172.21.15.57:0/3314085688 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f84d8051760 con 0x7f84e80c42a0 2023-10-30T14:22:18.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.270+0000 7f84edd3e640 1 -- 172.21.15.57:0/3314085688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84e80cb8d0 con 0x7f84e80c42a0 2023-10-30T14:22:18.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.270+0000 7f84cffff640 1 -- 172.21.15.57:0/3314085688 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f84d8007380 con 0x7f84e80c42a0 2023-10-30T14:22:18.271 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:22:18.271 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:22:12.973521+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:stopping","state_seq":28,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:18.272 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 60 2023-10-30T14:22:18.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.270+0000 7f84edd3e640 1 -- 172.21.15.57:0/3314085688 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f84d40664a0 msgr2=0x7f84d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:18.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.270+0000 7f84edd3e640 1 --2- 172.21.15.57:0/3314085688 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f84d40664a0 0x7f84d4068960 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f84e80b19f0 tx=0x7f84dc009040 comp rx=0 tx=0).stop 2023-10-30T14:22:18.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.270+0000 7f84edd3e640 1 -- 172.21.15.57:0/3314085688 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84e80c42a0 msgr2=0x7f84e8152950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:18.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.270+0000 7f84edd3e640 1 --2- 172.21.15.57:0/3314085688 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84e80c42a0 0x7f84e8152950 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f84d800d5f0 tx=0x7f84d800dab0 comp rx=0 tx=0).stop 2023-10-30T14:22:18.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.270+0000 7f84edd3e640 1 -- 172.21.15.57:0/3314085688 shutdown_connections 2023-10-30T14:22:18.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.270+0000 7f84edd3e640 1 --2- 172.21.15.57:0/3314085688 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f84d40664a0 0x7f84d4068960 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:18.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.270+0000 7f84edd3e640 1 --2- 172.21.15.57:0/3314085688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e8153c80 0x7f84e81f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:18.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.270+0000 7f84edd3e640 1 --2- 172.21.15.57:0/3314085688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84e80c4a10 0x7f84e8154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:18.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.270+0000 7f84edd3e640 1 --2- 172.21.15.57:0/3314085688 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84e80c42a0 0x7f84e8152950 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:18.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.270+0000 7f84edd3e640 1 -- 172.21.15.57:0/3314085688 >> 172.21.15.57:0/3314085688 conn(0x7f84e80715c0 msgr2=0x7f84e8158790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:18.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.270+0000 7f84edd3e640 1 -- 172.21.15.57:0/3314085688 shutdown_connections 2023-10-30T14:22:18.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:18.270+0000 7f84edd3e640 1 -- 172.21.15.57:0/3314085688 wait complete. 2023-10-30T14:22:18.292 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T14:22:18.293 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 60, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 60, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5078": { "addr": "172.21.15.88:6839/1677924344", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1677924344, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1677924344, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5078, "incarnation": 48, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:stopping", "state_seq": 28 }, "gid_5133": { "addr": "172.21.15.57:6834/2912050965", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2912050965, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2912050965, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5133, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 15 }, "gid_5289": { "addr": "172.21.15.57:6836/2427081813", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2427081813, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2427081813, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5289, "incarnation": 35, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:22:12.973521+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5289, "mds_1": 5133, "mds_2": 5078 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.57:6837/862088802", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 862088802, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 862088802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 54, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5586, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:22:20.295 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:20.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.446+0000 7fbb81aea640 1 Processor -- start 2023-10-30T14:22:20.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb81aea640 1 -- start start 2023-10-30T14:22:20.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb81aea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbb7c14fc50 0x7fbb7c06fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:20.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb81aea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb7c06be90 0x7fbb7c06c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:20.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb81aea640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb7c0c55f0 con 0x7fbb7c0c6320 2023-10-30T14:22:20.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb81aea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb7c0c5f00 con 0x7fbb7c14fc50 2023-10-30T14:22:20.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb81aea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7c1567b0 con 0x7fbb7c06be90 2023-10-30T14:22:20.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7affd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fbb7c0c6320 0x7fbb7c070e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:56548/0 (socket says 172.21.15.57:56548) 2023-10-30T14:22:20.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7affd640 1 -- 172.21.15.57:0/50213245 learned_addr learned my addr 172.21.15.57:0/50213245 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:22:20.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7b7fe640 1 --2- 172.21.15.57:0/50213245 >> v2:172.21.15.88:3300/0 conn(0x7fbb7c14fc50 0x7fbb7c06fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:20.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7bfff640 1 --2- 172.21.15.57:0/50213245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb7c06be90 0x7fbb7c06c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:20.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7a7fc640 1 -- 172.21.15.57:0/50213245 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514376442 0 0) 0x7fbb7c0c55f0 con 0x7fbb7c0c6320 2023-10-30T14:22:20.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7a7fc640 1 -- 172.21.15.57:0/50213245 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb64003530 con 0x7fbb7c0c6320 2023-10-30T14:22:20.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7a7fc640 1 -- 172.21.15.57:0/50213245 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 128509666 0 0) 0x7fbb64003530 con 0x7fbb7c0c6320 2023-10-30T14:22:20.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7a7fc640 1 -- 172.21.15.57:0/50213245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb7c06be90 msgr2=0x7fbb7c06c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:20.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7a7fc640 1 --2- 172.21.15.57:0/50213245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb7c06be90 0x7fbb7c06c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:20.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7a7fc640 1 -- 172.21.15.57:0/50213245 >> v2:172.21.15.88:3300/0 conn(0x7fbb7c14fc50 msgr2=0x7fbb7c06fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:20.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7a7fc640 1 --2- 172.21.15.57:0/50213245 >> v2:172.21.15.88:3300/0 conn(0x7fbb7c14fc50 0x7fbb7c06fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:20.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7a7fc640 1 -- 172.21.15.57:0/50213245 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7c156a70 con 0x7fbb7c0c6320 2023-10-30T14:22:20.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7a7fc640 1 -- 172.21.15.57:0/50213245 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fbb70001680 con 0x7fbb7c0c6320 2023-10-30T14:22:20.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7a7fc640 1 -- 172.21.15.57:0/50213245 >> v1:172.21.15.57:6789/0 conn(0x7fbb7c0c6320 legacy=0x7fbb7c070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:20.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7a7fc640 1 --2- 172.21.15.57:0/50213245 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb64003980 0x7fbb64003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:20.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7a7fc640 1 --2- 172.21.15.57:0/50213245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb7c06be90 0x7fbb640047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:20.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7a7fc640 1 --2- 172.21.15.57:0/50213245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb64004d10 0x7fbb64005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:20.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7a7fc640 1 -- 172.21.15.57:0/50213245 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7c0c5f00 con 0x7fbb64003980 2023-10-30T14:22:20.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7a7fc640 1 -- 172.21.15.57:0/50213245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7c1567b0 con 0x7fbb64004d10 2023-10-30T14:22:20.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7a7fc640 1 -- 172.21.15.57:0/50213245 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6400b990 con 0x7fbb7c06be90 2023-10-30T14:22:20.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7b7fe640 1 --2- 172.21.15.57:0/50213245 >> v2:172.21.15.88:3300/0 conn(0x7fbb7c14fc50 0x7fbb7c06fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:22:20.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7b7fe640 1 --2- 172.21.15.57:0/50213245 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb64003980 0x7fbb64003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:20.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7bfff640 1 --2- 172.21.15.57:0/50213245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb64004d10 0x7fbb64005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:20.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7affd640 1 --2- 172.21.15.57:0/50213245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb7c06be90 0x7fbb640047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:20.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7b7fe640 1 -- 172.21.15.57:0/50213245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb7c06be90 msgr2=0x7fbb640047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:20.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7b7fe640 1 --2- 172.21.15.57:0/50213245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb7c06be90 0x7fbb640047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:20.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7b7fe640 1 -- 172.21.15.57:0/50213245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb64004d10 msgr2=0x7fbb64005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:20.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7b7fe640 1 --2- 172.21.15.57:0/50213245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb64004d10 0x7fbb64005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:20.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7b7fe640 1 -- 172.21.15.57:0/50213245 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb64003530 con 0x7fbb64003980 2023-10-30T14:22:20.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.450+0000 7fbb7affd640 1 --2- 172.21.15.57:0/50213245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb7c06be90 0x7fbb640047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:22:20.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.454+0000 7fbb7b7fe640 1 --2- 172.21.15.57:0/50213245 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb64003980 0x7fbb64003da0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c00aba0 tx=0x7fbb6c01d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe40f9c1be9a4f04 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:20.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.454+0000 7fbb7a7fc640 1 -- 172.21.15.57:0/50213245 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c004220 con 0x7fbb64003980 2023-10-30T14:22:20.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.454+0000 7fbb7a7fc640 1 -- 172.21.15.57:0/50213245 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6c0a8050 con 0x7fbb64003980 2023-10-30T14:22:20.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.454+0000 7fbb7a7fc640 1 -- 172.21.15.57:0/50213245 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c003620 con 0x7fbb64003980 2023-10-30T14:22:20.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.454+0000 7fbb81aea640 1 -- 172.21.15.57:0/50213245 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb64003980 msgr2=0x7fbb64003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:20.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.454+0000 7fbb81aea640 1 --2- 172.21.15.57:0/50213245 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb64003980 0x7fbb64003da0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c00aba0 tx=0x7fbb6c01d070 comp rx=0 tx=0).stop 2023-10-30T14:22:20.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.454+0000 7fbb7bfff640 1 -- 172.21.15.57:0/50213245 reap_dead start 2023-10-30T14:22:20.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.454+0000 7fbb81aea640 1 -- 172.21.15.57:0/50213245 shutdown_connections 2023-10-30T14:22:20.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.454+0000 7fbb81aea640 1 -- 172.21.15.57:0/50213245 >> 172.21.15.57:0/50213245 conn(0x7fbb7c0cac70 msgr2=0x7fbb7c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:20.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.454+0000 7fbb81aea640 1 -- 172.21.15.57:0/50213245 shutdown_connections 2023-10-30T14:22:20.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.454+0000 7fbb81aea640 1 -- 172.21.15.57:0/50213245 wait complete. 2023-10-30T14:22:20.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb81aea640 1 Processor -- start 2023-10-30T14:22:20.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb81aea640 1 -- start start 2023-10-30T14:22:20.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb81aea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb64003980 0x7fbb7c157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:20.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb81aea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb7c071ce0 0x7fbb7c0c4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:20.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb81aea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb7c1598e0 0x7fbb7c0c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:20.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb81aea640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7c149560 con 0x7fbb7c071ce0 2023-10-30T14:22:20.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb81aea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7c13ef40 con 0x7fbb7c1598e0 2023-10-30T14:22:20.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb81aea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7c13d360 con 0x7fbb64003980 2023-10-30T14:22:20.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb7affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb7c071ce0 0x7fbb7c0c4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:20.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb7affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb7c071ce0 0x7fbb7c0c4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45424/0 (socket says 172.21.15.57:45424) 2023-10-30T14:22:20.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb7affd640 1 -- 172.21.15.57:0/3410652073 learned_addr learned my addr 172.21.15.57:0/3410652073 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:20.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb7b7fe640 1 --2- 172.21.15.57:0/3410652073 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb64003980 0x7fbb7c157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:20.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb7bfff640 1 --2- 172.21.15.57:0/3410652073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb7c1598e0 0x7fbb7c0c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:20.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb7affd640 1 -- 172.21.15.57:0/3410652073 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb64003980 msgr2=0x7fbb7c157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:20.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb7affd640 1 --2- 172.21.15.57:0/3410652073 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb64003980 0x7fbb7c157060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:20.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb7affd640 1 -- 172.21.15.57:0/3410652073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb7c1598e0 msgr2=0x7fbb7c0c46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:20.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb7affd640 1 --2- 172.21.15.57:0/3410652073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb7c1598e0 0x7fbb7c0c46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:20.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb7affd640 1 -- 172.21.15.57:0/3410652073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7c149a90 con 0x7fbb7c071ce0 2023-10-30T14:22:20.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb7affd640 1 --2- 172.21.15.57:0/3410652073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb7c071ce0 0x7fbb7c0c4160 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fbb70001000 tx=0x7fbb7000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:20.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb63fff640 1 -- 172.21.15.57:0/3410652073 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb700098b0 con 0x7fbb7c071ce0 2023-10-30T14:22:20.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb63fff640 1 -- 172.21.15.57:0/3410652073 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb70009a10 con 0x7fbb7c071ce0 2023-10-30T14:22:20.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb63fff640 1 -- 172.21.15.57:0/3410652073 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb70009d80 con 0x7fbb7c071ce0 2023-10-30T14:22:20.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb81aea640 1 -- 172.21.15.57:0/3410652073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb7c1497b0 con 0x7fbb7c071ce0 2023-10-30T14:22:20.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.458+0000 7fbb81aea640 1 -- 172.21.15.57:0/3410652073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb7c154b20 con 0x7fbb7c071ce0 2023-10-30T14:22:20.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.462+0000 7fbb81aea640 1 -- 172.21.15.57:0/3410652073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb7c13f0b0 con 0x7fbb7c071ce0 2023-10-30T14:22:20.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.462+0000 7fbb63fff640 1 -- 172.21.15.57:0/3410652073 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbb70014400 con 0x7fbb7c071ce0 2023-10-30T14:22:20.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.462+0000 7fbb63fff640 1 --2- 172.21.15.57:0/3410652073 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbb680664a0 0x7fbb68068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:20.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.462+0000 7fbb63fff640 1 -- 172.21.15.57:0/3410652073 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fbb700838d0 con 0x7fbb7c071ce0 2023-10-30T14:22:20.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.462+0000 7fbb7b7fe640 1 --2- 172.21.15.57:0/3410652073 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbb680664a0 0x7fbb68068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:20.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.462+0000 7fbb7b7fe640 1 --2- 172.21.15.57:0/3410652073 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbb680664a0 0x7fbb68068960 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0ad000 tx=0x7fbb6c0046d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:20.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.466+0000 7fbb63fff640 1 -- 172.21.15.57:0/3410652073 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbb70050bd0 con 0x7fbb7c071ce0 2023-10-30T14:22:20.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.662+0000 7fbb81aea640 1 -- 172.21.15.57:0/3410652073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb7c04e170 con 0x7fbb7c071ce0 2023-10-30T14:22:20.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.662+0000 7fbb63fff640 1 -- 172.21.15.57:0/3410652073 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6532 (secure 0 0 0) 0x7fbb70019020 con 0x7fbb7c071ce0 2023-10-30T14:22:20.664 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:22:20.665 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:22:12.973521+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:stopping","state_seq":28,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:20.665 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 60 2023-10-30T14:22:20.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.662+0000 7fbb81aea640 1 -- 172.21.15.57:0/3410652073 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbb680664a0 msgr2=0x7fbb68068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:20.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.662+0000 7fbb81aea640 1 --2- 172.21.15.57:0/3410652073 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbb680664a0 0x7fbb68068960 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0ad000 tx=0x7fbb6c0046d0 comp rx=0 tx=0).stop 2023-10-30T14:22:20.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.662+0000 7fbb81aea640 1 -- 172.21.15.57:0/3410652073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb7c071ce0 msgr2=0x7fbb7c0c4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:20.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.662+0000 7fbb81aea640 1 --2- 172.21.15.57:0/3410652073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb7c071ce0 0x7fbb7c0c4160 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fbb70001000 tx=0x7fbb7000eb90 comp rx=0 tx=0).stop 2023-10-30T14:22:20.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.662+0000 7fbb81aea640 1 -- 172.21.15.57:0/3410652073 shutdown_connections 2023-10-30T14:22:20.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.662+0000 7fbb81aea640 1 --2- 172.21.15.57:0/3410652073 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbb680664a0 0x7fbb68068960 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:20.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.662+0000 7fbb81aea640 1 --2- 172.21.15.57:0/3410652073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb7c1598e0 0x7fbb7c0c46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:20.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.662+0000 7fbb81aea640 1 --2- 172.21.15.57:0/3410652073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb7c071ce0 0x7fbb7c0c4160 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:20.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.662+0000 7fbb81aea640 1 --2- 172.21.15.57:0/3410652073 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb64003980 0x7fbb7c157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:20.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.662+0000 7fbb81aea640 1 -- 172.21.15.57:0/3410652073 >> 172.21.15.57:0/3410652073 conn(0x7fbb7c0cac70 msgr2=0x7fbb7c1422d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:20.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.666+0000 7fbb81aea640 1 -- 172.21.15.57:0/3410652073 shutdown_connections 2023-10-30T14:22:20.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:20.666+0000 7fbb81aea640 1 -- 172.21.15.57:0/3410652073 wait complete. 2023-10-30T14:22:20.685 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:22:22.686 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:22.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.838+0000 7fe5c36bb640 1 Processor -- start 2023-10-30T14:22:22.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.838+0000 7fe5c36bb640 1 -- start start 2023-10-30T14:22:22.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.838+0000 7fe5c36bb640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe5bc151840 0x7fe5bc0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:22.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.838+0000 7fe5c36bb640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5bc0c5030 con 0x7fe5bc0cb8b0 2023-10-30T14:22:22.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.838+0000 7fe5c36bb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5bc157830 con 0x7fe5bc151c20 2023-10-30T14:22:22.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.838+0000 7fe5c36bb640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5bc157b10 con 0x7fe5bc151840 2023-10-30T14:22:22.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.838+0000 7fe5c1457640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe5bc151840 0x7fe5bc0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:22.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.838+0000 7fe5c1457640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe5bc151840 0x7fe5bc0c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45428/0 (socket says 172.21.15.57:45428) 2023-10-30T14:22:22.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.838+0000 7fe5c1457640 1 -- 172.21.15.57:0/3176435314 learned_addr learned my addr 172.21.15.57:0/3176435314 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:22.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.838+0000 7fe5c1c58640 1 --1- 172.21.15.57:0/3176435314 >> v1:172.21.15.57:6789/0 conn(0x7fe5bc0cb8b0 0x7fe5bc06b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:56556/0 (socket says 172.21.15.57:56556) 2023-10-30T14:22:22.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.842+0000 7fe5c1457640 1 -- 172.21.15.57:0/3176435314 >> v1:172.21.15.88:6789/0 conn(0x7fe5bc151c20 legacy=0x7fe5bc072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:22.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.842+0000 7fe5c1457640 1 -- 172.21.15.57:0/3176435314 >> v1:172.21.15.57:6789/0 conn(0x7fe5bc0cb8b0 legacy=0x7fe5bc06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:22.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.842+0000 7fe5c1457640 1 -- 172.21.15.57:0/3176435314 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5bc157df0 con 0x7fe5bc151840 2023-10-30T14:22:22.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.842+0000 7fe5c1457640 1 --2- 172.21.15.57:0/3176435314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5bc151840 0x7fe5bc0c49a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8009870 tx=0x7fe5b801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d9b5667ac4975d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:22.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.842+0000 7fe5b3fff640 1 -- 172.21.15.57:0/3176435314 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b80a8a30 con 0x7fe5bc151840 2023-10-30T14:22:22.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.842+0000 7fe5b3fff640 1 -- 172.21.15.57:0/3176435314 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b80a8b90 con 0x7fe5bc151840 2023-10-30T14:22:22.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.842+0000 7fe5b3fff640 1 -- 172.21.15.57:0/3176435314 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b80b0440 con 0x7fe5bc151840 2023-10-30T14:22:22.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.842+0000 7fe5c36bb640 1 -- 172.21.15.57:0/3176435314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5bc151840 msgr2=0x7fe5bc0c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:22.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.842+0000 7fe5c36bb640 1 --2- 172.21.15.57:0/3176435314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5bc151840 0x7fe5bc0c49a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8009870 tx=0x7fe5b801ed30 comp rx=0 tx=0).stop 2023-10-30T14:22:22.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c36bb640 1 -- 172.21.15.57:0/3176435314 shutdown_connections 2023-10-30T14:22:22.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c36bb640 1 --2- 172.21.15.57:0/3176435314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5bc151840 0x7fe5bc0c49a0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:22.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c36bb640 1 -- 172.21.15.57:0/3176435314 >> 172.21.15.57:0/3176435314 conn(0x7fe5bc0cac70 msgr2=0x7fe5bc0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:22.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c36bb640 1 -- 172.21.15.57:0/3176435314 shutdown_connections 2023-10-30T14:22:22.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c36bb640 1 -- 172.21.15.57:0/3176435314 wait complete. 2023-10-30T14:22:22.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c36bb640 1 Processor -- start 2023-10-30T14:22:22.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c36bb640 1 -- start start 2023-10-30T14:22:22.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c36bb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5bc0cb8b0 0x7fe5bc156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:22.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c36bb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5bc151840 0x7fe5bc143710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:22.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c36bb640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5bc151c20 0x7fe5bc143c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:22.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c36bb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc1527b0 con 0x7fe5bc0cb8b0 2023-10-30T14:22:22.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c36bb640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc14fbd0 con 0x7fe5bc151840 2023-10-30T14:22:22.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c36bb640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc154ec0 con 0x7fe5bc151c20 2023-10-30T14:22:22.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c1457640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5bc0cb8b0 0x7fe5bc156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:22.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c1457640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5bc0cb8b0 0x7fe5bc156560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45438/0 (socket says 172.21.15.57:45438) 2023-10-30T14:22:22.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c1457640 1 -- 172.21.15.57:0/3864155615 learned_addr learned my addr 172.21.15.57:0/3864155615 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:22.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c1c58640 1 --2- 172.21.15.57:0/3864155615 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5bc151c20 0x7fe5bc143c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:22.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c0c56640 1 --2- 172.21.15.57:0/3864155615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5bc151840 0x7fe5bc143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:22.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c1457640 1 -- 172.21.15.57:0/3864155615 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5bc151c20 msgr2=0x7fe5bc143c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:22.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c1457640 1 --2- 172.21.15.57:0/3864155615 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5bc151c20 0x7fe5bc143c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:22.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c1457640 1 -- 172.21.15.57:0/3864155615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5bc151840 msgr2=0x7fe5bc143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:22.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c1457640 1 --2- 172.21.15.57:0/3864155615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5bc151840 0x7fe5bc143710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:22.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c1457640 1 -- 172.21.15.57:0/3864155615 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b80094d0 con 0x7fe5bc0cb8b0 2023-10-30T14:22:22.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c1c58640 1 --2- 172.21.15.57:0/3864155615 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5bc151c20 0x7fe5bc143c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:22:22.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.846+0000 7fe5c1457640 1 --2- 172.21.15.57:0/3864155615 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5bc0cb8b0 0x7fe5bc156560 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fe5b80b3040 tx=0x7fe5b801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:22.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.850+0000 7fe5b1ffb640 1 -- 172.21.15.57:0/3864155615 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b80be070 con 0x7fe5bc0cb8b0 2023-10-30T14:22:22.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.850+0000 7fe5c36bb640 1 -- 172.21.15.57:0/3864155615 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5bc152a00 con 0x7fe5bc0cb8b0 2023-10-30T14:22:22.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.850+0000 7fe5b1ffb640 1 -- 172.21.15.57:0/3864155615 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b80ba400 con 0x7fe5bc0cb8b0 2023-10-30T14:22:22.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.850+0000 7fe5b1ffb640 1 -- 172.21.15.57:0/3864155615 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b80ba7b0 con 0x7fe5bc0cb8b0 2023-10-30T14:22:22.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.850+0000 7fe5c36bb640 1 -- 172.21.15.57:0/3864155615 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5bc152fc0 con 0x7fe5bc0cb8b0 2023-10-30T14:22:22.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.850+0000 7fe5c36bb640 1 -- 172.21.15.57:0/3864155615 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5bc04e170 con 0x7fe5bc0cb8b0 2023-10-30T14:22:22.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.850+0000 7fe5b1ffb640 1 -- 172.21.15.57:0/3864155615 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe5b80ba910 con 0x7fe5bc0cb8b0 2023-10-30T14:22:22.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.850+0000 7fe5b1ffb640 1 --2- 172.21.15.57:0/3864155615 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5a00664d0 0x7fe5a0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:22.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.850+0000 7fe5b1ffb640 1 -- 172.21.15.57:0/3864155615 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fe5b8121fe0 con 0x7fe5bc0cb8b0 2023-10-30T14:22:22.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.850+0000 7fe5c0c56640 1 --2- 172.21.15.57:0/3864155615 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5a00664d0 0x7fe5a0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:22.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.854+0000 7fe5c0c56640 1 --2- 172.21.15.57:0/3864155615 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5a00664d0 0x7fe5a0068990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc148480 tx=0x7fe5ac009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:22.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:22.854+0000 7fe5b1ffb640 1 -- 172.21.15.57:0/3864155615 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe5b80ef180 con 0x7fe5bc0cb8b0 2023-10-30T14:22:23.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:23.046+0000 7fe5c36bb640 1 -- 172.21.15.57:0/3864155615 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5bc066510 con 0x7fe5bc0cb8b0 2023-10-30T14:22:23.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:23.046+0000 7fe5b1ffb640 1 -- 172.21.15.57:0/3864155615 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+6531 (secure 0 0 0) 0x7fe5b80f69a0 con 0x7fe5bc0cb8b0 2023-10-30T14:22:23.050 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:22:23.050 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:22:22.973067+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:stopping","state_seq":28,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:23.050 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 61 2023-10-30T14:22:23.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:23.050+0000 7fe5c36bb640 1 -- 172.21.15.57:0/3864155615 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5a00664d0 msgr2=0x7fe5a0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:23.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:23.050+0000 7fe5c36bb640 1 --2- 172.21.15.57:0/3864155615 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5a00664d0 0x7fe5a0068990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc148480 tx=0x7fe5ac009040 comp rx=0 tx=0).stop 2023-10-30T14:22:23.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:23.050+0000 7fe5c36bb640 1 -- 172.21.15.57:0/3864155615 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5bc0cb8b0 msgr2=0x7fe5bc156560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:23.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:23.050+0000 7fe5c36bb640 1 --2- 172.21.15.57:0/3864155615 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5bc0cb8b0 0x7fe5bc156560 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fe5b80b3040 tx=0x7fe5b801bad0 comp rx=0 tx=0).stop 2023-10-30T14:22:23.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:23.050+0000 7fe5c36bb640 1 -- 172.21.15.57:0/3864155615 shutdown_connections 2023-10-30T14:22:23.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:23.050+0000 7fe5c36bb640 1 --2- 172.21.15.57:0/3864155615 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5a00664d0 0x7fe5a0068990 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:23.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:23.050+0000 7fe5c36bb640 1 --2- 172.21.15.57:0/3864155615 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5bc151c20 0x7fe5bc143c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:23.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:23.050+0000 7fe5c36bb640 1 --2- 172.21.15.57:0/3864155615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5bc151840 0x7fe5bc143710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:23.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:23.050+0000 7fe5c36bb640 1 --2- 172.21.15.57:0/3864155615 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5bc0cb8b0 0x7fe5bc156560 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:23.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:23.050+0000 7fe5c36bb640 1 -- 172.21.15.57:0/3864155615 >> 172.21.15.57:0/3864155615 conn(0x7fe5bc0cac70 msgr2=0x7fe5bc0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:23.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:23.050+0000 7fe5c36bb640 1 -- 172.21.15.57:0/3864155615 shutdown_connections 2023-10-30T14:22:23.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:23.050+0000 7fe5c36bb640 1 -- 172.21.15.57:0/3864155615 wait complete. 2023-10-30T14:22:23.070 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:22:25.072 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:25.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.222+0000 7f923ec54640 1 Processor -- start 2023-10-30T14:22:25.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.226+0000 7f923ec54640 1 -- start start 2023-10-30T14:22:25.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.226+0000 7f923ec54640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9238150830 0x7f923806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:25.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.226+0000 7f923ec54640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f92380cb7d0 0x7f92380c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:25.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.226+0000 7f923ec54640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92380c2310 con 0x7f92380c3e40 2023-10-30T14:22:25.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.226+0000 7f923ec54640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9238156710 con 0x7f92380cb7d0 2023-10-30T14:22:25.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.226+0000 7f923ec54640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92380c4360 con 0x7f9238150830 2023-10-30T14:22:25.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.226+0000 7f923c9f0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9238150830 0x7f923806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:25.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.226+0000 7f9237fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f92380cb7d0 0x7f92380c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:25.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.226+0000 7f923c9f0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9238150830 0x7f923806b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:33906/0 (socket says 172.21.15.57:33906) 2023-10-30T14:22:25.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.226+0000 7f923c9f0640 1 -- 172.21.15.57:0/185747383 learned_addr learned my addr 172.21.15.57:0/185747383 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:25.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.226+0000 7f923c9f0640 1 -- 172.21.15.57:0/185747383 >> v2:172.21.15.88:3300/0 conn(0x7f92380cb7d0 msgr2=0x7f92380c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:25.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.226+0000 7f923c9f0640 1 --2- 172.21.15.57:0/185747383 >> v2:172.21.15.88:3300/0 conn(0x7f92380cb7d0 0x7f92380c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:25.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.226+0000 7f923c9f0640 1 -- 172.21.15.57:0/185747383 >> v1:172.21.15.88:6789/0 conn(0x7f92380c3e40 legacy=0x7f9238070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:25.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.226+0000 7f923c9f0640 1 -- 172.21.15.57:0/185747383 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92381569b0 con 0x7f9238150830 2023-10-30T14:22:25.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.230+0000 7f923c9f0640 1 --2- 172.21.15.57:0/185747383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9238150830 0x7f923806b500 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f922c009870 tx=0x7f922c01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5fbe536811ded1d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:25.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.230+0000 7f92377fe640 1 -- 172.21.15.57:0/185747383 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c0a8a30 con 0x7f9238150830 2023-10-30T14:22:25.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.230+0000 7f92377fe640 1 -- 172.21.15.57:0/185747383 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922c0a8b90 con 0x7f9238150830 2023-10-30T14:22:25.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.230+0000 7f92377fe640 1 -- 172.21.15.57:0/185747383 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c0b0440 con 0x7f9238150830 2023-10-30T14:22:25.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.230+0000 7f923ec54640 1 -- 172.21.15.57:0/185747383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9238150830 msgr2=0x7f923806b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:25.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.230+0000 7f923ec54640 1 --2- 172.21.15.57:0/185747383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9238150830 0x7f923806b500 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f922c009870 tx=0x7f922c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:22:25.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.230+0000 7f923ec54640 1 -- 172.21.15.57:0/185747383 shutdown_connections 2023-10-30T14:22:25.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.230+0000 7f923ec54640 1 --2- 172.21.15.57:0/185747383 >> v2:172.21.15.88:3300/0 conn(0x7f92380cb7d0 0x7f92380c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:25.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.230+0000 7f923ec54640 1 --2- 172.21.15.57:0/185747383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9238150830 0x7f923806b500 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:25.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.230+0000 7f923ec54640 1 -- 172.21.15.57:0/185747383 >> 172.21.15.57:0/185747383 conn(0x7f92380cac70 msgr2=0x7f92380cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:25.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.230+0000 7f923ec54640 1 -- 172.21.15.57:0/185747383 shutdown_connections 2023-10-30T14:22:25.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.230+0000 7f923ec54640 1 -- 172.21.15.57:0/185747383 wait complete. 2023-10-30T14:22:25.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923ec54640 1 Processor -- start 2023-10-30T14:22:25.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923ec54640 1 -- start start 2023-10-30T14:22:25.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923ec54640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92380c3e40 0x7f9238073620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:25.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923ec54640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92380cb7d0 0x7f9238151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:25.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923ec54640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9238150830 0x7f9238149be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:25.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923ec54640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9238157860 con 0x7f9238150830 2023-10-30T14:22:25.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923ec54640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9238157b00 con 0x7f92380c3e40 2023-10-30T14:22:25.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923ec54640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9238154490 con 0x7f92380cb7d0 2023-10-30T14:22:25.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923d1f1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9238150830 0x7f9238149be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:25.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923d1f1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9238150830 0x7f9238149be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43592/0 (socket says 172.21.15.57:43592) 2023-10-30T14:22:25.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923d1f1640 1 -- 172.21.15.57:0/2911408125 learned_addr learned my addr 172.21.15.57:0/2911408125 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:25.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923c9f0640 1 --2- 172.21.15.57:0/2911408125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92380c3e40 0x7f9238073620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:25.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f9237fff640 1 --2- 172.21.15.57:0/2911408125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92380cb7d0 0x7f9238151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:25.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923d1f1640 1 -- 172.21.15.57:0/2911408125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92380cb7d0 msgr2=0x7f9238151b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:25.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923d1f1640 1 --2- 172.21.15.57:0/2911408125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92380cb7d0 0x7f9238151b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:25.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923d1f1640 1 -- 172.21.15.57:0/2911408125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92380c3e40 msgr2=0x7f9238073620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:25.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923d1f1640 1 --2- 172.21.15.57:0/2911408125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92380c3e40 0x7f9238073620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:25.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923d1f1640 1 -- 172.21.15.57:0/2911408125 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922c0094d0 con 0x7f9238150830 2023-10-30T14:22:25.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923d1f1640 1 --2- 172.21.15.57:0/2911408125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9238150830 0x7f9238149be0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f923000c7a0 tx=0x7f923000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:25.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f92357fa640 1 -- 172.21.15.57:0/2911408125 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9230003b80 con 0x7f9238150830 2023-10-30T14:22:25.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f92357fa640 1 -- 172.21.15.57:0/2911408125 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9230003ce0 con 0x7f9238150830 2023-10-30T14:22:25.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923ec54640 1 -- 172.21.15.57:0/2911408125 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9238154770 con 0x7f9238150830 2023-10-30T14:22:25.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f92357fa640 1 -- 172.21.15.57:0/2911408125 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923000aa30 con 0x7f9238150830 2023-10-30T14:22:25.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923ec54640 1 -- 172.21.15.57:0/2911408125 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9238154d30 con 0x7f9238150830 2023-10-30T14:22:25.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.234+0000 7f923ec54640 1 -- 172.21.15.57:0/2911408125 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9238154f20 con 0x7f9238150830 2023-10-30T14:22:25.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.238+0000 7f92357fa640 1 -- 172.21.15.57:0/2911408125 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f923000ab90 con 0x7f9238150830 2023-10-30T14:22:25.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.238+0000 7f92357fa640 1 --2- 172.21.15.57:0/2911408125 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f921c0664a0 0x7f921c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:25.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.238+0000 7f92357fa640 1 -- 172.21.15.57:0/2911408125 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f9230083ec0 con 0x7f9238150830 2023-10-30T14:22:25.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.238+0000 7f923c9f0640 1 --2- 172.21.15.57:0/2911408125 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f921c0664a0 0x7f921c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:25.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.238+0000 7f923c9f0640 1 --2- 172.21.15.57:0/2911408125 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f921c0664a0 0x7f921c068960 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f922c0b2000 tx=0x7f922c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:25.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.242+0000 7f92357fa640 1 -- 172.21.15.57:0/2911408125 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f92300510a0 con 0x7f9238150830 2023-10-30T14:22:25.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.434+0000 7f923ec54640 1 -- 172.21.15.57:0/2911408125 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f923804e170 con 0x7f9238150830 2023-10-30T14:22:25.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.434+0000 7f92357fa640 1 -- 172.21.15.57:0/2911408125 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f92300546c0 con 0x7f9238150830 2023-10-30T14:22:25.437 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:22:25.437 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:22:22.973067+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:stopping","state_seq":28,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:25.438 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 61 2023-10-30T14:22:25.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.438+0000 7f923ec54640 1 -- 172.21.15.57:0/2911408125 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f921c0664a0 msgr2=0x7f921c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:25.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.438+0000 7f923ec54640 1 --2- 172.21.15.57:0/2911408125 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f921c0664a0 0x7f921c068960 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f922c0b2000 tx=0x7f922c009040 comp rx=0 tx=0).stop 2023-10-30T14:22:25.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.438+0000 7f923ec54640 1 -- 172.21.15.57:0/2911408125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9238150830 msgr2=0x7f9238149be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:25.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.438+0000 7f923ec54640 1 --2- 172.21.15.57:0/2911408125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9238150830 0x7f9238149be0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f923000c7a0 tx=0x7f923000cc60 comp rx=0 tx=0).stop 2023-10-30T14:22:25.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.438+0000 7f923ec54640 1 -- 172.21.15.57:0/2911408125 shutdown_connections 2023-10-30T14:22:25.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.438+0000 7f923ec54640 1 --2- 172.21.15.57:0/2911408125 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f921c0664a0 0x7f921c068960 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:25.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.438+0000 7f923ec54640 1 --2- 172.21.15.57:0/2911408125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9238150830 0x7f9238149be0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:25.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.438+0000 7f923ec54640 1 --2- 172.21.15.57:0/2911408125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92380cb7d0 0x7f9238151b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:25.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.438+0000 7f923ec54640 1 --2- 172.21.15.57:0/2911408125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92380c3e40 0x7f9238073620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:25.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.438+0000 7f923ec54640 1 -- 172.21.15.57:0/2911408125 >> 172.21.15.57:0/2911408125 conn(0x7f92380cac70 msgr2=0x7f923814e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:25.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.438+0000 7f923ec54640 1 -- 172.21.15.57:0/2911408125 shutdown_connections 2023-10-30T14:22:25.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:25.438+0000 7f923ec54640 1 -- 172.21.15.57:0/2911408125 wait complete. 2023-10-30T14:22:25.457 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:22:27.459 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:27.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.610+0000 7f516e86b640 1 Processor -- start 2023-10-30T14:22:27.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.610+0000 7f516e86b640 1 -- start start 2023-10-30T14:22:27.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.614+0000 7f516e86b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f516814fc50 0x7f516806fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:27.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.614+0000 7f516e86b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51680c6320 0x7f5168070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:27.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.614+0000 7f516e86b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51680c55f0 con 0x7f516806be90 2023-10-30T14:22:27.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.614+0000 7f516e86b640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51680c5f00 con 0x7f516814fc50 2023-10-30T14:22:27.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.614+0000 7f516e86b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5168071350 con 0x7f51680c6320 2023-10-30T14:22:27.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.614+0000 7f5167fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f516814fc50 0x7f516806fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:27.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.614+0000 7f5167fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f516814fc50 0x7f516806fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43596/0 (socket says 172.21.15.57:43596) 2023-10-30T14:22:27.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.614+0000 7f5167fff640 1 -- 172.21.15.57:0/1257212918 learned_addr learned my addr 172.21.15.57:0/1257212918 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:27.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.614+0000 7f51677fe640 1 --2- 172.21.15.57:0/1257212918 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51680c6320 0x7f5168070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:27.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.614+0000 7f5167fff640 1 -- 172.21.15.57:0/1257212918 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51680c6320 msgr2=0x7f5168070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:27.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.614+0000 7f5167fff640 1 --2- 172.21.15.57:0/1257212918 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51680c6320 0x7f5168070e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:27.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.614+0000 7f5167fff640 1 -- 172.21.15.57:0/1257212918 >> v1:172.21.15.88:6789/0 conn(0x7f516806be90 legacy=0x7f5168155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:27.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.614+0000 7f5167fff640 1 -- 172.21.15.57:0/1257212918 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51681567b0 con 0x7f516814fc50 2023-10-30T14:22:27.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.618+0000 7f5167fff640 1 --2- 172.21.15.57:0/1257212918 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f516814fc50 0x7f516806fdf0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f5158006fd0 tx=0x7f515801e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e02cae36aa13f756 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:27.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.618+0000 7f5166ffd640 1 -- 172.21.15.57:0/1257212918 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5158004230 con 0x7f516814fc50 2023-10-30T14:22:27.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.618+0000 7f5166ffd640 1 -- 172.21.15.57:0/1257212918 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51580ab070 con 0x7f516814fc50 2023-10-30T14:22:27.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.618+0000 7f5166ffd640 1 -- 172.21.15.57:0/1257212918 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51580a7790 con 0x7f516814fc50 2023-10-30T14:22:27.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.618+0000 7f516e86b640 1 -- 172.21.15.57:0/1257212918 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f516814fc50 msgr2=0x7f516806fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:27.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.618+0000 7f516e86b640 1 --2- 172.21.15.57:0/1257212918 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f516814fc50 0x7f516806fdf0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f5158006fd0 tx=0x7f515801e940 comp rx=0 tx=0).stop 2023-10-30T14:22:27.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.618+0000 7f516e86b640 1 -- 172.21.15.57:0/1257212918 shutdown_connections 2023-10-30T14:22:27.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.618+0000 7f516e86b640 1 --2- 172.21.15.57:0/1257212918 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51680c6320 0x7f5168070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:27.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.618+0000 7f516e86b640 1 --2- 172.21.15.57:0/1257212918 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f516814fc50 0x7f516806fdf0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:27.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.618+0000 7f516e86b640 1 -- 172.21.15.57:0/1257212918 >> 172.21.15.57:0/1257212918 conn(0x7f51680cac70 msgr2=0x7f51680cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:27.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.618+0000 7f516e86b640 1 -- 172.21.15.57:0/1257212918 shutdown_connections 2023-10-30T14:22:27.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.618+0000 7f516e86b640 1 -- 172.21.15.57:0/1257212918 wait complete. 2023-10-30T14:22:27.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.618+0000 7f516e86b640 1 Processor -- start 2023-10-30T14:22:27.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.618+0000 7f516e86b640 1 -- start start 2023-10-30T14:22:27.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f516e86b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f516806be90 0x7f51680c9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:27.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f516e86b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51680c6320 0x7f51681f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:27.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f516e86b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f516814fc50 0x7f51681f2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:27.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f516e86b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51681f2180 con 0x7f516806be90 2023-10-30T14:22:27.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f516e86b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51681f5870 con 0x7f516814fc50 2023-10-30T14:22:27.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f516e86b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51681f5b50 con 0x7f51680c6320 2023-10-30T14:22:27.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f5167fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f516806be90 0x7f51680c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:27.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f5167fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f516806be90 0x7f51680c9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43598/0 (socket says 172.21.15.57:43598) 2023-10-30T14:22:27.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f5167fff640 1 -- 172.21.15.57:0/943853337 learned_addr learned my addr 172.21.15.57:0/943853337 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:27.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f51677fe640 1 --2- 172.21.15.57:0/943853337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51680c6320 0x7f51681f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:27.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f516ce08640 1 --2- 172.21.15.57:0/943853337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f516814fc50 0x7f51681f2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:27.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f5167fff640 1 -- 172.21.15.57:0/943853337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51680c6320 msgr2=0x7f51681f25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:27.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f5167fff640 1 --2- 172.21.15.57:0/943853337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51680c6320 0x7f51681f25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:27.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f5167fff640 1 -- 172.21.15.57:0/943853337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f516814fc50 msgr2=0x7f51681f2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:27.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f5167fff640 1 --2- 172.21.15.57:0/943853337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f516814fc50 0x7f51681f2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:27.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f5167fff640 1 -- 172.21.15.57:0/943853337 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515800b780 con 0x7f516806be90 2023-10-30T14:22:27.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f51677fe640 1 --2- 172.21.15.57:0/943853337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51680c6320 0x7f51681f25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:22:27.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f516ce08640 1 --2- 172.21.15.57:0/943853337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f516814fc50 0x7f51681f2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:22:27.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f5167fff640 1 --2- 172.21.15.57:0/943853337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f516806be90 0x7f51680c9700 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f515801c570 tx=0x7f515801b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:27.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f5164ff9640 1 -- 172.21.15.57:0/943853337 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51580a7d20 con 0x7f516806be90 2023-10-30T14:22:27.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f5164ff9640 1 -- 172.21.15.57:0/943853337 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51580a7e80 con 0x7f516806be90 2023-10-30T14:22:27.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f5164ff9640 1 -- 172.21.15.57:0/943853337 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5158003840 con 0x7f516806be90 2023-10-30T14:22:27.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f516e86b640 1 -- 172.21.15.57:0/943853337 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51681f5e30 con 0x7f516806be90 2023-10-30T14:22:27.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f516e86b640 1 -- 172.21.15.57:0/943853337 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51681f63f0 con 0x7f516806be90 2023-10-30T14:22:27.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f516e86b640 1 -- 172.21.15.57:0/943853337 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5168073de0 con 0x7f516806be90 2023-10-30T14:22:27.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.622+0000 7f5164ff9640 1 -- 172.21.15.57:0/943853337 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f51580039a0 con 0x7f516806be90 2023-10-30T14:22:27.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.626+0000 7f5164ff9640 1 --2- 172.21.15.57:0/943853337 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51540664a0 0x7f5154068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:27.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.626+0000 7f5164ff9640 1 -- 172.21.15.57:0/943853337 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f515811e3d0 con 0x7f516806be90 2023-10-30T14:22:27.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.626+0000 7f51677fe640 1 --2- 172.21.15.57:0/943853337 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51540664a0 0x7f5154068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:27.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.626+0000 7f51677fe640 1 --2- 172.21.15.57:0/943853337 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51540664a0 0x7f5154068960 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f51680c9230 tx=0x7f515c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:27.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.630+0000 7f5164ff9640 1 -- 172.21.15.57:0/943853337 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f51580eb740 con 0x7f516806be90 2023-10-30T14:22:27.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.826+0000 7f516e86b640 1 -- 172.21.15.57:0/943853337 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51681f22d0 con 0x7f516806be90 2023-10-30T14:22:27.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.826+0000 7f5164ff9640 1 -- 172.21.15.57:0/943853337 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f51681f22d0 con 0x7f516806be90 2023-10-30T14:22:27.828 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:22:27.828 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:22:25.990261+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":5289,"mds_1":5133,"mds_2":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"f","rank":2,"incarnation":48,"state":"up:stopping","state_seq":28,"addr":"172.21.15.88:6839/1677924344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1677924344},{"type":"v1","addr":"172.21.15.88:6839","nonce":1677924344}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:27.829 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 62 2023-10-30T14:22:27.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.826+0000 7f516e86b640 1 -- 172.21.15.57:0/943853337 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51540664a0 msgr2=0x7f5154068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:27.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.826+0000 7f516e86b640 1 --2- 172.21.15.57:0/943853337 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51540664a0 0x7f5154068960 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f51680c9230 tx=0x7f515c008040 comp rx=0 tx=0).stop 2023-10-30T14:22:27.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.826+0000 7f516e86b640 1 -- 172.21.15.57:0/943853337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f516806be90 msgr2=0x7f51680c9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:27.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.826+0000 7f516e86b640 1 --2- 172.21.15.57:0/943853337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f516806be90 0x7f51680c9700 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f515801c570 tx=0x7f515801b6e0 comp rx=0 tx=0).stop 2023-10-30T14:22:27.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.826+0000 7f516e86b640 1 -- 172.21.15.57:0/943853337 shutdown_connections 2023-10-30T14:22:27.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.826+0000 7f516e86b640 1 --2- 172.21.15.57:0/943853337 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51540664a0 0x7f5154068960 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:27.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.826+0000 7f516e86b640 1 --2- 172.21.15.57:0/943853337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f516814fc50 0x7f51681f2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:27.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.826+0000 7f516e86b640 1 --2- 172.21.15.57:0/943853337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51680c6320 0x7f51681f25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:27.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.826+0000 7f516e86b640 1 --2- 172.21.15.57:0/943853337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f516806be90 0x7f51680c9700 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:27.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.826+0000 7f516e86b640 1 -- 172.21.15.57:0/943853337 >> 172.21.15.57:0/943853337 conn(0x7f51680cac70 msgr2=0x7f516815f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:27.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.826+0000 7f516e86b640 1 -- 172.21.15.57:0/943853337 shutdown_connections 2023-10-30T14:22:27.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:27.830+0000 7f516e86b640 1 -- 172.21.15.57:0/943853337 wait complete. 2023-10-30T14:22:27.849 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:22:29.842 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/21' 2023-10-30T14:22:29.842 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/27' 2023-10-30T14:22:29.842 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/31' 2023-10-30T14:22:29.842 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/36' 2023-10-30T14:22:29.843 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/20' 2023-10-30T14:22:29.843 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/39' 2023-10-30T14:22:29.843 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/34' 2023-10-30T14:22:29.843 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/38' 2023-10-30T14:22:29.843 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/28' 2023-10-30T14:22:29.843 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/24' 2023-10-30T14:22:29.844 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/29' 2023-10-30T14:22:29.844 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/32' 2023-10-30T14:22:29.844 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/22' 2023-10-30T14:22:29.844 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/25' 2023-10-30T14:22:29.844 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/30' 2023-10-30T14:22:29.844 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/33' 2023-10-30T14:22:29.845 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/23' 2023-10-30T14:22:29.845 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/35' 2023-10-30T14:22:29.845 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/26' 2023-10-30T14:22:29.845 INFO:tasks.workunit.client.0.smithi057.stdout:removed './287/37' 2023-10-30T14:22:29.845 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './287' 2023-10-30T14:22:29.846 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/21' 2023-10-30T14:22:29.846 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/27' 2023-10-30T14:22:29.846 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/31' 2023-10-30T14:22:29.846 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/36' 2023-10-30T14:22:29.846 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/20' 2023-10-30T14:22:29.846 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/39' 2023-10-30T14:22:29.847 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/34' 2023-10-30T14:22:29.847 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/38' 2023-10-30T14:22:29.847 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/28' 2023-10-30T14:22:29.847 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/24' 2023-10-30T14:22:29.847 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/29' 2023-10-30T14:22:29.848 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/32' 2023-10-30T14:22:29.848 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/22' 2023-10-30T14:22:29.848 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/25' 2023-10-30T14:22:29.848 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/30' 2023-10-30T14:22:29.848 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/33' 2023-10-30T14:22:29.848 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/23' 2023-10-30T14:22:29.849 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/35' 2023-10-30T14:22:29.849 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/26' 2023-10-30T14:22:29.849 INFO:tasks.workunit.client.0.smithi057.stdout:removed './288/37' 2023-10-30T14:22:29.849 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './288' 2023-10-30T14:22:29.849 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/21' 2023-10-30T14:22:29.850 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/27' 2023-10-30T14:22:29.850 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/31' 2023-10-30T14:22:29.850 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/36' 2023-10-30T14:22:29.850 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/20' 2023-10-30T14:22:29.850 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/39' 2023-10-30T14:22:29.850 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/34' 2023-10-30T14:22:29.851 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/38' 2023-10-30T14:22:29.851 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/28' 2023-10-30T14:22:29.851 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/24' 2023-10-30T14:22:29.851 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/29' 2023-10-30T14:22:29.851 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/32' 2023-10-30T14:22:29.851 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/22' 2023-10-30T14:22:29.852 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/25' 2023-10-30T14:22:29.852 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/30' 2023-10-30T14:22:29.852 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/33' 2023-10-30T14:22:29.852 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/23' 2023-10-30T14:22:29.852 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/35' 2023-10-30T14:22:29.853 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/26' 2023-10-30T14:22:29.853 INFO:tasks.workunit.client.0.smithi057.stdout:removed './289/37' 2023-10-30T14:22:29.853 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './289' 2023-10-30T14:22:29.853 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/21' 2023-10-30T14:22:29.853 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/27' 2023-10-30T14:22:29.853 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/31' 2023-10-30T14:22:29.854 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/36' 2023-10-30T14:22:29.854 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/20' 2023-10-30T14:22:29.854 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/39' 2023-10-30T14:22:29.854 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/34' 2023-10-30T14:22:29.854 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/38' 2023-10-30T14:22:29.854 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/28' 2023-10-30T14:22:29.855 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/24' 2023-10-30T14:22:29.855 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/29' 2023-10-30T14:22:29.855 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/32' 2023-10-30T14:22:29.855 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/22' 2023-10-30T14:22:29.855 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/25' 2023-10-30T14:22:29.856 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/30' 2023-10-30T14:22:29.856 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/33' 2023-10-30T14:22:29.856 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/23' 2023-10-30T14:22:29.856 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/35' 2023-10-30T14:22:29.856 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/26' 2023-10-30T14:22:29.856 INFO:tasks.workunit.client.0.smithi057.stdout:removed './290/37' 2023-10-30T14:22:29.857 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './290' 2023-10-30T14:22:29.857 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/21' 2023-10-30T14:22:29.857 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/27' 2023-10-30T14:22:29.857 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/31' 2023-10-30T14:22:29.857 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/36' 2023-10-30T14:22:29.858 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/20' 2023-10-30T14:22:29.858 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/39' 2023-10-30T14:22:29.858 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/34' 2023-10-30T14:22:29.858 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/38' 2023-10-30T14:22:29.858 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/28' 2023-10-30T14:22:29.858 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/24' 2023-10-30T14:22:29.859 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/29' 2023-10-30T14:22:29.859 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/32' 2023-10-30T14:22:29.859 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/22' 2023-10-30T14:22:29.859 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/25' 2023-10-30T14:22:29.859 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/30' 2023-10-30T14:22:29.860 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/33' 2023-10-30T14:22:29.860 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/23' 2023-10-30T14:22:29.860 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/35' 2023-10-30T14:22:29.860 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/26' 2023-10-30T14:22:29.860 INFO:tasks.workunit.client.0.smithi057.stdout:removed './291/37' 2023-10-30T14:22:29.860 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './291' 2023-10-30T14:22:29.861 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/21' 2023-10-30T14:22:29.861 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/27' 2023-10-30T14:22:29.861 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/31' 2023-10-30T14:22:29.861 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/36' 2023-10-30T14:22:29.861 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/20' 2023-10-30T14:22:29.862 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/39' 2023-10-30T14:22:29.862 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/34' 2023-10-30T14:22:29.862 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/38' 2023-10-30T14:22:29.862 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/28' 2023-10-30T14:22:29.862 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/24' 2023-10-30T14:22:29.862 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/29' 2023-10-30T14:22:29.863 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/32' 2023-10-30T14:22:29.863 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/22' 2023-10-30T14:22:29.863 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/25' 2023-10-30T14:22:29.863 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/30' 2023-10-30T14:22:29.863 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/33' 2023-10-30T14:22:29.864 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/23' 2023-10-30T14:22:29.864 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/35' 2023-10-30T14:22:29.864 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/26' 2023-10-30T14:22:29.864 INFO:tasks.workunit.client.0.smithi057.stdout:removed './292/37' 2023-10-30T14:22:29.864 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './292' 2023-10-30T14:22:29.864 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/21' 2023-10-30T14:22:29.865 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/27' 2023-10-30T14:22:29.865 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/31' 2023-10-30T14:22:29.865 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/36' 2023-10-30T14:22:29.865 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/20' 2023-10-30T14:22:29.865 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/39' 2023-10-30T14:22:29.865 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/34' 2023-10-30T14:22:29.866 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/38' 2023-10-30T14:22:29.866 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/28' 2023-10-30T14:22:29.866 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/24' 2023-10-30T14:22:29.866 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/29' 2023-10-30T14:22:29.866 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/32' 2023-10-30T14:22:29.867 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/22' 2023-10-30T14:22:29.867 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/25' 2023-10-30T14:22:29.867 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/30' 2023-10-30T14:22:29.867 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/33' 2023-10-30T14:22:29.867 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/23' 2023-10-30T14:22:29.867 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/35' 2023-10-30T14:22:29.868 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/26' 2023-10-30T14:22:29.868 INFO:tasks.workunit.client.0.smithi057.stdout:removed './293/37' 2023-10-30T14:22:29.868 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './293' 2023-10-30T14:22:29.868 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/21' 2023-10-30T14:22:29.868 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/27' 2023-10-30T14:22:29.869 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/31' 2023-10-30T14:22:29.869 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/36' 2023-10-30T14:22:29.869 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/20' 2023-10-30T14:22:29.869 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/39' 2023-10-30T14:22:29.869 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/34' 2023-10-30T14:22:29.869 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/38' 2023-10-30T14:22:29.870 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/28' 2023-10-30T14:22:29.870 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/24' 2023-10-30T14:22:29.870 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/29' 2023-10-30T14:22:29.870 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/32' 2023-10-30T14:22:29.870 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/22' 2023-10-30T14:22:29.871 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/25' 2023-10-30T14:22:29.871 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/30' 2023-10-30T14:22:29.871 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/33' 2023-10-30T14:22:29.871 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/23' 2023-10-30T14:22:29.871 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/35' 2023-10-30T14:22:29.872 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/26' 2023-10-30T14:22:29.872 INFO:tasks.workunit.client.0.smithi057.stdout:removed './294/37' 2023-10-30T14:22:29.872 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './294' 2023-10-30T14:22:29.872 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/21' 2023-10-30T14:22:29.872 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/27' 2023-10-30T14:22:29.872 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/31' 2023-10-30T14:22:29.873 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/36' 2023-10-30T14:22:29.873 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/20' 2023-10-30T14:22:29.873 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/39' 2023-10-30T14:22:29.873 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/34' 2023-10-30T14:22:29.873 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/38' 2023-10-30T14:22:29.874 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/28' 2023-10-30T14:22:29.874 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/24' 2023-10-30T14:22:29.874 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/29' 2023-10-30T14:22:29.874 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/32' 2023-10-30T14:22:29.874 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/22' 2023-10-30T14:22:29.874 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/25' 2023-10-30T14:22:29.875 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/30' 2023-10-30T14:22:29.875 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/33' 2023-10-30T14:22:29.875 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/23' 2023-10-30T14:22:29.875 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/35' 2023-10-30T14:22:29.875 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/26' 2023-10-30T14:22:29.875 INFO:tasks.workunit.client.0.smithi057.stdout:removed './295/37' 2023-10-30T14:22:29.876 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './295' 2023-10-30T14:22:29.876 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/21' 2023-10-30T14:22:29.876 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/27' 2023-10-30T14:22:29.876 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/31' 2023-10-30T14:22:29.876 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/36' 2023-10-30T14:22:29.877 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/20' 2023-10-30T14:22:29.877 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/39' 2023-10-30T14:22:29.877 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/34' 2023-10-30T14:22:29.877 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/38' 2023-10-30T14:22:29.877 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/28' 2023-10-30T14:22:29.877 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/24' 2023-10-30T14:22:29.878 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/29' 2023-10-30T14:22:29.878 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/32' 2023-10-30T14:22:29.878 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/22' 2023-10-30T14:22:29.878 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/25' 2023-10-30T14:22:29.878 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/30' 2023-10-30T14:22:29.878 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/33' 2023-10-30T14:22:29.879 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/23' 2023-10-30T14:22:29.879 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/35' 2023-10-30T14:22:29.879 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/26' 2023-10-30T14:22:29.879 INFO:tasks.workunit.client.0.smithi057.stdout:removed './296/37' 2023-10-30T14:22:29.879 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './296' 2023-10-30T14:22:29.880 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/21' 2023-10-30T14:22:29.880 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/27' 2023-10-30T14:22:29.880 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:29.881 INFO:tasks.ceph.mds.f.smithi088.stdout:starting mds.f at 2023-10-30T14:22:30.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.026+0000 7f9c9a446640 1 Processor -- start 2023-10-30T14:22:30.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.026+0000 7f9c9a446640 1 -- start start 2023-10-30T14:22:30.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.030+0000 7f9c9a446640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9c940c4a10 0x7f9c9415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:30.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.030+0000 7f9c9a446640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9c940c42a0 0x7f9c940ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:30.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.030+0000 7f9c9a446640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c94159f20 con 0x7f9c94153c80 2023-10-30T14:22:30.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.030+0000 7f9c9a446640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c9415dec0 con 0x7f9c940c42a0 2023-10-30T14:22:30.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.030+0000 7f9c9a446640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c941599a0 con 0x7f9c940c4a10 2023-10-30T14:22:30.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.030+0000 7f9c989e3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9c940c42a0 0x7f9c940ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:30.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.030+0000 7f9c989e3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9c940c42a0 0x7f9c940ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43604/0 (socket says 172.21.15.57:43604) 2023-10-30T14:22:30.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.030+0000 7f9c989e3640 1 -- 172.21.15.57:0/1705752104 learned_addr learned my addr 172.21.15.57:0/1705752104 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:30.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.030+0000 7f9c937fe640 1 --2- 172.21.15.57:0/1705752104 >> v2:172.21.15.88:3300/0 conn(0x7f9c940c4a10 0x7f9c9415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:30.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.030+0000 7f9c989e3640 1 -- 172.21.15.57:0/1705752104 >> v2:172.21.15.88:3300/0 conn(0x7f9c940c4a10 msgr2=0x7f9c9415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:30.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.030+0000 7f9c989e3640 1 --2- 172.21.15.57:0/1705752104 >> v2:172.21.15.88:3300/0 conn(0x7f9c940c4a10 0x7f9c9415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.030+0000 7f9c989e3640 1 -- 172.21.15.57:0/1705752104 >> v1:172.21.15.88:6789/0 conn(0x7f9c94153c80 legacy=0x7f9c9415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:30.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.030+0000 7f9c989e3640 1 -- 172.21.15.57:0/1705752104 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9415d620 con 0x7f9c940c42a0 2023-10-30T14:22:30.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.030+0000 7f9c989e3640 1 --2- 172.21.15.57:0/1705752104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c940c42a0 0x7f9c940ca1e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9c9406be50 tx=0x7f9c8c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de8193509fda7d24 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:30.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.030+0000 7f9c92ffd640 1 -- 172.21.15.57:0/1705752104 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c0af400 con 0x7f9c940c42a0 2023-10-30T14:22:30.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.030+0000 7f9c92ffd640 1 -- 172.21.15.57:0/1705752104 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8c0ab050 con 0x7f9c940c42a0 2023-10-30T14:22:30.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c92ffd640 1 -- 172.21.15.57:0/1705752104 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c0af780 con 0x7f9c940c42a0 2023-10-30T14:22:30.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c9a446640 1 -- 172.21.15.57:0/1705752104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c940c42a0 msgr2=0x7f9c940ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:30.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c9a446640 1 --2- 172.21.15.57:0/1705752104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c940c42a0 0x7f9c940ca1e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9c9406be50 tx=0x7f9c8c01f1c0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c9a446640 1 -- 172.21.15.57:0/1705752104 shutdown_connections 2023-10-30T14:22:30.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c9a446640 1 --2- 172.21.15.57:0/1705752104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c940c42a0 0x7f9c940ca1e0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c9a446640 1 --2- 172.21.15.57:0/1705752104 >> v2:172.21.15.88:3300/0 conn(0x7f9c940c4a10 0x7f9c9415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c9a446640 1 -- 172.21.15.57:0/1705752104 >> 172.21.15.57:0/1705752104 conn(0x7f9c940715c0 msgr2=0x7f9c940719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:30.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c9a446640 1 -- 172.21.15.57:0/1705752104 shutdown_connections 2023-10-30T14:22:30.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c9a446640 1 -- 172.21.15.57:0/1705752104 wait complete. 2023-10-30T14:22:30.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c9a446640 1 Processor -- start 2023-10-30T14:22:30.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c9a446640 1 -- start start 2023-10-30T14:22:30.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c9a446640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c940c42a0 0x7f9c941f2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:30.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c9a446640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c940c4a10 0x7f9c940ce000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:30.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c9a446640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c94153c80 0x7f9c940ce540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:30.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c9a446640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c941f2f80 con 0x7f9c940c42a0 2023-10-30T14:22:30.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c9a446640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c941f2ca0 con 0x7f9c940c4a10 2023-10-30T14:22:30.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c9a446640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c940cfa30 con 0x7f9c94153c80 2023-10-30T14:22:30.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c93fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c940c42a0 0x7f9c941f2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:30.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c93fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c940c42a0 0x7f9c941f2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43616/0 (socket says 172.21.15.57:43616) 2023-10-30T14:22:30.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c93fff640 1 -- 172.21.15.57:0/2346213273 learned_addr learned my addr 172.21.15.57:0/2346213273 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:30.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c989e3640 1 --2- 172.21.15.57:0/2346213273 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c94153c80 0x7f9c940ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:30.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.034+0000 7f9c937fe640 1 --2- 172.21.15.57:0/2346213273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c940c4a10 0x7f9c940ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:30.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.038+0000 7f9c93fff640 1 -- 172.21.15.57:0/2346213273 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c94153c80 msgr2=0x7f9c940ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:30.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.038+0000 7f9c93fff640 1 --2- 172.21.15.57:0/2346213273 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c94153c80 0x7f9c940ce540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.038+0000 7f9c93fff640 1 -- 172.21.15.57:0/2346213273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c940c4a10 msgr2=0x7f9c940ce000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:30.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.038+0000 7f9c93fff640 1 --2- 172.21.15.57:0/2346213273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c940c4a10 0x7f9c940ce000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.038+0000 7f9c93fff640 1 -- 172.21.15.57:0/2346213273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c8c009b30 con 0x7f9c940c42a0 2023-10-30T14:22:30.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.038+0000 7f9c93fff640 1 --2- 172.21.15.57:0/2346213273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c940c42a0 0x7f9c941f2240 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f9c8400c3c0 tx=0x7f9c8400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:30.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.038+0000 7f9c90ff9640 1 -- 172.21.15.57:0/2346213273 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c840052d0 con 0x7f9c940c42a0 2023-10-30T14:22:30.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.038+0000 7f9c90ff9640 1 -- 172.21.15.57:0/2346213273 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c84005430 con 0x7f9c940c42a0 2023-10-30T14:22:30.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.038+0000 7f9c90ff9640 1 -- 172.21.15.57:0/2346213273 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c84003840 con 0x7f9c940c42a0 2023-10-30T14:22:30.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.038+0000 7f9c9a446640 1 -- 172.21.15.57:0/2346213273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c940cfd10 con 0x7f9c940c42a0 2023-10-30T14:22:30.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.038+0000 7f9c9a446640 1 -- 172.21.15.57:0/2346213273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c940cdae0 con 0x7f9c940c42a0 2023-10-30T14:22:30.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.038+0000 7f9c9a446640 1 -- 172.21.15.57:0/2346213273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c9404e170 con 0x7f9c940c42a0 2023-10-30T14:22:30.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.038+0000 7f9c90ff9640 1 -- 172.21.15.57:0/2346213273 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9c840039a0 con 0x7f9c940c42a0 2023-10-30T14:22:30.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.038+0000 7f9c90ff9640 1 --2- 172.21.15.57:0/2346213273 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9c800664a0 0x7f9c80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:30.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.038+0000 7f9c90ff9640 1 -- 172.21.15.57:0/2346213273 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f9c84083750 con 0x7f9c940c42a0 2023-10-30T14:22:30.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.038+0000 7f9c937fe640 1 --2- 172.21.15.57:0/2346213273 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9c800664a0 0x7f9c80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:30.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.042+0000 7f9c937fe640 1 --2- 172.21.15.57:0/2346213273 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9c800664a0 0x7f9c80068960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9c940cbb20 tx=0x7f9c88008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:30.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.042+0000 7f9c90ff9640 1 -- 172.21.15.57:0/2346213273 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9c84050880 con 0x7f9c940c42a0 2023-10-30T14:22:30.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.234+0000 7f9c9a446640 1 -- 172.21.15.57:0/2346213273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c940cdcd0 con 0x7f9c940c42a0 2023-10-30T14:22:30.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.234+0000 7f9c90ff9640 1 -- 172.21.15.57:0/2346213273 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5719 (secure 0 0 0) 0x7f9c84053ea0 con 0x7f9c940c42a0 2023-10-30T14:22:30.235 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:22:30.235 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:22:28.838814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5133},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:30.236 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 63 2023-10-30T14:22:30.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.234+0000 7f9c9a446640 1 -- 172.21.15.57:0/2346213273 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9c800664a0 msgr2=0x7f9c80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:30.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.234+0000 7f9c9a446640 1 --2- 172.21.15.57:0/2346213273 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9c800664a0 0x7f9c80068960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9c940cbb20 tx=0x7f9c88008040 comp rx=0 tx=0).stop 2023-10-30T14:22:30.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.234+0000 7f9c9a446640 1 -- 172.21.15.57:0/2346213273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c940c42a0 msgr2=0x7f9c941f2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:30.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.234+0000 7f9c9a446640 1 --2- 172.21.15.57:0/2346213273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c940c42a0 0x7f9c941f2240 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f9c8400c3c0 tx=0x7f9c8400c880 comp rx=0 tx=0).stop 2023-10-30T14:22:30.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.234+0000 7f9c9a446640 1 -- 172.21.15.57:0/2346213273 shutdown_connections 2023-10-30T14:22:30.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.234+0000 7f9c9a446640 1 --2- 172.21.15.57:0/2346213273 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9c800664a0 0x7f9c80068960 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.234+0000 7f9c9a446640 1 --2- 172.21.15.57:0/2346213273 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c94153c80 0x7f9c940ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.234+0000 7f9c9a446640 1 --2- 172.21.15.57:0/2346213273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c940c4a10 0x7f9c940ce000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.234+0000 7f9c9a446640 1 --2- 172.21.15.57:0/2346213273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c940c42a0 0x7f9c941f2240 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.234+0000 7f9c9a446640 1 -- 172.21.15.57:0/2346213273 >> 172.21.15.57:0/2346213273 conn(0x7f9c940715c0 msgr2=0x7f9c94151450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:30.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.234+0000 7f9c9a446640 1 -- 172.21.15.57:0/2346213273 shutdown_connections 2023-10-30T14:22:30.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.234+0000 7f9c9a446640 1 -- 172.21.15.57:0/2346213273 wait complete. 2023-10-30T14:22:30.256 INFO:tasks.mds_thrash.fs.[cephfs]:mds cluster has 2 alive and active, now stable! 2023-10-30T14:22:30.256 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=2) 2023-10-30T14:22:30.256 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T14:22:30.257 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:22:30.271 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T14:22:30.272 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:30.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.414+0000 7f1d2a846640 1 Processor -- start 2023-10-30T14:22:30.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.414+0000 7f1d2a846640 1 -- start start 2023-10-30T14:22:30.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.414+0000 7f1d2a846640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1d240c4a10 0x7f1d2415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:30.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.414+0000 7f1d2a846640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1d240c42a0 0x7f1d240ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:30.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.414+0000 7f1d2a846640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d24159f20 con 0x7f1d24153c80 2023-10-30T14:22:30.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.414+0000 7f1d2a846640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d2415dec0 con 0x7f1d240c4a10 2023-10-30T14:22:30.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.414+0000 7f1d2a846640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d241599a0 con 0x7f1d240c42a0 2023-10-30T14:22:30.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.414+0000 7f1d23fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1d24153c80 0x7f1d2415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:54142/0 (socket says 172.21.15.57:54142) 2023-10-30T14:22:30.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.414+0000 7f1d23fff640 1 -- 172.21.15.57:0/3493384084 learned_addr learned my addr 172.21.15.57:0/3493384084 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:22:30.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.414+0000 7f1d237fe640 1 --2- 172.21.15.57:0/3493384084 >> v2:172.21.15.57:3300/0 conn(0x7f1d240c4a10 0x7f1d2415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:30.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.414+0000 7f1d28de3640 1 --2- 172.21.15.57:0/3493384084 >> v2:172.21.15.88:3300/0 conn(0x7f1d240c42a0 0x7f1d240ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:30.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.414+0000 7f1d237fe640 1 -- 172.21.15.57:0/3493384084 >> v2:172.21.15.88:3300/0 conn(0x7f1d240c42a0 msgr2=0x7f1d240ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:30.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.414+0000 7f1d237fe640 1 --2- 172.21.15.57:0/3493384084 >> v2:172.21.15.88:3300/0 conn(0x7f1d240c42a0 0x7f1d240ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.414+0000 7f1d237fe640 1 -- 172.21.15.57:0/3493384084 >> v1:172.21.15.57:6789/0 conn(0x7f1d24153c80 legacy=0x7f1d2415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:30.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.414+0000 7f1d237fe640 1 -- 172.21.15.57:0/3493384084 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2415d620 con 0x7f1d240c4a10 2023-10-30T14:22:30.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.414+0000 7f1d22ffd640 1 -- 172.21.15.57:0/3493384084 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924137151 0 0) 0x7f1d24159f20 con 0x7f1d24153c80 2023-10-30T14:22:30.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.418+0000 7f1d237fe640 1 --2- 172.21.15.57:0/3493384084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d240c4a10 0x7f1d2415cf90 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1d240b1d40 tx=0x7f1d1801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3d7f070bc90552 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:30.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.418+0000 7f1d22ffd640 1 -- 172.21.15.57:0/3493384084 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d180a8950 con 0x7f1d240c4a10 2023-10-30T14:22:30.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.418+0000 7f1d22ffd640 1 -- 172.21.15.57:0/3493384084 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d180a8ab0 con 0x7f1d240c4a10 2023-10-30T14:22:30.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.418+0000 7f1d22ffd640 1 -- 172.21.15.57:0/3493384084 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d180a8e40 con 0x7f1d240c4a10 2023-10-30T14:22:30.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.418+0000 7f1d2a846640 1 -- 172.21.15.57:0/3493384084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d240c4a10 msgr2=0x7f1d2415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:30.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.418+0000 7f1d2a846640 1 --2- 172.21.15.57:0/3493384084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d240c4a10 0x7f1d2415cf90 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1d240b1d40 tx=0x7f1d1801ec50 comp rx=0 tx=0).stop 2023-10-30T14:22:30.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.418+0000 7f1d2a846640 1 -- 172.21.15.57:0/3493384084 shutdown_connections 2023-10-30T14:22:30.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.418+0000 7f1d2a846640 1 --2- 172.21.15.57:0/3493384084 >> v2:172.21.15.88:3300/0 conn(0x7f1d240c42a0 0x7f1d240ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.418+0000 7f1d2a846640 1 --2- 172.21.15.57:0/3493384084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d240c4a10 0x7f1d2415cf90 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.418+0000 7f1d2a846640 1 -- 172.21.15.57:0/3493384084 >> 172.21.15.57:0/3493384084 conn(0x7f1d240715c0 msgr2=0x7f1d240719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:30.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.418+0000 7f1d2a846640 1 -- 172.21.15.57:0/3493384084 shutdown_connections 2023-10-30T14:22:30.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.418+0000 7f1d2a846640 1 -- 172.21.15.57:0/3493384084 wait complete. 2023-10-30T14:22:30.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d2a846640 1 Processor -- start 2023-10-30T14:22:30.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d2a846640 1 -- start start 2023-10-30T14:22:30.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d2a846640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d240c42a0 0x7f1d2414bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:30.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d2a846640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1d240c4a10 0x7f1d24149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:30.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d2a846640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d24153c80 0x7f1d2415bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:30.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d2a846640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d241f2220 con 0x7f1d24153c80 2023-10-30T14:22:30.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d2a846640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d241f30a0 con 0x7f1d240c42a0 2023-10-30T14:22:30.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d2a846640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2414a470 con 0x7f1d240c4a10 2023-10-30T14:22:30.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d28de3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d24153c80 0x7f1d2415bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:30.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d28de3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d24153c80 0x7f1d2415bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43636/0 (socket says 172.21.15.57:43636) 2023-10-30T14:22:30.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d28de3640 1 -- 172.21.15.57:0/2833389353 learned_addr learned my addr 172.21.15.57:0/2833389353 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:30.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d23fff640 1 --2- 172.21.15.57:0/2833389353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d240c42a0 0x7f1d2414bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:30.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d237fe640 1 --2- 172.21.15.57:0/2833389353 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1d240c4a10 0x7f1d24149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:30.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d28de3640 1 -- 172.21.15.57:0/2833389353 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1d240c4a10 msgr2=0x7f1d24149f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:30.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d28de3640 1 --2- 172.21.15.57:0/2833389353 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1d240c4a10 0x7f1d24149f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d28de3640 1 -- 172.21.15.57:0/2833389353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d240c42a0 msgr2=0x7f1d2414bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:30.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d28de3640 1 --2- 172.21.15.57:0/2833389353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d240c42a0 0x7f1d2414bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d28de3640 1 -- 172.21.15.57:0/2833389353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d18009420 con 0x7f1d24153c80 2023-10-30T14:22:30.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d237fe640 1 --2- 172.21.15.57:0/2833389353 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1d240c4a10 0x7f1d24149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:22:30.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d28de3640 1 --2- 172.21.15.57:0/2833389353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d24153c80 0x7f1d2415bef0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c007e40 tx=0x7f1d1c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:30.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d20ff9640 1 -- 172.21.15.57:0/2833389353 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c014620 con 0x7f1d24153c80 2023-10-30T14:22:30.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d20ff9640 1 -- 172.21.15.57:0/2833389353 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d1c00ee90 con 0x7f1d24153c80 2023-10-30T14:22:30.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d20ff9640 1 -- 172.21.15.57:0/2833389353 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c0149e0 con 0x7f1d24153c80 2023-10-30T14:22:30.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d2a846640 1 -- 172.21.15.57:0/2833389353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d241f2490 con 0x7f1d24153c80 2023-10-30T14:22:30.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.422+0000 7f1d2a846640 1 -- 172.21.15.57:0/2833389353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d241f2a50 con 0x7f1d24153c80 2023-10-30T14:22:30.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.426+0000 7f1d2a846640 1 -- 172.21.15.57:0/2833389353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d2404e170 con 0x7f1d24153c80 2023-10-30T14:22:30.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.426+0000 7f1d20ff9640 1 -- 172.21.15.57:0/2833389353 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1d1c014b40 con 0x7f1d24153c80 2023-10-30T14:22:30.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.426+0000 7f1d20ff9640 1 --2- 172.21.15.57:0/2833389353 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1d100664a0 0x7f1d10068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:30.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.426+0000 7f1d20ff9640 1 -- 172.21.15.57:0/2833389353 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f1d1c0849c0 con 0x7f1d24153c80 2023-10-30T14:22:30.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.426+0000 7f1d23fff640 1 --2- 172.21.15.57:0/2833389353 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1d100664a0 0x7f1d10068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:30.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.426+0000 7f1d23fff640 1 --2- 172.21.15.57:0/2833389353 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1d100664a0 0x7f1d10068960 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1d14000f70 tx=0x7f1d14007df0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:30.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.430+0000 7f1d20ff9640 1 -- 172.21.15.57:0/2833389353 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1d1c051c20 con 0x7f1d24153c80 2023-10-30T14:22:30.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.622+0000 7f1d2a846640 1 -- 172.21.15.57:0/2833389353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d240b5510 con 0x7f1d24153c80 2023-10-30T14:22:30.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.622+0000 7f1d20ff9640 1 -- 172.21.15.57:0/2833389353 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5719 (secure 0 0 0) 0x7f1d1c055240 con 0x7f1d24153c80 2023-10-30T14:22:30.624 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:22:30.624 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:22:28.838814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5133},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:30.625 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 63 2023-10-30T14:22:30.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.626+0000 7f1d2a846640 1 -- 172.21.15.57:0/2833389353 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1d100664a0 msgr2=0x7f1d10068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:30.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.626+0000 7f1d2a846640 1 --2- 172.21.15.57:0/2833389353 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1d100664a0 0x7f1d10068960 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1d14000f70 tx=0x7f1d14007df0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.626+0000 7f1d2a846640 1 -- 172.21.15.57:0/2833389353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d24153c80 msgr2=0x7f1d2415bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:30.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.626+0000 7f1d2a846640 1 --2- 172.21.15.57:0/2833389353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d24153c80 0x7f1d2415bef0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c007e40 tx=0x7f1d1c00e4b0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.626+0000 7f1d2a846640 1 -- 172.21.15.57:0/2833389353 shutdown_connections 2023-10-30T14:22:30.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.626+0000 7f1d2a846640 1 --2- 172.21.15.57:0/2833389353 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1d100664a0 0x7f1d10068960 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.626+0000 7f1d2a846640 1 --2- 172.21.15.57:0/2833389353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d24153c80 0x7f1d2415bef0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.626+0000 7f1d2a846640 1 --2- 172.21.15.57:0/2833389353 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1d240c4a10 0x7f1d24149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.626+0000 7f1d2a846640 1 --2- 172.21.15.57:0/2833389353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d240c42a0 0x7f1d2414bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.626+0000 7f1d2a846640 1 -- 172.21.15.57:0/2833389353 >> 172.21.15.57:0/2833389353 conn(0x7f1d240715c0 msgr2=0x7f1d240caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:30.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.626+0000 7f1d2a846640 1 -- 172.21.15.57:0/2833389353 shutdown_connections 2023-10-30T14:22:30.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.626+0000 7f1d2a846640 1 -- 172.21.15.57:0/2833389353 wait complete. 2023-10-30T14:22:30.645 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-30T14:22:30.645 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:22:30.646 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:30.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.810+0000 7fac8f009640 1 Processor -- start 2023-10-30T14:22:30.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.810+0000 7fac8f009640 1 -- start start 2023-10-30T14:22:30.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.810+0000 7fac8f009640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fac88151c20 0x7fac880c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:30.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.810+0000 7fac8f009640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fac880cb8b0 0x7fac8806b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:30.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.810+0000 7fac8f009640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac880c5030 con 0x7fac88151840 2023-10-30T14:22:30.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.810+0000 7fac8f009640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac88157830 con 0x7fac88151c20 2023-10-30T14:22:30.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.810+0000 7fac8f009640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac88157b10 con 0x7fac880cb8b0 2023-10-30T14:22:30.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.810+0000 7fac8cda5640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fac88151840 0x7fac88072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:54146/0 (socket says 172.21.15.57:54146) 2023-10-30T14:22:30.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.810+0000 7fac8cda5640 1 -- 172.21.15.57:0/3772996467 learned_addr learned my addr 172.21.15.57:0/3772996467 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:22:30.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.810+0000 7fac87fff640 1 --2- 172.21.15.57:0/3772996467 >> v2:172.21.15.57:3300/0 conn(0x7fac88151c20 0x7fac880c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:30.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.810+0000 7fac8d5a6640 1 --2- 172.21.15.57:0/3772996467 >> v2:172.21.15.88:3300/0 conn(0x7fac880cb8b0 0x7fac8806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:30.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.810+0000 7fac877fe640 1 -- 172.21.15.57:0/3772996467 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97596693 0 0) 0x7fac880c5030 con 0x7fac88151840 2023-10-30T14:22:30.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.810+0000 7fac877fe640 1 -- 172.21.15.57:0/3772996467 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac78003530 con 0x7fac88151840 2023-10-30T14:22:30.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.810+0000 7fac87fff640 1 -- 172.21.15.57:0/3772996467 >> v2:172.21.15.88:3300/0 conn(0x7fac880cb8b0 msgr2=0x7fac8806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:30.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.810+0000 7fac87fff640 1 --2- 172.21.15.57:0/3772996467 >> v2:172.21.15.88:3300/0 conn(0x7fac880cb8b0 0x7fac8806b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.810+0000 7fac87fff640 1 -- 172.21.15.57:0/3772996467 >> v1:172.21.15.57:6789/0 conn(0x7fac88151840 legacy=0x7fac88072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:30.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.810+0000 7fac87fff640 1 -- 172.21.15.57:0/3772996467 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac88157df0 con 0x7fac88151c20 2023-10-30T14:22:30.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.818+0000 7fac87fff640 1 --2- 172.21.15.57:0/3772996467 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac88151c20 0x7fac880c49a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fac880c6cb0 tx=0x7fac7401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a03f58f73cd8cdf6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:30.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.818+0000 7fac877fe640 1 -- 172.21.15.57:0/3772996467 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac740a8950 con 0x7fac88151c20 2023-10-30T14:22:30.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.818+0000 7fac877fe640 1 -- 172.21.15.57:0/3772996467 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac740a8ab0 con 0x7fac88151c20 2023-10-30T14:22:30.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.818+0000 7fac877fe640 1 -- 172.21.15.57:0/3772996467 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac740a8e20 con 0x7fac88151c20 2023-10-30T14:22:30.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.818+0000 7fac8f009640 1 -- 172.21.15.57:0/3772996467 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac88151c20 msgr2=0x7fac880c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:30.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.818+0000 7fac8f009640 1 --2- 172.21.15.57:0/3772996467 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac88151c20 0x7fac880c49a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fac880c6cb0 tx=0x7fac7401ec50 comp rx=0 tx=0).stop 2023-10-30T14:22:30.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.818+0000 7fac8f009640 1 -- 172.21.15.57:0/3772996467 shutdown_connections 2023-10-30T14:22:30.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.818+0000 7fac8f009640 1 --2- 172.21.15.57:0/3772996467 >> v2:172.21.15.88:3300/0 conn(0x7fac880cb8b0 0x7fac8806b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.818+0000 7fac8f009640 1 --2- 172.21.15.57:0/3772996467 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac88151c20 0x7fac880c49a0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.818+0000 7fac8f009640 1 -- 172.21.15.57:0/3772996467 >> 172.21.15.57:0/3772996467 conn(0x7fac880cac70 msgr2=0x7fac880cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:30.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.818+0000 7fac8f009640 1 -- 172.21.15.57:0/3772996467 shutdown_connections 2023-10-30T14:22:30.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.818+0000 7fac8f009640 1 -- 172.21.15.57:0/3772996467 wait complete. 2023-10-30T14:22:30.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8f009640 1 Processor -- start 2023-10-30T14:22:30.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8f009640 1 -- start start 2023-10-30T14:22:30.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8f009640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac880cb8b0 0x7fac88145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:30.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8f009640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac88151840 0x7fac880c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:30.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8f009640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fac88151c20 0x7fac881e2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:30.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8f009640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac88070950 con 0x7fac880cb8b0 2023-10-30T14:22:30.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8f009640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac881e25f0 con 0x7fac88151840 2023-10-30T14:22:30.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8f009640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac881e1fe0 con 0x7fac88151c20 2023-10-30T14:22:30.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8cda5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac880cb8b0 0x7fac88145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:30.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8cda5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac880cb8b0 0x7fac88145a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43660/0 (socket says 172.21.15.57:43660) 2023-10-30T14:22:30.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8cda5640 1 -- 172.21.15.57:0/206712154 learned_addr learned my addr 172.21.15.57:0/206712154 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:30.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac87fff640 1 --2- 172.21.15.57:0/206712154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac88151840 0x7fac880c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:30.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8d5a6640 1 --2- 172.21.15.57:0/206712154 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fac88151c20 0x7fac881e2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:30.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8cda5640 1 -- 172.21.15.57:0/206712154 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fac88151c20 msgr2=0x7fac881e2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:30.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8cda5640 1 --2- 172.21.15.57:0/206712154 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fac88151c20 0x7fac881e2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8cda5640 1 -- 172.21.15.57:0/206712154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac88151840 msgr2=0x7fac880c9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:30.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8cda5640 1 --2- 172.21.15.57:0/206712154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac88151840 0x7fac880c9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:30.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8cda5640 1 -- 172.21.15.57:0/206712154 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac74009420 con 0x7fac880cb8b0 2023-10-30T14:22:30.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8d5a6640 1 --2- 172.21.15.57:0/206712154 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fac88151c20 0x7fac881e2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:22:30.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8cda5640 1 --2- 172.21.15.57:0/206712154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac880cb8b0 0x7fac88145a40 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fac7c00c480 tx=0x7fac7c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:30.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac857fa640 1 -- 172.21.15.57:0/206712154 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c005cf0 con 0x7fac880cb8b0 2023-10-30T14:22:30.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac857fa640 1 -- 172.21.15.57:0/206712154 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac7c005e50 con 0x7fac880cb8b0 2023-10-30T14:22:30.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8f009640 1 -- 172.21.15.57:0/206712154 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac880c8af0 con 0x7fac880cb8b0 2023-10-30T14:22:30.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac857fa640 1 -- 172.21.15.57:0/206712154 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c007290 con 0x7fac880cb8b0 2023-10-30T14:22:30.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.822+0000 7fac8f009640 1 -- 172.21.15.57:0/206712154 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac881e2280 con 0x7fac880cb8b0 2023-10-30T14:22:30.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.826+0000 7fac8f009640 1 -- 172.21.15.57:0/206712154 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac8804e170 con 0x7fac880cb8b0 2023-10-30T14:22:30.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.826+0000 7fac857fa640 1 -- 172.21.15.57:0/206712154 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fac7c0073f0 con 0x7fac880cb8b0 2023-10-30T14:22:30.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.826+0000 7fac857fa640 1 --2- 172.21.15.57:0/206712154 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fac6c0664a0 0x7fac6c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:30.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.826+0000 7fac857fa640 1 -- 172.21.15.57:0/206712154 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fac7c083690 con 0x7fac880cb8b0 2023-10-30T14:22:30.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.826+0000 7fac87fff640 1 --2- 172.21.15.57:0/206712154 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fac6c0664a0 0x7fac6c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:30.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.830+0000 7fac87fff640 1 --2- 172.21.15.57:0/206712154 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fac6c0664a0 0x7fac6c068960 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fac740b2000 tx=0x7fac740aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:30.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:30.830+0000 7fac857fa640 1 -- 172.21.15.57:0/206712154 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fac7c0508e0 con 0x7fac880cb8b0 2023-10-30T14:22:31.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:31.022+0000 7fac8f009640 1 -- 172.21.15.57:0/206712154 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac88155060 con 0x7fac880cb8b0 2023-10-30T14:22:31.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:31.022+0000 7fac857fa640 1 -- 172.21.15.57:0/206712154 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+6510 (secure 0 0 0) 0x7fac7c017090 con 0x7fac880cb8b0 2023-10-30T14:22:31.024 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:22:31.024 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":5644,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1331276377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1331276377},{"type":"v1","addr":"172.21.15.88:6839","nonce":1331276377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:22:28.838814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5133},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:31.025 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 64 2023-10-30T14:22:31.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:31.022+0000 7fac8f009640 1 -- 172.21.15.57:0/206712154 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fac6c0664a0 msgr2=0x7fac6c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:31.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:31.022+0000 7fac8f009640 1 --2- 172.21.15.57:0/206712154 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fac6c0664a0 0x7fac6c068960 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fac740b2000 tx=0x7fac740aa070 comp rx=0 tx=0).stop 2023-10-30T14:22:31.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:31.022+0000 7fac8f009640 1 -- 172.21.15.57:0/206712154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac880cb8b0 msgr2=0x7fac88145a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:31.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:31.022+0000 7fac8f009640 1 --2- 172.21.15.57:0/206712154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac880cb8b0 0x7fac88145a40 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fac7c00c480 tx=0x7fac7c00c940 comp rx=0 tx=0).stop 2023-10-30T14:22:31.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:31.026+0000 7fac8f009640 1 -- 172.21.15.57:0/206712154 shutdown_connections 2023-10-30T14:22:31.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:31.026+0000 7fac8f009640 1 --2- 172.21.15.57:0/206712154 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fac6c0664a0 0x7fac6c068960 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:31.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:31.026+0000 7fac8f009640 1 --2- 172.21.15.57:0/206712154 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fac88151c20 0x7fac881e2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:31.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:31.026+0000 7fac8f009640 1 --2- 172.21.15.57:0/206712154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac88151840 0x7fac880c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:31.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:31.026+0000 7fac8f009640 1 --2- 172.21.15.57:0/206712154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac880cb8b0 0x7fac88145a40 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:31.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:31.026+0000 7fac8f009640 1 -- 172.21.15.57:0/206712154 >> 172.21.15.57:0/206712154 conn(0x7fac880cac70 msgr2=0x7fac880733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:31.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:31.026+0000 7fac8f009640 1 -- 172.21.15.57:0/206712154 shutdown_connections 2023-10-30T14:22:31.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:31.026+0000 7fac8f009640 1 -- 172.21.15.57:0/206712154 wait complete. 2023-10-30T14:22:31.045 INFO:tasks.mds_thrash.fs.[cephfs]:cluster has 2 actives (max_mds is 2), no MDS can replace rank 2 2023-10-30T14:22:31.046 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.f 2023-10-30T14:22:31.121 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:22:31.126 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:22:35.392 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/31' 2023-10-30T14:22:35.393 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/36' 2023-10-30T14:22:35.393 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/20' 2023-10-30T14:22:35.393 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/39' 2023-10-30T14:22:35.393 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/34' 2023-10-30T14:22:35.393 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/38' 2023-10-30T14:22:35.394 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/28' 2023-10-30T14:22:35.394 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/24' 2023-10-30T14:22:35.394 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/29' 2023-10-30T14:22:35.394 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/32' 2023-10-30T14:22:35.394 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/22' 2023-10-30T14:22:35.395 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/25' 2023-10-30T14:22:35.395 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/30' 2023-10-30T14:22:35.395 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/33' 2023-10-30T14:22:35.395 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/23' 2023-10-30T14:22:35.395 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/35' 2023-10-30T14:22:35.396 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/26' 2023-10-30T14:22:35.396 INFO:tasks.workunit.client.0.smithi057.stdout:removed './297/37' 2023-10-30T14:22:35.396 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './297' 2023-10-30T14:22:35.396 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/21' 2023-10-30T14:22:35.396 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/27' 2023-10-30T14:22:35.396 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/31' 2023-10-30T14:22:35.397 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/36' 2023-10-30T14:22:35.397 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/20' 2023-10-30T14:22:35.397 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/39' 2023-10-30T14:22:35.397 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/34' 2023-10-30T14:22:35.397 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/38' 2023-10-30T14:22:35.398 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/28' 2023-10-30T14:22:35.398 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/24' 2023-10-30T14:22:35.398 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/29' 2023-10-30T14:22:35.398 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/32' 2023-10-30T14:22:35.398 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/22' 2023-10-30T14:22:35.398 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/25' 2023-10-30T14:22:35.399 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/30' 2023-10-30T14:22:35.399 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/33' 2023-10-30T14:22:35.399 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/23' 2023-10-30T14:22:35.399 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/35' 2023-10-30T14:22:35.399 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/26' 2023-10-30T14:22:35.400 INFO:tasks.workunit.client.0.smithi057.stdout:removed './298/37' 2023-10-30T14:22:35.400 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './298' 2023-10-30T14:22:35.400 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/21' 2023-10-30T14:22:35.400 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/27' 2023-10-30T14:22:35.400 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/31' 2023-10-30T14:22:35.400 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/36' 2023-10-30T14:22:35.401 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/20' 2023-10-30T14:22:35.401 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/39' 2023-10-30T14:22:35.401 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/34' 2023-10-30T14:22:35.401 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/38' 2023-10-30T14:22:35.401 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/28' 2023-10-30T14:22:35.402 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/24' 2023-10-30T14:22:35.402 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/29' 2023-10-30T14:22:35.402 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/32' 2023-10-30T14:22:35.402 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/22' 2023-10-30T14:22:35.402 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/25' 2023-10-30T14:22:35.402 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/30' 2023-10-30T14:22:35.403 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/33' 2023-10-30T14:22:35.403 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/23' 2023-10-30T14:22:35.403 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/35' 2023-10-30T14:22:35.403 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/26' 2023-10-30T14:22:35.403 INFO:tasks.workunit.client.0.smithi057.stdout:removed './299/37' 2023-10-30T14:22:35.404 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './299' 2023-10-30T14:22:35.404 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/166/19' 2023-10-30T14:22:35.404 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/166/15' 2023-10-30T14:22:35.404 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/166/20' 2023-10-30T14:22:35.404 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/166/12' 2023-10-30T14:22:35.404 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/166/10' 2023-10-30T14:22:35.405 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/166/11' 2023-10-30T14:22:35.405 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/166/16' 2023-10-30T14:22:35.405 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/166/14' 2023-10-30T14:22:35.405 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/166/18' 2023-10-30T14:22:35.405 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/166/13' 2023-10-30T14:22:35.405 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/166/17' 2023-10-30T14:22:35.406 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/166' 2023-10-30T14:22:35.406 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/147/19' 2023-10-30T14:22:35.406 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/147/15' 2023-10-30T14:22:35.406 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/147/20' 2023-10-30T14:22:35.406 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/147/12' 2023-10-30T14:22:35.407 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/147/10' 2023-10-30T14:22:35.407 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/147/11' 2023-10-30T14:22:35.407 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/147/16' 2023-10-30T14:22:35.407 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/147/14' 2023-10-30T14:22:35.407 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/147/18' 2023-10-30T14:22:35.408 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/147/13' 2023-10-30T14:22:35.408 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/147/17' 2023-10-30T14:22:35.408 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/147' 2023-10-30T14:22:35.408 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/151/19' 2023-10-30T14:22:35.408 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/151/15' 2023-10-30T14:22:35.408 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/151/20' 2023-10-30T14:22:35.409 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/151/12' 2023-10-30T14:22:35.409 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/151/10' 2023-10-30T14:22:35.409 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/151/11' 2023-10-30T14:22:35.409 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/151/16' 2023-10-30T14:22:35.409 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/151/14' 2023-10-30T14:22:35.409 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/151/18' 2023-10-30T14:22:35.410 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/151/13' 2023-10-30T14:22:35.410 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/151/17' 2023-10-30T14:22:35.410 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/151' 2023-10-30T14:22:35.410 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/170/19' 2023-10-30T14:22:35.410 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/170/15' 2023-10-30T14:22:35.411 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/170/20' 2023-10-30T14:22:35.411 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/170/12' 2023-10-30T14:22:35.411 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/170/10' 2023-10-30T14:22:35.411 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/170/11' 2023-10-30T14:22:35.411 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/170/16' 2023-10-30T14:22:35.412 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/170/14' 2023-10-30T14:22:35.412 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/170/18' 2023-10-30T14:22:35.412 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/170/13' 2023-10-30T14:22:35.412 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/170/17' 2023-10-30T14:22:35.412 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/170' 2023-10-30T14:22:35.412 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/167/19' 2023-10-30T14:22:35.413 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/167/15' 2023-10-30T14:22:35.413 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/167/20' 2023-10-30T14:22:35.413 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/167/12' 2023-10-30T14:22:35.413 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/167/10' 2023-10-30T14:22:35.413 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/167/11' 2023-10-30T14:22:35.413 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/167/16' 2023-10-30T14:22:35.414 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/167/14' 2023-10-30T14:22:35.414 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/167/18' 2023-10-30T14:22:35.414 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/167/13' 2023-10-30T14:22:35.414 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/167/17' 2023-10-30T14:22:35.414 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/167' 2023-10-30T14:22:35.415 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/140/19' 2023-10-30T14:22:35.415 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/140/15' 2023-10-30T14:22:35.415 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/140/20' 2023-10-30T14:22:35.415 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/140/12' 2023-10-30T14:22:35.415 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/140/10' 2023-10-30T14:22:35.416 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/140/11' 2023-10-30T14:22:35.416 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/140/16' 2023-10-30T14:22:35.416 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/140/14' 2023-10-30T14:22:35.416 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/140/18' 2023-10-30T14:22:35.416 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/140/13' 2023-10-30T14:22:35.416 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/140/17' 2023-10-30T14:22:35.417 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/140' 2023-10-30T14:22:35.417 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/142/19' 2023-10-30T14:22:35.417 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/142/15' 2023-10-30T14:22:35.417 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/142/20' 2023-10-30T14:22:35.417 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/142/12' 2023-10-30T14:22:35.418 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/142/10' 2023-10-30T14:22:38.186 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/142/11' 2023-10-30T14:22:38.186 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/142/16' 2023-10-30T14:22:38.186 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/142/14' 2023-10-30T14:22:38.186 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/142/18' 2023-10-30T14:22:38.187 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/142/13' 2023-10-30T14:22:38.187 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/142/17' 2023-10-30T14:22:38.187 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/142' 2023-10-30T14:22:38.187 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/150/19' 2023-10-30T14:22:38.187 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/150/15' 2023-10-30T14:22:38.188 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/150/20' 2023-10-30T14:22:38.188 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/150/12' 2023-10-30T14:22:38.188 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/150/10' 2023-10-30T14:22:38.188 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/150/11' 2023-10-30T14:22:38.188 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/150/16' 2023-10-30T14:22:38.188 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/150/14' 2023-10-30T14:22:38.189 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/150/18' 2023-10-30T14:22:38.189 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/150/13' 2023-10-30T14:22:38.189 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/150/17' 2023-10-30T14:22:38.189 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/150' 2023-10-30T14:22:38.189 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/105/19' 2023-10-30T14:22:38.190 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/105/15' 2023-10-30T14:22:38.190 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/105/20' 2023-10-30T14:22:38.190 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/105/12' 2023-10-30T14:22:38.190 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/105/10' 2023-10-30T14:22:38.190 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/105/11' 2023-10-30T14:22:38.190 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/105/16' 2023-10-30T14:22:38.191 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/105/14' 2023-10-30T14:22:38.191 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/105/18' 2023-10-30T14:22:38.191 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/105/13' 2023-10-30T14:22:38.191 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/105/17' 2023-10-30T14:22:38.191 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/105' 2023-10-30T14:22:38.192 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/149/19' 2023-10-30T14:22:38.192 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/149/15' 2023-10-30T14:22:38.192 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/149/20' 2023-10-30T14:22:38.192 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/149/12' 2023-10-30T14:22:38.192 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/149/10' 2023-10-30T14:22:38.192 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/149/11' 2023-10-30T14:22:38.193 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/149/16' 2023-10-30T14:22:38.193 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/149/14' 2023-10-30T14:22:38.193 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/149/18' 2023-10-30T14:22:38.193 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/149/13' 2023-10-30T14:22:38.194 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/149/17' 2023-10-30T14:22:38.194 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/149' 2023-10-30T14:22:38.194 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/109/19' 2023-10-30T14:22:38.194 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/109/15' 2023-10-30T14:22:38.194 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/109/20' 2023-10-30T14:22:38.195 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/109/12' 2023-10-30T14:22:38.195 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/109/10' 2023-10-30T14:22:38.195 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/109/11' 2023-10-30T14:22:38.195 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/109/16' 2023-10-30T14:22:38.195 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/109/14' 2023-10-30T14:22:38.196 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/109/18' 2023-10-30T14:22:38.196 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/109/13' 2023-10-30T14:22:38.196 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/109/17' 2023-10-30T14:22:38.196 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/109' 2023-10-30T14:22:38.196 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/158/19' 2023-10-30T14:22:38.197 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/158/15' 2023-10-30T14:22:38.197 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/158/20' 2023-10-30T14:22:38.197 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/158/12' 2023-10-30T14:22:38.197 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/158/10' 2023-10-30T14:22:38.197 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/158/11' 2023-10-30T14:22:38.197 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/158/16' 2023-10-30T14:22:38.198 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/158/14' 2023-10-30T14:22:38.198 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/158/18' 2023-10-30T14:22:38.198 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/158/13' 2023-10-30T14:22:38.198 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/158/17' 2023-10-30T14:22:38.198 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/158' 2023-10-30T14:22:38.199 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/106/19' 2023-10-30T14:22:38.199 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/106/15' 2023-10-30T14:22:38.199 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/106/20' 2023-10-30T14:22:38.199 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/106/12' 2023-10-30T14:22:38.199 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/106/10' 2023-10-30T14:22:38.199 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/106/11' 2023-10-30T14:22:38.200 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/106/16' 2023-10-30T14:22:38.200 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/106/14' 2023-10-30T14:22:38.200 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/106/18' 2023-10-30T14:22:38.200 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/106/13' 2023-10-30T14:22:38.201 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/106/17' 2023-10-30T14:22:38.201 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/106' 2023-10-30T14:22:38.201 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/189/19' 2023-10-30T14:22:38.201 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/189/15' 2023-10-30T14:22:38.201 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/189/20' 2023-10-30T14:22:38.201 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/189/12' 2023-10-30T14:22:38.202 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/189/10' 2023-10-30T14:22:38.202 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/189/11' 2023-10-30T14:22:38.202 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/189/16' 2023-10-30T14:22:38.202 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/189/14' 2023-10-30T14:22:38.203 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/189/18' 2023-10-30T14:22:38.203 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/189/13' 2023-10-30T14:22:38.203 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/189/17' 2023-10-30T14:22:38.203 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/189' 2023-10-30T14:22:38.203 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/133/19' 2023-10-30T14:22:38.203 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/133/15' 2023-10-30T14:22:38.204 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/133/20' 2023-10-30T14:22:38.204 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/133/12' 2023-10-30T14:22:38.204 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/133/10' 2023-10-30T14:22:38.204 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/133/11' 2023-10-30T14:22:38.205 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/133/16' 2023-10-30T14:22:38.205 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/133/14' 2023-10-30T14:22:38.205 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/133/18' 2023-10-30T14:22:38.205 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/133/13' 2023-10-30T14:22:38.206 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/133/17' 2023-10-30T14:22:38.206 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/133' 2023-10-30T14:22:38.206 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/173/19' 2023-10-30T14:22:38.206 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/173/15' 2023-10-30T14:22:38.206 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/173/20' 2023-10-30T14:22:38.207 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/173/12' 2023-10-30T14:22:38.207 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/173/10' 2023-10-30T14:22:38.207 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/173/11' 2023-10-30T14:22:39.047 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-30T14:22:39.047 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T14:22:39.048 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T14:22:39.051 INFO:tasks.ceph.mds.f:Started 2023-10-30T14:22:39.051 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:39.127 INFO:tasks.ceph.mds.f.smithi088.stdout:starting mds.f at 2023-10-30T14:22:39.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.194+0000 7f6a2189e640 1 Processor -- start 2023-10-30T14:22:39.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.194+0000 7f6a2189e640 1 -- start start 2023-10-30T14:22:39.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.194+0000 7f6a2189e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a1c150830 0x7f6a1c06b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:39.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.194+0000 7f6a2189e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6a1c0cb7d0 0x7f6a1c0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:39.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.194+0000 7f6a2189e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6a1c0c3e40 0x7f6a1c070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:39.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.194+0000 7f6a2189e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a1c0c2310 con 0x7f6a1c0cb7d0 2023-10-30T14:22:39.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.194+0000 7f6a2189e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a1c156710 con 0x7f6a1c0c3e40 2023-10-30T14:22:39.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.194+0000 7f6a2189e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1c0c4360 con 0x7f6a1c150830 2023-10-30T14:22:39.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.194+0000 7f6a1a7fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6a1c0cb7d0 0x7f6a1c0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:39.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.194+0000 7f6a1a7fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6a1c0cb7d0 0x7f6a1c0c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54504/0 (socket says 172.21.15.57:54504) 2023-10-30T14:22:39.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.194+0000 7f6a1a7fc640 1 -- 172.21.15.57:0/405844861 learned_addr learned my addr 172.21.15.57:0/405844861 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:39.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.194+0000 7f6a1b7fe640 1 --2- 172.21.15.57:0/405844861 >> v2:172.21.15.88:3300/0 conn(0x7f6a1c0c3e40 0x7f6a1c070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:39.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.194+0000 7f6a1affd640 1 --2- 172.21.15.57:0/405844861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a1c150830 0x7f6a1c06b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:39.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.198+0000 7f6a1a7fc640 1 -- 172.21.15.57:0/405844861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a1c150830 msgr2=0x7f6a1c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:39.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.198+0000 7f6a1a7fc640 1 --2- 172.21.15.57:0/405844861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a1c150830 0x7f6a1c06b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:39.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.198+0000 7f6a1a7fc640 1 -- 172.21.15.57:0/405844861 >> v2:172.21.15.88:3300/0 conn(0x7f6a1c0c3e40 msgr2=0x7f6a1c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:39.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.198+0000 7f6a1a7fc640 1 --2- 172.21.15.57:0/405844861 >> v2:172.21.15.88:3300/0 conn(0x7f6a1c0c3e40 0x7f6a1c070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:39.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.198+0000 7f6a1a7fc640 1 -- 172.21.15.57:0/405844861 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a1c1569b0 con 0x7f6a1c0cb7d0 2023-10-30T14:22:39.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.198+0000 7f6a1a7fc640 1 --2- 172.21.15.57:0/405844861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c0cb7d0 0x7f6a1c0c3900 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c06fc60 tx=0x7f6a1001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd4dbea4410abc8b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:39.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.198+0000 7f6a19ffb640 1 -- 172.21.15.57:0/405844861 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a100a89b0 con 0x7f6a1c0cb7d0 2023-10-30T14:22:39.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.198+0000 7f6a19ffb640 1 -- 172.21.15.57:0/405844861 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a100a8b10 con 0x7f6a1c0cb7d0 2023-10-30T14:22:39.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.198+0000 7f6a19ffb640 1 -- 172.21.15.57:0/405844861 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a100a8ea0 con 0x7f6a1c0cb7d0 2023-10-30T14:22:39.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.198+0000 7f6a2189e640 1 -- 172.21.15.57:0/405844861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c0cb7d0 msgr2=0x7f6a1c0c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:39.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.198+0000 7f6a2189e640 1 --2- 172.21.15.57:0/405844861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c0cb7d0 0x7f6a1c0c3900 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c06fc60 tx=0x7f6a1001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:22:39.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.198+0000 7f6a2189e640 1 -- 172.21.15.57:0/405844861 shutdown_connections 2023-10-30T14:22:39.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.198+0000 7f6a2189e640 1 --2- 172.21.15.57:0/405844861 >> v2:172.21.15.88:3300/0 conn(0x7f6a1c0c3e40 0x7f6a1c070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:39.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.198+0000 7f6a2189e640 1 --2- 172.21.15.57:0/405844861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c0cb7d0 0x7f6a1c0c3900 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:39.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.198+0000 7f6a2189e640 1 --2- 172.21.15.57:0/405844861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a1c150830 0x7f6a1c06b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:39.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.198+0000 7f6a2189e640 1 -- 172.21.15.57:0/405844861 >> 172.21.15.57:0/405844861 conn(0x7f6a1c0cac70 msgr2=0x7f6a1c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:39.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.198+0000 7f6a2189e640 1 -- 172.21.15.57:0/405844861 shutdown_connections 2023-10-30T14:22:39.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.198+0000 7f6a2189e640 1 -- 172.21.15.57:0/405844861 wait complete. 2023-10-30T14:22:39.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a2189e640 1 Processor -- start 2023-10-30T14:22:39.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a2189e640 1 -- start start 2023-10-30T14:22:39.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a2189e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a1c0c3e40 0x7f6a1c06bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:39.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a2189e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c0cb7d0 0x7f6a1c1eab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:39.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a2189e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a1c150830 0x7f6a1c0c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:39.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a2189e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1c147300 con 0x7f6a1c0cb7d0 2023-10-30T14:22:39.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a2189e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1c1547b0 con 0x7f6a1c150830 2023-10-30T14:22:39.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a2189e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1c1ee0c0 con 0x7f6a1c0c3e40 2023-10-30T14:22:39.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a1a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c0cb7d0 0x7f6a1c1eab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:39.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a1a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c0cb7d0 0x7f6a1c1eab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54520/0 (socket says 172.21.15.57:54520) 2023-10-30T14:22:39.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a1a7fc640 1 -- 172.21.15.57:0/3533566093 learned_addr learned my addr 172.21.15.57:0/3533566093 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:22:39.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a1affd640 1 --2- 172.21.15.57:0/3533566093 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a1c0c3e40 0x7f6a1c06bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:39.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a1b7fe640 1 --2- 172.21.15.57:0/3533566093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a1c150830 0x7f6a1c0c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:39.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a1a7fc640 1 -- 172.21.15.57:0/3533566093 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a1c0c3e40 msgr2=0x7f6a1c06bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:39.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a1a7fc640 1 --2- 172.21.15.57:0/3533566093 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a1c0c3e40 0x7f6a1c06bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:39.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a1a7fc640 1 -- 172.21.15.57:0/3533566093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a1c150830 msgr2=0x7f6a1c0c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:39.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a1a7fc640 1 --2- 172.21.15.57:0/3533566093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a1c150830 0x7f6a1c0c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:39.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a1a7fc640 1 -- 172.21.15.57:0/3533566093 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a100094d0 con 0x7f6a1c0cb7d0 2023-10-30T14:22:39.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a1a7fc640 1 --2- 172.21.15.57:0/3533566093 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c0cb7d0 0x7f6a1c1eab50 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f6a1001d3a0 tx=0x7f6a1001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:39.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a03fff640 1 -- 172.21.15.57:0/3533566093 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a100b0d40 con 0x7f6a1c0cb7d0 2023-10-30T14:22:39.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a03fff640 1 -- 172.21.15.57:0/3533566093 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a100bb460 con 0x7f6a1c0cb7d0 2023-10-30T14:22:39.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a03fff640 1 -- 172.21.15.57:0/3533566093 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a100b0ea0 con 0x7f6a1c0cb7d0 2023-10-30T14:22:39.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a2189e640 1 -- 172.21.15.57:0/3533566093 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a1c154a50 con 0x7f6a1c0cb7d0 2023-10-30T14:22:39.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.202+0000 7f6a2189e640 1 -- 172.21.15.57:0/3533566093 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a1c1ee640 con 0x7f6a1c0cb7d0 2023-10-30T14:22:39.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.206+0000 7f6a2189e640 1 -- 172.21.15.57:0/3533566093 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a1c066510 con 0x7f6a1c0cb7d0 2023-10-30T14:22:39.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.206+0000 7f6a03fff640 1 -- 172.21.15.57:0/3533566093 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6a100b83f0 con 0x7f6a1c0cb7d0 2023-10-30T14:22:39.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.206+0000 7f6a03fff640 1 --2- 172.21.15.57:0/3533566093 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6a080664a0 0x7f6a08068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:39.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.206+0000 7f6a03fff640 1 -- 172.21.15.57:0/3533566093 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f6a101219e0 con 0x7f6a1c0cb7d0 2023-10-30T14:22:39.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.206+0000 7f6a1affd640 1 --2- 172.21.15.57:0/3533566093 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6a080664a0 0x7f6a08068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:39.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.206+0000 7f6a1affd640 1 --2- 172.21.15.57:0/3533566093 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6a080664a0 0x7f6a08068960 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c009af0 tx=0x7f6a0c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:39.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.210+0000 7f6a03fff640 1 -- 172.21.15.57:0/3533566093 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6a100eeb10 con 0x7f6a1c0cb7d0 2023-10-30T14:22:39.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.402+0000 7f6a2189e640 1 -- 172.21.15.57:0/3533566093 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a1c04e170 con 0x7f6a1c0cb7d0 2023-10-30T14:22:39.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.402+0000 7f6a03fff640 1 -- 172.21.15.57:0/3533566093 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f6a100b5020 con 0x7f6a1c0cb7d0 2023-10-30T14:22:39.404 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:22:39.404 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":5644,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1331276377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1331276377},{"type":"v1","addr":"172.21.15.88:6839","nonce":1331276377}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:22:37.974156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5133},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:39.405 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 65 2023-10-30T14:22:39.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.406+0000 7f6a2189e640 1 -- 172.21.15.57:0/3533566093 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6a080664a0 msgr2=0x7f6a08068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:39.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.406+0000 7f6a2189e640 1 --2- 172.21.15.57:0/3533566093 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6a080664a0 0x7f6a08068960 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c009af0 tx=0x7f6a0c008040 comp rx=0 tx=0).stop 2023-10-30T14:22:39.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.406+0000 7f6a2189e640 1 -- 172.21.15.57:0/3533566093 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c0cb7d0 msgr2=0x7f6a1c1eab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:39.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.406+0000 7f6a2189e640 1 --2- 172.21.15.57:0/3533566093 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c0cb7d0 0x7f6a1c1eab50 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f6a1001d3a0 tx=0x7f6a1001ba90 comp rx=0 tx=0).stop 2023-10-30T14:22:39.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.406+0000 7f6a2189e640 1 -- 172.21.15.57:0/3533566093 shutdown_connections 2023-10-30T14:22:39.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.406+0000 7f6a2189e640 1 --2- 172.21.15.57:0/3533566093 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6a080664a0 0x7f6a08068960 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:39.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.406+0000 7f6a2189e640 1 --2- 172.21.15.57:0/3533566093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a1c150830 0x7f6a1c0c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:39.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.406+0000 7f6a2189e640 1 --2- 172.21.15.57:0/3533566093 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c0cb7d0 0x7f6a1c1eab50 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:39.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.406+0000 7f6a2189e640 1 --2- 172.21.15.57:0/3533566093 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a1c0c3e40 0x7f6a1c06bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:39.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.406+0000 7f6a2189e640 1 -- 172.21.15.57:0/3533566093 >> 172.21.15.57:0/3533566093 conn(0x7f6a1c0cac70 msgr2=0x7f6a1c070520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:39.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.406+0000 7f6a2189e640 1 -- 172.21.15.57:0/3533566093 shutdown_connections 2023-10-30T14:22:39.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:22:39.406+0000 7f6a2189e640 1 -- 172.21.15.57:0/3533566093 wait complete. 2023-10-30T14:22:39.425 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-30T14:22:39.425 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 60 secs before thrashing 2023-10-30T14:22:39.656 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/173/16' 2023-10-30T14:22:39.657 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/173/14' 2023-10-30T14:22:39.657 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/173/18' 2023-10-30T14:22:39.657 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/173/13' 2023-10-30T14:22:39.657 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/173/17' 2023-10-30T14:22:39.657 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/173' 2023-10-30T14:22:39.658 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/174/19' 2023-10-30T14:22:39.658 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/174/15' 2023-10-30T14:22:39.658 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/174/20' 2023-10-30T14:22:39.658 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/174/12' 2023-10-30T14:22:39.659 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/174/10' 2023-10-30T14:22:39.659 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/174/11' 2023-10-30T14:22:39.659 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/174/16' 2023-10-30T14:22:39.659 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/174/14' 2023-10-30T14:22:39.659 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/174/18' 2023-10-30T14:22:39.660 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/174/13' 2023-10-30T14:22:39.660 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/174/17' 2023-10-30T14:22:39.660 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/174' 2023-10-30T14:22:39.660 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/155/19' 2023-10-30T14:22:39.660 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/155/15' 2023-10-30T14:22:39.660 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/155/20' 2023-10-30T14:22:39.661 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/155/12' 2023-10-30T14:22:39.661 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/155/10' 2023-10-30T14:22:39.661 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/155/11' 2023-10-30T14:22:39.661 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/155/16' 2023-10-30T14:22:39.661 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/155/14' 2023-10-30T14:22:39.662 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/155/18' 2023-10-30T14:22:39.662 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/155/13' 2023-10-30T14:22:39.662 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/155/17' 2023-10-30T14:22:39.662 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/155' 2023-10-30T14:22:39.662 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/114/19' 2023-10-30T14:22:39.663 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/114/15' 2023-10-30T14:22:39.663 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/114/20' 2023-10-30T14:22:39.663 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/114/12' 2023-10-30T14:22:39.663 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/114/10' 2023-10-30T14:22:39.663 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/114/11' 2023-10-30T14:22:39.663 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/114/16' 2023-10-30T14:22:39.664 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/114/14' 2023-10-30T14:22:39.664 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/114/18' 2023-10-30T14:22:39.664 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/114/13' 2023-10-30T14:22:39.664 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/114/17' 2023-10-30T14:22:39.664 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/114' 2023-10-30T14:22:39.665 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/159/19' 2023-10-30T14:22:39.665 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/159/15' 2023-10-30T14:22:39.665 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/159/20' 2023-10-30T14:22:39.665 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/159/12' 2023-10-30T14:22:39.665 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/159/10' 2023-10-30T14:22:39.666 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/159/11' 2023-10-30T14:22:39.666 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/159/16' 2023-10-30T14:22:39.666 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/159/14' 2023-10-30T14:22:39.666 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/159/18' 2023-10-30T14:22:39.666 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/159/13' 2023-10-30T14:22:39.666 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/159/17' 2023-10-30T14:22:39.667 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/159' 2023-10-30T14:22:39.667 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/187/19' 2023-10-30T14:22:39.667 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/187/15' 2023-10-30T14:22:39.667 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/187/20' 2023-10-30T14:22:39.668 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/187/12' 2023-10-30T14:22:39.668 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/187/10' 2023-10-30T14:22:39.668 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/187/11' 2023-10-30T14:22:39.668 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/187/16' 2023-10-30T14:22:39.668 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/187/14' 2023-10-30T14:22:39.668 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/187/18' 2023-10-30T14:22:39.669 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/187/13' 2023-10-30T14:22:39.669 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/187/17' 2023-10-30T14:22:39.669 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/187' 2023-10-30T14:22:39.669 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/185/19' 2023-10-30T14:22:39.669 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/185/15' 2023-10-30T14:22:39.670 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/185/20' 2023-10-30T14:22:39.670 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/185/12' 2023-10-30T14:22:39.670 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/185/10' 2023-10-30T14:22:39.670 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/185/11' 2023-10-30T14:22:39.670 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/185/16' 2023-10-30T14:22:39.671 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/185/14' 2023-10-30T14:22:39.671 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/185/18' 2023-10-30T14:22:39.671 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/185/13' 2023-10-30T14:22:39.671 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/185/17' 2023-10-30T14:22:39.671 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/185' 2023-10-30T14:22:39.671 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/120/19' 2023-10-30T14:22:39.672 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/120/15' 2023-10-30T14:22:39.672 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/120/20' 2023-10-30T14:22:39.672 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/120/12' 2023-10-30T14:22:39.672 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/120/10' 2023-10-30T14:22:39.673 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/120/11' 2023-10-30T14:22:39.673 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/120/16' 2023-10-30T14:22:39.673 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/120/14' 2023-10-30T14:22:39.673 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/120/18' 2023-10-30T14:22:39.673 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/120/13' 2023-10-30T14:22:39.673 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/120/17' 2023-10-30T14:22:39.674 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/120' 2023-10-30T14:22:39.674 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/110/19' 2023-10-30T14:22:39.674 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/110/15' 2023-10-30T14:22:39.674 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/110/20' 2023-10-30T14:22:39.675 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/110/12' 2023-10-30T14:22:39.675 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/110/10' 2023-10-30T14:22:39.675 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/110/11' 2023-10-30T14:22:39.675 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/110/16' 2023-10-30T14:22:39.675 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/110/14' 2023-10-30T14:22:39.676 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/110/18' 2023-10-30T14:22:39.676 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/110/13' 2023-10-30T14:22:39.676 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/110/17' 2023-10-30T14:22:39.676 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/110' 2023-10-30T14:22:39.677 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/137/19' 2023-10-30T14:22:39.677 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/137/15' 2023-10-30T14:22:39.677 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/137/20' 2023-10-30T14:22:39.677 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/137/12' 2023-10-30T14:22:39.677 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/137/10' 2023-10-30T14:22:39.678 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/137/11' 2023-10-30T14:22:39.678 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/137/16' 2023-10-30T14:22:41.013 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/137/14' 2023-10-30T14:22:41.013 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/137/18' 2023-10-30T14:22:41.014 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/137/13' 2023-10-30T14:22:41.014 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/137/17' 2023-10-30T14:22:41.014 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/137' 2023-10-30T14:22:41.014 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/198/19' 2023-10-30T14:22:41.014 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/198/15' 2023-10-30T14:22:41.015 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/198/20' 2023-10-30T14:22:41.015 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/198/12' 2023-10-30T14:22:41.015 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/198/10' 2023-10-30T14:22:41.015 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/198/11' 2023-10-30T14:22:41.015 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/198/16' 2023-10-30T14:22:41.016 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/198/14' 2023-10-30T14:22:41.016 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/198/18' 2023-10-30T14:22:41.016 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/198/13' 2023-10-30T14:22:41.016 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/198/17' 2023-10-30T14:22:41.016 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/198' 2023-10-30T14:22:41.017 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/136/19' 2023-10-30T14:22:41.017 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/136/15' 2023-10-30T14:22:41.017 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/136/20' 2023-10-30T14:22:41.017 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/136/12' 2023-10-30T14:22:41.017 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/136/10' 2023-10-30T14:22:41.017 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/136/11' 2023-10-30T14:22:41.018 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/136/16' 2023-10-30T14:22:41.018 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/136/14' 2023-10-30T14:22:41.018 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/136/18' 2023-10-30T14:22:41.018 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/136/13' 2023-10-30T14:22:41.018 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/136/17' 2023-10-30T14:22:41.019 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/136' 2023-10-30T14:22:41.019 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/125/19' 2023-10-30T14:22:41.019 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/125/15' 2023-10-30T14:22:41.019 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/125/20' 2023-10-30T14:22:41.019 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/125/12' 2023-10-30T14:22:41.020 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/125/10' 2023-10-30T14:22:41.020 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/125/11' 2023-10-30T14:22:41.020 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/125/16' 2023-10-30T14:22:41.020 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/125/14' 2023-10-30T14:22:41.020 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/125/18' 2023-10-30T14:22:41.021 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/125/13' 2023-10-30T14:22:41.021 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/125/17' 2023-10-30T14:22:41.021 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/125' 2023-10-30T14:22:41.021 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/194/19' 2023-10-30T14:22:41.021 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/194/15' 2023-10-30T14:22:41.021 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/194/20' 2023-10-30T14:22:41.022 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/194/12' 2023-10-30T14:22:41.022 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/194/10' 2023-10-30T14:22:41.022 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/194/11' 2023-10-30T14:22:41.022 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/194/16' 2023-10-30T14:22:41.022 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/194/14' 2023-10-30T14:22:41.023 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/194/18' 2023-10-30T14:22:41.023 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/194/13' 2023-10-30T14:22:41.023 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/194/17' 2023-10-30T14:22:41.023 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/194' 2023-10-30T14:22:41.023 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/119/19' 2023-10-30T14:22:41.024 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/119/15' 2023-10-30T14:22:41.024 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/119/20' 2023-10-30T14:22:41.024 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/119/12' 2023-10-30T14:22:41.024 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/119/10' 2023-10-30T14:22:41.024 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/119/11' 2023-10-30T14:22:41.024 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/119/16' 2023-10-30T14:22:41.025 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/119/14' 2023-10-30T14:22:41.025 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/119/18' 2023-10-30T14:22:41.025 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/119/13' 2023-10-30T14:22:41.025 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/119/17' 2023-10-30T14:22:41.025 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/119' 2023-10-30T14:22:41.026 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/157/19' 2023-10-30T14:22:41.026 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/157/15' 2023-10-30T14:22:41.026 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/157/20' 2023-10-30T14:22:41.026 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/157/12' 2023-10-30T14:22:41.026 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/157/10' 2023-10-30T14:22:41.026 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/157/11' 2023-10-30T14:22:41.027 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/157/16' 2023-10-30T14:22:41.027 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/157/14' 2023-10-30T14:22:41.027 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/157/18' 2023-10-30T14:22:41.027 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/157/13' 2023-10-30T14:22:41.027 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/157/17' 2023-10-30T14:22:41.028 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/157' 2023-10-30T14:22:41.028 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/172/19' 2023-10-30T14:22:41.028 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/172/15' 2023-10-30T14:22:41.028 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/172/20' 2023-10-30T14:22:41.028 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/172/12' 2023-10-30T14:22:41.029 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/172/10' 2023-10-30T14:22:41.029 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/172/11' 2023-10-30T14:22:41.029 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/172/16' 2023-10-30T14:22:41.029 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/172/14' 2023-10-30T14:22:41.029 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/172/18' 2023-10-30T14:22:41.029 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/172/13' 2023-10-30T14:22:41.030 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/172/17' 2023-10-30T14:22:41.030 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/172' 2023-10-30T14:22:41.030 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/186/19' 2023-10-30T14:22:41.030 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/186/15' 2023-10-30T14:22:41.030 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/186/20' 2023-10-30T14:22:41.031 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/186/12' 2023-10-30T14:22:41.031 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/186/10' 2023-10-30T14:22:41.031 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/186/11' 2023-10-30T14:22:41.031 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/186/16' 2023-10-30T14:22:41.031 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/186/14' 2023-10-30T14:22:41.031 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/186/18' 2023-10-30T14:22:41.032 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/186/13' 2023-10-30T14:22:41.032 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/186/17' 2023-10-30T14:22:41.032 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/186' 2023-10-30T14:22:41.032 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/190/19' 2023-10-30T14:22:41.032 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/190/15' 2023-10-30T14:22:41.033 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/190/20' 2023-10-30T14:22:41.033 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/190/12' 2023-10-30T14:22:41.033 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/190/10' 2023-10-30T14:22:41.033 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/190/11' 2023-10-30T14:22:41.033 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/190/16' 2023-10-30T14:22:41.034 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/190/14' 2023-10-30T14:22:46.893 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/190/18' 2023-10-30T14:22:46.893 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/190/13' 2023-10-30T14:22:46.893 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/190/17' 2023-10-30T14:22:46.894 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/190' 2023-10-30T14:22:46.894 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/104/19' 2023-10-30T14:22:46.894 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/104/15' 2023-10-30T14:22:46.894 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/104/20' 2023-10-30T14:22:46.894 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/104/12' 2023-10-30T14:22:46.894 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/104/10' 2023-10-30T14:22:46.895 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/104/11' 2023-10-30T14:22:46.895 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/104/16' 2023-10-30T14:22:46.895 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/104/14' 2023-10-30T14:22:46.895 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/104/18' 2023-10-30T14:22:46.895 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/104/13' 2023-10-30T14:22:46.896 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/104/17' 2023-10-30T14:22:46.896 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/104' 2023-10-30T14:22:46.896 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/164/19' 2023-10-30T14:22:46.896 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/164/15' 2023-10-30T14:22:46.896 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/164/20' 2023-10-30T14:22:46.896 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/164/12' 2023-10-30T14:22:46.897 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/164/10' 2023-10-30T14:22:46.897 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/164/11' 2023-10-30T14:22:46.897 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/164/16' 2023-10-30T14:22:46.897 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/164/14' 2023-10-30T14:22:46.897 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/164/18' 2023-10-30T14:22:46.897 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/164/13' 2023-10-30T14:22:46.898 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/164/17' 2023-10-30T14:22:46.898 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/164' 2023-10-30T14:22:46.898 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/112/19' 2023-10-30T14:22:46.898 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/112/15' 2023-10-30T14:22:46.898 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/112/20' 2023-10-30T14:22:46.899 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/112/12' 2023-10-30T14:22:46.899 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/112/10' 2023-10-30T14:22:46.899 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/112/11' 2023-10-30T14:22:46.899 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/112/16' 2023-10-30T14:22:46.899 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/112/14' 2023-10-30T14:22:46.899 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/112/18' 2023-10-30T14:22:46.900 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/112/13' 2023-10-30T14:22:46.900 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/112/17' 2023-10-30T14:22:46.900 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/112' 2023-10-30T14:22:46.900 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/100/19' 2023-10-30T14:22:46.900 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/100/15' 2023-10-30T14:22:46.901 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/100/20' 2023-10-30T14:22:46.901 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/100/12' 2023-10-30T14:22:46.901 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/100/10' 2023-10-30T14:22:46.901 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/100/11' 2023-10-30T14:22:46.901 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/100/16' 2023-10-30T14:22:46.901 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/100/14' 2023-10-30T14:22:46.902 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/100/18' 2023-10-30T14:22:46.902 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/100/13' 2023-10-30T14:22:46.902 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/100/17' 2023-10-30T14:22:46.902 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/100' 2023-10-30T14:22:46.902 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/134/19' 2023-10-30T14:22:46.902 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/134/15' 2023-10-30T14:22:46.903 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/134/20' 2023-10-30T14:22:46.903 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/134/12' 2023-10-30T14:22:46.903 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/134/10' 2023-10-30T14:22:46.903 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/134/11' 2023-10-30T14:22:46.903 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/134/16' 2023-10-30T14:22:46.903 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/134/14' 2023-10-30T14:22:46.904 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/134/18' 2023-10-30T14:22:46.904 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/134/13' 2023-10-30T14:22:46.904 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/134/17' 2023-10-30T14:22:46.904 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/134' 2023-10-30T14:22:46.904 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/115/19' 2023-10-30T14:22:46.904 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/115/15' 2023-10-30T14:22:46.905 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/115/20' 2023-10-30T14:22:46.905 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/115/12' 2023-10-30T14:22:46.905 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/115/10' 2023-10-30T14:22:46.905 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/115/11' 2023-10-30T14:22:46.905 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/115/16' 2023-10-30T14:22:46.906 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/115/14' 2023-10-30T14:22:46.906 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/115/18' 2023-10-30T14:22:46.906 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/115/13' 2023-10-30T14:22:46.906 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/115/17' 2023-10-30T14:22:46.906 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/115' 2023-10-30T14:22:46.906 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/116/19' 2023-10-30T14:22:46.907 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/116/15' 2023-10-30T14:22:46.907 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/116/20' 2023-10-30T14:22:46.907 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/116/12' 2023-10-30T14:22:46.907 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/116/10' 2023-10-30T14:22:46.907 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/116/11' 2023-10-30T14:22:46.907 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/116/16' 2023-10-30T14:22:46.908 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/116/14' 2023-10-30T14:22:46.908 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/116/18' 2023-10-30T14:22:46.908 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/116/13' 2023-10-30T14:22:46.908 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/116/17' 2023-10-30T14:22:46.908 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/116' 2023-10-30T14:22:46.909 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/195/19' 2023-10-30T14:22:46.909 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/195/15' 2023-10-30T14:22:46.909 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/195/20' 2023-10-30T14:22:46.909 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/195/12' 2023-10-30T14:22:46.909 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/195/10' 2023-10-30T14:22:46.909 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/195/11' 2023-10-30T14:22:46.910 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/195/16' 2023-10-30T14:22:46.910 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/195/14' 2023-10-30T14:22:46.910 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/195/18' 2023-10-30T14:22:46.910 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/195/13' 2023-10-30T14:22:46.910 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/195/17' 2023-10-30T14:22:46.911 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/195' 2023-10-30T14:22:46.911 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/138/19' 2023-10-30T14:22:46.911 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/138/15' 2023-10-30T14:22:46.911 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/138/20' 2023-10-30T14:22:46.911 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/138/12' 2023-10-30T14:22:46.911 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/138/10' 2023-10-30T14:22:46.912 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/138/11' 2023-10-30T14:22:46.912 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/138/16' 2023-10-30T14:22:46.912 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/138/14' 2023-10-30T14:22:46.912 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/138/18' 2023-10-30T14:22:49.287 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/138/13' 2023-10-30T14:22:49.287 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/138/17' 2023-10-30T14:22:49.287 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/138' 2023-10-30T14:22:49.288 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/135/19' 2023-10-30T14:22:49.288 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/135/15' 2023-10-30T14:22:49.288 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/135/20' 2023-10-30T14:22:49.288 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/135/12' 2023-10-30T14:22:49.289 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/135/10' 2023-10-30T14:22:49.289 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/135/11' 2023-10-30T14:22:49.289 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/135/16' 2023-10-30T14:22:49.289 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/135/14' 2023-10-30T14:22:49.289 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/135/18' 2023-10-30T14:22:49.289 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/135/13' 2023-10-30T14:22:49.290 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/135/17' 2023-10-30T14:22:49.290 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/135' 2023-10-30T14:22:49.290 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/117/19' 2023-10-30T14:22:49.290 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/117/15' 2023-10-30T14:22:49.290 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/117/20' 2023-10-30T14:22:49.290 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/117/12' 2023-10-30T14:22:49.291 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/117/10' 2023-10-30T14:22:49.291 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/117/11' 2023-10-30T14:22:49.291 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/117/16' 2023-10-30T14:22:49.291 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/117/14' 2023-10-30T14:22:49.291 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/117/18' 2023-10-30T14:22:49.291 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/117/13' 2023-10-30T14:22:49.292 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/117/17' 2023-10-30T14:22:49.292 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/117' 2023-10-30T14:22:49.292 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/143/19' 2023-10-30T14:22:49.292 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/143/15' 2023-10-30T14:22:49.292 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/143/20' 2023-10-30T14:22:49.293 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/143/12' 2023-10-30T14:22:49.293 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/143/10' 2023-10-30T14:22:49.293 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/143/11' 2023-10-30T14:22:49.293 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/143/16' 2023-10-30T14:22:49.293 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/143/14' 2023-10-30T14:22:49.293 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/143/18' 2023-10-30T14:22:49.294 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/143/13' 2023-10-30T14:22:49.294 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/143/17' 2023-10-30T14:22:49.294 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/143' 2023-10-30T14:22:49.294 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/180/19' 2023-10-30T14:22:49.294 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/180/15' 2023-10-30T14:22:49.295 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/180/20' 2023-10-30T14:22:49.295 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/180/12' 2023-10-30T14:22:49.295 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/180/10' 2023-10-30T14:22:49.295 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/180/11' 2023-10-30T14:22:49.295 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/180/16' 2023-10-30T14:22:49.295 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/180/14' 2023-10-30T14:22:49.296 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/180/18' 2023-10-30T14:22:49.296 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/180/13' 2023-10-30T14:22:49.296 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/180/17' 2023-10-30T14:22:49.296 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/180' 2023-10-30T14:22:49.296 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/126/19' 2023-10-30T14:22:49.296 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/126/15' 2023-10-30T14:22:49.297 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/126/20' 2023-10-30T14:22:49.297 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/126/12' 2023-10-30T14:22:49.297 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/126/10' 2023-10-30T14:22:49.297 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/126/11' 2023-10-30T14:22:49.297 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/126/16' 2023-10-30T14:22:49.297 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/126/14' 2023-10-30T14:22:49.298 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/126/18' 2023-10-30T14:22:49.298 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/126/13' 2023-10-30T14:22:49.298 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/126/17' 2023-10-30T14:22:49.298 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/126' 2023-10-30T14:22:49.298 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/161/19' 2023-10-30T14:22:49.299 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/161/15' 2023-10-30T14:22:49.299 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/161/20' 2023-10-30T14:22:49.299 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/161/12' 2023-10-30T14:22:49.299 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/161/10' 2023-10-30T14:22:49.299 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/161/11' 2023-10-30T14:22:49.300 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/161/16' 2023-10-30T14:22:49.300 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/161/14' 2023-10-30T14:22:49.300 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/161/18' 2023-10-30T14:22:49.300 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/161/13' 2023-10-30T14:22:49.300 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/161/17' 2023-10-30T14:22:49.301 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/161' 2023-10-30T14:22:49.301 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/176/19' 2023-10-30T14:22:49.301 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/176/15' 2023-10-30T14:22:49.301 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/176/20' 2023-10-30T14:22:49.301 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/176/12' 2023-10-30T14:22:49.301 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/176/10' 2023-10-30T14:22:49.302 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/176/11' 2023-10-30T14:22:49.302 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/176/16' 2023-10-30T14:22:49.302 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/176/14' 2023-10-30T14:22:49.302 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/176/18' 2023-10-30T14:22:49.302 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/176/13' 2023-10-30T14:22:49.303 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/176/17' 2023-10-30T14:22:49.303 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/176' 2023-10-30T14:22:49.303 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/111/19' 2023-10-30T14:22:49.303 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/111/15' 2023-10-30T14:22:49.303 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/111/20' 2023-10-30T14:22:49.303 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/111/12' 2023-10-30T14:22:49.304 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/111/10' 2023-10-30T14:22:49.304 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/111/11' 2023-10-30T14:22:49.304 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/111/16' 2023-10-30T14:22:49.304 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/111/14' 2023-10-30T14:22:49.304 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/111/18' 2023-10-30T14:22:49.304 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/111/13' 2023-10-30T14:22:49.305 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/111/17' 2023-10-30T14:22:49.305 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/111' 2023-10-30T14:22:49.305 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/191/19' 2023-10-30T14:22:49.305 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/191/15' 2023-10-30T14:22:49.305 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/191/20' 2023-10-30T14:22:49.306 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/191/12' 2023-10-30T14:22:49.306 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/191/10' 2023-10-30T14:22:49.306 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/191/11' 2023-10-30T14:22:49.306 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/191/16' 2023-10-30T14:22:49.306 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/191/14' 2023-10-30T14:22:49.306 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/191/18' 2023-10-30T14:22:49.307 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/191/13' 2023-10-30T14:22:50.131 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/191/17' 2023-10-30T14:22:50.132 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/191' 2023-10-30T14:22:50.132 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/184/19' 2023-10-30T14:22:50.132 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/184/15' 2023-10-30T14:22:50.132 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/184/20' 2023-10-30T14:22:50.132 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/184/12' 2023-10-30T14:22:50.133 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/184/10' 2023-10-30T14:22:50.133 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/184/11' 2023-10-30T14:22:50.133 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/184/16' 2023-10-30T14:22:50.133 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/184/14' 2023-10-30T14:22:50.133 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/184/18' 2023-10-30T14:22:50.134 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/184/13' 2023-10-30T14:22:50.134 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/184/17' 2023-10-30T14:22:50.134 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/184' 2023-10-30T14:22:50.134 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/192/19' 2023-10-30T14:22:50.134 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/192/15' 2023-10-30T14:22:50.135 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/192/20' 2023-10-30T14:22:50.135 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/192/12' 2023-10-30T14:22:50.135 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/192/10' 2023-10-30T14:22:50.135 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/192/11' 2023-10-30T14:22:50.135 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/192/16' 2023-10-30T14:22:50.136 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/192/14' 2023-10-30T14:22:50.136 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/192/18' 2023-10-30T14:22:50.136 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/192/13' 2023-10-30T14:22:50.136 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/192/17' 2023-10-30T14:22:50.136 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/192' 2023-10-30T14:22:50.136 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/103/19' 2023-10-30T14:22:50.137 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/103/15' 2023-10-30T14:22:50.137 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/103/20' 2023-10-30T14:22:50.137 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/103/12' 2023-10-30T14:22:50.137 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/103/10' 2023-10-30T14:22:50.137 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/103/11' 2023-10-30T14:22:50.138 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/103/16' 2023-10-30T14:22:50.138 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/103/14' 2023-10-30T14:22:50.138 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/103/18' 2023-10-30T14:22:50.138 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/103/13' 2023-10-30T14:22:50.138 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/103/17' 2023-10-30T14:22:50.138 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/103' 2023-10-30T14:22:50.139 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/128/19' 2023-10-30T14:22:50.139 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/128/15' 2023-10-30T14:22:50.139 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/128/20' 2023-10-30T14:22:50.139 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/128/12' 2023-10-30T14:22:50.139 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/128/10' 2023-10-30T14:22:50.140 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/128/11' 2023-10-30T14:22:50.140 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/128/16' 2023-10-30T14:22:50.140 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/128/14' 2023-10-30T14:22:50.140 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/128/18' 2023-10-30T14:22:50.140 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/128/13' 2023-10-30T14:22:50.141 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/128/17' 2023-10-30T14:22:50.141 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/128' 2023-10-30T14:22:50.141 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/144/19' 2023-10-30T14:22:50.141 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/144/15' 2023-10-30T14:22:50.141 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/144/20' 2023-10-30T14:22:50.141 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/144/12' 2023-10-30T14:22:50.142 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/144/10' 2023-10-30T14:22:50.142 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/144/11' 2023-10-30T14:22:50.142 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/144/16' 2023-10-30T14:22:50.142 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/144/14' 2023-10-30T14:22:50.142 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/144/18' 2023-10-30T14:22:50.143 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/144/13' 2023-10-30T14:22:50.143 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/144/17' 2023-10-30T14:22:50.143 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/144' 2023-10-30T14:22:50.143 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/146/19' 2023-10-30T14:22:50.143 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/146/15' 2023-10-30T14:22:50.144 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/146/20' 2023-10-30T14:22:50.144 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/146/12' 2023-10-30T14:22:50.144 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/146/10' 2023-10-30T14:22:50.144 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/146/11' 2023-10-30T14:22:50.144 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/146/16' 2023-10-30T14:22:50.144 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/146/14' 2023-10-30T14:22:50.145 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/146/18' 2023-10-30T14:22:50.145 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/146/13' 2023-10-30T14:22:50.145 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/146/17' 2023-10-30T14:22:50.145 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/146' 2023-10-30T14:22:50.145 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/141/19' 2023-10-30T14:22:50.146 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/141/15' 2023-10-30T14:22:50.146 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/141/20' 2023-10-30T14:22:50.146 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/141/12' 2023-10-30T14:22:50.146 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/141/10' 2023-10-30T14:22:50.146 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/141/11' 2023-10-30T14:22:50.146 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/141/16' 2023-10-30T14:22:50.147 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/141/14' 2023-10-30T14:22:50.147 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/141/18' 2023-10-30T14:22:50.147 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/141/13' 2023-10-30T14:22:50.147 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/141/17' 2023-10-30T14:22:50.147 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/141' 2023-10-30T14:22:50.148 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/182/19' 2023-10-30T14:22:50.148 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/182/15' 2023-10-30T14:22:50.148 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/182/20' 2023-10-30T14:22:50.148 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/182/12' 2023-10-30T14:22:50.148 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/182/10' 2023-10-30T14:22:50.148 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/182/11' 2023-10-30T14:22:50.149 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/182/16' 2023-10-30T14:22:50.149 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/182/14' 2023-10-30T14:22:50.149 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/182/18' 2023-10-30T14:22:50.149 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/182/13' 2023-10-30T14:22:50.149 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/182/17' 2023-10-30T14:22:50.150 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/182' 2023-10-30T14:22:50.150 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/107/19' 2023-10-30T14:22:50.150 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/107/15' 2023-10-30T14:22:50.150 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/107/20' 2023-10-30T14:22:50.150 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/107/12' 2023-10-30T14:22:50.151 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/107/10' 2023-10-30T14:22:50.151 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/107/11' 2023-10-30T14:22:50.151 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/107/16' 2023-10-30T14:22:50.151 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/107/14' 2023-10-30T14:22:50.151 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/107/18' 2023-10-30T14:22:50.151 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/107/13' 2023-10-30T14:22:50.152 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/107/17' 2023-10-30T14:22:50.845 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/107' 2023-10-30T14:22:50.846 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/178/19' 2023-10-30T14:22:50.846 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/178/15' 2023-10-30T14:22:50.846 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/178/20' 2023-10-30T14:22:50.846 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/178/12' 2023-10-30T14:22:50.846 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/178/10' 2023-10-30T14:22:50.847 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/178/11' 2023-10-30T14:22:50.847 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/178/16' 2023-10-30T14:22:50.847 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/178/14' 2023-10-30T14:22:50.847 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/178/18' 2023-10-30T14:22:50.847 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/178/13' 2023-10-30T14:22:50.848 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/178/17' 2023-10-30T14:22:50.848 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/178' 2023-10-30T14:22:50.848 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/156/19' 2023-10-30T14:22:50.848 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/156/15' 2023-10-30T14:22:50.848 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/156/20' 2023-10-30T14:22:50.849 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/156/12' 2023-10-30T14:22:50.849 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/156/10' 2023-10-30T14:22:50.849 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/156/11' 2023-10-30T14:22:50.849 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/156/16' 2023-10-30T14:22:50.849 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/156/14' 2023-10-30T14:22:50.850 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/156/18' 2023-10-30T14:22:50.850 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/156/13' 2023-10-30T14:22:50.850 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/156/17' 2023-10-30T14:22:50.850 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/156' 2023-10-30T14:22:50.850 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/131/19' 2023-10-30T14:22:50.850 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/131/15' 2023-10-30T14:22:50.851 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/131/20' 2023-10-30T14:22:50.851 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/131/12' 2023-10-30T14:22:50.851 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/131/10' 2023-10-30T14:22:50.851 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/131/11' 2023-10-30T14:22:50.851 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/131/16' 2023-10-30T14:22:50.852 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/131/14' 2023-10-30T14:22:50.852 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/131/18' 2023-10-30T14:22:50.852 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/131/13' 2023-10-30T14:22:50.852 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/131/17' 2023-10-30T14:22:50.852 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/131' 2023-10-30T14:22:50.852 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/123/19' 2023-10-30T14:22:50.853 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/123/15' 2023-10-30T14:22:50.853 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/123/20' 2023-10-30T14:22:50.853 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/123/12' 2023-10-30T14:22:50.853 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/123/10' 2023-10-30T14:22:50.853 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/123/11' 2023-10-30T14:22:50.854 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/123/16' 2023-10-30T14:22:50.854 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/123/14' 2023-10-30T14:22:50.854 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/123/18' 2023-10-30T14:22:50.854 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/123/13' 2023-10-30T14:22:50.854 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/123/17' 2023-10-30T14:22:50.855 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/123' 2023-10-30T14:22:50.855 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/160/19' 2023-10-30T14:22:50.855 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/160/15' 2023-10-30T14:22:50.855 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/160/20' 2023-10-30T14:22:50.855 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/160/12' 2023-10-30T14:22:50.855 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/160/10' 2023-10-30T14:22:50.856 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/160/11' 2023-10-30T14:22:50.856 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/160/16' 2023-10-30T14:22:50.856 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/160/14' 2023-10-30T14:22:50.856 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/160/18' 2023-10-30T14:22:50.856 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/160/13' 2023-10-30T14:22:50.857 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/160/17' 2023-10-30T14:22:50.857 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/160' 2023-10-30T14:22:50.857 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/165/19' 2023-10-30T14:22:50.857 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/165/15' 2023-10-30T14:22:50.857 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/165/20' 2023-10-30T14:22:50.857 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/165/12' 2023-10-30T14:22:50.858 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/165/10' 2023-10-30T14:22:50.858 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/165/11' 2023-10-30T14:22:50.858 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/165/16' 2023-10-30T14:22:50.858 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/165/14' 2023-10-30T14:22:50.858 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/165/18' 2023-10-30T14:22:50.859 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/165/13' 2023-10-30T14:22:50.859 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/165/17' 2023-10-30T14:22:50.859 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/165' 2023-10-30T14:22:50.859 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/101/19' 2023-10-30T14:22:50.859 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/101/15' 2023-10-30T14:22:50.860 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/101/20' 2023-10-30T14:22:50.860 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/101/12' 2023-10-30T14:22:50.860 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/101/10' 2023-10-30T14:22:50.860 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/101/11' 2023-10-30T14:22:50.860 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/101/16' 2023-10-30T14:22:50.860 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/101/14' 2023-10-30T14:22:50.861 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/101/18' 2023-10-30T14:22:50.861 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/101/13' 2023-10-30T14:22:50.861 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/101/17' 2023-10-30T14:22:50.861 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/101' 2023-10-30T14:22:50.861 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/148/19' 2023-10-30T14:22:50.861 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/148/15' 2023-10-30T14:22:50.862 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/148/20' 2023-10-30T14:22:50.862 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/148/12' 2023-10-30T14:22:50.862 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/148/10' 2023-10-30T14:22:50.862 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/148/11' 2023-10-30T14:22:50.862 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/148/16' 2023-10-30T14:22:50.862 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/148/14' 2023-10-30T14:22:50.863 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/148/18' 2023-10-30T14:22:50.863 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/148/13' 2023-10-30T14:22:50.863 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/148/17' 2023-10-30T14:22:50.863 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/148' 2023-10-30T14:22:50.863 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/168/19' 2023-10-30T14:22:50.864 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/168/15' 2023-10-30T14:22:50.864 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/168/20' 2023-10-30T14:22:50.864 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/168/12' 2023-10-30T14:22:50.864 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/168/10' 2023-10-30T14:22:50.864 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/168/11' 2023-10-30T14:22:50.864 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/168/16' 2023-10-30T14:22:50.865 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/168/14' 2023-10-30T14:22:50.865 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/168/18' 2023-10-30T14:22:50.865 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/168/13' 2023-10-30T14:22:50.865 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/168/17' 2023-10-30T14:22:50.865 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/168' 2023-10-30T14:22:54.162 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/179/19' 2023-10-30T14:22:54.162 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/179/15' 2023-10-30T14:22:54.163 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/179/20' 2023-10-30T14:22:54.163 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/179/12' 2023-10-30T14:22:54.163 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/179/10' 2023-10-30T14:22:54.163 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/179/11' 2023-10-30T14:22:54.164 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/179/16' 2023-10-30T14:22:54.164 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/179/14' 2023-10-30T14:22:54.164 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/179/18' 2023-10-30T14:22:54.164 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/179/13' 2023-10-30T14:22:54.164 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/179/17' 2023-10-30T14:22:54.165 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/179' 2023-10-30T14:22:54.165 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/132/19' 2023-10-30T14:22:54.165 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/132/15' 2023-10-30T14:22:54.165 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/132/20' 2023-10-30T14:22:54.165 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/132/12' 2023-10-30T14:22:54.166 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/132/10' 2023-10-30T14:22:54.166 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/132/11' 2023-10-30T14:22:54.166 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/132/16' 2023-10-30T14:22:54.166 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/132/14' 2023-10-30T14:22:54.166 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/132/18' 2023-10-30T14:22:54.166 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/132/13' 2023-10-30T14:22:54.167 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/132/17' 2023-10-30T14:22:54.167 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/132' 2023-10-30T14:22:54.167 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/199/19' 2023-10-30T14:22:54.167 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/199/15' 2023-10-30T14:22:54.168 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/199/20' 2023-10-30T14:22:54.168 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/199/12' 2023-10-30T14:22:54.168 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/199/10' 2023-10-30T14:22:54.168 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/199/11' 2023-10-30T14:22:54.168 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/199/16' 2023-10-30T14:22:54.168 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/199/14' 2023-10-30T14:22:54.169 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/199/18' 2023-10-30T14:22:54.169 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/199/13' 2023-10-30T14:22:54.169 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/199/17' 2023-10-30T14:22:54.169 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/199' 2023-10-30T14:22:54.169 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/163/19' 2023-10-30T14:22:54.170 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/163/15' 2023-10-30T14:22:54.170 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/163/20' 2023-10-30T14:22:54.170 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/163/12' 2023-10-30T14:22:54.170 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/163/10' 2023-10-30T14:22:54.170 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/163/11' 2023-10-30T14:22:54.171 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/163/16' 2023-10-30T14:22:54.171 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/163/14' 2023-10-30T14:22:54.171 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/163/18' 2023-10-30T14:22:54.171 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/163/13' 2023-10-30T14:22:54.171 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/163/17' 2023-10-30T14:22:54.171 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/163' 2023-10-30T14:22:54.172 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/153/19' 2023-10-30T14:22:54.172 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/153/15' 2023-10-30T14:22:54.172 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/153/20' 2023-10-30T14:22:54.172 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/153/12' 2023-10-30T14:22:54.172 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/153/10' 2023-10-30T14:22:54.173 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/153/11' 2023-10-30T14:22:54.173 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/153/16' 2023-10-30T14:22:54.173 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/153/14' 2023-10-30T14:22:54.173 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/153/18' 2023-10-30T14:22:54.173 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/153/13' 2023-10-30T14:22:54.174 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/153/17' 2023-10-30T14:22:54.174 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/153' 2023-10-30T14:22:54.174 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/122/19' 2023-10-30T14:22:54.174 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/122/15' 2023-10-30T14:22:54.174 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/122/20' 2023-10-30T14:22:54.175 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/122/12' 2023-10-30T14:22:54.175 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/122/10' 2023-10-30T14:22:54.175 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/122/11' 2023-10-30T14:22:54.175 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/122/16' 2023-10-30T14:22:54.175 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/122/14' 2023-10-30T14:22:54.176 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/122/18' 2023-10-30T14:22:54.176 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/122/13' 2023-10-30T14:22:54.176 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/122/17' 2023-10-30T14:22:54.176 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/122' 2023-10-30T14:22:54.176 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/162/19' 2023-10-30T14:22:54.177 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/162/15' 2023-10-30T14:22:54.177 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/162/20' 2023-10-30T14:22:54.177 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/162/12' 2023-10-30T14:22:54.177 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/162/10' 2023-10-30T14:22:54.177 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/162/11' 2023-10-30T14:22:54.177 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/162/16' 2023-10-30T14:22:54.178 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/162/14' 2023-10-30T14:22:54.178 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/162/18' 2023-10-30T14:22:54.178 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/162/13' 2023-10-30T14:22:54.178 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/162/17' 2023-10-30T14:22:54.178 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/162' 2023-10-30T14:22:54.179 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/197/19' 2023-10-30T14:22:54.179 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/197/15' 2023-10-30T14:22:54.179 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/197/20' 2023-10-30T14:22:54.179 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/197/12' 2023-10-30T14:22:54.179 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/197/10' 2023-10-30T14:22:54.180 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/197/11' 2023-10-30T14:22:54.180 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/197/16' 2023-10-30T14:22:54.180 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/197/14' 2023-10-30T14:22:54.180 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/197/18' 2023-10-30T14:22:54.180 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/197/13' 2023-10-30T14:22:54.180 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/197/17' 2023-10-30T14:22:54.181 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/197' 2023-10-30T14:22:54.181 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/113/19' 2023-10-30T14:22:54.181 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/113/15' 2023-10-30T14:22:54.181 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/113/20' 2023-10-30T14:22:54.181 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/113/12' 2023-10-30T14:22:54.182 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/113/10' 2023-10-30T14:22:54.182 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/113/11' 2023-10-30T14:22:54.182 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/113/16' 2023-10-30T14:22:54.182 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/113/14' 2023-10-30T14:22:54.182 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/113/18' 2023-10-30T14:22:54.182 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/113/13' 2023-10-30T14:22:54.183 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/113/17' 2023-10-30T14:22:54.183 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/113' 2023-10-30T14:22:54.183 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/152/19' 2023-10-30T14:22:55.444 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/152/15' 2023-10-30T14:22:55.444 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/152/20' 2023-10-30T14:22:55.445 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/152/12' 2023-10-30T14:22:55.445 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/152/10' 2023-10-30T14:22:55.445 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/152/11' 2023-10-30T14:22:55.445 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/152/16' 2023-10-30T14:22:55.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/152/14' 2023-10-30T14:22:55.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/152/18' 2023-10-30T14:22:55.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/152/13' 2023-10-30T14:22:55.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/152/17' 2023-10-30T14:22:55.446 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/152' 2023-10-30T14:22:55.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/121/19' 2023-10-30T14:22:55.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/121/15' 2023-10-30T14:22:55.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/121/20' 2023-10-30T14:22:55.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/121/12' 2023-10-30T14:22:55.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/121/10' 2023-10-30T14:22:55.447 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/121/11' 2023-10-30T14:22:55.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/121/16' 2023-10-30T14:22:55.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/121/14' 2023-10-30T14:22:55.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/121/18' 2023-10-30T14:22:55.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/121/13' 2023-10-30T14:22:55.448 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/121/17' 2023-10-30T14:22:55.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/121' 2023-10-30T14:22:55.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/118/19' 2023-10-30T14:22:55.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/118/15' 2023-10-30T14:22:55.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/118/20' 2023-10-30T14:22:55.449 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/118/12' 2023-10-30T14:22:55.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/118/10' 2023-10-30T14:22:55.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/118/11' 2023-10-30T14:22:55.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/118/16' 2023-10-30T14:22:55.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/118/14' 2023-10-30T14:22:55.450 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/118/18' 2023-10-30T14:22:55.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/118/13' 2023-10-30T14:22:55.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/118/17' 2023-10-30T14:22:55.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/118' 2023-10-30T14:22:55.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/196/19' 2023-10-30T14:22:55.451 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/196/15' 2023-10-30T14:22:55.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/196/20' 2023-10-30T14:22:55.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/196/12' 2023-10-30T14:22:55.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/196/10' 2023-10-30T14:22:55.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/196/11' 2023-10-30T14:22:55.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/196/16' 2023-10-30T14:22:55.452 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/196/14' 2023-10-30T14:22:55.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/196/18' 2023-10-30T14:22:55.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/196/13' 2023-10-30T14:22:55.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/196/17' 2023-10-30T14:22:55.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/196' 2023-10-30T14:22:55.453 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/124/19' 2023-10-30T14:22:55.454 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/124/15' 2023-10-30T14:22:55.454 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/124/20' 2023-10-30T14:22:55.454 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/124/12' 2023-10-30T14:22:55.454 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/124/10' 2023-10-30T14:22:55.454 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/124/11' 2023-10-30T14:22:55.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/124/16' 2023-10-30T14:22:55.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/124/14' 2023-10-30T14:22:55.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/124/18' 2023-10-30T14:22:55.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/124/13' 2023-10-30T14:22:55.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/124/17' 2023-10-30T14:22:55.455 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/124' 2023-10-30T14:22:55.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/188/19' 2023-10-30T14:22:55.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/188/15' 2023-10-30T14:22:55.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/188/20' 2023-10-30T14:22:55.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/188/12' 2023-10-30T14:22:55.456 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/188/10' 2023-10-30T14:22:55.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/188/11' 2023-10-30T14:22:55.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/188/16' 2023-10-30T14:22:55.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/188/14' 2023-10-30T14:22:55.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/188/18' 2023-10-30T14:22:55.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/188/13' 2023-10-30T14:22:55.457 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/188/17' 2023-10-30T14:22:55.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/188' 2023-10-30T14:22:55.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/145/19' 2023-10-30T14:22:55.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/145/15' 2023-10-30T14:22:55.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/145/20' 2023-10-30T14:22:55.458 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/145/12' 2023-10-30T14:22:55.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/145/10' 2023-10-30T14:22:55.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/145/11' 2023-10-30T14:22:55.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/145/16' 2023-10-30T14:22:55.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/145/14' 2023-10-30T14:22:55.459 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/145/18' 2023-10-30T14:22:55.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/145/13' 2023-10-30T14:22:55.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/145/17' 2023-10-30T14:22:55.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/145' 2023-10-30T14:22:55.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/130/19' 2023-10-30T14:22:55.460 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/130/15' 2023-10-30T14:22:55.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/130/20' 2023-10-30T14:22:55.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/130/12' 2023-10-30T14:22:55.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/130/10' 2023-10-30T14:22:55.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/130/11' 2023-10-30T14:22:55.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/130/16' 2023-10-30T14:22:55.461 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/130/14' 2023-10-30T14:22:55.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/130/18' 2023-10-30T14:22:55.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/130/13' 2023-10-30T14:22:55.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/130/17' 2023-10-30T14:22:55.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/130' 2023-10-30T14:22:55.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/193/19' 2023-10-30T14:22:55.462 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/193/15' 2023-10-30T14:22:55.463 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/193/20' 2023-10-30T14:22:55.463 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/193/12' 2023-10-30T14:22:55.463 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/193/10' 2023-10-30T14:22:55.463 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/193/11' 2023-10-30T14:22:55.463 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/193/16' 2023-10-30T14:22:55.464 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/193/14' 2023-10-30T14:22:55.464 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/193/18' 2023-10-30T14:22:55.464 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/193/13' 2023-10-30T14:22:55.464 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/193/17' 2023-10-30T14:22:55.464 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/193' 2023-10-30T14:22:55.465 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/175/19' 2023-10-30T14:22:55.465 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/175/15' 2023-10-30T14:22:56.891 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/175/20' 2023-10-30T14:22:56.891 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/175/12' 2023-10-30T14:22:56.891 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/175/10' 2023-10-30T14:22:56.891 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/175/11' 2023-10-30T14:22:56.892 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/175/16' 2023-10-30T14:22:56.892 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/175/14' 2023-10-30T14:22:56.892 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/175/18' 2023-10-30T14:22:56.892 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/175/13' 2023-10-30T14:22:56.892 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/175/17' 2023-10-30T14:22:56.892 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/175' 2023-10-30T14:22:56.893 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/181/19' 2023-10-30T14:22:56.893 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/181/15' 2023-10-30T14:22:56.893 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/181/20' 2023-10-30T14:22:56.893 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/181/12' 2023-10-30T14:22:56.893 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/181/10' 2023-10-30T14:22:56.894 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/181/11' 2023-10-30T14:22:56.894 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/181/16' 2023-10-30T14:22:56.894 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/181/14' 2023-10-30T14:22:56.894 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/181/18' 2023-10-30T14:22:56.894 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/181/13' 2023-10-30T14:22:56.895 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/181/17' 2023-10-30T14:22:56.895 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/181' 2023-10-30T14:22:56.895 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/127/19' 2023-10-30T14:22:56.895 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/127/15' 2023-10-30T14:22:56.895 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/127/20' 2023-10-30T14:22:56.895 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/127/12' 2023-10-30T14:22:56.896 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/127/10' 2023-10-30T14:22:56.896 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/127/11' 2023-10-30T14:22:56.896 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/127/16' 2023-10-30T14:22:56.896 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/127/14' 2023-10-30T14:22:56.896 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/127/18' 2023-10-30T14:22:56.897 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/127/13' 2023-10-30T14:22:56.897 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/127/17' 2023-10-30T14:22:56.897 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/127' 2023-10-30T14:22:56.897 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/108/19' 2023-10-30T14:22:56.897 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/108/15' 2023-10-30T14:22:56.898 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/108/20' 2023-10-30T14:22:56.898 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/108/12' 2023-10-30T14:22:56.898 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/108/10' 2023-10-30T14:22:56.898 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/108/11' 2023-10-30T14:22:56.898 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/108/16' 2023-10-30T14:22:56.899 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/108/14' 2023-10-30T14:22:56.899 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/108/18' 2023-10-30T14:22:56.899 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/108/13' 2023-10-30T14:22:56.899 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/108/17' 2023-10-30T14:22:56.899 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/108' 2023-10-30T14:22:56.900 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/183/19' 2023-10-30T14:22:56.900 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/183/15' 2023-10-30T14:22:56.900 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/183/20' 2023-10-30T14:22:56.900 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/183/12' 2023-10-30T14:22:56.900 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/183/10' 2023-10-30T14:22:56.900 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/183/11' 2023-10-30T14:22:56.901 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/183/16' 2023-10-30T14:22:56.901 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/183/14' 2023-10-30T14:22:56.901 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/183/18' 2023-10-30T14:22:56.901 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/183/13' 2023-10-30T14:22:56.901 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/183/17' 2023-10-30T14:22:56.902 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/183' 2023-10-30T14:22:56.902 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/177/19' 2023-10-30T14:22:56.902 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/177/15' 2023-10-30T14:22:56.902 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/177/20' 2023-10-30T14:22:56.902 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/177/12' 2023-10-30T14:22:56.903 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/177/10' 2023-10-30T14:22:56.903 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/177/11' 2023-10-30T14:22:56.903 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/177/16' 2023-10-30T14:22:56.903 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/177/14' 2023-10-30T14:22:56.903 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/177/18' 2023-10-30T14:22:56.903 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/177/13' 2023-10-30T14:22:56.904 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/177/17' 2023-10-30T14:22:56.904 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/177' 2023-10-30T14:22:56.904 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/169/19' 2023-10-30T14:22:56.904 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/169/15' 2023-10-30T14:22:56.904 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/169/20' 2023-10-30T14:22:56.905 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/169/12' 2023-10-30T14:22:56.905 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/169/10' 2023-10-30T14:22:56.905 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/169/11' 2023-10-30T14:22:56.905 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/169/16' 2023-10-30T14:22:56.905 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/169/14' 2023-10-30T14:22:56.906 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/169/18' 2023-10-30T14:22:56.906 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/169/13' 2023-10-30T14:22:56.906 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/169/17' 2023-10-30T14:22:56.906 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/169' 2023-10-30T14:22:56.906 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/154/19' 2023-10-30T14:22:56.906 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/154/15' 2023-10-30T14:22:56.907 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/154/20' 2023-10-30T14:22:56.907 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/154/12' 2023-10-30T14:22:56.907 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/154/10' 2023-10-30T14:22:56.907 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/154/11' 2023-10-30T14:22:56.907 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/154/16' 2023-10-30T14:22:56.908 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/154/14' 2023-10-30T14:22:56.908 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/154/18' 2023-10-30T14:22:56.908 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/154/13' 2023-10-30T14:22:56.908 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/154/17' 2023-10-30T14:22:56.908 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/154' 2023-10-30T14:22:56.909 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/171/19' 2023-10-30T14:22:56.909 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/171/15' 2023-10-30T14:22:56.909 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/171/20' 2023-10-30T14:22:56.909 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/171/12' 2023-10-30T14:22:56.909 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/171/10' 2023-10-30T14:22:56.910 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/171/11' 2023-10-30T14:22:56.910 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/171/16' 2023-10-30T14:22:56.910 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/171/14' 2023-10-30T14:22:56.910 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/171/18' 2023-10-30T14:22:56.910 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/171/13' 2023-10-30T14:22:56.911 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/171/17' 2023-10-30T14:22:56.911 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/171' 2023-10-30T14:22:56.911 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/139/19' 2023-10-30T14:22:56.911 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/139/15' 2023-10-30T14:22:56.911 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/139/20' 2023-10-30T14:22:58.272 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/139/12' 2023-10-30T14:22:58.272 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/139/10' 2023-10-30T14:22:58.272 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/139/11' 2023-10-30T14:22:58.272 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/139/16' 2023-10-30T14:22:58.273 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/139/14' 2023-10-30T14:22:58.273 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/139/18' 2023-10-30T14:22:58.273 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/139/13' 2023-10-30T14:22:58.273 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/139/17' 2023-10-30T14:22:58.273 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/139' 2023-10-30T14:22:58.274 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/102/19' 2023-10-30T14:22:58.274 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/102/15' 2023-10-30T14:22:58.274 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/102/20' 2023-10-30T14:22:58.274 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/102/12' 2023-10-30T14:22:58.274 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/102/10' 2023-10-30T14:22:58.274 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/102/11' 2023-10-30T14:22:58.275 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/102/16' 2023-10-30T14:22:58.275 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/102/14' 2023-10-30T14:22:58.275 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/102/18' 2023-10-30T14:22:58.275 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/102/13' 2023-10-30T14:22:58.275 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/102/17' 2023-10-30T14:22:58.276 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/102' 2023-10-30T14:22:58.276 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/129/19' 2023-10-30T14:22:58.276 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/129/15' 2023-10-30T14:22:58.276 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/129/20' 2023-10-30T14:22:58.276 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/129/12' 2023-10-30T14:22:58.277 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/129/10' 2023-10-30T14:22:58.277 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/129/11' 2023-10-30T14:22:58.277 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/129/16' 2023-10-30T14:22:58.277 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/129/14' 2023-10-30T14:22:58.277 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/129/18' 2023-10-30T14:22:58.277 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/129/13' 2023-10-30T14:22:58.278 INFO:tasks.workunit.client.0.smithi057.stdout:removed './copyofsnap1/test1/129/17' 2023-10-30T14:22:58.278 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1/129' 2023-10-30T14:22:58.278 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1/test1' 2023-10-30T14:22:58.278 INFO:tasks.workunit.client.0.smithi057.stdout:removed directory './copyofsnap1' 2023-10-30T14:22:58.279 INFO:tasks.workunit.client.0.smithi057.stdout:OK 2023-10-30T14:22:58.280 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T14:22:58.280 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T14:22:58.323 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2023-10-30T14:22:58.323 DEBUG:teuthology.orchestra.run.smithi057:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2023-10-30T14:22:58.386 INFO:tasks.workunit.client.0.smithi057.stderr:+ set -e 2023-10-30T14:22:58.387 INFO:tasks.workunit.client.0.smithi057.stderr:+ touch foo 2023-10-30T14:22:58.389 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod +x foo 2023-10-30T14:22:58.390 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/s 2023-10-30T14:22:59.314 INFO:tasks.workunit.client.0.smithi057.stderr:+ find .snap/s/foo -executable 2023-10-30T14:22:59.314 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep foo 2023-10-30T14:22:59.317 INFO:tasks.workunit.client.0.smithi057.stdout:.snap/s/foo 2023-10-30T14:22:59.318 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir .snap/s 2023-10-30T14:22:59.329 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm foo 2023-10-30T14:22:59.335 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo OK 2023-10-30T14:22:59.336 INFO:tasks.workunit.client.0.smithi057.stdout:OK 2023-10-30T14:22:59.336 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T14:22:59.337 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T14:22:59.410 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2023-10-30T14:22:59.410 DEBUG:teuthology.orchestra.run.smithi057:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2023-10-30T14:23:01.176 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:23:01.180 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:23:02.998 INFO:tasks.workunit.client.0.smithi057.stderr:+ set -e 2023-10-30T14:23:02.998 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir foo 2023-10-30T14:23:03.001 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo x 2023-10-30T14:23:03.001 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:03.592 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir foo/.snap/ss 2023-10-30T14:23:03.604 INFO:tasks.workunit.client.0.smithi057.stderr:+ ln foo/x foo/xx 2023-10-30T14:23:03.607 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat foo/.snap/ss/x 2023-10-30T14:23:03.615 INFO:tasks.workunit.client.0.smithi057.stdout:x 2023-10-30T14:23:03.615 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir foo/.snap/ss 2023-10-30T14:23:03.625 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo a 2023-10-30T14:23:03.626 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo b 2023-10-30T14:23:03.626 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir foo/.snap/s 2023-10-30T14:23:03.641 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat foo/.snap/s/a 2023-10-30T14:23:03.653 INFO:tasks.workunit.client.0.smithi057.stderr:+ r=a 2023-10-30T14:23:03.653 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ -z a ] 2023-10-30T14:23:03.653 INFO:tasks.workunit.client.0.smithi057.stderr:+ ln foo/b foo/b2 2023-10-30T14:23:03.655 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat foo/.snap/s/b 2023-10-30T14:23:03.665 INFO:tasks.workunit.client.0.smithi057.stdout:b 2023-10-30T14:23:03.666 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo this used to hang: 2023-10-30T14:23:03.666 INFO:tasks.workunit.client.0.smithi057.stdout:this used to hang: 2023-10-30T14:23:03.666 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo more 2023-10-30T14:23:03.667 INFO:tasks.workunit.client.0.smithi057.stdout:oh, it didn't hang! good job. 2023-10-30T14:23:03.667 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo oh, it didn't hang! good job. 2023-10-30T14:23:03.668 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat foo/b 2023-10-30T14:23:03.668 INFO:tasks.workunit.client.0.smithi057.stdout:b 2023-10-30T14:23:03.668 INFO:tasks.workunit.client.0.smithi057.stdout:more 2023-10-30T14:23:03.668 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir foo/.snap/s 2023-10-30T14:23:03.677 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm -r foo 2023-10-30T14:23:03.697 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo OK 2023-10-30T14:23:03.698 INFO:tasks.workunit.client.0.smithi057.stdout:OK 2023-10-30T14:23:03.698 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T14:23:03.699 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T14:23:04.475 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2023-10-30T14:23:04.475 DEBUG:teuthology.orchestra.run.smithi057:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2023-10-30T14:23:07.990 INFO:tasks.workunit.client.0.smithi057.stderr:+ set -e 2023-10-30T14:23:07.991 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:07.993 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/bar 2023-10-30T14:23:08.827 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/foo 2023-10-30T14:23:08.838 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/foo 2023-10-30T14:23:08.849 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:08.853 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/bar/a 2023-10-30T14:23:12.982 INFO:tasks.workunit.client.0.smithi057.stdout: File: .snap/bar/a 2023-10-30T14:23:12.982 INFO:tasks.workunit.client.0.smithi057.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-10-30T14:23:12.982 INFO:tasks.workunit.client.0.smithi057.stdout:Device: fah/250d Inode: 1099511633266 Links: 2 2023-10-30T14:23:12.983 INFO:tasks.workunit.client.0.smithi057.stdout:Access: (0775/drwxrwxr-x) Uid: ( 1000/ ubuntu) Gid: ( 1271/ ubuntu) 2023-10-30T14:23:12.983 INFO:tasks.workunit.client.0.smithi057.stdout:Access: 2023-10-30 14:23:07.990190266 +0000 2023-10-30T14:23:12.983 INFO:tasks.workunit.client.0.smithi057.stdout:Modify: 2023-10-30 14:23:07.990190266 +0000 2023-10-30T14:23:12.983 INFO:tasks.workunit.client.0.smithi057.stdout:Change: 2023-10-30 14:23:07.990190266 +0000 2023-10-30T14:23:12.983 INFO:tasks.workunit.client.0.smithi057.stdout: Birth: 2023-10-30 14:23:07.990190266 +0000 2023-10-30T14:23:12.984 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir .snap/bar 2023-10-30T14:23:12.995 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo OK 2023-10-30T14:23:12.996 INFO:tasks.workunit.client.0.smithi057.stdout:OK 2023-10-30T14:23:12.997 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T14:23:12.997 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T14:23:13.027 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2023-10-30T14:23:13.027 DEBUG:teuthology.orchestra.run.smithi057:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2023-10-30T14:23:17.992 INFO:tasks.workunit.client.0.smithi057.stderr:+ set -e 2023-10-30T14:23:17.993 INFO:tasks.workunit.client.0.smithi057.stderr:+ seq 1 20 2023-10-30T14:23:17.994 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:17.999 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:18.001 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:18.034 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:18.044 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:18.045 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:18.298 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:18.314 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:18.315 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:18.315 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:18.315 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:18.315 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:18.315 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:18.316 INFO:tasks.workunit.client.0.smithi057.stdout: 51465 ? 00:00:00 ps 2023-10-30T14:23:18.316 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:18.768 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:18.771 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:18.780 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:18.783 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:18.784 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:18.801 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:18.811 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:18.813 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:18.866 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:18.879 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:18.880 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:18.880 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:18.880 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:18.880 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:18.880 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:18.881 INFO:tasks.workunit.client.0.smithi057.stdout: 51475 ? 00:00:00 ps 2023-10-30T14:23:18.881 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:18.901 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:18.903 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:18.909 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:18.912 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:18.913 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:18.923 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:18.930 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:18.931 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:18.997 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:19.019 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:19.019 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:19.019 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:19.019 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:19.019 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:19.020 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:19.020 INFO:tasks.workunit.client.0.smithi057.stdout: 51485 ? 00:00:00 ps 2023-10-30T14:23:19.020 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:19.031 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:19.033 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:19.039 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:19.041 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:19.042 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:19.051 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:19.058 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:19.060 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:19.131 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:19.146 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:19.146 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:19.146 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:19.146 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:19.147 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:19.147 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:19.147 INFO:tasks.workunit.client.0.smithi057.stdout: 51495 ? 00:00:00 ps 2023-10-30T14:23:19.148 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:19.156 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:19.158 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:19.164 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:19.167 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:19.168 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:19.176 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:19.182 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:19.183 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:19.239 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:19.254 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:19.254 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:19.254 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:19.255 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:19.255 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:19.255 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:19.255 INFO:tasks.workunit.client.0.smithi057.stdout: 51505 ? 00:00:00 ps 2023-10-30T14:23:19.256 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:19.265 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:19.268 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:19.274 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:19.277 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:19.279 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:19.287 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:19.293 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:19.294 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:19.465 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:19.480 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:19.480 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:19.481 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:19.481 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:19.481 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:19.481 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:19.481 INFO:tasks.workunit.client.0.smithi057.stdout: 51515 ? 00:00:00 ps 2023-10-30T14:23:19.482 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:19.491 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:19.494 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:19.501 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:19.504 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:19.505 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:19.514 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:19.767 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:19.768 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:19.907 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:19.922 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:19.922 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:19.922 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:19.923 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:19.923 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:19.923 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:19.923 INFO:tasks.workunit.client.0.smithi057.stdout: 51526 ? 00:00:00 ps 2023-10-30T14:23:19.923 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:20.221 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:20.223 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:20.230 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:20.233 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:20.235 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:20.243 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:20.250 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:20.251 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:20.374 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:20.389 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:20.390 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:20.390 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:20.390 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:20.390 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:20.391 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:20.391 INFO:tasks.workunit.client.0.smithi057.stdout: 51536 ? 00:00:00 ps 2023-10-30T14:23:20.391 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:20.401 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:20.403 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:20.409 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:20.411 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:20.412 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:20.422 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:20.428 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:20.429 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:20.482 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:20.497 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:20.497 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:20.498 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:20.498 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:20.498 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:20.498 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:20.498 INFO:tasks.workunit.client.0.smithi057.stdout: 51546 ? 00:00:00 ps 2023-10-30T14:23:20.499 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:20.508 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:20.511 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:20.516 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:20.518 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:20.519 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:20.528 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:20.534 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:20.535 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:20.598 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:20.613 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:20.613 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:20.614 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:20.614 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:20.614 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:20.614 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:20.615 INFO:tasks.workunit.client.0.smithi057.stdout: 51556 ? 00:00:00 ps 2023-10-30T14:23:20.615 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:20.624 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:20.626 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:20.631 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:20.634 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:20.635 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:20.643 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:20.650 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:20.651 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:20.716 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:20.731 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:20.731 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:20.731 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:20.732 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:20.732 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:20.732 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:20.732 INFO:tasks.workunit.client.0.smithi057.stdout: 51566 ? 00:00:00 ps 2023-10-30T14:23:20.733 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:21.087 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:21.093 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:21.100 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:21.103 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:21.104 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:21.115 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:21.122 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:21.123 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:21.298 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:21.313 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:21.314 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:21.314 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:21.314 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:21.314 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:21.314 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:21.315 INFO:tasks.workunit.client.0.smithi057.stdout: 51576 ? 00:00:00 ps 2023-10-30T14:23:21.315 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:21.326 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:21.328 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:21.333 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:21.336 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:21.337 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:21.346 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:21.352 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:21.353 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:21.441 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:21.457 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:21.457 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:21.457 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:21.457 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:21.457 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:21.458 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:21.458 INFO:tasks.workunit.client.0.smithi057.stdout: 51586 ? 00:00:00 ps 2023-10-30T14:23:21.458 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:21.467 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:21.469 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:21.476 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:21.479 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:21.480 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:21.491 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:21.498 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:21.499 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:21.575 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:21.589 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:21.590 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:21.590 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:21.590 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:21.590 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:21.590 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:21.591 INFO:tasks.workunit.client.0.smithi057.stdout: 51596 ? 00:00:00 ps 2023-10-30T14:23:21.591 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:21.601 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:21.604 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:21.610 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:21.612 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:21.613 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:21.623 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:21.630 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:21.631 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:21.675 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:21.691 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:21.691 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:21.691 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:21.691 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:21.691 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:21.691 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:21.692 INFO:tasks.workunit.client.0.smithi057.stdout: 51606 ? 00:00:00 ps 2023-10-30T14:23:21.692 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:21.703 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:21.705 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:21.712 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:21.715 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:21.716 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:21.726 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:21.733 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:21.734 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:21.783 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:21.798 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:21.799 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:21.799 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:21.799 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:21.799 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:21.799 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:21.800 INFO:tasks.workunit.client.0.smithi057.stdout: 51616 ? 00:00:00 ps 2023-10-30T14:23:21.800 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:21.810 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:21.813 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:21.819 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:21.821 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:21.822 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:21.832 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:22.588 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:22.590 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:22.751 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:22.766 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:22.766 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:22.766 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:22.767 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:22.767 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:22.767 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:22.767 INFO:tasks.workunit.client.0.smithi057.stdout: 51626 ? 00:00:00 ps 2023-10-30T14:23:22.768 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:22.787 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:22.790 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:22.795 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:22.798 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:22.799 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:22.808 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:22.815 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:22.816 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:22.901 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:22.916 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:22.916 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:22.916 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:22.916 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:22.917 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:22.917 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:22.917 INFO:tasks.workunit.client.0.smithi057.stdout: 51636 ? 00:00:00 ps 2023-10-30T14:23:22.917 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:22.927 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:22.930 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:22.936 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:22.939 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:22.942 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:22.950 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:22.957 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:22.958 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:23.037 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:23.056 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:23.056 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:23.056 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:23.056 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:23.056 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:23.057 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:23.057 INFO:tasks.workunit.client.0.smithi057.stdout: 51646 ? 00:00:00 ps 2023-10-30T14:23:23.059 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:23.123 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:23.128 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:23.147 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a 2023-10-30T14:23:23.154 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/one 2023-10-30T14:23:23.155 INFO:tasks.workunit.client.0.smithi057.stderr:+ cat 2023-10-30T14:23:23.185 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir a/.snap/two 2023-10-30T14:23:23.196 INFO:tasks.workunit.client.0.smithi057.stderr:+ chmod 777 a/foo 2023-10-30T14:23:23.197 INFO:tasks.workunit.client.0.smithi057.stderr:+ sync 2023-10-30T14:23:23.361 INFO:tasks.workunit.client.0.smithi057.stderr:+ ps 2023-10-30T14:23:23.379 INFO:tasks.workunit.client.0.smithi057.stdout: PID TTY TIME CMD 2023-10-30T14:23:23.379 INFO:tasks.workunit.client.0.smithi057.stdout: 1194 ? 00:00:00 systemd 2023-10-30T14:23:23.379 INFO:tasks.workunit.client.0.smithi057.stdout: 1195 ? 00:00:00 (sd-pam) 2023-10-30T14:23:23.380 INFO:tasks.workunit.client.0.smithi057.stdout: 27864 ? 00:00:02 sshd 2023-10-30T14:23:23.380 INFO:tasks.workunit.client.0.smithi057.stdout: 51453 ? 00:00:00 timeout 2023-10-30T14:23:23.380 INFO:tasks.workunit.client.0.smithi057.stdout: 51456 ? 00:00:00 snaptest-double 2023-10-30T14:23:23.380 INFO:tasks.workunit.client.0.smithi057.stdout: 51656 ? 00:00:00 ps 2023-10-30T14:23:23.381 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T14:23:23.392 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm a/foo 2023-10-30T14:23:23.394 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir a 2023-10-30T14:23:23.399 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo OK 2023-10-30T14:23:23.400 INFO:tasks.workunit.client.0.smithi057.stdout:OK 2023-10-30T14:23:23.401 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T14:23:23.402 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T14:23:23.429 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2023-10-30T14:23:23.429 DEBUG:teuthology.orchestra.run.smithi057:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2023-10-30T14:23:23.486 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/foo 2023-10-30T14:23:23.751 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo We want ENOENT, not ESTALE, here. 2023-10-30T14:23:23.751 INFO:tasks.workunit.client.0.smithi057.stdout:We want ENOENT, not ESTALE, here. 2023-10-30T14:23:23.752 INFO:tasks.workunit.client.0.smithi057.stderr:+ seq 1 100 2023-10-30T14:23:23.753 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/1 2023-10-30T14:23:23.753 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.755 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2023-10-30T14:23:23.756 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/2 2023-10-30T14:23:23.757 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.758 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2023-10-30T14:23:23.759 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/3 2023-10-30T14:23:23.759 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.760 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2023-10-30T14:23:23.761 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/4 2023-10-30T14:23:23.761 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.763 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2023-10-30T14:23:23.765 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/5 2023-10-30T14:23:23.766 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.766 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2023-10-30T14:23:23.767 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/6 2023-10-30T14:23:23.767 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.768 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2023-10-30T14:23:23.768 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/7 2023-10-30T14:23:23.769 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.770 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2023-10-30T14:23:23.771 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/8 2023-10-30T14:23:23.854 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.855 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2023-10-30T14:23:23.856 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/9 2023-10-30T14:23:23.856 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.857 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2023-10-30T14:23:23.857 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/10 2023-10-30T14:23:23.857 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.858 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2023-10-30T14:23:23.859 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/11 2023-10-30T14:23:23.859 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.859 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2023-10-30T14:23:23.860 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/12 2023-10-30T14:23:23.860 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.860 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2023-10-30T14:23:23.861 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/13 2023-10-30T14:23:23.861 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.862 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2023-10-30T14:23:23.863 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/14 2023-10-30T14:23:23.863 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.863 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2023-10-30T14:23:23.864 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/15 2023-10-30T14:23:23.864 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.865 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2023-10-30T14:23:23.865 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/16 2023-10-30T14:23:23.865 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.866 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2023-10-30T14:23:23.866 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/17 2023-10-30T14:23:23.867 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.867 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2023-10-30T14:23:23.868 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/18 2023-10-30T14:23:23.868 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.868 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2023-10-30T14:23:23.869 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/19 2023-10-30T14:23:23.869 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.870 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2023-10-30T14:23:23.870 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/20 2023-10-30T14:23:23.870 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.871 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2023-10-30T14:23:23.871 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/21 2023-10-30T14:23:23.872 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.872 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2023-10-30T14:23:23.873 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/22 2023-10-30T14:23:23.873 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.873 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2023-10-30T14:23:23.874 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/23 2023-10-30T14:23:23.874 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.874 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2023-10-30T14:23:23.875 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/24 2023-10-30T14:23:23.875 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.875 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2023-10-30T14:23:23.876 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/25 2023-10-30T14:23:23.876 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.877 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2023-10-30T14:23:23.877 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/26 2023-10-30T14:23:23.878 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.878 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2023-10-30T14:23:23.879 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/27 2023-10-30T14:23:23.879 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.879 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2023-10-30T14:23:23.880 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/28 2023-10-30T14:23:23.880 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.880 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2023-10-30T14:23:23.881 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/29 2023-10-30T14:23:23.881 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.882 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2023-10-30T14:23:23.882 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/30 2023-10-30T14:23:23.883 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.883 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2023-10-30T14:23:23.884 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/31 2023-10-30T14:23:23.884 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.884 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2023-10-30T14:23:23.885 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/32 2023-10-30T14:23:23.885 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.885 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2023-10-30T14:23:23.886 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/33 2023-10-30T14:23:23.886 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.887 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2023-10-30T14:23:23.888 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/34 2023-10-30T14:23:23.888 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.888 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2023-10-30T14:23:23.889 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/35 2023-10-30T14:23:23.889 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.889 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2023-10-30T14:23:23.890 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/36 2023-10-30T14:23:23.890 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.891 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2023-10-30T14:23:23.891 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/37 2023-10-30T14:23:23.892 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.892 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2023-10-30T14:23:23.893 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/38 2023-10-30T14:23:23.893 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.893 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2023-10-30T14:23:23.894 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/39 2023-10-30T14:23:23.894 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.895 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2023-10-30T14:23:23.895 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/40 2023-10-30T14:23:23.895 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.896 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2023-10-30T14:23:23.896 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/41 2023-10-30T14:23:23.897 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.897 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2023-10-30T14:23:23.898 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/42 2023-10-30T14:23:23.898 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.898 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2023-10-30T14:23:23.899 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/43 2023-10-30T14:23:23.899 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.900 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2023-10-30T14:23:23.900 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/44 2023-10-30T14:23:23.901 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.901 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2023-10-30T14:23:23.902 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/45 2023-10-30T14:23:23.902 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.902 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2023-10-30T14:23:23.903 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/46 2023-10-30T14:23:23.903 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.904 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2023-10-30T14:23:23.904 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/47 2023-10-30T14:23:23.904 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.905 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2023-10-30T14:23:23.906 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/48 2023-10-30T14:23:23.906 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.906 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2023-10-30T14:23:23.907 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/49 2023-10-30T14:23:23.907 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.908 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2023-10-30T14:23:23.908 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/50 2023-10-30T14:23:23.909 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.909 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2023-10-30T14:23:23.910 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/51 2023-10-30T14:23:23.910 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.910 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2023-10-30T14:23:23.911 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/52 2023-10-30T14:23:23.911 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.911 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2023-10-30T14:23:23.912 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/53 2023-10-30T14:23:23.912 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.913 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2023-10-30T14:23:23.913 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/54 2023-10-30T14:23:23.913 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.914 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2023-10-30T14:23:23.914 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/55 2023-10-30T14:23:23.915 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.915 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2023-10-30T14:23:23.916 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/56 2023-10-30T14:23:23.916 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.916 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2023-10-30T14:23:23.917 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/57 2023-10-30T14:23:23.917 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.918 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2023-10-30T14:23:23.918 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/58 2023-10-30T14:23:23.919 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.919 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2023-10-30T14:23:23.920 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/59 2023-10-30T14:23:23.920 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.920 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2023-10-30T14:23:23.921 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/60 2023-10-30T14:23:23.921 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.922 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2023-10-30T14:23:23.922 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/61 2023-10-30T14:23:23.923 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.923 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2023-10-30T14:23:23.924 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/62 2023-10-30T14:23:23.924 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.924 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2023-10-30T14:23:23.925 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/63 2023-10-30T14:23:23.925 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.925 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2023-10-30T14:23:23.926 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/64 2023-10-30T14:23:23.926 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.927 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2023-10-30T14:23:23.928 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/65 2023-10-30T14:23:23.928 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.928 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2023-10-30T14:23:23.929 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/66 2023-10-30T14:23:23.929 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.930 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2023-10-30T14:23:23.930 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/67 2023-10-30T14:23:23.930 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.931 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2023-10-30T14:23:23.932 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/68 2023-10-30T14:23:23.932 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.932 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2023-10-30T14:23:23.933 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/69 2023-10-30T14:23:23.933 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.933 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2023-10-30T14:23:23.934 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/70 2023-10-30T14:23:23.934 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.935 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2023-10-30T14:23:23.935 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/71 2023-10-30T14:23:23.936 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.936 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2023-10-30T14:23:23.937 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/72 2023-10-30T14:23:23.937 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.937 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2023-10-30T14:23:23.938 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/73 2023-10-30T14:23:23.938 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.939 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2023-10-30T14:23:23.939 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/74 2023-10-30T14:23:23.939 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.940 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2023-10-30T14:23:23.940 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/75 2023-10-30T14:23:23.941 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.941 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2023-10-30T14:23:23.942 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/76 2023-10-30T14:23:23.942 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.942 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2023-10-30T14:23:23.943 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/77 2023-10-30T14:23:23.943 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.943 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2023-10-30T14:23:23.944 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/78 2023-10-30T14:23:23.944 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.945 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2023-10-30T14:23:23.945 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/79 2023-10-30T14:23:23.946 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.946 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2023-10-30T14:23:23.947 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/80 2023-10-30T14:23:23.947 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.947 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2023-10-30T14:23:23.948 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/81 2023-10-30T14:23:23.948 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.949 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2023-10-30T14:23:23.949 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/82 2023-10-30T14:23:23.949 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.950 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2023-10-30T14:23:23.950 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/83 2023-10-30T14:23:23.951 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.951 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2023-10-30T14:23:23.952 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/84 2023-10-30T14:23:23.952 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.952 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2023-10-30T14:23:23.953 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/85 2023-10-30T14:23:23.953 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.953 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2023-10-30T14:23:23.954 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/86 2023-10-30T14:23:23.954 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.955 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2023-10-30T14:23:23.955 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/87 2023-10-30T14:23:23.956 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.956 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2023-10-30T14:23:23.957 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/88 2023-10-30T14:23:23.957 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.959 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2023-10-30T14:23:23.960 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/89 2023-10-30T14:23:23.960 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.961 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2023-10-30T14:23:23.962 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/90 2023-10-30T14:23:23.962 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.963 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2023-10-30T14:23:23.964 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/91 2023-10-30T14:23:23.965 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.966 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2023-10-30T14:23:23.970 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/92 2023-10-30T14:23:23.971 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.971 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2023-10-30T14:23:23.972 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/93 2023-10-30T14:23:23.972 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.972 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2023-10-30T14:23:23.973 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/94 2023-10-30T14:23:23.973 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.974 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2023-10-30T14:23:23.974 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/95 2023-10-30T14:23:23.974 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.975 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2023-10-30T14:23:23.976 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/96 2023-10-30T14:23:23.977 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.978 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2023-10-30T14:23:23.979 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/97 2023-10-30T14:23:23.979 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.980 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2023-10-30T14:23:23.981 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/98 2023-10-30T14:23:23.981 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.982 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2023-10-30T14:23:23.983 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/99 2023-10-30T14:23:23.984 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.985 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2023-10-30T14:23:23.986 INFO:tasks.workunit.client.0.smithi057.stderr:+ stat .snap/foo/100 2023-10-30T14:23:23.986 INFO:tasks.workunit.client.0.smithi057.stderr:+ grep No such file 2023-10-30T14:23:23.987 INFO:tasks.workunit.client.0.smithi057.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2023-10-30T14:23:23.988 INFO:tasks.workunit.client.0.smithi057.stderr:+ rmdir .snap/foo 2023-10-30T14:23:23.995 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo OK 2023-10-30T14:23:23.995 INFO:tasks.workunit.client.0.smithi057.stdout:OK 2023-10-30T14:23:23.996 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T14:23:23.996 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T14:23:24.062 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2023-10-30T14:23:24.062 DEBUG:teuthology.orchestra.run.smithi057:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2023-10-30T14:23:28.418 INFO:tasks.workunit.client.0.smithi057.stderr:+ set -e 2023-10-30T14:23:28.419 INFO:tasks.workunit.client.0.smithi057.stderr:+ sudo git config --global http.sslVerify false 2023-10-30T14:23:28.437 INFO:tasks.workunit.client.0.smithi057.stderr:+ sudo git config --global http.postBuffer 1048576000 2023-10-30T14:23:28.452 INFO:tasks.workunit.client.0.smithi057.stderr:+ retried=false 2023-10-30T14:23:28.452 INFO:tasks.workunit.client.0.smithi057.stderr:+ trap -- retry EXIT 2023-10-30T14:23:28.452 INFO:tasks.workunit.client.0.smithi057.stderr:+ rm -rf ceph 2023-10-30T14:23:28.453 INFO:tasks.workunit.client.0.smithi057.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2023-10-30T14:23:28.457 INFO:tasks.workunit.client.0.smithi057.stderr:Cloning into 'ceph'... 2023-10-30T14:23:31.237 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:23:31.242 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:23:39.427 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:39.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc6749f4640 1 Processor -- start 2023-10-30T14:23:39.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc6749f4640 1 -- start start 2023-10-30T14:23:39.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc6749f4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6700c8eb0 0x7fc67015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:39.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc6749f4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc67015dcc0 con 0x7fc67015ac70 2023-10-30T14:23:39.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc6749f4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc670159d80 con 0x7fc67006aeb0 2023-10-30T14:23:39.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc6749f4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc670159800 con 0x7fc6700c8eb0 2023-10-30T14:23:39.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc667fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc67015ac70 0x7fc67015a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:55714/0 (socket says 172.21.15.57:55714) 2023-10-30T14:23:39.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc667fff640 1 -- 172.21.15.57:0/4145234671 learned_addr learned my addr 172.21.15.57:0/4145234671 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:23:39.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66ed9d640 1 --2- 172.21.15.57:0/4145234671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6700c8eb0 0x7fc67015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:39.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66e59c640 1 -- 172.21.15.57:0/4145234671 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231216960 0 0) 0x7fc658001120 con 0x7fc67015ac70 2023-10-30T14:23:39.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66e59c640 1 -- 172.21.15.57:0/4145234671 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc65c003530 con 0x7fc67015ac70 2023-10-30T14:23:39.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66e59c640 1 -- 172.21.15.57:0/4145234671 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3851336800 0 0) 0x7fc65c003530 con 0x7fc67015ac70 2023-10-30T14:23:39.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66e59c640 1 -- 172.21.15.57:0/4145234671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6700c8eb0 msgr2=0x7fc67015cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:39.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66e59c640 1 --2- 172.21.15.57:0/4145234671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6700c8eb0 0x7fc67015cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:39.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66e59c640 1 -- 172.21.15.57:0/4145234671 >> v1:172.21.15.88:6789/0 conn(0x7fc67006aeb0 legacy=0x7fc670069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:39.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66e59c640 1 -- 172.21.15.57:0/4145234671 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc67015d420 con 0x7fc67015ac70 2023-10-30T14:23:39.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66e59c640 1 -- 172.21.15.57:0/4145234671 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fc658001810 con 0x7fc67015ac70 2023-10-30T14:23:39.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66e59c640 1 -- 172.21.15.57:0/4145234671 >> v1:172.21.15.57:6789/0 conn(0x7fc67015ac70 legacy=0x7fc67015a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:39.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66e59c640 1 --2- 172.21.15.57:0/4145234671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc65c0039b0 0x7fc65c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:39.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66e59c640 1 --2- 172.21.15.57:0/4145234671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6700c8eb0 0x7fc65c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:39.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66e59c640 1 --2- 172.21.15.57:0/4145234671 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c004d10 0x7fc65c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:39.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66e59c640 1 -- 172.21.15.57:0/4145234671 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc670159800 con 0x7fc65c004d10 2023-10-30T14:23:39.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66e59c640 1 -- 172.21.15.57:0/4145234671 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c00b990 con 0x7fc65c0039b0 2023-10-30T14:23:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66e59c640 1 -- 172.21.15.57:0/4145234671 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c00bb50 con 0x7fc6700c8eb0 2023-10-30T14:23:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66ed9d640 1 --2- 172.21.15.57:0/4145234671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc65c0039b0 0x7fc65c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66f59e640 1 --2- 172.21.15.57:0/4145234671 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c004d10 0x7fc65c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc667fff640 1 --2- 172.21.15.57:0/4145234671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6700c8eb0 0x7fc65c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66f59e640 1 -- 172.21.15.57:0/4145234671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6700c8eb0 msgr2=0x7fc65c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:39.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66f59e640 1 --2- 172.21.15.57:0/4145234671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6700c8eb0 0x7fc65c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:39.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66f59e640 1 -- 172.21.15.57:0/4145234671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc65c0039b0 msgr2=0x7fc65c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:39.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66f59e640 1 --2- 172.21.15.57:0/4145234671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc65c0039b0 0x7fc65c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:39.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.581+0000 7fc66f59e640 1 -- 172.21.15.57:0/4145234671 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c003530 con 0x7fc65c004d10 2023-10-30T14:23:39.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.585+0000 7fc66f59e640 1 --2- 172.21.15.57:0/4145234671 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c004d10 0x7fc65c005190 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc668001f60 tx=0x7fc66801c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcf70a5f49bb5140 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:39.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.585+0000 7fc66e59c640 1 -- 172.21.15.57:0/4145234671 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6680a5750 con 0x7fc65c004d10 2023-10-30T14:23:39.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.585+0000 7fc66e59c640 1 -- 172.21.15.57:0/4145234671 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6680a58b0 con 0x7fc65c004d10 2023-10-30T14:23:39.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.585+0000 7fc66e59c640 1 -- 172.21.15.57:0/4145234671 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6680a5c60 con 0x7fc65c004d10 2023-10-30T14:23:39.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.585+0000 7fc6749f4640 1 -- 172.21.15.57:0/4145234671 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c004d10 msgr2=0x7fc65c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:39.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.585+0000 7fc6749f4640 1 --2- 172.21.15.57:0/4145234671 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c004d10 0x7fc65c005190 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc668001f60 tx=0x7fc66801c860 comp rx=0 tx=0).stop 2023-10-30T14:23:39.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.585+0000 7fc66f59e640 1 -- 172.21.15.57:0/4145234671 reap_dead start 2023-10-30T14:23:39.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.585+0000 7fc6749f4640 1 -- 172.21.15.57:0/4145234671 shutdown_connections 2023-10-30T14:23:39.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.585+0000 7fc6749f4640 1 -- 172.21.15.57:0/4145234671 >> 172.21.15.57:0/4145234671 conn(0x7fc6700cef00 msgr2=0x7fc6700cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:39.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.585+0000 7fc6749f4640 1 -- 172.21.15.57:0/4145234671 shutdown_connections 2023-10-30T14:23:39.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.585+0000 7fc6749f4640 1 -- 172.21.15.57:0/4145234671 wait complete. 2023-10-30T14:23:39.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc6749f4640 1 Processor -- start 2023-10-30T14:23:39.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc6749f4640 1 -- start start 2023-10-30T14:23:39.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc6749f4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c004d10 0x7fc6700c76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:39.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc6749f4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6701500c0 0x7fc6701ef790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:39.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc6749f4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6701efcd0 0x7fc6701564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:39.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc6749f4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6701ef300 con 0x7fc65c004d10 2023-10-30T14:23:39.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc6749f4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6701f0420 con 0x7fc6701efcd0 2023-10-30T14:23:39.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc6749f4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6701eefb0 con 0x7fc6701500c0 2023-10-30T14:23:39.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc66ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c004d10 0x7fc6700c76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:39.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc66ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c004d10 0x7fc6700c76e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42538/0 (socket says 172.21.15.57:42538) 2023-10-30T14:23:39.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc66ed9d640 1 -- 172.21.15.57:0/284662425 learned_addr learned my addr 172.21.15.57:0/284662425 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:23:39.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc66f59e640 1 --2- 172.21.15.57:0/284662425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6701efcd0 0x7fc6701564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:39.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc667fff640 1 --2- 172.21.15.57:0/284662425 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6701500c0 0x7fc6701ef790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:39.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc66ed9d640 1 -- 172.21.15.57:0/284662425 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6701500c0 msgr2=0x7fc6701ef790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:39.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc66ed9d640 1 --2- 172.21.15.57:0/284662425 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6701500c0 0x7fc6701ef790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:39.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc66ed9d640 1 -- 172.21.15.57:0/284662425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6701efcd0 msgr2=0x7fc6701564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:39.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc66ed9d640 1 --2- 172.21.15.57:0/284662425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6701efcd0 0x7fc6701564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:39.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc66ed9d640 1 -- 172.21.15.57:0/284662425 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc67014e390 con 0x7fc65c004d10 2023-10-30T14:23:39.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc66ed9d640 1 --2- 172.21.15.57:0/284662425 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c004d10 0x7fc6700c76e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc6600119d0 tx=0x7fc660011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:39.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc6677fe640 1 -- 172.21.15.57:0/284662425 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66000fb50 con 0x7fc65c004d10 2023-10-30T14:23:39.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc6677fe640 1 -- 172.21.15.57:0/284662425 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66000fcb0 con 0x7fc65c004d10 2023-10-30T14:23:39.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc6677fe640 1 -- 172.21.15.57:0/284662425 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc660014440 con 0x7fc65c004d10 2023-10-30T14:23:39.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc6749f4640 1 -- 172.21.15.57:0/284662425 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6701508b0 con 0x7fc65c004d10 2023-10-30T14:23:39.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.589+0000 7fc6749f4640 1 -- 172.21.15.57:0/284662425 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6700c8780 con 0x7fc65c004d10 2023-10-30T14:23:39.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.593+0000 7fc6749f4640 1 -- 172.21.15.57:0/284662425 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc67004e170 con 0x7fc65c004d10 2023-10-30T14:23:39.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.593+0000 7fc6677fe640 1 -- 172.21.15.57:0/284662425 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc6600145a0 con 0x7fc65c004d10 2023-10-30T14:23:39.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.593+0000 7fc6677fe640 1 --2- 172.21.15.57:0/284662425 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc650066520 0x7fc6500689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:39.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.593+0000 7fc6677fe640 1 -- 172.21.15.57:0/284662425 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fc6600848d0 con 0x7fc65c004d10 2023-10-30T14:23:39.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.593+0000 7fc667fff640 1 --2- 172.21.15.57:0/284662425 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc650066520 0x7fc6500689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:39.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.593+0000 7fc667fff640 1 --2- 172.21.15.57:0/284662425 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc650066520 0x7fc6500689e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc658001280 tx=0x7fc658009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:39.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.597+0000 7fc6677fe640 1 -- 172.21.15.57:0/284662425 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc660051c40 con 0x7fc65c004d10 2023-10-30T14:23:39.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.793+0000 7fc6749f4640 1 -- 172.21.15.57:0/284662425 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6700ca560 con 0x7fc65c004d10 2023-10-30T14:23:39.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.793+0000 7fc6677fe640 1 -- 172.21.15.57:0/284662425 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6508 (secure 0 0 0) 0x7fc660055260 con 0x7fc65c004d10 2023-10-30T14:23:39.795 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:23:39.795 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:22:37.974156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5133},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:39.796 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 66 2023-10-30T14:23:39.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.793+0000 7fc6749f4640 1 -- 172.21.15.57:0/284662425 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc650066520 msgr2=0x7fc6500689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:39.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.793+0000 7fc6749f4640 1 --2- 172.21.15.57:0/284662425 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc650066520 0x7fc6500689e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc658001280 tx=0x7fc658009040 comp rx=0 tx=0).stop 2023-10-30T14:23:39.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.793+0000 7fc6749f4640 1 -- 172.21.15.57:0/284662425 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c004d10 msgr2=0x7fc6700c76e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:39.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.793+0000 7fc6749f4640 1 --2- 172.21.15.57:0/284662425 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c004d10 0x7fc6700c76e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc6600119d0 tx=0x7fc660011e90 comp rx=0 tx=0).stop 2023-10-30T14:23:39.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.793+0000 7fc6749f4640 1 -- 172.21.15.57:0/284662425 shutdown_connections 2023-10-30T14:23:39.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.793+0000 7fc6749f4640 1 --2- 172.21.15.57:0/284662425 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc650066520 0x7fc6500689e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:39.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.793+0000 7fc6749f4640 1 --2- 172.21.15.57:0/284662425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6701efcd0 0x7fc6701564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:39.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.793+0000 7fc6749f4640 1 --2- 172.21.15.57:0/284662425 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc6701500c0 0x7fc6701ef790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:39.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.793+0000 7fc6749f4640 1 --2- 172.21.15.57:0/284662425 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc65c004d10 0x7fc6700c76e0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:39.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.793+0000 7fc6749f4640 1 -- 172.21.15.57:0/284662425 >> 172.21.15.57:0/284662425 conn(0x7fc6700cef00 msgr2=0x7fc670069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:39.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.793+0000 7fc6749f4640 1 -- 172.21.15.57:0/284662425 shutdown_connections 2023-10-30T14:23:39.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:39.793+0000 7fc6749f4640 1 -- 172.21.15.57:0/284662425 wait complete. 2023-10-30T14:23:39.815 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=1) 2023-10-30T14:23:39.816 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T14:23:39.816 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:23:39.832 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T14:23:39.833 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:40.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.017+0000 7f64d126b640 1 Processor -- start 2023-10-30T14:23:40.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.017+0000 7f64d126b640 1 -- start start 2023-10-30T14:23:40.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64d126b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f64cc06fdf0 0x7f64cc155c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:40.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64d126b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64cc0c55f0 con 0x7f64cc0c6320 2023-10-30T14:23:40.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64d126b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64cc1567b0 con 0x7f64cc14fc50 2023-10-30T14:23:40.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64d126b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64cc0c5f00 con 0x7f64cc06fdf0 2023-10-30T14:23:40.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64c3fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f64cc0c6320 0x7f64cc06be90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:55724/0 (socket says 172.21.15.57:55724) 2023-10-30T14:23:40.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64c3fff640 1 -- 172.21.15.57:0/1515626036 learned_addr learned my addr 172.21.15.57:0/1515626036 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:23:40.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64cb7fe640 1 --2- 172.21.15.57:0/1515626036 >> v2:172.21.15.88:3300/0 conn(0x7f64cc06fdf0 0x7f64cc155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:40.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64ca7fc640 1 -- 172.21.15.57:0/1515626036 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652618513 0 0) 0x7f64cc0c55f0 con 0x7f64cc0c6320 2023-10-30T14:23:40.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64ca7fc640 1 -- 172.21.15.57:0/1515626036 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b8003530 con 0x7f64cc0c6320 2023-10-30T14:23:40.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64ca7fc640 1 -- 172.21.15.57:0/1515626036 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535505252 0 0) 0x7f64cc1567b0 con 0x7f64cc14fc50 2023-10-30T14:23:40.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64ca7fc640 1 -- 172.21.15.57:0/1515626036 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64cc0c55f0 con 0x7f64cc14fc50 2023-10-30T14:23:40.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64ca7fc640 1 -- 172.21.15.57:0/1515626036 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3526986326 0 0) 0x7f64b8003530 con 0x7f64cc0c6320 2023-10-30T14:23:40.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64ca7fc640 1 -- 172.21.15.57:0/1515626036 >> v2:172.21.15.88:3300/0 conn(0x7f64cc06fdf0 msgr2=0x7f64cc155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:40.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64ca7fc640 1 --2- 172.21.15.57:0/1515626036 >> v2:172.21.15.88:3300/0 conn(0x7f64cc06fdf0 0x7f64cc155c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:40.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64ca7fc640 1 -- 172.21.15.57:0/1515626036 >> v1:172.21.15.88:6789/0 conn(0x7f64cc14fc50 legacy=0x7f64cc070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:40.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64ca7fc640 1 -- 172.21.15.57:0/1515626036 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc156a70 con 0x7f64cc0c6320 2023-10-30T14:23:40.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64ca7fc640 1 -- 172.21.15.57:0/1515626036 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f64b4001600 con 0x7f64cc0c6320 2023-10-30T14:23:40.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64ca7fc640 1 -- 172.21.15.57:0/1515626036 >> v1:172.21.15.57:6789/0 conn(0x7f64cc0c6320 legacy=0x7f64cc06be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:40.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64ca7fc640 1 --2- 172.21.15.57:0/1515626036 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64b8003b60 0x7f64b8003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:40.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64ca7fc640 1 --2- 172.21.15.57:0/1515626036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64b80047d0 0x7f64b8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:40.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64ca7fc640 1 --2- 172.21.15.57:0/1515626036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64b80051e0 0x7f64b800dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:40.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64ca7fc640 1 -- 172.21.15.57:0/1515626036 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc0c5f00 con 0x7f64b80051e0 2023-10-30T14:23:40.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64ca7fc640 1 -- 172.21.15.57:0/1515626036 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b800e320 con 0x7f64b80047d0 2023-10-30T14:23:40.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64ca7fc640 1 -- 172.21.15.57:0/1515626036 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b800e4e0 con 0x7f64b8003b60 2023-10-30T14:23:40.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64cb7fe640 1 --2- 172.21.15.57:0/1515626036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64b80051e0 0x7f64b800dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:40.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64c3fff640 1 --2- 172.21.15.57:0/1515626036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64b80047d0 0x7f64b8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:40.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64caffd640 1 --2- 172.21.15.57:0/1515626036 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64b8003b60 0x7f64b8003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:40.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64cb7fe640 1 -- 172.21.15.57:0/1515626036 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64b8003b60 msgr2=0x7f64b8003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:40.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64cb7fe640 1 --2- 172.21.15.57:0/1515626036 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64b8003b60 0x7f64b8003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:40.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64cb7fe640 1 -- 172.21.15.57:0/1515626036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64b80047d0 msgr2=0x7f64b8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:40.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64cb7fe640 1 --2- 172.21.15.57:0/1515626036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64b80047d0 0x7f64b8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:40.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64cb7fe640 1 -- 172.21.15.57:0/1515626036 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b8003530 con 0x7f64b80051e0 2023-10-30T14:23:40.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.021+0000 7f64c3fff640 1 --2- 172.21.15.57:0/1515626036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64b80047d0 0x7f64b8004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:23:40.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.025+0000 7f64cb7fe640 1 -- 172.21.15.57:0/1515626036 reap_dead start 2023-10-30T14:23:40.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.025+0000 7f64cb7fe640 1 --2- 172.21.15.57:0/1515626036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64b80051e0 0x7f64b800dba0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009fc0 tx=0x7f64c401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=993b35ae1797f704 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:40.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.025+0000 7f64ca7fc640 1 -- 172.21.15.57:0/1515626036 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c40a5cb0 con 0x7f64b80051e0 2023-10-30T14:23:40.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.025+0000 7f64ca7fc640 1 -- 172.21.15.57:0/1515626036 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c400f040 con 0x7f64b80051e0 2023-10-30T14:23:40.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.025+0000 7f64ca7fc640 1 -- 172.21.15.57:0/1515626036 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4003be0 con 0x7f64b80051e0 2023-10-30T14:23:40.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.025+0000 7f64d126b640 1 -- 172.21.15.57:0/1515626036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64b80051e0 msgr2=0x7f64b800dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:40.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.025+0000 7f64d126b640 1 --2- 172.21.15.57:0/1515626036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64b80051e0 0x7f64b800dba0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009fc0 tx=0x7f64c401cca0 comp rx=0 tx=0).stop 2023-10-30T14:23:40.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.025+0000 7f64d126b640 1 -- 172.21.15.57:0/1515626036 shutdown_connections 2023-10-30T14:23:40.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.025+0000 7f64d126b640 1 --2- 172.21.15.57:0/1515626036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64b80051e0 0x7f64b800dba0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:40.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.025+0000 7f64d126b640 1 -- 172.21.15.57:0/1515626036 >> 172.21.15.57:0/1515626036 conn(0x7f64cc0cac70 msgr2=0x7f64cc0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:40.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.025+0000 7f64d126b640 1 -- 172.21.15.57:0/1515626036 shutdown_connections 2023-10-30T14:23:40.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.025+0000 7f64d126b640 1 -- 172.21.15.57:0/1515626036 wait complete. 2023-10-30T14:23:40.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.025+0000 7f64d126b640 1 Processor -- start 2023-10-30T14:23:40.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64d126b640 1 -- start start 2023-10-30T14:23:40.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64d126b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64b80051e0 0x7f64cc0c32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:40.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64d126b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc155940 0x7f64cc145010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:40.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64d126b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64cc153a40 0x7f64cc0c3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:40.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64d126b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc149500 con 0x7f64b80051e0 2023-10-30T14:23:40.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64d126b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc14ee20 con 0x7f64cc153a40 2023-10-30T14:23:40.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64d126b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc0c4a50 con 0x7f64cc155940 2023-10-30T14:23:40.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64caffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64b80051e0 0x7f64cc0c32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:40.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64caffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64b80051e0 0x7f64cc0c32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42552/0 (socket says 172.21.15.57:42552) 2023-10-30T14:23:40.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64caffd640 1 -- 172.21.15.57:0/3180282456 learned_addr learned my addr 172.21.15.57:0/3180282456 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:23:40.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64c3fff640 1 --2- 172.21.15.57:0/3180282456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc155940 0x7f64cc145010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:40.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64cb7fe640 1 --2- 172.21.15.57:0/3180282456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64cc153a40 0x7f64cc0c3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:40.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64caffd640 1 -- 172.21.15.57:0/3180282456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc155940 msgr2=0x7f64cc145010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:40.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64caffd640 1 --2- 172.21.15.57:0/3180282456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc155940 0x7f64cc145010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:40.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64caffd640 1 -- 172.21.15.57:0/3180282456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64cc153a40 msgr2=0x7f64cc0c3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:40.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64caffd640 1 --2- 172.21.15.57:0/3180282456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64cc153a40 0x7f64cc0c3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:40.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64caffd640 1 -- 172.21.15.57:0/3180282456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc0c4d30 con 0x7f64b80051e0 2023-10-30T14:23:40.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64c3fff640 1 --2- 172.21.15.57:0/3180282456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc155940 0x7f64cc145010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:23:40.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64caffd640 1 --2- 172.21.15.57:0/3180282456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64b80051e0 0x7f64cc0c32e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f64bc007b70 tx=0x7f64bc00f540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:40.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64c37fe640 1 -- 172.21.15.57:0/3180282456 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64bc015610 con 0x7f64b80051e0 2023-10-30T14:23:40.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64c37fe640 1 -- 172.21.15.57:0/3180282456 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64bc0157b0 con 0x7f64b80051e0 2023-10-30T14:23:40.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64d126b640 1 -- 172.21.15.57:0/3180282456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64cc149750 con 0x7f64b80051e0 2023-10-30T14:23:40.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64c37fe640 1 -- 172.21.15.57:0/3180282456 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64bc015b60 con 0x7f64b80051e0 2023-10-30T14:23:40.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64d126b640 1 -- 172.21.15.57:0/3180282456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64cc156150 con 0x7f64b80051e0 2023-10-30T14:23:40.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.029+0000 7f64d126b640 1 -- 172.21.15.57:0/3180282456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64cc156340 con 0x7f64b80051e0 2023-10-30T14:23:40.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.033+0000 7f64c37fe640 1 -- 172.21.15.57:0/3180282456 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f64bc015cc0 con 0x7f64b80051e0 2023-10-30T14:23:40.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.033+0000 7f64c37fe640 1 --2- 172.21.15.57:0/3180282456 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f64ac0664a0 0x7f64ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:40.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.033+0000 7f64c37fe640 1 -- 172.21.15.57:0/3180282456 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f64bc087ab0 con 0x7f64b80051e0 2023-10-30T14:23:40.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.033+0000 7f64c3fff640 1 --2- 172.21.15.57:0/3180282456 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f64ac0664a0 0x7f64ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:40.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.033+0000 7f64c3fff640 1 --2- 172.21.15.57:0/3180282456 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f64ac0664a0 0x7f64ac068960 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f64cc156a40 tx=0x7f64b4000ee0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:40.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.037+0000 7f64c37fe640 1 -- 172.21.15.57:0/3180282456 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f64bc054c50 con 0x7f64b80051e0 2023-10-30T14:23:40.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.225+0000 7f64d126b640 1 -- 172.21.15.57:0/3180282456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64cc04e170 con 0x7f64b80051e0 2023-10-30T14:23:40.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.225+0000 7f64c37fe640 1 -- 172.21.15.57:0/3180282456 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6508 (secure 0 0 0) 0x7f64bc058270 con 0x7f64b80051e0 2023-10-30T14:23:40.228 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:23:40.229 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:22:37.974156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5133},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:40.229 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 66 2023-10-30T14:23:40.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.229+0000 7f64d126b640 1 -- 172.21.15.57:0/3180282456 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f64ac0664a0 msgr2=0x7f64ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:40.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.229+0000 7f64d126b640 1 --2- 172.21.15.57:0/3180282456 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f64ac0664a0 0x7f64ac068960 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f64cc156a40 tx=0x7f64b4000ee0 comp rx=0 tx=0).stop 2023-10-30T14:23:40.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.229+0000 7f64d126b640 1 -- 172.21.15.57:0/3180282456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64b80051e0 msgr2=0x7f64cc0c32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:40.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.229+0000 7f64d126b640 1 --2- 172.21.15.57:0/3180282456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64b80051e0 0x7f64cc0c32e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f64bc007b70 tx=0x7f64bc00f540 comp rx=0 tx=0).stop 2023-10-30T14:23:40.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.229+0000 7f64d126b640 1 -- 172.21.15.57:0/3180282456 shutdown_connections 2023-10-30T14:23:40.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.229+0000 7f64d126b640 1 --2- 172.21.15.57:0/3180282456 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f64ac0664a0 0x7f64ac068960 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:40.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.229+0000 7f64d126b640 1 --2- 172.21.15.57:0/3180282456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64cc153a40 0x7f64cc0c3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:40.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.229+0000 7f64d126b640 1 --2- 172.21.15.57:0/3180282456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc155940 0x7f64cc145010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:40.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.229+0000 7f64d126b640 1 --2- 172.21.15.57:0/3180282456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64b80051e0 0x7f64cc0c32e0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:40.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.229+0000 7f64d126b640 1 -- 172.21.15.57:0/3180282456 >> 172.21.15.57:0/3180282456 conn(0x7f64cc0cac70 msgr2=0x7f64cc0723f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:40.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.229+0000 7f64d126b640 1 -- 172.21.15.57:0/3180282456 shutdown_connections 2023-10-30T14:23:40.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:40.229+0000 7f64d126b640 1 -- 172.21.15.57:0/3180282456 wait complete. 2023-10-30T14:23:40.250 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:23:42.250 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:42.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.401+0000 7f25d68ea640 1 Processor -- start 2023-10-30T14:23:42.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.401+0000 7f25d68ea640 1 -- start start 2023-10-30T14:23:42.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.405+0000 7f25d68ea640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f25d0071340 0x7f25d0065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:42.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.405+0000 7f25d68ea640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d006e390 con 0x7f25d0153c10 2023-10-30T14:23:42.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.405+0000 7f25d68ea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d00660e0 con 0x7f25d0159970 2023-10-30T14:23:42.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.405+0000 7f25d68ea640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d0072390 con 0x7f25d0071340 2023-10-30T14:23:42.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.405+0000 7f25cf7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f25d0071340 0x7f25d0065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:42.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.405+0000 7f25cffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f25d0153c10 0x7f25d0070c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:55732/0 (socket says 172.21.15.57:55732) 2023-10-30T14:23:42.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.405+0000 7f25cffff640 1 -- 172.21.15.57:0/2536234490 learned_addr learned my addr 172.21.15.57:0/2536234490 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:23:42.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.405+0000 7f25cf7fe640 1 -- 172.21.15.57:0/2536234490 >> v1:172.21.15.88:6789/0 conn(0x7f25d0159970 legacy=0x7f25d0159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:42.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.405+0000 7f25cf7fe640 1 -- 172.21.15.57:0/2536234490 >> v1:172.21.15.57:6789/0 conn(0x7f25d0153c10 legacy=0x7f25d0070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:42.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.405+0000 7f25cf7fe640 1 -- 172.21.15.57:0/2536234490 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d00bec10 con 0x7f25d0071340 2023-10-30T14:23:42.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.405+0000 7f25ceffd640 1 -- 172.21.15.57:0/2536234490 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409457664 0 0) 0x7f25d006e390 con 0x7f25d0153c10 2023-10-30T14:23:42.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.409+0000 7f25cf7fe640 1 --2- 172.21.15.57:0/2536234490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25d0071340 0x7f25d0065ba0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f25c40097c0 tx=0x7f25c401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e615fbffc253de7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:42.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.409+0000 7f25ceffd640 1 -- 172.21.15.57:0/2536234490 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c40ab600 con 0x7f25d0071340 2023-10-30T14:23:42.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.409+0000 7f25ceffd640 1 -- 172.21.15.57:0/2536234490 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c40a7050 con 0x7f25d0071340 2023-10-30T14:23:42.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.409+0000 7f25ceffd640 1 -- 172.21.15.57:0/2536234490 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c40ab980 con 0x7f25d0071340 2023-10-30T14:23:42.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.409+0000 7f25d68ea640 1 -- 172.21.15.57:0/2536234490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25d0071340 msgr2=0x7f25d0065ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:42.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.409+0000 7f25d68ea640 1 --2- 172.21.15.57:0/2536234490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25d0071340 0x7f25d0065ba0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f25c40097c0 tx=0x7f25c401c3b0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.409+0000 7f25d68ea640 1 -- 172.21.15.57:0/2536234490 shutdown_connections 2023-10-30T14:23:42.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.409+0000 7f25d68ea640 1 --2- 172.21.15.57:0/2536234490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25d0071340 0x7f25d0065ba0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.409+0000 7f25d68ea640 1 -- 172.21.15.57:0/2536234490 >> 172.21.15.57:0/2536234490 conn(0x7f25d00c7f60 msgr2=0x7f25d00c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:42.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.409+0000 7f25d68ea640 1 -- 172.21.15.57:0/2536234490 shutdown_connections 2023-10-30T14:23:42.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.409+0000 7f25d68ea640 1 -- 172.21.15.57:0/2536234490 wait complete. 2023-10-30T14:23:42.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.409+0000 7f25d68ea640 1 Processor -- start 2023-10-30T14:23:42.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.409+0000 7f25d68ea640 1 -- start start 2023-10-30T14:23:42.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25d68ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25d0071340 0x7f25d01f6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:42.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25d68ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25d0153c10 0x7f25d0157a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:42.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25d68ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25d0159970 0x7f25d014faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:42.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25d68ea640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d00cb000 con 0x7f25d0153c10 2023-10-30T14:23:42.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25d68ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d0151990 con 0x7f25d0071340 2023-10-30T14:23:42.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25d68ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d00ca130 con 0x7f25d0159970 2023-10-30T14:23:42.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25cf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25d0153c10 0x7f25d0157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:42.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25cf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25d0153c10 0x7f25d0157a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42580/0 (socket says 172.21.15.57:42580) 2023-10-30T14:23:42.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25cf7fe640 1 -- 172.21.15.57:0/4152724864 learned_addr learned my addr 172.21.15.57:0/4152724864 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:23:42.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25cffff640 1 --2- 172.21.15.57:0/4152724864 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25d0071340 0x7f25d01f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:42.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25d4e87640 1 --2- 172.21.15.57:0/4152724864 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25d0159970 0x7f25d014faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:42.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25cf7fe640 1 -- 172.21.15.57:0/4152724864 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25d0159970 msgr2=0x7f25d014faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:42.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25cf7fe640 1 --2- 172.21.15.57:0/4152724864 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25d0159970 0x7f25d014faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25cf7fe640 1 -- 172.21.15.57:0/4152724864 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25d0071340 msgr2=0x7f25d01f6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:42.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25cf7fe640 1 --2- 172.21.15.57:0/4152724864 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25d0071340 0x7f25d01f6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25cf7fe640 1 -- 172.21.15.57:0/4152724864 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c4009420 con 0x7f25d0153c10 2023-10-30T14:23:42.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25cf7fe640 1 --2- 172.21.15.57:0/4152724864 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25d0153c10 0x7f25d0157a40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f25c4018fc0 tx=0x7f25c401abf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:42.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25ccff9640 1 -- 172.21.15.57:0/4152724864 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c40b7650 con 0x7f25d0153c10 2023-10-30T14:23:42.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25ccff9640 1 -- 172.21.15.57:0/4152724864 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c40b77b0 con 0x7f25d0153c10 2023-10-30T14:23:42.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25d68ea640 1 -- 172.21.15.57:0/4152724864 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d00ca380 con 0x7f25d0153c10 2023-10-30T14:23:42.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25ccff9640 1 -- 172.21.15.57:0/4152724864 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c40b7b60 con 0x7f25d0153c10 2023-10-30T14:23:42.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25d68ea640 1 -- 172.21.15.57:0/4152724864 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d00ca940 con 0x7f25d0153c10 2023-10-30T14:23:42.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25d68ea640 1 -- 172.21.15.57:0/4152724864 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d00cb140 con 0x7f25d0153c10 2023-10-30T14:23:42.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.413+0000 7f25ccff9640 1 -- 172.21.15.57:0/4152724864 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f25c40b7cc0 con 0x7f25d0153c10 2023-10-30T14:23:42.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.417+0000 7f25ccff9640 1 --2- 172.21.15.57:0/4152724864 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f25bc0664a0 0x7f25bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:42.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.417+0000 7f25ccff9640 1 -- 172.21.15.57:0/4152724864 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f25c411e990 con 0x7f25d0153c10 2023-10-30T14:23:42.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.417+0000 7f25cffff640 1 --2- 172.21.15.57:0/4152724864 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f25bc0664a0 0x7f25bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:42.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.417+0000 7f25cffff640 1 --2- 172.21.15.57:0/4152724864 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f25bc0664a0 0x7f25bc068960 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f25c0007fd0 tx=0x7f25c0007d60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:42.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.421+0000 7f25ccff9640 1 -- 172.21.15.57:0/4152724864 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f25d00cb140 con 0x7f25d0153c10 2023-10-30T14:23:42.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.609+0000 7f25d68ea640 1 -- 172.21.15.57:0/4152724864 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25d00cb140 con 0x7f25d0153c10 2023-10-30T14:23:42.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.609+0000 7f25ccff9640 1 -- 172.21.15.57:0/4152724864 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6508 (secure 0 0 0) 0x7f25d00cb140 con 0x7f25d0153c10 2023-10-30T14:23:42.613 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:23:42.613 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:22:37.974156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5133},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:42.613 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 66 2023-10-30T14:23:42.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.613+0000 7f25d68ea640 1 -- 172.21.15.57:0/4152724864 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f25bc0664a0 msgr2=0x7f25bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:42.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.613+0000 7f25d68ea640 1 --2- 172.21.15.57:0/4152724864 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f25bc0664a0 0x7f25bc068960 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f25c0007fd0 tx=0x7f25c0007d60 comp rx=0 tx=0).stop 2023-10-30T14:23:42.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.613+0000 7f25d68ea640 1 -- 172.21.15.57:0/4152724864 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25d0153c10 msgr2=0x7f25d0157a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:42.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.613+0000 7f25d68ea640 1 --2- 172.21.15.57:0/4152724864 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25d0153c10 0x7f25d0157a40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f25c4018fc0 tx=0x7f25c401abf0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.613+0000 7f25d68ea640 1 -- 172.21.15.57:0/4152724864 shutdown_connections 2023-10-30T14:23:42.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.613+0000 7f25d68ea640 1 --2- 172.21.15.57:0/4152724864 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f25bc0664a0 0x7f25bc068960 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.613+0000 7f25d68ea640 1 --2- 172.21.15.57:0/4152724864 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25d0159970 0x7f25d014faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.613+0000 7f25d68ea640 1 --2- 172.21.15.57:0/4152724864 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25d0153c10 0x7f25d0157a40 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.613+0000 7f25d68ea640 1 --2- 172.21.15.57:0/4152724864 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25d0071340 0x7f25d01f6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.613+0000 7f25d68ea640 1 -- 172.21.15.57:0/4152724864 >> 172.21.15.57:0/4152724864 conn(0x7f25d00c7f60 msgr2=0x7f25d0153260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:42.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.613+0000 7f25d68ea640 1 -- 172.21.15.57:0/4152724864 shutdown_connections 2023-10-30T14:23:42.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:42.613+0000 7f25d68ea640 1 -- 172.21.15.57:0/4152724864 wait complete. 2023-10-30T14:23:42.634 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:23:44.635 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:44.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.785+0000 7f658ab65640 1 Processor -- start 2023-10-30T14:23:44.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.785+0000 7f658ab65640 1 -- start start 2023-10-30T14:23:44.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f658ab65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6584151740 0x7f6584158040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:44.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f658ab65640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6584066160 con 0x7f6584151b20 2023-10-30T14:23:44.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f658ab65640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658415b640 con 0x7f65840c7880 2023-10-30T14:23:44.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f658ab65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6584158580 con 0x7f6584151740 2023-10-30T14:23:44.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f6583fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f6584151b20 0x7f65840c28e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50336/0 (socket says 172.21.15.57:50336) 2023-10-30T14:23:44.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f6583fff640 1 -- 172.21.15.57:0/1981932570 learned_addr learned my addr 172.21.15.57:0/1981932570 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:23:44.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f6588901640 1 --2- 172.21.15.57:0/1981932570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6584151740 0x7f6584158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:44.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f65837fe640 1 -- 172.21.15.57:0/1981932570 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401695424 0 0) 0x7f6584066160 con 0x7f6584151b20 2023-10-30T14:23:44.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f65837fe640 1 -- 172.21.15.57:0/1981932570 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6574003530 con 0x7f6584151b20 2023-10-30T14:23:44.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f65837fe640 1 -- 172.21.15.57:0/1981932570 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3135739173 0 0) 0x7f6574003530 con 0x7f6584151b20 2023-10-30T14:23:44.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f65837fe640 1 -- 172.21.15.57:0/1981932570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6584151740 msgr2=0x7f6584158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:44.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f65837fe640 1 --2- 172.21.15.57:0/1981932570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6584151740 0x7f6584158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:44.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f65837fe640 1 -- 172.21.15.57:0/1981932570 >> v1:172.21.15.88:6789/0 conn(0x7f65840c7880 legacy=0x7f65840c4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:44.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f65837fe640 1 -- 172.21.15.57:0/1981932570 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658415bad0 con 0x7f6584151b20 2023-10-30T14:23:44.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f65837fe640 1 -- 172.21.15.57:0/1981932570 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f6570001680 con 0x7f6584151b20 2023-10-30T14:23:44.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f6588901640 1 --2- 172.21.15.57:0/1981932570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6584151740 0x7f6584158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:23:44.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f65837fe640 1 -- 172.21.15.57:0/1981932570 >> v1:172.21.15.57:6789/0 conn(0x7f6584151b20 legacy=0x7f65840c28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:44.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f65837fe640 1 --2- 172.21.15.57:0/1981932570 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65740039b0 0x7f6574003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:44.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f65837fe640 1 --2- 172.21.15.57:0/1981932570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6584151740 0x7f65740047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:44.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f65837fe640 1 --2- 172.21.15.57:0/1981932570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6574004d10 0x7f6574005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:44.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f65837fe640 1 -- 172.21.15.57:0/1981932570 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6584158580 con 0x7f65740039b0 2023-10-30T14:23:44.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f65837fe640 1 -- 172.21.15.57:0/1981932570 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657400ba40 con 0x7f6574004d10 2023-10-30T14:23:44.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f65837fe640 1 -- 172.21.15.57:0/1981932570 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657400bbb0 con 0x7f6584151740 2023-10-30T14:23:44.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f6588901640 1 --2- 172.21.15.57:0/1981932570 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65740039b0 0x7f6574003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:44.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f6583fff640 1 --2- 172.21.15.57:0/1981932570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6584151740 0x7f65740047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:44.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f6589102640 1 --2- 172.21.15.57:0/1981932570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6574004d10 0x7f6574005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:44.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f6588901640 1 -- 172.21.15.57:0/1981932570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6584151740 msgr2=0x7f65740047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:44.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f6588901640 1 --2- 172.21.15.57:0/1981932570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6584151740 0x7f65740047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:44.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f6588901640 1 -- 172.21.15.57:0/1981932570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6574004d10 msgr2=0x7f6574005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:44.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f6588901640 1 --2- 172.21.15.57:0/1981932570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6574004d10 0x7f6574005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:44.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f6588901640 1 -- 172.21.15.57:0/1981932570 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6578009420 con 0x7f65740039b0 2023-10-30T14:23:44.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.789+0000 7f6589102640 1 --2- 172.21.15.57:0/1981932570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6574004d10 0x7f6574005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:23:44.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.793+0000 7f6588901640 1 --2- 172.21.15.57:0/1981932570 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65740039b0 0x7f6574003dd0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6578009cb0 tx=0x7f657801cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caed090b7428e077 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:44.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.793+0000 7f65837fe640 1 -- 172.21.15.57:0/1981932570 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65780a5770 con 0x7f65740039b0 2023-10-30T14:23:44.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.793+0000 7f65837fe640 1 -- 172.21.15.57:0/1981932570 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65780a58d0 con 0x7f65740039b0 2023-10-30T14:23:44.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.793+0000 7f65837fe640 1 -- 172.21.15.57:0/1981932570 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65780a5c80 con 0x7f65740039b0 2023-10-30T14:23:44.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.793+0000 7f658ab65640 1 -- 172.21.15.57:0/1981932570 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65740039b0 msgr2=0x7f6574003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:44.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.793+0000 7f658ab65640 1 --2- 172.21.15.57:0/1981932570 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65740039b0 0x7f6574003dd0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6578009cb0 tx=0x7f657801cbc0 comp rx=0 tx=0).stop 2023-10-30T14:23:44.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.793+0000 7f6589102640 1 -- 172.21.15.57:0/1981932570 reap_dead start 2023-10-30T14:23:44.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.793+0000 7f658ab65640 1 -- 172.21.15.57:0/1981932570 shutdown_connections 2023-10-30T14:23:44.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.793+0000 7f658ab65640 1 -- 172.21.15.57:0/1981932570 >> 172.21.15.57:0/1981932570 conn(0x7f65840c7f60 msgr2=0x7f65840c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:44.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.793+0000 7f658ab65640 1 -- 172.21.15.57:0/1981932570 shutdown_connections 2023-10-30T14:23:44.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.793+0000 7f658ab65640 1 -- 172.21.15.57:0/1981932570 wait complete. 2023-10-30T14:23:44.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.793+0000 7f658ab65640 1 Processor -- start 2023-10-30T14:23:44.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f658ab65640 1 -- start start 2023-10-30T14:23:44.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f658ab65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65740039b0 0x7f6584150280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:44.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f658ab65640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f658415a4c0 0x7f6584155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:44.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f658ab65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6584149db0 0x7f658414a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:44.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f658ab65640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6584150960 con 0x7f658415a4c0 2023-10-30T14:23:44.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f658ab65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6584157270 con 0x7f65740039b0 2023-10-30T14:23:44.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f658ab65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658414e460 con 0x7f6584149db0 2023-10-30T14:23:44.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f6583fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f658415a4c0 0x7f6584155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:44.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f6583fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f658415a4c0 0x7f6584155630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48138/0 (socket says 172.21.15.57:48138) 2023-10-30T14:23:44.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f6583fff640 1 -- 172.21.15.57:0/3366453400 learned_addr learned my addr 172.21.15.57:0/3366453400 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:23:44.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f6588901640 1 --2- 172.21.15.57:0/3366453400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65740039b0 0x7f6584150280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:44.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f6589102640 1 --2- 172.21.15.57:0/3366453400 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6584149db0 0x7f658414a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:44.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f6583fff640 1 -- 172.21.15.57:0/3366453400 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6584149db0 msgr2=0x7f658414a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:44.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f6583fff640 1 --2- 172.21.15.57:0/3366453400 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6584149db0 0x7f658414a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:44.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f6583fff640 1 -- 172.21.15.57:0/3366453400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65740039b0 msgr2=0x7f6584150280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:44.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f6583fff640 1 --2- 172.21.15.57:0/3366453400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65740039b0 0x7f6584150280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:44.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f6583fff640 1 -- 172.21.15.57:0/3366453400 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658414a770 con 0x7f658415a4c0 2023-10-30T14:23:44.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f6588901640 1 --2- 172.21.15.57:0/3366453400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65740039b0 0x7f6584150280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:23:44.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f6583fff640 1 --2- 172.21.15.57:0/3366453400 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f658415a4c0 0x7f6584155630 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6570007290 tx=0x7f6570007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:44.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f65817fa640 1 -- 172.21.15.57:0/3366453400 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657000f8c0 con 0x7f658415a4c0 2023-10-30T14:23:44.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f65817fa640 1 -- 172.21.15.57:0/3366453400 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657000fa20 con 0x7f658415a4c0 2023-10-30T14:23:44.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f65817fa640 1 -- 172.21.15.57:0/3366453400 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657000fdd0 con 0x7f658415a4c0 2023-10-30T14:23:44.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f658ab65640 1 -- 172.21.15.57:0/3366453400 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65841576f0 con 0x7f658415a4c0 2023-10-30T14:23:44.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f658ab65640 1 -- 172.21.15.57:0/3366453400 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f658414e6b0 con 0x7f658415a4c0 2023-10-30T14:23:44.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.797+0000 7f658ab65640 1 -- 172.21.15.57:0/3366453400 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6584157ba0 con 0x7f658415a4c0 2023-10-30T14:23:44.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.801+0000 7f65817fa640 1 -- 172.21.15.57:0/3366453400 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f657001d460 con 0x7f658415a4c0 2023-10-30T14:23:44.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.801+0000 7f65817fa640 1 --2- 172.21.15.57:0/3366453400 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65680664d0 0x7f6568068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:44.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.801+0000 7f65817fa640 1 -- 172.21.15.57:0/3366453400 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f65700843c0 con 0x7f658415a4c0 2023-10-30T14:23:44.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.801+0000 7f6588901640 1 --2- 172.21.15.57:0/3366453400 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65680664d0 0x7f6568068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:44.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.801+0000 7f6588901640 1 --2- 172.21.15.57:0/3366453400 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65680664d0 0x7f6568068990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f65780af000 tx=0x7f65780046d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:44.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.805+0000 7f65817fa640 1 -- 172.21.15.57:0/3366453400 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6570051560 con 0x7f658415a4c0 2023-10-30T14:23:44.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.993+0000 7f658ab65640 1 -- 172.21.15.57:0/3366453400 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f658404e170 con 0x7f658415a4c0 2023-10-30T14:23:44.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.993+0000 7f65817fa640 1 -- 172.21.15.57:0/3366453400 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6508 (secure 0 0 0) 0x7f6570054b80 con 0x7f658415a4c0 2023-10-30T14:23:44.996 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:23:44.996 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:22:37.974156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5133},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:44.997 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 66 2023-10-30T14:23:44.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.997+0000 7f658ab65640 1 -- 172.21.15.57:0/3366453400 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65680664d0 msgr2=0x7f6568068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:44.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.997+0000 7f658ab65640 1 --2- 172.21.15.57:0/3366453400 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65680664d0 0x7f6568068990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f65780af000 tx=0x7f65780046d0 comp rx=0 tx=0).stop 2023-10-30T14:23:44.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.997+0000 7f658ab65640 1 -- 172.21.15.57:0/3366453400 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f658415a4c0 msgr2=0x7f6584155630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:44.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.997+0000 7f658ab65640 1 --2- 172.21.15.57:0/3366453400 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f658415a4c0 0x7f6584155630 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6570007290 tx=0x7f6570007750 comp rx=0 tx=0).stop 2023-10-30T14:23:44.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.997+0000 7f658ab65640 1 -- 172.21.15.57:0/3366453400 shutdown_connections 2023-10-30T14:23:44.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.997+0000 7f658ab65640 1 --2- 172.21.15.57:0/3366453400 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65680664d0 0x7f6568068990 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:44.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.997+0000 7f658ab65640 1 --2- 172.21.15.57:0/3366453400 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6584149db0 0x7f658414a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:44.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.997+0000 7f658ab65640 1 --2- 172.21.15.57:0/3366453400 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f658415a4c0 0x7f6584155630 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:45.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.997+0000 7f658ab65640 1 --2- 172.21.15.57:0/3366453400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65740039b0 0x7f6584150280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:45.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.997+0000 7f658ab65640 1 -- 172.21.15.57:0/3366453400 >> 172.21.15.57:0/3366453400 conn(0x7f65840c7f60 msgr2=0x7f658414b540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:45.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.997+0000 7f658ab65640 1 -- 172.21.15.57:0/3366453400 shutdown_connections 2023-10-30T14:23:45.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:44.997+0000 7f658ab65640 1 -- 172.21.15.57:0/3366453400 wait complete. 2023-10-30T14:23:45.017 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:23:47.019 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:47.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.169+0000 7fc78ef65640 1 Processor -- start 2023-10-30T14:23:47.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.169+0000 7fc78ef65640 1 -- start start 2023-10-30T14:23:47.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.173+0000 7fc78ef65640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc788153c10 0x7fc788065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:47.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.173+0000 7fc78ef65640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc788070c30 0x7fc788071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:47.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.173+0000 7fc78ef65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc788159970 0x7fc788159e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:47.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.173+0000 7fc78ef65640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc788072390 con 0x7fc788153c10 2023-10-30T14:23:47.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.173+0000 7fc78ef65640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc78806e390 con 0x7fc788070c30 2023-10-30T14:23:47.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.173+0000 7fc78ef65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7880660e0 con 0x7fc788159970 2023-10-30T14:23:47.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.173+0000 7fc78cd01640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc788153c10 0x7fc788065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:47.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.173+0000 7fc78cd01640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc788153c10 0x7fc788065ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48150/0 (socket says 172.21.15.57:48150) 2023-10-30T14:23:47.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.173+0000 7fc78cd01640 1 -- 172.21.15.57:0/1428728787 learned_addr learned my addr 172.21.15.57:0/1428728787 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:23:47.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.173+0000 7fc787fff640 1 --2- 172.21.15.57:0/1428728787 >> v2:172.21.15.88:3300/0 conn(0x7fc788070c30 0x7fc788071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:47.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.173+0000 7fc78d502640 1 --2- 172.21.15.57:0/1428728787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc788159970 0x7fc788159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:47.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.173+0000 7fc78cd01640 1 -- 172.21.15.57:0/1428728787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc788159970 msgr2=0x7fc788159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:47.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.173+0000 7fc78cd01640 1 --2- 172.21.15.57:0/1428728787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc788159970 0x7fc788159e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.173+0000 7fc78cd01640 1 -- 172.21.15.57:0/1428728787 >> v2:172.21.15.88:3300/0 conn(0x7fc788070c30 msgr2=0x7fc788071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:47.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.173+0000 7fc78cd01640 1 --2- 172.21.15.57:0/1428728787 >> v2:172.21.15.88:3300/0 conn(0x7fc788070c30 0x7fc788071070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.173+0000 7fc78cd01640 1 -- 172.21.15.57:0/1428728787 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7880715b0 con 0x7fc788153c10 2023-10-30T14:23:47.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.177+0000 7fc78cd01640 1 --2- 172.21.15.57:0/1428728787 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc788153c10 0x7fc788065ba0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc7880717a0 tx=0x7fc77c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d0466f0e41eeb8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:47.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.177+0000 7fc7877fe640 1 -- 172.21.15.57:0/1428728787 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77c0ab600 con 0x7fc788153c10 2023-10-30T14:23:47.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.177+0000 7fc7877fe640 1 -- 172.21.15.57:0/1428728787 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc77c0a7050 con 0x7fc788153c10 2023-10-30T14:23:47.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.177+0000 7fc7877fe640 1 -- 172.21.15.57:0/1428728787 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77c0ab980 con 0x7fc788153c10 2023-10-30T14:23:47.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.177+0000 7fc78ef65640 1 -- 172.21.15.57:0/1428728787 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc788153c10 msgr2=0x7fc788065ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:47.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.177+0000 7fc78ef65640 1 --2- 172.21.15.57:0/1428728787 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc788153c10 0x7fc788065ba0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc7880717a0 tx=0x7fc77c01c470 comp rx=0 tx=0).stop 2023-10-30T14:23:47.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.177+0000 7fc78ef65640 1 -- 172.21.15.57:0/1428728787 shutdown_connections 2023-10-30T14:23:47.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.177+0000 7fc78ef65640 1 --2- 172.21.15.57:0/1428728787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc788159970 0x7fc788159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.177+0000 7fc78ef65640 1 --2- 172.21.15.57:0/1428728787 >> v2:172.21.15.88:3300/0 conn(0x7fc788070c30 0x7fc788071070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.177+0000 7fc78ef65640 1 --2- 172.21.15.57:0/1428728787 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc788153c10 0x7fc788065ba0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.177+0000 7fc78ef65640 1 -- 172.21.15.57:0/1428728787 >> 172.21.15.57:0/1428728787 conn(0x7fc7880c7f60 msgr2=0x7fc7880c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:47.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.177+0000 7fc78ef65640 1 -- 172.21.15.57:0/1428728787 shutdown_connections 2023-10-30T14:23:47.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.177+0000 7fc78ef65640 1 -- 172.21.15.57:0/1428728787 wait complete. 2023-10-30T14:23:47.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.177+0000 7fc78ef65640 1 Processor -- start 2023-10-30T14:23:47.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.177+0000 7fc78ef65640 1 -- start start 2023-10-30T14:23:47.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc78ef65640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc788070c30 0x7fc7881532d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:47.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc78ef65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc788153c10 0x7fc788149b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:47.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc78ef65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc788159970 0x7fc78814a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:47.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc78ef65640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78814f4b0 con 0x7fc788070c30 2023-10-30T14:23:47.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc78ef65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc788153810 con 0x7fc788159970 2023-10-30T14:23:47.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc78ef65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7881f3000 con 0x7fc788153c10 2023-10-30T14:23:47.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc78cd01640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc788070c30 0x7fc7881532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:47.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc78cd01640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc788070c30 0x7fc7881532d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48166/0 (socket says 172.21.15.57:48166) 2023-10-30T14:23:47.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc78cd01640 1 -- 172.21.15.57:0/3362550997 learned_addr learned my addr 172.21.15.57:0/3362550997 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:23:47.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc78d502640 1 --2- 172.21.15.57:0/3362550997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc788159970 0x7fc78814a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:47.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc787fff640 1 --2- 172.21.15.57:0/3362550997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc788153c10 0x7fc788149b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:47.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc78cd01640 1 -- 172.21.15.57:0/3362550997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc788153c10 msgr2=0x7fc788149b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:47.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc78cd01640 1 --2- 172.21.15.57:0/3362550997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc788153c10 0x7fc788149b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc78cd01640 1 -- 172.21.15.57:0/3362550997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc788159970 msgr2=0x7fc78814a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:47.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc78cd01640 1 --2- 172.21.15.57:0/3362550997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc788159970 0x7fc78814a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc78cd01640 1 -- 172.21.15.57:0/3362550997 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77c0094d0 con 0x7fc788070c30 2023-10-30T14:23:47.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc787fff640 1 --2- 172.21.15.57:0/3362550997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc788153c10 0x7fc788149b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:23:47.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc78cd01640 1 --2- 172.21.15.57:0/3362550997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc788070c30 0x7fc7881532d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc77c01cd10 tx=0x7fc77c01acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:47.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc7857fa640 1 -- 172.21.15.57:0/3362550997 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77c0b6620 con 0x7fc788070c30 2023-10-30T14:23:47.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc7857fa640 1 -- 172.21.15.57:0/3362550997 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc77c0abe90 con 0x7fc788070c30 2023-10-30T14:23:47.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc7857fa640 1 -- 172.21.15.57:0/3362550997 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77c0b69c0 con 0x7fc788070c30 2023-10-30T14:23:47.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc78ef65640 1 -- 172.21.15.57:0/3362550997 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7880cb320 con 0x7fc788070c30 2023-10-30T14:23:47.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc78ef65640 1 -- 172.21.15.57:0/3362550997 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7880cb040 con 0x7fc788070c30 2023-10-30T14:23:47.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc78ef65640 1 -- 172.21.15.57:0/3362550997 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc78804e170 con 0x7fc788070c30 2023-10-30T14:23:47.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.181+0000 7fc7857fa640 1 -- 172.21.15.57:0/3362550997 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc77c0b2040 con 0x7fc788070c30 2023-10-30T14:23:47.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.185+0000 7fc7857fa640 1 --2- 172.21.15.57:0/3362550997 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc76c0664a0 0x7fc76c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:47.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.185+0000 7fc7857fa640 1 -- 172.21.15.57:0/3362550997 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fc77c11e540 con 0x7fc788070c30 2023-10-30T14:23:47.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.185+0000 7fc787fff640 1 --2- 172.21.15.57:0/3362550997 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc76c0664a0 0x7fc76c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:47.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.185+0000 7fc787fff640 1 --2- 172.21.15.57:0/3362550997 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc76c0664a0 0x7fc76c068960 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc774004660 tx=0x7fc774008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:47.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.189+0000 7fc7857fa640 1 -- 172.21.15.57:0/3362550997 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc77c0eb790 con 0x7fc788070c30 2023-10-30T14:23:47.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.377+0000 7fc78ef65640 1 -- 172.21.15.57:0/3362550997 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7880706e0 con 0x7fc788070c30 2023-10-30T14:23:47.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.377+0000 7fc7857fa640 1 -- 172.21.15.57:0/3362550997 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6508 (secure 0 0 0) 0x7fc77c0eedb0 con 0x7fc788070c30 2023-10-30T14:23:47.380 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:23:47.381 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:22:37.974156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5133},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:47.381 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 66 2023-10-30T14:23:47.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.381+0000 7fc78ef65640 1 -- 172.21.15.57:0/3362550997 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc76c0664a0 msgr2=0x7fc76c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:47.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.381+0000 7fc78ef65640 1 --2- 172.21.15.57:0/3362550997 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc76c0664a0 0x7fc76c068960 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc774004660 tx=0x7fc774008040 comp rx=0 tx=0).stop 2023-10-30T14:23:47.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.381+0000 7fc78ef65640 1 -- 172.21.15.57:0/3362550997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc788070c30 msgr2=0x7fc7881532d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:47.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.381+0000 7fc78ef65640 1 --2- 172.21.15.57:0/3362550997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc788070c30 0x7fc7881532d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc77c01cd10 tx=0x7fc77c01acb0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.381+0000 7fc78ef65640 1 -- 172.21.15.57:0/3362550997 shutdown_connections 2023-10-30T14:23:47.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.381+0000 7fc78ef65640 1 --2- 172.21.15.57:0/3362550997 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc76c0664a0 0x7fc76c068960 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.381+0000 7fc78ef65640 1 --2- 172.21.15.57:0/3362550997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc788159970 0x7fc78814a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.381+0000 7fc78ef65640 1 --2- 172.21.15.57:0/3362550997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc788153c10 0x7fc788149b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.381+0000 7fc78ef65640 1 --2- 172.21.15.57:0/3362550997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc788070c30 0x7fc7881532d0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.381+0000 7fc78ef65640 1 -- 172.21.15.57:0/3362550997 >> 172.21.15.57:0/3362550997 conn(0x7fc7880c7f60 msgr2=0x7fc78814cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:47.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.381+0000 7fc78ef65640 1 -- 172.21.15.57:0/3362550997 shutdown_connections 2023-10-30T14:23:47.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:47.381+0000 7fc78ef65640 1 -- 172.21.15.57:0/3362550997 wait complete. 2023-10-30T14:23:47.402 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:23:49.402 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:49.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.557+0000 7f1b69b3b640 1 Processor -- start 2023-10-30T14:23:49.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.557+0000 7f1b69b3b640 1 -- start start 2023-10-30T14:23:49.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.557+0000 7f1b69b3b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1b640c8eb0 0x7f1b6415cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:49.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.557+0000 7f1b69b3b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1b6415ac70 0x7f1b6415a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:49.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.557+0000 7f1b69b3b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6415dcc0 con 0x7f1b6406aeb0 2023-10-30T14:23:49.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.557+0000 7f1b69b3b640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b64159800 con 0x7f1b640c8eb0 2023-10-30T14:23:49.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.557+0000 7f1b69b3b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b64159d80 con 0x7f1b6415ac70 2023-10-30T14:23:49.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.557+0000 7f1b68b39640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1b640c8eb0 0x7f1b6415cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:49.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.557+0000 7f1b6933a640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1b6406aeb0 0x7f1b64069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50346/0 (socket says 172.21.15.57:50346) 2023-10-30T14:23:49.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.557+0000 7f1b6933a640 1 -- 172.21.15.57:0/2550761420 learned_addr learned my addr 172.21.15.57:0/2550761420 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:23:49.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.557+0000 7f1b63fff640 1 --2- 172.21.15.57:0/2550761420 >> v2:172.21.15.88:3300/0 conn(0x7f1b6415ac70 0x7f1b6415a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:49.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.557+0000 7f1b68b39640 1 -- 172.21.15.57:0/2550761420 >> v2:172.21.15.88:3300/0 conn(0x7f1b6415ac70 msgr2=0x7f1b6415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:49.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.557+0000 7f1b68b39640 1 --2- 172.21.15.57:0/2550761420 >> v2:172.21.15.88:3300/0 conn(0x7f1b6415ac70 0x7f1b6415a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.557+0000 7f1b68b39640 1 -- 172.21.15.57:0/2550761420 >> v1:172.21.15.57:6789/0 conn(0x7f1b6406aeb0 legacy=0x7f1b64069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:49.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.557+0000 7f1b68b39640 1 -- 172.21.15.57:0/2550761420 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6415a750 con 0x7f1b640c8eb0 2023-10-30T14:23:49.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.557+0000 7f1b637fe640 1 -- 172.21.15.57:0/2550761420 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259728890 0 0) 0x7f1b5c0014a0 con 0x7f1b6406aeb0 2023-10-30T14:23:49.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.561+0000 7f1b68b39640 1 --2- 172.21.15.57:0/2550761420 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b640c8eb0 0x7f1b6415cd70 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1b640b4130 tx=0x7f1b5801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ee2acf86c098aaa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:49.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.561+0000 7f1b637fe640 1 -- 172.21.15.57:0/2550761420 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b580a8950 con 0x7f1b640c8eb0 2023-10-30T14:23:49.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.561+0000 7f1b637fe640 1 -- 172.21.15.57:0/2550761420 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b580a8ab0 con 0x7f1b640c8eb0 2023-10-30T14:23:49.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.561+0000 7f1b637fe640 1 -- 172.21.15.57:0/2550761420 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b580a8e40 con 0x7f1b640c8eb0 2023-10-30T14:23:49.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.561+0000 7f1b69b3b640 1 -- 172.21.15.57:0/2550761420 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b640c8eb0 msgr2=0x7f1b6415cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:49.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.561+0000 7f1b69b3b640 1 --2- 172.21.15.57:0/2550761420 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b640c8eb0 0x7f1b6415cd70 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1b640b4130 tx=0x7f1b5801ec50 comp rx=0 tx=0).stop 2023-10-30T14:23:49.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.561+0000 7f1b69b3b640 1 -- 172.21.15.57:0/2550761420 shutdown_connections 2023-10-30T14:23:49.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.561+0000 7f1b69b3b640 1 --2- 172.21.15.57:0/2550761420 >> v2:172.21.15.88:3300/0 conn(0x7f1b6415ac70 0x7f1b6415a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.561+0000 7f1b69b3b640 1 --2- 172.21.15.57:0/2550761420 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b640c8eb0 0x7f1b6415cd70 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.561+0000 7f1b69b3b640 1 -- 172.21.15.57:0/2550761420 >> 172.21.15.57:0/2550761420 conn(0x7f1b640cef00 msgr2=0x7f1b640cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:49.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.561+0000 7f1b69b3b640 1 -- 172.21.15.57:0/2550761420 shutdown_connections 2023-10-30T14:23:49.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.561+0000 7f1b69b3b640 1 -- 172.21.15.57:0/2550761420 wait complete. 2023-10-30T14:23:49.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.561+0000 7f1b69b3b640 1 Processor -- start 2023-10-30T14:23:49.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.561+0000 7f1b69b3b640 1 -- start start 2023-10-30T14:23:49.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b69b3b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b6406aeb0 0x7f1b640c3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:49.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b69b3b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b640c8eb0 0x7f1b640c57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:49.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b69b3b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b6415ac70 0x7f1b6414f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:49.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b69b3b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b640c5320 con 0x7f1b640c8eb0 2023-10-30T14:23:49.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b69b3b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b640c3940 con 0x7f1b6406aeb0 2023-10-30T14:23:49.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b69b3b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b640c4430 con 0x7f1b6415ac70 2023-10-30T14:23:49.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b63fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b640c8eb0 0x7f1b640c57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:49.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b63fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b640c8eb0 0x7f1b640c57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48190/0 (socket says 172.21.15.57:48190) 2023-10-30T14:23:49.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b63fff640 1 -- 172.21.15.57:0/2722323456 learned_addr learned my addr 172.21.15.57:0/2722323456 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:23:49.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b68b39640 1 --2- 172.21.15.57:0/2722323456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b6406aeb0 0x7f1b640c3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:49.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b6933a640 1 --2- 172.21.15.57:0/2722323456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b6415ac70 0x7f1b6414f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:49.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b63fff640 1 -- 172.21.15.57:0/2722323456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b6415ac70 msgr2=0x7f1b6414f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:49.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b63fff640 1 --2- 172.21.15.57:0/2722323456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b6415ac70 0x7f1b6414f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b63fff640 1 -- 172.21.15.57:0/2722323456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b6406aeb0 msgr2=0x7f1b640c3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:49.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b63fff640 1 --2- 172.21.15.57:0/2722323456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b6406aeb0 0x7f1b640c3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b63fff640 1 -- 172.21.15.57:0/2722323456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b58009420 con 0x7f1b640c8eb0 2023-10-30T14:23:49.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b68b39640 1 --2- 172.21.15.57:0/2722323456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b6406aeb0 0x7f1b640c3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:23:49.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b63fff640 1 --2- 172.21.15.57:0/2722323456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b640c8eb0 0x7f1b640c57b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1b5000e960 tx=0x7f1b5000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:49.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b617fa640 1 -- 172.21.15.57:0/2722323456 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b50009690 con 0x7f1b640c8eb0 2023-10-30T14:23:49.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b617fa640 1 -- 172.21.15.57:0/2722323456 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5000ca00 con 0x7f1b640c8eb0 2023-10-30T14:23:49.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b617fa640 1 -- 172.21.15.57:0/2722323456 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5000cd70 con 0x7f1b640c8eb0 2023-10-30T14:23:49.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b69b3b640 1 -- 172.21.15.57:0/2722323456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b640c4710 con 0x7f1b640c8eb0 2023-10-30T14:23:49.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b69b3b640 1 -- 172.21.15.57:0/2722323456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b640c4cd0 con 0x7f1b640c8eb0 2023-10-30T14:23:49.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.565+0000 7f1b69b3b640 1 -- 172.21.15.57:0/2722323456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b640c54d0 con 0x7f1b640c8eb0 2023-10-30T14:23:49.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.569+0000 7f1b617fa640 1 -- 172.21.15.57:0/2722323456 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1b50013470 con 0x7f1b640c8eb0 2023-10-30T14:23:49.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.569+0000 7f1b617fa640 1 --2- 172.21.15.57:0/2722323456 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1b480664a0 0x7f1b48068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:49.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.569+0000 7f1b617fa640 1 -- 172.21.15.57:0/2722323456 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f1b50084460 con 0x7f1b640c8eb0 2023-10-30T14:23:49.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.569+0000 7f1b68b39640 1 --2- 172.21.15.57:0/2722323456 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1b480664a0 0x7f1b48068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:49.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.569+0000 7f1b68b39640 1 --2- 172.21.15.57:0/2722323456 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1b480664a0 0x7f1b48068960 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1b580b2000 tx=0x7f1b58008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:49.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.573+0000 7f1b617fa640 1 -- 172.21.15.57:0/2722323456 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1b50051570 con 0x7f1b640c8eb0 2023-10-30T14:23:49.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.765+0000 7f1b69b3b640 1 -- 172.21.15.57:0/2722323456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b6404e170 con 0x7f1b640c8eb0 2023-10-30T14:23:49.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.769+0000 7f1b617fa640 1 -- 172.21.15.57:0/2722323456 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6508 (secure 0 0 0) 0x7f1b50054b90 con 0x7f1b640c8eb0 2023-10-30T14:23:49.770 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:23:49.770 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:22:37.974156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5133},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:49.771 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 66 2023-10-30T14:23:49.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.769+0000 7f1b69b3b640 1 -- 172.21.15.57:0/2722323456 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1b480664a0 msgr2=0x7f1b48068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:49.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.769+0000 7f1b69b3b640 1 --2- 172.21.15.57:0/2722323456 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1b480664a0 0x7f1b48068960 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1b580b2000 tx=0x7f1b58008040 comp rx=0 tx=0).stop 2023-10-30T14:23:49.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.769+0000 7f1b69b3b640 1 -- 172.21.15.57:0/2722323456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b640c8eb0 msgr2=0x7f1b640c57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:49.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.769+0000 7f1b69b3b640 1 --2- 172.21.15.57:0/2722323456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b640c8eb0 0x7f1b640c57b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1b5000e960 tx=0x7f1b5000ee20 comp rx=0 tx=0).stop 2023-10-30T14:23:49.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.769+0000 7f1b69b3b640 1 -- 172.21.15.57:0/2722323456 shutdown_connections 2023-10-30T14:23:49.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.769+0000 7f1b69b3b640 1 --2- 172.21.15.57:0/2722323456 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1b480664a0 0x7f1b48068960 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.769+0000 7f1b69b3b640 1 --2- 172.21.15.57:0/2722323456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b6415ac70 0x7f1b6414f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.769+0000 7f1b69b3b640 1 --2- 172.21.15.57:0/2722323456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b640c8eb0 0x7f1b640c57b0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.769+0000 7f1b69b3b640 1 --2- 172.21.15.57:0/2722323456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b6406aeb0 0x7f1b640c3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.769+0000 7f1b69b3b640 1 -- 172.21.15.57:0/2722323456 >> 172.21.15.57:0/2722323456 conn(0x7f1b640cef00 msgr2=0x7f1b64158c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:49.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.769+0000 7f1b69b3b640 1 -- 172.21.15.57:0/2722323456 shutdown_connections 2023-10-30T14:23:49.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:49.769+0000 7f1b69b3b640 1 -- 172.21.15.57:0/2722323456 wait complete. 2023-10-30T14:23:49.791 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:23:51.792 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:51.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.945+0000 7fe944a46640 1 Processor -- start 2023-10-30T14:23:51.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.945+0000 7fe944a46640 1 -- start start 2023-10-30T14:23:51.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.945+0000 7fe944a46640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe940151840 0x7fe9400c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:51.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.945+0000 7fe944a46640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe940151c20 0x7fe940072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:51.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.945+0000 7fe944a46640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9400c5030 con 0x7fe9400cb8b0 2023-10-30T14:23:51.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.945+0000 7fe944a46640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe940157830 con 0x7fe940151840 2023-10-30T14:23:51.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.945+0000 7fe944a46640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe940157b10 con 0x7fe940151c20 2023-10-30T14:23:51.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.949+0000 7fe93e59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe940151840 0x7fe9400c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:51.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.949+0000 7fe937fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe940151c20 0x7fe940072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:51.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.949+0000 7fe93e59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe940151840 0x7fe9400c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:57496/0 (socket says 172.21.15.57:57496) 2023-10-30T14:23:51.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.949+0000 7fe93e59c640 1 -- 172.21.15.57:0/1536118527 learned_addr learned my addr 172.21.15.57:0/1536118527 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:23:51.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.949+0000 7fe93e59c640 1 -- 172.21.15.57:0/1536118527 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe940151c20 msgr2=0x7fe940072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:51.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.949+0000 7fe93e59c640 1 --2- 172.21.15.57:0/1536118527 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe940151c20 0x7fe940072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:51.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.949+0000 7fe93e59c640 1 -- 172.21.15.57:0/1536118527 >> v1:172.21.15.88:6789/0 conn(0x7fe9400cb8b0 legacy=0x7fe94006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:51.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.949+0000 7fe93e59c640 1 -- 172.21.15.57:0/1536118527 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe940157df0 con 0x7fe940151840 2023-10-30T14:23:51.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.949+0000 7fe937fff640 1 --2- 172.21.15.57:0/1536118527 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe940151c20 0x7fe940072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:23:51.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.949+0000 7fe93e59c640 1 --2- 172.21.15.57:0/1536118527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe940151840 0x7fe9400c49a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe930009870 tx=0x7fe93001ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c53a28f545c8909d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:51.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.953+0000 7fe93dd9b640 1 -- 172.21.15.57:0/1536118527 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9300a8a30 con 0x7fe940151840 2023-10-30T14:23:51.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.953+0000 7fe93dd9b640 1 -- 172.21.15.57:0/1536118527 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9300a8b90 con 0x7fe940151840 2023-10-30T14:23:51.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.953+0000 7fe93dd9b640 1 -- 172.21.15.57:0/1536118527 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9300b0440 con 0x7fe940151840 2023-10-30T14:23:51.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.953+0000 7fe944a46640 1 -- 172.21.15.57:0/1536118527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe940151840 msgr2=0x7fe9400c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:51.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.953+0000 7fe944a46640 1 --2- 172.21.15.57:0/1536118527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe940151840 0x7fe9400c49a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe930009870 tx=0x7fe93001ed30 comp rx=0 tx=0).stop 2023-10-30T14:23:51.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.953+0000 7fe944a46640 1 -- 172.21.15.57:0/1536118527 shutdown_connections 2023-10-30T14:23:51.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.953+0000 7fe944a46640 1 --2- 172.21.15.57:0/1536118527 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe940151c20 0x7fe940072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:51.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.953+0000 7fe944a46640 1 --2- 172.21.15.57:0/1536118527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe940151840 0x7fe9400c49a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:51.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.953+0000 7fe944a46640 1 -- 172.21.15.57:0/1536118527 >> 172.21.15.57:0/1536118527 conn(0x7fe9400cac70 msgr2=0x7fe9400cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:51.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.953+0000 7fe944a46640 1 -- 172.21.15.57:0/1536118527 shutdown_connections 2023-10-30T14:23:51.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.953+0000 7fe944a46640 1 -- 172.21.15.57:0/1536118527 wait complete. 2023-10-30T14:23:51.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.953+0000 7fe944a46640 1 Processor -- start 2023-10-30T14:23:51.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.953+0000 7fe944a46640 1 -- start start 2023-10-30T14:23:51.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe944a46640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9400cb8b0 0x7fe940145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:51.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe944a46640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe940151840 0x7fe9400c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:51.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe944a46640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe940151c20 0x7fe9400ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:51.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe944a46640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe940146530 con 0x7fe9400cb8b0 2023-10-30T14:23:51.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe944a46640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe940156dc0 con 0x7fe940151c20 2023-10-30T14:23:51.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe944a46640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe940146780 con 0x7fe940151840 2023-10-30T14:23:51.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe93e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9400cb8b0 0x7fe940145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:51.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe93e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9400cb8b0 0x7fe940145a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48192/0 (socket says 172.21.15.57:48192) 2023-10-30T14:23:51.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe93e59c640 1 -- 172.21.15.57:0/26594880 learned_addr learned my addr 172.21.15.57:0/26594880 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:23:51.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe937fff640 1 --2- 172.21.15.57:0/26594880 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe940151840 0x7fe9400c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:51.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe93ed9d640 1 --2- 172.21.15.57:0/26594880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe940151c20 0x7fe9400ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:51.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe93e59c640 1 -- 172.21.15.57:0/26594880 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe940151840 msgr2=0x7fe9400c9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:51.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe93e59c640 1 --2- 172.21.15.57:0/26594880 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe940151840 0x7fe9400c9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:51.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe93e59c640 1 -- 172.21.15.57:0/26594880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe940151c20 msgr2=0x7fe9400ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:51.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe93e59c640 1 --2- 172.21.15.57:0/26594880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe940151c20 0x7fe9400ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:51.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe93e59c640 1 -- 172.21.15.57:0/26594880 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe928009420 con 0x7fe9400cb8b0 2023-10-30T14:23:51.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe93e59c640 1 --2- 172.21.15.57:0/26594880 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9400cb8b0 0x7fe940145a40 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe930000c10 tx=0x7fe93001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:51.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe936ffd640 1 -- 172.21.15.57:0/26594880 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9300bf070 con 0x7fe9400cb8b0 2023-10-30T14:23:51.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe936ffd640 1 -- 172.21.15.57:0/26594880 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9300b1be0 con 0x7fe9400cb8b0 2023-10-30T14:23:51.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe944a46640 1 -- 172.21.15.57:0/26594880 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9300094d0 con 0x7fe9400cb8b0 2023-10-30T14:23:51.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe936ffd640 1 -- 172.21.15.57:0/26594880 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9300b9410 con 0x7fe9400cb8b0 2023-10-30T14:23:51.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe944a46640 1 -- 172.21.15.57:0/26594880 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe940157310 con 0x7fe9400cb8b0 2023-10-30T14:23:51.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe944a46640 1 -- 172.21.15.57:0/26594880 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9401574d0 con 0x7fe9400cb8b0 2023-10-30T14:23:51.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.957+0000 7fe936ffd640 1 -- 172.21.15.57:0/26594880 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe9300b7040 con 0x7fe9400cb8b0 2023-10-30T14:23:51.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.961+0000 7fe936ffd640 1 --2- 172.21.15.57:0/26594880 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe9200664d0 0x7fe920068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:51.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.961+0000 7fe936ffd640 1 -- 172.21.15.57:0/26594880 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fe9301213d0 con 0x7fe9400cb8b0 2023-10-30T14:23:51.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.961+0000 7fe937fff640 1 --2- 172.21.15.57:0/26594880 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe9200664d0 0x7fe920068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:51.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.961+0000 7fe937fff640 1 --2- 172.21.15.57:0/26594880 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe9200664d0 0x7fe920068990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe9401e2d00 tx=0x7fe928009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:51.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:51.965+0000 7fe936ffd640 1 -- 172.21.15.57:0/26594880 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe9300ee620 con 0x7fe9400cb8b0 2023-10-30T14:23:52.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:52.157+0000 7fe944a46640 1 -- 172.21.15.57:0/26594880 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe940066510 con 0x7fe9400cb8b0 2023-10-30T14:23:52.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:52.157+0000 7fe936ffd640 1 -- 172.21.15.57:0/26594880 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6508 (secure 0 0 0) 0x7fe9300b5090 con 0x7fe9400cb8b0 2023-10-30T14:23:52.160 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:23:52.160 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:22:37.974156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5133},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:52.161 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 66 2023-10-30T14:23:52.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:52.161+0000 7fe944a46640 1 -- 172.21.15.57:0/26594880 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe9200664d0 msgr2=0x7fe920068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:52.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:52.161+0000 7fe944a46640 1 --2- 172.21.15.57:0/26594880 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe9200664d0 0x7fe920068990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe9401e2d00 tx=0x7fe928009190 comp rx=0 tx=0).stop 2023-10-30T14:23:52.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:52.161+0000 7fe944a46640 1 -- 172.21.15.57:0/26594880 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9400cb8b0 msgr2=0x7fe940145a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:52.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:52.161+0000 7fe944a46640 1 --2- 172.21.15.57:0/26594880 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9400cb8b0 0x7fe940145a40 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe930000c10 tx=0x7fe93001bad0 comp rx=0 tx=0).stop 2023-10-30T14:23:52.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:52.161+0000 7fe944a46640 1 -- 172.21.15.57:0/26594880 shutdown_connections 2023-10-30T14:23:52.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:52.161+0000 7fe944a46640 1 --2- 172.21.15.57:0/26594880 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe9200664d0 0x7fe920068990 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:52.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:52.161+0000 7fe944a46640 1 --2- 172.21.15.57:0/26594880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe940151c20 0x7fe9400ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:52.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:52.161+0000 7fe944a46640 1 --2- 172.21.15.57:0/26594880 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe940151840 0x7fe9400c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:52.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:52.161+0000 7fe944a46640 1 --2- 172.21.15.57:0/26594880 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9400cb8b0 0x7fe940145a40 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:52.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:52.161+0000 7fe944a46640 1 -- 172.21.15.57:0/26594880 >> 172.21.15.57:0/26594880 conn(0x7fe9400cac70 msgr2=0x7fe9401de090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:52.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:52.161+0000 7fe944a46640 1 -- 172.21.15.57:0/26594880 shutdown_connections 2023-10-30T14:23:52.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:52.161+0000 7fe944a46640 1 -- 172.21.15.57:0/26594880 wait complete. 2023-10-30T14:23:52.181 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:23:54.182 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:54.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.333+0000 7fa30ea32640 1 Processor -- start 2023-10-30T14:23:54.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.333+0000 7fa30ea32640 1 -- start start 2023-10-30T14:23:54.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.337+0000 7fa30ea32640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa308150830 0x7fa30806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:54.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.337+0000 7fa30ea32640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3080c2310 con 0x7fa308070520 2023-10-30T14:23:54.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.337+0000 7fa30ea32640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa308156710 con 0x7fa3080cb7d0 2023-10-30T14:23:54.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.337+0000 7fa30ea32640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3080c4360 con 0x7fa308150830 2023-10-30T14:23:54.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.337+0000 7fa307fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa308150830 0x7fa30806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:54.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.337+0000 7fa30cfcf640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa308070520 0x7fa308160f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:52510/0 (socket says 172.21.15.57:52510) 2023-10-30T14:23:54.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.337+0000 7fa30cfcf640 1 -- 172.21.15.57:0/1152370187 learned_addr learned my addr 172.21.15.57:0/1152370187 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:23:54.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.337+0000 7fa306ffd640 1 -- 172.21.15.57:0/1152370187 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713653667 0 0) 0x7fa3080c2310 con 0x7fa308070520 2023-10-30T14:23:54.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.337+0000 7fa306ffd640 1 -- 172.21.15.57:0/1152370187 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f0003530 con 0x7fa308070520 2023-10-30T14:23:54.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.337+0000 7fa307fff640 1 -- 172.21.15.57:0/1152370187 >> v1:172.21.15.88:6789/0 conn(0x7fa3080cb7d0 legacy=0x7fa3080c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:54.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.337+0000 7fa307fff640 1 -- 172.21.15.57:0/1152370187 >> v1:172.21.15.57:6789/0 conn(0x7fa308070520 legacy=0x7fa308160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:54.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.337+0000 7fa307fff640 1 -- 172.21.15.57:0/1152370187 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3081569b0 con 0x7fa308150830 2023-10-30T14:23:54.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.341+0000 7fa307fff640 1 --2- 172.21.15.57:0/1152370187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa308150830 0x7fa30806b500 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa2f80097c0 tx=0x7fa2f801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c01a89031bd6c1f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:54.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.341+0000 7fa306ffd640 1 -- 172.21.15.57:0/1152370187 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f80a8970 con 0x7fa308150830 2023-10-30T14:23:54.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.341+0000 7fa306ffd640 1 -- 172.21.15.57:0/1152370187 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f80a8ad0 con 0x7fa308150830 2023-10-30T14:23:54.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.341+0000 7fa306ffd640 1 -- 172.21.15.57:0/1152370187 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f80a8e60 con 0x7fa308150830 2023-10-30T14:23:54.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.341+0000 7fa30ea32640 1 -- 172.21.15.57:0/1152370187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa308150830 msgr2=0x7fa30806b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:54.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.341+0000 7fa30ea32640 1 --2- 172.21.15.57:0/1152370187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa308150830 0x7fa30806b500 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa2f80097c0 tx=0x7fa2f801ec70 comp rx=0 tx=0).stop 2023-10-30T14:23:54.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.341+0000 7fa30ea32640 1 -- 172.21.15.57:0/1152370187 shutdown_connections 2023-10-30T14:23:54.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.341+0000 7fa30ea32640 1 --2- 172.21.15.57:0/1152370187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa308150830 0x7fa30806b500 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:54.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.341+0000 7fa30ea32640 1 -- 172.21.15.57:0/1152370187 >> 172.21.15.57:0/1152370187 conn(0x7fa3080cac70 msgr2=0x7fa3080cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:54.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.341+0000 7fa30ea32640 1 -- 172.21.15.57:0/1152370187 shutdown_connections 2023-10-30T14:23:54.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.341+0000 7fa30ea32640 1 -- 172.21.15.57:0/1152370187 wait complete. 2023-10-30T14:23:54.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.341+0000 7fa30ea32640 1 Processor -- start 2023-10-30T14:23:54.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.341+0000 7fa30ea32640 1 -- start start 2023-10-30T14:23:54.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa30ea32640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa308070520 0x7fa308159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:54.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa30ea32640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3080cb7d0 0x7fa30814b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:54.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa30ea32640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa308150830 0x7fa3081584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:54.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa30ea32640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3080c7140 con 0x7fa3080cb7d0 2023-10-30T14:23:54.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa30ea32640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30815a310 con 0x7fa308150830 2023-10-30T14:23:54.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa30ea32640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3080c7fe0 con 0x7fa308070520 2023-10-30T14:23:54.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa3077fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3080cb7d0 0x7fa30814b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:54.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa3077fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3080cb7d0 0x7fa30814b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45996/0 (socket says 172.21.15.57:45996) 2023-10-30T14:23:54.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa3077fe640 1 -- 172.21.15.57:0/3512505419 learned_addr learned my addr 172.21.15.57:0/3512505419 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:23:54.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa30cfcf640 1 --2- 172.21.15.57:0/3512505419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa308150830 0x7fa3081584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:54.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa307fff640 1 --2- 172.21.15.57:0/3512505419 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa308070520 0x7fa308159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:54.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa3077fe640 1 -- 172.21.15.57:0/3512505419 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa308070520 msgr2=0x7fa308159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:54.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa3077fe640 1 --2- 172.21.15.57:0/3512505419 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa308070520 0x7fa308159930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:54.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa3077fe640 1 -- 172.21.15.57:0/3512505419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa308150830 msgr2=0x7fa3081584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:54.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa3077fe640 1 --2- 172.21.15.57:0/3512505419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa308150830 0x7fa3081584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:54.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa3077fe640 1 -- 172.21.15.57:0/3512505419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f8009420 con 0x7fa3080cb7d0 2023-10-30T14:23:54.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa3077fe640 1 --2- 172.21.15.57:0/3512505419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3080cb7d0 0x7fa30814b2c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00c500 tx=0x7fa2fc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:54.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa304ff9640 1 -- 172.21.15.57:0/3512505419 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc005220 con 0x7fa3080cb7d0 2023-10-30T14:23:54.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa304ff9640 1 -- 172.21.15.57:0/3512505419 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2fc005380 con 0x7fa3080cb7d0 2023-10-30T14:23:54.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa30ea32640 1 -- 172.21.15.57:0/3512505419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa30815a720 con 0x7fa3080cb7d0 2023-10-30T14:23:54.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa304ff9640 1 -- 172.21.15.57:0/3512505419 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc007270 con 0x7fa3080cb7d0 2023-10-30T14:23:54.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa30ea32640 1 -- 172.21.15.57:0/3512505419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa308158fc0 con 0x7fa3080cb7d0 2023-10-30T14:23:54.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa30ea32640 1 -- 172.21.15.57:0/3512505419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa30815a960 con 0x7fa3080cb7d0 2023-10-30T14:23:54.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.345+0000 7fa304ff9640 1 -- 172.21.15.57:0/3512505419 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa2fc0073d0 con 0x7fa3080cb7d0 2023-10-30T14:23:54.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.349+0000 7fa304ff9640 1 --2- 172.21.15.57:0/3512505419 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2f40664d0 0x7fa2f4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:54.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.349+0000 7fa304ff9640 1 -- 172.21.15.57:0/3512505419 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fa2fc0835a0 con 0x7fa3080cb7d0 2023-10-30T14:23:54.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.349+0000 7fa307fff640 1 --2- 172.21.15.57:0/3512505419 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2f40664d0 0x7fa2f4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:54.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.349+0000 7fa307fff640 1 --2- 172.21.15.57:0/3512505419 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2f40664d0 0x7fa2f4068990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa2f80b2000 tx=0x7fa2f8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:54.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.353+0000 7fa304ff9640 1 -- 172.21.15.57:0/3512505419 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa30815a960 con 0x7fa3080cb7d0 2023-10-30T14:23:54.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.541+0000 7fa30ea32640 1 -- 172.21.15.57:0/3512505419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa30815a910 con 0x7fa3080cb7d0 2023-10-30T14:23:54.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.541+0000 7fa304ff9640 1 -- 172.21.15.57:0/3512505419 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6508 (secure 0 0 0) 0x7fa2fc050800 con 0x7fa3080cb7d0 2023-10-30T14:23:54.544 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:23:54.544 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":5586,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:22:37.974156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5133},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5133":{"gid":5133,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":15,"addr":"172.21.15.57:6834/2912050965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2912050965},{"type":"v1","addr":"172.21.15.57:6834","nonce":2912050965}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:54.544 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 66 2023-10-30T14:23:54.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.541+0000 7fa30ea32640 1 -- 172.21.15.57:0/3512505419 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2f40664d0 msgr2=0x7fa2f4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:54.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.541+0000 7fa30ea32640 1 --2- 172.21.15.57:0/3512505419 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2f40664d0 0x7fa2f4068990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa2f80b2000 tx=0x7fa2f8008040 comp rx=0 tx=0).stop 2023-10-30T14:23:54.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.541+0000 7fa30ea32640 1 -- 172.21.15.57:0/3512505419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3080cb7d0 msgr2=0x7fa30814b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:54.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.541+0000 7fa30ea32640 1 --2- 172.21.15.57:0/3512505419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3080cb7d0 0x7fa30814b2c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00c500 tx=0x7fa2fc00c9c0 comp rx=0 tx=0).stop 2023-10-30T14:23:54.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.541+0000 7fa30ea32640 1 -- 172.21.15.57:0/3512505419 shutdown_connections 2023-10-30T14:23:54.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.541+0000 7fa30ea32640 1 --2- 172.21.15.57:0/3512505419 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2f40664d0 0x7fa2f4068990 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:54.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.541+0000 7fa30ea32640 1 --2- 172.21.15.57:0/3512505419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa308150830 0x7fa3081584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:54.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.541+0000 7fa30ea32640 1 --2- 172.21.15.57:0/3512505419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3080cb7d0 0x7fa30814b2c0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:54.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.541+0000 7fa30ea32640 1 --2- 172.21.15.57:0/3512505419 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa308070520 0x7fa308159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:54.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.541+0000 7fa30ea32640 1 -- 172.21.15.57:0/3512505419 >> 172.21.15.57:0/3512505419 conn(0x7fa3080cac70 msgr2=0x7fa30814e840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:54.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.541+0000 7fa30ea32640 1 -- 172.21.15.57:0/3512505419 shutdown_connections 2023-10-30T14:23:54.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:54.541+0000 7fa30ea32640 1 -- 172.21.15.57:0/3512505419 wait complete. 2023-10-30T14:23:54.565 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:23:56.565 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:56.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.713+0000 7f4cb811d640 1 Processor -- start 2023-10-30T14:23:56.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.713+0000 7f4cb811d640 1 -- start start 2023-10-30T14:23:56.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.717+0000 7f4cb811d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4cb0153c80 0x7f4cb015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:56.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.717+0000 7f4cb811d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb00c42a0 0x7f4cb00ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:56.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.717+0000 7f4cb811d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb0159f20 con 0x7f4cb00c4a10 2023-10-30T14:23:56.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.717+0000 7f4cb811d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb015dec0 con 0x7f4cb0153c80 2023-10-30T14:23:56.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.717+0000 7f4cb811d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb01599a0 con 0x7f4cb00c42a0 2023-10-30T14:23:56.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.717+0000 7f4cb5eb9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4cb0153c80 0x7f4cb015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:56.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.717+0000 7f4cb5eb9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4cb0153c80 0x7f4cb015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45998/0 (socket says 172.21.15.57:45998) 2023-10-30T14:23:56.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.717+0000 7f4cb5eb9640 1 -- 172.21.15.57:0/2483377886 learned_addr learned my addr 172.21.15.57:0/2483377886 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:23:56.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.717+0000 7f4cb66ba640 1 --2- 172.21.15.57:0/2483377886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb00c42a0 0x7f4cb00ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:56.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.717+0000 7f4cb5eb9640 1 -- 172.21.15.57:0/2483377886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb00c42a0 msgr2=0x7f4cb00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:56.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.717+0000 7f4cb5eb9640 1 --2- 172.21.15.57:0/2483377886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb00c42a0 0x7f4cb00ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:56.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.717+0000 7f4cb5eb9640 1 -- 172.21.15.57:0/2483377886 >> v1:172.21.15.88:6789/0 conn(0x7f4cb00c4a10 legacy=0x7f4cb015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:56.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.717+0000 7f4cb5eb9640 1 -- 172.21.15.57:0/2483377886 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb015d620 con 0x7f4cb0153c80 2023-10-30T14:23:56.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.717+0000 7f4cb66ba640 1 --2- 172.21.15.57:0/2483377886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb00c42a0 0x7f4cb00ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:23:56.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.717+0000 7f4cb5eb9640 1 --2- 172.21.15.57:0/2483377886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cb0153c80 0x7f4cb015cf90 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009870 tx=0x7f4cac01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa9517d29a4c5683 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:56.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.721+0000 7f4cb4eb7640 1 -- 172.21.15.57:0/2483377886 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac0a8a30 con 0x7f4cb0153c80 2023-10-30T14:23:56.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.721+0000 7f4cb4eb7640 1 -- 172.21.15.57:0/2483377886 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cac0a8b90 con 0x7f4cb0153c80 2023-10-30T14:23:56.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.721+0000 7f4cb4eb7640 1 -- 172.21.15.57:0/2483377886 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac0b0440 con 0x7f4cb0153c80 2023-10-30T14:23:56.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.721+0000 7f4cb811d640 1 -- 172.21.15.57:0/2483377886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cb0153c80 msgr2=0x7f4cb015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:56.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.721+0000 7f4cb811d640 1 --2- 172.21.15.57:0/2483377886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cb0153c80 0x7f4cb015cf90 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009870 tx=0x7f4cac01ed30 comp rx=0 tx=0).stop 2023-10-30T14:23:56.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.721+0000 7f4cb811d640 1 -- 172.21.15.57:0/2483377886 shutdown_connections 2023-10-30T14:23:56.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.721+0000 7f4cb811d640 1 --2- 172.21.15.57:0/2483377886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb00c42a0 0x7f4cb00ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:56.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.721+0000 7f4cb811d640 1 --2- 172.21.15.57:0/2483377886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cb0153c80 0x7f4cb015cf90 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:56.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.721+0000 7f4cb811d640 1 -- 172.21.15.57:0/2483377886 >> 172.21.15.57:0/2483377886 conn(0x7f4cb00715c0 msgr2=0x7f4cb00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:56.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.721+0000 7f4cb811d640 1 -- 172.21.15.57:0/2483377886 shutdown_connections 2023-10-30T14:23:56.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.721+0000 7f4cb811d640 1 -- 172.21.15.57:0/2483377886 wait complete. 2023-10-30T14:23:56.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.721+0000 7f4cb811d640 1 Processor -- start 2023-10-30T14:23:56.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.721+0000 7f4cb811d640 1 -- start start 2023-10-30T14:23:56.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.721+0000 7f4cb811d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cb00c42a0 0x7f4cb0151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:56.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.721+0000 7f4cb811d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb00c4a10 0x7f4cb00c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:56.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4cb811d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb0153c80 0x7f4cb00c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:56.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4cb811d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb0154950 con 0x7f4cb00c42a0 2023-10-30T14:23:56.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4cb811d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb0156e20 con 0x7f4cb00c4a10 2023-10-30T14:23:56.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4cb811d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb01f23e0 con 0x7f4cb0153c80 2023-10-30T14:23:56.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4cb5eb9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cb00c42a0 0x7f4cb0151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:56.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4cb5eb9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cb00c42a0 0x7f4cb0151940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46002/0 (socket says 172.21.15.57:46002) 2023-10-30T14:23:56.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4cb5eb9640 1 -- 172.21.15.57:0/369023283 learned_addr learned my addr 172.21.15.57:0/369023283 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:23:56.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4cb56b8640 1 --2- 172.21.15.57:0/369023283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb00c4a10 0x7f4cb00c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:56.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4cb66ba640 1 --2- 172.21.15.57:0/369023283 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb0153c80 0x7f4cb00c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:56.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4cb5eb9640 1 -- 172.21.15.57:0/369023283 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb0153c80 msgr2=0x7f4cb00c8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:56.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4cb5eb9640 1 --2- 172.21.15.57:0/369023283 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb0153c80 0x7f4cb00c8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:56.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4cb5eb9640 1 -- 172.21.15.57:0/369023283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb00c4a10 msgr2=0x7f4cb00c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:56.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4cb5eb9640 1 --2- 172.21.15.57:0/369023283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb00c4a10 0x7f4cb00c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:56.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4cb5eb9640 1 -- 172.21.15.57:0/369023283 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cac0094d0 con 0x7f4cb00c42a0 2023-10-30T14:23:56.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4cb56b8640 1 --2- 172.21.15.57:0/369023283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb00c4a10 0x7f4cb00c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:23:56.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4cb5eb9640 1 --2- 172.21.15.57:0/369023283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cb00c42a0 0x7f4cb0151940 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f4cac0b3040 tx=0x7f4cac01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:56.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4ca67fc640 1 -- 172.21.15.57:0/369023283 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac0be070 con 0x7f4cb00c42a0 2023-10-30T14:23:56.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4ca67fc640 1 -- 172.21.15.57:0/369023283 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cac0ba400 con 0x7f4cb00c42a0 2023-10-30T14:23:56.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4ca67fc640 1 -- 172.21.15.57:0/369023283 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac0ba7d0 con 0x7f4cb00c42a0 2023-10-30T14:23:56.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4cb811d640 1 -- 172.21.15.57:0/369023283 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb0154b60 con 0x7f4cb00c42a0 2023-10-30T14:23:56.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4cb811d640 1 -- 172.21.15.57:0/369023283 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb0155120 con 0x7f4cb00c42a0 2023-10-30T14:23:56.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4cb811d640 1 -- 172.21.15.57:0/369023283 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb004e170 con 0x7f4cb00c42a0 2023-10-30T14:23:56.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.725+0000 7f4ca67fc640 1 -- 172.21.15.57:0/369023283 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4cac0c3410 con 0x7f4cb00c42a0 2023-10-30T14:23:56.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.729+0000 7f4ca67fc640 1 --2- 172.21.15.57:0/369023283 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4c940664a0 0x7f4c94068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:56.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.729+0000 7f4ca67fc640 1 -- 172.21.15.57:0/369023283 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f4cac121cf0 con 0x7f4cb00c42a0 2023-10-30T14:23:56.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.729+0000 7f4cb56b8640 1 --2- 172.21.15.57:0/369023283 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4c940664a0 0x7f4c94068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:56.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.729+0000 7f4cb56b8640 1 --2- 172.21.15.57:0/369023283 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4c940664a0 0x7f4c94068960 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4cb00ca7b0 tx=0x7f4ca0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:56.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.733+0000 7f4ca67fc640 1 -- 172.21.15.57:0/369023283 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4cac0eeee0 con 0x7f4cb00c42a0 2023-10-30T14:23:56.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.937+0000 7f4cb811d640 1 -- 172.21.15.57:0/369023283 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cb00cad40 con 0x7f4cb00c42a0 2023-10-30T14:23:56.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.937+0000 7f4ca67fc640 1 -- 172.21.15.57:0/369023283 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5715 (secure 0 0 0) 0x7f4cac0b5090 con 0x7f4cb00c42a0 2023-10-30T14:23:56.940 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:23:56.940 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:23:54.764193+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:56.941 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 67 2023-10-30T14:23:56.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.941+0000 7f4cb811d640 1 -- 172.21.15.57:0/369023283 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4c940664a0 msgr2=0x7f4c94068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:56.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.941+0000 7f4cb811d640 1 --2- 172.21.15.57:0/369023283 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4c940664a0 0x7f4c94068960 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4cb00ca7b0 tx=0x7f4ca0009040 comp rx=0 tx=0).stop 2023-10-30T14:23:56.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.941+0000 7f4cb811d640 1 -- 172.21.15.57:0/369023283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cb00c42a0 msgr2=0x7f4cb0151940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:56.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.941+0000 7f4cb811d640 1 --2- 172.21.15.57:0/369023283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cb00c42a0 0x7f4cb0151940 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f4cac0b3040 tx=0x7f4cac01bad0 comp rx=0 tx=0).stop 2023-10-30T14:23:56.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.941+0000 7f4cb811d640 1 -- 172.21.15.57:0/369023283 shutdown_connections 2023-10-30T14:23:56.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.941+0000 7f4cb811d640 1 --2- 172.21.15.57:0/369023283 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4c940664a0 0x7f4c94068960 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:56.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.941+0000 7f4cb811d640 1 --2- 172.21.15.57:0/369023283 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb0153c80 0x7f4cb00c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:56.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.941+0000 7f4cb811d640 1 --2- 172.21.15.57:0/369023283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb00c4a10 0x7f4cb00c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:56.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.941+0000 7f4cb811d640 1 --2- 172.21.15.57:0/369023283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cb00c42a0 0x7f4cb0151940 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:56.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.941+0000 7f4cb811d640 1 -- 172.21.15.57:0/369023283 >> 172.21.15.57:0/369023283 conn(0x7f4cb00715c0 msgr2=0x7f4cb00c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:56.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.941+0000 7f4cb811d640 1 -- 172.21.15.57:0/369023283 shutdown_connections 2023-10-30T14:23:56.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:56.941+0000 7f4cb811d640 1 -- 172.21.15.57:0/369023283 wait complete. 2023-10-30T14:23:56.964 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T14:23:56.964 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:23:56.964 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:57.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.109+0000 7f6203780640 1 Processor -- start 2023-10-30T14:23:57.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.109+0000 7f6203780640 1 -- start start 2023-10-30T14:23:57.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.109+0000 7f6203780640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61fc06ba00 0x7f61fc15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:57.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.109+0000 7f6203780640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61fc0c2aa0 con 0x7f61fc0701a0 2023-10-30T14:23:57.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.109+0000 7f6203780640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61fc0708e0 con 0x7f61fc1537f0 2023-10-30T14:23:57.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.109+0000 7f6203780640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc0c3ab0 con 0x7f61fc06ba00 2023-10-30T14:23:57.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.113+0000 7f6201d1d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61fc06ba00 0x7f61fc15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:57.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.113+0000 7f620151c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f61fc1537f0 0x7f61fc0ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:44500/0 (socket says 172.21.15.57:44500) 2023-10-30T14:23:57.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.113+0000 7f620151c640 1 -- 172.21.15.57:0/3671752721 learned_addr learned my addr 172.21.15.57:0/3671752721 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:23:57.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.113+0000 7f61f3fff640 1 -- 172.21.15.57:0/3671752721 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906285321 0 0) 0x7f61fc0c2aa0 con 0x7f61fc0701a0 2023-10-30T14:23:57.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.113+0000 7f61f3fff640 1 -- 172.21.15.57:0/3671752721 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e8003530 con 0x7f61fc0701a0 2023-10-30T14:23:57.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.113+0000 7f61f3fff640 1 -- 172.21.15.57:0/3671752721 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925516457 0 0) 0x7f61fc0708e0 con 0x7f61fc1537f0 2023-10-30T14:23:57.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.113+0000 7f6201d1d640 1 -- 172.21.15.57:0/3671752721 >> v1:172.21.15.88:6789/0 conn(0x7f61fc1537f0 legacy=0x7f61fc0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:57.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.113+0000 7f6201d1d640 1 -- 172.21.15.57:0/3671752721 >> v1:172.21.15.57:6789/0 conn(0x7f61fc0701a0 legacy=0x7f61fc06b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:57.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.113+0000 7f6201d1d640 1 -- 172.21.15.57:0/3671752721 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61fc0c1eb0 con 0x7f61fc06ba00 2023-10-30T14:23:57.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.117+0000 7f6201d1d640 1 --2- 172.21.15.57:0/3671752721 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61fc06ba00 0x7f61fc15a2e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f61f4009dc0 tx=0x7f61f401f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c4969067df77084 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:57.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.117+0000 7f61f3fff640 1 -- 172.21.15.57:0/3671752721 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f40af600 con 0x7f61fc06ba00 2023-10-30T14:23:57.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.117+0000 7f61f3fff640 1 -- 172.21.15.57:0/3671752721 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f40ab050 con 0x7f61fc06ba00 2023-10-30T14:23:57.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.117+0000 7f61f3fff640 1 -- 172.21.15.57:0/3671752721 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f40af980 con 0x7f61fc06ba00 2023-10-30T14:23:57.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.117+0000 7f6203780640 1 -- 172.21.15.57:0/3671752721 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61fc06ba00 msgr2=0x7f61fc15a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:57.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.117+0000 7f6203780640 1 --2- 172.21.15.57:0/3671752721 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61fc06ba00 0x7f61fc15a2e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f61f4009dc0 tx=0x7f61f401f150 comp rx=0 tx=0).stop 2023-10-30T14:23:57.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.117+0000 7f6203780640 1 -- 172.21.15.57:0/3671752721 shutdown_connections 2023-10-30T14:23:57.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.117+0000 7f6203780640 1 --2- 172.21.15.57:0/3671752721 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61fc06ba00 0x7f61fc15a2e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.117+0000 7f6203780640 1 -- 172.21.15.57:0/3671752721 >> 172.21.15.57:0/3671752721 conn(0x7f61fc071fb0 msgr2=0x7f61fc0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:57.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.117+0000 7f6203780640 1 -- 172.21.15.57:0/3671752721 shutdown_connections 2023-10-30T14:23:57.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.117+0000 7f6203780640 1 -- 172.21.15.57:0/3671752721 wait complete. 2023-10-30T14:23:57.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.117+0000 7f6203780640 1 Processor -- start 2023-10-30T14:23:57.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.117+0000 7f6203780640 1 -- start start 2023-10-30T14:23:57.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f6203780640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61fc06ba00 0x7f61fc156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:57.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f6203780640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61fc0701a0 0x7f61fc154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:57.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f6203780640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61fc1537f0 0x7f61fc154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:57.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f6203780640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc159780 con 0x7f61fc1537f0 2023-10-30T14:23:57.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f6203780640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc149bd0 con 0x7f61fc06ba00 2023-10-30T14:23:57.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f6203780640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc158a10 con 0x7f61fc0701a0 2023-10-30T14:23:57.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f6201d1d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61fc1537f0 0x7f61fc154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:57.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f6201d1d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61fc1537f0 0x7f61fc154860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46010/0 (socket says 172.21.15.57:46010) 2023-10-30T14:23:57.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f6201d1d640 1 -- 172.21.15.57:0/3231703003 learned_addr learned my addr 172.21.15.57:0/3231703003 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:23:57.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f620151c640 1 --2- 172.21.15.57:0/3231703003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61fc06ba00 0x7f61fc156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:57.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f6200d1b640 1 --2- 172.21.15.57:0/3231703003 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61fc0701a0 0x7f61fc154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:57.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f6201d1d640 1 -- 172.21.15.57:0/3231703003 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61fc0701a0 msgr2=0x7f61fc154320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:57.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f6201d1d640 1 --2- 172.21.15.57:0/3231703003 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61fc0701a0 0x7f61fc154320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f6201d1d640 1 -- 172.21.15.57:0/3231703003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61fc06ba00 msgr2=0x7f61fc156830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:57.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f6201d1d640 1 --2- 172.21.15.57:0/3231703003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61fc06ba00 0x7f61fc156830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f6201d1d640 1 -- 172.21.15.57:0/3231703003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f4009a70 con 0x7f61fc1537f0 2023-10-30T14:23:57.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f620151c640 1 --2- 172.21.15.57:0/3231703003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61fc06ba00 0x7f61fc156830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:23:57.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f6201d1d640 1 --2- 172.21.15.57:0/3231703003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61fc1537f0 0x7f61fc154860 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f61f40b2040 tx=0x7f61f401bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:57.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f61f1ffb640 1 -- 172.21.15.57:0/3231703003 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f4009450 con 0x7f61fc1537f0 2023-10-30T14:23:57.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f61f1ffb640 1 -- 172.21.15.57:0/3231703003 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f40b9410 con 0x7f61fc1537f0 2023-10-30T14:23:57.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f6203780640 1 -- 172.21.15.57:0/3231703003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61fc158c60 con 0x7f61fc1537f0 2023-10-30T14:23:57.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f61f1ffb640 1 -- 172.21.15.57:0/3231703003 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f40b9780 con 0x7f61fc1537f0 2023-10-30T14:23:57.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f6203780640 1 -- 172.21.15.57:0/3231703003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61fc1591c0 con 0x7f61fc1537f0 2023-10-30T14:23:57.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.121+0000 7f6203780640 1 -- 172.21.15.57:0/3231703003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61fc159020 con 0x7f61fc1537f0 2023-10-30T14:23:57.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.125+0000 7f61f1ffb640 1 -- 172.21.15.57:0/3231703003 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f61f40b6050 con 0x7f61fc1537f0 2023-10-30T14:23:57.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.125+0000 7f61f1ffb640 1 --2- 172.21.15.57:0/3231703003 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f61e00664a0 0x7f61e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:57.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.125+0000 7f61f1ffb640 1 -- 172.21.15.57:0/3231703003 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f61f41222e0 con 0x7f61fc1537f0 2023-10-30T14:23:57.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.125+0000 7f620151c640 1 --2- 172.21.15.57:0/3231703003 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f61e00664a0 0x7f61e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:57.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.125+0000 7f620151c640 1 --2- 172.21.15.57:0/3231703003 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f61e00664a0 0x7f61e0068960 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f61f8000f70 tx=0x7f61f8007cb0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:57.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.129+0000 7f61f1ffb640 1 -- 172.21.15.57:0/3231703003 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f61fc159020 con 0x7f61fc1537f0 2023-10-30T14:23:57.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.321+0000 7f6203780640 1 -- 172.21.15.57:0/3231703003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61fc159020 con 0x7f61fc1537f0 2023-10-30T14:23:57.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.321+0000 7f61f1ffb640 1 -- 172.21.15.57:0/3231703003 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5715 (secure 0 0 0) 0x7f61fc159020 con 0x7f61fc1537f0 2023-10-30T14:23:57.324 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:23:57.324 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:23:54.764193+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:57.324 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 67 2023-10-30T14:23:57.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.325+0000 7f6203780640 1 -- 172.21.15.57:0/3231703003 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f61e00664a0 msgr2=0x7f61e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:57.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.325+0000 7f6203780640 1 --2- 172.21.15.57:0/3231703003 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f61e00664a0 0x7f61e0068960 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f61f8000f70 tx=0x7f61f8007cb0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.325+0000 7f6203780640 1 -- 172.21.15.57:0/3231703003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61fc1537f0 msgr2=0x7f61fc154860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:57.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.325+0000 7f6203780640 1 --2- 172.21.15.57:0/3231703003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61fc1537f0 0x7f61fc154860 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f61f40b2040 tx=0x7f61f401bef0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.325+0000 7f6203780640 1 -- 172.21.15.57:0/3231703003 shutdown_connections 2023-10-30T14:23:57.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.325+0000 7f6203780640 1 --2- 172.21.15.57:0/3231703003 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f61e00664a0 0x7f61e0068960 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.325+0000 7f6203780640 1 --2- 172.21.15.57:0/3231703003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61fc1537f0 0x7f61fc154860 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.325+0000 7f6203780640 1 --2- 172.21.15.57:0/3231703003 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61fc0701a0 0x7f61fc154320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.325+0000 7f6203780640 1 --2- 172.21.15.57:0/3231703003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61fc06ba00 0x7f61fc156830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.325+0000 7f6203780640 1 -- 172.21.15.57:0/3231703003 >> 172.21.15.57:0/3231703003 conn(0x7f61fc071fb0 msgr2=0x7f61fc06c660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:57.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.325+0000 7f6203780640 1 -- 172.21.15.57:0/3231703003 shutdown_connections 2023-10-30T14:23:57.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:57.325+0000 7f6203780640 1 -- 172.21.15.57:0/3231703003 wait complete. 2023-10-30T14:23:57.347 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 67, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 67, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5289": { "addr": "172.21.15.57:6836/2427081813", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2427081813, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2427081813, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5289, "incarnation": 35, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 21 }, "gid_5586": { "addr": "172.21.15.57:6837/862088802", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 862088802, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 862088802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5586, "incarnation": 67, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:23:54.764193+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5289, "mds_1": 5586 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6839/358521496", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 358521496, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 358521496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 66, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5315, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:23:59.347 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:59.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.493+0000 7fb436406640 1 Processor -- start 2023-10-30T14:23:59.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.493+0000 7fb436406640 1 -- start start 2023-10-30T14:23:59.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.497+0000 7fb436406640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4300c8eb0 0x7fb43015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:59.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.497+0000 7fb436406640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb43015ac70 0x7fb43015a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:59.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.497+0000 7fb436406640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb43006aeb0 0x7fb43006b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:59.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.497+0000 7fb436406640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb430159800 con 0x7fb43015ac70 2023-10-30T14:23:59.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.497+0000 7fb436406640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb43015dcc0 con 0x7fb43006aeb0 2023-10-30T14:23:59.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.497+0000 7fb436406640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb430159d80 con 0x7fb4300c8eb0 2023-10-30T14:23:59.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.497+0000 7fb434c03640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb43015ac70 0x7fb43015a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:59.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.497+0000 7fb435c05640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb43006aeb0 0x7fb43006b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:59.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.497+0000 7fb434c03640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb43015ac70 0x7fb43015a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46012/0 (socket says 172.21.15.57:46012) 2023-10-30T14:23:59.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.497+0000 7fb434c03640 1 -- 172.21.15.57:0/3197255103 learned_addr learned my addr 172.21.15.57:0/3197255103 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:23:59.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.497+0000 7fb435404640 1 --2- 172.21.15.57:0/3197255103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4300c8eb0 0x7fb43015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:59.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.497+0000 7fb434c03640 1 -- 172.21.15.57:0/3197255103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4300c8eb0 msgr2=0x7fb43015cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:59.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.497+0000 7fb434c03640 1 --2- 172.21.15.57:0/3197255103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4300c8eb0 0x7fb43015cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.497+0000 7fb434c03640 1 -- 172.21.15.57:0/3197255103 >> v2:172.21.15.88:3300/0 conn(0x7fb43006aeb0 msgr2=0x7fb43006b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:59.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.497+0000 7fb434c03640 1 --2- 172.21.15.57:0/3197255103 >> v2:172.21.15.88:3300/0 conn(0x7fb43006aeb0 0x7fb43006b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.497+0000 7fb434c03640 1 -- 172.21.15.57:0/3197255103 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43015a750 con 0x7fb43015ac70 2023-10-30T14:23:59.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.501+0000 7fb434c03640 1 --2- 172.21.15.57:0/3197255103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb43015ac70 0x7fb43015a210 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb43008cfd0 tx=0x7fb42001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78fdcc2ad9062d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:59.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.501+0000 7fb427fff640 1 -- 172.21.15.57:0/3197255103 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4200a89b0 con 0x7fb43015ac70 2023-10-30T14:23:59.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.501+0000 7fb427fff640 1 -- 172.21.15.57:0/3197255103 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4200a8b10 con 0x7fb43015ac70 2023-10-30T14:23:59.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.501+0000 7fb427fff640 1 -- 172.21.15.57:0/3197255103 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4200a8ea0 con 0x7fb43015ac70 2023-10-30T14:23:59.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.501+0000 7fb436406640 1 -- 172.21.15.57:0/3197255103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb43015ac70 msgr2=0x7fb43015a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:59.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.501+0000 7fb436406640 1 --2- 172.21.15.57:0/3197255103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb43015ac70 0x7fb43015a210 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb43008cfd0 tx=0x7fb42001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.501+0000 7fb436406640 1 -- 172.21.15.57:0/3197255103 shutdown_connections 2023-10-30T14:23:59.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.501+0000 7fb436406640 1 --2- 172.21.15.57:0/3197255103 >> v2:172.21.15.88:3300/0 conn(0x7fb43006aeb0 0x7fb43006b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.501+0000 7fb436406640 1 --2- 172.21.15.57:0/3197255103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb43015ac70 0x7fb43015a210 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.501+0000 7fb436406640 1 --2- 172.21.15.57:0/3197255103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4300c8eb0 0x7fb43015cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.501+0000 7fb436406640 1 -- 172.21.15.57:0/3197255103 >> 172.21.15.57:0/3197255103 conn(0x7fb4300cef00 msgr2=0x7fb4300cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:59.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.501+0000 7fb436406640 1 -- 172.21.15.57:0/3197255103 shutdown_connections 2023-10-30T14:23:59.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.501+0000 7fb436406640 1 -- 172.21.15.57:0/3197255103 wait complete. 2023-10-30T14:23:59.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb436406640 1 Processor -- start 2023-10-30T14:23:59.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb436406640 1 -- start start 2023-10-30T14:23:59.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb436406640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb43006aeb0 0x7fb4301529d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:59.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb436406640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4300c8eb0 0x7fb4301ef040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:59.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb436406640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb43015ac70 0x7fb43014d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:59.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb436406640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4301eebb0 con 0x7fb4300c8eb0 2023-10-30T14:23:59.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb436406640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4301e6320 con 0x7fb43015ac70 2023-10-30T14:23:59.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb436406640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4300c8830 con 0x7fb43006aeb0 2023-10-30T14:23:59.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb434c03640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4300c8eb0 0x7fb4301ef040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:59.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb434c03640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4300c8eb0 0x7fb4301ef040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46028/0 (socket says 172.21.15.57:46028) 2023-10-30T14:23:59.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb434c03640 1 -- 172.21.15.57:0/1997955662 learned_addr learned my addr 172.21.15.57:0/1997955662 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:23:59.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb435404640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb43006aeb0 0x7fb4301529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:59.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb435c05640 1 --2- 172.21.15.57:0/1997955662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb43015ac70 0x7fb43014d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:59.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb434c03640 1 -- 172.21.15.57:0/1997955662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb43006aeb0 msgr2=0x7fb4301529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:59.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb434c03640 1 --2- 172.21.15.57:0/1997955662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb43006aeb0 0x7fb4301529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb434c03640 1 -- 172.21.15.57:0/1997955662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb43015ac70 msgr2=0x7fb43014d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:59.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb434c03640 1 --2- 172.21.15.57:0/1997955662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb43015ac70 0x7fb43014d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb434c03640 1 -- 172.21.15.57:0/1997955662 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4200094d0 con 0x7fb4300c8eb0 2023-10-30T14:23:59.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb434c03640 1 --2- 172.21.15.57:0/1997955662 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4300c8eb0 0x7fb4301ef040 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb42001d3a0 tx=0x7fb42001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:59.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb425ffb640 1 -- 172.21.15.57:0/1997955662 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4200ba600 con 0x7fb4300c8eb0 2023-10-30T14:23:59.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb425ffb640 1 -- 172.21.15.57:0/1997955662 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4200ba760 con 0x7fb4300c8eb0 2023-10-30T14:23:59.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb425ffb640 1 -- 172.21.15.57:0/1997955662 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4200bab10 con 0x7fb4300c8eb0 2023-10-30T14:23:59.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb436406640 1 -- 172.21.15.57:0/1997955662 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4301e65c0 con 0x7fb4300c8eb0 2023-10-30T14:23:59.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb436406640 1 -- 172.21.15.57:0/1997955662 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb43014f0c0 con 0x7fb4300c8eb0 2023-10-30T14:23:59.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.505+0000 7fb436406640 1 -- 172.21.15.57:0/1997955662 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4301e6750 con 0x7fb4300c8eb0 2023-10-30T14:23:59.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.509+0000 7fb425ffb640 1 -- 172.21.15.57:0/1997955662 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb4200bac70 con 0x7fb4300c8eb0 2023-10-30T14:23:59.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.509+0000 7fb425ffb640 1 --2- 172.21.15.57:0/1997955662 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4140664a0 0x7fb414068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:59.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.509+0000 7fb425ffb640 1 -- 172.21.15.57:0/1997955662 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6341+0+0 (secure 0 0 0) 0x7fb420122510 con 0x7fb4300c8eb0 2023-10-30T14:23:59.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.509+0000 7fb435404640 1 --2- 172.21.15.57:0/1997955662 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4140664a0 0x7fb414068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:59.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.509+0000 7fb435404640 1 --2- 172.21.15.57:0/1997955662 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4140664a0 0x7fb414068960 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb42c007a60 tx=0x7fb42c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:59.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.513+0000 7fb425ffb640 1 -- 172.21.15.57:0/1997955662 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb4200ef850 con 0x7fb4300c8eb0 2023-10-30T14:23:59.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.713+0000 7fb436406640 1 -- 172.21.15.57:0/1997955662 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4301ee9a0 con 0x7fb4300c8eb0 2023-10-30T14:23:59.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.713+0000 7fb425ffb640 1 -- 172.21.15.57:0/1997955662 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5715 (secure 0 0 0) 0x7fb4200f2e70 con 0x7fb4300c8eb0 2023-10-30T14:23:59.716 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:23:59.716 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:23:54.764193+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:59.717 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 67 2023-10-30T14:23:59.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.717+0000 7fb436406640 1 -- 172.21.15.57:0/1997955662 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4140664a0 msgr2=0x7fb414068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:59.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.717+0000 7fb436406640 1 --2- 172.21.15.57:0/1997955662 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4140664a0 0x7fb414068960 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb42c007a60 tx=0x7fb42c008040 comp rx=0 tx=0).stop 2023-10-30T14:23:59.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.717+0000 7fb436406640 1 -- 172.21.15.57:0/1997955662 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4300c8eb0 msgr2=0x7fb4301ef040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:59.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.717+0000 7fb436406640 1 --2- 172.21.15.57:0/1997955662 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4300c8eb0 0x7fb4301ef040 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb42001d3a0 tx=0x7fb42001ba90 comp rx=0 tx=0).stop 2023-10-30T14:23:59.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.717+0000 7fb436406640 1 -- 172.21.15.57:0/1997955662 shutdown_connections 2023-10-30T14:23:59.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.717+0000 7fb436406640 1 --2- 172.21.15.57:0/1997955662 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4140664a0 0x7fb414068960 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.717+0000 7fb436406640 1 --2- 172.21.15.57:0/1997955662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb43015ac70 0x7fb43014d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.717+0000 7fb436406640 1 --2- 172.21.15.57:0/1997955662 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4300c8eb0 0x7fb4301ef040 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.717+0000 7fb436406640 1 --2- 172.21.15.57:0/1997955662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb43006aeb0 0x7fb4301529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.717+0000 7fb436406640 1 -- 172.21.15.57:0/1997955662 >> 172.21.15.57:0/1997955662 conn(0x7fb4300cef00 msgr2=0x7fb43015b840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:59.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.717+0000 7fb436406640 1 -- 172.21.15.57:0/1997955662 shutdown_connections 2023-10-30T14:23:59.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:23:59.717+0000 7fb436406640 1 -- 172.21.15.57:0/1997955662 wait complete. 2023-10-30T14:23:59.742 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:24:01.299 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:24:01.304 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:24:01.743 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:01.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.889+0000 7fa89bbaa640 1 Processor -- start 2023-10-30T14:24:01.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.889+0000 7fa89bbaa640 1 -- start start 2023-10-30T14:24:01.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.889+0000 7fa89bbaa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa894153c80 0x7fa89415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:01.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.889+0000 7fa89bbaa640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa8940c4a10 0x7fa8940c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:01.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.889+0000 7fa89bbaa640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa894159f20 con 0x7fa89415a3b0 2023-10-30T14:24:01.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.889+0000 7fa89bbaa640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa89415dec0 con 0x7fa8940c4a10 2023-10-30T14:24:01.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.889+0000 7fa89bbaa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8941599a0 con 0x7fa894153c80 2023-10-30T14:24:01.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.889+0000 7fa89a147640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa89415a3b0 0x7fa8940ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:52534/0 (socket says 172.21.15.57:52534) 2023-10-30T14:24:01.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.889+0000 7fa899145640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa8940c4a10 0x7fa8940c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:01.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.889+0000 7fa89a147640 1 -- 172.21.15.57:0/1195518645 learned_addr learned my addr 172.21.15.57:0/1195518645 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:24:01.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.889+0000 7fa899946640 1 --2- 172.21.15.57:0/1195518645 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa894153c80 0x7fa89415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:01.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.889+0000 7fa899145640 1 -- 172.21.15.57:0/1195518645 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa894153c80 msgr2=0x7fa89415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:01.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.889+0000 7fa899145640 1 --2- 172.21.15.57:0/1195518645 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa894153c80 0x7fa89415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:01.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.889+0000 7fa899145640 1 -- 172.21.15.57:0/1195518645 >> v1:172.21.15.57:6789/0 conn(0x7fa89415a3b0 legacy=0x7fa8940ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:01.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.889+0000 7fa899145640 1 -- 172.21.15.57:0/1195518645 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8940c47e0 con 0x7fa8940c4a10 2023-10-30T14:24:01.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.889+0000 7fa898944640 1 -- 172.21.15.57:0/1195518645 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740781870 0 0) 0x7fa894159f20 con 0x7fa89415a3b0 2023-10-30T14:24:01.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.893+0000 7fa899145640 1 --2- 172.21.15.57:0/1195518645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8940c4a10 0x7fa8940c42a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fa8840097c0 tx=0x7fa88401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a91f29d0556a40ca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:01.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.893+0000 7fa898944640 1 -- 172.21.15.57:0/1195518645 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8840a8970 con 0x7fa8940c4a10 2023-10-30T14:24:01.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.893+0000 7fa898944640 1 -- 172.21.15.57:0/1195518645 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8840a8ad0 con 0x7fa8940c4a10 2023-10-30T14:24:01.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.893+0000 7fa898944640 1 -- 172.21.15.57:0/1195518645 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8840a8e60 con 0x7fa8940c4a10 2023-10-30T14:24:01.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.893+0000 7fa89bbaa640 1 -- 172.21.15.57:0/1195518645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8940c4a10 msgr2=0x7fa8940c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:01.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.893+0000 7fa89bbaa640 1 --2- 172.21.15.57:0/1195518645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8940c4a10 0x7fa8940c42a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fa8840097c0 tx=0x7fa88401ec70 comp rx=0 tx=0).stop 2023-10-30T14:24:01.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.893+0000 7fa89bbaa640 1 -- 172.21.15.57:0/1195518645 shutdown_connections 2023-10-30T14:24:01.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.893+0000 7fa89bbaa640 1 --2- 172.21.15.57:0/1195518645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8940c4a10 0x7fa8940c42a0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:01.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.893+0000 7fa89bbaa640 1 --2- 172.21.15.57:0/1195518645 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa894153c80 0x7fa89415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:01.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.893+0000 7fa89bbaa640 1 -- 172.21.15.57:0/1195518645 >> 172.21.15.57:0/1195518645 conn(0x7fa8940715c0 msgr2=0x7fa8940719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:01.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.893+0000 7fa89bbaa640 1 -- 172.21.15.57:0/1195518645 shutdown_connections 2023-10-30T14:24:01.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.893+0000 7fa89bbaa640 1 -- 172.21.15.57:0/1195518645 wait complete. 2023-10-30T14:24:01.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa89bbaa640 1 Processor -- start 2023-10-30T14:24:01.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa89bbaa640 1 -- start start 2023-10-30T14:24:01.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa89bbaa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8940c4a10 0x7fa8940ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:01.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa89bbaa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa894153c80 0x7fa8940b4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:01.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa89bbaa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa89415a3b0 0x7fa8940b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:01.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa89bbaa640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8940b4070 con 0x7fa89415a3b0 2023-10-30T14:24:01.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa89bbaa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8940b3180 con 0x7fa894153c80 2023-10-30T14:24:01.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa89bbaa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8940b3460 con 0x7fa8940c4a10 2023-10-30T14:24:01.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa89a147640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa89415a3b0 0x7fa8940b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:01.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa899946640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8940c4a10 0x7fa8940ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:01.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa899145640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa894153c80 0x7fa8940b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:01.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa89a147640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa89415a3b0 0x7fa8940b4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46048/0 (socket says 172.21.15.57:46048) 2023-10-30T14:24:01.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa89a147640 1 -- 172.21.15.57:0/3079886490 learned_addr learned my addr 172.21.15.57:0/3079886490 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:24:01.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa89a147640 1 -- 172.21.15.57:0/3079886490 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8940c4a10 msgr2=0x7fa8940ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:01.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa89a147640 1 --2- 172.21.15.57:0/3079886490 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8940c4a10 0x7fa8940ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:01.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa89a147640 1 -- 172.21.15.57:0/3079886490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa894153c80 msgr2=0x7fa8940b4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:01.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa89a147640 1 --2- 172.21.15.57:0/3079886490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa894153c80 0x7fa8940b4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:01.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa89a147640 1 -- 172.21.15.57:0/3079886490 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa884009420 con 0x7fa89415a3b0 2023-10-30T14:24:01.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa89a147640 1 --2- 172.21.15.57:0/3079886490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa89415a3b0 0x7fa8940b4ec0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fa88c00c780 tx=0x7fa88c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:01.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa88a7fc640 1 -- 172.21.15.57:0/3079886490 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c00ce90 con 0x7fa89415a3b0 2023-10-30T14:24:01.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa88a7fc640 1 -- 172.21.15.57:0/3079886490 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa88c00a640 con 0x7fa89415a3b0 2023-10-30T14:24:01.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa88a7fc640 1 -- 172.21.15.57:0/3079886490 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c00a9d0 con 0x7fa89415a3b0 2023-10-30T14:24:01.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa89bbaa640 1 -- 172.21.15.57:0/3079886490 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8940b3740 con 0x7fa89415a3b0 2023-10-30T14:24:01.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.897+0000 7fa89bbaa640 1 -- 172.21.15.57:0/3079886490 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8940b3d00 con 0x7fa89415a3b0 2023-10-30T14:24:01.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.901+0000 7fa89bbaa640 1 -- 172.21.15.57:0/3079886490 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8940b4220 con 0x7fa89415a3b0 2023-10-30T14:24:01.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.901+0000 7fa88a7fc640 1 -- 172.21.15.57:0/3079886490 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa88c00ab30 con 0x7fa89415a3b0 2023-10-30T14:24:01.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.901+0000 7fa88a7fc640 1 --2- 172.21.15.57:0/3079886490 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa8780664a0 0x7fa878068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:01.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.901+0000 7fa88a7fc640 1 -- 172.21.15.57:0/3079886490 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6341+0+0 (secure 0 0 0) 0x7fa88c083630 con 0x7fa89415a3b0 2023-10-30T14:24:01.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.901+0000 7fa899946640 1 --2- 172.21.15.57:0/3079886490 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa8780664a0 0x7fa878068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:01.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.901+0000 7fa899946640 1 --2- 172.21.15.57:0/3079886490 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa8780664a0 0x7fa878068960 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa890009b60 tx=0x7fa890009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:01.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:01.905+0000 7fa88a7fc640 1 -- 172.21.15.57:0/3079886490 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa8940b4220 con 0x7fa89415a3b0 2023-10-30T14:24:02.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:02.117+0000 7fa89bbaa640 1 -- 172.21.15.57:0/3079886490 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8940b4220 con 0x7fa89415a3b0 2023-10-30T14:24:02.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:02.117+0000 7fa88a7fc640 1 -- 172.21.15.57:0/3079886490 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+5717 (secure 0 0 0) 0x7fa8940b4220 con 0x7fa89415a3b0 2023-10-30T14:24:02.123 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:24:02.123 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:24:00.268818+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:resolve","state_seq":45,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:02.124 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 68 2023-10-30T14:24:02.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:02.121+0000 7fa89bbaa640 1 -- 172.21.15.57:0/3079886490 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa8780664a0 msgr2=0x7fa878068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:02.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:02.121+0000 7fa89bbaa640 1 --2- 172.21.15.57:0/3079886490 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa8780664a0 0x7fa878068960 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa890009b60 tx=0x7fa890009190 comp rx=0 tx=0).stop 2023-10-30T14:24:02.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:02.121+0000 7fa89bbaa640 1 -- 172.21.15.57:0/3079886490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa89415a3b0 msgr2=0x7fa8940b4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:02.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:02.121+0000 7fa89bbaa640 1 --2- 172.21.15.57:0/3079886490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa89415a3b0 0x7fa8940b4ec0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fa88c00c780 tx=0x7fa88c00cc40 comp rx=0 tx=0).stop 2023-10-30T14:24:02.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:02.121+0000 7fa89bbaa640 1 -- 172.21.15.57:0/3079886490 shutdown_connections 2023-10-30T14:24:02.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:02.121+0000 7fa89bbaa640 1 --2- 172.21.15.57:0/3079886490 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa8780664a0 0x7fa878068960 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:02.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:02.121+0000 7fa89bbaa640 1 --2- 172.21.15.57:0/3079886490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa89415a3b0 0x7fa8940b4ec0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:02.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:02.121+0000 7fa89bbaa640 1 --2- 172.21.15.57:0/3079886490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa894153c80 0x7fa8940b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:02.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:02.121+0000 7fa89bbaa640 1 --2- 172.21.15.57:0/3079886490 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8940c4a10 0x7fa8940ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:02.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:02.121+0000 7fa89bbaa640 1 -- 172.21.15.57:0/3079886490 >> 172.21.15.57:0/3079886490 conn(0x7fa8940715c0 msgr2=0x7fa894157a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:02.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:02.121+0000 7fa89bbaa640 1 -- 172.21.15.57:0/3079886490 shutdown_connections 2023-10-30T14:24:02.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:02.121+0000 7fa89bbaa640 1 -- 172.21.15.57:0/3079886490 wait complete. 2023-10-30T14:24:02.144 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:24:04.145 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:04.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.293+0000 7f9a9c817640 1 Processor -- start 2023-10-30T14:24:04.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.293+0000 7f9a9c817640 1 -- start start 2023-10-30T14:24:04.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.297+0000 7f9a9c817640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a980c8eb0 0x7f9a9815cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:04.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.297+0000 7f9a9c817640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9a9806aeb0 0x7f9a98069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:04.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.297+0000 7f9a9c817640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9815dcc0 con 0x7f9a9815ac70 2023-10-30T14:24:04.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.297+0000 7f9a9c817640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a98159800 con 0x7f9a9806aeb0 2023-10-30T14:24:04.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.297+0000 7f9a9c817640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a98159d80 con 0x7f9a980c8eb0 2023-10-30T14:24:04.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.297+0000 7f9a96ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f9a9815ac70 0x7f9a9815a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:51344/0 (socket says 172.21.15.57:51344) 2023-10-30T14:24:04.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.297+0000 7f9a96ffd640 1 -- 172.21.15.57:0/3767505370 learned_addr learned my addr 172.21.15.57:0/3767505370 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:24:04.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.297+0000 7f9a977fe640 1 --2- 172.21.15.57:0/3767505370 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a980c8eb0 0x7f9a9815cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:04.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.297+0000 7f9a97fff640 1 --2- 172.21.15.57:0/3767505370 >> v2:172.21.15.57:3300/0 conn(0x7f9a9806aeb0 0x7f9a98069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:04.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.297+0000 7f9a967fc640 1 -- 172.21.15.57:0/3767505370 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296911563 0 0) 0x7f9a8c001120 con 0x7f9a9815ac70 2023-10-30T14:24:04.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.297+0000 7f9a967fc640 1 -- 172.21.15.57:0/3767505370 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a80003530 con 0x7f9a9815ac70 2023-10-30T14:24:04.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.297+0000 7f9a967fc640 1 -- 172.21.15.57:0/3767505370 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3948955949 0 0) 0x7f9a80003530 con 0x7f9a9815ac70 2023-10-30T14:24:04.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.297+0000 7f9a977fe640 1 -- 172.21.15.57:0/3767505370 >> v2:172.21.15.57:3300/0 conn(0x7f9a9806aeb0 msgr2=0x7f9a98069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:04.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.297+0000 7f9a977fe640 1 --2- 172.21.15.57:0/3767505370 >> v2:172.21.15.57:3300/0 conn(0x7f9a9806aeb0 0x7f9a98069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:04.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.297+0000 7f9a977fe640 1 -- 172.21.15.57:0/3767505370 >> v1:172.21.15.57:6789/0 conn(0x7f9a9815ac70 legacy=0x7f9a9815a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:04.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.297+0000 7f9a977fe640 1 -- 172.21.15.57:0/3767505370 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9815d420 con 0x7f9a980c8eb0 2023-10-30T14:24:04.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.297+0000 7f9a97fff640 1 --2- 172.21.15.57:0/3767505370 >> v2:172.21.15.57:3300/0 conn(0x7f9a9806aeb0 0x7f9a98069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:24:04.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.301+0000 7f9a977fe640 1 --2- 172.21.15.57:0/3767505370 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a980c8eb0 0x7f9a9815cd70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9a880097c0 tx=0x7f9a8801eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9a83239fc55a284 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:04.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.301+0000 7f9a967fc640 1 -- 172.21.15.57:0/3767505370 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c001810 con 0x7f9a980c8eb0 2023-10-30T14:24:04.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.301+0000 7f9a967fc640 1 -- 172.21.15.57:0/3767505370 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a880a85d0 con 0x7f9a980c8eb0 2023-10-30T14:24:04.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.301+0000 7f9a967fc640 1 -- 172.21.15.57:0/3767505370 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a880a8940 con 0x7f9a980c8eb0 2023-10-30T14:24:04.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.301+0000 7f9a9c817640 1 -- 172.21.15.57:0/3767505370 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a980c8eb0 msgr2=0x7f9a9815cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:04.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.301+0000 7f9a9c817640 1 --2- 172.21.15.57:0/3767505370 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a980c8eb0 0x7f9a9815cd70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9a880097c0 tx=0x7f9a8801eb90 comp rx=0 tx=0).stop 2023-10-30T14:24:04.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.301+0000 7f9a9c817640 1 -- 172.21.15.57:0/3767505370 shutdown_connections 2023-10-30T14:24:04.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.301+0000 7f9a9c817640 1 --2- 172.21.15.57:0/3767505370 >> v2:172.21.15.57:3300/0 conn(0x7f9a9806aeb0 0x7f9a98069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:04.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.301+0000 7f9a9c817640 1 --2- 172.21.15.57:0/3767505370 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a980c8eb0 0x7f9a9815cd70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:04.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.301+0000 7f9a9c817640 1 -- 172.21.15.57:0/3767505370 >> 172.21.15.57:0/3767505370 conn(0x7f9a980cef00 msgr2=0x7f9a980cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:04.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.301+0000 7f9a9c817640 1 -- 172.21.15.57:0/3767505370 shutdown_connections 2023-10-30T14:24:04.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.301+0000 7f9a9c817640 1 -- 172.21.15.57:0/3767505370 wait complete. 2023-10-30T14:24:04.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.301+0000 7f9a9c817640 1 Processor -- start 2023-10-30T14:24:04.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.301+0000 7f9a9c817640 1 -- start start 2023-10-30T14:24:04.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a9c817640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a9806aeb0 0x7f9a981f2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:04.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a9c817640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a980c8eb0 0x7f9a98150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:04.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a9c817640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a9815ac70 0x7f9a98152930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:04.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a9c817640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a981e5bf0 con 0x7f9a980c8eb0 2023-10-30T14:24:04.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a9c817640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a981e5070 con 0x7f9a9815ac70 2023-10-30T14:24:04.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a9c817640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a981f3100 con 0x7f9a9806aeb0 2023-10-30T14:24:04.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a96ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a980c8eb0 0x7f9a98150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:04.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a96ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a980c8eb0 0x7f9a98150c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55482/0 (socket says 172.21.15.57:55482) 2023-10-30T14:24:04.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a96ffd640 1 -- 172.21.15.57:0/4038735380 learned_addr learned my addr 172.21.15.57:0/4038735380 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:24:04.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a97fff640 1 --2- 172.21.15.57:0/4038735380 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a9815ac70 0x7f9a98152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:04.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a977fe640 1 --2- 172.21.15.57:0/4038735380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a9806aeb0 0x7f9a981f2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:04.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a96ffd640 1 -- 172.21.15.57:0/4038735380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a9806aeb0 msgr2=0x7f9a981f2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:04.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a96ffd640 1 --2- 172.21.15.57:0/4038735380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a9806aeb0 0x7f9a981f2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:04.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a96ffd640 1 -- 172.21.15.57:0/4038735380 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a9815ac70 msgr2=0x7f9a98152930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:04.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a96ffd640 1 --2- 172.21.15.57:0/4038735380 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a9815ac70 0x7f9a98152930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:04.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a96ffd640 1 -- 172.21.15.57:0/4038735380 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a90009a70 con 0x7f9a980c8eb0 2023-10-30T14:24:04.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a97fff640 1 --2- 172.21.15.57:0/4038735380 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a9815ac70 0x7f9a98152930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:24:04.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a96ffd640 1 --2- 172.21.15.57:0/4038735380 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a980c8eb0 0x7f9a98150c10 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c00c780 tx=0x7f9a8c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:04.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a7ffff640 1 -- 172.21.15.57:0/4038735380 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c005ae0 con 0x7f9a980c8eb0 2023-10-30T14:24:04.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a7ffff640 1 -- 172.21.15.57:0/4038735380 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a8c005c40 con 0x7f9a980c8eb0 2023-10-30T14:24:04.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a7ffff640 1 -- 172.21.15.57:0/4038735380 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c007070 con 0x7f9a980c8eb0 2023-10-30T14:24:04.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a9c817640 1 -- 172.21.15.57:0/4038735380 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a88009420 con 0x7f9a980c8eb0 2023-10-30T14:24:04.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a9c817640 1 -- 172.21.15.57:0/4038735380 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a98150250 con 0x7f9a980c8eb0 2023-10-30T14:24:04.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.305+0000 7f9a9c817640 1 -- 172.21.15.57:0/4038735380 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a9804e170 con 0x7f9a980c8eb0 2023-10-30T14:24:04.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.309+0000 7f9a7ffff640 1 -- 172.21.15.57:0/4038735380 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9a8c0071f0 con 0x7f9a980c8eb0 2023-10-30T14:24:04.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.309+0000 7f9a7ffff640 1 --2- 172.21.15.57:0/4038735380 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9a840664d0 0x7f9a84068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:04.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.309+0000 7f9a7ffff640 1 -- 172.21.15.57:0/4038735380 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f9a8c083da0 con 0x7f9a980c8eb0 2023-10-30T14:24:04.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.309+0000 7f9a977fe640 1 --2- 172.21.15.57:0/4038735380 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9a840664d0 0x7f9a84068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:04.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.309+0000 7f9a977fe640 1 --2- 172.21.15.57:0/4038735380 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9a840664d0 0x7f9a84068990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9a880b2000 tx=0x7f9a880aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:04.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.313+0000 7f9a7ffff640 1 -- 172.21.15.57:0/4038735380 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9a8c050f10 con 0x7f9a980c8eb0 2023-10-30T14:24:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.501+0000 7f9a9c817640 1 -- 172.21.15.57:0/4038735380 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a9815dcc0 con 0x7f9a980c8eb0 2023-10-30T14:24:04.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.505+0000 7f9a7ffff640 1 -- 172.21.15.57:0/4038735380 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+5717 (secure 0 0 0) 0x7f9a9815dcc0 con 0x7f9a980c8eb0 2023-10-30T14:24:04.506 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:24:04.506 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:24:00.268818+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:resolve","state_seq":45,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:04.507 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 68 2023-10-30T14:24:04.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.505+0000 7f9a9c817640 1 -- 172.21.15.57:0/4038735380 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9a840664d0 msgr2=0x7f9a84068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:04.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.505+0000 7f9a9c817640 1 --2- 172.21.15.57:0/4038735380 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9a840664d0 0x7f9a84068990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9a880b2000 tx=0x7f9a880aa070 comp rx=0 tx=0).stop 2023-10-30T14:24:04.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.505+0000 7f9a9c817640 1 -- 172.21.15.57:0/4038735380 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a980c8eb0 msgr2=0x7f9a98150c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:04.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.505+0000 7f9a9c817640 1 --2- 172.21.15.57:0/4038735380 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a980c8eb0 0x7f9a98150c10 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c00c780 tx=0x7f9a8c00cc40 comp rx=0 tx=0).stop 2023-10-30T14:24:04.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.505+0000 7f9a9c817640 1 -- 172.21.15.57:0/4038735380 shutdown_connections 2023-10-30T14:24:04.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.505+0000 7f9a9c817640 1 --2- 172.21.15.57:0/4038735380 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9a840664d0 0x7f9a84068990 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:04.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.505+0000 7f9a9c817640 1 --2- 172.21.15.57:0/4038735380 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a9815ac70 0x7f9a98152930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:04.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.505+0000 7f9a9c817640 1 --2- 172.21.15.57:0/4038735380 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a980c8eb0 0x7f9a98150c10 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:04.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.505+0000 7f9a9c817640 1 --2- 172.21.15.57:0/4038735380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a9806aeb0 0x7f9a981f2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:04.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.505+0000 7f9a9c817640 1 -- 172.21.15.57:0/4038735380 >> 172.21.15.57:0/4038735380 conn(0x7f9a980cef00 msgr2=0x7f9a980c65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:04.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.505+0000 7f9a9c817640 1 -- 172.21.15.57:0/4038735380 shutdown_connections 2023-10-30T14:24:04.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:04.505+0000 7f9a9c817640 1 -- 172.21.15.57:0/4038735380 wait complete. 2023-10-30T14:24:04.528 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:24:06.529 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.737+0000 7f43ee910640 1 Processor -- start 2023-10-30T14:24:06.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.741+0000 7f43ee910640 1 -- start start 2023-10-30T14:24:06.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.741+0000 7f43ee910640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43e814f830 0x7f43e80c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:06.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.741+0000 7f43ee910640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f43e806d4f0 0x7f43e806d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:06.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.741+0000 7f43ee910640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e80c5b10 con 0x7f43e814fc10 2023-10-30T14:24:06.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.741+0000 7f43ee910640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e80c6a70 con 0x7f43e806d4f0 2023-10-30T14:24:06.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.741+0000 7f43ee910640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e80c5df0 con 0x7f43e814f830 2023-10-30T14:24:06.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.741+0000 7f43ed90e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43e814f830 0x7f43e80c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:06.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.741+0000 7f43ed90e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43e814f830 0x7f43e80c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:50078/0 (socket says 172.21.15.57:50078) 2023-10-30T14:24:06.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.741+0000 7f43ed90e640 1 -- 172.21.15.57:0/1939230185 learned_addr learned my addr 172.21.15.57:0/1939230185 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:24:06.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.741+0000 7f43ee10f640 1 --2- 172.21.15.57:0/1939230185 >> v2:172.21.15.88:3300/0 conn(0x7f43e806d4f0 0x7f43e806d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:06.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.745+0000 7f43ed90e640 1 -- 172.21.15.57:0/1939230185 >> v2:172.21.15.88:3300/0 conn(0x7f43e806d4f0 msgr2=0x7f43e806d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:06.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.745+0000 7f43ed90e640 1 --2- 172.21.15.57:0/1939230185 >> v2:172.21.15.88:3300/0 conn(0x7f43e806d4f0 0x7f43e806d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:06.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.745+0000 7f43ed90e640 1 -- 172.21.15.57:0/1939230185 >> v1:172.21.15.88:6789/0 conn(0x7f43e814fc10 legacy=0x7f43e8156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:06.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.745+0000 7f43ed90e640 1 -- 172.21.15.57:0/1939230185 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e80c60d0 con 0x7f43e814f830 2023-10-30T14:24:06.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.749+0000 7f43ed90e640 1 --2- 172.21.15.57:0/1939230185 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43e814f830 0x7f43e80c5460 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f43e400bb10 tx=0x7f43e401ebe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=441279592170eb26 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:06.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.749+0000 7f43ec90c640 1 -- 172.21.15.57:0/1939230185 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e40a7cb0 con 0x7f43e814f830 2023-10-30T14:24:06.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.749+0000 7f43ec90c640 1 -- 172.21.15.57:0/1939230185 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e40a7e10 con 0x7f43e814f830 2023-10-30T14:24:06.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.749+0000 7f43ec90c640 1 -- 172.21.15.57:0/1939230185 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e40af660 con 0x7f43e814f830 2023-10-30T14:24:06.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.749+0000 7f43ee910640 1 -- 172.21.15.57:0/1939230185 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43e814f830 msgr2=0x7f43e80c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:06.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.749+0000 7f43ee910640 1 --2- 172.21.15.57:0/1939230185 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43e814f830 0x7f43e80c5460 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f43e400bb10 tx=0x7f43e401ebe0 comp rx=0 tx=0).stop 2023-10-30T14:24:06.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.749+0000 7f43ee910640 1 -- 172.21.15.57:0/1939230185 shutdown_connections 2023-10-30T14:24:06.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.749+0000 7f43ee910640 1 --2- 172.21.15.57:0/1939230185 >> v2:172.21.15.88:3300/0 conn(0x7f43e806d4f0 0x7f43e806d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:06.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.749+0000 7f43ee910640 1 --2- 172.21.15.57:0/1939230185 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43e814f830 0x7f43e80c5460 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:06.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.749+0000 7f43ee910640 1 -- 172.21.15.57:0/1939230185 >> 172.21.15.57:0/1939230185 conn(0x7f43e80caae0 msgr2=0x7f43e80caec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:06.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.749+0000 7f43ee910640 1 -- 172.21.15.57:0/1939230185 shutdown_connections 2023-10-30T14:24:06.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.749+0000 7f43ee910640 1 -- 172.21.15.57:0/1939230185 wait complete. 2023-10-30T14:24:06.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.753+0000 7f43ee910640 1 Processor -- start 2023-10-30T14:24:06.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.753+0000 7f43ee910640 1 -- start start 2023-10-30T14:24:06.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.753+0000 7f43ee910640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43e806d4f0 0x7f43e80c4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:06.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.753+0000 7f43ee910640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f43e814f830 0x7f43e8153d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:06.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.753+0000 7f43ee910640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43e814fc10 0x7f43e81542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:06.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.753+0000 7f43ee910640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e81ee320 con 0x7f43e814f830 2023-10-30T14:24:06.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.753+0000 7f43ee910640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e81f0e80 con 0x7f43e814fc10 2023-10-30T14:24:06.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.753+0000 7f43ee910640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e81ed6e0 con 0x7f43e806d4f0 2023-10-30T14:24:06.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.753+0000 7f43ed90e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43e806d4f0 0x7f43e80c4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:06.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.753+0000 7f43ed90e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43e806d4f0 0x7f43e80c4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:50094/0 (socket says 172.21.15.57:50094) 2023-10-30T14:24:06.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.753+0000 7f43ed90e640 1 -- 172.21.15.57:0/506909756 learned_addr learned my addr 172.21.15.57:0/506909756 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:24:06.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.753+0000 7f43ee10f640 1 --2- 172.21.15.57:0/506909756 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43e814fc10 0x7f43e81542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:06.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.753+0000 7f43ed10d640 1 --2- 172.21.15.57:0/506909756 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f43e814f830 0x7f43e8153d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:06.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.757+0000 7f43ed90e640 1 -- 172.21.15.57:0/506909756 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43e814fc10 msgr2=0x7f43e81542b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:06.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.757+0000 7f43ed90e640 1 --2- 172.21.15.57:0/506909756 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43e814fc10 0x7f43e81542b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:06.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.757+0000 7f43ed90e640 1 -- 172.21.15.57:0/506909756 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f43e814f830 msgr2=0x7f43e8153d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:06.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.757+0000 7f43ed90e640 1 --2- 172.21.15.57:0/506909756 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f43e814f830 0x7f43e8153d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:06.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.757+0000 7f43ed90e640 1 -- 172.21.15.57:0/506909756 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e400b770 con 0x7f43e806d4f0 2023-10-30T14:24:06.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.757+0000 7f43ed10d640 1 --2- 172.21.15.57:0/506909756 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f43e814f830 0x7f43e8153d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:24:06.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.757+0000 7f43ed90e640 1 --2- 172.21.15.57:0/506909756 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43e806d4f0 0x7f43e80c4180 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f43e400b2c0 tx=0x7f43e401b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:06.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.757+0000 7f43de7fc640 1 -- 172.21.15.57:0/506909756 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e40ba660 con 0x7f43e806d4f0 2023-10-30T14:24:06.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.757+0000 7f43de7fc640 1 -- 172.21.15.57:0/506909756 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e40ba7c0 con 0x7f43e806d4f0 2023-10-30T14:24:06.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.757+0000 7f43de7fc640 1 -- 172.21.15.57:0/506909756 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e40bab70 con 0x7f43e806d4f0 2023-10-30T14:24:06.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.757+0000 7f43ee910640 1 -- 172.21.15.57:0/506909756 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e81ed980 con 0x7f43e806d4f0 2023-10-30T14:24:06.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.757+0000 7f43ee910640 1 -- 172.21.15.57:0/506909756 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e81edf40 con 0x7f43e806d4f0 2023-10-30T14:24:06.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.761+0000 7f43ee910640 1 -- 172.21.15.57:0/506909756 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e804e170 con 0x7f43e806d4f0 2023-10-30T14:24:06.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.761+0000 7f43de7fc640 1 -- 172.21.15.57:0/506909756 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f43e40c2360 con 0x7f43e806d4f0 2023-10-30T14:24:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.761+0000 7f43de7fc640 1 --2- 172.21.15.57:0/506909756 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f43cc0664d0 0x7f43cc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.761+0000 7f43de7fc640 1 -- 172.21.15.57:0/506909756 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f43e411ef90 con 0x7f43e806d4f0 2023-10-30T14:24:06.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.761+0000 7f43ed10d640 1 --2- 172.21.15.57:0/506909756 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f43cc0664d0 0x7f43cc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:06.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.765+0000 7f43ed10d640 1 --2- 172.21.15.57:0/506909756 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f43cc0664d0 0x7f43cc068990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f43e80c96a0 tx=0x7f43d8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:06.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:06.773+0000 7f43de7fc640 1 -- 172.21.15.57:0/506909756 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f43e40ec1a0 con 0x7f43e806d4f0 2023-10-30T14:24:07.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:07.125+0000 7f43ee910640 1 -- 172.21.15.57:0/506909756 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43e806e4b0 con 0x7f43e806d4f0 2023-10-30T14:24:07.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:07.129+0000 7f43de7fc640 1 -- 172.21.15.57:0/506909756 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+5719 (secure 0 0 0) 0x7f43e40b4020 con 0x7f43e806d4f0 2023-10-30T14:24:07.133 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:24:07.133 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:24:05.900311+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:reconnect","state_seq":47,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:07.133 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 69 2023-10-30T14:24:07.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:07.133+0000 7f43ee910640 1 -- 172.21.15.57:0/506909756 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f43cc0664d0 msgr2=0x7f43cc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:07.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:07.133+0000 7f43ee910640 1 --2- 172.21.15.57:0/506909756 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f43cc0664d0 0x7f43cc068990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f43e80c96a0 tx=0x7f43d8009040 comp rx=0 tx=0).stop 2023-10-30T14:24:07.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:07.137+0000 7f43ee910640 1 -- 172.21.15.57:0/506909756 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43e806d4f0 msgr2=0x7f43e80c4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:07.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:07.137+0000 7f43ee910640 1 --2- 172.21.15.57:0/506909756 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43e806d4f0 0x7f43e80c4180 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f43e400b2c0 tx=0x7f43e401b980 comp rx=0 tx=0).stop 2023-10-30T14:24:07.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:07.137+0000 7f43ee910640 1 -- 172.21.15.57:0/506909756 shutdown_connections 2023-10-30T14:24:07.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:07.137+0000 7f43ee910640 1 --2- 172.21.15.57:0/506909756 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f43cc0664d0 0x7f43cc068990 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:07.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:07.137+0000 7f43ee910640 1 --2- 172.21.15.57:0/506909756 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43e814fc10 0x7f43e81542b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:07.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:07.137+0000 7f43ee910640 1 --2- 172.21.15.57:0/506909756 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f43e814f830 0x7f43e8153d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:07.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:07.137+0000 7f43ee910640 1 --2- 172.21.15.57:0/506909756 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43e806d4f0 0x7f43e80c4180 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:07.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:07.137+0000 7f43ee910640 1 -- 172.21.15.57:0/506909756 >> 172.21.15.57:0/506909756 conn(0x7f43e80caae0 msgr2=0x7f43e8156180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:07.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:07.137+0000 7f43ee910640 1 -- 172.21.15.57:0/506909756 shutdown_connections 2023-10-30T14:24:07.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:07.137+0000 7f43ee910640 1 -- 172.21.15.57:0/506909756 wait complete. 2023-10-30T14:24:07.176 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:24:09.177 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:09.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.325+0000 7f9764971640 1 Processor -- start 2023-10-30T14:24:09.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.325+0000 7f9764971640 1 -- start start 2023-10-30T14:24:09.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.325+0000 7f9764971640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97600cb180 0x7f97601597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:09.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.325+0000 7f9764971640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f976015a770 con 0x7f97600c62b0 2023-10-30T14:24:09.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.325+0000 7f9764971640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97600c9d40 con 0x7f9760153b50 2023-10-30T14:24:09.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.325+0000 7f9764971640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9760159cf0 con 0x7f97600cb180 2023-10-30T14:24:09.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.325+0000 7f975e59c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f97600c62b0 0x7f976006ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:51358/0 (socket says 172.21.15.57:51358) 2023-10-30T14:24:09.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.325+0000 7f975e59c640 1 -- 172.21.15.57:0/3886742066 learned_addr learned my addr 172.21.15.57:0/3886742066 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:24:09.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.325+0000 7f975f59e640 1 --2- 172.21.15.57:0/3886742066 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97600cb180 0x7f97601597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:09.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.325+0000 7f975dd9b640 1 -- 172.21.15.57:0/3886742066 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82447845 0 0) 0x7f9754001120 con 0x7f97600c62b0 2023-10-30T14:24:09.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.325+0000 7f975dd9b640 1 -- 172.21.15.57:0/3886742066 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9748003530 con 0x7f97600c62b0 2023-10-30T14:24:09.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.329+0000 7f975dd9b640 1 -- 172.21.15.57:0/3886742066 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729076212 0 0) 0x7f97600c9d40 con 0x7f9760153b50 2023-10-30T14:24:09.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.329+0000 7f975dd9b640 1 -- 172.21.15.57:0/3886742066 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9754001120 con 0x7f9760153b50 2023-10-30T14:24:09.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.329+0000 7f975f59e640 1 -- 172.21.15.57:0/3886742066 >> v1:172.21.15.88:6789/0 conn(0x7f9760153b50 legacy=0x7f97600ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:09.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.329+0000 7f975f59e640 1 -- 172.21.15.57:0/3886742066 >> v1:172.21.15.57:6789/0 conn(0x7f97600c62b0 legacy=0x7f976006ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:09.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.329+0000 7f975dd9b640 1 -- 172.21.15.57:0/3886742066 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2588394199 0 0) 0x7f9748003530 con 0x7f97600c62b0 2023-10-30T14:24:09.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.329+0000 7f975f59e640 1 -- 172.21.15.57:0/3886742066 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9760153f30 con 0x7f97600cb180 2023-10-30T14:24:09.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.333+0000 7f975f59e640 1 --2- 172.21.15.57:0/3886742066 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97600cb180 0x7f97601597b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9758009dc0 tx=0x7f975801f080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5db314f5b97f57e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:09.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.333+0000 7f975dd9b640 1 -- 172.21.15.57:0/3886742066 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9754001810 con 0x7f97600cb180 2023-10-30T14:24:09.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.333+0000 7f975dd9b640 1 -- 172.21.15.57:0/3886742066 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97580a8c10 con 0x7f97600cb180 2023-10-30T14:24:09.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.333+0000 7f975dd9b640 1 -- 172.21.15.57:0/3886742066 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97580b0440 con 0x7f97600cb180 2023-10-30T14:24:09.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.333+0000 7f9764971640 1 -- 172.21.15.57:0/3886742066 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97600cb180 msgr2=0x7f97601597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:09.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.333+0000 7f9764971640 1 --2- 172.21.15.57:0/3886742066 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97600cb180 0x7f97601597b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9758009dc0 tx=0x7f975801f080 comp rx=0 tx=0).stop 2023-10-30T14:24:09.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.333+0000 7f9764971640 1 -- 172.21.15.57:0/3886742066 shutdown_connections 2023-10-30T14:24:09.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.333+0000 7f9764971640 1 --2- 172.21.15.57:0/3886742066 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97600cb180 0x7f97601597b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:09.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.333+0000 7f9764971640 1 -- 172.21.15.57:0/3886742066 >> 172.21.15.57:0/3886742066 conn(0x7f9760065a80 msgr2=0x7f9760065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:09.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.333+0000 7f9764971640 1 -- 172.21.15.57:0/3886742066 shutdown_connections 2023-10-30T14:24:09.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.333+0000 7f9764971640 1 -- 172.21.15.57:0/3886742066 wait complete. 2023-10-30T14:24:09.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f9764971640 1 Processor -- start 2023-10-30T14:24:09.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f9764971640 1 -- start start 2023-10-30T14:24:09.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f9764971640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97600c62b0 0x7f976014c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:09.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f9764971640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97600cb180 0x7f97600c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:09.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f9764971640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9760153b50 0x7f97601494b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:09.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f9764971640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976014ec50 con 0x7f97600cb180 2023-10-30T14:24:09.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f9764971640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9760159160 con 0x7f97600c62b0 2023-10-30T14:24:09.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f9764971640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9760159420 con 0x7f9760153b50 2023-10-30T14:24:09.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f975e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97600cb180 0x7f97600c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:09.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f975f59e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9760153b50 0x7f97601494b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:09.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f975ed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97600c62b0 0x7f976014c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:09.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f975e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97600cb180 0x7f97600c2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55502/0 (socket says 172.21.15.57:55502) 2023-10-30T14:24:09.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f975e59c640 1 -- 172.21.15.57:0/3773688323 learned_addr learned my addr 172.21.15.57:0/3773688323 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:24:09.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f975e59c640 1 -- 172.21.15.57:0/3773688323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9760153b50 msgr2=0x7f97601494b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:09.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f975e59c640 1 --2- 172.21.15.57:0/3773688323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9760153b50 0x7f97601494b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:09.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f975e59c640 1 -- 172.21.15.57:0/3773688323 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97600c62b0 msgr2=0x7f976014c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:09.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f975e59c640 1 --2- 172.21.15.57:0/3773688323 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97600c62b0 0x7f976014c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:09.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f975e59c640 1 -- 172.21.15.57:0/3773688323 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9758009a70 con 0x7f97600cb180 2023-10-30T14:24:09.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f975e59c640 1 --2- 172.21.15.57:0/3773688323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97600cb180 0x7f97600c2c40 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f975400cac0 tx=0x7f975400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:09.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f97477fe640 1 -- 172.21.15.57:0/3773688323 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9754005cf0 con 0x7f97600cb180 2023-10-30T14:24:09.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f97477fe640 1 -- 172.21.15.57:0/3773688323 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9754005e50 con 0x7f97600cb180 2023-10-30T14:24:09.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f97477fe640 1 -- 172.21.15.57:0/3773688323 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97540072b0 con 0x7f97600cb180 2023-10-30T14:24:09.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f9764971640 1 -- 172.21.15.57:0/3773688323 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9760159700 con 0x7f97600cb180 2023-10-30T14:24:09.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.337+0000 7f9764971640 1 -- 172.21.15.57:0/3773688323 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9760158490 con 0x7f97600cb180 2023-10-30T14:24:09.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.341+0000 7f9764971640 1 -- 172.21.15.57:0/3773688323 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9760157fe0 con 0x7f97600cb180 2023-10-30T14:24:09.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.341+0000 7f97477fe640 1 -- 172.21.15.57:0/3773688323 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9754007410 con 0x7f97600cb180 2023-10-30T14:24:09.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.341+0000 7f97477fe640 1 --2- 172.21.15.57:0/3773688323 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f974c0664a0 0x7f974c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:09.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.341+0000 7f97477fe640 1 -- 172.21.15.57:0/3773688323 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f97540837f0 con 0x7f97600cb180 2023-10-30T14:24:09.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.341+0000 7f975ed9d640 1 --2- 172.21.15.57:0/3773688323 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f974c0664a0 0x7f974c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:09.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.341+0000 7f975ed9d640 1 --2- 172.21.15.57:0/3773688323 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f974c0664a0 0x7f974c068960 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f97500011f0 tx=0x7f9750009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:09.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.345+0000 7f97477fe640 1 -- 172.21.15.57:0/3773688323 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9754050a00 con 0x7f97600cb180 2023-10-30T14:24:09.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.581+0000 7f9764971640 1 -- 172.21.15.57:0/3773688323 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f976015a770 con 0x7f97600cb180 2023-10-30T14:24:09.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.581+0000 7f97477fe640 1 -- 172.21.15.57:0/3773688323 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+5719 (secure 0 0 0) 0x7f976015a770 con 0x7f97600cb180 2023-10-30T14:24:09.583 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:24:09.583 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:24:05.900311+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:reconnect","state_seq":47,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:09.584 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 69 2023-10-30T14:24:09.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.581+0000 7f9764971640 1 -- 172.21.15.57:0/3773688323 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f974c0664a0 msgr2=0x7f974c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:09.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.581+0000 7f9764971640 1 --2- 172.21.15.57:0/3773688323 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f974c0664a0 0x7f974c068960 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f97500011f0 tx=0x7f9750009040 comp rx=0 tx=0).stop 2023-10-30T14:24:09.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.581+0000 7f9764971640 1 -- 172.21.15.57:0/3773688323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97600cb180 msgr2=0x7f97600c2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:09.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.581+0000 7f9764971640 1 --2- 172.21.15.57:0/3773688323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97600cb180 0x7f97600c2c40 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f975400cac0 tx=0x7f975400cf80 comp rx=0 tx=0).stop 2023-10-30T14:24:09.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.581+0000 7f9764971640 1 -- 172.21.15.57:0/3773688323 shutdown_connections 2023-10-30T14:24:09.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.581+0000 7f9764971640 1 --2- 172.21.15.57:0/3773688323 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f974c0664a0 0x7f974c068960 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:09.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.581+0000 7f9764971640 1 --2- 172.21.15.57:0/3773688323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9760153b50 0x7f97601494b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:09.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.581+0000 7f9764971640 1 --2- 172.21.15.57:0/3773688323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97600cb180 0x7f97600c2c40 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:09.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.581+0000 7f9764971640 1 --2- 172.21.15.57:0/3773688323 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97600c62b0 0x7f976014c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:09.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.581+0000 7f9764971640 1 -- 172.21.15.57:0/3773688323 >> 172.21.15.57:0/3773688323 conn(0x7f9760065a80 msgr2=0x7f976014de40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:09.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.581+0000 7f9764971640 1 -- 172.21.15.57:0/3773688323 shutdown_connections 2023-10-30T14:24:09.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:09.585+0000 7f9764971640 1 -- 172.21.15.57:0/3773688323 wait complete. 2023-10-30T14:24:09.607 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 69, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 69, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5289": { "addr": "172.21.15.57:6836/2427081813", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2427081813, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2427081813, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5289, "incarnation": 35, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 21 }, "gid_5586": { "addr": "172.21.15.57:6837/862088802", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 862088802, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 862088802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5586, "incarnation": 67, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:reconnect", "state_seq": 47 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:24:05.900311+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5289, "mds_1": 5586 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6839/358521496", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 358521496, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 358521496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 66, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5315, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:24:11.608 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:11.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.757+0000 7f4f17501640 1 Processor -- start 2023-10-30T14:24:11.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.757+0000 7f4f17501640 1 -- start start 2023-10-30T14:24:11.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.757+0000 7f4f17501640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4f100c4a10 0x7f4f1015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:11.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.757+0000 7f4f17501640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f100c42a0 0x7f4f100ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:11.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.757+0000 7f4f17501640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f10159f20 con 0x7f4f10153c80 2023-10-30T14:24:11.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.757+0000 7f4f17501640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1015dec0 con 0x7f4f100c4a10 2023-10-30T14:24:11.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.757+0000 7f4f17501640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f101599a0 con 0x7f4f100c42a0 2023-10-30T14:24:11.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.761+0000 7f4f15a9e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f100c42a0 0x7f4f100ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:11.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.761+0000 7f4f14a9c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4f100c4a10 0x7f4f1015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:11.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.761+0000 7f4f15a9e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f100c42a0 0x7f4f100ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:50136/0 (socket says 172.21.15.57:50136) 2023-10-30T14:24:11.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.761+0000 7f4f1529d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4f10153c80 0x7f4f1015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:51636/0 (socket says 172.21.15.57:51636) 2023-10-30T14:24:11.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.761+0000 7f4f15a9e640 1 -- 172.21.15.57:0/2441991750 learned_addr learned my addr 172.21.15.57:0/2441991750 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:24:11.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.761+0000 7f4f14a9c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4f100c4a10 0x7f4f1015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:45360/0 (socket says 172.21.15.57:45360) 2023-10-30T14:24:11.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.761+0000 7f4f07fff640 1 -- 172.21.15.57:0/2441991750 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425392281 0 0) 0x7f4f10159f20 con 0x7f4f10153c80 2023-10-30T14:24:11.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.761+0000 7f4f15a9e640 1 -- 172.21.15.57:0/2441991750 >> v2:172.21.15.88:3300/0 conn(0x7f4f100c4a10 msgr2=0x7f4f1015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:11.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.761+0000 7f4f15a9e640 1 --2- 172.21.15.57:0/2441991750 >> v2:172.21.15.88:3300/0 conn(0x7f4f100c4a10 0x7f4f1015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.761+0000 7f4f15a9e640 1 -- 172.21.15.57:0/2441991750 >> v1:172.21.15.88:6789/0 conn(0x7f4f10153c80 legacy=0x7f4f1015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:11.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.761+0000 7f4f15a9e640 1 -- 172.21.15.57:0/2441991750 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1015d620 con 0x7f4f100c42a0 2023-10-30T14:24:11.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.761+0000 7f4f15a9e640 1 --2- 172.21.15.57:0/2441991750 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f100c42a0 0x7f4f100ca1e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4f08009e50 tx=0x7f4f0801f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c81e9f3b847e89d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:11.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.761+0000 7f4f07fff640 1 -- 172.21.15.57:0/2441991750 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f080af400 con 0x7f4f100c42a0 2023-10-30T14:24:11.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.765+0000 7f4f07fff640 1 -- 172.21.15.57:0/2441991750 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f080ab050 con 0x7f4f100c42a0 2023-10-30T14:24:11.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.765+0000 7f4f07fff640 1 -- 172.21.15.57:0/2441991750 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f080af780 con 0x7f4f100c42a0 2023-10-30T14:24:11.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.765+0000 7f4f17501640 1 -- 172.21.15.57:0/2441991750 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f100c42a0 msgr2=0x7f4f100ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:11.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.765+0000 7f4f17501640 1 --2- 172.21.15.57:0/2441991750 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f100c42a0 0x7f4f100ca1e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4f08009e50 tx=0x7f4f0801f1f0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.765+0000 7f4f17501640 1 -- 172.21.15.57:0/2441991750 shutdown_connections 2023-10-30T14:24:11.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.765+0000 7f4f17501640 1 --2- 172.21.15.57:0/2441991750 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f100c42a0 0x7f4f100ca1e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.765+0000 7f4f17501640 1 --2- 172.21.15.57:0/2441991750 >> v2:172.21.15.88:3300/0 conn(0x7f4f100c4a10 0x7f4f1015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.765+0000 7f4f17501640 1 -- 172.21.15.57:0/2441991750 >> 172.21.15.57:0/2441991750 conn(0x7f4f100715c0 msgr2=0x7f4f100719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:11.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.765+0000 7f4f17501640 1 -- 172.21.15.57:0/2441991750 shutdown_connections 2023-10-30T14:24:11.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.765+0000 7f4f17501640 1 -- 172.21.15.57:0/2441991750 wait complete. 2023-10-30T14:24:11.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.765+0000 7f4f17501640 1 Processor -- start 2023-10-30T14:24:11.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.765+0000 7f4f17501640 1 -- start start 2023-10-30T14:24:11.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.765+0000 7f4f17501640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f100c42a0 0x7f4f101f2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:11.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.765+0000 7f4f17501640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f100c4a10 0x7f4f100c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:11.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.765+0000 7f4f17501640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f10153c80 0x7f4f10156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:11.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.765+0000 7f4f17501640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f101ef830 con 0x7f4f100c4a10 2023-10-30T14:24:11.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.765+0000 7f4f17501640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f101f3030 con 0x7f4f100c42a0 2023-10-30T14:24:11.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.765+0000 7f4f17501640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1015cd80 con 0x7f4f10153c80 2023-10-30T14:24:11.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.765+0000 7f4f14a9c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f100c4a10 0x7f4f100c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:11.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.769+0000 7f4f14a9c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f100c4a10 0x7f4f100c8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55518/0 (socket says 172.21.15.57:55518) 2023-10-30T14:24:11.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.769+0000 7f4f14a9c640 1 -- 172.21.15.57:0/1170705844 learned_addr learned my addr 172.21.15.57:0/1170705844 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:24:11.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.769+0000 7f4f15a9e640 1 --2- 172.21.15.57:0/1170705844 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f10153c80 0x7f4f10156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:11.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.769+0000 7f4f1529d640 1 --2- 172.21.15.57:0/1170705844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f100c42a0 0x7f4f101f2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:11.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.769+0000 7f4f14a9c640 1 -- 172.21.15.57:0/1170705844 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f10153c80 msgr2=0x7f4f10156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:11.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.769+0000 7f4f14a9c640 1 --2- 172.21.15.57:0/1170705844 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f10153c80 0x7f4f10156b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.769+0000 7f4f14a9c640 1 -- 172.21.15.57:0/1170705844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f100c42a0 msgr2=0x7f4f101f2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:11.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.769+0000 7f4f14a9c640 1 --2- 172.21.15.57:0/1170705844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f100c42a0 0x7f4f101f2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.769+0000 7f4f14a9c640 1 -- 172.21.15.57:0/1170705844 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f08009b30 con 0x7f4f100c4a10 2023-10-30T14:24:11.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.769+0000 7f4f14a9c640 1 --2- 172.21.15.57:0/1170705844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f100c4a10 0x7f4f100c8ee0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f4f0000e900 tx=0x7f4f0000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:11.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.769+0000 7f4f05ffb640 1 -- 172.21.15.57:0/1170705844 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f00009690 con 0x7f4f100c4a10 2023-10-30T14:24:11.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.769+0000 7f4f05ffb640 1 -- 172.21.15.57:0/1170705844 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0000ca00 con 0x7f4f100c4a10 2023-10-30T14:24:11.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.769+0000 7f4f05ffb640 1 -- 172.21.15.57:0/1170705844 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0000cdb0 con 0x7f4f100c4a10 2023-10-30T14:24:11.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.769+0000 7f4f17501640 1 -- 172.21.15.57:0/1170705844 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f101eee90 con 0x7f4f100c4a10 2023-10-30T14:24:11.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.769+0000 7f4f17501640 1 -- 172.21.15.57:0/1170705844 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f101ef450 con 0x7f4f100c4a10 2023-10-30T14:24:11.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.769+0000 7f4f17501640 1 -- 172.21.15.57:0/1170705844 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1004e170 con 0x7f4f100c4a10 2023-10-30T14:24:11.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.769+0000 7f4f05ffb640 1 -- 172.21.15.57:0/1170705844 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4f00013470 con 0x7f4f100c4a10 2023-10-30T14:24:11.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.773+0000 7f4f05ffb640 1 --2- 172.21.15.57:0/1170705844 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4ef40664a0 0x7f4ef4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:11.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.773+0000 7f4f05ffb640 1 -- 172.21.15.57:0/1170705844 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f4f000842a0 con 0x7f4f100c4a10 2023-10-30T14:24:11.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.773+0000 7f4f1529d640 1 --2- 172.21.15.57:0/1170705844 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4ef40664a0 0x7f4ef4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:11.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.773+0000 7f4f1529d640 1 --2- 172.21.15.57:0/1170705844 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4ef40664a0 0x7f4ef4068960 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c007fd0 tx=0x7f4f0c007d60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:11.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.781+0000 7f4f05ffb640 1 -- 172.21.15.57:0/1170705844 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4f00051550 con 0x7f4f100c4a10 2023-10-30T14:24:11.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.969+0000 7f4f17501640 1 -- 172.21.15.57:0/1170705844 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f100cad40 con 0x7f4f100c4a10 2023-10-30T14:24:11.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.969+0000 7f4f05ffb640 1 -- 172.21.15.57:0/1170705844 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f4f00054b70 con 0x7f4f100c4a10 2023-10-30T14:24:11.973 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:24:11.973 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:24:10.921507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:rejoin","state_seq":49,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:11.974 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 70 2023-10-30T14:24:11.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.973+0000 7f4f17501640 1 -- 172.21.15.57:0/1170705844 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4ef40664a0 msgr2=0x7f4ef4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:11.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.973+0000 7f4f17501640 1 --2- 172.21.15.57:0/1170705844 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4ef40664a0 0x7f4ef4068960 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c007fd0 tx=0x7f4f0c007d60 comp rx=0 tx=0).stop 2023-10-30T14:24:11.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.973+0000 7f4f17501640 1 -- 172.21.15.57:0/1170705844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f100c4a10 msgr2=0x7f4f100c8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:11.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.973+0000 7f4f17501640 1 --2- 172.21.15.57:0/1170705844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f100c4a10 0x7f4f100c8ee0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f4f0000e900 tx=0x7f4f0000edc0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.973+0000 7f4f17501640 1 -- 172.21.15.57:0/1170705844 shutdown_connections 2023-10-30T14:24:11.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.973+0000 7f4f17501640 1 --2- 172.21.15.57:0/1170705844 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4ef40664a0 0x7f4ef4068960 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.973+0000 7f4f17501640 1 --2- 172.21.15.57:0/1170705844 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f10153c80 0x7f4f10156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.973+0000 7f4f17501640 1 --2- 172.21.15.57:0/1170705844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f100c4a10 0x7f4f100c8ee0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.973+0000 7f4f17501640 1 --2- 172.21.15.57:0/1170705844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f100c42a0 0x7f4f101f2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.973+0000 7f4f17501640 1 -- 172.21.15.57:0/1170705844 >> 172.21.15.57:0/1170705844 conn(0x7f4f100715c0 msgr2=0x7f4f100c5870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:11.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.973+0000 7f4f17501640 1 -- 172.21.15.57:0/1170705844 shutdown_connections 2023-10-30T14:24:11.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:11.973+0000 7f4f17501640 1 -- 172.21.15.57:0/1170705844 wait complete. 2023-10-30T14:24:11.997 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:24:13.997 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:14.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.145+0000 7f83efb65640 1 Processor -- start 2023-10-30T14:24:14.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.145+0000 7f83efb65640 1 -- start start 2023-10-30T14:24:14.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.149+0000 7f83efb65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83e8153c80 0x7f83e815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:14.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.149+0000 7f83efb65640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f83e80c4a10 0x7f83e80c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:14.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.149+0000 7f83efb65640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e8159f20 con 0x7f83e815a3b0 2023-10-30T14:24:14.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.149+0000 7f83efb65640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e815dec0 con 0x7f83e80c4a10 2023-10-30T14:24:14.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.149+0000 7f83efb65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e81599a0 con 0x7f83e8153c80 2023-10-30T14:24:14.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.149+0000 7f83ed100640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f83e80c4a10 0x7f83e80c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:14.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.149+0000 7f83ed901640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83e8153c80 0x7f83e815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:14.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.149+0000 7f83ed100640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f83e80c4a10 0x7f83e80c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44734/0 (socket says 172.21.15.57:44734) 2023-10-30T14:24:14.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.149+0000 7f83ed100640 1 -- 172.21.15.57:0/552632067 learned_addr learned my addr 172.21.15.57:0/552632067 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:24:14.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.149+0000 7f83ed100640 1 -- 172.21.15.57:0/552632067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83e8153c80 msgr2=0x7f83e815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:14.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.149+0000 7f83ed100640 1 --2- 172.21.15.57:0/552632067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83e8153c80 0x7f83e815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:14.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.149+0000 7f83ed100640 1 -- 172.21.15.57:0/552632067 >> v1:172.21.15.88:6789/0 conn(0x7f83e815a3b0 legacy=0x7f83e80ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:14.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.149+0000 7f83ed100640 1 -- 172.21.15.57:0/552632067 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e80c47e0 con 0x7f83e80c4a10 2023-10-30T14:24:14.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.153+0000 7f83ed100640 1 --2- 172.21.15.57:0/552632067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83e80c4a10 0x7f83e80c42a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f83d8009870 tx=0x7f83d801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc3956fd00441ce server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:14.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.153+0000 7f83ec8ff640 1 -- 172.21.15.57:0/552632067 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d80a8a30 con 0x7f83e80c4a10 2023-10-30T14:24:14.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.153+0000 7f83ec8ff640 1 -- 172.21.15.57:0/552632067 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d80a8b90 con 0x7f83e80c4a10 2023-10-30T14:24:14.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.153+0000 7f83ec8ff640 1 -- 172.21.15.57:0/552632067 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d80b0440 con 0x7f83e80c4a10 2023-10-30T14:24:14.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.153+0000 7f83efb65640 1 -- 172.21.15.57:0/552632067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83e80c4a10 msgr2=0x7f83e80c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:14.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.153+0000 7f83efb65640 1 --2- 172.21.15.57:0/552632067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83e80c4a10 0x7f83e80c42a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f83d8009870 tx=0x7f83d801ed30 comp rx=0 tx=0).stop 2023-10-30T14:24:14.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.153+0000 7f83efb65640 1 -- 172.21.15.57:0/552632067 shutdown_connections 2023-10-30T14:24:14.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.153+0000 7f83efb65640 1 --2- 172.21.15.57:0/552632067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83e80c4a10 0x7f83e80c42a0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:14.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.153+0000 7f83efb65640 1 --2- 172.21.15.57:0/552632067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83e8153c80 0x7f83e815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:14.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.153+0000 7f83efb65640 1 -- 172.21.15.57:0/552632067 >> 172.21.15.57:0/552632067 conn(0x7f83e80715c0 msgr2=0x7f83e80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:14.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.153+0000 7f83efb65640 1 -- 172.21.15.57:0/552632067 shutdown_connections 2023-10-30T14:24:14.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.153+0000 7f83efb65640 1 -- 172.21.15.57:0/552632067 wait complete. 2023-10-30T14:24:14.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.153+0000 7f83efb65640 1 Processor -- start 2023-10-30T14:24:14.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83efb65640 1 -- start start 2023-10-30T14:24:14.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83efb65640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83e80c4a10 0x7f83e80b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:14.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83efb65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83e8153c80 0x7f83e80ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:14.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83efb65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83e815a3b0 0x7f83e80b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:14.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83efb65640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e81ef230 con 0x7f83e80c4a10 2023-10-30T14:24:14.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83efb65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e80b28d0 con 0x7f83e8153c80 2023-10-30T14:24:14.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83efb65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e81ef4f0 con 0x7f83e815a3b0 2023-10-30T14:24:14.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83ed901640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83e80c4a10 0x7f83e80b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:14.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83ed901640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83e80c4a10 0x7f83e80b2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44750/0 (socket says 172.21.15.57:44750) 2023-10-30T14:24:14.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83ed901640 1 -- 172.21.15.57:0/564392158 learned_addr learned my addr 172.21.15.57:0/564392158 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:24:14.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83ed100640 1 --2- 172.21.15.57:0/564392158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83e8153c80 0x7f83e80ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:14.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83ee102640 1 --2- 172.21.15.57:0/564392158 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83e815a3b0 0x7f83e80b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:14.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83ed901640 1 -- 172.21.15.57:0/564392158 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83e815a3b0 msgr2=0x7f83e80b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:14.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83ed901640 1 --2- 172.21.15.57:0/564392158 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83e815a3b0 0x7f83e80b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:14.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83ed901640 1 -- 172.21.15.57:0/564392158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83e8153c80 msgr2=0x7f83e80ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:14.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83ed901640 1 --2- 172.21.15.57:0/564392158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83e8153c80 0x7f83e80ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:14.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83ed901640 1 -- 172.21.15.57:0/564392158 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d80094d0 con 0x7f83e80c4a10 2023-10-30T14:24:14.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83ed901640 1 --2- 172.21.15.57:0/564392158 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83e80c4a10 0x7f83e80b2020 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f83e400e930 tx=0x7f83e400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:14.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83de7fc640 1 -- 172.21.15.57:0/564392158 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e4009690 con 0x7f83e80c4a10 2023-10-30T14:24:14.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83de7fc640 1 -- 172.21.15.57:0/564392158 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e400ca00 con 0x7f83e80c4a10 2023-10-30T14:24:14.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83de7fc640 1 -- 172.21.15.57:0/564392158 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e400cd70 con 0x7f83e80c4a10 2023-10-30T14:24:14.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83efb65640 1 -- 172.21.15.57:0/564392158 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e81ef7d0 con 0x7f83e80c4a10 2023-10-30T14:24:14.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83efb65640 1 -- 172.21.15.57:0/564392158 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e81efd90 con 0x7f83e80c4a10 2023-10-30T14:24:14.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.157+0000 7f83efb65640 1 -- 172.21.15.57:0/564392158 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e81eff80 con 0x7f83e80c4a10 2023-10-30T14:24:14.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.161+0000 7f83de7fc640 1 -- 172.21.15.57:0/564392158 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f83e4013470 con 0x7f83e80c4a10 2023-10-30T14:24:14.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.161+0000 7f83de7fc640 1 --2- 172.21.15.57:0/564392158 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f83cc0664d0 0x7f83cc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:14.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.161+0000 7f83de7fc640 1 -- 172.21.15.57:0/564392158 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f83e4084310 con 0x7f83e80c4a10 2023-10-30T14:24:14.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.161+0000 7f83ed100640 1 --2- 172.21.15.57:0/564392158 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f83cc0664d0 0x7f83cc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:14.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.161+0000 7f83ed100640 1 --2- 172.21.15.57:0/564392158 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f83cc0664d0 0x7f83cc068990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f83d80b2000 tx=0x7f83d8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:14.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.165+0000 7f83de7fc640 1 -- 172.21.15.57:0/564392158 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f83e4051670 con 0x7f83e80c4a10 2023-10-30T14:24:14.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.357+0000 7f83efb65640 1 -- 172.21.15.57:0/564392158 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83e804e170 con 0x7f83e80c4a10 2023-10-30T14:24:14.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.357+0000 7f83de7fc640 1 -- 172.21.15.57:0/564392158 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f83e4054c90 con 0x7f83e80c4a10 2023-10-30T14:24:14.359 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:24:14.359 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:24:10.921507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:rejoin","state_seq":49,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:14.359 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 70 2023-10-30T14:24:14.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.357+0000 7f83efb65640 1 -- 172.21.15.57:0/564392158 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f83cc0664d0 msgr2=0x7f83cc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:14.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.357+0000 7f83efb65640 1 --2- 172.21.15.57:0/564392158 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f83cc0664d0 0x7f83cc068990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f83d80b2000 tx=0x7f83d8009040 comp rx=0 tx=0).stop 2023-10-30T14:24:14.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.357+0000 7f83efb65640 1 -- 172.21.15.57:0/564392158 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83e80c4a10 msgr2=0x7f83e80b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:14.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.357+0000 7f83efb65640 1 --2- 172.21.15.57:0/564392158 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83e80c4a10 0x7f83e80b2020 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f83e400e930 tx=0x7f83e400edf0 comp rx=0 tx=0).stop 2023-10-30T14:24:14.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.357+0000 7f83efb65640 1 -- 172.21.15.57:0/564392158 shutdown_connections 2023-10-30T14:24:14.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.357+0000 7f83efb65640 1 --2- 172.21.15.57:0/564392158 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f83cc0664d0 0x7f83cc068990 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:14.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.357+0000 7f83efb65640 1 --2- 172.21.15.57:0/564392158 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83e815a3b0 0x7f83e80b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:14.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.357+0000 7f83efb65640 1 --2- 172.21.15.57:0/564392158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83e8153c80 0x7f83e80ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:14.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.357+0000 7f83efb65640 1 --2- 172.21.15.57:0/564392158 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83e80c4a10 0x7f83e80b2020 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:14.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.357+0000 7f83efb65640 1 -- 172.21.15.57:0/564392158 >> 172.21.15.57:0/564392158 conn(0x7f83e80715c0 msgr2=0x7f83e8157970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:14.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.357+0000 7f83efb65640 1 -- 172.21.15.57:0/564392158 shutdown_connections 2023-10-30T14:24:14.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:14.357+0000 7f83efb65640 1 -- 172.21.15.57:0/564392158 wait complete. 2023-10-30T14:24:14.381 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:24:16.382 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:16.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.537+0000 7f85e45c4640 1 Processor -- start 2023-10-30T14:24:16.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.537+0000 7f85e45c4640 1 -- start start 2023-10-30T14:24:16.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.541+0000 7f85e45c4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f85dc153c80 0x7f85dc15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:16.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.541+0000 7f85e45c4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85dc159f20 con 0x7f85dc0c4a10 2023-10-30T14:24:16.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.541+0000 7f85e45c4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85dc15d620 con 0x7f85dc0c42a0 2023-10-30T14:24:16.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.541+0000 7f85e45c4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85dc15dec0 con 0x7f85dc153c80 2023-10-30T14:24:16.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.541+0000 7f85e1b5f640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f85dc0c4a10 0x7f85dc15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:51198/0 (socket says 172.21.15.57:51198) 2023-10-30T14:24:16.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.541+0000 7f85e1b5f640 1 -- 172.21.15.57:0/1992692648 learned_addr learned my addr 172.21.15.57:0/1992692648 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:24:16.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.541+0000 7f85e2360640 1 --2- 172.21.15.57:0/1992692648 >> v2:172.21.15.88:3300/0 conn(0x7f85dc153c80 0x7f85dc15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:16.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.541+0000 7f85e135e640 1 -- 172.21.15.57:0/1992692648 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3275764941 0 0) 0x7f85dc159f20 con 0x7f85dc0c4a10 2023-10-30T14:24:16.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.541+0000 7f85e135e640 1 -- 172.21.15.57:0/1992692648 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85c8003530 con 0x7f85dc0c4a10 2023-10-30T14:24:16.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.541+0000 7f85e2360640 1 -- 172.21.15.57:0/1992692648 >> v1:172.21.15.88:6789/0 conn(0x7f85dc0c42a0 legacy=0x7f85dc0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:16.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.541+0000 7f85e2360640 1 -- 172.21.15.57:0/1992692648 >> v1:172.21.15.57:6789/0 conn(0x7f85dc0c4a10 legacy=0x7f85dc15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:16.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.541+0000 7f85e2360640 1 -- 172.21.15.57:0/1992692648 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc15d900 con 0x7f85dc153c80 2023-10-30T14:24:16.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.545+0000 7f85e2360640 1 --2- 172.21.15.57:0/1992692648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85dc153c80 0x7f85dc15cf90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f85d80097c0 tx=0x7f85d801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a7aa6d86ad560e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:16.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.549+0000 7f85e135e640 1 -- 172.21.15.57:0/1992692648 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d80a8970 con 0x7f85dc153c80 2023-10-30T14:24:16.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.549+0000 7f85e135e640 1 -- 172.21.15.57:0/1992692648 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d80a8ad0 con 0x7f85dc153c80 2023-10-30T14:24:16.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.549+0000 7f85e135e640 1 -- 172.21.15.57:0/1992692648 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d80a8e60 con 0x7f85dc153c80 2023-10-30T14:24:16.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.549+0000 7f85e45c4640 1 -- 172.21.15.57:0/1992692648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85dc153c80 msgr2=0x7f85dc15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:16.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.549+0000 7f85e45c4640 1 --2- 172.21.15.57:0/1992692648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85dc153c80 0x7f85dc15cf90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f85d80097c0 tx=0x7f85d801ec70 comp rx=0 tx=0).stop 2023-10-30T14:24:16.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.549+0000 7f85e45c4640 1 -- 172.21.15.57:0/1992692648 shutdown_connections 2023-10-30T14:24:16.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.549+0000 7f85e45c4640 1 --2- 172.21.15.57:0/1992692648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85dc153c80 0x7f85dc15cf90 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:16.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.549+0000 7f85e45c4640 1 -- 172.21.15.57:0/1992692648 >> 172.21.15.57:0/1992692648 conn(0x7f85dc0715c0 msgr2=0x7f85dc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:16.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.549+0000 7f85e45c4640 1 -- 172.21.15.57:0/1992692648 shutdown_connections 2023-10-30T14:24:16.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.549+0000 7f85e45c4640 1 -- 172.21.15.57:0/1992692648 wait complete. 2023-10-30T14:24:16.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.549+0000 7f85e45c4640 1 Processor -- start 2023-10-30T14:24:16.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.549+0000 7f85e45c4640 1 -- start start 2023-10-30T14:24:16.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.549+0000 7f85e45c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85dc0c42a0 0x7f85dc1528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:16.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.549+0000 7f85e45c4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f85dc0c4a10 0x7f85dc158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:16.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.549+0000 7f85e45c4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85dc153c80 0x7f85dc0c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:16.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.549+0000 7f85e45c4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc14ad30 con 0x7f85dc153c80 2023-10-30T14:24:16.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.549+0000 7f85e45c4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc158910 con 0x7f85dc0c42a0 2023-10-30T14:24:16.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.549+0000 7f85e45c4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc159950 con 0x7f85dc0c4a10 2023-10-30T14:24:16.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.553+0000 7f85e1b5f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f85dc0c4a10 0x7f85dc158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:16.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.553+0000 7f85e2b61640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85dc153c80 0x7f85dc0c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:16.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.553+0000 7f85e2360640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85dc0c42a0 0x7f85dc1528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:16.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.553+0000 7f85e1b5f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f85dc0c4a10 0x7f85dc158af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:48096/0 (socket says 172.21.15.57:48096) 2023-10-30T14:24:16.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.553+0000 7f85e1b5f640 1 -- 172.21.15.57:0/2133812998 learned_addr learned my addr 172.21.15.57:0/2133812998 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:24:16.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.553+0000 7f85e2b61640 1 -- 172.21.15.57:0/2133812998 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f85dc0c4a10 msgr2=0x7f85dc158af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:16.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.553+0000 7f85e2b61640 1 --2- 172.21.15.57:0/2133812998 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f85dc0c4a10 0x7f85dc158af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:16.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.553+0000 7f85e2b61640 1 -- 172.21.15.57:0/2133812998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85dc0c42a0 msgr2=0x7f85dc1528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:16.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.553+0000 7f85e2b61640 1 --2- 172.21.15.57:0/2133812998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85dc0c42a0 0x7f85dc1528f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:16.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.553+0000 7f85e2b61640 1 -- 172.21.15.57:0/2133812998 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d8009420 con 0x7f85dc153c80 2023-10-30T14:24:16.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.553+0000 7f85e2b61640 1 --2- 172.21.15.57:0/2133812998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85dc153c80 0x7f85dc0c88d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f85d400c700 tx=0x7f85d400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:16.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.553+0000 7f85d2ffd640 1 -- 172.21.15.57:0/2133812998 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d4003b80 con 0x7f85dc153c80 2023-10-30T14:24:16.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.553+0000 7f85d2ffd640 1 -- 172.21.15.57:0/2133812998 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d4003ce0 con 0x7f85dc153c80 2023-10-30T14:24:16.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.553+0000 7f85e45c4640 1 -- 172.21.15.57:0/2133812998 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85dc1593d0 con 0x7f85dc153c80 2023-10-30T14:24:16.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.553+0000 7f85d2ffd640 1 -- 172.21.15.57:0/2133812998 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d400aa30 con 0x7f85dc153c80 2023-10-30T14:24:16.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.553+0000 7f85e45c4640 1 -- 172.21.15.57:0/2133812998 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85dc158350 con 0x7f85dc153c80 2023-10-30T14:24:16.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.553+0000 7f85d2ffd640 1 -- 172.21.15.57:0/2133812998 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f85d400ab90 con 0x7f85dc153c80 2023-10-30T14:24:16.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.553+0000 7f85e45c4640 1 -- 172.21.15.57:0/2133812998 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85dc14bc60 con 0x7f85dc153c80 2023-10-30T14:24:16.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.557+0000 7f85d2ffd640 1 --2- 172.21.15.57:0/2133812998 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85c00664a0 0x7f85c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:16.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.557+0000 7f85d2ffd640 1 -- 172.21.15.57:0/2133812998 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f85d40834e0 con 0x7f85dc153c80 2023-10-30T14:24:16.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.557+0000 7f85e2360640 1 --2- 172.21.15.57:0/2133812998 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85c00664a0 0x7f85c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:16.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.557+0000 7f85e2360640 1 --2- 172.21.15.57:0/2133812998 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85c00664a0 0x7f85c0068960 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f85d80b2000 tx=0x7f85d8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:16.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.565+0000 7f85d2ffd640 1 -- 172.21.15.57:0/2133812998 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f85dc14bc60 con 0x7f85dc153c80 2023-10-30T14:24:16.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.785+0000 7f85e45c4640 1 -- 172.21.15.57:0/2133812998 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85dc14bc60 con 0x7f85dc153c80 2023-10-30T14:24:16.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.785+0000 7f85d2ffd640 1 -- 172.21.15.57:0/2133812998 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f85dc14bc60 con 0x7f85dc153c80 2023-10-30T14:24:16.787 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:24:16.787 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:24:15.976636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:16.788 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 71 2023-10-30T14:24:16.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.789+0000 7f85e45c4640 1 -- 172.21.15.57:0/2133812998 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85c00664a0 msgr2=0x7f85c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:16.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.789+0000 7f85e45c4640 1 --2- 172.21.15.57:0/2133812998 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85c00664a0 0x7f85c0068960 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f85d80b2000 tx=0x7f85d8008040 comp rx=0 tx=0).stop 2023-10-30T14:24:16.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.789+0000 7f85e45c4640 1 -- 172.21.15.57:0/2133812998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85dc153c80 msgr2=0x7f85dc0c88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:16.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.789+0000 7f85e45c4640 1 --2- 172.21.15.57:0/2133812998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85dc153c80 0x7f85dc0c88d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f85d400c700 tx=0x7f85d400cbc0 comp rx=0 tx=0).stop 2023-10-30T14:24:16.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.789+0000 7f85e45c4640 1 -- 172.21.15.57:0/2133812998 shutdown_connections 2023-10-30T14:24:16.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.789+0000 7f85e45c4640 1 --2- 172.21.15.57:0/2133812998 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85c00664a0 0x7f85c0068960 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:16.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.789+0000 7f85e45c4640 1 --2- 172.21.15.57:0/2133812998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85dc153c80 0x7f85dc0c88d0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:16.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.789+0000 7f85e45c4640 1 --2- 172.21.15.57:0/2133812998 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f85dc0c4a10 0x7f85dc158af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:16.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.789+0000 7f85e45c4640 1 --2- 172.21.15.57:0/2133812998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85dc0c42a0 0x7f85dc1528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:16.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.789+0000 7f85e45c4640 1 -- 172.21.15.57:0/2133812998 >> 172.21.15.57:0/2133812998 conn(0x7f85dc0715c0 msgr2=0x7f85dc0c5800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:16.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.789+0000 7f85e45c4640 1 -- 172.21.15.57:0/2133812998 shutdown_connections 2023-10-30T14:24:16.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:16.789+0000 7f85e45c4640 1 -- 172.21.15.57:0/2133812998 wait complete. 2023-10-30T14:24:16.818 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=1, replacing gid=5133 2023-10-30T14:24:16.818 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.a 2023-10-30T14:24:16.818 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T14:24:16.818 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T14:24:16.819 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T14:24:16.824 INFO:tasks.ceph.mds.a:Started 2023-10-30T14:24:16.824 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:16.937 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-30T14:24:17.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.049+0000 7fe533e17640 1 Processor -- start 2023-10-30T14:24:17.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.049+0000 7fe533e17640 1 -- start start 2023-10-30T14:24:17.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.049+0000 7fe533e17640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe52c15cf90 0x7fe52c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:17.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.049+0000 7fe533e17640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52c159f20 con 0x7fe52c0c4a10 2023-10-30T14:24:17.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.049+0000 7fe533e17640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52c0c47e0 con 0x7fe52c153c80 2023-10-30T14:24:17.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.049+0000 7fe533e17640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c15dec0 con 0x7fe52c15cf90 2023-10-30T14:24:17.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.049+0000 7fe5313b2640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fe52c0c4a10 0x7fe52c0ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:51212/0 (socket says 172.21.15.57:51212) 2023-10-30T14:24:17.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.049+0000 7fe5313b2640 1 -- 172.21.15.57:0/277169120 learned_addr learned my addr 172.21.15.57:0/277169120 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:24:17.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.049+0000 7fe5323b4640 1 --2- 172.21.15.57:0/277169120 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe52c15cf90 0x7fe52c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:17.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.053+0000 7fe530bb1640 1 -- 172.21.15.57:0/277169120 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916533613 0 0) 0x7fe52c159f20 con 0x7fe52c0c4a10 2023-10-30T14:24:17.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.053+0000 7fe5323b4640 1 -- 172.21.15.57:0/277169120 >> v1:172.21.15.88:6789/0 conn(0x7fe52c153c80 legacy=0x7fe52c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:17.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.053+0000 7fe5323b4640 1 -- 172.21.15.57:0/277169120 >> v1:172.21.15.57:6789/0 conn(0x7fe52c0c4a10 legacy=0x7fe52c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:17.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.053+0000 7fe5323b4640 1 -- 172.21.15.57:0/277169120 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52c15d620 con 0x7fe52c15cf90 2023-10-30T14:24:17.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.057+0000 7fe5323b4640 1 --2- 172.21.15.57:0/277169120 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe52c15cf90 0x7fe52c0c42a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe524009dc0 tx=0x7fe52401c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=347c2284b18cc295 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:17.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.057+0000 7fe530bb1640 1 -- 172.21.15.57:0/277169120 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5240a5910 con 0x7fe52c15cf90 2023-10-30T14:24:17.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.057+0000 7fe530bb1640 1 -- 172.21.15.57:0/277169120 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5240a5a70 con 0x7fe52c15cf90 2023-10-30T14:24:17.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.057+0000 7fe530bb1640 1 -- 172.21.15.57:0/277169120 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5240a5e00 con 0x7fe52c15cf90 2023-10-30T14:24:17.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.057+0000 7fe533e17640 1 -- 172.21.15.57:0/277169120 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe52c15cf90 msgr2=0x7fe52c0c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:17.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.057+0000 7fe533e17640 1 --2- 172.21.15.57:0/277169120 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe52c15cf90 0x7fe52c0c42a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe524009dc0 tx=0x7fe52401c8b0 comp rx=0 tx=0).stop 2023-10-30T14:24:17.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.057+0000 7fe533e17640 1 -- 172.21.15.57:0/277169120 shutdown_connections 2023-10-30T14:24:17.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.057+0000 7fe533e17640 1 --2- 172.21.15.57:0/277169120 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe52c15cf90 0x7fe52c0c42a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:17.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.057+0000 7fe533e17640 1 -- 172.21.15.57:0/277169120 >> 172.21.15.57:0/277169120 conn(0x7fe52c0715c0 msgr2=0x7fe52c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:17.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.057+0000 7fe533e17640 1 -- 172.21.15.57:0/277169120 shutdown_connections 2023-10-30T14:24:17.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.057+0000 7fe533e17640 1 -- 172.21.15.57:0/277169120 wait complete. 2023-10-30T14:24:17.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.061+0000 7fe533e17640 1 Processor -- start 2023-10-30T14:24:17.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.061+0000 7fe533e17640 1 -- start start 2023-10-30T14:24:17.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.061+0000 7fe533e17640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe52c0c4a10 0x7fe52c1ef2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:17.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.061+0000 7fe533e17640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe52c153c80 0x7fe52c1503b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:17.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.061+0000 7fe533e17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe52c15cf90 0x7fe52c14f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:17.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.061+0000 7fe533e17640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c159250 con 0x7fe52c0c4a10 2023-10-30T14:24:17.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.061+0000 7fe533e17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c1594f0 con 0x7fe52c15cf90 2023-10-30T14:24:17.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.061+0000 7fe533e17640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c1597b0 con 0x7fe52c153c80 2023-10-30T14:24:17.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.061+0000 7fe531bb3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe52c0c4a10 0x7fe52c1ef2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:17.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.061+0000 7fe531bb3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe52c0c4a10 0x7fe52c1ef2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44780/0 (socket says 172.21.15.57:44780) 2023-10-30T14:24:17.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.061+0000 7fe531bb3640 1 -- 172.21.15.57:0/1491482879 learned_addr learned my addr 172.21.15.57:0/1491482879 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:24:17.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.061+0000 7fe5313b2640 1 --2- 172.21.15.57:0/1491482879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe52c153c80 0x7fe52c1503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:17.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.061+0000 7fe5323b4640 1 --2- 172.21.15.57:0/1491482879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe52c15cf90 0x7fe52c14f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:17.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.061+0000 7fe531bb3640 1 -- 172.21.15.57:0/1491482879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe52c153c80 msgr2=0x7fe52c1503b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:17.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.061+0000 7fe531bb3640 1 --2- 172.21.15.57:0/1491482879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe52c153c80 0x7fe52c1503b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:17.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.061+0000 7fe531bb3640 1 -- 172.21.15.57:0/1491482879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe52c15cf90 msgr2=0x7fe52c14f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:17.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.061+0000 7fe531bb3640 1 --2- 172.21.15.57:0/1491482879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe52c15cf90 0x7fe52c14f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:17.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.065+0000 7fe531bb3640 1 -- 172.21.15.57:0/1491482879 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe524009a70 con 0x7fe52c0c4a10 2023-10-30T14:24:17.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.065+0000 7fe531bb3640 1 --2- 172.21.15.57:0/1491482879 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe52c0c4a10 0x7fe52c1ef2a0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fe528009eb0 tx=0x7fe52800c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:17.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.065+0000 7fe5227fc640 1 -- 172.21.15.57:0/1491482879 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52800ce10 con 0x7fe52c0c4a10 2023-10-30T14:24:17.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.065+0000 7fe5227fc640 1 -- 172.21.15.57:0/1491482879 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe528004020 con 0x7fe52c0c4a10 2023-10-30T14:24:17.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.065+0000 7fe5227fc640 1 -- 172.21.15.57:0/1491482879 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5280043d0 con 0x7fe52c0c4a10 2023-10-30T14:24:17.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.065+0000 7fe533e17640 1 -- 172.21.15.57:0/1491482879 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52c14e800 con 0x7fe52c0c4a10 2023-10-30T14:24:17.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.065+0000 7fe533e17640 1 -- 172.21.15.57:0/1491482879 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52c158f40 con 0x7fe52c0c4a10 2023-10-30T14:24:17.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.065+0000 7fe533e17640 1 -- 172.21.15.57:0/1491482879 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe52c14e640 con 0x7fe52c0c4a10 2023-10-30T14:24:17.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.069+0000 7fe5227fc640 1 -- 172.21.15.57:0/1491482879 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe528003600 con 0x7fe52c0c4a10 2023-10-30T14:24:17.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.069+0000 7fe5227fc640 1 --2- 172.21.15.57:0/1491482879 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5100664a0 0x7fe510068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:17.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.069+0000 7fe5227fc640 1 -- 172.21.15.57:0/1491482879 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6341+0+0 (secure 0 0 0) 0x7fe5280866d0 con 0x7fe52c0c4a10 2023-10-30T14:24:17.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.069+0000 7fe5313b2640 1 --2- 172.21.15.57:0/1491482879 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5100664a0 0x7fe510068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:17.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.069+0000 7fe5313b2640 1 --2- 172.21.15.57:0/1491482879 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5100664a0 0x7fe510068960 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe51c001120 tx=0x7fe51c001090 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:17.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.073+0000 7fe5227fc640 1 -- 172.21.15.57:0/1491482879 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe528053a10 con 0x7fe52c0c4a10 2023-10-30T14:24:17.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.413+0000 7fe533e17640 1 -- 172.21.15.57:0/1491482879 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe52c14ed20 con 0x7fe52c0c4a10 2023-10-30T14:24:17.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.413+0000 7fe5227fc640 1 -- 172.21.15.57:0/1491482879 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6507 (secure 0 0 0) 0x7fe528057030 con 0x7fe52c0c4a10 2023-10-30T14:24:17.416 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:24:17.417 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":5797,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:24:15.976636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:17.417 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 72 2023-10-30T14:24:17.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.417+0000 7fe533e17640 1 -- 172.21.15.57:0/1491482879 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5100664a0 msgr2=0x7fe510068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:17.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.417+0000 7fe533e17640 1 --2- 172.21.15.57:0/1491482879 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5100664a0 0x7fe510068960 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe51c001120 tx=0x7fe51c001090 comp rx=0 tx=0).stop 2023-10-30T14:24:17.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.417+0000 7fe533e17640 1 -- 172.21.15.57:0/1491482879 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe52c0c4a10 msgr2=0x7fe52c1ef2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:17.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.417+0000 7fe533e17640 1 --2- 172.21.15.57:0/1491482879 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe52c0c4a10 0x7fe52c1ef2a0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fe528009eb0 tx=0x7fe52800c430 comp rx=0 tx=0).stop 2023-10-30T14:24:17.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.417+0000 7fe533e17640 1 -- 172.21.15.57:0/1491482879 shutdown_connections 2023-10-30T14:24:17.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.417+0000 7fe533e17640 1 --2- 172.21.15.57:0/1491482879 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5100664a0 0x7fe510068960 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:17.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.417+0000 7fe533e17640 1 --2- 172.21.15.57:0/1491482879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe52c15cf90 0x7fe52c14f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:17.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.417+0000 7fe533e17640 1 --2- 172.21.15.57:0/1491482879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe52c153c80 0x7fe52c1503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:17.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.417+0000 7fe533e17640 1 --2- 172.21.15.57:0/1491482879 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe52c0c4a10 0x7fe52c1ef2a0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:17.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.417+0000 7fe533e17640 1 -- 172.21.15.57:0/1491482879 >> 172.21.15.57:0/1491482879 conn(0x7fe52c0715c0 msgr2=0x7fe52c0c58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:17.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.417+0000 7fe533e17640 1 -- 172.21.15.57:0/1491482879 shutdown_connections 2023-10-30T14:24:17.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:24:17.417+0000 7fe533e17640 1 -- 172.21.15.57:0/1491482879 wait complete. 2023-10-30T14:24:17.440 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T14:24:17.440 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 114 secs before thrashing 2023-10-30T14:24:31.402 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:24:31.408 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:25:01.522 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:25:01.527 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:25:31.662 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:25:31.666 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:26:01.795 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:26:01.800 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:26:11.441 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:11.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.584+0000 7fd767017640 1 Processor -- start 2023-10-30T14:26:11.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.584+0000 7fd767017640 1 -- start start 2023-10-30T14:26:11.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.584+0000 7fd767017640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd7600cb7d0 0x7fd7600c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:11.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.584+0000 7fd767017640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd7600c3e40 0x7fd760070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:11.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.584+0000 7fd767017640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7600c2310 con 0x7fd760150830 2023-10-30T14:26:11.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.584+0000 7fd767017640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd76006bc10 con 0x7fd7600cb7d0 2023-10-30T14:26:11.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.584+0000 7fd767017640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd760156710 con 0x7fd7600c3e40 2023-10-30T14:26:11.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.584+0000 7fd75ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd7600cb7d0 0x7fd7600c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:11.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.584+0000 7fd764db3640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fd760150830 0x7fd76006b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:55072/0 (socket says 172.21.15.57:55072) 2023-10-30T14:26:11.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.584+0000 7fd764db3640 1 -- 172.21.15.57:0/2365453042 learned_addr learned my addr 172.21.15.57:0/2365453042 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:26:11.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.584+0000 7fd7655b4640 1 --2- 172.21.15.57:0/2365453042 >> v2:172.21.15.88:3300/0 conn(0x7fd7600c3e40 0x7fd760070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:11.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.588+0000 7fd75ffff640 1 -- 172.21.15.57:0/2365453042 >> v2:172.21.15.88:3300/0 conn(0x7fd7600c3e40 msgr2=0x7fd760070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:11.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.588+0000 7fd75ffff640 1 --2- 172.21.15.57:0/2365453042 >> v2:172.21.15.88:3300/0 conn(0x7fd7600c3e40 0x7fd760070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:11.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.588+0000 7fd75ffff640 1 -- 172.21.15.57:0/2365453042 >> v1:172.21.15.57:6789/0 conn(0x7fd760150830 legacy=0x7fd76006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:11.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.588+0000 7fd75ffff640 1 -- 172.21.15.57:0/2365453042 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7600c4360 con 0x7fd7600cb7d0 2023-10-30T14:26:11.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.588+0000 7fd7655b4640 1 --2- 172.21.15.57:0/2365453042 >> v2:172.21.15.88:3300/0 conn(0x7fd7600c3e40 0x7fd760070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:11.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.588+0000 7fd75ffff640 1 --2- 172.21.15.57:0/2365453042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7600cb7d0 0x7fd7600c3900 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd760066e10 tx=0x7fd74c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15771741a553951a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:11.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.588+0000 7fd75f7fe640 1 -- 172.21.15.57:0/2365453042 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c0a8950 con 0x7fd7600cb7d0 2023-10-30T14:26:11.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.588+0000 7fd75f7fe640 1 -- 172.21.15.57:0/2365453042 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74c0a8ab0 con 0x7fd7600cb7d0 2023-10-30T14:26:11.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.588+0000 7fd75f7fe640 1 -- 172.21.15.57:0/2365453042 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c0a8e40 con 0x7fd7600cb7d0 2023-10-30T14:26:11.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd767017640 1 -- 172.21.15.57:0/2365453042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7600cb7d0 msgr2=0x7fd7600c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:11.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd767017640 1 --2- 172.21.15.57:0/2365453042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7600cb7d0 0x7fd7600c3900 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd760066e10 tx=0x7fd74c01ec50 comp rx=0 tx=0).stop 2023-10-30T14:26:11.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd767017640 1 -- 172.21.15.57:0/2365453042 shutdown_connections 2023-10-30T14:26:11.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd767017640 1 --2- 172.21.15.57:0/2365453042 >> v2:172.21.15.88:3300/0 conn(0x7fd7600c3e40 0x7fd760070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:11.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd767017640 1 --2- 172.21.15.57:0/2365453042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7600cb7d0 0x7fd7600c3900 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:11.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd767017640 1 -- 172.21.15.57:0/2365453042 >> 172.21.15.57:0/2365453042 conn(0x7fd7600cac70 msgr2=0x7fd7600cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:11.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd767017640 1 -- 172.21.15.57:0/2365453042 shutdown_connections 2023-10-30T14:26:11.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd767017640 1 -- 172.21.15.57:0/2365453042 wait complete. 2023-10-30T14:26:11.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd767017640 1 Processor -- start 2023-10-30T14:26:11.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd767017640 1 -- start start 2023-10-30T14:26:11.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd767017640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7600c3e40 0x7fd760154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:11.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd767017640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7600cb7d0 0x7fd7600c5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:11.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd767017640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd760150830 0x7fd7601478d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:11.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd767017640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7601550c0 con 0x7fd760150830 2023-10-30T14:26:11.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd767017640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd760155460 con 0x7fd7600c3e40 2023-10-30T14:26:11.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd767017640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd760155720 con 0x7fd7600cb7d0 2023-10-30T14:26:11.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd7655b4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd760150830 0x7fd7601478d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:11.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd764db3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7600c3e40 0x7fd760154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:11.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd7655b4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd760150830 0x7fd7601478d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40710/0 (socket says 172.21.15.57:40710) 2023-10-30T14:26:11.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd7655b4640 1 -- 172.21.15.57:0/867582392 learned_addr learned my addr 172.21.15.57:0/867582392 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:11.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.592+0000 7fd75ffff640 1 --2- 172.21.15.57:0/867582392 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7600cb7d0 0x7fd7600c5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:11.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.596+0000 7fd7655b4640 1 -- 172.21.15.57:0/867582392 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7600cb7d0 msgr2=0x7fd7600c5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:11.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.596+0000 7fd7655b4640 1 --2- 172.21.15.57:0/867582392 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7600cb7d0 0x7fd7600c5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:11.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.596+0000 7fd7655b4640 1 -- 172.21.15.57:0/867582392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7600c3e40 msgr2=0x7fd760154b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:11.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.596+0000 7fd7655b4640 1 --2- 172.21.15.57:0/867582392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7600c3e40 0x7fd760154b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:11.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.596+0000 7fd7655b4640 1 -- 172.21.15.57:0/867582392 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74c009420 con 0x7fd760150830 2023-10-30T14:26:11.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.596+0000 7fd7655b4640 1 --2- 172.21.15.57:0/867582392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd760150830 0x7fd7601478d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fd758007dc0 tx=0x7fd75800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:11.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.596+0000 7fd75d7fa640 1 -- 172.21.15.57:0/867582392 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7580039a0 con 0x7fd760150830 2023-10-30T14:26:11.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.596+0000 7fd75d7fa640 1 -- 172.21.15.57:0/867582392 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75800f040 con 0x7fd760150830 2023-10-30T14:26:11.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.596+0000 7fd75d7fa640 1 -- 172.21.15.57:0/867582392 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd758003d40 con 0x7fd760150830 2023-10-30T14:26:11.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.596+0000 7fd767017640 1 -- 172.21.15.57:0/867582392 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76014ad30 con 0x7fd760150830 2023-10-30T14:26:11.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.596+0000 7fd767017640 1 -- 172.21.15.57:0/867582392 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd760155fa0 con 0x7fd760150830 2023-10-30T14:26:11.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.596+0000 7fd767017640 1 -- 172.21.15.57:0/867582392 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd76004e170 con 0x7fd760150830 2023-10-30T14:26:11.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.596+0000 7fd75d7fa640 1 -- 172.21.15.57:0/867582392 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd758004030 con 0x7fd760150830 2023-10-30T14:26:11.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.596+0000 7fd75d7fa640 1 --2- 172.21.15.57:0/867582392 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd7440664a0 0x7fd744068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:11.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.596+0000 7fd75d7fa640 1 -- 172.21.15.57:0/867582392 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6341+0+0 (secure 0 0 0) 0x7fd758084330 con 0x7fd760150830 2023-10-30T14:26:11.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.600+0000 7fd764db3640 1 --2- 172.21.15.57:0/867582392 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd7440664a0 0x7fd744068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:11.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.600+0000 7fd764db3640 1 --2- 172.21.15.57:0/867582392 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd7440664a0 0x7fd744068960 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd754000f70 tx=0x7fd754009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:11.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.600+0000 7fd75d7fa640 1 -- 172.21.15.57:0/867582392 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd7580515d0 con 0x7fd760150830 2023-10-30T14:26:11.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.796+0000 7fd767017640 1 -- 172.21.15.57:0/867582392 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd760156f40 con 0x7fd760150830 2023-10-30T14:26:11.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.796+0000 7fd75d7fa640 1 -- 172.21.15.57:0/867582392 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6507 (secure 0 0 0) 0x7fd758054bf0 con 0x7fd760150830 2023-10-30T14:26:11.800 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:11.800 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":5797,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:24:15.976636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:11.800 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 72 2023-10-30T14:26:11.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.800+0000 7fd767017640 1 -- 172.21.15.57:0/867582392 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd7440664a0 msgr2=0x7fd744068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:11.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.800+0000 7fd767017640 1 --2- 172.21.15.57:0/867582392 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd7440664a0 0x7fd744068960 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd754000f70 tx=0x7fd754009040 comp rx=0 tx=0).stop 2023-10-30T14:26:11.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.800+0000 7fd767017640 1 -- 172.21.15.57:0/867582392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd760150830 msgr2=0x7fd7601478d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:11.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.800+0000 7fd767017640 1 --2- 172.21.15.57:0/867582392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd760150830 0x7fd7601478d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fd758007dc0 tx=0x7fd75800d4a0 comp rx=0 tx=0).stop 2023-10-30T14:26:11.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.800+0000 7fd767017640 1 -- 172.21.15.57:0/867582392 shutdown_connections 2023-10-30T14:26:11.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.800+0000 7fd767017640 1 --2- 172.21.15.57:0/867582392 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd7440664a0 0x7fd744068960 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:11.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.800+0000 7fd767017640 1 --2- 172.21.15.57:0/867582392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd760150830 0x7fd7601478d0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:11.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.800+0000 7fd767017640 1 --2- 172.21.15.57:0/867582392 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7600cb7d0 0x7fd7600c5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:11.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.800+0000 7fd767017640 1 --2- 172.21.15.57:0/867582392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7600c3e40 0x7fd760154b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:11.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.800+0000 7fd767017640 1 -- 172.21.15.57:0/867582392 >> 172.21.15.57:0/867582392 conn(0x7fd7600cac70 msgr2=0x7fd76006b500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:11.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.800+0000 7fd767017640 1 -- 172.21.15.57:0/867582392 shutdown_connections 2023-10-30T14:26:11.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:11.800+0000 7fd767017640 1 -- 172.21.15.57:0/867582392 wait complete. 2023-10-30T14:26:11.820 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T14:26:11.821 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T14:26:11.821 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:26:11.842 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T14:26:11.842 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:12.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.024+0000 7fb4f6dc5640 1 Processor -- start 2023-10-30T14:26:12.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.024+0000 7fb4f6dc5640 1 -- start start 2023-10-30T14:26:12.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.024+0000 7fb4f6dc5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb4f0151840 0x7fb4f00c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:12.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.024+0000 7fb4f6dc5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb4f00cb8b0 0x7fb4f006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:12.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.024+0000 7fb4f6dc5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f00c5030 con 0x7fb4f0151c20 2023-10-30T14:26:12.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.024+0000 7fb4f6dc5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f0157830 con 0x7fb4f00cb8b0 2023-10-30T14:26:12.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.024+0000 7fb4f6dc5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f0157b10 con 0x7fb4f0151840 2023-10-30T14:26:12.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.028+0000 7fb4f5362640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb4f00cb8b0 0x7fb4f006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:12.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.028+0000 7fb4f5362640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb4f00cb8b0 0x7fb4f006b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40712/0 (socket says 172.21.15.57:40712) 2023-10-30T14:26:12.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.028+0000 7fb4f5362640 1 -- 172.21.15.57:0/477050910 learned_addr learned my addr 172.21.15.57:0/477050910 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:12.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.028+0000 7fb4f4b61640 1 --2- 172.21.15.57:0/477050910 >> v2:172.21.15.88:3300/0 conn(0x7fb4f0151840 0x7fb4f00c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:12.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.028+0000 7fb4f5362640 1 -- 172.21.15.57:0/477050910 >> v2:172.21.15.88:3300/0 conn(0x7fb4f0151840 msgr2=0x7fb4f00c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:12.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.028+0000 7fb4f5362640 1 --2- 172.21.15.57:0/477050910 >> v2:172.21.15.88:3300/0 conn(0x7fb4f0151840 0x7fb4f00c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:12.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.028+0000 7fb4f5362640 1 -- 172.21.15.57:0/477050910 >> v1:172.21.15.88:6789/0 conn(0x7fb4f0151c20 legacy=0x7fb4f0072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:12.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.028+0000 7fb4f5362640 1 -- 172.21.15.57:0/477050910 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f0157df0 con 0x7fb4f00cb8b0 2023-10-30T14:26:12.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.028+0000 7fb4f5362640 1 --2- 172.21.15.57:0/477050910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f00cb8b0 0x7fb4f006b600 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0072b40 tx=0x7fb4e801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dabb4e87136ab63 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:12.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.028+0000 7fb4ef7fe640 1 -- 172.21.15.57:0/477050910 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e80af400 con 0x7fb4f00cb8b0 2023-10-30T14:26:12.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.028+0000 7fb4ef7fe640 1 -- 172.21.15.57:0/477050910 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e80ab050 con 0x7fb4f00cb8b0 2023-10-30T14:26:12.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.028+0000 7fb4ef7fe640 1 -- 172.21.15.57:0/477050910 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e80af780 con 0x7fb4f00cb8b0 2023-10-30T14:26:12.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4f6dc5640 1 -- 172.21.15.57:0/477050910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f00cb8b0 msgr2=0x7fb4f006b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:12.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4f6dc5640 1 --2- 172.21.15.57:0/477050910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f00cb8b0 0x7fb4f006b600 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0072b40 tx=0x7fb4e801f1c0 comp rx=0 tx=0).stop 2023-10-30T14:26:12.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4f6dc5640 1 -- 172.21.15.57:0/477050910 shutdown_connections 2023-10-30T14:26:12.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4f6dc5640 1 --2- 172.21.15.57:0/477050910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f00cb8b0 0x7fb4f006b600 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:12.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4f6dc5640 1 --2- 172.21.15.57:0/477050910 >> v2:172.21.15.88:3300/0 conn(0x7fb4f0151840 0x7fb4f00c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:12.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4f6dc5640 1 -- 172.21.15.57:0/477050910 >> 172.21.15.57:0/477050910 conn(0x7fb4f00cac70 msgr2=0x7fb4f00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:12.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4f6dc5640 1 -- 172.21.15.57:0/477050910 shutdown_connections 2023-10-30T14:26:12.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4f6dc5640 1 -- 172.21.15.57:0/477050910 wait complete. 2023-10-30T14:26:12.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4f6dc5640 1 Processor -- start 2023-10-30T14:26:12.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4f6dc5640 1 -- start start 2023-10-30T14:26:12.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4f6dc5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4f00cb8b0 0x7fb4f01ee7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:12.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4f6dc5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f0151840 0x7fb4f006b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:12.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4f6dc5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4f0151c20 0x7fb4f014fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:12.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4f6dc5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f0156030 con 0x7fb4f0151840 2023-10-30T14:26:12.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4f6dc5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f006bb90 con 0x7fb4f0151c20 2023-10-30T14:26:12.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4f6dc5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f006be50 con 0x7fb4f00cb8b0 2023-10-30T14:26:12.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4effff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f0151840 0x7fb4f006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:12.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4effff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f0151840 0x7fb4f006b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40728/0 (socket says 172.21.15.57:40728) 2023-10-30T14:26:12.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4effff640 1 -- 172.21.15.57:0/1618206027 learned_addr learned my addr 172.21.15.57:0/1618206027 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:12.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4f4b61640 1 --2- 172.21.15.57:0/1618206027 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4f00cb8b0 0x7fb4f01ee7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:12.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4f5362640 1 --2- 172.21.15.57:0/1618206027 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4f0151c20 0x7fb4f014fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:12.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.032+0000 7fb4effff640 1 -- 172.21.15.57:0/1618206027 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4f00cb8b0 msgr2=0x7fb4f01ee7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:12.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.036+0000 7fb4effff640 1 --2- 172.21.15.57:0/1618206027 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4f00cb8b0 0x7fb4f01ee7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:12.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.036+0000 7fb4effff640 1 -- 172.21.15.57:0/1618206027 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4f0151c20 msgr2=0x7fb4f014fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:12.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.036+0000 7fb4effff640 1 --2- 172.21.15.57:0/1618206027 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4f0151c20 0x7fb4f014fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:12.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.036+0000 7fb4effff640 1 -- 172.21.15.57:0/1618206027 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e8009b30 con 0x7fb4f0151840 2023-10-30T14:26:12.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.036+0000 7fb4f5362640 1 --2- 172.21.15.57:0/1618206027 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4f0151c20 0x7fb4f014fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:12.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.036+0000 7fb4effff640 1 --2- 172.21.15.57:0/1618206027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f0151840 0x7fb4f006b500 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc00c3c0 tx=0x7fb4dc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:12.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.036+0000 7fb4ed7fa640 1 -- 172.21.15.57:0/1618206027 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc0052d0 con 0x7fb4f0151840 2023-10-30T14:26:12.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.036+0000 7fb4ed7fa640 1 -- 172.21.15.57:0/1618206027 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4dc005430 con 0x7fb4f0151840 2023-10-30T14:26:12.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.036+0000 7fb4ed7fa640 1 -- 172.21.15.57:0/1618206027 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc003840 con 0x7fb4f0151840 2023-10-30T14:26:12.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.036+0000 7fb4f6dc5640 1 -- 172.21.15.57:0/1618206027 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f006c130 con 0x7fb4f0151840 2023-10-30T14:26:12.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.036+0000 7fb4f6dc5640 1 -- 172.21.15.57:0/1618206027 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f0156230 con 0x7fb4f0151840 2023-10-30T14:26:12.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.036+0000 7fb4f6dc5640 1 -- 172.21.15.57:0/1618206027 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f006c650 con 0x7fb4f0151840 2023-10-30T14:26:12.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.036+0000 7fb4ed7fa640 1 -- 172.21.15.57:0/1618206027 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb4dc0039a0 con 0x7fb4f0151840 2023-10-30T14:26:12.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.036+0000 7fb4ed7fa640 1 --2- 172.21.15.57:0/1618206027 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4d40664a0 0x7fb4d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:12.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.036+0000 7fb4ed7fa640 1 -- 172.21.15.57:0/1618206027 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6341+0+0 (secure 0 0 0) 0x7fb4dc0835a0 con 0x7fb4f0151840 2023-10-30T14:26:12.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.036+0000 7fb4f4b61640 1 --2- 172.21.15.57:0/1618206027 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4d40664a0 0x7fb4d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:12.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.040+0000 7fb4f4b61640 1 --2- 172.21.15.57:0/1618206027 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4d40664a0 0x7fb4d4068960 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4007a60 tx=0x7fb4e4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:12.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.040+0000 7fb4ed7fa640 1 -- 172.21.15.57:0/1618206027 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb4f006c650 con 0x7fb4f0151840 2023-10-30T14:26:12.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.232+0000 7fb4f6dc5640 1 -- 172.21.15.57:0/1618206027 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4f006c650 con 0x7fb4f0151840 2023-10-30T14:26:12.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.232+0000 7fb4ed7fa640 1 -- 172.21.15.57:0/1618206027 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6507 (secure 0 0 0) 0x7fb4f006c650 con 0x7fb4f0151840 2023-10-30T14:26:12.236 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:12.237 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":5797,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:24:15.976636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:12.237 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 72 2023-10-30T14:26:12.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.236+0000 7fb4f6dc5640 1 -- 172.21.15.57:0/1618206027 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4d40664a0 msgr2=0x7fb4d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:12.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.236+0000 7fb4f6dc5640 1 --2- 172.21.15.57:0/1618206027 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4d40664a0 0x7fb4d4068960 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4007a60 tx=0x7fb4e4008040 comp rx=0 tx=0).stop 2023-10-30T14:26:12.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.236+0000 7fb4f6dc5640 1 -- 172.21.15.57:0/1618206027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f0151840 msgr2=0x7fb4f006b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:12.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.236+0000 7fb4f6dc5640 1 --2- 172.21.15.57:0/1618206027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f0151840 0x7fb4f006b500 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc00c3c0 tx=0x7fb4dc00c880 comp rx=0 tx=0).stop 2023-10-30T14:26:12.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.236+0000 7fb4f6dc5640 1 -- 172.21.15.57:0/1618206027 shutdown_connections 2023-10-30T14:26:12.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.236+0000 7fb4f6dc5640 1 --2- 172.21.15.57:0/1618206027 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4d40664a0 0x7fb4d4068960 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:12.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.236+0000 7fb4f6dc5640 1 --2- 172.21.15.57:0/1618206027 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4f0151c20 0x7fb4f014fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:12.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.236+0000 7fb4f6dc5640 1 --2- 172.21.15.57:0/1618206027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f0151840 0x7fb4f006b500 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:12.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.236+0000 7fb4f6dc5640 1 --2- 172.21.15.57:0/1618206027 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4f00cb8b0 0x7fb4f01ee7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:12.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.236+0000 7fb4f6dc5640 1 -- 172.21.15.57:0/1618206027 >> 172.21.15.57:0/1618206027 conn(0x7fb4f00cac70 msgr2=0x7fb4f00733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:12.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.236+0000 7fb4f6dc5640 1 -- 172.21.15.57:0/1618206027 shutdown_connections 2023-10-30T14:26:12.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:12.236+0000 7fb4f6dc5640 1 -- 172.21.15.57:0/1618206027 wait complete. 2023-10-30T14:26:12.258 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:26:14.259 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:14.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.544+0000 7f6720d10640 1 Processor -- start 2023-10-30T14:26:14.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.544+0000 7f6720d10640 1 -- start start 2023-10-30T14:26:14.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.548+0000 7f6720d10640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f671c15ac70 0x7f671c15cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:14.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.548+0000 7f6720d10640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f671c06aeb0 0x7f671c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:14.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.548+0000 7f6720d10640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c15dcc0 con 0x7f671c0c8eb0 2023-10-30T14:26:14.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.548+0000 7f6720d10640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f671c159800 con 0x7f671c15ac70 2023-10-30T14:26:14.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.548+0000 7f6720d10640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671c159d80 con 0x7f671c06aeb0 2023-10-30T14:26:14.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.548+0000 7f671b7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f671c0c8eb0 0x7f671c15a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:54544/0 (socket says 172.21.15.57:54544) 2023-10-30T14:26:14.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.548+0000 7f671b7fe640 1 -- 172.21.15.57:0/3085390130 learned_addr learned my addr 172.21.15.57:0/3085390130 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:26:14.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.548+0000 7f671bfff640 1 --2- 172.21.15.57:0/3085390130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f671c06aeb0 0x7f671c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:14.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.548+0000 7f671affd640 1 --2- 172.21.15.57:0/3085390130 >> v2:172.21.15.88:3300/0 conn(0x7f671c15ac70 0x7f671c15cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:14.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.548+0000 7f671a7fc640 1 -- 172.21.15.57:0/3085390130 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465147623 0 0) 0x7f670c001120 con 0x7f671c0c8eb0 2023-10-30T14:26:14.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.548+0000 7f671a7fc640 1 -- 172.21.15.57:0/3085390130 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6704003530 con 0x7f671c0c8eb0 2023-10-30T14:26:14.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.548+0000 7f671bfff640 1 -- 172.21.15.57:0/3085390130 >> v2:172.21.15.88:3300/0 conn(0x7f671c15ac70 msgr2=0x7f671c15cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:14.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.548+0000 7f671bfff640 1 --2- 172.21.15.57:0/3085390130 >> v2:172.21.15.88:3300/0 conn(0x7f671c15ac70 0x7f671c15cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:14.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.548+0000 7f671bfff640 1 -- 172.21.15.57:0/3085390130 >> v1:172.21.15.57:6789/0 conn(0x7f671c0c8eb0 legacy=0x7f671c15a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:14.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.548+0000 7f671bfff640 1 -- 172.21.15.57:0/3085390130 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c15d420 con 0x7f671c06aeb0 2023-10-30T14:26:14.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.552+0000 7f671bfff640 1 --2- 172.21.15.57:0/3085390130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f671c06aeb0 0x7f671c069e90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6714009d90 tx=0x7f671401f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f18da35a4d8c5192 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:14.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.552+0000 7f671a7fc640 1 -- 172.21.15.57:0/3085390130 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67140af600 con 0x7f671c06aeb0 2023-10-30T14:26:14.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.552+0000 7f671a7fc640 1 -- 172.21.15.57:0/3085390130 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67140ab050 con 0x7f671c06aeb0 2023-10-30T14:26:14.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.552+0000 7f671a7fc640 1 -- 172.21.15.57:0/3085390130 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67140af980 con 0x7f671c06aeb0 2023-10-30T14:26:14.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.552+0000 7f6720d10640 1 -- 172.21.15.57:0/3085390130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f671c06aeb0 msgr2=0x7f671c069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:14.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.552+0000 7f6720d10640 1 --2- 172.21.15.57:0/3085390130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f671c06aeb0 0x7f671c069e90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6714009d90 tx=0x7f671401f130 comp rx=0 tx=0).stop 2023-10-30T14:26:14.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.552+0000 7f6720d10640 1 -- 172.21.15.57:0/3085390130 shutdown_connections 2023-10-30T14:26:14.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.552+0000 7f6720d10640 1 --2- 172.21.15.57:0/3085390130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f671c06aeb0 0x7f671c069e90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:14.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.552+0000 7f6720d10640 1 --2- 172.21.15.57:0/3085390130 >> v2:172.21.15.88:3300/0 conn(0x7f671c15ac70 0x7f671c15cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:14.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.552+0000 7f6720d10640 1 -- 172.21.15.57:0/3085390130 >> 172.21.15.57:0/3085390130 conn(0x7f671c0cef00 msgr2=0x7f671c0cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:14.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.552+0000 7f6720d10640 1 -- 172.21.15.57:0/3085390130 shutdown_connections 2023-10-30T14:26:14.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.552+0000 7f6720d10640 1 -- 172.21.15.57:0/3085390130 wait complete. 2023-10-30T14:26:14.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.552+0000 7f6720d10640 1 Processor -- start 2023-10-30T14:26:14.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.552+0000 7f6720d10640 1 -- start start 2023-10-30T14:26:14.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f6720d10640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f671c06aeb0 0x7f671c14f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:14.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f6720d10640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f671c0c8eb0 0x7f671c158df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:14.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f6720d10640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f671c15ac70 0x7f671c154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:14.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f6720d10640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c14fed0 con 0x7f671c0c8eb0 2023-10-30T14:26:14.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f6720d10640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c0c7330 con 0x7f671c15ac70 2023-10-30T14:26:14.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f6720d10640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671c157110 con 0x7f671c06aeb0 2023-10-30T14:26:14.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f671affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f671c0c8eb0 0x7f671c158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:14.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f671affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f671c0c8eb0 0x7f671c158df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56736/0 (socket says 172.21.15.57:56736) 2023-10-30T14:26:14.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f671affd640 1 -- 172.21.15.57:0/544610799 learned_addr learned my addr 172.21.15.57:0/544610799 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:14.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f671b7fe640 1 --2- 172.21.15.57:0/544610799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f671c06aeb0 0x7f671c14f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:14.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f671bfff640 1 --2- 172.21.15.57:0/544610799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f671c15ac70 0x7f671c154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:14.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f671affd640 1 -- 172.21.15.57:0/544610799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f671c06aeb0 msgr2=0x7f671c14f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:14.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f671affd640 1 --2- 172.21.15.57:0/544610799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f671c06aeb0 0x7f671c14f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:14.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f671affd640 1 -- 172.21.15.57:0/544610799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f671c15ac70 msgr2=0x7f671c154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:14.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f671affd640 1 --2- 172.21.15.57:0/544610799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f671c15ac70 0x7f671c154170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:14.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f671affd640 1 -- 172.21.15.57:0/544610799 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6714009a70 con 0x7f671c0c8eb0 2023-10-30T14:26:14.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f671affd640 1 --2- 172.21.15.57:0/544610799 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f671c0c8eb0 0x7f671c158df0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f671000e870 tx=0x7f671000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:14.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f6703fff640 1 -- 172.21.15.57:0/544610799 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6710009510 con 0x7f671c0c8eb0 2023-10-30T14:26:14.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f6703fff640 1 -- 172.21.15.57:0/544610799 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f671000eec0 con 0x7f671c0c8eb0 2023-10-30T14:26:14.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f6720d10640 1 -- 172.21.15.57:0/544610799 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671c0c7560 con 0x7f671c0c8eb0 2023-10-30T14:26:14.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f6703fff640 1 -- 172.21.15.57:0/544610799 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671000cb40 con 0x7f671c0c8eb0 2023-10-30T14:26:14.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f6720d10640 1 -- 172.21.15.57:0/544610799 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671c156e30 con 0x7f671c0c8eb0 2023-10-30T14:26:14.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.556+0000 7f6720d10640 1 -- 172.21.15.57:0/544610799 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671c04e170 con 0x7f671c0c8eb0 2023-10-30T14:26:14.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.560+0000 7f6703fff640 1 -- 172.21.15.57:0/544610799 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f671000cca0 con 0x7f671c0c8eb0 2023-10-30T14:26:14.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.560+0000 7f6703fff640 1 --2- 172.21.15.57:0/544610799 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f67080664a0 0x7f6708068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:14.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.560+0000 7f6703fff640 1 -- 172.21.15.57:0/544610799 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f6710084190 con 0x7f671c0c8eb0 2023-10-30T14:26:14.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.560+0000 7f671b7fe640 1 --2- 172.21.15.57:0/544610799 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f67080664a0 0x7f6708068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:14.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.560+0000 7f671b7fe640 1 --2- 172.21.15.57:0/544610799 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f67080664a0 0x7f6708068960 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f670c001280 tx=0x7f670c007bc0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:14.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.564+0000 7f6703fff640 1 -- 172.21.15.57:0/544610799 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6710051430 con 0x7f671c0c8eb0 2023-10-30T14:26:14.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.776+0000 7f6720d10640 1 -- 172.21.15.57:0/544610799 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f671c0b2d30 con 0x7f671c0c8eb0 2023-10-30T14:26:14.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.776+0000 7f6703fff640 1 -- 172.21.15.57:0/544610799 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6507 (secure 0 0 0) 0x7f6710054a50 con 0x7f671c0c8eb0 2023-10-30T14:26:14.780 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:14.780 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":5797,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:24:15.976636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:14.781 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 72 2023-10-30T14:26:14.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.780+0000 7f6720d10640 1 -- 172.21.15.57:0/544610799 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f67080664a0 msgr2=0x7f6708068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:14.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.780+0000 7f6720d10640 1 --2- 172.21.15.57:0/544610799 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f67080664a0 0x7f6708068960 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f670c001280 tx=0x7f670c007bc0 comp rx=0 tx=0).stop 2023-10-30T14:26:14.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.780+0000 7f6720d10640 1 -- 172.21.15.57:0/544610799 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f671c0c8eb0 msgr2=0x7f671c158df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:14.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.780+0000 7f6720d10640 1 --2- 172.21.15.57:0/544610799 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f671c0c8eb0 0x7f671c158df0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f671000e870 tx=0x7f671000ec70 comp rx=0 tx=0).stop 2023-10-30T14:26:14.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.780+0000 7f6720d10640 1 -- 172.21.15.57:0/544610799 shutdown_connections 2023-10-30T14:26:14.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.780+0000 7f6720d10640 1 --2- 172.21.15.57:0/544610799 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f67080664a0 0x7f6708068960 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:14.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.780+0000 7f6720d10640 1 --2- 172.21.15.57:0/544610799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f671c15ac70 0x7f671c154170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:14.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.780+0000 7f6720d10640 1 --2- 172.21.15.57:0/544610799 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f671c0c8eb0 0x7f671c158df0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:14.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.780+0000 7f6720d10640 1 --2- 172.21.15.57:0/544610799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f671c06aeb0 0x7f671c14f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:14.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.780+0000 7f6720d10640 1 -- 172.21.15.57:0/544610799 >> 172.21.15.57:0/544610799 conn(0x7f671c0cef00 msgr2=0x7f671c0733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:14.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.780+0000 7f6720d10640 1 -- 172.21.15.57:0/544610799 shutdown_connections 2023-10-30T14:26:14.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:14.780+0000 7f6720d10640 1 -- 172.21.15.57:0/544610799 wait complete. 2023-10-30T14:26:14.801 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:26:16.803 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:16.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.956+0000 7f29ff009640 1 Processor -- start 2023-10-30T14:26:16.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.956+0000 7f29ff009640 1 -- start start 2023-10-30T14:26:16.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.956+0000 7f29ff009640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f29f815cf90 0x7f29f80c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:16.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.956+0000 7f29ff009640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f8159f20 con 0x7f29f8153c80 2023-10-30T14:26:16.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.956+0000 7f29ff009640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f80c47e0 con 0x7f29f80c4a10 2023-10-30T14:26:16.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.956+0000 7f29ff009640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29f815dec0 con 0x7f29f815cf90 2023-10-30T14:26:16.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.956+0000 7f29fd5a6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f29f815cf90 0x7f29f80c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:16.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.956+0000 7f29fcda5640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f29f8153c80 0x7f29f815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:54552/0 (socket says 172.21.15.57:54552) 2023-10-30T14:26:16.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.956+0000 7f29fd5a6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f29f815cf90 0x7f29f80c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56738/0 (socket says 172.21.15.57:56738) 2023-10-30T14:26:16.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.956+0000 7f29fcda5640 1 -- 172.21.15.57:0/1596234364 learned_addr learned my addr 172.21.15.57:0/1596234364 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:26:16.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.956+0000 7f29fd5a6640 1 -- 172.21.15.57:0/1596234364 >> v1:172.21.15.88:6789/0 conn(0x7f29f80c4a10 legacy=0x7f29f80ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:16.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.956+0000 7f29fd5a6640 1 -- 172.21.15.57:0/1596234364 >> v1:172.21.15.57:6789/0 conn(0x7f29f8153c80 legacy=0x7f29f815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:16.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.956+0000 7f29f77fe640 1 -- 172.21.15.57:0/1596234364 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074472371 0 0) 0x7f29f8159f20 con 0x7f29f8153c80 2023-10-30T14:26:16.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.956+0000 7f29fd5a6640 1 -- 172.21.15.57:0/1596234364 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f815d620 con 0x7f29f815cf90 2023-10-30T14:26:16.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.960+0000 7f29fd5a6640 1 --2- 172.21.15.57:0/1596234364 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29f815cf90 0x7f29f80c42a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f29f0009d90 tx=0x7f29f001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b083a03a655b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:16.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.960+0000 7f29f77fe640 1 -- 172.21.15.57:0/1596234364 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f00af600 con 0x7f29f815cf90 2023-10-30T14:26:16.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.960+0000 7f29f77fe640 1 -- 172.21.15.57:0/1596234364 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f00ab050 con 0x7f29f815cf90 2023-10-30T14:26:16.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.960+0000 7f29f77fe640 1 -- 172.21.15.57:0/1596234364 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f00af980 con 0x7f29f815cf90 2023-10-30T14:26:16.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.960+0000 7f29ff009640 1 -- 172.21.15.57:0/1596234364 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29f815cf90 msgr2=0x7f29f80c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:16.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.960+0000 7f29ff009640 1 --2- 172.21.15.57:0/1596234364 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29f815cf90 0x7f29f80c42a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f29f0009d90 tx=0x7f29f001f130 comp rx=0 tx=0).stop 2023-10-30T14:26:16.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.960+0000 7f29ff009640 1 -- 172.21.15.57:0/1596234364 shutdown_connections 2023-10-30T14:26:16.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.960+0000 7f29ff009640 1 --2- 172.21.15.57:0/1596234364 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29f815cf90 0x7f29f80c42a0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:16.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.960+0000 7f29ff009640 1 -- 172.21.15.57:0/1596234364 >> 172.21.15.57:0/1596234364 conn(0x7f29f80715c0 msgr2=0x7f29f80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:16.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.960+0000 7f29ff009640 1 -- 172.21.15.57:0/1596234364 shutdown_connections 2023-10-30T14:26:16.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.960+0000 7f29ff009640 1 -- 172.21.15.57:0/1596234364 wait complete. 2023-10-30T14:26:16.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29ff009640 1 Processor -- start 2023-10-30T14:26:16.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29ff009640 1 -- start start 2023-10-30T14:26:16.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29ff009640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29f80c4a10 0x7f29f80cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:16.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29ff009640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29f8153c80 0x7f29f80cf310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:16.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29ff009640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29f815cf90 0x7f29f81f2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:16.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29ff009640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f8151900 con 0x7f29f815cf90 2023-10-30T14:26:16.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29ff009640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f8156f60 con 0x7f29f8153c80 2023-10-30T14:26:16.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29ff009640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f8157220 con 0x7f29f80c4a10 2023-10-30T14:26:16.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29fd5a6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29f815cf90 0x7f29f81f2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:16.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29fd5a6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29f815cf90 0x7f29f81f2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56752/0 (socket says 172.21.15.57:56752) 2023-10-30T14:26:16.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29fd5a6640 1 -- 172.21.15.57:0/806105267 learned_addr learned my addr 172.21.15.57:0/806105267 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:16.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29fcda5640 1 --2- 172.21.15.57:0/806105267 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29f80c4a10 0x7f29f80cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:16.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29f7fff640 1 --2- 172.21.15.57:0/806105267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29f8153c80 0x7f29f80cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:16.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29fd5a6640 1 -- 172.21.15.57:0/806105267 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29f80c4a10 msgr2=0x7f29f80cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:16.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29fd5a6640 1 --2- 172.21.15.57:0/806105267 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29f80c4a10 0x7f29f80cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:16.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29fd5a6640 1 -- 172.21.15.57:0/806105267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29f8153c80 msgr2=0x7f29f80cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:16.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29fd5a6640 1 --2- 172.21.15.57:0/806105267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29f8153c80 0x7f29f80cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:16.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29fd5a6640 1 -- 172.21.15.57:0/806105267 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f0009a70 con 0x7f29f815cf90 2023-10-30T14:26:16.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29fd5a6640 1 --2- 172.21.15.57:0/806105267 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29f815cf90 0x7f29f81f2260 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f29f00b2040 tx=0x7f29f001bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:16.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29f57fa640 1 -- 172.21.15.57:0/806105267 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f00ba680 con 0x7f29f815cf90 2023-10-30T14:26:16.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29f57fa640 1 -- 172.21.15.57:0/806105267 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f00ba7e0 con 0x7f29f815cf90 2023-10-30T14:26:16.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29f57fa640 1 -- 172.21.15.57:0/806105267 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f00bab90 con 0x7f29f815cf90 2023-10-30T14:26:16.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29ff009640 1 -- 172.21.15.57:0/806105267 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f81548e0 con 0x7f29f815cf90 2023-10-30T14:26:16.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.964+0000 7f29ff009640 1 -- 172.21.15.57:0/806105267 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f8154ea0 con 0x7f29f815cf90 2023-10-30T14:26:16.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.968+0000 7f29ff009640 1 -- 172.21.15.57:0/806105267 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f81553c0 con 0x7f29f815cf90 2023-10-30T14:26:16.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.968+0000 7f29f57fa640 1 -- 172.21.15.57:0/806105267 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f29f00b6040 con 0x7f29f815cf90 2023-10-30T14:26:16.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.968+0000 7f29f57fa640 1 --2- 172.21.15.57:0/806105267 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29dc0664a0 0x7f29dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:16.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.968+0000 7f29f57fa640 1 -- 172.21.15.57:0/806105267 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f29f0122ab0 con 0x7f29f815cf90 2023-10-30T14:26:16.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.968+0000 7f29fcda5640 1 --2- 172.21.15.57:0/806105267 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29dc0664a0 0x7f29dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:16.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.968+0000 7f29fcda5640 1 --2- 172.21.15.57:0/806105267 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29dc0664a0 0x7f29dc068960 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f29ec000f70 tx=0x7f29ec007240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:16.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:16.972+0000 7f29f57fa640 1 -- 172.21.15.57:0/806105267 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f29f81553c0 con 0x7f29f815cf90 2023-10-30T14:26:17.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:17.164+0000 7f29ff009640 1 -- 172.21.15.57:0/806105267 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29f81553c0 con 0x7f29f815cf90 2023-10-30T14:26:17.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:17.164+0000 7f29f57fa640 1 -- 172.21.15.57:0/806105267 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6507 (secure 0 0 0) 0x7f29f81553c0 con 0x7f29f815cf90 2023-10-30T14:26:17.168 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:17.168 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":5797,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:24:15.976636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:17.169 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 72 2023-10-30T14:26:17.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:17.168+0000 7f29ff009640 1 -- 172.21.15.57:0/806105267 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29dc0664a0 msgr2=0x7f29dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:17.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:17.168+0000 7f29ff009640 1 --2- 172.21.15.57:0/806105267 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29dc0664a0 0x7f29dc068960 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f29ec000f70 tx=0x7f29ec007240 comp rx=0 tx=0).stop 2023-10-30T14:26:17.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:17.168+0000 7f29ff009640 1 -- 172.21.15.57:0/806105267 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29f815cf90 msgr2=0x7f29f81f2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:17.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:17.168+0000 7f29ff009640 1 --2- 172.21.15.57:0/806105267 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29f815cf90 0x7f29f81f2260 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f29f00b2040 tx=0x7f29f001bed0 comp rx=0 tx=0).stop 2023-10-30T14:26:17.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:17.168+0000 7f29ff009640 1 -- 172.21.15.57:0/806105267 shutdown_connections 2023-10-30T14:26:17.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:17.168+0000 7f29ff009640 1 --2- 172.21.15.57:0/806105267 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29dc0664a0 0x7f29dc068960 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:17.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:17.168+0000 7f29ff009640 1 --2- 172.21.15.57:0/806105267 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29f815cf90 0x7f29f81f2260 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:17.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:17.168+0000 7f29ff009640 1 --2- 172.21.15.57:0/806105267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29f8153c80 0x7f29f80cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:17.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:17.168+0000 7f29ff009640 1 --2- 172.21.15.57:0/806105267 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29f80c4a10 0x7f29f80cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:17.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:17.168+0000 7f29ff009640 1 -- 172.21.15.57:0/806105267 >> 172.21.15.57:0/806105267 conn(0x7f29f80715c0 msgr2=0x7f29f81520f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:17.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:17.168+0000 7f29ff009640 1 -- 172.21.15.57:0/806105267 shutdown_connections 2023-10-30T14:26:17.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:17.168+0000 7f29ff009640 1 -- 172.21.15.57:0/806105267 wait complete. 2023-10-30T14:26:17.189 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:26:19.190 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:19.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.344+0000 7f6129db5640 1 Processor -- start 2023-10-30T14:26:19.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.344+0000 7f6129db5640 1 -- start start 2023-10-30T14:26:19.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.344+0000 7f6129db5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f61240c8eb0 0x7f612415cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:19.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.344+0000 7f6129db5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612415dcc0 con 0x7f612415ac70 2023-10-30T14:26:19.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.344+0000 7f6129db5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6124159d80 con 0x7f612406aeb0 2023-10-30T14:26:19.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.344+0000 7f6129db5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6124159800 con 0x7f61240c8eb0 2023-10-30T14:26:19.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.344+0000 7f6128db3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f61240c8eb0 0x7f612415cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:19.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.344+0000 7f6123fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f612415ac70 0x7f612415a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:54562/0 (socket says 172.21.15.57:54562) 2023-10-30T14:26:19.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.344+0000 7f6123fff640 1 -- 172.21.15.57:0/652069043 learned_addr learned my addr 172.21.15.57:0/652069043 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:26:19.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.344+0000 7f6128db3640 1 -- 172.21.15.57:0/652069043 >> v1:172.21.15.88:6789/0 conn(0x7f612406aeb0 legacy=0x7f6124069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:19.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.344+0000 7f61237fe640 1 -- 172.21.15.57:0/652069043 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338272483 0 0) 0x7f6110001120 con 0x7f612415ac70 2023-10-30T14:26:19.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.344+0000 7f6128db3640 1 -- 172.21.15.57:0/652069043 >> v1:172.21.15.57:6789/0 conn(0x7f612415ac70 legacy=0x7f612415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:19.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.344+0000 7f6128db3640 1 -- 172.21.15.57:0/652069043 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612415d420 con 0x7f61240c8eb0 2023-10-30T14:26:19.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.348+0000 7f6128db3640 1 --2- 172.21.15.57:0/652069043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61240c8eb0 0x7f612415cd70 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f61180097c0 tx=0x7f611801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58fdce941a14d4a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:19.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.348+0000 7f61237fe640 1 -- 172.21.15.57:0/652069043 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61180a8970 con 0x7f61240c8eb0 2023-10-30T14:26:19.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.348+0000 7f61237fe640 1 -- 172.21.15.57:0/652069043 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61180a8ad0 con 0x7f61240c8eb0 2023-10-30T14:26:19.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.348+0000 7f61237fe640 1 -- 172.21.15.57:0/652069043 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61180a8e60 con 0x7f61240c8eb0 2023-10-30T14:26:19.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.348+0000 7f6129db5640 1 -- 172.21.15.57:0/652069043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61240c8eb0 msgr2=0x7f612415cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:19.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.348+0000 7f6129db5640 1 --2- 172.21.15.57:0/652069043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61240c8eb0 0x7f612415cd70 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f61180097c0 tx=0x7f611801ec70 comp rx=0 tx=0).stop 2023-10-30T14:26:19.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.348+0000 7f6129db5640 1 -- 172.21.15.57:0/652069043 shutdown_connections 2023-10-30T14:26:19.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.348+0000 7f6129db5640 1 --2- 172.21.15.57:0/652069043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61240c8eb0 0x7f612415cd70 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:19.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.348+0000 7f6129db5640 1 -- 172.21.15.57:0/652069043 >> 172.21.15.57:0/652069043 conn(0x7f61240cef00 msgr2=0x7f61240cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:19.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.348+0000 7f6129db5640 1 -- 172.21.15.57:0/652069043 shutdown_connections 2023-10-30T14:26:19.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.348+0000 7f6129db5640 1 -- 172.21.15.57:0/652069043 wait complete. 2023-10-30T14:26:19.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f6129db5640 1 Processor -- start 2023-10-30T14:26:19.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f6129db5640 1 -- start start 2023-10-30T14:26:19.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f6129db5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612406aeb0 0x7f61241f2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:19.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f6129db5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61240c8eb0 0x7f61241578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:19.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f6129db5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f612415ac70 0x7f612415c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:19.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f6129db5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61241efd60 con 0x7f612415ac70 2023-10-30T14:26:19.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f6129db5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61240c3150 con 0x7f612406aeb0 2023-10-30T14:26:19.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f6129db5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61241f24b0 con 0x7f61240c8eb0 2023-10-30T14:26:19.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f6128db3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612406aeb0 0x7f61241f2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:19.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f6123fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61240c8eb0 0x7f61241578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:19.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f6128db3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612406aeb0 0x7f61241f2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:38230/0 (socket says 172.21.15.57:38230) 2023-10-30T14:26:19.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f6128db3640 1 -- 172.21.15.57:0/2233779691 learned_addr learned my addr 172.21.15.57:0/2233779691 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:19.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f61295b4640 1 --2- 172.21.15.57:0/2233779691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f612415ac70 0x7f612415c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:19.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f61295b4640 1 -- 172.21.15.57:0/2233779691 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61240c8eb0 msgr2=0x7f61241578d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:19.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f61295b4640 1 --2- 172.21.15.57:0/2233779691 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61240c8eb0 0x7f61241578d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:19.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f61295b4640 1 -- 172.21.15.57:0/2233779691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612406aeb0 msgr2=0x7f61241f2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:19.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f61295b4640 1 --2- 172.21.15.57:0/2233779691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612406aeb0 0x7f61241f2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:19.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f61295b4640 1 -- 172.21.15.57:0/2233779691 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6118009420 con 0x7f612415ac70 2023-10-30T14:26:19.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f6128db3640 1 --2- 172.21.15.57:0/2233779691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612406aeb0 0x7f61241f2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:26:19.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f61295b4640 1 --2- 172.21.15.57:0/2233779691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f612415ac70 0x7f612415c0d0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f611c00c730 tx=0x7f611c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:19.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f61217fa640 1 -- 172.21.15.57:0/2233779691 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c003b80 con 0x7f612415ac70 2023-10-30T14:26:19.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f61217fa640 1 -- 172.21.15.57:0/2233779691 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f611c003ce0 con 0x7f612415ac70 2023-10-30T14:26:19.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f61217fa640 1 -- 172.21.15.57:0/2233779691 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c00aa30 con 0x7f612415ac70 2023-10-30T14:26:19.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.352+0000 7f6129db5640 1 -- 172.21.15.57:0/2233779691 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61241f3190 con 0x7f612415ac70 2023-10-30T14:26:19.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.356+0000 7f6129db5640 1 -- 172.21.15.57:0/2233779691 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61240c2e70 con 0x7f612415ac70 2023-10-30T14:26:19.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.356+0000 7f6129db5640 1 -- 172.21.15.57:0/2233779691 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61241f2310 con 0x7f612415ac70 2023-10-30T14:26:19.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.356+0000 7f61217fa640 1 -- 172.21.15.57:0/2233779691 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f611c00ab90 con 0x7f612415ac70 2023-10-30T14:26:19.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.360+0000 7f61217fa640 1 --2- 172.21.15.57:0/2233779691 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f61080664a0 0x7f6108068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:19.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.360+0000 7f61217fa640 1 -- 172.21.15.57:0/2233779691 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f611c083da0 con 0x7f612415ac70 2023-10-30T14:26:19.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.360+0000 7f6128db3640 1 --2- 172.21.15.57:0/2233779691 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f61080664a0 0x7f6108068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:19.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.360+0000 7f6128db3640 1 --2- 172.21.15.57:0/2233779691 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f61080664a0 0x7f6108068960 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f61180097c0 tx=0x7f6118008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:19.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.368+0000 7f61217fa640 1 -- 172.21.15.57:0/2233779691 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f61241f2310 con 0x7f612415ac70 2023-10-30T14:26:19.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.688+0000 7f6129db5640 1 -- 172.21.15.57:0/2233779691 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61241f2310 con 0x7f612415ac70 2023-10-30T14:26:19.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.692+0000 7f61217fa640 1 -- 172.21.15.57:0/2233779691 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6507 (secure 0 0 0) 0x7f61241f2310 con 0x7f612415ac70 2023-10-30T14:26:19.696 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:19.696 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":5797,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:24:15.976636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:19.696 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 72 2023-10-30T14:26:19.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.696+0000 7f6129db5640 1 -- 172.21.15.57:0/2233779691 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f61080664a0 msgr2=0x7f6108068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:19.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.696+0000 7f6129db5640 1 --2- 172.21.15.57:0/2233779691 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f61080664a0 0x7f6108068960 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f61180097c0 tx=0x7f6118008040 comp rx=0 tx=0).stop 2023-10-30T14:26:19.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.696+0000 7f6129db5640 1 -- 172.21.15.57:0/2233779691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f612415ac70 msgr2=0x7f612415c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:19.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.696+0000 7f6129db5640 1 --2- 172.21.15.57:0/2233779691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f612415ac70 0x7f612415c0d0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f611c00c730 tx=0x7f611c00cbf0 comp rx=0 tx=0).stop 2023-10-30T14:26:19.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.700+0000 7f6129db5640 1 -- 172.21.15.57:0/2233779691 shutdown_connections 2023-10-30T14:26:19.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.700+0000 7f6129db5640 1 --2- 172.21.15.57:0/2233779691 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f61080664a0 0x7f6108068960 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:19.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.700+0000 7f6129db5640 1 --2- 172.21.15.57:0/2233779691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f612415ac70 0x7f612415c0d0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:19.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.700+0000 7f6129db5640 1 --2- 172.21.15.57:0/2233779691 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61240c8eb0 0x7f61241578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:19.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.700+0000 7f6129db5640 1 --2- 172.21.15.57:0/2233779691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612406aeb0 0x7f61241f2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:19.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.700+0000 7f6129db5640 1 -- 172.21.15.57:0/2233779691 >> 172.21.15.57:0/2233779691 conn(0x7f61240cef00 msgr2=0x7f61240c7080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:19.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.700+0000 7f6129db5640 1 -- 172.21.15.57:0/2233779691 shutdown_connections 2023-10-30T14:26:19.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:19.700+0000 7f6129db5640 1 -- 172.21.15.57:0/2233779691 wait complete. 2023-10-30T14:26:19.733 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:26:21.734 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:21.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.884+0000 7f2d53854640 1 Processor -- start 2023-10-30T14:26:21.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.884+0000 7f2d53854640 1 -- start start 2023-10-30T14:26:21.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d53854640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2d4c06b760 0x7f2d4c15e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:21.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d53854640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d4c071e00 con 0x7f2d4c06c420 2023-10-30T14:26:21.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d53854640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d4c070020 con 0x7f2d4c06bb40 2023-10-30T14:26:21.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d53854640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d4c15eb90 con 0x7f2d4c06b760 2023-10-30T14:26:21.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d51df1640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f2d4c06c420 0x7f2d4c0c57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:54570/0 (socket says 172.21.15.57:54570) 2023-10-30T14:26:21.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d51df1640 1 -- 172.21.15.57:0/3582260055 learned_addr learned my addr 172.21.15.57:0/3582260055 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:26:21.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d515f0640 1 --2- 172.21.15.57:0/3582260055 >> v2:172.21.15.88:3300/0 conn(0x7f2d4c06b760 0x7f2d4c15e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:21.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d43fff640 1 -- 172.21.15.57:0/3582260055 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680486185 0 0) 0x7f2d4c071e00 con 0x7f2d4c06c420 2023-10-30T14:26:21.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d43fff640 1 -- 172.21.15.57:0/3582260055 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d38003530 con 0x7f2d4c06c420 2023-10-30T14:26:21.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d43fff640 1 -- 172.21.15.57:0/3582260055 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2033162015 0 0) 0x7f2d38003530 con 0x7f2d4c06c420 2023-10-30T14:26:21.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d43fff640 1 -- 172.21.15.57:0/3582260055 >> v2:172.21.15.88:3300/0 conn(0x7f2d4c06b760 msgr2=0x7f2d4c15e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:21.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d43fff640 1 --2- 172.21.15.57:0/3582260055 >> v2:172.21.15.88:3300/0 conn(0x7f2d4c06b760 0x7f2d4c15e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:21.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d43fff640 1 -- 172.21.15.57:0/3582260055 >> v1:172.21.15.88:6789/0 conn(0x7f2d4c06bb40 legacy=0x7f2d4c157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:21.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d43fff640 1 -- 172.21.15.57:0/3582260055 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d4c070f10 con 0x7f2d4c06c420 2023-10-30T14:26:21.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d43fff640 1 -- 172.21.15.57:0/3582260055 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f2d44001a00 con 0x7f2d4c06c420 2023-10-30T14:26:21.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d43fff640 1 -- 172.21.15.57:0/3582260055 >> v1:172.21.15.57:6789/0 conn(0x7f2d4c06c420 legacy=0x7f2d4c0c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:21.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d43fff640 1 --2- 172.21.15.57:0/3582260055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d38003980 0x7f2d38003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:21.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d43fff640 1 --2- 172.21.15.57:0/3582260055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d380047d0 0x7f2d38004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:21.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d43fff640 1 --2- 172.21.15.57:0/3582260055 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d38005190 0x7f2d3800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:21.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d43fff640 1 -- 172.21.15.57:0/3582260055 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4c15eb90 con 0x7f2d38003980 2023-10-30T14:26:21.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d43fff640 1 -- 172.21.15.57:0/3582260055 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3800c030 con 0x7f2d380047d0 2023-10-30T14:26:21.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d43fff640 1 -- 172.21.15.57:0/3582260055 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3800c1f0 con 0x7f2d38005190 2023-10-30T14:26:21.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d515f0640 1 --2- 172.21.15.57:0/3582260055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d38003980 0x7f2d38003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:21.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d51df1640 1 --2- 172.21.15.57:0/3582260055 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d38005190 0x7f2d3800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:21.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d50def640 1 --2- 172.21.15.57:0/3582260055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d380047d0 0x7f2d38004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:21.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d515f0640 1 -- 172.21.15.57:0/3582260055 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d38005190 msgr2=0x7f2d3800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:21.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d515f0640 1 --2- 172.21.15.57:0/3582260055 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d38005190 0x7f2d3800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:21.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d515f0640 1 -- 172.21.15.57:0/3582260055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d380047d0 msgr2=0x7f2d38004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:21.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d515f0640 1 --2- 172.21.15.57:0/3582260055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d380047d0 0x7f2d38004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:21.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d515f0640 1 -- 172.21.15.57:0/3582260055 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d38003530 con 0x7f2d38003980 2023-10-30T14:26:21.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.888+0000 7f2d51df1640 1 -- 172.21.15.57:0/3582260055 reap_dead start 2023-10-30T14:26:21.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.892+0000 7f2d515f0640 1 --2- 172.21.15.57:0/3582260055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d38003980 0x7f2d38003da0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009ae0 tx=0x7f2d4801ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=571202f162b7ce38 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:21.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.892+0000 7f2d43fff640 1 -- 172.21.15.57:0/3582260055 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d480a5610 con 0x7f2d38003980 2023-10-30T14:26:21.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.892+0000 7f2d43fff640 1 -- 172.21.15.57:0/3582260055 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d480a5770 con 0x7f2d38003980 2023-10-30T14:26:21.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.892+0000 7f2d43fff640 1 -- 172.21.15.57:0/3582260055 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d480a5b20 con 0x7f2d38003980 2023-10-30T14:26:21.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.892+0000 7f2d53854640 1 -- 172.21.15.57:0/3582260055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d38003980 msgr2=0x7f2d38003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:21.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.892+0000 7f2d53854640 1 --2- 172.21.15.57:0/3582260055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d38003980 0x7f2d38003da0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009ae0 tx=0x7f2d4801ca20 comp rx=0 tx=0).stop 2023-10-30T14:26:21.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.892+0000 7f2d53854640 1 -- 172.21.15.57:0/3582260055 shutdown_connections 2023-10-30T14:26:21.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.892+0000 7f2d53854640 1 --2- 172.21.15.57:0/3582260055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d38003980 0x7f2d38003da0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:21.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.892+0000 7f2d53854640 1 -- 172.21.15.57:0/3582260055 >> 172.21.15.57:0/3582260055 conn(0x7f2d4c0cac70 msgr2=0x7f2d4c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:21.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.892+0000 7f2d53854640 1 -- 172.21.15.57:0/3582260055 shutdown_connections 2023-10-30T14:26:21.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.892+0000 7f2d53854640 1 -- 172.21.15.57:0/3582260055 wait complete. 2023-10-30T14:26:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d53854640 1 Processor -- start 2023-10-30T14:26:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d53854640 1 -- start start 2023-10-30T14:26:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d53854640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d38003980 0x7f2d4c1595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d53854640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d4c15afa0 0x7f2d4c0c3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d53854640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d4c14bdd0 0x7f2d4c14c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:21.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d53854640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4c15ace0 con 0x7f2d4c15afa0 2023-10-30T14:26:21.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d53854640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4c1f9490 con 0x7f2d38003980 2023-10-30T14:26:21.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d53854640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4c1f9770 con 0x7f2d4c14bdd0 2023-10-30T14:26:21.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d50def640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d4c15afa0 0x7f2d4c0c3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:21.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d50def640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d4c15afa0 0x7f2d4c0c3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56772/0 (socket says 172.21.15.57:56772) 2023-10-30T14:26:21.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d50def640 1 -- 172.21.15.57:0/1622284739 learned_addr learned my addr 172.21.15.57:0/1622284739 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:21.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d515f0640 1 --2- 172.21.15.57:0/1622284739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d38003980 0x7f2d4c1595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:21.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d51df1640 1 --2- 172.21.15.57:0/1622284739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d4c14bdd0 0x7f2d4c14c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:21.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d50def640 1 -- 172.21.15.57:0/1622284739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d4c14bdd0 msgr2=0x7f2d4c14c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:21.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d50def640 1 --2- 172.21.15.57:0/1622284739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d4c14bdd0 0x7f2d4c14c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:21.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d50def640 1 -- 172.21.15.57:0/1622284739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d38003980 msgr2=0x7f2d4c1595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:21.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d50def640 1 --2- 172.21.15.57:0/1622284739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d38003980 0x7f2d4c1595e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:21.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d50def640 1 -- 172.21.15.57:0/1622284739 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d4c1f9a50 con 0x7f2d4c15afa0 2023-10-30T14:26:21.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d50def640 1 --2- 172.21.15.57:0/1622284739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d4c15afa0 0x7f2d4c0c3b50 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c00f7f0 tx=0x7f2d3c00fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:21.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d41ffb640 1 -- 172.21.15.57:0/1622284739 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c0050d0 con 0x7f2d4c15afa0 2023-10-30T14:26:21.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d41ffb640 1 -- 172.21.15.57:0/1622284739 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d3c005230 con 0x7f2d4c15afa0 2023-10-30T14:26:21.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d41ffb640 1 -- 172.21.15.57:0/1622284739 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c00d730 con 0x7f2d4c15afa0 2023-10-30T14:26:21.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d53854640 1 -- 172.21.15.57:0/1622284739 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d4c1f9d30 con 0x7f2d4c15afa0 2023-10-30T14:26:21.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.896+0000 7f2d53854640 1 -- 172.21.15.57:0/1622284739 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d4c15b420 con 0x7f2d4c15afa0 2023-10-30T14:26:21.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.900+0000 7f2d53854640 1 -- 172.21.15.57:0/1622284739 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d4c04e170 con 0x7f2d4c15afa0 2023-10-30T14:26:21.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.900+0000 7f2d41ffb640 1 -- 172.21.15.57:0/1622284739 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2d3c00d890 con 0x7f2d4c15afa0 2023-10-30T14:26:21.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.900+0000 7f2d41ffb640 1 --2- 172.21.15.57:0/1622284739 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2d300664d0 0x7f2d30068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:21.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.900+0000 7f2d41ffb640 1 -- 172.21.15.57:0/1622284739 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f2d3c083f00 con 0x7f2d4c15afa0 2023-10-30T14:26:21.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.900+0000 7f2d515f0640 1 --2- 172.21.15.57:0/1622284739 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2d300664d0 0x7f2d30068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:21.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.900+0000 7f2d515f0640 1 --2- 172.21.15.57:0/1622284739 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2d300664d0 0x7f2d30068990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2d480af000 tx=0x7f2d480046d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:21.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:21.904+0000 7f2d41ffb640 1 -- 172.21.15.57:0/1622284739 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2d3c0511a0 con 0x7f2d4c15afa0 2023-10-30T14:26:22.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:22.096+0000 7f2d53854640 1 -- 172.21.15.57:0/1622284739 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d4c1573c0 con 0x7f2d4c15afa0 2023-10-30T14:26:22.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:22.096+0000 7f2d41ffb640 1 -- 172.21.15.57:0/1622284739 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6507 (secure 0 0 0) 0x7f2d3c0547c0 con 0x7f2d4c15afa0 2023-10-30T14:26:22.099 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:22.100 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":5797,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:24:15.976636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:22.100 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 72 2023-10-30T14:26:22.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:22.100+0000 7f2d53854640 1 -- 172.21.15.57:0/1622284739 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2d300664d0 msgr2=0x7f2d30068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:22.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:22.100+0000 7f2d53854640 1 --2- 172.21.15.57:0/1622284739 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2d300664d0 0x7f2d30068990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2d480af000 tx=0x7f2d480046d0 comp rx=0 tx=0).stop 2023-10-30T14:26:22.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:22.100+0000 7f2d53854640 1 -- 172.21.15.57:0/1622284739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d4c15afa0 msgr2=0x7f2d4c0c3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:22.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:22.100+0000 7f2d53854640 1 --2- 172.21.15.57:0/1622284739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d4c15afa0 0x7f2d4c0c3b50 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c00f7f0 tx=0x7f2d3c00fcb0 comp rx=0 tx=0).stop 2023-10-30T14:26:22.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:22.100+0000 7f2d53854640 1 -- 172.21.15.57:0/1622284739 shutdown_connections 2023-10-30T14:26:22.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:22.100+0000 7f2d53854640 1 --2- 172.21.15.57:0/1622284739 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2d300664d0 0x7f2d30068990 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:22.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:22.100+0000 7f2d53854640 1 --2- 172.21.15.57:0/1622284739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d4c14bdd0 0x7f2d4c14c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:22.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:22.100+0000 7f2d53854640 1 --2- 172.21.15.57:0/1622284739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d4c15afa0 0x7f2d4c0c3b50 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:22.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:22.100+0000 7f2d53854640 1 --2- 172.21.15.57:0/1622284739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d38003980 0x7f2d4c1595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:22.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:22.100+0000 7f2d53854640 1 -- 172.21.15.57:0/1622284739 >> 172.21.15.57:0/1622284739 conn(0x7f2d4c0cac70 msgr2=0x7f2d4c152800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:22.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:22.100+0000 7f2d53854640 1 -- 172.21.15.57:0/1622284739 shutdown_connections 2023-10-30T14:26:22.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:22.100+0000 7f2d53854640 1 -- 172.21.15.57:0/1622284739 wait complete. 2023-10-30T14:26:22.121 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:26:24.122 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:24.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.276+0000 7f42df7f8640 1 Processor -- start 2023-10-30T14:26:24.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.276+0000 7f42df7f8640 1 -- start start 2023-10-30T14:26:24.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.276+0000 7f42df7f8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f42d80c4a10 0x7f42d815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:24.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.276+0000 7f42df7f8640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f42d80c42a0 0x7f42d80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:24.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.276+0000 7f42df7f8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d8159f20 con 0x7f42d8153c80 2023-10-30T14:26:24.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.276+0000 7f42df7f8640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42d815dec0 con 0x7f42d80c42a0 2023-10-30T14:26:24.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.276+0000 7f42df7f8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42d81599a0 con 0x7f42d80c4a10 2023-10-30T14:26:24.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.276+0000 7f42ddd95640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f42d80c42a0 0x7f42d80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:24.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.276+0000 7f42ddd95640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f42d80c42a0 0x7f42d80ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39988/0 (socket says 172.21.15.57:39988) 2023-10-30T14:26:24.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.276+0000 7f42ddd95640 1 -- 172.21.15.57:0/3598030145 learned_addr learned my addr 172.21.15.57:0/3598030145 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:24.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.280+0000 7f42dcd93640 1 --2- 172.21.15.57:0/3598030145 >> v2:172.21.15.88:3300/0 conn(0x7f42d80c4a10 0x7f42d815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:24.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.280+0000 7f42ddd95640 1 -- 172.21.15.57:0/3598030145 >> v2:172.21.15.88:3300/0 conn(0x7f42d80c4a10 msgr2=0x7f42d815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:24.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.280+0000 7f42ddd95640 1 --2- 172.21.15.57:0/3598030145 >> v2:172.21.15.88:3300/0 conn(0x7f42d80c4a10 0x7f42d815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:24.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.280+0000 7f42ddd95640 1 -- 172.21.15.57:0/3598030145 >> v1:172.21.15.88:6789/0 conn(0x7f42d8153c80 legacy=0x7f42d815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:24.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.280+0000 7f42ddd95640 1 -- 172.21.15.57:0/3598030145 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d815d620 con 0x7f42d80c42a0 2023-10-30T14:26:24.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.280+0000 7f42ddd95640 1 --2- 172.21.15.57:0/3598030145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42d80c42a0 0x7f42d80ca1e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f42d806be50 tx=0x7f42d001f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38e1e70e8a89f356 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:24.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.280+0000 7f42cffff640 1 -- 172.21.15.57:0/3598030145 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d00af400 con 0x7f42d80c42a0 2023-10-30T14:26:24.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.280+0000 7f42cffff640 1 -- 172.21.15.57:0/3598030145 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d00ab050 con 0x7f42d80c42a0 2023-10-30T14:26:24.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.280+0000 7f42cffff640 1 -- 172.21.15.57:0/3598030145 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d00af780 con 0x7f42d80c42a0 2023-10-30T14:26:24.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42df7f8640 1 -- 172.21.15.57:0/3598030145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42d80c42a0 msgr2=0x7f42d80ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:24.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42df7f8640 1 --2- 172.21.15.57:0/3598030145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42d80c42a0 0x7f42d80ca1e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f42d806be50 tx=0x7f42d001f180 comp rx=0 tx=0).stop 2023-10-30T14:26:24.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42df7f8640 1 -- 172.21.15.57:0/3598030145 shutdown_connections 2023-10-30T14:26:24.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42df7f8640 1 --2- 172.21.15.57:0/3598030145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42d80c42a0 0x7f42d80ca1e0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:24.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42df7f8640 1 --2- 172.21.15.57:0/3598030145 >> v2:172.21.15.88:3300/0 conn(0x7f42d80c4a10 0x7f42d815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:24.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42df7f8640 1 -- 172.21.15.57:0/3598030145 >> 172.21.15.57:0/3598030145 conn(0x7f42d80715c0 msgr2=0x7f42d80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:24.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42df7f8640 1 -- 172.21.15.57:0/3598030145 shutdown_connections 2023-10-30T14:26:24.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42df7f8640 1 -- 172.21.15.57:0/3598030145 wait complete. 2023-10-30T14:26:24.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42df7f8640 1 Processor -- start 2023-10-30T14:26:24.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42df7f8640 1 -- start start 2023-10-30T14:26:24.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42df7f8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42d80c42a0 0x7f42d81ef480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:24.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42df7f8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42d80c4a10 0x7f42d815c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:24.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42df7f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42d8153c80 0x7f42d80cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:24.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42df7f8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d80cf170 con 0x7f42d80c42a0 2023-10-30T14:26:24.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42df7f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d81eeff0 con 0x7f42d8153c80 2023-10-30T14:26:24.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42df7f8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d81f2940 con 0x7f42d80c4a10 2023-10-30T14:26:24.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42dd594640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42d80c42a0 0x7f42d81ef480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:24.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42dd594640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42d80c42a0 0x7f42d81ef480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39996/0 (socket says 172.21.15.57:39996) 2023-10-30T14:26:24.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42dd594640 1 -- 172.21.15.57:0/3534945535 learned_addr learned my addr 172.21.15.57:0/3534945535 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:24.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42dcd93640 1 --2- 172.21.15.57:0/3534945535 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42d80c4a10 0x7f42d815c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:24.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42ddd95640 1 --2- 172.21.15.57:0/3534945535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42d8153c80 0x7f42d80cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:24.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42dd594640 1 -- 172.21.15.57:0/3534945535 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42d80c4a10 msgr2=0x7f42d815c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:24.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42dd594640 1 --2- 172.21.15.57:0/3534945535 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42d80c4a10 0x7f42d815c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:24.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42dd594640 1 -- 172.21.15.57:0/3534945535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42d8153c80 msgr2=0x7f42d80cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:24.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42dd594640 1 --2- 172.21.15.57:0/3534945535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42d8153c80 0x7f42d80cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:24.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42dd594640 1 -- 172.21.15.57:0/3534945535 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d0009b30 con 0x7f42d80c42a0 2023-10-30T14:26:24.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42ddd95640 1 --2- 172.21.15.57:0/3534945535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42d8153c80 0x7f42d80cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:24.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.284+0000 7f42dcd93640 1 --2- 172.21.15.57:0/3534945535 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42d80c4a10 0x7f42d815c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:24.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.288+0000 7f42dd594640 1 --2- 172.21.15.57:0/3534945535 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42d80c42a0 0x7f42d81ef480 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f42d400b410 tx=0x7f42d400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:24.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.288+0000 7f42cdffb640 1 -- 172.21.15.57:0/3534945535 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d4004020 con 0x7f42d80c42a0 2023-10-30T14:26:24.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.288+0000 7f42cdffb640 1 -- 172.21.15.57:0/3534945535 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d4004180 con 0x7f42d80c42a0 2023-10-30T14:26:24.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.288+0000 7f42cdffb640 1 -- 172.21.15.57:0/3534945535 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d4011630 con 0x7f42d80c42a0 2023-10-30T14:26:24.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.288+0000 7f42df7f8640 1 -- 172.21.15.57:0/3534945535 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d81ee9a0 con 0x7f42d80c42a0 2023-10-30T14:26:24.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.288+0000 7f42df7f8640 1 -- 172.21.15.57:0/3534945535 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d8071e30 con 0x7f42d80c42a0 2023-10-30T14:26:24.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.288+0000 7f42df7f8640 1 -- 172.21.15.57:0/3534945535 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42d804e170 con 0x7f42d80c42a0 2023-10-30T14:26:24.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.288+0000 7f42cdffb640 1 -- 172.21.15.57:0/3534945535 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f42d4011790 con 0x7f42d80c42a0 2023-10-30T14:26:24.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.288+0000 7f42cdffb640 1 --2- 172.21.15.57:0/3534945535 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f42bc0664a0 0x7f42bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:24.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.288+0000 7f42cdffb640 1 -- 172.21.15.57:0/3534945535 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6341+0+0 (secure 0 0 0) 0x7f42d4083f60 con 0x7f42d80c42a0 2023-10-30T14:26:24.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.292+0000 7f42dcd93640 1 --2- 172.21.15.57:0/3534945535 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f42bc0664a0 0x7f42bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:24.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.292+0000 7f42dcd93640 1 --2- 172.21.15.57:0/3534945535 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f42bc0664a0 0x7f42bc068960 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f42d80c4e30 tx=0x7f42c8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:24.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.292+0000 7f42cdffb640 1 -- 172.21.15.57:0/3534945535 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f42d40510c0 con 0x7f42d80c42a0 2023-10-30T14:26:24.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.484+0000 7f42df7f8640 1 -- 172.21.15.57:0/3534945535 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42d80cb9d0 con 0x7f42d80c42a0 2023-10-30T14:26:24.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.484+0000 7f42cdffb640 1 -- 172.21.15.57:0/3534945535 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6507 (secure 0 0 0) 0x7f42d40546e0 con 0x7f42d80c42a0 2023-10-30T14:26:24.487 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:24.488 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":5797,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:24:15.976636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5289,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5289":{"gid":5289,"name":"c","rank":0,"incarnation":35,"state":"up:active","state_seq":21,"addr":"172.21.15.57:6836/2427081813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2427081813},{"type":"v1","addr":"172.21.15.57:6836","nonce":2427081813}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:24.488 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 72 2023-10-30T14:26:24.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.488+0000 7f42df7f8640 1 -- 172.21.15.57:0/3534945535 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f42bc0664a0 msgr2=0x7f42bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:24.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.488+0000 7f42df7f8640 1 --2- 172.21.15.57:0/3534945535 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f42bc0664a0 0x7f42bc068960 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f42d80c4e30 tx=0x7f42c8008040 comp rx=0 tx=0).stop 2023-10-30T14:26:24.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.488+0000 7f42df7f8640 1 -- 172.21.15.57:0/3534945535 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42d80c42a0 msgr2=0x7f42d81ef480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:24.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.488+0000 7f42df7f8640 1 --2- 172.21.15.57:0/3534945535 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42d80c42a0 0x7f42d81ef480 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f42d400b410 tx=0x7f42d400b8d0 comp rx=0 tx=0).stop 2023-10-30T14:26:24.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.488+0000 7f42df7f8640 1 -- 172.21.15.57:0/3534945535 shutdown_connections 2023-10-30T14:26:24.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.488+0000 7f42df7f8640 1 --2- 172.21.15.57:0/3534945535 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f42bc0664a0 0x7f42bc068960 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:24.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.488+0000 7f42df7f8640 1 --2- 172.21.15.57:0/3534945535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42d8153c80 0x7f42d80cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:24.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.488+0000 7f42df7f8640 1 --2- 172.21.15.57:0/3534945535 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42d80c4a10 0x7f42d815c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:24.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.488+0000 7f42df7f8640 1 --2- 172.21.15.57:0/3534945535 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42d80c42a0 0x7f42d81ef480 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:24.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.488+0000 7f42df7f8640 1 -- 172.21.15.57:0/3534945535 >> 172.21.15.57:0/3534945535 conn(0x7f42d80715c0 msgr2=0x7f42d806bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:24.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.488+0000 7f42df7f8640 1 -- 172.21.15.57:0/3534945535 shutdown_connections 2023-10-30T14:26:24.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:24.488+0000 7f42df7f8640 1 -- 172.21.15.57:0/3534945535 wait complete. 2023-10-30T14:26:24.509 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:26:26.511 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:26.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.664+0000 7f8b0d5aa640 1 Processor -- start 2023-10-30T14:26:26.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.664+0000 7f8b0d5aa640 1 -- start start 2023-10-30T14:26:26.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.664+0000 7f8b0d5aa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8b080cb7d0 0x7f8b080c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:26.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.664+0000 7f8b0d5aa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b080c3e40 0x7f8b08070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:26.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.664+0000 7f8b0d5aa640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b080c2310 con 0x7f8b08150830 2023-10-30T14:26:26.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.664+0000 7f8b0d5aa640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b0806bc10 con 0x7f8b080cb7d0 2023-10-30T14:26:26.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.664+0000 7f8b0d5aa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b08156710 con 0x7f8b080c3e40 2023-10-30T14:26:26.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.664+0000 7f8b06ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f8b08150830 0x7f8b0806b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:40188/0 (socket says 172.21.15.57:40188) 2023-10-30T14:26:26.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.664+0000 7f8b06ffd640 1 -- 172.21.15.57:0/850886718 learned_addr learned my addr 172.21.15.57:0/850886718 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:26:26.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.664+0000 7f8b077fe640 1 --2- 172.21.15.57:0/850886718 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b080c3e40 0x7f8b08070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:26.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.664+0000 7f8afffff640 1 --2- 172.21.15.57:0/850886718 >> v2:172.21.15.88:3300/0 conn(0x7f8b080cb7d0 0x7f8b080c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:26.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.664+0000 7f8b067fc640 1 -- 172.21.15.57:0/850886718 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485778730 0 0) 0x7f8b080c2310 con 0x7f8b08150830 2023-10-30T14:26:26.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.664+0000 7f8b067fc640 1 -- 172.21.15.57:0/850886718 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af4003530 con 0x7f8b08150830 2023-10-30T14:26:26.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.664+0000 7f8b077fe640 1 -- 172.21.15.57:0/850886718 >> v2:172.21.15.88:3300/0 conn(0x7f8b080cb7d0 msgr2=0x7f8b080c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:26.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.664+0000 7f8b077fe640 1 --2- 172.21.15.57:0/850886718 >> v2:172.21.15.88:3300/0 conn(0x7f8b080cb7d0 0x7f8b080c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:26.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.664+0000 7f8b067fc640 1 -- 172.21.15.57:0/850886718 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2394948531 0 0) 0x7f8af4003530 con 0x7f8b08150830 2023-10-30T14:26:26.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.664+0000 7f8b077fe640 1 -- 172.21.15.57:0/850886718 >> v1:172.21.15.57:6789/0 conn(0x7f8b08150830 legacy=0x7f8b0806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:26.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.664+0000 7f8b077fe640 1 -- 172.21.15.57:0/850886718 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b080c4360 con 0x7f8b080c3e40 2023-10-30T14:26:26.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.668+0000 7f8b077fe640 1 --2- 172.21.15.57:0/850886718 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b080c3e40 0x7f8b08070520 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8b00009d90 tx=0x7f8b0001f060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a17b72d95976f08e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:26.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.668+0000 7f8b067fc640 1 -- 172.21.15.57:0/850886718 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af8001680 con 0x7f8b080c3e40 2023-10-30T14:26:26.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.668+0000 7f8b067fc640 1 -- 172.21.15.57:0/850886718 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b000a8bb0 con 0x7f8b080c3e40 2023-10-30T14:26:26.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.668+0000 7f8b067fc640 1 -- 172.21.15.57:0/850886718 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b000b0440 con 0x7f8b080c3e40 2023-10-30T14:26:26.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.668+0000 7f8b0d5aa640 1 -- 172.21.15.57:0/850886718 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b080c3e40 msgr2=0x7f8b08070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:26.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.668+0000 7f8b0d5aa640 1 --2- 172.21.15.57:0/850886718 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b080c3e40 0x7f8b08070520 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8b00009d90 tx=0x7f8b0001f060 comp rx=0 tx=0).stop 2023-10-30T14:26:26.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.668+0000 7f8b0d5aa640 1 -- 172.21.15.57:0/850886718 shutdown_connections 2023-10-30T14:26:26.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.668+0000 7f8b0d5aa640 1 --2- 172.21.15.57:0/850886718 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b080c3e40 0x7f8b08070520 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:26.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.668+0000 7f8b0d5aa640 1 --2- 172.21.15.57:0/850886718 >> v2:172.21.15.88:3300/0 conn(0x7f8b080cb7d0 0x7f8b080c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:26.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.668+0000 7f8b0d5aa640 1 -- 172.21.15.57:0/850886718 >> 172.21.15.57:0/850886718 conn(0x7f8b080cac70 msgr2=0x7f8b080cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:26.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.668+0000 7f8b0d5aa640 1 -- 172.21.15.57:0/850886718 shutdown_connections 2023-10-30T14:26:26.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.668+0000 7f8b0d5aa640 1 -- 172.21.15.57:0/850886718 wait complete. 2023-10-30T14:26:26.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b0d5aa640 1 Processor -- start 2023-10-30T14:26:26.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b0d5aa640 1 -- start start 2023-10-30T14:26:26.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b0d5aa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b080c3e40 0x7f8b0806bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:26.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b0d5aa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b080cb7d0 0x7f8b08144980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:26.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b0d5aa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b08150830 0x7f8b08144ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:26.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b0d5aa640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b08145400 con 0x7f8b080c3e40 2023-10-30T14:26:26.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b0d5aa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b081f1c50 con 0x7f8b080cb7d0 2023-10-30T14:26:26.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b0d5aa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0814b2c0 con 0x7f8b08150830 2023-10-30T14:26:26.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b06ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b080c3e40 0x7f8b0806bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:26.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b06ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b080c3e40 0x7f8b0806bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40006/0 (socket says 172.21.15.57:40006) 2023-10-30T14:26:26.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b06ffd640 1 -- 172.21.15.57:0/1634238004 learned_addr learned my addr 172.21.15.57:0/1634238004 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:26.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b077fe640 1 --2- 172.21.15.57:0/1634238004 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b08150830 0x7f8b08144ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:26.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8afffff640 1 --2- 172.21.15.57:0/1634238004 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b080cb7d0 0x7f8b08144980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:26.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b06ffd640 1 -- 172.21.15.57:0/1634238004 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b08150830 msgr2=0x7f8b08144ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:26.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b06ffd640 1 --2- 172.21.15.57:0/1634238004 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b08150830 0x7f8b08144ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:26.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b06ffd640 1 -- 172.21.15.57:0/1634238004 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b080cb7d0 msgr2=0x7f8b08144980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:26.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b06ffd640 1 --2- 172.21.15.57:0/1634238004 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b080cb7d0 0x7f8b08144980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:26.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b06ffd640 1 -- 172.21.15.57:0/1634238004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b00009a70 con 0x7f8b080c3e40 2023-10-30T14:26:26.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b077fe640 1 --2- 172.21.15.57:0/1634238004 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b08150830 0x7f8b08144ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:26.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8afffff640 1 --2- 172.21.15.57:0/1634238004 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b080cb7d0 0x7f8b08144980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:26.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b06ffd640 1 --2- 172.21.15.57:0/1634238004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b080c3e40 0x7f8b0806bfb0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8af800c750 tx=0x7f8af800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:26.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8aff7fe640 1 -- 172.21.15.57:0/1634238004 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af8005ae0 con 0x7f8b080c3e40 2023-10-30T14:26:26.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8aff7fe640 1 -- 172.21.15.57:0/1634238004 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8af8005c40 con 0x7f8b080c3e40 2023-10-30T14:26:26.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b0d5aa640 1 -- 172.21.15.57:0/1634238004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b0814b5a0 con 0x7f8b080c3e40 2023-10-30T14:26:26.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8aff7fe640 1 -- 172.21.15.57:0/1634238004 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af8007070 con 0x7f8b080c3e40 2023-10-30T14:26:26.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b0d5aa640 1 -- 172.21.15.57:0/1634238004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b0814bb60 con 0x7f8b080c3e40 2023-10-30T14:26:26.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.672+0000 7f8b0d5aa640 1 -- 172.21.15.57:0/1634238004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b0804e170 con 0x7f8b080c3e40 2023-10-30T14:26:26.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.676+0000 7f8aff7fe640 1 -- 172.21.15.57:0/1634238004 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8af80071f0 con 0x7f8b080c3e40 2023-10-30T14:26:26.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.676+0000 7f8aff7fe640 1 --2- 172.21.15.57:0/1634238004 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8ae80664a0 0x7f8ae8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:26.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.676+0000 7f8aff7fe640 1 -- 172.21.15.57:0/1634238004 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f8af8084000 con 0x7f8b080c3e40 2023-10-30T14:26:26.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.676+0000 7f8afffff640 1 --2- 172.21.15.57:0/1634238004 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8ae80664a0 0x7f8ae8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:26.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.676+0000 7f8afffff640 1 --2- 172.21.15.57:0/1634238004 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8ae80664a0 0x7f8ae8068960 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8b08073910 tx=0x7f8af0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:26.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.680+0000 7f8aff7fe640 1 -- 172.21.15.57:0/1634238004 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8af8051110 con 0x7f8b080c3e40 2023-10-30T14:26:26.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.868+0000 7f8b0d5aa640 1 -- 172.21.15.57:0/1634238004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b08070300 con 0x7f8b080c3e40 2023-10-30T14:26:26.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.868+0000 7f8aff7fe640 1 -- 172.21.15.57:0/1634238004 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5714 (secure 0 0 0) 0x7f8af8054730 con 0x7f8b080c3e40 2023-10-30T14:26:26.871 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:26.872 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:26:24.805500+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:26.872 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 73 2023-10-30T14:26:26.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.872+0000 7f8b0d5aa640 1 -- 172.21.15.57:0/1634238004 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8ae80664a0 msgr2=0x7f8ae8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:26.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.872+0000 7f8b0d5aa640 1 --2- 172.21.15.57:0/1634238004 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8ae80664a0 0x7f8ae8068960 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8b08073910 tx=0x7f8af0008040 comp rx=0 tx=0).stop 2023-10-30T14:26:26.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.872+0000 7f8b0d5aa640 1 -- 172.21.15.57:0/1634238004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b080c3e40 msgr2=0x7f8b0806bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:26.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.872+0000 7f8b0d5aa640 1 --2- 172.21.15.57:0/1634238004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b080c3e40 0x7f8b0806bfb0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8af800c750 tx=0x7f8af800cc10 comp rx=0 tx=0).stop 2023-10-30T14:26:26.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.872+0000 7f8b0d5aa640 1 -- 172.21.15.57:0/1634238004 shutdown_connections 2023-10-30T14:26:26.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.872+0000 7f8b0d5aa640 1 --2- 172.21.15.57:0/1634238004 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8ae80664a0 0x7f8ae8068960 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:26.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.872+0000 7f8b0d5aa640 1 --2- 172.21.15.57:0/1634238004 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b08150830 0x7f8b08144ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:26.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.872+0000 7f8b0d5aa640 1 --2- 172.21.15.57:0/1634238004 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b080cb7d0 0x7f8b08144980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:26.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.872+0000 7f8b0d5aa640 1 --2- 172.21.15.57:0/1634238004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b080c3e40 0x7f8b0806bfb0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:26.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.872+0000 7f8b0d5aa640 1 -- 172.21.15.57:0/1634238004 >> 172.21.15.57:0/1634238004 conn(0x7f8b080cac70 msgr2=0x7f8b0814d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:26.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.872+0000 7f8b0d5aa640 1 -- 172.21.15.57:0/1634238004 shutdown_connections 2023-10-30T14:26:26.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:26.872+0000 7f8b0d5aa640 1 -- 172.21.15.57:0/1634238004 wait complete. 2023-10-30T14:26:26.893 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T14:26:26.893 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:26:26.894 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:27.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.036+0000 7fe73919d640 1 Processor -- start 2023-10-30T14:26:27.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.036+0000 7fe73919d640 1 -- start start 2023-10-30T14:26:27.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.040+0000 7fe73919d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe7340ccf60 0x7fe7340cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:27.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.040+0000 7fe73919d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7340c2500 0x7fe7340c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:27.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.040+0000 7fe73919d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe734151aa0 con 0x7fe7341516c0 2023-10-30T14:26:27.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.040+0000 7fe73919d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7340c74a0 con 0x7fe7340ccf60 2023-10-30T14:26:27.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.040+0000 7fe73919d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7340c2e70 con 0x7fe7340c2500 2023-10-30T14:26:27.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.040+0000 7fe72bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe7340ccf60 0x7fe7340cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:27.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.040+0000 7fe72bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe7340ccf60 0x7fe7340cd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40008/0 (socket says 172.21.15.57:40008) 2023-10-30T14:26:27.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.040+0000 7fe72bfff640 1 -- 172.21.15.57:0/922122513 learned_addr learned my addr 172.21.15.57:0/922122513 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:27.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.040+0000 7fe73359e640 1 --2- 172.21.15.57:0/922122513 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7340c2500 0x7fe7340c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:27.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.040+0000 7fe72bfff640 1 -- 172.21.15.57:0/922122513 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7340c2500 msgr2=0x7fe7340c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:27.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.040+0000 7fe72bfff640 1 --2- 172.21.15.57:0/922122513 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7340c2500 0x7fe7340c2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:27.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.040+0000 7fe72bfff640 1 -- 172.21.15.57:0/922122513 >> v1:172.21.15.88:6789/0 conn(0x7fe7341516c0 legacy=0x7fe7340c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:27.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.040+0000 7fe72bfff640 1 -- 172.21.15.57:0/922122513 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe734158af0 con 0x7fe7340ccf60 2023-10-30T14:26:27.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.044+0000 7fe72bfff640 1 --2- 172.21.15.57:0/922122513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe7340ccf60 0x7fe7340cd3a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe71c009870 tx=0x7fe71c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed6d6c071fec94b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:27.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.044+0000 7fe73259c640 1 -- 172.21.15.57:0/922122513 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c0ab600 con 0x7fe7340ccf60 2023-10-30T14:26:27.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.044+0000 7fe73259c640 1 -- 172.21.15.57:0/922122513 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe71c0a7050 con 0x7fe7340ccf60 2023-10-30T14:26:27.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.044+0000 7fe73259c640 1 -- 172.21.15.57:0/922122513 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c0ab980 con 0x7fe7340ccf60 2023-10-30T14:26:27.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.044+0000 7fe73919d640 1 -- 172.21.15.57:0/922122513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe7340ccf60 msgr2=0x7fe7340cd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:27.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.044+0000 7fe73919d640 1 --2- 172.21.15.57:0/922122513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe7340ccf60 0x7fe7340cd3a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe71c009870 tx=0x7fe71c01c540 comp rx=0 tx=0).stop 2023-10-30T14:26:27.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.044+0000 7fe73919d640 1 -- 172.21.15.57:0/922122513 shutdown_connections 2023-10-30T14:26:27.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.044+0000 7fe73919d640 1 --2- 172.21.15.57:0/922122513 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7340c2500 0x7fe7340c2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:27.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.044+0000 7fe73919d640 1 --2- 172.21.15.57:0/922122513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe7340ccf60 0x7fe7340cd3a0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:27.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.044+0000 7fe73919d640 1 -- 172.21.15.57:0/922122513 >> 172.21.15.57:0/922122513 conn(0x7fe734071fb0 msgr2=0x7fe7340723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:27.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.044+0000 7fe73919d640 1 -- 172.21.15.57:0/922122513 shutdown_connections 2023-10-30T14:26:27.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.044+0000 7fe73919d640 1 -- 172.21.15.57:0/922122513 wait complete. 2023-10-30T14:26:27.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.044+0000 7fe73919d640 1 Processor -- start 2023-10-30T14:26:27.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.044+0000 7fe73919d640 1 -- start start 2023-10-30T14:26:27.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe73919d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe7340c2500 0x7fe734150c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:27.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe73919d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7340ccf60 0x7fe73414ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:27.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe73919d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7341516c0 0x7fe7341f1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:27.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe73919d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7341f1990 con 0x7fe7340c2500 2023-10-30T14:26:27.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe73919d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73406afa0 con 0x7fe7340ccf60 2023-10-30T14:26:27.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe73919d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7341f59b0 con 0x7fe7341516c0 2023-10-30T14:26:27.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe732d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe7340c2500 0x7fe734150c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:27.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe732d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe7340c2500 0x7fe734150c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40014/0 (socket says 172.21.15.57:40014) 2023-10-30T14:26:27.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe732d9d640 1 -- 172.21.15.57:0/2061979889 learned_addr learned my addr 172.21.15.57:0/2061979889 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:27.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe72bfff640 1 --2- 172.21.15.57:0/2061979889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7340ccf60 0x7fe73414ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:27.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe73359e640 1 --2- 172.21.15.57:0/2061979889 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7341516c0 0x7fe7341f1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:27.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe732d9d640 1 -- 172.21.15.57:0/2061979889 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7341516c0 msgr2=0x7fe7341f1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:27.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe732d9d640 1 --2- 172.21.15.57:0/2061979889 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7341516c0 0x7fe7341f1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:27.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe732d9d640 1 -- 172.21.15.57:0/2061979889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7340ccf60 msgr2=0x7fe73414ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:27.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe732d9d640 1 --2- 172.21.15.57:0/2061979889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7340ccf60 0x7fe73414ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:27.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe732d9d640 1 -- 172.21.15.57:0/2061979889 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe71c0094d0 con 0x7fe7340c2500 2023-10-30T14:26:27.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe73359e640 1 --2- 172.21.15.57:0/2061979889 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7341516c0 0x7fe7341f1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:27.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe732d9d640 1 --2- 172.21.15.57:0/2061979889 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe7340c2500 0x7fe734150c90 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe72400c500 tx=0x7fe72400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:27.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe72b7fe640 1 -- 172.21.15.57:0/2061979889 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe724005220 con 0x7fe7340c2500 2023-10-30T14:26:27.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe72b7fe640 1 -- 172.21.15.57:0/2061979889 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe724005380 con 0x7fe7340c2500 2023-10-30T14:26:27.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe72b7fe640 1 -- 172.21.15.57:0/2061979889 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe724007270 con 0x7fe7340c2500 2023-10-30T14:26:27.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe73919d640 1 -- 172.21.15.57:0/2061979889 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7341f5c90 con 0x7fe7340c2500 2023-10-30T14:26:27.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe73919d640 1 -- 172.21.15.57:0/2061979889 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe73406b4d0 con 0x7fe7340c2500 2023-10-30T14:26:27.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe73919d640 1 -- 172.21.15.57:0/2061979889 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe73404e170 con 0x7fe7340c2500 2023-10-30T14:26:27.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.048+0000 7fe72b7fe640 1 -- 172.21.15.57:0/2061979889 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe7240073d0 con 0x7fe7340c2500 2023-10-30T14:26:27.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.052+0000 7fe72b7fe640 1 --2- 172.21.15.57:0/2061979889 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe7140664a0 0x7fe714068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:27.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.052+0000 7fe72b7fe640 1 -- 172.21.15.57:0/2061979889 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7fe724083a90 con 0x7fe7340c2500 2023-10-30T14:26:27.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.052+0000 7fe72bfff640 1 --2- 172.21.15.57:0/2061979889 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe7140664a0 0x7fe714068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:27.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.052+0000 7fe72bfff640 1 --2- 172.21.15.57:0/2061979889 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe7140664a0 0x7fe714068960 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe7340cdb70 tx=0x7fe71c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:27.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.052+0000 7fe72b7fe640 1 -- 172.21.15.57:0/2061979889 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe724050bb0 con 0x7fe7340c2500 2023-10-30T14:26:27.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.244+0000 7fe73919d640 1 -- 172.21.15.57:0/2061979889 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7340c7b70 con 0x7fe7340c2500 2023-10-30T14:26:27.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.244+0000 7fe72b7fe640 1 -- 172.21.15.57:0/2061979889 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5714 (secure 0 0 0) 0x7fe7240541d0 con 0x7fe7340c2500 2023-10-30T14:26:27.246 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:27.246 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:26:24.805500+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:27.247 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 73 2023-10-30T14:26:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.244+0000 7fe73919d640 1 -- 172.21.15.57:0/2061979889 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe7140664a0 msgr2=0x7fe714068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.244+0000 7fe73919d640 1 --2- 172.21.15.57:0/2061979889 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe7140664a0 0x7fe714068960 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe7340cdb70 tx=0x7fe71c008040 comp rx=0 tx=0).stop 2023-10-30T14:26:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.244+0000 7fe73919d640 1 -- 172.21.15.57:0/2061979889 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe7340c2500 msgr2=0x7fe734150c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.244+0000 7fe73919d640 1 --2- 172.21.15.57:0/2061979889 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe7340c2500 0x7fe734150c90 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe72400c500 tx=0x7fe72400c9c0 comp rx=0 tx=0).stop 2023-10-30T14:26:27.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.244+0000 7fe73919d640 1 -- 172.21.15.57:0/2061979889 shutdown_connections 2023-10-30T14:26:27.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.244+0000 7fe73919d640 1 --2- 172.21.15.57:0/2061979889 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe7140664a0 0x7fe714068960 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:27.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.244+0000 7fe73919d640 1 --2- 172.21.15.57:0/2061979889 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7341516c0 0x7fe7341f1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:27.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.244+0000 7fe73919d640 1 --2- 172.21.15.57:0/2061979889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7340ccf60 0x7fe73414ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:27.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.244+0000 7fe73919d640 1 --2- 172.21.15.57:0/2061979889 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe7340c2500 0x7fe734150c90 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:27.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.244+0000 7fe73919d640 1 -- 172.21.15.57:0/2061979889 >> 172.21.15.57:0/2061979889 conn(0x7fe734071fb0 msgr2=0x7fe734160320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:27.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.244+0000 7fe73919d640 1 -- 172.21.15.57:0/2061979889 shutdown_connections 2023-10-30T14:26:27.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:27.244+0000 7fe73919d640 1 -- 172.21.15.57:0/2061979889 wait complete. 2023-10-30T14:26:27.268 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 73, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 73, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5586": { "addr": "172.21.15.57:6837/862088802", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 862088802, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 862088802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5586, "incarnation": 67, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 51 }, "gid_5797": { "addr": "172.21.15.57:6834/2940934016", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2940934016, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2940934016, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5797, "incarnation": 73, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:26:24.805500+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5797, "mds_1": 5586 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6839/358521496", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 358521496, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 358521496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 66, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5315, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:26:29.269 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:29.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.428+0000 7fb8ff565640 1 Processor -- start 2023-10-30T14:26:29.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.428+0000 7fb8ff565640 1 -- start start 2023-10-30T14:26:29.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.428+0000 7fb8ff565640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8f80c4a10 0x7fb8f815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:29.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.428+0000 7fb8ff565640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb8f80c42a0 0x7fb8f80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:29.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.428+0000 7fb8ff565640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f8159f20 con 0x7fb8f8153c80 2023-10-30T14:26:29.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.428+0000 7fb8ff565640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f815dec0 con 0x7fb8f80c42a0 2023-10-30T14:26:29.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.428+0000 7fb8ff565640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f81599a0 con 0x7fb8f80c4a10 2023-10-30T14:26:29.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.428+0000 7fb8fdb02640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb8f80c42a0 0x7fb8f80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:29.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.428+0000 7fb8fdb02640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb8f80c42a0 0x7fb8f80ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40016/0 (socket says 172.21.15.57:40016) 2023-10-30T14:26:29.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.428+0000 7fb8fdb02640 1 -- 172.21.15.57:0/474558336 learned_addr learned my addr 172.21.15.57:0/474558336 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:29.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.428+0000 7fb8fcb00640 1 --2- 172.21.15.57:0/474558336 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8f80c4a10 0x7fb8f815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:29.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.428+0000 7fb8fdb02640 1 -- 172.21.15.57:0/474558336 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8f80c4a10 msgr2=0x7fb8f815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:29.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.428+0000 7fb8fdb02640 1 --2- 172.21.15.57:0/474558336 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8f80c4a10 0x7fb8f815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:29.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.428+0000 7fb8fdb02640 1 -- 172.21.15.57:0/474558336 >> v1:172.21.15.88:6789/0 conn(0x7fb8f8153c80 legacy=0x7fb8f815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:29.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.428+0000 7fb8fdb02640 1 -- 172.21.15.57:0/474558336 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f815d620 con 0x7fb8f80c42a0 2023-10-30T14:26:29.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.432+0000 7fb8fdb02640 1 --2- 172.21.15.57:0/474558336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8f80c42a0 0x7fb8f80ca1e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0009e50 tx=0x7fb8f001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d8e7b057425cd3f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:29.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.432+0000 7fb8effff640 1 -- 172.21.15.57:0/474558336 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f00af400 con 0x7fb8f80c42a0 2023-10-30T14:26:29.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.432+0000 7fb8effff640 1 -- 172.21.15.57:0/474558336 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f00ab050 con 0x7fb8f80c42a0 2023-10-30T14:26:29.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.432+0000 7fb8effff640 1 -- 172.21.15.57:0/474558336 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f00af780 con 0x7fb8f80c42a0 2023-10-30T14:26:29.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.432+0000 7fb8ff565640 1 -- 172.21.15.57:0/474558336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8f80c42a0 msgr2=0x7fb8f80ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:29.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.432+0000 7fb8ff565640 1 --2- 172.21.15.57:0/474558336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8f80c42a0 0x7fb8f80ca1e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0009e50 tx=0x7fb8f001f1f0 comp rx=0 tx=0).stop 2023-10-30T14:26:29.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.432+0000 7fb8ff565640 1 -- 172.21.15.57:0/474558336 shutdown_connections 2023-10-30T14:26:29.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.432+0000 7fb8ff565640 1 --2- 172.21.15.57:0/474558336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8f80c42a0 0x7fb8f80ca1e0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:29.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.432+0000 7fb8ff565640 1 --2- 172.21.15.57:0/474558336 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8f80c4a10 0x7fb8f815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:29.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.432+0000 7fb8ff565640 1 -- 172.21.15.57:0/474558336 >> 172.21.15.57:0/474558336 conn(0x7fb8f80715c0 msgr2=0x7fb8f80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:29.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.432+0000 7fb8ff565640 1 -- 172.21.15.57:0/474558336 shutdown_connections 2023-10-30T14:26:29.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.432+0000 7fb8ff565640 1 -- 172.21.15.57:0/474558336 wait complete. 2023-10-30T14:26:29.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8ff565640 1 Processor -- start 2023-10-30T14:26:29.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8ff565640 1 -- start start 2023-10-30T14:26:29.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8ff565640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8f80c42a0 0x7fb8f814ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:29.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8ff565640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8f80c4a10 0x7fb8f81542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:29.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8ff565640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8f8153c80 0x7fb8f80cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:29.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8ff565640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f81eedd0 con 0x7fb8f80c4a10 2023-10-30T14:26:29.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8ff565640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f8150700 con 0x7fb8f80c42a0 2023-10-30T14:26:29.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8ff565640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f8150270 con 0x7fb8f8153c80 2023-10-30T14:26:29.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8fcb00640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8f80c4a10 0x7fb8f81542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:29.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8fcb00640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8f80c4a10 0x7fb8f81542c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40032/0 (socket says 172.21.15.57:40032) 2023-10-30T14:26:29.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8fcb00640 1 -- 172.21.15.57:0/1852007494 learned_addr learned my addr 172.21.15.57:0/1852007494 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:29.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8fdb02640 1 --2- 172.21.15.57:0/1852007494 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8f8153c80 0x7fb8f80cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:29.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8fd301640 1 --2- 172.21.15.57:0/1852007494 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8f80c42a0 0x7fb8f814ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:29.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8fcb00640 1 -- 172.21.15.57:0/1852007494 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8f8153c80 msgr2=0x7fb8f80cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:29.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8fcb00640 1 --2- 172.21.15.57:0/1852007494 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8f8153c80 0x7fb8f80cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:29.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8fcb00640 1 -- 172.21.15.57:0/1852007494 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8f80c42a0 msgr2=0x7fb8f814ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:29.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8fcb00640 1 --2- 172.21.15.57:0/1852007494 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8f80c42a0 0x7fb8f814ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:29.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8fcb00640 1 -- 172.21.15.57:0/1852007494 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f0009b30 con 0x7fb8f80c4a10 2023-10-30T14:26:29.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8fcb00640 1 --2- 172.21.15.57:0/1852007494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8f80c4a10 0x7fb8f81542c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fb8e800d800 tx=0x7fb8e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:29.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8edffb640 1 -- 172.21.15.57:0/1852007494 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e800b6c0 con 0x7fb8f80c4a10 2023-10-30T14:26:29.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8edffb640 1 -- 172.21.15.57:0/1852007494 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e800b840 con 0x7fb8f80c4a10 2023-10-30T14:26:29.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8edffb640 1 -- 172.21.15.57:0/1852007494 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e800bbf0 con 0x7fb8f80c4a10 2023-10-30T14:26:29.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.436+0000 7fb8ff565640 1 -- 172.21.15.57:0/1852007494 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f81eefc0 con 0x7fb8f80c4a10 2023-10-30T14:26:29.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.440+0000 7fb8ff565640 1 -- 172.21.15.57:0/1852007494 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f814fc20 con 0x7fb8f80c4a10 2023-10-30T14:26:29.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.440+0000 7fb8ff565640 1 -- 172.21.15.57:0/1852007494 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f814f6d0 con 0x7fb8f80c4a10 2023-10-30T14:26:29.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.440+0000 7fb8edffb640 1 -- 172.21.15.57:0/1852007494 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb8e800bd50 con 0x7fb8f80c4a10 2023-10-30T14:26:29.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.440+0000 7fb8edffb640 1 --2- 172.21.15.57:0/1852007494 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8dc0664a0 0x7fb8dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:29.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.440+0000 7fb8edffb640 1 -- 172.21.15.57:0/1852007494 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7fb8e8084620 con 0x7fb8f80c4a10 2023-10-30T14:26:29.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.440+0000 7fb8fd301640 1 --2- 172.21.15.57:0/1852007494 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8dc0664a0 0x7fb8dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:29.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.444+0000 7fb8fd301640 1 --2- 172.21.15.57:0/1852007494 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8dc0664a0 0x7fb8dc068960 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4007fd0 tx=0x7fb8f4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:29.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.444+0000 7fb8edffb640 1 -- 172.21.15.57:0/1852007494 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb8e80517e0 con 0x7fb8f80c4a10 2023-10-30T14:26:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.648+0000 7fb8ff565640 1 -- 172.21.15.57:0/1852007494 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8f804e170 con 0x7fb8f80c4a10 2023-10-30T14:26:29.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.648+0000 7fb8edffb640 1 -- 172.21.15.57:0/1852007494 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5714 (secure 0 0 0) 0x7fb8e8054e00 con 0x7fb8f80c4a10 2023-10-30T14:26:29.653 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:29.653 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:26:24.805500+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:29.654 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 73 2023-10-30T14:26:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.652+0000 7fb8ff565640 1 -- 172.21.15.57:0/1852007494 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8dc0664a0 msgr2=0x7fb8dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.652+0000 7fb8ff565640 1 --2- 172.21.15.57:0/1852007494 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8dc0664a0 0x7fb8dc068960 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4007fd0 tx=0x7fb8f4009040 comp rx=0 tx=0).stop 2023-10-30T14:26:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.652+0000 7fb8ff565640 1 -- 172.21.15.57:0/1852007494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8f80c4a10 msgr2=0x7fb8f81542c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.652+0000 7fb8ff565640 1 --2- 172.21.15.57:0/1852007494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8f80c4a10 0x7fb8f81542c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fb8e800d800 tx=0x7fb8e800dcc0 comp rx=0 tx=0).stop 2023-10-30T14:26:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.652+0000 7fb8ff565640 1 -- 172.21.15.57:0/1852007494 shutdown_connections 2023-10-30T14:26:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.652+0000 7fb8ff565640 1 --2- 172.21.15.57:0/1852007494 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8dc0664a0 0x7fb8dc068960 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.652+0000 7fb8ff565640 1 --2- 172.21.15.57:0/1852007494 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8f8153c80 0x7fb8f80cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.652+0000 7fb8ff565640 1 --2- 172.21.15.57:0/1852007494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8f80c4a10 0x7fb8f81542c0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.652+0000 7fb8ff565640 1 --2- 172.21.15.57:0/1852007494 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8f80c42a0 0x7fb8f814ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:29.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.652+0000 7fb8ff565640 1 -- 172.21.15.57:0/1852007494 >> 172.21.15.57:0/1852007494 conn(0x7fb8f80715c0 msgr2=0x7fb8f8152580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:29.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.652+0000 7fb8ff565640 1 -- 172.21.15.57:0/1852007494 shutdown_connections 2023-10-30T14:26:29.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:29.652+0000 7fb8ff565640 1 -- 172.21.15.57:0/1852007494 wait complete. 2023-10-30T14:26:29.675 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:26:31.677 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:31.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.815+0000 7f7621756640 1 Processor -- start 2023-10-30T14:26:31.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.819+0000 7f7621756640 1 -- start start 2023-10-30T14:26:31.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.819+0000 7f7621756640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f761c0c8eb0 0x7f761c15cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:31.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.819+0000 7f7621756640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f761c15ac70 0x7f761c15a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:31.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.819+0000 7f7621756640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c15dcc0 con 0x7f761c06aeb0 2023-10-30T14:26:31.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.819+0000 7f7621756640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f761c159800 con 0x7f761c15ac70 2023-10-30T14:26:31.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.819+0000 7f7621756640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f761c159d80 con 0x7f761c0c8eb0 2023-10-30T14:26:31.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.819+0000 7f761b7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f761c15ac70 0x7f761c15a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:31.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.819+0000 7f761b7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f761c15ac70 0x7f761c15a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40040/0 (socket says 172.21.15.57:40040) 2023-10-30T14:26:31.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.819+0000 7f761bfff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f761c0c8eb0 0x7f761c15cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:31.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.819+0000 7f761b7fe640 1 -- 172.21.15.57:0/3427153960 learned_addr learned my addr 172.21.15.57:0/3427153960 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:31.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.819+0000 7f761b7fe640 1 -- 172.21.15.57:0/3427153960 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f761c0c8eb0 msgr2=0x7f761c15cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:31.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.819+0000 7f761b7fe640 1 --2- 172.21.15.57:0/3427153960 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f761c0c8eb0 0x7f761c15cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:31.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.819+0000 7f761b7fe640 1 -- 172.21.15.57:0/3427153960 >> v1:172.21.15.88:6789/0 conn(0x7f761c06aeb0 legacy=0x7f761c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:31.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.819+0000 7f761b7fe640 1 -- 172.21.15.57:0/3427153960 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c15a750 con 0x7f761c15ac70 2023-10-30T14:26:31.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.827+0000 7f761b7fe640 1 --2- 172.21.15.57:0/3427153960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f761c15ac70 0x7f761c15a210 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7610009870 tx=0x7f761001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6671e84666cae935 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:31.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.827+0000 7f761affd640 1 -- 172.21.15.57:0/3427153960 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76100a8a30 con 0x7f761c15ac70 2023-10-30T14:26:31.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.827+0000 7f761affd640 1 -- 172.21.15.57:0/3427153960 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76100a8b90 con 0x7f761c15ac70 2023-10-30T14:26:31.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.827+0000 7f761affd640 1 -- 172.21.15.57:0/3427153960 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76100b0440 con 0x7f761c15ac70 2023-10-30T14:26:31.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.827+0000 7f7621756640 1 -- 172.21.15.57:0/3427153960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f761c15ac70 msgr2=0x7f761c15a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:31.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.827+0000 7f7621756640 1 --2- 172.21.15.57:0/3427153960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f761c15ac70 0x7f761c15a210 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7610009870 tx=0x7f761001ed30 comp rx=0 tx=0).stop 2023-10-30T14:26:31.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.827+0000 7f7621756640 1 -- 172.21.15.57:0/3427153960 shutdown_connections 2023-10-30T14:26:31.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.827+0000 7f7621756640 1 --2- 172.21.15.57:0/3427153960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f761c15ac70 0x7f761c15a210 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:31.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.827+0000 7f7621756640 1 --2- 172.21.15.57:0/3427153960 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f761c0c8eb0 0x7f761c15cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:31.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.827+0000 7f7621756640 1 -- 172.21.15.57:0/3427153960 >> 172.21.15.57:0/3427153960 conn(0x7f761c0cef00 msgr2=0x7f761c0cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:31.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.827+0000 7f7621756640 1 -- 172.21.15.57:0/3427153960 shutdown_connections 2023-10-30T14:26:31.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.827+0000 7f7621756640 1 -- 172.21.15.57:0/3427153960 wait complete. 2023-10-30T14:26:31.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.827+0000 7f7621756640 1 Processor -- start 2023-10-30T14:26:31.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.827+0000 7f7621756640 1 -- start start 2023-10-30T14:26:31.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f7621756640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f761c06aeb0 0x7f761c1e4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:31.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f7621756640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f761c0c8eb0 0x7f761c1e6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:31.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f7621756640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f761c15ac70 0x7f761c1e66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:31.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f7621756640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761c1e5770 con 0x7f761c0c8eb0 2023-10-30T14:26:31.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f7621756640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761c1e55d0 con 0x7f761c06aeb0 2023-10-30T14:26:31.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f7621756640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f761c1e5cf0 con 0x7f761c15ac70 2023-10-30T14:26:31.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f761b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f761c0c8eb0 0x7f761c1e6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:31.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f761bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f761c06aeb0 0x7f761c1e4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:31.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f761bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f761c06aeb0 0x7f761c1e4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:36882/0 (socket says 172.21.15.57:36882) 2023-10-30T14:26:31.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f761bfff640 1 -- 172.21.15.57:0/3916374923 learned_addr learned my addr 172.21.15.57:0/3916374923 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:31.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f761b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f761c0c8eb0 0x7f761c1e6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40050/0 (socket says 172.21.15.57:40050) 2023-10-30T14:26:31.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f7620f55640 1 --2- 172.21.15.57:0/3916374923 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f761c15ac70 0x7f761c1e66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:31.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f761b7fe640 1 -- 172.21.15.57:0/3916374923 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f761c15ac70 msgr2=0x7f761c1e66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:31.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f761b7fe640 1 --2- 172.21.15.57:0/3916374923 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f761c15ac70 0x7f761c1e66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:31.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f761b7fe640 1 -- 172.21.15.57:0/3916374923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f761c06aeb0 msgr2=0x7f761c1e4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:31.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f761b7fe640 1 --2- 172.21.15.57:0/3916374923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f761c06aeb0 0x7f761c1e4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:31.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f761b7fe640 1 -- 172.21.15.57:0/3916374923 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76100094d0 con 0x7f761c0c8eb0 2023-10-30T14:26:31.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f761b7fe640 1 --2- 172.21.15.57:0/3916374923 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f761c0c8eb0 0x7f761c1e6180 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f761c08c9d0 tx=0x7f761001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:31.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f7618ff9640 1 -- 172.21.15.57:0/3916374923 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76100be070 con 0x7f761c0c8eb0 2023-10-30T14:26:31.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f7618ff9640 1 -- 172.21.15.57:0/3916374923 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76100ba400 con 0x7f761c0c8eb0 2023-10-30T14:26:31.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f7618ff9640 1 -- 172.21.15.57:0/3916374923 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76100ba7d0 con 0x7f761c0c8eb0 2023-10-30T14:26:31.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f7621756640 1 -- 172.21.15.57:0/3916374923 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f761c0c7330 con 0x7f761c0c8eb0 2023-10-30T14:26:31.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f7621756640 1 -- 172.21.15.57:0/3916374923 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f761c1e5980 con 0x7f761c0c8eb0 2023-10-30T14:26:31.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.831+0000 7f7621756640 1 -- 172.21.15.57:0/3916374923 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f761c04e170 con 0x7f761c0c8eb0 2023-10-30T14:26:31.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.835+0000 7f7618ff9640 1 -- 172.21.15.57:0/3916374923 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f76100c3410 con 0x7f761c0c8eb0 2023-10-30T14:26:31.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.835+0000 7f7618ff9640 1 --2- 172.21.15.57:0/3916374923 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76080664a0 0x7f7608068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:31.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.835+0000 7f7618ff9640 1 -- 172.21.15.57:0/3916374923 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f7610121a90 con 0x7f761c0c8eb0 2023-10-30T14:26:31.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.835+0000 7f761bfff640 1 --2- 172.21.15.57:0/3916374923 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76080664a0 0x7f7608068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:31.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.835+0000 7f761bfff640 1 --2- 172.21.15.57:0/3916374923 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76080664a0 0x7f7608068960 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f760c009c20 tx=0x7f760c009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:31.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:31.839+0000 7f7618ff9640 1 -- 172.21.15.57:0/3916374923 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f76100eecf0 con 0x7f761c0c8eb0 2023-10-30T14:26:32.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:32.051+0000 7f7621756640 1 -- 172.21.15.57:0/3916374923 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f761c06b5f0 con 0x7f761c0c8eb0 2023-10-30T14:26:32.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:32.051+0000 7f7618ff9640 1 -- 172.21.15.57:0/3916374923 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5714 (secure 0 0 0) 0x7f76100f2310 con 0x7f761c0c8eb0 2023-10-30T14:26:32.055 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:32.056 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:26:24.805500+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:32.056 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 73 2023-10-30T14:26:32.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:32.055+0000 7f7621756640 1 -- 172.21.15.57:0/3916374923 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76080664a0 msgr2=0x7f7608068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:32.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:32.055+0000 7f7621756640 1 --2- 172.21.15.57:0/3916374923 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76080664a0 0x7f7608068960 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f760c009c20 tx=0x7f760c009190 comp rx=0 tx=0).stop 2023-10-30T14:26:32.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:32.055+0000 7f7621756640 1 -- 172.21.15.57:0/3916374923 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f761c0c8eb0 msgr2=0x7f761c1e6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:32.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:32.055+0000 7f7621756640 1 --2- 172.21.15.57:0/3916374923 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f761c0c8eb0 0x7f761c1e6180 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f761c08c9d0 tx=0x7f761001bad0 comp rx=0 tx=0).stop 2023-10-30T14:26:32.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:32.055+0000 7f7621756640 1 -- 172.21.15.57:0/3916374923 shutdown_connections 2023-10-30T14:26:32.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:32.055+0000 7f7621756640 1 --2- 172.21.15.57:0/3916374923 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76080664a0 0x7f7608068960 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:32.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:32.055+0000 7f7621756640 1 --2- 172.21.15.57:0/3916374923 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f761c15ac70 0x7f761c1e66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:32.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:32.055+0000 7f7621756640 1 --2- 172.21.15.57:0/3916374923 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f761c0c8eb0 0x7f761c1e6180 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:32.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:32.055+0000 7f7621756640 1 --2- 172.21.15.57:0/3916374923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f761c06aeb0 0x7f761c1e4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:32.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:32.055+0000 7f7621756640 1 -- 172.21.15.57:0/3916374923 >> 172.21.15.57:0/3916374923 conn(0x7f761c0cef00 msgr2=0x7f761c15c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:32.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:32.055+0000 7f7621756640 1 -- 172.21.15.57:0/3916374923 shutdown_connections 2023-10-30T14:26:32.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:32.055+0000 7f7621756640 1 -- 172.21.15.57:0/3916374923 wait complete. 2023-10-30T14:26:32.078 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:26:33.701 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:26:33.705 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:26:34.079 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:34.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.231+0000 7fda26b65640 1 Processor -- start 2023-10-30T14:26:34.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.231+0000 7fda26b65640 1 -- start start 2023-10-30T14:26:34.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.231+0000 7fda26b65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda2014fc50 0x7fda2006fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:34.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.231+0000 7fda26b65640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fda200c6320 0x7fda20070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:34.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.231+0000 7fda26b65640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fda2006be90 0x7fda2006c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:34.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.231+0000 7fda26b65640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda200c5f00 con 0x7fda2006be90 2023-10-30T14:26:34.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.231+0000 7fda26b65640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda200c55f0 con 0x7fda200c6320 2023-10-30T14:26:34.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.231+0000 7fda26b65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda20071350 con 0x7fda2014fc50 2023-10-30T14:26:34.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.231+0000 7fda25102640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fda2006be90 0x7fda2006c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:34.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.235+0000 7fda25102640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fda2006be90 0x7fda2006c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34250/0 (socket says 172.21.15.57:34250) 2023-10-30T14:26:34.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.235+0000 7fda25102640 1 -- 172.21.15.57:0/821303703 learned_addr learned my addr 172.21.15.57:0/821303703 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:34.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.235+0000 7fda24901640 1 --2- 172.21.15.57:0/821303703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda2014fc50 0x7fda2006fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:34.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.235+0000 7fda1ffff640 1 --2- 172.21.15.57:0/821303703 >> v2:172.21.15.88:3300/0 conn(0x7fda200c6320 0x7fda20070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:34.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.235+0000 7fda25102640 1 -- 172.21.15.57:0/821303703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda2014fc50 msgr2=0x7fda2006fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:34.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.235+0000 7fda25102640 1 --2- 172.21.15.57:0/821303703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda2014fc50 0x7fda2006fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:34.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.235+0000 7fda25102640 1 -- 172.21.15.57:0/821303703 >> v2:172.21.15.88:3300/0 conn(0x7fda200c6320 msgr2=0x7fda20070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:34.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.235+0000 7fda25102640 1 --2- 172.21.15.57:0/821303703 >> v2:172.21.15.88:3300/0 conn(0x7fda200c6320 0x7fda20070e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:34.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.235+0000 7fda25102640 1 -- 172.21.15.57:0/821303703 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda201567b0 con 0x7fda2006be90 2023-10-30T14:26:34.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.235+0000 7fda25102640 1 --2- 172.21.15.57:0/821303703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda2006be90 0x7fda2006c320 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fda2006c980 tx=0x7fda1801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad5b83f5f108081 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:34.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.235+0000 7fda1f7fe640 1 -- 172.21.15.57:0/821303703 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda180a5740 con 0x7fda2006be90 2023-10-30T14:26:34.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.235+0000 7fda1f7fe640 1 -- 172.21.15.57:0/821303703 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda180a58a0 con 0x7fda2006be90 2023-10-30T14:26:34.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.235+0000 7fda1f7fe640 1 -- 172.21.15.57:0/821303703 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda180a5c30 con 0x7fda2006be90 2023-10-30T14:26:34.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.239+0000 7fda26b65640 1 -- 172.21.15.57:0/821303703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda2006be90 msgr2=0x7fda2006c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:34.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.239+0000 7fda26b65640 1 --2- 172.21.15.57:0/821303703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda2006be90 0x7fda2006c320 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fda2006c980 tx=0x7fda1801c940 comp rx=0 tx=0).stop 2023-10-30T14:26:34.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.239+0000 7fda26b65640 1 -- 172.21.15.57:0/821303703 shutdown_connections 2023-10-30T14:26:34.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.239+0000 7fda26b65640 1 --2- 172.21.15.57:0/821303703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda2006be90 0x7fda2006c320 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:34.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.239+0000 7fda26b65640 1 --2- 172.21.15.57:0/821303703 >> v2:172.21.15.88:3300/0 conn(0x7fda200c6320 0x7fda20070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:34.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.239+0000 7fda26b65640 1 --2- 172.21.15.57:0/821303703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda2014fc50 0x7fda2006fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:34.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.239+0000 7fda26b65640 1 -- 172.21.15.57:0/821303703 >> 172.21.15.57:0/821303703 conn(0x7fda200cac70 msgr2=0x7fda200cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:34.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.239+0000 7fda26b65640 1 -- 172.21.15.57:0/821303703 shutdown_connections 2023-10-30T14:26:34.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.239+0000 7fda26b65640 1 -- 172.21.15.57:0/821303703 wait complete. 2023-10-30T14:26:34.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.239+0000 7fda26b65640 1 Processor -- start 2023-10-30T14:26:34.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.239+0000 7fda26b65640 1 -- start start 2023-10-30T14:26:34.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.239+0000 7fda26b65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda2006be90 0x7fda20152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:34.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.239+0000 7fda26b65640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda200c6320 0x7fda20159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:34.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.239+0000 7fda26b65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda2014fc50 0x7fda2014ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:34.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.239+0000 7fda26b65640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda201ef750 con 0x7fda200c6320 2023-10-30T14:26:34.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.239+0000 7fda26b65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda201f3c30 con 0x7fda2014fc50 2023-10-30T14:26:34.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.239+0000 7fda26b65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda20159970 con 0x7fda2006be90 2023-10-30T14:26:34.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.239+0000 7fda1ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda200c6320 0x7fda20159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:34.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda1ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda200c6320 0x7fda20159430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34264/0 (socket says 172.21.15.57:34264) 2023-10-30T14:26:34.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda1ffff640 1 -- 172.21.15.57:0/3990368740 learned_addr learned my addr 172.21.15.57:0/3990368740 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:34.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda25102640 1 --2- 172.21.15.57:0/3990368740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda2014fc50 0x7fda2014ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:34.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda24901640 1 --2- 172.21.15.57:0/3990368740 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda2006be90 0x7fda20152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:34.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda1ffff640 1 -- 172.21.15.57:0/3990368740 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda2006be90 msgr2=0x7fda20152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:34.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda1ffff640 1 --2- 172.21.15.57:0/3990368740 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda2006be90 0x7fda20152be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:34.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda1ffff640 1 -- 172.21.15.57:0/3990368740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda2014fc50 msgr2=0x7fda2014ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:34.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda1ffff640 1 --2- 172.21.15.57:0/3990368740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda2014fc50 0x7fda2014ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:34.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda1ffff640 1 -- 172.21.15.57:0/3990368740 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda18009b30 con 0x7fda200c6320 2023-10-30T14:26:34.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda25102640 1 --2- 172.21.15.57:0/3990368740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda2014fc50 0x7fda2014ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:34.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda1ffff640 1 --2- 172.21.15.57:0/3990368740 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda200c6320 0x7fda20159430 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00d860 tx=0x7fda0c00dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:34.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda1d7fa640 1 -- 172.21.15.57:0/3990368740 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c00de50 con 0x7fda200c6320 2023-10-30T14:26:34.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda1d7fa640 1 -- 172.21.15.57:0/3990368740 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda0c009850 con 0x7fda200c6320 2023-10-30T14:26:34.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda1d7fa640 1 -- 172.21.15.57:0/3990368740 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c009bc0 con 0x7fda200c6320 2023-10-30T14:26:34.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda26b65640 1 -- 172.21.15.57:0/3990368740 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda201f2dd0 con 0x7fda200c6320 2023-10-30T14:26:34.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda26b65640 1 -- 172.21.15.57:0/3990368740 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda201f3390 con 0x7fda200c6320 2023-10-30T14:26:34.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda26b65640 1 -- 172.21.15.57:0/3990368740 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2004e170 con 0x7fda200c6320 2023-10-30T14:26:34.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda1d7fa640 1 -- 172.21.15.57:0/3990368740 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fda0c009d20 con 0x7fda200c6320 2023-10-30T14:26:34.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda1d7fa640 1 --2- 172.21.15.57:0/3990368740 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fda040664d0 0x7fda04068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:34.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.243+0000 7fda1d7fa640 1 -- 172.21.15.57:0/3990368740 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7fda0c084820 con 0x7fda200c6320 2023-10-30T14:26:34.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.247+0000 7fda24901640 1 --2- 172.21.15.57:0/3990368740 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fda040664d0 0x7fda04068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:34.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.247+0000 7fda24901640 1 --2- 172.21.15.57:0/3990368740 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fda040664d0 0x7fda04068990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fda1400bb10 tx=0x7fda14008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:34.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.247+0000 7fda1d7fa640 1 -- 172.21.15.57:0/3990368740 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fda0c051930 con 0x7fda200c6320 2023-10-30T14:26:34.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.439+0000 7fda26b65640 1 -- 172.21.15.57:0/3990368740 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda20149ce0 con 0x7fda200c6320 2023-10-30T14:26:34.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.439+0000 7fda1d7fa640 1 -- 172.21.15.57:0/3990368740 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5716 (secure 0 0 0) 0x7fda0c054f50 con 0x7fda200c6320 2023-10-30T14:26:34.444 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:34.444 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:26:32.594825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:resolve","state_seq":35,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:34.445 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 74 2023-10-30T14:26:34.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.443+0000 7fda26b65640 1 -- 172.21.15.57:0/3990368740 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fda040664d0 msgr2=0x7fda04068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:34.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.443+0000 7fda26b65640 1 --2- 172.21.15.57:0/3990368740 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fda040664d0 0x7fda04068990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fda1400bb10 tx=0x7fda14008040 comp rx=0 tx=0).stop 2023-10-30T14:26:34.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.443+0000 7fda26b65640 1 -- 172.21.15.57:0/3990368740 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda200c6320 msgr2=0x7fda20159430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:34.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.443+0000 7fda26b65640 1 --2- 172.21.15.57:0/3990368740 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda200c6320 0x7fda20159430 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00d860 tx=0x7fda0c00dd20 comp rx=0 tx=0).stop 2023-10-30T14:26:34.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.443+0000 7fda26b65640 1 -- 172.21.15.57:0/3990368740 shutdown_connections 2023-10-30T14:26:34.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.443+0000 7fda26b65640 1 --2- 172.21.15.57:0/3990368740 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fda040664d0 0x7fda04068990 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:34.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.443+0000 7fda26b65640 1 --2- 172.21.15.57:0/3990368740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda2014fc50 0x7fda2014ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:34.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.443+0000 7fda26b65640 1 --2- 172.21.15.57:0/3990368740 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda200c6320 0x7fda20159430 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:34.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.443+0000 7fda26b65640 1 --2- 172.21.15.57:0/3990368740 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda2006be90 0x7fda20152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:34.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.443+0000 7fda26b65640 1 -- 172.21.15.57:0/3990368740 >> 172.21.15.57:0/3990368740 conn(0x7fda200cac70 msgr2=0x7fda20142340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:34.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.443+0000 7fda26b65640 1 -- 172.21.15.57:0/3990368740 shutdown_connections 2023-10-30T14:26:34.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:34.443+0000 7fda26b65640 1 -- 172.21.15.57:0/3990368740 wait complete. 2023-10-30T14:26:34.465 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:26:36.466 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:36.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.619+0000 7f1dea046640 1 Processor -- start 2023-10-30T14:26:36.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.619+0000 7f1dea046640 1 -- start start 2023-10-30T14:26:36.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.619+0000 7f1dea046640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1de4153c80 0x7f1de415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:36.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.619+0000 7f1dea046640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1de40c42a0 0x7f1de40ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:36.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.623+0000 7f1dea046640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de4159f20 con 0x7f1de40c4a10 2023-10-30T14:26:36.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.623+0000 7f1dea046640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de415dec0 con 0x7f1de4153c80 2023-10-30T14:26:36.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.623+0000 7f1dea046640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de41599a0 con 0x7f1de40c42a0 2023-10-30T14:26:36.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.623+0000 7f1de2ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1de40c4a10 0x7f1de415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46130/0 (socket says 172.21.15.57:46130) 2023-10-30T14:26:36.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.623+0000 7f1de2ffd640 1 -- 172.21.15.57:0/2922307576 learned_addr learned my addr 172.21.15.57:0/2922307576 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:26:36.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.623+0000 7f1de3fff640 1 --2- 172.21.15.57:0/2922307576 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1de40c42a0 0x7f1de40ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:36.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.623+0000 7f1de37fe640 1 --2- 172.21.15.57:0/2922307576 >> v2:172.21.15.88:3300/0 conn(0x7f1de4153c80 0x7f1de415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:36.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.623+0000 7f1de27fc640 1 -- 172.21.15.57:0/2922307576 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679624418 0 0) 0x7f1de4159f20 con 0x7f1de40c4a10 2023-10-30T14:26:36.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.623+0000 7f1de27fc640 1 -- 172.21.15.57:0/2922307576 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dcc003530 con 0x7f1de40c4a10 2023-10-30T14:26:36.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.623+0000 7f1de27fc640 1 -- 172.21.15.57:0/2922307576 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3441222259 0 0) 0x7f1dcc003530 con 0x7f1de40c4a10 2023-10-30T14:26:36.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.623+0000 7f1de3fff640 1 -- 172.21.15.57:0/2922307576 >> v2:172.21.15.88:3300/0 conn(0x7f1de4153c80 msgr2=0x7f1de415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:36.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.623+0000 7f1de3fff640 1 --2- 172.21.15.57:0/2922307576 >> v2:172.21.15.88:3300/0 conn(0x7f1de4153c80 0x7f1de415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.623+0000 7f1de3fff640 1 -- 172.21.15.57:0/2922307576 >> v1:172.21.15.57:6789/0 conn(0x7f1de40c4a10 legacy=0x7f1de415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.623+0000 7f1de3fff640 1 -- 172.21.15.57:0/2922307576 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de415d620 con 0x7f1de40c42a0 2023-10-30T14:26:36.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.627+0000 7f1de3fff640 1 --2- 172.21.15.57:0/2922307576 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1de40c42a0 0x7f1de40ca1e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc009d90 tx=0x7f1ddc01f060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cf6320dfac95a0f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:36.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.627+0000 7f1de27fc640 1 -- 172.21.15.57:0/2922307576 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8001680 con 0x7f1de40c42a0 2023-10-30T14:26:36.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.627+0000 7f1de27fc640 1 -- 172.21.15.57:0/2922307576 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ddc0a8bb0 con 0x7f1de40c42a0 2023-10-30T14:26:36.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.627+0000 7f1de27fc640 1 -- 172.21.15.57:0/2922307576 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc0b0440 con 0x7f1de40c42a0 2023-10-30T14:26:36.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.627+0000 7f1dea046640 1 -- 172.21.15.57:0/2922307576 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1de40c42a0 msgr2=0x7f1de40ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.627+0000 7f1dea046640 1 --2- 172.21.15.57:0/2922307576 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1de40c42a0 0x7f1de40ca1e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc009d90 tx=0x7f1ddc01f060 comp rx=0 tx=0).stop 2023-10-30T14:26:36.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.627+0000 7f1dea046640 1 -- 172.21.15.57:0/2922307576 shutdown_connections 2023-10-30T14:26:36.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.627+0000 7f1dea046640 1 --2- 172.21.15.57:0/2922307576 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1de40c42a0 0x7f1de40ca1e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.627+0000 7f1dea046640 1 --2- 172.21.15.57:0/2922307576 >> v2:172.21.15.88:3300/0 conn(0x7f1de4153c80 0x7f1de415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.627+0000 7f1dea046640 1 -- 172.21.15.57:0/2922307576 >> 172.21.15.57:0/2922307576 conn(0x7f1de40715c0 msgr2=0x7f1de40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:36.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.627+0000 7f1dea046640 1 -- 172.21.15.57:0/2922307576 shutdown_connections 2023-10-30T14:26:36.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.627+0000 7f1dea046640 1 -- 172.21.15.57:0/2922307576 wait complete. 2023-10-30T14:26:36.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.627+0000 7f1dea046640 1 Processor -- start 2023-10-30T14:26:36.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.627+0000 7f1dea046640 1 -- start start 2023-10-30T14:26:36.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1dea046640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1de40c42a0 0x7f1de414d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:36.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1dea046640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1de40c4a10 0x7f1de4145f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:36.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1dea046640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1de4153c80 0x7f1de40cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:36.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1dea046640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de414d8d0 con 0x7f1de4153c80 2023-10-30T14:26:36.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1dea046640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de414daf0 con 0x7f1de40c4a10 2023-10-30T14:26:36.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1dea046640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de415b230 con 0x7f1de40c42a0 2023-10-30T14:26:36.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1de3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1de4153c80 0x7f1de40cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:36.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1de37fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1de40c42a0 0x7f1de414d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:36.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1de3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1de4153c80 0x7f1de40cf1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34272/0 (socket says 172.21.15.57:34272) 2023-10-30T14:26:36.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1de3fff640 1 -- 172.21.15.57:0/4258104928 learned_addr learned my addr 172.21.15.57:0/4258104928 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:36.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1de2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1de40c4a10 0x7f1de4145f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:36.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1de3fff640 1 -- 172.21.15.57:0/4258104928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1de40c42a0 msgr2=0x7f1de414d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1de3fff640 1 --2- 172.21.15.57:0/4258104928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1de40c42a0 0x7f1de414d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1de3fff640 1 -- 172.21.15.57:0/4258104928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1de40c4a10 msgr2=0x7f1de4145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1de3fff640 1 --2- 172.21.15.57:0/4258104928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1de40c4a10 0x7f1de4145f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1de3fff640 1 -- 172.21.15.57:0/4258104928 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ddc009a70 con 0x7f1de4153c80 2023-10-30T14:26:36.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1de3fff640 1 --2- 172.21.15.57:0/4258104928 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1de4153c80 0x7f1de40cf1c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc009d90 tx=0x7f1ddc01be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:36.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1dcbfff640 1 -- 172.21.15.57:0/4258104928 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc009450 con 0x7f1de4153c80 2023-10-30T14:26:36.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1dcbfff640 1 -- 172.21.15.57:0/4258104928 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ddc0ba410 con 0x7f1de4153c80 2023-10-30T14:26:36.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1dea046640 1 -- 172.21.15.57:0/4258104928 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de41eeeb0 con 0x7f1de4153c80 2023-10-30T14:26:36.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1dcbfff640 1 -- 172.21.15.57:0/4258104928 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc0ba7a0 con 0x7f1de4153c80 2023-10-30T14:26:36.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1dea046640 1 -- 172.21.15.57:0/4258104928 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de415b7b0 con 0x7f1de4153c80 2023-10-30T14:26:36.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.631+0000 7f1dea046640 1 -- 172.21.15.57:0/4258104928 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de41ef070 con 0x7f1de4153c80 2023-10-30T14:26:36.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.635+0000 7f1dcbfff640 1 -- 172.21.15.57:0/4258104928 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1ddc0b7050 con 0x7f1de4153c80 2023-10-30T14:26:36.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.635+0000 7f1dcbfff640 1 --2- 172.21.15.57:0/4258104928 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1dd00664d0 0x7f1dd0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:36.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.635+0000 7f1dcbfff640 1 -- 172.21.15.57:0/4258104928 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f1ddc122500 con 0x7f1de4153c80 2023-10-30T14:26:36.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.635+0000 7f1de37fe640 1 --2- 172.21.15.57:0/4258104928 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1dd00664d0 0x7f1dd0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:36.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.635+0000 7f1de37fe640 1 --2- 172.21.15.57:0/4258104928 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1dd00664d0 0x7f1dd0068990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009ac0 tx=0x7f1dd4009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:36.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.639+0000 7f1dcbfff640 1 -- 172.21.15.57:0/4258104928 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1ddc0ef700 con 0x7f1de4153c80 2023-10-30T14:26:36.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.835+0000 7f1dea046640 1 -- 172.21.15.57:0/4258104928 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1de404e170 con 0x7f1de4153c80 2023-10-30T14:26:36.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.835+0000 7f1dcbfff640 1 -- 172.21.15.57:0/4258104928 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f1ddc0f2d20 con 0x7f1de4153c80 2023-10-30T14:26:36.840 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:36.840 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:26:32.594825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:resolve","state_seq":35,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:36.840 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 74 2023-10-30T14:26:36.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.839+0000 7f1dea046640 1 -- 172.21.15.57:0/4258104928 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1dd00664d0 msgr2=0x7f1dd0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.839+0000 7f1dea046640 1 --2- 172.21.15.57:0/4258104928 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1dd00664d0 0x7f1dd0068990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009ac0 tx=0x7f1dd4009190 comp rx=0 tx=0).stop 2023-10-30T14:26:36.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.839+0000 7f1dea046640 1 -- 172.21.15.57:0/4258104928 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1de4153c80 msgr2=0x7f1de40cf1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.839+0000 7f1dea046640 1 --2- 172.21.15.57:0/4258104928 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1de4153c80 0x7f1de40cf1c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc009d90 tx=0x7f1ddc01be00 comp rx=0 tx=0).stop 2023-10-30T14:26:36.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.839+0000 7f1dea046640 1 -- 172.21.15.57:0/4258104928 shutdown_connections 2023-10-30T14:26:36.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.839+0000 7f1dea046640 1 --2- 172.21.15.57:0/4258104928 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1dd00664d0 0x7f1dd0068990 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.839+0000 7f1dea046640 1 --2- 172.21.15.57:0/4258104928 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1de4153c80 0x7f1de40cf1c0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.839+0000 7f1dea046640 1 --2- 172.21.15.57:0/4258104928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1de40c4a10 0x7f1de4145f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.839+0000 7f1dea046640 1 --2- 172.21.15.57:0/4258104928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1de40c42a0 0x7f1de414d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.839+0000 7f1dea046640 1 -- 172.21.15.57:0/4258104928 >> 172.21.15.57:0/4258104928 conn(0x7f1de40715c0 msgr2=0x7f1de41577d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:36.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.839+0000 7f1dea046640 1 -- 172.21.15.57:0/4258104928 shutdown_connections 2023-10-30T14:26:36.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:36.839+0000 7f1dea046640 1 -- 172.21.15.57:0/4258104928 wait complete. 2023-10-30T14:26:36.861 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:26:38.862 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:39.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.015+0000 7fb89c446640 1 Processor -- start 2023-10-30T14:26:39.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.015+0000 7fb89c446640 1 -- start start 2023-10-30T14:26:39.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.015+0000 7fb89c446640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb8940c4a10 0x7fb89415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:39.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.015+0000 7fb89c446640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8940c42a0 0x7fb8940ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:39.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.015+0000 7fb89c446640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb894159f20 con 0x7fb894153c80 2023-10-30T14:26:39.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.015+0000 7fb89c446640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb89415dec0 con 0x7fb8940c4a10 2023-10-30T14:26:39.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.015+0000 7fb89c446640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8941599a0 con 0x7fb8940c42a0 2023-10-30T14:26:39.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.015+0000 7fb8999e1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb8940c4a10 0x7fb89415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:39.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.015+0000 7fb8999e1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb8940c4a10 0x7fb89415cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34284/0 (socket says 172.21.15.57:34284) 2023-10-30T14:26:39.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.015+0000 7fb8999e1640 1 -- 172.21.15.57:0/690576191 learned_addr learned my addr 172.21.15.57:0/690576191 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:39.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.015+0000 7fb89a9e3640 1 --2- 172.21.15.57:0/690576191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8940c42a0 0x7fb8940ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:39.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.015+0000 7fb8999e1640 1 -- 172.21.15.57:0/690576191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8940c42a0 msgr2=0x7fb8940ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:39.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.015+0000 7fb8999e1640 1 --2- 172.21.15.57:0/690576191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8940c42a0 0x7fb8940ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:39.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.015+0000 7fb8999e1640 1 -- 172.21.15.57:0/690576191 >> v1:172.21.15.88:6789/0 conn(0x7fb894153c80 legacy=0x7fb89415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:39.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.015+0000 7fb8999e1640 1 -- 172.21.15.57:0/690576191 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89415d620 con 0x7fb8940c4a10 2023-10-30T14:26:39.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.019+0000 7fb8999e1640 1 --2- 172.21.15.57:0/690576191 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8940c4a10 0x7fb89415cf90 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb884009870 tx=0x7fb88401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b640b24a49f2712c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:39.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.019+0000 7fb8991e0640 1 -- 172.21.15.57:0/690576191 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8840a8a30 con 0x7fb8940c4a10 2023-10-30T14:26:39.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.019+0000 7fb8991e0640 1 -- 172.21.15.57:0/690576191 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8840a8b90 con 0x7fb8940c4a10 2023-10-30T14:26:39.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.019+0000 7fb8991e0640 1 -- 172.21.15.57:0/690576191 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8840b0440 con 0x7fb8940c4a10 2023-10-30T14:26:39.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.019+0000 7fb89c446640 1 -- 172.21.15.57:0/690576191 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8940c4a10 msgr2=0x7fb89415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:39.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.019+0000 7fb89c446640 1 --2- 172.21.15.57:0/690576191 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8940c4a10 0x7fb89415cf90 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb884009870 tx=0x7fb88401ed30 comp rx=0 tx=0).stop 2023-10-30T14:26:39.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.019+0000 7fb89c446640 1 -- 172.21.15.57:0/690576191 shutdown_connections 2023-10-30T14:26:39.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.019+0000 7fb89c446640 1 --2- 172.21.15.57:0/690576191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8940c42a0 0x7fb8940ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:39.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.019+0000 7fb89c446640 1 --2- 172.21.15.57:0/690576191 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8940c4a10 0x7fb89415cf90 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:39.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.019+0000 7fb89c446640 1 -- 172.21.15.57:0/690576191 >> 172.21.15.57:0/690576191 conn(0x7fb8940715c0 msgr2=0x7fb8940719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:39.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.019+0000 7fb89c446640 1 -- 172.21.15.57:0/690576191 shutdown_connections 2023-10-30T14:26:39.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.019+0000 7fb89c446640 1 -- 172.21.15.57:0/690576191 wait complete. 2023-10-30T14:26:39.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb89c446640 1 Processor -- start 2023-10-30T14:26:39.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb89c446640 1 -- start start 2023-10-30T14:26:39.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb89c446640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8940c42a0 0x7fb89414d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:39.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb89c446640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8940c4a10 0x7fb89414ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:39.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb89c446640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb894153c80 0x7fb89414f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:39.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb89c446640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89415c380 con 0x7fb894153c80 2023-10-30T14:26:39.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb89c446640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89415c620 con 0x7fb8940c4a10 2023-10-30T14:26:39.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb89c446640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89415c900 con 0x7fb8940c42a0 2023-10-30T14:26:39.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb89a9e3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb894153c80 0x7fb89414f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:39.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb89a9e3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb894153c80 0x7fb89414f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34288/0 (socket says 172.21.15.57:34288) 2023-10-30T14:26:39.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb89a9e3640 1 -- 172.21.15.57:0/1695573983 learned_addr learned my addr 172.21.15.57:0/1695573983 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:39.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb89a1e2640 1 --2- 172.21.15.57:0/1695573983 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8940c42a0 0x7fb89414d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:39.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb8999e1640 1 --2- 172.21.15.57:0/1695573983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8940c4a10 0x7fb89414ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:39.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb89a9e3640 1 -- 172.21.15.57:0/1695573983 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8940c42a0 msgr2=0x7fb89414d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:39.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb89a9e3640 1 --2- 172.21.15.57:0/1695573983 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8940c42a0 0x7fb89414d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:39.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb89a9e3640 1 -- 172.21.15.57:0/1695573983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8940c4a10 msgr2=0x7fb89414ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:39.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb89a9e3640 1 --2- 172.21.15.57:0/1695573983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8940c4a10 0x7fb89414ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:39.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb89a9e3640 1 -- 172.21.15.57:0/1695573983 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8840094d0 con 0x7fb894153c80 2023-10-30T14:26:39.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb8999e1640 1 --2- 172.21.15.57:0/1695573983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8940c4a10 0x7fb89414ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:39.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb89a9e3640 1 --2- 172.21.15.57:0/1695573983 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb894153c80 0x7fb89414f4c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00efc0 tx=0x7fb88c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:39.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb88affd640 1 -- 172.21.15.57:0/1695573983 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c009ba0 con 0x7fb894153c80 2023-10-30T14:26:39.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.023+0000 7fb88affd640 1 -- 172.21.15.57:0/1695573983 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88c00f040 con 0x7fb894153c80 2023-10-30T14:26:39.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.027+0000 7fb89c446640 1 -- 172.21.15.57:0/1695573983 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb89415cbe0 con 0x7fb894153c80 2023-10-30T14:26:39.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.027+0000 7fb88affd640 1 -- 172.21.15.57:0/1695573983 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c0146c0 con 0x7fb894153c80 2023-10-30T14:26:39.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.027+0000 7fb89c446640 1 -- 172.21.15.57:0/1695573983 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb89415d1a0 con 0x7fb894153c80 2023-10-30T14:26:39.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.027+0000 7fb89c446640 1 -- 172.21.15.57:0/1695573983 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb89404e170 con 0x7fb894153c80 2023-10-30T14:26:39.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.027+0000 7fb88affd640 1 -- 172.21.15.57:0/1695573983 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb88c014820 con 0x7fb894153c80 2023-10-30T14:26:39.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.027+0000 7fb88affd640 1 --2- 172.21.15.57:0/1695573983 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8780664f0 0x7fb8780689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:39.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.027+0000 7fb88affd640 1 -- 172.21.15.57:0/1695573983 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7fb88c084b70 con 0x7fb894153c80 2023-10-30T14:26:39.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.027+0000 7fb89a1e2640 1 --2- 172.21.15.57:0/1695573983 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8780664f0 0x7fb8780689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:39.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.031+0000 7fb89a1e2640 1 --2- 172.21.15.57:0/1695573983 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8780664f0 0x7fb8780689b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb890001810 tx=0x7fb890009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:39.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.031+0000 7fb88affd640 1 -- 172.21.15.57:0/1695573983 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb88c051d70 con 0x7fb894153c80 2023-10-30T14:26:39.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.227+0000 7fb89c446640 1 -- 172.21.15.57:0/1695573983 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8940cad40 con 0x7fb894153c80 2023-10-30T14:26:39.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.231+0000 7fb88affd640 1 -- 172.21.15.57:0/1695573983 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5718 (secure 0 0 0) 0x7fb88c018020 con 0x7fb894153c80 2023-10-30T14:26:39.234 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:39.234 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:26:38.032694+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:39.234 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 75 2023-10-30T14:26:39.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.231+0000 7fb89c446640 1 -- 172.21.15.57:0/1695573983 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8780664f0 msgr2=0x7fb8780689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:39.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.231+0000 7fb89c446640 1 --2- 172.21.15.57:0/1695573983 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8780664f0 0x7fb8780689b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb890001810 tx=0x7fb890009040 comp rx=0 tx=0).stop 2023-10-30T14:26:39.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.231+0000 7fb89c446640 1 -- 172.21.15.57:0/1695573983 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb894153c80 msgr2=0x7fb89414f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:39.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.231+0000 7fb89c446640 1 --2- 172.21.15.57:0/1695573983 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb894153c80 0x7fb89414f4c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00efc0 tx=0x7fb88c00c490 comp rx=0 tx=0).stop 2023-10-30T14:26:39.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.231+0000 7fb89c446640 1 -- 172.21.15.57:0/1695573983 shutdown_connections 2023-10-30T14:26:39.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.231+0000 7fb89c446640 1 --2- 172.21.15.57:0/1695573983 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8780664f0 0x7fb8780689b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:39.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.231+0000 7fb89c446640 1 --2- 172.21.15.57:0/1695573983 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb894153c80 0x7fb89414f4c0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:39.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.231+0000 7fb89c446640 1 --2- 172.21.15.57:0/1695573983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8940c4a10 0x7fb89414ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:39.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.231+0000 7fb89c446640 1 --2- 172.21.15.57:0/1695573983 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8940c42a0 0x7fb89414d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:39.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.231+0000 7fb89c446640 1 -- 172.21.15.57:0/1695573983 >> 172.21.15.57:0/1695573983 conn(0x7fb8940715c0 msgr2=0x7fb894158250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:39.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.231+0000 7fb89c446640 1 -- 172.21.15.57:0/1695573983 shutdown_connections 2023-10-30T14:26:39.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:39.231+0000 7fb89c446640 1 -- 172.21.15.57:0/1695573983 wait complete. 2023-10-30T14:26:39.257 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 75, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 75, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5586": { "addr": "172.21.15.57:6837/862088802", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 862088802, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 862088802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5586, "incarnation": 67, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 51 }, "gid_5797": { "addr": "172.21.15.57:6834/2940934016", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2940934016, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2940934016, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5797, "incarnation": 73, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 37 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:26:38.032694+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5797, "mds_1": 5586 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6839/358521496", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 358521496, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 358521496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 66, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5315, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:26:41.258 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:41.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.407+0000 7f71f066b640 1 Processor -- start 2023-10-30T14:26:41.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.407+0000 7f71f066b640 1 -- start start 2023-10-30T14:26:41.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.411+0000 7f71f066b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71e8153bb0 0x7f71e80c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.411+0000 7f71f066b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f71e806c420 0x7f71e80c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.411+0000 7f71f066b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e80c5f80 con 0x7f71e80c7170 2023-10-30T14:26:41.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.411+0000 7f71f066b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e80c5690 con 0x7f71e806c420 2023-10-30T14:26:41.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.411+0000 7f71f066b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e815a580 con 0x7f71e8153bb0 2023-10-30T14:26:41.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.411+0000 7f71eec08640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f71e80c7170 0x7f71e8159980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46132/0 (socket says 172.21.15.57:46132) 2023-10-30T14:26:41.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.411+0000 7f71eec08640 1 -- 172.21.15.57:0/2388513272 learned_addr learned my addr 172.21.15.57:0/2388513272 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:26:41.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.411+0000 7f71ee407640 1 --2- 172.21.15.57:0/2388513272 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71e8153bb0 0x7f71e80c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.411+0000 7f71edc06640 1 --2- 172.21.15.57:0/2388513272 >> v2:172.21.15.88:3300/0 conn(0x7f71e806c420 0x7f71e80c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.411+0000 7f71ed405640 1 -- 172.21.15.57:0/2388513272 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521957180 0 0) 0x7f71e80c5f80 con 0x7f71e80c7170 2023-10-30T14:26:41.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.411+0000 7f71ed405640 1 -- 172.21.15.57:0/2388513272 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71d4003530 con 0x7f71e80c7170 2023-10-30T14:26:41.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.411+0000 7f71ee407640 1 -- 172.21.15.57:0/2388513272 >> v2:172.21.15.88:3300/0 conn(0x7f71e806c420 msgr2=0x7f71e80c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:41.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.411+0000 7f71ee407640 1 --2- 172.21.15.57:0/2388513272 >> v2:172.21.15.88:3300/0 conn(0x7f71e806c420 0x7f71e80c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.411+0000 7f71ee407640 1 -- 172.21.15.57:0/2388513272 >> v1:172.21.15.57:6789/0 conn(0x7f71e80c7170 legacy=0x7f71e8159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.411+0000 7f71ee407640 1 -- 172.21.15.57:0/2388513272 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e80c5930 con 0x7f71e8153bb0 2023-10-30T14:26:41.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.411+0000 7f71edc06640 1 --2- 172.21.15.57:0/2388513272 >> v2:172.21.15.88:3300/0 conn(0x7f71e806c420 0x7f71e80c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:26:41.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.415+0000 7f71ee407640 1 --2- 172.21.15.57:0/2388513272 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71e8153bb0 0x7f71e80c6410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f71e40097c0 tx=0x7f71e401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d2d1ebde5c87947 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:41.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.415+0000 7f71ed405640 1 -- 172.21.15.57:0/2388513272 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e40a8970 con 0x7f71e8153bb0 2023-10-30T14:26:41.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.415+0000 7f71ed405640 1 -- 172.21.15.57:0/2388513272 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e40a8ad0 con 0x7f71e8153bb0 2023-10-30T14:26:41.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.415+0000 7f71ed405640 1 -- 172.21.15.57:0/2388513272 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e40a8e40 con 0x7f71e8153bb0 2023-10-30T14:26:41.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.415+0000 7f71f066b640 1 -- 172.21.15.57:0/2388513272 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71e8153bb0 msgr2=0x7f71e80c6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.415+0000 7f71f066b640 1 --2- 172.21.15.57:0/2388513272 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71e8153bb0 0x7f71e80c6410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f71e40097c0 tx=0x7f71e401ec70 comp rx=0 tx=0).stop 2023-10-30T14:26:41.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.415+0000 7f71f066b640 1 -- 172.21.15.57:0/2388513272 shutdown_connections 2023-10-30T14:26:41.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.415+0000 7f71f066b640 1 --2- 172.21.15.57:0/2388513272 >> v2:172.21.15.88:3300/0 conn(0x7f71e806c420 0x7f71e80c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.415+0000 7f71f066b640 1 --2- 172.21.15.57:0/2388513272 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71e8153bb0 0x7f71e80c6410 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.415+0000 7f71f066b640 1 -- 172.21.15.57:0/2388513272 >> 172.21.15.57:0/2388513272 conn(0x7f71e80cac70 msgr2=0x7f71e80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:41.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.415+0000 7f71f066b640 1 -- 172.21.15.57:0/2388513272 shutdown_connections 2023-10-30T14:26:41.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.415+0000 7f71f066b640 1 -- 172.21.15.57:0/2388513272 wait complete. 2023-10-30T14:26:41.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.415+0000 7f71f066b640 1 Processor -- start 2023-10-30T14:26:41.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.415+0000 7f71f066b640 1 -- start start 2023-10-30T14:26:41.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71f066b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e806c420 0x7f71e80bccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71f066b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71e80c7170 0x7f71e8156610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71f066b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71e8153bb0 0x7f71e8156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71f066b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e80c9030 con 0x7f71e8153bb0 2023-10-30T14:26:41.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71f066b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e81ee4c0 con 0x7f71e806c420 2023-10-30T14:26:41.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71f066b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e80c44a0 con 0x7f71e80c7170 2023-10-30T14:26:41.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71eec08640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71e8153bb0 0x7f71e8156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71eec08640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71e8153bb0 0x7f71e8156b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34294/0 (socket says 172.21.15.57:34294) 2023-10-30T14:26:41.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71eec08640 1 -- 172.21.15.57:0/2410750548 learned_addr learned my addr 172.21.15.57:0/2410750548 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:41.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71edc06640 1 --2- 172.21.15.57:0/2410750548 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71e80c7170 0x7f71e8156610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71ee407640 1 --2- 172.21.15.57:0/2410750548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e806c420 0x7f71e80bccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71eec08640 1 -- 172.21.15.57:0/2410750548 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71e80c7170 msgr2=0x7f71e8156610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71eec08640 1 --2- 172.21.15.57:0/2410750548 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71e80c7170 0x7f71e8156610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71eec08640 1 -- 172.21.15.57:0/2410750548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e806c420 msgr2=0x7f71e80bccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71eec08640 1 --2- 172.21.15.57:0/2410750548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e806c420 0x7f71e80bccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71eec08640 1 -- 172.21.15.57:0/2410750548 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d8009420 con 0x7f71e8153bb0 2023-10-30T14:26:41.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71eec08640 1 --2- 172.21.15.57:0/2410750548 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71e8153bb0 0x7f71e8156b50 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f71e000caa0 tx=0x7f71e000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:41.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71deffd640 1 -- 172.21.15.57:0/2410750548 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e000ac00 con 0x7f71e8153bb0 2023-10-30T14:26:41.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71deffd640 1 -- 172.21.15.57:0/2410750548 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e000ad60 con 0x7f71e8153bb0 2023-10-30T14:26:41.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71deffd640 1 -- 172.21.15.57:0/2410750548 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e0012620 con 0x7f71e8153bb0 2023-10-30T14:26:41.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71f066b640 1 -- 172.21.15.57:0/2410750548 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e4009420 con 0x7f71e8153bb0 2023-10-30T14:26:41.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71f066b640 1 -- 172.21.15.57:0/2410750548 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e80c4a60 con 0x7f71e8153bb0 2023-10-30T14:26:41.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.419+0000 7f71f066b640 1 -- 172.21.15.57:0/2410750548 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e81ee680 con 0x7f71e8153bb0 2023-10-30T14:26:41.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.423+0000 7f71deffd640 1 -- 172.21.15.57:0/2410750548 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f71e0007070 con 0x7f71e8153bb0 2023-10-30T14:26:41.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.423+0000 7f71deffd640 1 --2- 172.21.15.57:0/2410750548 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f71cc0664a0 0x7f71cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.423+0000 7f71deffd640 1 -- 172.21.15.57:0/2410750548 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f71e00840b0 con 0x7f71e8153bb0 2023-10-30T14:26:41.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.423+0000 7f71ee407640 1 --2- 172.21.15.57:0/2410750548 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f71cc0664a0 0x7f71cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.423+0000 7f71ee407640 1 --2- 172.21.15.57:0/2410750548 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f71cc0664a0 0x7f71cc068960 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f71e40b2000 tx=0x7f71e40aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:41.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.427+0000 7f71deffd640 1 -- 172.21.15.57:0/2410750548 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f71e0051280 con 0x7f71e8153bb0 2023-10-30T14:26:41.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.615+0000 7f71f066b640 1 -- 172.21.15.57:0/2410750548 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71e80c5be0 con 0x7f71e8153bb0 2023-10-30T14:26:41.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.615+0000 7f71deffd640 1 -- 172.21.15.57:0/2410750548 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5718 (secure 0 0 0) 0x7f71e0017090 con 0x7f71e8153bb0 2023-10-30T14:26:41.619 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:41.620 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:26:38.032694+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:41.620 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 75 2023-10-30T14:26:41.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.619+0000 7f71f066b640 1 -- 172.21.15.57:0/2410750548 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f71cc0664a0 msgr2=0x7f71cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.619+0000 7f71f066b640 1 --2- 172.21.15.57:0/2410750548 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f71cc0664a0 0x7f71cc068960 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f71e40b2000 tx=0x7f71e40aa070 comp rx=0 tx=0).stop 2023-10-30T14:26:41.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.619+0000 7f71f066b640 1 -- 172.21.15.57:0/2410750548 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71e8153bb0 msgr2=0x7f71e8156b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.619+0000 7f71f066b640 1 --2- 172.21.15.57:0/2410750548 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71e8153bb0 0x7f71e8156b50 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f71e000caa0 tx=0x7f71e000cf60 comp rx=0 tx=0).stop 2023-10-30T14:26:41.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.619+0000 7f71f066b640 1 -- 172.21.15.57:0/2410750548 shutdown_connections 2023-10-30T14:26:41.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.619+0000 7f71f066b640 1 --2- 172.21.15.57:0/2410750548 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f71cc0664a0 0x7f71cc068960 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.619+0000 7f71f066b640 1 --2- 172.21.15.57:0/2410750548 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71e8153bb0 0x7f71e8156b50 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.619+0000 7f71f066b640 1 --2- 172.21.15.57:0/2410750548 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71e80c7170 0x7f71e8156610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.619+0000 7f71f066b640 1 --2- 172.21.15.57:0/2410750548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e806c420 0x7f71e80bccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.619+0000 7f71f066b640 1 -- 172.21.15.57:0/2410750548 >> 172.21.15.57:0/2410750548 conn(0x7f71e80cac70 msgr2=0x7f71e81526d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:41.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.619+0000 7f71f066b640 1 -- 172.21.15.57:0/2410750548 shutdown_connections 2023-10-30T14:26:41.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:41.619+0000 7f71f066b640 1 -- 172.21.15.57:0/2410750548 wait complete. 2023-10-30T14:26:41.640 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:26:43.642 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:43.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.795+0000 7f0122165640 1 Processor -- start 2023-10-30T14:26:43.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.795+0000 7f0122165640 1 -- start start 2023-10-30T14:26:43.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.795+0000 7f0122165640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f011c0c4a10 0x7f011c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.795+0000 7f0122165640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f011c0c42a0 0x7f011c0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.795+0000 7f0122165640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011c159f20 con 0x7f011c153c80 2023-10-30T14:26:43.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.795+0000 7f0122165640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f011c15dec0 con 0x7f011c0c4a10 2023-10-30T14:26:43.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.795+0000 7f0122165640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f011c1599a0 con 0x7f011c0c42a0 2023-10-30T14:26:43.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.795+0000 7f011affd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f011c0c4a10 0x7f011c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.795+0000 7f011affd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f011c0c4a10 0x7f011c15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54924/0 (socket says 172.21.15.57:54924) 2023-10-30T14:26:43.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.795+0000 7f011affd640 1 -- 172.21.15.57:0/2190116219 learned_addr learned my addr 172.21.15.57:0/2190116219 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:43.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.795+0000 7f011bfff640 1 --2- 172.21.15.57:0/2190116219 >> v2:172.21.15.88:3300/0 conn(0x7f011c0c42a0 0x7f011c0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.795+0000 7f011affd640 1 -- 172.21.15.57:0/2190116219 >> v2:172.21.15.88:3300/0 conn(0x7f011c0c42a0 msgr2=0x7f011c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:43.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.795+0000 7f011affd640 1 --2- 172.21.15.57:0/2190116219 >> v2:172.21.15.88:3300/0 conn(0x7f011c0c42a0 0x7f011c0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.795+0000 7f011affd640 1 -- 172.21.15.57:0/2190116219 >> v1:172.21.15.88:6789/0 conn(0x7f011c153c80 legacy=0x7f011c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.795+0000 7f011affd640 1 -- 172.21.15.57:0/2190116219 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011c15d620 con 0x7f011c0c4a10 2023-10-30T14:26:43.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.799+0000 7f011affd640 1 --2- 172.21.15.57:0/2190116219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f011c0c4a10 0x7f011c15cf90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f011c0b1d40 tx=0x7f011001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17c19cc471303852 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:43.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.799+0000 7f011a7fc640 1 -- 172.21.15.57:0/2190116219 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01100a89f0 con 0x7f011c0c4a10 2023-10-30T14:26:43.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.799+0000 7f011a7fc640 1 -- 172.21.15.57:0/2190116219 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01100a8b50 con 0x7f011c0c4a10 2023-10-30T14:26:43.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.799+0000 7f011a7fc640 1 -- 172.21.15.57:0/2190116219 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01100b0440 con 0x7f011c0c4a10 2023-10-30T14:26:43.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.799+0000 7f0122165640 1 -- 172.21.15.57:0/2190116219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f011c0c4a10 msgr2=0x7f011c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.799+0000 7f0122165640 1 --2- 172.21.15.57:0/2190116219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f011c0c4a10 0x7f011c15cf90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f011c0b1d40 tx=0x7f011001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.799+0000 7f0122165640 1 -- 172.21.15.57:0/2190116219 shutdown_connections 2023-10-30T14:26:43.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.799+0000 7f0122165640 1 --2- 172.21.15.57:0/2190116219 >> v2:172.21.15.88:3300/0 conn(0x7f011c0c42a0 0x7f011c0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.799+0000 7f0122165640 1 --2- 172.21.15.57:0/2190116219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f011c0c4a10 0x7f011c15cf90 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.799+0000 7f0122165640 1 -- 172.21.15.57:0/2190116219 >> 172.21.15.57:0/2190116219 conn(0x7f011c0715c0 msgr2=0x7f011c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:43.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.799+0000 7f0122165640 1 -- 172.21.15.57:0/2190116219 shutdown_connections 2023-10-30T14:26:43.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.799+0000 7f0122165640 1 -- 172.21.15.57:0/2190116219 wait complete. 2023-10-30T14:26:43.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f0122165640 1 Processor -- start 2023-10-30T14:26:43.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f0122165640 1 -- start start 2023-10-30T14:26:43.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f0122165640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f011c0c42a0 0x7f011c150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f0122165640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f011c0c4a10 0x7f011c0cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f0122165640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f011c153c80 0x7f011c149e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f0122165640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011c151940 con 0x7f011c0c42a0 2023-10-30T14:26:43.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f0122165640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011c14fd40 con 0x7f011c153c80 2023-10-30T14:26:43.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f0122165640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f011c150020 con 0x7f011c0c4a10 2023-10-30T14:26:43.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f011b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f011c0c42a0 0x7f011c150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f011b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f011c0c42a0 0x7f011c150ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54928/0 (socket says 172.21.15.57:54928) 2023-10-30T14:26:43.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f011b7fe640 1 -- 172.21.15.57:0/1602043707 learned_addr learned my addr 172.21.15.57:0/1602043707 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:43.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f011affd640 1 --2- 172.21.15.57:0/1602043707 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f011c0c4a10 0x7f011c0cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f011bfff640 1 --2- 172.21.15.57:0/1602043707 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f011c153c80 0x7f011c149e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f011b7fe640 1 -- 172.21.15.57:0/1602043707 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f011c0c4a10 msgr2=0x7f011c0cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f011b7fe640 1 --2- 172.21.15.57:0/1602043707 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f011c0c4a10 0x7f011c0cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f011b7fe640 1 -- 172.21.15.57:0/1602043707 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f011c153c80 msgr2=0x7f011c149e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f011b7fe640 1 --2- 172.21.15.57:0/1602043707 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f011c153c80 0x7f011c149e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f011b7fe640 1 -- 172.21.15.57:0/1602043707 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01100094d0 con 0x7f011c0c42a0 2023-10-30T14:26:43.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f011b7fe640 1 --2- 172.21.15.57:0/1602043707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f011c0c42a0 0x7f011c150ff0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f010c00c3c0 tx=0x7f010c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:43.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f0103fff640 1 -- 172.21.15.57:0/1602043707 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c0052d0 con 0x7f011c0c42a0 2023-10-30T14:26:43.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f0103fff640 1 -- 172.21.15.57:0/1602043707 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010c005430 con 0x7f011c0c42a0 2023-10-30T14:26:43.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f0103fff640 1 -- 172.21.15.57:0/1602043707 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c003840 con 0x7f011c0c42a0 2023-10-30T14:26:43.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f0122165640 1 -- 172.21.15.57:0/1602043707 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f011c1f1f10 con 0x7f011c0c42a0 2023-10-30T14:26:43.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.803+0000 7f0122165640 1 -- 172.21.15.57:0/1602043707 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f011c1f24d0 con 0x7f011c0c42a0 2023-10-30T14:26:43.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.807+0000 7f0122165640 1 -- 172.21.15.57:0/1602043707 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f011c04e170 con 0x7f011c0c42a0 2023-10-30T14:26:43.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.807+0000 7f0103fff640 1 -- 172.21.15.57:0/1602043707 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f010c0039a0 con 0x7f011c0c42a0 2023-10-30T14:26:43.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.807+0000 7f0103fff640 1 --2- 172.21.15.57:0/1602043707 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f01080664a0 0x7f0108068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.807+0000 7f0103fff640 1 -- 172.21.15.57:0/1602043707 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f010c0837a0 con 0x7f011c0c42a0 2023-10-30T14:26:43.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.807+0000 7f011affd640 1 --2- 172.21.15.57:0/1602043707 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f01080664a0 0x7f0108068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.807+0000 7f011affd640 1 --2- 172.21.15.57:0/1602043707 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f01080664a0 0x7f0108068960 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f01100b2000 tx=0x7f0110008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:43.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:43.811+0000 7f0103fff640 1 -- 172.21.15.57:0/1602043707 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f010c050820 con 0x7f011c0c42a0 2023-10-30T14:26:44.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:44.003+0000 7f0122165640 1 -- 172.21.15.57:0/1602043707 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f011c0cad40 con 0x7f011c0c42a0 2023-10-30T14:26:44.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:44.003+0000 7f0103fff640 1 -- 172.21.15.57:0/1602043707 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5718 (secure 0 0 0) 0x7f010c053e40 con 0x7f011c0c42a0 2023-10-30T14:26:44.006 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:44.006 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:26:38.032694+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:44.007 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 75 2023-10-30T14:26:44.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:44.003+0000 7f0122165640 1 -- 172.21.15.57:0/1602043707 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f01080664a0 msgr2=0x7f0108068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:44.003+0000 7f0122165640 1 --2- 172.21.15.57:0/1602043707 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f01080664a0 0x7f0108068960 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f01100b2000 tx=0x7f0110008040 comp rx=0 tx=0).stop 2023-10-30T14:26:44.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:44.003+0000 7f0122165640 1 -- 172.21.15.57:0/1602043707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f011c0c42a0 msgr2=0x7f011c150ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:44.003+0000 7f0122165640 1 --2- 172.21.15.57:0/1602043707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f011c0c42a0 0x7f011c150ff0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f010c00c3c0 tx=0x7f010c00c880 comp rx=0 tx=0).stop 2023-10-30T14:26:44.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:44.003+0000 7f0122165640 1 -- 172.21.15.57:0/1602043707 shutdown_connections 2023-10-30T14:26:44.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:44.003+0000 7f0122165640 1 --2- 172.21.15.57:0/1602043707 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f01080664a0 0x7f0108068960 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:44.003+0000 7f0122165640 1 --2- 172.21.15.57:0/1602043707 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f011c153c80 0x7f011c149e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:44.003+0000 7f0122165640 1 --2- 172.21.15.57:0/1602043707 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f011c0c4a10 0x7f011c0cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:44.003+0000 7f0122165640 1 --2- 172.21.15.57:0/1602043707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f011c0c42a0 0x7f011c150ff0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:44.003+0000 7f0122165640 1 -- 172.21.15.57:0/1602043707 >> 172.21.15.57:0/1602043707 conn(0x7f011c0715c0 msgr2=0x7f011c0c4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:44.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:44.003+0000 7f0122165640 1 -- 172.21.15.57:0/1602043707 shutdown_connections 2023-10-30T14:26:44.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:44.003+0000 7f0122165640 1 -- 172.21.15.57:0/1602043707 wait complete. 2023-10-30T14:26:44.028 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:26:46.029 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:46.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.179+0000 7f1022772640 1 Processor -- start 2023-10-30T14:26:46.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.179+0000 7f1022772640 1 -- start start 2023-10-30T14:26:46.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.179+0000 7f1022772640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f101c153bb0 0x7f101c0c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:46.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.179+0000 7f1022772640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f101c159980 0x7f101c159df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:46.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.179+0000 7f1022772640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101c0c5f80 con 0x7f101c06c420 2023-10-30T14:26:46.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.179+0000 7f1022772640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f101c15a330 con 0x7f101c159980 2023-10-30T14:26:46.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.179+0000 7f1022772640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f101c0c5690 con 0x7f101c153bb0 2023-10-30T14:26:46.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.179+0000 7f1020d0f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f101c159980 0x7f101c159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:46.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.179+0000 7f101b7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f101c06c420 0x7f101c0c6c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44088/0 (socket says 172.21.15.57:44088) 2023-10-30T14:26:46.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.179+0000 7f101b7fe640 1 -- 172.21.15.57:0/1675130361 learned_addr learned my addr 172.21.15.57:0/1675130361 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:26:46.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.179+0000 7f101bfff640 1 --2- 172.21.15.57:0/1675130361 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f101c153bb0 0x7f101c0c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:46.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.179+0000 7f1020d0f640 1 -- 172.21.15.57:0/1675130361 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f101c153bb0 msgr2=0x7f101c0c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:46.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.179+0000 7f1020d0f640 1 --2- 172.21.15.57:0/1675130361 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f101c153bb0 0x7f101c0c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:46.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.179+0000 7f1020d0f640 1 -- 172.21.15.57:0/1675130361 >> v1:172.21.15.57:6789/0 conn(0x7f101c06c420 legacy=0x7f101c0c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:46.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.179+0000 7f1020d0f640 1 -- 172.21.15.57:0/1675130361 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c15a580 con 0x7f101c159980 2023-10-30T14:26:46.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.179+0000 7f101bfff640 1 --2- 172.21.15.57:0/1675130361 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f101c153bb0 0x7f101c0c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:46.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.183+0000 7f1020d0f640 1 --2- 172.21.15.57:0/1675130361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f101c159980 0x7f101c159df0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f1014009d90 tx=0x7f101401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68efeac11a7c52ac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:46.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.183+0000 7f101affd640 1 -- 172.21.15.57:0/1675130361 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10140af600 con 0x7f101c159980 2023-10-30T14:26:46.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.183+0000 7f101affd640 1 -- 172.21.15.57:0/1675130361 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10140ab050 con 0x7f101c159980 2023-10-30T14:26:46.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.183+0000 7f101affd640 1 -- 172.21.15.57:0/1675130361 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10140af980 con 0x7f101c159980 2023-10-30T14:26:46.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.183+0000 7f1022772640 1 -- 172.21.15.57:0/1675130361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f101c159980 msgr2=0x7f101c159df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:46.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.183+0000 7f1022772640 1 --2- 172.21.15.57:0/1675130361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f101c159980 0x7f101c159df0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f1014009d90 tx=0x7f101401f130 comp rx=0 tx=0).stop 2023-10-30T14:26:46.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.183+0000 7f1022772640 1 -- 172.21.15.57:0/1675130361 shutdown_connections 2023-10-30T14:26:46.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.183+0000 7f1022772640 1 --2- 172.21.15.57:0/1675130361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f101c159980 0x7f101c159df0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:46.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.183+0000 7f1022772640 1 --2- 172.21.15.57:0/1675130361 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f101c153bb0 0x7f101c0c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:46.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.183+0000 7f1022772640 1 -- 172.21.15.57:0/1675130361 >> 172.21.15.57:0/1675130361 conn(0x7f101c0cac70 msgr2=0x7f101c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:46.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.183+0000 7f1022772640 1 -- 172.21.15.57:0/1675130361 shutdown_connections 2023-10-30T14:26:46.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.183+0000 7f1022772640 1 -- 172.21.15.57:0/1675130361 wait complete. 2023-10-30T14:26:46.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f1022772640 1 Processor -- start 2023-10-30T14:26:46.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f1022772640 1 -- start start 2023-10-30T14:26:46.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f1022772640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f101c06c420 0x7f101c1ee690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:46.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f1022772640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f101c153bb0 0x7f101c150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:46.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f1022772640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f101c159980 0x7f101c0bc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:46.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f1022772640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101c0bd250 con 0x7f101c06c420 2023-10-30T14:26:46.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f1022772640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101c0bd440 con 0x7f101c159980 2023-10-30T14:26:46.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f1022772640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f101c0bd720 con 0x7f101c153bb0 2023-10-30T14:26:46.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f101bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f101c06c420 0x7f101c1ee690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:46.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f101bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f101c06c420 0x7f101c1ee690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54948/0 (socket says 172.21.15.57:54948) 2023-10-30T14:26:46.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f101bfff640 1 -- 172.21.15.57:0/1306648013 learned_addr learned my addr 172.21.15.57:0/1306648013 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:46.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f101b7fe640 1 --2- 172.21.15.57:0/1306648013 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f101c153bb0 0x7f101c150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:46.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f1020d0f640 1 --2- 172.21.15.57:0/1306648013 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f101c159980 0x7f101c0bc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:46.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f101bfff640 1 -- 172.21.15.57:0/1306648013 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f101c153bb0 msgr2=0x7f101c150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:46.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f101bfff640 1 --2- 172.21.15.57:0/1306648013 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f101c153bb0 0x7f101c150960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:46.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f101bfff640 1 -- 172.21.15.57:0/1306648013 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f101c159980 msgr2=0x7f101c0bc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:46.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f101bfff640 1 --2- 172.21.15.57:0/1306648013 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f101c159980 0x7f101c0bc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:46.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f101bfff640 1 -- 172.21.15.57:0/1306648013 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1014009a70 con 0x7f101c06c420 2023-10-30T14:26:46.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f101b7fe640 1 --2- 172.21.15.57:0/1306648013 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f101c153bb0 0x7f101c150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:46.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f101bfff640 1 --2- 172.21.15.57:0/1306648013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f101c06c420 0x7f101c1ee690 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f100c00d800 tx=0x7f100c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:46.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f1018ff9640 1 -- 172.21.15.57:0/1306648013 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c0096c0 con 0x7f101c06c420 2023-10-30T14:26:46.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f1018ff9640 1 -- 172.21.15.57:0/1306648013 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f100c009820 con 0x7f101c06c420 2023-10-30T14:26:46.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f1022772640 1 -- 172.21.15.57:0/1306648013 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f101c0bda00 con 0x7f101c06c420 2023-10-30T14:26:46.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f1018ff9640 1 -- 172.21.15.57:0/1306648013 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c009bd0 con 0x7f101c06c420 2023-10-30T14:26:46.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f1022772640 1 -- 172.21.15.57:0/1306648013 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f101c0bdfc0 con 0x7f101c06c420 2023-10-30T14:26:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.187+0000 7f1022772640 1 -- 172.21.15.57:0/1306648013 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f101c04e170 con 0x7f101c06c420 2023-10-30T14:26:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.191+0000 7f1018ff9640 1 -- 172.21.15.57:0/1306648013 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f100c009d30 con 0x7f101c06c420 2023-10-30T14:26:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.191+0000 7f1018ff9640 1 --2- 172.21.15.57:0/1306648013 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f10080664d0 0x7f1008068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.191+0000 7f1018ff9640 1 -- 172.21.15.57:0/1306648013 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f100c084810 con 0x7f101c06c420 2023-10-30T14:26:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.191+0000 7f101b7fe640 1 --2- 172.21.15.57:0/1306648013 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f10080664d0 0x7f1008068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.191+0000 7f101b7fe640 1 --2- 172.21.15.57:0/1306648013 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f10080664d0 0x7f1008068990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f101c06d430 tx=0x7f1010009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.195+0000 7f1018ff9640 1 -- 172.21.15.57:0/1306648013 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f100c0519b0 con 0x7f101c06c420 2023-10-30T14:26:46.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.383+0000 7f1022772640 1 -- 172.21.15.57:0/1306648013 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f101c0c5be0 con 0x7f101c06c420 2023-10-30T14:26:46.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.383+0000 7f1018ff9640 1 -- 172.21.15.57:0/1306648013 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5715 (secure 0 0 0) 0x7f100c054fd0 con 0x7f101c06c420 2023-10-30T14:26:46.387 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:46.387 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:26:44.047443+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:46.388 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 76 2023-10-30T14:26:46.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.383+0000 7f1022772640 1 -- 172.21.15.57:0/1306648013 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f10080664d0 msgr2=0x7f1008068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:46.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.383+0000 7f1022772640 1 --2- 172.21.15.57:0/1306648013 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f10080664d0 0x7f1008068990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f101c06d430 tx=0x7f1010009040 comp rx=0 tx=0).stop 2023-10-30T14:26:46.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.383+0000 7f1022772640 1 -- 172.21.15.57:0/1306648013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f101c06c420 msgr2=0x7f101c1ee690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:46.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.383+0000 7f1022772640 1 --2- 172.21.15.57:0/1306648013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f101c06c420 0x7f101c1ee690 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f100c00d800 tx=0x7f100c00dcc0 comp rx=0 tx=0).stop 2023-10-30T14:26:46.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.387+0000 7f1022772640 1 -- 172.21.15.57:0/1306648013 shutdown_connections 2023-10-30T14:26:46.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.387+0000 7f1022772640 1 --2- 172.21.15.57:0/1306648013 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f10080664d0 0x7f1008068990 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:46.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.387+0000 7f1022772640 1 --2- 172.21.15.57:0/1306648013 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f101c159980 0x7f101c0bc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:46.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.387+0000 7f1022772640 1 --2- 172.21.15.57:0/1306648013 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f101c153bb0 0x7f101c150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:46.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.387+0000 7f1022772640 1 --2- 172.21.15.57:0/1306648013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f101c06c420 0x7f101c1ee690 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:46.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.387+0000 7f1022772640 1 -- 172.21.15.57:0/1306648013 >> 172.21.15.57:0/1306648013 conn(0x7f101c0cac70 msgr2=0x7f101c0c49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:46.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.387+0000 7f1022772640 1 -- 172.21.15.57:0/1306648013 shutdown_connections 2023-10-30T14:26:46.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:46.387+0000 7f1022772640 1 -- 172.21.15.57:0/1306648013 wait complete. 2023-10-30T14:26:46.409 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:26:48.410 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:48.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.559+0000 7f972f11d640 1 Processor -- start 2023-10-30T14:26:48.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.559+0000 7f972f11d640 1 -- start start 2023-10-30T14:26:48.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.563+0000 7f972f11d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f97280cb7d0 0x7f97280c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:48.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.563+0000 7f972f11d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97280c3e40 0x7f9728070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:48.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.563+0000 7f972f11d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97280c2310 con 0x7f9728150830 2023-10-30T14:26:48.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.563+0000 7f972f11d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f972806bc10 con 0x7f97280cb7d0 2023-10-30T14:26:48.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.563+0000 7f972f11d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9728156710 con 0x7f97280c3e40 2023-10-30T14:26:48.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.563+0000 7f9727fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f97280cb7d0 0x7f97280c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:48.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.563+0000 7f972d6ba640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97280c3e40 0x7f9728070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:48.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.563+0000 7f9727fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f97280cb7d0 0x7f97280c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:40070/0 (socket says 172.21.15.57:40070) 2023-10-30T14:26:48.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.563+0000 7f972d6ba640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97280c3e40 0x7f9728070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:34506/0 (socket says 172.21.15.57:34506) 2023-10-30T14:26:48.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.563+0000 7f9727fff640 1 -- 172.21.15.57:0/3471991225 learned_addr learned my addr 172.21.15.57:0/3471991225 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:48.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.563+0000 7f9727fff640 1 -- 172.21.15.57:0/3471991225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97280c3e40 msgr2=0x7f9728070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:48.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.563+0000 7f9727fff640 1 --2- 172.21.15.57:0/3471991225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97280c3e40 0x7f9728070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:48.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.563+0000 7f9727fff640 1 -- 172.21.15.57:0/3471991225 >> v1:172.21.15.88:6789/0 conn(0x7f9728150830 legacy=0x7f972806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:48.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.563+0000 7f9727fff640 1 -- 172.21.15.57:0/3471991225 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97280c4360 con 0x7f97280cb7d0 2023-10-30T14:26:48.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.563+0000 7f972d6ba640 1 --2- 172.21.15.57:0/3471991225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97280c3e40 0x7f9728070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:26:48.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.567+0000 7f9727fff640 1 --2- 172.21.15.57:0/3471991225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97280cb7d0 0x7f97280c3900 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9714009870 tx=0x7f9714020f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49aac7b4af9069d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:48.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.567+0000 7f97277fe640 1 -- 172.21.15.57:0/3471991225 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97140aac40 con 0x7f97280cb7d0 2023-10-30T14:26:48.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.567+0000 7f97277fe640 1 -- 172.21.15.57:0/3471991225 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97140aada0 con 0x7f97280cb7d0 2023-10-30T14:26:48.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.567+0000 7f97277fe640 1 -- 172.21.15.57:0/3471991225 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97140b2660 con 0x7f97280cb7d0 2023-10-30T14:26:48.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.567+0000 7f972f11d640 1 -- 172.21.15.57:0/3471991225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97280cb7d0 msgr2=0x7f97280c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:48.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.567+0000 7f972f11d640 1 --2- 172.21.15.57:0/3471991225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97280cb7d0 0x7f97280c3900 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9714009870 tx=0x7f9714020f40 comp rx=0 tx=0).stop 2023-10-30T14:26:48.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.567+0000 7f972f11d640 1 -- 172.21.15.57:0/3471991225 shutdown_connections 2023-10-30T14:26:48.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.567+0000 7f972f11d640 1 --2- 172.21.15.57:0/3471991225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97280c3e40 0x7f9728070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:48.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.567+0000 7f972f11d640 1 --2- 172.21.15.57:0/3471991225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97280cb7d0 0x7f97280c3900 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:48.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.567+0000 7f972f11d640 1 -- 172.21.15.57:0/3471991225 >> 172.21.15.57:0/3471991225 conn(0x7f97280cac70 msgr2=0x7f97280cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:48.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.567+0000 7f972f11d640 1 -- 172.21.15.57:0/3471991225 shutdown_connections 2023-10-30T14:26:48.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.567+0000 7f972f11d640 1 -- 172.21.15.57:0/3471991225 wait complete. 2023-10-30T14:26:48.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.567+0000 7f972f11d640 1 Processor -- start 2023-10-30T14:26:48.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.567+0000 7f972f11d640 1 -- start start 2023-10-30T14:26:48.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f972f11d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97280c3e40 0x7f9728148aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:48.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f972f11d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97280cb7d0 0x7f9728148fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:48.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f972f11d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9728150830 0x7f9728157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:48.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f972f11d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9728158910 con 0x7f9728150830 2023-10-30T14:26:48.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f972f11d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97281485c0 con 0x7f97280cb7d0 2023-10-30T14:26:48.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f972f11d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97281eac50 con 0x7f97280c3e40 2023-10-30T14:26:48.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f972d6ba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9728150830 0x7f9728157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:48.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f972d6ba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9728150830 0x7f9728157a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54962/0 (socket says 172.21.15.57:54962) 2023-10-30T14:26:48.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f972d6ba640 1 -- 172.21.15.57:0/2008360747 learned_addr learned my addr 172.21.15.57:0/2008360747 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:48.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f972ceb9640 1 --2- 172.21.15.57:0/2008360747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97280c3e40 0x7f9728148aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:48.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f9727fff640 1 --2- 172.21.15.57:0/2008360747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97280cb7d0 0x7f9728148fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:48.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f972d6ba640 1 -- 172.21.15.57:0/2008360747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97280c3e40 msgr2=0x7f9728148aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:48.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f972d6ba640 1 --2- 172.21.15.57:0/2008360747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97280c3e40 0x7f9728148aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:48.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f972d6ba640 1 -- 172.21.15.57:0/2008360747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97280cb7d0 msgr2=0x7f9728148fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:48.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f972d6ba640 1 --2- 172.21.15.57:0/2008360747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97280cb7d0 0x7f9728148fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:48.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f972d6ba640 1 -- 172.21.15.57:0/2008360747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9720009a70 con 0x7f9728150830 2023-10-30T14:26:48.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f972d6ba640 1 --2- 172.21.15.57:0/2008360747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9728150830 0x7f9728157a20 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f972000efb0 tx=0x7f972000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:48.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f97257fa640 1 -- 172.21.15.57:0/2008360747 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9720009d50 con 0x7f9728150830 2023-10-30T14:26:48.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f97257fa640 1 -- 172.21.15.57:0/2008360747 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972000f040 con 0x7f9728150830 2023-10-30T14:26:48.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f97257fa640 1 -- 172.21.15.57:0/2008360747 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97200146a0 con 0x7f9728150830 2023-10-30T14:26:48.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f972f11d640 1 -- 172.21.15.57:0/2008360747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97140094d0 con 0x7f9728150830 2023-10-30T14:26:48.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f972f11d640 1 -- 172.21.15.57:0/2008360747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97281eb1f0 con 0x7f9728150830 2023-10-30T14:26:48.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.571+0000 7f972f11d640 1 -- 172.21.15.57:0/2008360747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9728148780 con 0x7f9728150830 2023-10-30T14:26:48.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.575+0000 7f97257fa640 1 -- 172.21.15.57:0/2008360747 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9720014800 con 0x7f9728150830 2023-10-30T14:26:48.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.575+0000 7f97257fa640 1 --2- 172.21.15.57:0/2008360747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f970c0664a0 0x7f970c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:48.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.575+0000 7f97257fa640 1 -- 172.21.15.57:0/2008360747 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f97200850d0 con 0x7f9728150830 2023-10-30T14:26:48.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.575+0000 7f972ceb9640 1 --2- 172.21.15.57:0/2008360747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f970c0664a0 0x7f970c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:48.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.575+0000 7f972ceb9640 1 --2- 172.21.15.57:0/2008360747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f970c0664a0 0x7f970c068960 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f971c007fd0 tx=0x7f971c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:48.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.579+0000 7f97257fa640 1 -- 172.21.15.57:0/2008360747 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9720052290 con 0x7f9728150830 2023-10-30T14:26:48.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.767+0000 7f972f11d640 1 -- 172.21.15.57:0/2008360747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f972804e170 con 0x7f9728150830 2023-10-30T14:26:48.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.767+0000 7f97257fa640 1 -- 172.21.15.57:0/2008360747 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5715 (secure 0 0 0) 0x7f97200558b0 con 0x7f9728150830 2023-10-30T14:26:48.771 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:48.771 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:26:44.047443+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:48.772 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 76 2023-10-30T14:26:48.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.767+0000 7f972f11d640 1 -- 172.21.15.57:0/2008360747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f970c0664a0 msgr2=0x7f970c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:48.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.767+0000 7f972f11d640 1 --2- 172.21.15.57:0/2008360747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f970c0664a0 0x7f970c068960 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f971c007fd0 tx=0x7f971c009040 comp rx=0 tx=0).stop 2023-10-30T14:26:48.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.767+0000 7f972f11d640 1 -- 172.21.15.57:0/2008360747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9728150830 msgr2=0x7f9728157a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:48.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.767+0000 7f972f11d640 1 --2- 172.21.15.57:0/2008360747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9728150830 0x7f9728157a20 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f972000efb0 tx=0x7f972000c490 comp rx=0 tx=0).stop 2023-10-30T14:26:48.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.771+0000 7f972f11d640 1 -- 172.21.15.57:0/2008360747 shutdown_connections 2023-10-30T14:26:48.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.771+0000 7f972f11d640 1 --2- 172.21.15.57:0/2008360747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f970c0664a0 0x7f970c068960 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:48.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.771+0000 7f972f11d640 1 --2- 172.21.15.57:0/2008360747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9728150830 0x7f9728157a20 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:48.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.771+0000 7f972f11d640 1 --2- 172.21.15.57:0/2008360747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97280cb7d0 0x7f9728148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:48.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.771+0000 7f972f11d640 1 --2- 172.21.15.57:0/2008360747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97280c3e40 0x7f9728148aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:48.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.771+0000 7f972f11d640 1 -- 172.21.15.57:0/2008360747 >> 172.21.15.57:0/2008360747 conn(0x7f97280cac70 msgr2=0x7f9728070520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:48.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.771+0000 7f972f11d640 1 -- 172.21.15.57:0/2008360747 shutdown_connections 2023-10-30T14:26:48.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:48.771+0000 7f972f11d640 1 -- 172.21.15.57:0/2008360747 wait complete. 2023-10-30T14:26:48.793 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:26:50.795 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:50.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.947+0000 7f6da6794640 1 Processor -- start 2023-10-30T14:26:50.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.947+0000 7f6da6794640 1 -- start start 2023-10-30T14:26:50.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.947+0000 7f6da6794640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6da0150830 0x7f6da006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:50.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.947+0000 7f6da6794640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6da00cb7d0 0x7f6da00c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:50.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.947+0000 7f6da6794640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6da00c3e40 0x7f6da0070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:50.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.947+0000 7f6da6794640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6da00c2310 con 0x7f6da0150830 2023-10-30T14:26:50.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.947+0000 7f6da6794640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6da0156710 con 0x7f6da00c3e40 2023-10-30T14:26:50.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.947+0000 7f6da6794640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da00c4360 con 0x7f6da00cb7d0 2023-10-30T14:26:50.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.951+0000 7f6d9ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6da0150830 0x7f6da006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:50.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.951+0000 7f6d9ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6da0150830 0x7f6da006b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54966/0 (socket says 172.21.15.57:54966) 2023-10-30T14:26:50.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.951+0000 7f6d9ffff640 1 -- 172.21.15.57:0/315020099 learned_addr learned my addr 172.21.15.57:0/315020099 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:50.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.951+0000 7f6da4d31640 1 --2- 172.21.15.57:0/315020099 >> v2:172.21.15.88:3300/0 conn(0x7f6da00c3e40 0x7f6da0070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:50.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.951+0000 7f6d9f7fe640 1 --2- 172.21.15.57:0/315020099 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6da00cb7d0 0x7f6da00c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:50.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.951+0000 7f6d9ffff640 1 -- 172.21.15.57:0/315020099 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6da00cb7d0 msgr2=0x7f6da00c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:50.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.951+0000 7f6d9ffff640 1 --2- 172.21.15.57:0/315020099 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6da00cb7d0 0x7f6da00c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:50.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.951+0000 7f6d9ffff640 1 -- 172.21.15.57:0/315020099 >> v2:172.21.15.88:3300/0 conn(0x7f6da00c3e40 msgr2=0x7f6da0070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:50.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.951+0000 7f6d9ffff640 1 --2- 172.21.15.57:0/315020099 >> v2:172.21.15.88:3300/0 conn(0x7f6da00c3e40 0x7f6da0070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:50.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.951+0000 7f6d9ffff640 1 -- 172.21.15.57:0/315020099 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da01569b0 con 0x7f6da0150830 2023-10-30T14:26:50.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.951+0000 7f6d9ffff640 1 --2- 172.21.15.57:0/315020099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6da0150830 0x7f6da006b500 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f6da006fc60 tx=0x7f6d900213c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e791233ab4fede13 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:50.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.951+0000 7f6d9effd640 1 -- 172.21.15.57:0/315020099 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d900b1600 con 0x7f6da0150830 2023-10-30T14:26:50.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6d9effd640 1 -- 172.21.15.57:0/315020099 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d900ad050 con 0x7f6da0150830 2023-10-30T14:26:50.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6d9effd640 1 -- 172.21.15.57:0/315020099 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d900b1980 con 0x7f6da0150830 2023-10-30T14:26:50.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6da6794640 1 -- 172.21.15.57:0/315020099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6da0150830 msgr2=0x7f6da006b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:50.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6da6794640 1 --2- 172.21.15.57:0/315020099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6da0150830 0x7f6da006b500 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f6da006fc60 tx=0x7f6d900213c0 comp rx=0 tx=0).stop 2023-10-30T14:26:50.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6da6794640 1 -- 172.21.15.57:0/315020099 shutdown_connections 2023-10-30T14:26:50.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6da6794640 1 --2- 172.21.15.57:0/315020099 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6da00cb7d0 0x7f6da00c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:50.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6da6794640 1 --2- 172.21.15.57:0/315020099 >> v2:172.21.15.88:3300/0 conn(0x7f6da00c3e40 0x7f6da0070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:50.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6da6794640 1 --2- 172.21.15.57:0/315020099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6da0150830 0x7f6da006b500 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:50.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6da6794640 1 -- 172.21.15.57:0/315020099 >> 172.21.15.57:0/315020099 conn(0x7f6da00cac70 msgr2=0x7f6da00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:50.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6da6794640 1 -- 172.21.15.57:0/315020099 shutdown_connections 2023-10-30T14:26:50.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6da6794640 1 -- 172.21.15.57:0/315020099 wait complete. 2023-10-30T14:26:50.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6da6794640 1 Processor -- start 2023-10-30T14:26:50.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6da6794640 1 -- start start 2023-10-30T14:26:50.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6da6794640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6da00c3e40 0x7f6da0151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:50.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6da6794640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6da00cb7d0 0x7f6da00ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:50.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6da6794640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6da0150830 0x7f6da01f6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:50.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6da6794640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da0153620 con 0x7f6da00cb7d0 2023-10-30T14:26:50.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6da6794640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da01523d0 con 0x7f6da0150830 2023-10-30T14:26:50.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6da6794640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da0157fe0 con 0x7f6da00c3e40 2023-10-30T14:26:50.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6d9f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6da00cb7d0 0x7f6da00ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:50.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6d9f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6da00cb7d0 0x7f6da00ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54972/0 (socket says 172.21.15.57:54972) 2023-10-30T14:26:50.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.955+0000 7f6d9f7fe640 1 -- 172.21.15.57:0/1770238963 learned_addr learned my addr 172.21.15.57:0/1770238963 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:50.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.959+0000 7f6d9ffff640 1 --2- 172.21.15.57:0/1770238963 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6da00c3e40 0x7f6da0151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:50.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.959+0000 7f6da4d31640 1 --2- 172.21.15.57:0/1770238963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6da0150830 0x7f6da01f6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:50.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.959+0000 7f6d9f7fe640 1 -- 172.21.15.57:0/1770238963 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6da00c3e40 msgr2=0x7f6da0151460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:50.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.959+0000 7f6d9f7fe640 1 --2- 172.21.15.57:0/1770238963 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6da00c3e40 0x7f6da0151460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:50.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.959+0000 7f6d9f7fe640 1 -- 172.21.15.57:0/1770238963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6da0150830 msgr2=0x7f6da01f6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:50.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.959+0000 7f6d9f7fe640 1 --2- 172.21.15.57:0/1770238963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6da0150830 0x7f6da01f6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:50.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.959+0000 7f6d9f7fe640 1 -- 172.21.15.57:0/1770238963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9000b770 con 0x7f6da00cb7d0 2023-10-30T14:26:50.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.959+0000 7f6d9f7fe640 1 --2- 172.21.15.57:0/1770238963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6da00cb7d0 0x7f6da00ca2a0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6d9400e870 tx=0x7f6d9400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:50.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.959+0000 7f6d9cff9640 1 -- 172.21.15.57:0/1770238963 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d94009510 con 0x7f6da00cb7d0 2023-10-30T14:26:50.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.959+0000 7f6d9cff9640 1 -- 172.21.15.57:0/1770238963 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9400eec0 con 0x7f6da00cb7d0 2023-10-30T14:26:50.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.959+0000 7f6da6794640 1 -- 172.21.15.57:0/1770238963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6da01564a0 con 0x7f6da00cb7d0 2023-10-30T14:26:50.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.959+0000 7f6d9cff9640 1 -- 172.21.15.57:0/1770238963 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9400cb40 con 0x7f6da00cb7d0 2023-10-30T14:26:50.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.959+0000 7f6da6794640 1 -- 172.21.15.57:0/1770238963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da0155c00 con 0x7f6da00cb7d0 2023-10-30T14:26:50.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.959+0000 7f6da6794640 1 -- 172.21.15.57:0/1770238963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6da01f6230 con 0x7f6da00cb7d0 2023-10-30T14:26:50.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.959+0000 7f6d9cff9640 1 -- 172.21.15.57:0/1770238963 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6d9400cca0 con 0x7f6da00cb7d0 2023-10-30T14:26:50.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.959+0000 7f6d9cff9640 1 --2- 172.21.15.57:0/1770238963 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6d8c0664a0 0x7f6d8c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:50.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.963+0000 7f6d9cff9640 1 -- 172.21.15.57:0/1770238963 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f6d940846b0 con 0x7f6da00cb7d0 2023-10-30T14:26:50.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.963+0000 7f6d9ffff640 1 --2- 172.21.15.57:0/1770238963 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6d8c0664a0 0x7f6d8c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:50.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.963+0000 7f6d9ffff640 1 --2- 172.21.15.57:0/1770238963 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6d8c0664a0 0x7f6d8c068960 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6d900b3000 tx=0x7f6d900ac000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:50.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:50.963+0000 7f6d9cff9640 1 -- 172.21.15.57:0/1770238963 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6d940517e0 con 0x7f6da00cb7d0 2023-10-30T14:26:51.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:51.155+0000 7f6da6794640 1 -- 172.21.15.57:0/1770238963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6da004e170 con 0x7f6da00cb7d0 2023-10-30T14:26:51.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:51.155+0000 7f6d9cff9640 1 -- 172.21.15.57:0/1770238963 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+5721 (secure 0 0 0) 0x7f6d94054e00 con 0x7f6da00cb7d0 2023-10-30T14:26:51.161 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:51.161 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:26:49.113369+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:clientreplay","state_seq":41,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:51.161 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 77 2023-10-30T14:26:51.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:51.159+0000 7f6da6794640 1 -- 172.21.15.57:0/1770238963 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6d8c0664a0 msgr2=0x7f6d8c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:51.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:51.159+0000 7f6da6794640 1 --2- 172.21.15.57:0/1770238963 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6d8c0664a0 0x7f6d8c068960 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6d900b3000 tx=0x7f6d900ac000 comp rx=0 tx=0).stop 2023-10-30T14:26:51.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:51.159+0000 7f6da6794640 1 -- 172.21.15.57:0/1770238963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6da00cb7d0 msgr2=0x7f6da00ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:51.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:51.159+0000 7f6da6794640 1 --2- 172.21.15.57:0/1770238963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6da00cb7d0 0x7f6da00ca2a0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6d9400e870 tx=0x7f6d9400ec70 comp rx=0 tx=0).stop 2023-10-30T14:26:51.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:51.159+0000 7f6da6794640 1 -- 172.21.15.57:0/1770238963 shutdown_connections 2023-10-30T14:26:51.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:51.159+0000 7f6da6794640 1 --2- 172.21.15.57:0/1770238963 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6d8c0664a0 0x7f6d8c068960 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:51.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:51.159+0000 7f6da6794640 1 --2- 172.21.15.57:0/1770238963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6da0150830 0x7f6da01f6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:51.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:51.159+0000 7f6da6794640 1 --2- 172.21.15.57:0/1770238963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6da00cb7d0 0x7f6da00ca2a0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:51.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:51.159+0000 7f6da6794640 1 --2- 172.21.15.57:0/1770238963 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6da00c3e40 0x7f6da0151460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:51.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:51.159+0000 7f6da6794640 1 -- 172.21.15.57:0/1770238963 >> 172.21.15.57:0/1770238963 conn(0x7f6da00cac70 msgr2=0x7f6da00c54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:51.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:51.159+0000 7f6da6794640 1 -- 172.21.15.57:0/1770238963 shutdown_connections 2023-10-30T14:26:51.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:51.159+0000 7f6da6794640 1 -- 172.21.15.57:0/1770238963 wait complete. 2023-10-30T14:26:51.182 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 77, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 77, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5586": { "addr": "172.21.15.57:6837/862088802", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 862088802, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 862088802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5586, "incarnation": 67, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 51 }, "gid_5797": { "addr": "172.21.15.57:6834/2940934016", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2940934016, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2940934016, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5797, "incarnation": 73, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:clientreplay", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:26:49.113369+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5797, "mds_1": 5586 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6839/358521496", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 358521496, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 358521496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 66, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5315, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:26:53.183 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:53.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.335+0000 7fcef9446640 1 Processor -- start 2023-10-30T14:26:53.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.335+0000 7fcef9446640 1 -- start start 2023-10-30T14:26:53.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.335+0000 7fcef9446640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcef4153c80 0x7fcef415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:53.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.335+0000 7fcef9446640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcef40c4a10 0x7fcef40c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:53.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.335+0000 7fcef9446640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef4159f20 con 0x7fcef415a3b0 2023-10-30T14:26:53.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.335+0000 7fcef9446640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcef415dec0 con 0x7fcef40c4a10 2023-10-30T14:26:53.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.335+0000 7fcef9446640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcef41599a0 con 0x7fcef4153c80 2023-10-30T14:26:53.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.335+0000 7fceebfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcef40c4a10 0x7fcef40c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:53.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.335+0000 7fcef37fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fcef415a3b0 0x7fcef40ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44090/0 (socket says 172.21.15.57:44090) 2023-10-30T14:26:53.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.335+0000 7fcef37fe640 1 -- 172.21.15.57:0/3130876134 learned_addr learned my addr 172.21.15.57:0/3130876134 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:26:53.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.335+0000 7fcef2ffd640 1 --2- 172.21.15.57:0/3130876134 >> v2:172.21.15.88:3300/0 conn(0x7fcef4153c80 0x7fcef415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:53.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.335+0000 7fceebfff640 1 -- 172.21.15.57:0/3130876134 >> v2:172.21.15.88:3300/0 conn(0x7fcef4153c80 msgr2=0x7fcef415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:53.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.335+0000 7fceebfff640 1 --2- 172.21.15.57:0/3130876134 >> v2:172.21.15.88:3300/0 conn(0x7fcef4153c80 0x7fcef415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:53.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.335+0000 7fceebfff640 1 -- 172.21.15.57:0/3130876134 >> v1:172.21.15.57:6789/0 conn(0x7fcef415a3b0 legacy=0x7fcef40ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:53.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.335+0000 7fcef27fc640 1 -- 172.21.15.57:0/3130876134 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413972551 0 0) 0x7fcef4159f20 con 0x7fcef415a3b0 2023-10-30T14:26:53.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.335+0000 7fceebfff640 1 -- 172.21.15.57:0/3130876134 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef40c47e0 con 0x7fcef40c4a10 2023-10-30T14:26:53.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.339+0000 7fceebfff640 1 --2- 172.21.15.57:0/3130876134 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcef40c4a10 0x7fcef40c42a0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fcef40d00f0 tx=0x7fcedc01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31667aabb893d52 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:53.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.339+0000 7fcef27fc640 1 -- 172.21.15.57:0/3130876134 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcedc0a8950 con 0x7fcef40c4a10 2023-10-30T14:26:53.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.339+0000 7fcef27fc640 1 -- 172.21.15.57:0/3130876134 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcedc0a8ab0 con 0x7fcef40c4a10 2023-10-30T14:26:53.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.339+0000 7fcef27fc640 1 -- 172.21.15.57:0/3130876134 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcedc0a8e40 con 0x7fcef40c4a10 2023-10-30T14:26:53.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.339+0000 7fcef9446640 1 -- 172.21.15.57:0/3130876134 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcef40c4a10 msgr2=0x7fcef40c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:53.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.339+0000 7fcef9446640 1 --2- 172.21.15.57:0/3130876134 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcef40c4a10 0x7fcef40c42a0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fcef40d00f0 tx=0x7fcedc01ec50 comp rx=0 tx=0).stop 2023-10-30T14:26:53.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.339+0000 7fcef9446640 1 -- 172.21.15.57:0/3130876134 shutdown_connections 2023-10-30T14:26:53.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.339+0000 7fcef9446640 1 --2- 172.21.15.57:0/3130876134 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcef40c4a10 0x7fcef40c42a0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:53.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.339+0000 7fcef9446640 1 --2- 172.21.15.57:0/3130876134 >> v2:172.21.15.88:3300/0 conn(0x7fcef4153c80 0x7fcef415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:53.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.339+0000 7fcef9446640 1 -- 172.21.15.57:0/3130876134 >> 172.21.15.57:0/3130876134 conn(0x7fcef40715c0 msgr2=0x7fcef40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:53.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.339+0000 7fcef9446640 1 -- 172.21.15.57:0/3130876134 shutdown_connections 2023-10-30T14:26:53.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.339+0000 7fcef9446640 1 -- 172.21.15.57:0/3130876134 wait complete. 2023-10-30T14:26:53.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef9446640 1 Processor -- start 2023-10-30T14:26:53.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef9446640 1 -- start start 2023-10-30T14:26:53.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef9446640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcef40c4a10 0x7fcef414e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:53.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef9446640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef4153c80 0x7fcef4152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:53.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef9446640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcef415a3b0 0x7fcef41eea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:53.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef9446640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef4150ab0 con 0x7fcef40c4a10 2023-10-30T14:26:53.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef9446640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef4150d70 con 0x7fcef415a3b0 2023-10-30T14:26:53.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef9446640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcef41f1fe0 con 0x7fcef4153c80 2023-10-30T14:26:53.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcef40c4a10 0x7fcef414e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:53.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcef40c4a10 0x7fcef414e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54988/0 (socket says 172.21.15.57:54988) 2023-10-30T14:26:53.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef2ffd640 1 -- 172.21.15.57:0/3542768615 learned_addr learned my addr 172.21.15.57:0/3542768615 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:53.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fceebfff640 1 --2- 172.21.15.57:0/3542768615 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef4153c80 0x7fcef4152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:53.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef37fe640 1 --2- 172.21.15.57:0/3542768615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcef415a3b0 0x7fcef41eea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:53.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef2ffd640 1 -- 172.21.15.57:0/3542768615 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef4153c80 msgr2=0x7fcef4152e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:53.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef2ffd640 1 --2- 172.21.15.57:0/3542768615 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef4153c80 0x7fcef4152e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:53.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef2ffd640 1 -- 172.21.15.57:0/3542768615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcef415a3b0 msgr2=0x7fcef41eea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:53.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef2ffd640 1 --2- 172.21.15.57:0/3542768615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcef415a3b0 0x7fcef41eea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:53.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef2ffd640 1 -- 172.21.15.57:0/3542768615 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcedc009420 con 0x7fcef40c4a10 2023-10-30T14:26:53.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fceebfff640 1 --2- 172.21.15.57:0/3542768615 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef4153c80 0x7fcef4152e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:26:53.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef37fe640 1 --2- 172.21.15.57:0/3542768615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcef415a3b0 0x7fcef41eea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:26:53.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef2ffd640 1 --2- 172.21.15.57:0/3542768615 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcef40c4a10 0x7fcef414e240 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fcee400d800 tx=0x7fcee400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:53.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fceeb7fe640 1 -- 172.21.15.57:0/3542768615 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee400b6c0 con 0x7fcef40c4a10 2023-10-30T14:26:53.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fceeb7fe640 1 -- 172.21.15.57:0/3542768615 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee400b820 con 0x7fcef40c4a10 2023-10-30T14:26:53.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fceeb7fe640 1 -- 172.21.15.57:0/3542768615 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee400bb90 con 0x7fcef40c4a10 2023-10-30T14:26:53.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef9446640 1 -- 172.21.15.57:0/3542768615 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef41f22c0 con 0x7fcef40c4a10 2023-10-30T14:26:53.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.343+0000 7fcef9446640 1 -- 172.21.15.57:0/3542768615 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcef41f2880 con 0x7fcef40c4a10 2023-10-30T14:26:53.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.347+0000 7fcef9446640 1 -- 172.21.15.57:0/3542768615 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcef404e170 con 0x7fcef40c4a10 2023-10-30T14:26:53.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.347+0000 7fceeb7fe640 1 -- 172.21.15.57:0/3542768615 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcee400bcf0 con 0x7fcef40c4a10 2023-10-30T14:26:53.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.347+0000 7fceeb7fe640 1 --2- 172.21.15.57:0/3542768615 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fced40664a0 0x7fced4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:53.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.347+0000 7fceeb7fe640 1 -- 172.21.15.57:0/3542768615 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7fcee4084780 con 0x7fcef40c4a10 2023-10-30T14:26:53.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.347+0000 7fceebfff640 1 --2- 172.21.15.57:0/3542768615 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fced40664a0 0x7fced4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:53.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.351+0000 7fceebfff640 1 --2- 172.21.15.57:0/3542768615 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fced40664a0 0x7fced4068960 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fcedc0b2000 tx=0x7fcedc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:53.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.351+0000 7fceeb7fe640 1 -- 172.21.15.57:0/3542768615 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcee4051890 con 0x7fcef40c4a10 2023-10-30T14:26:53.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.539+0000 7fcef9446640 1 -- 172.21.15.57:0/3542768615 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcef40cad40 con 0x7fcef40c4a10 2023-10-30T14:26:53.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.539+0000 7fceeb7fe640 1 -- 172.21.15.57:0/3542768615 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+5721 (secure 0 0 0) 0x7fcee4011340 con 0x7fcef40c4a10 2023-10-30T14:26:53.545 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:53.545 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:26:49.113369+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:clientreplay","state_seq":41,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:53.546 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 77 2023-10-30T14:26:53.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.543+0000 7fcef9446640 1 -- 172.21.15.57:0/3542768615 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fced40664a0 msgr2=0x7fced4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:53.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.543+0000 7fcef9446640 1 --2- 172.21.15.57:0/3542768615 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fced40664a0 0x7fced4068960 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fcedc0b2000 tx=0x7fcedc008040 comp rx=0 tx=0).stop 2023-10-30T14:26:53.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.543+0000 7fcef9446640 1 -- 172.21.15.57:0/3542768615 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcef40c4a10 msgr2=0x7fcef414e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:53.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.543+0000 7fcef9446640 1 --2- 172.21.15.57:0/3542768615 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcef40c4a10 0x7fcef414e240 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fcee400d800 tx=0x7fcee400dcc0 comp rx=0 tx=0).stop 2023-10-30T14:26:53.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.543+0000 7fcef9446640 1 -- 172.21.15.57:0/3542768615 shutdown_connections 2023-10-30T14:26:53.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.543+0000 7fcef9446640 1 --2- 172.21.15.57:0/3542768615 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fced40664a0 0x7fced4068960 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:53.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.543+0000 7fcef9446640 1 --2- 172.21.15.57:0/3542768615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcef415a3b0 0x7fcef41eea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:53.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.543+0000 7fcef9446640 1 --2- 172.21.15.57:0/3542768615 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef4153c80 0x7fcef4152e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:53.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.543+0000 7fcef9446640 1 --2- 172.21.15.57:0/3542768615 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcef40c4a10 0x7fcef414e240 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:53.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.543+0000 7fcef9446640 1 -- 172.21.15.57:0/3542768615 >> 172.21.15.57:0/3542768615 conn(0x7fcef40715c0 msgr2=0x7fcef40cb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:53.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.543+0000 7fcef9446640 1 -- 172.21.15.57:0/3542768615 shutdown_connections 2023-10-30T14:26:53.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:53.543+0000 7fcef9446640 1 -- 172.21.15.57:0/3542768615 wait complete. 2023-10-30T14:26:53.567 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:26:55.568 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:55.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.743+0000 7fe60092b640 1 Processor -- start 2023-10-30T14:26:55.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.743+0000 7fe60092b640 1 -- start start 2023-10-30T14:26:55.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.743+0000 7fe60092b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe5fc0c62b0 0x7fe5fc1597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:55.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.743+0000 7fe60092b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe5fc0cb180 0x7fe5fc06ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:55.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.743+0000 7fe60092b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5fc15a770 con 0x7fe5fc153b50 2023-10-30T14:26:55.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.743+0000 7fe60092b640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5fc159cf0 con 0x7fe5fc0c62b0 2023-10-30T14:26:55.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.743+0000 7fe60092b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5fc06aa30 con 0x7fe5fc0cb180 2023-10-30T14:26:55.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.747+0000 7fe5faffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe5fc0c62b0 0x7fe5fc1597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:55.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.747+0000 7fe5fb7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fe5fc153b50 0x7fe5fc0ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:57736/0 (socket says 172.21.15.57:57736) 2023-10-30T14:26:55.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.747+0000 7fe5faffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe5fc0c62b0 0x7fe5fc1597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54330/0 (socket says 172.21.15.57:54330) 2023-10-30T14:26:55.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.747+0000 7fe5fb7fe640 1 -- 172.21.15.57:0/259441547 learned_addr learned my addr 172.21.15.57:0/259441547 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:26:55.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.747+0000 7fe5fbfff640 1 --2- 172.21.15.57:0/259441547 >> v2:172.21.15.88:3300/0 conn(0x7fe5fc0cb180 0x7fe5fc06ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:55.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.747+0000 7fe5faffd640 1 -- 172.21.15.57:0/259441547 >> v2:172.21.15.88:3300/0 conn(0x7fe5fc0cb180 msgr2=0x7fe5fc06ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:55.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.747+0000 7fe5faffd640 1 --2- 172.21.15.57:0/259441547 >> v2:172.21.15.88:3300/0 conn(0x7fe5fc0cb180 0x7fe5fc06ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:55.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.747+0000 7fe5faffd640 1 -- 172.21.15.57:0/259441547 >> v1:172.21.15.57:6789/0 conn(0x7fe5fc153b50 legacy=0x7fe5fc0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:55.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.747+0000 7fe5faffd640 1 -- 172.21.15.57:0/259441547 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5fc0c9d40 con 0x7fe5fc0c62b0 2023-10-30T14:26:55.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.747+0000 7fe5fa7fc640 1 -- 172.21.15.57:0/259441547 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289245533 0 0) 0x7fe5ec001120 con 0x7fe5fc153b50 2023-10-30T14:26:55.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.751+0000 7fe5faffd640 1 --2- 172.21.15.57:0/259441547 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5fc0c62b0 0x7fe5fc1597b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc0c5d20 tx=0x7fe5f001ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aacd39e264027b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:55.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.751+0000 7fe5fa7fc640 1 -- 172.21.15.57:0/259441547 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f00a8930 con 0x7fe5fc0c62b0 2023-10-30T14:26:55.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.751+0000 7fe5fa7fc640 1 -- 172.21.15.57:0/259441547 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f00a8a90 con 0x7fe5fc0c62b0 2023-10-30T14:26:55.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.751+0000 7fe5fa7fc640 1 -- 172.21.15.57:0/259441547 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f00a8e20 con 0x7fe5fc0c62b0 2023-10-30T14:26:55.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.751+0000 7fe60092b640 1 -- 172.21.15.57:0/259441547 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5fc0c62b0 msgr2=0x7fe5fc1597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:55.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.751+0000 7fe60092b640 1 --2- 172.21.15.57:0/259441547 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5fc0c62b0 0x7fe5fc1597b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc0c5d20 tx=0x7fe5f001ec30 comp rx=0 tx=0).stop 2023-10-30T14:26:55.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.751+0000 7fe60092b640 1 -- 172.21.15.57:0/259441547 shutdown_connections 2023-10-30T14:26:55.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.751+0000 7fe60092b640 1 --2- 172.21.15.57:0/259441547 >> v2:172.21.15.88:3300/0 conn(0x7fe5fc0cb180 0x7fe5fc06ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.751+0000 7fe60092b640 1 --2- 172.21.15.57:0/259441547 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5fc0c62b0 0x7fe5fc1597b0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.751+0000 7fe60092b640 1 -- 172.21.15.57:0/259441547 >> 172.21.15.57:0/259441547 conn(0x7fe5fc065a80 msgr2=0x7fe5fc065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.751+0000 7fe60092b640 1 -- 172.21.15.57:0/259441547 shutdown_connections 2023-10-30T14:26:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.751+0000 7fe60092b640 1 -- 172.21.15.57:0/259441547 wait complete. 2023-10-30T14:26:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe60092b640 1 Processor -- start 2023-10-30T14:26:55.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe60092b640 1 -- start start 2023-10-30T14:26:55.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe60092b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5fc0c62b0 0x7fe5fc0c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:55.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe60092b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5fc0cb180 0x7fe5fc1ee680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:55.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe60092b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5fc153b50 0x7fe5fc14c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:55.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe60092b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc0c3250 con 0x7fe5fc0c62b0 2023-10-30T14:26:55.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe60092b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc150a40 con 0x7fe5fc153b50 2023-10-30T14:26:55.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe60092b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc0c2c40 con 0x7fe5fc0cb180 2023-10-30T14:26:55.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe5fb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5fc0c62b0 0x7fe5fc0c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:55.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe5fb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5fc0c62b0 0x7fe5fc0c36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54332/0 (socket says 172.21.15.57:54332) 2023-10-30T14:26:55.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe5fb7fe640 1 -- 172.21.15.57:0/3102892456 learned_addr learned my addr 172.21.15.57:0/3102892456 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:55.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe5fbfff640 1 --2- 172.21.15.57:0/3102892456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5fc153b50 0x7fe5fc14c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:55.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe5faffd640 1 --2- 172.21.15.57:0/3102892456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5fc0cb180 0x7fe5fc1ee680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:55.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe5fb7fe640 1 -- 172.21.15.57:0/3102892456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5fc0cb180 msgr2=0x7fe5fc1ee680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:55.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe5fb7fe640 1 --2- 172.21.15.57:0/3102892456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5fc0cb180 0x7fe5fc1ee680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:55.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe5fb7fe640 1 -- 172.21.15.57:0/3102892456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5fc153b50 msgr2=0x7fe5fc14c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:55.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe5fb7fe640 1 --2- 172.21.15.57:0/3102892456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5fc153b50 0x7fe5fc14c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:55.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe5fb7fe640 1 -- 172.21.15.57:0/3102892456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f0009420 con 0x7fe5fc0c62b0 2023-10-30T14:26:55.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe5fbfff640 1 --2- 172.21.15.57:0/3102892456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5fc153b50 0x7fe5fc14c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:55.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe5fb7fe640 1 --2- 172.21.15.57:0/3102892456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5fc0c62b0 0x7fe5fc0c36e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec00c480 tx=0x7fe5ec00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:55.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe5e3fff640 1 -- 172.21.15.57:0/3102892456 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec0074e0 con 0x7fe5fc0c62b0 2023-10-30T14:26:55.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe5e3fff640 1 -- 172.21.15.57:0/3102892456 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ec007660 con 0x7fe5fc0c62b0 2023-10-30T14:26:55.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe5e3fff640 1 -- 172.21.15.57:0/3102892456 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec004a30 con 0x7fe5fc0c62b0 2023-10-30T14:26:55.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe60092b640 1 -- 172.21.15.57:0/3102892456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5fc150ce0 con 0x7fe5fc0c62b0 2023-10-30T14:26:55.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe60092b640 1 -- 172.21.15.57:0/3102892456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5fc151ec0 con 0x7fe5fc0c62b0 2023-10-30T14:26:55.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.755+0000 7fe60092b640 1 -- 172.21.15.57:0/3102892456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5fc067540 con 0x7fe5fc0c62b0 2023-10-30T14:26:55.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.759+0000 7fe5e3fff640 1 -- 172.21.15.57:0/3102892456 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe5ec004b90 con 0x7fe5fc0c62b0 2023-10-30T14:26:55.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.759+0000 7fe5e3fff640 1 --2- 172.21.15.57:0/3102892456 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5e80664a0 0x7fe5e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:55.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.759+0000 7fe5e3fff640 1 -- 172.21.15.57:0/3102892456 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7fe5ec083160 con 0x7fe5fc0c62b0 2023-10-30T14:26:55.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.759+0000 7fe5faffd640 1 --2- 172.21.15.57:0/3102892456 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5e80664a0 0x7fe5e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:55.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.759+0000 7fe5faffd640 1 --2- 172.21.15.57:0/3102892456 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5e80664a0 0x7fe5e8068960 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00b2000 tx=0x7fe5f0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:55.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.763+0000 7fe5e3fff640 1 -- 172.21.15.57:0/3102892456 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe5ec053e10 con 0x7fe5fc0c62b0 2023-10-30T14:26:55.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.955+0000 7fe60092b640 1 -- 172.21.15.57:0/3102892456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5fc15a770 con 0x7fe5fc0c62b0 2023-10-30T14:26:55.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.955+0000 7fe5e3fff640 1 -- 172.21.15.57:0/3102892456 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+5715 (secure 0 0 0) 0x7fe5fc15a770 con 0x7fe5fc0c62b0 2023-10-30T14:26:55.958 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:55.958 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:26:55.544573+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:55.958 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 78 2023-10-30T14:26:55.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.955+0000 7fe60092b640 1 -- 172.21.15.57:0/3102892456 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5e80664a0 msgr2=0x7fe5e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:55.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.955+0000 7fe60092b640 1 --2- 172.21.15.57:0/3102892456 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5e80664a0 0x7fe5e8068960 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00b2000 tx=0x7fe5f0008040 comp rx=0 tx=0).stop 2023-10-30T14:26:55.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.955+0000 7fe60092b640 1 -- 172.21.15.57:0/3102892456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5fc0c62b0 msgr2=0x7fe5fc0c36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:55.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.955+0000 7fe60092b640 1 --2- 172.21.15.57:0/3102892456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5fc0c62b0 0x7fe5fc0c36e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec00c480 tx=0x7fe5ec00c940 comp rx=0 tx=0).stop 2023-10-30T14:26:55.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.955+0000 7fe60092b640 1 -- 172.21.15.57:0/3102892456 shutdown_connections 2023-10-30T14:26:55.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.955+0000 7fe60092b640 1 --2- 172.21.15.57:0/3102892456 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5e80664a0 0x7fe5e8068960 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:55.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.955+0000 7fe60092b640 1 --2- 172.21.15.57:0/3102892456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5fc153b50 0x7fe5fc14c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:55.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.955+0000 7fe60092b640 1 --2- 172.21.15.57:0/3102892456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5fc0cb180 0x7fe5fc1ee680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:55.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.955+0000 7fe60092b640 1 --2- 172.21.15.57:0/3102892456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5fc0c62b0 0x7fe5fc0c36e0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:55.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.955+0000 7fe60092b640 1 -- 172.21.15.57:0/3102892456 >> 172.21.15.57:0/3102892456 conn(0x7fe5fc065a80 msgr2=0x7fe5fc0c6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:55.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.955+0000 7fe60092b640 1 -- 172.21.15.57:0/3102892456 shutdown_connections 2023-10-30T14:26:55.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:55.955+0000 7fe60092b640 1 -- 172.21.15.57:0/3102892456 wait complete. 2023-10-30T14:26:55.979 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5289 2023-10-30T14:26:55.979 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.c 2023-10-30T14:26:58.980 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T14:26:58.980 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T14:26:58.981 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T14:26:58.985 INFO:tasks.ceph.mds.c:Started 2023-10-30T14:26:58.985 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:26:59.083 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-30T14:26:59.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.135+0000 7f69a759e640 1 Processor -- start 2023-10-30T14:26:59.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.135+0000 7f69a759e640 1 -- start start 2023-10-30T14:26:59.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.135+0000 7f69a759e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f69a806dfc0 0x7f69a806d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:59.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.135+0000 7f69a759e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f69a80706a0 0x7f69a80c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:59.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.135+0000 7f69a759e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a806fc30 con 0x7f69a80c5460 2023-10-30T14:26:59.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.135+0000 7f69a759e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a815b170 con 0x7f69a806dfc0 2023-10-30T14:26:59.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.135+0000 7f69a759e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a815b360 con 0x7f69a80706a0 2023-10-30T14:26:59.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.139+0000 7f69a659c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f69a806dfc0 0x7f69a806d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:59.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.139+0000 7f69a659c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f69a806dfc0 0x7f69a806d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54362/0 (socket says 172.21.15.57:54362) 2023-10-30T14:26:59.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.139+0000 7f69a659c640 1 -- 172.21.15.57:0/1499978016 learned_addr learned my addr 172.21.15.57:0/1499978016 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:59.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.139+0000 7f69a6d9d640 1 --2- 172.21.15.57:0/1499978016 >> v2:172.21.15.88:3300/0 conn(0x7f69a80706a0 0x7f69a80c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:59.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.139+0000 7f69a659c640 1 -- 172.21.15.57:0/1499978016 >> v2:172.21.15.88:3300/0 conn(0x7f69a80706a0 msgr2=0x7f69a80c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:59.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.139+0000 7f69a659c640 1 --2- 172.21.15.57:0/1499978016 >> v2:172.21.15.88:3300/0 conn(0x7f69a80706a0 0x7f69a80c40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:59.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.139+0000 7f69a659c640 1 -- 172.21.15.57:0/1499978016 >> v1:172.21.15.57:6789/0 conn(0x7f69a80c5460 legacy=0x7f69a815aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:59.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.139+0000 7f69a5d9b640 1 -- 172.21.15.57:0/1499978016 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330733968 0 0) 0x7f69a806fc30 con 0x7f69a80c5460 2023-10-30T14:26:59.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.139+0000 7f69a659c640 1 -- 172.21.15.57:0/1499978016 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a815b620 con 0x7f69a806dfc0 2023-10-30T14:26:59.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.139+0000 7f69a659c640 1 --2- 172.21.15.57:0/1499978016 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a806dfc0 0x7f69a806d4f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f69a8072020 tx=0x7f699801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f83bc28555d45bda server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:59.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.139+0000 7f69a5d9b640 1 -- 172.21.15.57:0/1499978016 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69980ac640 con 0x7f69a806dfc0 2023-10-30T14:26:59.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.139+0000 7f69a5d9b640 1 -- 172.21.15.57:0/1499978016 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69980ac7a0 con 0x7f69a806dfc0 2023-10-30T14:26:59.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.139+0000 7f69a5d9b640 1 -- 172.21.15.57:0/1499978016 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69980acb30 con 0x7f69a806dfc0 2023-10-30T14:26:59.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a759e640 1 -- 172.21.15.57:0/1499978016 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a806dfc0 msgr2=0x7f69a806d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:59.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a759e640 1 --2- 172.21.15.57:0/1499978016 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a806dfc0 0x7f69a806d4f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f69a8072020 tx=0x7f699801c4d0 comp rx=0 tx=0).stop 2023-10-30T14:26:59.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a759e640 1 -- 172.21.15.57:0/1499978016 shutdown_connections 2023-10-30T14:26:59.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a759e640 1 --2- 172.21.15.57:0/1499978016 >> v2:172.21.15.88:3300/0 conn(0x7f69a80706a0 0x7f69a80c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:59.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a759e640 1 --2- 172.21.15.57:0/1499978016 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a806dfc0 0x7f69a806d4f0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:59.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a759e640 1 -- 172.21.15.57:0/1499978016 >> 172.21.15.57:0/1499978016 conn(0x7f69a80caae0 msgr2=0x7f69a80caec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:59.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a759e640 1 -- 172.21.15.57:0/1499978016 shutdown_connections 2023-10-30T14:26:59.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a759e640 1 -- 172.21.15.57:0/1499978016 wait complete. 2023-10-30T14:26:59.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a759e640 1 Processor -- start 2023-10-30T14:26:59.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a759e640 1 -- start start 2023-10-30T14:26:59.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a759e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69a806dfc0 0x7f69a814f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:59.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a759e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69a80706a0 0x7f69a81f4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:59.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a759e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a80c5460 0x7f69a814b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:59.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a759e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a80c7890 con 0x7f69a80c5460 2023-10-30T14:26:59.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a759e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a81540a0 con 0x7f69a806dfc0 2023-10-30T14:26:59.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a759e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a80ca0b0 con 0x7f69a80706a0 2023-10-30T14:26:59.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a80c5460 0x7f69a814b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:59.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a80c5460 0x7f69a814b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54370/0 (socket says 172.21.15.57:54370) 2023-10-30T14:26:59.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a6d9d640 1 -- 172.21.15.57:0/3240753584 learned_addr learned my addr 172.21.15.57:0/3240753584 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:26:59.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f699ffff640 1 --2- 172.21.15.57:0/3240753584 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69a80706a0 0x7f69a81f4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:59.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.143+0000 7f69a659c640 1 --2- 172.21.15.57:0/3240753584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69a806dfc0 0x7f69a814f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:59.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.147+0000 7f69a6d9d640 1 -- 172.21.15.57:0/3240753584 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69a80706a0 msgr2=0x7f69a81f4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:59.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.147+0000 7f69a6d9d640 1 --2- 172.21.15.57:0/3240753584 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69a80706a0 0x7f69a81f4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:59.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.147+0000 7f69a6d9d640 1 -- 172.21.15.57:0/3240753584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69a806dfc0 msgr2=0x7f69a814f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:59.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.147+0000 7f69a6d9d640 1 --2- 172.21.15.57:0/3240753584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69a806dfc0 0x7f69a814f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:59.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.147+0000 7f69a6d9d640 1 -- 172.21.15.57:0/3240753584 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69980094d0 con 0x7f69a80c5460 2023-10-30T14:26:59.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.147+0000 7f69a6d9d640 1 --2- 172.21.15.57:0/3240753584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a80c5460 0x7f69a814b360 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f69a0007e70 tx=0x7f69a000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:59.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.147+0000 7f699effd640 1 -- 172.21.15.57:0/3240753584 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a0014420 con 0x7f69a80c5460 2023-10-30T14:26:59.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.147+0000 7f699effd640 1 -- 172.21.15.57:0/3240753584 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a00145a0 con 0x7f69a80c5460 2023-10-30T14:26:59.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.147+0000 7f699effd640 1 -- 172.21.15.57:0/3240753584 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a0014950 con 0x7f69a80c5460 2023-10-30T14:26:59.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.147+0000 7f69a759e640 1 -- 172.21.15.57:0/3240753584 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a81592c0 con 0x7f69a80c5460 2023-10-30T14:26:59.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.147+0000 7f69a759e640 1 -- 172.21.15.57:0/3240753584 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a80c71d0 con 0x7f69a80c5460 2023-10-30T14:26:59.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.147+0000 7f69a759e640 1 -- 172.21.15.57:0/3240753584 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a80c79d0 con 0x7f69a80c5460 2023-10-30T14:26:59.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.147+0000 7f699effd640 1 -- 172.21.15.57:0/3240753584 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f69a0014ab0 con 0x7f69a80c5460 2023-10-30T14:26:59.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.147+0000 7f699effd640 1 --2- 172.21.15.57:0/3240753584 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f69880664a0 0x7f6988068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:59.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.147+0000 7f699effd640 1 -- 172.21.15.57:0/3240753584 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f69a0084a90 con 0x7f69a80c5460 2023-10-30T14:26:59.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.151+0000 7f69a659c640 1 --2- 172.21.15.57:0/3240753584 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f69880664a0 0x7f6988068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:59.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.151+0000 7f69a659c640 1 --2- 172.21.15.57:0/3240753584 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f69880664a0 0x7f6988068960 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f69980ae000 tx=0x7f6998008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:59.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.151+0000 7f699effd640 1 -- 172.21.15.57:0/3240753584 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f69a80c79d0 con 0x7f69a80c5460 2023-10-30T14:26:59.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.343+0000 7f69a759e640 1 -- 172.21.15.57:0/3240753584 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69a80c79d0 con 0x7f69a80c5460 2023-10-30T14:26:59.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.343+0000 7f699effd640 1 -- 172.21.15.57:0/3240753584 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+5715 (secure 0 0 0) 0x7f69a80c79d0 con 0x7f69a80c5460 2023-10-30T14:26:59.348 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:26:59.349 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:26:55.544573+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:26:59.349 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 78 2023-10-30T14:26:59.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.347+0000 7f69a759e640 1 -- 172.21.15.57:0/3240753584 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f69880664a0 msgr2=0x7f6988068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:59.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.347+0000 7f69a759e640 1 --2- 172.21.15.57:0/3240753584 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f69880664a0 0x7f6988068960 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f69980ae000 tx=0x7f6998008040 comp rx=0 tx=0).stop 2023-10-30T14:26:59.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.347+0000 7f69a759e640 1 -- 172.21.15.57:0/3240753584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a80c5460 msgr2=0x7f69a814b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:59.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.347+0000 7f69a759e640 1 --2- 172.21.15.57:0/3240753584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a80c5460 0x7f69a814b360 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f69a0007e70 tx=0x7f69a000e4b0 comp rx=0 tx=0).stop 2023-10-30T14:26:59.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.347+0000 7f69a759e640 1 -- 172.21.15.57:0/3240753584 shutdown_connections 2023-10-30T14:26:59.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.347+0000 7f69a759e640 1 --2- 172.21.15.57:0/3240753584 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f69880664a0 0x7f6988068960 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:59.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.347+0000 7f69a759e640 1 --2- 172.21.15.57:0/3240753584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a80c5460 0x7f69a814b360 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:59.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.347+0000 7f69a759e640 1 --2- 172.21.15.57:0/3240753584 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69a80706a0 0x7f69a81f4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:59.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.347+0000 7f69a759e640 1 --2- 172.21.15.57:0/3240753584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69a806dfc0 0x7f69a814f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:59.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.347+0000 7f69a759e640 1 -- 172.21.15.57:0/3240753584 >> 172.21.15.57:0/3240753584 conn(0x7f69a80caae0 msgr2=0x7f69a814e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:59.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.347+0000 7f69a759e640 1 -- 172.21.15.57:0/3240753584 shutdown_connections 2023-10-30T14:26:59.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:26:59.347+0000 7f69a759e640 1 -- 172.21.15.57:0/3240753584 wait complete. 2023-10-30T14:26:59.369 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T14:27:01.370 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:27:01.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.519+0000 7f8dd0165640 1 Processor -- start 2023-10-30T14:27:01.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.519+0000 7f8dd0165640 1 -- start start 2023-10-30T14:27:01.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.519+0000 7f8dd0165640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dc8153bb0 0x7f8dc80c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:01.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dd0165640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc80c5f80 con 0x7f8dc806c420 2023-10-30T14:27:01.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dd0165640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc80c5690 con 0x7f8dc8159980 2023-10-30T14:27:01.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dd0165640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc815a580 con 0x7f8dc8153bb0 2023-10-30T14:27:01.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcd700640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f8dc806c420 0x7f8dc80c6c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:57758/0 (socket says 172.21.15.57:57758) 2023-10-30T14:27:01.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcd700640 1 -- 172.21.15.57:0/2791036461 learned_addr learned my addr 172.21.15.57:0/2791036461 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:27:01.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcdf01640 1 --2- 172.21.15.57:0/2791036461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dc8153bb0 0x7f8dc80c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:01.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcceff640 1 -- 172.21.15.57:0/2791036461 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894339080 0 0) 0x7f8dc80c5f80 con 0x7f8dc806c420 2023-10-30T14:27:01.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcceff640 1 -- 172.21.15.57:0/2791036461 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8db4003530 con 0x7f8dc806c420 2023-10-30T14:27:01.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcceff640 1 -- 172.21.15.57:0/2791036461 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2002902026 0 0) 0x7f8db4003530 con 0x7f8dc806c420 2023-10-30T14:27:01.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcceff640 1 -- 172.21.15.57:0/2791036461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dc8153bb0 msgr2=0x7f8dc80c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:01.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcceff640 1 --2- 172.21.15.57:0/2791036461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dc8153bb0 0x7f8dc80c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:01.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcceff640 1 -- 172.21.15.57:0/2791036461 >> v1:172.21.15.88:6789/0 conn(0x7f8dc8159980 legacy=0x7f8dc8164220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:01.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcceff640 1 -- 172.21.15.57:0/2791036461 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc80c5930 con 0x7f8dc806c420 2023-10-30T14:27:01.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcceff640 1 -- 172.21.15.57:0/2791036461 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f8db8001680 con 0x7f8dc806c420 2023-10-30T14:27:01.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcceff640 1 -- 172.21.15.57:0/2791036461 >> v1:172.21.15.57:6789/0 conn(0x7f8dc806c420 legacy=0x7f8dc80c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:01.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcceff640 1 --2- 172.21.15.57:0/2791036461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dc8153bb0 0x7f8db40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:01.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcceff640 1 --2- 172.21.15.57:0/2791036461 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8db4003ef0 0x7f8db40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:01.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcceff640 1 --2- 172.21.15.57:0/2791036461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4004d10 0x7f8db4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:01.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcceff640 1 -- 172.21.15.57:0/2791036461 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc815a580 con 0x7f8db4003ef0 2023-10-30T14:27:01.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcceff640 1 -- 172.21.15.57:0/2791036461 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db400b9f0 con 0x7f8db4004d10 2023-10-30T14:27:01.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcceff640 1 -- 172.21.15.57:0/2791036461 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db400bbb0 con 0x7f8dc8153bb0 2023-10-30T14:27:01.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcd700640 1 --2- 172.21.15.57:0/2791036461 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8db4003ef0 0x7f8db40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:01.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcdf01640 1 --2- 172.21.15.57:0/2791036461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dc8153bb0 0x7f8db40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:01.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dce702640 1 --2- 172.21.15.57:0/2791036461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4004d10 0x7f8db4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:01.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcd700640 1 -- 172.21.15.57:0/2791036461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dc8153bb0 msgr2=0x7f8db40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:01.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcd700640 1 --2- 172.21.15.57:0/2791036461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dc8153bb0 0x7f8db40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:01.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcd700640 1 -- 172.21.15.57:0/2791036461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4004d10 msgr2=0x7f8db4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:01.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcd700640 1 --2- 172.21.15.57:0/2791036461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4004d10 0x7f8db4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:01.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dcd700640 1 -- 172.21.15.57:0/2791036461 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db4003530 con 0x7f8db4003ef0 2023-10-30T14:27:01.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.523+0000 7f8dce702640 1 --2- 172.21.15.57:0/2791036461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4004d10 0x7f8db4005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:27:01.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.527+0000 7f8dcd700640 1 --2- 172.21.15.57:0/2791036461 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8db4003ef0 0x7f8db40047d0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f8db800c210 tx=0x7f8db801f2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c33e0bd130d0fe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:01.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.527+0000 7f8dcceff640 1 -- 172.21.15.57:0/2791036461 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db80058e0 con 0x7f8db4003ef0 2023-10-30T14:27:01.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.527+0000 7f8dcceff640 1 -- 172.21.15.57:0/2791036461 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db8005a60 con 0x7f8db4003ef0 2023-10-30T14:27:01.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.527+0000 7f8dcceff640 1 -- 172.21.15.57:0/2791036461 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db8005e50 con 0x7f8db4003ef0 2023-10-30T14:27:01.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.527+0000 7f8dd0165640 1 -- 172.21.15.57:0/2791036461 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8db4003ef0 msgr2=0x7f8db40047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:01.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.527+0000 7f8dd0165640 1 --2- 172.21.15.57:0/2791036461 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8db4003ef0 0x7f8db40047d0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f8db800c210 tx=0x7f8db801f2d0 comp rx=0 tx=0).stop 2023-10-30T14:27:01.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.527+0000 7f8dce702640 1 -- 172.21.15.57:0/2791036461 reap_dead start 2023-10-30T14:27:01.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.527+0000 7f8dd0165640 1 -- 172.21.15.57:0/2791036461 shutdown_connections 2023-10-30T14:27:01.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.527+0000 7f8dd0165640 1 -- 172.21.15.57:0/2791036461 >> 172.21.15.57:0/2791036461 conn(0x7f8dc80cac70 msgr2=0x7f8dc80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:01.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.527+0000 7f8dd0165640 1 -- 172.21.15.57:0/2791036461 shutdown_connections 2023-10-30T14:27:01.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.527+0000 7f8dd0165640 1 -- 172.21.15.57:0/2791036461 wait complete. 2023-10-30T14:27:01.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.527+0000 7f8dd0165640 1 Processor -- start 2023-10-30T14:27:01.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.527+0000 7f8dd0165640 1 -- start start 2023-10-30T14:27:01.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dd0165640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4003ef0 0x7f8dc814d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:01.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dd0165640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc80c8b50 0x7f8dc80c9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:01.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dd0165640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dc80c9540 0x7f8dc8146450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:01.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dd0165640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc81ef1d0 con 0x7f8dc80c8b50 2023-10-30T14:27:01.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dd0165640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc81ee620 con 0x7f8db4003ef0 2023-10-30T14:27:01.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dd0165640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc81f1d10 con 0x7f8dc80c9540 2023-10-30T14:27:01.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dcd700640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc80c8b50 0x7f8dc80c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:01.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dcd700640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc80c8b50 0x7f8dc80c9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54390/0 (socket says 172.21.15.57:54390) 2023-10-30T14:27:01.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dcd700640 1 -- 172.21.15.57:0/2973977652 learned_addr learned my addr 172.21.15.57:0/2973977652 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:01.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dce702640 1 --2- 172.21.15.57:0/2973977652 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dc80c9540 0x7f8dc8146450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:01.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dcdf01640 1 --2- 172.21.15.57:0/2973977652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4003ef0 0x7f8dc814d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:01.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dcd700640 1 -- 172.21.15.57:0/2973977652 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dc80c9540 msgr2=0x7f8dc8146450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:01.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dcd700640 1 --2- 172.21.15.57:0/2973977652 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dc80c9540 0x7f8dc8146450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:01.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dcd700640 1 -- 172.21.15.57:0/2973977652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4003ef0 msgr2=0x7f8dc814d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:01.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dcd700640 1 --2- 172.21.15.57:0/2973977652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4003ef0 0x7f8dc814d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:01.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dcd700640 1 -- 172.21.15.57:0/2973977652 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc81ee8a0 con 0x7f8dc80c8b50 2023-10-30T14:27:01.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dcdf01640 1 --2- 172.21.15.57:0/2973977652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4003ef0 0x7f8dc814d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:27:01.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dcd700640 1 --2- 172.21.15.57:0/2973977652 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc80c8b50 0x7f8dc80c9000 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f8db800c210 tx=0x7f8db4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:01.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dbe7fc640 1 -- 172.21.15.57:0/2973977652 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db80b7070 con 0x7f8dc80c8b50 2023-10-30T14:27:01.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dbe7fc640 1 -- 172.21.15.57:0/2973977652 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db8001500 con 0x7f8dc80c8b50 2023-10-30T14:27:01.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dd0165640 1 -- 172.21.15.57:0/2973977652 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc81eeb80 con 0x7f8dc80c8b50 2023-10-30T14:27:01.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dbe7fc640 1 -- 172.21.15.57:0/2973977652 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db80049f0 con 0x7f8dc80c8b50 2023-10-30T14:27:01.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dd0165640 1 -- 172.21.15.57:0/2973977652 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc80ca900 con 0x7f8dc80c8b50 2023-10-30T14:27:01.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.531+0000 7f8dd0165640 1 -- 172.21.15.57:0/2973977652 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc8066510 con 0x7f8dc80c8b50 2023-10-30T14:27:01.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.535+0000 7f8dbe7fc640 1 -- 172.21.15.57:0/2973977652 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8db8004b50 con 0x7f8dc80c8b50 2023-10-30T14:27:01.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.535+0000 7f8dbe7fc640 1 --2- 172.21.15.57:0/2973977652 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dac0664a0 0x7f8dac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:01.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.535+0000 7f8dbe7fc640 1 -- 172.21.15.57:0/2973977652 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f8db81211a0 con 0x7f8dc80c8b50 2023-10-30T14:27:01.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.535+0000 7f8dcdf01640 1 --2- 172.21.15.57:0/2973977652 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dac0664a0 0x7f8dac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:01.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.535+0000 7f8dcdf01640 1 --2- 172.21.15.57:0/2973977652 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dac0664a0 0x7f8dac068960 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4009be0 tx=0x7f8dc4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:01.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.539+0000 7f8dbe7fc640 1 -- 172.21.15.57:0/2973977652 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8db80ee340 con 0x7f8dc80c8b50 2023-10-30T14:27:01.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.759+0000 7f8dd0165640 1 -- 172.21.15.57:0/2973977652 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8dc804e170 con 0x7f8dc80c8b50 2023-10-30T14:27:01.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.759+0000 7f8dbe7fc640 1 -- 172.21.15.57:0/2973977652 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6506 (secure 0 0 0) 0x7f8db80f1960 con 0x7f8dc80c8b50 2023-10-30T14:27:01.763 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:27:01.763 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":6027,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:26:55.544573+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:27:01.763 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 79 2023-10-30T14:27:01.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.759+0000 7f8dd0165640 1 -- 172.21.15.57:0/2973977652 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dac0664a0 msgr2=0x7f8dac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:01.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.759+0000 7f8dd0165640 1 --2- 172.21.15.57:0/2973977652 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dac0664a0 0x7f8dac068960 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4009be0 tx=0x7f8dc4008040 comp rx=0 tx=0).stop 2023-10-30T14:27:01.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.759+0000 7f8dd0165640 1 -- 172.21.15.57:0/2973977652 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc80c8b50 msgr2=0x7f8dc80c9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:01.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.759+0000 7f8dd0165640 1 --2- 172.21.15.57:0/2973977652 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc80c8b50 0x7f8dc80c9000 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f8db800c210 tx=0x7f8db4005800 comp rx=0 tx=0).stop 2023-10-30T14:27:01.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.759+0000 7f8dd0165640 1 -- 172.21.15.57:0/2973977652 shutdown_connections 2023-10-30T14:27:01.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.759+0000 7f8dd0165640 1 --2- 172.21.15.57:0/2973977652 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dac0664a0 0x7f8dac068960 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:01.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.759+0000 7f8dd0165640 1 --2- 172.21.15.57:0/2973977652 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dc80c9540 0x7f8dc8146450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:01.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.759+0000 7f8dd0165640 1 --2- 172.21.15.57:0/2973977652 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc80c8b50 0x7f8dc80c9000 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:01.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.759+0000 7f8dd0165640 1 --2- 172.21.15.57:0/2973977652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4003ef0 0x7f8dc814d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:01.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.759+0000 7f8dd0165640 1 -- 172.21.15.57:0/2973977652 >> 172.21.15.57:0/2973977652 conn(0x7f8dc80cac70 msgr2=0x7f8dc80c7640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:01.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.763+0000 7f8dd0165640 1 -- 172.21.15.57:0/2973977652 shutdown_connections 2023-10-30T14:27:01.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:01.763+0000 7f8dd0165640 1 -- 172.21.15.57:0/2973977652 wait complete. 2023-10-30T14:27:01.784 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T14:27:01.784 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 31 secs before thrashing 2023-10-30T14:27:02.038 INFO:tasks.ceph.mds.a.smithi057.stderr:2023-10-30T14:27:02.035+0000 7f80eda8e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T14:27:02.039 INFO:tasks.ceph.mds.a.smithi057.stderr:2023-10-30T14:27:02.035+0000 7f80eda8e640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:27:03.756 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:27:03.760 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:27:17.848 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 1% (141/12071) Updating files: 1% (237/12071) Updating files: 2% (242/12071) Updating files: 3% (363/12071) Updating files: 3% (474/12071) Updating files: 4% (483/12071) Updating files: 4% (497/12071) Updating files: 4% (584/12071) Updating files: 5% (604/12071) Updating files: 5% (615/12071) Updating files: 6% (725/12071) Updating files: 7% (845/12071) Updating files: 7% (878/12071) Updating files: 7% (898/12071) Updating files: 8% (966/12071) Updating files: 8% (1079/12071) Updating files: 9% (1087/12071) Updating files: 9% (1147/12071) Updating files: 10% (1208/12071) Updating files: 11% (1328/12071) Updating files: 11% (1379/12071) Updating files: 12% (1449/12071) Updating files: 13% (1570/12071) Updating files: 13% (1618/12071) Updating files: 14% (1690/12071) Updating files: 14% (1716/12071) Updating files: 15% (1811/12071) Updating files: 15% (1863/12071) Updating files: 16% (1932/12071) Updating files: 16% (2016/12071) Updating files: 16% (2039/12071) Updating files: 17% (2053/12071) Updating files: 17% (2060/12071) Updating files: 18% (2173/12071) Updating files: 18% (2192/12071) Updating files: 19% (2294/12071) Updating files: 19% (2310/12071) Updating files: 20% (2415/12071) Updating files: 20% (2478/12071) Updating files: 20% (2519/12071) Updating files: 21% (2535/12071) Updating files: 22% (2656/12071) Updating files: 22% (2667/12071) Updating files: 22% (2719/12071) Updating files: 22% (2770/12071) Updating files: 23% (2777/12071) Updating files: 24% (2898/12071) Updating files: 24% (2972/12071) Updating files: 25% (3018/12071) Updating files: 26% (3139/12071) Updating files: 26% (3184/12071) Updating files: 27% (3260/12071) Updating files: 27% (3292/12071) Updating files: 27% (3345/12071) Updating files: 28% (3380/12071) Updating files: 28% (3447/12071) Updating files: 29% (3501/12071) Updating files: 29% (3562/12071) Updating files: 30% (3622/12071) Updating files: 30% (3685/12071) Updating files: 31% (3743/12071) Updating files: 32% (3863/12071) Updating files: 33% (3984/12071) Updating files: 33% (4005/12071) Updating files: 33% (4020/12071) Updating files: 33% (4097/12071) Updating files: 34% (4105/12071) Updating files: 34% (4188/12071) Updating files: 35% (4225/12071) Updating files: 35% (4266/12071) Updating files: 36% (4346/12071) Updating files: 36% (4388/12071) Updating files: 36% (4400/12071) Updating files: 37% (4467/12071) Updating files: 38% (4587/12071) Updating files: 38% (4617/12071) Updating files: 38% (4678/12071) Updating files: 38% (4692/12071) Updating files: 39% (4708/12071) Updating files: 40% (4829/12071) Updating files: 40% (4869/12071) Updating files: 41% (4950/12071) Updating files: 41% (4986/12071) Updating files: 42% (5070/12071) Updating files: 42% (5083/12071) Updating files: 42% (5136/12071) Updating files: 43% (5191/12071) Updating files: 44% (5312/12071) Updating files: 45% (5432/12071) Updating files: 46% (5553/12071) Updating files: 46% (5594/12071) Updating files: 47% (5674/12071) Updating files: 47% (5678/12071) Updating files: 47% (5743/12071) Updating files: 48% (5795/12071) Updating files: 48% (5832/12071) Updating files: 48% (5847/12071) Updating files: 49% (5915/12071) Updating files: 49% (5955/12071) Updating files: 50% (6036/12071) Updating files: 51% (6157/12071) Updating files: 51% (6189/12071) Updating files: 52% (6277/12071) Updating files: 52% (6299/12071) Updating files: 53% (6398/12071) Updating files: 53% (6427/12071) Updating files: 54% (6519/12071) Updating files: 54% (6558/12071) Updating files: 54% (6607/12071) Updating files: 55% (6640/12071) Updating files: 55% (6679/12071) Updating files: 55% (6721/12071) Updating files: 56% (6760/12071) Updating files: 56% (6845/12071) Updating files: 57% (6881/12071) Updating files: 57% (6982/12071) Updating files: 58% (7002/12071) Updating files: 58% (7035/12071) Updating files: 59% (7122/12071) Updating files: 59% (7233/12071) Updating files: 60% (7243/12071) Updating files: 61% (7364/12071) Updating files: 62% (7485/12071) Updating files: 62% (7532/12071) Updating files: 63% (7605/12071) Updating files: 63% (7624/12071) Updating files: 64% (7726/12071) Updating files: 64% (7787/12071) Updating files: 64% (7829/12071) Updating files: 65% (7847/12071) Updating files: 65% (7928/12071) Updating files: 66% (7967/12071) Updating files: 66% (8004/12071) Updating files: 67% (8088/12071) Updating files: 67% (8101/12071) Updating files: 67% (8143/12071) Updating files: 68% (8209/12071) Updating files: 68% (8212/12071) Updating files: 68% (8309/12071) Updating files: 69% (8329/12071) Updating files: 69% (8340/12071) Updating files: 70% (8450/12071) Updating files: 70% (8485/12071) Updating files: 71% (8571/12071) Updating files: 71% (8589/12071) Updating files: 71% (8601/12071) Updating files: 72% (8692/12071) Updating files: 73% (8812/12071) Updating files: 73% (8931/12071) Updating files: 74% (8933/12071) Updating files: 74% (8982/12071) Updating files: 74% (9039/12071) Updating files: 74% (9043/12071) Updating files: 75% (9054/12071) Updating files: 75% (9077/12071) Updating files: 75% (9163/12071) Updating files: 76% (9174/12071) Updating files: 77% (9295/12071) Updating files: 77% (9393/12071) Updating files: 77% (9394/12071) Updating files: 78% (9416/12071) Updating files: 78% (9470/12071) Updating files: 79% (9537/12071) Updating files: 79% (9611/12071) Updating files: 80% (9657/12071) Updating files: 80% (9773/12071) Updating files: 81% (9778/12071) Updating files: 81% (9867/12071) Updating files: 82% (9899/12071) Updating files: 83% (10019/12071) Updating files: 83% (10047/12071) Updating files: 83% (10132/12071) Updating files: 84% (10140/12071) Updating files: 85% (10261/12071) Updating files: 86% (10382/12071) Updating files: 86% (10436/12071) Updating files: 87% (10502/12071) Updating files: 87% (10620/12071) Updating files: 88% (10623/12071) Updating files: 89% (10744/12071) Updating files: 90% (10864/12071) Updating files: 90% (10923/12071) Updating files: 91% (10985/12071) Updating files: 91% (11027/12071) Updating files: 92% (11106/12071) Updating files: 92% (11224/12071) Updating files: 93% (11227/12071) Updating files: 94% (11347/12071) Updating files: 94% (11376/12071) Updating files: 95% (11468/12071) Updating files: 95% (11493/12071) Updating files: 96% (11589/12071) Updating files: 96% (11603/12071) Updating files: 97% (11709/12071) Updating files: 98% (11830/12071) Updating files: 98% (11894/12071) Updating files: 99% (11951/12071) Updating files: 100% (12071/12071) Updating files: 100% (12071/12071), done. 2023-10-30T14:27:22.878 INFO:tasks.workunit.client.0.smithi057.stderr:+ trap - EXIT 2023-10-30T14:27:22.879 INFO:tasks.workunit.client.0.smithi057.stderr:+ cd ceph 2023-10-30T14:27:22.879 INFO:tasks.workunit.client.0.smithi057.stderr:+ seq 1 90 2023-10-30T14:27:22.880 INFO:tasks.workunit.client.0.smithi057.stderr:+ versions=1 2023-10-30T14:27:22.880 INFO:tasks.workunit.client.0.smithi057.stderr:2 2023-10-30T14:27:22.880 INFO:tasks.workunit.client.0.smithi057.stderr:3 2023-10-30T14:27:22.881 INFO:tasks.workunit.client.0.smithi057.stderr:4 2023-10-30T14:27:22.881 INFO:tasks.workunit.client.0.smithi057.stderr:5 2023-10-30T14:27:22.881 INFO:tasks.workunit.client.0.smithi057.stderr:6 2023-10-30T14:27:22.881 INFO:tasks.workunit.client.0.smithi057.stderr:7 2023-10-30T14:27:22.881 INFO:tasks.workunit.client.0.smithi057.stderr:8 2023-10-30T14:27:22.881 INFO:tasks.workunit.client.0.smithi057.stderr:9 2023-10-30T14:27:22.882 INFO:tasks.workunit.client.0.smithi057.stderr:10 2023-10-30T14:27:22.882 INFO:tasks.workunit.client.0.smithi057.stderr:11 2023-10-30T14:27:22.882 INFO:tasks.workunit.client.0.smithi057.stderr:12 2023-10-30T14:27:22.882 INFO:tasks.workunit.client.0.smithi057.stderr:13 2023-10-30T14:27:22.882 INFO:tasks.workunit.client.0.smithi057.stderr:14 2023-10-30T14:27:22.882 INFO:tasks.workunit.client.0.smithi057.stderr:15 2023-10-30T14:27:22.883 INFO:tasks.workunit.client.0.smithi057.stderr:16 2023-10-30T14:27:22.883 INFO:tasks.workunit.client.0.smithi057.stderr:17 2023-10-30T14:27:22.883 INFO:tasks.workunit.client.0.smithi057.stderr:18 2023-10-30T14:27:22.883 INFO:tasks.workunit.client.0.smithi057.stderr:19 2023-10-30T14:27:22.883 INFO:tasks.workunit.client.0.smithi057.stderr:20 2023-10-30T14:27:22.884 INFO:tasks.workunit.client.0.smithi057.stderr:21 2023-10-30T14:27:22.884 INFO:tasks.workunit.client.0.smithi057.stderr:22 2023-10-30T14:27:22.884 INFO:tasks.workunit.client.0.smithi057.stderr:23 2023-10-30T14:27:22.884 INFO:tasks.workunit.client.0.smithi057.stderr:24 2023-10-30T14:27:22.884 INFO:tasks.workunit.client.0.smithi057.stderr:25 2023-10-30T14:27:22.885 INFO:tasks.workunit.client.0.smithi057.stderr:26 2023-10-30T14:27:22.885 INFO:tasks.workunit.client.0.smithi057.stderr:27 2023-10-30T14:27:22.885 INFO:tasks.workunit.client.0.smithi057.stderr:28 2023-10-30T14:27:22.885 INFO:tasks.workunit.client.0.smithi057.stderr:29 2023-10-30T14:27:22.885 INFO:tasks.workunit.client.0.smithi057.stderr:30 2023-10-30T14:27:22.886 INFO:tasks.workunit.client.0.smithi057.stderr:31 2023-10-30T14:27:22.886 INFO:tasks.workunit.client.0.smithi057.stderr:32 2023-10-30T14:27:22.886 INFO:tasks.workunit.client.0.smithi057.stderr:33 2023-10-30T14:27:22.886 INFO:tasks.workunit.client.0.smithi057.stderr:34 2023-10-30T14:27:22.886 INFO:tasks.workunit.client.0.smithi057.stderr:35 2023-10-30T14:27:22.886 INFO:tasks.workunit.client.0.smithi057.stderr:36 2023-10-30T14:27:22.887 INFO:tasks.workunit.client.0.smithi057.stderr:37 2023-10-30T14:27:22.887 INFO:tasks.workunit.client.0.smithi057.stderr:38 2023-10-30T14:27:22.887 INFO:tasks.workunit.client.0.smithi057.stderr:39 2023-10-30T14:27:22.887 INFO:tasks.workunit.client.0.smithi057.stderr:40 2023-10-30T14:27:22.887 INFO:tasks.workunit.client.0.smithi057.stderr:41 2023-10-30T14:27:22.888 INFO:tasks.workunit.client.0.smithi057.stderr:42 2023-10-30T14:27:22.888 INFO:tasks.workunit.client.0.smithi057.stderr:43 2023-10-30T14:27:22.888 INFO:tasks.workunit.client.0.smithi057.stderr:44 2023-10-30T14:27:22.888 INFO:tasks.workunit.client.0.smithi057.stderr:45 2023-10-30T14:27:22.888 INFO:tasks.workunit.client.0.smithi057.stderr:46 2023-10-30T14:27:22.888 INFO:tasks.workunit.client.0.smithi057.stderr:47 2023-10-30T14:27:22.889 INFO:tasks.workunit.client.0.smithi057.stderr:48 2023-10-30T14:27:22.889 INFO:tasks.workunit.client.0.smithi057.stderr:49 2023-10-30T14:27:22.889 INFO:tasks.workunit.client.0.smithi057.stderr:50 2023-10-30T14:27:22.889 INFO:tasks.workunit.client.0.smithi057.stderr:51 2023-10-30T14:27:22.889 INFO:tasks.workunit.client.0.smithi057.stderr:52 2023-10-30T14:27:22.890 INFO:tasks.workunit.client.0.smithi057.stderr:53 2023-10-30T14:27:22.890 INFO:tasks.workunit.client.0.smithi057.stderr:54 2023-10-30T14:27:22.890 INFO:tasks.workunit.client.0.smithi057.stderr:55 2023-10-30T14:27:22.890 INFO:tasks.workunit.client.0.smithi057.stderr:56 2023-10-30T14:27:22.890 INFO:tasks.workunit.client.0.smithi057.stderr:57 2023-10-30T14:27:22.890 INFO:tasks.workunit.client.0.smithi057.stderr:58 2023-10-30T14:27:22.891 INFO:tasks.workunit.client.0.smithi057.stderr:59 2023-10-30T14:27:22.891 INFO:tasks.workunit.client.0.smithi057.stderr:60 2023-10-30T14:27:22.891 INFO:tasks.workunit.client.0.smithi057.stderr:61 2023-10-30T14:27:22.891 INFO:tasks.workunit.client.0.smithi057.stderr:62 2023-10-30T14:27:22.891 INFO:tasks.workunit.client.0.smithi057.stderr:63 2023-10-30T14:27:22.891 INFO:tasks.workunit.client.0.smithi057.stderr:64 2023-10-30T14:27:22.892 INFO:tasks.workunit.client.0.smithi057.stderr:65 2023-10-30T14:27:22.892 INFO:tasks.workunit.client.0.smithi057.stderr:66 2023-10-30T14:27:22.892 INFO:tasks.workunit.client.0.smithi057.stderr:67 2023-10-30T14:27:22.892 INFO:tasks.workunit.client.0.smithi057.stderr:68 2023-10-30T14:27:22.892 INFO:tasks.workunit.client.0.smithi057.stderr:69 2023-10-30T14:27:22.892 INFO:tasks.workunit.client.0.smithi057.stderr:70 2023-10-30T14:27:22.893 INFO:tasks.workunit.client.0.smithi057.stderr:71 2023-10-30T14:27:22.893 INFO:tasks.workunit.client.0.smithi057.stderr:72 2023-10-30T14:27:22.893 INFO:tasks.workunit.client.0.smithi057.stderr:73 2023-10-30T14:27:22.893 INFO:tasks.workunit.client.0.smithi057.stderr:74 2023-10-30T14:27:22.893 INFO:tasks.workunit.client.0.smithi057.stderr:75 2023-10-30T14:27:22.893 INFO:tasks.workunit.client.0.smithi057.stderr:76 2023-10-30T14:27:22.894 INFO:tasks.workunit.client.0.smithi057.stderr:77 2023-10-30T14:27:22.894 INFO:tasks.workunit.client.0.smithi057.stderr:78 2023-10-30T14:27:22.894 INFO:tasks.workunit.client.0.smithi057.stderr:79 2023-10-30T14:27:22.894 INFO:tasks.workunit.client.0.smithi057.stderr:80 2023-10-30T14:27:22.894 INFO:tasks.workunit.client.0.smithi057.stderr:81 2023-10-30T14:27:22.895 INFO:tasks.workunit.client.0.smithi057.stderr:82 2023-10-30T14:27:22.895 INFO:tasks.workunit.client.0.smithi057.stderr:83 2023-10-30T14:27:22.895 INFO:tasks.workunit.client.0.smithi057.stderr:84 2023-10-30T14:27:22.895 INFO:tasks.workunit.client.0.smithi057.stderr:85 2023-10-30T14:27:22.895 INFO:tasks.workunit.client.0.smithi057.stderr:86 2023-10-30T14:27:22.895 INFO:tasks.workunit.client.0.smithi057.stderr:87 2023-10-30T14:27:22.896 INFO:tasks.workunit.client.0.smithi057.stderr:88 2023-10-30T14:27:22.896 INFO:tasks.workunit.client.0.smithi057.stderr:89 2023-10-30T14:27:22.896 INFO:tasks.workunit.client.0.smithi057.stderr:90 2023-10-30T14:27:22.896 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 1 -eq 48 ] 2023-10-30T14:27:22.897 INFO:tasks.workunit.client.0.smithi057.stdout:v0.1 2023-10-30T14:27:22.897 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.1 2023-10-30T14:27:22.897 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.1 2023-10-30T14:27:22.897 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.1 2023-10-30T14:27:32.786 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:27:32.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.931+0000 7f3d2efc5640 1 Processor -- start 2023-10-30T14:27:32.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.931+0000 7f3d2efc5640 1 -- start start 2023-10-30T14:27:32.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.931+0000 7f3d2efc5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3d28153c80 0x7f3d2815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:32.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.931+0000 7f3d2efc5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3d280c4a10 0x7f3d280c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:32.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.931+0000 7f3d2efc5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3d2815a3b0 0x7f3d280ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:32.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.931+0000 7f3d2efc5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d2815dec0 con 0x7f3d28153c80 2023-10-30T14:27:32.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.931+0000 7f3d2efc5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d281599a0 con 0x7f3d280c4a10 2023-10-30T14:27:32.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.931+0000 7f3d2efc5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d28159f20 con 0x7f3d2815a3b0 2023-10-30T14:27:32.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.931+0000 7f3d2cd61640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3d28153c80 0x7f3d2815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:32.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.931+0000 7f3d2cd61640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3d28153c80 0x7f3d2815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34754/0 (socket says 172.21.15.57:34754) 2023-10-30T14:27:32.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.931+0000 7f3d2cd61640 1 -- 172.21.15.57:0/1598069886 learned_addr learned my addr 172.21.15.57:0/1598069886 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:32.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.931+0000 7f3d2d562640 1 --2- 172.21.15.57:0/1598069886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3d2815a3b0 0x7f3d280ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:32.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.931+0000 7f3d27fff640 1 --2- 172.21.15.57:0/1598069886 >> v2:172.21.15.88:3300/0 conn(0x7f3d280c4a10 0x7f3d280c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:32.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.931+0000 7f3d2cd61640 1 -- 172.21.15.57:0/1598069886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3d2815a3b0 msgr2=0x7f3d280ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:32.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.931+0000 7f3d2cd61640 1 --2- 172.21.15.57:0/1598069886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3d2815a3b0 0x7f3d280ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:32.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.931+0000 7f3d2cd61640 1 -- 172.21.15.57:0/1598069886 >> v2:172.21.15.88:3300/0 conn(0x7f3d280c4a10 msgr2=0x7f3d280c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:32.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.931+0000 7f3d2cd61640 1 --2- 172.21.15.57:0/1598069886 >> v2:172.21.15.88:3300/0 conn(0x7f3d280c4a10 0x7f3d280c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:32.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.931+0000 7f3d2cd61640 1 -- 172.21.15.57:0/1598069886 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d280c47e0 con 0x7f3d28153c80 2023-10-30T14:27:32.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.935+0000 7f3d2cd61640 1 --2- 172.21.15.57:0/1598069886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d28153c80 0x7f3d2815cf90 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3d2806be50 tx=0x7f3d1c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1df47820c24e4cf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:32.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.935+0000 7f3d277fe640 1 -- 172.21.15.57:0/1598069886 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0a89b0 con 0x7f3d28153c80 2023-10-30T14:27:32.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.935+0000 7f3d277fe640 1 -- 172.21.15.57:0/1598069886 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c0a8b10 con 0x7f3d28153c80 2023-10-30T14:27:32.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.935+0000 7f3d277fe640 1 -- 172.21.15.57:0/1598069886 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0a8ea0 con 0x7f3d28153c80 2023-10-30T14:27:32.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.935+0000 7f3d2efc5640 1 -- 172.21.15.57:0/1598069886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d28153c80 msgr2=0x7f3d2815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:32.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.935+0000 7f3d2efc5640 1 --2- 172.21.15.57:0/1598069886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d28153c80 0x7f3d2815cf90 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3d2806be50 tx=0x7f3d1c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:27:32.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.935+0000 7f3d2efc5640 1 -- 172.21.15.57:0/1598069886 shutdown_connections 2023-10-30T14:27:32.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.935+0000 7f3d2efc5640 1 --2- 172.21.15.57:0/1598069886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3d2815a3b0 0x7f3d280ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:32.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.935+0000 7f3d2efc5640 1 --2- 172.21.15.57:0/1598069886 >> v2:172.21.15.88:3300/0 conn(0x7f3d280c4a10 0x7f3d280c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:32.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.935+0000 7f3d2efc5640 1 --2- 172.21.15.57:0/1598069886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d28153c80 0x7f3d2815cf90 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:32.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.935+0000 7f3d2efc5640 1 -- 172.21.15.57:0/1598069886 >> 172.21.15.57:0/1598069886 conn(0x7f3d280715c0 msgr2=0x7f3d280719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:32.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.935+0000 7f3d2efc5640 1 -- 172.21.15.57:0/1598069886 shutdown_connections 2023-10-30T14:27:32.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.935+0000 7f3d2efc5640 1 -- 172.21.15.57:0/1598069886 wait complete. 2023-10-30T14:27:32.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.935+0000 7f3d2efc5640 1 Processor -- start 2023-10-30T14:27:32.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2efc5640 1 -- start start 2023-10-30T14:27:32.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2efc5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d280c4a10 0x7f3d28156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:32.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2efc5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3d28153c80 0x7f3d280ce280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:32.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2efc5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d2815a3b0 0x7f3d280ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:32.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2efc5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d280cfe30 con 0x7f3d2815a3b0 2023-10-30T14:27:32.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2efc5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d28150d40 con 0x7f3d280c4a10 2023-10-30T14:27:32.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2efc5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d280cefd0 con 0x7f3d28153c80 2023-10-30T14:27:32.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2d562640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d2815a3b0 0x7f3d280ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:32.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2d562640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d2815a3b0 0x7f3d280ce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34760/0 (socket says 172.21.15.57:34760) 2023-10-30T14:27:32.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2d562640 1 -- 172.21.15.57:0/1267014018 learned_addr learned my addr 172.21.15.57:0/1267014018 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:32.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d27fff640 1 --2- 172.21.15.57:0/1267014018 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3d28153c80 0x7f3d280ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:32.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2cd61640 1 --2- 172.21.15.57:0/1267014018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d280c4a10 0x7f3d28156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:32.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2d562640 1 -- 172.21.15.57:0/1267014018 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3d28153c80 msgr2=0x7f3d280ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:32.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2d562640 1 --2- 172.21.15.57:0/1267014018 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3d28153c80 0x7f3d280ce280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:32.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2d562640 1 -- 172.21.15.57:0/1267014018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d280c4a10 msgr2=0x7f3d28156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:32.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2d562640 1 --2- 172.21.15.57:0/1267014018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d280c4a10 0x7f3d28156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:32.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2d562640 1 -- 172.21.15.57:0/1267014018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c0094d0 con 0x7f3d2815a3b0 2023-10-30T14:27:32.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2d562640 1 --2- 172.21.15.57:0/1267014018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d2815a3b0 0x7f3d280ce7c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3d20007e60 tx=0x7f3d2000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:32.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d257fa640 1 -- 172.21.15.57:0/1267014018 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20014420 con 0x7f3d2815a3b0 2023-10-30T14:27:32.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d257fa640 1 -- 172.21.15.57:0/1267014018 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d200145a0 con 0x7f3d2815a3b0 2023-10-30T14:27:32.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d257fa640 1 -- 172.21.15.57:0/1267014018 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20014950 con 0x7f3d2815a3b0 2023-10-30T14:27:32.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2efc5640 1 -- 172.21.15.57:0/1267014018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d280cf2b0 con 0x7f3d2815a3b0 2023-10-30T14:27:32.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2efc5640 1 -- 172.21.15.57:0/1267014018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d280cf870 con 0x7f3d2815a3b0 2023-10-30T14:27:32.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.939+0000 7f3d2efc5640 1 -- 172.21.15.57:0/1267014018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d2804e170 con 0x7f3d2815a3b0 2023-10-30T14:27:32.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.943+0000 7f3d257fa640 1 -- 172.21.15.57:0/1267014018 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3d20014ab0 con 0x7f3d2815a3b0 2023-10-30T14:27:32.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.943+0000 7f3d257fa640 1 --2- 172.21.15.57:0/1267014018 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3d0c0664a0 0x7f3d0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:32.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.943+0000 7f3d257fa640 1 -- 172.21.15.57:0/1267014018 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f3d20084aa0 con 0x7f3d2815a3b0 2023-10-30T14:27:32.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.943+0000 7f3d2cd61640 1 --2- 172.21.15.57:0/1267014018 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3d0c0664a0 0x7f3d0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:32.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.943+0000 7f3d2cd61640 1 --2- 172.21.15.57:0/1267014018 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3d0c0664a0 0x7f3d0c068960 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0b2000 tx=0x7f3d1c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:32.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:32.947+0000 7f3d257fa640 1 -- 172.21.15.57:0/1267014018 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3d20051bd0 con 0x7f3d2815a3b0 2023-10-30T14:27:33.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.139+0000 7f3d2efc5640 1 -- 172.21.15.57:0/1267014018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d280cad40 con 0x7f3d2815a3b0 2023-10-30T14:27:33.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.139+0000 7f3d257fa640 1 -- 172.21.15.57:0/1267014018 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6507 (secure 0 0 0) 0x7f3d200551f0 con 0x7f3d2815a3b0 2023-10-30T14:27:33.139 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:27:33.139 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":6027,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:27:05.900123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:27:33.140 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 80 2023-10-30T14:27:33.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.139+0000 7f3d2efc5640 1 -- 172.21.15.57:0/1267014018 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3d0c0664a0 msgr2=0x7f3d0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:33.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.139+0000 7f3d2efc5640 1 --2- 172.21.15.57:0/1267014018 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3d0c0664a0 0x7f3d0c068960 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0b2000 tx=0x7f3d1c009040 comp rx=0 tx=0).stop 2023-10-30T14:27:33.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.139+0000 7f3d2efc5640 1 -- 172.21.15.57:0/1267014018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d2815a3b0 msgr2=0x7f3d280ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:33.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.139+0000 7f3d2efc5640 1 --2- 172.21.15.57:0/1267014018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d2815a3b0 0x7f3d280ce7c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3d20007e60 tx=0x7f3d2000e4b0 comp rx=0 tx=0).stop 2023-10-30T14:27:33.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.139+0000 7f3d2efc5640 1 -- 172.21.15.57:0/1267014018 shutdown_connections 2023-10-30T14:27:33.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.139+0000 7f3d2efc5640 1 --2- 172.21.15.57:0/1267014018 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3d0c0664a0 0x7f3d0c068960 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:33.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.139+0000 7f3d2efc5640 1 --2- 172.21.15.57:0/1267014018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d2815a3b0 0x7f3d280ce7c0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:33.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.139+0000 7f3d2efc5640 1 --2- 172.21.15.57:0/1267014018 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3d28153c80 0x7f3d280ce280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:33.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.139+0000 7f3d2efc5640 1 --2- 172.21.15.57:0/1267014018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d280c4a10 0x7f3d28156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:33.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.139+0000 7f3d2efc5640 1 -- 172.21.15.57:0/1267014018 >> 172.21.15.57:0/1267014018 conn(0x7f3d280715c0 msgr2=0x7f3d281513e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:33.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.139+0000 7f3d2efc5640 1 -- 172.21.15.57:0/1267014018 shutdown_connections 2023-10-30T14:27:33.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.139+0000 7f3d2efc5640 1 -- 172.21.15.57:0/1267014018 wait complete. 2023-10-30T14:27:33.159 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=1) 2023-10-30T14:27:33.160 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T14:27:33.160 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:27:33.179 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T14:27:33.179 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:27:33.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.363+0000 7f07a4d94640 1 Processor -- start 2023-10-30T14:27:33.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.363+0000 7f07a4d94640 1 -- start start 2023-10-30T14:27:33.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.363+0000 7f07a4d94640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07a0153c80 0x7f07a015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:33.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.363+0000 7f07a4d94640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f07a00c4a10 0x7f07a00c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:33.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.363+0000 7f07a4d94640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a0159f20 con 0x7f07a015a3b0 2023-10-30T14:27:33.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.363+0000 7f07a4d94640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a015dec0 con 0x7f07a00c4a10 2023-10-30T14:27:33.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.363+0000 7f07a4d94640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a01599a0 con 0x7f07a0153c80 2023-10-30T14:27:33.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.367+0000 7f0797fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f07a00c4a10 0x7f07a00c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:33.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.367+0000 7f0797fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f07a00c4a10 0x7f07a00c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34772/0 (socket says 172.21.15.57:34772) 2023-10-30T14:27:33.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.367+0000 7f0797fff640 1 -- 172.21.15.57:0/3326132183 learned_addr learned my addr 172.21.15.57:0/3326132183 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:33.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.367+0000 7f079e59c640 1 --2- 172.21.15.57:0/3326132183 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07a0153c80 0x7f07a015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:33.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.367+0000 7f0797fff640 1 -- 172.21.15.57:0/3326132183 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07a0153c80 msgr2=0x7f07a015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:33.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.367+0000 7f0797fff640 1 --2- 172.21.15.57:0/3326132183 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07a0153c80 0x7f07a015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:33.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.367+0000 7f0797fff640 1 -- 172.21.15.57:0/3326132183 >> v1:172.21.15.57:6789/0 conn(0x7f07a015a3b0 legacy=0x7f07a00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:33.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.367+0000 7f0797fff640 1 -- 172.21.15.57:0/3326132183 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a00c47e0 con 0x7f07a00c4a10 2023-10-30T14:27:33.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.367+0000 7f079dd9b640 1 -- 172.21.15.57:0/3326132183 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915505874 0 0) 0x7f07a0159f20 con 0x7f07a015a3b0 2023-10-30T14:27:33.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.367+0000 7f0797fff640 1 --2- 172.21.15.57:0/3326132183 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07a00c4a10 0x7f07a00c42a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f0788009870 tx=0x7f078801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad2b6cab4742bda server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:33.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.367+0000 7f079dd9b640 1 -- 172.21.15.57:0/3326132183 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07880a8a30 con 0x7f07a00c4a10 2023-10-30T14:27:33.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.367+0000 7f079dd9b640 1 -- 172.21.15.57:0/3326132183 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07880a8b90 con 0x7f07a00c4a10 2023-10-30T14:27:33.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.367+0000 7f079dd9b640 1 -- 172.21.15.57:0/3326132183 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07880b0440 con 0x7f07a00c4a10 2023-10-30T14:27:33.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f07a4d94640 1 -- 172.21.15.57:0/3326132183 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07a00c4a10 msgr2=0x7f07a00c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:33.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f07a4d94640 1 --2- 172.21.15.57:0/3326132183 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07a00c4a10 0x7f07a00c42a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f0788009870 tx=0x7f078801ed30 comp rx=0 tx=0).stop 2023-10-30T14:27:33.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f07a4d94640 1 -- 172.21.15.57:0/3326132183 shutdown_connections 2023-10-30T14:27:33.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f07a4d94640 1 --2- 172.21.15.57:0/3326132183 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07a00c4a10 0x7f07a00c42a0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:33.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f07a4d94640 1 --2- 172.21.15.57:0/3326132183 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07a0153c80 0x7f07a015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:33.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f07a4d94640 1 -- 172.21.15.57:0/3326132183 >> 172.21.15.57:0/3326132183 conn(0x7f07a00715c0 msgr2=0x7f07a00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:33.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f07a4d94640 1 -- 172.21.15.57:0/3326132183 shutdown_connections 2023-10-30T14:27:33.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f07a4d94640 1 -- 172.21.15.57:0/3326132183 wait complete. 2023-10-30T14:27:33.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f07a4d94640 1 Processor -- start 2023-10-30T14:27:33.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f07a4d94640 1 -- start start 2023-10-30T14:27:33.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f07a4d94640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07a00c4a10 0x7f07a00ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:33.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f07a4d94640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07a0153c80 0x7f07a00b4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:33.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f07a4d94640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07a015a3b0 0x7f07a00b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:33.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f07a4d94640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a00b4070 con 0x7f07a0153c80 2023-10-30T14:27:33.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f07a4d94640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a00b3180 con 0x7f07a015a3b0 2023-10-30T14:27:33.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f07a4d94640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a00b3460 con 0x7f07a00c4a10 2023-10-30T14:27:33.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f0797fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07a0153c80 0x7f07a00b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:33.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f0797fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07a0153c80 0x7f07a00b4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34788/0 (socket says 172.21.15.57:34788) 2023-10-30T14:27:33.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f0797fff640 1 -- 172.21.15.57:0/2621024644 learned_addr learned my addr 172.21.15.57:0/2621024644 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:33.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f079e59c640 1 --2- 172.21.15.57:0/2621024644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07a00c4a10 0x7f07a00ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:33.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f079ed9d640 1 --2- 172.21.15.57:0/2621024644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07a015a3b0 0x7f07a00b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:33.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f0797fff640 1 -- 172.21.15.57:0/2621024644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07a00c4a10 msgr2=0x7f07a00ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:33.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f0797fff640 1 --2- 172.21.15.57:0/2621024644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07a00c4a10 0x7f07a00ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:33.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f0797fff640 1 -- 172.21.15.57:0/2621024644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07a015a3b0 msgr2=0x7f07a00b4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:33.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f0797fff640 1 --2- 172.21.15.57:0/2621024644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07a015a3b0 0x7f07a00b4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:33.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f0797fff640 1 -- 172.21.15.57:0/2621024644 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07880094d0 con 0x7f07a0153c80 2023-10-30T14:27:33.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.371+0000 7f079ed9d640 1 --2- 172.21.15.57:0/2621024644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07a015a3b0 0x7f07a00b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:27:33.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.375+0000 7f0797fff640 1 --2- 172.21.15.57:0/2621024644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07a0153c80 0x7f07a00b4980 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f07880b3040 tx=0x7f078801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:33.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.375+0000 7f0796ffd640 1 -- 172.21.15.57:0/2621024644 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07880b0dd0 con 0x7f07a0153c80 2023-10-30T14:27:33.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.375+0000 7f0796ffd640 1 -- 172.21.15.57:0/2621024644 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07880bb460 con 0x7f07a0153c80 2023-10-30T14:27:33.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.375+0000 7f0796ffd640 1 -- 172.21.15.57:0/2621024644 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07880bb810 con 0x7f07a0153c80 2023-10-30T14:27:33.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.375+0000 7f07a4d94640 1 -- 172.21.15.57:0/2621024644 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a00b3740 con 0x7f07a0153c80 2023-10-30T14:27:33.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.375+0000 7f07a4d94640 1 -- 172.21.15.57:0/2621024644 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a00b3d00 con 0x7f07a0153c80 2023-10-30T14:27:33.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.375+0000 7f07a4d94640 1 -- 172.21.15.57:0/2621024644 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a004e170 con 0x7f07a0153c80 2023-10-30T14:27:33.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.375+0000 7f0796ffd640 1 -- 172.21.15.57:0/2621024644 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f07880b83f0 con 0x7f07a0153c80 2023-10-30T14:27:33.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.375+0000 7f0796ffd640 1 --2- 172.21.15.57:0/2621024644 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07800664a0 0x7f0780068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:33.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.375+0000 7f0796ffd640 1 -- 172.21.15.57:0/2621024644 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f0788121ab0 con 0x7f07a0153c80 2023-10-30T14:27:33.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.375+0000 7f079e59c640 1 --2- 172.21.15.57:0/2621024644 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07800664a0 0x7f0780068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:33.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.379+0000 7f079e59c640 1 --2- 172.21.15.57:0/2621024644 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07800664a0 0x7f0780068960 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0790009b60 tx=0x7f0790009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:33.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.379+0000 7f0796ffd640 1 -- 172.21.15.57:0/2621024644 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f07880eebe0 con 0x7f07a0153c80 2023-10-30T14:27:33.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.571+0000 7f07a4d94640 1 -- 172.21.15.57:0/2621024644 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07a00b1b20 con 0x7f07a0153c80 2023-10-30T14:27:33.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.575+0000 7f0796ffd640 1 -- 172.21.15.57:0/2621024644 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6507 (secure 0 0 0) 0x7f07880b5020 con 0x7f07a0153c80 2023-10-30T14:27:33.573 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:27:33.573 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":6027,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:27:05.900123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:27:33.573 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 80 2023-10-30T14:27:33.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.575+0000 7f07a4d94640 1 -- 172.21.15.57:0/2621024644 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07800664a0 msgr2=0x7f0780068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:33.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.575+0000 7f07a4d94640 1 --2- 172.21.15.57:0/2621024644 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07800664a0 0x7f0780068960 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0790009b60 tx=0x7f0790009190 comp rx=0 tx=0).stop 2023-10-30T14:27:33.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.575+0000 7f07a4d94640 1 -- 172.21.15.57:0/2621024644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07a0153c80 msgr2=0x7f07a00b4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:33.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.575+0000 7f07a4d94640 1 --2- 172.21.15.57:0/2621024644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07a0153c80 0x7f07a00b4980 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f07880b3040 tx=0x7f078801bad0 comp rx=0 tx=0).stop 2023-10-30T14:27:33.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.575+0000 7f07a4d94640 1 -- 172.21.15.57:0/2621024644 shutdown_connections 2023-10-30T14:27:33.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.575+0000 7f07a4d94640 1 --2- 172.21.15.57:0/2621024644 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07800664a0 0x7f0780068960 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:33.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.575+0000 7f07a4d94640 1 --2- 172.21.15.57:0/2621024644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07a015a3b0 0x7f07a00b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:33.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.575+0000 7f07a4d94640 1 --2- 172.21.15.57:0/2621024644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07a0153c80 0x7f07a00b4980 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:33.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.575+0000 7f07a4d94640 1 --2- 172.21.15.57:0/2621024644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07a00c4a10 0x7f07a00ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:33.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.575+0000 7f07a4d94640 1 -- 172.21.15.57:0/2621024644 >> 172.21.15.57:0/2621024644 conn(0x7f07a00715c0 msgr2=0x7f07a0157a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:33.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.575+0000 7f07a4d94640 1 -- 172.21.15.57:0/2621024644 shutdown_connections 2023-10-30T14:27:33.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:33.575+0000 7f07a4d94640 1 -- 172.21.15.57:0/2621024644 wait complete. 2023-10-30T14:27:33.595 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:27:33.896 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:27:33.900 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:27:35.596 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:27:35.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.751+0000 7fb675301640 1 Processor -- start 2023-10-30T14:27:35.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.751+0000 7fb675301640 1 -- start start 2023-10-30T14:27:35.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.751+0000 7fb675301640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb67014fc50 0x7fb67006fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:35.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.751+0000 7fb675301640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67006be90 0x7fb67006c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:35.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.751+0000 7fb675301640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6700c55f0 con 0x7fb6700c6320 2023-10-30T14:27:35.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.751+0000 7fb675301640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6700c5f00 con 0x7fb67014fc50 2023-10-30T14:27:35.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.751+0000 7fb675301640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6701567b0 con 0x7fb67006be90 2023-10-30T14:27:35.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.751+0000 7fb667fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb6700c6320 0x7fb670070e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:34000/0 (socket says 172.21.15.57:34000) 2023-10-30T14:27:35.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.751+0000 7fb667fff640 1 -- 172.21.15.57:0/799319751 learned_addr learned my addr 172.21.15.57:0/799319751 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:27:35.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.751+0000 7fb66effd640 1 --2- 172.21.15.57:0/799319751 >> v2:172.21.15.88:3300/0 conn(0x7fb67014fc50 0x7fb67006fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:35.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.751+0000 7fb66f7fe640 1 --2- 172.21.15.57:0/799319751 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67006be90 0x7fb67006c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:35.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.755+0000 7fb66e7fc640 1 -- 172.21.15.57:0/799319751 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176957191 0 0) 0x7fb6700c55f0 con 0x7fb6700c6320 2023-10-30T14:27:35.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.755+0000 7fb66e7fc640 1 -- 172.21.15.57:0/799319751 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb65c003450 con 0x7fb6700c6320 2023-10-30T14:27:35.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.755+0000 7fb66f7fe640 1 -- 172.21.15.57:0/799319751 >> v2:172.21.15.88:3300/0 conn(0x7fb67014fc50 msgr2=0x7fb67006fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:35.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.755+0000 7fb66f7fe640 1 --2- 172.21.15.57:0/799319751 >> v2:172.21.15.88:3300/0 conn(0x7fb67014fc50 0x7fb67006fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:35.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.755+0000 7fb66e7fc640 1 -- 172.21.15.57:0/799319751 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1124610403 0 0) 0x7fb65c003450 con 0x7fb6700c6320 2023-10-30T14:27:35.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.755+0000 7fb66f7fe640 1 -- 172.21.15.57:0/799319751 >> v1:172.21.15.57:6789/0 conn(0x7fb6700c6320 legacy=0x7fb670070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:35.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.755+0000 7fb66f7fe640 1 -- 172.21.15.57:0/799319751 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb670156a70 con 0x7fb67006be90 2023-10-30T14:27:35.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.755+0000 7fb66f7fe640 1 --2- 172.21.15.57:0/799319751 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67006be90 0x7fb67006c320 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb668009d90 tx=0x7fb66801c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=720054b01f70480e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:35.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.755+0000 7fb66e7fc640 1 -- 172.21.15.57:0/799319751 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb658001680 con 0x7fb67006be90 2023-10-30T14:27:35.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.755+0000 7fb66e7fc640 1 -- 172.21.15.57:0/799319751 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6680a5560 con 0x7fb67006be90 2023-10-30T14:27:35.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.755+0000 7fb66e7fc640 1 -- 172.21.15.57:0/799319751 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6680a58d0 con 0x7fb67006be90 2023-10-30T14:27:35.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb675301640 1 -- 172.21.15.57:0/799319751 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67006be90 msgr2=0x7fb67006c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:35.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb675301640 1 --2- 172.21.15.57:0/799319751 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67006be90 0x7fb67006c320 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb668009d90 tx=0x7fb66801c7e0 comp rx=0 tx=0).stop 2023-10-30T14:27:35.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb675301640 1 -- 172.21.15.57:0/799319751 shutdown_connections 2023-10-30T14:27:35.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb675301640 1 --2- 172.21.15.57:0/799319751 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67006be90 0x7fb67006c320 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:35.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb675301640 1 --2- 172.21.15.57:0/799319751 >> v2:172.21.15.88:3300/0 conn(0x7fb67014fc50 0x7fb67006fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:35.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb675301640 1 -- 172.21.15.57:0/799319751 >> 172.21.15.57:0/799319751 conn(0x7fb6700cac70 msgr2=0x7fb6700cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:35.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb675301640 1 -- 172.21.15.57:0/799319751 shutdown_connections 2023-10-30T14:27:35.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb675301640 1 -- 172.21.15.57:0/799319751 wait complete. 2023-10-30T14:27:35.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb675301640 1 Processor -- start 2023-10-30T14:27:35.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb675301640 1 -- start start 2023-10-30T14:27:35.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb675301640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67006be90 0x7fb67013ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:35.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb675301640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6700c6320 0x7fb67015e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:35.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb675301640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb67014fc50 0x7fb67013f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:35.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb675301640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6700c9990 con 0x7fb67014fc50 2023-10-30T14:27:35.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb675301640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6701f2220 con 0x7fb6700c6320 2023-10-30T14:27:35.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb675301640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6701f2500 con 0x7fb67006be90 2023-10-30T14:27:35.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb66f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb67014fc50 0x7fb67013f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:35.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb66f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb67014fc50 0x7fb67013f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57908/0 (socket says 172.21.15.57:57908) 2023-10-30T14:27:35.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb66f7fe640 1 -- 172.21.15.57:0/505954484 learned_addr learned my addr 172.21.15.57:0/505954484 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:35.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb66effd640 1 --2- 172.21.15.57:0/505954484 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67006be90 0x7fb67013ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:35.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.759+0000 7fb667fff640 1 --2- 172.21.15.57:0/505954484 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6700c6320 0x7fb67015e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:35.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.763+0000 7fb66f7fe640 1 -- 172.21.15.57:0/505954484 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67006be90 msgr2=0x7fb67013ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:35.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.763+0000 7fb66f7fe640 1 --2- 172.21.15.57:0/505954484 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67006be90 0x7fb67013ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:35.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.763+0000 7fb66f7fe640 1 -- 172.21.15.57:0/505954484 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6700c6320 msgr2=0x7fb67015e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:35.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.763+0000 7fb66f7fe640 1 --2- 172.21.15.57:0/505954484 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6700c6320 0x7fb67015e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:35.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.763+0000 7fb66f7fe640 1 -- 172.21.15.57:0/505954484 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb668009a70 con 0x7fb67014fc50 2023-10-30T14:27:35.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.763+0000 7fb66f7fe640 1 --2- 172.21.15.57:0/505954484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb67014fc50 0x7fb67013f390 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb66801d080 tx=0x7fb668019580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:35.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.763+0000 7fb6677fe640 1 -- 172.21.15.57:0/505954484 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6680a58e0 con 0x7fb67014fc50 2023-10-30T14:27:35.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.763+0000 7fb675301640 1 -- 172.21.15.57:0/505954484 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6700c9c00 con 0x7fb67014fc50 2023-10-30T14:27:35.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.763+0000 7fb6677fe640 1 -- 172.21.15.57:0/505954484 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6680a5a40 con 0x7fb67014fc50 2023-10-30T14:27:35.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.763+0000 7fb6677fe640 1 -- 172.21.15.57:0/505954484 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6680a5db0 con 0x7fb67014fc50 2023-10-30T14:27:35.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.763+0000 7fb675301640 1 -- 172.21.15.57:0/505954484 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6700ca1c0 con 0x7fb67014fc50 2023-10-30T14:27:35.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.763+0000 7fb675301640 1 -- 172.21.15.57:0/505954484 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6700ca3b0 con 0x7fb67014fc50 2023-10-30T14:27:35.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.763+0000 7fb6677fe640 1 -- 172.21.15.57:0/505954484 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb6680b4070 con 0x7fb67014fc50 2023-10-30T14:27:35.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.763+0000 7fb6677fe640 1 --2- 172.21.15.57:0/505954484 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb6500664a0 0x7fb650068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:35.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.763+0000 7fb6677fe640 1 -- 172.21.15.57:0/505954484 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7fb66811e5e0 con 0x7fb67014fc50 2023-10-30T14:27:35.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.767+0000 7fb66effd640 1 --2- 172.21.15.57:0/505954484 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb6500664a0 0x7fb650068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:35.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.767+0000 7fb66effd640 1 --2- 172.21.15.57:0/505954484 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb6500664a0 0x7fb650068960 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb660009ac0 tx=0x7fb660009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:35.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.767+0000 7fb6677fe640 1 -- 172.21.15.57:0/505954484 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb6680eb720 con 0x7fb67014fc50 2023-10-30T14:27:35.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.963+0000 7fb675301640 1 -- 172.21.15.57:0/505954484 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb67004e170 con 0x7fb67014fc50 2023-10-30T14:27:35.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.963+0000 7fb6677fe640 1 -- 172.21.15.57:0/505954484 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6507 (secure 0 0 0) 0x7fb6680eed40 con 0x7fb67014fc50 2023-10-30T14:27:35.962 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:27:35.962 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":6027,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:27:05.900123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:27:35.963 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 80 2023-10-30T14:27:35.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.963+0000 7fb675301640 1 -- 172.21.15.57:0/505954484 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb6500664a0 msgr2=0x7fb650068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:35.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.963+0000 7fb675301640 1 --2- 172.21.15.57:0/505954484 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb6500664a0 0x7fb650068960 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb660009ac0 tx=0x7fb660009190 comp rx=0 tx=0).stop 2023-10-30T14:27:35.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.963+0000 7fb675301640 1 -- 172.21.15.57:0/505954484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb67014fc50 msgr2=0x7fb67013f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:35.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.963+0000 7fb675301640 1 --2- 172.21.15.57:0/505954484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb67014fc50 0x7fb67013f390 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb66801d080 tx=0x7fb668019580 comp rx=0 tx=0).stop 2023-10-30T14:27:35.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.963+0000 7fb675301640 1 -- 172.21.15.57:0/505954484 shutdown_connections 2023-10-30T14:27:35.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.963+0000 7fb675301640 1 --2- 172.21.15.57:0/505954484 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb6500664a0 0x7fb650068960 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:35.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.963+0000 7fb675301640 1 --2- 172.21.15.57:0/505954484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb67014fc50 0x7fb67013f390 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:35.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.963+0000 7fb675301640 1 --2- 172.21.15.57:0/505954484 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6700c6320 0x7fb67015e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:35.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.963+0000 7fb675301640 1 --2- 172.21.15.57:0/505954484 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67006be90 0x7fb67013ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:35.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.963+0000 7fb675301640 1 -- 172.21.15.57:0/505954484 >> 172.21.15.57:0/505954484 conn(0x7fb6700cac70 msgr2=0x7fb67014eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:35.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.963+0000 7fb675301640 1 -- 172.21.15.57:0/505954484 shutdown_connections 2023-10-30T14:27:35.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:35.963+0000 7fb675301640 1 -- 172.21.15.57:0/505954484 wait complete. 2023-10-30T14:27:35.984 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:27:37.985 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:27:38.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.139+0000 7f85cfdc4640 1 Processor -- start 2023-10-30T14:27:38.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.139+0000 7f85cfdc4640 1 -- start start 2023-10-30T14:27:38.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.143+0000 7f85cfdc4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f85c806c420 0x7f85c80c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:38.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.143+0000 7f85cfdc4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c8071e00 con 0x7f85c806b760 2023-10-30T14:27:38.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.143+0000 7f85cfdc4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c8070020 con 0x7f85c806bb40 2023-10-30T14:27:38.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.143+0000 7f85cfdc4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c8070f10 con 0x7f85c806c420 2023-10-30T14:27:38.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.143+0000 7f85cdb60640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f85c806b760 0x7f85c815e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:34014/0 (socket says 172.21.15.57:34014) 2023-10-30T14:27:38.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.143+0000 7f85cdb60640 1 -- 172.21.15.57:0/1651248478 learned_addr learned my addr 172.21.15.57:0/1651248478 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:27:38.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.143+0000 7f85ce361640 1 --2- 172.21.15.57:0/1651248478 >> v2:172.21.15.57:3300/0 conn(0x7f85c806c420 0x7f85c80c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:38.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.143+0000 7f85ce361640 1 -- 172.21.15.57:0/1651248478 >> v1:172.21.15.88:6789/0 conn(0x7f85c806bb40 legacy=0x7f85c8157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:38.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.143+0000 7f85ce361640 1 -- 172.21.15.57:0/1651248478 >> v1:172.21.15.57:6789/0 conn(0x7f85c806b760 legacy=0x7f85c815e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:38.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.143+0000 7f85ce361640 1 -- 172.21.15.57:0/1651248478 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c80711f0 con 0x7f85c806c420 2023-10-30T14:27:38.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.143+0000 7f85ccb5e640 1 -- 172.21.15.57:0/1651248478 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265501766 0 0) 0x7f85c8071e00 con 0x7f85c806b760 2023-10-30T14:27:38.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.147+0000 7f85ce361640 1 --2- 172.21.15.57:0/1651248478 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85c806c420 0x7f85c80c57a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f85c0009d90 tx=0x7f85c001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35328ac83da4c6d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:38.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.147+0000 7f85ccb5e640 1 -- 172.21.15.57:0/1651248478 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c00a59e0 con 0x7f85c806c420 2023-10-30T14:27:38.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.147+0000 7f85ccb5e640 1 -- 172.21.15.57:0/1651248478 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c00a5b40 con 0x7f85c806c420 2023-10-30T14:27:38.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.147+0000 7f85ccb5e640 1 -- 172.21.15.57:0/1651248478 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c00ad440 con 0x7f85c806c420 2023-10-30T14:27:38.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.147+0000 7f85cfdc4640 1 -- 172.21.15.57:0/1651248478 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85c806c420 msgr2=0x7f85c80c57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:38.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.147+0000 7f85cfdc4640 1 --2- 172.21.15.57:0/1651248478 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85c806c420 0x7f85c80c57a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f85c0009d90 tx=0x7f85c001c920 comp rx=0 tx=0).stop 2023-10-30T14:27:38.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.147+0000 7f85cfdc4640 1 -- 172.21.15.57:0/1651248478 shutdown_connections 2023-10-30T14:27:38.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.147+0000 7f85cfdc4640 1 --2- 172.21.15.57:0/1651248478 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85c806c420 0x7f85c80c57a0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:38.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.147+0000 7f85cfdc4640 1 -- 172.21.15.57:0/1651248478 >> 172.21.15.57:0/1651248478 conn(0x7f85c80cac70 msgr2=0x7f85c80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:38.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.147+0000 7f85cfdc4640 1 -- 172.21.15.57:0/1651248478 shutdown_connections 2023-10-30T14:27:38.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.147+0000 7f85cfdc4640 1 -- 172.21.15.57:0/1651248478 wait complete. 2023-10-30T14:27:38.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.147+0000 7f85cfdc4640 1 Processor -- start 2023-10-30T14:27:38.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.147+0000 7f85cfdc4640 1 -- start start 2023-10-30T14:27:38.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85cfdc4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85c806b760 0x7f85c815e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:38.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85cfdc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85c806bb40 0x7f85c80c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:38.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85cfdc4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f85c806c420 0x7f85c8145700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:38.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85cfdc4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c814e160 con 0x7f85c806b760 2023-10-30T14:27:38.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85cfdc4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c80c9800 con 0x7f85c806bb40 2023-10-30T14:27:38.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85cfdc4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c80c8bf0 con 0x7f85c806c420 2023-10-30T14:27:38.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85cdb60640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85c806b760 0x7f85c815e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:38.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85cdb60640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85c806b760 0x7f85c815e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57930/0 (socket says 172.21.15.57:57930) 2023-10-30T14:27:38.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85cdb60640 1 -- 172.21.15.57:0/3719522411 learned_addr learned my addr 172.21.15.57:0/3719522411 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:38.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85cd35f640 1 --2- 172.21.15.57:0/3719522411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85c806bb40 0x7f85c80c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:38.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85ce361640 1 --2- 172.21.15.57:0/3719522411 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f85c806c420 0x7f85c8145700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:38.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85cdb60640 1 -- 172.21.15.57:0/3719522411 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f85c806c420 msgr2=0x7f85c8145700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:38.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85cdb60640 1 --2- 172.21.15.57:0/3719522411 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f85c806c420 0x7f85c8145700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:38.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85cdb60640 1 -- 172.21.15.57:0/3719522411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85c806bb40 msgr2=0x7f85c80c9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:38.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85cdb60640 1 --2- 172.21.15.57:0/3719522411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85c806bb40 0x7f85c80c9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:38.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85cdb60640 1 -- 172.21.15.57:0/3719522411 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c0009a70 con 0x7f85c806b760 2023-10-30T14:27:38.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85ce361640 1 --2- 172.21.15.57:0/3719522411 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f85c806c420 0x7f85c8145700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:27:38.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85cdb60640 1 --2- 172.21.15.57:0/3719522411 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85c806b760 0x7f85c815e8e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f85c400c4a0 tx=0x7f85c400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:38.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85be7fc640 1 -- 172.21.15.57:0/3719522411 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c40074e0 con 0x7f85c806b760 2023-10-30T14:27:38.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85be7fc640 1 -- 172.21.15.57:0/3719522411 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c4007640 con 0x7f85c806b760 2023-10-30T14:27:38.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85be7fc640 1 -- 172.21.15.57:0/3719522411 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c4004a30 con 0x7f85c806b760 2023-10-30T14:27:38.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85cfdc4640 1 -- 172.21.15.57:0/3719522411 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c80c8ed0 con 0x7f85c806b760 2023-10-30T14:27:38.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85cfdc4640 1 -- 172.21.15.57:0/3719522411 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c80c9490 con 0x7f85c806b760 2023-10-30T14:27:38.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85cfdc4640 1 -- 172.21.15.57:0/3719522411 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c806c250 con 0x7f85c806b760 2023-10-30T14:27:38.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.151+0000 7f85be7fc640 1 -- 172.21.15.57:0/3719522411 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f85c4004b90 con 0x7f85c806b760 2023-10-30T14:27:38.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.155+0000 7f85be7fc640 1 --2- 172.21.15.57:0/3719522411 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85ac0664d0 0x7f85ac068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:38.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.155+0000 7f85be7fc640 1 -- 172.21.15.57:0/3719522411 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f85c4083250 con 0x7f85c806b760 2023-10-30T14:27:38.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.155+0000 7f85cd35f640 1 --2- 172.21.15.57:0/3719522411 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85ac0664d0 0x7f85ac068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:38.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.155+0000 7f85cd35f640 1 --2- 172.21.15.57:0/3719522411 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85ac0664d0 0x7f85ac068990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f85c80ca7a0 tx=0x7f85b8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:38.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.159+0000 7f85be7fc640 1 -- 172.21.15.57:0/3719522411 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f85c4050410 con 0x7f85c806b760 2023-10-30T14:27:38.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.347+0000 7f85cfdc4640 1 -- 172.21.15.57:0/3719522411 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85c804e170 con 0x7f85c806b760 2023-10-30T14:27:38.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.351+0000 7f85be7fc640 1 -- 172.21.15.57:0/3719522411 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6507 (secure 0 0 0) 0x7f85c4053a30 con 0x7f85c806b760 2023-10-30T14:27:38.349 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:27:38.349 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":6027,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:27:05.900123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:27:38.350 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 80 2023-10-30T14:27:38.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.351+0000 7f85cfdc4640 1 -- 172.21.15.57:0/3719522411 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85ac0664d0 msgr2=0x7f85ac068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:38.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.351+0000 7f85cfdc4640 1 --2- 172.21.15.57:0/3719522411 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85ac0664d0 0x7f85ac068990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f85c80ca7a0 tx=0x7f85b8009040 comp rx=0 tx=0).stop 2023-10-30T14:27:38.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.351+0000 7f85cfdc4640 1 -- 172.21.15.57:0/3719522411 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85c806b760 msgr2=0x7f85c815e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:38.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.351+0000 7f85cfdc4640 1 --2- 172.21.15.57:0/3719522411 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85c806b760 0x7f85c815e8e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f85c400c4a0 tx=0x7f85c400c960 comp rx=0 tx=0).stop 2023-10-30T14:27:38.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.351+0000 7f85cfdc4640 1 -- 172.21.15.57:0/3719522411 shutdown_connections 2023-10-30T14:27:38.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.351+0000 7f85cfdc4640 1 --2- 172.21.15.57:0/3719522411 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85ac0664d0 0x7f85ac068990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:38.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.351+0000 7f85cfdc4640 1 --2- 172.21.15.57:0/3719522411 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f85c806c420 0x7f85c8145700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:38.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.351+0000 7f85cfdc4640 1 --2- 172.21.15.57:0/3719522411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85c806bb40 0x7f85c80c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:38.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.351+0000 7f85cfdc4640 1 --2- 172.21.15.57:0/3719522411 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85c806b760 0x7f85c815e8e0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:38.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.351+0000 7f85cfdc4640 1 -- 172.21.15.57:0/3719522411 >> 172.21.15.57:0/3719522411 conn(0x7f85c80cac70 msgr2=0x7f85c81538e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:38.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.351+0000 7f85cfdc4640 1 -- 172.21.15.57:0/3719522411 shutdown_connections 2023-10-30T14:27:38.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:38.351+0000 7f85cfdc4640 1 -- 172.21.15.57:0/3719522411 wait complete. 2023-10-30T14:27:38.370 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:27:40.371 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:27:40.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.527+0000 7f140d5c5640 1 Processor -- start 2023-10-30T14:27:40.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.527+0000 7f140d5c5640 1 -- start start 2023-10-30T14:27:40.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.527+0000 7f140d5c5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1408150830 0x7f140806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:40.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.527+0000 7f140d5c5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1408070520 0x7f14080709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:40.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.527+0000 7f140d5c5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14080c2310 con 0x7f14080cb7d0 2023-10-30T14:27:40.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.527+0000 7f140d5c5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1408156710 con 0x7f1408070520 2023-10-30T14:27:40.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.527+0000 7f140d5c5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14080c4360 con 0x7f1408150830 2023-10-30T14:27:40.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.527+0000 7f13fffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f14080cb7d0 0x7f14080c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:34024/0 (socket says 172.21.15.57:34024) 2023-10-30T14:27:40.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.527+0000 7f14077fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1408070520 0x7f14080709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:40.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.527+0000 7f13fffff640 1 -- 172.21.15.57:0/2551400983 learned_addr learned my addr 172.21.15.57:0/2551400983 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:27:40.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.527+0000 7f1406ffd640 1 --2- 172.21.15.57:0/2551400983 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1408150830 0x7f140806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:40.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.527+0000 7f14067fc640 1 -- 172.21.15.57:0/2551400983 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529280893 0 0) 0x7f14080c2310 con 0x7f14080cb7d0 2023-10-30T14:27:40.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.527+0000 7f14067fc640 1 -- 172.21.15.57:0/2551400983 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13f4005740 con 0x7f14080cb7d0 2023-10-30T14:27:40.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.527+0000 7f14077fe640 1 -- 172.21.15.57:0/2551400983 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1408150830 msgr2=0x7f140806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:40.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.527+0000 7f14077fe640 1 --2- 172.21.15.57:0/2551400983 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1408150830 0x7f140806b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:40.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.527+0000 7f14077fe640 1 -- 172.21.15.57:0/2551400983 >> v1:172.21.15.57:6789/0 conn(0x7f14080cb7d0 legacy=0x7f14080c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:40.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.527+0000 7f14077fe640 1 -- 172.21.15.57:0/2551400983 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14081569b0 con 0x7f1408070520 2023-10-30T14:27:40.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.527+0000 7f1406ffd640 1 --2- 172.21.15.57:0/2551400983 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1408150830 0x7f140806b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:27:40.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.531+0000 7f14077fe640 1 --2- 172.21.15.57:0/2551400983 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1408070520 0x7f14080709b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1400009d90 tx=0x7f140001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1093f84002ced528 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:40.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.531+0000 7f14067fc640 1 -- 172.21.15.57:0/2551400983 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14000af600 con 0x7f1408070520 2023-10-30T14:27:40.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.531+0000 7f14067fc640 1 -- 172.21.15.57:0/2551400983 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14000ab050 con 0x7f1408070520 2023-10-30T14:27:40.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.531+0000 7f14067fc640 1 -- 172.21.15.57:0/2551400983 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14000af980 con 0x7f1408070520 2023-10-30T14:27:40.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.531+0000 7f140d5c5640 1 -- 172.21.15.57:0/2551400983 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1408070520 msgr2=0x7f14080709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:40.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.531+0000 7f140d5c5640 1 --2- 172.21.15.57:0/2551400983 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1408070520 0x7f14080709b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1400009d90 tx=0x7f140001f130 comp rx=0 tx=0).stop 2023-10-30T14:27:40.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.531+0000 7f140d5c5640 1 -- 172.21.15.57:0/2551400983 shutdown_connections 2023-10-30T14:27:40.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.531+0000 7f140d5c5640 1 --2- 172.21.15.57:0/2551400983 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1408070520 0x7f14080709b0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:40.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.531+0000 7f140d5c5640 1 --2- 172.21.15.57:0/2551400983 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1408150830 0x7f140806b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:40.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.531+0000 7f140d5c5640 1 -- 172.21.15.57:0/2551400983 >> 172.21.15.57:0/2551400983 conn(0x7f14080cac70 msgr2=0x7f14080cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:40.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.531+0000 7f140d5c5640 1 -- 172.21.15.57:0/2551400983 shutdown_connections 2023-10-30T14:27:40.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.531+0000 7f140d5c5640 1 -- 172.21.15.57:0/2551400983 wait complete. 2023-10-30T14:27:40.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f140d5c5640 1 Processor -- start 2023-10-30T14:27:40.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f140d5c5640 1 -- start start 2023-10-30T14:27:40.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f140d5c5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1408070520 0x7f1408154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:40.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f140d5c5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14080cb7d0 0x7f14081f0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:40.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f140d5c5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1408150830 0x7f14081ef610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:40.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f140d5c5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1408154190 con 0x7f14080cb7d0 2023-10-30T14:27:40.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f140d5c5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14081efb50 con 0x7f1408150830 2023-10-30T14:27:40.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f140d5c5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14081dff50 con 0x7f1408070520 2023-10-30T14:27:40.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f13fffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14080cb7d0 0x7f14081f0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:40.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f13fffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14080cb7d0 0x7f14081f0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57948/0 (socket says 172.21.15.57:57948) 2023-10-30T14:27:40.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f13fffff640 1 -- 172.21.15.57:0/822836953 learned_addr learned my addr 172.21.15.57:0/822836953 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:40.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f1406ffd640 1 --2- 172.21.15.57:0/822836953 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1408070520 0x7f1408154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:40.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f14077fe640 1 --2- 172.21.15.57:0/822836953 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1408150830 0x7f14081ef610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:40.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f13fffff640 1 -- 172.21.15.57:0/822836953 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1408070520 msgr2=0x7f1408154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:40.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f13fffff640 1 --2- 172.21.15.57:0/822836953 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1408070520 0x7f1408154ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:40.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f13fffff640 1 -- 172.21.15.57:0/822836953 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1408150830 msgr2=0x7f14081ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:40.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f13fffff640 1 --2- 172.21.15.57:0/822836953 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1408150830 0x7f14081ef610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:40.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f13fffff640 1 -- 172.21.15.57:0/822836953 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1400009a70 con 0x7f14080cb7d0 2023-10-30T14:27:40.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f13fffff640 1 --2- 172.21.15.57:0/822836953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14080cb7d0 0x7f14081f0000 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f13f000c4d0 tx=0x7f13f000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:40.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f13ff7fe640 1 -- 172.21.15.57:0/822836953 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f0005cf0 con 0x7f14080cb7d0 2023-10-30T14:27:40.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f13ff7fe640 1 -- 172.21.15.57:0/822836953 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f0005e50 con 0x7f14080cb7d0 2023-10-30T14:27:40.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f13ff7fe640 1 -- 172.21.15.57:0/822836953 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f0007270 con 0x7f14080cb7d0 2023-10-30T14:27:40.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f140d5c5640 1 -- 172.21.15.57:0/822836953 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14081e01f0 con 0x7f14080cb7d0 2023-10-30T14:27:40.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.535+0000 7f140d5c5640 1 -- 172.21.15.57:0/822836953 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14081546a0 con 0x7f14080cb7d0 2023-10-30T14:27:40.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.539+0000 7f140d5c5640 1 -- 172.21.15.57:0/822836953 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f140804e170 con 0x7f14080cb7d0 2023-10-30T14:27:40.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.539+0000 7f13ff7fe640 1 -- 172.21.15.57:0/822836953 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f13f00073d0 con 0x7f14080cb7d0 2023-10-30T14:27:40.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.539+0000 7f13ff7fe640 1 --2- 172.21.15.57:0/822836953 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f13e80664a0 0x7f13e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:40.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.539+0000 7f13ff7fe640 1 -- 172.21.15.57:0/822836953 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f13f0083740 con 0x7f14080cb7d0 2023-10-30T14:27:40.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.539+0000 7f1406ffd640 1 --2- 172.21.15.57:0/822836953 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f13e80664a0 0x7f13e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:40.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.543+0000 7f1406ffd640 1 --2- 172.21.15.57:0/822836953 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f13e80664a0 0x7f13e8068960 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f13f800ba10 tx=0x7f13f8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:40.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.543+0000 7f13ff7fe640 1 -- 172.21.15.57:0/822836953 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f13f0050870 con 0x7f14080cb7d0 2023-10-30T14:27:40.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.735+0000 7f140d5c5640 1 -- 172.21.15.57:0/822836953 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1408070300 con 0x7f14080cb7d0 2023-10-30T14:27:40.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.739+0000 7f13ff7fe640 1 -- 172.21.15.57:0/822836953 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6507 (secure 0 0 0) 0x7f13f0017090 con 0x7f14080cb7d0 2023-10-30T14:27:40.737 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:27:40.737 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":6027,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:27:05.900123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:27:40.738 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 80 2023-10-30T14:27:40.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.739+0000 7f140d5c5640 1 -- 172.21.15.57:0/822836953 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f13e80664a0 msgr2=0x7f13e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:40.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.739+0000 7f140d5c5640 1 --2- 172.21.15.57:0/822836953 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f13e80664a0 0x7f13e8068960 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f13f800ba10 tx=0x7f13f8008040 comp rx=0 tx=0).stop 2023-10-30T14:27:40.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.739+0000 7f140d5c5640 1 -- 172.21.15.57:0/822836953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14080cb7d0 msgr2=0x7f14081f0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:40.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.739+0000 7f140d5c5640 1 --2- 172.21.15.57:0/822836953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14080cb7d0 0x7f14081f0000 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f13f000c4d0 tx=0x7f13f000c990 comp rx=0 tx=0).stop 2023-10-30T14:27:40.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.739+0000 7f140d5c5640 1 -- 172.21.15.57:0/822836953 shutdown_connections 2023-10-30T14:27:40.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.739+0000 7f140d5c5640 1 --2- 172.21.15.57:0/822836953 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f13e80664a0 0x7f13e8068960 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:40.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.739+0000 7f140d5c5640 1 --2- 172.21.15.57:0/822836953 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1408150830 0x7f14081ef610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:40.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.739+0000 7f140d5c5640 1 --2- 172.21.15.57:0/822836953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14080cb7d0 0x7f14081f0000 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:40.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.739+0000 7f140d5c5640 1 --2- 172.21.15.57:0/822836953 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1408070520 0x7f1408154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:40.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.739+0000 7f140d5c5640 1 -- 172.21.15.57:0/822836953 >> 172.21.15.57:0/822836953 conn(0x7f14080cac70 msgr2=0x7f14080c6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:40.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.739+0000 7f140d5c5640 1 -- 172.21.15.57:0/822836953 shutdown_connections 2023-10-30T14:27:40.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:40.739+0000 7f140d5c5640 1 -- 172.21.15.57:0/822836953 wait complete. 2023-10-30T14:27:40.759 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:27:42.760 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:27:42.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.919+0000 7f9be5617640 1 Processor -- start 2023-10-30T14:27:42.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.919+0000 7f9be5617640 1 -- start start 2023-10-30T14:27:42.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.919+0000 7f9be5617640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9be0153b10 0x7f9be00c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:42.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.919+0000 7f9be5617640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9be00ca8e0 0x7f9be006bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:42.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.919+0000 7f9be5617640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be015d070 con 0x7f9be006c480 2023-10-30T14:27:42.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.919+0000 7f9be5617640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be0159900 con 0x7f9be0153b10 2023-10-30T14:27:42.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.919+0000 7f9be5617640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be015d350 con 0x7f9be00ca8e0 2023-10-30T14:27:42.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.919+0000 7f9bdeffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9be0153b10 0x7f9be00c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:42.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.919+0000 7f9bdeffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9be0153b10 0x7f9be00c9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57950/0 (socket says 172.21.15.57:57950) 2023-10-30T14:27:42.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.919+0000 7f9bdeffd640 1 -- 172.21.15.57:0/3989712021 learned_addr learned my addr 172.21.15.57:0/3989712021 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:42.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.919+0000 7f9bdf7fe640 1 --1- 172.21.15.57:0/3989712021 >> v1:172.21.15.57:6789/0 conn(0x7f9be006c480 0x7f9be0159d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:34028/0 (socket says 172.21.15.57:34028) 2023-10-30T14:27:42.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.919+0000 7f9bd7fff640 1 --2- 172.21.15.57:0/3989712021 >> v2:172.21.15.88:3300/0 conn(0x7f9be00ca8e0 0x7f9be006bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:42.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.919+0000 7f9bdeffd640 1 -- 172.21.15.57:0/3989712021 >> v2:172.21.15.88:3300/0 conn(0x7f9be00ca8e0 msgr2=0x7f9be006bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:42.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.919+0000 7f9bdeffd640 1 --2- 172.21.15.57:0/3989712021 >> v2:172.21.15.88:3300/0 conn(0x7f9be00ca8e0 0x7f9be006bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:42.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.919+0000 7f9bdeffd640 1 -- 172.21.15.57:0/3989712021 >> v1:172.21.15.57:6789/0 conn(0x7f9be006c480 legacy=0x7f9be0159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:42.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.919+0000 7f9bdeffd640 1 -- 172.21.15.57:0/3989712021 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be015d630 con 0x7f9be0153b10 2023-10-30T14:27:42.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.919+0000 7f9bde7fc640 1 -- 172.21.15.57:0/3989712021 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589422963 0 0) 0x7f9be015d070 con 0x7f9be006c480 2023-10-30T14:27:42.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.923+0000 7f9bdeffd640 1 --2- 172.21.15.57:0/3989712021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9be0153b10 0x7f9be00c9b90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f9be00b1dd0 tx=0x7f9bd001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c607ba8e32b5f2aa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:42.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.923+0000 7f9bde7fc640 1 -- 172.21.15.57:0/3989712021 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd00ab610 con 0x7f9be0153b10 2023-10-30T14:27:42.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.923+0000 7f9bde7fc640 1 -- 172.21.15.57:0/3989712021 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd00ab770 con 0x7f9be0153b10 2023-10-30T14:27:42.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.923+0000 7f9bde7fc640 1 -- 172.21.15.57:0/3989712021 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd00abb00 con 0x7f9be0153b10 2023-10-30T14:27:42.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.923+0000 7f9be5617640 1 -- 172.21.15.57:0/3989712021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9be0153b10 msgr2=0x7f9be00c9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:42.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.923+0000 7f9be5617640 1 --2- 172.21.15.57:0/3989712021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9be0153b10 0x7f9be00c9b90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f9be00b1dd0 tx=0x7f9bd001c500 comp rx=0 tx=0).stop 2023-10-30T14:27:42.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.923+0000 7f9be5617640 1 -- 172.21.15.57:0/3989712021 shutdown_connections 2023-10-30T14:27:42.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.923+0000 7f9be5617640 1 --2- 172.21.15.57:0/3989712021 >> v2:172.21.15.88:3300/0 conn(0x7f9be00ca8e0 0x7f9be006bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:42.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.923+0000 7f9be5617640 1 --2- 172.21.15.57:0/3989712021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9be0153b10 0x7f9be00c9b90 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:42.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.923+0000 7f9be5617640 1 -- 172.21.15.57:0/3989712021 >> 172.21.15.57:0/3989712021 conn(0x7f9be006a240 msgr2=0x7f9be006a650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:42.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.923+0000 7f9be5617640 1 -- 172.21.15.57:0/3989712021 shutdown_connections 2023-10-30T14:27:42.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.923+0000 7f9be5617640 1 -- 172.21.15.57:0/3989712021 wait complete. 2023-10-30T14:27:42.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9be5617640 1 Processor -- start 2023-10-30T14:27:42.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9be5617640 1 -- start start 2023-10-30T14:27:42.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9be5617640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9be006c480 0x7f9be01551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:42.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9be5617640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9be00ca8e0 0x7f9be01549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:42.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9be5617640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9be0153b10 0x7f9be01f1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:42.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9be5617640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be01eb850 con 0x7f9be0153b10 2023-10-30T14:27:42.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9be5617640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be01f2bf0 con 0x7f9be006c480 2023-10-30T14:27:42.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9be5617640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be01ebaf0 con 0x7f9be00ca8e0 2023-10-30T14:27:42.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9bdf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9be0153b10 0x7f9be01f1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:42.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9bdf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9be0153b10 0x7f9be01f1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57958/0 (socket says 172.21.15.57:57958) 2023-10-30T14:27:42.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9bdf7fe640 1 -- 172.21.15.57:0/103636697 learned_addr learned my addr 172.21.15.57:0/103636697 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:42.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9bd7fff640 1 --2- 172.21.15.57:0/103636697 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9be00ca8e0 0x7f9be01549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:42.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9bdeffd640 1 --2- 172.21.15.57:0/103636697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9be006c480 0x7f9be01551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:42.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9bdf7fe640 1 -- 172.21.15.57:0/103636697 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9be00ca8e0 msgr2=0x7f9be01549a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:42.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9bdf7fe640 1 --2- 172.21.15.57:0/103636697 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9be00ca8e0 0x7f9be01549a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:42.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9bdf7fe640 1 -- 172.21.15.57:0/103636697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9be006c480 msgr2=0x7f9be01551f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:42.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9bdf7fe640 1 --2- 172.21.15.57:0/103636697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9be006c480 0x7f9be01551f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:42.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9bdf7fe640 1 -- 172.21.15.57:0/103636697 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd00094d0 con 0x7f9be0153b10 2023-10-30T14:27:42.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9bdf7fe640 1 --2- 172.21.15.57:0/103636697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9be0153b10 0x7f9be01f1ff0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8001570 tx=0x7f9bd800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:42.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9bd77fe640 1 -- 172.21.15.57:0/103636697 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd800a640 con 0x7f9be0153b10 2023-10-30T14:27:42.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9bd77fe640 1 -- 172.21.15.57:0/103636697 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd800a7c0 con 0x7f9be0153b10 2023-10-30T14:27:42.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9be5617640 1 -- 172.21.15.57:0/103636697 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be01ebdd0 con 0x7f9be0153b10 2023-10-30T14:27:42.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9bd77fe640 1 -- 172.21.15.57:0/103636697 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd800ab70 con 0x7f9be0153b10 2023-10-30T14:27:42.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.927+0000 7f9be5617640 1 -- 172.21.15.57:0/103636697 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be01f2ed0 con 0x7f9be0153b10 2023-10-30T14:27:42.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.931+0000 7f9be5617640 1 -- 172.21.15.57:0/103636697 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be0067540 con 0x7f9be0153b10 2023-10-30T14:27:42.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.931+0000 7f9bd77fe640 1 -- 172.21.15.57:0/103636697 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9bd800acd0 con 0x7f9be0153b10 2023-10-30T14:27:42.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.931+0000 7f9bd77fe640 1 --2- 172.21.15.57:0/103636697 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9bc00664a0 0x7f9bc0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:42.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.931+0000 7f9bd77fe640 1 -- 172.21.15.57:0/103636697 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f9bd8083940 con 0x7f9be0153b10 2023-10-30T14:27:42.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.931+0000 7f9bdeffd640 1 --2- 172.21.15.57:0/103636697 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9bc00664a0 0x7f9bc0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:42.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.931+0000 7f9bdeffd640 1 --2- 172.21.15.57:0/103636697 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9bc00664a0 0x7f9bc0068960 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00af000 tx=0x7f9bd00b6040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:42.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:42.935+0000 7f9bd77fe640 1 -- 172.21.15.57:0/103636697 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9bd8050b00 con 0x7f9be0153b10 2023-10-30T14:27:43.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:43.127+0000 7f9be5617640 1 -- 172.21.15.57:0/103636697 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9be01ec280 con 0x7f9be0153b10 2023-10-30T14:27:43.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:43.127+0000 7f9bd77fe640 1 -- 172.21.15.57:0/103636697 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6507 (secure 0 0 0) 0x7f9bd8018090 con 0x7f9be0153b10 2023-10-30T14:27:43.127 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:27:43.127 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":6027,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:27:05.900123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:27:43.128 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 80 2023-10-30T14:27:43.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:43.131+0000 7f9be5617640 1 -- 172.21.15.57:0/103636697 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9bc00664a0 msgr2=0x7f9bc0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:43.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:43.131+0000 7f9be5617640 1 --2- 172.21.15.57:0/103636697 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9bc00664a0 0x7f9bc0068960 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00af000 tx=0x7f9bd00b6040 comp rx=0 tx=0).stop 2023-10-30T14:27:43.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:43.131+0000 7f9be5617640 1 -- 172.21.15.57:0/103636697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9be0153b10 msgr2=0x7f9be01f1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:43.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:43.131+0000 7f9be5617640 1 --2- 172.21.15.57:0/103636697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9be0153b10 0x7f9be01f1ff0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8001570 tx=0x7f9bd800cca0 comp rx=0 tx=0).stop 2023-10-30T14:27:43.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:43.131+0000 7f9be5617640 1 -- 172.21.15.57:0/103636697 shutdown_connections 2023-10-30T14:27:43.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:43.131+0000 7f9be5617640 1 --2- 172.21.15.57:0/103636697 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9bc00664a0 0x7f9bc0068960 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:43.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:43.131+0000 7f9be5617640 1 --2- 172.21.15.57:0/103636697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9be0153b10 0x7f9be01f1ff0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:43.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:43.131+0000 7f9be5617640 1 --2- 172.21.15.57:0/103636697 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9be00ca8e0 0x7f9be01549a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:43.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:43.131+0000 7f9be5617640 1 --2- 172.21.15.57:0/103636697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9be006c480 0x7f9be01551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:43.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:43.131+0000 7f9be5617640 1 -- 172.21.15.57:0/103636697 >> 172.21.15.57:0/103636697 conn(0x7f9be006a240 msgr2=0x7f9be00ce3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:43.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:43.131+0000 7f9be5617640 1 -- 172.21.15.57:0/103636697 shutdown_connections 2023-10-30T14:27:43.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:43.131+0000 7f9be5617640 1 -- 172.21.15.57:0/103636697 wait complete. 2023-10-30T14:27:43.148 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:27:45.150 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:27:45.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.303+0000 7f8e1d4ea640 1 Processor -- start 2023-10-30T14:27:45.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.303+0000 7f8e1d4ea640 1 -- start start 2023-10-30T14:27:45.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.307+0000 7f8e1d4ea640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8e18153bb0 0x7f8e180c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:45.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.307+0000 7f8e1d4ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e1806c420 0x7f8e180c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:45.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.307+0000 7f8e1d4ea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e180c5f80 con 0x7f8e180c7170 2023-10-30T14:27:45.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.307+0000 7f8e1d4ea640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e180c5690 con 0x7f8e18153bb0 2023-10-30T14:27:45.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.307+0000 7f8e1d4ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1815a580 con 0x7f8e1806c420 2023-10-30T14:27:45.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.307+0000 7f8e16ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8e18153bb0 0x7f8e180c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:45.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.307+0000 7f8e16ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8e18153bb0 0x7f8e180c6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51828/0 (socket says 172.21.15.57:51828) 2023-10-30T14:27:45.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.307+0000 7f8e16ffd640 1 -- 172.21.15.57:0/2145907435 learned_addr learned my addr 172.21.15.57:0/2145907435 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:45.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.307+0000 7f8e0ffff640 1 --2- 172.21.15.57:0/2145907435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e1806c420 0x7f8e180c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:45.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.307+0000 7f8e16ffd640 1 -- 172.21.15.57:0/2145907435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e1806c420 msgr2=0x7f8e180c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:45.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.307+0000 7f8e16ffd640 1 --2- 172.21.15.57:0/2145907435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e1806c420 0x7f8e180c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:45.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.307+0000 7f8e16ffd640 1 -- 172.21.15.57:0/2145907435 >> v1:172.21.15.88:6789/0 conn(0x7f8e180c7170 legacy=0x7f8e18159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:45.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.307+0000 7f8e16ffd640 1 -- 172.21.15.57:0/2145907435 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e180c5930 con 0x7f8e18153bb0 2023-10-30T14:27:45.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e16ffd640 1 --2- 172.21.15.57:0/2145907435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e18153bb0 0x7f8e180c6410 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f8e08009870 tx=0x7f8e0801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=910c92b3dcb6e45c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:45.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e167fc640 1 -- 172.21.15.57:0/2145907435 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e080a8a30 con 0x7f8e18153bb0 2023-10-30T14:27:45.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e167fc640 1 -- 172.21.15.57:0/2145907435 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e080a8b90 con 0x7f8e18153bb0 2023-10-30T14:27:45.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e167fc640 1 -- 172.21.15.57:0/2145907435 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e080b0440 con 0x7f8e18153bb0 2023-10-30T14:27:45.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e1d4ea640 1 -- 172.21.15.57:0/2145907435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e18153bb0 msgr2=0x7f8e180c6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:45.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e1d4ea640 1 --2- 172.21.15.57:0/2145907435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e18153bb0 0x7f8e180c6410 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f8e08009870 tx=0x7f8e0801ed30 comp rx=0 tx=0).stop 2023-10-30T14:27:45.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e1d4ea640 1 -- 172.21.15.57:0/2145907435 shutdown_connections 2023-10-30T14:27:45.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e1d4ea640 1 --2- 172.21.15.57:0/2145907435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e1806c420 0x7f8e180c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:45.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e1d4ea640 1 --2- 172.21.15.57:0/2145907435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e18153bb0 0x7f8e180c6410 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:45.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e1d4ea640 1 -- 172.21.15.57:0/2145907435 >> 172.21.15.57:0/2145907435 conn(0x7f8e180cac70 msgr2=0x7f8e180cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:45.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e1d4ea640 1 -- 172.21.15.57:0/2145907435 shutdown_connections 2023-10-30T14:27:45.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e1d4ea640 1 -- 172.21.15.57:0/2145907435 wait complete. 2023-10-30T14:27:45.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e1d4ea640 1 Processor -- start 2023-10-30T14:27:45.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e1d4ea640 1 -- start start 2023-10-30T14:27:45.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e1d4ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e1806c420 0x7f8e181490a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:45.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e1d4ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e180c7170 0x7f8e1814fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:45.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e1d4ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e18153bb0 0x7f8e1814ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:45.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e1d4ea640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e180c0eb0 con 0x7f8e180c7170 2023-10-30T14:27:45.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e1d4ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1814d130 con 0x7f8e18153bb0 2023-10-30T14:27:45.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.311+0000 7f8e1d4ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1814d3d0 con 0x7f8e1806c420 2023-10-30T14:27:45.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.315+0000 7f8e0ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e180c7170 0x7f8e1814fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:45.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.315+0000 7f8e0ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e180c7170 0x7f8e1814fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51832/0 (socket says 172.21.15.57:51832) 2023-10-30T14:27:45.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.315+0000 7f8e0ffff640 1 -- 172.21.15.57:0/1148163018 learned_addr learned my addr 172.21.15.57:0/1148163018 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:45.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.315+0000 7f8e177fe640 1 --2- 172.21.15.57:0/1148163018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e18153bb0 0x7f8e1814ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:45.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.315+0000 7f8e16ffd640 1 --2- 172.21.15.57:0/1148163018 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e1806c420 0x7f8e181490a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:45.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.315+0000 7f8e0ffff640 1 -- 172.21.15.57:0/1148163018 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e1806c420 msgr2=0x7f8e181490a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:45.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.315+0000 7f8e0ffff640 1 --2- 172.21.15.57:0/1148163018 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e1806c420 0x7f8e181490a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:45.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.315+0000 7f8e0ffff640 1 -- 172.21.15.57:0/1148163018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e18153bb0 msgr2=0x7f8e1814ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:45.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.315+0000 7f8e0ffff640 1 --2- 172.21.15.57:0/1148163018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e18153bb0 0x7f8e1814ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:45.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.315+0000 7f8e0ffff640 1 -- 172.21.15.57:0/1148163018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e080094d0 con 0x7f8e180c7170 2023-10-30T14:27:45.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.315+0000 7f8e0ffff640 1 --2- 172.21.15.57:0/1148163018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e180c7170 0x7f8e1814fa10 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000d5f0 tx=0x7f8e0000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:45.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.315+0000 7f8e0f7fe640 1 -- 172.21.15.57:0/1148163018 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e000125f0 con 0x7f8e180c7170 2023-10-30T14:27:45.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.315+0000 7f8e0f7fe640 1 -- 172.21.15.57:0/1148163018 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e00012750 con 0x7f8e180c7170 2023-10-30T14:27:45.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.315+0000 7f8e0f7fe640 1 -- 172.21.15.57:0/1148163018 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00012b00 con 0x7f8e180c7170 2023-10-30T14:27:45.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.315+0000 7f8e1d4ea640 1 -- 172.21.15.57:0/1148163018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1814d670 con 0x7f8e180c7170 2023-10-30T14:27:45.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.315+0000 7f8e1d4ea640 1 -- 172.21.15.57:0/1148163018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e1814dc30 con 0x7f8e180c7170 2023-10-30T14:27:45.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.315+0000 7f8e1d4ea640 1 -- 172.21.15.57:0/1148163018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e1804e170 con 0x7f8e180c7170 2023-10-30T14:27:45.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.315+0000 7f8e0f7fe640 1 -- 172.21.15.57:0/1148163018 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8e00012c60 con 0x7f8e180c7170 2023-10-30T14:27:45.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.319+0000 7f8e0f7fe640 1 --2- 172.21.15.57:0/1148163018 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8df80664a0 0x7f8df8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:45.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.319+0000 7f8e0f7fe640 1 -- 172.21.15.57:0/1148163018 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f8e000845f0 con 0x7f8e180c7170 2023-10-30T14:27:45.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.319+0000 7f8e16ffd640 1 --2- 172.21.15.57:0/1148163018 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8df80664a0 0x7f8df8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:45.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.319+0000 7f8e16ffd640 1 --2- 172.21.15.57:0/1148163018 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8df80664a0 0x7f8df8068960 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8e080b2000 tx=0x7f8e08009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:45.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.323+0000 7f8e0f7fe640 1 -- 172.21.15.57:0/1148163018 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8e000517b0 con 0x7f8e180c7170 2023-10-30T14:27:45.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.511+0000 7f8e1d4ea640 1 -- 172.21.15.57:0/1148163018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e1806b500 con 0x7f8e180c7170 2023-10-30T14:27:45.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.511+0000 7f8e0f7fe640 1 -- 172.21.15.57:0/1148163018 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6507 (secure 0 0 0) 0x7f8e00054dd0 con 0x7f8e180c7170 2023-10-30T14:27:45.512 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:27:45.513 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":6027,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:27:05.900123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:27:45.513 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 80 2023-10-30T14:27:45.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.515+0000 7f8e1d4ea640 1 -- 172.21.15.57:0/1148163018 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8df80664a0 msgr2=0x7f8df8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:45.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.515+0000 7f8e1d4ea640 1 --2- 172.21.15.57:0/1148163018 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8df80664a0 0x7f8df8068960 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8e080b2000 tx=0x7f8e08009040 comp rx=0 tx=0).stop 2023-10-30T14:27:45.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.515+0000 7f8e1d4ea640 1 -- 172.21.15.57:0/1148163018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e180c7170 msgr2=0x7f8e1814fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:45.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.515+0000 7f8e1d4ea640 1 --2- 172.21.15.57:0/1148163018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e180c7170 0x7f8e1814fa10 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000d5f0 tx=0x7f8e0000dab0 comp rx=0 tx=0).stop 2023-10-30T14:27:45.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.515+0000 7f8e1d4ea640 1 -- 172.21.15.57:0/1148163018 shutdown_connections 2023-10-30T14:27:45.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.515+0000 7f8e1d4ea640 1 --2- 172.21.15.57:0/1148163018 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8df80664a0 0x7f8df8068960 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:45.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.515+0000 7f8e1d4ea640 1 --2- 172.21.15.57:0/1148163018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e18153bb0 0x7f8e1814ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:45.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.515+0000 7f8e1d4ea640 1 --2- 172.21.15.57:0/1148163018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e180c7170 0x7f8e1814fa10 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:45.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.515+0000 7f8e1d4ea640 1 --2- 172.21.15.57:0/1148163018 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e1806c420 0x7f8e181490a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:45.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.515+0000 7f8e1d4ea640 1 -- 172.21.15.57:0/1148163018 >> 172.21.15.57:0/1148163018 conn(0x7f8e180cac70 msgr2=0x7f8e180c9020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:45.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.515+0000 7f8e1d4ea640 1 -- 172.21.15.57:0/1148163018 shutdown_connections 2023-10-30T14:27:45.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:45.515+0000 7f8e1d4ea640 1 -- 172.21.15.57:0/1148163018 wait complete. 2023-10-30T14:27:45.533 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:27:47.535 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:27:47.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7fb772640 1 Processor -- start 2023-10-30T14:27:47.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7fb772640 1 -- start start 2023-10-30T14:27:47.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7fb772640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb7f40c4a10 0x7fb7f415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:47.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7fb772640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7f40c42a0 0x7fb7f40ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:47.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7fb772640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f4159f20 con 0x7fb7f4153c80 2023-10-30T14:27:47.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7fb772640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f415dec0 con 0x7fb7f40c4a10 2023-10-30T14:27:47.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7fb772640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f41599a0 con 0x7fb7f40c42a0 2023-10-30T14:27:47.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7f950e640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb7f4153c80 0x7fb7f415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:57364/0 (socket says 172.21.15.57:57364) 2023-10-30T14:27:47.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7f950e640 1 -- 172.21.15.57:0/2210617063 learned_addr learned my addr 172.21.15.57:0/2210617063 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:27:47.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7f8d0d640 1 --2- 172.21.15.57:0/2210617063 >> v2:172.21.15.88:3300/0 conn(0x7fb7f40c4a10 0x7fb7f415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:47.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7f9d0f640 1 --2- 172.21.15.57:0/2210617063 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7f40c42a0 0x7fb7f40ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:47.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7ebfff640 1 -- 172.21.15.57:0/2210617063 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408076357 0 0) 0x7fb7f4159f20 con 0x7fb7f4153c80 2023-10-30T14:27:47.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7ebfff640 1 -- 172.21.15.57:0/2210617063 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7e0003530 con 0x7fb7f4153c80 2023-10-30T14:27:47.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7ebfff640 1 -- 172.21.15.57:0/2210617063 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3242317003 0 0) 0x7fb7e0003530 con 0x7fb7f4153c80 2023-10-30T14:27:47.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7ebfff640 1 -- 172.21.15.57:0/2210617063 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7f40c42a0 msgr2=0x7fb7f40ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:47.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7ebfff640 1 --2- 172.21.15.57:0/2210617063 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7f40c42a0 0x7fb7f40ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:47.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7ebfff640 1 -- 172.21.15.57:0/2210617063 >> v2:172.21.15.88:3300/0 conn(0x7fb7f40c4a10 msgr2=0x7fb7f415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:47.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7ebfff640 1 --2- 172.21.15.57:0/2210617063 >> v2:172.21.15.88:3300/0 conn(0x7fb7f40c4a10 0x7fb7f415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:47.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7ebfff640 1 -- 172.21.15.57:0/2210617063 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f415d620 con 0x7fb7f4153c80 2023-10-30T14:27:47.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7ebfff640 1 -- 172.21.15.57:0/2210617063 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fb7f0001680 con 0x7fb7f4153c80 2023-10-30T14:27:47.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7ebfff640 1 -- 172.21.15.57:0/2210617063 >> v1:172.21.15.57:6789/0 conn(0x7fb7f4153c80 legacy=0x7fb7f415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:47.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7ebfff640 1 --2- 172.21.15.57:0/2210617063 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7f40c42a0 0x7fb7e0003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:47.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7ebfff640 1 --2- 172.21.15.57:0/2210617063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7e0003f00 0x7fb7e00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:47.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7ebfff640 1 --2- 172.21.15.57:0/2210617063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7e0004d10 0x7fb7e0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:47.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7ebfff640 1 -- 172.21.15.57:0/2210617063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f415dec0 con 0x7fb7e0003f00 2023-10-30T14:27:47.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7ebfff640 1 -- 172.21.15.57:0/2210617063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f41599a0 con 0x7fb7e0004d10 2023-10-30T14:27:47.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7ebfff640 1 -- 172.21.15.57:0/2210617063 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e000b9c0 con 0x7fb7f40c42a0 2023-10-30T14:27:47.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7f8d0d640 1 --2- 172.21.15.57:0/2210617063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7e0003f00 0x7fb7e00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:47.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7f950e640 1 --2- 172.21.15.57:0/2210617063 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7f40c42a0 0x7fb7e0003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:47.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.690+0000 7fb7f9d0f640 1 --2- 172.21.15.57:0/2210617063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7e0004d10 0x7fb7e0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:47.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.694+0000 7fb7f8d0d640 1 -- 172.21.15.57:0/2210617063 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7f40c42a0 msgr2=0x7fb7e0003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:47.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.694+0000 7fb7f8d0d640 1 --2- 172.21.15.57:0/2210617063 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7f40c42a0 0x7fb7e0003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:47.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.694+0000 7fb7f8d0d640 1 -- 172.21.15.57:0/2210617063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7e0004d10 msgr2=0x7fb7e0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:47.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.694+0000 7fb7f8d0d640 1 --2- 172.21.15.57:0/2210617063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7e0004d10 0x7fb7e0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:47.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.694+0000 7fb7f8d0d640 1 -- 172.21.15.57:0/2210617063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e0003530 con 0x7fb7e0003f00 2023-10-30T14:27:47.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.694+0000 7fb7f8d0d640 1 --2- 172.21.15.57:0/2210617063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7e0003f00 0x7fb7e00047d0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4009ab0 tx=0x7fb7e401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf56de96c3102cc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:47.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.694+0000 7fb7ebfff640 1 -- 172.21.15.57:0/2210617063 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e40a5660 con 0x7fb7e0003f00 2023-10-30T14:27:47.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.694+0000 7fb7ebfff640 1 -- 172.21.15.57:0/2210617063 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e40a57c0 con 0x7fb7e0003f00 2023-10-30T14:27:47.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.694+0000 7fb7ebfff640 1 -- 172.21.15.57:0/2210617063 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e40a5b70 con 0x7fb7e0003f00 2023-10-30T14:27:47.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.694+0000 7fb7fb772640 1 -- 172.21.15.57:0/2210617063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7e0003f00 msgr2=0x7fb7e00047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:47.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.694+0000 7fb7fb772640 1 --2- 172.21.15.57:0/2210617063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7e0003f00 0x7fb7e00047d0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4009ab0 tx=0x7fb7e401ca70 comp rx=0 tx=0).stop 2023-10-30T14:27:47.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.694+0000 7fb7f9d0f640 1 -- 172.21.15.57:0/2210617063 reap_dead start 2023-10-30T14:27:47.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7fb772640 1 -- 172.21.15.57:0/2210617063 shutdown_connections 2023-10-30T14:27:47.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7fb772640 1 -- 172.21.15.57:0/2210617063 >> 172.21.15.57:0/2210617063 conn(0x7fb7f40715c0 msgr2=0x7fb7f40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:47.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7fb772640 1 -- 172.21.15.57:0/2210617063 shutdown_connections 2023-10-30T14:27:47.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7fb772640 1 -- 172.21.15.57:0/2210617063 wait complete. 2023-10-30T14:27:47.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7fb772640 1 Processor -- start 2023-10-30T14:27:47.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7fb772640 1 -- start start 2023-10-30T14:27:47.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7fb772640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7e0003f00 0x7fb7f40b29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:47.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7fb772640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7f415c030 0x7fb7f415c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:47.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7fb772640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7f415b360 0x7fb7f415b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:47.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7fb772640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f4146a30 con 0x7fb7e0003f00 2023-10-30T14:27:47.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7fb772640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f40b38a0 con 0x7fb7f415c030 2023-10-30T14:27:47.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7fb772640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f414f020 con 0x7fb7f415b360 2023-10-30T14:27:47.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7f950e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7e0003f00 0x7fb7f40b29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:47.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7f950e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7e0003f00 0x7fb7f40b29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51850/0 (socket says 172.21.15.57:51850) 2023-10-30T14:27:47.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7f950e640 1 -- 172.21.15.57:0/3493402764 learned_addr learned my addr 172.21.15.57:0/3493402764 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:47.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7f9d0f640 1 --2- 172.21.15.57:0/3493402764 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7f415b360 0x7fb7f415b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:47.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7f8d0d640 1 --2- 172.21.15.57:0/3493402764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7f415c030 0x7fb7f415c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:47.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7f950e640 1 -- 172.21.15.57:0/3493402764 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7f415b360 msgr2=0x7fb7f415b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:47.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7f950e640 1 --2- 172.21.15.57:0/3493402764 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7f415b360 0x7fb7f415b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:47.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7f950e640 1 -- 172.21.15.57:0/3493402764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7f415c030 msgr2=0x7fb7f415c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:47.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7f950e640 1 --2- 172.21.15.57:0/3493402764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7f415c030 0x7fb7f415c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:47.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7f950e640 1 -- 172.21.15.57:0/3493402764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f414f300 con 0x7fb7e0003f00 2023-10-30T14:27:47.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7f8d0d640 1 --2- 172.21.15.57:0/3493402764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7f415c030 0x7fb7f415c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:27:47.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7f950e640 1 --2- 172.21.15.57:0/3493402764 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7e0003f00 0x7fb7f40b29e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0007340 tx=0x7fb7f0007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:47.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.698+0000 7fb7e9ffb640 1 -- 172.21.15.57:0/3493402764 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f000f9c0 con 0x7fb7e0003f00 2023-10-30T14:27:47.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.702+0000 7fb7e9ffb640 1 -- 172.21.15.57:0/3493402764 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f000fb20 con 0x7fb7e0003f00 2023-10-30T14:27:47.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.702+0000 7fb7fb772640 1 -- 172.21.15.57:0/3493402764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f414f5e0 con 0x7fb7e0003f00 2023-10-30T14:27:47.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.702+0000 7fb7e9ffb640 1 -- 172.21.15.57:0/3493402764 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f000feb0 con 0x7fb7e0003f00 2023-10-30T14:27:47.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.702+0000 7fb7fb772640 1 -- 172.21.15.57:0/3493402764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f415ab60 con 0x7fb7e0003f00 2023-10-30T14:27:47.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.702+0000 7fb7fb772640 1 -- 172.21.15.57:0/3493402764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f404e170 con 0x7fb7e0003f00 2023-10-30T14:27:47.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.702+0000 7fb7e9ffb640 1 -- 172.21.15.57:0/3493402764 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb7f001d460 con 0x7fb7e0003f00 2023-10-30T14:27:47.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.702+0000 7fb7e9ffb640 1 --2- 172.21.15.57:0/3493402764 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb7d80664a0 0x7fb7d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:47.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.702+0000 7fb7e9ffb640 1 -- 172.21.15.57:0/3493402764 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6427+0+0 (secure 0 0 0) 0x7fb7f0084320 con 0x7fb7e0003f00 2023-10-30T14:27:47.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.702+0000 7fb7f8d0d640 1 --2- 172.21.15.57:0/3493402764 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb7d80664a0 0x7fb7d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:47.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.706+0000 7fb7f8d0d640 1 --2- 172.21.15.57:0/3493402764 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb7d80664a0 0x7fb7d8068960 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb7e40af000 tx=0x7fb7e40046d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:47.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.706+0000 7fb7e9ffb640 1 -- 172.21.15.57:0/3493402764 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb7f0051590 con 0x7fb7e0003f00 2023-10-30T14:27:47.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.898+0000 7fb7fb772640 1 -- 172.21.15.57:0/3493402764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7f41595f0 con 0x7fb7e0003f00 2023-10-30T14:27:47.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.898+0000 7fb7e9ffb640 1 -- 172.21.15.57:0/3493402764 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6507 (secure 0 0 0) 0x7fb7f0054bb0 con 0x7fb7e0003f00 2023-10-30T14:27:47.901 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:27:47.901 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":6027,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:27:05.900123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":5586},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5586":{"gid":5586,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":51,"addr":"172.21.15.57:6837/862088802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":862088802},{"type":"v1","addr":"172.21.15.57:6837","nonce":862088802}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:27:47.901 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 80 2023-10-30T14:27:47.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.902+0000 7fb7fb772640 1 -- 172.21.15.57:0/3493402764 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb7d80664a0 msgr2=0x7fb7d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:47.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.902+0000 7fb7fb772640 1 --2- 172.21.15.57:0/3493402764 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb7d80664a0 0x7fb7d8068960 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb7e40af000 tx=0x7fb7e40046d0 comp rx=0 tx=0).stop 2023-10-30T14:27:47.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.902+0000 7fb7fb772640 1 -- 172.21.15.57:0/3493402764 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7e0003f00 msgr2=0x7fb7f40b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:47.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.902+0000 7fb7fb772640 1 --2- 172.21.15.57:0/3493402764 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7e0003f00 0x7fb7f40b29e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0007340 tx=0x7fb7f0007800 comp rx=0 tx=0).stop 2023-10-30T14:27:47.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.902+0000 7fb7fb772640 1 -- 172.21.15.57:0/3493402764 shutdown_connections 2023-10-30T14:27:47.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.902+0000 7fb7fb772640 1 --2- 172.21.15.57:0/3493402764 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb7d80664a0 0x7fb7d8068960 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:47.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.902+0000 7fb7fb772640 1 --2- 172.21.15.57:0/3493402764 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7f415b360 0x7fb7f415b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:47.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.902+0000 7fb7fb772640 1 --2- 172.21.15.57:0/3493402764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7f415c030 0x7fb7f415c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:47.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.902+0000 7fb7fb772640 1 --2- 172.21.15.57:0/3493402764 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7e0003f00 0x7fb7f40b29e0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:47.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.902+0000 7fb7fb772640 1 -- 172.21.15.57:0/3493402764 >> 172.21.15.57:0/3493402764 conn(0x7fb7f40715c0 msgr2=0x7fb7f4157670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:47.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.902+0000 7fb7fb772640 1 -- 172.21.15.57:0/3493402764 shutdown_connections 2023-10-30T14:27:47.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:47.902+0000 7fb7fb772640 1 -- 172.21.15.57:0/3493402764 wait complete. 2023-10-30T14:27:47.922 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:27:49.924 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:27:50.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.078+0000 7f168633e640 1 Processor -- start 2023-10-30T14:27:50.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.078+0000 7f168633e640 1 -- start start 2023-10-30T14:27:50.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.078+0000 7f168633e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f16800cb7d0 0x7f16800c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:50.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.078+0000 7f168633e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f16800c3e40 0x7f1680070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:50.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.078+0000 7f168633e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16800c2310 con 0x7f1680150830 2023-10-30T14:27:50.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.078+0000 7f168633e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f168006bc10 con 0x7f16800c3e40 2023-10-30T14:27:50.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.078+0000 7f168633e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1680156710 con 0x7f16800cb7d0 2023-10-30T14:27:50.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.078+0000 7f16848db640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f16800c3e40 0x7f1680070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:50.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.078+0000 7f16848db640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f16800c3e40 0x7f1680070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51852/0 (socket says 172.21.15.57:51852) 2023-10-30T14:27:50.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.078+0000 7f16848db640 1 -- 172.21.15.57:0/1806477909 learned_addr learned my addr 172.21.15.57:0/1806477909 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:50.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.078+0000 7f167f7fe640 1 --2- 172.21.15.57:0/1806477909 >> v2:172.21.15.88:3300/0 conn(0x7f16800cb7d0 0x7f16800c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:50.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.082+0000 7f16848db640 1 -- 172.21.15.57:0/1806477909 >> v2:172.21.15.88:3300/0 conn(0x7f16800cb7d0 msgr2=0x7f16800c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:50.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.082+0000 7f16848db640 1 --2- 172.21.15.57:0/1806477909 >> v2:172.21.15.88:3300/0 conn(0x7f16800cb7d0 0x7f16800c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.082+0000 7f16848db640 1 -- 172.21.15.57:0/1806477909 >> v1:172.21.15.88:6789/0 conn(0x7f1680150830 legacy=0x7f168006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:50.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.082+0000 7f16848db640 1 -- 172.21.15.57:0/1806477909 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16800c4360 con 0x7f16800c3e40 2023-10-30T14:27:50.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.086+0000 7f16848db640 1 --2- 172.21.15.57:0/1806477909 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f16800c3e40 0x7f1680070520 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f168006f9f0 tx=0x7f167801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e8f94074e6da08 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:50.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.086+0000 7f167effd640 1 -- 172.21.15.57:0/1806477909 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16780af400 con 0x7f16800c3e40 2023-10-30T14:27:50.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.086+0000 7f167effd640 1 -- 172.21.15.57:0/1806477909 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16780ab050 con 0x7f16800c3e40 2023-10-30T14:27:50.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.086+0000 7f167effd640 1 -- 172.21.15.57:0/1806477909 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16780af780 con 0x7f16800c3e40 2023-10-30T14:27:50.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.086+0000 7f168633e640 1 -- 172.21.15.57:0/1806477909 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f16800c3e40 msgr2=0x7f1680070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:50.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.086+0000 7f168633e640 1 --2- 172.21.15.57:0/1806477909 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f16800c3e40 0x7f1680070520 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f168006f9f0 tx=0x7f167801f1c0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.086+0000 7f168633e640 1 -- 172.21.15.57:0/1806477909 shutdown_connections 2023-10-30T14:27:50.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.086+0000 7f168633e640 1 --2- 172.21.15.57:0/1806477909 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f16800c3e40 0x7f1680070520 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.086+0000 7f168633e640 1 --2- 172.21.15.57:0/1806477909 >> v2:172.21.15.88:3300/0 conn(0x7f16800cb7d0 0x7f16800c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.086+0000 7f168633e640 1 -- 172.21.15.57:0/1806477909 >> 172.21.15.57:0/1806477909 conn(0x7f16800cac70 msgr2=0x7f16800cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:50.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.086+0000 7f168633e640 1 -- 172.21.15.57:0/1806477909 shutdown_connections 2023-10-30T14:27:50.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.086+0000 7f168633e640 1 -- 172.21.15.57:0/1806477909 wait complete. 2023-10-30T14:27:50.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f168633e640 1 Processor -- start 2023-10-30T14:27:50.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f168633e640 1 -- start start 2023-10-30T14:27:50.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f168633e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16800c3e40 0x7f16800c6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:50.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f168633e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16800cb7d0 0x7f16801e40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:50.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f168633e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1680150830 0x7f168014a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:50.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f168633e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168006c590 con 0x7f1680150830 2023-10-30T14:27:50.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f168633e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168015a140 con 0x7f16800cb7d0 2023-10-30T14:27:50.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f168633e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16801ef970 con 0x7f16800c3e40 2023-10-30T14:27:50.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f16848db640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1680150830 0x7f168014a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:50.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f16848db640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1680150830 0x7f168014a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51864/0 (socket says 172.21.15.57:51864) 2023-10-30T14:27:50.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f16848db640 1 -- 172.21.15.57:0/4202505834 learned_addr learned my addr 172.21.15.57:0/4202505834 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:50.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f167f7fe640 1 --2- 172.21.15.57:0/4202505834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16800cb7d0 0x7f16801e40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:50.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f167ffff640 1 --2- 172.21.15.57:0/4202505834 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16800c3e40 0x7f16800c6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:50.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f16848db640 1 -- 172.21.15.57:0/4202505834 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16800c3e40 msgr2=0x7f16800c6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:50.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f16848db640 1 --2- 172.21.15.57:0/4202505834 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16800c3e40 0x7f16800c6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f16848db640 1 -- 172.21.15.57:0/4202505834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16800cb7d0 msgr2=0x7f16801e40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:50.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f16848db640 1 --2- 172.21.15.57:0/4202505834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16800cb7d0 0x7f16801e40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f16848db640 1 -- 172.21.15.57:0/4202505834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1678009b30 con 0x7f1680150830 2023-10-30T14:27:50.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f16848db640 1 --2- 172.21.15.57:0/4202505834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1680150830 0x7f168014a4f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f167801cdf0 tx=0x7f167801bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:50.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f167cff9640 1 -- 172.21.15.57:0/4202505834 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16780afcb0 con 0x7f1680150830 2023-10-30T14:27:50.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f167cff9640 1 -- 172.21.15.57:0/4202505834 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16780afe30 con 0x7f1680150830 2023-10-30T14:27:50.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f168633e640 1 -- 172.21.15.57:0/4202505834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168015a500 con 0x7f1680150830 2023-10-30T14:27:50.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f167cff9640 1 -- 172.21.15.57:0/4202505834 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16780ba680 con 0x7f1680150830 2023-10-30T14:27:50.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.090+0000 7f168633e640 1 -- 172.21.15.57:0/4202505834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1680159dd0 con 0x7f1680150830 2023-10-30T14:27:50.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.094+0000 7f168633e640 1 -- 172.21.15.57:0/4202505834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1680159c30 con 0x7f1680150830 2023-10-30T14:27:50.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.094+0000 7f167cff9640 1 -- 172.21.15.57:0/4202505834 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f16780b6040 con 0x7f1680150830 2023-10-30T14:27:50.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.094+0000 7f167cff9640 1 --2- 172.21.15.57:0/4202505834 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f166c0664a0 0x7f166c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:50.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.094+0000 7f167cff9640 1 -- 172.21.15.57:0/4202505834 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f1678122290 con 0x7f1680150830 2023-10-30T14:27:50.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.094+0000 7f167ffff640 1 --2- 172.21.15.57:0/4202505834 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f166c0664a0 0x7f166c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:50.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.094+0000 7f167ffff640 1 --2- 172.21.15.57:0/4202505834 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f166c0664a0 0x7f166c068960 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1670007fd0 tx=0x7f1670009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:50.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.098+0000 7f167cff9640 1 -- 172.21.15.57:0/4202505834 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1680159c30 con 0x7f1680150830 2023-10-30T14:27:50.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.294+0000 7f168633e640 1 -- 172.21.15.57:0/4202505834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1680159c30 con 0x7f1680150830 2023-10-30T14:27:50.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.294+0000 7f167cff9640 1 -- 172.21.15.57:0/4202505834 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5718 (secure 0 0 0) 0x7f1680159c30 con 0x7f1680150830 2023-10-30T14:27:50.294 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:27:50.294 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:27:49.829968+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":6027},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:27:50.295 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 81 2023-10-30T14:27:50.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.294+0000 7f168633e640 1 -- 172.21.15.57:0/4202505834 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f166c0664a0 msgr2=0x7f166c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:50.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.294+0000 7f168633e640 1 --2- 172.21.15.57:0/4202505834 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f166c0664a0 0x7f166c068960 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1670007fd0 tx=0x7f1670009040 comp rx=0 tx=0).stop 2023-10-30T14:27:50.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.294+0000 7f168633e640 1 -- 172.21.15.57:0/4202505834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1680150830 msgr2=0x7f168014a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:50.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.294+0000 7f168633e640 1 --2- 172.21.15.57:0/4202505834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1680150830 0x7f168014a4f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f167801cdf0 tx=0x7f167801bf60 comp rx=0 tx=0).stop 2023-10-30T14:27:50.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.294+0000 7f168633e640 1 -- 172.21.15.57:0/4202505834 shutdown_connections 2023-10-30T14:27:50.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.294+0000 7f168633e640 1 --2- 172.21.15.57:0/4202505834 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f166c0664a0 0x7f166c068960 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.294+0000 7f168633e640 1 --2- 172.21.15.57:0/4202505834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1680150830 0x7f168014a4f0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.294+0000 7f168633e640 1 --2- 172.21.15.57:0/4202505834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16800cb7d0 0x7f16801e40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.294+0000 7f168633e640 1 --2- 172.21.15.57:0/4202505834 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16800c3e40 0x7f16800c6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.294+0000 7f168633e640 1 -- 172.21.15.57:0/4202505834 >> 172.21.15.57:0/4202505834 conn(0x7f16800cac70 msgr2=0x7f1680070f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:50.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.294+0000 7f168633e640 1 -- 172.21.15.57:0/4202505834 shutdown_connections 2023-10-30T14:27:50.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.294+0000 7f168633e640 1 -- 172.21.15.57:0/4202505834 wait complete. 2023-10-30T14:27:50.316 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T14:27:50.316 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:27:50.316 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:27:50.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.458+0000 7fa4afcea640 1 Processor -- start 2023-10-30T14:27:50.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.458+0000 7fa4afcea640 1 -- start start 2023-10-30T14:27:50.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.458+0000 7fa4afcea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4a8153c80 0x7fa4a815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:50.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.458+0000 7fa4afcea640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa4a80c4a10 0x7fa4a80c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:50.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.458+0000 7fa4afcea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa4a815a3b0 0x7fa4a80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:50.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.458+0000 7fa4afcea640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a815dec0 con 0x7fa4a80c4a10 2023-10-30T14:27:50.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.458+0000 7fa4afcea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a81599a0 con 0x7fa4a815a3b0 2023-10-30T14:27:50.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.458+0000 7fa4afcea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a8159f20 con 0x7fa4a8153c80 2023-10-30T14:27:50.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.458+0000 7fa4ad285640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa4a80c4a10 0x7fa4a80c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:50.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.458+0000 7fa4ad285640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa4a80c4a10 0x7fa4a80c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51880/0 (socket says 172.21.15.57:51880) 2023-10-30T14:27:50.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.458+0000 7fa4ad285640 1 -- 172.21.15.57:0/1674901898 learned_addr learned my addr 172.21.15.57:0/1674901898 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:50.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.458+0000 7fa4ada86640 1 --2- 172.21.15.57:0/1674901898 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4a8153c80 0x7fa4a815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:50.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.458+0000 7fa4ae287640 1 --2- 172.21.15.57:0/1674901898 >> v2:172.21.15.88:3300/0 conn(0x7fa4a815a3b0 0x7fa4a80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:50.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.458+0000 7fa4ad285640 1 -- 172.21.15.57:0/1674901898 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4a8153c80 msgr2=0x7fa4a815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:50.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.458+0000 7fa4ad285640 1 --2- 172.21.15.57:0/1674901898 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4a8153c80 0x7fa4a815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.458+0000 7fa4ad285640 1 -- 172.21.15.57:0/1674901898 >> v2:172.21.15.88:3300/0 conn(0x7fa4a815a3b0 msgr2=0x7fa4a80ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:50.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.458+0000 7fa4ad285640 1 --2- 172.21.15.57:0/1674901898 >> v2:172.21.15.88:3300/0 conn(0x7fa4a815a3b0 0x7fa4a80ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.458+0000 7fa4ad285640 1 -- 172.21.15.57:0/1674901898 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a80c47e0 con 0x7fa4a80c4a10 2023-10-30T14:27:50.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.458+0000 7fa4ae287640 1 --2- 172.21.15.57:0/1674901898 >> v2:172.21.15.88:3300/0 conn(0x7fa4a815a3b0 0x7fa4a80ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:27:50.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.462+0000 7fa4ad285640 1 --2- 172.21.15.57:0/1674901898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4a80c4a10 0x7fa4a80c42a0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa4a80cbbc0 tx=0x7fa49801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84f382270a3812 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:50.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.462+0000 7fa4aca84640 1 -- 172.21.15.57:0/1674901898 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4980a89b0 con 0x7fa4a80c4a10 2023-10-30T14:27:50.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.462+0000 7fa4aca84640 1 -- 172.21.15.57:0/1674901898 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4980a8b10 con 0x7fa4a80c4a10 2023-10-30T14:27:50.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.462+0000 7fa4aca84640 1 -- 172.21.15.57:0/1674901898 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4980a8ea0 con 0x7fa4a80c4a10 2023-10-30T14:27:50.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.462+0000 7fa4afcea640 1 -- 172.21.15.57:0/1674901898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4a80c4a10 msgr2=0x7fa4a80c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:50.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.462+0000 7fa4afcea640 1 --2- 172.21.15.57:0/1674901898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4a80c4a10 0x7fa4a80c42a0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa4a80cbbc0 tx=0x7fa49801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.462+0000 7fa4afcea640 1 -- 172.21.15.57:0/1674901898 shutdown_connections 2023-10-30T14:27:50.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.462+0000 7fa4afcea640 1 --2- 172.21.15.57:0/1674901898 >> v2:172.21.15.88:3300/0 conn(0x7fa4a815a3b0 0x7fa4a80ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.462+0000 7fa4afcea640 1 --2- 172.21.15.57:0/1674901898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4a80c4a10 0x7fa4a80c42a0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.462+0000 7fa4afcea640 1 --2- 172.21.15.57:0/1674901898 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4a8153c80 0x7fa4a815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.462+0000 7fa4afcea640 1 -- 172.21.15.57:0/1674901898 >> 172.21.15.57:0/1674901898 conn(0x7fa4a80715c0 msgr2=0x7fa4a80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:50.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.462+0000 7fa4afcea640 1 -- 172.21.15.57:0/1674901898 shutdown_connections 2023-10-30T14:27:50.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.462+0000 7fa4afcea640 1 -- 172.21.15.57:0/1674901898 wait complete. 2023-10-30T14:27:50.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4afcea640 1 Processor -- start 2023-10-30T14:27:50.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4afcea640 1 -- start start 2023-10-30T14:27:50.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4afcea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4a80c4a10 0x7fa4a80ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:50.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4afcea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4a8153c80 0x7fa4a815ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:50.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4afcea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4a815a3b0 0x7fa4a815b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:50.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4afcea640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a80cece0 con 0x7fa4a80c4a10 2023-10-30T14:27:50.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4afcea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a80b4100 con 0x7fa4a8153c80 2023-10-30T14:27:50.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4afcea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a80b3210 con 0x7fa4a815a3b0 2023-10-30T14:27:50.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4ada86640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4a80c4a10 0x7fa4a80ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:50.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4ada86640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4a80c4a10 0x7fa4a80ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51888/0 (socket says 172.21.15.57:51888) 2023-10-30T14:27:50.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4ada86640 1 -- 172.21.15.57:0/3597287472 learned_addr learned my addr 172.21.15.57:0/3597287472 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:50.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4ae287640 1 --2- 172.21.15.57:0/3597287472 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4a815a3b0 0x7fa4a815b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:50.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4ad285640 1 --2- 172.21.15.57:0/3597287472 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4a8153c80 0x7fa4a815ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:50.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4ada86640 1 -- 172.21.15.57:0/3597287472 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4a815a3b0 msgr2=0x7fa4a815b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:50.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4ada86640 1 --2- 172.21.15.57:0/3597287472 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4a815a3b0 0x7fa4a815b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4ada86640 1 -- 172.21.15.57:0/3597287472 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4a8153c80 msgr2=0x7fa4a815ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:50.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4ada86640 1 --2- 172.21.15.57:0/3597287472 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4a8153c80 0x7fa4a815ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4ada86640 1 -- 172.21.15.57:0/3597287472 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4980094d0 con 0x7fa4a80c4a10 2023-10-30T14:27:50.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4ada86640 1 --2- 172.21.15.57:0/3597287472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4a80c4a10 0x7fa4a80ce2f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400e8b0 tx=0x7fa4a400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:50.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa49e7fc640 1 -- 172.21.15.57:0/3597287472 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a400c940 con 0x7fa4a80c4a10 2023-10-30T14:27:50.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa49e7fc640 1 -- 172.21.15.57:0/3597287472 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a400caa0 con 0x7fa4a80c4a10 2023-10-30T14:27:50.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4afcea640 1 -- 172.21.15.57:0/3597287472 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a80b34f0 con 0x7fa4a80c4a10 2023-10-30T14:27:50.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa49e7fc640 1 -- 172.21.15.57:0/3597287472 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a400ce30 con 0x7fa4a80c4a10 2023-10-30T14:27:50.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.466+0000 7fa4afcea640 1 -- 172.21.15.57:0/3597287472 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a80b3ab0 con 0x7fa4a80c4a10 2023-10-30T14:27:50.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.470+0000 7fa4afcea640 1 -- 172.21.15.57:0/3597287472 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a804e170 con 0x7fa4a80c4a10 2023-10-30T14:27:50.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.470+0000 7fa49e7fc640 1 -- 172.21.15.57:0/3597287472 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa4a4013470 con 0x7fa4a80c4a10 2023-10-30T14:27:50.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.470+0000 7fa49e7fc640 1 --2- 172.21.15.57:0/3597287472 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa48c0664a0 0x7fa48c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:50.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.470+0000 7fa49e7fc640 1 -- 172.21.15.57:0/3597287472 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7fa4a4084480 con 0x7fa4a80c4a10 2023-10-30T14:27:50.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.470+0000 7fa4ad285640 1 --2- 172.21.15.57:0/3597287472 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa48c0664a0 0x7fa48c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:50.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.470+0000 7fa4ad285640 1 --2- 172.21.15.57:0/3597287472 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa48c0664a0 0x7fa48c068960 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa4980b2000 tx=0x7fa498008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:50.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.474+0000 7fa49e7fc640 1 -- 172.21.15.57:0/3597287472 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa4a40514a0 con 0x7fa4a80c4a10 2023-10-30T14:27:50.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.666+0000 7fa4afcea640 1 -- 172.21.15.57:0/3597287472 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4a81575f0 con 0x7fa4a80c4a10 2023-10-30T14:27:50.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.666+0000 7fa49e7fc640 1 -- 172.21.15.57:0/3597287472 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5718 (secure 0 0 0) 0x7fa4a4054ac0 con 0x7fa4a80c4a10 2023-10-30T14:27:50.667 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:27:50.668 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:27:49.829968+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":6027},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:27:50.668 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 81 2023-10-30T14:27:50.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.670+0000 7fa4afcea640 1 -- 172.21.15.57:0/3597287472 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa48c0664a0 msgr2=0x7fa48c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:50.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.670+0000 7fa4afcea640 1 --2- 172.21.15.57:0/3597287472 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa48c0664a0 0x7fa48c068960 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa4980b2000 tx=0x7fa498008040 comp rx=0 tx=0).stop 2023-10-30T14:27:50.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.670+0000 7fa4afcea640 1 -- 172.21.15.57:0/3597287472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4a80c4a10 msgr2=0x7fa4a80ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:50.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.670+0000 7fa4afcea640 1 --2- 172.21.15.57:0/3597287472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4a80c4a10 0x7fa4a80ce2f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400e8b0 tx=0x7fa4a400ed70 comp rx=0 tx=0).stop 2023-10-30T14:27:50.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.670+0000 7fa4afcea640 1 -- 172.21.15.57:0/3597287472 shutdown_connections 2023-10-30T14:27:50.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.670+0000 7fa4afcea640 1 --2- 172.21.15.57:0/3597287472 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa48c0664a0 0x7fa48c068960 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.670+0000 7fa4afcea640 1 --2- 172.21.15.57:0/3597287472 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4a815a3b0 0x7fa4a815b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.670+0000 7fa4afcea640 1 --2- 172.21.15.57:0/3597287472 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4a8153c80 0x7fa4a815ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.670+0000 7fa4afcea640 1 --2- 172.21.15.57:0/3597287472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4a80c4a10 0x7fa4a80ce2f0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:50.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.670+0000 7fa4afcea640 1 -- 172.21.15.57:0/3597287472 >> 172.21.15.57:0/3597287472 conn(0x7fa4a80715c0 msgr2=0x7fa4a80c9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:50.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.670+0000 7fa4afcea640 1 -- 172.21.15.57:0/3597287472 shutdown_connections 2023-10-30T14:27:50.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:50.670+0000 7fa4afcea640 1 -- 172.21.15.57:0/3597287472 wait complete. 2023-10-30T14:27:50.689 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 81, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 81, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5797": { "addr": "172.21.15.57:6834/2940934016", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2940934016, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2940934016, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5797, "incarnation": 73, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 43 }, "gid_6027": { "addr": "172.21.15.57:6836/2733323524", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2733323524, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2733323524, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6027, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:27:49.829968+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5797, "mds_1": 6027 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6839/358521496", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 358521496, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 358521496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 66, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5315, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:27:52.690 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:27:52.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.842+0000 7f2ff5f01640 1 Processor -- start 2023-10-30T14:27:52.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.842+0000 7f2ff5f01640 1 -- start start 2023-10-30T14:27:52.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2ff5f01640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ff01516c0 0x7f2ff00c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:52.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2ff5f01640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff0151aa0 con 0x7f2ff00c82d0 2023-10-30T14:27:52.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2ff5f01640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff0158af0 con 0x7f2ff00c2500 2023-10-30T14:27:52.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2ff5f01640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff00c74a0 con 0x7f2ff01516c0 2023-10-30T14:27:52.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2feeffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f2ff00c82d0 0x7f2ff00ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:57374/0 (socket says 172.21.15.57:57374) 2023-10-30T14:27:52.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2feeffd640 1 -- 172.21.15.57:0/1730352479 learned_addr learned my addr 172.21.15.57:0/1730352479 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:27:52.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fef7fe640 1 --2- 172.21.15.57:0/1730352479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ff01516c0 0x7f2ff00c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:52.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fee7fc640 1 -- 172.21.15.57:0/1730352479 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685382259 0 0) 0x7f2ff0151aa0 con 0x7f2ff00c82d0 2023-10-30T14:27:52.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fee7fc640 1 -- 172.21.15.57:0/1730352479 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd8003530 con 0x7f2ff00c82d0 2023-10-30T14:27:52.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fee7fc640 1 -- 172.21.15.57:0/1730352479 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327187382 0 0) 0x7f2ff0158af0 con 0x7f2ff00c2500 2023-10-30T14:27:52.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fee7fc640 1 -- 172.21.15.57:0/1730352479 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff0151aa0 con 0x7f2ff00c2500 2023-10-30T14:27:52.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fee7fc640 1 -- 172.21.15.57:0/1730352479 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 184416843 0 0) 0x7f2fd8003530 con 0x7f2ff00c82d0 2023-10-30T14:27:52.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fee7fc640 1 -- 172.21.15.57:0/1730352479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ff01516c0 msgr2=0x7f2ff00c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:52.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fee7fc640 1 --2- 172.21.15.57:0/1730352479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ff01516c0 0x7f2ff00c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:52.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fee7fc640 1 -- 172.21.15.57:0/1730352479 >> v1:172.21.15.88:6789/0 conn(0x7f2ff00c2500 legacy=0x7f2ff0161f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:52.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fee7fc640 1 -- 172.21.15.57:0/1730352479 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff0158510 con 0x7f2ff00c82d0 2023-10-30T14:27:52.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fee7fc640 1 -- 172.21.15.57:0/1730352479 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f2fe4001680 con 0x7f2ff00c82d0 2023-10-30T14:27:52.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fef7fe640 1 --2- 172.21.15.57:0/1730352479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ff01516c0 0x7f2ff00c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:27:52.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fee7fc640 1 -- 172.21.15.57:0/1730352479 >> v1:172.21.15.57:6789/0 conn(0x7f2ff00c82d0 legacy=0x7f2ff00ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:52.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fee7fc640 1 --2- 172.21.15.57:0/1730352479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fd8003b40 0x7f2fd8003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:52.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fee7fc640 1 --2- 172.21.15.57:0/1730352479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fd80047d0 0x7f2fd8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:52.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fee7fc640 1 --2- 172.21.15.57:0/1730352479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ff01516c0 0x7f2fd8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:52.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fee7fc640 1 -- 172.21.15.57:0/1730352479 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff00c74a0 con 0x7f2fd8003b40 2023-10-30T14:27:52.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fee7fc640 1 -- 172.21.15.57:0/1730352479 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd800b9c0 con 0x7f2fd80047d0 2023-10-30T14:27:52.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fee7fc640 1 -- 172.21.15.57:0/1730352479 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd800bb80 con 0x7f2ff01516c0 2023-10-30T14:27:52.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fef7fe640 1 --2- 172.21.15.57:0/1730352479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fd8003b40 0x7f2fd8003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:52.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2feffff640 1 --2- 172.21.15.57:0/1730352479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ff01516c0 0x7f2fd8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:52.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2feeffd640 1 --2- 172.21.15.57:0/1730352479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fd80047d0 0x7f2fd8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:52.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fef7fe640 1 -- 172.21.15.57:0/1730352479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ff01516c0 msgr2=0x7f2fd8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:52.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fef7fe640 1 --2- 172.21.15.57:0/1730352479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ff01516c0 0x7f2fd8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:52.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fef7fe640 1 -- 172.21.15.57:0/1730352479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fd80047d0 msgr2=0x7f2fd8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:52.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fef7fe640 1 --2- 172.21.15.57:0/1730352479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fd80047d0 0x7f2fd8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:52.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.846+0000 7f2fef7fe640 1 -- 172.21.15.57:0/1730352479 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe0009420 con 0x7f2fd8003b40 2023-10-30T14:27:52.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.850+0000 7f2fef7fe640 1 --2- 172.21.15.57:0/1730352479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fd8003b40 0x7f2fd8003f40 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009c80 tx=0x7f2fe001cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692d11d27727927a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:52.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.850+0000 7f2fee7fc640 1 -- 172.21.15.57:0/1730352479 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe00a5630 con 0x7f2fd8003b40 2023-10-30T14:27:52.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.850+0000 7f2fee7fc640 1 -- 172.21.15.57:0/1730352479 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe00a5790 con 0x7f2fd8003b40 2023-10-30T14:27:52.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.850+0000 7f2fee7fc640 1 -- 172.21.15.57:0/1730352479 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe00a5b40 con 0x7f2fd8003b40 2023-10-30T14:27:52.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.850+0000 7f2ff5f01640 1 -- 172.21.15.57:0/1730352479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fd8003b40 msgr2=0x7f2fd8003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:52.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.850+0000 7f2ff5f01640 1 --2- 172.21.15.57:0/1730352479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fd8003b40 0x7f2fd8003f40 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009c80 tx=0x7f2fe001cb90 comp rx=0 tx=0).stop 2023-10-30T14:27:52.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.850+0000 7f2feffff640 1 -- 172.21.15.57:0/1730352479 reap_dead start 2023-10-30T14:27:52.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.850+0000 7f2ff5f01640 1 -- 172.21.15.57:0/1730352479 shutdown_connections 2023-10-30T14:27:52.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.850+0000 7f2ff5f01640 1 -- 172.21.15.57:0/1730352479 >> 172.21.15.57:0/1730352479 conn(0x7f2ff0071fb0 msgr2=0x7f2ff00723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:52.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.850+0000 7f2ff5f01640 1 -- 172.21.15.57:0/1730352479 shutdown_connections 2023-10-30T14:27:52.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.850+0000 7f2ff5f01640 1 -- 172.21.15.57:0/1730352479 wait complete. 2023-10-30T14:27:52.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2ff5f01640 1 Processor -- start 2023-10-30T14:27:52.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2ff5f01640 1 -- start start 2023-10-30T14:27:52.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2ff5f01640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fd8003b40 0x7f2ff014ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:52.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2ff5f01640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ff006afa0 0x7f2ff01fa1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:52.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2ff5f01640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ff014fa90 0x7f2ff01fa6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:52.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2ff5f01640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff015a280 con 0x7f2fd8003b40 2023-10-30T14:27:52.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2ff5f01640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff01f9cd0 con 0x7f2ff014fa90 2023-10-30T14:27:52.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2ff5f01640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff0158960 con 0x7f2ff006afa0 2023-10-30T14:27:52.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2fef7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fd8003b40 0x7f2ff014ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:52.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2fef7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fd8003b40 0x7f2ff014ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51892/0 (socket says 172.21.15.57:51892) 2023-10-30T14:27:52.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2fef7fe640 1 -- 172.21.15.57:0/1824751749 learned_addr learned my addr 172.21.15.57:0/1824751749 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:52.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2feeffd640 1 --2- 172.21.15.57:0/1824751749 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ff006afa0 0x7f2ff01fa1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:52.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2feffff640 1 --2- 172.21.15.57:0/1824751749 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ff014fa90 0x7f2ff01fa6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:52.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2fef7fe640 1 -- 172.21.15.57:0/1824751749 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ff006afa0 msgr2=0x7f2ff01fa1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:52.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2fef7fe640 1 --2- 172.21.15.57:0/1824751749 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ff006afa0 0x7f2ff01fa1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:52.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2fef7fe640 1 -- 172.21.15.57:0/1824751749 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ff014fa90 msgr2=0x7f2ff01fa6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:52.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2fef7fe640 1 --2- 172.21.15.57:0/1824751749 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ff014fa90 0x7f2ff01fa6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:52.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2fef7fe640 1 -- 172.21.15.57:0/1824751749 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff0158c40 con 0x7f2fd8003b40 2023-10-30T14:27:52.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2fef7fe640 1 --2- 172.21.15.57:0/1824751749 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fd8003b40 0x7f2ff014ef40 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009c80 tx=0x7f2fd8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:52.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2fd7fff640 1 -- 172.21.15.57:0/1824751749 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe00be070 con 0x7f2fd8003b40 2023-10-30T14:27:52.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2fd7fff640 1 -- 172.21.15.57:0/1824751749 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe0006e40 con 0x7f2fd8003b40 2023-10-30T14:27:52.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2fd7fff640 1 -- 172.21.15.57:0/1824751749 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0004260 con 0x7f2fd8003b40 2023-10-30T14:27:52.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2ff5f01640 1 -- 172.21.15.57:0/1824751749 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff0158f20 con 0x7f2fd8003b40 2023-10-30T14:27:52.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2ff5f01640 1 -- 172.21.15.57:0/1824751749 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff014f480 con 0x7f2fd8003b40 2023-10-30T14:27:52.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.854+0000 7f2ff5f01640 1 -- 172.21.15.57:0/1824751749 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff004e170 con 0x7f2fd8003b40 2023-10-30T14:27:52.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.858+0000 7f2fd7fff640 1 -- 172.21.15.57:0/1824751749 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2fe00b4040 con 0x7f2fd8003b40 2023-10-30T14:27:52.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.858+0000 7f2fd7fff640 1 --2- 172.21.15.57:0/1824751749 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2fdc0664a0 0x7f2fdc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:52.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.858+0000 7f2fd7fff640 1 -- 172.21.15.57:0/1824751749 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f2fe011e320 con 0x7f2fd8003b40 2023-10-30T14:27:52.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.858+0000 7f2feeffd640 1 --2- 172.21.15.57:0/1824751749 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2fdc0664a0 0x7f2fdc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:52.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.858+0000 7f2feeffd640 1 --2- 172.21.15.57:0/1824751749 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2fdc0664a0 0x7f2fdc068960 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00b2510 tx=0x7f2fe4007040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:52.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:52.862+0000 7f2fd7fff640 1 -- 172.21.15.57:0/1824751749 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2fe00eb510 con 0x7f2fd8003b40 2023-10-30T14:27:53.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:53.054+0000 7f2ff5f01640 1 -- 172.21.15.57:0/1824751749 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ff0154f20 con 0x7f2fd8003b40 2023-10-30T14:27:53.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:53.054+0000 7f2fd7fff640 1 -- 172.21.15.57:0/1824751749 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5718 (secure 0 0 0) 0x7f2fe00eeb30 con 0x7f2fd8003b40 2023-10-30T14:27:53.057 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:27:53.057 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:27:49.829968+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":6027},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:27:53.057 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 81 2023-10-30T14:27:53.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:53.058+0000 7f2ff5f01640 1 -- 172.21.15.57:0/1824751749 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2fdc0664a0 msgr2=0x7f2fdc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:53.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:53.058+0000 7f2ff5f01640 1 --2- 172.21.15.57:0/1824751749 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2fdc0664a0 0x7f2fdc068960 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00b2510 tx=0x7f2fe4007040 comp rx=0 tx=0).stop 2023-10-30T14:27:53.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:53.058+0000 7f2ff5f01640 1 -- 172.21.15.57:0/1824751749 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fd8003b40 msgr2=0x7f2ff014ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:53.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:53.058+0000 7f2ff5f01640 1 --2- 172.21.15.57:0/1824751749 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fd8003b40 0x7f2ff014ef40 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009c80 tx=0x7f2fd8005800 comp rx=0 tx=0).stop 2023-10-30T14:27:53.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:53.058+0000 7f2ff5f01640 1 -- 172.21.15.57:0/1824751749 shutdown_connections 2023-10-30T14:27:53.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:53.058+0000 7f2ff5f01640 1 --2- 172.21.15.57:0/1824751749 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2fdc0664a0 0x7f2fdc068960 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:53.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:53.058+0000 7f2ff5f01640 1 --2- 172.21.15.57:0/1824751749 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ff014fa90 0x7f2ff01fa6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:53.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:53.058+0000 7f2ff5f01640 1 --2- 172.21.15.57:0/1824751749 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ff006afa0 0x7f2ff01fa1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:53.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:53.058+0000 7f2ff5f01640 1 --2- 172.21.15.57:0/1824751749 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fd8003b40 0x7f2ff014ef40 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:53.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:53.058+0000 7f2ff5f01640 1 -- 172.21.15.57:0/1824751749 >> 172.21.15.57:0/1824751749 conn(0x7f2ff0071fb0 msgr2=0x7f2ff0156f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:53.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:53.058+0000 7f2ff5f01640 1 -- 172.21.15.57:0/1824751749 shutdown_connections 2023-10-30T14:27:53.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:53.058+0000 7f2ff5f01640 1 -- 172.21.15.57:0/1824751749 wait complete. 2023-10-30T14:27:53.078 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:27:55.079 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:27:55.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.222+0000 7ffa00eea640 1 Processor -- start 2023-10-30T14:27:55.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.222+0000 7ffa00eea640 1 -- start start 2023-10-30T14:27:55.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.222+0000 7ffa00eea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff9fc151c20 0x7ff9fc0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:55.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.222+0000 7ffa00eea640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9fc0c5030 con 0x7ff9fc151840 2023-10-30T14:27:55.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.226+0000 7ffa00eea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9fc157830 con 0x7ff9fc0cb8b0 2023-10-30T14:27:55.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.226+0000 7ffa00eea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9fc157b10 con 0x7ff9fc151c20 2023-10-30T14:27:55.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.226+0000 7ff9fa59c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff9fc151840 0x7ff9fc072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44632/0 (socket says 172.21.15.57:44632) 2023-10-30T14:27:55.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.226+0000 7ff9fa59c640 1 -- 172.21.15.57:0/3980983090 learned_addr learned my addr 172.21.15.57:0/3980983090 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:27:55.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.226+0000 7ff9f9d9b640 1 --2- 172.21.15.57:0/3980983090 >> v2:172.21.15.88:3300/0 conn(0x7ff9fc151c20 0x7ff9fc0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:55.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.226+0000 7ff9f959a640 1 -- 172.21.15.57:0/3980983090 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237457017 0 0) 0x7ff9fc0c5030 con 0x7ff9fc151840 2023-10-30T14:27:55.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.226+0000 7ff9f959a640 1 -- 172.21.15.57:0/3980983090 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9e4003530 con 0x7ff9fc151840 2023-10-30T14:27:55.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.226+0000 7ff9f9d9b640 1 -- 172.21.15.57:0/3980983090 >> v1:172.21.15.88:6789/0 conn(0x7ff9fc0cb8b0 legacy=0x7ff9fc06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:55.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.226+0000 7ff9f9d9b640 1 -- 172.21.15.57:0/3980983090 >> v1:172.21.15.57:6789/0 conn(0x7ff9fc151840 legacy=0x7ff9fc072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:55.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.226+0000 7ff9f959a640 1 -- 172.21.15.57:0/3980983090 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3672853979 0 0) 0x7ff9e4003530 con 0x7ff9fc151840 2023-10-30T14:27:55.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.226+0000 7ff9f9d9b640 1 -- 172.21.15.57:0/3980983090 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9fc157df0 con 0x7ff9fc151c20 2023-10-30T14:27:55.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.230+0000 7ff9f9d9b640 1 --2- 172.21.15.57:0/3980983090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9fc151c20 0x7ff9fc0c49a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff9f00097c0 tx=0x7ff9f001eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e796e8182a671ea0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:55.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.230+0000 7ff9f959a640 1 -- 172.21.15.57:0/3980983090 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec001680 con 0x7ff9fc151c20 2023-10-30T14:27:55.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.230+0000 7ff9f959a640 1 -- 172.21.15.57:0/3980983090 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f00a8610 con 0x7ff9fc151c20 2023-10-30T14:27:55.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.230+0000 7ff9f959a640 1 -- 172.21.15.57:0/3980983090 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f00a8980 con 0x7ff9fc151c20 2023-10-30T14:27:55.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.230+0000 7ffa00eea640 1 -- 172.21.15.57:0/3980983090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9fc151c20 msgr2=0x7ff9fc0c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:55.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.230+0000 7ffa00eea640 1 --2- 172.21.15.57:0/3980983090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9fc151c20 0x7ff9fc0c49a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff9f00097c0 tx=0x7ff9f001eb90 comp rx=0 tx=0).stop 2023-10-30T14:27:55.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.230+0000 7ffa00eea640 1 -- 172.21.15.57:0/3980983090 shutdown_connections 2023-10-30T14:27:55.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.230+0000 7ffa00eea640 1 --2- 172.21.15.57:0/3980983090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9fc151c20 0x7ff9fc0c49a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:55.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.230+0000 7ffa00eea640 1 -- 172.21.15.57:0/3980983090 >> 172.21.15.57:0/3980983090 conn(0x7ff9fc0cac70 msgr2=0x7ff9fc0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:55.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.230+0000 7ffa00eea640 1 -- 172.21.15.57:0/3980983090 shutdown_connections 2023-10-30T14:27:55.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.230+0000 7ffa00eea640 1 -- 172.21.15.57:0/3980983090 wait complete. 2023-10-30T14:27:55.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.230+0000 7ffa00eea640 1 Processor -- start 2023-10-30T14:27:55.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.230+0000 7ffa00eea640 1 -- start start 2023-10-30T14:27:55.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ffa00eea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9fc0cb8b0 0x7ff9fc1f0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:55.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ffa00eea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9fc151840 0x7ff9fc1f12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:55.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ffa00eea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9fc151c20 0x7ff9fc1475d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:55.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ffa00eea640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9fc0c8280 con 0x7ff9fc151840 2023-10-30T14:27:55.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ffa00eea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9fc155d80 con 0x7ff9fc151c20 2023-10-30T14:27:55.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ffa00eea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9fc0c7cc0 con 0x7ff9fc0cb8b0 2023-10-30T14:27:55.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ff9f9d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9fc151840 0x7ff9fc1f12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:55.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ff9f9d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9fc151840 0x7ff9fc1f12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59528/0 (socket says 172.21.15.57:59528) 2023-10-30T14:27:55.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ff9f9d9b640 1 -- 172.21.15.57:0/178652826 learned_addr learned my addr 172.21.15.57:0/178652826 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:55.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ff9fad9d640 1 --2- 172.21.15.57:0/178652826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9fc151c20 0x7ff9fc1475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:55.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ff9fa59c640 1 --2- 172.21.15.57:0/178652826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9fc0cb8b0 0x7ff9fc1f0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:55.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ff9f9d9b640 1 -- 172.21.15.57:0/178652826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9fc0cb8b0 msgr2=0x7ff9fc1f0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:55.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ff9f9d9b640 1 --2- 172.21.15.57:0/178652826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9fc0cb8b0 0x7ff9fc1f0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:55.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ff9f9d9b640 1 -- 172.21.15.57:0/178652826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9fc151c20 msgr2=0x7ff9fc1475d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:55.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ff9f9d9b640 1 --2- 172.21.15.57:0/178652826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9fc151c20 0x7ff9fc1475d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:55.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ff9f9d9b640 1 -- 172.21.15.57:0/178652826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f0009420 con 0x7ff9fc151840 2023-10-30T14:27:55.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ff9fad9d640 1 --2- 172.21.15.57:0/178652826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9fc151c20 0x7ff9fc1475d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:27:55.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ff9f9d9b640 1 --2- 172.21.15.57:0/178652826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9fc151840 0x7ff9fc1f12c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff9f001b7a0 tx=0x7ff9f001b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:55.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ff9eaffd640 1 -- 172.21.15.57:0/178652826 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f00b0850 con 0x7ff9fc151840 2023-10-30T14:27:55.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ff9eaffd640 1 -- 172.21.15.57:0/178652826 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f00b09b0 con 0x7ff9fc151840 2023-10-30T14:27:55.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ff9eaffd640 1 -- 172.21.15.57:0/178652826 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f00b0d20 con 0x7ff9fc151840 2023-10-30T14:27:55.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ffa00eea640 1 -- 172.21.15.57:0/178652826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9fc0c7fa0 con 0x7ff9fc151840 2023-10-30T14:27:55.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ffa00eea640 1 -- 172.21.15.57:0/178652826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9fc155a10 con 0x7ff9fc151840 2023-10-30T14:27:55.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ffa00eea640 1 -- 172.21.15.57:0/178652826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9fc155590 con 0x7ff9fc151840 2023-10-30T14:27:55.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.234+0000 7ff9eaffd640 1 -- 172.21.15.57:0/178652826 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff9f00b7040 con 0x7ff9fc151840 2023-10-30T14:27:55.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.238+0000 7ff9eaffd640 1 --2- 172.21.15.57:0/178652826 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff9dc0664a0 0x7ff9dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:55.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.238+0000 7ff9eaffd640 1 -- 172.21.15.57:0/178652826 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7ff9f01217f0 con 0x7ff9fc151840 2023-10-30T14:27:55.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.238+0000 7ff9fa59c640 1 --2- 172.21.15.57:0/178652826 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff9dc0664a0 0x7ff9dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:55.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.238+0000 7ff9fa59c640 1 --2- 172.21.15.57:0/178652826 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff9dc0664a0 0x7ff9dc068960 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec0011f0 tx=0x7ff9ec009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:55.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.242+0000 7ff9eaffd640 1 -- 172.21.15.57:0/178652826 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff9fc155590 con 0x7ff9fc151840 2023-10-30T14:27:55.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.434+0000 7ffa00eea640 1 -- 172.21.15.57:0/178652826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9fc1f0aa0 con 0x7ff9fc151840 2023-10-30T14:27:55.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.434+0000 7ff9eaffd640 1 -- 172.21.15.57:0/178652826 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5718 (secure 0 0 0) 0x7ff9fc1f0aa0 con 0x7ff9fc151840 2023-10-30T14:27:55.435 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:27:55.435 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:27:49.829968+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":6027},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:27:55.435 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 81 2023-10-30T14:27:55.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.434+0000 7ffa00eea640 1 -- 172.21.15.57:0/178652826 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff9dc0664a0 msgr2=0x7ff9dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:55.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.434+0000 7ffa00eea640 1 --2- 172.21.15.57:0/178652826 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff9dc0664a0 0x7ff9dc068960 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec0011f0 tx=0x7ff9ec009040 comp rx=0 tx=0).stop 2023-10-30T14:27:55.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.434+0000 7ffa00eea640 1 -- 172.21.15.57:0/178652826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9fc151840 msgr2=0x7ff9fc1f12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:55.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.434+0000 7ffa00eea640 1 --2- 172.21.15.57:0/178652826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9fc151840 0x7ff9fc1f12c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff9f001b7a0 tx=0x7ff9f001b930 comp rx=0 tx=0).stop 2023-10-30T14:27:55.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.434+0000 7ffa00eea640 1 -- 172.21.15.57:0/178652826 shutdown_connections 2023-10-30T14:27:55.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.434+0000 7ffa00eea640 1 --2- 172.21.15.57:0/178652826 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff9dc0664a0 0x7ff9dc068960 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:55.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.434+0000 7ffa00eea640 1 --2- 172.21.15.57:0/178652826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9fc151c20 0x7ff9fc1475d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:55.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.434+0000 7ffa00eea640 1 --2- 172.21.15.57:0/178652826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9fc151840 0x7ff9fc1f12c0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:55.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.434+0000 7ffa00eea640 1 --2- 172.21.15.57:0/178652826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9fc0cb8b0 0x7ff9fc1f0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:55.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.434+0000 7ffa00eea640 1 -- 172.21.15.57:0/178652826 >> 172.21.15.57:0/178652826 conn(0x7ff9fc0cac70 msgr2=0x7ff9fc14fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:55.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.438+0000 7ffa00eea640 1 -- 172.21.15.57:0/178652826 shutdown_connections 2023-10-30T14:27:55.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:55.438+0000 7ffa00eea640 1 -- 172.21.15.57:0/178652826 wait complete. 2023-10-30T14:27:55.456 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:27:57.457 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:27:57.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.610+0000 7f365b33e640 1 Processor -- start 2023-10-30T14:27:57.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.610+0000 7f365b33e640 1 -- start start 2023-10-30T14:27:57.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.614+0000 7f365b33e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3654151840 0x7f36540c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:57.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.614+0000 7f365b33e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36540cb8b0 0x7f365406b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:57.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.614+0000 7f365b33e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36540c5030 con 0x7f3654151c20 2023-10-30T14:27:57.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.614+0000 7f365b33e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3654157830 con 0x7f3654151840 2023-10-30T14:27:57.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.614+0000 7f365b33e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3654157b10 con 0x7f36540cb8b0 2023-10-30T14:27:57.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.614+0000 7f36588d9640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f3654151c20 0x7f3654072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44636/0 (socket says 172.21.15.57:44636) 2023-10-30T14:27:57.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.614+0000 7f36588d9640 1 -- 172.21.15.57:0/4162577506 learned_addr learned my addr 172.21.15.57:0/4162577506 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:27:57.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.614+0000 7f36590da640 1 --2- 172.21.15.57:0/4162577506 >> v2:172.21.15.88:3300/0 conn(0x7f3654151840 0x7f36540c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:57.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.614+0000 7f36598db640 1 --2- 172.21.15.57:0/4162577506 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36540cb8b0 0x7f365406b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:57.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.614+0000 7f364bfff640 1 -- 172.21.15.57:0/4162577506 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573934373 0 0) 0x7f36540c5030 con 0x7f3654151c20 2023-10-30T14:27:57.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.614+0000 7f364bfff640 1 -- 172.21.15.57:0/4162577506 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3640003530 con 0x7f3654151c20 2023-10-30T14:27:57.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.614+0000 7f36590da640 1 -- 172.21.15.57:0/4162577506 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36540cb8b0 msgr2=0x7f365406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:57.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.614+0000 7f36590da640 1 --2- 172.21.15.57:0/4162577506 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36540cb8b0 0x7f365406b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:57.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.614+0000 7f36590da640 1 -- 172.21.15.57:0/4162577506 >> v1:172.21.15.57:6789/0 conn(0x7f3654151c20 legacy=0x7f3654072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:57.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.614+0000 7f36590da640 1 -- 172.21.15.57:0/4162577506 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3654157df0 con 0x7f3654151840 2023-10-30T14:27:57.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.618+0000 7f36590da640 1 --2- 172.21.15.57:0/4162577506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3654151840 0x7f36540c49a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f36500097c0 tx=0x7f365001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce4518eefbbee628 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:57.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.618+0000 7f364bfff640 1 -- 172.21.15.57:0/4162577506 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36500a8970 con 0x7f3654151840 2023-10-30T14:27:57.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.618+0000 7f364bfff640 1 -- 172.21.15.57:0/4162577506 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36500a8ad0 con 0x7f3654151840 2023-10-30T14:27:57.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.618+0000 7f364bfff640 1 -- 172.21.15.57:0/4162577506 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36500a8e60 con 0x7f3654151840 2023-10-30T14:27:57.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.618+0000 7f365b33e640 1 -- 172.21.15.57:0/4162577506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3654151840 msgr2=0x7f36540c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:57.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.618+0000 7f365b33e640 1 --2- 172.21.15.57:0/4162577506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3654151840 0x7f36540c49a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f36500097c0 tx=0x7f365001ec70 comp rx=0 tx=0).stop 2023-10-30T14:27:57.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.618+0000 7f365b33e640 1 -- 172.21.15.57:0/4162577506 shutdown_connections 2023-10-30T14:27:57.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.618+0000 7f365b33e640 1 --2- 172.21.15.57:0/4162577506 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36540cb8b0 0x7f365406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:57.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.618+0000 7f365b33e640 1 --2- 172.21.15.57:0/4162577506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3654151840 0x7f36540c49a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:57.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.618+0000 7f365b33e640 1 -- 172.21.15.57:0/4162577506 >> 172.21.15.57:0/4162577506 conn(0x7f36540cac70 msgr2=0x7f36540cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:57.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.618+0000 7f365b33e640 1 -- 172.21.15.57:0/4162577506 shutdown_connections 2023-10-30T14:27:57.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.618+0000 7f365b33e640 1 -- 172.21.15.57:0/4162577506 wait complete. 2023-10-30T14:27:57.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.618+0000 7f365b33e640 1 Processor -- start 2023-10-30T14:27:57.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.618+0000 7f365b33e640 1 -- start start 2023-10-30T14:27:57.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f365b33e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36540cb8b0 0x7f3654146ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:57.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f365b33e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3654151840 0x7f36540c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:57.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f365b33e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3654151c20 0x7f36540c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:57.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f365b33e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3654147020 con 0x7f3654151c20 2023-10-30T14:27:57.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f365b33e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36540ca570 con 0x7f36540cb8b0 2023-10-30T14:27:57.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f365b33e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36540ca850 con 0x7f3654151840 2023-10-30T14:27:57.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f36598db640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3654151c20 0x7f36540c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:57.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f36590da640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36540cb8b0 0x7f3654146ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:57.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f36598db640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3654151c20 0x7f36540c9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59538/0 (socket says 172.21.15.57:59538) 2023-10-30T14:27:57.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f36598db640 1 -- 172.21.15.57:0/3303012991 learned_addr learned my addr 172.21.15.57:0/3303012991 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:27:57.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f36588d9640 1 --2- 172.21.15.57:0/3303012991 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3654151840 0x7f36540c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:57.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f36598db640 1 -- 172.21.15.57:0/3303012991 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3654151840 msgr2=0x7f36540c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:57.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f36598db640 1 --2- 172.21.15.57:0/3303012991 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3654151840 0x7f36540c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:57.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f36598db640 1 -- 172.21.15.57:0/3303012991 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36540cb8b0 msgr2=0x7f3654146ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:57.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f36598db640 1 --2- 172.21.15.57:0/3303012991 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36540cb8b0 0x7f3654146ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:57.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f36598db640 1 -- 172.21.15.57:0/3303012991 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3650009420 con 0x7f3654151c20 2023-10-30T14:27:57.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f36588d9640 1 --2- 172.21.15.57:0/3303012991 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3654151840 0x7f36540c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:27:57.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f36598db640 1 --2- 172.21.15.57:0/3303012991 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3654151c20 0x7f36540c9fe0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f364c00efc0 tx=0x7f364c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:57.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f3649ffb640 1 -- 172.21.15.57:0/3303012991 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364c014480 con 0x7f3654151c20 2023-10-30T14:27:57.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f3649ffb640 1 -- 172.21.15.57:0/3303012991 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f364c00f040 con 0x7f3654151c20 2023-10-30T14:27:57.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f365b33e640 1 -- 172.21.15.57:0/3303012991 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36541e10d0 con 0x7f3654151c20 2023-10-30T14:27:57.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f3649ffb640 1 -- 172.21.15.57:0/3303012991 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364c014820 con 0x7f3654151c20 2023-10-30T14:27:57.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f365b33e640 1 -- 172.21.15.57:0/3303012991 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36541e1690 con 0x7f3654151c20 2023-10-30T14:27:57.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.622+0000 7f365b33e640 1 -- 172.21.15.57:0/3303012991 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36541e18b0 con 0x7f3654151c20 2023-10-30T14:27:57.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.626+0000 7f3649ffb640 1 -- 172.21.15.57:0/3303012991 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f364c014980 con 0x7f3654151c20 2023-10-30T14:27:57.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.626+0000 7f3649ffb640 1 --2- 172.21.15.57:0/3303012991 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f36380664a0 0x7f3638068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:57.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.626+0000 7f3649ffb640 1 -- 172.21.15.57:0/3303012991 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f364c084d30 con 0x7f3654151c20 2023-10-30T14:27:57.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.626+0000 7f36590da640 1 --2- 172.21.15.57:0/3303012991 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f36380664a0 0x7f3638068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:57.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.626+0000 7f36590da640 1 --2- 172.21.15.57:0/3303012991 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f36380664a0 0x7f3638068960 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f36500b2000 tx=0x7f3650008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:57.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.630+0000 7f3649ffb640 1 -- 172.21.15.57:0/3303012991 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f364c051de0 con 0x7f3654151c20 2023-10-30T14:27:57.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.822+0000 7f365b33e640 1 -- 172.21.15.57:0/3303012991 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f365404e170 con 0x7f3654151c20 2023-10-30T14:27:57.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.822+0000 7f3649ffb640 1 -- 172.21.15.57:0/3303012991 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5720 (secure 0 0 0) 0x7f364c018020 con 0x7f3654151c20 2023-10-30T14:27:57.825 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:27:57.825 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:27:55.707756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":6027},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:resolve","state_seq":16,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:27:57.825 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 82 2023-10-30T14:27:57.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.826+0000 7f365b33e640 1 -- 172.21.15.57:0/3303012991 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f36380664a0 msgr2=0x7f3638068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:57.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.826+0000 7f365b33e640 1 --2- 172.21.15.57:0/3303012991 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f36380664a0 0x7f3638068960 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f36500b2000 tx=0x7f3650008040 comp rx=0 tx=0).stop 2023-10-30T14:27:57.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.826+0000 7f365b33e640 1 -- 172.21.15.57:0/3303012991 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3654151c20 msgr2=0x7f36540c9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:57.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.826+0000 7f365b33e640 1 --2- 172.21.15.57:0/3303012991 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3654151c20 0x7f36540c9fe0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f364c00efc0 tx=0x7f364c00c490 comp rx=0 tx=0).stop 2023-10-30T14:27:57.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.826+0000 7f365b33e640 1 -- 172.21.15.57:0/3303012991 shutdown_connections 2023-10-30T14:27:57.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.826+0000 7f365b33e640 1 --2- 172.21.15.57:0/3303012991 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f36380664a0 0x7f3638068960 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:57.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.826+0000 7f365b33e640 1 --2- 172.21.15.57:0/3303012991 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3654151c20 0x7f36540c9fe0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:57.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.826+0000 7f365b33e640 1 --2- 172.21.15.57:0/3303012991 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3654151840 0x7f36540c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:57.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.826+0000 7f365b33e640 1 --2- 172.21.15.57:0/3303012991 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36540cb8b0 0x7f3654146ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:57.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.826+0000 7f365b33e640 1 -- 172.21.15.57:0/3303012991 >> 172.21.15.57:0/3303012991 conn(0x7f36540cac70 msgr2=0x7f3654147180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:57.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.826+0000 7f365b33e640 1 -- 172.21.15.57:0/3303012991 shutdown_connections 2023-10-30T14:27:57.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:27:57.826+0000 7f365b33e640 1 -- 172.21.15.57:0/3303012991 wait complete. 2023-10-30T14:27:57.845 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:27:59.846 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:00.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.002+0000 7f2b86fe4640 1 Processor -- start 2023-10-30T14:28:00.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.002+0000 7f2b86fe4640 1 -- start start 2023-10-30T14:28:00.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.002+0000 7f2b86fe4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2b800c8eb0 0x7f2b8015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:00.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.002+0000 7f2b86fe4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b8015ac70 0x7f2b8015a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:00.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.002+0000 7f2b86fe4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8015dcc0 con 0x7f2b8006aeb0 2023-10-30T14:28:00.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.002+0000 7f2b86fe4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b80159800 con 0x7f2b800c8eb0 2023-10-30T14:28:00.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.002+0000 7f2b86fe4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b80159d80 con 0x7f2b8015ac70 2023-10-30T14:28:00.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.002+0000 7f2b85fe2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2b800c8eb0 0x7f2b8015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:00.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.002+0000 7f2b85fe2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2b800c8eb0 0x7f2b8015cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59552/0 (socket says 172.21.15.57:59552) 2023-10-30T14:28:00.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.002+0000 7f2b85fe2640 1 -- 172.21.15.57:0/2028372470 learned_addr learned my addr 172.21.15.57:0/2028372470 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:00.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.002+0000 7f2b857e1640 1 --2- 172.21.15.57:0/2028372470 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b8015ac70 0x7f2b8015a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:00.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.002+0000 7f2b85fe2640 1 -- 172.21.15.57:0/2028372470 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b8015ac70 msgr2=0x7f2b8015a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:00.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.002+0000 7f2b85fe2640 1 --2- 172.21.15.57:0/2028372470 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b8015ac70 0x7f2b8015a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:00.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.002+0000 7f2b85fe2640 1 -- 172.21.15.57:0/2028372470 >> v1:172.21.15.88:6789/0 conn(0x7f2b8006aeb0 legacy=0x7f2b80069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:00.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.002+0000 7f2b85fe2640 1 -- 172.21.15.57:0/2028372470 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8015a750 con 0x7f2b800c8eb0 2023-10-30T14:28:00.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.006+0000 7f2b85fe2640 1 --2- 172.21.15.57:0/2028372470 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b800c8eb0 0x7f2b8015cd70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009870 tx=0x7f2b7c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d1de0e9b4da28a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:00.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.006+0000 7f2b84fe0640 1 -- 172.21.15.57:0/2028372470 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c0a8a30 con 0x7f2b800c8eb0 2023-10-30T14:28:00.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.006+0000 7f2b84fe0640 1 -- 172.21.15.57:0/2028372470 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7c0a8b90 con 0x7f2b800c8eb0 2023-10-30T14:28:00.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.006+0000 7f2b84fe0640 1 -- 172.21.15.57:0/2028372470 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c0b0440 con 0x7f2b800c8eb0 2023-10-30T14:28:00.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.006+0000 7f2b86fe4640 1 -- 172.21.15.57:0/2028372470 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b800c8eb0 msgr2=0x7f2b8015cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:00.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.006+0000 7f2b86fe4640 1 --2- 172.21.15.57:0/2028372470 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b800c8eb0 0x7f2b8015cd70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009870 tx=0x7f2b7c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:28:00.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.006+0000 7f2b86fe4640 1 -- 172.21.15.57:0/2028372470 shutdown_connections 2023-10-30T14:28:00.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.006+0000 7f2b86fe4640 1 --2- 172.21.15.57:0/2028372470 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b8015ac70 0x7f2b8015a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:00.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.006+0000 7f2b86fe4640 1 --2- 172.21.15.57:0/2028372470 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b800c8eb0 0x7f2b8015cd70 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:00.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.006+0000 7f2b86fe4640 1 -- 172.21.15.57:0/2028372470 >> 172.21.15.57:0/2028372470 conn(0x7f2b800cef00 msgr2=0x7f2b800cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:00.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.006+0000 7f2b86fe4640 1 -- 172.21.15.57:0/2028372470 shutdown_connections 2023-10-30T14:28:00.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.006+0000 7f2b86fe4640 1 -- 172.21.15.57:0/2028372470 wait complete. 2023-10-30T14:28:00.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b86fe4640 1 Processor -- start 2023-10-30T14:28:00.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b86fe4640 1 -- start start 2023-10-30T14:28:00.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b86fe4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b8006aeb0 0x7f2b8014a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:00.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b86fe4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b800c8eb0 0x7f2b8014e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:00.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b86fe4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b8015ac70 0x7f2b8014bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:00.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b86fe4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b801f3220 con 0x7f2b8006aeb0 2023-10-30T14:28:00.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b86fe4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8014b360 con 0x7f2b8015ac70 2023-10-30T14:28:00.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b86fe4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8014ac50 con 0x7f2b800c8eb0 2023-10-30T14:28:00.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b85fe2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b8006aeb0 0x7f2b8014a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:00.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b85fe2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b8006aeb0 0x7f2b8014a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59560/0 (socket says 172.21.15.57:59560) 2023-10-30T14:28:00.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b85fe2640 1 -- 172.21.15.57:0/937694688 learned_addr learned my addr 172.21.15.57:0/937694688 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:00.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b857e1640 1 --2- 172.21.15.57:0/937694688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b800c8eb0 0x7f2b8014e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:00.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b867e3640 1 --2- 172.21.15.57:0/937694688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b8015ac70 0x7f2b8014bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:00.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b85fe2640 1 -- 172.21.15.57:0/937694688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b800c8eb0 msgr2=0x7f2b8014e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:00.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b85fe2640 1 --2- 172.21.15.57:0/937694688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b800c8eb0 0x7f2b8014e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:00.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b85fe2640 1 -- 172.21.15.57:0/937694688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b8015ac70 msgr2=0x7f2b8014bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:00.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b85fe2640 1 --2- 172.21.15.57:0/937694688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b8015ac70 0x7f2b8014bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:00.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b85fe2640 1 -- 172.21.15.57:0/937694688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7c0094d0 con 0x7f2b8006aeb0 2023-10-30T14:28:00.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b85fe2640 1 --2- 172.21.15.57:0/937694688 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b8006aeb0 0x7f2b8014a710 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0b3040 tx=0x7f2b7c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:00.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b767fc640 1 -- 172.21.15.57:0/937694688 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c0b0dd0 con 0x7f2b8006aeb0 2023-10-30T14:28:00.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b767fc640 1 -- 172.21.15.57:0/937694688 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7c0bb460 con 0x7f2b8006aeb0 2023-10-30T14:28:00.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b86fe4640 1 -- 172.21.15.57:0/937694688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b800ce770 con 0x7f2b8006aeb0 2023-10-30T14:28:00.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b767fc640 1 -- 172.21.15.57:0/937694688 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c0bb810 con 0x7f2b8006aeb0 2023-10-30T14:28:00.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b86fe4640 1 -- 172.21.15.57:0/937694688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b8014b5d0 con 0x7f2b8006aeb0 2023-10-30T14:28:00.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.010+0000 7f2b86fe4640 1 -- 172.21.15.57:0/937694688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b8004e170 con 0x7f2b8006aeb0 2023-10-30T14:28:00.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.014+0000 7f2b767fc640 1 -- 172.21.15.57:0/937694688 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2b7c0b83f0 con 0x7f2b8006aeb0 2023-10-30T14:28:00.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.014+0000 7f2b767fc640 1 --2- 172.21.15.57:0/937694688 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b640664a0 0x7f2b64068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:00.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.014+0000 7f2b767fc640 1 -- 172.21.15.57:0/937694688 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f2b7c121bf0 con 0x7f2b8006aeb0 2023-10-30T14:28:00.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.014+0000 7f2b857e1640 1 --2- 172.21.15.57:0/937694688 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b640664a0 0x7f2b64068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:00.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.014+0000 7f2b857e1640 1 --2- 172.21.15.57:0/937694688 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b640664a0 0x7f2b64068960 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2b801505b0 tx=0x7f2b70009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:00.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.018+0000 7f2b767fc640 1 -- 172.21.15.57:0/937694688 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2b7c0eed50 con 0x7f2b8006aeb0 2023-10-30T14:28:00.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.206+0000 7f2b86fe4640 1 -- 172.21.15.57:0/937694688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b80157330 con 0x7f2b8006aeb0 2023-10-30T14:28:00.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.210+0000 7f2b767fc640 1 -- 172.21.15.57:0/937694688 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5720 (secure 0 0 0) 0x7f2b7c0c0090 con 0x7f2b8006aeb0 2023-10-30T14:28:00.210 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:28:00.210 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:27:55.707756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":6027},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:resolve","state_seq":16,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:00.211 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 82 2023-10-30T14:28:00.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.210+0000 7f2b86fe4640 1 -- 172.21.15.57:0/937694688 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b640664a0 msgr2=0x7f2b64068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:00.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.210+0000 7f2b86fe4640 1 --2- 172.21.15.57:0/937694688 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b640664a0 0x7f2b64068960 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2b801505b0 tx=0x7f2b70009190 comp rx=0 tx=0).stop 2023-10-30T14:28:00.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.210+0000 7f2b86fe4640 1 -- 172.21.15.57:0/937694688 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b8006aeb0 msgr2=0x7f2b8014a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:00.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.210+0000 7f2b86fe4640 1 --2- 172.21.15.57:0/937694688 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b8006aeb0 0x7f2b8014a710 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0b3040 tx=0x7f2b7c01bad0 comp rx=0 tx=0).stop 2023-10-30T14:28:00.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.210+0000 7f2b86fe4640 1 -- 172.21.15.57:0/937694688 shutdown_connections 2023-10-30T14:28:00.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.210+0000 7f2b86fe4640 1 --2- 172.21.15.57:0/937694688 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b640664a0 0x7f2b64068960 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:00.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.210+0000 7f2b86fe4640 1 --2- 172.21.15.57:0/937694688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b8015ac70 0x7f2b8014bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:00.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.210+0000 7f2b86fe4640 1 --2- 172.21.15.57:0/937694688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b800c8eb0 0x7f2b8014e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:00.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.210+0000 7f2b86fe4640 1 --2- 172.21.15.57:0/937694688 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b8006aeb0 0x7f2b8014a710 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:00.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.210+0000 7f2b86fe4640 1 -- 172.21.15.57:0/937694688 >> 172.21.15.57:0/937694688 conn(0x7f2b800cef00 msgr2=0x7f2b800c64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:00.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.210+0000 7f2b86fe4640 1 -- 172.21.15.57:0/937694688 shutdown_connections 2023-10-30T14:28:00.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:00.210+0000 7f2b86fe4640 1 -- 172.21.15.57:0/937694688 wait complete. 2023-10-30T14:28:00.230 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:28:02.232 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:02.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.378+0000 7f11c0254640 1 Processor -- start 2023-10-30T14:28:02.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.378+0000 7f11c0254640 1 -- start start 2023-10-30T14:28:02.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.382+0000 7f11c0254640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f11b806b760 0x7f11b815e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:02.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.382+0000 7f11c0254640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f11b806bb40 0x7f11b8157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:02.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.382+0000 7f11c0254640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b806c420 0x7f11b8157f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:02.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.382+0000 7f11c0254640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b8071e00 con 0x7f11b806bb40 2023-10-30T14:28:02.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.382+0000 7f11c0254640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b8070020 con 0x7f11b806b760 2023-10-30T14:28:02.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.382+0000 7f11c0254640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b815eb90 con 0x7f11b806c420 2023-10-30T14:28:02.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.382+0000 7f11bd7ef640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f11b806bb40 0x7f11b8157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:02.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.382+0000 7f11bd7ef640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f11b806bb40 0x7f11b8157a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59564/0 (socket says 172.21.15.57:59564) 2023-10-30T14:28:02.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.382+0000 7f11bd7ef640 1 -- 172.21.15.57:0/514130444 learned_addr learned my addr 172.21.15.57:0/514130444 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:02.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.382+0000 7f11bdff0640 1 --2- 172.21.15.57:0/514130444 >> v2:172.21.15.88:3300/0 conn(0x7f11b806b760 0x7f11b815e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:02.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.382+0000 7f11be7f1640 1 --2- 172.21.15.57:0/514130444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b806c420 0x7f11b8157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:02.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.382+0000 7f11bd7ef640 1 -- 172.21.15.57:0/514130444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b806c420 msgr2=0x7f11b8157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:02.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.382+0000 7f11bd7ef640 1 --2- 172.21.15.57:0/514130444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b806c420 0x7f11b8157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:02.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.382+0000 7f11bd7ef640 1 -- 172.21.15.57:0/514130444 >> v2:172.21.15.88:3300/0 conn(0x7f11b806b760 msgr2=0x7f11b815e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:02.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.382+0000 7f11bd7ef640 1 --2- 172.21.15.57:0/514130444 >> v2:172.21.15.88:3300/0 conn(0x7f11b806b760 0x7f11b815e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:02.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.382+0000 7f11bd7ef640 1 -- 172.21.15.57:0/514130444 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b8070f10 con 0x7f11b806bb40 2023-10-30T14:28:02.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.382+0000 7f11bdff0640 1 --2- 172.21.15.57:0/514130444 >> v2:172.21.15.88:3300/0 conn(0x7f11b806b760 0x7f11b815e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:28:02.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.386+0000 7f11bd7ef640 1 --2- 172.21.15.57:0/514130444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11b806bb40 0x7f11b8157a20 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f11b80cb840 tx=0x7f11a801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69b3fb80a7bb2950 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:02.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.386+0000 7f11bcfee640 1 -- 172.21.15.57:0/514130444 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a80aa610 con 0x7f11b806bb40 2023-10-30T14:28:02.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.386+0000 7f11bcfee640 1 -- 172.21.15.57:0/514130444 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a80a6040 con 0x7f11b806bb40 2023-10-30T14:28:02.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.386+0000 7f11bcfee640 1 -- 172.21.15.57:0/514130444 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a80aa990 con 0x7f11b806bb40 2023-10-30T14:28:02.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.386+0000 7f11c0254640 1 -- 172.21.15.57:0/514130444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11b806bb40 msgr2=0x7f11b8157a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:02.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.386+0000 7f11c0254640 1 --2- 172.21.15.57:0/514130444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11b806bb40 0x7f11b8157a20 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f11b80cb840 tx=0x7f11a801c4d0 comp rx=0 tx=0).stop 2023-10-30T14:28:02.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.386+0000 7f11c0254640 1 -- 172.21.15.57:0/514130444 shutdown_connections 2023-10-30T14:28:02.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.386+0000 7f11c0254640 1 --2- 172.21.15.57:0/514130444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b806c420 0x7f11b8157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:02.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.386+0000 7f11c0254640 1 --2- 172.21.15.57:0/514130444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11b806bb40 0x7f11b8157a20 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:02.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.386+0000 7f11c0254640 1 --2- 172.21.15.57:0/514130444 >> v2:172.21.15.88:3300/0 conn(0x7f11b806b760 0x7f11b815e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:02.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.386+0000 7f11c0254640 1 -- 172.21.15.57:0/514130444 >> 172.21.15.57:0/514130444 conn(0x7f11b80cac70 msgr2=0x7f11b80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:02.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.386+0000 7f11c0254640 1 -- 172.21.15.57:0/514130444 shutdown_connections 2023-10-30T14:28:02.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.386+0000 7f11c0254640 1 -- 172.21.15.57:0/514130444 wait complete. 2023-10-30T14:28:02.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.386+0000 7f11c0254640 1 Processor -- start 2023-10-30T14:28:02.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.386+0000 7f11c0254640 1 -- start start 2023-10-30T14:28:02.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11c0254640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b806b760 0x7f11b80c2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:02.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11c0254640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11b806bb40 0x7f11b814f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:02.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11c0254640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b806c420 0x7f11b814fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:02.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11c0254640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b81efcb0 con 0x7f11b806bb40 2023-10-30T14:28:02.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11c0254640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b81f9d20 con 0x7f11b806b760 2023-10-30T14:28:02.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11c0254640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b81ef310 con 0x7f11b806c420 2023-10-30T14:28:02.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11bd7ef640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11b806bb40 0x7f11b814f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:02.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11bd7ef640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11b806bb40 0x7f11b814f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59578/0 (socket says 172.21.15.57:59578) 2023-10-30T14:28:02.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11bd7ef640 1 -- 172.21.15.57:0/71490088 learned_addr learned my addr 172.21.15.57:0/71490088 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:02.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11bdff0640 1 --2- 172.21.15.57:0/71490088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b806b760 0x7f11b80c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:02.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11be7f1640 1 --2- 172.21.15.57:0/71490088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b806c420 0x7f11b814fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:02.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11bd7ef640 1 -- 172.21.15.57:0/71490088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b806c420 msgr2=0x7f11b814fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:02.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11bd7ef640 1 --2- 172.21.15.57:0/71490088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b806c420 0x7f11b814fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:02.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11bd7ef640 1 -- 172.21.15.57:0/71490088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b806b760 msgr2=0x7f11b80c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:02.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11bd7ef640 1 --2- 172.21.15.57:0/71490088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b806b760 0x7f11b80c2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:02.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11bd7ef640 1 -- 172.21.15.57:0/71490088 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a80094d0 con 0x7f11b806bb40 2023-10-30T14:28:02.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11be7f1640 1 --2- 172.21.15.57:0/71490088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b806c420 0x7f11b814fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:28:02.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11bd7ef640 1 --2- 172.21.15.57:0/71490088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11b806bb40 0x7f11b814f160 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f11a801ab80 tx=0x7f11a801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:02.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11ae7fc640 1 -- 172.21.15.57:0/71490088 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a80aad80 con 0x7f11b806bb40 2023-10-30T14:28:02.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11ae7fc640 1 -- 172.21.15.57:0/71490088 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a80bd3f0 con 0x7f11b806bb40 2023-10-30T14:28:02.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11ae7fc640 1 -- 172.21.15.57:0/71490088 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a80bd780 con 0x7f11b806bb40 2023-10-30T14:28:02.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11c0254640 1 -- 172.21.15.57:0/71490088 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b81ef5f0 con 0x7f11b806bb40 2023-10-30T14:28:02.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11c0254640 1 -- 172.21.15.57:0/71490088 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b814f6a0 con 0x7f11b806bb40 2023-10-30T14:28:02.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11c0254640 1 -- 172.21.15.57:0/71490088 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b81efdf0 con 0x7f11b806bb40 2023-10-30T14:28:02.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.390+0000 7f11ae7fc640 1 -- 172.21.15.57:0/71490088 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f11a80b33f0 con 0x7f11b806bb40 2023-10-30T14:28:02.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.394+0000 7f11ae7fc640 1 --2- 172.21.15.57:0/71490088 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f119c0664a0 0x7f119c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:02.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.394+0000 7f11ae7fc640 1 -- 172.21.15.57:0/71490088 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f11a811e290 con 0x7f11b806bb40 2023-10-30T14:28:02.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.394+0000 7f11bdff0640 1 --2- 172.21.15.57:0/71490088 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f119c0664a0 0x7f119c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:02.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.394+0000 7f11bdff0640 1 --2- 172.21.15.57:0/71490088 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f119c0664a0 0x7f119c068960 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f11b40098d0 tx=0x7f11b4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:02.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.398+0000 7f11ae7fc640 1 -- 172.21.15.57:0/71490088 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f11b81efdf0 con 0x7f11b806bb40 2023-10-30T14:28:02.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.586+0000 7f11c0254640 1 -- 172.21.15.57:0/71490088 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11b81efdf0 con 0x7f11b806bb40 2023-10-30T14:28:02.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.586+0000 7f11ae7fc640 1 -- 172.21.15.57:0/71490088 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5722 (secure 0 0 0) 0x7f11b81efdf0 con 0x7f11b806bb40 2023-10-30T14:28:02.588 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:28:02.588 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:01.002233+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":6027},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:02.588 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 83 2023-10-30T14:28:02.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.590+0000 7f11c0254640 1 -- 172.21.15.57:0/71490088 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f119c0664a0 msgr2=0x7f119c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:02.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.590+0000 7f11c0254640 1 --2- 172.21.15.57:0/71490088 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f119c0664a0 0x7f119c068960 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f11b40098d0 tx=0x7f11b4008040 comp rx=0 tx=0).stop 2023-10-30T14:28:02.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.590+0000 7f11c0254640 1 -- 172.21.15.57:0/71490088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11b806bb40 msgr2=0x7f11b814f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:02.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.590+0000 7f11c0254640 1 --2- 172.21.15.57:0/71490088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11b806bb40 0x7f11b814f160 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f11a801ab80 tx=0x7f11a801ad10 comp rx=0 tx=0).stop 2023-10-30T14:28:02.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.590+0000 7f11c0254640 1 -- 172.21.15.57:0/71490088 shutdown_connections 2023-10-30T14:28:02.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.590+0000 7f11c0254640 1 --2- 172.21.15.57:0/71490088 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f119c0664a0 0x7f119c068960 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:02.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.590+0000 7f11c0254640 1 --2- 172.21.15.57:0/71490088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b806c420 0x7f11b814fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:02.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.590+0000 7f11c0254640 1 --2- 172.21.15.57:0/71490088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11b806bb40 0x7f11b814f160 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:02.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.590+0000 7f11c0254640 1 --2- 172.21.15.57:0/71490088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b806b760 0x7f11b80c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:02.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.590+0000 7f11c0254640 1 -- 172.21.15.57:0/71490088 >> 172.21.15.57:0/71490088 conn(0x7f11b80cac70 msgr2=0x7f11b81530d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:02.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.590+0000 7f11c0254640 1 -- 172.21.15.57:0/71490088 shutdown_connections 2023-10-30T14:28:02.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:02.590+0000 7f11c0254640 1 -- 172.21.15.57:0/71490088 wait complete. 2023-10-30T14:28:02.609 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 83, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 83, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5797": { "addr": "172.21.15.57:6834/2940934016", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2940934016, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2940934016, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5797, "incarnation": 73, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 43 }, "gid_6027": { "addr": "172.21.15.57:6836/2733323524", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2733323524, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2733323524, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6027, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:reconnect", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:28:01.002233+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5797, "mds_1": 6027 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6839/358521496", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 358521496, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 358521496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 66, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5315, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:28:04.611 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:04.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.766+0000 7f98f350f640 1 Processor -- start 2023-10-30T14:28:04.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.766+0000 7f98f350f640 1 -- start start 2023-10-30T14:28:04.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.770+0000 7f98f350f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f98ec06bb40 0x7f98ec157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:04.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.770+0000 7f98f350f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98ec06c420 0x7f98ec157f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:04.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.770+0000 7f98f350f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ec071e00 con 0x7f98ec06b760 2023-10-30T14:28:04.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.770+0000 7f98f350f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98ec070020 con 0x7f98ec06bb40 2023-10-30T14:28:04.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.770+0000 7f98f350f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec070f10 con 0x7f98ec06c420 2023-10-30T14:28:04.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.770+0000 7f98f12ab640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f98ec06b760 0x7f98ec15e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44578/0 (socket says 172.21.15.57:44578) 2023-10-30T14:28:04.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.770+0000 7f98f12ab640 1 -- 172.21.15.57:0/3597008892 learned_addr learned my addr 172.21.15.57:0/3597008892 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:28:04.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.770+0000 7f98f0aaa640 1 --2- 172.21.15.57:0/3597008892 >> v2:172.21.15.57:3300/0 conn(0x7f98ec06bb40 0x7f98ec157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:04.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.770+0000 7f98f1aac640 1 --2- 172.21.15.57:0/3597008892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98ec06c420 0x7f98ec157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:04.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.770+0000 7f98e3fff640 1 -- 172.21.15.57:0/3597008892 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023746136 0 0) 0x7f98ec071e00 con 0x7f98ec06b760 2023-10-30T14:28:04.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.770+0000 7f98e3fff640 1 -- 172.21.15.57:0/3597008892 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d8003530 con 0x7f98ec06b760 2023-10-30T14:28:04.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.770+0000 7f98f0aaa640 1 -- 172.21.15.57:0/3597008892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98ec06c420 msgr2=0x7f98ec157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:04.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.770+0000 7f98f0aaa640 1 --2- 172.21.15.57:0/3597008892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98ec06c420 0x7f98ec157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:04.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.770+0000 7f98f0aaa640 1 -- 172.21.15.57:0/3597008892 >> v1:172.21.15.57:6789/0 conn(0x7f98ec06b760 legacy=0x7f98ec15e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:04.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.770+0000 7f98f0aaa640 1 -- 172.21.15.57:0/3597008892 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ec0711f0 con 0x7f98ec06bb40 2023-10-30T14:28:04.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.770+0000 7f98f1aac640 1 --2- 172.21.15.57:0/3597008892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98ec06c420 0x7f98ec157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:28:04.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.774+0000 7f98f0aaa640 1 --2- 172.21.15.57:0/3597008892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98ec06bb40 0x7f98ec157a20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f98dc0097c0 tx=0x7f98dc01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed3bd8f43359b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:04.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.774+0000 7f98e3fff640 1 -- 172.21.15.57:0/3597008892 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc0ab650 con 0x7f98ec06bb40 2023-10-30T14:28:04.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.774+0000 7f98e3fff640 1 -- 172.21.15.57:0/3597008892 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98dc0ab7b0 con 0x7f98ec06bb40 2023-10-30T14:28:04.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.774+0000 7f98e3fff640 1 -- 172.21.15.57:0/3597008892 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc0abb20 con 0x7f98ec06bb40 2023-10-30T14:28:04.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.774+0000 7f98f350f640 1 -- 172.21.15.57:0/3597008892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98ec06bb40 msgr2=0x7f98ec157a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:04.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.774+0000 7f98f350f640 1 --2- 172.21.15.57:0/3597008892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98ec06bb40 0x7f98ec157a20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f98dc0097c0 tx=0x7f98dc01c450 comp rx=0 tx=0).stop 2023-10-30T14:28:04.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.774+0000 7f98f350f640 1 -- 172.21.15.57:0/3597008892 shutdown_connections 2023-10-30T14:28:04.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.774+0000 7f98f350f640 1 --2- 172.21.15.57:0/3597008892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98ec06c420 0x7f98ec157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:04.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.774+0000 7f98f350f640 1 --2- 172.21.15.57:0/3597008892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98ec06bb40 0x7f98ec157a20 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:04.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.774+0000 7f98f350f640 1 -- 172.21.15.57:0/3597008892 >> 172.21.15.57:0/3597008892 conn(0x7f98ec0cac70 msgr2=0x7f98ec0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:04.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.774+0000 7f98f350f640 1 -- 172.21.15.57:0/3597008892 shutdown_connections 2023-10-30T14:28:04.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.774+0000 7f98f350f640 1 -- 172.21.15.57:0/3597008892 wait complete. 2023-10-30T14:28:04.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.774+0000 7f98f350f640 1 Processor -- start 2023-10-30T14:28:04.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.774+0000 7f98f350f640 1 -- start start 2023-10-30T14:28:04.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f350f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98ec06b760 0x7f98ec0c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:04.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f350f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98ec06bb40 0x7f98ec161040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:04.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f350f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98ec06c420 0x7f98ec161580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:04.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f350f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec151d50 con 0x7f98ec06c420 2023-10-30T14:28:04.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f350f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec158c40 con 0x7f98ec06b760 2023-10-30T14:28:04.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f350f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec160b60 con 0x7f98ec06bb40 2023-10-30T14:28:04.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f1aac640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98ec06c420 0x7f98ec161580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:04.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f1aac640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98ec06c420 0x7f98ec161580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45200/0 (socket says 172.21.15.57:45200) 2023-10-30T14:28:04.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f1aac640 1 -- 172.21.15.57:0/1478272605 learned_addr learned my addr 172.21.15.57:0/1478272605 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:04.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f0aaa640 1 --2- 172.21.15.57:0/1478272605 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98ec06bb40 0x7f98ec161040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:04.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f12ab640 1 --2- 172.21.15.57:0/1478272605 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98ec06b760 0x7f98ec0c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:04.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f1aac640 1 -- 172.21.15.57:0/1478272605 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98ec06bb40 msgr2=0x7f98ec161040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:04.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f1aac640 1 --2- 172.21.15.57:0/1478272605 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98ec06bb40 0x7f98ec161040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:04.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f1aac640 1 -- 172.21.15.57:0/1478272605 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98ec06b760 msgr2=0x7f98ec0c3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:04.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f1aac640 1 --2- 172.21.15.57:0/1478272605 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98ec06b760 0x7f98ec0c3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:04.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f1aac640 1 -- 172.21.15.57:0/1478272605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc009420 con 0x7f98ec06c420 2023-10-30T14:28:04.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f12ab640 1 --2- 172.21.15.57:0/1478272605 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98ec06b760 0x7f98ec0c3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:28:04.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f1aac640 1 --2- 172.21.15.57:0/1478272605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98ec06c420 0x7f98ec161580 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f98e4007e20 tx=0x7f98e400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:04.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98e1ffb640 1 -- 172.21.15.57:0/1478272605 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e4003ba0 con 0x7f98ec06c420 2023-10-30T14:28:04.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98e1ffb640 1 -- 172.21.15.57:0/1478272605 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e400f040 con 0x7f98ec06c420 2023-10-30T14:28:04.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98e1ffb640 1 -- 172.21.15.57:0/1478272605 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e400de80 con 0x7f98ec06c420 2023-10-30T14:28:04.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f350f640 1 -- 172.21.15.57:0/1478272605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98ec158e90 con 0x7f98ec06c420 2023-10-30T14:28:04.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f350f640 1 -- 172.21.15.57:0/1478272605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ec159450 con 0x7f98ec06c420 2023-10-30T14:28:04.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98f350f640 1 -- 172.21.15.57:0/1478272605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98ec160d20 con 0x7f98ec06c420 2023-10-30T14:28:04.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.778+0000 7f98e1ffb640 1 -- 172.21.15.57:0/1478272605 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f98e4004030 con 0x7f98ec06c420 2023-10-30T14:28:04.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.782+0000 7f98e1ffb640 1 --2- 172.21.15.57:0/1478272605 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f98d00664d0 0x7f98d0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:04.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.782+0000 7f98e1ffb640 1 -- 172.21.15.57:0/1478272605 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f98e4084450 con 0x7f98ec06c420 2023-10-30T14:28:04.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.782+0000 7f98f12ab640 1 --2- 172.21.15.57:0/1478272605 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f98d00664d0 0x7f98d0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:04.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.782+0000 7f98f12ab640 1 --2- 172.21.15.57:0/1478272605 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f98d00664d0 0x7f98d0068990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f98e8005890 tx=0x7f98e8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:04.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.786+0000 7f98e1ffb640 1 -- 172.21.15.57:0/1478272605 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f98e40515b0 con 0x7f98ec06c420 2023-10-30T14:28:04.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.986+0000 7f98f350f640 1 -- 172.21.15.57:0/1478272605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98ec04e170 con 0x7f98ec06c420 2023-10-30T14:28:04.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.986+0000 7f98e1ffb640 1 -- 172.21.15.57:0/1478272605 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5722 (secure 0 0 0) 0x7f98e4054bd0 con 0x7f98ec06c420 2023-10-30T14:28:04.987 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:28:04.988 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:01.002233+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":6027},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:04.988 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 83 2023-10-30T14:28:04.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.990+0000 7f98f350f640 1 -- 172.21.15.57:0/1478272605 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f98d00664d0 msgr2=0x7f98d0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:04.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.990+0000 7f98f350f640 1 --2- 172.21.15.57:0/1478272605 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f98d00664d0 0x7f98d0068990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f98e8005890 tx=0x7f98e8009040 comp rx=0 tx=0).stop 2023-10-30T14:28:04.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.990+0000 7f98f350f640 1 -- 172.21.15.57:0/1478272605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98ec06c420 msgr2=0x7f98ec161580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:04.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.990+0000 7f98f350f640 1 --2- 172.21.15.57:0/1478272605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98ec06c420 0x7f98ec161580 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f98e4007e20 tx=0x7f98e400d4a0 comp rx=0 tx=0).stop 2023-10-30T14:28:04.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.990+0000 7f98f350f640 1 -- 172.21.15.57:0/1478272605 shutdown_connections 2023-10-30T14:28:04.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.990+0000 7f98f350f640 1 --2- 172.21.15.57:0/1478272605 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f98d00664d0 0x7f98d0068990 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:04.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.990+0000 7f98f350f640 1 --2- 172.21.15.57:0/1478272605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98ec06c420 0x7f98ec161580 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:04.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.990+0000 7f98f350f640 1 --2- 172.21.15.57:0/1478272605 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98ec06bb40 0x7f98ec161040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:04.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.990+0000 7f98f350f640 1 --2- 172.21.15.57:0/1478272605 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98ec06b760 0x7f98ec0c3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:04.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.990+0000 7f98f350f640 1 -- 172.21.15.57:0/1478272605 >> 172.21.15.57:0/1478272605 conn(0x7f98ec0cac70 msgr2=0x7f98ec153e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:04.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.990+0000 7f98f350f640 1 -- 172.21.15.57:0/1478272605 shutdown_connections 2023-10-30T14:28:04.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:04.990+0000 7f98f350f640 1 -- 172.21.15.57:0/1478272605 wait complete. 2023-10-30T14:28:05.009 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:28:05.127 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:28:05.131 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:28:07.010 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:07.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.162+0000 7f8245994640 1 Processor -- start 2023-10-30T14:28:07.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.166+0000 7f8245994640 1 -- start start 2023-10-30T14:28:07.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.166+0000 7f8245994640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f82400c4a10 0x7f824015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.166+0000 7f8245994640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f82400c42a0 0x7f82400ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.166+0000 7f8245994640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8240159f20 con 0x7f8240153c80 2023-10-30T14:28:07.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.166+0000 7f8245994640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f824015dec0 con 0x7f82400c4a10 2023-10-30T14:28:07.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.166+0000 7f8245994640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82401599a0 con 0x7f82400c42a0 2023-10-30T14:28:07.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.166+0000 7f823e7fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f82400c4a10 0x7f824015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.166+0000 7f823e7fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f82400c4a10 0x7f824015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45208/0 (socket says 172.21.15.57:45208) 2023-10-30T14:28:07.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.166+0000 7f823e7fc640 1 -- 172.21.15.57:0/1913353755 learned_addr learned my addr 172.21.15.57:0/1913353755 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:07.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.166+0000 7f823f7fe640 1 --2- 172.21.15.57:0/1913353755 >> v2:172.21.15.88:3300/0 conn(0x7f82400c42a0 0x7f82400ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.166+0000 7f823e7fc640 1 -- 172.21.15.57:0/1913353755 >> v2:172.21.15.88:3300/0 conn(0x7f82400c42a0 msgr2=0x7f82400ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:07.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.166+0000 7f823e7fc640 1 --2- 172.21.15.57:0/1913353755 >> v2:172.21.15.88:3300/0 conn(0x7f82400c42a0 0x7f82400ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.166+0000 7f823e7fc640 1 -- 172.21.15.57:0/1913353755 >> v1:172.21.15.88:6789/0 conn(0x7f8240153c80 legacy=0x7f824015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:07.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.166+0000 7f823e7fc640 1 -- 172.21.15.57:0/1913353755 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824015d620 con 0x7f82400c4a10 2023-10-30T14:28:07.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.170+0000 7f823e7fc640 1 --2- 172.21.15.57:0/1913353755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82400c4a10 0x7f824015cf90 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f82400b1d40 tx=0x7f823401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0ebec3d3e43cb2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:07.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.170+0000 7f823dffb640 1 -- 172.21.15.57:0/1913353755 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82340a89f0 con 0x7f82400c4a10 2023-10-30T14:28:07.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.170+0000 7f823dffb640 1 -- 172.21.15.57:0/1913353755 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82340a8b50 con 0x7f82400c4a10 2023-10-30T14:28:07.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.170+0000 7f823dffb640 1 -- 172.21.15.57:0/1913353755 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82340b0440 con 0x7f82400c4a10 2023-10-30T14:28:07.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.170+0000 7f8245994640 1 -- 172.21.15.57:0/1913353755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82400c4a10 msgr2=0x7f824015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.170+0000 7f8245994640 1 --2- 172.21.15.57:0/1913353755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82400c4a10 0x7f824015cf90 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f82400b1d40 tx=0x7f823401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.170+0000 7f8245994640 1 -- 172.21.15.57:0/1913353755 shutdown_connections 2023-10-30T14:28:07.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.170+0000 7f8245994640 1 --2- 172.21.15.57:0/1913353755 >> v2:172.21.15.88:3300/0 conn(0x7f82400c42a0 0x7f82400ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.170+0000 7f8245994640 1 --2- 172.21.15.57:0/1913353755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82400c4a10 0x7f824015cf90 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.170+0000 7f8245994640 1 -- 172.21.15.57:0/1913353755 >> 172.21.15.57:0/1913353755 conn(0x7f82400715c0 msgr2=0x7f82400719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:07.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.170+0000 7f8245994640 1 -- 172.21.15.57:0/1913353755 shutdown_connections 2023-10-30T14:28:07.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.170+0000 7f8245994640 1 -- 172.21.15.57:0/1913353755 wait complete. 2023-10-30T14:28:07.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.170+0000 7f8245994640 1 Processor -- start 2023-10-30T14:28:07.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f8245994640 1 -- start start 2023-10-30T14:28:07.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f8245994640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82400c42a0 0x7f8240150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f8245994640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82400c4a10 0x7f82400cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f8245994640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8240153c80 0x7f8240149e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f8245994640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8240151940 con 0x7f82400c4a10 2023-10-30T14:28:07.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f8245994640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824014fd40 con 0x7f8240153c80 2023-10-30T14:28:07.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f8245994640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8240150020 con 0x7f82400c42a0 2023-10-30T14:28:07.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f823e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82400c4a10 0x7f82400cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f823e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82400c4a10 0x7f82400cd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45218/0 (socket says 172.21.15.57:45218) 2023-10-30T14:28:07.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f823e7fc640 1 -- 172.21.15.57:0/3251676367 learned_addr learned my addr 172.21.15.57:0/3251676367 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:07.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f823f7fe640 1 --2- 172.21.15.57:0/3251676367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8240153c80 0x7f8240149e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f823effd640 1 --2- 172.21.15.57:0/3251676367 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82400c42a0 0x7f8240150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f823e7fc640 1 -- 172.21.15.57:0/3251676367 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82400c42a0 msgr2=0x7f8240150ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f823e7fc640 1 --2- 172.21.15.57:0/3251676367 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82400c42a0 0x7f8240150ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f823e7fc640 1 -- 172.21.15.57:0/3251676367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8240153c80 msgr2=0x7f8240149e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f823e7fc640 1 --2- 172.21.15.57:0/3251676367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8240153c80 0x7f8240149e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f823e7fc640 1 -- 172.21.15.57:0/3251676367 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82340094d0 con 0x7f82400c4a10 2023-10-30T14:28:07.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f823e7fc640 1 --2- 172.21.15.57:0/3251676367 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82400c4a10 0x7f82400cd040 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f82340b3040 tx=0x7f823401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:07.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f8227fff640 1 -- 172.21.15.57:0/3251676367 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82340b0d50 con 0x7f82400c4a10 2023-10-30T14:28:07.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f8227fff640 1 -- 172.21.15.57:0/3251676367 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82340bb470 con 0x7f82400c4a10 2023-10-30T14:28:07.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f8227fff640 1 -- 172.21.15.57:0/3251676367 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82340bb820 con 0x7f82400c4a10 2023-10-30T14:28:07.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f8245994640 1 -- 172.21.15.57:0/3251676367 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82401f1f10 con 0x7f82400c4a10 2023-10-30T14:28:07.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f8245994640 1 -- 172.21.15.57:0/3251676367 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82401f24d0 con 0x7f82400c4a10 2023-10-30T14:28:07.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.174+0000 7f8245994640 1 -- 172.21.15.57:0/3251676367 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f824004e170 con 0x7f82400c4a10 2023-10-30T14:28:07.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.178+0000 7f8227fff640 1 -- 172.21.15.57:0/3251676367 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f82340b7070 con 0x7f82400c4a10 2023-10-30T14:28:07.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.178+0000 7f8227fff640 1 --2- 172.21.15.57:0/3251676367 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f822c0664a0 0x7f822c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.178+0000 7f8227fff640 1 -- 172.21.15.57:0/3251676367 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f82341225b0 con 0x7f82400c4a10 2023-10-30T14:28:07.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.178+0000 7f823effd640 1 --2- 172.21.15.57:0/3251676367 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f822c0664a0 0x7f822c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.178+0000 7f823effd640 1 --2- 172.21.15.57:0/3251676367 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f822c0664a0 0x7f822c068960 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8230000f70 tx=0x7f8230009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:07.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.182+0000 7f8227fff640 1 -- 172.21.15.57:0/3251676367 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f82340ef5d0 con 0x7f82400c4a10 2023-10-30T14:28:07.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.370+0000 7f8245994640 1 -- 172.21.15.57:0/3251676367 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82400cad40 con 0x7f82400c4a10 2023-10-30T14:28:07.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.370+0000 7f8227fff640 1 -- 172.21.15.57:0/3251676367 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5722 (secure 0 0 0) 0x7f82340b5090 con 0x7f82400c4a10 2023-10-30T14:28:07.372 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:28:07.372 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:01.002233+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":6027},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:07.372 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 83 2023-10-30T14:28:07.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.374+0000 7f8245994640 1 -- 172.21.15.57:0/3251676367 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f822c0664a0 msgr2=0x7f822c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.374+0000 7f8245994640 1 --2- 172.21.15.57:0/3251676367 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f822c0664a0 0x7f822c068960 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8230000f70 tx=0x7f8230009040 comp rx=0 tx=0).stop 2023-10-30T14:28:07.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.374+0000 7f8245994640 1 -- 172.21.15.57:0/3251676367 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82400c4a10 msgr2=0x7f82400cd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.374+0000 7f8245994640 1 --2- 172.21.15.57:0/3251676367 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82400c4a10 0x7f82400cd040 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f82340b3040 tx=0x7f823401ba90 comp rx=0 tx=0).stop 2023-10-30T14:28:07.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.374+0000 7f8245994640 1 -- 172.21.15.57:0/3251676367 shutdown_connections 2023-10-30T14:28:07.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.374+0000 7f8245994640 1 --2- 172.21.15.57:0/3251676367 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f822c0664a0 0x7f822c068960 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.374+0000 7f8245994640 1 --2- 172.21.15.57:0/3251676367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8240153c80 0x7f8240149e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.374+0000 7f8245994640 1 --2- 172.21.15.57:0/3251676367 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82400c4a10 0x7f82400cd040 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.374+0000 7f8245994640 1 --2- 172.21.15.57:0/3251676367 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82400c42a0 0x7f8240150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.374+0000 7f8245994640 1 -- 172.21.15.57:0/3251676367 >> 172.21.15.57:0/3251676367 conn(0x7f82400715c0 msgr2=0x7f82400c4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:07.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.374+0000 7f8245994640 1 -- 172.21.15.57:0/3251676367 shutdown_connections 2023-10-30T14:28:07.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:07.374+0000 7f8245994640 1 -- 172.21.15.57:0/3251676367 wait complete. 2023-10-30T14:28:07.393 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:28:09.394 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:09.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.550+0000 7f8e0e7b5640 1 Processor -- start 2023-10-30T14:28:09.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.550+0000 7f8e0e7b5640 1 -- start start 2023-10-30T14:28:09.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.550+0000 7f8e0e7b5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e080c8eb0 0x7f8e0815cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:09.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.550+0000 7f8e0e7b5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8e0806aeb0 0x7f8e08069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:09.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.550+0000 7f8e0e7b5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0815dcc0 con 0x7f8e0815ac70 2023-10-30T14:28:09.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.550+0000 7f8e0e7b5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e08159800 con 0x7f8e0806aeb0 2023-10-30T14:28:09.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.550+0000 7f8e0e7b5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e08159d80 con 0x7f8e080c8eb0 2023-10-30T14:28:09.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.550+0000 7f8e0d7b3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e080c8eb0 0x7f8e0815cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:09.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.550+0000 7f8e0dfb4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8e0806aeb0 0x7f8e08069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:09.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.550+0000 7f8e0d7b3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e080c8eb0 0x7f8e0815cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:56042/0 (socket says 172.21.15.57:56042) 2023-10-30T14:28:09.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.550+0000 7f8e0d7b3640 1 -- 172.21.15.57:0/945062494 learned_addr learned my addr 172.21.15.57:0/945062494 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:09.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.550+0000 7f8e0d7b3640 1 -- 172.21.15.57:0/945062494 >> v2:172.21.15.88:3300/0 conn(0x7f8e0806aeb0 msgr2=0x7f8e08069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:09.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.550+0000 7f8e0d7b3640 1 --2- 172.21.15.57:0/945062494 >> v2:172.21.15.88:3300/0 conn(0x7f8e0806aeb0 0x7f8e08069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.550+0000 7f8e0d7b3640 1 -- 172.21.15.57:0/945062494 >> v1:172.21.15.88:6789/0 conn(0x7f8e0815ac70 legacy=0x7f8e0815a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:09.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.550+0000 7f8e0d7b3640 1 -- 172.21.15.57:0/945062494 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0815d420 con 0x7f8e080c8eb0 2023-10-30T14:28:09.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.554+0000 7f8e0d7b3640 1 --2- 172.21.15.57:0/945062494 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e080c8eb0 0x7f8e0815cd70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8e04009870 tx=0x7f8e0401ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=822d35cf439ccbe8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:09.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.554+0000 7f8dfffff640 1 -- 172.21.15.57:0/945062494 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e040a8a30 con 0x7f8e080c8eb0 2023-10-30T14:28:09.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.554+0000 7f8dfffff640 1 -- 172.21.15.57:0/945062494 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e040a8b90 con 0x7f8e080c8eb0 2023-10-30T14:28:09.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.554+0000 7f8dfffff640 1 -- 172.21.15.57:0/945062494 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e040b0440 con 0x7f8e080c8eb0 2023-10-30T14:28:09.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.554+0000 7f8e0e7b5640 1 -- 172.21.15.57:0/945062494 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e080c8eb0 msgr2=0x7f8e0815cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:09.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.554+0000 7f8e0e7b5640 1 --2- 172.21.15.57:0/945062494 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e080c8eb0 0x7f8e0815cd70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8e04009870 tx=0x7f8e0401ed30 comp rx=0 tx=0).stop 2023-10-30T14:28:09.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.554+0000 7f8e0e7b5640 1 -- 172.21.15.57:0/945062494 shutdown_connections 2023-10-30T14:28:09.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.554+0000 7f8e0e7b5640 1 --2- 172.21.15.57:0/945062494 >> v2:172.21.15.88:3300/0 conn(0x7f8e0806aeb0 0x7f8e08069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.554+0000 7f8e0e7b5640 1 --2- 172.21.15.57:0/945062494 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e080c8eb0 0x7f8e0815cd70 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.554+0000 7f8e0e7b5640 1 -- 172.21.15.57:0/945062494 >> 172.21.15.57:0/945062494 conn(0x7f8e080cef00 msgr2=0x7f8e080cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:09.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.554+0000 7f8e0e7b5640 1 -- 172.21.15.57:0/945062494 shutdown_connections 2023-10-30T14:28:09.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.554+0000 7f8e0e7b5640 1 -- 172.21.15.57:0/945062494 wait complete. 2023-10-30T14:28:09.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0e7b5640 1 Processor -- start 2023-10-30T14:28:09.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0e7b5640 1 -- start start 2023-10-30T14:28:09.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0e7b5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e0806aeb0 0x7f8e080b3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:09.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0e7b5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e080c8eb0 0x7f8e08156f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:09.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0e7b5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e0815ac70 0x7f8e08157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:09.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0e7b5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080b3b40 con 0x7f8e0806aeb0 2023-10-30T14:28:09.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0e7b5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e08156ab0 con 0x7f8e080c8eb0 2023-10-30T14:28:09.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0e7b5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e081eea90 con 0x7f8e0815ac70 2023-10-30T14:28:09.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0d7b3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e0806aeb0 0x7f8e080b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:09.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0d7b3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e0806aeb0 0x7f8e080b3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45222/0 (socket says 172.21.15.57:45222) 2023-10-30T14:28:09.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0d7b3640 1 -- 172.21.15.57:0/1111765887 learned_addr learned my addr 172.21.15.57:0/1111765887 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:09.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0dfb4640 1 --2- 172.21.15.57:0/1111765887 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e0815ac70 0x7f8e08157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:09.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0cfb2640 1 --2- 172.21.15.57:0/1111765887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e080c8eb0 0x7f8e08156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:09.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0d7b3640 1 -- 172.21.15.57:0/1111765887 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e0815ac70 msgr2=0x7f8e08157480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:09.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0d7b3640 1 --2- 172.21.15.57:0/1111765887 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e0815ac70 0x7f8e08157480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0d7b3640 1 -- 172.21.15.57:0/1111765887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e080c8eb0 msgr2=0x7f8e08156f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:09.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0d7b3640 1 --2- 172.21.15.57:0/1111765887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e080c8eb0 0x7f8e08156f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0d7b3640 1 -- 172.21.15.57:0/1111765887 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e040094d0 con 0x7f8e0806aeb0 2023-10-30T14:28:09.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0dfb4640 1 --2- 172.21.15.57:0/1111765887 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e0815ac70 0x7f8e08157480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:28:09.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0d7b3640 1 --2- 172.21.15.57:0/1111765887 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e0806aeb0 0x7f8e080b3600 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f8e040b3040 tx=0x7f8e0401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:09.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8dfdffb640 1 -- 172.21.15.57:0/1111765887 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e040b0dd0 con 0x7f8e0806aeb0 2023-10-30T14:28:09.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8dfdffb640 1 -- 172.21.15.57:0/1111765887 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e040bb460 con 0x7f8e0806aeb0 2023-10-30T14:28:09.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8dfdffb640 1 -- 172.21.15.57:0/1111765887 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e040bb810 con 0x7f8e0806aeb0 2023-10-30T14:28:09.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0e7b5640 1 -- 172.21.15.57:0/1111765887 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e081eed30 con 0x7f8e0806aeb0 2023-10-30T14:28:09.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0e7b5640 1 -- 172.21.15.57:0/1111765887 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e081ef2f0 con 0x7f8e0806aeb0 2023-10-30T14:28:09.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.558+0000 7f8e0e7b5640 1 -- 172.21.15.57:0/1111765887 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e0804e170 con 0x7f8e0806aeb0 2023-10-30T14:28:09.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.562+0000 7f8dfdffb640 1 -- 172.21.15.57:0/1111765887 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8e040b83f0 con 0x7f8e0806aeb0 2023-10-30T14:28:09.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.562+0000 7f8dfdffb640 1 --2- 172.21.15.57:0/1111765887 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dec0664a0 0x7f8dec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:09.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.562+0000 7f8dfdffb640 1 -- 172.21.15.57:0/1111765887 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f8e04121b90 con 0x7f8e0806aeb0 2023-10-30T14:28:09.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.562+0000 7f8e0cfb2640 1 --2- 172.21.15.57:0/1111765887 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dec0664a0 0x7f8dec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:09.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.562+0000 7f8e0cfb2640 1 --2- 172.21.15.57:0/1111765887 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dec0664a0 0x7f8dec068960 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8e081e93d0 tx=0x7f8df8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:09.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.566+0000 7f8dfdffb640 1 -- 172.21.15.57:0/1111765887 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8e040eec40 con 0x7f8e0806aeb0 2023-10-30T14:28:09.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.754+0000 7f8e0e7b5640 1 -- 172.21.15.57:0/1111765887 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e080ca960 con 0x7f8e0806aeb0 2023-10-30T14:28:09.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.754+0000 7f8dfdffb640 1 -- 172.21.15.57:0/1111765887 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5719 (secure 0 0 0) 0x7f8e040b5090 con 0x7f8e0806aeb0 2023-10-30T14:28:09.756 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:28:09.756 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:06.996512+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":6027},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:09.757 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 84 2023-10-30T14:28:09.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.758+0000 7f8e0e7b5640 1 -- 172.21.15.57:0/1111765887 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dec0664a0 msgr2=0x7f8dec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:09.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.758+0000 7f8e0e7b5640 1 --2- 172.21.15.57:0/1111765887 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dec0664a0 0x7f8dec068960 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8e081e93d0 tx=0x7f8df8009040 comp rx=0 tx=0).stop 2023-10-30T14:28:09.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.758+0000 7f8e0e7b5640 1 -- 172.21.15.57:0/1111765887 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e0806aeb0 msgr2=0x7f8e080b3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:09.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.758+0000 7f8e0e7b5640 1 --2- 172.21.15.57:0/1111765887 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e0806aeb0 0x7f8e080b3600 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f8e040b3040 tx=0x7f8e0401bad0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.758+0000 7f8e0e7b5640 1 -- 172.21.15.57:0/1111765887 shutdown_connections 2023-10-30T14:28:09.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.758+0000 7f8e0e7b5640 1 --2- 172.21.15.57:0/1111765887 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dec0664a0 0x7f8dec068960 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.758+0000 7f8e0e7b5640 1 --2- 172.21.15.57:0/1111765887 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e0815ac70 0x7f8e08157480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.758+0000 7f8e0e7b5640 1 --2- 172.21.15.57:0/1111765887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e080c8eb0 0x7f8e08156f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.758+0000 7f8e0e7b5640 1 --2- 172.21.15.57:0/1111765887 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e0806aeb0 0x7f8e080b3600 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.758+0000 7f8e0e7b5640 1 -- 172.21.15.57:0/1111765887 >> 172.21.15.57:0/1111765887 conn(0x7f8e080cef00 msgr2=0x7f8e080cadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:09.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.758+0000 7f8e0e7b5640 1 -- 172.21.15.57:0/1111765887 shutdown_connections 2023-10-30T14:28:09.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:09.758+0000 7f8e0e7b5640 1 -- 172.21.15.57:0/1111765887 wait complete. 2023-10-30T14:28:09.777 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:28:11.778 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:11.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.934+0000 7f07bc5d3640 1 Processor -- start 2023-10-30T14:28:11.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.934+0000 7f07bc5d3640 1 -- start start 2023-10-30T14:28:11.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.934+0000 7f07bc5d3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f07b40c4a10 0x7f07b415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:11.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.934+0000 7f07bc5d3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f07b40c42a0 0x7f07b40ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:11.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.934+0000 7f07bc5d3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b4159f20 con 0x7f07b4153c80 2023-10-30T14:28:11.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.934+0000 7f07bc5d3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b415dec0 con 0x7f07b40c42a0 2023-10-30T14:28:11.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.934+0000 7f07bc5d3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b41599a0 con 0x7f07b40c4a10 2023-10-30T14:28:11.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.938+0000 7f07ba36f640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f07b4153c80 0x7f07b415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44580/0 (socket says 172.21.15.57:44580) 2023-10-30T14:28:11.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.938+0000 7f07ba36f640 1 -- 172.21.15.57:0/2076832682 learned_addr learned my addr 172.21.15.57:0/2076832682 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:28:11.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.938+0000 7f07bab70640 1 --2- 172.21.15.57:0/2076832682 >> v2:172.21.15.57:3300/0 conn(0x7f07b40c42a0 0x7f07b40ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:11.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.938+0000 7f07b9b6e640 1 --2- 172.21.15.57:0/2076832682 >> v2:172.21.15.88:3300/0 conn(0x7f07b40c4a10 0x7f07b415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:11.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.938+0000 7f07b936d640 1 -- 172.21.15.57:0/2076832682 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185217857 0 0) 0x7f07b4159f20 con 0x7f07b4153c80 2023-10-30T14:28:11.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.938+0000 7f07bab70640 1 -- 172.21.15.57:0/2076832682 >> v2:172.21.15.88:3300/0 conn(0x7f07b40c4a10 msgr2=0x7f07b415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:11.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.938+0000 7f07bab70640 1 --2- 172.21.15.57:0/2076832682 >> v2:172.21.15.88:3300/0 conn(0x7f07b40c4a10 0x7f07b415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:11.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.938+0000 7f07bab70640 1 -- 172.21.15.57:0/2076832682 >> v1:172.21.15.57:6789/0 conn(0x7f07b4153c80 legacy=0x7f07b415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:11.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.938+0000 7f07bab70640 1 -- 172.21.15.57:0/2076832682 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b415d620 con 0x7f07b40c42a0 2023-10-30T14:28:11.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.938+0000 7f07bab70640 1 --2- 172.21.15.57:0/2076832682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b40c42a0 0x7f07b40ca1e0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f07b406be50 tx=0x7f07ac01f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a73a57b931c3bf33 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:11.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.938+0000 7f07b936d640 1 -- 172.21.15.57:0/2076832682 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac0b0440 con 0x7f07b40c42a0 2023-10-30T14:28:11.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.938+0000 7f07b936d640 1 -- 172.21.15.57:0/2076832682 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07ac0b05a0 con 0x7f07b40c42a0 2023-10-30T14:28:11.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.938+0000 7f07b936d640 1 -- 172.21.15.57:0/2076832682 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac0b0930 con 0x7f07b40c42a0 2023-10-30T14:28:11.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07bc5d3640 1 -- 172.21.15.57:0/2076832682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b40c42a0 msgr2=0x7f07b40ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:11.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07bc5d3640 1 --2- 172.21.15.57:0/2076832682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b40c42a0 0x7f07b40ca1e0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f07b406be50 tx=0x7f07ac01f0f0 comp rx=0 tx=0).stop 2023-10-30T14:28:11.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07bc5d3640 1 -- 172.21.15.57:0/2076832682 shutdown_connections 2023-10-30T14:28:11.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07bc5d3640 1 --2- 172.21.15.57:0/2076832682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b40c42a0 0x7f07b40ca1e0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:11.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07bc5d3640 1 --2- 172.21.15.57:0/2076832682 >> v2:172.21.15.88:3300/0 conn(0x7f07b40c4a10 0x7f07b415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:11.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07bc5d3640 1 -- 172.21.15.57:0/2076832682 >> 172.21.15.57:0/2076832682 conn(0x7f07b40715c0 msgr2=0x7f07b40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:11.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07bc5d3640 1 -- 172.21.15.57:0/2076832682 shutdown_connections 2023-10-30T14:28:11.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07bc5d3640 1 -- 172.21.15.57:0/2076832682 wait complete. 2023-10-30T14:28:11.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07bc5d3640 1 Processor -- start 2023-10-30T14:28:11.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07bc5d3640 1 -- start start 2023-10-30T14:28:11.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07bc5d3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b40c42a0 0x7f07b4154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:11.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07bc5d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07b40c4a10 0x7f07b41506e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:11.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07bc5d3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07b4153c80 0x7f07b4150c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:11.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07bc5d3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b40ce760 con 0x7f07b40c42a0 2023-10-30T14:28:11.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07bc5d3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b4154650 con 0x7f07b4153c80 2023-10-30T14:28:11.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07bc5d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b40c9190 con 0x7f07b40c4a10 2023-10-30T14:28:11.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07ba36f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b40c42a0 0x7f07b4154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:11.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07ba36f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b40c42a0 0x7f07b4154df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45234/0 (socket says 172.21.15.57:45234) 2023-10-30T14:28:11.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07ba36f640 1 -- 172.21.15.57:0/4185233321 learned_addr learned my addr 172.21.15.57:0/4185233321 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:11.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07b9b6e640 1 --2- 172.21.15.57:0/4185233321 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07b40c4a10 0x7f07b41506e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:11.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07bab70640 1 --2- 172.21.15.57:0/4185233321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07b4153c80 0x7f07b4150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:11.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07ba36f640 1 -- 172.21.15.57:0/4185233321 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07b40c4a10 msgr2=0x7f07b41506e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:11.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07ba36f640 1 --2- 172.21.15.57:0/4185233321 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07b40c4a10 0x7f07b41506e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:11.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07ba36f640 1 -- 172.21.15.57:0/4185233321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07b4153c80 msgr2=0x7f07b4150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:11.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07ba36f640 1 --2- 172.21.15.57:0/4185233321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07b4153c80 0x7f07b4150c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:11.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.942+0000 7f07ba36f640 1 -- 172.21.15.57:0/4185233321 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac009a70 con 0x7f07b40c42a0 2023-10-30T14:28:11.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.946+0000 7f07ba36f640 1 --2- 172.21.15.57:0/4185233321 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b40c42a0 0x7f07b4154df0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f07b000c4b0 tx=0x7f07b000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:11.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.946+0000 7f07aaffd640 1 -- 172.21.15.57:0/4185233321 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b00074e0 con 0x7f07b40c42a0 2023-10-30T14:28:11.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.946+0000 7f07aaffd640 1 -- 172.21.15.57:0/4185233321 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b0007660 con 0x7f07b40c42a0 2023-10-30T14:28:11.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.946+0000 7f07aaffd640 1 -- 172.21.15.57:0/4185233321 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b0004a30 con 0x7f07b40c42a0 2023-10-30T14:28:11.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.946+0000 7f07bc5d3640 1 -- 172.21.15.57:0/4185233321 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b40c9470 con 0x7f07b40c42a0 2023-10-30T14:28:11.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.946+0000 7f07bc5d3640 1 -- 172.21.15.57:0/4185233321 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b40cec90 con 0x7f07b40c42a0 2023-10-30T14:28:11.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.946+0000 7f07bc5d3640 1 -- 172.21.15.57:0/4185233321 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b4154ac0 con 0x7f07b40c42a0 2023-10-30T14:28:11.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.946+0000 7f07aaffd640 1 -- 172.21.15.57:0/4185233321 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f07b0004b90 con 0x7f07b40c42a0 2023-10-30T14:28:11.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.946+0000 7f07aaffd640 1 --2- 172.21.15.57:0/4185233321 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07980664a0 0x7f0798068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:11.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.946+0000 7f07aaffd640 1 -- 172.21.15.57:0/4185233321 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f07b00832f0 con 0x7f07b40c42a0 2023-10-30T14:28:11.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.946+0000 7f07b9b6e640 1 --2- 172.21.15.57:0/4185233321 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07980664a0 0x7f0798068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:11.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.950+0000 7f07b9b6e640 1 --2- 172.21.15.57:0/4185233321 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07980664a0 0x7f0798068960 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f07b406cc80 tx=0x7f07a4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:11.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:11.950+0000 7f07aaffd640 1 -- 172.21.15.57:0/4185233321 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f07b4154ac0 con 0x7f07b40c42a0 2023-10-30T14:28:12.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:12.142+0000 7f07bc5d3640 1 -- 172.21.15.57:0/4185233321 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07b4154ac0 con 0x7f07b40c42a0 2023-10-30T14:28:12.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:12.142+0000 7f07aaffd640 1 -- 172.21.15.57:0/4185233321 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5719 (secure 0 0 0) 0x7f07b4154ac0 con 0x7f07b40c42a0 2023-10-30T14:28:12.143 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:28:12.143 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:06.996512+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":6027},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:12.144 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 84 2023-10-30T14:28:12.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:12.142+0000 7f07bc5d3640 1 -- 172.21.15.57:0/4185233321 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07980664a0 msgr2=0x7f0798068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:12.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:12.142+0000 7f07bc5d3640 1 --2- 172.21.15.57:0/4185233321 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07980664a0 0x7f0798068960 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f07b406cc80 tx=0x7f07a4008040 comp rx=0 tx=0).stop 2023-10-30T14:28:12.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:12.142+0000 7f07bc5d3640 1 -- 172.21.15.57:0/4185233321 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b40c42a0 msgr2=0x7f07b4154df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:12.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:12.142+0000 7f07bc5d3640 1 --2- 172.21.15.57:0/4185233321 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b40c42a0 0x7f07b4154df0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f07b000c4b0 tx=0x7f07b000c970 comp rx=0 tx=0).stop 2023-10-30T14:28:12.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:12.142+0000 7f07bc5d3640 1 -- 172.21.15.57:0/4185233321 shutdown_connections 2023-10-30T14:28:12.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:12.142+0000 7f07bc5d3640 1 --2- 172.21.15.57:0/4185233321 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07980664a0 0x7f0798068960 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:12.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:12.142+0000 7f07bc5d3640 1 --2- 172.21.15.57:0/4185233321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07b4153c80 0x7f07b4150c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:12.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:12.142+0000 7f07bc5d3640 1 --2- 172.21.15.57:0/4185233321 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07b40c4a10 0x7f07b41506e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:12.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:12.142+0000 7f07bc5d3640 1 --2- 172.21.15.57:0/4185233321 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b40c42a0 0x7f07b4154df0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:12.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:12.142+0000 7f07bc5d3640 1 -- 172.21.15.57:0/4185233321 >> 172.21.15.57:0/4185233321 conn(0x7f07b40715c0 msgr2=0x7f07b40719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:12.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:12.142+0000 7f07bc5d3640 1 -- 172.21.15.57:0/4185233321 shutdown_connections 2023-10-30T14:28:12.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:12.146+0000 7f07bc5d3640 1 -- 172.21.15.57:0/4185233321 wait complete. 2023-10-30T14:28:12.164 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:28:14.165 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:14.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.318+0000 7fe8d2e9c640 1 Processor -- start 2023-10-30T14:28:14.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.318+0000 7fe8d2e9c640 1 -- start start 2023-10-30T14:28:14.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.322+0000 7fe8d2e9c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe8cc153c80 0x7fe8cc15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:14.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.322+0000 7fe8d2e9c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe8cc0c42a0 0x7fe8cc0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:14.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.322+0000 7fe8d2e9c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8cc159f20 con 0x7fe8cc0c4a10 2023-10-30T14:28:14.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.322+0000 7fe8d2e9c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8cc15dec0 con 0x7fe8cc0c42a0 2023-10-30T14:28:14.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.322+0000 7fe8d2e9c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8cc1599a0 con 0x7fe8cc153c80 2023-10-30T14:28:14.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.322+0000 7fe8d1439640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe8cc0c42a0 0x7fe8cc0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:14.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.322+0000 7fe8d1439640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe8cc0c42a0 0x7fe8cc0ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40752/0 (socket says 172.21.15.57:40752) 2023-10-30T14:28:14.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.322+0000 7fe8d1439640 1 -- 172.21.15.57:0/968712593 learned_addr learned my addr 172.21.15.57:0/968712593 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:14.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.322+0000 7fe8d0c38640 1 --2- 172.21.15.57:0/968712593 >> v2:172.21.15.88:3300/0 conn(0x7fe8cc153c80 0x7fe8cc15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:14.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.322+0000 7fe8d1439640 1 -- 172.21.15.57:0/968712593 >> v2:172.21.15.88:3300/0 conn(0x7fe8cc153c80 msgr2=0x7fe8cc15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:14.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.322+0000 7fe8d1439640 1 --2- 172.21.15.57:0/968712593 >> v2:172.21.15.88:3300/0 conn(0x7fe8cc153c80 0x7fe8cc15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:14.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.322+0000 7fe8d1439640 1 -- 172.21.15.57:0/968712593 >> v1:172.21.15.88:6789/0 conn(0x7fe8cc0c4a10 legacy=0x7fe8cc15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:14.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.322+0000 7fe8d1439640 1 -- 172.21.15.57:0/968712593 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8cc15d620 con 0x7fe8cc0c42a0 2023-10-30T14:28:14.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.322+0000 7fe8d0c38640 1 --2- 172.21.15.57:0/968712593 >> v2:172.21.15.88:3300/0 conn(0x7fe8cc153c80 0x7fe8cc15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:28:14.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.322+0000 7fe8d1439640 1 --2- 172.21.15.57:0/968712593 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8cc0c42a0 0x7fe8cc0ca1e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc0d00f0 tx=0x7fe8c401f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a880c4c099f314e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:14.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8cb7fe640 1 -- 172.21.15.57:0/968712593 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c40af400 con 0x7fe8cc0c42a0 2023-10-30T14:28:14.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8cb7fe640 1 -- 172.21.15.57:0/968712593 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c40ab050 con 0x7fe8cc0c42a0 2023-10-30T14:28:14.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8cb7fe640 1 -- 172.21.15.57:0/968712593 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c40af780 con 0x7fe8cc0c42a0 2023-10-30T14:28:14.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8d2e9c640 1 -- 172.21.15.57:0/968712593 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8cc0c42a0 msgr2=0x7fe8cc0ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:14.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8d2e9c640 1 --2- 172.21.15.57:0/968712593 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8cc0c42a0 0x7fe8cc0ca1e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc0d00f0 tx=0x7fe8c401f180 comp rx=0 tx=0).stop 2023-10-30T14:28:14.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8d2e9c640 1 -- 172.21.15.57:0/968712593 shutdown_connections 2023-10-30T14:28:14.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8d2e9c640 1 --2- 172.21.15.57:0/968712593 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8cc0c42a0 0x7fe8cc0ca1e0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:14.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8d2e9c640 1 --2- 172.21.15.57:0/968712593 >> v2:172.21.15.88:3300/0 conn(0x7fe8cc153c80 0x7fe8cc15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:14.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8d2e9c640 1 -- 172.21.15.57:0/968712593 >> 172.21.15.57:0/968712593 conn(0x7fe8cc0715c0 msgr2=0x7fe8cc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:14.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8d2e9c640 1 -- 172.21.15.57:0/968712593 shutdown_connections 2023-10-30T14:28:14.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8d2e9c640 1 -- 172.21.15.57:0/968712593 wait complete. 2023-10-30T14:28:14.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8d2e9c640 1 Processor -- start 2023-10-30T14:28:14.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8d2e9c640 1 -- start start 2023-10-30T14:28:14.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8d2e9c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8cc0c42a0 0x7fe8cc152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:14.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8d2e9c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8cc0c4a10 0x7fe8cc15ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:14.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8d2e9c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8cc153c80 0x7fe8cc149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:14.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8d2e9c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc0b35b0 con 0x7fe8cc0c4a10 2023-10-30T14:28:14.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8d2e9c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc15c420 con 0x7fe8cc153c80 2023-10-30T14:28:14.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8d2e9c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc0b37c0 con 0x7fe8cc0c42a0 2023-10-30T14:28:14.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8cbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8cc0c4a10 0x7fe8cc15ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:14.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8cbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8cc0c4a10 0x7fe8cc15ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40756/0 (socket says 172.21.15.57:40756) 2023-10-30T14:28:14.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.326+0000 7fe8cbfff640 1 -- 172.21.15.57:0/365339549 learned_addr learned my addr 172.21.15.57:0/365339549 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:14.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.330+0000 7fe8d0c38640 1 --2- 172.21.15.57:0/365339549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8cc0c42a0 0x7fe8cc152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:14.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.330+0000 7fe8d1439640 1 --2- 172.21.15.57:0/365339549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8cc153c80 0x7fe8cc149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:14.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.330+0000 7fe8cbfff640 1 -- 172.21.15.57:0/365339549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8cc0c42a0 msgr2=0x7fe8cc152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:14.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.330+0000 7fe8cbfff640 1 --2- 172.21.15.57:0/365339549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8cc0c42a0 0x7fe8cc152850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:14.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.330+0000 7fe8cbfff640 1 -- 172.21.15.57:0/365339549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8cc153c80 msgr2=0x7fe8cc149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:14.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.330+0000 7fe8cbfff640 1 --2- 172.21.15.57:0/365339549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8cc153c80 0x7fe8cc149b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:14.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.330+0000 7fe8cbfff640 1 -- 172.21.15.57:0/365339549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c4009b30 con 0x7fe8cc0c4a10 2023-10-30T14:28:14.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.330+0000 7fe8cbfff640 1 --2- 172.21.15.57:0/365339549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8cc0c4a10 0x7fe8cc15ab60 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe8b800c410 tx=0x7fe8b800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:14.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.330+0000 7fe8c97fa640 1 -- 172.21.15.57:0/365339549 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b8011790 con 0x7fe8cc0c4a10 2023-10-30T14:28:14.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.330+0000 7fe8c97fa640 1 -- 172.21.15.57:0/365339549 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b80118f0 con 0x7fe8cc0c4a10 2023-10-30T14:28:14.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.330+0000 7fe8c97fa640 1 -- 172.21.15.57:0/365339549 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b8011ce0 con 0x7fe8cc0c4a10 2023-10-30T14:28:14.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.330+0000 7fe8d2e9c640 1 -- 172.21.15.57:0/365339549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8cc0b3aa0 con 0x7fe8cc0c4a10 2023-10-30T14:28:14.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.330+0000 7fe8d2e9c640 1 -- 172.21.15.57:0/365339549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8cc0b4060 con 0x7fe8cc0c4a10 2023-10-30T14:28:14.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.330+0000 7fe8d2e9c640 1 -- 172.21.15.57:0/365339549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8cc073c30 con 0x7fe8cc0c4a10 2023-10-30T14:28:14.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.330+0000 7fe8c97fa640 1 -- 172.21.15.57:0/365339549 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe8b801c460 con 0x7fe8cc0c4a10 2023-10-30T14:28:14.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.330+0000 7fe8c97fa640 1 --2- 172.21.15.57:0/365339549 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe8b00664a0 0x7fe8b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:14.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.330+0000 7fe8c97fa640 1 -- 172.21.15.57:0/365339549 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7fe8b8083e40 con 0x7fe8cc0c4a10 2023-10-30T14:28:14.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.334+0000 7fe8d0c38640 1 --2- 172.21.15.57:0/365339549 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe8b00664a0 0x7fe8b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:14.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.334+0000 7fe8d0c38640 1 --2- 172.21.15.57:0/365339549 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe8b00664a0 0x7fe8b0068960 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe8c00045f0 tx=0x7fe8c0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:14.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.334+0000 7fe8c97fa640 1 -- 172.21.15.57:0/365339549 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe8b8050fa0 con 0x7fe8cc0c4a10 2023-10-30T14:28:14.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.530+0000 7fe8d2e9c640 1 -- 172.21.15.57:0/365339549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8cc04e170 con 0x7fe8cc0c4a10 2023-10-30T14:28:14.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.534+0000 7fe8c97fa640 1 -- 172.21.15.57:0/365339549 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5725 (secure 0 0 0) 0x7fe8b80545c0 con 0x7fe8cc0c4a10 2023-10-30T14:28:14.533 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:28:14.534 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:12.971467+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":6027},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:14.534 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 85 2023-10-30T14:28:14.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.534+0000 7fe8d2e9c640 1 -- 172.21.15.57:0/365339549 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe8b00664a0 msgr2=0x7fe8b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:14.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.534+0000 7fe8d2e9c640 1 --2- 172.21.15.57:0/365339549 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe8b00664a0 0x7fe8b0068960 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe8c00045f0 tx=0x7fe8c0008040 comp rx=0 tx=0).stop 2023-10-30T14:28:14.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.534+0000 7fe8d2e9c640 1 -- 172.21.15.57:0/365339549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8cc0c4a10 msgr2=0x7fe8cc15ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:14.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.534+0000 7fe8d2e9c640 1 --2- 172.21.15.57:0/365339549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8cc0c4a10 0x7fe8cc15ab60 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe8b800c410 tx=0x7fe8b800c8d0 comp rx=0 tx=0).stop 2023-10-30T14:28:14.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.534+0000 7fe8d2e9c640 1 -- 172.21.15.57:0/365339549 shutdown_connections 2023-10-30T14:28:14.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.534+0000 7fe8d2e9c640 1 --2- 172.21.15.57:0/365339549 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe8b00664a0 0x7fe8b0068960 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:14.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.534+0000 7fe8d2e9c640 1 --2- 172.21.15.57:0/365339549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8cc153c80 0x7fe8cc149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:14.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.534+0000 7fe8d2e9c640 1 --2- 172.21.15.57:0/365339549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8cc0c4a10 0x7fe8cc15ab60 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:14.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.534+0000 7fe8d2e9c640 1 --2- 172.21.15.57:0/365339549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8cc0c42a0 0x7fe8cc152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:14.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.534+0000 7fe8d2e9c640 1 -- 172.21.15.57:0/365339549 >> 172.21.15.57:0/365339549 conn(0x7fe8cc0715c0 msgr2=0x7fe8cc1575b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:14.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.534+0000 7fe8d2e9c640 1 -- 172.21.15.57:0/365339549 shutdown_connections 2023-10-30T14:28:14.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:14.534+0000 7fe8d2e9c640 1 -- 172.21.15.57:0/365339549 wait complete. 2023-10-30T14:28:14.554 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 85, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 85, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5797": { "addr": "172.21.15.57:6834/2940934016", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2940934016, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2940934016, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5797, "incarnation": 73, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 43 }, "gid_6027": { "addr": "172.21.15.57:6836/2733323524", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2733323524, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2733323524, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6027, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:clientreplay", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:28:12.971467+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5797, "mds_1": 6027 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6839/358521496", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 358521496, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 358521496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 66, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5315, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:28:16.555 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:16.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.710+0000 7fcb4bcc9640 1 Processor -- start 2023-10-30T14:28:16.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.710+0000 7fcb4bcc9640 1 -- start start 2023-10-30T14:28:16.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.710+0000 7fcb4bcc9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcb44151840 0x7fcb440c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:16.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.710+0000 7fcb4bcc9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcb44151c20 0x7fcb44072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:16.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.710+0000 7fcb4bcc9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb440c5030 con 0x7fcb440cb8b0 2023-10-30T14:28:16.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.710+0000 7fcb4bcc9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb44157830 con 0x7fcb44151840 2023-10-30T14:28:16.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.710+0000 7fcb4bcc9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb44157b10 con 0x7fcb44151c20 2023-10-30T14:28:16.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.710+0000 7fcb49a65640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcb44151840 0x7fcb440c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:16.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.710+0000 7fcb49a65640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcb44151840 0x7fcb440c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40758/0 (socket says 172.21.15.57:40758) 2023-10-30T14:28:16.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.710+0000 7fcb4a266640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fcb440cb8b0 0x7fcb4406b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:59778/0 (socket says 172.21.15.57:59778) 2023-10-30T14:28:16.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.710+0000 7fcb49a65640 1 -- 172.21.15.57:0/2404486164 learned_addr learned my addr 172.21.15.57:0/2404486164 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:16.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.710+0000 7fcb49264640 1 --2- 172.21.15.57:0/2404486164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcb44151c20 0x7fcb44072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:16.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.710+0000 7fcb49a65640 1 -- 172.21.15.57:0/2404486164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcb44151c20 msgr2=0x7fcb44072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:16.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.710+0000 7fcb49a65640 1 --2- 172.21.15.57:0/2404486164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcb44151c20 0x7fcb44072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:16.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.710+0000 7fcb49a65640 1 -- 172.21.15.57:0/2404486164 >> v1:172.21.15.57:6789/0 conn(0x7fcb440cb8b0 legacy=0x7fcb4406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:16.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.710+0000 7fcb49a65640 1 -- 172.21.15.57:0/2404486164 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb44157df0 con 0x7fcb44151840 2023-10-30T14:28:16.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.714+0000 7fcb49a65640 1 --2- 172.21.15.57:0/2404486164 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb44151840 0x7fcb440c49a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fcb40009870 tx=0x7fcb4001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b665da7b5841227c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:16.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.714+0000 7fcb48a63640 1 -- 172.21.15.57:0/2404486164 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb400a8a30 con 0x7fcb44151840 2023-10-30T14:28:16.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.714+0000 7fcb48a63640 1 -- 172.21.15.57:0/2404486164 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb400a8b90 con 0x7fcb44151840 2023-10-30T14:28:16.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.714+0000 7fcb48a63640 1 -- 172.21.15.57:0/2404486164 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb400b0440 con 0x7fcb44151840 2023-10-30T14:28:16.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.714+0000 7fcb4bcc9640 1 -- 172.21.15.57:0/2404486164 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb44151840 msgr2=0x7fcb440c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:16.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.714+0000 7fcb4bcc9640 1 --2- 172.21.15.57:0/2404486164 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb44151840 0x7fcb440c49a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fcb40009870 tx=0x7fcb4001ed30 comp rx=0 tx=0).stop 2023-10-30T14:28:16.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.714+0000 7fcb4bcc9640 1 -- 172.21.15.57:0/2404486164 shutdown_connections 2023-10-30T14:28:16.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.714+0000 7fcb4bcc9640 1 --2- 172.21.15.57:0/2404486164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcb44151c20 0x7fcb44072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:16.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.714+0000 7fcb4bcc9640 1 --2- 172.21.15.57:0/2404486164 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb44151840 0x7fcb440c49a0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:16.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.714+0000 7fcb4bcc9640 1 -- 172.21.15.57:0/2404486164 >> 172.21.15.57:0/2404486164 conn(0x7fcb440cac70 msgr2=0x7fcb440cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:16.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.714+0000 7fcb4bcc9640 1 -- 172.21.15.57:0/2404486164 shutdown_connections 2023-10-30T14:28:16.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.714+0000 7fcb4bcc9640 1 -- 172.21.15.57:0/2404486164 wait complete. 2023-10-30T14:28:16.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb4bcc9640 1 Processor -- start 2023-10-30T14:28:16.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb4bcc9640 1 -- start start 2023-10-30T14:28:16.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb4bcc9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb440cb8b0 0x7fcb4414e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:16.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb4bcc9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb44151840 0x7fcb4414b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:16.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb4bcc9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcb44151c20 0x7fcb44154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:16.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb4bcc9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4414cee0 con 0x7fcb44151840 2023-10-30T14:28:16.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb4bcc9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4414ca90 con 0x7fcb440cb8b0 2023-10-30T14:28:16.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb4bcc9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4414eb50 con 0x7fcb44151c20 2023-10-30T14:28:16.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb49264640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb44151840 0x7fcb4414b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:16.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb49264640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb44151840 0x7fcb4414b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40772/0 (socket says 172.21.15.57:40772) 2023-10-30T14:28:16.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb49264640 1 -- 172.21.15.57:0/2424697109 learned_addr learned my addr 172.21.15.57:0/2424697109 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:16.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb49a65640 1 --2- 172.21.15.57:0/2424697109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb440cb8b0 0x7fcb4414e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:16.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb4a266640 1 --2- 172.21.15.57:0/2424697109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcb44151c20 0x7fcb44154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:16.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb49264640 1 -- 172.21.15.57:0/2424697109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcb44151c20 msgr2=0x7fcb44154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:16.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb49264640 1 --2- 172.21.15.57:0/2424697109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcb44151c20 0x7fcb44154200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:16.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb49264640 1 -- 172.21.15.57:0/2424697109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb440cb8b0 msgr2=0x7fcb4414e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:16.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb49264640 1 --2- 172.21.15.57:0/2424697109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb440cb8b0 0x7fcb4414e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:16.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb49264640 1 -- 172.21.15.57:0/2424697109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb400094d0 con 0x7fcb44151840 2023-10-30T14:28:16.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb49264640 1 --2- 172.21.15.57:0/2424697109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb44151840 0x7fcb4414b0e0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fcb3400d7d0 tx=0x7fcb3400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:16.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb3a7fc640 1 -- 172.21.15.57:0/2424697109 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3400b660 con 0x7fcb44151840 2023-10-30T14:28:16.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb3a7fc640 1 -- 172.21.15.57:0/2424697109 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3400b7c0 con 0x7fcb44151840 2023-10-30T14:28:16.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb3a7fc640 1 -- 172.21.15.57:0/2424697109 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3400bb70 con 0x7fcb44151840 2023-10-30T14:28:16.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb4bcc9640 1 -- 172.21.15.57:0/2424697109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4414bf10 con 0x7fcb44151840 2023-10-30T14:28:16.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.718+0000 7fcb4bcc9640 1 -- 172.21.15.57:0/2424697109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4414c4d0 con 0x7fcb44151840 2023-10-30T14:28:16.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.722+0000 7fcb4bcc9640 1 -- 172.21.15.57:0/2424697109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4414d020 con 0x7fcb44151840 2023-10-30T14:28:16.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.722+0000 7fcb3a7fc640 1 -- 172.21.15.57:0/2424697109 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcb3400bcd0 con 0x7fcb44151840 2023-10-30T14:28:16.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.722+0000 7fcb3a7fc640 1 --2- 172.21.15.57:0/2424697109 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcb280664a0 0x7fcb28068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:16.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.722+0000 7fcb3a7fc640 1 -- 172.21.15.57:0/2424697109 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7fcb34084540 con 0x7fcb44151840 2023-10-30T14:28:16.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.722+0000 7fcb49a65640 1 --2- 172.21.15.57:0/2424697109 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcb280664a0 0x7fcb28068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:16.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.722+0000 7fcb49a65640 1 --2- 172.21.15.57:0/2424697109 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcb280664a0 0x7fcb28068960 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcb400b2000 tx=0x7fcb40009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:16.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.726+0000 7fcb3a7fc640 1 -- 172.21.15.57:0/2424697109 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcb4414d020 con 0x7fcb44151840 2023-10-30T14:28:16.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.918+0000 7fcb4bcc9640 1 -- 172.21.15.57:0/2424697109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb4414d020 con 0x7fcb44151840 2023-10-30T14:28:16.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.918+0000 7fcb3a7fc640 1 -- 172.21.15.57:0/2424697109 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5725 (secure 0 0 0) 0x7fcb4414d020 con 0x7fcb44151840 2023-10-30T14:28:16.924 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:28:16.924 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:12.971467+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":6027},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:16.925 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 85 2023-10-30T14:28:16.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.922+0000 7fcb4bcc9640 1 -- 172.21.15.57:0/2424697109 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcb280664a0 msgr2=0x7fcb28068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:16.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.922+0000 7fcb4bcc9640 1 --2- 172.21.15.57:0/2424697109 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcb280664a0 0x7fcb28068960 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcb400b2000 tx=0x7fcb40009040 comp rx=0 tx=0).stop 2023-10-30T14:28:16.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.922+0000 7fcb4bcc9640 1 -- 172.21.15.57:0/2424697109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb44151840 msgr2=0x7fcb4414b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:16.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.922+0000 7fcb4bcc9640 1 --2- 172.21.15.57:0/2424697109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb44151840 0x7fcb4414b0e0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fcb3400d7d0 tx=0x7fcb3400dc90 comp rx=0 tx=0).stop 2023-10-30T14:28:16.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.922+0000 7fcb4bcc9640 1 -- 172.21.15.57:0/2424697109 shutdown_connections 2023-10-30T14:28:16.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.922+0000 7fcb4bcc9640 1 --2- 172.21.15.57:0/2424697109 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcb280664a0 0x7fcb28068960 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:16.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.922+0000 7fcb4bcc9640 1 --2- 172.21.15.57:0/2424697109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcb44151c20 0x7fcb44154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:16.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.922+0000 7fcb4bcc9640 1 --2- 172.21.15.57:0/2424697109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb44151840 0x7fcb4414b0e0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:16.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.922+0000 7fcb4bcc9640 1 --2- 172.21.15.57:0/2424697109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb440cb8b0 0x7fcb4414e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:16.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.922+0000 7fcb4bcc9640 1 -- 172.21.15.57:0/2424697109 >> 172.21.15.57:0/2424697109 conn(0x7fcb440cac70 msgr2=0x7fcb4414fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:16.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.922+0000 7fcb4bcc9640 1 -- 172.21.15.57:0/2424697109 shutdown_connections 2023-10-30T14:28:16.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:16.922+0000 7fcb4bcc9640 1 -- 172.21.15.57:0/2424697109 wait complete. 2023-10-30T14:28:16.942 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:28:18.943 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:19.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.146+0000 7efd06074640 1 Processor -- start 2023-10-30T14:28:19.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.150+0000 7efd06074640 1 -- start start 2023-10-30T14:28:19.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.150+0000 7efd06074640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efd0015ac70 0x7efd0015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:19.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.150+0000 7efd06074640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0015dcc0 con 0x7efd0006aeb0 2023-10-30T14:28:19.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.150+0000 7efd06074640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd00159d80 con 0x7efd000c8eb0 2023-10-30T14:28:19.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.150+0000 7efd06074640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd00159800 con 0x7efd0015ac70 2023-10-30T14:28:19.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.150+0000 7efd04871640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efd0015ac70 0x7efd0015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:19.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.150+0000 7efd05873640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7efd0006aeb0 0x7efd00069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:59782/0 (socket says 172.21.15.57:59782) 2023-10-30T14:28:19.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.150+0000 7efd05873640 1 -- 172.21.15.57:0/769003606 learned_addr learned my addr 172.21.15.57:0/769003606 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:28:19.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.150+0000 7efcf7fff640 1 -- 172.21.15.57:0/769003606 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425535017 0 0) 0x7efcf80014a0 con 0x7efd0006aeb0 2023-10-30T14:28:19.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.150+0000 7efcf7fff640 1 -- 172.21.15.57:0/769003606 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcec003530 con 0x7efd0006aeb0 2023-10-30T14:28:19.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.150+0000 7efd04871640 1 -- 172.21.15.57:0/769003606 >> v1:172.21.15.88:6789/0 conn(0x7efd000c8eb0 legacy=0x7efd0015a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:19.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.150+0000 7efcf7fff640 1 -- 172.21.15.57:0/769003606 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166346738 0 0) 0x7efd00159d80 con 0x7efd000c8eb0 2023-10-30T14:28:19.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.150+0000 7efd04871640 1 -- 172.21.15.57:0/769003606 >> v1:172.21.15.57:6789/0 conn(0x7efd0006aeb0 legacy=0x7efd00069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:19.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.150+0000 7efd04871640 1 -- 172.21.15.57:0/769003606 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0015d420 con 0x7efd0015ac70 2023-10-30T14:28:19.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.154+0000 7efd04871640 1 --2- 172.21.15.57:0/769003606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd0015ac70 0x7efd0015cd70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efcf00097c0 tx=0x7efcf001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa59b122a3949766 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:19.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.154+0000 7efcf7fff640 1 -- 172.21.15.57:0/769003606 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf00a8970 con 0x7efd0015ac70 2023-10-30T14:28:19.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.154+0000 7efcf7fff640 1 -- 172.21.15.57:0/769003606 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf00a8ad0 con 0x7efd0015ac70 2023-10-30T14:28:19.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.154+0000 7efcf7fff640 1 -- 172.21.15.57:0/769003606 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf00a8e60 con 0x7efd0015ac70 2023-10-30T14:28:19.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.154+0000 7efd06074640 1 -- 172.21.15.57:0/769003606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd0015ac70 msgr2=0x7efd0015cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:19.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.154+0000 7efd06074640 1 --2- 172.21.15.57:0/769003606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd0015ac70 0x7efd0015cd70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efcf00097c0 tx=0x7efcf001ec70 comp rx=0 tx=0).stop 2023-10-30T14:28:19.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.154+0000 7efd06074640 1 -- 172.21.15.57:0/769003606 shutdown_connections 2023-10-30T14:28:19.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.154+0000 7efd06074640 1 --2- 172.21.15.57:0/769003606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd0015ac70 0x7efd0015cd70 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:19.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.154+0000 7efd06074640 1 -- 172.21.15.57:0/769003606 >> 172.21.15.57:0/769003606 conn(0x7efd000cef00 msgr2=0x7efd000cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:19.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.154+0000 7efd06074640 1 -- 172.21.15.57:0/769003606 shutdown_connections 2023-10-30T14:28:19.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.154+0000 7efd06074640 1 -- 172.21.15.57:0/769003606 wait complete. 2023-10-30T14:28:19.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd06074640 1 Processor -- start 2023-10-30T14:28:19.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd06074640 1 -- start start 2023-10-30T14:28:19.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd06074640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0006aeb0 0x7efd0014d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:19.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd06074640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd000c8eb0 0x7efd0014d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:19.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd06074640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd0015ac70 0x7efd001529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:19.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd06074640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd000c6ca0 con 0x7efd000c8eb0 2023-10-30T14:28:19.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd06074640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd001ee7f0 con 0x7efd0015ac70 2023-10-30T14:28:19.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd06074640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd001eea00 con 0x7efd0006aeb0 2023-10-30T14:28:19.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd05072640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0006aeb0 0x7efd0014d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:19.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd04871640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd000c8eb0 0x7efd0014d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:19.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd04871640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd000c8eb0 0x7efd0014d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40774/0 (socket says 172.21.15.57:40774) 2023-10-30T14:28:19.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd05072640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0006aeb0 0x7efd0014d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:37044/0 (socket says 172.21.15.57:37044) 2023-10-30T14:28:19.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd04871640 1 -- 172.21.15.57:0/2558802213 learned_addr learned my addr 172.21.15.57:0/2558802213 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:19.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd05873640 1 --2- 172.21.15.57:0/2558802213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd0015ac70 0x7efd001529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:19.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd05873640 1 -- 172.21.15.57:0/2558802213 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0006aeb0 msgr2=0x7efd0014d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:19.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd05873640 1 --2- 172.21.15.57:0/2558802213 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0006aeb0 0x7efd0014d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:19.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd05873640 1 -- 172.21.15.57:0/2558802213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd000c8eb0 msgr2=0x7efd0014d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:19.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd05873640 1 --2- 172.21.15.57:0/2558802213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd000c8eb0 0x7efd0014d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:19.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd05873640 1 -- 172.21.15.57:0/2558802213 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf0009420 con 0x7efd0015ac70 2023-10-30T14:28:19.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.158+0000 7efd04871640 1 --2- 172.21.15.57:0/2558802213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd000c8eb0 0x7efd0014d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:28:19.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.162+0000 7efd05873640 1 --2- 172.21.15.57:0/2558802213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd0015ac70 0x7efd001529b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efcf800caa0 tx=0x7efcf800cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:19.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.162+0000 7efcf5ffb640 1 -- 172.21.15.57:0/2558802213 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf800ac00 con 0x7efd0015ac70 2023-10-30T14:28:19.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.162+0000 7efcf5ffb640 1 -- 172.21.15.57:0/2558802213 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf800ad60 con 0x7efd0015ac70 2023-10-30T14:28:19.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.162+0000 7efcf5ffb640 1 -- 172.21.15.57:0/2558802213 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf8012620 con 0x7efd0015ac70 2023-10-30T14:28:19.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.162+0000 7efd06074640 1 -- 172.21.15.57:0/2558802213 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd001eece0 con 0x7efd0015ac70 2023-10-30T14:28:19.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.162+0000 7efd06074640 1 -- 172.21.15.57:0/2558802213 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd001ef2a0 con 0x7efd0015ac70 2023-10-30T14:28:19.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.162+0000 7efd06074640 1 -- 172.21.15.57:0/2558802213 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd0004e170 con 0x7efd0015ac70 2023-10-30T14:28:19.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.166+0000 7efcf5ffb640 1 -- 172.21.15.57:0/2558802213 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efcf8007070 con 0x7efd0015ac70 2023-10-30T14:28:19.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.166+0000 7efcf5ffb640 1 --2- 172.21.15.57:0/2558802213 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efce40664a0 0x7efce4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:19.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.166+0000 7efcf5ffb640 1 -- 172.21.15.57:0/2558802213 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7efcf8084100 con 0x7efd0015ac70 2023-10-30T14:28:19.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.166+0000 7efd05072640 1 --2- 172.21.15.57:0/2558802213 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efce40664a0 0x7efce4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:19.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.166+0000 7efd05072640 1 --2- 172.21.15.57:0/2558802213 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efce40664a0 0x7efce4068960 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7efcfc007fd0 tx=0x7efcfc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:19.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.170+0000 7efcf5ffb640 1 -- 172.21.15.57:0/2558802213 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efcf8051250 con 0x7efd0015ac70 2023-10-30T14:28:19.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.378+0000 7efd06074640 1 -- 172.21.15.57:0/2558802213 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd0015dcc0 con 0x7efd0015ac70 2023-10-30T14:28:19.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.378+0000 7efcf5ffb640 1 -- 172.21.15.57:0/2558802213 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5725 (secure 0 0 0) 0x7efd0015dcc0 con 0x7efd0015ac70 2023-10-30T14:28:19.379 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:28:19.379 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:12.971467+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":6027},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:19.379 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 85 2023-10-30T14:28:19.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.378+0000 7efd06074640 1 -- 172.21.15.57:0/2558802213 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efce40664a0 msgr2=0x7efce4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:19.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.378+0000 7efd06074640 1 --2- 172.21.15.57:0/2558802213 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efce40664a0 0x7efce4068960 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7efcfc007fd0 tx=0x7efcfc009040 comp rx=0 tx=0).stop 2023-10-30T14:28:19.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.378+0000 7efd06074640 1 -- 172.21.15.57:0/2558802213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd0015ac70 msgr2=0x7efd001529b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:19.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.378+0000 7efd06074640 1 --2- 172.21.15.57:0/2558802213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd0015ac70 0x7efd001529b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efcf800caa0 tx=0x7efcf800cf60 comp rx=0 tx=0).stop 2023-10-30T14:28:19.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.378+0000 7efd06074640 1 -- 172.21.15.57:0/2558802213 shutdown_connections 2023-10-30T14:28:19.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.378+0000 7efd06074640 1 --2- 172.21.15.57:0/2558802213 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efce40664a0 0x7efce4068960 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:19.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.378+0000 7efd06074640 1 --2- 172.21.15.57:0/2558802213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd0015ac70 0x7efd001529b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:19.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.378+0000 7efd06074640 1 --2- 172.21.15.57:0/2558802213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd000c8eb0 0x7efd0014d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:19.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.378+0000 7efd06074640 1 --2- 172.21.15.57:0/2558802213 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0006aeb0 0x7efd0014d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:19.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.378+0000 7efd06074640 1 -- 172.21.15.57:0/2558802213 >> 172.21.15.57:0/2558802213 conn(0x7efd000cef00 msgr2=0x7efd000c7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:19.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.382+0000 7efd06074640 1 -- 172.21.15.57:0/2558802213 shutdown_connections 2023-10-30T14:28:19.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:19.382+0000 7efd06074640 1 -- 172.21.15.57:0/2558802213 wait complete. 2023-10-30T14:28:19.399 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:28:21.401 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:21.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4ee0d6640 1 Processor -- start 2023-10-30T14:28:21.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4ee0d6640 1 -- start start 2023-10-30T14:28:21.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4ee0d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa4e8153c80 0x7fa4e815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:21.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4ee0d6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e8159f20 con 0x7fa4e80c4a10 2023-10-30T14:28:21.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4ee0d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e815d620 con 0x7fa4e80c42a0 2023-10-30T14:28:21.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4ee0d6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4e815dec0 con 0x7fa4e8153c80 2023-10-30T14:28:21.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e6ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa4e80c4a10 0x7fa4e815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:59794/0 (socket says 172.21.15.57:59794) 2023-10-30T14:28:21.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e6ffd640 1 -- 172.21.15.57:0/3049836990 learned_addr learned my addr 172.21.15.57:0/3049836990 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:28:21.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e77fe640 1 --2- 172.21.15.57:0/3049836990 >> v2:172.21.15.88:3300/0 conn(0x7fa4e8153c80 0x7fa4e815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:21.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e67fc640 1 -- 172.21.15.57:0/3049836990 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3932315700 0 0) 0x7fa4e8159f20 con 0x7fa4e80c4a10 2023-10-30T14:28:21.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e67fc640 1 -- 172.21.15.57:0/3049836990 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4d0003530 con 0x7fa4e80c4a10 2023-10-30T14:28:21.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e67fc640 1 -- 172.21.15.57:0/3049836990 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3320687243 0 0) 0x7fa4d0003530 con 0x7fa4e80c4a10 2023-10-30T14:28:21.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e67fc640 1 -- 172.21.15.57:0/3049836990 >> v2:172.21.15.88:3300/0 conn(0x7fa4e8153c80 msgr2=0x7fa4e815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:21.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e67fc640 1 --2- 172.21.15.57:0/3049836990 >> v2:172.21.15.88:3300/0 conn(0x7fa4e8153c80 0x7fa4e815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:21.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e67fc640 1 -- 172.21.15.57:0/3049836990 >> v1:172.21.15.88:6789/0 conn(0x7fa4e80c42a0 legacy=0x7fa4e80ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:21.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e67fc640 1 -- 172.21.15.57:0/3049836990 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e815d900 con 0x7fa4e80c4a10 2023-10-30T14:28:21.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e67fc640 1 -- 172.21.15.57:0/3049836990 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fa4dc001680 con 0x7fa4e80c4a10 2023-10-30T14:28:21.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e67fc640 1 -- 172.21.15.57:0/3049836990 >> v1:172.21.15.57:6789/0 conn(0x7fa4e80c4a10 legacy=0x7fa4e815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:21.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e67fc640 1 --2- 172.21.15.57:0/3049836990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4d0003980 0x7fa4d0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:21.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e67fc640 1 --2- 172.21.15.57:0/3049836990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4d00047d0 0x7fa4d0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:21.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e67fc640 1 --2- 172.21.15.57:0/3049836990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4d0005190 0x7fa4d000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:21.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e67fc640 1 -- 172.21.15.57:0/3049836990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e815dec0 con 0x7fa4d0003980 2023-10-30T14:28:21.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e67fc640 1 -- 172.21.15.57:0/3049836990 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d000c040 con 0x7fa4d0005190 2023-10-30T14:28:21.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e67fc640 1 -- 172.21.15.57:0/3049836990 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d000c200 con 0x7fa4d00047d0 2023-10-30T14:28:21.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e77fe640 1 --2- 172.21.15.57:0/3049836990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4d0003980 0x7fa4d0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:21.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e6ffd640 1 --2- 172.21.15.57:0/3049836990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4d00047d0 0x7fa4d0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:21.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e7fff640 1 --2- 172.21.15.57:0/3049836990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4d0005190 0x7fa4d000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:21.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e77fe640 1 -- 172.21.15.57:0/3049836990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4d00047d0 msgr2=0x7fa4d0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:21.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e77fe640 1 --2- 172.21.15.57:0/3049836990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4d00047d0 0x7fa4d0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:21.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e77fe640 1 -- 172.21.15.57:0/3049836990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4d0005190 msgr2=0x7fa4d000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:21.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e77fe640 1 --2- 172.21.15.57:0/3049836990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4d0005190 0x7fa4d000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:21.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e77fe640 1 -- 172.21.15.57:0/3049836990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d0003530 con 0x7fa4d0003980 2023-10-30T14:28:21.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.558+0000 7fa4e7fff640 1 -- 172.21.15.57:0/3049836990 reap_dead start 2023-10-30T14:28:21.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.562+0000 7fa4e77fe640 1 --2- 172.21.15.57:0/3049836990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4d0003980 0x7fa4d0003da0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8009ae0 tx=0x7fa4d801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59ccecc9d269331a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:21.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.562+0000 7fa4e67fc640 1 -- 172.21.15.57:0/3049836990 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d80a5530 con 0x7fa4d0003980 2023-10-30T14:28:21.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.562+0000 7fa4e67fc640 1 -- 172.21.15.57:0/3049836990 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d80a5690 con 0x7fa4d0003980 2023-10-30T14:28:21.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.562+0000 7fa4e67fc640 1 -- 172.21.15.57:0/3049836990 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d80a5a40 con 0x7fa4d0003980 2023-10-30T14:28:21.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.562+0000 7fa4ee0d6640 1 -- 172.21.15.57:0/3049836990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4d0003980 msgr2=0x7fa4d0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:21.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.562+0000 7fa4ee0d6640 1 --2- 172.21.15.57:0/3049836990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4d0003980 0x7fa4d0003da0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8009ae0 tx=0x7fa4d801ca50 comp rx=0 tx=0).stop 2023-10-30T14:28:21.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.562+0000 7fa4ee0d6640 1 -- 172.21.15.57:0/3049836990 shutdown_connections 2023-10-30T14:28:21.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.562+0000 7fa4ee0d6640 1 --2- 172.21.15.57:0/3049836990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4d0003980 0x7fa4d0003da0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:21.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.562+0000 7fa4ee0d6640 1 -- 172.21.15.57:0/3049836990 >> 172.21.15.57:0/3049836990 conn(0x7fa4e80715c0 msgr2=0x7fa4e80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:21.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.562+0000 7fa4ee0d6640 1 -- 172.21.15.57:0/3049836990 shutdown_connections 2023-10-30T14:28:21.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.562+0000 7fa4ee0d6640 1 -- 172.21.15.57:0/3049836990 wait complete. 2023-10-30T14:28:21.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4ee0d6640 1 Processor -- start 2023-10-30T14:28:21.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4ee0d6640 1 -- start start 2023-10-30T14:28:21.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4ee0d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4d0003980 0x7fa4e814fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:21.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4ee0d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4e8149ba0 0x7fa4e814a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:21.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4ee0d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4e81eff30 0x7fa4e81f03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:21.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4ee0d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e80b25e0 con 0x7fa4e81eff30 2023-10-30T14:28:21.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4ee0d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e81f08f0 con 0x7fa4e8149ba0 2023-10-30T14:28:21.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4ee0d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e815ad00 con 0x7fa4d0003980 2023-10-30T14:28:21.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4e7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4e81eff30 0x7fa4e81f03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:21.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4e7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4e81eff30 0x7fa4e81f03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40788/0 (socket says 172.21.15.57:40788) 2023-10-30T14:28:21.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4e7fff640 1 -- 172.21.15.57:0/4161068250 learned_addr learned my addr 172.21.15.57:0/4161068250 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:21.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4e6ffd640 1 --2- 172.21.15.57:0/4161068250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4e8149ba0 0x7fa4e814a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:21.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4e77fe640 1 --2- 172.21.15.57:0/4161068250 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4d0003980 0x7fa4e814fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:21.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4e7fff640 1 -- 172.21.15.57:0/4161068250 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4d0003980 msgr2=0x7fa4e814fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:21.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4e7fff640 1 --2- 172.21.15.57:0/4161068250 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4d0003980 0x7fa4e814fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:21.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4e7fff640 1 -- 172.21.15.57:0/4161068250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4e8149ba0 msgr2=0x7fa4e814a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:21.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4e7fff640 1 --2- 172.21.15.57:0/4161068250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4e8149ba0 0x7fa4e814a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:21.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4e7fff640 1 -- 172.21.15.57:0/4161068250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e815afe0 con 0x7fa4e81eff30 2023-10-30T14:28:21.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4e7fff640 1 --2- 172.21.15.57:0/4161068250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4e81eff30 0x7fa4e81f03b0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fa4e80c97d0 tx=0x7fa4e000e920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:21.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4cffff640 1 -- 172.21.15.57:0/4161068250 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0006e30 con 0x7fa4e81eff30 2023-10-30T14:28:21.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4cffff640 1 -- 172.21.15.57:0/4161068250 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e0003980 con 0x7fa4e81eff30 2023-10-30T14:28:21.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4cffff640 1 -- 172.21.15.57:0/4161068250 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0003d30 con 0x7fa4e81eff30 2023-10-30T14:28:21.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.566+0000 7fa4ee0d6640 1 -- 172.21.15.57:0/4161068250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e815b2c0 con 0x7fa4e81eff30 2023-10-30T14:28:21.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.570+0000 7fa4ee0d6640 1 -- 172.21.15.57:0/4161068250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e815b880 con 0x7fa4e81eff30 2023-10-30T14:28:21.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.570+0000 7fa4ee0d6640 1 -- 172.21.15.57:0/4161068250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e81efd10 con 0x7fa4e81eff30 2023-10-30T14:28:21.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.570+0000 7fa4cffff640 1 -- 172.21.15.57:0/4161068250 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa4e0005070 con 0x7fa4e81eff30 2023-10-30T14:28:21.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.570+0000 7fa4cffff640 1 --2- 172.21.15.57:0/4161068250 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa4d40664a0 0x7fa4d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:21.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.570+0000 7fa4cffff640 1 -- 172.21.15.57:0/4161068250 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7fa4e0083670 con 0x7fa4e81eff30 2023-10-30T14:28:21.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.570+0000 7fa4e77fe640 1 --2- 172.21.15.57:0/4161068250 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa4d40664a0 0x7fa4d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:21.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.574+0000 7fa4e77fe640 1 --2- 172.21.15.57:0/4161068250 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa4d40664a0 0x7fa4d4068960 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa4d80af000 tx=0x7fa4d80046d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:21.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.574+0000 7fa4cffff640 1 -- 172.21.15.57:0/4161068250 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa4e0050740 con 0x7fa4e81eff30 2023-10-30T14:28:21.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.766+0000 7fa4ee0d6640 1 -- 172.21.15.57:0/4161068250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4e804e170 con 0x7fa4e81eff30 2023-10-30T14:28:21.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.766+0000 7fa4cffff640 1 -- 172.21.15.57:0/4161068250 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5719 (secure 0 0 0) 0x7fa4e0053d60 con 0x7fa4e81eff30 2023-10-30T14:28:21.766 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:28:21.766 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:19.215998+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":6027},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:21.767 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 86 2023-10-30T14:28:21.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.766+0000 7fa4ee0d6640 1 -- 172.21.15.57:0/4161068250 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa4d40664a0 msgr2=0x7fa4d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:21.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.766+0000 7fa4ee0d6640 1 --2- 172.21.15.57:0/4161068250 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa4d40664a0 0x7fa4d4068960 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa4d80af000 tx=0x7fa4d80046d0 comp rx=0 tx=0).stop 2023-10-30T14:28:21.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.766+0000 7fa4ee0d6640 1 -- 172.21.15.57:0/4161068250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4e81eff30 msgr2=0x7fa4e81f03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:21.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.766+0000 7fa4ee0d6640 1 --2- 172.21.15.57:0/4161068250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4e81eff30 0x7fa4e81f03b0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fa4e80c97d0 tx=0x7fa4e000e920 comp rx=0 tx=0).stop 2023-10-30T14:28:21.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.766+0000 7fa4ee0d6640 1 -- 172.21.15.57:0/4161068250 shutdown_connections 2023-10-30T14:28:21.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.766+0000 7fa4ee0d6640 1 --2- 172.21.15.57:0/4161068250 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa4d40664a0 0x7fa4d4068960 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:21.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.766+0000 7fa4ee0d6640 1 --2- 172.21.15.57:0/4161068250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4e81eff30 0x7fa4e81f03b0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:21.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.766+0000 7fa4ee0d6640 1 --2- 172.21.15.57:0/4161068250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4e8149ba0 0x7fa4e814a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:21.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.766+0000 7fa4ee0d6640 1 --2- 172.21.15.57:0/4161068250 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4d0003980 0x7fa4e814fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:21.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.766+0000 7fa4ee0d6640 1 -- 172.21.15.57:0/4161068250 >> 172.21.15.57:0/4161068250 conn(0x7fa4e80715c0 msgr2=0x7fa4e8151140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:21.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.766+0000 7fa4ee0d6640 1 -- 172.21.15.57:0/4161068250 shutdown_connections 2023-10-30T14:28:21.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:21.766+0000 7fa4ee0d6640 1 -- 172.21.15.57:0/4161068250 wait complete. 2023-10-30T14:28:21.787 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=1, replacing gid=5586 2023-10-30T14:28:21.788 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.e 2023-10-30T14:28:26.788 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T14:28:26.789 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T14:28:26.789 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T14:28:26.793 INFO:tasks.ceph.mds.e:Started 2023-10-30T14:28:26.793 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:26.889 INFO:tasks.ceph.mds.e.smithi057.stdout:starting mds.e at 2023-10-30T14:28:26.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.942+0000 7fe5a529c640 1 Processor -- start 2023-10-30T14:28:26.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.942+0000 7fe5a529c640 1 -- start start 2023-10-30T14:28:26.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.942+0000 7fe5a529c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe5a0150830 0x7fe5a006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:26.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.942+0000 7fe5a529c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe5a00cb7d0 0x7fe5a00c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:26.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.942+0000 7fe5a529c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a00c2310 con 0x7fe5a00c3e40 2023-10-30T14:28:26.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.942+0000 7fe5a529c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a0156710 con 0x7fe5a00cb7d0 2023-10-30T14:28:26.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.942+0000 7fe5a529c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a00c4360 con 0x7fe5a0150830 2023-10-30T14:28:26.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.942+0000 7fe597fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe5a00cb7d0 0x7fe5a00c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:26.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.942+0000 7fe59f7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fe5a00c3e40 0x7fe5a0070520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44504/0 (socket says 172.21.15.57:44504) 2023-10-30T14:28:26.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.942+0000 7fe59f7fe640 1 -- 172.21.15.57:0/4136364948 learned_addr learned my addr 172.21.15.57:0/4136364948 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:28:26.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.942+0000 7fe597fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe5a00cb7d0 0x7fe5a00c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56458/0 (socket says 172.21.15.57:56458) 2023-10-30T14:28:26.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.942+0000 7fe59effd640 1 --2- 172.21.15.57:0/4136364948 >> v2:172.21.15.88:3300/0 conn(0x7fe5a0150830 0x7fe5a006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:26.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.942+0000 7fe597fff640 1 -- 172.21.15.57:0/4136364948 >> v2:172.21.15.88:3300/0 conn(0x7fe5a0150830 msgr2=0x7fe5a006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:26.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.942+0000 7fe597fff640 1 --2- 172.21.15.57:0/4136364948 >> v2:172.21.15.88:3300/0 conn(0x7fe5a0150830 0x7fe5a006b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:26.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.942+0000 7fe597fff640 1 -- 172.21.15.57:0/4136364948 >> v1:172.21.15.57:6789/0 conn(0x7fe5a00c3e40 legacy=0x7fe5a0070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:26.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.942+0000 7fe597fff640 1 -- 172.21.15.57:0/4136364948 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a01569b0 con 0x7fe5a00cb7d0 2023-10-30T14:28:26.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.942+0000 7fe59e7fc640 1 -- 172.21.15.57:0/4136364948 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368595125 0 0) 0x7fe5a00c2310 con 0x7fe5a00c3e40 2023-10-30T14:28:26.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.946+0000 7fe59effd640 1 --2- 172.21.15.57:0/4136364948 >> v2:172.21.15.88:3300/0 conn(0x7fe5a0150830 0x7fe5a006b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:28:26.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.946+0000 7fe597fff640 1 --2- 172.21.15.57:0/4136364948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5a00cb7d0 0x7fe5a00c3900 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fe5a006fa40 tx=0x7fe58801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73f663ae1a3c997d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:26.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.946+0000 7fe59e7fc640 1 -- 172.21.15.57:0/4136364948 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5880a8950 con 0x7fe5a00cb7d0 2023-10-30T14:28:26.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.946+0000 7fe59e7fc640 1 -- 172.21.15.57:0/4136364948 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5880a8ab0 con 0x7fe5a00cb7d0 2023-10-30T14:28:26.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.946+0000 7fe59e7fc640 1 -- 172.21.15.57:0/4136364948 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5880a8e40 con 0x7fe5a00cb7d0 2023-10-30T14:28:26.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.946+0000 7fe5a529c640 1 -- 172.21.15.57:0/4136364948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5a00cb7d0 msgr2=0x7fe5a00c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:26.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.946+0000 7fe5a529c640 1 --2- 172.21.15.57:0/4136364948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5a00cb7d0 0x7fe5a00c3900 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fe5a006fa40 tx=0x7fe58801ec50 comp rx=0 tx=0).stop 2023-10-30T14:28:26.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.946+0000 7fe5a529c640 1 -- 172.21.15.57:0/4136364948 shutdown_connections 2023-10-30T14:28:26.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.946+0000 7fe5a529c640 1 --2- 172.21.15.57:0/4136364948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5a00cb7d0 0x7fe5a00c3900 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:26.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.946+0000 7fe5a529c640 1 --2- 172.21.15.57:0/4136364948 >> v2:172.21.15.88:3300/0 conn(0x7fe5a0150830 0x7fe5a006b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:26.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.946+0000 7fe5a529c640 1 -- 172.21.15.57:0/4136364948 >> 172.21.15.57:0/4136364948 conn(0x7fe5a00cac70 msgr2=0x7fe5a00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:26.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.946+0000 7fe5a529c640 1 -- 172.21.15.57:0/4136364948 shutdown_connections 2023-10-30T14:28:26.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.946+0000 7fe5a529c640 1 -- 172.21.15.57:0/4136364948 wait complete. 2023-10-30T14:28:26.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe5a529c640 1 Processor -- start 2023-10-30T14:28:26.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe5a529c640 1 -- start start 2023-10-30T14:28:26.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe5a529c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5a00c3e40 0x7fe5a006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:26.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe5a529c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5a00cb7d0 0x7fe5a01ee310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:26.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe5a529c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5a0150830 0x7fe5a01577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:26.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe5a529c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a006c290 con 0x7fe5a00cb7d0 2023-10-30T14:28:26.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe5a529c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a0159530 con 0x7fe5a00c3e40 2023-10-30T14:28:26.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe5a529c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a01edb50 con 0x7fe5a0150830 2023-10-30T14:28:26.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe597fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5a00cb7d0 0x7fe5a01ee310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:26.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe597fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5a00cb7d0 0x7fe5a01ee310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56474/0 (socket says 172.21.15.57:56474) 2023-10-30T14:28:26.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe597fff640 1 -- 172.21.15.57:0/607700150 learned_addr learned my addr 172.21.15.57:0/607700150 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:26.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe59effd640 1 --2- 172.21.15.57:0/607700150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5a00c3e40 0x7fe5a006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:26.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe59f7fe640 1 --2- 172.21.15.57:0/607700150 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5a0150830 0x7fe5a01577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:26.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe597fff640 1 -- 172.21.15.57:0/607700150 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5a0150830 msgr2=0x7fe5a01577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:26.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe597fff640 1 --2- 172.21.15.57:0/607700150 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5a0150830 0x7fe5a01577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:26.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe597fff640 1 -- 172.21.15.57:0/607700150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5a00c3e40 msgr2=0x7fe5a006b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:26.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe597fff640 1 --2- 172.21.15.57:0/607700150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5a00c3e40 0x7fe5a006b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:26.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe597fff640 1 -- 172.21.15.57:0/607700150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe588009420 con 0x7fe5a00cb7d0 2023-10-30T14:28:26.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe59effd640 1 --2- 172.21.15.57:0/607700150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5a00c3e40 0x7fe5a006b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:28:26.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe597fff640 1 --2- 172.21.15.57:0/607700150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5a00cb7d0 0x7fe5a01ee310 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fe5880b3040 tx=0x7fe58801b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:26.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe5977fe640 1 -- 172.21.15.57:0/607700150 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5880a8950 con 0x7fe5a00cb7d0 2023-10-30T14:28:26.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe5977fe640 1 -- 172.21.15.57:0/607700150 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5880b9410 con 0x7fe5a00cb7d0 2023-10-30T14:28:26.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe5977fe640 1 -- 172.21.15.57:0/607700150 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5880b97c0 con 0x7fe5a00cb7d0 2023-10-30T14:28:26.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.950+0000 7fe5a529c640 1 -- 172.21.15.57:0/607700150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a0159780 con 0x7fe5a00cb7d0 2023-10-30T14:28:26.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.954+0000 7fe5a529c640 1 -- 172.21.15.57:0/607700150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a0159d20 con 0x7fe5a00cb7d0 2023-10-30T14:28:26.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.954+0000 7fe5a529c640 1 -- 172.21.15.57:0/607700150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a0066510 con 0x7fe5a00cb7d0 2023-10-30T14:28:26.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.954+0000 7fe5977fe640 1 -- 172.21.15.57:0/607700150 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe5880b7040 con 0x7fe5a00cb7d0 2023-10-30T14:28:26.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.954+0000 7fe5977fe640 1 --2- 172.21.15.57:0/607700150 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5800664a0 0x7fe580068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:26.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.954+0000 7fe5977fe640 1 -- 172.21.15.57:0/607700150 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7fe588121ec0 con 0x7fe5a00cb7d0 2023-10-30T14:28:26.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.954+0000 7fe59effd640 1 --2- 172.21.15.57:0/607700150 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5800664a0 0x7fe580068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:26.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.958+0000 7fe59effd640 1 --2- 172.21.15.57:0/607700150 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5800664a0 0x7fe580068960 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe59000b990 tx=0x7fe590008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:26.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:26.962+0000 7fe5977fe640 1 -- 172.21.15.57:0/607700150 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe5880eeff0 con 0x7fe5a00cb7d0 2023-10-30T14:28:27.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:27.174+0000 7fe5a529c640 1 -- 172.21.15.57:0/607700150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5a004e170 con 0x7fe5a00cb7d0 2023-10-30T14:28:27.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:27.174+0000 7fe5977fe640 1 -- 172.21.15.57:0/607700150 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+6510 (secure 0 0 0) 0x7fe5880b5020 con 0x7fe5a00cb7d0 2023-10-30T14:28:27.175 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:28:27.175 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":6204,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:19.215998+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":6027},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:27.175 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 87 2023-10-30T14:28:27.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:27.174+0000 7fe5a529c640 1 -- 172.21.15.57:0/607700150 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5800664a0 msgr2=0x7fe580068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:27.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:27.174+0000 7fe5a529c640 1 --2- 172.21.15.57:0/607700150 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5800664a0 0x7fe580068960 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe59000b990 tx=0x7fe590008040 comp rx=0 tx=0).stop 2023-10-30T14:28:27.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:27.174+0000 7fe5a529c640 1 -- 172.21.15.57:0/607700150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5a00cb7d0 msgr2=0x7fe5a01ee310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:27.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:27.174+0000 7fe5a529c640 1 --2- 172.21.15.57:0/607700150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5a00cb7d0 0x7fe5a01ee310 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fe5880b3040 tx=0x7fe58801b9f0 comp rx=0 tx=0).stop 2023-10-30T14:28:27.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:27.174+0000 7fe5a529c640 1 -- 172.21.15.57:0/607700150 shutdown_connections 2023-10-30T14:28:27.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:27.174+0000 7fe5a529c640 1 --2- 172.21.15.57:0/607700150 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5800664a0 0x7fe580068960 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:27.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:27.174+0000 7fe5a529c640 1 --2- 172.21.15.57:0/607700150 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5a0150830 0x7fe5a01577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:27.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:27.174+0000 7fe5a529c640 1 --2- 172.21.15.57:0/607700150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5a00cb7d0 0x7fe5a01ee310 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:27.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:27.174+0000 7fe5a529c640 1 --2- 172.21.15.57:0/607700150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5a00c3e40 0x7fe5a006b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:27.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:27.174+0000 7fe5a529c640 1 -- 172.21.15.57:0/607700150 >> 172.21.15.57:0/607700150 conn(0x7fe5a00cac70 msgr2=0x7fe5a00733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:27.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:27.174+0000 7fe5a529c640 1 -- 172.21.15.57:0/607700150 shutdown_connections 2023-10-30T14:28:27.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:27.174+0000 7fe5a529c640 1 -- 172.21.15.57:0/607700150 wait complete. 2023-10-30T14:28:27.196 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T14:28:27.196 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 19 secs before thrashing 2023-10-30T14:28:35.178 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:28:35.182 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:28:46.197 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:46.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.342+0000 7f1ec449e640 1 Processor -- start 2023-10-30T14:28:46.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.342+0000 7f1ec449e640 1 -- start start 2023-10-30T14:28:46.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.342+0000 7f1ec449e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1ebc0cb7d0 0x7f1ebc0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:46.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.342+0000 7f1ec449e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1ebc0c3e40 0x7f1ebc070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:46.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.342+0000 7f1ec449e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc0c2310 con 0x7f1ebc150830 2023-10-30T14:28:46.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.346+0000 7f1ec449e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ebc06bc10 con 0x7f1ebc0cb7d0 2023-10-30T14:28:46.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.346+0000 7f1ec449e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ebc156710 con 0x7f1ebc0c3e40 2023-10-30T14:28:46.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.346+0000 7f1ec1a39640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1ebc0cb7d0 0x7f1ebc0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:46.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.346+0000 7f1ec1a39640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1ebc0cb7d0 0x7f1ebc0c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35274/0 (socket says 172.21.15.57:35274) 2023-10-30T14:28:46.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.346+0000 7f1ec1a39640 1 -- 172.21.15.57:0/1627289144 learned_addr learned my addr 172.21.15.57:0/1627289144 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:46.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.346+0000 7f1ec223a640 1 --1- 172.21.15.57:0/1627289144 >> v1:172.21.15.57:6789/0 conn(0x7f1ebc150830 0x7f1ebc06b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:35474/0 (socket says 172.21.15.57:35474) 2023-10-30T14:28:46.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.346+0000 7f1ec2a3b640 1 --2- 172.21.15.57:0/1627289144 >> v2:172.21.15.88:3300/0 conn(0x7f1ebc0c3e40 0x7f1ebc070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:46.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.346+0000 7f1ec1a39640 1 -- 172.21.15.57:0/1627289144 >> v2:172.21.15.88:3300/0 conn(0x7f1ebc0c3e40 msgr2=0x7f1ebc070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:46.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.346+0000 7f1ec1a39640 1 --2- 172.21.15.57:0/1627289144 >> v2:172.21.15.88:3300/0 conn(0x7f1ebc0c3e40 0x7f1ebc070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:46.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.346+0000 7f1ec1a39640 1 -- 172.21.15.57:0/1627289144 >> v1:172.21.15.57:6789/0 conn(0x7f1ebc150830 legacy=0x7f1ebc06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:46.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.346+0000 7f1ec1238640 1 -- 172.21.15.57:0/1627289144 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703279688 0 0) 0x7f1ebc0c2310 con 0x7f1ebc150830 2023-10-30T14:28:46.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.346+0000 7f1ec1a39640 1 -- 172.21.15.57:0/1627289144 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc0c4360 con 0x7f1ebc0cb7d0 2023-10-30T14:28:46.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.346+0000 7f1ec1a39640 1 --2- 172.21.15.57:0/1627289144 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ebc0cb7d0 0x7f1ebc0c3900 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc066e10 tx=0x7f1eac01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd40e2c4e90b2d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:46.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.346+0000 7f1ec1238640 1 -- 172.21.15.57:0/1627289144 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac0a89f0 con 0x7f1ebc0cb7d0 2023-10-30T14:28:46.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.346+0000 7f1ec1238640 1 -- 172.21.15.57:0/1627289144 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac0a8b50 con 0x7f1ebc0cb7d0 2023-10-30T14:28:46.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.346+0000 7f1ec1238640 1 -- 172.21.15.57:0/1627289144 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac0b0440 con 0x7f1ebc0cb7d0 2023-10-30T14:28:46.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec449e640 1 -- 172.21.15.57:0/1627289144 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ebc0cb7d0 msgr2=0x7f1ebc0c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:46.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec449e640 1 --2- 172.21.15.57:0/1627289144 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ebc0cb7d0 0x7f1ebc0c3900 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc066e10 tx=0x7f1eac01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:28:46.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec449e640 1 -- 172.21.15.57:0/1627289144 shutdown_connections 2023-10-30T14:28:46.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec449e640 1 --2- 172.21.15.57:0/1627289144 >> v2:172.21.15.88:3300/0 conn(0x7f1ebc0c3e40 0x7f1ebc070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:46.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec449e640 1 --2- 172.21.15.57:0/1627289144 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ebc0cb7d0 0x7f1ebc0c3900 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:46.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec449e640 1 -- 172.21.15.57:0/1627289144 >> 172.21.15.57:0/1627289144 conn(0x7f1ebc0cac70 msgr2=0x7f1ebc0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:46.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec449e640 1 -- 172.21.15.57:0/1627289144 shutdown_connections 2023-10-30T14:28:46.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec449e640 1 -- 172.21.15.57:0/1627289144 wait complete. 2023-10-30T14:28:46.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec449e640 1 Processor -- start 2023-10-30T14:28:46.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec449e640 1 -- start start 2023-10-30T14:28:46.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec449e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ebc0c3e40 0x7f1ebc1596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:46.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec449e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ebc0cb7d0 0x7f1ebc0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:46.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec449e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ebc150830 0x7f1ebc159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:46.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec449e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc152980 con 0x7f1ebc0c3e40 2023-10-30T14:28:46.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec449e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc1ee5e0 con 0x7f1ebc150830 2023-10-30T14:28:46.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec449e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc1eee40 con 0x7f1ebc0cb7d0 2023-10-30T14:28:46.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec223a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ebc0c3e40 0x7f1ebc1596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:46.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec223a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ebc0c3e40 0x7f1ebc1596c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35290/0 (socket says 172.21.15.57:35290) 2023-10-30T14:28:46.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec223a640 1 -- 172.21.15.57:0/1481299555 learned_addr learned my addr 172.21.15.57:0/1481299555 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:46.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec2a3b640 1 --2- 172.21.15.57:0/1481299555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ebc150830 0x7f1ebc159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:46.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.350+0000 7f1ec1a39640 1 --2- 172.21.15.57:0/1481299555 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ebc0cb7d0 0x7f1ebc0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:46.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.354+0000 7f1ec223a640 1 -- 172.21.15.57:0/1481299555 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ebc0cb7d0 msgr2=0x7f1ebc0c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:46.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.354+0000 7f1ec223a640 1 --2- 172.21.15.57:0/1481299555 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ebc0cb7d0 0x7f1ebc0c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:46.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.354+0000 7f1ec223a640 1 -- 172.21.15.57:0/1481299555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ebc150830 msgr2=0x7f1ebc159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:46.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.354+0000 7f1ec223a640 1 --2- 172.21.15.57:0/1481299555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ebc150830 0x7f1ebc159fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:46.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.354+0000 7f1ec223a640 1 -- 172.21.15.57:0/1481299555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eac0094d0 con 0x7f1ebc0c3e40 2023-10-30T14:28:46.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.354+0000 7f1ec223a640 1 --2- 172.21.15.57:0/1481299555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ebc0c3e40 0x7f1ebc1596c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800c4b0 tx=0x7f1eb800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:46.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.354+0000 7f1eb2ffd640 1 -- 172.21.15.57:0/1481299555 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8007480 con 0x7f1ebc0c3e40 2023-10-30T14:28:46.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.354+0000 7f1eb2ffd640 1 -- 172.21.15.57:0/1481299555 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb80075e0 con 0x7f1ebc0c3e40 2023-10-30T14:28:46.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.354+0000 7f1eb2ffd640 1 -- 172.21.15.57:0/1481299555 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8004a30 con 0x7f1ebc0c3e40 2023-10-30T14:28:46.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.354+0000 7f1ec449e640 1 -- 172.21.15.57:0/1481299555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ebc153800 con 0x7f1ebc0c3e40 2023-10-30T14:28:46.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.354+0000 7f1ec449e640 1 -- 172.21.15.57:0/1481299555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ebc153dc0 con 0x7f1ebc0c3e40 2023-10-30T14:28:46.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.354+0000 7f1ec449e640 1 -- 172.21.15.57:0/1481299555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ebc073e30 con 0x7f1ebc0c3e40 2023-10-30T14:28:46.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.354+0000 7f1eb2ffd640 1 -- 172.21.15.57:0/1481299555 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1eb8004b90 con 0x7f1ebc0c3e40 2023-10-30T14:28:46.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.358+0000 7f1eb2ffd640 1 --2- 172.21.15.57:0/1481299555 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1ea00664d0 0x7f1ea0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:46.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.358+0000 7f1eb2ffd640 1 -- 172.21.15.57:0/1481299555 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f1eb8083220 con 0x7f1ebc0c3e40 2023-10-30T14:28:46.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.358+0000 7f1ec1a39640 1 --2- 172.21.15.57:0/1481299555 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1ea00664d0 0x7f1ea0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:46.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.358+0000 7f1ec1a39640 1 --2- 172.21.15.57:0/1481299555 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1ea00664d0 0x7f1ea0068990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1eac0b2000 tx=0x7f1eac009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:46.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.362+0000 7f1eb2ffd640 1 -- 172.21.15.57:0/1481299555 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1eb8050570 con 0x7f1ebc0c3e40 2023-10-30T14:28:46.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.554+0000 7f1ec449e640 1 -- 172.21.15.57:0/1481299555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ebc04e170 con 0x7f1ebc0c3e40 2023-10-30T14:28:46.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.554+0000 7f1eb2ffd640 1 -- 172.21.15.57:0/1481299555 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f1eb8053b90 con 0x7f1ebc0c3e40 2023-10-30T14:28:46.554 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:28:46.554 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66},{"gid":6204,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:29.955079+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5797,"mds_1":6027},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:46.555 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 88 2023-10-30T14:28:46.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.554+0000 7f1ec449e640 1 -- 172.21.15.57:0/1481299555 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1ea00664d0 msgr2=0x7f1ea0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:46.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.554+0000 7f1ec449e640 1 --2- 172.21.15.57:0/1481299555 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1ea00664d0 0x7f1ea0068990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1eac0b2000 tx=0x7f1eac009040 comp rx=0 tx=0).stop 2023-10-30T14:28:46.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.554+0000 7f1ec449e640 1 -- 172.21.15.57:0/1481299555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ebc0c3e40 msgr2=0x7f1ebc1596c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:46.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.554+0000 7f1ec449e640 1 --2- 172.21.15.57:0/1481299555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ebc0c3e40 0x7f1ebc1596c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800c4b0 tx=0x7f1eb800c970 comp rx=0 tx=0).stop 2023-10-30T14:28:46.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.554+0000 7f1ec449e640 1 -- 172.21.15.57:0/1481299555 shutdown_connections 2023-10-30T14:28:46.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.558+0000 7f1ec449e640 1 --2- 172.21.15.57:0/1481299555 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1ea00664d0 0x7f1ea0068990 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:46.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.558+0000 7f1ec449e640 1 --2- 172.21.15.57:0/1481299555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ebc150830 0x7f1ebc159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:46.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.558+0000 7f1ec449e640 1 --2- 172.21.15.57:0/1481299555 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ebc0cb7d0 0x7f1ebc0c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:46.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.558+0000 7f1ec449e640 1 --2- 172.21.15.57:0/1481299555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ebc0c3e40 0x7f1ebc1596c0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:46.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.558+0000 7f1ec449e640 1 -- 172.21.15.57:0/1481299555 >> 172.21.15.57:0/1481299555 conn(0x7f1ebc0cac70 msgr2=0x7f1ebc0733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:46.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.558+0000 7f1ec449e640 1 -- 172.21.15.57:0/1481299555 shutdown_connections 2023-10-30T14:28:46.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.558+0000 7f1ec449e640 1 -- 172.21.15.57:0/1481299555 wait complete. 2023-10-30T14:28:46.578 INFO:tasks.mds_thrash.fs.[cephfs]:thrashing max_mds: 2 -> 3 2023-10-30T14:28:46.578 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-10-30T14:28:46.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.722+0000 7fc3e2765640 1 Processor -- start 2023-10-30T14:28:46.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.722+0000 7fc3e2765640 1 -- start start 2023-10-30T14:28:46.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.726+0000 7fc3e2765640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc3dc0c1430 0x7fc3dc159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:46.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.726+0000 7fc3e2765640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3dc15a040 0x7fc3dc0cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:46.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.726+0000 7fc3e2765640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3dc157100 con 0x7fc3dc150630 2023-10-30T14:28:46.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.726+0000 7fc3e2765640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3dc150a10 con 0x7fc3dc0c1430 2023-10-30T14:28:46.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.726+0000 7fc3e2765640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc072000 con 0x7fc3dc15a040 2023-10-30T14:28:46.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.726+0000 7fc3db7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc3dc0c1430 0x7fc3dc159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:46.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.726+0000 7fc3db7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc3dc0c1430 0x7fc3dc159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35296/0 (socket says 172.21.15.57:35296) 2023-10-30T14:28:46.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.726+0000 7fc3db7fe640 1 -- 172.21.15.57:0/2361200510 learned_addr learned my addr 172.21.15.57:0/2361200510 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:46.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.726+0000 7fc3e0d02640 1 --2- 172.21.15.57:0/2361200510 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3dc15a040 0x7fc3dc0cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:46.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.726+0000 7fc3db7fe640 1 -- 172.21.15.57:0/2361200510 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3dc15a040 msgr2=0x7fc3dc0cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:46.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.726+0000 7fc3db7fe640 1 --2- 172.21.15.57:0/2361200510 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3dc15a040 0x7fc3dc0cacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:46.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.726+0000 7fc3db7fe640 1 -- 172.21.15.57:0/2361200510 >> v1:172.21.15.88:6789/0 conn(0x7fc3dc150630 legacy=0x7fc3dc0c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:46.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.726+0000 7fc3db7fe640 1 -- 172.21.15.57:0/2361200510 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3dc156510 con 0x7fc3dc0c1430 2023-10-30T14:28:46.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.730+0000 7fc3db7fe640 1 --2- 172.21.15.57:0/2361200510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3dc0c1430 0x7fc3dc159b00 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0009870 tx=0x7fc3d001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=263847d7590873a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:46.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.730+0000 7fc3daffd640 1 -- 172.21.15.57:0/2361200510 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d00ab600 con 0x7fc3dc0c1430 2023-10-30T14:28:46.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.730+0000 7fc3daffd640 1 -- 172.21.15.57:0/2361200510 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d00a7050 con 0x7fc3dc0c1430 2023-10-30T14:28:46.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.730+0000 7fc3daffd640 1 -- 172.21.15.57:0/2361200510 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d00ab980 con 0x7fc3dc0c1430 2023-10-30T14:28:46.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.730+0000 7fc3e2765640 1 -- 172.21.15.57:0/2361200510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3dc0c1430 msgr2=0x7fc3dc159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:46.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.730+0000 7fc3e2765640 1 --2- 172.21.15.57:0/2361200510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3dc0c1430 0x7fc3dc159b00 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0009870 tx=0x7fc3d001c540 comp rx=0 tx=0).stop 2023-10-30T14:28:46.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.730+0000 7fc3e2765640 1 -- 172.21.15.57:0/2361200510 shutdown_connections 2023-10-30T14:28:46.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.730+0000 7fc3e2765640 1 --2- 172.21.15.57:0/2361200510 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3dc15a040 0x7fc3dc0cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:46.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.730+0000 7fc3e2765640 1 --2- 172.21.15.57:0/2361200510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3dc0c1430 0x7fc3dc159b00 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:46.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.730+0000 7fc3e2765640 1 -- 172.21.15.57:0/2361200510 >> 172.21.15.57:0/2361200510 conn(0x7fc3dc06da40 msgr2=0x7fc3dc06de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:46.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.730+0000 7fc3e2765640 1 -- 172.21.15.57:0/2361200510 shutdown_connections 2023-10-30T14:28:46.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.730+0000 7fc3e2765640 1 -- 172.21.15.57:0/2361200510 wait complete. 2023-10-30T14:28:46.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.730+0000 7fc3e2765640 1 Processor -- start 2023-10-30T14:28:46.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.730+0000 7fc3e2765640 1 -- start start 2023-10-30T14:28:46.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3e2765640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3dc0c1430 0x7fc3dc159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:46.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3e2765640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3dc150630 0x7fc3dc0c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:46.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3e2765640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3dc15a040 0x7fc3dc1f5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:46.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3e2765640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc159bd0 con 0x7fc3dc15a040 2023-10-30T14:28:46.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3e2765640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc1583c0 con 0x7fc3dc150630 2023-10-30T14:28:46.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3e2765640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc1591f0 con 0x7fc3dc0c1430 2023-10-30T14:28:46.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3e0d02640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3dc15a040 0x7fc3dc1f5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:46.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3e0d02640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3dc15a040 0x7fc3dc1f5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35304/0 (socket says 172.21.15.57:35304) 2023-10-30T14:28:46.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3e0d02640 1 -- 172.21.15.57:0/1699834125 learned_addr learned my addr 172.21.15.57:0/1699834125 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:46.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3dbfff640 1 --2- 172.21.15.57:0/1699834125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3dc0c1430 0x7fc3dc159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:46.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3db7fe640 1 --2- 172.21.15.57:0/1699834125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3dc150630 0x7fc3dc0c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:46.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3e0d02640 1 -- 172.21.15.57:0/1699834125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3dc0c1430 msgr2=0x7fc3dc159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:46.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3e0d02640 1 --2- 172.21.15.57:0/1699834125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3dc0c1430 0x7fc3dc159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:46.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3e0d02640 1 -- 172.21.15.57:0/1699834125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3dc150630 msgr2=0x7fc3dc0c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:46.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3e0d02640 1 --2- 172.21.15.57:0/1699834125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3dc150630 0x7fc3dc0c9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:46.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3e0d02640 1 -- 172.21.15.57:0/1699834125 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d00094d0 con 0x7fc3dc15a040 2023-10-30T14:28:46.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3dbfff640 1 --2- 172.21.15.57:0/1699834125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3dc0c1430 0x7fc3dc159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:28:46.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3e0d02640 1 --2- 172.21.15.57:0/1699834125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3dc15a040 0x7fc3dc1f5620 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4009d30 tx=0x7fc3d400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:46.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3d8ff9640 1 -- 172.21.15.57:0/1699834125 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4009ba0 con 0x7fc3dc15a040 2023-10-30T14:28:46.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3d8ff9640 1 -- 172.21.15.57:0/1699834125 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d400f040 con 0x7fc3dc15a040 2023-10-30T14:28:46.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3d8ff9640 1 -- 172.21.15.57:0/1699834125 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d40146c0 con 0x7fc3dc15a040 2023-10-30T14:28:46.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3e2765640 1 -- 172.21.15.57:0/1699834125 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3dc160aa0 con 0x7fc3dc15a040 2023-10-30T14:28:46.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3e2765640 1 -- 172.21.15.57:0/1699834125 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3dc160170 con 0x7fc3dc15a040 2023-10-30T14:28:46.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3e2765640 1 -- 172.21.15.57:0/1699834125 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3dc0c5350 con 0x7fc3dc15a040 2023-10-30T14:28:46.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.734+0000 7fc3d8ff9640 1 -- 172.21.15.57:0/1699834125 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc3d4014820 con 0x7fc3dc15a040 2023-10-30T14:28:46.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.738+0000 7fc3d8ff9640 1 --2- 172.21.15.57:0/1699834125 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc3c80664d0 0x7fc3c8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:46.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.738+0000 7fc3d8ff9640 1 -- 172.21.15.57:0/1699834125 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7fc3d4084a50 con 0x7fc3dc15a040 2023-10-30T14:28:46.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.738+0000 7fc3dbfff640 1 --2- 172.21.15.57:0/1699834125 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc3c80664d0 0x7fc3c8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:46.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.738+0000 7fc3dbfff640 1 --2- 172.21.15.57:0/1699834125 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc3c80664d0 0x7fc3c8068990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc000f70 tx=0x7fc3cc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:46.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.742+0000 7fc3d8ff9640 1 -- 172.21.15.57:0/1699834125 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc3d4051b90 con 0x7fc3dc15a040 2023-10-30T14:28:46.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:46.954+0000 7fc3e2765640 1 -- 172.21.15.57:0/1699834125 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7fc3dc160aa0 con 0x7fc3dc15a040 2023-10-30T14:28:47.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:47.906+0000 7fc3d8ff9640 1 -- 172.21.15.57:0/1699834125 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v89) v1 ==== 105+0+0 (secure 0 0 0) 0x7fc3dc160aa0 con 0x7fc3dc15a040 2023-10-30T14:28:47.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:47.906+0000 7fc3e2765640 1 -- 172.21.15.57:0/1699834125 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc3c80664d0 msgr2=0x7fc3c8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:47.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:47.906+0000 7fc3e2765640 1 --2- 172.21.15.57:0/1699834125 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc3c80664d0 0x7fc3c8068990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc000f70 tx=0x7fc3cc009040 comp rx=0 tx=0).stop 2023-10-30T14:28:47.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:47.906+0000 7fc3e2765640 1 -- 172.21.15.57:0/1699834125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3dc15a040 msgr2=0x7fc3dc1f5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:47.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:47.906+0000 7fc3e2765640 1 --2- 172.21.15.57:0/1699834125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3dc15a040 0x7fc3dc1f5620 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4009d30 tx=0x7fc3d400c490 comp rx=0 tx=0).stop 2023-10-30T14:28:47.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:47.906+0000 7fc3e2765640 1 -- 172.21.15.57:0/1699834125 shutdown_connections 2023-10-30T14:28:47.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:47.906+0000 7fc3e2765640 1 --2- 172.21.15.57:0/1699834125 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc3c80664d0 0x7fc3c8068990 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:47.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:47.906+0000 7fc3e2765640 1 --2- 172.21.15.57:0/1699834125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3dc15a040 0x7fc3dc1f5620 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:47.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:47.906+0000 7fc3e2765640 1 --2- 172.21.15.57:0/1699834125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3dc150630 0x7fc3dc0c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:47.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:47.906+0000 7fc3e2765640 1 --2- 172.21.15.57:0/1699834125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3dc0c1430 0x7fc3dc159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:47.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:47.906+0000 7fc3e2765640 1 -- 172.21.15.57:0/1699834125 >> 172.21.15.57:0/1699834125 conn(0x7fc3dc06da40 msgr2=0x7fc3dc070780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:47.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:47.906+0000 7fc3e2765640 1 -- 172.21.15.57:0/1699834125 shutdown_connections 2023-10-30T14:28:47.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:47.906+0000 7fc3e2765640 1 -- 172.21.15.57:0/1699834125 wait complete. 2023-10-30T14:28:47.928 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:28:47.928 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:48.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.074+0000 7f86981aa640 1 Processor -- start 2023-10-30T14:28:48.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.074+0000 7f86981aa640 1 -- start start 2023-10-30T14:28:48.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.078+0000 7f86981aa640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8690153c80 0x7f869015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:48.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.078+0000 7f86981aa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f86900c42a0 0x7f86900ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:48.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.078+0000 7f86981aa640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8690159f20 con 0x7f86900c4a10 2023-10-30T14:28:48.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.078+0000 7f86981aa640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f869015dec0 con 0x7f8690153c80 2023-10-30T14:28:48.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.078+0000 7f86981aa640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86901599a0 con 0x7f86900c42a0 2023-10-30T14:28:48.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.078+0000 7f8695f46640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8690153c80 0x7f869015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:48.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.078+0000 7f8695f46640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8690153c80 0x7f869015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35320/0 (socket says 172.21.15.57:35320) 2023-10-30T14:28:48.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.078+0000 7f8695f46640 1 -- 172.21.15.57:0/2688981052 learned_addr learned my addr 172.21.15.57:0/2688981052 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:48.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.078+0000 7f8695745640 1 --1- 172.21.15.57:0/2688981052 >> v1:172.21.15.57:6789/0 conn(0x7f86900c4a10 0x7f869015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:35478/0 (socket says 172.21.15.57:35478) 2023-10-30T14:28:48.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.078+0000 7f8696747640 1 --2- 172.21.15.57:0/2688981052 >> v2:172.21.15.88:3300/0 conn(0x7f86900c42a0 0x7f86900ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:48.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.078+0000 7f8694f44640 1 -- 172.21.15.57:0/2688981052 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30425235 0 0) 0x7f8690159f20 con 0x7f86900c4a10 2023-10-30T14:28:48.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.078+0000 7f8694f44640 1 -- 172.21.15.57:0/2688981052 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8678003530 con 0x7f86900c4a10 2023-10-30T14:28:48.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.078+0000 7f8695f46640 1 -- 172.21.15.57:0/2688981052 >> v2:172.21.15.88:3300/0 conn(0x7f86900c42a0 msgr2=0x7f86900ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:48.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.078+0000 7f8695f46640 1 --2- 172.21.15.57:0/2688981052 >> v2:172.21.15.88:3300/0 conn(0x7f86900c42a0 0x7f86900ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:48.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.078+0000 7f8695f46640 1 -- 172.21.15.57:0/2688981052 >> v1:172.21.15.57:6789/0 conn(0x7f86900c4a10 legacy=0x7f869015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:48.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.078+0000 7f8695f46640 1 -- 172.21.15.57:0/2688981052 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869015d620 con 0x7f8690153c80 2023-10-30T14:28:48.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.078+0000 7f8696747640 1 --2- 172.21.15.57:0/2688981052 >> v2:172.21.15.88:3300/0 conn(0x7f86900c42a0 0x7f86900ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:28:48.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.082+0000 7f8695f46640 1 --2- 172.21.15.57:0/2688981052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8690153c80 0x7f869015cf90 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f86900b1d40 tx=0x7f868001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e78f6ba35392252 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:48.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.082+0000 7f8694f44640 1 -- 172.21.15.57:0/2688981052 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86800af400 con 0x7f8690153c80 2023-10-30T14:28:48.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.082+0000 7f8694f44640 1 -- 172.21.15.57:0/2688981052 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86800ab050 con 0x7f8690153c80 2023-10-30T14:28:48.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.082+0000 7f8694f44640 1 -- 172.21.15.57:0/2688981052 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86800af780 con 0x7f8690153c80 2023-10-30T14:28:48.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.082+0000 7f86981aa640 1 -- 172.21.15.57:0/2688981052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8690153c80 msgr2=0x7f869015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:48.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.082+0000 7f86981aa640 1 --2- 172.21.15.57:0/2688981052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8690153c80 0x7f869015cf90 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f86900b1d40 tx=0x7f868001f1c0 comp rx=0 tx=0).stop 2023-10-30T14:28:48.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.082+0000 7f86981aa640 1 -- 172.21.15.57:0/2688981052 shutdown_connections 2023-10-30T14:28:48.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.082+0000 7f86981aa640 1 --2- 172.21.15.57:0/2688981052 >> v2:172.21.15.88:3300/0 conn(0x7f86900c42a0 0x7f86900ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:48.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.082+0000 7f86981aa640 1 --2- 172.21.15.57:0/2688981052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8690153c80 0x7f869015cf90 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:48.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.082+0000 7f86981aa640 1 -- 172.21.15.57:0/2688981052 >> 172.21.15.57:0/2688981052 conn(0x7f86900715c0 msgr2=0x7f86900719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:48.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.086+0000 7f86981aa640 1 -- 172.21.15.57:0/2688981052 shutdown_connections 2023-10-30T14:28:48.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.086+0000 7f86981aa640 1 -- 172.21.15.57:0/2688981052 wait complete. 2023-10-30T14:28:48.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.086+0000 7f86981aa640 1 Processor -- start 2023-10-30T14:28:48.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.086+0000 7f86981aa640 1 -- start start 2023-10-30T14:28:48.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.086+0000 7f86981aa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86900c42a0 0x7f869014fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:48.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.086+0000 7f86981aa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86900c4a10 0x7f8690154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:48.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.086+0000 7f86981aa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8690153c80 0x7f869014df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:48.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.086+0000 7f86981aa640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8690157340 con 0x7f86900c42a0 2023-10-30T14:28:48.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.086+0000 7f86981aa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86901efb20 con 0x7f8690153c80 2023-10-30T14:28:48.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.086+0000 7f86981aa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86900b5530 con 0x7f86900c4a10 2023-10-30T14:28:48.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.090+0000 7f8695f46640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86900c42a0 0x7f869014fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:48.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.090+0000 7f8695f46640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86900c42a0 0x7f869014fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35322/0 (socket says 172.21.15.57:35322) 2023-10-30T14:28:48.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.090+0000 7f8695f46640 1 -- 172.21.15.57:0/1372601689 learned_addr learned my addr 172.21.15.57:0/1372601689 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:48.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.090+0000 7f8695f46640 1 -- 172.21.15.57:0/1372601689 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86900c4a10 msgr2=0x7f8690154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:48.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.090+0000 7f8695745640 1 --2- 172.21.15.57:0/1372601689 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86900c4a10 0x7f8690154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:48.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.090+0000 7f8695f46640 1 --2- 172.21.15.57:0/1372601689 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86900c4a10 0x7f8690154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:48.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.090+0000 7f8695f46640 1 -- 172.21.15.57:0/1372601689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8690153c80 msgr2=0x7f869014df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:48.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.090+0000 7f8695f46640 1 --2- 172.21.15.57:0/1372601689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8690153c80 0x7f869014df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:48.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.090+0000 7f8695f46640 1 -- 172.21.15.57:0/1372601689 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8688009420 con 0x7f86900c42a0 2023-10-30T14:28:48.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.090+0000 7f8695745640 1 --2- 172.21.15.57:0/1372601689 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86900c4a10 0x7f8690154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:28:48.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.090+0000 7f8695f46640 1 --2- 172.21.15.57:0/1372601689 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86900c42a0 0x7f869014fdf0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f86800b2040 tx=0x7f868001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:48.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.090+0000 7f868e7fc640 1 -- 172.21.15.57:0/1372601689 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86800c0070 con 0x7f86900c42a0 2023-10-30T14:28:48.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.090+0000 7f868e7fc640 1 -- 172.21.15.57:0/1372601689 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8680009510 con 0x7f86900c42a0 2023-10-30T14:28:48.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.090+0000 7f868e7fc640 1 -- 172.21.15.57:0/1372601689 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86800afd40 con 0x7f86900c42a0 2023-10-30T14:28:48.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.090+0000 7f86981aa640 1 -- 172.21.15.57:0/1372601689 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8680009b30 con 0x7f86900c42a0 2023-10-30T14:28:48.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.090+0000 7f86981aa640 1 -- 172.21.15.57:0/1372601689 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8690157060 con 0x7f86900c42a0 2023-10-30T14:28:48.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.090+0000 7f86981aa640 1 -- 172.21.15.57:0/1372601689 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869004e170 con 0x7f86900c42a0 2023-10-30T14:28:48.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.094+0000 7f868e7fc640 1 -- 172.21.15.57:0/1372601689 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f86800b6040 con 0x7f86900c42a0 2023-10-30T14:28:48.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.094+0000 7f868e7fc640 1 --2- 172.21.15.57:0/1372601689 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f867c0664d0 0x7f867c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:48.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.094+0000 7f868e7fc640 1 -- 172.21.15.57:0/1372601689 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f86801215e0 con 0x7f86900c42a0 2023-10-30T14:28:48.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.094+0000 7f8695745640 1 --2- 172.21.15.57:0/1372601689 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f867c0664d0 0x7f867c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:48.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.094+0000 7f8695745640 1 --2- 172.21.15.57:0/1372601689 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f867c0664d0 0x7f867c068990 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f86900cb2a0 tx=0x7f8684009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:48.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.098+0000 7f868e7fc640 1 -- 172.21.15.57:0/1372601689 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f86800ee830 con 0x7f86900c42a0 2023-10-30T14:28:48.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.306+0000 7f86981aa640 1 -- 172.21.15.57:0/1372601689 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86900c5ef0 con 0x7f86900c42a0 2023-10-30T14:28:48.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.306+0000 7f868e7fc640 1 -- 172.21.15.57:0/1372601689 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+6526 (secure 0 0 0) 0x7f86800f1e50 con 0x7f86900c42a0 2023-10-30T14:28:48.306 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:28:48.306 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:47.910869+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5797,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:starting","state_seq":1,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:48.307 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 90 2023-10-30T14:28:48.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.306+0000 7f86981aa640 1 -- 172.21.15.57:0/1372601689 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f867c0664d0 msgr2=0x7f867c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:48.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.306+0000 7f86981aa640 1 --2- 172.21.15.57:0/1372601689 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f867c0664d0 0x7f867c068990 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f86900cb2a0 tx=0x7f8684009040 comp rx=0 tx=0).stop 2023-10-30T14:28:48.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.306+0000 7f86981aa640 1 -- 172.21.15.57:0/1372601689 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86900c42a0 msgr2=0x7f869014fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:48.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.306+0000 7f86981aa640 1 --2- 172.21.15.57:0/1372601689 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86900c42a0 0x7f869014fdf0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f86800b2040 tx=0x7f868001bf60 comp rx=0 tx=0).stop 2023-10-30T14:28:48.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.306+0000 7f86981aa640 1 -- 172.21.15.57:0/1372601689 shutdown_connections 2023-10-30T14:28:48.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.306+0000 7f86981aa640 1 --2- 172.21.15.57:0/1372601689 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f867c0664d0 0x7f867c068990 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:48.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.306+0000 7f86981aa640 1 --2- 172.21.15.57:0/1372601689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8690153c80 0x7f869014df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:48.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.306+0000 7f86981aa640 1 --2- 172.21.15.57:0/1372601689 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86900c4a10 0x7f8690154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:48.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.306+0000 7f86981aa640 1 --2- 172.21.15.57:0/1372601689 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86900c42a0 0x7f869014fdf0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:48.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.306+0000 7f86981aa640 1 -- 172.21.15.57:0/1372601689 >> 172.21.15.57:0/1372601689 conn(0x7f86900715c0 msgr2=0x7f86900c4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:48.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.306+0000 7f86981aa640 1 -- 172.21.15.57:0/1372601689 shutdown_connections 2023-10-30T14:28:48.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:48.306+0000 7f86981aa640 1 -- 172.21.15.57:0/1372601689 wait complete. 2023-10-30T14:28:48.328 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 90, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 90, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5797": { "addr": "172.21.15.57:6834/2940934016", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2940934016, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2940934016, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5797, "incarnation": 73, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 43 }, "gid_6027": { "addr": "172.21.15.57:6836/2733323524", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2733323524, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2733323524, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6027, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 24 }, "gid_6204": { "addr": "172.21.15.57:6837/1843528309", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 1843528309, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 1843528309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6204, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:starting", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:28:47.910869+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5797, "mds_1": 6027, "mds_2": 6204 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6839/358521496", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 358521496, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 358521496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 66, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5315, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:28:50.329 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:50.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.478+0000 7fe1d06c9640 1 Processor -- start 2023-10-30T14:28:50.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.478+0000 7fe1d06c9640 1 -- start start 2023-10-30T14:28:50.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.478+0000 7fe1d06c9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe1c8151c20 0x7fe1c80c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:50.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.478+0000 7fe1d06c9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c80c5030 con 0x7fe1c8151840 2023-10-30T14:28:50.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.478+0000 7fe1d06c9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c8157830 con 0x7fe1c80cb8b0 2023-10-30T14:28:50.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.478+0000 7fe1d06c9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c8157b10 con 0x7fe1c8151c20 2023-10-30T14:28:50.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.478+0000 7fe1cdc64640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe1c8151c20 0x7fe1c80c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:50.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.478+0000 7fe1ce465640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fe1c8151840 0x7fe1c8072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:35488/0 (socket says 172.21.15.57:35488) 2023-10-30T14:28:50.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.478+0000 7fe1ce465640 1 -- 172.21.15.57:0/2668212988 learned_addr learned my addr 172.21.15.57:0/2668212988 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:28:50.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.478+0000 7fe1cdc64640 1 -- 172.21.15.57:0/2668212988 >> v1:172.21.15.88:6789/0 conn(0x7fe1c80cb8b0 legacy=0x7fe1c806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:50.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.478+0000 7fe1cdc64640 1 -- 172.21.15.57:0/2668212988 >> v1:172.21.15.57:6789/0 conn(0x7fe1c8151840 legacy=0x7fe1c8072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:50.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.478+0000 7fe1cdc64640 1 -- 172.21.15.57:0/2668212988 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c8157df0 con 0x7fe1c8151c20 2023-10-30T14:28:50.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.482+0000 7fe1cdc64640 1 --2- 172.21.15.57:0/2668212988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1c8151c20 0x7fe1c80c49a0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fe1b80097c0 tx=0x7fe1b801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e27125135ad6dd4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:50.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.482+0000 7fe1cd463640 1 -- 172.21.15.57:0/2668212988 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b80a8970 con 0x7fe1c8151c20 2023-10-30T14:28:50.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.482+0000 7fe1cd463640 1 -- 172.21.15.57:0/2668212988 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b80a8ad0 con 0x7fe1c8151c20 2023-10-30T14:28:50.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.482+0000 7fe1cd463640 1 -- 172.21.15.57:0/2668212988 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b80a8e60 con 0x7fe1c8151c20 2023-10-30T14:28:50.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.482+0000 7fe1d06c9640 1 -- 172.21.15.57:0/2668212988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1c8151c20 msgr2=0x7fe1c80c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:50.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.482+0000 7fe1d06c9640 1 --2- 172.21.15.57:0/2668212988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1c8151c20 0x7fe1c80c49a0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fe1b80097c0 tx=0x7fe1b801ec70 comp rx=0 tx=0).stop 2023-10-30T14:28:50.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.482+0000 7fe1d06c9640 1 -- 172.21.15.57:0/2668212988 shutdown_connections 2023-10-30T14:28:50.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.482+0000 7fe1d06c9640 1 --2- 172.21.15.57:0/2668212988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1c8151c20 0x7fe1c80c49a0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:50.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.482+0000 7fe1d06c9640 1 -- 172.21.15.57:0/2668212988 >> 172.21.15.57:0/2668212988 conn(0x7fe1c80cac70 msgr2=0x7fe1c80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:50.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.482+0000 7fe1d06c9640 1 -- 172.21.15.57:0/2668212988 shutdown_connections 2023-10-30T14:28:50.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.482+0000 7fe1d06c9640 1 -- 172.21.15.57:0/2668212988 wait complete. 2023-10-30T14:28:50.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1d06c9640 1 Processor -- start 2023-10-30T14:28:50.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1d06c9640 1 -- start start 2023-10-30T14:28:50.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1d06c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1c80cb8b0 0x7fe1c8070250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:50.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1d06c9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c8151840 0x7fe1c814c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:50.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1d06c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1c8151c20 0x7fe1c814cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:50.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1d06c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c81547f0 con 0x7fe1c80cb8b0 2023-10-30T14:28:50.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1d06c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c81529d0 con 0x7fe1c8151c20 2023-10-30T14:28:50.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1d06c9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c80c9bd0 con 0x7fe1c8151840 2023-10-30T14:28:50.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1ce465640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1c80cb8b0 0x7fe1c8070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:50.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1ce465640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1c80cb8b0 0x7fe1c8070250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35340/0 (socket says 172.21.15.57:35340) 2023-10-30T14:28:50.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1ce465640 1 -- 172.21.15.57:0/1816186436 learned_addr learned my addr 172.21.15.57:0/1816186436 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:50.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1cec66640 1 --2- 172.21.15.57:0/1816186436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1c8151c20 0x7fe1c814cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:50.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1cdc64640 1 --2- 172.21.15.57:0/1816186436 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c8151840 0x7fe1c814c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:50.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1ce465640 1 -- 172.21.15.57:0/1816186436 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c8151840 msgr2=0x7fe1c814c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:50.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1ce465640 1 --2- 172.21.15.57:0/1816186436 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c8151840 0x7fe1c814c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:50.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1ce465640 1 -- 172.21.15.57:0/1816186436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1c8151c20 msgr2=0x7fe1c814cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:50.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1ce465640 1 --2- 172.21.15.57:0/1816186436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1c8151c20 0x7fe1c814cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:50.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1ce465640 1 -- 172.21.15.57:0/1816186436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b8009420 con 0x7fe1c80cb8b0 2023-10-30T14:28:50.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1cec66640 1 --2- 172.21.15.57:0/1816186436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1c8151c20 0x7fe1c814cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:28:50.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1cdc64640 1 --2- 172.21.15.57:0/1816186436 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c8151840 0x7fe1c814c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:28:50.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1ce465640 1 --2- 172.21.15.57:0/1816186436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1c80cb8b0 0x7fe1c8070250 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe1c400c500 tx=0x7fe1c400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:50.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1beffd640 1 -- 172.21.15.57:0/1816186436 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c4005cf0 con 0x7fe1c80cb8b0 2023-10-30T14:28:50.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1beffd640 1 -- 172.21.15.57:0/1816186436 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c4005e50 con 0x7fe1c80cb8b0 2023-10-30T14:28:50.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1beffd640 1 -- 172.21.15.57:0/1816186436 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c40072f0 con 0x7fe1c80cb8b0 2023-10-30T14:28:50.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1d06c9640 1 -- 172.21.15.57:0/1816186436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c8154d10 con 0x7fe1c80cb8b0 2023-10-30T14:28:50.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1d06c9640 1 -- 172.21.15.57:0/1816186436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c80ca100 con 0x7fe1c80cb8b0 2023-10-30T14:28:50.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.486+0000 7fe1d06c9640 1 -- 172.21.15.57:0/1816186436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c804e170 con 0x7fe1c80cb8b0 2023-10-30T14:28:50.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.490+0000 7fe1beffd640 1 -- 172.21.15.57:0/1816186436 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe1c4007450 con 0x7fe1c80cb8b0 2023-10-30T14:28:50.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.490+0000 7fe1beffd640 1 --2- 172.21.15.57:0/1816186436 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe1ac0664d0 0x7fe1ac068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:50.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.490+0000 7fe1beffd640 1 -- 172.21.15.57:0/1816186436 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7fe1c40836c0 con 0x7fe1c80cb8b0 2023-10-30T14:28:50.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.490+0000 7fe1cdc64640 1 --2- 172.21.15.57:0/1816186436 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe1ac0664d0 0x7fe1ac068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:50.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.490+0000 7fe1cdc64640 1 --2- 172.21.15.57:0/1816186436 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe1ac0664d0 0x7fe1ac068990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe1b80b2000 tx=0x7fe1b80aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:50.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.494+0000 7fe1beffd640 1 -- 172.21.15.57:0/1816186436 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe1c40508b0 con 0x7fe1c80cb8b0 2023-10-30T14:28:50.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.690+0000 7fe1d06c9640 1 -- 172.21.15.57:0/1816186436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1c814c4d0 con 0x7fe1c80cb8b0 2023-10-30T14:28:50.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.690+0000 7fe1beffd640 1 -- 172.21.15.57:0/1816186436 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+6526 (secure 0 0 0) 0x7fe1c4053ed0 con 0x7fe1c80cb8b0 2023-10-30T14:28:50.693 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:28:50.693 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:47.910869+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5797,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:starting","state_seq":1,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:50.693 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 90 2023-10-30T14:28:50.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.694+0000 7fe1d06c9640 1 -- 172.21.15.57:0/1816186436 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe1ac0664d0 msgr2=0x7fe1ac068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:50.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.694+0000 7fe1d06c9640 1 --2- 172.21.15.57:0/1816186436 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe1ac0664d0 0x7fe1ac068990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe1b80b2000 tx=0x7fe1b80aa070 comp rx=0 tx=0).stop 2023-10-30T14:28:50.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.694+0000 7fe1d06c9640 1 -- 172.21.15.57:0/1816186436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1c80cb8b0 msgr2=0x7fe1c8070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:50.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.694+0000 7fe1d06c9640 1 --2- 172.21.15.57:0/1816186436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1c80cb8b0 0x7fe1c8070250 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe1c400c500 tx=0x7fe1c400c9c0 comp rx=0 tx=0).stop 2023-10-30T14:28:50.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.694+0000 7fe1d06c9640 1 -- 172.21.15.57:0/1816186436 shutdown_connections 2023-10-30T14:28:50.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.694+0000 7fe1d06c9640 1 --2- 172.21.15.57:0/1816186436 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe1ac0664d0 0x7fe1ac068990 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:50.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.694+0000 7fe1d06c9640 1 --2- 172.21.15.57:0/1816186436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1c8151c20 0x7fe1c814cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:50.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.694+0000 7fe1d06c9640 1 --2- 172.21.15.57:0/1816186436 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c8151840 0x7fe1c814c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:50.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.694+0000 7fe1d06c9640 1 --2- 172.21.15.57:0/1816186436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1c80cb8b0 0x7fe1c8070250 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:50.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.694+0000 7fe1d06c9640 1 -- 172.21.15.57:0/1816186436 >> 172.21.15.57:0/1816186436 conn(0x7fe1c80cac70 msgr2=0x7fe1c80cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:50.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.694+0000 7fe1d06c9640 1 -- 172.21.15.57:0/1816186436 shutdown_connections 2023-10-30T14:28:50.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:50.694+0000 7fe1d06c9640 1 -- 172.21.15.57:0/1816186436 wait complete. 2023-10-30T14:28:50.713 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:28:52.714 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:52.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.882+0000 7f3f31777640 1 Processor -- start 2023-10-30T14:28:52.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.882+0000 7f3f31777640 1 -- start start 2023-10-30T14:28:52.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.886+0000 7f3f31777640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3f2c0c62b0 0x7f3f2c1597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:52.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.886+0000 7f3f31777640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3f2c0cb180 0x7f3f2c06ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:52.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.886+0000 7f3f31777640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2c15a770 con 0x7f3f2c153b50 2023-10-30T14:28:52.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.886+0000 7f3f31777640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f2c159cf0 con 0x7f3f2c0c62b0 2023-10-30T14:28:52.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.886+0000 7f3f31777640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f2c06aa30 con 0x7f3f2c0cb180 2023-10-30T14:28:52.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.886+0000 7f3f23fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3f2c0c62b0 0x7f3f2c1597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:52.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.886+0000 7f3f23fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3f2c0c62b0 0x7f3f2c1597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35350/0 (socket says 172.21.15.57:35350) 2023-10-30T14:28:52.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.886+0000 7f3f23fff640 1 -- 172.21.15.57:0/1424188264 learned_addr learned my addr 172.21.15.57:0/1424188264 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:52.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.886+0000 7f3f30f76640 1 --2- 172.21.15.57:0/1424188264 >> v2:172.21.15.88:3300/0 conn(0x7f3f2c0cb180 0x7f3f2c06ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:52.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.886+0000 7f3f23fff640 1 -- 172.21.15.57:0/1424188264 >> v2:172.21.15.88:3300/0 conn(0x7f3f2c0cb180 msgr2=0x7f3f2c06ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:52.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.886+0000 7f3f23fff640 1 --2- 172.21.15.57:0/1424188264 >> v2:172.21.15.88:3300/0 conn(0x7f3f2c0cb180 0x7f3f2c06ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:52.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.886+0000 7f3f23fff640 1 -- 172.21.15.57:0/1424188264 >> v1:172.21.15.88:6789/0 conn(0x7f3f2c153b50 legacy=0x7f3f2c0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:52.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.886+0000 7f3f23fff640 1 -- 172.21.15.57:0/1424188264 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2c0c9d40 con 0x7f3f2c0c62b0 2023-10-30T14:28:52.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.890+0000 7f3f23fff640 1 --2- 172.21.15.57:0/1424188264 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c0c62b0 0x7f3f2c1597b0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c0c5d20 tx=0x7f3f1401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96dff8d2b9144c4c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:52.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.890+0000 7f3f2b7fe640 1 -- 172.21.15.57:0/1424188264 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f140a89f0 con 0x7f3f2c0c62b0 2023-10-30T14:28:52.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.890+0000 7f3f2b7fe640 1 -- 172.21.15.57:0/1424188264 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f140a8b50 con 0x7f3f2c0c62b0 2023-10-30T14:28:52.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.890+0000 7f3f2b7fe640 1 -- 172.21.15.57:0/1424188264 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f140b0440 con 0x7f3f2c0c62b0 2023-10-30T14:28:52.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.890+0000 7f3f31777640 1 -- 172.21.15.57:0/1424188264 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c0c62b0 msgr2=0x7f3f2c1597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:52.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.890+0000 7f3f31777640 1 --2- 172.21.15.57:0/1424188264 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c0c62b0 0x7f3f2c1597b0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c0c5d20 tx=0x7f3f1401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:28:52.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.890+0000 7f3f31777640 1 -- 172.21.15.57:0/1424188264 shutdown_connections 2023-10-30T14:28:52.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.890+0000 7f3f31777640 1 --2- 172.21.15.57:0/1424188264 >> v2:172.21.15.88:3300/0 conn(0x7f3f2c0cb180 0x7f3f2c06ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:52.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.890+0000 7f3f31777640 1 --2- 172.21.15.57:0/1424188264 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c0c62b0 0x7f3f2c1597b0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:52.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.890+0000 7f3f31777640 1 -- 172.21.15.57:0/1424188264 >> 172.21.15.57:0/1424188264 conn(0x7f3f2c065a80 msgr2=0x7f3f2c065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:52.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.890+0000 7f3f31777640 1 -- 172.21.15.57:0/1424188264 shutdown_connections 2023-10-30T14:28:52.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.890+0000 7f3f31777640 1 -- 172.21.15.57:0/1424188264 wait complete. 2023-10-30T14:28:52.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f31777640 1 Processor -- start 2023-10-30T14:28:52.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f31777640 1 -- start start 2023-10-30T14:28:52.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f31777640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3f2c0c62b0 0x7f3f2c14c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:52.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f31777640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c0cb180 0x7f3f2c1effa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:52.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f31777640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2c153b50 0x7f3f2c149390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:52.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f31777640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c158630 con 0x7f3f2c0cb180 2023-10-30T14:28:52.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f31777640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c157620 con 0x7f3f2c153b50 2023-10-30T14:28:52.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f31777640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c1581a0 con 0x7f3f2c0c62b0 2023-10-30T14:28:52.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f23fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c0cb180 0x7f3f2c1effa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:52.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f23fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c0cb180 0x7f3f2c1effa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35366/0 (socket says 172.21.15.57:35366) 2023-10-30T14:28:52.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f23fff640 1 -- 172.21.15.57:0/3906893544 learned_addr learned my addr 172.21.15.57:0/3906893544 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:52.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f30f76640 1 --2- 172.21.15.57:0/3906893544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2c153b50 0x7f3f2c149390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:52.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f2bfff640 1 --2- 172.21.15.57:0/3906893544 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3f2c0c62b0 0x7f3f2c14c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:52.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f23fff640 1 -- 172.21.15.57:0/3906893544 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3f2c0c62b0 msgr2=0x7f3f2c14c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:52.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f23fff640 1 --2- 172.21.15.57:0/3906893544 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3f2c0c62b0 0x7f3f2c14c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:52.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f23fff640 1 -- 172.21.15.57:0/3906893544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2c153b50 msgr2=0x7f3f2c149390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:52.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f23fff640 1 --2- 172.21.15.57:0/3906893544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2c153b50 0x7f3f2c149390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:52.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f23fff640 1 -- 172.21.15.57:0/3906893544 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f140094d0 con 0x7f3f2c0cb180 2023-10-30T14:28:52.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f30f76640 1 --2- 172.21.15.57:0/3906893544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2c153b50 0x7f3f2c149390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:28:52.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f23fff640 1 --2- 172.21.15.57:0/3906893544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c0cb180 0x7f3f2c1effa0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3f140b3040 tx=0x7f3f1401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:52.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f297fa640 1 -- 172.21.15.57:0/3906893544 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f140be070 con 0x7f3f2c0cb180 2023-10-30T14:28:52.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f297fa640 1 -- 172.21.15.57:0/3906893544 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f140ba400 con 0x7f3f2c0cb180 2023-10-30T14:28:52.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f297fa640 1 -- 172.21.15.57:0/3906893544 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f140ba7d0 con 0x7f3f2c0cb180 2023-10-30T14:28:52.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.894+0000 7f3f31777640 1 -- 172.21.15.57:0/3906893544 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2c14a210 con 0x7f3f2c0cb180 2023-10-30T14:28:52.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.898+0000 7f3f31777640 1 -- 172.21.15.57:0/3906893544 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2c157b50 con 0x7f3f2c0cb180 2023-10-30T14:28:52.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.898+0000 7f3f31777640 1 -- 172.21.15.57:0/3906893544 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f2c067540 con 0x7f3f2c0cb180 2023-10-30T14:28:52.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.898+0000 7f3f297fa640 1 -- 172.21.15.57:0/3906893544 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3f140c3410 con 0x7f3f2c0cb180 2023-10-30T14:28:52.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.898+0000 7f3f297fa640 1 --2- 172.21.15.57:0/3906893544 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3f0c0664a0 0x7f3f0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:52.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.898+0000 7f3f297fa640 1 -- 172.21.15.57:0/3906893544 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f3f14121d30 con 0x7f3f2c0cb180 2023-10-30T14:28:52.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.898+0000 7f3f2bfff640 1 --2- 172.21.15.57:0/3906893544 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3f0c0664a0 0x7f3f0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:52.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.902+0000 7f3f2bfff640 1 --2- 172.21.15.57:0/3906893544 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3f0c0664a0 0x7f3f0c068960 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c000f70 tx=0x7f3f1c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:52.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:52.902+0000 7f3f297fa640 1 -- 172.21.15.57:0/3906893544 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3f140eef10 con 0x7f3f2c0cb180 2023-10-30T14:28:53.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.126+0000 7f3f31777640 1 -- 172.21.15.57:0/3906893544 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f2c151930 con 0x7f3f2c0cb180 2023-10-30T14:28:53.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.126+0000 7f3f297fa640 1 -- 172.21.15.57:0/3906893544 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f3f140f2530 con 0x7f3f2c0cb180 2023-10-30T14:28:53.128 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:28:53.128 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:52.928186+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5797,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:53.129 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 91 2023-10-30T14:28:53.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.130+0000 7f3f31777640 1 -- 172.21.15.57:0/3906893544 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3f0c0664a0 msgr2=0x7f3f0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:53.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.130+0000 7f3f31777640 1 --2- 172.21.15.57:0/3906893544 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3f0c0664a0 0x7f3f0c068960 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c000f70 tx=0x7f3f1c009040 comp rx=0 tx=0).stop 2023-10-30T14:28:53.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.130+0000 7f3f31777640 1 -- 172.21.15.57:0/3906893544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c0cb180 msgr2=0x7f3f2c1effa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:53.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.130+0000 7f3f31777640 1 --2- 172.21.15.57:0/3906893544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c0cb180 0x7f3f2c1effa0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3f140b3040 tx=0x7f3f1401ba90 comp rx=0 tx=0).stop 2023-10-30T14:28:53.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.130+0000 7f3f31777640 1 -- 172.21.15.57:0/3906893544 shutdown_connections 2023-10-30T14:28:53.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.130+0000 7f3f31777640 1 --2- 172.21.15.57:0/3906893544 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3f0c0664a0 0x7f3f0c068960 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:53.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.130+0000 7f3f31777640 1 --2- 172.21.15.57:0/3906893544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2c153b50 0x7f3f2c149390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:53.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.130+0000 7f3f31777640 1 --2- 172.21.15.57:0/3906893544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c0cb180 0x7f3f2c1effa0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:53.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.130+0000 7f3f31777640 1 --2- 172.21.15.57:0/3906893544 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3f2c0c62b0 0x7f3f2c14c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:53.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.130+0000 7f3f31777640 1 -- 172.21.15.57:0/3906893544 >> 172.21.15.57:0/3906893544 conn(0x7f3f2c065a80 msgr2=0x7f3f2c14e910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:53.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.130+0000 7f3f31777640 1 -- 172.21.15.57:0/3906893544 shutdown_connections 2023-10-30T14:28:53.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.130+0000 7f3f31777640 1 -- 172.21.15.57:0/3906893544 wait complete. 2023-10-30T14:28:53.157 INFO:tasks.mds_thrash.fs.[cephfs]:mds cluster has 3 alive and active, now stable! 2023-10-30T14:28:53.157 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2023-10-30T14:28:53.158 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T14:28:53.158 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:53.180 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T14:28:53.180 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:53.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.362+0000 7f05788ea640 1 Processor -- start 2023-10-30T14:28:53.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.362+0000 7f05788ea640 1 -- start start 2023-10-30T14:28:53.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.362+0000 7f05788ea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0574153c80 0x7f057415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:53.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.362+0000 7f05788ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f05740c4a10 0x7f05740c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:53.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.362+0000 7f05788ea640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f057415a3b0 0x7f05740ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:53.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.362+0000 7f05788ea640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f057415dec0 con 0x7f057415a3b0 2023-10-30T14:28:53.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.362+0000 7f05788ea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05741599a0 con 0x7f0574153c80 2023-10-30T14:28:53.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.362+0000 7f05788ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0574159f20 con 0x7f05740c4a10 2023-10-30T14:28:53.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.362+0000 7f0572d9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f057415a3b0 0x7f05740ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:53.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.362+0000 7f0572d9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f057415a3b0 0x7f05740ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35382/0 (socket says 172.21.15.57:35382) 2023-10-30T14:28:53.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.362+0000 7f0572d9d640 1 -- 172.21.15.57:0/1103366525 learned_addr learned my addr 172.21.15.57:0/1103366525 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:53.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.362+0000 7f056bfff640 1 --2- 172.21.15.57:0/1103366525 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f05740c4a10 0x7f05740c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:53.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.362+0000 7f057259c640 1 --2- 172.21.15.57:0/1103366525 >> v2:172.21.15.88:3300/0 conn(0x7f0574153c80 0x7f057415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:53.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.362+0000 7f0572d9d640 1 -- 172.21.15.57:0/1103366525 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f05740c4a10 msgr2=0x7f05740c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:53.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.362+0000 7f0572d9d640 1 --2- 172.21.15.57:0/1103366525 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f05740c4a10 0x7f05740c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:53.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.362+0000 7f0572d9d640 1 -- 172.21.15.57:0/1103366525 >> v2:172.21.15.88:3300/0 conn(0x7f0574153c80 msgr2=0x7f057415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:53.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.362+0000 7f0572d9d640 1 --2- 172.21.15.57:0/1103366525 >> v2:172.21.15.88:3300/0 conn(0x7f0574153c80 0x7f057415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:53.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.362+0000 7f0572d9d640 1 -- 172.21.15.57:0/1103366525 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05740c47e0 con 0x7f057415a3b0 2023-10-30T14:28:53.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.366+0000 7f0572d9d640 1 --2- 172.21.15.57:0/1103366525 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f057415a3b0 0x7f05740ca1e0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f05740d00f0 tx=0x7f056c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4e68f2665e92d71 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:53.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.366+0000 7f0571d9b640 1 -- 172.21.15.57:0/1103366525 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c0af400 con 0x7f057415a3b0 2023-10-30T14:28:53.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.366+0000 7f0571d9b640 1 -- 172.21.15.57:0/1103366525 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056c0ab050 con 0x7f057415a3b0 2023-10-30T14:28:53.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.366+0000 7f0571d9b640 1 -- 172.21.15.57:0/1103366525 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c0af780 con 0x7f057415a3b0 2023-10-30T14:28:53.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.366+0000 7f05788ea640 1 -- 172.21.15.57:0/1103366525 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f057415a3b0 msgr2=0x7f05740ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:53.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.366+0000 7f05788ea640 1 --2- 172.21.15.57:0/1103366525 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f057415a3b0 0x7f05740ca1e0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f05740d00f0 tx=0x7f056c01f1c0 comp rx=0 tx=0).stop 2023-10-30T14:28:53.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.366+0000 7f05788ea640 1 -- 172.21.15.57:0/1103366525 shutdown_connections 2023-10-30T14:28:53.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.366+0000 7f05788ea640 1 --2- 172.21.15.57:0/1103366525 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f05740c4a10 0x7f05740c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:53.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.366+0000 7f05788ea640 1 --2- 172.21.15.57:0/1103366525 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f057415a3b0 0x7f05740ca1e0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:53.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.366+0000 7f05788ea640 1 --2- 172.21.15.57:0/1103366525 >> v2:172.21.15.88:3300/0 conn(0x7f0574153c80 0x7f057415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:53.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.366+0000 7f05788ea640 1 -- 172.21.15.57:0/1103366525 >> 172.21.15.57:0/1103366525 conn(0x7f05740715c0 msgr2=0x7f05740719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:53.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.366+0000 7f05788ea640 1 -- 172.21.15.57:0/1103366525 shutdown_connections 2023-10-30T14:28:53.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.366+0000 7f05788ea640 1 -- 172.21.15.57:0/1103366525 wait complete. 2023-10-30T14:28:53.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f05788ea640 1 Processor -- start 2023-10-30T14:28:53.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f05788ea640 1 -- start start 2023-10-30T14:28:53.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f05788ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05740c4a10 0x7f057414c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:53.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f05788ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0574153c80 0x7f057414c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:53.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f05788ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f057415a3b0 0x7f057415ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:53.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f05788ea640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05741588f0 con 0x7f0574153c80 2023-10-30T14:28:53.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f05788ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0574158dc0 con 0x7f05740c4a10 2023-10-30T14:28:53.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f05788ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05740cd450 con 0x7f057415a3b0 2023-10-30T14:28:53.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f056bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0574153c80 0x7f057414c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:53.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f056bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0574153c80 0x7f057414c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35386/0 (socket says 172.21.15.57:35386) 2023-10-30T14:28:53.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f056bfff640 1 -- 172.21.15.57:0/2288757928 learned_addr learned my addr 172.21.15.57:0/2288757928 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:53.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f0572d9d640 1 --2- 172.21.15.57:0/2288757928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f057415a3b0 0x7f057415ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:53.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f057259c640 1 --2- 172.21.15.57:0/2288757928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05740c4a10 0x7f057414c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:53.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f056bfff640 1 -- 172.21.15.57:0/2288757928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f057415a3b0 msgr2=0x7f057415ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:53.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f056bfff640 1 --2- 172.21.15.57:0/2288757928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f057415a3b0 0x7f057415ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:53.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f056bfff640 1 -- 172.21.15.57:0/2288757928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05740c4a10 msgr2=0x7f057414c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:53.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f056bfff640 1 --2- 172.21.15.57:0/2288757928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05740c4a10 0x7f057414c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:53.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f056bfff640 1 -- 172.21.15.57:0/2288757928 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c009b30 con 0x7f0574153c80 2023-10-30T14:28:53.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f057259c640 1 --2- 172.21.15.57:0/2288757928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05740c4a10 0x7f057414c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:28:53.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f056bfff640 1 --2- 172.21.15.57:0/2288757928 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0574153c80 0x7f057414c930 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f055c00d7d0 tx=0x7f055c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:53.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f056affd640 1 -- 172.21.15.57:0/2288757928 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c009640 con 0x7f0574153c80 2023-10-30T14:28:53.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f056affd640 1 -- 172.21.15.57:0/2288757928 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f055c0097a0 con 0x7f0574153c80 2023-10-30T14:28:53.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f056affd640 1 -- 172.21.15.57:0/2288757928 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c009b50 con 0x7f0574153c80 2023-10-30T14:28:53.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f05788ea640 1 -- 172.21.15.57:0/2288757928 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0574158b20 con 0x7f0574153c80 2023-10-30T14:28:53.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f05788ea640 1 -- 172.21.15.57:0/2288757928 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05741efd90 con 0x7f0574153c80 2023-10-30T14:28:53.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.370+0000 7f05788ea640 1 -- 172.21.15.57:0/2288757928 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f057415a790 con 0x7f0574153c80 2023-10-30T14:28:53.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.374+0000 7f056affd640 1 -- 172.21.15.57:0/2288757928 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f055c009cb0 con 0x7f0574153c80 2023-10-30T14:28:53.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.374+0000 7f056affd640 1 --2- 172.21.15.57:0/2288757928 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05540664a0 0x7f0554068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:53.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.374+0000 7f056affd640 1 -- 172.21.15.57:0/2288757928 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f055c0846a0 con 0x7f0574153c80 2023-10-30T14:28:53.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.374+0000 7f057259c640 1 --2- 172.21.15.57:0/2288757928 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05540664a0 0x7f0554068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:53.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.374+0000 7f057259c640 1 --2- 172.21.15.57:0/2288757928 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05540664a0 0x7f0554068960 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0564007a70 tx=0x7f0564008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:53.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.378+0000 7f056affd640 1 -- 172.21.15.57:0/2288757928 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f055c051750 con 0x7f0574153c80 2023-10-30T14:28:53.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.566+0000 7f05788ea640 1 -- 172.21.15.57:0/2288757928 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05740cd590 con 0x7f0574153c80 2023-10-30T14:28:53.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.566+0000 7f056affd640 1 -- 172.21.15.57:0/2288757928 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f055c054d70 con 0x7f0574153c80 2023-10-30T14:28:53.568 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:28:53.568 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:52.928186+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5797,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:53.568 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 91 2023-10-30T14:28:53.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.570+0000 7f05788ea640 1 -- 172.21.15.57:0/2288757928 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05540664a0 msgr2=0x7f0554068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:53.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.570+0000 7f05788ea640 1 --2- 172.21.15.57:0/2288757928 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05540664a0 0x7f0554068960 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0564007a70 tx=0x7f0564008040 comp rx=0 tx=0).stop 2023-10-30T14:28:53.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.570+0000 7f05788ea640 1 -- 172.21.15.57:0/2288757928 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0574153c80 msgr2=0x7f057414c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:53.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.570+0000 7f05788ea640 1 --2- 172.21.15.57:0/2288757928 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0574153c80 0x7f057414c930 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f055c00d7d0 tx=0x7f055c00dc90 comp rx=0 tx=0).stop 2023-10-30T14:28:53.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.570+0000 7f05788ea640 1 -- 172.21.15.57:0/2288757928 shutdown_connections 2023-10-30T14:28:53.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.570+0000 7f05788ea640 1 --2- 172.21.15.57:0/2288757928 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05540664a0 0x7f0554068960 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:53.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.570+0000 7f05788ea640 1 --2- 172.21.15.57:0/2288757928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f057415a3b0 0x7f057415ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:53.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.570+0000 7f05788ea640 1 --2- 172.21.15.57:0/2288757928 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0574153c80 0x7f057414c930 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:53.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.570+0000 7f05788ea640 1 --2- 172.21.15.57:0/2288757928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05740c4a10 0x7f057414c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:53.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.570+0000 7f05788ea640 1 -- 172.21.15.57:0/2288757928 >> 172.21.15.57:0/2288757928 conn(0x7f05740715c0 msgr2=0x7f05740cb360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:53.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.570+0000 7f05788ea640 1 -- 172.21.15.57:0/2288757928 shutdown_connections 2023-10-30T14:28:53.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:53.570+0000 7f05788ea640 1 -- 172.21.15.57:0/2288757928 wait complete. 2023-10-30T14:28:53.589 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:28:55.590 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:55.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.742+0000 7fcea69d3640 1 Processor -- start 2023-10-30T14:28:55.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.742+0000 7fcea69d3640 1 -- start start 2023-10-30T14:28:55.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.746+0000 7fcea69d3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcea0151c20 0x7fcea00c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:55.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.746+0000 7fcea69d3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcea00cb8b0 0x7fcea006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:55.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.746+0000 7fcea69d3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea00c5030 con 0x7fcea0151840 2023-10-30T14:28:55.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.746+0000 7fcea69d3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea0157830 con 0x7fcea00cb8b0 2023-10-30T14:28:55.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.746+0000 7fcea69d3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea0157b10 con 0x7fcea0151c20 2023-10-30T14:28:55.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.746+0000 7fce9ffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fcea0151840 0x7fcea0072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44426/0 (socket says 172.21.15.57:44426) 2023-10-30T14:28:55.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.746+0000 7fce9ffff640 1 -- 172.21.15.57:0/500886397 learned_addr learned my addr 172.21.15.57:0/500886397 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:28:55.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.746+0000 7fcea4f70640 1 --2- 172.21.15.57:0/500886397 >> v2:172.21.15.57:3300/0 conn(0x7fcea00cb8b0 0x7fcea006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:55.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.746+0000 7fce9f7fe640 1 --2- 172.21.15.57:0/500886397 >> v2:172.21.15.88:3300/0 conn(0x7fcea0151c20 0x7fcea00c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:55.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.746+0000 7fce9effd640 1 -- 172.21.15.57:0/500886397 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051392959 0 0) 0x7fcea00c5030 con 0x7fcea0151840 2023-10-30T14:28:55.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.746+0000 7fce9effd640 1 -- 172.21.15.57:0/500886397 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce88003530 con 0x7fcea0151840 2023-10-30T14:28:55.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.746+0000 7fcea4f70640 1 -- 172.21.15.57:0/500886397 >> v2:172.21.15.88:3300/0 conn(0x7fcea0151c20 msgr2=0x7fcea00c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:55.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.746+0000 7fcea4f70640 1 --2- 172.21.15.57:0/500886397 >> v2:172.21.15.88:3300/0 conn(0x7fcea0151c20 0x7fcea00c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:55.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.746+0000 7fcea4f70640 1 -- 172.21.15.57:0/500886397 >> v1:172.21.15.57:6789/0 conn(0x7fcea0151840 legacy=0x7fcea0072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:55.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.746+0000 7fcea4f70640 1 -- 172.21.15.57:0/500886397 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea0157df0 con 0x7fcea00cb8b0 2023-10-30T14:28:55.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.750+0000 7fcea4f70640 1 --2- 172.21.15.57:0/500886397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea00cb8b0 0x7fcea006b600 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fcea006d4d0 tx=0x7fce9801f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b60200b1a69782 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:55.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.750+0000 7fce9effd640 1 -- 172.21.15.57:0/500886397 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce980b0440 con 0x7fcea00cb8b0 2023-10-30T14:28:55.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.750+0000 7fce9effd640 1 -- 172.21.15.57:0/500886397 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce980b05a0 con 0x7fcea00cb8b0 2023-10-30T14:28:55.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.750+0000 7fce9effd640 1 -- 172.21.15.57:0/500886397 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce980b0930 con 0x7fcea00cb8b0 2023-10-30T14:28:55.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.750+0000 7fcea69d3640 1 -- 172.21.15.57:0/500886397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea00cb8b0 msgr2=0x7fcea006b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:55.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.750+0000 7fcea69d3640 1 --2- 172.21.15.57:0/500886397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea00cb8b0 0x7fcea006b600 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fcea006d4d0 tx=0x7fce9801f0f0 comp rx=0 tx=0).stop 2023-10-30T14:28:55.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.750+0000 7fcea69d3640 1 -- 172.21.15.57:0/500886397 shutdown_connections 2023-10-30T14:28:55.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.750+0000 7fcea69d3640 1 --2- 172.21.15.57:0/500886397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea00cb8b0 0x7fcea006b600 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:55.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.750+0000 7fcea69d3640 1 --2- 172.21.15.57:0/500886397 >> v2:172.21.15.88:3300/0 conn(0x7fcea0151c20 0x7fcea00c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:55.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.750+0000 7fcea69d3640 1 -- 172.21.15.57:0/500886397 >> 172.21.15.57:0/500886397 conn(0x7fcea00cac70 msgr2=0x7fcea00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:55.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.750+0000 7fcea69d3640 1 -- 172.21.15.57:0/500886397 shutdown_connections 2023-10-30T14:28:55.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.750+0000 7fcea69d3640 1 -- 172.21.15.57:0/500886397 wait complete. 2023-10-30T14:28:55.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.750+0000 7fcea69d3640 1 Processor -- start 2023-10-30T14:28:55.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.750+0000 7fcea69d3640 1 -- start start 2023-10-30T14:28:55.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fcea69d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcea00cb8b0 0x7fcea00c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:55.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fcea69d3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcea0151840 0x7fcea0143590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:55.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fcea69d3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea0151c20 0x7fcea0143ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:55.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fcea69d3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea0144010 con 0x7fcea0151c20 2023-10-30T14:28:55.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fcea69d3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea00c9190 con 0x7fcea0151840 2023-10-30T14:28:55.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fcea69d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea00c9470 con 0x7fcea00cb8b0 2023-10-30T14:28:55.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fcea4f70640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea0151c20 0x7fcea0143ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fcea4f70640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea0151c20 0x7fcea0143ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55846/0 (socket says 172.21.15.57:55846) 2023-10-30T14:28:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fcea4f70640 1 -- 172.21.15.57:0/3409874237 learned_addr learned my addr 172.21.15.57:0/3409874237 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fce9ffff640 1 --2- 172.21.15.57:0/3409874237 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcea00cb8b0 0x7fcea00c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fce9f7fe640 1 --2- 172.21.15.57:0/3409874237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcea0151840 0x7fcea0143590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fcea4f70640 1 -- 172.21.15.57:0/3409874237 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcea00cb8b0 msgr2=0x7fcea00c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fcea4f70640 1 --2- 172.21.15.57:0/3409874237 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcea00cb8b0 0x7fcea00c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fcea4f70640 1 -- 172.21.15.57:0/3409874237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcea0151840 msgr2=0x7fcea0143590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fcea4f70640 1 --2- 172.21.15.57:0/3409874237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcea0151840 0x7fcea0143590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fcea4f70640 1 -- 172.21.15.57:0/3409874237 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce98009a70 con 0x7fcea0151c20 2023-10-30T14:28:55.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fce9f7fe640 1 --2- 172.21.15.57:0/3409874237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcea0151840 0x7fcea0143590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:28:55.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fcea4f70640 1 --2- 172.21.15.57:0/3409874237 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea0151c20 0x7fcea0143ad0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fce9801bd00 tx=0x7fce9801be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:55.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fce9cff9640 1 -- 172.21.15.57:0/3409874237 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce980b0440 con 0x7fcea0151c20 2023-10-30T14:28:55.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fce9cff9640 1 -- 172.21.15.57:0/3409874237 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce980bc650 con 0x7fcea0151c20 2023-10-30T14:28:55.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fce9cff9640 1 -- 172.21.15.57:0/3409874237 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce980bca00 con 0x7fcea0151c20 2023-10-30T14:28:55.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fcea69d3640 1 -- 172.21.15.57:0/3409874237 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea01ef320 con 0x7fcea0151c20 2023-10-30T14:28:55.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fcea69d3640 1 -- 172.21.15.57:0/3409874237 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea0156ed0 con 0x7fcea0151c20 2023-10-30T14:28:55.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fcea69d3640 1 -- 172.21.15.57:0/3409874237 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea0156d30 con 0x7fcea0151c20 2023-10-30T14:28:55.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.754+0000 7fce9cff9640 1 -- 172.21.15.57:0/3409874237 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fce980bcb60 con 0x7fcea0151c20 2023-10-30T14:28:55.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.758+0000 7fce9cff9640 1 --2- 172.21.15.57:0/3409874237 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fce8c0664a0 0x7fce8c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:55.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.758+0000 7fce9cff9640 1 -- 172.21.15.57:0/3409874237 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7fce98122f20 con 0x7fcea0151c20 2023-10-30T14:28:55.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.758+0000 7fce9ffff640 1 --2- 172.21.15.57:0/3409874237 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fce8c0664a0 0x7fce8c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:55.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.758+0000 7fce9ffff640 1 --2- 172.21.15.57:0/3409874237 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fce8c0664a0 0x7fce8c068960 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fce900011f0 tx=0x7fce90009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:55.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.762+0000 7fce9cff9640 1 -- 172.21.15.57:0/3409874237 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcea0156d30 con 0x7fcea0151c20 2023-10-30T14:28:55.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.950+0000 7fcea69d3640 1 -- 172.21.15.57:0/3409874237 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcea0156d30 con 0x7fcea0151c20 2023-10-30T14:28:55.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.954+0000 7fce9cff9640 1 -- 172.21.15.57:0/3409874237 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fcea0156d30 con 0x7fcea0151c20 2023-10-30T14:28:55.953 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:28:55.954 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:52.928186+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5797,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:55.954 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 91 2023-10-30T14:28:55.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.954+0000 7fcea69d3640 1 -- 172.21.15.57:0/3409874237 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fce8c0664a0 msgr2=0x7fce8c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:55.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.954+0000 7fcea69d3640 1 --2- 172.21.15.57:0/3409874237 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fce8c0664a0 0x7fce8c068960 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fce900011f0 tx=0x7fce90009040 comp rx=0 tx=0).stop 2023-10-30T14:28:55.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.954+0000 7fcea69d3640 1 -- 172.21.15.57:0/3409874237 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea0151c20 msgr2=0x7fcea0143ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:55.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.954+0000 7fcea69d3640 1 --2- 172.21.15.57:0/3409874237 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea0151c20 0x7fcea0143ad0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fce9801bd00 tx=0x7fce9801be90 comp rx=0 tx=0).stop 2023-10-30T14:28:55.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.954+0000 7fcea69d3640 1 -- 172.21.15.57:0/3409874237 shutdown_connections 2023-10-30T14:28:55.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.954+0000 7fcea69d3640 1 --2- 172.21.15.57:0/3409874237 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fce8c0664a0 0x7fce8c068960 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:55.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.954+0000 7fcea69d3640 1 --2- 172.21.15.57:0/3409874237 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea0151c20 0x7fcea0143ad0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:55.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.954+0000 7fcea69d3640 1 --2- 172.21.15.57:0/3409874237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcea0151840 0x7fcea0143590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:55.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.954+0000 7fcea69d3640 1 --2- 172.21.15.57:0/3409874237 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcea00cb8b0 0x7fcea00c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:55.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.954+0000 7fcea69d3640 1 -- 172.21.15.57:0/3409874237 >> 172.21.15.57:0/3409874237 conn(0x7fcea00cac70 msgr2=0x7fcea0072000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:55.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.954+0000 7fcea69d3640 1 -- 172.21.15.57:0/3409874237 shutdown_connections 2023-10-30T14:28:55.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:55.954+0000 7fcea69d3640 1 -- 172.21.15.57:0/3409874237 wait complete. 2023-10-30T14:28:55.976 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:28:57.977 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:58.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.130+0000 7f479fa6b640 1 Processor -- start 2023-10-30T14:28:58.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.130+0000 7f479fa6b640 1 -- start start 2023-10-30T14:28:58.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.130+0000 7f479fa6b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4798151840 0x7f47980c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:58.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.130+0000 7f479fa6b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47980cb8b0 0x7f479806b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:58.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.130+0000 7f479fa6b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47980c5030 con 0x7f4798151c20 2023-10-30T14:28:58.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.130+0000 7f479fa6b640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4798157830 con 0x7f4798151840 2023-10-30T14:28:58.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.130+0000 7f479fa6b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4798157b10 con 0x7f47980cb8b0 2023-10-30T14:28:58.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.130+0000 7f479d807640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4798151840 0x7f47980c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:58.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.130+0000 7f479d807640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4798151840 0x7f47980c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55862/0 (socket says 172.21.15.57:55862) 2023-10-30T14:28:58.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.134+0000 7f479d807640 1 -- 172.21.15.57:0/1568543442 learned_addr learned my addr 172.21.15.57:0/1568543442 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:58.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.134+0000 7f479e008640 1 --2- 172.21.15.57:0/1568543442 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47980cb8b0 0x7f479806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:58.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.134+0000 7f479d807640 1 -- 172.21.15.57:0/1568543442 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47980cb8b0 msgr2=0x7f479806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:58.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.134+0000 7f479d807640 1 --2- 172.21.15.57:0/1568543442 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47980cb8b0 0x7f479806b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:58.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.134+0000 7f479d807640 1 -- 172.21.15.57:0/1568543442 >> v1:172.21.15.88:6789/0 conn(0x7f4798151c20 legacy=0x7f4798072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:58.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.134+0000 7f479d807640 1 -- 172.21.15.57:0/1568543442 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4798157df0 con 0x7f4798151840 2023-10-30T14:28:58.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.134+0000 7f479d807640 1 --2- 172.21.15.57:0/1568543442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4798151840 0x7f47980c49a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f4794009870 tx=0x7f479401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97067f21faf18c49 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:58.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.134+0000 7f479c805640 1 -- 172.21.15.57:0/1568543442 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47940a8a30 con 0x7f4798151840 2023-10-30T14:28:58.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.134+0000 7f479c805640 1 -- 172.21.15.57:0/1568543442 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47940a8b90 con 0x7f4798151840 2023-10-30T14:28:58.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.134+0000 7f479c805640 1 -- 172.21.15.57:0/1568543442 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47940b0440 con 0x7f4798151840 2023-10-30T14:28:58.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479fa6b640 1 -- 172.21.15.57:0/1568543442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4798151840 msgr2=0x7f47980c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:58.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479fa6b640 1 --2- 172.21.15.57:0/1568543442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4798151840 0x7f47980c49a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f4794009870 tx=0x7f479401ed30 comp rx=0 tx=0).stop 2023-10-30T14:28:58.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479fa6b640 1 -- 172.21.15.57:0/1568543442 shutdown_connections 2023-10-30T14:28:58.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479fa6b640 1 --2- 172.21.15.57:0/1568543442 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47980cb8b0 0x7f479806b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:58.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479fa6b640 1 --2- 172.21.15.57:0/1568543442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4798151840 0x7f47980c49a0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:58.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479fa6b640 1 -- 172.21.15.57:0/1568543442 >> 172.21.15.57:0/1568543442 conn(0x7f47980cac70 msgr2=0x7f47980cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:58.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479fa6b640 1 -- 172.21.15.57:0/1568543442 shutdown_connections 2023-10-30T14:28:58.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479fa6b640 1 -- 172.21.15.57:0/1568543442 wait complete. 2023-10-30T14:28:58.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479fa6b640 1 Processor -- start 2023-10-30T14:28:58.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479fa6b640 1 -- start start 2023-10-30T14:28:58.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479fa6b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47980cb8b0 0x7f47981548d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:58.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479fa6b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4798151840 0x7f4798156840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:58.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479fa6b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4798151c20 0x7f47981525b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:58.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479fa6b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47981520d0 con 0x7f4798151840 2023-10-30T14:28:58.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479fa6b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479814b5d0 con 0x7f47980cb8b0 2023-10-30T14:28:58.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479fa6b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4798148710 con 0x7f4798151c20 2023-10-30T14:28:58.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479d006640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4798151840 0x7f4798156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:58.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479d006640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4798151840 0x7f4798156840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55878/0 (socket says 172.21.15.57:55878) 2023-10-30T14:28:58.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479d006640 1 -- 172.21.15.57:0/3773157482 learned_addr learned my addr 172.21.15.57:0/3773157482 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:28:58.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479e008640 1 --2- 172.21.15.57:0/3773157482 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4798151c20 0x7f47981525b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:58.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479d807640 1 --2- 172.21.15.57:0/3773157482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47980cb8b0 0x7f47981548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:58.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479d006640 1 -- 172.21.15.57:0/3773157482 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4798151c20 msgr2=0x7f47981525b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:58.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479d006640 1 --2- 172.21.15.57:0/3773157482 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4798151c20 0x7f47981525b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:58.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479d006640 1 -- 172.21.15.57:0/3773157482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47980cb8b0 msgr2=0x7f47981548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:58.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479d006640 1 --2- 172.21.15.57:0/3773157482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47980cb8b0 0x7f47981548d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:58.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479d006640 1 -- 172.21.15.57:0/3773157482 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47940094d0 con 0x7f4798151840 2023-10-30T14:28:58.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.138+0000 7f479d807640 1 --2- 172.21.15.57:0/3773157482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47980cb8b0 0x7f47981548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:28:58.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.142+0000 7f479d006640 1 --2- 172.21.15.57:0/3773157482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4798151840 0x7f4798156840 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f478800c3c0 tx=0x7f478800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:58.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.142+0000 7f478e7fc640 1 -- 172.21.15.57:0/3773157482 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47880052d0 con 0x7f4798151840 2023-10-30T14:28:58.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.142+0000 7f478e7fc640 1 -- 172.21.15.57:0/3773157482 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4788005430 con 0x7f4798151840 2023-10-30T14:28:58.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.142+0000 7f478e7fc640 1 -- 172.21.15.57:0/3773157482 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788003840 con 0x7f4798151840 2023-10-30T14:28:58.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.142+0000 7f479fa6b640 1 -- 172.21.15.57:0/3773157482 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47981489f0 con 0x7f4798151840 2023-10-30T14:28:58.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.142+0000 7f479fa6b640 1 -- 172.21.15.57:0/3773157482 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4798148fb0 con 0x7f4798151840 2023-10-30T14:28:58.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.142+0000 7f479fa6b640 1 -- 172.21.15.57:0/3773157482 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f479804e170 con 0x7f4798151840 2023-10-30T14:28:58.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.142+0000 7f478e7fc640 1 -- 172.21.15.57:0/3773157482 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f47880039a0 con 0x7f4798151840 2023-10-30T14:28:58.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.142+0000 7f478e7fc640 1 --2- 172.21.15.57:0/3773157482 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f477c0664a0 0x7f477c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:58.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.142+0000 7f478e7fc640 1 -- 172.21.15.57:0/3773157482 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f4788083800 con 0x7f4798151840 2023-10-30T14:28:58.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.146+0000 7f479d807640 1 --2- 172.21.15.57:0/3773157482 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f477c0664a0 0x7f477c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:58.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.146+0000 7f479d807640 1 --2- 172.21.15.57:0/3773157482 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f477c0664a0 0x7f477c068960 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f47940b2000 tx=0x7f4794008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:58.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.146+0000 7f478e7fc640 1 -- 172.21.15.57:0/3773157482 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4788050820 con 0x7f4798151840 2023-10-30T14:28:58.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.338+0000 7f479fa6b640 1 -- 172.21.15.57:0/3773157482 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47980c39c0 con 0x7f4798151840 2023-10-30T14:28:58.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.342+0000 7f478e7fc640 1 -- 172.21.15.57:0/3773157482 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f4788053e40 con 0x7f4798151840 2023-10-30T14:28:58.341 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:28:58.341 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:52.928186+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5797,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:58.342 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 91 2023-10-30T14:28:58.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.342+0000 7f479fa6b640 1 -- 172.21.15.57:0/3773157482 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f477c0664a0 msgr2=0x7f477c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:58.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.342+0000 7f479fa6b640 1 --2- 172.21.15.57:0/3773157482 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f477c0664a0 0x7f477c068960 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f47940b2000 tx=0x7f4794008040 comp rx=0 tx=0).stop 2023-10-30T14:28:58.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.342+0000 7f479fa6b640 1 -- 172.21.15.57:0/3773157482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4798151840 msgr2=0x7f4798156840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:58.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.342+0000 7f479fa6b640 1 --2- 172.21.15.57:0/3773157482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4798151840 0x7f4798156840 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f478800c3c0 tx=0x7f478800c880 comp rx=0 tx=0).stop 2023-10-30T14:28:58.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.342+0000 7f479fa6b640 1 -- 172.21.15.57:0/3773157482 shutdown_connections 2023-10-30T14:28:58.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.342+0000 7f479fa6b640 1 --2- 172.21.15.57:0/3773157482 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f477c0664a0 0x7f477c068960 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:58.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.342+0000 7f479fa6b640 1 --2- 172.21.15.57:0/3773157482 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4798151c20 0x7f47981525b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:58.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.342+0000 7f479fa6b640 1 --2- 172.21.15.57:0/3773157482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4798151840 0x7f4798156840 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:58.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.342+0000 7f479fa6b640 1 --2- 172.21.15.57:0/3773157482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47980cb8b0 0x7f47981548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:58.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.342+0000 7f479fa6b640 1 -- 172.21.15.57:0/3773157482 >> 172.21.15.57:0/3773157482 conn(0x7f47980cac70 msgr2=0x7f4798072000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:58.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.342+0000 7f479fa6b640 1 -- 172.21.15.57:0/3773157482 shutdown_connections 2023-10-30T14:28:58.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:28:58.342+0000 7f479fa6b640 1 -- 172.21.15.57:0/3773157482 wait complete. 2023-10-30T14:28:58.362 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:29:00.363 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:00.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.518+0000 7f359d646640 1 Processor -- start 2023-10-30T14:29:00.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.518+0000 7f359d646640 1 -- start start 2023-10-30T14:29:00.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.518+0000 7f359d646640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f35980c4a10 0x7f359815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:00.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.518+0000 7f359d646640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35980c42a0 0x7f35980ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:00.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.518+0000 7f359d646640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3598159f20 con 0x7f3598153c80 2023-10-30T14:29:00.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.518+0000 7f359d646640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f359815dec0 con 0x7f35980c4a10 2023-10-30T14:29:00.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.518+0000 7f359d646640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35981599a0 con 0x7f35980c42a0 2023-10-30T14:29:00.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.518+0000 7f3596ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f3598153c80 0x7f359815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44428/0 (socket says 172.21.15.57:44428) 2023-10-30T14:29:00.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.518+0000 7f3596ffd640 1 -- 172.21.15.57:0/369109195 learned_addr learned my addr 172.21.15.57:0/369109195 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:29:00.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.518+0000 7f358ffff640 1 --2- 172.21.15.57:0/369109195 >> v2:172.21.15.57:3300/0 conn(0x7f35980c4a10 0x7f359815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:00.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.518+0000 7f35977fe640 1 --2- 172.21.15.57:0/369109195 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35980c42a0 0x7f35980ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:00.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.518+0000 7f35967fc640 1 -- 172.21.15.57:0/369109195 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860307 0 0) 0x7f3598159f20 con 0x7f3598153c80 2023-10-30T14:29:00.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.518+0000 7f358ffff640 1 -- 172.21.15.57:0/369109195 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35980c42a0 msgr2=0x7f35980ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:00.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.518+0000 7f358ffff640 1 --2- 172.21.15.57:0/369109195 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35980c42a0 0x7f35980ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:00.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.518+0000 7f358ffff640 1 -- 172.21.15.57:0/369109195 >> v1:172.21.15.57:6789/0 conn(0x7f3598153c80 legacy=0x7f359815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:00.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.518+0000 7f358ffff640 1 -- 172.21.15.57:0/369109195 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359815d620 con 0x7f35980c4a10 2023-10-30T14:29:00.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.518+0000 7f35977fe640 1 --2- 172.21.15.57:0/369109195 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35980c42a0 0x7f35980ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:00.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.522+0000 7f358ffff640 1 --2- 172.21.15.57:0/369109195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35980c4a10 0x7f359815cf90 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f35800097c0 tx=0x7f358001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d42112223dfb644 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:00.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.522+0000 7f35967fc640 1 -- 172.21.15.57:0/369109195 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35800a8970 con 0x7f35980c4a10 2023-10-30T14:29:00.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.522+0000 7f35967fc640 1 -- 172.21.15.57:0/369109195 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35800a8ad0 con 0x7f35980c4a10 2023-10-30T14:29:00.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.522+0000 7f35967fc640 1 -- 172.21.15.57:0/369109195 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35800a8e60 con 0x7f35980c4a10 2023-10-30T14:29:00.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.522+0000 7f359d646640 1 -- 172.21.15.57:0/369109195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35980c4a10 msgr2=0x7f359815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:00.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.522+0000 7f359d646640 1 --2- 172.21.15.57:0/369109195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35980c4a10 0x7f359815cf90 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f35800097c0 tx=0x7f358001ec70 comp rx=0 tx=0).stop 2023-10-30T14:29:00.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.522+0000 7f359d646640 1 -- 172.21.15.57:0/369109195 shutdown_connections 2023-10-30T14:29:00.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.522+0000 7f359d646640 1 --2- 172.21.15.57:0/369109195 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35980c42a0 0x7f35980ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:00.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.522+0000 7f359d646640 1 --2- 172.21.15.57:0/369109195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35980c4a10 0x7f359815cf90 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:00.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.522+0000 7f359d646640 1 -- 172.21.15.57:0/369109195 >> 172.21.15.57:0/369109195 conn(0x7f35980715c0 msgr2=0x7f35980719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:00.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.522+0000 7f359d646640 1 -- 172.21.15.57:0/369109195 shutdown_connections 2023-10-30T14:29:00.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.522+0000 7f359d646640 1 -- 172.21.15.57:0/369109195 wait complete. 2023-10-30T14:29:00.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f359d646640 1 Processor -- start 2023-10-30T14:29:00.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f359d646640 1 -- start start 2023-10-30T14:29:00.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f359d646640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35980c42a0 0x7f35981542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:00.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f359d646640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35980c4a10 0x7f35981508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:00.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f359d646640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3598153c80 0x7f35981f23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:00.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f359d646640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3598154830 con 0x7f3598153c80 2023-10-30T14:29:00.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f359d646640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35981550d0 con 0x7f35980c42a0 2023-10-30T14:29:00.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f359d646640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3598155390 con 0x7f35980c4a10 2023-10-30T14:29:00.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f35977fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3598153c80 0x7f35981f23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:00.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f35977fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3598153c80 0x7f35981f23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55902/0 (socket says 172.21.15.57:55902) 2023-10-30T14:29:00.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f35977fe640 1 -- 172.21.15.57:0/3654831861 learned_addr learned my addr 172.21.15.57:0/3654831861 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:00.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f358ffff640 1 --2- 172.21.15.57:0/3654831861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35980c4a10 0x7f35981508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:00.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f3596ffd640 1 --2- 172.21.15.57:0/3654831861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35980c42a0 0x7f35981542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:00.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f35977fe640 1 -- 172.21.15.57:0/3654831861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35980c4a10 msgr2=0x7f35981508c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:00.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f35977fe640 1 --2- 172.21.15.57:0/3654831861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35980c4a10 0x7f35981508c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:00.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f35977fe640 1 -- 172.21.15.57:0/3654831861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35980c42a0 msgr2=0x7f35981542f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:00.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f35977fe640 1 --2- 172.21.15.57:0/3654831861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35980c42a0 0x7f35981542f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:00.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f35977fe640 1 -- 172.21.15.57:0/3654831861 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3580009420 con 0x7f3598153c80 2023-10-30T14:29:00.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f3596ffd640 1 --2- 172.21.15.57:0/3654831861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35980c42a0 0x7f35981542f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:00.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f35977fe640 1 --2- 172.21.15.57:0/3654831861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3598153c80 0x7f35981f23f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3590007ef0 tx=0x7f359000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:00.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f358f7fe640 1 -- 172.21.15.57:0/3654831861 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3590003ba0 con 0x7f3598153c80 2023-10-30T14:29:00.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f358f7fe640 1 -- 172.21.15.57:0/3654831861 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359000f040 con 0x7f3598153c80 2023-10-30T14:29:00.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f359d646640 1 -- 172.21.15.57:0/3654831861 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3598155670 con 0x7f3598153c80 2023-10-30T14:29:00.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f358f7fe640 1 -- 172.21.15.57:0/3654831861 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359000de80 con 0x7f3598153c80 2023-10-30T14:29:00.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f359d646640 1 -- 172.21.15.57:0/3654831861 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35981f1bf0 con 0x7f3598153c80 2023-10-30T14:29:00.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.526+0000 7f359d646640 1 -- 172.21.15.57:0/3654831861 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359804e170 con 0x7f3598153c80 2023-10-30T14:29:00.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.530+0000 7f358f7fe640 1 -- 172.21.15.57:0/3654831861 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3590004030 con 0x7f3598153c80 2023-10-30T14:29:00.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.530+0000 7f358f7fe640 1 --2- 172.21.15.57:0/3654831861 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f35780664a0 0x7f3578068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:00.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.530+0000 7f358f7fe640 1 -- 172.21.15.57:0/3654831861 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f3590084550 con 0x7f3598153c80 2023-10-30T14:29:00.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.530+0000 7f3596ffd640 1 --2- 172.21.15.57:0/3654831861 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f35780664a0 0x7f3578068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:00.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.530+0000 7f3596ffd640 1 --2- 172.21.15.57:0/3654831861 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f35780664a0 0x7f3578068960 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3588000f70 tx=0x7f3588007cb0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:00.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.534+0000 7f358f7fe640 1 -- 172.21.15.57:0/3654831861 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3590051620 con 0x7f3598153c80 2023-10-30T14:29:00.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.726+0000 7f359d646640 1 -- 172.21.15.57:0/3654831861 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35980cad40 con 0x7f3598153c80 2023-10-30T14:29:00.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.726+0000 7f358f7fe640 1 -- 172.21.15.57:0/3654831861 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f3590054c40 con 0x7f3598153c80 2023-10-30T14:29:00.729 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:00.729 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:52.928186+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5797,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:00.729 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 91 2023-10-30T14:29:00.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.730+0000 7f359d646640 1 -- 172.21.15.57:0/3654831861 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f35780664a0 msgr2=0x7f3578068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:00.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.730+0000 7f359d646640 1 --2- 172.21.15.57:0/3654831861 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f35780664a0 0x7f3578068960 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3588000f70 tx=0x7f3588007cb0 comp rx=0 tx=0).stop 2023-10-30T14:29:00.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.730+0000 7f359d646640 1 -- 172.21.15.57:0/3654831861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3598153c80 msgr2=0x7f35981f23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:00.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.730+0000 7f359d646640 1 --2- 172.21.15.57:0/3654831861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3598153c80 0x7f35981f23f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3590007ef0 tx=0x7f359000d4a0 comp rx=0 tx=0).stop 2023-10-30T14:29:00.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.730+0000 7f359d646640 1 -- 172.21.15.57:0/3654831861 shutdown_connections 2023-10-30T14:29:00.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.730+0000 7f359d646640 1 --2- 172.21.15.57:0/3654831861 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f35780664a0 0x7f3578068960 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:00.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.730+0000 7f359d646640 1 --2- 172.21.15.57:0/3654831861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3598153c80 0x7f35981f23f0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:00.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.730+0000 7f359d646640 1 --2- 172.21.15.57:0/3654831861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35980c4a10 0x7f35981508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:00.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.730+0000 7f359d646640 1 --2- 172.21.15.57:0/3654831861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35980c42a0 0x7f35981542f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:00.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.730+0000 7f359d646640 1 -- 172.21.15.57:0/3654831861 >> 172.21.15.57:0/3654831861 conn(0x7f35980715c0 msgr2=0x7f359806bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:00.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.730+0000 7f359d646640 1 -- 172.21.15.57:0/3654831861 shutdown_connections 2023-10-30T14:29:00.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:00.730+0000 7f359d646640 1 -- 172.21.15.57:0/3654831861 wait complete. 2023-10-30T14:29:00.750 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:29:02.751 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:02.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.905+0000 7f32bb2a9640 1 Processor -- start 2023-10-30T14:29:02.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.905+0000 7f32bb2a9640 1 -- start start 2023-10-30T14:29:02.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.905+0000 7f32bb2a9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f32b4153c80 0x7f32b415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:02.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.905+0000 7f32bb2a9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32b40c4a10 0x7f32b40c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:02.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.905+0000 7f32bb2a9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f32b415a3b0 0x7f32b40ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:02.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.905+0000 7f32bb2a9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b415dec0 con 0x7f32b415a3b0 2023-10-30T14:29:02.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.905+0000 7f32bb2a9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b41599a0 con 0x7f32b4153c80 2023-10-30T14:29:02.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.905+0000 7f32bb2a9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b4159f20 con 0x7f32b40c4a10 2023-10-30T14:29:02.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.905+0000 7f32b9846640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f32b415a3b0 0x7f32b40ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:02.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.905+0000 7f32b9846640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f32b415a3b0 0x7f32b40ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55908/0 (socket says 172.21.15.57:55908) 2023-10-30T14:29:02.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.905+0000 7f32b9846640 1 -- 172.21.15.57:0/2562603954 learned_addr learned my addr 172.21.15.57:0/2562603954 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:02.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.905+0000 7f32b8844640 1 --2- 172.21.15.57:0/2562603954 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32b40c4a10 0x7f32b40c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:02.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.905+0000 7f32b9045640 1 --2- 172.21.15.57:0/2562603954 >> v2:172.21.15.88:3300/0 conn(0x7f32b4153c80 0x7f32b415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:02.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.905+0000 7f32b9846640 1 -- 172.21.15.57:0/2562603954 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32b40c4a10 msgr2=0x7f32b40c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:02.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.905+0000 7f32b9846640 1 --2- 172.21.15.57:0/2562603954 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32b40c4a10 0x7f32b40c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:02.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.905+0000 7f32b9846640 1 -- 172.21.15.57:0/2562603954 >> v2:172.21.15.88:3300/0 conn(0x7f32b4153c80 msgr2=0x7f32b415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:02.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.905+0000 7f32b9846640 1 --2- 172.21.15.57:0/2562603954 >> v2:172.21.15.88:3300/0 conn(0x7f32b4153c80 0x7f32b415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:02.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.905+0000 7f32b9846640 1 -- 172.21.15.57:0/2562603954 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b40c47e0 con 0x7f32b415a3b0 2023-10-30T14:29:02.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.909+0000 7f32b9846640 1 --2- 172.21.15.57:0/2562603954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b415a3b0 0x7f32b40ca1e0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f32b40d00f0 tx=0x7f32ac01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3459cf45172ef32 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:02.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.909+0000 7f32abfff640 1 -- 172.21.15.57:0/2562603954 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ac0af400 con 0x7f32b415a3b0 2023-10-30T14:29:02.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.909+0000 7f32abfff640 1 -- 172.21.15.57:0/2562603954 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32ac0ab050 con 0x7f32b415a3b0 2023-10-30T14:29:02.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.909+0000 7f32abfff640 1 -- 172.21.15.57:0/2562603954 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ac0af780 con 0x7f32b415a3b0 2023-10-30T14:29:02.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.909+0000 7f32bb2a9640 1 -- 172.21.15.57:0/2562603954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b415a3b0 msgr2=0x7f32b40ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:02.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.909+0000 7f32bb2a9640 1 --2- 172.21.15.57:0/2562603954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b415a3b0 0x7f32b40ca1e0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f32b40d00f0 tx=0x7f32ac01f1c0 comp rx=0 tx=0).stop 2023-10-30T14:29:02.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.909+0000 7f32bb2a9640 1 -- 172.21.15.57:0/2562603954 shutdown_connections 2023-10-30T14:29:02.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.909+0000 7f32bb2a9640 1 --2- 172.21.15.57:0/2562603954 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32b40c4a10 0x7f32b40c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:02.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.909+0000 7f32bb2a9640 1 --2- 172.21.15.57:0/2562603954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b415a3b0 0x7f32b40ca1e0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:02.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.909+0000 7f32bb2a9640 1 --2- 172.21.15.57:0/2562603954 >> v2:172.21.15.88:3300/0 conn(0x7f32b4153c80 0x7f32b415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:02.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.909+0000 7f32bb2a9640 1 -- 172.21.15.57:0/2562603954 >> 172.21.15.57:0/2562603954 conn(0x7f32b40715c0 msgr2=0x7f32b40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:02.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.909+0000 7f32bb2a9640 1 -- 172.21.15.57:0/2562603954 shutdown_connections 2023-10-30T14:29:02.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.909+0000 7f32bb2a9640 1 -- 172.21.15.57:0/2562603954 wait complete. 2023-10-30T14:29:02.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32bb2a9640 1 Processor -- start 2023-10-30T14:29:02.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32bb2a9640 1 -- start start 2023-10-30T14:29:02.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32bb2a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b40c4a10 0x7f32b414c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:02.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32bb2a9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32b4153c80 0x7f32b414c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:02.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32bb2a9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32b415a3b0 0x7f32b415ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:02.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32bb2a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b41588f0 con 0x7f32b40c4a10 2023-10-30T14:29:02.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32bb2a9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b4158dc0 con 0x7f32b415a3b0 2023-10-30T14:29:02.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32bb2a9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b40cd450 con 0x7f32b4153c80 2023-10-30T14:29:02.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32b9045640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b40c4a10 0x7f32b414c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:02.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32b9045640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b40c4a10 0x7f32b414c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55910/0 (socket says 172.21.15.57:55910) 2023-10-30T14:29:02.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32b9045640 1 -- 172.21.15.57:0/3837132754 learned_addr learned my addr 172.21.15.57:0/3837132754 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:02.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32b8844640 1 --2- 172.21.15.57:0/3837132754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32b4153c80 0x7f32b414c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:02.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32b9846640 1 --2- 172.21.15.57:0/3837132754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32b415a3b0 0x7f32b415ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:02.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32b9045640 1 -- 172.21.15.57:0/3837132754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32b4153c80 msgr2=0x7f32b414c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:02.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32b9045640 1 --2- 172.21.15.57:0/3837132754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32b4153c80 0x7f32b414c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:02.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32b9045640 1 -- 172.21.15.57:0/3837132754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32b415a3b0 msgr2=0x7f32b415ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:02.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32b9045640 1 --2- 172.21.15.57:0/3837132754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32b415a3b0 0x7f32b415ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:02.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32b9045640 1 -- 172.21.15.57:0/3837132754 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ac009b30 con 0x7f32b40c4a10 2023-10-30T14:29:02.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32b8844640 1 --2- 172.21.15.57:0/3837132754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32b4153c80 0x7f32b414c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:02.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32b9045640 1 --2- 172.21.15.57:0/3837132754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b40c4a10 0x7f32b414c3f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f32b000e930 tx=0x7f32b000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:02.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32a9ffb640 1 -- 172.21.15.57:0/3837132754 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b0009690 con 0x7f32b40c4a10 2023-10-30T14:29:02.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32a9ffb640 1 -- 172.21.15.57:0/3837132754 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b000c9a0 con 0x7f32b40c4a10 2023-10-30T14:29:02.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32bb2a9640 1 -- 172.21.15.57:0/3837132754 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b4158b20 con 0x7f32b40c4a10 2023-10-30T14:29:02.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32a9ffb640 1 -- 172.21.15.57:0/3837132754 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b000cd10 con 0x7f32b40c4a10 2023-10-30T14:29:02.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.913+0000 7f32bb2a9640 1 -- 172.21.15.57:0/3837132754 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b41efd90 con 0x7f32b40c4a10 2023-10-30T14:29:02.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.917+0000 7f32bb2a9640 1 -- 172.21.15.57:0/3837132754 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b415a790 con 0x7f32b40c4a10 2023-10-30T14:29:02.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.917+0000 7f32a9ffb640 1 -- 172.21.15.57:0/3837132754 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f32b0013470 con 0x7f32b40c4a10 2023-10-30T14:29:02.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.917+0000 7f32a9ffb640 1 --2- 172.21.15.57:0/3837132754 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32980664a0 0x7f3298068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:02.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.917+0000 7f32a9ffb640 1 -- 172.21.15.57:0/3837132754 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f32b0084800 con 0x7f32b40c4a10 2023-10-30T14:29:02.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.917+0000 7f32b8844640 1 --2- 172.21.15.57:0/3837132754 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32980664a0 0x7f3298068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:02.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.917+0000 7f32b8844640 1 --2- 172.21.15.57:0/3837132754 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32980664a0 0x7f3298068960 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f32b40b52f0 tx=0x7f32a4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:02.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:02.921+0000 7f32a9ffb640 1 -- 172.21.15.57:0/3837132754 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f32b0051970 con 0x7f32b40c4a10 2023-10-30T14:29:03.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:03.113+0000 7f32bb2a9640 1 -- 172.21.15.57:0/3837132754 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32b404e170 con 0x7f32b40c4a10 2023-10-30T14:29:03.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:03.113+0000 7f32a9ffb640 1 -- 172.21.15.57:0/3837132754 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f32b0054f90 con 0x7f32b40c4a10 2023-10-30T14:29:03.114 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:03.114 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:52.928186+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5797,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:03.114 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 91 2023-10-30T14:29:03.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:03.113+0000 7f32bb2a9640 1 -- 172.21.15.57:0/3837132754 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32980664a0 msgr2=0x7f3298068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:03.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:03.113+0000 7f32bb2a9640 1 --2- 172.21.15.57:0/3837132754 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32980664a0 0x7f3298068960 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f32b40b52f0 tx=0x7f32a4008040 comp rx=0 tx=0).stop 2023-10-30T14:29:03.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:03.113+0000 7f32bb2a9640 1 -- 172.21.15.57:0/3837132754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b40c4a10 msgr2=0x7f32b414c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:03.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:03.113+0000 7f32bb2a9640 1 --2- 172.21.15.57:0/3837132754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b40c4a10 0x7f32b414c3f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f32b000e930 tx=0x7f32b000edf0 comp rx=0 tx=0).stop 2023-10-30T14:29:03.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:03.113+0000 7f32bb2a9640 1 -- 172.21.15.57:0/3837132754 shutdown_connections 2023-10-30T14:29:03.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:03.113+0000 7f32bb2a9640 1 --2- 172.21.15.57:0/3837132754 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f32980664a0 0x7f3298068960 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:03.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:03.113+0000 7f32bb2a9640 1 --2- 172.21.15.57:0/3837132754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32b415a3b0 0x7f32b415ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:03.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:03.113+0000 7f32bb2a9640 1 --2- 172.21.15.57:0/3837132754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32b4153c80 0x7f32b414c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:03.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:03.113+0000 7f32bb2a9640 1 --2- 172.21.15.57:0/3837132754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b40c4a10 0x7f32b414c3f0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:03.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:03.113+0000 7f32bb2a9640 1 -- 172.21.15.57:0/3837132754 >> 172.21.15.57:0/3837132754 conn(0x7f32b40715c0 msgr2=0x7f32b40cb360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:03.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:03.113+0000 7f32bb2a9640 1 -- 172.21.15.57:0/3837132754 shutdown_connections 2023-10-30T14:29:03.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:03.113+0000 7f32bb2a9640 1 -- 172.21.15.57:0/3837132754 wait complete. 2023-10-30T14:29:03.135 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:29:05.136 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:05.223 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:29:05.227 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:29:05.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.289+0000 7f7c66a9c640 1 Processor -- start 2023-10-30T14:29:05.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.289+0000 7f7c66a9c640 1 -- start start 2023-10-30T14:29:05.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.289+0000 7f7c66a9c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7c601537f0 0x7f7c600701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:05.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.289+0000 7f7c66a9c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7c6015a2e0 0x7f7c600ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:05.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.289+0000 7f7c66a9c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c600c2aa0 con 0x7f7c600cd4a0 2023-10-30T14:29:05.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.289+0000 7f7c66a9c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c600cd8b0 con 0x7f7c6015a2e0 2023-10-30T14:29:05.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.293+0000 7f7c66a9c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c600c3ab0 con 0x7f7c601537f0 2023-10-30T14:29:05.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.293+0000 7f7c5ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7c6015a2e0 0x7f7c600ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:05.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.293+0000 7f7c5ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7c6015a2e0 0x7f7c600ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51522/0 (socket says 172.21.15.57:51522) 2023-10-30T14:29:05.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.293+0000 7f7c5ffff640 1 -- 172.21.15.57:0/2349356769 learned_addr learned my addr 172.21.15.57:0/2349356769 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:05.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.293+0000 7f7c64838640 1 --2- 172.21.15.57:0/2349356769 >> v2:172.21.15.88:3300/0 conn(0x7f7c601537f0 0x7f7c600701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:05.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.293+0000 7f7c5ffff640 1 -- 172.21.15.57:0/2349356769 >> v2:172.21.15.88:3300/0 conn(0x7f7c601537f0 msgr2=0x7f7c600701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:05.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.293+0000 7f7c5ffff640 1 --2- 172.21.15.57:0/2349356769 >> v2:172.21.15.88:3300/0 conn(0x7f7c601537f0 0x7f7c600701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:05.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.293+0000 7f7c5ffff640 1 -- 172.21.15.57:0/2349356769 >> v1:172.21.15.88:6789/0 conn(0x7f7c600cd4a0 legacy=0x7f7c6006b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:05.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.293+0000 7f7c5ffff640 1 -- 172.21.15.57:0/2349356769 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c600708e0 con 0x7f7c6015a2e0 2023-10-30T14:29:05.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.293+0000 7f7c5ffff640 1 --2- 172.21.15.57:0/2349356769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c6015a2e0 0x7f7c600ccf60 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f7c600cdb70 tx=0x7f7c4c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba9aaef398403b22 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:05.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.293+0000 7f7c5f7fe640 1 -- 172.21.15.57:0/2349356769 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c0a89f0 con 0x7f7c6015a2e0 2023-10-30T14:29:05.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.293+0000 7f7c5f7fe640 1 -- 172.21.15.57:0/2349356769 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4c0a8b50 con 0x7f7c6015a2e0 2023-10-30T14:29:05.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.293+0000 7f7c5f7fe640 1 -- 172.21.15.57:0/2349356769 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c0b0440 con 0x7f7c6015a2e0 2023-10-30T14:29:05.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c66a9c640 1 -- 172.21.15.57:0/2349356769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c6015a2e0 msgr2=0x7f7c600ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:05.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c66a9c640 1 --2- 172.21.15.57:0/2349356769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c6015a2e0 0x7f7c600ccf60 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f7c600cdb70 tx=0x7f7c4c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:29:05.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c66a9c640 1 -- 172.21.15.57:0/2349356769 shutdown_connections 2023-10-30T14:29:05.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c66a9c640 1 --2- 172.21.15.57:0/2349356769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c6015a2e0 0x7f7c600ccf60 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:05.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c66a9c640 1 --2- 172.21.15.57:0/2349356769 >> v2:172.21.15.88:3300/0 conn(0x7f7c601537f0 0x7f7c600701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:05.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c66a9c640 1 -- 172.21.15.57:0/2349356769 >> 172.21.15.57:0/2349356769 conn(0x7f7c60071fb0 msgr2=0x7f7c600723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:05.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c66a9c640 1 -- 172.21.15.57:0/2349356769 shutdown_connections 2023-10-30T14:29:05.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c66a9c640 1 -- 172.21.15.57:0/2349356769 wait complete. 2023-10-30T14:29:05.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c66a9c640 1 Processor -- start 2023-10-30T14:29:05.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c66a9c640 1 -- start start 2023-10-30T14:29:05.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c66a9c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c600cd4a0 0x7f7c6014cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:05.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c66a9c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7c601537f0 0x7f7c601f1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:05.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c66a9c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c6015a2e0 0x7f7c6006b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:05.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c66a9c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c600c82e0 con 0x7f7c6015a2e0 2023-10-30T14:29:05.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c66a9c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c60155610 con 0x7f7c600cd4a0 2023-10-30T14:29:05.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c66a9c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c600c5e20 con 0x7f7c601537f0 2023-10-30T14:29:05.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c65039640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c6015a2e0 0x7f7c6006b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:05.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c65039640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c6015a2e0 0x7f7c6006b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51532/0 (socket says 172.21.15.57:51532) 2023-10-30T14:29:05.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c65039640 1 -- 172.21.15.57:0/1793737331 learned_addr learned my addr 172.21.15.57:0/1793737331 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:05.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c64838640 1 --2- 172.21.15.57:0/1793737331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c600cd4a0 0x7f7c6014cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:05.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.297+0000 7f7c5ffff640 1 --2- 172.21.15.57:0/1793737331 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7c601537f0 0x7f7c601f1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:05.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.301+0000 7f7c65039640 1 -- 172.21.15.57:0/1793737331 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7c601537f0 msgr2=0x7f7c601f1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:05.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.301+0000 7f7c65039640 1 --2- 172.21.15.57:0/1793737331 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7c601537f0 0x7f7c601f1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:05.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.301+0000 7f7c65039640 1 -- 172.21.15.57:0/1793737331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c600cd4a0 msgr2=0x7f7c6014cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:05.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.301+0000 7f7c65039640 1 --2- 172.21.15.57:0/1793737331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c600cd4a0 0x7f7c6014cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:05.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.301+0000 7f7c65039640 1 -- 172.21.15.57:0/1793737331 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c4c0094d0 con 0x7f7c6015a2e0 2023-10-30T14:29:05.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.301+0000 7f7c65039640 1 --2- 172.21.15.57:0/1793737331 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c6015a2e0 0x7f7c6006b210 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800c760 tx=0x7f7c5800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:05.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.301+0000 7f7c5d7fa640 1 -- 172.21.15.57:0/1793737331 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c58003b80 con 0x7f7c6015a2e0 2023-10-30T14:29:05.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.301+0000 7f7c5d7fa640 1 -- 172.21.15.57:0/1793737331 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c58003ce0 con 0x7f7c6015a2e0 2023-10-30T14:29:05.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.301+0000 7f7c66a9c640 1 -- 172.21.15.57:0/1793737331 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c6014d2c0 con 0x7f7c6015a2e0 2023-10-30T14:29:05.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.301+0000 7f7c5d7fa640 1 -- 172.21.15.57:0/1793737331 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5800aa30 con 0x7f7c6015a2e0 2023-10-30T14:29:05.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.301+0000 7f7c66a9c640 1 -- 172.21.15.57:0/1793737331 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c600c6050 con 0x7f7c6015a2e0 2023-10-30T14:29:05.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.301+0000 7f7c66a9c640 1 -- 172.21.15.57:0/1793737331 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c6004e170 con 0x7f7c6015a2e0 2023-10-30T14:29:05.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.301+0000 7f7c5d7fa640 1 -- 172.21.15.57:0/1793737331 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7c5800ab90 con 0x7f7c6015a2e0 2023-10-30T14:29:05.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.301+0000 7f7c5d7fa640 1 --2- 172.21.15.57:0/1793737331 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7c440664a0 0x7f7c44068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:05.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.301+0000 7f7c5d7fa640 1 -- 172.21.15.57:0/1793737331 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f7c58083f30 con 0x7f7c6015a2e0 2023-10-30T14:29:05.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.301+0000 7f7c64838640 1 --2- 172.21.15.57:0/1793737331 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7c440664a0 0x7f7c44068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:05.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.305+0000 7f7c64838640 1 --2- 172.21.15.57:0/1793737331 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7c440664a0 0x7f7c44068960 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7c54007a60 tx=0x7f7c54008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:05.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.305+0000 7f7c5d7fa640 1 -- 172.21.15.57:0/1793737331 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7c58051000 con 0x7f7c6015a2e0 2023-10-30T14:29:05.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.497+0000 7f7c66a9c640 1 -- 172.21.15.57:0/1793737331 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c60073de0 con 0x7f7c6015a2e0 2023-10-30T14:29:05.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.497+0000 7f7c5d7fa640 1 -- 172.21.15.57:0/1793737331 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f7c58054620 con 0x7f7c6015a2e0 2023-10-30T14:29:05.498 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:05.498 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:52.928186+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5797,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:05.499 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 91 2023-10-30T14:29:05.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.497+0000 7f7c66a9c640 1 -- 172.21.15.57:0/1793737331 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7c440664a0 msgr2=0x7f7c44068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:05.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.497+0000 7f7c66a9c640 1 --2- 172.21.15.57:0/1793737331 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7c440664a0 0x7f7c44068960 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7c54007a60 tx=0x7f7c54008040 comp rx=0 tx=0).stop 2023-10-30T14:29:05.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.497+0000 7f7c66a9c640 1 -- 172.21.15.57:0/1793737331 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c6015a2e0 msgr2=0x7f7c6006b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:05.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.497+0000 7f7c66a9c640 1 --2- 172.21.15.57:0/1793737331 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c6015a2e0 0x7f7c6006b210 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800c760 tx=0x7f7c5800cc20 comp rx=0 tx=0).stop 2023-10-30T14:29:05.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.497+0000 7f7c66a9c640 1 -- 172.21.15.57:0/1793737331 shutdown_connections 2023-10-30T14:29:05.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.497+0000 7f7c66a9c640 1 --2- 172.21.15.57:0/1793737331 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7c440664a0 0x7f7c44068960 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:05.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.497+0000 7f7c66a9c640 1 --2- 172.21.15.57:0/1793737331 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c6015a2e0 0x7f7c6006b210 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:05.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.497+0000 7f7c66a9c640 1 --2- 172.21.15.57:0/1793737331 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7c601537f0 0x7f7c601f1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:05.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.497+0000 7f7c66a9c640 1 --2- 172.21.15.57:0/1793737331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c600cd4a0 0x7f7c6014cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:05.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.497+0000 7f7c66a9c640 1 -- 172.21.15.57:0/1793737331 >> 172.21.15.57:0/1793737331 conn(0x7f7c60071fb0 msgr2=0x7f7c60072390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:05.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.497+0000 7f7c66a9c640 1 -- 172.21.15.57:0/1793737331 shutdown_connections 2023-10-30T14:29:05.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:05.497+0000 7f7c66a9c640 1 -- 172.21.15.57:0/1793737331 wait complete. 2023-10-30T14:29:05.519 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:29:07.521 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:07.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.673+0000 7f64d2a79640 1 Processor -- start 2023-10-30T14:29:07.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.673+0000 7f64d2a79640 1 -- start start 2023-10-30T14:29:07.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64d2a79640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f64cc0701a0 0x7f64cc15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:07.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64d2a79640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc06b2f0 0x7f64cc06b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:07.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64d2a79640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64cc0c2aa0 con 0x7f64cc1537f0 2023-10-30T14:29:07.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64d2a79640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64cc0c3ab0 con 0x7f64cc0701a0 2023-10-30T14:29:07.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64d2a79640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc15a820 con 0x7f64cc06b2f0 2023-10-30T14:29:07.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64d0815640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f64cc1537f0 0x7f64cc0ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:49338/0 (socket says 172.21.15.57:49338) 2023-10-30T14:29:07.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64d0815640 1 -- 172.21.15.57:0/548119217 learned_addr learned my addr 172.21.15.57:0/548119217 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:29:07.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64cbfff640 1 --2- 172.21.15.57:0/548119217 >> v2:172.21.15.88:3300/0 conn(0x7f64cc0701a0 0x7f64cc15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:07.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64d1016640 1 --2- 172.21.15.57:0/548119217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc06b2f0 0x7f64cc06b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:07.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64cb7fe640 1 -- 172.21.15.57:0/548119217 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2153112023 0 0) 0x7f64cc0c2aa0 con 0x7f64cc1537f0 2023-10-30T14:29:07.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64cb7fe640 1 -- 172.21.15.57:0/548119217 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64bc003530 con 0x7f64cc1537f0 2023-10-30T14:29:07.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64cb7fe640 1 -- 172.21.15.57:0/548119217 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3256625599 0 0) 0x7f64bc003530 con 0x7f64cc1537f0 2023-10-30T14:29:07.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64cb7fe640 1 -- 172.21.15.57:0/548119217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc06b2f0 msgr2=0x7f64cc06b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:07.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64cb7fe640 1 --2- 172.21.15.57:0/548119217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc06b2f0 0x7f64cc06b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:07.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64cb7fe640 1 -- 172.21.15.57:0/548119217 >> v2:172.21.15.88:3300/0 conn(0x7f64cc0701a0 msgr2=0x7f64cc15a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:07.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64cb7fe640 1 --2- 172.21.15.57:0/548119217 >> v2:172.21.15.88:3300/0 conn(0x7f64cc0701a0 0x7f64cc15a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:07.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64cb7fe640 1 -- 172.21.15.57:0/548119217 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc06bc80 con 0x7f64cc1537f0 2023-10-30T14:29:07.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64cb7fe640 1 -- 172.21.15.57:0/548119217 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f64c0001680 con 0x7f64cc1537f0 2023-10-30T14:29:07.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64cb7fe640 1 -- 172.21.15.57:0/548119217 >> v1:172.21.15.57:6789/0 conn(0x7f64cc1537f0 legacy=0x7f64cc0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:07.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64cb7fe640 1 --2- 172.21.15.57:0/548119217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc06b2f0 0x7f64bc003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:07.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64cb7fe640 1 --2- 172.21.15.57:0/548119217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64bc003f00 0x7f64bc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:07.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64cb7fe640 1 --2- 172.21.15.57:0/548119217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64bc004d10 0x7f64bc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:07.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64cb7fe640 1 -- 172.21.15.57:0/548119217 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc15a820 con 0x7f64bc004d10 2023-10-30T14:29:07.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64cb7fe640 1 -- 172.21.15.57:0/548119217 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc00b990 con 0x7f64bc003f00 2023-10-30T14:29:07.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64cb7fe640 1 -- 172.21.15.57:0/548119217 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc00bb00 con 0x7f64cc06b2f0 2023-10-30T14:29:07.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64cbfff640 1 --2- 172.21.15.57:0/548119217 >> v2:172.21.15.88:3300/0 conn(0x7f64cc0701a0 0x7f64cc15a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:29:07.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64d1016640 1 --2- 172.21.15.57:0/548119217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64bc004d10 0x7f64bc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:07.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64d0815640 1 --2- 172.21.15.57:0/548119217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc06b2f0 0x7f64bc003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:07.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64cbfff640 1 --2- 172.21.15.57:0/548119217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64bc003f00 0x7f64bc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:07.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64d1016640 1 -- 172.21.15.57:0/548119217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc06b2f0 msgr2=0x7f64bc003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:07.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64d1016640 1 --2- 172.21.15.57:0/548119217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc06b2f0 0x7f64bc003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:07.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64d1016640 1 -- 172.21.15.57:0/548119217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64bc003f00 msgr2=0x7f64bc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:07.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64d1016640 1 --2- 172.21.15.57:0/548119217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64bc003f00 0x7f64bc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:07.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64d1016640 1 -- 172.21.15.57:0/548119217 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64bc003530 con 0x7f64bc004d10 2023-10-30T14:29:07.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.677+0000 7f64d0815640 1 --2- 172.21.15.57:0/548119217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc06b2f0 0x7f64bc003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:07.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.681+0000 7f64d1016640 1 --2- 172.21.15.57:0/548119217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64bc004d10 0x7f64bc005190 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009fc0 tx=0x7f64c401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1547adb9d4d2cbf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:07.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.681+0000 7f64cb7fe640 1 -- 172.21.15.57:0/548119217 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c40a5c70 con 0x7f64bc004d10 2023-10-30T14:29:07.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.681+0000 7f64cb7fe640 1 -- 172.21.15.57:0/548119217 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c400f040 con 0x7f64bc004d10 2023-10-30T14:29:07.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.681+0000 7f64cb7fe640 1 -- 172.21.15.57:0/548119217 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c40ad680 con 0x7f64bc004d10 2023-10-30T14:29:07.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.681+0000 7f64d2a79640 1 -- 172.21.15.57:0/548119217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64bc004d10 msgr2=0x7f64bc005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:07.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.681+0000 7f64d2a79640 1 --2- 172.21.15.57:0/548119217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64bc004d10 0x7f64bc005190 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009fc0 tx=0x7f64c401cc60 comp rx=0 tx=0).stop 2023-10-30T14:29:07.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.681+0000 7f64d1016640 1 -- 172.21.15.57:0/548119217 reap_dead start 2023-10-30T14:29:07.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.681+0000 7f64d2a79640 1 -- 172.21.15.57:0/548119217 shutdown_connections 2023-10-30T14:29:07.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.681+0000 7f64d2a79640 1 -- 172.21.15.57:0/548119217 >> 172.21.15.57:0/548119217 conn(0x7f64cc071fb0 msgr2=0x7f64cc0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:07.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.681+0000 7f64d2a79640 1 -- 172.21.15.57:0/548119217 shutdown_connections 2023-10-30T14:29:07.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.681+0000 7f64d2a79640 1 -- 172.21.15.57:0/548119217 wait complete. 2023-10-30T14:29:07.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.681+0000 7f64d2a79640 1 Processor -- start 2023-10-30T14:29:07.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.681+0000 7f64d2a79640 1 -- start start 2023-10-30T14:29:07.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64d2a79640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64bc004d10 0x7f64cc0c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:07.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64d2a79640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64cc149930 0x7f64cc149d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:07.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64d2a79640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc0cc570 0x7f64cc0cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:07.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64d2a79640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc14a2b0 con 0x7f64bc004d10 2023-10-30T14:29:07.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64d2a79640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc1ef6e0 con 0x7f64cc149930 2023-10-30T14:29:07.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64d2a79640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc1ef180 con 0x7f64cc0cc570 2023-10-30T14:29:07.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64d0815640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64bc004d10 0x7f64cc0c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:07.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64d0815640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64bc004d10 0x7f64cc0c8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51544/0 (socket says 172.21.15.57:51544) 2023-10-30T14:29:07.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64d0815640 1 -- 172.21.15.57:0/1732952763 learned_addr learned my addr 172.21.15.57:0/1732952763 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:07.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64cbfff640 1 --2- 172.21.15.57:0/1732952763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64cc149930 0x7f64cc149d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:07.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64d1016640 1 --2- 172.21.15.57:0/1732952763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc0cc570 0x7f64cc0cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:07.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64d0815640 1 -- 172.21.15.57:0/1732952763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc0cc570 msgr2=0x7f64cc0cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:07.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64d0815640 1 --2- 172.21.15.57:0/1732952763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc0cc570 0x7f64cc0cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:07.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64d0815640 1 -- 172.21.15.57:0/1732952763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64cc149930 msgr2=0x7f64cc149d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:07.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64d0815640 1 --2- 172.21.15.57:0/1732952763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64cc149930 0x7f64cc149d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:07.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64d0815640 1 -- 172.21.15.57:0/1732952763 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc0c70c0 con 0x7f64bc004d10 2023-10-30T14:29:07.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64d0815640 1 --2- 172.21.15.57:0/1732952763 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64bc004d10 0x7f64cc0c8f60 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f64c000e880 tx=0x7f64c000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:07.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64c97fa640 1 -- 172.21.15.57:0/1732952763 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c000c8d0 con 0x7f64bc004d10 2023-10-30T14:29:07.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64c97fa640 1 -- 172.21.15.57:0/1732952763 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c000ca30 con 0x7f64bc004d10 2023-10-30T14:29:07.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64c97fa640 1 -- 172.21.15.57:0/1732952763 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c000cda0 con 0x7f64bc004d10 2023-10-30T14:29:07.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64d2a79640 1 -- 172.21.15.57:0/1732952763 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64cc1ef400 con 0x7f64bc004d10 2023-10-30T14:29:07.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64d2a79640 1 -- 172.21.15.57:0/1732952763 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64cc0c8a40 con 0x7f64bc004d10 2023-10-30T14:29:07.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.685+0000 7f64d2a79640 1 -- 172.21.15.57:0/1732952763 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64cc04e170 con 0x7f64bc004d10 2023-10-30T14:29:07.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.689+0000 7f64c97fa640 1 -- 172.21.15.57:0/1732952763 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f64c001d460 con 0x7f64bc004d10 2023-10-30T14:29:07.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.689+0000 7f64c97fa640 1 --2- 172.21.15.57:0/1732952763 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f64b00664a0 0x7f64b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:07.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.689+0000 7f64c97fa640 1 -- 172.21.15.57:0/1732952763 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f64c0084300 con 0x7f64bc004d10 2023-10-30T14:29:07.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.689+0000 7f64cbfff640 1 --2- 172.21.15.57:0/1732952763 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f64b00664a0 0x7f64b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:07.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.689+0000 7f64cbfff640 1 --2- 172.21.15.57:0/1732952763 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f64b00664a0 0x7f64b0068960 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f64cc1ec360 tx=0x7f64b8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:07.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.693+0000 7f64c97fa640 1 -- 172.21.15.57:0/1732952763 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f64c0051440 con 0x7f64bc004d10 2023-10-30T14:29:07.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.885+0000 7f64d2a79640 1 -- 172.21.15.57:0/1732952763 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64cc0c1eb0 con 0x7f64bc004d10 2023-10-30T14:29:07.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.885+0000 7f64c97fa640 1 -- 172.21.15.57:0/1732952763 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f64c0054a60 con 0x7f64bc004d10 2023-10-30T14:29:07.886 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:07.886 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5315,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:28:52.928186+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5797,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5797":{"gid":5797,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":43,"addr":"172.21.15.57:6834/2940934016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2940934016},{"type":"v1","addr":"172.21.15.57:6834","nonce":2940934016}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:07.887 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 91 2023-10-30T14:29:07.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.885+0000 7f64d2a79640 1 -- 172.21.15.57:0/1732952763 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f64b00664a0 msgr2=0x7f64b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:07.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.885+0000 7f64d2a79640 1 --2- 172.21.15.57:0/1732952763 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f64b00664a0 0x7f64b0068960 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f64cc1ec360 tx=0x7f64b8008040 comp rx=0 tx=0).stop 2023-10-30T14:29:07.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.885+0000 7f64d2a79640 1 -- 172.21.15.57:0/1732952763 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64bc004d10 msgr2=0x7f64cc0c8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:07.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.885+0000 7f64d2a79640 1 --2- 172.21.15.57:0/1732952763 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64bc004d10 0x7f64cc0c8f60 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f64c000e880 tx=0x7f64c000ed40 comp rx=0 tx=0).stop 2023-10-30T14:29:07.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.885+0000 7f64d2a79640 1 -- 172.21.15.57:0/1732952763 shutdown_connections 2023-10-30T14:29:07.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.885+0000 7f64d2a79640 1 --2- 172.21.15.57:0/1732952763 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f64b00664a0 0x7f64b0068960 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:07.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.885+0000 7f64d2a79640 1 --2- 172.21.15.57:0/1732952763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64cc0cc570 0x7f64cc0cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:07.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.885+0000 7f64d2a79640 1 --2- 172.21.15.57:0/1732952763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64cc149930 0x7f64cc149d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:07.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.885+0000 7f64d2a79640 1 --2- 172.21.15.57:0/1732952763 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64bc004d10 0x7f64cc0c8f60 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:07.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.885+0000 7f64d2a79640 1 -- 172.21.15.57:0/1732952763 >> 172.21.15.57:0/1732952763 conn(0x7f64cc071fb0 msgr2=0x7f64cc06c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:07.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.885+0000 7f64d2a79640 1 -- 172.21.15.57:0/1732952763 shutdown_connections 2023-10-30T14:29:07.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:07.885+0000 7f64d2a79640 1 -- 172.21.15.57:0/1732952763 wait complete. 2023-10-30T14:29:07.908 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:29:09.909 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:10.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.061+0000 7f7d774ad640 1 Processor -- start 2023-10-30T14:29:10.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.061+0000 7f7d774ad640 1 -- start start 2023-10-30T14:29:10.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.061+0000 7f7d774ad640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7d7015ac70 0x7f7d7015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:10.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.061+0000 7f7d774ad640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7d7006aeb0 0x7f7d7006b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:10.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.061+0000 7f7d774ad640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d7015dcc0 con 0x7f7d700c8eb0 2023-10-30T14:29:10.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.061+0000 7f7d774ad640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d70159800 con 0x7f7d7006aeb0 2023-10-30T14:29:10.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.061+0000 7f7d774ad640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d70159d80 con 0x7f7d7015ac70 2023-10-30T14:29:10.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.061+0000 7f7d764ab640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f7d700c8eb0 0x7f7d7015a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:49340/0 (socket says 172.21.15.57:49340) 2023-10-30T14:29:10.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.061+0000 7f7d764ab640 1 -- 172.21.15.57:0/1626572913 learned_addr learned my addr 172.21.15.57:0/1626572913 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:29:10.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.061+0000 7f7d76cac640 1 --2- 172.21.15.57:0/1626572913 >> v2:172.21.15.57:3300/0 conn(0x7f7d7006aeb0 0x7f7d7006b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:10.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.061+0000 7f7d75caa640 1 --2- 172.21.15.57:0/1626572913 >> v2:172.21.15.88:3300/0 conn(0x7f7d7015ac70 0x7f7d7015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:10.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.061+0000 7f7d754a9640 1 -- 172.21.15.57:0/1626572913 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635671726 0 0) 0x7f7d6c001120 con 0x7f7d700c8eb0 2023-10-30T14:29:10.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.061+0000 7f7d76cac640 1 -- 172.21.15.57:0/1626572913 >> v2:172.21.15.88:3300/0 conn(0x7f7d7015ac70 msgr2=0x7f7d7015cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:10.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.061+0000 7f7d76cac640 1 --2- 172.21.15.57:0/1626572913 >> v2:172.21.15.88:3300/0 conn(0x7f7d7015ac70 0x7f7d7015cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.061+0000 7f7d76cac640 1 -- 172.21.15.57:0/1626572913 >> v1:172.21.15.57:6789/0 conn(0x7f7d700c8eb0 legacy=0x7f7d7015a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:10.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.061+0000 7f7d76cac640 1 -- 172.21.15.57:0/1626572913 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7015d420 con 0x7f7d7006aeb0 2023-10-30T14:29:10.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.065+0000 7f7d76cac640 1 --2- 172.21.15.57:0/1626572913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d7006aeb0 0x7f7d7006b300 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7d700b4130 tx=0x7f7d6801f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=573a66dc1985afaa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:10.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.065+0000 7f7d754a9640 1 -- 172.21.15.57:0/1626572913 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d680b0440 con 0x7f7d7006aeb0 2023-10-30T14:29:10.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.065+0000 7f7d754a9640 1 -- 172.21.15.57:0/1626572913 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d680b05a0 con 0x7f7d7006aeb0 2023-10-30T14:29:10.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.065+0000 7f7d754a9640 1 -- 172.21.15.57:0/1626572913 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d680b0930 con 0x7f7d7006aeb0 2023-10-30T14:29:10.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.065+0000 7f7d774ad640 1 -- 172.21.15.57:0/1626572913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d7006aeb0 msgr2=0x7f7d7006b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:10.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.065+0000 7f7d774ad640 1 --2- 172.21.15.57:0/1626572913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d7006aeb0 0x7f7d7006b300 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7d700b4130 tx=0x7f7d6801f0f0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.065+0000 7f7d774ad640 1 -- 172.21.15.57:0/1626572913 shutdown_connections 2023-10-30T14:29:10.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.065+0000 7f7d774ad640 1 --2- 172.21.15.57:0/1626572913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d7006aeb0 0x7f7d7006b300 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.065+0000 7f7d774ad640 1 --2- 172.21.15.57:0/1626572913 >> v2:172.21.15.88:3300/0 conn(0x7f7d7015ac70 0x7f7d7015cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.065+0000 7f7d774ad640 1 -- 172.21.15.57:0/1626572913 >> 172.21.15.57:0/1626572913 conn(0x7f7d700cef00 msgr2=0x7f7d700cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:10.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.065+0000 7f7d774ad640 1 -- 172.21.15.57:0/1626572913 shutdown_connections 2023-10-30T14:29:10.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.065+0000 7f7d774ad640 1 -- 172.21.15.57:0/1626572913 wait complete. 2023-10-30T14:29:10.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d774ad640 1 Processor -- start 2023-10-30T14:29:10.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d774ad640 1 -- start start 2023-10-30T14:29:10.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d774ad640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7d7006aeb0 0x7f7d7014f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:10.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d774ad640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d700c8eb0 0x7f7d7014ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:10.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d774ad640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d7015ac70 0x7f7d7006a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:10.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d774ad640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d70151130 con 0x7f7d7015ac70 2023-10-30T14:29:10.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d774ad640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7014fec0 con 0x7f7d700c8eb0 2023-10-30T14:29:10.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d774ad640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7014e5f0 con 0x7f7d7006aeb0 2023-10-30T14:29:10.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d76cac640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d7015ac70 0x7f7d7006a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:10.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d76cac640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d7015ac70 0x7f7d7006a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51552/0 (socket says 172.21.15.57:51552) 2023-10-30T14:29:10.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d76cac640 1 -- 172.21.15.57:0/3199467409 learned_addr learned my addr 172.21.15.57:0/3199467409 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:10.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d75caa640 1 --2- 172.21.15.57:0/3199467409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d700c8eb0 0x7f7d7014ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:10.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d764ab640 1 --2- 172.21.15.57:0/3199467409 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7d7006aeb0 0x7f7d7014f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:10.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d76cac640 1 -- 172.21.15.57:0/3199467409 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7d7006aeb0 msgr2=0x7f7d7014f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:10.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d76cac640 1 --2- 172.21.15.57:0/3199467409 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7d7006aeb0 0x7f7d7014f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d76cac640 1 -- 172.21.15.57:0/3199467409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d700c8eb0 msgr2=0x7f7d7014ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:10.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d76cac640 1 --2- 172.21.15.57:0/3199467409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d700c8eb0 0x7f7d7014ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d76cac640 1 -- 172.21.15.57:0/3199467409 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d68009a70 con 0x7f7d7015ac70 2023-10-30T14:29:10.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d75caa640 1 --2- 172.21.15.57:0/3199467409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d700c8eb0 0x7f7d7014ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:10.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d764ab640 1 --2- 172.21.15.57:0/3199467409 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7d7006aeb0 0x7f7d7014f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:10.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d76cac640 1 --2- 172.21.15.57:0/3199467409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d7015ac70 0x7f7d7006a0f0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f7d6801bd00 tx=0x7f7d6801be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:10.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d66ffd640 1 -- 172.21.15.57:0/3199467409 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d680b0440 con 0x7f7d7015ac70 2023-10-30T14:29:10.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d66ffd640 1 -- 172.21.15.57:0/3199467409 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d680bc860 con 0x7f7d7015ac70 2023-10-30T14:29:10.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d774ad640 1 -- 172.21.15.57:0/3199467409 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d701508d0 con 0x7f7d7015ac70 2023-10-30T14:29:10.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d66ffd640 1 -- 172.21.15.57:0/3199467409 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d680bcc10 con 0x7f7d7015ac70 2023-10-30T14:29:10.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.069+0000 7f7d774ad640 1 -- 172.21.15.57:0/3199467409 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d70150e50 con 0x7f7d7015ac70 2023-10-30T14:29:10.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.073+0000 7f7d774ad640 1 -- 172.21.15.57:0/3199467409 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d700b4050 con 0x7f7d7015ac70 2023-10-30T14:29:10.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.073+0000 7f7d66ffd640 1 -- 172.21.15.57:0/3199467409 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7d680c7460 con 0x7f7d7015ac70 2023-10-30T14:29:10.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.073+0000 7f7d66ffd640 1 --2- 172.21.15.57:0/3199467409 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7d540664a0 0x7f7d54068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:10.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.073+0000 7f7d66ffd640 1 -- 172.21.15.57:0/3199467409 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f7d681230e0 con 0x7f7d7015ac70 2023-10-30T14:29:10.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.073+0000 7f7d764ab640 1 --2- 172.21.15.57:0/3199467409 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7d540664a0 0x7f7d54068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:10.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.073+0000 7f7d764ab640 1 --2- 172.21.15.57:0/3199467409 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7d540664a0 0x7f7d54068960 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c000f70 tx=0x7f7d6c007a60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:10.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.077+0000 7f7d66ffd640 1 -- 172.21.15.57:0/3199467409 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7d680f0260 con 0x7f7d7015ac70 2023-10-30T14:29:10.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.273+0000 7f7d774ad640 1 -- 172.21.15.57:0/3199467409 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d7014e730 con 0x7f7d7015ac70 2023-10-30T14:29:10.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.273+0000 7f7d66ffd640 1 -- 172.21.15.57:0/3199467409 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5731 (secure 0 0 0) 0x7f7d7014e730 con 0x7f7d7015ac70 2023-10-30T14:29:10.276 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:10.276 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:09.852404+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:10.276 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 92 2023-10-30T14:29:10.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.277+0000 7f7d774ad640 1 -- 172.21.15.57:0/3199467409 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7d540664a0 msgr2=0x7f7d54068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:10.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.277+0000 7f7d774ad640 1 --2- 172.21.15.57:0/3199467409 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7d540664a0 0x7f7d54068960 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c000f70 tx=0x7f7d6c007a60 comp rx=0 tx=0).stop 2023-10-30T14:29:10.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.277+0000 7f7d774ad640 1 -- 172.21.15.57:0/3199467409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d7015ac70 msgr2=0x7f7d7006a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:10.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.277+0000 7f7d774ad640 1 --2- 172.21.15.57:0/3199467409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d7015ac70 0x7f7d7006a0f0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f7d6801bd00 tx=0x7f7d6801be90 comp rx=0 tx=0).stop 2023-10-30T14:29:10.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.277+0000 7f7d774ad640 1 -- 172.21.15.57:0/3199467409 shutdown_connections 2023-10-30T14:29:10.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.277+0000 7f7d774ad640 1 --2- 172.21.15.57:0/3199467409 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7d540664a0 0x7f7d54068960 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.277+0000 7f7d774ad640 1 --2- 172.21.15.57:0/3199467409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d7015ac70 0x7f7d7006a0f0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.277+0000 7f7d774ad640 1 --2- 172.21.15.57:0/3199467409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d700c8eb0 0x7f7d7014ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.277+0000 7f7d774ad640 1 --2- 172.21.15.57:0/3199467409 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7d7006aeb0 0x7f7d7014f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.277+0000 7f7d774ad640 1 -- 172.21.15.57:0/3199467409 >> 172.21.15.57:0/3199467409 conn(0x7f7d700cef00 msgr2=0x7f7d700c84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:10.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.277+0000 7f7d774ad640 1 -- 172.21.15.57:0/3199467409 shutdown_connections 2023-10-30T14:29:10.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.277+0000 7f7d774ad640 1 -- 172.21.15.57:0/3199467409 wait complete. 2023-10-30T14:29:10.568 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T14:29:10.568 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:29:10.568 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:10.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.717+0000 7f58f3394640 1 Processor -- start 2023-10-30T14:29:10.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.717+0000 7f58f3394640 1 -- start start 2023-10-30T14:29:10.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.717+0000 7f58f3394640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f58ec153c80 0x7f58ec15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:10.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.717+0000 7f58f3394640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f58ec0c4a10 0x7f58ec0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:10.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.717+0000 7f58f3394640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58ec15a3b0 0x7f58ec0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:10.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.717+0000 7f58f3394640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ec15dec0 con 0x7f58ec153c80 2023-10-30T14:29:10.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.717+0000 7f58f3394640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ec1599a0 con 0x7f58ec0c4a10 2023-10-30T14:29:10.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.717+0000 7f58f3394640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec159f20 con 0x7f58ec15a3b0 2023-10-30T14:29:10.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.717+0000 7f58f1130640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f58ec153c80 0x7f58ec15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:10.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.717+0000 7f58f1130640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f58ec153c80 0x7f58ec15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51558/0 (socket says 172.21.15.57:51558) 2023-10-30T14:29:10.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.717+0000 7f58f1130640 1 -- 172.21.15.57:0/1543915203 learned_addr learned my addr 172.21.15.57:0/1543915203 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:10.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.717+0000 7f58f092f640 1 --2- 172.21.15.57:0/1543915203 >> v2:172.21.15.88:3300/0 conn(0x7f58ec0c4a10 0x7f58ec0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:10.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.717+0000 7f58f1931640 1 --2- 172.21.15.57:0/1543915203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58ec15a3b0 0x7f58ec0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:10.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.717+0000 7f58f1130640 1 -- 172.21.15.57:0/1543915203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58ec15a3b0 msgr2=0x7f58ec0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:10.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.717+0000 7f58f1130640 1 --2- 172.21.15.57:0/1543915203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58ec15a3b0 0x7f58ec0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.717+0000 7f58f1130640 1 -- 172.21.15.57:0/1543915203 >> v2:172.21.15.88:3300/0 conn(0x7f58ec0c4a10 msgr2=0x7f58ec0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:10.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.717+0000 7f58f1130640 1 --2- 172.21.15.57:0/1543915203 >> v2:172.21.15.88:3300/0 conn(0x7f58ec0c4a10 0x7f58ec0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.717+0000 7f58f1130640 1 -- 172.21.15.57:0/1543915203 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ec0c47e0 con 0x7f58ec153c80 2023-10-30T14:29:10.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.721+0000 7f58f1130640 1 --2- 172.21.15.57:0/1543915203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58ec153c80 0x7f58ec15cf90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f58ec06be50 tx=0x7f58e801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80bd6551a19a4da8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:10.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.721+0000 7f58e3fff640 1 -- 172.21.15.57:0/1543915203 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e80a89b0 con 0x7f58ec153c80 2023-10-30T14:29:10.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.721+0000 7f58e3fff640 1 -- 172.21.15.57:0/1543915203 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e80a8b10 con 0x7f58ec153c80 2023-10-30T14:29:10.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.721+0000 7f58e3fff640 1 -- 172.21.15.57:0/1543915203 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e80a8ea0 con 0x7f58ec153c80 2023-10-30T14:29:10.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.721+0000 7f58f3394640 1 -- 172.21.15.57:0/1543915203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58ec153c80 msgr2=0x7f58ec15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:10.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.721+0000 7f58f3394640 1 --2- 172.21.15.57:0/1543915203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58ec153c80 0x7f58ec15cf90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f58ec06be50 tx=0x7f58e801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.721+0000 7f58f3394640 1 -- 172.21.15.57:0/1543915203 shutdown_connections 2023-10-30T14:29:10.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.721+0000 7f58f3394640 1 --2- 172.21.15.57:0/1543915203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58ec15a3b0 0x7f58ec0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.721+0000 7f58f3394640 1 --2- 172.21.15.57:0/1543915203 >> v2:172.21.15.88:3300/0 conn(0x7f58ec0c4a10 0x7f58ec0c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.721+0000 7f58f3394640 1 --2- 172.21.15.57:0/1543915203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58ec153c80 0x7f58ec15cf90 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.721+0000 7f58f3394640 1 -- 172.21.15.57:0/1543915203 >> 172.21.15.57:0/1543915203 conn(0x7f58ec0715c0 msgr2=0x7f58ec0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:10.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.721+0000 7f58f3394640 1 -- 172.21.15.57:0/1543915203 shutdown_connections 2023-10-30T14:29:10.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.721+0000 7f58f3394640 1 -- 172.21.15.57:0/1543915203 wait complete. 2023-10-30T14:29:10.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f3394640 1 Processor -- start 2023-10-30T14:29:10.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f3394640 1 -- start start 2023-10-30T14:29:10.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f3394640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58ec0c4a10 0x7f58ec0c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:10.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f3394640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58ec153c80 0x7f58ec151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:10.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f3394640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58ec15a3b0 0x7f58ec1f24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:10.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f3394640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec0c9110 con 0x7f58ec0c4a10 2023-10-30T14:29:10.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f3394640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec0c93d0 con 0x7f58ec15a3b0 2023-10-30T14:29:10.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f3394640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec15b1b0 con 0x7f58ec153c80 2023-10-30T14:29:10.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f1130640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58ec0c4a10 0x7f58ec0c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:10.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f1130640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58ec0c4a10 0x7f58ec0c8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51562/0 (socket says 172.21.15.57:51562) 2023-10-30T14:29:10.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f1130640 1 -- 172.21.15.57:0/3921628608 learned_addr learned my addr 172.21.15.57:0/3921628608 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:10.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f092f640 1 --2- 172.21.15.57:0/3921628608 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58ec153c80 0x7f58ec151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:10.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f1931640 1 --2- 172.21.15.57:0/3921628608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58ec15a3b0 0x7f58ec1f24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:10.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f1130640 1 -- 172.21.15.57:0/3921628608 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58ec153c80 msgr2=0x7f58ec151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:10.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f1130640 1 --2- 172.21.15.57:0/3921628608 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58ec153c80 0x7f58ec151e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f1130640 1 -- 172.21.15.57:0/3921628608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58ec15a3b0 msgr2=0x7f58ec1f24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:10.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f1130640 1 --2- 172.21.15.57:0/3921628608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58ec15a3b0 0x7f58ec1f24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f1130640 1 -- 172.21.15.57:0/3921628608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e80094d0 con 0x7f58ec0c4a10 2023-10-30T14:29:10.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f1931640 1 --2- 172.21.15.57:0/3921628608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58ec15a3b0 0x7f58ec1f24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:10.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f1130640 1 --2- 172.21.15.57:0/3921628608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58ec0c4a10 0x7f58ec0c8af0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f58e801d3a0 tx=0x7f58e801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:10.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58e1ffb640 1 -- 172.21.15.57:0/3921628608 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e80ba600 con 0x7f58ec0c4a10 2023-10-30T14:29:10.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58e1ffb640 1 -- 172.21.15.57:0/3921628608 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e80ba760 con 0x7f58ec0c4a10 2023-10-30T14:29:10.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f3394640 1 -- 172.21.15.57:0/3921628608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ec15b490 con 0x7f58ec0c4a10 2023-10-30T14:29:10.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58e1ffb640 1 -- 172.21.15.57:0/3921628608 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e80bab10 con 0x7f58ec0c4a10 2023-10-30T14:29:10.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.725+0000 7f58f3394640 1 -- 172.21.15.57:0/3921628608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ec15bc90 con 0x7f58ec0c4a10 2023-10-30T14:29:10.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.729+0000 7f58f3394640 1 -- 172.21.15.57:0/3921628608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ec15b010 con 0x7f58ec0c4a10 2023-10-30T14:29:10.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.729+0000 7f58e1ffb640 1 -- 172.21.15.57:0/3921628608 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f58e80bac70 con 0x7f58ec0c4a10 2023-10-30T14:29:10.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.729+0000 7f58e1ffb640 1 --2- 172.21.15.57:0/3921628608 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f58d00664a0 0x7f58d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:10.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.729+0000 7f58e1ffb640 1 -- 172.21.15.57:0/3921628608 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f58e8122920 con 0x7f58ec0c4a10 2023-10-30T14:29:10.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.729+0000 7f58f092f640 1 --2- 172.21.15.57:0/3921628608 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f58d00664a0 0x7f58d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:10.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.729+0000 7f58f092f640 1 --2- 172.21.15.57:0/3921628608 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f58d00664a0 0x7f58d0068960 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f58ec158310 tx=0x7f58dc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:10.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.733+0000 7f58e1ffb640 1 -- 172.21.15.57:0/3921628608 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f58ec15b010 con 0x7f58ec0c4a10 2023-10-30T14:29:10.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.921+0000 7f58f3394640 1 -- 172.21.15.57:0/3921628608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58ec15b9b0 con 0x7f58ec0c4a10 2023-10-30T14:29:10.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.921+0000 7f58e1ffb640 1 -- 172.21.15.57:0/3921628608 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5731 (secure 0 0 0) 0x7f58ec15b9b0 con 0x7f58ec0c4a10 2023-10-30T14:29:10.924 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:10.924 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:09.852404+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:10.924 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 92 2023-10-30T14:29:10.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.925+0000 7f58f3394640 1 -- 172.21.15.57:0/3921628608 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f58d00664a0 msgr2=0x7f58d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:10.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.925+0000 7f58f3394640 1 --2- 172.21.15.57:0/3921628608 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f58d00664a0 0x7f58d0068960 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f58ec158310 tx=0x7f58dc008040 comp rx=0 tx=0).stop 2023-10-30T14:29:10.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.925+0000 7f58f3394640 1 -- 172.21.15.57:0/3921628608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58ec0c4a10 msgr2=0x7f58ec0c8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:10.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.925+0000 7f58f3394640 1 --2- 172.21.15.57:0/3921628608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58ec0c4a10 0x7f58ec0c8af0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f58e801d3a0 tx=0x7f58e801ba90 comp rx=0 tx=0).stop 2023-10-30T14:29:10.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.925+0000 7f58f3394640 1 -- 172.21.15.57:0/3921628608 shutdown_connections 2023-10-30T14:29:10.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.925+0000 7f58f3394640 1 --2- 172.21.15.57:0/3921628608 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f58d00664a0 0x7f58d0068960 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.925+0000 7f58f3394640 1 --2- 172.21.15.57:0/3921628608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58ec15a3b0 0x7f58ec1f24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.925+0000 7f58f3394640 1 --2- 172.21.15.57:0/3921628608 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58ec153c80 0x7f58ec151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.925+0000 7f58f3394640 1 --2- 172.21.15.57:0/3921628608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58ec0c4a10 0x7f58ec0c8af0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:10.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.925+0000 7f58f3394640 1 -- 172.21.15.57:0/3921628608 >> 172.21.15.57:0/3921628608 conn(0x7f58ec0715c0 msgr2=0x7f58ec06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:10.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.925+0000 7f58f3394640 1 -- 172.21.15.57:0/3921628608 shutdown_connections 2023-10-30T14:29:10.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:10.925+0000 7f58f3394640 1 -- 172.21.15.57:0/3921628608 wait complete. 2023-10-30T14:29:10.946 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 92, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 92, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5315": { "addr": "172.21.15.88:6839/358521496", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 358521496, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 358521496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5315, "incarnation": 92, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_6027": { "addr": "172.21.15.57:6836/2733323524", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2733323524, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2733323524, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6027, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 24 }, "gid_6204": { "addr": "172.21.15.57:6837/1843528309", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 1843528309, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 1843528309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6204, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:29:09.852404+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5315, "mds_1": 6027, "mds_2": 6204 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:29:12.947 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:13.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.101+0000 7f687ee79640 1 Processor -- start 2023-10-30T14:29:13.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.101+0000 7f687ee79640 1 -- start start 2023-10-30T14:29:13.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.101+0000 7f687ee79640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6878150830 0x7f687806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:13.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.101+0000 7f687ee79640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6878070520 0x7f6878070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:13.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.101+0000 7f687ee79640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68780c2310 con 0x7f68780cb7d0 2023-10-30T14:29:13.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.101+0000 7f687ee79640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6878156710 con 0x7f6878070520 2023-10-30T14:29:13.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.101+0000 7f687ee79640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68780c4360 con 0x7f6878150830 2023-10-30T14:29:13.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.101+0000 7f6877fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f68780cb7d0 0x7f68780c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:49348/0 (socket says 172.21.15.57:49348) 2023-10-30T14:29:13.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.101+0000 7f6877fff640 1 -- 172.21.15.57:0/2069383258 learned_addr learned my addr 172.21.15.57:0/2069383258 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:29:13.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.101+0000 7f687d416640 1 --2- 172.21.15.57:0/2069383258 >> v2:172.21.15.57:3300/0 conn(0x7f6878070520 0x7f6878070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:13.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.101+0000 7f687cc15640 1 --2- 172.21.15.57:0/2069383258 >> v2:172.21.15.88:3300/0 conn(0x7f6878150830 0x7f687806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:13.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.101+0000 7f68777fe640 1 -- 172.21.15.57:0/2069383258 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223250466 0 0) 0x7f68780c2310 con 0x7f68780cb7d0 2023-10-30T14:29:13.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.101+0000 7f68777fe640 1 -- 172.21.15.57:0/2069383258 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6868003530 con 0x7f68780cb7d0 2023-10-30T14:29:13.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.105+0000 7f687d416640 1 -- 172.21.15.57:0/2069383258 >> v2:172.21.15.88:3300/0 conn(0x7f6878150830 msgr2=0x7f687806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:13.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.105+0000 7f687d416640 1 --2- 172.21.15.57:0/2069383258 >> v2:172.21.15.88:3300/0 conn(0x7f6878150830 0x7f687806b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:13.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.105+0000 7f687d416640 1 -- 172.21.15.57:0/2069383258 >> v1:172.21.15.57:6789/0 conn(0x7f68780cb7d0 legacy=0x7f68780c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:13.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.105+0000 7f687d416640 1 -- 172.21.15.57:0/2069383258 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68781569b0 con 0x7f6878070520 2023-10-30T14:29:13.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.105+0000 7f687d416640 1 --2- 172.21.15.57:0/2069383258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6878070520 0x7f6878070990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f687806fa40 tx=0x7f687001f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c714eaddc03e9939 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:13.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.105+0000 7f68777fe640 1 -- 172.21.15.57:0/2069383258 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68700b0440 con 0x7f6878070520 2023-10-30T14:29:13.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.105+0000 7f68777fe640 1 -- 172.21.15.57:0/2069383258 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68700b05a0 con 0x7f6878070520 2023-10-30T14:29:13.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.105+0000 7f68777fe640 1 -- 172.21.15.57:0/2069383258 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68700b0930 con 0x7f6878070520 2023-10-30T14:29:13.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.105+0000 7f687ee79640 1 -- 172.21.15.57:0/2069383258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6878070520 msgr2=0x7f6878070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:13.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.105+0000 7f687ee79640 1 --2- 172.21.15.57:0/2069383258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6878070520 0x7f6878070990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f687806fa40 tx=0x7f687001f0f0 comp rx=0 tx=0).stop 2023-10-30T14:29:13.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.105+0000 7f687ee79640 1 -- 172.21.15.57:0/2069383258 shutdown_connections 2023-10-30T14:29:13.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.105+0000 7f687ee79640 1 --2- 172.21.15.57:0/2069383258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6878070520 0x7f6878070990 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:13.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.105+0000 7f687ee79640 1 --2- 172.21.15.57:0/2069383258 >> v2:172.21.15.88:3300/0 conn(0x7f6878150830 0x7f687806b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:13.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.105+0000 7f687ee79640 1 -- 172.21.15.57:0/2069383258 >> 172.21.15.57:0/2069383258 conn(0x7f68780cac70 msgr2=0x7f68780cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:13.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687ee79640 1 -- 172.21.15.57:0/2069383258 shutdown_connections 2023-10-30T14:29:13.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687ee79640 1 -- 172.21.15.57:0/2069383258 wait complete. 2023-10-30T14:29:13.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687ee79640 1 Processor -- start 2023-10-30T14:29:13.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687ee79640 1 -- start start 2023-10-30T14:29:13.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687ee79640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6878070520 0x7f68781f2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:13.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687ee79640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68780cb7d0 0x7f68781ef5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:13.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687ee79640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6878150830 0x7f6878159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:13.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687ee79640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6878151040 con 0x7f6878150830 2023-10-30T14:29:13.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687ee79640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6878151320 con 0x7f6878070520 2023-10-30T14:29:13.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687ee79640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6878151600 con 0x7f68780cb7d0 2023-10-30T14:29:13.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687d416640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6878150830 0x7f6878159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:13.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687d416640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6878150830 0x7f6878159ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51590/0 (socket says 172.21.15.57:51590) 2023-10-30T14:29:13.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687d416640 1 -- 172.21.15.57:0/1921755043 learned_addr learned my addr 172.21.15.57:0/1921755043 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:13.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f6877fff640 1 --2- 172.21.15.57:0/1921755043 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68780cb7d0 0x7f68781ef5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:13.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687cc15640 1 --2- 172.21.15.57:0/1921755043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6878070520 0x7f68781f2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:13.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687d416640 1 -- 172.21.15.57:0/1921755043 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68780cb7d0 msgr2=0x7f68781ef5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:13.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687d416640 1 --2- 172.21.15.57:0/1921755043 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68780cb7d0 0x7f68781ef5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:13.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687d416640 1 -- 172.21.15.57:0/1921755043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6878070520 msgr2=0x7f68781f2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:13.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687d416640 1 --2- 172.21.15.57:0/1921755043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6878070520 0x7f68781f2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:13.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687d416640 1 -- 172.21.15.57:0/1921755043 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6870009a70 con 0x7f6878150830 2023-10-30T14:29:13.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f6877fff640 1 --2- 172.21.15.57:0/1921755043 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68780cb7d0 0x7f68781ef5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:13.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687cc15640 1 --2- 172.21.15.57:0/1921755043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6878070520 0x7f68781f2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:13.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.109+0000 7f687d416640 1 --2- 172.21.15.57:0/1921755043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6878150830 0x7f6878159ca0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f687001bd00 tx=0x7f687001be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:13.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.113+0000 7f68757fa640 1 -- 172.21.15.57:0/1921755043 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68700b0440 con 0x7f6878150830 2023-10-30T14:29:13.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.113+0000 7f68757fa640 1 -- 172.21.15.57:0/1921755043 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68700bc650 con 0x7f6878150830 2023-10-30T14:29:13.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.113+0000 7f687ee79640 1 -- 172.21.15.57:0/1921755043 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68781eedd0 con 0x7f6878150830 2023-10-30T14:29:13.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.113+0000 7f68757fa640 1 -- 172.21.15.57:0/1921755043 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68700bca00 con 0x7f6878150830 2023-10-30T14:29:13.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.113+0000 7f687ee79640 1 -- 172.21.15.57:0/1921755043 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6878156230 con 0x7f6878150830 2023-10-30T14:29:13.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.113+0000 7f687ee79640 1 -- 172.21.15.57:0/1921755043 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68781ef2f0 con 0x7f6878150830 2023-10-30T14:29:13.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.113+0000 7f68757fa640 1 -- 172.21.15.57:0/1921755043 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f68700bcb60 con 0x7f6878150830 2023-10-30T14:29:13.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.113+0000 7f68757fa640 1 --2- 172.21.15.57:0/1921755043 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f685c0664a0 0x7f685c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:13.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.113+0000 7f68757fa640 1 -- 172.21.15.57:0/1921755043 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f6870122ee0 con 0x7f6878150830 2023-10-30T14:29:13.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.113+0000 7f687cc15640 1 --2- 172.21.15.57:0/1921755043 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f685c0664a0 0x7f685c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:13.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.117+0000 7f687cc15640 1 --2- 172.21.15.57:0/1921755043 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f685c0664a0 0x7f685c068960 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f686c009b10 tx=0x7f686c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:13.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.117+0000 7f68757fa640 1 -- 172.21.15.57:0/1921755043 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f68781ef2f0 con 0x7f6878150830 2023-10-30T14:29:13.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.313+0000 7f687ee79640 1 -- 172.21.15.57:0/1921755043 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68781ef2f0 con 0x7f6878150830 2023-10-30T14:29:13.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.313+0000 7f68757fa640 1 -- 172.21.15.57:0/1921755043 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5731 (secure 0 0 0) 0x7f68781ef2f0 con 0x7f6878150830 2023-10-30T14:29:13.314 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:13.315 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:09.852404+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:13.315 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 92 2023-10-30T14:29:13.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.313+0000 7f687ee79640 1 -- 172.21.15.57:0/1921755043 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f685c0664a0 msgr2=0x7f685c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:13.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.313+0000 7f687ee79640 1 --2- 172.21.15.57:0/1921755043 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f685c0664a0 0x7f685c068960 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f686c009b10 tx=0x7f686c008040 comp rx=0 tx=0).stop 2023-10-30T14:29:13.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.313+0000 7f687ee79640 1 -- 172.21.15.57:0/1921755043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6878150830 msgr2=0x7f6878159ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:13.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.313+0000 7f687ee79640 1 --2- 172.21.15.57:0/1921755043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6878150830 0x7f6878159ca0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f687001bd00 tx=0x7f687001be90 comp rx=0 tx=0).stop 2023-10-30T14:29:13.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.313+0000 7f687ee79640 1 -- 172.21.15.57:0/1921755043 shutdown_connections 2023-10-30T14:29:13.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.313+0000 7f687ee79640 1 --2- 172.21.15.57:0/1921755043 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f685c0664a0 0x7f685c068960 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:13.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.313+0000 7f687ee79640 1 --2- 172.21.15.57:0/1921755043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6878150830 0x7f6878159ca0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:13.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.313+0000 7f687ee79640 1 --2- 172.21.15.57:0/1921755043 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68780cb7d0 0x7f68781ef5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:13.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.313+0000 7f687ee79640 1 --2- 172.21.15.57:0/1921755043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6878070520 0x7f68781f2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:13.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.313+0000 7f687ee79640 1 -- 172.21.15.57:0/1921755043 >> 172.21.15.57:0/1921755043 conn(0x7f68780cac70 msgr2=0x7f68780c64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:13.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.313+0000 7f687ee79640 1 -- 172.21.15.57:0/1921755043 shutdown_connections 2023-10-30T14:29:13.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:13.313+0000 7f687ee79640 1 -- 172.21.15.57:0/1921755043 wait complete. 2023-10-30T14:29:13.336 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:29:15.337 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:15.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.493+0000 7faf7d92a640 1 Processor -- start 2023-10-30T14:29:15.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.493+0000 7faf7d92a640 1 -- start start 2023-10-30T14:29:15.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.493+0000 7faf7d92a640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7faf780718d0 0x7faf780c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:15.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.493+0000 7faf7d92a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7815d0a0 con 0x7faf78153300 2023-10-30T14:29:15.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.493+0000 7faf7d92a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7815d380 con 0x7faf78071cb0 2023-10-30T14:29:15.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.493+0000 7faf7d92a640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf780cca10 con 0x7faf780718d0 2023-10-30T14:29:15.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.493+0000 7faf76ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7faf780718d0 0x7faf780c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:15.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.493+0000 7faf76ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7faf780718d0 0x7faf780c9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38810/0 (socket says 172.21.15.57:38810) 2023-10-30T14:29:15.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.493+0000 7faf76ffd640 1 -- 172.21.15.57:0/3149280227 learned_addr learned my addr 172.21.15.57:0/3149280227 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:15.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.493+0000 7faf777fe640 1 --1- 172.21.15.57:0/3149280227 >> v1:172.21.15.57:6789/0 conn(0x7faf78153300 0x7faf78153750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50094/0 (socket says 172.21.15.57:50094) 2023-10-30T14:29:15.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.493+0000 7faf76ffd640 1 -- 172.21.15.57:0/3149280227 >> v1:172.21.15.88:6789/0 conn(0x7faf78071cb0 legacy=0x7faf7815a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:15.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.493+0000 7faf76ffd640 1 -- 172.21.15.57:0/3149280227 >> v1:172.21.15.57:6789/0 conn(0x7faf78153300 legacy=0x7faf78153750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:15.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.493+0000 7faf76ffd640 1 -- 172.21.15.57:0/3149280227 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7815d660 con 0x7faf780718d0 2023-10-30T14:29:15.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.497+0000 7faf76ffd640 1 --2- 172.21.15.57:0/3149280227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf780718d0 0x7faf780c9a10 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7faf68009870 tx=0x7faf6801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f997b5d147e89537 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:15.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.497+0000 7faf75ffb640 1 -- 172.21.15.57:0/3149280227 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf680a8a30 con 0x7faf780718d0 2023-10-30T14:29:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.497+0000 7faf75ffb640 1 -- 172.21.15.57:0/3149280227 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf680a8b90 con 0x7faf780718d0 2023-10-30T14:29:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.497+0000 7faf75ffb640 1 -- 172.21.15.57:0/3149280227 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf680b0440 con 0x7faf780718d0 2023-10-30T14:29:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.497+0000 7faf7d92a640 1 -- 172.21.15.57:0/3149280227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf780718d0 msgr2=0x7faf780c9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.497+0000 7faf7d92a640 1 --2- 172.21.15.57:0/3149280227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf780718d0 0x7faf780c9a10 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7faf68009870 tx=0x7faf6801ed30 comp rx=0 tx=0).stop 2023-10-30T14:29:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.497+0000 7faf7d92a640 1 -- 172.21.15.57:0/3149280227 shutdown_connections 2023-10-30T14:29:15.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.497+0000 7faf7d92a640 1 --2- 172.21.15.57:0/3149280227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf780718d0 0x7faf780c9a10 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:15.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.497+0000 7faf7d92a640 1 -- 172.21.15.57:0/3149280227 >> 172.21.15.57:0/3149280227 conn(0x7faf7806ab00 msgr2=0x7faf78065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:15.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.497+0000 7faf7d92a640 1 -- 172.21.15.57:0/3149280227 shutdown_connections 2023-10-30T14:29:15.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.497+0000 7faf7d92a640 1 -- 172.21.15.57:0/3149280227 wait complete. 2023-10-30T14:29:15.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.497+0000 7faf7d92a640 1 Processor -- start 2023-10-30T14:29:15.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf7d92a640 1 -- start start 2023-10-30T14:29:15.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf7d92a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf780718d0 0x7faf7815a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:15.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf7d92a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf78071cb0 0x7faf78158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:15.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf7d92a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf78153300 0x7faf781f29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:15.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf7d92a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7815aeb0 con 0x7faf78153300 2023-10-30T14:29:15.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf7d92a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf781f2530 con 0x7faf78071cb0 2023-10-30T14:29:15.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf7d92a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf781f2200 con 0x7faf780718d0 2023-10-30T14:29:15.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf777fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf78153300 0x7faf781f29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:15.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf76ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf780718d0 0x7faf7815a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:15.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf767fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf78071cb0 0x7faf78158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:15.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf76ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf780718d0 0x7faf7815a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:37900/0 (socket says 172.21.15.57:37900) 2023-10-30T14:29:15.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf777fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf78153300 0x7faf781f29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38812/0 (socket says 172.21.15.57:38812) 2023-10-30T14:29:15.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf76ffd640 1 -- 172.21.15.57:0/629478657 learned_addr learned my addr 172.21.15.57:0/629478657 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:15.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf777fe640 1 -- 172.21.15.57:0/629478657 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf780718d0 msgr2=0x7faf7815a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:15.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf777fe640 1 --2- 172.21.15.57:0/629478657 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf780718d0 0x7faf7815a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:15.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf777fe640 1 -- 172.21.15.57:0/629478657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf78071cb0 msgr2=0x7faf78158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:15.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf777fe640 1 --2- 172.21.15.57:0/629478657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf78071cb0 0x7faf78158c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:15.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf777fe640 1 -- 172.21.15.57:0/629478657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf680094d0 con 0x7faf78153300 2023-10-30T14:29:15.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf777fe640 1 --2- 172.21.15.57:0/629478657 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf78153300 0x7faf781f29c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7faf7000cad0 tx=0x7faf7000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:15.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf5ffff640 1 -- 172.21.15.57:0/629478657 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7000abd0 con 0x7faf78153300 2023-10-30T14:29:15.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf5ffff640 1 -- 172.21.15.57:0/629478657 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf7000ad30 con 0x7faf78153300 2023-10-30T14:29:15.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf5ffff640 1 -- 172.21.15.57:0/629478657 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7000aea0 con 0x7faf78153300 2023-10-30T14:29:15.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf7d92a640 1 -- 172.21.15.57:0/629478657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf78151b20 con 0x7faf78153300 2023-10-30T14:29:15.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf7d92a640 1 -- 172.21.15.57:0/629478657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf781520e0 con 0x7faf78153300 2023-10-30T14:29:15.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.501+0000 7faf7d92a640 1 -- 172.21.15.57:0/629478657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf7804e170 con 0x7faf78153300 2023-10-30T14:29:15.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.505+0000 7faf5ffff640 1 -- 172.21.15.57:0/629478657 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faf700072b0 con 0x7faf78153300 2023-10-30T14:29:15.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.505+0000 7faf5ffff640 1 --2- 172.21.15.57:0/629478657 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faf640664d0 0x7faf64068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:15.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.505+0000 7faf5ffff640 1 -- 172.21.15.57:0/629478657 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7faf70084100 con 0x7faf78153300 2023-10-30T14:29:15.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.505+0000 7faf76ffd640 1 --2- 172.21.15.57:0/629478657 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faf640664d0 0x7faf64068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:15.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.505+0000 7faf76ffd640 1 --2- 172.21.15.57:0/629478657 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faf640664d0 0x7faf64068990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7faf6801ed70 tx=0x7faf68009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:15.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.509+0000 7faf5ffff640 1 -- 172.21.15.57:0/629478657 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faf700511c0 con 0x7faf78153300 2023-10-30T14:29:15.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.701+0000 7faf7d92a640 1 -- 172.21.15.57:0/629478657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf780c9f80 con 0x7faf78153300 2023-10-30T14:29:15.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.705+0000 7faf5ffff640 1 -- 172.21.15.57:0/629478657 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5731 (secure 0 0 0) 0x7faf70017020 con 0x7faf78153300 2023-10-30T14:29:15.705 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:15.705 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:09.852404+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:15.706 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 92 2023-10-30T14:29:15.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.705+0000 7faf7d92a640 1 -- 172.21.15.57:0/629478657 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faf640664d0 msgr2=0x7faf64068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:15.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.705+0000 7faf7d92a640 1 --2- 172.21.15.57:0/629478657 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faf640664d0 0x7faf64068990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7faf6801ed70 tx=0x7faf68009040 comp rx=0 tx=0).stop 2023-10-30T14:29:15.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.705+0000 7faf7d92a640 1 -- 172.21.15.57:0/629478657 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf78153300 msgr2=0x7faf781f29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:15.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.705+0000 7faf7d92a640 1 --2- 172.21.15.57:0/629478657 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf78153300 0x7faf781f29c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7faf7000cad0 tx=0x7faf7000cf90 comp rx=0 tx=0).stop 2023-10-30T14:29:15.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.705+0000 7faf7d92a640 1 -- 172.21.15.57:0/629478657 shutdown_connections 2023-10-30T14:29:15.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.705+0000 7faf7d92a640 1 --2- 172.21.15.57:0/629478657 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faf640664d0 0x7faf64068990 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:15.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.705+0000 7faf7d92a640 1 --2- 172.21.15.57:0/629478657 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf78153300 0x7faf781f29c0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:15.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.705+0000 7faf7d92a640 1 --2- 172.21.15.57:0/629478657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf78071cb0 0x7faf78158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:15.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.705+0000 7faf7d92a640 1 --2- 172.21.15.57:0/629478657 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf780718d0 0x7faf7815a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:15.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.705+0000 7faf7d92a640 1 -- 172.21.15.57:0/629478657 >> 172.21.15.57:0/629478657 conn(0x7faf7806ab00 msgr2=0x7faf78073510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:15.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.705+0000 7faf7d92a640 1 -- 172.21.15.57:0/629478657 shutdown_connections 2023-10-30T14:29:15.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:15.705+0000 7faf7d92a640 1 -- 172.21.15.57:0/629478657 wait complete. 2023-10-30T14:29:15.727 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:29:17.727 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:17.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.881+0000 7fa7391d3640 1 Processor -- start 2023-10-30T14:29:17.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.881+0000 7fa7391d3640 1 -- start start 2023-10-30T14:29:17.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.881+0000 7fa7391d3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa734151c20 0x7fa7340c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:17.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.881+0000 7fa7391d3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa7340cb8b0 0x7fa73406b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:17.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.881+0000 7fa7391d3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7340c5030 con 0x7fa734151840 2023-10-30T14:29:17.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.881+0000 7fa7391d3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa734157830 con 0x7fa7340cb8b0 2023-10-30T14:29:17.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.881+0000 7fa7391d3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa734157b10 con 0x7fa734151c20 2023-10-30T14:29:17.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.881+0000 7fa73359e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa7340cb8b0 0x7fa73406b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:17.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.881+0000 7fa73359e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa7340cb8b0 0x7fa73406b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38824/0 (socket says 172.21.15.57:38824) 2023-10-30T14:29:17.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.881+0000 7fa73359e640 1 -- 172.21.15.57:0/4037421984 learned_addr learned my addr 172.21.15.57:0/4037421984 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:17.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.881+0000 7fa73259c640 1 --2- 172.21.15.57:0/4037421984 >> v2:172.21.15.88:3300/0 conn(0x7fa734151c20 0x7fa7340c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:17.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.881+0000 7fa73359e640 1 -- 172.21.15.57:0/4037421984 >> v2:172.21.15.88:3300/0 conn(0x7fa734151c20 msgr2=0x7fa7340c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:17.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.881+0000 7fa73359e640 1 --2- 172.21.15.57:0/4037421984 >> v2:172.21.15.88:3300/0 conn(0x7fa734151c20 0x7fa7340c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:17.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.881+0000 7fa73359e640 1 -- 172.21.15.57:0/4037421984 >> v1:172.21.15.88:6789/0 conn(0x7fa734151840 legacy=0x7fa734072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:17.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.881+0000 7fa73359e640 1 -- 172.21.15.57:0/4037421984 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa734157df0 con 0x7fa7340cb8b0 2023-10-30T14:29:17.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.885+0000 7fa73359e640 1 --2- 172.21.15.57:0/4037421984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7340cb8b0 0x7fa73406b600 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa73406d4d0 tx=0x7fa72c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a1609a2b18772b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:17.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.885+0000 7fa731d9b640 1 -- 172.21.15.57:0/4037421984 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c0af400 con 0x7fa7340cb8b0 2023-10-30T14:29:17.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.885+0000 7fa731d9b640 1 -- 172.21.15.57:0/4037421984 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72c0ab050 con 0x7fa7340cb8b0 2023-10-30T14:29:17.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.885+0000 7fa731d9b640 1 -- 172.21.15.57:0/4037421984 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c0af780 con 0x7fa7340cb8b0 2023-10-30T14:29:17.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.885+0000 7fa7391d3640 1 -- 172.21.15.57:0/4037421984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7340cb8b0 msgr2=0x7fa73406b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:17.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.885+0000 7fa7391d3640 1 --2- 172.21.15.57:0/4037421984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7340cb8b0 0x7fa73406b600 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa73406d4d0 tx=0x7fa72c01f1c0 comp rx=0 tx=0).stop 2023-10-30T14:29:17.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.885+0000 7fa7391d3640 1 -- 172.21.15.57:0/4037421984 shutdown_connections 2023-10-30T14:29:17.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.885+0000 7fa7391d3640 1 --2- 172.21.15.57:0/4037421984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7340cb8b0 0x7fa73406b600 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:17.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.885+0000 7fa7391d3640 1 --2- 172.21.15.57:0/4037421984 >> v2:172.21.15.88:3300/0 conn(0x7fa734151c20 0x7fa7340c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:17.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.885+0000 7fa7391d3640 1 -- 172.21.15.57:0/4037421984 >> 172.21.15.57:0/4037421984 conn(0x7fa7340cac70 msgr2=0x7fa7340cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:17.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.885+0000 7fa7391d3640 1 -- 172.21.15.57:0/4037421984 shutdown_connections 2023-10-30T14:29:17.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.885+0000 7fa7391d3640 1 -- 172.21.15.57:0/4037421984 wait complete. 2023-10-30T14:29:17.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa7391d3640 1 Processor -- start 2023-10-30T14:29:17.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa7391d3640 1 -- start start 2023-10-30T14:29:17.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa7391d3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7340cb8b0 0x7fa7340ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:17.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa7391d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa734151840 0x7fa7341478f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:17.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa7391d3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa734151c20 0x7fa7340733c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:17.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa7391d3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7341eecc0 con 0x7fa734151c20 2023-10-30T14:29:17.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa7391d3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7340ca610 con 0x7fa7340cb8b0 2023-10-30T14:29:17.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa7391d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7340ca850 con 0x7fa734151840 2023-10-30T14:29:17.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa73259c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa734151840 0x7fa7341478f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:17.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa73359e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa734151c20 0x7fa7340733c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:17.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa732d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7340cb8b0 0x7fa7340ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:17.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa73259c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa734151840 0x7fa7341478f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:37910/0 (socket says 172.21.15.57:37910) 2023-10-30T14:29:17.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa73259c640 1 -- 172.21.15.57:0/2223094402 learned_addr learned my addr 172.21.15.57:0/2223094402 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:17.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa73359e640 1 -- 172.21.15.57:0/2223094402 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa734151840 msgr2=0x7fa7341478f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:17.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa73359e640 1 --2- 172.21.15.57:0/2223094402 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa734151840 0x7fa7341478f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:17.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa73359e640 1 -- 172.21.15.57:0/2223094402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7340cb8b0 msgr2=0x7fa7340ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:17.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa73359e640 1 --2- 172.21.15.57:0/2223094402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7340cb8b0 0x7fa7340ca0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:17.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa73359e640 1 -- 172.21.15.57:0/2223094402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72c009b30 con 0x7fa734151c20 2023-10-30T14:29:17.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa73359e640 1 --2- 172.21.15.57:0/2223094402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa734151c20 0x7fa7340733c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fa72c0ae000 tx=0x7fa72c01bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:17.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa71b7fe640 1 -- 172.21.15.57:0/2223094402 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c0c0070 con 0x7fa734151c20 2023-10-30T14:29:17.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa71b7fe640 1 -- 172.21.15.57:0/2223094402 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72c009510 con 0x7fa734151c20 2023-10-30T14:29:17.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa71b7fe640 1 -- 172.21.15.57:0/2223094402 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c0afe60 con 0x7fa734151c20 2023-10-30T14:29:17.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa7391d3640 1 -- 172.21.15.57:0/2223094402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa734155590 con 0x7fa734151c20 2023-10-30T14:29:17.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa7391d3640 1 -- 172.21.15.57:0/2223094402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa734155b50 con 0x7fa734151c20 2023-10-30T14:29:17.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.889+0000 7fa7391d3640 1 -- 172.21.15.57:0/2223094402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7341eee30 con 0x7fa734151c20 2023-10-30T14:29:17.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.893+0000 7fa71b7fe640 1 -- 172.21.15.57:0/2223094402 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa72c0b6040 con 0x7fa734151c20 2023-10-30T14:29:17.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.893+0000 7fa71b7fe640 1 --2- 172.21.15.57:0/2223094402 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa7200664d0 0x7fa720068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:17.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.893+0000 7fa71b7fe640 1 -- 172.21.15.57:0/2223094402 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7fa72c121ae0 con 0x7fa734151c20 2023-10-30T14:29:17.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.893+0000 7fa732d9d640 1 --2- 172.21.15.57:0/2223094402 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa7200664d0 0x7fa720068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:17.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.893+0000 7fa732d9d640 1 --2- 172.21.15.57:0/2223094402 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa7200664d0 0x7fa720068990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa724007fd0 tx=0x7fa724009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:17.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:17.897+0000 7fa71b7fe640 1 -- 172.21.15.57:0/2223094402 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa7341eee30 con 0x7fa734151c20 2023-10-30T14:29:18.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:18.085+0000 7fa7391d3640 1 -- 172.21.15.57:0/2223094402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa734066510 con 0x7fa734151c20 2023-10-30T14:29:18.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:18.085+0000 7fa71b7fe640 1 -- 172.21.15.57:0/2223094402 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5731 (secure 0 0 0) 0x7fa72c0eed40 con 0x7fa734151c20 2023-10-30T14:29:18.087 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:18.087 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:09.852404+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:18.088 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 92 2023-10-30T14:29:18.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:18.089+0000 7fa7391d3640 1 -- 172.21.15.57:0/2223094402 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa7200664d0 msgr2=0x7fa720068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:18.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:18.089+0000 7fa7391d3640 1 --2- 172.21.15.57:0/2223094402 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa7200664d0 0x7fa720068990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa724007fd0 tx=0x7fa724009040 comp rx=0 tx=0).stop 2023-10-30T14:29:18.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:18.089+0000 7fa7391d3640 1 -- 172.21.15.57:0/2223094402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa734151c20 msgr2=0x7fa7340733c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:18.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:18.089+0000 7fa7391d3640 1 --2- 172.21.15.57:0/2223094402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa734151c20 0x7fa7340733c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fa72c0ae000 tx=0x7fa72c01bf60 comp rx=0 tx=0).stop 2023-10-30T14:29:18.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:18.089+0000 7fa7391d3640 1 -- 172.21.15.57:0/2223094402 shutdown_connections 2023-10-30T14:29:18.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:18.089+0000 7fa7391d3640 1 --2- 172.21.15.57:0/2223094402 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa7200664d0 0x7fa720068990 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:18.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:18.089+0000 7fa7391d3640 1 --2- 172.21.15.57:0/2223094402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa734151c20 0x7fa7340733c0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:18.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:18.089+0000 7fa7391d3640 1 --2- 172.21.15.57:0/2223094402 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa734151840 0x7fa7341478f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:18.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:18.089+0000 7fa7391d3640 1 --2- 172.21.15.57:0/2223094402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7340cb8b0 0x7fa7340ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:18.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:18.089+0000 7fa7391d3640 1 -- 172.21.15.57:0/2223094402 >> 172.21.15.57:0/2223094402 conn(0x7fa7340cac70 msgr2=0x7fa73406b500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:18.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:18.089+0000 7fa7391d3640 1 -- 172.21.15.57:0/2223094402 shutdown_connections 2023-10-30T14:29:18.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:18.089+0000 7fa7391d3640 1 -- 172.21.15.57:0/2223094402 wait complete. 2023-10-30T14:29:18.108 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:29:20.109 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:20.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.265+0000 7fa0cc580640 1 Processor -- start 2023-10-30T14:29:20.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.265+0000 7fa0cc580640 1 -- start start 2023-10-30T14:29:20.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.265+0000 7fa0cc580640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa0c4153c80 0x7fa0c415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:20.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.265+0000 7fa0cc580640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c4159f20 con 0x7fa0c40c42a0 2023-10-30T14:29:20.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.265+0000 7fa0cc580640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c415d620 con 0x7fa0c40c4a10 2023-10-30T14:29:20.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.265+0000 7fa0cc580640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c415dec0 con 0x7fa0c4153c80 2023-10-30T14:29:20.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.265+0000 7fa0cab1d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa0c40c42a0 0x7fa0c40ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50096/0 (socket says 172.21.15.57:50096) 2023-10-30T14:29:20.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.265+0000 7fa0cab1d640 1 -- 172.21.15.57:0/2911097260 learned_addr learned my addr 172.21.15.57:0/2911097260 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:29:20.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.265+0000 7fa0ca31c640 1 --2- 172.21.15.57:0/2911097260 >> v2:172.21.15.88:3300/0 conn(0x7fa0c4153c80 0x7fa0c415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:20.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.265+0000 7fa0c931a640 1 -- 172.21.15.57:0/2911097260 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339788206 0 0) 0x7fa0c4159f20 con 0x7fa0c40c42a0 2023-10-30T14:29:20.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.265+0000 7fa0c931a640 1 -- 172.21.15.57:0/2911097260 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0b0003530 con 0x7fa0c40c42a0 2023-10-30T14:29:20.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.265+0000 7fa0ca31c640 1 -- 172.21.15.57:0/2911097260 >> v1:172.21.15.88:6789/0 conn(0x7fa0c40c4a10 legacy=0x7fa0c415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:20.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.265+0000 7fa0ca31c640 1 -- 172.21.15.57:0/2911097260 >> v1:172.21.15.57:6789/0 conn(0x7fa0c40c42a0 legacy=0x7fa0c40ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:20.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.265+0000 7fa0c931a640 1 -- 172.21.15.57:0/2911097260 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2029729993 0 0) 0x7fa0b0003530 con 0x7fa0c40c42a0 2023-10-30T14:29:20.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.265+0000 7fa0ca31c640 1 -- 172.21.15.57:0/2911097260 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c415d900 con 0x7fa0c4153c80 2023-10-30T14:29:20.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.269+0000 7fa0ca31c640 1 --2- 172.21.15.57:0/2911097260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0c4153c80 0x7fa0c415cf90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa0c00097c0 tx=0x7fa0c001eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a60fc462920fe380 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:20.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.269+0000 7fa0c931a640 1 -- 172.21.15.57:0/2911097260 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc001a00 con 0x7fa0c4153c80 2023-10-30T14:29:20.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.269+0000 7fa0c931a640 1 -- 172.21.15.57:0/2911097260 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c00a8610 con 0x7fa0c4153c80 2023-10-30T14:29:20.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.269+0000 7fa0c931a640 1 -- 172.21.15.57:0/2911097260 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c00a89a0 con 0x7fa0c4153c80 2023-10-30T14:29:20.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.269+0000 7fa0cc580640 1 -- 172.21.15.57:0/2911097260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0c4153c80 msgr2=0x7fa0c415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:20.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.269+0000 7fa0cc580640 1 --2- 172.21.15.57:0/2911097260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0c4153c80 0x7fa0c415cf90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa0c00097c0 tx=0x7fa0c001eb90 comp rx=0 tx=0).stop 2023-10-30T14:29:20.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.269+0000 7fa0cc580640 1 -- 172.21.15.57:0/2911097260 shutdown_connections 2023-10-30T14:29:20.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.269+0000 7fa0cc580640 1 --2- 172.21.15.57:0/2911097260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0c4153c80 0x7fa0c415cf90 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:20.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.269+0000 7fa0cc580640 1 -- 172.21.15.57:0/2911097260 >> 172.21.15.57:0/2911097260 conn(0x7fa0c40715c0 msgr2=0x7fa0c40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:20.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.269+0000 7fa0cc580640 1 -- 172.21.15.57:0/2911097260 shutdown_connections 2023-10-30T14:29:20.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.269+0000 7fa0cc580640 1 -- 172.21.15.57:0/2911097260 wait complete. 2023-10-30T14:29:20.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0cc580640 1 Processor -- start 2023-10-30T14:29:20.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0cc580640 1 -- start start 2023-10-30T14:29:20.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0cc580640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c40c42a0 0x7fa0c4159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:20.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0cc580640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0c40c4a10 0x7fa0c406c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:20.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0cc580640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0c4153c80 0x7fa0c4149ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:20.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0cc580640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c41561f0 con 0x7fa0c40c42a0 2023-10-30T14:29:20.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0cc580640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c4154e10 con 0x7fa0c40c4a10 2023-10-30T14:29:20.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0cc580640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c4158870 con 0x7fa0c4153c80 2023-10-30T14:29:20.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0ca31c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c40c42a0 0x7fa0c4159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:20.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0ca31c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c40c42a0 0x7fa0c4159450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38846/0 (socket says 172.21.15.57:38846) 2023-10-30T14:29:20.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0ca31c640 1 -- 172.21.15.57:0/2133940037 learned_addr learned my addr 172.21.15.57:0/2133940037 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:20.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0cab1d640 1 --2- 172.21.15.57:0/2133940037 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0c4153c80 0x7fa0c4149ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:20.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0c9b1b640 1 --2- 172.21.15.57:0/2133940037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0c40c4a10 0x7fa0c406c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:20.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0ca31c640 1 -- 172.21.15.57:0/2133940037 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0c4153c80 msgr2=0x7fa0c4149ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:20.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0ca31c640 1 --2- 172.21.15.57:0/2133940037 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0c4153c80 0x7fa0c4149ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:20.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0ca31c640 1 -- 172.21.15.57:0/2133940037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0c40c4a10 msgr2=0x7fa0c406c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:20.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0ca31c640 1 --2- 172.21.15.57:0/2133940037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0c40c4a10 0x7fa0c406c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:20.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0ca31c640 1 -- 172.21.15.57:0/2133940037 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c0009420 con 0x7fa0c40c42a0 2023-10-30T14:29:20.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0c9b1b640 1 --2- 172.21.15.57:0/2133940037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0c40c4a10 0x7fa0c406c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:29:20.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0ca31c640 1 --2- 172.21.15.57:0/2133940037 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c40c42a0 0x7fa0c4159450 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fa0c00b3040 tx=0x7fa0c001b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:20.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0baffd640 1 -- 172.21.15.57:0/2133940037 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c00b0850 con 0x7fa0c40c42a0 2023-10-30T14:29:20.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0baffd640 1 -- 172.21.15.57:0/2133940037 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c00b09b0 con 0x7fa0c40c42a0 2023-10-30T14:29:20.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0baffd640 1 -- 172.21.15.57:0/2133940037 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c00b0d60 con 0x7fa0c40c42a0 2023-10-30T14:29:20.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0cc580640 1 -- 172.21.15.57:0/2133940037 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c4155440 con 0x7fa0c40c42a0 2023-10-30T14:29:20.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.273+0000 7fa0cc580640 1 -- 172.21.15.57:0/2133940037 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c4159990 con 0x7fa0c40c42a0 2023-10-30T14:29:20.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.277+0000 7fa0cc580640 1 -- 172.21.15.57:0/2133940037 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c4159b80 con 0x7fa0c40c42a0 2023-10-30T14:29:20.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.277+0000 7fa0baffd640 1 -- 172.21.15.57:0/2133940037 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa0c00b7040 con 0x7fa0c40c42a0 2023-10-30T14:29:20.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.277+0000 7fa0baffd640 1 --2- 172.21.15.57:0/2133940037 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa0a80664d0 0x7fa0a8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:20.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.277+0000 7fa0baffd640 1 -- 172.21.15.57:0/2133940037 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7fa0c0121920 con 0x7fa0c40c42a0 2023-10-30T14:29:20.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.277+0000 7fa0c9b1b640 1 --2- 172.21.15.57:0/2133940037 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa0a80664d0 0x7fa0a8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:20.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.277+0000 7fa0c9b1b640 1 --2- 172.21.15.57:0/2133940037 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa0a80664d0 0x7fa0a8068990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa0c415d480 tx=0x7fa0b4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:20.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.281+0000 7fa0baffd640 1 -- 172.21.15.57:0/2133940037 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa0c00ee980 con 0x7fa0c40c42a0 2023-10-30T14:29:20.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.473+0000 7fa0cc580640 1 -- 172.21.15.57:0/2133940037 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0c404e170 con 0x7fa0c40c42a0 2023-10-30T14:29:20.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.473+0000 7fa0baffd640 1 -- 172.21.15.57:0/2133940037 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5734 (secure 0 0 0) 0x7fa0c00b5020 con 0x7fa0c40c42a0 2023-10-30T14:29:20.476 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:20.477 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:18.867806+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:resolve","state_seq":101,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:20.477 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 93 2023-10-30T14:29:20.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.477+0000 7fa0cc580640 1 -- 172.21.15.57:0/2133940037 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa0a80664d0 msgr2=0x7fa0a8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:20.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.477+0000 7fa0cc580640 1 --2- 172.21.15.57:0/2133940037 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa0a80664d0 0x7fa0a8068990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa0c415d480 tx=0x7fa0b4009040 comp rx=0 tx=0).stop 2023-10-30T14:29:20.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.477+0000 7fa0cc580640 1 -- 172.21.15.57:0/2133940037 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c40c42a0 msgr2=0x7fa0c4159450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:20.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.477+0000 7fa0cc580640 1 --2- 172.21.15.57:0/2133940037 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c40c42a0 0x7fa0c4159450 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fa0c00b3040 tx=0x7fa0c001b930 comp rx=0 tx=0).stop 2023-10-30T14:29:20.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.477+0000 7fa0cc580640 1 -- 172.21.15.57:0/2133940037 shutdown_connections 2023-10-30T14:29:20.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.477+0000 7fa0cc580640 1 --2- 172.21.15.57:0/2133940037 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa0a80664d0 0x7fa0a8068990 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:20.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.477+0000 7fa0cc580640 1 --2- 172.21.15.57:0/2133940037 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0c4153c80 0x7fa0c4149ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:20.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.477+0000 7fa0cc580640 1 --2- 172.21.15.57:0/2133940037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0c40c4a10 0x7fa0c406c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:20.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.477+0000 7fa0cc580640 1 --2- 172.21.15.57:0/2133940037 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c40c42a0 0x7fa0c4159450 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:20.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.477+0000 7fa0cc580640 1 -- 172.21.15.57:0/2133940037 >> 172.21.15.57:0/2133940037 conn(0x7fa0c40715c0 msgr2=0x7fa0c4152410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:20.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.477+0000 7fa0cc580640 1 -- 172.21.15.57:0/2133940037 shutdown_connections 2023-10-30T14:29:20.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:20.477+0000 7fa0cc580640 1 -- 172.21.15.57:0/2133940037 wait complete. 2023-10-30T14:29:20.498 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:29:22.498 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:22.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.653+0000 7f19fb59e640 1 Processor -- start 2023-10-30T14:29:22.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.653+0000 7f19fb59e640 1 -- start start 2023-10-30T14:29:22.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.653+0000 7f19fb59e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f19fc0c8eb0 0x7f19fc15cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:22.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.653+0000 7f19fb59e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f19fc06aeb0 0x7f19fc069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:22.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.653+0000 7f19fb59e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19fc15dcc0 con 0x7f19fc15ac70 2023-10-30T14:29:22.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.653+0000 7f19fb59e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19fc159800 con 0x7f19fc06aeb0 2023-10-30T14:29:22.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.653+0000 7f19fb59e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc159d80 con 0x7f19fc0c8eb0 2023-10-30T14:29:22.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.653+0000 7f19fad9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f19fc06aeb0 0x7f19fc069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:22.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.653+0000 7f19fad9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f19fc06aeb0 0x7f19fc069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38858/0 (socket says 172.21.15.57:38858) 2023-10-30T14:29:22.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.653+0000 7f19fad9d640 1 -- 172.21.15.57:0/3607390873 learned_addr learned my addr 172.21.15.57:0/3607390873 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:22.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.653+0000 7f19fa59c640 1 --2- 172.21.15.57:0/3607390873 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f19fc0c8eb0 0x7f19fc15cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:22.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.653+0000 7f19fad9d640 1 -- 172.21.15.57:0/3607390873 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f19fc0c8eb0 msgr2=0x7f19fc15cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:22.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.653+0000 7f19fad9d640 1 --2- 172.21.15.57:0/3607390873 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f19fc0c8eb0 0x7f19fc15cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:22.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.653+0000 7f19fad9d640 1 -- 172.21.15.57:0/3607390873 >> v1:172.21.15.88:6789/0 conn(0x7f19fc15ac70 legacy=0x7f19fc15a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:22.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.653+0000 7f19fad9d640 1 -- 172.21.15.57:0/3607390873 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19fc15d420 con 0x7f19fc06aeb0 2023-10-30T14:29:22.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.657+0000 7f19fad9d640 1 --2- 172.21.15.57:0/3607390873 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19fc06aeb0 0x7f19fc069e90 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f19f4009e50 tx=0x7f19f401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a528d8cda4e7b99 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:22.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.657+0000 7f19f959a640 1 -- 172.21.15.57:0/3607390873 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f40af400 con 0x7f19fc06aeb0 2023-10-30T14:29:22.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.657+0000 7f19f959a640 1 -- 172.21.15.57:0/3607390873 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f40ab050 con 0x7f19fc06aeb0 2023-10-30T14:29:22.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.657+0000 7f19f959a640 1 -- 172.21.15.57:0/3607390873 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f40af780 con 0x7f19fc06aeb0 2023-10-30T14:29:22.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.657+0000 7f19fb59e640 1 -- 172.21.15.57:0/3607390873 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19fc06aeb0 msgr2=0x7f19fc069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:22.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.657+0000 7f19fb59e640 1 --2- 172.21.15.57:0/3607390873 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19fc06aeb0 0x7f19fc069e90 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f19f4009e50 tx=0x7f19f401f1f0 comp rx=0 tx=0).stop 2023-10-30T14:29:22.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.657+0000 7f19fb59e640 1 -- 172.21.15.57:0/3607390873 shutdown_connections 2023-10-30T14:29:22.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.657+0000 7f19fb59e640 1 --2- 172.21.15.57:0/3607390873 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19fc06aeb0 0x7f19fc069e90 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:22.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.657+0000 7f19fb59e640 1 --2- 172.21.15.57:0/3607390873 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f19fc0c8eb0 0x7f19fc15cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:22.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.657+0000 7f19fb59e640 1 -- 172.21.15.57:0/3607390873 >> 172.21.15.57:0/3607390873 conn(0x7f19fc0cef00 msgr2=0x7f19fc0cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:22.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.657+0000 7f19fb59e640 1 -- 172.21.15.57:0/3607390873 shutdown_connections 2023-10-30T14:29:22.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.657+0000 7f19fb59e640 1 -- 172.21.15.57:0/3607390873 wait complete. 2023-10-30T14:29:22.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fb59e640 1 Processor -- start 2023-10-30T14:29:22.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fb59e640 1 -- start start 2023-10-30T14:29:22.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fb59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19fc06aeb0 0x7f19fc14f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:22.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fb59e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f19fc0c8eb0 0x7f19fc0ce540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:22.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fb59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19fc15ac70 0x7f19fc14cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:22.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fb59e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc1e4db0 con 0x7f19fc06aeb0 2023-10-30T14:29:22.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fb59e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc150a10 con 0x7f19fc15ac70 2023-10-30T14:29:22.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fb59e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc1eee40 con 0x7f19fc0c8eb0 2023-10-30T14:29:22.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fa59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19fc06aeb0 0x7f19fc14f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:22.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fa59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19fc06aeb0 0x7f19fc14f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38870/0 (socket says 172.21.15.57:38870) 2023-10-30T14:29:22.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fa59c640 1 -- 172.21.15.57:0/1786157968 learned_addr learned my addr 172.21.15.57:0/1786157968 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:22.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19f9d9b640 1 --2- 172.21.15.57:0/1786157968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f19fc0c8eb0 0x7f19fc0ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:22.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fad9d640 1 --2- 172.21.15.57:0/1786157968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19fc15ac70 0x7f19fc14cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:22.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fa59c640 1 -- 172.21.15.57:0/1786157968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f19fc0c8eb0 msgr2=0x7f19fc0ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:22.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fa59c640 1 --2- 172.21.15.57:0/1786157968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f19fc0c8eb0 0x7f19fc0ce540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:22.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fa59c640 1 -- 172.21.15.57:0/1786157968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19fc15ac70 msgr2=0x7f19fc14cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:22.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fa59c640 1 --2- 172.21.15.57:0/1786157968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19fc15ac70 0x7f19fc14cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:22.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fa59c640 1 -- 172.21.15.57:0/1786157968 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f4009b30 con 0x7f19fc06aeb0 2023-10-30T14:29:22.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fa59c640 1 --2- 172.21.15.57:0/1786157968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19fc06aeb0 0x7f19fc14f0c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00d800 tx=0x7f19ec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:22.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19eaffd640 1 -- 172.21.15.57:0/1786157968 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec00b6c0 con 0x7f19fc06aeb0 2023-10-30T14:29:22.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19eaffd640 1 -- 172.21.15.57:0/1786157968 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ec00b820 con 0x7f19fc06aeb0 2023-10-30T14:29:22.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19eaffd640 1 -- 172.21.15.57:0/1786157968 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec00bb90 con 0x7f19fc06aeb0 2023-10-30T14:29:22.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fb59e640 1 -- 172.21.15.57:0/1786157968 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19fc150c60 con 0x7f19fc06aeb0 2023-10-30T14:29:22.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fb59e640 1 -- 172.21.15.57:0/1786157968 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19fc1ef100 con 0x7f19fc06aeb0 2023-10-30T14:29:22.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.661+0000 7f19fb59e640 1 -- 172.21.15.57:0/1786157968 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19fc04e170 con 0x7f19fc06aeb0 2023-10-30T14:29:22.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.665+0000 7f19eaffd640 1 -- 172.21.15.57:0/1786157968 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f19ec00bcf0 con 0x7f19fc06aeb0 2023-10-30T14:29:22.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.665+0000 7f19eaffd640 1 --2- 172.21.15.57:0/1786157968 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f19dc0664a0 0x7f19dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:22.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.665+0000 7f19eaffd640 1 -- 172.21.15.57:0/1786157968 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f19ec084610 con 0x7f19fc06aeb0 2023-10-30T14:29:22.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.665+0000 7f19f9d9b640 1 --2- 172.21.15.57:0/1786157968 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f19dc0664a0 0x7f19dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:22.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.665+0000 7f19f9d9b640 1 --2- 172.21.15.57:0/1786157968 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f19dc0664a0 0x7f19dc068960 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f19fc0c9880 tx=0x7f19f0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:22.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.669+0000 7f19eaffd640 1 -- 172.21.15.57:0/1786157968 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f19ec0517b0 con 0x7f19fc06aeb0 2023-10-30T14:29:22.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.857+0000 7f19fb59e640 1 -- 172.21.15.57:0/1786157968 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19fc0c95d0 con 0x7f19fc06aeb0 2023-10-30T14:29:22.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.861+0000 7f19eaffd640 1 -- 172.21.15.57:0/1786157968 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f19ec054dd0 con 0x7f19fc06aeb0 2023-10-30T14:29:22.861 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:22.862 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:18.867806+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:resolve","state_seq":101,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:22.862 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 93 2023-10-30T14:29:22.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.861+0000 7f19fb59e640 1 -- 172.21.15.57:0/1786157968 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f19dc0664a0 msgr2=0x7f19dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:22.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.861+0000 7f19fb59e640 1 --2- 172.21.15.57:0/1786157968 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f19dc0664a0 0x7f19dc068960 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f19fc0c9880 tx=0x7f19f0009040 comp rx=0 tx=0).stop 2023-10-30T14:29:22.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.861+0000 7f19fb59e640 1 -- 172.21.15.57:0/1786157968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19fc06aeb0 msgr2=0x7f19fc14f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:22.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.861+0000 7f19fb59e640 1 --2- 172.21.15.57:0/1786157968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19fc06aeb0 0x7f19fc14f0c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00d800 tx=0x7f19ec00dcc0 comp rx=0 tx=0).stop 2023-10-30T14:29:22.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.861+0000 7f19fb59e640 1 -- 172.21.15.57:0/1786157968 shutdown_connections 2023-10-30T14:29:22.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.861+0000 7f19fb59e640 1 --2- 172.21.15.57:0/1786157968 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f19dc0664a0 0x7f19dc068960 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:22.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.861+0000 7f19fb59e640 1 --2- 172.21.15.57:0/1786157968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19fc15ac70 0x7f19fc14cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:22.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.861+0000 7f19fb59e640 1 --2- 172.21.15.57:0/1786157968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f19fc0c8eb0 0x7f19fc0ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:22.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.861+0000 7f19fb59e640 1 --2- 172.21.15.57:0/1786157968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19fc06aeb0 0x7f19fc14f0c0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:22.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.861+0000 7f19fb59e640 1 -- 172.21.15.57:0/1786157968 >> 172.21.15.57:0/1786157968 conn(0x7f19fc0cef00 msgr2=0x7f19fc0cb100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:22.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.861+0000 7f19fb59e640 1 -- 172.21.15.57:0/1786157968 shutdown_connections 2023-10-30T14:29:22.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:22.861+0000 7f19fb59e640 1 -- 172.21.15.57:0/1786157968 wait complete. 2023-10-30T14:29:22.883 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 93, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 93, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5315": { "addr": "172.21.15.88:6839/358521496", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 358521496, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 358521496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5315, "incarnation": 92, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:resolve", "state_seq": 101 }, "gid_6027": { "addr": "172.21.15.57:6836/2733323524", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2733323524, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2733323524, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6027, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 24 }, "gid_6204": { "addr": "172.21.15.57:6837/1843528309", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 1843528309, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 1843528309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6204, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:29:18.867806+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5315, "mds_1": 6027, "mds_2": 6204 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:29:24.885 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:25.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.037+0000 7f5eb2b3e640 1 Processor -- start 2023-10-30T14:29:25.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.037+0000 7f5eb2b3e640 1 -- start start 2023-10-30T14:29:25.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.041+0000 7f5eb2b3e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5eac06c420 0x7f5eac0c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:25.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.041+0000 7f5eb2b3e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5eac0c7170 0x7f5eac159980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:25.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.041+0000 7f5eb2b3e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eac0c5f80 con 0x7f5eac153bb0 2023-10-30T14:29:25.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.041+0000 7f5eb2b3e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eac0c7580 con 0x7f5eac0c7170 2023-10-30T14:29:25.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.041+0000 7f5eb2b3e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eac0c5690 con 0x7f5eac06c420 2023-10-30T14:29:25.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.041+0000 7f5eb10db640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5eac0c7170 0x7f5eac159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:25.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.041+0000 7f5eb10db640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5eac0c7170 0x7f5eac159980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51956/0 (socket says 172.21.15.57:51956) 2023-10-30T14:29:25.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.041+0000 7f5eb10db640 1 -- 172.21.15.57:0/1228768722 learned_addr learned my addr 172.21.15.57:0/1228768722 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:25.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.041+0000 7f5eabfff640 1 --2- 172.21.15.57:0/1228768722 >> v2:172.21.15.88:3300/0 conn(0x7f5eac06c420 0x7f5eac0c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:25.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.041+0000 7f5eb10db640 1 -- 172.21.15.57:0/1228768722 >> v2:172.21.15.88:3300/0 conn(0x7f5eac06c420 msgr2=0x7f5eac0c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:25.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.041+0000 7f5eb10db640 1 --2- 172.21.15.57:0/1228768722 >> v2:172.21.15.88:3300/0 conn(0x7f5eac06c420 0x7f5eac0c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:25.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.041+0000 7f5eb10db640 1 -- 172.21.15.57:0/1228768722 >> v1:172.21.15.88:6789/0 conn(0x7f5eac153bb0 legacy=0x7f5eac0c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:25.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.041+0000 7f5eb10db640 1 -- 172.21.15.57:0/1228768722 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eac15a580 con 0x7f5eac0c7170 2023-10-30T14:29:25.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.041+0000 7f5eb10db640 1 --2- 172.21.15.57:0/1228768722 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eac0c7170 0x7f5eac159980 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f5eac06f800 tx=0x7f5ea401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b6da2d9681aad6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:25.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.045+0000 7f5eab7fe640 1 -- 172.21.15.57:0/1228768722 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea40af400 con 0x7f5eac0c7170 2023-10-30T14:29:25.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.045+0000 7f5eab7fe640 1 -- 172.21.15.57:0/1228768722 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea40ab050 con 0x7f5eac0c7170 2023-10-30T14:29:25.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.045+0000 7f5eab7fe640 1 -- 172.21.15.57:0/1228768722 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea40af780 con 0x7f5eac0c7170 2023-10-30T14:29:25.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.045+0000 7f5eb2b3e640 1 -- 172.21.15.57:0/1228768722 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eac0c7170 msgr2=0x7f5eac159980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:25.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.045+0000 7f5eb2b3e640 1 --2- 172.21.15.57:0/1228768722 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eac0c7170 0x7f5eac159980 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f5eac06f800 tx=0x7f5ea401f1c0 comp rx=0 tx=0).stop 2023-10-30T14:29:25.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.045+0000 7f5eb2b3e640 1 -- 172.21.15.57:0/1228768722 shutdown_connections 2023-10-30T14:29:25.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.045+0000 7f5eb2b3e640 1 --2- 172.21.15.57:0/1228768722 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eac0c7170 0x7f5eac159980 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:25.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.045+0000 7f5eb2b3e640 1 --2- 172.21.15.57:0/1228768722 >> v2:172.21.15.88:3300/0 conn(0x7f5eac06c420 0x7f5eac0c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:25.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.045+0000 7f5eb2b3e640 1 -- 172.21.15.57:0/1228768722 >> 172.21.15.57:0/1228768722 conn(0x7f5eac0cac70 msgr2=0x7f5eac0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:25.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.045+0000 7f5eb2b3e640 1 -- 172.21.15.57:0/1228768722 shutdown_connections 2023-10-30T14:29:25.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.045+0000 7f5eb2b3e640 1 -- 172.21.15.57:0/1228768722 wait complete. 2023-10-30T14:29:25.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.045+0000 7f5eb2b3e640 1 Processor -- start 2023-10-30T14:29:25.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.045+0000 7f5eb2b3e640 1 -- start start 2023-10-30T14:29:25.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.045+0000 7f5eb2b3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5eac06c420 0x7f5eac1ef180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:25.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.045+0000 7f5eb2b3e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5eac0c7170 0x7f5eac0c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:25.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.045+0000 7f5eb2b3e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eac153bb0 0x7f5eac0bfb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:25.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.045+0000 7f5eb2b3e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac1ef6c0 con 0x7f5eac153bb0 2023-10-30T14:29:25.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.045+0000 7f5eb2b3e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac070ac0 con 0x7f5eac06c420 2023-10-30T14:29:25.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.045+0000 7f5eb2b3e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac0bf800 con 0x7f5eac0c7170 2023-10-30T14:29:25.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5eb10db640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eac153bb0 0x7f5eac0bfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:25.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5eb10db640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eac153bb0 0x7f5eac0bfb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51958/0 (socket says 172.21.15.57:51958) 2023-10-30T14:29:25.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5eb10db640 1 -- 172.21.15.57:0/3342251811 learned_addr learned my addr 172.21.15.57:0/3342251811 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:25.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5eabfff640 1 --2- 172.21.15.57:0/3342251811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5eac0c7170 0x7f5eac0c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:25.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5eb08da640 1 --2- 172.21.15.57:0/3342251811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5eac06c420 0x7f5eac1ef180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:25.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5eb10db640 1 -- 172.21.15.57:0/3342251811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5eac0c7170 msgr2=0x7f5eac0c3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:25.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5eb10db640 1 --2- 172.21.15.57:0/3342251811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5eac0c7170 0x7f5eac0c3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:25.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5eb10db640 1 -- 172.21.15.57:0/3342251811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5eac06c420 msgr2=0x7f5eac1ef180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:25.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5eb10db640 1 --2- 172.21.15.57:0/3342251811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5eac06c420 0x7f5eac1ef180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:25.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5eb10db640 1 -- 172.21.15.57:0/3342251811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea4009b30 con 0x7f5eac153bb0 2023-10-30T14:29:25.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5eabfff640 1 --2- 172.21.15.57:0/3342251811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5eac0c7170 0x7f5eac0c3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:25.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5eb10db640 1 --2- 172.21.15.57:0/3342251811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eac153bb0 0x7f5eac0bfb10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f5ea40b2040 tx=0x7f5ea401bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:25.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5ea97fa640 1 -- 172.21.15.57:0/3342251811 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea40afcb0 con 0x7f5eac153bb0 2023-10-30T14:29:25.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5ea97fa640 1 -- 172.21.15.57:0/3342251811 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea40afe30 con 0x7f5eac153bb0 2023-10-30T14:29:25.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5ea97fa640 1 -- 172.21.15.57:0/3342251811 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea40ba660 con 0x7f5eac153bb0 2023-10-30T14:29:25.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5eb2b3e640 1 -- 172.21.15.57:0/3342251811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eac1f1cb0 con 0x7f5eac153bb0 2023-10-30T14:29:25.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5eb2b3e640 1 -- 172.21.15.57:0/3342251811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eac1f2270 con 0x7f5eac153bb0 2023-10-30T14:29:25.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5eb2b3e640 1 -- 172.21.15.57:0/3342251811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eac04e170 con 0x7f5eac153bb0 2023-10-30T14:29:25.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5ea97fa640 1 -- 172.21.15.57:0/3342251811 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5ea40b6040 con 0x7f5eac153bb0 2023-10-30T14:29:25.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5ea97fa640 1 --2- 172.21.15.57:0/3342251811 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5e900664d0 0x7f5e90068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:25.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.049+0000 7f5ea97fa640 1 -- 172.21.15.57:0/3342251811 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f5ea4122330 con 0x7f5eac153bb0 2023-10-30T14:29:25.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.053+0000 7f5eb08da640 1 --2- 172.21.15.57:0/3342251811 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5e900664d0 0x7f5e90068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:25.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.053+0000 7f5eb08da640 1 --2- 172.21.15.57:0/3342251811 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5e900664d0 0x7f5e90068990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0007fd0 tx=0x7f5ea0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:25.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.057+0000 7f5ea97fa640 1 -- 172.21.15.57:0/3342251811 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5ea40ef380 con 0x7f5eac153bb0 2023-10-30T14:29:25.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.245+0000 7f5eb2b3e640 1 -- 172.21.15.57:0/3342251811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5eac066510 con 0x7f5eac153bb0 2023-10-30T14:29:25.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.245+0000 7f5ea97fa640 1 -- 172.21.15.57:0/3342251811 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f5ea4009510 con 0x7f5eac153bb0 2023-10-30T14:29:25.249 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:25.249 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:24.192200+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:reconnect","state_seq":103,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:25.249 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 94 2023-10-30T14:29:25.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.249+0000 7f5eb2b3e640 1 -- 172.21.15.57:0/3342251811 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5e900664d0 msgr2=0x7f5e90068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:25.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.249+0000 7f5eb2b3e640 1 --2- 172.21.15.57:0/3342251811 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5e900664d0 0x7f5e90068990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0007fd0 tx=0x7f5ea0009040 comp rx=0 tx=0).stop 2023-10-30T14:29:25.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.249+0000 7f5eb2b3e640 1 -- 172.21.15.57:0/3342251811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eac153bb0 msgr2=0x7f5eac0bfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:25.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.249+0000 7f5eb2b3e640 1 --2- 172.21.15.57:0/3342251811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eac153bb0 0x7f5eac0bfb10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f5ea40b2040 tx=0x7f5ea401bf60 comp rx=0 tx=0).stop 2023-10-30T14:29:25.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.249+0000 7f5eb2b3e640 1 -- 172.21.15.57:0/3342251811 shutdown_connections 2023-10-30T14:29:25.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.249+0000 7f5eb2b3e640 1 --2- 172.21.15.57:0/3342251811 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5e900664d0 0x7f5e90068990 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:25.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.249+0000 7f5eb2b3e640 1 --2- 172.21.15.57:0/3342251811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eac153bb0 0x7f5eac0bfb10 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:25.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.249+0000 7f5eb2b3e640 1 --2- 172.21.15.57:0/3342251811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5eac0c7170 0x7f5eac0c3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:25.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.249+0000 7f5eb2b3e640 1 --2- 172.21.15.57:0/3342251811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5eac06c420 0x7f5eac1ef180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:25.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.249+0000 7f5eb2b3e640 1 -- 172.21.15.57:0/3342251811 >> 172.21.15.57:0/3342251811 conn(0x7f5eac0cac70 msgr2=0x7f5eac070170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:25.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.249+0000 7f5eb2b3e640 1 -- 172.21.15.57:0/3342251811 shutdown_connections 2023-10-30T14:29:25.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:25.249+0000 7f5eb2b3e640 1 -- 172.21.15.57:0/3342251811 wait complete. 2023-10-30T14:29:25.270 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:29:27.271 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:27.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.425+0000 7ff88c565640 1 Processor -- start 2023-10-30T14:29:27.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.425+0000 7ff88c565640 1 -- start start 2023-10-30T14:29:27.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.425+0000 7ff88c565640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff884151840 0x7ff8840c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:27.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.425+0000 7ff88c565640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8840c5030 con 0x7ff884151c20 2023-10-30T14:29:27.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.425+0000 7ff88c565640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff884157830 con 0x7ff8840cb8b0 2023-10-30T14:29:27.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.425+0000 7ff88c565640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff884157b10 con 0x7ff884151840 2023-10-30T14:29:27.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.425+0000 7ff889b00640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff884151c20 0x7ff884072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:60514/0 (socket says 172.21.15.57:60514) 2023-10-30T14:29:27.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.425+0000 7ff889b00640 1 -- 172.21.15.57:0/3443164085 learned_addr learned my addr 172.21.15.57:0/3443164085 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:29:27.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.425+0000 7ff88a301640 1 --2- 172.21.15.57:0/3443164085 >> v2:172.21.15.88:3300/0 conn(0x7ff884151840 0x7ff8840c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:27.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.425+0000 7ff8892ff640 1 -- 172.21.15.57:0/3443164085 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167632160 0 0) 0x7ff8840c5030 con 0x7ff884151c20 2023-10-30T14:29:27.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.425+0000 7ff8892ff640 1 -- 172.21.15.57:0/3443164085 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff870003530 con 0x7ff884151c20 2023-10-30T14:29:27.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.425+0000 7ff88a301640 1 -- 172.21.15.57:0/3443164085 >> v1:172.21.15.88:6789/0 conn(0x7ff8840cb8b0 legacy=0x7ff88406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:27.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.425+0000 7ff88a301640 1 -- 172.21.15.57:0/3443164085 >> v1:172.21.15.57:6789/0 conn(0x7ff884151c20 legacy=0x7ff884072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:27.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.425+0000 7ff88a301640 1 -- 172.21.15.57:0/3443164085 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff884157df0 con 0x7ff884151840 2023-10-30T14:29:27.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.429+0000 7ff88a301640 1 --2- 172.21.15.57:0/3443164085 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff884151840 0x7ff8840c49a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff8800097c0 tx=0x7ff88001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cecb71e3e2352b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:27.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.429+0000 7ff8892ff640 1 -- 172.21.15.57:0/3443164085 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8800a8970 con 0x7ff884151840 2023-10-30T14:29:27.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.429+0000 7ff8892ff640 1 -- 172.21.15.57:0/3443164085 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8800a8ad0 con 0x7ff884151840 2023-10-30T14:29:27.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.429+0000 7ff8892ff640 1 -- 172.21.15.57:0/3443164085 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8800a8e60 con 0x7ff884151840 2023-10-30T14:29:27.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.429+0000 7ff88c565640 1 -- 172.21.15.57:0/3443164085 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff884151840 msgr2=0x7ff8840c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:27.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.429+0000 7ff88c565640 1 --2- 172.21.15.57:0/3443164085 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff884151840 0x7ff8840c49a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff8800097c0 tx=0x7ff88001ec70 comp rx=0 tx=0).stop 2023-10-30T14:29:27.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.429+0000 7ff88c565640 1 -- 172.21.15.57:0/3443164085 shutdown_connections 2023-10-30T14:29:27.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.429+0000 7ff88c565640 1 --2- 172.21.15.57:0/3443164085 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff884151840 0x7ff8840c49a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:27.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.429+0000 7ff88c565640 1 -- 172.21.15.57:0/3443164085 >> 172.21.15.57:0/3443164085 conn(0x7ff8840cac70 msgr2=0x7ff8840cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:27.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.429+0000 7ff88c565640 1 -- 172.21.15.57:0/3443164085 shutdown_connections 2023-10-30T14:29:27.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.429+0000 7ff88c565640 1 -- 172.21.15.57:0/3443164085 wait complete. 2023-10-30T14:29:27.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88c565640 1 Processor -- start 2023-10-30T14:29:27.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88c565640 1 -- start start 2023-10-30T14:29:27.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88c565640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8840cb8b0 0x7ff8841f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:27.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88c565640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff884151840 0x7ff8841e2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:27.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88c565640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff884151c20 0x7ff8841465c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:27.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88c565640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8841e07c0 con 0x7ff884151c20 2023-10-30T14:29:27.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88c565640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8841e0130 con 0x7ff8840cb8b0 2023-10-30T14:29:27.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88c565640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8841e0410 con 0x7ff884151840 2023-10-30T14:29:27.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88ab02640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff884151c20 0x7ff8841465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:27.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88ab02640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff884151c20 0x7ff8841465c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51970/0 (socket says 172.21.15.57:51970) 2023-10-30T14:29:27.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88ab02640 1 -- 172.21.15.57:0/1561449583 learned_addr learned my addr 172.21.15.57:0/1561449583 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:27.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88a301640 1 --2- 172.21.15.57:0/1561449583 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8840cb8b0 0x7ff8841f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:27.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff889b00640 1 --2- 172.21.15.57:0/1561449583 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff884151840 0x7ff8841e2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:27.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88ab02640 1 -- 172.21.15.57:0/1561449583 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff884151840 msgr2=0x7ff8841e2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:27.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88ab02640 1 --2- 172.21.15.57:0/1561449583 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff884151840 0x7ff8841e2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:27.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88ab02640 1 -- 172.21.15.57:0/1561449583 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8840cb8b0 msgr2=0x7ff8841f2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:27.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88ab02640 1 --2- 172.21.15.57:0/1561449583 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8840cb8b0 0x7ff8841f2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:27.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88ab02640 1 -- 172.21.15.57:0/1561449583 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff880009420 con 0x7ff884151c20 2023-10-30T14:29:27.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88a301640 1 --2- 172.21.15.57:0/1561449583 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8840cb8b0 0x7ff8841f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:27.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88ab02640 1 --2- 172.21.15.57:0/1561449583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff884151c20 0x7ff8841465c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ff87c00c770 tx=0x7ff87c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:27.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff87affd640 1 -- 172.21.15.57:0/1561449583 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c003b80 con 0x7ff884151c20 2023-10-30T14:29:27.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff87affd640 1 -- 172.21.15.57:0/1561449583 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87c003ce0 con 0x7ff884151c20 2023-10-30T14:29:27.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff87affd640 1 -- 172.21.15.57:0/1561449583 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c00aa90 con 0x7ff884151c20 2023-10-30T14:29:27.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88c565640 1 -- 172.21.15.57:0/1561449583 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8841e09d0 con 0x7ff884151c20 2023-10-30T14:29:27.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.433+0000 7ff88c565640 1 -- 172.21.15.57:0/1561449583 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8841e0f90 con 0x7ff884151c20 2023-10-30T14:29:27.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.437+0000 7ff88c565640 1 -- 172.21.15.57:0/1561449583 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88404e170 con 0x7ff884151c20 2023-10-30T14:29:27.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.437+0000 7ff87affd640 1 -- 172.21.15.57:0/1561449583 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff87c00abf0 con 0x7ff884151c20 2023-10-30T14:29:27.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.437+0000 7ff87affd640 1 --2- 172.21.15.57:0/1561449583 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff8680664a0 0x7ff868068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:27.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.437+0000 7ff87affd640 1 -- 172.21.15.57:0/1561449583 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7ff87c084100 con 0x7ff884151c20 2023-10-30T14:29:27.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.437+0000 7ff88a301640 1 --2- 172.21.15.57:0/1561449583 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff8680664a0 0x7ff868068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:27.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.437+0000 7ff88a301640 1 --2- 172.21.15.57:0/1561449583 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff8680664a0 0x7ff868068960 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff8800b2000 tx=0x7ff880008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:27.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.441+0000 7ff87affd640 1 -- 172.21.15.57:0/1561449583 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff87c051160 con 0x7ff884151c20 2023-10-30T14:29:27.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.629+0000 7ff88c565640 1 -- 172.21.15.57:0/1561449583 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff884155060 con 0x7ff884151c20 2023-10-30T14:29:27.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.633+0000 7ff87affd640 1 -- 172.21.15.57:0/1561449583 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5736 (secure 0 0 0) 0x7ff87c054780 con 0x7ff884151c20 2023-10-30T14:29:27.633 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:27.633 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:24.192200+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:reconnect","state_seq":103,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:27.634 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 94 2023-10-30T14:29:27.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.633+0000 7ff88c565640 1 -- 172.21.15.57:0/1561449583 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff8680664a0 msgr2=0x7ff868068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:27.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.633+0000 7ff88c565640 1 --2- 172.21.15.57:0/1561449583 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff8680664a0 0x7ff868068960 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff8800b2000 tx=0x7ff880008040 comp rx=0 tx=0).stop 2023-10-30T14:29:27.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.633+0000 7ff88c565640 1 -- 172.21.15.57:0/1561449583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff884151c20 msgr2=0x7ff8841465c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:27.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.633+0000 7ff88c565640 1 --2- 172.21.15.57:0/1561449583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff884151c20 0x7ff8841465c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ff87c00c770 tx=0x7ff87c00cc30 comp rx=0 tx=0).stop 2023-10-30T14:29:27.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.633+0000 7ff88c565640 1 -- 172.21.15.57:0/1561449583 shutdown_connections 2023-10-30T14:29:27.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.633+0000 7ff88c565640 1 --2- 172.21.15.57:0/1561449583 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff8680664a0 0x7ff868068960 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:27.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.633+0000 7ff88c565640 1 --2- 172.21.15.57:0/1561449583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff884151c20 0x7ff8841465c0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:27.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.633+0000 7ff88c565640 1 --2- 172.21.15.57:0/1561449583 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff884151840 0x7ff8841e2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:27.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.633+0000 7ff88c565640 1 --2- 172.21.15.57:0/1561449583 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8840cb8b0 0x7ff8841f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:27.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.633+0000 7ff88c565640 1 -- 172.21.15.57:0/1561449583 >> 172.21.15.57:0/1561449583 conn(0x7ff8840cac70 msgr2=0x7ff88414ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:27.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.633+0000 7ff88c565640 1 -- 172.21.15.57:0/1561449583 shutdown_connections 2023-10-30T14:29:27.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:27.633+0000 7ff88c565640 1 -- 172.21.15.57:0/1561449583 wait complete. 2023-10-30T14:29:27.654 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:29:29.656 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:29.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.805+0000 7f25f9f65640 1 Processor -- start 2023-10-30T14:29:29.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.805+0000 7f25f9f65640 1 -- start start 2023-10-30T14:29:29.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.809+0000 7f25f9f65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25f40701a0 0x7f25f415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:29.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.809+0000 7f25f9f65640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f25f406b2f0 0x7f25f406b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:29.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.809+0000 7f25f9f65640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f40c2aa0 con 0x7f25f41537f0 2023-10-30T14:29:29.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.809+0000 7f25f9f65640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f40c3ab0 con 0x7f25f406b2f0 2023-10-30T14:29:29.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.809+0000 7f25f9f65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f415a820 con 0x7f25f40701a0 2023-10-30T14:29:29.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.809+0000 7f25f37fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f25f41537f0 0x7f25f40ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:60528/0 (socket says 172.21.15.57:60528) 2023-10-30T14:29:29.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.809+0000 7f25f37fe640 1 -- 172.21.15.57:0/2160342739 learned_addr learned my addr 172.21.15.57:0/2160342739 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:29:29.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.809+0000 7f25f3fff640 1 --2- 172.21.15.57:0/2160342739 >> v2:172.21.15.57:3300/0 conn(0x7f25f406b2f0 0x7f25f406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:29.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.809+0000 7f25f2ffd640 1 --2- 172.21.15.57:0/2160342739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25f40701a0 0x7f25f415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:29.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.809+0000 7f25f3fff640 1 -- 172.21.15.57:0/2160342739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25f40701a0 msgr2=0x7f25f415a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:29.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.809+0000 7f25f3fff640 1 --2- 172.21.15.57:0/2160342739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25f40701a0 0x7f25f415a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:29.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.809+0000 7f25f3fff640 1 -- 172.21.15.57:0/2160342739 >> v1:172.21.15.57:6789/0 conn(0x7f25f41537f0 legacy=0x7f25f40ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:29.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.809+0000 7f25f3fff640 1 -- 172.21.15.57:0/2160342739 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f406bca0 con 0x7f25f406b2f0 2023-10-30T14:29:29.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.809+0000 7f25f27fc640 1 -- 172.21.15.57:0/2160342739 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807981594 0 0) 0x7f25f40c2aa0 con 0x7f25f41537f0 2023-10-30T14:29:29.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.809+0000 7f25f2ffd640 1 --2- 172.21.15.57:0/2160342739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25f40701a0 0x7f25f415a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:29:29.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f3fff640 1 --2- 172.21.15.57:0/2160342739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25f406b2f0 0x7f25f406b760 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f25ec009d90 tx=0x7f25ec01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb4ca44ebc3f6daf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:29.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f27fc640 1 -- 172.21.15.57:0/2160342739 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec0af600 con 0x7f25f406b2f0 2023-10-30T14:29:29.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f27fc640 1 -- 172.21.15.57:0/2160342739 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ec0ab050 con 0x7f25f406b2f0 2023-10-30T14:29:29.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f27fc640 1 -- 172.21.15.57:0/2160342739 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec0af980 con 0x7f25f406b2f0 2023-10-30T14:29:29.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f9f65640 1 -- 172.21.15.57:0/2160342739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25f406b2f0 msgr2=0x7f25f406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:29.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f9f65640 1 --2- 172.21.15.57:0/2160342739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25f406b2f0 0x7f25f406b760 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f25ec009d90 tx=0x7f25ec01f130 comp rx=0 tx=0).stop 2023-10-30T14:29:29.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f9f65640 1 -- 172.21.15.57:0/2160342739 shutdown_connections 2023-10-30T14:29:29.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f9f65640 1 --2- 172.21.15.57:0/2160342739 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25f40701a0 0x7f25f415a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:29.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f9f65640 1 --2- 172.21.15.57:0/2160342739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25f406b2f0 0x7f25f406b760 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:29.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f9f65640 1 -- 172.21.15.57:0/2160342739 >> 172.21.15.57:0/2160342739 conn(0x7f25f4071fb0 msgr2=0x7f25f40723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:29.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f9f65640 1 -- 172.21.15.57:0/2160342739 shutdown_connections 2023-10-30T14:29:29.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f9f65640 1 -- 172.21.15.57:0/2160342739 wait complete. 2023-10-30T14:29:29.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f9f65640 1 Processor -- start 2023-10-30T14:29:29.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f9f65640 1 -- start start 2023-10-30T14:29:29.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f9f65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25f406b2f0 0x7f25f40c45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:29.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f9f65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25f40701a0 0x7f25f41efd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:29.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f9f65640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25f41537f0 0x7f25f41f02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:29.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f9f65640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f4155770 con 0x7f25f41537f0 2023-10-30T14:29:29.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f9f65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f4155a50 con 0x7f25f40701a0 2023-10-30T14:29:29.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.813+0000 7f25f9f65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f4155d30 con 0x7f25f406b2f0 2023-10-30T14:29:29.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25f3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25f41537f0 0x7f25f41f02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:29.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25f3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25f41537f0 0x7f25f41f02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51990/0 (socket says 172.21.15.57:51990) 2023-10-30T14:29:29.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25f3fff640 1 -- 172.21.15.57:0/90488596 learned_addr learned my addr 172.21.15.57:0/90488596 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:29.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25f37fe640 1 --2- 172.21.15.57:0/90488596 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25f406b2f0 0x7f25f40c45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:29.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25f2ffd640 1 --2- 172.21.15.57:0/90488596 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25f40701a0 0x7f25f41efd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:29.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25f3fff640 1 -- 172.21.15.57:0/90488596 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25f406b2f0 msgr2=0x7f25f40c45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:29.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25f3fff640 1 --2- 172.21.15.57:0/90488596 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25f406b2f0 0x7f25f40c45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:29.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25f3fff640 1 -- 172.21.15.57:0/90488596 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25f40701a0 msgr2=0x7f25f41efd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:29.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25f3fff640 1 --2- 172.21.15.57:0/90488596 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25f40701a0 0x7f25f41efd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:29.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25f3fff640 1 -- 172.21.15.57:0/90488596 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec009a70 con 0x7f25f41537f0 2023-10-30T14:29:29.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25f37fe640 1 --2- 172.21.15.57:0/90488596 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25f406b2f0 0x7f25f40c45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:29.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25f2ffd640 1 --2- 172.21.15.57:0/90488596 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25f40701a0 0x7f25f41efd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:29.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25f3fff640 1 --2- 172.21.15.57:0/90488596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25f41537f0 0x7f25f41f02b0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f25ec0b2040 tx=0x7f25ec01bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:29.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25dbfff640 1 -- 172.21.15.57:0/90488596 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec0b9610 con 0x7f25f41537f0 2023-10-30T14:29:29.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25dbfff640 1 -- 172.21.15.57:0/90488596 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ec0b9770 con 0x7f25f41537f0 2023-10-30T14:29:29.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25dbfff640 1 -- 172.21.15.57:0/90488596 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec0b9b20 con 0x7f25f41537f0 2023-10-30T14:29:29.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25f9f65640 1 -- 172.21.15.57:0/90488596 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f4156010 con 0x7f25f41537f0 2023-10-30T14:29:29.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25f9f65640 1 -- 172.21.15.57:0/90488596 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f41565d0 con 0x7f25f41537f0 2023-10-30T14:29:29.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25f9f65640 1 -- 172.21.15.57:0/90488596 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f41567c0 con 0x7f25f41537f0 2023-10-30T14:29:29.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.817+0000 7f25dbfff640 1 -- 172.21.15.57:0/90488596 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f25ec0b6050 con 0x7f25f41537f0 2023-10-30T14:29:29.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.821+0000 7f25dbfff640 1 --2- 172.21.15.57:0/90488596 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f25e00664a0 0x7f25e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:29.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.821+0000 7f25dbfff640 1 -- 172.21.15.57:0/90488596 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f25ec122800 con 0x7f25f41537f0 2023-10-30T14:29:29.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.821+0000 7f25f37fe640 1 --2- 172.21.15.57:0/90488596 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f25e00664a0 0x7f25e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:29.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.821+0000 7f25f37fe640 1 --2- 172.21.15.57:0/90488596 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f25e00664a0 0x7f25e0068960 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f25e4007fd0 tx=0x7f25e4007c20 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:29.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:29.825+0000 7f25dbfff640 1 -- 172.21.15.57:0/90488596 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f25ec0ef9b0 con 0x7f25f41537f0 2023-10-30T14:29:30.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:30.013+0000 7f25f9f65640 1 -- 172.21.15.57:0/90488596 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25f404e170 con 0x7f25f41537f0 2023-10-30T14:29:30.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:30.017+0000 7f25dbfff640 1 -- 172.21.15.57:0/90488596 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f25ec0f2fd0 con 0x7f25f41537f0 2023-10-30T14:29:30.017 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:30.018 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:24.192200+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:reconnect","state_seq":103,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:30.018 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 94 2023-10-30T14:29:30.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:30.017+0000 7f25f9f65640 1 -- 172.21.15.57:0/90488596 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f25e00664a0 msgr2=0x7f25e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:30.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:30.017+0000 7f25f9f65640 1 --2- 172.21.15.57:0/90488596 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f25e00664a0 0x7f25e0068960 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f25e4007fd0 tx=0x7f25e4007c20 comp rx=0 tx=0).stop 2023-10-30T14:29:30.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:30.017+0000 7f25f9f65640 1 -- 172.21.15.57:0/90488596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25f41537f0 msgr2=0x7f25f41f02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:30.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:30.017+0000 7f25f9f65640 1 --2- 172.21.15.57:0/90488596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25f41537f0 0x7f25f41f02b0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f25ec0b2040 tx=0x7f25ec01bed0 comp rx=0 tx=0).stop 2023-10-30T14:29:30.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:30.017+0000 7f25f9f65640 1 -- 172.21.15.57:0/90488596 shutdown_connections 2023-10-30T14:29:30.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:30.017+0000 7f25f9f65640 1 --2- 172.21.15.57:0/90488596 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f25e00664a0 0x7f25e0068960 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:30.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:30.017+0000 7f25f9f65640 1 --2- 172.21.15.57:0/90488596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25f41537f0 0x7f25f41f02b0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:30.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:30.017+0000 7f25f9f65640 1 --2- 172.21.15.57:0/90488596 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25f40701a0 0x7f25f41efd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:30.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:30.017+0000 7f25f9f65640 1 --2- 172.21.15.57:0/90488596 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25f406b2f0 0x7f25f40c45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:30.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:30.017+0000 7f25f9f65640 1 -- 172.21.15.57:0/90488596 >> 172.21.15.57:0/90488596 conn(0x7f25f4071fb0 msgr2=0x7f25f40cc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:30.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:30.017+0000 7f25f9f65640 1 -- 172.21.15.57:0/90488596 shutdown_connections 2023-10-30T14:29:30.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:30.017+0000 7f25f9f65640 1 -- 172.21.15.57:0/90488596 wait complete. 2023-10-30T14:29:30.038 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:29:32.040 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:32.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.193+0000 7fd6275aa640 1 Processor -- start 2023-10-30T14:29:32.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.193+0000 7fd6275aa640 1 -- start start 2023-10-30T14:29:32.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.193+0000 7fd6275aa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd620151840 0x7fd6200c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:32.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.193+0000 7fd6275aa640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd620151c20 0x7fd620072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:32.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.193+0000 7fd6275aa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd6200cb8b0 0x7fd620072540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:32.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.193+0000 7fd6275aa640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6200c5030 con 0x7fd620151c20 2023-10-30T14:29:32.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.193+0000 7fd6275aa640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd620157830 con 0x7fd6200cb8b0 2023-10-30T14:29:32.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.193+0000 7fd6275aa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd620157b10 con 0x7fd620151840 2023-10-30T14:29:32.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.197+0000 7fd624b45640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd620151c20 0x7fd620072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:32.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.197+0000 7fd624b45640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd620151c20 0x7fd620072000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52010/0 (socket says 172.21.15.57:52010) 2023-10-30T14:29:32.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.197+0000 7fd624b45640 1 -- 172.21.15.57:0/349230564 learned_addr learned my addr 172.21.15.57:0/349230564 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:32.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.197+0000 7fd625b47640 1 --2- 172.21.15.57:0/349230564 >> v2:172.21.15.88:3300/0 conn(0x7fd6200cb8b0 0x7fd620072540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:32.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.197+0000 7fd625346640 1 --2- 172.21.15.57:0/349230564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd620151840 0x7fd6200c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:32.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.197+0000 7fd624b45640 1 -- 172.21.15.57:0/349230564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd620151840 msgr2=0x7fd6200c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:32.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.197+0000 7fd624b45640 1 --2- 172.21.15.57:0/349230564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd620151840 0x7fd6200c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:32.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.197+0000 7fd624b45640 1 -- 172.21.15.57:0/349230564 >> v2:172.21.15.88:3300/0 conn(0x7fd6200cb8b0 msgr2=0x7fd620072540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:32.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.197+0000 7fd624b45640 1 --2- 172.21.15.57:0/349230564 >> v2:172.21.15.88:3300/0 conn(0x7fd6200cb8b0 0x7fd620072540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:32.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.197+0000 7fd624b45640 1 -- 172.21.15.57:0/349230564 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd620157df0 con 0x7fd620151c20 2023-10-30T14:29:32.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.197+0000 7fd624b45640 1 --2- 172.21.15.57:0/349230564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd620151c20 0x7fd620072000 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fd6200cb820 tx=0x7fd61001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e967b96d8c06cdd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:32.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.197+0000 7fd617fff640 1 -- 172.21.15.57:0/349230564 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6100a89b0 con 0x7fd620151c20 2023-10-30T14:29:32.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.197+0000 7fd617fff640 1 -- 172.21.15.57:0/349230564 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6100a8b10 con 0x7fd620151c20 2023-10-30T14:29:32.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.197+0000 7fd617fff640 1 -- 172.21.15.57:0/349230564 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6100a8ea0 con 0x7fd620151c20 2023-10-30T14:29:32.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd6275aa640 1 -- 172.21.15.57:0/349230564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd620151c20 msgr2=0x7fd620072000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:32.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd6275aa640 1 --2- 172.21.15.57:0/349230564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd620151c20 0x7fd620072000 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fd6200cb820 tx=0x7fd61001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:29:32.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd6275aa640 1 -- 172.21.15.57:0/349230564 shutdown_connections 2023-10-30T14:29:32.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd6275aa640 1 --2- 172.21.15.57:0/349230564 >> v2:172.21.15.88:3300/0 conn(0x7fd6200cb8b0 0x7fd620072540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:32.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd6275aa640 1 --2- 172.21.15.57:0/349230564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd620151c20 0x7fd620072000 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:32.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd6275aa640 1 --2- 172.21.15.57:0/349230564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd620151840 0x7fd6200c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:32.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd6275aa640 1 -- 172.21.15.57:0/349230564 >> 172.21.15.57:0/349230564 conn(0x7fd6200cac70 msgr2=0x7fd6200cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:32.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd6275aa640 1 -- 172.21.15.57:0/349230564 shutdown_connections 2023-10-30T14:29:32.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd6275aa640 1 -- 172.21.15.57:0/349230564 wait complete. 2023-10-30T14:29:32.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd6275aa640 1 Processor -- start 2023-10-30T14:29:32.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd6275aa640 1 -- start start 2023-10-30T14:29:32.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd6275aa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6200cb8b0 0x7fd6201f1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:32.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd6275aa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd620151840 0x7fd6201ef210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:32.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd6275aa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd620151c20 0x7fd6200c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:32.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd6275aa640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6201eedf0 con 0x7fd620151840 2023-10-30T14:29:32.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd6275aa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6201f2d10 con 0x7fd6200cb8b0 2023-10-30T14:29:32.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd6275aa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6201f2470 con 0x7fd620151c20 2023-10-30T14:29:32.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd624b45640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd620151840 0x7fd6201ef210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:32.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd624b45640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd620151840 0x7fd6201ef210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52018/0 (socket says 172.21.15.57:52018) 2023-10-30T14:29:32.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd624b45640 1 -- 172.21.15.57:0/687973302 learned_addr learned my addr 172.21.15.57:0/687973302 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:32.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd625346640 1 --2- 172.21.15.57:0/687973302 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6200cb8b0 0x7fd6201f1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:32.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd625b47640 1 --2- 172.21.15.57:0/687973302 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd620151c20 0x7fd6200c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:32.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd624b45640 1 -- 172.21.15.57:0/687973302 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd620151c20 msgr2=0x7fd6200c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:32.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd624b45640 1 --2- 172.21.15.57:0/687973302 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd620151c20 0x7fd6200c9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:32.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd624b45640 1 -- 172.21.15.57:0/687973302 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6200cb8b0 msgr2=0x7fd6201f1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:32.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd624b45640 1 --2- 172.21.15.57:0/687973302 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6200cb8b0 0x7fd6201f1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:32.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd624b45640 1 -- 172.21.15.57:0/687973302 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6100094d0 con 0x7fd620151840 2023-10-30T14:29:32.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.201+0000 7fd625346640 1 --2- 172.21.15.57:0/687973302 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6200cb8b0 0x7fd6201f1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:32.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.205+0000 7fd624b45640 1 --2- 172.21.15.57:0/687973302 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd620151840 0x7fd6201ef210 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fd61001d3a0 tx=0x7fd61001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:32.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.205+0000 7fd615ffb640 1 -- 172.21.15.57:0/687973302 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6100ba600 con 0x7fd620151840 2023-10-30T14:29:32.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.205+0000 7fd615ffb640 1 -- 172.21.15.57:0/687973302 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6100ba760 con 0x7fd620151840 2023-10-30T14:29:32.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.205+0000 7fd6275aa640 1 -- 172.21.15.57:0/687973302 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6201f2750 con 0x7fd620151840 2023-10-30T14:29:32.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.205+0000 7fd615ffb640 1 -- 172.21.15.57:0/687973302 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6100bab10 con 0x7fd620151840 2023-10-30T14:29:32.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.205+0000 7fd6275aa640 1 -- 172.21.15.57:0/687973302 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6201eea10 con 0x7fd620151840 2023-10-30T14:29:32.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.205+0000 7fd6275aa640 1 -- 172.21.15.57:0/687973302 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6201ee870 con 0x7fd620151840 2023-10-30T14:29:32.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.205+0000 7fd615ffb640 1 -- 172.21.15.57:0/687973302 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd6100bac70 con 0x7fd620151840 2023-10-30T14:29:32.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.205+0000 7fd615ffb640 1 --2- 172.21.15.57:0/687973302 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd6040664d0 0x7fd604068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:32.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.205+0000 7fd615ffb640 1 -- 172.21.15.57:0/687973302 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7fd610122710 con 0x7fd620151840 2023-10-30T14:29:32.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.209+0000 7fd625346640 1 --2- 172.21.15.57:0/687973302 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd6040664d0 0x7fd604068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:32.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.209+0000 7fd625346640 1 --2- 172.21.15.57:0/687973302 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd6040664d0 0x7fd604068990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd61c007a70 tx=0x7fd61c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:32.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.209+0000 7fd615ffb640 1 -- 172.21.15.57:0/687973302 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd6201ee870 con 0x7fd620151840 2023-10-30T14:29:32.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.401+0000 7fd6275aa640 1 -- 172.21.15.57:0/687973302 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6201ee870 con 0x7fd620151840 2023-10-30T14:29:32.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.401+0000 7fd615ffb640 1 -- 172.21.15.57:0/687973302 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5733 (secure 0 0 0) 0x7fd6201ee870 con 0x7fd620151840 2023-10-30T14:29:32.403 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:32.403 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:30.110189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:rejoin","state_seq":105,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:32.403 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 95 2023-10-30T14:29:32.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.401+0000 7fd6275aa640 1 -- 172.21.15.57:0/687973302 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd6040664d0 msgr2=0x7fd604068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:32.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.401+0000 7fd6275aa640 1 --2- 172.21.15.57:0/687973302 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd6040664d0 0x7fd604068990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd61c007a70 tx=0x7fd61c008040 comp rx=0 tx=0).stop 2023-10-30T14:29:32.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.401+0000 7fd6275aa640 1 -- 172.21.15.57:0/687973302 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd620151840 msgr2=0x7fd6201ef210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:32.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.401+0000 7fd6275aa640 1 --2- 172.21.15.57:0/687973302 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd620151840 0x7fd6201ef210 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fd61001d3a0 tx=0x7fd61001ba90 comp rx=0 tx=0).stop 2023-10-30T14:29:32.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.401+0000 7fd6275aa640 1 -- 172.21.15.57:0/687973302 shutdown_connections 2023-10-30T14:29:32.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.401+0000 7fd6275aa640 1 --2- 172.21.15.57:0/687973302 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd6040664d0 0x7fd604068990 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:32.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.401+0000 7fd6275aa640 1 --2- 172.21.15.57:0/687973302 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd620151c20 0x7fd6200c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:32.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.401+0000 7fd6275aa640 1 --2- 172.21.15.57:0/687973302 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd620151840 0x7fd6201ef210 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:32.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.401+0000 7fd6275aa640 1 --2- 172.21.15.57:0/687973302 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6200cb8b0 0x7fd6201f1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:32.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.401+0000 7fd6275aa640 1 -- 172.21.15.57:0/687973302 >> 172.21.15.57:0/687973302 conn(0x7fd6200cac70 msgr2=0x7fd6201e3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:32.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.401+0000 7fd6275aa640 1 -- 172.21.15.57:0/687973302 shutdown_connections 2023-10-30T14:29:32.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:32.401+0000 7fd6275aa640 1 -- 172.21.15.57:0/687973302 wait complete. 2023-10-30T14:29:32.424 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:29:34.425 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:34.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.577+0000 7f9bb13aa640 1 Processor -- start 2023-10-30T14:29:34.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.577+0000 7f9bb13aa640 1 -- start start 2023-10-30T14:29:34.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.581+0000 7f9bb13aa640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9bac153bb0 0x7f9bac0c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:34.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.581+0000 7f9bb13aa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9bac159980 0x7f9bac159d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:34.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.581+0000 7f9bb13aa640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bac0c5f80 con 0x7f9bac06c420 2023-10-30T14:29:34.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.581+0000 7f9bb13aa640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bac0c5690 con 0x7f9bac153bb0 2023-10-30T14:29:34.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.581+0000 7f9bb13aa640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bac15a2d0 con 0x7f9bac159980 2023-10-30T14:29:34.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.581+0000 7f9baaffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9bac153bb0 0x7f9bac0c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:34.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.581+0000 7f9baaffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9bac153bb0 0x7f9bac0c6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54784/0 (socket says 172.21.15.57:54784) 2023-10-30T14:29:34.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.581+0000 7f9baaffd640 1 -- 172.21.15.57:0/3320577793 learned_addr learned my addr 172.21.15.57:0/3320577793 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:34.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.581+0000 7f9bab7fe640 1 --2- 172.21.15.57:0/3320577793 >> v2:172.21.15.88:3300/0 conn(0x7f9bac159980 0x7f9bac159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:34.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.581+0000 7f9baaffd640 1 -- 172.21.15.57:0/3320577793 >> v2:172.21.15.88:3300/0 conn(0x7f9bac159980 msgr2=0x7f9bac159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:34.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.581+0000 7f9baaffd640 1 --2- 172.21.15.57:0/3320577793 >> v2:172.21.15.88:3300/0 conn(0x7f9bac159980 0x7f9bac159d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:34.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.581+0000 7f9baaffd640 1 -- 172.21.15.57:0/3320577793 >> v1:172.21.15.88:6789/0 conn(0x7f9bac06c420 legacy=0x7f9bac0c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:34.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.581+0000 7f9baaffd640 1 -- 172.21.15.57:0/3320577793 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac15a580 con 0x7f9bac153bb0 2023-10-30T14:29:34.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9baaffd640 1 --2- 172.21.15.57:0/3320577793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9bac153bb0 0x7f9bac0c6410 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f9bac0c8b20 tx=0x7f9b9c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eae55aa143d7119 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:34.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9baa7fc640 1 -- 172.21.15.57:0/3320577793 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c0a89f0 con 0x7f9bac153bb0 2023-10-30T14:29:34.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9baa7fc640 1 -- 172.21.15.57:0/3320577793 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9c0a8b50 con 0x7f9bac153bb0 2023-10-30T14:29:34.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9baa7fc640 1 -- 172.21.15.57:0/3320577793 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c0b0440 con 0x7f9bac153bb0 2023-10-30T14:29:34.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9bb13aa640 1 -- 172.21.15.57:0/3320577793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9bac153bb0 msgr2=0x7f9bac0c6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:34.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9bb13aa640 1 --2- 172.21.15.57:0/3320577793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9bac153bb0 0x7f9bac0c6410 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f9bac0c8b20 tx=0x7f9b9c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:29:34.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9bb13aa640 1 -- 172.21.15.57:0/3320577793 shutdown_connections 2023-10-30T14:29:34.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9bb13aa640 1 --2- 172.21.15.57:0/3320577793 >> v2:172.21.15.88:3300/0 conn(0x7f9bac159980 0x7f9bac159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:34.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9bb13aa640 1 --2- 172.21.15.57:0/3320577793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9bac153bb0 0x7f9bac0c6410 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:34.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9bb13aa640 1 -- 172.21.15.57:0/3320577793 >> 172.21.15.57:0/3320577793 conn(0x7f9bac0cac70 msgr2=0x7f9bac0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:34.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9bb13aa640 1 -- 172.21.15.57:0/3320577793 shutdown_connections 2023-10-30T14:29:34.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9bb13aa640 1 -- 172.21.15.57:0/3320577793 wait complete. 2023-10-30T14:29:34.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9bb13aa640 1 Processor -- start 2023-10-30T14:29:34.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9bb13aa640 1 -- start start 2023-10-30T14:29:34.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9bb13aa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9bac06c420 0x7f9bac0c2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:34.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9bb13aa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bac153bb0 0x7f9bac1f1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:34.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9bb13aa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bac159980 0x7f9bac14d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:34.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9bb13aa640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac1560c0 con 0x7f9bac06c420 2023-10-30T14:29:34.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9bb13aa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac1573b0 con 0x7f9bac153bb0 2023-10-30T14:29:34.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.585+0000 7f9bb13aa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac1587a0 con 0x7f9bac159980 2023-10-30T14:29:34.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9baaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9bac06c420 0x7f9bac0c2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:34.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9baaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9bac06c420 0x7f9bac0c2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54788/0 (socket says 172.21.15.57:54788) 2023-10-30T14:29:34.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9baaffd640 1 -- 172.21.15.57:0/2428838508 learned_addr learned my addr 172.21.15.57:0/2428838508 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:34.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9ba3fff640 1 --2- 172.21.15.57:0/2428838508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bac153bb0 0x7f9bac1f1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:34.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9bab7fe640 1 --2- 172.21.15.57:0/2428838508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bac159980 0x7f9bac14d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:34.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9baaffd640 1 -- 172.21.15.57:0/2428838508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bac159980 msgr2=0x7f9bac14d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:34.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9baaffd640 1 --2- 172.21.15.57:0/2428838508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bac159980 0x7f9bac14d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:34.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9baaffd640 1 -- 172.21.15.57:0/2428838508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bac153bb0 msgr2=0x7f9bac1f1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:34.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9baaffd640 1 --2- 172.21.15.57:0/2428838508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bac153bb0 0x7f9bac1f1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:34.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9baaffd640 1 -- 172.21.15.57:0/2428838508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c0094d0 con 0x7f9bac06c420 2023-10-30T14:29:34.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9ba3fff640 1 --2- 172.21.15.57:0/2428838508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bac153bb0 0x7f9bac1f1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:34.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9baaffd640 1 --2- 172.21.15.57:0/2428838508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9bac06c420 0x7f9bac0c2670 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c0b3040 tx=0x7f9b9c01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:34.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9ba37fe640 1 -- 172.21.15.57:0/2428838508 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c0ba600 con 0x7f9bac06c420 2023-10-30T14:29:34.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9ba37fe640 1 -- 172.21.15.57:0/2428838508 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9c0ba760 con 0x7f9bac06c420 2023-10-30T14:29:34.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9ba37fe640 1 -- 172.21.15.57:0/2428838508 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c0bab10 con 0x7f9bac06c420 2023-10-30T14:29:34.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9bb13aa640 1 -- 172.21.15.57:0/2428838508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bac1f2b10 con 0x7f9bac06c420 2023-10-30T14:29:34.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9bb13aa640 1 -- 172.21.15.57:0/2428838508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bac155a00 con 0x7f9bac06c420 2023-10-30T14:29:34.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9bb13aa640 1 -- 172.21.15.57:0/2428838508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bac066510 con 0x7f9bac06c420 2023-10-30T14:29:34.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9ba37fe640 1 -- 172.21.15.57:0/2428838508 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9b9c0bac70 con 0x7f9bac06c420 2023-10-30T14:29:34.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.589+0000 7f9ba37fe640 1 --2- 172.21.15.57:0/2428838508 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9b8c0664d0 0x7f9b8c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:34.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.593+0000 7f9ba37fe640 1 -- 172.21.15.57:0/2428838508 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f9b9c122780 con 0x7f9bac06c420 2023-10-30T14:29:34.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.593+0000 7f9ba3fff640 1 --2- 172.21.15.57:0/2428838508 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9b8c0664d0 0x7f9b8c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:34.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.593+0000 7f9ba3fff640 1 --2- 172.21.15.57:0/2428838508 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9b8c0664d0 0x7f9b8c068990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9bac06c2a0 tx=0x7f9b94009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:34.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.597+0000 7f9ba37fe640 1 -- 172.21.15.57:0/2428838508 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9b9c0ef7e0 con 0x7f9bac06c420 2023-10-30T14:29:34.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.789+0000 7f9bb13aa640 1 -- 172.21.15.57:0/2428838508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bac04e170 con 0x7f9bac06c420 2023-10-30T14:29:34.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.789+0000 7f9ba37fe640 1 -- 172.21.15.57:0/2428838508 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5733 (secure 0 0 0) 0x7f9b9c0f2e00 con 0x7f9bac06c420 2023-10-30T14:29:34.790 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:34.790 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:30.110189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:rejoin","state_seq":105,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:34.791 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 95 2023-10-30T14:29:34.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.789+0000 7f9bb13aa640 1 -- 172.21.15.57:0/2428838508 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9b8c0664d0 msgr2=0x7f9b8c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:34.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.789+0000 7f9bb13aa640 1 --2- 172.21.15.57:0/2428838508 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9b8c0664d0 0x7f9b8c068990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9bac06c2a0 tx=0x7f9b94009040 comp rx=0 tx=0).stop 2023-10-30T14:29:34.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.789+0000 7f9bb13aa640 1 -- 172.21.15.57:0/2428838508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9bac06c420 msgr2=0x7f9bac0c2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:34.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.789+0000 7f9bb13aa640 1 --2- 172.21.15.57:0/2428838508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9bac06c420 0x7f9bac0c2670 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c0b3040 tx=0x7f9b9c01ba90 comp rx=0 tx=0).stop 2023-10-30T14:29:34.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.789+0000 7f9bb13aa640 1 -- 172.21.15.57:0/2428838508 shutdown_connections 2023-10-30T14:29:34.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.789+0000 7f9bb13aa640 1 --2- 172.21.15.57:0/2428838508 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9b8c0664d0 0x7f9b8c068990 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:34.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.789+0000 7f9bb13aa640 1 --2- 172.21.15.57:0/2428838508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bac159980 0x7f9bac14d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:34.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.789+0000 7f9bb13aa640 1 --2- 172.21.15.57:0/2428838508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bac153bb0 0x7f9bac1f1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:34.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.789+0000 7f9bb13aa640 1 --2- 172.21.15.57:0/2428838508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9bac06c420 0x7f9bac0c2670 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:34.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.789+0000 7f9bb13aa640 1 -- 172.21.15.57:0/2428838508 >> 172.21.15.57:0/2428838508 conn(0x7f9bac0cac70 msgr2=0x7f9bac071230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:34.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.789+0000 7f9bb13aa640 1 -- 172.21.15.57:0/2428838508 shutdown_connections 2023-10-30T14:29:34.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:34.789+0000 7f9bb13aa640 1 -- 172.21.15.57:0/2428838508 wait complete. 2023-10-30T14:29:34.812 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 95, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 95, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5315": { "addr": "172.21.15.88:6839/358521496", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 358521496, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 358521496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5315, "incarnation": 92, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:rejoin", "state_seq": 105 }, "gid_6027": { "addr": "172.21.15.57:6836/2733323524", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2733323524, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2733323524, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6027, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 24 }, "gid_6204": { "addr": "172.21.15.57:6837/1843528309", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 1843528309, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 1843528309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6204, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:29:30.110189+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5315, "mds_1": 6027, "mds_2": 6204 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:29:35.297 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:29:35.301 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:29:36.813 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:36.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.969+0000 7f3be43a1640 1 Processor -- start 2023-10-30T14:29:36.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.969+0000 7f3be43a1640 1 -- start start 2023-10-30T14:29:36.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.973+0000 7f3be43a1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3bdc150830 0x7f3bdc06b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:36.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.973+0000 7f3be43a1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3bdc0cb7d0 0x7f3bdc0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:36.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.973+0000 7f3be43a1640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bdc0c2310 con 0x7f3bdc0c3e40 2023-10-30T14:29:36.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.973+0000 7f3be43a1640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bdc156710 con 0x7f3bdc150830 2023-10-30T14:29:36.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.973+0000 7f3be43a1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bdc0c4360 con 0x7f3bdc0cb7d0 2023-10-30T14:29:36.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.973+0000 7f3be213d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3bdc150830 0x7f3bdc06b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:36.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.973+0000 7f3be293e640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f3bdc0c3e40 0x7f3bdc070520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:33802/0 (socket says 172.21.15.57:33802) 2023-10-30T14:29:36.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.973+0000 7f3be213d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3bdc150830 0x7f3bdc06b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54790/0 (socket says 172.21.15.57:54790) 2023-10-30T14:29:36.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.973+0000 7f3be293e640 1 -- 172.21.15.57:0/1247091384 learned_addr learned my addr 172.21.15.57:0/1247091384 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:29:36.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.973+0000 7f3be193c640 1 --2- 172.21.15.57:0/1247091384 >> v2:172.21.15.88:3300/0 conn(0x7f3bdc0cb7d0 0x7f3bdc0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:36.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.973+0000 7f3be213d640 1 -- 172.21.15.57:0/1247091384 >> v2:172.21.15.88:3300/0 conn(0x7f3bdc0cb7d0 msgr2=0x7f3bdc0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:36.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.973+0000 7f3be213d640 1 --2- 172.21.15.57:0/1247091384 >> v2:172.21.15.88:3300/0 conn(0x7f3bdc0cb7d0 0x7f3bdc0c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:36.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.973+0000 7f3be213d640 1 -- 172.21.15.57:0/1247091384 >> v1:172.21.15.57:6789/0 conn(0x7f3bdc0c3e40 legacy=0x7f3bdc070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:36.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.973+0000 7f3be213d640 1 -- 172.21.15.57:0/1247091384 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bdc1569b0 con 0x7f3bdc150830 2023-10-30T14:29:36.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.973+0000 7f3be113b640 1 -- 172.21.15.57:0/1247091384 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 387823623 0 0) 0x7f3bdc0c2310 con 0x7f3bdc0c3e40 2023-10-30T14:29:36.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.977+0000 7f3be213d640 1 --2- 172.21.15.57:0/1247091384 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3bdc150830 0x7f3bdc06b500 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc06f9f0 tx=0x7f3bd801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f14509fa12cb78 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:36.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.977+0000 7f3be113b640 1 -- 172.21.15.57:0/1247091384 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd80a8950 con 0x7f3bdc150830 2023-10-30T14:29:36.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.977+0000 7f3be113b640 1 -- 172.21.15.57:0/1247091384 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd80a8ab0 con 0x7f3bdc150830 2023-10-30T14:29:36.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.977+0000 7f3be113b640 1 -- 172.21.15.57:0/1247091384 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd80a8e40 con 0x7f3bdc150830 2023-10-30T14:29:36.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.977+0000 7f3be43a1640 1 -- 172.21.15.57:0/1247091384 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3bdc150830 msgr2=0x7f3bdc06b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:36.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.977+0000 7f3be43a1640 1 --2- 172.21.15.57:0/1247091384 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3bdc150830 0x7f3bdc06b500 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc06f9f0 tx=0x7f3bd801ec50 comp rx=0 tx=0).stop 2023-10-30T14:29:36.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.977+0000 7f3be43a1640 1 -- 172.21.15.57:0/1247091384 shutdown_connections 2023-10-30T14:29:36.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.977+0000 7f3be43a1640 1 --2- 172.21.15.57:0/1247091384 >> v2:172.21.15.88:3300/0 conn(0x7f3bdc0cb7d0 0x7f3bdc0c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:36.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.977+0000 7f3be43a1640 1 --2- 172.21.15.57:0/1247091384 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3bdc150830 0x7f3bdc06b500 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:36.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.977+0000 7f3be43a1640 1 -- 172.21.15.57:0/1247091384 >> 172.21.15.57:0/1247091384 conn(0x7f3bdc0cac70 msgr2=0x7f3bdc0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:36.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.977+0000 7f3be43a1640 1 -- 172.21.15.57:0/1247091384 shutdown_connections 2023-10-30T14:29:36.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.977+0000 7f3be43a1640 1 -- 172.21.15.57:0/1247091384 wait complete. 2023-10-30T14:29:36.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.977+0000 7f3be43a1640 1 Processor -- start 2023-10-30T14:29:36.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.977+0000 7f3be43a1640 1 -- start start 2023-10-30T14:29:36.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be43a1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3bdc0c3e40 0x7f3bdc13afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:36.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be43a1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3bdc0cb7d0 0x7f3bdc13c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:36.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be43a1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bdc150830 0x7f3bdc13cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:36.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be43a1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc163fd0 con 0x7f3bdc0c3e40 2023-10-30T14:29:36.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be43a1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc13f040 con 0x7f3bdc150830 2023-10-30T14:29:36.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be43a1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc1efac0 con 0x7f3bdc0cb7d0 2023-10-30T14:29:36.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be213d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3bdc0c3e40 0x7f3bdc13afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:36.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be213d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3bdc0c3e40 0x7f3bdc13afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54798/0 (socket says 172.21.15.57:54798) 2023-10-30T14:29:36.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be213d640 1 -- 172.21.15.57:0/554833731 learned_addr learned my addr 172.21.15.57:0/554833731 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:36.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be193c640 1 --2- 172.21.15.57:0/554833731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3bdc0cb7d0 0x7f3bdc13c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:36.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be293e640 1 --2- 172.21.15.57:0/554833731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bdc150830 0x7f3bdc13cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:36.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be213d640 1 -- 172.21.15.57:0/554833731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3bdc0cb7d0 msgr2=0x7f3bdc13c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:36.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be213d640 1 --2- 172.21.15.57:0/554833731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3bdc0cb7d0 0x7f3bdc13c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:36.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be213d640 1 -- 172.21.15.57:0/554833731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bdc150830 msgr2=0x7f3bdc13cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:36.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be213d640 1 --2- 172.21.15.57:0/554833731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bdc150830 0x7f3bdc13cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:36.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be213d640 1 -- 172.21.15.57:0/554833731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd8009420 con 0x7f3bdc0c3e40 2023-10-30T14:29:36.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be293e640 1 --2- 172.21.15.57:0/554833731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bdc150830 0x7f3bdc13cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:29:36.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be213d640 1 --2- 172.21.15.57:0/554833731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3bdc0c3e40 0x7f3bdc13afc0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3bd80b3040 tx=0x7f3bd801b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:36.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3bd2ffd640 1 -- 172.21.15.57:0/554833731 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd80b0df0 con 0x7f3bdc0c3e40 2023-10-30T14:29:36.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3bd2ffd640 1 -- 172.21.15.57:0/554833731 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd80b9410 con 0x7f3bdc0c3e40 2023-10-30T14:29:36.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be43a1640 1 -- 172.21.15.57:0/554833731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bdc1efda0 con 0x7f3bdc0c3e40 2023-10-30T14:29:36.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3bd2ffd640 1 -- 172.21.15.57:0/554833731 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd80b97c0 con 0x7f3bdc0c3e40 2023-10-30T14:29:36.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be43a1640 1 -- 172.21.15.57:0/554833731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bdc163bf0 con 0x7f3bdc0c3e40 2023-10-30T14:29:36.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.981+0000 7f3be43a1640 1 -- 172.21.15.57:0/554833731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bdc066510 con 0x7f3bdc0c3e40 2023-10-30T14:29:36.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.985+0000 7f3bd2ffd640 1 -- 172.21.15.57:0/554833731 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3bd80b7040 con 0x7f3bdc0c3e40 2023-10-30T14:29:36.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.985+0000 7f3bd2ffd640 1 --2- 172.21.15.57:0/554833731 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3bc00664a0 0x7f3bc0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:36.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.985+0000 7f3bd2ffd640 1 -- 172.21.15.57:0/554833731 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f3bd8121e20 con 0x7f3bdc0c3e40 2023-10-30T14:29:36.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.985+0000 7f3be193c640 1 --2- 172.21.15.57:0/554833731 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3bc00664a0 0x7f3bc0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:36.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.985+0000 7f3be193c640 1 --2- 172.21.15.57:0/554833731 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3bc00664a0 0x7f3bc0068960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc0728f0 tx=0x7f3bcc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:36.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:36.989+0000 7f3bd2ffd640 1 -- 172.21.15.57:0/554833731 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3bd80eee80 con 0x7f3bdc0c3e40 2023-10-30T14:29:37.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:37.181+0000 7f3be43a1640 1 -- 172.21.15.57:0/554833731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bdc13c430 con 0x7f3bdc0c3e40 2023-10-30T14:29:37.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:37.181+0000 7f3bd2ffd640 1 -- 172.21.15.57:0/554833731 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f3bd80b5070 con 0x7f3bdc0c3e40 2023-10-30T14:29:37.184 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:37.184 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:36.071458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:clientreplay","state_seq":107,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:37.184 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 96 2023-10-30T14:29:37.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:37.185+0000 7f3be43a1640 1 -- 172.21.15.57:0/554833731 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3bc00664a0 msgr2=0x7f3bc0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:37.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:37.185+0000 7f3be43a1640 1 --2- 172.21.15.57:0/554833731 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3bc00664a0 0x7f3bc0068960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc0728f0 tx=0x7f3bcc008040 comp rx=0 tx=0).stop 2023-10-30T14:29:37.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:37.185+0000 7f3be43a1640 1 -- 172.21.15.57:0/554833731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3bdc0c3e40 msgr2=0x7f3bdc13afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:37.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:37.185+0000 7f3be43a1640 1 --2- 172.21.15.57:0/554833731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3bdc0c3e40 0x7f3bdc13afc0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3bd80b3040 tx=0x7f3bd801b9f0 comp rx=0 tx=0).stop 2023-10-30T14:29:37.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:37.185+0000 7f3be43a1640 1 -- 172.21.15.57:0/554833731 shutdown_connections 2023-10-30T14:29:37.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:37.185+0000 7f3be43a1640 1 --2- 172.21.15.57:0/554833731 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3bc00664a0 0x7f3bc0068960 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:37.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:37.185+0000 7f3be43a1640 1 --2- 172.21.15.57:0/554833731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bdc150830 0x7f3bdc13cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:37.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:37.185+0000 7f3be43a1640 1 --2- 172.21.15.57:0/554833731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3bdc0cb7d0 0x7f3bdc13c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:37.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:37.185+0000 7f3be43a1640 1 --2- 172.21.15.57:0/554833731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3bdc0c3e40 0x7f3bdc13afc0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:37.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:37.185+0000 7f3be43a1640 1 -- 172.21.15.57:0/554833731 >> 172.21.15.57:0/554833731 conn(0x7f3bdc0cac70 msgr2=0x7f3bdc147ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:37.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:37.185+0000 7f3be43a1640 1 -- 172.21.15.57:0/554833731 shutdown_connections 2023-10-30T14:29:37.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:37.185+0000 7f3be43a1640 1 -- 172.21.15.57:0/554833731 wait complete. 2023-10-30T14:29:37.204 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:29:39.206 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:39.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.357+0000 7fc5f4646640 1 Processor -- start 2023-10-30T14:29:39.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.357+0000 7fc5f4646640 1 -- start start 2023-10-30T14:29:39.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.361+0000 7fc5f4646640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5ec151840 0x7fc5ec0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:39.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.361+0000 7fc5f4646640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ec0c5030 con 0x7fc5ec151c20 2023-10-30T14:29:39.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.361+0000 7fc5f4646640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ec157830 con 0x7fc5ec0cb8b0 2023-10-30T14:29:39.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.361+0000 7fc5f4646640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec157b10 con 0x7fc5ec151840 2023-10-30T14:29:39.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.361+0000 7fc5f1be1640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc5ec151c20 0x7fc5ec072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:33814/0 (socket says 172.21.15.57:33814) 2023-10-30T14:29:39.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.361+0000 7fc5f1be1640 1 -- 172.21.15.57:0/4229627495 learned_addr learned my addr 172.21.15.57:0/4229627495 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:29:39.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.361+0000 7fc5f23e2640 1 --2- 172.21.15.57:0/4229627495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5ec151840 0x7fc5ec0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:39.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.361+0000 7fc5f13e0640 1 -- 172.21.15.57:0/4229627495 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92069762 0 0) 0x7fc5ec0c5030 con 0x7fc5ec151c20 2023-10-30T14:29:39.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.361+0000 7fc5f13e0640 1 -- 172.21.15.57:0/4229627495 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d8003530 con 0x7fc5ec151c20 2023-10-30T14:29:39.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.361+0000 7fc5f13e0640 1 -- 172.21.15.57:0/4229627495 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846681005 0 0) 0x7fc5ec157830 con 0x7fc5ec0cb8b0 2023-10-30T14:29:39.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.361+0000 7fc5f13e0640 1 -- 172.21.15.57:0/4229627495 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5ec0c5030 con 0x7fc5ec0cb8b0 2023-10-30T14:29:39.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.361+0000 7fc5f23e2640 1 -- 172.21.15.57:0/4229627495 >> v1:172.21.15.88:6789/0 conn(0x7fc5ec0cb8b0 legacy=0x7fc5ec06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:39.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.361+0000 7fc5f23e2640 1 -- 172.21.15.57:0/4229627495 >> v1:172.21.15.57:6789/0 conn(0x7fc5ec151c20 legacy=0x7fc5ec072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:39.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.361+0000 7fc5f23e2640 1 -- 172.21.15.57:0/4229627495 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec157df0 con 0x7fc5ec151840 2023-10-30T14:29:39.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.365+0000 7fc5f23e2640 1 --2- 172.21.15.57:0/4229627495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5ec151840 0x7fc5ec0c49a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc5e80097f0 tx=0x7fc5e801ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c827876c95bd08a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:39.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.365+0000 7fc5f13e0640 1 -- 172.21.15.57:0/4229627495 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e80a89b0 con 0x7fc5ec151840 2023-10-30T14:29:39.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.365+0000 7fc5f13e0640 1 -- 172.21.15.57:0/4229627495 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e80a8b10 con 0x7fc5ec151840 2023-10-30T14:29:39.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.365+0000 7fc5f13e0640 1 -- 172.21.15.57:0/4229627495 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e80a8ea0 con 0x7fc5ec151840 2023-10-30T14:29:39.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.365+0000 7fc5f4646640 1 -- 172.21.15.57:0/4229627495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5ec151840 msgr2=0x7fc5ec0c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:39.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.365+0000 7fc5f4646640 1 --2- 172.21.15.57:0/4229627495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5ec151840 0x7fc5ec0c49a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc5e80097f0 tx=0x7fc5e801ecb0 comp rx=0 tx=0).stop 2023-10-30T14:29:39.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.365+0000 7fc5f4646640 1 -- 172.21.15.57:0/4229627495 shutdown_connections 2023-10-30T14:29:39.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.365+0000 7fc5f4646640 1 --2- 172.21.15.57:0/4229627495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5ec151840 0x7fc5ec0c49a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:39.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.365+0000 7fc5f4646640 1 -- 172.21.15.57:0/4229627495 >> 172.21.15.57:0/4229627495 conn(0x7fc5ec0cac70 msgr2=0x7fc5ec0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:39.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.365+0000 7fc5f4646640 1 -- 172.21.15.57:0/4229627495 shutdown_connections 2023-10-30T14:29:39.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.365+0000 7fc5f4646640 1 -- 172.21.15.57:0/4229627495 wait complete. 2023-10-30T14:29:39.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f4646640 1 Processor -- start 2023-10-30T14:29:39.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f4646640 1 -- start start 2023-10-30T14:29:39.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f4646640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5ec0cb8b0 0x7fc5ec14cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:39.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f4646640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5ec151840 0x7fc5ec152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:39.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f4646640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5ec151c20 0x7fc5ec152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:39.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f4646640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec153f70 con 0x7fc5ec151840 2023-10-30T14:29:39.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f4646640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec1ef4c0 con 0x7fc5ec0cb8b0 2023-10-30T14:29:39.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f4646640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec153360 con 0x7fc5ec151c20 2023-10-30T14:29:39.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f1be1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5ec151840 0x7fc5ec152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:39.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f1be1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5ec151840 0x7fc5ec152030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54804/0 (socket says 172.21.15.57:54804) 2023-10-30T14:29:39.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f1be1640 1 -- 172.21.15.57:0/3289012267 learned_addr learned my addr 172.21.15.57:0/3289012267 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:39.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f2be3640 1 --2- 172.21.15.57:0/3289012267 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5ec151c20 0x7fc5ec152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:39.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f23e2640 1 --2- 172.21.15.57:0/3289012267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5ec0cb8b0 0x7fc5ec14cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:39.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f1be1640 1 -- 172.21.15.57:0/3289012267 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5ec151c20 msgr2=0x7fc5ec152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:39.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f1be1640 1 --2- 172.21.15.57:0/3289012267 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5ec151c20 0x7fc5ec152570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:39.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f1be1640 1 -- 172.21.15.57:0/3289012267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5ec0cb8b0 msgr2=0x7fc5ec14cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:39.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f1be1640 1 --2- 172.21.15.57:0/3289012267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5ec0cb8b0 0x7fc5ec14cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:39.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f1be1640 1 -- 172.21.15.57:0/3289012267 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e8009420 con 0x7fc5ec151840 2023-10-30T14:29:39.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f23e2640 1 --2- 172.21.15.57:0/3289012267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5ec0cb8b0 0x7fc5ec14cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:39.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f1be1640 1 --2- 172.21.15.57:0/3289012267 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5ec151840 0x7fc5ec152030 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc005890 tx=0x7fc5dc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:39.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5e2ffd640 1 -- 172.21.15.57:0/3289012267 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc005cf0 con 0x7fc5ec151840 2023-10-30T14:29:39.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5e2ffd640 1 -- 172.21.15.57:0/3289012267 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5dc005e70 con 0x7fc5ec151840 2023-10-30T14:29:39.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f4646640 1 -- 172.21.15.57:0/3289012267 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ec153640 con 0x7fc5ec151840 2023-10-30T14:29:39.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5e2ffd640 1 -- 172.21.15.57:0/3289012267 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc0072f0 con 0x7fc5ec151840 2023-10-30T14:29:39.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.369+0000 7fc5f4646640 1 -- 172.21.15.57:0/3289012267 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5ec153c00 con 0x7fc5ec151840 2023-10-30T14:29:39.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.373+0000 7fc5f4646640 1 -- 172.21.15.57:0/3289012267 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5ec04e170 con 0x7fc5ec151840 2023-10-30T14:29:39.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.373+0000 7fc5e2ffd640 1 -- 172.21.15.57:0/3289012267 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc5dc007450 con 0x7fc5ec151840 2023-10-30T14:29:39.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.373+0000 7fc5e2ffd640 1 --2- 172.21.15.57:0/3289012267 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5d00664a0 0x7fc5d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:39.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.373+0000 7fc5e2ffd640 1 -- 172.21.15.57:0/3289012267 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7fc5dc083810 con 0x7fc5ec151840 2023-10-30T14:29:39.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.373+0000 7fc5f23e2640 1 --2- 172.21.15.57:0/3289012267 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5d00664a0 0x7fc5d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:39.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.373+0000 7fc5f23e2640 1 --2- 172.21.15.57:0/3289012267 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5d00664a0 0x7fc5d0068960 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc5e80b2000 tx=0x7fc5e8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:39.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.377+0000 7fc5e2ffd640 1 -- 172.21.15.57:0/3289012267 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc5dc050930 con 0x7fc5ec151840 2023-10-30T14:29:39.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.569+0000 7fc5f4646640 1 -- 172.21.15.57:0/3289012267 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5ec155060 con 0x7fc5ec151840 2023-10-30T14:29:39.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.573+0000 7fc5e2ffd640 1 -- 172.21.15.57:0/3289012267 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+5739 (secure 0 0 0) 0x7fc5dc017090 con 0x7fc5ec151840 2023-10-30T14:29:39.573 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:39.574 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:36.071458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:clientreplay","state_seq":107,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:39.574 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 96 2023-10-30T14:29:39.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.573+0000 7fc5f4646640 1 -- 172.21.15.57:0/3289012267 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5d00664a0 msgr2=0x7fc5d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:39.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.573+0000 7fc5f4646640 1 --2- 172.21.15.57:0/3289012267 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5d00664a0 0x7fc5d0068960 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc5e80b2000 tx=0x7fc5e8008040 comp rx=0 tx=0).stop 2023-10-30T14:29:39.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.573+0000 7fc5f4646640 1 -- 172.21.15.57:0/3289012267 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5ec151840 msgr2=0x7fc5ec152030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:39.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.573+0000 7fc5f4646640 1 --2- 172.21.15.57:0/3289012267 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5ec151840 0x7fc5ec152030 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc005890 tx=0x7fc5dc00c8b0 comp rx=0 tx=0).stop 2023-10-30T14:29:39.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.573+0000 7fc5f4646640 1 -- 172.21.15.57:0/3289012267 shutdown_connections 2023-10-30T14:29:39.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.573+0000 7fc5f4646640 1 --2- 172.21.15.57:0/3289012267 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5d00664a0 0x7fc5d0068960 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:39.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.573+0000 7fc5f4646640 1 --2- 172.21.15.57:0/3289012267 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5ec151c20 0x7fc5ec152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:39.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.573+0000 7fc5f4646640 1 --2- 172.21.15.57:0/3289012267 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5ec151840 0x7fc5ec152030 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:39.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.573+0000 7fc5f4646640 1 --2- 172.21.15.57:0/3289012267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5ec0cb8b0 0x7fc5ec14cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:39.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.573+0000 7fc5f4646640 1 -- 172.21.15.57:0/3289012267 >> 172.21.15.57:0/3289012267 conn(0x7fc5ec0cac70 msgr2=0x7fc5ec0c30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:39.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.573+0000 7fc5f4646640 1 -- 172.21.15.57:0/3289012267 shutdown_connections 2023-10-30T14:29:39.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:39.573+0000 7fc5f4646640 1 -- 172.21.15.57:0/3289012267 wait complete. 2023-10-30T14:29:39.595 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:29:41.595 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:41.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0efdbc5640 1 Processor -- start 2023-10-30T14:29:41.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0efdbc5640 1 -- start start 2023-10-30T14:29:41.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0efdbc5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef81537f0 0x7f0ef80701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:41.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0efdbc5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ef80c2aa0 con 0x7f0ef815a2e0 2023-10-30T14:29:41.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0efdbc5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ef80708e0 con 0x7f0ef806b2f0 2023-10-30T14:29:41.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0efdbc5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef80c3ab0 con 0x7f0ef81537f0 2023-10-30T14:29:41.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef6ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f0ef815a2e0 0x7f0ef80ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:33824/0 (socket says 172.21.15.57:33824) 2023-10-30T14:29:41.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef6ffd640 1 -- 172.21.15.57:0/2659961968 learned_addr learned my addr 172.21.15.57:0/2659961968 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:29:41.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef77fe640 1 --2- 172.21.15.57:0/2659961968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef81537f0 0x7f0ef80701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:41.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef67fc640 1 -- 172.21.15.57:0/2659961968 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430620720 0 0) 0x7f0ef80c2aa0 con 0x7f0ef815a2e0 2023-10-30T14:29:41.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef67fc640 1 -- 172.21.15.57:0/2659961968 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ee0003530 con 0x7f0ef815a2e0 2023-10-30T14:29:41.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef67fc640 1 -- 172.21.15.57:0/2659961968 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216043407 0 0) 0x7f0ef80708e0 con 0x7f0ef806b2f0 2023-10-30T14:29:41.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef67fc640 1 -- 172.21.15.57:0/2659961968 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ef80c2aa0 con 0x7f0ef806b2f0 2023-10-30T14:29:41.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef67fc640 1 -- 172.21.15.57:0/2659961968 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 455452876 0 0) 0x7f0ee0003530 con 0x7f0ef815a2e0 2023-10-30T14:29:41.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef67fc640 1 -- 172.21.15.57:0/2659961968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef81537f0 msgr2=0x7f0ef80701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:41.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef67fc640 1 --2- 172.21.15.57:0/2659961968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef81537f0 0x7f0ef80701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:41.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef67fc640 1 -- 172.21.15.57:0/2659961968 >> v1:172.21.15.88:6789/0 conn(0x7f0ef806b2f0 legacy=0x7f0ef81641e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:41.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef67fc640 1 -- 172.21.15.57:0/2659961968 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef80c1eb0 con 0x7f0ef815a2e0 2023-10-30T14:29:41.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef67fc640 1 -- 172.21.15.57:0/2659961968 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f0eec001680 con 0x7f0ef815a2e0 2023-10-30T14:29:41.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef67fc640 1 -- 172.21.15.57:0/2659961968 >> v1:172.21.15.57:6789/0 conn(0x7f0ef815a2e0 legacy=0x7f0ef80ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:41.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef67fc640 1 --2- 172.21.15.57:0/2659961968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ee0003b40 0x7f0ee0003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:41.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef67fc640 1 --2- 172.21.15.57:0/2659961968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef81537f0 0x7f0ee00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:41.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef67fc640 1 --2- 172.21.15.57:0/2659961968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ee0004d10 0x7f0ee0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:41.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef67fc640 1 -- 172.21.15.57:0/2659961968 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee000ba40 con 0x7f0ee0003b40 2023-10-30T14:29:41.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef67fc640 1 -- 172.21.15.57:0/2659961968 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee000bb80 con 0x7f0ee0004d10 2023-10-30T14:29:41.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef67fc640 1 -- 172.21.15.57:0/2659961968 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee000bcf0 con 0x7f0ef81537f0 2023-10-30T14:29:41.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef77fe640 1 --2- 172.21.15.57:0/2659961968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ee0003b40 0x7f0ee0003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:41.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef6ffd640 1 --2- 172.21.15.57:0/2659961968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef81537f0 0x7f0ee00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:41.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.749+0000 7f0ef7fff640 1 --2- 172.21.15.57:0/2659961968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ee0004d10 0x7f0ee0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:41.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.753+0000 7f0ef77fe640 1 -- 172.21.15.57:0/2659961968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef81537f0 msgr2=0x7f0ee00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:41.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.753+0000 7f0ef77fe640 1 --2- 172.21.15.57:0/2659961968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef81537f0 0x7f0ee00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:41.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.753+0000 7f0ef77fe640 1 -- 172.21.15.57:0/2659961968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ee0004d10 msgr2=0x7f0ee0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:41.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.753+0000 7f0ef77fe640 1 --2- 172.21.15.57:0/2659961968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ee0004d10 0x7f0ee0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:41.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.753+0000 7f0ef77fe640 1 -- 172.21.15.57:0/2659961968 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee0003530 con 0x7f0ee0003b40 2023-10-30T14:29:41.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.753+0000 7f0ef6ffd640 1 --2- 172.21.15.57:0/2659961968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef81537f0 0x7f0ee00047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:29:41.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.753+0000 7f0ef7fff640 1 --2- 172.21.15.57:0/2659961968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ee0004d10 0x7f0ee0005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:29:41.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.753+0000 7f0ef77fe640 1 --2- 172.21.15.57:0/2659961968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ee0003b40 0x7f0ee0003f40 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009ae0 tx=0x7f0ee801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537c90895ff9807b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:41.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.753+0000 7f0ef67fc640 1 -- 172.21.15.57:0/2659961968 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee80a5530 con 0x7f0ee0003b40 2023-10-30T14:29:41.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.753+0000 7f0ef67fc640 1 -- 172.21.15.57:0/2659961968 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee80a5690 con 0x7f0ee0003b40 2023-10-30T14:29:41.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.753+0000 7f0ef67fc640 1 -- 172.21.15.57:0/2659961968 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee80a5a40 con 0x7f0ee0003b40 2023-10-30T14:29:41.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.753+0000 7f0efdbc5640 1 -- 172.21.15.57:0/2659961968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ee0003b40 msgr2=0x7f0ee0003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:41.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.753+0000 7f0efdbc5640 1 --2- 172.21.15.57:0/2659961968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ee0003b40 0x7f0ee0003f40 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009ae0 tx=0x7f0ee801ca50 comp rx=0 tx=0).stop 2023-10-30T14:29:41.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.753+0000 7f0ef7fff640 1 -- 172.21.15.57:0/2659961968 reap_dead start 2023-10-30T14:29:41.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.753+0000 7f0efdbc5640 1 -- 172.21.15.57:0/2659961968 shutdown_connections 2023-10-30T14:29:41.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.753+0000 7f0efdbc5640 1 -- 172.21.15.57:0/2659961968 >> 172.21.15.57:0/2659961968 conn(0x7f0ef8071fb0 msgr2=0x7f0ef80723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:41.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.753+0000 7f0efdbc5640 1 -- 172.21.15.57:0/2659961968 shutdown_connections 2023-10-30T14:29:41.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.753+0000 7f0efdbc5640 1 -- 172.21.15.57:0/2659961968 wait complete. 2023-10-30T14:29:41.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0efdbc5640 1 Processor -- start 2023-10-30T14:29:41.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0efdbc5640 1 -- start start 2023-10-30T14:29:41.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0efdbc5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ee0003b40 0x7f0ef80c8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:41.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0efdbc5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ef8155570 0x7f0ef8156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:41.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0efdbc5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef81f2220 0x7f0ef8151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:41.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0efdbc5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef81541f0 con 0x7f0ef8155570 2023-10-30T14:29:41.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0efdbc5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef81544d0 con 0x7f0ee0003b40 2023-10-30T14:29:41.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0efdbc5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef81547b0 con 0x7f0ef81f2220 2023-10-30T14:29:41.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0ef6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ef8155570 0x7f0ef8156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:41.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0ef6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ef8155570 0x7f0ef8156210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54820/0 (socket says 172.21.15.57:54820) 2023-10-30T14:29:41.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0ef6ffd640 1 -- 172.21.15.57:0/3270821254 learned_addr learned my addr 172.21.15.57:0/3270821254 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:41.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0ef7fff640 1 --2- 172.21.15.57:0/3270821254 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef81f2220 0x7f0ef8151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:41.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0ef77fe640 1 --2- 172.21.15.57:0/3270821254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ee0003b40 0x7f0ef80c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:41.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0ef6ffd640 1 -- 172.21.15.57:0/3270821254 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef81f2220 msgr2=0x7f0ef8151740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:41.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0ef6ffd640 1 --2- 172.21.15.57:0/3270821254 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef81f2220 0x7f0ef8151740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:41.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0ef6ffd640 1 -- 172.21.15.57:0/3270821254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ee0003b40 msgr2=0x7f0ef80c8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:41.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0ef6ffd640 1 --2- 172.21.15.57:0/3270821254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ee0003b40 0x7f0ef80c8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:41.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0ef6ffd640 1 -- 172.21.15.57:0/3270821254 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef8154a90 con 0x7f0ef8155570 2023-10-30T14:29:41.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0ef7fff640 1 --2- 172.21.15.57:0/3270821254 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef81f2220 0x7f0ef8151740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:41.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0ef77fe640 1 --2- 172.21.15.57:0/3270821254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ee0003b40 0x7f0ef80c8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:41.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0ef6ffd640 1 --2- 172.21.15.57:0/3270821254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ef8155570 0x7f0ef8156210 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00d7e0 tx=0x7f0eec00dca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:41.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0edffff640 1 -- 172.21.15.57:0/3270821254 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec0074e0 con 0x7f0ef8155570 2023-10-30T14:29:41.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0edffff640 1 -- 172.21.15.57:0/3270821254 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eec007640 con 0x7f0ef8155570 2023-10-30T14:29:41.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0edffff640 1 -- 172.21.15.57:0/3270821254 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec004a30 con 0x7f0ef8155570 2023-10-30T14:29:41.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.757+0000 7f0efdbc5640 1 -- 172.21.15.57:0/3270821254 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef8154d70 con 0x7f0ef8155570 2023-10-30T14:29:41.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.761+0000 7f0efdbc5640 1 -- 172.21.15.57:0/3270821254 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef8156ed0 con 0x7f0ef8155570 2023-10-30T14:29:41.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.761+0000 7f0efdbc5640 1 -- 172.21.15.57:0/3270821254 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ef81548f0 con 0x7f0ef8155570 2023-10-30T14:29:41.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.761+0000 7f0edffff640 1 -- 172.21.15.57:0/3270821254 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0eec004b90 con 0x7f0ef8155570 2023-10-30T14:29:41.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.761+0000 7f0edffff640 1 --2- 172.21.15.57:0/3270821254 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0ee40664a0 0x7f0ee4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:41.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.761+0000 7f0edffff640 1 -- 172.21.15.57:0/3270821254 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f0eec083c20 con 0x7f0ef8155570 2023-10-30T14:29:41.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.761+0000 7f0ef77fe640 1 --2- 172.21.15.57:0/3270821254 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0ee40664a0 0x7f0ee4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:41.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.765+0000 7f0ef77fe640 1 --2- 172.21.15.57:0/3270821254 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0ee40664a0 0x7f0ee4068960 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80af000 tx=0x7f0ee80046d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:41.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.765+0000 7f0edffff640 1 -- 172.21.15.57:0/3270821254 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0ef81548f0 con 0x7f0ef8155570 2023-10-30T14:29:41.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.957+0000 7f0efdbc5640 1 -- 172.21.15.57:0/3270821254 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ef81548f0 con 0x7f0ef8155570 2023-10-30T14:29:41.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.957+0000 7f0edffff640 1 -- 172.21.15.57:0/3270821254 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f0ef81548f0 con 0x7f0ef8155570 2023-10-30T14:29:41.960 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:41.960 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:36.071458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:clientreplay","state_seq":107,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:41.961 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 96 2023-10-30T14:29:41.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.961+0000 7f0efdbc5640 1 -- 172.21.15.57:0/3270821254 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0ee40664a0 msgr2=0x7f0ee4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:41.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.961+0000 7f0efdbc5640 1 --2- 172.21.15.57:0/3270821254 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0ee40664a0 0x7f0ee4068960 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80af000 tx=0x7f0ee80046d0 comp rx=0 tx=0).stop 2023-10-30T14:29:41.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.961+0000 7f0efdbc5640 1 -- 172.21.15.57:0/3270821254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ef8155570 msgr2=0x7f0ef8156210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:41.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.961+0000 7f0efdbc5640 1 --2- 172.21.15.57:0/3270821254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ef8155570 0x7f0ef8156210 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00d7e0 tx=0x7f0eec00dca0 comp rx=0 tx=0).stop 2023-10-30T14:29:41.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.961+0000 7f0efdbc5640 1 -- 172.21.15.57:0/3270821254 shutdown_connections 2023-10-30T14:29:41.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.961+0000 7f0efdbc5640 1 --2- 172.21.15.57:0/3270821254 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0ee40664a0 0x7f0ee4068960 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:41.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.961+0000 7f0efdbc5640 1 --2- 172.21.15.57:0/3270821254 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef81f2220 0x7f0ef8151740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:41.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.961+0000 7f0efdbc5640 1 --2- 172.21.15.57:0/3270821254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ef8155570 0x7f0ef8156210 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:41.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.961+0000 7f0efdbc5640 1 --2- 172.21.15.57:0/3270821254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ee0003b40 0x7f0ef80c8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:41.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.961+0000 7f0efdbc5640 1 -- 172.21.15.57:0/3270821254 >> 172.21.15.57:0/3270821254 conn(0x7f0ef8071fb0 msgr2=0x7f0ef8152a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:41.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.961+0000 7f0efdbc5640 1 -- 172.21.15.57:0/3270821254 shutdown_connections 2023-10-30T14:29:41.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:41.961+0000 7f0efdbc5640 1 -- 172.21.15.57:0/3270821254 wait complete. 2023-10-30T14:29:41.982 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:29:43.983 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:44.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.137+0000 7fa3bf5d3640 1 Processor -- start 2023-10-30T14:29:44.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.137+0000 7fa3bf5d3640 1 -- start start 2023-10-30T14:29:44.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.137+0000 7fa3bf5d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3b8151c20 0x7fa3b80c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:44.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.137+0000 7fa3bf5d3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa3b80cb8b0 0x7fa3b806b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:44.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.137+0000 7fa3bf5d3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b80c5030 con 0x7fa3b8151840 2023-10-30T14:29:44.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.137+0000 7fa3bf5d3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3b8157830 con 0x7fa3b80cb8b0 2023-10-30T14:29:44.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.137+0000 7fa3bf5d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b8157b10 con 0x7fa3b8151c20 2023-10-30T14:29:44.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.137+0000 7fa3bdb70640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa3b80cb8b0 0x7fa3b806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:44.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.137+0000 7fa3bdb70640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa3b80cb8b0 0x7fa3b806b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47620/0 (socket says 172.21.15.57:47620) 2023-10-30T14:29:44.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.137+0000 7fa3bdb70640 1 -- 172.21.15.57:0/3734380198 learned_addr learned my addr 172.21.15.57:0/3734380198 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:44.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.137+0000 7fa3bcb6e640 1 --2- 172.21.15.57:0/3734380198 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3b8151c20 0x7fa3b80c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:44.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.137+0000 7fa3bdb70640 1 -- 172.21.15.57:0/3734380198 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3b8151c20 msgr2=0x7fa3b80c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:44.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.137+0000 7fa3bdb70640 1 --2- 172.21.15.57:0/3734380198 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3b8151c20 0x7fa3b80c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:44.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.137+0000 7fa3bdb70640 1 -- 172.21.15.57:0/3734380198 >> v1:172.21.15.88:6789/0 conn(0x7fa3b8151840 legacy=0x7fa3b8072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:44.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.137+0000 7fa3bdb70640 1 -- 172.21.15.57:0/3734380198 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b8157df0 con 0x7fa3b80cb8b0 2023-10-30T14:29:44.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.141+0000 7fa3bdb70640 1 --2- 172.21.15.57:0/3734380198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3b80cb8b0 0x7fa3b806b600 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0009e50 tx=0x7fa3b001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30b4aa910782362c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:44.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.141+0000 7fa3affff640 1 -- 172.21.15.57:0/3734380198 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b00af400 con 0x7fa3b80cb8b0 2023-10-30T14:29:44.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.141+0000 7fa3affff640 1 -- 172.21.15.57:0/3734380198 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b00ab050 con 0x7fa3b80cb8b0 2023-10-30T14:29:44.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.141+0000 7fa3affff640 1 -- 172.21.15.57:0/3734380198 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b00af780 con 0x7fa3b80cb8b0 2023-10-30T14:29:44.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.141+0000 7fa3bf5d3640 1 -- 172.21.15.57:0/3734380198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3b80cb8b0 msgr2=0x7fa3b806b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:44.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.141+0000 7fa3bf5d3640 1 --2- 172.21.15.57:0/3734380198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3b80cb8b0 0x7fa3b806b600 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0009e50 tx=0x7fa3b001f1f0 comp rx=0 tx=0).stop 2023-10-30T14:29:44.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.141+0000 7fa3bf5d3640 1 -- 172.21.15.57:0/3734380198 shutdown_connections 2023-10-30T14:29:44.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.141+0000 7fa3bf5d3640 1 --2- 172.21.15.57:0/3734380198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3b80cb8b0 0x7fa3b806b600 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:44.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.141+0000 7fa3bf5d3640 1 --2- 172.21.15.57:0/3734380198 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3b8151c20 0x7fa3b80c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:44.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.141+0000 7fa3bf5d3640 1 -- 172.21.15.57:0/3734380198 >> 172.21.15.57:0/3734380198 conn(0x7fa3b80cac70 msgr2=0x7fa3b80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:44.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.141+0000 7fa3bf5d3640 1 -- 172.21.15.57:0/3734380198 shutdown_connections 2023-10-30T14:29:44.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.141+0000 7fa3bf5d3640 1 -- 172.21.15.57:0/3734380198 wait complete. 2023-10-30T14:29:44.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bf5d3640 1 Processor -- start 2023-10-30T14:29:44.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bf5d3640 1 -- start start 2023-10-30T14:29:44.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bf5d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3b80cb8b0 0x7fa3b81e38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:44.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bf5d3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3b8151840 0x7fa3b80c7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:44.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bf5d3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3b8151c20 0x7fa3b80c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:44.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bf5d3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b81ef0b0 con 0x7fa3b8151840 2023-10-30T14:29:44.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bf5d3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b81ef390 con 0x7fa3b8151c20 2023-10-30T14:29:44.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bf5d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b81ef670 con 0x7fa3b80cb8b0 2023-10-30T14:29:44.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bcb6e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3b8151840 0x7fa3b80c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:44.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bcb6e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3b8151840 0x7fa3b80c7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47632/0 (socket says 172.21.15.57:47632) 2023-10-30T14:29:44.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bcb6e640 1 -- 172.21.15.57:0/867799329 learned_addr learned my addr 172.21.15.57:0/867799329 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:44.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bd36f640 1 --2- 172.21.15.57:0/867799329 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3b80cb8b0 0x7fa3b81e38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:44.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bdb70640 1 --2- 172.21.15.57:0/867799329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3b8151c20 0x7fa3b80c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:44.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bcb6e640 1 -- 172.21.15.57:0/867799329 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3b80cb8b0 msgr2=0x7fa3b81e38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:44.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bcb6e640 1 --2- 172.21.15.57:0/867799329 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3b80cb8b0 0x7fa3b81e38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:44.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bcb6e640 1 -- 172.21.15.57:0/867799329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3b8151c20 msgr2=0x7fa3b80c7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:44.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bcb6e640 1 --2- 172.21.15.57:0/867799329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3b8151c20 0x7fa3b80c7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:44.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bcb6e640 1 -- 172.21.15.57:0/867799329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b0009b30 con 0x7fa3b8151840 2023-10-30T14:29:44.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bcb6e640 1 --2- 172.21.15.57:0/867799329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3b8151840 0x7fa3b80c7680 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8006e90 tx=0x7fa3a800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:44.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3adffb640 1 -- 172.21.15.57:0/867799329 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a800b6c0 con 0x7fa3b8151840 2023-10-30T14:29:44.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3adffb640 1 -- 172.21.15.57:0/867799329 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a800b840 con 0x7fa3b8151840 2023-10-30T14:29:44.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3adffb640 1 -- 172.21.15.57:0/867799329 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a800bbf0 con 0x7fa3b8151840 2023-10-30T14:29:44.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bf5d3640 1 -- 172.21.15.57:0/867799329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b81ef950 con 0x7fa3b8151840 2023-10-30T14:29:44.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.145+0000 7fa3bf5d3640 1 -- 172.21.15.57:0/867799329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b81e2070 con 0x7fa3b8151840 2023-10-30T14:29:44.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.149+0000 7fa3bf5d3640 1 -- 172.21.15.57:0/867799329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b81efe70 con 0x7fa3b8151840 2023-10-30T14:29:44.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.149+0000 7fa3adffb640 1 -- 172.21.15.57:0/867799329 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa3a800bd50 con 0x7fa3b8151840 2023-10-30T14:29:44.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.149+0000 7fa3adffb640 1 --2- 172.21.15.57:0/867799329 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa39c0664d0 0x7fa39c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:44.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.149+0000 7fa3adffb640 1 -- 172.21.15.57:0/867799329 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7fa3a80845e0 con 0x7fa3b8151840 2023-10-30T14:29:44.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.149+0000 7fa3bd36f640 1 --2- 172.21.15.57:0/867799329 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa39c0664d0 0x7fa39c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:44.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.149+0000 7fa3bd36f640 1 --2- 172.21.15.57:0/867799329 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa39c0664d0 0x7fa39c068990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4007fd0 tx=0x7fa3b4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:44.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.153+0000 7fa3adffb640 1 -- 172.21.15.57:0/867799329 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa3b81efe70 con 0x7fa3b8151840 2023-10-30T14:29:44.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.341+0000 7fa3bf5d3640 1 -- 172.21.15.57:0/867799329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3b81efe70 con 0x7fa3b8151840 2023-10-30T14:29:44.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.341+0000 7fa3adffb640 1 -- 172.21.15.57:0/867799329 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5733 (secure 0 0 0) 0x7fa3b81efe70 con 0x7fa3b8151840 2023-10-30T14:29:44.344 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:44.345 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:42.182782+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:active","state_seq":109,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:44.345 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 97 2023-10-30T14:29:44.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.345+0000 7fa3bf5d3640 1 -- 172.21.15.57:0/867799329 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa39c0664d0 msgr2=0x7fa39c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:44.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.345+0000 7fa3bf5d3640 1 --2- 172.21.15.57:0/867799329 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa39c0664d0 0x7fa39c068990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4007fd0 tx=0x7fa3b4009040 comp rx=0 tx=0).stop 2023-10-30T14:29:44.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.345+0000 7fa3bf5d3640 1 -- 172.21.15.57:0/867799329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3b8151840 msgr2=0x7fa3b80c7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:44.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.345+0000 7fa3bf5d3640 1 --2- 172.21.15.57:0/867799329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3b8151840 0x7fa3b80c7680 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8006e90 tx=0x7fa3a800dc60 comp rx=0 tx=0).stop 2023-10-30T14:29:44.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.345+0000 7fa3bf5d3640 1 -- 172.21.15.57:0/867799329 shutdown_connections 2023-10-30T14:29:44.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.345+0000 7fa3bf5d3640 1 --2- 172.21.15.57:0/867799329 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa39c0664d0 0x7fa39c068990 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:44.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.345+0000 7fa3bf5d3640 1 --2- 172.21.15.57:0/867799329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3b8151c20 0x7fa3b80c7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:44.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.345+0000 7fa3bf5d3640 1 --2- 172.21.15.57:0/867799329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3b8151840 0x7fa3b80c7680 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:44.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.345+0000 7fa3bf5d3640 1 --2- 172.21.15.57:0/867799329 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3b80cb8b0 0x7fa3b81e38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:44.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.345+0000 7fa3bf5d3640 1 -- 172.21.15.57:0/867799329 >> 172.21.15.57:0/867799329 conn(0x7fa3b80cac70 msgr2=0x7fa3b81dd800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:44.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.345+0000 7fa3bf5d3640 1 -- 172.21.15.57:0/867799329 shutdown_connections 2023-10-30T14:29:44.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:44.345+0000 7fa3bf5d3640 1 -- 172.21.15.57:0/867799329 wait complete. 2023-10-30T14:29:44.365 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=0, replacing gid=5797 2023-10-30T14:29:44.366 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.a 2023-10-30T14:29:46.367 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T14:29:46.367 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T14:29:46.368 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T14:29:46.372 INFO:tasks.ceph.mds.a:Started 2023-10-30T14:29:46.372 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:46.467 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-30T14:29:46.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.521+0000 7f1f321d3640 1 Processor -- start 2023-10-30T14:29:46.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.521+0000 7f1f321d3640 1 -- start start 2023-10-30T14:29:46.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.525+0000 7f1f321d3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1f2c0701a0 0x7f1f2c15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:46.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.525+0000 7f1f321d3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c0c2aa0 con 0x7f1f2c1537f0 2023-10-30T14:29:46.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.525+0000 7f1f321d3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c0708e0 con 0x7f1f2c06b2f0 2023-10-30T14:29:46.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.525+0000 7f1f321d3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2c0c3ab0 con 0x7f1f2c0701a0 2023-10-30T14:29:46.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.525+0000 7f1f2b7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1f2c1537f0 0x7f1f2c0ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:54716/0 (socket says 172.21.15.57:54716) 2023-10-30T14:29:46.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.525+0000 7f1f2b7fe640 1 -- 172.21.15.57:0/1543626407 learned_addr learned my addr 172.21.15.57:0/1543626407 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:29:46.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.525+0000 7f1f2affd640 1 --2- 172.21.15.57:0/1543626407 >> v2:172.21.15.57:3300/0 conn(0x7f1f2c0701a0 0x7f1f2c15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:46.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.525+0000 7f1f2a7fc640 1 -- 172.21.15.57:0/1543626407 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773516750 0 0) 0x7f1f2c0c2aa0 con 0x7f1f2c1537f0 2023-10-30T14:29:46.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.525+0000 7f1f2a7fc640 1 -- 172.21.15.57:0/1543626407 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f14003530 con 0x7f1f2c1537f0 2023-10-30T14:29:46.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.525+0000 7f1f2affd640 1 -- 172.21.15.57:0/1543626407 >> v1:172.21.15.88:6789/0 conn(0x7f1f2c06b2f0 legacy=0x7f1f2c06b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:46.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.525+0000 7f1f2affd640 1 -- 172.21.15.57:0/1543626407 >> v1:172.21.15.57:6789/0 conn(0x7f1f2c1537f0 legacy=0x7f1f2c0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:46.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.525+0000 7f1f2affd640 1 -- 172.21.15.57:0/1543626407 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c0c1eb0 con 0x7f1f2c0701a0 2023-10-30T14:29:46.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.529+0000 7f1f2affd640 1 --2- 172.21.15.57:0/1543626407 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f2c0701a0 0x7f1f2c15a2e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f1f200097c0 tx=0x7f1f2001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b15e84b9a3400d3f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:46.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.529+0000 7f1f2a7fc640 1 -- 172.21.15.57:0/1543626407 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f200a8970 con 0x7f1f2c0701a0 2023-10-30T14:29:46.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.529+0000 7f1f2a7fc640 1 -- 172.21.15.57:0/1543626407 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f200a8ad0 con 0x7f1f2c0701a0 2023-10-30T14:29:46.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.529+0000 7f1f2a7fc640 1 -- 172.21.15.57:0/1543626407 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f200a8e60 con 0x7f1f2c0701a0 2023-10-30T14:29:46.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.529+0000 7f1f321d3640 1 -- 172.21.15.57:0/1543626407 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f2c0701a0 msgr2=0x7f1f2c15a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:46.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.529+0000 7f1f321d3640 1 --2- 172.21.15.57:0/1543626407 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f2c0701a0 0x7f1f2c15a2e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f1f200097c0 tx=0x7f1f2001ec70 comp rx=0 tx=0).stop 2023-10-30T14:29:46.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.529+0000 7f1f321d3640 1 -- 172.21.15.57:0/1543626407 shutdown_connections 2023-10-30T14:29:46.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.529+0000 7f1f321d3640 1 --2- 172.21.15.57:0/1543626407 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f2c0701a0 0x7f1f2c15a2e0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:46.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.529+0000 7f1f321d3640 1 -- 172.21.15.57:0/1543626407 >> 172.21.15.57:0/1543626407 conn(0x7f1f2c071fb0 msgr2=0x7f1f2c0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:46.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.529+0000 7f1f321d3640 1 -- 172.21.15.57:0/1543626407 shutdown_connections 2023-10-30T14:29:46.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.529+0000 7f1f321d3640 1 -- 172.21.15.57:0/1543626407 wait complete. 2023-10-30T14:29:46.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.529+0000 7f1f321d3640 1 Processor -- start 2023-10-30T14:29:46.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.529+0000 7f1f321d3640 1 -- start start 2023-10-30T14:29:46.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f321d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f2c06b2f0 0x7f1f2c14c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:46.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f321d3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f2c0701a0 0x7f1f2c158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:46.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f321d3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f2c1537f0 0x7f1f2c1591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:46.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f321d3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c0c7200 con 0x7f1f2c1537f0 2023-10-30T14:29:46.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f321d3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c0ca580 con 0x7f1f2c0701a0 2023-10-30T14:29:46.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f321d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c0c9ce0 con 0x7f1f2c06b2f0 2023-10-30T14:29:46.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f2bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f2c1537f0 0x7f1f2c1591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:46.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f2bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f2c1537f0 0x7f1f2c1591e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47652/0 (socket says 172.21.15.57:47652) 2023-10-30T14:29:46.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f2bfff640 1 -- 172.21.15.57:0/3566193128 learned_addr learned my addr 172.21.15.57:0/3566193128 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:46.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f2b7fe640 1 --2- 172.21.15.57:0/3566193128 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f2c06b2f0 0x7f1f2c14c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:46.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f2affd640 1 --2- 172.21.15.57:0/3566193128 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f2c0701a0 0x7f1f2c158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:46.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f2bfff640 1 -- 172.21.15.57:0/3566193128 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f2c06b2f0 msgr2=0x7f1f2c14c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:46.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f2bfff640 1 --2- 172.21.15.57:0/3566193128 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f2c06b2f0 0x7f1f2c14c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:46.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f2bfff640 1 -- 172.21.15.57:0/3566193128 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f2c0701a0 msgr2=0x7f1f2c158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:46.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f2bfff640 1 --2- 172.21.15.57:0/3566193128 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f2c0701a0 0x7f1f2c158ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:46.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f2bfff640 1 -- 172.21.15.57:0/3566193128 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f20009420 con 0x7f1f2c1537f0 2023-10-30T14:29:46.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f2bfff640 1 --2- 172.21.15.57:0/3566193128 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f2c1537f0 0x7f1f2c1591e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f1f2400c770 tx=0x7f1f2400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:46.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f13fff640 1 -- 172.21.15.57:0/3566193128 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f24003b80 con 0x7f1f2c1537f0 2023-10-30T14:29:46.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f13fff640 1 -- 172.21.15.57:0/3566193128 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f24003ce0 con 0x7f1f2c1537f0 2023-10-30T14:29:46.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f321d3640 1 -- 172.21.15.57:0/3566193128 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2c0c9fc0 con 0x7f1f2c1537f0 2023-10-30T14:29:46.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f13fff640 1 -- 172.21.15.57:0/3566193128 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2400aa30 con 0x7f1f2c1537f0 2023-10-30T14:29:46.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f321d3640 1 -- 172.21.15.57:0/3566193128 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2c0c5210 con 0x7f1f2c1537f0 2023-10-30T14:29:46.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f321d3640 1 -- 172.21.15.57:0/3566193128 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f2c0c5050 con 0x7f1f2c1537f0 2023-10-30T14:29:46.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.533+0000 7f1f13fff640 1 -- 172.21.15.57:0/3566193128 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1f2400ab90 con 0x7f1f2c1537f0 2023-10-30T14:29:46.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.537+0000 7f1f13fff640 1 --2- 172.21.15.57:0/3566193128 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1f180664a0 0x7f1f18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:46.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.537+0000 7f1f13fff640 1 -- 172.21.15.57:0/3566193128 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f1f24083ec0 con 0x7f1f2c1537f0 2023-10-30T14:29:46.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.537+0000 7f1f2b7fe640 1 --2- 172.21.15.57:0/3566193128 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1f180664a0 0x7f1f18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:46.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.537+0000 7f1f2b7fe640 1 --2- 172.21.15.57:0/3566193128 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1f180664a0 0x7f1f18068960 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0011f0 tx=0x7f1f1c007d40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:46.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.541+0000 7f1f13fff640 1 -- 172.21.15.57:0/3566193128 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1f24051060 con 0x7f1f2c1537f0 2023-10-30T14:29:46.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.729+0000 7f1f321d3640 1 -- 172.21.15.57:0/3566193128 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f2c0c5400 con 0x7f1f2c1537f0 2023-10-30T14:29:46.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.729+0000 7f1f13fff640 1 -- 172.21.15.57:0/3566193128 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5733 (secure 0 0 0) 0x7f1f24054680 con 0x7f1f2c1537f0 2023-10-30T14:29:46.730 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:46.730 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:42.182782+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:active","state_seq":109,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:46.730 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 97 2023-10-30T14:29:46.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.729+0000 7f1f321d3640 1 -- 172.21.15.57:0/3566193128 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1f180664a0 msgr2=0x7f1f18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:46.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.729+0000 7f1f321d3640 1 --2- 172.21.15.57:0/3566193128 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1f180664a0 0x7f1f18068960 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0011f0 tx=0x7f1f1c007d40 comp rx=0 tx=0).stop 2023-10-30T14:29:46.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.729+0000 7f1f321d3640 1 -- 172.21.15.57:0/3566193128 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f2c1537f0 msgr2=0x7f1f2c1591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:46.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.729+0000 7f1f321d3640 1 --2- 172.21.15.57:0/3566193128 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f2c1537f0 0x7f1f2c1591e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f1f2400c770 tx=0x7f1f2400cc30 comp rx=0 tx=0).stop 2023-10-30T14:29:46.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.729+0000 7f1f321d3640 1 -- 172.21.15.57:0/3566193128 shutdown_connections 2023-10-30T14:29:46.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.729+0000 7f1f321d3640 1 --2- 172.21.15.57:0/3566193128 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1f180664a0 0x7f1f18068960 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:46.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.729+0000 7f1f321d3640 1 --2- 172.21.15.57:0/3566193128 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f2c1537f0 0x7f1f2c1591e0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:46.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.729+0000 7f1f321d3640 1 --2- 172.21.15.57:0/3566193128 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f2c0701a0 0x7f1f2c158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:46.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.729+0000 7f1f321d3640 1 --2- 172.21.15.57:0/3566193128 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f2c06b2f0 0x7f1f2c14c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:46.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.729+0000 7f1f321d3640 1 -- 172.21.15.57:0/3566193128 >> 172.21.15.57:0/3566193128 conn(0x7f1f2c071fb0 msgr2=0x7f1f2c0cb830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:46.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.729+0000 7f1f321d3640 1 -- 172.21.15.57:0/3566193128 shutdown_connections 2023-10-30T14:29:46.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:46.729+0000 7f1f321d3640 1 -- 172.21.15.57:0/3566193128 wait complete. 2023-10-30T14:29:46.751 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-30T14:29:48.752 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:48.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.905+0000 7fc88a0ea640 1 Processor -- start 2023-10-30T14:29:48.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.905+0000 7fc88a0ea640 1 -- start start 2023-10-30T14:29:48.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.905+0000 7fc88a0ea640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc8840cb7d0 0x7fc8840c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:48.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.905+0000 7fc88a0ea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc8840c3e40 0x7fc884070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:48.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.905+0000 7fc88a0ea640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8840c2310 con 0x7fc884150830 2023-10-30T14:29:48.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.905+0000 7fc88a0ea640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc88406bc10 con 0x7fc8840cb7d0 2023-10-30T14:29:48.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.905+0000 7fc88a0ea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc884156710 con 0x7fc8840c3e40 2023-10-30T14:29:48.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.905+0000 7fc882ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc8840cb7d0 0x7fc8840c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:48.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.905+0000 7fc8837fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc884150830 0x7fc88406b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:54720/0 (socket says 172.21.15.57:54720) 2023-10-30T14:29:48.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.905+0000 7fc8837fe640 1 -- 172.21.15.57:0/1067868749 learned_addr learned my addr 172.21.15.57:0/1067868749 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:29:48.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.905+0000 7fc883fff640 1 --2- 172.21.15.57:0/1067868749 >> v2:172.21.15.88:3300/0 conn(0x7fc8840c3e40 0x7fc884070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:48.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.905+0000 7fc882ffd640 1 -- 172.21.15.57:0/1067868749 >> v2:172.21.15.88:3300/0 conn(0x7fc8840c3e40 msgr2=0x7fc884070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:48.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.905+0000 7fc882ffd640 1 --2- 172.21.15.57:0/1067868749 >> v2:172.21.15.88:3300/0 conn(0x7fc8840c3e40 0x7fc884070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:48.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.905+0000 7fc882ffd640 1 -- 172.21.15.57:0/1067868749 >> v1:172.21.15.57:6789/0 conn(0x7fc884150830 legacy=0x7fc88406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:48.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.909+0000 7fc882ffd640 1 -- 172.21.15.57:0/1067868749 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8840c4360 con 0x7fc8840cb7d0 2023-10-30T14:29:48.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.905+0000 7fc8827fc640 1 -- 172.21.15.57:0/1067868749 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3843792226 0 0) 0x7fc8840c2310 con 0x7fc884150830 2023-10-30T14:29:48.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.909+0000 7fc882ffd640 1 --2- 172.21.15.57:0/1067868749 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8840cb7d0 0x7fc8840c3900 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fc884066e10 tx=0x7fc87801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a9efc0accef3b9c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:48.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.909+0000 7fc8827fc640 1 -- 172.21.15.57:0/1067868749 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8780a8950 con 0x7fc8840cb7d0 2023-10-30T14:29:48.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.909+0000 7fc8827fc640 1 -- 172.21.15.57:0/1067868749 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8780a8ab0 con 0x7fc8840cb7d0 2023-10-30T14:29:48.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.909+0000 7fc8827fc640 1 -- 172.21.15.57:0/1067868749 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8780a8e40 con 0x7fc8840cb7d0 2023-10-30T14:29:48.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.909+0000 7fc88a0ea640 1 -- 172.21.15.57:0/1067868749 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8840cb7d0 msgr2=0x7fc8840c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:48.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.909+0000 7fc88a0ea640 1 --2- 172.21.15.57:0/1067868749 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8840cb7d0 0x7fc8840c3900 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fc884066e10 tx=0x7fc87801ec50 comp rx=0 tx=0).stop 2023-10-30T14:29:48.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.909+0000 7fc88a0ea640 1 -- 172.21.15.57:0/1067868749 shutdown_connections 2023-10-30T14:29:48.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.909+0000 7fc88a0ea640 1 --2- 172.21.15.57:0/1067868749 >> v2:172.21.15.88:3300/0 conn(0x7fc8840c3e40 0x7fc884070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:48.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.909+0000 7fc88a0ea640 1 --2- 172.21.15.57:0/1067868749 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8840cb7d0 0x7fc8840c3900 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:48.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.909+0000 7fc88a0ea640 1 -- 172.21.15.57:0/1067868749 >> 172.21.15.57:0/1067868749 conn(0x7fc8840cac70 msgr2=0x7fc8840cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:48.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.909+0000 7fc88a0ea640 1 -- 172.21.15.57:0/1067868749 shutdown_connections 2023-10-30T14:29:48.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.909+0000 7fc88a0ea640 1 -- 172.21.15.57:0/1067868749 wait complete. 2023-10-30T14:29:48.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc88a0ea640 1 Processor -- start 2023-10-30T14:29:48.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc88a0ea640 1 -- start start 2023-10-30T14:29:48.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc88a0ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8840c3e40 0x7fc8841ef6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:48.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc88a0ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8840cb7d0 0x7fc8841f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:48.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc88a0ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc884150830 0x7fc88414a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:48.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc88a0ea640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8840c8fe0 con 0x7fc8840cb7d0 2023-10-30T14:29:48.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc88a0ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8841ef230 con 0x7fc884150830 2023-10-30T14:29:48.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc88a0ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88414bab0 con 0x7fc8840c3e40 2023-10-30T14:29:48.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc882ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8840cb7d0 0x7fc8841f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:48.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc882ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8840cb7d0 0x7fc8841f2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47684/0 (socket says 172.21.15.57:47684) 2023-10-30T14:29:48.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc882ffd640 1 -- 172.21.15.57:0/3519599917 learned_addr learned my addr 172.21.15.57:0/3519599917 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:48.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc883fff640 1 --2- 172.21.15.57:0/3519599917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc884150830 0x7fc88414a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:48.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc8837fe640 1 --2- 172.21.15.57:0/3519599917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8840c3e40 0x7fc8841ef6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:48.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc882ffd640 1 -- 172.21.15.57:0/3519599917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8840c3e40 msgr2=0x7fc8841ef6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:48.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc882ffd640 1 --2- 172.21.15.57:0/3519599917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8840c3e40 0x7fc8841ef6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:48.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc882ffd640 1 -- 172.21.15.57:0/3519599917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc884150830 msgr2=0x7fc88414a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:48.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc882ffd640 1 --2- 172.21.15.57:0/3519599917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc884150830 0x7fc88414a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:48.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc882ffd640 1 -- 172.21.15.57:0/3519599917 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc878009420 con 0x7fc8840cb7d0 2023-10-30T14:29:48.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc883fff640 1 --2- 172.21.15.57:0/3519599917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc884150830 0x7fc88414a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:48.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc882ffd640 1 --2- 172.21.15.57:0/3519599917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8840cb7d0 0x7fc8841f2040 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fc87801b860 tx=0x7fc87801d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:48.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc86bfff640 1 -- 172.21.15.57:0/3519599917 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8780a8950 con 0x7fc8840cb7d0 2023-10-30T14:29:48.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc86bfff640 1 -- 172.21.15.57:0/3519599917 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc878006e20 con 0x7fc8840cb7d0 2023-10-30T14:29:48.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc86bfff640 1 -- 172.21.15.57:0/3519599917 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc878004230 con 0x7fc8840cb7d0 2023-10-30T14:29:48.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.913+0000 7fc88a0ea640 1 -- 172.21.15.57:0/3519599917 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc88414bd70 con 0x7fc8840cb7d0 2023-10-30T14:29:48.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.917+0000 7fc88a0ea640 1 -- 172.21.15.57:0/3519599917 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8841eeec0 con 0x7fc8840cb7d0 2023-10-30T14:29:48.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.917+0000 7fc88a0ea640 1 -- 172.21.15.57:0/3519599917 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc884156f50 con 0x7fc8840cb7d0 2023-10-30T14:29:48.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.917+0000 7fc86bfff640 1 -- 172.21.15.57:0/3519599917 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc878004390 con 0x7fc8840cb7d0 2023-10-30T14:29:48.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.917+0000 7fc86bfff640 1 --2- 172.21.15.57:0/3519599917 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8700664a0 0x7fc870068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:48.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.917+0000 7fc86bfff640 1 -- 172.21.15.57:0/3519599917 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7fc878121b90 con 0x7fc8840cb7d0 2023-10-30T14:29:48.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.917+0000 7fc8837fe640 1 --2- 172.21.15.57:0/3519599917 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8700664a0 0x7fc870068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:48.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.921+0000 7fc8837fe640 1 --2- 172.21.15.57:0/3519599917 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8700664a0 0x7fc870068960 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc874000f70 tx=0x7fc874009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:48.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:48.921+0000 7fc86bfff640 1 -- 172.21.15.57:0/3519599917 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc884156f50 con 0x7fc8840cb7d0 2023-10-30T14:29:49.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:49.125+0000 7fc88a0ea640 1 -- 172.21.15.57:0/3519599917 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc884156f50 con 0x7fc8840cb7d0 2023-10-30T14:29:49.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:49.125+0000 7fc86bfff640 1 -- 172.21.15.57:0/3519599917 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fc884156f50 con 0x7fc8840cb7d0 2023-10-30T14:29:49.127 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:49.128 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6414,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:42.182782+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:active","state_seq":109,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:49.128 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 98 2023-10-30T14:29:49.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:49.125+0000 7fc88a0ea640 1 -- 172.21.15.57:0/3519599917 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8700664a0 msgr2=0x7fc870068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:49.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:49.125+0000 7fc88a0ea640 1 --2- 172.21.15.57:0/3519599917 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8700664a0 0x7fc870068960 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc874000f70 tx=0x7fc874009040 comp rx=0 tx=0).stop 2023-10-30T14:29:49.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:49.125+0000 7fc88a0ea640 1 -- 172.21.15.57:0/3519599917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8840cb7d0 msgr2=0x7fc8841f2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:49.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:49.125+0000 7fc88a0ea640 1 --2- 172.21.15.57:0/3519599917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8840cb7d0 0x7fc8841f2040 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fc87801b860 tx=0x7fc87801d490 comp rx=0 tx=0).stop 2023-10-30T14:29:49.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:49.129+0000 7fc88a0ea640 1 -- 172.21.15.57:0/3519599917 shutdown_connections 2023-10-30T14:29:49.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:49.129+0000 7fc88a0ea640 1 --2- 172.21.15.57:0/3519599917 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8700664a0 0x7fc870068960 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:49.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:49.129+0000 7fc88a0ea640 1 --2- 172.21.15.57:0/3519599917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc884150830 0x7fc88414a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:49.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:49.129+0000 7fc88a0ea640 1 --2- 172.21.15.57:0/3519599917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8840cb7d0 0x7fc8841f2040 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:49.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:49.129+0000 7fc88a0ea640 1 --2- 172.21.15.57:0/3519599917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8840c3e40 0x7fc8841ef6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:49.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:49.129+0000 7fc88a0ea640 1 -- 172.21.15.57:0/3519599917 >> 172.21.15.57:0/3519599917 conn(0x7fc8840cac70 msgr2=0x7fc884070520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:49.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:49.129+0000 7fc88a0ea640 1 -- 172.21.15.57:0/3519599917 shutdown_connections 2023-10-30T14:29:49.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:49.129+0000 7fc88a0ea640 1 -- 172.21.15.57:0/3519599917 wait complete. 2023-10-30T14:29:49.148 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T14:29:49.149 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before thrashing 2023-10-30T14:29:49.867 INFO:tasks.ceph.mds.f.smithi088.stderr:2023-10-30T14:29:49.868+0000 7fcf56163640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T14:29:49.868 INFO:tasks.ceph.mds.f.smithi088.stderr:2023-10-30T14:29:49.868+0000 7fcf56163640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:29:49.868 INFO:tasks.ceph.mds.f.smithi088.stderr:2023-10-30T14:29:49.868+0000 7fcf56163640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T14:29:53.149 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:53.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.301+0000 7f8dc791d640 1 Processor -- start 2023-10-30T14:29:53.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.301+0000 7f8dc791d640 1 -- start start 2023-10-30T14:29:53.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.305+0000 7f8dc791d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8dc0070520 0x7f8dc0070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:53.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.305+0000 7f8dc791d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc00c2310 con 0x7f8dc0150830 2023-10-30T14:29:53.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.305+0000 7f8dc791d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc0156710 con 0x7f8dc00cb7d0 2023-10-30T14:29:53.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.305+0000 7f8dc791d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc006bc10 con 0x7f8dc0070520 2023-10-30T14:29:53.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.305+0000 7f8dc56b9640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f8dc0150830 0x7f8dc006b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:54736/0 (socket says 172.21.15.57:54736) 2023-10-30T14:29:53.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.305+0000 7f8dc5eba640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8dc0070520 0x7f8dc0070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:53.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.305+0000 7f8dc56b9640 1 -- 172.21.15.57:0/3878687735 learned_addr learned my addr 172.21.15.57:0/3878687735 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:29:53.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.305+0000 7f8dc5eba640 1 -- 172.21.15.57:0/3878687735 >> v1:172.21.15.88:6789/0 conn(0x7f8dc00cb7d0 legacy=0x7f8dc00c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:53.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.305+0000 7f8dc5eba640 1 -- 172.21.15.57:0/3878687735 >> v1:172.21.15.57:6789/0 conn(0x7f8dc0150830 legacy=0x7f8dc006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:53.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.305+0000 7f8dc5eba640 1 -- 172.21.15.57:0/3878687735 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc00c4360 con 0x7f8dc0070520 2023-10-30T14:29:53.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.305+0000 7f8db7fff640 1 -- 172.21.15.57:0/3878687735 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 131440510 0 0) 0x7f8dc00c2310 con 0x7f8dc0150830 2023-10-30T14:29:53.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.309+0000 7f8dc5eba640 1 --2- 172.21.15.57:0/3878687735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc0070520 0x7f8dc0070990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f8db8009d90 tx=0x7f8db801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccf47662574202b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:53.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.309+0000 7f8db7fff640 1 -- 172.21.15.57:0/3878687735 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db80af600 con 0x7f8dc0070520 2023-10-30T14:29:53.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.309+0000 7f8db7fff640 1 -- 172.21.15.57:0/3878687735 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db80ab050 con 0x7f8dc0070520 2023-10-30T14:29:53.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.309+0000 7f8db7fff640 1 -- 172.21.15.57:0/3878687735 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db80af980 con 0x7f8dc0070520 2023-10-30T14:29:53.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.309+0000 7f8dc791d640 1 -- 172.21.15.57:0/3878687735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc0070520 msgr2=0x7f8dc0070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:53.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.309+0000 7f8dc791d640 1 --2- 172.21.15.57:0/3878687735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc0070520 0x7f8dc0070990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f8db8009d90 tx=0x7f8db801f130 comp rx=0 tx=0).stop 2023-10-30T14:29:53.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.309+0000 7f8dc791d640 1 -- 172.21.15.57:0/3878687735 shutdown_connections 2023-10-30T14:29:53.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.309+0000 7f8dc791d640 1 --2- 172.21.15.57:0/3878687735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc0070520 0x7f8dc0070990 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:53.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.309+0000 7f8dc791d640 1 -- 172.21.15.57:0/3878687735 >> 172.21.15.57:0/3878687735 conn(0x7f8dc00cac70 msgr2=0x7f8dc00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:53.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.309+0000 7f8dc791d640 1 -- 172.21.15.57:0/3878687735 shutdown_connections 2023-10-30T14:29:53.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.309+0000 7f8dc791d640 1 -- 172.21.15.57:0/3878687735 wait complete. 2023-10-30T14:29:53.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.309+0000 7f8dc791d640 1 Processor -- start 2023-10-30T14:29:53.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.309+0000 7f8dc791d640 1 -- start start 2023-10-30T14:29:53.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc791d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dc0070520 0x7f8dc014d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:53.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc791d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dc00cb7d0 0x7f8dc0162bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:53.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc791d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc0150830 0x7f8dc0163130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:53.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc791d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc0162930 con 0x7f8dc0150830 2023-10-30T14:29:53.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc791d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc013cd80 con 0x7f8dc00cb7d0 2023-10-30T14:29:53.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc791d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc013be90 con 0x7f8dc0070520 2023-10-30T14:29:53.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc5eba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc0150830 0x7f8dc0163130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:53.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc5eba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc0150830 0x7f8dc0163130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47704/0 (socket says 172.21.15.57:47704) 2023-10-30T14:29:53.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc5eba640 1 -- 172.21.15.57:0/3455115444 learned_addr learned my addr 172.21.15.57:0/3455115444 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:53.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc4eb8640 1 --2- 172.21.15.57:0/3455115444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dc00cb7d0 0x7f8dc0162bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:53.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc56b9640 1 --2- 172.21.15.57:0/3455115444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dc0070520 0x7f8dc014d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:53.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc5eba640 1 -- 172.21.15.57:0/3455115444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dc0070520 msgr2=0x7f8dc014d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:53.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc5eba640 1 --2- 172.21.15.57:0/3455115444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dc0070520 0x7f8dc014d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:53.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc5eba640 1 -- 172.21.15.57:0/3455115444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dc00cb7d0 msgr2=0x7f8dc0162bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:53.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc5eba640 1 --2- 172.21.15.57:0/3455115444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dc00cb7d0 0x7f8dc0162bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:53.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc5eba640 1 -- 172.21.15.57:0/3455115444 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db8009a70 con 0x7f8dc0150830 2023-10-30T14:29:53.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc56b9640 1 --2- 172.21.15.57:0/3455115444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dc0070520 0x7f8dc014d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:53.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc5eba640 1 --2- 172.21.15.57:0/3455115444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc0150830 0x7f8dc0163130 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f8db801cd60 tx=0x7f8db801bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:53.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8db5ffb640 1 -- 172.21.15.57:0/3455115444 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db80b9610 con 0x7f8dc0150830 2023-10-30T14:29:53.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8db5ffb640 1 -- 172.21.15.57:0/3455115444 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db80b9770 con 0x7f8dc0150830 2023-10-30T14:29:53.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8db5ffb640 1 -- 172.21.15.57:0/3455115444 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db80b9b00 con 0x7f8dc0150830 2023-10-30T14:29:53.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc791d640 1 -- 172.21.15.57:0/3455115444 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc013c170 con 0x7f8dc0150830 2023-10-30T14:29:53.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc791d640 1 -- 172.21.15.57:0/3455115444 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc013c730 con 0x7f8dc0150830 2023-10-30T14:29:53.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8dc791d640 1 -- 172.21.15.57:0/3455115444 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc013cf30 con 0x7f8dc0150830 2023-10-30T14:29:53.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.313+0000 7f8db5ffb640 1 -- 172.21.15.57:0/3455115444 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8db80b6050 con 0x7f8dc0150830 2023-10-30T14:29:53.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.317+0000 7f8db5ffb640 1 --2- 172.21.15.57:0/3455115444 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8da40664a0 0x7f8da4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:53.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.317+0000 7f8db5ffb640 1 -- 172.21.15.57:0/3455115444 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f8db81228f0 con 0x7f8dc0150830 2023-10-30T14:29:53.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.317+0000 7f8dc56b9640 1 --2- 172.21.15.57:0/3455115444 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8da40664a0 0x7f8da4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:53.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.317+0000 7f8dc56b9640 1 --2- 172.21.15.57:0/3455115444 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8da40664a0 0x7f8da4068960 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc000f70 tx=0x7f8dbc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:53.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.321+0000 7f8db5ffb640 1 -- 172.21.15.57:0/3455115444 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8dc013cf30 con 0x7f8dc0150830 2023-10-30T14:29:53.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.509+0000 7f8dc791d640 1 -- 172.21.15.57:0/3455115444 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8dc013cf30 con 0x7f8dc0150830 2023-10-30T14:29:53.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.509+0000 7f8db5ffb640 1 -- 172.21.15.57:0/3455115444 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f8dc013cf30 con 0x7f8dc0150830 2023-10-30T14:29:53.511 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:53.512 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6414,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:42.182782+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:active","state_seq":109,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:53.512 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 98 2023-10-30T14:29:53.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.513+0000 7f8dc791d640 1 -- 172.21.15.57:0/3455115444 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8da40664a0 msgr2=0x7f8da4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:53.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.513+0000 7f8dc791d640 1 --2- 172.21.15.57:0/3455115444 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8da40664a0 0x7f8da4068960 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc000f70 tx=0x7f8dbc009040 comp rx=0 tx=0).stop 2023-10-30T14:29:53.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.513+0000 7f8dc791d640 1 -- 172.21.15.57:0/3455115444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc0150830 msgr2=0x7f8dc0163130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:53.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.513+0000 7f8dc791d640 1 --2- 172.21.15.57:0/3455115444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc0150830 0x7f8dc0163130 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f8db801cd60 tx=0x7f8db801bed0 comp rx=0 tx=0).stop 2023-10-30T14:29:53.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.513+0000 7f8dc791d640 1 -- 172.21.15.57:0/3455115444 shutdown_connections 2023-10-30T14:29:53.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.513+0000 7f8dc791d640 1 --2- 172.21.15.57:0/3455115444 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8da40664a0 0x7f8da4068960 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:53.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.513+0000 7f8dc791d640 1 --2- 172.21.15.57:0/3455115444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc0150830 0x7f8dc0163130 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:53.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.513+0000 7f8dc791d640 1 --2- 172.21.15.57:0/3455115444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dc00cb7d0 0x7f8dc0162bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:53.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.513+0000 7f8dc791d640 1 --2- 172.21.15.57:0/3455115444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dc0070520 0x7f8dc014d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:53.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.513+0000 7f8dc791d640 1 -- 172.21.15.57:0/3455115444 >> 172.21.15.57:0/3455115444 conn(0x7f8dc00cac70 msgr2=0x7f8dc006bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:53.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.513+0000 7f8dc791d640 1 -- 172.21.15.57:0/3455115444 shutdown_connections 2023-10-30T14:29:53.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.513+0000 7f8dc791d640 1 -- 172.21.15.57:0/3455115444 wait complete. 2023-10-30T14:29:53.533 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2023-10-30T14:29:53.533 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T14:29:53.533 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:29:53.554 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T14:29:53.554 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:53.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa27b52a640 1 Processor -- start 2023-10-30T14:29:53.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa27b52a640 1 -- start start 2023-10-30T14:29:53.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa27b52a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2740c4a10 0x7fa27415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:53.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa27b52a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa274159f20 con 0x7fa2740c42a0 2023-10-30T14:29:53.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa27b52a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa27415d620 con 0x7fa274153c80 2023-10-30T14:29:53.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa27b52a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa27415dec0 con 0x7fa2740c4a10 2023-10-30T14:29:53.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa279ac7640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa2740c42a0 0x7fa2740ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:56476/0 (socket says 172.21.15.57:56476) 2023-10-30T14:29:53.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa279ac7640 1 -- 172.21.15.57:0/4100054959 learned_addr learned my addr 172.21.15.57:0/4100054959 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:29:53.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa278ac5640 1 --2- 172.21.15.57:0/4100054959 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2740c4a10 0x7fa27415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:53.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa26bfff640 1 -- 172.21.15.57:0/4100054959 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927718542 0 0) 0x7fa274159f20 con 0x7fa2740c42a0 2023-10-30T14:29:53.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa26bfff640 1 -- 172.21.15.57:0/4100054959 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa260003530 con 0x7fa2740c42a0 2023-10-30T14:29:53.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa26bfff640 1 -- 172.21.15.57:0/4100054959 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2275916499 0 0) 0x7fa260003530 con 0x7fa2740c42a0 2023-10-30T14:29:53.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa26bfff640 1 -- 172.21.15.57:0/4100054959 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2740c4a10 msgr2=0x7fa27415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:53.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa26bfff640 1 --2- 172.21.15.57:0/4100054959 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2740c4a10 0x7fa27415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:53.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa26bfff640 1 -- 172.21.15.57:0/4100054959 >> v1:172.21.15.88:6789/0 conn(0x7fa274153c80 legacy=0x7fa27415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:53.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa26bfff640 1 -- 172.21.15.57:0/4100054959 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27415d900 con 0x7fa2740c42a0 2023-10-30T14:29:53.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa26bfff640 1 -- 172.21.15.57:0/4100054959 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fa26c001a00 con 0x7fa2740c42a0 2023-10-30T14:29:53.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa26bfff640 1 -- 172.21.15.57:0/4100054959 >> v1:172.21.15.57:6789/0 conn(0x7fa2740c42a0 legacy=0x7fa2740ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:53.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa26bfff640 1 --2- 172.21.15.57:0/4100054959 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2740c4a10 0x7fa2600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:53.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa26bfff640 1 --2- 172.21.15.57:0/4100054959 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa260003f30 0x7fa2600047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:53.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa26bfff640 1 --2- 172.21.15.57:0/4100054959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa260004d10 0x7fa260005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:53.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa26bfff640 1 -- 172.21.15.57:0/4100054959 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27415dec0 con 0x7fa260003f30 2023-10-30T14:29:53.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa26bfff640 1 -- 172.21.15.57:0/4100054959 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26000bb00 con 0x7fa260004d10 2023-10-30T14:29:53.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa26bfff640 1 -- 172.21.15.57:0/4100054959 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa26000bcc0 con 0x7fa2740c4a10 2023-10-30T14:29:53.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa278ac5640 1 --2- 172.21.15.57:0/4100054959 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa260003f30 0x7fa2600047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:53.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa2792c6640 1 --2- 172.21.15.57:0/4100054959 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2740c4a10 0x7fa2600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:53.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa279ac7640 1 --2- 172.21.15.57:0/4100054959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa260004d10 0x7fa260005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:53.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa278ac5640 1 -- 172.21.15.57:0/4100054959 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2740c4a10 msgr2=0x7fa2600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:53.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa278ac5640 1 --2- 172.21.15.57:0/4100054959 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2740c4a10 0x7fa2600039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:53.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa278ac5640 1 -- 172.21.15.57:0/4100054959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa260004d10 msgr2=0x7fa260005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:53.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa278ac5640 1 --2- 172.21.15.57:0/4100054959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa260004d10 0x7fa260005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:53.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.697+0000 7fa278ac5640 1 -- 172.21.15.57:0/4100054959 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa260003530 con 0x7fa260003f30 2023-10-30T14:29:53.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.701+0000 7fa278ac5640 1 --2- 172.21.15.57:0/4100054959 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa260003f30 0x7fa2600047d0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fa264009ab0 tx=0x7fa26401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30818dadd0f3b35a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:53.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.701+0000 7fa26bfff640 1 -- 172.21.15.57:0/4100054959 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2640a5620 con 0x7fa260003f30 2023-10-30T14:29:53.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.701+0000 7fa26bfff640 1 -- 172.21.15.57:0/4100054959 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2640a5780 con 0x7fa260003f30 2023-10-30T14:29:53.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.701+0000 7fa26bfff640 1 -- 172.21.15.57:0/4100054959 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2640a5b30 con 0x7fa260003f30 2023-10-30T14:29:53.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.701+0000 7fa27b52a640 1 -- 172.21.15.57:0/4100054959 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa260003f30 msgr2=0x7fa2600047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:53.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.701+0000 7fa27b52a640 1 --2- 172.21.15.57:0/4100054959 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa260003f30 0x7fa2600047d0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fa264009ab0 tx=0x7fa26401ca30 comp rx=0 tx=0).stop 2023-10-30T14:29:53.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.701+0000 7fa279ac7640 1 -- 172.21.15.57:0/4100054959 reap_dead start 2023-10-30T14:29:53.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.701+0000 7fa27b52a640 1 -- 172.21.15.57:0/4100054959 shutdown_connections 2023-10-30T14:29:53.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.701+0000 7fa27b52a640 1 -- 172.21.15.57:0/4100054959 >> 172.21.15.57:0/4100054959 conn(0x7fa2740715c0 msgr2=0x7fa2740719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:53.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.701+0000 7fa27b52a640 1 -- 172.21.15.57:0/4100054959 shutdown_connections 2023-10-30T14:29:53.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.701+0000 7fa27b52a640 1 -- 172.21.15.57:0/4100054959 wait complete. 2023-10-30T14:29:53.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa27b52a640 1 Processor -- start 2023-10-30T14:29:53.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa27b52a640 1 -- start start 2023-10-30T14:29:53.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa27b52a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa260003f30 0x7fa274157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:53.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa27b52a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2741f2150 0x7fa274157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:53.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa27b52a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa27415c5c0 0x7fa274149c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:53.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa27b52a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa274154b50 con 0x7fa27415c5c0 2023-10-30T14:29:53.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa27b52a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa274156c00 con 0x7fa2741f2150 2023-10-30T14:29:53.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa27b52a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa274155d30 con 0x7fa260003f30 2023-10-30T14:29:53.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa279ac7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa27415c5c0 0x7fa274149c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:53.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa279ac7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa27415c5c0 0x7fa274149c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58102/0 (socket says 172.21.15.57:58102) 2023-10-30T14:29:53.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa279ac7640 1 -- 172.21.15.57:0/2197523228 learned_addr learned my addr 172.21.15.57:0/2197523228 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:53.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa278ac5640 1 --2- 172.21.15.57:0/2197523228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2741f2150 0x7fa274157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:53.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa2792c6640 1 --2- 172.21.15.57:0/2197523228 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa260003f30 0x7fa274157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:53.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa279ac7640 1 -- 172.21.15.57:0/2197523228 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa260003f30 msgr2=0x7fa274157090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:53.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa279ac7640 1 --2- 172.21.15.57:0/2197523228 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa260003f30 0x7fa274157090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:53.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa279ac7640 1 -- 172.21.15.57:0/2197523228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2741f2150 msgr2=0x7fa274157cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:53.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa279ac7640 1 --2- 172.21.15.57:0/2197523228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2741f2150 0x7fa274157cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:53.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa279ac7640 1 -- 172.21.15.57:0/2197523228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa274155ff0 con 0x7fa27415c5c0 2023-10-30T14:29:53.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa278ac5640 1 --2- 172.21.15.57:0/2197523228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2741f2150 0x7fa274157cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:53.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa279ac7640 1 --2- 172.21.15.57:0/2197523228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa27415c5c0 0x7fa274149c40 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fa2740c97d0 tx=0x7fa26c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:53.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa269ffb640 1 -- 172.21.15.57:0/2197523228 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c007ae0 con 0x7fa27415c5c0 2023-10-30T14:29:53.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa269ffb640 1 -- 172.21.15.57:0/2197523228 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa26c00edf0 con 0x7fa27415c5c0 2023-10-30T14:29:53.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa269ffb640 1 -- 172.21.15.57:0/2197523228 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c014670 con 0x7fa27415c5c0 2023-10-30T14:29:53.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.705+0000 7fa27b52a640 1 -- 172.21.15.57:0/2197523228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2741562d0 con 0x7fa27415c5c0 2023-10-30T14:29:53.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.709+0000 7fa27b52a640 1 -- 172.21.15.57:0/2197523228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa274156890 con 0x7fa27415c5c0 2023-10-30T14:29:53.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.709+0000 7fa27b52a640 1 -- 172.21.15.57:0/2197523228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa274156a20 con 0x7fa27415c5c0 2023-10-30T14:29:53.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.709+0000 7fa269ffb640 1 -- 172.21.15.57:0/2197523228 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa26c0147d0 con 0x7fa27415c5c0 2023-10-30T14:29:53.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.709+0000 7fa269ffb640 1 --2- 172.21.15.57:0/2197523228 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2580664d0 0x7fa258068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:53.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.709+0000 7fa269ffb640 1 -- 172.21.15.57:0/2197523228 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7fa26c083fe0 con 0x7fa27415c5c0 2023-10-30T14:29:53.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.709+0000 7fa2792c6640 1 --2- 172.21.15.57:0/2197523228 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2580664d0 0x7fa258068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:53.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.713+0000 7fa2792c6640 1 --2- 172.21.15.57:0/2197523228 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2580664d0 0x7fa258068990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa270005490 tx=0x7fa270009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:53.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.713+0000 7fa269ffb640 1 -- 172.21.15.57:0/2197523228 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa26c050ff0 con 0x7fa27415c5c0 2023-10-30T14:29:53.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~0s 2023-10-30T14:29:53.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.901+0000 7fa27b52a640 1 -- 172.21.15.57:0/2197523228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa27404e170 con 0x7fa27415c5c0 2023-10-30T14:29:53.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.901+0000 7fa269ffb640 1 -- 172.21.15.57:0/2197523228 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fa26c054610 con 0x7fa27415c5c0 2023-10-30T14:29:53.905 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:53.905 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6414,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:42.182782+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:active","state_seq":109,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:53.905 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 98 2023-10-30T14:29:53.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.905+0000 7fa27b52a640 1 -- 172.21.15.57:0/2197523228 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2580664d0 msgr2=0x7fa258068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:53.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.905+0000 7fa27b52a640 1 --2- 172.21.15.57:0/2197523228 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2580664d0 0x7fa258068990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa270005490 tx=0x7fa270009040 comp rx=0 tx=0).stop 2023-10-30T14:29:53.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.905+0000 7fa27b52a640 1 -- 172.21.15.57:0/2197523228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa27415c5c0 msgr2=0x7fa274149c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:53.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.905+0000 7fa27b52a640 1 --2- 172.21.15.57:0/2197523228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa27415c5c0 0x7fa274149c40 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fa2740c97d0 tx=0x7fa26c00e530 comp rx=0 tx=0).stop 2023-10-30T14:29:53.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.905+0000 7fa27b52a640 1 -- 172.21.15.57:0/2197523228 shutdown_connections 2023-10-30T14:29:53.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.905+0000 7fa27b52a640 1 --2- 172.21.15.57:0/2197523228 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2580664d0 0x7fa258068990 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:53.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.905+0000 7fa27b52a640 1 --2- 172.21.15.57:0/2197523228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa27415c5c0 0x7fa274149c40 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:53.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.905+0000 7fa27b52a640 1 --2- 172.21.15.57:0/2197523228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2741f2150 0x7fa274157cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:53.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.905+0000 7fa27b52a640 1 --2- 172.21.15.57:0/2197523228 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa260003f30 0x7fa274157090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:53.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.905+0000 7fa27b52a640 1 -- 172.21.15.57:0/2197523228 >> 172.21.15.57:0/2197523228 conn(0x7fa2740715c0 msgr2=0x7fa27406bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:53.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.905+0000 7fa27b52a640 1 -- 172.21.15.57:0/2197523228 shutdown_connections 2023-10-30T14:29:53.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:53.905+0000 7fa27b52a640 1 -- 172.21.15.57:0/2197523228 wait complete. 2023-10-30T14:29:53.925 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:29:55.926 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:56.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.081+0000 7f4fde69c640 1 Processor -- start 2023-10-30T14:29:56.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.081+0000 7f4fde69c640 1 -- start start 2023-10-30T14:29:56.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.081+0000 7f4fde69c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4fd8153aa0 0x7f4fd815d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:56.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.081+0000 7f4fde69c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fd80c7b00 con 0x7f4fd806b3b0 2023-10-30T14:29:56.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.081+0000 7f4fde69c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fd80c2050 con 0x7f4fd80701e0 2023-10-30T14:29:56.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.081+0000 7f4fde69c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd8070a50 con 0x7f4fd8153aa0 2023-10-30T14:29:56.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.081+0000 7f4fdcc39640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f4fd806b3b0 0x7f4fd806b840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:56486/0 (socket says 172.21.15.57:56486) 2023-10-30T14:29:56.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.081+0000 7f4fdcc39640 1 -- 172.21.15.57:0/765347460 learned_addr learned my addr 172.21.15.57:0/765347460 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:29:56.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.081+0000 7f4fd7fff640 1 --2- 172.21.15.57:0/765347460 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4fd8153aa0 0x7f4fd815d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:56.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.081+0000 7f4fd6ffd640 1 -- 172.21.15.57:0/765347460 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709609742 0 0) 0x7f4fd80c7b00 con 0x7f4fd806b3b0 2023-10-30T14:29:56.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.081+0000 7f4fd6ffd640 1 -- 172.21.15.57:0/765347460 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fc0003530 con 0x7f4fd806b3b0 2023-10-30T14:29:56.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd6ffd640 1 -- 172.21.15.57:0/765347460 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239920091 0 0) 0x7f4fd80c2050 con 0x7f4fd80701e0 2023-10-30T14:29:56.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd6ffd640 1 -- 172.21.15.57:0/765347460 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd80c7b00 con 0x7f4fd80701e0 2023-10-30T14:29:56.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd6ffd640 1 -- 172.21.15.57:0/765347460 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3944746831 0 0) 0x7f4fc0003530 con 0x7f4fd806b3b0 2023-10-30T14:29:56.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd6ffd640 1 -- 172.21.15.57:0/765347460 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4fd8153aa0 msgr2=0x7f4fd815d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:56.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd6ffd640 1 --2- 172.21.15.57:0/765347460 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4fd8153aa0 0x7f4fd815d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:56.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd6ffd640 1 -- 172.21.15.57:0/765347460 >> v1:172.21.15.88:6789/0 conn(0x7f4fd80701e0 legacy=0x7f4fd80ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:56.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd6ffd640 1 -- 172.21.15.57:0/765347460 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd80c2330 con 0x7f4fd806b3b0 2023-10-30T14:29:56.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd6ffd640 1 -- 172.21.15.57:0/765347460 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f4fd0001a00 con 0x7f4fd806b3b0 2023-10-30T14:29:56.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd6ffd640 1 -- 172.21.15.57:0/765347460 >> v1:172.21.15.57:6789/0 conn(0x7f4fd806b3b0 legacy=0x7f4fd806b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:56.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd6ffd640 1 --2- 172.21.15.57:0/765347460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fc0003b40 0x7f4fc0003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:56.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd6ffd640 1 --2- 172.21.15.57:0/765347460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fc00047d0 0x7f4fc0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:56.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd6ffd640 1 --2- 172.21.15.57:0/765347460 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4fd8153aa0 0x7f4fc0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:56.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd6ffd640 1 -- 172.21.15.57:0/765347460 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd8070a50 con 0x7f4fc00047d0 2023-10-30T14:29:56.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd6ffd640 1 -- 172.21.15.57:0/765347460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc000ba40 con 0x7f4fc0003b40 2023-10-30T14:29:56.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd6ffd640 1 -- 172.21.15.57:0/765347460 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc000bc00 con 0x7f4fd8153aa0 2023-10-30T14:29:56.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd77fe640 1 --2- 172.21.15.57:0/765347460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fc00047d0 0x7f4fc0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:56.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd7fff640 1 --2- 172.21.15.57:0/765347460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fc0003b40 0x7f4fc0003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:56.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fdcc39640 1 --2- 172.21.15.57:0/765347460 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4fd8153aa0 0x7f4fc0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:56.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd77fe640 1 -- 172.21.15.57:0/765347460 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4fd8153aa0 msgr2=0x7f4fc0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:56.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd77fe640 1 --2- 172.21.15.57:0/765347460 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4fd8153aa0 0x7f4fc0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:56.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd77fe640 1 -- 172.21.15.57:0/765347460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fc0003b40 msgr2=0x7f4fc0003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:56.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd77fe640 1 --2- 172.21.15.57:0/765347460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fc0003b40 0x7f4fc0003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:56.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.085+0000 7f4fd77fe640 1 -- 172.21.15.57:0/765347460 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc0003530 con 0x7f4fc00047d0 2023-10-30T14:29:56.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.089+0000 7f4fd77fe640 1 --2- 172.21.15.57:0/765347460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fc00047d0 0x7f4fc0004c50 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc0018d0 tx=0x7f4fcc01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1193126a56f025f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:56.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.089+0000 7f4fd6ffd640 1 -- 172.21.15.57:0/765347460 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc0a6040 con 0x7f4fc00047d0 2023-10-30T14:29:56.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.089+0000 7f4fd6ffd640 1 -- 172.21.15.57:0/765347460 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fcc0052f0 con 0x7f4fc00047d0 2023-10-30T14:29:56.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.089+0000 7f4fd6ffd640 1 -- 172.21.15.57:0/765347460 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc003600 con 0x7f4fc00047d0 2023-10-30T14:29:56.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.089+0000 7f4fde69c640 1 -- 172.21.15.57:0/765347460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fc00047d0 msgr2=0x7f4fc0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:56.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.089+0000 7f4fde69c640 1 --2- 172.21.15.57:0/765347460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fc00047d0 0x7f4fc0004c50 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc0018d0 tx=0x7f4fcc01c560 comp rx=0 tx=0).stop 2023-10-30T14:29:56.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.089+0000 7f4fdcc39640 1 -- 172.21.15.57:0/765347460 reap_dead start 2023-10-30T14:29:56.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.089+0000 7f4fde69c640 1 -- 172.21.15.57:0/765347460 shutdown_connections 2023-10-30T14:29:56.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.089+0000 7f4fde69c640 1 -- 172.21.15.57:0/765347460 >> 172.21.15.57:0/765347460 conn(0x7f4fd8071fb0 msgr2=0x7f4fd80723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:56.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.089+0000 7f4fde69c640 1 -- 172.21.15.57:0/765347460 shutdown_connections 2023-10-30T14:29:56.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.089+0000 7f4fde69c640 1 -- 172.21.15.57:0/765347460 wait complete. 2023-10-30T14:29:56.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.089+0000 7f4fde69c640 1 Processor -- start 2023-10-30T14:29:56.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.089+0000 7f4fde69c640 1 -- start start 2023-10-30T14:29:56.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.089+0000 7f4fde69c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fc00047d0 0x7f4fd814f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:56.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.089+0000 7f4fde69c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fd8154c10 0x7f4fd81f6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:56.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.089+0000 7f4fde69c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4fd8155d20 0x7f4fd81f6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:56.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.089+0000 7f4fde69c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd815a6c0 con 0x7f4fd8154c10 2023-10-30T14:29:56.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.089+0000 7f4fde69c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd814a210 con 0x7f4fc00047d0 2023-10-30T14:29:56.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.089+0000 7f4fde69c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd81545c0 con 0x7f4fd8155d20 2023-10-30T14:29:56.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.093+0000 7f4fd77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fd8154c10 0x7f4fd81f6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:56.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.093+0000 7f4fd77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fd8154c10 0x7f4fd81f6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58112/0 (socket says 172.21.15.57:58112) 2023-10-30T14:29:56.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.093+0000 7f4fd77fe640 1 -- 172.21.15.57:0/2394194940 learned_addr learned my addr 172.21.15.57:0/2394194940 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:56.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.093+0000 7f4fd7fff640 1 --2- 172.21.15.57:0/2394194940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fc00047d0 0x7f4fd814f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:56.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.093+0000 7f4fdcc39640 1 --2- 172.21.15.57:0/2394194940 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4fd8155d20 0x7f4fd81f6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:56.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.093+0000 7f4fd77fe640 1 -- 172.21.15.57:0/2394194940 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4fd8155d20 msgr2=0x7f4fd81f6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:56.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.093+0000 7f4fd77fe640 1 --2- 172.21.15.57:0/2394194940 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4fd8155d20 0x7f4fd81f6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:56.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.093+0000 7f4fd77fe640 1 -- 172.21.15.57:0/2394194940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fc00047d0 msgr2=0x7f4fd814f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:56.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.093+0000 7f4fd77fe640 1 --2- 172.21.15.57:0/2394194940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fc00047d0 0x7f4fd814f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:56.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.093+0000 7f4fd77fe640 1 -- 172.21.15.57:0/2394194940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd81547a0 con 0x7f4fd8154c10 2023-10-30T14:29:56.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.093+0000 7f4fdcc39640 1 --2- 172.21.15.57:0/2394194940 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4fd8155d20 0x7f4fd81f6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:56.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.093+0000 7f4fd77fe640 1 --2- 172.21.15.57:0/2394194940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fd8154c10 0x7f4fd81f6640 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc0ad040 tx=0x7f4fc0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:56.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.093+0000 7f4fd4ff9640 1 -- 172.21.15.57:0/2394194940 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc0b4470 con 0x7f4fd8154c10 2023-10-30T14:29:56.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.093+0000 7f4fd4ff9640 1 -- 172.21.15.57:0/2394194940 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fcc0b45d0 con 0x7f4fd8154c10 2023-10-30T14:29:56.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.093+0000 7f4fde69c640 1 -- 172.21.15.57:0/2394194940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fd8149f30 con 0x7f4fd8154c10 2023-10-30T14:29:56.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.093+0000 7f4fd4ff9640 1 -- 172.21.15.57:0/2394194940 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc0b4980 con 0x7f4fd8154c10 2023-10-30T14:29:56.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.093+0000 7f4fde69c640 1 -- 172.21.15.57:0/2394194940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fd815a2e0 con 0x7f4fd8154c10 2023-10-30T14:29:56.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.093+0000 7f4fde69c640 1 -- 172.21.15.57:0/2394194940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fd81f6490 con 0x7f4fd8154c10 2023-10-30T14:29:56.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.093+0000 7f4fd4ff9640 1 -- 172.21.15.57:0/2394194940 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4fcc0b1040 con 0x7f4fd8154c10 2023-10-30T14:29:56.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.097+0000 7f4fd4ff9640 1 --2- 172.21.15.57:0/2394194940 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4fc40664a0 0x7f4fc4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:56.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.097+0000 7f4fd4ff9640 1 -- 172.21.15.57:0/2394194940 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f4fcc11e310 con 0x7f4fd8154c10 2023-10-30T14:29:56.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.097+0000 7f4fd7fff640 1 --2- 172.21.15.57:0/2394194940 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4fc40664a0 0x7f4fc4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:56.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.097+0000 7f4fd7fff640 1 --2- 172.21.15.57:0/2394194940 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4fc40664a0 0x7f4fc4068960 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8009a80 tx=0x7f4fc8009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:56.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.101+0000 7f4fd4ff9640 1 -- 172.21.15.57:0/2394194940 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4fcc0eb400 con 0x7f4fd8154c10 2023-10-30T14:29:56.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.297+0000 7f4fde69c640 1 -- 172.21.15.57:0/2394194940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fd804e170 con 0x7f4fd8154c10 2023-10-30T14:29:56.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.297+0000 7f4fd4ff9640 1 -- 172.21.15.57:0/2394194940 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f4fcc0eea20 con 0x7f4fd8154c10 2023-10-30T14:29:56.298 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:56.299 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6414,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:42.182782+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:active","state_seq":109,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:56.299 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 98 2023-10-30T14:29:56.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.297+0000 7f4fde69c640 1 -- 172.21.15.57:0/2394194940 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4fc40664a0 msgr2=0x7f4fc4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:56.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.297+0000 7f4fde69c640 1 --2- 172.21.15.57:0/2394194940 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4fc40664a0 0x7f4fc4068960 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8009a80 tx=0x7f4fc8009190 comp rx=0 tx=0).stop 2023-10-30T14:29:56.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.297+0000 7f4fde69c640 1 -- 172.21.15.57:0/2394194940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fd8154c10 msgr2=0x7f4fd81f6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:56.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.297+0000 7f4fde69c640 1 --2- 172.21.15.57:0/2394194940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fd8154c10 0x7f4fd81f6640 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc0ad040 tx=0x7f4fc0005800 comp rx=0 tx=0).stop 2023-10-30T14:29:56.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.297+0000 7f4fde69c640 1 -- 172.21.15.57:0/2394194940 shutdown_connections 2023-10-30T14:29:56.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.297+0000 7f4fde69c640 1 --2- 172.21.15.57:0/2394194940 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4fc40664a0 0x7f4fc4068960 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:56.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.297+0000 7f4fde69c640 1 --2- 172.21.15.57:0/2394194940 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4fd8155d20 0x7f4fd81f6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:56.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.297+0000 7f4fde69c640 1 --2- 172.21.15.57:0/2394194940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fd8154c10 0x7f4fd81f6640 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:56.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.297+0000 7f4fde69c640 1 --2- 172.21.15.57:0/2394194940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fc00047d0 0x7f4fd814f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:56.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.297+0000 7f4fde69c640 1 -- 172.21.15.57:0/2394194940 >> 172.21.15.57:0/2394194940 conn(0x7f4fd8071fb0 msgr2=0x7f4fd814e330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:56.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.297+0000 7f4fde69c640 1 -- 172.21.15.57:0/2394194940 shutdown_connections 2023-10-30T14:29:56.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:56.297+0000 7f4fde69c640 1 -- 172.21.15.57:0/2394194940 wait complete. 2023-10-30T14:29:56.320 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:29:58.321 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:29:58.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.472+0000 7f194a1d3640 1 Processor -- start 2023-10-30T14:29:58.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.472+0000 7f194a1d3640 1 -- start start 2023-10-30T14:29:58.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.472+0000 7f194a1d3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f194406bd10 0x7f19440c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:58.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.472+0000 7f194a1d3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19440c5030 con 0x7f1944151840 2023-10-30T14:29:58.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.472+0000 7f194a1d3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1944157830 con 0x7f1944151c20 2023-10-30T14:29:58.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.472+0000 7f194a1d3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1944157b10 con 0x7f194406bd10 2023-10-30T14:29:58.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.472+0000 7f19437fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1944151840 0x7f1944072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:56488/0 (socket says 172.21.15.57:56488) 2023-10-30T14:29:58.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.472+0000 7f19437fe640 1 -- 172.21.15.57:0/649239201 learned_addr learned my addr 172.21.15.57:0/649239201 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:29:58.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.472+0000 7f1943fff640 1 --2- 172.21.15.57:0/649239201 >> v2:172.21.15.88:3300/0 conn(0x7f194406bd10 0x7f19440c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:58.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.472+0000 7f19427fc640 1 -- 172.21.15.57:0/649239201 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787608200 0 0) 0x7f19440c5030 con 0x7f1944151840 2023-10-30T14:29:58.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.472+0000 7f19427fc640 1 -- 172.21.15.57:0/649239201 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f192c003530 con 0x7f1944151840 2023-10-30T14:29:58.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f19427fc640 1 -- 172.21.15.57:0/649239201 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 177623988 0 0) 0x7f192c003530 con 0x7f1944151840 2023-10-30T14:29:58.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f19427fc640 1 -- 172.21.15.57:0/649239201 >> v2:172.21.15.88:3300/0 conn(0x7f194406bd10 msgr2=0x7f19440c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:58.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f19427fc640 1 --2- 172.21.15.57:0/649239201 >> v2:172.21.15.88:3300/0 conn(0x7f194406bd10 0x7f19440c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:58.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f19427fc640 1 -- 172.21.15.57:0/649239201 >> v1:172.21.15.88:6789/0 conn(0x7f1944151c20 legacy=0x7f194406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:58.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f19427fc640 1 -- 172.21.15.57:0/649239201 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1944157df0 con 0x7f1944151840 2023-10-30T14:29:58.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f19427fc640 1 -- 172.21.15.57:0/649239201 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f1934001680 con 0x7f1944151840 2023-10-30T14:29:58.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f19427fc640 1 -- 172.21.15.57:0/649239201 >> v1:172.21.15.57:6789/0 conn(0x7f1944151840 legacy=0x7f1944072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:58.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f19427fc640 1 --2- 172.21.15.57:0/649239201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f192c003980 0x7f192c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:58.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f19427fc640 1 --2- 172.21.15.57:0/649239201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f192c0047d0 0x7f192c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:58.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f19427fc640 1 --2- 172.21.15.57:0/649239201 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f192c005190 0x7f192c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:58.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f19427fc640 1 -- 172.21.15.57:0/649239201 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1944157b10 con 0x7f192c003980 2023-10-30T14:29:58.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f19427fc640 1 -- 172.21.15.57:0/649239201 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192c00c040 con 0x7f192c0047d0 2023-10-30T14:29:58.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f19427fc640 1 -- 172.21.15.57:0/649239201 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f192c00c200 con 0x7f192c005190 2023-10-30T14:29:58.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f19437fe640 1 --2- 172.21.15.57:0/649239201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f192c003980 0x7f192c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:58.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f1942ffd640 1 --2- 172.21.15.57:0/649239201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f192c0047d0 0x7f192c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:58.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f1943fff640 1 --2- 172.21.15.57:0/649239201 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f192c005190 0x7f192c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:58.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f19437fe640 1 -- 172.21.15.57:0/649239201 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f192c005190 msgr2=0x7f192c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:58.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f19437fe640 1 --2- 172.21.15.57:0/649239201 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f192c005190 0x7f192c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:58.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f19437fe640 1 -- 172.21.15.57:0/649239201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f192c0047d0 msgr2=0x7f192c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:29:58.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f19437fe640 1 --2- 172.21.15.57:0/649239201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f192c0047d0 0x7f192c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:58.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f19437fe640 1 -- 172.21.15.57:0/649239201 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f192c003530 con 0x7f192c003980 2023-10-30T14:29:58.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f1942ffd640 1 --2- 172.21.15.57:0/649239201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f192c0047d0 0x7f192c004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:29:58.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.476+0000 7f1943fff640 1 -- 172.21.15.57:0/649239201 reap_dead start 2023-10-30T14:29:58.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.480+0000 7f19437fe640 1 --2- 172.21.15.57:0/649239201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f192c003980 0x7f192c003da0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f193400c240 tx=0x7f193401f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e29de97a5022a5da server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:58.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.480+0000 7f19427fc640 1 -- 172.21.15.57:0/649239201 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1934005ae0 con 0x7f192c003980 2023-10-30T14:29:58.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.480+0000 7f19427fc640 1 -- 172.21.15.57:0/649239201 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19340ab070 con 0x7f192c003980 2023-10-30T14:29:58.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.480+0000 7f19427fc640 1 -- 172.21.15.57:0/649239201 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1934007070 con 0x7f192c003980 2023-10-30T14:29:58.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.480+0000 7f194a1d3640 1 -- 172.21.15.57:0/649239201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f192c003980 msgr2=0x7f192c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:58.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.480+0000 7f194a1d3640 1 --2- 172.21.15.57:0/649239201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f192c003980 0x7f192c003da0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f193400c240 tx=0x7f193401f330 comp rx=0 tx=0).stop 2023-10-30T14:29:58.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.480+0000 7f194a1d3640 1 -- 172.21.15.57:0/649239201 shutdown_connections 2023-10-30T14:29:58.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.480+0000 7f194a1d3640 1 --2- 172.21.15.57:0/649239201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f192c003980 0x7f192c003da0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:58.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.480+0000 7f194a1d3640 1 -- 172.21.15.57:0/649239201 >> 172.21.15.57:0/649239201 conn(0x7f19440cac70 msgr2=0x7f19440cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:58.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.480+0000 7f194a1d3640 1 -- 172.21.15.57:0/649239201 shutdown_connections 2023-10-30T14:29:58.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.480+0000 7f194a1d3640 1 -- 172.21.15.57:0/649239201 wait complete. 2023-10-30T14:29:58.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.480+0000 7f194a1d3640 1 Processor -- start 2023-10-30T14:29:58.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.480+0000 7f194a1d3640 1 -- start start 2023-10-30T14:29:58.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.480+0000 7f194a1d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f192c003980 0x7f194414d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:58.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.480+0000 7f194a1d3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19441437b0 0x7f194413b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:58.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.480+0000 7f194a1d3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19441ef390 0x7f194413bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:58.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.480+0000 7f194a1d3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19441f2c50 con 0x7f19441ef390 2023-10-30T14:29:58.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.480+0000 7f194a1d3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19441f2040 con 0x7f19441437b0 2023-10-30T14:29:58.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.480+0000 7f194a1d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19441f2320 con 0x7f192c003980 2023-10-30T14:29:58.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.484+0000 7f1943fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19441ef390 0x7f194413bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:58.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.484+0000 7f1943fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19441ef390 0x7f194413bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58134/0 (socket says 172.21.15.57:58134) 2023-10-30T14:29:58.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.484+0000 7f1943fff640 1 -- 172.21.15.57:0/126759935 learned_addr learned my addr 172.21.15.57:0/126759935 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:29:58.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.484+0000 7f19437fe640 1 --2- 172.21.15.57:0/126759935 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f192c003980 0x7f194414d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:58.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.484+0000 7f1942ffd640 1 --2- 172.21.15.57:0/126759935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19441437b0 0x7f194413b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:58.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.484+0000 7f1943fff640 1 -- 172.21.15.57:0/126759935 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f192c003980 msgr2=0x7f194414d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:58.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.484+0000 7f1943fff640 1 --2- 172.21.15.57:0/126759935 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f192c003980 0x7f194414d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:58.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.484+0000 7f1943fff640 1 -- 172.21.15.57:0/126759935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19441437b0 msgr2=0x7f194413b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:58.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.484+0000 7f1943fff640 1 --2- 172.21.15.57:0/126759935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19441437b0 0x7f194413b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:58.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.484+0000 7f1943fff640 1 -- 172.21.15.57:0/126759935 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19441f2600 con 0x7f19441ef390 2023-10-30T14:29:58.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.484+0000 7f1943fff640 1 --2- 172.21.15.57:0/126759935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19441ef390 0x7f194413bc80 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f1944155280 tx=0x7f193c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:58.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.484+0000 7f192bfff640 1 -- 172.21.15.57:0/126759935 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c00e070 con 0x7f19441ef390 2023-10-30T14:29:58.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.484+0000 7f192bfff640 1 -- 172.21.15.57:0/126759935 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193c011db0 con 0x7f19441ef390 2023-10-30T14:29:58.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.484+0000 7f192bfff640 1 -- 172.21.15.57:0/126759935 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c0039a0 con 0x7f19441ef390 2023-10-30T14:29:58.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.484+0000 7f194a1d3640 1 -- 172.21.15.57:0/126759935 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19441f28e0 con 0x7f19441ef390 2023-10-30T14:29:58.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.484+0000 7f194a1d3640 1 -- 172.21.15.57:0/126759935 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f194414d580 con 0x7f19441ef390 2023-10-30T14:29:58.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.484+0000 7f194a1d3640 1 -- 172.21.15.57:0/126759935 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19441f2dc0 con 0x7f19441ef390 2023-10-30T14:29:58.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.484+0000 7f192bfff640 1 -- 172.21.15.57:0/126759935 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f193c003b00 con 0x7f19441ef390 2023-10-30T14:29:58.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.488+0000 7f192bfff640 1 --2- 172.21.15.57:0/126759935 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f19300664d0 0x7f1930068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:29:58.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.488+0000 7f192bfff640 1 -- 172.21.15.57:0/126759935 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f193c084a20 con 0x7f19441ef390 2023-10-30T14:29:58.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.488+0000 7f19437fe640 1 --2- 172.21.15.57:0/126759935 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f19300664d0 0x7f1930068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:29:58.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.488+0000 7f19437fe640 1 --2- 172.21.15.57:0/126759935 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f19300664d0 0x7f1930068990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f19340af000 tx=0x7f19340aa000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:29:58.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.488+0000 7f192bfff640 1 -- 172.21.15.57:0/126759935 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f193c051aa0 con 0x7f19441ef390 2023-10-30T14:29:58.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.680+0000 7f194a1d3640 1 -- 172.21.15.57:0/126759935 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f194404e170 con 0x7f19441ef390 2023-10-30T14:29:58.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.680+0000 7f192bfff640 1 -- 172.21.15.57:0/126759935 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f193c003db0 con 0x7f19441ef390 2023-10-30T14:29:58.683 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:29:58.683 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6414,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:42.182782+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:active","state_seq":109,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:29:58.683 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 98 2023-10-30T14:29:58.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.680+0000 7f194a1d3640 1 -- 172.21.15.57:0/126759935 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f19300664d0 msgr2=0x7f1930068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:58.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.680+0000 7f194a1d3640 1 --2- 172.21.15.57:0/126759935 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f19300664d0 0x7f1930068990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f19340af000 tx=0x7f19340aa000 comp rx=0 tx=0).stop 2023-10-30T14:29:58.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.680+0000 7f194a1d3640 1 -- 172.21.15.57:0/126759935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19441ef390 msgr2=0x7f194413bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:29:58.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.680+0000 7f194a1d3640 1 --2- 172.21.15.57:0/126759935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19441ef390 0x7f194413bc80 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f1944155280 tx=0x7f193c0115a0 comp rx=0 tx=0).stop 2023-10-30T14:29:58.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.680+0000 7f194a1d3640 1 -- 172.21.15.57:0/126759935 shutdown_connections 2023-10-30T14:29:58.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.680+0000 7f194a1d3640 1 --2- 172.21.15.57:0/126759935 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f19300664d0 0x7f1930068990 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:58.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.680+0000 7f194a1d3640 1 --2- 172.21.15.57:0/126759935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19441ef390 0x7f194413bc80 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:58.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.680+0000 7f194a1d3640 1 --2- 172.21.15.57:0/126759935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19441437b0 0x7f194413b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:58.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.680+0000 7f194a1d3640 1 --2- 172.21.15.57:0/126759935 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f192c003980 0x7f194414d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:29:58.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.680+0000 7f194a1d3640 1 -- 172.21.15.57:0/126759935 >> 172.21.15.57:0/126759935 conn(0x7f19440cac70 msgr2=0x7f19441479e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:29:58.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.684+0000 7f194a1d3640 1 -- 172.21.15.57:0/126759935 shutdown_connections 2023-10-30T14:29:58.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:29:58.684+0000 7f194a1d3640 1 -- 172.21.15.57:0/126759935 wait complete. 2023-10-30T14:29:58.704 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:30:00.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f has been restored 2023-10-30T14:30:00.705 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:00.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.856+0000 7fcd6039d640 1 Processor -- start 2023-10-30T14:30:00.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.856+0000 7fcd6039d640 1 -- start start 2023-10-30T14:30:00.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.856+0000 7fcd6039d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcd58151740 0x7fcd58158040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:00.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.856+0000 7fcd6039d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcd580c7880 0x7fcd580c4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:00.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.856+0000 7fcd6039d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd58066160 con 0x7fcd58151b20 2023-10-30T14:30:00.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.856+0000 7fcd6039d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd5815b640 con 0x7fcd58151740 2023-10-30T14:30:00.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.856+0000 7fcd6039d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd58158580 con 0x7fcd580c7880 2023-10-30T14:30:00.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.860+0000 7fcd5e139640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcd58151740 0x7fcd58158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:00.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.860+0000 7fcd5d938640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fcd58151b20 0x7fcd580c28e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:56502/0 (socket says 172.21.15.57:56502) 2023-10-30T14:30:00.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.860+0000 7fcd5d938640 1 -- 172.21.15.57:0/2407412458 learned_addr learned my addr 172.21.15.57:0/2407412458 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:30:00.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.860+0000 7fcd5e93a640 1 --2- 172.21.15.57:0/2407412458 >> v2:172.21.15.88:3300/0 conn(0x7fcd580c7880 0x7fcd580c4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:00.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.860+0000 7fcd5e139640 1 -- 172.21.15.57:0/2407412458 >> v2:172.21.15.88:3300/0 conn(0x7fcd580c7880 msgr2=0x7fcd580c4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:00.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.860+0000 7fcd5e139640 1 --2- 172.21.15.57:0/2407412458 >> v2:172.21.15.88:3300/0 conn(0x7fcd580c7880 0x7fcd580c4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:00.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.860+0000 7fcd5e139640 1 -- 172.21.15.57:0/2407412458 >> v1:172.21.15.57:6789/0 conn(0x7fcd58151b20 legacy=0x7fcd580c28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:00.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.860+0000 7fcd5e139640 1 -- 172.21.15.57:0/2407412458 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5815bad0 con 0x7fcd58151740 2023-10-30T14:30:00.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.860+0000 7fcd5e93a640 1 --2- 172.21.15.57:0/2407412458 >> v2:172.21.15.88:3300/0 conn(0x7fcd580c7880 0x7fcd580c4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:30:00.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.860+0000 7fcd5d137640 1 -- 172.21.15.57:0/2407412458 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496847859 0 0) 0x7fcd58066160 con 0x7fcd58151b20 2023-10-30T14:30:00.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.860+0000 7fcd5e139640 1 --2- 172.21.15.57:0/2407412458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd58151740 0x7fcd58158040 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fcd58072be0 tx=0x7fcd5401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a4ed348f1033b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:00.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.860+0000 7fcd5d137640 1 -- 172.21.15.57:0/2407412458 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd540ab600 con 0x7fcd58151740 2023-10-30T14:30:00.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.860+0000 7fcd5d137640 1 -- 172.21.15.57:0/2407412458 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd540a7050 con 0x7fcd58151740 2023-10-30T14:30:00.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.860+0000 7fcd5d137640 1 -- 172.21.15.57:0/2407412458 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd540ab980 con 0x7fcd58151740 2023-10-30T14:30:00.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd6039d640 1 -- 172.21.15.57:0/2407412458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd58151740 msgr2=0x7fcd58158040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:00.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd6039d640 1 --2- 172.21.15.57:0/2407412458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd58151740 0x7fcd58158040 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fcd58072be0 tx=0x7fcd5401c430 comp rx=0 tx=0).stop 2023-10-30T14:30:00.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd6039d640 1 -- 172.21.15.57:0/2407412458 shutdown_connections 2023-10-30T14:30:00.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd6039d640 1 --2- 172.21.15.57:0/2407412458 >> v2:172.21.15.88:3300/0 conn(0x7fcd580c7880 0x7fcd580c4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:00.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd6039d640 1 --2- 172.21.15.57:0/2407412458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd58151740 0x7fcd58158040 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:00.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd6039d640 1 -- 172.21.15.57:0/2407412458 >> 172.21.15.57:0/2407412458 conn(0x7fcd580c7f60 msgr2=0x7fcd580c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:00.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd6039d640 1 -- 172.21.15.57:0/2407412458 shutdown_connections 2023-10-30T14:30:00.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd6039d640 1 -- 172.21.15.57:0/2407412458 wait complete. 2023-10-30T14:30:00.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd6039d640 1 Processor -- start 2023-10-30T14:30:00.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd6039d640 1 -- start start 2023-10-30T14:30:00.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd6039d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd580c7880 0x7fcd580c6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:00.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd6039d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd58151740 0x7fcd5814aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:00.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd6039d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd58151b20 0x7fcd5814b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:00.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd6039d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd581f2da0 con 0x7fcd580c7880 2023-10-30T14:30:00.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd6039d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd581f1f60 con 0x7fcd58151740 2023-10-30T14:30:00.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd6039d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd581f2220 con 0x7fcd58151b20 2023-10-30T14:30:00.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd5e139640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd580c7880 0x7fcd580c6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:00.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd5e139640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd580c7880 0x7fcd580c6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58156/0 (socket says 172.21.15.57:58156) 2023-10-30T14:30:00.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd5e139640 1 -- 172.21.15.57:0/2063743838 learned_addr learned my addr 172.21.15.57:0/2063743838 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:00.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd5e93a640 1 --2- 172.21.15.57:0/2063743838 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd58151b20 0x7fcd5814b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:00.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd5d938640 1 --2- 172.21.15.57:0/2063743838 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd58151740 0x7fcd5814aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:00.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd5e139640 1 -- 172.21.15.57:0/2063743838 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd58151b20 msgr2=0x7fcd5814b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:00.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd5e139640 1 --2- 172.21.15.57:0/2063743838 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd58151b20 0x7fcd5814b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:00.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd5e139640 1 -- 172.21.15.57:0/2063743838 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd58151740 msgr2=0x7fcd5814aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:00.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd5e139640 1 --2- 172.21.15.57:0/2063743838 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd58151740 0x7fcd5814aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:00.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd5e139640 1 -- 172.21.15.57:0/2063743838 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd54009420 con 0x7fcd580c7880 2023-10-30T14:30:00.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.864+0000 7fcd5d938640 1 --2- 172.21.15.57:0/2063743838 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd58151740 0x7fcd5814aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:30:00.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.868+0000 7fcd5e139640 1 --2- 172.21.15.57:0/2063743838 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd580c7880 0x7fcd580c6340 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fcd5401a060 tx=0x7fcd5401ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:00.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.868+0000 7fcd4effd640 1 -- 172.21.15.57:0/2063743838 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd540b37d0 con 0x7fcd580c7880 2023-10-30T14:30:00.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.868+0000 7fcd4effd640 1 -- 172.21.15.57:0/2063743838 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd540b3930 con 0x7fcd580c7880 2023-10-30T14:30:00.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.868+0000 7fcd4effd640 1 -- 172.21.15.57:0/2063743838 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd540b3cc0 con 0x7fcd580c7880 2023-10-30T14:30:00.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.868+0000 7fcd6039d640 1 -- 172.21.15.57:0/2063743838 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd581f2500 con 0x7fcd580c7880 2023-10-30T14:30:00.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.868+0000 7fcd6039d640 1 -- 172.21.15.57:0/2063743838 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd581f2ac0 con 0x7fcd580c7880 2023-10-30T14:30:00.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.868+0000 7fcd6039d640 1 -- 172.21.15.57:0/2063743838 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd5804e170 con 0x7fcd580c7880 2023-10-30T14:30:00.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.868+0000 7fcd4effd640 1 -- 172.21.15.57:0/2063743838 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcd540b2070 con 0x7fcd580c7880 2023-10-30T14:30:00.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.868+0000 7fcd4effd640 1 --2- 172.21.15.57:0/2063743838 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcd3c0664a0 0x7fcd3c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:00.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.868+0000 7fcd4effd640 1 -- 172.21.15.57:0/2063743838 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7fcd5411e570 con 0x7fcd580c7880 2023-10-30T14:30:00.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.868+0000 7fcd5d938640 1 --2- 172.21.15.57:0/2063743838 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcd3c0664a0 0x7fcd3c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:00.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.872+0000 7fcd5d938640 1 --2- 172.21.15.57:0/2063743838 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcd3c0664a0 0x7fcd3c068960 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fcd5814bb30 tx=0x7fcd48009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:00.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:00.872+0000 7fcd4effd640 1 -- 172.21.15.57:0/2063743838 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcd540eb5d0 con 0x7fcd580c7880 2023-10-30T14:30:01.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:01.064+0000 7fcd6039d640 1 -- 172.21.15.57:0/2063743838 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd580c7450 con 0x7fcd580c7880 2023-10-30T14:30:01.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:01.064+0000 7fcd4effd640 1 -- 172.21.15.57:0/2063743838 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fcd540eebf0 con 0x7fcd580c7880 2023-10-30T14:30:01.067 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:01.068 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6414,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:42.182782+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:active","state_seq":109,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:01.068 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 98 2023-10-30T14:30:01.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:01.068+0000 7fcd6039d640 1 -- 172.21.15.57:0/2063743838 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcd3c0664a0 msgr2=0x7fcd3c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:01.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:01.068+0000 7fcd6039d640 1 --2- 172.21.15.57:0/2063743838 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcd3c0664a0 0x7fcd3c068960 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fcd5814bb30 tx=0x7fcd48009040 comp rx=0 tx=0).stop 2023-10-30T14:30:01.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:01.068+0000 7fcd6039d640 1 -- 172.21.15.57:0/2063743838 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd580c7880 msgr2=0x7fcd580c6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:01.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:01.068+0000 7fcd6039d640 1 --2- 172.21.15.57:0/2063743838 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd580c7880 0x7fcd580c6340 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fcd5401a060 tx=0x7fcd5401ac70 comp rx=0 tx=0).stop 2023-10-30T14:30:01.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:01.068+0000 7fcd6039d640 1 -- 172.21.15.57:0/2063743838 shutdown_connections 2023-10-30T14:30:01.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:01.068+0000 7fcd6039d640 1 --2- 172.21.15.57:0/2063743838 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcd3c0664a0 0x7fcd3c068960 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:01.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:01.068+0000 7fcd6039d640 1 --2- 172.21.15.57:0/2063743838 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd58151b20 0x7fcd5814b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:01.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:01.068+0000 7fcd6039d640 1 --2- 172.21.15.57:0/2063743838 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd58151740 0x7fcd5814aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:01.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:01.068+0000 7fcd6039d640 1 --2- 172.21.15.57:0/2063743838 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd580c7880 0x7fcd580c6340 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:01.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:01.068+0000 7fcd6039d640 1 -- 172.21.15.57:0/2063743838 >> 172.21.15.57:0/2063743838 conn(0x7fcd580c7f60 msgr2=0x7fcd581491f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:01.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:01.068+0000 7fcd6039d640 1 -- 172.21.15.57:0/2063743838 shutdown_connections 2023-10-30T14:30:01.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:01.068+0000 7fcd6039d640 1 -- 172.21.15.57:0/2063743838 wait complete. 2023-10-30T14:30:01.088 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:30:03.090 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:03.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.240+0000 7f802d279640 1 Processor -- start 2023-10-30T14:30:03.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.240+0000 7f802d279640 1 -- start start 2023-10-30T14:30:03.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.244+0000 7f802d279640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f80280c4a10 0x7f802815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:03.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.244+0000 7f802d279640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80280c42a0 0x7f80280ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:03.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.244+0000 7f802d279640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8028159f20 con 0x7f8028153c80 2023-10-30T14:30:03.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.244+0000 7f802d279640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f802815dec0 con 0x7f80280c4a10 2023-10-30T14:30:03.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.244+0000 7f802d279640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80281599a0 con 0x7f80280c42a0 2023-10-30T14:30:03.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.244+0000 7f801ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f80280c4a10 0x7f802815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:03.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.244+0000 7f8026ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f8028153c80 0x7f802815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:56504/0 (socket says 172.21.15.57:56504) 2023-10-30T14:30:03.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.244+0000 7f8026ffd640 1 -- 172.21.15.57:0/1149598778 learned_addr learned my addr 172.21.15.57:0/1149598778 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:30:03.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.244+0000 7f80277fe640 1 --2- 172.21.15.57:0/1149598778 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80280c42a0 0x7f80280ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:03.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.244+0000 7f801ffff640 1 -- 172.21.15.57:0/1149598778 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80280c42a0 msgr2=0x7f80280ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:03.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.244+0000 7f801ffff640 1 --2- 172.21.15.57:0/1149598778 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80280c42a0 0x7f80280ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:03.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.244+0000 7f801ffff640 1 -- 172.21.15.57:0/1149598778 >> v1:172.21.15.57:6789/0 conn(0x7f8028153c80 legacy=0x7f802815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:03.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.244+0000 7f801ffff640 1 -- 172.21.15.57:0/1149598778 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802815d620 con 0x7f80280c4a10 2023-10-30T14:30:03.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.244+0000 7f80267fc640 1 -- 172.21.15.57:0/1149598778 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000838984 0 0) 0x7f8028159f20 con 0x7f8028153c80 2023-10-30T14:30:03.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.248+0000 7f801ffff640 1 --2- 172.21.15.57:0/1149598778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80280c4a10 0x7f802815cf90 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f80100097c0 tx=0x7f801001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f132ebbfc75f2f0c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:03.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.248+0000 7f80267fc640 1 -- 172.21.15.57:0/1149598778 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80100a8970 con 0x7f80280c4a10 2023-10-30T14:30:03.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.248+0000 7f80267fc640 1 -- 172.21.15.57:0/1149598778 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80100a8ad0 con 0x7f80280c4a10 2023-10-30T14:30:03.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.248+0000 7f80267fc640 1 -- 172.21.15.57:0/1149598778 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80100a8e60 con 0x7f80280c4a10 2023-10-30T14:30:03.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.248+0000 7f802d279640 1 -- 172.21.15.57:0/1149598778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80280c4a10 msgr2=0x7f802815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:03.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.248+0000 7f802d279640 1 --2- 172.21.15.57:0/1149598778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80280c4a10 0x7f802815cf90 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f80100097c0 tx=0x7f801001ec70 comp rx=0 tx=0).stop 2023-10-30T14:30:03.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.248+0000 7f802d279640 1 -- 172.21.15.57:0/1149598778 shutdown_connections 2023-10-30T14:30:03.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.248+0000 7f802d279640 1 --2- 172.21.15.57:0/1149598778 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80280c42a0 0x7f80280ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:03.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.248+0000 7f802d279640 1 --2- 172.21.15.57:0/1149598778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80280c4a10 0x7f802815cf90 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:03.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.248+0000 7f802d279640 1 -- 172.21.15.57:0/1149598778 >> 172.21.15.57:0/1149598778 conn(0x7f80280715c0 msgr2=0x7f80280719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:03.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.248+0000 7f802d279640 1 -- 172.21.15.57:0/1149598778 shutdown_connections 2023-10-30T14:30:03.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.248+0000 7f802d279640 1 -- 172.21.15.57:0/1149598778 wait complete. 2023-10-30T14:30:03.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.248+0000 7f802d279640 1 Processor -- start 2023-10-30T14:30:03.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.248+0000 7f802d279640 1 -- start start 2023-10-30T14:30:03.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f802d279640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80280c42a0 0x7f80281f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:03.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f802d279640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80280c4a10 0x7f80280c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:03.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f802d279640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8028153c80 0x7f80281ef6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:03.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f802d279640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80281ef220 con 0x7f80280c4a10 2023-10-30T14:30:03.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f802d279640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802814df00 con 0x7f80280c42a0 2023-10-30T14:30:03.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f802d279640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f802815d2a0 con 0x7f8028153c80 2023-10-30T14:30:03.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f801ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80280c4a10 0x7f80280c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:03.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f801ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80280c4a10 0x7f80280c5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58176/0 (socket says 172.21.15.57:58176) 2023-10-30T14:30:03.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f801ffff640 1 -- 172.21.15.57:0/2194580434 learned_addr learned my addr 172.21.15.57:0/2194580434 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:03.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f8026ffd640 1 --2- 172.21.15.57:0/2194580434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80280c42a0 0x7f80281f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:03.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f80277fe640 1 --2- 172.21.15.57:0/2194580434 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8028153c80 0x7f80281ef6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:03.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f801ffff640 1 -- 172.21.15.57:0/2194580434 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8028153c80 msgr2=0x7f80281ef6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:03.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f801ffff640 1 --2- 172.21.15.57:0/2194580434 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8028153c80 0x7f80281ef6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:03.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f801ffff640 1 -- 172.21.15.57:0/2194580434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80280c42a0 msgr2=0x7f80281f2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:03.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f801ffff640 1 --2- 172.21.15.57:0/2194580434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80280c42a0 0x7f80281f2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:03.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f801ffff640 1 -- 172.21.15.57:0/2194580434 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8010009420 con 0x7f80280c4a10 2023-10-30T14:30:03.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f801ffff640 1 --2- 172.21.15.57:0/2194580434 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80280c4a10 0x7f80280c5a60 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f801001c8a0 tx=0x7f801001d4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:03.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f801f7fe640 1 -- 172.21.15.57:0/2194580434 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80100a8970 con 0x7f80280c4a10 2023-10-30T14:30:03.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f801f7fe640 1 -- 172.21.15.57:0/2194580434 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8010006e20 con 0x7f80280c4a10 2023-10-30T14:30:03.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f802d279640 1 -- 172.21.15.57:0/2194580434 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f802814e1c0 con 0x7f80280c4a10 2023-10-30T14:30:03.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f801f7fe640 1 -- 172.21.15.57:0/2194580434 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010004230 con 0x7f80280c4a10 2023-10-30T14:30:03.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f802d279640 1 -- 172.21.15.57:0/2194580434 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802814e780 con 0x7f80280c4a10 2023-10-30T14:30:03.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f802d279640 1 -- 172.21.15.57:0/2194580434 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f802804e170 con 0x7f80280c4a10 2023-10-30T14:30:03.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.252+0000 7f801f7fe640 1 -- 172.21.15.57:0/2194580434 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8010004390 con 0x7f80280c4a10 2023-10-30T14:30:03.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.256+0000 7f801f7fe640 1 --2- 172.21.15.57:0/2194580434 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80080664a0 0x7f8008068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:03.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.256+0000 7f801f7fe640 1 -- 172.21.15.57:0/2194580434 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f8010121f70 con 0x7f80280c4a10 2023-10-30T14:30:03.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.256+0000 7f8026ffd640 1 --2- 172.21.15.57:0/2194580434 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80080664a0 0x7f8008068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:03.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.256+0000 7f8026ffd640 1 --2- 172.21.15.57:0/2194580434 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80080664a0 0x7f8008068960 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8018000f70 tx=0x7f8018007b60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:03.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.260+0000 7f801f7fe640 1 -- 172.21.15.57:0/2194580434 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f80100eefd0 con 0x7f80280c4a10 2023-10-30T14:30:03.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.448+0000 7f802d279640 1 -- 172.21.15.57:0/2194580434 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8028159b80 con 0x7f80280c4a10 2023-10-30T14:30:03.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.452+0000 7f801f7fe640 1 -- 172.21.15.57:0/2194580434 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f80100b5090 con 0x7f80280c4a10 2023-10-30T14:30:03.454 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:03.454 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6414,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:42.182782+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:active","state_seq":109,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:03.454 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 98 2023-10-30T14:30:03.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.452+0000 7f802d279640 1 -- 172.21.15.57:0/2194580434 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80080664a0 msgr2=0x7f8008068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:03.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.452+0000 7f802d279640 1 --2- 172.21.15.57:0/2194580434 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80080664a0 0x7f8008068960 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8018000f70 tx=0x7f8018007b60 comp rx=0 tx=0).stop 2023-10-30T14:30:03.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.452+0000 7f802d279640 1 -- 172.21.15.57:0/2194580434 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80280c4a10 msgr2=0x7f80280c5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:03.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.452+0000 7f802d279640 1 --2- 172.21.15.57:0/2194580434 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80280c4a10 0x7f80280c5a60 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f801001c8a0 tx=0x7f801001d4b0 comp rx=0 tx=0).stop 2023-10-30T14:30:03.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.452+0000 7f802d279640 1 -- 172.21.15.57:0/2194580434 shutdown_connections 2023-10-30T14:30:03.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.452+0000 7f802d279640 1 --2- 172.21.15.57:0/2194580434 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80080664a0 0x7f8008068960 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:03.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.452+0000 7f802d279640 1 --2- 172.21.15.57:0/2194580434 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8028153c80 0x7f80281ef6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:03.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.452+0000 7f802d279640 1 --2- 172.21.15.57:0/2194580434 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80280c4a10 0x7f80280c5a60 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:03.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.452+0000 7f802d279640 1 --2- 172.21.15.57:0/2194580434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80280c42a0 0x7f80281f2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:03.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.452+0000 7f802d279640 1 -- 172.21.15.57:0/2194580434 >> 172.21.15.57:0/2194580434 conn(0x7f80280715c0 msgr2=0x7f8028157810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:03.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.452+0000 7f802d279640 1 -- 172.21.15.57:0/2194580434 shutdown_connections 2023-10-30T14:30:03.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:03.452+0000 7f802d279640 1 -- 172.21.15.57:0/2194580434 wait complete. 2023-10-30T14:30:03.474 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:30:05.345 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:30:05.349 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:30:05.476 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:05.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1ec150f640 1 Processor -- start 2023-10-30T14:30:05.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1ec150f640 1 -- start start 2023-10-30T14:30:05.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1ec150f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1ebc15cf90 0x7f1ebc0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:05.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1ec150f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc159f20 con 0x7f1ebc153c80 2023-10-30T14:30:05.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1ec150f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc0c47e0 con 0x7f1ebc0c4a10 2023-10-30T14:30:05.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1ec150f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ebc15dec0 con 0x7f1ebc15cf90 2023-10-30T14:30:05.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1ebaffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1ebc153c80 0x7f1ebc15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46442/0 (socket says 172.21.15.57:46442) 2023-10-30T14:30:05.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1ebaffd640 1 -- 172.21.15.57:0/2101939252 learned_addr learned my addr 172.21.15.57:0/2101939252 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:30:05.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1ebb7fe640 1 --2- 172.21.15.57:0/2101939252 >> v2:172.21.15.88:3300/0 conn(0x7f1ebc15cf90 0x7f1ebc0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:05.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1eba7fc640 1 -- 172.21.15.57:0/2101939252 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281004188 0 0) 0x7f1ebc159f20 con 0x7f1ebc153c80 2023-10-30T14:30:05.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1eba7fc640 1 -- 172.21.15.57:0/2101939252 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea8003530 con 0x7f1ebc153c80 2023-10-30T14:30:05.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1eba7fc640 1 -- 172.21.15.57:0/2101939252 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1926365257 0 0) 0x7f1ea8003530 con 0x7f1ebc153c80 2023-10-30T14:30:05.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1eba7fc640 1 -- 172.21.15.57:0/2101939252 >> v2:172.21.15.88:3300/0 conn(0x7f1ebc15cf90 msgr2=0x7f1ebc0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:05.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1eba7fc640 1 --2- 172.21.15.57:0/2101939252 >> v2:172.21.15.88:3300/0 conn(0x7f1ebc15cf90 0x7f1ebc0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:05.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1eba7fc640 1 -- 172.21.15.57:0/2101939252 >> v1:172.21.15.88:6789/0 conn(0x7f1ebc0c4a10 legacy=0x7f1ebc0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:05.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1eba7fc640 1 -- 172.21.15.57:0/2101939252 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc15d620 con 0x7f1ebc153c80 2023-10-30T14:30:05.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1eba7fc640 1 -- 172.21.15.57:0/2101939252 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f1eac001680 con 0x7f1ebc153c80 2023-10-30T14:30:05.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1eba7fc640 1 -- 172.21.15.57:0/2101939252 >> v1:172.21.15.57:6789/0 conn(0x7f1ebc153c80 legacy=0x7f1ebc15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:05.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1eba7fc640 1 --2- 172.21.15.57:0/2101939252 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ea8003980 0x7f1ea8003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:05.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1eba7fc640 1 --2- 172.21.15.57:0/2101939252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ea80043a0 0x7f1ea8004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:05.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1eba7fc640 1 --2- 172.21.15.57:0/2101939252 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ea8005050 0x7f1ea800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:05.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1eba7fc640 1 -- 172.21.15.57:0/2101939252 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc15dec0 con 0x7f1ea8003980 2023-10-30T14:30:05.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1eba7fc640 1 -- 172.21.15.57:0/2101939252 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea800bf90 con 0x7f1ea8005050 2023-10-30T14:30:05.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1eba7fc640 1 -- 172.21.15.57:0/2101939252 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea800c150 con 0x7f1ea80043a0 2023-10-30T14:30:05.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1ebaffd640 1 --2- 172.21.15.57:0/2101939252 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ea8003980 0x7f1ea8003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:05.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.628+0000 7f1eb3fff640 1 --2- 172.21.15.57:0/2101939252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ea80043a0 0x7f1ea8004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:05.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.632+0000 7f1ebb7fe640 1 --2- 172.21.15.57:0/2101939252 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ea8005050 0x7f1ea800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:05.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.632+0000 7f1ebaffd640 1 -- 172.21.15.57:0/2101939252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ea80043a0 msgr2=0x7f1ea8004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:05.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.632+0000 7f1ebaffd640 1 --2- 172.21.15.57:0/2101939252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ea80043a0 0x7f1ea8004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:05.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.632+0000 7f1ebaffd640 1 -- 172.21.15.57:0/2101939252 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ea8005050 msgr2=0x7f1ea800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:05.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.632+0000 7f1ebaffd640 1 --2- 172.21.15.57:0/2101939252 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ea8005050 0x7f1ea800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:05.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.632+0000 7f1ebaffd640 1 -- 172.21.15.57:0/2101939252 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea8003530 con 0x7f1ea8003980 2023-10-30T14:30:05.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.632+0000 7f1ebb7fe640 1 -- 172.21.15.57:0/2101939252 reap_dead start 2023-10-30T14:30:05.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.632+0000 7f1ebaffd640 1 --2- 172.21.15.57:0/2101939252 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ea8003980 0x7f1ea8003d80 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f1eac00c550 tx=0x7f1eac01f600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95137b640fc83b5d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:05.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.632+0000 7f1eba7fc640 1 -- 172.21.15.57:0/2101939252 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac005ae0 con 0x7f1ea8003980 2023-10-30T14:30:05.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.632+0000 7f1eba7fc640 1 -- 172.21.15.57:0/2101939252 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac0ab070 con 0x7f1ea8003980 2023-10-30T14:30:05.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.632+0000 7f1eba7fc640 1 -- 172.21.15.57:0/2101939252 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac007070 con 0x7f1ea8003980 2023-10-30T14:30:05.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.632+0000 7f1ec150f640 1 -- 172.21.15.57:0/2101939252 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ea8003980 msgr2=0x7f1ea8003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:05.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.632+0000 7f1ec150f640 1 --2- 172.21.15.57:0/2101939252 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ea8003980 0x7f1ea8003d80 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f1eac00c550 tx=0x7f1eac01f600 comp rx=0 tx=0).stop 2023-10-30T14:30:05.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.632+0000 7f1ec150f640 1 -- 172.21.15.57:0/2101939252 shutdown_connections 2023-10-30T14:30:05.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.632+0000 7f1ec150f640 1 --2- 172.21.15.57:0/2101939252 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ea8003980 0x7f1ea8003d80 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:05.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.632+0000 7f1ec150f640 1 -- 172.21.15.57:0/2101939252 >> 172.21.15.57:0/2101939252 conn(0x7f1ebc0715c0 msgr2=0x7f1ebc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:05.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.632+0000 7f1ec150f640 1 -- 172.21.15.57:0/2101939252 shutdown_connections 2023-10-30T14:30:05.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.632+0000 7f1ec150f640 1 -- 172.21.15.57:0/2101939252 wait complete. 2023-10-30T14:30:05.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1ec150f640 1 Processor -- start 2023-10-30T14:30:05.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1ec150f640 1 -- start start 2023-10-30T14:30:05.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1ec150f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ea8003980 0x7f1ebc15c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:05.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1ec150f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ebc14ed90 0x7f1ebc156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:05.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1ec150f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ebc0ce1c0 0x7f1ebc157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:05.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1ec150f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc154c50 con 0x7f1ebc14ed90 2023-10-30T14:30:05.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1ec150f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc0c8ba0 con 0x7f1ea8003980 2023-10-30T14:30:05.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1ec150f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc1ef590 con 0x7f1ebc0ce1c0 2023-10-30T14:30:05.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1eb3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ebc14ed90 0x7f1ebc156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:05.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1ebaffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ea8003980 0x7f1ebc15c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:05.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1eb3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ebc14ed90 0x7f1ebc156ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54098/0 (socket says 172.21.15.57:54098) 2023-10-30T14:30:05.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1ebaffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ea8003980 0x7f1ebc15c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:52894/0 (socket says 172.21.15.57:52894) 2023-10-30T14:30:05.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1eb3fff640 1 -- 172.21.15.57:0/4169123132 learned_addr learned my addr 172.21.15.57:0/4169123132 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:05.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1ebb7fe640 1 --2- 172.21.15.57:0/4169123132 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ebc0ce1c0 0x7f1ebc157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:05.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1eb3fff640 1 -- 172.21.15.57:0/4169123132 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ebc0ce1c0 msgr2=0x7f1ebc157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:05.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1eb3fff640 1 --2- 172.21.15.57:0/4169123132 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ebc0ce1c0 0x7f1ebc157020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:05.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1eb3fff640 1 -- 172.21.15.57:0/4169123132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ea8003980 msgr2=0x7f1ebc15c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:05.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1eb3fff640 1 --2- 172.21.15.57:0/4169123132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ea8003980 0x7f1ebc15c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:05.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1eb3fff640 1 -- 172.21.15.57:0/4169123132 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc1ee9e0 con 0x7f1ebc14ed90 2023-10-30T14:30:05.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1eb3fff640 1 --2- 172.21.15.57:0/4169123132 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ebc14ed90 0x7f1ebc156ae0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f1ea400e410 tx=0x7f1ea400e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:05.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.636+0000 7f1eb37fe640 1 -- 172.21.15.57:0/4169123132 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea40052e0 con 0x7f1ebc14ed90 2023-10-30T14:30:05.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.640+0000 7f1eb37fe640 1 -- 172.21.15.57:0/4169123132 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea4005440 con 0x7f1ebc14ed90 2023-10-30T14:30:05.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.640+0000 7f1eb37fe640 1 -- 172.21.15.57:0/4169123132 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4003820 con 0x7f1ebc14ed90 2023-10-30T14:30:05.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.640+0000 7f1ec150f640 1 -- 172.21.15.57:0/4169123132 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ebc0c8d90 con 0x7f1ebc14ed90 2023-10-30T14:30:05.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.640+0000 7f1ec150f640 1 -- 172.21.15.57:0/4169123132 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ebc1eef40 con 0x7f1ebc14ed90 2023-10-30T14:30:05.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.640+0000 7f1ec150f640 1 -- 172.21.15.57:0/4169123132 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ebc04e170 con 0x7f1ebc14ed90 2023-10-30T14:30:05.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.640+0000 7f1eb37fe640 1 -- 172.21.15.57:0/4169123132 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1ea4003980 con 0x7f1ebc14ed90 2023-10-30T14:30:05.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.640+0000 7f1eb37fe640 1 --2- 172.21.15.57:0/4169123132 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1e9c0664a0 0x7f1e9c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:05.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.640+0000 7f1eb37fe640 1 -- 172.21.15.57:0/4169123132 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f1ea4083a80 con 0x7f1ebc14ed90 2023-10-30T14:30:05.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.640+0000 7f1ebaffd640 1 --2- 172.21.15.57:0/4169123132 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1e9c0664a0 0x7f1e9c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:05.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.644+0000 7f1ebaffd640 1 --2- 172.21.15.57:0/4169123132 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1e9c0664a0 0x7f1e9c068960 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1eac00c520 tx=0x7f1eac007040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:05.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.644+0000 7f1eb37fe640 1 -- 172.21.15.57:0/4169123132 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1ea4050ba0 con 0x7f1ebc14ed90 2023-10-30T14:30:05.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.832+0000 7f1ec150f640 1 -- 172.21.15.57:0/4169123132 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ebc0cad40 con 0x7f1ebc14ed90 2023-10-30T14:30:05.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.832+0000 7f1eb37fe640 1 -- 172.21.15.57:0/4169123132 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f1ea4019070 con 0x7f1ebc14ed90 2023-10-30T14:30:05.837 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:05.837 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6414,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:42.182782+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:active","state_seq":109,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:05.838 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 98 2023-10-30T14:30:05.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.836+0000 7f1ec150f640 1 -- 172.21.15.57:0/4169123132 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1e9c0664a0 msgr2=0x7f1e9c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:05.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.836+0000 7f1ec150f640 1 --2- 172.21.15.57:0/4169123132 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1e9c0664a0 0x7f1e9c068960 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1eac00c520 tx=0x7f1eac007040 comp rx=0 tx=0).stop 2023-10-30T14:30:05.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.836+0000 7f1ec150f640 1 -- 172.21.15.57:0/4169123132 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ebc14ed90 msgr2=0x7f1ebc156ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:05.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.836+0000 7f1ec150f640 1 --2- 172.21.15.57:0/4169123132 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ebc14ed90 0x7f1ebc156ae0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f1ea400e410 tx=0x7f1ea400e8d0 comp rx=0 tx=0).stop 2023-10-30T14:30:05.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.836+0000 7f1ec150f640 1 -- 172.21.15.57:0/4169123132 shutdown_connections 2023-10-30T14:30:05.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.836+0000 7f1ec150f640 1 --2- 172.21.15.57:0/4169123132 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1e9c0664a0 0x7f1e9c068960 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:05.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.836+0000 7f1ec150f640 1 --2- 172.21.15.57:0/4169123132 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ebc0ce1c0 0x7f1ebc157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:05.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.836+0000 7f1ec150f640 1 --2- 172.21.15.57:0/4169123132 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ebc14ed90 0x7f1ebc156ae0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:05.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.836+0000 7f1ec150f640 1 --2- 172.21.15.57:0/4169123132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ea8003980 0x7f1ebc15c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:05.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.836+0000 7f1ec150f640 1 -- 172.21.15.57:0/4169123132 >> 172.21.15.57:0/4169123132 conn(0x7f1ebc0715c0 msgr2=0x7f1ebc0c5090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:05.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.836+0000 7f1ec150f640 1 -- 172.21.15.57:0/4169123132 shutdown_connections 2023-10-30T14:30:05.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:05.836+0000 7f1ec150f640 1 -- 172.21.15.57:0/4169123132 wait complete. 2023-10-30T14:30:05.858 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:30:07.859 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:08.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.012+0000 7f934f7c5640 1 Processor -- start 2023-10-30T14:30:08.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.012+0000 7f934f7c5640 1 -- start start 2023-10-30T14:30:08.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.012+0000 7f934f7c5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f93481537f0 0x7f93480701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:08.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.012+0000 7f934f7c5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93480c2aa0 con 0x7f934806b2f0 2023-10-30T14:30:08.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.012+0000 7f934f7c5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93480708e0 con 0x7f934815a2e0 2023-10-30T14:30:08.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.012+0000 7f934f7c5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93480c3ab0 con 0x7f93481537f0 2023-10-30T14:30:08.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.012+0000 7f934d561640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f93481537f0 0x7f93480701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:08.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.012+0000 7f934dd62640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f934806b2f0 0x7f934806b700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46446/0 (socket says 172.21.15.57:46446) 2023-10-30T14:30:08.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.012+0000 7f934dd62640 1 -- 172.21.15.57:0/3551907944 learned_addr learned my addr 172.21.15.57:0/3551907944 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:30:08.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.012+0000 7f934d561640 1 -- 172.21.15.57:0/3551907944 >> v1:172.21.15.88:6789/0 conn(0x7f934815a2e0 legacy=0x7f93480ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:08.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.012+0000 7f934d561640 1 -- 172.21.15.57:0/3551907944 >> v1:172.21.15.57:6789/0 conn(0x7f934806b2f0 legacy=0x7f934806b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:08.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.012+0000 7f933ffff640 1 -- 172.21.15.57:0/3551907944 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288296790 0 0) 0x7f93480c2aa0 con 0x7f934806b2f0 2023-10-30T14:30:08.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.012+0000 7f934d561640 1 -- 172.21.15.57:0/3551907944 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93480c1eb0 con 0x7f93481537f0 2023-10-30T14:30:08.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.016+0000 7f934d561640 1 --2- 172.21.15.57:0/3551907944 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93481537f0 0x7f93480701a0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f93440097c0 tx=0x7f934401ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7c3c9d0ef0ec8f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:08.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.016+0000 7f933ffff640 1 -- 172.21.15.57:0/3551907944 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93440a8930 con 0x7f93481537f0 2023-10-30T14:30:08.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.016+0000 7f933ffff640 1 -- 172.21.15.57:0/3551907944 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93440a8a90 con 0x7f93481537f0 2023-10-30T14:30:08.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.016+0000 7f933ffff640 1 -- 172.21.15.57:0/3551907944 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93440a8e20 con 0x7f93481537f0 2023-10-30T14:30:08.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.016+0000 7f934f7c5640 1 -- 172.21.15.57:0/3551907944 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93481537f0 msgr2=0x7f93480701a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:08.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.016+0000 7f934f7c5640 1 --2- 172.21.15.57:0/3551907944 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93481537f0 0x7f93480701a0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f93440097c0 tx=0x7f934401ec30 comp rx=0 tx=0).stop 2023-10-30T14:30:08.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.016+0000 7f934f7c5640 1 -- 172.21.15.57:0/3551907944 shutdown_connections 2023-10-30T14:30:08.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.016+0000 7f934f7c5640 1 --2- 172.21.15.57:0/3551907944 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93481537f0 0x7f93480701a0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:08.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.016+0000 7f934f7c5640 1 -- 172.21.15.57:0/3551907944 >> 172.21.15.57:0/3551907944 conn(0x7f9348071fb0 msgr2=0x7f93480723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:08.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.016+0000 7f934f7c5640 1 -- 172.21.15.57:0/3551907944 shutdown_connections 2023-10-30T14:30:08.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.016+0000 7f934f7c5640 1 -- 172.21.15.57:0/3551907944 wait complete. 2023-10-30T14:30:08.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934f7c5640 1 Processor -- start 2023-10-30T14:30:08.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934f7c5640 1 -- start start 2023-10-30T14:30:08.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934f7c5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f934806b2f0 0x7f93480c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:08.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934f7c5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93481537f0 0x7f934814e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:08.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934f7c5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f934815a2e0 0x7f93480c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:08.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934f7c5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93480ca400 con 0x7f934806b2f0 2023-10-30T14:30:08.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934f7c5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93481ef280 con 0x7f934815a2e0 2023-10-30T14:30:08.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934f7c5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93481f2510 con 0x7f93481537f0 2023-10-30T14:30:08.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934d561640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f934806b2f0 0x7f93480c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:08.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934d561640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f934806b2f0 0x7f93480c9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54120/0 (socket says 172.21.15.57:54120) 2023-10-30T14:30:08.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934d561640 1 -- 172.21.15.57:0/1153571751 learned_addr learned my addr 172.21.15.57:0/1153571751 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:08.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934dd62640 1 --2- 172.21.15.57:0/1153571751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f934815a2e0 0x7f93480c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:08.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934cd60640 1 --2- 172.21.15.57:0/1153571751 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93481537f0 0x7f934814e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:08.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934d561640 1 -- 172.21.15.57:0/1153571751 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93481537f0 msgr2=0x7f934814e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:08.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934d561640 1 --2- 172.21.15.57:0/1153571751 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93481537f0 0x7f934814e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:08.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934d561640 1 -- 172.21.15.57:0/1153571751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f934815a2e0 msgr2=0x7f93480c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:08.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934d561640 1 --2- 172.21.15.57:0/1153571751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f934815a2e0 0x7f93480c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:08.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934d561640 1 -- 172.21.15.57:0/1153571751 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9344009420 con 0x7f934806b2f0 2023-10-30T14:30:08.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934dd62640 1 --2- 172.21.15.57:0/1153571751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f934815a2e0 0x7f93480c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:30:08.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934cd60640 1 --2- 172.21.15.57:0/1153571751 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93481537f0 0x7f934814e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:30:08.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934d561640 1 --2- 172.21.15.57:0/1153571751 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f934806b2f0 0x7f93480c9880 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f93440b3040 tx=0x7f934401b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:08.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f933dffb640 1 -- 172.21.15.57:0/1153571751 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93440a8930 con 0x7f934806b2f0 2023-10-30T14:30:08.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f933dffb640 1 -- 172.21.15.57:0/1153571751 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93440b9410 con 0x7f934806b2f0 2023-10-30T14:30:08.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f933dffb640 1 -- 172.21.15.57:0/1153571751 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93440b97e0 con 0x7f934806b2f0 2023-10-30T14:30:08.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934f7c5640 1 -- 172.21.15.57:0/1153571751 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93481ef7b0 con 0x7f934806b2f0 2023-10-30T14:30:08.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934f7c5640 1 -- 172.21.15.57:0/1153571751 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93481f1ec0 con 0x7f934806b2f0 2023-10-30T14:30:08.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.020+0000 7f934f7c5640 1 -- 172.21.15.57:0/1153571751 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f934804e170 con 0x7f934806b2f0 2023-10-30T14:30:08.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.024+0000 7f933dffb640 1 -- 172.21.15.57:0/1153571751 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f93440b7040 con 0x7f934806b2f0 2023-10-30T14:30:08.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.024+0000 7f933dffb640 1 --2- 172.21.15.57:0/1153571751 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f932c0664d0 0x7f932c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:08.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.024+0000 7f933dffb640 1 -- 172.21.15.57:0/1153571751 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f9344121f70 con 0x7f934806b2f0 2023-10-30T14:30:08.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.024+0000 7f934cd60640 1 --2- 172.21.15.57:0/1153571751 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f932c0664d0 0x7f932c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:08.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.024+0000 7f934cd60640 1 --2- 172.21.15.57:0/1153571751 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f932c0664d0 0x7f932c068990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f93480ccb20 tx=0x7f9338008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:08.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.028+0000 7f933dffb640 1 -- 172.21.15.57:0/1153571751 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f93440eefe0 con 0x7f934806b2f0 2023-10-30T14:30:08.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.220+0000 7f934f7c5640 1 -- 172.21.15.57:0/1153571751 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f934804e170 con 0x7f934806b2f0 2023-10-30T14:30:08.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.220+0000 7f933dffb640 1 -- 172.21.15.57:0/1153571751 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f93440b5070 con 0x7f934806b2f0 2023-10-30T14:30:08.225 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:08.225 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6414,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:29:42.182782+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5315,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5315":{"gid":5315,"name":"f","rank":0,"incarnation":92,"state":"up:active","state_seq":109,"addr":"172.21.15.88:6839/358521496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":358521496},{"type":"v1","addr":"172.21.15.88:6839","nonce":358521496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:08.226 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 98 2023-10-30T14:30:08.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.224+0000 7f934f7c5640 1 -- 172.21.15.57:0/1153571751 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f932c0664d0 msgr2=0x7f932c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:08.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.224+0000 7f934f7c5640 1 --2- 172.21.15.57:0/1153571751 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f932c0664d0 0x7f932c068990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f93480ccb20 tx=0x7f9338008040 comp rx=0 tx=0).stop 2023-10-30T14:30:08.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.224+0000 7f934f7c5640 1 -- 172.21.15.57:0/1153571751 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f934806b2f0 msgr2=0x7f93480c9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:08.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.224+0000 7f934f7c5640 1 --2- 172.21.15.57:0/1153571751 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f934806b2f0 0x7f93480c9880 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f93440b3040 tx=0x7f934401b9d0 comp rx=0 tx=0).stop 2023-10-30T14:30:08.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.224+0000 7f934f7c5640 1 -- 172.21.15.57:0/1153571751 shutdown_connections 2023-10-30T14:30:08.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.224+0000 7f934f7c5640 1 --2- 172.21.15.57:0/1153571751 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f932c0664d0 0x7f932c068990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:08.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.224+0000 7f934f7c5640 1 --2- 172.21.15.57:0/1153571751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f934815a2e0 0x7f93480c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:08.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.224+0000 7f934f7c5640 1 --2- 172.21.15.57:0/1153571751 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93481537f0 0x7f934814e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:08.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.224+0000 7f934f7c5640 1 --2- 172.21.15.57:0/1153571751 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f934806b2f0 0x7f93480c9880 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:08.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.224+0000 7f934f7c5640 1 -- 172.21.15.57:0/1153571751 >> 172.21.15.57:0/1153571751 conn(0x7f9348071fb0 msgr2=0x7f93481513c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:08.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.224+0000 7f934f7c5640 1 -- 172.21.15.57:0/1153571751 shutdown_connections 2023-10-30T14:30:08.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:08.224+0000 7f934f7c5640 1 -- 172.21.15.57:0/1153571751 wait complete. 2023-10-30T14:30:08.246 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:30:10.247 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:10.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.400+0000 7fcd095b8640 1 Processor -- start 2023-10-30T14:30:10.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.400+0000 7fcd095b8640 1 -- start start 2023-10-30T14:30:10.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.400+0000 7fcd095b8640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcd041537f0 0x7fcd040701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:10.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.400+0000 7fcd095b8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcd0415a2e0 0x7fcd040ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:10.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.400+0000 7fcd095b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd040cd4a0 0x7fcd0406b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:10.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.400+0000 7fcd095b8640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd040c3ab0 con 0x7fcd041537f0 2023-10-30T14:30:10.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.400+0000 7fcd095b8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd040706e0 con 0x7fcd0415a2e0 2023-10-30T14:30:10.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.400+0000 7fcd095b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd040c2aa0 con 0x7fcd040cd4a0 2023-10-30T14:30:10.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.400+0000 7fcd02ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcd041537f0 0x7fcd040701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:10.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.400+0000 7fcd02ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcd041537f0 0x7fcd040701a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54130/0 (socket says 172.21.15.57:54130) 2023-10-30T14:30:10.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.400+0000 7fcd02ffd640 1 -- 172.21.15.57:0/3946100145 learned_addr learned my addr 172.21.15.57:0/3946100145 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:10.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.400+0000 7fccfbfff640 1 --2- 172.21.15.57:0/3946100145 >> v2:172.21.15.88:3300/0 conn(0x7fcd0415a2e0 0x7fcd040ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:10.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.400+0000 7fcd037fe640 1 --2- 172.21.15.57:0/3946100145 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd040cd4a0 0x7fcd0406b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:10.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.400+0000 7fcd02ffd640 1 -- 172.21.15.57:0/3946100145 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd040cd4a0 msgr2=0x7fcd0406b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:10.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.400+0000 7fcd02ffd640 1 --2- 172.21.15.57:0/3946100145 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd040cd4a0 0x7fcd0406b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.400+0000 7fcd02ffd640 1 -- 172.21.15.57:0/3946100145 >> v2:172.21.15.88:3300/0 conn(0x7fcd0415a2e0 msgr2=0x7fcd040ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:10.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.400+0000 7fcd02ffd640 1 --2- 172.21.15.57:0/3946100145 >> v2:172.21.15.88:3300/0 conn(0x7fcd0415a2e0 0x7fcd040ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.400+0000 7fcd02ffd640 1 -- 172.21.15.57:0/3946100145 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd040708e0 con 0x7fcd041537f0 2023-10-30T14:30:10.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.404+0000 7fcd02ffd640 1 --2- 172.21.15.57:0/3946100145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd041537f0 0x7fcd040701a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fcd0406de40 tx=0x7fccf401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff855c259ab7ed39 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:10.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.404+0000 7fcd027fc640 1 -- 172.21.15.57:0/3946100145 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf40a89b0 con 0x7fcd041537f0 2023-10-30T14:30:10.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.404+0000 7fcd027fc640 1 -- 172.21.15.57:0/3946100145 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf40a8b10 con 0x7fcd041537f0 2023-10-30T14:30:10.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.404+0000 7fcd027fc640 1 -- 172.21.15.57:0/3946100145 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf40a8ea0 con 0x7fcd041537f0 2023-10-30T14:30:10.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.404+0000 7fcd095b8640 1 -- 172.21.15.57:0/3946100145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd041537f0 msgr2=0x7fcd040701a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:10.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.404+0000 7fcd095b8640 1 --2- 172.21.15.57:0/3946100145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd041537f0 0x7fcd040701a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fcd0406de40 tx=0x7fccf401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.404+0000 7fcd095b8640 1 -- 172.21.15.57:0/3946100145 shutdown_connections 2023-10-30T14:30:10.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.404+0000 7fcd095b8640 1 --2- 172.21.15.57:0/3946100145 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd040cd4a0 0x7fcd0406b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.404+0000 7fcd095b8640 1 --2- 172.21.15.57:0/3946100145 >> v2:172.21.15.88:3300/0 conn(0x7fcd0415a2e0 0x7fcd040ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.404+0000 7fcd095b8640 1 --2- 172.21.15.57:0/3946100145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd041537f0 0x7fcd040701a0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.404+0000 7fcd095b8640 1 -- 172.21.15.57:0/3946100145 >> 172.21.15.57:0/3946100145 conn(0x7fcd04071fb0 msgr2=0x7fcd040723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:10.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.404+0000 7fcd095b8640 1 -- 172.21.15.57:0/3946100145 shutdown_connections 2023-10-30T14:30:10.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.404+0000 7fcd095b8640 1 -- 172.21.15.57:0/3946100145 wait complete. 2023-10-30T14:30:10.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd095b8640 1 Processor -- start 2023-10-30T14:30:10.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd095b8640 1 -- start start 2023-10-30T14:30:10.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd095b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd040cd4a0 0x7fcd04157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:10.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd095b8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd041537f0 0x7fcd0414fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:10.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd095b8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd0415a2e0 0x7fcd041559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:10.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd095b8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd041597f0 con 0x7fcd0415a2e0 2023-10-30T14:30:10.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd095b8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd04157170 con 0x7fcd041537f0 2023-10-30T14:30:10.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd095b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd04158420 con 0x7fcd040cd4a0 2023-10-30T14:30:10.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd037fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd0415a2e0 0x7fcd041559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:10.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd037fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd0415a2e0 0x7fcd041559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54142/0 (socket says 172.21.15.57:54142) 2023-10-30T14:30:10.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd037fe640 1 -- 172.21.15.57:0/2549825634 learned_addr learned my addr 172.21.15.57:0/2549825634 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:10.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fccfbfff640 1 --2- 172.21.15.57:0/2549825634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd041537f0 0x7fcd0414fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:10.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd02ffd640 1 --2- 172.21.15.57:0/2549825634 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd040cd4a0 0x7fcd04157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:10.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd037fe640 1 -- 172.21.15.57:0/2549825634 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd040cd4a0 msgr2=0x7fcd04157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:10.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd037fe640 1 --2- 172.21.15.57:0/2549825634 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd040cd4a0 0x7fcd04157ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd037fe640 1 -- 172.21.15.57:0/2549825634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd041537f0 msgr2=0x7fcd0414fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:10.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd037fe640 1 --2- 172.21.15.57:0/2549825634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd041537f0 0x7fcd0414fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd037fe640 1 -- 172.21.15.57:0/2549825634 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf40094d0 con 0x7fcd0415a2e0 2023-10-30T14:30:10.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd02ffd640 1 --2- 172.21.15.57:0/2549825634 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd040cd4a0 0x7fcd04157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:30:10.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd037fe640 1 --2- 172.21.15.57:0/2549825634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd0415a2e0 0x7fcd041559e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fccfc007d20 tx=0x7fccfc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:10.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fccfb7fe640 1 -- 172.21.15.57:0/2549825634 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc014420 con 0x7fcd0415a2e0 2023-10-30T14:30:10.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fccfb7fe640 1 -- 172.21.15.57:0/2549825634 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccfc0145a0 con 0x7fcd0415a2e0 2023-10-30T14:30:10.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd095b8640 1 -- 172.21.15.57:0/2549825634 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd04158f50 con 0x7fcd0415a2e0 2023-10-30T14:30:10.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fccfb7fe640 1 -- 172.21.15.57:0/2549825634 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc014950 con 0x7fcd0415a2e0 2023-10-30T14:30:10.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.408+0000 7fcd095b8640 1 -- 172.21.15.57:0/2549825634 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd04159510 con 0x7fcd0415a2e0 2023-10-30T14:30:10.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.412+0000 7fcd095b8640 1 -- 172.21.15.57:0/2549825634 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd0404e170 con 0x7fcd0415a2e0 2023-10-30T14:30:10.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.412+0000 7fccfb7fe640 1 -- 172.21.15.57:0/2549825634 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fccfc014ab0 con 0x7fcd0415a2e0 2023-10-30T14:30:10.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.412+0000 7fccfb7fe640 1 --2- 172.21.15.57:0/2549825634 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcce40664a0 0x7fcce4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:10.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.412+0000 7fccfb7fe640 1 -- 172.21.15.57:0/2549825634 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7fccfc084b80 con 0x7fcd0415a2e0 2023-10-30T14:30:10.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.412+0000 7fcd02ffd640 1 --2- 172.21.15.57:0/2549825634 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcce40664a0 0x7fcce4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:10.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.412+0000 7fcd02ffd640 1 --2- 172.21.15.57:0/2549825634 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcce40664a0 0x7fcce4068960 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fccf40b2000 tx=0x7fccf4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:10.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.416+0000 7fccfb7fe640 1 -- 172.21.15.57:0/2549825634 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fccfc051b00 con 0x7fcd0415a2e0 2023-10-30T14:30:10.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.608+0000 7fcd095b8640 1 -- 172.21.15.57:0/2549825634 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd0406cdf0 con 0x7fcd0415a2e0 2023-10-30T14:30:10.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.608+0000 7fccfb7fe640 1 -- 172.21.15.57:0/2549825634 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5734 (secure 0 0 0) 0x7fccfc084e60 con 0x7fcd0415a2e0 2023-10-30T14:30:10.610 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:10.611 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:30:09.871551+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:10.611 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 99 2023-10-30T14:30:10.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.608+0000 7fcd095b8640 1 -- 172.21.15.57:0/2549825634 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcce40664a0 msgr2=0x7fcce4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:10.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.608+0000 7fcd095b8640 1 --2- 172.21.15.57:0/2549825634 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcce40664a0 0x7fcce4068960 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fccf40b2000 tx=0x7fccf4008040 comp rx=0 tx=0).stop 2023-10-30T14:30:10.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.608+0000 7fcd095b8640 1 -- 172.21.15.57:0/2549825634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd0415a2e0 msgr2=0x7fcd041559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:10.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.608+0000 7fcd095b8640 1 --2- 172.21.15.57:0/2549825634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd0415a2e0 0x7fcd041559e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fccfc007d20 tx=0x7fccfc00e4b0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.608+0000 7fcd095b8640 1 -- 172.21.15.57:0/2549825634 shutdown_connections 2023-10-30T14:30:10.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.608+0000 7fcd095b8640 1 --2- 172.21.15.57:0/2549825634 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcce40664a0 0x7fcce4068960 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.608+0000 7fcd095b8640 1 --2- 172.21.15.57:0/2549825634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd0415a2e0 0x7fcd041559e0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.608+0000 7fcd095b8640 1 --2- 172.21.15.57:0/2549825634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd041537f0 0x7fcd0414fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.608+0000 7fcd095b8640 1 --2- 172.21.15.57:0/2549825634 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd040cd4a0 0x7fcd04157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.608+0000 7fcd095b8640 1 -- 172.21.15.57:0/2549825634 >> 172.21.15.57:0/2549825634 conn(0x7fcd04071fb0 msgr2=0x7fcd04151de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:10.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.608+0000 7fcd095b8640 1 -- 172.21.15.57:0/2549825634 shutdown_connections 2023-10-30T14:30:10.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.608+0000 7fcd095b8640 1 -- 172.21.15.57:0/2549825634 wait complete. 2023-10-30T14:30:10.631 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-30T14:30:10.631 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:30:10.631 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:10.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.772+0000 7f5773fd3640 1 Processor -- start 2023-10-30T14:30:10.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.772+0000 7f5773fd3640 1 -- start start 2023-10-30T14:30:10.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.772+0000 7f5773fd3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c151c20 0x7f576c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:10.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.772+0000 7f5773fd3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f576c0cb8b0 0x7f576c06b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:10.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.772+0000 7f5773fd3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576c0c5030 con 0x7f576c151840 2023-10-30T14:30:10.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.772+0000 7f5773fd3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f576c157830 con 0x7f576c0cb8b0 2023-10-30T14:30:10.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.772+0000 7f5773fd3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576c157b10 con 0x7f576c151c20 2023-10-30T14:30:10.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.772+0000 7f5772570640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f576c0cb8b0 0x7f576c06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:10.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.772+0000 7f5771d6f640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f576c151840 0x7f576c072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46454/0 (socket says 172.21.15.57:46454) 2023-10-30T14:30:10.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.772+0000 7f5771d6f640 1 -- 172.21.15.57:0/3862932580 learned_addr learned my addr 172.21.15.57:0/3862932580 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:30:10.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.772+0000 7f577156e640 1 --2- 172.21.15.57:0/3862932580 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c151c20 0x7f576c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:10.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.772+0000 7f5772570640 1 -- 172.21.15.57:0/3862932580 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c151c20 msgr2=0x7f576c0c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:10.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.772+0000 7f5772570640 1 --2- 172.21.15.57:0/3862932580 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c151c20 0x7f576c0c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.772+0000 7f5772570640 1 -- 172.21.15.57:0/3862932580 >> v1:172.21.15.57:6789/0 conn(0x7f576c151840 legacy=0x7f576c072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:10.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.772+0000 7f5772570640 1 -- 172.21.15.57:0/3862932580 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c157df0 con 0x7f576c0cb8b0 2023-10-30T14:30:10.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.772+0000 7f5770d6d640 1 -- 172.21.15.57:0/3862932580 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981747430 0 0) 0x7f576c0c5030 con 0x7f576c151840 2023-10-30T14:30:10.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.776+0000 7f5772570640 1 --2- 172.21.15.57:0/3862932580 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f576c0cb8b0 0x7f576c06b600 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f5764009d90 tx=0x7f576401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2985042f11cf549 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:10.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.776+0000 7f5770d6d640 1 -- 172.21.15.57:0/3862932580 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57640af600 con 0x7f576c0cb8b0 2023-10-30T14:30:10.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.776+0000 7f5770d6d640 1 -- 172.21.15.57:0/3862932580 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57640ab050 con 0x7f576c0cb8b0 2023-10-30T14:30:10.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.776+0000 7f5770d6d640 1 -- 172.21.15.57:0/3862932580 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57640af980 con 0x7f576c0cb8b0 2023-10-30T14:30:10.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.776+0000 7f5773fd3640 1 -- 172.21.15.57:0/3862932580 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f576c0cb8b0 msgr2=0x7f576c06b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:10.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.776+0000 7f5773fd3640 1 --2- 172.21.15.57:0/3862932580 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f576c0cb8b0 0x7f576c06b600 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f5764009d90 tx=0x7f576401f130 comp rx=0 tx=0).stop 2023-10-30T14:30:10.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.776+0000 7f5773fd3640 1 -- 172.21.15.57:0/3862932580 shutdown_connections 2023-10-30T14:30:10.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.776+0000 7f5773fd3640 1 --2- 172.21.15.57:0/3862932580 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c151c20 0x7f576c0c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.776+0000 7f5773fd3640 1 --2- 172.21.15.57:0/3862932580 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f576c0cb8b0 0x7f576c06b600 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.776+0000 7f5773fd3640 1 -- 172.21.15.57:0/3862932580 >> 172.21.15.57:0/3862932580 conn(0x7f576c0cac70 msgr2=0x7f576c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:10.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.776+0000 7f5773fd3640 1 -- 172.21.15.57:0/3862932580 shutdown_connections 2023-10-30T14:30:10.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.776+0000 7f5773fd3640 1 -- 172.21.15.57:0/3862932580 wait complete. 2023-10-30T14:30:10.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5773fd3640 1 Processor -- start 2023-10-30T14:30:10.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5773fd3640 1 -- start start 2023-10-30T14:30:10.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5773fd3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f576c0cb8b0 0x7f576c1ed880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:10.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5773fd3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f576c151840 0x7f576c148570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:10.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5773fd3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c151c20 0x7f576c1e6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:10.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5773fd3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c1547d0 con 0x7f576c0cb8b0 2023-10-30T14:30:10.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5773fd3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c154ab0 con 0x7f576c151840 2023-10-30T14:30:10.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5773fd3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576c1ecda0 con 0x7f576c151c20 2023-10-30T14:30:10.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5771d6f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f576c0cb8b0 0x7f576c1ed880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:10.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5771d6f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f576c0cb8b0 0x7f576c1ed880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54164/0 (socket says 172.21.15.57:54164) 2023-10-30T14:30:10.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5771d6f640 1 -- 172.21.15.57:0/943504922 learned_addr learned my addr 172.21.15.57:0/943504922 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:10.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f577156e640 1 --2- 172.21.15.57:0/943504922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f576c151840 0x7f576c148570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:10.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5772570640 1 --2- 172.21.15.57:0/943504922 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c151c20 0x7f576c1e6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:10.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5771d6f640 1 -- 172.21.15.57:0/943504922 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c151c20 msgr2=0x7f576c1e6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:10.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5771d6f640 1 --2- 172.21.15.57:0/943504922 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c151c20 0x7f576c1e6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5771d6f640 1 -- 172.21.15.57:0/943504922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f576c151840 msgr2=0x7f576c148570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:10.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5771d6f640 1 --2- 172.21.15.57:0/943504922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f576c151840 0x7f576c148570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5771d6f640 1 -- 172.21.15.57:0/943504922 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5764009a70 con 0x7f576c0cb8b0 2023-10-30T14:30:10.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5771d6f640 1 --2- 172.21.15.57:0/943504922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f576c0cb8b0 0x7f576c1ed880 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f57680011f0 tx=0x7f576800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:10.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f57627fc640 1 -- 172.21.15.57:0/943504922 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57680074e0 con 0x7f576c0cb8b0 2023-10-30T14:30:10.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f57627fc640 1 -- 172.21.15.57:0/943504922 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5768007640 con 0x7f576c0cb8b0 2023-10-30T14:30:10.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5773fd3640 1 -- 172.21.15.57:0/943504922 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f576c1ed080 con 0x7f576c0cb8b0 2023-10-30T14:30:10.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f57627fc640 1 -- 172.21.15.57:0/943504922 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5768004a10 con 0x7f576c0cb8b0 2023-10-30T14:30:10.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.780+0000 7f5773fd3640 1 -- 172.21.15.57:0/943504922 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f576c1e7380 con 0x7f576c0cb8b0 2023-10-30T14:30:10.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.784+0000 7f5773fd3640 1 -- 172.21.15.57:0/943504922 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f576c04e170 con 0x7f576c0cb8b0 2023-10-30T14:30:10.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.784+0000 7f57627fc640 1 -- 172.21.15.57:0/943504922 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5768004b70 con 0x7f576c0cb8b0 2023-10-30T14:30:10.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.784+0000 7f57627fc640 1 --2- 172.21.15.57:0/943504922 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f57500664a0 0x7f5750068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:10.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.784+0000 7f57627fc640 1 -- 172.21.15.57:0/943504922 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f57680835f0 con 0x7f576c0cb8b0 2023-10-30T14:30:10.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.784+0000 7f577156e640 1 --2- 172.21.15.57:0/943504922 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f57500664a0 0x7f5750068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:10.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.784+0000 7f577156e640 1 --2- 172.21.15.57:0/943504922 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f57500664a0 0x7f5750068960 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f576c0c61c0 tx=0x7f575c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:10.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.788+0000 7f57627fc640 1 -- 172.21.15.57:0/943504922 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5768050600 con 0x7f576c0cb8b0 2023-10-30T14:30:10.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.976+0000 7f5773fd3640 1 -- 172.21.15.57:0/943504922 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f576c1ed530 con 0x7f576c0cb8b0 2023-10-30T14:30:10.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.980+0000 7f57627fc640 1 -- 172.21.15.57:0/943504922 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f5768018090 con 0x7f576c0cb8b0 2023-10-30T14:30:10.981 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:10.982 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:30:09.871551+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:10.982 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 99 2023-10-30T14:30:10.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.980+0000 7f5773fd3640 1 -- 172.21.15.57:0/943504922 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f57500664a0 msgr2=0x7f5750068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:10.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.980+0000 7f5773fd3640 1 --2- 172.21.15.57:0/943504922 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f57500664a0 0x7f5750068960 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f576c0c61c0 tx=0x7f575c008040 comp rx=0 tx=0).stop 2023-10-30T14:30:10.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.980+0000 7f5773fd3640 1 -- 172.21.15.57:0/943504922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f576c0cb8b0 msgr2=0x7f576c1ed880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:10.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.980+0000 7f5773fd3640 1 --2- 172.21.15.57:0/943504922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f576c0cb8b0 0x7f576c1ed880 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f57680011f0 tx=0x7f576800c9b0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.980+0000 7f5773fd3640 1 -- 172.21.15.57:0/943504922 shutdown_connections 2023-10-30T14:30:10.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.980+0000 7f5773fd3640 1 --2- 172.21.15.57:0/943504922 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f57500664a0 0x7f5750068960 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.980+0000 7f5773fd3640 1 --2- 172.21.15.57:0/943504922 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c151c20 0x7f576c1e6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.980+0000 7f5773fd3640 1 --2- 172.21.15.57:0/943504922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f576c151840 0x7f576c148570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.980+0000 7f5773fd3640 1 --2- 172.21.15.57:0/943504922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f576c0cb8b0 0x7f576c1ed880 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:10.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.980+0000 7f5773fd3640 1 -- 172.21.15.57:0/943504922 >> 172.21.15.57:0/943504922 conn(0x7f576c0cac70 msgr2=0x7f576c0c59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:10.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.980+0000 7f5773fd3640 1 -- 172.21.15.57:0/943504922 shutdown_connections 2023-10-30T14:30:10.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:10.980+0000 7f5773fd3640 1 -- 172.21.15.57:0/943504922 wait complete. 2023-10-30T14:30:11.002 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 99, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 99, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6027": { "addr": "172.21.15.57:6836/2733323524", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2733323524, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2733323524, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6027, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 24 }, "gid_6204": { "addr": "172.21.15.57:6837/1843528309", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 1843528309, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 1843528309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6204, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_6414": { "addr": "172.21.15.57:6834/2321408149", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2321408149, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2321408149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6414, "incarnation": 99, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:30:09.871551+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6414, "mds_1": 6027, "mds_2": 6204 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:30:13.003 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:13.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.156+0000 7f39ed565640 1 Processor -- start 2023-10-30T14:30:13.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.156+0000 7f39ed565640 1 -- start start 2023-10-30T14:30:13.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.160+0000 7f39ed565640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39e81516c0 0x7f39e80c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:13.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.160+0000 7f39ed565640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f39e80c82d0 0x7f39e80ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:13.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.160+0000 7f39ed565640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e8151aa0 con 0x7f39e80cd4a0 2023-10-30T14:30:13.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.160+0000 7f39ed565640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39e80c74a0 con 0x7f39e80c82d0 2023-10-30T14:30:13.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.160+0000 7f39ed565640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e8158510 con 0x7f39e81516c0 2023-10-30T14:30:13.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.160+0000 7f39e77fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f39e80cd4a0 0x7f39e80c2500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46460/0 (socket says 172.21.15.57:46460) 2023-10-30T14:30:13.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.160+0000 7f39e77fe640 1 -- 172.21.15.57:0/3909632461 learned_addr learned my addr 172.21.15.57:0/3909632461 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:30:13.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.160+0000 7f39dffff640 1 --2- 172.21.15.57:0/3909632461 >> v2:172.21.15.88:3300/0 conn(0x7f39e80c82d0 0x7f39e80ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:13.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.160+0000 7f39e6ffd640 1 --2- 172.21.15.57:0/3909632461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39e81516c0 0x7f39e80c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:13.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.160+0000 7f39e67fc640 1 -- 172.21.15.57:0/3909632461 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995502511 0 0) 0x7f39e8151aa0 con 0x7f39e80cd4a0 2023-10-30T14:30:13.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.160+0000 7f39e67fc640 1 -- 172.21.15.57:0/3909632461 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39d4003530 con 0x7f39e80cd4a0 2023-10-30T14:30:13.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.160+0000 7f39e67fc640 1 -- 172.21.15.57:0/3909632461 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 515782983 0 0) 0x7f39d4003530 con 0x7f39e80cd4a0 2023-10-30T14:30:13.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.160+0000 7f39dffff640 1 -- 172.21.15.57:0/3909632461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39e81516c0 msgr2=0x7f39e80c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:13.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.160+0000 7f39dffff640 1 --2- 172.21.15.57:0/3909632461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39e81516c0 0x7f39e80c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:13.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.160+0000 7f39dffff640 1 -- 172.21.15.57:0/3909632461 >> v1:172.21.15.57:6789/0 conn(0x7f39e80cd4a0 legacy=0x7f39e80c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:13.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.160+0000 7f39dffff640 1 -- 172.21.15.57:0/3909632461 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e8158af0 con 0x7f39e80c82d0 2023-10-30T14:30:13.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.164+0000 7f39dffff640 1 --2- 172.21.15.57:0/3909632461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39e80c82d0 0x7f39e80ccf60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f39d00097c0 tx=0x7f39d001c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a2a8872941ec78f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:13.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.164+0000 7f39e67fc640 1 -- 172.21.15.57:0/3909632461 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e0001a00 con 0x7f39e80c82d0 2023-10-30T14:30:13.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.164+0000 7f39e67fc640 1 -- 172.21.15.57:0/3909632461 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d00a7050 con 0x7f39e80c82d0 2023-10-30T14:30:13.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.164+0000 7f39e67fc640 1 -- 172.21.15.57:0/3909632461 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d00ab620 con 0x7f39e80c82d0 2023-10-30T14:30:13.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.164+0000 7f39ed565640 1 -- 172.21.15.57:0/3909632461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39e80c82d0 msgr2=0x7f39e80ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:13.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.164+0000 7f39ed565640 1 --2- 172.21.15.57:0/3909632461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39e80c82d0 0x7f39e80ccf60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f39d00097c0 tx=0x7f39d001c3a0 comp rx=0 tx=0).stop 2023-10-30T14:30:13.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.164+0000 7f39ed565640 1 -- 172.21.15.57:0/3909632461 shutdown_connections 2023-10-30T14:30:13.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.164+0000 7f39ed565640 1 --2- 172.21.15.57:0/3909632461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39e80c82d0 0x7f39e80ccf60 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:13.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.164+0000 7f39ed565640 1 --2- 172.21.15.57:0/3909632461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39e81516c0 0x7f39e80c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:13.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.164+0000 7f39ed565640 1 -- 172.21.15.57:0/3909632461 >> 172.21.15.57:0/3909632461 conn(0x7f39e8071fb0 msgr2=0x7f39e80723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:13.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.164+0000 7f39ed565640 1 -- 172.21.15.57:0/3909632461 shutdown_connections 2023-10-30T14:30:13.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.164+0000 7f39ed565640 1 -- 172.21.15.57:0/3909632461 wait complete. 2023-10-30T14:30:13.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.164+0000 7f39ed565640 1 Processor -- start 2023-10-30T14:30:13.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.164+0000 7f39ed565640 1 -- start start 2023-10-30T14:30:13.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39ed565640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39e80c82d0 0x7f39e81ef890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:13.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39ed565640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39e80cd4a0 0x7f39e813acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:13.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39ed565640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39e81516c0 0x7f39e8144760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:13.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39ed565640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e813f940 con 0x7f39e80c82d0 2023-10-30T14:30:13.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39ed565640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e81efed0 con 0x7f39e80cd4a0 2023-10-30T14:30:13.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39ed565640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e81f01b0 con 0x7f39e81516c0 2023-10-30T14:30:13.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39e6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39e80c82d0 0x7f39e81ef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:13.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39e6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39e80c82d0 0x7f39e81ef890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54174/0 (socket says 172.21.15.57:54174) 2023-10-30T14:30:13.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39e6ffd640 1 -- 172.21.15.57:0/2868274498 learned_addr learned my addr 172.21.15.57:0/2868274498 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:13.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39dffff640 1 --2- 172.21.15.57:0/2868274498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39e80cd4a0 0x7f39e813acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:13.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39e77fe640 1 --2- 172.21.15.57:0/2868274498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39e81516c0 0x7f39e8144760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:13.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39e6ffd640 1 -- 172.21.15.57:0/2868274498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39e81516c0 msgr2=0x7f39e8144760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:13.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39e6ffd640 1 --2- 172.21.15.57:0/2868274498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39e81516c0 0x7f39e8144760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:13.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39e6ffd640 1 -- 172.21.15.57:0/2868274498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39e80cd4a0 msgr2=0x7f39e813acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:13.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39e6ffd640 1 --2- 172.21.15.57:0/2868274498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39e80cd4a0 0x7f39e813acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:13.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39e6ffd640 1 -- 172.21.15.57:0/2868274498 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d0009420 con 0x7f39e80c82d0 2023-10-30T14:30:13.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39e6ffd640 1 --2- 172.21.15.57:0/2868274498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39e80c82d0 0x7f39e81ef890 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f39d800e870 tx=0x7f39d800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:13.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39df7fe640 1 -- 172.21.15.57:0/2868274498 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d8009510 con 0x7f39e80c82d0 2023-10-30T14:30:13.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39df7fe640 1 -- 172.21.15.57:0/2868274498 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d800eec0 con 0x7f39e80c82d0 2023-10-30T14:30:13.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39df7fe640 1 -- 172.21.15.57:0/2868274498 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d800cb40 con 0x7f39e80c82d0 2023-10-30T14:30:13.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39ed565640 1 -- 172.21.15.57:0/2868274498 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e81f0490 con 0x7f39e80c82d0 2023-10-30T14:30:13.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39ed565640 1 -- 172.21.15.57:0/2868274498 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e81f0770 con 0x7f39e80c82d0 2023-10-30T14:30:13.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39ed565640 1 -- 172.21.15.57:0/2868274498 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e804e170 con 0x7f39e80c82d0 2023-10-30T14:30:13.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.168+0000 7f39df7fe640 1 -- 172.21.15.57:0/2868274498 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f39d800cca0 con 0x7f39e80c82d0 2023-10-30T14:30:13.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.172+0000 7f39df7fe640 1 --2- 172.21.15.57:0/2868274498 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f39c80664a0 0x7f39c8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:13.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.172+0000 7f39df7fe640 1 -- 172.21.15.57:0/2868274498 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f39d8084400 con 0x7f39e80c82d0 2023-10-30T14:30:13.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.172+0000 7f39dffff640 1 --2- 172.21.15.57:0/2868274498 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f39c80664a0 0x7f39c8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:13.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.172+0000 7f39dffff640 1 --2- 172.21.15.57:0/2868274498 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f39c80664a0 0x7f39c8068960 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f39d00ad000 tx=0x7f39d0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:13.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.176+0000 7f39df7fe640 1 -- 172.21.15.57:0/2868274498 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f39d8051430 con 0x7f39e80c82d0 2023-10-30T14:30:13.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.364+0000 7f39ed565640 1 -- 172.21.15.57:0/2868274498 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39e8161d60 con 0x7f39e80c82d0 2023-10-30T14:30:13.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.364+0000 7f39df7fe640 1 -- 172.21.15.57:0/2868274498 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f39d8054a50 con 0x7f39e80c82d0 2023-10-30T14:30:13.368 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:13.369 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:30:09.871551+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:13.369 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 99 2023-10-30T14:30:13.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.368+0000 7f39ed565640 1 -- 172.21.15.57:0/2868274498 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f39c80664a0 msgr2=0x7f39c8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:13.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.368+0000 7f39ed565640 1 --2- 172.21.15.57:0/2868274498 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f39c80664a0 0x7f39c8068960 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f39d00ad000 tx=0x7f39d0008040 comp rx=0 tx=0).stop 2023-10-30T14:30:13.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.368+0000 7f39ed565640 1 -- 172.21.15.57:0/2868274498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39e80c82d0 msgr2=0x7f39e81ef890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:13.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.368+0000 7f39ed565640 1 --2- 172.21.15.57:0/2868274498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39e80c82d0 0x7f39e81ef890 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f39d800e870 tx=0x7f39d800ec70 comp rx=0 tx=0).stop 2023-10-30T14:30:13.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.368+0000 7f39ed565640 1 -- 172.21.15.57:0/2868274498 shutdown_connections 2023-10-30T14:30:13.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.368+0000 7f39ed565640 1 --2- 172.21.15.57:0/2868274498 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f39c80664a0 0x7f39c8068960 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:13.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.368+0000 7f39ed565640 1 --2- 172.21.15.57:0/2868274498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39e81516c0 0x7f39e8144760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:13.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.368+0000 7f39ed565640 1 --2- 172.21.15.57:0/2868274498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39e80cd4a0 0x7f39e813acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:13.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.368+0000 7f39ed565640 1 --2- 172.21.15.57:0/2868274498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39e80c82d0 0x7f39e81ef890 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:13.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.368+0000 7f39ed565640 1 -- 172.21.15.57:0/2868274498 >> 172.21.15.57:0/2868274498 conn(0x7f39e8071fb0 msgr2=0x7f39e8072390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:13.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.368+0000 7f39ed565640 1 -- 172.21.15.57:0/2868274498 shutdown_connections 2023-10-30T14:30:13.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:13.368+0000 7f39ed565640 1 -- 172.21.15.57:0/2868274498 wait complete. 2023-10-30T14:30:13.391 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:30:15.393 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:15.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.536+0000 7f3e11d65640 1 Processor -- start 2023-10-30T14:30:15.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.536+0000 7f3e11d65640 1 -- start start 2023-10-30T14:30:15.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.540+0000 7f3e11d65640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3e0c0c6320 0x7f3e0c06fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:15.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.540+0000 7f3e11d65640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3e0c06be90 0x7f3e0c06c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:15.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.540+0000 7f3e11d65640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e0c0c55f0 con 0x7f3e0c14fc50 2023-10-30T14:30:15.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.540+0000 7f3e11d65640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e0c0c5f00 con 0x7f3e0c06be90 2023-10-30T14:30:15.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.540+0000 7f3e11d65640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e0c1567b0 con 0x7f3e0c0c6320 2023-10-30T14:30:15.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.540+0000 7f3e0bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3e0c06be90 0x7f3e0c06c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:15.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.540+0000 7f3e0bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3e0c06be90 0x7f3e0c06c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36682/0 (socket says 172.21.15.57:36682) 2023-10-30T14:30:15.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.540+0000 7f3e0bfff640 1 -- 172.21.15.57:0/3884789045 learned_addr learned my addr 172.21.15.57:0/3884789045 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:15.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.540+0000 7f3e0affd640 1 --2- 172.21.15.57:0/3884789045 >> v2:172.21.15.88:3300/0 conn(0x7f3e0c0c6320 0x7f3e0c06fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:15.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.540+0000 7f3e0bfff640 1 -- 172.21.15.57:0/3884789045 >> v2:172.21.15.88:3300/0 conn(0x7f3e0c0c6320 msgr2=0x7f3e0c06fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:15.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.540+0000 7f3e0bfff640 1 --2- 172.21.15.57:0/3884789045 >> v2:172.21.15.88:3300/0 conn(0x7f3e0c0c6320 0x7f3e0c06fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:15.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.540+0000 7f3e0bfff640 1 -- 172.21.15.57:0/3884789045 >> v1:172.21.15.88:6789/0 conn(0x7f3e0c14fc50 legacy=0x7f3e0c070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:15.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.540+0000 7f3e0bfff640 1 -- 172.21.15.57:0/3884789045 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0c156a70 con 0x7f3e0c06be90 2023-10-30T14:30:15.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.540+0000 7f3e0affd640 1 --2- 172.21.15.57:0/3884789045 >> v2:172.21.15.88:3300/0 conn(0x7f3e0c0c6320 0x7f3e0c06fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:30:15.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.544+0000 7f3e0bfff640 1 --2- 172.21.15.57:0/3884789045 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e0c06be90 0x7f3e0c06c320 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c0667f0 tx=0x7f3e0401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b7bcc90abacb7a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:15.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.544+0000 7f3e0a7fc640 1 -- 172.21.15.57:0/3884789045 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e040a5780 con 0x7f3e0c06be90 2023-10-30T14:30:15.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.544+0000 7f3e0a7fc640 1 -- 172.21.15.57:0/3884789045 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e040a58e0 con 0x7f3e0c06be90 2023-10-30T14:30:15.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.544+0000 7f3e0a7fc640 1 -- 172.21.15.57:0/3884789045 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e040a5c70 con 0x7f3e0c06be90 2023-10-30T14:30:15.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.544+0000 7f3e11d65640 1 -- 172.21.15.57:0/3884789045 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e0c06be90 msgr2=0x7f3e0c06c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:15.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.544+0000 7f3e11d65640 1 --2- 172.21.15.57:0/3884789045 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e0c06be90 0x7f3e0c06c320 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c0667f0 tx=0x7f3e0401c940 comp rx=0 tx=0).stop 2023-10-30T14:30:15.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.544+0000 7f3e11d65640 1 -- 172.21.15.57:0/3884789045 shutdown_connections 2023-10-30T14:30:15.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.544+0000 7f3e11d65640 1 --2- 172.21.15.57:0/3884789045 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e0c06be90 0x7f3e0c06c320 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:15.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.544+0000 7f3e11d65640 1 --2- 172.21.15.57:0/3884789045 >> v2:172.21.15.88:3300/0 conn(0x7f3e0c0c6320 0x7f3e0c06fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:15.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.544+0000 7f3e11d65640 1 -- 172.21.15.57:0/3884789045 >> 172.21.15.57:0/3884789045 conn(0x7f3e0c0cac70 msgr2=0x7f3e0c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:15.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.544+0000 7f3e11d65640 1 -- 172.21.15.57:0/3884789045 shutdown_connections 2023-10-30T14:30:15.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.544+0000 7f3e11d65640 1 -- 172.21.15.57:0/3884789045 wait complete. 2023-10-30T14:30:15.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.544+0000 7f3e11d65640 1 Processor -- start 2023-10-30T14:30:15.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.544+0000 7f3e11d65640 1 -- start start 2023-10-30T14:30:15.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e11d65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e0c06be90 0x7f3e0c158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:15.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e11d65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3e0c0c6320 0x7f3e0c14b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:15.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e11d65640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e0c14fc50 0x7f3e0c14c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:15.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e11d65640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c157610 con 0x7f3e0c14fc50 2023-10-30T14:30:15.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e11d65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c14b1f0 con 0x7f3e0c06be90 2023-10-30T14:30:15.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e11d65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c156e00 con 0x7f3e0c0c6320 2023-10-30T14:30:15.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e0bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e0c14fc50 0x7f3e0c14c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:15.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e0bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e0c14fc50 0x7f3e0c14c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36698/0 (socket says 172.21.15.57:36698) 2023-10-30T14:30:15.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e0bfff640 1 -- 172.21.15.57:0/1589448150 learned_addr learned my addr 172.21.15.57:0/1589448150 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:15.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e0b7fe640 1 --2- 172.21.15.57:0/1589448150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e0c06be90 0x7f3e0c158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:15.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e0affd640 1 --2- 172.21.15.57:0/1589448150 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3e0c0c6320 0x7f3e0c14b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:15.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e0bfff640 1 -- 172.21.15.57:0/1589448150 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3e0c0c6320 msgr2=0x7f3e0c14b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:15.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e0bfff640 1 --2- 172.21.15.57:0/1589448150 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3e0c0c6320 0x7f3e0c14b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:15.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e0bfff640 1 -- 172.21.15.57:0/1589448150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e0c06be90 msgr2=0x7f3e0c158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:15.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e0bfff640 1 --2- 172.21.15.57:0/1589448150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e0c06be90 0x7f3e0c158150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:15.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e0bfff640 1 -- 172.21.15.57:0/1589448150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e04009b30 con 0x7f3e0c14fc50 2023-10-30T14:30:15.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e0bfff640 1 --2- 172.21.15.57:0/1589448150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e0c14fc50 0x7f3e0c14c6e0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f3e040af040 tx=0x7f3e040196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:15.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3df3fff640 1 -- 172.21.15.57:0/1589448150 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e040b8670 con 0x7f3e0c14fc50 2023-10-30T14:30:15.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3df3fff640 1 -- 172.21.15.57:0/1589448150 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e040b87d0 con 0x7f3e0c14fc50 2023-10-30T14:30:15.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e11d65640 1 -- 172.21.15.57:0/1589448150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e0c14b490 con 0x7f3e0c14fc50 2023-10-30T14:30:15.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3df3fff640 1 -- 172.21.15.57:0/1589448150 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e040b8b60 con 0x7f3e0c14fc50 2023-10-30T14:30:15.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e11d65640 1 -- 172.21.15.57:0/1589448150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e0c159f10 con 0x7f3e0c14fc50 2023-10-30T14:30:15.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.548+0000 7f3e11d65640 1 -- 172.21.15.57:0/1589448150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e0c14b620 con 0x7f3e0c14fc50 2023-10-30T14:30:15.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.552+0000 7f3df3fff640 1 -- 172.21.15.57:0/1589448150 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3e040b8cc0 con 0x7f3e0c14fc50 2023-10-30T14:30:15.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.552+0000 7f3df3fff640 1 --2- 172.21.15.57:0/1589448150 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3df80664a0 0x7f3df8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:15.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.552+0000 7f3df3fff640 1 -- 172.21.15.57:0/1589448150 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f3e0411f520 con 0x7f3e0c14fc50 2023-10-30T14:30:15.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.552+0000 7f3e0b7fe640 1 --2- 172.21.15.57:0/1589448150 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3df80664a0 0x7f3df8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:15.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.552+0000 7f3e0b7fe640 1 --2- 172.21.15.57:0/1589448150 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3df80664a0 0x7f3df8068960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc007fd0 tx=0x7f3dfc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:15.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.556+0000 7f3df3fff640 1 -- 172.21.15.57:0/1589448150 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3e040ec7d0 con 0x7f3e0c14fc50 2023-10-30T14:30:15.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.748+0000 7f3e11d65640 1 -- 172.21.15.57:0/1589448150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e0c04e170 con 0x7f3e0c14fc50 2023-10-30T14:30:15.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.748+0000 7f3df3fff640 1 -- 172.21.15.57:0/1589448150 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f3e040efdf0 con 0x7f3e0c14fc50 2023-10-30T14:30:15.752 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:15.753 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:30:09.871551+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:15.753 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 99 2023-10-30T14:30:15.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.752+0000 7f3e11d65640 1 -- 172.21.15.57:0/1589448150 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3df80664a0 msgr2=0x7f3df8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:15.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.752+0000 7f3e11d65640 1 --2- 172.21.15.57:0/1589448150 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3df80664a0 0x7f3df8068960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc007fd0 tx=0x7f3dfc009040 comp rx=0 tx=0).stop 2023-10-30T14:30:15.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.752+0000 7f3e11d65640 1 -- 172.21.15.57:0/1589448150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e0c14fc50 msgr2=0x7f3e0c14c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:15.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.752+0000 7f3e11d65640 1 --2- 172.21.15.57:0/1589448150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e0c14fc50 0x7f3e0c14c6e0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f3e040af040 tx=0x7f3e040196e0 comp rx=0 tx=0).stop 2023-10-30T14:30:15.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.752+0000 7f3e11d65640 1 -- 172.21.15.57:0/1589448150 shutdown_connections 2023-10-30T14:30:15.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.752+0000 7f3e11d65640 1 --2- 172.21.15.57:0/1589448150 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3df80664a0 0x7f3df8068960 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:15.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.752+0000 7f3e11d65640 1 --2- 172.21.15.57:0/1589448150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e0c14fc50 0x7f3e0c14c6e0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:15.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.752+0000 7f3e11d65640 1 --2- 172.21.15.57:0/1589448150 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3e0c0c6320 0x7f3e0c14b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:15.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.752+0000 7f3e11d65640 1 --2- 172.21.15.57:0/1589448150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e0c06be90 0x7f3e0c158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:15.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.752+0000 7f3e11d65640 1 -- 172.21.15.57:0/1589448150 >> 172.21.15.57:0/1589448150 conn(0x7f3e0c0cac70 msgr2=0x7f3e0c14edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:15.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.752+0000 7f3e11d65640 1 -- 172.21.15.57:0/1589448150 shutdown_connections 2023-10-30T14:30:15.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:15.752+0000 7f3e11d65640 1 -- 172.21.15.57:0/1589448150 wait complete. 2023-10-30T14:30:15.774 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:30:17.775 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:17.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.916+0000 7f6754b0f640 1 Processor -- start 2023-10-30T14:30:17.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.916+0000 7f6754b0f640 1 -- start start 2023-10-30T14:30:17.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.920+0000 7f6754b0f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6750153c80 0x7f675015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:17.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.920+0000 7f6754b0f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f67500c4a10 0x7f67500c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:17.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.920+0000 7f6754b0f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f675015a3b0 0x7f67500ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:17.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.920+0000 7f6754b0f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f675015dec0 con 0x7f6750153c80 2023-10-30T14:30:17.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.920+0000 7f6754b0f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67501599a0 con 0x7f67500c4a10 2023-10-30T14:30:17.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.920+0000 7f6754b0f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6750159f20 con 0x7f675015a3b0 2023-10-30T14:30:17.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.920+0000 7f674e59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6750153c80 0x7f675015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:17.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.920+0000 7f674e59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6750153c80 0x7f675015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36704/0 (socket says 172.21.15.57:36704) 2023-10-30T14:30:17.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.920+0000 7f674e59c640 1 -- 172.21.15.57:0/511575393 learned_addr learned my addr 172.21.15.57:0/511575393 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:17.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.920+0000 7f674ed9d640 1 --2- 172.21.15.57:0/511575393 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f675015a3b0 0x7f67500ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:17.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.920+0000 7f674dd9b640 1 --2- 172.21.15.57:0/511575393 >> v2:172.21.15.88:3300/0 conn(0x7f67500c4a10 0x7f67500c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:17.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.920+0000 7f674e59c640 1 -- 172.21.15.57:0/511575393 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f675015a3b0 msgr2=0x7f67500ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:17.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.920+0000 7f674e59c640 1 --2- 172.21.15.57:0/511575393 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f675015a3b0 0x7f67500ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:17.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.920+0000 7f674e59c640 1 -- 172.21.15.57:0/511575393 >> v2:172.21.15.88:3300/0 conn(0x7f67500c4a10 msgr2=0x7f67500c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:17.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.920+0000 7f674e59c640 1 --2- 172.21.15.57:0/511575393 >> v2:172.21.15.88:3300/0 conn(0x7f67500c4a10 0x7f67500c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:17.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.920+0000 7f674e59c640 1 -- 172.21.15.57:0/511575393 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67500c47e0 con 0x7f6750153c80 2023-10-30T14:30:17.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.924+0000 7f674e59c640 1 --2- 172.21.15.57:0/511575393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6750153c80 0x7f675015cf90 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f675006be50 tx=0x7f674001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b09b673a92e73c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:17.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.924+0000 7f674d59a640 1 -- 172.21.15.57:0/511575393 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67400a89b0 con 0x7f6750153c80 2023-10-30T14:30:17.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.924+0000 7f674d59a640 1 -- 172.21.15.57:0/511575393 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67400a8b10 con 0x7f6750153c80 2023-10-30T14:30:17.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.924+0000 7f674d59a640 1 -- 172.21.15.57:0/511575393 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67400a8ea0 con 0x7f6750153c80 2023-10-30T14:30:17.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.924+0000 7f6754b0f640 1 -- 172.21.15.57:0/511575393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6750153c80 msgr2=0x7f675015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:17.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.924+0000 7f6754b0f640 1 --2- 172.21.15.57:0/511575393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6750153c80 0x7f675015cf90 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f675006be50 tx=0x7f674001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:30:17.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.924+0000 7f6754b0f640 1 -- 172.21.15.57:0/511575393 shutdown_connections 2023-10-30T14:30:17.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.924+0000 7f6754b0f640 1 --2- 172.21.15.57:0/511575393 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f675015a3b0 0x7f67500ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:17.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.924+0000 7f6754b0f640 1 --2- 172.21.15.57:0/511575393 >> v2:172.21.15.88:3300/0 conn(0x7f67500c4a10 0x7f67500c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:17.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.924+0000 7f6754b0f640 1 --2- 172.21.15.57:0/511575393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6750153c80 0x7f675015cf90 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:17.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.924+0000 7f6754b0f640 1 -- 172.21.15.57:0/511575393 >> 172.21.15.57:0/511575393 conn(0x7f67500715c0 msgr2=0x7f67500719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:17.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.924+0000 7f6754b0f640 1 -- 172.21.15.57:0/511575393 shutdown_connections 2023-10-30T14:30:17.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.924+0000 7f6754b0f640 1 -- 172.21.15.57:0/511575393 wait complete. 2023-10-30T14:30:17.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.924+0000 7f6754b0f640 1 Processor -- start 2023-10-30T14:30:17.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f6754b0f640 1 -- start start 2023-10-30T14:30:17.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f6754b0f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500c4a10 0x7f6750156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:17.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f6754b0f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6750153c80 0x7f67500ce280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:17.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f6754b0f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f675015a3b0 0x7f67500ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:17.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f6754b0f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67500cfe30 con 0x7f675015a3b0 2023-10-30T14:30:17.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f6754b0f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6750150d40 con 0x7f67500c4a10 2023-10-30T14:30:17.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f6754b0f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67500cefd0 con 0x7f6750153c80 2023-10-30T14:30:17.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f674ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f675015a3b0 0x7f67500ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:17.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f674ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f675015a3b0 0x7f67500ce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36710/0 (socket says 172.21.15.57:36710) 2023-10-30T14:30:17.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f674ed9d640 1 -- 172.21.15.57:0/736924481 learned_addr learned my addr 172.21.15.57:0/736924481 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:17.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f674dd9b640 1 --2- 172.21.15.57:0/736924481 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6750153c80 0x7f67500ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:17.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f674e59c640 1 --2- 172.21.15.57:0/736924481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500c4a10 0x7f6750156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:17.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f674ed9d640 1 -- 172.21.15.57:0/736924481 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6750153c80 msgr2=0x7f67500ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:17.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f674ed9d640 1 --2- 172.21.15.57:0/736924481 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6750153c80 0x7f67500ce280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:17.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f674ed9d640 1 -- 172.21.15.57:0/736924481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500c4a10 msgr2=0x7f6750156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:17.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f674ed9d640 1 --2- 172.21.15.57:0/736924481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500c4a10 0x7f6750156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:17.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f674ed9d640 1 -- 172.21.15.57:0/736924481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67400094d0 con 0x7f675015a3b0 2023-10-30T14:30:17.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f674e59c640 1 --2- 172.21.15.57:0/736924481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500c4a10 0x7f6750156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:30:17.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f674ed9d640 1 --2- 172.21.15.57:0/736924481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f675015a3b0 0x7f67500ce7c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f6748007e60 tx=0x7f674800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:17.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f673effd640 1 -- 172.21.15.57:0/736924481 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6748014420 con 0x7f675015a3b0 2023-10-30T14:30:17.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f673effd640 1 -- 172.21.15.57:0/736924481 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67480145a0 con 0x7f675015a3b0 2023-10-30T14:30:17.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f6754b0f640 1 -- 172.21.15.57:0/736924481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67500cf2b0 con 0x7f675015a3b0 2023-10-30T14:30:17.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f673effd640 1 -- 172.21.15.57:0/736924481 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6748014950 con 0x7f675015a3b0 2023-10-30T14:30:17.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f6754b0f640 1 -- 172.21.15.57:0/736924481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67500cf870 con 0x7f675015a3b0 2023-10-30T14:30:17.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.928+0000 7f6754b0f640 1 -- 172.21.15.57:0/736924481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f675004e170 con 0x7f675015a3b0 2023-10-30T14:30:17.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.932+0000 7f673effd640 1 -- 172.21.15.57:0/736924481 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6748014ab0 con 0x7f675015a3b0 2023-10-30T14:30:17.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.932+0000 7f673effd640 1 --2- 172.21.15.57:0/736924481 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f67300664a0 0x7f6730068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:17.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.932+0000 7f673effd640 1 -- 172.21.15.57:0/736924481 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f6748084d10 con 0x7f675015a3b0 2023-10-30T14:30:17.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.932+0000 7f674e59c640 1 --2- 172.21.15.57:0/736924481 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f67300664a0 0x7f6730068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:17.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.936+0000 7f674e59c640 1 --2- 172.21.15.57:0/736924481 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f67300664a0 0x7f6730068960 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f67400b2000 tx=0x7f6740008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:17.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:17.936+0000 7f673effd640 1 -- 172.21.15.57:0/736924481 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f674808a020 con 0x7f675015a3b0 2023-10-30T14:30:18.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:18.128+0000 7f6754b0f640 1 -- 172.21.15.57:0/736924481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67500cad40 con 0x7f675015a3b0 2023-10-30T14:30:18.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:18.128+0000 7f673effd640 1 -- 172.21.15.57:0/736924481 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f6748055720 con 0x7f675015a3b0 2023-10-30T14:30:18.131 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:18.131 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:30:09.871551+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:18.131 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 99 2023-10-30T14:30:18.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:18.128+0000 7f6754b0f640 1 -- 172.21.15.57:0/736924481 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f67300664a0 msgr2=0x7f6730068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:18.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:18.128+0000 7f6754b0f640 1 --2- 172.21.15.57:0/736924481 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f67300664a0 0x7f6730068960 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f67400b2000 tx=0x7f6740008040 comp rx=0 tx=0).stop 2023-10-30T14:30:18.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:18.128+0000 7f6754b0f640 1 -- 172.21.15.57:0/736924481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f675015a3b0 msgr2=0x7f67500ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:18.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:18.128+0000 7f6754b0f640 1 --2- 172.21.15.57:0/736924481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f675015a3b0 0x7f67500ce7c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f6748007e60 tx=0x7f674800e4b0 comp rx=0 tx=0).stop 2023-10-30T14:30:18.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:18.128+0000 7f6754b0f640 1 -- 172.21.15.57:0/736924481 shutdown_connections 2023-10-30T14:30:18.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:18.128+0000 7f6754b0f640 1 --2- 172.21.15.57:0/736924481 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f67300664a0 0x7f6730068960 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:18.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:18.128+0000 7f6754b0f640 1 --2- 172.21.15.57:0/736924481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f675015a3b0 0x7f67500ce7c0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:18.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:18.128+0000 7f6754b0f640 1 --2- 172.21.15.57:0/736924481 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6750153c80 0x7f67500ce280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:18.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:18.128+0000 7f6754b0f640 1 --2- 172.21.15.57:0/736924481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500c4a10 0x7f6750156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:18.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:18.128+0000 7f6754b0f640 1 -- 172.21.15.57:0/736924481 >> 172.21.15.57:0/736924481 conn(0x7f67500715c0 msgr2=0x7f67501513e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:18.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:18.128+0000 7f6754b0f640 1 -- 172.21.15.57:0/736924481 shutdown_connections 2023-10-30T14:30:18.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:18.128+0000 7f6754b0f640 1 -- 172.21.15.57:0/736924481 wait complete. 2023-10-30T14:30:18.152 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:30:20.152 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:20.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.304+0000 7fb4fccd6640 1 Processor -- start 2023-10-30T14:30:20.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.304+0000 7fb4fccd6640 1 -- start start 2023-10-30T14:30:20.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.304+0000 7fb4fccd6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4f806a0f0 0x7fb4f8153830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:20.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.304+0000 7fb4fccd6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb4f806a4d0 0x7fb4f8153d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:20.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.304+0000 7fb4fccd6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f80c1e60 con 0x7fb4f80c0260 2023-10-30T14:30:20.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.304+0000 7fb4fccd6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f815a3f0 con 0x7fb4f806a4d0 2023-10-30T14:30:20.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.304+0000 7fb4fccd6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f80cb2b0 con 0x7fb4f806a0f0 2023-10-30T14:30:20.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.304+0000 7fb4effff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb4f806a4d0 0x7fb4f8153d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:20.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.304+0000 7fb4effff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb4f806a4d0 0x7fb4f8153d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36722/0 (socket says 172.21.15.57:36722) 2023-10-30T14:30:20.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.304+0000 7fb4effff640 1 -- 172.21.15.57:0/3659058010 learned_addr learned my addr 172.21.15.57:0/3659058010 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:20.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.304+0000 7fb4f659c640 1 --2- 172.21.15.57:0/3659058010 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4f806a0f0 0x7fb4f8153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:20.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.308+0000 7fb4effff640 1 -- 172.21.15.57:0/3659058010 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4f806a0f0 msgr2=0x7fb4f8153830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:20.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.308+0000 7fb4effff640 1 --2- 172.21.15.57:0/3659058010 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4f806a0f0 0x7fb4f8153830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:20.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.308+0000 7fb4effff640 1 -- 172.21.15.57:0/3659058010 >> v1:172.21.15.88:6789/0 conn(0x7fb4f80c0260 legacy=0x7fb4f80cb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:20.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.308+0000 7fb4effff640 1 -- 172.21.15.57:0/3659058010 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f815a8c0 con 0x7fb4f806a4d0 2023-10-30T14:30:20.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.308+0000 7fb4effff640 1 --2- 172.21.15.57:0/3659058010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f806a4d0 0x7fb4f8153d70 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0009870 tx=0x7fb4e001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adfcb8ac1d902560 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:20.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.308+0000 7fb4f5d9b640 1 -- 172.21.15.57:0/3659058010 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e00aa670 con 0x7fb4f806a4d0 2023-10-30T14:30:20.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.308+0000 7fb4f5d9b640 1 -- 172.21.15.57:0/3659058010 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e00aa7d0 con 0x7fb4f806a4d0 2023-10-30T14:30:20.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.308+0000 7fb4f5d9b640 1 -- 172.21.15.57:0/3659058010 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e00aab60 con 0x7fb4f806a4d0 2023-10-30T14:30:20.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.308+0000 7fb4fccd6640 1 -- 172.21.15.57:0/3659058010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f806a4d0 msgr2=0x7fb4f8153d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:20.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.308+0000 7fb4fccd6640 1 --2- 172.21.15.57:0/3659058010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f806a4d0 0x7fb4f8153d70 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0009870 tx=0x7fb4e001c510 comp rx=0 tx=0).stop 2023-10-30T14:30:20.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4fccd6640 1 -- 172.21.15.57:0/3659058010 shutdown_connections 2023-10-30T14:30:20.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4fccd6640 1 --2- 172.21.15.57:0/3659058010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f806a4d0 0x7fb4f8153d70 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:20.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4fccd6640 1 --2- 172.21.15.57:0/3659058010 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4f806a0f0 0x7fb4f8153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:20.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4fccd6640 1 -- 172.21.15.57:0/3659058010 >> 172.21.15.57:0/3659058010 conn(0x7fb4f806de90 msgr2=0x7fb4f806e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:20.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4fccd6640 1 -- 172.21.15.57:0/3659058010 shutdown_connections 2023-10-30T14:30:20.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4fccd6640 1 -- 172.21.15.57:0/3659058010 wait complete. 2023-10-30T14:30:20.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4fccd6640 1 Processor -- start 2023-10-30T14:30:20.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4fccd6640 1 -- start start 2023-10-30T14:30:20.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4fccd6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f806a0f0 0x7fb4f80af0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:20.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4fccd6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4f806a4d0 0x7fb4f81f53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:20.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4fccd6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4f80c0260 0x7fb4f814cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:20.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4fccd6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f8071b40 con 0x7fb4f806a0f0 2023-10-30T14:30:20.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4fccd6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f815d270 con 0x7fb4f80c0260 2023-10-30T14:30:20.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4fccd6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f815d550 con 0x7fb4f806a4d0 2023-10-30T14:30:20.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4f659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f806a0f0 0x7fb4f80af0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:20.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4f659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f806a0f0 0x7fb4f80af0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36726/0 (socket says 172.21.15.57:36726) 2023-10-30T14:30:20.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4f659c640 1 -- 172.21.15.57:0/139922463 learned_addr learned my addr 172.21.15.57:0/139922463 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:20.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4effff640 1 --2- 172.21.15.57:0/139922463 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4f806a4d0 0x7fb4f81f53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:20.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4f6d9d640 1 --2- 172.21.15.57:0/139922463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4f80c0260 0x7fb4f814cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:20.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4f659c640 1 -- 172.21.15.57:0/139922463 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4f806a4d0 msgr2=0x7fb4f81f53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:20.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4f659c640 1 --2- 172.21.15.57:0/139922463 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4f806a4d0 0x7fb4f81f53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:20.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4f659c640 1 -- 172.21.15.57:0/139922463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4f80c0260 msgr2=0x7fb4f814cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:20.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4f659c640 1 --2- 172.21.15.57:0/139922463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4f80c0260 0x7fb4f814cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:20.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.312+0000 7fb4f659c640 1 -- 172.21.15.57:0/139922463 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e00094d0 con 0x7fb4f806a0f0 2023-10-30T14:30:20.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.316+0000 7fb4f659c640 1 --2- 172.21.15.57:0/139922463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f806a0f0 0x7fb4f80af0c0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb4e800e930 tx=0x7fb4e800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:20.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.316+0000 7fb4eeffd640 1 -- 172.21.15.57:0/139922463 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8009690 con 0x7fb4f806a0f0 2023-10-30T14:30:20.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.316+0000 7fb4eeffd640 1 -- 172.21.15.57:0/139922463 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e800ca00 con 0x7fb4f806a0f0 2023-10-30T14:30:20.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.316+0000 7fb4fccd6640 1 -- 172.21.15.57:0/139922463 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f80714f0 con 0x7fb4f806a0f0 2023-10-30T14:30:20.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.316+0000 7fb4eeffd640 1 -- 172.21.15.57:0/139922463 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e800cd70 con 0x7fb4f806a0f0 2023-10-30T14:30:20.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.316+0000 7fb4fccd6640 1 -- 172.21.15.57:0/139922463 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f815c610 con 0x7fb4f806a0f0 2023-10-30T14:30:20.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.316+0000 7fb4fccd6640 1 -- 172.21.15.57:0/139922463 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f8071350 con 0x7fb4f806a0f0 2023-10-30T14:30:20.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.316+0000 7fb4eeffd640 1 -- 172.21.15.57:0/139922463 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb4e8013470 con 0x7fb4f806a0f0 2023-10-30T14:30:20.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.316+0000 7fb4eeffd640 1 --2- 172.21.15.57:0/139922463 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4d80664d0 0x7fb4d8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:20.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.316+0000 7fb4eeffd640 1 -- 172.21.15.57:0/139922463 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7fb4e80844e0 con 0x7fb4f806a0f0 2023-10-30T14:30:20.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.316+0000 7fb4effff640 1 --2- 172.21.15.57:0/139922463 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4d80664d0 0x7fb4d8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:20.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.320+0000 7fb4effff640 1 --2- 172.21.15.57:0/139922463 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4d80664d0 0x7fb4d8068990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00ae000 tx=0x7fb4e00b5040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:20.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.320+0000 7fb4eeffd640 1 -- 172.21.15.57:0/139922463 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb4f8071350 con 0x7fb4f806a0f0 2023-10-30T14:30:20.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.508+0000 7fb4fccd6640 1 -- 172.21.15.57:0/139922463 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4f8071350 con 0x7fb4f806a0f0 2023-10-30T14:30:20.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.508+0000 7fb4eeffd640 1 -- 172.21.15.57:0/139922463 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5738 (secure 0 0 0) 0x7fb4f8071350 con 0x7fb4f806a0f0 2023-10-30T14:30:20.513 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:20.513 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:30:18.661785+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:resolve","state_seq":10,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:20.513 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 100 2023-10-30T14:30:20.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.512+0000 7fb4fccd6640 1 -- 172.21.15.57:0/139922463 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4d80664d0 msgr2=0x7fb4d8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:20.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.512+0000 7fb4fccd6640 1 --2- 172.21.15.57:0/139922463 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4d80664d0 0x7fb4d8068990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00ae000 tx=0x7fb4e00b5040 comp rx=0 tx=0).stop 2023-10-30T14:30:20.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.512+0000 7fb4fccd6640 1 -- 172.21.15.57:0/139922463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f806a0f0 msgr2=0x7fb4f80af0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:20.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.512+0000 7fb4fccd6640 1 --2- 172.21.15.57:0/139922463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f806a0f0 0x7fb4f80af0c0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb4e800e930 tx=0x7fb4e800edf0 comp rx=0 tx=0).stop 2023-10-30T14:30:20.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.512+0000 7fb4fccd6640 1 -- 172.21.15.57:0/139922463 shutdown_connections 2023-10-30T14:30:20.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.512+0000 7fb4fccd6640 1 --2- 172.21.15.57:0/139922463 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4d80664d0 0x7fb4d8068990 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:20.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.512+0000 7fb4fccd6640 1 --2- 172.21.15.57:0/139922463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4f80c0260 0x7fb4f814cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:20.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.512+0000 7fb4fccd6640 1 --2- 172.21.15.57:0/139922463 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4f806a4d0 0x7fb4f81f53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:20.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.512+0000 7fb4fccd6640 1 --2- 172.21.15.57:0/139922463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4f806a0f0 0x7fb4f80af0c0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:20.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.512+0000 7fb4fccd6640 1 -- 172.21.15.57:0/139922463 >> 172.21.15.57:0/139922463 conn(0x7fb4f806de90 msgr2=0x7fb4f8159190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:20.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.512+0000 7fb4fccd6640 1 -- 172.21.15.57:0/139922463 shutdown_connections 2023-10-30T14:30:20.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:20.512+0000 7fb4fccd6640 1 -- 172.21.15.57:0/139922463 wait complete. 2023-10-30T14:30:20.534 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:30:22.535 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:22.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.688+0000 7f6450e17640 1 Processor -- start 2023-10-30T14:30:22.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.688+0000 7f6450e17640 1 -- start start 2023-10-30T14:30:22.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.688+0000 7f6450e17640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f644c153c80 0x7f644c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:22.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.688+0000 7f6450e17640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f644c0c42a0 0x7f644c0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:22.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.688+0000 7f6450e17640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f644c159f20 con 0x7f644c0c4a10 2023-10-30T14:30:22.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.688+0000 7f6450e17640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f644c15dec0 con 0x7f644c153c80 2023-10-30T14:30:22.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.688+0000 7f6450e17640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644c1599a0 con 0x7f644c0c42a0 2023-10-30T14:30:22.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.688+0000 7f644a59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f644c153c80 0x7f644c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:22.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.688+0000 7f644a59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f644c153c80 0x7f644c15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36736/0 (socket says 172.21.15.57:36736) 2023-10-30T14:30:22.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.688+0000 7f644a59c640 1 -- 172.21.15.57:0/156539779 learned_addr learned my addr 172.21.15.57:0/156539779 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:22.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.688+0000 7f6443fff640 1 --1- 172.21.15.57:0/156539779 >> v1:172.21.15.57:6789/0 conn(0x7f644c0c4a10 0x7f644c15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48220/0 (socket says 172.21.15.57:48220) 2023-10-30T14:30:22.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.688+0000 7f644ad9d640 1 --2- 172.21.15.57:0/156539779 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f644c0c42a0 0x7f644c0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:22.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.688+0000 7f644a59c640 1 -- 172.21.15.57:0/156539779 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f644c0c42a0 msgr2=0x7f644c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:22.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.688+0000 7f644a59c640 1 --2- 172.21.15.57:0/156539779 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f644c0c42a0 0x7f644c0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:22.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.688+0000 7f644a59c640 1 -- 172.21.15.57:0/156539779 >> v1:172.21.15.57:6789/0 conn(0x7f644c0c4a10 legacy=0x7f644c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:22.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.688+0000 7f644a59c640 1 -- 172.21.15.57:0/156539779 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c15d620 con 0x7f644c153c80 2023-10-30T14:30:22.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.688+0000 7f6449d9b640 1 -- 172.21.15.57:0/156539779 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236916217 0 0) 0x7f644c159f20 con 0x7f644c0c4a10 2023-10-30T14:30:22.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.692+0000 7f644a59c640 1 --2- 172.21.15.57:0/156539779 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f644c153c80 0x7f644c15cf90 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f643c009870 tx=0x7f643c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b11fa87e68b3dc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:22.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.692+0000 7f6449d9b640 1 -- 172.21.15.57:0/156539779 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c0a8a30 con 0x7f644c153c80 2023-10-30T14:30:22.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.692+0000 7f6449d9b640 1 -- 172.21.15.57:0/156539779 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643c0a8b90 con 0x7f644c153c80 2023-10-30T14:30:22.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.692+0000 7f6449d9b640 1 -- 172.21.15.57:0/156539779 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c0b0440 con 0x7f644c153c80 2023-10-30T14:30:22.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.692+0000 7f6450e17640 1 -- 172.21.15.57:0/156539779 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f644c153c80 msgr2=0x7f644c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:22.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.692+0000 7f6450e17640 1 --2- 172.21.15.57:0/156539779 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f644c153c80 0x7f644c15cf90 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f643c009870 tx=0x7f643c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:30:22.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.692+0000 7f6450e17640 1 -- 172.21.15.57:0/156539779 shutdown_connections 2023-10-30T14:30:22.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.692+0000 7f6450e17640 1 --2- 172.21.15.57:0/156539779 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f644c0c42a0 0x7f644c0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:22.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.692+0000 7f6450e17640 1 --2- 172.21.15.57:0/156539779 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f644c153c80 0x7f644c15cf90 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:22.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.692+0000 7f6450e17640 1 -- 172.21.15.57:0/156539779 >> 172.21.15.57:0/156539779 conn(0x7f644c0715c0 msgr2=0x7f644c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:22.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.692+0000 7f6450e17640 1 -- 172.21.15.57:0/156539779 shutdown_connections 2023-10-30T14:30:22.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.692+0000 7f6450e17640 1 -- 172.21.15.57:0/156539779 wait complete. 2023-10-30T14:30:22.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f6450e17640 1 Processor -- start 2023-10-30T14:30:22.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f6450e17640 1 -- start start 2023-10-30T14:30:22.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f6450e17640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f644c0c42a0 0x7f644c154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:22.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f6450e17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f644c0c4a10 0x7f644c154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:22.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f6450e17640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f644c153c80 0x7f644c155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:22.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f6450e17640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644c156620 con 0x7f644c153c80 2023-10-30T14:30:22.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f6450e17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644c0ceac0 con 0x7f644c0c4a10 2023-10-30T14:30:22.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f6450e17640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644c154850 con 0x7f644c0c42a0 2023-10-30T14:30:22.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f644ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f644c153c80 0x7f644c155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:22.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f644ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f644c153c80 0x7f644c155220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36750/0 (socket says 172.21.15.57:36750) 2023-10-30T14:30:22.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f644ad9d640 1 -- 172.21.15.57:0/745577123 learned_addr learned my addr 172.21.15.57:0/745577123 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:22.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f6443fff640 1 --2- 172.21.15.57:0/745577123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f644c0c4a10 0x7f644c154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:22.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f644a59c640 1 --2- 172.21.15.57:0/745577123 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f644c0c42a0 0x7f644c154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:22.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f644ad9d640 1 -- 172.21.15.57:0/745577123 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f644c0c42a0 msgr2=0x7f644c154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:22.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f644ad9d640 1 --2- 172.21.15.57:0/745577123 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f644c0c42a0 0x7f644c154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:22.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f644ad9d640 1 -- 172.21.15.57:0/745577123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f644c0c4a10 msgr2=0x7f644c154ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:22.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f644ad9d640 1 --2- 172.21.15.57:0/745577123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f644c0c4a10 0x7f644c154ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:22.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f644ad9d640 1 -- 172.21.15.57:0/745577123 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643c0094d0 con 0x7f644c153c80 2023-10-30T14:30:22.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f644ad9d640 1 --2- 172.21.15.57:0/745577123 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f644c153c80 0x7f644c155220 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f6444007e70 tx=0x7f644400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:22.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f6442ffd640 1 -- 172.21.15.57:0/745577123 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6444014620 con 0x7f644c153c80 2023-10-30T14:30:22.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f6442ffd640 1 -- 172.21.15.57:0/745577123 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64440147a0 con 0x7f644c153c80 2023-10-30T14:30:22.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f6442ffd640 1 -- 172.21.15.57:0/745577123 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6444014b50 con 0x7f644c153c80 2023-10-30T14:30:22.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f6450e17640 1 -- 172.21.15.57:0/745577123 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f644c1f29f0 con 0x7f644c153c80 2023-10-30T14:30:22.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.696+0000 7f6450e17640 1 -- 172.21.15.57:0/745577123 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f644c1f2fa0 con 0x7f644c153c80 2023-10-30T14:30:22.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.700+0000 7f6450e17640 1 -- 172.21.15.57:0/745577123 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f644c04e170 con 0x7f644c153c80 2023-10-30T14:30:22.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.700+0000 7f6442ffd640 1 -- 172.21.15.57:0/745577123 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6444014cb0 con 0x7f644c153c80 2023-10-30T14:30:22.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.700+0000 7f6442ffd640 1 --2- 172.21.15.57:0/745577123 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f642c0664a0 0x7f642c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:22.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.700+0000 7f6442ffd640 1 -- 172.21.15.57:0/745577123 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f6444084cf0 con 0x7f644c153c80 2023-10-30T14:30:22.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.700+0000 7f644a59c640 1 --2- 172.21.15.57:0/745577123 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f642c0664a0 0x7f642c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:22.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.700+0000 7f644a59c640 1 --2- 172.21.15.57:0/745577123 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f642c0664a0 0x7f642c068960 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f643c0b2000 tx=0x7f643c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:22.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.704+0000 7f6442ffd640 1 -- 172.21.15.57:0/745577123 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6444051c70 con 0x7f644c153c80 2023-10-30T14:30:22.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.896+0000 7f6450e17640 1 -- 172.21.15.57:0/745577123 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f644c14dae0 con 0x7f644c153c80 2023-10-30T14:30:22.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.896+0000 7f6442ffd640 1 -- 172.21.15.57:0/745577123 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5738 (secure 0 0 0) 0x7f6444055290 con 0x7f644c153c80 2023-10-30T14:30:22.900 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:22.900 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:30:18.661785+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:resolve","state_seq":10,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:22.900 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 100 2023-10-30T14:30:22.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.900+0000 7f6450e17640 1 -- 172.21.15.57:0/745577123 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f642c0664a0 msgr2=0x7f642c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:22.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.900+0000 7f6450e17640 1 --2- 172.21.15.57:0/745577123 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f642c0664a0 0x7f642c068960 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f643c0b2000 tx=0x7f643c008040 comp rx=0 tx=0).stop 2023-10-30T14:30:22.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.900+0000 7f6450e17640 1 -- 172.21.15.57:0/745577123 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f644c153c80 msgr2=0x7f644c155220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:22.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.900+0000 7f6450e17640 1 --2- 172.21.15.57:0/745577123 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f644c153c80 0x7f644c155220 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f6444007e70 tx=0x7f644400e4b0 comp rx=0 tx=0).stop 2023-10-30T14:30:22.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.900+0000 7f6450e17640 1 -- 172.21.15.57:0/745577123 shutdown_connections 2023-10-30T14:30:22.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.900+0000 7f6450e17640 1 --2- 172.21.15.57:0/745577123 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f642c0664a0 0x7f642c068960 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:22.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.900+0000 7f6450e17640 1 --2- 172.21.15.57:0/745577123 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f644c153c80 0x7f644c155220 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:22.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.900+0000 7f6450e17640 1 --2- 172.21.15.57:0/745577123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f644c0c4a10 0x7f644c154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:22.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.900+0000 7f6450e17640 1 --2- 172.21.15.57:0/745577123 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f644c0c42a0 0x7f644c154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:22.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.900+0000 7f6450e17640 1 -- 172.21.15.57:0/745577123 >> 172.21.15.57:0/745577123 conn(0x7f644c0715c0 msgr2=0x7f644c158450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:22.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.900+0000 7f6450e17640 1 -- 172.21.15.57:0/745577123 shutdown_connections 2023-10-30T14:30:22.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:22.900+0000 7f6450e17640 1 -- 172.21.15.57:0/745577123 wait complete. 2023-10-30T14:30:22.922 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 100, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 100, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6027": { "addr": "172.21.15.57:6836/2733323524", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2733323524, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2733323524, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6027, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 24 }, "gid_6204": { "addr": "172.21.15.57:6837/1843528309", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 1843528309, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 1843528309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6204, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_6414": { "addr": "172.21.15.57:6834/2321408149", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2321408149, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2321408149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6414, "incarnation": 99, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:resolve", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:30:18.661785+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6414, "mds_1": 6027, "mds_2": 6204 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:30:24.923 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:25.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.072+0000 7fd14aad6640 1 Processor -- start 2023-10-30T14:30:25.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.072+0000 7fd14aad6640 1 -- start start 2023-10-30T14:30:25.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.072+0000 7fd14aad6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd14406bd10 0x7fd1440c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:25.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.072+0000 7fd14aad6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1440c5030 con 0x7fd144151840 2023-10-30T14:30:25.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.072+0000 7fd14aad6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd144157830 con 0x7fd144151c20 2023-10-30T14:30:25.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.072+0000 7fd14aad6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd144157b10 con 0x7fd14406bd10 2023-10-30T14:30:25.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.072+0000 7fd149073640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd14406bd10 0x7fd1440c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:25.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.072+0000 7fd149073640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd14406bd10 0x7fd1440c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35304/0 (socket says 172.21.15.57:35304) 2023-10-30T14:30:25.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.072+0000 7fd149073640 1 -- 172.21.15.57:0/644107941 learned_addr learned my addr 172.21.15.57:0/644107941 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:25.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.072+0000 7fd148872640 1 --1- 172.21.15.57:0/644107941 >> v1:172.21.15.57:6789/0 conn(0x7fd144151840 0x7fd144072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46792/0 (socket says 172.21.15.57:46792) 2023-10-30T14:30:25.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.076+0000 7fd149073640 1 -- 172.21.15.57:0/644107941 >> v1:172.21.15.88:6789/0 conn(0x7fd144151c20 legacy=0x7fd14406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:25.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.076+0000 7fd149073640 1 -- 172.21.15.57:0/644107941 >> v1:172.21.15.57:6789/0 conn(0x7fd144151840 legacy=0x7fd144072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:25.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.076+0000 7fd149073640 1 -- 172.21.15.57:0/644107941 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd144157df0 con 0x7fd14406bd10 2023-10-30T14:30:25.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.076+0000 7fd149073640 1 --2- 172.21.15.57:0/644107941 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd14406bd10 0x7fd1440c49a0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009e50 tx=0x7fd13c01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2385c47eb8382fcd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:25.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.076+0000 7fd1437fe640 1 -- 172.21.15.57:0/644107941 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c0af400 con 0x7fd14406bd10 2023-10-30T14:30:25.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.076+0000 7fd1437fe640 1 -- 172.21.15.57:0/644107941 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13c0ab050 con 0x7fd14406bd10 2023-10-30T14:30:25.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.076+0000 7fd1437fe640 1 -- 172.21.15.57:0/644107941 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c0af780 con 0x7fd14406bd10 2023-10-30T14:30:25.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.076+0000 7fd14aad6640 1 -- 172.21.15.57:0/644107941 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd14406bd10 msgr2=0x7fd1440c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:25.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.076+0000 7fd14aad6640 1 --2- 172.21.15.57:0/644107941 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd14406bd10 0x7fd1440c49a0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009e50 tx=0x7fd13c01f1f0 comp rx=0 tx=0).stop 2023-10-30T14:30:25.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.076+0000 7fd14aad6640 1 -- 172.21.15.57:0/644107941 shutdown_connections 2023-10-30T14:30:25.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.076+0000 7fd14aad6640 1 --2- 172.21.15.57:0/644107941 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd14406bd10 0x7fd1440c49a0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:25.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.076+0000 7fd14aad6640 1 -- 172.21.15.57:0/644107941 >> 172.21.15.57:0/644107941 conn(0x7fd1440cac70 msgr2=0x7fd1440cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:25.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.076+0000 7fd14aad6640 1 -- 172.21.15.57:0/644107941 shutdown_connections 2023-10-30T14:30:25.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd14aad6640 1 -- 172.21.15.57:0/644107941 wait complete. 2023-10-30T14:30:25.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd14aad6640 1 Processor -- start 2023-10-30T14:30:25.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd14aad6640 1 -- start start 2023-10-30T14:30:25.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd14aad6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd14406bd10 0x7fd144070380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:25.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd14aad6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd144151840 0x7fd144150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:25.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd14aad6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd144151c20 0x7fd1441e20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:25.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd14aad6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1440708c0 con 0x7fd144151840 2023-10-30T14:30:25.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd14aad6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd144152310 con 0x7fd14406bd10 2023-10-30T14:30:25.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd14aad6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14414d3b0 con 0x7fd144151c20 2023-10-30T14:30:25.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd143fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd144151840 0x7fd144150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:25.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd143fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd144151840 0x7fd144150450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35316/0 (socket says 172.21.15.57:35316) 2023-10-30T14:30:25.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd143fff640 1 -- 172.21.15.57:0/3193816187 learned_addr learned my addr 172.21.15.57:0/3193816187 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:25.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd148872640 1 --2- 172.21.15.57:0/3193816187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd14406bd10 0x7fd144070380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:25.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd149073640 1 --2- 172.21.15.57:0/3193816187 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd144151c20 0x7fd1441e20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:25.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd143fff640 1 -- 172.21.15.57:0/3193816187 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd144151c20 msgr2=0x7fd1441e20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd143fff640 1 --2- 172.21.15.57:0/3193816187 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd144151c20 0x7fd1441e20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd143fff640 1 -- 172.21.15.57:0/3193816187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd14406bd10 msgr2=0x7fd144070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd143fff640 1 --2- 172.21.15.57:0/3193816187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd14406bd10 0x7fd144070380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd143fff640 1 -- 172.21.15.57:0/3193816187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13c009b30 con 0x7fd144151840 2023-10-30T14:30:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd143fff640 1 --2- 172.21.15.57:0/3193816187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd144151840 0x7fd144150450 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fd13000c3c0 tx=0x7fd13000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:25.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd1417fa640 1 -- 172.21.15.57:0/3193816187 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1300052d0 con 0x7fd144151840 2023-10-30T14:30:25.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.080+0000 7fd1417fa640 1 -- 172.21.15.57:0/3193816187 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd130005430 con 0x7fd144151840 2023-10-30T14:30:25.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.084+0000 7fd1417fa640 1 -- 172.21.15.57:0/3193816187 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd130003840 con 0x7fd144151840 2023-10-30T14:30:25.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.084+0000 7fd14aad6640 1 -- 172.21.15.57:0/3193816187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1441e1c10 con 0x7fd144151840 2023-10-30T14:30:25.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.084+0000 7fd14aad6640 1 -- 172.21.15.57:0/3193816187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd144070f60 con 0x7fd144151840 2023-10-30T14:30:25.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.084+0000 7fd14aad6640 1 -- 172.21.15.57:0/3193816187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd14404e170 con 0x7fd144151840 2023-10-30T14:30:25.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.084+0000 7fd1417fa640 1 -- 172.21.15.57:0/3193816187 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd1300039a0 con 0x7fd144151840 2023-10-30T14:30:25.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.084+0000 7fd1417fa640 1 --2- 172.21.15.57:0/3193816187 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd1280664a0 0x7fd128068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:25.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.084+0000 7fd1417fa640 1 -- 172.21.15.57:0/3193816187 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7fd130083680 con 0x7fd144151840 2023-10-30T14:30:25.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.084+0000 7fd148872640 1 --2- 172.21.15.57:0/3193816187 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd1280664a0 0x7fd128068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:25.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.088+0000 7fd148872640 1 --2- 172.21.15.57:0/3193816187 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd1280664a0 0x7fd128068960 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd138000f70 tx=0x7fd138007e50 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:25.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.088+0000 7fd1417fa640 1 -- 172.21.15.57:0/3193816187 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd1300507d0 con 0x7fd144151840 2023-10-30T14:30:25.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.276+0000 7fd14aad6640 1 -- 172.21.15.57:0/3193816187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1440c48c0 con 0x7fd144151840 2023-10-30T14:30:25.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.276+0000 7fd1417fa640 1 -- 172.21.15.57:0/3193816187 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5740 (secure 0 0 0) 0x7fd130053df0 con 0x7fd144151840 2023-10-30T14:30:25.280 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:25.281 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:30:23.995194+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:25.281 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 101 2023-10-30T14:30:25.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.280+0000 7fd14aad6640 1 -- 172.21.15.57:0/3193816187 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd1280664a0 msgr2=0x7fd128068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:25.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.280+0000 7fd14aad6640 1 --2- 172.21.15.57:0/3193816187 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd1280664a0 0x7fd128068960 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd138000f70 tx=0x7fd138007e50 comp rx=0 tx=0).stop 2023-10-30T14:30:25.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.280+0000 7fd14aad6640 1 -- 172.21.15.57:0/3193816187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd144151840 msgr2=0x7fd144150450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:25.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.280+0000 7fd14aad6640 1 --2- 172.21.15.57:0/3193816187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd144151840 0x7fd144150450 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fd13000c3c0 tx=0x7fd13000c880 comp rx=0 tx=0).stop 2023-10-30T14:30:25.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.280+0000 7fd14aad6640 1 -- 172.21.15.57:0/3193816187 shutdown_connections 2023-10-30T14:30:25.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.280+0000 7fd14aad6640 1 --2- 172.21.15.57:0/3193816187 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd1280664a0 0x7fd128068960 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:25.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.280+0000 7fd14aad6640 1 --2- 172.21.15.57:0/3193816187 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd144151c20 0x7fd1441e20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:25.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.280+0000 7fd14aad6640 1 --2- 172.21.15.57:0/3193816187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd144151840 0x7fd144150450 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:25.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.280+0000 7fd14aad6640 1 --2- 172.21.15.57:0/3193816187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd14406bd10 0x7fd144070380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:25.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.280+0000 7fd14aad6640 1 -- 172.21.15.57:0/3193816187 >> 172.21.15.57:0/3193816187 conn(0x7fd1440cac70 msgr2=0x7fd1440c6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:25.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.280+0000 7fd14aad6640 1 -- 172.21.15.57:0/3193816187 shutdown_connections 2023-10-30T14:30:25.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:25.280+0000 7fd14aad6640 1 -- 172.21.15.57:0/3193816187 wait complete. 2023-10-30T14:30:25.301 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:30:27.302 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:27.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.456+0000 7ff991291640 1 Processor -- start 2023-10-30T14:30:27.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.456+0000 7ff991291640 1 -- start start 2023-10-30T14:30:27.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.456+0000 7ff991291640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff98c1516c0 0x7ff98c0c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:27.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.456+0000 7ff991291640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff98c0c2500 0x7ff98c0c2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:27.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.456+0000 7ff991291640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff98c151aa0 con 0x7ff98c0c82d0 2023-10-30T14:30:27.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.456+0000 7ff991291640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff98c0c74a0 con 0x7ff98c0c2500 2023-10-30T14:30:27.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.456+0000 7ff991291640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98c0c2e90 con 0x7ff98c1516c0 2023-10-30T14:30:27.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.456+0000 7ff98b7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff98c0c2500 0x7ff98c0c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:27.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.456+0000 7ff98affd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff98c1516c0 0x7ff98c0c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:27.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.456+0000 7ff98b7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff98c0c2500 0x7ff98c0c2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:55038/0 (socket says 172.21.15.57:55038) 2023-10-30T14:30:27.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.456+0000 7ff98b7fe640 1 -- 172.21.15.57:0/1489819228 learned_addr learned my addr 172.21.15.57:0/1489819228 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:27.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.456+0000 7ff98b7fe640 1 -- 172.21.15.57:0/1489819228 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff98c1516c0 msgr2=0x7ff98c0c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:27.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.456+0000 7ff98b7fe640 1 --2- 172.21.15.57:0/1489819228 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff98c1516c0 0x7ff98c0c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:27.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.456+0000 7ff98b7fe640 1 -- 172.21.15.57:0/1489819228 >> v1:172.21.15.88:6789/0 conn(0x7ff98c0c82d0 legacy=0x7ff98c0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:27.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.456+0000 7ff98b7fe640 1 -- 172.21.15.57:0/1489819228 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98c158af0 con 0x7ff98c0c2500 2023-10-30T14:30:27.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.456+0000 7ff98affd640 1 --2- 172.21.15.57:0/1489819228 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff98c1516c0 0x7ff98c0c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:30:27.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.460+0000 7ff98b7fe640 1 --2- 172.21.15.57:0/1489819228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff98c0c2500 0x7ff98c0c2950 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff984009e80 tx=0x7ff98401c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d96ec50c88e226b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:27.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.460+0000 7ff98a7fc640 1 -- 172.21.15.57:0/1489819228 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840a56c0 con 0x7ff98c0c2500 2023-10-30T14:30:27.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.460+0000 7ff98a7fc640 1 -- 172.21.15.57:0/1489819228 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9840a5820 con 0x7ff98c0c2500 2023-10-30T14:30:27.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.460+0000 7ff98a7fc640 1 -- 172.21.15.57:0/1489819228 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840a5bb0 con 0x7ff98c0c2500 2023-10-30T14:30:27.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.460+0000 7ff991291640 1 -- 172.21.15.57:0/1489819228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff98c0c2500 msgr2=0x7ff98c0c2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:27.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.460+0000 7ff991291640 1 --2- 172.21.15.57:0/1489819228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff98c0c2500 0x7ff98c0c2950 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff984009e80 tx=0x7ff98401c9f0 comp rx=0 tx=0).stop 2023-10-30T14:30:27.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.460+0000 7ff991291640 1 -- 172.21.15.57:0/1489819228 shutdown_connections 2023-10-30T14:30:27.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.460+0000 7ff991291640 1 --2- 172.21.15.57:0/1489819228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff98c0c2500 0x7ff98c0c2950 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:27.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.460+0000 7ff991291640 1 --2- 172.21.15.57:0/1489819228 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff98c1516c0 0x7ff98c0c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:27.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.460+0000 7ff991291640 1 -- 172.21.15.57:0/1489819228 >> 172.21.15.57:0/1489819228 conn(0x7ff98c071fb0 msgr2=0x7ff98c0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:27.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.460+0000 7ff991291640 1 -- 172.21.15.57:0/1489819228 shutdown_connections 2023-10-30T14:30:27.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.460+0000 7ff991291640 1 -- 172.21.15.57:0/1489819228 wait complete. 2023-10-30T14:30:27.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff991291640 1 Processor -- start 2023-10-30T14:30:27.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff991291640 1 -- start start 2023-10-30T14:30:27.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff991291640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff98c0c2500 0x7ff98c1f5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:27.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff991291640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff98c0c82d0 0x7ff98c144a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:27.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff991291640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff98c1516c0 0x7ff98c146d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:27.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff991291640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98c1f23d0 con 0x7ff98c0c82d0 2023-10-30T14:30:27.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff991291640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98c1596c0 con 0x7ff98c1516c0 2023-10-30T14:30:27.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff991291640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98c1f1b00 con 0x7ff98c0c2500 2023-10-30T14:30:27.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff983fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff98c0c82d0 0x7ff98c144a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:27.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff983fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff98c0c82d0 0x7ff98c144a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35326/0 (socket says 172.21.15.57:35326) 2023-10-30T14:30:27.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff983fff640 1 -- 172.21.15.57:0/2211243189 learned_addr learned my addr 172.21.15.57:0/2211243189 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:27.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff98b7fe640 1 --2- 172.21.15.57:0/2211243189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff98c1516c0 0x7ff98c146d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:27.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff98affd640 1 --2- 172.21.15.57:0/2211243189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff98c0c2500 0x7ff98c1f5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:27.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff983fff640 1 -- 172.21.15.57:0/2211243189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff98c0c2500 msgr2=0x7ff98c1f5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:27.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff983fff640 1 --2- 172.21.15.57:0/2211243189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff98c0c2500 0x7ff98c1f5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:27.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff983fff640 1 -- 172.21.15.57:0/2211243189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff98c1516c0 msgr2=0x7ff98c146d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:27.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff983fff640 1 --2- 172.21.15.57:0/2211243189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff98c1516c0 0x7ff98c146d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:27.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff983fff640 1 -- 172.21.15.57:0/2211243189 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97c009420 con 0x7ff98c0c82d0 2023-10-30T14:30:27.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff983fff640 1 --2- 172.21.15.57:0/2211243189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff98c0c82d0 0x7ff98c144a00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7ff97400c370 tx=0x7ff97400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:27.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff9837fe640 1 -- 172.21.15.57:0/2211243189 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9740052d0 con 0x7ff98c0c82d0 2023-10-30T14:30:27.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff9837fe640 1 -- 172.21.15.57:0/2211243189 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff974005430 con 0x7ff98c0c82d0 2023-10-30T14:30:27.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff9837fe640 1 -- 172.21.15.57:0/2211243189 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff974007260 con 0x7ff98c0c82d0 2023-10-30T14:30:27.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.464+0000 7ff991291640 1 -- 172.21.15.57:0/2211243189 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff984009b30 con 0x7ff98c0c82d0 2023-10-30T14:30:27.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.468+0000 7ff991291640 1 -- 172.21.15.57:0/2211243189 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98c1f5fc0 con 0x7ff98c0c82d0 2023-10-30T14:30:27.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.468+0000 7ff991291640 1 -- 172.21.15.57:0/2211243189 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff98c145dd0 con 0x7ff98c0c82d0 2023-10-30T14:30:27.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.468+0000 7ff9837fe640 1 -- 172.21.15.57:0/2211243189 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff9740073c0 con 0x7ff98c0c82d0 2023-10-30T14:30:27.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.468+0000 7ff9837fe640 1 --2- 172.21.15.57:0/2211243189 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff96c0664a0 0x7ff96c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:27.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.468+0000 7ff9837fe640 1 -- 172.21.15.57:0/2211243189 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7ff974083470 con 0x7ff98c0c82d0 2023-10-30T14:30:27.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.468+0000 7ff98affd640 1 --2- 172.21.15.57:0/2211243189 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff96c0664a0 0x7ff96c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:27.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.472+0000 7ff98affd640 1 --2- 172.21.15.57:0/2211243189 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff96c0664a0 0x7ff96c068960 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009cf0 tx=0x7ff97c009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:27.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.472+0000 7ff9837fe640 1 -- 172.21.15.57:0/2211243189 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff9740504a0 con 0x7ff98c0c82d0 2023-10-30T14:30:27.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.676+0000 7ff991291640 1 -- 172.21.15.57:0/2211243189 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff98c1f5fc0 con 0x7ff98c0c82d0 2023-10-30T14:30:27.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.676+0000 7ff9837fe640 1 -- 172.21.15.57:0/2211243189 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5740 (secure 0 0 0) 0x7ff98c1f5fc0 con 0x7ff98c0c82d0 2023-10-30T14:30:27.681 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:27.681 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:30:23.995194+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:27.682 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 101 2023-10-30T14:30:27.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.680+0000 7ff991291640 1 -- 172.21.15.57:0/2211243189 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff96c0664a0 msgr2=0x7ff96c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:27.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.680+0000 7ff991291640 1 --2- 172.21.15.57:0/2211243189 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff96c0664a0 0x7ff96c068960 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009cf0 tx=0x7ff97c009190 comp rx=0 tx=0).stop 2023-10-30T14:30:27.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.680+0000 7ff991291640 1 -- 172.21.15.57:0/2211243189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff98c0c82d0 msgr2=0x7ff98c144a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:27.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.680+0000 7ff991291640 1 --2- 172.21.15.57:0/2211243189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff98c0c82d0 0x7ff98c144a00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7ff97400c370 tx=0x7ff97400c830 comp rx=0 tx=0).stop 2023-10-30T14:30:27.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.680+0000 7ff991291640 1 -- 172.21.15.57:0/2211243189 shutdown_connections 2023-10-30T14:30:27.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.680+0000 7ff991291640 1 --2- 172.21.15.57:0/2211243189 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff96c0664a0 0x7ff96c068960 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:27.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.680+0000 7ff991291640 1 --2- 172.21.15.57:0/2211243189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff98c1516c0 0x7ff98c146d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:27.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.680+0000 7ff991291640 1 --2- 172.21.15.57:0/2211243189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff98c0c82d0 0x7ff98c144a00 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:27.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.680+0000 7ff991291640 1 --2- 172.21.15.57:0/2211243189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff98c0c2500 0x7ff98c1f5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:27.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.680+0000 7ff991291640 1 -- 172.21.15.57:0/2211243189 >> 172.21.15.57:0/2211243189 conn(0x7ff98c071fb0 msgr2=0x7ff98c072390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:27.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.680+0000 7ff991291640 1 -- 172.21.15.57:0/2211243189 shutdown_connections 2023-10-30T14:30:27.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:27.680+0000 7ff991291640 1 -- 172.21.15.57:0/2211243189 wait complete. 2023-10-30T14:30:27.702 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:30:29.703 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:29.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.856+0000 7f712530f640 1 Processor -- start 2023-10-30T14:30:29.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.856+0000 7f712530f640 1 -- start start 2023-10-30T14:30:29.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.860+0000 7f712530f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7120150830 0x7f712006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:29.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.860+0000 7f712530f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f71200cb7d0 0x7f71200c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:29.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.860+0000 7f712530f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71200c2310 con 0x7f71200c3e40 2023-10-30T14:30:29.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.860+0000 7f712530f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7120156710 con 0x7f71200cb7d0 2023-10-30T14:30:29.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.860+0000 7f712530f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71200c4360 con 0x7f7120150830 2023-10-30T14:30:29.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.860+0000 7f711e7fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f71200cb7d0 0x7f71200c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:29.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.860+0000 7f711f7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f71200c3e40 0x7f7120070520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46798/0 (socket says 172.21.15.57:46798) 2023-10-30T14:30:29.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.860+0000 7f711e7fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f71200cb7d0 0x7f71200c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35332/0 (socket says 172.21.15.57:35332) 2023-10-30T14:30:29.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.860+0000 7f711f7fe640 1 -- 172.21.15.57:0/4070293245 learned_addr learned my addr 172.21.15.57:0/4070293245 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:30:29.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.860+0000 7f711effd640 1 --2- 172.21.15.57:0/4070293245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7120150830 0x7f712006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:29.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.860+0000 7f711e7fc640 1 -- 172.21.15.57:0/4070293245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7120150830 msgr2=0x7f712006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:29.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.860+0000 7f711e7fc640 1 --2- 172.21.15.57:0/4070293245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7120150830 0x7f712006b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:29.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.860+0000 7f711e7fc640 1 -- 172.21.15.57:0/4070293245 >> v1:172.21.15.57:6789/0 conn(0x7f71200c3e40 legacy=0x7f7120070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:29.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.860+0000 7f711e7fc640 1 -- 172.21.15.57:0/4070293245 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71201569b0 con 0x7f71200cb7d0 2023-10-30T14:30:29.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.860+0000 7f711effd640 1 --2- 172.21.15.57:0/4070293245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7120150830 0x7f712006b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:30:29.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.860+0000 7f711dffb640 1 -- 172.21.15.57:0/4070293245 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462940986 0 0) 0x7f71200c2310 con 0x7f71200c3e40 2023-10-30T14:30:29.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.860+0000 7f711e7fc640 1 --2- 172.21.15.57:0/4070293245 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71200cb7d0 0x7f71200c3900 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f71140097c0 tx=0x7f711401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f1c99e1010cef8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:29.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.864+0000 7f711dffb640 1 -- 172.21.15.57:0/4070293245 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71140a8970 con 0x7f71200cb7d0 2023-10-30T14:30:29.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.864+0000 7f711dffb640 1 -- 172.21.15.57:0/4070293245 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71140a8ad0 con 0x7f71200cb7d0 2023-10-30T14:30:29.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.864+0000 7f711dffb640 1 -- 172.21.15.57:0/4070293245 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71140a8e60 con 0x7f71200cb7d0 2023-10-30T14:30:29.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.864+0000 7f712530f640 1 -- 172.21.15.57:0/4070293245 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71200cb7d0 msgr2=0x7f71200c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:29.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.864+0000 7f712530f640 1 --2- 172.21.15.57:0/4070293245 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71200cb7d0 0x7f71200c3900 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f71140097c0 tx=0x7f711401ec70 comp rx=0 tx=0).stop 2023-10-30T14:30:29.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.864+0000 7f712530f640 1 -- 172.21.15.57:0/4070293245 shutdown_connections 2023-10-30T14:30:29.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.864+0000 7f712530f640 1 --2- 172.21.15.57:0/4070293245 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71200cb7d0 0x7f71200c3900 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:29.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.864+0000 7f712530f640 1 --2- 172.21.15.57:0/4070293245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7120150830 0x7f712006b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:29.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.864+0000 7f712530f640 1 -- 172.21.15.57:0/4070293245 >> 172.21.15.57:0/4070293245 conn(0x7f71200cac70 msgr2=0x7f71200cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:29.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.864+0000 7f712530f640 1 -- 172.21.15.57:0/4070293245 shutdown_connections 2023-10-30T14:30:29.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.864+0000 7f712530f640 1 -- 172.21.15.57:0/4070293245 wait complete. 2023-10-30T14:30:29.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.864+0000 7f712530f640 1 Processor -- start 2023-10-30T14:30:29.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.864+0000 7f712530f640 1 -- start start 2023-10-30T14:30:29.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f712530f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71200c3e40 0x7f7120155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:29.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f712530f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71200cb7d0 0x7f71201efd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:29.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f712530f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7120150830 0x7f71201f2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:29.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f712530f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71200b9e40 con 0x7f71200cb7d0 2023-10-30T14:30:29.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f712530f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71200b8b60 con 0x7f71200c3e40 2023-10-30T14:30:29.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f712530f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71201f2d90 con 0x7f7120150830 2023-10-30T14:30:29.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f711e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71200cb7d0 0x7f71201efd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:29.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f711e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71200cb7d0 0x7f71201efd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35334/0 (socket says 172.21.15.57:35334) 2023-10-30T14:30:29.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f711e7fc640 1 -- 172.21.15.57:0/4290161120 learned_addr learned my addr 172.21.15.57:0/4290161120 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:29.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f711effd640 1 --2- 172.21.15.57:0/4290161120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71200c3e40 0x7f7120155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:29.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f711f7fe640 1 --2- 172.21.15.57:0/4290161120 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7120150830 0x7f71201f2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:29.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f711e7fc640 1 -- 172.21.15.57:0/4290161120 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7120150830 msgr2=0x7f71201f2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:29.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f711e7fc640 1 --2- 172.21.15.57:0/4290161120 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7120150830 0x7f71201f2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:29.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f711e7fc640 1 -- 172.21.15.57:0/4290161120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71200c3e40 msgr2=0x7f7120155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:29.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f711e7fc640 1 --2- 172.21.15.57:0/4290161120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71200c3e40 0x7f7120155320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:29.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f711e7fc640 1 -- 172.21.15.57:0/4290161120 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7114009420 con 0x7f71200cb7d0 2023-10-30T14:30:29.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f711effd640 1 --2- 172.21.15.57:0/4290161120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71200c3e40 0x7f7120155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:30:29.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f711e7fc640 1 --2- 172.21.15.57:0/4290161120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71200cb7d0 0x7f71201efd10 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f71140097c0 tx=0x7f711401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:29.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f71077fe640 1 -- 172.21.15.57:0/4290161120 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71140b9630 con 0x7f71200cb7d0 2023-10-30T14:30:29.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f71077fe640 1 -- 172.21.15.57:0/4290161120 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71140b97b0 con 0x7f71200cb7d0 2023-10-30T14:30:29.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f71077fe640 1 -- 172.21.15.57:0/4290161120 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71140b9b60 con 0x7f71200cb7d0 2023-10-30T14:30:29.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f712530f640 1 -- 172.21.15.57:0/4290161120 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71200b8de0 con 0x7f71200cb7d0 2023-10-30T14:30:29.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f712530f640 1 -- 172.21.15.57:0/4290161120 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71200c6220 con 0x7f71200cb7d0 2023-10-30T14:30:29.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f712530f640 1 -- 172.21.15.57:0/4290161120 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7120066510 con 0x7f71200cb7d0 2023-10-30T14:30:29.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.868+0000 7f71077fe640 1 -- 172.21.15.57:0/4290161120 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f71140b7040 con 0x7f71200cb7d0 2023-10-30T14:30:29.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.872+0000 7f71077fe640 1 --2- 172.21.15.57:0/4290161120 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f710c0664a0 0x7f710c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:29.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.872+0000 7f71077fe640 1 -- 172.21.15.57:0/4290161120 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f7114122440 con 0x7f71200cb7d0 2023-10-30T14:30:29.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.872+0000 7f711effd640 1 --2- 172.21.15.57:0/4290161120 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f710c0664a0 0x7f710c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:29.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.872+0000 7f711effd640 1 --2- 172.21.15.57:0/4290161120 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f710c0664a0 0x7f710c068960 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f711000b990 tx=0x7f7110008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:29.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:29.876+0000 7f71077fe640 1 -- 172.21.15.57:0/4290161120 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f71140ef4f0 con 0x7f71200cb7d0 2023-10-30T14:30:30.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:30.064+0000 7f712530f640 1 -- 172.21.15.57:0/4290161120 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f712004e170 con 0x7f71200cb7d0 2023-10-30T14:30:30.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:30.064+0000 7f71077fe640 1 -- 172.21.15.57:0/4290161120 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5740 (secure 0 0 0) 0x7f71140b5090 con 0x7f71200cb7d0 2023-10-30T14:30:30.068 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:30.068 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:30:23.995194+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:30.069 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 101 2023-10-30T14:30:30.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:30.068+0000 7f712530f640 1 -- 172.21.15.57:0/4290161120 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f710c0664a0 msgr2=0x7f710c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:30.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:30.068+0000 7f712530f640 1 --2- 172.21.15.57:0/4290161120 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f710c0664a0 0x7f710c068960 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f711000b990 tx=0x7f7110008040 comp rx=0 tx=0).stop 2023-10-30T14:30:30.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:30.068+0000 7f712530f640 1 -- 172.21.15.57:0/4290161120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71200cb7d0 msgr2=0x7f71201efd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:30.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:30.068+0000 7f712530f640 1 --2- 172.21.15.57:0/4290161120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71200cb7d0 0x7f71201efd10 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f71140097c0 tx=0x7f711401ba10 comp rx=0 tx=0).stop 2023-10-30T14:30:30.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:30.068+0000 7f712530f640 1 -- 172.21.15.57:0/4290161120 shutdown_connections 2023-10-30T14:30:30.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:30.068+0000 7f712530f640 1 --2- 172.21.15.57:0/4290161120 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f710c0664a0 0x7f710c068960 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:30.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:30.068+0000 7f712530f640 1 --2- 172.21.15.57:0/4290161120 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7120150830 0x7f71201f2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:30.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:30.068+0000 7f712530f640 1 --2- 172.21.15.57:0/4290161120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71200cb7d0 0x7f71201efd10 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:30.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:30.068+0000 7f712530f640 1 --2- 172.21.15.57:0/4290161120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71200c3e40 0x7f7120155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:30.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:30.068+0000 7f712530f640 1 -- 172.21.15.57:0/4290161120 >> 172.21.15.57:0/4290161120 conn(0x7f71200cac70 msgr2=0x7f7120070520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:30.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:30.068+0000 7f712530f640 1 -- 172.21.15.57:0/4290161120 shutdown_connections 2023-10-30T14:30:30.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:30.068+0000 7f712530f640 1 -- 172.21.15.57:0/4290161120 wait complete. 2023-10-30T14:30:30.089 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:30:32.090 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:32.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.244+0000 7fc097c9c640 1 Processor -- start 2023-10-30T14:30:32.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.244+0000 7fc097c9c640 1 -- start start 2023-10-30T14:30:32.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.244+0000 7fc097c9c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc090153c80 0x7fc09015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:32.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.244+0000 7fc097c9c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc0900c4a10 0x7fc0900c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:32.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.244+0000 7fc097c9c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc090159f20 con 0x7fc09015a3b0 2023-10-30T14:30:32.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.244+0000 7fc097c9c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc09015dec0 con 0x7fc090153c80 2023-10-30T14:30:32.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.244+0000 7fc097c9c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0901599a0 con 0x7fc0900c4a10 2023-10-30T14:30:32.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.244+0000 7fc095a38640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc090153c80 0x7fc09015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:32.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.244+0000 7fc096239640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc09015a3b0 0x7fc0900ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46800/0 (socket says 172.21.15.57:46800) 2023-10-30T14:30:32.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.244+0000 7fc096239640 1 -- 172.21.15.57:0/3945361428 learned_addr learned my addr 172.21.15.57:0/3945361428 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:30:32.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.244+0000 7fc095237640 1 --2- 172.21.15.57:0/3945361428 >> v2:172.21.15.88:3300/0 conn(0x7fc0900c4a10 0x7fc0900c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:32.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.244+0000 7fc095a38640 1 -- 172.21.15.57:0/3945361428 >> v2:172.21.15.88:3300/0 conn(0x7fc0900c4a10 msgr2=0x7fc0900c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:32.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.244+0000 7fc095a38640 1 --2- 172.21.15.57:0/3945361428 >> v2:172.21.15.88:3300/0 conn(0x7fc0900c4a10 0x7fc0900c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:32.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.244+0000 7fc095a38640 1 -- 172.21.15.57:0/3945361428 >> v1:172.21.15.57:6789/0 conn(0x7fc09015a3b0 legacy=0x7fc0900ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:32.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.244+0000 7fc094a36640 1 -- 172.21.15.57:0/3945361428 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070467655 0 0) 0x7fc090159f20 con 0x7fc09015a3b0 2023-10-30T14:30:32.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.244+0000 7fc095a38640 1 -- 172.21.15.57:0/3945361428 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0900c47e0 con 0x7fc090153c80 2023-10-30T14:30:32.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.244+0000 7fc095237640 1 --2- 172.21.15.57:0/3945361428 >> v2:172.21.15.88:3300/0 conn(0x7fc0900c4a10 0x7fc0900c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:30:32.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.248+0000 7fc095a38640 1 --2- 172.21.15.57:0/3945361428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc090153c80 0x7fc09015cf90 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fc09006be50 tx=0x7fc08c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b0cd7f144619d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:32.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.248+0000 7fc094a36640 1 -- 172.21.15.57:0/3945361428 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c0a8950 con 0x7fc090153c80 2023-10-30T14:30:32.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.248+0000 7fc094a36640 1 -- 172.21.15.57:0/3945361428 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08c0a8ab0 con 0x7fc090153c80 2023-10-30T14:30:32.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.248+0000 7fc094a36640 1 -- 172.21.15.57:0/3945361428 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c0a8e40 con 0x7fc090153c80 2023-10-30T14:30:32.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.248+0000 7fc097c9c640 1 -- 172.21.15.57:0/3945361428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc090153c80 msgr2=0x7fc09015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:32.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.248+0000 7fc097c9c640 1 --2- 172.21.15.57:0/3945361428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc090153c80 0x7fc09015cf90 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fc09006be50 tx=0x7fc08c01ec50 comp rx=0 tx=0).stop 2023-10-30T14:30:32.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.248+0000 7fc097c9c640 1 -- 172.21.15.57:0/3945361428 shutdown_connections 2023-10-30T14:30:32.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.248+0000 7fc097c9c640 1 --2- 172.21.15.57:0/3945361428 >> v2:172.21.15.88:3300/0 conn(0x7fc0900c4a10 0x7fc0900c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:32.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.248+0000 7fc097c9c640 1 --2- 172.21.15.57:0/3945361428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc090153c80 0x7fc09015cf90 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:32.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.248+0000 7fc097c9c640 1 -- 172.21.15.57:0/3945361428 >> 172.21.15.57:0/3945361428 conn(0x7fc0900715c0 msgr2=0x7fc0900719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:32.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.248+0000 7fc097c9c640 1 -- 172.21.15.57:0/3945361428 shutdown_connections 2023-10-30T14:30:32.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.248+0000 7fc097c9c640 1 -- 172.21.15.57:0/3945361428 wait complete. 2023-10-30T14:30:32.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc097c9c640 1 Processor -- start 2023-10-30T14:30:32.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc097c9c640 1 -- start start 2023-10-30T14:30:32.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc097c9c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0900c4a10 0x7fc0901529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:32.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc097c9c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc090153c80 0x7fc090154160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:32.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc097c9c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc09015a3b0 0x7fc090148e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:32.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc097c9c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09015d280 con 0x7fc09015a3b0 2023-10-30T14:30:32.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc097c9c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc090154720 con 0x7fc0900c4a10 2023-10-30T14:30:32.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc097c9c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc090154a00 con 0x7fc090153c80 2023-10-30T14:30:32.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc096239640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc09015a3b0 0x7fc090148e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:32.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc096239640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc09015a3b0 0x7fc090148e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35350/0 (socket says 172.21.15.57:35350) 2023-10-30T14:30:32.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc096239640 1 -- 172.21.15.57:0/4155281669 learned_addr learned my addr 172.21.15.57:0/4155281669 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:32.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc095a38640 1 --2- 172.21.15.57:0/4155281669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0900c4a10 0x7fc0901529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:32.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc095237640 1 --2- 172.21.15.57:0/4155281669 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc090153c80 0x7fc090154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:32.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc096239640 1 -- 172.21.15.57:0/4155281669 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc090153c80 msgr2=0x7fc090154160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:32.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc096239640 1 --2- 172.21.15.57:0/4155281669 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc090153c80 0x7fc090154160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:32.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc096239640 1 -- 172.21.15.57:0/4155281669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0900c4a10 msgr2=0x7fc0901529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:32.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc096239640 1 --2- 172.21.15.57:0/4155281669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0900c4a10 0x7fc0901529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:32.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc096239640 1 -- 172.21.15.57:0/4155281669 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c009420 con 0x7fc09015a3b0 2023-10-30T14:30:32.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc096239640 1 --2- 172.21.15.57:0/4155281669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc09015a3b0 0x7fc090148e20 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fc08800c7b0 tx=0x7fc08800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:32.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc0867fc640 1 -- 172.21.15.57:0/4155281669 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08800a640 con 0x7fc09015a3b0 2023-10-30T14:30:32.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc0867fc640 1 -- 172.21.15.57:0/4155281669 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08800a7c0 con 0x7fc09015a3b0 2023-10-30T14:30:32.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc0867fc640 1 -- 172.21.15.57:0/4155281669 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08800ab70 con 0x7fc09015a3b0 2023-10-30T14:30:32.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc097c9c640 1 -- 172.21.15.57:0/4155281669 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc090156ff0 con 0x7fc09015a3b0 2023-10-30T14:30:32.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc097c9c640 1 -- 172.21.15.57:0/4155281669 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0901ef670 con 0x7fc09015a3b0 2023-10-30T14:30:32.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.252+0000 7fc097c9c640 1 -- 172.21.15.57:0/4155281669 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09004e170 con 0x7fc09015a3b0 2023-10-30T14:30:32.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.256+0000 7fc0867fc640 1 -- 172.21.15.57:0/4155281669 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc08800acd0 con 0x7fc09015a3b0 2023-10-30T14:30:32.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.256+0000 7fc0867fc640 1 --2- 172.21.15.57:0/4155281669 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0740664d0 0x7fc074068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:32.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.256+0000 7fc0867fc640 1 -- 172.21.15.57:0/4155281669 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7fc0880835e0 con 0x7fc09015a3b0 2023-10-30T14:30:32.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.256+0000 7fc095a38640 1 --2- 172.21.15.57:0/4155281669 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0740664d0 0x7fc074068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:32.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.256+0000 7fc095a38640 1 --2- 172.21.15.57:0/4155281669 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0740664d0 0x7fc074068990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc08c0b2000 tx=0x7fc08c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:32.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.260+0000 7fc0867fc640 1 -- 172.21.15.57:0/4155281669 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc088050720 con 0x7fc09015a3b0 2023-10-30T14:30:32.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.452+0000 7fc097c9c640 1 -- 172.21.15.57:0/4155281669 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc090154b40 con 0x7fc09015a3b0 2023-10-30T14:30:32.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.452+0000 7fc0867fc640 1 -- 172.21.15.57:0/4155281669 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5737 (secure 0 0 0) 0x7fc088053d40 con 0x7fc09015a3b0 2023-10-30T14:30:32.455 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:32.455 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:30:29.961962+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:32.456 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 102 2023-10-30T14:30:32.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.452+0000 7fc097c9c640 1 -- 172.21.15.57:0/4155281669 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0740664d0 msgr2=0x7fc074068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:32.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.452+0000 7fc097c9c640 1 --2- 172.21.15.57:0/4155281669 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0740664d0 0x7fc074068990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc08c0b2000 tx=0x7fc08c008040 comp rx=0 tx=0).stop 2023-10-30T14:30:32.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.452+0000 7fc097c9c640 1 -- 172.21.15.57:0/4155281669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc09015a3b0 msgr2=0x7fc090148e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:32.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.452+0000 7fc097c9c640 1 --2- 172.21.15.57:0/4155281669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc09015a3b0 0x7fc090148e20 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fc08800c7b0 tx=0x7fc08800cc70 comp rx=0 tx=0).stop 2023-10-30T14:30:32.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.452+0000 7fc097c9c640 1 -- 172.21.15.57:0/4155281669 shutdown_connections 2023-10-30T14:30:32.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.452+0000 7fc097c9c640 1 --2- 172.21.15.57:0/4155281669 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0740664d0 0x7fc074068990 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:32.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.452+0000 7fc097c9c640 1 --2- 172.21.15.57:0/4155281669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc09015a3b0 0x7fc090148e20 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:32.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.452+0000 7fc097c9c640 1 --2- 172.21.15.57:0/4155281669 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc090153c80 0x7fc090154160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:32.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.452+0000 7fc097c9c640 1 --2- 172.21.15.57:0/4155281669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0900c4a10 0x7fc0901529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:32.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.452+0000 7fc097c9c640 1 -- 172.21.15.57:0/4155281669 >> 172.21.15.57:0/4155281669 conn(0x7fc0900715c0 msgr2=0x7fc090149e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:32.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.452+0000 7fc097c9c640 1 -- 172.21.15.57:0/4155281669 shutdown_connections 2023-10-30T14:30:32.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:32.452+0000 7fc097c9c640 1 -- 172.21.15.57:0/4155281669 wait complete. 2023-10-30T14:30:32.475 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:30:34.476 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:34.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.628+0000 7fa00c33e640 1 Processor -- start 2023-10-30T14:30:34.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.628+0000 7fa00c33e640 1 -- start start 2023-10-30T14:30:34.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.628+0000 7fa00c33e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa004069e90 0x7fa00406f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:34.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.628+0000 7fa00c33e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa0040c9c40 0x7fa00406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:34.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.628+0000 7fa00c33e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0040c8440 con 0x7fa0041597d0 2023-10-30T14:30:34.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.628+0000 7fa00c33e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa00406a7b0 con 0x7fa0040c9c40 2023-10-30T14:30:34.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.628+0000 7fa00c33e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa004153f60 con 0x7fa004069e90 2023-10-30T14:30:34.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.632+0000 7fa00a0da640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa004069e90 0x7fa00406f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:34.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.632+0000 7fa0098d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa0040c9c40 0x7fa00406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:34.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.632+0000 7fa00a0da640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa004069e90 0x7fa00406f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:52202/0 (socket says 172.21.15.57:52202) 2023-10-30T14:30:34.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.632+0000 7fa00a0da640 1 -- 172.21.15.57:0/70505540 learned_addr learned my addr 172.21.15.57:0/70505540 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:34.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.632+0000 7fa00a0da640 1 -- 172.21.15.57:0/70505540 >> v2:172.21.15.88:3300/0 conn(0x7fa0040c9c40 msgr2=0x7fa00406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:34.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.632+0000 7fa00a0da640 1 --2- 172.21.15.57:0/70505540 >> v2:172.21.15.88:3300/0 conn(0x7fa0040c9c40 0x7fa00406a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:34.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.632+0000 7fa00a0da640 1 -- 172.21.15.57:0/70505540 >> v1:172.21.15.88:6789/0 conn(0x7fa0041597d0 legacy=0x7fa0040c8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:34.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.632+0000 7fa00a0da640 1 -- 172.21.15.57:0/70505540 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0040c89f0 con 0x7fa004069e90 2023-10-30T14:30:34.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.632+0000 7fa00a0da640 1 --2- 172.21.15.57:0/70505540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa004069e90 0x7fa00406f1d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa000009870 tx=0x7fa00001c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ac4b87847440dfa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:34.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa0090d8640 1 -- 172.21.15.57:0/70505540 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0000ab600 con 0x7fa004069e90 2023-10-30T14:30:34.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa0090d8640 1 -- 172.21.15.57:0/70505540 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0000a7050 con 0x7fa004069e90 2023-10-30T14:30:34.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa0090d8640 1 -- 172.21.15.57:0/70505540 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0000ab980 con 0x7fa004069e90 2023-10-30T14:30:34.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa00c33e640 1 -- 172.21.15.57:0/70505540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa004069e90 msgr2=0x7fa00406f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:34.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa00c33e640 1 --2- 172.21.15.57:0/70505540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa004069e90 0x7fa00406f1d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa000009870 tx=0x7fa00001c540 comp rx=0 tx=0).stop 2023-10-30T14:30:34.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa00c33e640 1 -- 172.21.15.57:0/70505540 shutdown_connections 2023-10-30T14:30:34.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa00c33e640 1 --2- 172.21.15.57:0/70505540 >> v2:172.21.15.88:3300/0 conn(0x7fa0040c9c40 0x7fa00406a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:34.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa00c33e640 1 --2- 172.21.15.57:0/70505540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa004069e90 0x7fa00406f1d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:34.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa00c33e640 1 -- 172.21.15.57:0/70505540 >> 172.21.15.57:0/70505540 conn(0x7fa00406afa0 msgr2=0x7fa00406b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:34.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa00c33e640 1 -- 172.21.15.57:0/70505540 shutdown_connections 2023-10-30T14:30:34.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa00c33e640 1 -- 172.21.15.57:0/70505540 wait complete. 2023-10-30T14:30:34.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa00c33e640 1 Processor -- start 2023-10-30T14:30:34.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa00c33e640 1 -- start start 2023-10-30T14:30:34.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa00c33e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa004069e90 0x7fa004150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:34.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa00c33e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0040c9c40 0x7fa00415abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:34.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa00c33e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0041597d0 0x7fa00414aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:34.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa00c33e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00415ba70 con 0x7fa0040c9c40 2023-10-30T14:30:34.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa00c33e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa004150880 con 0x7fa004069e90 2023-10-30T14:30:34.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa00c33e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0041512c0 con 0x7fa0041597d0 2023-10-30T14:30:34.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa0098d9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0040c9c40 0x7fa00415abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa0098d9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0040c9c40 0x7fa00415abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43916/0 (socket says 172.21.15.57:43916) 2023-10-30T14:30:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa0098d9640 1 -- 172.21.15.57:0/1625697003 learned_addr learned my addr 172.21.15.57:0/1625697003 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.636+0000 7fa00a0da640 1 --2- 172.21.15.57:0/1625697003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa004069e90 0x7fa004150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.640+0000 7fa00a8db640 1 --2- 172.21.15.57:0/1625697003 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0041597d0 0x7fa00414aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.640+0000 7fa0098d9640 1 -- 172.21.15.57:0/1625697003 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0041597d0 msgr2=0x7fa00414aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.640+0000 7fa0098d9640 1 --2- 172.21.15.57:0/1625697003 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0041597d0 0x7fa00414aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.640+0000 7fa0098d9640 1 -- 172.21.15.57:0/1625697003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa004069e90 msgr2=0x7fa004150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.640+0000 7fa0098d9640 1 --2- 172.21.15.57:0/1625697003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa004069e90 0x7fa004150d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.640+0000 7fa0098d9640 1 -- 172.21.15.57:0/1625697003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0000094d0 con 0x7fa0040c9c40 2023-10-30T14:30:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.640+0000 7fa0098d9640 1 --2- 172.21.15.57:0/1625697003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0040c9c40 0x7fa00415abf0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400e870 tx=0x7f9ff400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.640+0000 7f9ffaffd640 1 -- 172.21.15.57:0/1625697003 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff4009510 con 0x7fa0040c9c40 2023-10-30T14:30:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.640+0000 7f9ffaffd640 1 -- 172.21.15.57:0/1625697003 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff400eec0 con 0x7fa0040c9c40 2023-10-30T14:30:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.640+0000 7f9ffaffd640 1 -- 172.21.15.57:0/1625697003 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff400cb40 con 0x7fa0040c9c40 2023-10-30T14:30:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.640+0000 7fa00c33e640 1 -- 172.21.15.57:0/1625697003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0041529c0 con 0x7fa0040c9c40 2023-10-30T14:30:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.640+0000 7fa00c33e640 1 -- 172.21.15.57:0/1625697003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa004152f80 con 0x7fa0040c9c40 2023-10-30T14:30:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.640+0000 7fa00c33e640 1 -- 172.21.15.57:0/1625697003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa00404e170 con 0x7fa0040c9c40 2023-10-30T14:30:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.640+0000 7f9ffaffd640 1 -- 172.21.15.57:0/1625697003 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9ff400cca0 con 0x7fa0040c9c40 2023-10-30T14:30:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.640+0000 7f9ffaffd640 1 --2- 172.21.15.57:0/1625697003 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9fe80664a0 0x7f9fe8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.640+0000 7f9ffaffd640 1 -- 172.21.15.57:0/1625697003 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f9ff40842d0 con 0x7fa0040c9c40 2023-10-30T14:30:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.640+0000 7fa00a0da640 1 --2- 172.21.15.57:0/1625697003 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9fe80664a0 0x7f9fe8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.644+0000 7fa00a0da640 1 --2- 172.21.15.57:0/1625697003 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9fe80664a0 0x7f9fe8068960 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa00406f7f0 tx=0x7fa000008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.644+0000 7f9ffaffd640 1 -- 172.21.15.57:0/1625697003 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9ff4051430 con 0x7fa0040c9c40 2023-10-30T14:30:34.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.836+0000 7fa00c33e640 1 -- 172.21.15.57:0/1625697003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0040c8650 con 0x7fa0040c9c40 2023-10-30T14:30:34.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.836+0000 7f9ffaffd640 1 -- 172.21.15.57:0/1625697003 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5737 (secure 0 0 0) 0x7f9ff4054a50 con 0x7fa0040c9c40 2023-10-30T14:30:34.841 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:34.841 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:30:29.961962+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:34.841 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 102 2023-10-30T14:30:34.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.840+0000 7fa00c33e640 1 -- 172.21.15.57:0/1625697003 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9fe80664a0 msgr2=0x7f9fe8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:34.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.840+0000 7fa00c33e640 1 --2- 172.21.15.57:0/1625697003 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9fe80664a0 0x7f9fe8068960 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa00406f7f0 tx=0x7fa000008040 comp rx=0 tx=0).stop 2023-10-30T14:30:34.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.840+0000 7fa00c33e640 1 -- 172.21.15.57:0/1625697003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0040c9c40 msgr2=0x7fa00415abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:34.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.840+0000 7fa00c33e640 1 --2- 172.21.15.57:0/1625697003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0040c9c40 0x7fa00415abf0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400e870 tx=0x7f9ff400ec70 comp rx=0 tx=0).stop 2023-10-30T14:30:34.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.840+0000 7fa00c33e640 1 -- 172.21.15.57:0/1625697003 shutdown_connections 2023-10-30T14:30:34.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.840+0000 7fa00c33e640 1 --2- 172.21.15.57:0/1625697003 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9fe80664a0 0x7f9fe8068960 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:34.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.840+0000 7fa00c33e640 1 --2- 172.21.15.57:0/1625697003 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0041597d0 0x7fa00414aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:34.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.840+0000 7fa00c33e640 1 --2- 172.21.15.57:0/1625697003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0040c9c40 0x7fa00415abf0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:34.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.840+0000 7fa00c33e640 1 --2- 172.21.15.57:0/1625697003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa004069e90 0x7fa004150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:34.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.840+0000 7fa00c33e640 1 -- 172.21.15.57:0/1625697003 >> 172.21.15.57:0/1625697003 conn(0x7fa00406afa0 msgr2=0x7fa004071780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:34.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.840+0000 7fa00c33e640 1 -- 172.21.15.57:0/1625697003 shutdown_connections 2023-10-30T14:30:34.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:34.840+0000 7fa00c33e640 1 -- 172.21.15.57:0/1625697003 wait complete. 2023-10-30T14:30:34.862 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 102, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 102, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6027": { "addr": "172.21.15.57:6836/2733323524", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2733323524, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2733323524, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6027, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 24 }, "gid_6204": { "addr": "172.21.15.57:6837/1843528309", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 1843528309, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 1843528309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6204, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_6414": { "addr": "172.21.15.57:6834/2321408149", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2321408149, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2321408149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6414, "incarnation": 99, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:30:29.961962+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6414, "mds_1": 6027, "mds_2": 6204 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:30:35.484 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:30:35.488 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:30:36.863 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:37.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.012+0000 7f0608b56640 1 Processor -- start 2023-10-30T14:30:37.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.012+0000 7f0608b56640 1 -- start start 2023-10-30T14:30:37.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.012+0000 7f0608b56640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0604153b50 0x7f06041597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:37.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.012+0000 7f0608b56640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f06040c62b0 0x7f06040ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:37.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.012+0000 7f0608b56640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f060415a770 con 0x7f06040cb180 2023-10-30T14:30:37.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.012+0000 7f0608b56640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0604159cf0 con 0x7f06040c62b0 2023-10-30T14:30:37.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.012+0000 7f0608b56640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f060406aa30 con 0x7f0604153b50 2023-10-30T14:30:37.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.012+0000 7f0602ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f06040c62b0 0x7f06040ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:37.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.012+0000 7f0603fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f06040cb180 0x7f060406ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:36708/0 (socket says 172.21.15.57:36708) 2023-10-30T14:30:37.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.012+0000 7f0603fff640 1 -- 172.21.15.57:0/2748915586 learned_addr learned my addr 172.21.15.57:0/2748915586 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:30:37.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.012+0000 7f06037fe640 1 --2- 172.21.15.57:0/2748915586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0604153b50 0x7f06041597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:37.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.012+0000 7f0602ffd640 1 -- 172.21.15.57:0/2748915586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0604153b50 msgr2=0x7f06041597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:37.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.012+0000 7f0602ffd640 1 --2- 172.21.15.57:0/2748915586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0604153b50 0x7f06041597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:37.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.012+0000 7f0602ffd640 1 -- 172.21.15.57:0/2748915586 >> v1:172.21.15.57:6789/0 conn(0x7f06040cb180 legacy=0x7f060406ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:37.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.012+0000 7f0602ffd640 1 -- 172.21.15.57:0/2748915586 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06040c9d40 con 0x7f06040c62b0 2023-10-30T14:30:37.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.012+0000 7f06027fc640 1 -- 172.21.15.57:0/2748915586 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2184191268 0 0) 0x7f05fc0014a0 con 0x7f06040cb180 2023-10-30T14:30:37.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.016+0000 7f0602ffd640 1 --2- 172.21.15.57:0/2748915586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06040c62b0 0x7f06040ca1d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f05f80097c0 tx=0x7f05f801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f9e4c0b84ad3617 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:37.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.016+0000 7f06027fc640 1 -- 172.21.15.57:0/2748915586 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f80ab600 con 0x7f06040c62b0 2023-10-30T14:30:37.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.016+0000 7f06027fc640 1 -- 172.21.15.57:0/2748915586 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f80a7050 con 0x7f06040c62b0 2023-10-30T14:30:37.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.016+0000 7f06027fc640 1 -- 172.21.15.57:0/2748915586 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f80ab980 con 0x7f06040c62b0 2023-10-30T14:30:37.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.016+0000 7f0608b56640 1 -- 172.21.15.57:0/2748915586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06040c62b0 msgr2=0x7f06040ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:37.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.016+0000 7f0608b56640 1 --2- 172.21.15.57:0/2748915586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06040c62b0 0x7f06040ca1d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f05f80097c0 tx=0x7f05f801c480 comp rx=0 tx=0).stop 2023-10-30T14:30:37.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.016+0000 7f0608b56640 1 -- 172.21.15.57:0/2748915586 shutdown_connections 2023-10-30T14:30:37.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.016+0000 7f0608b56640 1 --2- 172.21.15.57:0/2748915586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06040c62b0 0x7f06040ca1d0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:37.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.016+0000 7f0608b56640 1 --2- 172.21.15.57:0/2748915586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0604153b50 0x7f06041597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:37.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.016+0000 7f0608b56640 1 -- 172.21.15.57:0/2748915586 >> 172.21.15.57:0/2748915586 conn(0x7f0604065a80 msgr2=0x7f0604065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:37.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.016+0000 7f0608b56640 1 -- 172.21.15.57:0/2748915586 shutdown_connections 2023-10-30T14:30:37.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.016+0000 7f0608b56640 1 -- 172.21.15.57:0/2748915586 wait complete. 2023-10-30T14:30:37.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.016+0000 7f0608b56640 1 Processor -- start 2023-10-30T14:30:37.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.016+0000 7f0608b56640 1 -- start start 2023-10-30T14:30:37.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0608b56640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06040c62b0 0x7f0604157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:37.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0608b56640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06040cb180 0x7f0604158330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:37.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0608b56640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0604153b50 0x7f06040cded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:37.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0608b56640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0604149400 con 0x7f0604153b50 2023-10-30T14:30:37.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0608b56640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06041574e0 con 0x7f06040c62b0 2023-10-30T14:30:37.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0608b56640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0604157960 con 0x7f06040cb180 2023-10-30T14:30:37.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0603fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0604153b50 0x7f06040cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:37.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0603fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0604153b50 0x7f06040cded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43932/0 (socket says 172.21.15.57:43932) 2023-10-30T14:30:37.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0603fff640 1 -- 172.21.15.57:0/1939400258 learned_addr learned my addr 172.21.15.57:0/1939400258 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:37.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f06037fe640 1 --2- 172.21.15.57:0/1939400258 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06040c62b0 0x7f0604157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:37.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0602ffd640 1 --2- 172.21.15.57:0/1939400258 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06040cb180 0x7f0604158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:37.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0603fff640 1 -- 172.21.15.57:0/1939400258 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06040cb180 msgr2=0x7f0604158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:37.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0603fff640 1 --2- 172.21.15.57:0/1939400258 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06040cb180 0x7f0604158330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:37.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0603fff640 1 -- 172.21.15.57:0/1939400258 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06040c62b0 msgr2=0x7f0604157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:37.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0603fff640 1 --2- 172.21.15.57:0/1939400258 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06040c62b0 0x7f0604157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:37.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0603fff640 1 -- 172.21.15.57:0/1939400258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f8009420 con 0x7f0604153b50 2023-10-30T14:30:37.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0602ffd640 1 --2- 172.21.15.57:0/1939400258 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06040cb180 0x7f0604158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:30:37.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0603fff640 1 --2- 172.21.15.57:0/1939400258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0604153b50 0x7f06040cded0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00c7d0 tx=0x7f05fc00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:37.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f05ebfff640 1 -- 172.21.15.57:0/1939400258 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05fc00a5e0 con 0x7f0604153b50 2023-10-30T14:30:37.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f05ebfff640 1 -- 172.21.15.57:0/1939400258 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05fc00a740 con 0x7f0604153b50 2023-10-30T14:30:37.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f05ebfff640 1 -- 172.21.15.57:0/1939400258 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05fc00aaf0 con 0x7f0604153b50 2023-10-30T14:30:37.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0608b56640 1 -- 172.21.15.57:0/1939400258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0604143c80 con 0x7f0604153b50 2023-10-30T14:30:37.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0608b56640 1 -- 172.21.15.57:0/1939400258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06041433e0 con 0x7f0604153b50 2023-10-30T14:30:37.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f0608b56640 1 -- 172.21.15.57:0/1939400258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0604150010 con 0x7f0604153b50 2023-10-30T14:30:37.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.020+0000 7f05ebfff640 1 -- 172.21.15.57:0/1939400258 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f05fc00ac50 con 0x7f0604153b50 2023-10-30T14:30:37.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.024+0000 7f05ebfff640 1 --2- 172.21.15.57:0/1939400258 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05f00664a0 0x7f05f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:37.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.024+0000 7f05ebfff640 1 -- 172.21.15.57:0/1939400258 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f05fc083510 con 0x7f0604153b50 2023-10-30T14:30:37.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.024+0000 7f06037fe640 1 --2- 172.21.15.57:0/1939400258 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05f00664a0 0x7f05f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:37.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.024+0000 7f06037fe640 1 --2- 172.21.15.57:0/1939400258 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05f00664a0 0x7f05f0068960 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f05f400b930 tx=0x7f05f4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:37.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.028+0000 7f05ebfff640 1 -- 172.21.15.57:0/1939400258 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f05fc0505b0 con 0x7f0604153b50 2023-10-30T14:30:37.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.224+0000 7f0608b56640 1 -- 172.21.15.57:0/1939400258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f060415a770 con 0x7f0604153b50 2023-10-30T14:30:37.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.224+0000 7f05ebfff640 1 -- 172.21.15.57:0/1939400258 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5743 (secure 0 0 0) 0x7f060415a770 con 0x7f0604153b50 2023-10-30T14:30:37.228 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:37.228 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:30:35.996360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:clientreplay","state_seq":16,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:37.230 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 103 2023-10-30T14:30:37.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.228+0000 7f0608b56640 1 -- 172.21.15.57:0/1939400258 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05f00664a0 msgr2=0x7f05f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:37.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.228+0000 7f0608b56640 1 --2- 172.21.15.57:0/1939400258 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05f00664a0 0x7f05f0068960 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f05f400b930 tx=0x7f05f4008040 comp rx=0 tx=0).stop 2023-10-30T14:30:37.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.228+0000 7f0608b56640 1 -- 172.21.15.57:0/1939400258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0604153b50 msgr2=0x7f06040cded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:37.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.228+0000 7f0608b56640 1 --2- 172.21.15.57:0/1939400258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0604153b50 0x7f06040cded0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00c7d0 tx=0x7f05fc00cc90 comp rx=0 tx=0).stop 2023-10-30T14:30:37.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.228+0000 7f0608b56640 1 -- 172.21.15.57:0/1939400258 shutdown_connections 2023-10-30T14:30:37.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.228+0000 7f0608b56640 1 --2- 172.21.15.57:0/1939400258 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05f00664a0 0x7f05f0068960 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:37.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.228+0000 7f0608b56640 1 --2- 172.21.15.57:0/1939400258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0604153b50 0x7f06040cded0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:37.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.228+0000 7f0608b56640 1 --2- 172.21.15.57:0/1939400258 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06040cb180 0x7f0604158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:37.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.228+0000 7f0608b56640 1 --2- 172.21.15.57:0/1939400258 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06040c62b0 0x7f0604157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:37.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.228+0000 7f0608b56640 1 -- 172.21.15.57:0/1939400258 >> 172.21.15.57:0/1939400258 conn(0x7f0604065a80 msgr2=0x7f0604069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:37.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.228+0000 7f0608b56640 1 -- 172.21.15.57:0/1939400258 shutdown_connections 2023-10-30T14:30:37.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:37.228+0000 7f0608b56640 1 -- 172.21.15.57:0/1939400258 wait complete. 2023-10-30T14:30:37.249 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:30:39.250 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:39.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.404+0000 7fc7ba688640 1 Processor -- start 2023-10-30T14:30:39.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.404+0000 7fc7ba688640 1 -- start start 2023-10-30T14:30:39.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.404+0000 7fc7ba688640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7b4153b50 0x7fc7b41597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:39.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.404+0000 7fc7ba688640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc7b40cb180 0x7fc7b406ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:39.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.404+0000 7fc7ba688640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b415a770 con 0x7fc7b40c62b0 2023-10-30T14:30:39.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.404+0000 7fc7ba688640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7b4159cf0 con 0x7fc7b40cb180 2023-10-30T14:30:39.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.404+0000 7fc7ba688640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b406aa30 con 0x7fc7b4153b50 2023-10-30T14:30:39.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.404+0000 7fc7b8e85640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc7b40c62b0 0x7fc7b40ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:36714/0 (socket says 172.21.15.57:36714) 2023-10-30T14:30:39.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.404+0000 7fc7b8e85640 1 -- 172.21.15.57:0/2324858430 learned_addr learned my addr 172.21.15.57:0/2324858430 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:30:39.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.404+0000 7fc7b9e87640 1 --2- 172.21.15.57:0/2324858430 >> v2:172.21.15.57:3300/0 conn(0x7fc7b40cb180 0x7fc7b406ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:39.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.404+0000 7fc7b9686640 1 --2- 172.21.15.57:0/2324858430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7b4153b50 0x7fc7b41597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:39.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.404+0000 7fc7abfff640 1 -- 172.21.15.57:0/2324858430 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1614475037 0 0) 0x7fc7a4001120 con 0x7fc7b40c62b0 2023-10-30T14:30:39.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.404+0000 7fc7abfff640 1 -- 172.21.15.57:0/2324858430 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a0003530 con 0x7fc7b40c62b0 2023-10-30T14:30:39.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.404+0000 7fc7b9e87640 1 -- 172.21.15.57:0/2324858430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7b4153b50 msgr2=0x7fc7b41597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:39.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.404+0000 7fc7b9e87640 1 --2- 172.21.15.57:0/2324858430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7b4153b50 0x7fc7b41597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:39.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.404+0000 7fc7b9e87640 1 -- 172.21.15.57:0/2324858430 >> v1:172.21.15.57:6789/0 conn(0x7fc7b40c62b0 legacy=0x7fc7b40ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:39.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.404+0000 7fc7b9e87640 1 -- 172.21.15.57:0/2324858430 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b40c9d40 con 0x7fc7b40cb180 2023-10-30T14:30:39.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.408+0000 7fc7b9e87640 1 --2- 172.21.15.57:0/2324858430 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b40cb180 0x7fc7b406ae00 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009d90 tx=0x7fc7ac01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6a179f4ac190c78 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:39.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.408+0000 7fc7abfff640 1 -- 172.21.15.57:0/2324858430 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac0a5810 con 0x7fc7b40cb180 2023-10-30T14:30:39.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.408+0000 7fc7abfff640 1 -- 172.21.15.57:0/2324858430 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ac0a5970 con 0x7fc7b40cb180 2023-10-30T14:30:39.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.408+0000 7fc7abfff640 1 -- 172.21.15.57:0/2324858430 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac0a5d00 con 0x7fc7b40cb180 2023-10-30T14:30:39.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.408+0000 7fc7ba688640 1 -- 172.21.15.57:0/2324858430 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b40cb180 msgr2=0x7fc7b406ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:39.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.408+0000 7fc7ba688640 1 --2- 172.21.15.57:0/2324858430 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b40cb180 0x7fc7b406ae00 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009d90 tx=0x7fc7ac01c950 comp rx=0 tx=0).stop 2023-10-30T14:30:39.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.408+0000 7fc7ba688640 1 -- 172.21.15.57:0/2324858430 shutdown_connections 2023-10-30T14:30:39.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.408+0000 7fc7ba688640 1 --2- 172.21.15.57:0/2324858430 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b40cb180 0x7fc7b406ae00 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:39.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.408+0000 7fc7ba688640 1 --2- 172.21.15.57:0/2324858430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7b4153b50 0x7fc7b41597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:39.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.408+0000 7fc7ba688640 1 -- 172.21.15.57:0/2324858430 >> 172.21.15.57:0/2324858430 conn(0x7fc7b4065a80 msgr2=0x7fc7b4065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:39.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.408+0000 7fc7ba688640 1 -- 172.21.15.57:0/2324858430 shutdown_connections 2023-10-30T14:30:39.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.408+0000 7fc7ba688640 1 -- 172.21.15.57:0/2324858430 wait complete. 2023-10-30T14:30:39.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.408+0000 7fc7ba688640 1 Processor -- start 2023-10-30T14:30:39.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7ba688640 1 -- start start 2023-10-30T14:30:39.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7ba688640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7b40c62b0 0x7fc7b4154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:39.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7ba688640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7b40cb180 0x7fc7b41f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:39.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7ba688640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b4153b50 0x7fc7b4158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:39.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7ba688640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b40ce720 con 0x7fc7b4153b50 2023-10-30T14:30:39.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7ba688640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b4154660 con 0x7fc7b40c62b0 2023-10-30T14:30:39.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7ba688640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b41f2f10 con 0x7fc7b40cb180 2023-10-30T14:30:39.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7b9e87640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b4153b50 0x7fc7b4158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:39.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7b9e87640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b4153b50 0x7fc7b4158fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43946/0 (socket says 172.21.15.57:43946) 2023-10-30T14:30:39.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7b9e87640 1 -- 172.21.15.57:0/850666252 learned_addr learned my addr 172.21.15.57:0/850666252 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:39.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7b8e85640 1 --2- 172.21.15.57:0/850666252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7b40cb180 0x7fc7b41f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:39.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7b9686640 1 --2- 172.21.15.57:0/850666252 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7b40c62b0 0x7fc7b4154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:39.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7b9e87640 1 -- 172.21.15.57:0/850666252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7b40cb180 msgr2=0x7fc7b41f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:39.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7b9e87640 1 --2- 172.21.15.57:0/850666252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7b40cb180 0x7fc7b41f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:39.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7b9e87640 1 -- 172.21.15.57:0/850666252 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7b40c62b0 msgr2=0x7fc7b4154120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:39.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7b9e87640 1 --2- 172.21.15.57:0/850666252 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7b40c62b0 0x7fc7b4154120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:39.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7b9e87640 1 -- 172.21.15.57:0/850666252 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac009a70 con 0x7fc7b4153b50 2023-10-30T14:30:39.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7b9e87640 1 --2- 172.21.15.57:0/850666252 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b4153b50 0x7fc7b4158fc0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009d90 tx=0x7fc7ac0196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:39.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7a9ffb640 1 -- 172.21.15.57:0/850666252 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac0b7850 con 0x7fc7b4153b50 2023-10-30T14:30:39.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7a9ffb640 1 -- 172.21.15.57:0/850666252 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ac0b79b0 con 0x7fc7b4153b50 2023-10-30T14:30:39.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7a9ffb640 1 -- 172.21.15.57:0/850666252 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac0b7d60 con 0x7fc7b4153b50 2023-10-30T14:30:39.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7ba688640 1 -- 172.21.15.57:0/850666252 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b4144050 con 0x7fc7b4153b50 2023-10-30T14:30:39.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7ba688640 1 -- 172.21.15.57:0/850666252 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b40cdd80 con 0x7fc7b4153b50 2023-10-30T14:30:39.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.412+0000 7fc7ba688640 1 -- 172.21.15.57:0/850666252 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b40ce860 con 0x7fc7b4153b50 2023-10-30T14:30:39.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.416+0000 7fc7a9ffb640 1 -- 172.21.15.57:0/850666252 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc7ac0c2460 con 0x7fc7b4153b50 2023-10-30T14:30:39.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.416+0000 7fc7a9ffb640 1 --2- 172.21.15.57:0/850666252 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc7980664a0 0x7fc798068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:39.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.416+0000 7fc7a9ffb640 1 -- 172.21.15.57:0/850666252 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7fc7ac11f260 con 0x7fc7b4153b50 2023-10-30T14:30:39.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.416+0000 7fc7b9686640 1 --2- 172.21.15.57:0/850666252 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc7980664a0 0x7fc798068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:39.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.416+0000 7fc7b9686640 1 --2- 172.21.15.57:0/850666252 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc7980664a0 0x7fc798068960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc7b000b930 tx=0x7fc7b0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:39.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.420+0000 7fc7a9ffb640 1 -- 172.21.15.57:0/850666252 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc7b40ce860 con 0x7fc7b4153b50 2023-10-30T14:30:39.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.608+0000 7fc7ba688640 1 -- 172.21.15.57:0/850666252 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7b40ce860 con 0x7fc7b4153b50 2023-10-30T14:30:39.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.608+0000 7fc7a9ffb640 1 -- 172.21.15.57:0/850666252 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5743 (secure 0 0 0) 0x7fc7b40ce860 con 0x7fc7b4153b50 2023-10-30T14:30:39.611 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:39.612 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:30:35.996360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:clientreplay","state_seq":16,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:39.612 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 103 2023-10-30T14:30:39.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.612+0000 7fc7ba688640 1 -- 172.21.15.57:0/850666252 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc7980664a0 msgr2=0x7fc798068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:39.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.612+0000 7fc7ba688640 1 --2- 172.21.15.57:0/850666252 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc7980664a0 0x7fc798068960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc7b000b930 tx=0x7fc7b0008040 comp rx=0 tx=0).stop 2023-10-30T14:30:39.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.612+0000 7fc7ba688640 1 -- 172.21.15.57:0/850666252 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b4153b50 msgr2=0x7fc7b4158fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:39.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.612+0000 7fc7ba688640 1 --2- 172.21.15.57:0/850666252 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b4153b50 0x7fc7b4158fc0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009d90 tx=0x7fc7ac0196f0 comp rx=0 tx=0).stop 2023-10-30T14:30:39.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.612+0000 7fc7ba688640 1 -- 172.21.15.57:0/850666252 shutdown_connections 2023-10-30T14:30:39.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.612+0000 7fc7ba688640 1 --2- 172.21.15.57:0/850666252 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc7980664a0 0x7fc798068960 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:39.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.612+0000 7fc7ba688640 1 --2- 172.21.15.57:0/850666252 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b4153b50 0x7fc7b4158fc0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:39.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.612+0000 7fc7ba688640 1 --2- 172.21.15.57:0/850666252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7b40cb180 0x7fc7b41f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:39.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.612+0000 7fc7ba688640 1 --2- 172.21.15.57:0/850666252 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7b40c62b0 0x7fc7b4154120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:39.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.612+0000 7fc7ba688640 1 -- 172.21.15.57:0/850666252 >> 172.21.15.57:0/850666252 conn(0x7fc7b4065a80 msgr2=0x7fc7b4149c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:39.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.612+0000 7fc7ba688640 1 -- 172.21.15.57:0/850666252 shutdown_connections 2023-10-30T14:30:39.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:39.612+0000 7fc7ba688640 1 -- 172.21.15.57:0/850666252 wait complete. 2023-10-30T14:30:39.632 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:30:41.633 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:41.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe56e3c5640 1 Processor -- start 2023-10-30T14:30:41.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe56e3c5640 1 -- start start 2023-10-30T14:30:41.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe56e3c5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5681537f0 0x7fe5680701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:41.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe56e3c5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5680c2aa0 con 0x7fe56806b2f0 2023-10-30T14:30:41.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe56e3c5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5680708e0 con 0x7fe56815a2e0 2023-10-30T14:30:41.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe56e3c5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5680c3ab0 con 0x7fe5681537f0 2023-10-30T14:30:41.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe56c962640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fe56806b2f0 0x7fe5681641e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:36728/0 (socket says 172.21.15.57:36728) 2023-10-30T14:30:41.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe56c962640 1 -- 172.21.15.57:0/2486720187 learned_addr learned my addr 172.21.15.57:0/2486720187 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:30:41.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe567fff640 1 --2- 172.21.15.57:0/2486720187 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5681537f0 0x7fe5680701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:41.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe566ffd640 1 -- 172.21.15.57:0/2486720187 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492120136 0 0) 0x7fe5680c2aa0 con 0x7fe56806b2f0 2023-10-30T14:30:41.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe566ffd640 1 -- 172.21.15.57:0/2486720187 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe550003530 con 0x7fe56806b2f0 2023-10-30T14:30:41.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe566ffd640 1 -- 172.21.15.57:0/2486720187 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417951337 0 0) 0x7fe5680708e0 con 0x7fe56815a2e0 2023-10-30T14:30:41.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe566ffd640 1 -- 172.21.15.57:0/2486720187 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5680c2aa0 con 0x7fe56815a2e0 2023-10-30T14:30:41.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe566ffd640 1 -- 172.21.15.57:0/2486720187 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2807000060 0 0) 0x7fe550003530 con 0x7fe56806b2f0 2023-10-30T14:30:41.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe566ffd640 1 -- 172.21.15.57:0/2486720187 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5681537f0 msgr2=0x7fe5680701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:41.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe566ffd640 1 --2- 172.21.15.57:0/2486720187 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5681537f0 0x7fe5680701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:41.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe566ffd640 1 -- 172.21.15.57:0/2486720187 >> v1:172.21.15.88:6789/0 conn(0x7fe56815a2e0 legacy=0x7fe5680ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:41.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe566ffd640 1 -- 172.21.15.57:0/2486720187 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5680c1eb0 con 0x7fe56806b2f0 2023-10-30T14:30:41.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe566ffd640 1 -- 172.21.15.57:0/2486720187 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fe560001a00 con 0x7fe56806b2f0 2023-10-30T14:30:41.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe566ffd640 1 -- 172.21.15.57:0/2486720187 >> v1:172.21.15.57:6789/0 conn(0x7fe56806b2f0 legacy=0x7fe5681641e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:41.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe566ffd640 1 --2- 172.21.15.57:0/2486720187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe550003b40 0x7fe550003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:41.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe566ffd640 1 --2- 172.21.15.57:0/2486720187 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5681537f0 0x7fe5500047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:41.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe566ffd640 1 --2- 172.21.15.57:0/2486720187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe550004d10 0x7fe550005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:41.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe566ffd640 1 -- 172.21.15.57:0/2486720187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55000ba40 con 0x7fe550004d10 2023-10-30T14:30:41.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe566ffd640 1 -- 172.21.15.57:0/2486720187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55000bb80 con 0x7fe550003b40 2023-10-30T14:30:41.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.780+0000 7fe566ffd640 1 -- 172.21.15.57:0/2486720187 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe55000bcf0 con 0x7fe5681537f0 2023-10-30T14:30:41.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.784+0000 7fe567fff640 1 --2- 172.21.15.57:0/2486720187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe550003b40 0x7fe550003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:41.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.784+0000 7fe56c962640 1 --2- 172.21.15.57:0/2486720187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe550004d10 0x7fe550005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:41.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.784+0000 7fe5677fe640 1 --2- 172.21.15.57:0/2486720187 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5681537f0 0x7fe5500047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:41.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.784+0000 7fe56c962640 1 -- 172.21.15.57:0/2486720187 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5681537f0 msgr2=0x7fe5500047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:41.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.784+0000 7fe56c962640 1 --2- 172.21.15.57:0/2486720187 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5681537f0 0x7fe5500047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:41.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.784+0000 7fe56c962640 1 -- 172.21.15.57:0/2486720187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe550003b40 msgr2=0x7fe550003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:41.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.784+0000 7fe56c962640 1 --2- 172.21.15.57:0/2486720187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe550003b40 0x7fe550003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:41.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.784+0000 7fe56c962640 1 -- 172.21.15.57:0/2486720187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe550003530 con 0x7fe550004d10 2023-10-30T14:30:41.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.784+0000 7fe56c962640 1 --2- 172.21.15.57:0/2486720187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe550004d10 0x7fe550005190 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe56000a7a0 tx=0x7fe56001d2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f7c50a5f7350667 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:41.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.784+0000 7fe566ffd640 1 -- 172.21.15.57:0/2486720187 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5600ad630 con 0x7fe550004d10 2023-10-30T14:30:41.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.784+0000 7fe566ffd640 1 -- 172.21.15.57:0/2486720187 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5600a8070 con 0x7fe550004d10 2023-10-30T14:30:41.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.784+0000 7fe566ffd640 1 -- 172.21.15.57:0/2486720187 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5600ad9f0 con 0x7fe550004d10 2023-10-30T14:30:41.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56e3c5640 1 -- 172.21.15.57:0/2486720187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe550004d10 msgr2=0x7fe550005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:41.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56e3c5640 1 --2- 172.21.15.57:0/2486720187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe550004d10 0x7fe550005190 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe56000a7a0 tx=0x7fe56001d2a0 comp rx=0 tx=0).stop 2023-10-30T14:30:41.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56c962640 1 -- 172.21.15.57:0/2486720187 reap_dead start 2023-10-30T14:30:41.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56e3c5640 1 -- 172.21.15.57:0/2486720187 shutdown_connections 2023-10-30T14:30:41.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56e3c5640 1 -- 172.21.15.57:0/2486720187 >> 172.21.15.57:0/2486720187 conn(0x7fe568071fb0 msgr2=0x7fe5680723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:41.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56e3c5640 1 -- 172.21.15.57:0/2486720187 shutdown_connections 2023-10-30T14:30:41.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56e3c5640 1 -- 172.21.15.57:0/2486720187 wait complete. 2023-10-30T14:30:41.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56e3c5640 1 Processor -- start 2023-10-30T14:30:41.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56e3c5640 1 -- start start 2023-10-30T14:30:41.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56e3c5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe56814c1a0 0x7fe56814feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:41.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56e3c5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5681503f0 0x7fe5680c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:41.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56e3c5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5680c61f0 0x7fe5681f1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:41.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56e3c5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5681585f0 con 0x7fe5680c61f0 2023-10-30T14:30:41.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56e3c5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5680c5820 con 0x7fe56814c1a0 2023-10-30T14:30:41.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56e3c5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5680c6670 con 0x7fe5681503f0 2023-10-30T14:30:41.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56c962640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5680c61f0 0x7fe5681f1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:41.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56c962640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5680c61f0 0x7fe5681f1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43966/0 (socket says 172.21.15.57:43966) 2023-10-30T14:30:41.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56c962640 1 -- 172.21.15.57:0/1708043508 learned_addr learned my addr 172.21.15.57:0/1708043508 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe567fff640 1 --2- 172.21.15.57:0/1708043508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe56814c1a0 0x7fe56814feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe5677fe640 1 --2- 172.21.15.57:0/1708043508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5681503f0 0x7fe5680c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56c962640 1 -- 172.21.15.57:0/1708043508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5681503f0 msgr2=0x7fe5680c5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56c962640 1 --2- 172.21.15.57:0/1708043508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5681503f0 0x7fe5680c5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56c962640 1 -- 172.21.15.57:0/1708043508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe56814c1a0 msgr2=0x7fe56814feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56c962640 1 --2- 172.21.15.57:0/1708043508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe56814c1a0 0x7fe56814feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe56c962640 1 -- 172.21.15.57:0/1708043508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe568072780 con 0x7fe5680c61f0 2023-10-30T14:30:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe5677fe640 1 --2- 172.21.15.57:0/1708043508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5681503f0 0x7fe5680c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:30:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.788+0000 7fe567fff640 1 --2- 172.21.15.57:0/1708043508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe56814c1a0 0x7fe56814feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:30:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.792+0000 7fe56c962640 1 --2- 172.21.15.57:0/1708043508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5680c61f0 0x7fe5681f1f30 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fe56000a7a0 tx=0x7fe550005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:41.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.792+0000 7fe564ff9640 1 -- 172.21.15.57:0/1708043508 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5600011d0 con 0x7fe5680c61f0 2023-10-30T14:30:41.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.792+0000 7fe564ff9640 1 -- 172.21.15.57:0/1708043508 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe560001f60 con 0x7fe5680c61f0 2023-10-30T14:30:41.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.792+0000 7fe564ff9640 1 -- 172.21.15.57:0/1708043508 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5600ad630 con 0x7fe5680c61f0 2023-10-30T14:30:41.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.792+0000 7fe56e3c5640 1 -- 172.21.15.57:0/1708043508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe568158840 con 0x7fe5680c61f0 2023-10-30T14:30:41.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.792+0000 7fe56e3c5640 1 -- 172.21.15.57:0/1708043508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5680c54b0 con 0x7fe5680c61f0 2023-10-30T14:30:41.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.792+0000 7fe56e3c5640 1 -- 172.21.15.57:0/1708043508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5680c2710 con 0x7fe5680c61f0 2023-10-30T14:30:41.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.792+0000 7fe564ff9640 1 -- 172.21.15.57:0/1708043508 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe5600aa030 con 0x7fe5680c61f0 2023-10-30T14:30:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.792+0000 7fe564ff9640 1 --2- 172.21.15.57:0/1708043508 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5540664a0 0x7fe554068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.792+0000 7fe564ff9640 1 -- 172.21.15.57:0/1708043508 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7fe56011dae0 con 0x7fe5680c61f0 2023-10-30T14:30:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.796+0000 7fe567fff640 1 --2- 172.21.15.57:0/1708043508 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5540664a0 0x7fe554068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.796+0000 7fe567fff640 1 --2- 172.21.15.57:0/1708043508 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5540664a0 0x7fe554068960 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe558009be0 tx=0x7fe558008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.796+0000 7fe564ff9640 1 -- 172.21.15.57:0/1708043508 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe5680c2710 con 0x7fe5680c61f0 2023-10-30T14:30:41.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.988+0000 7fe56e3c5640 1 -- 172.21.15.57:0/1708043508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5680c2710 con 0x7fe5680c61f0 2023-10-30T14:30:41.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.988+0000 7fe564ff9640 1 -- 172.21.15.57:0/1708043508 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5743 (secure 0 0 0) 0x7fe5680c2710 con 0x7fe5680c61f0 2023-10-30T14:30:41.992 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:41.992 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:30:35.996360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:clientreplay","state_seq":16,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:41.992 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 103 2023-10-30T14:30:41.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.992+0000 7fe56e3c5640 1 -- 172.21.15.57:0/1708043508 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5540664a0 msgr2=0x7fe554068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:41.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.992+0000 7fe56e3c5640 1 --2- 172.21.15.57:0/1708043508 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5540664a0 0x7fe554068960 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe558009be0 tx=0x7fe558008040 comp rx=0 tx=0).stop 2023-10-30T14:30:41.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.992+0000 7fe56e3c5640 1 -- 172.21.15.57:0/1708043508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5680c61f0 msgr2=0x7fe5681f1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:41.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.992+0000 7fe56e3c5640 1 --2- 172.21.15.57:0/1708043508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5680c61f0 0x7fe5681f1f30 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fe56000a7a0 tx=0x7fe550005800 comp rx=0 tx=0).stop 2023-10-30T14:30:41.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.992+0000 7fe56e3c5640 1 -- 172.21.15.57:0/1708043508 shutdown_connections 2023-10-30T14:30:41.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.992+0000 7fe56e3c5640 1 --2- 172.21.15.57:0/1708043508 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe5540664a0 0x7fe554068960 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:41.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.992+0000 7fe56e3c5640 1 --2- 172.21.15.57:0/1708043508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5680c61f0 0x7fe5681f1f30 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:41.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.992+0000 7fe56e3c5640 1 --2- 172.21.15.57:0/1708043508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5681503f0 0x7fe5680c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:41.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.992+0000 7fe56e3c5640 1 --2- 172.21.15.57:0/1708043508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe56814c1a0 0x7fe56814feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:41.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.992+0000 7fe56e3c5640 1 -- 172.21.15.57:0/1708043508 >> 172.21.15.57:0/1708043508 conn(0x7fe568071fb0 msgr2=0x7fe5680ca070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:41.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.992+0000 7fe56e3c5640 1 -- 172.21.15.57:0/1708043508 shutdown_connections 2023-10-30T14:30:41.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:41.992+0000 7fe56e3c5640 1 -- 172.21.15.57:0/1708043508 wait complete. 2023-10-30T14:30:42.013 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:30:44.014 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:44.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.168+0000 7ff57989e640 1 Processor -- start 2023-10-30T14:30:44.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.168+0000 7ff57989e640 1 -- start start 2023-10-30T14:30:44.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.168+0000 7ff57989e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff57414fc50 0x7ff5741616d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:44.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.168+0000 7ff57989e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5740c55f0 con 0x7ff5740c6320 2023-10-30T14:30:44.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.168+0000 7ff57989e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5741567b0 con 0x7ff57406fdf0 2023-10-30T14:30:44.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.168+0000 7ff57989e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5740c5f00 con 0x7ff57414fc50 2023-10-30T14:30:44.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.168+0000 7ff5737fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff57414fc50 0x7ff5741616d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:44.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.168+0000 7ff5737fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff57414fc50 0x7ff5741616d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50384/0 (socket says 172.21.15.57:50384) 2023-10-30T14:30:44.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.168+0000 7ff5737fe640 1 -- 172.21.15.57:0/3830573270 learned_addr learned my addr 172.21.15.57:0/3830573270 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:44.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.168+0000 7ff572ffd640 1 --1- 172.21.15.57:0/3830573270 >> v1:172.21.15.57:6789/0 conn(0x7ff5740c6320 0x7ff574070e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:56936/0 (socket says 172.21.15.57:56936) 2023-10-30T14:30:44.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.168+0000 7ff5737fe640 1 -- 172.21.15.57:0/3830573270 >> v1:172.21.15.88:6789/0 conn(0x7ff57406fdf0 legacy=0x7ff574155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:44.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.168+0000 7ff5737fe640 1 -- 172.21.15.57:0/3830573270 >> v1:172.21.15.57:6789/0 conn(0x7ff5740c6320 legacy=0x7ff574070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:44.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.168+0000 7ff5737fe640 1 -- 172.21.15.57:0/3830573270 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff574156a70 con 0x7ff57414fc50 2023-10-30T14:30:44.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.168+0000 7ff5727fc640 1 -- 172.21.15.57:0/3830573270 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686874984 0 0) 0x7ff5740c55f0 con 0x7ff5740c6320 2023-10-30T14:30:44.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.172+0000 7ff5737fe640 1 --2- 172.21.15.57:0/3830573270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff57414fc50 0x7ff5741616d0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7ff56c009e50 tx=0x7ff56c01c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50cd8226852e860 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:44.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.172+0000 7ff5727fc640 1 -- 172.21.15.57:0/3830573270 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c0a5650 con 0x7ff57414fc50 2023-10-30T14:30:44.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.172+0000 7ff5727fc640 1 -- 172.21.15.57:0/3830573270 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56c0a57b0 con 0x7ff57414fc50 2023-10-30T14:30:44.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.172+0000 7ff5727fc640 1 -- 172.21.15.57:0/3830573270 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c0a5b40 con 0x7ff57414fc50 2023-10-30T14:30:44.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.172+0000 7ff57989e640 1 -- 172.21.15.57:0/3830573270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff57414fc50 msgr2=0x7ff5741616d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:44.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.172+0000 7ff57989e640 1 --2- 172.21.15.57:0/3830573270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff57414fc50 0x7ff5741616d0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7ff56c009e50 tx=0x7ff56c01c980 comp rx=0 tx=0).stop 2023-10-30T14:30:44.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.172+0000 7ff57989e640 1 -- 172.21.15.57:0/3830573270 shutdown_connections 2023-10-30T14:30:44.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.172+0000 7ff57989e640 1 --2- 172.21.15.57:0/3830573270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff57414fc50 0x7ff5741616d0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:44.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.172+0000 7ff57989e640 1 -- 172.21.15.57:0/3830573270 >> 172.21.15.57:0/3830573270 conn(0x7ff5740cac70 msgr2=0x7ff5740cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:44.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.172+0000 7ff57989e640 1 -- 172.21.15.57:0/3830573270 shutdown_connections 2023-10-30T14:30:44.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.172+0000 7ff57989e640 1 -- 172.21.15.57:0/3830573270 wait complete. 2023-10-30T14:30:44.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff57989e640 1 Processor -- start 2023-10-30T14:30:44.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff57989e640 1 -- start start 2023-10-30T14:30:44.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff57989e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff57406fdf0 0x7ff5740c39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:44.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff57989e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5740c6320 0x7ff57413afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:44.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff57989e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff57414fc50 0x7ff57415fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:44.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff57989e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5741616c0 con 0x7ff57406fdf0 2023-10-30T14:30:44.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff57989e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5741f4de0 con 0x7ff5740c6320 2023-10-30T14:30:44.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff57989e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5741f5640 con 0x7ff57414fc50 2023-10-30T14:30:44.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff572ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff57406fdf0 0x7ff5740c39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:44.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff572ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff57406fdf0 0x7ff5740c39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50396/0 (socket says 172.21.15.57:50396) 2023-10-30T14:30:44.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff572ffd640 1 -- 172.21.15.57:0/1772080807 learned_addr learned my addr 172.21.15.57:0/1772080807 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:44.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff56bfff640 1 --2- 172.21.15.57:0/1772080807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5740c6320 0x7ff57413afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:44.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff5737fe640 1 --2- 172.21.15.57:0/1772080807 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff57414fc50 0x7ff57415fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff572ffd640 1 -- 172.21.15.57:0/1772080807 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff57414fc50 msgr2=0x7ff57415fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff572ffd640 1 --2- 172.21.15.57:0/1772080807 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff57414fc50 0x7ff57415fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff572ffd640 1 -- 172.21.15.57:0/1772080807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5740c6320 msgr2=0x7ff57413afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff572ffd640 1 --2- 172.21.15.57:0/1772080807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5740c6320 0x7ff57413afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff572ffd640 1 -- 172.21.15.57:0/1772080807 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c009b30 con 0x7ff57406fdf0 2023-10-30T14:30:44.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff572ffd640 1 --2- 172.21.15.57:0/1772080807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff57406fdf0 0x7ff5740c39f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7ff56400c4a0 tx=0x7ff56400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:44.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff570ff9640 1 -- 172.21.15.57:0/1772080807 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff564006480 con 0x7ff57406fdf0 2023-10-30T14:30:44.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff570ff9640 1 -- 172.21.15.57:0/1772080807 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5640065e0 con 0x7ff57406fdf0 2023-10-30T14:30:44.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff57989e640 1 -- 172.21.15.57:0/1772080807 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57406c790 con 0x7ff57406fdf0 2023-10-30T14:30:44.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff570ff9640 1 -- 172.21.15.57:0/1772080807 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff564004a10 con 0x7ff57406fdf0 2023-10-30T14:30:44.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.176+0000 7ff57989e640 1 -- 172.21.15.57:0/1772080807 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5740c4be0 con 0x7ff57406fdf0 2023-10-30T14:30:44.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.180+0000 7ff57989e640 1 -- 172.21.15.57:0/1772080807 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff57404e170 con 0x7ff57406fdf0 2023-10-30T14:30:44.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.180+0000 7ff570ff9640 1 -- 172.21.15.57:0/1772080807 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff564004b70 con 0x7ff57406fdf0 2023-10-30T14:30:44.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.180+0000 7ff570ff9640 1 --2- 172.21.15.57:0/1772080807 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff5540664a0 0x7ff554068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:44.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.180+0000 7ff570ff9640 1 -- 172.21.15.57:0/1772080807 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7ff564083880 con 0x7ff57406fdf0 2023-10-30T14:30:44.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.180+0000 7ff56bfff640 1 --2- 172.21.15.57:0/1772080807 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff5540664a0 0x7ff554068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:44.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.180+0000 7ff56bfff640 1 --2- 172.21.15.57:0/1772080807 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff5540664a0 0x7ff554068960 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff5740c4a60 tx=0x7ff55c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:44.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.184+0000 7ff570ff9640 1 -- 172.21.15.57:0/1772080807 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff564050890 con 0x7ff57406fdf0 2023-10-30T14:30:44.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.372+0000 7ff57989e640 1 -- 172.21.15.57:0/1772080807 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5740ca9e0 con 0x7ff57406fdf0 2023-10-30T14:30:44.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.372+0000 7ff570ff9640 1 -- 172.21.15.57:0/1772080807 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5737 (secure 0 0 0) 0x7ff564053eb0 con 0x7ff57406fdf0 2023-10-30T14:30:44.377 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:44.377 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:30:41.224973+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:44.379 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 104 2023-10-30T14:30:44.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.376+0000 7ff57989e640 1 -- 172.21.15.57:0/1772080807 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff5540664a0 msgr2=0x7ff554068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:44.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.376+0000 7ff57989e640 1 --2- 172.21.15.57:0/1772080807 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff5540664a0 0x7ff554068960 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff5740c4a60 tx=0x7ff55c009040 comp rx=0 tx=0).stop 2023-10-30T14:30:44.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.376+0000 7ff57989e640 1 -- 172.21.15.57:0/1772080807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff57406fdf0 msgr2=0x7ff5740c39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:44.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.376+0000 7ff57989e640 1 --2- 172.21.15.57:0/1772080807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff57406fdf0 0x7ff5740c39f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7ff56400c4a0 tx=0x7ff56400c960 comp rx=0 tx=0).stop 2023-10-30T14:30:44.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.376+0000 7ff57989e640 1 -- 172.21.15.57:0/1772080807 shutdown_connections 2023-10-30T14:30:44.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.376+0000 7ff57989e640 1 --2- 172.21.15.57:0/1772080807 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff5540664a0 0x7ff554068960 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:44.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.376+0000 7ff57989e640 1 --2- 172.21.15.57:0/1772080807 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff57414fc50 0x7ff57415fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:44.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.376+0000 7ff57989e640 1 --2- 172.21.15.57:0/1772080807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5740c6320 0x7ff57413afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:44.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.376+0000 7ff57989e640 1 --2- 172.21.15.57:0/1772080807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff57406fdf0 0x7ff5740c39f0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:44.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.376+0000 7ff57989e640 1 -- 172.21.15.57:0/1772080807 >> 172.21.15.57:0/1772080807 conn(0x7ff5740cac70 msgr2=0x7ff5740c6790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:44.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.376+0000 7ff57989e640 1 -- 172.21.15.57:0/1772080807 shutdown_connections 2023-10-30T14:30:44.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:44.376+0000 7ff57989e640 1 -- 172.21.15.57:0/1772080807 wait complete. 2023-10-30T14:30:44.397 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5315 2023-10-30T14:30:44.398 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.f 2023-10-30T14:30:51.398 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-30T14:30:51.399 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T14:30:51.399 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T14:30:51.402 INFO:tasks.ceph.mds.f:Started 2023-10-30T14:30:51.402 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:51.478 INFO:tasks.ceph.mds.f.smithi088.stdout:starting mds.f at 2023-10-30T14:30:51.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.564+0000 7f66ddd63640 1 Processor -- start 2023-10-30T14:30:51.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.564+0000 7f66ddd63640 1 -- start start 2023-10-30T14:30:51.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.564+0000 7f66ddd63640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f66d80c62b0 0x7f66d81597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:51.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.564+0000 7f66ddd63640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f66d80cb180 0x7f66d806ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:51.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.564+0000 7f66ddd63640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d815a770 con 0x7f66d8153b50 2023-10-30T14:30:51.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.564+0000 7f66ddd63640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d8159cf0 con 0x7f66d80cb180 2023-10-30T14:30:51.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.564+0000 7f66ddd63640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d806aa30 con 0x7f66d80c62b0 2023-10-30T14:30:51.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.564+0000 7f66dd562640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f66d80cb180 0x7f66d806ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:51.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.564+0000 7f66dd562640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f66d80cb180 0x7f66d806ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50400/0 (socket says 172.21.15.57:50400) 2023-10-30T14:30:51.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.564+0000 7f66dd562640 1 -- 172.21.15.57:0/1313064270 learned_addr learned my addr 172.21.15.57:0/1313064270 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:51.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.564+0000 7f66d7fff640 1 --2- 172.21.15.57:0/1313064270 >> v2:172.21.15.88:3300/0 conn(0x7f66d80c62b0 0x7f66d81597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:51.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.564+0000 7f66dd562640 1 -- 172.21.15.57:0/1313064270 >> v2:172.21.15.88:3300/0 conn(0x7f66d80c62b0 msgr2=0x7f66d81597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:51.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.564+0000 7f66dd562640 1 --2- 172.21.15.57:0/1313064270 >> v2:172.21.15.88:3300/0 conn(0x7f66d80c62b0 0x7f66d81597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:51.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.564+0000 7f66dd562640 1 -- 172.21.15.57:0/1313064270 >> v1:172.21.15.88:6789/0 conn(0x7f66d8153b50 legacy=0x7f66d80ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:51.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.564+0000 7f66dd562640 1 -- 172.21.15.57:0/1313064270 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d80c9d40 con 0x7f66d80cb180 2023-10-30T14:30:51.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.564+0000 7f66d7fff640 1 --2- 172.21.15.57:0/1313064270 >> v2:172.21.15.88:3300/0 conn(0x7f66d80c62b0 0x7f66d81597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:30:51.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.568+0000 7f66dd562640 1 --2- 172.21.15.57:0/1313064270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66d80cb180 0x7f66d806ae00 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f66d806d180 tx=0x7f66d001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba918a0567b582a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:51.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.568+0000 7f66d77fe640 1 -- 172.21.15.57:0/1313064270 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d00af400 con 0x7f66d80cb180 2023-10-30T14:30:51.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.568+0000 7f66d77fe640 1 -- 172.21.15.57:0/1313064270 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d00ab050 con 0x7f66d80cb180 2023-10-30T14:30:51.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.568+0000 7f66d77fe640 1 -- 172.21.15.57:0/1313064270 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d00af780 con 0x7f66d80cb180 2023-10-30T14:30:51.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.568+0000 7f66ddd63640 1 -- 172.21.15.57:0/1313064270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66d80cb180 msgr2=0x7f66d806ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:51.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.568+0000 7f66ddd63640 1 --2- 172.21.15.57:0/1313064270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66d80cb180 0x7f66d806ae00 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f66d806d180 tx=0x7f66d001f1c0 comp rx=0 tx=0).stop 2023-10-30T14:30:51.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.568+0000 7f66ddd63640 1 -- 172.21.15.57:0/1313064270 shutdown_connections 2023-10-30T14:30:51.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.568+0000 7f66ddd63640 1 --2- 172.21.15.57:0/1313064270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66d80cb180 0x7f66d806ae00 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:51.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.568+0000 7f66ddd63640 1 --2- 172.21.15.57:0/1313064270 >> v2:172.21.15.88:3300/0 conn(0x7f66d80c62b0 0x7f66d81597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:51.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.568+0000 7f66ddd63640 1 -- 172.21.15.57:0/1313064270 >> 172.21.15.57:0/1313064270 conn(0x7f66d8065a80 msgr2=0x7f66d8065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:51.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.568+0000 7f66ddd63640 1 -- 172.21.15.57:0/1313064270 shutdown_connections 2023-10-30T14:30:51.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.568+0000 7f66ddd63640 1 -- 172.21.15.57:0/1313064270 wait complete. 2023-10-30T14:30:51.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66ddd63640 1 Processor -- start 2023-10-30T14:30:51.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66ddd63640 1 -- start start 2023-10-30T14:30:51.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66ddd63640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66d80c62b0 0x7f66d81eefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:51.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66ddd63640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66d80cb180 0x7f66d80c2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:51.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66ddd63640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66d8153b50 0x7f66d80c3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:51.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66ddd63640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d81ef4f0 con 0x7f66d80cb180 2023-10-30T14:30:51.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66ddd63640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d80c3960 con 0x7f66d8153b50 2023-10-30T14:30:51.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66ddd63640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d814c4f0 con 0x7f66d80c62b0 2023-10-30T14:30:51.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66d7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66d80cb180 0x7f66d80c2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:51.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66d7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66d80cb180 0x7f66d80c2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50406/0 (socket says 172.21.15.57:50406) 2023-10-30T14:30:51.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66d7fff640 1 -- 172.21.15.57:0/2231672889 learned_addr learned my addr 172.21.15.57:0/2231672889 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:51.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66dcd61640 1 --2- 172.21.15.57:0/2231672889 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66d80c62b0 0x7f66d81eefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:51.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66dd562640 1 --2- 172.21.15.57:0/2231672889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66d8153b50 0x7f66d80c3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:51.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66d7fff640 1 -- 172.21.15.57:0/2231672889 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66d80c62b0 msgr2=0x7f66d81eefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:51.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66d7fff640 1 --2- 172.21.15.57:0/2231672889 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66d80c62b0 0x7f66d81eefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:51.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66d7fff640 1 -- 172.21.15.57:0/2231672889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66d8153b50 msgr2=0x7f66d80c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:51.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66d7fff640 1 --2- 172.21.15.57:0/2231672889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66d8153b50 0x7f66d80c3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:51.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66d7fff640 1 -- 172.21.15.57:0/2231672889 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d0009b30 con 0x7f66d80cb180 2023-10-30T14:30:51.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66d7fff640 1 --2- 172.21.15.57:0/2231672889 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66d80cb180 0x7f66d80c2ea0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f66c400e7e0 tx=0x7f66c400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:51.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66d57fa640 1 -- 172.21.15.57:0/2231672889 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c400c6d0 con 0x7f66d80cb180 2023-10-30T14:30:51.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66d57fa640 1 -- 172.21.15.57:0/2231672889 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c400c830 con 0x7f66d80cb180 2023-10-30T14:30:51.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66d57fa640 1 -- 172.21.15.57:0/2231672889 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c400cbe0 con 0x7f66d80cb180 2023-10-30T14:30:51.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66ddd63640 1 -- 172.21.15.57:0/2231672889 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d814c7d0 con 0x7f66d80cb180 2023-10-30T14:30:51.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.572+0000 7f66ddd63640 1 -- 172.21.15.57:0/2231672889 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d814cd90 con 0x7f66d80cb180 2023-10-30T14:30:51.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.576+0000 7f66ddd63640 1 -- 172.21.15.57:0/2231672889 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d814cfb0 con 0x7f66d80cb180 2023-10-30T14:30:51.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.576+0000 7f66d57fa640 1 -- 172.21.15.57:0/2231672889 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f66c400cd40 con 0x7f66d80cb180 2023-10-30T14:30:51.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.576+0000 7f66d57fa640 1 --2- 172.21.15.57:0/2231672889 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66bc0664a0 0x7f66bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:51.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.576+0000 7f66d57fa640 1 -- 172.21.15.57:0/2231672889 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f66c4084210 con 0x7f66d80cb180 2023-10-30T14:30:51.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.576+0000 7f66dcd61640 1 --2- 172.21.15.57:0/2231672889 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66bc0664a0 0x7f66bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:51.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.576+0000 7f66dcd61640 1 --2- 172.21.15.57:0/2231672889 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66bc0664a0 0x7f66bc068960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f66cc007fd0 tx=0x7f66cc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:51.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.580+0000 7f66d57fa640 1 -- 172.21.15.57:0/2231672889 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f66c4051240 con 0x7f66d80cb180 2023-10-30T14:30:51.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.772+0000 7f66ddd63640 1 -- 172.21.15.57:0/2231672889 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66d80c3750 con 0x7f66d80cb180 2023-10-30T14:30:51.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.772+0000 7f66d57fa640 1 -- 172.21.15.57:0/2231672889 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5737 (secure 0 0 0) 0x7f66c4054860 con 0x7f66d80cb180 2023-10-30T14:30:51.775 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:51.775 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:30:41.224973+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:51.775 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 104 2023-10-30T14:30:51.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.772+0000 7f66ddd63640 1 -- 172.21.15.57:0/2231672889 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66bc0664a0 msgr2=0x7f66bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:51.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.772+0000 7f66ddd63640 1 --2- 172.21.15.57:0/2231672889 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66bc0664a0 0x7f66bc068960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f66cc007fd0 tx=0x7f66cc009040 comp rx=0 tx=0).stop 2023-10-30T14:30:51.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.772+0000 7f66ddd63640 1 -- 172.21.15.57:0/2231672889 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66d80cb180 msgr2=0x7f66d80c2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:51.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.772+0000 7f66ddd63640 1 --2- 172.21.15.57:0/2231672889 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66d80cb180 0x7f66d80c2ea0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f66c400e7e0 tx=0x7f66c400eca0 comp rx=0 tx=0).stop 2023-10-30T14:30:51.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.772+0000 7f66ddd63640 1 -- 172.21.15.57:0/2231672889 shutdown_connections 2023-10-30T14:30:51.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.772+0000 7f66ddd63640 1 --2- 172.21.15.57:0/2231672889 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66bc0664a0 0x7f66bc068960 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:51.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.772+0000 7f66ddd63640 1 --2- 172.21.15.57:0/2231672889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66d8153b50 0x7f66d80c3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:51.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.772+0000 7f66ddd63640 1 --2- 172.21.15.57:0/2231672889 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66d80cb180 0x7f66d80c2ea0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:51.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.772+0000 7f66ddd63640 1 --2- 172.21.15.57:0/2231672889 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66d80c62b0 0x7f66d81eefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:51.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.772+0000 7f66ddd63640 1 -- 172.21.15.57:0/2231672889 >> 172.21.15.57:0/2231672889 conn(0x7f66d8065a80 msgr2=0x7f66d8069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:51.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.772+0000 7f66ddd63640 1 -- 172.21.15.57:0/2231672889 shutdown_connections 2023-10-30T14:30:51.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:51.772+0000 7f66ddd63640 1 -- 172.21.15.57:0/2231672889 wait complete. 2023-10-30T14:30:51.796 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-30T14:30:52.007 INFO:tasks.ceph.mds.a.smithi057.stderr:2023-10-30T14:30:52.004+0000 7f5ba0392640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T14:30:52.007 INFO:tasks.ceph.mds.a.smithi057.stderr:2023-10-30T14:30:52.004+0000 7f5ba0392640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:30:52.007 INFO:tasks.ceph.mds.a.smithi057.stderr:2023-10-30T14:30:52.004+0000 7f5ba0392640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T14:30:53.796 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:30:53.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.948+0000 7fbe27772640 1 Processor -- start 2023-10-30T14:30:53.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.948+0000 7fbe27772640 1 -- start start 2023-10-30T14:30:53.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.948+0000 7fbe27772640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbe20150830 0x7fbe2006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:53.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.948+0000 7fbe27772640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe200cb7d0 0x7fbe200c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:53.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.948+0000 7fbe27772640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbe200c3e40 0x7fbe20070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:53.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.948+0000 7fbe27772640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe200c2310 con 0x7fbe200c3e40 2023-10-30T14:30:53.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.948+0000 7fbe27772640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe20156710 con 0x7fbe20150830 2023-10-30T14:30:53.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.948+0000 7fbe27772640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe200c4360 con 0x7fbe200cb7d0 2023-10-30T14:30:53.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.948+0000 7fbe25d0f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbe200c3e40 0x7fbe20070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:53.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.948+0000 7fbe25d0f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbe200c3e40 0x7fbe20070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51920/0 (socket says 172.21.15.57:51920) 2023-10-30T14:30:53.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.948+0000 7fbe25d0f640 1 -- 172.21.15.57:0/3742474241 learned_addr learned my addr 172.21.15.57:0/3742474241 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:53.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.948+0000 7fbe2550e640 1 --2- 172.21.15.57:0/3742474241 >> v2:172.21.15.88:3300/0 conn(0x7fbe20150830 0x7fbe2006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:53.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.948+0000 7fbe24d0d640 1 --2- 172.21.15.57:0/3742474241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe200cb7d0 0x7fbe200c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:53.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.948+0000 7fbe25d0f640 1 -- 172.21.15.57:0/3742474241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe200cb7d0 msgr2=0x7fbe200c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:53.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.948+0000 7fbe25d0f640 1 --2- 172.21.15.57:0/3742474241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe200cb7d0 0x7fbe200c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:53.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.948+0000 7fbe25d0f640 1 -- 172.21.15.57:0/3742474241 >> v2:172.21.15.88:3300/0 conn(0x7fbe20150830 msgr2=0x7fbe2006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:30:53.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.948+0000 7fbe25d0f640 1 --2- 172.21.15.57:0/3742474241 >> v2:172.21.15.88:3300/0 conn(0x7fbe20150830 0x7fbe2006b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:53.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.948+0000 7fbe25d0f640 1 -- 172.21.15.57:0/3742474241 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe201569b0 con 0x7fbe200c3e40 2023-10-30T14:30:53.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.952+0000 7fbe25d0f640 1 --2- 172.21.15.57:0/3742474241 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe200c3e40 0x7fbe20070520 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fbe2006fa40 tx=0x7fbe180213c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e5df20a1a2e69d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:53.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.952+0000 7fbe17fff640 1 -- 172.21.15.57:0/3742474241 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1800b2d0 con 0x7fbe200c3e40 2023-10-30T14:30:53.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.952+0000 7fbe17fff640 1 -- 172.21.15.57:0/3742474241 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe180ad050 con 0x7fbe200c3e40 2023-10-30T14:30:53.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.952+0000 7fbe17fff640 1 -- 172.21.15.57:0/3742474241 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe18009820 con 0x7fbe200c3e40 2023-10-30T14:30:53.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.952+0000 7fbe27772640 1 -- 172.21.15.57:0/3742474241 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe200c3e40 msgr2=0x7fbe20070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:53.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.952+0000 7fbe27772640 1 --2- 172.21.15.57:0/3742474241 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe200c3e40 0x7fbe20070520 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fbe2006fa40 tx=0x7fbe180213c0 comp rx=0 tx=0).stop 2023-10-30T14:30:53.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.952+0000 7fbe27772640 1 -- 172.21.15.57:0/3742474241 shutdown_connections 2023-10-30T14:30:53.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.952+0000 7fbe27772640 1 --2- 172.21.15.57:0/3742474241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe200cb7d0 0x7fbe200c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:53.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.952+0000 7fbe27772640 1 --2- 172.21.15.57:0/3742474241 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe200c3e40 0x7fbe20070520 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:53.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.952+0000 7fbe27772640 1 --2- 172.21.15.57:0/3742474241 >> v2:172.21.15.88:3300/0 conn(0x7fbe20150830 0x7fbe2006b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:53.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.952+0000 7fbe27772640 1 -- 172.21.15.57:0/3742474241 >> 172.21.15.57:0/3742474241 conn(0x7fbe200cac70 msgr2=0x7fbe200cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:53.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.952+0000 7fbe27772640 1 -- 172.21.15.57:0/3742474241 shutdown_connections 2023-10-30T14:30:53.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.952+0000 7fbe27772640 1 -- 172.21.15.57:0/3742474241 wait complete. 2023-10-30T14:30:53.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe27772640 1 Processor -- start 2023-10-30T14:30:53.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe27772640 1 -- start start 2023-10-30T14:30:53.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe27772640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe200c3e40 0x7fbe20140000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:53.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe27772640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe200cb7d0 0x7fbe200c5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:53.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe27772640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe20150830 0x7fbe2013e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:53.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe27772640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe20155320 con 0x7fbe200cb7d0 2023-10-30T14:30:53.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe27772640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe20155dd0 con 0x7fbe20150830 2023-10-30T14:30:53.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe27772640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe20147360 con 0x7fbe200c3e40 2023-10-30T14:30:53.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe24d0d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe200cb7d0 0x7fbe200c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:53.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe24d0d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe200cb7d0 0x7fbe200c5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51936/0 (socket says 172.21.15.57:51936) 2023-10-30T14:30:53.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe24d0d640 1 -- 172.21.15.57:0/2408111317 learned_addr learned my addr 172.21.15.57:0/2408111317 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:30:53.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe25d0f640 1 --2- 172.21.15.57:0/2408111317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe20150830 0x7fbe2013e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:53.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe2550e640 1 --2- 172.21.15.57:0/2408111317 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe200c3e40 0x7fbe20140000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:53.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe24d0d640 1 -- 172.21.15.57:0/2408111317 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe200c3e40 msgr2=0x7fbe20140000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:53.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe24d0d640 1 --2- 172.21.15.57:0/2408111317 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe200c3e40 0x7fbe20140000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:53.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe24d0d640 1 -- 172.21.15.57:0/2408111317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe20150830 msgr2=0x7fbe2013e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:53.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe24d0d640 1 --2- 172.21.15.57:0/2408111317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe20150830 0x7fbe2013e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:53.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe24d0d640 1 -- 172.21.15.57:0/2408111317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1800bd40 con 0x7fbe200cb7d0 2023-10-30T14:30:53.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe2550e640 1 --2- 172.21.15.57:0/2408111317 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe200c3e40 0x7fbe20140000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:30:53.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe24d0d640 1 --2- 172.21.15.57:0/2408111317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe200cb7d0 0x7fbe200c5750 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fbe1000e8b0 tx=0x7fbe1000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:53.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe15ffb640 1 -- 172.21.15.57:0/2408111317 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1000c940 con 0x7fbe200cb7d0 2023-10-30T14:30:53.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe15ffb640 1 -- 172.21.15.57:0/2408111317 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe1000caa0 con 0x7fbe200cb7d0 2023-10-30T14:30:53.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe27772640 1 -- 172.21.15.57:0/2408111317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe20147620 con 0x7fbe200cb7d0 2023-10-30T14:30:53.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe15ffb640 1 -- 172.21.15.57:0/2408111317 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1000ce30 con 0x7fbe200cb7d0 2023-10-30T14:30:53.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe27772640 1 -- 172.21.15.57:0/2408111317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe20154c60 con 0x7fbe200cb7d0 2023-10-30T14:30:53.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.956+0000 7fbe27772640 1 -- 172.21.15.57:0/2408111317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe2004e170 con 0x7fbe200cb7d0 2023-10-30T14:30:53.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.960+0000 7fbe15ffb640 1 -- 172.21.15.57:0/2408111317 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbe10013470 con 0x7fbe200cb7d0 2023-10-30T14:30:53.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.960+0000 7fbe15ffb640 1 --2- 172.21.15.57:0/2408111317 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbe040664a0 0x7fbe04068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:30:53.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.960+0000 7fbe15ffb640 1 -- 172.21.15.57:0/2408111317 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7fbe10084350 con 0x7fbe200cb7d0 2023-10-30T14:30:53.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.960+0000 7fbe2550e640 1 --2- 172.21.15.57:0/2408111317 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbe040664a0 0x7fbe04068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:30:53.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.960+0000 7fbe2550e640 1 --2- 172.21.15.57:0/2408111317 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbe040664a0 0x7fbe04068960 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c009bb0 tx=0x7fbe1c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:30:53.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:53.964+0000 7fbe15ffb640 1 -- 172.21.15.57:0/2408111317 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbe100514a0 con 0x7fbe200cb7d0 2023-10-30T14:30:54.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:54.176+0000 7fbe27772640 1 -- 172.21.15.57:0/2408111317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe20071480 con 0x7fbe200cb7d0 2023-10-30T14:30:54.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:54.180+0000 7fbe15ffb640 1 -- 172.21.15.57:0/2408111317 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+6529 (secure 0 0 0) 0x7fbe10054ac0 con 0x7fbe200cb7d0 2023-10-30T14:30:54.181 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:30:54.182 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5891,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:30:41.224973+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:30:54.182 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 105 2023-10-30T14:30:54.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:54.180+0000 7fbe27772640 1 -- 172.21.15.57:0/2408111317 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbe040664a0 msgr2=0x7fbe04068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:54.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:54.180+0000 7fbe27772640 1 --2- 172.21.15.57:0/2408111317 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbe040664a0 0x7fbe04068960 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c009bb0 tx=0x7fbe1c008040 comp rx=0 tx=0).stop 2023-10-30T14:30:54.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:54.180+0000 7fbe27772640 1 -- 172.21.15.57:0/2408111317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe200cb7d0 msgr2=0x7fbe200c5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:30:54.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:54.180+0000 7fbe27772640 1 --2- 172.21.15.57:0/2408111317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe200cb7d0 0x7fbe200c5750 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fbe1000e8b0 tx=0x7fbe1000ed70 comp rx=0 tx=0).stop 2023-10-30T14:30:54.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:54.180+0000 7fbe27772640 1 -- 172.21.15.57:0/2408111317 shutdown_connections 2023-10-30T14:30:54.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:54.180+0000 7fbe27772640 1 --2- 172.21.15.57:0/2408111317 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbe040664a0 0x7fbe04068960 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:54.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:54.180+0000 7fbe27772640 1 --2- 172.21.15.57:0/2408111317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe20150830 0x7fbe2013e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:54.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:54.180+0000 7fbe27772640 1 --2- 172.21.15.57:0/2408111317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe200cb7d0 0x7fbe200c5750 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:54.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:54.180+0000 7fbe27772640 1 --2- 172.21.15.57:0/2408111317 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe200c3e40 0x7fbe20140000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:30:54.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:54.180+0000 7fbe27772640 1 -- 172.21.15.57:0/2408111317 >> 172.21.15.57:0/2408111317 conn(0x7fbe200cac70 msgr2=0x7fbe200c25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:30:54.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:54.180+0000 7fbe27772640 1 -- 172.21.15.57:0/2408111317 shutdown_connections 2023-10-30T14:30:54.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:30:54.180+0000 7fbe27772640 1 -- 172.21.15.57:0/2408111317 wait complete. 2023-10-30T14:30:54.203 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-30T14:30:54.203 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 108 secs before thrashing 2023-10-30T14:31:05.545 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:31:05.550 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:31:36.400 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:31:36.405 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:32:06.464 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:32:06.471 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:32:36.536 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:32:36.541 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:32:42.205 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:32:42.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51fe3e4640 1 Processor -- start 2023-10-30T14:32:42.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51fe3e4640 1 -- start start 2023-10-30T14:32:42.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51fe3e4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51f80c8eb0 0x7f51f815cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:42.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51fe3e4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f51f806aeb0 0x7f51f8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:42.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51fe3e4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f815dcc0 con 0x7f51f815ac70 2023-10-30T14:32:42.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51fe3e4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f8159800 con 0x7f51f806aeb0 2023-10-30T14:32:42.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51fe3e4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f8159d80 con 0x7f51f80c8eb0 2023-10-30T14:32:42.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51fcbe1640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f51f815ac70 0x7f51f815a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:59352/0 (socket says 172.21.15.57:59352) 2023-10-30T14:32:42.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51fcbe1640 1 -- 172.21.15.57:0/881649081 learned_addr learned my addr 172.21.15.57:0/881649081 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:32:42.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51fd3e2640 1 --2- 172.21.15.57:0/881649081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51f80c8eb0 0x7f51f815cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:42.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51fdbe3640 1 --2- 172.21.15.57:0/881649081 >> v2:172.21.15.88:3300/0 conn(0x7f51f806aeb0 0x7f51f8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:42.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51effff640 1 -- 172.21.15.57:0/881649081 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201295729 0 0) 0x7f51e8001120 con 0x7f51f815ac70 2023-10-30T14:32:42.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51effff640 1 -- 172.21.15.57:0/881649081 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51e4003530 con 0x7f51f815ac70 2023-10-30T14:32:42.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51effff640 1 -- 172.21.15.57:0/881649081 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1837958589 0 0) 0x7f51e4003530 con 0x7f51f815ac70 2023-10-30T14:32:42.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51effff640 1 -- 172.21.15.57:0/881649081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51f80c8eb0 msgr2=0x7f51f815cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:32:42.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51effff640 1 --2- 172.21.15.57:0/881649081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51f80c8eb0 0x7f51f815cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:42.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51effff640 1 -- 172.21.15.57:0/881649081 >> v2:172.21.15.88:3300/0 conn(0x7f51f806aeb0 msgr2=0x7f51f8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:32:42.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51effff640 1 --2- 172.21.15.57:0/881649081 >> v2:172.21.15.88:3300/0 conn(0x7f51f806aeb0 0x7f51f8069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:42.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51effff640 1 -- 172.21.15.57:0/881649081 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f815d420 con 0x7f51f815ac70 2023-10-30T14:32:42.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51effff640 1 -- 172.21.15.57:0/881649081 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f51e8001810 con 0x7f51f815ac70 2023-10-30T14:32:42.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51effff640 1 -- 172.21.15.57:0/881649081 >> v1:172.21.15.57:6789/0 conn(0x7f51f815ac70 legacy=0x7f51f815a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:42.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51effff640 1 --2- 172.21.15.57:0/881649081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51e4003980 0x7f51e4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:42.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51effff640 1 --2- 172.21.15.57:0/881649081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51e40047d0 0x7f51e4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:42.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51effff640 1 --2- 172.21.15.57:0/881649081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51f80c8eb0 0x7f51e4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:42.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51effff640 1 -- 172.21.15.57:0/881649081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f8159800 con 0x7f51e40047d0 2023-10-30T14:32:42.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51effff640 1 -- 172.21.15.57:0/881649081 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f8159d80 con 0x7f51e4003980 2023-10-30T14:32:42.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51effff640 1 -- 172.21.15.57:0/881649081 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e400b990 con 0x7f51f80c8eb0 2023-10-30T14:32:42.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51fcbe1640 1 --2- 172.21.15.57:0/881649081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51e40047d0 0x7f51e4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:42.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.394+0000 7f51fd3e2640 1 --2- 172.21.15.57:0/881649081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51e4003980 0x7f51e4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:42.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.398+0000 7f51fdbe3640 1 --2- 172.21.15.57:0/881649081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51f80c8eb0 0x7f51e4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:42.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.398+0000 7f51fcbe1640 1 -- 172.21.15.57:0/881649081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51f80c8eb0 msgr2=0x7f51e4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:32:42.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.398+0000 7f51fcbe1640 1 --2- 172.21.15.57:0/881649081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51f80c8eb0 0x7f51e4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:42.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.398+0000 7f51fcbe1640 1 -- 172.21.15.57:0/881649081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51e4003980 msgr2=0x7f51e4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:32:42.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.398+0000 7f51fcbe1640 1 --2- 172.21.15.57:0/881649081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51e4003980 0x7f51e4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:42.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.398+0000 7f51fcbe1640 1 -- 172.21.15.57:0/881649081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e4003530 con 0x7f51e40047d0 2023-10-30T14:32:42.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.398+0000 7f51fcbe1640 1 --2- 172.21.15.57:0/881649081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51e40047d0 0x7f51e4004c50 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f51f815b090 tx=0x7f51e801cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f748a04e250a51fb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:42.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.398+0000 7f51effff640 1 -- 172.21.15.57:0/881649081 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e80058e0 con 0x7f51e40047d0 2023-10-30T14:32:42.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.398+0000 7f51effff640 1 -- 172.21.15.57:0/881649081 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51e8005a40 con 0x7f51e40047d0 2023-10-30T14:32:42.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.398+0000 7f51effff640 1 -- 172.21.15.57:0/881649081 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e8005df0 con 0x7f51e40047d0 2023-10-30T14:32:42.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fe3e4640 1 -- 172.21.15.57:0/881649081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51e40047d0 msgr2=0x7f51e4004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:42.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fe3e4640 1 --2- 172.21.15.57:0/881649081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51e40047d0 0x7f51e4004c50 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f51f815b090 tx=0x7f51e801cc00 comp rx=0 tx=0).stop 2023-10-30T14:32:42.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fdbe3640 1 -- 172.21.15.57:0/881649081 reap_dead start 2023-10-30T14:32:42.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fe3e4640 1 -- 172.21.15.57:0/881649081 shutdown_connections 2023-10-30T14:32:42.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fe3e4640 1 -- 172.21.15.57:0/881649081 >> 172.21.15.57:0/881649081 conn(0x7f51f80cef00 msgr2=0x7f51f80cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:32:42.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fe3e4640 1 -- 172.21.15.57:0/881649081 shutdown_connections 2023-10-30T14:32:42.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fe3e4640 1 -- 172.21.15.57:0/881649081 wait complete. 2023-10-30T14:32:42.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fe3e4640 1 Processor -- start 2023-10-30T14:32:42.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fe3e4640 1 -- start start 2023-10-30T14:32:42.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fe3e4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51e40047d0 0x7f51f814c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:42.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fe3e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51f81512c0 0x7f51f81f0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:42.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fe3e4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51f81f06a0 0x7f51f81491b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:42.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fe3e4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f814d020 con 0x7f51e40047d0 2023-10-30T14:32:42.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fe3e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f814c190 con 0x7f51f81512c0 2023-10-30T14:32:42.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fe3e4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f81f0b50 con 0x7f51f81f06a0 2023-10-30T14:32:42.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fd3e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51e40047d0 0x7f51f814c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:42.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fd3e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51e40047d0 0x7f51f814c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48836/0 (socket says 172.21.15.57:48836) 2023-10-30T14:32:42.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fd3e2640 1 -- 172.21.15.57:0/2959936882 learned_addr learned my addr 172.21.15.57:0/2959936882 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:32:42.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fcbe1640 1 --2- 172.21.15.57:0/2959936882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51f81512c0 0x7f51f81f0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:42.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.402+0000 7f51fdbe3640 1 --2- 172.21.15.57:0/2959936882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51f81f06a0 0x7f51f81491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:42.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.406+0000 7f51fd3e2640 1 -- 172.21.15.57:0/2959936882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51f81f06a0 msgr2=0x7f51f81491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:42.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.406+0000 7f51fd3e2640 1 --2- 172.21.15.57:0/2959936882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51f81f06a0 0x7f51f81491b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:42.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.406+0000 7f51fd3e2640 1 -- 172.21.15.57:0/2959936882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51f81512c0 msgr2=0x7f51f81f0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:42.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.406+0000 7f51fd3e2640 1 --2- 172.21.15.57:0/2959936882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51f81512c0 0x7f51f81f0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:42.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.406+0000 7f51fd3e2640 1 -- 172.21.15.57:0/2959936882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f814c500 con 0x7f51e40047d0 2023-10-30T14:32:42.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.406+0000 7f51fd3e2640 1 --2- 172.21.15.57:0/2959936882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51e40047d0 0x7f51f814c7e0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f51f4011990 tx=0x7f51f4011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:42.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.406+0000 7f51edffb640 1 -- 172.21.15.57:0/2959936882 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f400fad0 con 0x7f51e40047d0 2023-10-30T14:32:42.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.406+0000 7f51edffb640 1 -- 172.21.15.57:0/2959936882 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f400fc30 con 0x7f51e40047d0 2023-10-30T14:32:42.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.406+0000 7f51edffb640 1 -- 172.21.15.57:0/2959936882 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f4014440 con 0x7f51e40047d0 2023-10-30T14:32:42.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.406+0000 7f51fe3e4640 1 -- 172.21.15.57:0/2959936882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f81f2270 con 0x7f51e40047d0 2023-10-30T14:32:42.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.406+0000 7f51fe3e4640 1 -- 172.21.15.57:0/2959936882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f81f2810 con 0x7f51e40047d0 2023-10-30T14:32:42.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.406+0000 7f51fe3e4640 1 -- 172.21.15.57:0/2959936882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f804e170 con 0x7f51e40047d0 2023-10-30T14:32:42.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.406+0000 7f51edffb640 1 -- 172.21.15.57:0/2959936882 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f51f40145a0 con 0x7f51e40047d0 2023-10-30T14:32:42.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.410+0000 7f51edffb640 1 --2- 172.21.15.57:0/2959936882 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51dc0664a0 0x7f51dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:42.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.410+0000 7f51edffb640 1 -- 172.21.15.57:0/2959936882 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f51f40849b0 con 0x7f51e40047d0 2023-10-30T14:32:42.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.410+0000 7f51fcbe1640 1 --2- 172.21.15.57:0/2959936882 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51dc0664a0 0x7f51dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:42.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.410+0000 7f51fcbe1640 1 --2- 172.21.15.57:0/2959936882 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51dc0664a0 0x7f51dc068960 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f51e80aa000 tx=0x7f51e8000ff0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:42.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.414+0000 7f51edffb640 1 -- 172.21.15.57:0/2959936882 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f51f40519e0 con 0x7f51e40047d0 2023-10-30T14:32:42.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.606+0000 7f51fe3e4640 1 -- 172.21.15.57:0/2959936882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51f80ca960 con 0x7f51e40047d0 2023-10-30T14:32:42.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.606+0000 7f51edffb640 1 -- 172.21.15.57:0/2959936882 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+6533 (secure 0 0 0) 0x7f51f4055000 con 0x7f51e40047d0 2023-10-30T14:32:42.608 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:32:42.608 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5891,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:32:34.961195+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:32:42.608 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 110 2023-10-30T14:32:42.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.610+0000 7f51fe3e4640 1 -- 172.21.15.57:0/2959936882 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51dc0664a0 msgr2=0x7f51dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:42.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.610+0000 7f51fe3e4640 1 --2- 172.21.15.57:0/2959936882 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51dc0664a0 0x7f51dc068960 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f51e80aa000 tx=0x7f51e8000ff0 comp rx=0 tx=0).stop 2023-10-30T14:32:42.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.610+0000 7f51fe3e4640 1 -- 172.21.15.57:0/2959936882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51e40047d0 msgr2=0x7f51f814c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:42.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.610+0000 7f51fe3e4640 1 --2- 172.21.15.57:0/2959936882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51e40047d0 0x7f51f814c7e0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f51f4011990 tx=0x7f51f4011e50 comp rx=0 tx=0).stop 2023-10-30T14:32:42.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.610+0000 7f51fe3e4640 1 -- 172.21.15.57:0/2959936882 shutdown_connections 2023-10-30T14:32:42.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.610+0000 7f51fe3e4640 1 --2- 172.21.15.57:0/2959936882 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51dc0664a0 0x7f51dc068960 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:42.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.610+0000 7f51fe3e4640 1 --2- 172.21.15.57:0/2959936882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51f81f06a0 0x7f51f81491b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:42.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.610+0000 7f51fe3e4640 1 --2- 172.21.15.57:0/2959936882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51f81512c0 0x7f51f81f0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:42.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.610+0000 7f51fe3e4640 1 --2- 172.21.15.57:0/2959936882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51e40047d0 0x7f51f814c7e0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:42.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.610+0000 7f51fe3e4640 1 -- 172.21.15.57:0/2959936882 >> 172.21.15.57:0/2959936882 conn(0x7f51f80cef00 msgr2=0x7f51f80c5390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:32:42.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.610+0000 7f51fe3e4640 1 -- 172.21.15.57:0/2959936882 shutdown_connections 2023-10-30T14:32:42.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.610+0000 7f51fe3e4640 1 -- 172.21.15.57:0/2959936882 wait complete. 2023-10-30T14:32:42.634 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=1) 2023-10-30T14:32:42.634 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T14:32:42.635 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:32:42.654 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T14:32:42.655 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:32:42.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.846+0000 7f9154409640 1 Processor -- start 2023-10-30T14:32:42.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.846+0000 7f9154409640 1 -- start start 2023-10-30T14:32:42.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.846+0000 7f9154409640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f914c153c80 0x7f914c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:42.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.846+0000 7f9154409640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f914c159f20 con 0x7f914c0c4a10 2023-10-30T14:32:42.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.846+0000 7f9154409640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f914c15d620 con 0x7f914c0c42a0 2023-10-30T14:32:42.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.846+0000 7f9154409640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914c15dec0 con 0x7f914c153c80 2023-10-30T14:32:42.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.850+0000 7f91519a4640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f914c0c4a10 0x7f914c15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:59356/0 (socket says 172.21.15.57:59356) 2023-10-30T14:32:42.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.850+0000 7f91519a4640 1 -- 172.21.15.57:0/3614414557 learned_addr learned my addr 172.21.15.57:0/3614414557 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:32:42.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.850+0000 7f91521a5640 1 --2- 172.21.15.57:0/3614414557 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f914c153c80 0x7f914c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:42.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.850+0000 7f91511a3640 1 -- 172.21.15.57:0/3614414557 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 129235782 0 0) 0x7f914c159f20 con 0x7f914c0c4a10 2023-10-30T14:32:42.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.850+0000 7f91511a3640 1 -- 172.21.15.57:0/3614414557 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9138003530 con 0x7f914c0c4a10 2023-10-30T14:32:42.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.850+0000 7f91511a3640 1 -- 172.21.15.57:0/3614414557 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723113856 0 0) 0x7f914c15d620 con 0x7f914c0c42a0 2023-10-30T14:32:42.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.850+0000 7f91511a3640 1 -- 172.21.15.57:0/3614414557 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f914c159f20 con 0x7f914c0c42a0 2023-10-30T14:32:42.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.850+0000 7f91521a5640 1 -- 172.21.15.57:0/3614414557 >> v1:172.21.15.88:6789/0 conn(0x7f914c0c42a0 legacy=0x7f914c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:42.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.850+0000 7f91521a5640 1 -- 172.21.15.57:0/3614414557 >> v1:172.21.15.57:6789/0 conn(0x7f914c0c4a10 legacy=0x7f914c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:42.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.850+0000 7f91521a5640 1 -- 172.21.15.57:0/3614414557 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914c15d900 con 0x7f914c153c80 2023-10-30T14:32:42.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.854+0000 7f91521a5640 1 --2- 172.21.15.57:0/3614414557 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f914c153c80 0x7f914c15cf90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f91480097f0 tx=0x7f914801ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc0bf4251966e37c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:42.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.854+0000 7f91511a3640 1 -- 172.21.15.57:0/3614414557 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91480a89b0 con 0x7f914c153c80 2023-10-30T14:32:42.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.854+0000 7f91511a3640 1 -- 172.21.15.57:0/3614414557 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91480a8b10 con 0x7f914c153c80 2023-10-30T14:32:42.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.854+0000 7f91511a3640 1 -- 172.21.15.57:0/3614414557 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91480a8ea0 con 0x7f914c153c80 2023-10-30T14:32:42.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.854+0000 7f9154409640 1 -- 172.21.15.57:0/3614414557 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f914c153c80 msgr2=0x7f914c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:42.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.854+0000 7f9154409640 1 --2- 172.21.15.57:0/3614414557 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f914c153c80 0x7f914c15cf90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f91480097f0 tx=0x7f914801ecb0 comp rx=0 tx=0).stop 2023-10-30T14:32:42.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.854+0000 7f9154409640 1 -- 172.21.15.57:0/3614414557 shutdown_connections 2023-10-30T14:32:42.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.854+0000 7f9154409640 1 --2- 172.21.15.57:0/3614414557 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f914c153c80 0x7f914c15cf90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:42.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.854+0000 7f9154409640 1 -- 172.21.15.57:0/3614414557 >> 172.21.15.57:0/3614414557 conn(0x7f914c0715c0 msgr2=0x7f914c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:32:42.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.854+0000 7f9154409640 1 -- 172.21.15.57:0/3614414557 shutdown_connections 2023-10-30T14:32:42.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.854+0000 7f9154409640 1 -- 172.21.15.57:0/3614414557 wait complete. 2023-10-30T14:32:42.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f9154409640 1 Processor -- start 2023-10-30T14:32:42.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f9154409640 1 -- start start 2023-10-30T14:32:42.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f9154409640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f914c0c42a0 0x7f914c1f20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:42.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f9154409640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f914c0c4a10 0x7f914c158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:42.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f9154409640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f914c153c80 0x7f914c146970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:42.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f9154409640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c15b6d0 con 0x7f914c0c4a10 2023-10-30T14:32:42.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f9154409640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c0cd6e0 con 0x7f914c153c80 2023-10-30T14:32:42.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f9154409640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914c0cd9a0 con 0x7f914c0c42a0 2023-10-30T14:32:42.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f91519a4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f914c0c4a10 0x7f914c158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:42.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f91519a4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f914c0c4a10 0x7f914c158bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48844/0 (socket says 172.21.15.57:48844) 2023-10-30T14:32:42.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f91519a4640 1 -- 172.21.15.57:0/3434182917 learned_addr learned my addr 172.21.15.57:0/3434182917 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:32:42.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f91521a5640 1 --2- 172.21.15.57:0/3434182917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f914c0c42a0 0x7f914c1f20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:42.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f91529a6640 1 --2- 172.21.15.57:0/3434182917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f914c153c80 0x7f914c146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:42.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f91519a4640 1 -- 172.21.15.57:0/3434182917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f914c0c42a0 msgr2=0x7f914c1f20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:42.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f91519a4640 1 --2- 172.21.15.57:0/3434182917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f914c0c42a0 0x7f914c1f20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:42.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f91519a4640 1 -- 172.21.15.57:0/3434182917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f914c153c80 msgr2=0x7f914c146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:42.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f91519a4640 1 --2- 172.21.15.57:0/3434182917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f914c153c80 0x7f914c146970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:42.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f91519a4640 1 -- 172.21.15.57:0/3434182917 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9148009420 con 0x7f914c0c4a10 2023-10-30T14:32:42.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f91519a4640 1 --2- 172.21.15.57:0/3434182917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f914c0c4a10 0x7f914c158bc0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f913c005890 tx=0x7f913c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:42.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f9142ffd640 1 -- 172.21.15.57:0/3434182917 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c005cf0 con 0x7f914c0c4a10 2023-10-30T14:32:42.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f9142ffd640 1 -- 172.21.15.57:0/3434182917 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913c005e50 con 0x7f914c0c4a10 2023-10-30T14:32:42.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f9142ffd640 1 -- 172.21.15.57:0/3434182917 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c007270 con 0x7f914c0c4a10 2023-10-30T14:32:42.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f9154409640 1 -- 172.21.15.57:0/3434182917 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914c0ce890 con 0x7f914c0c4a10 2023-10-30T14:32:42.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.858+0000 7f9154409640 1 -- 172.21.15.57:0/3434182917 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f914c15bd30 con 0x7f914c0c4a10 2023-10-30T14:32:42.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.862+0000 7f9154409640 1 -- 172.21.15.57:0/3434182917 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f914c04e170 con 0x7f914c0c4a10 2023-10-30T14:32:42.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.862+0000 7f9142ffd640 1 -- 172.21.15.57:0/3434182917 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f913c0073d0 con 0x7f914c0c4a10 2023-10-30T14:32:42.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.862+0000 7f9142ffd640 1 --2- 172.21.15.57:0/3434182917 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f91300664a0 0x7f9130068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:42.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.862+0000 7f9142ffd640 1 -- 172.21.15.57:0/3434182917 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f913c083780 con 0x7f914c0c4a10 2023-10-30T14:32:42.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.862+0000 7f91521a5640 1 --2- 172.21.15.57:0/3434182917 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f91300664a0 0x7f9130068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:42.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.862+0000 7f91521a5640 1 --2- 172.21.15.57:0/3434182917 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f91300664a0 0x7f9130068960 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f91480b2000 tx=0x7f91480aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:42.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:42.866+0000 7f9142ffd640 1 -- 172.21.15.57:0/3434182917 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f913c050840 con 0x7f914c0c4a10 2023-10-30T14:32:43.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:43.070+0000 7f9154409640 1 -- 172.21.15.57:0/3434182917 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f914c0cad40 con 0x7f914c0c4a10 2023-10-30T14:32:43.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:43.070+0000 7f9142ffd640 1 -- 172.21.15.57:0/3434182917 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+6533 (secure 0 0 0) 0x7f913c053e60 con 0x7f914c0c4a10 2023-10-30T14:32:43.072 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:32:43.072 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5891,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:32:34.961195+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:32:43.073 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 110 2023-10-30T14:32:43.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:43.074+0000 7f9154409640 1 -- 172.21.15.57:0/3434182917 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f91300664a0 msgr2=0x7f9130068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:43.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:43.074+0000 7f9154409640 1 --2- 172.21.15.57:0/3434182917 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f91300664a0 0x7f9130068960 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f91480b2000 tx=0x7f91480aa070 comp rx=0 tx=0).stop 2023-10-30T14:32:43.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:43.074+0000 7f9154409640 1 -- 172.21.15.57:0/3434182917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f914c0c4a10 msgr2=0x7f914c158bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:43.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:43.074+0000 7f9154409640 1 --2- 172.21.15.57:0/3434182917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f914c0c4a10 0x7f914c158bc0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f913c005890 tx=0x7f913c00c960 comp rx=0 tx=0).stop 2023-10-30T14:32:43.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:43.074+0000 7f9154409640 1 -- 172.21.15.57:0/3434182917 shutdown_connections 2023-10-30T14:32:43.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:43.074+0000 7f9154409640 1 --2- 172.21.15.57:0/3434182917 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f91300664a0 0x7f9130068960 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:43.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:43.074+0000 7f9154409640 1 --2- 172.21.15.57:0/3434182917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f914c153c80 0x7f914c146970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:43.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:43.074+0000 7f9154409640 1 --2- 172.21.15.57:0/3434182917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f914c0c4a10 0x7f914c158bc0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:43.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:43.074+0000 7f9154409640 1 --2- 172.21.15.57:0/3434182917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f914c0c42a0 0x7f914c1f20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:43.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:43.074+0000 7f9154409640 1 -- 172.21.15.57:0/3434182917 >> 172.21.15.57:0/3434182917 conn(0x7f914c0715c0 msgr2=0x7f914c1523e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:32:43.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:43.074+0000 7f9154409640 1 -- 172.21.15.57:0/3434182917 shutdown_connections 2023-10-30T14:32:43.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:43.074+0000 7f9154409640 1 -- 172.21.15.57:0/3434182917 wait complete. 2023-10-30T14:32:43.095 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:32:45.096 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:32:45.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.242+0000 7f9465bd3640 1 Processor -- start 2023-10-30T14:32:45.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.242+0000 7f9465bd3640 1 -- start start 2023-10-30T14:32:45.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.242+0000 7f9465bd3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9460153c80 0x7f946015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:45.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.242+0000 7f9465bd3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f94600c42a0 0x7f94600ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:45.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.242+0000 7f9465bd3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9460159f20 con 0x7f94600c4a10 2023-10-30T14:32:45.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.242+0000 7f9465bd3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f946015dec0 con 0x7f94600c42a0 2023-10-30T14:32:45.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.242+0000 7f9465bd3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94601599a0 con 0x7f9460153c80 2023-10-30T14:32:45.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.242+0000 7f945effd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f94600c4a10 0x7f946015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:59148/0 (socket says 172.21.15.57:59148) 2023-10-30T14:32:45.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.242+0000 7f945ffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f94600c42a0 0x7f94600ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:45.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.242+0000 7f945f7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9460153c80 0x7f946015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:45.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.242+0000 7f945effd640 1 -- 172.21.15.57:0/69450775 learned_addr learned my addr 172.21.15.57:0/69450775 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:32:45.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.242+0000 7f945e7fc640 1 -- 172.21.15.57:0/69450775 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405240086 0 0) 0x7f9460159f20 con 0x7f94600c4a10 2023-10-30T14:32:45.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.242+0000 7f945e7fc640 1 -- 172.21.15.57:0/69450775 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9448003530 con 0x7f94600c4a10 2023-10-30T14:32:45.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.242+0000 7f945ffff640 1 -- 172.21.15.57:0/69450775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9460153c80 msgr2=0x7f946015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:32:45.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.242+0000 7f945ffff640 1 --2- 172.21.15.57:0/69450775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9460153c80 0x7f946015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:45.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.242+0000 7f945ffff640 1 -- 172.21.15.57:0/69450775 >> v1:172.21.15.88:6789/0 conn(0x7f94600c4a10 legacy=0x7f946015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:45.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.242+0000 7f945ffff640 1 -- 172.21.15.57:0/69450775 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946015d620 con 0x7f94600c42a0 2023-10-30T14:32:45.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.242+0000 7f945f7fe640 1 --2- 172.21.15.57:0/69450775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9460153c80 0x7f946015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:32:45.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.246+0000 7f945ffff640 1 --2- 172.21.15.57:0/69450775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94600c42a0 0x7f94600ca1e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9458009d90 tx=0x7f945801f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce26ef5583bd532a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:45.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.246+0000 7f945e7fc640 1 -- 172.21.15.57:0/69450775 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94580af600 con 0x7f94600c42a0 2023-10-30T14:32:45.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.246+0000 7f945e7fc640 1 -- 172.21.15.57:0/69450775 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94580ab050 con 0x7f94600c42a0 2023-10-30T14:32:45.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.246+0000 7f945e7fc640 1 -- 172.21.15.57:0/69450775 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94580af980 con 0x7f94600c42a0 2023-10-30T14:32:45.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.246+0000 7f9465bd3640 1 -- 172.21.15.57:0/69450775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94600c42a0 msgr2=0x7f94600ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:45.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.246+0000 7f9465bd3640 1 --2- 172.21.15.57:0/69450775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94600c42a0 0x7f94600ca1e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9458009d90 tx=0x7f945801f130 comp rx=0 tx=0).stop 2023-10-30T14:32:45.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.246+0000 7f9465bd3640 1 -- 172.21.15.57:0/69450775 shutdown_connections 2023-10-30T14:32:45.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.246+0000 7f9465bd3640 1 --2- 172.21.15.57:0/69450775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94600c42a0 0x7f94600ca1e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:45.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.246+0000 7f9465bd3640 1 --2- 172.21.15.57:0/69450775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9460153c80 0x7f946015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:45.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.246+0000 7f9465bd3640 1 -- 172.21.15.57:0/69450775 >> 172.21.15.57:0/69450775 conn(0x7f94600715c0 msgr2=0x7f94600719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:32:45.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f9465bd3640 1 -- 172.21.15.57:0/69450775 shutdown_connections 2023-10-30T14:32:45.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f9465bd3640 1 -- 172.21.15.57:0/69450775 wait complete. 2023-10-30T14:32:45.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f9465bd3640 1 Processor -- start 2023-10-30T14:32:45.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f9465bd3640 1 -- start start 2023-10-30T14:32:45.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f9465bd3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94600c42a0 0x7f94600c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:45.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f9465bd3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94600c4a10 0x7f94600b2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:45.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f9465bd3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9460153c80 0x7f94600b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:45.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f9465bd3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94601ee800 con 0x7f94600c42a0 2023-10-30T14:32:45.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f9465bd3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94601eea70 con 0x7f9460153c80 2023-10-30T14:32:45.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f9465bd3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94601eed50 con 0x7f94600c4a10 2023-10-30T14:32:45.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f945f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94600c42a0 0x7f94600c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:45.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f945f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94600c42a0 0x7f94600c8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48076/0 (socket says 172.21.15.57:48076) 2023-10-30T14:32:45.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f945f7fe640 1 -- 172.21.15.57:0/1427038397 learned_addr learned my addr 172.21.15.57:0/1427038397 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:32:45.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f945effd640 1 --2- 172.21.15.57:0/1427038397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94600c4a10 0x7f94600b2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:45.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f945ffff640 1 --2- 172.21.15.57:0/1427038397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9460153c80 0x7f94600b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:45.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f945f7fe640 1 -- 172.21.15.57:0/1427038397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94600c4a10 msgr2=0x7f94600b2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:45.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f945f7fe640 1 --2- 172.21.15.57:0/1427038397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94600c4a10 0x7f94600b2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:45.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f945f7fe640 1 -- 172.21.15.57:0/1427038397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9460153c80 msgr2=0x7f94600b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:45.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f945f7fe640 1 --2- 172.21.15.57:0/1427038397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9460153c80 0x7f94600b2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:45.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.250+0000 7f945f7fe640 1 -- 172.21.15.57:0/1427038397 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9450009420 con 0x7f94600c42a0 2023-10-30T14:32:45.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.254+0000 7f945f7fe640 1 --2- 172.21.15.57:0/1427038397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94600c42a0 0x7f94600c8de0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f945000e980 tx=0x7f945000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:45.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.254+0000 7f9447fff640 1 -- 172.21.15.57:0/1427038397 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945000ca00 con 0x7f94600c42a0 2023-10-30T14:32:45.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.254+0000 7f9447fff640 1 -- 172.21.15.57:0/1427038397 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945000cb60 con 0x7f94600c42a0 2023-10-30T14:32:45.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.254+0000 7f9447fff640 1 -- 172.21.15.57:0/1427038397 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9450013470 con 0x7f94600c42a0 2023-10-30T14:32:45.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.254+0000 7f9465bd3640 1 -- 172.21.15.57:0/1427038397 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9458009a70 con 0x7f94600c42a0 2023-10-30T14:32:45.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.254+0000 7f9465bd3640 1 -- 172.21.15.57:0/1427038397 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94601ef310 con 0x7f94600c42a0 2023-10-30T14:32:45.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.254+0000 7f9465bd3640 1 -- 172.21.15.57:0/1427038397 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946004e170 con 0x7f94600c42a0 2023-10-30T14:32:45.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.254+0000 7f9447fff640 1 -- 172.21.15.57:0/1427038397 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f94500135d0 con 0x7f94600c42a0 2023-10-30T14:32:45.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.254+0000 7f9447fff640 1 --2- 172.21.15.57:0/1427038397 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f944c0664d0 0x7f944c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:45.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.254+0000 7f9447fff640 1 -- 172.21.15.57:0/1427038397 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f94500848d0 con 0x7f94600c42a0 2023-10-30T14:32:45.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.254+0000 7f945effd640 1 --2- 172.21.15.57:0/1427038397 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f944c0664d0 0x7f944c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:45.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.258+0000 7f945effd640 1 --2- 172.21.15.57:0/1427038397 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f944c0664d0 0x7f944c068990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f94600c7070 tx=0x7f9454007d40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:45.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.258+0000 7f9447fff640 1 -- 172.21.15.57:0/1427038397 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9450051900 con 0x7f94600c42a0 2023-10-30T14:32:45.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.498+0000 7f9465bd3640 1 -- 172.21.15.57:0/1427038397 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94600cad40 con 0x7f94600c42a0 2023-10-30T14:32:45.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.498+0000 7f9447fff640 1 -- 172.21.15.57:0/1427038397 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+6533 (secure 0 0 0) 0x7f9450017090 con 0x7f94600c42a0 2023-10-30T14:32:45.500 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:32:45.501 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5891,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:32:34.961195+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:32:45.501 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 110 2023-10-30T14:32:45.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.502+0000 7f9465bd3640 1 -- 172.21.15.57:0/1427038397 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f944c0664d0 msgr2=0x7f944c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:45.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.502+0000 7f9465bd3640 1 --2- 172.21.15.57:0/1427038397 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f944c0664d0 0x7f944c068990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f94600c7070 tx=0x7f9454007d40 comp rx=0 tx=0).stop 2023-10-30T14:32:45.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.502+0000 7f9465bd3640 1 -- 172.21.15.57:0/1427038397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94600c42a0 msgr2=0x7f94600c8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:45.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.502+0000 7f9465bd3640 1 --2- 172.21.15.57:0/1427038397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94600c42a0 0x7f94600c8de0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f945000e980 tx=0x7f945000ee40 comp rx=0 tx=0).stop 2023-10-30T14:32:45.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.502+0000 7f9465bd3640 1 -- 172.21.15.57:0/1427038397 shutdown_connections 2023-10-30T14:32:45.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.502+0000 7f9465bd3640 1 --2- 172.21.15.57:0/1427038397 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f944c0664d0 0x7f944c068990 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:45.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.502+0000 7f9465bd3640 1 --2- 172.21.15.57:0/1427038397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9460153c80 0x7f94600b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:45.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.502+0000 7f9465bd3640 1 --2- 172.21.15.57:0/1427038397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94600c4a10 0x7f94600b2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:45.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.502+0000 7f9465bd3640 1 --2- 172.21.15.57:0/1427038397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94600c42a0 0x7f94600c8de0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:45.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.502+0000 7f9465bd3640 1 -- 172.21.15.57:0/1427038397 >> 172.21.15.57:0/1427038397 conn(0x7f94600715c0 msgr2=0x7f9460158590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:32:45.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.502+0000 7f9465bd3640 1 -- 172.21.15.57:0/1427038397 shutdown_connections 2023-10-30T14:32:45.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:45.502+0000 7f9465bd3640 1 -- 172.21.15.57:0/1427038397 wait complete. 2023-10-30T14:32:45.521 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:32:47.522 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:32:47.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.674+0000 7f8a34ba7640 1 Processor -- start 2023-10-30T14:32:47.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.674+0000 7f8a34ba7640 1 -- start start 2023-10-30T14:32:47.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.674+0000 7f8a34ba7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8a300c8eb0 0x7f8a3015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:47.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.674+0000 7f8a34ba7640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8a3006aeb0 0x7f8a3006b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:47.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.674+0000 7f8a34ba7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a3015dcc0 con 0x7f8a3015ac70 2023-10-30T14:32:47.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.674+0000 7f8a34ba7640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a30159800 con 0x7f8a3006aeb0 2023-10-30T14:32:47.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.674+0000 7f8a34ba7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a30159d80 con 0x7f8a300c8eb0 2023-10-30T14:32:47.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.678+0000 7f8a2ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8a3006aeb0 0x7f8a3006b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:47.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.678+0000 7f8a2ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8a3006aeb0 0x7f8a3006b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48084/0 (socket says 172.21.15.57:48084) 2023-10-30T14:32:47.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.678+0000 7f8a2ffff640 1 -- 172.21.15.57:0/578015631 learned_addr learned my addr 172.21.15.57:0/578015631 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:32:47.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.678+0000 7f8a2f7fe640 1 --2- 172.21.15.57:0/578015631 >> v2:172.21.15.88:3300/0 conn(0x7f8a300c8eb0 0x7f8a3015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:47.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.678+0000 7f8a2ffff640 1 -- 172.21.15.57:0/578015631 >> v2:172.21.15.88:3300/0 conn(0x7f8a300c8eb0 msgr2=0x7f8a3015cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:32:47.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.678+0000 7f8a2ffff640 1 --2- 172.21.15.57:0/578015631 >> v2:172.21.15.88:3300/0 conn(0x7f8a300c8eb0 0x7f8a3015cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:47.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.678+0000 7f8a2ffff640 1 -- 172.21.15.57:0/578015631 >> v1:172.21.15.88:6789/0 conn(0x7f8a3015ac70 legacy=0x7f8a3015a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:47.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.678+0000 7f8a2ffff640 1 -- 172.21.15.57:0/578015631 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3015d420 con 0x7f8a3006aeb0 2023-10-30T14:32:47.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.678+0000 7f8a2ffff640 1 --2- 172.21.15.57:0/578015631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a3006aeb0 0x7f8a3006b300 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f8a3006aa40 tx=0x7f8a2801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=154415797cabb916 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:47.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.678+0000 7f8a2e7fc640 1 -- 172.21.15.57:0/578015631 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a280af400 con 0x7f8a3006aeb0 2023-10-30T14:32:47.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.678+0000 7f8a2e7fc640 1 -- 172.21.15.57:0/578015631 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a280ab050 con 0x7f8a3006aeb0 2023-10-30T14:32:47.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.678+0000 7f8a2e7fc640 1 -- 172.21.15.57:0/578015631 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a280af780 con 0x7f8a3006aeb0 2023-10-30T14:32:47.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.682+0000 7f8a34ba7640 1 -- 172.21.15.57:0/578015631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a3006aeb0 msgr2=0x7f8a3006b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:47.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.682+0000 7f8a34ba7640 1 --2- 172.21.15.57:0/578015631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a3006aeb0 0x7f8a3006b300 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f8a3006aa40 tx=0x7f8a2801f1c0 comp rx=0 tx=0).stop 2023-10-30T14:32:47.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.682+0000 7f8a34ba7640 1 -- 172.21.15.57:0/578015631 shutdown_connections 2023-10-30T14:32:47.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.682+0000 7f8a34ba7640 1 --2- 172.21.15.57:0/578015631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a3006aeb0 0x7f8a3006b300 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:47.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.682+0000 7f8a34ba7640 1 --2- 172.21.15.57:0/578015631 >> v2:172.21.15.88:3300/0 conn(0x7f8a300c8eb0 0x7f8a3015cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:47.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.682+0000 7f8a34ba7640 1 -- 172.21.15.57:0/578015631 >> 172.21.15.57:0/578015631 conn(0x7f8a300cef00 msgr2=0x7f8a300cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:32:47.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.682+0000 7f8a34ba7640 1 -- 172.21.15.57:0/578015631 shutdown_connections 2023-10-30T14:32:47.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.682+0000 7f8a34ba7640 1 -- 172.21.15.57:0/578015631 wait complete. 2023-10-30T14:32:47.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.682+0000 7f8a34ba7640 1 Processor -- start 2023-10-30T14:32:47.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a34ba7640 1 -- start start 2023-10-30T14:32:47.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a34ba7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a3006aeb0 0x7f8a30158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:47.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a34ba7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a300c8eb0 0x7f8a3014dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:47.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a34ba7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a3015ac70 0x7f8a301f2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:47.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a34ba7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a301ef200 con 0x7f8a300c8eb0 2023-10-30T14:32:47.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a34ba7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a301f01d0 con 0x7f8a3006aeb0 2023-10-30T14:32:47.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a34ba7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a30156fb0 con 0x7f8a3015ac70 2023-10-30T14:32:47.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a2effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a300c8eb0 0x7f8a3014dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:47.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a2effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a300c8eb0 0x7f8a3014dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48092/0 (socket says 172.21.15.57:48092) 2023-10-30T14:32:47.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a2effd640 1 -- 172.21.15.57:0/2877445798 learned_addr learned my addr 172.21.15.57:0/2877445798 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:32:47.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a2f7fe640 1 --2- 172.21.15.57:0/2877445798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a3006aeb0 0x7f8a30158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:47.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a2ffff640 1 --2- 172.21.15.57:0/2877445798 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a3015ac70 0x7f8a301f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:47.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a2effd640 1 -- 172.21.15.57:0/2877445798 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a3015ac70 msgr2=0x7f8a301f2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:47.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a2effd640 1 --2- 172.21.15.57:0/2877445798 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a3015ac70 0x7f8a301f2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:47.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a2effd640 1 -- 172.21.15.57:0/2877445798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a3006aeb0 msgr2=0x7f8a30158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:47.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a2effd640 1 --2- 172.21.15.57:0/2877445798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a3006aeb0 0x7f8a30158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:47.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a2effd640 1 -- 172.21.15.57:0/2877445798 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a28009b30 con 0x7f8a300c8eb0 2023-10-30T14:32:47.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a2effd640 1 --2- 172.21.15.57:0/2877445798 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a300c8eb0 0x7f8a3014dd90 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f8a2400c3c0 tx=0x7f8a2400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:47.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a17fff640 1 -- 172.21.15.57:0/2877445798 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a240052d0 con 0x7f8a300c8eb0 2023-10-30T14:32:47.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a17fff640 1 -- 172.21.15.57:0/2877445798 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a24005430 con 0x7f8a300c8eb0 2023-10-30T14:32:47.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a17fff640 1 -- 172.21.15.57:0/2877445798 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a24003840 con 0x7f8a300c8eb0 2023-10-30T14:32:47.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a34ba7640 1 -- 172.21.15.57:0/2877445798 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a301ef410 con 0x7f8a300c8eb0 2023-10-30T14:32:47.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.686+0000 7f8a34ba7640 1 -- 172.21.15.57:0/2877445798 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a301ef9d0 con 0x7f8a300c8eb0 2023-10-30T14:32:47.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.690+0000 7f8a34ba7640 1 -- 172.21.15.57:0/2877445798 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a3004e170 con 0x7f8a300c8eb0 2023-10-30T14:32:47.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.690+0000 7f8a17fff640 1 -- 172.21.15.57:0/2877445798 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8a240039a0 con 0x7f8a300c8eb0 2023-10-30T14:32:47.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.690+0000 7f8a17fff640 1 --2- 172.21.15.57:0/2877445798 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8a1c0664a0 0x7f8a1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:47.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.690+0000 7f8a17fff640 1 -- 172.21.15.57:0/2877445798 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f8a240834d0 con 0x7f8a300c8eb0 2023-10-30T14:32:47.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.690+0000 7f8a2f7fe640 1 --2- 172.21.15.57:0/2877445798 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8a1c0664a0 0x7f8a1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:47.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.694+0000 7f8a2f7fe640 1 --2- 172.21.15.57:0/2877445798 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8a1c0664a0 0x7f8a1c068960 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8a20009b60 tx=0x7f8a20009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:47.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.694+0000 7f8a17fff640 1 -- 172.21.15.57:0/2877445798 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8a240506c0 con 0x7f8a300c8eb0 2023-10-30T14:32:47.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.930+0000 7f8a34ba7640 1 -- 172.21.15.57:0/2877445798 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a3015dcc0 con 0x7f8a300c8eb0 2023-10-30T14:32:47.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.930+0000 7f8a17fff640 1 -- 172.21.15.57:0/2877445798 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+6533 (secure 0 0 0) 0x7f8a3015dcc0 con 0x7f8a300c8eb0 2023-10-30T14:32:47.932 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:32:47.932 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5891,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:32:34.961195+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:32:47.932 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 110 2023-10-30T14:32:47.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.934+0000 7f8a34ba7640 1 -- 172.21.15.57:0/2877445798 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8a1c0664a0 msgr2=0x7f8a1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:47.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.934+0000 7f8a34ba7640 1 --2- 172.21.15.57:0/2877445798 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8a1c0664a0 0x7f8a1c068960 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8a20009b60 tx=0x7f8a20009190 comp rx=0 tx=0).stop 2023-10-30T14:32:47.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.934+0000 7f8a34ba7640 1 -- 172.21.15.57:0/2877445798 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a300c8eb0 msgr2=0x7f8a3014dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:47.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.934+0000 7f8a34ba7640 1 --2- 172.21.15.57:0/2877445798 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a300c8eb0 0x7f8a3014dd90 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f8a2400c3c0 tx=0x7f8a2400c880 comp rx=0 tx=0).stop 2023-10-30T14:32:47.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.934+0000 7f8a34ba7640 1 -- 172.21.15.57:0/2877445798 shutdown_connections 2023-10-30T14:32:47.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.934+0000 7f8a34ba7640 1 --2- 172.21.15.57:0/2877445798 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8a1c0664a0 0x7f8a1c068960 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:47.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.934+0000 7f8a34ba7640 1 --2- 172.21.15.57:0/2877445798 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a3015ac70 0x7f8a301f2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:47.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.934+0000 7f8a34ba7640 1 --2- 172.21.15.57:0/2877445798 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a300c8eb0 0x7f8a3014dd90 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:47.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.934+0000 7f8a34ba7640 1 --2- 172.21.15.57:0/2877445798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a3006aeb0 0x7f8a30158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:47.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.934+0000 7f8a34ba7640 1 -- 172.21.15.57:0/2877445798 >> 172.21.15.57:0/2877445798 conn(0x7f8a300cef00 msgr2=0x7f8a300c6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:32:47.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.934+0000 7f8a34ba7640 1 -- 172.21.15.57:0/2877445798 shutdown_connections 2023-10-30T14:32:47.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:47.934+0000 7f8a34ba7640 1 -- 172.21.15.57:0/2877445798 wait complete. 2023-10-30T14:32:47.952 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:32:49.953 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:32:50.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.106+0000 7f163ef80640 1 Processor -- start 2023-10-30T14:32:50.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.106+0000 7f163ef80640 1 -- start start 2023-10-30T14:32:50.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f163ef80640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16380c4a10 0x7f163815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:50.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f163ef80640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1638159f20 con 0x7f16380c42a0 2023-10-30T14:32:50.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f163ef80640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f163815d620 con 0x7f1638153c80 2023-10-30T14:32:50.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f163ef80640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163815dec0 con 0x7f16380c4a10 2023-10-30T14:32:50.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f163d51d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f16380c42a0 0x7f16380ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:34616/0 (socket says 172.21.15.57:34616) 2023-10-30T14:32:50.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f163d51d640 1 -- 172.21.15.57:0/1837927079 learned_addr learned my addr 172.21.15.57:0/1837927079 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:32:50.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f1637fff640 1 --2- 172.21.15.57:0/1837927079 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16380c4a10 0x7f163815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:50.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f16377fe640 1 -- 172.21.15.57:0/1837927079 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408100343 0 0) 0x7f1638159f20 con 0x7f16380c42a0 2023-10-30T14:32:50.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f16377fe640 1 -- 172.21.15.57:0/1837927079 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1628003530 con 0x7f16380c42a0 2023-10-30T14:32:50.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f16377fe640 1 -- 172.21.15.57:0/1837927079 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3158184065 0 0) 0x7f1628003530 con 0x7f16380c42a0 2023-10-30T14:32:50.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f16377fe640 1 -- 172.21.15.57:0/1837927079 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16380c4a10 msgr2=0x7f163815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:32:50.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f16377fe640 1 --2- 172.21.15.57:0/1837927079 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16380c4a10 0x7f163815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:50.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f16377fe640 1 -- 172.21.15.57:0/1837927079 >> v1:172.21.15.88:6789/0 conn(0x7f1638153c80 legacy=0x7f163815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:50.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f16377fe640 1 -- 172.21.15.57:0/1837927079 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163815d900 con 0x7f16380c42a0 2023-10-30T14:32:50.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f16377fe640 1 -- 172.21.15.57:0/1837927079 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f1630001a00 con 0x7f16380c42a0 2023-10-30T14:32:50.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f16377fe640 1 -- 172.21.15.57:0/1837927079 >> v1:172.21.15.57:6789/0 conn(0x7f16380c42a0 legacy=0x7f16380ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:50.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f16377fe640 1 --2- 172.21.15.57:0/1837927079 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16380c4a10 0x7f16280039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:50.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f16377fe640 1 --2- 172.21.15.57:0/1837927079 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1628003f30 0x7f16280047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:50.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f16377fe640 1 --2- 172.21.15.57:0/1837927079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1628004d10 0x7f1628005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:50.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f16377fe640 1 -- 172.21.15.57:0/1837927079 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163815dec0 con 0x7f1628003f30 2023-10-30T14:32:50.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f16377fe640 1 -- 172.21.15.57:0/1837927079 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162800bb00 con 0x7f1628004d10 2023-10-30T14:32:50.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f16377fe640 1 -- 172.21.15.57:0/1837927079 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f162800bcc0 con 0x7f16380c4a10 2023-10-30T14:32:50.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f1637fff640 1 --2- 172.21.15.57:0/1837927079 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1628003f30 0x7f16280047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:50.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f163cd1c640 1 --2- 172.21.15.57:0/1837927079 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16380c4a10 0x7f16280039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:50.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f163d51d640 1 --2- 172.21.15.57:0/1837927079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1628004d10 0x7f1628005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:50.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f1637fff640 1 -- 172.21.15.57:0/1837927079 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16380c4a10 msgr2=0x7f16280039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:32:50.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f1637fff640 1 --2- 172.21.15.57:0/1837927079 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16380c4a10 0x7f16280039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:50.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f1637fff640 1 -- 172.21.15.57:0/1837927079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1628004d10 msgr2=0x7f1628005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:32:50.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f1637fff640 1 --2- 172.21.15.57:0/1837927079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1628004d10 0x7f1628005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:50.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.110+0000 7f1637fff640 1 -- 172.21.15.57:0/1837927079 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1628003530 con 0x7f1628003f30 2023-10-30T14:32:50.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.114+0000 7f1637fff640 1 --2- 172.21.15.57:0/1837927079 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1628003f30 0x7f16280047d0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f1624009ab0 tx=0x7f162401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b30b2dfff9c810d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:50.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.114+0000 7f16377fe640 1 -- 172.21.15.57:0/1837927079 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16240a5620 con 0x7f1628003f30 2023-10-30T14:32:50.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.114+0000 7f16377fe640 1 -- 172.21.15.57:0/1837927079 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16240a5780 con 0x7f1628003f30 2023-10-30T14:32:50.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.114+0000 7f16377fe640 1 -- 172.21.15.57:0/1837927079 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16240a5b30 con 0x7f1628003f30 2023-10-30T14:32:50.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.114+0000 7f163ef80640 1 -- 172.21.15.57:0/1837927079 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1628003f30 msgr2=0x7f16280047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:50.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.114+0000 7f163ef80640 1 --2- 172.21.15.57:0/1837927079 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1628003f30 0x7f16280047d0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f1624009ab0 tx=0x7f162401ca30 comp rx=0 tx=0).stop 2023-10-30T14:32:50.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.114+0000 7f163d51d640 1 -- 172.21.15.57:0/1837927079 reap_dead start 2023-10-30T14:32:50.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.114+0000 7f163ef80640 1 -- 172.21.15.57:0/1837927079 shutdown_connections 2023-10-30T14:32:50.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.114+0000 7f163ef80640 1 -- 172.21.15.57:0/1837927079 >> 172.21.15.57:0/1837927079 conn(0x7f16380715c0 msgr2=0x7f16380719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:32:50.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.114+0000 7f163ef80640 1 -- 172.21.15.57:0/1837927079 shutdown_connections 2023-10-30T14:32:50.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.114+0000 7f163ef80640 1 -- 172.21.15.57:0/1837927079 wait complete. 2023-10-30T14:32:50.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163ef80640 1 Processor -- start 2023-10-30T14:32:50.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163ef80640 1 -- start start 2023-10-30T14:32:50.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163ef80640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1628003f30 0x7f16380c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:50.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163ef80640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f163815ced0 0x7f163815bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:50.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163ef80640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16381f2300 0x7f16381f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:50.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163ef80640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16381eed30 con 0x7f1628003f30 2023-10-30T14:32:50.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163ef80640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163815c5d0 con 0x7f16381f2300 2023-10-30T14:32:50.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163ef80640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16381eeff0 con 0x7f163815ced0 2023-10-30T14:32:50.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163cd1c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1628003f30 0x7f16380c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:50.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163cd1c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1628003f30 0x7f16380c8e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48106/0 (socket says 172.21.15.57:48106) 2023-10-30T14:32:50.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163cd1c640 1 -- 172.21.15.57:0/831318293 learned_addr learned my addr 172.21.15.57:0/831318293 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:32:50.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f1637fff640 1 --2- 172.21.15.57:0/831318293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f163815ced0 0x7f163815bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:50.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163d51d640 1 --2- 172.21.15.57:0/831318293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16381f2300 0x7f16381f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:50.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163cd1c640 1 -- 172.21.15.57:0/831318293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f163815ced0 msgr2=0x7f163815bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:50.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163cd1c640 1 --2- 172.21.15.57:0/831318293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f163815ced0 0x7f163815bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:50.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163cd1c640 1 -- 172.21.15.57:0/831318293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16381f2300 msgr2=0x7f16381f27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:50.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163cd1c640 1 --2- 172.21.15.57:0/831318293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16381f2300 0x7f16381f27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:50.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163cd1c640 1 -- 172.21.15.57:0/831318293 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16381ef2d0 con 0x7f1628003f30 2023-10-30T14:32:50.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163cd1c640 1 --2- 172.21.15.57:0/831318293 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1628003f30 0x7f16380c8e60 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f162c00f7f0 tx=0x7f162c00fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:50.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f16357fa640 1 -- 172.21.15.57:0/831318293 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c0050d0 con 0x7f1628003f30 2023-10-30T14:32:50.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f16357fa640 1 -- 172.21.15.57:0/831318293 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c005230 con 0x7f1628003f30 2023-10-30T14:32:50.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f16357fa640 1 -- 172.21.15.57:0/831318293 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c00d730 con 0x7f1628003f30 2023-10-30T14:32:50.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163ef80640 1 -- 172.21.15.57:0/831318293 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16381ef5b0 con 0x7f1628003f30 2023-10-30T14:32:50.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163ef80640 1 -- 172.21.15.57:0/831318293 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f163815afa0 con 0x7f1628003f30 2023-10-30T14:32:50.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.118+0000 7f163ef80640 1 -- 172.21.15.57:0/831318293 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1638159b70 con 0x7f1628003f30 2023-10-30T14:32:50.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.122+0000 7f16357fa640 1 -- 172.21.15.57:0/831318293 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f162c00d890 con 0x7f1628003f30 2023-10-30T14:32:50.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.122+0000 7f16357fa640 1 --2- 172.21.15.57:0/831318293 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f161c0664d0 0x7f161c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:50.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.122+0000 7f16357fa640 1 -- 172.21.15.57:0/831318293 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f162c0842c0 con 0x7f1628003f30 2023-10-30T14:32:50.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.122+0000 7f1637fff640 1 --2- 172.21.15.57:0/831318293 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f161c0664d0 0x7f161c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:50.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.122+0000 7f1637fff640 1 --2- 172.21.15.57:0/831318293 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f161c0664d0 0x7f161c068990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f16240af000 tx=0x7f16240046d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:50.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.126+0000 7f16357fa640 1 -- 172.21.15.57:0/831318293 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f162c0512d0 con 0x7f1628003f30 2023-10-30T14:32:50.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.318+0000 7f163ef80640 1 -- 172.21.15.57:0/831318293 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f163804e170 con 0x7f1628003f30 2023-10-30T14:32:50.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.318+0000 7f16357fa640 1 -- 172.21.15.57:0/831318293 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+6533 (secure 0 0 0) 0x7f162c0548f0 con 0x7f1628003f30 2023-10-30T14:32:50.319 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:32:50.319 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5891,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:32:34.961195+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:32:50.320 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 110 2023-10-30T14:32:50.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.318+0000 7f163ef80640 1 -- 172.21.15.57:0/831318293 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f161c0664d0 msgr2=0x7f161c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:50.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.318+0000 7f163ef80640 1 --2- 172.21.15.57:0/831318293 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f161c0664d0 0x7f161c068990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f16240af000 tx=0x7f16240046d0 comp rx=0 tx=0).stop 2023-10-30T14:32:50.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.318+0000 7f163ef80640 1 -- 172.21.15.57:0/831318293 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1628003f30 msgr2=0x7f16380c8e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:50.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.318+0000 7f163ef80640 1 --2- 172.21.15.57:0/831318293 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1628003f30 0x7f16380c8e60 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f162c00f7f0 tx=0x7f162c00fcb0 comp rx=0 tx=0).stop 2023-10-30T14:32:50.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.318+0000 7f163ef80640 1 -- 172.21.15.57:0/831318293 shutdown_connections 2023-10-30T14:32:50.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.318+0000 7f163ef80640 1 --2- 172.21.15.57:0/831318293 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f161c0664d0 0x7f161c068990 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:50.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.318+0000 7f163ef80640 1 --2- 172.21.15.57:0/831318293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16381f2300 0x7f16381f27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:50.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.318+0000 7f163ef80640 1 --2- 172.21.15.57:0/831318293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f163815ced0 0x7f163815bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:50.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.318+0000 7f163ef80640 1 --2- 172.21.15.57:0/831318293 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1628003f30 0x7f16380c8e60 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:50.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.318+0000 7f163ef80640 1 -- 172.21.15.57:0/831318293 >> 172.21.15.57:0/831318293 conn(0x7f16380715c0 msgr2=0x7f1638152210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:32:50.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.318+0000 7f163ef80640 1 -- 172.21.15.57:0/831318293 shutdown_connections 2023-10-30T14:32:50.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:50.322+0000 7f163ef80640 1 -- 172.21.15.57:0/831318293 wait complete. 2023-10-30T14:32:50.340 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:32:52.341 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:32:52.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.494+0000 7fe881891640 1 Processor -- start 2023-10-30T14:32:52.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.494+0000 7fe881891640 1 -- start start 2023-10-30T14:32:52.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.494+0000 7fe881891640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe87c0c55d0 0x7fe87c06e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:52.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.494+0000 7fe881891640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe87c151b90 0x7fe87c15b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:52.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.494+0000 7fe881891640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe87c1578e0 con 0x7fe87c06eb70 2023-10-30T14:32:52.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.494+0000 7fe881891640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe87c158720 con 0x7fe87c151b90 2023-10-30T14:32:52.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.494+0000 7fe881891640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c1576f0 con 0x7fe87c0c55d0 2023-10-30T14:32:52.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.498+0000 7fe87b7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe87c151b90 0x7fe87c15b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:52.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.498+0000 7fe87b7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe87c151b90 0x7fe87c15b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48116/0 (socket says 172.21.15.57:48116) 2023-10-30T14:32:52.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.498+0000 7fe87b7fe640 1 -- 172.21.15.57:0/2640172447 learned_addr learned my addr 172.21.15.57:0/2640172447 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:32:52.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.498+0000 7fe87affd640 1 --2- 172.21.15.57:0/2640172447 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe87c0c55d0 0x7fe87c06e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:52.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.498+0000 7fe87b7fe640 1 -- 172.21.15.57:0/2640172447 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe87c0c55d0 msgr2=0x7fe87c06e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:32:52.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.498+0000 7fe87b7fe640 1 --2- 172.21.15.57:0/2640172447 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe87c0c55d0 0x7fe87c06e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:52.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.498+0000 7fe87b7fe640 1 -- 172.21.15.57:0/2640172447 >> v1:172.21.15.88:6789/0 conn(0x7fe87c06eb70 legacy=0x7fe87c151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:52.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.498+0000 7fe87b7fe640 1 -- 172.21.15.57:0/2640172447 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c157ba0 con 0x7fe87c151b90 2023-10-30T14:32:52.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.498+0000 7fe87b7fe640 1 --2- 172.21.15.57:0/2640172447 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87c151b90 0x7fe87c15b0c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fe874009e50 tx=0x7fe87401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=698158b9afcc4696 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:52.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.498+0000 7fe87a7fc640 1 -- 172.21.15.57:0/2640172447 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8740af400 con 0x7fe87c151b90 2023-10-30T14:32:52.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.498+0000 7fe87a7fc640 1 -- 172.21.15.57:0/2640172447 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8740ab050 con 0x7fe87c151b90 2023-10-30T14:32:52.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.498+0000 7fe87a7fc640 1 -- 172.21.15.57:0/2640172447 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8740af780 con 0x7fe87c151b90 2023-10-30T14:32:52.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe881891640 1 -- 172.21.15.57:0/2640172447 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87c151b90 msgr2=0x7fe87c15b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:52.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe881891640 1 --2- 172.21.15.57:0/2640172447 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87c151b90 0x7fe87c15b0c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fe874009e50 tx=0x7fe87401f1f0 comp rx=0 tx=0).stop 2023-10-30T14:32:52.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe881891640 1 -- 172.21.15.57:0/2640172447 shutdown_connections 2023-10-30T14:32:52.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe881891640 1 --2- 172.21.15.57:0/2640172447 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87c151b90 0x7fe87c15b0c0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:52.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe881891640 1 --2- 172.21.15.57:0/2640172447 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe87c0c55d0 0x7fe87c06e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:52.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe881891640 1 -- 172.21.15.57:0/2640172447 >> 172.21.15.57:0/2640172447 conn(0x7fe87c06afa0 msgr2=0x7fe87c06b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:32:52.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe881891640 1 -- 172.21.15.57:0/2640172447 shutdown_connections 2023-10-30T14:32:52.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe881891640 1 -- 172.21.15.57:0/2640172447 wait complete. 2023-10-30T14:32:52.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe881891640 1 Processor -- start 2023-10-30T14:32:52.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe881891640 1 -- start start 2023-10-30T14:32:52.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe881891640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe87c06eb70 0x7fe87c0c2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:52.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe881891640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87c0c55d0 0x7fe87c0721e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:52.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe881891640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe87c151b90 0x7fe87c072720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:52.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe881891640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c1efb40 con 0x7fe87c0c55d0 2023-10-30T14:32:52.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe881891640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c1eec70 con 0x7fe87c06eb70 2023-10-30T14:32:52.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe881891640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c0b13a0 con 0x7fe87c151b90 2023-10-30T14:32:52.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe873fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87c0c55d0 0x7fe87c0721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:52.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe873fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87c0c55d0 0x7fe87c0721e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48122/0 (socket says 172.21.15.57:48122) 2023-10-30T14:32:52.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe873fff640 1 -- 172.21.15.57:0/3051663834 learned_addr learned my addr 172.21.15.57:0/3051663834 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:32:52.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe87affd640 1 --2- 172.21.15.57:0/3051663834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe87c06eb70 0x7fe87c0c2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:52.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.502+0000 7fe87b7fe640 1 --2- 172.21.15.57:0/3051663834 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe87c151b90 0x7fe87c072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:52.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.506+0000 7fe873fff640 1 -- 172.21.15.57:0/3051663834 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe87c151b90 msgr2=0x7fe87c072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:52.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.506+0000 7fe873fff640 1 --2- 172.21.15.57:0/3051663834 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe87c151b90 0x7fe87c072720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:52.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.506+0000 7fe873fff640 1 -- 172.21.15.57:0/3051663834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe87c06eb70 msgr2=0x7fe87c0c2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:52.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.506+0000 7fe873fff640 1 --2- 172.21.15.57:0/3051663834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe87c06eb70 0x7fe87c0c2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:52.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.506+0000 7fe873fff640 1 -- 172.21.15.57:0/3051663834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe874009b30 con 0x7fe87c0c55d0 2023-10-30T14:32:52.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.506+0000 7fe873fff640 1 --2- 172.21.15.57:0/3051663834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87c0c55d0 0x7fe87c0721e0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fe86400c530 tx=0x7fe86400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:52.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.506+0000 7fe878ff9640 1 -- 172.21.15.57:0/3051663834 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe864005220 con 0x7fe87c0c55d0 2023-10-30T14:32:52.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.506+0000 7fe878ff9640 1 -- 172.21.15.57:0/3051663834 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe864005380 con 0x7fe87c0c55d0 2023-10-30T14:32:52.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.506+0000 7fe878ff9640 1 -- 172.21.15.57:0/3051663834 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe864007270 con 0x7fe87c0c55d0 2023-10-30T14:32:52.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.506+0000 7fe881891640 1 -- 172.21.15.57:0/3051663834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe87c1eeec0 con 0x7fe87c0c55d0 2023-10-30T14:32:52.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.506+0000 7fe881891640 1 -- 172.21.15.57:0/3051663834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe87c1ef480 con 0x7fe87c0c55d0 2023-10-30T14:32:52.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.506+0000 7fe881891640 1 -- 172.21.15.57:0/3051663834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe87c04e170 con 0x7fe87c0c55d0 2023-10-30T14:32:52.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.506+0000 7fe878ff9640 1 -- 172.21.15.57:0/3051663834 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe8640073d0 con 0x7fe87c0c55d0 2023-10-30T14:32:52.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.510+0000 7fe878ff9640 1 --2- 172.21.15.57:0/3051663834 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe85c0664a0 0x7fe85c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:52.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.510+0000 7fe878ff9640 1 -- 172.21.15.57:0/3051663834 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7fe864083990 con 0x7fe87c0c55d0 2023-10-30T14:32:52.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.510+0000 7fe87affd640 1 --2- 172.21.15.57:0/3051663834 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe85c0664a0 0x7fe85c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:52.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.510+0000 7fe87affd640 1 --2- 172.21.15.57:0/3051663834 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe85c0664a0 0x7fe85c068960 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe86c009b60 tx=0x7fe86c009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:52.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.514+0000 7fe878ff9640 1 -- 172.21.15.57:0/3051663834 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe864050b80 con 0x7fe87c0c55d0 2023-10-30T14:32:52.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.706+0000 7fe881891640 1 -- 172.21.15.57:0/3051663834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe87c0c8950 con 0x7fe87c0c55d0 2023-10-30T14:32:52.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.706+0000 7fe878ff9640 1 -- 172.21.15.57:0/3051663834 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+6533 (secure 0 0 0) 0x7fe8640541a0 con 0x7fe87c0c55d0 2023-10-30T14:32:52.707 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:32:52.707 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5891,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:32:34.961195+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6027,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6836/2733323524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2733323524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2733323524}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:32:52.707 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 110 2023-10-30T14:32:52.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.706+0000 7fe881891640 1 -- 172.21.15.57:0/3051663834 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe85c0664a0 msgr2=0x7fe85c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:52.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.706+0000 7fe881891640 1 --2- 172.21.15.57:0/3051663834 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe85c0664a0 0x7fe85c068960 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe86c009b60 tx=0x7fe86c009190 comp rx=0 tx=0).stop 2023-10-30T14:32:52.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.706+0000 7fe881891640 1 -- 172.21.15.57:0/3051663834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87c0c55d0 msgr2=0x7fe87c0721e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:52.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.706+0000 7fe881891640 1 --2- 172.21.15.57:0/3051663834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87c0c55d0 0x7fe87c0721e0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fe86400c530 tx=0x7fe86400c9f0 comp rx=0 tx=0).stop 2023-10-30T14:32:52.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.706+0000 7fe881891640 1 -- 172.21.15.57:0/3051663834 shutdown_connections 2023-10-30T14:32:52.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.706+0000 7fe881891640 1 --2- 172.21.15.57:0/3051663834 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe85c0664a0 0x7fe85c068960 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:52.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.706+0000 7fe881891640 1 --2- 172.21.15.57:0/3051663834 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe87c151b90 0x7fe87c072720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:52.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.706+0000 7fe881891640 1 --2- 172.21.15.57:0/3051663834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87c0c55d0 0x7fe87c0721e0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:52.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.706+0000 7fe881891640 1 --2- 172.21.15.57:0/3051663834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe87c06eb70 0x7fe87c0c2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:52.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.706+0000 7fe881891640 1 -- 172.21.15.57:0/3051663834 >> 172.21.15.57:0/3051663834 conn(0x7fe87c06afa0 msgr2=0x7fe87c0c7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:32:52.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.706+0000 7fe881891640 1 -- 172.21.15.57:0/3051663834 shutdown_connections 2023-10-30T14:32:52.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:52.706+0000 7fe881891640 1 -- 172.21.15.57:0/3051663834 wait complete. 2023-10-30T14:32:52.727 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:32:54.728 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:32:54.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.882+0000 7f4e95079640 1 Processor -- start 2023-10-30T14:32:54.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.882+0000 7f4e95079640 1 -- start start 2023-10-30T14:32:54.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.882+0000 7f4e95079640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4e900cb7d0 0x7f4e900c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:54.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.882+0000 7f4e95079640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4e900c3e40 0x7f4e90070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:54.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.882+0000 7f4e95079640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e900c2310 con 0x7f4e90150830 2023-10-30T14:32:54.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.882+0000 7f4e95079640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e9006bc10 con 0x7f4e900c3e40 2023-10-30T14:32:54.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.882+0000 7f4e95079640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e90156710 con 0x7f4e900cb7d0 2023-10-30T14:32:54.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.882+0000 7f4e8f59e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4e900c3e40 0x7f4e90070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:54.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.882+0000 7f4e8f59e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4e900c3e40 0x7f4e90070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57970/0 (socket says 172.21.15.57:57970) 2023-10-30T14:32:54.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.882+0000 7f4e8f59e640 1 -- 172.21.15.57:0/3881274517 learned_addr learned my addr 172.21.15.57:0/3881274517 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:32:54.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.882+0000 7f4e8e59c640 1 --2- 172.21.15.57:0/3881274517 >> v2:172.21.15.88:3300/0 conn(0x7f4e900cb7d0 0x7f4e900c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:54.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.882+0000 7f4e8f59e640 1 -- 172.21.15.57:0/3881274517 >> v2:172.21.15.88:3300/0 conn(0x7f4e900cb7d0 msgr2=0x7f4e900c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:32:54.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.882+0000 7f4e8f59e640 1 --2- 172.21.15.57:0/3881274517 >> v2:172.21.15.88:3300/0 conn(0x7f4e900cb7d0 0x7f4e900c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:54.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.882+0000 7f4e8f59e640 1 -- 172.21.15.57:0/3881274517 >> v1:172.21.15.88:6789/0 conn(0x7f4e90150830 legacy=0x7f4e9006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:54.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.882+0000 7f4e8f59e640 1 -- 172.21.15.57:0/3881274517 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e900c4360 con 0x7f4e900c3e40 2023-10-30T14:32:54.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.886+0000 7f4e8f59e640 1 --2- 172.21.15.57:0/3881274517 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e900c3e40 0x7f4e90070520 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f4e9006f9f0 tx=0x7f4e8801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d2da3b009f522ed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:54.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.886+0000 7f4e8dd9b640 1 -- 172.21.15.57:0/3881274517 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e880af400 con 0x7f4e900c3e40 2023-10-30T14:32:54.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.886+0000 7f4e8dd9b640 1 -- 172.21.15.57:0/3881274517 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e880ab050 con 0x7f4e900c3e40 2023-10-30T14:32:54.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.886+0000 7f4e8dd9b640 1 -- 172.21.15.57:0/3881274517 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e880af780 con 0x7f4e900c3e40 2023-10-30T14:32:54.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.890+0000 7f4e95079640 1 -- 172.21.15.57:0/3881274517 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e900c3e40 msgr2=0x7f4e90070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:54.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.890+0000 7f4e95079640 1 --2- 172.21.15.57:0/3881274517 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e900c3e40 0x7f4e90070520 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f4e9006f9f0 tx=0x7f4e8801f1c0 comp rx=0 tx=0).stop 2023-10-30T14:32:54.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.890+0000 7f4e95079640 1 -- 172.21.15.57:0/3881274517 shutdown_connections 2023-10-30T14:32:54.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.890+0000 7f4e95079640 1 --2- 172.21.15.57:0/3881274517 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e900c3e40 0x7f4e90070520 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:54.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.890+0000 7f4e95079640 1 --2- 172.21.15.57:0/3881274517 >> v2:172.21.15.88:3300/0 conn(0x7f4e900cb7d0 0x7f4e900c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:54.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.890+0000 7f4e95079640 1 -- 172.21.15.57:0/3881274517 >> 172.21.15.57:0/3881274517 conn(0x7f4e900cac70 msgr2=0x7f4e900cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:32:54.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.890+0000 7f4e95079640 1 -- 172.21.15.57:0/3881274517 shutdown_connections 2023-10-30T14:32:54.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.890+0000 7f4e95079640 1 -- 172.21.15.57:0/3881274517 wait complete. 2023-10-30T14:32:54.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.890+0000 7f4e95079640 1 Processor -- start 2023-10-30T14:32:54.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.890+0000 7f4e95079640 1 -- start start 2023-10-30T14:32:54.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.890+0000 7f4e95079640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e900c3e40 0x7f4e900ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:54.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.890+0000 7f4e95079640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e900cb7d0 0x7f4e90073620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:54.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.890+0000 7f4e95079640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e90150830 0x7f4e901eea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:54.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.890+0000 7f4e95079640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e90152720 con 0x7f4e90150830 2023-10-30T14:32:54.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.890+0000 7f4e95079640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e900c9700 con 0x7f4e900c3e40 2023-10-30T14:32:54.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.890+0000 7f4e95079640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e90153fd0 con 0x7f4e900cb7d0 2023-10-30T14:32:54.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.890+0000 7f4e8f59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e90150830 0x7f4e901eea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:54.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.890+0000 7f4e8f59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e90150830 0x7f4e901eea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57982/0 (socket says 172.21.15.57:57982) 2023-10-30T14:32:54.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.890+0000 7f4e8f59e640 1 -- 172.21.15.57:0/1360427513 learned_addr learned my addr 172.21.15.57:0/1360427513 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:32:54.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.894+0000 7f4e8ed9d640 1 --2- 172.21.15.57:0/1360427513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e900c3e40 0x7f4e900ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:54.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.894+0000 7f4e8e59c640 1 --2- 172.21.15.57:0/1360427513 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e900cb7d0 0x7f4e90073620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:54.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.894+0000 7f4e8f59e640 1 -- 172.21.15.57:0/1360427513 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e900cb7d0 msgr2=0x7f4e90073620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:54.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.894+0000 7f4e8f59e640 1 --2- 172.21.15.57:0/1360427513 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e900cb7d0 0x7f4e90073620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:54.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.894+0000 7f4e8f59e640 1 -- 172.21.15.57:0/1360427513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e900c3e40 msgr2=0x7f4e900ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:54.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.894+0000 7f4e8f59e640 1 --2- 172.21.15.57:0/1360427513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e900c3e40 0x7f4e900ca380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:54.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.894+0000 7f4e8f59e640 1 -- 172.21.15.57:0/1360427513 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e88009b30 con 0x7f4e90150830 2023-10-30T14:32:54.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.894+0000 7f4e8f59e640 1 --2- 172.21.15.57:0/1360427513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e90150830 0x7f4e901eea40 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f4e880b2040 tx=0x7f4e8801bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:54.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.894+0000 7f4e777fe640 1 -- 172.21.15.57:0/1360427513 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e880afd60 con 0x7f4e90150830 2023-10-30T14:32:54.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.894+0000 7f4e777fe640 1 -- 172.21.15.57:0/1360427513 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e88009510 con 0x7f4e90150830 2023-10-30T14:32:54.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.894+0000 7f4e777fe640 1 -- 172.21.15.57:0/1360427513 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e880ba660 con 0x7f4e90150830 2023-10-30T14:32:54.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.894+0000 7f4e95079640 1 -- 172.21.15.57:0/1360427513 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e900caa30 con 0x7f4e90150830 2023-10-30T14:32:54.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.894+0000 7f4e95079640 1 -- 172.21.15.57:0/1360427513 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e900c8dd0 con 0x7f4e90150830 2023-10-30T14:32:54.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.894+0000 7f4e95079640 1 -- 172.21.15.57:0/1360427513 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e9004e170 con 0x7f4e90150830 2023-10-30T14:32:54.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.894+0000 7f4e777fe640 1 -- 172.21.15.57:0/1360427513 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4e880b6040 con 0x7f4e90150830 2023-10-30T14:32:54.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.894+0000 7f4e777fe640 1 --2- 172.21.15.57:0/1360427513 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4e7c0664d0 0x7f4e7c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:54.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.898+0000 7f4e777fe640 1 -- 172.21.15.57:0/1360427513 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6685+0+0 (secure 0 0 0) 0x7f4e88122330 con 0x7f4e90150830 2023-10-30T14:32:54.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.898+0000 7f4e8ed9d640 1 --2- 172.21.15.57:0/1360427513 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4e7c0664d0 0x7f4e7c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:54.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.898+0000 7f4e8ed9d640 1 --2- 172.21.15.57:0/1360427513 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4e7c0664d0 0x7f4e7c068990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f4e80007fd0 tx=0x7f4e80009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:54.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:54.902+0000 7f4e777fe640 1 -- 172.21.15.57:0/1360427513 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4e880ef510 con 0x7f4e90150830 2023-10-30T14:32:55.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.106+0000 7f4e95079640 1 -- 172.21.15.57:0/1360427513 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e90066510 con 0x7f4e90150830 2023-10-30T14:32:55.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.106+0000 7f4e777fe640 1 -- 172.21.15.57:0/1360427513 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5740 (secure 0 0 0) 0x7f4e880f2b30 con 0x7f4e90150830 2023-10-30T14:32:55.109 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:32:55.109 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:32:54.940721+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:32:55.110 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 111 2023-10-30T14:32:55.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.110+0000 7f4e95079640 1 -- 172.21.15.57:0/1360427513 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4e7c0664d0 msgr2=0x7f4e7c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:55.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.110+0000 7f4e95079640 1 --2- 172.21.15.57:0/1360427513 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4e7c0664d0 0x7f4e7c068990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f4e80007fd0 tx=0x7f4e80009040 comp rx=0 tx=0).stop 2023-10-30T14:32:55.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.110+0000 7f4e95079640 1 -- 172.21.15.57:0/1360427513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e90150830 msgr2=0x7f4e901eea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:55.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.110+0000 7f4e95079640 1 --2- 172.21.15.57:0/1360427513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e90150830 0x7f4e901eea40 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f4e880b2040 tx=0x7f4e8801bf60 comp rx=0 tx=0).stop 2023-10-30T14:32:55.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.110+0000 7f4e95079640 1 -- 172.21.15.57:0/1360427513 shutdown_connections 2023-10-30T14:32:55.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.110+0000 7f4e95079640 1 --2- 172.21.15.57:0/1360427513 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4e7c0664d0 0x7f4e7c068990 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:55.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.110+0000 7f4e95079640 1 --2- 172.21.15.57:0/1360427513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e90150830 0x7f4e901eea40 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:55.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.110+0000 7f4e95079640 1 --2- 172.21.15.57:0/1360427513 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e900cb7d0 0x7f4e90073620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:55.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.110+0000 7f4e95079640 1 --2- 172.21.15.57:0/1360427513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e900c3e40 0x7f4e900ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:55.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.110+0000 7f4e95079640 1 -- 172.21.15.57:0/1360427513 >> 172.21.15.57:0/1360427513 conn(0x7f4e900cac70 msgr2=0x7f4e90154470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:32:55.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.110+0000 7f4e95079640 1 -- 172.21.15.57:0/1360427513 shutdown_connections 2023-10-30T14:32:55.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.110+0000 7f4e95079640 1 -- 172.21.15.57:0/1360427513 wait complete. 2023-10-30T14:32:55.138 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T14:32:55.138 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:32:55.139 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:32:55.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.282+0000 7f84dba79640 1 Processor -- start 2023-10-30T14:32:55.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.282+0000 7f84dba79640 1 -- start start 2023-10-30T14:32:55.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.282+0000 7f84dba79640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f84d4150830 0x7f84d406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:55.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.282+0000 7f84dba79640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84d4070520 0x7f84d40709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:55.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.282+0000 7f84dba79640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d40c2310 con 0x7f84d40cb7d0 2023-10-30T14:32:55.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.282+0000 7f84dba79640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84d4156710 con 0x7f84d4150830 2023-10-30T14:32:55.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.282+0000 7f84dba79640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d40c4360 con 0x7f84d4070520 2023-10-30T14:32:55.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.286+0000 7f84d9815640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f84d4150830 0x7f84d406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:55.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.286+0000 7f84d9815640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f84d4150830 0x7f84d406b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57984/0 (socket says 172.21.15.57:57984) 2023-10-30T14:32:55.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.286+0000 7f84d9815640 1 -- 172.21.15.57:0/3992472872 learned_addr learned my addr 172.21.15.57:0/3992472872 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:32:55.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.286+0000 7f84da016640 1 --2- 172.21.15.57:0/3992472872 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84d4070520 0x7f84d40709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:55.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.286+0000 7f84d9815640 1 -- 172.21.15.57:0/3992472872 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84d4070520 msgr2=0x7f84d40709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:32:55.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.286+0000 7f84d9815640 1 --2- 172.21.15.57:0/3992472872 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84d4070520 0x7f84d40709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:55.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.286+0000 7f84d9815640 1 -- 172.21.15.57:0/3992472872 >> v1:172.21.15.88:6789/0 conn(0x7f84d40cb7d0 legacy=0x7f84d40c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:55.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.286+0000 7f84d9815640 1 -- 172.21.15.57:0/3992472872 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d41569b0 con 0x7f84d4150830 2023-10-30T14:32:55.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.286+0000 7f84d9815640 1 --2- 172.21.15.57:0/3992472872 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84d4150830 0x7f84d406b500 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009870 tx=0x7f84d001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde408fab4862b97 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:55.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.286+0000 7f84d8813640 1 -- 172.21.15.57:0/3992472872 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d00a8a30 con 0x7f84d4150830 2023-10-30T14:32:55.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.286+0000 7f84d8813640 1 -- 172.21.15.57:0/3992472872 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d00a8b90 con 0x7f84d4150830 2023-10-30T14:32:55.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.286+0000 7f84d8813640 1 -- 172.21.15.57:0/3992472872 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d00b0440 con 0x7f84d4150830 2023-10-30T14:32:55.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84dba79640 1 -- 172.21.15.57:0/3992472872 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84d4150830 msgr2=0x7f84d406b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:55.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84dba79640 1 --2- 172.21.15.57:0/3992472872 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84d4150830 0x7f84d406b500 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009870 tx=0x7f84d001ed30 comp rx=0 tx=0).stop 2023-10-30T14:32:55.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84dba79640 1 -- 172.21.15.57:0/3992472872 shutdown_connections 2023-10-30T14:32:55.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84dba79640 1 --2- 172.21.15.57:0/3992472872 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84d4070520 0x7f84d40709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:55.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84dba79640 1 --2- 172.21.15.57:0/3992472872 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84d4150830 0x7f84d406b500 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:55.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84dba79640 1 -- 172.21.15.57:0/3992472872 >> 172.21.15.57:0/3992472872 conn(0x7f84d40cac70 msgr2=0x7f84d40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:32:55.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84dba79640 1 -- 172.21.15.57:0/3992472872 shutdown_connections 2023-10-30T14:32:55.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84dba79640 1 -- 172.21.15.57:0/3992472872 wait complete. 2023-10-30T14:32:55.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84dba79640 1 Processor -- start 2023-10-30T14:32:55.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84dba79640 1 -- start start 2023-10-30T14:32:55.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84dba79640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84d4070520 0x7f84d406bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:55.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84dba79640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84d40cb7d0 0x7f84d41547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:55.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84dba79640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84d4150830 0x7f84d4154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:55.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84dba79640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d4159cd0 con 0x7f84d40cb7d0 2023-10-30T14:32:55.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84dba79640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d4158ee0 con 0x7f84d4150830 2023-10-30T14:32:55.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84dba79640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d4159150 con 0x7f84d4070520 2023-10-30T14:32:55.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84d9014640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84d40cb7d0 0x7f84d41547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:55.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84d9014640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84d40cb7d0 0x7f84d41547a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57998/0 (socket says 172.21.15.57:57998) 2023-10-30T14:32:55.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84d9014640 1 -- 172.21.15.57:0/3726369003 learned_addr learned my addr 172.21.15.57:0/3726369003 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:32:55.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84d9815640 1 --2- 172.21.15.57:0/3726369003 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84d4070520 0x7f84d406bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:55.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.290+0000 7f84da016640 1 --2- 172.21.15.57:0/3726369003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84d4150830 0x7f84d4154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:55.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.294+0000 7f84d9014640 1 -- 172.21.15.57:0/3726369003 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84d4070520 msgr2=0x7f84d406bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:55.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.294+0000 7f84d9014640 1 --2- 172.21.15.57:0/3726369003 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84d4070520 0x7f84d406bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:55.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.294+0000 7f84d9014640 1 -- 172.21.15.57:0/3726369003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84d4150830 msgr2=0x7f84d4154ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:55.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.294+0000 7f84d9014640 1 --2- 172.21.15.57:0/3726369003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84d4150830 0x7f84d4154ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:55.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.294+0000 7f84d9014640 1 -- 172.21.15.57:0/3726369003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d00094d0 con 0x7f84d40cb7d0 2023-10-30T14:32:55.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.294+0000 7f84da016640 1 --2- 172.21.15.57:0/3726369003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84d4150830 0x7f84d4154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:32:55.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.294+0000 7f84d9014640 1 --2- 172.21.15.57:0/3726369003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84d40cb7d0 0x7f84d41547a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f84c400c530 tx=0x7f84c400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:55.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.294+0000 7f84ca7fc640 1 -- 172.21.15.57:0/3726369003 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c4005220 con 0x7f84d40cb7d0 2023-10-30T14:32:55.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.294+0000 7f84ca7fc640 1 -- 172.21.15.57:0/3726369003 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c4005380 con 0x7f84d40cb7d0 2023-10-30T14:32:55.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.294+0000 7f84dba79640 1 -- 172.21.15.57:0/3726369003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d4159430 con 0x7f84d40cb7d0 2023-10-30T14:32:55.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.294+0000 7f84ca7fc640 1 -- 172.21.15.57:0/3726369003 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c4007270 con 0x7f84d40cb7d0 2023-10-30T14:32:55.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.294+0000 7f84dba79640 1 -- 172.21.15.57:0/3726369003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d41599f0 con 0x7f84d40cb7d0 2023-10-30T14:32:55.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.294+0000 7f84dba79640 1 -- 172.21.15.57:0/3726369003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d406c960 con 0x7f84d40cb7d0 2023-10-30T14:32:55.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.294+0000 7f84ca7fc640 1 -- 172.21.15.57:0/3726369003 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f84c40073d0 con 0x7f84d40cb7d0 2023-10-30T14:32:55.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.294+0000 7f84ca7fc640 1 --2- 172.21.15.57:0/3726369003 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f84b80664a0 0x7f84b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:55.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.294+0000 7f84ca7fc640 1 -- 172.21.15.57:0/3726369003 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f84c4083b10 con 0x7f84d40cb7d0 2023-10-30T14:32:55.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.298+0000 7f84d9815640 1 --2- 172.21.15.57:0/3726369003 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f84b80664a0 0x7f84b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:55.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.298+0000 7f84d9815640 1 --2- 172.21.15.57:0/3726369003 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f84b80664a0 0x7f84b8068960 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f84d00b2000 tx=0x7f84d0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:55.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.298+0000 7f84ca7fc640 1 -- 172.21.15.57:0/3726369003 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f84c4050b50 con 0x7f84d40cb7d0 2023-10-30T14:32:55.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.490+0000 7f84dba79640 1 -- 172.21.15.57:0/3726369003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84d404e170 con 0x7f84d40cb7d0 2023-10-30T14:32:55.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.490+0000 7f84ca7fc640 1 -- 172.21.15.57:0/3726369003 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5740 (secure 0 0 0) 0x7f84c4054170 con 0x7f84d40cb7d0 2023-10-30T14:32:55.490 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:32:55.491 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:32:54.940721+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:32:55.491 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 111 2023-10-30T14:32:55.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.490+0000 7f84dba79640 1 -- 172.21.15.57:0/3726369003 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f84b80664a0 msgr2=0x7f84b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:55.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.490+0000 7f84dba79640 1 --2- 172.21.15.57:0/3726369003 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f84b80664a0 0x7f84b8068960 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f84d00b2000 tx=0x7f84d0009040 comp rx=0 tx=0).stop 2023-10-30T14:32:55.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.490+0000 7f84dba79640 1 -- 172.21.15.57:0/3726369003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84d40cb7d0 msgr2=0x7f84d41547a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:55.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.490+0000 7f84dba79640 1 --2- 172.21.15.57:0/3726369003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84d40cb7d0 0x7f84d41547a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f84c400c530 tx=0x7f84c400c9f0 comp rx=0 tx=0).stop 2023-10-30T14:32:55.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.490+0000 7f84dba79640 1 -- 172.21.15.57:0/3726369003 shutdown_connections 2023-10-30T14:32:55.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.490+0000 7f84dba79640 1 --2- 172.21.15.57:0/3726369003 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f84b80664a0 0x7f84b8068960 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:55.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.490+0000 7f84dba79640 1 --2- 172.21.15.57:0/3726369003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84d4150830 0x7f84d4154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:55.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.490+0000 7f84dba79640 1 --2- 172.21.15.57:0/3726369003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84d40cb7d0 0x7f84d41547a0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:55.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.490+0000 7f84dba79640 1 --2- 172.21.15.57:0/3726369003 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84d4070520 0x7f84d406bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:55.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.490+0000 7f84dba79640 1 -- 172.21.15.57:0/3726369003 >> 172.21.15.57:0/3726369003 conn(0x7f84d40cac70 msgr2=0x7f84d414e770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:32:55.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.490+0000 7f84dba79640 1 -- 172.21.15.57:0/3726369003 shutdown_connections 2023-10-30T14:32:55.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:55.490+0000 7f84dba79640 1 -- 172.21.15.57:0/3726369003 wait complete. 2023-10-30T14:32:55.514 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 111, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 111, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5891": { "addr": "172.21.15.88:6839/3604033338", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 3604033338, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 3604033338, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5891, "incarnation": 111, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:replay", "state_seq": 1 }, "gid_6204": { "addr": "172.21.15.57:6837/1843528309", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 1843528309, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 1843528309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6204, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_6414": { "addr": "172.21.15.57:6834/2321408149", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2321408149, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2321408149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6414, "incarnation": 99, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:32:54.940721+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6414, "mds_1": 5891, "mds_2": 6204 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:32:57.516 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:32:57.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.670+0000 7facee217640 1 Processor -- start 2023-10-30T14:32:57.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.670+0000 7facee217640 1 -- start start 2023-10-30T14:32:57.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.670+0000 7facee217640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7face806a4d0 0x7face80cb790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:57.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.670+0000 7facee217640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face80c1e60 con 0x7face80c0260 2023-10-30T14:32:57.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.670+0000 7facee217640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face80cb2b0 con 0x7face806a0f0 2023-10-30T14:32:57.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.670+0000 7facee217640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face815a3f0 con 0x7face806a4d0 2023-10-30T14:32:57.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.670+0000 7face7fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7face80c0260 0x7face8164320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:34782/0 (socket says 172.21.15.57:34782) 2023-10-30T14:32:57.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.670+0000 7face7fff640 1 -- 172.21.15.57:0/3010255761 learned_addr learned my addr 172.21.15.57:0/3010255761 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:32:57.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.670+0000 7face6ffd640 1 --2- 172.21.15.57:0/3010255761 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7face806a4d0 0x7face80cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:57.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.670+0000 7face67fc640 1 -- 172.21.15.57:0/3010255761 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2066899027 0 0) 0x7face80c1e60 con 0x7face80c0260 2023-10-30T14:32:57.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.670+0000 7face67fc640 1 -- 172.21.15.57:0/3010255761 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facd0003530 con 0x7face80c0260 2023-10-30T14:32:57.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face67fc640 1 -- 172.21.15.57:0/3010255761 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1091910817 0 0) 0x7facd0003530 con 0x7face80c0260 2023-10-30T14:32:57.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face67fc640 1 -- 172.21.15.57:0/3010255761 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7face806a4d0 msgr2=0x7face80cb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:32:57.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face67fc640 1 --2- 172.21.15.57:0/3010255761 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7face806a4d0 0x7face80cb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:57.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face67fc640 1 -- 172.21.15.57:0/3010255761 >> v1:172.21.15.88:6789/0 conn(0x7face806a0f0 legacy=0x7face8153830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:57.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face67fc640 1 -- 172.21.15.57:0/3010255761 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face815a8c0 con 0x7face80c0260 2023-10-30T14:32:57.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face67fc640 1 -- 172.21.15.57:0/3010255761 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7face0001a00 con 0x7face80c0260 2023-10-30T14:32:57.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face67fc640 1 -- 172.21.15.57:0/3010255761 >> v1:172.21.15.57:6789/0 conn(0x7face80c0260 legacy=0x7face8164320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:57.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face67fc640 1 --2- 172.21.15.57:0/3010255761 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facd00039b0 0x7facd0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:57.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face67fc640 1 --2- 172.21.15.57:0/3010255761 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd00047d0 0x7facd0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:57.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face67fc640 1 --2- 172.21.15.57:0/3010255761 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7face806a4d0 0x7facd0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:57.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face67fc640 1 -- 172.21.15.57:0/3010255761 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd000ba40 con 0x7facd00047d0 2023-10-30T14:32:57.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face67fc640 1 -- 172.21.15.57:0/3010255761 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd000bb80 con 0x7facd00039b0 2023-10-30T14:32:57.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face67fc640 1 -- 172.21.15.57:0/3010255761 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd000bd40 con 0x7face806a4d0 2023-10-30T14:32:57.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face6ffd640 1 --2- 172.21.15.57:0/3010255761 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd00047d0 0x7facd0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:57.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face77fe640 1 --2- 172.21.15.57:0/3010255761 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facd00039b0 0x7facd0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:57.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face6ffd640 1 -- 172.21.15.57:0/3010255761 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7face806a4d0 msgr2=0x7facd0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:32:57.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face6ffd640 1 --2- 172.21.15.57:0/3010255761 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7face806a4d0 0x7facd0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:57.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face6ffd640 1 -- 172.21.15.57:0/3010255761 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facd00039b0 msgr2=0x7facd0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:32:57.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face6ffd640 1 --2- 172.21.15.57:0/3010255761 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facd00039b0 0x7facd0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:57.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face6ffd640 1 -- 172.21.15.57:0/3010255761 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd0003530 con 0x7facd00047d0 2023-10-30T14:32:57.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face6ffd640 1 --2- 172.21.15.57:0/3010255761 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd00047d0 0x7facd0004c50 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7face806a8f0 tx=0x7facdc01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef329c460a9cc5ba server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:57.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face67fc640 1 -- 172.21.15.57:0/3010255761 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc0a5600 con 0x7facd00047d0 2023-10-30T14:32:57.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face67fc640 1 -- 172.21.15.57:0/3010255761 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc0a5760 con 0x7facd00047d0 2023-10-30T14:32:57.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.674+0000 7face67fc640 1 -- 172.21.15.57:0/3010255761 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc0a5b10 con 0x7facd00047d0 2023-10-30T14:32:57.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7facee217640 1 -- 172.21.15.57:0/3010255761 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd00047d0 msgr2=0x7facd0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:57.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7facee217640 1 --2- 172.21.15.57:0/3010255761 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd00047d0 0x7facd0004c50 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7face806a8f0 tx=0x7facdc01ca10 comp rx=0 tx=0).stop 2023-10-30T14:32:57.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7face7fff640 1 -- 172.21.15.57:0/3010255761 reap_dead start 2023-10-30T14:32:57.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7facee217640 1 -- 172.21.15.57:0/3010255761 shutdown_connections 2023-10-30T14:32:57.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7facee217640 1 -- 172.21.15.57:0/3010255761 >> 172.21.15.57:0/3010255761 conn(0x7face806de90 msgr2=0x7face806e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:32:57.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7facee217640 1 -- 172.21.15.57:0/3010255761 shutdown_connections 2023-10-30T14:32:57.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7facee217640 1 -- 172.21.15.57:0/3010255761 wait complete. 2023-10-30T14:32:57.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7facee217640 1 Processor -- start 2023-10-30T14:32:57.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7facee217640 1 -- start start 2023-10-30T14:32:57.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7facee217640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd00047d0 0x7face80afe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:57.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7facee217640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face80c7d50 0x7face814bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:57.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7facee217640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7face814c3e0 0x7face8158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:57.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7facee217640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face80c79e0 con 0x7facd00047d0 2023-10-30T14:32:57.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7facee217640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face81f6950 con 0x7face80c7d50 2023-10-30T14:32:57.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7facee217640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face8070680 con 0x7face814c3e0 2023-10-30T14:32:57.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7face77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd00047d0 0x7face80afe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:57.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7face77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd00047d0 0x7face80afe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58016/0 (socket says 172.21.15.57:58016) 2023-10-30T14:32:57.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7face77fe640 1 -- 172.21.15.57:0/201295274 learned_addr learned my addr 172.21.15.57:0/201295274 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:32:57.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7face7fff640 1 --2- 172.21.15.57:0/201295274 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7face814c3e0 0x7face8158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:57.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.678+0000 7face6ffd640 1 --2- 172.21.15.57:0/201295274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face80c7d50 0x7face814bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:57.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.682+0000 7face77fe640 1 -- 172.21.15.57:0/201295274 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7face814c3e0 msgr2=0x7face8158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:57.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.682+0000 7face77fe640 1 --2- 172.21.15.57:0/201295274 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7face814c3e0 0x7face8158990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:57.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.682+0000 7face77fe640 1 -- 172.21.15.57:0/201295274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face80c7d50 msgr2=0x7face814bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:57.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.682+0000 7face77fe640 1 --2- 172.21.15.57:0/201295274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face80c7d50 0x7face814bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:57.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.682+0000 7face77fe640 1 -- 172.21.15.57:0/201295274 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face8070920 con 0x7facd00047d0 2023-10-30T14:32:57.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.682+0000 7face77fe640 1 --2- 172.21.15.57:0/201295274 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd00047d0 0x7face80afe20 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7facd800e580 tx=0x7facd800ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:57.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.682+0000 7faccffff640 1 -- 172.21.15.57:0/201295274 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd80052e0 con 0x7facd00047d0 2023-10-30T14:32:57.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.682+0000 7faccffff640 1 -- 172.21.15.57:0/201295274 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd8005440 con 0x7facd00047d0 2023-10-30T14:32:57.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.682+0000 7faccffff640 1 -- 172.21.15.57:0/201295274 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd8003820 con 0x7facd00047d0 2023-10-30T14:32:57.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.682+0000 7facee217640 1 -- 172.21.15.57:0/201295274 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face81f6ba0 con 0x7facd00047d0 2023-10-30T14:32:57.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.682+0000 7facee217640 1 -- 172.21.15.57:0/201295274 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face81f7160 con 0x7facd00047d0 2023-10-30T14:32:57.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.682+0000 7facee217640 1 -- 172.21.15.57:0/201295274 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face804e170 con 0x7facd00047d0 2023-10-30T14:32:57.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.682+0000 7faccffff640 1 -- 172.21.15.57:0/201295274 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7facd8003980 con 0x7facd00047d0 2023-10-30T14:32:57.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.682+0000 7faccffff640 1 --2- 172.21.15.57:0/201295274 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7facd40664a0 0x7facd4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:32:57.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.682+0000 7faccffff640 1 -- 172.21.15.57:0/201295274 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7facd8083f60 con 0x7facd00047d0 2023-10-30T14:32:57.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.686+0000 7face6ffd640 1 --2- 172.21.15.57:0/201295274 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7facd40664a0 0x7facd4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:32:57.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.686+0000 7face6ffd640 1 --2- 172.21.15.57:0/201295274 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7facd40664a0 0x7facd4068960 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7facdc0af000 tx=0x7facdc0046d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:32:57.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.686+0000 7faccffff640 1 -- 172.21.15.57:0/201295274 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7facd8050fd0 con 0x7facd00047d0 2023-10-30T14:32:57.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.878+0000 7facee217640 1 -- 172.21.15.57:0/201295274 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7face80c5670 con 0x7facd00047d0 2023-10-30T14:32:57.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.882+0000 7faccffff640 1 -- 172.21.15.57:0/201295274 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5740 (secure 0 0 0) 0x7facd80545f0 con 0x7facd00047d0 2023-10-30T14:32:57.882 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:32:57.882 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:32:54.940721+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:32:57.883 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 111 2023-10-30T14:32:57.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.882+0000 7facee217640 1 -- 172.21.15.57:0/201295274 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7facd40664a0 msgr2=0x7facd4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:57.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.882+0000 7facee217640 1 --2- 172.21.15.57:0/201295274 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7facd40664a0 0x7facd4068960 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7facdc0af000 tx=0x7facdc0046d0 comp rx=0 tx=0).stop 2023-10-30T14:32:57.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.882+0000 7facee217640 1 -- 172.21.15.57:0/201295274 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd00047d0 msgr2=0x7face80afe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:32:57.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.882+0000 7facee217640 1 --2- 172.21.15.57:0/201295274 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd00047d0 0x7face80afe20 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7facd800e580 tx=0x7facd800ea40 comp rx=0 tx=0).stop 2023-10-30T14:32:57.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.882+0000 7facee217640 1 -- 172.21.15.57:0/201295274 shutdown_connections 2023-10-30T14:32:57.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.882+0000 7facee217640 1 --2- 172.21.15.57:0/201295274 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7facd40664a0 0x7facd4068960 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:57.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.882+0000 7facee217640 1 --2- 172.21.15.57:0/201295274 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7face814c3e0 0x7face8158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:57.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.882+0000 7facee217640 1 --2- 172.21.15.57:0/201295274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face80c7d50 0x7face814bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:57.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.882+0000 7facee217640 1 --2- 172.21.15.57:0/201295274 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facd00047d0 0x7face80afe20 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:32:57.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.882+0000 7facee217640 1 -- 172.21.15.57:0/201295274 >> 172.21.15.57:0/201295274 conn(0x7face806de90 msgr2=0x7face814e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:32:57.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.882+0000 7facee217640 1 -- 172.21.15.57:0/201295274 shutdown_connections 2023-10-30T14:32:57.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:32:57.882+0000 7facee217640 1 -- 172.21.15.57:0/201295274 wait complete. 2023-10-30T14:32:57.902 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:32:59.903 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:33:00.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.054+0000 7fd0d4bee640 1 Processor -- start 2023-10-30T14:33:00.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.054+0000 7fd0d4bee640 1 -- start start 2023-10-30T14:33:00.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.054+0000 7fd0d4bee640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd0d00701a0 0x7fd0d015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:00.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.054+0000 7fd0d4bee640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d00c2aa0 con 0x7fd0d01537f0 2023-10-30T14:33:00.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.054+0000 7fd0d4bee640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d00708e0 con 0x7fd0d006b2f0 2023-10-30T14:33:00.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.054+0000 7fd0d4bee640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d00c3ab0 con 0x7fd0d00701a0 2023-10-30T14:33:00.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.054+0000 7fd0ce59c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fd0d01537f0 0x7fd0d00ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:34790/0 (socket says 172.21.15.57:34790) 2023-10-30T14:33:00.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.054+0000 7fd0ce59c640 1 -- 172.21.15.57:0/2531113744 learned_addr learned my addr 172.21.15.57:0/2531113744 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:33:00.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.054+0000 7fd0c7fff640 1 --2- 172.21.15.57:0/2531113744 >> v2:172.21.15.88:3300/0 conn(0x7fd0d00701a0 0x7fd0d015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:00.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.054+0000 7fd0cdd9b640 1 -- 172.21.15.57:0/2531113744 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727055121 0 0) 0x7fd0d00c2aa0 con 0x7fd0d01537f0 2023-10-30T14:33:00.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.058+0000 7fd0cdd9b640 1 -- 172.21.15.57:0/2531113744 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0bc003530 con 0x7fd0d01537f0 2023-10-30T14:33:00.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.058+0000 7fd0c7fff640 1 -- 172.21.15.57:0/2531113744 >> v1:172.21.15.88:6789/0 conn(0x7fd0d006b2f0 legacy=0x7fd0d006b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:00.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.058+0000 7fd0c7fff640 1 -- 172.21.15.57:0/2531113744 >> v1:172.21.15.57:6789/0 conn(0x7fd0d01537f0 legacy=0x7fd0d00ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:00.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.058+0000 7fd0cdd9b640 1 -- 172.21.15.57:0/2531113744 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2255196643 0 0) 0x7fd0bc003530 con 0x7fd0d01537f0 2023-10-30T14:33:00.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.058+0000 7fd0c7fff640 1 -- 172.21.15.57:0/2531113744 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d00c1eb0 con 0x7fd0d00701a0 2023-10-30T14:33:00.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.058+0000 7fd0c7fff640 1 --2- 172.21.15.57:0/2531113744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d00701a0 0x7fd0d015a2e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd0b80097c0 tx=0x7fd0b801eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6acd7288014a03 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:00.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.058+0000 7fd0cdd9b640 1 -- 172.21.15.57:0/2531113744 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0001680 con 0x7fd0d00701a0 2023-10-30T14:33:00.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.058+0000 7fd0cdd9b640 1 -- 172.21.15.57:0/2531113744 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b80a8610 con 0x7fd0d00701a0 2023-10-30T14:33:00.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.058+0000 7fd0cdd9b640 1 -- 172.21.15.57:0/2531113744 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b80a89a0 con 0x7fd0d00701a0 2023-10-30T14:33:00.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0d4bee640 1 -- 172.21.15.57:0/2531113744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d00701a0 msgr2=0x7fd0d015a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:00.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0d4bee640 1 --2- 172.21.15.57:0/2531113744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d00701a0 0x7fd0d015a2e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd0b80097c0 tx=0x7fd0b801eb90 comp rx=0 tx=0).stop 2023-10-30T14:33:00.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0d4bee640 1 -- 172.21.15.57:0/2531113744 shutdown_connections 2023-10-30T14:33:00.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0d4bee640 1 --2- 172.21.15.57:0/2531113744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d00701a0 0x7fd0d015a2e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:00.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0d4bee640 1 -- 172.21.15.57:0/2531113744 >> 172.21.15.57:0/2531113744 conn(0x7fd0d0071fb0 msgr2=0x7fd0d00723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:00.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0d4bee640 1 -- 172.21.15.57:0/2531113744 shutdown_connections 2023-10-30T14:33:00.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0d4bee640 1 -- 172.21.15.57:0/2531113744 wait complete. 2023-10-30T14:33:00.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0d4bee640 1 Processor -- start 2023-10-30T14:33:00.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0d4bee640 1 -- start start 2023-10-30T14:33:00.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0d4bee640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0d006b2f0 0x7fd0d00cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:00.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0d4bee640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d00701a0 0x7fd0d0155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:00.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0d4bee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0d01537f0 0x7fd0d01555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:00.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0d4bee640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d01598a0 con 0x7fd0d006b2f0 2023-10-30T14:33:00.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0d4bee640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d01522a0 con 0x7fd0d00701a0 2023-10-30T14:33:00.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0d4bee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d0158e50 con 0x7fd0d01537f0 2023-10-30T14:33:00.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0ce59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0d006b2f0 0x7fd0d00cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:00.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0ce59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0d006b2f0 0x7fd0d00cbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58018/0 (socket says 172.21.15.57:58018) 2023-10-30T14:33:00.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0ce59c640 1 -- 172.21.15.57:0/1151451119 learned_addr learned my addr 172.21.15.57:0/1151451119 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:33:00.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0c7fff640 1 --2- 172.21.15.57:0/1151451119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d00701a0 0x7fd0d0155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:00.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0ced9d640 1 --2- 172.21.15.57:0/1151451119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0d01537f0 0x7fd0d01555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:00.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0ce59c640 1 -- 172.21.15.57:0/1151451119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0d01537f0 msgr2=0x7fd0d01555d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:00.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0ce59c640 1 --2- 172.21.15.57:0/1151451119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0d01537f0 0x7fd0d01555d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:00.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0ce59c640 1 -- 172.21.15.57:0/1151451119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d00701a0 msgr2=0x7fd0d0155090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:00.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0ce59c640 1 --2- 172.21.15.57:0/1151451119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d00701a0 0x7fd0d0155090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:00.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0ce59c640 1 -- 172.21.15.57:0/1151451119 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b8009420 con 0x7fd0d006b2f0 2023-10-30T14:33:00.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.062+0000 7fd0ce59c640 1 --2- 172.21.15.57:0/1151451119 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0d006b2f0 0x7fd0d00cbdc0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000c780 tx=0x7fd0c000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:00.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.066+0000 7fd0c6ffd640 1 -- 172.21.15.57:0/1151451119 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0005ae0 con 0x7fd0d006b2f0 2023-10-30T14:33:00.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.066+0000 7fd0c6ffd640 1 -- 172.21.15.57:0/1151451119 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c0005c60 con 0x7fd0d006b2f0 2023-10-30T14:33:00.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.066+0000 7fd0d4bee640 1 -- 172.21.15.57:0/1151451119 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d00c9ac0 con 0x7fd0d006b2f0 2023-10-30T14:33:00.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.066+0000 7fd0c6ffd640 1 -- 172.21.15.57:0/1151451119 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0007070 con 0x7fd0d006b2f0 2023-10-30T14:33:00.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.066+0000 7fd0d4bee640 1 -- 172.21.15.57:0/1151451119 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d00ca010 con 0x7fd0d006b2f0 2023-10-30T14:33:00.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.066+0000 7fd0d4bee640 1 -- 172.21.15.57:0/1151451119 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d004e170 con 0x7fd0d006b2f0 2023-10-30T14:33:00.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.066+0000 7fd0c6ffd640 1 -- 172.21.15.57:0/1151451119 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd0c00071d0 con 0x7fd0d006b2f0 2023-10-30T14:33:00.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.066+0000 7fd0c6ffd640 1 --2- 172.21.15.57:0/1151451119 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0b00664a0 0x7fd0b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:00.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.066+0000 7fd0c6ffd640 1 -- 172.21.15.57:0/1151451119 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7fd0c0083aa0 con 0x7fd0d006b2f0 2023-10-30T14:33:00.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.066+0000 7fd0c7fff640 1 --2- 172.21.15.57:0/1151451119 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0b00664a0 0x7fd0b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:00.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.070+0000 7fd0c7fff640 1 --2- 172.21.15.57:0/1151451119 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0b00664a0 0x7fd0b0068960 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd0b80b2000 tx=0x7fd0b8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:00.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.070+0000 7fd0c6ffd640 1 -- 172.21.15.57:0/1151451119 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd0c0050b30 con 0x7fd0d006b2f0 2023-10-30T14:33:00.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.262+0000 7fd0d4bee640 1 -- 172.21.15.57:0/1151451119 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0d00cc300 con 0x7fd0d006b2f0 2023-10-30T14:33:00.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.266+0000 7fd0c6ffd640 1 -- 172.21.15.57:0/1151451119 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5740 (secure 0 0 0) 0x7fd0c0017020 con 0x7fd0d006b2f0 2023-10-30T14:33:00.266 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:33:00.266 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:32:54.940721+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:33:00.266 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 111 2023-10-30T14:33:00.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.266+0000 7fd0d4bee640 1 -- 172.21.15.57:0/1151451119 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0b00664a0 msgr2=0x7fd0b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:00.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.266+0000 7fd0d4bee640 1 --2- 172.21.15.57:0/1151451119 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0b00664a0 0x7fd0b0068960 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd0b80b2000 tx=0x7fd0b8008040 comp rx=0 tx=0).stop 2023-10-30T14:33:00.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.266+0000 7fd0d4bee640 1 -- 172.21.15.57:0/1151451119 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0d006b2f0 msgr2=0x7fd0d00cbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:00.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.266+0000 7fd0d4bee640 1 --2- 172.21.15.57:0/1151451119 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0d006b2f0 0x7fd0d00cbdc0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000c780 tx=0x7fd0c000cc40 comp rx=0 tx=0).stop 2023-10-30T14:33:00.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.266+0000 7fd0d4bee640 1 -- 172.21.15.57:0/1151451119 shutdown_connections 2023-10-30T14:33:00.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.266+0000 7fd0d4bee640 1 --2- 172.21.15.57:0/1151451119 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0b00664a0 0x7fd0b0068960 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:00.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.266+0000 7fd0d4bee640 1 --2- 172.21.15.57:0/1151451119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0d01537f0 0x7fd0d01555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:00.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.266+0000 7fd0d4bee640 1 --2- 172.21.15.57:0/1151451119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d00701a0 0x7fd0d0155090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:00.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.266+0000 7fd0d4bee640 1 --2- 172.21.15.57:0/1151451119 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0d006b2f0 0x7fd0d00cbdc0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:00.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.266+0000 7fd0d4bee640 1 -- 172.21.15.57:0/1151451119 >> 172.21.15.57:0/1151451119 conn(0x7fd0d0071fb0 msgr2=0x7fd0d006b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:00.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.266+0000 7fd0d4bee640 1 -- 172.21.15.57:0/1151451119 shutdown_connections 2023-10-30T14:33:00.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:00.266+0000 7fd0d4bee640 1 -- 172.21.15.57:0/1151451119 wait complete. 2023-10-30T14:33:00.286 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:33:02.287 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:33:02.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.430+0000 7f06c589c640 1 Processor -- start 2023-10-30T14:33:02.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.430+0000 7f06c589c640 1 -- start start 2023-10-30T14:33:02.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.434+0000 7f06c589c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f06c0153bb0 0x7f06c00c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:02.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.434+0000 7f06c589c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f06c006c420 0x7f06c00c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:02.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.434+0000 7f06c589c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c00c5f80 con 0x7f06c00c7170 2023-10-30T14:33:02.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.434+0000 7f06c589c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c00c7580 con 0x7f06c006c420 2023-10-30T14:33:02.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.434+0000 7f06c589c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c00c5690 con 0x7f06c0153bb0 2023-10-30T14:33:02.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.434+0000 7f06b7fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f06c006c420 0x7f06c00c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:02.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.434+0000 7f06bf7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f06c00c7170 0x7f06c0159980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:34792/0 (socket says 172.21.15.57:34792) 2023-10-30T14:33:02.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.434+0000 7f06bf7fe640 1 -- 172.21.15.57:0/1478129441 learned_addr learned my addr 172.21.15.57:0/1478129441 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:33:02.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.434+0000 7f06beffd640 1 --2- 172.21.15.57:0/1478129441 >> v2:172.21.15.88:3300/0 conn(0x7f06c0153bb0 0x7f06c00c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:02.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.434+0000 7f06b7fff640 1 -- 172.21.15.57:0/1478129441 >> v2:172.21.15.88:3300/0 conn(0x7f06c0153bb0 msgr2=0x7f06c00c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:02.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.434+0000 7f06b7fff640 1 --2- 172.21.15.57:0/1478129441 >> v2:172.21.15.88:3300/0 conn(0x7f06c0153bb0 0x7f06c00c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:02.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.434+0000 7f06b7fff640 1 -- 172.21.15.57:0/1478129441 >> v1:172.21.15.57:6789/0 conn(0x7f06c00c7170 legacy=0x7f06c0159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:02.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.434+0000 7f06b7fff640 1 -- 172.21.15.57:0/1478129441 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c015a580 con 0x7f06c006c420 2023-10-30T14:33:02.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.434+0000 7f06be7fc640 1 -- 172.21.15.57:0/1478129441 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369074138 0 0) 0x7f06c00c5f80 con 0x7f06c00c7170 2023-10-30T14:33:02.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06b7fff640 1 --2- 172.21.15.57:0/1478129441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c006c420 0x7f06c00c6c30 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f06c00cb840 tx=0x7f06a801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbc2ca0b73b5179c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:02.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06be7fc640 1 -- 172.21.15.57:0/1478129441 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a80a8950 con 0x7f06c006c420 2023-10-30T14:33:02.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06be7fc640 1 -- 172.21.15.57:0/1478129441 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a80a8ab0 con 0x7f06c006c420 2023-10-30T14:33:02.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06be7fc640 1 -- 172.21.15.57:0/1478129441 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a80a8e40 con 0x7f06c006c420 2023-10-30T14:33:02.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06c589c640 1 -- 172.21.15.57:0/1478129441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c006c420 msgr2=0x7f06c00c6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:02.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06c589c640 1 --2- 172.21.15.57:0/1478129441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c006c420 0x7f06c00c6c30 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f06c00cb840 tx=0x7f06a801ec50 comp rx=0 tx=0).stop 2023-10-30T14:33:02.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06c589c640 1 -- 172.21.15.57:0/1478129441 shutdown_connections 2023-10-30T14:33:02.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06c589c640 1 --2- 172.21.15.57:0/1478129441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c006c420 0x7f06c00c6c30 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:02.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06c589c640 1 --2- 172.21.15.57:0/1478129441 >> v2:172.21.15.88:3300/0 conn(0x7f06c0153bb0 0x7f06c00c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:02.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06c589c640 1 -- 172.21.15.57:0/1478129441 >> 172.21.15.57:0/1478129441 conn(0x7f06c00cac70 msgr2=0x7f06c00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:02.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06c589c640 1 -- 172.21.15.57:0/1478129441 shutdown_connections 2023-10-30T14:33:02.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06c589c640 1 -- 172.21.15.57:0/1478129441 wait complete. 2023-10-30T14:33:02.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06c589c640 1 Processor -- start 2023-10-30T14:33:02.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06c589c640 1 -- start start 2023-10-30T14:33:02.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06c589c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c006c420 0x7f06c01ef440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:02.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06c589c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c00c7170 0x7f06c00c9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:02.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06c589c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c0153bb0 0x7f06c00c29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:02.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06c589c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c00c40a0 con 0x7f06c00c7170 2023-10-30T14:33:02.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06c589c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c00c42f0 con 0x7f06c006c420 2023-10-30T14:33:02.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.438+0000 7f06c589c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c00c45d0 con 0x7f06c0153bb0 2023-10-30T14:33:02.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06b7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c00c7170 0x7f06c00c9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:02.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06b7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c00c7170 0x7f06c00c9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58036/0 (socket says 172.21.15.57:58036) 2023-10-30T14:33:02.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06b7fff640 1 -- 172.21.15.57:0/2579365832 learned_addr learned my addr 172.21.15.57:0/2579365832 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:33:02.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06beffd640 1 --2- 172.21.15.57:0/2579365832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c006c420 0x7f06c01ef440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:02.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06bf7fe640 1 --2- 172.21.15.57:0/2579365832 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c0153bb0 0x7f06c00c29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:02.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06b7fff640 1 -- 172.21.15.57:0/2579365832 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c0153bb0 msgr2=0x7f06c00c29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:02.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06b7fff640 1 --2- 172.21.15.57:0/2579365832 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c0153bb0 0x7f06c00c29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:02.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06b7fff640 1 -- 172.21.15.57:0/2579365832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c006c420 msgr2=0x7f06c01ef440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:02.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06b7fff640 1 --2- 172.21.15.57:0/2579365832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c006c420 0x7f06c01ef440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:02.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06b7fff640 1 -- 172.21.15.57:0/2579365832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a8009420 con 0x7f06c00c7170 2023-10-30T14:33:02.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06beffd640 1 --2- 172.21.15.57:0/2579365832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c006c420 0x7f06c01ef440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:33:02.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06b7fff640 1 --2- 172.21.15.57:0/2579365832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c00c7170 0x7f06c00c9800 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f06a80b3040 tx=0x7f06a801b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:02.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06bcff9640 1 -- 172.21.15.57:0/2579365832 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a80b0dd0 con 0x7f06c00c7170 2023-10-30T14:33:02.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06bcff9640 1 -- 172.21.15.57:0/2579365832 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a80b9430 con 0x7f06c00c7170 2023-10-30T14:33:02.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06bcff9640 1 -- 172.21.15.57:0/2579365832 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a80b97e0 con 0x7f06c00c7170 2023-10-30T14:33:02.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06c589c640 1 -- 172.21.15.57:0/2579365832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c00c48b0 con 0x7f06c00c7170 2023-10-30T14:33:02.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06c589c640 1 -- 172.21.15.57:0/2579365832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c00c4e70 con 0x7f06c00c7170 2023-10-30T14:33:02.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06c589c640 1 -- 172.21.15.57:0/2579365832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c0066510 con 0x7f06c00c7170 2023-10-30T14:33:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06bcff9640 1 -- 172.21.15.57:0/2579365832 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f06a80b7040 con 0x7f06c00c7170 2023-10-30T14:33:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.442+0000 7f06bcff9640 1 --2- 172.21.15.57:0/2579365832 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06a00664a0 0x7f06a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.446+0000 7f06bcff9640 1 -- 172.21.15.57:0/2579365832 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f06a8121da0 con 0x7f06c00c7170 2023-10-30T14:33:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.446+0000 7f06beffd640 1 --2- 172.21.15.57:0/2579365832 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06a00664a0 0x7f06a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.446+0000 7f06beffd640 1 --2- 172.21.15.57:0/2579365832 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06a00664a0 0x7f06a0068960 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f06b0009bb0 tx=0x7f06b0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:02.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.446+0000 7f06bcff9640 1 -- 172.21.15.57:0/2579365832 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f06a80eee90 con 0x7f06c00c7170 2023-10-30T14:33:02.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.642+0000 7f06c589c640 1 -- 172.21.15.57:0/2579365832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06c0066510 con 0x7f06c00c7170 2023-10-30T14:33:02.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.642+0000 7f06bcff9640 1 -- 172.21.15.57:0/2579365832 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5742 (secure 0 0 0) 0x7f06a80b5020 con 0x7f06c00c7170 2023-10-30T14:33:02.643 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:33:02.643 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:33:01.327539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:resolve","state_seq":34,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:33:02.644 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 112 2023-10-30T14:33:02.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.642+0000 7f06c589c640 1 -- 172.21.15.57:0/2579365832 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06a00664a0 msgr2=0x7f06a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:02.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.642+0000 7f06c589c640 1 --2- 172.21.15.57:0/2579365832 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06a00664a0 0x7f06a0068960 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f06b0009bb0 tx=0x7f06b0008040 comp rx=0 tx=0).stop 2023-10-30T14:33:02.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.642+0000 7f06c589c640 1 -- 172.21.15.57:0/2579365832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c00c7170 msgr2=0x7f06c00c9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:02.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.642+0000 7f06c589c640 1 --2- 172.21.15.57:0/2579365832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c00c7170 0x7f06c00c9800 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f06a80b3040 tx=0x7f06a801b9f0 comp rx=0 tx=0).stop 2023-10-30T14:33:02.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.642+0000 7f06c589c640 1 -- 172.21.15.57:0/2579365832 shutdown_connections 2023-10-30T14:33:02.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.642+0000 7f06c589c640 1 --2- 172.21.15.57:0/2579365832 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06a00664a0 0x7f06a0068960 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:02.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.642+0000 7f06c589c640 1 --2- 172.21.15.57:0/2579365832 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c0153bb0 0x7f06c00c29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:02.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.642+0000 7f06c589c640 1 --2- 172.21.15.57:0/2579365832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c00c7170 0x7f06c00c9800 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:02.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.642+0000 7f06c589c640 1 --2- 172.21.15.57:0/2579365832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c006c420 0x7f06c01ef440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:02.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.642+0000 7f06c589c640 1 -- 172.21.15.57:0/2579365832 >> 172.21.15.57:0/2579365832 conn(0x7f06c00cac70 msgr2=0x7f06c01524f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:02.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.642+0000 7f06c589c640 1 -- 172.21.15.57:0/2579365832 shutdown_connections 2023-10-30T14:33:02.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:02.642+0000 7f06c589c640 1 -- 172.21.15.57:0/2579365832 wait complete. 2023-10-30T14:33:02.664 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:33:04.665 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:33:04.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.810+0000 7fde86f72640 1 Processor -- start 2023-10-30T14:33:04.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.810+0000 7fde86f72640 1 -- start start 2023-10-30T14:33:04.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde86f72640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fde800ccf60 0x7fde800cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:04.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde86f72640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde800c2500 0x7fde800c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:04.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde86f72640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde80151aa0 con 0x7fde801516c0 2023-10-30T14:33:04.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde86f72640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde800c74a0 con 0x7fde800ccf60 2023-10-30T14:33:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde86f72640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde800c2e70 con 0x7fde800c2500 2023-10-30T14:33:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde84d0e640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fde801516c0 0x7fde800c7d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:32796/0 (socket says 172.21.15.57:32796) 2023-10-30T14:33:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde84d0e640 1 -- 172.21.15.57:0/503272882 learned_addr learned my addr 172.21.15.57:0/503272882 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:33:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7ffff640 1 --2- 172.21.15.57:0/503272882 >> v2:172.21.15.88:3300/0 conn(0x7fde800ccf60 0x7fde800cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde8550f640 1 --2- 172.21.15.57:0/503272882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde800c2500 0x7fde800c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:04.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7f7fe640 1 -- 172.21.15.57:0/503272882 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129449092 0 0) 0x7fde80151aa0 con 0x7fde801516c0 2023-10-30T14:33:04.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7f7fe640 1 -- 172.21.15.57:0/503272882 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde70003530 con 0x7fde801516c0 2023-10-30T14:33:04.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7f7fe640 1 -- 172.21.15.57:0/503272882 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2101222695 0 0) 0x7fde70003530 con 0x7fde801516c0 2023-10-30T14:33:04.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7f7fe640 1 -- 172.21.15.57:0/503272882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde800c2500 msgr2=0x7fde800c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:04.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7f7fe640 1 --2- 172.21.15.57:0/503272882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde800c2500 0x7fde800c2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7f7fe640 1 -- 172.21.15.57:0/503272882 >> v2:172.21.15.88:3300/0 conn(0x7fde800ccf60 msgr2=0x7fde800cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7f7fe640 1 --2- 172.21.15.57:0/503272882 >> v2:172.21.15.88:3300/0 conn(0x7fde800ccf60 0x7fde800cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7f7fe640 1 -- 172.21.15.57:0/503272882 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde80158af0 con 0x7fde801516c0 2023-10-30T14:33:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7f7fe640 1 -- 172.21.15.57:0/503272882 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fde74001680 con 0x7fde801516c0 2023-10-30T14:33:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7f7fe640 1 -- 172.21.15.57:0/503272882 >> v1:172.21.15.57:6789/0 conn(0x7fde801516c0 legacy=0x7fde800c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:04.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7f7fe640 1 --2- 172.21.15.57:0/503272882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde800c2500 0x7fde70003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:04.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7f7fe640 1 --2- 172.21.15.57:0/503272882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde70003f00 0x7fde700047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:04.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7f7fe640 1 --2- 172.21.15.57:0/503272882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde70004d10 0x7fde70005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:04.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7f7fe640 1 -- 172.21.15.57:0/503272882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde800c74a0 con 0x7fde70003f00 2023-10-30T14:33:04.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7f7fe640 1 -- 172.21.15.57:0/503272882 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde800c2e70 con 0x7fde70004d10 2023-10-30T14:33:04.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7f7fe640 1 -- 172.21.15.57:0/503272882 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7000b990 con 0x7fde800c2500 2023-10-30T14:33:04.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7ffff640 1 --2- 172.21.15.57:0/503272882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde70003f00 0x7fde700047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:04.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde84d0e640 1 --2- 172.21.15.57:0/503272882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde800c2500 0x7fde70003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:04.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde8550f640 1 --2- 172.21.15.57:0/503272882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde70004d10 0x7fde70005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:04.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7ffff640 1 -- 172.21.15.57:0/503272882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde800c2500 msgr2=0x7fde70003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:04.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7ffff640 1 --2- 172.21.15.57:0/503272882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde800c2500 0x7fde70003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:04.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7ffff640 1 -- 172.21.15.57:0/503272882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde70004d10 msgr2=0x7fde70005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:04.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7ffff640 1 --2- 172.21.15.57:0/503272882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde70004d10 0x7fde70005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:04.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde7ffff640 1 -- 172.21.15.57:0/503272882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde70003530 con 0x7fde70003f00 2023-10-30T14:33:04.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.814+0000 7fde8550f640 1 --2- 172.21.15.57:0/503272882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde70004d10 0x7fde70005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:33:04.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.818+0000 7fde7ffff640 1 --2- 172.21.15.57:0/503272882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde70003f00 0x7fde700047d0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009ae0 tx=0x7fde6c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f79f07fbf66d0a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:04.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.818+0000 7fde7f7fe640 1 -- 172.21.15.57:0/503272882 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c0a55a0 con 0x7fde70003f00 2023-10-30T14:33:04.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.818+0000 7fde7f7fe640 1 -- 172.21.15.57:0/503272882 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6c0a5700 con 0x7fde70003f00 2023-10-30T14:33:04.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.818+0000 7fde7f7fe640 1 -- 172.21.15.57:0/503272882 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c0a5af0 con 0x7fde70003f00 2023-10-30T14:33:04.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.818+0000 7fde86f72640 1 -- 172.21.15.57:0/503272882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde70003f00 msgr2=0x7fde700047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:04.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.818+0000 7fde86f72640 1 --2- 172.21.15.57:0/503272882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde70003f00 0x7fde700047d0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009ae0 tx=0x7fde6c01ca30 comp rx=0 tx=0).stop 2023-10-30T14:33:04.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.818+0000 7fde8550f640 1 -- 172.21.15.57:0/503272882 reap_dead start 2023-10-30T14:33:04.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.818+0000 7fde86f72640 1 -- 172.21.15.57:0/503272882 shutdown_connections 2023-10-30T14:33:04.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.818+0000 7fde86f72640 1 -- 172.21.15.57:0/503272882 >> 172.21.15.57:0/503272882 conn(0x7fde80071fb0 msgr2=0x7fde800723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:04.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.818+0000 7fde86f72640 1 -- 172.21.15.57:0/503272882 shutdown_connections 2023-10-30T14:33:04.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.818+0000 7fde86f72640 1 -- 172.21.15.57:0/503272882 wait complete. 2023-10-30T14:33:04.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.818+0000 7fde86f72640 1 Processor -- start 2023-10-30T14:33:04.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde86f72640 1 -- start start 2023-10-30T14:33:04.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde86f72640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde70003f00 0x7fde8014baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:04.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde86f72640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde80159f90 0x7fde8006bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:04.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde86f72640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde8006c0f0 0x7fde80149930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:04.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde86f72640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde800c6270 con 0x7fde80159f90 2023-10-30T14:33:04.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde86f72640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde801503b0 con 0x7fde8006c0f0 2023-10-30T14:33:04.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde86f72640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde8006c570 con 0x7fde70003f00 2023-10-30T14:33:04.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde7ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde80159f90 0x7fde8006bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:04.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde7ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde80159f90 0x7fde8006bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56222/0 (socket says 172.21.15.57:56222) 2023-10-30T14:33:04.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde7ffff640 1 -- 172.21.15.57:0/1764271860 learned_addr learned my addr 172.21.15.57:0/1764271860 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:33:04.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde8550f640 1 --2- 172.21.15.57:0/1764271860 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde8006c0f0 0x7fde80149930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:04.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde84d0e640 1 --2- 172.21.15.57:0/1764271860 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde70003f00 0x7fde8014baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:04.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde7ffff640 1 -- 172.21.15.57:0/1764271860 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde70003f00 msgr2=0x7fde8014baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:04.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde7ffff640 1 --2- 172.21.15.57:0/1764271860 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde70003f00 0x7fde8014baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:04.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde7ffff640 1 -- 172.21.15.57:0/1764271860 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde8006c0f0 msgr2=0x7fde80149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:04.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde7ffff640 1 --2- 172.21.15.57:0/1764271860 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde8006c0f0 0x7fde80149930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:04.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde7ffff640 1 -- 172.21.15.57:0/1764271860 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde801f2c60 con 0x7fde80159f90 2023-10-30T14:33:04.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde8550f640 1 --2- 172.21.15.57:0/1764271860 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde8006c0f0 0x7fde80149930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:33:04.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde7ffff640 1 --2- 172.21.15.57:0/1764271860 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde80159f90 0x7fde8006bbb0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fde6c01d2d0 tx=0x7fde70005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:04.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde7d7fa640 1 -- 172.21.15.57:0/1764271860 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c0be070 con 0x7fde80159f90 2023-10-30T14:33:04.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde7d7fa640 1 -- 172.21.15.57:0/1764271860 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6c0a5d50 con 0x7fde80159f90 2023-10-30T14:33:04.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde7d7fa640 1 -- 172.21.15.57:0/1764271860 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c004260 con 0x7fde80159f90 2023-10-30T14:33:04.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde86f72640 1 -- 172.21.15.57:0/1764271860 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde800c5cb0 con 0x7fde80159f90 2023-10-30T14:33:04.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde86f72640 1 -- 172.21.15.57:0/1764271860 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde80150040 con 0x7fde80159f90 2023-10-30T14:33:04.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.822+0000 7fde86f72640 1 -- 172.21.15.57:0/1764271860 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde8014fea0 con 0x7fde80159f90 2023-10-30T14:33:04.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.826+0000 7fde7d7fa640 1 -- 172.21.15.57:0/1764271860 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fde6c0b4040 con 0x7fde80159f90 2023-10-30T14:33:04.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.826+0000 7fde7d7fa640 1 --2- 172.21.15.57:0/1764271860 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fde640664d0 0x7fde64068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:04.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.826+0000 7fde7d7fa640 1 -- 172.21.15.57:0/1764271860 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7fde6c11dd40 con 0x7fde80159f90 2023-10-30T14:33:04.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.826+0000 7fde84d0e640 1 --2- 172.21.15.57:0/1764271860 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fde640664d0 0x7fde64068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:04.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.826+0000 7fde84d0e640 1 --2- 172.21.15.57:0/1764271860 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fde640664d0 0x7fde64068990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fde74005180 tx=0x7fde74009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:04.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:04.830+0000 7fde7d7fa640 1 -- 172.21.15.57:0/1764271860 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fde8014fea0 con 0x7fde80159f90 2023-10-30T14:33:05.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:05.022+0000 7fde86f72640 1 -- 172.21.15.57:0/1764271860 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde8014fea0 con 0x7fde80159f90 2023-10-30T14:33:05.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:05.022+0000 7fde7d7fa640 1 -- 172.21.15.57:0/1764271860 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5742 (secure 0 0 0) 0x7fde8014fea0 con 0x7fde80159f90 2023-10-30T14:33:05.023 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:33:05.023 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:33:01.327539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:resolve","state_seq":34,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:33:05.024 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 112 2023-10-30T14:33:05.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:05.022+0000 7fde86f72640 1 -- 172.21.15.57:0/1764271860 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fde640664d0 msgr2=0x7fde64068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:05.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:05.022+0000 7fde86f72640 1 --2- 172.21.15.57:0/1764271860 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fde640664d0 0x7fde64068990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fde74005180 tx=0x7fde74009040 comp rx=0 tx=0).stop 2023-10-30T14:33:05.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:05.022+0000 7fde86f72640 1 -- 172.21.15.57:0/1764271860 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde80159f90 msgr2=0x7fde8006bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:05.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:05.022+0000 7fde86f72640 1 --2- 172.21.15.57:0/1764271860 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde80159f90 0x7fde8006bbb0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fde6c01d2d0 tx=0x7fde70005800 comp rx=0 tx=0).stop 2023-10-30T14:33:05.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:05.022+0000 7fde86f72640 1 -- 172.21.15.57:0/1764271860 shutdown_connections 2023-10-30T14:33:05.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:05.022+0000 7fde86f72640 1 --2- 172.21.15.57:0/1764271860 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fde640664d0 0x7fde64068990 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:05.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:05.022+0000 7fde86f72640 1 --2- 172.21.15.57:0/1764271860 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde8006c0f0 0x7fde80149930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:05.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:05.022+0000 7fde86f72640 1 --2- 172.21.15.57:0/1764271860 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde80159f90 0x7fde8006bbb0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:05.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:05.022+0000 7fde86f72640 1 --2- 172.21.15.57:0/1764271860 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde70003f00 0x7fde8014baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:05.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:05.022+0000 7fde86f72640 1 -- 172.21.15.57:0/1764271860 >> 172.21.15.57:0/1764271860 conn(0x7fde80071fb0 msgr2=0x7fde800cc110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:05.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:05.022+0000 7fde86f72640 1 -- 172.21.15.57:0/1764271860 shutdown_connections 2023-10-30T14:33:05.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:05.026+0000 7fde86f72640 1 -- 172.21.15.57:0/1764271860 wait complete. 2023-10-30T14:33:05.044 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:33:06.591 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:33:06.597 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:33:07.046 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:33:07.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.202+0000 7f44c809a640 1 Processor -- start 2023-10-30T14:33:07.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.202+0000 7f44c809a640 1 -- start start 2023-10-30T14:33:07.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.202+0000 7f44c809a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44c0151840 0x7f44c00c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:07.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.202+0000 7f44c809a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f44c0151c20 0x7f44c0072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:07.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.202+0000 7f44c809a640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f44c00cb8b0 0x7f44c0072540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:07.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.202+0000 7f44c809a640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c00c5030 con 0x7f44c00cb8b0 2023-10-30T14:33:07.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.202+0000 7f44c809a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c0157830 con 0x7f44c0151c20 2023-10-30T14:33:07.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.202+0000 7f44c809a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c0157b10 con 0x7f44c0151840 2023-10-30T14:33:07.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.202+0000 7f44c6637640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f44c00cb8b0 0x7f44c0072540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:07.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.202+0000 7f44c6637640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f44c00cb8b0 0x7f44c0072540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56238/0 (socket says 172.21.15.57:56238) 2023-10-30T14:33:07.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.202+0000 7f44c6637640 1 -- 172.21.15.57:0/2943941442 learned_addr learned my addr 172.21.15.57:0/2943941442 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:33:07.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.202+0000 7f44c5635640 1 --2- 172.21.15.57:0/2943941442 >> v2:172.21.15.88:3300/0 conn(0x7f44c0151c20 0x7f44c0072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:07.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.202+0000 7f44c5e36640 1 --2- 172.21.15.57:0/2943941442 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44c0151840 0x7f44c00c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:07.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.202+0000 7f44c6637640 1 -- 172.21.15.57:0/2943941442 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44c0151840 msgr2=0x7f44c00c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:07.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.202+0000 7f44c6637640 1 --2- 172.21.15.57:0/2943941442 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44c0151840 0x7f44c00c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:07.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.202+0000 7f44c6637640 1 -- 172.21.15.57:0/2943941442 >> v2:172.21.15.88:3300/0 conn(0x7f44c0151c20 msgr2=0x7f44c0072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:07.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.202+0000 7f44c6637640 1 --2- 172.21.15.57:0/2943941442 >> v2:172.21.15.88:3300/0 conn(0x7f44c0151c20 0x7f44c0072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:07.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.202+0000 7f44c6637640 1 -- 172.21.15.57:0/2943941442 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c0157df0 con 0x7f44c00cb8b0 2023-10-30T14:33:07.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.206+0000 7f44c6637640 1 --2- 172.21.15.57:0/2943941442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c00cb8b0 0x7f44c0072540 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f44c006fa70 tx=0x7f44b801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dedfe13f7cfe0ff5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:07.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.206+0000 7f44c4e34640 1 -- 172.21.15.57:0/2943941442 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b80af400 con 0x7f44c00cb8b0 2023-10-30T14:33:07.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.206+0000 7f44c4e34640 1 -- 172.21.15.57:0/2943941442 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b80ab050 con 0x7f44c00cb8b0 2023-10-30T14:33:07.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.206+0000 7f44c4e34640 1 -- 172.21.15.57:0/2943941442 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b80af780 con 0x7f44c00cb8b0 2023-10-30T14:33:07.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.206+0000 7f44c809a640 1 -- 172.21.15.57:0/2943941442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c00cb8b0 msgr2=0x7f44c0072540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:07.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.206+0000 7f44c809a640 1 --2- 172.21.15.57:0/2943941442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c00cb8b0 0x7f44c0072540 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f44c006fa70 tx=0x7f44b801f1c0 comp rx=0 tx=0).stop 2023-10-30T14:33:07.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.206+0000 7f44c809a640 1 -- 172.21.15.57:0/2943941442 shutdown_connections 2023-10-30T14:33:07.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.206+0000 7f44c809a640 1 --2- 172.21.15.57:0/2943941442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c00cb8b0 0x7f44c0072540 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:07.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.206+0000 7f44c809a640 1 --2- 172.21.15.57:0/2943941442 >> v2:172.21.15.88:3300/0 conn(0x7f44c0151c20 0x7f44c0072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:07.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.206+0000 7f44c809a640 1 --2- 172.21.15.57:0/2943941442 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44c0151840 0x7f44c00c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:07.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.206+0000 7f44c809a640 1 -- 172.21.15.57:0/2943941442 >> 172.21.15.57:0/2943941442 conn(0x7f44c00cac70 msgr2=0x7f44c00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:07.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.206+0000 7f44c809a640 1 -- 172.21.15.57:0/2943941442 shutdown_connections 2023-10-30T14:33:07.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.206+0000 7f44c809a640 1 -- 172.21.15.57:0/2943941442 wait complete. 2023-10-30T14:33:07.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c809a640 1 Processor -- start 2023-10-30T14:33:07.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c809a640 1 -- start start 2023-10-30T14:33:07.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c809a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c00cb8b0 0x7f44c0145530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:07.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c809a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44c0151840 0x7f44c01f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:07.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c809a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44c0151c20 0x7f44c01565e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:07.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c809a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c01e30d0 con 0x7f44c00cb8b0 2023-10-30T14:33:07.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c809a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c0150ac0 con 0x7f44c0151840 2023-10-30T14:33:07.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c809a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c014f610 con 0x7f44c0151c20 2023-10-30T14:33:07.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c5e36640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c00cb8b0 0x7f44c0145530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:07.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c5e36640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c00cb8b0 0x7f44c0145530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56244/0 (socket says 172.21.15.57:56244) 2023-10-30T14:33:07.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c5e36640 1 -- 172.21.15.57:0/1060990466 learned_addr learned my addr 172.21.15.57:0/1060990466 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:33:07.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c5635640 1 --2- 172.21.15.57:0/1060990466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44c0151840 0x7f44c01f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:07.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c6637640 1 --2- 172.21.15.57:0/1060990466 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44c0151c20 0x7f44c01565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:07.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c5e36640 1 -- 172.21.15.57:0/1060990466 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44c0151c20 msgr2=0x7f44c01565e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:07.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c5e36640 1 --2- 172.21.15.57:0/1060990466 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44c0151c20 0x7f44c01565e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:07.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c5e36640 1 -- 172.21.15.57:0/1060990466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44c0151840 msgr2=0x7f44c01f2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:07.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c5e36640 1 --2- 172.21.15.57:0/1060990466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44c0151840 0x7f44c01f2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:07.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c5e36640 1 -- 172.21.15.57:0/1060990466 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b8009b30 con 0x7f44c00cb8b0 2023-10-30T14:33:07.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c5e36640 1 --2- 172.21.15.57:0/1060990466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c00cb8b0 0x7f44c0145530 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00d830 tx=0x7f44bc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:07.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44b67fc640 1 -- 172.21.15.57:0/1060990466 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc0096a0 con 0x7f44c00cb8b0 2023-10-30T14:33:07.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44b67fc640 1 -- 172.21.15.57:0/1060990466 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44bc009800 con 0x7f44c00cb8b0 2023-10-30T14:33:07.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44b67fc640 1 -- 172.21.15.57:0/1060990466 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc009b70 con 0x7f44c00cb8b0 2023-10-30T14:33:07.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c809a640 1 -- 172.21.15.57:0/1060990466 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c01e2120 con 0x7f44c00cb8b0 2023-10-30T14:33:07.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c809a640 1 -- 172.21.15.57:0/1060990466 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c0145da0 con 0x7f44c00cb8b0 2023-10-30T14:33:07.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.210+0000 7f44c809a640 1 -- 172.21.15.57:0/1060990466 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c01580b0 con 0x7f44c00cb8b0 2023-10-30T14:33:07.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.214+0000 7f44b67fc640 1 -- 172.21.15.57:0/1060990466 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f44bc014410 con 0x7f44c00cb8b0 2023-10-30T14:33:07.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.214+0000 7f44b67fc640 1 --2- 172.21.15.57:0/1060990466 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f44a40664a0 0x7f44a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:07.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.214+0000 7f44b67fc640 1 -- 172.21.15.57:0/1060990466 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f44bc084d70 con 0x7f44c00cb8b0 2023-10-30T14:33:07.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.214+0000 7f44c5635640 1 --2- 172.21.15.57:0/1060990466 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f44a40664a0 0x7f44a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:07.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.214+0000 7f44c5635640 1 --2- 172.21.15.57:0/1060990466 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f44a40664a0 0x7f44a4068960 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f44c01501f0 tx=0x7f44b0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:07.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.218+0000 7f44b67fc640 1 -- 172.21.15.57:0/1060990466 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f44c01580b0 con 0x7f44c00cb8b0 2023-10-30T14:33:07.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.406+0000 7f44c809a640 1 -- 172.21.15.57:0/1060990466 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44c01580b0 con 0x7f44c00cb8b0 2023-10-30T14:33:07.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.406+0000 7f44b67fc640 1 -- 172.21.15.57:0/1060990466 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5742 (secure 0 0 0) 0x7f44c01580b0 con 0x7f44c00cb8b0 2023-10-30T14:33:07.409 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:33:07.410 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:33:01.327539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:resolve","state_seq":34,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:33:07.410 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 112 2023-10-30T14:33:07.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.410+0000 7f44c809a640 1 -- 172.21.15.57:0/1060990466 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f44a40664a0 msgr2=0x7f44a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:07.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.410+0000 7f44c809a640 1 --2- 172.21.15.57:0/1060990466 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f44a40664a0 0x7f44a4068960 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f44c01501f0 tx=0x7f44b0008040 comp rx=0 tx=0).stop 2023-10-30T14:33:07.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.410+0000 7f44c809a640 1 -- 172.21.15.57:0/1060990466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c00cb8b0 msgr2=0x7f44c0145530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:07.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.410+0000 7f44c809a640 1 --2- 172.21.15.57:0/1060990466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c00cb8b0 0x7f44c0145530 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00d830 tx=0x7f44bc00dcf0 comp rx=0 tx=0).stop 2023-10-30T14:33:07.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.410+0000 7f44c809a640 1 -- 172.21.15.57:0/1060990466 shutdown_connections 2023-10-30T14:33:07.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.410+0000 7f44c809a640 1 --2- 172.21.15.57:0/1060990466 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f44a40664a0 0x7f44a4068960 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:07.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.410+0000 7f44c809a640 1 --2- 172.21.15.57:0/1060990466 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44c0151c20 0x7f44c01565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:07.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.410+0000 7f44c809a640 1 --2- 172.21.15.57:0/1060990466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44c0151840 0x7f44c01f2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:07.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.410+0000 7f44c809a640 1 --2- 172.21.15.57:0/1060990466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c00cb8b0 0x7f44c0145530 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:07.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.410+0000 7f44c809a640 1 -- 172.21.15.57:0/1060990466 >> 172.21.15.57:0/1060990466 conn(0x7f44c00cac70 msgr2=0x7f44c0072000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:07.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.410+0000 7f44c809a640 1 -- 172.21.15.57:0/1060990466 shutdown_connections 2023-10-30T14:33:07.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:07.410+0000 7f44c809a640 1 -- 172.21.15.57:0/1060990466 wait complete. 2023-10-30T14:33:07.431 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 112, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 112, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5891": { "addr": "172.21.15.88:6839/3604033338", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 3604033338, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 3604033338, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5891, "incarnation": 111, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:resolve", "state_seq": 34 }, "gid_6204": { "addr": "172.21.15.57:6837/1843528309", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 1843528309, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 1843528309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6204, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_6414": { "addr": "172.21.15.57:6834/2321408149", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2321408149, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2321408149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6414, "incarnation": 99, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:33:01.327539+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6414, "mds_1": 5891, "mds_2": 6204 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:33:09.432 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:33:09.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.586+0000 7f4c24a91640 1 Processor -- start 2023-10-30T14:33:09.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.586+0000 7f4c24a91640 1 -- start start 2023-10-30T14:33:09.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.586+0000 7f4c24a91640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c2014fc50 0x7f4c2006fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:09.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.586+0000 7f4c24a91640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4c200c6320 0x7f4c20070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:09.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.586+0000 7f4c24a91640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4c2006be90 0x7f4c2006c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:09.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.586+0000 7f4c24a91640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c200c5f00 con 0x7f4c200c6320 2023-10-30T14:33:09.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.586+0000 7f4c24a91640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c200c55f0 con 0x7f4c2006be90 2023-10-30T14:33:09.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.586+0000 7f4c24a91640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c20071350 con 0x7f4c2014fc50 2023-10-30T14:33:09.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.586+0000 7f4c17fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4c200c6320 0x7f4c20070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:09.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.586+0000 7f4c17fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4c200c6320 0x7f4c20070e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56250/0 (socket says 172.21.15.57:56250) 2023-10-30T14:33:09.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.586+0000 7f4c17fff640 1 -- 172.21.15.57:0/1305859659 learned_addr learned my addr 172.21.15.57:0/1305859659 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:33:09.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.586+0000 7f4c1ed9d640 1 --2- 172.21.15.57:0/1305859659 >> v2:172.21.15.88:3300/0 conn(0x7f4c2006be90 0x7f4c2006c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:09.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.586+0000 7f4c1e59c640 1 --2- 172.21.15.57:0/1305859659 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c2014fc50 0x7f4c2006fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:09.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.590+0000 7f4c17fff640 1 -- 172.21.15.57:0/1305859659 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c2014fc50 msgr2=0x7f4c2006fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:09.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.590+0000 7f4c17fff640 1 --2- 172.21.15.57:0/1305859659 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c2014fc50 0x7f4c2006fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:09.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.590+0000 7f4c17fff640 1 -- 172.21.15.57:0/1305859659 >> v2:172.21.15.88:3300/0 conn(0x7f4c2006be90 msgr2=0x7f4c2006c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:09.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.590+0000 7f4c17fff640 1 --2- 172.21.15.57:0/1305859659 >> v2:172.21.15.88:3300/0 conn(0x7f4c2006be90 0x7f4c2006c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:09.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.590+0000 7f4c17fff640 1 -- 172.21.15.57:0/1305859659 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c201567b0 con 0x7f4c200c6320 2023-10-30T14:33:09.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.590+0000 7f4c17fff640 1 --2- 172.21.15.57:0/1305859659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c200c6320 0x7f4c20070e10 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f4c200c5270 tx=0x7f4c0801e920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=171c1ac38fd2205e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:09.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.590+0000 7f4c1dd9b640 1 -- 172.21.15.57:0/1305859659 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c08004230 con 0x7f4c200c6320 2023-10-30T14:33:09.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.590+0000 7f4c1dd9b640 1 -- 172.21.15.57:0/1305859659 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c080ab070 con 0x7f4c200c6320 2023-10-30T14:33:09.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.590+0000 7f4c1dd9b640 1 -- 172.21.15.57:0/1305859659 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c080a7730 con 0x7f4c200c6320 2023-10-30T14:33:09.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.590+0000 7f4c24a91640 1 -- 172.21.15.57:0/1305859659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c200c6320 msgr2=0x7f4c20070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:09.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.590+0000 7f4c24a91640 1 --2- 172.21.15.57:0/1305859659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c200c6320 0x7f4c20070e10 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f4c200c5270 tx=0x7f4c0801e920 comp rx=0 tx=0).stop 2023-10-30T14:33:09.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.590+0000 7f4c24a91640 1 -- 172.21.15.57:0/1305859659 shutdown_connections 2023-10-30T14:33:09.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.590+0000 7f4c24a91640 1 --2- 172.21.15.57:0/1305859659 >> v2:172.21.15.88:3300/0 conn(0x7f4c2006be90 0x7f4c2006c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:09.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.590+0000 7f4c24a91640 1 --2- 172.21.15.57:0/1305859659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c200c6320 0x7f4c20070e10 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:09.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.590+0000 7f4c24a91640 1 --2- 172.21.15.57:0/1305859659 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c2014fc50 0x7f4c2006fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:09.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.590+0000 7f4c24a91640 1 -- 172.21.15.57:0/1305859659 >> 172.21.15.57:0/1305859659 conn(0x7f4c200cac70 msgr2=0x7f4c200cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:09.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.590+0000 7f4c24a91640 1 -- 172.21.15.57:0/1305859659 shutdown_connections 2023-10-30T14:33:09.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.590+0000 7f4c24a91640 1 -- 172.21.15.57:0/1305859659 wait complete. 2023-10-30T14:33:09.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c24a91640 1 Processor -- start 2023-10-30T14:33:09.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c24a91640 1 -- start start 2023-10-30T14:33:09.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c24a91640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c2006be90 0x7f4c2014d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:09.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c24a91640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c200c6320 0x7f4c20137db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:09.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c24a91640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c2014fc50 0x7f4c20154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:09.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c24a91640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c200c8e00 con 0x7f4c200c6320 2023-10-30T14:33:09.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c24a91640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2014d950 con 0x7f4c2006be90 2023-10-30T14:33:09.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c24a91640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c201f4da0 con 0x7f4c2014fc50 2023-10-30T14:33:09.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c17fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c200c6320 0x7f4c20137db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:09.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c17fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c200c6320 0x7f4c20137db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56266/0 (socket says 172.21.15.57:56266) 2023-10-30T14:33:09.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c17fff640 1 -- 172.21.15.57:0/2530857168 learned_addr learned my addr 172.21.15.57:0/2530857168 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:33:09.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c1e59c640 1 --2- 172.21.15.57:0/2530857168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c2006be90 0x7f4c2014d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:09.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c1ed9d640 1 --2- 172.21.15.57:0/2530857168 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c2014fc50 0x7f4c20154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:09.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c17fff640 1 -- 172.21.15.57:0/2530857168 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c2014fc50 msgr2=0x7f4c20154db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:09.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c17fff640 1 --2- 172.21.15.57:0/2530857168 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c2014fc50 0x7f4c20154db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:09.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c17fff640 1 -- 172.21.15.57:0/2530857168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c2006be90 msgr2=0x7f4c2014d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:09.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c17fff640 1 --2- 172.21.15.57:0/2530857168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c2006be90 0x7f4c2014d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:09.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c17fff640 1 -- 172.21.15.57:0/2530857168 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0800b780 con 0x7f4c200c6320 2023-10-30T14:33:09.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c17fff640 1 --2- 172.21.15.57:0/2530857168 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c200c6320 0x7f4c20137db0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f4c0801cfd0 tx=0x7f4c0801b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:09.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.594+0000 7f4c16ffd640 1 -- 172.21.15.57:0/2530857168 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c080a79c0 con 0x7f4c200c6320 2023-10-30T14:33:09.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.598+0000 7f4c16ffd640 1 -- 172.21.15.57:0/2530857168 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c080a7b20 con 0x7f4c200c6320 2023-10-30T14:33:09.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.598+0000 7f4c24a91640 1 -- 172.21.15.57:0/2530857168 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c201f5080 con 0x7f4c200c6320 2023-10-30T14:33:09.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.598+0000 7f4c16ffd640 1 -- 172.21.15.57:0/2530857168 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c080a7cb0 con 0x7f4c200c6320 2023-10-30T14:33:09.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.598+0000 7f4c24a91640 1 -- 172.21.15.57:0/2530857168 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c201f5640 con 0x7f4c200c6320 2023-10-30T14:33:09.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.598+0000 7f4c24a91640 1 -- 172.21.15.57:0/2530857168 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c20155c00 con 0x7f4c200c6320 2023-10-30T14:33:09.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.598+0000 7f4c16ffd640 1 -- 172.21.15.57:0/2530857168 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4c08003620 con 0x7f4c200c6320 2023-10-30T14:33:09.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.598+0000 7f4c16ffd640 1 --2- 172.21.15.57:0/2530857168 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4c000664a0 0x7f4c00068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:09.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.598+0000 7f4c16ffd640 1 -- 172.21.15.57:0/2530857168 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f4c0811e900 con 0x7f4c200c6320 2023-10-30T14:33:09.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.598+0000 7f4c1e59c640 1 --2- 172.21.15.57:0/2530857168 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4c000664a0 0x7f4c00068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:09.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.602+0000 7f4c1e59c640 1 --2- 172.21.15.57:0/2530857168 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4c000664a0 0x7f4c00068960 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4c1000b930 tx=0x7f4c10008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:09.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.602+0000 7f4c16ffd640 1 -- 172.21.15.57:0/2530857168 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4c080eba60 con 0x7f4c200c6320 2023-10-30T14:33:09.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.790+0000 7f4c24a91640 1 -- 172.21.15.57:0/2530857168 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c2004e170 con 0x7f4c200c6320 2023-10-30T14:33:09.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.794+0000 7f4c16ffd640 1 -- 172.21.15.57:0/2530857168 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+5744 (secure 0 0 0) 0x7f4c080ef080 con 0x7f4c200c6320 2023-10-30T14:33:09.793 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:33:09.793 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:33:07.074168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:33:09.794 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 113 2023-10-30T14:33:09.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.794+0000 7f4c24a91640 1 -- 172.21.15.57:0/2530857168 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4c000664a0 msgr2=0x7f4c00068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:09.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.794+0000 7f4c24a91640 1 --2- 172.21.15.57:0/2530857168 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4c000664a0 0x7f4c00068960 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4c1000b930 tx=0x7f4c10008040 comp rx=0 tx=0).stop 2023-10-30T14:33:09.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.794+0000 7f4c24a91640 1 -- 172.21.15.57:0/2530857168 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c200c6320 msgr2=0x7f4c20137db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:09.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.794+0000 7f4c24a91640 1 --2- 172.21.15.57:0/2530857168 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c200c6320 0x7f4c20137db0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f4c0801cfd0 tx=0x7f4c0801b6c0 comp rx=0 tx=0).stop 2023-10-30T14:33:09.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.794+0000 7f4c24a91640 1 -- 172.21.15.57:0/2530857168 shutdown_connections 2023-10-30T14:33:09.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.794+0000 7f4c24a91640 1 --2- 172.21.15.57:0/2530857168 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4c000664a0 0x7f4c00068960 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:09.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.794+0000 7f4c24a91640 1 --2- 172.21.15.57:0/2530857168 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c2014fc50 0x7f4c20154db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:09.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.794+0000 7f4c24a91640 1 --2- 172.21.15.57:0/2530857168 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c200c6320 0x7f4c20137db0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:09.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.794+0000 7f4c24a91640 1 --2- 172.21.15.57:0/2530857168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c2006be90 0x7f4c2014d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:09.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.794+0000 7f4c24a91640 1 -- 172.21.15.57:0/2530857168 >> 172.21.15.57:0/2530857168 conn(0x7f4c200cac70 msgr2=0x7f4c201405e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:09.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.794+0000 7f4c24a91640 1 -- 172.21.15.57:0/2530857168 shutdown_connections 2023-10-30T14:33:09.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:09.794+0000 7f4c24a91640 1 -- 172.21.15.57:0/2530857168 wait complete. 2023-10-30T14:33:09.814 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:33:11.815 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:33:11.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.970+0000 7f1a7131d640 1 Processor -- start 2023-10-30T14:33:11.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.970+0000 7f1a7131d640 1 -- start start 2023-10-30T14:33:11.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.970+0000 7f1a7131d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1a6c14fc50 0x7f1a6c06fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:11.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.970+0000 7f1a7131d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1a6c06be90 0x7f1a6c06c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:11.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.970+0000 7f1a7131d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6c0c55f0 con 0x7f1a6c0c6320 2023-10-30T14:33:11.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.970+0000 7f1a7131d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a6c0c5f00 con 0x7f1a6c06be90 2023-10-30T14:33:11.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.970+0000 7f1a7131d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a6c1567b0 con 0x7f1a6c14fc50 2023-10-30T14:33:11.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.970+0000 7f1a63fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1a6c0c6320 0x7f1a6c070e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:32808/0 (socket says 172.21.15.57:32808) 2023-10-30T14:33:11.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.970+0000 7f1a63fff640 1 -- 172.21.15.57:0/723615913 learned_addr learned my addr 172.21.15.57:0/723615913 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:33:11.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.970+0000 7f1a6b7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1a6c06be90 0x7f1a6c06c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:11.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.970+0000 7f1a6affd640 1 --2- 172.21.15.57:0/723615913 >> v2:172.21.15.88:3300/0 conn(0x7f1a6c14fc50 0x7f1a6c06fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:11.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.970+0000 7f1a6a7fc640 1 -- 172.21.15.57:0/723615913 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485799856 0 0) 0x7f1a6c0c55f0 con 0x7f1a6c0c6320 2023-10-30T14:33:11.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.970+0000 7f1a6a7fc640 1 -- 172.21.15.57:0/723615913 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a58003530 con 0x7f1a6c0c6320 2023-10-30T14:33:11.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.970+0000 7f1a6b7fe640 1 -- 172.21.15.57:0/723615913 >> v2:172.21.15.88:3300/0 conn(0x7f1a6c14fc50 msgr2=0x7f1a6c06fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:11.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.970+0000 7f1a6b7fe640 1 --2- 172.21.15.57:0/723615913 >> v2:172.21.15.88:3300/0 conn(0x7f1a6c14fc50 0x7f1a6c06fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:11.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.970+0000 7f1a6b7fe640 1 -- 172.21.15.57:0/723615913 >> v1:172.21.15.57:6789/0 conn(0x7f1a6c0c6320 legacy=0x7f1a6c070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:11.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.970+0000 7f1a6b7fe640 1 -- 172.21.15.57:0/723615913 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c156a70 con 0x7f1a6c06be90 2023-10-30T14:33:11.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.974+0000 7f1a6b7fe640 1 --2- 172.21.15.57:0/723615913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a6c06be90 0x7f1a6c06c2e0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c06f6e0 tx=0x7f1a6401d9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1a5441ccbce0db server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:11.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.974+0000 7f1a6a7fc640 1 -- 172.21.15.57:0/723615913 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6400a2d0 con 0x7f1a6c06be90 2023-10-30T14:33:11.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.974+0000 7f1a6a7fc640 1 -- 172.21.15.57:0/723615913 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a640a67e0 con 0x7f1a6c06be90 2023-10-30T14:33:11.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.974+0000 7f1a6a7fc640 1 -- 172.21.15.57:0/723615913 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a640a6b50 con 0x7f1a6c06be90 2023-10-30T14:33:11.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.974+0000 7f1a7131d640 1 -- 172.21.15.57:0/723615913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a6c06be90 msgr2=0x7f1a6c06c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:11.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.974+0000 7f1a7131d640 1 --2- 172.21.15.57:0/723615913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a6c06be90 0x7f1a6c06c2e0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c06f6e0 tx=0x7f1a6401d9a0 comp rx=0 tx=0).stop 2023-10-30T14:33:11.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.974+0000 7f1a7131d640 1 -- 172.21.15.57:0/723615913 shutdown_connections 2023-10-30T14:33:11.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.974+0000 7f1a7131d640 1 --2- 172.21.15.57:0/723615913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a6c06be90 0x7f1a6c06c2e0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:11.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.974+0000 7f1a7131d640 1 --2- 172.21.15.57:0/723615913 >> v2:172.21.15.88:3300/0 conn(0x7f1a6c14fc50 0x7f1a6c06fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:11.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.974+0000 7f1a7131d640 1 -- 172.21.15.57:0/723615913 >> 172.21.15.57:0/723615913 conn(0x7f1a6c0cac70 msgr2=0x7f1a6c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:11.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.974+0000 7f1a7131d640 1 -- 172.21.15.57:0/723615913 shutdown_connections 2023-10-30T14:33:11.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.974+0000 7f1a7131d640 1 -- 172.21.15.57:0/723615913 wait complete. 2023-10-30T14:33:11.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a7131d640 1 Processor -- start 2023-10-30T14:33:11.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a7131d640 1 -- start start 2023-10-30T14:33:11.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a7131d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a6c06be90 0x7f1a6c0714d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:11.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a7131d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a6c0c6320 0x7f1a6c14c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:11.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a7131d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a6c14fc50 0x7f1a6c0ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:11.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a7131d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c14c640 con 0x7f1a6c0c6320 2023-10-30T14:33:11.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a7131d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c14a5e0 con 0x7f1a6c06be90 2023-10-30T14:33:11.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a7131d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c1f49f0 con 0x7f1a6c14fc50 2023-10-30T14:33:11.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a63fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a6c0c6320 0x7f1a6c14c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:11.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a63fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a6c0c6320 0x7f1a6c14c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56282/0 (socket says 172.21.15.57:56282) 2023-10-30T14:33:11.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a63fff640 1 -- 172.21.15.57:0/1141956452 learned_addr learned my addr 172.21.15.57:0/1141956452 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:33:11.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a6affd640 1 --2- 172.21.15.57:0/1141956452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a6c06be90 0x7f1a6c0714d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:11.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a6b7fe640 1 --2- 172.21.15.57:0/1141956452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a6c14fc50 0x7f1a6c0ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:11.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a63fff640 1 -- 172.21.15.57:0/1141956452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a6c14fc50 msgr2=0x7f1a6c0ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:11.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a63fff640 1 --2- 172.21.15.57:0/1141956452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a6c14fc50 0x7f1a6c0ca220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:11.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a63fff640 1 -- 172.21.15.57:0/1141956452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a6c06be90 msgr2=0x7f1a6c0714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:11.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a63fff640 1 --2- 172.21.15.57:0/1141956452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a6c06be90 0x7f1a6c0714d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:11.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a63fff640 1 -- 172.21.15.57:0/1141956452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6400ab80 con 0x7f1a6c0c6320 2023-10-30T14:33:11.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a6b7fe640 1 --2- 172.21.15.57:0/1141956452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a6c14fc50 0x7f1a6c0ca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:33:11.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a63fff640 1 --2- 172.21.15.57:0/1141956452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a6c0c6320 0x7f1a6c14c990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400c450 tx=0x7f1a5400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:11.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a637fe640 1 -- 172.21.15.57:0/1141956452 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54005cf0 con 0x7f1a6c0c6320 2023-10-30T14:33:11.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a637fe640 1 -- 172.21.15.57:0/1141956452 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a54005e50 con 0x7f1a6c0c6320 2023-10-30T14:33:11.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a637fe640 1 -- 172.21.15.57:0/1141956452 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54007290 con 0x7f1a6c0c6320 2023-10-30T14:33:11.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a7131d640 1 -- 172.21.15.57:0/1141956452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6c14a300 con 0x7f1a6c0c6320 2023-10-30T14:33:11.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.978+0000 7f1a7131d640 1 -- 172.21.15.57:0/1141956452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6c14bff0 con 0x7f1a6c0c6320 2023-10-30T14:33:11.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.982+0000 7f1a7131d640 1 -- 172.21.15.57:0/1141956452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6c04e170 con 0x7f1a6c0c6320 2023-10-30T14:33:11.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.982+0000 7f1a637fe640 1 -- 172.21.15.57:0/1141956452 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1a540073f0 con 0x7f1a6c0c6320 2023-10-30T14:33:11.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.982+0000 7f1a637fe640 1 --2- 172.21.15.57:0/1141956452 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1a4c0664a0 0x7f1a4c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:11.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.982+0000 7f1a637fe640 1 -- 172.21.15.57:0/1141956452 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f1a54083810 con 0x7f1a6c0c6320 2023-10-30T14:33:11.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.982+0000 7f1a6affd640 1 --2- 172.21.15.57:0/1141956452 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1a4c0664a0 0x7f1a4c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:11.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.982+0000 7f1a6affd640 1 --2- 172.21.15.57:0/1141956452 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1a4c0664a0 0x7f1a4c068960 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009ac0 tx=0x7f1a5c009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:11.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:11.986+0000 7f1a637fe640 1 -- 172.21.15.57:0/1141956452 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1a54050880 con 0x7f1a6c0c6320 2023-10-30T14:33:12.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:12.178+0000 7f1a7131d640 1 -- 172.21.15.57:0/1141956452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a6c0c7300 con 0x7f1a6c0c6320 2023-10-30T14:33:12.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:12.178+0000 7f1a637fe640 1 -- 172.21.15.57:0/1141956452 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+5744 (secure 0 0 0) 0x7f1a54017070 con 0x7f1a6c0c6320 2023-10-30T14:33:12.180 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:33:12.181 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:33:07.074168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:33:12.181 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 113 2023-10-30T14:33:12.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:12.182+0000 7f1a7131d640 1 -- 172.21.15.57:0/1141956452 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1a4c0664a0 msgr2=0x7f1a4c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:12.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:12.182+0000 7f1a7131d640 1 --2- 172.21.15.57:0/1141956452 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1a4c0664a0 0x7f1a4c068960 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009ac0 tx=0x7f1a5c009190 comp rx=0 tx=0).stop 2023-10-30T14:33:12.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:12.182+0000 7f1a7131d640 1 -- 172.21.15.57:0/1141956452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a6c0c6320 msgr2=0x7f1a6c14c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:12.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:12.182+0000 7f1a7131d640 1 --2- 172.21.15.57:0/1141956452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a6c0c6320 0x7f1a6c14c990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400c450 tx=0x7f1a5400c910 comp rx=0 tx=0).stop 2023-10-30T14:33:12.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:12.182+0000 7f1a7131d640 1 -- 172.21.15.57:0/1141956452 shutdown_connections 2023-10-30T14:33:12.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:12.182+0000 7f1a7131d640 1 --2- 172.21.15.57:0/1141956452 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1a4c0664a0 0x7f1a4c068960 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:12.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:12.182+0000 7f1a7131d640 1 --2- 172.21.15.57:0/1141956452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a6c14fc50 0x7f1a6c0ca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:12.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:12.182+0000 7f1a7131d640 1 --2- 172.21.15.57:0/1141956452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a6c0c6320 0x7f1a6c14c990 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:12.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:12.182+0000 7f1a7131d640 1 --2- 172.21.15.57:0/1141956452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a6c06be90 0x7f1a6c0714d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:12.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:12.182+0000 7f1a7131d640 1 -- 172.21.15.57:0/1141956452 >> 172.21.15.57:0/1141956452 conn(0x7f1a6c0cac70 msgr2=0x7f1a6c06fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:12.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:12.182+0000 7f1a7131d640 1 -- 172.21.15.57:0/1141956452 shutdown_connections 2023-10-30T14:33:12.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:12.182+0000 7f1a7131d640 1 -- 172.21.15.57:0/1141956452 wait complete. 2023-10-30T14:33:12.201 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:33:14.203 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:33:14.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.358+0000 7f06cd8ad640 1 Processor -- start 2023-10-30T14:33:14.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.358+0000 7f06cd8ad640 1 -- start start 2023-10-30T14:33:14.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.358+0000 7f06cd8ad640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f06c815ac70 0x7f06c815cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:14.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.358+0000 7f06cd8ad640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c815dcc0 con 0x7f06c806aeb0 2023-10-30T14:33:14.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.358+0000 7f06cd8ad640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c8159d80 con 0x7f06c80c8eb0 2023-10-30T14:33:14.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.358+0000 7f06cd8ad640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c8159800 con 0x7f06c815ac70 2023-10-30T14:33:14.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.358+0000 7f06cd0ac640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f06c806aeb0 0x7f06c8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:51530/0 (socket says 172.21.15.57:51530) 2023-10-30T14:33:14.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.358+0000 7f06c7fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f06c815ac70 0x7f06c815cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:14.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.358+0000 7f06cd0ac640 1 -- 172.21.15.57:0/877705217 learned_addr learned my addr 172.21.15.57:0/877705217 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:33:14.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.358+0000 7f06c7fff640 1 -- 172.21.15.57:0/877705217 >> v1:172.21.15.88:6789/0 conn(0x7f06c80c8eb0 legacy=0x7f06c815a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:14.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.358+0000 7f06c7fff640 1 -- 172.21.15.57:0/877705217 >> v1:172.21.15.57:6789/0 conn(0x7f06c806aeb0 legacy=0x7f06c8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:14.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.358+0000 7f06c7fff640 1 -- 172.21.15.57:0/877705217 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c815d420 con 0x7f06c815ac70 2023-10-30T14:33:14.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.362+0000 7f06c7fff640 1 --2- 172.21.15.57:0/877705217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c815ac70 0x7f06c815cd70 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f06b40097c0 tx=0x7f06b401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb8d7adf9cebd29 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:14.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.362+0000 7f06c77fe640 1 -- 172.21.15.57:0/877705217 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b40a8970 con 0x7f06c815ac70 2023-10-30T14:33:14.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.362+0000 7f06c77fe640 1 -- 172.21.15.57:0/877705217 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b40a8ad0 con 0x7f06c815ac70 2023-10-30T14:33:14.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.362+0000 7f06c77fe640 1 -- 172.21.15.57:0/877705217 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b40a8e60 con 0x7f06c815ac70 2023-10-30T14:33:14.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.362+0000 7f06cd8ad640 1 -- 172.21.15.57:0/877705217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c815ac70 msgr2=0x7f06c815cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:14.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.362+0000 7f06cd8ad640 1 --2- 172.21.15.57:0/877705217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c815ac70 0x7f06c815cd70 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f06b40097c0 tx=0x7f06b401ec70 comp rx=0 tx=0).stop 2023-10-30T14:33:14.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.362+0000 7f06cd8ad640 1 -- 172.21.15.57:0/877705217 shutdown_connections 2023-10-30T14:33:14.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.362+0000 7f06cd8ad640 1 --2- 172.21.15.57:0/877705217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c815ac70 0x7f06c815cd70 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:14.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.362+0000 7f06cd8ad640 1 -- 172.21.15.57:0/877705217 >> 172.21.15.57:0/877705217 conn(0x7f06c80cef00 msgr2=0x7f06c80cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:14.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.362+0000 7f06cd8ad640 1 -- 172.21.15.57:0/877705217 shutdown_connections 2023-10-30T14:33:14.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.362+0000 7f06cd8ad640 1 -- 172.21.15.57:0/877705217 wait complete. 2023-10-30T14:33:14.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.362+0000 7f06cd8ad640 1 Processor -- start 2023-10-30T14:33:14.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cd8ad640 1 -- start start 2023-10-30T14:33:14.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cd8ad640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c806aeb0 0x7f06c8158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:14.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cd8ad640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c80c8eb0 0x7f06c81e2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:14.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cd8ad640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c815ac70 0x7f06c814d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:14.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cd8ad640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c81e3ae0 con 0x7f06c815ac70 2023-10-30T14:33:14.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cd8ad640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c81e3420 con 0x7f06c806aeb0 2023-10-30T14:33:14.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cd8ad640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c81549e0 con 0x7f06c80c8eb0 2023-10-30T14:33:14.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cd0ac640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c815ac70 0x7f06c814d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:14.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cd0ac640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c815ac70 0x7f06c814d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36986/0 (socket says 172.21.15.57:36986) 2023-10-30T14:33:14.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cd0ac640 1 -- 172.21.15.57:0/2038964273 learned_addr learned my addr 172.21.15.57:0/2038964273 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:33:14.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06c7fff640 1 --2- 172.21.15.57:0/2038964273 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c80c8eb0 0x7f06c81e2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:14.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cc8ab640 1 --2- 172.21.15.57:0/2038964273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c806aeb0 0x7f06c8158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:14.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cd0ac640 1 -- 172.21.15.57:0/2038964273 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c80c8eb0 msgr2=0x7f06c81e2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:14.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cd0ac640 1 --2- 172.21.15.57:0/2038964273 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c80c8eb0 0x7f06c81e2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:14.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cd0ac640 1 -- 172.21.15.57:0/2038964273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c806aeb0 msgr2=0x7f06c8158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:14.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cd0ac640 1 --2- 172.21.15.57:0/2038964273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c806aeb0 0x7f06c8158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:14.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cd0ac640 1 -- 172.21.15.57:0/2038964273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b4009420 con 0x7f06c815ac70 2023-10-30T14:33:14.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06c7fff640 1 --2- 172.21.15.57:0/2038964273 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c80c8eb0 0x7f06c81e2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:33:14.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cc8ab640 1 --2- 172.21.15.57:0/2038964273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c806aeb0 0x7f06c8158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:33:14.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cd0ac640 1 --2- 172.21.15.57:0/2038964273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c815ac70 0x7f06c814d710 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f06c000cbc0 tx=0x7f06c000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:14.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06c57fa640 1 -- 172.21.15.57:0/2038964273 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c000ad40 con 0x7f06c815ac70 2023-10-30T14:33:14.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06c57fa640 1 -- 172.21.15.57:0/2038964273 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c0012400 con 0x7f06c815ac70 2023-10-30T14:33:14.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06c57fa640 1 -- 172.21.15.57:0/2038964273 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c000aea0 con 0x7f06c815ac70 2023-10-30T14:33:14.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cd8ad640 1 -- 172.21.15.57:0/2038964273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c81545f0 con 0x7f06c815ac70 2023-10-30T14:33:14.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cd8ad640 1 -- 172.21.15.57:0/2038964273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c8154310 con 0x7f06c815ac70 2023-10-30T14:33:14.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.366+0000 7f06cd8ad640 1 -- 172.21.15.57:0/2038964273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c81e2c00 con 0x7f06c815ac70 2023-10-30T14:33:14.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.370+0000 7f06c57fa640 1 -- 172.21.15.57:0/2038964273 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f06c0007070 con 0x7f06c815ac70 2023-10-30T14:33:14.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.370+0000 7f06c57fa640 1 --2- 172.21.15.57:0/2038964273 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06ac0664a0 0x7f06ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:14.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.370+0000 7f06c57fa640 1 -- 172.21.15.57:0/2038964273 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f06c0083fe0 con 0x7f06c815ac70 2023-10-30T14:33:14.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.370+0000 7f06cc8ab640 1 --2- 172.21.15.57:0/2038964273 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06ac0664a0 0x7f06ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:14.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.370+0000 7f06cc8ab640 1 --2- 172.21.15.57:0/2038964273 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06ac0664a0 0x7f06ac068960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f06bc000f70 tx=0x7f06bc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:14.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.374+0000 7f06c57fa640 1 -- 172.21.15.57:0/2038964273 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f06c81e2c00 con 0x7f06c815ac70 2023-10-30T14:33:14.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.569+0000 7f06cd8ad640 1 -- 172.21.15.57:0/2038964273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06c815dcc0 con 0x7f06c815ac70 2023-10-30T14:33:14.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.569+0000 7f06c57fa640 1 -- 172.21.15.57:0/2038964273 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5741 (secure 0 0 0) 0x7f06c815dcc0 con 0x7f06c815ac70 2023-10-30T14:33:14.571 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:33:14.571 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:33:12.968830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:33:14.572 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 114 2023-10-30T14:33:14.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.569+0000 7f06cd8ad640 1 -- 172.21.15.57:0/2038964273 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06ac0664a0 msgr2=0x7f06ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:14.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.569+0000 7f06cd8ad640 1 --2- 172.21.15.57:0/2038964273 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06ac0664a0 0x7f06ac068960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f06bc000f70 tx=0x7f06bc009040 comp rx=0 tx=0).stop 2023-10-30T14:33:14.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.569+0000 7f06cd8ad640 1 -- 172.21.15.57:0/2038964273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c815ac70 msgr2=0x7f06c814d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:14.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.569+0000 7f06cd8ad640 1 --2- 172.21.15.57:0/2038964273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c815ac70 0x7f06c814d710 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f06c000cbc0 tx=0x7f06c000a430 comp rx=0 tx=0).stop 2023-10-30T14:33:14.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.573+0000 7f06cd8ad640 1 -- 172.21.15.57:0/2038964273 shutdown_connections 2023-10-30T14:33:14.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.573+0000 7f06cd8ad640 1 --2- 172.21.15.57:0/2038964273 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06ac0664a0 0x7f06ac068960 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:14.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.573+0000 7f06cd8ad640 1 --2- 172.21.15.57:0/2038964273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06c815ac70 0x7f06c814d710 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:14.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.573+0000 7f06cd8ad640 1 --2- 172.21.15.57:0/2038964273 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06c80c8eb0 0x7f06c81e2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:14.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.573+0000 7f06cd8ad640 1 --2- 172.21.15.57:0/2038964273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06c806aeb0 0x7f06c8158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:14.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.573+0000 7f06cd8ad640 1 -- 172.21.15.57:0/2038964273 >> 172.21.15.57:0/2038964273 conn(0x7f06c80cef00 msgr2=0x7f06c80c9ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:14.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.573+0000 7f06cd8ad640 1 -- 172.21.15.57:0/2038964273 shutdown_connections 2023-10-30T14:33:14.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:14.573+0000 7f06cd8ad640 1 -- 172.21.15.57:0/2038964273 wait complete. 2023-10-30T14:33:14.592 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:33:16.593 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:33:16.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.749+0000 7fba5de91640 1 Processor -- start 2023-10-30T14:33:16.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.749+0000 7fba5de91640 1 -- start start 2023-10-30T14:33:16.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.749+0000 7fba5de91640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba58153c80 0x7fba5815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:16.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.749+0000 7fba5de91640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fba580c42a0 0x7fba580ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:16.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.749+0000 7fba5de91640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba58159f20 con 0x7fba580c4a10 2023-10-30T14:33:16.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.749+0000 7fba5de91640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba5815dec0 con 0x7fba580c42a0 2023-10-30T14:33:16.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.749+0000 7fba5de91640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba581599a0 con 0x7fba58153c80 2023-10-30T14:33:16.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.749+0000 7fba56ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fba580c4a10 0x7fba5815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:51546/0 (socket says 172.21.15.57:51546) 2023-10-30T14:33:16.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.749+0000 7fba56ffd640 1 -- 172.21.15.57:0/1405809709 learned_addr learned my addr 172.21.15.57:0/1405809709 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:33:16.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.749+0000 7fba57fff640 1 --2- 172.21.15.57:0/1405809709 >> v2:172.21.15.57:3300/0 conn(0x7fba580c42a0 0x7fba580ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:16.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.749+0000 7fba577fe640 1 --2- 172.21.15.57:0/1405809709 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba58153c80 0x7fba5815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:16.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.749+0000 7fba57fff640 1 -- 172.21.15.57:0/1405809709 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba58153c80 msgr2=0x7fba5815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:16.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.749+0000 7fba567fc640 1 -- 172.21.15.57:0/1405809709 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907686962 0 0) 0x7fba58159f20 con 0x7fba580c4a10 2023-10-30T14:33:16.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.749+0000 7fba57fff640 1 --2- 172.21.15.57:0/1405809709 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba58153c80 0x7fba5815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:16.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.749+0000 7fba57fff640 1 -- 172.21.15.57:0/1405809709 >> v1:172.21.15.57:6789/0 conn(0x7fba580c4a10 legacy=0x7fba5815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:16.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.749+0000 7fba57fff640 1 -- 172.21.15.57:0/1405809709 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5815d620 con 0x7fba580c42a0 2023-10-30T14:33:16.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.753+0000 7fba57fff640 1 --2- 172.21.15.57:0/1405809709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba580c42a0 0x7fba580ca1e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fba50009d90 tx=0x7fba5001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c23e682818fbc5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:16.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.753+0000 7fba567fc640 1 -- 172.21.15.57:0/1405809709 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba500af600 con 0x7fba580c42a0 2023-10-30T14:33:16.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.753+0000 7fba567fc640 1 -- 172.21.15.57:0/1405809709 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba500ab050 con 0x7fba580c42a0 2023-10-30T14:33:16.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.753+0000 7fba567fc640 1 -- 172.21.15.57:0/1405809709 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba500af980 con 0x7fba580c42a0 2023-10-30T14:33:16.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.753+0000 7fba5de91640 1 -- 172.21.15.57:0/1405809709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba580c42a0 msgr2=0x7fba580ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:16.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.753+0000 7fba5de91640 1 --2- 172.21.15.57:0/1405809709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba580c42a0 0x7fba580ca1e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fba50009d90 tx=0x7fba5001f130 comp rx=0 tx=0).stop 2023-10-30T14:33:16.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.753+0000 7fba5de91640 1 -- 172.21.15.57:0/1405809709 shutdown_connections 2023-10-30T14:33:16.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.753+0000 7fba5de91640 1 --2- 172.21.15.57:0/1405809709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba580c42a0 0x7fba580ca1e0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:16.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.753+0000 7fba5de91640 1 --2- 172.21.15.57:0/1405809709 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba58153c80 0x7fba5815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:16.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.753+0000 7fba5de91640 1 -- 172.21.15.57:0/1405809709 >> 172.21.15.57:0/1405809709 conn(0x7fba580715c0 msgr2=0x7fba580719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:16.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.753+0000 7fba5de91640 1 -- 172.21.15.57:0/1405809709 shutdown_connections 2023-10-30T14:33:16.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.753+0000 7fba5de91640 1 -- 172.21.15.57:0/1405809709 wait complete. 2023-10-30T14:33:16.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba5de91640 1 Processor -- start 2023-10-30T14:33:16.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba5de91640 1 -- start start 2023-10-30T14:33:16.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba5de91640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba580c42a0 0x7fba5815cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:16.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba5de91640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba580c4a10 0x7fba580cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:16.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba5de91640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba58153c80 0x7fba580cd580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:16.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba5de91640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba581434b0 con 0x7fba58153c80 2023-10-30T14:33:16.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba5de91640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba580cee40 con 0x7fba580c42a0 2023-10-30T14:33:16.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba5de91640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba581431d0 con 0x7fba580c4a10 2023-10-30T14:33:16.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba57fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba58153c80 0x7fba580cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:16.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba57fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba58153c80 0x7fba580cd580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37000/0 (socket says 172.21.15.57:37000) 2023-10-30T14:33:16.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba57fff640 1 -- 172.21.15.57:0/1722694986 learned_addr learned my addr 172.21.15.57:0/1722694986 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:33:16.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba56ffd640 1 --2- 172.21.15.57:0/1722694986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba580c4a10 0x7fba580cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:16.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba577fe640 1 --2- 172.21.15.57:0/1722694986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba580c42a0 0x7fba5815cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:16.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba57fff640 1 -- 172.21.15.57:0/1722694986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba580c4a10 msgr2=0x7fba580cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:16.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba57fff640 1 --2- 172.21.15.57:0/1722694986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba580c4a10 0x7fba580cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:16.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba57fff640 1 -- 172.21.15.57:0/1722694986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba580c42a0 msgr2=0x7fba5815cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:16.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba57fff640 1 --2- 172.21.15.57:0/1722694986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba580c42a0 0x7fba5815cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:16.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba57fff640 1 -- 172.21.15.57:0/1722694986 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba50009a70 con 0x7fba58153c80 2023-10-30T14:33:16.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba56ffd640 1 --2- 172.21.15.57:0/1722694986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba580c4a10 0x7fba580cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:33:16.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba577fe640 1 --2- 172.21.15.57:0/1722694986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba580c42a0 0x7fba5815cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:33:16.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba57fff640 1 --2- 172.21.15.57:0/1722694986 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba58153c80 0x7fba580cd580 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fba5001d7e0 tx=0x7fba5001bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:16.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba3ffff640 1 -- 172.21.15.57:0/1722694986 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba50009450 con 0x7fba58153c80 2023-10-30T14:33:16.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba3ffff640 1 -- 172.21.15.57:0/1722694986 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba500b9410 con 0x7fba58153c80 2023-10-30T14:33:16.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba3ffff640 1 -- 172.21.15.57:0/1722694986 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba500b97a0 con 0x7fba58153c80 2023-10-30T14:33:16.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba5de91640 1 -- 172.21.15.57:0/1722694986 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba5815bd10 con 0x7fba58153c80 2023-10-30T14:33:16.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.757+0000 7fba5de91640 1 -- 172.21.15.57:0/1722694986 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5815bff0 con 0x7fba58153c80 2023-10-30T14:33:16.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.761+0000 7fba5de91640 1 -- 172.21.15.57:0/1722694986 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba580cefb0 con 0x7fba58153c80 2023-10-30T14:33:16.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.761+0000 7fba3ffff640 1 -- 172.21.15.57:0/1722694986 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fba500b6050 con 0x7fba58153c80 2023-10-30T14:33:16.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.761+0000 7fba3ffff640 1 --2- 172.21.15.57:0/1722694986 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fba44066520 0x7fba440689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:16.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.761+0000 7fba3ffff640 1 -- 172.21.15.57:0/1722694986 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7fba501227c0 con 0x7fba58153c80 2023-10-30T14:33:16.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.761+0000 7fba577fe640 1 --2- 172.21.15.57:0/1722694986 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fba44066520 0x7fba440689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:16.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.761+0000 7fba577fe640 1 --2- 172.21.15.57:0/1722694986 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fba44066520 0x7fba440689e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fba48009b10 tx=0x7fba48008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:16.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.765+0000 7fba3ffff640 1 -- 172.21.15.57:0/1722694986 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fba500ef7f0 con 0x7fba58153c80 2023-10-30T14:33:16.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.957+0000 7fba5de91640 1 -- 172.21.15.57:0/1722694986 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba5804e170 con 0x7fba58153c80 2023-10-30T14:33:16.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.957+0000 7fba3ffff640 1 -- 172.21.15.57:0/1722694986 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5741 (secure 0 0 0) 0x7fba500f2e10 con 0x7fba58153c80 2023-10-30T14:33:16.960 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:33:16.960 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:33:12.968830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:33:16.960 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 114 2023-10-30T14:33:16.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.961+0000 7fba5de91640 1 -- 172.21.15.57:0/1722694986 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fba44066520 msgr2=0x7fba440689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:16.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.961+0000 7fba5de91640 1 --2- 172.21.15.57:0/1722694986 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fba44066520 0x7fba440689e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fba48009b10 tx=0x7fba48008040 comp rx=0 tx=0).stop 2023-10-30T14:33:16.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.961+0000 7fba5de91640 1 -- 172.21.15.57:0/1722694986 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba58153c80 msgr2=0x7fba580cd580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:16.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.961+0000 7fba5de91640 1 --2- 172.21.15.57:0/1722694986 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba58153c80 0x7fba580cd580 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fba5001d7e0 tx=0x7fba5001bed0 comp rx=0 tx=0).stop 2023-10-30T14:33:16.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.961+0000 7fba5de91640 1 -- 172.21.15.57:0/1722694986 shutdown_connections 2023-10-30T14:33:16.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.961+0000 7fba5de91640 1 --2- 172.21.15.57:0/1722694986 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fba44066520 0x7fba440689e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:16.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.961+0000 7fba5de91640 1 --2- 172.21.15.57:0/1722694986 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba58153c80 0x7fba580cd580 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:16.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.961+0000 7fba5de91640 1 --2- 172.21.15.57:0/1722694986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba580c4a10 0x7fba580cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:16.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.961+0000 7fba5de91640 1 --2- 172.21.15.57:0/1722694986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba580c42a0 0x7fba5815cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:16.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.961+0000 7fba5de91640 1 -- 172.21.15.57:0/1722694986 >> 172.21.15.57:0/1722694986 conn(0x7fba580715c0 msgr2=0x7fba580c9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:16.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.961+0000 7fba5de91640 1 -- 172.21.15.57:0/1722694986 shutdown_connections 2023-10-30T14:33:16.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:16.961+0000 7fba5de91640 1 -- 172.21.15.57:0/1722694986 wait complete. 2023-10-30T14:33:16.980 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:33:18.982 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:33:19.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4359e640 1 Processor -- start 2023-10-30T14:33:19.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4359e640 1 -- start start 2023-10-30T14:33:19.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4359e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fec440c62b0 0x7fec441597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:19.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4359e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec440cb180 0x7fec4406ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:19.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4359e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec4415a770 con 0x7fec44153b50 2023-10-30T14:33:19.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4359e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec44159cf0 con 0x7fec440c62b0 2023-10-30T14:33:19.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4359e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec4406aa30 con 0x7fec440cb180 2023-10-30T14:33:19.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4259c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fec44153b50 0x7fec440ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:51556/0 (socket says 172.21.15.57:51556) 2023-10-30T14:33:19.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4259c640 1 -- 172.21.15.57:0/541567725 learned_addr learned my addr 172.21.15.57:0/541567725 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:33:19.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec41d9b640 1 --2- 172.21.15.57:0/541567725 >> v2:172.21.15.88:3300/0 conn(0x7fec440c62b0 0x7fec441597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:19.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec42d9d640 1 --2- 172.21.15.57:0/541567725 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec440cb180 0x7fec4406ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:19.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4159a640 1 -- 172.21.15.57:0/541567725 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525545460 0 0) 0x7fec34001120 con 0x7fec44153b50 2023-10-30T14:33:19.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4159a640 1 -- 172.21.15.57:0/541567725 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec2c003530 con 0x7fec44153b50 2023-10-30T14:33:19.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4159a640 1 -- 172.21.15.57:0/541567725 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2853778458 0 0) 0x7fec2c003530 con 0x7fec44153b50 2023-10-30T14:33:19.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4159a640 1 -- 172.21.15.57:0/541567725 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec440cb180 msgr2=0x7fec4406ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:19.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4159a640 1 --2- 172.21.15.57:0/541567725 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec440cb180 0x7fec4406ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4159a640 1 -- 172.21.15.57:0/541567725 >> v2:172.21.15.88:3300/0 conn(0x7fec440c62b0 msgr2=0x7fec441597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:19.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4159a640 1 --2- 172.21.15.57:0/541567725 >> v2:172.21.15.88:3300/0 conn(0x7fec440c62b0 0x7fec441597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4159a640 1 -- 172.21.15.57:0/541567725 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec440c9d40 con 0x7fec44153b50 2023-10-30T14:33:19.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4159a640 1 -- 172.21.15.57:0/541567725 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fec34001810 con 0x7fec44153b50 2023-10-30T14:33:19.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec42d9d640 1 --2- 172.21.15.57:0/541567725 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec440cb180 0x7fec4406ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:33:19.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4159a640 1 -- 172.21.15.57:0/541567725 >> v1:172.21.15.57:6789/0 conn(0x7fec44153b50 legacy=0x7fec440ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:19.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4159a640 1 --2- 172.21.15.57:0/541567725 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec440cb180 0x7fec2c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:19.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4159a640 1 --2- 172.21.15.57:0/541567725 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec2c003ec0 0x7fec2c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:19.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4159a640 1 --2- 172.21.15.57:0/541567725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec2c004d10 0x7fec2c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:19.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4159a640 1 -- 172.21.15.57:0/541567725 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec44159cf0 con 0x7fec2c004d10 2023-10-30T14:33:19.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4159a640 1 -- 172.21.15.57:0/541567725 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4406aa30 con 0x7fec2c003ec0 2023-10-30T14:33:19.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4159a640 1 -- 172.21.15.57:0/541567725 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c00b990 con 0x7fec440cb180 2023-10-30T14:33:19.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec42d9d640 1 --2- 172.21.15.57:0/541567725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec2c004d10 0x7fec2c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:19.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec4259c640 1 --2- 172.21.15.57:0/541567725 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec440cb180 0x7fec2c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:19.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec41d9b640 1 --2- 172.21.15.57:0/541567725 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec2c003ec0 0x7fec2c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:19.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec42d9d640 1 -- 172.21.15.57:0/541567725 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec440cb180 msgr2=0x7fec2c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:19.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec42d9d640 1 --2- 172.21.15.57:0/541567725 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec440cb180 0x7fec2c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec42d9d640 1 -- 172.21.15.57:0/541567725 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec2c003ec0 msgr2=0x7fec2c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:19.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec42d9d640 1 --2- 172.21.15.57:0/541567725 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec2c003ec0 0x7fec2c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.133+0000 7fec42d9d640 1 -- 172.21.15.57:0/541567725 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3c009a70 con 0x7fec2c004d10 2023-10-30T14:33:19.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.137+0000 7fec42d9d640 1 --2- 172.21.15.57:0/541567725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec2c004d10 0x7fec2c005190 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00e380 tx=0x7fec3c01cdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=479ce8197e2173ed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:19.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.137+0000 7fec4159a640 1 -- 172.21.15.57:0/541567725 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3c0a5dd0 con 0x7fec2c004d10 2023-10-30T14:33:19.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.137+0000 7fec4159a640 1 -- 172.21.15.57:0/541567725 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3c00f040 con 0x7fec2c004d10 2023-10-30T14:33:19.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.137+0000 7fec4159a640 1 -- 172.21.15.57:0/541567725 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3c0ad640 con 0x7fec2c004d10 2023-10-30T14:33:19.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.137+0000 7fec4359e640 1 -- 172.21.15.57:0/541567725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec2c004d10 msgr2=0x7fec2c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:19.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.137+0000 7fec4359e640 1 --2- 172.21.15.57:0/541567725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec2c004d10 0x7fec2c005190 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00e380 tx=0x7fec3c01cdc0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.137+0000 7fec42d9d640 1 -- 172.21.15.57:0/541567725 reap_dead start 2023-10-30T14:33:19.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.141+0000 7fec4359e640 1 -- 172.21.15.57:0/541567725 shutdown_connections 2023-10-30T14:33:19.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.141+0000 7fec4359e640 1 -- 172.21.15.57:0/541567725 >> 172.21.15.57:0/541567725 conn(0x7fec44065a80 msgr2=0x7fec44065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:19.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.141+0000 7fec4359e640 1 -- 172.21.15.57:0/541567725 shutdown_connections 2023-10-30T14:33:19.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.141+0000 7fec4359e640 1 -- 172.21.15.57:0/541567725 wait complete. 2023-10-30T14:33:19.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.141+0000 7fec4359e640 1 Processor -- start 2023-10-30T14:33:19.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.141+0000 7fec4359e640 1 -- start start 2023-10-30T14:33:19.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.141+0000 7fec4359e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec2c004d10 0x7fec44154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:19.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.141+0000 7fec4359e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec440ca400 0x7fec441552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:19.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.141+0000 7fec4359e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec440c2380 0x7fec4414d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:19.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.141+0000 7fec4359e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec44156f50 con 0x7fec440ca400 2023-10-30T14:33:19.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.141+0000 7fec4359e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec44154910 con 0x7fec440c2380 2023-10-30T14:33:19.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.141+0000 7fec4359e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec44156990 con 0x7fec2c004d10 2023-10-30T14:33:19.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.141+0000 7fec41d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec440ca400 0x7fec441552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:19.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.141+0000 7fec41d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec440ca400 0x7fec441552e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37020/0 (socket says 172.21.15.57:37020) 2023-10-30T14:33:19.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.141+0000 7fec41d9b640 1 -- 172.21.15.57:0/1108503072 learned_addr learned my addr 172.21.15.57:0/1108503072 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:33:19.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.145+0000 7fec4259c640 1 --2- 172.21.15.57:0/1108503072 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec2c004d10 0x7fec44154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:19.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.145+0000 7fec42d9d640 1 --2- 172.21.15.57:0/1108503072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec440c2380 0x7fec4414d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:19.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.145+0000 7fec41d9b640 1 -- 172.21.15.57:0/1108503072 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec2c004d10 msgr2=0x7fec44154da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:19.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.145+0000 7fec41d9b640 1 --2- 172.21.15.57:0/1108503072 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec2c004d10 0x7fec44154da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.145+0000 7fec41d9b640 1 -- 172.21.15.57:0/1108503072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec440c2380 msgr2=0x7fec4414d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:19.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.145+0000 7fec41d9b640 1 --2- 172.21.15.57:0/1108503072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec440c2380 0x7fec4414d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.145+0000 7fec41d9b640 1 -- 172.21.15.57:0/1108503072 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec44156c70 con 0x7fec440ca400 2023-10-30T14:33:19.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.145+0000 7fec41d9b640 1 --2- 172.21.15.57:0/1108503072 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec440ca400 0x7fec441552e0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fec38006d50 tx=0x7fec38011d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:19.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.145+0000 7fec32ffd640 1 -- 172.21.15.57:0/1108503072 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3800fa30 con 0x7fec440ca400 2023-10-30T14:33:19.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.145+0000 7fec32ffd640 1 -- 172.21.15.57:0/1108503072 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3800fb90 con 0x7fec440ca400 2023-10-30T14:33:19.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.145+0000 7fec32ffd640 1 -- 172.21.15.57:0/1108503072 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec38004030 con 0x7fec440ca400 2023-10-30T14:33:19.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.145+0000 7fec4359e640 1 -- 172.21.15.57:0/1108503072 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec441542c0 con 0x7fec440ca400 2023-10-30T14:33:19.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.145+0000 7fec4359e640 1 -- 172.21.15.57:0/1108503072 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec4414d570 con 0x7fec440ca400 2023-10-30T14:33:19.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.145+0000 7fec4359e640 1 -- 172.21.15.57:0/1108503072 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec44154120 con 0x7fec440ca400 2023-10-30T14:33:19.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.145+0000 7fec32ffd640 1 -- 172.21.15.57:0/1108503072 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fec38004190 con 0x7fec440ca400 2023-10-30T14:33:19.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.145+0000 7fec32ffd640 1 --2- 172.21.15.57:0/1108503072 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fec240664a0 0x7fec24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:19.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.149+0000 7fec32ffd640 1 -- 172.21.15.57:0/1108503072 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7fec38084780 con 0x7fec440ca400 2023-10-30T14:33:19.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.149+0000 7fec4259c640 1 --2- 172.21.15.57:0/1108503072 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fec240664a0 0x7fec24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:19.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.149+0000 7fec4259c640 1 --2- 172.21.15.57:0/1108503072 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fec240664a0 0x7fec24068960 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fec34001280 tx=0x7fec34007a80 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:19.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.153+0000 7fec32ffd640 1 -- 172.21.15.57:0/1108503072 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fec44154120 con 0x7fec440ca400 2023-10-30T14:33:19.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.345+0000 7fec4359e640 1 -- 172.21.15.57:0/1108503072 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec44154120 con 0x7fec440ca400 2023-10-30T14:33:19.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.345+0000 7fec32ffd640 1 -- 172.21.15.57:0/1108503072 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5741 (secure 0 0 0) 0x7fec44154120 con 0x7fec440ca400 2023-10-30T14:33:19.347 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:33:19.347 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:33:18.087795+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:active","state_seq":40,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:33:19.348 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 115 2023-10-30T14:33:19.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.345+0000 7fec4359e640 1 -- 172.21.15.57:0/1108503072 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fec240664a0 msgr2=0x7fec24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:19.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.345+0000 7fec4359e640 1 --2- 172.21.15.57:0/1108503072 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fec240664a0 0x7fec24068960 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fec34001280 tx=0x7fec34007a80 comp rx=0 tx=0).stop 2023-10-30T14:33:19.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.345+0000 7fec4359e640 1 -- 172.21.15.57:0/1108503072 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec440ca400 msgr2=0x7fec441552e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:19.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.345+0000 7fec4359e640 1 --2- 172.21.15.57:0/1108503072 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec440ca400 0x7fec441552e0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fec38006d50 tx=0x7fec38011d20 comp rx=0 tx=0).stop 2023-10-30T14:33:19.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.345+0000 7fec4359e640 1 -- 172.21.15.57:0/1108503072 shutdown_connections 2023-10-30T14:33:19.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.345+0000 7fec4359e640 1 --2- 172.21.15.57:0/1108503072 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fec240664a0 0x7fec24068960 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.345+0000 7fec4359e640 1 --2- 172.21.15.57:0/1108503072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec440c2380 0x7fec4414d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.345+0000 7fec4359e640 1 --2- 172.21.15.57:0/1108503072 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec440ca400 0x7fec441552e0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.345+0000 7fec4359e640 1 --2- 172.21.15.57:0/1108503072 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec2c004d10 0x7fec44154da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.345+0000 7fec4359e640 1 -- 172.21.15.57:0/1108503072 >> 172.21.15.57:0/1108503072 conn(0x7fec44065a80 msgr2=0x7fec4414ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:19.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.345+0000 7fec4359e640 1 -- 172.21.15.57:0/1108503072 shutdown_connections 2023-10-30T14:33:19.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.345+0000 7fec4359e640 1 -- 172.21.15.57:0/1108503072 wait complete. 2023-10-30T14:33:19.367 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=1, replacing gid=6027 2023-10-30T14:33:19.368 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.c 2023-10-30T14:33:19.368 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T14:33:19.368 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T14:33:19.368 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T14:33:19.416 INFO:tasks.ceph.mds.c:Started 2023-10-30T14:33:19.417 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:33:19.507 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-30T14:33:19.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8d5f3e640 1 Processor -- start 2023-10-30T14:33:19.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8d5f3e640 1 -- start start 2023-10-30T14:33:19.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8d5f3e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff8d006bd10 0x7ff8d00c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:19.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8d5f3e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d00c5030 con 0x7ff8d0151c20 2023-10-30T14:33:19.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8d5f3e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d0157830 con 0x7ff8d0151840 2023-10-30T14:33:19.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8d5f3e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d0157b10 con 0x7ff8d006bd10 2023-10-30T14:33:19.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8ceffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff8d0151c20 0x7ff8d006b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:51568/0 (socket says 172.21.15.57:51568) 2023-10-30T14:33:19.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8ceffd640 1 -- 172.21.15.57:0/1627611304 learned_addr learned my addr 172.21.15.57:0/1627611304 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:33:19.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8cffff640 1 --2- 172.21.15.57:0/1627611304 >> v2:172.21.15.88:3300/0 conn(0x7ff8d006bd10 0x7ff8d00c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:19.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8ce7fc640 1 -- 172.21.15.57:0/1627611304 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396801784 0 0) 0x7ff8d00c5030 con 0x7ff8d0151c20 2023-10-30T14:33:19.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8ce7fc640 1 -- 172.21.15.57:0/1627611304 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8b8003530 con 0x7ff8d0151c20 2023-10-30T14:33:19.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8ce7fc640 1 -- 172.21.15.57:0/1627611304 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220960051 0 0) 0x7ff8d0157830 con 0x7ff8d0151840 2023-10-30T14:33:19.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8ce7fc640 1 -- 172.21.15.57:0/1627611304 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8d00c5030 con 0x7ff8d0151840 2023-10-30T14:33:19.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8ce7fc640 1 -- 172.21.15.57:0/1627611304 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1665754094 0 0) 0x7ff8b8003530 con 0x7ff8d0151c20 2023-10-30T14:33:19.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8ce7fc640 1 -- 172.21.15.57:0/1627611304 >> v2:172.21.15.88:3300/0 conn(0x7ff8d006bd10 msgr2=0x7ff8d00c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:19.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8ce7fc640 1 --2- 172.21.15.57:0/1627611304 >> v2:172.21.15.88:3300/0 conn(0x7ff8d006bd10 0x7ff8d00c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8ce7fc640 1 -- 172.21.15.57:0/1627611304 >> v1:172.21.15.88:6789/0 conn(0x7ff8d0151840 legacy=0x7ff8d0072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:19.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8ce7fc640 1 -- 172.21.15.57:0/1627611304 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d0157df0 con 0x7ff8d0151c20 2023-10-30T14:33:19.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8ce7fc640 1 -- 172.21.15.57:0/1627611304 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7ff8c4001680 con 0x7ff8d0151c20 2023-10-30T14:33:19.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8ce7fc640 1 -- 172.21.15.57:0/1627611304 >> v1:172.21.15.57:6789/0 conn(0x7ff8d0151c20 legacy=0x7ff8d006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:19.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8ce7fc640 1 --2- 172.21.15.57:0/1627611304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8b8003b60 0x7ff8b8003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:19.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8ce7fc640 1 --2- 172.21.15.57:0/1627611304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8b80047d0 0x7ff8b8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:19.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8ce7fc640 1 --2- 172.21.15.57:0/1627611304 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff8b8005190 0x7ff8b800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:19.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.565+0000 7ff8ce7fc640 1 -- 172.21.15.57:0/1627611304 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d0157b10 con 0x7ff8b80047d0 2023-10-30T14:33:19.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.569+0000 7ff8ce7fc640 1 -- 172.21.15.57:0/1627611304 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b800bfb0 con 0x7ff8b8003b60 2023-10-30T14:33:19.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.569+0000 7ff8ce7fc640 1 -- 172.21.15.57:0/1627611304 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b800c170 con 0x7ff8b8005190 2023-10-30T14:33:19.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.569+0000 7ff8ceffd640 1 --2- 172.21.15.57:0/1627611304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8b80047d0 0x7ff8b8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:19.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.569+0000 7ff8cffff640 1 --2- 172.21.15.57:0/1627611304 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff8b8005190 0x7ff8b800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:19.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.569+0000 7ff8cf7fe640 1 --2- 172.21.15.57:0/1627611304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8b8003b60 0x7ff8b8003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:19.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.569+0000 7ff8ceffd640 1 -- 172.21.15.57:0/1627611304 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff8b8005190 msgr2=0x7ff8b800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:19.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.569+0000 7ff8ceffd640 1 --2- 172.21.15.57:0/1627611304 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff8b8005190 0x7ff8b800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.569+0000 7ff8ceffd640 1 -- 172.21.15.57:0/1627611304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8b8003b60 msgr2=0x7ff8b8003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:19.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.569+0000 7ff8ceffd640 1 --2- 172.21.15.57:0/1627611304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8b8003b60 0x7ff8b8003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.569+0000 7ff8ceffd640 1 -- 172.21.15.57:0/1627611304 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b8003530 con 0x7ff8b80047d0 2023-10-30T14:33:19.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.569+0000 7ff8cf7fe640 1 --2- 172.21.15.57:0/1627611304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8b8003b60 0x7ff8b8003f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:33:19.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.569+0000 7ff8cffff640 1 -- 172.21.15.57:0/1627611304 reap_dead start 2023-10-30T14:33:19.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.569+0000 7ff8ceffd640 1 --2- 172.21.15.57:0/1627611304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8b80047d0 0x7ff8b8004c50 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4007fa0 tx=0x7ff8c401f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c0ad8d02816c2c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:19.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.569+0000 7ff8ce7fc640 1 -- 172.21.15.57:0/1627611304 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c40a8e30 con 0x7ff8b80047d0 2023-10-30T14:33:19.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.569+0000 7ff8ce7fc640 1 -- 172.21.15.57:0/1627611304 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c40058e0 con 0x7ff8b80047d0 2023-10-30T14:33:19.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.569+0000 7ff8ce7fc640 1 -- 172.21.15.57:0/1627611304 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4005cf0 con 0x7ff8b80047d0 2023-10-30T14:33:19.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8d5f3e640 1 -- 172.21.15.57:0/1627611304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8b80047d0 msgr2=0x7ff8b8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:19.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8d5f3e640 1 --2- 172.21.15.57:0/1627611304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8b80047d0 0x7ff8b8004c50 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4007fa0 tx=0x7ff8c401f210 comp rx=0 tx=0).stop 2023-10-30T14:33:19.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8d5f3e640 1 -- 172.21.15.57:0/1627611304 shutdown_connections 2023-10-30T14:33:19.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8d5f3e640 1 --2- 172.21.15.57:0/1627611304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8b80047d0 0x7ff8b8004c50 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8d5f3e640 1 -- 172.21.15.57:0/1627611304 >> 172.21.15.57:0/1627611304 conn(0x7ff8d00cac70 msgr2=0x7ff8d00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:19.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8d5f3e640 1 -- 172.21.15.57:0/1627611304 shutdown_connections 2023-10-30T14:33:19.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8d5f3e640 1 -- 172.21.15.57:0/1627611304 wait complete. 2023-10-30T14:33:19.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8d5f3e640 1 Processor -- start 2023-10-30T14:33:19.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8d5f3e640 1 -- start start 2023-10-30T14:33:19.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8d5f3e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8b80047d0 0x7ff8d00705d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:19.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8d5f3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8d00718a0 0x7ff8d01f1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:19.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8d5f3e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff8d01df440 0x7ff8d00ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:19.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8d5f3e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d0070140 con 0x7ff8b80047d0 2023-10-30T14:33:19.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8d5f3e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d014e640 con 0x7ff8d00718a0 2023-10-30T14:33:19.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8d5f3e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d01f24a0 con 0x7ff8d01df440 2023-10-30T14:33:19.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8cf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8b80047d0 0x7ff8d00705d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:19.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8cf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8b80047d0 0x7ff8d00705d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37058/0 (socket says 172.21.15.57:37058) 2023-10-30T14:33:19.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8cf7fe640 1 -- 172.21.15.57:0/2649063343 learned_addr learned my addr 172.21.15.57:0/2649063343 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:33:19.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8ceffd640 1 --2- 172.21.15.57:0/2649063343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8d00718a0 0x7ff8d01f1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:19.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8cffff640 1 --2- 172.21.15.57:0/2649063343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff8d01df440 0x7ff8d00ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:19.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8cf7fe640 1 -- 172.21.15.57:0/2649063343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff8d01df440 msgr2=0x7ff8d00ca2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:19.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8cf7fe640 1 --2- 172.21.15.57:0/2649063343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff8d01df440 0x7ff8d00ca2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8cf7fe640 1 -- 172.21.15.57:0/2649063343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8d00718a0 msgr2=0x7ff8d01f1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:19.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8cf7fe640 1 --2- 172.21.15.57:0/2649063343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8d00718a0 0x7ff8d01f1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.573+0000 7ff8cf7fe640 1 -- 172.21.15.57:0/2649063343 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d006fdf0 con 0x7ff8b80047d0 2023-10-30T14:33:19.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.577+0000 7ff8cf7fe640 1 --2- 172.21.15.57:0/2649063343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8b80047d0 0x7ff8d00705d0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000e560 tx=0x7ff8c000ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:19.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.577+0000 7ff8b7fff640 1 -- 172.21.15.57:0/2649063343 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0017070 con 0x7ff8b80047d0 2023-10-30T14:33:19.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.577+0000 7ff8b7fff640 1 -- 172.21.15.57:0/2649063343 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c00052e0 con 0x7ff8b80047d0 2023-10-30T14:33:19.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.577+0000 7ff8d5f3e640 1 -- 172.21.15.57:0/2649063343 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d0071ff0 con 0x7ff8b80047d0 2023-10-30T14:33:19.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.577+0000 7ff8b7fff640 1 -- 172.21.15.57:0/2649063343 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c00054b0 con 0x7ff8b80047d0 2023-10-30T14:33:19.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.577+0000 7ff8d5f3e640 1 -- 172.21.15.57:0/2649063343 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d00725b0 con 0x7ff8b80047d0 2023-10-30T14:33:19.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.577+0000 7ff8d5f3e640 1 -- 172.21.15.57:0/2649063343 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d006ff80 con 0x7ff8b80047d0 2023-10-30T14:33:19.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.577+0000 7ff8b7fff640 1 -- 172.21.15.57:0/2649063343 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff8c0007020 con 0x7ff8b80047d0 2023-10-30T14:33:19.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.577+0000 7ff8b7fff640 1 --2- 172.21.15.57:0/2649063343 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff8bc0664d0 0x7ff8bc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:19.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.577+0000 7ff8b7fff640 1 -- 172.21.15.57:0/2649063343 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7ff8c0083b40 con 0x7ff8b80047d0 2023-10-30T14:33:19.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.577+0000 7ff8ceffd640 1 --2- 172.21.15.57:0/2649063343 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff8bc0664d0 0x7ff8bc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:19.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.581+0000 7ff8ceffd640 1 --2- 172.21.15.57:0/2649063343 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff8bc0664d0 0x7ff8bc068990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40ae000 tx=0x7ff8c40aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:19.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.581+0000 7ff8b7fff640 1 -- 172.21.15.57:0/2649063343 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff8d006ff80 con 0x7ff8b80047d0 2023-10-30T14:33:19.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.777+0000 7ff8d5f3e640 1 -- 172.21.15.57:0/2649063343 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8d006ff80 con 0x7ff8b80047d0 2023-10-30T14:33:19.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.777+0000 7ff8b7fff640 1 -- 172.21.15.57:0/2649063343 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5741 (secure 0 0 0) 0x7ff8d006ff80 con 0x7ff8b80047d0 2023-10-30T14:33:19.781 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:33:19.781 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:33:18.087795+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:active","state_seq":40,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:33:19.782 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 115 2023-10-30T14:33:19.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.781+0000 7ff8d5f3e640 1 -- 172.21.15.57:0/2649063343 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff8bc0664d0 msgr2=0x7ff8bc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:19.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.781+0000 7ff8d5f3e640 1 --2- 172.21.15.57:0/2649063343 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff8bc0664d0 0x7ff8bc068990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40ae000 tx=0x7ff8c40aa070 comp rx=0 tx=0).stop 2023-10-30T14:33:19.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.781+0000 7ff8d5f3e640 1 -- 172.21.15.57:0/2649063343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8b80047d0 msgr2=0x7ff8d00705d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:19.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.781+0000 7ff8d5f3e640 1 --2- 172.21.15.57:0/2649063343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8b80047d0 0x7ff8d00705d0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000e560 tx=0x7ff8c000ea20 comp rx=0 tx=0).stop 2023-10-30T14:33:19.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.781+0000 7ff8d5f3e640 1 -- 172.21.15.57:0/2649063343 shutdown_connections 2023-10-30T14:33:19.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.781+0000 7ff8d5f3e640 1 --2- 172.21.15.57:0/2649063343 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff8bc0664d0 0x7ff8bc068990 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.781+0000 7ff8d5f3e640 1 --2- 172.21.15.57:0/2649063343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff8d01df440 0x7ff8d00ca2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.781+0000 7ff8d5f3e640 1 --2- 172.21.15.57:0/2649063343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8d00718a0 0x7ff8d01f1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.781+0000 7ff8d5f3e640 1 --2- 172.21.15.57:0/2649063343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8b80047d0 0x7ff8d00705d0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:19.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.781+0000 7ff8d5f3e640 1 -- 172.21.15.57:0/2649063343 >> 172.21.15.57:0/2649063343 conn(0x7ff8d00cac70 msgr2=0x7ff8d01552b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:19.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.781+0000 7ff8d5f3e640 1 -- 172.21.15.57:0/2649063343 shutdown_connections 2023-10-30T14:33:19.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:19.781+0000 7ff8d5f3e640 1 -- 172.21.15.57:0/2649063343 wait complete. 2023-10-30T14:33:19.802 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T14:33:21.804 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:33:21.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.969+0000 7f6f097d9640 1 Processor -- start 2023-10-30T14:33:21.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.969+0000 7f6f097d9640 1 -- start start 2023-10-30T14:33:21.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.969+0000 7f6f097d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6f04153c80 0x7f6f0415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:21.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.969+0000 7f6f097d9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6f040c4a10 0x7f6f040c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:21.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.969+0000 7f6f097d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6f0415a3b0 0x7f6f040ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:21.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.969+0000 7f6f097d9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f0415dec0 con 0x7f6f040c4a10 2023-10-30T14:33:21.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.969+0000 7f6f097d9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f041599a0 con 0x7f6f0415a3b0 2023-10-30T14:33:21.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.969+0000 7f6f097d9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f04159f20 con 0x7f6f04153c80 2023-10-30T14:33:21.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.969+0000 7f6efa7fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6f040c4a10 0x7f6f040c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:21.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.969+0000 7f6efa7fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6f040c4a10 0x7f6f040c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37072/0 (socket says 172.21.15.57:37072) 2023-10-30T14:33:21.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.969+0000 7f6efa7fc640 1 -- 172.21.15.57:0/406274365 learned_addr learned my addr 172.21.15.57:0/406274365 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:33:21.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.969+0000 7f6f02ffd640 1 --2- 172.21.15.57:0/406274365 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6f04153c80 0x7f6f0415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:21.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.969+0000 7f6f037fe640 1 --2- 172.21.15.57:0/406274365 >> v2:172.21.15.88:3300/0 conn(0x7f6f0415a3b0 0x7f6f040ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:21.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.969+0000 7f6efa7fc640 1 -- 172.21.15.57:0/406274365 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6f04153c80 msgr2=0x7f6f0415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:21.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.969+0000 7f6efa7fc640 1 --2- 172.21.15.57:0/406274365 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6f04153c80 0x7f6f0415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:21.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.969+0000 7f6efa7fc640 1 -- 172.21.15.57:0/406274365 >> v2:172.21.15.88:3300/0 conn(0x7f6f0415a3b0 msgr2=0x7f6f040ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:33:21.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.969+0000 7f6efa7fc640 1 --2- 172.21.15.57:0/406274365 >> v2:172.21.15.88:3300/0 conn(0x7f6f0415a3b0 0x7f6f040ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:21.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.969+0000 7f6efa7fc640 1 -- 172.21.15.57:0/406274365 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f040c47e0 con 0x7f6f040c4a10 2023-10-30T14:33:21.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.969+0000 7f6f037fe640 1 --2- 172.21.15.57:0/406274365 >> v2:172.21.15.88:3300/0 conn(0x7f6f0415a3b0 0x7f6f040ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:33:21.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.973+0000 7f6efa7fc640 1 --2- 172.21.15.57:0/406274365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f040c4a10 0x7f6f040c42a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f6f040cbbc0 tx=0x7f6eec01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a51d2d1b2886915c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:21.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.973+0000 7f6f027fc640 1 -- 172.21.15.57:0/406274365 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec0a89b0 con 0x7f6f040c4a10 2023-10-30T14:33:21.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.973+0000 7f6f027fc640 1 -- 172.21.15.57:0/406274365 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eec0a8b10 con 0x7f6f040c4a10 2023-10-30T14:33:21.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.973+0000 7f6f027fc640 1 -- 172.21.15.57:0/406274365 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec0a8ea0 con 0x7f6f040c4a10 2023-10-30T14:33:21.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.973+0000 7f6f097d9640 1 -- 172.21.15.57:0/406274365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f040c4a10 msgr2=0x7f6f040c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:21.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.973+0000 7f6f097d9640 1 --2- 172.21.15.57:0/406274365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f040c4a10 0x7f6f040c42a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f6f040cbbc0 tx=0x7f6eec01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:33:21.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.973+0000 7f6f097d9640 1 -- 172.21.15.57:0/406274365 shutdown_connections 2023-10-30T14:33:21.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.973+0000 7f6f097d9640 1 --2- 172.21.15.57:0/406274365 >> v2:172.21.15.88:3300/0 conn(0x7f6f0415a3b0 0x7f6f040ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:21.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.973+0000 7f6f097d9640 1 --2- 172.21.15.57:0/406274365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f040c4a10 0x7f6f040c42a0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:21.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.973+0000 7f6f097d9640 1 --2- 172.21.15.57:0/406274365 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6f04153c80 0x7f6f0415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:21.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.973+0000 7f6f097d9640 1 -- 172.21.15.57:0/406274365 >> 172.21.15.57:0/406274365 conn(0x7f6f040715c0 msgr2=0x7f6f040719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:21.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.973+0000 7f6f097d9640 1 -- 172.21.15.57:0/406274365 shutdown_connections 2023-10-30T14:33:21.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.973+0000 7f6f097d9640 1 -- 172.21.15.57:0/406274365 wait complete. 2023-10-30T14:33:21.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6f097d9640 1 Processor -- start 2023-10-30T14:33:21.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6f097d9640 1 -- start start 2023-10-30T14:33:21.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6f097d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6f040c4a10 0x7f6f041f4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:21.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6f097d9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f04153c80 0x7f6f040b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:21.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6f097d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f0415a3b0 0x7f6f041f3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:21.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6f097d9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f040b28d0 con 0x7f6f04153c80 2023-10-30T14:33:21.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6f097d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f040b44a0 con 0x7f6f0415a3b0 2023-10-30T14:33:21.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6f097d9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f041f3f90 con 0x7f6f040c4a10 2023-10-30T14:33:21.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6efa7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f04153c80 0x7f6f040b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:21.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6efa7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f04153c80 0x7f6f040b3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37084/0 (socket says 172.21.15.57:37084) 2023-10-30T14:33:21.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6efa7fc640 1 -- 172.21.15.57:0/1427989942 learned_addr learned my addr 172.21.15.57:0/1427989942 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:33:21.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6f037fe640 1 --2- 172.21.15.57:0/1427989942 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f0415a3b0 0x7f6f041f3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:21.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6f02ffd640 1 --2- 172.21.15.57:0/1427989942 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6f040c4a10 0x7f6f041f4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:21.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6efa7fc640 1 -- 172.21.15.57:0/1427989942 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6f040c4a10 msgr2=0x7f6f041f4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:21.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6efa7fc640 1 --2- 172.21.15.57:0/1427989942 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6f040c4a10 0x7f6f041f4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:21.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6efa7fc640 1 -- 172.21.15.57:0/1427989942 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f0415a3b0 msgr2=0x7f6f041f3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:21.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6efa7fc640 1 --2- 172.21.15.57:0/1427989942 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f0415a3b0 0x7f6f041f3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:21.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6efa7fc640 1 -- 172.21.15.57:0/1427989942 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eec0094d0 con 0x7f6f04153c80 2023-10-30T14:33:21.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6efa7fc640 1 --2- 172.21.15.57:0/1427989942 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f04153c80 0x7f6f040b3f60 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f6eec01d3a0 tx=0x7f6eec01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:21.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6efbfff640 1 -- 172.21.15.57:0/1427989942 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec0ba600 con 0x7f6f04153c80 2023-10-30T14:33:21.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6efbfff640 1 -- 172.21.15.57:0/1427989942 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eec0ba760 con 0x7f6f04153c80 2023-10-30T14:33:21.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6efbfff640 1 -- 172.21.15.57:0/1427989942 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec0bab10 con 0x7f6f04153c80 2023-10-30T14:33:21.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6f097d9640 1 -- 172.21.15.57:0/1427989942 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f04156ea0 con 0x7f6f04153c80 2023-10-30T14:33:21.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.977+0000 7f6f097d9640 1 -- 172.21.15.57:0/1427989942 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f040b25f0 con 0x7f6f04153c80 2023-10-30T14:33:21.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.981+0000 7f6f097d9640 1 -- 172.21.15.57:0/1427989942 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f0404e170 con 0x7f6f04153c80 2023-10-30T14:33:21.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.981+0000 7f6efbfff640 1 -- 172.21.15.57:0/1427989942 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6eec0bac70 con 0x7f6f04153c80 2023-10-30T14:33:21.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.981+0000 7f6efbfff640 1 --2- 172.21.15.57:0/1427989942 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ee40664a0 0x7f6ee4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:33:21.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.981+0000 7f6efbfff640 1 -- 172.21.15.57:0/1427989942 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f6eec1229d0 con 0x7f6f04153c80 2023-10-30T14:33:21.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.981+0000 7f6f02ffd640 1 --2- 172.21.15.57:0/1427989942 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ee40664a0 0x7f6ee4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:33:21.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.981+0000 7f6f02ffd640 1 --2- 172.21.15.57:0/1427989942 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ee40664a0 0x7f6ee4068960 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6ef400ba30 tx=0x7f6ef4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:33:21.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:21.985+0000 7f6efbfff640 1 -- 172.21.15.57:0/1427989942 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6eec0efa10 con 0x7f6f04153c80 2023-10-30T14:33:22.016 INFO:tasks.ceph.mds.a.smithi057.stderr:2023-10-30T14:33:22.013+0000 7f5ba0392640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:33:22.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:22.181+0000 7f6f097d9640 1 -- 172.21.15.57:0/1427989942 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f041575f0 con 0x7f6f04153c80 2023-10-30T14:33:22.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:22.181+0000 7f6efbfff640 1 -- 172.21.15.57:0/1427989942 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+6530 (secure 0 0 0) 0x7f6eec0b5090 con 0x7f6f04153c80 2023-10-30T14:33:22.183 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:33:22.183 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6747,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:33:18.087795+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:active","state_seq":40,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:33:22.183 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 116 2023-10-30T14:33:22.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:22.181+0000 7f6f097d9640 1 -- 172.21.15.57:0/1427989942 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ee40664a0 msgr2=0x7f6ee4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:22.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:22.181+0000 7f6f097d9640 1 --2- 172.21.15.57:0/1427989942 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ee40664a0 0x7f6ee4068960 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6ef400ba30 tx=0x7f6ef4008040 comp rx=0 tx=0).stop 2023-10-30T14:33:22.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:22.181+0000 7f6f097d9640 1 -- 172.21.15.57:0/1427989942 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f04153c80 msgr2=0x7f6f040b3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:33:22.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:22.181+0000 7f6f097d9640 1 --2- 172.21.15.57:0/1427989942 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f04153c80 0x7f6f040b3f60 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f6eec01d3a0 tx=0x7f6eec01ba90 comp rx=0 tx=0).stop 2023-10-30T14:33:22.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:22.181+0000 7f6f097d9640 1 -- 172.21.15.57:0/1427989942 shutdown_connections 2023-10-30T14:33:22.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:22.181+0000 7f6f097d9640 1 --2- 172.21.15.57:0/1427989942 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ee40664a0 0x7f6ee4068960 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:22.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:22.181+0000 7f6f097d9640 1 --2- 172.21.15.57:0/1427989942 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f0415a3b0 0x7f6f041f3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:22.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:22.181+0000 7f6f097d9640 1 --2- 172.21.15.57:0/1427989942 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f04153c80 0x7f6f040b3f60 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:22.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:22.181+0000 7f6f097d9640 1 --2- 172.21.15.57:0/1427989942 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6f040c4a10 0x7f6f041f4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:33:22.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:22.181+0000 7f6f097d9640 1 -- 172.21.15.57:0/1427989942 >> 172.21.15.57:0/1427989942 conn(0x7f6f040715c0 msgr2=0x7f6f040c9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:33:22.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:22.181+0000 7f6f097d9640 1 -- 172.21.15.57:0/1427989942 shutdown_connections 2023-10-30T14:33:22.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:33:22.181+0000 7f6f097d9640 1 -- 172.21.15.57:0/1427989942 wait complete. 2023-10-30T14:33:22.204 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T14:33:22.204 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 111 secs before thrashing 2023-10-30T14:33:36.696 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:33:36.700 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:34:06.833 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:34:06.838 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:34:34.481 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 1% (139/12861) Updating files: 1% (174/12861) Updating files: 1% (246/12861) Updating files: 2% (258/12861) Updating files: 3% (386/12861) Updating files: 3% (398/12861) Updating files: 3% (430/12861) Updating files: 4% (515/12861) Updating files: 4% (579/12861) Updating files: 4% (627/12861) Updating files: 5% (644/12861) Updating files: 5% (683/12861) Updating files: 5% (761/12861) Updating files: 5% (762/12861) Updating files: 5% (764/12861) Updating files: 6% (772/12861) Updating files: 6% (818/12861) Updating files: 6% (878/12861) Updating files: 7% (901/12861) Updating files: 7% (931/12861) Updating files: 7% (941/12861) Updating files: 7% (970/12861) Updating files: 7% (987/12861) Updating files: 8% (1029/12861) Updating files: 8% (1070/12861) Updating files: 8% (1142/12861) Updating files: 9% (1158/12861) Updating files: 9% (1218/12861) Updating files: 10% (1287/12861) Updating files: 10% (1320/12861) Updating files: 10% (1399/12861) Updating files: 11% (1415/12861) Updating files: 11% (1436/12861) Updating files: 11% (1493/12861) Updating files: 11% (1507/12861) Updating files: 11% (1512/12861) Updating files: 12% (1544/12861) Updating files: 12% (1595/12861) Updating files: 12% (1625/12861) Updating files: 12% (1633/12861) Updating files: 13% (1672/12861) Updating files: 13% (1680/12861) Updating files: 13% (1725/12861) Updating files: 13% (1751/12861) Updating files: 13% (1793/12861) Updating files: 14% (1801/12861) Updating files: 14% (1873/12861) Updating files: 14% (1880/12861) Updating files: 14% (1893/12861) Updating files: 14% (1920/12861) Updating files: 15% (1930/12861) Updating files: 15% (1967/12861) Updating files: 16% (2058/12861) Updating files: 16% (2076/12861) Updating files: 16% (2077/12861) Updating files: 16% (2086/12861) Updating files: 16% (2146/12861) Updating files: 16% (2185/12861) Updating files: 17% (2187/12861) Updating files: 17% (2213/12861) Updating files: 17% (2295/12861) Updating files: 17% (2310/12861) Updating files: 18% (2315/12861) Updating files: 18% (2411/12861) Updating files: 19% (2444/12861) Updating files: 19% (2468/12861) Updating files: 19% (2504/12861) Updating files: 20% (2573/12861) Updating files: 20% (2585/12861) Updating files: 20% (2672/12861) Updating files: 20% (2680/12861) Updating files: 21% (2701/12861) Updating files: 21% (2720/12861) Updating files: 21% (2750/12861) Updating files: 21% (2803/12861) Updating files: 22% (2830/12861) Updating files: 22% (2911/12861) Updating files: 23% (2959/12861) Updating files: 23% (3029/12861) Updating files: 23% (3041/12861) Updating files: 24% (3087/12861) Updating files: 24% (3141/12861) Updating files: 24% (3153/12861) Updating files: 25% (3216/12861) Updating files: 25% (3229/12861) Updating files: 25% (3278/12861) Updating files: 25% (3323/12861) Updating files: 26% (3344/12861) Updating files: 26% (3366/12861) Updating files: 26% (3388/12861) Updating files: 26% (3423/12861) Updating files: 27% (3473/12861) Updating files: 27% (3513/12861) Updating files: 27% (3565/12861) Updating files: 27% (3594/12861) Updating files: 28% (3602/12861) Updating files: 28% (3678/12861) Updating files: 28% (3721/12861) Updating files: 29% (3730/12861) Updating files: 29% (3794/12861) Updating files: 29% (3819/12861) Updating files: 29% (3829/12861) Updating files: 30% (3859/12861) Updating files: 30% (3903/12861) Updating files: 30% (3915/12861) Updating files: 30% (3924/12861) Updating files: 30% (3966/12861) Updating files: 31% (3987/12861) Updating files: 31% (4054/12861) Updating files: 31% (4083/12861) Updating files: 31% (4098/12861) Updating files: 31% (4101/12861) Updating files: 32% (4116/12861) Updating files: 32% (4148/12861) Updating files: 32% (4168/12861) Updating files: 33% (4245/12861) Updating files: 33% (4257/12861) Updating files: 33% (4271/12861) Updating files: 33% (4287/12861) Updating files: 33% (4333/12861) Updating files: 34% (4373/12861) Updating files: 34% (4402/12861) Updating files: 34% (4486/12861) Updating files: 35% (4502/12861) Updating files: 35% (4525/12861) Updating files: 35% (4535/12861) Updating files: 35% (4555/12861) Updating files: 36% (4630/12861) Updating files: 36% (4637/12861) Updating files: 36% (4646/12861) Updating files: 36% (4752/12861) Updating files: 37% (4759/12861) Updating files: 38% (4888/12861) Updating files: 38% (4936/12861) Updating files: 38% (4939/12861) Updating files: 39% (5016/12861) Updating files: 39% (5058/12861) Updating files: 39% (5063/12861) Updating files: 39% (5127/12861) Updating files: 39% (5135/12861) Updating files: 40% (5145/12861) Updating files: 40% (5188/12861) Updating files: 40% (5201/12861) Updating files: 40% (5249/12861) Updating files: 41% (5274/12861) Updating files: 41% (5346/12861) Updating files: 42% (5402/12861) Updating files: 42% (5449/12861) Updating files: 42% (5529/12861) Updating files: 43% (5531/12861) Updating files: 43% (5616/12861) Updating files: 43% (5651/12861) Updating files: 43% (5652/12861) Updating files: 44% (5659/12861) Updating files: 44% (5667/12861) Updating files: 44% (5738/12861) Updating files: 44% (5747/12861) Updating files: 45% (5788/12861) Updating files: 45% (5793/12861) Updating files: 45% (5864/12861) Updating files: 46% (5917/12861) Updating files: 46% (5924/12861) Updating files: 47% (6045/12861) Updating files: 47% (6155/12861) Updating files: 47% (6162/12861) Updating files: 48% (6174/12861) Updating files: 48% (6221/12861) Updating files: 48% (6236/12861) Updating files: 48% (6265/12861) Updating files: 49% (6302/12861) Updating files: 49% (6310/12861) Updating files: 49% (6316/12861) Updating files: 49% (6426/12861) Updating files: 50% (6431/12861) Updating files: 50% (6495/12861) Updating files: 50% (6511/12861) Updating files: 51% (6560/12861) Updating files: 51% (6669/12861) Updating files: 51% (6684/12861) Updating files: 52% (6688/12861) Updating files: 52% (6745/12861) Updating files: 52% (6777/12861) Updating files: 52% (6810/12861) Updating files: 53% (6817/12861) Updating files: 53% (6843/12861) Updating files: 53% (6943/12861) Updating files: 54% (6945/12861) Updating files: 54% (6981/12861) Updating files: 54% (7012/12861) Updating files: 55% (7074/12861) Updating files: 55% (7130/12861) Updating files: 55% (7174/12861) Updating files: 56% (7203/12861) Updating files: 56% (7255/12861) Updating files: 57% (7331/12861) Updating files: 58% (7460/12861) Updating files: 58% (7469/12861) Updating files: 58% (7482/12861) Updating files: 58% (7487/12861) Updating files: 58% (7540/12861) Updating files: 59% (7588/12861) Updating files: 59% (7700/12861) Updating files: 59% (7703/12861) Updating files: 60% (7717/12861) Updating files: 61% (7846/12861) Updating files: 61% (7902/12861) Updating files: 61% (7970/12861) Updating files: 62% (7974/12861) Updating files: 62% (8039/12861) Updating files: 62% (8064/12861) Updating files: 63% (8103/12861) Updating files: 63% (8126/12861) Updating files: 64% (8232/12861) Updating files: 64% (8272/12861) Updating files: 64% (8294/12861) Updating files: 64% (8302/12861) Updating files: 64% (8326/12861) Updating files: 65% (8360/12861) Updating files: 65% (8460/12861) Updating files: 65% (8464/12861) Updating files: 65% (8477/12861) Updating files: 66% (8489/12861) Updating files: 66% (8529/12861) Updating files: 67% (8617/12861) Updating files: 67% (8636/12861) Updating files: 67% (8683/12861) Updating files: 67% (8728/12861) Updating files: 68% (8746/12861) Updating files: 68% (8819/12861) Updating files: 69% (8875/12861) Updating files: 69% (8886/12861) Updating files: 69% (8904/12861) Updating files: 69% (8964/12861) Updating files: 69% (8977/12861) Updating files: 70% (9003/12861) Updating files: 70% (9091/12861) Updating files: 70% (9130/12861) Updating files: 71% (9132/12861) Updating files: 71% (9203/12861) Updating files: 71% (9244/12861) Updating files: 72% (9260/12861) Updating files: 73% (9389/12861) Updating files: 73% (9420/12861) Updating files: 73% (9441/12861) Updating files: 73% (9509/12861) Updating files: 74% (9518/12861) Updating files: 74% (9545/12861) Updating files: 74% (9613/12861) Updating files: 74% (9614/12861) Updating files: 74% (9639/12861) Updating files: 75% (9646/12861) Updating files: 75% (9647/12861) Updating files: 75% (9682/12861) Updating files: 75% (9763/12861) Updating files: 76% (9775/12861) Updating files: 76% (9776/12861) Updating files: 76% (9859/12861) Updating files: 76% (9900/12861) Updating files: 77% (9903/12861) Updating files: 78% (10032/12861) Updating files: 78% (10101/12861) Updating files: 79% (10161/12861) Updating files: 79% (10198/12861) Updating files: 79% (10204/12861) Updating files: 79% (10221/12861) Updating files: 79% (10264/12861) Updating files: 80% (10289/12861) Updating files: 80% (10314/12861) Updating files: 80% (10345/12861) Updating files: 81% (10418/12861) Updating files: 81% (10432/12861) Updating files: 81% (10481/12861) Updating files: 82% (10547/12861) Updating files: 82% (10567/12861) Updating files: 82% (10608/12861) Updating files: 82% (10673/12861) Updating files: 83% (10675/12861) Updating files: 83% (10676/12861) Updating files: 83% (10683/12861) Updating files: 83% (10775/12861) Updating files: 84% (10804/12861) Updating files: 84% (10845/12861) Updating files: 84% (10928/12861) Updating files: 85% (10932/12861) Updating files: 85% (10963/12861) Updating files: 85% (11005/12861) Updating files: 85% (11038/12861) Updating files: 85% (11051/12861) Updating files: 86% (11061/12861) Updating files: 86% (11064/12861) Updating files: 86% (11065/12861) Updating files: 86% (11112/12861) Updating files: 86% (11174/12861) Updating files: 87% (11190/12861) Updating files: 87% (11221/12861) Updating files: 87% (11237/12861) Updating files: 87% (11287/12861) Updating files: 88% (11318/12861) Updating files: 88% (11353/12861) Updating files: 88% (11358/12861) Updating files: 88% (11397/12861) Updating files: 88% (11441/12861) Updating files: 89% (11447/12861) Updating files: 89% (11556/12861) Updating files: 90% (11575/12861) Updating files: 90% (11576/12861) Updating files: 90% (11677/12861) Updating files: 91% (11704/12861) Updating files: 91% (11755/12861) Updating files: 91% (11783/12861) Updating files: 92% (11833/12861) Updating files: 92% (11857/12861) Updating files: 93% (11961/12861) Updating files: 93% (12037/12861) Updating files: 94% (12090/12861) Updating files: 95% (12218/12861) Updating files: 95% (12263/12861) Updating files: 95% (12295/12861) Updating files: 96% (12347/12861) Updating files: 96% (12401/12861) Updating files: 97% (12476/12861) Updating files: 97% (12558/12861) Updating files: 97% (12589/12861) Updating files: 98% (12604/12861) Updating files: 98% (12647/12861) Updating files: 98% (12683/12861) Updating files: 99% (12733/12861) Updating files: 99% (12749/12861) Updating files: 99% (12774/12861) Updating files: 100% (12861/12861) Updating files: 100% (12861/12861), done. 2023-10-30T14:34:34.565 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 204c175eea3 msg: blobhash on entity_name_t causes strange badness... should look into that, but avoid for now 2023-10-30T14:34:34.573 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.1 2023-10-30T14:34:34.859 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 2 -eq 48 ] 2023-10-30T14:34:34.859 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.2 2023-10-30T14:34:34.859 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.2 2023-10-30T14:34:34.860 INFO:tasks.workunit.client.0.smithi057.stdout:v0.2 2023-10-30T14:34:34.860 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.2 2023-10-30T14:34:36.926 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 26% (59/222) Updating files: 27% (60/222) Updating files: 28% (63/222) Updating files: 29% (65/222) Updating files: 30% (67/222) Updating files: 31% (69/222) Updating files: 32% (72/222) Updating files: 33% (74/222) Updating files: 34% (76/222) Updating files: 35% (78/222) Updating files: 36% (80/222) Updating files: 37% (83/222) Updating files: 38% (85/222) Updating files: 39% (87/222) Updating files: 40% (89/222) Updating files: 41% (92/222) Updating files: 42% (94/222) Updating files: 43% (96/222) Updating files: 44% (98/222) Updating files: 45% (100/222) Updating files: 46% (103/222) Updating files: 47% (105/222) Updating files: 48% (107/222) Updating files: 49% (109/222) Updating files: 50% (111/222) Updating files: 51% (114/222) Updating files: 52% (116/222) Updating files: 53% (118/222) Updating files: 54% (120/222) Updating files: 55% (123/222) Updating files: 56% (125/222) Updating files: 57% (127/222) Updating files: 58% (129/222) Updating files: 59% (131/222) Updating files: 60% (134/222) Updating files: 61% (136/222) Updating files: 62% (138/222) Updating files: 63% (140/222) Updating files: 64% (143/222) Updating files: 65% (145/222) Updating files: 66% (147/222) Updating files: 67% (149/222) Updating files: 68% (151/222) Updating files: 69% (154/222) Updating files: 70% (156/222) Updating files: 71% (158/222) Updating files: 72% (160/222) Updating files: 73% (163/222) Updating files: 74% (165/222) Updating files: 75% (167/222) Updating files: 76% (169/222) Updating files: 77% (171/222) Updating files: 78% (174/222) Updating files: 79% (176/222) Updating files: 80% (178/222) Updating files: 81% (180/222) Updating files: 82% (183/222) Updating files: 83% (185/222) Updating files: 84% (187/222) Updating files: 85% (189/222) Updating files: 86% (191/222) Updating files: 87% (194/222) Updating files: 88% (196/222) Updating files: 89% (198/222) Updating files: 90% (200/222) Updating files: 91% (203/222) Updating files: 92% (205/222) Updating files: 93% (207/222) Updating files: 94% (209/222) Updating files: 95% (211/222) Updating files: 96% (214/222) Updating files: 97% (216/222) Updating files: 98% (218/222) Updating files: 99% (220/222) Updating files: 100% (222/222) Updating files: 100% (222/222), done. 2023-10-30T14:34:36.953 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 2b5f76f48e9 readme 2023-10-30T14:34:36.959 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.2 2023-10-30T14:34:37.084 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 3 -eq 48 ] 2023-10-30T14:34:37.084 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.3 2023-10-30T14:34:37.085 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.3 2023-10-30T14:34:37.085 INFO:tasks.workunit.client.0.smithi057.stdout:v0.3 2023-10-30T14:34:37.085 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.3 2023-10-30T14:34:37.087 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:34:37.092 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:34:39.258 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 68% (203/295) Updating files: 69% (204/295) Updating files: 70% (207/295) Updating files: 71% (210/295) Updating files: 72% (213/295) Updating files: 73% (216/295) Updating files: 74% (219/295) Updating files: 75% (222/295) Updating files: 76% (225/295) Updating files: 77% (228/295) Updating files: 78% (231/295) Updating files: 79% (234/295) Updating files: 80% (236/295) Updating files: 81% (239/295) Updating files: 82% (242/295) Updating files: 83% (245/295) Updating files: 84% (248/295) Updating files: 85% (251/295) Updating files: 86% (254/295) Updating files: 87% (257/295) Updating files: 88% (260/295) Updating files: 89% (263/295) Updating files: 90% (266/295) Updating files: 91% (269/295) Updating files: 92% (272/295) Updating files: 93% (275/295) Updating files: 94% (278/295) Updating files: 95% (281/295) Updating files: 96% (284/295) Updating files: 97% (287/295) Updating files: 98% (290/295) Updating files: 99% (293/295) Updating files: 100% (295/295) Updating files: 100% (295/295), done. 2023-10-30T14:34:39.288 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 2e986fe9ddf include ceph.spec in configure.ac 2023-10-30T14:34:39.297 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.3 2023-10-30T14:34:39.809 INFO:tasks.workunit.client.0.smithi057.stdout:v0.4 2023-10-30T14:34:39.809 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 4 -eq 48 ] 2023-10-30T14:34:39.809 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.4 2023-10-30T14:34:39.810 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.4 2023-10-30T14:34:39.810 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.4 2023-10-30T14:34:44.168 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 21% (53/247) Updating files: 22% (55/247) Updating files: 23% (57/247) Updating files: 24% (60/247) Updating files: 25% (62/247) Updating files: 25% (64/247) Updating files: 26% (65/247) Updating files: 27% (67/247) Updating files: 28% (70/247) Updating files: 29% (72/247) Updating files: 30% (75/247) Updating files: 31% (77/247) Updating files: 32% (80/247) Updating files: 33% (82/247) Updating files: 34% (84/247) Updating files: 35% (87/247) Updating files: 36% (89/247) Updating files: 37% (92/247) Updating files: 38% (94/247) Updating files: 39% (97/247) Updating files: 40% (99/247) Updating files: 41% (102/247) Updating files: 42% (104/247) Updating files: 43% (107/247) Updating files: 44% (109/247) Updating files: 45% (112/247) Updating files: 46% (114/247) Updating files: 47% (117/247) Updating files: 48% (119/247) Updating files: 49% (122/247) Updating files: 50% (124/247) Updating files: 51% (126/247) Updating files: 52% (129/247) Updating files: 53% (131/247) Updating files: 54% (134/247) Updating files: 55% (136/247) Updating files: 56% (139/247) Updating files: 57% (141/247) Updating files: 58% (144/247) Updating files: 59% (146/247) Updating files: 60% (149/247) Updating files: 61% (151/247) Updating files: 62% (154/247) Updating files: 63% (156/247) Updating files: 64% (159/247) Updating files: 65% (161/247) Updating files: 66% (164/247) Updating files: 67% (166/247) Updating files: 68% (168/247) Updating files: 69% (171/247) Updating files: 70% (173/247) Updating files: 71% (176/247) Updating files: 72% (178/247) Updating files: 73% (181/247) Updating files: 74% (183/247) Updating files: 75% (186/247) Updating files: 76% (188/247) Updating files: 77% (191/247) Updating files: 78% (193/247) Updating files: 79% (196/247) Updating files: 80% (198/247) Updating files: 81% (201/247) Updating files: 82% (203/247) Updating files: 83% (206/247) Updating files: 84% (208/247) Updating files: 84% (209/247) Updating files: 85% (210/247) Updating files: 86% (213/247) Updating files: 87% (215/247) Updating files: 88% (218/247) Updating files: 89% (220/247) Updating files: 90% (223/247) Updating files: 91% (225/247) Updating files: 92% (228/247) Updating files: 93% (230/247) Updating files: 94% (233/247) Updating files: 95% (235/247) Updating files: 96% (238/247) Updating files: 97% (240/247) Updating files: 98% (243/247) Updating files: 99% (245/247) Updating files: 100% (247/247) Updating files: 100% (247/247), done. 2023-10-30T14:34:44.205 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 13b1bf7c492 kclient: initializing kaddr (merge fix) 2023-10-30T14:34:44.212 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.4 2023-10-30T14:34:44.280 INFO:tasks.workunit.client.0.smithi057.stdout:v0.5 2023-10-30T14:34:44.281 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 5 -eq 48 ] 2023-10-30T14:34:44.281 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.5 2023-10-30T14:34:44.281 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.5 2023-10-30T14:34:44.281 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.5 2023-10-30T14:34:50.867 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 3% (9/245) Updating files: 4% (10/245) Updating files: 5% (13/245) Updating files: 6% (15/245) Updating files: 7% (18/245) Updating files: 8% (20/245) Updating files: 8% (21/245) Updating files: 9% (23/245) Updating files: 10% (25/245) Updating files: 11% (27/245) Updating files: 12% (30/245) Updating files: 13% (32/245) Updating files: 14% (35/245) Updating files: 15% (37/245) Updating files: 16% (40/245) Updating files: 17% (42/245) Updating files: 18% (45/245) Updating files: 19% (47/245) Updating files: 20% (49/245) Updating files: 20% (50/245) Updating files: 21% (52/245) Updating files: 22% (54/245) Updating files: 23% (57/245) Updating files: 24% (59/245) Updating files: 25% (62/245) Updating files: 26% (64/245) Updating files: 27% (67/245) Updating files: 28% (69/245) Updating files: 29% (72/245) Updating files: 30% (74/245) Updating files: 31% (76/245) Updating files: 32% (79/245) Updating files: 33% (81/245) Updating files: 34% (84/245) Updating files: 35% (86/245) Updating files: 36% (89/245) Updating files: 37% (91/245) Updating files: 38% (94/245) Updating files: 39% (96/245) Updating files: 40% (98/245) Updating files: 41% (101/245) Updating files: 42% (103/245) Updating files: 43% (106/245) Updating files: 44% (108/245) Updating files: 45% (111/245) Updating files: 46% (113/245) Updating files: 47% (116/245) Updating files: 48% (118/245) Updating files: 49% (121/245) Updating files: 50% (123/245) Updating files: 51% (125/245) Updating files: 52% (128/245) Updating files: 53% (130/245) Updating files: 54% (133/245) Updating files: 55% (135/245) Updating files: 56% (138/245) Updating files: 57% (140/245) Updating files: 58% (143/245) Updating files: 59% (145/245) Updating files: 60% (147/245) Updating files: 61% (150/245) Updating files: 62% (152/245) Updating files: 63% (155/245) Updating files: 64% (157/245) Updating files: 65% (160/245) Updating files: 66% (162/245) Updating files: 67% (165/245) Updating files: 68% (167/245) Updating files: 68% (168/245) Updating files: 69% (170/245) Updating files: 70% (172/245) Updating files: 71% (174/245) Updating files: 72% (177/245) Updating files: 73% (179/245) Updating files: 74% (182/245) Updating files: 75% (184/245) Updating files: 76% (187/245) Updating files: 77% (189/245) Updating files: 78% (192/245) Updating files: 79% (194/245) Updating files: 80% (196/245) Updating files: 81% (199/245) Updating files: 82% (201/245) Updating files: 83% (204/245) Updating files: 84% (206/245) Updating files: 85% (209/245) Updating files: 86% (211/245) Updating files: 87% (214/245) Updating files: 88% (216/245) Updating files: 89% (219/245) Updating files: 90% (221/245) Updating files: 91% (223/245) Updating files: 92% (226/245) Updating files: 93% (228/245) Updating files: 94% (231/245) Updating files: 95% (233/245) Updating files: 96% (236/245) Updating files: 97% (238/245) Updating files: 98% (241/245) Updating files: 99% (243/245) Updating files: 100% (245/245) Updating files: 100% (245/245), done. 2023-10-30T14:34:50.894 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 1dd420957f9 journal: detect size of raw block devices properly 2023-10-30T14:34:50.900 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.5 2023-10-30T14:34:51.990 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 6 -eq 48 ] 2023-10-30T14:34:51.990 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.6 2023-10-30T14:34:51.991 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.6 2023-10-30T14:34:51.991 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.6 2023-10-30T14:34:51.991 INFO:tasks.workunit.client.0.smithi057.stdout:v0.6 2023-10-30T14:34:54.776 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 18% (43/227) Updating files: 19% (44/227) Updating files: 20% (46/227) Updating files: 21% (48/227) Updating files: 22% (50/227) Updating files: 23% (53/227) Updating files: 24% (55/227) Updating files: 25% (57/227) Updating files: 26% (60/227) Updating files: 27% (62/227) Updating files: 28% (64/227) Updating files: 29% (66/227) Updating files: 30% (69/227) Updating files: 31% (71/227) Updating files: 32% (73/227) Updating files: 33% (75/227) Updating files: 34% (78/227) Updating files: 35% (80/227) Updating files: 36% (82/227) Updating files: 37% (84/227) Updating files: 38% (87/227) Updating files: 39% (89/227) Updating files: 40% (91/227) Updating files: 41% (94/227) Updating files: 42% (96/227) Updating files: 43% (98/227) Updating files: 44% (100/227) Updating files: 45% (103/227) Updating files: 46% (105/227) Updating files: 47% (107/227) Updating files: 48% (109/227) Updating files: 49% (112/227) Updating files: 50% (114/227) Updating files: 51% (116/227) Updating files: 52% (119/227) Updating files: 53% (121/227) Updating files: 54% (123/227) Updating files: 55% (125/227) Updating files: 56% (128/227) Updating files: 57% (130/227) Updating files: 58% (132/227) Updating files: 59% (134/227) Updating files: 60% (137/227) Updating files: 61% (139/227) Updating files: 62% (141/227) Updating files: 63% (144/227) Updating files: 64% (146/227) Updating files: 65% (148/227) Updating files: 66% (150/227) Updating files: 67% (153/227) Updating files: 68% (155/227) Updating files: 69% (157/227) Updating files: 70% (159/227) Updating files: 71% (162/227) Updating files: 72% (164/227) Updating files: 73% (166/227) Updating files: 74% (168/227) Updating files: 75% (171/227) Updating files: 76% (173/227) Updating files: 77% (175/227) Updating files: 78% (178/227) Updating files: 79% (180/227) Updating files: 80% (182/227) Updating files: 81% (184/227) Updating files: 82% (187/227) Updating files: 83% (189/227) Updating files: 84% (191/227) Updating files: 85% (193/227) Updating files: 86% (196/227) Updating files: 87% (198/227) Updating files: 88% (200/227) Updating files: 89% (203/227) Updating files: 90% (205/227) Updating files: 91% (207/227) Updating files: 92% (209/227) Updating files: 93% (212/227) Updating files: 94% (214/227) Updating files: 95% (216/227) Updating files: 96% (218/227) Updating files: 96% (219/227) Updating files: 97% (221/227) Updating files: 98% (223/227) Updating files: 99% (225/227) Updating files: 100% (227/227) Updating files: 100% (227/227), done. 2023-10-30T14:34:55.057 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at a4c752df713 v0.6 2023-10-30T14:34:55.062 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.6 2023-10-30T14:34:55.780 INFO:tasks.workunit.client.0.smithi057.stdout:v0.7 2023-10-30T14:34:55.780 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 7 -eq 48 ] 2023-10-30T14:34:55.781 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.7 2023-10-30T14:34:55.781 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.7 2023-10-30T14:34:55.781 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.7 2023-10-30T14:35:07.150 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:35:07.155 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:35:09.590 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 8% (18/201) Updating files: 9% (19/201) Updating files: 10% (21/201) Updating files: 11% (23/201) Updating files: 11% (24/201) Updating files: 12% (25/201) Updating files: 13% (27/201) Updating files: 14% (29/201) Updating files: 15% (31/201) Updating files: 16% (33/201) Updating files: 17% (35/201) Updating files: 18% (37/201) Updating files: 19% (39/201) Updating files: 20% (41/201) Updating files: 21% (43/201) Updating files: 22% (45/201) Updating files: 23% (47/201) Updating files: 24% (49/201) Updating files: 25% (51/201) Updating files: 26% (53/201) Updating files: 27% (55/201) Updating files: 28% (57/201) Updating files: 29% (59/201) Updating files: 30% (61/201) Updating files: 31% (63/201) Updating files: 32% (65/201) Updating files: 33% (67/201) Updating files: 34% (69/201) Updating files: 35% (71/201) Updating files: 36% (73/201) Updating files: 37% (75/201) Updating files: 38% (77/201) Updating files: 39% (79/201) Updating files: 40% (81/201) Updating files: 41% (83/201) Updating files: 42% (85/201) Updating files: 43% (87/201) Updating files: 44% (89/201) Updating files: 45% (91/201) Updating files: 46% (93/201) Updating files: 47% (95/201) Updating files: 48% (97/201) Updating files: 49% (99/201) Updating files: 50% (101/201) Updating files: 51% (103/201) Updating files: 52% (105/201) Updating files: 53% (107/201) Updating files: 54% (109/201) Updating files: 55% (111/201) Updating files: 56% (113/201) Updating files: 57% (115/201) Updating files: 58% (117/201) Updating files: 59% (119/201) Updating files: 60% (121/201) Updating files: 61% (123/201) Updating files: 62% (125/201) Updating files: 63% (127/201) Updating files: 64% (129/201) Updating files: 65% (131/201) Updating files: 66% (133/201) Updating files: 67% (135/201) Updating files: 68% (137/201) Updating files: 69% (139/201) Updating files: 70% (141/201) Updating files: 71% (143/201) Updating files: 72% (145/201) Updating files: 73% (147/201) Updating files: 74% (149/201) Updating files: 75% (151/201) Updating files: 76% (153/201) Updating files: 77% (155/201) Updating files: 78% (157/201) Updating files: 79% (159/201) Updating files: 80% (161/201) Updating files: 81% (163/201) Updating files: 82% (165/201) Updating files: 83% (167/201) Updating files: 84% (169/201) Updating files: 85% (171/201) Updating files: 86% (173/201) Updating files: 87% (175/201) Updating files: 88% (177/201) Updating files: 89% (179/201) Updating files: 90% (181/201) Updating files: 91% (183/201) Updating files: 92% (185/201) Updating files: 93% (187/201) Updating files: 94% (189/201) Updating files: 95% (191/201) Updating files: 96% (193/201) Updating files: 97% (195/201) Updating files: 98% (197/201) Updating files: 99% (199/201) Updating files: 100% (201/201) Updating files: 100% (201/201), done. 2023-10-30T14:35:09.623 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at c46e1b320c6 mkcephfs: btrfsctl -a between mkfs and mount 2023-10-30T14:35:09.630 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.7 2023-10-30T14:35:09.760 INFO:tasks.workunit.client.0.smithi057.stdout:v0.8 2023-10-30T14:35:09.761 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 8 -eq 48 ] 2023-10-30T14:35:09.761 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.8 2023-10-30T14:35:09.761 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.8 2023-10-30T14:35:09.761 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.8 2023-10-30T14:35:13.205 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:13.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c6c9aa640 1 Processor -- start 2023-10-30T14:35:13.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c6c9aa640 1 -- start start 2023-10-30T14:35:13.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c6c9aa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c680cb7d0 0x7f0c680c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:13.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c6c9aa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0c680c3e40 0x7f0c68070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:13.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c6c9aa640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c680c2310 con 0x7f0c68150830 2023-10-30T14:35:13.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c6c9aa640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c6806bc10 con 0x7f0c680c3e40 2023-10-30T14:35:13.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c6c9aa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c68156710 con 0x7f0c680cb7d0 2023-10-30T14:35:13.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c6659c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f0c68150830 0x7f0c6806b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:59512/0 (socket says 172.21.15.57:59512) 2023-10-30T14:35:13.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c6659c640 1 -- 172.21.15.57:0/3720139167 learned_addr learned my addr 172.21.15.57:0/3720139167 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:35:13.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c65d9b640 1 --2- 172.21.15.57:0/3720139167 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c680cb7d0 0x7f0c680c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:13.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c66d9d640 1 --2- 172.21.15.57:0/3720139167 >> v2:172.21.15.88:3300/0 conn(0x7f0c680c3e40 0x7f0c68070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:13.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c6559a640 1 -- 172.21.15.57:0/3720139167 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359539059 0 0) 0x7f0c680c2310 con 0x7f0c68150830 2023-10-30T14:35:13.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c6559a640 1 -- 172.21.15.57:0/3720139167 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c50003530 con 0x7f0c68150830 2023-10-30T14:35:13.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c6559a640 1 -- 172.21.15.57:0/3720139167 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 594798707 0 0) 0x7f0c50003530 con 0x7f0c68150830 2023-10-30T14:35:13.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c6559a640 1 -- 172.21.15.57:0/3720139167 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c680cb7d0 msgr2=0x7f0c680c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:13.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c6559a640 1 --2- 172.21.15.57:0/3720139167 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c680cb7d0 0x7f0c680c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c6559a640 1 -- 172.21.15.57:0/3720139167 >> v2:172.21.15.88:3300/0 conn(0x7f0c680c3e40 msgr2=0x7f0c68070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:13.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c6559a640 1 --2- 172.21.15.57:0/3720139167 >> v2:172.21.15.88:3300/0 conn(0x7f0c680c3e40 0x7f0c68070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c6559a640 1 -- 172.21.15.57:0/3720139167 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c680c4360 con 0x7f0c68150830 2023-10-30T14:35:13.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.348+0000 7f0c6559a640 1 -- 172.21.15.57:0/3720139167 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f0c58001680 con 0x7f0c68150830 2023-10-30T14:35:13.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.352+0000 7f0c6559a640 1 -- 172.21.15.57:0/3720139167 >> v1:172.21.15.57:6789/0 conn(0x7f0c68150830 legacy=0x7f0c6806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:13.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.352+0000 7f0c6559a640 1 --2- 172.21.15.57:0/3720139167 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c680cb7d0 0x7f0c50003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:13.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.352+0000 7f0c6559a640 1 --2- 172.21.15.57:0/3720139167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c50003f00 0x7f0c500047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:13.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.352+0000 7f0c6559a640 1 --2- 172.21.15.57:0/3720139167 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c50004d10 0x7f0c50005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:13.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.352+0000 7f0c6559a640 1 -- 172.21.15.57:0/3720139167 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6806bc10 con 0x7f0c50004d10 2023-10-30T14:35:13.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.352+0000 7f0c6559a640 1 -- 172.21.15.57:0/3720139167 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c68156710 con 0x7f0c50003f00 2023-10-30T14:35:13.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.352+0000 7f0c6559a640 1 -- 172.21.15.57:0/3720139167 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5000b9c0 con 0x7f0c680cb7d0 2023-10-30T14:35:13.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.352+0000 7f0c66d9d640 1 --2- 172.21.15.57:0/3720139167 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c50004d10 0x7f0c50005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:13.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.352+0000 7f0c65d9b640 1 --2- 172.21.15.57:0/3720139167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c50003f00 0x7f0c500047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:13.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.352+0000 7f0c6659c640 1 --2- 172.21.15.57:0/3720139167 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c680cb7d0 0x7f0c50003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:13.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.352+0000 7f0c66d9d640 1 -- 172.21.15.57:0/3720139167 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c680cb7d0 msgr2=0x7f0c50003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:13.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.352+0000 7f0c66d9d640 1 --2- 172.21.15.57:0/3720139167 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c680cb7d0 0x7f0c50003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.352+0000 7f0c66d9d640 1 -- 172.21.15.57:0/3720139167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c50003f00 msgr2=0x7f0c500047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:13.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.352+0000 7f0c66d9d640 1 --2- 172.21.15.57:0/3720139167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c50003f00 0x7f0c500047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.352+0000 7f0c66d9d640 1 -- 172.21.15.57:0/3720139167 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c50003530 con 0x7f0c50004d10 2023-10-30T14:35:13.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.352+0000 7f0c66d9d640 1 --2- 172.21.15.57:0/3720139167 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c50004d10 0x7f0c50005190 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009bd0 tx=0x7f0c6001ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2edd58ccd06d45a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:13.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.352+0000 7f0c6559a640 1 -- 172.21.15.57:0/3720139167 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c600a7ea0 con 0x7f0c50004d10 2023-10-30T14:35:13.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.352+0000 7f0c6559a640 1 -- 172.21.15.57:0/3720139167 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6000f040 con 0x7f0c50004d10 2023-10-30T14:35:13.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.352+0000 7f0c6559a640 1 -- 172.21.15.57:0/3720139167 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c600af680 con 0x7f0c50004d10 2023-10-30T14:35:13.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.356+0000 7f0c6c9aa640 1 -- 172.21.15.57:0/3720139167 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c50004d10 msgr2=0x7f0c50005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:13.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.356+0000 7f0c6c9aa640 1 --2- 172.21.15.57:0/3720139167 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c50004d10 0x7f0c50005190 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009bd0 tx=0x7f0c6001ee90 comp rx=0 tx=0).stop 2023-10-30T14:35:13.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.356+0000 7f0c66d9d640 1 -- 172.21.15.57:0/3720139167 reap_dead start 2023-10-30T14:35:13.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.356+0000 7f0c6c9aa640 1 -- 172.21.15.57:0/3720139167 shutdown_connections 2023-10-30T14:35:13.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.356+0000 7f0c6c9aa640 1 -- 172.21.15.57:0/3720139167 >> 172.21.15.57:0/3720139167 conn(0x7f0c680cac70 msgr2=0x7f0c680cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:13.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.356+0000 7f0c6c9aa640 1 -- 172.21.15.57:0/3720139167 shutdown_connections 2023-10-30T14:35:13.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.356+0000 7f0c6c9aa640 1 -- 172.21.15.57:0/3720139167 wait complete. 2023-10-30T14:35:13.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.356+0000 7f0c6c9aa640 1 Processor -- start 2023-10-30T14:35:13.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.356+0000 7f0c6c9aa640 1 -- start start 2023-10-30T14:35:13.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.356+0000 7f0c6c9aa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c50004d10 0x7f0c681577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:13.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.356+0000 7f0c6c9aa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c680c2e40 0x7f0c68155580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:13.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.356+0000 7f0c6c9aa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c68148470 0x7f0c681488f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:13.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.356+0000 7f0c6c9aa640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6815a8f0 con 0x7f0c680c2e40 2023-10-30T14:35:13.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.356+0000 7f0c6c9aa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c68148e30 con 0x7f0c68148470 2023-10-30T14:35:13.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.356+0000 7f0c6c9aa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c680cab10 con 0x7f0c50004d10 2023-10-30T14:35:13.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.356+0000 7f0c65d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c680c2e40 0x7f0c68155580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:13.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.356+0000 7f0c65d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c680c2e40 0x7f0c68155580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59346/0 (socket says 172.21.15.57:59346) 2023-10-30T14:35:13.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.356+0000 7f0c65d9b640 1 -- 172.21.15.57:0/816445103 learned_addr learned my addr 172.21.15.57:0/816445103 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:13.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.360+0000 7f0c6659c640 1 --2- 172.21.15.57:0/816445103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c50004d10 0x7f0c681577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:13.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.360+0000 7f0c66d9d640 1 --2- 172.21.15.57:0/816445103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c68148470 0x7f0c681488f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:13.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.360+0000 7f0c65d9b640 1 -- 172.21.15.57:0/816445103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c50004d10 msgr2=0x7f0c681577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:13.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.360+0000 7f0c65d9b640 1 --2- 172.21.15.57:0/816445103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c50004d10 0x7f0c681577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.360+0000 7f0c65d9b640 1 -- 172.21.15.57:0/816445103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c68148470 msgr2=0x7f0c681488f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:13.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.360+0000 7f0c65d9b640 1 --2- 172.21.15.57:0/816445103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c68148470 0x7f0c681488f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.360+0000 7f0c65d9b640 1 -- 172.21.15.57:0/816445103 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c680b8100 con 0x7f0c680c2e40 2023-10-30T14:35:13.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.360+0000 7f0c65d9b640 1 --2- 172.21.15.57:0/816445103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c680c2e40 0x7f0c68155580 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c011590 tx=0x7f0c5c011990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:13.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.360+0000 7f0c56ffd640 1 -- 172.21.15.57:0/816445103 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c015930 con 0x7f0c680c2e40 2023-10-30T14:35:13.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.360+0000 7f0c56ffd640 1 -- 172.21.15.57:0/816445103 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5c015a90 con 0x7f0c680c2e40 2023-10-30T14:35:13.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.360+0000 7f0c56ffd640 1 -- 172.21.15.57:0/816445103 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c015e20 con 0x7f0c680c2e40 2023-10-30T14:35:13.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.360+0000 7f0c6c9aa640 1 -- 172.21.15.57:0/816445103 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c68155e50 con 0x7f0c680c2e40 2023-10-30T14:35:13.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.360+0000 7f0c6c9aa640 1 -- 172.21.15.57:0/816445103 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c680ca830 con 0x7f0c680c2e40 2023-10-30T14:35:13.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.360+0000 7f0c6c9aa640 1 -- 172.21.15.57:0/816445103 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c6804e170 con 0x7f0c680c2e40 2023-10-30T14:35:13.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.360+0000 7f0c56ffd640 1 -- 172.21.15.57:0/816445103 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0c5c01d400 con 0x7f0c680c2e40 2023-10-30T14:35:13.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.360+0000 7f0c56ffd640 1 --2- 172.21.15.57:0/816445103 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0c48066520 0x7f0c480689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:13.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.360+0000 7f0c56ffd640 1 -- 172.21.15.57:0/816445103 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f0c5c084940 con 0x7f0c680c2e40 2023-10-30T14:35:13.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.364+0000 7f0c6659c640 1 --2- 172.21.15.57:0/816445103 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0c48066520 0x7f0c480689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:13.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.364+0000 7f0c6659c640 1 --2- 172.21.15.57:0/816445103 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0c48066520 0x7f0c480689e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0c58001060 tx=0x7f0c58007040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:13.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.368+0000 7f0c56ffd640 1 -- 172.21.15.57:0/816445103 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0c5c051970 con 0x7f0c680c2e40 2023-10-30T14:35:13.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.556+0000 7f0c6c9aa640 1 -- 172.21.15.57:0/816445103 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c68071480 con 0x7f0c680c2e40 2023-10-30T14:35:13.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.556+0000 7f0c56ffd640 1 -- 172.21.15.57:0/816445103 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6535 (secure 0 0 0) 0x7f0c5c054f90 con 0x7f0c680c2e40 2023-10-30T14:35:13.559 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:13.559 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6747,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:34:12.961116+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:active","state_seq":40,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:13.560 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 119 2023-10-30T14:35:13.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.560+0000 7f0c6c9aa640 1 -- 172.21.15.57:0/816445103 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0c48066520 msgr2=0x7f0c480689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:13.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.560+0000 7f0c6c9aa640 1 --2- 172.21.15.57:0/816445103 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0c48066520 0x7f0c480689e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0c58001060 tx=0x7f0c58007040 comp rx=0 tx=0).stop 2023-10-30T14:35:13.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.560+0000 7f0c6c9aa640 1 -- 172.21.15.57:0/816445103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c680c2e40 msgr2=0x7f0c68155580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:13.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.560+0000 7f0c6c9aa640 1 --2- 172.21.15.57:0/816445103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c680c2e40 0x7f0c68155580 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c011590 tx=0x7f0c5c011990 comp rx=0 tx=0).stop 2023-10-30T14:35:13.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.560+0000 7f0c6c9aa640 1 -- 172.21.15.57:0/816445103 shutdown_connections 2023-10-30T14:35:13.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.560+0000 7f0c6c9aa640 1 --2- 172.21.15.57:0/816445103 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0c48066520 0x7f0c480689e0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.560+0000 7f0c6c9aa640 1 --2- 172.21.15.57:0/816445103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c68148470 0x7f0c681488f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.560+0000 7f0c6c9aa640 1 --2- 172.21.15.57:0/816445103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c680c2e40 0x7f0c68155580 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.560+0000 7f0c6c9aa640 1 --2- 172.21.15.57:0/816445103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c50004d10 0x7f0c681577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.560+0000 7f0c6c9aa640 1 -- 172.21.15.57:0/816445103 >> 172.21.15.57:0/816445103 conn(0x7f0c680cac70 msgr2=0x7f0c680c54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:13.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.560+0000 7f0c6c9aa640 1 -- 172.21.15.57:0/816445103 shutdown_connections 2023-10-30T14:35:13.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.560+0000 7f0c6c9aa640 1 -- 172.21.15.57:0/816445103 wait complete. 2023-10-30T14:35:13.580 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=1) 2023-10-30T14:35:13.580 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T14:35:13.581 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:35:13.597 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T14:35:13.598 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:13.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.764+0000 7ff115f9d640 1 Processor -- start 2023-10-30T14:35:13.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.764+0000 7ff115f9d640 1 -- start start 2023-10-30T14:35:13.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.764+0000 7ff115f9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff1100cb7d0 0x7ff1100c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:13.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.764+0000 7ff115f9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff1100c3e40 0x7ff110070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:13.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.764+0000 7ff115f9d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1100c2310 con 0x7ff110150830 2023-10-30T14:35:13.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.764+0000 7ff115f9d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff11006bc10 con 0x7ff1100cb7d0 2023-10-30T14:35:13.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.764+0000 7ff115f9d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff110156710 con 0x7ff1100c3e40 2023-10-30T14:35:13.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.764+0000 7ff10f7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff110150830 0x7ff11006b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:39374/0 (socket says 172.21.15.57:39374) 2023-10-30T14:35:13.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.764+0000 7ff10f7fe640 1 -- 172.21.15.57:0/529752613 learned_addr learned my addr 172.21.15.57:0/529752613 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:35:13.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.764+0000 7ff10effd640 1 --2- 172.21.15.57:0/529752613 >> v2:172.21.15.57:3300/0 conn(0x7ff1100cb7d0 0x7ff1100c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:13.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.764+0000 7ff10ffff640 1 --2- 172.21.15.57:0/529752613 >> v2:172.21.15.88:3300/0 conn(0x7ff1100c3e40 0x7ff110070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:13.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.764+0000 7ff10effd640 1 -- 172.21.15.57:0/529752613 >> v2:172.21.15.88:3300/0 conn(0x7ff1100c3e40 msgr2=0x7ff110070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:13.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.764+0000 7ff10effd640 1 --2- 172.21.15.57:0/529752613 >> v2:172.21.15.88:3300/0 conn(0x7ff1100c3e40 0x7ff110070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.764+0000 7ff10effd640 1 -- 172.21.15.57:0/529752613 >> v1:172.21.15.57:6789/0 conn(0x7ff110150830 legacy=0x7ff11006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:13.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.764+0000 7ff10effd640 1 -- 172.21.15.57:0/529752613 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1100c4360 con 0x7ff1100cb7d0 2023-10-30T14:35:13.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.764+0000 7ff10e7fc640 1 -- 172.21.15.57:0/529752613 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712466418 0 0) 0x7ff1100c2310 con 0x7ff110150830 2023-10-30T14:35:13.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.768+0000 7ff10effd640 1 --2- 172.21.15.57:0/529752613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1100cb7d0 0x7ff1100c3900 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7ff110066e10 tx=0x7ff10401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c2ca918dfa5ce0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:13.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.768+0000 7ff10e7fc640 1 -- 172.21.15.57:0/529752613 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1040a8950 con 0x7ff1100cb7d0 2023-10-30T14:35:13.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.768+0000 7ff10e7fc640 1 -- 172.21.15.57:0/529752613 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1040a8ab0 con 0x7ff1100cb7d0 2023-10-30T14:35:13.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.768+0000 7ff10e7fc640 1 -- 172.21.15.57:0/529752613 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1040a8e40 con 0x7ff1100cb7d0 2023-10-30T14:35:13.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.768+0000 7ff115f9d640 1 -- 172.21.15.57:0/529752613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1100cb7d0 msgr2=0x7ff1100c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:13.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.768+0000 7ff115f9d640 1 --2- 172.21.15.57:0/529752613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1100cb7d0 0x7ff1100c3900 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7ff110066e10 tx=0x7ff10401ec50 comp rx=0 tx=0).stop 2023-10-30T14:35:13.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.768+0000 7ff115f9d640 1 -- 172.21.15.57:0/529752613 shutdown_connections 2023-10-30T14:35:13.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.768+0000 7ff115f9d640 1 --2- 172.21.15.57:0/529752613 >> v2:172.21.15.88:3300/0 conn(0x7ff1100c3e40 0x7ff110070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.768+0000 7ff115f9d640 1 --2- 172.21.15.57:0/529752613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1100cb7d0 0x7ff1100c3900 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.768+0000 7ff115f9d640 1 -- 172.21.15.57:0/529752613 >> 172.21.15.57:0/529752613 conn(0x7ff1100cac70 msgr2=0x7ff1100cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:13.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.768+0000 7ff115f9d640 1 -- 172.21.15.57:0/529752613 shutdown_connections 2023-10-30T14:35:13.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.768+0000 7ff115f9d640 1 -- 172.21.15.57:0/529752613 wait complete. 2023-10-30T14:35:13.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff115f9d640 1 Processor -- start 2023-10-30T14:35:13.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff115f9d640 1 -- start start 2023-10-30T14:35:13.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff115f9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1100c3e40 0x7ff11006c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:13.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff115f9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1100cb7d0 0x7ff110155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:13.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff115f9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff110150830 0x7ff1101f22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:13.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff115f9d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1101529f0 con 0x7ff1100cb7d0 2023-10-30T14:35:13.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff115f9d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11006c7a0 con 0x7ff110150830 2023-10-30T14:35:13.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff115f9d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1101f1e10 con 0x7ff1100c3e40 2023-10-30T14:35:13.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff10effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1100cb7d0 0x7ff110155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:13.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff10effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1100cb7d0 0x7ff110155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43786/0 (socket says 172.21.15.57:43786) 2023-10-30T14:35:13.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff10effd640 1 -- 172.21.15.57:0/2734761438 learned_addr learned my addr 172.21.15.57:0/2734761438 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:13.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff10ffff640 1 --2- 172.21.15.57:0/2734761438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff110150830 0x7ff1101f22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:13.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff10f7fe640 1 --2- 172.21.15.57:0/2734761438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1100c3e40 0x7ff11006c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:13.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff10effd640 1 -- 172.21.15.57:0/2734761438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1100c3e40 msgr2=0x7ff11006c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:13.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff10effd640 1 --2- 172.21.15.57:0/2734761438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1100c3e40 0x7ff11006c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff10effd640 1 -- 172.21.15.57:0/2734761438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff110150830 msgr2=0x7ff1101f22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:13.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff10effd640 1 --2- 172.21.15.57:0/2734761438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff110150830 0x7ff1101f22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff10effd640 1 -- 172.21.15.57:0/2734761438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff104009420 con 0x7ff1100cb7d0 2023-10-30T14:35:13.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff10f7fe640 1 --2- 172.21.15.57:0/2734761438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1100c3e40 0x7ff11006c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:13.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff10effd640 1 --2- 172.21.15.57:0/2734761438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1100cb7d0 0x7ff110155850 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7ff1040b3040 tx=0x7ff10401b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:13.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff0f7fff640 1 -- 172.21.15.57:0/2734761438 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1040a8950 con 0x7ff1100cb7d0 2023-10-30T14:35:13.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff0f7fff640 1 -- 172.21.15.57:0/2734761438 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1040b9410 con 0x7ff1100cb7d0 2023-10-30T14:35:13.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff0f7fff640 1 -- 172.21.15.57:0/2734761438 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1040b97c0 con 0x7ff1100cb7d0 2023-10-30T14:35:13.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.772+0000 7ff115f9d640 1 -- 172.21.15.57:0/2734761438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff110159520 con 0x7ff1100cb7d0 2023-10-30T14:35:13.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.776+0000 7ff115f9d640 1 -- 172.21.15.57:0/2734761438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff110159ae0 con 0x7ff1100cb7d0 2023-10-30T14:35:13.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.776+0000 7ff115f9d640 1 -- 172.21.15.57:0/2734761438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff110066510 con 0x7ff1100cb7d0 2023-10-30T14:35:13.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.776+0000 7ff0f7fff640 1 -- 172.21.15.57:0/2734761438 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff1040b7040 con 0x7ff1100cb7d0 2023-10-30T14:35:13.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.776+0000 7ff0f7fff640 1 --2- 172.21.15.57:0/2734761438 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0fc0664a0 0x7ff0fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:13.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.776+0000 7ff0f7fff640 1 -- 172.21.15.57:0/2734761438 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7ff104121fe0 con 0x7ff1100cb7d0 2023-10-30T14:35:13.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.776+0000 7ff10f7fe640 1 --2- 172.21.15.57:0/2734761438 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0fc0664a0 0x7ff0fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:13.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.780+0000 7ff10f7fe640 1 --2- 172.21.15.57:0/2734761438 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0fc0664a0 0x7ff0fc068960 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff100000f70 tx=0x7ff100009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:13.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.780+0000 7ff0f7fff640 1 -- 172.21.15.57:0/2734761438 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff1040eef40 con 0x7ff1100cb7d0 2023-10-30T14:35:13.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.972+0000 7ff115f9d640 1 -- 172.21.15.57:0/2734761438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff11004e170 con 0x7ff1100cb7d0 2023-10-30T14:35:13.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.972+0000 7ff0f7fff640 1 -- 172.21.15.57:0/2734761438 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6535 (secure 0 0 0) 0x7ff1040b5030 con 0x7ff1100cb7d0 2023-10-30T14:35:13.974 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:13.974 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6747,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:34:12.961116+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:active","state_seq":40,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:13.975 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 119 2023-10-30T14:35:13.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.972+0000 7ff115f9d640 1 -- 172.21.15.57:0/2734761438 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0fc0664a0 msgr2=0x7ff0fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:13.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.972+0000 7ff115f9d640 1 --2- 172.21.15.57:0/2734761438 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0fc0664a0 0x7ff0fc068960 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff100000f70 tx=0x7ff100009040 comp rx=0 tx=0).stop 2023-10-30T14:35:13.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.972+0000 7ff115f9d640 1 -- 172.21.15.57:0/2734761438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1100cb7d0 msgr2=0x7ff110155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:13.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.972+0000 7ff115f9d640 1 --2- 172.21.15.57:0/2734761438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1100cb7d0 0x7ff110155850 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7ff1040b3040 tx=0x7ff10401b9f0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.972+0000 7ff115f9d640 1 -- 172.21.15.57:0/2734761438 shutdown_connections 2023-10-30T14:35:13.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.972+0000 7ff115f9d640 1 --2- 172.21.15.57:0/2734761438 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0fc0664a0 0x7ff0fc068960 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.972+0000 7ff115f9d640 1 --2- 172.21.15.57:0/2734761438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff110150830 0x7ff1101f22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.972+0000 7ff115f9d640 1 --2- 172.21.15.57:0/2734761438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1100cb7d0 0x7ff110155850 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.972+0000 7ff115f9d640 1 --2- 172.21.15.57:0/2734761438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1100c3e40 0x7ff11006c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:13.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.972+0000 7ff115f9d640 1 -- 172.21.15.57:0/2734761438 >> 172.21.15.57:0/2734761438 conn(0x7ff1100cac70 msgr2=0x7ff1100733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:13.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.972+0000 7ff115f9d640 1 -- 172.21.15.57:0/2734761438 shutdown_connections 2023-10-30T14:35:13.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:13.972+0000 7ff115f9d640 1 -- 172.21.15.57:0/2734761438 wait complete. 2023-10-30T14:35:13.995 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:35:15.996 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:16.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.148+0000 7eff266d6640 1 Processor -- start 2023-10-30T14:35:16.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.148+0000 7eff266d6640 1 -- start start 2023-10-30T14:35:16.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.148+0000 7eff266d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff200c55d0 0x7eff2006e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:16.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.148+0000 7eff266d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff20151b90 0x7eff2015b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:16.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.148+0000 7eff266d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff201578e0 con 0x7eff2006eb70 2023-10-30T14:35:16.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.148+0000 7eff266d6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff20158720 con 0x7eff200c55d0 2023-10-30T14:35:16.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.148+0000 7eff266d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff201576f0 con 0x7eff20151b90 2023-10-30T14:35:16.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.148+0000 7eff24c73640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff20151b90 0x7eff2015b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:16.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.148+0000 7eff1ffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff200c55d0 0x7eff2006e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:16.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.148+0000 7eff1ffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff200c55d0 0x7eff2006e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:46106/0 (socket says 172.21.15.57:46106) 2023-10-30T14:35:16.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.148+0000 7eff1ffff640 1 -- 172.21.15.57:0/4152768827 learned_addr learned my addr 172.21.15.57:0/4152768827 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:16.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.148+0000 7eff24c73640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff20151b90 0x7eff2015b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:57378/0 (socket says 172.21.15.57:57378) 2023-10-30T14:35:16.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.152+0000 7eff1ffff640 1 -- 172.21.15.57:0/4152768827 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff20151b90 msgr2=0x7eff2015b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:16.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.152+0000 7eff1ffff640 1 --2- 172.21.15.57:0/4152768827 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff20151b90 0x7eff2015b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:16.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.152+0000 7eff1ffff640 1 -- 172.21.15.57:0/4152768827 >> v1:172.21.15.88:6789/0 conn(0x7eff2006eb70 legacy=0x7eff20151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:16.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.152+0000 7eff1ffff640 1 -- 172.21.15.57:0/4152768827 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff20157ba0 con 0x7eff200c55d0 2023-10-30T14:35:16.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.152+0000 7eff1ffff640 1 --2- 172.21.15.57:0/4152768827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff200c55d0 0x7eff2006e630 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7eff100098a0 tx=0x7eff1001ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=887ac059c3ebbde8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:16.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.152+0000 7eff1effd640 1 -- 172.21.15.57:0/4152768827 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff100a8a10 con 0x7eff200c55d0 2023-10-30T14:35:16.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.152+0000 7eff1effd640 1 -- 172.21.15.57:0/4152768827 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff100a8b70 con 0x7eff200c55d0 2023-10-30T14:35:16.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.152+0000 7eff1effd640 1 -- 172.21.15.57:0/4152768827 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff100b0440 con 0x7eff200c55d0 2023-10-30T14:35:16.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff266d6640 1 -- 172.21.15.57:0/4152768827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff200c55d0 msgr2=0x7eff2006e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:16.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff266d6640 1 --2- 172.21.15.57:0/4152768827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff200c55d0 0x7eff2006e630 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7eff100098a0 tx=0x7eff1001ed10 comp rx=0 tx=0).stop 2023-10-30T14:35:16.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff266d6640 1 -- 172.21.15.57:0/4152768827 shutdown_connections 2023-10-30T14:35:16.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff266d6640 1 --2- 172.21.15.57:0/4152768827 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff20151b90 0x7eff2015b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:16.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff266d6640 1 --2- 172.21.15.57:0/4152768827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff200c55d0 0x7eff2006e630 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:16.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff266d6640 1 -- 172.21.15.57:0/4152768827 >> 172.21.15.57:0/4152768827 conn(0x7eff2006afa0 msgr2=0x7eff2006b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:16.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff266d6640 1 -- 172.21.15.57:0/4152768827 shutdown_connections 2023-10-30T14:35:16.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff266d6640 1 -- 172.21.15.57:0/4152768827 wait complete. 2023-10-30T14:35:16.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff266d6640 1 Processor -- start 2023-10-30T14:35:16.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff266d6640 1 -- start start 2023-10-30T14:35:16.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff266d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff2006eb70 0x7eff2014e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:16.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff266d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff200c55d0 0x7eff20159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:16.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff266d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff20151b90 0x7eff201521f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:16.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff266d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff200c4730 con 0x7eff200c55d0 2023-10-30T14:35:16.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff266d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff200c4a80 con 0x7eff20151b90 2023-10-30T14:35:16.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff266d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff200c4f70 con 0x7eff2006eb70 2023-10-30T14:35:16.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff1f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff200c55d0 0x7eff20159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:16.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff1f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff200c55d0 0x7eff20159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43792/0 (socket says 172.21.15.57:43792) 2023-10-30T14:35:16.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff1f7fe640 1 -- 172.21.15.57:0/510216236 learned_addr learned my addr 172.21.15.57:0/510216236 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:16.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff1ffff640 1 --2- 172.21.15.57:0/510216236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff2006eb70 0x7eff2014e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:16.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff24c73640 1 --2- 172.21.15.57:0/510216236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff20151b90 0x7eff201521f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:16.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff1f7fe640 1 -- 172.21.15.57:0/510216236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff2006eb70 msgr2=0x7eff2014e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:16.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff1f7fe640 1 --2- 172.21.15.57:0/510216236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff2006eb70 0x7eff2014e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:16.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff1f7fe640 1 -- 172.21.15.57:0/510216236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff20151b90 msgr2=0x7eff201521f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:16.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff1f7fe640 1 --2- 172.21.15.57:0/510216236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff20151b90 0x7eff201521f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:16.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff1f7fe640 1 -- 172.21.15.57:0/510216236 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff100094d0 con 0x7eff200c55d0 2023-10-30T14:35:16.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.156+0000 7eff24c73640 1 --2- 172.21.15.57:0/510216236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff20151b90 0x7eff201521f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:16.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.160+0000 7eff1f7fe640 1 --2- 172.21.15.57:0/510216236 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff200c55d0 0x7eff20159ce0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7eff1400c3c0 tx=0x7eff1400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:16.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.160+0000 7eff1cff9640 1 -- 172.21.15.57:0/510216236 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff140052d0 con 0x7eff200c55d0 2023-10-30T14:35:16.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.160+0000 7eff1cff9640 1 -- 172.21.15.57:0/510216236 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff14005430 con 0x7eff200c55d0 2023-10-30T14:35:16.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.160+0000 7eff266d6640 1 -- 172.21.15.57:0/510216236 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff200c3ca0 con 0x7eff200c55d0 2023-10-30T14:35:16.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.160+0000 7eff1cff9640 1 -- 172.21.15.57:0/510216236 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff14003840 con 0x7eff200c55d0 2023-10-30T14:35:16.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.160+0000 7eff266d6640 1 -- 172.21.15.57:0/510216236 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff200caa10 con 0x7eff200c55d0 2023-10-30T14:35:16.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.160+0000 7eff266d6640 1 -- 172.21.15.57:0/510216236 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff2004e170 con 0x7eff200c55d0 2023-10-30T14:35:16.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.160+0000 7eff1cff9640 1 -- 172.21.15.57:0/510216236 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7eff140039a0 con 0x7eff200c55d0 2023-10-30T14:35:16.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.160+0000 7eff1cff9640 1 --2- 172.21.15.57:0/510216236 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7eff0c0664a0 0x7eff0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:16.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.164+0000 7eff1cff9640 1 -- 172.21.15.57:0/510216236 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7eff14083810 con 0x7eff200c55d0 2023-10-30T14:35:16.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.164+0000 7eff1ffff640 1 --2- 172.21.15.57:0/510216236 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7eff0c0664a0 0x7eff0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:16.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.164+0000 7eff1ffff640 1 --2- 172.21.15.57:0/510216236 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7eff0c0664a0 0x7eff0c068960 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7eff100b2000 tx=0x7eff10009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:16.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.168+0000 7eff1cff9640 1 -- 172.21.15.57:0/510216236 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7eff14050880 con 0x7eff200c55d0 2023-10-30T14:35:16.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.356+0000 7eff266d6640 1 -- 172.21.15.57:0/510216236 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff200c3e30 con 0x7eff200c55d0 2023-10-30T14:35:16.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.356+0000 7eff1cff9640 1 -- 172.21.15.57:0/510216236 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6535 (secure 0 0 0) 0x7eff14053ea0 con 0x7eff200c55d0 2023-10-30T14:35:16.358 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:16.358 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6747,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:34:12.961116+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:active","state_seq":40,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:16.358 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 119 2023-10-30T14:35:16.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.356+0000 7eff266d6640 1 -- 172.21.15.57:0/510216236 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7eff0c0664a0 msgr2=0x7eff0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:16.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.356+0000 7eff266d6640 1 --2- 172.21.15.57:0/510216236 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7eff0c0664a0 0x7eff0c068960 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7eff100b2000 tx=0x7eff10009040 comp rx=0 tx=0).stop 2023-10-30T14:35:16.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.356+0000 7eff266d6640 1 -- 172.21.15.57:0/510216236 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff200c55d0 msgr2=0x7eff20159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:16.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.356+0000 7eff266d6640 1 --2- 172.21.15.57:0/510216236 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff200c55d0 0x7eff20159ce0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7eff1400c3c0 tx=0x7eff1400c880 comp rx=0 tx=0).stop 2023-10-30T14:35:16.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.356+0000 7eff266d6640 1 -- 172.21.15.57:0/510216236 shutdown_connections 2023-10-30T14:35:16.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.356+0000 7eff266d6640 1 --2- 172.21.15.57:0/510216236 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7eff0c0664a0 0x7eff0c068960 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:16.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.356+0000 7eff266d6640 1 --2- 172.21.15.57:0/510216236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff20151b90 0x7eff201521f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:16.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.356+0000 7eff266d6640 1 --2- 172.21.15.57:0/510216236 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff200c55d0 0x7eff20159ce0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:16.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.356+0000 7eff266d6640 1 --2- 172.21.15.57:0/510216236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff2006eb70 0x7eff2014e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:16.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.356+0000 7eff266d6640 1 -- 172.21.15.57:0/510216236 >> 172.21.15.57:0/510216236 conn(0x7eff2006afa0 msgr2=0x7eff200b1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:16.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.356+0000 7eff266d6640 1 -- 172.21.15.57:0/510216236 shutdown_connections 2023-10-30T14:35:16.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:16.356+0000 7eff266d6640 1 -- 172.21.15.57:0/510216236 wait complete. 2023-10-30T14:35:16.379 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:35:18.380 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:18.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.532+0000 7f7a5631d640 1 Processor -- start 2023-10-30T14:35:18.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.532+0000 7f7a5631d640 1 -- start start 2023-10-30T14:35:18.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.532+0000 7f7a5631d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a500cb7d0 0x7f7a500c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:18.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.532+0000 7f7a5631d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7a500c3e40 0x7f7a50070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:18.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.532+0000 7f7a5631d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a500c2310 con 0x7f7a50150830 2023-10-30T14:35:18.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.532+0000 7f7a5631d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a5006bc10 con 0x7f7a500c3e40 2023-10-30T14:35:18.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.532+0000 7f7a5631d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a50156710 con 0x7f7a500cb7d0 2023-10-30T14:35:18.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.536+0000 7f7a548ba640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7a500c3e40 0x7f7a50070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:18.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.536+0000 7f7a4f7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a500cb7d0 0x7f7a500c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:18.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.536+0000 7f7a4f7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a500cb7d0 0x7f7a500c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:57390/0 (socket says 172.21.15.57:57390) 2023-10-30T14:35:18.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.536+0000 7f7a548ba640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7a500c3e40 0x7f7a50070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:46134/0 (socket says 172.21.15.57:46134) 2023-10-30T14:35:18.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.536+0000 7f7a4f7fe640 1 -- 172.21.15.57:0/3062785486 learned_addr learned my addr 172.21.15.57:0/3062785486 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:18.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.536+0000 7f7a4ffff640 1 --1- 172.21.15.57:0/3062785486 >> v1:172.21.15.88:6789/0 conn(0x7f7a50150830 0x7f7a5006b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:39900/0 (socket says 172.21.15.57:39900) 2023-10-30T14:35:18.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.536+0000 7f7a4f7fe640 1 -- 172.21.15.57:0/3062785486 >> v2:172.21.15.88:3300/0 conn(0x7f7a500c3e40 msgr2=0x7f7a50070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:18.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.536+0000 7f7a4f7fe640 1 --2- 172.21.15.57:0/3062785486 >> v2:172.21.15.88:3300/0 conn(0x7f7a500c3e40 0x7f7a50070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:18.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.536+0000 7f7a4f7fe640 1 -- 172.21.15.57:0/3062785486 >> v1:172.21.15.88:6789/0 conn(0x7f7a50150830 legacy=0x7f7a5006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:18.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.536+0000 7f7a4f7fe640 1 -- 172.21.15.57:0/3062785486 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a500c4360 con 0x7f7a500cb7d0 2023-10-30T14:35:18.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.536+0000 7f7a548ba640 1 --2- 172.21.15.57:0/3062785486 >> v2:172.21.15.88:3300/0 conn(0x7f7a500c3e40 0x7f7a50070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:35:18.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.536+0000 7f7a4f7fe640 1 --2- 172.21.15.57:0/3062785486 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a500cb7d0 0x7f7a500c3900 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009870 tx=0x7f7a4401ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0053702acab83ad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:18.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a4effd640 1 -- 172.21.15.57:0/3062785486 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a440a8a30 con 0x7f7a500cb7d0 2023-10-30T14:35:18.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a4effd640 1 -- 172.21.15.57:0/3062785486 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a440a8b90 con 0x7f7a500cb7d0 2023-10-30T14:35:18.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a4effd640 1 -- 172.21.15.57:0/3062785486 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a440b0440 con 0x7f7a500cb7d0 2023-10-30T14:35:18.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a5631d640 1 -- 172.21.15.57:0/3062785486 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a500cb7d0 msgr2=0x7f7a500c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:18.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a5631d640 1 --2- 172.21.15.57:0/3062785486 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a500cb7d0 0x7f7a500c3900 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009870 tx=0x7f7a4401ed30 comp rx=0 tx=0).stop 2023-10-30T14:35:18.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a5631d640 1 -- 172.21.15.57:0/3062785486 shutdown_connections 2023-10-30T14:35:18.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a5631d640 1 --2- 172.21.15.57:0/3062785486 >> v2:172.21.15.88:3300/0 conn(0x7f7a500c3e40 0x7f7a50070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:18.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a5631d640 1 --2- 172.21.15.57:0/3062785486 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a500cb7d0 0x7f7a500c3900 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:18.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a5631d640 1 -- 172.21.15.57:0/3062785486 >> 172.21.15.57:0/3062785486 conn(0x7f7a500cac70 msgr2=0x7f7a500cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:18.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a5631d640 1 -- 172.21.15.57:0/3062785486 shutdown_connections 2023-10-30T14:35:18.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a5631d640 1 -- 172.21.15.57:0/3062785486 wait complete. 2023-10-30T14:35:18.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a5631d640 1 Processor -- start 2023-10-30T14:35:18.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a5631d640 1 -- start start 2023-10-30T14:35:18.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a5631d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a500c3e40 0x7f7a501528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:18.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a5631d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a500cb7d0 0x7f7a5014d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:18.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a5631d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a50150830 0x7f7a5014b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:18.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a5631d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a501532e0 con 0x7f7a500c3e40 2023-10-30T14:35:18.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a5631d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a50152de0 con 0x7f7a500cb7d0 2023-10-30T14:35:18.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a5631d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a50159fc0 con 0x7f7a50150830 2023-10-30T14:35:18.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.540+0000 7f7a4ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a500c3e40 0x7f7a501528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:18.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.544+0000 7f7a4ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a500c3e40 0x7f7a501528a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43802/0 (socket says 172.21.15.57:43802) 2023-10-30T14:35:18.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.544+0000 7f7a4ffff640 1 -- 172.21.15.57:0/4071709668 learned_addr learned my addr 172.21.15.57:0/4071709668 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:18.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.544+0000 7f7a548ba640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a50150830 0x7f7a5014b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:18.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.544+0000 7f7a4f7fe640 1 --2- 172.21.15.57:0/4071709668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a500cb7d0 0x7f7a5014d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:18.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.544+0000 7f7a4ffff640 1 -- 172.21.15.57:0/4071709668 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a50150830 msgr2=0x7f7a5014b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:18.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.544+0000 7f7a4ffff640 1 --2- 172.21.15.57:0/4071709668 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a50150830 0x7f7a5014b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:18.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.544+0000 7f7a4ffff640 1 -- 172.21.15.57:0/4071709668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a500cb7d0 msgr2=0x7f7a5014d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:18.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.544+0000 7f7a4ffff640 1 --2- 172.21.15.57:0/4071709668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a500cb7d0 0x7f7a5014d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:18.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.544+0000 7f7a4ffff640 1 -- 172.21.15.57:0/4071709668 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a48009a70 con 0x7f7a500c3e40 2023-10-30T14:35:18.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.544+0000 7f7a4ffff640 1 --2- 172.21.15.57:0/4071709668 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a500c3e40 0x7f7a501528a0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f7a4000c460 tx=0x7f7a4000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:18.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.544+0000 7f7a4cff9640 1 -- 172.21.15.57:0/4071709668 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a400074e0 con 0x7f7a500c3e40 2023-10-30T14:35:18.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.544+0000 7f7a4cff9640 1 -- 172.21.15.57:0/4071709668 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a40007640 con 0x7f7a500c3e40 2023-10-30T14:35:18.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.544+0000 7f7a5631d640 1 -- 172.21.15.57:0/4071709668 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a440094d0 con 0x7f7a500c3e40 2023-10-30T14:35:18.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.544+0000 7f7a4cff9640 1 -- 172.21.15.57:0/4071709668 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a40004a10 con 0x7f7a500c3e40 2023-10-30T14:35:18.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.544+0000 7f7a5631d640 1 -- 172.21.15.57:0/4071709668 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a5015a260 con 0x7f7a500c3e40 2023-10-30T14:35:18.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.544+0000 7f7a5631d640 1 -- 172.21.15.57:0/4071709668 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a5004e170 con 0x7f7a500c3e40 2023-10-30T14:35:18.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.544+0000 7f7a4cff9640 1 -- 172.21.15.57:0/4071709668 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7a40004b70 con 0x7f7a500c3e40 2023-10-30T14:35:18.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.548+0000 7f7a4cff9640 1 --2- 172.21.15.57:0/4071709668 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7a3c0664a0 0x7f7a3c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:18.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.548+0000 7f7a4cff9640 1 -- 172.21.15.57:0/4071709668 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f7a40083bc0 con 0x7f7a500c3e40 2023-10-30T14:35:18.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.548+0000 7f7a4f7fe640 1 --2- 172.21.15.57:0/4071709668 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7a3c0664a0 0x7f7a3c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:18.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.548+0000 7f7a4f7fe640 1 --2- 172.21.15.57:0/4071709668 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7a3c0664a0 0x7f7a3c068960 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7a440b2000 tx=0x7f7a44009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:18.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.552+0000 7f7a4cff9640 1 -- 172.21.15.57:0/4071709668 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7a40050b70 con 0x7f7a500c3e40 2023-10-30T14:35:18.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.740+0000 7f7a5631d640 1 -- 172.21.15.57:0/4071709668 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a50070300 con 0x7f7a500c3e40 2023-10-30T14:35:18.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.740+0000 7f7a4cff9640 1 -- 172.21.15.57:0/4071709668 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6535 (secure 0 0 0) 0x7f7a40054190 con 0x7f7a500c3e40 2023-10-30T14:35:18.743 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:18.743 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6747,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:34:12.961116+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:active","state_seq":40,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:18.744 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 119 2023-10-30T14:35:18.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.740+0000 7f7a5631d640 1 -- 172.21.15.57:0/4071709668 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7a3c0664a0 msgr2=0x7f7a3c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:18.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.740+0000 7f7a5631d640 1 --2- 172.21.15.57:0/4071709668 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7a3c0664a0 0x7f7a3c068960 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7a440b2000 tx=0x7f7a44009040 comp rx=0 tx=0).stop 2023-10-30T14:35:18.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.740+0000 7f7a5631d640 1 -- 172.21.15.57:0/4071709668 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a500c3e40 msgr2=0x7f7a501528a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:18.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.740+0000 7f7a5631d640 1 --2- 172.21.15.57:0/4071709668 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a500c3e40 0x7f7a501528a0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f7a4000c460 tx=0x7f7a4000c860 comp rx=0 tx=0).stop 2023-10-30T14:35:18.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.740+0000 7f7a5631d640 1 -- 172.21.15.57:0/4071709668 shutdown_connections 2023-10-30T14:35:18.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.740+0000 7f7a5631d640 1 --2- 172.21.15.57:0/4071709668 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7a3c0664a0 0x7f7a3c068960 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:18.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.740+0000 7f7a5631d640 1 --2- 172.21.15.57:0/4071709668 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a50150830 0x7f7a5014b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:18.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.740+0000 7f7a5631d640 1 --2- 172.21.15.57:0/4071709668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a500cb7d0 0x7f7a5014d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:18.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.740+0000 7f7a5631d640 1 --2- 172.21.15.57:0/4071709668 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a500c3e40 0x7f7a501528a0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:18.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.744+0000 7f7a5631d640 1 -- 172.21.15.57:0/4071709668 >> 172.21.15.57:0/4071709668 conn(0x7f7a500cac70 msgr2=0x7f7a500c65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:18.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.744+0000 7f7a5631d640 1 -- 172.21.15.57:0/4071709668 shutdown_connections 2023-10-30T14:35:18.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:18.744+0000 7f7a5631d640 1 -- 172.21.15.57:0/4071709668 wait complete. 2023-10-30T14:35:18.764 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:35:20.764 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:20.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.904+0000 7fc80aa46640 1 Processor -- start 2023-10-30T14:35:20.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.904+0000 7fc80aa46640 1 -- start start 2023-10-30T14:35:20.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.908+0000 7fc80aa46640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8041537f0 0x7fc8040701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:20.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.908+0000 7fc80aa46640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc80415a2e0 0x7fc8040ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:20.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.908+0000 7fc80aa46640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc8040cd4a0 0x7fc80406b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:20.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.908+0000 7fc80aa46640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8040c3ab0 con 0x7fc80415a2e0 2023-10-30T14:35:20.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.908+0000 7fc80aa46640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8040706e0 con 0x7fc8040cd4a0 2023-10-30T14:35:20.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.908+0000 7fc80aa46640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8040c2aa0 con 0x7fc8041537f0 2023-10-30T14:35:20.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.908+0000 7fc8037fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc80415a2e0 0x7fc8040ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:20.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.908+0000 7fc8037fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc80415a2e0 0x7fc8040ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43816/0 (socket says 172.21.15.57:43816) 2023-10-30T14:35:20.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.908+0000 7fc8037fe640 1 -- 172.21.15.57:0/3948885921 learned_addr learned my addr 172.21.15.57:0/3948885921 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:20.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.908+0000 7fc803fff640 1 --2- 172.21.15.57:0/3948885921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8041537f0 0x7fc8040701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:20.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.908+0000 7fc808fe3640 1 --2- 172.21.15.57:0/3948885921 >> v2:172.21.15.88:3300/0 conn(0x7fc8040cd4a0 0x7fc80406b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:20.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.908+0000 7fc8037fe640 1 -- 172.21.15.57:0/3948885921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8041537f0 msgr2=0x7fc8040701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:20.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.908+0000 7fc8037fe640 1 --2- 172.21.15.57:0/3948885921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8041537f0 0x7fc8040701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:20.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.908+0000 7fc8037fe640 1 -- 172.21.15.57:0/3948885921 >> v2:172.21.15.88:3300/0 conn(0x7fc8040cd4a0 msgr2=0x7fc80406b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:20.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.908+0000 7fc8037fe640 1 --2- 172.21.15.57:0/3948885921 >> v2:172.21.15.88:3300/0 conn(0x7fc8040cd4a0 0x7fc80406b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:20.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.908+0000 7fc8037fe640 1 -- 172.21.15.57:0/3948885921 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8040708e0 con 0x7fc80415a2e0 2023-10-30T14:35:20.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.908+0000 7fc808fe3640 1 --2- 172.21.15.57:0/3948885921 >> v2:172.21.15.88:3300/0 conn(0x7fc8040cd4a0 0x7fc80406b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:20.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc8037fe640 1 --2- 172.21.15.57:0/3948885921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc80415a2e0 0x7fc8040ccf60 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fc804071f60 tx=0x7fc7f801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e66e0f06ee2b9d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:20.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc802ffd640 1 -- 172.21.15.57:0/3948885921 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f80a89b0 con 0x7fc80415a2e0 2023-10-30T14:35:20.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc802ffd640 1 -- 172.21.15.57:0/3948885921 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f80a8b10 con 0x7fc80415a2e0 2023-10-30T14:35:20.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc802ffd640 1 -- 172.21.15.57:0/3948885921 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f80a8ea0 con 0x7fc80415a2e0 2023-10-30T14:35:20.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc80aa46640 1 -- 172.21.15.57:0/3948885921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc80415a2e0 msgr2=0x7fc8040ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:20.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc80aa46640 1 --2- 172.21.15.57:0/3948885921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc80415a2e0 0x7fc8040ccf60 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fc804071f60 tx=0x7fc7f801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:35:20.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc80aa46640 1 -- 172.21.15.57:0/3948885921 shutdown_connections 2023-10-30T14:35:20.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc80aa46640 1 --2- 172.21.15.57:0/3948885921 >> v2:172.21.15.88:3300/0 conn(0x7fc8040cd4a0 0x7fc80406b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:20.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc80aa46640 1 --2- 172.21.15.57:0/3948885921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc80415a2e0 0x7fc8040ccf60 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:20.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc80aa46640 1 --2- 172.21.15.57:0/3948885921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8041537f0 0x7fc8040701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:20.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc80aa46640 1 -- 172.21.15.57:0/3948885921 >> 172.21.15.57:0/3948885921 conn(0x7fc804071fb0 msgr2=0x7fc8040723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:20.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc80aa46640 1 -- 172.21.15.57:0/3948885921 shutdown_connections 2023-10-30T14:35:20.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc80aa46640 1 -- 172.21.15.57:0/3948885921 wait complete. 2023-10-30T14:35:20.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc80aa46640 1 Processor -- start 2023-10-30T14:35:20.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc80aa46640 1 -- start start 2023-10-30T14:35:20.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc80aa46640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8040cd4a0 0x7fc8041e2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:20.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc80aa46640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8041537f0 0x7fc8040cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:20.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc80aa46640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc80415a2e0 0x7fc8040c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:20.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc80aa46640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc804155510 con 0x7fc8040cd4a0 2023-10-30T14:35:20.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc80aa46640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc804156090 con 0x7fc8041537f0 2023-10-30T14:35:20.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.912+0000 7fc80aa46640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8041f2f00 con 0x7fc80415a2e0 2023-10-30T14:35:20.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.916+0000 7fc803fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8040cd4a0 0x7fc8041e2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:20.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.916+0000 7fc803fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8040cd4a0 0x7fc8041e2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43820/0 (socket says 172.21.15.57:43820) 2023-10-30T14:35:20.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.916+0000 7fc803fff640 1 -- 172.21.15.57:0/2919285572 learned_addr learned my addr 172.21.15.57:0/2919285572 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:20.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.916+0000 7fc8037fe640 1 --2- 172.21.15.57:0/2919285572 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8041537f0 0x7fc8040cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:20.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.916+0000 7fc808fe3640 1 --2- 172.21.15.57:0/2919285572 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc80415a2e0 0x7fc8040c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:20.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.916+0000 7fc803fff640 1 -- 172.21.15.57:0/2919285572 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc80415a2e0 msgr2=0x7fc8040c42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:20.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.916+0000 7fc803fff640 1 --2- 172.21.15.57:0/2919285572 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc80415a2e0 0x7fc8040c42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:20.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.916+0000 7fc803fff640 1 -- 172.21.15.57:0/2919285572 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8041537f0 msgr2=0x7fc8040cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:20.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.916+0000 7fc803fff640 1 --2- 172.21.15.57:0/2919285572 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8041537f0 0x7fc8040cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:20.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.916+0000 7fc803fff640 1 -- 172.21.15.57:0/2919285572 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f80094d0 con 0x7fc8040cd4a0 2023-10-30T14:35:20.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.916+0000 7fc803fff640 1 --2- 172.21.15.57:0/2919285572 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8040cd4a0 0x7fc8041e2cb0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fc7f400e930 tx=0x7fc7f400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:20.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.916+0000 7fc800ff9640 1 -- 172.21.15.57:0/2919285572 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f4009690 con 0x7fc8040cd4a0 2023-10-30T14:35:20.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.916+0000 7fc800ff9640 1 -- 172.21.15.57:0/2919285572 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f400ca00 con 0x7fc8040cd4a0 2023-10-30T14:35:20.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.916+0000 7fc800ff9640 1 -- 172.21.15.57:0/2919285572 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f400cd70 con 0x7fc8040cd4a0 2023-10-30T14:35:20.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.916+0000 7fc80aa46640 1 -- 172.21.15.57:0/2919285572 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc804155760 con 0x7fc8040cd4a0 2023-10-30T14:35:20.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.916+0000 7fc80aa46640 1 -- 172.21.15.57:0/2919285572 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc804155d20 con 0x7fc8040cd4a0 2023-10-30T14:35:20.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.916+0000 7fc80aa46640 1 -- 172.21.15.57:0/2919285572 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc80404e170 con 0x7fc8040cd4a0 2023-10-30T14:35:20.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.916+0000 7fc800ff9640 1 -- 172.21.15.57:0/2919285572 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc7f4013470 con 0x7fc8040cd4a0 2023-10-30T14:35:20.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.920+0000 7fc800ff9640 1 --2- 172.21.15.57:0/2919285572 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc7f00664d0 0x7fc7f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:20.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.920+0000 7fc800ff9640 1 -- 172.21.15.57:0/2919285572 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7fc7f40844b0 con 0x7fc8040cd4a0 2023-10-30T14:35:20.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.920+0000 7fc8037fe640 1 --2- 172.21.15.57:0/2919285572 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc7f00664d0 0x7fc7f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:20.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.920+0000 7fc8037fe640 1 --2- 172.21.15.57:0/2919285572 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc7f00664d0 0x7fc7f0068990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80b2000 tx=0x7fc7f80aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:20.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:20.924+0000 7fc800ff9640 1 -- 172.21.15.57:0/2919285572 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc7f4051660 con 0x7fc8040cd4a0 2023-10-30T14:35:21.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:21.112+0000 7fc80aa46640 1 -- 172.21.15.57:0/2919285572 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8040734c0 con 0x7fc8040cd4a0 2023-10-30T14:35:21.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:21.112+0000 7fc800ff9640 1 -- 172.21.15.57:0/2919285572 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6535 (secure 0 0 0) 0x7fc7f4054c80 con 0x7fc8040cd4a0 2023-10-30T14:35:21.115 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:21.115 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6747,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:34:12.961116+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:active","state_seq":40,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:21.115 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 119 2023-10-30T14:35:21.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:21.112+0000 7fc80aa46640 1 -- 172.21.15.57:0/2919285572 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc7f00664d0 msgr2=0x7fc7f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:21.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:21.112+0000 7fc80aa46640 1 --2- 172.21.15.57:0/2919285572 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc7f00664d0 0x7fc7f0068990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80b2000 tx=0x7fc7f80aa070 comp rx=0 tx=0).stop 2023-10-30T14:35:21.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:21.112+0000 7fc80aa46640 1 -- 172.21.15.57:0/2919285572 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8040cd4a0 msgr2=0x7fc8041e2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:21.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:21.112+0000 7fc80aa46640 1 --2- 172.21.15.57:0/2919285572 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8040cd4a0 0x7fc8041e2cb0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fc7f400e930 tx=0x7fc7f400edf0 comp rx=0 tx=0).stop 2023-10-30T14:35:21.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:21.112+0000 7fc80aa46640 1 -- 172.21.15.57:0/2919285572 shutdown_connections 2023-10-30T14:35:21.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:21.112+0000 7fc80aa46640 1 --2- 172.21.15.57:0/2919285572 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc7f00664d0 0x7fc7f0068990 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:21.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:21.112+0000 7fc80aa46640 1 --2- 172.21.15.57:0/2919285572 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc80415a2e0 0x7fc8040c42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:21.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:21.112+0000 7fc80aa46640 1 --2- 172.21.15.57:0/2919285572 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8041537f0 0x7fc8040cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:21.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:21.112+0000 7fc80aa46640 1 --2- 172.21.15.57:0/2919285572 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8040cd4a0 0x7fc8041e2cb0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:21.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:21.112+0000 7fc80aa46640 1 -- 172.21.15.57:0/2919285572 >> 172.21.15.57:0/2919285572 conn(0x7fc804071fb0 msgr2=0x7fc804072390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:21.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:21.112+0000 7fc80aa46640 1 -- 172.21.15.57:0/2919285572 shutdown_connections 2023-10-30T14:35:21.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:21.112+0000 7fc80aa46640 1 -- 172.21.15.57:0/2919285572 wait complete. 2023-10-30T14:35:21.136 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:35:23.136 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:23.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.280+0000 7f2c3ba46640 1 Processor -- start 2023-10-30T14:35:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.280+0000 7f2c3ba46640 1 -- start start 2023-10-30T14:35:23.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.280+0000 7f2c3ba46640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c341537f0 0x7f2c340701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:23.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.280+0000 7f2c3ba46640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2c3406b2f0 0x7f2c3406b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:23.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.280+0000 7f2c3ba46640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c340c2aa0 con 0x7f2c3415a2e0 2023-10-30T14:35:23.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.280+0000 7f2c3ba46640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c340c3ab0 con 0x7f2c3406b2f0 2023-10-30T14:35:23.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.280+0000 7f2c3ba46640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c340706e0 con 0x7f2c341537f0 2023-10-30T14:35:23.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.280+0000 7f2c38fe1640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f2c3415a2e0 0x7f2c340ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:39384/0 (socket says 172.21.15.57:39384) 2023-10-30T14:35:23.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.280+0000 7f2c38fe1640 1 -- 172.21.15.57:0/3324233241 learned_addr learned my addr 172.21.15.57:0/3324233241 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:35:23.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.280+0000 7f2c397e2640 1 --2- 172.21.15.57:0/3324233241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c341537f0 0x7f2c340701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:23.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.280+0000 7f2c39fe3640 1 --2- 172.21.15.57:0/3324233241 >> v2:172.21.15.88:3300/0 conn(0x7f2c3406b2f0 0x7f2c3406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:23.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.280+0000 7f2c2bfff640 1 -- 172.21.15.57:0/3324233241 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180358980 0 0) 0x7f2c340c2aa0 con 0x7f2c3415a2e0 2023-10-30T14:35:23.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c2bfff640 1 -- 172.21.15.57:0/3324233241 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c20003530 con 0x7f2c3415a2e0 2023-10-30T14:35:23.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c2bfff640 1 -- 172.21.15.57:0/3324233241 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 375137895 0 0) 0x7f2c20003530 con 0x7f2c3415a2e0 2023-10-30T14:35:23.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c2bfff640 1 -- 172.21.15.57:0/3324233241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c341537f0 msgr2=0x7f2c340701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:23.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c2bfff640 1 --2- 172.21.15.57:0/3324233241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c341537f0 0x7f2c340701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:23.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c2bfff640 1 -- 172.21.15.57:0/3324233241 >> v2:172.21.15.88:3300/0 conn(0x7f2c3406b2f0 msgr2=0x7f2c3406b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:23.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c2bfff640 1 --2- 172.21.15.57:0/3324233241 >> v2:172.21.15.88:3300/0 conn(0x7f2c3406b2f0 0x7f2c3406b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:23.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c2bfff640 1 -- 172.21.15.57:0/3324233241 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3406bca0 con 0x7f2c3415a2e0 2023-10-30T14:35:23.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c2bfff640 1 -- 172.21.15.57:0/3324233241 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f2c24001680 con 0x7f2c3415a2e0 2023-10-30T14:35:23.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c2bfff640 1 -- 172.21.15.57:0/3324233241 >> v1:172.21.15.57:6789/0 conn(0x7f2c3415a2e0 legacy=0x7f2c340ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:23.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c2bfff640 1 --2- 172.21.15.57:0/3324233241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c20003980 0x7f2c20003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:23.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c2bfff640 1 --2- 172.21.15.57:0/3324233241 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c200047d0 0x7f2c20004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:23.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c2bfff640 1 --2- 172.21.15.57:0/3324233241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c341537f0 0x7f2c20005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:23.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c2bfff640 1 -- 172.21.15.57:0/3324233241 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c340706e0 con 0x7f2c200047d0 2023-10-30T14:35:23.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c2bfff640 1 -- 172.21.15.57:0/3324233241 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2000b990 con 0x7f2c20003980 2023-10-30T14:35:23.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c2bfff640 1 -- 172.21.15.57:0/3324233241 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2000bad0 con 0x7f2c341537f0 2023-10-30T14:35:23.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c38fe1640 1 --2- 172.21.15.57:0/3324233241 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c200047d0 0x7f2c20004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:23.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c397e2640 1 --2- 172.21.15.57:0/3324233241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c20003980 0x7f2c20003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:23.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c39fe3640 1 --2- 172.21.15.57:0/3324233241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c341537f0 0x7f2c20005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:23.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c38fe1640 1 -- 172.21.15.57:0/3324233241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c341537f0 msgr2=0x7f2c20005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:23.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c38fe1640 1 --2- 172.21.15.57:0/3324233241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c341537f0 0x7f2c20005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:23.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c38fe1640 1 -- 172.21.15.57:0/3324233241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c20003980 msgr2=0x7f2c20003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:23.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c38fe1640 1 --2- 172.21.15.57:0/3324233241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c20003980 0x7f2c20003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:23.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c38fe1640 1 -- 172.21.15.57:0/3324233241 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c20003530 con 0x7f2c200047d0 2023-10-30T14:35:23.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c397e2640 1 --2- 172.21.15.57:0/3324233241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c20003980 0x7f2c20003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:23.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.284+0000 7f2c38fe1640 1 --2- 172.21.15.57:0/3324233241 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c200047d0 0x7f2c20004c50 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f2c2400c1c0 tx=0x7f2c2401cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=375b31d12adb5724 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:23.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.288+0000 7f2c2bfff640 1 -- 172.21.15.57:0/3324233241 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c240058e0 con 0x7f2c200047d0 2023-10-30T14:35:23.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.288+0000 7f2c2bfff640 1 -- 172.21.15.57:0/3324233241 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c24005a60 con 0x7f2c200047d0 2023-10-30T14:35:23.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.288+0000 7f2c2bfff640 1 -- 172.21.15.57:0/3324233241 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c24005e10 con 0x7f2c200047d0 2023-10-30T14:35:23.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.288+0000 7f2c3ba46640 1 -- 172.21.15.57:0/3324233241 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c200047d0 msgr2=0x7f2c20004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:23.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.288+0000 7f2c3ba46640 1 --2- 172.21.15.57:0/3324233241 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c200047d0 0x7f2c20004c50 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f2c2400c1c0 tx=0x7f2c2401cc40 comp rx=0 tx=0).stop 2023-10-30T14:35:23.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.288+0000 7f2c39fe3640 1 -- 172.21.15.57:0/3324233241 reap_dead start 2023-10-30T14:35:23.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.288+0000 7f2c3ba46640 1 -- 172.21.15.57:0/3324233241 shutdown_connections 2023-10-30T14:35:23.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.288+0000 7f2c3ba46640 1 -- 172.21.15.57:0/3324233241 >> 172.21.15.57:0/3324233241 conn(0x7f2c34071fb0 msgr2=0x7f2c340723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:23.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.288+0000 7f2c3ba46640 1 -- 172.21.15.57:0/3324233241 shutdown_connections 2023-10-30T14:35:23.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.288+0000 7f2c3ba46640 1 -- 172.21.15.57:0/3324233241 wait complete. 2023-10-30T14:35:23.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.288+0000 7f2c3ba46640 1 Processor -- start 2023-10-30T14:35:23.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.288+0000 7f2c3ba46640 1 -- start start 2023-10-30T14:35:23.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c3ba46640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c200047d0 0x7f2c340b2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:23.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c3ba46640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c3415d2e0 0x7f2c340b2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:23.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c3ba46640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c340c42d0 0x7f2c340c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:23.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c3ba46640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c341ee930 con 0x7f2c3415d2e0 2023-10-30T14:35:23.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c3ba46640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c341597d0 con 0x7f2c340c42d0 2023-10-30T14:35:23.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c3ba46640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c341eeb20 con 0x7f2c200047d0 2023-10-30T14:35:23.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c38fe1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c3415d2e0 0x7f2c340b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:23.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c38fe1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c3415d2e0 0x7f2c340b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43844/0 (socket says 172.21.15.57:43844) 2023-10-30T14:35:23.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c38fe1640 1 -- 172.21.15.57:0/59183331 learned_addr learned my addr 172.21.15.57:0/59183331 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:23.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c397e2640 1 --2- 172.21.15.57:0/59183331 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c200047d0 0x7f2c340b2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:23.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c39fe3640 1 --2- 172.21.15.57:0/59183331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c340c42d0 0x7f2c340c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:23.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c38fe1640 1 -- 172.21.15.57:0/59183331 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c200047d0 msgr2=0x7f2c340b2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:23.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c38fe1640 1 --2- 172.21.15.57:0/59183331 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c200047d0 0x7f2c340b2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:23.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c38fe1640 1 -- 172.21.15.57:0/59183331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c340c42d0 msgr2=0x7f2c340c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:23.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c38fe1640 1 --2- 172.21.15.57:0/59183331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c340c42d0 0x7f2c340c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:23.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c38fe1640 1 -- 172.21.15.57:0/59183331 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c341eedc0 con 0x7f2c3415d2e0 2023-10-30T14:35:23.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c39fe3640 1 --2- 172.21.15.57:0/59183331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c340c42d0 0x7f2c340c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:23.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c38fe1640 1 --2- 172.21.15.57:0/59183331 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c3415d2e0 0x7f2c340b2950 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f2c240ab040 tx=0x7f2c20005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:23.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c29ffb640 1 -- 172.21.15.57:0/59183331 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c240a5c70 con 0x7f2c3415d2e0 2023-10-30T14:35:23.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c29ffb640 1 -- 172.21.15.57:0/59183331 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c240a5dd0 con 0x7f2c3415d2e0 2023-10-30T14:35:23.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c3ba46640 1 -- 172.21.15.57:0/59183331 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c341ef0a0 con 0x7f2c3415d2e0 2023-10-30T14:35:23.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c29ffb640 1 -- 172.21.15.57:0/59183331 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c24004a30 con 0x7f2c3415d2e0 2023-10-30T14:35:23.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c3ba46640 1 -- 172.21.15.57:0/59183331 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c341ef660 con 0x7f2c3415d2e0 2023-10-30T14:35:23.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.292+0000 7f2c3ba46640 1 -- 172.21.15.57:0/59183331 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c3404e170 con 0x7f2c3415d2e0 2023-10-30T14:35:23.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.296+0000 7f2c29ffb640 1 -- 172.21.15.57:0/59183331 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2c24004b90 con 0x7f2c3415d2e0 2023-10-30T14:35:23.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.296+0000 7f2c29ffb640 1 --2- 172.21.15.57:0/59183331 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2c180664a0 0x7f2c18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:23.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.296+0000 7f2c29ffb640 1 -- 172.21.15.57:0/59183331 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f2c2411ddb0 con 0x7f2c3415d2e0 2023-10-30T14:35:23.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.296+0000 7f2c397e2640 1 --2- 172.21.15.57:0/59183331 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2c180664a0 0x7f2c18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:23.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.296+0000 7f2c397e2640 1 --2- 172.21.15.57:0/59183331 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2c180664a0 0x7f2c18068960 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2c30006f90 tx=0x7f2c30008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:23.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.300+0000 7f2c29ffb640 1 -- 172.21.15.57:0/59183331 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2c24122070 con 0x7f2c3415d2e0 2023-10-30T14:35:23.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.516+0000 7f2c3ba46640 1 -- 172.21.15.57:0/59183331 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c34159ae0 con 0x7f2c3415d2e0 2023-10-30T14:35:23.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.516+0000 7f2c29ffb640 1 -- 172.21.15.57:0/59183331 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6535 (secure 0 0 0) 0x7f2c240ead60 con 0x7f2c3415d2e0 2023-10-30T14:35:23.520 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:23.520 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6747,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:34:12.961116+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:active","state_seq":40,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:23.521 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 119 2023-10-30T14:35:23.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.520+0000 7f2c3ba46640 1 -- 172.21.15.57:0/59183331 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2c180664a0 msgr2=0x7f2c18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:23.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.520+0000 7f2c3ba46640 1 --2- 172.21.15.57:0/59183331 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2c180664a0 0x7f2c18068960 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2c30006f90 tx=0x7f2c30008040 comp rx=0 tx=0).stop 2023-10-30T14:35:23.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.520+0000 7f2c3ba46640 1 -- 172.21.15.57:0/59183331 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c3415d2e0 msgr2=0x7f2c340b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:23.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.520+0000 7f2c3ba46640 1 --2- 172.21.15.57:0/59183331 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c3415d2e0 0x7f2c340b2950 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f2c240ab040 tx=0x7f2c20005800 comp rx=0 tx=0).stop 2023-10-30T14:35:23.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.520+0000 7f2c3ba46640 1 -- 172.21.15.57:0/59183331 shutdown_connections 2023-10-30T14:35:23.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.520+0000 7f2c3ba46640 1 --2- 172.21.15.57:0/59183331 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2c180664a0 0x7f2c18068960 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:23.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.520+0000 7f2c3ba46640 1 --2- 172.21.15.57:0/59183331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c340c42d0 0x7f2c340c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:23.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.520+0000 7f2c3ba46640 1 --2- 172.21.15.57:0/59183331 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c3415d2e0 0x7f2c340b2950 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:23.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.520+0000 7f2c3ba46640 1 --2- 172.21.15.57:0/59183331 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c200047d0 0x7f2c340b2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:23.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.520+0000 7f2c3ba46640 1 -- 172.21.15.57:0/59183331 >> 172.21.15.57:0/59183331 conn(0x7f2c34071fb0 msgr2=0x7f2c3414e990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:23.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.520+0000 7f2c3ba46640 1 -- 172.21.15.57:0/59183331 shutdown_connections 2023-10-30T14:35:23.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:23.520+0000 7f2c3ba46640 1 -- 172.21.15.57:0/59183331 wait complete. 2023-10-30T14:35:23.541 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:35:25.542 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:25.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.696+0000 7f2baefa1640 1 Processor -- start 2023-10-30T14:35:25.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.696+0000 7f2baefa1640 1 -- start start 2023-10-30T14:35:25.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.696+0000 7f2baefa1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2ba80c4a10 0x7f2ba815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:25.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.696+0000 7f2baefa1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2ba80c42a0 0x7f2ba80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:25.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.696+0000 7f2baefa1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba8159f20 con 0x7f2ba8153c80 2023-10-30T14:35:25.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.696+0000 7f2baefa1640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ba815dec0 con 0x7f2ba80c42a0 2023-10-30T14:35:25.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.696+0000 7f2baefa1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ba81599a0 con 0x7f2ba80c4a10 2023-10-30T14:35:25.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.696+0000 7f2bad53e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2ba80c42a0 0x7f2ba80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:25.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.696+0000 7f2bad53e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2ba80c42a0 0x7f2ba80ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44878/0 (socket says 172.21.15.57:44878) 2023-10-30T14:35:25.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.696+0000 7f2bad53e640 1 -- 172.21.15.57:0/3937194544 learned_addr learned my addr 172.21.15.57:0/3937194544 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:25.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.696+0000 7f2ba7fff640 1 --2- 172.21.15.57:0/3937194544 >> v2:172.21.15.88:3300/0 conn(0x7f2ba80c4a10 0x7f2ba815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:25.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.696+0000 7f2bad53e640 1 -- 172.21.15.57:0/3937194544 >> v2:172.21.15.88:3300/0 conn(0x7f2ba80c4a10 msgr2=0x7f2ba815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:25.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.696+0000 7f2bad53e640 1 --2- 172.21.15.57:0/3937194544 >> v2:172.21.15.88:3300/0 conn(0x7f2ba80c4a10 0x7f2ba815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:25.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.696+0000 7f2bad53e640 1 -- 172.21.15.57:0/3937194544 >> v1:172.21.15.88:6789/0 conn(0x7f2ba8153c80 legacy=0x7f2ba815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:25.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.696+0000 7f2bad53e640 1 -- 172.21.15.57:0/3937194544 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba815d620 con 0x7f2ba80c42a0 2023-10-30T14:35:25.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.700+0000 7f2bad53e640 1 --2- 172.21.15.57:0/3937194544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ba80c42a0 0x7f2ba80ca1e0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f2ba806be50 tx=0x7f2ba001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0dea4634e7dd600 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:25.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.700+0000 7f2ba77fe640 1 -- 172.21.15.57:0/3937194544 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba00af400 con 0x7f2ba80c42a0 2023-10-30T14:35:25.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.700+0000 7f2ba77fe640 1 -- 172.21.15.57:0/3937194544 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba00ab050 con 0x7f2ba80c42a0 2023-10-30T14:35:25.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.700+0000 7f2ba77fe640 1 -- 172.21.15.57:0/3937194544 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba00af780 con 0x7f2ba80c42a0 2023-10-30T14:35:25.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.700+0000 7f2baefa1640 1 -- 172.21.15.57:0/3937194544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ba80c42a0 msgr2=0x7f2ba80ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:25.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.700+0000 7f2baefa1640 1 --2- 172.21.15.57:0/3937194544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ba80c42a0 0x7f2ba80ca1e0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f2ba806be50 tx=0x7f2ba001f1c0 comp rx=0 tx=0).stop 2023-10-30T14:35:25.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.700+0000 7f2baefa1640 1 -- 172.21.15.57:0/3937194544 shutdown_connections 2023-10-30T14:35:25.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.700+0000 7f2baefa1640 1 --2- 172.21.15.57:0/3937194544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ba80c42a0 0x7f2ba80ca1e0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:25.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.700+0000 7f2baefa1640 1 --2- 172.21.15.57:0/3937194544 >> v2:172.21.15.88:3300/0 conn(0x7f2ba80c4a10 0x7f2ba815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:25.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.700+0000 7f2baefa1640 1 -- 172.21.15.57:0/3937194544 >> 172.21.15.57:0/3937194544 conn(0x7f2ba80715c0 msgr2=0x7f2ba80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:25.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.700+0000 7f2baefa1640 1 -- 172.21.15.57:0/3937194544 shutdown_connections 2023-10-30T14:35:25.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.700+0000 7f2baefa1640 1 -- 172.21.15.57:0/3937194544 wait complete. 2023-10-30T14:35:25.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2baefa1640 1 Processor -- start 2023-10-30T14:35:25.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2baefa1640 1 -- start start 2023-10-30T14:35:25.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2baefa1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba80c42a0 0x7f2ba81f2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:25.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2baefa1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba80c4a10 0x7f2ba80ce000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:25.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2baefa1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ba8153c80 0x7f2ba80ce540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:25.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2baefa1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba81f2f80 con 0x7f2ba8153c80 2023-10-30T14:35:25.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2baefa1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba81f2ca0 con 0x7f2ba80c4a10 2023-10-30T14:35:25.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2baefa1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba80cfa30 con 0x7f2ba80c42a0 2023-10-30T14:35:25.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2bad53e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ba8153c80 0x7f2ba80ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:25.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2bad53e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ba8153c80 0x7f2ba80ce540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44894/0 (socket says 172.21.15.57:44894) 2023-10-30T14:35:25.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2bad53e640 1 -- 172.21.15.57:0/3794083770 learned_addr learned my addr 172.21.15.57:0/3794083770 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:25.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2ba7fff640 1 --2- 172.21.15.57:0/3794083770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba80c4a10 0x7f2ba80ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:25.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2bacd3d640 1 --2- 172.21.15.57:0/3794083770 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba80c42a0 0x7f2ba81f2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:25.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2bad53e640 1 -- 172.21.15.57:0/3794083770 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba80c42a0 msgr2=0x7f2ba81f2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:25.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2bad53e640 1 --2- 172.21.15.57:0/3794083770 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba80c42a0 0x7f2ba81f2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:25.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2bad53e640 1 -- 172.21.15.57:0/3794083770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba80c4a10 msgr2=0x7f2ba80ce000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:25.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2bad53e640 1 --2- 172.21.15.57:0/3794083770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba80c4a10 0x7f2ba80ce000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:25.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2bad53e640 1 -- 172.21.15.57:0/3794083770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba0009b30 con 0x7f2ba8153c80 2023-10-30T14:35:25.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2bad53e640 1 --2- 172.21.15.57:0/3794083770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ba8153c80 0x7f2ba80ce540 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f2ba00b2040 tx=0x7f2ba001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:25.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2ba57fa640 1 -- 172.21.15.57:0/3794083770 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba00afd10 con 0x7f2ba8153c80 2023-10-30T14:35:25.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2ba57fa640 1 -- 172.21.15.57:0/3794083770 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba00afe90 con 0x7f2ba8153c80 2023-10-30T14:35:25.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2ba57fa640 1 -- 172.21.15.57:0/3794083770 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba00b8640 con 0x7f2ba8153c80 2023-10-30T14:35:25.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2baefa1640 1 -- 172.21.15.57:0/3794083770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba80cfd10 con 0x7f2ba8153c80 2023-10-30T14:35:25.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.704+0000 7f2baefa1640 1 -- 172.21.15.57:0/3794083770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba80cdae0 con 0x7f2ba8153c80 2023-10-30T14:35:25.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.708+0000 7f2baefa1640 1 -- 172.21.15.57:0/3794083770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba804e170 con 0x7f2ba8153c80 2023-10-30T14:35:25.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.708+0000 7f2ba57fa640 1 -- 172.21.15.57:0/3794083770 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2ba00b6040 con 0x7f2ba8153c80 2023-10-30T14:35:25.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.708+0000 7f2ba57fa640 1 --2- 172.21.15.57:0/3794083770 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b8c0664a0 0x7f2b8c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:25.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.708+0000 7f2ba57fa640 1 -- 172.21.15.57:0/3794083770 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f2ba01222b0 con 0x7f2ba8153c80 2023-10-30T14:35:25.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.708+0000 7f2bacd3d640 1 --2- 172.21.15.57:0/3794083770 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b8c0664a0 0x7f2b8c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:25.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.708+0000 7f2bacd3d640 1 --2- 172.21.15.57:0/3794083770 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b8c0664a0 0x7f2b8c068960 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c000f70 tx=0x7f2b9c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:25.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.712+0000 7f2ba57fa640 1 -- 172.21.15.57:0/3794083770 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2ba00ef1d0 con 0x7f2ba8153c80 2023-10-30T14:35:25.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.900+0000 7f2baefa1640 1 -- 172.21.15.57:0/3794083770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ba80cad40 con 0x7f2ba8153c80 2023-10-30T14:35:25.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.904+0000 7f2ba57fa640 1 -- 172.21.15.57:0/3794083770 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6535 (secure 0 0 0) 0x7f2ba0009510 con 0x7f2ba8153c80 2023-10-30T14:35:25.905 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:25.905 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6747,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:34:12.961116+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:active","state_seq":40,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:25.906 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 119 2023-10-30T14:35:25.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.904+0000 7f2baefa1640 1 -- 172.21.15.57:0/3794083770 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b8c0664a0 msgr2=0x7f2b8c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:25.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.904+0000 7f2baefa1640 1 --2- 172.21.15.57:0/3794083770 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b8c0664a0 0x7f2b8c068960 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c000f70 tx=0x7f2b9c009040 comp rx=0 tx=0).stop 2023-10-30T14:35:25.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.904+0000 7f2baefa1640 1 -- 172.21.15.57:0/3794083770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ba8153c80 msgr2=0x7f2ba80ce540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:25.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.904+0000 7f2baefa1640 1 --2- 172.21.15.57:0/3794083770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ba8153c80 0x7f2ba80ce540 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f2ba00b2040 tx=0x7f2ba001bf60 comp rx=0 tx=0).stop 2023-10-30T14:35:25.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.904+0000 7f2baefa1640 1 -- 172.21.15.57:0/3794083770 shutdown_connections 2023-10-30T14:35:25.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.904+0000 7f2baefa1640 1 --2- 172.21.15.57:0/3794083770 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2b8c0664a0 0x7f2b8c068960 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:25.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.904+0000 7f2baefa1640 1 --2- 172.21.15.57:0/3794083770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ba8153c80 0x7f2ba80ce540 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:25.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.904+0000 7f2baefa1640 1 --2- 172.21.15.57:0/3794083770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba80c4a10 0x7f2ba80ce000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:25.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.904+0000 7f2baefa1640 1 --2- 172.21.15.57:0/3794083770 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba80c42a0 0x7f2ba81f2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:25.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.904+0000 7f2baefa1640 1 -- 172.21.15.57:0/3794083770 >> 172.21.15.57:0/3794083770 conn(0x7f2ba80715c0 msgr2=0x7f2ba8151450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:25.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.904+0000 7f2baefa1640 1 -- 172.21.15.57:0/3794083770 shutdown_connections 2023-10-30T14:35:25.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:25.904+0000 7f2baefa1640 1 -- 172.21.15.57:0/3794083770 wait complete. 2023-10-30T14:35:25.926 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:35:27.927 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:28.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.080+0000 7f69710d6640 1 Processor -- start 2023-10-30T14:35:28.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.080+0000 7f69710d6640 1 -- start start 2023-10-30T14:35:28.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.080+0000 7f69710d6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f696c070520 0x7f696c070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:28.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.080+0000 7f69710d6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c0c2310 con 0x7f696c0cb7d0 2023-10-30T14:35:28.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.080+0000 7f69710d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c156710 con 0x7f696c150830 2023-10-30T14:35:28.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.080+0000 7f69710d6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696c06bc10 con 0x7f696c070520 2023-10-30T14:35:28.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.080+0000 7f6963fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f696c0cb7d0 0x7f696c0c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:40566/0 (socket says 172.21.15.57:40566) 2023-10-30T14:35:28.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.080+0000 7f696b59e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f696c070520 0x7f696c070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:28.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.084+0000 7f6963fff640 1 -- 172.21.15.57:0/2547187974 learned_addr learned my addr 172.21.15.57:0/2547187974 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:35:28.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.084+0000 7f696b59e640 1 -- 172.21.15.57:0/2547187974 >> v1:172.21.15.88:6789/0 conn(0x7f696c150830 legacy=0x7f696c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:28.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.084+0000 7f696b59e640 1 -- 172.21.15.57:0/2547187974 >> v1:172.21.15.57:6789/0 conn(0x7f696c0cb7d0 legacy=0x7f696c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:28.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.084+0000 7f696b59e640 1 -- 172.21.15.57:0/2547187974 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c0c4360 con 0x7f696c070520 2023-10-30T14:35:28.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.084+0000 7f696a59c640 1 -- 172.21.15.57:0/2547187974 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568743197 0 0) 0x7f696c0c2310 con 0x7f696c0cb7d0 2023-10-30T14:35:28.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.084+0000 7f696b59e640 1 --2- 172.21.15.57:0/2547187974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c070520 0x7f696c070990 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f6964009d90 tx=0x7f696401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f54f05bc58caebc0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:28.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.084+0000 7f696a59c640 1 -- 172.21.15.57:0/2547187974 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69640af600 con 0x7f696c070520 2023-10-30T14:35:28.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.084+0000 7f696a59c640 1 -- 172.21.15.57:0/2547187974 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69640ab050 con 0x7f696c070520 2023-10-30T14:35:28.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.084+0000 7f696a59c640 1 -- 172.21.15.57:0/2547187974 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69640af980 con 0x7f696c070520 2023-10-30T14:35:28.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f69710d6640 1 -- 172.21.15.57:0/2547187974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c070520 msgr2=0x7f696c070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:28.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f69710d6640 1 --2- 172.21.15.57:0/2547187974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c070520 0x7f696c070990 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f6964009d90 tx=0x7f696401f130 comp rx=0 tx=0).stop 2023-10-30T14:35:28.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f69710d6640 1 -- 172.21.15.57:0/2547187974 shutdown_connections 2023-10-30T14:35:28.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f69710d6640 1 --2- 172.21.15.57:0/2547187974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c070520 0x7f696c070990 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:28.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f69710d6640 1 -- 172.21.15.57:0/2547187974 >> 172.21.15.57:0/2547187974 conn(0x7f696c0cac70 msgr2=0x7f696c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:28.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f69710d6640 1 -- 172.21.15.57:0/2547187974 shutdown_connections 2023-10-30T14:35:28.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f69710d6640 1 -- 172.21.15.57:0/2547187974 wait complete. 2023-10-30T14:35:28.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f69710d6640 1 Processor -- start 2023-10-30T14:35:28.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f69710d6640 1 -- start start 2023-10-30T14:35:28.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f69710d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f696c070520 0x7f696c06bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:28.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f69710d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c0cb7d0 0x7f696c154620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:28.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f69710d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f696c150830 0x7f696c154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:28.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f69710d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c1eeb90 con 0x7f696c0cb7d0 2023-10-30T14:35:28.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f69710d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c0c5800 con 0x7f696c150830 2023-10-30T14:35:28.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f69710d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696c1ef9b0 con 0x7f696c070520 2023-10-30T14:35:28.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f6963fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c0cb7d0 0x7f696c154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:28.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f6963fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c0cb7d0 0x7f696c154620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44922/0 (socket says 172.21.15.57:44922) 2023-10-30T14:35:28.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f6963fff640 1 -- 172.21.15.57:0/3213379998 learned_addr learned my addr 172.21.15.57:0/3213379998 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:28.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f696b59e640 1 --2- 172.21.15.57:0/3213379998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f696c150830 0x7f696c154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:28.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f696ad9d640 1 --2- 172.21.15.57:0/3213379998 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f696c070520 0x7f696c06bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:28.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f6963fff640 1 -- 172.21.15.57:0/3213379998 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f696c070520 msgr2=0x7f696c06bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:28.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f6963fff640 1 --2- 172.21.15.57:0/3213379998 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f696c070520 0x7f696c06bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:28.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f6963fff640 1 -- 172.21.15.57:0/3213379998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f696c150830 msgr2=0x7f696c154b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:28.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f6963fff640 1 --2- 172.21.15.57:0/3213379998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f696c150830 0x7f696c154b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:28.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f6963fff640 1 -- 172.21.15.57:0/3213379998 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6964009a70 con 0x7f696c0cb7d0 2023-10-30T14:35:28.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.088+0000 7f696b59e640 1 --2- 172.21.15.57:0/3213379998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f696c150830 0x7f696c154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:28.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.092+0000 7f6963fff640 1 --2- 172.21.15.57:0/3213379998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c0cb7d0 0x7f696c154620 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f695400c480 tx=0x7f695400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:28.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.092+0000 7f69637fe640 1 -- 172.21.15.57:0/3213379998 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6954005cf0 con 0x7f696c0cb7d0 2023-10-30T14:35:28.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.092+0000 7f69637fe640 1 -- 172.21.15.57:0/3213379998 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6954005e50 con 0x7f696c0cb7d0 2023-10-30T14:35:28.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.092+0000 7f69710d6640 1 -- 172.21.15.57:0/3213379998 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696c1eeda0 con 0x7f696c0cb7d0 2023-10-30T14:35:28.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.092+0000 7f69637fe640 1 -- 172.21.15.57:0/3213379998 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6954007290 con 0x7f696c0cb7d0 2023-10-30T14:35:28.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.092+0000 7f69710d6640 1 -- 172.21.15.57:0/3213379998 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696c1ef360 con 0x7f696c0cb7d0 2023-10-30T14:35:28.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.092+0000 7f69710d6640 1 -- 172.21.15.57:0/3213379998 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696c04e170 con 0x7f696c0cb7d0 2023-10-30T14:35:28.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.092+0000 7f69637fe640 1 -- 172.21.15.57:0/3213379998 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f69540073f0 con 0x7f696c0cb7d0 2023-10-30T14:35:28.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.092+0000 7f69637fe640 1 --2- 172.21.15.57:0/3213379998 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f694c0664a0 0x7f694c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:28.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.092+0000 7f69637fe640 1 -- 172.21.15.57:0/3213379998 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6771+0+0 (secure 0 0 0) 0x7f6954083f70 con 0x7f696c0cb7d0 2023-10-30T14:35:28.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.092+0000 7f696ad9d640 1 --2- 172.21.15.57:0/3213379998 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f694c0664a0 0x7f694c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:28.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.096+0000 7f696ad9d640 1 --2- 172.21.15.57:0/3213379998 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f694c0664a0 0x7f694c068960 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f695c000f70 tx=0x7f695c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:28.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.096+0000 7f69637fe640 1 -- 172.21.15.57:0/3213379998 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f69540b5090 con 0x7f696c0cb7d0 2023-10-30T14:35:28.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.292+0000 7f69710d6640 1 -- 172.21.15.57:0/3213379998 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f696c0c61c0 con 0x7f696c0cb7d0 2023-10-30T14:35:28.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.292+0000 7f69637fe640 1 -- 172.21.15.57:0/3213379998 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6535 (secure 0 0 0) 0x7f6954050f40 con 0x7f696c0cb7d0 2023-10-30T14:35:28.295 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:28.295 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6747,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:34:12.961116+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":5891,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5891":{"gid":5891,"name":"f","rank":1,"incarnation":111,"state":"up:active","state_seq":40,"addr":"172.21.15.88:6839/3604033338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3604033338},{"type":"v1","addr":"172.21.15.88:6839","nonce":3604033338}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:28.295 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 119 2023-10-30T14:35:28.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.292+0000 7f69710d6640 1 -- 172.21.15.57:0/3213379998 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f694c0664a0 msgr2=0x7f694c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:28.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.292+0000 7f69710d6640 1 --2- 172.21.15.57:0/3213379998 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f694c0664a0 0x7f694c068960 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f695c000f70 tx=0x7f695c009040 comp rx=0 tx=0).stop 2023-10-30T14:35:28.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.292+0000 7f69710d6640 1 -- 172.21.15.57:0/3213379998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c0cb7d0 msgr2=0x7f696c154620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:28.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.292+0000 7f69710d6640 1 --2- 172.21.15.57:0/3213379998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c0cb7d0 0x7f696c154620 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f695400c480 tx=0x7f695400c940 comp rx=0 tx=0).stop 2023-10-30T14:35:28.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.292+0000 7f69710d6640 1 -- 172.21.15.57:0/3213379998 shutdown_connections 2023-10-30T14:35:28.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.292+0000 7f69710d6640 1 --2- 172.21.15.57:0/3213379998 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f694c0664a0 0x7f694c068960 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:28.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.292+0000 7f69710d6640 1 --2- 172.21.15.57:0/3213379998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f696c150830 0x7f696c154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:28.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.292+0000 7f69710d6640 1 --2- 172.21.15.57:0/3213379998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c0cb7d0 0x7f696c154620 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:28.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.292+0000 7f69710d6640 1 --2- 172.21.15.57:0/3213379998 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f696c070520 0x7f696c06bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:28.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.292+0000 7f69710d6640 1 -- 172.21.15.57:0/3213379998 >> 172.21.15.57:0/3213379998 conn(0x7f696c0cac70 msgr2=0x7f696c0733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:28.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.292+0000 7f69710d6640 1 -- 172.21.15.57:0/3213379998 shutdown_connections 2023-10-30T14:35:28.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:28.292+0000 7f69710d6640 1 -- 172.21.15.57:0/3213379998 wait complete. 2023-10-30T14:35:28.315 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:35:30.316 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:30.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.468+0000 7f959cb72640 1 Processor -- start 2023-10-30T14:35:30.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.468+0000 7f959cb72640 1 -- start start 2023-10-30T14:35:30.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.468+0000 7f959cb72640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9598153c80 0x7f959815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:30.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.468+0000 7f959cb72640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f95980c4a10 0x7f95980c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:30.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.468+0000 7f959cb72640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9598159f20 con 0x7f959815a3b0 2023-10-30T14:35:30.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.468+0000 7f959cb72640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f959815dec0 con 0x7f95980c4a10 2023-10-30T14:35:30.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.468+0000 7f959cb72640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95981599a0 con 0x7f9598153c80 2023-10-30T14:35:30.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.468+0000 7f958ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f95980c4a10 0x7f95980c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:30.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.468+0000 7f9596d9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f959815a3b0 0x7f95980ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:40568/0 (socket says 172.21.15.57:40568) 2023-10-30T14:35:30.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.468+0000 7f9596d9d640 1 -- 172.21.15.57:0/2539389816 learned_addr learned my addr 172.21.15.57:0/2539389816 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:35:30.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.468+0000 7f959659c640 1 --2- 172.21.15.57:0/2539389816 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9598153c80 0x7f959815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:30.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.468+0000 7f958ffff640 1 -- 172.21.15.57:0/2539389816 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9598153c80 msgr2=0x7f959815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:30.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.468+0000 7f958ffff640 1 --2- 172.21.15.57:0/2539389816 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9598153c80 0x7f959815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:30.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.468+0000 7f958ffff640 1 -- 172.21.15.57:0/2539389816 >> v1:172.21.15.57:6789/0 conn(0x7f959815a3b0 legacy=0x7f95980ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:30.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.468+0000 7f958ffff640 1 -- 172.21.15.57:0/2539389816 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95980c47e0 con 0x7f95980c4a10 2023-10-30T14:35:30.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.472+0000 7f958ffff640 1 --2- 172.21.15.57:0/2539389816 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95980c4a10 0x7f95980c42a0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f95800097c0 tx=0x7f958001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36481c2c977d59 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:30.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.472+0000 7f9595d9b640 1 -- 172.21.15.57:0/2539389816 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95800a8970 con 0x7f95980c4a10 2023-10-30T14:35:30.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.472+0000 7f9595d9b640 1 -- 172.21.15.57:0/2539389816 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95800a8ad0 con 0x7f95980c4a10 2023-10-30T14:35:30.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.472+0000 7f9595d9b640 1 -- 172.21.15.57:0/2539389816 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95800a8e60 con 0x7f95980c4a10 2023-10-30T14:35:30.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.472+0000 7f959cb72640 1 -- 172.21.15.57:0/2539389816 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95980c4a10 msgr2=0x7f95980c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:30.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.472+0000 7f959cb72640 1 --2- 172.21.15.57:0/2539389816 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95980c4a10 0x7f95980c42a0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f95800097c0 tx=0x7f958001ec70 comp rx=0 tx=0).stop 2023-10-30T14:35:30.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.472+0000 7f959cb72640 1 -- 172.21.15.57:0/2539389816 shutdown_connections 2023-10-30T14:35:30.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.472+0000 7f959cb72640 1 --2- 172.21.15.57:0/2539389816 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95980c4a10 0x7f95980c42a0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:30.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.472+0000 7f959cb72640 1 --2- 172.21.15.57:0/2539389816 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9598153c80 0x7f959815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:30.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.472+0000 7f959cb72640 1 -- 172.21.15.57:0/2539389816 >> 172.21.15.57:0/2539389816 conn(0x7f95980715c0 msgr2=0x7f95980719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:30.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.472+0000 7f959cb72640 1 -- 172.21.15.57:0/2539389816 shutdown_connections 2023-10-30T14:35:30.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.472+0000 7f959cb72640 1 -- 172.21.15.57:0/2539389816 wait complete. 2023-10-30T14:35:30.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f959cb72640 1 Processor -- start 2023-10-30T14:35:30.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f959cb72640 1 -- start start 2023-10-30T14:35:30.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f959cb72640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95980c4a10 0x7f95980b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:30.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f959cb72640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9598153c80 0x7f95980b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:30.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f959cb72640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f959815a3b0 0x7f95981f0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:30.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f959cb72640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95980b45b0 con 0x7f95980c4a10 2023-10-30T14:35:30.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f959cb72640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95980b39a0 con 0x7f959815a3b0 2023-10-30T14:35:30.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f959cb72640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95980b26d0 con 0x7f9598153c80 2023-10-30T14:35:30.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f959659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95980c4a10 0x7f95980b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:30.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f959659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95980c4a10 0x7f95980b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44940/0 (socket says 172.21.15.57:44940) 2023-10-30T14:35:30.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f959659c640 1 -- 172.21.15.57:0/64858904 learned_addr learned my addr 172.21.15.57:0/64858904 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:30.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f9596d9d640 1 --2- 172.21.15.57:0/64858904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f959815a3b0 0x7f95981f0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:30.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f958ffff640 1 --2- 172.21.15.57:0/64858904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9598153c80 0x7f95980b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:30.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f959659c640 1 -- 172.21.15.57:0/64858904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9598153c80 msgr2=0x7f95980b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:30.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f959659c640 1 --2- 172.21.15.57:0/64858904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9598153c80 0x7f95980b2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:30.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f959659c640 1 -- 172.21.15.57:0/64858904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f959815a3b0 msgr2=0x7f95981f0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:30.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f959659c640 1 --2- 172.21.15.57:0/64858904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f959815a3b0 0x7f95981f0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:30.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f959659c640 1 -- 172.21.15.57:0/64858904 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9580009420 con 0x7f95980c4a10 2023-10-30T14:35:30.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f959659c640 1 --2- 172.21.15.57:0/64858904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95980c4a10 0x7f95980b3e30 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f958800d800 tx=0x7f958800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:30.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f958effd640 1 -- 172.21.15.57:0/64858904 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958800b6c0 con 0x7f95980c4a10 2023-10-30T14:35:30.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f958effd640 1 -- 172.21.15.57:0/64858904 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958800b820 con 0x7f95980c4a10 2023-10-30T14:35:30.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f959cb72640 1 -- 172.21.15.57:0/64858904 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95980b3070 con 0x7f95980c4a10 2023-10-30T14:35:30.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f958effd640 1 -- 172.21.15.57:0/64858904 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958800bb90 con 0x7f95980c4a10 2023-10-30T14:35:30.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.476+0000 7f959cb72640 1 -- 172.21.15.57:0/64858904 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95980b3630 con 0x7f95980c4a10 2023-10-30T14:35:30.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.480+0000 7f959cb72640 1 -- 172.21.15.57:0/64858904 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f959804e170 con 0x7f95980c4a10 2023-10-30T14:35:30.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.480+0000 7f958effd640 1 -- 172.21.15.57:0/64858904 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f958800bcf0 con 0x7f95980c4a10 2023-10-30T14:35:30.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.480+0000 7f958effd640 1 --2- 172.21.15.57:0/64858904 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95780664a0 0x7f9578068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:30.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.480+0000 7f958effd640 1 -- 172.21.15.57:0/64858904 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f95880849a0 con 0x7f95980c4a10 2023-10-30T14:35:30.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.480+0000 7f958ffff640 1 --2- 172.21.15.57:0/64858904 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95780664a0 0x7f9578068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:30.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.480+0000 7f958ffff640 1 --2- 172.21.15.57:0/64858904 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95780664a0 0x7f9578068960 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f95800b2000 tx=0x7f9580008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:30.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.484+0000 7f958effd640 1 -- 172.21.15.57:0/64858904 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9588051870 con 0x7f95980c4a10 2023-10-30T14:35:30.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.680+0000 7f959cb72640 1 -- 172.21.15.57:0/64858904 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95980b5050 con 0x7f95980c4a10 2023-10-30T14:35:30.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.680+0000 7f958effd640 1 -- 172.21.15.57:0/64858904 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5739 (secure 0 0 0) 0x7f9588054e90 con 0x7f95980c4a10 2023-10-30T14:35:30.683 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:30.683 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:35:29.980342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:30.684 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 120 2023-10-30T14:35:30.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.684+0000 7f959cb72640 1 -- 172.21.15.57:0/64858904 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95780664a0 msgr2=0x7f9578068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:30.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.684+0000 7f959cb72640 1 --2- 172.21.15.57:0/64858904 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95780664a0 0x7f9578068960 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f95800b2000 tx=0x7f9580008040 comp rx=0 tx=0).stop 2023-10-30T14:35:30.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.684+0000 7f959cb72640 1 -- 172.21.15.57:0/64858904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95980c4a10 msgr2=0x7f95980b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:30.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.684+0000 7f959cb72640 1 --2- 172.21.15.57:0/64858904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95980c4a10 0x7f95980b3e30 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f958800d800 tx=0x7f958800dcc0 comp rx=0 tx=0).stop 2023-10-30T14:35:30.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.684+0000 7f959cb72640 1 -- 172.21.15.57:0/64858904 shutdown_connections 2023-10-30T14:35:30.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.684+0000 7f959cb72640 1 --2- 172.21.15.57:0/64858904 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95780664a0 0x7f9578068960 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:30.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.684+0000 7f959cb72640 1 --2- 172.21.15.57:0/64858904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f959815a3b0 0x7f95981f0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:30.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.684+0000 7f959cb72640 1 --2- 172.21.15.57:0/64858904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9598153c80 0x7f95980b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:30.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.684+0000 7f959cb72640 1 --2- 172.21.15.57:0/64858904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95980c4a10 0x7f95980b3e30 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:30.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.684+0000 7f959cb72640 1 -- 172.21.15.57:0/64858904 >> 172.21.15.57:0/64858904 conn(0x7f95980715c0 msgr2=0x7f9598157970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:30.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.684+0000 7f959cb72640 1 -- 172.21.15.57:0/64858904 shutdown_connections 2023-10-30T14:35:30.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.684+0000 7f959cb72640 1 -- 172.21.15.57:0/64858904 wait complete. 2023-10-30T14:35:30.704 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-30T14:35:30.704 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:35:30.704 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:30.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.844+0000 7fee624c9640 1 Processor -- start 2023-10-30T14:35:30.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.844+0000 7fee624c9640 1 -- start start 2023-10-30T14:35:30.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.848+0000 7fee624c9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee5c150830 0x7fee5c06b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:30.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.848+0000 7fee624c9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fee5c0cb7d0 0x7fee5c0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:30.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.848+0000 7fee624c9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5c0c2310 con 0x7fee5c0c3e40 2023-10-30T14:35:30.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.848+0000 7fee624c9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee5c156710 con 0x7fee5c0cb7d0 2023-10-30T14:35:30.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.848+0000 7fee624c9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c0c4360 con 0x7fee5c150830 2023-10-30T14:35:30.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.848+0000 7fee5b7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fee5c0cb7d0 0x7fee5c0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:30.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.848+0000 7fee5b7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fee5c0cb7d0 0x7fee5c0c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44948/0 (socket says 172.21.15.57:44948) 2023-10-30T14:35:30.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.848+0000 7fee5b7fe640 1 -- 172.21.15.57:0/3623711942 learned_addr learned my addr 172.21.15.57:0/3623711942 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:30.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.848+0000 7fee5bfff640 1 --2- 172.21.15.57:0/3623711942 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee5c150830 0x7fee5c06b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:30.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.848+0000 7fee5b7fe640 1 -- 172.21.15.57:0/3623711942 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee5c150830 msgr2=0x7fee5c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:30.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.848+0000 7fee5b7fe640 1 --2- 172.21.15.57:0/3623711942 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee5c150830 0x7fee5c06b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:30.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.848+0000 7fee5b7fe640 1 -- 172.21.15.57:0/3623711942 >> v1:172.21.15.88:6789/0 conn(0x7fee5c0c3e40 legacy=0x7fee5c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:30.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.848+0000 7fee5b7fe640 1 -- 172.21.15.57:0/3623711942 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5c1569b0 con 0x7fee5c0cb7d0 2023-10-30T14:35:30.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.852+0000 7fee5b7fe640 1 --2- 172.21.15.57:0/3623711942 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee5c0cb7d0 0x7fee5c0c3900 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fee50009870 tx=0x7fee5001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=695e3ce9ae6cd99 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:30.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.852+0000 7fee5affd640 1 -- 172.21.15.57:0/3623711942 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee500a8a30 con 0x7fee5c0cb7d0 2023-10-30T14:35:30.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.852+0000 7fee5affd640 1 -- 172.21.15.57:0/3623711942 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee500a8b90 con 0x7fee5c0cb7d0 2023-10-30T14:35:30.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.852+0000 7fee5affd640 1 -- 172.21.15.57:0/3623711942 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee500b0440 con 0x7fee5c0cb7d0 2023-10-30T14:35:30.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.852+0000 7fee624c9640 1 -- 172.21.15.57:0/3623711942 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee5c0cb7d0 msgr2=0x7fee5c0c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:30.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.852+0000 7fee624c9640 1 --2- 172.21.15.57:0/3623711942 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee5c0cb7d0 0x7fee5c0c3900 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fee50009870 tx=0x7fee5001ed30 comp rx=0 tx=0).stop 2023-10-30T14:35:30.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.852+0000 7fee624c9640 1 -- 172.21.15.57:0/3623711942 shutdown_connections 2023-10-30T14:35:30.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.852+0000 7fee624c9640 1 --2- 172.21.15.57:0/3623711942 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee5c0cb7d0 0x7fee5c0c3900 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:30.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.852+0000 7fee624c9640 1 --2- 172.21.15.57:0/3623711942 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee5c150830 0x7fee5c06b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:30.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.852+0000 7fee624c9640 1 -- 172.21.15.57:0/3623711942 >> 172.21.15.57:0/3623711942 conn(0x7fee5c0cac70 msgr2=0x7fee5c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:30.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.852+0000 7fee624c9640 1 -- 172.21.15.57:0/3623711942 shutdown_connections 2023-10-30T14:35:30.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.852+0000 7fee624c9640 1 -- 172.21.15.57:0/3623711942 wait complete. 2023-10-30T14:35:30.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.852+0000 7fee624c9640 1 Processor -- start 2023-10-30T14:35:30.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.852+0000 7fee624c9640 1 -- start start 2023-10-30T14:35:30.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee624c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee5c0c3e40 0x7fee5c1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:30.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee624c9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee5c0cb7d0 0x7fee5c155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:30.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee624c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee5c150830 0x7fee5c1485a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:30.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee624c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c154390 con 0x7fee5c0c3e40 2023-10-30T14:35:30.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee624c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c14a1b0 con 0x7fee5c150830 2023-10-30T14:35:30.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee624c9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c156390 con 0x7fee5c0cb7d0 2023-10-30T14:35:30.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee5bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee5c0c3e40 0x7fee5c1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:30.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee5bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee5c0c3e40 0x7fee5c1550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44958/0 (socket says 172.21.15.57:44958) 2023-10-30T14:35:30.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee5bfff640 1 -- 172.21.15.57:0/2550485380 learned_addr learned my addr 172.21.15.57:0/2550485380 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:30.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee60a66640 1 --2- 172.21.15.57:0/2550485380 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee5c150830 0x7fee5c1485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:30.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee5b7fe640 1 --2- 172.21.15.57:0/2550485380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee5c0cb7d0 0x7fee5c155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:30.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee5bfff640 1 -- 172.21.15.57:0/2550485380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee5c0cb7d0 msgr2=0x7fee5c155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:30.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee5bfff640 1 --2- 172.21.15.57:0/2550485380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee5c0cb7d0 0x7fee5c155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:30.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee5bfff640 1 -- 172.21.15.57:0/2550485380 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee5c150830 msgr2=0x7fee5c1485a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:30.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee5bfff640 1 --2- 172.21.15.57:0/2550485380 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee5c150830 0x7fee5c1485a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:30.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee5bfff640 1 -- 172.21.15.57:0/2550485380 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee500094d0 con 0x7fee5c0c3e40 2023-10-30T14:35:30.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee5bfff640 1 --2- 172.21.15.57:0/2550485380 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee5c0c3e40 0x7fee5c1550c0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fee4c00d5f0 tx=0x7fee4c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:30.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee58ff9640 1 -- 172.21.15.57:0/2550485380 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c0125f0 con 0x7fee5c0c3e40 2023-10-30T14:35:30.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee58ff9640 1 -- 172.21.15.57:0/2550485380 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4c012750 con 0x7fee5c0c3e40 2023-10-30T14:35:30.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee624c9640 1 -- 172.21.15.57:0/2550485380 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee5c1545e0 con 0x7fee5c0c3e40 2023-10-30T14:35:30.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee58ff9640 1 -- 172.21.15.57:0/2550485380 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c012ae0 con 0x7fee5c0c3e40 2023-10-30T14:35:30.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee624c9640 1 -- 172.21.15.57:0/2550485380 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee5c154ba0 con 0x7fee5c0c3e40 2023-10-30T14:35:30.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee624c9640 1 -- 172.21.15.57:0/2550485380 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee5c04e170 con 0x7fee5c0c3e40 2023-10-30T14:35:30.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.856+0000 7fee58ff9640 1 -- 172.21.15.57:0/2550485380 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fee4c012c40 con 0x7fee5c0c3e40 2023-10-30T14:35:30.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.860+0000 7fee58ff9640 1 --2- 172.21.15.57:0/2550485380 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fee480664a0 0x7fee48068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:30.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.860+0000 7fee58ff9640 1 -- 172.21.15.57:0/2550485380 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fee4c084910 con 0x7fee5c0c3e40 2023-10-30T14:35:30.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.860+0000 7fee5b7fe640 1 --2- 172.21.15.57:0/2550485380 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fee480664a0 0x7fee48068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:30.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.860+0000 7fee5b7fe640 1 --2- 172.21.15.57:0/2550485380 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fee480664a0 0x7fee48068960 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fee500b2000 tx=0x7fee50008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:30.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:30.864+0000 7fee58ff9640 1 -- 172.21.15.57:0/2550485380 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fee4c051800 con 0x7fee5c0c3e40 2023-10-30T14:35:31.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:31.052+0000 7fee624c9640 1 -- 172.21.15.57:0/2550485380 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee5c14abc0 con 0x7fee5c0c3e40 2023-10-30T14:35:31.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:31.052+0000 7fee58ff9640 1 -- 172.21.15.57:0/2550485380 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5739 (secure 0 0 0) 0x7fee4c054e20 con 0x7fee5c0c3e40 2023-10-30T14:35:31.054 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:31.055 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:35:29.980342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:31.056 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 120 2023-10-30T14:35:31.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:31.052+0000 7fee624c9640 1 -- 172.21.15.57:0/2550485380 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fee480664a0 msgr2=0x7fee48068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:31.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:31.052+0000 7fee624c9640 1 --2- 172.21.15.57:0/2550485380 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fee480664a0 0x7fee48068960 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fee500b2000 tx=0x7fee50008040 comp rx=0 tx=0).stop 2023-10-30T14:35:31.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:31.052+0000 7fee624c9640 1 -- 172.21.15.57:0/2550485380 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee5c0c3e40 msgr2=0x7fee5c1550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:31.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:31.052+0000 7fee624c9640 1 --2- 172.21.15.57:0/2550485380 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee5c0c3e40 0x7fee5c1550c0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fee4c00d5f0 tx=0x7fee4c00dab0 comp rx=0 tx=0).stop 2023-10-30T14:35:31.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:31.052+0000 7fee624c9640 1 -- 172.21.15.57:0/2550485380 shutdown_connections 2023-10-30T14:35:31.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:31.052+0000 7fee624c9640 1 --2- 172.21.15.57:0/2550485380 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fee480664a0 0x7fee48068960 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:31.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:31.052+0000 7fee624c9640 1 --2- 172.21.15.57:0/2550485380 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee5c150830 0x7fee5c1485a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:31.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:31.052+0000 7fee624c9640 1 --2- 172.21.15.57:0/2550485380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee5c0cb7d0 0x7fee5c155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:31.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:31.052+0000 7fee624c9640 1 --2- 172.21.15.57:0/2550485380 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee5c0c3e40 0x7fee5c1550c0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:31.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:31.052+0000 7fee624c9640 1 -- 172.21.15.57:0/2550485380 >> 172.21.15.57:0/2550485380 conn(0x7fee5c0cac70 msgr2=0x7fee5c070520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:31.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:31.052+0000 7fee624c9640 1 -- 172.21.15.57:0/2550485380 shutdown_connections 2023-10-30T14:35:31.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:31.052+0000 7fee624c9640 1 -- 172.21.15.57:0/2550485380 wait complete. 2023-10-30T14:35:31.076 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 120, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 120, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6204": { "addr": "172.21.15.57:6837/1843528309", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 1843528309, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 1843528309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 6204, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_6414": { "addr": "172.21.15.57:6834/2321408149", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2321408149, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2321408149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6414, "incarnation": 99, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 18 }, "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:35:29.980342+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6414, "mds_1": 6747, "mds_2": 6204 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:35:33.078 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:33.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f806a5b8640 1 Processor -- start 2023-10-30T14:35:33.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f806a5b8640 1 -- start start 2023-10-30T14:35:33.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f806a5b8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8064153c80 0x7f806415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:33.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f806a5b8640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f80640c4a10 0x7f80640c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:33.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f806a5b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f806415a3b0 0x7f80640ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:33.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f806a5b8640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f806415dec0 con 0x7f80640c4a10 2023-10-30T14:35:33.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f806a5b8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80641599a0 con 0x7f8064153c80 2023-10-30T14:35:33.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f806a5b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8064159f20 con 0x7f806415a3b0 2023-10-30T14:35:33.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f80637fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f80640c4a10 0x7f80640c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:33.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f80637fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f80640c4a10 0x7f80640c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44970/0 (socket says 172.21.15.57:44970) 2023-10-30T14:35:33.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f80637fe640 1 -- 172.21.15.57:0/2354954240 learned_addr learned my addr 172.21.15.57:0/2354954240 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:33.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f8068b55640 1 --2- 172.21.15.57:0/2354954240 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f806415a3b0 0x7f80640ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:33.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f8063fff640 1 --2- 172.21.15.57:0/2354954240 >> v2:172.21.15.88:3300/0 conn(0x7f8064153c80 0x7f806415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:33.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f80637fe640 1 -- 172.21.15.57:0/2354954240 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f806415a3b0 msgr2=0x7f80640ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:33.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f80637fe640 1 --2- 172.21.15.57:0/2354954240 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f806415a3b0 0x7f80640ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:33.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f80637fe640 1 -- 172.21.15.57:0/2354954240 >> v2:172.21.15.88:3300/0 conn(0x7f8064153c80 msgr2=0x7f806415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:33.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f80637fe640 1 --2- 172.21.15.57:0/2354954240 >> v2:172.21.15.88:3300/0 conn(0x7f8064153c80 0x7f806415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:33.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f80637fe640 1 -- 172.21.15.57:0/2354954240 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80640c47e0 con 0x7f80640c4a10 2023-10-30T14:35:33.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f8068b55640 1 --2- 172.21.15.57:0/2354954240 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f806415a3b0 0x7f80640ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:33.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.232+0000 7f8063fff640 1 --2- 172.21.15.57:0/2354954240 >> v2:172.21.15.88:3300/0 conn(0x7f8064153c80 0x7f806415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:33.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.236+0000 7f80637fe640 1 --2- 172.21.15.57:0/2354954240 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80640c4a10 0x7f80640c42a0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f80640d00f0 tx=0x7f805801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e1bd761f37e826 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:33.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.236+0000 7f8062ffd640 1 -- 172.21.15.57:0/2354954240 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80580a89b0 con 0x7f80640c4a10 2023-10-30T14:35:33.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.236+0000 7f8062ffd640 1 -- 172.21.15.57:0/2354954240 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80580a8b10 con 0x7f80640c4a10 2023-10-30T14:35:33.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.236+0000 7f8062ffd640 1 -- 172.21.15.57:0/2354954240 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80580a8ea0 con 0x7f80640c4a10 2023-10-30T14:35:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.236+0000 7f806a5b8640 1 -- 172.21.15.57:0/2354954240 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80640c4a10 msgr2=0x7f80640c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.236+0000 7f806a5b8640 1 --2- 172.21.15.57:0/2354954240 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80640c4a10 0x7f80640c42a0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f80640d00f0 tx=0x7f805801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:35:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.236+0000 7f806a5b8640 1 -- 172.21.15.57:0/2354954240 shutdown_connections 2023-10-30T14:35:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.236+0000 7f806a5b8640 1 --2- 172.21.15.57:0/2354954240 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f806415a3b0 0x7f80640ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.236+0000 7f806a5b8640 1 --2- 172.21.15.57:0/2354954240 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80640c4a10 0x7f80640c42a0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.236+0000 7f806a5b8640 1 --2- 172.21.15.57:0/2354954240 >> v2:172.21.15.88:3300/0 conn(0x7f8064153c80 0x7f806415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.236+0000 7f806a5b8640 1 -- 172.21.15.57:0/2354954240 >> 172.21.15.57:0/2354954240 conn(0x7f80640715c0 msgr2=0x7f80640719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.236+0000 7f806a5b8640 1 -- 172.21.15.57:0/2354954240 shutdown_connections 2023-10-30T14:35:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.236+0000 7f806a5b8640 1 -- 172.21.15.57:0/2354954240 wait complete. 2023-10-30T14:35:33.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f806a5b8640 1 Processor -- start 2023-10-30T14:35:33.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f806a5b8640 1 -- start start 2023-10-30T14:35:33.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f806a5b8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80640c4a10 0x7f80641f2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:33.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f806a5b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8064153c80 0x7f806415bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:33.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f806a5b8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f806415a3b0 0x7f80640c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:33.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f806a5b8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80640b4b00 con 0x7f806415a3b0 2023-10-30T14:35:33.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f806a5b8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80641f2dc0 con 0x7f80640c4a10 2023-10-30T14:35:33.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f806a5b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80641ee900 con 0x7f8064153c80 2023-10-30T14:35:33.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f8068b55640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f806415a3b0 0x7f80640c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:33.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f8068b55640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f806415a3b0 0x7f80640c8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44974/0 (socket says 172.21.15.57:44974) 2023-10-30T14:35:33.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f8068b55640 1 -- 172.21.15.57:0/6321418 learned_addr learned my addr 172.21.15.57:0/6321418 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:33.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f8063fff640 1 --2- 172.21.15.57:0/6321418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80640c4a10 0x7f80641f2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:33.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f80637fe640 1 --2- 172.21.15.57:0/6321418 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8064153c80 0x7f806415bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:33.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f8068b55640 1 -- 172.21.15.57:0/6321418 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8064153c80 msgr2=0x7f806415bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:33.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f8068b55640 1 --2- 172.21.15.57:0/6321418 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8064153c80 0x7f806415bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:33.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f8068b55640 1 -- 172.21.15.57:0/6321418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80640c4a10 msgr2=0x7f80641f2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:33.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f8068b55640 1 --2- 172.21.15.57:0/6321418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80640c4a10 0x7f80641f2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:33.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f8068b55640 1 -- 172.21.15.57:0/6321418 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80580094d0 con 0x7f806415a3b0 2023-10-30T14:35:33.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f8063fff640 1 --2- 172.21.15.57:0/6321418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80640c4a10 0x7f80641f2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:33.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f8068b55640 1 --2- 172.21.15.57:0/6321418 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f806415a3b0 0x7f80640c8b00 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f805c007df0 tx=0x7f805c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:33.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f8060ff9640 1 -- 172.21.15.57:0/6321418 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c0039a0 con 0x7f806415a3b0 2023-10-30T14:35:33.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f8060ff9640 1 -- 172.21.15.57:0/6321418 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805c00f040 con 0x7f806415a3b0 2023-10-30T14:35:33.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f8060ff9640 1 -- 172.21.15.57:0/6321418 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c003d40 con 0x7f806415a3b0 2023-10-30T14:35:33.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f806a5b8640 1 -- 172.21.15.57:0/6321418 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80640b25b0 con 0x7f806415a3b0 2023-10-30T14:35:33.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f806a5b8640 1 -- 172.21.15.57:0/6321418 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80641eebc0 con 0x7f806415a3b0 2023-10-30T14:35:33.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.240+0000 7f806a5b8640 1 -- 172.21.15.57:0/6321418 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f806404e170 con 0x7f806415a3b0 2023-10-30T14:35:33.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.244+0000 7f8060ff9640 1 -- 172.21.15.57:0/6321418 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f805c004030 con 0x7f806415a3b0 2023-10-30T14:35:33.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.244+0000 7f8060ff9640 1 --2- 172.21.15.57:0/6321418 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80500664a0 0x7f8050068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:33.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.244+0000 7f8060ff9640 1 -- 172.21.15.57:0/6321418 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f805c0846e0 con 0x7f806415a3b0 2023-10-30T14:35:33.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.244+0000 7f8063fff640 1 --2- 172.21.15.57:0/6321418 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80500664a0 0x7f8050068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:33.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.244+0000 7f8063fff640 1 --2- 172.21.15.57:0/6321418 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80500664a0 0x7f8050068960 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f805400c940 tx=0x7f8054008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:33.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.248+0000 7f8060ff9640 1 -- 172.21.15.57:0/6321418 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f805c0515b0 con 0x7f806415a3b0 2023-10-30T14:35:33.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.436+0000 7f806a5b8640 1 -- 172.21.15.57:0/6321418 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80640cad40 con 0x7f806415a3b0 2023-10-30T14:35:33.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.436+0000 7f8060ff9640 1 -- 172.21.15.57:0/6321418 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5739 (secure 0 0 0) 0x7f805c054bd0 con 0x7f806415a3b0 2023-10-30T14:35:33.440 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:33.441 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:35:29.980342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:33.441 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 120 2023-10-30T14:35:33.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.440+0000 7f806a5b8640 1 -- 172.21.15.57:0/6321418 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80500664a0 msgr2=0x7f8050068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:33.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.440+0000 7f806a5b8640 1 --2- 172.21.15.57:0/6321418 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80500664a0 0x7f8050068960 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f805400c940 tx=0x7f8054008040 comp rx=0 tx=0).stop 2023-10-30T14:35:33.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.440+0000 7f806a5b8640 1 -- 172.21.15.57:0/6321418 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f806415a3b0 msgr2=0x7f80640c8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:33.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.440+0000 7f806a5b8640 1 --2- 172.21.15.57:0/6321418 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f806415a3b0 0x7f80640c8b00 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f805c007df0 tx=0x7f805c00d4a0 comp rx=0 tx=0).stop 2023-10-30T14:35:33.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.440+0000 7f806a5b8640 1 -- 172.21.15.57:0/6321418 shutdown_connections 2023-10-30T14:35:33.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.440+0000 7f806a5b8640 1 --2- 172.21.15.57:0/6321418 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80500664a0 0x7f8050068960 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:33.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.440+0000 7f806a5b8640 1 --2- 172.21.15.57:0/6321418 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f806415a3b0 0x7f80640c8b00 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:33.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.440+0000 7f806a5b8640 1 --2- 172.21.15.57:0/6321418 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8064153c80 0x7f806415bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:33.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.440+0000 7f806a5b8640 1 --2- 172.21.15.57:0/6321418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80640c4a10 0x7f80641f2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:33.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.440+0000 7f806a5b8640 1 -- 172.21.15.57:0/6321418 >> 172.21.15.57:0/6321418 conn(0x7f80640715c0 msgr2=0x7f8064158450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:33.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.440+0000 7f806a5b8640 1 -- 172.21.15.57:0/6321418 shutdown_connections 2023-10-30T14:35:33.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:33.440+0000 7f806a5b8640 1 -- 172.21.15.57:0/6321418 wait complete. 2023-10-30T14:35:33.461 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:35:35.463 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:35.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.608+0000 7f0771572640 1 Processor -- start 2023-10-30T14:35:35.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.608+0000 7f0771572640 1 -- start start 2023-10-30T14:35:35.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.612+0000 7f0771572640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f076c153c80 0x7f076c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:35.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.612+0000 7f0771572640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f076c0c4a10 0x7f076c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:35.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.612+0000 7f0771572640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076c159f20 con 0x7f076c15a3b0 2023-10-30T14:35:35.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.612+0000 7f0771572640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f076c15dec0 con 0x7f076c153c80 2023-10-30T14:35:35.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.612+0000 7f0771572640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f076c1599a0 con 0x7f076c0c4a10 2023-10-30T14:35:35.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.612+0000 7f076affd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f076c153c80 0x7f076c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:35.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.612+0000 7f076affd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f076c153c80 0x7f076c15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56210/0 (socket says 172.21.15.57:56210) 2023-10-30T14:35:35.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.612+0000 7f076b7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f076c15a3b0 0x7f076c0ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:60594/0 (socket says 172.21.15.57:60594) 2023-10-30T14:35:35.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.612+0000 7f076affd640 1 -- 172.21.15.57:0/979439774 learned_addr learned my addr 172.21.15.57:0/979439774 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:35.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.612+0000 7f076a7fc640 1 --2- 172.21.15.57:0/979439774 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f076c0c4a10 0x7f076c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:35.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.612+0000 7f076affd640 1 -- 172.21.15.57:0/979439774 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f076c0c4a10 msgr2=0x7f076c0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:35.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.612+0000 7f076affd640 1 --2- 172.21.15.57:0/979439774 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f076c0c4a10 0x7f076c0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:35.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.612+0000 7f076affd640 1 -- 172.21.15.57:0/979439774 >> v1:172.21.15.57:6789/0 conn(0x7f076c15a3b0 legacy=0x7f076c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:35.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.612+0000 7f0769ffb640 1 -- 172.21.15.57:0/979439774 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4207118877 0 0) 0x7f076c159f20 con 0x7f076c15a3b0 2023-10-30T14:35:35.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.612+0000 7f076affd640 1 -- 172.21.15.57:0/979439774 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c0c47e0 con 0x7f076c153c80 2023-10-30T14:35:35.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.612+0000 7f076affd640 1 --2- 172.21.15.57:0/979439774 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f076c153c80 0x7f076c15cf90 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f075c009870 tx=0x7f075c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ff8f8f2e55d1429 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:35.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.616+0000 7f0769ffb640 1 -- 172.21.15.57:0/979439774 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c0a8a30 con 0x7f076c153c80 2023-10-30T14:35:35.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.616+0000 7f0769ffb640 1 -- 172.21.15.57:0/979439774 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c0a8b90 con 0x7f076c153c80 2023-10-30T14:35:35.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.616+0000 7f0769ffb640 1 -- 172.21.15.57:0/979439774 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c0b0440 con 0x7f076c153c80 2023-10-30T14:35:35.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.616+0000 7f0771572640 1 -- 172.21.15.57:0/979439774 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f076c153c80 msgr2=0x7f076c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:35.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.616+0000 7f0771572640 1 --2- 172.21.15.57:0/979439774 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f076c153c80 0x7f076c15cf90 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f075c009870 tx=0x7f075c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:35:35.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.616+0000 7f0771572640 1 -- 172.21.15.57:0/979439774 shutdown_connections 2023-10-30T14:35:35.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.616+0000 7f0771572640 1 --2- 172.21.15.57:0/979439774 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f076c0c4a10 0x7f076c0c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:35.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.616+0000 7f0771572640 1 --2- 172.21.15.57:0/979439774 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f076c153c80 0x7f076c15cf90 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:35.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.616+0000 7f0771572640 1 -- 172.21.15.57:0/979439774 >> 172.21.15.57:0/979439774 conn(0x7f076c0715c0 msgr2=0x7f076c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:35.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.616+0000 7f0771572640 1 -- 172.21.15.57:0/979439774 shutdown_connections 2023-10-30T14:35:35.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.616+0000 7f0771572640 1 -- 172.21.15.57:0/979439774 wait complete. 2023-10-30T14:35:35.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f0771572640 1 Processor -- start 2023-10-30T14:35:35.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f0771572640 1 -- start start 2023-10-30T14:35:35.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f0771572640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f076c0c4a10 0x7f076c14efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:35.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f0771572640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f076c153c80 0x7f076c0c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:35.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f0771572640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f076c15a3b0 0x7f076c149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:35.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f0771572640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076c1ef0e0 con 0x7f076c0c4a10 2023-10-30T14:35:35.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f0771572640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076c1587c0 con 0x7f076c15a3b0 2023-10-30T14:35:35.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f0771572640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f076c14eb90 con 0x7f076c153c80 2023-10-30T14:35:35.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f076affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f076c0c4a10 0x7f076c14efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:35.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f076affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f076c0c4a10 0x7f076c14efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56218/0 (socket says 172.21.15.57:56218) 2023-10-30T14:35:35.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f076affd640 1 -- 172.21.15.57:0/3109228428 learned_addr learned my addr 172.21.15.57:0/3109228428 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:35.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f076a7fc640 1 --2- 172.21.15.57:0/3109228428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f076c153c80 0x7f076c0c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:35.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f076b7fe640 1 --2- 172.21.15.57:0/3109228428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f076c15a3b0 0x7f076c149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:35.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f076affd640 1 -- 172.21.15.57:0/3109228428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f076c153c80 msgr2=0x7f076c0c8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:35.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f076affd640 1 --2- 172.21.15.57:0/3109228428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f076c153c80 0x7f076c0c8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:35.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f076affd640 1 -- 172.21.15.57:0/3109228428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f076c15a3b0 msgr2=0x7f076c149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:35.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f076affd640 1 --2- 172.21.15.57:0/3109228428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f076c15a3b0 0x7f076c149b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:35.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f076affd640 1 -- 172.21.15.57:0/3109228428 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c0094d0 con 0x7f076c0c4a10 2023-10-30T14:35:35.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f076b7fe640 1 --2- 172.21.15.57:0/3109228428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f076c15a3b0 0x7f076c149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:35:35.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f076affd640 1 --2- 172.21.15.57:0/3109228428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f076c0c4a10 0x7f076c14efb0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f075c0b3040 tx=0x7f075c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:35.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f07537fe640 1 -- 172.21.15.57:0/3109228428 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c0a8a30 con 0x7f076c0c4a10 2023-10-30T14:35:35.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f07537fe640 1 -- 172.21.15.57:0/3109228428 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c0bb480 con 0x7f076c0c4a10 2023-10-30T14:35:35.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f0771572640 1 -- 172.21.15.57:0/3109228428 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076c1ee7d0 con 0x7f076c0c4a10 2023-10-30T14:35:35.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f07537fe640 1 -- 172.21.15.57:0/3109228428 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c0bb830 con 0x7f076c0c4a10 2023-10-30T14:35:35.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.620+0000 7f0771572640 1 -- 172.21.15.57:0/3109228428 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076c1f2ad0 con 0x7f076c0c4a10 2023-10-30T14:35:35.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.624+0000 7f0771572640 1 -- 172.21.15.57:0/3109228428 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f076c073c30 con 0x7f076c0c4a10 2023-10-30T14:35:35.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.624+0000 7f07537fe640 1 -- 172.21.15.57:0/3109228428 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f075c0bb990 con 0x7f076c0c4a10 2023-10-30T14:35:35.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.624+0000 7f07537fe640 1 --2- 172.21.15.57:0/3109228428 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07580664a0 0x7f0758068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:35.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.624+0000 7f07537fe640 1 -- 172.21.15.57:0/3109228428 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f075c1229e0 con 0x7f076c0c4a10 2023-10-30T14:35:35.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.624+0000 7f076a7fc640 1 --2- 172.21.15.57:0/3109228428 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07580664a0 0x7f0758068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:35.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.628+0000 7f076a7fc640 1 --2- 172.21.15.57:0/3109228428 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07580664a0 0x7f0758068960 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f076c0c7100 tx=0x7f0760008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:35.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.632+0000 7f07537fe640 1 -- 172.21.15.57:0/3109228428 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f075c0ef940 con 0x7f076c0c4a10 2023-10-30T14:35:35.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.864+0000 7f0771572640 1 -- 172.21.15.57:0/3109228428 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f076c04e170 con 0x7f076c0c4a10 2023-10-30T14:35:35.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.864+0000 7f07537fe640 1 -- 172.21.15.57:0/3109228428 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5739 (secure 0 0 0) 0x7f075c0b5090 con 0x7f076c0c4a10 2023-10-30T14:35:35.866 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:35.867 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:35:29.980342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:35.867 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 120 2023-10-30T14:35:35.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.864+0000 7f0771572640 1 -- 172.21.15.57:0/3109228428 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07580664a0 msgr2=0x7f0758068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:35.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.864+0000 7f0771572640 1 --2- 172.21.15.57:0/3109228428 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07580664a0 0x7f0758068960 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f076c0c7100 tx=0x7f0760008040 comp rx=0 tx=0).stop 2023-10-30T14:35:35.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.864+0000 7f0771572640 1 -- 172.21.15.57:0/3109228428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f076c0c4a10 msgr2=0x7f076c14efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:35.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.864+0000 7f0771572640 1 --2- 172.21.15.57:0/3109228428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f076c0c4a10 0x7f076c14efb0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f075c0b3040 tx=0x7f075c01bad0 comp rx=0 tx=0).stop 2023-10-30T14:35:35.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.864+0000 7f0771572640 1 -- 172.21.15.57:0/3109228428 shutdown_connections 2023-10-30T14:35:35.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.864+0000 7f0771572640 1 --2- 172.21.15.57:0/3109228428 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07580664a0 0x7f0758068960 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:35.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.864+0000 7f0771572640 1 --2- 172.21.15.57:0/3109228428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f076c15a3b0 0x7f076c149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:35.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.864+0000 7f0771572640 1 --2- 172.21.15.57:0/3109228428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f076c153c80 0x7f076c0c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:35.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.864+0000 7f0771572640 1 --2- 172.21.15.57:0/3109228428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f076c0c4a10 0x7f076c14efb0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:35.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.864+0000 7f0771572640 1 -- 172.21.15.57:0/3109228428 >> 172.21.15.57:0/3109228428 conn(0x7f076c0715c0 msgr2=0x7f076c152e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:35.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.864+0000 7f0771572640 1 -- 172.21.15.57:0/3109228428 shutdown_connections 2023-10-30T14:35:35.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:35.864+0000 7f0771572640 1 -- 172.21.15.57:0/3109228428 wait complete. 2023-10-30T14:35:35.888 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:35:37.210 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:35:37.215 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:35:37.888 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:38.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.040+0000 7f397ec68640 1 Processor -- start 2023-10-30T14:35:38.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.044+0000 7f397ec68640 1 -- start start 2023-10-30T14:35:38.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.044+0000 7f397ec68640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3978153c80 0x7f397815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:38.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.044+0000 7f397ec68640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39780c4a10 0x7f39780c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:38.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.044+0000 7f397ec68640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f397815a3b0 0x7f39780ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:38.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.044+0000 7f397ec68640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f397815dec0 con 0x7f397815a3b0 2023-10-30T14:35:38.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.044+0000 7f397ec68640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39781599a0 con 0x7f3978153c80 2023-10-30T14:35:38.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.044+0000 7f397ec68640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3978159f20 con 0x7f39780c4a10 2023-10-30T14:35:38.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.044+0000 7f397d205640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f397815a3b0 0x7f39780ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:38.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.044+0000 7f397d205640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f397815a3b0 0x7f39780ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56222/0 (socket says 172.21.15.57:56222) 2023-10-30T14:35:38.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.044+0000 7f397d205640 1 -- 172.21.15.57:0/108545373 learned_addr learned my addr 172.21.15.57:0/108545373 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:38.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.044+0000 7f3977fff640 1 --2- 172.21.15.57:0/108545373 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39780c4a10 0x7f39780c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:38.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.044+0000 7f397ca04640 1 --2- 172.21.15.57:0/108545373 >> v2:172.21.15.88:3300/0 conn(0x7f3978153c80 0x7f397815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:38.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.044+0000 7f397d205640 1 -- 172.21.15.57:0/108545373 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39780c4a10 msgr2=0x7f39780c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:38.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.044+0000 7f397d205640 1 --2- 172.21.15.57:0/108545373 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39780c4a10 0x7f39780c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:38.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.044+0000 7f397d205640 1 -- 172.21.15.57:0/108545373 >> v2:172.21.15.88:3300/0 conn(0x7f3978153c80 msgr2=0x7f397815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:38.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.044+0000 7f397d205640 1 --2- 172.21.15.57:0/108545373 >> v2:172.21.15.88:3300/0 conn(0x7f3978153c80 0x7f397815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:38.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.044+0000 7f397d205640 1 -- 172.21.15.57:0/108545373 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39780c47e0 con 0x7f397815a3b0 2023-10-30T14:35:38.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.044+0000 7f3977fff640 1 --2- 172.21.15.57:0/108545373 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39780c4a10 0x7f39780c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:38.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.048+0000 7f397d205640 1 --2- 172.21.15.57:0/108545373 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f397815a3b0 0x7f39780ca1e0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f39780d00f0 tx=0x7f397001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ad41cc34c0ac55f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:38.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.048+0000 7f39777fe640 1 -- 172.21.15.57:0/108545373 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39700af400 con 0x7f397815a3b0 2023-10-30T14:35:38.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.048+0000 7f39777fe640 1 -- 172.21.15.57:0/108545373 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39700ab050 con 0x7f397815a3b0 2023-10-30T14:35:38.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.048+0000 7f39777fe640 1 -- 172.21.15.57:0/108545373 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39700af780 con 0x7f397815a3b0 2023-10-30T14:35:38.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.048+0000 7f397ec68640 1 -- 172.21.15.57:0/108545373 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f397815a3b0 msgr2=0x7f39780ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:38.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.048+0000 7f397ec68640 1 --2- 172.21.15.57:0/108545373 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f397815a3b0 0x7f39780ca1e0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f39780d00f0 tx=0x7f397001f1c0 comp rx=0 tx=0).stop 2023-10-30T14:35:38.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.048+0000 7f397ec68640 1 -- 172.21.15.57:0/108545373 shutdown_connections 2023-10-30T14:35:38.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.048+0000 7f397ec68640 1 --2- 172.21.15.57:0/108545373 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39780c4a10 0x7f39780c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:38.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.048+0000 7f397ec68640 1 --2- 172.21.15.57:0/108545373 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f397815a3b0 0x7f39780ca1e0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:38.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.048+0000 7f397ec68640 1 --2- 172.21.15.57:0/108545373 >> v2:172.21.15.88:3300/0 conn(0x7f3978153c80 0x7f397815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:38.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.048+0000 7f397ec68640 1 -- 172.21.15.57:0/108545373 >> 172.21.15.57:0/108545373 conn(0x7f39780715c0 msgr2=0x7f39780719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:38.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.048+0000 7f397ec68640 1 -- 172.21.15.57:0/108545373 shutdown_connections 2023-10-30T14:35:38.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.048+0000 7f397ec68640 1 -- 172.21.15.57:0/108545373 wait complete. 2023-10-30T14:35:38.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.048+0000 7f397ec68640 1 Processor -- start 2023-10-30T14:35:38.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.048+0000 7f397ec68640 1 -- start start 2023-10-30T14:35:38.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397ec68640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39780c4a10 0x7f3978155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:38.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397ec68640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3978153c80 0x7f397814de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:38.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397ec68640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f397815a3b0 0x7f397814e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:38.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397ec68640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39780ce120 con 0x7f397815a3b0 2023-10-30T14:35:38.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397ec68640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397814fbf0 con 0x7f39780c4a10 2023-10-30T14:35:38.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397ec68640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39780cd1c0 con 0x7f3978153c80 2023-10-30T14:35:38.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397d205640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f397815a3b0 0x7f397814e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:38.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397d205640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f397815a3b0 0x7f397814e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56238/0 (socket says 172.21.15.57:56238) 2023-10-30T14:35:38.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397d205640 1 -- 172.21.15.57:0/1584836841 learned_addr learned my addr 172.21.15.57:0/1584836841 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:38.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397ca04640 1 --2- 172.21.15.57:0/1584836841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39780c4a10 0x7f3978155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:38.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f3977fff640 1 --2- 172.21.15.57:0/1584836841 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3978153c80 0x7f397814de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:38.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397d205640 1 -- 172.21.15.57:0/1584836841 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3978153c80 msgr2=0x7f397814de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:38.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397d205640 1 --2- 172.21.15.57:0/1584836841 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3978153c80 0x7f397814de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:38.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397d205640 1 -- 172.21.15.57:0/1584836841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39780c4a10 msgr2=0x7f3978155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:38.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397d205640 1 --2- 172.21.15.57:0/1584836841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39780c4a10 0x7f3978155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:38.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397d205640 1 -- 172.21.15.57:0/1584836841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3970009b30 con 0x7f397815a3b0 2023-10-30T14:35:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397ca04640 1 --2- 172.21.15.57:0/1584836841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39780c4a10 0x7f3978155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f3977fff640 1 --2- 172.21.15.57:0/1584836841 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3978153c80 0x7f397814de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397d205640 1 --2- 172.21.15.57:0/1584836841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f397815a3b0 0x7f397814e370 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f39700b2040 tx=0x7f397001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f39757fa640 1 -- 172.21.15.57:0/1584836841 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39700c0070 con 0x7f397815a3b0 2023-10-30T14:35:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f39757fa640 1 -- 172.21.15.57:0/1584836841 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3970009510 con 0x7f397815a3b0 2023-10-30T14:35:38.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f39757fa640 1 -- 172.21.15.57:0/1584836841 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39700afdc0 con 0x7f397815a3b0 2023-10-30T14:35:38.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397ec68640 1 -- 172.21.15.57:0/1584836841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39780ce380 con 0x7f397815a3b0 2023-10-30T14:35:38.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397ec68640 1 -- 172.21.15.57:0/1584836841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397814feb0 con 0x7f397815a3b0 2023-10-30T14:35:38.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.052+0000 7f397ec68640 1 -- 172.21.15.57:0/1584836841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39781500a0 con 0x7f397815a3b0 2023-10-30T14:35:38.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.056+0000 7f39757fa640 1 -- 172.21.15.57:0/1584836841 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f39700b6040 con 0x7f397815a3b0 2023-10-30T14:35:38.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.056+0000 7f39757fa640 1 --2- 172.21.15.57:0/1584836841 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f395c0664a0 0x7f395c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:38.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.056+0000 7f39757fa640 1 -- 172.21.15.57:0/1584836841 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f3970121ad0 con 0x7f397815a3b0 2023-10-30T14:35:38.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.056+0000 7f397ca04640 1 --2- 172.21.15.57:0/1584836841 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f395c0664a0 0x7f395c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:38.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.056+0000 7f397ca04640 1 --2- 172.21.15.57:0/1584836841 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f395c0664a0 0x7f395c068960 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f396c007a70 tx=0x7f396c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:38.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.060+0000 7f39757fa640 1 -- 172.21.15.57:0/1584836841 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f39700ee9a0 con 0x7f397815a3b0 2023-10-30T14:35:38.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.248+0000 7f397ec68640 1 -- 172.21.15.57:0/1584836841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f397804e170 con 0x7f397815a3b0 2023-10-30T14:35:38.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.248+0000 7f39757fa640 1 -- 172.21.15.57:0/1584836841 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+5741 (secure 0 0 0) 0x7f39700f1fc0 con 0x7f397815a3b0 2023-10-30T14:35:38.251 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:38.252 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:35:36.806292+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:resolve","state_seq":36,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:38.252 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 121 2023-10-30T14:35:38.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.252+0000 7f397ec68640 1 -- 172.21.15.57:0/1584836841 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f395c0664a0 msgr2=0x7f395c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:38.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.252+0000 7f397ec68640 1 --2- 172.21.15.57:0/1584836841 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f395c0664a0 0x7f395c068960 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f396c007a70 tx=0x7f396c008040 comp rx=0 tx=0).stop 2023-10-30T14:35:38.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.252+0000 7f397ec68640 1 -- 172.21.15.57:0/1584836841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f397815a3b0 msgr2=0x7f397814e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:38.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.252+0000 7f397ec68640 1 --2- 172.21.15.57:0/1584836841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f397815a3b0 0x7f397814e370 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f39700b2040 tx=0x7f397001bf60 comp rx=0 tx=0).stop 2023-10-30T14:35:38.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.252+0000 7f397ec68640 1 -- 172.21.15.57:0/1584836841 shutdown_connections 2023-10-30T14:35:38.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.252+0000 7f397ec68640 1 --2- 172.21.15.57:0/1584836841 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f395c0664a0 0x7f395c068960 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:38.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.252+0000 7f397ec68640 1 --2- 172.21.15.57:0/1584836841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f397815a3b0 0x7f397814e370 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:38.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.252+0000 7f397ec68640 1 --2- 172.21.15.57:0/1584836841 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3978153c80 0x7f397814de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:38.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.252+0000 7f397ec68640 1 --2- 172.21.15.57:0/1584836841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39780c4a10 0x7f3978155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:38.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.252+0000 7f397ec68640 1 -- 172.21.15.57:0/1584836841 >> 172.21.15.57:0/1584836841 conn(0x7f39780715c0 msgr2=0x7f3978158640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:38.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.252+0000 7f397ec68640 1 -- 172.21.15.57:0/1584836841 shutdown_connections 2023-10-30T14:35:38.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:38.252+0000 7f397ec68640 1 -- 172.21.15.57:0/1584836841 wait complete. 2023-10-30T14:35:38.272 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:35:40.272 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:40.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.424+0000 7f2f1a372640 1 Processor -- start 2023-10-30T14:35:40.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.424+0000 7f2f1a372640 1 -- start start 2023-10-30T14:35:40.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f1a372640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f140cb7d0 0x7f2f140c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:40.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f1a372640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f140c2310 con 0x7f2f14150830 2023-10-30T14:35:40.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f1a372640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f14156710 con 0x7f2f140c3e40 2023-10-30T14:35:40.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f1a372640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1406bc10 con 0x7f2f140cb7d0 2023-10-30T14:35:40.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f13fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f2f14150830 0x7f2f1406b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:60600/0 (socket says 172.21.15.57:60600) 2023-10-30T14:35:40.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f13fff640 1 -- 172.21.15.57:0/2870080381 learned_addr learned my addr 172.21.15.57:0/2870080381 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:35:40.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f137fe640 1 --2- 172.21.15.57:0/2870080381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f140cb7d0 0x7f2f140c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:40.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f12ffd640 1 -- 172.21.15.57:0/2870080381 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369335527 0 0) 0x7f2f140c2310 con 0x7f2f14150830 2023-10-30T14:35:40.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f12ffd640 1 -- 172.21.15.57:0/2870080381 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2efc003530 con 0x7f2f14150830 2023-10-30T14:35:40.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f12ffd640 1 -- 172.21.15.57:0/2870080381 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3830500365 0 0) 0x7f2f14156710 con 0x7f2f140c3e40 2023-10-30T14:35:40.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f12ffd640 1 -- 172.21.15.57:0/2870080381 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f140c2310 con 0x7f2f140c3e40 2023-10-30T14:35:40.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f12ffd640 1 -- 172.21.15.57:0/2870080381 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 555875319 0 0) 0x7f2efc003530 con 0x7f2f14150830 2023-10-30T14:35:40.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f12ffd640 1 -- 172.21.15.57:0/2870080381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f140cb7d0 msgr2=0x7f2f140c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:40.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f12ffd640 1 --2- 172.21.15.57:0/2870080381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f140cb7d0 0x7f2f140c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:40.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f12ffd640 1 -- 172.21.15.57:0/2870080381 >> v1:172.21.15.88:6789/0 conn(0x7f2f140c3e40 legacy=0x7f2f14070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:40.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f12ffd640 1 -- 172.21.15.57:0/2870080381 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f140c4360 con 0x7f2f14150830 2023-10-30T14:35:40.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f12ffd640 1 -- 172.21.15.57:0/2870080381 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f2f04001680 con 0x7f2f14150830 2023-10-30T14:35:40.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f12ffd640 1 -- 172.21.15.57:0/2870080381 >> v1:172.21.15.57:6789/0 conn(0x7f2f14150830 legacy=0x7f2f1406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:40.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f12ffd640 1 --2- 172.21.15.57:0/2870080381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2efc003b40 0x7f2efc003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:40.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f12ffd640 1 --2- 172.21.15.57:0/2870080381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2efc0047d0 0x7f2efc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:40.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f12ffd640 1 --2- 172.21.15.57:0/2870080381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f140cb7d0 0x7f2efc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:40.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f12ffd640 1 -- 172.21.15.57:0/2870080381 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1406bc10 con 0x7f2efc003b40 2023-10-30T14:35:40.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f12ffd640 1 -- 172.21.15.57:0/2870080381 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc00ba40 con 0x7f2efc0047d0 2023-10-30T14:35:40.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f12ffd640 1 -- 172.21.15.57:0/2870080381 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc00bbd0 con 0x7f2f140cb7d0 2023-10-30T14:35:40.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f13fff640 1 --2- 172.21.15.57:0/2870080381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2efc003b40 0x7f2efc003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:40.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f137fe640 1 --2- 172.21.15.57:0/2870080381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2efc0047d0 0x7f2efc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:40.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f1890f640 1 --2- 172.21.15.57:0/2870080381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f140cb7d0 0x7f2efc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:40.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f13fff640 1 -- 172.21.15.57:0/2870080381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f140cb7d0 msgr2=0x7f2efc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:40.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f13fff640 1 --2- 172.21.15.57:0/2870080381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f140cb7d0 0x7f2efc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:40.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f13fff640 1 -- 172.21.15.57:0/2870080381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2efc0047d0 msgr2=0x7f2efc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:40.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f13fff640 1 --2- 172.21.15.57:0/2870080381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2efc0047d0 0x7f2efc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:40.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f13fff640 1 -- 172.21.15.57:0/2870080381 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc003530 con 0x7f2efc003b40 2023-10-30T14:35:40.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.428+0000 7f2f1890f640 1 --2- 172.21.15.57:0/2870080381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f140cb7d0 0x7f2efc005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:40.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.432+0000 7f2f13fff640 1 --2- 172.21.15.57:0/2870080381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2efc003b40 0x7f2efc003f40 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f2f0400c160 tx=0x7f2f0401cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1c01076c9f4bcfb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:40.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.432+0000 7f2f12ffd640 1 -- 172.21.15.57:0/2870080381 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f040058e0 con 0x7f2efc003b40 2023-10-30T14:35:40.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.432+0000 7f2f12ffd640 1 -- 172.21.15.57:0/2870080381 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f04005a60 con 0x7f2efc003b40 2023-10-30T14:35:40.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.432+0000 7f2f12ffd640 1 -- 172.21.15.57:0/2870080381 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f04005e10 con 0x7f2efc003b40 2023-10-30T14:35:40.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.432+0000 7f2f1a372640 1 -- 172.21.15.57:0/2870080381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2efc003b40 msgr2=0x7f2efc003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:40.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.432+0000 7f2f1a372640 1 --2- 172.21.15.57:0/2870080381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2efc003b40 0x7f2efc003f40 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f2f0400c160 tx=0x7f2f0401cb60 comp rx=0 tx=0).stop 2023-10-30T14:35:40.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.432+0000 7f2f1890f640 1 -- 172.21.15.57:0/2870080381 reap_dead start 2023-10-30T14:35:40.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.432+0000 7f2f1a372640 1 -- 172.21.15.57:0/2870080381 shutdown_connections 2023-10-30T14:35:40.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.432+0000 7f2f1a372640 1 -- 172.21.15.57:0/2870080381 >> 172.21.15.57:0/2870080381 conn(0x7f2f140cac70 msgr2=0x7f2f140cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:40.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.432+0000 7f2f1a372640 1 -- 172.21.15.57:0/2870080381 shutdown_connections 2023-10-30T14:35:40.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.432+0000 7f2f1a372640 1 -- 172.21.15.57:0/2870080381 wait complete. 2023-10-30T14:35:40.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f1a372640 1 Processor -- start 2023-10-30T14:35:40.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f1a372640 1 -- start start 2023-10-30T14:35:40.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f1a372640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2efc003b40 0x7f2f14151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:40.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f1a372640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f14158a10 0x7f2f1414af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:40.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f1a372640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f141f2150 0x7f2f1414f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:40.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f1a372640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f14151c80 con 0x7f2f141f2150 2023-10-30T14:35:40.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f1a372640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f14150f80 con 0x7f2efc003b40 2023-10-30T14:35:40.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f1a372640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f14153640 con 0x7f2f14158a10 2023-10-30T14:35:40.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f1890f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f141f2150 0x7f2f1414f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:40.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f1890f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f141f2150 0x7f2f1414f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56250/0 (socket says 172.21.15.57:56250) 2023-10-30T14:35:40.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f1890f640 1 -- 172.21.15.57:0/1025459500 learned_addr learned my addr 172.21.15.57:0/1025459500 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:40.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f13fff640 1 --2- 172.21.15.57:0/1025459500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2efc003b40 0x7f2f14151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:40.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f137fe640 1 --2- 172.21.15.57:0/1025459500 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f14158a10 0x7f2f1414af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:40.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f1890f640 1 -- 172.21.15.57:0/1025459500 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f14158a10 msgr2=0x7f2f1414af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:40.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f1890f640 1 --2- 172.21.15.57:0/1025459500 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f14158a10 0x7f2f1414af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:40.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f1890f640 1 -- 172.21.15.57:0/1025459500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2efc003b40 msgr2=0x7f2f14151740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:40.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f1890f640 1 --2- 172.21.15.57:0/1025459500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2efc003b40 0x7f2f14151740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:40.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f1890f640 1 -- 172.21.15.57:0/1025459500 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f14153920 con 0x7f2f141f2150 2023-10-30T14:35:40.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f137fe640 1 --2- 172.21.15.57:0/1025459500 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f14158a10 0x7f2f1414af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:40.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f1890f640 1 --2- 172.21.15.57:0/1025459500 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f141f2150 0x7f2f1414f470 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c007e70 tx=0x7f2f0c00f540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:40.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f10ff9640 1 -- 172.21.15.57:0/1025459500 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c00fda0 con 0x7f2f141f2150 2023-10-30T14:35:40.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f10ff9640 1 -- 172.21.15.57:0/1025459500 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f0c015410 con 0x7f2f141f2150 2023-10-30T14:35:40.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f10ff9640 1 -- 172.21.15.57:0/1025459500 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c0157c0 con 0x7f2f141f2150 2023-10-30T14:35:40.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f1a372640 1 -- 172.21.15.57:0/1025459500 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f14153c00 con 0x7f2f141f2150 2023-10-30T14:35:40.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.436+0000 7f2f1a372640 1 -- 172.21.15.57:0/1025459500 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f14153ee0 con 0x7f2f141f2150 2023-10-30T14:35:40.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.440+0000 7f2f1a372640 1 -- 172.21.15.57:0/1025459500 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f1404e170 con 0x7f2f141f2150 2023-10-30T14:35:40.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.440+0000 7f2f10ff9640 1 -- 172.21.15.57:0/1025459500 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2f0c015920 con 0x7f2f141f2150 2023-10-30T14:35:40.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.440+0000 7f2f10ff9640 1 --2- 172.21.15.57:0/1025459500 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f000664d0 0x7f2f00068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:40.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.440+0000 7f2f10ff9640 1 -- 172.21.15.57:0/1025459500 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f2f0c087840 con 0x7f2f141f2150 2023-10-30T14:35:40.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.440+0000 7f2f13fff640 1 --2- 172.21.15.57:0/1025459500 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f000664d0 0x7f2f00068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:40.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.440+0000 7f2f13fff640 1 --2- 172.21.15.57:0/1025459500 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f000664d0 0x7f2f00068990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2f040aa000 tx=0x7f2f04001a00 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:40.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.444+0000 7f2f10ff9640 1 -- 172.21.15.57:0/1025459500 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2f0c054710 con 0x7f2f141f2150 2023-10-30T14:35:40.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.636+0000 7f2f1a372640 1 -- 172.21.15.57:0/1025459500 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f141513f0 con 0x7f2f141f2150 2023-10-30T14:35:40.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.640+0000 7f2f10ff9640 1 -- 172.21.15.57:0/1025459500 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+5741 (secure 0 0 0) 0x7f2f0c057d30 con 0x7f2f141f2150 2023-10-30T14:35:40.641 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:40.642 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:35:36.806292+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:resolve","state_seq":36,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:40.642 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 121 2023-10-30T14:35:40.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.640+0000 7f2f1a372640 1 -- 172.21.15.57:0/1025459500 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f000664d0 msgr2=0x7f2f00068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:40.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.640+0000 7f2f1a372640 1 --2- 172.21.15.57:0/1025459500 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f000664d0 0x7f2f00068990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2f040aa000 tx=0x7f2f04001a00 comp rx=0 tx=0).stop 2023-10-30T14:35:40.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.640+0000 7f2f1a372640 1 -- 172.21.15.57:0/1025459500 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f141f2150 msgr2=0x7f2f1414f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:40.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.640+0000 7f2f1a372640 1 --2- 172.21.15.57:0/1025459500 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f141f2150 0x7f2f1414f470 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c007e70 tx=0x7f2f0c00f540 comp rx=0 tx=0).stop 2023-10-30T14:35:40.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.640+0000 7f2f1a372640 1 -- 172.21.15.57:0/1025459500 shutdown_connections 2023-10-30T14:35:40.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.640+0000 7f2f1a372640 1 --2- 172.21.15.57:0/1025459500 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f000664d0 0x7f2f00068990 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:40.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.640+0000 7f2f1a372640 1 --2- 172.21.15.57:0/1025459500 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f141f2150 0x7f2f1414f470 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:40.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.640+0000 7f2f1a372640 1 --2- 172.21.15.57:0/1025459500 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f14158a10 0x7f2f1414af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:40.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.640+0000 7f2f1a372640 1 --2- 172.21.15.57:0/1025459500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2efc003b40 0x7f2f14151740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:40.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.640+0000 7f2f1a372640 1 -- 172.21.15.57:0/1025459500 >> 172.21.15.57:0/1025459500 conn(0x7f2f140cac70 msgr2=0x7f2f140c55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:40.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.640+0000 7f2f1a372640 1 -- 172.21.15.57:0/1025459500 shutdown_connections 2023-10-30T14:35:40.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:40.640+0000 7f2f1a372640 1 -- 172.21.15.57:0/1025459500 wait complete. 2023-10-30T14:35:40.664 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:35:42.665 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:42.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.812+0000 7f007a7b8640 1 Processor -- start 2023-10-30T14:35:42.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.812+0000 7f007a7b8640 1 -- start start 2023-10-30T14:35:42.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.812+0000 7f007a7b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f007406c420 0x7f00740c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:42.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.812+0000 7f007a7b8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f00740c7170 0x7f0074159980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:42.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.812+0000 7f007a7b8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00740c5f80 con 0x7f0074153bb0 2023-10-30T14:35:42.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.812+0000 7f007a7b8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00740c5690 con 0x7f00740c7170 2023-10-30T14:35:42.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.812+0000 7f007a7b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f007415a580 con 0x7f007406c420 2023-10-30T14:35:42.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.816+0000 7f00737fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f007406c420 0x7f00740c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:42.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.816+0000 7f0078d55640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f00740c7170 0x7f0074159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:42.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.816+0000 7f00737fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f007406c420 0x7f00740c6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:57410/0 (socket says 172.21.15.57:57410) 2023-10-30T14:35:42.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.816+0000 7f00737fe640 1 -- 172.21.15.57:0/2639150577 learned_addr learned my addr 172.21.15.57:0/2639150577 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:42.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.816+0000 7f0078d55640 1 -- 172.21.15.57:0/2639150577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f007406c420 msgr2=0x7f00740c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:42.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.816+0000 7f0078d55640 1 --2- 172.21.15.57:0/2639150577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f007406c420 0x7f00740c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:42.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.816+0000 7f0078d55640 1 -- 172.21.15.57:0/2639150577 >> v1:172.21.15.88:6789/0 conn(0x7f0074153bb0 legacy=0x7f00740c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:42.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.816+0000 7f0078d55640 1 -- 172.21.15.57:0/2639150577 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00740c5930 con 0x7f00740c7170 2023-10-30T14:35:42.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.816+0000 7f00737fe640 1 --2- 172.21.15.57:0/2639150577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f007406c420 0x7f00740c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:35:42.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.816+0000 7f0078d55640 1 --2- 172.21.15.57:0/2639150577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00740c7170 0x7f0074159980 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f006c009d90 tx=0x7f006c01f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4682b71e457e9798 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:42.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.816+0000 7f0072ffd640 1 -- 172.21.15.57:0/2639150577 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006c0af600 con 0x7f00740c7170 2023-10-30T14:35:42.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.816+0000 7f0072ffd640 1 -- 172.21.15.57:0/2639150577 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006c0ab050 con 0x7f00740c7170 2023-10-30T14:35:42.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.816+0000 7f0072ffd640 1 -- 172.21.15.57:0/2639150577 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006c0af980 con 0x7f00740c7170 2023-10-30T14:35:42.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f007a7b8640 1 -- 172.21.15.57:0/2639150577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00740c7170 msgr2=0x7f0074159980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:42.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f007a7b8640 1 --2- 172.21.15.57:0/2639150577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00740c7170 0x7f0074159980 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f006c009d90 tx=0x7f006c01f130 comp rx=0 tx=0).stop 2023-10-30T14:35:42.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f007a7b8640 1 -- 172.21.15.57:0/2639150577 shutdown_connections 2023-10-30T14:35:42.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f007a7b8640 1 --2- 172.21.15.57:0/2639150577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00740c7170 0x7f0074159980 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:42.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f007a7b8640 1 --2- 172.21.15.57:0/2639150577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f007406c420 0x7f00740c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:42.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f007a7b8640 1 -- 172.21.15.57:0/2639150577 >> 172.21.15.57:0/2639150577 conn(0x7f00740cac70 msgr2=0x7f00740cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:42.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f007a7b8640 1 -- 172.21.15.57:0/2639150577 shutdown_connections 2023-10-30T14:35:42.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f007a7b8640 1 -- 172.21.15.57:0/2639150577 wait complete. 2023-10-30T14:35:42.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f007a7b8640 1 Processor -- start 2023-10-30T14:35:42.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f007a7b8640 1 -- start start 2023-10-30T14:35:42.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f007a7b8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f007406c420 0x7f00740ba190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:42.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f007a7b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00740c7170 0x7f00740c08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:42.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f007a7b8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0074153bb0 0x7f0074071100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:42.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f007a7b8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00740c1180 con 0x7f007406c420 2023-10-30T14:35:42.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f007a7b8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00740bd500 con 0x7f0074153bb0 2023-10-30T14:35:42.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f007a7b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00740bd7e0 con 0x7f00740c7170 2023-10-30T14:35:42.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f0073fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f007406c420 0x7f00740ba190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:42.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f0073fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f007406c420 0x7f00740ba190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56266/0 (socket says 172.21.15.57:56266) 2023-10-30T14:35:42.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f0073fff640 1 -- 172.21.15.57:0/1168407644 learned_addr learned my addr 172.21.15.57:0/1168407644 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:42.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f0078d55640 1 --2- 172.21.15.57:0/1168407644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0074153bb0 0x7f0074071100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:42.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.820+0000 7f00737fe640 1 --2- 172.21.15.57:0/1168407644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00740c7170 0x7f00740c08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:42.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.824+0000 7f0073fff640 1 -- 172.21.15.57:0/1168407644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00740c7170 msgr2=0x7f00740c08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:42.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.824+0000 7f0073fff640 1 --2- 172.21.15.57:0/1168407644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00740c7170 0x7f00740c08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:42.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.824+0000 7f0073fff640 1 -- 172.21.15.57:0/1168407644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0074153bb0 msgr2=0x7f0074071100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:42.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.824+0000 7f0073fff640 1 --2- 172.21.15.57:0/1168407644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0074153bb0 0x7f0074071100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:42.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.824+0000 7f0073fff640 1 -- 172.21.15.57:0/1168407644 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00680094d0 con 0x7f007406c420 2023-10-30T14:35:42.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.824+0000 7f0073fff640 1 --2- 172.21.15.57:0/1168407644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f007406c420 0x7f00740ba190 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f006400c3c0 tx=0x7f006400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:42.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.824+0000 7f0070ff9640 1 -- 172.21.15.57:0/1168407644 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00640052d0 con 0x7f007406c420 2023-10-30T14:35:42.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.824+0000 7f0070ff9640 1 -- 172.21.15.57:0/1168407644 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0064005430 con 0x7f007406c420 2023-10-30T14:35:42.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.824+0000 7f0070ff9640 1 -- 172.21.15.57:0/1168407644 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064003840 con 0x7f007406c420 2023-10-30T14:35:42.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.824+0000 7f007a7b8640 1 -- 172.21.15.57:0/1168407644 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f006c009a70 con 0x7f007406c420 2023-10-30T14:35:42.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.824+0000 7f007a7b8640 1 -- 172.21.15.57:0/1168407644 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00740bdda0 con 0x7f007406c420 2023-10-30T14:35:42.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.824+0000 7f007a7b8640 1 -- 172.21.15.57:0/1168407644 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f007404e170 con 0x7f007406c420 2023-10-30T14:35:42.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.824+0000 7f0070ff9640 1 -- 172.21.15.57:0/1168407644 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f00640039a0 con 0x7f007406c420 2023-10-30T14:35:42.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.824+0000 7f0070ff9640 1 --2- 172.21.15.57:0/1168407644 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f00600664d0 0x7f0060068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:42.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.824+0000 7f0070ff9640 1 -- 172.21.15.57:0/1168407644 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f0064083810 con 0x7f007406c420 2023-10-30T14:35:42.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.824+0000 7f00737fe640 1 --2- 172.21.15.57:0/1168407644 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f00600664d0 0x7f0060068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:42.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.828+0000 7f00737fe640 1 --2- 172.21.15.57:0/1168407644 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f00600664d0 0x7f0060068990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f00741515a0 tx=0x7f0068009240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:42.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:42.828+0000 7f0070ff9640 1 -- 172.21.15.57:0/1168407644 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0064050820 con 0x7f007406c420 2023-10-30T14:35:43.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:43.024+0000 7f007a7b8640 1 -- 172.21.15.57:0/1168407644 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00740722b0 con 0x7f007406c420 2023-10-30T14:35:43.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:43.024+0000 7f0070ff9640 1 -- 172.21.15.57:0/1168407644 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5743 (secure 0 0 0) 0x7f0064053e40 con 0x7f007406c420 2023-10-30T14:35:43.026 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:43.027 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:35:42.022506+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:reconnect","state_seq":38,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:43.027 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 122 2023-10-30T14:35:43.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:43.024+0000 7f007a7b8640 1 -- 172.21.15.57:0/1168407644 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f00600664d0 msgr2=0x7f0060068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:43.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:43.024+0000 7f007a7b8640 1 --2- 172.21.15.57:0/1168407644 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f00600664d0 0x7f0060068990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f00741515a0 tx=0x7f0068009240 comp rx=0 tx=0).stop 2023-10-30T14:35:43.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:43.024+0000 7f007a7b8640 1 -- 172.21.15.57:0/1168407644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f007406c420 msgr2=0x7f00740ba190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:43.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:43.024+0000 7f007a7b8640 1 --2- 172.21.15.57:0/1168407644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f007406c420 0x7f00740ba190 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f006400c3c0 tx=0x7f006400c880 comp rx=0 tx=0).stop 2023-10-30T14:35:43.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:43.024+0000 7f007a7b8640 1 -- 172.21.15.57:0/1168407644 shutdown_connections 2023-10-30T14:35:43.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:43.024+0000 7f007a7b8640 1 --2- 172.21.15.57:0/1168407644 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f00600664d0 0x7f0060068990 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:43.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:43.024+0000 7f007a7b8640 1 --2- 172.21.15.57:0/1168407644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0074153bb0 0x7f0074071100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:43.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:43.024+0000 7f007a7b8640 1 --2- 172.21.15.57:0/1168407644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00740c7170 0x7f00740c08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:43.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:43.024+0000 7f007a7b8640 1 --2- 172.21.15.57:0/1168407644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f007406c420 0x7f00740ba190 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:43.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:43.024+0000 7f007a7b8640 1 -- 172.21.15.57:0/1168407644 >> 172.21.15.57:0/1168407644 conn(0x7f00740cac70 msgr2=0x7f00740c9000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:43.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:43.024+0000 7f007a7b8640 1 -- 172.21.15.57:0/1168407644 shutdown_connections 2023-10-30T14:35:43.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:43.024+0000 7f007a7b8640 1 -- 172.21.15.57:0/1168407644 wait complete. 2023-10-30T14:35:43.049 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 122, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 122, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6204": { "addr": "172.21.15.57:6837/1843528309", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 1843528309, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 1843528309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 6204, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_6414": { "addr": "172.21.15.57:6834/2321408149", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2321408149, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2321408149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6414, "incarnation": 99, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 18 }, "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:reconnect", "state_seq": 38 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:35:42.022506+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6414, "mds_1": 6747, "mds_2": 6204 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:35:45.050 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:45.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc8a270f640 1 Processor -- start 2023-10-30T14:35:45.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc8a270f640 1 -- start start 2023-10-30T14:35:45.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc8a270f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc89c06bd10 0x7fc89c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:45.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc8a270f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc89c0c5030 con 0x7fc89c151840 2023-10-30T14:35:45.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc8a270f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc89c157830 con 0x7fc89c151c20 2023-10-30T14:35:45.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc8a270f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c157b10 con 0x7fc89c06bd10 2023-10-30T14:35:45.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc89bfff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc89c151840 0x7fc89c072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48134/0 (socket says 172.21.15.57:48134) 2023-10-30T14:35:45.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc89bfff640 1 -- 172.21.15.57:0/2859717082 learned_addr learned my addr 172.21.15.57:0/2859717082 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:35:45.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc8a0cac640 1 --2- 172.21.15.57:0/2859717082 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc89c06bd10 0x7fc89c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:45.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc89affd640 1 -- 172.21.15.57:0/2859717082 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 491218416 0 0) 0x7fc89c0c5030 con 0x7fc89c151840 2023-10-30T14:35:45.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc89affd640 1 -- 172.21.15.57:0/2859717082 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc884003530 con 0x7fc89c151840 2023-10-30T14:35:45.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc89affd640 1 -- 172.21.15.57:0/2859717082 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1039219066 0 0) 0x7fc884003530 con 0x7fc89c151840 2023-10-30T14:35:45.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc89affd640 1 -- 172.21.15.57:0/2859717082 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc89c06bd10 msgr2=0x7fc89c0c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:45.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc89affd640 1 --2- 172.21.15.57:0/2859717082 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc89c06bd10 0x7fc89c0c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:45.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc89affd640 1 -- 172.21.15.57:0/2859717082 >> v1:172.21.15.88:6789/0 conn(0x7fc89c151c20 legacy=0x7fc89c06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:45.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc89affd640 1 -- 172.21.15.57:0/2859717082 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89c157df0 con 0x7fc89c151840 2023-10-30T14:35:45.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc89affd640 1 -- 172.21.15.57:0/2859717082 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fc88c001680 con 0x7fc89c151840 2023-10-30T14:35:45.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc89affd640 1 -- 172.21.15.57:0/2859717082 >> v1:172.21.15.57:6789/0 conn(0x7fc89c151840 legacy=0x7fc89c072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:45.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc89affd640 1 --2- 172.21.15.57:0/2859717082 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc89c06bd10 0x7fc8840039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:45.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc89affd640 1 --2- 172.21.15.57:0/2859717082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc884003f30 0x7fc8840047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:45.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc89affd640 1 --2- 172.21.15.57:0/2859717082 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc884004d10 0x7fc884005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:45.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc89affd640 1 -- 172.21.15.57:0/2859717082 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c157b10 con 0x7fc884004d10 2023-10-30T14:35:45.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc89affd640 1 -- 172.21.15.57:0/2859717082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88400bb00 con 0x7fc884003f30 2023-10-30T14:35:45.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc89affd640 1 -- 172.21.15.57:0/2859717082 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88400bcc0 con 0x7fc89c06bd10 2023-10-30T14:35:45.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc8a0cac640 1 --2- 172.21.15.57:0/2859717082 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc884004d10 0x7fc884005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:45.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc89bfff640 1 --2- 172.21.15.57:0/2859717082 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc89c06bd10 0x7fc8840039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:45.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc89b7fe640 1 --2- 172.21.15.57:0/2859717082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc884003f30 0x7fc8840047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:45.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc8a0cac640 1 -- 172.21.15.57:0/2859717082 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc89c06bd10 msgr2=0x7fc8840039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:45.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc8a0cac640 1 --2- 172.21.15.57:0/2859717082 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc89c06bd10 0x7fc8840039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:45.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc8a0cac640 1 -- 172.21.15.57:0/2859717082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc884003f30 msgr2=0x7fc8840047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:45.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc8a0cac640 1 --2- 172.21.15.57:0/2859717082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc884003f30 0x7fc8840047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:45.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.204+0000 7fc8a0cac640 1 -- 172.21.15.57:0/2859717082 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc884003530 con 0x7fc884004d10 2023-10-30T14:35:45.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.208+0000 7fc8a0cac640 1 --2- 172.21.15.57:0/2859717082 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc884004d10 0x7fc884005190 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fc894009fc0 tx=0x7fc89401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181941e8e407a2b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:45.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.208+0000 7fc89affd640 1 -- 172.21.15.57:0/2859717082 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8940a5c70 con 0x7fc884004d10 2023-10-30T14:35:45.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.208+0000 7fc89affd640 1 -- 172.21.15.57:0/2859717082 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc89400f040 con 0x7fc884004d10 2023-10-30T14:35:45.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.208+0000 7fc89affd640 1 -- 172.21.15.57:0/2859717082 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8940ad440 con 0x7fc884004d10 2023-10-30T14:35:45.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.208+0000 7fc8a270f640 1 -- 172.21.15.57:0/2859717082 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc884004d10 msgr2=0x7fc884005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:45.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.208+0000 7fc8a270f640 1 --2- 172.21.15.57:0/2859717082 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc884004d10 0x7fc884005190 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fc894009fc0 tx=0x7fc89401cc60 comp rx=0 tx=0).stop 2023-10-30T14:35:45.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.208+0000 7fc8a0cac640 1 -- 172.21.15.57:0/2859717082 reap_dead start 2023-10-30T14:35:45.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.208+0000 7fc8a270f640 1 -- 172.21.15.57:0/2859717082 shutdown_connections 2023-10-30T14:35:45.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.208+0000 7fc8a270f640 1 -- 172.21.15.57:0/2859717082 >> 172.21.15.57:0/2859717082 conn(0x7fc89c0cac70 msgr2=0x7fc89c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:45.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.208+0000 7fc8a270f640 1 -- 172.21.15.57:0/2859717082 shutdown_connections 2023-10-30T14:35:45.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.208+0000 7fc8a270f640 1 -- 172.21.15.57:0/2859717082 wait complete. 2023-10-30T14:35:45.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc8a270f640 1 Processor -- start 2023-10-30T14:35:45.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc8a270f640 1 -- start start 2023-10-30T14:35:45.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc8a270f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc884004d10 0x7fc89c14c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:45.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc8a270f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc89c14e800 0x7fc89c1edd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:45.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc8a270f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc89c0c3370 0x7fc89c1ee2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:45.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc8a270f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c155310 con 0x7fc884004d10 2023-10-30T14:35:45.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc8a270f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c14de10 con 0x7fc89c0c3370 2023-10-30T14:35:45.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc8a270f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c071410 con 0x7fc89c14e800 2023-10-30T14:35:45.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc89bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc884004d10 0x7fc89c14c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:45.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc89bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc884004d10 0x7fc89c14c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57822/0 (socket says 172.21.15.57:57822) 2023-10-30T14:35:45.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc89bfff640 1 -- 172.21.15.57:0/3859394100 learned_addr learned my addr 172.21.15.57:0/3859394100 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:45.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc89b7fe640 1 --2- 172.21.15.57:0/3859394100 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc89c14e800 0x7fc89c1edd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:45.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc8a0cac640 1 --2- 172.21.15.57:0/3859394100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc89c0c3370 0x7fc89c1ee2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:45.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc89bfff640 1 -- 172.21.15.57:0/3859394100 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc89c14e800 msgr2=0x7fc89c1edd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:45.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc89bfff640 1 --2- 172.21.15.57:0/3859394100 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc89c14e800 0x7fc89c1edd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:45.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc89bfff640 1 -- 172.21.15.57:0/3859394100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc89c0c3370 msgr2=0x7fc89c1ee2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:45.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc89bfff640 1 --2- 172.21.15.57:0/3859394100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc89c0c3370 0x7fc89c1ee2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:45.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc89bfff640 1 -- 172.21.15.57:0/3859394100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89c155560 con 0x7fc884004d10 2023-10-30T14:35:45.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc8a0cac640 1 --2- 172.21.15.57:0/3859394100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc89c0c3370 0x7fc89c1ee2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:35:45.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc89bfff640 1 --2- 172.21.15.57:0/3859394100 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc884004d10 0x7fc89c14c210 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fc88c00e8c0 tx=0x7fc88c00ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:45.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc898ff9640 1 -- 172.21.15.57:0/3859394100 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88c00c8d0 con 0x7fc884004d10 2023-10-30T14:35:45.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc898ff9640 1 -- 172.21.15.57:0/3859394100 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc88c00ca30 con 0x7fc884004d10 2023-10-30T14:35:45.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc8a270f640 1 -- 172.21.15.57:0/3859394100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc89c155840 con 0x7fc884004d10 2023-10-30T14:35:45.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc898ff9640 1 -- 172.21.15.57:0/3859394100 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88c00cde0 con 0x7fc884004d10 2023-10-30T14:35:45.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.212+0000 7fc8a270f640 1 -- 172.21.15.57:0/3859394100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc89c14c750 con 0x7fc884004d10 2023-10-30T14:35:45.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.216+0000 7fc8a270f640 1 -- 172.21.15.57:0/3859394100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc89c155cf0 con 0x7fc884004d10 2023-10-30T14:35:45.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.216+0000 7fc898ff9640 1 -- 172.21.15.57:0/3859394100 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc88c01d460 con 0x7fc884004d10 2023-10-30T14:35:45.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.216+0000 7fc898ff9640 1 --2- 172.21.15.57:0/3859394100 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8880664c0 0x7fc888068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:45.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.216+0000 7fc898ff9640 1 -- 172.21.15.57:0/3859394100 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fc88c0844b0 con 0x7fc884004d10 2023-10-30T14:35:45.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.216+0000 7fc89b7fe640 1 --2- 172.21.15.57:0/3859394100 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8880664c0 0x7fc888068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:45.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.216+0000 7fc89b7fe640 1 --2- 172.21.15.57:0/3859394100 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8880664c0 0x7fc888068980 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc89c072cd0 tx=0x7fc890009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:45.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.220+0000 7fc898ff9640 1 -- 172.21.15.57:0/3859394100 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc88c051410 con 0x7fc884004d10 2023-10-30T14:35:45.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.412+0000 7fc8a270f640 1 -- 172.21.15.57:0/3859394100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc89c066510 con 0x7fc884004d10 2023-10-30T14:35:45.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.412+0000 7fc898ff9640 1 -- 172.21.15.57:0/3859394100 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5743 (secure 0 0 0) 0x7fc88c054a30 con 0x7fc884004d10 2023-10-30T14:35:45.414 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:45.414 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:35:42.022506+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:reconnect","state_seq":38,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:45.415 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 122 2023-10-30T14:35:45.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.412+0000 7fc8a270f640 1 -- 172.21.15.57:0/3859394100 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8880664c0 msgr2=0x7fc888068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:45.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.412+0000 7fc8a270f640 1 --2- 172.21.15.57:0/3859394100 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8880664c0 0x7fc888068980 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc89c072cd0 tx=0x7fc890009040 comp rx=0 tx=0).stop 2023-10-30T14:35:45.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.412+0000 7fc8a270f640 1 -- 172.21.15.57:0/3859394100 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc884004d10 msgr2=0x7fc89c14c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:45.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.412+0000 7fc8a270f640 1 --2- 172.21.15.57:0/3859394100 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc884004d10 0x7fc89c14c210 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fc88c00e8c0 tx=0x7fc88c00ed80 comp rx=0 tx=0).stop 2023-10-30T14:35:45.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.412+0000 7fc8a270f640 1 -- 172.21.15.57:0/3859394100 shutdown_connections 2023-10-30T14:35:45.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.412+0000 7fc8a270f640 1 --2- 172.21.15.57:0/3859394100 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8880664c0 0x7fc888068980 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:45.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.412+0000 7fc8a270f640 1 --2- 172.21.15.57:0/3859394100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc89c0c3370 0x7fc89c1ee2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:45.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.412+0000 7fc8a270f640 1 --2- 172.21.15.57:0/3859394100 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc89c14e800 0x7fc89c1edd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:45.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.412+0000 7fc8a270f640 1 --2- 172.21.15.57:0/3859394100 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc884004d10 0x7fc89c14c210 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:45.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.412+0000 7fc8a270f640 1 -- 172.21.15.57:0/3859394100 >> 172.21.15.57:0/3859394100 conn(0x7fc89c0cac70 msgr2=0x7fc89c0c6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:45.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.412+0000 7fc8a270f640 1 -- 172.21.15.57:0/3859394100 shutdown_connections 2023-10-30T14:35:45.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:45.412+0000 7fc8a270f640 1 -- 172.21.15.57:0/3859394100 wait complete. 2023-10-30T14:35:45.434 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:35:47.435 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:47.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.588+0000 7f88b35c5640 1 Processor -- start 2023-10-30T14:35:47.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.588+0000 7f88b35c5640 1 -- start start 2023-10-30T14:35:47.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.588+0000 7f88b35c5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f88ac150830 0x7f88ac06b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:47.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.588+0000 7f88b35c5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88ac0c2310 con 0x7f88ac070520 2023-10-30T14:35:47.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.588+0000 7f88b35c5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88ac156710 con 0x7f88ac0cb7d0 2023-10-30T14:35:47.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.588+0000 7f88b35c5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88ac0c4360 con 0x7f88ac150830 2023-10-30T14:35:47.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.588+0000 7f88b1361640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f88ac150830 0x7f88ac06b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:47.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.588+0000 7f88b1b62640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f88ac070520 0x7f88ac160f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48148/0 (socket says 172.21.15.57:48148) 2023-10-30T14:35:47.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.588+0000 7f88b1b62640 1 -- 172.21.15.57:0/1003267374 learned_addr learned my addr 172.21.15.57:0/1003267374 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:35:47.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.588+0000 7f88b1361640 1 -- 172.21.15.57:0/1003267374 >> v1:172.21.15.88:6789/0 conn(0x7f88ac0cb7d0 legacy=0x7f88ac0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:47.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.588+0000 7f88b1361640 1 -- 172.21.15.57:0/1003267374 >> v1:172.21.15.57:6789/0 conn(0x7f88ac070520 legacy=0x7f88ac160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:47.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.588+0000 7f88b1361640 1 -- 172.21.15.57:0/1003267374 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ac1569b0 con 0x7f88ac150830 2023-10-30T14:35:47.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.588+0000 7f88a3fff640 1 -- 172.21.15.57:0/1003267374 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712102949 0 0) 0x7f88ac0c2310 con 0x7f88ac070520 2023-10-30T14:35:47.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.592+0000 7f88b1361640 1 --2- 172.21.15.57:0/1003267374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88ac150830 0x7f88ac06b500 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f88a80097c0 tx=0x7f88a801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f64be19486620133 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:47.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.592+0000 7f88a3fff640 1 -- 172.21.15.57:0/1003267374 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a80a8970 con 0x7f88ac150830 2023-10-30T14:35:47.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.592+0000 7f88a3fff640 1 -- 172.21.15.57:0/1003267374 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a80a8ad0 con 0x7f88ac150830 2023-10-30T14:35:47.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.592+0000 7f88a3fff640 1 -- 172.21.15.57:0/1003267374 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a80a8e60 con 0x7f88ac150830 2023-10-30T14:35:47.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.592+0000 7f88b35c5640 1 -- 172.21.15.57:0/1003267374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88ac150830 msgr2=0x7f88ac06b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:47.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.592+0000 7f88b35c5640 1 --2- 172.21.15.57:0/1003267374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88ac150830 0x7f88ac06b500 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f88a80097c0 tx=0x7f88a801ec70 comp rx=0 tx=0).stop 2023-10-30T14:35:47.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.592+0000 7f88b35c5640 1 -- 172.21.15.57:0/1003267374 shutdown_connections 2023-10-30T14:35:47.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.592+0000 7f88b35c5640 1 --2- 172.21.15.57:0/1003267374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88ac150830 0x7f88ac06b500 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:47.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.592+0000 7f88b35c5640 1 -- 172.21.15.57:0/1003267374 >> 172.21.15.57:0/1003267374 conn(0x7f88ac0cac70 msgr2=0x7f88ac0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:47.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.592+0000 7f88b35c5640 1 -- 172.21.15.57:0/1003267374 shutdown_connections 2023-10-30T14:35:47.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.592+0000 7f88b35c5640 1 -- 172.21.15.57:0/1003267374 wait complete. 2023-10-30T14:35:47.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b35c5640 1 Processor -- start 2023-10-30T14:35:47.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b35c5640 1 -- start start 2023-10-30T14:35:47.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b35c5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88ac070520 0x7f88ac1ee830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:47.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b35c5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88ac0cb7d0 0x7f88ac14ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:47.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b35c5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88ac150830 0x7f88ac152dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:47.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b35c5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac151e40 con 0x7f88ac150830 2023-10-30T14:35:47.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b35c5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac1512c0 con 0x7f88ac0cb7d0 2023-10-30T14:35:47.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b35c5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac1515a0 con 0x7f88ac070520 2023-10-30T14:35:47.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b1b62640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88ac150830 0x7f88ac152dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:47.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b1b62640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88ac150830 0x7f88ac152dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57838/0 (socket says 172.21.15.57:57838) 2023-10-30T14:35:47.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b1b62640 1 -- 172.21.15.57:0/144689627 learned_addr learned my addr 172.21.15.57:0/144689627 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:47.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b1361640 1 --2- 172.21.15.57:0/144689627 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88ac070520 0x7f88ac1ee830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:47.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b0b60640 1 --2- 172.21.15.57:0/144689627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88ac0cb7d0 0x7f88ac14ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:47.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b1b62640 1 -- 172.21.15.57:0/144689627 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88ac070520 msgr2=0x7f88ac1ee830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:47.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b1b62640 1 --2- 172.21.15.57:0/144689627 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88ac070520 0x7f88ac1ee830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:47.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b1b62640 1 -- 172.21.15.57:0/144689627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88ac0cb7d0 msgr2=0x7f88ac14ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:47.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b1b62640 1 --2- 172.21.15.57:0/144689627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88ac0cb7d0 0x7f88ac14ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:47.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b1b62640 1 -- 172.21.15.57:0/144689627 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a8009420 con 0x7f88ac150830 2023-10-30T14:35:47.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b1b62640 1 --2- 172.21.15.57:0/144689627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88ac150830 0x7f88ac152dc0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f88a400cb30 tx=0x7f88a400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:47.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88a1ffb640 1 -- 172.21.15.57:0/144689627 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a400aca0 con 0x7f88ac150830 2023-10-30T14:35:47.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88a1ffb640 1 -- 172.21.15.57:0/144689627 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a400ae20 con 0x7f88ac150830 2023-10-30T14:35:47.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b35c5640 1 -- 172.21.15.57:0/144689627 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88ac151880 con 0x7f88ac150830 2023-10-30T14:35:47.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88a1ffb640 1 -- 172.21.15.57:0/144689627 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a4012640 con 0x7f88ac150830 2023-10-30T14:35:47.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.596+0000 7f88b35c5640 1 -- 172.21.15.57:0/144689627 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88ac1528a0 con 0x7f88ac150830 2023-10-30T14:35:47.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.600+0000 7f88b35c5640 1 -- 172.21.15.57:0/144689627 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88ac152a90 con 0x7f88ac150830 2023-10-30T14:35:47.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.600+0000 7f88a1ffb640 1 -- 172.21.15.57:0/144689627 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f88a4007070 con 0x7f88ac150830 2023-10-30T14:35:47.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.600+0000 7f88a1ffb640 1 --2- 172.21.15.57:0/144689627 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f88900664a0 0x7f8890068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:47.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.600+0000 7f88a1ffb640 1 -- 172.21.15.57:0/144689627 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f88a40842a0 con 0x7f88ac150830 2023-10-30T14:35:47.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.600+0000 7f88b1361640 1 --2- 172.21.15.57:0/144689627 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f88900664a0 0x7f8890068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:47.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.600+0000 7f88b1361640 1 --2- 172.21.15.57:0/144689627 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f88900664a0 0x7f8890068960 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f88a80b2000 tx=0x7f88a8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:47.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.604+0000 7f88a1ffb640 1 -- 172.21.15.57:0/144689627 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f88a4051340 con 0x7f88ac150830 2023-10-30T14:35:47.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.796+0000 7f88b35c5640 1 -- 172.21.15.57:0/144689627 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88ac04e170 con 0x7f88ac150830 2023-10-30T14:35:47.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.796+0000 7f88a1ffb640 1 -- 172.21.15.57:0/144689627 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+5740 (secure 0 0 0) 0x7f88a4054960 con 0x7f88ac150830 2023-10-30T14:35:47.799 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:47.799 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:35:47.042495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:47.800 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 123 2023-10-30T14:35:47.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.800+0000 7f88b35c5640 1 -- 172.21.15.57:0/144689627 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f88900664a0 msgr2=0x7f8890068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:47.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.800+0000 7f88b35c5640 1 --2- 172.21.15.57:0/144689627 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f88900664a0 0x7f8890068960 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f88a80b2000 tx=0x7f88a8008040 comp rx=0 tx=0).stop 2023-10-30T14:35:47.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.800+0000 7f88b35c5640 1 -- 172.21.15.57:0/144689627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88ac150830 msgr2=0x7f88ac152dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:47.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.800+0000 7f88b35c5640 1 --2- 172.21.15.57:0/144689627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88ac150830 0x7f88ac152dc0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f88a400cb30 tx=0x7f88a400a060 comp rx=0 tx=0).stop 2023-10-30T14:35:47.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.800+0000 7f88b35c5640 1 -- 172.21.15.57:0/144689627 shutdown_connections 2023-10-30T14:35:47.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.800+0000 7f88b35c5640 1 --2- 172.21.15.57:0/144689627 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f88900664a0 0x7f8890068960 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:47.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.800+0000 7f88b35c5640 1 --2- 172.21.15.57:0/144689627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88ac150830 0x7f88ac152dc0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:47.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.800+0000 7f88b35c5640 1 --2- 172.21.15.57:0/144689627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88ac0cb7d0 0x7f88ac14ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:47.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.800+0000 7f88b35c5640 1 --2- 172.21.15.57:0/144689627 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88ac070520 0x7f88ac1ee830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:47.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.800+0000 7f88b35c5640 1 -- 172.21.15.57:0/144689627 >> 172.21.15.57:0/144689627 conn(0x7f88ac0cac70 msgr2=0x7f88ac06b500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:47.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.800+0000 7f88b35c5640 1 -- 172.21.15.57:0/144689627 shutdown_connections 2023-10-30T14:35:47.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:47.800+0000 7f88b35c5640 1 -- 172.21.15.57:0/144689627 wait complete. 2023-10-30T14:35:47.820 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:35:49.821 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:49.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.972+0000 7f3c7613e640 1 Processor -- start 2023-10-30T14:35:49.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.972+0000 7f3c7613e640 1 -- start start 2023-10-30T14:35:49.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.976+0000 7f3c7613e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c7006bd10 0x7f3c700c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:49.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.976+0000 7f3c7613e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c700c5030 con 0x7f3c70151840 2023-10-30T14:35:49.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.976+0000 7f3c7613e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c70157830 con 0x7f3c70151c20 2023-10-30T14:35:49.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.976+0000 7f3c7613e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c70157b10 con 0x7f3c7006bd10 2023-10-30T14:35:49.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.976+0000 7f3c6f7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f3c70151840 0x7f3c70072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48164/0 (socket says 172.21.15.57:48164) 2023-10-30T14:35:49.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.976+0000 7f3c6f7fe640 1 -- 172.21.15.57:0/964081462 learned_addr learned my addr 172.21.15.57:0/964081462 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:35:49.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.976+0000 7f3c6ffff640 1 --2- 172.21.15.57:0/964081462 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c7006bd10 0x7f3c700c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:49.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.976+0000 7f3c6e7fc640 1 -- 172.21.15.57:0/964081462 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106043592 0 0) 0x7f3c700c5030 con 0x7f3c70151840 2023-10-30T14:35:49.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.976+0000 7f3c6e7fc640 1 -- 172.21.15.57:0/964081462 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c58003530 con 0x7f3c70151840 2023-10-30T14:35:49.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.976+0000 7f3c6e7fc640 1 -- 172.21.15.57:0/964081462 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165946108 0 0) 0x7f3c70157830 con 0x7f3c70151c20 2023-10-30T14:35:49.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.976+0000 7f3c6e7fc640 1 -- 172.21.15.57:0/964081462 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c700c5030 con 0x7f3c70151c20 2023-10-30T14:35:49.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.976+0000 7f3c6ffff640 1 -- 172.21.15.57:0/964081462 >> v1:172.21.15.88:6789/0 conn(0x7f3c70151c20 legacy=0x7f3c7006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:49.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.976+0000 7f3c6ffff640 1 -- 172.21.15.57:0/964081462 >> v1:172.21.15.57:6789/0 conn(0x7f3c70151840 legacy=0x7f3c70072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:49.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.976+0000 7f3c6e7fc640 1 -- 172.21.15.57:0/964081462 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1416897003 0 0) 0x7f3c58003530 con 0x7f3c70151840 2023-10-30T14:35:49.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.976+0000 7f3c6ffff640 1 -- 172.21.15.57:0/964081462 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c70157df0 con 0x7f3c7006bd10 2023-10-30T14:35:49.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.980+0000 7f3c6ffff640 1 --2- 172.21.15.57:0/964081462 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c7006bd10 0x7f3c700c49a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3c68009dc0 tx=0x7f3c6801f080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8aec7b6d2de8f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:49.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.980+0000 7f3c6e7fc640 1 -- 172.21.15.57:0/964081462 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c60001680 con 0x7f3c7006bd10 2023-10-30T14:35:49.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.980+0000 7f3c6e7fc640 1 -- 172.21.15.57:0/964081462 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c680a8c10 con 0x7f3c7006bd10 2023-10-30T14:35:49.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.980+0000 7f3c6e7fc640 1 -- 172.21.15.57:0/964081462 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c680b0440 con 0x7f3c7006bd10 2023-10-30T14:35:49.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.980+0000 7f3c7613e640 1 -- 172.21.15.57:0/964081462 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c7006bd10 msgr2=0x7f3c700c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:49.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.980+0000 7f3c7613e640 1 --2- 172.21.15.57:0/964081462 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c7006bd10 0x7f3c700c49a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3c68009dc0 tx=0x7f3c6801f080 comp rx=0 tx=0).stop 2023-10-30T14:35:49.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.980+0000 7f3c7613e640 1 -- 172.21.15.57:0/964081462 shutdown_connections 2023-10-30T14:35:49.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.980+0000 7f3c7613e640 1 --2- 172.21.15.57:0/964081462 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c7006bd10 0x7f3c700c49a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:49.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.980+0000 7f3c7613e640 1 -- 172.21.15.57:0/964081462 >> 172.21.15.57:0/964081462 conn(0x7f3c700cac70 msgr2=0x7f3c700cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:49.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.980+0000 7f3c7613e640 1 -- 172.21.15.57:0/964081462 shutdown_connections 2023-10-30T14:35:49.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.980+0000 7f3c7613e640 1 -- 172.21.15.57:0/964081462 wait complete. 2023-10-30T14:35:49.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.980+0000 7f3c7613e640 1 Processor -- start 2023-10-30T14:35:49.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.980+0000 7f3c7613e640 1 -- start start 2023-10-30T14:35:49.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c7613e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c7006bd10 0x7f3c701dfe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:49.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c7613e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c70151840 0x7f3c701eaff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:49.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c7613e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c70151c20 0x7f3c70142480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:49.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c7613e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c701edcf0 con 0x7f3c7006bd10 2023-10-30T14:35:49.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c7613e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c701edf90 con 0x7f3c70151c20 2023-10-30T14:35:49.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c7613e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c701ee250 con 0x7f3c70151840 2023-10-30T14:35:49.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c6f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c7006bd10 0x7f3c701dfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:49.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c6f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c7006bd10 0x7f3c701dfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57842/0 (socket says 172.21.15.57:57842) 2023-10-30T14:35:49.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c6f7fe640 1 -- 172.21.15.57:0/393283177 learned_addr learned my addr 172.21.15.57:0/393283177 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:49.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c6ffff640 1 --2- 172.21.15.57:0/393283177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c70151c20 0x7f3c70142480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:49.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c6effd640 1 --2- 172.21.15.57:0/393283177 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c70151840 0x7f3c701eaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:49.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c6f7fe640 1 -- 172.21.15.57:0/393283177 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c70151840 msgr2=0x7f3c701eaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:49.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c6f7fe640 1 --2- 172.21.15.57:0/393283177 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c70151840 0x7f3c701eaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:49.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c6f7fe640 1 -- 172.21.15.57:0/393283177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c70151c20 msgr2=0x7f3c70142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:49.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c6f7fe640 1 --2- 172.21.15.57:0/393283177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c70151c20 0x7f3c70142480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:49.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c6f7fe640 1 -- 172.21.15.57:0/393283177 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c68009a70 con 0x7f3c7006bd10 2023-10-30T14:35:49.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c6f7fe640 1 --2- 172.21.15.57:0/393283177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c7006bd10 0x7f3c701dfe70 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f3c6000c750 tx=0x7f3c6000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:49.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c57fff640 1 -- 172.21.15.57:0/393283177 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c60005af0 con 0x7f3c7006bd10 2023-10-30T14:35:49.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c57fff640 1 -- 172.21.15.57:0/393283177 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c60005c50 con 0x7f3c7006bd10 2023-10-30T14:35:49.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c7613e640 1 -- 172.21.15.57:0/393283177 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c701ee530 con 0x7f3c7006bd10 2023-10-30T14:35:49.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c57fff640 1 -- 172.21.15.57:0/393283177 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c60007070 con 0x7f3c7006bd10 2023-10-30T14:35:49.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c7613e640 1 -- 172.21.15.57:0/393283177 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c701eeaf0 con 0x7f3c7006bd10 2023-10-30T14:35:49.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.984+0000 7f3c7613e640 1 -- 172.21.15.57:0/393283177 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c7004e170 con 0x7f3c7006bd10 2023-10-30T14:35:49.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.988+0000 7f3c57fff640 1 -- 172.21.15.57:0/393283177 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3c600071d0 con 0x7f3c7006bd10 2023-10-30T14:35:49.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.988+0000 7f3c57fff640 1 --2- 172.21.15.57:0/393283177 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3c5c0664d0 0x7f3c5c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:49.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.988+0000 7f3c57fff640 1 -- 172.21.15.57:0/393283177 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f3c60083b70 con 0x7f3c7006bd10 2023-10-30T14:35:49.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.988+0000 7f3c6effd640 1 --2- 172.21.15.57:0/393283177 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3c5c0664d0 0x7f3c5c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:49.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.988+0000 7f3c6effd640 1 --2- 172.21.15.57:0/393283177 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3c5c0664d0 0x7f3c5c068990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3c70150100 tx=0x7f3c64009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:49.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:49.992+0000 7f3c57fff640 1 -- 172.21.15.57:0/393283177 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3c60054580 con 0x7f3c7006bd10 2023-10-30T14:35:50.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:50.184+0000 7f3c7613e640 1 -- 172.21.15.57:0/393283177 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c70073610 con 0x7f3c7006bd10 2023-10-30T14:35:50.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:50.184+0000 7f3c57fff640 1 -- 172.21.15.57:0/393283177 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+5740 (secure 0 0 0) 0x7f3c60017020 con 0x7f3c7006bd10 2023-10-30T14:35:50.188 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:50.188 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:35:47.042495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:50.189 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 123 2023-10-30T14:35:50.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:50.188+0000 7f3c7613e640 1 -- 172.21.15.57:0/393283177 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3c5c0664d0 msgr2=0x7f3c5c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:50.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:50.188+0000 7f3c7613e640 1 --2- 172.21.15.57:0/393283177 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3c5c0664d0 0x7f3c5c068990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3c70150100 tx=0x7f3c64009040 comp rx=0 tx=0).stop 2023-10-30T14:35:50.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:50.188+0000 7f3c7613e640 1 -- 172.21.15.57:0/393283177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c7006bd10 msgr2=0x7f3c701dfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:50.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:50.188+0000 7f3c7613e640 1 --2- 172.21.15.57:0/393283177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c7006bd10 0x7f3c701dfe70 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f3c6000c750 tx=0x7f3c6000cc10 comp rx=0 tx=0).stop 2023-10-30T14:35:50.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:50.188+0000 7f3c7613e640 1 -- 172.21.15.57:0/393283177 shutdown_connections 2023-10-30T14:35:50.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:50.188+0000 7f3c7613e640 1 --2- 172.21.15.57:0/393283177 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3c5c0664d0 0x7f3c5c068990 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:50.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:50.188+0000 7f3c7613e640 1 --2- 172.21.15.57:0/393283177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c70151c20 0x7f3c70142480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:50.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:50.188+0000 7f3c7613e640 1 --2- 172.21.15.57:0/393283177 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c70151840 0x7f3c701eaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:50.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:50.188+0000 7f3c7613e640 1 --2- 172.21.15.57:0/393283177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c7006bd10 0x7f3c701dfe70 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:50.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:50.188+0000 7f3c7613e640 1 -- 172.21.15.57:0/393283177 >> 172.21.15.57:0/393283177 conn(0x7f3c700cac70 msgr2=0x7f3c70150ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:50.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:50.188+0000 7f3c7613e640 1 -- 172.21.15.57:0/393283177 shutdown_connections 2023-10-30T14:35:50.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:50.188+0000 7f3c7613e640 1 -- 172.21.15.57:0/393283177 wait complete. 2023-10-30T14:35:50.209 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:35:52.209 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:52.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.355+0000 7fc5a3b9d640 1 Processor -- start 2023-10-30T14:35:52.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.355+0000 7fc5a3b9d640 1 -- start start 2023-10-30T14:35:52.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.359+0000 7fc5a3b9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc59c151c20 0x7fc59c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:52.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.359+0000 7fc5a3b9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc59c0cb8b0 0x7fc59c06b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:52.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.359+0000 7fc5a3b9d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc59c0c5030 con 0x7fc59c151840 2023-10-30T14:35:52.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.359+0000 7fc5a3b9d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc59c157830 con 0x7fc59c151c20 2023-10-30T14:35:52.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.359+0000 7fc5a3b9d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c157b10 con 0x7fc59c0cb8b0 2023-10-30T14:35:52.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.359+0000 7fc5a1138640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc59c151c20 0x7fc59c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:52.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.359+0000 7fc5a1138640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc59c151c20 0x7fc59c0c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57848/0 (socket says 172.21.15.57:57848) 2023-10-30T14:35:52.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.359+0000 7fc5a1138640 1 -- 172.21.15.57:0/1502446562 learned_addr learned my addr 172.21.15.57:0/1502446562 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:52.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.359+0000 7fc5a213a640 1 --2- 172.21.15.57:0/1502446562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc59c0cb8b0 0x7fc59c06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:52.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.359+0000 7fc5a1138640 1 -- 172.21.15.57:0/1502446562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc59c0cb8b0 msgr2=0x7fc59c06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:52.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.359+0000 7fc5a1138640 1 --2- 172.21.15.57:0/1502446562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc59c0cb8b0 0x7fc59c06b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:52.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.359+0000 7fc5a1138640 1 -- 172.21.15.57:0/1502446562 >> v1:172.21.15.57:6789/0 conn(0x7fc59c151840 legacy=0x7fc59c072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:52.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.359+0000 7fc5a1138640 1 -- 172.21.15.57:0/1502446562 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc59c157df0 con 0x7fc59c151c20 2023-10-30T14:35:52.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.363+0000 7fc5a1138640 1 --2- 172.21.15.57:0/1502446562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc59c151c20 0x7fc59c0c49a0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fc58c009870 tx=0x7fc58c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a4abf68398fe575 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:52.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.363+0000 7fc5a0937640 1 -- 172.21.15.57:0/1502446562 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c0a8a30 con 0x7fc59c151c20 2023-10-30T14:35:52.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.363+0000 7fc5a0937640 1 -- 172.21.15.57:0/1502446562 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58c0a8b90 con 0x7fc59c151c20 2023-10-30T14:35:52.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.363+0000 7fc5a0937640 1 -- 172.21.15.57:0/1502446562 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c0b0440 con 0x7fc59c151c20 2023-10-30T14:35:52.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.363+0000 7fc5a3b9d640 1 -- 172.21.15.57:0/1502446562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc59c151c20 msgr2=0x7fc59c0c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:52.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.363+0000 7fc5a3b9d640 1 --2- 172.21.15.57:0/1502446562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc59c151c20 0x7fc59c0c49a0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fc58c009870 tx=0x7fc58c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:35:52.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.363+0000 7fc5a3b9d640 1 -- 172.21.15.57:0/1502446562 shutdown_connections 2023-10-30T14:35:52.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.363+0000 7fc5a3b9d640 1 --2- 172.21.15.57:0/1502446562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc59c0cb8b0 0x7fc59c06b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:52.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.363+0000 7fc5a3b9d640 1 --2- 172.21.15.57:0/1502446562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc59c151c20 0x7fc59c0c49a0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:52.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.363+0000 7fc5a3b9d640 1 -- 172.21.15.57:0/1502446562 >> 172.21.15.57:0/1502446562 conn(0x7fc59c0cac70 msgr2=0x7fc59c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:52.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.363+0000 7fc5a3b9d640 1 -- 172.21.15.57:0/1502446562 shutdown_connections 2023-10-30T14:35:52.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.363+0000 7fc5a3b9d640 1 -- 172.21.15.57:0/1502446562 wait complete. 2023-10-30T14:35:52.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.363+0000 7fc5a3b9d640 1 Processor -- start 2023-10-30T14:35:52.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.363+0000 7fc5a3b9d640 1 -- start start 2023-10-30T14:35:52.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a3b9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc59c0cb8b0 0x7fc59c073620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:52.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a3b9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc59c151840 0x7fc59c0c8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:52.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a3b9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc59c151c20 0x7fc59c0c9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:52.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a3b9d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c1e2090 con 0x7fc59c151840 2023-10-30T14:35:52.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a3b9d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c1ef810 con 0x7fc59c151c20 2023-10-30T14:35:52.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a3b9d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c1ee9b0 con 0x7fc59c0cb8b0 2023-10-30T14:35:52.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a1138640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc59c151840 0x7fc59c0c8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:52.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a1138640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc59c151840 0x7fc59c0c8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57862/0 (socket says 172.21.15.57:57862) 2023-10-30T14:35:52.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a1138640 1 -- 172.21.15.57:0/2254040335 learned_addr learned my addr 172.21.15.57:0/2254040335 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:52.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a1939640 1 --2- 172.21.15.57:0/2254040335 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc59c0cb8b0 0x7fc59c073620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:52.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a213a640 1 --2- 172.21.15.57:0/2254040335 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc59c151c20 0x7fc59c0c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:52.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a1138640 1 -- 172.21.15.57:0/2254040335 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc59c0cb8b0 msgr2=0x7fc59c073620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:52.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a1138640 1 --2- 172.21.15.57:0/2254040335 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc59c0cb8b0 0x7fc59c073620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:52.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a1138640 1 -- 172.21.15.57:0/2254040335 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc59c151c20 msgr2=0x7fc59c0c9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:52.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a1138640 1 --2- 172.21.15.57:0/2254040335 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc59c151c20 0x7fc59c0c9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:52.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a1138640 1 -- 172.21.15.57:0/2254040335 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58c0094d0 con 0x7fc59c151840 2023-10-30T14:35:52.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a213a640 1 --2- 172.21.15.57:0/2254040335 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc59c151c20 0x7fc59c0c9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:52.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a1939640 1 --2- 172.21.15.57:0/2254040335 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc59c0cb8b0 0x7fc59c073620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:52.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a1138640 1 --2- 172.21.15.57:0/2254040335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc59c151840 0x7fc59c0c8df0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fc58c01d3e0 tx=0x7fc58c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:52.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5927fc640 1 -- 172.21.15.57:0/2254040335 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c0ba600 con 0x7fc59c151840 2023-10-30T14:35:52.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5927fc640 1 -- 172.21.15.57:0/2254040335 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58c0ba780 con 0x7fc59c151840 2023-10-30T14:35:52.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a3b9d640 1 -- 172.21.15.57:0/2254040335 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59c1eec90 con 0x7fc59c151840 2023-10-30T14:35:52.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5927fc640 1 -- 172.21.15.57:0/2254040335 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c0bab30 con 0x7fc59c151840 2023-10-30T14:35:52.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a3b9d640 1 -- 172.21.15.57:0/2254040335 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59c1ef250 con 0x7fc59c151840 2023-10-30T14:35:52.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5a3b9d640 1 -- 172.21.15.57:0/2254040335 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc59c04e170 con 0x7fc59c151840 2023-10-30T14:35:52.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.367+0000 7fc5927fc640 1 -- 172.21.15.57:0/2254040335 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc58c0bac90 con 0x7fc59c151840 2023-10-30T14:35:52.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.371+0000 7fc5927fc640 1 --2- 172.21.15.57:0/2254040335 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5800664d0 0x7fc580068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:52.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.371+0000 7fc5927fc640 1 -- 172.21.15.57:0/2254040335 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fc58c122860 con 0x7fc59c151840 2023-10-30T14:35:52.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.371+0000 7fc5a1939640 1 --2- 172.21.15.57:0/2254040335 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5800664d0 0x7fc580068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:52.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.371+0000 7fc5a1939640 1 --2- 172.21.15.57:0/2254040335 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5800664d0 0x7fc580068990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc598000f70 tx=0x7fc598009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:52.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.375+0000 7fc5927fc640 1 -- 172.21.15.57:0/2254040335 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc58c0ef7c0 con 0x7fc59c151840 2023-10-30T14:35:52.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.563+0000 7fc5a3b9d640 1 -- 172.21.15.57:0/2254040335 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc59c066510 con 0x7fc59c151840 2023-10-30T14:35:52.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.567+0000 7fc5927fc640 1 -- 172.21.15.57:0/2254040335 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+5740 (secure 0 0 0) 0x7fc58c0f2de0 con 0x7fc59c151840 2023-10-30T14:35:52.569 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:52.570 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:35:47.042495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:52.570 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 123 2023-10-30T14:35:52.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.567+0000 7fc5a3b9d640 1 -- 172.21.15.57:0/2254040335 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5800664d0 msgr2=0x7fc580068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:52.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.567+0000 7fc5a3b9d640 1 --2- 172.21.15.57:0/2254040335 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5800664d0 0x7fc580068990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc598000f70 tx=0x7fc598009040 comp rx=0 tx=0).stop 2023-10-30T14:35:52.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.567+0000 7fc5a3b9d640 1 -- 172.21.15.57:0/2254040335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc59c151840 msgr2=0x7fc59c0c8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:52.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.567+0000 7fc5a3b9d640 1 --2- 172.21.15.57:0/2254040335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc59c151840 0x7fc59c0c8df0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fc58c01d3e0 tx=0x7fc58c01bad0 comp rx=0 tx=0).stop 2023-10-30T14:35:52.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.567+0000 7fc5a3b9d640 1 -- 172.21.15.57:0/2254040335 shutdown_connections 2023-10-30T14:35:52.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.567+0000 7fc5a3b9d640 1 --2- 172.21.15.57:0/2254040335 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc5800664d0 0x7fc580068990 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:52.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.567+0000 7fc5a3b9d640 1 --2- 172.21.15.57:0/2254040335 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc59c151c20 0x7fc59c0c9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:52.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.567+0000 7fc5a3b9d640 1 --2- 172.21.15.57:0/2254040335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc59c151840 0x7fc59c0c8df0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:52.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.567+0000 7fc5a3b9d640 1 --2- 172.21.15.57:0/2254040335 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc59c0cb8b0 0x7fc59c073620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:52.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.567+0000 7fc5a3b9d640 1 -- 172.21.15.57:0/2254040335 >> 172.21.15.57:0/2254040335 conn(0x7fc59c0cac70 msgr2=0x7fc59c0cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:52.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.567+0000 7fc5a3b9d640 1 -- 172.21.15.57:0/2254040335 shutdown_connections 2023-10-30T14:35:52.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:52.567+0000 7fc5a3b9d640 1 -- 172.21.15.57:0/2254040335 wait complete. 2023-10-30T14:35:52.590 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:35:54.591 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:54.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.743+0000 7f80b913e640 1 Processor -- start 2023-10-30T14:35:54.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.743+0000 7f80b913e640 1 -- start start 2023-10-30T14:35:54.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.743+0000 7f80b913e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80b4150830 0x7f80b406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:54.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.743+0000 7f80b913e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f80b40cb7d0 0x7f80b40c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:54.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.743+0000 7f80b913e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f80b40c3e40 0x7f80b4070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:54.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.743+0000 7f80b913e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b40c2310 con 0x7f80b40c3e40 2023-10-30T14:35:54.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.743+0000 7f80b913e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b4156710 con 0x7f80b40cb7d0 2023-10-30T14:35:54.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.743+0000 7f80b913e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b40c4360 con 0x7f80b4150830 2023-10-30T14:35:54.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.743+0000 7f80b359e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f80b40c3e40 0x7f80b4070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:54.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.743+0000 7f80b359e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f80b40c3e40 0x7f80b4070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50184/0 (socket says 172.21.15.57:50184) 2023-10-30T14:35:54.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.743+0000 7f80b359e640 1 -- 172.21.15.57:0/2656886975 learned_addr learned my addr 172.21.15.57:0/2656886975 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:54.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.743+0000 7f80b2d9d640 1 --2- 172.21.15.57:0/2656886975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80b4150830 0x7f80b406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:54.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.743+0000 7f80abfff640 1 --2- 172.21.15.57:0/2656886975 >> v2:172.21.15.88:3300/0 conn(0x7f80b40cb7d0 0x7f80b40c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:54.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.747+0000 7f80b359e640 1 -- 172.21.15.57:0/2656886975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80b4150830 msgr2=0x7f80b406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:54.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.747+0000 7f80b359e640 1 --2- 172.21.15.57:0/2656886975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80b4150830 0x7f80b406b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:54.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.747+0000 7f80b359e640 1 -- 172.21.15.57:0/2656886975 >> v2:172.21.15.88:3300/0 conn(0x7f80b40cb7d0 msgr2=0x7f80b40c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:54.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.747+0000 7f80b359e640 1 --2- 172.21.15.57:0/2656886975 >> v2:172.21.15.88:3300/0 conn(0x7f80b40cb7d0 0x7f80b40c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:54.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.747+0000 7f80b359e640 1 -- 172.21.15.57:0/2656886975 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b41569b0 con 0x7f80b40c3e40 2023-10-30T14:35:54.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.747+0000 7f80b2d9d640 1 --2- 172.21.15.57:0/2656886975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80b4150830 0x7f80b406b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:54.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.747+0000 7f80b359e640 1 --2- 172.21.15.57:0/2656886975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80b40c3e40 0x7f80b4070520 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f80b40cbd70 tx=0x7f80ac01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aac4158db3bcbbd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:54.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.747+0000 7f80b259c640 1 -- 172.21.15.57:0/2656886975 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac0af400 con 0x7f80b40c3e40 2023-10-30T14:35:54.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.747+0000 7f80b259c640 1 -- 172.21.15.57:0/2656886975 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac0ab050 con 0x7f80b40c3e40 2023-10-30T14:35:54.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.747+0000 7f80b259c640 1 -- 172.21.15.57:0/2656886975 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac0af780 con 0x7f80b40c3e40 2023-10-30T14:35:54.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.747+0000 7f80b913e640 1 -- 172.21.15.57:0/2656886975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80b40c3e40 msgr2=0x7f80b4070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:54.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.747+0000 7f80b913e640 1 --2- 172.21.15.57:0/2656886975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80b40c3e40 0x7f80b4070520 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f80b40cbd70 tx=0x7f80ac01f1c0 comp rx=0 tx=0).stop 2023-10-30T14:35:54.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.747+0000 7f80b913e640 1 -- 172.21.15.57:0/2656886975 shutdown_connections 2023-10-30T14:35:54.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.747+0000 7f80b913e640 1 --2- 172.21.15.57:0/2656886975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80b40c3e40 0x7f80b4070520 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:54.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.747+0000 7f80b913e640 1 --2- 172.21.15.57:0/2656886975 >> v2:172.21.15.88:3300/0 conn(0x7f80b40cb7d0 0x7f80b40c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:54.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.747+0000 7f80b913e640 1 --2- 172.21.15.57:0/2656886975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80b4150830 0x7f80b406b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:54.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.747+0000 7f80b913e640 1 -- 172.21.15.57:0/2656886975 >> 172.21.15.57:0/2656886975 conn(0x7f80b40cac70 msgr2=0x7f80b40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:54.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.747+0000 7f80b913e640 1 -- 172.21.15.57:0/2656886975 shutdown_connections 2023-10-30T14:35:54.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b913e640 1 -- 172.21.15.57:0/2656886975 wait complete. 2023-10-30T14:35:54.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b913e640 1 Processor -- start 2023-10-30T14:35:54.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b913e640 1 -- start start 2023-10-30T14:35:54.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b913e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80b40c3e40 0x7f80b4159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:54.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b913e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80b40cb7d0 0x7f80b4149c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:54.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b913e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80b4150830 0x7f80b414a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:54.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b913e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b414ac20 con 0x7f80b40c3e40 2023-10-30T14:35:54.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b913e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b4148bf0 con 0x7f80b4150830 2023-10-30T14:35:54.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b913e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b41ed4c0 con 0x7f80b40cb7d0 2023-10-30T14:35:54.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80b40c3e40 0x7f80b4159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:54.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80b40c3e40 0x7f80b4159780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50192/0 (socket says 172.21.15.57:50192) 2023-10-30T14:35:54.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b2d9d640 1 -- 172.21.15.57:0/959028313 learned_addr learned my addr 172.21.15.57:0/959028313 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:54.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80abfff640 1 --2- 172.21.15.57:0/959028313 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80b40cb7d0 0x7f80b4149c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:54.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b359e640 1 --2- 172.21.15.57:0/959028313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80b4150830 0x7f80b414a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:54.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b2d9d640 1 -- 172.21.15.57:0/959028313 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80b40cb7d0 msgr2=0x7f80b4149c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:54.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b2d9d640 1 --2- 172.21.15.57:0/959028313 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80b40cb7d0 0x7f80b4149c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:54.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b2d9d640 1 -- 172.21.15.57:0/959028313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80b4150830 msgr2=0x7f80b414a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:54.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b2d9d640 1 --2- 172.21.15.57:0/959028313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80b4150830 0x7f80b414a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:54.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b2d9d640 1 -- 172.21.15.57:0/959028313 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac009b30 con 0x7f80b40c3e40 2023-10-30T14:35:54.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b359e640 1 --2- 172.21.15.57:0/959028313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80b4150830 0x7f80b414a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:54.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80b2d9d640 1 --2- 172.21.15.57:0/959028313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80b40c3e40 0x7f80b4159780 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f80a400e810 tx=0x7f80a400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:54.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.751+0000 7f80ab7fe640 1 -- 172.21.15.57:0/959028313 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a400c8d0 con 0x7f80b40c3e40 2023-10-30T14:35:54.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.755+0000 7f80ab7fe640 1 -- 172.21.15.57:0/959028313 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a400ca30 con 0x7f80b40c3e40 2023-10-30T14:35:54.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.755+0000 7f80ab7fe640 1 -- 172.21.15.57:0/959028313 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a400cdc0 con 0x7f80b40c3e40 2023-10-30T14:35:54.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.755+0000 7f80b913e640 1 -- 172.21.15.57:0/959028313 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b4148e40 con 0x7f80b40c3e40 2023-10-30T14:35:54.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.755+0000 7f80b913e640 1 -- 172.21.15.57:0/959028313 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b4149400 con 0x7f80b40c3e40 2023-10-30T14:35:54.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.755+0000 7f80b913e640 1 -- 172.21.15.57:0/959028313 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b404e170 con 0x7f80b40c3e40 2023-10-30T14:35:54.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.755+0000 7f80ab7fe640 1 -- 172.21.15.57:0/959028313 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f80a4013410 con 0x7f80b40c3e40 2023-10-30T14:35:54.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.755+0000 7f80ab7fe640 1 --2- 172.21.15.57:0/959028313 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80940664a0 0x7f8094068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:54.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.755+0000 7f80ab7fe640 1 -- 172.21.15.57:0/959028313 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f80a4084200 con 0x7f80b40c3e40 2023-10-30T14:35:54.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.755+0000 7f80abfff640 1 --2- 172.21.15.57:0/959028313 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80940664a0 0x7f8094068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:54.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.759+0000 7f80abfff640 1 --2- 172.21.15.57:0/959028313 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80940664a0 0x7f8094068960 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f80b41562f0 tx=0x7f809c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:54.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.759+0000 7f80ab7fe640 1 -- 172.21.15.57:0/959028313 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f80a40510d0 con 0x7f80b40c3e40 2023-10-30T14:35:54.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.947+0000 7f80b913e640 1 -- 172.21.15.57:0/959028313 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80b4070300 con 0x7f80b40c3e40 2023-10-30T14:35:54.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.947+0000 7f80ab7fe640 1 -- 172.21.15.57:0/959028313 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+5746 (secure 0 0 0) 0x7f80a40546f0 con 0x7f80b40c3e40 2023-10-30T14:35:54.953 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:54.953 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:35:52.120615+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:clientreplay","state_seq":42,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:54.953 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 124 2023-10-30T14:35:54.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.951+0000 7f80b913e640 1 -- 172.21.15.57:0/959028313 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80940664a0 msgr2=0x7f8094068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:54.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.951+0000 7f80b913e640 1 --2- 172.21.15.57:0/959028313 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80940664a0 0x7f8094068960 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f80b41562f0 tx=0x7f809c008040 comp rx=0 tx=0).stop 2023-10-30T14:35:54.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.951+0000 7f80b913e640 1 -- 172.21.15.57:0/959028313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80b40c3e40 msgr2=0x7f80b4159780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:54.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.951+0000 7f80b913e640 1 --2- 172.21.15.57:0/959028313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80b40c3e40 0x7f80b4159780 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f80a400e810 tx=0x7f80a400ecd0 comp rx=0 tx=0).stop 2023-10-30T14:35:54.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.951+0000 7f80b913e640 1 -- 172.21.15.57:0/959028313 shutdown_connections 2023-10-30T14:35:54.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.951+0000 7f80b913e640 1 --2- 172.21.15.57:0/959028313 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80940664a0 0x7f8094068960 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:54.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.951+0000 7f80b913e640 1 --2- 172.21.15.57:0/959028313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80b4150830 0x7f80b414a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:54.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.951+0000 7f80b913e640 1 --2- 172.21.15.57:0/959028313 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80b40cb7d0 0x7f80b4149c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:54.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.951+0000 7f80b913e640 1 --2- 172.21.15.57:0/959028313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80b40c3e40 0x7f80b4159780 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:54.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.951+0000 7f80b913e640 1 -- 172.21.15.57:0/959028313 >> 172.21.15.57:0/959028313 conn(0x7f80b40cac70 msgr2=0x7f80b4070520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:54.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.951+0000 7f80b913e640 1 -- 172.21.15.57:0/959028313 shutdown_connections 2023-10-30T14:35:54.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:54.951+0000 7f80b913e640 1 -- 172.21.15.57:0/959028313 wait complete. 2023-10-30T14:35:54.975 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 124, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 124, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6204": { "addr": "172.21.15.57:6837/1843528309", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 1843528309, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 1843528309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 6204, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_6414": { "addr": "172.21.15.57:6834/2321408149", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2321408149, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2321408149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6414, "incarnation": 99, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 18 }, "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:clientreplay", "state_seq": 42 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:35:52.120615+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6414, "mds_1": 6747, "mds_2": 6204 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:35:56.975 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:57.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.127+0000 7f429bdc4640 1 Processor -- start 2023-10-30T14:35:57.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.127+0000 7f429bdc4640 1 -- start start 2023-10-30T14:35:57.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.127+0000 7f429bdc4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4294153c80 0x7f429415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:57.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.127+0000 7f429bdc4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f42940c4a10 0x7f42940c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:57.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.127+0000 7f429bdc4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4294159f20 con 0x7f429415a3b0 2023-10-30T14:35:57.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.127+0000 7f429bdc4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f429415dec0 con 0x7f42940c4a10 2023-10-30T14:35:57.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.127+0000 7f429bdc4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42941599a0 con 0x7f4294153c80 2023-10-30T14:35:57.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.127+0000 7f429a361640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f429415a3b0 0x7f42940ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:58046/0 (socket says 172.21.15.57:58046) 2023-10-30T14:35:57.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.127+0000 7f429a361640 1 -- 172.21.15.57:0/3226709448 learned_addr learned my addr 172.21.15.57:0/3226709448 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:35:57.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4299b60640 1 --2- 172.21.15.57:0/3226709448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4294153c80 0x7f429415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:57.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f429935f640 1 --2- 172.21.15.57:0/3226709448 >> v2:172.21.15.88:3300/0 conn(0x7f42940c4a10 0x7f42940c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:57.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4298b5e640 1 -- 172.21.15.57:0/3226709448 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109071595 0 0) 0x7f4294159f20 con 0x7f429415a3b0 2023-10-30T14:35:57.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4298b5e640 1 -- 172.21.15.57:0/3226709448 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4280003530 con 0x7f429415a3b0 2023-10-30T14:35:57.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4298b5e640 1 -- 172.21.15.57:0/3226709448 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1832183750 0 0) 0x7f4280003530 con 0x7f429415a3b0 2023-10-30T14:35:57.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4298b5e640 1 -- 172.21.15.57:0/3226709448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4294153c80 msgr2=0x7f429415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:57.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4298b5e640 1 --2- 172.21.15.57:0/3226709448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4294153c80 0x7f429415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:57.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4298b5e640 1 -- 172.21.15.57:0/3226709448 >> v2:172.21.15.88:3300/0 conn(0x7f42940c4a10 msgr2=0x7f42940c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:57.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4298b5e640 1 --2- 172.21.15.57:0/3226709448 >> v2:172.21.15.88:3300/0 conn(0x7f42940c4a10 0x7f42940c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:57.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4298b5e640 1 -- 172.21.15.57:0/3226709448 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42940c47e0 con 0x7f429415a3b0 2023-10-30T14:35:57.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4298b5e640 1 -- 172.21.15.57:0/3226709448 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f428c001a00 con 0x7f429415a3b0 2023-10-30T14:35:57.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4298b5e640 1 -- 172.21.15.57:0/3226709448 >> v1:172.21.15.57:6789/0 conn(0x7f429415a3b0 legacy=0x7f42940ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:57.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4298b5e640 1 --2- 172.21.15.57:0/3226709448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4280003980 0x7f4280003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:57.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4298b5e640 1 --2- 172.21.15.57:0/3226709448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4294153c80 0x7f42800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:57.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4298b5e640 1 --2- 172.21.15.57:0/3226709448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4280004d10 0x7f4280005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:57.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4298b5e640 1 -- 172.21.15.57:0/3226709448 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429415dec0 con 0x7f4280003980 2023-10-30T14:35:57.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4298b5e640 1 -- 172.21.15.57:0/3226709448 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42941599a0 con 0x7f4280004d10 2023-10-30T14:35:57.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4298b5e640 1 -- 172.21.15.57:0/3226709448 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f428000b990 con 0x7f4294153c80 2023-10-30T14:35:57.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4299b60640 1 --2- 172.21.15.57:0/3226709448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4280003980 0x7f4280003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:57.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f429935f640 1 --2- 172.21.15.57:0/3226709448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4294153c80 0x7f42800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:57.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f429a361640 1 --2- 172.21.15.57:0/3226709448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4280004d10 0x7f4280005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:57.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4299b60640 1 -- 172.21.15.57:0/3226709448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4294153c80 msgr2=0x7f42800047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:57.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4299b60640 1 --2- 172.21.15.57:0/3226709448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4294153c80 0x7f42800047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:57.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4299b60640 1 -- 172.21.15.57:0/3226709448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4280004d10 msgr2=0x7f4280005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:57.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4299b60640 1 --2- 172.21.15.57:0/3226709448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4280004d10 0x7f4280005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:57.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f4299b60640 1 -- 172.21.15.57:0/3226709448 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4280003530 con 0x7f4280003980 2023-10-30T14:35:57.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.131+0000 7f429a361640 1 --2- 172.21.15.57:0/3226709448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4280004d10 0x7f4280005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:35:57.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.135+0000 7f4299b60640 1 --2- 172.21.15.57:0/3226709448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4280003980 0x7f4280003da0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f4290009ae0 tx=0x7f429001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db8086fbb83d6a7d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:57.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.135+0000 7f4298b5e640 1 -- 172.21.15.57:0/3226709448 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42900a5640 con 0x7f4280003980 2023-10-30T14:35:57.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.135+0000 7f4298b5e640 1 -- 172.21.15.57:0/3226709448 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42900a57a0 con 0x7f4280003980 2023-10-30T14:35:57.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.135+0000 7f4298b5e640 1 -- 172.21.15.57:0/3226709448 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42900a5b50 con 0x7f4280003980 2023-10-30T14:35:57.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.135+0000 7f429bdc4640 1 -- 172.21.15.57:0/3226709448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4280003980 msgr2=0x7f4280003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:57.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.135+0000 7f429bdc4640 1 --2- 172.21.15.57:0/3226709448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4280003980 0x7f4280003da0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f4290009ae0 tx=0x7f429001ca50 comp rx=0 tx=0).stop 2023-10-30T14:35:57.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.135+0000 7f429a361640 1 -- 172.21.15.57:0/3226709448 reap_dead start 2023-10-30T14:35:57.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.135+0000 7f429bdc4640 1 -- 172.21.15.57:0/3226709448 shutdown_connections 2023-10-30T14:35:57.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.135+0000 7f429bdc4640 1 -- 172.21.15.57:0/3226709448 >> 172.21.15.57:0/3226709448 conn(0x7f42940715c0 msgr2=0x7f42940719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:57.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.135+0000 7f429bdc4640 1 -- 172.21.15.57:0/3226709448 shutdown_connections 2023-10-30T14:35:57.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.135+0000 7f429bdc4640 1 -- 172.21.15.57:0/3226709448 wait complete. 2023-10-30T14:35:57.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.135+0000 7f429bdc4640 1 Processor -- start 2023-10-30T14:35:57.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.135+0000 7f429bdc4640 1 -- start start 2023-10-30T14:35:57.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429bdc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4280003980 0x7f429414ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:57.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429bdc4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f429414f320 0x7f429414c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:57.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429bdc4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f429414cbb0 0x7f4294149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:57.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429bdc4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429414f7a0 con 0x7f429414cbb0 2023-10-30T14:35:57.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429bdc4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429414c200 con 0x7f4280003980 2023-10-30T14:35:57.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429bdc4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4294151070 con 0x7f429414f320 2023-10-30T14:35:57.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429a361640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f429414cbb0 0x7f4294149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:57.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429a361640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f429414cbb0 0x7f4294149b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50200/0 (socket says 172.21.15.57:50200) 2023-10-30T14:35:57.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f4299b60640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4280003980 0x7f429414ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:57.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429a361640 1 -- 172.21.15.57:0/3906233780 learned_addr learned my addr 172.21.15.57:0/3906233780 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:57.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429935f640 1 --2- 172.21.15.57:0/3906233780 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f429414f320 0x7f429414c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:57.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429a361640 1 -- 172.21.15.57:0/3906233780 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f429414f320 msgr2=0x7f429414c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:57.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429a361640 1 --2- 172.21.15.57:0/3906233780 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f429414f320 0x7f429414c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:57.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429a361640 1 -- 172.21.15.57:0/3906233780 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4280003980 msgr2=0x7f429414ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:57.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429a361640 1 --2- 172.21.15.57:0/3906233780 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4280003980 0x7f429414ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:57.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429a361640 1 -- 172.21.15.57:0/3906233780 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429414d030 con 0x7f429414cbb0 2023-10-30T14:35:57.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429935f640 1 --2- 172.21.15.57:0/3906233780 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f429414f320 0x7f429414c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:57.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429a361640 1 --2- 172.21.15.57:0/3906233780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f429414cbb0 0x7f4294149b30 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f429408d1f0 tx=0x7f428c010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:57.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f428a7fc640 1 -- 172.21.15.57:0/3906233780 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c00eac0 con 0x7f429414cbb0 2023-10-30T14:35:57.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f428a7fc640 1 -- 172.21.15.57:0/3906233780 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f428c00ec20 con 0x7f429414cbb0 2023-10-30T14:35:57.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f428a7fc640 1 -- 172.21.15.57:0/3906233780 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c015400 con 0x7f429414cbb0 2023-10-30T14:35:57.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429bdc4640 1 -- 172.21.15.57:0/3906233780 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f429414ab30 con 0x7f429414cbb0 2023-10-30T14:35:57.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429bdc4640 1 -- 172.21.15.57:0/3906233780 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f429414e020 con 0x7f429414cbb0 2023-10-30T14:35:57.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f429bdc4640 1 -- 172.21.15.57:0/3906233780 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f429414c390 con 0x7f429414cbb0 2023-10-30T14:35:57.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.139+0000 7f428a7fc640 1 -- 172.21.15.57:0/3906233780 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f428c01e3b0 con 0x7f429414cbb0 2023-10-30T14:35:57.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.143+0000 7f428a7fc640 1 --2- 172.21.15.57:0/3906233780 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f42780664a0 0x7f4278068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:57.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.143+0000 7f428a7fc640 1 -- 172.21.15.57:0/3906233780 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f428c086f70 con 0x7f429414cbb0 2023-10-30T14:35:57.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.143+0000 7f4299b60640 1 --2- 172.21.15.57:0/3906233780 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f42780664a0 0x7f4278068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:57.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.143+0000 7f4299b60640 1 --2- 172.21.15.57:0/3906233780 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f42780664a0 0x7f4278068960 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f42900af000 tx=0x7f42900046d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:57.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.147+0000 7f428a7fc640 1 -- 172.21.15.57:0/3906233780 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f429414c390 con 0x7f429414cbb0 2023-10-30T14:35:57.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.335+0000 7f429bdc4640 1 -- 172.21.15.57:0/3906233780 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f429414c390 con 0x7f429414cbb0 2023-10-30T14:35:57.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.335+0000 7f428a7fc640 1 -- 172.21.15.57:0/3906233780 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+5746 (secure 0 0 0) 0x7f429414c390 con 0x7f429414cbb0 2023-10-30T14:35:57.339 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:57.339 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:35:52.120615+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:clientreplay","state_seq":42,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:57.340 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 124 2023-10-30T14:35:57.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.339+0000 7f429bdc4640 1 -- 172.21.15.57:0/3906233780 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f42780664a0 msgr2=0x7f4278068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:57.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.339+0000 7f429bdc4640 1 --2- 172.21.15.57:0/3906233780 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f42780664a0 0x7f4278068960 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f42900af000 tx=0x7f42900046d0 comp rx=0 tx=0).stop 2023-10-30T14:35:57.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.339+0000 7f429bdc4640 1 -- 172.21.15.57:0/3906233780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f429414cbb0 msgr2=0x7f4294149b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:57.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.339+0000 7f429bdc4640 1 --2- 172.21.15.57:0/3906233780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f429414cbb0 0x7f4294149b30 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f429408d1f0 tx=0x7f428c010d70 comp rx=0 tx=0).stop 2023-10-30T14:35:57.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.339+0000 7f429bdc4640 1 -- 172.21.15.57:0/3906233780 shutdown_connections 2023-10-30T14:35:57.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.339+0000 7f429bdc4640 1 --2- 172.21.15.57:0/3906233780 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f42780664a0 0x7f4278068960 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:57.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.339+0000 7f429bdc4640 1 --2- 172.21.15.57:0/3906233780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f429414cbb0 0x7f4294149b30 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:57.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.339+0000 7f429bdc4640 1 --2- 172.21.15.57:0/3906233780 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f429414f320 0x7f429414c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:57.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.339+0000 7f429bdc4640 1 --2- 172.21.15.57:0/3906233780 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4280003980 0x7f429414ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:57.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.339+0000 7f429bdc4640 1 -- 172.21.15.57:0/3906233780 >> 172.21.15.57:0/3906233780 conn(0x7f42940715c0 msgr2=0x7f4294157fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:57.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.339+0000 7f429bdc4640 1 -- 172.21.15.57:0/3906233780 shutdown_connections 2023-10-30T14:35:57.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:57.339+0000 7f429bdc4640 1 -- 172.21.15.57:0/3906233780 wait complete. 2023-10-30T14:35:57.360 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:35:59.361 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:35:59.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.511+0000 7fec2be91640 1 Processor -- start 2023-10-30T14:35:59.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.511+0000 7fec2be91640 1 -- start start 2023-10-30T14:35:59.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.511+0000 7fec2be91640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fec24151840 0x7fec240c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:59.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.515+0000 7fec2be91640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec24151c20 0x7fec24072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:59.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.515+0000 7fec2be91640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec240c5030 con 0x7fec240cb8b0 2023-10-30T14:35:59.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.515+0000 7fec2be91640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec24157830 con 0x7fec24151840 2023-10-30T14:35:59.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.515+0000 7fec2be91640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec24157b10 con 0x7fec24151c20 2023-10-30T14:35:59.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.515+0000 7fec2942c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec24151c20 0x7fec24072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:59.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.515+0000 7fec29c2d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fec24151840 0x7fec240c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:59.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.515+0000 7fec2942c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec24151c20 0x7fec24072000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:51632/0 (socket says 172.21.15.57:51632) 2023-10-30T14:35:59.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.515+0000 7fec2942c640 1 -- 172.21.15.57:0/216982603 learned_addr learned my addr 172.21.15.57:0/216982603 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:59.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.515+0000 7fec29c2d640 1 -- 172.21.15.57:0/216982603 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec24151c20 msgr2=0x7fec24072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:35:59.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.515+0000 7fec29c2d640 1 --2- 172.21.15.57:0/216982603 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec24151c20 0x7fec24072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:59.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.515+0000 7fec29c2d640 1 -- 172.21.15.57:0/216982603 >> v1:172.21.15.88:6789/0 conn(0x7fec240cb8b0 legacy=0x7fec2406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:59.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.515+0000 7fec29c2d640 1 -- 172.21.15.57:0/216982603 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec24157df0 con 0x7fec24151840 2023-10-30T14:35:59.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.515+0000 7fec2942c640 1 --2- 172.21.15.57:0/216982603 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec24151c20 0x7fec24072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:35:59.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.515+0000 7fec28c2b640 1 -- 172.21.15.57:0/216982603 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269887179 0 0) 0x7fec240c5030 con 0x7fec240cb8b0 2023-10-30T14:35:59.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.519+0000 7fec29c2d640 1 --2- 172.21.15.57:0/216982603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec24151840 0x7fec240c49a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fec200097c0 tx=0x7fec2001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ae34e7196d96030 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:59.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.519+0000 7fec28c2b640 1 -- 172.21.15.57:0/216982603 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec200a8970 con 0x7fec24151840 2023-10-30T14:35:59.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.519+0000 7fec28c2b640 1 -- 172.21.15.57:0/216982603 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec200a8ad0 con 0x7fec24151840 2023-10-30T14:35:59.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.519+0000 7fec28c2b640 1 -- 172.21.15.57:0/216982603 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec200a8e60 con 0x7fec24151840 2023-10-30T14:35:59.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.519+0000 7fec2be91640 1 -- 172.21.15.57:0/216982603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec24151840 msgr2=0x7fec240c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:59.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.519+0000 7fec2be91640 1 --2- 172.21.15.57:0/216982603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec24151840 0x7fec240c49a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fec200097c0 tx=0x7fec2001ec70 comp rx=0 tx=0).stop 2023-10-30T14:35:59.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.519+0000 7fec2be91640 1 -- 172.21.15.57:0/216982603 shutdown_connections 2023-10-30T14:35:59.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.519+0000 7fec2be91640 1 --2- 172.21.15.57:0/216982603 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec24151c20 0x7fec24072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:59.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.519+0000 7fec2be91640 1 --2- 172.21.15.57:0/216982603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec24151840 0x7fec240c49a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:59.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.519+0000 7fec2be91640 1 -- 172.21.15.57:0/216982603 >> 172.21.15.57:0/216982603 conn(0x7fec240cac70 msgr2=0x7fec240cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:59.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.519+0000 7fec2be91640 1 -- 172.21.15.57:0/216982603 shutdown_connections 2023-10-30T14:35:59.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.519+0000 7fec2be91640 1 -- 172.21.15.57:0/216982603 wait complete. 2023-10-30T14:35:59.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.519+0000 7fec2be91640 1 Processor -- start 2023-10-30T14:35:59.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.519+0000 7fec2be91640 1 -- start start 2023-10-30T14:35:59.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2be91640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec240cb8b0 0x7fec2414c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:59.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2be91640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec24151840 0x7fec241ee870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:59.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2be91640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec24151c20 0x7fec2414a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:59.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2be91640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec24154700 con 0x7fec24151840 2023-10-30T14:35:59.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2be91640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec241f2820 con 0x7fec24151c20 2023-10-30T14:35:59.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2be91640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec241f2c40 con 0x7fec240cb8b0 2023-10-30T14:35:59.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2942c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec24151840 0x7fec241ee870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:59.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2942c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec24151840 0x7fec241ee870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50208/0 (socket says 172.21.15.57:50208) 2023-10-30T14:35:59.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2942c640 1 -- 172.21.15.57:0/154742255 learned_addr learned my addr 172.21.15.57:0/154742255 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:35:59.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec29c2d640 1 --2- 172.21.15.57:0/154742255 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec240cb8b0 0x7fec2414c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:59.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2a42e640 1 --2- 172.21.15.57:0/154742255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec24151c20 0x7fec2414a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:59.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2942c640 1 -- 172.21.15.57:0/154742255 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec240cb8b0 msgr2=0x7fec2414c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:59.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2942c640 1 --2- 172.21.15.57:0/154742255 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec240cb8b0 0x7fec2414c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:59.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2942c640 1 -- 172.21.15.57:0/154742255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec24151c20 msgr2=0x7fec2414a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:59.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2942c640 1 --2- 172.21.15.57:0/154742255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec24151c20 0x7fec2414a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:59.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2942c640 1 -- 172.21.15.57:0/154742255 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec140094d0 con 0x7fec24151840 2023-10-30T14:35:59.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2a42e640 1 --2- 172.21.15.57:0/154742255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec24151c20 0x7fec2414a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:35:59.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2942c640 1 --2- 172.21.15.57:0/154742255 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec24151840 0x7fec241ee870 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fec1400e900 tx=0x7fec1400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:59.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec1a7fc640 1 -- 172.21.15.57:0/154742255 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1400ca00 con 0x7fec24151840 2023-10-30T14:35:59.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec1a7fc640 1 -- 172.21.15.57:0/154742255 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1400cb60 con 0x7fec24151840 2023-10-30T14:35:59.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec1a7fc640 1 -- 172.21.15.57:0/154742255 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec14013470 con 0x7fec24151840 2023-10-30T14:35:59.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2be91640 1 -- 172.21.15.57:0/154742255 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec20009420 con 0x7fec24151840 2023-10-30T14:35:59.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2be91640 1 -- 172.21.15.57:0/154742255 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec24156140 con 0x7fec24151840 2023-10-30T14:35:59.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec2be91640 1 -- 172.21.15.57:0/154742255 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2404e170 con 0x7fec24151840 2023-10-30T14:35:59.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.523+0000 7fec1a7fc640 1 -- 172.21.15.57:0/154742255 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fec140135d0 con 0x7fec24151840 2023-10-30T14:35:59.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.527+0000 7fec1a7fc640 1 --2- 172.21.15.57:0/154742255 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fec080664a0 0x7fec08068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:35:59.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.527+0000 7fec1a7fc640 1 -- 172.21.15.57:0/154742255 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fec140847a0 con 0x7fec24151840 2023-10-30T14:35:59.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.527+0000 7fec29c2d640 1 --2- 172.21.15.57:0/154742255 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fec080664a0 0x7fec08068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:35:59.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.527+0000 7fec29c2d640 1 --2- 172.21.15.57:0/154742255 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fec080664a0 0x7fec08068960 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fec200b2000 tx=0x7fec20008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:35:59.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.531+0000 7fec1a7fc640 1 -- 172.21.15.57:0/154742255 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fec140518e0 con 0x7fec24151840 2023-10-30T14:35:59.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.719+0000 7fec2be91640 1 -- 172.21.15.57:0/154742255 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec24155060 con 0x7fec24151840 2023-10-30T14:35:59.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.719+0000 7fec1a7fc640 1 -- 172.21.15.57:0/154742255 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5740 (secure 0 0 0) 0x7fec14054f00 con 0x7fec24151840 2023-10-30T14:35:59.724 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:35:59.724 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:35:58.639193+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:35:59.725 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 125 2023-10-30T14:35:59.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.723+0000 7fec2be91640 1 -- 172.21.15.57:0/154742255 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fec080664a0 msgr2=0x7fec08068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:59.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.723+0000 7fec2be91640 1 --2- 172.21.15.57:0/154742255 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fec080664a0 0x7fec08068960 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fec200b2000 tx=0x7fec20008040 comp rx=0 tx=0).stop 2023-10-30T14:35:59.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.723+0000 7fec2be91640 1 -- 172.21.15.57:0/154742255 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec24151840 msgr2=0x7fec241ee870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:35:59.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.723+0000 7fec2be91640 1 --2- 172.21.15.57:0/154742255 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec24151840 0x7fec241ee870 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fec1400e900 tx=0x7fec1400edc0 comp rx=0 tx=0).stop 2023-10-30T14:35:59.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.723+0000 7fec2be91640 1 -- 172.21.15.57:0/154742255 shutdown_connections 2023-10-30T14:35:59.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.723+0000 7fec2be91640 1 --2- 172.21.15.57:0/154742255 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fec080664a0 0x7fec08068960 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:59.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.723+0000 7fec2be91640 1 --2- 172.21.15.57:0/154742255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec24151c20 0x7fec2414a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:59.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.723+0000 7fec2be91640 1 --2- 172.21.15.57:0/154742255 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec24151840 0x7fec241ee870 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:59.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.723+0000 7fec2be91640 1 --2- 172.21.15.57:0/154742255 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec240cb8b0 0x7fec2414c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:35:59.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.723+0000 7fec2be91640 1 -- 172.21.15.57:0/154742255 >> 172.21.15.57:0/154742255 conn(0x7fec240cac70 msgr2=0x7fec2414d740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:35:59.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.723+0000 7fec2be91640 1 -- 172.21.15.57:0/154742255 shutdown_connections 2023-10-30T14:35:59.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:35:59.723+0000 7fec2be91640 1 -- 172.21.15.57:0/154742255 wait complete. 2023-10-30T14:35:59.744 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=1, replacing gid=5891 2023-10-30T14:35:59.744 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.f 2023-10-30T14:36:02.026 INFO:tasks.ceph.mds.a.smithi057.stderr:2023-10-30T14:36:02.023+0000 7f5ba0392640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:36:05.097 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 22% (50/225) Updating files: 23% (52/225) Updating files: 24% (54/225) Updating files: 25% (57/225) Updating files: 26% (59/225) Updating files: 27% (61/225) Updating files: 28% (63/225) Updating files: 29% (66/225) Updating files: 30% (68/225) Updating files: 31% (70/225) Updating files: 32% (72/225) Updating files: 33% (75/225) Updating files: 34% (77/225) Updating files: 35% (79/225) Updating files: 36% (81/225) Updating files: 37% (84/225) Updating files: 38% (86/225) Updating files: 39% (88/225) Updating files: 40% (90/225) Updating files: 41% (93/225) Updating files: 42% (95/225) Updating files: 43% (97/225) Updating files: 44% (99/225) Updating files: 45% (102/225) Updating files: 46% (104/225) Updating files: 47% (106/225) Updating files: 48% (108/225) Updating files: 49% (111/225) Updating files: 50% (113/225) Updating files: 51% (115/225) Updating files: 52% (117/225) Updating files: 53% (120/225) Updating files: 54% (122/225) Updating files: 55% (124/225) Updating files: 56% (126/225) Updating files: 57% (129/225) Updating files: 58% (131/225) Updating files: 59% (133/225) Updating files: 60% (135/225) Updating files: 61% (138/225) Updating files: 62% (140/225) Updating files: 63% (142/225) Updating files: 64% (144/225) Updating files: 65% (147/225) Updating files: 66% (149/225) Updating files: 67% (151/225) Updating files: 68% (153/225) Updating files: 69% (156/225) Updating files: 70% (158/225) Updating files: 71% (160/225) Updating files: 72% (162/225) Updating files: 73% (165/225) Updating files: 74% (167/225) Updating files: 75% (169/225) Updating files: 76% (171/225) Updating files: 77% (174/225) Updating files: 78% (176/225) Updating files: 79% (178/225) Updating files: 80% (180/225) Updating files: 81% (183/225) Updating files: 82% (185/225) Updating files: 83% (187/225) Updating files: 84% (189/225) Updating files: 85% (192/225) Updating files: 86% (194/225) Updating files: 87% (196/225) Updating files: 88% (198/225) Updating files: 89% (201/225) Updating files: 90% (203/225) Updating files: 91% (205/225) Updating files: 92% (207/225) Updating files: 93% (210/225) Updating files: 94% (212/225) Updating files: 95% (214/225) Updating files: 96% (216/225) Updating files: 97% (219/225) Updating files: 98% (221/225) Updating files: 99% (223/225) Updating files: 100% (225/225) Updating files: 100% (225/225), done. 2023-10-30T14:36:05.700 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 284161c00ab kclient: kill dput/d_drop debug cruft 2023-10-30T14:36:05.703 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.8 2023-10-30T14:36:05.745 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-30T14:36:05.745 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T14:36:05.746 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T14:36:05.748 INFO:tasks.ceph.mds.f:Started 2023-10-30T14:36:05.749 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:36:05.766 INFO:tasks.workunit.client.0.smithi057.stdout:v0.9 2023-10-30T14:36:05.766 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 9 -eq 48 ] 2023-10-30T14:36:05.766 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.9 2023-10-30T14:36:05.766 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.9 2023-10-30T14:36:05.767 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.9 2023-10-30T14:36:05.815 INFO:tasks.ceph.mds.f.smithi088.stdout:starting mds.f at 2023-10-30T14:36:05.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.899+0000 7f3b6b879640 1 Processor -- start 2023-10-30T14:36:05.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.899+0000 7f3b6b879640 1 -- start start 2023-10-30T14:36:05.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.899+0000 7f3b6b879640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b641516c0 0x7f3b640c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:36:05.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.899+0000 7f3b6b879640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3b640c2500 0x7f3b640c2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:36:05.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.899+0000 7f3b6b879640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b64151aa0 con 0x7f3b640c82d0 2023-10-30T14:36:05.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.899+0000 7f3b6b879640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b640c74a0 con 0x7f3b640c2500 2023-10-30T14:36:05.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.899+0000 7f3b6b879640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b640c2e90 con 0x7f3b641516c0 2023-10-30T14:36:05.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.899+0000 7f3b69e16640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3b640c2500 0x7f3b640c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:36:05.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.899+0000 7f3b69e16640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3b640c2500 0x7f3b640c2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37286/0 (socket says 172.21.15.57:37286) 2023-10-30T14:36:05.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.899+0000 7f3b69e16640 1 -- 172.21.15.57:0/2145797274 learned_addr learned my addr 172.21.15.57:0/2145797274 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:36:05.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.899+0000 7f3b69615640 1 --2- 172.21.15.57:0/2145797274 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b641516c0 0x7f3b640c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:36:05.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.899+0000 7f3b69e16640 1 -- 172.21.15.57:0/2145797274 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b641516c0 msgr2=0x7f3b640c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:36:05.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.899+0000 7f3b69e16640 1 --2- 172.21.15.57:0/2145797274 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b641516c0 0x7f3b640c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:36:05.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.899+0000 7f3b69e16640 1 -- 172.21.15.57:0/2145797274 >> v1:172.21.15.88:6789/0 conn(0x7f3b640c82d0 legacy=0x7f3b640ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:36:05.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.899+0000 7f3b69e16640 1 -- 172.21.15.57:0/2145797274 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b64158af0 con 0x7f3b640c2500 2023-10-30T14:36:05.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.903+0000 7f3b69e16640 1 --2- 172.21.15.57:0/2145797274 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b640c2500 0x7f3b640c2950 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c009e80 tx=0x7f3b5c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5affd15d364d60a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:36:05.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.903+0000 7f3b5bfff640 1 -- 172.21.15.57:0/2145797274 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c0a56c0 con 0x7f3b640c2500 2023-10-30T14:36:05.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.903+0000 7f3b5bfff640 1 -- 172.21.15.57:0/2145797274 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5c0a5820 con 0x7f3b640c2500 2023-10-30T14:36:05.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.903+0000 7f3b5bfff640 1 -- 172.21.15.57:0/2145797274 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c0a5bb0 con 0x7f3b640c2500 2023-10-30T14:36:05.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.903+0000 7f3b6b879640 1 -- 172.21.15.57:0/2145797274 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b640c2500 msgr2=0x7f3b640c2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:36:05.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.903+0000 7f3b6b879640 1 --2- 172.21.15.57:0/2145797274 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b640c2500 0x7f3b640c2950 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c009e80 tx=0x7f3b5c01c9f0 comp rx=0 tx=0).stop 2023-10-30T14:36:05.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.903+0000 7f3b6b879640 1 -- 172.21.15.57:0/2145797274 shutdown_connections 2023-10-30T14:36:05.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.903+0000 7f3b6b879640 1 --2- 172.21.15.57:0/2145797274 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b640c2500 0x7f3b640c2950 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:36:05.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.903+0000 7f3b6b879640 1 --2- 172.21.15.57:0/2145797274 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b641516c0 0x7f3b640c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:36:05.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.903+0000 7f3b6b879640 1 -- 172.21.15.57:0/2145797274 >> 172.21.15.57:0/2145797274 conn(0x7f3b64071fb0 msgr2=0x7f3b640723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:36:05.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.903+0000 7f3b6b879640 1 -- 172.21.15.57:0/2145797274 shutdown_connections 2023-10-30T14:36:05.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.903+0000 7f3b6b879640 1 -- 172.21.15.57:0/2145797274 wait complete. 2023-10-30T14:36:05.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b6b879640 1 Processor -- start 2023-10-30T14:36:05.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b6b879640 1 -- start start 2023-10-30T14:36:05.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b6b879640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b640c2500 0x7f3b6414f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:36:05.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b6b879640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b640c82d0 0x7f3b641f5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:36:05.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b6b879640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b641516c0 0x7f3b641f62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:36:05.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b6b879640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b641f21e0 con 0x7f3b640c82d0 2023-10-30T14:36:05.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b6b879640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b640c8e70 con 0x7f3b641516c0 2023-10-30T14:36:05.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b6b879640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b641f58f0 con 0x7f3b640c2500 2023-10-30T14:36:05.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b68e14640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b640c82d0 0x7f3b641f5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:36:05.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b68e14640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b640c82d0 0x7f3b641f5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37296/0 (socket says 172.21.15.57:37296) 2023-10-30T14:36:05.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b68e14640 1 -- 172.21.15.57:0/251774892 learned_addr learned my addr 172.21.15.57:0/251774892 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:36:05.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b69615640 1 --2- 172.21.15.57:0/251774892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b640c2500 0x7f3b6414f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:36:05.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b69e16640 1 --2- 172.21.15.57:0/251774892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b641516c0 0x7f3b641f62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:36:05.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b68e14640 1 -- 172.21.15.57:0/251774892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b640c2500 msgr2=0x7f3b6414f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:36:05.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b68e14640 1 --2- 172.21.15.57:0/251774892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b640c2500 0x7f3b6414f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:36:05.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b68e14640 1 -- 172.21.15.57:0/251774892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b641516c0 msgr2=0x7f3b641f62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:36:05.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b68e14640 1 --2- 172.21.15.57:0/251774892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b641516c0 0x7f3b641f62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:36:05.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b68e14640 1 -- 172.21.15.57:0/251774892 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c009b30 con 0x7f3b640c82d0 2023-10-30T14:36:05.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b68e14640 1 --2- 172.21.15.57:0/251774892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b640c82d0 0x7f3b641f5d80 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f3b5400c3f0 tx=0x7f3b5400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:36:05.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b59ffb640 1 -- 172.21.15.57:0/251774892 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b540052d0 con 0x7f3b640c82d0 2023-10-30T14:36:05.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b59ffb640 1 -- 172.21.15.57:0/251774892 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b54005430 con 0x7f3b640c82d0 2023-10-30T14:36:05.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b59ffb640 1 -- 172.21.15.57:0/251774892 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b54003840 con 0x7f3b640c82d0 2023-10-30T14:36:05.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b6b879640 1 -- 172.21.15.57:0/251774892 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b64144fb0 con 0x7f3b640c82d0 2023-10-30T14:36:05.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.907+0000 7f3b6b879640 1 -- 172.21.15.57:0/251774892 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b641449f0 con 0x7f3b640c82d0 2023-10-30T14:36:05.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.911+0000 7f3b6b879640 1 -- 172.21.15.57:0/251774892 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b6404e170 con 0x7f3b640c82d0 2023-10-30T14:36:05.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.911+0000 7f3b59ffb640 1 -- 172.21.15.57:0/251774892 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3b540039a0 con 0x7f3b640c82d0 2023-10-30T14:36:05.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.911+0000 7f3b59ffb640 1 --2- 172.21.15.57:0/251774892 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3b480664a0 0x7f3b48068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:36:05.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.911+0000 7f3b59ffb640 1 -- 172.21.15.57:0/251774892 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f3b540838c0 con 0x7f3b640c82d0 2023-10-30T14:36:05.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.911+0000 7f3b69615640 1 --2- 172.21.15.57:0/251774892 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3b480664a0 0x7f3b48068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:36:05.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.911+0000 7f3b69615640 1 --2- 172.21.15.57:0/251774892 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3b480664a0 0x7f3b48068960 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3b60009b60 tx=0x7f3b60009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:36:05.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:05.915+0000 7f3b59ffb640 1 -- 172.21.15.57:0/251774892 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3b54050840 con 0x7f3b640c82d0 2023-10-30T14:36:06.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:06.107+0000 7f3b6b879640 1 -- 172.21.15.57:0/251774892 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b64154f20 con 0x7f3b640c82d0 2023-10-30T14:36:06.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:06.107+0000 7f3b59ffb640 1 -- 172.21.15.57:0/251774892 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5740 (secure 0 0 0) 0x7f3b54017090 con 0x7f3b640c82d0 2023-10-30T14:36:06.110 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:36:06.111 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:35:58.639193+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:36:06.111 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 125 2023-10-30T14:36:06.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:06.107+0000 7f3b6b879640 1 -- 172.21.15.57:0/251774892 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3b480664a0 msgr2=0x7f3b48068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:36:06.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:06.107+0000 7f3b6b879640 1 --2- 172.21.15.57:0/251774892 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3b480664a0 0x7f3b48068960 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3b60009b60 tx=0x7f3b60009190 comp rx=0 tx=0).stop 2023-10-30T14:36:06.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:06.107+0000 7f3b6b879640 1 -- 172.21.15.57:0/251774892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b640c82d0 msgr2=0x7f3b641f5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:36:06.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:06.107+0000 7f3b6b879640 1 --2- 172.21.15.57:0/251774892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b640c82d0 0x7f3b641f5d80 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f3b5400c3f0 tx=0x7f3b5400c8b0 comp rx=0 tx=0).stop 2023-10-30T14:36:06.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:06.107+0000 7f3b6b879640 1 -- 172.21.15.57:0/251774892 shutdown_connections 2023-10-30T14:36:06.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:06.107+0000 7f3b6b879640 1 --2- 172.21.15.57:0/251774892 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3b480664a0 0x7f3b48068960 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:36:06.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:06.107+0000 7f3b6b879640 1 --2- 172.21.15.57:0/251774892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b641516c0 0x7f3b641f62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:36:06.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:06.107+0000 7f3b6b879640 1 --2- 172.21.15.57:0/251774892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b640c82d0 0x7f3b641f5d80 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:36:06.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:06.107+0000 7f3b6b879640 1 --2- 172.21.15.57:0/251774892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b640c2500 0x7f3b6414f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:36:06.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:06.107+0000 7f3b6b879640 1 -- 172.21.15.57:0/251774892 >> 172.21.15.57:0/251774892 conn(0x7f3b64071fb0 msgr2=0x7f3b64145db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:36:06.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:06.107+0000 7f3b6b879640 1 -- 172.21.15.57:0/251774892 shutdown_connections 2023-10-30T14:36:06.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:06.107+0000 7f3b6b879640 1 -- 172.21.15.57:0/251774892 wait complete. 2023-10-30T14:36:06.131 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-30T14:36:07.255 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:36:07.259 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:36:08.132 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:36:08.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.279+0000 7f6c3aa17640 1 Processor -- start 2023-10-30T14:36:08.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.279+0000 7f6c3aa17640 1 -- start start 2023-10-30T14:36:08.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.279+0000 7f6c3aa17640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6c34071340 0x7f6c34065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:36:08.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.279+0000 7f6c3aa17640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3406e390 con 0x7f6c34153c10 2023-10-30T14:36:08.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.279+0000 7f6c3aa17640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c340660e0 con 0x7f6c34159970 2023-10-30T14:36:08.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.279+0000 7f6c3aa17640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c34072390 con 0x7f6c34071340 2023-10-30T14:36:08.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.279+0000 7f6c337fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6c34071340 0x7f6c34065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:36:08.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.283+0000 7f6c337fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6c34071340 0x7f6c34065ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37308/0 (socket says 172.21.15.57:37308) 2023-10-30T14:36:08.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.283+0000 7f6c337fe640 1 -- 172.21.15.57:0/3209455957 learned_addr learned my addr 172.21.15.57:0/3209455957 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:36:08.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.283+0000 7f6c33fff640 1 --1- 172.21.15.57:0/3209455957 >> v1:172.21.15.57:6789/0 conn(0x7f6c34153c10 0x7f6c34070c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:55292/0 (socket says 172.21.15.57:55292) 2023-10-30T14:36:08.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.283+0000 7f6c337fe640 1 -- 172.21.15.57:0/3209455957 >> v1:172.21.15.88:6789/0 conn(0x7f6c34159970 legacy=0x7f6c34159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:36:08.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.283+0000 7f6c337fe640 1 -- 172.21.15.57:0/3209455957 >> v1:172.21.15.57:6789/0 conn(0x7f6c34153c10 legacy=0x7f6c34070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:36:08.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.283+0000 7f6c337fe640 1 -- 172.21.15.57:0/3209455957 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c340bec10 con 0x7f6c34071340 2023-10-30T14:36:08.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.283+0000 7f6c32ffd640 1 -- 172.21.15.57:0/3209455957 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375256293 0 0) 0x7f6c3406e390 con 0x7f6c34153c10 2023-10-30T14:36:08.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.283+0000 7f6c337fe640 1 --2- 172.21.15.57:0/3209455957 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c34071340 0x7f6c34065ba0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f6c28009870 tx=0x7f6c2801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45a2169a926ce62f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:36:08.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.283+0000 7f6c32ffd640 1 -- 172.21.15.57:0/3209455957 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c280ab600 con 0x7f6c34071340 2023-10-30T14:36:08.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.283+0000 7f6c32ffd640 1 -- 172.21.15.57:0/3209455957 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c280a7050 con 0x7f6c34071340 2023-10-30T14:36:08.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.283+0000 7f6c32ffd640 1 -- 172.21.15.57:0/3209455957 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c280ab980 con 0x7f6c34071340 2023-10-30T14:36:08.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c3aa17640 1 -- 172.21.15.57:0/3209455957 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c34071340 msgr2=0x7f6c34065ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:36:08.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c3aa17640 1 --2- 172.21.15.57:0/3209455957 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c34071340 0x7f6c34065ba0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f6c28009870 tx=0x7f6c2801c4b0 comp rx=0 tx=0).stop 2023-10-30T14:36:08.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c3aa17640 1 -- 172.21.15.57:0/3209455957 shutdown_connections 2023-10-30T14:36:08.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c3aa17640 1 --2- 172.21.15.57:0/3209455957 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c34071340 0x7f6c34065ba0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:36:08.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c3aa17640 1 -- 172.21.15.57:0/3209455957 >> 172.21.15.57:0/3209455957 conn(0x7f6c340c7f60 msgr2=0x7f6c340c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:36:08.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c3aa17640 1 -- 172.21.15.57:0/3209455957 shutdown_connections 2023-10-30T14:36:08.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c3aa17640 1 -- 172.21.15.57:0/3209455957 wait complete. 2023-10-30T14:36:08.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c3aa17640 1 Processor -- start 2023-10-30T14:36:08.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c3aa17640 1 -- start start 2023-10-30T14:36:08.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c3aa17640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c34071340 0x7f6c34159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:36:08.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c3aa17640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c34153c10 0x7f6c3414d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:36:08.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c3aa17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c34159970 0x7f6c34151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:36:08.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c3aa17640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c34149b50 con 0x7f6c34071340 2023-10-30T14:36:08.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c3aa17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c34148fb0 con 0x7f6c34159970 2023-10-30T14:36:08.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c3aa17640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c340c5860 con 0x7f6c34153c10 2023-10-30T14:36:08.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c33fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c34071340 0x7f6c34159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:36:08.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c33fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c34071340 0x7f6c34159170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37310/0 (socket says 172.21.15.57:37310) 2023-10-30T14:36:08.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c33fff640 1 -- 172.21.15.57:0/318031946 learned_addr learned my addr 172.21.15.57:0/318031946 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:36:08.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c337fe640 1 --2- 172.21.15.57:0/318031946 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c34153c10 0x7f6c3414d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:36:08.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c38fb4640 1 --2- 172.21.15.57:0/318031946 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c34159970 0x7f6c34151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:36:08.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c33fff640 1 -- 172.21.15.57:0/318031946 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c34153c10 msgr2=0x7f6c3414d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:36:08.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c33fff640 1 --2- 172.21.15.57:0/318031946 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c34153c10 0x7f6c3414d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:36:08.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c33fff640 1 -- 172.21.15.57:0/318031946 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c34159970 msgr2=0x7f6c34151ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:36:08.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c33fff640 1 --2- 172.21.15.57:0/318031946 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c34159970 0x7f6c34151ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:36:08.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c33fff640 1 -- 172.21.15.57:0/318031946 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c280094d0 con 0x7f6c34071340 2023-10-30T14:36:08.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.287+0000 7f6c337fe640 1 --2- 172.21.15.57:0/318031946 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c34153c10 0x7f6c3414d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:36:08.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.291+0000 7f6c33fff640 1 --2- 172.21.15.57:0/318031946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c34071340 0x7f6c34159170 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400c4d0 tx=0x7f6c2400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:36:08.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.291+0000 7f6c30ff9640 1 -- 172.21.15.57:0/318031946 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c240074e0 con 0x7f6c34071340 2023-10-30T14:36:08.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.291+0000 7f6c30ff9640 1 -- 172.21.15.57:0/318031946 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c24007640 con 0x7f6c34071340 2023-10-30T14:36:08.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.291+0000 7f6c30ff9640 1 -- 172.21.15.57:0/318031946 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c24004a30 con 0x7f6c34071340 2023-10-30T14:36:08.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.291+0000 7f6c3aa17640 1 -- 172.21.15.57:0/318031946 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c34149220 con 0x7f6c34071340 2023-10-30T14:36:08.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.291+0000 7f6c3aa17640 1 -- 172.21.15.57:0/318031946 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c341497e0 con 0x7f6c34071340 2023-10-30T14:36:08.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.291+0000 7f6c3aa17640 1 -- 172.21.15.57:0/318031946 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c3404e170 con 0x7f6c34071340 2023-10-30T14:36:08.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.291+0000 7f6c30ff9640 1 -- 172.21.15.57:0/318031946 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6c24004b90 con 0x7f6c34071340 2023-10-30T14:36:08.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.295+0000 7f6c30ff9640 1 --2- 172.21.15.57:0/318031946 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6c200664a0 0x7f6c20068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:36:08.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.295+0000 7f6c30ff9640 1 -- 172.21.15.57:0/318031946 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f6c24083660 con 0x7f6c34071340 2023-10-30T14:36:08.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.295+0000 7f6c337fe640 1 --2- 172.21.15.57:0/318031946 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6c200664a0 0x7f6c20068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:36:08.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.295+0000 7f6c337fe640 1 --2- 172.21.15.57:0/318031946 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6c200664a0 0x7f6c20068960 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6c280ad000 tx=0x7f6c28008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:36:08.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.295+0000 7f6c30ff9640 1 -- 172.21.15.57:0/318031946 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6c240505c0 con 0x7f6c34071340 2023-10-30T14:36:08.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.527+0000 7f6c3aa17640 1 -- 172.21.15.57:0/318031946 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c340710f0 con 0x7f6c34071340 2023-10-30T14:36:08.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.527+0000 7f6c30ff9640 1 -- 172.21.15.57:0/318031946 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+6532 (secure 0 0 0) 0x7f6c24018090 con 0x7f6c34071340 2023-10-30T14:36:08.532 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:36:08.532 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":126,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6125,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:35:58.639193+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:36:08.533 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 126 2023-10-30T14:36:08.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.531+0000 7f6c3aa17640 1 -- 172.21.15.57:0/318031946 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6c200664a0 msgr2=0x7f6c20068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:36:08.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.531+0000 7f6c3aa17640 1 --2- 172.21.15.57:0/318031946 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6c200664a0 0x7f6c20068960 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6c280ad000 tx=0x7f6c28008040 comp rx=0 tx=0).stop 2023-10-30T14:36:08.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.531+0000 7f6c3aa17640 1 -- 172.21.15.57:0/318031946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c34071340 msgr2=0x7f6c34159170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:36:08.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.531+0000 7f6c3aa17640 1 --2- 172.21.15.57:0/318031946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c34071340 0x7f6c34159170 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400c4d0 tx=0x7f6c2400c990 comp rx=0 tx=0).stop 2023-10-30T14:36:08.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.531+0000 7f6c3aa17640 1 -- 172.21.15.57:0/318031946 shutdown_connections 2023-10-30T14:36:08.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.531+0000 7f6c3aa17640 1 --2- 172.21.15.57:0/318031946 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6c200664a0 0x7f6c20068960 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:36:08.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.531+0000 7f6c3aa17640 1 --2- 172.21.15.57:0/318031946 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c34159970 0x7f6c34151ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:36:08.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.531+0000 7f6c3aa17640 1 --2- 172.21.15.57:0/318031946 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c34153c10 0x7f6c3414d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:36:08.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.531+0000 7f6c3aa17640 1 --2- 172.21.15.57:0/318031946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c34071340 0x7f6c34159170 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:36:08.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.531+0000 7f6c3aa17640 1 -- 172.21.15.57:0/318031946 >> 172.21.15.57:0/318031946 conn(0x7f6c340c7f60 msgr2=0x7f6c34159d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:36:08.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.531+0000 7f6c3aa17640 1 -- 172.21.15.57:0/318031946 shutdown_connections 2023-10-30T14:36:08.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:36:08.531+0000 7f6c3aa17640 1 -- 172.21.15.57:0/318031946 wait complete. 2023-10-30T14:36:08.553 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-30T14:36:08.554 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 110 secs before thrashing 2023-10-30T14:36:13.509 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 35% (68/193) Updating files: 36% (70/193) Updating files: 37% (72/193) Updating files: 38% (74/193) Updating files: 39% (76/193) Updating files: 40% (78/193) Updating files: 41% (80/193) Updating files: 42% (82/193) Updating files: 43% (83/193) Updating files: 44% (85/193) Updating files: 45% (87/193) Updating files: 46% (89/193) Updating files: 47% (91/193) Updating files: 48% (93/193) Updating files: 49% (95/193) Updating files: 50% (97/193) Updating files: 51% (99/193) Updating files: 52% (101/193) Updating files: 53% (103/193) Updating files: 54% (105/193) Updating files: 55% (107/193) Updating files: 56% (109/193) Updating files: 57% (111/193) Updating files: 58% (112/193) Updating files: 59% (114/193) Updating files: 60% (116/193) Updating files: 61% (118/193) Updating files: 62% (120/193) Updating files: 63% (122/193) Updating files: 64% (124/193) Updating files: 65% (126/193) Updating files: 66% (128/193) Updating files: 67% (130/193) Updating files: 68% (132/193) Updating files: 69% (134/193) Updating files: 70% (136/193) Updating files: 71% (138/193) Updating files: 72% (139/193) Updating files: 73% (141/193) Updating files: 74% (143/193) Updating files: 75% (145/193) Updating files: 76% (147/193) Updating files: 77% (149/193) Updating files: 78% (151/193) Updating files: 79% (153/193) Updating files: 80% (155/193) Updating files: 81% (157/193) Updating files: 82% (159/193) Updating files: 83% (161/193) Updating files: 84% (163/193) Updating files: 85% (165/193) Updating files: 86% (166/193) Updating files: 87% (168/193) Updating files: 88% (170/193) Updating files: 88% (171/193) Updating files: 89% (172/193) Updating files: 90% (174/193) Updating files: 91% (176/193) Updating files: 92% (178/193) Updating files: 93% (180/193) Updating files: 94% (182/193) Updating files: 95% (184/193) Updating files: 96% (186/193) Updating files: 97% (188/193) Updating files: 98% (190/193) Updating files: 99% (192/193) Updating files: 100% (193/193) Updating files: 100% (193/193), done. 2023-10-30T14:36:13.539 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 41481853924 kclient: strip out kernel version compatibility cruft 2023-10-30T14:36:13.544 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.9 2023-10-30T14:36:13.645 INFO:tasks.workunit.client.0.smithi057.stdout:v0.10 2023-10-30T14:36:13.645 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 10 -eq 48 ] 2023-10-30T14:36:13.645 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.10 2023-10-30T14:36:13.646 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.10 2023-10-30T14:36:13.646 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.10 2023-10-30T14:36:21.052 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 41% (68/164) Updating files: 42% (69/164) Updating files: 43% (71/164) Updating files: 44% (73/164) Updating files: 45% (74/164) Updating files: 46% (76/164) Updating files: 47% (78/164) Updating files: 48% (79/164) Updating files: 49% (81/164) Updating files: 50% (82/164) Updating files: 51% (84/164) Updating files: 52% (86/164) Updating files: 53% (87/164) Updating files: 54% (89/164) Updating files: 55% (91/164) Updating files: 56% (92/164) Updating files: 57% (94/164) Updating files: 58% (96/164) Updating files: 59% (97/164) Updating files: 60% (99/164) Updating files: 61% (101/164) Updating files: 62% (102/164) Updating files: 63% (104/164) Updating files: 64% (105/164) Updating files: 65% (107/164) Updating files: 66% (109/164) Updating files: 67% (110/164) Updating files: 68% (112/164) Updating files: 69% (114/164) Updating files: 70% (115/164) Updating files: 71% (117/164) Updating files: 72% (119/164) Updating files: 73% (120/164) Updating files: 74% (122/164) Updating files: 75% (123/164) Updating files: 76% (125/164) Updating files: 77% (127/164) Updating files: 78% (128/164) Updating files: 79% (130/164) Updating files: 80% (132/164) Updating files: 81% (133/164) Updating files: 82% (135/164) Updating files: 83% (137/164) Updating files: 84% (138/164) Updating files: 85% (140/164) Updating files: 86% (142/164) Updating files: 87% (143/164) Updating files: 88% (145/164) Updating files: 89% (146/164) Updating files: 90% (148/164) Updating files: 91% (150/164) Updating files: 92% (151/164) Updating files: 93% (153/164) Updating files: 94% (155/164) Updating files: 95% (156/164) Updating files: 96% (158/164) Updating files: 97% (160/164) Updating files: 98% (161/164) Updating files: 99% (163/164) Updating files: 100% (164/164) Updating files: 100% (164/164), done. 2023-10-30T14:36:21.074 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 007313fe844 v0.10 2023-10-30T14:36:21.080 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.10 2023-10-30T14:36:21.126 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 11 -eq 48 ] 2023-10-30T14:36:21.126 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.11 2023-10-30T14:36:21.126 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.11 2023-10-30T14:36:21.127 INFO:tasks.workunit.client.0.smithi057.stdout:v0.11 2023-10-30T14:36:21.127 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.11 2023-10-30T14:36:23.844 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 92% (64/69) Updating files: 94% (65/69) Updating files: 95% (66/69) Updating files: 97% (67/69) Updating files: 98% (68/69) Updating files: 100% (69/69) Updating files: 100% (69/69), done. 2023-10-30T14:36:23.878 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 9fa6f5e7d09 v0.11 2023-10-30T14:36:23.881 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.11 2023-10-30T14:36:24.161 INFO:tasks.workunit.client.0.smithi057.stdout:v0.12 2023-10-30T14:36:24.162 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 12 -eq 48 ] 2023-10-30T14:36:24.162 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.12 2023-10-30T14:36:24.162 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.12 2023-10-30T14:36:24.162 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.12 2023-10-30T14:36:25.300 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at f524150822a v0.12 2023-10-30T14:36:25.304 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.12 2023-10-30T14:36:26.618 INFO:tasks.workunit.client.0.smithi057.stdout:v0.13 2023-10-30T14:36:26.619 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 13 -eq 48 ] 2023-10-30T14:36:26.619 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.13 2023-10-30T14:36:26.619 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.13 2023-10-30T14:36:26.619 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.13 2023-10-30T14:36:28.537 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 55% (74/133) Updating files: 56% (75/133) Updating files: 57% (76/133) Updating files: 58% (78/133) Updating files: 59% (79/133) Updating files: 60% (80/133) Updating files: 61% (82/133) Updating files: 62% (83/133) Updating files: 63% (84/133) Updating files: 64% (86/133) Updating files: 65% (87/133) Updating files: 66% (88/133) Updating files: 67% (90/133) Updating files: 68% (91/133) Updating files: 69% (92/133) Updating files: 70% (94/133) Updating files: 71% (95/133) Updating files: 72% (96/133) Updating files: 73% (98/133) Updating files: 74% (99/133) Updating files: 75% (100/133) Updating files: 76% (102/133) Updating files: 77% (103/133) Updating files: 78% (104/133) Updating files: 79% (106/133) Updating files: 80% (107/133) Updating files: 81% (108/133) Updating files: 82% (110/133) Updating files: 83% (111/133) Updating files: 84% (112/133) Updating files: 85% (114/133) Updating files: 86% (115/133) Updating files: 87% (116/133) Updating files: 88% (118/133) Updating files: 89% (119/133) Updating files: 90% (120/133) Updating files: 91% (122/133) Updating files: 92% (123/133) Updating files: 93% (124/133) Updating files: 94% (126/133) Updating files: 95% (127/133) Updating files: 96% (128/133) Updating files: 97% (130/133) Updating files: 98% (131/133) Updating files: 99% (132/133) Updating files: 100% (133/133) Updating files: 100% (133/133), done. 2023-10-30T14:36:28.569 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 2b32297267d v0.13 2023-10-30T14:36:28.573 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.13 2023-10-30T14:36:28.655 INFO:tasks.workunit.client.0.smithi057.stdout:v0.14 2023-10-30T14:36:28.656 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 14 -eq 48 ] 2023-10-30T14:36:28.656 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.14 2023-10-30T14:36:28.656 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.14 2023-10-30T14:36:28.656 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.14 2023-10-30T14:36:37.360 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:36:37.364 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:36:43.516 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 64% (98/151) Updating files: 65% (99/151) Updating files: 66% (100/151) Updating files: 67% (102/151) Updating files: 68% (103/151) Updating files: 69% (105/151) Updating files: 70% (106/151) Updating files: 71% (108/151) Updating files: 72% (109/151) Updating files: 73% (111/151) Updating files: 74% (112/151) Updating files: 75% (114/151) Updating files: 76% (115/151) Updating files: 77% (117/151) Updating files: 78% (118/151) Updating files: 79% (120/151) Updating files: 80% (121/151) Updating files: 81% (123/151) Updating files: 82% (124/151) Updating files: 83% (126/151) Updating files: 84% (127/151) Updating files: 85% (129/151) Updating files: 86% (130/151) Updating files: 87% (132/151) Updating files: 88% (133/151) Updating files: 89% (135/151) Updating files: 90% (136/151) Updating files: 91% (138/151) Updating files: 92% (139/151) Updating files: 93% (141/151) Updating files: 94% (142/151) Updating files: 95% (144/151) Updating files: 96% (145/151) Updating files: 97% (147/151) Updating files: 98% (148/151) Updating files: 99% (150/151) Updating files: 100% (151/151) Updating files: 100% (151/151), done. 2023-10-30T14:36:43.549 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 1ec1e8d6fb6 v0.14 2023-10-30T14:36:43.553 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.14 2023-10-30T14:36:44.063 INFO:tasks.workunit.client.0.smithi057.stdout:v0.15 2023-10-30T14:36:44.063 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 15 -eq 48 ] 2023-10-30T14:36:44.063 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.15 2023-10-30T14:36:44.064 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.15 2023-10-30T14:36:44.064 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.15 2023-10-30T14:37:03.806 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 11% (13/112) Updating files: 12% (14/112) Updating files: 13% (15/112) Updating files: 14% (16/112) Updating files: 15% (17/112) Updating files: 16% (18/112) Updating files: 17% (20/112) Updating files: 18% (21/112) Updating files: 19% (22/112) Updating files: 20% (23/112) Updating files: 21% (24/112) Updating files: 22% (25/112) Updating files: 23% (26/112) Updating files: 24% (27/112) Updating files: 25% (28/112) Updating files: 26% (30/112) Updating files: 27% (31/112) Updating files: 28% (32/112) Updating files: 29% (33/112) Updating files: 30% (34/112) Updating files: 31% (35/112) Updating files: 32% (36/112) Updating files: 33% (37/112) Updating files: 34% (39/112) Updating files: 35% (40/112) Updating files: 36% (41/112) Updating files: 37% (42/112) Updating files: 38% (43/112) Updating files: 39% (44/112) Updating files: 40% (45/112) Updating files: 41% (46/112) Updating files: 42% (48/112) Updating files: 43% (49/112) Updating files: 44% (50/112) Updating files: 45% (51/112) Updating files: 46% (52/112) Updating files: 47% (53/112) Updating files: 48% (54/112) Updating files: 49% (55/112) Updating files: 50% (56/112) Updating files: 51% (58/112) Updating files: 52% (59/112) Updating files: 53% (60/112) Updating files: 54% (61/112) Updating files: 55% (62/112) Updating files: 56% (63/112) Updating files: 57% (64/112) Updating files: 58% (65/112) Updating files: 59% (67/112) Updating files: 60% (68/112) Updating files: 61% (69/112) Updating files: 62% (70/112) Updating files: 63% (71/112) Updating files: 64% (72/112) Updating files: 65% (73/112) Updating files: 66% (74/112) Updating files: 67% (76/112) Updating files: 68% (77/112) Updating files: 69% (78/112) Updating files: 70% (79/112) Updating files: 71% (80/112) Updating files: 72% (81/112) Updating files: 73% (82/112) Updating files: 74% (83/112) Updating files: 75% (84/112) Updating files: 76% (86/112) Updating files: 77% (87/112) Updating files: 78% (88/112) Updating files: 79% (89/112) Updating files: 80% (90/112) Updating files: 81% (91/112) Updating files: 82% (92/112) Updating files: 83% (93/112) Updating files: 84% (95/112) Updating files: 85% (96/112) Updating files: 86% (97/112) Updating files: 87% (98/112) Updating files: 88% (99/112) Updating files: 89% (100/112) Updating files: 90% (101/112) Updating files: 91% (102/112) Updating files: 92% (104/112) Updating files: 93% (105/112) Updating files: 94% (106/112) Updating files: 95% (107/112) Updating files: 96% (108/112) Updating files: 97% (109/112) Updating files: 98% (110/112) Updating files: 99% (111/112) Updating files: 100% (112/112) Updating files: 100% (112/112), done. 2023-10-30T14:37:03.991 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at ab3c421eb95 v0.15 2023-10-30T14:37:03.995 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.15 2023-10-30T14:37:04.069 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 16 -eq 48 ] 2023-10-30T14:37:04.070 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.16 2023-10-30T14:37:04.070 INFO:tasks.workunit.client.0.smithi057.stdout:v0.16 2023-10-30T14:37:04.070 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.16 2023-10-30T14:37:04.071 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.16 2023-10-30T14:37:07.420 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:37:07.425 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:37:13.942 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 28% (31/107) Updating files: 29% (32/107) Updating files: 30% (33/107) Updating files: 31% (34/107) Updating files: 32% (35/107) Updating files: 33% (36/107) Updating files: 34% (37/107) Updating files: 35% (38/107) Updating files: 36% (39/107) Updating files: 37% (40/107) Updating files: 38% (41/107) Updating files: 39% (42/107) Updating files: 40% (43/107) Updating files: 41% (44/107) Updating files: 42% (45/107) Updating files: 43% (47/107) Updating files: 44% (48/107) Updating files: 45% (49/107) Updating files: 46% (50/107) Updating files: 47% (51/107) Updating files: 48% (52/107) Updating files: 49% (53/107) Updating files: 50% (54/107) Updating files: 51% (55/107) Updating files: 52% (56/107) Updating files: 53% (57/107) Updating files: 54% (58/107) Updating files: 55% (59/107) Updating files: 56% (60/107) Updating files: 57% (61/107) Updating files: 58% (63/107) Updating files: 59% (64/107) Updating files: 60% (65/107) Updating files: 61% (66/107) Updating files: 62% (67/107) Updating files: 63% (68/107) Updating files: 64% (69/107) Updating files: 65% (70/107) Updating files: 66% (71/107) Updating files: 67% (72/107) Updating files: 68% (73/107) Updating files: 69% (74/107) Updating files: 70% (75/107) Updating files: 71% (76/107) Updating files: 72% (78/107) Updating files: 73% (79/107) Updating files: 74% (80/107) Updating files: 75% (81/107) Updating files: 76% (82/107) Updating files: 77% (83/107) Updating files: 78% (84/107) Updating files: 79% (85/107) Updating files: 80% (86/107) Updating files: 81% (87/107) Updating files: 82% (88/107) Updating files: 83% (89/107) Updating files: 84% (90/107) Updating files: 85% (91/107) Updating files: 86% (93/107) Updating files: 87% (94/107) Updating files: 88% (95/107) Updating files: 89% (96/107) Updating files: 90% (97/107) Updating files: 91% (98/107) Updating files: 92% (99/107) Updating files: 93% (100/107) Updating files: 94% (101/107) Updating files: 95% (102/107) Updating files: 96% (103/107) Updating files: 97% (104/107) Updating files: 98% (105/107) Updating files: 99% (106/107) Updating files: 100% (107/107) Updating files: 100% (107/107), done. 2023-10-30T14:37:13.975 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 38dbaa5175c v0.16 2023-10-30T14:37:13.978 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.16 2023-10-30T14:37:14.010 INFO:tasks.workunit.client.0.smithi057.stdout:v0.17 2023-10-30T14:37:14.011 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 17 -eq 48 ] 2023-10-30T14:37:14.011 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.17 2023-10-30T14:37:14.011 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.17 2023-10-30T14:37:14.012 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.17 2023-10-30T14:37:17.937 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 44% (96/217) Updating files: 45% (98/217) Updating files: 46% (100/217) Updating files: 47% (102/217) Updating files: 48% (105/217) Updating files: 49% (107/217) Updating files: 50% (109/217) Updating files: 51% (111/217) Updating files: 52% (113/217) Updating files: 53% (116/217) Updating files: 54% (118/217) Updating files: 55% (120/217) Updating files: 56% (122/217) Updating files: 57% (124/217) Updating files: 58% (126/217) Updating files: 59% (129/217) Updating files: 60% (131/217) Updating files: 61% (133/217) Updating files: 62% (135/217) Updating files: 63% (137/217) Updating files: 64% (139/217) Updating files: 65% (142/217) Updating files: 66% (144/217) Updating files: 67% (146/217) Updating files: 68% (148/217) Updating files: 69% (150/217) Updating files: 70% (152/217) Updating files: 71% (155/217) Updating files: 72% (157/217) Updating files: 73% (159/217) Updating files: 74% (161/217) Updating files: 75% (163/217) Updating files: 76% (165/217) Updating files: 77% (168/217) Updating files: 78% (170/217) Updating files: 79% (172/217) Updating files: 80% (174/217) Updating files: 81% (176/217) Updating files: 82% (178/217) Updating files: 83% (181/217) Updating files: 84% (183/217) Updating files: 85% (185/217) Updating files: 86% (187/217) Updating files: 87% (189/217) Updating files: 88% (191/217) Updating files: 89% (194/217) Updating files: 90% (196/217) Updating files: 91% (198/217) Updating files: 92% (200/217) Updating files: 93% (202/217) Updating files: 94% (204/217) Updating files: 95% (207/217) Updating files: 96% (209/217) Updating files: 97% (211/217) Updating files: 98% (213/217) Updating files: 99% (215/217) Updating files: 100% (217/217) Updating files: 100% (217/217), done. 2023-10-30T14:37:17.958 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at dac3a8c35fb v0.17 2023-10-30T14:37:17.961 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.17 2023-10-30T14:37:18.732 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 18 -eq 48 ] 2023-10-30T14:37:18.733 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.18 2023-10-30T14:37:18.733 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.18 2023-10-30T14:37:18.734 INFO:tasks.workunit.client.0.smithi057.stdout:v0.18 2023-10-30T14:37:18.734 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.18 2023-10-30T14:37:29.469 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 6% (54/779) Updating files: 7% (55/779) Updating files: 8% (63/779) Updating files: 9% (71/779) Updating files: 10% (78/779) Updating files: 11% (86/779) Updating files: 12% (94/779) Updating files: 13% (102/779) Updating files: 14% (110/779) Updating files: 15% (117/779) Updating files: 16% (125/779) Updating files: 17% (133/779) Updating files: 18% (141/779) Updating files: 19% (149/779) Updating files: 20% (156/779) Updating files: 21% (164/779) Updating files: 22% (172/779) Updating files: 23% (180/779) Updating files: 24% (187/779) Updating files: 25% (195/779) Updating files: 26% (203/779) Updating files: 27% (211/779) Updating files: 27% (214/779) Updating files: 28% (219/779) Updating files: 29% (226/779) Updating files: 30% (234/779) Updating files: 31% (242/779) Updating files: 32% (250/779) Updating files: 33% (258/779) Updating files: 34% (265/779) Updating files: 35% (273/779) Updating files: 36% (281/779) Updating files: 37% (289/779) Updating files: 38% (297/779) Updating files: 38% (302/779) Updating files: 39% (304/779) Updating files: 40% (312/779) Updating files: 41% (320/779) Updating files: 42% (328/779) Updating files: 43% (335/779) Updating files: 44% (343/779) Updating files: 44% (350/779) Updating files: 45% (351/779) Updating files: 46% (359/779) Updating files: 47% (367/779) Updating files: 48% (374/779) Updating files: 49% (382/779) Updating files: 50% (390/779) Updating files: 51% (398/779) Updating files: 52% (406/779) Updating files: 53% (413/779) Updating files: 54% (421/779) Updating files: 55% (429/779) Updating files: 56% (437/779) Updating files: 57% (445/779) Updating files: 58% (452/779) Updating files: 59% (460/779) Updating files: 59% (464/779) Updating files: 60% (468/779) Updating files: 61% (476/779) Updating files: 61% (480/779) Updating files: 62% (483/779) Updating files: 63% (491/779) Updating files: 64% (499/779) Updating files: 65% (507/779) Updating files: 66% (515/779) Updating files: 67% (522/779) Updating files: 68% (530/779) Updating files: 69% (538/779) Updating files: 70% (546/779) Updating files: 70% (548/779) Updating files: 71% (554/779) Updating files: 72% (561/779) Updating files: 73% (569/779) Updating files: 74% (577/779) Updating files: 74% (581/779) Updating files: 75% (585/779) Updating files: 76% (593/779) Updating files: 77% (600/779) Updating files: 77% (601/779) Updating files: 78% (608/779) Updating files: 79% (616/779) Updating files: 80% (624/779) Updating files: 81% (631/779) Updating files: 82% (639/779) Updating files: 83% (647/779) Updating files: 84% (655/779) Updating files: 85% (663/779) Updating files: 86% (670/779) Updating files: 87% (678/779) Updating files: 88% (686/779) Updating files: 89% (694/779) Updating files: 90% (702/779) Updating files: 91% (709/779) Updating files: 91% (714/779) Updating files: 92% (717/779) Updating files: 93% (725/779) Updating files: 94% (733/779) Updating files: 95% (741/779) Updating files: 96% (748/779) Updating files: 97% (756/779) Updating files: 98% (764/779) Updating files: 99% (772/779) Updating files: 100% (779/779) Updating files: 100% (779/779), done. 2023-10-30T14:37:29.510 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 2f10137986a v0.18 2023-10-30T14:37:29.515 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.18 2023-10-30T14:37:29.594 INFO:tasks.workunit.client.0.smithi057.stdout:v0.19 2023-10-30T14:37:29.595 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 19 -eq 48 ] 2023-10-30T14:37:29.595 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.19 2023-10-30T14:37:29.595 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.19 2023-10-30T14:37:29.596 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.19 2023-10-30T14:37:37.490 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:37:37.498 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:37:38.455 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 11% (26/217) Updating files: 12% (27/217) Updating files: 13% (29/217) Updating files: 14% (31/217) Updating files: 15% (33/217) Updating files: 16% (35/217) Updating files: 17% (37/217) Updating files: 18% (40/217) Updating files: 19% (42/217) Updating files: 20% (44/217) Updating files: 21% (46/217) Updating files: 22% (48/217) Updating files: 23% (50/217) Updating files: 24% (53/217) Updating files: 25% (55/217) Updating files: 26% (57/217) Updating files: 27% (59/217) Updating files: 28% (61/217) Updating files: 29% (63/217) Updating files: 30% (66/217) Updating files: 31% (68/217) Updating files: 32% (70/217) Updating files: 33% (72/217) Updating files: 34% (74/217) Updating files: 35% (76/217) Updating files: 36% (79/217) Updating files: 37% (81/217) Updating files: 38% (83/217) Updating files: 39% (85/217) Updating files: 40% (87/217) Updating files: 41% (89/217) Updating files: 42% (92/217) Updating files: 43% (94/217) Updating files: 44% (96/217) Updating files: 45% (98/217) Updating files: 46% (100/217) Updating files: 47% (102/217) Updating files: 48% (105/217) Updating files: 49% (107/217) Updating files: 50% (109/217) Updating files: 51% (111/217) Updating files: 52% (113/217) Updating files: 53% (116/217) Updating files: 54% (118/217) Updating files: 55% (120/217) Updating files: 56% (122/217) Updating files: 57% (124/217) Updating files: 58% (126/217) Updating files: 59% (129/217) Updating files: 60% (131/217) Updating files: 61% (133/217) Updating files: 62% (135/217) Updating files: 63% (137/217) Updating files: 64% (139/217) Updating files: 65% (142/217) Updating files: 66% (144/217) Updating files: 67% (146/217) Updating files: 68% (148/217) Updating files: 69% (150/217) Updating files: 70% (152/217) Updating files: 71% (155/217) Updating files: 72% (157/217) Updating files: 73% (159/217) Updating files: 74% (161/217) Updating files: 75% (163/217) Updating files: 76% (165/217) Updating files: 77% (168/217) Updating files: 78% (170/217) Updating files: 79% (172/217) Updating files: 80% (174/217) Updating files: 81% (176/217) Updating files: 82% (178/217) Updating files: 83% (181/217) Updating files: 84% (183/217) Updating files: 85% (185/217) Updating files: 85% (186/217) Updating files: 86% (187/217) Updating files: 87% (189/217) Updating files: 88% (191/217) Updating files: 89% (194/217) Updating files: 90% (196/217) Updating files: 91% (198/217) Updating files: 92% (200/217) Updating files: 93% (202/217) Updating files: 94% (204/217) Updating files: 95% (207/217) Updating files: 96% (209/217) Updating files: 97% (211/217) Updating files: 98% (213/217) Updating files: 99% (215/217) Updating files: 100% (217/217) Updating files: 100% (217/217), done. 2023-10-30T14:37:38.530 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 98f5be53f85 v0.19 2023-10-30T14:37:38.534 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.19 2023-10-30T14:37:39.170 INFO:tasks.workunit.client.0.smithi057.stdout:v0.20 2023-10-30T14:37:39.171 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 20 -eq 48 ] 2023-10-30T14:37:39.171 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.20 2023-10-30T14:37:39.171 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.20 2023-10-30T14:37:39.171 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.20 2023-10-30T14:37:43.244 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 31% (77/245) Updating files: 32% (79/245) Updating files: 33% (81/245) Updating files: 34% (84/245) Updating files: 35% (86/245) Updating files: 36% (89/245) Updating files: 37% (91/245) Updating files: 38% (94/245) Updating files: 39% (96/245) Updating files: 40% (98/245) Updating files: 41% (101/245) Updating files: 42% (103/245) Updating files: 43% (106/245) Updating files: 44% (108/245) Updating files: 45% (111/245) Updating files: 46% (113/245) Updating files: 47% (116/245) Updating files: 48% (118/245) Updating files: 49% (121/245) Updating files: 50% (123/245) Updating files: 51% (125/245) Updating files: 51% (127/245) Updating files: 52% (128/245) Updating files: 53% (130/245) Updating files: 54% (133/245) Updating files: 55% (135/245) Updating files: 56% (138/245) Updating files: 57% (140/245) Updating files: 58% (143/245) Updating files: 59% (145/245) Updating files: 60% (147/245) Updating files: 61% (150/245) Updating files: 62% (152/245) Updating files: 63% (155/245) Updating files: 64% (157/245) Updating files: 65% (160/245) Updating files: 66% (162/245) Updating files: 67% (165/245) Updating files: 68% (167/245) Updating files: 69% (170/245) Updating files: 70% (172/245) Updating files: 71% (174/245) Updating files: 72% (177/245) Updating files: 73% (179/245) Updating files: 74% (182/245) Updating files: 75% (184/245) Updating files: 76% (187/245) Updating files: 77% (189/245) Updating files: 78% (192/245) Updating files: 79% (194/245) Updating files: 80% (196/245) Updating files: 81% (199/245) Updating files: 82% (201/245) Updating files: 83% (204/245) Updating files: 84% (206/245) Updating files: 85% (209/245) Updating files: 86% (211/245) Updating files: 87% (214/245) Updating files: 88% (216/245) Updating files: 89% (219/245) Updating files: 90% (221/245) Updating files: 91% (223/245) Updating files: 92% (226/245) Updating files: 93% (228/245) Updating files: 94% (231/245) Updating files: 95% (233/245) Updating files: 96% (236/245) Updating files: 97% (238/245) Updating files: 98% (241/245) Updating files: 99% (243/245) Updating files: 100% (245/245) Updating files: 100% (245/245), done. 2023-10-30T14:37:43.299 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 266d2076b6c debian: fix up debian scripts 2023-10-30T14:37:43.302 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.20 2023-10-30T14:37:44.136 INFO:tasks.workunit.client.0.smithi057.stdout:v0.21 2023-10-30T14:37:44.136 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 21 -eq 48 ] 2023-10-30T14:37:44.136 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.21 2023-10-30T14:37:44.136 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.21 2023-10-30T14:37:44.137 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.21 2023-10-30T14:37:52.048 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 24% (133/537) Updating files: 25% (135/537) Updating files: 26% (140/537) Updating files: 27% (145/537) Updating files: 28% (151/537) Updating files: 29% (156/537) Updating files: 30% (162/537) Updating files: 31% (167/537) Updating files: 32% (172/537) Updating files: 33% (178/537) Updating files: 34% (183/537) Updating files: 35% (188/537) Updating files: 36% (194/537) Updating files: 37% (199/537) Updating files: 38% (205/537) Updating files: 38% (206/537) Updating files: 39% (210/537) Updating files: 40% (215/537) Updating files: 41% (221/537) Updating files: 42% (226/537) Updating files: 43% (231/537) Updating files: 44% (237/537) Updating files: 45% (242/537) Updating files: 46% (248/537) Updating files: 47% (253/537) Updating files: 48% (258/537) Updating files: 49% (264/537) Updating files: 50% (269/537) Updating files: 51% (274/537) Updating files: 52% (280/537) Updating files: 53% (285/537) Updating files: 54% (290/537) Updating files: 55% (296/537) Updating files: 56% (301/537) Updating files: 57% (307/537) Updating files: 58% (312/537) Updating files: 59% (317/537) Updating files: 60% (323/537) Updating files: 61% (328/537) Updating files: 62% (333/537) Updating files: 63% (339/537) Updating files: 64% (344/537) Updating files: 65% (350/537) Updating files: 66% (355/537) Updating files: 67% (360/537) Updating files: 68% (366/537) Updating files: 69% (371/537) Updating files: 70% (376/537) Updating files: 71% (382/537) Updating files: 72% (387/537) Updating files: 73% (393/537) Updating files: 74% (398/537) Updating files: 75% (403/537) Updating files: 75% (406/537) Updating files: 76% (409/537) Updating files: 77% (414/537) Updating files: 78% (419/537) Updating files: 79% (425/537) Updating files: 80% (430/537) Updating files: 81% (435/537) Updating files: 82% (441/537) Updating files: 83% (446/537) Updating files: 84% (452/537) Updating files: 84% (455/537) Updating files: 85% (457/537) Updating files: 86% (462/537) Updating files: 87% (468/537) Updating files: 88% (473/537) Updating files: 89% (478/537) Updating files: 90% (484/537) Updating files: 91% (489/537) Updating files: 92% (495/537) Updating files: 93% (500/537) Updating files: 94% (505/537) Updating files: 95% (511/537) Updating files: 96% (516/537) Updating files: 97% (521/537) Updating files: 97% (525/537) Updating files: 98% (527/537) Updating files: 99% (532/537) Updating files: 100% (537/537) Updating files: 100% (537/537), done. 2023-10-30T14:37:52.084 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 090436f56ac v0.21 2023-10-30T14:37:52.090 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.21 2023-10-30T14:37:52.562 INFO:tasks.workunit.client.0.smithi057.stdout:v0.22 2023-10-30T14:37:52.563 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 22 -eq 48 ] 2023-10-30T14:37:52.563 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.22 2023-10-30T14:37:52.563 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.22 2023-10-30T14:37:52.563 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.22 2023-10-30T14:37:58.555 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:37:58.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.718+0000 7fec8b59e640 1 Processor -- start 2023-10-30T14:37:58.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.718+0000 7fec8b59e640 1 -- start start 2023-10-30T14:37:58.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.718+0000 7fec8b59e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec8c153b50 0x7fec8c1597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:37:58.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.718+0000 7fec8b59e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8c15a770 con 0x7fec8c0cb180 2023-10-30T14:37:58.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.718+0000 7fec8b59e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8c0c9d40 con 0x7fec8c0c62b0 2023-10-30T14:37:58.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.718+0000 7fec8b59e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c159cf0 con 0x7fec8c153b50 2023-10-30T14:37:58.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.718+0000 7fec8ad9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fec8c0cb180 0x7fec8c06ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50676/0 (socket says 172.21.15.57:50676) 2023-10-30T14:37:58.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.718+0000 7fec8ad9d640 1 -- 172.21.15.57:0/2910867191 learned_addr learned my addr 172.21.15.57:0/2910867191 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:37:58.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.718+0000 7fec8a59c640 1 --2- 172.21.15.57:0/2910867191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec8c153b50 0x7fec8c1597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:37:58.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.718+0000 7fec89d9b640 1 -- 172.21.15.57:0/2910867191 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540223427 0 0) 0x7fec840014a0 con 0x7fec8c0cb180 2023-10-30T14:37:58.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.718+0000 7fec89d9b640 1 -- 172.21.15.57:0/2910867191 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec78003530 con 0x7fec8c0cb180 2023-10-30T14:37:58.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.718+0000 7fec89d9b640 1 -- 172.21.15.57:0/2910867191 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360643539 0 0) 0x7fec8c0c9d40 con 0x7fec8c0c62b0 2023-10-30T14:37:58.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.718+0000 7fec89d9b640 1 -- 172.21.15.57:0/2910867191 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec840014a0 con 0x7fec8c0c62b0 2023-10-30T14:37:58.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.718+0000 7fec89d9b640 1 -- 172.21.15.57:0/2910867191 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4133162298 0 0) 0x7fec78003530 con 0x7fec8c0cb180 2023-10-30T14:37:58.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.718+0000 7fec8a59c640 1 -- 172.21.15.57:0/2910867191 >> v1:172.21.15.88:6789/0 conn(0x7fec8c0c62b0 legacy=0x7fec8c0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:37:58.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.718+0000 7fec8a59c640 1 -- 172.21.15.57:0/2910867191 >> v1:172.21.15.57:6789/0 conn(0x7fec8c0cb180 legacy=0x7fec8c06ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:37:58.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.718+0000 7fec8a59c640 1 -- 172.21.15.57:0/2910867191 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8c153f30 con 0x7fec8c153b50 2023-10-30T14:37:58.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.722+0000 7fec8a59c640 1 --2- 172.21.15.57:0/2910867191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec8c153b50 0x7fec8c1597b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0097f0 tx=0x7fec7c01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d6180a5630546b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:37:58.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.722+0000 7fec89d9b640 1 -- 172.21.15.57:0/2910867191 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec84001b90 con 0x7fec8c153b50 2023-10-30T14:37:58.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.722+0000 7fec89d9b640 1 -- 172.21.15.57:0/2910867191 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7c0a7050 con 0x7fec8c153b50 2023-10-30T14:37:58.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.722+0000 7fec89d9b640 1 -- 172.21.15.57:0/2910867191 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c0ab600 con 0x7fec8c153b50 2023-10-30T14:37:58.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.722+0000 7fec8b59e640 1 -- 172.21.15.57:0/2910867191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec8c153b50 msgr2=0x7fec8c1597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:37:58.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.722+0000 7fec8b59e640 1 --2- 172.21.15.57:0/2910867191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec8c153b50 0x7fec8c1597b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0097f0 tx=0x7fec7c01c3e0 comp rx=0 tx=0).stop 2023-10-30T14:37:58.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.722+0000 7fec8b59e640 1 -- 172.21.15.57:0/2910867191 shutdown_connections 2023-10-30T14:37:58.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.722+0000 7fec8b59e640 1 --2- 172.21.15.57:0/2910867191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec8c153b50 0x7fec8c1597b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:37:58.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.722+0000 7fec8b59e640 1 -- 172.21.15.57:0/2910867191 >> 172.21.15.57:0/2910867191 conn(0x7fec8c065a80 msgr2=0x7fec8c065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:37:58.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.722+0000 7fec8b59e640 1 -- 172.21.15.57:0/2910867191 shutdown_connections 2023-10-30T14:37:58.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.722+0000 7fec8b59e640 1 -- 172.21.15.57:0/2910867191 wait complete. 2023-10-30T14:37:58.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8b59e640 1 Processor -- start 2023-10-30T14:37:58.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8b59e640 1 -- start start 2023-10-30T14:37:58.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8b59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec8c0c62b0 0x7fec8c0cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:37:58.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8b59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec8c0cb180 0x7fec8c0c13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:37:58.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8b59e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec8c153b50 0x7fec8c1470f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:37:58.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8b59e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c157480 con 0x7fec8c0c62b0 2023-10-30T14:37:58.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8b59e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c157760 con 0x7fec8c0cb180 2023-10-30T14:37:58.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8b59e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c0ccdc0 con 0x7fec8c153b50 2023-10-30T14:37:58.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec8c0c62b0 0x7fec8c0cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:37:58.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec8c0c62b0 0x7fec8c0cd8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45992/0 (socket says 172.21.15.57:45992) 2023-10-30T14:37:58.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8a59c640 1 -- 172.21.15.57:0/3126071244 learned_addr learned my addr 172.21.15.57:0/3126071244 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:37:58.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec83fff640 1 --2- 172.21.15.57:0/3126071244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec8c0cb180 0x7fec8c0c13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:37:58.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8ad9d640 1 --2- 172.21.15.57:0/3126071244 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec8c153b50 0x7fec8c1470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:37:58.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8a59c640 1 -- 172.21.15.57:0/3126071244 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec8c153b50 msgr2=0x7fec8c1470f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:37:58.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8a59c640 1 --2- 172.21.15.57:0/3126071244 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec8c153b50 0x7fec8c1470f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:37:58.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8a59c640 1 -- 172.21.15.57:0/3126071244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec8c0cb180 msgr2=0x7fec8c0c13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:37:58.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8a59c640 1 --2- 172.21.15.57:0/3126071244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec8c0cb180 0x7fec8c0c13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:37:58.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8a59c640 1 -- 172.21.15.57:0/3126071244 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7c009420 con 0x7fec8c0c62b0 2023-10-30T14:37:58.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8a59c640 1 --2- 172.21.15.57:0/3126071244 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec8c0c62b0 0x7fec8c0cd8a0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fec7c01a010 tx=0x7fec7c019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:37:58.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec82ffd640 1 -- 172.21.15.57:0/3126071244 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c0038a0 con 0x7fec8c0c62b0 2023-10-30T14:37:58.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec82ffd640 1 -- 172.21.15.57:0/3126071244 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7c003a00 con 0x7fec8c0c62b0 2023-10-30T14:37:58.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8b59e640 1 -- 172.21.15.57:0/3126071244 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec8c0cd0a0 con 0x7fec8c0c62b0 2023-10-30T14:37:58.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec82ffd640 1 -- 172.21.15.57:0/3126071244 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c003d70 con 0x7fec8c0c62b0 2023-10-30T14:37:58.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8b59e640 1 -- 172.21.15.57:0/3126071244 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8c159a50 con 0x7fec8c0c62b0 2023-10-30T14:37:58.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.726+0000 7fec8b59e640 1 -- 172.21.15.57:0/3126071244 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec8c067540 con 0x7fec8c0c62b0 2023-10-30T14:37:58.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.730+0000 7fec82ffd640 1 -- 172.21.15.57:0/3126071244 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fec7c0b2040 con 0x7fec8c0c62b0 2023-10-30T14:37:58.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.730+0000 7fec82ffd640 1 --2- 172.21.15.57:0/3126071244 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fec6c0664a0 0x7fec6c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:37:58.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.730+0000 7fec82ffd640 1 -- 172.21.15.57:0/3126071244 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fec7c11de60 con 0x7fec8c0c62b0 2023-10-30T14:37:58.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.730+0000 7fec83fff640 1 --2- 172.21.15.57:0/3126071244 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fec6c0664a0 0x7fec6c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:37:58.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.730+0000 7fec83fff640 1 --2- 172.21.15.57:0/3126071244 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fec6c0664a0 0x7fec6c068960 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fec8c0c7f30 tx=0x7fec74009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:37:58.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.734+0000 7fec82ffd640 1 -- 172.21.15.57:0/3126071244 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fec7c0eae70 con 0x7fec8c0c62b0 2023-10-30T14:37:58.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.922+0000 7fec8b59e640 1 -- 172.21.15.57:0/3126071244 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec8c0cd550 con 0x7fec8c0c62b0 2023-10-30T14:37:58.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.922+0000 7fec82ffd640 1 -- 172.21.15.57:0/3126071244 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6535 (secure 0 0 0) 0x7fec7c0b0020 con 0x7fec8c0c62b0 2023-10-30T14:37:58.924 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:37:58.925 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6125,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:36:09.152736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:37:58.925 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 127 2023-10-30T14:37:58.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.926+0000 7fec8b59e640 1 -- 172.21.15.57:0/3126071244 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fec6c0664a0 msgr2=0x7fec6c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:37:58.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.926+0000 7fec8b59e640 1 --2- 172.21.15.57:0/3126071244 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fec6c0664a0 0x7fec6c068960 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fec8c0c7f30 tx=0x7fec74009040 comp rx=0 tx=0).stop 2023-10-30T14:37:58.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.926+0000 7fec8b59e640 1 -- 172.21.15.57:0/3126071244 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec8c0c62b0 msgr2=0x7fec8c0cd8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:37:58.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.926+0000 7fec8b59e640 1 --2- 172.21.15.57:0/3126071244 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec8c0c62b0 0x7fec8c0cd8a0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fec7c01a010 tx=0x7fec7c019180 comp rx=0 tx=0).stop 2023-10-30T14:37:58.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.926+0000 7fec8b59e640 1 -- 172.21.15.57:0/3126071244 shutdown_connections 2023-10-30T14:37:58.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.926+0000 7fec8b59e640 1 --2- 172.21.15.57:0/3126071244 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fec6c0664a0 0x7fec6c068960 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:37:58.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.926+0000 7fec8b59e640 1 --2- 172.21.15.57:0/3126071244 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec8c153b50 0x7fec8c1470f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:37:58.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.926+0000 7fec8b59e640 1 --2- 172.21.15.57:0/3126071244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec8c0cb180 0x7fec8c0c13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:37:58.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.926+0000 7fec8b59e640 1 --2- 172.21.15.57:0/3126071244 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec8c0c62b0 0x7fec8c0cd8a0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:37:58.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.926+0000 7fec8b59e640 1 -- 172.21.15.57:0/3126071244 >> 172.21.15.57:0/3126071244 conn(0x7fec8c065a80 msgr2=0x7fec8c1455e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:37:58.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.926+0000 7fec8b59e640 1 -- 172.21.15.57:0/3126071244 shutdown_connections 2023-10-30T14:37:58.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:58.926+0000 7fec8b59e640 1 -- 172.21.15.57:0/3126071244 wait complete. 2023-10-30T14:37:58.946 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=2) 2023-10-30T14:37:58.946 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T14:37:58.947 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:37:58.964 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T14:37:58.964 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:37:59.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.154+0000 7f2db4f2a640 1 Processor -- start 2023-10-30T14:37:59.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.154+0000 7f2db4f2a640 1 -- start start 2023-10-30T14:37:59.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.154+0000 7f2db4f2a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2db006b760 0x7f2db015e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:37:59.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.154+0000 7f2db4f2a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2db006c420 0x7f2db00c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:37:59.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.154+0000 7f2db4f2a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db0071e00 con 0x7f2db006bb40 2023-10-30T14:37:59.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.154+0000 7f2db4f2a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db0070020 con 0x7f2db006b760 2023-10-30T14:37:59.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.154+0000 7f2db4f2a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db015eb90 con 0x7f2db006c420 2023-10-30T14:37:59.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.158+0000 7f2dadd9b640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f2db006bb40 0x7f2db0157a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50690/0 (socket says 172.21.15.57:50690) 2023-10-30T14:37:59.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.158+0000 7f2dadd9b640 1 -- 172.21.15.57:0/760449467 learned_addr learned my addr 172.21.15.57:0/760449467 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:37:59.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.158+0000 7f2dae59c640 1 --2- 172.21.15.57:0/760449467 >> v2:172.21.15.88:3300/0 conn(0x7f2db006b760 0x7f2db015e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:37:59.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.162+0000 7f2dad59a640 1 -- 172.21.15.57:0/760449467 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123863331 0 0) 0x7f2db0071e00 con 0x7f2db006bb40 2023-10-30T14:37:59.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.162+0000 7f2dad59a640 1 -- 172.21.15.57:0/760449467 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d94003530 con 0x7f2db006bb40 2023-10-30T14:37:59.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.162+0000 7f2dad59a640 1 -- 172.21.15.57:0/760449467 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1091287514 0 0) 0x7f2d94003530 con 0x7f2db006bb40 2023-10-30T14:37:59.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.162+0000 7f2dad59a640 1 -- 172.21.15.57:0/760449467 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2db006c420 msgr2=0x7f2db00c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:37:59.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.162+0000 7f2dad59a640 1 --2- 172.21.15.57:0/760449467 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2db006c420 0x7f2db00c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:37:59.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.162+0000 7f2dad59a640 1 -- 172.21.15.57:0/760449467 >> v2:172.21.15.88:3300/0 conn(0x7f2db006b760 msgr2=0x7f2db015e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:37:59.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.162+0000 7f2dad59a640 1 --2- 172.21.15.57:0/760449467 >> v2:172.21.15.88:3300/0 conn(0x7f2db006b760 0x7f2db015e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:37:59.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.162+0000 7f2dad59a640 1 -- 172.21.15.57:0/760449467 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db0070f10 con 0x7f2db006bb40 2023-10-30T14:37:59.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.162+0000 7f2dad59a640 1 -- 172.21.15.57:0/760449467 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f2d9c001680 con 0x7f2db006bb40 2023-10-30T14:37:59.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.162+0000 7f2dad59a640 1 -- 172.21.15.57:0/760449467 >> v1:172.21.15.57:6789/0 conn(0x7f2db006bb40 legacy=0x7f2db0157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:37:59.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.162+0000 7f2dad59a640 1 --2- 172.21.15.57:0/760449467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d94003980 0x7f2d94003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:37:59.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.162+0000 7f2dad59a640 1 --2- 172.21.15.57:0/760449467 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d940047d0 0x7f2d94004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:37:59.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.162+0000 7f2dad59a640 1 --2- 172.21.15.57:0/760449467 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2db006c420 0x7f2d940051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:37:59.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.162+0000 7f2dad59a640 1 -- 172.21.15.57:0/760449467 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db0070020 con 0x7f2d940047d0 2023-10-30T14:37:59.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.162+0000 7f2dad59a640 1 -- 172.21.15.57:0/760449467 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db015eb90 con 0x7f2d94003980 2023-10-30T14:37:59.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.162+0000 7f2dad59a640 1 -- 172.21.15.57:0/760449467 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9400bb20 con 0x7f2db006c420 2023-10-30T14:37:59.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.166+0000 7f2daed9d640 1 --2- 172.21.15.57:0/760449467 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2db006c420 0x7f2d940051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:37:59.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.166+0000 7f2dadd9b640 1 --2- 172.21.15.57:0/760449467 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d940047d0 0x7f2d94004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:37:59.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.166+0000 7f2daed9d640 1 -- 172.21.15.57:0/760449467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d94003980 msgr2=0x7f2d94003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:37:59.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.166+0000 7f2dae59c640 1 --2- 172.21.15.57:0/760449467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d94003980 0x7f2d94003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:37:59.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.166+0000 7f2daed9d640 1 --2- 172.21.15.57:0/760449467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d94003980 0x7f2d94003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:37:59.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.166+0000 7f2daed9d640 1 -- 172.21.15.57:0/760449467 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d940047d0 msgr2=0x7f2d94004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:37:59.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.166+0000 7f2daed9d640 1 --2- 172.21.15.57:0/760449467 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d940047d0 0x7f2d94004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:37:59.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.166+0000 7f2daed9d640 1 -- 172.21.15.57:0/760449467 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db0070f10 con 0x7f2db006c420 2023-10-30T14:37:59.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.166+0000 7f2dadd9b640 1 --2- 172.21.15.57:0/760449467 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d940047d0 0x7f2d94004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:37:59.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.170+0000 7f2dae59c640 1 --2- 172.21.15.57:0/760449467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d94003980 0x7f2d94003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:37:59.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.174+0000 7f2daed9d640 1 --2- 172.21.15.57:0/760449467 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2db006c420 0x7f2d940051b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2da8009ca0 tx=0x7f2da801cc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b84ff5b831e00d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:37:59.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.174+0000 7f2dad59a640 1 -- 172.21.15.57:0/760449467 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da80a5c60 con 0x7f2db006c420 2023-10-30T14:37:59.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.174+0000 7f2dad59a640 1 -- 172.21.15.57:0/760449467 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da80a5de0 con 0x7f2db006c420 2023-10-30T14:37:59.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.174+0000 7f2dad59a640 1 -- 172.21.15.57:0/760449467 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da80ad680 con 0x7f2db006c420 2023-10-30T14:37:59.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.174+0000 7f2db4f2a640 1 -- 172.21.15.57:0/760449467 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2db006c420 msgr2=0x7f2d940051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:37:59.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.174+0000 7f2db4f2a640 1 --2- 172.21.15.57:0/760449467 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2db006c420 0x7f2d940051b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2da8009ca0 tx=0x7f2da801cc50 comp rx=0 tx=0).stop 2023-10-30T14:37:59.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.174+0000 7f2daed9d640 1 -- 172.21.15.57:0/760449467 reap_dead start 2023-10-30T14:37:59.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.174+0000 7f2db4f2a640 1 -- 172.21.15.57:0/760449467 shutdown_connections 2023-10-30T14:37:59.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.174+0000 7f2db4f2a640 1 -- 172.21.15.57:0/760449467 >> 172.21.15.57:0/760449467 conn(0x7f2db00cac70 msgr2=0x7f2db00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:37:59.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.174+0000 7f2db4f2a640 1 -- 172.21.15.57:0/760449467 shutdown_connections 2023-10-30T14:37:59.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.174+0000 7f2db4f2a640 1 -- 172.21.15.57:0/760449467 wait complete. 2023-10-30T14:37:59.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.178+0000 7f2db4f2a640 1 Processor -- start 2023-10-30T14:37:59.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.178+0000 7f2db4f2a640 1 -- start start 2023-10-30T14:37:59.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2db4f2a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2db006c420 0x7f2db00c4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:37:59.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2db4f2a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2db00c4030 0x7f2db0161930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:37:59.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2db4f2a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2db00ca650 0x7f2db014f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:37:59.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2db4f2a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db00c4a70 con 0x7f2db006c420 2023-10-30T14:37:59.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2db4f2a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db015e460 con 0x7f2db00c4030 2023-10-30T14:37:59.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2db4f2a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db01f6c70 con 0x7f2db00ca650 2023-10-30T14:37:59.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2dae59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2db006c420 0x7f2db00c4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:37:59.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2dae59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2db006c420 0x7f2db00c4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46006/0 (socket says 172.21.15.57:46006) 2023-10-30T14:37:59.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2dae59c640 1 -- 172.21.15.57:0/2655017856 learned_addr learned my addr 172.21.15.57:0/2655017856 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:37:59.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2dadd9b640 1 --2- 172.21.15.57:0/2655017856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2db00c4030 0x7f2db0161930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:37:59.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2daed9d640 1 --2- 172.21.15.57:0/2655017856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2db00ca650 0x7f2db014f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:37:59.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2dae59c640 1 -- 172.21.15.57:0/2655017856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2db00ca650 msgr2=0x7f2db014f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:37:59.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2dae59c640 1 --2- 172.21.15.57:0/2655017856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2db00ca650 0x7f2db014f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:37:59.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2dae59c640 1 -- 172.21.15.57:0/2655017856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2db00c4030 msgr2=0x7f2db0161930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:37:59.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2dae59c640 1 --2- 172.21.15.57:0/2655017856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2db00c4030 0x7f2db0161930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:37:59.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2dae59c640 1 -- 172.21.15.57:0/2655017856 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db015e700 con 0x7f2db006c420 2023-10-30T14:37:59.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2dadd9b640 1 --2- 172.21.15.57:0/2655017856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2db00c4030 0x7f2db0161930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:37:59.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2dae59c640 1 --2- 172.21.15.57:0/2655017856 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2db006c420 0x7f2db00c4470 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f2da000e810 tx=0x7f2da000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:37:59.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2da6ffd640 1 -- 172.21.15.57:0/2655017856 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da000c8d0 con 0x7f2db006c420 2023-10-30T14:37:59.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2da6ffd640 1 -- 172.21.15.57:0/2655017856 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da000ca30 con 0x7f2db006c420 2023-10-30T14:37:59.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2db4f2a640 1 -- 172.21.15.57:0/2655017856 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db014f890 con 0x7f2db006c420 2023-10-30T14:37:59.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2da6ffd640 1 -- 172.21.15.57:0/2655017856 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da000cda0 con 0x7f2db006c420 2023-10-30T14:37:59.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2db4f2a640 1 -- 172.21.15.57:0/2655017856 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db00c9d00 con 0x7f2db006c420 2023-10-30T14:37:59.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.182+0000 7f2db4f2a640 1 -- 172.21.15.57:0/2655017856 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db015e890 con 0x7f2db006c420 2023-10-30T14:37:59.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.186+0000 7f2da6ffd640 1 -- 172.21.15.57:0/2655017856 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2da0013410 con 0x7f2db006c420 2023-10-30T14:37:59.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.186+0000 7f2da6ffd640 1 --2- 172.21.15.57:0/2655017856 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2d980664d0 0x7f2d98068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:37:59.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.186+0000 7f2da6ffd640 1 -- 172.21.15.57:0/2655017856 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f2da0084160 con 0x7f2db006c420 2023-10-30T14:37:59.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.186+0000 7f2dadd9b640 1 --2- 172.21.15.57:0/2655017856 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2d980664d0 0x7f2d98068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:37:59.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.186+0000 7f2dadd9b640 1 --2- 172.21.15.57:0/2655017856 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2d980664d0 0x7f2d98068990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c000fc0 tx=0x7f2d9c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:37:59.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.194+0000 7f2da6ffd640 1 -- 172.21.15.57:0/2655017856 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2da0051150 con 0x7f2db006c420 2023-10-30T14:37:59.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.386+0000 7f2db4f2a640 1 -- 172.21.15.57:0/2655017856 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2db00c3e80 con 0x7f2db006c420 2023-10-30T14:37:59.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.386+0000 7f2da6ffd640 1 -- 172.21.15.57:0/2655017856 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6535 (secure 0 0 0) 0x7f2da0054770 con 0x7f2db006c420 2023-10-30T14:37:59.388 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:37:59.388 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6125,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:36:09.152736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:37:59.388 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 127 2023-10-30T14:37:59.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.390+0000 7f2db4f2a640 1 -- 172.21.15.57:0/2655017856 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2d980664d0 msgr2=0x7f2d98068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:37:59.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.390+0000 7f2db4f2a640 1 --2- 172.21.15.57:0/2655017856 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2d980664d0 0x7f2d98068990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c000fc0 tx=0x7f2d9c009040 comp rx=0 tx=0).stop 2023-10-30T14:37:59.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.390+0000 7f2db4f2a640 1 -- 172.21.15.57:0/2655017856 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2db006c420 msgr2=0x7f2db00c4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:37:59.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.390+0000 7f2db4f2a640 1 --2- 172.21.15.57:0/2655017856 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2db006c420 0x7f2db00c4470 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f2da000e810 tx=0x7f2da000ecd0 comp rx=0 tx=0).stop 2023-10-30T14:37:59.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.390+0000 7f2db4f2a640 1 -- 172.21.15.57:0/2655017856 shutdown_connections 2023-10-30T14:37:59.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.390+0000 7f2db4f2a640 1 --2- 172.21.15.57:0/2655017856 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2d980664d0 0x7f2d98068990 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:37:59.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.390+0000 7f2db4f2a640 1 --2- 172.21.15.57:0/2655017856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2db00ca650 0x7f2db014f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:37:59.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.390+0000 7f2db4f2a640 1 --2- 172.21.15.57:0/2655017856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2db00c4030 0x7f2db0161930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:37:59.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.390+0000 7f2db4f2a640 1 --2- 172.21.15.57:0/2655017856 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2db006c420 0x7f2db00c4470 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:37:59.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.390+0000 7f2db4f2a640 1 -- 172.21.15.57:0/2655017856 >> 172.21.15.57:0/2655017856 conn(0x7f2db00cac70 msgr2=0x7f2db014fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:37:59.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.390+0000 7f2db4f2a640 1 -- 172.21.15.57:0/2655017856 shutdown_connections 2023-10-30T14:37:59.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:37:59.390+0000 7f2db4f2a640 1 -- 172.21.15.57:0/2655017856 wait complete. 2023-10-30T14:37:59.409 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:38:01.410 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:01.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.566+0000 7f4874fc5640 1 Processor -- start 2023-10-30T14:38:01.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.566+0000 7f4874fc5640 1 -- start start 2023-10-30T14:38:01.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.566+0000 7f4874fc5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4870153c80 0x7f487015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:01.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.566+0000 7f4874fc5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f48700c4a10 0x7f48700c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:01.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.566+0000 7f4874fc5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f487015a3b0 0x7f48700ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:01.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.566+0000 7f4874fc5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f487015dec0 con 0x7f48700c4a10 2023-10-30T14:38:01.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.566+0000 7f4874fc5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48701599a0 con 0x7f487015a3b0 2023-10-30T14:38:01.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.566+0000 7f4874fc5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4870159f20 con 0x7f4870153c80 2023-10-30T14:38:01.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.566+0000 7f4867fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f48700c4a10 0x7f48700c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:01.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.566+0000 7f4867fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f48700c4a10 0x7f48700c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46020/0 (socket says 172.21.15.57:46020) 2023-10-30T14:38:01.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.566+0000 7f4867fff640 1 -- 172.21.15.57:0/1432867182 learned_addr learned my addr 172.21.15.57:0/1432867182 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:01.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.566+0000 7f486e59c640 1 --2- 172.21.15.57:0/1432867182 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4870153c80 0x7f487015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:01.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.566+0000 7f486ed9d640 1 --2- 172.21.15.57:0/1432867182 >> v2:172.21.15.88:3300/0 conn(0x7f487015a3b0 0x7f48700ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:01.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.566+0000 7f4867fff640 1 -- 172.21.15.57:0/1432867182 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4870153c80 msgr2=0x7f487015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:01.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.566+0000 7f4867fff640 1 --2- 172.21.15.57:0/1432867182 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4870153c80 0x7f487015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:01.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.566+0000 7f4867fff640 1 -- 172.21.15.57:0/1432867182 >> v2:172.21.15.88:3300/0 conn(0x7f487015a3b0 msgr2=0x7f48700ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:01.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.566+0000 7f4867fff640 1 --2- 172.21.15.57:0/1432867182 >> v2:172.21.15.88:3300/0 conn(0x7f487015a3b0 0x7f48700ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:01.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.566+0000 7f4867fff640 1 -- 172.21.15.57:0/1432867182 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48700c47e0 con 0x7f48700c4a10 2023-10-30T14:38:01.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.570+0000 7f4867fff640 1 --2- 172.21.15.57:0/1432867182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48700c4a10 0x7f48700c42a0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f48700cbbc0 tx=0x7f485801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1971ee9aa832f59 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:01.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.570+0000 7f486dd9b640 1 -- 172.21.15.57:0/1432867182 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48580a89b0 con 0x7f48700c4a10 2023-10-30T14:38:01.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.570+0000 7f486dd9b640 1 -- 172.21.15.57:0/1432867182 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48580a8b10 con 0x7f48700c4a10 2023-10-30T14:38:01.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.570+0000 7f486dd9b640 1 -- 172.21.15.57:0/1432867182 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48580a8ea0 con 0x7f48700c4a10 2023-10-30T14:38:01.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.570+0000 7f4874fc5640 1 -- 172.21.15.57:0/1432867182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48700c4a10 msgr2=0x7f48700c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:01.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.570+0000 7f4874fc5640 1 --2- 172.21.15.57:0/1432867182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48700c4a10 0x7f48700c42a0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f48700cbbc0 tx=0x7f485801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:38:01.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.570+0000 7f4874fc5640 1 -- 172.21.15.57:0/1432867182 shutdown_connections 2023-10-30T14:38:01.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.570+0000 7f4874fc5640 1 --2- 172.21.15.57:0/1432867182 >> v2:172.21.15.88:3300/0 conn(0x7f487015a3b0 0x7f48700ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:01.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.570+0000 7f4874fc5640 1 --2- 172.21.15.57:0/1432867182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48700c4a10 0x7f48700c42a0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:01.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.570+0000 7f4874fc5640 1 --2- 172.21.15.57:0/1432867182 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4870153c80 0x7f487015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:01.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.570+0000 7f4874fc5640 1 -- 172.21.15.57:0/1432867182 >> 172.21.15.57:0/1432867182 conn(0x7f48700715c0 msgr2=0x7f48700719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:01.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.570+0000 7f4874fc5640 1 -- 172.21.15.57:0/1432867182 shutdown_connections 2023-10-30T14:38:01.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.570+0000 7f4874fc5640 1 -- 172.21.15.57:0/1432867182 wait complete. 2023-10-30T14:38:01.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f4874fc5640 1 Processor -- start 2023-10-30T14:38:01.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f4874fc5640 1 -- start start 2023-10-30T14:38:01.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f4874fc5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48700c4a10 0x7f4870151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:01.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f4874fc5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4870153c80 0x7f48700b3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:01.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f4874fc5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f487015a3b0 0x7f4870149b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:01.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f4874fc5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48700b2b70 con 0x7f48700c4a10 2023-10-30T14:38:01.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f4874fc5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48701eea50 con 0x7f4870153c80 2023-10-30T14:38:01.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f4874fc5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48700b2e30 con 0x7f487015a3b0 2023-10-30T14:38:01.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f486e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48700c4a10 0x7f4870151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:01.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f486e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48700c4a10 0x7f4870151fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46032/0 (socket says 172.21.15.57:46032) 2023-10-30T14:38:01.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f486e59c640 1 -- 172.21.15.57:0/1969997165 learned_addr learned my addr 172.21.15.57:0/1969997165 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:01.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f4867fff640 1 --2- 172.21.15.57:0/1969997165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4870153c80 0x7f48700b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:01.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f486ed9d640 1 --2- 172.21.15.57:0/1969997165 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f487015a3b0 0x7f4870149b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:01.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f486e59c640 1 -- 172.21.15.57:0/1969997165 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f487015a3b0 msgr2=0x7f4870149b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:01.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f486e59c640 1 --2- 172.21.15.57:0/1969997165 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f487015a3b0 0x7f4870149b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:01.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f486e59c640 1 -- 172.21.15.57:0/1969997165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4870153c80 msgr2=0x7f48700b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:01.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f486e59c640 1 --2- 172.21.15.57:0/1969997165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4870153c80 0x7f48700b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:01.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f486e59c640 1 -- 172.21.15.57:0/1969997165 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48580094d0 con 0x7f48700c4a10 2023-10-30T14:38:01.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f486e59c640 1 --2- 172.21.15.57:0/1969997165 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48700c4a10 0x7f4870151fd0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f486000d800 tx=0x7f486000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:01.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f4866ffd640 1 -- 172.21.15.57:0/1969997165 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48600096a0 con 0x7f48700c4a10 2023-10-30T14:38:01.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f4866ffd640 1 -- 172.21.15.57:0/1969997165 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4860009800 con 0x7f48700c4a10 2023-10-30T14:38:01.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f4866ffd640 1 -- 172.21.15.57:0/1969997165 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860009b70 con 0x7f48700c4a10 2023-10-30T14:38:01.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f4874fc5640 1 -- 172.21.15.57:0/1969997165 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48701eed10 con 0x7f48700c4a10 2023-10-30T14:38:01.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.574+0000 7f4874fc5640 1 -- 172.21.15.57:0/1969997165 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48701ef2d0 con 0x7f48700c4a10 2023-10-30T14:38:01.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.578+0000 7f4874fc5640 1 -- 172.21.15.57:0/1969997165 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f487004e170 con 0x7f48700c4a10 2023-10-30T14:38:01.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.578+0000 7f4866ffd640 1 -- 172.21.15.57:0/1969997165 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4860009cd0 con 0x7f48700c4a10 2023-10-30T14:38:01.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.578+0000 7f4866ffd640 1 --2- 172.21.15.57:0/1969997165 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f48500664d0 0x7f4850068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:01.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.578+0000 7f4866ffd640 1 -- 172.21.15.57:0/1969997165 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f48600849e0 con 0x7f48700c4a10 2023-10-30T14:38:01.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.578+0000 7f4867fff640 1 --2- 172.21.15.57:0/1969997165 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f48500664d0 0x7f4850068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:01.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.578+0000 7f4867fff640 1 --2- 172.21.15.57:0/1969997165 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f48500664d0 0x7f4850068990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f48580b2000 tx=0x7f4858009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:01.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.582+0000 7f4866ffd640 1 -- 172.21.15.57:0/1969997165 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4860051960 con 0x7f48700c4a10 2023-10-30T14:38:01.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.770+0000 7f4874fc5640 1 -- 172.21.15.57:0/1969997165 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48701594b0 con 0x7f48700c4a10 2023-10-30T14:38:01.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.774+0000 7f4866ffd640 1 -- 172.21.15.57:0/1969997165 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6535 (secure 0 0 0) 0x7f4860054f80 con 0x7f48700c4a10 2023-10-30T14:38:01.773 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:01.774 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6125,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:36:09.152736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:01.774 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 127 2023-10-30T14:38:01.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.774+0000 7f4874fc5640 1 -- 172.21.15.57:0/1969997165 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f48500664d0 msgr2=0x7f4850068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:01.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.774+0000 7f4874fc5640 1 --2- 172.21.15.57:0/1969997165 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f48500664d0 0x7f4850068990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f48580b2000 tx=0x7f4858009040 comp rx=0 tx=0).stop 2023-10-30T14:38:01.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.774+0000 7f4874fc5640 1 -- 172.21.15.57:0/1969997165 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48700c4a10 msgr2=0x7f4870151fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:01.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.774+0000 7f4874fc5640 1 --2- 172.21.15.57:0/1969997165 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48700c4a10 0x7f4870151fd0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f486000d800 tx=0x7f486000dcc0 comp rx=0 tx=0).stop 2023-10-30T14:38:01.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.774+0000 7f4874fc5640 1 -- 172.21.15.57:0/1969997165 shutdown_connections 2023-10-30T14:38:01.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.774+0000 7f4874fc5640 1 --2- 172.21.15.57:0/1969997165 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f48500664d0 0x7f4850068990 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:01.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.774+0000 7f4874fc5640 1 --2- 172.21.15.57:0/1969997165 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f487015a3b0 0x7f4870149b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:01.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.774+0000 7f4874fc5640 1 --2- 172.21.15.57:0/1969997165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4870153c80 0x7f48700b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:01.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.774+0000 7f4874fc5640 1 --2- 172.21.15.57:0/1969997165 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48700c4a10 0x7f4870151fd0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:01.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.774+0000 7f4874fc5640 1 -- 172.21.15.57:0/1969997165 >> 172.21.15.57:0/1969997165 conn(0x7f48700715c0 msgr2=0x7f4870158450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:01.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.774+0000 7f4874fc5640 1 -- 172.21.15.57:0/1969997165 shutdown_connections 2023-10-30T14:38:01.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:01.774+0000 7f4874fc5640 1 -- 172.21.15.57:0/1969997165 wait complete. 2023-10-30T14:38:01.795 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:38:03.796 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.950+0000 7f9e36c9c640 1 Processor -- start 2023-10-30T14:38:04.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.950+0000 7f9e36c9c640 1 -- start start 2023-10-30T14:38:04.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.950+0000 7f9e36c9c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e30151840 0x7f9e300c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:04.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.950+0000 7f9e36c9c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9e30151c20 0x7f9e30072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:04.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.950+0000 7f9e36c9c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e300c5030 con 0x7f9e300cb8b0 2023-10-30T14:38:04.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.950+0000 7f9e36c9c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e30157830 con 0x7f9e30151c20 2023-10-30T14:38:04.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.950+0000 7f9e36c9c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e30157b10 con 0x7f9e30151840 2023-10-30T14:38:04.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.950+0000 7f9e2ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9e30151c20 0x7f9e30072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:04.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.950+0000 7f9e2ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9e30151c20 0x7f9e30072000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40762/0 (socket says 172.21.15.57:40762) 2023-10-30T14:38:04.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.950+0000 7f9e2ffff640 1 -- 172.21.15.57:0/4019082754 learned_addr learned my addr 172.21.15.57:0/4019082754 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:04.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.950+0000 7f9e34a38640 1 --2- 172.21.15.57:0/4019082754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e30151840 0x7f9e300c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:04.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.950+0000 7f9e2ffff640 1 -- 172.21.15.57:0/4019082754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e30151840 msgr2=0x7f9e300c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:04.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.950+0000 7f9e2ffff640 1 --2- 172.21.15.57:0/4019082754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e30151840 0x7f9e300c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:04.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.950+0000 7f9e2ffff640 1 -- 172.21.15.57:0/4019082754 >> v1:172.21.15.57:6789/0 conn(0x7f9e300cb8b0 legacy=0x7f9e3006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:04.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.950+0000 7f9e2ffff640 1 -- 172.21.15.57:0/4019082754 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e30157df0 con 0x7f9e30151c20 2023-10-30T14:38:04.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.954+0000 7f9e2ffff640 1 --2- 172.21.15.57:0/4019082754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e30151c20 0x7f9e30072000 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c009870 tx=0x7f9e1c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12f1a598a1973caa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:04.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.954+0000 7f9e2f7fe640 1 -- 172.21.15.57:0/4019082754 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c0a8a30 con 0x7f9e30151c20 2023-10-30T14:38:04.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.954+0000 7f9e2f7fe640 1 -- 172.21.15.57:0/4019082754 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1c0a8b90 con 0x7f9e30151c20 2023-10-30T14:38:04.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.954+0000 7f9e2f7fe640 1 -- 172.21.15.57:0/4019082754 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c0b0440 con 0x7f9e30151c20 2023-10-30T14:38:04.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.954+0000 7f9e36c9c640 1 -- 172.21.15.57:0/4019082754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e30151c20 msgr2=0x7f9e30072000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:04.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.954+0000 7f9e36c9c640 1 --2- 172.21.15.57:0/4019082754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e30151c20 0x7f9e30072000 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c009870 tx=0x7f9e1c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:38:04.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.954+0000 7f9e36c9c640 1 -- 172.21.15.57:0/4019082754 shutdown_connections 2023-10-30T14:38:04.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.954+0000 7f9e36c9c640 1 --2- 172.21.15.57:0/4019082754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e30151c20 0x7f9e30072000 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:04.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.954+0000 7f9e36c9c640 1 --2- 172.21.15.57:0/4019082754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e30151840 0x7f9e300c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:04.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.954+0000 7f9e36c9c640 1 -- 172.21.15.57:0/4019082754 >> 172.21.15.57:0/4019082754 conn(0x7f9e300cac70 msgr2=0x7f9e300cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:04.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.954+0000 7f9e36c9c640 1 -- 172.21.15.57:0/4019082754 shutdown_connections 2023-10-30T14:38:04.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.954+0000 7f9e36c9c640 1 -- 172.21.15.57:0/4019082754 wait complete. 2023-10-30T14:38:04.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e36c9c640 1 Processor -- start 2023-10-30T14:38:04.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e36c9c640 1 -- start start 2023-10-30T14:38:04.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e36c9c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e300cb8b0 0x7f9e301ef500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:04.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e36c9c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e30151840 0x7f9e301471f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:04.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e36c9c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e30151c20 0x7f9e30145520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:04.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e36c9c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e301efa40 con 0x7f9e300cb8b0 2023-10-30T14:38:04.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e36c9c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e301462a0 con 0x7f9e30151840 2023-10-30T14:38:04.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e36c9c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e301e28d0 con 0x7f9e30151c20 2023-10-30T14:38:04.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e34a38640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e300cb8b0 0x7f9e301ef500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:04.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e34a38640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e300cb8b0 0x7f9e301ef500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40770/0 (socket says 172.21.15.57:40770) 2023-10-30T14:38:04.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e34a38640 1 -- 172.21.15.57:0/2619562605 learned_addr learned my addr 172.21.15.57:0/2619562605 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:04.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e35239640 1 --2- 172.21.15.57:0/2619562605 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e30151c20 0x7f9e30145520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:04.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e2ffff640 1 --2- 172.21.15.57:0/2619562605 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e30151840 0x7f9e301471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:04.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e34a38640 1 -- 172.21.15.57:0/2619562605 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e30151c20 msgr2=0x7f9e30145520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:04.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e34a38640 1 --2- 172.21.15.57:0/2619562605 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e30151c20 0x7f9e30145520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:04.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e34a38640 1 -- 172.21.15.57:0/2619562605 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e30151840 msgr2=0x7f9e301471f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:04.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e34a38640 1 --2- 172.21.15.57:0/2619562605 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e30151840 0x7f9e301471f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:04.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e34a38640 1 -- 172.21.15.57:0/2619562605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1c0094d0 con 0x7f9e300cb8b0 2023-10-30T14:38:04.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e34a38640 1 --2- 172.21.15.57:0/2619562605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e300cb8b0 0x7f9e301ef500 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400d800 tx=0x7f9e2400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:04.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e2d7fa640 1 -- 172.21.15.57:0/2619562605 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2400b660 con 0x7f9e300cb8b0 2023-10-30T14:38:04.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e2d7fa640 1 -- 172.21.15.57:0/2619562605 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e2400b7c0 con 0x7f9e300cb8b0 2023-10-30T14:38:04.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e2d7fa640 1 -- 172.21.15.57:0/2619562605 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2400bb30 con 0x7f9e300cb8b0 2023-10-30T14:38:04.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e36c9c640 1 -- 172.21.15.57:0/2619562605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e301ddf60 con 0x7f9e300cb8b0 2023-10-30T14:38:04.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.958+0000 7f9e36c9c640 1 -- 172.21.15.57:0/2619562605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e300ca7f0 con 0x7f9e300cb8b0 2023-10-30T14:38:04.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.962+0000 7f9e36c9c640 1 -- 172.21.15.57:0/2619562605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e300c74f0 con 0x7f9e300cb8b0 2023-10-30T14:38:04.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.962+0000 7f9e2d7fa640 1 -- 172.21.15.57:0/2619562605 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9e2400bc90 con 0x7f9e300cb8b0 2023-10-30T14:38:04.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.962+0000 7f9e2d7fa640 1 --2- 172.21.15.57:0/2619562605 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9e140664a0 0x7f9e14068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:04.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.962+0000 7f9e2d7fa640 1 -- 172.21.15.57:0/2619562605 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f9e24084990 con 0x7f9e300cb8b0 2023-10-30T14:38:04.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.962+0000 7f9e2ffff640 1 --2- 172.21.15.57:0/2619562605 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9e140664a0 0x7f9e14068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:04.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.962+0000 7f9e2ffff640 1 --2- 172.21.15.57:0/2619562605 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9e140664a0 0x7f9e14068960 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0b2000 tx=0x7f9e1c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:04.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:03.966+0000 7f9e2d7fa640 1 -- 172.21.15.57:0/2619562605 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9e24051860 con 0x7f9e300cb8b0 2023-10-30T14:38:04.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:04.170+0000 7f9e36c9c640 1 -- 172.21.15.57:0/2619562605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e301de150 con 0x7f9e300cb8b0 2023-10-30T14:38:04.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:04.170+0000 7f9e2d7fa640 1 -- 172.21.15.57:0/2619562605 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6535 (secure 0 0 0) 0x7f9e24011340 con 0x7f9e300cb8b0 2023-10-30T14:38:04.172 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:04.172 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6125,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:36:09.152736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:04.172 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 127 2023-10-30T14:38:04.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:04.174+0000 7f9e36c9c640 1 -- 172.21.15.57:0/2619562605 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9e140664a0 msgr2=0x7f9e14068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:04.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:04.174+0000 7f9e36c9c640 1 --2- 172.21.15.57:0/2619562605 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9e140664a0 0x7f9e14068960 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0b2000 tx=0x7f9e1c008040 comp rx=0 tx=0).stop 2023-10-30T14:38:04.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:04.174+0000 7f9e36c9c640 1 -- 172.21.15.57:0/2619562605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e300cb8b0 msgr2=0x7f9e301ef500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:04.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:04.174+0000 7f9e36c9c640 1 --2- 172.21.15.57:0/2619562605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e300cb8b0 0x7f9e301ef500 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400d800 tx=0x7f9e2400dcc0 comp rx=0 tx=0).stop 2023-10-30T14:38:04.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:04.174+0000 7f9e36c9c640 1 -- 172.21.15.57:0/2619562605 shutdown_connections 2023-10-30T14:38:04.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:04.174+0000 7f9e36c9c640 1 --2- 172.21.15.57:0/2619562605 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9e140664a0 0x7f9e14068960 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:04.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:04.174+0000 7f9e36c9c640 1 --2- 172.21.15.57:0/2619562605 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e30151c20 0x7f9e30145520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:04.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:04.174+0000 7f9e36c9c640 1 --2- 172.21.15.57:0/2619562605 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e30151840 0x7f9e301471f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:04.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:04.174+0000 7f9e36c9c640 1 --2- 172.21.15.57:0/2619562605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e300cb8b0 0x7f9e301ef500 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:04.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:04.174+0000 7f9e36c9c640 1 -- 172.21.15.57:0/2619562605 >> 172.21.15.57:0/2619562605 conn(0x7f9e300cac70 msgr2=0x7f9e300cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:04.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:04.174+0000 7f9e36c9c640 1 -- 172.21.15.57:0/2619562605 shutdown_connections 2023-10-30T14:38:04.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:04.174+0000 7f9e36c9c640 1 -- 172.21.15.57:0/2619562605 wait complete. 2023-10-30T14:38:04.194 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:38:06.195 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:06.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.350+0000 7f8f88205640 1 Processor -- start 2023-10-30T14:38:06.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.350+0000 7f8f88205640 1 -- start start 2023-10-30T14:38:06.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.354+0000 7f8f88205640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8f80153bb0 0x7f8f800c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:06.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.354+0000 7f8f88205640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8f8006c420 0x7f8f800c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:06.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.354+0000 7f8f88205640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f800c7170 0x7f8f80159980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:06.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.354+0000 7f8f88205640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f800c5f80 con 0x7f8f8006c420 2023-10-30T14:38:06.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.354+0000 7f8f88205640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f800c5690 con 0x7f8f80153bb0 2023-10-30T14:38:06.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.354+0000 7f8f88205640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8015a580 con 0x7f8f800c7170 2023-10-30T14:38:06.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.354+0000 7f8f857a0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8f8006c420 0x7f8f800c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:06.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.354+0000 7f8f857a0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8f8006c420 0x7f8f800c6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40776/0 (socket says 172.21.15.57:40776) 2023-10-30T14:38:06.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.354+0000 7f8f857a0640 1 -- 172.21.15.57:0/4034575366 learned_addr learned my addr 172.21.15.57:0/4034575366 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:06.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.354+0000 7f8f85fa1640 1 --2- 172.21.15.57:0/4034575366 >> v2:172.21.15.88:3300/0 conn(0x7f8f80153bb0 0x7f8f800c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:06.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.354+0000 7f8f867a2640 1 --2- 172.21.15.57:0/4034575366 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f800c7170 0x7f8f80159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:06.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.354+0000 7f8f857a0640 1 -- 172.21.15.57:0/4034575366 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f800c7170 msgr2=0x7f8f80159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:06.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.354+0000 7f8f857a0640 1 --2- 172.21.15.57:0/4034575366 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f800c7170 0x7f8f80159980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:06.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.354+0000 7f8f857a0640 1 -- 172.21.15.57:0/4034575366 >> v2:172.21.15.88:3300/0 conn(0x7f8f80153bb0 msgr2=0x7f8f800c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:06.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.354+0000 7f8f857a0640 1 --2- 172.21.15.57:0/4034575366 >> v2:172.21.15.88:3300/0 conn(0x7f8f80153bb0 0x7f8f800c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:06.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.354+0000 7f8f857a0640 1 -- 172.21.15.57:0/4034575366 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f800c5930 con 0x7f8f8006c420 2023-10-30T14:38:06.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.358+0000 7f8f857a0640 1 --2- 172.21.15.57:0/4034575366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f8006c420 0x7f8f800c6c30 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f8f800cb840 tx=0x7f8f7001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff9cba8716e776f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:06.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.358+0000 7f8f84f9f640 1 -- 172.21.15.57:0/4034575366 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f700a89b0 con 0x7f8f8006c420 2023-10-30T14:38:06.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.358+0000 7f8f84f9f640 1 -- 172.21.15.57:0/4034575366 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f700a8b10 con 0x7f8f8006c420 2023-10-30T14:38:06.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.358+0000 7f8f84f9f640 1 -- 172.21.15.57:0/4034575366 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f700a8ea0 con 0x7f8f8006c420 2023-10-30T14:38:06.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.358+0000 7f8f88205640 1 -- 172.21.15.57:0/4034575366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f8006c420 msgr2=0x7f8f800c6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:06.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.358+0000 7f8f88205640 1 --2- 172.21.15.57:0/4034575366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f8006c420 0x7f8f800c6c30 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f8f800cb840 tx=0x7f8f7001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:38:06.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.358+0000 7f8f88205640 1 -- 172.21.15.57:0/4034575366 shutdown_connections 2023-10-30T14:38:06.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.358+0000 7f8f88205640 1 --2- 172.21.15.57:0/4034575366 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f800c7170 0x7f8f80159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:06.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.358+0000 7f8f88205640 1 --2- 172.21.15.57:0/4034575366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f8006c420 0x7f8f800c6c30 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:06.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.358+0000 7f8f88205640 1 --2- 172.21.15.57:0/4034575366 >> v2:172.21.15.88:3300/0 conn(0x7f8f80153bb0 0x7f8f800c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:06.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.358+0000 7f8f88205640 1 -- 172.21.15.57:0/4034575366 >> 172.21.15.57:0/4034575366 conn(0x7f8f800cac70 msgr2=0x7f8f800cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:06.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.358+0000 7f8f88205640 1 -- 172.21.15.57:0/4034575366 shutdown_connections 2023-10-30T14:38:06.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.358+0000 7f8f88205640 1 -- 172.21.15.57:0/4034575366 wait complete. 2023-10-30T14:38:06.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.358+0000 7f8f88205640 1 Processor -- start 2023-10-30T14:38:06.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.358+0000 7f8f88205640 1 -- start start 2023-10-30T14:38:06.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f88205640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f8006c420 0x7f8f800c3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:06.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f88205640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f800c7170 0x7f8f80156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:06.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f88205640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f80153bb0 0x7f8f80070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:06.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f88205640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f801ef700 con 0x7f8f800c7170 2023-10-30T14:38:06.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f88205640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f800c2770 con 0x7f8f8006c420 2023-10-30T14:38:06.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f88205640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f800c44e0 con 0x7f8f80153bb0 2023-10-30T14:38:06.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f857a0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f800c7170 0x7f8f80156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:06.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f857a0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f800c7170 0x7f8f80156e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40780/0 (socket says 172.21.15.57:40780) 2023-10-30T14:38:06.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f857a0640 1 -- 172.21.15.57:0/171279305 learned_addr learned my addr 172.21.15.57:0/171279305 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:06.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f85fa1640 1 --2- 172.21.15.57:0/171279305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f8006c420 0x7f8f800c3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:06.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f867a2640 1 --2- 172.21.15.57:0/171279305 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f80153bb0 0x7f8f80070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:06.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f857a0640 1 -- 172.21.15.57:0/171279305 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f80153bb0 msgr2=0x7f8f80070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:06.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f857a0640 1 --2- 172.21.15.57:0/171279305 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f80153bb0 0x7f8f80070bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:06.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f857a0640 1 -- 172.21.15.57:0/171279305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f8006c420 msgr2=0x7f8f800c3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:06.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f857a0640 1 --2- 172.21.15.57:0/171279305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f8006c420 0x7f8f800c3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:06.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f857a0640 1 -- 172.21.15.57:0/171279305 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f700094d0 con 0x7f8f800c7170 2023-10-30T14:38:06.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f857a0640 1 --2- 172.21.15.57:0/171279305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f800c7170 0x7f8f80156e90 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f8f7001d3a0 tx=0x7f8f7001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:06.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f767fc640 1 -- 172.21.15.57:0/171279305 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f700ba600 con 0x7f8f800c7170 2023-10-30T14:38:06.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f767fc640 1 -- 172.21.15.57:0/171279305 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f700ba760 con 0x7f8f800c7170 2023-10-30T14:38:06.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f767fc640 1 -- 172.21.15.57:0/171279305 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f700bab10 con 0x7f8f800c7170 2023-10-30T14:38:06.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f88205640 1 -- 172.21.15.57:0/171279305 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f801ef180 con 0x7f8f800c7170 2023-10-30T14:38:06.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f88205640 1 -- 172.21.15.57:0/171279305 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f800c4b40 con 0x7f8f800c7170 2023-10-30T14:38:06.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f88205640 1 -- 172.21.15.57:0/171279305 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f800c2920 con 0x7f8f800c7170 2023-10-30T14:38:06.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.362+0000 7f8f767fc640 1 -- 172.21.15.57:0/171279305 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8f700bac70 con 0x7f8f800c7170 2023-10-30T14:38:06.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.366+0000 7f8f767fc640 1 --2- 172.21.15.57:0/171279305 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f640664d0 0x7f8f64068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:06.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.366+0000 7f8f767fc640 1 -- 172.21.15.57:0/171279305 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f8f70122950 con 0x7f8f800c7170 2023-10-30T14:38:06.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.366+0000 7f8f85fa1640 1 --2- 172.21.15.57:0/171279305 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f640664d0 0x7f8f64068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:06.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.366+0000 7f8f85fa1640 1 --2- 172.21.15.57:0/171279305 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f640664d0 0x7f8f64068990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c00b9d0 tx=0x7f8f7c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:06.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.370+0000 7f8f767fc640 1 -- 172.21.15.57:0/171279305 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8f800c2920 con 0x7f8f800c7170 2023-10-30T14:38:06.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.558+0000 7f8f88205640 1 -- 172.21.15.57:0/171279305 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f800c2920 con 0x7f8f800c7170 2023-10-30T14:38:06.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.558+0000 7f8f767fc640 1 -- 172.21.15.57:0/171279305 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6535 (secure 0 0 0) 0x7f8f800c2920 con 0x7f8f800c7170 2023-10-30T14:38:06.560 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:06.561 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6125,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:36:09.152736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:06.562 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 127 2023-10-30T14:38:06.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.562+0000 7f8f88205640 1 -- 172.21.15.57:0/171279305 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f640664d0 msgr2=0x7f8f64068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:06.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.562+0000 7f8f88205640 1 --2- 172.21.15.57:0/171279305 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f640664d0 0x7f8f64068990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c00b9d0 tx=0x7f8f7c008040 comp rx=0 tx=0).stop 2023-10-30T14:38:06.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.562+0000 7f8f88205640 1 -- 172.21.15.57:0/171279305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f800c7170 msgr2=0x7f8f80156e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:06.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.562+0000 7f8f88205640 1 --2- 172.21.15.57:0/171279305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f800c7170 0x7f8f80156e90 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f8f7001d3a0 tx=0x7f8f7001ba90 comp rx=0 tx=0).stop 2023-10-30T14:38:06.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.562+0000 7f8f88205640 1 -- 172.21.15.57:0/171279305 shutdown_connections 2023-10-30T14:38:06.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.562+0000 7f8f88205640 1 --2- 172.21.15.57:0/171279305 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f640664d0 0x7f8f64068990 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:06.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.562+0000 7f8f88205640 1 --2- 172.21.15.57:0/171279305 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f80153bb0 0x7f8f80070bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:06.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.562+0000 7f8f88205640 1 --2- 172.21.15.57:0/171279305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f800c7170 0x7f8f80156e90 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:06.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.562+0000 7f8f88205640 1 --2- 172.21.15.57:0/171279305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f8006c420 0x7f8f800c3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:06.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.562+0000 7f8f88205640 1 -- 172.21.15.57:0/171279305 >> 172.21.15.57:0/171279305 conn(0x7f8f800cac70 msgr2=0x7f8f801513c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:06.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.562+0000 7f8f88205640 1 -- 172.21.15.57:0/171279305 shutdown_connections 2023-10-30T14:38:06.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:06.562+0000 7f8f88205640 1 -- 172.21.15.57:0/171279305 wait complete. 2023-10-30T14:38:06.581 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:38:07.611 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:38:07.615 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:38:08.582 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:08.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.738+0000 7f09d5565640 1 Processor -- start 2023-10-30T14:38:08.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.738+0000 7f09d5565640 1 -- start start 2023-10-30T14:38:08.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.738+0000 7f09d5565640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09d0157c90 0x7f09d0158a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:08.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.738+0000 7f09d5565640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d0158110 con 0x7f09d00724c0 2023-10-30T14:38:08.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.738+0000 7f09d5565640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d01583f0 con 0x7f09d00720e0 2023-10-30T14:38:08.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.738+0000 7f09d5565640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d00c1f80 con 0x7f09d0157c90 2023-10-30T14:38:08.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.738+0000 7f09cf7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f09d00724c0 0x7f09d0150f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:47840/0 (socket says 172.21.15.57:47840) 2023-10-30T14:38:08.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.738+0000 7f09cf7fe640 1 -- 172.21.15.57:0/2277688042 learned_addr learned my addr 172.21.15.57:0/2277688042 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:38:08.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.738+0000 7f09ce7fc640 1 --2- 172.21.15.57:0/2277688042 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09d0157c90 0x7f09d0158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:08.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.738+0000 7f09cdffb640 1 -- 172.21.15.57:0/2277688042 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1577675935 0 0) 0x7f09d0158110 con 0x7f09d00724c0 2023-10-30T14:38:08.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.738+0000 7f09cdffb640 1 -- 172.21.15.57:0/2277688042 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b8003530 con 0x7f09d00724c0 2023-10-30T14:38:08.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.738+0000 7f09cdffb640 1 -- 172.21.15.57:0/2277688042 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173617718 0 0) 0x7f09d01583f0 con 0x7f09d00720e0 2023-10-30T14:38:08.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.738+0000 7f09cdffb640 1 -- 172.21.15.57:0/2277688042 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09d0158110 con 0x7f09d00720e0 2023-10-30T14:38:08.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.738+0000 7f09ce7fc640 1 -- 172.21.15.57:0/2277688042 >> v1:172.21.15.88:6789/0 conn(0x7f09d00720e0 legacy=0x7f09d0157580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:08.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.738+0000 7f09ce7fc640 1 -- 172.21.15.57:0/2277688042 >> v1:172.21.15.57:6789/0 conn(0x7f09d00724c0 legacy=0x7f09d0150f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:08.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.738+0000 7f09cdffb640 1 -- 172.21.15.57:0/2277688042 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1786569486 0 0) 0x7f09b8003530 con 0x7f09d00724c0 2023-10-30T14:38:08.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.738+0000 7f09ce7fc640 1 -- 172.21.15.57:0/2277688042 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d00c2260 con 0x7f09d0157c90 2023-10-30T14:38:08.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.742+0000 7f09ce7fc640 1 --2- 172.21.15.57:0/2277688042 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09d0157c90 0x7f09d0158a70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f09c40097f0 tx=0x7f09c401ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2af33842e052ad0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:08.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.742+0000 7f09cdffb640 1 -- 172.21.15.57:0/2277688042 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c8001a00 con 0x7f09d0157c90 2023-10-30T14:38:08.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.742+0000 7f09cdffb640 1 -- 172.21.15.57:0/2277688042 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c40a8650 con 0x7f09d0157c90 2023-10-30T14:38:08.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.742+0000 7f09cdffb640 1 -- 172.21.15.57:0/2277688042 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c40a89e0 con 0x7f09d0157c90 2023-10-30T14:38:08.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.742+0000 7f09d5565640 1 -- 172.21.15.57:0/2277688042 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09d0157c90 msgr2=0x7f09d0158a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:08.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.742+0000 7f09d5565640 1 --2- 172.21.15.57:0/2277688042 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09d0157c90 0x7f09d0158a70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f09c40097f0 tx=0x7f09c401ebd0 comp rx=0 tx=0).stop 2023-10-30T14:38:08.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.742+0000 7f09d5565640 1 -- 172.21.15.57:0/2277688042 shutdown_connections 2023-10-30T14:38:08.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.742+0000 7f09d5565640 1 --2- 172.21.15.57:0/2277688042 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09d0157c90 0x7f09d0158a70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:08.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.742+0000 7f09d5565640 1 -- 172.21.15.57:0/2277688042 >> 172.21.15.57:0/2277688042 conn(0x7f09d006d810 msgr2=0x7f09d006dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:08.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.742+0000 7f09d5565640 1 -- 172.21.15.57:0/2277688042 shutdown_connections 2023-10-30T14:38:08.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.742+0000 7f09d5565640 1 -- 172.21.15.57:0/2277688042 wait complete. 2023-10-30T14:38:08.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09d5565640 1 Processor -- start 2023-10-30T14:38:08.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09d5565640 1 -- start start 2023-10-30T14:38:08.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09d5565640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09d00720e0 0x7f09d014fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:08.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09d5565640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09d00724c0 0x7f09d006fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:08.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09d5565640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09d0157c90 0x7f09d00701b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:08.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09d5565640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d0156a00 con 0x7f09d00720e0 2023-10-30T14:38:08.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09d5565640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d0156f80 con 0x7f09d00724c0 2023-10-30T14:38:08.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09d5565640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d0156ca0 con 0x7f09d0157c90 2023-10-30T14:38:08.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09ceffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09d00720e0 0x7f09d014fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:08.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09ceffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09d00720e0 0x7f09d014fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40784/0 (socket says 172.21.15.57:40784) 2023-10-30T14:38:08.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09ceffd640 1 -- 172.21.15.57:0/1124263171 learned_addr learned my addr 172.21.15.57:0/1124263171 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:08.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09ce7fc640 1 --2- 172.21.15.57:0/1124263171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09d00724c0 0x7f09d006fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:08.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09cf7fe640 1 --2- 172.21.15.57:0/1124263171 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09d0157c90 0x7f09d00701b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:08.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09ceffd640 1 -- 172.21.15.57:0/1124263171 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09d0157c90 msgr2=0x7f09d00701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:08.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09ceffd640 1 --2- 172.21.15.57:0/1124263171 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09d0157c90 0x7f09d00701b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:08.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09ceffd640 1 -- 172.21.15.57:0/1124263171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09d00724c0 msgr2=0x7f09d006fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:08.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09ceffd640 1 --2- 172.21.15.57:0/1124263171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09d00724c0 0x7f09d006fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:08.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09ceffd640 1 -- 172.21.15.57:0/1124263171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c4009420 con 0x7f09d00720e0 2023-10-30T14:38:08.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09ceffd640 1 --2- 172.21.15.57:0/1124263171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09d00720e0 0x7f09d014fa20 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f09c000c490 tx=0x7f09c000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:08.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09b77fe640 1 -- 172.21.15.57:0/1124263171 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c00052d0 con 0x7f09d00720e0 2023-10-30T14:38:08.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09b77fe640 1 -- 172.21.15.57:0/1124263171 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c0005430 con 0x7f09d00720e0 2023-10-30T14:38:08.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09b77fe640 1 -- 172.21.15.57:0/1124263171 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0012870 con 0x7f09d00720e0 2023-10-30T14:38:08.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09d5565640 1 -- 172.21.15.57:0/1124263171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d0071580 con 0x7f09d00720e0 2023-10-30T14:38:08.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.746+0000 7f09d5565640 1 -- 172.21.15.57:0/1124263171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d01508d0 con 0x7f09d00720e0 2023-10-30T14:38:08.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.750+0000 7f09d5565640 1 -- 172.21.15.57:0/1124263171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d0071aa0 con 0x7f09d00720e0 2023-10-30T14:38:08.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.750+0000 7f09b77fe640 1 -- 172.21.15.57:0/1124263171 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f09c00129d0 con 0x7f09d00720e0 2023-10-30T14:38:08.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.750+0000 7f09b77fe640 1 --2- 172.21.15.57:0/1124263171 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09bc0664d0 0x7f09bc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:08.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.750+0000 7f09b77fe640 1 -- 172.21.15.57:0/1124263171 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f09c0083e80 con 0x7f09d00720e0 2023-10-30T14:38:08.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.750+0000 7f09ce7fc640 1 --2- 172.21.15.57:0/1124263171 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09bc0664d0 0x7f09bc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:08.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.750+0000 7f09ce7fc640 1 --2- 172.21.15.57:0/1124263171 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09bc0664d0 0x7f09bc068990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f09c40b2000 tx=0x7f09c4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:08.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.754+0000 7f09b77fe640 1 -- 172.21.15.57:0/1124263171 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f09d0071aa0 con 0x7f09d00720e0 2023-10-30T14:38:08.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.946+0000 7f09d5565640 1 -- 172.21.15.57:0/1124263171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09d004e170 con 0x7f09d00720e0 2023-10-30T14:38:08.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.946+0000 7f09b77fe640 1 -- 172.21.15.57:0/1124263171 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6535 (secure 0 0 0) 0x7f09c0054880 con 0x7f09d00720e0 2023-10-30T14:38:08.946 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:08.946 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6125,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:36:09.152736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:08.946 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 127 2023-10-30T14:38:08.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.946+0000 7f09d5565640 1 -- 172.21.15.57:0/1124263171 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09bc0664d0 msgr2=0x7f09bc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:08.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.946+0000 7f09d5565640 1 --2- 172.21.15.57:0/1124263171 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09bc0664d0 0x7f09bc068990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f09c40b2000 tx=0x7f09c4008040 comp rx=0 tx=0).stop 2023-10-30T14:38:08.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.946+0000 7f09d5565640 1 -- 172.21.15.57:0/1124263171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09d00720e0 msgr2=0x7f09d014fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:08.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.946+0000 7f09d5565640 1 --2- 172.21.15.57:0/1124263171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09d00720e0 0x7f09d014fa20 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f09c000c490 tx=0x7f09c000c950 comp rx=0 tx=0).stop 2023-10-30T14:38:08.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.946+0000 7f09d5565640 1 -- 172.21.15.57:0/1124263171 shutdown_connections 2023-10-30T14:38:08.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.946+0000 7f09d5565640 1 --2- 172.21.15.57:0/1124263171 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f09bc0664d0 0x7f09bc068990 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:08.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.946+0000 7f09d5565640 1 --2- 172.21.15.57:0/1124263171 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f09d0157c90 0x7f09d00701b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:08.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.946+0000 7f09d5565640 1 --2- 172.21.15.57:0/1124263171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09d00724c0 0x7f09d006fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:08.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.946+0000 7f09d5565640 1 --2- 172.21.15.57:0/1124263171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09d00720e0 0x7f09d014fa20 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:08.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.946+0000 7f09d5565640 1 -- 172.21.15.57:0/1124263171 >> 172.21.15.57:0/1124263171 conn(0x7f09d006d810 msgr2=0x7f09d00c9700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:08.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.946+0000 7f09d5565640 1 -- 172.21.15.57:0/1124263171 shutdown_connections 2023-10-30T14:38:08.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:08.946+0000 7f09d5565640 1 -- 172.21.15.57:0/1124263171 wait complete. 2023-10-30T14:38:08.968 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:38:10.969 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:11.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.121+0000 7fe0279a1640 1 Processor -- start 2023-10-30T14:38:11.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.121+0000 7fe0279a1640 1 -- start start 2023-10-30T14:38:11.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.125+0000 7fe0279a1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe0200ccf60 0x7fe0200cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:11.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.125+0000 7fe0279a1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe0200c2500 0x7fe0200c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:11.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.125+0000 7fe0279a1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe020151aa0 con 0x7fe0201516c0 2023-10-30T14:38:11.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.125+0000 7fe0279a1640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0200c74a0 con 0x7fe0200ccf60 2023-10-30T14:38:11.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.125+0000 7fe0279a1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0200c2e70 con 0x7fe0200c2500 2023-10-30T14:38:11.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.125+0000 7fe024f3c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe0200ccf60 0x7fe0200cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:11.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.125+0000 7fe024f3c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe0200ccf60 0x7fe0200cd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40794/0 (socket says 172.21.15.57:40794) 2023-10-30T14:38:11.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.125+0000 7fe024f3c640 1 -- 172.21.15.57:0/3271633087 learned_addr learned my addr 172.21.15.57:0/3271633087 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:11.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.125+0000 7fe025f3e640 1 --2- 172.21.15.57:0/3271633087 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe0200c2500 0x7fe0200c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:11.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.125+0000 7fe024f3c640 1 -- 172.21.15.57:0/3271633087 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe0200c2500 msgr2=0x7fe0200c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:11.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.125+0000 7fe024f3c640 1 --2- 172.21.15.57:0/3271633087 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe0200c2500 0x7fe0200c2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:11.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.125+0000 7fe024f3c640 1 -- 172.21.15.57:0/3271633087 >> v1:172.21.15.88:6789/0 conn(0x7fe0201516c0 legacy=0x7fe0200c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:11.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.125+0000 7fe024f3c640 1 -- 172.21.15.57:0/3271633087 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe020158af0 con 0x7fe0200ccf60 2023-10-30T14:38:11.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe024f3c640 1 --2- 172.21.15.57:0/3271633087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0200ccf60 0x7fe0200cd3a0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fe010009870 tx=0x7fe01001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b5ef2973fabf10 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:11.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe017fff640 1 -- 172.21.15.57:0/3271633087 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0100ab600 con 0x7fe0200ccf60 2023-10-30T14:38:11.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe017fff640 1 -- 172.21.15.57:0/3271633087 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0100a7050 con 0x7fe0200ccf60 2023-10-30T14:38:11.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe017fff640 1 -- 172.21.15.57:0/3271633087 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0100ab980 con 0x7fe0200ccf60 2023-10-30T14:38:11.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe0279a1640 1 -- 172.21.15.57:0/3271633087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0200ccf60 msgr2=0x7fe0200cd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:11.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe0279a1640 1 --2- 172.21.15.57:0/3271633087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0200ccf60 0x7fe0200cd3a0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fe010009870 tx=0x7fe01001c540 comp rx=0 tx=0).stop 2023-10-30T14:38:11.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe0279a1640 1 -- 172.21.15.57:0/3271633087 shutdown_connections 2023-10-30T14:38:11.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe0279a1640 1 --2- 172.21.15.57:0/3271633087 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe0200c2500 0x7fe0200c2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:11.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe0279a1640 1 --2- 172.21.15.57:0/3271633087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0200ccf60 0x7fe0200cd3a0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:11.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe0279a1640 1 -- 172.21.15.57:0/3271633087 >> 172.21.15.57:0/3271633087 conn(0x7fe020071fb0 msgr2=0x7fe0200723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:11.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe0279a1640 1 -- 172.21.15.57:0/3271633087 shutdown_connections 2023-10-30T14:38:11.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe0279a1640 1 -- 172.21.15.57:0/3271633087 wait complete. 2023-10-30T14:38:11.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe0279a1640 1 Processor -- start 2023-10-30T14:38:11.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe0279a1640 1 -- start start 2023-10-30T14:38:11.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe0279a1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0200c2500 0x7fe0200cb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:11.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe0279a1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe0200ccf60 0x7fe02014b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:11.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe0279a1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0201516c0 0x7fe02014be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:11.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe0279a1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0200c4ee0 con 0x7fe0200c2500 2023-10-30T14:38:11.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe0279a1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0201f2d80 con 0x7fe0201516c0 2023-10-30T14:38:11.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.129+0000 7fe0279a1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0200c4640 con 0x7fe0200ccf60 2023-10-30T14:38:11.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe02573d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0200c2500 0x7fe0200cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:11.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe02573d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0200c2500 0x7fe0200cb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40800/0 (socket says 172.21.15.57:40800) 2023-10-30T14:38:11.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe02573d640 1 -- 172.21.15.57:0/3831421974 learned_addr learned my addr 172.21.15.57:0/3831421974 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:11.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe024f3c640 1 --2- 172.21.15.57:0/3831421974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe0200ccf60 0x7fe02014b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:11.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe025f3e640 1 --2- 172.21.15.57:0/3831421974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0201516c0 0x7fe02014be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:11.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe02573d640 1 -- 172.21.15.57:0/3831421974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe0200ccf60 msgr2=0x7fe02014b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:11.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe02573d640 1 --2- 172.21.15.57:0/3831421974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe0200ccf60 0x7fe02014b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:11.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe02573d640 1 -- 172.21.15.57:0/3831421974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0201516c0 msgr2=0x7fe02014be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:11.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe02573d640 1 --2- 172.21.15.57:0/3831421974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0201516c0 0x7fe02014be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:11.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe02573d640 1 -- 172.21.15.57:0/3831421974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0100094d0 con 0x7fe0200c2500 2023-10-30T14:38:11.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe024f3c640 1 --2- 172.21.15.57:0/3831421974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe0200ccf60 0x7fe02014b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:38:11.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe02573d640 1 --2- 172.21.15.57:0/3831421974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0200c2500 0x7fe0200cb010 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fe01c00c500 tx=0x7fe01c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:11.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe015ffb640 1 -- 172.21.15.57:0/3831421974 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c005220 con 0x7fe0200c2500 2023-10-30T14:38:11.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe015ffb640 1 -- 172.21.15.57:0/3831421974 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01c005380 con 0x7fe0200c2500 2023-10-30T14:38:11.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe015ffb640 1 -- 172.21.15.57:0/3831421974 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c007270 con 0x7fe0200c2500 2023-10-30T14:38:11.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe0279a1640 1 -- 172.21.15.57:0/3831421974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0200c4920 con 0x7fe0200c2500 2023-10-30T14:38:11.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe0279a1640 1 -- 172.21.15.57:0/3831421974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02006c070 con 0x7fe0200c2500 2023-10-30T14:38:11.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe0279a1640 1 -- 172.21.15.57:0/3831421974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe02006c290 con 0x7fe0200c2500 2023-10-30T14:38:11.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe015ffb640 1 -- 172.21.15.57:0/3831421974 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe01c0073d0 con 0x7fe0200c2500 2023-10-30T14:38:11.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.133+0000 7fe015ffb640 1 --2- 172.21.15.57:0/3831421974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe0040664a0 0x7fe004068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:11.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.137+0000 7fe015ffb640 1 -- 172.21.15.57:0/3831421974 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fe01c083c80 con 0x7fe0200c2500 2023-10-30T14:38:11.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.137+0000 7fe024f3c640 1 --2- 172.21.15.57:0/3831421974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe0040664a0 0x7fe004068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:11.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.137+0000 7fe024f3c640 1 --2- 172.21.15.57:0/3831421974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe0040664a0 0x7fe004068960 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe0200cdb70 tx=0x7fe010008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:11.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.141+0000 7fe015ffb640 1 -- 172.21.15.57:0/3831421974 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe01c050bf0 con 0x7fe0200c2500 2023-10-30T14:38:11.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.329+0000 7fe0279a1640 1 -- 172.21.15.57:0/3831421974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe02004e170 con 0x7fe0200c2500 2023-10-30T14:38:11.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.329+0000 7fe015ffb640 1 -- 172.21.15.57:0/3831421974 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6535 (secure 0 0 0) 0x7fe01c054210 con 0x7fe0200c2500 2023-10-30T14:38:11.332 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:11.332 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6125,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:36:09.152736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:11.333 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 127 2023-10-30T14:38:11.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.333+0000 7fe0279a1640 1 -- 172.21.15.57:0/3831421974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe0040664a0 msgr2=0x7fe004068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:11.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.333+0000 7fe0279a1640 1 --2- 172.21.15.57:0/3831421974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe0040664a0 0x7fe004068960 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe0200cdb70 tx=0x7fe010008040 comp rx=0 tx=0).stop 2023-10-30T14:38:11.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.333+0000 7fe0279a1640 1 -- 172.21.15.57:0/3831421974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0200c2500 msgr2=0x7fe0200cb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:11.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.333+0000 7fe0279a1640 1 --2- 172.21.15.57:0/3831421974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0200c2500 0x7fe0200cb010 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fe01c00c500 tx=0x7fe01c00c9c0 comp rx=0 tx=0).stop 2023-10-30T14:38:11.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.333+0000 7fe0279a1640 1 -- 172.21.15.57:0/3831421974 shutdown_connections 2023-10-30T14:38:11.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.333+0000 7fe0279a1640 1 --2- 172.21.15.57:0/3831421974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe0040664a0 0x7fe004068960 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:11.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.333+0000 7fe0279a1640 1 --2- 172.21.15.57:0/3831421974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0201516c0 0x7fe02014be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:11.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.333+0000 7fe0279a1640 1 --2- 172.21.15.57:0/3831421974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe0200ccf60 0x7fe02014b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:11.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.333+0000 7fe0279a1640 1 --2- 172.21.15.57:0/3831421974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0200c2500 0x7fe0200cb010 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:11.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.333+0000 7fe0279a1640 1 -- 172.21.15.57:0/3831421974 >> 172.21.15.57:0/3831421974 conn(0x7fe020071fb0 msgr2=0x7fe020155e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:11.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.333+0000 7fe0279a1640 1 -- 172.21.15.57:0/3831421974 shutdown_connections 2023-10-30T14:38:11.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:11.333+0000 7fe0279a1640 1 -- 172.21.15.57:0/3831421974 wait complete. 2023-10-30T14:38:11.353 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:38:13.354 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:13.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.505+0000 7f5b8cb72640 1 Processor -- start 2023-10-30T14:38:13.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.505+0000 7f5b8cb72640 1 -- start start 2023-10-30T14:38:13.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.509+0000 7f5b8cb72640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5b88151c20 0x7f5b880c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:13.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.509+0000 7f5b8cb72640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5b880cb8b0 0x7f5b8806b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:13.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.509+0000 7f5b8cb72640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b880c5030 con 0x7f5b88151840 2023-10-30T14:38:13.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.509+0000 7f5b8cb72640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b88157830 con 0x7f5b88151c20 2023-10-30T14:38:13.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.509+0000 7f5b8cb72640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b88157b10 con 0x7f5b880cb8b0 2023-10-30T14:38:13.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.509+0000 7f5b85d9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5b88151c20 0x7f5b880c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:13.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.509+0000 7f5b85d9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5b88151c20 0x7f5b880c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40806/0 (socket says 172.21.15.57:40806) 2023-10-30T14:38:13.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.509+0000 7f5b85d9b640 1 -- 172.21.15.57:0/3866046121 learned_addr learned my addr 172.21.15.57:0/3866046121 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:13.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.509+0000 7f5b86d9d640 1 --2- 172.21.15.57:0/3866046121 >> v2:172.21.15.88:3300/0 conn(0x7f5b880cb8b0 0x7f5b8806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:13.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.509+0000 7f5b85d9b640 1 -- 172.21.15.57:0/3866046121 >> v2:172.21.15.88:3300/0 conn(0x7f5b880cb8b0 msgr2=0x7f5b8806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:13.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.509+0000 7f5b85d9b640 1 --2- 172.21.15.57:0/3866046121 >> v2:172.21.15.88:3300/0 conn(0x7f5b880cb8b0 0x7f5b8806b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:13.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.509+0000 7f5b85d9b640 1 -- 172.21.15.57:0/3866046121 >> v1:172.21.15.88:6789/0 conn(0x7f5b88151840 legacy=0x7f5b88072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:13.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.509+0000 7f5b85d9b640 1 -- 172.21.15.57:0/3866046121 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b88157df0 con 0x7f5b88151c20 2023-10-30T14:38:13.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.513+0000 7f5b85d9b640 1 --2- 172.21.15.57:0/3866046121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b88151c20 0x7f5b880c49a0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f5b880c6cb0 tx=0x7f5b7c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f46498229ef16e4b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:13.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.513+0000 7f5b8559a640 1 -- 172.21.15.57:0/3866046121 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c0a89f0 con 0x7f5b88151c20 2023-10-30T14:38:13.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.513+0000 7f5b8559a640 1 -- 172.21.15.57:0/3866046121 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7c0a8b50 con 0x7f5b88151c20 2023-10-30T14:38:13.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.513+0000 7f5b8559a640 1 -- 172.21.15.57:0/3866046121 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c0b0440 con 0x7f5b88151c20 2023-10-30T14:38:13.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.513+0000 7f5b8cb72640 1 -- 172.21.15.57:0/3866046121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b88151c20 msgr2=0x7f5b880c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:13.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.513+0000 7f5b8cb72640 1 --2- 172.21.15.57:0/3866046121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b88151c20 0x7f5b880c49a0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f5b880c6cb0 tx=0x7f5b7c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:38:13.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.513+0000 7f5b8cb72640 1 -- 172.21.15.57:0/3866046121 shutdown_connections 2023-10-30T14:38:13.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.513+0000 7f5b8cb72640 1 --2- 172.21.15.57:0/3866046121 >> v2:172.21.15.88:3300/0 conn(0x7f5b880cb8b0 0x7f5b8806b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:13.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.513+0000 7f5b8cb72640 1 --2- 172.21.15.57:0/3866046121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b88151c20 0x7f5b880c49a0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:13.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.513+0000 7f5b8cb72640 1 -- 172.21.15.57:0/3866046121 >> 172.21.15.57:0/3866046121 conn(0x7f5b880cac70 msgr2=0x7f5b880cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:13.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.513+0000 7f5b8cb72640 1 -- 172.21.15.57:0/3866046121 shutdown_connections 2023-10-30T14:38:13.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.513+0000 7f5b8cb72640 1 -- 172.21.15.57:0/3866046121 wait complete. 2023-10-30T14:38:13.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.513+0000 7f5b8cb72640 1 Processor -- start 2023-10-30T14:38:13.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.513+0000 7f5b8cb72640 1 -- start start 2023-10-30T14:38:13.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b8cb72640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b880cb8b0 0x7f5b881f2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:13.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b8cb72640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b88151840 0x7f5b881eeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:13.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b8cb72640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b88151c20 0x7f5b880c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:13.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b8cb72640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881e1c10 con 0x7f5b88151840 2023-10-30T14:38:13.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b8cb72640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881ef0d0 con 0x7f5b880cb8b0 2023-10-30T14:38:13.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b8cb72640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881e1df0 con 0x7f5b88151c20 2023-10-30T14:38:13.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b85d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b88151840 0x7f5b881eeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:13.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b85d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b88151840 0x7f5b881eeb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40818/0 (socket says 172.21.15.57:40818) 2023-10-30T14:38:13.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b85d9b640 1 -- 172.21.15.57:0/1218492687 learned_addr learned my addr 172.21.15.57:0/1218492687 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:13.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b8659c640 1 --2- 172.21.15.57:0/1218492687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b880cb8b0 0x7f5b881f2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:13.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b86d9d640 1 --2- 172.21.15.57:0/1218492687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b88151c20 0x7f5b880c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:13.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b85d9b640 1 -- 172.21.15.57:0/1218492687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b88151c20 msgr2=0x7f5b880c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:13.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b85d9b640 1 --2- 172.21.15.57:0/1218492687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b88151c20 0x7f5b880c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:13.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b85d9b640 1 -- 172.21.15.57:0/1218492687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b880cb8b0 msgr2=0x7f5b881f2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:13.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b85d9b640 1 --2- 172.21.15.57:0/1218492687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b880cb8b0 0x7f5b881f2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:13.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b85d9b640 1 -- 172.21.15.57:0/1218492687 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c0094d0 con 0x7f5b88151840 2023-10-30T14:38:13.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b8659c640 1 --2- 172.21.15.57:0/1218492687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b880cb8b0 0x7f5b881f2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:38:13.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b85d9b640 1 --2- 172.21.15.57:0/1218492687 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b88151840 0x7f5b881eeb90 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c01b900 tx=0x7f5b7c01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:13.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b76ffd640 1 -- 172.21.15.57:0/1218492687 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c0ba600 con 0x7f5b88151840 2023-10-30T14:38:13.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b76ffd640 1 -- 172.21.15.57:0/1218492687 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7c0ba760 con 0x7f5b88151840 2023-10-30T14:38:13.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b8cb72640 1 -- 172.21.15.57:0/1218492687 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b881f21a0 con 0x7f5b88151840 2023-10-30T14:38:13.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b76ffd640 1 -- 172.21.15.57:0/1218492687 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c0bab10 con 0x7f5b88151840 2023-10-30T14:38:13.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b8cb72640 1 -- 172.21.15.57:0/1218492687 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b881f2760 con 0x7f5b88151840 2023-10-30T14:38:13.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b8cb72640 1 -- 172.21.15.57:0/1218492687 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b8804e170 con 0x7f5b88151840 2023-10-30T14:38:13.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.517+0000 7f5b76ffd640 1 -- 172.21.15.57:0/1218492687 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5b7c0bac70 con 0x7f5b88151840 2023-10-30T14:38:13.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.521+0000 7f5b76ffd640 1 --2- 172.21.15.57:0/1218492687 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5b680664a0 0x7f5b68068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:13.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.521+0000 7f5b76ffd640 1 -- 172.21.15.57:0/1218492687 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6857+0+0 (secure 0 0 0) 0x7f5b7c122810 con 0x7f5b88151840 2023-10-30T14:38:13.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.521+0000 7f5b8659c640 1 --2- 172.21.15.57:0/1218492687 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5b680664a0 0x7f5b68068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:13.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.521+0000 7f5b8659c640 1 --2- 172.21.15.57:0/1218492687 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5b680664a0 0x7f5b68068960 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f5b78000f70 tx=0x7f5b78009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:13.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.525+0000 7f5b76ffd640 1 -- 172.21.15.57:0/1218492687 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5b7c0ef920 con 0x7f5b88151840 2023-10-30T14:38:13.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.717+0000 7f5b8cb72640 1 -- 172.21.15.57:0/1218492687 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b88066510 con 0x7f5b88151840 2023-10-30T14:38:13.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.717+0000 7f5b76ffd640 1 -- 172.21.15.57:0/1218492687 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6535 (secure 0 0 0) 0x7f5b7c0f2f40 con 0x7f5b88151840 2023-10-30T14:38:13.718 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:13.719 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6125,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:36:09.152736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"e","rank":2,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.57:6837/1843528309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1843528309},{"type":"v1","addr":"172.21.15.57:6837","nonce":1843528309}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:13.719 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 127 2023-10-30T14:38:13.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.717+0000 7f5b8cb72640 1 -- 172.21.15.57:0/1218492687 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5b680664a0 msgr2=0x7f5b68068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:13.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.717+0000 7f5b8cb72640 1 --2- 172.21.15.57:0/1218492687 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5b680664a0 0x7f5b68068960 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f5b78000f70 tx=0x7f5b78009040 comp rx=0 tx=0).stop 2023-10-30T14:38:13.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.717+0000 7f5b8cb72640 1 -- 172.21.15.57:0/1218492687 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b88151840 msgr2=0x7f5b881eeb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:13.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.717+0000 7f5b8cb72640 1 --2- 172.21.15.57:0/1218492687 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b88151840 0x7f5b881eeb90 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c01b900 tx=0x7f5b7c01ba90 comp rx=0 tx=0).stop 2023-10-30T14:38:13.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.717+0000 7f5b8cb72640 1 -- 172.21.15.57:0/1218492687 shutdown_connections 2023-10-30T14:38:13.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.717+0000 7f5b8cb72640 1 --2- 172.21.15.57:0/1218492687 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5b680664a0 0x7f5b68068960 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:13.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.717+0000 7f5b8cb72640 1 --2- 172.21.15.57:0/1218492687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b88151c20 0x7f5b880c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:13.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.717+0000 7f5b8cb72640 1 --2- 172.21.15.57:0/1218492687 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b88151840 0x7f5b881eeb90 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:13.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.717+0000 7f5b8cb72640 1 --2- 172.21.15.57:0/1218492687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b880cb8b0 0x7f5b881f2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:13.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.717+0000 7f5b8cb72640 1 -- 172.21.15.57:0/1218492687 >> 172.21.15.57:0/1218492687 conn(0x7f5b880cac70 msgr2=0x7f5b880cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:13.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.717+0000 7f5b8cb72640 1 -- 172.21.15.57:0/1218492687 shutdown_connections 2023-10-30T14:38:13.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:13.717+0000 7f5b8cb72640 1 -- 172.21.15.57:0/1218492687 wait complete. 2023-10-30T14:38:13.739 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:38:15.740 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:15.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.889+0000 7f94d2c09640 1 Processor -- start 2023-10-30T14:38:15.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.889+0000 7f94d2c09640 1 -- start start 2023-10-30T14:38:15.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94d2c09640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f94cc153aa0 0x7f94cc15d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:15.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94d2c09640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94cc0701e0 0x7f94cc0ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:15.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94d2c09640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94cc0c7b00 con 0x7f94cc0cd420 2023-10-30T14:38:15.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94d2c09640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94cc070a50 con 0x7f94cc153aa0 2023-10-30T14:38:15.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94d2c09640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc15dd80 con 0x7f94cc0701e0 2023-10-30T14:38:15.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94d11a6640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f94cc0cd420 0x7f94cc06b3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50570/0 (socket says 172.21.15.57:50570) 2023-10-30T14:38:15.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94d11a6640 1 -- 172.21.15.57:0/169687417 learned_addr learned my addr 172.21.15.57:0/169687417 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:38:15.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94d09a5640 1 --2- 172.21.15.57:0/169687417 >> v2:172.21.15.88:3300/0 conn(0x7f94cc153aa0 0x7f94cc15d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:15.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94cbfff640 1 --2- 172.21.15.57:0/169687417 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94cc0701e0 0x7f94cc0ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:15.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94cb7fe640 1 -- 172.21.15.57:0/169687417 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69176030 0 0) 0x7f94cc0c7b00 con 0x7f94cc0cd420 2023-10-30T14:38:15.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94cb7fe640 1 -- 172.21.15.57:0/169687417 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94bc003530 con 0x7f94cc0cd420 2023-10-30T14:38:15.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94cb7fe640 1 -- 172.21.15.57:0/169687417 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2254725907 0 0) 0x7f94bc003530 con 0x7f94cc0cd420 2023-10-30T14:38:15.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94cb7fe640 1 -- 172.21.15.57:0/169687417 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94cc0701e0 msgr2=0x7f94cc0ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:15.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94cb7fe640 1 --2- 172.21.15.57:0/169687417 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94cc0701e0 0x7f94cc0ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:15.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94cb7fe640 1 -- 172.21.15.57:0/169687417 >> v2:172.21.15.88:3300/0 conn(0x7f94cc153aa0 msgr2=0x7f94cc15d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:15.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94cb7fe640 1 --2- 172.21.15.57:0/169687417 >> v2:172.21.15.88:3300/0 conn(0x7f94cc153aa0 0x7f94cc15d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:15.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94cb7fe640 1 -- 172.21.15.57:0/169687417 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94cc0c2050 con 0x7f94cc0cd420 2023-10-30T14:38:15.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94cb7fe640 1 -- 172.21.15.57:0/169687417 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f94c4001a00 con 0x7f94cc0cd420 2023-10-30T14:38:15.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94cb7fe640 1 -- 172.21.15.57:0/169687417 >> v1:172.21.15.57:6789/0 conn(0x7f94cc0cd420 legacy=0x7f94cc06b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:15.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94cb7fe640 1 --2- 172.21.15.57:0/169687417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94bc003980 0x7f94bc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:15.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94cb7fe640 1 --2- 172.21.15.57:0/169687417 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94cc0701e0 0x7f94bc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:15.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94cb7fe640 1 --2- 172.21.15.57:0/169687417 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94bc004d10 0x7f94bc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:15.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94cb7fe640 1 -- 172.21.15.57:0/169687417 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc070a50 con 0x7f94bc004d10 2023-10-30T14:38:15.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94cb7fe640 1 -- 172.21.15.57:0/169687417 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc15dd80 con 0x7f94bc003980 2023-10-30T14:38:15.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94cb7fe640 1 -- 172.21.15.57:0/169687417 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc00b990 con 0x7f94cc0701e0 2023-10-30T14:38:15.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94d09a5640 1 --2- 172.21.15.57:0/169687417 >> v2:172.21.15.88:3300/0 conn(0x7f94cc153aa0 0x7f94cc15d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:38:15.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94d11a6640 1 --2- 172.21.15.57:0/169687417 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94bc004d10 0x7f94bc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:15.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94cbfff640 1 --2- 172.21.15.57:0/169687417 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94cc0701e0 0x7f94bc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:15.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94d09a5640 1 --2- 172.21.15.57:0/169687417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94bc003980 0x7f94bc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:15.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94d11a6640 1 -- 172.21.15.57:0/169687417 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94cc0701e0 msgr2=0x7f94bc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:15.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94d11a6640 1 --2- 172.21.15.57:0/169687417 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94cc0701e0 0x7f94bc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:15.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94d11a6640 1 -- 172.21.15.57:0/169687417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94bc003980 msgr2=0x7f94bc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:15.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94d11a6640 1 --2- 172.21.15.57:0/169687417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94bc003980 0x7f94bc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:15.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94d11a6640 1 -- 172.21.15.57:0/169687417 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94bc003530 con 0x7f94bc004d10 2023-10-30T14:38:15.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94d09a5640 1 --2- 172.21.15.57:0/169687417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94bc003980 0x7f94bc003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:38:15.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.893+0000 7f94cbfff640 1 --2- 172.21.15.57:0/169687417 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94cc0701e0 0x7f94bc0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:38:15.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.897+0000 7f94d11a6640 1 --2- 172.21.15.57:0/169687417 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94bc004d10 0x7f94bc005190 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f94c400a830 tx=0x7f94c401d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dfe45c2c93560f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:15.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.897+0000 7f94cb7fe640 1 -- 172.21.15.57:0/169687417 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c40ad630 con 0x7f94bc004d10 2023-10-30T14:38:15.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.897+0000 7f94cb7fe640 1 -- 172.21.15.57:0/169687417 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c40a8070 con 0x7f94bc004d10 2023-10-30T14:38:15.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.897+0000 7f94cb7fe640 1 -- 172.21.15.57:0/169687417 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c40ad9f0 con 0x7f94bc004d10 2023-10-30T14:38:15.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.897+0000 7f94d2c09640 1 -- 172.21.15.57:0/169687417 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94bc004d10 msgr2=0x7f94bc005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:15.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.897+0000 7f94d2c09640 1 --2- 172.21.15.57:0/169687417 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94bc004d10 0x7f94bc005190 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f94c400a830 tx=0x7f94c401d360 comp rx=0 tx=0).stop 2023-10-30T14:38:15.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.897+0000 7f94d11a6640 1 -- 172.21.15.57:0/169687417 reap_dead start 2023-10-30T14:38:15.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.897+0000 7f94d2c09640 1 -- 172.21.15.57:0/169687417 shutdown_connections 2023-10-30T14:38:15.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.897+0000 7f94d2c09640 1 -- 172.21.15.57:0/169687417 >> 172.21.15.57:0/169687417 conn(0x7f94cc071fb0 msgr2=0x7f94cc0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:15.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.897+0000 7f94d2c09640 1 -- 172.21.15.57:0/169687417 shutdown_connections 2023-10-30T14:38:15.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.897+0000 7f94d2c09640 1 -- 172.21.15.57:0/169687417 wait complete. 2023-10-30T14:38:15.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94d2c09640 1 Processor -- start 2023-10-30T14:38:15.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94d2c09640 1 -- start start 2023-10-30T14:38:15.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94d2c09640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94bc004d10 0x7f94cc146b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:15.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94d2c09640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94cc156060 0x7f94cc0cca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:15.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94d2c09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94cc0ccfa0 0x7f94cc0c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:15.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94d2c09640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc0cc110 con 0x7f94cc156060 2023-10-30T14:38:15.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94d2c09640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc0cbdb0 con 0x7f94cc0ccfa0 2023-10-30T14:38:15.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94d2c09640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc0caec0 con 0x7f94bc004d10 2023-10-30T14:38:15.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94cbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94cc156060 0x7f94cc0cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:15.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94cbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94cc156060 0x7f94cc0cca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39074/0 (socket says 172.21.15.57:39074) 2023-10-30T14:38:15.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94cbfff640 1 -- 172.21.15.57:0/1816619818 learned_addr learned my addr 172.21.15.57:0/1816619818 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:15.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94d11a6640 1 --2- 172.21.15.57:0/1816619818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94cc0ccfa0 0x7f94cc0c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:15.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94d09a5640 1 --2- 172.21.15.57:0/1816619818 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94bc004d10 0x7f94cc146b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:15.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94cbfff640 1 -- 172.21.15.57:0/1816619818 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94bc004d10 msgr2=0x7f94cc146b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:15.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94cbfff640 1 --2- 172.21.15.57:0/1816619818 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94bc004d10 0x7f94cc146b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:15.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94cbfff640 1 -- 172.21.15.57:0/1816619818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94cc0ccfa0 msgr2=0x7f94cc0c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:15.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94cbfff640 1 --2- 172.21.15.57:0/1816619818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94cc0ccfa0 0x7f94cc0c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:15.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94cbfff640 1 -- 172.21.15.57:0/1816619818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94cc0cb1a0 con 0x7f94cc156060 2023-10-30T14:38:15.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94cbfff640 1 --2- 172.21.15.57:0/1816619818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94cc156060 0x7f94cc0cca60 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f94b8009fd0 tx=0x7f94b8011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:15.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94c97fa640 1 -- 172.21.15.57:0/1816619818 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b8004220 con 0x7f94cc156060 2023-10-30T14:38:15.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94c97fa640 1 -- 172.21.15.57:0/1816619818 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b8004380 con 0x7f94cc156060 2023-10-30T14:38:15.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94c97fa640 1 -- 172.21.15.57:0/1816619818 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b800f9e0 con 0x7f94cc156060 2023-10-30T14:38:15.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94d2c09640 1 -- 172.21.15.57:0/1816619818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94cc0cb480 con 0x7f94cc156060 2023-10-30T14:38:15.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94d2c09640 1 -- 172.21.15.57:0/1816619818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94cc0cba40 con 0x7f94cc156060 2023-10-30T14:38:15.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.901+0000 7f94d2c09640 1 -- 172.21.15.57:0/1816619818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94cc1efda0 con 0x7f94cc156060 2023-10-30T14:38:15.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.905+0000 7f94c97fa640 1 -- 172.21.15.57:0/1816619818 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f94b800fb40 con 0x7f94cc156060 2023-10-30T14:38:15.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.905+0000 7f94c97fa640 1 --2- 172.21.15.57:0/1816619818 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f94b00664d0 0x7f94b0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:15.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.905+0000 7f94c97fa640 1 -- 172.21.15.57:0/1816619818 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7f94b8084960 con 0x7f94cc156060 2023-10-30T14:38:15.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.905+0000 7f94d09a5640 1 --2- 172.21.15.57:0/1816619818 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f94b00664d0 0x7f94b0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:15.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.905+0000 7f94d09a5640 1 --2- 172.21.15.57:0/1816619818 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f94b00664d0 0x7f94b0068990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f94c0006f90 tx=0x7f94c0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:15.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:15.909+0000 7f94c97fa640 1 -- 172.21.15.57:0/1816619818 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f94cc1efda0 con 0x7f94cc156060 2023-10-30T14:38:16.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.101+0000 7f94d2c09640 1 -- 172.21.15.57:0/1816619818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94cc1efda0 con 0x7f94cc156060 2023-10-30T14:38:16.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.101+0000 7f94c97fa640 1 -- 172.21.15.57:0/1816619818 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+5741 (secure 0 0 0) 0x7f94cc1efda0 con 0x7f94cc156060 2023-10-30T14:38:16.102 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:16.102 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:15.025367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:16.102 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 128 2023-10-30T14:38:16.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.101+0000 7f94d2c09640 1 -- 172.21.15.57:0/1816619818 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f94b00664d0 msgr2=0x7f94b0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:16.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.101+0000 7f94d2c09640 1 --2- 172.21.15.57:0/1816619818 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f94b00664d0 0x7f94b0068990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f94c0006f90 tx=0x7f94c0008040 comp rx=0 tx=0).stop 2023-10-30T14:38:16.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.101+0000 7f94d2c09640 1 -- 172.21.15.57:0/1816619818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94cc156060 msgr2=0x7f94cc0cca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:16.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.101+0000 7f94d2c09640 1 --2- 172.21.15.57:0/1816619818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94cc156060 0x7f94cc0cca60 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f94b8009fd0 tx=0x7f94b8011b30 comp rx=0 tx=0).stop 2023-10-30T14:38:16.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.101+0000 7f94d2c09640 1 -- 172.21.15.57:0/1816619818 shutdown_connections 2023-10-30T14:38:16.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.101+0000 7f94d2c09640 1 --2- 172.21.15.57:0/1816619818 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f94b00664d0 0x7f94b0068990 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:16.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.101+0000 7f94d2c09640 1 --2- 172.21.15.57:0/1816619818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94cc0ccfa0 0x7f94cc0c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:16.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.101+0000 7f94d2c09640 1 --2- 172.21.15.57:0/1816619818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94cc156060 0x7f94cc0cca60 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:16.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.101+0000 7f94d2c09640 1 --2- 172.21.15.57:0/1816619818 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94bc004d10 0x7f94cc146b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:16.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.101+0000 7f94d2c09640 1 -- 172.21.15.57:0/1816619818 >> 172.21.15.57:0/1816619818 conn(0x7f94cc071fb0 msgr2=0x7f94cc149dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:16.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.101+0000 7f94d2c09640 1 -- 172.21.15.57:0/1816619818 shutdown_connections 2023-10-30T14:38:16.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.101+0000 7f94d2c09640 1 -- 172.21.15.57:0/1816619818 wait complete. 2023-10-30T14:38:16.123 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T14:38:16.123 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:38:16.123 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:16.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.265+0000 7f985dfc4640 1 Processor -- start 2023-10-30T14:38:16.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.265+0000 7f985dfc4640 1 -- start start 2023-10-30T14:38:16.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.265+0000 7f985dfc4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9858153c80 0x7f985815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:16.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.265+0000 7f985dfc4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98580c42a0 0x7f98580ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:16.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.265+0000 7f985dfc4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9858159f20 con 0x7f98580c4a10 2023-10-30T14:38:16.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.265+0000 7f985dfc4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f985815dec0 con 0x7f9858153c80 2023-10-30T14:38:16.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.265+0000 7f985dfc4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98581599a0 con 0x7f98580c42a0 2023-10-30T14:38:16.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.265+0000 7f98577fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9858153c80 0x7f985815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:16.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.265+0000 7f98577fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9858153c80 0x7f985815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39080/0 (socket says 172.21.15.57:39080) 2023-10-30T14:38:16.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.265+0000 7f98577fe640 1 -- 172.21.15.57:0/1714531406 learned_addr learned my addr 172.21.15.57:0/1714531406 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:16.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.265+0000 7f9857fff640 1 --2- 172.21.15.57:0/1714531406 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98580c42a0 0x7f98580ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:16.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.265+0000 7f98577fe640 1 -- 172.21.15.57:0/1714531406 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98580c42a0 msgr2=0x7f98580ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:16.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.265+0000 7f98577fe640 1 --2- 172.21.15.57:0/1714531406 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98580c42a0 0x7f98580ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:16.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.265+0000 7f98577fe640 1 -- 172.21.15.57:0/1714531406 >> v1:172.21.15.57:6789/0 conn(0x7f98580c4a10 legacy=0x7f985815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:16.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.265+0000 7f98577fe640 1 -- 172.21.15.57:0/1714531406 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985815d620 con 0x7f9858153c80 2023-10-30T14:38:16.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.269+0000 7f98577fe640 1 --2- 172.21.15.57:0/1714531406 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9858153c80 0x7f985815cf90 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f9848009870 tx=0x7f984801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf94707af6a6f58 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:16.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.269+0000 7f98567fc640 1 -- 172.21.15.57:0/1714531406 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98480a8a30 con 0x7f9858153c80 2023-10-30T14:38:16.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.269+0000 7f98567fc640 1 -- 172.21.15.57:0/1714531406 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98480a8b90 con 0x7f9858153c80 2023-10-30T14:38:16.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.269+0000 7f98567fc640 1 -- 172.21.15.57:0/1714531406 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98480b0440 con 0x7f9858153c80 2023-10-30T14:38:16.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.269+0000 7f985dfc4640 1 -- 172.21.15.57:0/1714531406 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9858153c80 msgr2=0x7f985815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:16.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.269+0000 7f985dfc4640 1 --2- 172.21.15.57:0/1714531406 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9858153c80 0x7f985815cf90 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f9848009870 tx=0x7f984801ed30 comp rx=0 tx=0).stop 2023-10-30T14:38:16.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.269+0000 7f985dfc4640 1 -- 172.21.15.57:0/1714531406 shutdown_connections 2023-10-30T14:38:16.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.269+0000 7f985dfc4640 1 --2- 172.21.15.57:0/1714531406 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98580c42a0 0x7f98580ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:16.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.269+0000 7f985dfc4640 1 --2- 172.21.15.57:0/1714531406 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9858153c80 0x7f985815cf90 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:16.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.269+0000 7f985dfc4640 1 -- 172.21.15.57:0/1714531406 >> 172.21.15.57:0/1714531406 conn(0x7f98580715c0 msgr2=0x7f98580719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:16.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.269+0000 7f985dfc4640 1 -- 172.21.15.57:0/1714531406 shutdown_connections 2023-10-30T14:38:16.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.269+0000 7f985dfc4640 1 -- 172.21.15.57:0/1714531406 wait complete. 2023-10-30T14:38:16.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f985dfc4640 1 Processor -- start 2023-10-30T14:38:16.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f985dfc4640 1 -- start start 2023-10-30T14:38:16.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f985dfc4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98580c42a0 0x7f9858155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:16.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f985dfc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98580c4a10 0x7f985814f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:16.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f985dfc4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9858153c80 0x7f98581511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:16.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f985dfc4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98581f3320 con 0x7f98580c42a0 2023-10-30T14:38:16.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f985dfc4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98581556d0 con 0x7f98580c4a10 2023-10-30T14:38:16.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f985dfc4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9858157ea0 con 0x7f9858153c80 2023-10-30T14:38:16.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f98577fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98580c42a0 0x7f9858155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:16.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f98577fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98580c42a0 0x7f9858155190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39092/0 (socket says 172.21.15.57:39092) 2023-10-30T14:38:16.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f98577fe640 1 -- 172.21.15.57:0/4083765793 learned_addr learned my addr 172.21.15.57:0/4083765793 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:16.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f9856ffd640 1 --2- 172.21.15.57:0/4083765793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98580c4a10 0x7f985814f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:16.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f9857fff640 1 --2- 172.21.15.57:0/4083765793 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9858153c80 0x7f98581511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:16.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f98577fe640 1 -- 172.21.15.57:0/4083765793 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9858153c80 msgr2=0x7f98581511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:16.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f98577fe640 1 --2- 172.21.15.57:0/4083765793 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9858153c80 0x7f98581511b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:16.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f98577fe640 1 -- 172.21.15.57:0/4083765793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98580c4a10 msgr2=0x7f985814f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:16.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f98577fe640 1 --2- 172.21.15.57:0/4083765793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98580c4a10 0x7f985814f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:16.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f98577fe640 1 -- 172.21.15.57:0/4083765793 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98480094d0 con 0x7f98580c42a0 2023-10-30T14:38:16.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f9857fff640 1 --2- 172.21.15.57:0/4083765793 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9858153c80 0x7f98581511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:38:16.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f98577fe640 1 --2- 172.21.15.57:0/4083765793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98580c42a0 0x7f9858155190 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f98480b3040 tx=0x7f984801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:16.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f983ffff640 1 -- 172.21.15.57:0/4083765793 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98480ba600 con 0x7f98580c42a0 2023-10-30T14:38:16.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f983ffff640 1 -- 172.21.15.57:0/4083765793 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98480ba780 con 0x7f98580c42a0 2023-10-30T14:38:16.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f983ffff640 1 -- 172.21.15.57:0/4083765793 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98480bab30 con 0x7f98580c42a0 2023-10-30T14:38:16.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.273+0000 7f985dfc4640 1 -- 172.21.15.57:0/4083765793 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98581f2430 con 0x7f98580c42a0 2023-10-30T14:38:16.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.277+0000 7f985dfc4640 1 -- 172.21.15.57:0/4083765793 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98581f29f0 con 0x7f98580c42a0 2023-10-30T14:38:16.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.277+0000 7f985dfc4640 1 -- 172.21.15.57:0/4083765793 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f985804e170 con 0x7f98580c42a0 2023-10-30T14:38:16.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.277+0000 7f983ffff640 1 -- 172.21.15.57:0/4083765793 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f98480bac90 con 0x7f98580c42a0 2023-10-30T14:38:16.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.277+0000 7f983ffff640 1 --2- 172.21.15.57:0/4083765793 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f98440664a0 0x7f9844068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:16.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.277+0000 7f983ffff640 1 -- 172.21.15.57:0/4083765793 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7f98481228d0 con 0x7f98580c42a0 2023-10-30T14:38:16.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.277+0000 7f9856ffd640 1 --2- 172.21.15.57:0/4083765793 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f98440664a0 0x7f9844068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:16.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.281+0000 7f9856ffd640 1 --2- 172.21.15.57:0/4083765793 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f98440664a0 0x7f9844068960 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9858150160 tx=0x7f984c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:16.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.281+0000 7f983ffff640 1 -- 172.21.15.57:0/4083765793 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f98480ef7e0 con 0x7f98580c42a0 2023-10-30T14:38:16.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.473+0000 7f985dfc4640 1 -- 172.21.15.57:0/4083765793 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98580cad40 con 0x7f98580c42a0 2023-10-30T14:38:16.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.473+0000 7f983ffff640 1 -- 172.21.15.57:0/4083765793 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+5741 (secure 0 0 0) 0x7f98480f2e00 con 0x7f98580c42a0 2023-10-30T14:38:16.474 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:16.474 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:15.025367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:16.475 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 128 2023-10-30T14:38:16.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.473+0000 7f985dfc4640 1 -- 172.21.15.57:0/4083765793 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f98440664a0 msgr2=0x7f9844068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:16.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.473+0000 7f985dfc4640 1 --2- 172.21.15.57:0/4083765793 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f98440664a0 0x7f9844068960 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9858150160 tx=0x7f984c009040 comp rx=0 tx=0).stop 2023-10-30T14:38:16.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.473+0000 7f985dfc4640 1 -- 172.21.15.57:0/4083765793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98580c42a0 msgr2=0x7f9858155190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:16.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.473+0000 7f985dfc4640 1 --2- 172.21.15.57:0/4083765793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98580c42a0 0x7f9858155190 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f98480b3040 tx=0x7f984801bad0 comp rx=0 tx=0).stop 2023-10-30T14:38:16.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.473+0000 7f985dfc4640 1 -- 172.21.15.57:0/4083765793 shutdown_connections 2023-10-30T14:38:16.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.473+0000 7f985dfc4640 1 --2- 172.21.15.57:0/4083765793 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f98440664a0 0x7f9844068960 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:16.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.473+0000 7f985dfc4640 1 --2- 172.21.15.57:0/4083765793 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9858153c80 0x7f98581511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:16.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.473+0000 7f985dfc4640 1 --2- 172.21.15.57:0/4083765793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98580c4a10 0x7f985814f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:16.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.473+0000 7f985dfc4640 1 --2- 172.21.15.57:0/4083765793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98580c42a0 0x7f9858155190 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:16.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.473+0000 7f985dfc4640 1 -- 172.21.15.57:0/4083765793 >> 172.21.15.57:0/4083765793 conn(0x7f98580715c0 msgr2=0x7f9858157670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:16.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.473+0000 7f985dfc4640 1 -- 172.21.15.57:0/4083765793 shutdown_connections 2023-10-30T14:38:16.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:16.473+0000 7f985dfc4640 1 -- 172.21.15.57:0/4083765793 wait complete. 2023-10-30T14:38:16.496 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 128, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 128, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6125": { "addr": "172.21.15.88:6839/2390289388", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 2390289388, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2390289388, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6125, "incarnation": 128, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:replay", "state_seq": 1 }, "gid_6414": { "addr": "172.21.15.57:6834/2321408149", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2321408149, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2321408149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6414, "incarnation": 99, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 18 }, "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 44 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:38:15.025367+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6414, "mds_1": 6747, "mds_2": 6125 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:38:18.497 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:18.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.649+0000 7fcbeafab640 1 Processor -- start 2023-10-30T14:38:18.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.649+0000 7fcbeafab640 1 -- start start 2023-10-30T14:38:18.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.649+0000 7fcbeafab640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcbe4150830 0x7fcbe406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:18.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.649+0000 7fcbeafab640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcbe4070520 0x7fcbe4070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:18.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.649+0000 7fcbeafab640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe40c2310 con 0x7fcbe40cb7d0 2023-10-30T14:38:18.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.649+0000 7fcbeafab640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe4156710 con 0x7fcbe4150830 2023-10-30T14:38:18.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.649+0000 7fcbeafab640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe40c4360 con 0x7fcbe4070520 2023-10-30T14:38:18.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.653+0000 7fcbe8d47640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcbe4150830 0x7fcbe406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:18.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.653+0000 7fcbe8d47640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcbe4150830 0x7fcbe406b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39098/0 (socket says 172.21.15.57:39098) 2023-10-30T14:38:18.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.653+0000 7fcbe8d47640 1 -- 172.21.15.57:0/3128786741 learned_addr learned my addr 172.21.15.57:0/3128786741 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:18.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.653+0000 7fcbe9548640 1 --2- 172.21.15.57:0/3128786741 >> v2:172.21.15.88:3300/0 conn(0x7fcbe4070520 0x7fcbe4070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:18.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.653+0000 7fcbe8d47640 1 -- 172.21.15.57:0/3128786741 >> v2:172.21.15.88:3300/0 conn(0x7fcbe4070520 msgr2=0x7fcbe4070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:18.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.653+0000 7fcbe8d47640 1 --2- 172.21.15.57:0/3128786741 >> v2:172.21.15.88:3300/0 conn(0x7fcbe4070520 0x7fcbe4070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:18.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.653+0000 7fcbe8d47640 1 -- 172.21.15.57:0/3128786741 >> v1:172.21.15.57:6789/0 conn(0x7fcbe40cb7d0 legacy=0x7fcbe40c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:18.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.653+0000 7fcbe8d47640 1 -- 172.21.15.57:0/3128786741 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe41569b0 con 0x7fcbe4150830 2023-10-30T14:38:18.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.653+0000 7fcbe8d47640 1 --2- 172.21.15.57:0/3128786741 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbe4150830 0x7fcbe406b500 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4066e10 tx=0x7fcbd801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ce8a6c70899c27b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:18.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.653+0000 7fcbe37fe640 1 -- 172.21.15.57:0/3128786741 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd80a89f0 con 0x7fcbe4150830 2023-10-30T14:38:18.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.653+0000 7fcbe37fe640 1 -- 172.21.15.57:0/3128786741 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd80a8b50 con 0x7fcbe4150830 2023-10-30T14:38:18.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.653+0000 7fcbe37fe640 1 -- 172.21.15.57:0/3128786741 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd80b0440 con 0x7fcbe4150830 2023-10-30T14:38:18.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbeafab640 1 -- 172.21.15.57:0/3128786741 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbe4150830 msgr2=0x7fcbe406b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:18.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbeafab640 1 --2- 172.21.15.57:0/3128786741 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbe4150830 0x7fcbe406b500 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4066e10 tx=0x7fcbd801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:38:18.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbeafab640 1 -- 172.21.15.57:0/3128786741 shutdown_connections 2023-10-30T14:38:18.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbeafab640 1 --2- 172.21.15.57:0/3128786741 >> v2:172.21.15.88:3300/0 conn(0x7fcbe4070520 0x7fcbe4070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:18.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbeafab640 1 --2- 172.21.15.57:0/3128786741 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbe4150830 0x7fcbe406b500 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:18.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbeafab640 1 -- 172.21.15.57:0/3128786741 >> 172.21.15.57:0/3128786741 conn(0x7fcbe40cac70 msgr2=0x7fcbe40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:18.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbeafab640 1 -- 172.21.15.57:0/3128786741 shutdown_connections 2023-10-30T14:38:18.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbeafab640 1 -- 172.21.15.57:0/3128786741 wait complete. 2023-10-30T14:38:18.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbeafab640 1 Processor -- start 2023-10-30T14:38:18.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbeafab640 1 -- start start 2023-10-30T14:38:18.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbeafab640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbe4070520 0x7fcbe40c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:18.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbeafab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbe40cb7d0 0x7fcbe41ef740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:18.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbeafab640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbe4150830 0x7fcbe4151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:18.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbeafab640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe4152d40 con 0x7fcbe40cb7d0 2023-10-30T14:38:18.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbeafab640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe41f2e90 con 0x7fcbe4070520 2023-10-30T14:38:18.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbeafab640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe41f33c0 con 0x7fcbe4150830 2023-10-30T14:38:18.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbe3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbe40cb7d0 0x7fcbe41ef740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:18.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbe3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbe40cb7d0 0x7fcbe41ef740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39102/0 (socket says 172.21.15.57:39102) 2023-10-30T14:38:18.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbe3fff640 1 -- 172.21.15.57:0/2830999193 learned_addr learned my addr 172.21.15.57:0/2830999193 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:18.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbe8d47640 1 --2- 172.21.15.57:0/2830999193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbe4070520 0x7fcbe40c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:18.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbe9548640 1 --2- 172.21.15.57:0/2830999193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbe4150830 0x7fcbe4151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:18.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbe3fff640 1 -- 172.21.15.57:0/2830999193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbe4150830 msgr2=0x7fcbe4151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:18.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbe3fff640 1 --2- 172.21.15.57:0/2830999193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbe4150830 0x7fcbe4151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:18.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbe3fff640 1 -- 172.21.15.57:0/2830999193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbe4070520 msgr2=0x7fcbe40c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:18.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbe3fff640 1 --2- 172.21.15.57:0/2830999193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbe4070520 0x7fcbe40c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:18.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.657+0000 7fcbe3fff640 1 -- 172.21.15.57:0/2830999193 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd80094d0 con 0x7fcbe40cb7d0 2023-10-30T14:38:18.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.661+0000 7fcbe3fff640 1 --2- 172.21.15.57:0/2830999193 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbe40cb7d0 0x7fcbe41ef740 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000c480 tx=0x7fcbd000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:18.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.661+0000 7fcbe17fa640 1 -- 172.21.15.57:0/2830999193 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd00052d0 con 0x7fcbe40cb7d0 2023-10-30T14:38:18.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.661+0000 7fcbe17fa640 1 -- 172.21.15.57:0/2830999193 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd0005450 con 0x7fcbe40cb7d0 2023-10-30T14:38:18.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.661+0000 7fcbe17fa640 1 -- 172.21.15.57:0/2830999193 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd0012870 con 0x7fcbe40cb7d0 2023-10-30T14:38:18.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.661+0000 7fcbeafab640 1 -- 172.21.15.57:0/2830999193 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe41ef2b0 con 0x7fcbe40cb7d0 2023-10-30T14:38:18.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.661+0000 7fcbeafab640 1 -- 172.21.15.57:0/2830999193 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe415a6e0 con 0x7fcbe40cb7d0 2023-10-30T14:38:18.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.661+0000 7fcbeafab640 1 -- 172.21.15.57:0/2830999193 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe41ef440 con 0x7fcbe40cb7d0 2023-10-30T14:38:18.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.661+0000 7fcbe17fa640 1 -- 172.21.15.57:0/2830999193 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcbd00129d0 con 0x7fcbe40cb7d0 2023-10-30T14:38:18.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.661+0000 7fcbe17fa640 1 --2- 172.21.15.57:0/2830999193 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcbc80664a0 0x7fcbc8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:18.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.661+0000 7fcbe17fa640 1 -- 172.21.15.57:0/2830999193 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7fcbd0083ef0 con 0x7fcbe40cb7d0 2023-10-30T14:38:18.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.661+0000 7fcbe8d47640 1 --2- 172.21.15.57:0/2830999193 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcbc80664a0 0x7fcbc8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:18.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.665+0000 7fcbe8d47640 1 --2- 172.21.15.57:0/2830999193 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcbc80664a0 0x7fcbc8068960 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcbd80b2000 tx=0x7fcbd8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:18.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.665+0000 7fcbe17fa640 1 -- 172.21.15.57:0/2830999193 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcbd0050e10 con 0x7fcbe40cb7d0 2023-10-30T14:38:18.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.857+0000 7fcbeafab640 1 -- 172.21.15.57:0/2830999193 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbe404e170 con 0x7fcbe40cb7d0 2023-10-30T14:38:18.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.857+0000 7fcbe17fa640 1 -- 172.21.15.57:0/2830999193 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+5741 (secure 0 0 0) 0x7fcbd0054430 con 0x7fcbe40cb7d0 2023-10-30T14:38:18.861 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:18.861 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:15.025367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:18.861 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 128 2023-10-30T14:38:18.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.861+0000 7fcbeafab640 1 -- 172.21.15.57:0/2830999193 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcbc80664a0 msgr2=0x7fcbc8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:18.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.861+0000 7fcbeafab640 1 --2- 172.21.15.57:0/2830999193 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcbc80664a0 0x7fcbc8068960 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcbd80b2000 tx=0x7fcbd8009040 comp rx=0 tx=0).stop 2023-10-30T14:38:18.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.861+0000 7fcbeafab640 1 -- 172.21.15.57:0/2830999193 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbe40cb7d0 msgr2=0x7fcbe41ef740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:18.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.861+0000 7fcbeafab640 1 --2- 172.21.15.57:0/2830999193 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbe40cb7d0 0x7fcbe41ef740 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000c480 tx=0x7fcbd000c940 comp rx=0 tx=0).stop 2023-10-30T14:38:18.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.861+0000 7fcbeafab640 1 -- 172.21.15.57:0/2830999193 shutdown_connections 2023-10-30T14:38:18.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.861+0000 7fcbeafab640 1 --2- 172.21.15.57:0/2830999193 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcbc80664a0 0x7fcbc8068960 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:18.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.861+0000 7fcbeafab640 1 --2- 172.21.15.57:0/2830999193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbe4150830 0x7fcbe4151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:18.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.861+0000 7fcbeafab640 1 --2- 172.21.15.57:0/2830999193 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbe40cb7d0 0x7fcbe41ef740 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:18.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.861+0000 7fcbeafab640 1 --2- 172.21.15.57:0/2830999193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbe4070520 0x7fcbe40c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:18.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.861+0000 7fcbeafab640 1 -- 172.21.15.57:0/2830999193 >> 172.21.15.57:0/2830999193 conn(0x7fcbe40cac70 msgr2=0x7fcbe40c64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:18.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.861+0000 7fcbeafab640 1 -- 172.21.15.57:0/2830999193 shutdown_connections 2023-10-30T14:38:18.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:18.861+0000 7fcbeafab640 1 -- 172.21.15.57:0/2830999193 wait complete. 2023-10-30T14:38:18.882 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:38:20.883 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:21.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.037+0000 7fb8f00d6640 1 Processor -- start 2023-10-30T14:38:21.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.037+0000 7fb8f00d6640 1 -- start start 2023-10-30T14:38:21.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.041+0000 7fb8f00d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb8e8153c80 0x7fb8e815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:21.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.041+0000 7fb8f00d6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e8159f20 con 0x7fb8e80c42a0 2023-10-30T14:38:21.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.041+0000 7fb8f00d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e815d620 con 0x7fb8e80c4a10 2023-10-30T14:38:21.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.041+0000 7fb8f00d6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e815dec0 con 0x7fb8e8153c80 2023-10-30T14:38:21.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.041+0000 7fb8ede72640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb8e8153c80 0x7fb8e815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:21.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.041+0000 7fb8ede72640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb8e8153c80 0x7fb8e815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:60052/0 (socket says 172.21.15.57:60052) 2023-10-30T14:38:21.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.041+0000 7fb8ede72640 1 -- 172.21.15.57:0/3348177850 learned_addr learned my addr 172.21.15.57:0/3348177850 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:21.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.041+0000 7fb8ee673640 1 --1- 172.21.15.57:0/3348177850 >> v1:172.21.15.57:6789/0 conn(0x7fb8e80c42a0 0x7fb8e80ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50590/0 (socket says 172.21.15.57:50590) 2023-10-30T14:38:21.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.041+0000 7fb8ede72640 1 -- 172.21.15.57:0/3348177850 >> v1:172.21.15.88:6789/0 conn(0x7fb8e80c4a10 legacy=0x7fb8e815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:21.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.041+0000 7fb8ede72640 1 -- 172.21.15.57:0/3348177850 >> v1:172.21.15.57:6789/0 conn(0x7fb8e80c42a0 legacy=0x7fb8e80ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:21.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.041+0000 7fb8ede72640 1 -- 172.21.15.57:0/3348177850 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e815d900 con 0x7fb8e8153c80 2023-10-30T14:38:21.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.041+0000 7fb8ece70640 1 -- 172.21.15.57:0/3348177850 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679945203 0 0) 0x7fb8e8159f20 con 0x7fb8e80c42a0 2023-10-30T14:38:21.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.045+0000 7fb8ede72640 1 --2- 172.21.15.57:0/3348177850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e8153c80 0x7fb8e815cf90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4009870 tx=0x7fb8e401ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b521489663eb71c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:21.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.045+0000 7fb8ece70640 1 -- 172.21.15.57:0/3348177850 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e40a8a30 con 0x7fb8e8153c80 2023-10-30T14:38:21.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.045+0000 7fb8ece70640 1 -- 172.21.15.57:0/3348177850 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e40a8b90 con 0x7fb8e8153c80 2023-10-30T14:38:21.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.045+0000 7fb8ece70640 1 -- 172.21.15.57:0/3348177850 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e40b0440 con 0x7fb8e8153c80 2023-10-30T14:38:21.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.045+0000 7fb8f00d6640 1 -- 172.21.15.57:0/3348177850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e8153c80 msgr2=0x7fb8e815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:21.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.045+0000 7fb8f00d6640 1 --2- 172.21.15.57:0/3348177850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e8153c80 0x7fb8e815cf90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4009870 tx=0x7fb8e401ed30 comp rx=0 tx=0).stop 2023-10-30T14:38:21.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.045+0000 7fb8f00d6640 1 -- 172.21.15.57:0/3348177850 shutdown_connections 2023-10-30T14:38:21.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.045+0000 7fb8f00d6640 1 --2- 172.21.15.57:0/3348177850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e8153c80 0x7fb8e815cf90 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:21.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.045+0000 7fb8f00d6640 1 -- 172.21.15.57:0/3348177850 >> 172.21.15.57:0/3348177850 conn(0x7fb8e80715c0 msgr2=0x7fb8e80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:21.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.045+0000 7fb8f00d6640 1 -- 172.21.15.57:0/3348177850 shutdown_connections 2023-10-30T14:38:21.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.045+0000 7fb8f00d6640 1 -- 172.21.15.57:0/3348177850 wait complete. 2023-10-30T14:38:21.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.045+0000 7fb8f00d6640 1 Processor -- start 2023-10-30T14:38:21.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.045+0000 7fb8f00d6640 1 -- start start 2023-10-30T14:38:21.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8f00d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8e80c42a0 0x7fb8e8149ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:21.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8f00d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8e80c4a10 0x7fb8e8158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:21.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8f00d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e8153c80 0x7fb8e8158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:21.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8f00d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e80c9670 con 0x7fb8e80c4a10 2023-10-30T14:38:21.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8f00d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e80c85e0 con 0x7fb8e8153c80 2023-10-30T14:38:21.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8f00d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e80c89f0 con 0x7fb8e80c42a0 2023-10-30T14:38:21.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8ed671640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8e80c4a10 0x7fb8e8158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:21.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8ee673640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e8153c80 0x7fb8e8158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:21.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8ed671640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8e80c4a10 0x7fb8e8158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39112/0 (socket says 172.21.15.57:39112) 2023-10-30T14:38:21.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8ed671640 1 -- 172.21.15.57:0/2299270057 learned_addr learned my addr 172.21.15.57:0/2299270057 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:21.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8ede72640 1 --2- 172.21.15.57:0/2299270057 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8e80c42a0 0x7fb8e8149ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:21.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8ed671640 1 -- 172.21.15.57:0/2299270057 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8e80c42a0 msgr2=0x7fb8e8149ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:21.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8ed671640 1 --2- 172.21.15.57:0/2299270057 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8e80c42a0 0x7fb8e8149ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:21.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8ed671640 1 -- 172.21.15.57:0/2299270057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e8153c80 msgr2=0x7fb8e8158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:21.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8ed671640 1 --2- 172.21.15.57:0/2299270057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e8153c80 0x7fb8e8158da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:21.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8ed671640 1 -- 172.21.15.57:0/2299270057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e40094d0 con 0x7fb8e80c4a10 2023-10-30T14:38:21.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8ee673640 1 --2- 172.21.15.57:0/2299270057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e8153c80 0x7fb8e8158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:38:21.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8ed671640 1 --2- 172.21.15.57:0/2299270057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8e80c4a10 0x7fb8e8158860 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800c4d0 tx=0x7fb8d800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:21.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8de7fc640 1 -- 172.21.15.57:0/2299270057 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d80052d0 con 0x7fb8e80c4a10 2023-10-30T14:38:21.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8de7fc640 1 -- 172.21.15.57:0/2299270057 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d8005430 con 0x7fb8e80c4a10 2023-10-30T14:38:21.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8de7fc640 1 -- 172.21.15.57:0/2299270057 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d8012830 con 0x7fb8e80c4a10 2023-10-30T14:38:21.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8f00d6640 1 -- 172.21.15.57:0/2299270057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e80c8cd0 con 0x7fb8e80c4a10 2023-10-30T14:38:21.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8f00d6640 1 -- 172.21.15.57:0/2299270057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e80c9290 con 0x7fb8e80c4a10 2023-10-30T14:38:21.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8f00d6640 1 -- 172.21.15.57:0/2299270057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e80c4710 con 0x7fb8e80c4a10 2023-10-30T14:38:21.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.049+0000 7fb8de7fc640 1 -- 172.21.15.57:0/2299270057 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb8d8012990 con 0x7fb8e80c4a10 2023-10-30T14:38:21.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.053+0000 7fb8de7fc640 1 --2- 172.21.15.57:0/2299270057 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8cc0664a0 0x7fb8cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:21.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.053+0000 7fb8de7fc640 1 -- 172.21.15.57:0/2299270057 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7fb8d8083d20 con 0x7fb8e80c4a10 2023-10-30T14:38:21.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.053+0000 7fb8ede72640 1 --2- 172.21.15.57:0/2299270057 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8cc0664a0 0x7fb8cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:21.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.053+0000 7fb8ede72640 1 --2- 172.21.15.57:0/2299270057 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8cc0664a0 0x7fb8cc068960 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40b2000 tx=0x7fb8e4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:21.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.057+0000 7fb8de7fc640 1 -- 172.21.15.57:0/2299270057 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb8e80c4710 con 0x7fb8e80c4a10 2023-10-30T14:38:21.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.245+0000 7fb8f00d6640 1 -- 172.21.15.57:0/2299270057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8e80c4710 con 0x7fb8e80c4a10 2023-10-30T14:38:21.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.245+0000 7fb8de7fc640 1 -- 172.21.15.57:0/2299270057 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+5743 (secure 0 0 0) 0x7fb8e80c4710 con 0x7fb8e80c4a10 2023-10-30T14:38:21.248 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:21.248 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:20.974621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:resolve","state_seq":35,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:21.249 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 129 2023-10-30T14:38:21.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.249+0000 7fb8f00d6640 1 -- 172.21.15.57:0/2299270057 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8cc0664a0 msgr2=0x7fb8cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:21.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.249+0000 7fb8f00d6640 1 --2- 172.21.15.57:0/2299270057 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8cc0664a0 0x7fb8cc068960 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40b2000 tx=0x7fb8e4008040 comp rx=0 tx=0).stop 2023-10-30T14:38:21.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.249+0000 7fb8f00d6640 1 -- 172.21.15.57:0/2299270057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8e80c4a10 msgr2=0x7fb8e8158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:21.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.249+0000 7fb8f00d6640 1 --2- 172.21.15.57:0/2299270057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8e80c4a10 0x7fb8e8158860 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800c4d0 tx=0x7fb8d800c990 comp rx=0 tx=0).stop 2023-10-30T14:38:21.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.249+0000 7fb8f00d6640 1 -- 172.21.15.57:0/2299270057 shutdown_connections 2023-10-30T14:38:21.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.249+0000 7fb8f00d6640 1 --2- 172.21.15.57:0/2299270057 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb8cc0664a0 0x7fb8cc068960 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:21.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.249+0000 7fb8f00d6640 1 --2- 172.21.15.57:0/2299270057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e8153c80 0x7fb8e8158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:21.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.249+0000 7fb8f00d6640 1 --2- 172.21.15.57:0/2299270057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8e80c4a10 0x7fb8e8158860 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:21.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.249+0000 7fb8f00d6640 1 --2- 172.21.15.57:0/2299270057 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8e80c42a0 0x7fb8e8149ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:21.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.249+0000 7fb8f00d6640 1 -- 172.21.15.57:0/2299270057 >> 172.21.15.57:0/2299270057 conn(0x7fb8e80715c0 msgr2=0x7fb8e80cb3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:21.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.249+0000 7fb8f00d6640 1 -- 172.21.15.57:0/2299270057 shutdown_connections 2023-10-30T14:38:21.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:21.249+0000 7fb8f00d6640 1 -- 172.21.15.57:0/2299270057 wait complete. 2023-10-30T14:38:21.269 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:38:23.270 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:23.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.425+0000 7f5dcc66b640 1 Processor -- start 2023-10-30T14:38:23.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.425+0000 7f5dcc66b640 1 -- start start 2023-10-30T14:38:23.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.425+0000 7f5dcc66b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5dc40c4a10 0x7f5dc415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:23.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.425+0000 7f5dcc66b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5dc40c42a0 0x7f5dc40ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:23.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.425+0000 7f5dcc66b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc4159f20 con 0x7f5dc4153c80 2023-10-30T14:38:23.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.425+0000 7f5dcc66b640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc415dec0 con 0x7f5dc40c42a0 2023-10-30T14:38:23.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.425+0000 7f5dcc66b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc41599a0 con 0x7f5dc40c4a10 2023-10-30T14:38:23.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.425+0000 7f5dca407640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f5dc4153c80 0x7f5dc415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50598/0 (socket says 172.21.15.57:50598) 2023-10-30T14:38:23.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.425+0000 7f5dca407640 1 -- 172.21.15.57:0/3840965933 learned_addr learned my addr 172.21.15.57:0/3840965933 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:38:23.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.425+0000 7f5dcac08640 1 --2- 172.21.15.57:0/3840965933 >> v2:172.21.15.57:3300/0 conn(0x7f5dc40c42a0 0x7f5dc40ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:23.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.425+0000 7f5dc9c06640 1 --2- 172.21.15.57:0/3840965933 >> v2:172.21.15.88:3300/0 conn(0x7f5dc40c4a10 0x7f5dc415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:23.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.425+0000 7f5dcac08640 1 -- 172.21.15.57:0/3840965933 >> v2:172.21.15.88:3300/0 conn(0x7f5dc40c4a10 msgr2=0x7f5dc415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:23.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.425+0000 7f5dcac08640 1 --2- 172.21.15.57:0/3840965933 >> v2:172.21.15.88:3300/0 conn(0x7f5dc40c4a10 0x7f5dc415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:23.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.425+0000 7f5dcac08640 1 -- 172.21.15.57:0/3840965933 >> v1:172.21.15.57:6789/0 conn(0x7f5dc4153c80 legacy=0x7f5dc415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:23.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.425+0000 7f5dcac08640 1 -- 172.21.15.57:0/3840965933 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc415d620 con 0x7f5dc40c42a0 2023-10-30T14:38:23.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.429+0000 7f5dcac08640 1 --2- 172.21.15.57:0/3840965933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dc40c42a0 0x7f5dc40ca1e0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f5dc406be50 tx=0x7f5dbc01f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5941c06672a86e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:23.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.429+0000 7f5dc9405640 1 -- 172.21.15.57:0/3840965933 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc0b0440 con 0x7f5dc40c42a0 2023-10-30T14:38:23.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.429+0000 7f5dc9405640 1 -- 172.21.15.57:0/3840965933 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dbc0b05a0 con 0x7f5dc40c42a0 2023-10-30T14:38:23.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.429+0000 7f5dc9405640 1 -- 172.21.15.57:0/3840965933 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc0b0930 con 0x7f5dc40c42a0 2023-10-30T14:38:23.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.429+0000 7f5dcc66b640 1 -- 172.21.15.57:0/3840965933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dc40c42a0 msgr2=0x7f5dc40ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:23.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.429+0000 7f5dcc66b640 1 --2- 172.21.15.57:0/3840965933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dc40c42a0 0x7f5dc40ca1e0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f5dc406be50 tx=0x7f5dbc01f0f0 comp rx=0 tx=0).stop 2023-10-30T14:38:23.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.429+0000 7f5dcc66b640 1 -- 172.21.15.57:0/3840965933 shutdown_connections 2023-10-30T14:38:23.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.429+0000 7f5dcc66b640 1 --2- 172.21.15.57:0/3840965933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dc40c42a0 0x7f5dc40ca1e0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:23.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.429+0000 7f5dcc66b640 1 --2- 172.21.15.57:0/3840965933 >> v2:172.21.15.88:3300/0 conn(0x7f5dc40c4a10 0x7f5dc415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:23.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.429+0000 7f5dcc66b640 1 -- 172.21.15.57:0/3840965933 >> 172.21.15.57:0/3840965933 conn(0x7f5dc40715c0 msgr2=0x7f5dc40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:23.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.429+0000 7f5dcc66b640 1 -- 172.21.15.57:0/3840965933 shutdown_connections 2023-10-30T14:38:23.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.429+0000 7f5dcc66b640 1 -- 172.21.15.57:0/3840965933 wait complete. 2023-10-30T14:38:23.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dcc66b640 1 Processor -- start 2023-10-30T14:38:23.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dcc66b640 1 -- start start 2023-10-30T14:38:23.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dcc66b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dc40c42a0 0x7f5dc41f2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:23.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dcc66b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5dc40c4a10 0x7f5dc40ce000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:23.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dcc66b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5dc4153c80 0x7f5dc40ce540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:23.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dcc66b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc41f2f80 con 0x7f5dc40c42a0 2023-10-30T14:38:23.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dcc66b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc41f2ca0 con 0x7f5dc4153c80 2023-10-30T14:38:23.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dcc66b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc40cfa30 con 0x7f5dc40c4a10 2023-10-30T14:38:23.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dca407640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dc40c42a0 0x7f5dc41f2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:23.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dca407640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dc40c42a0 0x7f5dc41f2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39134/0 (socket says 172.21.15.57:39134) 2023-10-30T14:38:23.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dca407640 1 -- 172.21.15.57:0/691898024 learned_addr learned my addr 172.21.15.57:0/691898024 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:23.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dcac08640 1 --2- 172.21.15.57:0/691898024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5dc4153c80 0x7f5dc40ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:23.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dc9c06640 1 --2- 172.21.15.57:0/691898024 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5dc40c4a10 0x7f5dc40ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:23.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dca407640 1 -- 172.21.15.57:0/691898024 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5dc40c4a10 msgr2=0x7f5dc40ce000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:23.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dca407640 1 --2- 172.21.15.57:0/691898024 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5dc40c4a10 0x7f5dc40ce000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:23.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dca407640 1 -- 172.21.15.57:0/691898024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5dc4153c80 msgr2=0x7f5dc40ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:23.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dca407640 1 --2- 172.21.15.57:0/691898024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5dc4153c80 0x7f5dc40ce540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:23.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dca407640 1 -- 172.21.15.57:0/691898024 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dbc009a70 con 0x7f5dc40c42a0 2023-10-30T14:38:23.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dca407640 1 --2- 172.21.15.57:0/691898024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dc40c42a0 0x7f5dc41f2240 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f5dc000c400 tx=0x7f5dc000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:23.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dbaffd640 1 -- 172.21.15.57:0/691898024 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0005cf0 con 0x7f5dc40c42a0 2023-10-30T14:38:23.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dbaffd640 1 -- 172.21.15.57:0/691898024 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc0005e50 con 0x7f5dc40c42a0 2023-10-30T14:38:23.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dbaffd640 1 -- 172.21.15.57:0/691898024 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc00072d0 con 0x7f5dc40c42a0 2023-10-30T14:38:23.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dcc66b640 1 -- 172.21.15.57:0/691898024 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc40cfd10 con 0x7f5dc40c42a0 2023-10-30T14:38:23.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dcc66b640 1 -- 172.21.15.57:0/691898024 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc40cdae0 con 0x7f5dc40c42a0 2023-10-30T14:38:23.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.433+0000 7f5dcc66b640 1 -- 172.21.15.57:0/691898024 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc404e170 con 0x7f5dc40c42a0 2023-10-30T14:38:23.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.437+0000 7f5dbaffd640 1 -- 172.21.15.57:0/691898024 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5dc0007430 con 0x7f5dc40c42a0 2023-10-30T14:38:23.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.437+0000 7f5dbaffd640 1 --2- 172.21.15.57:0/691898024 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5da80664a0 0x7f5da8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:23.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.437+0000 7f5dbaffd640 1 -- 172.21.15.57:0/691898024 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7f5dc0083910 con 0x7f5dc40c42a0 2023-10-30T14:38:23.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.437+0000 7f5dc9c06640 1 --2- 172.21.15.57:0/691898024 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5da80664a0 0x7f5da8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:23.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.437+0000 7f5dc9c06640 1 --2- 172.21.15.57:0/691898024 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5da80664a0 0x7f5da8068960 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40cbb20 tx=0x7f5db4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:23.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.441+0000 7f5dbaffd640 1 -- 172.21.15.57:0/691898024 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5dc0050840 con 0x7f5dc40c42a0 2023-10-30T14:38:23.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.633+0000 7f5dcc66b640 1 -- 172.21.15.57:0/691898024 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5dc40cdcd0 con 0x7f5dc40c42a0 2023-10-30T14:38:23.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.633+0000 7f5dbaffd640 1 -- 172.21.15.57:0/691898024 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+5743 (secure 0 0 0) 0x7f5dc0017090 con 0x7f5dc40c42a0 2023-10-30T14:38:23.635 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:23.636 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:20.974621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:resolve","state_seq":35,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:23.636 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 129 2023-10-30T14:38:23.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.637+0000 7f5dcc66b640 1 -- 172.21.15.57:0/691898024 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5da80664a0 msgr2=0x7f5da8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:23.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.637+0000 7f5dcc66b640 1 --2- 172.21.15.57:0/691898024 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5da80664a0 0x7f5da8068960 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40cbb20 tx=0x7f5db4008040 comp rx=0 tx=0).stop 2023-10-30T14:38:23.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.637+0000 7f5dcc66b640 1 -- 172.21.15.57:0/691898024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dc40c42a0 msgr2=0x7f5dc41f2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:23.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.637+0000 7f5dcc66b640 1 --2- 172.21.15.57:0/691898024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dc40c42a0 0x7f5dc41f2240 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f5dc000c400 tx=0x7f5dc000c8c0 comp rx=0 tx=0).stop 2023-10-30T14:38:23.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.637+0000 7f5dcc66b640 1 -- 172.21.15.57:0/691898024 shutdown_connections 2023-10-30T14:38:23.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.637+0000 7f5dcc66b640 1 --2- 172.21.15.57:0/691898024 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5da80664a0 0x7f5da8068960 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:23.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.637+0000 7f5dcc66b640 1 --2- 172.21.15.57:0/691898024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5dc4153c80 0x7f5dc40ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:23.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.637+0000 7f5dcc66b640 1 --2- 172.21.15.57:0/691898024 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5dc40c4a10 0x7f5dc40ce000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:23.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.637+0000 7f5dcc66b640 1 --2- 172.21.15.57:0/691898024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dc40c42a0 0x7f5dc41f2240 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:23.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.637+0000 7f5dcc66b640 1 -- 172.21.15.57:0/691898024 >> 172.21.15.57:0/691898024 conn(0x7f5dc40715c0 msgr2=0x7f5dc4151450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:23.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.637+0000 7f5dcc66b640 1 -- 172.21.15.57:0/691898024 shutdown_connections 2023-10-30T14:38:23.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:23.637+0000 7f5dcc66b640 1 -- 172.21.15.57:0/691898024 wait complete. 2023-10-30T14:38:23.656 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:38:25.657 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:25.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.809+0000 7f2acfad6640 1 Processor -- start 2023-10-30T14:38:25.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.809+0000 7f2acfad6640 1 -- start start 2023-10-30T14:38:25.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.813+0000 7f2acfad6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2ac806b760 0x7f2ac815e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:25.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.813+0000 7f2acfad6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2ac806c420 0x7f2ac80c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:25.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.813+0000 7f2acfad6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac8071e00 con 0x7f2ac806bb40 2023-10-30T14:38:25.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.813+0000 7f2acfad6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac8070020 con 0x7f2ac806b760 2023-10-30T14:38:25.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.813+0000 7f2acfad6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac815eb90 con 0x7f2ac806c420 2023-10-30T14:38:25.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.813+0000 7f2acd872640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2ac806b760 0x7f2ac815e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:25.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.813+0000 7f2acd872640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2ac806b760 0x7f2ac815e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57968/0 (socket says 172.21.15.57:57968) 2023-10-30T14:38:25.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.813+0000 7f2acd872640 1 -- 172.21.15.57:0/2978746595 learned_addr learned my addr 172.21.15.57:0/2978746595 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:25.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.813+0000 7f2ace073640 1 --2- 172.21.15.57:0/2978746595 >> v2:172.21.15.88:3300/0 conn(0x7f2ac806c420 0x7f2ac80c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:25.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.813+0000 7f2acd872640 1 -- 172.21.15.57:0/2978746595 >> v2:172.21.15.88:3300/0 conn(0x7f2ac806c420 msgr2=0x7f2ac80c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:25.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.813+0000 7f2acd872640 1 --2- 172.21.15.57:0/2978746595 >> v2:172.21.15.88:3300/0 conn(0x7f2ac806c420 0x7f2ac80c57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:25.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.813+0000 7f2acd872640 1 -- 172.21.15.57:0/2978746595 >> v1:172.21.15.57:6789/0 conn(0x7f2ac806bb40 legacy=0x7f2ac8157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:25.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.813+0000 7f2acd872640 1 -- 172.21.15.57:0/2978746595 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac8070f10 con 0x7f2ac806b760 2023-10-30T14:38:25.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.817+0000 7f2acd872640 1 --2- 172.21.15.57:0/2978746595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac806b760 0x7f2ac815e650 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f2ac815b7a0 tx=0x7f2ac401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=649dd72c5abd662b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:25.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.817+0000 7f2acc870640 1 -- 172.21.15.57:0/2978746595 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac40aa430 con 0x7f2ac806b760 2023-10-30T14:38:25.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.817+0000 7f2acc870640 1 -- 172.21.15.57:0/2978746595 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac40aa590 con 0x7f2ac806b760 2023-10-30T14:38:25.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.817+0000 7f2acc870640 1 -- 172.21.15.57:0/2978746595 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac40aa920 con 0x7f2ac806b760 2023-10-30T14:38:25.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.817+0000 7f2acfad6640 1 -- 172.21.15.57:0/2978746595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac806b760 msgr2=0x7f2ac815e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:25.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.817+0000 7f2acfad6640 1 --2- 172.21.15.57:0/2978746595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac806b760 0x7f2ac815e650 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f2ac815b7a0 tx=0x7f2ac401c4d0 comp rx=0 tx=0).stop 2023-10-30T14:38:25.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.817+0000 7f2acfad6640 1 -- 172.21.15.57:0/2978746595 shutdown_connections 2023-10-30T14:38:25.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.817+0000 7f2acfad6640 1 --2- 172.21.15.57:0/2978746595 >> v2:172.21.15.88:3300/0 conn(0x7f2ac806c420 0x7f2ac80c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:25.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.817+0000 7f2acfad6640 1 --2- 172.21.15.57:0/2978746595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac806b760 0x7f2ac815e650 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:25.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.817+0000 7f2acfad6640 1 -- 172.21.15.57:0/2978746595 >> 172.21.15.57:0/2978746595 conn(0x7f2ac80cac70 msgr2=0x7f2ac80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:25.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.817+0000 7f2acfad6640 1 -- 172.21.15.57:0/2978746595 shutdown_connections 2023-10-30T14:38:25.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.817+0000 7f2acfad6640 1 -- 172.21.15.57:0/2978746595 wait complete. 2023-10-30T14:38:25.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.817+0000 7f2acfad6640 1 Processor -- start 2023-10-30T14:38:25.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.817+0000 7f2acfad6640 1 -- start start 2023-10-30T14:38:25.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2acfad6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ac806b760 0x7f2ac80c2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:25.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2acfad6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ac806bb40 0x7f2ac814a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:25.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2acfad6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac806c420 0x7f2ac80c4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:25.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2acfad6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac80c9220 con 0x7f2ac806c420 2023-10-30T14:38:25.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2acfad6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac80c9380 con 0x7f2ac806b760 2023-10-30T14:38:25.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2acfad6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac80c3d50 con 0x7f2ac806bb40 2023-10-30T14:38:25.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2ace073640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac806c420 0x7f2ac80c4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:25.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2ace073640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac806c420 0x7f2ac80c4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57980/0 (socket says 172.21.15.57:57980) 2023-10-30T14:38:25.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2ace073640 1 -- 172.21.15.57:0/2450541933 learned_addr learned my addr 172.21.15.57:0/2450541933 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:25.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2acd071640 1 --2- 172.21.15.57:0/2450541933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ac806bb40 0x7f2ac814a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:25.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2acd872640 1 --2- 172.21.15.57:0/2450541933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ac806b760 0x7f2ac80c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:25.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2ace073640 1 -- 172.21.15.57:0/2450541933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ac806bb40 msgr2=0x7f2ac814a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:25.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2ace073640 1 --2- 172.21.15.57:0/2450541933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ac806bb40 0x7f2ac814a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:25.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2ace073640 1 -- 172.21.15.57:0/2450541933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ac806b760 msgr2=0x7f2ac80c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:25.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2ace073640 1 --2- 172.21.15.57:0/2450541933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ac806b760 0x7f2ac80c2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:25.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2ace073640 1 -- 172.21.15.57:0/2450541933 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac40094d0 con 0x7f2ac806c420 2023-10-30T14:38:25.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2acd872640 1 --2- 172.21.15.57:0/2450541933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ac806b760 0x7f2ac80c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:38:25.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2ace073640 1 --2- 172.21.15.57:0/2450541933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac806c420 0x7f2ac80c4360 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0007e90 tx=0x7f2ac000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:25.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2abe7fc640 1 -- 172.21.15.57:0/2450541933 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0014420 con 0x7f2ac806c420 2023-10-30T14:38:25.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2abe7fc640 1 -- 172.21.15.57:0/2450541933 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac0014580 con 0x7f2ac806c420 2023-10-30T14:38:25.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2acfad6640 1 -- 172.21.15.57:0/2450541933 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac80c8c60 con 0x7f2ac806c420 2023-10-30T14:38:25.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2abe7fc640 1 -- 172.21.15.57:0/2450541933 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0014930 con 0x7f2ac806c420 2023-10-30T14:38:25.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2acfad6640 1 -- 172.21.15.57:0/2450541933 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac80caa20 con 0x7f2ac806c420 2023-10-30T14:38:25.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.821+0000 7f2acfad6640 1 -- 172.21.15.57:0/2450541933 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac80c4060 con 0x7f2ac806c420 2023-10-30T14:38:25.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.825+0000 7f2abe7fc640 1 -- 172.21.15.57:0/2450541933 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2ac0014a90 con 0x7f2ac806c420 2023-10-30T14:38:25.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.825+0000 7f2abe7fc640 1 --2- 172.21.15.57:0/2450541933 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2aac0664a0 0x7f2aac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:25.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.825+0000 7f2abe7fc640 1 -- 172.21.15.57:0/2450541933 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7f2ac0084c60 con 0x7f2ac806c420 2023-10-30T14:38:25.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.825+0000 7f2acd872640 1 --2- 172.21.15.57:0/2450541933 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2aac0664a0 0x7f2aac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:25.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.825+0000 7f2acd872640 1 --2- 172.21.15.57:0/2450541933 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2aac0664a0 0x7f2aac068960 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2ac40ae000 tx=0x7f2ac40a7000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:25.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:25.829+0000 7f2abe7fc640 1 -- 172.21.15.57:0/2450541933 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2ac0051b70 con 0x7f2ac806c420 2023-10-30T14:38:26.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:26.021+0000 7f2acfad6640 1 -- 172.21.15.57:0/2450541933 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ac804e170 con 0x7f2ac806c420 2023-10-30T14:38:26.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:26.021+0000 7f2abe7fc640 1 -- 172.21.15.57:0/2450541933 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+5743 (secure 0 0 0) 0x7f2ac0055190 con 0x7f2ac806c420 2023-10-30T14:38:26.022 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:26.022 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:20.974621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:resolve","state_seq":35,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:26.023 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 129 2023-10-30T14:38:26.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:26.021+0000 7f2acfad6640 1 -- 172.21.15.57:0/2450541933 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2aac0664a0 msgr2=0x7f2aac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:26.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:26.021+0000 7f2acfad6640 1 --2- 172.21.15.57:0/2450541933 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2aac0664a0 0x7f2aac068960 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2ac40ae000 tx=0x7f2ac40a7000 comp rx=0 tx=0).stop 2023-10-30T14:38:26.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:26.021+0000 7f2acfad6640 1 -- 172.21.15.57:0/2450541933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac806c420 msgr2=0x7f2ac80c4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:26.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:26.021+0000 7f2acfad6640 1 --2- 172.21.15.57:0/2450541933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac806c420 0x7f2ac80c4360 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0007e90 tx=0x7f2ac000e4b0 comp rx=0 tx=0).stop 2023-10-30T14:38:26.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:26.021+0000 7f2acfad6640 1 -- 172.21.15.57:0/2450541933 shutdown_connections 2023-10-30T14:38:26.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:26.021+0000 7f2acfad6640 1 --2- 172.21.15.57:0/2450541933 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2aac0664a0 0x7f2aac068960 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:26.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:26.021+0000 7f2acfad6640 1 --2- 172.21.15.57:0/2450541933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac806c420 0x7f2ac80c4360 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:26.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:26.021+0000 7f2acfad6640 1 --2- 172.21.15.57:0/2450541933 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ac806bb40 0x7f2ac814a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:26.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:26.021+0000 7f2acfad6640 1 --2- 172.21.15.57:0/2450541933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ac806b760 0x7f2ac80c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:26.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:26.021+0000 7f2acfad6640 1 -- 172.21.15.57:0/2450541933 >> 172.21.15.57:0/2450541933 conn(0x7f2ac80cac70 msgr2=0x7f2ac8152520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:26.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:26.021+0000 7f2acfad6640 1 -- 172.21.15.57:0/2450541933 shutdown_connections 2023-10-30T14:38:26.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:26.021+0000 7f2acfad6640 1 -- 172.21.15.57:0/2450541933 wait complete. 2023-10-30T14:38:26.043 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:38:28.043 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:28.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.197+0000 7f6bcbd0e640 1 Processor -- start 2023-10-30T14:38:28.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.197+0000 7f6bcbd0e640 1 -- start start 2023-10-30T14:38:28.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.197+0000 7f6bcbd0e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bc4151c20 0x7f6bc40c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:28.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.197+0000 7f6bcbd0e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc40c5030 con 0x7f6bc40cb8b0 2023-10-30T14:38:28.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.197+0000 7f6bcbd0e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc4157830 con 0x7f6bc4151840 2023-10-30T14:38:28.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.197+0000 7f6bcbd0e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc4157b10 con 0x7f6bc4151c20 2023-10-30T14:38:28.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.197+0000 7f6bca2ab640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f6bc40cb8b0 0x7f6bc406b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46016/0 (socket says 172.21.15.57:46016) 2023-10-30T14:38:28.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.197+0000 7f6bca2ab640 1 -- 172.21.15.57:0/3120115461 learned_addr learned my addr 172.21.15.57:0/3120115461 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:38:28.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.197+0000 7f6bc92a9640 1 --2- 172.21.15.57:0/3120115461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bc4151c20 0x7f6bc40c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:28.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.197+0000 7f6bc8aa8640 1 -- 172.21.15.57:0/3120115461 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195894430 0 0) 0x7f6bc40c5030 con 0x7f6bc40cb8b0 2023-10-30T14:38:28.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.197+0000 7f6bc8aa8640 1 -- 172.21.15.57:0/3120115461 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bb0003530 con 0x7f6bc40cb8b0 2023-10-30T14:38:28.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.197+0000 7f6bc8aa8640 1 -- 172.21.15.57:0/3120115461 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905737300 0 0) 0x7f6bc4157830 con 0x7f6bc4151840 2023-10-30T14:38:28.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.197+0000 7f6bc8aa8640 1 -- 172.21.15.57:0/3120115461 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bc40c5030 con 0x7f6bc4151840 2023-10-30T14:38:28.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.197+0000 7f6bc92a9640 1 -- 172.21.15.57:0/3120115461 >> v1:172.21.15.88:6789/0 conn(0x7f6bc4151840 legacy=0x7f6bc4072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:28.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.197+0000 7f6bc92a9640 1 -- 172.21.15.57:0/3120115461 >> v1:172.21.15.57:6789/0 conn(0x7f6bc40cb8b0 legacy=0x7f6bc406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:28.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.197+0000 7f6bc8aa8640 1 -- 172.21.15.57:0/3120115461 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3298002625 0 0) 0x7f6bb0003530 con 0x7f6bc40cb8b0 2023-10-30T14:38:28.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.197+0000 7f6bc92a9640 1 -- 172.21.15.57:0/3120115461 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc4157df0 con 0x7f6bc4151c20 2023-10-30T14:38:28.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.201+0000 7f6bc92a9640 1 --2- 172.21.15.57:0/3120115461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bc4151c20 0x7f6bc40c49a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6bb40097f0 tx=0x7f6bb401ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef6b814f3842958e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:28.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.201+0000 7f6bc8aa8640 1 -- 172.21.15.57:0/3120115461 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc001a00 con 0x7f6bc4151c20 2023-10-30T14:38:28.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.201+0000 7f6bc8aa8640 1 -- 172.21.15.57:0/3120115461 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb40a8650 con 0x7f6bc4151c20 2023-10-30T14:38:28.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.201+0000 7f6bc8aa8640 1 -- 172.21.15.57:0/3120115461 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb40a89c0 con 0x7f6bc4151c20 2023-10-30T14:38:28.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.201+0000 7f6bcbd0e640 1 -- 172.21.15.57:0/3120115461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bc4151c20 msgr2=0x7f6bc40c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:28.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.201+0000 7f6bcbd0e640 1 --2- 172.21.15.57:0/3120115461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bc4151c20 0x7f6bc40c49a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6bb40097f0 tx=0x7f6bb401ebd0 comp rx=0 tx=0).stop 2023-10-30T14:38:28.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.201+0000 7f6bcbd0e640 1 -- 172.21.15.57:0/3120115461 shutdown_connections 2023-10-30T14:38:28.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.201+0000 7f6bcbd0e640 1 --2- 172.21.15.57:0/3120115461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bc4151c20 0x7f6bc40c49a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:28.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.201+0000 7f6bcbd0e640 1 -- 172.21.15.57:0/3120115461 >> 172.21.15.57:0/3120115461 conn(0x7f6bc40cac70 msgr2=0x7f6bc40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:28.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.201+0000 7f6bcbd0e640 1 -- 172.21.15.57:0/3120115461 shutdown_connections 2023-10-30T14:38:28.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.201+0000 7f6bcbd0e640 1 -- 172.21.15.57:0/3120115461 wait complete. 2023-10-30T14:38:28.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bcbd0e640 1 Processor -- start 2023-10-30T14:38:28.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bcbd0e640 1 -- start start 2023-10-30T14:38:28.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bcbd0e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bc40cb8b0 0x7f6bc41e44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:28.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bcbd0e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bc4151840 0x7f6bc406bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:28.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bcbd0e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bc4151c20 0x7f6bc41e1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:28.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bcbd0e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc41f3000 con 0x7f6bc40cb8b0 2023-10-30T14:38:28.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bcbd0e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc41e49f0 con 0x7f6bc4151840 2023-10-30T14:38:28.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bcbd0e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc414f9e0 con 0x7f6bc4151c20 2023-10-30T14:38:28.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bc9aaa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bc40cb8b0 0x7f6bc41e44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:28.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bc9aaa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bc40cb8b0 0x7f6bc41e44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57982/0 (socket says 172.21.15.57:57982) 2023-10-30T14:38:28.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bc9aaa640 1 -- 172.21.15.57:0/3518773105 learned_addr learned my addr 172.21.15.57:0/3518773105 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:28.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bc92a9640 1 --2- 172.21.15.57:0/3518773105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bc4151840 0x7f6bc406bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:28.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bca2ab640 1 --2- 172.21.15.57:0/3518773105 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bc4151c20 0x7f6bc41e1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:28.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bc9aaa640 1 -- 172.21.15.57:0/3518773105 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bc4151c20 msgr2=0x7f6bc41e1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:28.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bc9aaa640 1 --2- 172.21.15.57:0/3518773105 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bc4151c20 0x7f6bc41e1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:28.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bc9aaa640 1 -- 172.21.15.57:0/3518773105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bc4151840 msgr2=0x7f6bc406bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:28.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bc9aaa640 1 --2- 172.21.15.57:0/3518773105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bc4151840 0x7f6bc406bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:28.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bc9aaa640 1 -- 172.21.15.57:0/3518773105 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb4009420 con 0x7f6bc40cb8b0 2023-10-30T14:38:28.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bc9aaa640 1 --2- 172.21.15.57:0/3518773105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bc40cb8b0 0x7f6bc41e44b0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f6bc000c4c0 tx=0x7f6bc000c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:28.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bba7fc640 1 -- 172.21.15.57:0/3518773105 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc00052d0 con 0x7f6bc40cb8b0 2023-10-30T14:38:28.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bba7fc640 1 -- 172.21.15.57:0/3518773105 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc0005430 con 0x7f6bc40cb8b0 2023-10-30T14:38:28.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bba7fc640 1 -- 172.21.15.57:0/3518773105 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc0012870 con 0x7f6bc40cb8b0 2023-10-30T14:38:28.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bcbd0e640 1 -- 172.21.15.57:0/3518773105 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc41f2380 con 0x7f6bc40cb8b0 2023-10-30T14:38:28.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.205+0000 7f6bcbd0e640 1 -- 172.21.15.57:0/3518773105 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc41f2940 con 0x7f6bc40cb8b0 2023-10-30T14:38:28.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.209+0000 7f6bcbd0e640 1 -- 172.21.15.57:0/3518773105 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc41f3140 con 0x7f6bc40cb8b0 2023-10-30T14:38:28.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.209+0000 7f6bba7fc640 1 -- 172.21.15.57:0/3518773105 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6bc00129d0 con 0x7f6bc40cb8b0 2023-10-30T14:38:28.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.209+0000 7f6bba7fc640 1 --2- 172.21.15.57:0/3518773105 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ba80664a0 0x7f6ba8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:28.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.209+0000 7f6bba7fc640 1 -- 172.21.15.57:0/3518773105 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7f6bc0083f10 con 0x7f6bc40cb8b0 2023-10-30T14:38:28.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.209+0000 7f6bc92a9640 1 --2- 172.21.15.57:0/3518773105 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ba80664a0 0x7f6ba8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:28.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.213+0000 7f6bc92a9640 1 --2- 172.21.15.57:0/3518773105 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ba80664a0 0x7f6ba8068960 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6bb40b2000 tx=0x7f6bb40aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:28.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.213+0000 7f6bba7fc640 1 -- 172.21.15.57:0/3518773105 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6bc0050db0 con 0x7f6bc40cb8b0 2023-10-30T14:38:28.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.409+0000 7f6bcbd0e640 1 -- 172.21.15.57:0/3518773105 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bc40726b0 con 0x7f6bc40cb8b0 2023-10-30T14:38:28.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.409+0000 7f6bba7fc640 1 -- 172.21.15.57:0/3518773105 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+5745 (secure 0 0 0) 0x7f6bc00543d0 con 0x7f6bc40cb8b0 2023-10-30T14:38:28.410 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:28.411 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:26.080956+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:28.411 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 130 2023-10-30T14:38:28.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.409+0000 7f6bcbd0e640 1 -- 172.21.15.57:0/3518773105 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ba80664a0 msgr2=0x7f6ba8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:28.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.409+0000 7f6bcbd0e640 1 --2- 172.21.15.57:0/3518773105 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ba80664a0 0x7f6ba8068960 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6bb40b2000 tx=0x7f6bb40aa070 comp rx=0 tx=0).stop 2023-10-30T14:38:28.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.409+0000 7f6bcbd0e640 1 -- 172.21.15.57:0/3518773105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bc40cb8b0 msgr2=0x7f6bc41e44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:28.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.409+0000 7f6bcbd0e640 1 --2- 172.21.15.57:0/3518773105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bc40cb8b0 0x7f6bc41e44b0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f6bc000c4c0 tx=0x7f6bc000c980 comp rx=0 tx=0).stop 2023-10-30T14:38:28.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.409+0000 7f6bcbd0e640 1 -- 172.21.15.57:0/3518773105 shutdown_connections 2023-10-30T14:38:28.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.409+0000 7f6bcbd0e640 1 --2- 172.21.15.57:0/3518773105 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ba80664a0 0x7f6ba8068960 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:28.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.409+0000 7f6bcbd0e640 1 --2- 172.21.15.57:0/3518773105 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bc4151c20 0x7f6bc41e1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:28.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.409+0000 7f6bcbd0e640 1 --2- 172.21.15.57:0/3518773105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bc4151840 0x7f6bc406bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:28.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.409+0000 7f6bcbd0e640 1 --2- 172.21.15.57:0/3518773105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bc40cb8b0 0x7f6bc41e44b0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:28.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.409+0000 7f6bcbd0e640 1 -- 172.21.15.57:0/3518773105 >> 172.21.15.57:0/3518773105 conn(0x7f6bc40cac70 msgr2=0x7f6bc40cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:28.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.409+0000 7f6bcbd0e640 1 -- 172.21.15.57:0/3518773105 shutdown_connections 2023-10-30T14:38:28.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:28.409+0000 7f6bcbd0e640 1 -- 172.21.15.57:0/3518773105 wait complete. 2023-10-30T14:38:28.431 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 130, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 130, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6125": { "addr": "172.21.15.88:6839/2390289388", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 2390289388, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2390289388, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6125, "incarnation": 128, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:reconnect", "state_seq": 37 }, "gid_6414": { "addr": "172.21.15.57:6834/2321408149", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2321408149, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2321408149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6414, "incarnation": 99, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 18 }, "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 44 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:38:26.080956+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6414, "mds_1": 6747, "mds_2": 6125 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:38:30.432 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:30.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.585+0000 7fb6ba33e640 1 Processor -- start 2023-10-30T14:38:30.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.585+0000 7fb6ba33e640 1 -- start start 2023-10-30T14:38:30.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6ba33e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6b4159ec0 0x7fb6b40c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:30.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6ba33e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b415cda0 con 0x7fb6b40c5f80 2023-10-30T14:38:30.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6ba33e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b406f490 con 0x7fb6b40c5ba0 2023-10-30T14:38:30.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6ba33e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b415d990 con 0x7fb6b4159ec0 2023-10-30T14:38:30.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b37fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb6b40c5f80 0x7fb6b41597b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46022/0 (socket says 172.21.15.57:46022) 2023-10-30T14:38:30.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b37fe640 1 -- 172.21.15.57:0/4036780054 learned_addr learned my addr 172.21.15.57:0/4036780054 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:38:30.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b88db640 1 --2- 172.21.15.57:0/4036780054 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6b4159ec0 0x7fb6b40c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:30.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b2ffd640 1 -- 172.21.15.57:0/4036780054 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632862772 0 0) 0x7fb6b415cda0 con 0x7fb6b40c5f80 2023-10-30T14:38:30.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b2ffd640 1 -- 172.21.15.57:0/4036780054 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb69c003530 con 0x7fb6b40c5f80 2023-10-30T14:38:30.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b2ffd640 1 -- 172.21.15.57:0/4036780054 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554223062 0 0) 0x7fb6b406f490 con 0x7fb6b40c5ba0 2023-10-30T14:38:30.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b2ffd640 1 -- 172.21.15.57:0/4036780054 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b415cda0 con 0x7fb6b40c5ba0 2023-10-30T14:38:30.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b2ffd640 1 -- 172.21.15.57:0/4036780054 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1417321250 0 0) 0x7fb69c003530 con 0x7fb6b40c5f80 2023-10-30T14:38:30.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b2ffd640 1 -- 172.21.15.57:0/4036780054 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6b4159ec0 msgr2=0x7fb6b40c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:30.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b2ffd640 1 --2- 172.21.15.57:0/4036780054 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6b4159ec0 0x7fb6b40c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:30.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b2ffd640 1 -- 172.21.15.57:0/4036780054 >> v1:172.21.15.88:6789/0 conn(0x7fb6b40c5ba0 legacy=0x7fb6b4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:30.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b2ffd640 1 -- 172.21.15.57:0/4036780054 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b415cff0 con 0x7fb6b40c5f80 2023-10-30T14:38:30.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b2ffd640 1 -- 172.21.15.57:0/4036780054 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fb6a8001680 con 0x7fb6b40c5f80 2023-10-30T14:38:30.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b2ffd640 1 -- 172.21.15.57:0/4036780054 >> v1:172.21.15.57:6789/0 conn(0x7fb6b40c5f80 legacy=0x7fb6b41597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:30.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b2ffd640 1 --2- 172.21.15.57:0/4036780054 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb69c003b40 0x7fb69c003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:30.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b2ffd640 1 --2- 172.21.15.57:0/4036780054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb69c0047d0 0x7fb69c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:30.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b2ffd640 1 --2- 172.21.15.57:0/4036780054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69c005190 0x7fb69c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:30.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b2ffd640 1 -- 172.21.15.57:0/4036780054 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b415d990 con 0x7fb69c005190 2023-10-30T14:38:30.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b2ffd640 1 -- 172.21.15.57:0/4036780054 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c00bed0 con 0x7fb69c0047d0 2023-10-30T14:38:30.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b2ffd640 1 -- 172.21.15.57:0/4036780054 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c00c010 con 0x7fb69c003b40 2023-10-30T14:38:30.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b88db640 1 --2- 172.21.15.57:0/4036780054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69c005190 0x7fb69c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:30.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b3fff640 1 --2- 172.21.15.57:0/4036780054 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb69c003b40 0x7fb69c003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:30.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b37fe640 1 --2- 172.21.15.57:0/4036780054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb69c0047d0 0x7fb69c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:30.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b88db640 1 -- 172.21.15.57:0/4036780054 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb69c003b40 msgr2=0x7fb69c003f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:30.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b88db640 1 --2- 172.21.15.57:0/4036780054 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb69c003b40 0x7fb69c003f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:30.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b88db640 1 -- 172.21.15.57:0/4036780054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb69c0047d0 msgr2=0x7fb69c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:30.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b88db640 1 --2- 172.21.15.57:0/4036780054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb69c0047d0 0x7fb69c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:30.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.589+0000 7fb6b88db640 1 -- 172.21.15.57:0/4036780054 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69c003530 con 0x7fb69c005190 2023-10-30T14:38:30.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.593+0000 7fb6b88db640 1 --2- 172.21.15.57:0/4036780054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69c005190 0x7fb69c00b990 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac009f90 tx=0x7fb6ac01cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1afeaa6c1f0c143 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:30.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.593+0000 7fb6b2ffd640 1 -- 172.21.15.57:0/4036780054 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8007ea0 con 0x7fb69c005190 2023-10-30T14:38:30.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.593+0000 7fb6b2ffd640 1 -- 172.21.15.57:0/4036780054 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6ac00f040 con 0x7fb69c005190 2023-10-30T14:38:30.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.593+0000 7fb6b2ffd640 1 -- 172.21.15.57:0/4036780054 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac0a5d60 con 0x7fb69c005190 2023-10-30T14:38:30.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.593+0000 7fb6ba33e640 1 -- 172.21.15.57:0/4036780054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69c005190 msgr2=0x7fb69c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:30.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.593+0000 7fb6ba33e640 1 --2- 172.21.15.57:0/4036780054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69c005190 0x7fb69c00b990 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac009f90 tx=0x7fb6ac01cc20 comp rx=0 tx=0).stop 2023-10-30T14:38:30.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.593+0000 7fb6b88db640 1 -- 172.21.15.57:0/4036780054 reap_dead start 2023-10-30T14:38:30.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.593+0000 7fb6ba33e640 1 -- 172.21.15.57:0/4036780054 shutdown_connections 2023-10-30T14:38:30.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.593+0000 7fb6ba33e640 1 -- 172.21.15.57:0/4036780054 >> 172.21.15.57:0/4036780054 conn(0x7fb6b406afa0 msgr2=0x7fb6b406b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:30.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.593+0000 7fb6ba33e640 1 -- 172.21.15.57:0/4036780054 shutdown_connections 2023-10-30T14:38:30.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.593+0000 7fb6ba33e640 1 -- 172.21.15.57:0/4036780054 wait complete. 2023-10-30T14:38:30.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.593+0000 7fb6ba33e640 1 Processor -- start 2023-10-30T14:38:30.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6ba33e640 1 -- start start 2023-10-30T14:38:30.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6ba33e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69c005190 0x7fb6b40ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:30.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6ba33e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6b40b17c0 0x7fb6b415c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:30.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6ba33e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6b41f1dd0 0x7fb6b41f2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:30.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6ba33e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b41f2790 con 0x7fb69c005190 2023-10-30T14:38:30.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6ba33e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b415a3a0 con 0x7fb6b40b17c0 2023-10-30T14:38:30.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6ba33e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b415a0c0 con 0x7fb6b41f1dd0 2023-10-30T14:38:30.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6b3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69c005190 0x7fb6b40ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:30.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6b3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69c005190 0x7fb6b40ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57996/0 (socket says 172.21.15.57:57996) 2023-10-30T14:38:30.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6b3fff640 1 -- 172.21.15.57:0/2718477580 learned_addr learned my addr 172.21.15.57:0/2718477580 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:30.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6b88db640 1 --2- 172.21.15.57:0/2718477580 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6b41f1dd0 0x7fb6b41f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:30.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6b37fe640 1 --2- 172.21.15.57:0/2718477580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6b40b17c0 0x7fb6b415c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:30.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6b3fff640 1 -- 172.21.15.57:0/2718477580 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6b41f1dd0 msgr2=0x7fb6b41f2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:30.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6b3fff640 1 --2- 172.21.15.57:0/2718477580 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6b41f1dd0 0x7fb6b41f2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:30.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6b3fff640 1 -- 172.21.15.57:0/2718477580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6b40b17c0 msgr2=0x7fb6b415c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:30.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6b3fff640 1 --2- 172.21.15.57:0/2718477580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6b40b17c0 0x7fb6b415c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:30.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6b3fff640 1 -- 172.21.15.57:0/2718477580 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b40ae2c0 con 0x7fb69c005190 2023-10-30T14:38:30.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6b37fe640 1 --2- 172.21.15.57:0/2718477580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6b40b17c0 0x7fb6b415c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:38:30.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6b88db640 1 --2- 172.21.15.57:0/2718477580 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6b41f1dd0 0x7fb6b41f2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:38:30.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6b3fff640 1 --2- 172.21.15.57:0/2718477580 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69c005190 0x7fb6b40ad8b0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb6a400f770 tx=0x7fb6a400fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:30.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6b0ff9640 1 -- 172.21.15.57:0/2718477580 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a40014b0 con 0x7fb69c005190 2023-10-30T14:38:30.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6b0ff9640 1 -- 172.21.15.57:0/2718477580 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a400fe80 con 0x7fb69c005190 2023-10-30T14:38:30.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6ba33e640 1 -- 172.21.15.57:0/2718477580 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b40ae5a0 con 0x7fb69c005190 2023-10-30T14:38:30.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6b0ff9640 1 -- 172.21.15.57:0/2718477580 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a400d730 con 0x7fb69c005190 2023-10-30T14:38:30.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6ba33e640 1 -- 172.21.15.57:0/2718477580 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b40aeb60 con 0x7fb69c005190 2023-10-30T14:38:30.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.597+0000 7fb6ba33e640 1 -- 172.21.15.57:0/2718477580 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6b404e170 con 0x7fb69c005190 2023-10-30T14:38:30.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.601+0000 7fb6b0ff9640 1 -- 172.21.15.57:0/2718477580 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb6a400d890 con 0x7fb69c005190 2023-10-30T14:38:30.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.601+0000 7fb6b0ff9640 1 --2- 172.21.15.57:0/2718477580 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb6a00664d0 0x7fb6a0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:30.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.601+0000 7fb6b0ff9640 1 -- 172.21.15.57:0/2718477580 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7fb6a4084500 con 0x7fb69c005190 2023-10-30T14:38:30.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.601+0000 7fb6b37fe640 1 --2- 172.21.15.57:0/2718477580 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb6a00664d0 0x7fb6a0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:30.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.601+0000 7fb6b37fe640 1 --2- 172.21.15.57:0/2718477580 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb6a00664d0 0x7fb6a0068990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb6b406b530 tx=0x7fb6a8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:30.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.605+0000 7fb6b0ff9640 1 -- 172.21.15.57:0/2718477580 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb6a4051380 con 0x7fb69c005190 2023-10-30T14:38:30.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.797+0000 7fb6ba33e640 1 -- 172.21.15.57:0/2718477580 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6b40aed20 con 0x7fb69c005190 2023-10-30T14:38:30.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.797+0000 7fb6b0ff9640 1 -- 172.21.15.57:0/2718477580 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+5745 (secure 0 0 0) 0x7fb6a40549a0 con 0x7fb69c005190 2023-10-30T14:38:30.799 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:30.799 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:26.080956+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:30.800 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 130 2023-10-30T14:38:30.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.797+0000 7fb6ba33e640 1 -- 172.21.15.57:0/2718477580 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb6a00664d0 msgr2=0x7fb6a0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:30.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.797+0000 7fb6ba33e640 1 --2- 172.21.15.57:0/2718477580 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb6a00664d0 0x7fb6a0068990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb6b406b530 tx=0x7fb6a8009040 comp rx=0 tx=0).stop 2023-10-30T14:38:30.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.797+0000 7fb6ba33e640 1 -- 172.21.15.57:0/2718477580 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69c005190 msgr2=0x7fb6b40ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:30.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.797+0000 7fb6ba33e640 1 --2- 172.21.15.57:0/2718477580 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69c005190 0x7fb6b40ad8b0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb6a400f770 tx=0x7fb6a400fc30 comp rx=0 tx=0).stop 2023-10-30T14:38:30.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.797+0000 7fb6ba33e640 1 -- 172.21.15.57:0/2718477580 shutdown_connections 2023-10-30T14:38:30.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.797+0000 7fb6ba33e640 1 --2- 172.21.15.57:0/2718477580 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb6a00664d0 0x7fb6a0068990 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:30.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.797+0000 7fb6ba33e640 1 --2- 172.21.15.57:0/2718477580 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6b41f1dd0 0x7fb6b41f2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:30.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.797+0000 7fb6ba33e640 1 --2- 172.21.15.57:0/2718477580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6b40b17c0 0x7fb6b415c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:30.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.797+0000 7fb6ba33e640 1 --2- 172.21.15.57:0/2718477580 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69c005190 0x7fb6b40ad8b0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:30.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.797+0000 7fb6ba33e640 1 -- 172.21.15.57:0/2718477580 >> 172.21.15.57:0/2718477580 conn(0x7fb6b406afa0 msgr2=0x7fb6b40c69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:30.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.797+0000 7fb6ba33e640 1 -- 172.21.15.57:0/2718477580 shutdown_connections 2023-10-30T14:38:30.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:30.797+0000 7fb6ba33e640 1 -- 172.21.15.57:0/2718477580 wait complete. 2023-10-30T14:38:30.819 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:38:32.821 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:32.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.973+0000 7f609750f640 1 Processor -- start 2023-10-30T14:38:32.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.973+0000 7f609750f640 1 -- start start 2023-10-30T14:38:32.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.977+0000 7f609750f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60901537f0 0x7f60900701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:32.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.977+0000 7f609750f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f609006b2f0 0x7f609006b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:32.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.977+0000 7f609750f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60900c2aa0 con 0x7f609015a2e0 2023-10-30T14:38:32.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.977+0000 7f609750f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60900c3ab0 con 0x7f609006b2f0 2023-10-30T14:38:32.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.977+0000 7f609750f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60900706e0 con 0x7f60901537f0 2023-10-30T14:38:32.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.977+0000 7f6095aac640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f609006b2f0 0x7f609006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:32.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.977+0000 7f6095aac640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f609006b2f0 0x7f609006b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58002/0 (socket says 172.21.15.57:58002) 2023-10-30T14:38:32.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.977+0000 7f6095aac640 1 -- 172.21.15.57:0/3828583224 learned_addr learned my addr 172.21.15.57:0/3828583224 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:32.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.977+0000 7f60952ab640 1 --2- 172.21.15.57:0/3828583224 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60901537f0 0x7f60900701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:32.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.977+0000 7f6095aac640 1 -- 172.21.15.57:0/3828583224 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60901537f0 msgr2=0x7f60900701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:32.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.977+0000 7f6095aac640 1 --2- 172.21.15.57:0/3828583224 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60901537f0 0x7f60900701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:32.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.977+0000 7f6095aac640 1 -- 172.21.15.57:0/3828583224 >> v1:172.21.15.57:6789/0 conn(0x7f609015a2e0 legacy=0x7f60900ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:32.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.977+0000 7f6095aac640 1 -- 172.21.15.57:0/3828583224 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609006bca0 con 0x7f609006b2f0 2023-10-30T14:38:32.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.977+0000 7f60952ab640 1 --2- 172.21.15.57:0/3828583224 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60901537f0 0x7f60900701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:38:32.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.977+0000 7f6095aac640 1 --2- 172.21.15.57:0/3828583224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f609006b2f0 0x7f609006b760 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f6088009e50 tx=0x7f608801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa2a7a468cdd3e9f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:32.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f6087fff640 1 -- 172.21.15.57:0/3828583224 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60880af400 con 0x7f609006b2f0 2023-10-30T14:38:32.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f6087fff640 1 -- 172.21.15.57:0/3828583224 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60880ab050 con 0x7f609006b2f0 2023-10-30T14:38:32.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f6087fff640 1 -- 172.21.15.57:0/3828583224 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60880af780 con 0x7f609006b2f0 2023-10-30T14:38:32.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f609750f640 1 -- 172.21.15.57:0/3828583224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f609006b2f0 msgr2=0x7f609006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:32.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f609750f640 1 --2- 172.21.15.57:0/3828583224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f609006b2f0 0x7f609006b760 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f6088009e50 tx=0x7f608801f1f0 comp rx=0 tx=0).stop 2023-10-30T14:38:32.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f609750f640 1 -- 172.21.15.57:0/3828583224 shutdown_connections 2023-10-30T14:38:32.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f609750f640 1 --2- 172.21.15.57:0/3828583224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f609006b2f0 0x7f609006b760 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:32.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f609750f640 1 --2- 172.21.15.57:0/3828583224 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60901537f0 0x7f60900701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:32.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f609750f640 1 -- 172.21.15.57:0/3828583224 >> 172.21.15.57:0/3828583224 conn(0x7f6090071fb0 msgr2=0x7f60900723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:32.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f609750f640 1 -- 172.21.15.57:0/3828583224 shutdown_connections 2023-10-30T14:38:32.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f609750f640 1 -- 172.21.15.57:0/3828583224 wait complete. 2023-10-30T14:38:32.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f609750f640 1 Processor -- start 2023-10-30T14:38:32.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f609750f640 1 -- start start 2023-10-30T14:38:32.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f609750f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f609006b2f0 0x7f60901596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:32.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f609750f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60901537f0 0x7f609015d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:32.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f609750f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f609015a2e0 0x7f609015d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:32.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f609750f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60901f10e0 con 0x7f609006b2f0 2023-10-30T14:38:32.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f609750f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60901f0210 con 0x7f60901537f0 2023-10-30T14:38:32.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f609750f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60901f0460 con 0x7f609015a2e0 2023-10-30T14:38:32.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f60952ab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f609006b2f0 0x7f60901596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:32.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f60952ab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f609006b2f0 0x7f60901596c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58014/0 (socket says 172.21.15.57:58014) 2023-10-30T14:38:32.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f60952ab640 1 -- 172.21.15.57:0/2645231310 learned_addr learned my addr 172.21.15.57:0/2645231310 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:32.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f6095aac640 1 --2- 172.21.15.57:0/2645231310 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f609015a2e0 0x7f609015d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:32.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.981+0000 7f6094aaa640 1 --2- 172.21.15.57:0/2645231310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60901537f0 0x7f609015d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:32.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.985+0000 7f60952ab640 1 -- 172.21.15.57:0/2645231310 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f609015a2e0 msgr2=0x7f609015d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:32.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.985+0000 7f60952ab640 1 --2- 172.21.15.57:0/2645231310 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f609015a2e0 0x7f609015d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:32.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.985+0000 7f60952ab640 1 -- 172.21.15.57:0/2645231310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60901537f0 msgr2=0x7f609015d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:32.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.985+0000 7f60952ab640 1 --2- 172.21.15.57:0/2645231310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60901537f0 0x7f609015d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:32.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.985+0000 7f60952ab640 1 -- 172.21.15.57:0/2645231310 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6088009b30 con 0x7f609006b2f0 2023-10-30T14:38:32.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.985+0000 7f60952ab640 1 --2- 172.21.15.57:0/2645231310 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f609006b2f0 0x7f60901596c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f608c00e810 tx=0x7f608c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:32.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.985+0000 7f6085ffb640 1 -- 172.21.15.57:0/2645231310 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608c00c8d0 con 0x7f609006b2f0 2023-10-30T14:38:32.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.985+0000 7f6085ffb640 1 -- 172.21.15.57:0/2645231310 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f608c00ca30 con 0x7f609006b2f0 2023-10-30T14:38:32.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.985+0000 7f6085ffb640 1 -- 172.21.15.57:0/2645231310 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608c00cdc0 con 0x7f609006b2f0 2023-10-30T14:38:32.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.985+0000 7f609750f640 1 -- 172.21.15.57:0/2645231310 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60901f0740 con 0x7f609006b2f0 2023-10-30T14:38:32.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.985+0000 7f609750f640 1 -- 172.21.15.57:0/2645231310 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60901f0d00 con 0x7f609006b2f0 2023-10-30T14:38:32.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.985+0000 7f609750f640 1 -- 172.21.15.57:0/2645231310 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60901f1350 con 0x7f609006b2f0 2023-10-30T14:38:32.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.985+0000 7f6085ffb640 1 -- 172.21.15.57:0/2645231310 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f608c013410 con 0x7f609006b2f0 2023-10-30T14:38:32.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.989+0000 7f6085ffb640 1 --2- 172.21.15.57:0/2645231310 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f60740664a0 0x7f6074068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:32.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.989+0000 7f6085ffb640 1 -- 172.21.15.57:0/2645231310 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7f608c084140 con 0x7f609006b2f0 2023-10-30T14:38:32.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.989+0000 7f6094aaa640 1 --2- 172.21.15.57:0/2645231310 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f60740664a0 0x7f6074068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:32.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.989+0000 7f6094aaa640 1 --2- 172.21.15.57:0/2645231310 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f60740664a0 0x7f6074068960 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f609006bc70 tx=0x7f6080009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:32.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:32.993+0000 7f6085ffb640 1 -- 172.21.15.57:0/2645231310 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f608c0510d0 con 0x7f609006b2f0 2023-10-30T14:38:33.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:33.181+0000 7f609750f640 1 -- 172.21.15.57:0/2645231310 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60901f0e90 con 0x7f609006b2f0 2023-10-30T14:38:33.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:33.181+0000 7f6085ffb640 1 -- 172.21.15.57:0/2645231310 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+5742 (secure 0 0 0) 0x7f608c0546f0 con 0x7f609006b2f0 2023-10-30T14:38:33.183 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:33.184 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:32.087700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:33.184 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 131 2023-10-30T14:38:33.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:33.185+0000 7f609750f640 1 -- 172.21.15.57:0/2645231310 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f60740664a0 msgr2=0x7f6074068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:33.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:33.185+0000 7f609750f640 1 --2- 172.21.15.57:0/2645231310 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f60740664a0 0x7f6074068960 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f609006bc70 tx=0x7f6080009040 comp rx=0 tx=0).stop 2023-10-30T14:38:33.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:33.185+0000 7f609750f640 1 -- 172.21.15.57:0/2645231310 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f609006b2f0 msgr2=0x7f60901596c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:33.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:33.185+0000 7f609750f640 1 --2- 172.21.15.57:0/2645231310 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f609006b2f0 0x7f60901596c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f608c00e810 tx=0x7f608c00ecd0 comp rx=0 tx=0).stop 2023-10-30T14:38:33.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:33.185+0000 7f609750f640 1 -- 172.21.15.57:0/2645231310 shutdown_connections 2023-10-30T14:38:33.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:33.185+0000 7f609750f640 1 --2- 172.21.15.57:0/2645231310 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f60740664a0 0x7f6074068960 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:33.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:33.185+0000 7f609750f640 1 --2- 172.21.15.57:0/2645231310 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f609015a2e0 0x7f609015d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:33.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:33.185+0000 7f609750f640 1 --2- 172.21.15.57:0/2645231310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60901537f0 0x7f609015d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:33.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:33.185+0000 7f609750f640 1 --2- 172.21.15.57:0/2645231310 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f609006b2f0 0x7f60901596c0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:33.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:33.185+0000 7f609750f640 1 -- 172.21.15.57:0/2645231310 >> 172.21.15.57:0/2645231310 conn(0x7f6090071fb0 msgr2=0x7f609006c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:33.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:33.185+0000 7f609750f640 1 -- 172.21.15.57:0/2645231310 shutdown_connections 2023-10-30T14:38:33.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:33.185+0000 7f609750f640 1 -- 172.21.15.57:0/2645231310 wait complete. 2023-10-30T14:38:33.204 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:38:35.205 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:35.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.357+0000 7f4475f94640 1 Processor -- start 2023-10-30T14:38:35.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.357+0000 7f4475f94640 1 -- start start 2023-10-30T14:38:35.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.357+0000 7f4475f94640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4470150830 0x7f447006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:35.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.357+0000 7f4475f94640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44700cb7d0 0x7f44700c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:35.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.357+0000 7f4475f94640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f44700c3e40 0x7f4470070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:35.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.357+0000 7f4475f94640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44700c2310 con 0x7f44700c3e40 2023-10-30T14:38:35.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.357+0000 7f4475f94640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4470156710 con 0x7f4470150830 2023-10-30T14:38:35.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.357+0000 7f4475f94640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44700c4360 con 0x7f44700cb7d0 2023-10-30T14:38:35.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.361+0000 7f446ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f44700c3e40 0x7f4470070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:35.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.361+0000 7f446ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f44700c3e40 0x7f4470070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59010/0 (socket says 172.21.15.57:59010) 2023-10-30T14:38:35.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.361+0000 7f446ffff640 1 -- 172.21.15.57:0/924925808 learned_addr learned my addr 172.21.15.57:0/924925808 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:35.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.361+0000 7f446f7fe640 1 --2- 172.21.15.57:0/924925808 >> v2:172.21.15.88:3300/0 conn(0x7f4470150830 0x7f447006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:35.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.361+0000 7f446effd640 1 --2- 172.21.15.57:0/924925808 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44700cb7d0 0x7f44700c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:35.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.361+0000 7f446ffff640 1 -- 172.21.15.57:0/924925808 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44700cb7d0 msgr2=0x7f44700c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:35.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.361+0000 7f446ffff640 1 --2- 172.21.15.57:0/924925808 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44700cb7d0 0x7f44700c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:35.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.361+0000 7f446ffff640 1 -- 172.21.15.57:0/924925808 >> v2:172.21.15.88:3300/0 conn(0x7f4470150830 msgr2=0x7f447006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:35.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.361+0000 7f446ffff640 1 --2- 172.21.15.57:0/924925808 >> v2:172.21.15.88:3300/0 conn(0x7f4470150830 0x7f447006b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:35.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.361+0000 7f446ffff640 1 -- 172.21.15.57:0/924925808 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44701569b0 con 0x7f44700c3e40 2023-10-30T14:38:35.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.361+0000 7f446f7fe640 1 --2- 172.21.15.57:0/924925808 >> v2:172.21.15.88:3300/0 conn(0x7f4470150830 0x7f447006b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:38:35.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.361+0000 7f446ffff640 1 --2- 172.21.15.57:0/924925808 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44700c3e40 0x7f4470070520 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f447006fa40 tx=0x7f446801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7598b7872414386 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:35.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.361+0000 7f446e7fc640 1 -- 172.21.15.57:0/924925808 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44680af400 con 0x7f44700c3e40 2023-10-30T14:38:35.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.361+0000 7f446e7fc640 1 -- 172.21.15.57:0/924925808 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44680ab050 con 0x7f44700c3e40 2023-10-30T14:38:35.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.361+0000 7f446e7fc640 1 -- 172.21.15.57:0/924925808 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44680af780 con 0x7f44700c3e40 2023-10-30T14:38:35.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f4475f94640 1 -- 172.21.15.57:0/924925808 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44700c3e40 msgr2=0x7f4470070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:35.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f4475f94640 1 --2- 172.21.15.57:0/924925808 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44700c3e40 0x7f4470070520 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f447006fa40 tx=0x7f446801f1c0 comp rx=0 tx=0).stop 2023-10-30T14:38:35.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f4475f94640 1 -- 172.21.15.57:0/924925808 shutdown_connections 2023-10-30T14:38:35.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f4475f94640 1 --2- 172.21.15.57:0/924925808 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44700cb7d0 0x7f44700c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:35.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f4475f94640 1 --2- 172.21.15.57:0/924925808 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44700c3e40 0x7f4470070520 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:35.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f4475f94640 1 --2- 172.21.15.57:0/924925808 >> v2:172.21.15.88:3300/0 conn(0x7f4470150830 0x7f447006b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:35.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f4475f94640 1 -- 172.21.15.57:0/924925808 >> 172.21.15.57:0/924925808 conn(0x7f44700cac70 msgr2=0x7f44700cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:35.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f4475f94640 1 -- 172.21.15.57:0/924925808 shutdown_connections 2023-10-30T14:38:35.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f4475f94640 1 -- 172.21.15.57:0/924925808 wait complete. 2023-10-30T14:38:35.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f4475f94640 1 Processor -- start 2023-10-30T14:38:35.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f4475f94640 1 -- start start 2023-10-30T14:38:35.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f4475f94640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44700c3e40 0x7f4470154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:35.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f4475f94640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44700cb7d0 0x7f4470148f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:35.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f4475f94640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4470150830 0x7f44701494c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:35.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f4475f94640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4470148af0 con 0x7f44700cb7d0 2023-10-30T14:38:35.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f4475f94640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44701484e0 con 0x7f4470150830 2023-10-30T14:38:35.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f4475f94640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4470155440 con 0x7f44700c3e40 2023-10-30T14:38:35.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f446effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44700cb7d0 0x7f4470148f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:35.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f446effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44700cb7d0 0x7f4470148f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59018/0 (socket says 172.21.15.57:59018) 2023-10-30T14:38:35.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f446effd640 1 -- 172.21.15.57:0/3493332082 learned_addr learned my addr 172.21.15.57:0/3493332082 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:35.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.365+0000 7f446f7fe640 1 --2- 172.21.15.57:0/3493332082 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44700c3e40 0x7f4470154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:35.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.369+0000 7f446ffff640 1 --2- 172.21.15.57:0/3493332082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4470150830 0x7f44701494c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:35.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.369+0000 7f446effd640 1 -- 172.21.15.57:0/3493332082 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44700c3e40 msgr2=0x7f4470154e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:35.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.369+0000 7f446effd640 1 --2- 172.21.15.57:0/3493332082 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44700c3e40 0x7f4470154e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:35.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.369+0000 7f446effd640 1 -- 172.21.15.57:0/3493332082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4470150830 msgr2=0x7f44701494c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:35.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.369+0000 7f446effd640 1 --2- 172.21.15.57:0/3493332082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4470150830 0x7f44701494c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:35.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.369+0000 7f446effd640 1 -- 172.21.15.57:0/3493332082 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4468009b30 con 0x7f44700cb7d0 2023-10-30T14:38:35.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.369+0000 7f446f7fe640 1 --2- 172.21.15.57:0/3493332082 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44700c3e40 0x7f4470154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:38:35.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.369+0000 7f446effd640 1 --2- 172.21.15.57:0/3493332082 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44700cb7d0 0x7f4470148f80 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f446400d800 tx=0x7f446400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:35.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.369+0000 7f4457fff640 1 -- 172.21.15.57:0/3493332082 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44640096a0 con 0x7f44700cb7d0 2023-10-30T14:38:35.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.369+0000 7f4457fff640 1 -- 172.21.15.57:0/3493332082 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4464009800 con 0x7f44700cb7d0 2023-10-30T14:38:35.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.369+0000 7f4475f94640 1 -- 172.21.15.57:0/3493332082 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4470155720 con 0x7f44700cb7d0 2023-10-30T14:38:35.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.369+0000 7f4457fff640 1 -- 172.21.15.57:0/3493332082 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4464009990 con 0x7f44700cb7d0 2023-10-30T14:38:35.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.369+0000 7f4475f94640 1 -- 172.21.15.57:0/3493332082 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4470148780 con 0x7f44700cb7d0 2023-10-30T14:38:35.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.369+0000 7f4475f94640 1 -- 172.21.15.57:0/3493332082 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4470148ca0 con 0x7f44700cb7d0 2023-10-30T14:38:35.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.369+0000 7f4457fff640 1 -- 172.21.15.57:0/3493332082 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4464009af0 con 0x7f44700cb7d0 2023-10-30T14:38:35.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.369+0000 7f4457fff640 1 --2- 172.21.15.57:0/3493332082 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f445c0664d0 0x7f445c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:35.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.369+0000 7f4457fff640 1 -- 172.21.15.57:0/3493332082 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7f4464084730 con 0x7f44700cb7d0 2023-10-30T14:38:35.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.373+0000 7f446f7fe640 1 --2- 172.21.15.57:0/3493332082 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f445c0664d0 0x7f445c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:35.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.373+0000 7f446f7fe640 1 --2- 172.21.15.57:0/3493332082 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f445c0664d0 0x7f445c068990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4460009970 tx=0x7f4460008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:35.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.377+0000 7f4457fff640 1 -- 172.21.15.57:0/3493332082 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4470148ca0 con 0x7f44700cb7d0 2023-10-30T14:38:35.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.569+0000 7f4475f94640 1 -- 172.21.15.57:0/3493332082 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f447004e170 con 0x7f44700cb7d0 2023-10-30T14:38:35.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.569+0000 7f4457fff640 1 -- 172.21.15.57:0/3493332082 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+5742 (secure 0 0 0) 0x7f44640516c0 con 0x7f44700cb7d0 2023-10-30T14:38:35.598 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:35.598 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:32.087700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:35.600 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 131 2023-10-30T14:38:35.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.573+0000 7f4475f94640 1 -- 172.21.15.57:0/3493332082 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f445c0664d0 msgr2=0x7f445c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:35.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.573+0000 7f4475f94640 1 --2- 172.21.15.57:0/3493332082 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f445c0664d0 0x7f445c068990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4460009970 tx=0x7f4460008040 comp rx=0 tx=0).stop 2023-10-30T14:38:35.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.573+0000 7f4475f94640 1 -- 172.21.15.57:0/3493332082 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44700cb7d0 msgr2=0x7f4470148f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:35.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.573+0000 7f4475f94640 1 --2- 172.21.15.57:0/3493332082 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44700cb7d0 0x7f4470148f80 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f446400d800 tx=0x7f446400dcc0 comp rx=0 tx=0).stop 2023-10-30T14:38:35.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.573+0000 7f4475f94640 1 -- 172.21.15.57:0/3493332082 shutdown_connections 2023-10-30T14:38:35.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.573+0000 7f4475f94640 1 --2- 172.21.15.57:0/3493332082 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f445c0664d0 0x7f445c068990 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:35.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.573+0000 7f4475f94640 1 --2- 172.21.15.57:0/3493332082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4470150830 0x7f44701494c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:35.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.573+0000 7f4475f94640 1 --2- 172.21.15.57:0/3493332082 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44700cb7d0 0x7f4470148f80 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:35.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.573+0000 7f4475f94640 1 --2- 172.21.15.57:0/3493332082 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44700c3e40 0x7f4470154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:35.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.573+0000 7f4475f94640 1 -- 172.21.15.57:0/3493332082 >> 172.21.15.57:0/3493332082 conn(0x7f44700cac70 msgr2=0x7f4470147a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:35.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.573+0000 7f4475f94640 1 -- 172.21.15.57:0/3493332082 shutdown_connections 2023-10-30T14:38:35.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:35.573+0000 7f4475f94640 1 -- 172.21.15.57:0/3493332082 wait complete. 2023-10-30T14:38:35.604 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:38:37.605 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:37.670 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:38:37.676 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:38:37.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.761+0000 7f1cc9e17640 1 Processor -- start 2023-10-30T14:38:37.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.761+0000 7f1cc9e17640 1 -- start start 2023-10-30T14:38:37.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.761+0000 7f1cc9e17640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1cc40c4a10 0x7f1cc415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:37.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.761+0000 7f1cc9e17640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1cc40c42a0 0x7f1cc40ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:37.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.761+0000 7f1cc9e17640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc4159f20 con 0x7f1cc4153c80 2023-10-30T14:38:37.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.761+0000 7f1cc9e17640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc415dec0 con 0x7f1cc40c4a10 2023-10-30T14:38:37.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.761+0000 7f1cc9e17640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc41599a0 con 0x7f1cc40c42a0 2023-10-30T14:38:37.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.761+0000 7f1cc2ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1cc40c4a10 0x7f1cc415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:37.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.761+0000 7f1cc2ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1cc40c4a10 0x7f1cc415cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59024/0 (socket says 172.21.15.57:59024) 2023-10-30T14:38:37.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.761+0000 7f1cc2ffd640 1 -- 172.21.15.57:0/2234873109 learned_addr learned my addr 172.21.15.57:0/2234873109 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:37.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.761+0000 7f1cc3fff640 1 --2- 172.21.15.57:0/2234873109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1cc40c42a0 0x7f1cc40ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:37.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.761+0000 7f1cc2ffd640 1 -- 172.21.15.57:0/2234873109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1cc40c42a0 msgr2=0x7f1cc40ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:37.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.761+0000 7f1cc2ffd640 1 --2- 172.21.15.57:0/2234873109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1cc40c42a0 0x7f1cc40ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:37.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.761+0000 7f1cc2ffd640 1 -- 172.21.15.57:0/2234873109 >> v1:172.21.15.57:6789/0 conn(0x7f1cc4153c80 legacy=0x7f1cc415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:37.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.761+0000 7f1cc2ffd640 1 -- 172.21.15.57:0/2234873109 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc415d620 con 0x7f1cc40c4a10 2023-10-30T14:38:37.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.761+0000 7f1cc27fc640 1 -- 172.21.15.57:0/2234873109 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372410550 0 0) 0x7f1cc4159f20 con 0x7f1cc4153c80 2023-10-30T14:38:37.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.765+0000 7f1cc2ffd640 1 --2- 172.21.15.57:0/2234873109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc40c4a10 0x7f1cc415cf90 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009870 tx=0x7f1cb801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4919e6d8e29f174f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:37.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.765+0000 7f1cc27fc640 1 -- 172.21.15.57:0/2234873109 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb80a8a30 con 0x7f1cc40c4a10 2023-10-30T14:38:37.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.765+0000 7f1cc27fc640 1 -- 172.21.15.57:0/2234873109 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb80a8b90 con 0x7f1cc40c4a10 2023-10-30T14:38:37.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.765+0000 7f1cc27fc640 1 -- 172.21.15.57:0/2234873109 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb80b0440 con 0x7f1cc40c4a10 2023-10-30T14:38:37.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.765+0000 7f1cc9e17640 1 -- 172.21.15.57:0/2234873109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc40c4a10 msgr2=0x7f1cc415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:37.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.765+0000 7f1cc9e17640 1 --2- 172.21.15.57:0/2234873109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc40c4a10 0x7f1cc415cf90 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009870 tx=0x7f1cb801ed30 comp rx=0 tx=0).stop 2023-10-30T14:38:37.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.765+0000 7f1cc9e17640 1 -- 172.21.15.57:0/2234873109 shutdown_connections 2023-10-30T14:38:37.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.765+0000 7f1cc9e17640 1 --2- 172.21.15.57:0/2234873109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1cc40c42a0 0x7f1cc40ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:37.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.765+0000 7f1cc9e17640 1 --2- 172.21.15.57:0/2234873109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc40c4a10 0x7f1cc415cf90 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:37.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.765+0000 7f1cc9e17640 1 -- 172.21.15.57:0/2234873109 >> 172.21.15.57:0/2234873109 conn(0x7f1cc40715c0 msgr2=0x7f1cc40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:37.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.765+0000 7f1cc9e17640 1 -- 172.21.15.57:0/2234873109 shutdown_connections 2023-10-30T14:38:37.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.765+0000 7f1cc9e17640 1 -- 172.21.15.57:0/2234873109 wait complete. 2023-10-30T14:38:37.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc9e17640 1 Processor -- start 2023-10-30T14:38:37.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc9e17640 1 -- start start 2023-10-30T14:38:37.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc9e17640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1cc40c42a0 0x7f1cc40c59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:37.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc9e17640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc40c4a10 0x7f1cc40ce750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:37.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc9e17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1cc4153c80 0x7f1cc40b3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:37.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc9e17640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc40b23d0 con 0x7f1cc40c4a10 2023-10-30T14:38:37.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc9e17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc41f1dd0 con 0x7f1cc4153c80 2023-10-30T14:38:37.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc9e17640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc40b2f80 con 0x7f1cc40c42a0 2023-10-30T14:38:37.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc40c4a10 0x7f1cc40ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:37.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc40c4a10 0x7f1cc40ce750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59030/0 (socket says 172.21.15.57:59030) 2023-10-30T14:38:37.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc2ffd640 1 -- 172.21.15.57:0/551647284 learned_addr learned my addr 172.21.15.57:0/551647284 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:37.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc37fe640 1 --2- 172.21.15.57:0/551647284 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1cc40c42a0 0x7f1cc40c59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:37.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc3fff640 1 --2- 172.21.15.57:0/551647284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1cc4153c80 0x7f1cc40b3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:37.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc2ffd640 1 -- 172.21.15.57:0/551647284 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1cc40c42a0 msgr2=0x7f1cc40c59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:37.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc2ffd640 1 --2- 172.21.15.57:0/551647284 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1cc40c42a0 0x7f1cc40c59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:37.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc2ffd640 1 -- 172.21.15.57:0/551647284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1cc4153c80 msgr2=0x7f1cc40b3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:37.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc2ffd640 1 --2- 172.21.15.57:0/551647284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1cc4153c80 0x7f1cc40b3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:37.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc2ffd640 1 -- 172.21.15.57:0/551647284 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb80094d0 con 0x7f1cc40c4a10 2023-10-30T14:38:37.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc2ffd640 1 --2- 172.21.15.57:0/551647284 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc40c4a10 0x7f1cc40ce750 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f1cb80b3040 tx=0x7f1cb801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:37.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cabfff640 1 -- 172.21.15.57:0/551647284 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb80ba600 con 0x7f1cc40c4a10 2023-10-30T14:38:37.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cabfff640 1 -- 172.21.15.57:0/551647284 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb80ba760 con 0x7f1cc40c4a10 2023-10-30T14:38:37.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cabfff640 1 -- 172.21.15.57:0/551647284 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb80bab10 con 0x7f1cc40c4a10 2023-10-30T14:38:37.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.769+0000 7f1cc9e17640 1 -- 172.21.15.57:0/551647284 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc40b32f0 con 0x7f1cc40c4a10 2023-10-30T14:38:37.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.773+0000 7f1cc9e17640 1 -- 172.21.15.57:0/551647284 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc41f22e0 con 0x7f1cc40c4a10 2023-10-30T14:38:37.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.773+0000 7f1cc9e17640 1 -- 172.21.15.57:0/551647284 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc40b3480 con 0x7f1cc40c4a10 2023-10-30T14:38:37.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.773+0000 7f1cabfff640 1 -- 172.21.15.57:0/551647284 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1cb80bac70 con 0x7f1cc40c4a10 2023-10-30T14:38:37.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.773+0000 7f1cabfff640 1 --2- 172.21.15.57:0/551647284 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1cb0066520 0x7f1cb00689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:37.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.773+0000 7f1cabfff640 1 -- 172.21.15.57:0/551647284 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7f1cb81226f0 con 0x7f1cc40c4a10 2023-10-30T14:38:37.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.773+0000 7f1cc37fe640 1 --2- 172.21.15.57:0/551647284 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1cb0066520 0x7f1cb00689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:37.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.773+0000 7f1cc37fe640 1 --2- 172.21.15.57:0/551647284 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1cb0066520 0x7f1cb00689e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4007fd0 tx=0x7f1cb4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:37.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.777+0000 7f1cabfff640 1 -- 172.21.15.57:0/551647284 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1cb80ef830 con 0x7f1cc40c4a10 2023-10-30T14:38:37.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.969+0000 7f1cc9e17640 1 -- 172.21.15.57:0/551647284 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cc404e170 con 0x7f1cc40c4a10 2023-10-30T14:38:37.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.969+0000 7f1cabfff640 1 -- 172.21.15.57:0/551647284 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+5742 (secure 0 0 0) 0x7f1cb80f2e50 con 0x7f1cc40c4a10 2023-10-30T14:38:37.972 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:37.972 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:32.087700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:37.973 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 131 2023-10-30T14:38:37.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.973+0000 7f1cc9e17640 1 -- 172.21.15.57:0/551647284 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1cb0066520 msgr2=0x7f1cb00689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:37.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.973+0000 7f1cc9e17640 1 --2- 172.21.15.57:0/551647284 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1cb0066520 0x7f1cb00689e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4007fd0 tx=0x7f1cb4009040 comp rx=0 tx=0).stop 2023-10-30T14:38:37.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.973+0000 7f1cc9e17640 1 -- 172.21.15.57:0/551647284 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc40c4a10 msgr2=0x7f1cc40ce750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:37.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.973+0000 7f1cc9e17640 1 --2- 172.21.15.57:0/551647284 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc40c4a10 0x7f1cc40ce750 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f1cb80b3040 tx=0x7f1cb801bad0 comp rx=0 tx=0).stop 2023-10-30T14:38:37.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.973+0000 7f1cc9e17640 1 -- 172.21.15.57:0/551647284 shutdown_connections 2023-10-30T14:38:37.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.973+0000 7f1cc9e17640 1 --2- 172.21.15.57:0/551647284 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1cb0066520 0x7f1cb00689e0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:37.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.973+0000 7f1cc9e17640 1 --2- 172.21.15.57:0/551647284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1cc4153c80 0x7f1cc40b3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:37.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.973+0000 7f1cc9e17640 1 --2- 172.21.15.57:0/551647284 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc40c4a10 0x7f1cc40ce750 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:37.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.973+0000 7f1cc9e17640 1 --2- 172.21.15.57:0/551647284 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1cc40c42a0 0x7f1cc40c59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:37.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.973+0000 7f1cc9e17640 1 -- 172.21.15.57:0/551647284 >> 172.21.15.57:0/551647284 conn(0x7f1cc40715c0 msgr2=0x7f1cc406c170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:37.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.973+0000 7f1cc9e17640 1 -- 172.21.15.57:0/551647284 shutdown_connections 2023-10-30T14:38:37.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:37.973+0000 7f1cc9e17640 1 -- 172.21.15.57:0/551647284 wait complete. 2023-10-30T14:38:37.994 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:38:39.995 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:40.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.149+0000 7f2dd6a2f640 1 Processor -- start 2023-10-30T14:38:40.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.149+0000 7f2dd6a2f640 1 -- start start 2023-10-30T14:38:40.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.149+0000 7f2dd6a2f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2dd00c8eb0 0x7f2dd015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:40.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.153+0000 7f2dd6a2f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2dd015ac70 0x7f2dd015a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:40.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.153+0000 7f2dd6a2f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd015dcc0 con 0x7f2dd006aeb0 2023-10-30T14:38:40.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.153+0000 7f2dd6a2f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd0159800 con 0x7f2dd015ac70 2023-10-30T14:38:40.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.153+0000 7f2dd6a2f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd0159d80 con 0x7f2dd00c8eb0 2023-10-30T14:38:40.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.153+0000 7f2dd522c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2dd015ac70 0x7f2dd015a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:40.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.153+0000 7f2dd522c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2dd015ac70 0x7f2dd015a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59036/0 (socket says 172.21.15.57:59036) 2023-10-30T14:38:40.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.153+0000 7f2dd522c640 1 -- 172.21.15.57:0/1698829375 learned_addr learned my addr 172.21.15.57:0/1698829375 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:40.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.153+0000 7f2dd5a2d640 1 --2- 172.21.15.57:0/1698829375 >> v2:172.21.15.88:3300/0 conn(0x7f2dd00c8eb0 0x7f2dd015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:40.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.153+0000 7f2dd522c640 1 -- 172.21.15.57:0/1698829375 >> v2:172.21.15.88:3300/0 conn(0x7f2dd00c8eb0 msgr2=0x7f2dd015cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:40.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.153+0000 7f2dd522c640 1 --2- 172.21.15.57:0/1698829375 >> v2:172.21.15.88:3300/0 conn(0x7f2dd00c8eb0 0x7f2dd015cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:40.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.153+0000 7f2dd522c640 1 -- 172.21.15.57:0/1698829375 >> v1:172.21.15.57:6789/0 conn(0x7f2dd006aeb0 legacy=0x7f2dd0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:40.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.153+0000 7f2dd522c640 1 -- 172.21.15.57:0/1698829375 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd015a750 con 0x7f2dd015ac70 2023-10-30T14:38:40.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.153+0000 7f2dd522c640 1 --2- 172.21.15.57:0/1698829375 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2dd015ac70 0x7f2dd015a210 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f2dd006aa40 tx=0x7f2dc001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7ee7da0353c3bc1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:40.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.153+0000 7f2dd4a2b640 1 -- 172.21.15.57:0/1698829375 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc00a89f0 con 0x7f2dd015ac70 2023-10-30T14:38:40.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.153+0000 7f2dd4a2b640 1 -- 172.21.15.57:0/1698829375 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc00a8b50 con 0x7f2dd015ac70 2023-10-30T14:38:40.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.153+0000 7f2dd4a2b640 1 -- 172.21.15.57:0/1698829375 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc00b0440 con 0x7f2dd015ac70 2023-10-30T14:38:40.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd6a2f640 1 -- 172.21.15.57:0/1698829375 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2dd015ac70 msgr2=0x7f2dd015a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:40.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd6a2f640 1 --2- 172.21.15.57:0/1698829375 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2dd015ac70 0x7f2dd015a210 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f2dd006aa40 tx=0x7f2dc001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:38:40.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd6a2f640 1 -- 172.21.15.57:0/1698829375 shutdown_connections 2023-10-30T14:38:40.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd6a2f640 1 --2- 172.21.15.57:0/1698829375 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2dd015ac70 0x7f2dd015a210 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:40.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd6a2f640 1 --2- 172.21.15.57:0/1698829375 >> v2:172.21.15.88:3300/0 conn(0x7f2dd00c8eb0 0x7f2dd015cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:40.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd6a2f640 1 -- 172.21.15.57:0/1698829375 >> 172.21.15.57:0/1698829375 conn(0x7f2dd00cef00 msgr2=0x7f2dd00cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:40.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd6a2f640 1 -- 172.21.15.57:0/1698829375 shutdown_connections 2023-10-30T14:38:40.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd6a2f640 1 -- 172.21.15.57:0/1698829375 wait complete. 2023-10-30T14:38:40.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd6a2f640 1 Processor -- start 2023-10-30T14:38:40.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd6a2f640 1 -- start start 2023-10-30T14:38:40.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd6a2f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2dd006aeb0 0x7f2dd014a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:40.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd6a2f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2dd00c8eb0 0x7f2dd00ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:40.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd6a2f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dd015ac70 0x7f2dd01effe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:40.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd6a2f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd014ef80 con 0x7f2dd00c8eb0 2023-10-30T14:38:40.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd6a2f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd014f220 con 0x7f2dd015ac70 2023-10-30T14:38:40.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd6a2f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd01f20f0 con 0x7f2dd006aeb0 2023-10-30T14:38:40.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd522c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2dd00c8eb0 0x7f2dd00ce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:40.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd522c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2dd00c8eb0 0x7f2dd00ce0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59038/0 (socket says 172.21.15.57:59038) 2023-10-30T14:38:40.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd522c640 1 -- 172.21.15.57:0/4261565256 learned_addr learned my addr 172.21.15.57:0/4261565256 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:40.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd5a2d640 1 --2- 172.21.15.57:0/4261565256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2dd006aeb0 0x7f2dd014a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:40.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd522c640 1 -- 172.21.15.57:0/4261565256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2dd006aeb0 msgr2=0x7f2dd014a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:40.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd522c640 1 --2- 172.21.15.57:0/4261565256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2dd006aeb0 0x7f2dd014a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:40.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd622e640 1 --2- 172.21.15.57:0/4261565256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dd015ac70 0x7f2dd01effe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:40.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.157+0000 7f2dd522c640 1 -- 172.21.15.57:0/4261565256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dd015ac70 msgr2=0x7f2dd01effe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:40.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.161+0000 7f2dd522c640 1 --2- 172.21.15.57:0/4261565256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dd015ac70 0x7f2dd01effe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:40.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.161+0000 7f2dd522c640 1 -- 172.21.15.57:0/4261565256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc00094d0 con 0x7f2dd00c8eb0 2023-10-30T14:38:40.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.161+0000 7f2dd5a2d640 1 --2- 172.21.15.57:0/4261565256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2dd006aeb0 0x7f2dd014a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:38:40.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.161+0000 7f2dd622e640 1 --2- 172.21.15.57:0/4261565256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dd015ac70 0x7f2dd01effe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:38:40.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.161+0000 7f2dd522c640 1 --2- 172.21.15.57:0/4261565256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2dd00c8eb0 0x7f2dd00ce0c0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f2dc001b900 tx=0x7f2dc001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:40.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.161+0000 7f2dc67fc640 1 -- 172.21.15.57:0/4261565256 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc00ba600 con 0x7f2dd00c8eb0 2023-10-30T14:38:40.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.161+0000 7f2dc67fc640 1 -- 172.21.15.57:0/4261565256 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc00ba760 con 0x7f2dd00c8eb0 2023-10-30T14:38:40.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.161+0000 7f2dd6a2f640 1 -- 172.21.15.57:0/4261565256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd01f23d0 con 0x7f2dd00c8eb0 2023-10-30T14:38:40.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.161+0000 7f2dc67fc640 1 -- 172.21.15.57:0/4261565256 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc00bab10 con 0x7f2dd00c8eb0 2023-10-30T14:38:40.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.161+0000 7f2dd6a2f640 1 -- 172.21.15.57:0/4261565256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd01f2990 con 0x7f2dd00c8eb0 2023-10-30T14:38:40.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.161+0000 7f2dd6a2f640 1 -- 172.21.15.57:0/4261565256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd004e170 con 0x7f2dd00c8eb0 2023-10-30T14:38:40.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.161+0000 7f2dc67fc640 1 -- 172.21.15.57:0/4261565256 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2dc00bac70 con 0x7f2dd00c8eb0 2023-10-30T14:38:40.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.161+0000 7f2dc67fc640 1 --2- 172.21.15.57:0/4261565256 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2db40664a0 0x7f2db4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:40.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.161+0000 7f2dc67fc640 1 -- 172.21.15.57:0/4261565256 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7f2dc01229e0 con 0x7f2dd00c8eb0 2023-10-30T14:38:40.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.165+0000 7f2dd5a2d640 1 --2- 172.21.15.57:0/4261565256 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2db40664a0 0x7f2db4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:40.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.165+0000 7f2dd5a2d640 1 --2- 172.21.15.57:0/4261565256 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2db40664a0 0x7f2db4068960 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009840 tx=0x7f2dcc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:40.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.165+0000 7f2dc67fc640 1 -- 172.21.15.57:0/4261565256 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2dc00ef8f0 con 0x7f2dd00c8eb0 2023-10-30T14:38:40.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.357+0000 7f2dd6a2f640 1 -- 172.21.15.57:0/4261565256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2dd00ca960 con 0x7f2dd00c8eb0 2023-10-30T14:38:40.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.357+0000 7f2dc67fc640 1 -- 172.21.15.57:0/4261565256 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5742 (secure 0 0 0) 0x7f2dc00f2f10 con 0x7f2dd00c8eb0 2023-10-30T14:38:40.361 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:40.361 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:37.197465+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:40.362 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 132 2023-10-30T14:38:40.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.361+0000 7f2dd6a2f640 1 -- 172.21.15.57:0/4261565256 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2db40664a0 msgr2=0x7f2db4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:40.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.361+0000 7f2dd6a2f640 1 --2- 172.21.15.57:0/4261565256 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2db40664a0 0x7f2db4068960 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009840 tx=0x7f2dcc008040 comp rx=0 tx=0).stop 2023-10-30T14:38:40.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.361+0000 7f2dd6a2f640 1 -- 172.21.15.57:0/4261565256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2dd00c8eb0 msgr2=0x7f2dd00ce0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:40.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.361+0000 7f2dd6a2f640 1 --2- 172.21.15.57:0/4261565256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2dd00c8eb0 0x7f2dd00ce0c0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f2dc001b900 tx=0x7f2dc001ba90 comp rx=0 tx=0).stop 2023-10-30T14:38:40.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.361+0000 7f2dd6a2f640 1 -- 172.21.15.57:0/4261565256 shutdown_connections 2023-10-30T14:38:40.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.361+0000 7f2dd6a2f640 1 --2- 172.21.15.57:0/4261565256 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2db40664a0 0x7f2db4068960 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:40.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.361+0000 7f2dd6a2f640 1 --2- 172.21.15.57:0/4261565256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dd015ac70 0x7f2dd01effe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:40.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.361+0000 7f2dd6a2f640 1 --2- 172.21.15.57:0/4261565256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2dd00c8eb0 0x7f2dd00ce0c0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:40.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.361+0000 7f2dd6a2f640 1 --2- 172.21.15.57:0/4261565256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2dd006aeb0 0x7f2dd014a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:40.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.361+0000 7f2dd6a2f640 1 -- 172.21.15.57:0/4261565256 >> 172.21.15.57:0/4261565256 conn(0x7f2dd00cef00 msgr2=0x7f2dd00cb8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:40.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.361+0000 7f2dd6a2f640 1 -- 172.21.15.57:0/4261565256 shutdown_connections 2023-10-30T14:38:40.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:40.361+0000 7f2dd6a2f640 1 -- 172.21.15.57:0/4261565256 wait complete. 2023-10-30T14:38:40.381 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=2, replacing gid=6204 2023-10-30T14:38:40.381 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.e 2023-10-30T14:38:42.037 INFO:tasks.ceph.mds.a.smithi057.stderr:2023-10-30T14:38:42.037+0000 7f5ba0392640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T14:38:46.382 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T14:38:46.382 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T14:38:46.383 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T14:38:46.387 INFO:tasks.ceph.mds.e:Started 2023-10-30T14:38:46.387 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:46.485 INFO:tasks.ceph.mds.e.smithi057.stdout:starting mds.e at 2023-10-30T14:38:46.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.541+0000 7fbce2101640 1 Processor -- start 2023-10-30T14:38:46.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.541+0000 7fbce2101640 1 -- start start 2023-10-30T14:38:46.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.541+0000 7fbce2101640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbcdc151c20 0x7fbcdc0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:46.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.541+0000 7fbce2101640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbcdc0cb8b0 0x7fbcdc06b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:46.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.541+0000 7fbce2101640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcdc0c5030 con 0x7fbcdc151840 2023-10-30T14:38:46.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.541+0000 7fbce2101640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcdc157830 con 0x7fbcdc151c20 2023-10-30T14:38:46.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.541+0000 7fbce2101640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcdc157b10 con 0x7fbcdc0cb8b0 2023-10-30T14:38:46.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.545+0000 7fbcdaffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbcdc151c20 0x7fbcdc0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:46.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.545+0000 7fbcdaffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbcdc151c20 0x7fbcdc0c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43334/0 (socket says 172.21.15.57:43334) 2023-10-30T14:38:46.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.545+0000 7fbcdaffd640 1 -- 172.21.15.57:0/475336170 learned_addr learned my addr 172.21.15.57:0/475336170 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:46.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.545+0000 7fbcdbfff640 1 --2- 172.21.15.57:0/475336170 >> v2:172.21.15.88:3300/0 conn(0x7fbcdc0cb8b0 0x7fbcdc06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:46.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.545+0000 7fbcdaffd640 1 -- 172.21.15.57:0/475336170 >> v2:172.21.15.88:3300/0 conn(0x7fbcdc0cb8b0 msgr2=0x7fbcdc06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:38:46.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.545+0000 7fbcdaffd640 1 --2- 172.21.15.57:0/475336170 >> v2:172.21.15.88:3300/0 conn(0x7fbcdc0cb8b0 0x7fbcdc06b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:46.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.545+0000 7fbcdaffd640 1 -- 172.21.15.57:0/475336170 >> v1:172.21.15.88:6789/0 conn(0x7fbcdc151840 legacy=0x7fbcdc072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:46.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.545+0000 7fbcdaffd640 1 -- 172.21.15.57:0/475336170 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcdc157df0 con 0x7fbcdc151c20 2023-10-30T14:38:46.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.545+0000 7fbcdbfff640 1 --2- 172.21.15.57:0/475336170 >> v2:172.21.15.88:3300/0 conn(0x7fbcdc0cb8b0 0x7fbcdc06b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:38:46.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.545+0000 7fbcdaffd640 1 --2- 172.21.15.57:0/475336170 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcdc151c20 0x7fbcdc0c49a0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc0c6cb0 tx=0x7fbcd001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7379354c6778ca3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:46.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.545+0000 7fbcda7fc640 1 -- 172.21.15.57:0/475336170 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd00a89f0 con 0x7fbcdc151c20 2023-10-30T14:38:46.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.545+0000 7fbcda7fc640 1 -- 172.21.15.57:0/475336170 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd00a8b50 con 0x7fbcdc151c20 2023-10-30T14:38:46.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.545+0000 7fbcda7fc640 1 -- 172.21.15.57:0/475336170 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd00b0440 con 0x7fbcdc151c20 2023-10-30T14:38:46.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbce2101640 1 -- 172.21.15.57:0/475336170 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcdc151c20 msgr2=0x7fbcdc0c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:46.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbce2101640 1 --2- 172.21.15.57:0/475336170 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcdc151c20 0x7fbcdc0c49a0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc0c6cb0 tx=0x7fbcd001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:38:46.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbce2101640 1 -- 172.21.15.57:0/475336170 shutdown_connections 2023-10-30T14:38:46.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbce2101640 1 --2- 172.21.15.57:0/475336170 >> v2:172.21.15.88:3300/0 conn(0x7fbcdc0cb8b0 0x7fbcdc06b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:46.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbce2101640 1 --2- 172.21.15.57:0/475336170 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcdc151c20 0x7fbcdc0c49a0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:46.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbce2101640 1 -- 172.21.15.57:0/475336170 >> 172.21.15.57:0/475336170 conn(0x7fbcdc0cac70 msgr2=0x7fbcdc0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:46.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbce2101640 1 -- 172.21.15.57:0/475336170 shutdown_connections 2023-10-30T14:38:46.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbce2101640 1 -- 172.21.15.57:0/475336170 wait complete. 2023-10-30T14:38:46.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbce2101640 1 Processor -- start 2023-10-30T14:38:46.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbce2101640 1 -- start start 2023-10-30T14:38:46.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbce2101640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcdc0cb8b0 0x7fbcdc1f3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:46.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbce2101640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcdc151840 0x7fbcdc1eef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:46.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbce2101640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcdc151c20 0x7fbcdc0c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:46.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbce2101640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc1f2b70 con 0x7fbcdc0cb8b0 2023-10-30T14:38:46.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbce2101640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc1ef450 con 0x7fbcdc151840 2023-10-30T14:38:46.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbce2101640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc0713d0 con 0x7fbcdc151c20 2023-10-30T14:38:46.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbcdb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcdc0cb8b0 0x7fbcdc1f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:46.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbcdb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcdc0cb8b0 0x7fbcdc1f3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43336/0 (socket says 172.21.15.57:43336) 2023-10-30T14:38:46.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbcdb7fe640 1 -- 172.21.15.57:0/3595402828 learned_addr learned my addr 172.21.15.57:0/3595402828 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:46.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbcdbfff640 1 --2- 172.21.15.57:0/3595402828 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcdc151c20 0x7fbcdc0c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:46.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.549+0000 7fbcdaffd640 1 --2- 172.21.15.57:0/3595402828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcdc151840 0x7fbcdc1eef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:46.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.553+0000 7fbcdb7fe640 1 -- 172.21.15.57:0/3595402828 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcdc151c20 msgr2=0x7fbcdc0c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:46.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.553+0000 7fbcdb7fe640 1 --2- 172.21.15.57:0/3595402828 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcdc151c20 0x7fbcdc0c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:46.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.553+0000 7fbcdb7fe640 1 -- 172.21.15.57:0/3595402828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcdc151840 msgr2=0x7fbcdc1eef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:46.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.553+0000 7fbcdb7fe640 1 --2- 172.21.15.57:0/3595402828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcdc151840 0x7fbcdc1eef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:46.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.553+0000 7fbcdb7fe640 1 -- 172.21.15.57:0/3595402828 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd00094d0 con 0x7fbcdc0cb8b0 2023-10-30T14:38:46.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.553+0000 7fbcdb7fe640 1 --2- 172.21.15.57:0/3595402828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcdc0cb8b0 0x7fbcdc1f3000 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fbccc00c500 tx=0x7fbccc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:46.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.553+0000 7fbcc3fff640 1 -- 172.21.15.57:0/3595402828 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc005220 con 0x7fbcdc0cb8b0 2023-10-30T14:38:46.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.553+0000 7fbcc3fff640 1 -- 172.21.15.57:0/3595402828 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbccc005380 con 0x7fbcdc0cb8b0 2023-10-30T14:38:46.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.553+0000 7fbcc3fff640 1 -- 172.21.15.57:0/3595402828 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc007270 con 0x7fbcdc0cb8b0 2023-10-30T14:38:46.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.553+0000 7fbce2101640 1 -- 172.21.15.57:0/3595402828 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcdc070e10 con 0x7fbcdc0cb8b0 2023-10-30T14:38:46.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.553+0000 7fbce2101640 1 -- 172.21.15.57:0/3595402828 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcdc1f2520 con 0x7fbcdc0cb8b0 2023-10-30T14:38:46.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.553+0000 7fbce2101640 1 -- 172.21.15.57:0/3595402828 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcdc04e170 con 0x7fbcdc0cb8b0 2023-10-30T14:38:46.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.553+0000 7fbcc3fff640 1 -- 172.21.15.57:0/3595402828 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbccc0073d0 con 0x7fbcdc0cb8b0 2023-10-30T14:38:46.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.553+0000 7fbcc3fff640 1 --2- 172.21.15.57:0/3595402828 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbcc80664a0 0x7fbcc8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:46.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.553+0000 7fbcc3fff640 1 -- 172.21.15.57:0/3595402828 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7fbccc083bc0 con 0x7fbcdc0cb8b0 2023-10-30T14:38:46.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.553+0000 7fbcdaffd640 1 --2- 172.21.15.57:0/3595402828 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbcc80664a0 0x7fbcc8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:46.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.557+0000 7fbcdaffd640 1 --2- 172.21.15.57:0/3595402828 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbcc80664a0 0x7fbcc8068960 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fbcd00b2000 tx=0x7fbcd0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:46.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.557+0000 7fbcc3fff640 1 -- 172.21.15.57:0/3595402828 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbccc050b90 con 0x7fbcdc0cb8b0 2023-10-30T14:38:46.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.753+0000 7fbce2101640 1 -- 172.21.15.57:0/3595402828 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcdc1f28c0 con 0x7fbcdc0cb8b0 2023-10-30T14:38:46.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.753+0000 7fbcc3fff640 1 -- 172.21.15.57:0/3595402828 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5742 (secure 0 0 0) 0x7fbccc0541b0 con 0x7fbcdc0cb8b0 2023-10-30T14:38:46.755 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:46.756 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:37.197465+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:46.756 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 132 2023-10-30T14:38:46.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.757+0000 7fbce2101640 1 -- 172.21.15.57:0/3595402828 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbcc80664a0 msgr2=0x7fbcc8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:46.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.757+0000 7fbce2101640 1 --2- 172.21.15.57:0/3595402828 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbcc80664a0 0x7fbcc8068960 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fbcd00b2000 tx=0x7fbcd0009040 comp rx=0 tx=0).stop 2023-10-30T14:38:46.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.757+0000 7fbce2101640 1 -- 172.21.15.57:0/3595402828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcdc0cb8b0 msgr2=0x7fbcdc1f3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:46.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.757+0000 7fbce2101640 1 --2- 172.21.15.57:0/3595402828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcdc0cb8b0 0x7fbcdc1f3000 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fbccc00c500 tx=0x7fbccc00c9c0 comp rx=0 tx=0).stop 2023-10-30T14:38:46.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.757+0000 7fbce2101640 1 -- 172.21.15.57:0/3595402828 shutdown_connections 2023-10-30T14:38:46.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.757+0000 7fbce2101640 1 --2- 172.21.15.57:0/3595402828 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbcc80664a0 0x7fbcc8068960 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:46.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.757+0000 7fbce2101640 1 --2- 172.21.15.57:0/3595402828 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcdc151c20 0x7fbcdc0c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:46.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.757+0000 7fbce2101640 1 --2- 172.21.15.57:0/3595402828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcdc151840 0x7fbcdc1eef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:46.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.757+0000 7fbce2101640 1 --2- 172.21.15.57:0/3595402828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcdc0cb8b0 0x7fbcdc1f3000 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:46.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.757+0000 7fbce2101640 1 -- 172.21.15.57:0/3595402828 >> 172.21.15.57:0/3595402828 conn(0x7fbcdc0cac70 msgr2=0x7fbcdc1505c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:46.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.757+0000 7fbce2101640 1 -- 172.21.15.57:0/3595402828 shutdown_connections 2023-10-30T14:38:46.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:46.757+0000 7fbce2101640 1 -- 172.21.15.57:0/3595402828 wait complete. 2023-10-30T14:38:46.777 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-30T14:38:47.247 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 20% (42/204) Updating files: 21% (43/204) Updating files: 22% (45/204) Updating files: 23% (47/204) Updating files: 24% (49/204) Updating files: 25% (51/204) Updating files: 26% (54/204) Updating files: 26% (55/204) Updating files: 27% (56/204) Updating files: 28% (58/204) Updating files: 29% (60/204) Updating files: 30% (62/204) Updating files: 31% (64/204) Updating files: 32% (66/204) Updating files: 33% (68/204) Updating files: 34% (70/204) Updating files: 35% (72/204) Updating files: 36% (74/204) Updating files: 37% (76/204) Updating files: 38% (78/204) Updating files: 39% (80/204) Updating files: 40% (82/204) Updating files: 41% (84/204) Updating files: 42% (86/204) Updating files: 43% (88/204) Updating files: 44% (90/204) Updating files: 45% (92/204) Updating files: 46% (94/204) Updating files: 47% (96/204) Updating files: 48% (98/204) Updating files: 49% (100/204) Updating files: 50% (102/204) Updating files: 51% (105/204) Updating files: 52% (107/204) Updating files: 53% (109/204) Updating files: 54% (111/204) Updating files: 55% (113/204) Updating files: 56% (115/204) Updating files: 57% (117/204) Updating files: 58% (119/204) Updating files: 59% (121/204) Updating files: 60% (123/204) Updating files: 61% (125/204) Updating files: 62% (127/204) Updating files: 63% (129/204) Updating files: 64% (131/204) Updating files: 65% (133/204) Updating files: 66% (135/204) Updating files: 67% (137/204) Updating files: 68% (139/204) Updating files: 69% (141/204) Updating files: 70% (143/204) Updating files: 71% (145/204) Updating files: 72% (147/204) Updating files: 73% (149/204) Updating files: 74% (151/204) Updating files: 75% (153/204) Updating files: 76% (156/204) Updating files: 77% (158/204) Updating files: 78% (160/204) Updating files: 79% (162/204) Updating files: 80% (164/204) Updating files: 81% (166/204) Updating files: 81% (167/204) Updating files: 82% (168/204) Updating files: 83% (170/204) Updating files: 84% (172/204) Updating files: 85% (174/204) Updating files: 86% (176/204) Updating files: 86% (177/204) Updating files: 87% (178/204) Updating files: 88% (180/204) Updating files: 89% (182/204) Updating files: 90% (184/204) Updating files: 91% (186/204) Updating files: 92% (188/204) Updating files: 93% (190/204) Updating files: 94% (192/204) Updating files: 95% (194/204) Updating files: 96% (196/204) Updating files: 97% (198/204) Updating files: 98% (200/204) Updating files: 99% (202/204) Updating files: 100% (204/204) Updating files: 100% (204/204), done. 2023-10-30T14:38:47.330 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 8a7c95f60ad v0.22 2023-10-30T14:38:47.335 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.22 2023-10-30T14:38:47.371 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 23 -eq 48 ] 2023-10-30T14:38:47.371 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.23 2023-10-30T14:38:47.371 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.23 2023-10-30T14:38:47.371 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.23 2023-10-30T14:38:47.372 INFO:tasks.workunit.client.0.smithi057.stdout:v0.23 2023-10-30T14:38:48.778 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:38:48.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.933+0000 7f3fc4895640 1 Processor -- start 2023-10-30T14:38:48.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.933+0000 7f3fc4895640 1 -- start start 2023-10-30T14:38:48.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.937+0000 7f3fc4895640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3fc015cd70 0x7f3fc0069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:48.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.937+0000 7f3fc4895640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc015dcc0 con 0x7f3fc015ac70 2023-10-30T14:38:48.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.937+0000 7f3fc4895640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc0159d80 con 0x7f3fc00c8eb0 2023-10-30T14:38:48.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.937+0000 7f3fc4895640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fc0159800 con 0x7f3fc015cd70 2023-10-30T14:38:48.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.937+0000 7f3fbffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3fc015cd70 0x7f3fc0069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:48.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.937+0000 7f3fbeffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f3fc015ac70 0x7f3fc006aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50826/0 (socket says 172.21.15.57:50826) 2023-10-30T14:38:48.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.937+0000 7f3fbffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3fc015cd70 0x7f3fc0069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43344/0 (socket says 172.21.15.57:43344) 2023-10-30T14:38:48.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.937+0000 7f3fbeffd640 1 -- 172.21.15.57:0/100073635 learned_addr learned my addr 172.21.15.57:0/100073635 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:38:48.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.937+0000 7f3fbffff640 1 -- 172.21.15.57:0/100073635 >> v1:172.21.15.88:6789/0 conn(0x7f3fc00c8eb0 legacy=0x7f3fc015a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:48.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.937+0000 7f3fbffff640 1 -- 172.21.15.57:0/100073635 >> v1:172.21.15.57:6789/0 conn(0x7f3fc015ac70 legacy=0x7f3fc006aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:48.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.937+0000 7f3fbffff640 1 -- 172.21.15.57:0/100073635 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc015d1c0 con 0x7f3fc015cd70 2023-10-30T14:38:48.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.937+0000 7f3fbe7fc640 1 -- 172.21.15.57:0/100073635 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997884314 0 0) 0x7f3fb4001120 con 0x7f3fc015ac70 2023-10-30T14:38:48.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.941+0000 7f3fbffff640 1 --2- 172.21.15.57:0/100073635 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc015cd70 0x7f3fc0069e90 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8009d90 tx=0x7f3fb801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13cc680ad2540a24 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:48.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.941+0000 7f3fbe7fc640 1 -- 172.21.15.57:0/100073635 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb80af600 con 0x7f3fc015cd70 2023-10-30T14:38:48.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.941+0000 7f3fbe7fc640 1 -- 172.21.15.57:0/100073635 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb80ab050 con 0x7f3fc015cd70 2023-10-30T14:38:48.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.941+0000 7f3fbe7fc640 1 -- 172.21.15.57:0/100073635 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb80af980 con 0x7f3fc015cd70 2023-10-30T14:38:48.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.941+0000 7f3fc4895640 1 -- 172.21.15.57:0/100073635 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc015cd70 msgr2=0x7f3fc0069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:48.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.941+0000 7f3fc4895640 1 --2- 172.21.15.57:0/100073635 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc015cd70 0x7f3fc0069e90 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8009d90 tx=0x7f3fb801f130 comp rx=0 tx=0).stop 2023-10-30T14:38:48.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.941+0000 7f3fc4895640 1 -- 172.21.15.57:0/100073635 shutdown_connections 2023-10-30T14:38:48.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.941+0000 7f3fc4895640 1 --2- 172.21.15.57:0/100073635 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc015cd70 0x7f3fc0069e90 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:48.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.941+0000 7f3fc4895640 1 -- 172.21.15.57:0/100073635 >> 172.21.15.57:0/100073635 conn(0x7f3fc00cef00 msgr2=0x7f3fc00cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:48.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.941+0000 7f3fc4895640 1 -- 172.21.15.57:0/100073635 shutdown_connections 2023-10-30T14:38:48.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.941+0000 7f3fc4895640 1 -- 172.21.15.57:0/100073635 wait complete. 2023-10-30T14:38:48.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.941+0000 7f3fc4895640 1 Processor -- start 2023-10-30T14:38:48.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.941+0000 7f3fc4895640 1 -- start start 2023-10-30T14:38:48.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fc4895640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fc00c8eb0 0x7f3fc0150880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:48.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fc4895640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc015ac70 0x7f3fc00c51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:48.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fc4895640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fc015cd70 0x7f3fc00ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:48.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fc4895640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc00c5a00 con 0x7f3fc015ac70 2023-10-30T14:38:48.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fc4895640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc00c2b20 con 0x7f3fc015cd70 2023-10-30T14:38:48.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fc4895640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc00c2e00 con 0x7f3fc00c8eb0 2023-10-30T14:38:48.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fbeffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc015ac70 0x7f3fc00c51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:48.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fbeffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc015ac70 0x7f3fc00c51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43356/0 (socket says 172.21.15.57:43356) 2023-10-30T14:38:48.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fbeffd640 1 -- 172.21.15.57:0/2709777826 learned_addr learned my addr 172.21.15.57:0/2709777826 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:38:48.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fbf7fe640 1 --2- 172.21.15.57:0/2709777826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fc00c8eb0 0x7f3fc0150880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:48.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fbffff640 1 --2- 172.21.15.57:0/2709777826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fc015cd70 0x7f3fc00ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:48.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fbeffd640 1 -- 172.21.15.57:0/2709777826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fc00c8eb0 msgr2=0x7f3fc0150880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:48.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fbeffd640 1 --2- 172.21.15.57:0/2709777826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fc00c8eb0 0x7f3fc0150880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:48.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fbeffd640 1 -- 172.21.15.57:0/2709777826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fc015cd70 msgr2=0x7f3fc00ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:48.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fbeffd640 1 --2- 172.21.15.57:0/2709777826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fc015cd70 0x7f3fc00ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:48.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fbeffd640 1 -- 172.21.15.57:0/2709777826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb8009a70 con 0x7f3fc015ac70 2023-10-30T14:38:48.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fbf7fe640 1 --2- 172.21.15.57:0/2709777826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fc00c8eb0 0x7f3fc0150880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:38:48.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fbffff640 1 --2- 172.21.15.57:0/2709777826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fc015cd70 0x7f3fc00ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:38:48.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fbeffd640 1 --2- 172.21.15.57:0/2709777826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc015ac70 0x7f3fc00c51c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f3fb400c780 tx=0x7f3fb400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:48.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fa7fff640 1 -- 172.21.15.57:0/2709777826 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb40072d0 con 0x7f3fc015ac70 2023-10-30T14:38:48.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fa7fff640 1 -- 172.21.15.57:0/2709777826 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb4007430 con 0x7f3fc015ac70 2023-10-30T14:38:48.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fa7fff640 1 -- 172.21.15.57:0/2709777826 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4001d20 con 0x7f3fc015ac70 2023-10-30T14:38:48.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fc4895640 1 -- 172.21.15.57:0/2709777826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc00c30e0 con 0x7f3fc015ac70 2023-10-30T14:38:48.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fc4895640 1 -- 172.21.15.57:0/2709777826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc00c36a0 con 0x7f3fc015ac70 2023-10-30T14:38:48.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.945+0000 7f3fc4895640 1 -- 172.21.15.57:0/2709777826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc006a730 con 0x7f3fc015ac70 2023-10-30T14:38:48.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.949+0000 7f3fa7fff640 1 -- 172.21.15.57:0/2709777826 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3fb40047f0 con 0x7f3fc015ac70 2023-10-30T14:38:48.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.949+0000 7f3fa7fff640 1 --2- 172.21.15.57:0/2709777826 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3fac0664a0 0x7f3fac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:38:48.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.949+0000 7f3fa7fff640 1 -- 172.21.15.57:0/2709777826 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7f3fb40834f0 con 0x7f3fc015ac70 2023-10-30T14:38:48.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.949+0000 7f3fbf7fe640 1 --2- 172.21.15.57:0/2709777826 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3fac0664a0 0x7f3fac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:38:48.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.949+0000 7f3fbf7fe640 1 --2- 172.21.15.57:0/2709777826 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3fac0664a0 0x7f3fac068960 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0000f70 tx=0x7f3fb0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:38:48.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:48.953+0000 7f3fa7fff640 1 -- 172.21.15.57:0/2709777826 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3fb40504a0 con 0x7f3fc015ac70 2023-10-30T14:38:49.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:49.153+0000 7f3fc4895640 1 -- 172.21.15.57:0/2709777826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fc015dcc0 con 0x7f3fc015ac70 2023-10-30T14:38:49.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:49.153+0000 7f3fa7fff640 1 -- 172.21.15.57:0/2709777826 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6534 (secure 0 0 0) 0x7f3fc015dcc0 con 0x7f3fc015ac70 2023-10-30T14:38:49.154 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:38:49.155 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7056,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:37.197465+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:38:49.155 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 133 2023-10-30T14:38:49.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:49.153+0000 7f3fc4895640 1 -- 172.21.15.57:0/2709777826 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3fac0664a0 msgr2=0x7f3fac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:49.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:49.153+0000 7f3fc4895640 1 --2- 172.21.15.57:0/2709777826 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3fac0664a0 0x7f3fac068960 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0000f70 tx=0x7f3fb0009040 comp rx=0 tx=0).stop 2023-10-30T14:38:49.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:49.153+0000 7f3fc4895640 1 -- 172.21.15.57:0/2709777826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc015ac70 msgr2=0x7f3fc00c51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:38:49.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:49.153+0000 7f3fc4895640 1 --2- 172.21.15.57:0/2709777826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc015ac70 0x7f3fc00c51c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f3fb400c780 tx=0x7f3fb400cc40 comp rx=0 tx=0).stop 2023-10-30T14:38:49.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:49.153+0000 7f3fc4895640 1 -- 172.21.15.57:0/2709777826 shutdown_connections 2023-10-30T14:38:49.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:49.153+0000 7f3fc4895640 1 --2- 172.21.15.57:0/2709777826 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3fac0664a0 0x7f3fac068960 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:49.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:49.153+0000 7f3fc4895640 1 --2- 172.21.15.57:0/2709777826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fc015cd70 0x7f3fc00ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:49.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:49.153+0000 7f3fc4895640 1 --2- 172.21.15.57:0/2709777826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc015ac70 0x7f3fc00c51c0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:49.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:49.153+0000 7f3fc4895640 1 --2- 172.21.15.57:0/2709777826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fc00c8eb0 0x7f3fc0150880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:38:49.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:49.153+0000 7f3fc4895640 1 -- 172.21.15.57:0/2709777826 >> 172.21.15.57:0/2709777826 conn(0x7f3fc00cef00 msgr2=0x7f3fc00c65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:38:49.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:49.153+0000 7f3fc4895640 1 -- 172.21.15.57:0/2709777826 shutdown_connections 2023-10-30T14:38:49.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:38:49.153+0000 7f3fc4895640 1 -- 172.21.15.57:0/2709777826 wait complete. 2023-10-30T14:38:49.176 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T14:38:49.176 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 37 secs before thrashing 2023-10-30T14:39:03.784 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 22% (36/163) Updating files: 23% (38/163) Updating files: 24% (40/163) Updating files: 25% (41/163) Updating files: 26% (43/163) Updating files: 27% (45/163) Updating files: 28% (46/163) Updating files: 29% (48/163) Updating files: 30% (49/163) Updating files: 31% (51/163) Updating files: 32% (53/163) Updating files: 33% (54/163) Updating files: 34% (56/163) Updating files: 35% (58/163) Updating files: 36% (59/163) Updating files: 37% (61/163) Updating files: 38% (62/163) Updating files: 39% (64/163) Updating files: 40% (66/163) Updating files: 41% (67/163) Updating files: 42% (69/163) Updating files: 43% (71/163) Updating files: 44% (72/163) Updating files: 45% (74/163) Updating files: 46% (75/163) Updating files: 47% (77/163) Updating files: 48% (79/163) Updating files: 49% (80/163) Updating files: 50% (82/163) Updating files: 51% (84/163) Updating files: 52% (85/163) Updating files: 53% (87/163) Updating files: 54% (89/163) Updating files: 55% (90/163) Updating files: 56% (92/163) Updating files: 57% (93/163) Updating files: 58% (95/163) Updating files: 59% (97/163) Updating files: 60% (98/163) Updating files: 61% (100/163) Updating files: 62% (102/163) Updating files: 63% (103/163) Updating files: 64% (105/163) Updating files: 65% (106/163) Updating files: 66% (108/163) Updating files: 67% (110/163) Updating files: 68% (111/163) Updating files: 69% (113/163) Updating files: 70% (115/163) Updating files: 71% (116/163) Updating files: 72% (118/163) Updating files: 73% (119/163) Updating files: 73% (120/163) Updating files: 74% (121/163) Updating files: 75% (123/163) Updating files: 76% (124/163) Updating files: 77% (126/163) Updating files: 78% (128/163) Updating files: 79% (129/163) Updating files: 80% (131/163) Updating files: 81% (133/163) Updating files: 82% (134/163) Updating files: 83% (136/163) Updating files: 84% (137/163) Updating files: 85% (139/163) Updating files: 86% (141/163) Updating files: 87% (142/163) Updating files: 88% (144/163) Updating files: 89% (146/163) Updating files: 90% (147/163) Updating files: 91% (149/163) Updating files: 92% (150/163) Updating files: 93% (152/163) Updating files: 94% (154/163) Updating files: 95% (155/163) Updating files: 96% (157/163) Updating files: 97% (159/163) Updating files: 98% (160/163) Updating files: 99% (162/163) Updating files: 100% (163/163) Updating files: 100% (163/163), done. 2023-10-30T14:39:03.841 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 5d1d8d0c460 v0.23 2023-10-30T14:39:03.846 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.23 2023-10-30T14:39:03.885 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 24 -eq 48 ] 2023-10-30T14:39:03.886 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.24 2023-10-30T14:39:03.886 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.24 2023-10-30T14:39:03.886 INFO:tasks.workunit.client.0.smithi057.stdout:v0.24 2023-10-30T14:39:03.887 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.24 2023-10-30T14:39:07.726 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:39:07.731 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:39:09.090 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 36% (67/185) Updating files: 36% (68/185) Updating files: 37% (69/185) Updating files: 38% (71/185) Updating files: 39% (73/185) Updating files: 40% (74/185) Updating files: 41% (76/185) Updating files: 42% (78/185) Updating files: 43% (80/185) Updating files: 44% (82/185) Updating files: 45% (84/185) Updating files: 46% (86/185) Updating files: 47% (87/185) Updating files: 48% (89/185) Updating files: 49% (91/185) Updating files: 50% (93/185) Updating files: 51% (95/185) Updating files: 52% (97/185) Updating files: 53% (99/185) Updating files: 54% (100/185) Updating files: 55% (102/185) Updating files: 56% (104/185) Updating files: 57% (106/185) Updating files: 58% (108/185) Updating files: 59% (110/185) Updating files: 60% (111/185) Updating files: 61% (113/185) Updating files: 62% (115/185) Updating files: 63% (117/185) Updating files: 64% (119/185) Updating files: 65% (121/185) Updating files: 66% (123/185) Updating files: 67% (124/185) Updating files: 68% (126/185) Updating files: 69% (128/185) Updating files: 70% (130/185) Updating files: 71% (132/185) Updating files: 72% (134/185) Updating files: 73% (136/185) Updating files: 74% (137/185) Updating files: 75% (139/185) Updating files: 75% (140/185) Updating files: 76% (141/185) Updating files: 77% (143/185) Updating files: 78% (145/185) Updating files: 79% (147/185) Updating files: 80% (148/185) Updating files: 81% (150/185) Updating files: 82% (152/185) Updating files: 83% (154/185) Updating files: 84% (156/185) Updating files: 85% (158/185) Updating files: 86% (160/185) Updating files: 87% (161/185) Updating files: 88% (163/185) Updating files: 89% (165/185) Updating files: 90% (167/185) Updating files: 91% (169/185) Updating files: 92% (171/185) Updating files: 93% (173/185) Updating files: 94% (174/185) Updating files: 95% (176/185) Updating files: 96% (178/185) Updating files: 97% (180/185) Updating files: 98% (182/185) Updating files: 99% (184/185) Updating files: 100% (185/185) Updating files: 100% (185/185), done. 2023-10-30T14:39:09.125 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 180a4176035 v0.24 2023-10-30T14:39:09.130 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.24 2023-10-30T14:39:09.725 INFO:tasks.workunit.client.0.smithi057.stdout:v0.25 2023-10-30T14:39:09.726 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 25 -eq 48 ] 2023-10-30T14:39:09.726 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.25 2023-10-30T14:39:09.726 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.25 2023-10-30T14:39:09.726 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.25 2023-10-30T14:39:21.645 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 4% (27/622) Updating files: 5% (32/622) Updating files: 6% (38/622) Updating files: 7% (44/622) Updating files: 8% (50/622) Updating files: 9% (56/622) Updating files: 10% (63/622) Updating files: 10% (66/622) Updating files: 11% (69/622) Updating files: 12% (75/622) Updating files: 13% (81/622) Updating files: 14% (88/622) Updating files: 15% (94/622) Updating files: 15% (97/622) Updating files: 16% (100/622) Updating files: 17% (106/622) Updating files: 18% (112/622) Updating files: 19% (119/622) Updating files: 20% (125/622) Updating files: 21% (131/622) Updating files: 22% (137/622) Updating files: 23% (144/622) Updating files: 24% (150/622) Updating files: 25% (156/622) Updating files: 26% (162/622) Updating files: 26% (165/622) Updating files: 27% (168/622) Updating files: 28% (175/622) Updating files: 29% (181/622) Updating files: 30% (187/622) Updating files: 31% (193/622) Updating files: 32% (200/622) Updating files: 33% (206/622) Updating files: 33% (208/622) Updating files: 34% (212/622) Updating files: 35% (218/622) Updating files: 36% (224/622) Updating files: 37% (231/622) Updating files: 38% (237/622) Updating files: 39% (243/622) Updating files: 40% (249/622) Updating files: 41% (256/622) Updating files: 42% (262/622) Updating files: 43% (268/622) Updating files: 43% (271/622) Updating files: 44% (274/622) Updating files: 45% (280/622) Updating files: 46% (287/622) Updating files: 47% (293/622) Updating files: 48% (299/622) Updating files: 49% (305/622) Updating files: 50% (311/622) Updating files: 51% (318/622) Updating files: 52% (324/622) Updating files: 53% (330/622) Updating files: 54% (336/622) Updating files: 55% (343/622) Updating files: 56% (349/622) Updating files: 57% (355/622) Updating files: 58% (361/622) Updating files: 59% (367/622) Updating files: 60% (374/622) Updating files: 61% (380/622) Updating files: 62% (386/622) Updating files: 62% (390/622) Updating files: 63% (392/622) Updating files: 64% (399/622) Updating files: 65% (405/622) Updating files: 66% (411/622) Updating files: 66% (416/622) Updating files: 67% (417/622) Updating files: 68% (423/622) Updating files: 68% (429/622) Updating files: 69% (430/622) Updating files: 70% (436/622) Updating files: 71% (442/622) Updating files: 72% (448/622) Updating files: 73% (455/622) Updating files: 74% (461/622) Updating files: 75% (467/622) Updating files: 76% (473/622) Updating files: 77% (479/622) Updating files: 78% (486/622) Updating files: 79% (492/622) Updating files: 80% (498/622) Updating files: 81% (504/622) Updating files: 82% (511/622) Updating files: 83% (517/622) Updating files: 84% (523/622) Updating files: 85% (529/622) Updating files: 86% (535/622) Updating files: 87% (542/622) Updating files: 88% (548/622) Updating files: 89% (554/622) Updating files: 90% (560/622) Updating files: 91% (567/622) Updating files: 92% (573/622) Updating files: 93% (579/622) Updating files: 94% (585/622) Updating files: 95% (591/622) Updating files: 96% (598/622) Updating files: 97% (604/622) Updating files: 98% (610/622) Updating files: 99% (616/622) Updating files: 100% (622/622) Updating files: 100% (622/622), done. 2023-10-30T14:39:21.710 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 3fa121bf0a8 v0.25 2023-10-30T14:39:21.715 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.25 2023-10-30T14:39:21.981 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 26 -eq 48 ] 2023-10-30T14:39:21.981 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.26 2023-10-30T14:39:21.981 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.26 2023-10-30T14:39:21.982 INFO:tasks.workunit.client.0.smithi057.stdout:v0.26 2023-10-30T14:39:21.982 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.26 2023-10-30T14:39:25.551 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 65% (157/239) Updating files: 66% (158/239) Updating files: 67% (161/239) Updating files: 68% (163/239) Updating files: 69% (165/239) Updating files: 70% (168/239) Updating files: 71% (170/239) Updating files: 72% (173/239) Updating files: 73% (175/239) Updating files: 74% (177/239) Updating files: 75% (180/239) Updating files: 76% (182/239) Updating files: 76% (183/239) Updating files: 77% (185/239) Updating files: 78% (187/239) Updating files: 79% (189/239) Updating files: 80% (192/239) Updating files: 81% (194/239) Updating files: 82% (196/239) Updating files: 83% (199/239) Updating files: 84% (201/239) Updating files: 85% (204/239) Updating files: 86% (206/239) Updating files: 87% (208/239) Updating files: 88% (211/239) Updating files: 89% (213/239) Updating files: 90% (216/239) Updating files: 91% (218/239) Updating files: 92% (220/239) Updating files: 92% (221/239) Updating files: 93% (223/239) Updating files: 94% (225/239) Updating files: 95% (228/239) Updating files: 96% (230/239) Updating files: 97% (232/239) Updating files: 98% (235/239) Updating files: 99% (237/239) Updating files: 100% (239/239) Updating files: 100% (239/239), done. 2023-10-30T14:39:25.587 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 9981ff90968 v0.26 2023-10-30T14:39:25.593 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.26 2023-10-30T14:39:25.865 INFO:tasks.workunit.client.0.smithi057.stdout:v0.27 2023-10-30T14:39:25.865 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 27 -eq 48 ] 2023-10-30T14:39:25.866 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.27 2023-10-30T14:39:25.866 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.27 2023-10-30T14:39:25.866 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.27 2023-10-30T14:39:26.177 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:39:26.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.324+0000 7f29d7068640 1 Processor -- start 2023-10-30T14:39:26.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.324+0000 7f29d7068640 1 -- start start 2023-10-30T14:39:26.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.324+0000 7f29d7068640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f29d0150830 0x7f29d006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:26.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.324+0000 7f29d7068640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f29d00cb7d0 0x7f29d00c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:26.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.324+0000 7f29d7068640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29d00c3e40 0x7f29d0070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:26.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.324+0000 7f29d7068640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d00c2310 con 0x7f29d00cb7d0 2023-10-30T14:39:26.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.324+0000 7f29d7068640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d0156710 con 0x7f29d0150830 2023-10-30T14:39:26.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.324+0000 7f29d7068640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d00c4360 con 0x7f29d00c3e40 2023-10-30T14:39:26.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.324+0000 7f29cffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f29d00cb7d0 0x7f29d00c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:26.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.324+0000 7f29cffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f29d00cb7d0 0x7f29d00c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53092/0 (socket says 172.21.15.57:53092) 2023-10-30T14:39:26.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.324+0000 7f29cffff640 1 -- 172.21.15.57:0/2652256835 learned_addr learned my addr 172.21.15.57:0/2652256835 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:26.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.328+0000 7f29d4e04640 1 --2- 172.21.15.57:0/2652256835 >> v2:172.21.15.88:3300/0 conn(0x7f29d0150830 0x7f29d006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:26.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.328+0000 7f29d5605640 1 --2- 172.21.15.57:0/2652256835 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29d00c3e40 0x7f29d0070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:26.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.328+0000 7f29cffff640 1 -- 172.21.15.57:0/2652256835 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29d00c3e40 msgr2=0x7f29d0070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:26.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.328+0000 7f29cffff640 1 --2- 172.21.15.57:0/2652256835 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29d00c3e40 0x7f29d0070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.328+0000 7f29cffff640 1 -- 172.21.15.57:0/2652256835 >> v2:172.21.15.88:3300/0 conn(0x7f29d0150830 msgr2=0x7f29d006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:26.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.328+0000 7f29cffff640 1 --2- 172.21.15.57:0/2652256835 >> v2:172.21.15.88:3300/0 conn(0x7f29d0150830 0x7f29d006b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.328+0000 7f29cffff640 1 -- 172.21.15.57:0/2652256835 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d01569b0 con 0x7f29d00cb7d0 2023-10-30T14:39:26.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.328+0000 7f29cffff640 1 --2- 172.21.15.57:0/2652256835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29d00cb7d0 0x7f29d00c3900 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f29d006fa40 tx=0x7f29bc0213c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2abfcefc825db9a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:26.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.328+0000 7f29cf7fe640 1 -- 172.21.15.57:0/2652256835 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc0b1600 con 0x7f29d00cb7d0 2023-10-30T14:39:26.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.328+0000 7f29cf7fe640 1 -- 172.21.15.57:0/2652256835 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29bc0ad050 con 0x7f29d00cb7d0 2023-10-30T14:39:26.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.328+0000 7f29cf7fe640 1 -- 172.21.15.57:0/2652256835 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc0b1980 con 0x7f29d00cb7d0 2023-10-30T14:39:26.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d7068640 1 -- 172.21.15.57:0/2652256835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29d00cb7d0 msgr2=0x7f29d00c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:26.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d7068640 1 --2- 172.21.15.57:0/2652256835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29d00cb7d0 0x7f29d00c3900 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f29d006fa40 tx=0x7f29bc0213c0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d7068640 1 -- 172.21.15.57:0/2652256835 shutdown_connections 2023-10-30T14:39:26.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d7068640 1 --2- 172.21.15.57:0/2652256835 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29d00c3e40 0x7f29d0070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d7068640 1 --2- 172.21.15.57:0/2652256835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29d00cb7d0 0x7f29d00c3900 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d7068640 1 --2- 172.21.15.57:0/2652256835 >> v2:172.21.15.88:3300/0 conn(0x7f29d0150830 0x7f29d006b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d7068640 1 -- 172.21.15.57:0/2652256835 >> 172.21.15.57:0/2652256835 conn(0x7f29d00cac70 msgr2=0x7f29d00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:26.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d7068640 1 -- 172.21.15.57:0/2652256835 shutdown_connections 2023-10-30T14:39:26.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d7068640 1 -- 172.21.15.57:0/2652256835 wait complete. 2023-10-30T14:39:26.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d7068640 1 Processor -- start 2023-10-30T14:39:26.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d7068640 1 -- start start 2023-10-30T14:39:26.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d7068640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29d00c3e40 0x7f29d006bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:26.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d7068640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29d00cb7d0 0x7f29d01effc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:26.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d7068640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29d0150830 0x7f29d01f2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:26.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d7068640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d01532d0 con 0x7f29d00c3e40 2023-10-30T14:39:26.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d7068640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d0151520 con 0x7f29d0150830 2023-10-30T14:39:26.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d7068640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d0153e10 con 0x7f29d00cb7d0 2023-10-30T14:39:26.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d4e04640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29d00c3e40 0x7f29d006bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:26.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d4e04640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29d00c3e40 0x7f29d006bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53096/0 (socket says 172.21.15.57:53096) 2023-10-30T14:39:26.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d4e04640 1 -- 172.21.15.57:0/91258477 learned_addr learned my addr 172.21.15.57:0/91258477 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:26.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29cffff640 1 --2- 172.21.15.57:0/91258477 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29d00cb7d0 0x7f29d01effc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:26.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d5605640 1 --2- 172.21.15.57:0/91258477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29d0150830 0x7f29d01f2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:26.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d4e04640 1 -- 172.21.15.57:0/91258477 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29d00cb7d0 msgr2=0x7f29d01effc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:26.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.332+0000 7f29d4e04640 1 --2- 172.21.15.57:0/91258477 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29d00cb7d0 0x7f29d01effc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.336+0000 7f29d4e04640 1 -- 172.21.15.57:0/91258477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29d0150830 msgr2=0x7f29d01f2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:26.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.336+0000 7f29d4e04640 1 --2- 172.21.15.57:0/91258477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29d0150830 0x7f29d01f2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.336+0000 7f29d4e04640 1 -- 172.21.15.57:0/91258477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29bc00b770 con 0x7f29d00c3e40 2023-10-30T14:39:26.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.336+0000 7f29d4e04640 1 --2- 172.21.15.57:0/91258477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29d00c3e40 0x7f29d006bf20 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f29c400d800 tx=0x7f29c400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:26.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.336+0000 7f29cd7fa640 1 -- 172.21.15.57:0/91258477 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c400b6c0 con 0x7f29d00c3e40 2023-10-30T14:39:26.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.336+0000 7f29cd7fa640 1 -- 172.21.15.57:0/91258477 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c400b820 con 0x7f29d00c3e40 2023-10-30T14:39:26.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.336+0000 7f29cd7fa640 1 -- 172.21.15.57:0/91258477 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c400bb90 con 0x7f29d00c3e40 2023-10-30T14:39:26.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.336+0000 7f29d7068640 1 -- 172.21.15.57:0/91258477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d0152510 con 0x7f29d00c3e40 2023-10-30T14:39:26.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.336+0000 7f29d7068640 1 -- 172.21.15.57:0/91258477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d0152ad0 con 0x7f29d00c3e40 2023-10-30T14:39:26.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.336+0000 7f29d7068640 1 -- 172.21.15.57:0/91258477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d004e170 con 0x7f29d00c3e40 2023-10-30T14:39:26.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.336+0000 7f29cd7fa640 1 -- 172.21.15.57:0/91258477 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f29c400bcf0 con 0x7f29d00c3e40 2023-10-30T14:39:26.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.336+0000 7f29cd7fa640 1 --2- 172.21.15.57:0/91258477 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29b40664d0 0x7f29b4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:26.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.336+0000 7f29cd7fa640 1 -- 172.21.15.57:0/91258477 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7f29c40847c0 con 0x7f29d00c3e40 2023-10-30T14:39:26.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.336+0000 7f29cffff640 1 --2- 172.21.15.57:0/91258477 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29b40664d0 0x7f29b4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:26.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.340+0000 7f29cffff640 1 --2- 172.21.15.57:0/91258477 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29b40664d0 0x7f29b4068990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f29bc0b3000 tx=0x7f29bc0ac000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:26.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.340+0000 7f29cd7fa640 1 -- 172.21.15.57:0/91258477 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f29c4051780 con 0x7f29d00c3e40 2023-10-30T14:39:26.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.536+0000 7f29d7068640 1 -- 172.21.15.57:0/91258477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29d0070300 con 0x7f29d00c3e40 2023-10-30T14:39:26.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.536+0000 7f29cd7fa640 1 -- 172.21.15.57:0/91258477 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+6537 (secure 0 0 0) 0x7f29c4054da0 con 0x7f29d00c3e40 2023-10-30T14:39:26.541 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:39:26.541 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7056,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:58.376369+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:39:26.541 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 135 2023-10-30T14:39:26.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.540+0000 7f29d7068640 1 -- 172.21.15.57:0/91258477 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29b40664d0 msgr2=0x7f29b4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:26.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.540+0000 7f29d7068640 1 --2- 172.21.15.57:0/91258477 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29b40664d0 0x7f29b4068990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f29bc0b3000 tx=0x7f29bc0ac000 comp rx=0 tx=0).stop 2023-10-30T14:39:26.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.540+0000 7f29d7068640 1 -- 172.21.15.57:0/91258477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29d00c3e40 msgr2=0x7f29d006bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:26.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.540+0000 7f29d7068640 1 --2- 172.21.15.57:0/91258477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29d00c3e40 0x7f29d006bf20 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f29c400d800 tx=0x7f29c400dcc0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.540+0000 7f29d7068640 1 -- 172.21.15.57:0/91258477 shutdown_connections 2023-10-30T14:39:26.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.540+0000 7f29d7068640 1 --2- 172.21.15.57:0/91258477 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29b40664d0 0x7f29b4068990 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.540+0000 7f29d7068640 1 --2- 172.21.15.57:0/91258477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29d0150830 0x7f29d01f2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.540+0000 7f29d7068640 1 --2- 172.21.15.57:0/91258477 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29d00cb7d0 0x7f29d01effc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.540+0000 7f29d7068640 1 --2- 172.21.15.57:0/91258477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29d00c3e40 0x7f29d006bf20 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.540+0000 7f29d7068640 1 -- 172.21.15.57:0/91258477 >> 172.21.15.57:0/91258477 conn(0x7f29d00cac70 msgr2=0x7f29d014e840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:26.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.540+0000 7f29d7068640 1 -- 172.21.15.57:0/91258477 shutdown_connections 2023-10-30T14:39:26.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.540+0000 7f29d7068640 1 -- 172.21.15.57:0/91258477 wait complete. 2023-10-30T14:39:26.562 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=2) 2023-10-30T14:39:26.562 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T14:39:26.563 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:39:26.578 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T14:39:26.578 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:39:26.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9da086b640 1 Processor -- start 2023-10-30T14:39:26.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9da086b640 1 -- start start 2023-10-30T14:39:26.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9da086b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9d9c153c80 0x7f9d9c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:26.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9da086b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d9c0c4a10 0x7f9d9c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:26.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9da086b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9c159f20 con 0x7f9d9c15a3b0 2023-10-30T14:39:26.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9da086b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d9c15dec0 con 0x7f9d9c153c80 2023-10-30T14:39:26.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9da086b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c1599a0 con 0x7f9d9c0c4a10 2023-10-30T14:39:26.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9ad9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f9d9c15a3b0 0x7f9d9c0ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:38810/0 (socket says 172.21.15.57:38810) 2023-10-30T14:39:26.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9ad9d640 1 -- 172.21.15.57:0/1981847692 learned_addr learned my addr 172.21.15.57:0/1981847692 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:39:26.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d99d9b640 1 --2- 172.21.15.57:0/1981847692 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d9c0c4a10 0x7f9d9c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:26.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9a59c640 1 --2- 172.21.15.57:0/1981847692 >> v2:172.21.15.88:3300/0 conn(0x7f9d9c153c80 0x7f9d9c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:26.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9959a640 1 -- 172.21.15.57:0/1981847692 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8658896 0 0) 0x7f9d9c159f20 con 0x7f9d9c15a3b0 2023-10-30T14:39:26.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9959a640 1 -- 172.21.15.57:0/1981847692 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d84003530 con 0x7f9d9c15a3b0 2023-10-30T14:39:26.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9959a640 1 -- 172.21.15.57:0/1981847692 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2707490022 0 0) 0x7f9d84003530 con 0x7f9d9c15a3b0 2023-10-30T14:39:26.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9959a640 1 -- 172.21.15.57:0/1981847692 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d9c0c4a10 msgr2=0x7f9d9c0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:26.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9959a640 1 --2- 172.21.15.57:0/1981847692 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d9c0c4a10 0x7f9d9c0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9959a640 1 -- 172.21.15.57:0/1981847692 >> v2:172.21.15.88:3300/0 conn(0x7f9d9c153c80 msgr2=0x7f9d9c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:26.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9959a640 1 --2- 172.21.15.57:0/1981847692 >> v2:172.21.15.88:3300/0 conn(0x7f9d9c153c80 0x7f9d9c15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9959a640 1 -- 172.21.15.57:0/1981847692 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c0c47e0 con 0x7f9d9c15a3b0 2023-10-30T14:39:26.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9959a640 1 -- 172.21.15.57:0/1981847692 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f9d94001a00 con 0x7f9d9c15a3b0 2023-10-30T14:39:26.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9959a640 1 -- 172.21.15.57:0/1981847692 >> v1:172.21.15.57:6789/0 conn(0x7f9d9c15a3b0 legacy=0x7f9d9c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:26.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9959a640 1 --2- 172.21.15.57:0/1981847692 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d84003980 0x7f9d84003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:26.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9959a640 1 --2- 172.21.15.57:0/1981847692 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d9c0c4a10 0x7f9d840047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:26.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9959a640 1 --2- 172.21.15.57:0/1981847692 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d84004d10 0x7f9d84005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:26.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9959a640 1 -- 172.21.15.57:0/1981847692 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c15dec0 con 0x7f9d84004d10 2023-10-30T14:39:26.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9959a640 1 -- 172.21.15.57:0/1981847692 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c1599a0 con 0x7f9d84003980 2023-10-30T14:39:26.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9959a640 1 -- 172.21.15.57:0/1981847692 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8400b990 con 0x7f9d9c0c4a10 2023-10-30T14:39:26.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9ad9d640 1 --2- 172.21.15.57:0/1981847692 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d84004d10 0x7f9d84005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:26.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d99d9b640 1 --2- 172.21.15.57:0/1981847692 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d9c0c4a10 0x7f9d840047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:26.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9a59c640 1 --2- 172.21.15.57:0/1981847692 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d84003980 0x7f9d84003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:26.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9ad9d640 1 -- 172.21.15.57:0/1981847692 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d9c0c4a10 msgr2=0x7f9d840047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:26.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9ad9d640 1 --2- 172.21.15.57:0/1981847692 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d9c0c4a10 0x7f9d840047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9ad9d640 1 -- 172.21.15.57:0/1981847692 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d84003980 msgr2=0x7f9d84003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:26.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9ad9d640 1 --2- 172.21.15.57:0/1981847692 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d84003980 0x7f9d84003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.720+0000 7f9d9ad9d640 1 -- 172.21.15.57:0/1981847692 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d84003530 con 0x7f9d84004d10 2023-10-30T14:39:26.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.724+0000 7f9d9ad9d640 1 --2- 172.21.15.57:0/1981847692 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d84004d10 0x7f9d84005190 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f9d9400a860 tx=0x7f9d9401d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b5af37412e071e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:26.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.724+0000 7f9d9959a640 1 -- 172.21.15.57:0/1981847692 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d940ad630 con 0x7f9d84004d10 2023-10-30T14:39:26.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.724+0000 7f9d9959a640 1 -- 172.21.15.57:0/1981847692 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d940a8070 con 0x7f9d84004d10 2023-10-30T14:39:26.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.724+0000 7f9d9959a640 1 -- 172.21.15.57:0/1981847692 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d940ad9f0 con 0x7f9d84004d10 2023-10-30T14:39:26.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.724+0000 7f9da086b640 1 -- 172.21.15.57:0/1981847692 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d84004d10 msgr2=0x7f9d84005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:26.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.724+0000 7f9da086b640 1 --2- 172.21.15.57:0/1981847692 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d84004d10 0x7f9d84005190 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f9d9400a860 tx=0x7f9d9401d360 comp rx=0 tx=0).stop 2023-10-30T14:39:26.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.724+0000 7f9d9ad9d640 1 -- 172.21.15.57:0/1981847692 reap_dead start 2023-10-30T14:39:26.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.724+0000 7f9da086b640 1 -- 172.21.15.57:0/1981847692 shutdown_connections 2023-10-30T14:39:26.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.724+0000 7f9da086b640 1 -- 172.21.15.57:0/1981847692 >> 172.21.15.57:0/1981847692 conn(0x7f9d9c0715c0 msgr2=0x7f9d9c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:26.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.724+0000 7f9da086b640 1 -- 172.21.15.57:0/1981847692 shutdown_connections 2023-10-30T14:39:26.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.724+0000 7f9da086b640 1 -- 172.21.15.57:0/1981847692 wait complete. 2023-10-30T14:39:26.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9da086b640 1 Processor -- start 2023-10-30T14:39:26.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9da086b640 1 -- start start 2023-10-30T14:39:26.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9da086b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d84004d10 0x7f9d9c1ef3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:26.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9da086b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d9c0cdb20 0x7f9d9c15bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:26.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9da086b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d9c0cdf80 0x7f9d9c15c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:26.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9da086b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c1ef930 con 0x7f9d84004d10 2023-10-30T14:39:26.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9da086b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c0ce560 con 0x7f9d9c0cdb20 2023-10-30T14:39:26.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9da086b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c15d0d0 con 0x7f9d9c0cdf80 2023-10-30T14:39:26.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9d9a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d84004d10 0x7f9d9c1ef3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:26.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9d9a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d84004d10 0x7f9d9c1ef3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53118/0 (socket says 172.21.15.57:53118) 2023-10-30T14:39:26.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9d9a59c640 1 -- 172.21.15.57:0/2741624662 learned_addr learned my addr 172.21.15.57:0/2741624662 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:26.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9d9ad9d640 1 --2- 172.21.15.57:0/2741624662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d9c0cdf80 0x7f9d9c15c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:26.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9d99d9b640 1 --2- 172.21.15.57:0/2741624662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d9c0cdb20 0x7f9d9c15bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:26.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9d9a59c640 1 -- 172.21.15.57:0/2741624662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d9c0cdf80 msgr2=0x7f9d9c15c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:26.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9d9a59c640 1 --2- 172.21.15.57:0/2741624662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d9c0cdf80 0x7f9d9c15c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9d9a59c640 1 -- 172.21.15.57:0/2741624662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d9c0cdb20 msgr2=0x7f9d9c15bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:26.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9d9a59c640 1 --2- 172.21.15.57:0/2741624662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d9c0cdb20 0x7f9d9c15bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9d9a59c640 1 -- 172.21.15.57:0/2741624662 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c0cd1c0 con 0x7f9d84004d10 2023-10-30T14:39:26.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9d99d9b640 1 --2- 172.21.15.57:0/2741624662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d9c0cdb20 0x7f9d9c15bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:39:26.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.728+0000 7f9d9a59c640 1 --2- 172.21.15.57:0/2741624662 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d84004d10 0x7f9d9c1ef3f0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c0119d0 tx=0x7f9d8c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:26.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.732+0000 7f9d8affd640 1 -- 172.21.15.57:0/2741624662 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c00fb50 con 0x7f9d84004d10 2023-10-30T14:39:26.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.732+0000 7f9d8affd640 1 -- 172.21.15.57:0/2741624662 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8c00fcb0 con 0x7f9d84004d10 2023-10-30T14:39:26.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.732+0000 7f9d8affd640 1 -- 172.21.15.57:0/2741624662 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c014440 con 0x7f9d84004d10 2023-10-30T14:39:26.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.732+0000 7f9da086b640 1 -- 172.21.15.57:0/2741624662 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d9c0cd4a0 con 0x7f9d84004d10 2023-10-30T14:39:26.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.732+0000 7f9da086b640 1 -- 172.21.15.57:0/2741624662 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9c0b4f60 con 0x7f9d84004d10 2023-10-30T14:39:26.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.732+0000 7f9da086b640 1 -- 172.21.15.57:0/2741624662 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9c0cd690 con 0x7f9d84004d10 2023-10-30T14:39:26.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.732+0000 7f9d8affd640 1 -- 172.21.15.57:0/2741624662 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9d8c0145a0 con 0x7f9d84004d10 2023-10-30T14:39:26.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.732+0000 7f9d8affd640 1 --2- 172.21.15.57:0/2741624662 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9d7c0664d0 0x7f9d7c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:26.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.732+0000 7f9d8affd640 1 -- 172.21.15.57:0/2741624662 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7f9d8c084c60 con 0x7f9d84004d10 2023-10-30T14:39:26.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.732+0000 7f9d99d9b640 1 --2- 172.21.15.57:0/2741624662 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9d7c0664d0 0x7f9d7c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:26.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.736+0000 7f9d99d9b640 1 --2- 172.21.15.57:0/2741624662 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9d7c0664d0 0x7f9d7c068990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c0cb110 tx=0x7f9d90008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:26.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.736+0000 7f9d8affd640 1 -- 172.21.15.57:0/2741624662 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9d8c051ba0 con 0x7f9d84004d10 2023-10-30T14:39:26.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.928+0000 7f9da086b640 1 -- 172.21.15.57:0/2741624662 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d9c0ce360 con 0x7f9d84004d10 2023-10-30T14:39:26.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.928+0000 7f9d8affd640 1 -- 172.21.15.57:0/2741624662 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+6537 (secure 0 0 0) 0x7f9d8c022070 con 0x7f9d84004d10 2023-10-30T14:39:26.930 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:39:26.931 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7056,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:58.376369+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:39:26.931 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 135 2023-10-30T14:39:26.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.928+0000 7f9da086b640 1 -- 172.21.15.57:0/2741624662 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9d7c0664d0 msgr2=0x7f9d7c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:26.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.928+0000 7f9da086b640 1 --2- 172.21.15.57:0/2741624662 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9d7c0664d0 0x7f9d7c068990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c0cb110 tx=0x7f9d90008040 comp rx=0 tx=0).stop 2023-10-30T14:39:26.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.928+0000 7f9da086b640 1 -- 172.21.15.57:0/2741624662 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d84004d10 msgr2=0x7f9d9c1ef3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:26.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.928+0000 7f9da086b640 1 --2- 172.21.15.57:0/2741624662 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d84004d10 0x7f9d9c1ef3f0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c0119d0 tx=0x7f9d8c011e90 comp rx=0 tx=0).stop 2023-10-30T14:39:26.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.928+0000 7f9da086b640 1 -- 172.21.15.57:0/2741624662 shutdown_connections 2023-10-30T14:39:26.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.928+0000 7f9da086b640 1 --2- 172.21.15.57:0/2741624662 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9d7c0664d0 0x7f9d7c068990 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.928+0000 7f9da086b640 1 --2- 172.21.15.57:0/2741624662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d9c0cdf80 0x7f9d9c15c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.928+0000 7f9da086b640 1 --2- 172.21.15.57:0/2741624662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d9c0cdb20 0x7f9d9c15bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.928+0000 7f9da086b640 1 --2- 172.21.15.57:0/2741624662 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d84004d10 0x7f9d9c1ef3f0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:26.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.928+0000 7f9da086b640 1 -- 172.21.15.57:0/2741624662 >> 172.21.15.57:0/2741624662 conn(0x7f9d9c0715c0 msgr2=0x7f9d9c150ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:26.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.928+0000 7f9da086b640 1 -- 172.21.15.57:0/2741624662 shutdown_connections 2023-10-30T14:39:26.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:26.928+0000 7f9da086b640 1 -- 172.21.15.57:0/2741624662 wait complete. 2023-10-30T14:39:26.951 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:39:28.952 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:39:29.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.104+0000 7f6adb846640 1 Processor -- start 2023-10-30T14:39:29.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.104+0000 7f6adb846640 1 -- start start 2023-10-30T14:39:29.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.104+0000 7f6adb846640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ad406b760 0x7f6ad415e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:29.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.104+0000 7f6adb846640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6ad406bb40 0x7f6ad4157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:29.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.104+0000 7f6adb846640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad4071e00 con 0x7f6ad406c420 2023-10-30T14:39:29.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.104+0000 7f6adb846640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad4070020 con 0x7f6ad406bb40 2023-10-30T14:39:29.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.104+0000 7f6adb846640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad415eb90 con 0x7f6ad406b760 2023-10-30T14:39:29.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.104+0000 7f6ad8de1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6ad406bb40 0x7f6ad4157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:29.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.104+0000 7f6ad9de3640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f6ad406c420 0x7f6ad40c57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:38824/0 (socket says 172.21.15.57:38824) 2023-10-30T14:39:29.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.104+0000 7f6ad9de3640 1 -- 172.21.15.57:0/2817971188 learned_addr learned my addr 172.21.15.57:0/2817971188 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:39:29.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.108+0000 7f6ad95e2640 1 --2- 172.21.15.57:0/2817971188 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ad406b760 0x7f6ad415e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:29.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.108+0000 7f6ad8de1640 1 -- 172.21.15.57:0/2817971188 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ad406b760 msgr2=0x7f6ad415e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:29.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.108+0000 7f6ad8de1640 1 --2- 172.21.15.57:0/2817971188 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ad406b760 0x7f6ad415e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:29.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.108+0000 7f6ad8de1640 1 -- 172.21.15.57:0/2817971188 >> v1:172.21.15.57:6789/0 conn(0x7f6ad406c420 legacy=0x7f6ad40c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:29.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.108+0000 7f6ad8de1640 1 -- 172.21.15.57:0/2817971188 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad4070f10 con 0x7f6ad406bb40 2023-10-30T14:39:29.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.108+0000 7f6acbfff640 1 -- 172.21.15.57:0/2817971188 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251486061 0 0) 0x7f6ad4071e00 con 0x7f6ad406c420 2023-10-30T14:39:29.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.108+0000 7f6ad8de1640 1 --2- 172.21.15.57:0/2817971188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ad406bb40 0x7f6ad4157a20 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40097c0 tx=0x7f6ac401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca434bfde75a5162 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:29.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.108+0000 7f6acbfff640 1 -- 172.21.15.57:0/2817971188 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac40ab670 con 0x7f6ad406bb40 2023-10-30T14:39:29.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.108+0000 7f6acbfff640 1 -- 172.21.15.57:0/2817971188 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac40ab7d0 con 0x7f6ad406bb40 2023-10-30T14:39:29.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.108+0000 7f6acbfff640 1 -- 172.21.15.57:0/2817971188 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac40abb60 con 0x7f6ad406bb40 2023-10-30T14:39:29.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6adb846640 1 -- 172.21.15.57:0/2817971188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ad406bb40 msgr2=0x7f6ad4157a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:29.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6adb846640 1 --2- 172.21.15.57:0/2817971188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ad406bb40 0x7f6ad4157a20 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40097c0 tx=0x7f6ac401c450 comp rx=0 tx=0).stop 2023-10-30T14:39:29.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6adb846640 1 -- 172.21.15.57:0/2817971188 shutdown_connections 2023-10-30T14:39:29.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6adb846640 1 --2- 172.21.15.57:0/2817971188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ad406bb40 0x7f6ad4157a20 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:29.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6adb846640 1 --2- 172.21.15.57:0/2817971188 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ad406b760 0x7f6ad415e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:29.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6adb846640 1 -- 172.21.15.57:0/2817971188 >> 172.21.15.57:0/2817971188 conn(0x7f6ad40cac70 msgr2=0x7f6ad40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:29.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6adb846640 1 -- 172.21.15.57:0/2817971188 shutdown_connections 2023-10-30T14:39:29.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6adb846640 1 -- 172.21.15.57:0/2817971188 wait complete. 2023-10-30T14:39:29.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6adb846640 1 Processor -- start 2023-10-30T14:39:29.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6adb846640 1 -- start start 2023-10-30T14:39:29.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6adb846640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ad406b760 0x7f6ad4146040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:29.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6adb846640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ad406bb40 0x7f6ad41f9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:29.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6adb846640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ad406c420 0x7f6ad41f9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:29.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6adb846640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad415c500 con 0x7f6ad406c420 2023-10-30T14:39:29.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6adb846640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad415d1c0 con 0x7f6ad406bb40 2023-10-30T14:39:29.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6adb846640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad40caa00 con 0x7f6ad406b760 2023-10-30T14:39:29.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6ad9de3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ad406c420 0x7f6ad41f9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:29.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6ad9de3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ad406c420 0x7f6ad41f9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53138/0 (socket says 172.21.15.57:53138) 2023-10-30T14:39:29.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6ad9de3640 1 -- 172.21.15.57:0/1860979357 learned_addr learned my addr 172.21.15.57:0/1860979357 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:29.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6ad95e2640 1 --2- 172.21.15.57:0/1860979357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ad406b760 0x7f6ad4146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:29.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6ad8de1640 1 --2- 172.21.15.57:0/1860979357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ad406bb40 0x7f6ad41f9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:29.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6ad9de3640 1 -- 172.21.15.57:0/1860979357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ad406b760 msgr2=0x7f6ad4146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:29.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6ad9de3640 1 --2- 172.21.15.57:0/1860979357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ad406b760 0x7f6ad4146040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:29.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6ad9de3640 1 -- 172.21.15.57:0/1860979357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ad406bb40 msgr2=0x7f6ad41f9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:29.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6ad9de3640 1 --2- 172.21.15.57:0/1860979357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ad406bb40 0x7f6ad41f9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:29.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6ad9de3640 1 -- 172.21.15.57:0/1860979357 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac4009420 con 0x7f6ad406c420 2023-10-30T14:39:29.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.112+0000 7f6ad8de1640 1 --2- 172.21.15.57:0/1860979357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ad406bb40 0x7f6ad41f9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:39:29.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.116+0000 7f6ad9de3640 1 --2- 172.21.15.57:0/1860979357 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ad406c420 0x7f6ad41f9da0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f6acc00c750 tx=0x7f6acc00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:29.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.116+0000 7f6ac9ffb640 1 -- 172.21.15.57:0/1860979357 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6acc00ce60 con 0x7f6ad406c420 2023-10-30T14:39:29.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.116+0000 7f6ac9ffb640 1 -- 172.21.15.57:0/1860979357 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6acc00a640 con 0x7f6ad406c420 2023-10-30T14:39:29.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.116+0000 7f6ac9ffb640 1 -- 172.21.15.57:0/1860979357 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6acc00a9b0 con 0x7f6ad406c420 2023-10-30T14:39:29.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.116+0000 7f6adb846640 1 -- 172.21.15.57:0/1860979357 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad414b0a0 con 0x7f6ad406c420 2023-10-30T14:39:29.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.116+0000 7f6adb846640 1 -- 172.21.15.57:0/1860979357 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad415d400 con 0x7f6ad406c420 2023-10-30T14:39:29.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.116+0000 7f6adb846640 1 -- 172.21.15.57:0/1860979357 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad41f9580 con 0x7f6ad406c420 2023-10-30T14:39:29.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.116+0000 7f6ac9ffb640 1 -- 172.21.15.57:0/1860979357 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6acc00ab10 con 0x7f6ad406c420 2023-10-30T14:39:29.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.116+0000 7f6ac9ffb640 1 --2- 172.21.15.57:0/1860979357 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ab8066520 0x7f6ab80689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:29.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.116+0000 7f6ac9ffb640 1 -- 172.21.15.57:0/1860979357 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7f6acc083ac0 con 0x7f6ad406c420 2023-10-30T14:39:29.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.116+0000 7f6ad95e2640 1 --2- 172.21.15.57:0/1860979357 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ab8066520 0x7f6ab80689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:29.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.120+0000 7f6ad95e2640 1 --2- 172.21.15.57:0/1860979357 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ab8066520 0x7f6ab80689e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0009b60 tx=0x7f6ad0009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:29.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.120+0000 7f6ac9ffb640 1 -- 172.21.15.57:0/1860979357 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6ad41f9580 con 0x7f6ad406c420 2023-10-30T14:39:29.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.316+0000 7f6adb846640 1 -- 172.21.15.57:0/1860979357 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ad41f9580 con 0x7f6ad406c420 2023-10-30T14:39:29.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.316+0000 7f6ac9ffb640 1 -- 172.21.15.57:0/1860979357 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+6537 (secure 0 0 0) 0x7f6ad41f9580 con 0x7f6ad406c420 2023-10-30T14:39:29.320 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:39:29.320 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7056,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:58.376369+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:39:29.321 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 135 2023-10-30T14:39:29.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.320+0000 7f6adb846640 1 -- 172.21.15.57:0/1860979357 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ab8066520 msgr2=0x7f6ab80689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:29.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.320+0000 7f6adb846640 1 --2- 172.21.15.57:0/1860979357 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ab8066520 0x7f6ab80689e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0009b60 tx=0x7f6ad0009190 comp rx=0 tx=0).stop 2023-10-30T14:39:29.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.320+0000 7f6adb846640 1 -- 172.21.15.57:0/1860979357 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ad406c420 msgr2=0x7f6ad41f9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:29.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.320+0000 7f6adb846640 1 --2- 172.21.15.57:0/1860979357 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ad406c420 0x7f6ad41f9da0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f6acc00c750 tx=0x7f6acc00cc10 comp rx=0 tx=0).stop 2023-10-30T14:39:29.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.320+0000 7f6adb846640 1 -- 172.21.15.57:0/1860979357 shutdown_connections 2023-10-30T14:39:29.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.320+0000 7f6adb846640 1 --2- 172.21.15.57:0/1860979357 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6ab8066520 0x7f6ab80689e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:29.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.320+0000 7f6adb846640 1 --2- 172.21.15.57:0/1860979357 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ad406c420 0x7f6ad41f9da0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:29.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.320+0000 7f6adb846640 1 --2- 172.21.15.57:0/1860979357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ad406bb40 0x7f6ad41f9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:29.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.320+0000 7f6adb846640 1 --2- 172.21.15.57:0/1860979357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ad406b760 0x7f6ad4146040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:29.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.320+0000 7f6adb846640 1 -- 172.21.15.57:0/1860979357 >> 172.21.15.57:0/1860979357 conn(0x7f6ad40cac70 msgr2=0x7f6ad4152b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:29.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.320+0000 7f6adb846640 1 -- 172.21.15.57:0/1860979357 shutdown_connections 2023-10-30T14:39:29.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:29.320+0000 7f6adb846640 1 -- 172.21.15.57:0/1860979357 wait complete. 2023-10-30T14:39:29.341 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:39:31.341 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:39:31.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.492+0000 7fc8f806b640 1 Processor -- start 2023-10-30T14:39:31.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.492+0000 7fc8f806b640 1 -- start start 2023-10-30T14:39:31.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.496+0000 7fc8f806b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8f0150830 0x7fc8f006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:31.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.496+0000 7fc8f806b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f00c2310 con 0x7fc8f00cb7d0 2023-10-30T14:39:31.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.496+0000 7fc8f806b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f0156710 con 0x7fc8f0070520 2023-10-30T14:39:31.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.496+0000 7fc8f806b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f00c4360 con 0x7fc8f0150830 2023-10-30T14:39:31.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.496+0000 7fc8f5606640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc8f00cb7d0 0x7fc8f00c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:38838/0 (socket says 172.21.15.57:38838) 2023-10-30T14:39:31.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.496+0000 7fc8f5606640 1 -- 172.21.15.57:0/563654267 learned_addr learned my addr 172.21.15.57:0/563654267 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:39:31.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.496+0000 7fc8f5e07640 1 --2- 172.21.15.57:0/563654267 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8f0150830 0x7fc8f006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:31.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.496+0000 7fc8f4e05640 1 -- 172.21.15.57:0/563654267 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282377130 0 0) 0x7fc8f00c2310 con 0x7fc8f00cb7d0 2023-10-30T14:39:31.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.496+0000 7fc8f4e05640 1 -- 172.21.15.57:0/563654267 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8dc003530 con 0x7fc8f00cb7d0 2023-10-30T14:39:31.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.496+0000 7fc8f4e05640 1 -- 172.21.15.57:0/563654267 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220797202 0 0) 0x7fc8f0156710 con 0x7fc8f0070520 2023-10-30T14:39:31.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.496+0000 7fc8f4e05640 1 -- 172.21.15.57:0/563654267 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f00c2310 con 0x7fc8f0070520 2023-10-30T14:39:31.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.496+0000 7fc8f4e05640 1 -- 172.21.15.57:0/563654267 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 200774345 0 0) 0x7fc8dc003530 con 0x7fc8f00cb7d0 2023-10-30T14:39:31.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.496+0000 7fc8f5e07640 1 -- 172.21.15.57:0/563654267 >> v1:172.21.15.88:6789/0 conn(0x7fc8f0070520 legacy=0x7fc8f0160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:31.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.496+0000 7fc8f5e07640 1 -- 172.21.15.57:0/563654267 >> v1:172.21.15.57:6789/0 conn(0x7fc8f00cb7d0 legacy=0x7fc8f00c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:31.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.496+0000 7fc8f5e07640 1 -- 172.21.15.57:0/563654267 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f01569b0 con 0x7fc8f0150830 2023-10-30T14:39:31.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.500+0000 7fc8f5e07640 1 --2- 172.21.15.57:0/563654267 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8f0150830 0x7fc8f006b500 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec0097f0 tx=0x7fc8ec01ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8bf88e76a6ed41d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:31.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.500+0000 7fc8f4e05640 1 -- 172.21.15.57:0/563654267 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0001680 con 0x7fc8f0150830 2023-10-30T14:39:31.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.500+0000 7fc8f4e05640 1 -- 172.21.15.57:0/563654267 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ec0a8650 con 0x7fc8f0150830 2023-10-30T14:39:31.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.500+0000 7fc8f4e05640 1 -- 172.21.15.57:0/563654267 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec0a89c0 con 0x7fc8f0150830 2023-10-30T14:39:31.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.500+0000 7fc8f806b640 1 -- 172.21.15.57:0/563654267 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8f0150830 msgr2=0x7fc8f006b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:31.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.500+0000 7fc8f806b640 1 --2- 172.21.15.57:0/563654267 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8f0150830 0x7fc8f006b500 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec0097f0 tx=0x7fc8ec01ebd0 comp rx=0 tx=0).stop 2023-10-30T14:39:31.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.500+0000 7fc8f806b640 1 -- 172.21.15.57:0/563654267 shutdown_connections 2023-10-30T14:39:31.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.500+0000 7fc8f806b640 1 --2- 172.21.15.57:0/563654267 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8f0150830 0x7fc8f006b500 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:31.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.500+0000 7fc8f806b640 1 -- 172.21.15.57:0/563654267 >> 172.21.15.57:0/563654267 conn(0x7fc8f00cac70 msgr2=0x7fc8f00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:31.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.500+0000 7fc8f806b640 1 -- 172.21.15.57:0/563654267 shutdown_connections 2023-10-30T14:39:31.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.500+0000 7fc8f806b640 1 -- 172.21.15.57:0/563654267 wait complete. 2023-10-30T14:39:31.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f806b640 1 Processor -- start 2023-10-30T14:39:31.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f806b640 1 -- start start 2023-10-30T14:39:31.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f806b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8f0070520 0x7fc8f01eb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:31.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f806b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8f00cb7d0 0x7fc8f0157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:31.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f806b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8f0150830 0x7fc8f01ee150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:31.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f806b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f01585c0 con 0x7fc8f0150830 2023-10-30T14:39:31.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f806b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f01eabe0 con 0x7fc8f00cb7d0 2023-10-30T14:39:31.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f806b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f014c030 con 0x7fc8f0070520 2023-10-30T14:39:31.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f6608640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8f0150830 0x7fc8f01ee150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:31.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f6608640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8f0150830 0x7fc8f01ee150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53154/0 (socket says 172.21.15.57:53154) 2023-10-30T14:39:31.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f6608640 1 -- 172.21.15.57:0/1624194116 learned_addr learned my addr 172.21.15.57:0/1624194116 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:31.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f5606640 1 --2- 172.21.15.57:0/1624194116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8f00cb7d0 0x7fc8f0157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:31.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f5e07640 1 --2- 172.21.15.57:0/1624194116 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8f0070520 0x7fc8f01eb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:31.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f6608640 1 -- 172.21.15.57:0/1624194116 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8f0070520 msgr2=0x7fc8f01eb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:31.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f6608640 1 --2- 172.21.15.57:0/1624194116 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8f0070520 0x7fc8f01eb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:31.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f6608640 1 -- 172.21.15.57:0/1624194116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8f00cb7d0 msgr2=0x7fc8f0157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:31.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f6608640 1 --2- 172.21.15.57:0/1624194116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8f00cb7d0 0x7fc8f0157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:31.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f6608640 1 -- 172.21.15.57:0/1624194116 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec009420 con 0x7fc8f0150830 2023-10-30T14:39:31.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f6608640 1 --2- 172.21.15.57:0/1624194116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8f0150830 0x7fc8f01ee150 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fc8e800c7c0 tx=0x7fc8e800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:31.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8e67fc640 1 -- 172.21.15.57:0/1624194116 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e8003b80 con 0x7fc8f0150830 2023-10-30T14:39:31.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8e67fc640 1 -- 172.21.15.57:0/1624194116 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e8003ce0 con 0x7fc8f0150830 2023-10-30T14:39:31.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f806b640 1 -- 172.21.15.57:0/1624194116 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f01eadf0 con 0x7fc8f0150830 2023-10-30T14:39:31.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8e67fc640 1 -- 172.21.15.57:0/1624194116 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e800aa50 con 0x7fc8f0150830 2023-10-30T14:39:31.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f806b640 1 -- 172.21.15.57:0/1624194116 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f014bd50 con 0x7fc8f0150830 2023-10-30T14:39:31.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.504+0000 7fc8f806b640 1 -- 172.21.15.57:0/1624194116 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f004e170 con 0x7fc8f0150830 2023-10-30T14:39:31.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.508+0000 7fc8e67fc640 1 -- 172.21.15.57:0/1624194116 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc8e800abb0 con 0x7fc8f0150830 2023-10-30T14:39:31.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.508+0000 7fc8e67fc640 1 --2- 172.21.15.57:0/1624194116 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8d40664d0 0x7fc8d4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:31.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.508+0000 7fc8e67fc640 1 -- 172.21.15.57:0/1624194116 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7fc8e80841c0 con 0x7fc8f0150830 2023-10-30T14:39:31.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.508+0000 7fc8f5e07640 1 --2- 172.21.15.57:0/1624194116 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8d40664d0 0x7fc8d4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:31.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.508+0000 7fc8f5e07640 1 --2- 172.21.15.57:0/1624194116 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8d40664d0 0x7fc8d4068990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec0b2000 tx=0x7fc8ec0aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:31.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.512+0000 7fc8e67fc640 1 -- 172.21.15.57:0/1624194116 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc8e8051100 con 0x7fc8f0150830 2023-10-30T14:39:31.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.720+0000 7fc8f806b640 1 -- 172.21.15.57:0/1624194116 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8f0072a10 con 0x7fc8f0150830 2023-10-30T14:39:31.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.720+0000 7fc8e67fc640 1 -- 172.21.15.57:0/1624194116 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+6537 (secure 0 0 0) 0x7fc8e800ae60 con 0x7fc8f0150830 2023-10-30T14:39:31.724 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:39:31.724 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7056,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:58.376369+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:39:31.724 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 135 2023-10-30T14:39:31.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.724+0000 7fc8f806b640 1 -- 172.21.15.57:0/1624194116 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8d40664d0 msgr2=0x7fc8d4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:31.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.724+0000 7fc8f806b640 1 --2- 172.21.15.57:0/1624194116 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8d40664d0 0x7fc8d4068990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec0b2000 tx=0x7fc8ec0aa070 comp rx=0 tx=0).stop 2023-10-30T14:39:31.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.724+0000 7fc8f806b640 1 -- 172.21.15.57:0/1624194116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8f0150830 msgr2=0x7fc8f01ee150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:31.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.724+0000 7fc8f806b640 1 --2- 172.21.15.57:0/1624194116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8f0150830 0x7fc8f01ee150 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fc8e800c7c0 tx=0x7fc8e800cc80 comp rx=0 tx=0).stop 2023-10-30T14:39:31.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.724+0000 7fc8f806b640 1 -- 172.21.15.57:0/1624194116 shutdown_connections 2023-10-30T14:39:31.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.724+0000 7fc8f806b640 1 --2- 172.21.15.57:0/1624194116 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc8d40664d0 0x7fc8d4068990 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:31.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.724+0000 7fc8f806b640 1 --2- 172.21.15.57:0/1624194116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8f0150830 0x7fc8f01ee150 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:31.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.724+0000 7fc8f806b640 1 --2- 172.21.15.57:0/1624194116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8f00cb7d0 0x7fc8f0157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:31.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.724+0000 7fc8f806b640 1 --2- 172.21.15.57:0/1624194116 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8f0070520 0x7fc8f01eb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:31.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.724+0000 7fc8f806b640 1 -- 172.21.15.57:0/1624194116 >> 172.21.15.57:0/1624194116 conn(0x7fc8f00cac70 msgr2=0x7fc8f00c3820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:31.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.724+0000 7fc8f806b640 1 -- 172.21.15.57:0/1624194116 shutdown_connections 2023-10-30T14:39:31.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:31.724+0000 7fc8f806b640 1 -- 172.21.15.57:0/1624194116 wait complete. 2023-10-30T14:39:31.745 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:39:33.746 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:39:33.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.896+0000 7f8fbced6640 1 Processor -- start 2023-10-30T14:39:33.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.900+0000 7f8fbced6640 1 -- start start 2023-10-30T14:39:33.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.900+0000 7f8fbced6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8fb8151840 0x7f8fb80c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:33.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.900+0000 7f8fbced6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8fb8151c20 0x7f8fb8072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:33.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.900+0000 7f8fbced6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fb80cb8b0 0x7f8fb8072540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:33.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.900+0000 7f8fbced6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fb80c5030 con 0x7f8fb8151840 2023-10-30T14:39:33.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.900+0000 7f8fbced6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fb8157830 con 0x7f8fb8151c20 2023-10-30T14:39:33.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.900+0000 7f8fbced6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb8157b10 con 0x7f8fb80cb8b0 2023-10-30T14:39:33.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.900+0000 7f8fb659c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8fb8151840 0x7f8fb80c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:33.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.900+0000 7f8fb659c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8fb8151840 0x7f8fb80c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46848/0 (socket says 172.21.15.57:46848) 2023-10-30T14:39:33.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.900+0000 7f8fb659c640 1 -- 172.21.15.57:0/4255371322 learned_addr learned my addr 172.21.15.57:0/4255371322 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:33.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.900+0000 7f8faffff640 1 --2- 172.21.15.57:0/4255371322 >> v2:172.21.15.88:3300/0 conn(0x7f8fb8151c20 0x7f8fb8072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:33.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.900+0000 7f8fb6d9d640 1 --2- 172.21.15.57:0/4255371322 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fb80cb8b0 0x7f8fb8072540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:33.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.900+0000 7f8fb659c640 1 -- 172.21.15.57:0/4255371322 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fb80cb8b0 msgr2=0x7f8fb8072540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:33.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.900+0000 7f8fb659c640 1 --2- 172.21.15.57:0/4255371322 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fb80cb8b0 0x7f8fb8072540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:33.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.900+0000 7f8fb659c640 1 -- 172.21.15.57:0/4255371322 >> v2:172.21.15.88:3300/0 conn(0x7f8fb8151c20 msgr2=0x7f8fb8072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:33.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.900+0000 7f8fb659c640 1 --2- 172.21.15.57:0/4255371322 >> v2:172.21.15.88:3300/0 conn(0x7f8fb8151c20 0x7f8fb8072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:33.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.900+0000 7f8fb659c640 1 -- 172.21.15.57:0/4255371322 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb8157df0 con 0x7f8fb8151840 2023-10-30T14:39:33.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.904+0000 7f8fb659c640 1 --2- 172.21.15.57:0/4255371322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fb8151840 0x7f8fb80c49a0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f8fb806d4d0 tx=0x7f8fa801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38ebab458cc4d046 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:33.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.904+0000 7f8fb5d9b640 1 -- 172.21.15.57:0/4255371322 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa80a89b0 con 0x7f8fb8151840 2023-10-30T14:39:33.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.904+0000 7f8fb5d9b640 1 -- 172.21.15.57:0/4255371322 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa80a8b10 con 0x7f8fb8151840 2023-10-30T14:39:33.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.904+0000 7f8fb5d9b640 1 -- 172.21.15.57:0/4255371322 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa80a8ea0 con 0x7f8fb8151840 2023-10-30T14:39:33.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.904+0000 7f8fbced6640 1 -- 172.21.15.57:0/4255371322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fb8151840 msgr2=0x7f8fb80c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:33.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.904+0000 7f8fbced6640 1 --2- 172.21.15.57:0/4255371322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fb8151840 0x7f8fb80c49a0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f8fb806d4d0 tx=0x7f8fa801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:39:33.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.904+0000 7f8fbced6640 1 -- 172.21.15.57:0/4255371322 shutdown_connections 2023-10-30T14:39:33.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.904+0000 7f8fbced6640 1 --2- 172.21.15.57:0/4255371322 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fb80cb8b0 0x7f8fb8072540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:33.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.904+0000 7f8fbced6640 1 --2- 172.21.15.57:0/4255371322 >> v2:172.21.15.88:3300/0 conn(0x7f8fb8151c20 0x7f8fb8072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:33.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.904+0000 7f8fbced6640 1 --2- 172.21.15.57:0/4255371322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fb8151840 0x7f8fb80c49a0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:33.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.904+0000 7f8fbced6640 1 -- 172.21.15.57:0/4255371322 >> 172.21.15.57:0/4255371322 conn(0x7f8fb80cac70 msgr2=0x7f8fb80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:33.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.904+0000 7f8fbced6640 1 -- 172.21.15.57:0/4255371322 shutdown_connections 2023-10-30T14:39:33.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.904+0000 7f8fbced6640 1 -- 172.21.15.57:0/4255371322 wait complete. 2023-10-30T14:39:33.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fbced6640 1 Processor -- start 2023-10-30T14:39:33.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fbced6640 1 -- start start 2023-10-30T14:39:33.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fbced6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fb80cb8b0 0x7f8fb8146a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:33.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fbced6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fb8151840 0x7f8fb80c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:33.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fbced6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fb8151c20 0x7f8fb80ca240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:33.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fbced6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb81548c0 con 0x7f8fb80cb8b0 2023-10-30T14:39:33.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fbced6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb814b0e0 con 0x7f8fb8151840 2023-10-30T14:39:33.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fbced6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb8149590 con 0x7f8fb8151c20 2023-10-30T14:39:33.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fb659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fb80cb8b0 0x7f8fb8146a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:33.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fb659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fb80cb8b0 0x7f8fb8146a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46860/0 (socket says 172.21.15.57:46860) 2023-10-30T14:39:33.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fb659c640 1 -- 172.21.15.57:0/3237666487 learned_addr learned my addr 172.21.15.57:0/3237666487 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:33.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8faffff640 1 --2- 172.21.15.57:0/3237666487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fb8151840 0x7f8fb80c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:33.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fb6d9d640 1 --2- 172.21.15.57:0/3237666487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fb8151c20 0x7f8fb80ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:33.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fb659c640 1 -- 172.21.15.57:0/3237666487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fb8151c20 msgr2=0x7f8fb80ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:33.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fb659c640 1 --2- 172.21.15.57:0/3237666487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fb8151c20 0x7f8fb80ca240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:33.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fb659c640 1 -- 172.21.15.57:0/3237666487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fb8151840 msgr2=0x7f8fb80c9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:33.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fb659c640 1 --2- 172.21.15.57:0/3237666487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fb8151840 0x7f8fb80c9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:33.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fb659c640 1 -- 172.21.15.57:0/3237666487 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa80094d0 con 0x7f8fb80cb8b0 2023-10-30T14:39:33.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fb659c640 1 --2- 172.21.15.57:0/3237666487 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fb80cb8b0 0x7f8fb8146a40 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f8fa801d3a0 tx=0x7f8fa801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:33.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8faeffd640 1 -- 172.21.15.57:0/3237666487 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa80ba600 con 0x7f8fb80cb8b0 2023-10-30T14:39:33.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8faeffd640 1 -- 172.21.15.57:0/3237666487 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa80ba760 con 0x7f8fb80cb8b0 2023-10-30T14:39:33.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fbced6640 1 -- 172.21.15.57:0/3237666487 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fb8148fb0 con 0x7f8fb80cb8b0 2023-10-30T14:39:33.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8faeffd640 1 -- 172.21.15.57:0/3237666487 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa80bab10 con 0x7f8fb80cb8b0 2023-10-30T14:39:33.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fbced6640 1 -- 172.21.15.57:0/3237666487 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fb814b330 con 0x7f8fb80cb8b0 2023-10-30T14:39:33.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.908+0000 7f8fbced6640 1 -- 172.21.15.57:0/3237666487 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fb804e170 con 0x7f8fb80cb8b0 2023-10-30T14:39:33.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.912+0000 7f8faeffd640 1 -- 172.21.15.57:0/3237666487 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8fa80bac70 con 0x7f8fb80cb8b0 2023-10-30T14:39:33.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.912+0000 7f8faeffd640 1 --2- 172.21.15.57:0/3237666487 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f980664a0 0x7f8f98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:33.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.912+0000 7f8faeffd640 1 -- 172.21.15.57:0/3237666487 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7f8fa8122930 con 0x7f8fb80cb8b0 2023-10-30T14:39:33.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.912+0000 7f8faffff640 1 --2- 172.21.15.57:0/3237666487 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f980664a0 0x7f8f98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:33.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.912+0000 7f8faffff640 1 --2- 172.21.15.57:0/3237666487 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f980664a0 0x7f8f98068960 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8fb814ccd0 tx=0x7f8fa0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:33.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:33.916+0000 7f8faeffd640 1 -- 172.21.15.57:0/3237666487 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8fa80ef8d0 con 0x7f8fb80cb8b0 2023-10-30T14:39:34.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:34.108+0000 7f8fbced6640 1 -- 172.21.15.57:0/3237666487 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fb81580a0 con 0x7f8fb80cb8b0 2023-10-30T14:39:34.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:34.108+0000 7f8faeffd640 1 -- 172.21.15.57:0/3237666487 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+6537 (secure 0 0 0) 0x7f8fa80f2ef0 con 0x7f8fb80cb8b0 2023-10-30T14:39:34.111 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:39:34.111 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7056,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:58.376369+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:39:34.112 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 135 2023-10-30T14:39:34.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:34.108+0000 7f8fbced6640 1 -- 172.21.15.57:0/3237666487 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f980664a0 msgr2=0x7f8f98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:34.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:34.108+0000 7f8fbced6640 1 --2- 172.21.15.57:0/3237666487 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f980664a0 0x7f8f98068960 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8fb814ccd0 tx=0x7f8fa0008040 comp rx=0 tx=0).stop 2023-10-30T14:39:34.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:34.108+0000 7f8fbced6640 1 -- 172.21.15.57:0/3237666487 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fb80cb8b0 msgr2=0x7f8fb8146a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:34.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:34.108+0000 7f8fbced6640 1 --2- 172.21.15.57:0/3237666487 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fb80cb8b0 0x7f8fb8146a40 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f8fa801d3a0 tx=0x7f8fa801ba90 comp rx=0 tx=0).stop 2023-10-30T14:39:34.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:34.108+0000 7f8fbced6640 1 -- 172.21.15.57:0/3237666487 shutdown_connections 2023-10-30T14:39:34.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:34.108+0000 7f8fbced6640 1 --2- 172.21.15.57:0/3237666487 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f980664a0 0x7f8f98068960 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:34.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:34.108+0000 7f8fbced6640 1 --2- 172.21.15.57:0/3237666487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fb8151c20 0x7f8fb80ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:34.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:34.108+0000 7f8fbced6640 1 --2- 172.21.15.57:0/3237666487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fb8151840 0x7f8fb80c9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:34.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:34.108+0000 7f8fbced6640 1 --2- 172.21.15.57:0/3237666487 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fb80cb8b0 0x7f8fb8146a40 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:34.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:34.108+0000 7f8fbced6640 1 -- 172.21.15.57:0/3237666487 >> 172.21.15.57:0/3237666487 conn(0x7f8fb80cac70 msgr2=0x7f8fb80c59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:34.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:34.108+0000 7f8fbced6640 1 -- 172.21.15.57:0/3237666487 shutdown_connections 2023-10-30T14:39:34.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:34.108+0000 7f8fbced6640 1 -- 172.21.15.57:0/3237666487 wait complete. 2023-10-30T14:39:34.132 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:39:36.133 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:39:36.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.284+0000 7fa7ad772640 1 Processor -- start 2023-10-30T14:39:36.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.284+0000 7fa7ad772640 1 -- start start 2023-10-30T14:39:36.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.288+0000 7fa7ad772640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa7a81516c0 0x7fa7a80c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:36.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.288+0000 7fa7ad772640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7a80c82d0 0x7fa7a80ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:36.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.288+0000 7fa7ad772640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa7a80cd4a0 0x7fa7a80c2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:36.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.288+0000 7fa7ad772640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7a80c74a0 con 0x7fa7a81516c0 2023-10-30T14:39:36.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.288+0000 7fa7ad772640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7a8151aa0 con 0x7fa7a80cd4a0 2023-10-30T14:39:36.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.288+0000 7fa7ad772640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a8158af0 con 0x7fa7a80c82d0 2023-10-30T14:39:36.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.288+0000 7fa7a6ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa7a81516c0 0x7fa7a80c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:36.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.288+0000 7fa7a6ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa7a81516c0 0x7fa7a80c7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46862/0 (socket says 172.21.15.57:46862) 2023-10-30T14:39:36.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.288+0000 7fa7a6ffd640 1 -- 172.21.15.57:0/1090368903 learned_addr learned my addr 172.21.15.57:0/1090368903 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:36.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.288+0000 7fa79ffff640 1 --2- 172.21.15.57:0/1090368903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7a80c82d0 0x7fa7a80ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:36.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.288+0000 7fa7a77fe640 1 --2- 172.21.15.57:0/1090368903 >> v2:172.21.15.88:3300/0 conn(0x7fa7a80cd4a0 0x7fa7a80c2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:36.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.288+0000 7fa7a6ffd640 1 -- 172.21.15.57:0/1090368903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7a80c82d0 msgr2=0x7fa7a80ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:36.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.288+0000 7fa7a6ffd640 1 --2- 172.21.15.57:0/1090368903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7a80c82d0 0x7fa7a80ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:36.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.288+0000 7fa7a6ffd640 1 -- 172.21.15.57:0/1090368903 >> v2:172.21.15.88:3300/0 conn(0x7fa7a80cd4a0 msgr2=0x7fa7a80c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:36.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.288+0000 7fa7a6ffd640 1 --2- 172.21.15.57:0/1090368903 >> v2:172.21.15.88:3300/0 conn(0x7fa7a80cd4a0 0x7fa7a80c2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:36.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.288+0000 7fa7a6ffd640 1 -- 172.21.15.57:0/1090368903 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a8158510 con 0x7fa7a81516c0 2023-10-30T14:39:36.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.292+0000 7fa7a6ffd640 1 --2- 172.21.15.57:0/1090368903 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7a81516c0 0x7fa7a80c7d90 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8071f60 tx=0x7fa79801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c1a18d8ba5da62 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:36.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.292+0000 7fa7a67fc640 1 -- 172.21.15.57:0/1090368903 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7980ab600 con 0x7fa7a81516c0 2023-10-30T14:39:36.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.292+0000 7fa7a67fc640 1 -- 172.21.15.57:0/1090368903 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7980a7050 con 0x7fa7a81516c0 2023-10-30T14:39:36.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.292+0000 7fa7a67fc640 1 -- 172.21.15.57:0/1090368903 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7980ab980 con 0x7fa7a81516c0 2023-10-30T14:39:36.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.292+0000 7fa7ad772640 1 -- 172.21.15.57:0/1090368903 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7a81516c0 msgr2=0x7fa7a80c7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:36.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.292+0000 7fa7ad772640 1 --2- 172.21.15.57:0/1090368903 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7a81516c0 0x7fa7a80c7d90 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8071f60 tx=0x7fa79801c500 comp rx=0 tx=0).stop 2023-10-30T14:39:36.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.292+0000 7fa7ad772640 1 -- 172.21.15.57:0/1090368903 shutdown_connections 2023-10-30T14:39:36.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.292+0000 7fa7ad772640 1 --2- 172.21.15.57:0/1090368903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7a80c82d0 0x7fa7a80ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:36.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.292+0000 7fa7ad772640 1 --2- 172.21.15.57:0/1090368903 >> v2:172.21.15.88:3300/0 conn(0x7fa7a80cd4a0 0x7fa7a80c2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:36.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.292+0000 7fa7ad772640 1 --2- 172.21.15.57:0/1090368903 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7a81516c0 0x7fa7a80c7d90 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:36.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.292+0000 7fa7ad772640 1 -- 172.21.15.57:0/1090368903 >> 172.21.15.57:0/1090368903 conn(0x7fa7a8071fb0 msgr2=0x7fa7a80723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:36.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.292+0000 7fa7ad772640 1 -- 172.21.15.57:0/1090368903 shutdown_connections 2023-10-30T14:39:36.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.292+0000 7fa7ad772640 1 -- 172.21.15.57:0/1090368903 wait complete. 2023-10-30T14:39:36.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.292+0000 7fa7ad772640 1 Processor -- start 2023-10-30T14:39:36.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.292+0000 7fa7ad772640 1 -- start start 2023-10-30T14:39:36.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa7ad772640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7a80c82d0 0x7fa7a81f3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:36.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa7ad772640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7a80cd4a0 0x7fa7a814eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:36.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa7ad772640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7a81516c0 0x7fa7a80cc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:36.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa7ad772640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a814e6d0 con 0x7fa7a80c82d0 2023-10-30T14:39:36.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa7ad772640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a80704d0 con 0x7fa7a81516c0 2023-10-30T14:39:36.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa7ad772640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a814ddc0 con 0x7fa7a80cd4a0 2023-10-30T14:39:36.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa7a6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7a80c82d0 0x7fa7a81f3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:36.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa7a6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7a80c82d0 0x7fa7a81f3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46870/0 (socket says 172.21.15.57:46870) 2023-10-30T14:39:36.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa7a6ffd640 1 -- 172.21.15.57:0/753783735 learned_addr learned my addr 172.21.15.57:0/753783735 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:36.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa7a77fe640 1 --2- 172.21.15.57:0/753783735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7a81516c0 0x7fa7a80cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:36.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa79ffff640 1 --2- 172.21.15.57:0/753783735 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7a80cd4a0 0x7fa7a814eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:36.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa7a6ffd640 1 -- 172.21.15.57:0/753783735 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7a80cd4a0 msgr2=0x7fa7a814eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:36.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa7a6ffd640 1 --2- 172.21.15.57:0/753783735 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7a80cd4a0 0x7fa7a814eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:36.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa7a6ffd640 1 -- 172.21.15.57:0/753783735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7a81516c0 msgr2=0x7fa7a80cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:36.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa7a6ffd640 1 --2- 172.21.15.57:0/753783735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7a81516c0 0x7fa7a80cc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:36.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa7a6ffd640 1 -- 172.21.15.57:0/753783735 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7980094d0 con 0x7fa7a80c82d0 2023-10-30T14:39:36.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa7a6ffd640 1 --2- 172.21.15.57:0/753783735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7a80c82d0 0x7fa7a81f3050 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fa7980ae040 tx=0x7fa79801ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:36.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa79f7fe640 1 -- 172.21.15.57:0/753783735 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7980b6620 con 0x7fa7a80c82d0 2023-10-30T14:39:36.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa79f7fe640 1 -- 172.21.15.57:0/753783735 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7980b67a0 con 0x7fa7a80c82d0 2023-10-30T14:39:36.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa7ad772640 1 -- 172.21.15.57:0/753783735 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a814f030 con 0x7fa7a80c82d0 2023-10-30T14:39:36.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa79f7fe640 1 -- 172.21.15.57:0/753783735 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7980b6b50 con 0x7fa7a80c82d0 2023-10-30T14:39:36.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa7ad772640 1 -- 172.21.15.57:0/753783735 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a814e010 con 0x7fa7a80c82d0 2023-10-30T14:39:36.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa7ad772640 1 -- 172.21.15.57:0/753783735 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a814e940 con 0x7fa7a80c82d0 2023-10-30T14:39:36.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.296+0000 7fa79f7fe640 1 -- 172.21.15.57:0/753783735 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa7980b2040 con 0x7fa7a80c82d0 2023-10-30T14:39:36.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.300+0000 7fa79f7fe640 1 --2- 172.21.15.57:0/753783735 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa7880664a0 0x7fa788068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:36.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.300+0000 7fa79f7fe640 1 -- 172.21.15.57:0/753783735 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7fa79811e980 con 0x7fa7a80c82d0 2023-10-30T14:39:36.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.300+0000 7fa79ffff640 1 --2- 172.21.15.57:0/753783735 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa7880664a0 0x7fa788068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:36.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.300+0000 7fa79ffff640 1 --2- 172.21.15.57:0/753783735 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa7880664a0 0x7fa788068960 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8151de0 tx=0x7fa790008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:36.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.304+0000 7fa79f7fe640 1 -- 172.21.15.57:0/753783735 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa7980eb920 con 0x7fa7a80c82d0 2023-10-30T14:39:36.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.492+0000 7fa7ad772640 1 -- 172.21.15.57:0/753783735 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7a804e170 con 0x7fa7a80c82d0 2023-10-30T14:39:36.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.492+0000 7fa79f7fe640 1 -- 172.21.15.57:0/753783735 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+6537 (secure 0 0 0) 0x7fa7980eef40 con 0x7fa7a80c82d0 2023-10-30T14:39:36.496 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:39:36.496 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7056,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:38:58.376369+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:39:36.496 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 135 2023-10-30T14:39:36.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.496+0000 7fa7ad772640 1 -- 172.21.15.57:0/753783735 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa7880664a0 msgr2=0x7fa788068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:36.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.496+0000 7fa7ad772640 1 --2- 172.21.15.57:0/753783735 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa7880664a0 0x7fa788068960 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8151de0 tx=0x7fa790008040 comp rx=0 tx=0).stop 2023-10-30T14:39:36.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.496+0000 7fa7ad772640 1 -- 172.21.15.57:0/753783735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7a80c82d0 msgr2=0x7fa7a81f3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:36.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.496+0000 7fa7ad772640 1 --2- 172.21.15.57:0/753783735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7a80c82d0 0x7fa7a81f3050 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fa7980ae040 tx=0x7fa79801ad40 comp rx=0 tx=0).stop 2023-10-30T14:39:36.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.496+0000 7fa7ad772640 1 -- 172.21.15.57:0/753783735 shutdown_connections 2023-10-30T14:39:36.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.496+0000 7fa7ad772640 1 --2- 172.21.15.57:0/753783735 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa7880664a0 0x7fa788068960 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:36.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.496+0000 7fa7ad772640 1 --2- 172.21.15.57:0/753783735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7a81516c0 0x7fa7a80cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:36.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.496+0000 7fa7ad772640 1 --2- 172.21.15.57:0/753783735 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7a80cd4a0 0x7fa7a814eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:36.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.496+0000 7fa7ad772640 1 --2- 172.21.15.57:0/753783735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7a80c82d0 0x7fa7a81f3050 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:36.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.496+0000 7fa7ad772640 1 -- 172.21.15.57:0/753783735 >> 172.21.15.57:0/753783735 conn(0x7fa7a8071fb0 msgr2=0x7fa7a8072390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:36.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.496+0000 7fa7ad772640 1 -- 172.21.15.57:0/753783735 shutdown_connections 2023-10-30T14:39:36.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:36.496+0000 7fa7ad772640 1 -- 172.21.15.57:0/753783735 wait complete. 2023-10-30T14:39:36.517 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:39:37.862 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:39:37.867 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:39:38.518 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:39:38.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.668+0000 7f36582ea640 1 Processor -- start 2023-10-30T14:39:38.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.668+0000 7f36582ea640 1 -- start start 2023-10-30T14:39:38.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f36582ea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f36501537f0 0x7f36500701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:38.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f36582ea640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36500c2aa0 con 0x7f365006b2f0 2023-10-30T14:39:38.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f36582ea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36500708e0 con 0x7f365015a2e0 2023-10-30T14:39:38.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f36582ea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36500c3ab0 con 0x7f36501537f0 2023-10-30T14:39:38.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3656887640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f365006b2f0 0x7f365006b700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:57784/0 (socket says 172.21.15.57:57784) 2023-10-30T14:39:38.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3656887640 1 -- 172.21.15.57:0/4130037076 learned_addr learned my addr 172.21.15.57:0/4130037076 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:39:38.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3656086640 1 --2- 172.21.15.57:0/4130037076 >> v2:172.21.15.88:3300/0 conn(0x7f36501537f0 0x7f36500701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:38.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3655084640 1 -- 172.21.15.57:0/4130037076 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040185431 0 0) 0x7f36500c2aa0 con 0x7f365006b2f0 2023-10-30T14:39:38.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3655084640 1 -- 172.21.15.57:0/4130037076 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f363c003530 con 0x7f365006b2f0 2023-10-30T14:39:38.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3655084640 1 -- 172.21.15.57:0/4130037076 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1784883661 0 0) 0x7f363c003530 con 0x7f365006b2f0 2023-10-30T14:39:38.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3655084640 1 -- 172.21.15.57:0/4130037076 >> v2:172.21.15.88:3300/0 conn(0x7f36501537f0 msgr2=0x7f36500701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:38.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3655084640 1 --2- 172.21.15.57:0/4130037076 >> v2:172.21.15.88:3300/0 conn(0x7f36501537f0 0x7f36500701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:38.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3655084640 1 -- 172.21.15.57:0/4130037076 >> v1:172.21.15.88:6789/0 conn(0x7f365015a2e0 legacy=0x7f36500ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:38.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3655084640 1 -- 172.21.15.57:0/4130037076 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36500c1eb0 con 0x7f365006b2f0 2023-10-30T14:39:38.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3655084640 1 -- 172.21.15.57:0/4130037076 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f3648001a00 con 0x7f365006b2f0 2023-10-30T14:39:38.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3655084640 1 -- 172.21.15.57:0/4130037076 >> v1:172.21.15.57:6789/0 conn(0x7f365006b2f0 legacy=0x7f365006b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:38.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3655084640 1 --2- 172.21.15.57:0/4130037076 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f363c003980 0x7f363c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:38.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3655084640 1 --2- 172.21.15.57:0/4130037076 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f363c0047d0 0x7f363c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:38.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3655084640 1 --2- 172.21.15.57:0/4130037076 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f363c005190 0x7f363c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:38.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3655084640 1 -- 172.21.15.57:0/4130037076 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363c00c010 con 0x7f363c005190 2023-10-30T14:39:38.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3655084640 1 -- 172.21.15.57:0/4130037076 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363c00c180 con 0x7f363c0047d0 2023-10-30T14:39:38.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3655084640 1 -- 172.21.15.57:0/4130037076 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363c00c340 con 0x7f363c003980 2023-10-30T14:39:38.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3656887640 1 --2- 172.21.15.57:0/4130037076 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f363c005190 0x7f363c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:38.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3656086640 1 --2- 172.21.15.57:0/4130037076 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f363c003980 0x7f363c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:38.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3655885640 1 --2- 172.21.15.57:0/4130037076 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f363c0047d0 0x7f363c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:38.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3656887640 1 -- 172.21.15.57:0/4130037076 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f363c003980 msgr2=0x7f363c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:38.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3656887640 1 --2- 172.21.15.57:0/4130037076 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f363c003980 0x7f363c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:38.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3656887640 1 -- 172.21.15.57:0/4130037076 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f363c0047d0 msgr2=0x7f363c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:38.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3656887640 1 --2- 172.21.15.57:0/4130037076 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f363c0047d0 0x7f363c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:38.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.672+0000 7f3656887640 1 -- 172.21.15.57:0/4130037076 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c003530 con 0x7f363c005190 2023-10-30T14:39:38.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.676+0000 7f3656887640 1 -- 172.21.15.57:0/4130037076 reap_dead start 2023-10-30T14:39:38.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.676+0000 7f3656887640 1 --2- 172.21.15.57:0/4130037076 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f363c005190 0x7f363c00b990 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f364800a820 tx=0x7f364801d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89fa09bad5449f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:38.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.676+0000 7f3655084640 1 -- 172.21.15.57:0/4130037076 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36480072d0 con 0x7f363c005190 2023-10-30T14:39:38.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.676+0000 7f3655084640 1 -- 172.21.15.57:0/4130037076 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36480a8070 con 0x7f363c005190 2023-10-30T14:39:38.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.676+0000 7f3655084640 1 -- 172.21.15.57:0/4130037076 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36480076b0 con 0x7f363c005190 2023-10-30T14:39:38.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.676+0000 7f36582ea640 1 -- 172.21.15.57:0/4130037076 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f363c005190 msgr2=0x7f363c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:38.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.676+0000 7f36582ea640 1 --2- 172.21.15.57:0/4130037076 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f363c005190 0x7f363c00b990 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f364800a820 tx=0x7f364801d320 comp rx=0 tx=0).stop 2023-10-30T14:39:38.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.676+0000 7f36582ea640 1 -- 172.21.15.57:0/4130037076 shutdown_connections 2023-10-30T14:39:38.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.676+0000 7f36582ea640 1 --2- 172.21.15.57:0/4130037076 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f363c005190 0x7f363c00b990 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:38.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.676+0000 7f36582ea640 1 -- 172.21.15.57:0/4130037076 >> 172.21.15.57:0/4130037076 conn(0x7f3650071fb0 msgr2=0x7f36500723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:38.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.676+0000 7f36582ea640 1 -- 172.21.15.57:0/4130037076 shutdown_connections 2023-10-30T14:39:38.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.676+0000 7f36582ea640 1 -- 172.21.15.57:0/4130037076 wait complete. 2023-10-30T14:39:38.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.676+0000 7f36582ea640 1 Processor -- start 2023-10-30T14:39:38.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.676+0000 7f36582ea640 1 -- start start 2023-10-30T14:39:38.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f36582ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f363c005190 0x7f36501ee830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:38.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f36582ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3650070250 0x7f36501f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:38.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f36582ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36500c6710 0x7f36501592c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:38.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f36582ea640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3650155b60 con 0x7f3650070250 2023-10-30T14:39:38.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f36582ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36501f22e0 con 0x7f36500c6710 2023-10-30T14:39:38.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f36582ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36501ef7d0 con 0x7f363c005190 2023-10-30T14:39:38.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f3655885640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3650070250 0x7f36501f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:38.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f3655885640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3650070250 0x7f36501f1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46884/0 (socket says 172.21.15.57:46884) 2023-10-30T14:39:38.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f3655885640 1 -- 172.21.15.57:0/3285464372 learned_addr learned my addr 172.21.15.57:0/3285464372 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:38.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f3656086640 1 --2- 172.21.15.57:0/3285464372 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f363c005190 0x7f36501ee830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:38.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f3656887640 1 --2- 172.21.15.57:0/3285464372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36500c6710 0x7f36501592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:38.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f3655885640 1 -- 172.21.15.57:0/3285464372 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f363c005190 msgr2=0x7f36501ee830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:38.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f3655885640 1 --2- 172.21.15.57:0/3285464372 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f363c005190 0x7f36501ee830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:38.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f3655885640 1 -- 172.21.15.57:0/3285464372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36500c6710 msgr2=0x7f36501592c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:38.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f3655885640 1 --2- 172.21.15.57:0/3285464372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36500c6710 0x7f36501592c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:38.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f3655885640 1 -- 172.21.15.57:0/3285464372 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36501f24e0 con 0x7f3650070250 2023-10-30T14:39:38.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f3656086640 1 --2- 172.21.15.57:0/3285464372 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f363c005190 0x7f36501ee830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:39:38.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f3655885640 1 --2- 172.21.15.57:0/3285464372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3650070250 0x7f36501f1da0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f364000e4b0 tx=0x7f364000e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:38.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f3646ffd640 1 -- 172.21.15.57:0/3285464372 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3640007ea0 con 0x7f3650070250 2023-10-30T14:39:38.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f3646ffd640 1 -- 172.21.15.57:0/3285464372 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36400052e0 con 0x7f3650070250 2023-10-30T14:39:38.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f3646ffd640 1 -- 172.21.15.57:0/3285464372 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3640003840 con 0x7f3650070250 2023-10-30T14:39:38.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f36582ea640 1 -- 172.21.15.57:0/3285464372 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36501f27c0 con 0x7f3650070250 2023-10-30T14:39:38.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f36582ea640 1 -- 172.21.15.57:0/3285464372 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3650155780 con 0x7f3650070250 2023-10-30T14:39:38.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f36582ea640 1 -- 172.21.15.57:0/3285464372 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36501555e0 con 0x7f3650070250 2023-10-30T14:39:38.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.680+0000 7f3646ffd640 1 -- 172.21.15.57:0/3285464372 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f36400039a0 con 0x7f3650070250 2023-10-30T14:39:38.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.684+0000 7f3646ffd640 1 --2- 172.21.15.57:0/3285464372 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f36340664a0 0x7f3634068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:38.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.684+0000 7f3646ffd640 1 -- 172.21.15.57:0/3285464372 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7f3640084120 con 0x7f3650070250 2023-10-30T14:39:38.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.684+0000 7f3656086640 1 --2- 172.21.15.57:0/3285464372 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f36340664a0 0x7f3634068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:38.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.684+0000 7f3656086640 1 --2- 172.21.15.57:0/3285464372 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f36340664a0 0x7f3634068960 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f364c009a70 tx=0x7f364c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:38.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.688+0000 7f3646ffd640 1 -- 172.21.15.57:0/3285464372 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f36501555e0 con 0x7f3650070250 2023-10-30T14:39:38.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.876+0000 7f36582ea640 1 -- 172.21.15.57:0/3285464372 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3650155ca0 con 0x7f3650070250 2023-10-30T14:39:38.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.876+0000 7f3646ffd640 1 -- 172.21.15.57:0/3285464372 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+6535 (secure 0 0 0) 0x7f3650155ca0 con 0x7f3650070250 2023-10-30T14:39:38.881 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:39:38.881 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":136,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7056,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:39:37.108413+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:39:38.882 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 136 2023-10-30T14:39:38.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.880+0000 7f36582ea640 1 -- 172.21.15.57:0/3285464372 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f36340664a0 msgr2=0x7f3634068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:38.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.880+0000 7f36582ea640 1 --2- 172.21.15.57:0/3285464372 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f36340664a0 0x7f3634068960 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f364c009a70 tx=0x7f364c008040 comp rx=0 tx=0).stop 2023-10-30T14:39:38.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.880+0000 7f36582ea640 1 -- 172.21.15.57:0/3285464372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3650070250 msgr2=0x7f36501f1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:38.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.880+0000 7f36582ea640 1 --2- 172.21.15.57:0/3285464372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3650070250 0x7f36501f1da0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f364000e4b0 tx=0x7f364000e970 comp rx=0 tx=0).stop 2023-10-30T14:39:38.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.880+0000 7f36582ea640 1 -- 172.21.15.57:0/3285464372 shutdown_connections 2023-10-30T14:39:38.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.880+0000 7f36582ea640 1 --2- 172.21.15.57:0/3285464372 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f36340664a0 0x7f3634068960 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:38.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.880+0000 7f36582ea640 1 --2- 172.21.15.57:0/3285464372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36500c6710 0x7f36501592c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:38.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.880+0000 7f36582ea640 1 --2- 172.21.15.57:0/3285464372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3650070250 0x7f36501f1da0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:38.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.880+0000 7f36582ea640 1 --2- 172.21.15.57:0/3285464372 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f363c005190 0x7f36501ee830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:38.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.880+0000 7f36582ea640 1 -- 172.21.15.57:0/3285464372 >> 172.21.15.57:0/3285464372 conn(0x7f3650071fb0 msgr2=0x7f36501520b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:38.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.880+0000 7f36582ea640 1 -- 172.21.15.57:0/3285464372 shutdown_connections 2023-10-30T14:39:38.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:38.880+0000 7f36582ea640 1 -- 172.21.15.57:0/3285464372 wait complete. 2023-10-30T14:39:38.902 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:39:40.903 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:39:41.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.056+0000 7fac40b3e640 1 Processor -- start 2023-10-30T14:39:41.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.056+0000 7fac40b3e640 1 -- start start 2023-10-30T14:39:41.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.056+0000 7fac40b3e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fac3c153c80 0x7fac3c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:41.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.056+0000 7fac40b3e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fac3c0c4a10 0x7fac3c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:41.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.056+0000 7fac40b3e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3c159f20 con 0x7fac3c15a3b0 2023-10-30T14:39:41.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.056+0000 7fac40b3e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac3c15dec0 con 0x7fac3c153c80 2023-10-30T14:39:41.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.056+0000 7fac40b3e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c1599a0 con 0x7fac3c0c4a10 2023-10-30T14:39:41.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.056+0000 7fac3a59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fac3c153c80 0x7fac3c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:41.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.056+0000 7fac3a59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fac3c153c80 0x7fac3c15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46890/0 (socket says 172.21.15.57:46890) 2023-10-30T14:39:41.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.056+0000 7fac3a59c640 1 -- 172.21.15.57:0/1663763792 learned_addr learned my addr 172.21.15.57:0/1663763792 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:41.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.056+0000 7fac3ad9d640 1 --1- 172.21.15.57:0/1663763792 >> v1:172.21.15.57:6789/0 conn(0x7fac3c15a3b0 0x7fac3c0ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:57788/0 (socket says 172.21.15.57:57788) 2023-10-30T14:39:41.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.056+0000 7fac33fff640 1 --2- 172.21.15.57:0/1663763792 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fac3c0c4a10 0x7fac3c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:41.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.056+0000 7fac3a59c640 1 -- 172.21.15.57:0/1663763792 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fac3c0c4a10 msgr2=0x7fac3c0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:41.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.056+0000 7fac3a59c640 1 --2- 172.21.15.57:0/1663763792 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fac3c0c4a10 0x7fac3c0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:41.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.056+0000 7fac3a59c640 1 -- 172.21.15.57:0/1663763792 >> v1:172.21.15.57:6789/0 conn(0x7fac3c15a3b0 legacy=0x7fac3c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:41.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.056+0000 7fac3a59c640 1 -- 172.21.15.57:0/1663763792 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c0c47e0 con 0x7fac3c153c80 2023-10-30T14:39:41.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.056+0000 7fac39d9b640 1 -- 172.21.15.57:0/1663763792 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314730571 0 0) 0x7fac3c159f20 con 0x7fac3c15a3b0 2023-10-30T14:39:41.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.060+0000 7fac3a59c640 1 --2- 172.21.15.57:0/1663763792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac3c153c80 0x7fac3c15cf90 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fac2c009870 tx=0x7fac2c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf426a2c45954ea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:41.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.060+0000 7fac39d9b640 1 -- 172.21.15.57:0/1663763792 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0a8a30 con 0x7fac3c153c80 2023-10-30T14:39:41.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.060+0000 7fac39d9b640 1 -- 172.21.15.57:0/1663763792 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2c0a8b90 con 0x7fac3c153c80 2023-10-30T14:39:41.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.060+0000 7fac39d9b640 1 -- 172.21.15.57:0/1663763792 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0b0440 con 0x7fac3c153c80 2023-10-30T14:39:41.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.060+0000 7fac40b3e640 1 -- 172.21.15.57:0/1663763792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac3c153c80 msgr2=0x7fac3c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:41.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.060+0000 7fac40b3e640 1 --2- 172.21.15.57:0/1663763792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac3c153c80 0x7fac3c15cf90 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fac2c009870 tx=0x7fac2c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:39:41.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.060+0000 7fac40b3e640 1 -- 172.21.15.57:0/1663763792 shutdown_connections 2023-10-30T14:39:41.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.060+0000 7fac40b3e640 1 --2- 172.21.15.57:0/1663763792 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fac3c0c4a10 0x7fac3c0c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:41.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.060+0000 7fac40b3e640 1 --2- 172.21.15.57:0/1663763792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac3c153c80 0x7fac3c15cf90 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:41.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.060+0000 7fac40b3e640 1 -- 172.21.15.57:0/1663763792 >> 172.21.15.57:0/1663763792 conn(0x7fac3c0715c0 msgr2=0x7fac3c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:41.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.060+0000 7fac40b3e640 1 -- 172.21.15.57:0/1663763792 shutdown_connections 2023-10-30T14:39:41.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.060+0000 7fac40b3e640 1 -- 172.21.15.57:0/1663763792 wait complete. 2023-10-30T14:39:41.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac40b3e640 1 Processor -- start 2023-10-30T14:39:41.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac40b3e640 1 -- start start 2023-10-30T14:39:41.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac40b3e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fac3c0c4a10 0x7fac3c14efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:41.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac40b3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac3c153c80 0x7fac3c0c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:41.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac40b3e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac3c15a3b0 0x7fac3c149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:41.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac40b3e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c1ef0e0 con 0x7fac3c15a3b0 2023-10-30T14:39:41.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac40b3e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c1587c0 con 0x7fac3c153c80 2023-10-30T14:39:41.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac40b3e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c14eb90 con 0x7fac3c0c4a10 2023-10-30T14:39:41.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac3ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac3c15a3b0 0x7fac3c149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:41.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac3ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac3c15a3b0 0x7fac3c149b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46906/0 (socket says 172.21.15.57:46906) 2023-10-30T14:39:41.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac3ad9d640 1 -- 172.21.15.57:0/1715692775 learned_addr learned my addr 172.21.15.57:0/1715692775 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:41.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac33fff640 1 --2- 172.21.15.57:0/1715692775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac3c153c80 0x7fac3c0c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:41.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac3a59c640 1 --2- 172.21.15.57:0/1715692775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fac3c0c4a10 0x7fac3c14efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:41.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac3ad9d640 1 -- 172.21.15.57:0/1715692775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fac3c0c4a10 msgr2=0x7fac3c14efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:41.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac3ad9d640 1 --2- 172.21.15.57:0/1715692775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fac3c0c4a10 0x7fac3c14efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:41.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac3ad9d640 1 -- 172.21.15.57:0/1715692775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac3c153c80 msgr2=0x7fac3c0c8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:41.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac3ad9d640 1 --2- 172.21.15.57:0/1715692775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac3c153c80 0x7fac3c0c8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:41.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac3ad9d640 1 -- 172.21.15.57:0/1715692775 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2c0094d0 con 0x7fac3c15a3b0 2023-10-30T14:39:41.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac3ad9d640 1 --2- 172.21.15.57:0/1715692775 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac3c15a3b0 0x7fac3c149b30 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fac3400c780 tx=0x7fac3400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:41.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac32ffd640 1 -- 172.21.15.57:0/1715692775 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3400ce90 con 0x7fac3c15a3b0 2023-10-30T14:39:41.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac32ffd640 1 -- 172.21.15.57:0/1715692775 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3400a640 con 0x7fac3c15a3b0 2023-10-30T14:39:41.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac32ffd640 1 -- 172.21.15.57:0/1715692775 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3400a9d0 con 0x7fac3c15a3b0 2023-10-30T14:39:41.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac40b3e640 1 -- 172.21.15.57:0/1715692775 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac3c1ee7d0 con 0x7fac3c15a3b0 2023-10-30T14:39:41.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.064+0000 7fac40b3e640 1 -- 172.21.15.57:0/1715692775 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac3c1f2ad0 con 0x7fac3c15a3b0 2023-10-30T14:39:41.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.068+0000 7fac40b3e640 1 -- 172.21.15.57:0/1715692775 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac3c073c30 con 0x7fac3c15a3b0 2023-10-30T14:39:41.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.068+0000 7fac32ffd640 1 -- 172.21.15.57:0/1715692775 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fac3400ab30 con 0x7fac3c15a3b0 2023-10-30T14:39:41.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.068+0000 7fac32ffd640 1 --2- 172.21.15.57:0/1715692775 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fac1c0664a0 0x7fac1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:41.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.068+0000 7fac32ffd640 1 -- 172.21.15.57:0/1715692775 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7fac34083870 con 0x7fac3c15a3b0 2023-10-30T14:39:41.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.068+0000 7fac3a59c640 1 --2- 172.21.15.57:0/1715692775 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fac1c0664a0 0x7fac1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:41.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.068+0000 7fac3a59c640 1 --2- 172.21.15.57:0/1715692775 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fac1c0664a0 0x7fac1c068960 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fac2c0b2000 tx=0x7fac2c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:41.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.072+0000 7fac32ffd640 1 -- 172.21.15.57:0/1715692775 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fac34050830 con 0x7fac3c15a3b0 2023-10-30T14:39:41.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.264+0000 7fac40b3e640 1 -- 172.21.15.57:0/1715692775 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac3c04e170 con 0x7fac3c15a3b0 2023-10-30T14:39:41.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.264+0000 7fac32ffd640 1 -- 172.21.15.57:0/1715692775 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+6535 (secure 0 0 0) 0x7fac34018090 con 0x7fac3c15a3b0 2023-10-30T14:39:41.267 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:39:41.267 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":136,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7056,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:39:37.108413+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:39:41.268 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 136 2023-10-30T14:39:41.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.268+0000 7fac40b3e640 1 -- 172.21.15.57:0/1715692775 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fac1c0664a0 msgr2=0x7fac1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:41.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.268+0000 7fac40b3e640 1 --2- 172.21.15.57:0/1715692775 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fac1c0664a0 0x7fac1c068960 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fac2c0b2000 tx=0x7fac2c009040 comp rx=0 tx=0).stop 2023-10-30T14:39:41.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.268+0000 7fac40b3e640 1 -- 172.21.15.57:0/1715692775 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac3c15a3b0 msgr2=0x7fac3c149b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:41.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.268+0000 7fac40b3e640 1 --2- 172.21.15.57:0/1715692775 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac3c15a3b0 0x7fac3c149b30 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fac3400c780 tx=0x7fac3400cc40 comp rx=0 tx=0).stop 2023-10-30T14:39:41.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.268+0000 7fac40b3e640 1 -- 172.21.15.57:0/1715692775 shutdown_connections 2023-10-30T14:39:41.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.268+0000 7fac40b3e640 1 --2- 172.21.15.57:0/1715692775 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fac1c0664a0 0x7fac1c068960 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:41.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.268+0000 7fac40b3e640 1 --2- 172.21.15.57:0/1715692775 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac3c15a3b0 0x7fac3c149b30 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:41.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.268+0000 7fac40b3e640 1 --2- 172.21.15.57:0/1715692775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac3c153c80 0x7fac3c0c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:41.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.268+0000 7fac40b3e640 1 --2- 172.21.15.57:0/1715692775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fac3c0c4a10 0x7fac3c14efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:41.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.268+0000 7fac40b3e640 1 -- 172.21.15.57:0/1715692775 >> 172.21.15.57:0/1715692775 conn(0x7fac3c0715c0 msgr2=0x7fac3c152e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:41.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.268+0000 7fac40b3e640 1 -- 172.21.15.57:0/1715692775 shutdown_connections 2023-10-30T14:39:41.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:41.268+0000 7fac40b3e640 1 -- 172.21.15.57:0/1715692775 wait complete. 2023-10-30T14:39:41.288 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:39:43.289 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:39:43.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.440+0000 7f5ce3994640 1 Processor -- start 2023-10-30T14:39:43.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.440+0000 7f5ce3994640 1 -- start start 2023-10-30T14:39:43.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.440+0000 7f5ce3994640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5cdc153aa0 0x7f5cdc15d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:43.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.440+0000 7f5ce3994640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cdc0c7b00 con 0x7f5cdc06b3b0 2023-10-30T14:39:43.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.440+0000 7f5ce3994640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cdc0c2050 con 0x7f5cdc0701e0 2023-10-30T14:39:43.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.440+0000 7f5ce3994640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cdc070a50 con 0x7f5cdc153aa0 2023-10-30T14:39:43.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.440+0000 7f5ce1730640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5cdc153aa0 0x7f5cdc15d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:43.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.440+0000 7f5ce1730640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5cdc153aa0 0x7f5cdc15d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46916/0 (socket says 172.21.15.57:46916) 2023-10-30T14:39:43.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.440+0000 7f5ce1730640 1 -- 172.21.15.57:0/1437244678 learned_addr learned my addr 172.21.15.57:0/1437244678 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:43.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.440+0000 7f5ce1f31640 1 --1- 172.21.15.57:0/1437244678 >> v1:172.21.15.57:6789/0 conn(0x7f5cdc06b3b0 0x7f5cdc06b7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:57802/0 (socket says 172.21.15.57:57802) 2023-10-30T14:39:43.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.440+0000 7f5ce1730640 1 -- 172.21.15.57:0/1437244678 >> v1:172.21.15.88:6789/0 conn(0x7f5cdc0701e0 legacy=0x7f5cdc0ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:43.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.440+0000 7f5ce1730640 1 -- 172.21.15.57:0/1437244678 >> v1:172.21.15.57:6789/0 conn(0x7f5cdc06b3b0 legacy=0x7f5cdc06b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:43.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.440+0000 7f5ce1730640 1 -- 172.21.15.57:0/1437244678 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cdc0c2330 con 0x7f5cdc153aa0 2023-10-30T14:39:43.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.444+0000 7f5ce1730640 1 --2- 172.21.15.57:0/1437244678 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5cdc153aa0 0x7f5cdc15d840 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8009870 tx=0x7f5cd801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c963f6c45e2f415e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:43.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.444+0000 7f5cd3fff640 1 -- 172.21.15.57:0/1437244678 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd80a8a30 con 0x7f5cdc153aa0 2023-10-30T14:39:43.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.444+0000 7f5cd3fff640 1 -- 172.21.15.57:0/1437244678 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd80a8b90 con 0x7f5cdc153aa0 2023-10-30T14:39:43.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.444+0000 7f5cd3fff640 1 -- 172.21.15.57:0/1437244678 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd80b0440 con 0x7f5cdc153aa0 2023-10-30T14:39:43.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.444+0000 7f5ce3994640 1 -- 172.21.15.57:0/1437244678 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5cdc153aa0 msgr2=0x7f5cdc15d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:43.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.444+0000 7f5ce3994640 1 --2- 172.21.15.57:0/1437244678 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5cdc153aa0 0x7f5cdc15d840 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8009870 tx=0x7f5cd801ed30 comp rx=0 tx=0).stop 2023-10-30T14:39:43.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.444+0000 7f5ce3994640 1 -- 172.21.15.57:0/1437244678 shutdown_connections 2023-10-30T14:39:43.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.444+0000 7f5ce3994640 1 --2- 172.21.15.57:0/1437244678 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5cdc153aa0 0x7f5cdc15d840 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:43.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.444+0000 7f5ce3994640 1 -- 172.21.15.57:0/1437244678 >> 172.21.15.57:0/1437244678 conn(0x7f5cdc071fb0 msgr2=0x7f5cdc0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:43.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.444+0000 7f5ce3994640 1 -- 172.21.15.57:0/1437244678 shutdown_connections 2023-10-30T14:39:43.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.444+0000 7f5ce3994640 1 -- 172.21.15.57:0/1437244678 wait complete. 2023-10-30T14:39:43.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce3994640 1 Processor -- start 2023-10-30T14:39:43.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce3994640 1 -- start start 2023-10-30T14:39:43.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce3994640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc06b3b0 0x7f5cdc0ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:43.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce3994640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5cdc0701e0 0x7f5cdc0cbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:43.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce3994640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cdc153aa0 0x7f5cdc156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:43.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce3994640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc14e890 con 0x7f5cdc0701e0 2023-10-30T14:39:43.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce3994640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc156280 con 0x7f5cdc06b3b0 2023-10-30T14:39:43.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce3994640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc1552f0 con 0x7f5cdc153aa0 2023-10-30T14:39:43.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce0f2f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5cdc0701e0 0x7f5cdc0cbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:43.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce0f2f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5cdc0701e0 0x7f5cdc0cbcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46926/0 (socket says 172.21.15.57:46926) 2023-10-30T14:39:43.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce0f2f640 1 -- 172.21.15.57:0/3659418504 learned_addr learned my addr 172.21.15.57:0/3659418504 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:43.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce1730640 1 --2- 172.21.15.57:0/3659418504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc06b3b0 0x7f5cdc0ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:43.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce1f31640 1 --2- 172.21.15.57:0/3659418504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cdc153aa0 0x7f5cdc156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:43.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce0f2f640 1 -- 172.21.15.57:0/3659418504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cdc153aa0 msgr2=0x7f5cdc156710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:43.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce0f2f640 1 --2- 172.21.15.57:0/3659418504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cdc153aa0 0x7f5cdc156710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:43.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce0f2f640 1 -- 172.21.15.57:0/3659418504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc06b3b0 msgr2=0x7f5cdc0ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:43.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce0f2f640 1 --2- 172.21.15.57:0/3659418504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc06b3b0 0x7f5cdc0ccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:43.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce0f2f640 1 -- 172.21.15.57:0/3659418504 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd80094d0 con 0x7f5cdc0701e0 2023-10-30T14:39:43.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce0f2f640 1 --2- 172.21.15.57:0/3659418504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5cdc0701e0 0x7f5cdc0cbcd0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00c500 tx=0x7f5ccc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:43.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5cd1ffb640 1 -- 172.21.15.57:0/3659418504 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc005220 con 0x7f5cdc0701e0 2023-10-30T14:39:43.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5cd1ffb640 1 -- 172.21.15.57:0/3659418504 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ccc005380 con 0x7f5cdc0701e0 2023-10-30T14:39:43.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce3994640 1 -- 172.21.15.57:0/3659418504 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cdc0cc710 con 0x7f5cdc0701e0 2023-10-30T14:39:43.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5cd1ffb640 1 -- 172.21.15.57:0/3659418504 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc007270 con 0x7f5cdc0701e0 2023-10-30T14:39:43.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce3994640 1 -- 172.21.15.57:0/3659418504 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cdc155950 con 0x7f5cdc0701e0 2023-10-30T14:39:43.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.448+0000 7f5ce3994640 1 -- 172.21.15.57:0/3659418504 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cdc0672b0 con 0x7f5cdc0701e0 2023-10-30T14:39:43.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.452+0000 7f5cd1ffb640 1 -- 172.21.15.57:0/3659418504 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5ccc0073d0 con 0x7f5cdc0701e0 2023-10-30T14:39:43.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.452+0000 7f5cd1ffb640 1 --2- 172.21.15.57:0/3659418504 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5cc00664d0 0x7f5cc0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:43.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.452+0000 7f5cd1ffb640 1 -- 172.21.15.57:0/3659418504 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6943+0+0 (secure 0 0 0) 0x7f5ccc083840 con 0x7f5cdc0701e0 2023-10-30T14:39:43.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.452+0000 7f5ce1730640 1 --2- 172.21.15.57:0/3659418504 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5cc00664d0 0x7f5cc0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:43.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.452+0000 7f5ce1730640 1 --2- 172.21.15.57:0/3659418504 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5cc00664d0 0x7f5cc0068990 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5cd80b2000 tx=0x7f5cd8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:43.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.456+0000 7f5cd1ffb640 1 -- 172.21.15.57:0/3659418504 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5ccc0506d0 con 0x7f5cdc0701e0 2023-10-30T14:39:43.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.648+0000 7f5ce3994640 1 -- 172.21.15.57:0/3659418504 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5cdc04e170 con 0x7f5cdc0701e0 2023-10-30T14:39:43.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.652+0000 7f5cd1ffb640 1 -- 172.21.15.57:0/3659418504 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+6535 (secure 0 0 0) 0x7f5ccc070e60 con 0x7f5cdc0701e0 2023-10-30T14:39:43.653 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:39:43.654 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":136,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7056,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:39:37.108413+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"f","rank":2,"incarnation":128,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6839/2390289388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":2390289388},{"type":"v1","addr":"172.21.15.88:6839","nonce":2390289388}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:39:43.654 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 136 2023-10-30T14:39:43.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.652+0000 7f5ce3994640 1 -- 172.21.15.57:0/3659418504 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5cc00664d0 msgr2=0x7f5cc0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:43.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.652+0000 7f5ce3994640 1 --2- 172.21.15.57:0/3659418504 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5cc00664d0 0x7f5cc0068990 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5cd80b2000 tx=0x7f5cd8009040 comp rx=0 tx=0).stop 2023-10-30T14:39:43.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.652+0000 7f5ce3994640 1 -- 172.21.15.57:0/3659418504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5cdc0701e0 msgr2=0x7f5cdc0cbcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:43.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.652+0000 7f5ce3994640 1 --2- 172.21.15.57:0/3659418504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5cdc0701e0 0x7f5cdc0cbcd0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00c500 tx=0x7f5ccc00c9c0 comp rx=0 tx=0).stop 2023-10-30T14:39:43.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.652+0000 7f5ce3994640 1 -- 172.21.15.57:0/3659418504 shutdown_connections 2023-10-30T14:39:43.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.652+0000 7f5ce3994640 1 --2- 172.21.15.57:0/3659418504 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5cc00664d0 0x7f5cc0068990 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:43.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.652+0000 7f5ce3994640 1 --2- 172.21.15.57:0/3659418504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cdc153aa0 0x7f5cdc156710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:43.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.652+0000 7f5ce3994640 1 --2- 172.21.15.57:0/3659418504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5cdc0701e0 0x7f5cdc0cbcd0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:43.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.652+0000 7f5ce3994640 1 --2- 172.21.15.57:0/3659418504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc06b3b0 0x7f5cdc0ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:43.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.652+0000 7f5ce3994640 1 -- 172.21.15.57:0/3659418504 >> 172.21.15.57:0/3659418504 conn(0x7f5cdc071fb0 msgr2=0x7f5cdc152150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:43.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.652+0000 7f5ce3994640 1 -- 172.21.15.57:0/3659418504 shutdown_connections 2023-10-30T14:39:43.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:43.652+0000 7f5ce3994640 1 -- 172.21.15.57:0/3659418504 wait complete. 2023-10-30T14:39:43.674 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:39:45.675 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:39:45.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.828+0000 7ff2a8454640 1 Processor -- start 2023-10-30T14:39:45.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.828+0000 7ff2a8454640 1 -- start start 2023-10-30T14:39:45.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.828+0000 7ff2a8454640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff2a0153c80 0x7ff2a015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:45.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.828+0000 7ff2a8454640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a0159f20 con 0x7ff2a00c42a0 2023-10-30T14:39:45.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.828+0000 7ff2a8454640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a015d620 con 0x7ff2a00c4a10 2023-10-30T14:39:45.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.828+0000 7ff2a8454640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a015dec0 con 0x7ff2a0153c80 2023-10-30T14:39:45.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.828+0000 7ff2a69f1640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff2a00c42a0 0x7ff2a00ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:35820/0 (socket says 172.21.15.57:35820) 2023-10-30T14:39:45.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.828+0000 7ff2a69f1640 1 -- 172.21.15.57:0/2985612419 learned_addr learned my addr 172.21.15.57:0/2985612419 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:39:45.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.828+0000 7ff2a61f0640 1 --2- 172.21.15.57:0/2985612419 >> v2:172.21.15.88:3300/0 conn(0x7ff2a0153c80 0x7ff2a015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:45.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a51ee640 1 -- 172.21.15.57:0/2985612419 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630544666 0 0) 0x7ff2a0159f20 con 0x7ff2a00c42a0 2023-10-30T14:39:45.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a51ee640 1 -- 172.21.15.57:0/2985612419 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff28c003530 con 0x7ff2a00c42a0 2023-10-30T14:39:45.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a51ee640 1 -- 172.21.15.57:0/2985612419 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 913284410 0 0) 0x7ff28c003530 con 0x7ff2a00c42a0 2023-10-30T14:39:45.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a51ee640 1 -- 172.21.15.57:0/2985612419 >> v2:172.21.15.88:3300/0 conn(0x7ff2a0153c80 msgr2=0x7ff2a015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:45.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a51ee640 1 --2- 172.21.15.57:0/2985612419 >> v2:172.21.15.88:3300/0 conn(0x7ff2a0153c80 0x7ff2a015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:45.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a51ee640 1 -- 172.21.15.57:0/2985612419 >> v1:172.21.15.88:6789/0 conn(0x7ff2a00c4a10 legacy=0x7ff2a015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:45.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a51ee640 1 -- 172.21.15.57:0/2985612419 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a015d900 con 0x7ff2a00c42a0 2023-10-30T14:39:45.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a51ee640 1 -- 172.21.15.57:0/2985612419 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7ff298001a00 con 0x7ff2a00c42a0 2023-10-30T14:39:45.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a51ee640 1 -- 172.21.15.57:0/2985612419 >> v1:172.21.15.57:6789/0 conn(0x7ff2a00c42a0 legacy=0x7ff2a00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:45.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a51ee640 1 --2- 172.21.15.57:0/2985612419 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff28c003980 0x7ff28c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:45.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a51ee640 1 --2- 172.21.15.57:0/2985612419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff28c0047d0 0x7ff28c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:45.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a51ee640 1 --2- 172.21.15.57:0/2985612419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff28c005190 0x7ff28c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:45.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a51ee640 1 -- 172.21.15.57:0/2985612419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a015dec0 con 0x7ff28c0047d0 2023-10-30T14:39:45.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a51ee640 1 -- 172.21.15.57:0/2985612419 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c00c040 con 0x7ff28c005190 2023-10-30T14:39:45.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a51ee640 1 -- 172.21.15.57:0/2985612419 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c00c200 con 0x7ff28c003980 2023-10-30T14:39:45.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a59ef640 1 --2- 172.21.15.57:0/2985612419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff28c0047d0 0x7ff28c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:45.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a69f1640 1 --2- 172.21.15.57:0/2985612419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff28c005190 0x7ff28c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:45.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a61f0640 1 --2- 172.21.15.57:0/2985612419 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff28c003980 0x7ff28c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:45.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a59ef640 1 -- 172.21.15.57:0/2985612419 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff28c003980 msgr2=0x7ff28c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:45.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a59ef640 1 --2- 172.21.15.57:0/2985612419 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff28c003980 0x7ff28c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:45.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a59ef640 1 -- 172.21.15.57:0/2985612419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff28c005190 msgr2=0x7ff28c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:45.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a59ef640 1 --2- 172.21.15.57:0/2985612419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff28c005190 0x7ff28c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:45.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a59ef640 1 -- 172.21.15.57:0/2985612419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28c003530 con 0x7ff28c0047d0 2023-10-30T14:39:45.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.832+0000 7ff2a69f1640 1 -- 172.21.15.57:0/2985612419 reap_dead start 2023-10-30T14:39:45.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.836+0000 7ff2a59ef640 1 --2- 172.21.15.57:0/2985612419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff28c0047d0 0x7ff28c004c50 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7ff290001750 tx=0x7ff29001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ed68721b143bda server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:45.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.836+0000 7ff2a51ee640 1 -- 172.21.15.57:0/2985612419 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2900052d0 con 0x7ff28c0047d0 2023-10-30T14:39:45.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.836+0000 7ff2a51ee640 1 -- 172.21.15.57:0/2985612419 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2900a7050 con 0x7ff28c0047d0 2023-10-30T14:39:45.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.836+0000 7ff2a51ee640 1 -- 172.21.15.57:0/2985612419 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff290003600 con 0x7ff28c0047d0 2023-10-30T14:39:45.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.836+0000 7ff2a8454640 1 -- 172.21.15.57:0/2985612419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff28c0047d0 msgr2=0x7ff28c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:45.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.836+0000 7ff2a8454640 1 --2- 172.21.15.57:0/2985612419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff28c0047d0 0x7ff28c004c50 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7ff290001750 tx=0x7ff29001c3a0 comp rx=0 tx=0).stop 2023-10-30T14:39:45.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.836+0000 7ff2a8454640 1 -- 172.21.15.57:0/2985612419 shutdown_connections 2023-10-30T14:39:45.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.836+0000 7ff2a8454640 1 --2- 172.21.15.57:0/2985612419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff28c0047d0 0x7ff28c004c50 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:45.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.836+0000 7ff2a8454640 1 -- 172.21.15.57:0/2985612419 >> 172.21.15.57:0/2985612419 conn(0x7ff2a00715c0 msgr2=0x7ff2a00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:45.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.836+0000 7ff2a8454640 1 -- 172.21.15.57:0/2985612419 shutdown_connections 2023-10-30T14:39:45.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.836+0000 7ff2a8454640 1 -- 172.21.15.57:0/2985612419 wait complete. 2023-10-30T14:39:45.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.836+0000 7ff2a8454640 1 Processor -- start 2023-10-30T14:39:45.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.836+0000 7ff2a8454640 1 -- start start 2023-10-30T14:39:45.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.836+0000 7ff2a8454640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff28c0047d0 0x7ff2a00b2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:45.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.836+0000 7ff2a8454640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a00b5110 0x7ff2a015c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:45.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.836+0000 7ff2a8454640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2a01f2f20 0x7ff2a01f33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:45.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff2a8454640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a0154310 con 0x7ff2a01f2f20 2023-10-30T14:39:45.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff2a8454640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a01f2a90 con 0x7ff2a00b5110 2023-10-30T14:39:45.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff2a8454640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a01f38e0 con 0x7ff28c0047d0 2023-10-30T14:39:45.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff2a69f1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2a01f2f20 0x7ff2a01f33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:45.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff2a69f1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2a01f2f20 0x7ff2a01f33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43976/0 (socket says 172.21.15.57:43976) 2023-10-30T14:39:45.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff2a69f1640 1 -- 172.21.15.57:0/2736657848 learned_addr learned my addr 172.21.15.57:0/2736657848 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:45.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff2a61f0640 1 --2- 172.21.15.57:0/2736657848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff28c0047d0 0x7ff2a00b2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:45.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff2a59ef640 1 --2- 172.21.15.57:0/2736657848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a00b5110 0x7ff2a015c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:45.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff2a69f1640 1 -- 172.21.15.57:0/2736657848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff28c0047d0 msgr2=0x7ff2a00b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:45.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff2a69f1640 1 --2- 172.21.15.57:0/2736657848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff28c0047d0 0x7ff2a00b2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:45.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff2a69f1640 1 -- 172.21.15.57:0/2736657848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a00b5110 msgr2=0x7ff2a015c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:45.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff2a69f1640 1 --2- 172.21.15.57:0/2736657848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a00b5110 0x7ff2a015c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:45.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff2a69f1640 1 -- 172.21.15.57:0/2736657848 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a0154540 con 0x7ff2a01f2f20 2023-10-30T14:39:45.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff2a61f0640 1 --2- 172.21.15.57:0/2736657848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff28c0047d0 0x7ff2a00b2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:39:45.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff2a59ef640 1 --2- 172.21.15.57:0/2736657848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a00b5110 0x7ff2a015c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:39:45.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff2a69f1640 1 --2- 172.21.15.57:0/2736657848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2a01f2f20 0x7ff2a01f33a0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0072840 tx=0x7ff29800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:45.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff296ffd640 1 -- 172.21.15.57:0/2736657848 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29800ec40 con 0x7ff2a01f2f20 2023-10-30T14:39:45.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff296ffd640 1 -- 172.21.15.57:0/2736657848 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29800eda0 con 0x7ff2a01f2f20 2023-10-30T14:39:45.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff2a8454640 1 -- 172.21.15.57:0/2736657848 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a0154820 con 0x7ff2a01f2f20 2023-10-30T14:39:45.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff296ffd640 1 -- 172.21.15.57:0/2736657848 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff298014670 con 0x7ff2a01f2f20 2023-10-30T14:39:45.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff2a8454640 1 -- 172.21.15.57:0/2736657848 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a0159550 con 0x7ff2a01f2f20 2023-10-30T14:39:45.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff2a8454640 1 -- 172.21.15.57:0/2736657848 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a01f2c40 con 0x7ff2a01f2f20 2023-10-30T14:39:45.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.840+0000 7ff296ffd640 1 -- 172.21.15.57:0/2736657848 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff2980147d0 con 0x7ff2a01f2f20 2023-10-30T14:39:45.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.844+0000 7ff296ffd640 1 --2- 172.21.15.57:0/2736657848 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff2840664a0 0x7ff284068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:45.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.844+0000 7ff296ffd640 1 -- 172.21.15.57:0/2736657848 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7ff2980842f0 con 0x7ff2a01f2f20 2023-10-30T14:39:45.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.844+0000 7ff2a61f0640 1 --2- 172.21.15.57:0/2736657848 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff2840664a0 0x7ff284068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:45.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.844+0000 7ff2a61f0640 1 --2- 172.21.15.57:0/2736657848 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff2840664a0 0x7ff284068960 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0063a60 tx=0x7ff29c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:45.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:45.848+0000 7ff296ffd640 1 -- 172.21.15.57:0/2736657848 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff2a01f2c40 con 0x7ff2a01f2f20 2023-10-30T14:39:46.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.036+0000 7ff2a8454640 1 -- 172.21.15.57:0/2736657848 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2a01f2c40 con 0x7ff2a01f2f20 2023-10-30T14:39:46.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.036+0000 7ff296ffd640 1 -- 172.21.15.57:0/2736657848 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+5739 (secure 0 0 0) 0x7ff2a01f2c40 con 0x7ff2a01f2f20 2023-10-30T14:39:46.039 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:39:46.039 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:39:45.086783+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:39:46.040 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 137 2023-10-30T14:39:46.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.036+0000 7ff2a8454640 1 -- 172.21.15.57:0/2736657848 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff2840664a0 msgr2=0x7ff284068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:46.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.036+0000 7ff2a8454640 1 --2- 172.21.15.57:0/2736657848 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff2840664a0 0x7ff284068960 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0063a60 tx=0x7ff29c008040 comp rx=0 tx=0).stop 2023-10-30T14:39:46.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.036+0000 7ff2a8454640 1 -- 172.21.15.57:0/2736657848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2a01f2f20 msgr2=0x7ff2a01f33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:46.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.036+0000 7ff2a8454640 1 --2- 172.21.15.57:0/2736657848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2a01f2f20 0x7ff2a01f33a0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0072840 tx=0x7ff29800e530 comp rx=0 tx=0).stop 2023-10-30T14:39:46.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.040+0000 7ff2a8454640 1 -- 172.21.15.57:0/2736657848 shutdown_connections 2023-10-30T14:39:46.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.040+0000 7ff2a8454640 1 --2- 172.21.15.57:0/2736657848 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff2840664a0 0x7ff284068960 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:46.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.040+0000 7ff2a8454640 1 --2- 172.21.15.57:0/2736657848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2a01f2f20 0x7ff2a01f33a0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:46.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.040+0000 7ff2a8454640 1 --2- 172.21.15.57:0/2736657848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a00b5110 0x7ff2a015c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:46.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.040+0000 7ff2a8454640 1 --2- 172.21.15.57:0/2736657848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff28c0047d0 0x7ff2a00b2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:46.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.040+0000 7ff2a8454640 1 -- 172.21.15.57:0/2736657848 >> 172.21.15.57:0/2736657848 conn(0x7ff2a00715c0 msgr2=0x7ff2a00cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:46.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.040+0000 7ff2a8454640 1 -- 172.21.15.57:0/2736657848 shutdown_connections 2023-10-30T14:39:46.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.040+0000 7ff2a8454640 1 -- 172.21.15.57:0/2736657848 wait complete. 2023-10-30T14:39:46.060 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-30T14:39:46.060 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:39:46.061 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:39:46.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.200+0000 7feec23f5640 1 Processor -- start 2023-10-30T14:39:46.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.200+0000 7feec23f5640 1 -- start start 2023-10-30T14:39:46.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.200+0000 7feec23f5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feebc1537f0 0x7feebc0701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:46.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feec23f5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7feebc06b2f0 0x7feebc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:46.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feec23f5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feebc0c2aa0 con 0x7feebc15a2e0 2023-10-30T14:39:46.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feec23f5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feebc0c3ab0 con 0x7feebc06b2f0 2023-10-30T14:39:46.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feec23f5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feebc0706e0 con 0x7feebc1537f0 2023-10-30T14:39:46.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebb7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7feebc15a2e0 0x7feebc0ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:35836/0 (socket says 172.21.15.57:35836) 2023-10-30T14:39:46.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebb7fe640 1 -- 172.21.15.57:0/2561202129 learned_addr learned my addr 172.21.15.57:0/2561202129 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:39:46.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feec0992640 1 --2- 172.21.15.57:0/2561202129 >> v2:172.21.15.88:3300/0 conn(0x7feebc06b2f0 0x7feebc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:46.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebbfff640 1 --2- 172.21.15.57:0/2561202129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feebc1537f0 0x7feebc0701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:46.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebaffd640 1 -- 172.21.15.57:0/2561202129 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024998194 0 0) 0x7feebc0c2aa0 con 0x7feebc15a2e0 2023-10-30T14:39:46.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebaffd640 1 -- 172.21.15.57:0/2561202129 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feea4003530 con 0x7feebc15a2e0 2023-10-30T14:39:46.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebaffd640 1 -- 172.21.15.57:0/2561202129 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 552630989 0 0) 0x7feea4003530 con 0x7feebc15a2e0 2023-10-30T14:39:46.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebaffd640 1 -- 172.21.15.57:0/2561202129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feebc1537f0 msgr2=0x7feebc0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:46.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebaffd640 1 --2- 172.21.15.57:0/2561202129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feebc1537f0 0x7feebc0701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:46.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebaffd640 1 -- 172.21.15.57:0/2561202129 >> v2:172.21.15.88:3300/0 conn(0x7feebc06b2f0 msgr2=0x7feebc06b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:46.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebaffd640 1 --2- 172.21.15.57:0/2561202129 >> v2:172.21.15.88:3300/0 conn(0x7feebc06b2f0 0x7feebc06b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:46.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebaffd640 1 -- 172.21.15.57:0/2561202129 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feebc06bca0 con 0x7feebc15a2e0 2023-10-30T14:39:46.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebaffd640 1 -- 172.21.15.57:0/2561202129 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7feeb0001680 con 0x7feebc15a2e0 2023-10-30T14:39:46.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebaffd640 1 -- 172.21.15.57:0/2561202129 >> v1:172.21.15.57:6789/0 conn(0x7feebc15a2e0 legacy=0x7feebc0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:46.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebaffd640 1 --2- 172.21.15.57:0/2561202129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feebc1537f0 0x7feea4003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:46.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebaffd640 1 --2- 172.21.15.57:0/2561202129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feea4003f00 0x7feea40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:46.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebaffd640 1 --2- 172.21.15.57:0/2561202129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feea4004d10 0x7feea4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:46.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebaffd640 1 -- 172.21.15.57:0/2561202129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feebc0706e0 con 0x7feea4003f00 2023-10-30T14:39:46.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebaffd640 1 -- 172.21.15.57:0/2561202129 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea400b990 con 0x7feea4004d10 2023-10-30T14:39:46.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebaffd640 1 -- 172.21.15.57:0/2561202129 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea400bb00 con 0x7feebc1537f0 2023-10-30T14:39:46.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebb7fe640 1 --2- 172.21.15.57:0/2561202129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feea4003f00 0x7feea40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:46.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebbfff640 1 --2- 172.21.15.57:0/2561202129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feebc1537f0 0x7feea4003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:46.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feec0992640 1 --2- 172.21.15.57:0/2561202129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feea4004d10 0x7feea4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:46.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebb7fe640 1 -- 172.21.15.57:0/2561202129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feebc1537f0 msgr2=0x7feea4003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:46.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebb7fe640 1 --2- 172.21.15.57:0/2561202129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feebc1537f0 0x7feea4003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:46.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebb7fe640 1 -- 172.21.15.57:0/2561202129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feea4004d10 msgr2=0x7feea4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:46.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebb7fe640 1 --2- 172.21.15.57:0/2561202129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feea4004d10 0x7feea4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:46.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feebb7fe640 1 -- 172.21.15.57:0/2561202129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea4003530 con 0x7feea4003f00 2023-10-30T14:39:46.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.204+0000 7feec0992640 1 --2- 172.21.15.57:0/2561202129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feea4004d10 0x7feea4005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:39:46.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.208+0000 7feebb7fe640 1 --2- 172.21.15.57:0/2561202129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feea4003f00 0x7feea40047d0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7feeb000c1c0 tx=0x7feeb001cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=510da73bea876515 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:46.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.208+0000 7feebaffd640 1 -- 172.21.15.57:0/2561202129 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb00a5940 con 0x7feea4003f00 2023-10-30T14:39:46.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.208+0000 7feebaffd640 1 -- 172.21.15.57:0/2561202129 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb00a5ac0 con 0x7feea4003f00 2023-10-30T14:39:46.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.208+0000 7feebaffd640 1 -- 172.21.15.57:0/2561202129 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb00058e0 con 0x7feea4003f00 2023-10-30T14:39:46.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.208+0000 7feec23f5640 1 -- 172.21.15.57:0/2561202129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feea4003f00 msgr2=0x7feea40047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:46.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.208+0000 7feec23f5640 1 --2- 172.21.15.57:0/2561202129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feea4003f00 0x7feea40047d0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7feeb000c1c0 tx=0x7feeb001cc40 comp rx=0 tx=0).stop 2023-10-30T14:39:46.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.208+0000 7feec0992640 1 -- 172.21.15.57:0/2561202129 reap_dead start 2023-10-30T14:39:46.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.208+0000 7feec23f5640 1 -- 172.21.15.57:0/2561202129 shutdown_connections 2023-10-30T14:39:46.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.208+0000 7feec23f5640 1 -- 172.21.15.57:0/2561202129 >> 172.21.15.57:0/2561202129 conn(0x7feebc071fb0 msgr2=0x7feebc0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:46.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.208+0000 7feec23f5640 1 -- 172.21.15.57:0/2561202129 shutdown_connections 2023-10-30T14:39:46.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.208+0000 7feec23f5640 1 -- 172.21.15.57:0/2561202129 wait complete. 2023-10-30T14:39:46.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.208+0000 7feec23f5640 1 Processor -- start 2023-10-30T14:39:46.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.208+0000 7feec23f5640 1 -- start start 2023-10-30T14:39:46.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feec23f5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feea4003f00 0x7feebc1f28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:46.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feec23f5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feebc0c3980 0x7feebc06c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:46.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feec23f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feebc15aa20 0x7feebc1ee800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:46.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feec23f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feebc1eed40 con 0x7feebc15aa20 2023-10-30T14:39:46.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feec23f5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feebc15be00 con 0x7feea4003f00 2023-10-30T14:39:46.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feec23f5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feebc15c0e0 con 0x7feebc0c3980 2023-10-30T14:39:46.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feec0992640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feebc15aa20 0x7feebc1ee800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:46.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feec0992640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feebc15aa20 0x7feebc1ee800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43984/0 (socket says 172.21.15.57:43984) 2023-10-30T14:39:46.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feec0992640 1 -- 172.21.15.57:0/1075589837 learned_addr learned my addr 172.21.15.57:0/1075589837 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:46.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feebbfff640 1 --2- 172.21.15.57:0/1075589837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feea4003f00 0x7feebc1f28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:46.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feebb7fe640 1 --2- 172.21.15.57:0/1075589837 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feebc0c3980 0x7feebc06c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:46.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feec0992640 1 -- 172.21.15.57:0/1075589837 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feebc0c3980 msgr2=0x7feebc06c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:46.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feec0992640 1 --2- 172.21.15.57:0/1075589837 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feebc0c3980 0x7feebc06c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:46.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feec0992640 1 -- 172.21.15.57:0/1075589837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feea4003f00 msgr2=0x7feebc1f28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:46.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feec0992640 1 --2- 172.21.15.57:0/1075589837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feea4003f00 0x7feebc1f28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:46.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feec0992640 1 -- 172.21.15.57:0/1075589837 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feebc15c3c0 con 0x7feebc15aa20 2023-10-30T14:39:46.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feec0992640 1 --2- 172.21.15.57:0/1075589837 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feebc15aa20 0x7feebc1ee800 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7feebc06d180 tx=0x7feeb4010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:46.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feeb8ff9640 1 -- 172.21.15.57:0/1075589837 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb401b070 con 0x7feebc15aa20 2023-10-30T14:39:46.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feeb8ff9640 1 -- 172.21.15.57:0/1075589837 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb400f040 con 0x7feebc15aa20 2023-10-30T14:39:46.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feeb8ff9640 1 -- 172.21.15.57:0/1075589837 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb400e050 con 0x7feebc15aa20 2023-10-30T14:39:46.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feec23f5640 1 -- 172.21.15.57:0/1075589837 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feebc1f1e10 con 0x7feebc15aa20 2023-10-30T14:39:46.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feec23f5640 1 -- 172.21.15.57:0/1075589837 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feebc1f23d0 con 0x7feebc15aa20 2023-10-30T14:39:46.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.212+0000 7feec23f5640 1 -- 172.21.15.57:0/1075589837 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feebc04e170 con 0x7feebc15aa20 2023-10-30T14:39:46.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.216+0000 7feeb8ff9640 1 -- 172.21.15.57:0/1075589837 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feeb4015440 con 0x7feebc15aa20 2023-10-30T14:39:46.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.216+0000 7feeb8ff9640 1 --2- 172.21.15.57:0/1075589837 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7feea80664a0 0x7feea8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:46.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.216+0000 7feeb8ff9640 1 -- 172.21.15.57:0/1075589837 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7feeb4084770 con 0x7feebc15aa20 2023-10-30T14:39:46.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.216+0000 7feebbfff640 1 --2- 172.21.15.57:0/1075589837 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7feea80664a0 0x7feea8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:46.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.216+0000 7feebbfff640 1 --2- 172.21.15.57:0/1075589837 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7feea80664a0 0x7feea8068960 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7feeac009b00 tx=0x7feeac009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:46.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.220+0000 7feeb8ff9640 1 -- 172.21.15.57:0/1075589837 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feeb4051620 con 0x7feebc15aa20 2023-10-30T14:39:46.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.504+0000 7feec23f5640 1 -- 172.21.15.57:0/1075589837 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feebc06b920 con 0x7feebc15aa20 2023-10-30T14:39:46.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.508+0000 7feeb8ff9640 1 -- 172.21.15.57:0/1075589837 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+5739 (secure 0 0 0) 0x7feeb405b090 con 0x7feebc15aa20 2023-10-30T14:39:46.509 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:39:46.510 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:39:45.086783+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:39:46.510 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 137 2023-10-30T14:39:46.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.508+0000 7feec23f5640 1 -- 172.21.15.57:0/1075589837 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7feea80664a0 msgr2=0x7feea8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:46.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.508+0000 7feec23f5640 1 --2- 172.21.15.57:0/1075589837 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7feea80664a0 0x7feea8068960 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7feeac009b00 tx=0x7feeac009190 comp rx=0 tx=0).stop 2023-10-30T14:39:46.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.508+0000 7feec23f5640 1 -- 172.21.15.57:0/1075589837 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feebc15aa20 msgr2=0x7feebc1ee800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:46.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.508+0000 7feec23f5640 1 --2- 172.21.15.57:0/1075589837 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feebc15aa20 0x7feebc1ee800 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7feebc06d180 tx=0x7feeb4010590 comp rx=0 tx=0).stop 2023-10-30T14:39:46.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.508+0000 7feec23f5640 1 -- 172.21.15.57:0/1075589837 shutdown_connections 2023-10-30T14:39:46.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.508+0000 7feec23f5640 1 --2- 172.21.15.57:0/1075589837 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7feea80664a0 0x7feea8068960 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:46.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.508+0000 7feec23f5640 1 --2- 172.21.15.57:0/1075589837 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feebc15aa20 0x7feebc1ee800 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:46.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.508+0000 7feec23f5640 1 --2- 172.21.15.57:0/1075589837 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feebc0c3980 0x7feebc06c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:46.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.508+0000 7feec23f5640 1 --2- 172.21.15.57:0/1075589837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feea4003f00 0x7feebc1f28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:46.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.508+0000 7feec23f5640 1 -- 172.21.15.57:0/1075589837 >> 172.21.15.57:0/1075589837 conn(0x7feebc071fb0 msgr2=0x7feebc152720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:46.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.508+0000 7feec23f5640 1 -- 172.21.15.57:0/1075589837 shutdown_connections 2023-10-30T14:39:46.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:46.508+0000 7feec23f5640 1 -- 172.21.15.57:0/1075589837 wait complete. 2023-10-30T14:39:46.531 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 137, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 137, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6414": { "addr": "172.21.15.57:6834/2321408149", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2321408149, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2321408149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6414, "incarnation": 99, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 18 }, "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 44 }, "gid_7056": { "addr": "172.21.15.57:6837/2307041430", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 2307041430, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 2307041430, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7056, "incarnation": 137, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:39:45.086783+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6414, "mds_1": 6747, "mds_2": 7056 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:39:48.532 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:39:48.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.684+0000 7faeea571640 1 Processor -- start 2023-10-30T14:39:48.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.684+0000 7faeea571640 1 -- start start 2023-10-30T14:39:48.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.688+0000 7faeea571640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faee40c8eb0 0x7faee415cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:48.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.688+0000 7faeea571640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7faee415ac70 0x7faee415a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:48.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.688+0000 7faeea571640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faee406aeb0 0x7faee406b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:48.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.688+0000 7faeea571640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee4159800 con 0x7faee415ac70 2023-10-30T14:39:48.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.688+0000 7faeea571640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee415dcc0 con 0x7faee406aeb0 2023-10-30T14:39:48.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.688+0000 7faeea571640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee4159d80 con 0x7faee40c8eb0 2023-10-30T14:39:48.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.688+0000 7faee8d6e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7faee415ac70 0x7faee415a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:48.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.688+0000 7faee8d6e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7faee415ac70 0x7faee415a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43986/0 (socket says 172.21.15.57:43986) 2023-10-30T14:39:48.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.688+0000 7faee8d6e640 1 -- 172.21.15.57:0/4264768140 learned_addr learned my addr 172.21.15.57:0/4264768140 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:48.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.688+0000 7faee956f640 1 --2- 172.21.15.57:0/4264768140 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faee40c8eb0 0x7faee415cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:48.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.688+0000 7faee9d70640 1 --2- 172.21.15.57:0/4264768140 >> v2:172.21.15.88:3300/0 conn(0x7faee406aeb0 0x7faee406b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:48.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.688+0000 7faee8d6e640 1 -- 172.21.15.57:0/4264768140 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faee40c8eb0 msgr2=0x7faee415cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:48.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.688+0000 7faee8d6e640 1 --2- 172.21.15.57:0/4264768140 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faee40c8eb0 0x7faee415cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:48.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.688+0000 7faee8d6e640 1 -- 172.21.15.57:0/4264768140 >> v2:172.21.15.88:3300/0 conn(0x7faee406aeb0 msgr2=0x7faee406b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:48.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.688+0000 7faee8d6e640 1 --2- 172.21.15.57:0/4264768140 >> v2:172.21.15.88:3300/0 conn(0x7faee406aeb0 0x7faee406b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:48.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.688+0000 7faee8d6e640 1 -- 172.21.15.57:0/4264768140 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee415a750 con 0x7faee415ac70 2023-10-30T14:39:48.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.692+0000 7faee8d6e640 1 --2- 172.21.15.57:0/4264768140 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faee415ac70 0x7faee415a210 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7faee408cfd0 tx=0x7faed401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9c706988c0e628 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:48.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.692+0000 7faedbfff640 1 -- 172.21.15.57:0/4264768140 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed40a89b0 con 0x7faee415ac70 2023-10-30T14:39:48.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.692+0000 7faedbfff640 1 -- 172.21.15.57:0/4264768140 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed40a8b10 con 0x7faee415ac70 2023-10-30T14:39:48.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.692+0000 7faedbfff640 1 -- 172.21.15.57:0/4264768140 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed40a8ea0 con 0x7faee415ac70 2023-10-30T14:39:48.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.692+0000 7faeea571640 1 -- 172.21.15.57:0/4264768140 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faee415ac70 msgr2=0x7faee415a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:48.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.692+0000 7faeea571640 1 --2- 172.21.15.57:0/4264768140 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faee415ac70 0x7faee415a210 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7faee408cfd0 tx=0x7faed401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:39:48.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.692+0000 7faeea571640 1 -- 172.21.15.57:0/4264768140 shutdown_connections 2023-10-30T14:39:48.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.692+0000 7faeea571640 1 --2- 172.21.15.57:0/4264768140 >> v2:172.21.15.88:3300/0 conn(0x7faee406aeb0 0x7faee406b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:48.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.692+0000 7faeea571640 1 --2- 172.21.15.57:0/4264768140 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faee415ac70 0x7faee415a210 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:48.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.692+0000 7faeea571640 1 --2- 172.21.15.57:0/4264768140 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faee40c8eb0 0x7faee415cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:48.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.692+0000 7faeea571640 1 -- 172.21.15.57:0/4264768140 >> 172.21.15.57:0/4264768140 conn(0x7faee40cef00 msgr2=0x7faee40cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:48.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.692+0000 7faeea571640 1 -- 172.21.15.57:0/4264768140 shutdown_connections 2023-10-30T14:39:48.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.692+0000 7faeea571640 1 -- 172.21.15.57:0/4264768140 wait complete. 2023-10-30T14:39:48.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.692+0000 7faeea571640 1 Processor -- start 2023-10-30T14:39:48.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.692+0000 7faeea571640 1 -- start start 2023-10-30T14:39:48.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faeea571640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faee406aeb0 0x7faee41ef930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:48.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faeea571640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faee40c8eb0 0x7faee4154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:48.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faeea571640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faee415ac70 0x7faee40cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:48.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faeea571640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee41efe70 con 0x7faee40c8eb0 2023-10-30T14:39:48.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faeea571640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee40c9b00 con 0x7faee406aeb0 2023-10-30T14:39:48.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faeea571640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee41e63c0 con 0x7faee415ac70 2023-10-30T14:39:48.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faee8d6e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faee40c8eb0 0x7faee4154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:48.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faee8d6e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faee40c8eb0 0x7faee4154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43988/0 (socket says 172.21.15.57:43988) 2023-10-30T14:39:48.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faee8d6e640 1 -- 172.21.15.57:0/2529161808 learned_addr learned my addr 172.21.15.57:0/2529161808 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:48.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faee956f640 1 --2- 172.21.15.57:0/2529161808 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faee406aeb0 0x7faee41ef930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:48.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faee9d70640 1 --2- 172.21.15.57:0/2529161808 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faee415ac70 0x7faee40cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:48.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faee8d6e640 1 -- 172.21.15.57:0/2529161808 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faee415ac70 msgr2=0x7faee40cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:48.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faee8d6e640 1 --2- 172.21.15.57:0/2529161808 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faee415ac70 0x7faee40cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:48.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faee8d6e640 1 -- 172.21.15.57:0/2529161808 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faee406aeb0 msgr2=0x7faee41ef930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:48.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faee8d6e640 1 --2- 172.21.15.57:0/2529161808 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faee406aeb0 0x7faee41ef930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:48.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faee8d6e640 1 -- 172.21.15.57:0/2529161808 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed40094d0 con 0x7faee40c8eb0 2023-10-30T14:39:48.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faee8d6e640 1 --2- 172.21.15.57:0/2529161808 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faee40c8eb0 0x7faee4154170 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7faed401d3a0 tx=0x7faed401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:48.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faed9ffb640 1 -- 172.21.15.57:0/2529161808 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed40ba600 con 0x7faee40c8eb0 2023-10-30T14:39:48.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faed9ffb640 1 -- 172.21.15.57:0/2529161808 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed40ba760 con 0x7faee40c8eb0 2023-10-30T14:39:48.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faed9ffb640 1 -- 172.21.15.57:0/2529161808 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed40bab10 con 0x7faee40c8eb0 2023-10-30T14:39:48.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faeea571640 1 -- 172.21.15.57:0/2529161808 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee40c9d50 con 0x7faee40c8eb0 2023-10-30T14:39:48.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faeea571640 1 -- 172.21.15.57:0/2529161808 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee41e5d70 con 0x7faee40c8eb0 2023-10-30T14:39:48.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.696+0000 7faeea571640 1 -- 172.21.15.57:0/2529161808 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee404e170 con 0x7faee40c8eb0 2023-10-30T14:39:48.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.700+0000 7faed9ffb640 1 -- 172.21.15.57:0/2529161808 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faed40bac70 con 0x7faee40c8eb0 2023-10-30T14:39:48.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.700+0000 7faed9ffb640 1 --2- 172.21.15.57:0/2529161808 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faec80664a0 0x7faec8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:48.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.700+0000 7faed9ffb640 1 -- 172.21.15.57:0/2529161808 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7faed41229d0 con 0x7faee40c8eb0 2023-10-30T14:39:48.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.700+0000 7faee956f640 1 --2- 172.21.15.57:0/2529161808 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faec80664a0 0x7faec8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:48.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.700+0000 7faee956f640 1 --2- 172.21.15.57:0/2529161808 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faec80664a0 0x7faec8068960 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7faee000b9d0 tx=0x7faee0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:48.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.704+0000 7faed9ffb640 1 -- 172.21.15.57:0/2529161808 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faed40ef9a0 con 0x7faee40c8eb0 2023-10-30T14:39:48.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.892+0000 7faeea571640 1 -- 172.21.15.57:0/2529161808 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faee415a350 con 0x7faee40c8eb0 2023-10-30T14:39:48.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.892+0000 7faed9ffb640 1 -- 172.21.15.57:0/2529161808 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+5739 (secure 0 0 0) 0x7faed40b5090 con 0x7faee40c8eb0 2023-10-30T14:39:48.897 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:39:48.897 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:39:45.086783+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:39:48.897 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 137 2023-10-30T14:39:48.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.896+0000 7faeea571640 1 -- 172.21.15.57:0/2529161808 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faec80664a0 msgr2=0x7faec8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:48.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.896+0000 7faeea571640 1 --2- 172.21.15.57:0/2529161808 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faec80664a0 0x7faec8068960 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7faee000b9d0 tx=0x7faee0008040 comp rx=0 tx=0).stop 2023-10-30T14:39:48.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.896+0000 7faeea571640 1 -- 172.21.15.57:0/2529161808 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faee40c8eb0 msgr2=0x7faee4154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:48.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.896+0000 7faeea571640 1 --2- 172.21.15.57:0/2529161808 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faee40c8eb0 0x7faee4154170 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7faed401d3a0 tx=0x7faed401ba90 comp rx=0 tx=0).stop 2023-10-30T14:39:48.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.896+0000 7faeea571640 1 -- 172.21.15.57:0/2529161808 shutdown_connections 2023-10-30T14:39:48.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.896+0000 7faeea571640 1 --2- 172.21.15.57:0/2529161808 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faec80664a0 0x7faec8068960 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:48.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.896+0000 7faeea571640 1 --2- 172.21.15.57:0/2529161808 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faee415ac70 0x7faee40cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:48.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.896+0000 7faeea571640 1 --2- 172.21.15.57:0/2529161808 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faee40c8eb0 0x7faee4154170 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:48.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.896+0000 7faeea571640 1 --2- 172.21.15.57:0/2529161808 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faee406aeb0 0x7faee41ef930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:48.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.896+0000 7faeea571640 1 -- 172.21.15.57:0/2529161808 >> 172.21.15.57:0/2529161808 conn(0x7faee40cef00 msgr2=0x7faee40c6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:48.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.896+0000 7faeea571640 1 -- 172.21.15.57:0/2529161808 shutdown_connections 2023-10-30T14:39:48.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:48.896+0000 7faeea571640 1 -- 172.21.15.57:0/2529161808 wait complete. 2023-10-30T14:39:48.917 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:39:50.919 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:39:51.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.060+0000 7f49ce963640 1 Processor -- start 2023-10-30T14:39:51.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.060+0000 7f49ce963640 1 -- start start 2023-10-30T14:39:51.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.064+0000 7f49ce963640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49c815ac70 0x7f49c815cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:51.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.064+0000 7f49ce963640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f49c806aeb0 0x7f49c8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:51.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.064+0000 7f49ce963640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c815dcc0 con 0x7f49c80c8eb0 2023-10-30T14:39:51.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.064+0000 7f49ce963640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c8159800 con 0x7f49c806aeb0 2023-10-30T14:39:51.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.064+0000 7f49ce963640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c8159d80 con 0x7f49c815ac70 2023-10-30T14:39:51.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.064+0000 7f49ce162640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f49c806aeb0 0x7f49c8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:51.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.064+0000 7f49ce162640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f49c806aeb0 0x7f49c8069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44002/0 (socket says 172.21.15.57:44002) 2023-10-30T14:39:51.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.064+0000 7f49ce162640 1 -- 172.21.15.57:0/3561910433 learned_addr learned my addr 172.21.15.57:0/3561910433 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:51.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.064+0000 7f49cd160640 1 --2- 172.21.15.57:0/3561910433 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49c815ac70 0x7f49c815cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:51.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.064+0000 7f49ce162640 1 -- 172.21.15.57:0/3561910433 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49c815ac70 msgr2=0x7f49c815cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:51.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.064+0000 7f49ce162640 1 --2- 172.21.15.57:0/3561910433 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49c815ac70 0x7f49c815cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:51.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.064+0000 7f49ce162640 1 -- 172.21.15.57:0/3561910433 >> v1:172.21.15.88:6789/0 conn(0x7f49c80c8eb0 legacy=0x7f49c815a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:51.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.064+0000 7f49ce162640 1 -- 172.21.15.57:0/3561910433 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c815d420 con 0x7f49c806aeb0 2023-10-30T14:39:51.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.068+0000 7f49ce162640 1 --2- 172.21.15.57:0/3561910433 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49c806aeb0 0x7f49c8069e90 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009e50 tx=0x7f49c001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3642c20cd89979b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:51.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.068+0000 7f49cc95f640 1 -- 172.21.15.57:0/3561910433 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c00af400 con 0x7f49c806aeb0 2023-10-30T14:39:51.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.068+0000 7f49cc95f640 1 -- 172.21.15.57:0/3561910433 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c00ab050 con 0x7f49c806aeb0 2023-10-30T14:39:51.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.068+0000 7f49cc95f640 1 -- 172.21.15.57:0/3561910433 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c00af780 con 0x7f49c806aeb0 2023-10-30T14:39:51.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.068+0000 7f49ce963640 1 -- 172.21.15.57:0/3561910433 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49c806aeb0 msgr2=0x7f49c8069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:51.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.068+0000 7f49ce963640 1 --2- 172.21.15.57:0/3561910433 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49c806aeb0 0x7f49c8069e90 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009e50 tx=0x7f49c001f1f0 comp rx=0 tx=0).stop 2023-10-30T14:39:51.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.068+0000 7f49ce963640 1 -- 172.21.15.57:0/3561910433 shutdown_connections 2023-10-30T14:39:51.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.068+0000 7f49ce963640 1 --2- 172.21.15.57:0/3561910433 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49c806aeb0 0x7f49c8069e90 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:51.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.068+0000 7f49ce963640 1 --2- 172.21.15.57:0/3561910433 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49c815ac70 0x7f49c815cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:51.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.068+0000 7f49ce963640 1 -- 172.21.15.57:0/3561910433 >> 172.21.15.57:0/3561910433 conn(0x7f49c80cef00 msgr2=0x7f49c80cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:51.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.068+0000 7f49ce963640 1 -- 172.21.15.57:0/3561910433 shutdown_connections 2023-10-30T14:39:51.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.068+0000 7f49ce963640 1 -- 172.21.15.57:0/3561910433 wait complete. 2023-10-30T14:39:51.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49ce963640 1 Processor -- start 2023-10-30T14:39:51.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49ce963640 1 -- start start 2023-10-30T14:39:51.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49ce963640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49c806aeb0 0x7f49c80c76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:51.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49ce963640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49c80c8eb0 0x7f49c80c4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:51.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49ce963640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49c815ac70 0x7f49c80c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:51.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49ce963640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c814e070 con 0x7f49c815ac70 2023-10-30T14:39:51.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49ce963640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c814e280 con 0x7f49c80c8eb0 2023-10-30T14:39:51.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49ce963640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c814e560 con 0x7f49c806aeb0 2023-10-30T14:39:51.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49ce162640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49c815ac70 0x7f49c80c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:51.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49ce162640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49c815ac70 0x7f49c80c4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44012/0 (socket says 172.21.15.57:44012) 2023-10-30T14:39:51.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49ce162640 1 -- 172.21.15.57:0/2196307720 learned_addr learned my addr 172.21.15.57:0/2196307720 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:51.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49cd961640 1 --2- 172.21.15.57:0/2196307720 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49c806aeb0 0x7f49c80c76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:51.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49cd160640 1 --2- 172.21.15.57:0/2196307720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49c80c8eb0 0x7f49c80c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:51.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49ce162640 1 -- 172.21.15.57:0/2196307720 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49c806aeb0 msgr2=0x7f49c80c76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:51.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49ce162640 1 --2- 172.21.15.57:0/2196307720 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49c806aeb0 0x7f49c80c76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:51.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49ce162640 1 -- 172.21.15.57:0/2196307720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49c80c8eb0 msgr2=0x7f49c80c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:51.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49ce162640 1 --2- 172.21.15.57:0/2196307720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49c80c8eb0 0x7f49c80c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:51.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49ce162640 1 -- 172.21.15.57:0/2196307720 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c0009b30 con 0x7f49c815ac70 2023-10-30T14:39:51.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49cd160640 1 --2- 172.21.15.57:0/2196307720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49c80c8eb0 0x7f49c80c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:39:51.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49ce162640 1 --2- 172.21.15.57:0/2196307720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49c815ac70 0x7f49c80c4a70 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f49c001ce20 tx=0x7f49c001bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:51.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49be7fc640 1 -- 172.21.15.57:0/2196307720 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c00c0070 con 0x7f49c815ac70 2023-10-30T14:39:51.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49be7fc640 1 -- 172.21.15.57:0/2196307720 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c0009510 con 0x7f49c815ac70 2023-10-30T14:39:51.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49be7fc640 1 -- 172.21.15.57:0/2196307720 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c00afcf0 con 0x7f49c815ac70 2023-10-30T14:39:51.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49ce963640 1 -- 172.21.15.57:0/2196307720 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c814e840 con 0x7f49c815ac70 2023-10-30T14:39:51.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.072+0000 7f49ce963640 1 -- 172.21.15.57:0/2196307720 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c814ee00 con 0x7f49c815ac70 2023-10-30T14:39:51.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.076+0000 7f49ce963640 1 -- 172.21.15.57:0/2196307720 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c814eff0 con 0x7f49c815ac70 2023-10-30T14:39:51.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.076+0000 7f49be7fc640 1 -- 172.21.15.57:0/2196307720 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f49c00b6040 con 0x7f49c815ac70 2023-10-30T14:39:51.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.076+0000 7f49be7fc640 1 --2- 172.21.15.57:0/2196307720 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f49ac0664a0 0x7f49ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:51.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.076+0000 7f49be7fc640 1 -- 172.21.15.57:0/2196307720 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7f49c0121c60 con 0x7f49c815ac70 2023-10-30T14:39:51.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.076+0000 7f49cd961640 1 --2- 172.21.15.57:0/2196307720 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f49ac0664a0 0x7f49ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:51.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.080+0000 7f49cd961640 1 --2- 172.21.15.57:0/2196307720 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f49ac0664a0 0x7f49ac068960 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f49c4007fd0 tx=0x7f49c4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:51.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.080+0000 7f49be7fc640 1 -- 172.21.15.57:0/2196307720 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f49c00eecf0 con 0x7f49c815ac70 2023-10-30T14:39:51.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.276+0000 7f49ce963640 1 -- 172.21.15.57:0/2196307720 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49c804e170 con 0x7f49c815ac70 2023-10-30T14:39:51.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.276+0000 7f49be7fc640 1 -- 172.21.15.57:0/2196307720 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+5739 (secure 0 0 0) 0x7f49c00f2310 con 0x7f49c815ac70 2023-10-30T14:39:51.279 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:39:51.279 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:39:45.086783+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:39:51.280 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 137 2023-10-30T14:39:51.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.276+0000 7f49ce963640 1 -- 172.21.15.57:0/2196307720 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f49ac0664a0 msgr2=0x7f49ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:51.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.276+0000 7f49ce963640 1 --2- 172.21.15.57:0/2196307720 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f49ac0664a0 0x7f49ac068960 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f49c4007fd0 tx=0x7f49c4009040 comp rx=0 tx=0).stop 2023-10-30T14:39:51.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.276+0000 7f49ce963640 1 -- 172.21.15.57:0/2196307720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49c815ac70 msgr2=0x7f49c80c4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:51.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.276+0000 7f49ce963640 1 --2- 172.21.15.57:0/2196307720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49c815ac70 0x7f49c80c4a70 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f49c001ce20 tx=0x7f49c001bf90 comp rx=0 tx=0).stop 2023-10-30T14:39:51.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.276+0000 7f49ce963640 1 -- 172.21.15.57:0/2196307720 shutdown_connections 2023-10-30T14:39:51.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.276+0000 7f49ce963640 1 --2- 172.21.15.57:0/2196307720 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f49ac0664a0 0x7f49ac068960 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:51.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.276+0000 7f49ce963640 1 --2- 172.21.15.57:0/2196307720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49c815ac70 0x7f49c80c4a70 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:51.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.276+0000 7f49ce963640 1 --2- 172.21.15.57:0/2196307720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49c80c8eb0 0x7f49c80c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:51.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.276+0000 7f49ce963640 1 --2- 172.21.15.57:0/2196307720 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49c806aeb0 0x7f49c80c76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:51.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.276+0000 7f49ce963640 1 -- 172.21.15.57:0/2196307720 >> 172.21.15.57:0/2196307720 conn(0x7f49c80cef00 msgr2=0x7f49c81585c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:51.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.276+0000 7f49ce963640 1 -- 172.21.15.57:0/2196307720 shutdown_connections 2023-10-30T14:39:51.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:51.276+0000 7f49ce963640 1 -- 172.21.15.57:0/2196307720 wait complete. 2023-10-30T14:39:51.299 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:39:53.300 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:39:53.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.455+0000 7f73b4d0f640 1 Processor -- start 2023-10-30T14:39:53.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.455+0000 7f73b4d0f640 1 -- start start 2023-10-30T14:39:53.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.455+0000 7f73b4d0f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73b0150830 0x7f73b006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:53.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.455+0000 7f73b4d0f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f73b00cb7d0 0x7f73b00c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:53.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.455+0000 7f73b4d0f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b00c2310 con 0x7f73b00c3e40 2023-10-30T14:39:53.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.455+0000 7f73b4d0f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b0156710 con 0x7f73b00cb7d0 2023-10-30T14:39:53.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.455+0000 7f73b4d0f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00c4360 con 0x7f73b0150830 2023-10-30T14:39:53.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.455+0000 7f73a7fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f73b00cb7d0 0x7f73b00c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:53.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.455+0000 7f73a7fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f73b00cb7d0 0x7f73b00c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44028/0 (socket says 172.21.15.57:44028) 2023-10-30T14:39:53.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.455+0000 7f73a7fff640 1 -- 172.21.15.57:0/938680763 learned_addr learned my addr 172.21.15.57:0/938680763 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:53.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.455+0000 7f73ae59c640 1 --2- 172.21.15.57:0/938680763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73b0150830 0x7f73b006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:53.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.455+0000 7f73a7fff640 1 -- 172.21.15.57:0/938680763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73b0150830 msgr2=0x7f73b006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:53.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.455+0000 7f73a7fff640 1 --2- 172.21.15.57:0/938680763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73b0150830 0x7f73b006b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:53.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.455+0000 7f73a7fff640 1 -- 172.21.15.57:0/938680763 >> v1:172.21.15.88:6789/0 conn(0x7f73b00c3e40 legacy=0x7f73b0070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:53.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.455+0000 7f73a7fff640 1 -- 172.21.15.57:0/938680763 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b01569b0 con 0x7f73b00cb7d0 2023-10-30T14:39:53.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.455+0000 7f73ae59c640 1 --2- 172.21.15.57:0/938680763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73b0150830 0x7f73b006b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:39:53.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.459+0000 7f73a7fff640 1 --2- 172.21.15.57:0/938680763 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73b00cb7d0 0x7f73b00c3900 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f7398009870 tx=0x7f739801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae62a22ae1b988e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:53.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.459+0000 7f73add9b640 1 -- 172.21.15.57:0/938680763 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73980a8a30 con 0x7f73b00cb7d0 2023-10-30T14:39:53.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.459+0000 7f73add9b640 1 -- 172.21.15.57:0/938680763 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73980a8b90 con 0x7f73b00cb7d0 2023-10-30T14:39:53.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.459+0000 7f73add9b640 1 -- 172.21.15.57:0/938680763 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73980b0440 con 0x7f73b00cb7d0 2023-10-30T14:39:53.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.459+0000 7f73b4d0f640 1 -- 172.21.15.57:0/938680763 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73b00cb7d0 msgr2=0x7f73b00c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:53.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.459+0000 7f73b4d0f640 1 --2- 172.21.15.57:0/938680763 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73b00cb7d0 0x7f73b00c3900 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f7398009870 tx=0x7f739801ed30 comp rx=0 tx=0).stop 2023-10-30T14:39:53.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.459+0000 7f73b4d0f640 1 -- 172.21.15.57:0/938680763 shutdown_connections 2023-10-30T14:39:53.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.459+0000 7f73b4d0f640 1 --2- 172.21.15.57:0/938680763 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73b00cb7d0 0x7f73b00c3900 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:53.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.459+0000 7f73b4d0f640 1 --2- 172.21.15.57:0/938680763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73b0150830 0x7f73b006b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:53.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.459+0000 7f73b4d0f640 1 -- 172.21.15.57:0/938680763 >> 172.21.15.57:0/938680763 conn(0x7f73b00cac70 msgr2=0x7f73b00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:53.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.459+0000 7f73b4d0f640 1 -- 172.21.15.57:0/938680763 shutdown_connections 2023-10-30T14:39:53.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.459+0000 7f73b4d0f640 1 -- 172.21.15.57:0/938680763 wait complete. 2023-10-30T14:39:53.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73b4d0f640 1 Processor -- start 2023-10-30T14:39:53.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73b4d0f640 1 -- start start 2023-10-30T14:39:53.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73b4d0f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73b00c3e40 0x7f73b01581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:53.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73b4d0f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73b00cb7d0 0x7f73b01eeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:53.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73b4d0f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73b0150830 0x7f73b00bc310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:53.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73b4d0f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b01577c0 con 0x7f73b0150830 2023-10-30T14:39:53.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73b4d0f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b0157d40 con 0x7f73b00cb7d0 2023-10-30T14:39:53.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73b4d0f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b015a320 con 0x7f73b00c3e40 2023-10-30T14:39:53.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73aed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73b0150830 0x7f73b00bc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:53.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73aed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73b0150830 0x7f73b00bc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44034/0 (socket says 172.21.15.57:44034) 2023-10-30T14:39:53.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73aed9d640 1 -- 172.21.15.57:0/1688824345 learned_addr learned my addr 172.21.15.57:0/1688824345 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:53.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73a7fff640 1 --2- 172.21.15.57:0/1688824345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73b00cb7d0 0x7f73b01eeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:53.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73ae59c640 1 --2- 172.21.15.57:0/1688824345 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73b00c3e40 0x7f73b01581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:53.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73aed9d640 1 -- 172.21.15.57:0/1688824345 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73b00c3e40 msgr2=0x7f73b01581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:53.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73aed9d640 1 --2- 172.21.15.57:0/1688824345 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73b00c3e40 0x7f73b01581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:53.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73aed9d640 1 -- 172.21.15.57:0/1688824345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73b00cb7d0 msgr2=0x7f73b01eeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:53.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73aed9d640 1 --2- 172.21.15.57:0/1688824345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73b00cb7d0 0x7f73b01eeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:53.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73aed9d640 1 -- 172.21.15.57:0/1688824345 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73980094d0 con 0x7f73b0150830 2023-10-30T14:39:53.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73aed9d640 1 --2- 172.21.15.57:0/1688824345 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73b0150830 0x7f73b00bc310 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f73a800c7d0 tx=0x7f73a800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:53.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73a6ffd640 1 -- 172.21.15.57:0/1688824345 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a8003b80 con 0x7f73b0150830 2023-10-30T14:39:53.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73a6ffd640 1 -- 172.21.15.57:0/1688824345 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a8003ce0 con 0x7f73b0150830 2023-10-30T14:39:53.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73b4d0f640 1 -- 172.21.15.57:0/1688824345 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b015a600 con 0x7f73b0150830 2023-10-30T14:39:53.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73a6ffd640 1 -- 172.21.15.57:0/1688824345 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a800aa30 con 0x7f73b0150830 2023-10-30T14:39:53.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.463+0000 7f73b4d0f640 1 -- 172.21.15.57:0/1688824345 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b01579d0 con 0x7f73b0150830 2023-10-30T14:39:53.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.467+0000 7f73b4d0f640 1 -- 172.21.15.57:0/1688824345 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b0157ef0 con 0x7f73b0150830 2023-10-30T14:39:53.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.467+0000 7f73a6ffd640 1 -- 172.21.15.57:0/1688824345 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f73a800ab90 con 0x7f73b0150830 2023-10-30T14:39:53.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.467+0000 7f73a6ffd640 1 --2- 172.21.15.57:0/1688824345 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f73900664a0 0x7f7390068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:53.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.467+0000 7f73a6ffd640 1 -- 172.21.15.57:0/1688824345 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7f73a8084140 con 0x7f73b0150830 2023-10-30T14:39:53.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.467+0000 7f73ae59c640 1 --2- 172.21.15.57:0/1688824345 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f73900664a0 0x7f7390068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:53.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.467+0000 7f73ae59c640 1 --2- 172.21.15.57:0/1688824345 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f73900664a0 0x7f7390068960 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f73a00045f0 tx=0x7f73a0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:53.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.471+0000 7f73a6ffd640 1 -- 172.21.15.57:0/1688824345 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f73b0157ef0 con 0x7f73b0150830 2023-10-30T14:39:53.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.659+0000 7f73b4d0f640 1 -- 172.21.15.57:0/1688824345 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73b0157ef0 con 0x7f73b0150830 2023-10-30T14:39:53.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.659+0000 7f73a6ffd640 1 -- 172.21.15.57:0/1688824345 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+5740 (secure 0 0 0) 0x7f73b0157ef0 con 0x7f73b0150830 2023-10-30T14:39:53.664 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:39:53.664 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:39:52.104561+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:resolve","state_seq":18,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:39:53.665 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 139 2023-10-30T14:39:53.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.659+0000 7f73b4d0f640 1 -- 172.21.15.57:0/1688824345 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f73900664a0 msgr2=0x7f7390068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:53.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.659+0000 7f73b4d0f640 1 --2- 172.21.15.57:0/1688824345 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f73900664a0 0x7f7390068960 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f73a00045f0 tx=0x7f73a0008040 comp rx=0 tx=0).stop 2023-10-30T14:39:53.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.659+0000 7f73b4d0f640 1 -- 172.21.15.57:0/1688824345 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73b0150830 msgr2=0x7f73b00bc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:53.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.659+0000 7f73b4d0f640 1 --2- 172.21.15.57:0/1688824345 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73b0150830 0x7f73b00bc310 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f73a800c7d0 tx=0x7f73a800cc90 comp rx=0 tx=0).stop 2023-10-30T14:39:53.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.659+0000 7f73b4d0f640 1 -- 172.21.15.57:0/1688824345 shutdown_connections 2023-10-30T14:39:53.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.659+0000 7f73b4d0f640 1 --2- 172.21.15.57:0/1688824345 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f73900664a0 0x7f7390068960 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:53.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.659+0000 7f73b4d0f640 1 --2- 172.21.15.57:0/1688824345 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73b0150830 0x7f73b00bc310 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:53.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.659+0000 7f73b4d0f640 1 --2- 172.21.15.57:0/1688824345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73b00cb7d0 0x7f73b01eeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:53.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.659+0000 7f73b4d0f640 1 --2- 172.21.15.57:0/1688824345 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73b00c3e40 0x7f73b01581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:53.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.659+0000 7f73b4d0f640 1 -- 172.21.15.57:0/1688824345 >> 172.21.15.57:0/1688824345 conn(0x7f73b00cac70 msgr2=0x7f73b014ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:53.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.659+0000 7f73b4d0f640 1 -- 172.21.15.57:0/1688824345 shutdown_connections 2023-10-30T14:39:53.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:53.659+0000 7f73b4d0f640 1 -- 172.21.15.57:0/1688824345 wait complete. 2023-10-30T14:39:53.684 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:39:55.686 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:39:55.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.839+0000 7fa5f209c640 1 Processor -- start 2023-10-30T14:39:55.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.839+0000 7fa5f209c640 1 -- start start 2023-10-30T14:39:55.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.839+0000 7fa5f209c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa5ec15cf90 0x7fa5ec0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:55.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.839+0000 7fa5f209c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec159f20 con 0x7fa5ec153c80 2023-10-30T14:39:55.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.839+0000 7fa5f209c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec0c47e0 con 0x7fa5ec0c4a10 2023-10-30T14:39:55.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.839+0000 7fa5f209c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5ec15dec0 con 0x7fa5ec15cf90 2023-10-30T14:39:55.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.839+0000 7fa5eb7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa5ec153c80 0x7fa5ec15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44782/0 (socket says 172.21.15.57:44782) 2023-10-30T14:39:55.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.839+0000 7fa5eb7fe640 1 -- 172.21.15.57:0/4217842229 learned_addr learned my addr 172.21.15.57:0/4217842229 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:39:55.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.839+0000 7fa5ebfff640 1 --2- 172.21.15.57:0/4217842229 >> v2:172.21.15.57:3300/0 conn(0x7fa5ec15cf90 0x7fa5ec0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:55.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.839+0000 7fa5ebfff640 1 -- 172.21.15.57:0/4217842229 >> v1:172.21.15.88:6789/0 conn(0x7fa5ec0c4a10 legacy=0x7fa5ec0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:55.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.839+0000 7fa5ebfff640 1 -- 172.21.15.57:0/4217842229 >> v1:172.21.15.57:6789/0 conn(0x7fa5ec153c80 legacy=0x7fa5ec15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:55.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.839+0000 7fa5ea7fc640 1 -- 172.21.15.57:0/4217842229 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424484556 0 0) 0x7fa5ec159f20 con 0x7fa5ec153c80 2023-10-30T14:39:55.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.839+0000 7fa5ebfff640 1 -- 172.21.15.57:0/4217842229 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec15d620 con 0x7fa5ec15cf90 2023-10-30T14:39:55.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.843+0000 7fa5ebfff640 1 --2- 172.21.15.57:0/4217842229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5ec15cf90 0x7fa5ec0c42a0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009d90 tx=0x7fa5e401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2956881f3ae347c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:55.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.843+0000 7fa5ea7fc640 1 -- 172.21.15.57:0/4217842229 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e40af600 con 0x7fa5ec15cf90 2023-10-30T14:39:55.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.843+0000 7fa5ea7fc640 1 -- 172.21.15.57:0/4217842229 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e40ab050 con 0x7fa5ec15cf90 2023-10-30T14:39:55.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.843+0000 7fa5ea7fc640 1 -- 172.21.15.57:0/4217842229 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e40af980 con 0x7fa5ec15cf90 2023-10-30T14:39:55.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.843+0000 7fa5f209c640 1 -- 172.21.15.57:0/4217842229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5ec15cf90 msgr2=0x7fa5ec0c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:55.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.843+0000 7fa5f209c640 1 --2- 172.21.15.57:0/4217842229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5ec15cf90 0x7fa5ec0c42a0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009d90 tx=0x7fa5e401f130 comp rx=0 tx=0).stop 2023-10-30T14:39:55.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.843+0000 7fa5f209c640 1 -- 172.21.15.57:0/4217842229 shutdown_connections 2023-10-30T14:39:55.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.843+0000 7fa5f209c640 1 --2- 172.21.15.57:0/4217842229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5ec15cf90 0x7fa5ec0c42a0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:55.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.843+0000 7fa5f209c640 1 -- 172.21.15.57:0/4217842229 >> 172.21.15.57:0/4217842229 conn(0x7fa5ec0715c0 msgr2=0x7fa5ec0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:55.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.843+0000 7fa5f209c640 1 -- 172.21.15.57:0/4217842229 shutdown_connections 2023-10-30T14:39:55.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.843+0000 7fa5f209c640 1 -- 172.21.15.57:0/4217842229 wait complete. 2023-10-30T14:39:55.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5f209c640 1 Processor -- start 2023-10-30T14:39:55.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5f209c640 1 -- start start 2023-10-30T14:39:55.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5f209c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5ec0c4a10 0x7fa5ec0cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:55.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5f209c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5ec153c80 0x7fa5ec0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:55.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5f209c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5ec15cf90 0x7fa5ec1f2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:55.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5f209c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec151900 con 0x7fa5ec15cf90 2023-10-30T14:39:55.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5f209c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec156f60 con 0x7fa5ec153c80 2023-10-30T14:39:55.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5f209c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec157220 con 0x7fa5ec0c4a10 2023-10-30T14:39:55.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5ebfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5ec15cf90 0x7fa5ec1f2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:55.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5ebfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5ec15cf90 0x7fa5ec1f2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54850/0 (socket says 172.21.15.57:54850) 2023-10-30T14:39:55.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5ebfff640 1 -- 172.21.15.57:0/4080544228 learned_addr learned my addr 172.21.15.57:0/4080544228 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:55.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5eb7fe640 1 --2- 172.21.15.57:0/4080544228 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5ec0c4a10 0x7fa5ec0cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:55.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5eaffd640 1 --2- 172.21.15.57:0/4080544228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5ec153c80 0x7fa5ec0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:55.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5ebfff640 1 -- 172.21.15.57:0/4080544228 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5ec0c4a10 msgr2=0x7fa5ec0cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:55.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5ebfff640 1 --2- 172.21.15.57:0/4080544228 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5ec0c4a10 0x7fa5ec0cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:55.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5ebfff640 1 -- 172.21.15.57:0/4080544228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5ec153c80 msgr2=0x7fa5ec0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:55.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5ebfff640 1 --2- 172.21.15.57:0/4080544228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5ec153c80 0x7fa5ec0cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:55.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5ebfff640 1 -- 172.21.15.57:0/4080544228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e4009a70 con 0x7fa5ec15cf90 2023-10-30T14:39:55.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5eb7fe640 1 --2- 172.21.15.57:0/4080544228 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5ec0c4a10 0x7fa5ec0cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:39:55.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5ebfff640 1 --2- 172.21.15.57:0/4080544228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5ec15cf90 0x7fa5ec1f2260 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fa5e40b2040 tx=0x7fa5e401bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:55.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5d3fff640 1 -- 172.21.15.57:0/4080544228 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e40ba680 con 0x7fa5ec15cf90 2023-10-30T14:39:55.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5d3fff640 1 -- 172.21.15.57:0/4080544228 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e40ba7e0 con 0x7fa5ec15cf90 2023-10-30T14:39:55.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5d3fff640 1 -- 172.21.15.57:0/4080544228 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e40bab90 con 0x7fa5ec15cf90 2023-10-30T14:39:55.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5f209c640 1 -- 172.21.15.57:0/4080544228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ec1548e0 con 0x7fa5ec15cf90 2023-10-30T14:39:55.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.847+0000 7fa5f209c640 1 -- 172.21.15.57:0/4080544228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ec154ea0 con 0x7fa5ec15cf90 2023-10-30T14:39:55.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.851+0000 7fa5f209c640 1 -- 172.21.15.57:0/4080544228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5ec1553c0 con 0x7fa5ec15cf90 2023-10-30T14:39:55.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.851+0000 7fa5d3fff640 1 -- 172.21.15.57:0/4080544228 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa5e40b6040 con 0x7fa5ec15cf90 2023-10-30T14:39:55.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.851+0000 7fa5d3fff640 1 --2- 172.21.15.57:0/4080544228 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa5d80664a0 0x7fa5d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:55.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.851+0000 7fa5d3fff640 1 -- 172.21.15.57:0/4080544228 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7fa5e4122f00 con 0x7fa5ec15cf90 2023-10-30T14:39:55.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.851+0000 7fa5eb7fe640 1 --2- 172.21.15.57:0/4080544228 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa5d80664a0 0x7fa5d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:55.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.851+0000 7fa5eb7fe640 1 --2- 172.21.15.57:0/4080544228 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa5d80664a0 0x7fa5d8068960 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc000f70 tx=0x7fa5dc007c60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:55.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:55.855+0000 7fa5d3fff640 1 -- 172.21.15.57:0/4080544228 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa5ec1553c0 con 0x7fa5ec15cf90 2023-10-30T14:39:56.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:56.043+0000 7fa5f209c640 1 -- 172.21.15.57:0/4080544228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5ec1553c0 con 0x7fa5ec15cf90 2023-10-30T14:39:56.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:56.043+0000 7fa5d3fff640 1 -- 172.21.15.57:0/4080544228 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+5738 (secure 0 0 0) 0x7fa5ec1553c0 con 0x7fa5ec15cf90 2023-10-30T14:39:56.047 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:39:56.048 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:39:54.158767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:resolve","state_seq":18,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:39:56.048 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 140 2023-10-30T14:39:56.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:56.043+0000 7fa5f209c640 1 -- 172.21.15.57:0/4080544228 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa5d80664a0 msgr2=0x7fa5d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:56.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:56.043+0000 7fa5f209c640 1 --2- 172.21.15.57:0/4080544228 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa5d80664a0 0x7fa5d8068960 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc000f70 tx=0x7fa5dc007c60 comp rx=0 tx=0).stop 2023-10-30T14:39:56.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:56.043+0000 7fa5f209c640 1 -- 172.21.15.57:0/4080544228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5ec15cf90 msgr2=0x7fa5ec1f2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:56.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:56.043+0000 7fa5f209c640 1 --2- 172.21.15.57:0/4080544228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5ec15cf90 0x7fa5ec1f2260 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fa5e40b2040 tx=0x7fa5e401bed0 comp rx=0 tx=0).stop 2023-10-30T14:39:56.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:56.047+0000 7fa5f209c640 1 -- 172.21.15.57:0/4080544228 shutdown_connections 2023-10-30T14:39:56.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:56.047+0000 7fa5f209c640 1 --2- 172.21.15.57:0/4080544228 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa5d80664a0 0x7fa5d8068960 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:56.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:56.047+0000 7fa5f209c640 1 --2- 172.21.15.57:0/4080544228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5ec15cf90 0x7fa5ec1f2260 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:56.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:56.047+0000 7fa5f209c640 1 --2- 172.21.15.57:0/4080544228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5ec153c80 0x7fa5ec0cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:56.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:56.047+0000 7fa5f209c640 1 --2- 172.21.15.57:0/4080544228 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5ec0c4a10 0x7fa5ec0cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:56.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:56.047+0000 7fa5f209c640 1 -- 172.21.15.57:0/4080544228 >> 172.21.15.57:0/4080544228 conn(0x7fa5ec0715c0 msgr2=0x7fa5ec1520f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:56.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:56.047+0000 7fa5f209c640 1 -- 172.21.15.57:0/4080544228 shutdown_connections 2023-10-30T14:39:56.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:56.047+0000 7fa5f209c640 1 -- 172.21.15.57:0/4080544228 wait complete. 2023-10-30T14:39:56.068 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:39:58.070 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:39:58.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.219+0000 7fdb0fe17640 1 Processor -- start 2023-10-30T14:39:58.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.219+0000 7fdb0fe17640 1 -- start start 2023-10-30T14:39:58.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0fe17640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb080c5ba0 0x7fdb080c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:58.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0fe17640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0815cda0 con 0x7fdb080c5f80 2023-10-30T14:39:58.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0fe17640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0806f490 con 0x7fdb080c9b40 2023-10-30T14:39:58.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0fe17640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0815d990 con 0x7fdb080c5ba0 2023-10-30T14:39:58.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0d3b2640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fdb080c5f80 0x7fdb08069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44784/0 (socket says 172.21.15.57:44784) 2023-10-30T14:39:58.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0d3b2640 1 -- 172.21.15.57:0/4190543832 learned_addr learned my addr 172.21.15.57:0/4190543832 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:39:58.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0dbb3640 1 --2- 172.21.15.57:0/4190543832 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb080c5ba0 0x7fdb080c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:58.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0cbb1640 1 -- 172.21.15.57:0/4190543832 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264195156 0 0) 0x7fdb0815cda0 con 0x7fdb080c5f80 2023-10-30T14:39:58.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0cbb1640 1 -- 172.21.15.57:0/4190543832 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaf4003530 con 0x7fdb080c5f80 2023-10-30T14:39:58.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0cbb1640 1 -- 172.21.15.57:0/4190543832 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 701068271 0 0) 0x7fdaf4003530 con 0x7fdb080c5f80 2023-10-30T14:39:58.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0cbb1640 1 -- 172.21.15.57:0/4190543832 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb080c5ba0 msgr2=0x7fdb080c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:58.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0cbb1640 1 --2- 172.21.15.57:0/4190543832 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb080c5ba0 0x7fdb080c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:58.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0cbb1640 1 -- 172.21.15.57:0/4190543832 >> v1:172.21.15.88:6789/0 conn(0x7fdb080c9b40 legacy=0x7fdb081597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:58.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0cbb1640 1 -- 172.21.15.57:0/4190543832 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0815cff0 con 0x7fdb080c5f80 2023-10-30T14:39:58.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0cbb1640 1 -- 172.21.15.57:0/4190543832 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fdaf8001680 con 0x7fdb080c5f80 2023-10-30T14:39:58.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0cbb1640 1 -- 172.21.15.57:0/4190543832 >> v1:172.21.15.57:6789/0 conn(0x7fdb080c5f80 legacy=0x7fdb08069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:58.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0cbb1640 1 --2- 172.21.15.57:0/4190543832 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb080c5ba0 0x7fdaf40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:58.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0cbb1640 1 --2- 172.21.15.57:0/4190543832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdaf4003ef0 0x7fdaf40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:58.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0cbb1640 1 --2- 172.21.15.57:0/4190543832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdaf4004d10 0x7fdaf4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:58.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0cbb1640 1 -- 172.21.15.57:0/4190543832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0815d990 con 0x7fdaf4003ef0 2023-10-30T14:39:58.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0cbb1640 1 -- 172.21.15.57:0/4190543832 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf400b9f0 con 0x7fdaf4004d10 2023-10-30T14:39:58.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0cbb1640 1 -- 172.21.15.57:0/4190543832 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf400bbb0 con 0x7fdb080c5ba0 2023-10-30T14:39:58.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0d3b2640 1 --2- 172.21.15.57:0/4190543832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdaf4003ef0 0x7fdaf40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:58.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0dbb3640 1 --2- 172.21.15.57:0/4190543832 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb080c5ba0 0x7fdaf40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:58.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0e3b4640 1 --2- 172.21.15.57:0/4190543832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdaf4004d10 0x7fdaf4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:58.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0d3b2640 1 -- 172.21.15.57:0/4190543832 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb080c5ba0 msgr2=0x7fdaf40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:58.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0d3b2640 1 --2- 172.21.15.57:0/4190543832 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb080c5ba0 0x7fdaf40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:58.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0d3b2640 1 -- 172.21.15.57:0/4190543832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdaf4004d10 msgr2=0x7fdaf4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:39:58.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0d3b2640 1 --2- 172.21.15.57:0/4190543832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdaf4004d10 0x7fdaf4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:58.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.223+0000 7fdb0d3b2640 1 -- 172.21.15.57:0/4190543832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf4003530 con 0x7fdaf4003ef0 2023-10-30T14:39:58.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.227+0000 7fdb0d3b2640 1 --2- 172.21.15.57:0/4190543832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdaf4003ef0 0x7fdaf40047d0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fdaf800c190 tx=0x7fdaf801cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d060cafbc58a4f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:58.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.227+0000 7fdb0cbb1640 1 -- 172.21.15.57:0/4190543832 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8005ae0 con 0x7fdaf4003ef0 2023-10-30T14:39:58.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.227+0000 7fdb0cbb1640 1 -- 172.21.15.57:0/4190543832 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf8005c40 con 0x7fdaf4003ef0 2023-10-30T14:39:58.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.227+0000 7fdb0cbb1640 1 -- 172.21.15.57:0/4190543832 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8007070 con 0x7fdaf4003ef0 2023-10-30T14:39:58.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.227+0000 7fdb0fe17640 1 -- 172.21.15.57:0/4190543832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdaf4003ef0 msgr2=0x7fdaf40047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:58.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.227+0000 7fdb0fe17640 1 --2- 172.21.15.57:0/4190543832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdaf4003ef0 0x7fdaf40047d0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fdaf800c190 tx=0x7fdaf801cb40 comp rx=0 tx=0).stop 2023-10-30T14:39:58.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.227+0000 7fdb0e3b4640 1 -- 172.21.15.57:0/4190543832 reap_dead start 2023-10-30T14:39:58.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.227+0000 7fdb0fe17640 1 -- 172.21.15.57:0/4190543832 shutdown_connections 2023-10-30T14:39:58.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.227+0000 7fdb0fe17640 1 -- 172.21.15.57:0/4190543832 >> 172.21.15.57:0/4190543832 conn(0x7fdb0806afa0 msgr2=0x7fdb0806b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:58.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.227+0000 7fdb0fe17640 1 -- 172.21.15.57:0/4190543832 shutdown_connections 2023-10-30T14:39:58.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.227+0000 7fdb0fe17640 1 -- 172.21.15.57:0/4190543832 wait complete. 2023-10-30T14:39:58.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0fe17640 1 Processor -- start 2023-10-30T14:39:58.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0fe17640 1 -- start start 2023-10-30T14:39:58.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0fe17640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdaf4003ef0 0x7fdb080c4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:58.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0fe17640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb08158920 0x7fdb081f5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:58.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0fe17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb080bf850 0x7fdb08151db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:58.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0fe17640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb081570d0 con 0x7fdb08158920 2023-10-30T14:39:58.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0fe17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0806a050 con 0x7fdb080bf850 2023-10-30T14:39:58.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0fe17640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0815a3a0 con 0x7fdaf4003ef0 2023-10-30T14:39:58.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0d3b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb08158920 0x7fdb081f5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:58.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0d3b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb08158920 0x7fdb081f5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54872/0 (socket says 172.21.15.57:54872) 2023-10-30T14:39:58.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0d3b2640 1 -- 172.21.15.57:0/2279622487 learned_addr learned my addr 172.21.15.57:0/2279622487 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:39:58.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0dbb3640 1 --2- 172.21.15.57:0/2279622487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdaf4003ef0 0x7fdb080c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:58.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0e3b4640 1 --2- 172.21.15.57:0/2279622487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb080bf850 0x7fdb08151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:58.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0d3b2640 1 -- 172.21.15.57:0/2279622487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdaf4003ef0 msgr2=0x7fdb080c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:58.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0d3b2640 1 --2- 172.21.15.57:0/2279622487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdaf4003ef0 0x7fdb080c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:58.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0d3b2640 1 -- 172.21.15.57:0/2279622487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb080bf850 msgr2=0x7fdb08151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:58.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0d3b2640 1 --2- 172.21.15.57:0/2279622487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb080bf850 0x7fdb08151db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:58.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0d3b2640 1 -- 172.21.15.57:0/2279622487 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb08156b10 con 0x7fdb08158920 2023-10-30T14:39:58.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0e3b4640 1 --2- 172.21.15.57:0/2279622487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb080bf850 0x7fdb08151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:39:58.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0dbb3640 1 --2- 172.21.15.57:0/2279622487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdaf4003ef0 0x7fdb080c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:39:58.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0d3b2640 1 --2- 172.21.15.57:0/2279622487 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb08158920 0x7fdb081f5180 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fdaf800c190 tx=0x7fdaf4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:58.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdafe7fc640 1 -- 172.21.15.57:0/2279622487 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf80b4070 con 0x7fdb08158920 2023-10-30T14:39:58.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdafe7fc640 1 -- 172.21.15.57:0/2279622487 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf8001620 con 0x7fdb08158920 2023-10-30T14:39:58.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdafe7fc640 1 -- 172.21.15.57:0/2279622487 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8007270 con 0x7fdb08158920 2023-10-30T14:39:58.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0fe17640 1 -- 172.21.15.57:0/2279622487 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb08156df0 con 0x7fdb08158920 2023-10-30T14:39:58.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0fe17640 1 -- 172.21.15.57:0/2279622487 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0815a030 con 0x7fdb08158920 2023-10-30T14:39:58.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.231+0000 7fdb0fe17640 1 -- 172.21.15.57:0/2279622487 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb08159bb0 con 0x7fdb08158920 2023-10-30T14:39:58.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.235+0000 7fdafe7fc640 1 -- 172.21.15.57:0/2279622487 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdaf80073d0 con 0x7fdb08158920 2023-10-30T14:39:58.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.235+0000 7fdafe7fc640 1 --2- 172.21.15.57:0/2279622487 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdaec0664d0 0x7fdaec068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:39:58.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.235+0000 7fdafe7fc640 1 -- 172.21.15.57:0/2279622487 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7fdaf811dd60 con 0x7fdb08158920 2023-10-30T14:39:58.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.235+0000 7fdb0dbb3640 1 --2- 172.21.15.57:0/2279622487 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdaec0664d0 0x7fdaec068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:39:58.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.235+0000 7fdb0dbb3640 1 --2- 172.21.15.57:0/2279622487 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdaec0664d0 0x7fdaec068990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fdb04009be0 tx=0x7fdb04008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:39:58.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.239+0000 7fdafe7fc640 1 -- 172.21.15.57:0/2279622487 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdb08159bb0 con 0x7fdb08158920 2023-10-30T14:39:58.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.431+0000 7fdb0fe17640 1 -- 172.21.15.57:0/2279622487 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb08159bb0 con 0x7fdb08158920 2023-10-30T14:39:58.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.431+0000 7fdafe7fc640 1 -- 172.21.15.57:0/2279622487 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+5740 (secure 0 0 0) 0x7fdb08159bb0 con 0x7fdb08158920 2023-10-30T14:39:58.436 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:39:58.436 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":141,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:39:57.112308+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:39:58.436 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 141 2023-10-30T14:39:58.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.435+0000 7fdb0fe17640 1 -- 172.21.15.57:0/2279622487 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdaec0664d0 msgr2=0x7fdaec068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:58.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.435+0000 7fdb0fe17640 1 --2- 172.21.15.57:0/2279622487 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdaec0664d0 0x7fdaec068990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fdb04009be0 tx=0x7fdb04008040 comp rx=0 tx=0).stop 2023-10-30T14:39:58.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.435+0000 7fdb0fe17640 1 -- 172.21.15.57:0/2279622487 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb08158920 msgr2=0x7fdb081f5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:39:58.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.435+0000 7fdb0fe17640 1 --2- 172.21.15.57:0/2279622487 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb08158920 0x7fdb081f5180 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fdaf800c190 tx=0x7fdaf4005800 comp rx=0 tx=0).stop 2023-10-30T14:39:58.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.435+0000 7fdb0fe17640 1 -- 172.21.15.57:0/2279622487 shutdown_connections 2023-10-30T14:39:58.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.435+0000 7fdb0fe17640 1 --2- 172.21.15.57:0/2279622487 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdaec0664d0 0x7fdaec068990 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:58.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.435+0000 7fdb0fe17640 1 --2- 172.21.15.57:0/2279622487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb080bf850 0x7fdb08151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:58.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.435+0000 7fdb0fe17640 1 --2- 172.21.15.57:0/2279622487 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb08158920 0x7fdb081f5180 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:58.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.435+0000 7fdb0fe17640 1 --2- 172.21.15.57:0/2279622487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdaf4003ef0 0x7fdb080c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:39:58.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.435+0000 7fdb0fe17640 1 -- 172.21.15.57:0/2279622487 >> 172.21.15.57:0/2279622487 conn(0x7fdb0806afa0 msgr2=0x7fdb080c3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:39:58.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.435+0000 7fdb0fe17640 1 -- 172.21.15.57:0/2279622487 shutdown_connections 2023-10-30T14:39:58.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:39:58.435+0000 7fdb0fe17640 1 -- 172.21.15.57:0/2279622487 wait complete. 2023-10-30T14:39:58.457 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 141, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 141, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6414": { "addr": "172.21.15.57:6834/2321408149", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 2321408149, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 2321408149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6414, "incarnation": 99, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 18 }, "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 44 }, "gid_7056": { "addr": "172.21.15.57:6837/2307041430", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 2307041430, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 2307041430, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7056, "incarnation": 137, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:reconnect", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:39:57.112308+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6414, "mds_1": 6747, "mds_2": 7056 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:40:00.458 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:40:00.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.607+0000 7f3ca9617640 1 Processor -- start 2023-10-30T14:40:00.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.607+0000 7f3ca9617640 1 -- start start 2023-10-30T14:40:00.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca9617640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3ca40701a0 0x7f3ca415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:00.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca9617640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ca406b2f0 0x7f3ca406b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:00.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca9617640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca40c2aa0 con 0x7f3ca41537f0 2023-10-30T14:40:00.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca9617640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ca40c3ab0 con 0x7f3ca40701a0 2023-10-30T14:40:00.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca9617640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca415a820 con 0x7f3ca406b2f0 2023-10-30T14:40:00.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca2ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f3ca41537f0 0x7f3ca40ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44792/0 (socket says 172.21.15.57:44792) 2023-10-30T14:40:00.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca2ffd640 1 -- 172.21.15.57:0/1719288203 learned_addr learned my addr 172.21.15.57:0/1719288203 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:40:00.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca37fe640 1 --2- 172.21.15.57:0/1719288203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ca406b2f0 0x7f3ca406b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:00.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3c9bfff640 1 --2- 172.21.15.57:0/1719288203 >> v2:172.21.15.88:3300/0 conn(0x7f3ca40701a0 0x7f3ca415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:00.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca27fc640 1 -- 172.21.15.57:0/1719288203 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566512071 0 0) 0x7f3ca40c2aa0 con 0x7f3ca41537f0 2023-10-30T14:40:00.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca27fc640 1 -- 172.21.15.57:0/1719288203 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c90003530 con 0x7f3ca41537f0 2023-10-30T14:40:00.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca27fc640 1 -- 172.21.15.57:0/1719288203 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3582907326 0 0) 0x7f3c90003530 con 0x7f3ca41537f0 2023-10-30T14:40:00.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca27fc640 1 -- 172.21.15.57:0/1719288203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ca406b2f0 msgr2=0x7f3ca406b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:40:00.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca27fc640 1 --2- 172.21.15.57:0/1719288203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ca406b2f0 0x7f3ca406b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:00.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca27fc640 1 -- 172.21.15.57:0/1719288203 >> v2:172.21.15.88:3300/0 conn(0x7f3ca40701a0 msgr2=0x7f3ca415a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:40:00.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca27fc640 1 --2- 172.21.15.57:0/1719288203 >> v2:172.21.15.88:3300/0 conn(0x7f3ca40701a0 0x7f3ca415a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:00.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca27fc640 1 -- 172.21.15.57:0/1719288203 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca406bc80 con 0x7f3ca41537f0 2023-10-30T14:40:00.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca27fc640 1 -- 172.21.15.57:0/1719288203 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f3c94001680 con 0x7f3ca41537f0 2023-10-30T14:40:00.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca27fc640 1 -- 172.21.15.57:0/1719288203 >> v1:172.21.15.57:6789/0 conn(0x7f3ca41537f0 legacy=0x7f3ca40ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:00.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca27fc640 1 --2- 172.21.15.57:0/1719288203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c90003980 0x7f3c90003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:00.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca27fc640 1 --2- 172.21.15.57:0/1719288203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c900047d0 0x7f3c90004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:00.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca27fc640 1 --2- 172.21.15.57:0/1719288203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ca406b2f0 0x7f3c90005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:00.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca27fc640 1 -- 172.21.15.57:0/1719288203 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca415a820 con 0x7f3c90003980 2023-10-30T14:40:00.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca27fc640 1 -- 172.21.15.57:0/1719288203 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9000b990 con 0x7f3c900047d0 2023-10-30T14:40:00.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca27fc640 1 -- 172.21.15.57:0/1719288203 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9000bad0 con 0x7f3ca406b2f0 2023-10-30T14:40:00.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca2ffd640 1 --2- 172.21.15.57:0/1719288203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c90003980 0x7f3c90003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:00.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3c9bfff640 1 --2- 172.21.15.57:0/1719288203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c900047d0 0x7f3c90004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:00.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca37fe640 1 --2- 172.21.15.57:0/1719288203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ca406b2f0 0x7f3c90005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:00.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca2ffd640 1 -- 172.21.15.57:0/1719288203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ca406b2f0 msgr2=0x7f3c90005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:40:00.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca2ffd640 1 --2- 172.21.15.57:0/1719288203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ca406b2f0 0x7f3c90005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:00.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca2ffd640 1 -- 172.21.15.57:0/1719288203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c900047d0 msgr2=0x7f3c90004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:40:00.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca2ffd640 1 --2- 172.21.15.57:0/1719288203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c900047d0 0x7f3c90004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:00.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.611+0000 7f3ca2ffd640 1 -- 172.21.15.57:0/1719288203 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c90003530 con 0x7f3c90003980 2023-10-30T14:40:00.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.615+0000 7f3ca2ffd640 1 --2- 172.21.15.57:0/1719288203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c90003980 0x7f3c90003da0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f3c9400c210 tx=0x7f3c9401f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba27c0a62730199 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:00.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.615+0000 7f3ca27fc640 1 -- 172.21.15.57:0/1719288203 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c940a8ce0 con 0x7f3c90003980 2023-10-30T14:40:00.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.615+0000 7f3ca27fc640 1 -- 172.21.15.57:0/1719288203 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c940a8e40 con 0x7f3c90003980 2023-10-30T14:40:00.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.615+0000 7f3ca27fc640 1 -- 172.21.15.57:0/1719288203 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c94005b00 con 0x7f3c90003980 2023-10-30T14:40:00.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.615+0000 7f3ca9617640 1 -- 172.21.15.57:0/1719288203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c90003980 msgr2=0x7f3c90003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:00.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.615+0000 7f3ca9617640 1 --2- 172.21.15.57:0/1719288203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c90003980 0x7f3c90003da0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f3c9400c210 tx=0x7f3c9401f3b0 comp rx=0 tx=0).stop 2023-10-30T14:40:00.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.615+0000 7f3ca37fe640 1 -- 172.21.15.57:0/1719288203 reap_dead start 2023-10-30T14:40:00.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.615+0000 7f3ca9617640 1 -- 172.21.15.57:0/1719288203 shutdown_connections 2023-10-30T14:40:00.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.615+0000 7f3ca9617640 1 -- 172.21.15.57:0/1719288203 >> 172.21.15.57:0/1719288203 conn(0x7f3ca4071fb0 msgr2=0x7f3ca40723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:40:00.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.615+0000 7f3ca9617640 1 -- 172.21.15.57:0/1719288203 shutdown_connections 2023-10-30T14:40:00.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.615+0000 7f3ca9617640 1 -- 172.21.15.57:0/1719288203 wait complete. 2023-10-30T14:40:00.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.615+0000 7f3ca9617640 1 Processor -- start 2023-10-30T14:40:00.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.615+0000 7f3ca9617640 1 -- start start 2023-10-30T14:40:00.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3ca9617640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c90003980 0x7f3ca41505a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:00.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3ca9617640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ca40c8780 0x7f3ca40c9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:00.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3ca9617640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ca4145130 0x7f3ca414abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:00.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3ca9617640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca4151090 con 0x7f3c90003980 2023-10-30T14:40:00.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3ca9617640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca4150110 con 0x7f3ca4145130 2023-10-30T14:40:00.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3ca9617640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca41f2ef0 con 0x7f3ca40c8780 2023-10-30T14:40:00.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3ca2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c90003980 0x7f3ca41505a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:00.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3ca2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c90003980 0x7f3ca41505a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54890/0 (socket says 172.21.15.57:54890) 2023-10-30T14:40:00.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3ca2ffd640 1 -- 172.21.15.57:0/2282182530 learned_addr learned my addr 172.21.15.57:0/2282182530 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:40:00.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3c9bfff640 1 --2- 172.21.15.57:0/2282182530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ca40c8780 0x7f3ca40c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:00.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3ca37fe640 1 --2- 172.21.15.57:0/2282182530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ca4145130 0x7f3ca414abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:00.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3ca2ffd640 1 -- 172.21.15.57:0/2282182530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ca40c8780 msgr2=0x7f3ca40c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:00.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3ca2ffd640 1 --2- 172.21.15.57:0/2282182530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ca40c8780 0x7f3ca40c9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:00.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3ca2ffd640 1 -- 172.21.15.57:0/2282182530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ca4145130 msgr2=0x7f3ca414abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:00.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3ca2ffd640 1 --2- 172.21.15.57:0/2282182530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ca4145130 0x7f3ca414abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:00.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3ca2ffd640 1 -- 172.21.15.57:0/2282182530 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca414f7e0 con 0x7f3c90003980 2023-10-30T14:40:00.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3ca2ffd640 1 --2- 172.21.15.57:0/2282182530 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c90003980 0x7f3ca41505a0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f3c9400c210 tx=0x7f3c90005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:00.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3c9b7fe640 1 -- 172.21.15.57:0/2282182530 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c94004ab0 con 0x7f3c90003980 2023-10-30T14:40:00.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3c9b7fe640 1 -- 172.21.15.57:0/2282182530 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c940b6070 con 0x7f3c90003980 2023-10-30T14:40:00.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3ca9617640 1 -- 172.21.15.57:0/2282182530 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca414fac0 con 0x7f3c90003980 2023-10-30T14:40:00.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3c9b7fe640 1 -- 172.21.15.57:0/2282182530 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c94004e30 con 0x7f3c90003980 2023-10-30T14:40:00.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3ca9617640 1 -- 172.21.15.57:0/2282182530 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca41f31b0 con 0x7f3c90003980 2023-10-30T14:40:00.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.619+0000 7f3ca9617640 1 -- 172.21.15.57:0/2282182530 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca404e170 con 0x7f3c90003980 2023-10-30T14:40:00.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.623+0000 7f3c9b7fe640 1 -- 172.21.15.57:0/2282182530 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3c94007070 con 0x7f3c90003980 2023-10-30T14:40:00.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.623+0000 7f3c9b7fe640 1 --2- 172.21.15.57:0/2282182530 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3c840664d0 0x7f3c84068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:00.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.623+0000 7f3c9b7fe640 1 -- 172.21.15.57:0/2282182530 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7f3c940b8070 con 0x7f3c90003980 2023-10-30T14:40:00.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.623+0000 7f3c9bfff640 1 --2- 172.21.15.57:0/2282182530 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3c840664d0 0x7f3c84068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:00.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.623+0000 7f3c9bfff640 1 --2- 172.21.15.57:0/2282182530 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3c840664d0 0x7f3c84068990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3ca41531b0 tx=0x7f3c8c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:00.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.627+0000 7f3c9b7fe640 1 -- 172.21.15.57:0/2282182530 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3c940ee6f0 con 0x7f3c90003980 2023-10-30T14:40:00.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.815+0000 7f3ca9617640 1 -- 172.21.15.57:0/2282182530 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ca41494e0 con 0x7f3c90003980 2023-10-30T14:40:00.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.815+0000 7f3c9b7fe640 1 -- 172.21.15.57:0/2282182530 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+5740 (secure 0 0 0) 0x7f3c940f1d10 con 0x7f3c90003980 2023-10-30T14:40:00.819 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:40:00.819 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":141,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:39:57.112308+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:40:00.820 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 141 2023-10-30T14:40:00.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.815+0000 7f3ca9617640 1 -- 172.21.15.57:0/2282182530 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3c840664d0 msgr2=0x7f3c84068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:00.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.815+0000 7f3ca9617640 1 --2- 172.21.15.57:0/2282182530 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3c840664d0 0x7f3c84068990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3ca41531b0 tx=0x7f3c8c008040 comp rx=0 tx=0).stop 2023-10-30T14:40:00.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.815+0000 7f3ca9617640 1 -- 172.21.15.57:0/2282182530 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c90003980 msgr2=0x7f3ca41505a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:00.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.815+0000 7f3ca9617640 1 --2- 172.21.15.57:0/2282182530 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c90003980 0x7f3ca41505a0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f3c9400c210 tx=0x7f3c90005800 comp rx=0 tx=0).stop 2023-10-30T14:40:00.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.815+0000 7f3ca9617640 1 -- 172.21.15.57:0/2282182530 shutdown_connections 2023-10-30T14:40:00.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.815+0000 7f3ca9617640 1 --2- 172.21.15.57:0/2282182530 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3c840664d0 0x7f3c84068990 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:00.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.815+0000 7f3ca9617640 1 --2- 172.21.15.57:0/2282182530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ca4145130 0x7f3ca414abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:00.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.815+0000 7f3ca9617640 1 --2- 172.21.15.57:0/2282182530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ca40c8780 0x7f3ca40c9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:00.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.815+0000 7f3ca9617640 1 --2- 172.21.15.57:0/2282182530 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c90003980 0x7f3ca41505a0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:00.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.815+0000 7f3ca9617640 1 -- 172.21.15.57:0/2282182530 >> 172.21.15.57:0/2282182530 conn(0x7f3ca4071fb0 msgr2=0x7f3ca40ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:40:00.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.819+0000 7f3ca9617640 1 -- 172.21.15.57:0/2282182530 shutdown_connections 2023-10-30T14:40:00.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:00.819+0000 7f3ca9617640 1 -- 172.21.15.57:0/2282182530 wait complete. 2023-10-30T14:40:00.840 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:40:02.841 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:40:02.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.991+0000 7fe1d029e640 1 Processor -- start 2023-10-30T14:40:02.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.991+0000 7fe1d029e640 1 -- start start 2023-10-30T14:40:02.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1d029e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c806bd10 0x7fe1c80c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:02.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1d029e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c80c5030 con 0x7fe1c8151c20 2023-10-30T14:40:02.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1d029e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c8157830 con 0x7fe1c8151840 2023-10-30T14:40:02.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1d029e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c8157b10 con 0x7fe1c806bd10 2023-10-30T14:40:02.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd839640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fe1c8151c20 0x7fe1c806b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44800/0 (socket says 172.21.15.57:44800) 2023-10-30T14:40:02.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd839640 1 -- 172.21.15.57:0/275332735 learned_addr learned my addr 172.21.15.57:0/275332735 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:40:02.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1ce83b640 1 --2- 172.21.15.57:0/275332735 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c806bd10 0x7fe1c80c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:03.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd038640 1 -- 172.21.15.57:0/275332735 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090205443 0 0) 0x7fe1c80c5030 con 0x7fe1c8151c20 2023-10-30T14:40:03.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd038640 1 -- 172.21.15.57:0/275332735 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1b4003530 con 0x7fe1c8151c20 2023-10-30T14:40:03.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd038640 1 -- 172.21.15.57:0/275332735 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109484561 0 0) 0x7fe1c8157830 con 0x7fe1c8151840 2023-10-30T14:40:03.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd038640 1 -- 172.21.15.57:0/275332735 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c80c5030 con 0x7fe1c8151840 2023-10-30T14:40:03.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd038640 1 -- 172.21.15.57:0/275332735 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1983220652 0 0) 0x7fe1b4003530 con 0x7fe1c8151c20 2023-10-30T14:40:03.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd038640 1 -- 172.21.15.57:0/275332735 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c806bd10 msgr2=0x7fe1c80c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:40:03.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd038640 1 --2- 172.21.15.57:0/275332735 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c806bd10 0x7fe1c80c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:03.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd038640 1 -- 172.21.15.57:0/275332735 >> v1:172.21.15.88:6789/0 conn(0x7fe1c8151840 legacy=0x7fe1c8072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:03.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd038640 1 -- 172.21.15.57:0/275332735 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c8157df0 con 0x7fe1c8151c20 2023-10-30T14:40:03.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd038640 1 -- 172.21.15.57:0/275332735 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fe1b8001680 con 0x7fe1c8151c20 2023-10-30T14:40:03.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1ce83b640 1 --2- 172.21.15.57:0/275332735 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c806bd10 0x7fe1c80c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:40:03.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd038640 1 -- 172.21.15.57:0/275332735 >> v1:172.21.15.57:6789/0 conn(0x7fe1c8151c20 legacy=0x7fe1c806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:03.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd038640 1 --2- 172.21.15.57:0/275332735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1b4003b40 0x7fe1b4003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:03.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd038640 1 --2- 172.21.15.57:0/275332735 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c806bd10 0x7fe1b40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:03.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd038640 1 --2- 172.21.15.57:0/275332735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1b4004d10 0x7fe1b4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:03.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd038640 1 -- 172.21.15.57:0/275332735 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c8157b10 con 0x7fe1b4004d10 2023-10-30T14:40:03.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd038640 1 -- 172.21.15.57:0/275332735 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b400ba40 con 0x7fe1b4003b40 2023-10-30T14:40:03.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd038640 1 -- 172.21.15.57:0/275332735 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b400bc00 con 0x7fe1c806bd10 2023-10-30T14:40:03.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1ce83b640 1 --2- 172.21.15.57:0/275332735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1b4004d10 0x7fe1b4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:03.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1ce03a640 1 --2- 172.21.15.57:0/275332735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1b4003b40 0x7fe1b4003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:03.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1cd839640 1 --2- 172.21.15.57:0/275332735 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c806bd10 0x7fe1b40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:03.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1ce83b640 1 -- 172.21.15.57:0/275332735 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c806bd10 msgr2=0x7fe1b40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:40:03.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1ce83b640 1 --2- 172.21.15.57:0/275332735 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c806bd10 0x7fe1b40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:03.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1ce83b640 1 -- 172.21.15.57:0/275332735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1b4003b40 msgr2=0x7fe1b4003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:40:03.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1ce83b640 1 --2- 172.21.15.57:0/275332735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1b4003b40 0x7fe1b4003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:03.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.995+0000 7fe1ce83b640 1 -- 172.21.15.57:0/275332735 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c0009a70 con 0x7fe1b4004d10 2023-10-30T14:40:03.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.999+0000 7fe1ce83b640 1 --2- 172.21.15.57:0/275332735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1b4004d10 0x7fe1b4005190 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000e350 tx=0x7fe1c001cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e3c80d2d369e8b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:03.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.999+0000 7fe1cd038640 1 -- 172.21.15.57:0/275332735 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c00a5df0 con 0x7fe1b4004d10 2023-10-30T14:40:03.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.999+0000 7fe1cd038640 1 -- 172.21.15.57:0/275332735 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c000f040 con 0x7fe1b4004d10 2023-10-30T14:40:03.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.999+0000 7fe1cd038640 1 -- 172.21.15.57:0/275332735 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c00ad640 con 0x7fe1b4004d10 2023-10-30T14:40:03.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.999+0000 7fe1d029e640 1 -- 172.21.15.57:0/275332735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1b4004d10 msgr2=0x7fe1b4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:03.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.999+0000 7fe1d029e640 1 --2- 172.21.15.57:0/275332735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1b4004d10 0x7fe1b4005190 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000e350 tx=0x7fe1c001cde0 comp rx=0 tx=0).stop 2023-10-30T14:40:03.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.999+0000 7fe1ce83b640 1 -- 172.21.15.57:0/275332735 reap_dead start 2023-10-30T14:40:03.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.999+0000 7fe1d029e640 1 -- 172.21.15.57:0/275332735 shutdown_connections 2023-10-30T14:40:03.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.999+0000 7fe1d029e640 1 -- 172.21.15.57:0/275332735 >> 172.21.15.57:0/275332735 conn(0x7fe1c80cac70 msgr2=0x7fe1c80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:40:03.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.999+0000 7fe1d029e640 1 -- 172.21.15.57:0/275332735 shutdown_connections 2023-10-30T14:40:03.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.999+0000 7fe1d029e640 1 -- 172.21.15.57:0/275332735 wait complete. 2023-10-30T14:40:03.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.999+0000 7fe1d029e640 1 Processor -- start 2023-10-30T14:40:03.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:02.999+0000 7fe1d029e640 1 -- start start 2023-10-30T14:40:03.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1d029e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1b4004d10 0x7fe1c81ee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:03.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1d029e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c81f25c0 0x7fe1c8071da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:03.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1d029e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1c814bfb0 0x7fe1c81473d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:03.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1d029e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c81f2a00 con 0x7fe1c814bfb0 2023-10-30T14:40:03.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1d029e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c814d790 con 0x7fe1b4004d10 2023-10-30T14:40:03.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1d029e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c81f1b00 con 0x7fe1c81f25c0 2023-10-30T14:40:03.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1ce83b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1c814bfb0 0x7fe1c81473d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:03.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1ce83b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1c814bfb0 0x7fe1c81473d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54918/0 (socket says 172.21.15.57:54918) 2023-10-30T14:40:03.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1ce83b640 1 -- 172.21.15.57:0/873774106 learned_addr learned my addr 172.21.15.57:0/873774106 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:40:03.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1cd839640 1 --2- 172.21.15.57:0/873774106 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c81f25c0 0x7fe1c8071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:03.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1ce03a640 1 --2- 172.21.15.57:0/873774106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1b4004d10 0x7fe1c81ee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:03.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1ce83b640 1 -- 172.21.15.57:0/873774106 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c81f25c0 msgr2=0x7fe1c8071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:03.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1ce83b640 1 --2- 172.21.15.57:0/873774106 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c81f25c0 0x7fe1c8071da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:03.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1ce83b640 1 -- 172.21.15.57:0/873774106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1b4004d10 msgr2=0x7fe1c81ee590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:03.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1ce83b640 1 --2- 172.21.15.57:0/873774106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1b4004d10 0x7fe1c81ee590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:03.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1ce83b640 1 -- 172.21.15.57:0/873774106 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c81e3570 con 0x7fe1c814bfb0 2023-10-30T14:40:03.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1ce83b640 1 --2- 172.21.15.57:0/873774106 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1c814bfb0 0x7fe1c81473d0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fe1c00b0040 tx=0x7fe1c0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:03.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1beffd640 1 -- 172.21.15.57:0/873774106 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c00039a0 con 0x7fe1c814bfb0 2023-10-30T14:40:03.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1beffd640 1 -- 172.21.15.57:0/873774106 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c0003b00 con 0x7fe1c814bfb0 2023-10-30T14:40:03.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1beffd640 1 -- 172.21.15.57:0/873774106 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0003e90 con 0x7fe1c814bfb0 2023-10-30T14:40:03.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1d029e640 1 -- 172.21.15.57:0/873774106 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c81e3850 con 0x7fe1c814bfb0 2023-10-30T14:40:03.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1d029e640 1 -- 172.21.15.57:0/873774106 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c81f1dc0 con 0x7fe1c814bfb0 2023-10-30T14:40:03.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.003+0000 7fe1d029e640 1 -- 172.21.15.57:0/873774106 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c804e170 con 0x7fe1c814bfb0 2023-10-30T14:40:03.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.007+0000 7fe1beffd640 1 -- 172.21.15.57:0/873774106 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe1c00b4040 con 0x7fe1c814bfb0 2023-10-30T14:40:03.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.007+0000 7fe1beffd640 1 --2- 172.21.15.57:0/873774106 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe1ac0664d0 0x7fe1ac068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:03.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.007+0000 7fe1beffd640 1 -- 172.21.15.57:0/873774106 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7fe1c011e870 con 0x7fe1c814bfb0 2023-10-30T14:40:03.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.007+0000 7fe1ce03a640 1 --2- 172.21.15.57:0/873774106 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe1ac0664d0 0x7fe1ac068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:03.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.007+0000 7fe1ce03a640 1 --2- 172.21.15.57:0/873774106 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe1ac0664d0 0x7fe1ac068990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4007af0 tx=0x7fe1c4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:03.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.011+0000 7fe1beffd640 1 -- 172.21.15.57:0/873774106 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe1c00eb730 con 0x7fe1c814bfb0 2023-10-30T14:40:03.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.199+0000 7fe1d029e640 1 -- 172.21.15.57:0/873774106 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1c806b6a0 con 0x7fe1c814bfb0 2023-10-30T14:40:03.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.199+0000 7fe1beffd640 1 -- 172.21.15.57:0/873774106 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+5737 (secure 0 0 0) 0x7fe1c0009de0 con 0x7fe1c814bfb0 2023-10-30T14:40:03.204 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:40:03.204 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:40:02.161922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:40:03.204 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 142 2023-10-30T14:40:03.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.203+0000 7fe1d029e640 1 -- 172.21.15.57:0/873774106 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe1ac0664d0 msgr2=0x7fe1ac068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:03.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.203+0000 7fe1d029e640 1 --2- 172.21.15.57:0/873774106 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe1ac0664d0 0x7fe1ac068990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4007af0 tx=0x7fe1c4009040 comp rx=0 tx=0).stop 2023-10-30T14:40:03.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.203+0000 7fe1d029e640 1 -- 172.21.15.57:0/873774106 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1c814bfb0 msgr2=0x7fe1c81473d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:03.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.203+0000 7fe1d029e640 1 --2- 172.21.15.57:0/873774106 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1c814bfb0 0x7fe1c81473d0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fe1c00b0040 tx=0x7fe1c0009750 comp rx=0 tx=0).stop 2023-10-30T14:40:03.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.203+0000 7fe1d029e640 1 -- 172.21.15.57:0/873774106 shutdown_connections 2023-10-30T14:40:03.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.203+0000 7fe1d029e640 1 --2- 172.21.15.57:0/873774106 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe1ac0664d0 0x7fe1ac068990 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:03.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.203+0000 7fe1d029e640 1 --2- 172.21.15.57:0/873774106 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1c814bfb0 0x7fe1c81473d0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:03.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.203+0000 7fe1d029e640 1 --2- 172.21.15.57:0/873774106 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1c81f25c0 0x7fe1c8071da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:03.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.203+0000 7fe1d029e640 1 --2- 172.21.15.57:0/873774106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1b4004d10 0x7fe1c81ee590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:03.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.203+0000 7fe1d029e640 1 -- 172.21.15.57:0/873774106 >> 172.21.15.57:0/873774106 conn(0x7fe1c80cac70 msgr2=0x7fe1c80733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:40:03.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.203+0000 7fe1d029e640 1 -- 172.21.15.57:0/873774106 shutdown_connections 2023-10-30T14:40:03.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:03.203+0000 7fe1d029e640 1 -- 172.21.15.57:0/873774106 wait complete. 2023-10-30T14:40:03.225 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:40:05.227 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:40:05.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.379+0000 7f6678fab640 1 Processor -- start 2023-10-30T14:40:05.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.379+0000 7f6678fab640 1 -- start start 2023-10-30T14:40:05.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.379+0000 7f6678fab640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6674153c80 0x7f667415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:05.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.379+0000 7f6678fab640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f66740c4a10 0x7f66740c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:05.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.379+0000 7f6678fab640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6674159f20 con 0x7f667415a3b0 2023-10-30T14:40:05.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.379+0000 7f6678fab640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f667415dec0 con 0x7f66740c4a10 2023-10-30T14:40:05.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.379+0000 7f6678fab640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66741599a0 con 0x7f6674153c80 2023-10-30T14:40:05.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.379+0000 7f666bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f66740c4a10 0x7f66740c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:05.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.379+0000 7f667259c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6674153c80 0x7f667415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:05.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.379+0000 7f666bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f66740c4a10 0x7f66740c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:47616/0 (socket says 172.21.15.57:47616) 2023-10-30T14:40:05.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.379+0000 7f666bfff640 1 -- 172.21.15.57:0/1177047214 learned_addr learned my addr 172.21.15.57:0/1177047214 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:40:05.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.379+0000 7f666bfff640 1 -- 172.21.15.57:0/1177047214 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6674153c80 msgr2=0x7f667415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:40:05.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.379+0000 7f666bfff640 1 --2- 172.21.15.57:0/1177047214 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6674153c80 0x7f667415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:05.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.383+0000 7f666bfff640 1 -- 172.21.15.57:0/1177047214 >> v1:172.21.15.88:6789/0 conn(0x7f667415a3b0 legacy=0x7f66740ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:05.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.383+0000 7f666bfff640 1 -- 172.21.15.57:0/1177047214 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66740c47e0 con 0x7f66740c4a10 2023-10-30T14:40:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.383+0000 7f666bfff640 1 --2- 172.21.15.57:0/1177047214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66740c4a10 0x7f66740c42a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f665c009870 tx=0x7f665c01ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78bca75270295970 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.383+0000 7f6671d9b640 1 -- 172.21.15.57:0/1177047214 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c0a8a30 con 0x7f66740c4a10 2023-10-30T14:40:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.383+0000 7f6671d9b640 1 -- 172.21.15.57:0/1177047214 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f665c0a8b90 con 0x7f66740c4a10 2023-10-30T14:40:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.383+0000 7f6671d9b640 1 -- 172.21.15.57:0/1177047214 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c0b0440 con 0x7f66740c4a10 2023-10-30T14:40:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.383+0000 7f6678fab640 1 -- 172.21.15.57:0/1177047214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66740c4a10 msgr2=0x7f66740c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.383+0000 7f6678fab640 1 --2- 172.21.15.57:0/1177047214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66740c4a10 0x7f66740c42a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f665c009870 tx=0x7f665c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:40:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.383+0000 7f6678fab640 1 -- 172.21.15.57:0/1177047214 shutdown_connections 2023-10-30T14:40:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.383+0000 7f6678fab640 1 --2- 172.21.15.57:0/1177047214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66740c4a10 0x7f66740c42a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.383+0000 7f6678fab640 1 --2- 172.21.15.57:0/1177047214 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6674153c80 0x7f667415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.383+0000 7f6678fab640 1 -- 172.21.15.57:0/1177047214 >> 172.21.15.57:0/1177047214 conn(0x7f66740715c0 msgr2=0x7f66740719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:40:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.383+0000 7f6678fab640 1 -- 172.21.15.57:0/1177047214 shutdown_connections 2023-10-30T14:40:05.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f6678fab640 1 -- 172.21.15.57:0/1177047214 wait complete. 2023-10-30T14:40:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f6678fab640 1 Processor -- start 2023-10-30T14:40:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f6678fab640 1 -- start start 2023-10-30T14:40:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f6678fab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66740c4a10 0x7f66740b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f6678fab640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6674153c80 0x7f66740ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f6678fab640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f667415a3b0 0x7f66740b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:05.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f6678fab640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66741ef230 con 0x7f66740c4a10 2023-10-30T14:40:05.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f6678fab640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66740b28d0 con 0x7f667415a3b0 2023-10-30T14:40:05.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f6678fab640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66741ef4f0 con 0x7f6674153c80 2023-10-30T14:40:05.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f667259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66740c4a10 0x7f66740b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:05.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f667259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66740c4a10 0x7f66740b2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51320/0 (socket says 172.21.15.57:51320) 2023-10-30T14:40:05.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f667259c640 1 -- 172.21.15.57:0/132237067 learned_addr learned my addr 172.21.15.57:0/132237067 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:40:05.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f666bfff640 1 --2- 172.21.15.57:0/132237067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6674153c80 0x7f66740ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:05.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f6672d9d640 1 --2- 172.21.15.57:0/132237067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f667415a3b0 0x7f66740b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:05.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f667259c640 1 -- 172.21.15.57:0/132237067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6674153c80 msgr2=0x7f66740ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:05.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f667259c640 1 --2- 172.21.15.57:0/132237067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6674153c80 0x7f66740ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:05.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f667259c640 1 -- 172.21.15.57:0/132237067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f667415a3b0 msgr2=0x7f66740b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:05.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f667259c640 1 --2- 172.21.15.57:0/132237067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f667415a3b0 0x7f66740b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:05.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f667259c640 1 -- 172.21.15.57:0/132237067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665c0094d0 con 0x7f66740c4a10 2023-10-30T14:40:05.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f667259c640 1 --2- 172.21.15.57:0/132237067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66740c4a10 0x7f66740b2020 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f666400e930 tx=0x7f666400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:05.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.387+0000 7f666affd640 1 -- 172.21.15.57:0/132237067 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6664009690 con 0x7f66740c4a10 2023-10-30T14:40:05.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.391+0000 7f666affd640 1 -- 172.21.15.57:0/132237067 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666400ca00 con 0x7f66740c4a10 2023-10-30T14:40:05.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.391+0000 7f6678fab640 1 -- 172.21.15.57:0/132237067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66741ef7d0 con 0x7f66740c4a10 2023-10-30T14:40:05.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.391+0000 7f666affd640 1 -- 172.21.15.57:0/132237067 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666400cd70 con 0x7f66740c4a10 2023-10-30T14:40:05.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.391+0000 7f6678fab640 1 -- 172.21.15.57:0/132237067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66741efd90 con 0x7f66740c4a10 2023-10-30T14:40:05.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.391+0000 7f6678fab640 1 -- 172.21.15.57:0/132237067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f667404e170 con 0x7f66740c4a10 2023-10-30T14:40:05.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.391+0000 7f666affd640 1 -- 172.21.15.57:0/132237067 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6664013470 con 0x7f66740c4a10 2023-10-30T14:40:05.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.391+0000 7f666affd640 1 --2- 172.21.15.57:0/132237067 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66540664d0 0x7f6654068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:05.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.391+0000 7f666affd640 1 -- 172.21.15.57:0/132237067 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7f66640846f0 con 0x7f66740c4a10 2023-10-30T14:40:05.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.391+0000 7f666bfff640 1 --2- 172.21.15.57:0/132237067 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66540664d0 0x7f6654068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:05.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.395+0000 7f666bfff640 1 --2- 172.21.15.57:0/132237067 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66540664d0 0x7f6654068990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f665c0b2000 tx=0x7f665c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:05.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.395+0000 7f666affd640 1 -- 172.21.15.57:0/132237067 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6664051670 con 0x7f66740c4a10 2023-10-30T14:40:05.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.587+0000 7f6678fab640 1 -- 172.21.15.57:0/132237067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66740b5050 con 0x7f66740c4a10 2023-10-30T14:40:05.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.587+0000 7f666affd640 1 -- 172.21.15.57:0/132237067 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f6664054c90 con 0x7f66740c4a10 2023-10-30T14:40:05.592 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:40:05.592 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:40:04.158908+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:40:05.592 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 143 2023-10-30T14:40:05.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.591+0000 7f6678fab640 1 -- 172.21.15.57:0/132237067 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66540664d0 msgr2=0x7f6654068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:05.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.591+0000 7f6678fab640 1 --2- 172.21.15.57:0/132237067 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66540664d0 0x7f6654068990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f665c0b2000 tx=0x7f665c008040 comp rx=0 tx=0).stop 2023-10-30T14:40:05.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.591+0000 7f6678fab640 1 -- 172.21.15.57:0/132237067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66740c4a10 msgr2=0x7f66740b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:05.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.591+0000 7f6678fab640 1 --2- 172.21.15.57:0/132237067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66740c4a10 0x7f66740b2020 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f666400e930 tx=0x7f666400edf0 comp rx=0 tx=0).stop 2023-10-30T14:40:05.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.591+0000 7f6678fab640 1 -- 172.21.15.57:0/132237067 shutdown_connections 2023-10-30T14:40:05.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.591+0000 7f6678fab640 1 --2- 172.21.15.57:0/132237067 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66540664d0 0x7f6654068990 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:05.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.591+0000 7f6678fab640 1 --2- 172.21.15.57:0/132237067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f667415a3b0 0x7f66740b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:05.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.591+0000 7f6678fab640 1 --2- 172.21.15.57:0/132237067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6674153c80 0x7f66740ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:05.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.591+0000 7f6678fab640 1 --2- 172.21.15.57:0/132237067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66740c4a10 0x7f66740b2020 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:05.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.591+0000 7f6678fab640 1 -- 172.21.15.57:0/132237067 >> 172.21.15.57:0/132237067 conn(0x7f66740715c0 msgr2=0x7f6674157970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:40:05.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.591+0000 7f6678fab640 1 -- 172.21.15.57:0/132237067 shutdown_connections 2023-10-30T14:40:05.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:05.591+0000 7f6678fab640 1 -- 172.21.15.57:0/132237067 wait complete. 2023-10-30T14:40:05.613 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:40:07.614 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:40:07.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.763+0000 7f4245005640 1 Processor -- start 2023-10-30T14:40:07.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.763+0000 7f4245005640 1 -- start start 2023-10-30T14:40:07.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.767+0000 7f4245005640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4240153c10 0x7f4240065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:07.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.767+0000 7f4245005640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4240159970 0x7f4240159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:07.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.767+0000 7f4245005640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424006e390 con 0x7f4240070c30 2023-10-30T14:40:07.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.767+0000 7f4245005640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4240072390 con 0x7f4240153c10 2023-10-30T14:40:07.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.767+0000 7f4245005640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42400660e0 con 0x7f4240159970 2023-10-30T14:40:07.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.767+0000 7f423ed9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4240153c10 0x7f4240065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:07.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.767+0000 7f423ed9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4240153c10 0x7f4240065ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51326/0 (socket says 172.21.15.57:51326) 2023-10-30T14:40:07.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.767+0000 7f423ed9d640 1 -- 172.21.15.57:0/3615667341 learned_addr learned my addr 172.21.15.57:0/3615667341 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:40:07.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.767+0000 7f423f59e640 1 --2- 172.21.15.57:0/3615667341 >> v2:172.21.15.88:3300/0 conn(0x7f4240159970 0x7f4240159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:07.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.767+0000 7f423ed9d640 1 -- 172.21.15.57:0/3615667341 >> v2:172.21.15.88:3300/0 conn(0x7f4240159970 msgr2=0x7f4240159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:40:07.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.767+0000 7f423ed9d640 1 --2- 172.21.15.57:0/3615667341 >> v2:172.21.15.88:3300/0 conn(0x7f4240159970 0x7f4240159dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:07.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.767+0000 7f423ed9d640 1 -- 172.21.15.57:0/3615667341 >> v1:172.21.15.88:6789/0 conn(0x7f4240070c30 legacy=0x7f4240071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:40:07.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.767+0000 7f423ed9d640 1 -- 172.21.15.57:0/3615667341 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42400bec10 con 0x7f4240153c10 2023-10-30T14:40:07.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.767+0000 7f423ed9d640 1 --2- 172.21.15.57:0/3615667341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4240153c10 0x7f4240065ba0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f424015df80 tx=0x7f423001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b7b776b1fcef29b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:07.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f423dd9b640 1 -- 172.21.15.57:0/3615667341 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42300ab600 con 0x7f4240153c10 2023-10-30T14:40:07.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f423dd9b640 1 -- 172.21.15.57:0/3615667341 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42300a7050 con 0x7f4240153c10 2023-10-30T14:40:07.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f423dd9b640 1 -- 172.21.15.57:0/3615667341 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42300ab980 con 0x7f4240153c10 2023-10-30T14:40:07.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f4245005640 1 -- 172.21.15.57:0/3615667341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4240153c10 msgr2=0x7f4240065ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:07.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f4245005640 1 --2- 172.21.15.57:0/3615667341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4240153c10 0x7f4240065ba0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f424015df80 tx=0x7f423001c430 comp rx=0 tx=0).stop 2023-10-30T14:40:07.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f4245005640 1 -- 172.21.15.57:0/3615667341 shutdown_connections 2023-10-30T14:40:07.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f4245005640 1 --2- 172.21.15.57:0/3615667341 >> v2:172.21.15.88:3300/0 conn(0x7f4240159970 0x7f4240159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:07.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f4245005640 1 --2- 172.21.15.57:0/3615667341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4240153c10 0x7f4240065ba0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:07.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f4245005640 1 -- 172.21.15.57:0/3615667341 >> 172.21.15.57:0/3615667341 conn(0x7f42400c7f60 msgr2=0x7f42400c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:40:07.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f4245005640 1 -- 172.21.15.57:0/3615667341 shutdown_connections 2023-10-30T14:40:07.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f4245005640 1 -- 172.21.15.57:0/3615667341 wait complete. 2023-10-30T14:40:07.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f4245005640 1 Processor -- start 2023-10-30T14:40:07.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f4245005640 1 -- start start 2023-10-30T14:40:07.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f4245005640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4240070c30 0x7f424014d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:07.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f4245005640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4240153c10 0x7f424014a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:07.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f4245005640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4240159970 0x7f424014e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:07.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f4245005640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42401f1ee0 con 0x7f4240159970 2023-10-30T14:40:07.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f4245005640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42401f21a0 con 0x7f4240153c10 2023-10-30T14:40:07.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f4245005640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42401f2480 con 0x7f4240070c30 2023-10-30T14:40:07.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f423f59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4240159970 0x7f424014e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:07.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f423f59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4240159970 0x7f424014e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51328/0 (socket says 172.21.15.57:51328) 2023-10-30T14:40:07.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.771+0000 7f423f59e640 1 -- 172.21.15.57:0/1744367121 learned_addr learned my addr 172.21.15.57:0/1744367121 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:40:07.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.775+0000 7f423ed9d640 1 --2- 172.21.15.57:0/1744367121 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4240070c30 0x7f424014d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:07.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.775+0000 7f423e59c640 1 --2- 172.21.15.57:0/1744367121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4240153c10 0x7f424014a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:07.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.775+0000 7f423f59e640 1 -- 172.21.15.57:0/1744367121 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4240070c30 msgr2=0x7f424014d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:07.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.775+0000 7f423f59e640 1 --2- 172.21.15.57:0/1744367121 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4240070c30 0x7f424014d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:07.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.775+0000 7f423f59e640 1 -- 172.21.15.57:0/1744367121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4240153c10 msgr2=0x7f424014a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:07.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.775+0000 7f423f59e640 1 --2- 172.21.15.57:0/1744367121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4240153c10 0x7f424014a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:07.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.775+0000 7f423f59e640 1 -- 172.21.15.57:0/1744367121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42300094d0 con 0x7f4240159970 2023-10-30T14:40:07.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.775+0000 7f423e59c640 1 --2- 172.21.15.57:0/1744367121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4240153c10 0x7f424014a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:40:07.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.775+0000 7f423f59e640 1 --2- 172.21.15.57:0/1744367121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4240159970 0x7f424014e610 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f4238007e60 tx=0x7f423800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:07.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.775+0000 7f42277fe640 1 -- 172.21.15.57:0/1744367121 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238014420 con 0x7f4240159970 2023-10-30T14:40:07.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.775+0000 7f42277fe640 1 -- 172.21.15.57:0/1744367121 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42380145a0 con 0x7f4240159970 2023-10-30T14:40:07.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.775+0000 7f42277fe640 1 -- 172.21.15.57:0/1744367121 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238014950 con 0x7f4240159970 2023-10-30T14:40:07.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.775+0000 7f4245005640 1 -- 172.21.15.57:0/1744367121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42401f2760 con 0x7f4240159970 2023-10-30T14:40:07.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.775+0000 7f4245005640 1 -- 172.21.15.57:0/1744367121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f424014de10 con 0x7f4240159970 2023-10-30T14:40:07.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.775+0000 7f4245005640 1 -- 172.21.15.57:0/1744367121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f424004e170 con 0x7f4240159970 2023-10-30T14:40:07.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.775+0000 7f42277fe640 1 -- 172.21.15.57:0/1744367121 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4238014ab0 con 0x7f4240159970 2023-10-30T14:40:07.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.775+0000 7f42277fe640 1 --2- 172.21.15.57:0/1744367121 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f422c0664a0 0x7f422c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:07.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.779+0000 7f42277fe640 1 -- 172.21.15.57:0/1744367121 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7f4238084cc0 con 0x7f4240159970 2023-10-30T14:40:07.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.779+0000 7f423ed9d640 1 --2- 172.21.15.57:0/1744367121 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f422c0664a0 0x7f422c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:07.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.779+0000 7f423ed9d640 1 --2- 172.21.15.57:0/1744367121 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f422c0664a0 0x7f422c068960 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f42300ad000 tx=0x7f4230008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:07.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.779+0000 7f42277fe640 1 -- 172.21.15.57:0/1744367121 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4238051b80 con 0x7f4240159970 2023-10-30T14:40:07.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.967+0000 7f4245005640 1 -- 172.21.15.57:0/1744367121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f424014ca30 con 0x7f4240159970 2023-10-30T14:40:07.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.967+0000 7f42277fe640 1 -- 172.21.15.57:0/1744367121 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f42380551a0 con 0x7f4240159970 2023-10-30T14:40:07.973 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:40:07.974 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:40:04.158908+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:40:07.974 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 143 2023-10-30T14:40:07.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.971+0000 7f4245005640 1 -- 172.21.15.57:0/1744367121 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f422c0664a0 msgr2=0x7f422c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:07.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.971+0000 7f4245005640 1 --2- 172.21.15.57:0/1744367121 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f422c0664a0 0x7f422c068960 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f42300ad000 tx=0x7f4230008040 comp rx=0 tx=0).stop 2023-10-30T14:40:07.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.971+0000 7f4245005640 1 -- 172.21.15.57:0/1744367121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4240159970 msgr2=0x7f424014e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:07.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.971+0000 7f4245005640 1 --2- 172.21.15.57:0/1744367121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4240159970 0x7f424014e610 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f4238007e60 tx=0x7f423800e4b0 comp rx=0 tx=0).stop 2023-10-30T14:40:07.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.971+0000 7f4245005640 1 -- 172.21.15.57:0/1744367121 shutdown_connections 2023-10-30T14:40:07.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.971+0000 7f4245005640 1 --2- 172.21.15.57:0/1744367121 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f422c0664a0 0x7f422c068960 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:07.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.971+0000 7f4245005640 1 --2- 172.21.15.57:0/1744367121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4240159970 0x7f424014e610 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:07.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.971+0000 7f4245005640 1 --2- 172.21.15.57:0/1744367121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4240153c10 0x7f424014a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:07.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.971+0000 7f4245005640 1 --2- 172.21.15.57:0/1744367121 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4240070c30 0x7f424014d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:07.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.971+0000 7f4245005640 1 -- 172.21.15.57:0/1744367121 >> 172.21.15.57:0/1744367121 conn(0x7f42400c7f60 msgr2=0x7f4240157d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:40:07.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.971+0000 7f4245005640 1 -- 172.21.15.57:0/1744367121 shutdown_connections 2023-10-30T14:40:07.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:07.971+0000 7f4245005640 1 -- 172.21.15.57:0/1744367121 wait complete. 2023-10-30T14:40:07.991 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:40:07.994 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:40:07.996 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:40:09.996 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:40:10.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.147+0000 7fb6e791d640 1 Processor -- start 2023-10-30T14:40:10.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.147+0000 7fb6e791d640 1 -- start start 2023-10-30T14:40:10.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.151+0000 7fb6e791d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb6e0150830 0x7fb6e006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:10.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.151+0000 7fb6e791d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6e00cb7d0 0x7fb6e00c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:10.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.151+0000 7fb6e791d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e00c2310 con 0x7fb6e00c3e40 2023-10-30T14:40:10.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.151+0000 7fb6e791d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6e0156710 con 0x7fb6e0150830 2023-10-30T14:40:10.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.151+0000 7fb6e791d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e00c4360 con 0x7fb6e00cb7d0 2023-10-30T14:40:10.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.151+0000 7fb6e4eb8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6e00cb7d0 0x7fb6e00c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:10.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.151+0000 7fb6e56b9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb6e0150830 0x7fb6e006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:10.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.151+0000 7fb6e4eb8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6e00cb7d0 0x7fb6e00c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:59288/0 (socket says 172.21.15.57:59288) 2023-10-30T14:40:10.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.151+0000 7fb6e4eb8640 1 -- 172.21.15.57:0/3089422257 learned_addr learned my addr 172.21.15.57:0/3089422257 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:40:10.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.151+0000 7fb6e56b9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb6e0150830 0x7fb6e006b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:47658/0 (socket says 172.21.15.57:47658) 2023-10-30T14:40:10.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.151+0000 7fb6e56b9640 1 -- 172.21.15.57:0/3089422257 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6e00cb7d0 msgr2=0x7fb6e00c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:40:10.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.151+0000 7fb6e56b9640 1 --2- 172.21.15.57:0/3089422257 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6e00cb7d0 0x7fb6e00c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:10.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.151+0000 7fb6e56b9640 1 -- 172.21.15.57:0/3089422257 >> v1:172.21.15.88:6789/0 conn(0x7fb6e00c3e40 legacy=0x7fb6e0070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:10.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.151+0000 7fb6e56b9640 1 -- 172.21.15.57:0/3089422257 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e01569b0 con 0x7fb6e0150830 2023-10-30T14:40:10.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.151+0000 7fb6e4eb8640 1 --2- 172.21.15.57:0/3089422257 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6e00cb7d0 0x7fb6e00c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:40:10.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.155+0000 7fb6e56b9640 1 --2- 172.21.15.57:0/3089422257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6e0150830 0x7fb6e006b500 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0097c0 tx=0x7fb6dc01ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5bbe00d0bf821f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:10.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.155+0000 7fb6d7fff640 1 -- 172.21.15.57:0/3089422257 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc0a8970 con 0x7fb6e0150830 2023-10-30T14:40:10.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.155+0000 7fb6d7fff640 1 -- 172.21.15.57:0/3089422257 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6dc0a8ad0 con 0x7fb6e0150830 2023-10-30T14:40:10.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.155+0000 7fb6d7fff640 1 -- 172.21.15.57:0/3089422257 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc0a8e60 con 0x7fb6e0150830 2023-10-30T14:40:10.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.155+0000 7fb6e791d640 1 -- 172.21.15.57:0/3089422257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6e0150830 msgr2=0x7fb6e006b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:10.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.155+0000 7fb6e791d640 1 --2- 172.21.15.57:0/3089422257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6e0150830 0x7fb6e006b500 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0097c0 tx=0x7fb6dc01ec70 comp rx=0 tx=0).stop 2023-10-30T14:40:10.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.155+0000 7fb6e791d640 1 -- 172.21.15.57:0/3089422257 shutdown_connections 2023-10-30T14:40:10.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.155+0000 7fb6e791d640 1 --2- 172.21.15.57:0/3089422257 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6e00cb7d0 0x7fb6e00c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:10.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.155+0000 7fb6e791d640 1 --2- 172.21.15.57:0/3089422257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6e0150830 0x7fb6e006b500 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:10.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.155+0000 7fb6e791d640 1 -- 172.21.15.57:0/3089422257 >> 172.21.15.57:0/3089422257 conn(0x7fb6e00cac70 msgr2=0x7fb6e00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:40:10.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.155+0000 7fb6e791d640 1 -- 172.21.15.57:0/3089422257 shutdown_connections 2023-10-30T14:40:10.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.155+0000 7fb6e791d640 1 -- 172.21.15.57:0/3089422257 wait complete. 2023-10-30T14:40:10.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.155+0000 7fb6e791d640 1 Processor -- start 2023-10-30T14:40:10.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.155+0000 7fb6e791d640 1 -- start start 2023-10-30T14:40:10.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e791d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6e00c3e40 0x7fb6e01f25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:10.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e791d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6e00cb7d0 0x7fb6e01f09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:10.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e791d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6e0150830 0x7fb6e01f0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:10.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e791d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e01519f0 con 0x7fb6e00cb7d0 2023-10-30T14:40:10.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e791d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e01f0280 con 0x7fb6e0150830 2023-10-30T14:40:10.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e791d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e014bd10 con 0x7fb6e00c3e40 2023-10-30T14:40:10.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e4eb8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6e00cb7d0 0x7fb6e01f09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:10.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e4eb8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6e00cb7d0 0x7fb6e01f09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51344/0 (socket says 172.21.15.57:51344) 2023-10-30T14:40:10.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e4eb8640 1 -- 172.21.15.57:0/1829976408 learned_addr learned my addr 172.21.15.57:0/1829976408 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:40:10.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e56b9640 1 --2- 172.21.15.57:0/1829976408 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6e00c3e40 0x7fb6e01f25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:10.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e5eba640 1 --2- 172.21.15.57:0/1829976408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6e0150830 0x7fb6e01f0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:10.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e4eb8640 1 -- 172.21.15.57:0/1829976408 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6e00c3e40 msgr2=0x7fb6e01f25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:10.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e4eb8640 1 --2- 172.21.15.57:0/1829976408 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6e00c3e40 0x7fb6e01f25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:10.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e4eb8640 1 -- 172.21.15.57:0/1829976408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6e0150830 msgr2=0x7fb6e01f0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:10.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e4eb8640 1 --2- 172.21.15.57:0/1829976408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6e0150830 0x7fb6e01f0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:10.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e4eb8640 1 -- 172.21.15.57:0/1829976408 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d000b770 con 0x7fb6e00cb7d0 2023-10-30T14:40:10.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e4eb8640 1 --2- 172.21.15.57:0/1829976408 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6e00cb7d0 0x7fb6e01f09f0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fb6d000eda0 tx=0x7fb6d000d720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:10.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6d5ffb640 1 -- 172.21.15.57:0/1829976408 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d0010850 con 0x7fb6e00cb7d0 2023-10-30T14:40:10.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6d5ffb640 1 -- 172.21.15.57:0/1829976408 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d00109b0 con 0x7fb6e00cb7d0 2023-10-30T14:40:10.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6d5ffb640 1 -- 172.21.15.57:0/1829976408 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d0010d20 con 0x7fb6e00cb7d0 2023-10-30T14:40:10.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e791d640 1 -- 172.21.15.57:0/1829976408 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6dc009420 con 0x7fb6e00cb7d0 2023-10-30T14:40:10.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e791d640 1 -- 172.21.15.57:0/1829976408 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e014bf80 con 0x7fb6e00cb7d0 2023-10-30T14:40:10.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6e791d640 1 -- 172.21.15.57:0/1829976408 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6e004e170 con 0x7fb6e00cb7d0 2023-10-30T14:40:10.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.159+0000 7fb6d5ffb640 1 -- 172.21.15.57:0/1829976408 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb6d000f3d0 con 0x7fb6e00cb7d0 2023-10-30T14:40:10.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.163+0000 7fb6d5ffb640 1 --2- 172.21.15.57:0/1829976408 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb6c40664a0 0x7fb6c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:10.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.163+0000 7fb6d5ffb640 1 -- 172.21.15.57:0/1829976408 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7fb6d00849c0 con 0x7fb6e00cb7d0 2023-10-30T14:40:10.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.163+0000 7fb6e56b9640 1 --2- 172.21.15.57:0/1829976408 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb6c40664a0 0x7fb6c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:10.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.163+0000 7fb6e56b9640 1 --2- 172.21.15.57:0/1829976408 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb6c40664a0 0x7fb6c4068960 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0b2000 tx=0x7fb6dc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:10.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.167+0000 7fb6d5ffb640 1 -- 172.21.15.57:0/1829976408 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb6d0051920 con 0x7fb6e00cb7d0 2023-10-30T14:40:10.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.359+0000 7fb6e791d640 1 -- 172.21.15.57:0/1829976408 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6e0070300 con 0x7fb6e00cb7d0 2023-10-30T14:40:10.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.363+0000 7fb6d5ffb640 1 -- 172.21.15.57:0/1829976408 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+5736 (secure 0 0 0) 0x7fb6d0054f40 con 0x7fb6e00cb7d0 2023-10-30T14:40:10.366 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:40:10.366 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:40:07.263309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:40:10.366 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 144 2023-10-30T14:40:10.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.363+0000 7fb6e791d640 1 -- 172.21.15.57:0/1829976408 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb6c40664a0 msgr2=0x7fb6c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:10.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.363+0000 7fb6e791d640 1 --2- 172.21.15.57:0/1829976408 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb6c40664a0 0x7fb6c4068960 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0b2000 tx=0x7fb6dc008040 comp rx=0 tx=0).stop 2023-10-30T14:40:10.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.363+0000 7fb6e791d640 1 -- 172.21.15.57:0/1829976408 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6e00cb7d0 msgr2=0x7fb6e01f09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:10.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.363+0000 7fb6e791d640 1 --2- 172.21.15.57:0/1829976408 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6e00cb7d0 0x7fb6e01f09f0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fb6d000eda0 tx=0x7fb6d000d720 comp rx=0 tx=0).stop 2023-10-30T14:40:10.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.363+0000 7fb6e791d640 1 -- 172.21.15.57:0/1829976408 shutdown_connections 2023-10-30T14:40:10.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.363+0000 7fb6e791d640 1 --2- 172.21.15.57:0/1829976408 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb6c40664a0 0x7fb6c4068960 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:10.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.363+0000 7fb6e791d640 1 --2- 172.21.15.57:0/1829976408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6e0150830 0x7fb6e01f0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:10.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.363+0000 7fb6e791d640 1 --2- 172.21.15.57:0/1829976408 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6e00cb7d0 0x7fb6e01f09f0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:10.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.363+0000 7fb6e791d640 1 --2- 172.21.15.57:0/1829976408 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6e00c3e40 0x7fb6e01f25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:10.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.363+0000 7fb6e791d640 1 -- 172.21.15.57:0/1829976408 >> 172.21.15.57:0/1829976408 conn(0x7fb6e00cac70 msgr2=0x7fb6e014da60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:40:10.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.363+0000 7fb6e791d640 1 -- 172.21.15.57:0/1829976408 shutdown_connections 2023-10-30T14:40:10.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:10.363+0000 7fb6e791d640 1 -- 172.21.15.57:0/1829976408 wait complete. 2023-10-30T14:40:10.387 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=2, replacing gid=6125 2023-10-30T14:40:10.387 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.f 2023-10-30T14:40:12.044 INFO:tasks.ceph.mds.a.smithi057.stderr:2023-10-30T14:40:12.043+0000 7f5ba0392640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T14:40:17.388 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-30T14:40:17.389 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T14:40:17.389 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T14:40:17.392 INFO:tasks.ceph.mds.f:Started 2023-10-30T14:40:17.392 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:40:17.469 INFO:tasks.ceph.mds.f.smithi088.stdout:starting mds.f at 2023-10-30T14:40:17.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.535+0000 7f48635a1640 1 Processor -- start 2023-10-30T14:40:17.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.535+0000 7f48635a1640 1 -- start start 2023-10-30T14:40:17.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.535+0000 7f48635a1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f485c06bb40 0x7f485c157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:17.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.535+0000 7f48635a1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f485c06c420 0x7f485c157f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:17.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.535+0000 7f48635a1640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c071e00 con 0x7f485c06b760 2023-10-30T14:40:17.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.535+0000 7f48635a1640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485c070020 con 0x7f485c06bb40 2023-10-30T14:40:17.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.535+0000 7f48635a1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485c070f10 con 0x7f485c06c420 2023-10-30T14:40:17.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.535+0000 7f4860b3c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f485c06bb40 0x7f485c157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:17.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.535+0000 7f4860b3c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f485c06bb40 0x7f485c157a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50652/0 (socket says 172.21.15.57:50652) 2023-10-30T14:40:17.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.535+0000 7f4860b3c640 1 -- 172.21.15.57:0/3922600592 learned_addr learned my addr 172.21.15.57:0/3922600592 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:40:17.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.535+0000 7f486133d640 1 --1- 172.21.15.57:0/3922600592 >> v1:172.21.15.57:6789/0 conn(0x7f485c06b760 0x7f485c15e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:33130/0 (socket says 172.21.15.57:33130) 2023-10-30T14:40:17.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.535+0000 7f4861b3e640 1 --2- 172.21.15.57:0/3922600592 >> v2:172.21.15.88:3300/0 conn(0x7f485c06c420 0x7f485c157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:17.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.535+0000 7f4860b3c640 1 -- 172.21.15.57:0/3922600592 >> v2:172.21.15.88:3300/0 conn(0x7f485c06c420 msgr2=0x7f485c157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:40:17.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.535+0000 7f4860b3c640 1 --2- 172.21.15.57:0/3922600592 >> v2:172.21.15.88:3300/0 conn(0x7f485c06c420 0x7f485c157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:17.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.535+0000 7f4860b3c640 1 -- 172.21.15.57:0/3922600592 >> v1:172.21.15.57:6789/0 conn(0x7f485c06b760 legacy=0x7f485c15e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:17.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.535+0000 7f4860b3c640 1 -- 172.21.15.57:0/3922600592 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c0711f0 con 0x7f485c06bb40 2023-10-30T14:40:17.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.535+0000 7f4853fff640 1 -- 172.21.15.57:0/3922600592 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113872115 0 0) 0x7f485c071e00 con 0x7f485c06b760 2023-10-30T14:40:17.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.539+0000 7f4860b3c640 1 --2- 172.21.15.57:0/3922600592 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f485c06bb40 0x7f485c157a20 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f485c15b7a0 tx=0x7f484c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f23306ef9f505a4d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:17.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.539+0000 7f4853fff640 1 -- 172.21.15.57:0/3922600592 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c0aa430 con 0x7f485c06bb40 2023-10-30T14:40:17.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.539+0000 7f4853fff640 1 -- 172.21.15.57:0/3922600592 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484c0aa590 con 0x7f485c06bb40 2023-10-30T14:40:17.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.539+0000 7f4853fff640 1 -- 172.21.15.57:0/3922600592 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c0aa920 con 0x7f485c06bb40 2023-10-30T14:40:17.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.539+0000 7f48635a1640 1 -- 172.21.15.57:0/3922600592 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f485c06bb40 msgr2=0x7f485c157a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:17.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.539+0000 7f48635a1640 1 --2- 172.21.15.57:0/3922600592 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f485c06bb40 0x7f485c157a20 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f485c15b7a0 tx=0x7f484c01c4d0 comp rx=0 tx=0).stop 2023-10-30T14:40:17.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.539+0000 7f48635a1640 1 -- 172.21.15.57:0/3922600592 shutdown_connections 2023-10-30T14:40:17.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.539+0000 7f48635a1640 1 --2- 172.21.15.57:0/3922600592 >> v2:172.21.15.88:3300/0 conn(0x7f485c06c420 0x7f485c157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:17.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.539+0000 7f48635a1640 1 --2- 172.21.15.57:0/3922600592 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f485c06bb40 0x7f485c157a20 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:17.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.539+0000 7f48635a1640 1 -- 172.21.15.57:0/3922600592 >> 172.21.15.57:0/3922600592 conn(0x7f485c0cac70 msgr2=0x7f485c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:40:17.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.539+0000 7f48635a1640 1 -- 172.21.15.57:0/3922600592 shutdown_connections 2023-10-30T14:40:17.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.539+0000 7f48635a1640 1 -- 172.21.15.57:0/3922600592 wait complete. 2023-10-30T14:40:17.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f48635a1640 1 Processor -- start 2023-10-30T14:40:17.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f48635a1640 1 -- start start 2023-10-30T14:40:17.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f48635a1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f485c06b760 0x7f485c15ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:17.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f48635a1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f485c06bb40 0x7f485c1509c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:17.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f48635a1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f485c06c420 0x7f485c1482e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:17.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f48635a1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c1503d0 con 0x7f485c06c420 2023-10-30T14:40:17.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f48635a1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c15c9d0 con 0x7f485c06bb40 2023-10-30T14:40:17.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f48635a1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485c0c9690 con 0x7f485c06b760 2023-10-30T14:40:17.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f4861b3e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f485c06c420 0x7f485c1482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:17.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f4861b3e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f485c06c420 0x7f485c1482e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50658/0 (socket says 172.21.15.57:50658) 2023-10-30T14:40:17.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f4861b3e640 1 -- 172.21.15.57:0/152262877 learned_addr learned my addr 172.21.15.57:0/152262877 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:40:17.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f486133d640 1 --2- 172.21.15.57:0/152262877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f485c06b760 0x7f485c15ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:17.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f4860b3c640 1 --2- 172.21.15.57:0/152262877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f485c06bb40 0x7f485c1509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:17.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f4861b3e640 1 -- 172.21.15.57:0/152262877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f485c06b760 msgr2=0x7f485c15ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:17.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f4861b3e640 1 --2- 172.21.15.57:0/152262877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f485c06b760 0x7f485c15ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:17.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f4861b3e640 1 -- 172.21.15.57:0/152262877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f485c06bb40 msgr2=0x7f485c1509c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:17.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f4861b3e640 1 --2- 172.21.15.57:0/152262877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f485c06bb40 0x7f485c1509c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:17.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f4861b3e640 1 -- 172.21.15.57:0/152262877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c0094d0 con 0x7f485c06c420 2023-10-30T14:40:17.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f4861b3e640 1 --2- 172.21.15.57:0/152262877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f485c06c420 0x7f485c1482e0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f4854007e60 tx=0x7f485400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:17.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f4851ffb640 1 -- 172.21.15.57:0/152262877 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4854014420 con 0x7f485c06c420 2023-10-30T14:40:17.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f4851ffb640 1 -- 172.21.15.57:0/152262877 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48540145a0 con 0x7f485c06c420 2023-10-30T14:40:17.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f48635a1640 1 -- 172.21.15.57:0/152262877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485c15bae0 con 0x7f485c06c420 2023-10-30T14:40:17.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f4851ffb640 1 -- 172.21.15.57:0/152262877 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4854014950 con 0x7f485c06c420 2023-10-30T14:40:17.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.543+0000 7f48635a1640 1 -- 172.21.15.57:0/152262877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485c15c0a0 con 0x7f485c06c420 2023-10-30T14:40:17.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.547+0000 7f48635a1640 1 -- 172.21.15.57:0/152262877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f485c15cb40 con 0x7f485c06c420 2023-10-30T14:40:17.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.547+0000 7f4851ffb640 1 -- 172.21.15.57:0/152262877 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4854014ab0 con 0x7f485c06c420 2023-10-30T14:40:17.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.547+0000 7f4851ffb640 1 --2- 172.21.15.57:0/152262877 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f48400664f0 0x7f48400689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:17.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.547+0000 7f4851ffb640 1 -- 172.21.15.57:0/152262877 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7f4854084c90 con 0x7f485c06c420 2023-10-30T14:40:17.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.547+0000 7f486133d640 1 --2- 172.21.15.57:0/152262877 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f48400664f0 0x7f48400689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:17.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.547+0000 7f486133d640 1 --2- 172.21.15.57:0/152262877 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f48400664f0 0x7f48400689b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f4858000f70 tx=0x7f4858007b60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:17.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.551+0000 7f4851ffb640 1 -- 172.21.15.57:0/152262877 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4854051c50 con 0x7f485c06c420 2023-10-30T14:40:17.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.739+0000 7f48635a1640 1 -- 172.21.15.57:0/152262877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f485c04e170 con 0x7f485c06c420 2023-10-30T14:40:17.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.739+0000 7f4851ffb640 1 -- 172.21.15.57:0/152262877 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f4854055270 con 0x7f485c06c420 2023-10-30T14:40:17.745 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:40:17.745 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:40:07.263309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:40:17.746 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 144 2023-10-30T14:40:17.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.743+0000 7f48635a1640 1 -- 172.21.15.57:0/152262877 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f48400664f0 msgr2=0x7f48400689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:17.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.743+0000 7f48635a1640 1 --2- 172.21.15.57:0/152262877 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f48400664f0 0x7f48400689b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f4858000f70 tx=0x7f4858007b60 comp rx=0 tx=0).stop 2023-10-30T14:40:17.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.743+0000 7f48635a1640 1 -- 172.21.15.57:0/152262877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f485c06c420 msgr2=0x7f485c1482e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:17.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.743+0000 7f48635a1640 1 --2- 172.21.15.57:0/152262877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f485c06c420 0x7f485c1482e0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f4854007e60 tx=0x7f485400e4b0 comp rx=0 tx=0).stop 2023-10-30T14:40:17.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.743+0000 7f48635a1640 1 -- 172.21.15.57:0/152262877 shutdown_connections 2023-10-30T14:40:17.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.743+0000 7f48635a1640 1 --2- 172.21.15.57:0/152262877 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f48400664f0 0x7f48400689b0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:17.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.743+0000 7f48635a1640 1 --2- 172.21.15.57:0/152262877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f485c06c420 0x7f485c1482e0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:17.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.743+0000 7f48635a1640 1 --2- 172.21.15.57:0/152262877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f485c06bb40 0x7f485c1509c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:17.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.743+0000 7f48635a1640 1 --2- 172.21.15.57:0/152262877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f485c06b760 0x7f485c15ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:17.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.743+0000 7f48635a1640 1 -- 172.21.15.57:0/152262877 >> 172.21.15.57:0/152262877 conn(0x7f485c0cac70 msgr2=0x7f485c073450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:40:17.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.743+0000 7f48635a1640 1 -- 172.21.15.57:0/152262877 shutdown_connections 2023-10-30T14:40:17.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:17.743+0000 7f48635a1640 1 -- 172.21.15.57:0/152262877 wait complete. 2023-10-30T14:40:17.766 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-30T14:40:19.767 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:40:19.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f202b71d640 1 Processor -- start 2023-10-30T14:40:19.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f202b71d640 1 -- start start 2023-10-30T14:40:19.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f202b71d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2024153c80 0x7f202415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:19.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f202b71d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f20240c42a0 0x7f20240ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:19.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f202b71d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2024159f20 con 0x7f20240c4a10 2023-10-30T14:40:19.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f202b71d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f202415dec0 con 0x7f20240c42a0 2023-10-30T14:40:19.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f202b71d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20241599a0 con 0x7f2024153c80 2023-10-30T14:40:19.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f2028cb8640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f20240c4a10 0x7f202415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:33142/0 (socket says 172.21.15.57:33142) 2023-10-30T14:40:19.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f2028cb8640 1 -- 172.21.15.57:0/2627920125 learned_addr learned my addr 172.21.15.57:0/2627920125 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:40:19.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f2029cba640 1 --2- 172.21.15.57:0/2627920125 >> v2:172.21.15.88:3300/0 conn(0x7f20240c42a0 0x7f20240ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:19.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f20294b9640 1 --2- 172.21.15.57:0/2627920125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2024153c80 0x7f202415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:19.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f201bfff640 1 -- 172.21.15.57:0/2627920125 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 391952878 0 0) 0x7f2024159f20 con 0x7f20240c4a10 2023-10-30T14:40:19.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f201bfff640 1 -- 172.21.15.57:0/2627920125 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2010003530 con 0x7f20240c4a10 2023-10-30T14:40:19.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f201bfff640 1 -- 172.21.15.57:0/2627920125 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4155827942 0 0) 0x7f2010003530 con 0x7f20240c4a10 2023-10-30T14:40:19.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f201bfff640 1 -- 172.21.15.57:0/2627920125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2024153c80 msgr2=0x7f202415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:40:19.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f201bfff640 1 --2- 172.21.15.57:0/2627920125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2024153c80 0x7f202415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:19.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f201bfff640 1 -- 172.21.15.57:0/2627920125 >> v2:172.21.15.88:3300/0 conn(0x7f20240c42a0 msgr2=0x7f20240ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:40:19.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f201bfff640 1 --2- 172.21.15.57:0/2627920125 >> v2:172.21.15.88:3300/0 conn(0x7f20240c42a0 0x7f20240ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:19.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f201bfff640 1 -- 172.21.15.57:0/2627920125 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202415d620 con 0x7f20240c4a10 2023-10-30T14:40:19.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f201bfff640 1 -- 172.21.15.57:0/2627920125 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f2014001680 con 0x7f20240c4a10 2023-10-30T14:40:19.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f201bfff640 1 -- 172.21.15.57:0/2627920125 >> v1:172.21.15.57:6789/0 conn(0x7f20240c4a10 legacy=0x7f202415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:19.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f201bfff640 1 --2- 172.21.15.57:0/2627920125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2024153c80 0x7f2010003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:19.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f201bfff640 1 --2- 172.21.15.57:0/2627920125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2010003f00 0x7f20100047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:19.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f201bfff640 1 --2- 172.21.15.57:0/2627920125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2010004d10 0x7f2010005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:19.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f201bfff640 1 -- 172.21.15.57:0/2627920125 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202415dec0 con 0x7f2010004d10 2023-10-30T14:40:19.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f201bfff640 1 -- 172.21.15.57:0/2627920125 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20241599a0 con 0x7f2010003f00 2023-10-30T14:40:19.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.911+0000 7f201bfff640 1 -- 172.21.15.57:0/2627920125 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f201000b9c0 con 0x7f2024153c80 2023-10-30T14:40:19.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.915+0000 7f2029cba640 1 --2- 172.21.15.57:0/2627920125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2010004d10 0x7f2010005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:19.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.915+0000 7f2028cb8640 1 --2- 172.21.15.57:0/2627920125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2010003f00 0x7f20100047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:19.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.915+0000 7f20294b9640 1 --2- 172.21.15.57:0/2627920125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2024153c80 0x7f2010003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:19.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.915+0000 7f2029cba640 1 -- 172.21.15.57:0/2627920125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2024153c80 msgr2=0x7f2010003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:40:19.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.915+0000 7f2029cba640 1 --2- 172.21.15.57:0/2627920125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2024153c80 0x7f2010003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:19.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.915+0000 7f2029cba640 1 -- 172.21.15.57:0/2627920125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2010003f00 msgr2=0x7f20100047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:40:19.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.915+0000 7f2029cba640 1 --2- 172.21.15.57:0/2627920125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2010003f00 0x7f20100047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:19.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.915+0000 7f2029cba640 1 -- 172.21.15.57:0/2627920125 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2010003530 con 0x7f2010004d10 2023-10-30T14:40:19.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.915+0000 7f2028cb8640 1 --2- 172.21.15.57:0/2627920125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2010003f00 0x7f20100047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:40:19.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.915+0000 7f2029cba640 1 --2- 172.21.15.57:0/2627920125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2010004d10 0x7f2010005190 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f201c009fc0 tx=0x7f201c01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6392b4c2dae9761a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:19.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.915+0000 7f201bfff640 1 -- 172.21.15.57:0/2627920125 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c0a5cb0 con 0x7f2010004d10 2023-10-30T14:40:19.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.915+0000 7f201bfff640 1 -- 172.21.15.57:0/2627920125 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f201c00f040 con 0x7f2010004d10 2023-10-30T14:40:19.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.915+0000 7f201bfff640 1 -- 172.21.15.57:0/2627920125 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c0ad680 con 0x7f2010004d10 2023-10-30T14:40:19.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.915+0000 7f202b71d640 1 -- 172.21.15.57:0/2627920125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2010004d10 msgr2=0x7f2010005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:19.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.915+0000 7f202b71d640 1 --2- 172.21.15.57:0/2627920125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2010004d10 0x7f2010005190 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f201c009fc0 tx=0x7f201c01cca0 comp rx=0 tx=0).stop 2023-10-30T14:40:19.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.915+0000 7f2029cba640 1 -- 172.21.15.57:0/2627920125 reap_dead start 2023-10-30T14:40:19.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f202b71d640 1 -- 172.21.15.57:0/2627920125 shutdown_connections 2023-10-30T14:40:19.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f202b71d640 1 -- 172.21.15.57:0/2627920125 >> 172.21.15.57:0/2627920125 conn(0x7f20240715c0 msgr2=0x7f20240719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:40:19.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f202b71d640 1 -- 172.21.15.57:0/2627920125 shutdown_connections 2023-10-30T14:40:19.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f202b71d640 1 -- 172.21.15.57:0/2627920125 wait complete. 2023-10-30T14:40:19.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f202b71d640 1 Processor -- start 2023-10-30T14:40:19.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f202b71d640 1 -- start start 2023-10-30T14:40:19.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f202b71d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2010004d10 0x7f2024156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:19.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f202b71d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20240c5b10 0x7f20241541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:19.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f202b71d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20240c50c0 0x7f20241ee5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:19.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f202b71d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2024155790 con 0x7f20240c50c0 2023-10-30T14:40:19.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f202b71d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20241546f0 con 0x7f2010004d10 2023-10-30T14:40:19.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f202b71d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20241f2440 con 0x7f20240c5b10 2023-10-30T14:40:19.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f2029cba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20240c50c0 0x7f20241ee5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:19.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f2029cba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20240c50c0 0x7f20241ee5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50664/0 (socket says 172.21.15.57:50664) 2023-10-30T14:40:19.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f2029cba640 1 -- 172.21.15.57:0/3048046297 learned_addr learned my addr 172.21.15.57:0/3048046297 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:40:19.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f20294b9640 1 --2- 172.21.15.57:0/3048046297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2010004d10 0x7f2024156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:19.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f2028cb8640 1 --2- 172.21.15.57:0/3048046297 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20240c5b10 0x7f20241541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:19.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f2029cba640 1 -- 172.21.15.57:0/3048046297 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20240c5b10 msgr2=0x7f20241541b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:19.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f2029cba640 1 --2- 172.21.15.57:0/3048046297 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20240c5b10 0x7f20241541b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:19.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f2029cba640 1 -- 172.21.15.57:0/3048046297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2010004d10 msgr2=0x7f2024156d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:19.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f2029cba640 1 --2- 172.21.15.57:0/3048046297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2010004d10 0x7f2024156d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:19.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f2029cba640 1 -- 172.21.15.57:0/3048046297 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2024154910 con 0x7f20240c50c0 2023-10-30T14:40:19.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.919+0000 7f2028cb8640 1 --2- 172.21.15.57:0/3048046297 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20240c5b10 0x7f20241541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:40:19.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.923+0000 7f2029cba640 1 --2- 172.21.15.57:0/3048046297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20240c50c0 0x7f20241ee5a0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f201c0b0040 tx=0x7f201c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:19.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.923+0000 7f2019ffb640 1 -- 172.21.15.57:0/3048046297 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c0039a0 con 0x7f20240c50c0 2023-10-30T14:40:19.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.923+0000 7f2019ffb640 1 -- 172.21.15.57:0/3048046297 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f201c003b00 con 0x7f20240c50c0 2023-10-30T14:40:19.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.923+0000 7f2019ffb640 1 -- 172.21.15.57:0/3048046297 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c003e70 con 0x7f20240c50c0 2023-10-30T14:40:19.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.923+0000 7f202b71d640 1 -- 172.21.15.57:0/3048046297 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2024154bf0 con 0x7f20240c50c0 2023-10-30T14:40:19.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.923+0000 7f202b71d640 1 -- 172.21.15.57:0/3048046297 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20241f20d0 con 0x7f20240c50c0 2023-10-30T14:40:19.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.923+0000 7f202b71d640 1 -- 172.21.15.57:0/3048046297 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f202404e170 con 0x7f20240c50c0 2023-10-30T14:40:19.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.923+0000 7f2019ffb640 1 -- 172.21.15.57:0/3048046297 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f201c0b4040 con 0x7f20240c50c0 2023-10-30T14:40:19.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.923+0000 7f2019ffb640 1 --2- 172.21.15.57:0/3048046297 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20080664d0 0x7f2008068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:40:19.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.923+0000 7f2019ffb640 1 -- 172.21.15.57:0/3048046297 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7f201c11e290 con 0x7f20240c50c0 2023-10-30T14:40:19.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.923+0000 7f20294b9640 1 --2- 172.21.15.57:0/3048046297 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20080664d0 0x7f2008068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:40:19.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.927+0000 7f20294b9640 1 --2- 172.21.15.57:0/3048046297 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20080664d0 0x7f2008068990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2020009be0 tx=0x7f2020009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:40:19.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:19.927+0000 7f2019ffb640 1 -- 172.21.15.57:0/3048046297 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f201c0eb160 con 0x7f20240c50c0 2023-10-30T14:40:20.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:20.123+0000 7f202b71d640 1 -- 172.21.15.57:0/3048046297 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20241f2580 con 0x7f20240c50c0 2023-10-30T14:40:20.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:20.123+0000 7f2019ffb640 1 -- 172.21.15.57:0/3048046297 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+6528 (secure 0 0 0) 0x7f201c0ee780 con 0x7f20240c50c0 2023-10-30T14:40:20.128 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:40:20.128 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6356,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:40:07.263309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:40:20.128 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 145 2023-10-30T14:40:20.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:20.123+0000 7f202b71d640 1 -- 172.21.15.57:0/3048046297 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20080664d0 msgr2=0x7f2008068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:20.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:20.123+0000 7f202b71d640 1 --2- 172.21.15.57:0/3048046297 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20080664d0 0x7f2008068990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2020009be0 tx=0x7f2020009190 comp rx=0 tx=0).stop 2023-10-30T14:40:20.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:20.123+0000 7f202b71d640 1 -- 172.21.15.57:0/3048046297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20240c50c0 msgr2=0x7f20241ee5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:40:20.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:20.123+0000 7f202b71d640 1 --2- 172.21.15.57:0/3048046297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20240c50c0 0x7f20241ee5a0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f201c0b0040 tx=0x7f201c009750 comp rx=0 tx=0).stop 2023-10-30T14:40:20.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:20.127+0000 7f202b71d640 1 -- 172.21.15.57:0/3048046297 shutdown_connections 2023-10-30T14:40:20.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:20.127+0000 7f202b71d640 1 --2- 172.21.15.57:0/3048046297 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20080664d0 0x7f2008068990 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:20.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:20.127+0000 7f202b71d640 1 --2- 172.21.15.57:0/3048046297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20240c50c0 0x7f20241ee5a0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:20.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:20.127+0000 7f202b71d640 1 --2- 172.21.15.57:0/3048046297 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20240c5b10 0x7f20241541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:20.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:20.127+0000 7f202b71d640 1 --2- 172.21.15.57:0/3048046297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2010004d10 0x7f2024156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:40:20.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:20.127+0000 7f202b71d640 1 -- 172.21.15.57:0/3048046297 >> 172.21.15.57:0/3048046297 conn(0x7f20240715c0 msgr2=0x7f2024148c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:40:20.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:20.127+0000 7f202b71d640 1 -- 172.21.15.57:0/3048046297 shutdown_connections 2023-10-30T14:40:20.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:40:20.127+0000 7f202b71d640 1 -- 172.21.15.57:0/3048046297 wait complete. 2023-10-30T14:40:20.148 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-30T14:40:20.148 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 118 secs before thrashing 2023-10-30T14:40:21.118 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 1% (3/253) Updating files: 1% (5/253) Updating files: 2% (6/253) Updating files: 3% (8/253) Updating files: 4% (11/253) Updating files: 5% (13/253) Updating files: 6% (16/253) Updating files: 7% (18/253) Updating files: 8% (21/253) Updating files: 9% (23/253) Updating files: 10% (26/253) Updating files: 11% (28/253) Updating files: 12% (31/253) Updating files: 13% (33/253) Updating files: 14% (36/253) Updating files: 15% (38/253) Updating files: 16% (41/253) Updating files: 16% (43/253) Updating files: 17% (44/253) Updating files: 18% (46/253) Updating files: 19% (49/253) Updating files: 20% (51/253) Updating files: 21% (54/253) Updating files: 22% (56/253) Updating files: 23% (59/253) Updating files: 24% (61/253) Updating files: 25% (64/253) Updating files: 26% (66/253) Updating files: 27% (69/253) Updating files: 28% (71/253) Updating files: 29% (74/253) Updating files: 30% (76/253) Updating files: 31% (79/253) Updating files: 32% (81/253) Updating files: 33% (84/253) Updating files: 34% (87/253) Updating files: 35% (89/253) Updating files: 36% (92/253) Updating files: 37% (94/253) Updating files: 38% (97/253) Updating files: 39% (99/253) Updating files: 40% (102/253) Updating files: 41% (104/253) Updating files: 42% (107/253) Updating files: 43% (109/253) Updating files: 44% (112/253) Updating files: 45% (114/253) Updating files: 46% (117/253) Updating files: 47% (119/253) Updating files: 48% (122/253) Updating files: 49% (124/253) Updating files: 50% (127/253) Updating files: 51% (130/253) Updating files: 52% (132/253) Updating files: 53% (135/253) Updating files: 54% (137/253) Updating files: 55% (140/253) Updating files: 56% (142/253) Updating files: 57% (145/253) Updating files: 58% (147/253) Updating files: 59% (150/253) Updating files: 60% (152/253) Updating files: 61% (155/253) Updating files: 62% (157/253) Updating files: 63% (160/253) Updating files: 64% (162/253) Updating files: 65% (165/253) Updating files: 66% (167/253) Updating files: 67% (170/253) Updating files: 68% (173/253) Updating files: 69% (175/253) Updating files: 70% (178/253) Updating files: 71% (180/253) Updating files: 72% (183/253) Updating files: 73% (185/253) Updating files: 74% (188/253) Updating files: 75% (190/253) Updating files: 76% (193/253) Updating files: 77% (195/253) Updating files: 78% (198/253) Updating files: 79% (200/253) Updating files: 80% (203/253) Updating files: 81% (205/253) Updating files: 82% (208/253) Updating files: 83% (210/253) Updating files: 84% (213/253) Updating files: 85% (216/253) Updating files: 86% (218/253) Updating files: 87% (221/253) Updating files: 88% (223/253) Updating files: 89% (226/253) Updating files: 90% (228/253) Updating files: 91% (231/253) Updating files: 92% (233/253) Updating files: 92% (234/253) Updating files: 93% (236/253) Updating files: 94% (238/253) Updating files: 95% (241/253) Updating files: 96% (243/253) Updating files: 97% (246/253) Updating files: 98% (248/253) Updating files: 99% (251/253) Updating files: 100% (253/253) Updating files: 100% (253/253), done. 2023-10-30T14:40:21.739 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 793034c62c8 v0.27 2023-10-30T14:40:21.744 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.27 2023-10-30T14:40:21.780 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 28 -eq 48 ] 2023-10-30T14:40:21.780 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.28 2023-10-30T14:40:21.781 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.28 2023-10-30T14:40:21.781 INFO:tasks.workunit.client.0.smithi057.stdout:v0.28 2023-10-30T14:40:21.781 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.28 2023-10-30T14:40:28.785 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 15% (36/227) Updating files: 16% (37/227) Updating files: 17% (39/227) Updating files: 18% (41/227) Updating files: 19% (44/227) Updating files: 20% (46/227) Updating files: 21% (48/227) Updating files: 22% (50/227) Updating files: 23% (53/227) Updating files: 24% (55/227) Updating files: 25% (57/227) Updating files: 26% (60/227) Updating files: 27% (62/227) Updating files: 28% (64/227) Updating files: 29% (66/227) Updating files: 30% (69/227) Updating files: 31% (71/227) Updating files: 32% (73/227) Updating files: 33% (75/227) Updating files: 34% (78/227) Updating files: 35% (80/227) Updating files: 36% (82/227) Updating files: 37% (84/227) Updating files: 38% (87/227) Updating files: 39% (89/227) Updating files: 40% (91/227) Updating files: 41% (94/227) Updating files: 42% (96/227) Updating files: 43% (98/227) Updating files: 44% (100/227) Updating files: 45% (103/227) Updating files: 46% (105/227) Updating files: 47% (107/227) Updating files: 48% (109/227) Updating files: 49% (112/227) Updating files: 50% (114/227) Updating files: 51% (116/227) Updating files: 52% (119/227) Updating files: 52% (120/227) Updating files: 53% (121/227) Updating files: 54% (123/227) Updating files: 55% (125/227) Updating files: 55% (126/227) Updating files: 56% (128/227) Updating files: 57% (130/227) Updating files: 58% (132/227) Updating files: 59% (134/227) Updating files: 60% (137/227) Updating files: 61% (139/227) Updating files: 62% (141/227) Updating files: 63% (144/227) Updating files: 64% (146/227) Updating files: 65% (148/227) Updating files: 66% (150/227) Updating files: 67% (153/227) Updating files: 68% (155/227) Updating files: 69% (157/227) Updating files: 70% (159/227) Updating files: 71% (162/227) Updating files: 72% (164/227) Updating files: 73% (166/227) Updating files: 74% (168/227) Updating files: 75% (171/227) Updating files: 76% (173/227) Updating files: 77% (175/227) Updating files: 78% (178/227) Updating files: 79% (180/227) Updating files: 80% (182/227) Updating files: 81% (184/227) Updating files: 82% (187/227) Updating files: 83% (189/227) Updating files: 84% (191/227) Updating files: 85% (193/227) Updating files: 86% (196/227) Updating files: 87% (198/227) Updating files: 88% (200/227) Updating files: 89% (203/227) Updating files: 90% (205/227) Updating files: 91% (207/227) Updating files: 92% (209/227) Updating files: 93% (212/227) Updating files: 94% (214/227) Updating files: 95% (216/227) Updating files: 96% (218/227) Updating files: 97% (221/227) Updating files: 98% (223/227) Updating files: 99% (225/227) Updating files: 100% (227/227) Updating files: 100% (227/227), done. 2023-10-30T14:40:29.291 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 071881d7e55 v0.28 2023-10-30T14:40:29.296 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.28 2023-10-30T14:40:29.325 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 29 -eq 48 ] 2023-10-30T14:40:29.326 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.29 2023-10-30T14:40:29.326 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.29 2023-10-30T14:40:29.327 INFO:tasks.workunit.client.0.smithi057.stdout:v0.29 2023-10-30T14:40:29.327 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.29 2023-10-30T14:40:33.299 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 15% (24/159) Updating files: 16% (26/159) Updating files: 17% (28/159) Updating files: 18% (29/159) Updating files: 19% (31/159) Updating files: 20% (32/159) Updating files: 21% (34/159) Updating files: 22% (35/159) Updating files: 23% (37/159) Updating files: 24% (39/159) Updating files: 25% (40/159) Updating files: 26% (42/159) Updating files: 27% (43/159) Updating files: 28% (45/159) Updating files: 29% (47/159) Updating files: 30% (48/159) Updating files: 31% (50/159) Updating files: 32% (51/159) Updating files: 33% (53/159) Updating files: 34% (55/159) Updating files: 35% (56/159) Updating files: 36% (58/159) Updating files: 37% (59/159) Updating files: 38% (61/159) Updating files: 39% (63/159) Updating files: 40% (64/159) Updating files: 41% (66/159) Updating files: 42% (67/159) Updating files: 43% (69/159) Updating files: 44% (70/159) Updating files: 45% (72/159) Updating files: 46% (74/159) Updating files: 47% (75/159) Updating files: 48% (77/159) Updating files: 49% (78/159) Updating files: 50% (80/159) Updating files: 51% (82/159) Updating files: 52% (83/159) Updating files: 53% (85/159) Updating files: 54% (86/159) Updating files: 55% (88/159) Updating files: 56% (90/159) Updating files: 57% (91/159) Updating files: 58% (93/159) Updating files: 59% (94/159) Updating files: 60% (96/159) Updating files: 61% (97/159) Updating files: 61% (98/159) Updating files: 62% (99/159) Updating files: 63% (101/159) Updating files: 64% (102/159) Updating files: 65% (104/159) Updating files: 66% (105/159) Updating files: 67% (107/159) Updating files: 68% (109/159) Updating files: 69% (110/159) Updating files: 70% (112/159) Updating files: 71% (113/159) Updating files: 72% (115/159) Updating files: 73% (117/159) Updating files: 74% (118/159) Updating files: 75% (120/159) Updating files: 76% (121/159) Updating files: 77% (123/159) Updating files: 78% (125/159) Updating files: 79% (126/159) Updating files: 80% (128/159) Updating files: 81% (129/159) Updating files: 82% (131/159) Updating files: 83% (132/159) Updating files: 84% (134/159) Updating files: 85% (136/159) Updating files: 86% (137/159) Updating files: 87% (139/159) Updating files: 88% (140/159) Updating files: 89% (142/159) Updating files: 90% (144/159) Updating files: 91% (145/159) Updating files: 92% (147/159) Updating files: 93% (148/159) Updating files: 94% (150/159) Updating files: 95% (152/159) Updating files: 96% (153/159) Updating files: 97% (155/159) Updating files: 98% (156/159) Updating files: 99% (158/159) Updating files: 100% (159/159) Updating files: 100% (159/159), done. 2023-10-30T14:40:33.336 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 8e69c39f699 v0.29 2023-10-30T14:40:33.342 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.29 2023-10-30T14:40:34.296 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 30 -eq 48 ] 2023-10-30T14:40:34.296 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.30 2023-10-30T14:40:34.296 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.30 2023-10-30T14:40:34.297 INFO:tasks.workunit.client.0.smithi057.stdout:v0.30 2023-10-30T14:40:34.297 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.30 2023-10-30T14:40:38.094 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:40:38.098 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:40:38.706 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 51% (147/285) Updating files: 52% (149/285) Updating files: 53% (152/285) Updating files: 54% (154/285) Updating files: 55% (157/285) Updating files: 56% (160/285) Updating files: 57% (163/285) Updating files: 58% (166/285) Updating files: 59% (169/285) Updating files: 60% (171/285) Updating files: 61% (174/285) Updating files: 62% (177/285) Updating files: 63% (180/285) Updating files: 64% (183/285) Updating files: 65% (186/285) Updating files: 66% (189/285) Updating files: 67% (191/285) Updating files: 68% (194/285) Updating files: 69% (197/285) Updating files: 70% (200/285) Updating files: 70% (202/285) Updating files: 71% (203/285) Updating files: 72% (206/285) Updating files: 72% (208/285) Updating files: 73% (209/285) Updating files: 74% (211/285) Updating files: 75% (214/285) Updating files: 76% (217/285) Updating files: 77% (220/285) Updating files: 78% (223/285) Updating files: 79% (226/285) Updating files: 80% (228/285) Updating files: 81% (231/285) Updating files: 82% (234/285) Updating files: 83% (237/285) Updating files: 84% (240/285) Updating files: 85% (243/285) Updating files: 86% (246/285) Updating files: 87% (248/285) Updating files: 88% (251/285) Updating files: 89% (254/285) Updating files: 90% (257/285) Updating files: 91% (260/285) Updating files: 92% (263/285) Updating files: 93% (266/285) Updating files: 94% (268/285) Updating files: 95% (271/285) Updating files: 96% (274/285) Updating files: 97% (277/285) Updating files: 98% (280/285) Updating files: 99% (283/285) Updating files: 100% (285/285) Updating files: 100% (285/285), done. 2023-10-30T14:40:39.031 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 64b1b2c70f0 v0.30 2023-10-30T14:40:39.036 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.30 2023-10-30T14:40:39.159 INFO:tasks.workunit.client.0.smithi057.stdout:v0.31 2023-10-30T14:40:39.159 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 31 -eq 48 ] 2023-10-30T14:40:39.159 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.31 2023-10-30T14:40:39.160 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.31 2023-10-30T14:40:39.160 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.31 2023-10-30T14:40:44.679 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 9% (38/398) Updating files: 10% (40/398) Updating files: 11% (44/398) Updating files: 12% (48/398) Updating files: 13% (52/398) Updating files: 14% (56/398) Updating files: 15% (60/398) Updating files: 16% (64/398) Updating files: 17% (68/398) Updating files: 18% (72/398) Updating files: 19% (76/398) Updating files: 20% (80/398) Updating files: 21% (84/398) Updating files: 22% (88/398) Updating files: 23% (92/398) Updating files: 24% (96/398) Updating files: 25% (100/398) Updating files: 26% (104/398) Updating files: 27% (108/398) Updating files: 28% (112/398) Updating files: 29% (116/398) Updating files: 30% (120/398) Updating files: 31% (124/398) Updating files: 32% (128/398) Updating files: 33% (132/398) Updating files: 34% (136/398) Updating files: 35% (140/398) Updating files: 36% (144/398) Updating files: 37% (148/398) Updating files: 38% (152/398) Updating files: 39% (156/398) Updating files: 40% (160/398) Updating files: 41% (164/398) Updating files: 42% (168/398) Updating files: 43% (172/398) Updating files: 44% (176/398) Updating files: 45% (180/398) Updating files: 46% (184/398) Updating files: 47% (188/398) Updating files: 48% (192/398) Updating files: 49% (196/398) Updating files: 50% (199/398) Updating files: 51% (203/398) Updating files: 52% (207/398) Updating files: 53% (211/398) Updating files: 54% (215/398) Updating files: 55% (219/398) Updating files: 56% (223/398) Updating files: 57% (227/398) Updating files: 58% (231/398) Updating files: 59% (235/398) Updating files: 60% (239/398) Updating files: 61% (243/398) Updating files: 62% (247/398) Updating files: 62% (250/398) Updating files: 63% (251/398) Updating files: 64% (255/398) Updating files: 65% (259/398) Updating files: 66% (263/398) Updating files: 67% (267/398) Updating files: 68% (271/398) Updating files: 69% (275/398) Updating files: 70% (279/398) Updating files: 70% (282/398) Updating files: 71% (283/398) Updating files: 72% (287/398) Updating files: 73% (291/398) Updating files: 74% (295/398) Updating files: 75% (299/398) Updating files: 76% (303/398) Updating files: 77% (307/398) Updating files: 78% (311/398) Updating files: 79% (315/398) Updating files: 80% (319/398) Updating files: 81% (323/398) Updating files: 82% (327/398) Updating files: 83% (331/398) Updating files: 84% (335/398) Updating files: 85% (339/398) Updating files: 86% (343/398) Updating files: 87% (347/398) Updating files: 88% (351/398) Updating files: 89% (355/398) Updating files: 90% (359/398) Updating files: 91% (363/398) Updating files: 92% (367/398) Updating files: 93% (371/398) Updating files: 94% (375/398) Updating files: 95% (379/398) Updating files: 96% (383/398) Updating files: 97% (387/398) Updating files: 98% (391/398) Updating files: 98% (393/398) Updating files: 99% (395/398) Updating files: 100% (398/398) Updating files: 100% (398/398), done. 2023-10-30T14:40:44.711 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 9019c6ce640 v0.31 2023-10-30T14:40:44.718 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.31 2023-10-30T14:40:44.784 INFO:tasks.workunit.client.0.smithi057.stdout:v0.32 2023-10-30T14:40:44.784 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 32 -eq 48 ] 2023-10-30T14:40:44.785 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.32 2023-10-30T14:40:44.785 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.32 2023-10-30T14:40:44.785 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.32 2023-10-30T14:41:04.275 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 33% (66/195) Updating files: 34% (67/195) Updating files: 35% (69/195) Updating files: 36% (71/195) Updating files: 37% (73/195) Updating files: 38% (75/195) Updating files: 39% (77/195) Updating files: 40% (78/195) Updating files: 41% (80/195) Updating files: 42% (82/195) Updating files: 43% (84/195) Updating files: 44% (86/195) Updating files: 45% (88/195) Updating files: 46% (90/195) Updating files: 47% (92/195) Updating files: 48% (94/195) Updating files: 49% (96/195) Updating files: 50% (98/195) Updating files: 51% (100/195) Updating files: 52% (102/195) Updating files: 53% (104/195) Updating files: 54% (106/195) Updating files: 55% (108/195) Updating files: 56% (110/195) Updating files: 57% (112/195) Updating files: 58% (114/195) Updating files: 59% (116/195) Updating files: 60% (117/195) Updating files: 61% (119/195) Updating files: 62% (121/195) Updating files: 63% (123/195) Updating files: 64% (125/195) Updating files: 65% (127/195) Updating files: 66% (129/195) Updating files: 67% (131/195) Updating files: 68% (133/195) Updating files: 69% (135/195) Updating files: 70% (137/195) Updating files: 71% (139/195) Updating files: 72% (141/195) Updating files: 73% (143/195) Updating files: 73% (144/195) Updating files: 74% (145/195) Updating files: 75% (147/195) Updating files: 76% (149/195) Updating files: 77% (151/195) Updating files: 78% (153/195) Updating files: 79% (155/195) Updating files: 80% (156/195) Updating files: 81% (158/195) Updating files: 82% (160/195) Updating files: 83% (162/195) Updating files: 84% (164/195) Updating files: 85% (166/195) Updating files: 86% (168/195) Updating files: 87% (170/195) Updating files: 88% (172/195) Updating files: 89% (174/195) Updating files: 90% (176/195) Updating files: 91% (178/195) Updating files: 92% (180/195) Updating files: 93% (182/195) Updating files: 94% (184/195) Updating files: 95% (186/195) Updating files: 96% (188/195) Updating files: 97% (190/195) Updating files: 98% (192/195) Updating files: 99% (194/195) Updating files: 100% (195/195) Updating files: 100% (195/195), done. 2023-10-30T14:41:04.334 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at c08d08baa6a v0.32 2023-10-30T14:41:04.338 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.32 2023-10-30T14:41:04.361 INFO:tasks.workunit.client.0.smithi057.stdout:v0.33 2023-10-30T14:41:04.362 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 33 -eq 48 ] 2023-10-30T14:41:04.362 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.33 2023-10-30T14:41:04.362 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.33 2023-10-30T14:41:04.362 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.33 2023-10-30T14:41:08.206 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:41:08.210 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:41:10.118 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 340b4348bf6 v0.33 2023-10-30T14:41:10.121 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.33 2023-10-30T14:41:10.208 INFO:tasks.workunit.client.0.smithi057.stdout:v0.34 2023-10-30T14:41:10.208 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 34 -eq 48 ] 2023-10-30T14:41:10.208 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.34 2023-10-30T14:41:10.208 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.34 2023-10-30T14:41:10.208 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.34 2023-10-30T14:41:20.163 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 12% (16/129) Updating files: 13% (17/129) Updating files: 14% (19/129) Updating files: 15% (20/129) Updating files: 16% (21/129) Updating files: 17% (22/129) Updating files: 18% (24/129) Updating files: 19% (25/129) Updating files: 20% (26/129) Updating files: 21% (28/129) Updating files: 22% (29/129) Updating files: 23% (30/129) Updating files: 24% (31/129) Updating files: 25% (33/129) Updating files: 26% (34/129) Updating files: 27% (35/129) Updating files: 28% (37/129) Updating files: 29% (38/129) Updating files: 30% (39/129) Updating files: 31% (40/129) Updating files: 32% (42/129) Updating files: 33% (43/129) Updating files: 34% (44/129) Updating files: 35% (46/129) Updating files: 36% (47/129) Updating files: 37% (48/129) Updating files: 38% (50/129) Updating files: 39% (51/129) Updating files: 40% (52/129) Updating files: 41% (53/129) Updating files: 42% (55/129) Updating files: 43% (56/129) Updating files: 44% (57/129) Updating files: 45% (59/129) Updating files: 46% (60/129) Updating files: 47% (61/129) Updating files: 48% (62/129) Updating files: 49% (64/129) Updating files: 50% (65/129) Updating files: 51% (66/129) Updating files: 52% (68/129) Updating files: 53% (69/129) Updating files: 54% (70/129) Updating files: 55% (71/129) Updating files: 56% (73/129) Updating files: 57% (74/129) Updating files: 58% (75/129) Updating files: 59% (77/129) Updating files: 60% (78/129) Updating files: 61% (79/129) Updating files: 62% (80/129) Updating files: 63% (82/129) Updating files: 64% (83/129) Updating files: 65% (84/129) Updating files: 66% (86/129) Updating files: 67% (87/129) Updating files: 68% (88/129) Updating files: 69% (90/129) Updating files: 70% (91/129) Updating files: 71% (92/129) Updating files: 72% (93/129) Updating files: 73% (95/129) Updating files: 74% (96/129) Updating files: 75% (97/129) Updating files: 76% (99/129) Updating files: 77% (100/129) Updating files: 78% (101/129) Updating files: 79% (102/129) Updating files: 80% (104/129) Updating files: 81% (105/129) Updating files: 82% (106/129) Updating files: 82% (107/129) Updating files: 83% (108/129) Updating files: 84% (109/129) Updating files: 85% (110/129) Updating files: 86% (111/129) Updating files: 87% (113/129) Updating files: 88% (114/129) Updating files: 89% (115/129) Updating files: 90% (117/129) Updating files: 91% (118/129) Updating files: 92% (119/129) Updating files: 93% (120/129) Updating files: 94% (122/129) Updating files: 95% (123/129) Updating files: 96% (124/129) Updating files: 97% (126/129) Updating files: 98% (127/129) Updating files: 99% (128/129) Updating files: 100% (129/129) Updating files: 100% (129/129), done. 2023-10-30T14:41:20.201 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 2f039eeeb74 v0.34 2023-10-30T14:41:20.204 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.34 2023-10-30T14:41:20.283 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 35 -eq 48 ] 2023-10-30T14:41:20.283 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.35 2023-10-30T14:41:20.283 INFO:tasks.workunit.client.0.smithi057.stdout:v0.35 2023-10-30T14:41:20.284 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.35 2023-10-30T14:41:20.284 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.35 2023-10-30T14:41:31.182 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 19% (66/334) Updating files: 20% (67/334) Updating files: 21% (71/334) Updating files: 22% (74/334) Updating files: 23% (77/334) Updating files: 24% (81/334) Updating files: 25% (84/334) Updating files: 26% (87/334) Updating files: 27% (91/334) Updating files: 28% (94/334) Updating files: 29% (97/334) Updating files: 30% (101/334) Updating files: 30% (103/334) Updating files: 31% (104/334) Updating files: 32% (107/334) Updating files: 33% (111/334) Updating files: 33% (113/334) Updating files: 34% (114/334) Updating files: 35% (117/334) Updating files: 36% (121/334) Updating files: 37% (124/334) Updating files: 38% (127/334) Updating files: 38% (130/334) Updating files: 39% (131/334) Updating files: 40% (134/334) Updating files: 41% (137/334) Updating files: 42% (141/334) Updating files: 43% (144/334) Updating files: 44% (147/334) Updating files: 45% (151/334) Updating files: 46% (154/334) Updating files: 47% (157/334) Updating files: 48% (161/334) Updating files: 49% (164/334) Updating files: 50% (167/334) Updating files: 51% (171/334) Updating files: 52% (174/334) Updating files: 53% (178/334) Updating files: 54% (181/334) Updating files: 55% (184/334) Updating files: 56% (188/334) Updating files: 57% (191/334) Updating files: 58% (194/334) Updating files: 59% (198/334) Updating files: 60% (201/334) Updating files: 61% (204/334) Updating files: 62% (208/334) Updating files: 62% (209/334) Updating files: 63% (211/334) Updating files: 64% (214/334) Updating files: 65% (218/334) Updating files: 66% (221/334) Updating files: 67% (224/334) Updating files: 68% (228/334) Updating files: 69% (231/334) Updating files: 70% (234/334) Updating files: 71% (238/334) Updating files: 72% (241/334) Updating files: 73% (244/334) Updating files: 74% (248/334) Updating files: 75% (251/334) Updating files: 75% (252/334) Updating files: 76% (254/334) Updating files: 77% (258/334) Updating files: 78% (261/334) Updating files: 79% (264/334) Updating files: 80% (268/334) Updating files: 81% (271/334) Updating files: 82% (274/334) Updating files: 83% (278/334) Updating files: 84% (281/334) Updating files: 85% (284/334) Updating files: 86% (288/334) Updating files: 87% (291/334) Updating files: 88% (294/334) Updating files: 89% (298/334) Updating files: 90% (301/334) Updating files: 91% (304/334) Updating files: 92% (308/334) Updating files: 93% (311/334) Updating files: 94% (314/334) Updating files: 95% (318/334) Updating files: 96% (321/334) Updating files: 97% (324/334) Updating files: 98% (328/334) Updating files: 99% (331/334) Updating files: 100% (334/334) Updating files: 100% (334/334), done. 2023-10-30T14:41:31.219 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 0afda378124 v0.35 2023-10-30T14:41:31.220 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.35 2023-10-30T14:41:31.289 INFO:tasks.workunit.client.0.smithi057.stdout:v0.36 2023-10-30T14:41:31.290 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 36 -eq 48 ] 2023-10-30T14:41:31.290 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.36 2023-10-30T14:41:31.290 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.36 2023-10-30T14:41:31.290 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.36 2023-10-30T14:41:38.284 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:41:38.288 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:41:57.851 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 11% (29/244) Updating files: 12% (30/244) Updating files: 13% (32/244) Updating files: 14% (35/244) Updating files: 15% (37/244) Updating files: 16% (40/244) Updating files: 17% (42/244) Updating files: 18% (44/244) Updating files: 19% (47/244) Updating files: 20% (49/244) Updating files: 21% (52/244) Updating files: 22% (54/244) Updating files: 23% (57/244) Updating files: 24% (59/244) Updating files: 25% (61/244) Updating files: 26% (64/244) Updating files: 27% (66/244) Updating files: 28% (69/244) Updating files: 29% (71/244) Updating files: 30% (74/244) Updating files: 31% (76/244) Updating files: 32% (79/244) Updating files: 33% (81/244) Updating files: 34% (83/244) Updating files: 35% (86/244) Updating files: 36% (88/244) Updating files: 37% (91/244) Updating files: 38% (93/244) Updating files: 39% (96/244) Updating files: 40% (98/244) Updating files: 41% (101/244) Updating files: 42% (103/244) Updating files: 43% (105/244) Updating files: 44% (108/244) Updating files: 45% (110/244) Updating files: 46% (113/244) Updating files: 47% (115/244) Updating files: 48% (118/244) Updating files: 49% (120/244) Updating files: 50% (122/244) Updating files: 51% (125/244) Updating files: 52% (127/244) Updating files: 53% (130/244) Updating files: 54% (132/244) Updating files: 55% (135/244) Updating files: 56% (137/244) Updating files: 57% (140/244) Updating files: 58% (142/244) Updating files: 59% (144/244) Updating files: 60% (147/244) Updating files: 61% (149/244) Updating files: 62% (152/244) Updating files: 63% (154/244) Updating files: 64% (157/244) Updating files: 65% (159/244) Updating files: 66% (162/244) Updating files: 67% (164/244) Updating files: 68% (166/244) Updating files: 69% (169/244) Updating files: 70% (171/244) Updating files: 71% (174/244) Updating files: 72% (176/244) Updating files: 73% (179/244) Updating files: 74% (181/244) Updating files: 75% (183/244) Updating files: 76% (186/244) Updating files: 77% (188/244) Updating files: 78% (191/244) Updating files: 79% (193/244) Updating files: 80% (196/244) Updating files: 81% (198/244) Updating files: 82% (201/244) Updating files: 83% (203/244) Updating files: 84% (205/244) Updating files: 84% (206/244) Updating files: 85% (208/244) Updating files: 86% (210/244) Updating files: 87% (213/244) Updating files: 88% (215/244) Updating files: 89% (218/244) Updating files: 90% (220/244) Updating files: 91% (223/244) Updating files: 92% (225/244) Updating files: 93% (227/244) Updating files: 94% (230/244) Updating files: 95% (232/244) Updating files: 96% (235/244) Updating files: 97% (237/244) Updating files: 98% (240/244) Updating files: 99% (242/244) Updating files: 100% (244/244) Updating files: 100% (244/244), done. 2023-10-30T14:41:57.979 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 877cacbbdbb v0.36 2023-10-30T14:41:57.981 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.36 2023-10-30T14:41:58.011 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 37 -eq 48 ] 2023-10-30T14:41:58.012 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.37 2023-10-30T14:41:58.012 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.37 2023-10-30T14:41:58.012 INFO:tasks.workunit.client.0.smithi057.stdout:v0.37 2023-10-30T14:41:58.013 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.37 2023-10-30T14:42:01.216 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 34% (63/181) Updating files: 35% (64/181) Updating files: 36% (66/181) Updating files: 37% (67/181) Updating files: 38% (69/181) Updating files: 39% (71/181) Updating files: 40% (73/181) Updating files: 41% (75/181) Updating files: 42% (77/181) Updating files: 43% (78/181) Updating files: 44% (80/181) Updating files: 45% (82/181) Updating files: 46% (84/181) Updating files: 47% (86/181) Updating files: 48% (87/181) Updating files: 49% (89/181) Updating files: 50% (91/181) Updating files: 51% (93/181) Updating files: 52% (95/181) Updating files: 53% (96/181) Updating files: 54% (98/181) Updating files: 55% (100/181) Updating files: 56% (102/181) Updating files: 57% (104/181) Updating files: 58% (105/181) Updating files: 59% (107/181) Updating files: 60% (109/181) Updating files: 61% (111/181) Updating files: 62% (113/181) Updating files: 63% (115/181) Updating files: 64% (116/181) Updating files: 65% (118/181) Updating files: 66% (120/181) Updating files: 67% (122/181) Updating files: 68% (124/181) Updating files: 69% (125/181) Updating files: 70% (127/181) Updating files: 71% (129/181) Updating files: 72% (131/181) Updating files: 73% (133/181) Updating files: 74% (134/181) Updating files: 75% (136/181) Updating files: 76% (138/181) Updating files: 77% (140/181) Updating files: 78% (142/181) Updating files: 79% (143/181) Updating files: 80% (145/181) Updating files: 81% (147/181) Updating files: 82% (149/181) Updating files: 83% (151/181) Updating files: 84% (153/181) Updating files: 85% (154/181) Updating files: 86% (156/181) Updating files: 87% (158/181) Updating files: 88% (160/181) Updating files: 89% (162/181) Updating files: 90% (163/181) Updating files: 91% (165/181) Updating files: 92% (167/181) Updating files: 93% (169/181) Updating files: 94% (171/181) Updating files: 95% (172/181) Updating files: 96% (174/181) Updating files: 97% (176/181) Updating files: 98% (178/181) Updating files: 99% (180/181) Updating files: 100% (181/181) Updating files: 100% (181/181), done. 2023-10-30T14:42:01.252 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at a6f3bbb744a v0.37 2023-10-30T14:42:01.255 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.37 2023-10-30T14:42:01.334 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 38 -eq 48 ] 2023-10-30T14:42:01.334 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.38 2023-10-30T14:42:01.334 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.38 2023-10-30T14:42:01.335 INFO:tasks.workunit.client.0.smithi057.stdout:v0.38 2023-10-30T14:42:01.335 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.38 2023-10-30T14:42:03.616 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at b600ec2ac7c v0.38 2023-10-30T14:42:03.619 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.38 2023-10-30T14:42:03.661 INFO:tasks.workunit.client.0.smithi057.stdout:v0.39 2023-10-30T14:42:03.662 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 39 -eq 48 ] 2023-10-30T14:42:03.662 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.39 2023-10-30T14:42:03.662 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.39 2023-10-30T14:42:03.662 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.39 2023-10-30T14:42:08.332 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:42:08.337 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:42:18.150 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:18.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.300+0000 7f68355f5640 1 Processor -- start 2023-10-30T14:42:18.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.300+0000 7f68355f5640 1 -- start start 2023-10-30T14:42:18.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.304+0000 7f68355f5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6830151c20 0x7f68300c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:18.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.304+0000 7f68355f5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68300cb8b0 0x7f683006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:18.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.304+0000 7f68355f5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68300c5030 con 0x7f6830151840 2023-10-30T14:42:18.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.304+0000 7f68355f5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6830157830 con 0x7f6830151c20 2023-10-30T14:42:18.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.304+0000 7f68355f5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6830157b10 con 0x7f68300cb8b0 2023-10-30T14:42:18.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.304+0000 7f6827fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6830151c20 0x7f68300c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:18.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.304+0000 7f682effd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f6830151840 0x7f6830072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48612/0 (socket says 172.21.15.57:48612) 2023-10-30T14:42:18.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.304+0000 7f682effd640 1 -- 172.21.15.57:0/23657904 learned_addr learned my addr 172.21.15.57:0/23657904 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:42:18.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.304+0000 7f682f7fe640 1 --2- 172.21.15.57:0/23657904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68300cb8b0 0x7f683006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:18.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.304+0000 7f6827fff640 1 -- 172.21.15.57:0/23657904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68300cb8b0 msgr2=0x7f683006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:18.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.304+0000 7f6827fff640 1 --2- 172.21.15.57:0/23657904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68300cb8b0 0x7f683006b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.304+0000 7f6827fff640 1 -- 172.21.15.57:0/23657904 >> v1:172.21.15.57:6789/0 conn(0x7f6830151840 legacy=0x7f6830072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:18.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.304+0000 7f682e7fc640 1 -- 172.21.15.57:0/23657904 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101603753 0 0) 0x7f68300c5030 con 0x7f6830151840 2023-10-30T14:42:18.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.304+0000 7f6827fff640 1 -- 172.21.15.57:0/23657904 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6830157df0 con 0x7f6830151c20 2023-10-30T14:42:18.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f6827fff640 1 --2- 172.21.15.57:0/23657904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6830151c20 0x7f68300c49a0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f68180097c0 tx=0x7f681801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8f8846ca3acbfb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:18.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f682e7fc640 1 -- 172.21.15.57:0/23657904 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68180a8970 con 0x7f6830151c20 2023-10-30T14:42:18.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f682e7fc640 1 -- 172.21.15.57:0/23657904 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68180a8ad0 con 0x7f6830151c20 2023-10-30T14:42:18.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f682e7fc640 1 -- 172.21.15.57:0/23657904 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68180a8e60 con 0x7f6830151c20 2023-10-30T14:42:18.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f68355f5640 1 -- 172.21.15.57:0/23657904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6830151c20 msgr2=0x7f68300c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:18.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f68355f5640 1 --2- 172.21.15.57:0/23657904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6830151c20 0x7f68300c49a0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f68180097c0 tx=0x7f681801ec70 comp rx=0 tx=0).stop 2023-10-30T14:42:18.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f68355f5640 1 -- 172.21.15.57:0/23657904 shutdown_connections 2023-10-30T14:42:18.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f68355f5640 1 --2- 172.21.15.57:0/23657904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68300cb8b0 0x7f683006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f68355f5640 1 --2- 172.21.15.57:0/23657904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6830151c20 0x7f68300c49a0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f68355f5640 1 -- 172.21.15.57:0/23657904 >> 172.21.15.57:0/23657904 conn(0x7f68300cac70 msgr2=0x7f68300cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:18.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f68355f5640 1 -- 172.21.15.57:0/23657904 shutdown_connections 2023-10-30T14:42:18.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f68355f5640 1 -- 172.21.15.57:0/23657904 wait complete. 2023-10-30T14:42:18.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f68355f5640 1 Processor -- start 2023-10-30T14:42:18.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f68355f5640 1 -- start start 2023-10-30T14:42:18.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f68355f5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68300cb8b0 0x7f683014a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:18.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f68355f5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6830151840 0x7f683014f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:18.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f68355f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6830151c20 0x7f683014f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:18.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f68355f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6830154410 con 0x7f6830151c20 2023-10-30T14:42:18.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f68355f5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6830154c00 con 0x7f6830151840 2023-10-30T14:42:18.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f68355f5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68301f1dc0 con 0x7f68300cb8b0 2023-10-30T14:42:18.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f682f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6830151c20 0x7f683014f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:18.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f682f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6830151c20 0x7f683014f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35770/0 (socket says 172.21.15.57:35770) 2023-10-30T14:42:18.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.308+0000 7f682f7fe640 1 -- 172.21.15.57:0/3427179159 learned_addr learned my addr 172.21.15.57:0/3427179159 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:18.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.312+0000 7f682effd640 1 --2- 172.21.15.57:0/3427179159 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68300cb8b0 0x7f683014a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:18.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.312+0000 7f6827fff640 1 --2- 172.21.15.57:0/3427179159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6830151840 0x7f683014f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:18.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.312+0000 7f682f7fe640 1 -- 172.21.15.57:0/3427179159 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68300cb8b0 msgr2=0x7f683014a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:18.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.312+0000 7f682f7fe640 1 --2- 172.21.15.57:0/3427179159 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68300cb8b0 0x7f683014a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.312+0000 7f682f7fe640 1 -- 172.21.15.57:0/3427179159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6830151840 msgr2=0x7f683014f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:18.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.312+0000 7f682f7fe640 1 --2- 172.21.15.57:0/3427179159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6830151840 0x7f683014f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.312+0000 7f682f7fe640 1 -- 172.21.15.57:0/3427179159 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6818009420 con 0x7f6830151c20 2023-10-30T14:42:18.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.312+0000 7f6827fff640 1 --2- 172.21.15.57:0/3427179159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6830151840 0x7f683014f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:42:18.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.312+0000 7f682f7fe640 1 --2- 172.21.15.57:0/3427179159 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6830151c20 0x7f683014f5b0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f6828007e70 tx=0x7f682800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:18.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.312+0000 7f68277fe640 1 -- 172.21.15.57:0/3427179159 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6828014620 con 0x7f6830151c20 2023-10-30T14:42:18.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.312+0000 7f68277fe640 1 -- 172.21.15.57:0/3427179159 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68280147a0 con 0x7f6830151c20 2023-10-30T14:42:18.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.312+0000 7f68277fe640 1 -- 172.21.15.57:0/3427179159 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6828014b50 con 0x7f6830151c20 2023-10-30T14:42:18.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.312+0000 7f68355f5640 1 -- 172.21.15.57:0/3427179159 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68301f2940 con 0x7f6830151c20 2023-10-30T14:42:18.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.312+0000 7f68355f5640 1 -- 172.21.15.57:0/3427179159 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68301f22f0 con 0x7f6830151c20 2023-10-30T14:42:18.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.312+0000 7f68355f5640 1 -- 172.21.15.57:0/3427179159 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68301f2ad0 con 0x7f6830151c20 2023-10-30T14:42:18.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.312+0000 7f68277fe640 1 -- 172.21.15.57:0/3427179159 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6828014cb0 con 0x7f6830151c20 2023-10-30T14:42:18.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.316+0000 7f68277fe640 1 --2- 172.21.15.57:0/3427179159 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f68100664d0 0x7f6810068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:18.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.316+0000 7f68277fe640 1 -- 172.21.15.57:0/3427179159 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7f6828084d90 con 0x7f6830151c20 2023-10-30T14:42:18.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.316+0000 7f682effd640 1 --2- 172.21.15.57:0/3427179159 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f68100664d0 0x7f6810068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:18.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.316+0000 7f682effd640 1 --2- 172.21.15.57:0/3427179159 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f68100664d0 0x7f6810068990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6820007fd0 tx=0x7f6820007d60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:18.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.320+0000 7f68277fe640 1 -- 172.21.15.57:0/3427179159 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6828051c40 con 0x7f6830151c20 2023-10-30T14:42:18.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.524+0000 7f68355f5640 1 -- 172.21.15.57:0/3427179159 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f683004e170 con 0x7f6830151c20 2023-10-30T14:42:18.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.524+0000 7f68277fe640 1 -- 172.21.15.57:0/3427179159 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6535 (secure 0 0 0) 0x7f6828055260 con 0x7f6830151c20 2023-10-30T14:42:18.528 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:18.528 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6356,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:41:28.336080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:18.528 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 150 2023-10-30T14:42:18.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.524+0000 7f68355f5640 1 -- 172.21.15.57:0/3427179159 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f68100664d0 msgr2=0x7f6810068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:18.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.524+0000 7f68355f5640 1 --2- 172.21.15.57:0/3427179159 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f68100664d0 0x7f6810068990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6820007fd0 tx=0x7f6820007d60 comp rx=0 tx=0).stop 2023-10-30T14:42:18.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.524+0000 7f68355f5640 1 -- 172.21.15.57:0/3427179159 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6830151c20 msgr2=0x7f683014f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:18.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.524+0000 7f68355f5640 1 --2- 172.21.15.57:0/3427179159 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6830151c20 0x7f683014f5b0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f6828007e70 tx=0x7f682800e4b0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.524+0000 7f68355f5640 1 -- 172.21.15.57:0/3427179159 shutdown_connections 2023-10-30T14:42:18.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.524+0000 7f68355f5640 1 --2- 172.21.15.57:0/3427179159 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f68100664d0 0x7f6810068990 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.524+0000 7f68355f5640 1 --2- 172.21.15.57:0/3427179159 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6830151c20 0x7f683014f5b0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.524+0000 7f68355f5640 1 --2- 172.21.15.57:0/3427179159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6830151840 0x7f683014f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.524+0000 7f68355f5640 1 --2- 172.21.15.57:0/3427179159 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68300cb8b0 0x7f683014a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.524+0000 7f68355f5640 1 -- 172.21.15.57:0/3427179159 >> 172.21.15.57:0/3427179159 conn(0x7f68300cac70 msgr2=0x7f6830072000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:18.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.524+0000 7f68355f5640 1 -- 172.21.15.57:0/3427179159 shutdown_connections 2023-10-30T14:42:18.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.524+0000 7f68355f5640 1 -- 172.21.15.57:0/3427179159 wait complete. 2023-10-30T14:42:18.549 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2023-10-30T14:42:18.549 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T14:42:18.549 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:42:18.573 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T14:42:18.573 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:18.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.756+0000 7f4d455b8640 1 Processor -- start 2023-10-30T14:42:18.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.756+0000 7f4d455b8640 1 -- start start 2023-10-30T14:42:18.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.756+0000 7f4d455b8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4d40153c80 0x7f4d4015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:18.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.756+0000 7f4d455b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d400c4a10 0x7f4d400c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:18.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.756+0000 7f4d455b8640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d40159f20 con 0x7f4d4015a3b0 2023-10-30T14:42:18.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.756+0000 7f4d455b8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d4015dec0 con 0x7f4d40153c80 2023-10-30T14:42:18.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.756+0000 7f4d455b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d401599a0 con 0x7f4d400c4a10 2023-10-30T14:42:18.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.756+0000 7f4d3f7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f4d4015a3b0 0x7f4d400ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48618/0 (socket says 172.21.15.57:48618) 2023-10-30T14:42:18.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.756+0000 7f4d3f7fe640 1 -- 172.21.15.57:0/30654449 learned_addr learned my addr 172.21.15.57:0/30654449 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:42:18.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.756+0000 7f4d3effd640 1 --2- 172.21.15.57:0/30654449 >> v2:172.21.15.88:3300/0 conn(0x7f4d40153c80 0x7f4d4015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:18.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.756+0000 7f4d37fff640 1 --2- 172.21.15.57:0/30654449 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d400c4a10 0x7f4d400c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:18.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.756+0000 7f4d3e7fc640 1 -- 172.21.15.57:0/30654449 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147817178 0 0) 0x7f4d40159f20 con 0x7f4d4015a3b0 2023-10-30T14:42:18.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.756+0000 7f4d3e7fc640 1 -- 172.21.15.57:0/30654449 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d2c003530 con 0x7f4d4015a3b0 2023-10-30T14:42:18.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.756+0000 7f4d3effd640 1 -- 172.21.15.57:0/30654449 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d400c4a10 msgr2=0x7f4d400c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:18.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.756+0000 7f4d3e7fc640 1 -- 172.21.15.57:0/30654449 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3604383473 0 0) 0x7f4d2c003530 con 0x7f4d4015a3b0 2023-10-30T14:42:18.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.756+0000 7f4d3effd640 1 --2- 172.21.15.57:0/30654449 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d400c4a10 0x7f4d400c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.756+0000 7f4d3effd640 1 -- 172.21.15.57:0/30654449 >> v1:172.21.15.57:6789/0 conn(0x7f4d4015a3b0 legacy=0x7f4d400ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:18.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.756+0000 7f4d3effd640 1 -- 172.21.15.57:0/30654449 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d400c47e0 con 0x7f4d40153c80 2023-10-30T14:42:18.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.756+0000 7f4d37fff640 1 --2- 172.21.15.57:0/30654449 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d400c4a10 0x7f4d400c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:42:18.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.760+0000 7f4d3effd640 1 --2- 172.21.15.57:0/30654449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d40153c80 0x7f4d4015cf90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4d300097c0 tx=0x7f4d3001eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a143b846c4e20a7c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:18.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.760+0000 7f4d3e7fc640 1 -- 172.21.15.57:0/30654449 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d38001a00 con 0x7f4d40153c80 2023-10-30T14:42:18.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.760+0000 7f4d3e7fc640 1 -- 172.21.15.57:0/30654449 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d300a85d0 con 0x7f4d40153c80 2023-10-30T14:42:18.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.760+0000 7f4d3e7fc640 1 -- 172.21.15.57:0/30654449 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d300a8960 con 0x7f4d40153c80 2023-10-30T14:42:18.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.760+0000 7f4d455b8640 1 -- 172.21.15.57:0/30654449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d40153c80 msgr2=0x7f4d4015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:18.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.760+0000 7f4d455b8640 1 --2- 172.21.15.57:0/30654449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d40153c80 0x7f4d4015cf90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4d300097c0 tx=0x7f4d3001eb90 comp rx=0 tx=0).stop 2023-10-30T14:42:18.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.760+0000 7f4d455b8640 1 -- 172.21.15.57:0/30654449 shutdown_connections 2023-10-30T14:42:18.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.760+0000 7f4d455b8640 1 --2- 172.21.15.57:0/30654449 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d400c4a10 0x7f4d400c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.760+0000 7f4d455b8640 1 --2- 172.21.15.57:0/30654449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d40153c80 0x7f4d4015cf90 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.760+0000 7f4d455b8640 1 -- 172.21.15.57:0/30654449 >> 172.21.15.57:0/30654449 conn(0x7f4d400715c0 msgr2=0x7f4d400719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:18.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d455b8640 1 -- 172.21.15.57:0/30654449 shutdown_connections 2023-10-30T14:42:18.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d455b8640 1 -- 172.21.15.57:0/30654449 wait complete. 2023-10-30T14:42:18.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d455b8640 1 Processor -- start 2023-10-30T14:42:18.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d455b8640 1 -- start start 2023-10-30T14:42:18.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d455b8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d400c4a10 0x7f4d401ef460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:18.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d455b8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d40153c80 0x7f4d4015c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:18.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d455b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d4015a3b0 0x7f4d40155990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:18.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d455b8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d40155290 con 0x7f4d40153c80 2023-10-30T14:42:18.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d455b8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d401ef9a0 con 0x7f4d400c4a10 2023-10-30T14:42:18.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d455b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4014f700 con 0x7f4d4015a3b0 2023-10-30T14:42:18.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d37fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d40153c80 0x7f4d4015c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:18.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d37fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d40153c80 0x7f4d4015c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35778/0 (socket says 172.21.15.57:35778) 2023-10-30T14:42:18.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d37fff640 1 -- 172.21.15.57:0/672628594 learned_addr learned my addr 172.21.15.57:0/672628594 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:18.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d3effd640 1 --2- 172.21.15.57:0/672628594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d400c4a10 0x7f4d401ef460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:18.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d3f7fe640 1 --2- 172.21.15.57:0/672628594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d4015a3b0 0x7f4d40155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:18.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d37fff640 1 -- 172.21.15.57:0/672628594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d4015a3b0 msgr2=0x7f4d40155990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:18.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d37fff640 1 --2- 172.21.15.57:0/672628594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d4015a3b0 0x7f4d40155990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d37fff640 1 -- 172.21.15.57:0/672628594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d400c4a10 msgr2=0x7f4d401ef460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:18.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d37fff640 1 --2- 172.21.15.57:0/672628594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d400c4a10 0x7f4d401ef460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d37fff640 1 -- 172.21.15.57:0/672628594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d28009420 con 0x7f4d40153c80 2023-10-30T14:42:18.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.764+0000 7f4d37fff640 1 --2- 172.21.15.57:0/672628594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d40153c80 0x7f4d4015c150 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f4d2800ea00 tx=0x7f4d2800eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:18.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.768+0000 7f4d377fe640 1 -- 172.21.15.57:0/672628594 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2800ca00 con 0x7f4d40153c80 2023-10-30T14:42:18.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.768+0000 7f4d377fe640 1 -- 172.21.15.57:0/672628594 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2800cb60 con 0x7f4d40153c80 2023-10-30T14:42:18.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.768+0000 7f4d377fe640 1 -- 172.21.15.57:0/672628594 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d28013470 con 0x7f4d40153c80 2023-10-30T14:42:18.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.768+0000 7f4d455b8640 1 -- 172.21.15.57:0/672628594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d30009420 con 0x7f4d40153c80 2023-10-30T14:42:18.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.768+0000 7f4d455b8640 1 -- 172.21.15.57:0/672628594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d400cd380 con 0x7f4d40153c80 2023-10-30T14:42:18.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.768+0000 7f4d455b8640 1 -- 172.21.15.57:0/672628594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d4004e170 con 0x7f4d40153c80 2023-10-30T14:42:18.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.768+0000 7f4d377fe640 1 -- 172.21.15.57:0/672628594 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4d280135d0 con 0x7f4d40153c80 2023-10-30T14:42:18.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.768+0000 7f4d377fe640 1 --2- 172.21.15.57:0/672628594 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4d200664d0 0x7f4d20068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:18.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.768+0000 7f4d377fe640 1 -- 172.21.15.57:0/672628594 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7f4d28084630 con 0x7f4d40153c80 2023-10-30T14:42:18.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.768+0000 7f4d3effd640 1 --2- 172.21.15.57:0/672628594 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4d200664d0 0x7f4d20068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:18.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.772+0000 7f4d3effd640 1 --2- 172.21.15.57:0/672628594 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4d200664d0 0x7f4d20068990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4d300b2000 tx=0x7f4d30008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:18.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.772+0000 7f4d377fe640 1 -- 172.21.15.57:0/672628594 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4d28051590 con 0x7f4d40153c80 2023-10-30T14:42:18.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.960+0000 7f4d455b8640 1 -- 172.21.15.57:0/672628594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d4014f980 con 0x7f4d40153c80 2023-10-30T14:42:18.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.960+0000 7f4d377fe640 1 -- 172.21.15.57:0/672628594 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6535 (secure 0 0 0) 0x7f4d28054bb0 con 0x7f4d40153c80 2023-10-30T14:42:18.967 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:18.967 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6356,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:41:28.336080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:18.968 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 150 2023-10-30T14:42:18.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.964+0000 7f4d455b8640 1 -- 172.21.15.57:0/672628594 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4d200664d0 msgr2=0x7f4d20068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:18.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.964+0000 7f4d455b8640 1 --2- 172.21.15.57:0/672628594 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4d200664d0 0x7f4d20068990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4d300b2000 tx=0x7f4d30008040 comp rx=0 tx=0).stop 2023-10-30T14:42:18.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.964+0000 7f4d455b8640 1 -- 172.21.15.57:0/672628594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d40153c80 msgr2=0x7f4d4015c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:18.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.964+0000 7f4d455b8640 1 --2- 172.21.15.57:0/672628594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d40153c80 0x7f4d4015c150 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f4d2800ea00 tx=0x7f4d2800eec0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.964+0000 7f4d455b8640 1 -- 172.21.15.57:0/672628594 shutdown_connections 2023-10-30T14:42:18.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.964+0000 7f4d455b8640 1 --2- 172.21.15.57:0/672628594 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4d200664d0 0x7f4d20068990 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.964+0000 7f4d455b8640 1 --2- 172.21.15.57:0/672628594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d4015a3b0 0x7f4d40155990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.964+0000 7f4d455b8640 1 --2- 172.21.15.57:0/672628594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d40153c80 0x7f4d4015c150 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.964+0000 7f4d455b8640 1 --2- 172.21.15.57:0/672628594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d400c4a10 0x7f4d401ef460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:18.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.964+0000 7f4d455b8640 1 -- 172.21.15.57:0/672628594 >> 172.21.15.57:0/672628594 conn(0x7f4d400715c0 msgr2=0x7f4d400ca5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:18.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.964+0000 7f4d455b8640 1 -- 172.21.15.57:0/672628594 shutdown_connections 2023-10-30T14:42:18.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:18.964+0000 7f4d455b8640 1 -- 172.21.15.57:0/672628594 wait complete. 2023-10-30T14:42:18.987 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:42:20.988 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:21.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.140+0000 7f8609d0f640 1 Processor -- start 2023-10-30T14:42:21.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.140+0000 7f8609d0f640 1 -- start start 2023-10-30T14:42:21.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.140+0000 7f8609d0f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8604151840 0x7f86040c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:21.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.140+0000 7f8609d0f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f86040cb8b0 0x7f860406b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:21.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.140+0000 7f8609d0f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86040c5030 con 0x7f8604151c20 2023-10-30T14:42:21.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.140+0000 7f8609d0f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8604157830 con 0x7f8604151840 2023-10-30T14:42:21.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.140+0000 7f8609d0f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8604157b10 con 0x7f86040cb8b0 2023-10-30T14:42:21.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.140+0000 7f86037fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8604151840 0x7f86040c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:21.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.140+0000 7f86037fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8604151840 0x7f86040c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35788/0 (socket says 172.21.15.57:35788) 2023-10-30T14:42:21.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.140+0000 7f86037fe640 1 -- 172.21.15.57:0/3567505215 learned_addr learned my addr 172.21.15.57:0/3567505215 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:21.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.140+0000 7f8603fff640 1 --2- 172.21.15.57:0/3567505215 >> v2:172.21.15.88:3300/0 conn(0x7f86040cb8b0 0x7f860406b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:21.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.140+0000 7f86037fe640 1 -- 172.21.15.57:0/3567505215 >> v2:172.21.15.88:3300/0 conn(0x7f86040cb8b0 msgr2=0x7f860406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:21.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.140+0000 7f86037fe640 1 --2- 172.21.15.57:0/3567505215 >> v2:172.21.15.88:3300/0 conn(0x7f86040cb8b0 0x7f860406b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:21.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.140+0000 7f86037fe640 1 -- 172.21.15.57:0/3567505215 >> v1:172.21.15.57:6789/0 conn(0x7f8604151c20 legacy=0x7f8604072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:21.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.140+0000 7f86037fe640 1 -- 172.21.15.57:0/3567505215 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8604157df0 con 0x7f8604151840 2023-10-30T14:42:21.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.144+0000 7f86037fe640 1 --2- 172.21.15.57:0/3567505215 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8604151840 0x7f86040c49a0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f86040c6cb0 tx=0x7f85f401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=739b5148e57f98ee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:21.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.144+0000 7f8602ffd640 1 -- 172.21.15.57:0/3567505215 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f40a89f0 con 0x7f8604151840 2023-10-30T14:42:21.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.144+0000 7f8602ffd640 1 -- 172.21.15.57:0/3567505215 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f40a8b50 con 0x7f8604151840 2023-10-30T14:42:21.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.144+0000 7f8602ffd640 1 -- 172.21.15.57:0/3567505215 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f40b0440 con 0x7f8604151840 2023-10-30T14:42:21.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.144+0000 7f8609d0f640 1 -- 172.21.15.57:0/3567505215 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8604151840 msgr2=0x7f86040c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:21.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.144+0000 7f8609d0f640 1 --2- 172.21.15.57:0/3567505215 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8604151840 0x7f86040c49a0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f86040c6cb0 tx=0x7f85f401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:42:21.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.144+0000 7f8609d0f640 1 -- 172.21.15.57:0/3567505215 shutdown_connections 2023-10-30T14:42:21.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.144+0000 7f8609d0f640 1 --2- 172.21.15.57:0/3567505215 >> v2:172.21.15.88:3300/0 conn(0x7f86040cb8b0 0x7f860406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:21.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.144+0000 7f8609d0f640 1 --2- 172.21.15.57:0/3567505215 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8604151840 0x7f86040c49a0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:21.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.144+0000 7f8609d0f640 1 -- 172.21.15.57:0/3567505215 >> 172.21.15.57:0/3567505215 conn(0x7f86040cac70 msgr2=0x7f86040cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:21.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.144+0000 7f8609d0f640 1 -- 172.21.15.57:0/3567505215 shutdown_connections 2023-10-30T14:42:21.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.144+0000 7f8609d0f640 1 -- 172.21.15.57:0/3567505215 wait complete. 2023-10-30T14:42:21.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8609d0f640 1 Processor -- start 2023-10-30T14:42:21.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8609d0f640 1 -- start start 2023-10-30T14:42:21.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8609d0f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86040cb8b0 0x7f86040712f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:21.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8609d0f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8604151840 0x7f860414fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:21.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8609d0f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8604151c20 0x7f86041ef0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:21.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8609d0f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86041f3580 con 0x7f8604151c20 2023-10-30T14:42:21.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8609d0f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8604070c60 con 0x7f86040cb8b0 2023-10-30T14:42:21.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8609d0f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8604071830 con 0x7f8604151840 2023-10-30T14:42:21.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8603fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8604151c20 0x7f86041ef0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:21.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8603fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8604151c20 0x7f86041ef0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35794/0 (socket says 172.21.15.57:35794) 2023-10-30T14:42:21.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8603fff640 1 -- 172.21.15.57:0/2685976107 learned_addr learned my addr 172.21.15.57:0/2685976107 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:21.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f86037fe640 1 --2- 172.21.15.57:0/2685976107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86040cb8b0 0x7f86040712f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:21.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f85fbfff640 1 --2- 172.21.15.57:0/2685976107 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8604151840 0x7f860414fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:21.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8603fff640 1 -- 172.21.15.57:0/2685976107 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8604151840 msgr2=0x7f860414fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:21.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8603fff640 1 --2- 172.21.15.57:0/2685976107 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8604151840 0x7f860414fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:21.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8603fff640 1 -- 172.21.15.57:0/2685976107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86040cb8b0 msgr2=0x7f86040712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:21.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8603fff640 1 --2- 172.21.15.57:0/2685976107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86040cb8b0 0x7f86040712f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:21.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8603fff640 1 -- 172.21.15.57:0/2685976107 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f40094d0 con 0x7f8604151c20 2023-10-30T14:42:21.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8603fff640 1 --2- 172.21.15.57:0/2685976107 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8604151c20 0x7f86041ef0d0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f85fc007e70 tx=0x7f85fc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:21.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8600ff9640 1 -- 172.21.15.57:0/2685976107 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc014620 con 0x7f8604151c20 2023-10-30T14:42:21.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8600ff9640 1 -- 172.21.15.57:0/2685976107 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85fc0147a0 con 0x7f8604151c20 2023-10-30T14:42:21.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8600ff9640 1 -- 172.21.15.57:0/2685976107 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc014b50 con 0x7f8604151c20 2023-10-30T14:42:21.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8609d0f640 1 -- 172.21.15.57:0/2685976107 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8604071ac0 con 0x7f8604151c20 2023-10-30T14:42:21.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.148+0000 7f8609d0f640 1 -- 172.21.15.57:0/2685976107 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86041f2aa0 con 0x7f8604151c20 2023-10-30T14:42:21.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.152+0000 7f8609d0f640 1 -- 172.21.15.57:0/2685976107 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860404e170 con 0x7f8604151c20 2023-10-30T14:42:21.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.152+0000 7f8600ff9640 1 -- 172.21.15.57:0/2685976107 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f85fc014cb0 con 0x7f8604151c20 2023-10-30T14:42:21.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.152+0000 7f8600ff9640 1 --2- 172.21.15.57:0/2685976107 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85e40664a0 0x7f85e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:21.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.152+0000 7f8600ff9640 1 -- 172.21.15.57:0/2685976107 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7f85fc084cc0 con 0x7f8604151c20 2023-10-30T14:42:21.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.152+0000 7f86037fe640 1 --2- 172.21.15.57:0/2685976107 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85e40664a0 0x7f85e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:21.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.152+0000 7f86037fe640 1 --2- 172.21.15.57:0/2685976107 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85e40664a0 0x7f85e4068960 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f85f40b2000 tx=0x7f85f4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:21.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.156+0000 7f8600ff9640 1 -- 172.21.15.57:0/2685976107 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f85fc08b080 con 0x7f8604151c20 2023-10-30T14:42:21.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.348+0000 7f8609d0f640 1 -- 172.21.15.57:0/2685976107 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8604155060 con 0x7f8604151c20 2023-10-30T14:42:21.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.348+0000 7f8600ff9640 1 -- 172.21.15.57:0/2685976107 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6535 (secure 0 0 0) 0x7f85fc051b90 con 0x7f8604151c20 2023-10-30T14:42:21.352 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:21.353 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6356,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:41:28.336080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:21.353 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 150 2023-10-30T14:42:21.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.352+0000 7f8609d0f640 1 -- 172.21.15.57:0/2685976107 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85e40664a0 msgr2=0x7f85e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:21.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.352+0000 7f8609d0f640 1 --2- 172.21.15.57:0/2685976107 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85e40664a0 0x7f85e4068960 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f85f40b2000 tx=0x7f85f4009040 comp rx=0 tx=0).stop 2023-10-30T14:42:21.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.352+0000 7f8609d0f640 1 -- 172.21.15.57:0/2685976107 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8604151c20 msgr2=0x7f86041ef0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:21.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.352+0000 7f8609d0f640 1 --2- 172.21.15.57:0/2685976107 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8604151c20 0x7f86041ef0d0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f85fc007e70 tx=0x7f85fc00e4b0 comp rx=0 tx=0).stop 2023-10-30T14:42:21.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.352+0000 7f8609d0f640 1 -- 172.21.15.57:0/2685976107 shutdown_connections 2023-10-30T14:42:21.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.352+0000 7f8609d0f640 1 --2- 172.21.15.57:0/2685976107 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f85e40664a0 0x7f85e4068960 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:21.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.352+0000 7f8609d0f640 1 --2- 172.21.15.57:0/2685976107 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8604151c20 0x7f86041ef0d0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:21.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.352+0000 7f8609d0f640 1 --2- 172.21.15.57:0/2685976107 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8604151840 0x7f860414fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:21.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.352+0000 7f8609d0f640 1 --2- 172.21.15.57:0/2685976107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86040cb8b0 0x7f86040712f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:21.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.352+0000 7f8609d0f640 1 -- 172.21.15.57:0/2685976107 >> 172.21.15.57:0/2685976107 conn(0x7f86040cac70 msgr2=0x7f86040c59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:21.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.352+0000 7f8609d0f640 1 -- 172.21.15.57:0/2685976107 shutdown_connections 2023-10-30T14:42:21.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:21.352+0000 7f8609d0f640 1 -- 172.21.15.57:0/2685976107 wait complete. 2023-10-30T14:42:21.374 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:42:23.375 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:23.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.524+0000 7ff38fdab640 1 Processor -- start 2023-10-30T14:42:23.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.524+0000 7ff38fdab640 1 -- start start 2023-10-30T14:42:23.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.524+0000 7ff38fdab640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff3880c4a10 0x7ff38815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:23.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.524+0000 7ff38fdab640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff388159f20 con 0x7ff388153c80 2023-10-30T14:42:23.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.524+0000 7ff38fdab640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38815d620 con 0x7ff3880c42a0 2023-10-30T14:42:23.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.524+0000 7ff38fdab640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff38815dec0 con 0x7ff3880c4a10 2023-10-30T14:42:23.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.524+0000 7ff38db47640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff388153c80 0x7ff38815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48626/0 (socket says 172.21.15.57:48626) 2023-10-30T14:42:23.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.524+0000 7ff38db47640 1 -- 172.21.15.57:0/3768229533 learned_addr learned my addr 172.21.15.57:0/3768229533 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:42:23.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.524+0000 7ff38d346640 1 --2- 172.21.15.57:0/3768229533 >> v2:172.21.15.57:3300/0 conn(0x7ff3880c4a10 0x7ff38815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:23.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.524+0000 7ff38cb45640 1 -- 172.21.15.57:0/3768229533 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386075924 0 0) 0x7ff388159f20 con 0x7ff388153c80 2023-10-30T14:42:23.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.524+0000 7ff38cb45640 1 -- 172.21.15.57:0/3768229533 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff374003530 con 0x7ff388153c80 2023-10-30T14:42:23.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.524+0000 7ff38d346640 1 -- 172.21.15.57:0/3768229533 >> v1:172.21.15.88:6789/0 conn(0x7ff3880c42a0 legacy=0x7ff3880ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:23.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.524+0000 7ff38d346640 1 -- 172.21.15.57:0/3768229533 >> v1:172.21.15.57:6789/0 conn(0x7ff388153c80 legacy=0x7ff38815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:23.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.524+0000 7ff38d346640 1 -- 172.21.15.57:0/3768229533 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38815d900 con 0x7ff3880c4a10 2023-10-30T14:42:23.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.528+0000 7ff38d346640 1 --2- 172.21.15.57:0/3768229533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3880c4a10 0x7ff38815cf90 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7ff3780097c0 tx=0x7ff37801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecfad356daaa615e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:23.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.528+0000 7ff38cb45640 1 -- 172.21.15.57:0/3768229533 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3780a8970 con 0x7ff3880c4a10 2023-10-30T14:42:23.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.528+0000 7ff38cb45640 1 -- 172.21.15.57:0/3768229533 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3780a8ad0 con 0x7ff3880c4a10 2023-10-30T14:42:23.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.528+0000 7ff38cb45640 1 -- 172.21.15.57:0/3768229533 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3780a8e60 con 0x7ff3880c4a10 2023-10-30T14:42:23.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.528+0000 7ff38fdab640 1 -- 172.21.15.57:0/3768229533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3880c4a10 msgr2=0x7ff38815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:23.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.528+0000 7ff38fdab640 1 --2- 172.21.15.57:0/3768229533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3880c4a10 0x7ff38815cf90 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7ff3780097c0 tx=0x7ff37801ec70 comp rx=0 tx=0).stop 2023-10-30T14:42:23.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.528+0000 7ff38fdab640 1 -- 172.21.15.57:0/3768229533 shutdown_connections 2023-10-30T14:42:23.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.528+0000 7ff38fdab640 1 --2- 172.21.15.57:0/3768229533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3880c4a10 0x7ff38815cf90 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:23.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.528+0000 7ff38fdab640 1 -- 172.21.15.57:0/3768229533 >> 172.21.15.57:0/3768229533 conn(0x7ff3880715c0 msgr2=0x7ff3880719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:23.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.528+0000 7ff38fdab640 1 -- 172.21.15.57:0/3768229533 shutdown_connections 2023-10-30T14:42:23.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.528+0000 7ff38fdab640 1 -- 172.21.15.57:0/3768229533 wait complete. 2023-10-30T14:42:23.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38fdab640 1 Processor -- start 2023-10-30T14:42:23.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38fdab640 1 -- start start 2023-10-30T14:42:23.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38fdab640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3880c42a0 0x7ff388158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:23.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38fdab640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3880c4a10 0x7ff388158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:23.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38fdab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff388153c80 0x7ff3881f26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:23.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38fdab640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3880cf850 con 0x7ff388153c80 2023-10-30T14:42:23.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38fdab640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3880cfb30 con 0x7ff3880c42a0 2023-10-30T14:42:23.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38fdab640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff388154490 con 0x7ff3880c4a10 2023-10-30T14:42:23.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38e348640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff388153c80 0x7ff3881f26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:23.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38e348640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff388153c80 0x7ff3881f26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35814/0 (socket says 172.21.15.57:35814) 2023-10-30T14:42:23.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38e348640 1 -- 172.21.15.57:0/2406231007 learned_addr learned my addr 172.21.15.57:0/2406231007 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:23.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38db47640 1 --2- 172.21.15.57:0/2406231007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3880c42a0 0x7ff388158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:23.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38d346640 1 --2- 172.21.15.57:0/2406231007 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3880c4a10 0x7ff388158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:23.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38e348640 1 -- 172.21.15.57:0/2406231007 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3880c4a10 msgr2=0x7ff388158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:23.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38e348640 1 --2- 172.21.15.57:0/2406231007 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3880c4a10 0x7ff388158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:23.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38e348640 1 -- 172.21.15.57:0/2406231007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3880c42a0 msgr2=0x7ff388158350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:23.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38e348640 1 --2- 172.21.15.57:0/2406231007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3880c42a0 0x7ff388158350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:23.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38e348640 1 -- 172.21.15.57:0/2406231007 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff378009420 con 0x7ff388153c80 2023-10-30T14:42:23.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38db47640 1 --2- 172.21.15.57:0/2406231007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3880c42a0 0x7ff388158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:42:23.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38e348640 1 --2- 172.21.15.57:0/2406231007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff388153c80 0x7ff3881f26a0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7ff38000c770 tx=0x7ff38000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:23.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff37e7fc640 1 -- 172.21.15.57:0/2406231007 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff380003b80 con 0x7ff388153c80 2023-10-30T14:42:23.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff37e7fc640 1 -- 172.21.15.57:0/2406231007 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff380003ce0 con 0x7ff388153c80 2023-10-30T14:42:23.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38fdab640 1 -- 172.21.15.57:0/2406231007 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff388154770 con 0x7ff388153c80 2023-10-30T14:42:23.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff37e7fc640 1 -- 172.21.15.57:0/2406231007 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38000aa90 con 0x7ff388153c80 2023-10-30T14:42:23.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.532+0000 7ff38fdab640 1 -- 172.21.15.57:0/2406231007 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38814fc40 con 0x7ff388153c80 2023-10-30T14:42:23.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.536+0000 7ff38fdab640 1 -- 172.21.15.57:0/2406231007 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3881542f0 con 0x7ff388153c80 2023-10-30T14:42:23.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.536+0000 7ff37e7fc640 1 -- 172.21.15.57:0/2406231007 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff38000abf0 con 0x7ff388153c80 2023-10-30T14:42:23.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.536+0000 7ff37e7fc640 1 --2- 172.21.15.57:0/2406231007 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff36c0664a0 0x7ff36c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:23.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.536+0000 7ff37e7fc640 1 -- 172.21.15.57:0/2406231007 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7ff380084270 con 0x7ff388153c80 2023-10-30T14:42:23.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.536+0000 7ff38db47640 1 --2- 172.21.15.57:0/2406231007 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff36c0664a0 0x7ff36c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:23.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.536+0000 7ff38db47640 1 --2- 172.21.15.57:0/2406231007 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff36c0664a0 0x7ff36c068960 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff3840011f0 tx=0x7ff384009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:23.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.540+0000 7ff37e7fc640 1 -- 172.21.15.57:0/2406231007 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff3881542f0 con 0x7ff388153c80 2023-10-30T14:42:23.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.732+0000 7ff38fdab640 1 -- 172.21.15.57:0/2406231007 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3881542f0 con 0x7ff388153c80 2023-10-30T14:42:23.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.732+0000 7ff37e7fc640 1 -- 172.21.15.57:0/2406231007 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6535 (secure 0 0 0) 0x7ff3881542f0 con 0x7ff388153c80 2023-10-30T14:42:23.736 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:23.736 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6356,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:41:28.336080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:23.736 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 150 2023-10-30T14:42:23.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.732+0000 7ff38fdab640 1 -- 172.21.15.57:0/2406231007 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff36c0664a0 msgr2=0x7ff36c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:23.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.732+0000 7ff38fdab640 1 --2- 172.21.15.57:0/2406231007 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff36c0664a0 0x7ff36c068960 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff3840011f0 tx=0x7ff384009040 comp rx=0 tx=0).stop 2023-10-30T14:42:23.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.732+0000 7ff38fdab640 1 -- 172.21.15.57:0/2406231007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff388153c80 msgr2=0x7ff3881f26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:23.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.732+0000 7ff38fdab640 1 --2- 172.21.15.57:0/2406231007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff388153c80 0x7ff3881f26a0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7ff38000c770 tx=0x7ff38000cc30 comp rx=0 tx=0).stop 2023-10-30T14:42:23.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.732+0000 7ff38fdab640 1 -- 172.21.15.57:0/2406231007 shutdown_connections 2023-10-30T14:42:23.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.732+0000 7ff38fdab640 1 --2- 172.21.15.57:0/2406231007 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff36c0664a0 0x7ff36c068960 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:23.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.732+0000 7ff38fdab640 1 --2- 172.21.15.57:0/2406231007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff388153c80 0x7ff3881f26a0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:23.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.732+0000 7ff38fdab640 1 --2- 172.21.15.57:0/2406231007 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3880c4a10 0x7ff388158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:23.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.732+0000 7ff38fdab640 1 --2- 172.21.15.57:0/2406231007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3880c42a0 0x7ff388158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:23.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.732+0000 7ff38fdab640 1 -- 172.21.15.57:0/2406231007 >> 172.21.15.57:0/2406231007 conn(0x7ff3880715c0 msgr2=0x7ff388152510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:23.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.732+0000 7ff38fdab640 1 -- 172.21.15.57:0/2406231007 shutdown_connections 2023-10-30T14:42:23.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:23.732+0000 7ff38fdab640 1 -- 172.21.15.57:0/2406231007 wait complete. 2023-10-30T14:42:23.757 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:42:25.759 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:25.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.912+0000 7ffb71668640 1 Processor -- start 2023-10-30T14:42:25.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.912+0000 7ffb71668640 1 -- start start 2023-10-30T14:42:25.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.912+0000 7ffb71668640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6c153c80 0x7ffb6c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:25.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.912+0000 7ffb71668640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ffb6c0c42a0 0x7ffb6c0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:25.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.912+0000 7ffb71668640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6c159f20 con 0x7ffb6c0c4a10 2023-10-30T14:42:25.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.912+0000 7ffb71668640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb6c15dec0 con 0x7ffb6c0c42a0 2023-10-30T14:42:25.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.912+0000 7ffb71668640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c1599a0 con 0x7ffb6c153c80 2023-10-30T14:42:25.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.912+0000 7ffb6b7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ffb6c0c42a0 0x7ffb6c0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:25.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.912+0000 7ffb63fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ffb6c0c4a10 0x7ffb6c15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48856/0 (socket says 172.21.15.57:48856) 2023-10-30T14:42:25.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.912+0000 7ffb63fff640 1 -- 172.21.15.57:0/2412726753 learned_addr learned my addr 172.21.15.57:0/2412726753 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:42:25.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.912+0000 7ffb6b7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ffb6c0c42a0 0x7ffb6c0ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42346/0 (socket says 172.21.15.57:42346) 2023-10-30T14:42:25.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.912+0000 7ffb6affd640 1 --2- 172.21.15.57:0/2412726753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6c153c80 0x7ffb6c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:25.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.912+0000 7ffb6b7fe640 1 -- 172.21.15.57:0/2412726753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6c153c80 msgr2=0x7ffb6c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:25.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.912+0000 7ffb6b7fe640 1 --2- 172.21.15.57:0/2412726753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6c153c80 0x7ffb6c15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:25.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.912+0000 7ffb6b7fe640 1 -- 172.21.15.57:0/2412726753 >> v1:172.21.15.57:6789/0 conn(0x7ffb6c0c4a10 legacy=0x7ffb6c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:25.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.912+0000 7ffb6b7fe640 1 -- 172.21.15.57:0/2412726753 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6c15d620 con 0x7ffb6c0c42a0 2023-10-30T14:42:25.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.912+0000 7ffb6affd640 1 --2- 172.21.15.57:0/2412726753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6c153c80 0x7ffb6c15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:42:25.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.912+0000 7ffb6a7fc640 1 -- 172.21.15.57:0/2412726753 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454188415 0 0) 0x7ffb6c159f20 con 0x7ffb6c0c4a10 2023-10-30T14:42:25.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.916+0000 7ffb6b7fe640 1 --2- 172.21.15.57:0/2412726753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c0c42a0 0x7ffb6c0ca1e0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7ffb64009d90 tx=0x7ffb6401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b3177560c5ed496 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:25.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.916+0000 7ffb6a7fc640 1 -- 172.21.15.57:0/2412726753 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb640af600 con 0x7ffb6c0c42a0 2023-10-30T14:42:25.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.916+0000 7ffb6a7fc640 1 -- 172.21.15.57:0/2412726753 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb640ab050 con 0x7ffb6c0c42a0 2023-10-30T14:42:25.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.916+0000 7ffb6a7fc640 1 -- 172.21.15.57:0/2412726753 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb640af980 con 0x7ffb6c0c42a0 2023-10-30T14:42:25.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.916+0000 7ffb71668640 1 -- 172.21.15.57:0/2412726753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c0c42a0 msgr2=0x7ffb6c0ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:25.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.916+0000 7ffb71668640 1 --2- 172.21.15.57:0/2412726753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c0c42a0 0x7ffb6c0ca1e0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7ffb64009d90 tx=0x7ffb6401f130 comp rx=0 tx=0).stop 2023-10-30T14:42:25.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.916+0000 7ffb71668640 1 -- 172.21.15.57:0/2412726753 shutdown_connections 2023-10-30T14:42:25.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.916+0000 7ffb71668640 1 --2- 172.21.15.57:0/2412726753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c0c42a0 0x7ffb6c0ca1e0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:25.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.916+0000 7ffb71668640 1 --2- 172.21.15.57:0/2412726753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6c153c80 0x7ffb6c15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:25.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.916+0000 7ffb71668640 1 -- 172.21.15.57:0/2412726753 >> 172.21.15.57:0/2412726753 conn(0x7ffb6c0715c0 msgr2=0x7ffb6c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:25.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.916+0000 7ffb71668640 1 -- 172.21.15.57:0/2412726753 shutdown_connections 2023-10-30T14:42:25.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.916+0000 7ffb71668640 1 -- 172.21.15.57:0/2412726753 wait complete. 2023-10-30T14:42:25.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb71668640 1 Processor -- start 2023-10-30T14:42:25.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb71668640 1 -- start start 2023-10-30T14:42:25.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb71668640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb6c0c42a0 0x7ffb6c1eed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:25.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb71668640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6c0c4a10 0x7ffb6c0b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:25.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb71668640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c153c80 0x7ffb6c0c8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:25.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb71668640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c0b4160 con 0x7ffb6c153c80 2023-10-30T14:42:25.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb71668640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c1f28c0 con 0x7ffb6c0c42a0 2023-10-30T14:42:25.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb71668640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c1f1a80 con 0x7ffb6c0c4a10 2023-10-30T14:42:25.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb6b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c153c80 0x7ffb6c0c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:25.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb6b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c153c80 0x7ffb6c0c8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42358/0 (socket says 172.21.15.57:42358) 2023-10-30T14:42:25.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb6b7fe640 1 -- 172.21.15.57:0/2271888899 learned_addr learned my addr 172.21.15.57:0/2271888899 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:25.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb6affd640 1 --2- 172.21.15.57:0/2271888899 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb6c0c42a0 0x7ffb6c1eed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:25.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb63fff640 1 --2- 172.21.15.57:0/2271888899 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6c0c4a10 0x7ffb6c0b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:25.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb6b7fe640 1 -- 172.21.15.57:0/2271888899 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6c0c4a10 msgr2=0x7ffb6c0b4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:25.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb6b7fe640 1 --2- 172.21.15.57:0/2271888899 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6c0c4a10 0x7ffb6c0b4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:25.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb6b7fe640 1 -- 172.21.15.57:0/2271888899 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb6c0c42a0 msgr2=0x7ffb6c1eed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:25.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb6b7fe640 1 --2- 172.21.15.57:0/2271888899 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb6c0c42a0 0x7ffb6c1eed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:25.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb6b7fe640 1 -- 172.21.15.57:0/2271888899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb64009a70 con 0x7ffb6c153c80 2023-10-30T14:42:25.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb6b7fe640 1 --2- 172.21.15.57:0/2271888899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c153c80 0x7ffb6c0c8140 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7ffb6401d7e0 tx=0x7ffb6401bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:25.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb637fe640 1 -- 172.21.15.57:0/2271888899 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb640b9610 con 0x7ffb6c153c80 2023-10-30T14:42:25.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb637fe640 1 -- 172.21.15.57:0/2271888899 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb640b9770 con 0x7ffb6c153c80 2023-10-30T14:42:25.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb637fe640 1 -- 172.21.15.57:0/2271888899 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb640b9b00 con 0x7ffb6c153c80 2023-10-30T14:42:25.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb71668640 1 -- 172.21.15.57:0/2271888899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb6c1f1d40 con 0x7ffb6c153c80 2023-10-30T14:42:25.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.920+0000 7ffb71668640 1 -- 172.21.15.57:0/2271888899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb6c1f2300 con 0x7ffb6c153c80 2023-10-30T14:42:25.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.924+0000 7ffb71668640 1 -- 172.21.15.57:0/2271888899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6c04e170 con 0x7ffb6c153c80 2023-10-30T14:42:25.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.924+0000 7ffb637fe640 1 -- 172.21.15.57:0/2271888899 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffb640b6050 con 0x7ffb6c153c80 2023-10-30T14:42:25.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.924+0000 7ffb637fe640 1 --2- 172.21.15.57:0/2271888899 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffb4c0664a0 0x7ffb4c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:25.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.924+0000 7ffb637fe640 1 -- 172.21.15.57:0/2271888899 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7ffb641228d0 con 0x7ffb6c153c80 2023-10-30T14:42:25.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.924+0000 7ffb6affd640 1 --2- 172.21.15.57:0/2271888899 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffb4c0664a0 0x7ffb4c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:25.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.924+0000 7ffb6affd640 1 --2- 172.21.15.57:0/2271888899 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffb4c0664a0 0x7ffb4c068960 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c0045f0 tx=0x7ffb5c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:25.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:25.928+0000 7ffb637fe640 1 -- 172.21.15.57:0/2271888899 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffb640ef830 con 0x7ffb6c153c80 2023-10-30T14:42:26.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:26.120+0000 7ffb71668640 1 -- 172.21.15.57:0/2271888899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb6c0cad40 con 0x7ffb6c153c80 2023-10-30T14:42:26.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:26.120+0000 7ffb637fe640 1 -- 172.21.15.57:0/2271888899 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6535 (secure 0 0 0) 0x7ffb640b4090 con 0x7ffb6c153c80 2023-10-30T14:42:26.124 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:26.124 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6356,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:41:28.336080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:26.124 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 150 2023-10-30T14:42:26.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:26.120+0000 7ffb71668640 1 -- 172.21.15.57:0/2271888899 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffb4c0664a0 msgr2=0x7ffb4c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:26.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:26.120+0000 7ffb71668640 1 --2- 172.21.15.57:0/2271888899 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffb4c0664a0 0x7ffb4c068960 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c0045f0 tx=0x7ffb5c008040 comp rx=0 tx=0).stop 2023-10-30T14:42:26.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:26.120+0000 7ffb71668640 1 -- 172.21.15.57:0/2271888899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c153c80 msgr2=0x7ffb6c0c8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:26.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:26.120+0000 7ffb71668640 1 --2- 172.21.15.57:0/2271888899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c153c80 0x7ffb6c0c8140 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7ffb6401d7e0 tx=0x7ffb6401bed0 comp rx=0 tx=0).stop 2023-10-30T14:42:26.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:26.120+0000 7ffb71668640 1 -- 172.21.15.57:0/2271888899 shutdown_connections 2023-10-30T14:42:26.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:26.120+0000 7ffb71668640 1 --2- 172.21.15.57:0/2271888899 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffb4c0664a0 0x7ffb4c068960 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:26.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:26.120+0000 7ffb71668640 1 --2- 172.21.15.57:0/2271888899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb6c153c80 0x7ffb6c0c8140 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:26.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:26.120+0000 7ffb71668640 1 --2- 172.21.15.57:0/2271888899 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffb6c0c4a10 0x7ffb6c0b4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:26.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:26.120+0000 7ffb71668640 1 --2- 172.21.15.57:0/2271888899 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb6c0c42a0 0x7ffb6c1eed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:26.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:26.120+0000 7ffb71668640 1 -- 172.21.15.57:0/2271888899 >> 172.21.15.57:0/2271888899 conn(0x7ffb6c0715c0 msgr2=0x7ffb6c0c9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:26.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:26.120+0000 7ffb71668640 1 -- 172.21.15.57:0/2271888899 shutdown_connections 2023-10-30T14:42:26.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:26.120+0000 7ffb71668640 1 -- 172.21.15.57:0/2271888899 wait complete. 2023-10-30T14:42:26.145 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:42:28.147 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:28.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.300+0000 7fb39a3b8640 1 Processor -- start 2023-10-30T14:42:28.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.300+0000 7fb39a3b8640 1 -- start start 2023-10-30T14:42:28.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.300+0000 7fb39a3b8640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb3940c4a10 0x7fb39415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:28.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.300+0000 7fb39a3b8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb3940c42a0 0x7fb3940ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:28.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.300+0000 7fb39a3b8640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb394159f20 con 0x7fb394153c80 2023-10-30T14:42:28.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.300+0000 7fb39a3b8640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb39415dec0 con 0x7fb3940c4a10 2023-10-30T14:42:28.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.300+0000 7fb39a3b8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3941599a0 con 0x7fb3940c42a0 2023-10-30T14:42:28.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.300+0000 7fb393fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb394153c80 0x7fb39415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48864/0 (socket says 172.21.15.57:48864) 2023-10-30T14:42:28.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.300+0000 7fb393fff640 1 -- 172.21.15.57:0/3736360022 learned_addr learned my addr 172.21.15.57:0/3736360022 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:42:28.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.300+0000 7fb3937fe640 1 --2- 172.21.15.57:0/3736360022 >> v2:172.21.15.57:3300/0 conn(0x7fb3940c4a10 0x7fb39415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:28.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.300+0000 7fb398955640 1 --2- 172.21.15.57:0/3736360022 >> v2:172.21.15.88:3300/0 conn(0x7fb3940c42a0 0x7fb3940ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:28.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.300+0000 7fb392ffd640 1 -- 172.21.15.57:0/3736360022 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226507036 0 0) 0x7fb394159f20 con 0x7fb394153c80 2023-10-30T14:42:28.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.300+0000 7fb392ffd640 1 -- 172.21.15.57:0/3736360022 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb37c003530 con 0x7fb394153c80 2023-10-30T14:42:28.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.300+0000 7fb3937fe640 1 -- 172.21.15.57:0/3736360022 >> v2:172.21.15.88:3300/0 conn(0x7fb3940c42a0 msgr2=0x7fb3940ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:28.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.300+0000 7fb3937fe640 1 --2- 172.21.15.57:0/3736360022 >> v2:172.21.15.88:3300/0 conn(0x7fb3940c42a0 0x7fb3940ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:28.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.300+0000 7fb3937fe640 1 -- 172.21.15.57:0/3736360022 >> v1:172.21.15.57:6789/0 conn(0x7fb394153c80 legacy=0x7fb39415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:28.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.300+0000 7fb3937fe640 1 -- 172.21.15.57:0/3736360022 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39415d620 con 0x7fb3940c4a10 2023-10-30T14:42:28.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.304+0000 7fb3937fe640 1 --2- 172.21.15.57:0/3736360022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3940c4a10 0x7fb39415cf90 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fb3940b1d40 tx=0x7fb38801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a11e677d21d7a16 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:28.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.304+0000 7fb392ffd640 1 -- 172.21.15.57:0/3736360022 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3880a8950 con 0x7fb3940c4a10 2023-10-30T14:42:28.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.304+0000 7fb392ffd640 1 -- 172.21.15.57:0/3736360022 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3880a8ab0 con 0x7fb3940c4a10 2023-10-30T14:42:28.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.304+0000 7fb392ffd640 1 -- 172.21.15.57:0/3736360022 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3880a8e40 con 0x7fb3940c4a10 2023-10-30T14:42:28.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.304+0000 7fb39a3b8640 1 -- 172.21.15.57:0/3736360022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3940c4a10 msgr2=0x7fb39415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:28.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.304+0000 7fb39a3b8640 1 --2- 172.21.15.57:0/3736360022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3940c4a10 0x7fb39415cf90 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fb3940b1d40 tx=0x7fb38801ec50 comp rx=0 tx=0).stop 2023-10-30T14:42:28.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.304+0000 7fb39a3b8640 1 -- 172.21.15.57:0/3736360022 shutdown_connections 2023-10-30T14:42:28.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.304+0000 7fb39a3b8640 1 --2- 172.21.15.57:0/3736360022 >> v2:172.21.15.88:3300/0 conn(0x7fb3940c42a0 0x7fb3940ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:28.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.304+0000 7fb39a3b8640 1 --2- 172.21.15.57:0/3736360022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3940c4a10 0x7fb39415cf90 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:28.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.304+0000 7fb39a3b8640 1 -- 172.21.15.57:0/3736360022 >> 172.21.15.57:0/3736360022 conn(0x7fb3940715c0 msgr2=0x7fb3940719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:28.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.304+0000 7fb39a3b8640 1 -- 172.21.15.57:0/3736360022 shutdown_connections 2023-10-30T14:42:28.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.304+0000 7fb39a3b8640 1 -- 172.21.15.57:0/3736360022 wait complete. 2023-10-30T14:42:28.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb39a3b8640 1 Processor -- start 2023-10-30T14:42:28.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb39a3b8640 1 -- start start 2023-10-30T14:42:28.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb39a3b8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3940c42a0 0x7fb39415b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:28.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb39a3b8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3940c4a10 0x7fb39414e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:28.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb39a3b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb394153c80 0x7fb39414eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:28.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb39a3b8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb394155320 con 0x7fb3940c42a0 2023-10-30T14:42:28.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb39a3b8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3940ce090 con 0x7fb3940c4a10 2023-10-30T14:42:28.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb39a3b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3940ce350 con 0x7fb394153c80 2023-10-30T14:42:28.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb393fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3940c42a0 0x7fb39415b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:28.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb393fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3940c42a0 0x7fb39415b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42370/0 (socket says 172.21.15.57:42370) 2023-10-30T14:42:28.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb393fff640 1 -- 172.21.15.57:0/510092114 learned_addr learned my addr 172.21.15.57:0/510092114 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:28.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb398955640 1 --2- 172.21.15.57:0/510092114 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb394153c80 0x7fb39414eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:28.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb3937fe640 1 --2- 172.21.15.57:0/510092114 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3940c4a10 0x7fb39414e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:28.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb393fff640 1 -- 172.21.15.57:0/510092114 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb394153c80 msgr2=0x7fb39414eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:28.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb393fff640 1 --2- 172.21.15.57:0/510092114 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb394153c80 0x7fb39414eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:28.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb393fff640 1 -- 172.21.15.57:0/510092114 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3940c4a10 msgr2=0x7fb39414e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:28.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb393fff640 1 --2- 172.21.15.57:0/510092114 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3940c4a10 0x7fb39414e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:28.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb393fff640 1 -- 172.21.15.57:0/510092114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb388009420 con 0x7fb3940c42a0 2023-10-30T14:42:28.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb398955640 1 --2- 172.21.15.57:0/510092114 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb394153c80 0x7fb39414eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:42:28.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb3937fe640 1 --2- 172.21.15.57:0/510092114 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3940c4a10 0x7fb39414e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:42:28.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb393fff640 1 --2- 172.21.15.57:0/510092114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3940c42a0 0x7fb39415b490 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fb38400c420 tx=0x7fb38400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:28.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb390ff9640 1 -- 172.21.15.57:0/510092114 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb384005cf0 con 0x7fb3940c42a0 2023-10-30T14:42:28.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb390ff9640 1 -- 172.21.15.57:0/510092114 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb384005e50 con 0x7fb3940c42a0 2023-10-30T14:42:28.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb390ff9640 1 -- 172.21.15.57:0/510092114 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3840072d0 con 0x7fb3940c42a0 2023-10-30T14:42:28.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb39a3b8640 1 -- 172.21.15.57:0/510092114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3940ce630 con 0x7fb3940c42a0 2023-10-30T14:42:28.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb39a3b8640 1 -- 172.21.15.57:0/510092114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb39414e0d0 con 0x7fb3940c42a0 2023-10-30T14:42:28.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.308+0000 7fb39a3b8640 1 -- 172.21.15.57:0/510092114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb39414df30 con 0x7fb3940c42a0 2023-10-30T14:42:28.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.312+0000 7fb390ff9640 1 -- 172.21.15.57:0/510092114 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb384007430 con 0x7fb3940c42a0 2023-10-30T14:42:28.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.312+0000 7fb390ff9640 1 --2- 172.21.15.57:0/510092114 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb3800664a0 0x7fb380068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:28.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.312+0000 7fb390ff9640 1 -- 172.21.15.57:0/510092114 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7fb3840839f0 con 0x7fb3940c42a0 2023-10-30T14:42:28.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.312+0000 7fb3937fe640 1 --2- 172.21.15.57:0/510092114 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb3800664a0 0x7fb380068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:28.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.312+0000 7fb3937fe640 1 --2- 172.21.15.57:0/510092114 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb3800664a0 0x7fb380068960 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb3880b2000 tx=0x7fb388008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:28.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.316+0000 7fb390ff9640 1 -- 172.21.15.57:0/510092114 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb39414df30 con 0x7fb3940c42a0 2023-10-30T14:42:28.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.508+0000 7fb39a3b8640 1 -- 172.21.15.57:0/510092114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb39414df30 con 0x7fb3940c42a0 2023-10-30T14:42:28.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.508+0000 7fb390ff9640 1 -- 172.21.15.57:0/510092114 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6535 (secure 0 0 0) 0x7fb39414df30 con 0x7fb3940c42a0 2023-10-30T14:42:28.512 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:28.512 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6356,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:41:28.336080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:28.513 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 150 2023-10-30T14:42:28.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.508+0000 7fb39a3b8640 1 -- 172.21.15.57:0/510092114 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb3800664a0 msgr2=0x7fb380068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:28.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.508+0000 7fb39a3b8640 1 --2- 172.21.15.57:0/510092114 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb3800664a0 0x7fb380068960 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb3880b2000 tx=0x7fb388008040 comp rx=0 tx=0).stop 2023-10-30T14:42:28.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.508+0000 7fb39a3b8640 1 -- 172.21.15.57:0/510092114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3940c42a0 msgr2=0x7fb39415b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:28.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.508+0000 7fb39a3b8640 1 --2- 172.21.15.57:0/510092114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3940c42a0 0x7fb39415b490 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fb38400c420 tx=0x7fb38400c8e0 comp rx=0 tx=0).stop 2023-10-30T14:42:28.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.508+0000 7fb39a3b8640 1 -- 172.21.15.57:0/510092114 shutdown_connections 2023-10-30T14:42:28.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.508+0000 7fb39a3b8640 1 --2- 172.21.15.57:0/510092114 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb3800664a0 0x7fb380068960 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:28.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.508+0000 7fb39a3b8640 1 --2- 172.21.15.57:0/510092114 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb394153c80 0x7fb39414eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:28.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.508+0000 7fb39a3b8640 1 --2- 172.21.15.57:0/510092114 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3940c4a10 0x7fb39414e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:28.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.508+0000 7fb39a3b8640 1 --2- 172.21.15.57:0/510092114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3940c42a0 0x7fb39415b490 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:28.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.508+0000 7fb39a3b8640 1 -- 172.21.15.57:0/510092114 >> 172.21.15.57:0/510092114 conn(0x7fb3940715c0 msgr2=0x7fb394152eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:28.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.512+0000 7fb39a3b8640 1 -- 172.21.15.57:0/510092114 shutdown_connections 2023-10-30T14:42:28.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:28.512+0000 7fb39a3b8640 1 -- 172.21.15.57:0/510092114 wait complete. 2023-10-30T14:42:28.534 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:42:30.535 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:30.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.688+0000 7eff13609640 1 Processor -- start 2023-10-30T14:42:30.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.688+0000 7eff13609640 1 -- start start 2023-10-30T14:42:30.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.688+0000 7eff13609640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff0c0c6320 0x7eff0c06fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:30.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.688+0000 7eff13609640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7eff0c06be90 0x7eff0c06c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:30.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.688+0000 7eff13609640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0c0c55f0 con 0x7eff0c14fc50 2023-10-30T14:42:30.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.688+0000 7eff13609640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff0c0c5f00 con 0x7eff0c06be90 2023-10-30T14:42:30.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.688+0000 7eff13609640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff0c1567b0 con 0x7eff0c0c6320 2023-10-30T14:42:30.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.688+0000 7eff113a5640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7eff0c14fc50 0x7eff0c070e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48872/0 (socket says 172.21.15.57:48872) 2023-10-30T14:42:30.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.688+0000 7eff113a5640 1 -- 172.21.15.57:0/4202568814 learned_addr learned my addr 172.21.15.57:0/4202568814 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:42:30.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.688+0000 7eff11ba6640 1 --2- 172.21.15.57:0/4202568814 >> v2:172.21.15.57:3300/0 conn(0x7eff0c06be90 0x7eff0c06c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:30.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.688+0000 7eff10ba4640 1 --2- 172.21.15.57:0/4202568814 >> v2:172.21.15.88:3300/0 conn(0x7eff0c0c6320 0x7eff0c06fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:30.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.688+0000 7eff03fff640 1 -- 172.21.15.57:0/4202568814 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839545421 0 0) 0x7eff0c0c55f0 con 0x7eff0c14fc50 2023-10-30T14:42:30.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.688+0000 7eff03fff640 1 -- 172.21.15.57:0/4202568814 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efef8003530 con 0x7eff0c14fc50 2023-10-30T14:42:30.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.688+0000 7eff11ba6640 1 -- 172.21.15.57:0/4202568814 >> v2:172.21.15.88:3300/0 conn(0x7eff0c0c6320 msgr2=0x7eff0c06fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:30.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.688+0000 7eff11ba6640 1 --2- 172.21.15.57:0/4202568814 >> v2:172.21.15.88:3300/0 conn(0x7eff0c0c6320 0x7eff0c06fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:30.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.688+0000 7eff11ba6640 1 -- 172.21.15.57:0/4202568814 >> v1:172.21.15.57:6789/0 conn(0x7eff0c14fc50 legacy=0x7eff0c070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:30.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.688+0000 7eff11ba6640 1 -- 172.21.15.57:0/4202568814 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0c156a70 con 0x7eff0c06be90 2023-10-30T14:42:30.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.688+0000 7eff10ba4640 1 --2- 172.21.15.57:0/4202568814 >> v2:172.21.15.88:3300/0 conn(0x7eff0c0c6320 0x7eff0c06fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:42:30.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.692+0000 7eff11ba6640 1 --2- 172.21.15.57:0/4202568814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff0c06be90 0x7eff0c06c320 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7eff0c0667f0 tx=0x7eff0401d9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f9344f5dcdd2955 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:30.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.692+0000 7eff03fff640 1 -- 172.21.15.57:0/4202568814 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0400a2d0 con 0x7eff0c06be90 2023-10-30T14:42:30.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.692+0000 7eff03fff640 1 -- 172.21.15.57:0/4202568814 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff040a67e0 con 0x7eff0c06be90 2023-10-30T14:42:30.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.692+0000 7eff03fff640 1 -- 172.21.15.57:0/4202568814 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff040a6b70 con 0x7eff0c06be90 2023-10-30T14:42:30.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.692+0000 7eff13609640 1 -- 172.21.15.57:0/4202568814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff0c06be90 msgr2=0x7eff0c06c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:30.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.692+0000 7eff13609640 1 --2- 172.21.15.57:0/4202568814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff0c06be90 0x7eff0c06c320 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7eff0c0667f0 tx=0x7eff0401d9a0 comp rx=0 tx=0).stop 2023-10-30T14:42:30.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.692+0000 7eff13609640 1 -- 172.21.15.57:0/4202568814 shutdown_connections 2023-10-30T14:42:30.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.692+0000 7eff13609640 1 --2- 172.21.15.57:0/4202568814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff0c06be90 0x7eff0c06c320 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:30.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.692+0000 7eff13609640 1 --2- 172.21.15.57:0/4202568814 >> v2:172.21.15.88:3300/0 conn(0x7eff0c0c6320 0x7eff0c06fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:30.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.692+0000 7eff13609640 1 -- 172.21.15.57:0/4202568814 >> 172.21.15.57:0/4202568814 conn(0x7eff0c0cac70 msgr2=0x7eff0c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:30.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.692+0000 7eff13609640 1 -- 172.21.15.57:0/4202568814 shutdown_connections 2023-10-30T14:42:30.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.692+0000 7eff13609640 1 -- 172.21.15.57:0/4202568814 wait complete. 2023-10-30T14:42:30.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff13609640 1 Processor -- start 2023-10-30T14:42:30.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff13609640 1 -- start start 2023-10-30T14:42:30.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff13609640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff0c06be90 0x7eff0c136950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:30.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff13609640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff0c0c6320 0x7eff0c140500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:30.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff13609640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff0c14fc50 0x7eff0c140a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:30.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff13609640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c071730 con 0x7eff0c14fc50 2023-10-30T14:42:30.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff13609640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c1f33b0 con 0x7eff0c0c6320 2023-10-30T14:42:30.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff13609640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c13e940 con 0x7eff0c06be90 2023-10-30T14:42:30.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff11ba6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff0c14fc50 0x7eff0c140a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:30.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff11ba6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff0c14fc50 0x7eff0c140a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42380/0 (socket says 172.21.15.57:42380) 2023-10-30T14:42:30.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff11ba6640 1 -- 172.21.15.57:0/1512743516 learned_addr learned my addr 172.21.15.57:0/1512743516 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:30.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff113a5640 1 --2- 172.21.15.57:0/1512743516 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff0c06be90 0x7eff0c136950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:30.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff10ba4640 1 --2- 172.21.15.57:0/1512743516 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff0c0c6320 0x7eff0c140500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:30.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff11ba6640 1 -- 172.21.15.57:0/1512743516 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff0c06be90 msgr2=0x7eff0c136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:30.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff11ba6640 1 --2- 172.21.15.57:0/1512743516 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff0c06be90 0x7eff0c136950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:30.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff11ba6640 1 -- 172.21.15.57:0/1512743516 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff0c0c6320 msgr2=0x7eff0c140500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:30.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff11ba6640 1 --2- 172.21.15.57:0/1512743516 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff0c0c6320 0x7eff0c140500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:30.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff11ba6640 1 -- 172.21.15.57:0/1512743516 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0400ab80 con 0x7eff0c14fc50 2023-10-30T14:42:30.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff11ba6640 1 --2- 172.21.15.57:0/1512743516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff0c14fc50 0x7eff0c140a40 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7eff040b0040 tx=0x7eff0401a740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:30.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff01ffb640 1 -- 172.21.15.57:0/1512743516 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff040b7680 con 0x7eff0c14fc50 2023-10-30T14:42:30.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff01ffb640 1 -- 172.21.15.57:0/1512743516 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff040b77e0 con 0x7eff0c14fc50 2023-10-30T14:42:30.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff01ffb640 1 -- 172.21.15.57:0/1512743516 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff040b7b90 con 0x7eff0c14fc50 2023-10-30T14:42:30.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.696+0000 7eff13609640 1 -- 172.21.15.57:0/1512743516 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0c13ec20 con 0x7eff0c14fc50 2023-10-30T14:42:30.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.700+0000 7eff13609640 1 -- 172.21.15.57:0/1512743516 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0c142790 con 0x7eff0c14fc50 2023-10-30T14:42:30.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.700+0000 7eff13609640 1 -- 172.21.15.57:0/1512743516 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff0c0718a0 con 0x7eff0c14fc50 2023-10-30T14:42:30.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.700+0000 7eff01ffb640 1 -- 172.21.15.57:0/1512743516 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7eff040b7cf0 con 0x7eff0c14fc50 2023-10-30T14:42:30.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.700+0000 7eff01ffb640 1 --2- 172.21.15.57:0/1512743516 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efef00664a0 0x7efef0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:30.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.700+0000 7eff01ffb640 1 -- 172.21.15.57:0/1512743516 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7eff0411f4c0 con 0x7eff0c14fc50 2023-10-30T14:42:30.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.700+0000 7eff113a5640 1 --2- 172.21.15.57:0/1512743516 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efef00664a0 0x7efef0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:30.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.704+0000 7eff113a5640 1 --2- 172.21.15.57:0/1512743516 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efef00664a0 0x7efef0068960 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7eff080011f0 tx=0x7eff08007d40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:30.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.704+0000 7eff01ffb640 1 -- 172.21.15.57:0/1512743516 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7eff040ec570 con 0x7eff0c14fc50 2023-10-30T14:42:30.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.892+0000 7eff13609640 1 -- 172.21.15.57:0/1512743516 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff0c04e170 con 0x7eff0c14fc50 2023-10-30T14:42:30.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.896+0000 7eff01ffb640 1 -- 172.21.15.57:0/1512743516 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6535 (secure 0 0 0) 0x7eff040efb90 con 0x7eff0c14fc50 2023-10-30T14:42:30.899 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:30.899 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6356,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:41:28.336080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:30.900 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 150 2023-10-30T14:42:30.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.896+0000 7eff13609640 1 -- 172.21.15.57:0/1512743516 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efef00664a0 msgr2=0x7efef0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:30.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.896+0000 7eff13609640 1 --2- 172.21.15.57:0/1512743516 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efef00664a0 0x7efef0068960 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7eff080011f0 tx=0x7eff08007d40 comp rx=0 tx=0).stop 2023-10-30T14:42:30.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.896+0000 7eff13609640 1 -- 172.21.15.57:0/1512743516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff0c14fc50 msgr2=0x7eff0c140a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:30.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.896+0000 7eff13609640 1 --2- 172.21.15.57:0/1512743516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff0c14fc50 0x7eff0c140a40 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7eff040b0040 tx=0x7eff0401a740 comp rx=0 tx=0).stop 2023-10-30T14:42:30.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.896+0000 7eff13609640 1 -- 172.21.15.57:0/1512743516 shutdown_connections 2023-10-30T14:42:30.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.896+0000 7eff13609640 1 --2- 172.21.15.57:0/1512743516 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efef00664a0 0x7efef0068960 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:30.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.896+0000 7eff13609640 1 --2- 172.21.15.57:0/1512743516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff0c14fc50 0x7eff0c140a40 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:30.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.896+0000 7eff13609640 1 --2- 172.21.15.57:0/1512743516 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff0c0c6320 0x7eff0c140500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:30.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.896+0000 7eff13609640 1 --2- 172.21.15.57:0/1512743516 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff0c06be90 0x7eff0c136950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:30.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.896+0000 7eff13609640 1 -- 172.21.15.57:0/1512743516 >> 172.21.15.57:0/1512743516 conn(0x7eff0c0cac70 msgr2=0x7eff0c0c92b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:30.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.896+0000 7eff13609640 1 -- 172.21.15.57:0/1512743516 shutdown_connections 2023-10-30T14:42:30.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:30.896+0000 7eff13609640 1 -- 172.21.15.57:0/1512743516 wait complete. 2023-10-30T14:42:30.920 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:42:32.920 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:33.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.072+0000 7fbfc15e4640 1 Processor -- start 2023-10-30T14:42:33.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.072+0000 7fbfc15e4640 1 -- start start 2023-10-30T14:42:33.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.072+0000 7fbfc15e4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbfbc0c62b0 0x7fbfbc1597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:33.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.072+0000 7fbfc15e4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfbc0cb180 0x7fbfbc06ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:33.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.072+0000 7fbfc15e4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfbc15a770 con 0x7fbfbc153b50 2023-10-30T14:42:33.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.072+0000 7fbfc15e4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfbc159cf0 con 0x7fbfbc0c62b0 2023-10-30T14:42:33.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.072+0000 7fbfc15e4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfbc06aa30 con 0x7fbfbc0cb180 2023-10-30T14:42:33.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.072+0000 7fbfbb7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbfbc0c62b0 0x7fbfbc1597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:33.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.072+0000 7fbfbb7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbfbc0c62b0 0x7fbfbc1597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42390/0 (socket says 172.21.15.57:42390) 2023-10-30T14:42:33.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.072+0000 7fbfbb7fe640 1 -- 172.21.15.57:0/3861108271 learned_addr learned my addr 172.21.15.57:0/3861108271 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:33.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.072+0000 7fbfc0de3640 1 --2- 172.21.15.57:0/3861108271 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfbc0cb180 0x7fbfbc06ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:33.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.072+0000 7fbfbb7fe640 1 -- 172.21.15.57:0/3861108271 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfbc0cb180 msgr2=0x7fbfbc06ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:33.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.072+0000 7fbfbb7fe640 1 --2- 172.21.15.57:0/3861108271 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfbc0cb180 0x7fbfbc06ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:33.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.072+0000 7fbfbb7fe640 1 -- 172.21.15.57:0/3861108271 >> v1:172.21.15.88:6789/0 conn(0x7fbfbc153b50 legacy=0x7fbfbc0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:33.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.072+0000 7fbfbb7fe640 1 -- 172.21.15.57:0/3861108271 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfbc0c9d40 con 0x7fbfbc0c62b0 2023-10-30T14:42:33.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.076+0000 7fbfbb7fe640 1 --2- 172.21.15.57:0/3861108271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfbc0c62b0 0x7fbfbc1597b0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009870 tx=0x7fbfb001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7cf3b598d49b449 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:33.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.076+0000 7fbfbaffd640 1 -- 172.21.15.57:0/3861108271 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb00a8a30 con 0x7fbfbc0c62b0 2023-10-30T14:42:33.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.076+0000 7fbfbaffd640 1 -- 172.21.15.57:0/3861108271 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb00a8b90 con 0x7fbfbc0c62b0 2023-10-30T14:42:33.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.076+0000 7fbfbaffd640 1 -- 172.21.15.57:0/3861108271 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb00b0440 con 0x7fbfbc0c62b0 2023-10-30T14:42:33.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.076+0000 7fbfc15e4640 1 -- 172.21.15.57:0/3861108271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfbc0c62b0 msgr2=0x7fbfbc1597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:33.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.076+0000 7fbfc15e4640 1 --2- 172.21.15.57:0/3861108271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfbc0c62b0 0x7fbfbc1597b0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009870 tx=0x7fbfb001ed30 comp rx=0 tx=0).stop 2023-10-30T14:42:33.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.076+0000 7fbfc15e4640 1 -- 172.21.15.57:0/3861108271 shutdown_connections 2023-10-30T14:42:33.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.076+0000 7fbfc15e4640 1 --2- 172.21.15.57:0/3861108271 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfbc0cb180 0x7fbfbc06ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:33.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.076+0000 7fbfc15e4640 1 --2- 172.21.15.57:0/3861108271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfbc0c62b0 0x7fbfbc1597b0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:33.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.076+0000 7fbfc15e4640 1 -- 172.21.15.57:0/3861108271 >> 172.21.15.57:0/3861108271 conn(0x7fbfbc065a80 msgr2=0x7fbfbc065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:33.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.076+0000 7fbfc15e4640 1 -- 172.21.15.57:0/3861108271 shutdown_connections 2023-10-30T14:42:33.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.076+0000 7fbfc15e4640 1 -- 172.21.15.57:0/3861108271 wait complete. 2023-10-30T14:42:33.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfc15e4640 1 Processor -- start 2023-10-30T14:42:33.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfc15e4640 1 -- start start 2023-10-30T14:42:33.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfc15e4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfbc0c62b0 0x7fbfbc0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:33.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfc15e4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfbc0cb180 0x7fbfbc0c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:33.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfc15e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfbc153b50 0x7fbfbc1f1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:33.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfc15e4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfbc158a90 con 0x7fbfbc0c62b0 2023-10-30T14:42:33.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfc15e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfbc159960 con 0x7fbfbc153b50 2023-10-30T14:42:33.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfc15e4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfbc159680 con 0x7fbfbc0cb180 2023-10-30T14:42:33.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfbbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfbc0c62b0 0x7fbfbc0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:33.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfbbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfbc0c62b0 0x7fbfbc0ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42406/0 (socket says 172.21.15.57:42406) 2023-10-30T14:42:33.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfbbfff640 1 -- 172.21.15.57:0/3875741638 learned_addr learned my addr 172.21.15.57:0/3875741638 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:33.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfbb7fe640 1 --2- 172.21.15.57:0/3875741638 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfbc0cb180 0x7fbfbc0c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:33.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfc0de3640 1 --2- 172.21.15.57:0/3875741638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfbc153b50 0x7fbfbc1f1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:33.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfbbfff640 1 -- 172.21.15.57:0/3875741638 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfbc0cb180 msgr2=0x7fbfbc0c2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:33.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfbbfff640 1 --2- 172.21.15.57:0/3875741638 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfbc0cb180 0x7fbfbc0c2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:33.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfbbfff640 1 -- 172.21.15.57:0/3875741638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfbc153b50 msgr2=0x7fbfbc1f1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:33.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfbbfff640 1 --2- 172.21.15.57:0/3875741638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfbc153b50 0x7fbfbc1f1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:33.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfbbfff640 1 -- 172.21.15.57:0/3875741638 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb00094d0 con 0x7fbfbc0c62b0 2023-10-30T14:42:33.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfc0de3640 1 --2- 172.21.15.57:0/3875741638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfbc153b50 0x7fbfbc1f1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:42:33.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfbbfff640 1 --2- 172.21.15.57:0/3875741638 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfbc0c62b0 0x7fbfbc0ce4e0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00c3c0 tx=0x7fbfac00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:33.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfb8ff9640 1 -- 172.21.15.57:0/3875741638 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac0052d0 con 0x7fbfbc0c62b0 2023-10-30T14:42:33.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfb8ff9640 1 -- 172.21.15.57:0/3875741638 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfac005430 con 0x7fbfbc0c62b0 2023-10-30T14:42:33.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfc15e4640 1 -- 172.21.15.57:0/3875741638 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfbc158160 con 0x7fbfbc0c62b0 2023-10-30T14:42:33.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfb8ff9640 1 -- 172.21.15.57:0/3875741638 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac003840 con 0x7fbfbc0c62b0 2023-10-30T14:42:33.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.080+0000 7fbfc15e4640 1 -- 172.21.15.57:0/3875741638 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfbc158720 con 0x7fbfbc0c62b0 2023-10-30T14:42:33.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.084+0000 7fbfc15e4640 1 -- 172.21.15.57:0/3875741638 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfbc067540 con 0x7fbfbc0c62b0 2023-10-30T14:42:33.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.084+0000 7fbfb8ff9640 1 -- 172.21.15.57:0/3875741638 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbfac0039a0 con 0x7fbfbc0c62b0 2023-10-30T14:42:33.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.084+0000 7fbfb8ff9640 1 --2- 172.21.15.57:0/3875741638 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbfa80664a0 0x7fbfa8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:33.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.084+0000 7fbfb8ff9640 1 -- 172.21.15.57:0/3875741638 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7029+0+0 (secure 0 0 0) 0x7fbfac0836e0 con 0x7fbfbc0c62b0 2023-10-30T14:42:33.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.084+0000 7fbfbb7fe640 1 --2- 172.21.15.57:0/3875741638 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbfa80664a0 0x7fbfa8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:33.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.084+0000 7fbfbb7fe640 1 --2- 172.21.15.57:0/3875741638 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbfa80664a0 0x7fbfa8068960 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00b2000 tx=0x7fbfb0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:33.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.088+0000 7fbfb8ff9640 1 -- 172.21.15.57:0/3875741638 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbfac0506c0 con 0x7fbfbc0c62b0 2023-10-30T14:42:33.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.276+0000 7fbfc15e4640 1 -- 172.21.15.57:0/3875741638 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfbc15a740 con 0x7fbfbc0c62b0 2023-10-30T14:42:33.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.284+0000 7fbfb8ff9640 1 -- 172.21.15.57:0/3875741638 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6535 (secure 0 0 0) 0x7fbfac053ce0 con 0x7fbfbc0c62b0 2023-10-30T14:42:33.288 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:33.288 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6356,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:41:28.336080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6414,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6834/2321408149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2321408149},{"type":"v1","addr":"172.21.15.57:6834","nonce":2321408149}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:33.288 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 150 2023-10-30T14:42:33.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.284+0000 7fbfc15e4640 1 -- 172.21.15.57:0/3875741638 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbfa80664a0 msgr2=0x7fbfa8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:33.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.284+0000 7fbfc15e4640 1 --2- 172.21.15.57:0/3875741638 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbfa80664a0 0x7fbfa8068960 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00b2000 tx=0x7fbfb0009040 comp rx=0 tx=0).stop 2023-10-30T14:42:33.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.284+0000 7fbfc15e4640 1 -- 172.21.15.57:0/3875741638 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfbc0c62b0 msgr2=0x7fbfbc0ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:33.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.284+0000 7fbfc15e4640 1 --2- 172.21.15.57:0/3875741638 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfbc0c62b0 0x7fbfbc0ce4e0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00c3c0 tx=0x7fbfac00c880 comp rx=0 tx=0).stop 2023-10-30T14:42:33.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.284+0000 7fbfc15e4640 1 -- 172.21.15.57:0/3875741638 shutdown_connections 2023-10-30T14:42:33.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.284+0000 7fbfc15e4640 1 --2- 172.21.15.57:0/3875741638 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbfa80664a0 0x7fbfa8068960 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:33.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.284+0000 7fbfc15e4640 1 --2- 172.21.15.57:0/3875741638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfbc153b50 0x7fbfbc1f1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:33.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.284+0000 7fbfc15e4640 1 --2- 172.21.15.57:0/3875741638 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfbc0cb180 0x7fbfbc0c2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:33.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.284+0000 7fbfc15e4640 1 --2- 172.21.15.57:0/3875741638 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfbc0c62b0 0x7fbfbc0ce4e0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:33.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.284+0000 7fbfc15e4640 1 -- 172.21.15.57:0/3875741638 >> 172.21.15.57:0/3875741638 conn(0x7fbfbc065a80 msgr2=0x7fbfbc14e310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:33.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.284+0000 7fbfc15e4640 1 -- 172.21.15.57:0/3875741638 shutdown_connections 2023-10-30T14:42:33.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:33.284+0000 7fbfc15e4640 1 -- 172.21.15.57:0/3875741638 wait complete. 2023-10-30T14:42:33.309 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:42:35.310 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:35.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.460+0000 7f7cf7b65640 1 Processor -- start 2023-10-30T14:42:35.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.460+0000 7f7cf7b65640 1 -- start start 2023-10-30T14:42:35.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.460+0000 7f7cf7b65640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7cf006b760 0x7f7cf015e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:35.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.460+0000 7f7cf7b65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cf006bb40 0x7f7cf0157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:35.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.460+0000 7f7cf7b65640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf0071e00 con 0x7f7cf006c420 2023-10-30T14:42:35.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.460+0000 7f7cf7b65640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf0070020 con 0x7f7cf006b760 2023-10-30T14:42:35.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.460+0000 7f7cf7b65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf015eb90 con 0x7f7cf006bb40 2023-10-30T14:42:35.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.460+0000 7f7cf6102640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f7cf006c420 0x7f7cf00c57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:59230/0 (socket says 172.21.15.57:59230) 2023-10-30T14:42:35.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.460+0000 7f7cf6102640 1 -- 172.21.15.57:0/2453677881 learned_addr learned my addr 172.21.15.57:0/2453677881 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:42:35.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.460+0000 7f7cf5100640 1 --2- 172.21.15.57:0/2453677881 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cf006bb40 0x7f7cf0157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:35.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf5901640 1 --2- 172.21.15.57:0/2453677881 >> v2:172.21.15.88:3300/0 conn(0x7f7cf006b760 0x7f7cf015e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:35.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf48ff640 1 -- 172.21.15.57:0/2453677881 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398158674 0 0) 0x7f7cf0071e00 con 0x7f7cf006c420 2023-10-30T14:42:35.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf48ff640 1 -- 172.21.15.57:0/2453677881 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cdc003530 con 0x7f7cf006c420 2023-10-30T14:42:35.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf48ff640 1 -- 172.21.15.57:0/2453677881 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 312845176 0 0) 0x7f7cdc003530 con 0x7f7cf006c420 2023-10-30T14:42:35.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf48ff640 1 -- 172.21.15.57:0/2453677881 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cf006bb40 msgr2=0x7f7cf0157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:35.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf48ff640 1 --2- 172.21.15.57:0/2453677881 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cf006bb40 0x7f7cf0157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:35.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf48ff640 1 -- 172.21.15.57:0/2453677881 >> v2:172.21.15.88:3300/0 conn(0x7f7cf006b760 msgr2=0x7f7cf015e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:35.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf48ff640 1 --2- 172.21.15.57:0/2453677881 >> v2:172.21.15.88:3300/0 conn(0x7f7cf006b760 0x7f7cf015e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:35.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf48ff640 1 -- 172.21.15.57:0/2453677881 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf0070f10 con 0x7f7cf006c420 2023-10-30T14:42:35.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf48ff640 1 -- 172.21.15.57:0/2453677881 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f7ce8001a00 con 0x7f7cf006c420 2023-10-30T14:42:35.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf48ff640 1 -- 172.21.15.57:0/2453677881 >> v1:172.21.15.57:6789/0 conn(0x7f7cf006c420 legacy=0x7f7cf00c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:35.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf48ff640 1 --2- 172.21.15.57:0/2453677881 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cdc003980 0x7f7cdc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:35.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf48ff640 1 --2- 172.21.15.57:0/2453677881 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cf006bb40 0x7f7cdc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:35.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf48ff640 1 --2- 172.21.15.57:0/2453677881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cdc004d10 0x7f7cdc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:35.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf48ff640 1 -- 172.21.15.57:0/2453677881 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf0070020 con 0x7f7cdc004d10 2023-10-30T14:42:35.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf48ff640 1 -- 172.21.15.57:0/2453677881 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf015eb90 con 0x7f7cdc003980 2023-10-30T14:42:35.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf48ff640 1 -- 172.21.15.57:0/2453677881 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc00b990 con 0x7f7cf006bb40 2023-10-30T14:42:35.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf6102640 1 --2- 172.21.15.57:0/2453677881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cdc004d10 0x7f7cdc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:35.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf5901640 1 --2- 172.21.15.57:0/2453677881 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cdc003980 0x7f7cdc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:35.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf5100640 1 --2- 172.21.15.57:0/2453677881 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cf006bb40 0x7f7cdc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:35.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf6102640 1 -- 172.21.15.57:0/2453677881 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cf006bb40 msgr2=0x7f7cdc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:35.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf6102640 1 --2- 172.21.15.57:0/2453677881 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cf006bb40 0x7f7cdc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:35.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf6102640 1 -- 172.21.15.57:0/2453677881 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cdc003980 msgr2=0x7f7cdc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:35.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf6102640 1 --2- 172.21.15.57:0/2453677881 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cdc003980 0x7f7cdc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:35.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.464+0000 7f7cf6102640 1 -- 172.21.15.57:0/2453677881 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cdc003530 con 0x7f7cdc004d10 2023-10-30T14:42:35.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.468+0000 7f7cf6102640 1 --2- 172.21.15.57:0/2453677881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cdc004d10 0x7f7cdc005190 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f7ce800a830 tx=0x7f7ce801d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20f80d9676a85479 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:35.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.468+0000 7f7cf48ff640 1 -- 172.21.15.57:0/2453677881 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80ad630 con 0x7f7cdc004d10 2023-10-30T14:42:35.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.468+0000 7f7cf48ff640 1 -- 172.21.15.57:0/2453677881 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce80a8070 con 0x7f7cdc004d10 2023-10-30T14:42:35.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.468+0000 7f7cf48ff640 1 -- 172.21.15.57:0/2453677881 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80ad9f0 con 0x7f7cdc004d10 2023-10-30T14:42:35.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.468+0000 7f7cf7b65640 1 -- 172.21.15.57:0/2453677881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cdc004d10 msgr2=0x7f7cdc005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:35.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.468+0000 7f7cf7b65640 1 --2- 172.21.15.57:0/2453677881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cdc004d10 0x7f7cdc005190 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f7ce800a830 tx=0x7f7ce801d340 comp rx=0 tx=0).stop 2023-10-30T14:42:35.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.468+0000 7f7cf6102640 1 -- 172.21.15.57:0/2453677881 reap_dead start 2023-10-30T14:42:35.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.468+0000 7f7cf7b65640 1 -- 172.21.15.57:0/2453677881 shutdown_connections 2023-10-30T14:42:35.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.468+0000 7f7cf7b65640 1 -- 172.21.15.57:0/2453677881 >> 172.21.15.57:0/2453677881 conn(0x7f7cf00cac70 msgr2=0x7f7cf00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:35.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.468+0000 7f7cf7b65640 1 -- 172.21.15.57:0/2453677881 shutdown_connections 2023-10-30T14:42:35.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.468+0000 7f7cf7b65640 1 -- 172.21.15.57:0/2453677881 wait complete. 2023-10-30T14:42:35.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.468+0000 7f7cf7b65640 1 Processor -- start 2023-10-30T14:42:35.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.468+0000 7f7cf7b65640 1 -- start start 2023-10-30T14:42:35.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.468+0000 7f7cf7b65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cdc004d10 0x7f7cf01f6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:35.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.468+0000 7f7cf7b65640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cf015ed90 0x7f7cf0150370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:35.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.468+0000 7f7cf7b65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cf014ddc0 0x7f7cf014e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:35.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7cf7b65640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf0161a60 con 0x7f7cf015ed90 2023-10-30T14:42:35.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7cf7b65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf0158a80 con 0x7f7cdc004d10 2023-10-30T14:42:35.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7cf7b65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf015f660 con 0x7f7cf014ddc0 2023-10-30T14:42:35.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7cf5100640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cf015ed90 0x7f7cf0150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:35.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7cf5100640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cf015ed90 0x7f7cf0150370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46996/0 (socket says 172.21.15.57:46996) 2023-10-30T14:42:35.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7cf5100640 1 -- 172.21.15.57:0/2435128170 learned_addr learned my addr 172.21.15.57:0/2435128170 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:35.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7cf6102640 1 --2- 172.21.15.57:0/2435128170 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cf014ddc0 0x7f7cf014e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:35.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7cf5901640 1 --2- 172.21.15.57:0/2435128170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cdc004d10 0x7f7cf01f6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:35.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7cf5100640 1 -- 172.21.15.57:0/2435128170 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cf014ddc0 msgr2=0x7f7cf014e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:35.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7cf5100640 1 --2- 172.21.15.57:0/2435128170 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cf014ddc0 0x7f7cf014e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:35.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7cf5100640 1 -- 172.21.15.57:0/2435128170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cdc004d10 msgr2=0x7f7cf01f6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:35.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7cf5100640 1 --2- 172.21.15.57:0/2435128170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cdc004d10 0x7f7cf01f6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:35.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7cf5100640 1 -- 172.21.15.57:0/2435128170 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf00c3cb0 con 0x7f7cf015ed90 2023-10-30T14:42:35.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7cf5100640 1 --2- 172.21.15.57:0/2435128170 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cf015ed90 0x7f7cf0150370 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0011930 tx=0x7f7ce0011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:35.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7ce67fc640 1 -- 172.21.15.57:0/2435128170 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce00095b0 con 0x7f7cf015ed90 2023-10-30T14:42:35.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7ce67fc640 1 -- 172.21.15.57:0/2435128170 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce000faf0 con 0x7f7cf015ed90 2023-10-30T14:42:35.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7ce67fc640 1 -- 172.21.15.57:0/2435128170 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce000fe80 con 0x7f7cf015ed90 2023-10-30T14:42:35.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7cf7b65640 1 -- 172.21.15.57:0/2435128170 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf00c3f90 con 0x7f7cf015ed90 2023-10-30T14:42:35.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7cf7b65640 1 -- 172.21.15.57:0/2435128170 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf015f900 con 0x7f7cf015ed90 2023-10-30T14:42:35.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7cf7b65640 1 -- 172.21.15.57:0/2435128170 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf004e170 con 0x7f7cf015ed90 2023-10-30T14:42:35.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.472+0000 7f7ce67fc640 1 -- 172.21.15.57:0/2435128170 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7ce0014440 con 0x7f7cf015ed90 2023-10-30T14:42:35.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.476+0000 7f7ce67fc640 1 --2- 172.21.15.57:0/2435128170 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7cd40664d0 0x7f7cd4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:35.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.476+0000 7f7ce67fc640 1 -- 172.21.15.57:0/2435128170 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f7ce0084d70 con 0x7f7cf015ed90 2023-10-30T14:42:35.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.476+0000 7f7cf5901640 1 --2- 172.21.15.57:0/2435128170 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7cd40664d0 0x7f7cd4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:35.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.476+0000 7f7cf5901640 1 --2- 172.21.15.57:0/2435128170 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7cd40664d0 0x7f7cd4068990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009b00 tx=0x7f7cec008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:35.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.480+0000 7f7ce67fc640 1 -- 172.21.15.57:0/2435128170 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7ce0051bd0 con 0x7f7cf015ed90 2023-10-30T14:42:35.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.672+0000 7f7cf7b65640 1 -- 172.21.15.57:0/2435128170 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cf01556b0 con 0x7f7cf015ed90 2023-10-30T14:42:35.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.672+0000 7f7ce67fc640 1 -- 172.21.15.57:0/2435128170 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+5740 (secure 0 0 0) 0x7f7ce0022090 con 0x7f7cf015ed90 2023-10-30T14:42:35.677 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:35.677 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:42:35.131527+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:35.677 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 151 2023-10-30T14:42:35.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.676+0000 7f7cf7b65640 1 -- 172.21.15.57:0/2435128170 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7cd40664d0 msgr2=0x7f7cd4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:35.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.676+0000 7f7cf7b65640 1 --2- 172.21.15.57:0/2435128170 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7cd40664d0 0x7f7cd4068990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009b00 tx=0x7f7cec008040 comp rx=0 tx=0).stop 2023-10-30T14:42:35.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.676+0000 7f7cf7b65640 1 -- 172.21.15.57:0/2435128170 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cf015ed90 msgr2=0x7f7cf0150370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:35.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.676+0000 7f7cf7b65640 1 --2- 172.21.15.57:0/2435128170 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cf015ed90 0x7f7cf0150370 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0011930 tx=0x7f7ce0011df0 comp rx=0 tx=0).stop 2023-10-30T14:42:35.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.676+0000 7f7cf7b65640 1 -- 172.21.15.57:0/2435128170 shutdown_connections 2023-10-30T14:42:35.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.676+0000 7f7cf7b65640 1 --2- 172.21.15.57:0/2435128170 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7cd40664d0 0x7f7cd4068990 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:35.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.676+0000 7f7cf7b65640 1 --2- 172.21.15.57:0/2435128170 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cf014ddc0 0x7f7cf014e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:35.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.676+0000 7f7cf7b65640 1 --2- 172.21.15.57:0/2435128170 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cf015ed90 0x7f7cf0150370 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:35.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.676+0000 7f7cf7b65640 1 --2- 172.21.15.57:0/2435128170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cdc004d10 0x7f7cf01f6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:35.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.676+0000 7f7cf7b65640 1 -- 172.21.15.57:0/2435128170 >> 172.21.15.57:0/2435128170 conn(0x7f7cf00cac70 msgr2=0x7f7cf01577f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:35.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.676+0000 7f7cf7b65640 1 -- 172.21.15.57:0/2435128170 shutdown_connections 2023-10-30T14:42:35.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.676+0000 7f7cf7b65640 1 -- 172.21.15.57:0/2435128170 wait complete. 2023-10-30T14:42:35.697 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T14:42:35.698 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:42:35.698 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:35.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.840+0000 7fa6e01c5640 1 Processor -- start 2023-10-30T14:42:35.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.840+0000 7fa6e01c5640 1 -- start start 2023-10-30T14:42:35.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.840+0000 7fa6e01c5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6d8150830 0x7fa6d806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:35.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.840+0000 7fa6e01c5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa6d80cb7d0 0x7fa6d80c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:35.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.840+0000 7fa6e01c5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d80c2310 con 0x7fa6d80c3e40 2023-10-30T14:42:35.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.840+0000 7fa6e01c5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d8156710 con 0x7fa6d80cb7d0 2023-10-30T14:42:35.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.840+0000 7fa6e01c5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d80c4360 con 0x7fa6d8150830 2023-10-30T14:42:35.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.844+0000 7fa6de762640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa6d80c3e40 0x7fa6d8070520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:59240/0 (socket says 172.21.15.57:59240) 2023-10-30T14:42:35.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.844+0000 7fa6de762640 1 -- 172.21.15.57:0/3220649302 learned_addr learned my addr 172.21.15.57:0/3220649302 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:42:35.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.844+0000 7fa6ddf61640 1 --2- 172.21.15.57:0/3220649302 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6d8150830 0x7fa6d806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:35.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.844+0000 7fa6dd760640 1 --2- 172.21.15.57:0/3220649302 >> v2:172.21.15.88:3300/0 conn(0x7fa6d80cb7d0 0x7fa6d80c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:35.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.844+0000 7fa6dcf5f640 1 -- 172.21.15.57:0/3220649302 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1922400497 0 0) 0x7fa6d80c2310 con 0x7fa6d80c3e40 2023-10-30T14:42:35.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.844+0000 7fa6dcf5f640 1 -- 172.21.15.57:0/3220649302 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c4003530 con 0x7fa6d80c3e40 2023-10-30T14:42:35.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.844+0000 7fa6ddf61640 1 -- 172.21.15.57:0/3220649302 >> v2:172.21.15.88:3300/0 conn(0x7fa6d80cb7d0 msgr2=0x7fa6d80c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:35.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.844+0000 7fa6ddf61640 1 --2- 172.21.15.57:0/3220649302 >> v2:172.21.15.88:3300/0 conn(0x7fa6d80cb7d0 0x7fa6d80c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:35.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.844+0000 7fa6ddf61640 1 -- 172.21.15.57:0/3220649302 >> v1:172.21.15.57:6789/0 conn(0x7fa6d80c3e40 legacy=0x7fa6d8070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:35.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.844+0000 7fa6ddf61640 1 -- 172.21.15.57:0/3220649302 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d81569b0 con 0x7fa6d8150830 2023-10-30T14:42:35.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.844+0000 7fa6ddf61640 1 --2- 172.21.15.57:0/3220649302 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6d8150830 0x7fa6d806b500 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa6d40097c0 tx=0x7fa6d401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=753404247f29a084 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:35.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.844+0000 7fa6dcf5f640 1 -- 172.21.15.57:0/3220649302 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d40a8970 con 0x7fa6d8150830 2023-10-30T14:42:35.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6dcf5f640 1 -- 172.21.15.57:0/3220649302 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d40a8ad0 con 0x7fa6d8150830 2023-10-30T14:42:35.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6dcf5f640 1 -- 172.21.15.57:0/3220649302 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d40a8e40 con 0x7fa6d8150830 2023-10-30T14:42:35.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6e01c5640 1 -- 172.21.15.57:0/3220649302 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6d8150830 msgr2=0x7fa6d806b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:35.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6e01c5640 1 --2- 172.21.15.57:0/3220649302 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6d8150830 0x7fa6d806b500 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa6d40097c0 tx=0x7fa6d401ec70 comp rx=0 tx=0).stop 2023-10-30T14:42:35.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6e01c5640 1 -- 172.21.15.57:0/3220649302 shutdown_connections 2023-10-30T14:42:35.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6e01c5640 1 --2- 172.21.15.57:0/3220649302 >> v2:172.21.15.88:3300/0 conn(0x7fa6d80cb7d0 0x7fa6d80c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:35.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6e01c5640 1 --2- 172.21.15.57:0/3220649302 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6d8150830 0x7fa6d806b500 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:35.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6e01c5640 1 -- 172.21.15.57:0/3220649302 >> 172.21.15.57:0/3220649302 conn(0x7fa6d80cac70 msgr2=0x7fa6d80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:35.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6e01c5640 1 -- 172.21.15.57:0/3220649302 shutdown_connections 2023-10-30T14:42:35.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6e01c5640 1 -- 172.21.15.57:0/3220649302 wait complete. 2023-10-30T14:42:35.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6e01c5640 1 Processor -- start 2023-10-30T14:42:35.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6e01c5640 1 -- start start 2023-10-30T14:42:35.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6e01c5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6d80c3e40 0x7fa6d81f2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:35.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6e01c5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6d80cb7d0 0x7fa6d81599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:35.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6e01c5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6d8150830 0x7fa6d8159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:35.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6e01c5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d8155fa0 con 0x7fa6d80cb7d0 2023-10-30T14:42:35.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6e01c5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d81564f0 con 0x7fa6d8150830 2023-10-30T14:42:35.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6e01c5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d81f2d50 con 0x7fa6d80c3e40 2023-10-30T14:42:35.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6dd760640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6d80cb7d0 0x7fa6d81599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:35.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6dd760640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6d80cb7d0 0x7fa6d81599f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47002/0 (socket says 172.21.15.57:47002) 2023-10-30T14:42:35.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6dd760640 1 -- 172.21.15.57:0/484647582 learned_addr learned my addr 172.21.15.57:0/484647582 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:35.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6ddf61640 1 --2- 172.21.15.57:0/484647582 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6d80c3e40 0x7fa6d81f2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:35.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.848+0000 7fa6de762640 1 --2- 172.21.15.57:0/484647582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6d8150830 0x7fa6d8159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:35.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.852+0000 7fa6dd760640 1 -- 172.21.15.57:0/484647582 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6d80c3e40 msgr2=0x7fa6d81f2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:35.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.852+0000 7fa6dd760640 1 --2- 172.21.15.57:0/484647582 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6d80c3e40 0x7fa6d81f2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:35.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.852+0000 7fa6dd760640 1 -- 172.21.15.57:0/484647582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6d8150830 msgr2=0x7fa6d8159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:35.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.852+0000 7fa6dd760640 1 --2- 172.21.15.57:0/484647582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6d8150830 0x7fa6d8159f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:35.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.852+0000 7fa6dd760640 1 -- 172.21.15.57:0/484647582 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d4009420 con 0x7fa6d80cb7d0 2023-10-30T14:42:35.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.852+0000 7fa6dd760640 1 --2- 172.21.15.57:0/484647582 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6d80cb7d0 0x7fa6d81599f0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800e930 tx=0x7fa6c800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:35.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.852+0000 7fa6ce7fc640 1 -- 172.21.15.57:0/484647582 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8009690 con 0x7fa6d80cb7d0 2023-10-30T14:42:35.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.852+0000 7fa6ce7fc640 1 -- 172.21.15.57:0/484647582 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c800ca00 con 0x7fa6d80cb7d0 2023-10-30T14:42:35.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.852+0000 7fa6ce7fc640 1 -- 172.21.15.57:0/484647582 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c800cd70 con 0x7fa6d80cb7d0 2023-10-30T14:42:35.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.852+0000 7fa6e01c5640 1 -- 172.21.15.57:0/484647582 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d8156210 con 0x7fa6d80cb7d0 2023-10-30T14:42:35.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.852+0000 7fa6e01c5640 1 -- 172.21.15.57:0/484647582 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d8155850 con 0x7fa6d80cb7d0 2023-10-30T14:42:35.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.852+0000 7fa6e01c5640 1 -- 172.21.15.57:0/484647582 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d804e170 con 0x7fa6d80cb7d0 2023-10-30T14:42:35.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.852+0000 7fa6ce7fc640 1 -- 172.21.15.57:0/484647582 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa6c8013470 con 0x7fa6d80cb7d0 2023-10-30T14:42:35.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.852+0000 7fa6ce7fc640 1 --2- 172.21.15.57:0/484647582 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa6bc0664d0 0x7fa6bc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:35.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.852+0000 7fa6ce7fc640 1 -- 172.21.15.57:0/484647582 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7fa6c80846b0 con 0x7fa6d80cb7d0 2023-10-30T14:42:35.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.856+0000 7fa6ddf61640 1 --2- 172.21.15.57:0/484647582 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa6bc0664d0 0x7fa6bc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:35.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.856+0000 7fa6ddf61640 1 --2- 172.21.15.57:0/484647582 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa6bc0664d0 0x7fa6bc068990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa6d40b2000 tx=0x7fa6d40aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:35.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:35.856+0000 7fa6ce7fc640 1 -- 172.21.15.57:0/484647582 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa6c80515f0 con 0x7fa6d80cb7d0 2023-10-30T14:42:36.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:36.056+0000 7fa6e01c5640 1 -- 172.21.15.57:0/484647582 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6d8156c60 con 0x7fa6d80cb7d0 2023-10-30T14:42:36.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:36.056+0000 7fa6ce7fc640 1 -- 172.21.15.57:0/484647582 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+5740 (secure 0 0 0) 0x7fa6c8054c10 con 0x7fa6d80cb7d0 2023-10-30T14:42:36.062 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:36.062 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:42:35.131527+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:36.063 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 151 2023-10-30T14:42:36.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:36.060+0000 7fa6e01c5640 1 -- 172.21.15.57:0/484647582 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa6bc0664d0 msgr2=0x7fa6bc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:36.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:36.060+0000 7fa6e01c5640 1 --2- 172.21.15.57:0/484647582 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa6bc0664d0 0x7fa6bc068990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa6d40b2000 tx=0x7fa6d40aa070 comp rx=0 tx=0).stop 2023-10-30T14:42:36.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:36.060+0000 7fa6e01c5640 1 -- 172.21.15.57:0/484647582 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6d80cb7d0 msgr2=0x7fa6d81599f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:36.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:36.060+0000 7fa6e01c5640 1 --2- 172.21.15.57:0/484647582 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6d80cb7d0 0x7fa6d81599f0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800e930 tx=0x7fa6c800edf0 comp rx=0 tx=0).stop 2023-10-30T14:42:36.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:36.060+0000 7fa6e01c5640 1 -- 172.21.15.57:0/484647582 shutdown_connections 2023-10-30T14:42:36.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:36.060+0000 7fa6e01c5640 1 --2- 172.21.15.57:0/484647582 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa6bc0664d0 0x7fa6bc068990 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:36.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:36.060+0000 7fa6e01c5640 1 --2- 172.21.15.57:0/484647582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6d8150830 0x7fa6d8159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:36.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:36.060+0000 7fa6e01c5640 1 --2- 172.21.15.57:0/484647582 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6d80cb7d0 0x7fa6d81599f0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:36.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:36.060+0000 7fa6e01c5640 1 --2- 172.21.15.57:0/484647582 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6d80c3e40 0x7fa6d81f2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:36.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:36.060+0000 7fa6e01c5640 1 -- 172.21.15.57:0/484647582 >> 172.21.15.57:0/484647582 conn(0x7fa6d80cac70 msgr2=0x7fa6d806b500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:36.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:36.060+0000 7fa6e01c5640 1 -- 172.21.15.57:0/484647582 shutdown_connections 2023-10-30T14:42:36.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:36.060+0000 7fa6e01c5640 1 -- 172.21.15.57:0/484647582 wait complete. 2023-10-30T14:42:36.083 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 151, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 151, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6356": { "addr": "172.21.15.88:6839/1744808283", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1744808283, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1744808283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6356, "incarnation": 151, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 44 }, "gid_7056": { "addr": "172.21.15.57:6837/2307041430", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 2307041430, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 2307041430, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 7056, "incarnation": 137, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:42:35.131527+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6356, "mds_1": 6747, "mds_2": 7056 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:42:38.085 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:38.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.236+0000 7f1ffde9a640 1 Processor -- start 2023-10-30T14:42:38.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.236+0000 7f1ffde9a640 1 -- start start 2023-10-30T14:42:38.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.236+0000 7f1ffde9a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1ff8150830 0x7f1ff806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:38.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.236+0000 7f1ffde9a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ff8070520 0x7f1ff80709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:38.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.236+0000 7f1ffde9a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff80c2310 con 0x7f1ff80cb7d0 2023-10-30T14:42:38.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.236+0000 7f1ffde9a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff8156710 con 0x7f1ff8150830 2023-10-30T14:42:38.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.236+0000 7f1ffde9a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff80c4360 con 0x7f1ff8070520 2023-10-30T14:42:38.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.240+0000 7f1ff77fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1ff8150830 0x7f1ff806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:38.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.240+0000 7f1ff7fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ff8070520 0x7f1ff80709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:38.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.240+0000 7f1ff77fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1ff8150830 0x7f1ff806b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:49806/0 (socket says 172.21.15.57:49806) 2023-10-30T14:42:38.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.240+0000 7f1ff77fe640 1 -- 172.21.15.57:0/1833284161 learned_addr learned my addr 172.21.15.57:0/1833284161 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:38.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.240+0000 7f1ff7fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ff8070520 0x7f1ff80709b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:46000/0 (socket says 172.21.15.57:46000) 2023-10-30T14:42:38.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.240+0000 7f1ff6ffd640 1 --1- 172.21.15.57:0/1833284161 >> v1:172.21.15.88:6789/0 conn(0x7f1ff80cb7d0 0x7f1ff80c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:44474/0 (socket says 172.21.15.57:44474) 2023-10-30T14:42:38.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.240+0000 7f1ff77fe640 1 -- 172.21.15.57:0/1833284161 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ff8070520 msgr2=0x7f1ff80709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:38.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.240+0000 7f1ff77fe640 1 --2- 172.21.15.57:0/1833284161 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ff8070520 0x7f1ff80709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:38.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.240+0000 7f1ff77fe640 1 -- 172.21.15.57:0/1833284161 >> v1:172.21.15.88:6789/0 conn(0x7f1ff80cb7d0 legacy=0x7f1ff80c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:38.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.240+0000 7f1ff77fe640 1 -- 172.21.15.57:0/1833284161 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff81569b0 con 0x7f1ff8150830 2023-10-30T14:42:38.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.244+0000 7f1ff77fe640 1 --2- 172.21.15.57:0/1833284161 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff8150830 0x7f1ff806b500 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8009870 tx=0x7f1fe801ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ca9a70bf986c2b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:38.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.244+0000 7f1ff67fc640 1 -- 172.21.15.57:0/1833284161 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe80a8a30 con 0x7f1ff8150830 2023-10-30T14:42:38.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.244+0000 7f1ff67fc640 1 -- 172.21.15.57:0/1833284161 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe80a8b90 con 0x7f1ff8150830 2023-10-30T14:42:38.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.244+0000 7f1ff67fc640 1 -- 172.21.15.57:0/1833284161 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe80b0440 con 0x7f1ff8150830 2023-10-30T14:42:38.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.244+0000 7f1ffde9a640 1 -- 172.21.15.57:0/1833284161 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff8150830 msgr2=0x7f1ff806b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:38.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.244+0000 7f1ffde9a640 1 --2- 172.21.15.57:0/1833284161 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff8150830 0x7f1ff806b500 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8009870 tx=0x7f1fe801ed30 comp rx=0 tx=0).stop 2023-10-30T14:42:38.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.244+0000 7f1ffde9a640 1 -- 172.21.15.57:0/1833284161 shutdown_connections 2023-10-30T14:42:38.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.244+0000 7f1ffde9a640 1 --2- 172.21.15.57:0/1833284161 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ff8070520 0x7f1ff80709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:38.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.244+0000 7f1ffde9a640 1 --2- 172.21.15.57:0/1833284161 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff8150830 0x7f1ff806b500 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:38.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.244+0000 7f1ffde9a640 1 -- 172.21.15.57:0/1833284161 >> 172.21.15.57:0/1833284161 conn(0x7f1ff80cac70 msgr2=0x7f1ff80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:38.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.244+0000 7f1ffde9a640 1 -- 172.21.15.57:0/1833284161 shutdown_connections 2023-10-30T14:42:38.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.244+0000 7f1ffde9a640 1 -- 172.21.15.57:0/1833284161 wait complete. 2023-10-30T14:42:38.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.244+0000 7f1ffde9a640 1 Processor -- start 2023-10-30T14:42:38.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.244+0000 7f1ffde9a640 1 -- start start 2023-10-30T14:42:38.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ffde9a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ff8070520 0x7f1ff81ebcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:38.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ffde9a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff80cb7d0 0x7f1ff81558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:38.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ffde9a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ff8150830 0x7f1ff81e2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:38.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ffde9a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff81dfa30 con 0x7f1ff8070520 2023-10-30T14:42:38.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ffde9a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff81ec1f0 con 0x7f1ff80cb7d0 2023-10-30T14:42:38.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ffde9a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff81dffb0 con 0x7f1ff8150830 2023-10-30T14:42:38.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ff77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ff8070520 0x7f1ff81ebcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:38.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ff77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ff8070520 0x7f1ff81ebcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47018/0 (socket says 172.21.15.57:47018) 2023-10-30T14:42:38.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ff77fe640 1 -- 172.21.15.57:0/1843011092 learned_addr learned my addr 172.21.15.57:0/1843011092 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:38.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ff7fff640 1 --2- 172.21.15.57:0/1843011092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ff8150830 0x7f1ff81e2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:38.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ff6ffd640 1 --2- 172.21.15.57:0/1843011092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff80cb7d0 0x7f1ff81558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:38.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ff77fe640 1 -- 172.21.15.57:0/1843011092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ff8150830 msgr2=0x7f1ff81e2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:38.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ff77fe640 1 --2- 172.21.15.57:0/1843011092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ff8150830 0x7f1ff81e2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:38.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ff77fe640 1 -- 172.21.15.57:0/1843011092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff80cb7d0 msgr2=0x7f1ff81558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:38.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ff77fe640 1 --2- 172.21.15.57:0/1843011092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff80cb7d0 0x7f1ff81558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:38.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ff77fe640 1 -- 172.21.15.57:0/1843011092 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe80094d0 con 0x7f1ff8070520 2023-10-30T14:42:38.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ff7fff640 1 --2- 172.21.15.57:0/1843011092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ff8150830 0x7f1ff81e2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:42:38.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ff77fe640 1 --2- 172.21.15.57:0/1843011092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ff8070520 0x7f1ff81ebcb0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f1fe801d3e0 tx=0x7f1fe801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:38.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1fdffff640 1 -- 172.21.15.57:0/1843011092 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe80b0dd0 con 0x7f1ff8070520 2023-10-30T14:42:38.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1fdffff640 1 -- 172.21.15.57:0/1843011092 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe80bb460 con 0x7f1ff8070520 2023-10-30T14:42:38.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1fdffff640 1 -- 172.21.15.57:0/1843011092 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe80bb810 con 0x7f1ff8070520 2023-10-30T14:42:38.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ffde9a640 1 -- 172.21.15.57:0/1843011092 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff8154220 con 0x7f1ff8070520 2023-10-30T14:42:38.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ffde9a640 1 -- 172.21.15.57:0/1843011092 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff81547c0 con 0x7f1ff8070520 2023-10-30T14:42:38.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1ffde9a640 1 -- 172.21.15.57:0/1843011092 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff8066510 con 0x7f1ff8070520 2023-10-30T14:42:38.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.248+0000 7f1fdffff640 1 -- 172.21.15.57:0/1843011092 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1fe80b83f0 con 0x7f1ff8070520 2023-10-30T14:42:38.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.252+0000 7f1fdffff640 1 --2- 172.21.15.57:0/1843011092 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1fe40664a0 0x7f1fe4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:38.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.252+0000 7f1fdffff640 1 -- 172.21.15.57:0/1843011092 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f1fe8121d30 con 0x7f1ff8070520 2023-10-30T14:42:38.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.252+0000 7f1ff6ffd640 1 --2- 172.21.15.57:0/1843011092 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1fe40664a0 0x7f1fe4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:38.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.252+0000 7f1ff6ffd640 1 --2- 172.21.15.57:0/1843011092 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1fe40664a0 0x7f1fe4068960 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1ff81541c0 tx=0x7f1fec007c90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:38.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.256+0000 7f1fdffff640 1 -- 172.21.15.57:0/1843011092 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1fe80eeb90 con 0x7f1ff8070520 2023-10-30T14:42:38.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.452+0000 7f1ffde9a640 1 -- 172.21.15.57:0/1843011092 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ff81549b0 con 0x7f1ff8070520 2023-10-30T14:42:38.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.452+0000 7f1fdffff640 1 -- 172.21.15.57:0/1843011092 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+5740 (secure 0 0 0) 0x7f1fe80f21b0 con 0x7f1ff8070520 2023-10-30T14:42:38.466 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:38.466 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:42:35.131527+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:38.467 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 151 2023-10-30T14:42:38.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.452+0000 7f1ffde9a640 1 -- 172.21.15.57:0/1843011092 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1fe40664a0 msgr2=0x7f1fe4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:38.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.452+0000 7f1ffde9a640 1 --2- 172.21.15.57:0/1843011092 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1fe40664a0 0x7f1fe4068960 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1ff81541c0 tx=0x7f1fec007c90 comp rx=0 tx=0).stop 2023-10-30T14:42:38.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.452+0000 7f1ffde9a640 1 -- 172.21.15.57:0/1843011092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ff8070520 msgr2=0x7f1ff81ebcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:38.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.452+0000 7f1ffde9a640 1 --2- 172.21.15.57:0/1843011092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ff8070520 0x7f1ff81ebcb0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f1fe801d3e0 tx=0x7f1fe801bad0 comp rx=0 tx=0).stop 2023-10-30T14:42:38.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.452+0000 7f1ffde9a640 1 -- 172.21.15.57:0/1843011092 shutdown_connections 2023-10-30T14:42:38.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.452+0000 7f1ffde9a640 1 --2- 172.21.15.57:0/1843011092 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1fe40664a0 0x7f1fe4068960 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:38.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.452+0000 7f1ffde9a640 1 --2- 172.21.15.57:0/1843011092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ff8150830 0x7f1ff81e2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:38.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.452+0000 7f1ffde9a640 1 --2- 172.21.15.57:0/1843011092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff80cb7d0 0x7f1ff81558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:38.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.452+0000 7f1ffde9a640 1 --2- 172.21.15.57:0/1843011092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ff8070520 0x7f1ff81ebcb0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:38.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.452+0000 7f1ffde9a640 1 -- 172.21.15.57:0/1843011092 >> 172.21.15.57:0/1843011092 conn(0x7f1ff80cac70 msgr2=0x7f1ff80c36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:38.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.452+0000 7f1ffde9a640 1 -- 172.21.15.57:0/1843011092 shutdown_connections 2023-10-30T14:42:38.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:38.452+0000 7f1ffde9a640 1 -- 172.21.15.57:0/1843011092 wait complete. 2023-10-30T14:42:38.477 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:42:38.568 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:42:38.579 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:42:40.478 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:40.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.624+0000 7f9eb33a7640 1 Processor -- start 2023-10-30T14:42:40.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.624+0000 7f9eb33a7640 1 -- start start 2023-10-30T14:42:40.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.628+0000 7f9eb33a7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9eac0c8eb0 0x7f9eac15cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:40.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.628+0000 7f9eb33a7640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9eac15ac70 0x7f9eac15a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:40.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.628+0000 7f9eb33a7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9eac06aeb0 0x7f9eac069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:40.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.628+0000 7f9eb33a7640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eac159800 con 0x7f9eac15ac70 2023-10-30T14:42:40.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.628+0000 7f9eb33a7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eac15dcc0 con 0x7f9eac0c8eb0 2023-10-30T14:42:40.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.628+0000 7f9eb33a7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac159d80 con 0x7f9eac06aeb0 2023-10-30T14:42:40.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.628+0000 7f9eb1ba4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9eac15ac70 0x7f9eac15a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:40.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.628+0000 7f9eb1ba4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9eac15ac70 0x7f9eac15a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47034/0 (socket says 172.21.15.57:47034) 2023-10-30T14:42:40.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.628+0000 7f9eb1ba4640 1 -- 172.21.15.57:0/1699464810 learned_addr learned my addr 172.21.15.57:0/1699464810 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:40.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.628+0000 7f9eb23a5640 1 --2- 172.21.15.57:0/1699464810 >> v2:172.21.15.88:3300/0 conn(0x7f9eac0c8eb0 0x7f9eac15cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:40.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.628+0000 7f9eb2ba6640 1 --2- 172.21.15.57:0/1699464810 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9eac06aeb0 0x7f9eac069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:40.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.628+0000 7f9eb1ba4640 1 -- 172.21.15.57:0/1699464810 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9eac06aeb0 msgr2=0x7f9eac069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:40.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.628+0000 7f9eb1ba4640 1 --2- 172.21.15.57:0/1699464810 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9eac06aeb0 0x7f9eac069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:40.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.628+0000 7f9eb1ba4640 1 -- 172.21.15.57:0/1699464810 >> v2:172.21.15.88:3300/0 conn(0x7f9eac0c8eb0 msgr2=0x7f9eac15cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:40.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.628+0000 7f9eb1ba4640 1 --2- 172.21.15.57:0/1699464810 >> v2:172.21.15.88:3300/0 conn(0x7f9eac0c8eb0 0x7f9eac15cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:40.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.628+0000 7f9eb1ba4640 1 -- 172.21.15.57:0/1699464810 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eac15a750 con 0x7f9eac15ac70 2023-10-30T14:42:40.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb1ba4640 1 --2- 172.21.15.57:0/1699464810 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9eac15ac70 0x7f9eac15a210 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f9eac06aa40 tx=0x7f9e9c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=400f5926df0fc0ef server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:40.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb13a3640 1 -- 172.21.15.57:0/1699464810 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c0a89b0 con 0x7f9eac15ac70 2023-10-30T14:42:40.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb13a3640 1 -- 172.21.15.57:0/1699464810 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9c0a8b10 con 0x7f9eac15ac70 2023-10-30T14:42:40.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb13a3640 1 -- 172.21.15.57:0/1699464810 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c0a8ea0 con 0x7f9eac15ac70 2023-10-30T14:42:40.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb33a7640 1 -- 172.21.15.57:0/1699464810 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9eac15ac70 msgr2=0x7f9eac15a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:40.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb33a7640 1 --2- 172.21.15.57:0/1699464810 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9eac15ac70 0x7f9eac15a210 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f9eac06aa40 tx=0x7f9e9c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:42:40.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb33a7640 1 -- 172.21.15.57:0/1699464810 shutdown_connections 2023-10-30T14:42:40.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb33a7640 1 --2- 172.21.15.57:0/1699464810 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9eac06aeb0 0x7f9eac069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:40.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb33a7640 1 --2- 172.21.15.57:0/1699464810 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9eac15ac70 0x7f9eac15a210 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:40.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb33a7640 1 --2- 172.21.15.57:0/1699464810 >> v2:172.21.15.88:3300/0 conn(0x7f9eac0c8eb0 0x7f9eac15cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:40.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb33a7640 1 -- 172.21.15.57:0/1699464810 >> 172.21.15.57:0/1699464810 conn(0x7f9eac0cef00 msgr2=0x7f9eac0cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:40.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb33a7640 1 -- 172.21.15.57:0/1699464810 shutdown_connections 2023-10-30T14:42:40.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb33a7640 1 -- 172.21.15.57:0/1699464810 wait complete. 2023-10-30T14:42:40.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb33a7640 1 Processor -- start 2023-10-30T14:42:40.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb33a7640 1 -- start start 2023-10-30T14:42:40.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb33a7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9eac06aeb0 0x7f9eac0cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:40.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb33a7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9eac0c8eb0 0x7f9eac1586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:40.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb33a7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9eac15ac70 0x7f9eac1eee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:40.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb33a7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac1eea00 con 0x7f9eac06aeb0 2023-10-30T14:42:40.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb33a7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac1f2c70 con 0x7f9eac15ac70 2023-10-30T14:42:40.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.632+0000 7f9eb33a7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac159430 con 0x7f9eac0c8eb0 2023-10-30T14:42:40.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.636+0000 7f9eb23a5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9eac06aeb0 0x7f9eac0cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:40.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.636+0000 7f9eb23a5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9eac06aeb0 0x7f9eac0cc230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47048/0 (socket says 172.21.15.57:47048) 2023-10-30T14:42:40.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.636+0000 7f9eb23a5640 1 -- 172.21.15.57:0/1735081499 learned_addr learned my addr 172.21.15.57:0/1735081499 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:40.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.636+0000 7f9eb1ba4640 1 --2- 172.21.15.57:0/1735081499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9eac0c8eb0 0x7f9eac1586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:40.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.636+0000 7f9eb2ba6640 1 --2- 172.21.15.57:0/1735081499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9eac15ac70 0x7f9eac1eee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:40.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.636+0000 7f9eb23a5640 1 -- 172.21.15.57:0/1735081499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9eac0c8eb0 msgr2=0x7f9eac1586f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:40.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.636+0000 7f9eb23a5640 1 --2- 172.21.15.57:0/1735081499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9eac0c8eb0 0x7f9eac1586f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:40.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.636+0000 7f9eb23a5640 1 -- 172.21.15.57:0/1735081499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9eac15ac70 msgr2=0x7f9eac1eee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:40.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.636+0000 7f9eb23a5640 1 --2- 172.21.15.57:0/1735081499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9eac15ac70 0x7f9eac1eee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:40.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.636+0000 7f9eb23a5640 1 -- 172.21.15.57:0/1735081499 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9c0094d0 con 0x7f9eac06aeb0 2023-10-30T14:42:40.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.636+0000 7f9eb23a5640 1 --2- 172.21.15.57:0/1735081499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9eac06aeb0 0x7f9eac0cc230 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f9ea800d800 tx=0x7f9ea800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:40.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.636+0000 7f9ea2ffd640 1 -- 172.21.15.57:0/1735081499 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea800b6c0 con 0x7f9eac06aeb0 2023-10-30T14:42:40.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.636+0000 7f9ea2ffd640 1 -- 172.21.15.57:0/1735081499 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea800b820 con 0x7f9eac06aeb0 2023-10-30T14:42:40.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.636+0000 7f9eb33a7640 1 -- 172.21.15.57:0/1735081499 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eac0cc770 con 0x7f9eac06aeb0 2023-10-30T14:42:40.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.636+0000 7f9ea2ffd640 1 -- 172.21.15.57:0/1735081499 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea800bb90 con 0x7f9eac06aeb0 2023-10-30T14:42:40.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.636+0000 7f9eb33a7640 1 -- 172.21.15.57:0/1735081499 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eac1f23d0 con 0x7f9eac06aeb0 2023-10-30T14:42:40.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.636+0000 7f9eb33a7640 1 -- 172.21.15.57:0/1735081499 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eac04e170 con 0x7f9eac06aeb0 2023-10-30T14:42:40.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.636+0000 7f9ea2ffd640 1 -- 172.21.15.57:0/1735081499 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9ea800bcf0 con 0x7f9eac06aeb0 2023-10-30T14:42:40.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.640+0000 7f9ea2ffd640 1 --2- 172.21.15.57:0/1735081499 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9e900664a0 0x7f9e90068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:40.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.640+0000 7f9ea2ffd640 1 -- 172.21.15.57:0/1735081499 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f9ea8084a10 con 0x7f9eac06aeb0 2023-10-30T14:42:40.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.640+0000 7f9eb1ba4640 1 --2- 172.21.15.57:0/1735081499 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9e900664a0 0x7f9e90068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:40.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.640+0000 7f9eb1ba4640 1 --2- 172.21.15.57:0/1735081499 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9e900664a0 0x7f9e90068960 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0b2000 tx=0x7f9e9c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:40.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.644+0000 7f9ea2ffd640 1 -- 172.21.15.57:0/1735081499 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9ea8051890 con 0x7f9eac06aeb0 2023-10-30T14:42:40.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.836+0000 7f9eb33a7640 1 -- 172.21.15.57:0/1735081499 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9eac1ee7d0 con 0x7f9eac06aeb0 2023-10-30T14:42:40.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.836+0000 7f9ea2ffd640 1 -- 172.21.15.57:0/1735081499 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+5740 (secure 0 0 0) 0x7f9ea8054eb0 con 0x7f9eac06aeb0 2023-10-30T14:42:40.841 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:40.841 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:42:35.131527+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:40.842 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 151 2023-10-30T14:42:40.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.836+0000 7f9eb33a7640 1 -- 172.21.15.57:0/1735081499 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9e900664a0 msgr2=0x7f9e90068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:40.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.836+0000 7f9eb33a7640 1 --2- 172.21.15.57:0/1735081499 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9e900664a0 0x7f9e90068960 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0b2000 tx=0x7f9e9c009040 comp rx=0 tx=0).stop 2023-10-30T14:42:40.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.836+0000 7f9eb33a7640 1 -- 172.21.15.57:0/1735081499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9eac06aeb0 msgr2=0x7f9eac0cc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:40.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.836+0000 7f9eb33a7640 1 --2- 172.21.15.57:0/1735081499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9eac06aeb0 0x7f9eac0cc230 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f9ea800d800 tx=0x7f9ea800dcc0 comp rx=0 tx=0).stop 2023-10-30T14:42:40.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.836+0000 7f9eb33a7640 1 -- 172.21.15.57:0/1735081499 shutdown_connections 2023-10-30T14:42:40.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.836+0000 7f9eb33a7640 1 --2- 172.21.15.57:0/1735081499 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9e900664a0 0x7f9e90068960 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:40.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.836+0000 7f9eb33a7640 1 --2- 172.21.15.57:0/1735081499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9eac15ac70 0x7f9eac1eee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:40.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.836+0000 7f9eb33a7640 1 --2- 172.21.15.57:0/1735081499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9eac0c8eb0 0x7f9eac1586f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:40.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.836+0000 7f9eb33a7640 1 --2- 172.21.15.57:0/1735081499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9eac06aeb0 0x7f9eac0cc230 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:40.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.836+0000 7f9eb33a7640 1 -- 172.21.15.57:0/1735081499 >> 172.21.15.57:0/1735081499 conn(0x7f9eac0cef00 msgr2=0x7f9eac0c7690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:40.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.836+0000 7f9eb33a7640 1 -- 172.21.15.57:0/1735081499 shutdown_connections 2023-10-30T14:42:40.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:40.836+0000 7f9eb33a7640 1 -- 172.21.15.57:0/1735081499 wait complete. 2023-10-30T14:42:40.861 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:42:42.862 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:43.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.016+0000 7f933189e640 1 Processor -- start 2023-10-30T14:42:43.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.016+0000 7f933189e640 1 -- start start 2023-10-30T14:42:43.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.016+0000 7f933189e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f932c150830 0x7f932c06b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:43.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.016+0000 7f933189e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f932c0cb7d0 0x7f932c0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:43.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.016+0000 7f933189e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932c0c2310 con 0x7f932c0c3e40 2023-10-30T14:42:43.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.016+0000 7f933189e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f932c156710 con 0x7f932c150830 2023-10-30T14:42:43.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.016+0000 7f933189e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f932c0c4360 con 0x7f932c0cb7d0 2023-10-30T14:42:43.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.016+0000 7f932b7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f932c0c3e40 0x7f932c070520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:59244/0 (socket says 172.21.15.57:59244) 2023-10-30T14:42:43.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.016+0000 7f932b7fe640 1 -- 172.21.15.57:0/3814380631 learned_addr learned my addr 172.21.15.57:0/3814380631 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:42:43.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.016+0000 7f932a7fc640 1 --2- 172.21.15.57:0/3814380631 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f932c0cb7d0 0x7f932c0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:43.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.016+0000 7f932affd640 1 --2- 172.21.15.57:0/3814380631 >> v2:172.21.15.88:3300/0 conn(0x7f932c150830 0x7f932c06b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:43.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.016+0000 7f9329ffb640 1 -- 172.21.15.57:0/3814380631 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808863228 0 0) 0x7f932c0c2310 con 0x7f932c0c3e40 2023-10-30T14:42:43.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.016+0000 7f9329ffb640 1 -- 172.21.15.57:0/3814380631 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9314003530 con 0x7f932c0c3e40 2023-10-30T14:42:43.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.016+0000 7f932a7fc640 1 -- 172.21.15.57:0/3814380631 >> v2:172.21.15.88:3300/0 conn(0x7f932c150830 msgr2=0x7f932c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:43.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.016+0000 7f932a7fc640 1 --2- 172.21.15.57:0/3814380631 >> v2:172.21.15.88:3300/0 conn(0x7f932c150830 0x7f932c06b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:43.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.016+0000 7f9329ffb640 1 -- 172.21.15.57:0/3814380631 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2205869845 0 0) 0x7f9314003530 con 0x7f932c0c3e40 2023-10-30T14:42:43.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.016+0000 7f932a7fc640 1 -- 172.21.15.57:0/3814380631 >> v1:172.21.15.57:6789/0 conn(0x7f932c0c3e40 legacy=0x7f932c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:43.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.016+0000 7f932a7fc640 1 -- 172.21.15.57:0/3814380631 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932c1569b0 con 0x7f932c0cb7d0 2023-10-30T14:42:43.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.016+0000 7f932affd640 1 --2- 172.21.15.57:0/3814380631 >> v2:172.21.15.88:3300/0 conn(0x7f932c150830 0x7f932c06b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:42:43.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.020+0000 7f932a7fc640 1 --2- 172.21.15.57:0/3814380631 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f932c0cb7d0 0x7f932c0c3900 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f93200097c0 tx=0x7f932001eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f2d5435f533c44e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:43.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.020+0000 7f9329ffb640 1 -- 172.21.15.57:0/3814380631 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9324001a00 con 0x7f932c0cb7d0 2023-10-30T14:42:43.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.020+0000 7f9329ffb640 1 -- 172.21.15.57:0/3814380631 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93200a85d0 con 0x7f932c0cb7d0 2023-10-30T14:42:43.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.020+0000 7f9329ffb640 1 -- 172.21.15.57:0/3814380631 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93200a8940 con 0x7f932c0cb7d0 2023-10-30T14:42:43.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.020+0000 7f933189e640 1 -- 172.21.15.57:0/3814380631 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f932c0cb7d0 msgr2=0x7f932c0c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:43.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.020+0000 7f933189e640 1 --2- 172.21.15.57:0/3814380631 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f932c0cb7d0 0x7f932c0c3900 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f93200097c0 tx=0x7f932001eb90 comp rx=0 tx=0).stop 2023-10-30T14:42:43.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.020+0000 7f933189e640 1 -- 172.21.15.57:0/3814380631 shutdown_connections 2023-10-30T14:42:43.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.020+0000 7f933189e640 1 --2- 172.21.15.57:0/3814380631 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f932c0cb7d0 0x7f932c0c3900 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:43.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.020+0000 7f933189e640 1 --2- 172.21.15.57:0/3814380631 >> v2:172.21.15.88:3300/0 conn(0x7f932c150830 0x7f932c06b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:43.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.020+0000 7f933189e640 1 -- 172.21.15.57:0/3814380631 >> 172.21.15.57:0/3814380631 conn(0x7f932c0cac70 msgr2=0x7f932c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:43.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.020+0000 7f933189e640 1 -- 172.21.15.57:0/3814380631 shutdown_connections 2023-10-30T14:42:43.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.020+0000 7f933189e640 1 -- 172.21.15.57:0/3814380631 wait complete. 2023-10-30T14:42:43.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f933189e640 1 Processor -- start 2023-10-30T14:42:43.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f933189e640 1 -- start start 2023-10-30T14:42:43.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f933189e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f932c0c3e40 0x7f932c1577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:43.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f933189e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f932c0cb7d0 0x7f932c1554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:43.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f933189e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f932c150830 0x7f932c155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:43.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f933189e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c154070 con 0x7f932c0c3e40 2023-10-30T14:42:43.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f933189e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c159b10 con 0x7f932c0cb7d0 2023-10-30T14:42:43.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f933189e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f932c159870 con 0x7f932c150830 2023-10-30T14:42:43.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f932affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f932c0c3e40 0x7f932c1577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:43.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f932affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f932c0c3e40 0x7f932c1577c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47064/0 (socket says 172.21.15.57:47064) 2023-10-30T14:42:43.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f932affd640 1 -- 172.21.15.57:0/702590764 learned_addr learned my addr 172.21.15.57:0/702590764 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:43.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f932b7fe640 1 --2- 172.21.15.57:0/702590764 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f932c150830 0x7f932c155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:43.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f932a7fc640 1 --2- 172.21.15.57:0/702590764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f932c0cb7d0 0x7f932c1554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:43.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f932affd640 1 -- 172.21.15.57:0/702590764 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f932c150830 msgr2=0x7f932c155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:43.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f932affd640 1 --2- 172.21.15.57:0/702590764 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f932c150830 0x7f932c155a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:43.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f932affd640 1 -- 172.21.15.57:0/702590764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f932c0cb7d0 msgr2=0x7f932c1554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:43.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f932affd640 1 --2- 172.21.15.57:0/702590764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f932c0cb7d0 0x7f932c1554c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:43.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f932affd640 1 -- 172.21.15.57:0/702590764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931c009420 con 0x7f932c0c3e40 2023-10-30T14:42:43.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f932affd640 1 --2- 172.21.15.57:0/702590764 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f932c0c3e40 0x7f932c1577c0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f931c00e980 tx=0x7f931c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:43.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f9313fff640 1 -- 172.21.15.57:0/702590764 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931c00ca00 con 0x7f932c0c3e40 2023-10-30T14:42:43.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f9313fff640 1 -- 172.21.15.57:0/702590764 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931c00cb60 con 0x7f932c0c3e40 2023-10-30T14:42:43.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f9313fff640 1 -- 172.21.15.57:0/702590764 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931c013470 con 0x7f932c0c3e40 2023-10-30T14:42:43.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.024+0000 7f933189e640 1 -- 172.21.15.57:0/702590764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9320009420 con 0x7f932c0c3e40 2023-10-30T14:42:43.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.028+0000 7f933189e640 1 -- 172.21.15.57:0/702590764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f932c153ab0 con 0x7f932c0c3e40 2023-10-30T14:42:43.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.028+0000 7f933189e640 1 -- 172.21.15.57:0/702590764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f932c159d30 con 0x7f932c0c3e40 2023-10-30T14:42:43.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.028+0000 7f9313fff640 1 -- 172.21.15.57:0/702590764 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f931c0135d0 con 0x7f932c0c3e40 2023-10-30T14:42:43.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.028+0000 7f9313fff640 1 --2- 172.21.15.57:0/702590764 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f93180664d0 0x7f9318068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:43.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.028+0000 7f9313fff640 1 -- 172.21.15.57:0/702590764 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f931c084590 con 0x7f932c0c3e40 2023-10-30T14:42:43.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.028+0000 7f932a7fc640 1 --2- 172.21.15.57:0/702590764 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f93180664d0 0x7f9318068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:43.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.032+0000 7f932a7fc640 1 --2- 172.21.15.57:0/702590764 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f93180664d0 0x7f9318068990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f93200b2000 tx=0x7f93200aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:43.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.032+0000 7f9313fff640 1 -- 172.21.15.57:0/702590764 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f932c159d30 con 0x7f932c0c3e40 2023-10-30T14:42:43.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.228+0000 7f933189e640 1 -- 172.21.15.57:0/702590764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f932c159d30 con 0x7f932c0c3e40 2023-10-30T14:42:43.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.228+0000 7f9313fff640 1 -- 172.21.15.57:0/702590764 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+5740 (secure 0 0 0) 0x7f932c159d30 con 0x7f932c0c3e40 2023-10-30T14:42:43.231 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:43.232 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:42:35.131527+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:43.232 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 151 2023-10-30T14:42:43.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.228+0000 7f933189e640 1 -- 172.21.15.57:0/702590764 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f93180664d0 msgr2=0x7f9318068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:43.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.228+0000 7f933189e640 1 --2- 172.21.15.57:0/702590764 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f93180664d0 0x7f9318068990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f93200b2000 tx=0x7f93200aa070 comp rx=0 tx=0).stop 2023-10-30T14:42:43.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.228+0000 7f933189e640 1 -- 172.21.15.57:0/702590764 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f932c0c3e40 msgr2=0x7f932c1577c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:43.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.228+0000 7f933189e640 1 --2- 172.21.15.57:0/702590764 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f932c0c3e40 0x7f932c1577c0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f931c00e980 tx=0x7f931c00ee40 comp rx=0 tx=0).stop 2023-10-30T14:42:43.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.228+0000 7f933189e640 1 -- 172.21.15.57:0/702590764 shutdown_connections 2023-10-30T14:42:43.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.228+0000 7f933189e640 1 --2- 172.21.15.57:0/702590764 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f93180664d0 0x7f9318068990 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:43.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.228+0000 7f933189e640 1 --2- 172.21.15.57:0/702590764 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f932c150830 0x7f932c155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:43.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.228+0000 7f933189e640 1 --2- 172.21.15.57:0/702590764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f932c0cb7d0 0x7f932c1554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:43.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.228+0000 7f933189e640 1 --2- 172.21.15.57:0/702590764 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f932c0c3e40 0x7f932c1577c0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:43.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.228+0000 7f933189e640 1 -- 172.21.15.57:0/702590764 >> 172.21.15.57:0/702590764 conn(0x7f932c0cac70 msgr2=0x7f932c14dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:43.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.228+0000 7f933189e640 1 -- 172.21.15.57:0/702590764 shutdown_connections 2023-10-30T14:42:43.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:43.228+0000 7f933189e640 1 -- 172.21.15.57:0/702590764 wait complete. 2023-10-30T14:42:43.253 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:42:45.254 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:45.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.404+0000 7fb0a2701640 1 Processor -- start 2023-10-30T14:42:45.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.404+0000 7fb0a2701640 1 -- start start 2023-10-30T14:42:45.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.408+0000 7fb0a2701640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb09c1516c0 0x7fb09c0c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:45.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.408+0000 7fb0a2701640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb09c0c82d0 0x7fb09c0ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:45.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.408+0000 7fb0a2701640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09c151aa0 con 0x7fb09c0cd4a0 2023-10-30T14:42:45.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.408+0000 7fb0a2701640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb09c0c74a0 con 0x7fb09c0c82d0 2023-10-30T14:42:45.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.408+0000 7fb0a2701640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb09c158510 con 0x7fb09c1516c0 2023-10-30T14:42:45.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.408+0000 7fb09b7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb09c0c82d0 0x7fb09c0ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:45.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.408+0000 7fb09b7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb09c0c82d0 0x7fb09c0ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39794/0 (socket says 172.21.15.57:39794) 2023-10-30T14:42:45.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.408+0000 7fb09b7fe640 1 -- 172.21.15.57:0/3103696946 learned_addr learned my addr 172.21.15.57:0/3103696946 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:45.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.408+0000 7fb0a0c9e640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb09c0cd4a0 0x7fb09c0c2500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48866/0 (socket says 172.21.15.57:48866) 2023-10-30T14:42:45.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.408+0000 7fb09bfff640 1 --2- 172.21.15.57:0/3103696946 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb09c1516c0 0x7fb09c0c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:45.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.408+0000 7fb09b7fe640 1 -- 172.21.15.57:0/3103696946 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb09c1516c0 msgr2=0x7fb09c0c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:45.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.408+0000 7fb09b7fe640 1 --2- 172.21.15.57:0/3103696946 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb09c1516c0 0x7fb09c0c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:45.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.408+0000 7fb09b7fe640 1 -- 172.21.15.57:0/3103696946 >> v1:172.21.15.57:6789/0 conn(0x7fb09c0cd4a0 legacy=0x7fb09c0c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:45.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.408+0000 7fb09b7fe640 1 -- 172.21.15.57:0/3103696946 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09c158af0 con 0x7fb09c0c82d0 2023-10-30T14:42:45.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.408+0000 7fb09affd640 1 -- 172.21.15.57:0/3103696946 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802194944 0 0) 0x7fb09c151aa0 con 0x7fb09c0cd4a0 2023-10-30T14:42:45.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.412+0000 7fb09b7fe640 1 --2- 172.21.15.57:0/3103696946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb09c0c82d0 0x7fb09c0ccf60 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fb090009870 tx=0x7fb09001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4419c16edd6ec1e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:45.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.412+0000 7fb09affd640 1 -- 172.21.15.57:0/3103696946 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0900ab600 con 0x7fb09c0c82d0 2023-10-30T14:42:45.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.412+0000 7fb09affd640 1 -- 172.21.15.57:0/3103696946 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0900a7050 con 0x7fb09c0c82d0 2023-10-30T14:42:45.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.412+0000 7fb09affd640 1 -- 172.21.15.57:0/3103696946 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0900ab980 con 0x7fb09c0c82d0 2023-10-30T14:42:45.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.412+0000 7fb0a2701640 1 -- 172.21.15.57:0/3103696946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb09c0c82d0 msgr2=0x7fb09c0ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:45.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.412+0000 7fb0a2701640 1 --2- 172.21.15.57:0/3103696946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb09c0c82d0 0x7fb09c0ccf60 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fb090009870 tx=0x7fb09001c540 comp rx=0 tx=0).stop 2023-10-30T14:42:45.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.412+0000 7fb0a2701640 1 -- 172.21.15.57:0/3103696946 shutdown_connections 2023-10-30T14:42:45.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.412+0000 7fb0a2701640 1 --2- 172.21.15.57:0/3103696946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb09c0c82d0 0x7fb09c0ccf60 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:45.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.412+0000 7fb0a2701640 1 --2- 172.21.15.57:0/3103696946 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb09c1516c0 0x7fb09c0c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:45.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.412+0000 7fb0a2701640 1 -- 172.21.15.57:0/3103696946 >> 172.21.15.57:0/3103696946 conn(0x7fb09c071fb0 msgr2=0x7fb09c0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:45.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.412+0000 7fb0a2701640 1 -- 172.21.15.57:0/3103696946 shutdown_connections 2023-10-30T14:42:45.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.412+0000 7fb0a2701640 1 -- 172.21.15.57:0/3103696946 wait complete. 2023-10-30T14:42:45.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.412+0000 7fb0a2701640 1 Processor -- start 2023-10-30T14:42:45.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.412+0000 7fb0a2701640 1 -- start start 2023-10-30T14:42:45.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb0a2701640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb09c0c82d0 0x7fb09c1f3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:45.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb0a2701640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb09c0cd4a0 0x7fb09c13ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:45.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb0a2701640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb09c1516c0 0x7fb09c1eff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:45.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb0a2701640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09c1f3b40 con 0x7fb09c1516c0 2023-10-30T14:42:45.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb0a2701640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09c1efb00 con 0x7fb09c0c82d0 2023-10-30T14:42:45.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb0a2701640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb09c13b2a0 con 0x7fb09c0cd4a0 2023-10-30T14:42:45.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb0a0c9e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb09c1516c0 0x7fb09c1eff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:45.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb0a0c9e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb09c1516c0 0x7fb09c1eff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39798/0 (socket says 172.21.15.57:39798) 2023-10-30T14:42:45.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb0a0c9e640 1 -- 172.21.15.57:0/3363844378 learned_addr learned my addr 172.21.15.57:0/3363844378 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:45.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb09b7fe640 1 --2- 172.21.15.57:0/3363844378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb09c0cd4a0 0x7fb09c13ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:45.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb09bfff640 1 --2- 172.21.15.57:0/3363844378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb09c0c82d0 0x7fb09c1f3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:45.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb0a0c9e640 1 -- 172.21.15.57:0/3363844378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb09c0cd4a0 msgr2=0x7fb09c13ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:45.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb0a0c9e640 1 --2- 172.21.15.57:0/3363844378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb09c0cd4a0 0x7fb09c13ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:45.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb0a0c9e640 1 -- 172.21.15.57:0/3363844378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb09c0c82d0 msgr2=0x7fb09c1f3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:45.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb0a0c9e640 1 --2- 172.21.15.57:0/3363844378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb09c0c82d0 0x7fb09c1f3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:45.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb0a0c9e640 1 -- 172.21.15.57:0/3363844378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0900094d0 con 0x7fb09c1516c0 2023-10-30T14:42:45.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb0a0c9e640 1 --2- 172.21.15.57:0/3363844378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb09c1516c0 0x7fb09c1eff90 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fb094001570 tx=0x7fb09400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:45.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb098ff9640 1 -- 172.21.15.57:0/3363844378 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09400a640 con 0x7fb09c1516c0 2023-10-30T14:42:45.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb098ff9640 1 -- 172.21.15.57:0/3363844378 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09400a7c0 con 0x7fb09c1516c0 2023-10-30T14:42:45.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb098ff9640 1 -- 172.21.15.57:0/3363844378 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09400ab70 con 0x7fb09c1516c0 2023-10-30T14:42:45.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb0a2701640 1 -- 172.21.15.57:0/3363844378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb09c145880 con 0x7fb09c1516c0 2023-10-30T14:42:45.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb0a2701640 1 -- 172.21.15.57:0/3363844378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb09c145e40 con 0x7fb09c1516c0 2023-10-30T14:42:45.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb0a2701640 1 -- 172.21.15.57:0/3363844378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb09c04e170 con 0x7fb09c1516c0 2023-10-30T14:42:45.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.416+0000 7fb098ff9640 1 -- 172.21.15.57:0/3363844378 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb09400acd0 con 0x7fb09c1516c0 2023-10-30T14:42:45.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.420+0000 7fb098ff9640 1 --2- 172.21.15.57:0/3363844378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb0880664d0 0x7fb088068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:45.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.420+0000 7fb098ff9640 1 -- 172.21.15.57:0/3363844378 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7fb094083840 con 0x7fb09c1516c0 2023-10-30T14:42:45.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.420+0000 7fb09bfff640 1 --2- 172.21.15.57:0/3363844378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb0880664d0 0x7fb088068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:45.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.420+0000 7fb09bfff640 1 --2- 172.21.15.57:0/3363844378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb0880664d0 0x7fb088068990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb08c009b60 tx=0x7fb08c009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:45.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.424+0000 7fb098ff9640 1 -- 172.21.15.57:0/3363844378 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb0940507e0 con 0x7fb09c1516c0 2023-10-30T14:42:45.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.612+0000 7fb0a2701640 1 -- 172.21.15.57:0/3363844378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb09c1efc40 con 0x7fb09c1516c0 2023-10-30T14:42:45.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.616+0000 7fb098ff9640 1 -- 172.21.15.57:0/3363844378 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+5740 (secure 0 0 0) 0x7fb094053e00 con 0x7fb09c1516c0 2023-10-30T14:42:45.619 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:45.620 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:42:35.131527+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:45.621 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 151 2023-10-30T14:42:45.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.616+0000 7fb0a2701640 1 -- 172.21.15.57:0/3363844378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb0880664d0 msgr2=0x7fb088068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:45.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.616+0000 7fb0a2701640 1 --2- 172.21.15.57:0/3363844378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb0880664d0 0x7fb088068990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb08c009b60 tx=0x7fb08c009190 comp rx=0 tx=0).stop 2023-10-30T14:42:45.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.616+0000 7fb0a2701640 1 -- 172.21.15.57:0/3363844378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb09c1516c0 msgr2=0x7fb09c1eff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:45.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.616+0000 7fb0a2701640 1 --2- 172.21.15.57:0/3363844378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb09c1516c0 0x7fb09c1eff90 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fb094001570 tx=0x7fb09400cca0 comp rx=0 tx=0).stop 2023-10-30T14:42:45.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.616+0000 7fb0a2701640 1 -- 172.21.15.57:0/3363844378 shutdown_connections 2023-10-30T14:42:45.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.616+0000 7fb0a2701640 1 --2- 172.21.15.57:0/3363844378 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb0880664d0 0x7fb088068990 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:45.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.616+0000 7fb0a2701640 1 --2- 172.21.15.57:0/3363844378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb09c1516c0 0x7fb09c1eff90 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:45.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.616+0000 7fb0a2701640 1 --2- 172.21.15.57:0/3363844378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb09c0cd4a0 0x7fb09c13ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:45.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.616+0000 7fb0a2701640 1 --2- 172.21.15.57:0/3363844378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb09c0c82d0 0x7fb09c1f3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:45.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.616+0000 7fb0a2701640 1 -- 172.21.15.57:0/3363844378 >> 172.21.15.57:0/3363844378 conn(0x7fb09c071fb0 msgr2=0x7fb09c158c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:45.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.616+0000 7fb0a2701640 1 -- 172.21.15.57:0/3363844378 shutdown_connections 2023-10-30T14:42:45.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:45.616+0000 7fb0a2701640 1 -- 172.21.15.57:0/3363844378 wait complete. 2023-10-30T14:42:45.641 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:42:47.642 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:47.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.792+0000 7faa26b9d640 1 Processor -- start 2023-10-30T14:42:47.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.792+0000 7faa26b9d640 1 -- start start 2023-10-30T14:42:47.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa26b9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faa20153aa0 0x7faa2015d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:47.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa26b9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa2006b3b0 0x7faa2006b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:47.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa26b9d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa200c7b00 con 0x7faa200701e0 2023-10-30T14:42:47.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa26b9d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa20070a50 con 0x7faa20153aa0 2023-10-30T14:42:47.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa26b9d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa2015dd80 con 0x7faa2006b3b0 2023-10-30T14:42:47.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1ffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7faa200701e0 0x7faa200ccee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48874/0 (socket says 172.21.15.57:48874) 2023-10-30T14:42:47.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1ffff640 1 -- 172.21.15.57:0/4165398888 learned_addr learned my addr 172.21.15.57:0/4165398888 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:42:47.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa24939640 1 --2- 172.21.15.57:0/4165398888 >> v2:172.21.15.88:3300/0 conn(0x7faa20153aa0 0x7faa2015d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:47.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa2513a640 1 --2- 172.21.15.57:0/4165398888 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa2006b3b0 0x7faa2006b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:47.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1f7fe640 1 -- 172.21.15.57:0/4165398888 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839803503 0 0) 0x7faa200c7b00 con 0x7faa200701e0 2023-10-30T14:42:47.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1f7fe640 1 -- 172.21.15.57:0/4165398888 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa10003530 con 0x7faa200701e0 2023-10-30T14:42:47.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1f7fe640 1 -- 172.21.15.57:0/4165398888 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2917343421 0 0) 0x7faa10003530 con 0x7faa200701e0 2023-10-30T14:42:47.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1f7fe640 1 -- 172.21.15.57:0/4165398888 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa2006b3b0 msgr2=0x7faa2006b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:47.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1f7fe640 1 --2- 172.21.15.57:0/4165398888 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa2006b3b0 0x7faa2006b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:47.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1f7fe640 1 -- 172.21.15.57:0/4165398888 >> v2:172.21.15.88:3300/0 conn(0x7faa20153aa0 msgr2=0x7faa2015d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:47.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1f7fe640 1 --2- 172.21.15.57:0/4165398888 >> v2:172.21.15.88:3300/0 conn(0x7faa20153aa0 0x7faa2015d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:47.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1f7fe640 1 -- 172.21.15.57:0/4165398888 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2006bd60 con 0x7faa200701e0 2023-10-30T14:42:47.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1f7fe640 1 -- 172.21.15.57:0/4165398888 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7faa0c001680 con 0x7faa200701e0 2023-10-30T14:42:47.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1f7fe640 1 -- 172.21.15.57:0/4165398888 >> v1:172.21.15.57:6789/0 conn(0x7faa200701e0 legacy=0x7faa200ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:47.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1f7fe640 1 --2- 172.21.15.57:0/4165398888 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa2006b3b0 0x7faa10003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:47.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1f7fe640 1 --2- 172.21.15.57:0/4165398888 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa10003f00 0x7faa100047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:47.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1f7fe640 1 --2- 172.21.15.57:0/4165398888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa10004d10 0x7faa10005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:47.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1f7fe640 1 -- 172.21.15.57:0/4165398888 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa20070a50 con 0x7faa10003f00 2023-10-30T14:42:47.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1f7fe640 1 -- 172.21.15.57:0/4165398888 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2015dd80 con 0x7faa10004d10 2023-10-30T14:42:47.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1f7fe640 1 -- 172.21.15.57:0/4165398888 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1000b9c0 con 0x7faa2006b3b0 2023-10-30T14:42:47.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1ffff640 1 --2- 172.21.15.57:0/4165398888 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa10003f00 0x7faa100047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:47.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa24939640 1 --2- 172.21.15.57:0/4165398888 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa2006b3b0 0x7faa10003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:47.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa2513a640 1 --2- 172.21.15.57:0/4165398888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa10004d10 0x7faa10005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:47.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1ffff640 1 -- 172.21.15.57:0/4165398888 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa2006b3b0 msgr2=0x7faa10003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:47.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1ffff640 1 --2- 172.21.15.57:0/4165398888 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa2006b3b0 0x7faa10003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:47.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1ffff640 1 -- 172.21.15.57:0/4165398888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa10004d10 msgr2=0x7faa10005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:47.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1ffff640 1 --2- 172.21.15.57:0/4165398888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa10004d10 0x7faa10005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:47.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.796+0000 7faa1ffff640 1 -- 172.21.15.57:0/4165398888 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa10003530 con 0x7faa10003f00 2023-10-30T14:42:47.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.800+0000 7faa1ffff640 1 --2- 172.21.15.57:0/4165398888 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa10003f00 0x7faa100047d0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7faa0c00c210 tx=0x7faa0c01cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5589631a07a7c32 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:47.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.800+0000 7faa1f7fe640 1 -- 172.21.15.57:0/4165398888 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0c0a5940 con 0x7faa10003f00 2023-10-30T14:42:47.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.800+0000 7faa1f7fe640 1 -- 172.21.15.57:0/4165398888 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0c0a5aa0 con 0x7faa10003f00 2023-10-30T14:42:47.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.800+0000 7faa1f7fe640 1 -- 172.21.15.57:0/4165398888 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0c0a5e50 con 0x7faa10003f00 2023-10-30T14:42:47.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.800+0000 7faa26b9d640 1 -- 172.21.15.57:0/4165398888 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa10003f00 msgr2=0x7faa100047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:47.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.800+0000 7faa26b9d640 1 --2- 172.21.15.57:0/4165398888 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa10003f00 0x7faa100047d0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7faa0c00c210 tx=0x7faa0c01cc40 comp rx=0 tx=0).stop 2023-10-30T14:42:47.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.800+0000 7faa2513a640 1 -- 172.21.15.57:0/4165398888 reap_dead start 2023-10-30T14:42:47.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.800+0000 7faa26b9d640 1 -- 172.21.15.57:0/4165398888 shutdown_connections 2023-10-30T14:42:47.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.800+0000 7faa26b9d640 1 -- 172.21.15.57:0/4165398888 >> 172.21.15.57:0/4165398888 conn(0x7faa20071fb0 msgr2=0x7faa200723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:47.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.800+0000 7faa26b9d640 1 -- 172.21.15.57:0/4165398888 shutdown_connections 2023-10-30T14:42:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.800+0000 7faa26b9d640 1 -- 172.21.15.57:0/4165398888 wait complete. 2023-10-30T14:42:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa26b9d640 1 Processor -- start 2023-10-30T14:42:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa26b9d640 1 -- start start 2023-10-30T14:42:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa26b9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa10003f00 0x7faa2015a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa26b9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa20155410 0x7faa200cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa26b9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa201566e0 0x7faa20156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa26b9d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa201570d0 con 0x7faa201566e0 2023-10-30T14:42:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa26b9d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa20155a20 con 0x7faa10003f00 2023-10-30T14:42:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa26b9d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa2015a610 con 0x7faa20155410 2023-10-30T14:42:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa2513a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa201566e0 0x7faa20156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa2513a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa201566e0 0x7faa20156b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39820/0 (socket says 172.21.15.57:39820) 2023-10-30T14:42:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa2513a640 1 -- 172.21.15.57:0/1751598983 learned_addr learned my addr 172.21.15.57:0/1751598983 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa1ffff640 1 --2- 172.21.15.57:0/1751598983 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa20155410 0x7faa200cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:47.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa24939640 1 --2- 172.21.15.57:0/1751598983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa10003f00 0x7faa2015a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:47.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa2513a640 1 -- 172.21.15.57:0/1751598983 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa20155410 msgr2=0x7faa200cb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:47.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa2513a640 1 --2- 172.21.15.57:0/1751598983 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa20155410 0x7faa200cb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:47.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa2513a640 1 -- 172.21.15.57:0/1751598983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa10003f00 msgr2=0x7faa2015a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:47.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa2513a640 1 --2- 172.21.15.57:0/1751598983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa10003f00 0x7faa2015a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:47.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa2513a640 1 -- 172.21.15.57:0/1751598983 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa20154820 con 0x7faa201566e0 2023-10-30T14:42:47.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa1ffff640 1 --2- 172.21.15.57:0/1751598983 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa20155410 0x7faa200cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:42:47.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa2513a640 1 --2- 172.21.15.57:0/1751598983 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa201566e0 0x7faa20156b90 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7faa20071870 tx=0x7faa1800f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:47.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa1d7fa640 1 -- 172.21.15.57:0/1751598983 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1800e040 con 0x7faa201566e0 2023-10-30T14:42:47.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa1d7fa640 1 -- 172.21.15.57:0/1751598983 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1800f040 con 0x7faa201566e0 2023-10-30T14:42:47.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa26b9d640 1 -- 172.21.15.57:0/1751598983 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa20153f80 con 0x7faa201566e0 2023-10-30T14:42:47.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa1d7fa640 1 -- 172.21.15.57:0/1751598983 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18015430 con 0x7faa201566e0 2023-10-30T14:42:47.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa26b9d640 1 -- 172.21.15.57:0/1751598983 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa20154540 con 0x7faa201566e0 2023-10-30T14:42:47.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.804+0000 7faa26b9d640 1 -- 172.21.15.57:0/1751598983 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa2004e170 con 0x7faa201566e0 2023-10-30T14:42:47.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.808+0000 7faa1d7fa640 1 -- 172.21.15.57:0/1751598983 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faa18015590 con 0x7faa201566e0 2023-10-30T14:42:47.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.808+0000 7faa1d7fa640 1 --2- 172.21.15.57:0/1751598983 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa040664a0 0x7faa04068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:47.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.808+0000 7faa1d7fa640 1 -- 172.21.15.57:0/1751598983 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7faa18084d60 con 0x7faa201566e0 2023-10-30T14:42:47.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.808+0000 7faa24939640 1 --2- 172.21.15.57:0/1751598983 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa040664a0 0x7faa04068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:47.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.808+0000 7faa24939640 1 --2- 172.21.15.57:0/1751598983 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa040664a0 0x7faa04068960 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7faa14009be0 tx=0x7faa14009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:47.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:47.812+0000 7faa1d7fa640 1 -- 172.21.15.57:0/1751598983 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faa18051c50 con 0x7faa201566e0 2023-10-30T14:42:48.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:48.000+0000 7faa26b9d640 1 -- 172.21.15.57:0/1751598983 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa2014e790 con 0x7faa201566e0 2023-10-30T14:42:48.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:48.000+0000 7faa1d7fa640 1 -- 172.21.15.57:0/1751598983 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+5740 (secure 0 0 0) 0x7faa18055270 con 0x7faa201566e0 2023-10-30T14:42:48.005 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:48.005 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:42:35.131527+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:48.006 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 151 2023-10-30T14:42:48.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:48.004+0000 7faa26b9d640 1 -- 172.21.15.57:0/1751598983 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa040664a0 msgr2=0x7faa04068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:48.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:48.004+0000 7faa26b9d640 1 --2- 172.21.15.57:0/1751598983 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa040664a0 0x7faa04068960 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7faa14009be0 tx=0x7faa14009190 comp rx=0 tx=0).stop 2023-10-30T14:42:48.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:48.004+0000 7faa26b9d640 1 -- 172.21.15.57:0/1751598983 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa201566e0 msgr2=0x7faa20156b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:48.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:48.004+0000 7faa26b9d640 1 --2- 172.21.15.57:0/1751598983 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa201566e0 0x7faa20156b90 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7faa20071870 tx=0x7faa1800f1b0 comp rx=0 tx=0).stop 2023-10-30T14:42:48.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:48.004+0000 7faa26b9d640 1 -- 172.21.15.57:0/1751598983 shutdown_connections 2023-10-30T14:42:48.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:48.004+0000 7faa26b9d640 1 --2- 172.21.15.57:0/1751598983 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa040664a0 0x7faa04068960 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:48.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:48.004+0000 7faa26b9d640 1 --2- 172.21.15.57:0/1751598983 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa201566e0 0x7faa20156b90 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:48.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:48.004+0000 7faa26b9d640 1 --2- 172.21.15.57:0/1751598983 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa20155410 0x7faa200cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:48.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:48.004+0000 7faa26b9d640 1 --2- 172.21.15.57:0/1751598983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa10003f00 0x7faa2015a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:48.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:48.004+0000 7faa26b9d640 1 -- 172.21.15.57:0/1751598983 >> 172.21.15.57:0/1751598983 conn(0x7faa20071fb0 msgr2=0x7faa2014fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:48.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:48.004+0000 7faa26b9d640 1 -- 172.21.15.57:0/1751598983 shutdown_connections 2023-10-30T14:42:48.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:48.004+0000 7faa26b9d640 1 -- 172.21.15.57:0/1751598983 wait complete. 2023-10-30T14:42:48.027 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 151, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 151, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6356": { "addr": "172.21.15.88:6839/1744808283", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1744808283, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1744808283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6356, "incarnation": 151, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 44 }, "gid_7056": { "addr": "172.21.15.57:6837/2307041430", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 2307041430, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 2307041430, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 7056, "incarnation": 137, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:42:35.131527+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6356, "mds_1": 6747, "mds_2": 7056 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:42:50.028 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:50.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.176+0000 7f9f99ed6640 1 Processor -- start 2023-10-30T14:42:50.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.176+0000 7f9f99ed6640 1 -- start start 2023-10-30T14:42:50.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.176+0000 7f9f99ed6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9f940cb7d0 0x7f9f940c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:50.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.176+0000 7f9f99ed6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f940c3e40 0x7f9f94070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:50.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.176+0000 7f9f99ed6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f940c2310 con 0x7f9f94150830 2023-10-30T14:42:50.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.176+0000 7f9f99ed6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f9406bc10 con 0x7f9f940cb7d0 2023-10-30T14:42:50.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.176+0000 7f9f99ed6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f94156710 con 0x7f9f940c3e40 2023-10-30T14:42:50.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.176+0000 7f9f92ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9f940cb7d0 0x7f9f940c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:50.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.176+0000 7f9f93fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f940c3e40 0x7f9f94070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:50.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.176+0000 7f9f92ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9f940cb7d0 0x7f9f940c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:59848/0 (socket says 172.21.15.57:59848) 2023-10-30T14:42:50.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.176+0000 7f9f92ffd640 1 -- 172.21.15.57:0/4140333504 learned_addr learned my addr 172.21.15.57:0/4140333504 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:50.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.176+0000 7f9f93fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f940c3e40 0x7f9f94070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:45426/0 (socket says 172.21.15.57:45426) 2023-10-30T14:42:50.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.176+0000 7f9f92ffd640 1 -- 172.21.15.57:0/4140333504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f940c3e40 msgr2=0x7f9f94070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:50.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.176+0000 7f9f92ffd640 1 --2- 172.21.15.57:0/4140333504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f940c3e40 0x7f9f94070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:50.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.176+0000 7f9f92ffd640 1 -- 172.21.15.57:0/4140333504 >> v1:172.21.15.88:6789/0 conn(0x7f9f94150830 legacy=0x7f9f9406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:50.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.176+0000 7f9f92ffd640 1 -- 172.21.15.57:0/4140333504 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f940c4360 con 0x7f9f940cb7d0 2023-10-30T14:42:50.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.180+0000 7f9f92ffd640 1 --2- 172.21.15.57:0/4140333504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f940cb7d0 0x7f9f940c3900 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9f88009870 tx=0x7f9f8801ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be6e624588350ac3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:50.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.180+0000 7f9f927fc640 1 -- 172.21.15.57:0/4140333504 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f880a8a30 con 0x7f9f940cb7d0 2023-10-30T14:42:50.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.180+0000 7f9f927fc640 1 -- 172.21.15.57:0/4140333504 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f880a8b90 con 0x7f9f940cb7d0 2023-10-30T14:42:50.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.180+0000 7f9f927fc640 1 -- 172.21.15.57:0/4140333504 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f880b0440 con 0x7f9f940cb7d0 2023-10-30T14:42:50.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.180+0000 7f9f99ed6640 1 -- 172.21.15.57:0/4140333504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f940cb7d0 msgr2=0x7f9f940c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:50.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.180+0000 7f9f99ed6640 1 --2- 172.21.15.57:0/4140333504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f940cb7d0 0x7f9f940c3900 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9f88009870 tx=0x7f9f8801ed30 comp rx=0 tx=0).stop 2023-10-30T14:42:50.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.180+0000 7f9f99ed6640 1 -- 172.21.15.57:0/4140333504 shutdown_connections 2023-10-30T14:42:50.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.180+0000 7f9f99ed6640 1 --2- 172.21.15.57:0/4140333504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f940c3e40 0x7f9f94070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:50.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.180+0000 7f9f99ed6640 1 --2- 172.21.15.57:0/4140333504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f940cb7d0 0x7f9f940c3900 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:50.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.180+0000 7f9f99ed6640 1 -- 172.21.15.57:0/4140333504 >> 172.21.15.57:0/4140333504 conn(0x7f9f940cac70 msgr2=0x7f9f940cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:50.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.180+0000 7f9f99ed6640 1 -- 172.21.15.57:0/4140333504 shutdown_connections 2023-10-30T14:42:50.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.180+0000 7f9f99ed6640 1 -- 172.21.15.57:0/4140333504 wait complete. 2023-10-30T14:42:50.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f99ed6640 1 Processor -- start 2023-10-30T14:42:50.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f99ed6640 1 -- start start 2023-10-30T14:42:50.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f99ed6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f940c3e40 0x7f9f94148df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:50.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f99ed6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f940cb7d0 0x7f9f94155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:50.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f99ed6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f94150830 0x7f9f940c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:50.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f99ed6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f94149330 con 0x7f9f940cb7d0 2023-10-30T14:42:50.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f99ed6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f941ef040 con 0x7f9f94150830 2023-10-30T14:42:50.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f99ed6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f94146690 con 0x7f9f940c3e40 2023-10-30T14:42:50.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f92ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f940cb7d0 0x7f9f94155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:50.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f92ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f940cb7d0 0x7f9f94155960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39844/0 (socket says 172.21.15.57:39844) 2023-10-30T14:42:50.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f92ffd640 1 -- 172.21.15.57:0/3779694775 learned_addr learned my addr 172.21.15.57:0/3779694775 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:50.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f93fff640 1 --2- 172.21.15.57:0/3779694775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f94150830 0x7f9f940c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:50.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f937fe640 1 --2- 172.21.15.57:0/3779694775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f940c3e40 0x7f9f94148df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:50.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f92ffd640 1 -- 172.21.15.57:0/3779694775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f940c3e40 msgr2=0x7f9f94148df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:50.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f92ffd640 1 --2- 172.21.15.57:0/3779694775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f940c3e40 0x7f9f94148df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:50.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f92ffd640 1 -- 172.21.15.57:0/3779694775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f94150830 msgr2=0x7f9f940c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:50.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f92ffd640 1 --2- 172.21.15.57:0/3779694775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f94150830 0x7f9f940c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:50.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f92ffd640 1 -- 172.21.15.57:0/3779694775 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f880094d0 con 0x7f9f940cb7d0 2023-10-30T14:42:50.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f937fe640 1 --2- 172.21.15.57:0/3779694775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f940c3e40 0x7f9f94148df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:42:50.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f92ffd640 1 --2- 172.21.15.57:0/3779694775 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f940cb7d0 0x7f9f94155960 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f9f880b3040 tx=0x7f9f8801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:50.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f7bfff640 1 -- 172.21.15.57:0/3779694775 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f880b0dd0 con 0x7f9f940cb7d0 2023-10-30T14:42:50.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f7bfff640 1 -- 172.21.15.57:0/3779694775 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f880bb460 con 0x7f9f940cb7d0 2023-10-30T14:42:50.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f99ed6640 1 -- 172.21.15.57:0/3779694775 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f941ee430 con 0x7f9f940cb7d0 2023-10-30T14:42:50.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f7bfff640 1 -- 172.21.15.57:0/3779694775 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f880bb810 con 0x7f9f940cb7d0 2023-10-30T14:42:50.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.184+0000 7f9f99ed6640 1 -- 172.21.15.57:0/3779694775 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f941ee9f0 con 0x7f9f940cb7d0 2023-10-30T14:42:50.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.188+0000 7f9f99ed6640 1 -- 172.21.15.57:0/3779694775 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f94066510 con 0x7f9f940cb7d0 2023-10-30T14:42:50.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.188+0000 7f9f7bfff640 1 -- 172.21.15.57:0/3779694775 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9f880b83f0 con 0x7f9f940cb7d0 2023-10-30T14:42:50.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.188+0000 7f9f7bfff640 1 --2- 172.21.15.57:0/3779694775 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9f800664a0 0x7f9f80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:50.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.188+0000 7f9f7bfff640 1 -- 172.21.15.57:0/3779694775 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f9f88121ee0 con 0x7f9f940cb7d0 2023-10-30T14:42:50.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.188+0000 7f9f937fe640 1 --2- 172.21.15.57:0/3779694775 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9f800664a0 0x7f9f80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:50.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.188+0000 7f9f937fe640 1 --2- 172.21.15.57:0/3779694775 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9f800664a0 0x7f9f80068960 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9f84000f70 tx=0x7f9f84009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:50.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.192+0000 7f9f7bfff640 1 -- 172.21.15.57:0/3779694775 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9f880eed60 con 0x7f9f940cb7d0 2023-10-30T14:42:50.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.380+0000 7f9f99ed6640 1 -- 172.21.15.57:0/3779694775 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f9404e170 con 0x7f9f940cb7d0 2023-10-30T14:42:50.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.380+0000 7f9f7bfff640 1 -- 172.21.15.57:0/3779694775 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+5742 (secure 0 0 0) 0x7f9f880b5020 con 0x7f9f940cb7d0 2023-10-30T14:42:50.385 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:50.385 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:42:47.464671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:resolve","state_seq":39,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:50.385 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 152 2023-10-30T14:42:50.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.384+0000 7f9f99ed6640 1 -- 172.21.15.57:0/3779694775 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9f800664a0 msgr2=0x7f9f80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:50.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.384+0000 7f9f99ed6640 1 --2- 172.21.15.57:0/3779694775 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9f800664a0 0x7f9f80068960 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9f84000f70 tx=0x7f9f84009040 comp rx=0 tx=0).stop 2023-10-30T14:42:50.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.384+0000 7f9f99ed6640 1 -- 172.21.15.57:0/3779694775 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f940cb7d0 msgr2=0x7f9f94155960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:50.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.384+0000 7f9f99ed6640 1 --2- 172.21.15.57:0/3779694775 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f940cb7d0 0x7f9f94155960 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f9f880b3040 tx=0x7f9f8801bad0 comp rx=0 tx=0).stop 2023-10-30T14:42:50.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.384+0000 7f9f99ed6640 1 -- 172.21.15.57:0/3779694775 shutdown_connections 2023-10-30T14:42:50.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.384+0000 7f9f99ed6640 1 --2- 172.21.15.57:0/3779694775 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9f800664a0 0x7f9f80068960 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:50.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.384+0000 7f9f99ed6640 1 --2- 172.21.15.57:0/3779694775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f94150830 0x7f9f940c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:50.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.384+0000 7f9f99ed6640 1 --2- 172.21.15.57:0/3779694775 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f940cb7d0 0x7f9f94155960 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:50.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.384+0000 7f9f99ed6640 1 --2- 172.21.15.57:0/3779694775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f940c3e40 0x7f9f94148df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:50.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.384+0000 7f9f99ed6640 1 -- 172.21.15.57:0/3779694775 >> 172.21.15.57:0/3779694775 conn(0x7f9f940cac70 msgr2=0x7f9f940c65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:50.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.384+0000 7f9f99ed6640 1 -- 172.21.15.57:0/3779694775 shutdown_connections 2023-10-30T14:42:50.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:50.384+0000 7f9f99ed6640 1 -- 172.21.15.57:0/3779694775 wait complete. 2023-10-30T14:42:50.406 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:42:52.408 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:52.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.560+0000 7fee164ea640 1 Processor -- start 2023-10-30T14:42:52.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.560+0000 7fee164ea640 1 -- start start 2023-10-30T14:42:52.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.560+0000 7fee164ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee10151c20 0x7fee100c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:52.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.560+0000 7fee164ea640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee100c5030 con 0x7fee100cb8b0 2023-10-30T14:42:52.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.560+0000 7fee164ea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee10157830 con 0x7fee10151840 2023-10-30T14:42:52.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.560+0000 7fee164ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee10157b10 con 0x7fee10151c20 2023-10-30T14:42:52.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.560+0000 7fee14a87640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fee100cb8b0 0x7fee1006b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48886/0 (socket says 172.21.15.57:48886) 2023-10-30T14:42:52.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.560+0000 7fee14a87640 1 -- 172.21.15.57:0/3375044732 learned_addr learned my addr 172.21.15.57:0/3375044732 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:42:52.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.560+0000 7fee0f7fe640 1 --2- 172.21.15.57:0/3375044732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee10151c20 0x7fee100c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:52.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.560+0000 7fee0effd640 1 -- 172.21.15.57:0/3375044732 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229827179 0 0) 0x7fee100c5030 con 0x7fee100cb8b0 2023-10-30T14:42:52.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.560+0000 7fee0effd640 1 -- 172.21.15.57:0/3375044732 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf8003530 con 0x7fee100cb8b0 2023-10-30T14:42:52.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee0effd640 1 -- 172.21.15.57:0/3375044732 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195240127 0 0) 0x7fee10157830 con 0x7fee10151840 2023-10-30T14:42:52.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee0effd640 1 -- 172.21.15.57:0/3375044732 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee100c5030 con 0x7fee10151840 2023-10-30T14:42:52.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee0effd640 1 -- 172.21.15.57:0/3375044732 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 641440811 0 0) 0x7fedf8003530 con 0x7fee100cb8b0 2023-10-30T14:42:52.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee0effd640 1 -- 172.21.15.57:0/3375044732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee10151c20 msgr2=0x7fee100c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:52.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee0effd640 1 --2- 172.21.15.57:0/3375044732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee10151c20 0x7fee100c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:52.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee0effd640 1 -- 172.21.15.57:0/3375044732 >> v1:172.21.15.88:6789/0 conn(0x7fee10151840 legacy=0x7fee10072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:52.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee0effd640 1 -- 172.21.15.57:0/3375044732 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee10157df0 con 0x7fee100cb8b0 2023-10-30T14:42:52.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee0effd640 1 -- 172.21.15.57:0/3375044732 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fee08001a00 con 0x7fee100cb8b0 2023-10-30T14:42:52.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee0effd640 1 -- 172.21.15.57:0/3375044732 >> v1:172.21.15.57:6789/0 conn(0x7fee100cb8b0 legacy=0x7fee1006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:52.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee0effd640 1 --2- 172.21.15.57:0/3375044732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedf8003b40 0x7fedf8003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:52.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee0effd640 1 --2- 172.21.15.57:0/3375044732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee10151c20 0x7fedf80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:52.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee0effd640 1 --2- 172.21.15.57:0/3375044732 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedf8004d10 0x7fedf8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:52.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee0effd640 1 -- 172.21.15.57:0/3375044732 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee10157b10 con 0x7fedf8004d10 2023-10-30T14:42:52.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee0effd640 1 -- 172.21.15.57:0/3375044732 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf800ba40 con 0x7fedf8003b40 2023-10-30T14:42:52.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee0effd640 1 -- 172.21.15.57:0/3375044732 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf800bc00 con 0x7fee10151c20 2023-10-30T14:42:52.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee14a87640 1 --2- 172.21.15.57:0/3375044732 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedf8004d10 0x7fedf8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:52.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee0f7fe640 1 --2- 172.21.15.57:0/3375044732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee10151c20 0x7fedf80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:52.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee0ffff640 1 --2- 172.21.15.57:0/3375044732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedf8003b40 0x7fedf8003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:52.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee14a87640 1 -- 172.21.15.57:0/3375044732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee10151c20 msgr2=0x7fedf80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:52.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee14a87640 1 --2- 172.21.15.57:0/3375044732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee10151c20 0x7fedf80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:52.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee14a87640 1 -- 172.21.15.57:0/3375044732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedf8003b40 msgr2=0x7fedf8003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:52.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee14a87640 1 --2- 172.21.15.57:0/3375044732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedf8003b40 0x7fedf8003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:52.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee14a87640 1 -- 172.21.15.57:0/3375044732 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf8003530 con 0x7fedf8004d10 2023-10-30T14:42:52.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee0ffff640 1 --2- 172.21.15.57:0/3375044732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedf8003b40 0x7fedf8003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:42:52.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.564+0000 7fee14a87640 1 --2- 172.21.15.57:0/3375044732 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedf8004d10 0x7fedf8005190 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fee0800a7c0 tx=0x7fee0801d2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80f17ab45799e3d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:52.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee0effd640 1 -- 172.21.15.57:0/3375044732 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee080ad630 con 0x7fedf8004d10 2023-10-30T14:42:52.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee0effd640 1 -- 172.21.15.57:0/3375044732 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee080a8070 con 0x7fedf8004d10 2023-10-30T14:42:52.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee0effd640 1 -- 172.21.15.57:0/3375044732 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee080ad9f0 con 0x7fedf8004d10 2023-10-30T14:42:52.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee164ea640 1 -- 172.21.15.57:0/3375044732 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedf8004d10 msgr2=0x7fedf8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:52.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee164ea640 1 --2- 172.21.15.57:0/3375044732 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedf8004d10 0x7fedf8005190 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fee0800a7c0 tx=0x7fee0801d2a0 comp rx=0 tx=0).stop 2023-10-30T14:42:52.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee14a87640 1 -- 172.21.15.57:0/3375044732 reap_dead start 2023-10-30T14:42:52.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee164ea640 1 -- 172.21.15.57:0/3375044732 shutdown_connections 2023-10-30T14:42:52.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee164ea640 1 -- 172.21.15.57:0/3375044732 >> 172.21.15.57:0/3375044732 conn(0x7fee100cac70 msgr2=0x7fee100cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:52.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee164ea640 1 -- 172.21.15.57:0/3375044732 shutdown_connections 2023-10-30T14:42:52.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee164ea640 1 -- 172.21.15.57:0/3375044732 wait complete. 2023-10-30T14:42:52.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee164ea640 1 Processor -- start 2023-10-30T14:42:52.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee164ea640 1 -- start start 2023-10-30T14:42:52.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee164ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedf8004d10 0x7fee101dfd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:52.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee164ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee10150080 0x7fee10071660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:52.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee164ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee10147980 0x7fee10143a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:52.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee164ea640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee101e16a0 con 0x7fee10147980 2023-10-30T14:42:52.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee164ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee100711d0 con 0x7fedf8004d10 2023-10-30T14:42:52.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee164ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee10144c80 con 0x7fee10150080 2023-10-30T14:42:52.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee14a87640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee10147980 0x7fee10143a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:52.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee14a87640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee10147980 0x7fee10143a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39872/0 (socket says 172.21.15.57:39872) 2023-10-30T14:42:52.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.568+0000 7fee14a87640 1 -- 172.21.15.57:0/2411813733 learned_addr learned my addr 172.21.15.57:0/2411813733 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:52.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.572+0000 7fee0ffff640 1 --2- 172.21.15.57:0/2411813733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedf8004d10 0x7fee101dfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:52.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.572+0000 7fee0f7fe640 1 --2- 172.21.15.57:0/2411813733 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee10150080 0x7fee10071660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:52.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.572+0000 7fee14a87640 1 -- 172.21.15.57:0/2411813733 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee10150080 msgr2=0x7fee10071660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:52.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.572+0000 7fee14a87640 1 --2- 172.21.15.57:0/2411813733 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee10150080 0x7fee10071660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:52.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.572+0000 7fee14a87640 1 -- 172.21.15.57:0/2411813733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedf8004d10 msgr2=0x7fee101dfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:52.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.572+0000 7fee14a87640 1 --2- 172.21.15.57:0/2411813733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedf8004d10 0x7fee101dfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:52.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.572+0000 7fee14a87640 1 -- 172.21.15.57:0/2411813733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee10070b80 con 0x7fee10147980 2023-10-30T14:42:52.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.572+0000 7fee0ffff640 1 --2- 172.21.15.57:0/2411813733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedf8004d10 0x7fee101dfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:42:52.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.572+0000 7fee14a87640 1 --2- 172.21.15.57:0/2411813733 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee10147980 0x7fee10143a40 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fee0801db40 tx=0x7fedf8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:52.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.572+0000 7fee0cff9640 1 -- 172.21.15.57:0/2411813733 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee08001f60 con 0x7fee10147980 2023-10-30T14:42:52.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.572+0000 7fee0cff9640 1 -- 172.21.15.57:0/2411813733 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee08001800 con 0x7fee10147980 2023-10-30T14:42:52.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.572+0000 7fee0cff9640 1 -- 172.21.15.57:0/2411813733 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee08004d70 con 0x7fee10147980 2023-10-30T14:42:52.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.572+0000 7fee164ea640 1 -- 172.21.15.57:0/2411813733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee10070e60 con 0x7fee10147980 2023-10-30T14:42:52.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.572+0000 7fee164ea640 1 -- 172.21.15.57:0/2411813733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee100ca680 con 0x7fee10147980 2023-10-30T14:42:52.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.572+0000 7fee164ea640 1 -- 172.21.15.57:0/2411813733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee100709c0 con 0x7fee10147980 2023-10-30T14:42:52.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.572+0000 7fee0cff9640 1 -- 172.21.15.57:0/2411813733 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fee080aa030 con 0x7fee10147980 2023-10-30T14:42:52.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.572+0000 7fee0cff9640 1 --2- 172.21.15.57:0/2411813733 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fedfc0664a0 0x7fedfc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:52.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.572+0000 7fee0cff9640 1 -- 172.21.15.57:0/2411813733 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7fee0811e1a0 con 0x7fee10147980 2023-10-30T14:42:52.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.576+0000 7fee0ffff640 1 --2- 172.21.15.57:0/2411813733 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fedfc0664a0 0x7fedfc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:52.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.576+0000 7fee0ffff640 1 --2- 172.21.15.57:0/2411813733 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fedfc0664a0 0x7fedfc068960 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fee00000f70 tx=0x7fee00009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:52.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.576+0000 7fee0cff9640 1 -- 172.21.15.57:0/2411813733 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fee080eb000 con 0x7fee10147980 2023-10-30T14:42:52.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.772+0000 7fee164ea640 1 -- 172.21.15.57:0/2411813733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee10066510 con 0x7fee10147980 2023-10-30T14:42:52.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.772+0000 7fee0cff9640 1 -- 172.21.15.57:0/2411813733 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+5742 (secure 0 0 0) 0x7fee080ee620 con 0x7fee10147980 2023-10-30T14:42:52.777 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:52.777 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:42:47.464671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:resolve","state_seq":39,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:52.779 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 152 2023-10-30T14:42:52.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.772+0000 7fee164ea640 1 -- 172.21.15.57:0/2411813733 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fedfc0664a0 msgr2=0x7fedfc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:52.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.772+0000 7fee164ea640 1 --2- 172.21.15.57:0/2411813733 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fedfc0664a0 0x7fedfc068960 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fee00000f70 tx=0x7fee00009040 comp rx=0 tx=0).stop 2023-10-30T14:42:52.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.776+0000 7fee164ea640 1 -- 172.21.15.57:0/2411813733 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee10147980 msgr2=0x7fee10143a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:52.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.776+0000 7fee164ea640 1 --2- 172.21.15.57:0/2411813733 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee10147980 0x7fee10143a40 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fee0801db40 tx=0x7fedf8005800 comp rx=0 tx=0).stop 2023-10-30T14:42:52.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.776+0000 7fee164ea640 1 -- 172.21.15.57:0/2411813733 shutdown_connections 2023-10-30T14:42:52.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.776+0000 7fee164ea640 1 --2- 172.21.15.57:0/2411813733 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fedfc0664a0 0x7fedfc068960 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:52.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.776+0000 7fee164ea640 1 --2- 172.21.15.57:0/2411813733 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee10147980 0x7fee10143a40 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:52.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.776+0000 7fee164ea640 1 --2- 172.21.15.57:0/2411813733 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee10150080 0x7fee10071660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:52.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.776+0000 7fee164ea640 1 --2- 172.21.15.57:0/2411813733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedf8004d10 0x7fee101dfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:52.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.776+0000 7fee164ea640 1 -- 172.21.15.57:0/2411813733 >> 172.21.15.57:0/2411813733 conn(0x7fee100cac70 msgr2=0x7fee10146640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:52.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.776+0000 7fee164ea640 1 -- 172.21.15.57:0/2411813733 shutdown_connections 2023-10-30T14:42:52.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:52.776+0000 7fee164ea640 1 -- 172.21.15.57:0/2411813733 wait complete. 2023-10-30T14:42:52.798 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:42:54.799 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:54.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.948+0000 7f33bc73e640 1 Processor -- start 2023-10-30T14:42:54.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.948+0000 7f33bc73e640 1 -- start start 2023-10-30T14:42:54.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.952+0000 7f33bc73e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f33b4153c80 0x7f33b415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:54.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.952+0000 7f33bc73e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b4159f20 con 0x7f33b40c42a0 2023-10-30T14:42:54.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.952+0000 7f33bc73e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b415d620 con 0x7f33b40c4a10 2023-10-30T14:42:54.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.952+0000 7f33bc73e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b415dec0 con 0x7f33b4153c80 2023-10-30T14:42:54.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.952+0000 7f33bacdb640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f33b40c42a0 0x7f33b40ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44160/0 (socket says 172.21.15.57:44160) 2023-10-30T14:42:54.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.952+0000 7f33bacdb640 1 -- 172.21.15.57:0/2263696246 learned_addr learned my addr 172.21.15.57:0/2263696246 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:42:54.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.952+0000 7f33ba4da640 1 --2- 172.21.15.57:0/2263696246 >> v2:172.21.15.88:3300/0 conn(0x7f33b4153c80 0x7f33b415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:54.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.952+0000 7f33b94d8640 1 -- 172.21.15.57:0/2263696246 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079801763 0 0) 0x7f33b4159f20 con 0x7f33b40c42a0 2023-10-30T14:42:54.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.952+0000 7f33b94d8640 1 -- 172.21.15.57:0/2263696246 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33a0003530 con 0x7f33b40c42a0 2023-10-30T14:42:54.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.952+0000 7f33ba4da640 1 -- 172.21.15.57:0/2263696246 >> v1:172.21.15.88:6789/0 conn(0x7f33b40c4a10 legacy=0x7f33b415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:54.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.952+0000 7f33ba4da640 1 -- 172.21.15.57:0/2263696246 >> v1:172.21.15.57:6789/0 conn(0x7f33b40c42a0 legacy=0x7f33b40ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:54.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.952+0000 7f33b94d8640 1 -- 172.21.15.57:0/2263696246 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1882006671 0 0) 0x7f33a0003530 con 0x7f33b40c42a0 2023-10-30T14:42:54.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.952+0000 7f33ba4da640 1 -- 172.21.15.57:0/2263696246 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b415d900 con 0x7f33b4153c80 2023-10-30T14:42:54.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.956+0000 7f33ba4da640 1 --2- 172.21.15.57:0/2263696246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33b4153c80 0x7f33b415cf90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f33b00097c0 tx=0x7f33b001eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84a374f77c1cc9a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:54.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.956+0000 7f33b94d8640 1 -- 172.21.15.57:0/2263696246 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac001a00 con 0x7f33b4153c80 2023-10-30T14:42:54.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.956+0000 7f33b94d8640 1 -- 172.21.15.57:0/2263696246 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b00a8610 con 0x7f33b4153c80 2023-10-30T14:42:54.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.956+0000 7f33b94d8640 1 -- 172.21.15.57:0/2263696246 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b00a89a0 con 0x7f33b4153c80 2023-10-30T14:42:54.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.956+0000 7f33bc73e640 1 -- 172.21.15.57:0/2263696246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33b4153c80 msgr2=0x7f33b415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:54.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.956+0000 7f33bc73e640 1 --2- 172.21.15.57:0/2263696246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33b4153c80 0x7f33b415cf90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f33b00097c0 tx=0x7f33b001eb90 comp rx=0 tx=0).stop 2023-10-30T14:42:54.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.956+0000 7f33bc73e640 1 -- 172.21.15.57:0/2263696246 shutdown_connections 2023-10-30T14:42:54.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.956+0000 7f33bc73e640 1 --2- 172.21.15.57:0/2263696246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33b4153c80 0x7f33b415cf90 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:54.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.956+0000 7f33bc73e640 1 -- 172.21.15.57:0/2263696246 >> 172.21.15.57:0/2263696246 conn(0x7f33b40715c0 msgr2=0x7f33b40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:54.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.956+0000 7f33bc73e640 1 -- 172.21.15.57:0/2263696246 shutdown_connections 2023-10-30T14:42:54.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.956+0000 7f33bc73e640 1 -- 172.21.15.57:0/2263696246 wait complete. 2023-10-30T14:42:54.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33bc73e640 1 Processor -- start 2023-10-30T14:42:54.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33bc73e640 1 -- start start 2023-10-30T14:42:54.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33bc73e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33b40c42a0 0x7f33b4155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:54.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33bc73e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33b40c4a10 0x7f33b41f02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:54.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33bc73e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33b4153c80 0x7f33b41f0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:54.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33bc73e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b41526f0 con 0x7f33b40c4a10 2023-10-30T14:42:54.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33bc73e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b415c7d0 con 0x7f33b40c42a0 2023-10-30T14:42:54.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33bc73e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b415cd50 con 0x7f33b4153c80 2023-10-30T14:42:54.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33b9cd9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33b40c4a10 0x7f33b41f02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:54.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33b9cd9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33b40c4a10 0x7f33b41f02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40254/0 (socket says 172.21.15.57:40254) 2023-10-30T14:42:54.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33b9cd9640 1 -- 172.21.15.57:0/1572049703 learned_addr learned my addr 172.21.15.57:0/1572049703 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:54.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33ba4da640 1 --2- 172.21.15.57:0/1572049703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33b40c42a0 0x7f33b4155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:54.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33bacdb640 1 --2- 172.21.15.57:0/1572049703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33b4153c80 0x7f33b41f0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:54.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33b9cd9640 1 -- 172.21.15.57:0/1572049703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33b4153c80 msgr2=0x7f33b41f0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:54.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33b9cd9640 1 --2- 172.21.15.57:0/1572049703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33b4153c80 0x7f33b41f0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:54.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33b9cd9640 1 -- 172.21.15.57:0/1572049703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33b40c42a0 msgr2=0x7f33b4155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:54.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33b9cd9640 1 --2- 172.21.15.57:0/1572049703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33b40c42a0 0x7f33b4155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:54.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33b9cd9640 1 -- 172.21.15.57:0/1572049703 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b0009420 con 0x7f33b40c4a10 2023-10-30T14:42:54.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33bacdb640 1 --2- 172.21.15.57:0/1572049703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33b4153c80 0x7f33b41f0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:42:54.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33b9cd9640 1 --2- 172.21.15.57:0/1572049703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33b40c4a10 0x7f33b41f02d0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f33a400c500 tx=0x7f33a400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:54.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33aaffd640 1 -- 172.21.15.57:0/1572049703 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a40052d0 con 0x7f33b40c4a10 2023-10-30T14:42:54.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33aaffd640 1 -- 172.21.15.57:0/1572049703 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a4005430 con 0x7f33b40c4a10 2023-10-30T14:42:54.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33aaffd640 1 -- 172.21.15.57:0/1572049703 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a4012830 con 0x7f33b40c4a10 2023-10-30T14:42:54.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33bc73e640 1 -- 172.21.15.57:0/1572049703 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b4156810 con 0x7f33b40c4a10 2023-10-30T14:42:54.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33bc73e640 1 -- 172.21.15.57:0/1572049703 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b415c9e0 con 0x7f33b40c4a10 2023-10-30T14:42:54.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.960+0000 7f33bc73e640 1 -- 172.21.15.57:0/1572049703 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b404e170 con 0x7f33b40c4a10 2023-10-30T14:42:54.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.964+0000 7f33aaffd640 1 -- 172.21.15.57:0/1572049703 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f33a4012990 con 0x7f33b40c4a10 2023-10-30T14:42:54.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.964+0000 7f33aaffd640 1 --2- 172.21.15.57:0/1572049703 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f33980664a0 0x7f3398068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:54.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.964+0000 7f33aaffd640 1 -- 172.21.15.57:0/1572049703 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f33a4083f70 con 0x7f33b40c4a10 2023-10-30T14:42:54.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.964+0000 7f33ba4da640 1 --2- 172.21.15.57:0/1572049703 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f33980664a0 0x7f3398068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:54.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.964+0000 7f33ba4da640 1 --2- 172.21.15.57:0/1572049703 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f33980664a0 0x7f3398068960 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f33b00b2000 tx=0x7f33b0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:54.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:54.968+0000 7f33aaffd640 1 -- 172.21.15.57:0/1572049703 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f33a4050de0 con 0x7f33b40c4a10 2023-10-30T14:42:55.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:55.156+0000 7f33bc73e640 1 -- 172.21.15.57:0/1572049703 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33b415ce90 con 0x7f33b40c4a10 2023-10-30T14:42:55.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:55.156+0000 7f33aaffd640 1 -- 172.21.15.57:0/1572049703 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+5744 (secure 0 0 0) 0x7f33a4054400 con 0x7f33b40c4a10 2023-10-30T14:42:55.161 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:55.161 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:42:53.290619+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:reconnect","state_seq":41,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:55.162 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 153 2023-10-30T14:42:55.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:55.160+0000 7f33bc73e640 1 -- 172.21.15.57:0/1572049703 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f33980664a0 msgr2=0x7f3398068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:55.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:55.160+0000 7f33bc73e640 1 --2- 172.21.15.57:0/1572049703 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f33980664a0 0x7f3398068960 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f33b00b2000 tx=0x7f33b0008040 comp rx=0 tx=0).stop 2023-10-30T14:42:55.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:55.160+0000 7f33bc73e640 1 -- 172.21.15.57:0/1572049703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33b40c4a10 msgr2=0x7f33b41f02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:55.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:55.160+0000 7f33bc73e640 1 --2- 172.21.15.57:0/1572049703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33b40c4a10 0x7f33b41f02d0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f33a400c500 tx=0x7f33a400c9c0 comp rx=0 tx=0).stop 2023-10-30T14:42:55.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:55.160+0000 7f33bc73e640 1 -- 172.21.15.57:0/1572049703 shutdown_connections 2023-10-30T14:42:55.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:55.160+0000 7f33bc73e640 1 --2- 172.21.15.57:0/1572049703 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f33980664a0 0x7f3398068960 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:55.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:55.160+0000 7f33bc73e640 1 --2- 172.21.15.57:0/1572049703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33b4153c80 0x7f33b41f0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:55.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:55.160+0000 7f33bc73e640 1 --2- 172.21.15.57:0/1572049703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33b40c4a10 0x7f33b41f02d0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:55.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:55.160+0000 7f33bc73e640 1 --2- 172.21.15.57:0/1572049703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33b40c42a0 0x7f33b4155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:55.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:55.160+0000 7f33bc73e640 1 -- 172.21.15.57:0/1572049703 >> 172.21.15.57:0/1572049703 conn(0x7f33b40715c0 msgr2=0x7f33b406bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:55.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:55.160+0000 7f33bc73e640 1 -- 172.21.15.57:0/1572049703 shutdown_connections 2023-10-30T14:42:55.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:55.160+0000 7f33bc73e640 1 -- 172.21.15.57:0/1572049703 wait complete. 2023-10-30T14:42:55.182 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:42:57.183 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:57.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.336+0000 7ff9bf7b8640 1 Processor -- start 2023-10-30T14:42:57.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.336+0000 7ff9bf7b8640 1 -- start start 2023-10-30T14:42:57.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.336+0000 7ff9bf7b8640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff9b8153c80 0x7ff9b815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:57.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.336+0000 7ff9bf7b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9b80c42a0 0x7ff9b80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:57.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.336+0000 7ff9bf7b8640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b8159f20 con 0x7ff9b80c4a10 2023-10-30T14:42:57.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.336+0000 7ff9bf7b8640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b815dec0 con 0x7ff9b8153c80 2023-10-30T14:42:57.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.336+0000 7ff9bf7b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b81599a0 con 0x7ff9b80c42a0 2023-10-30T14:42:57.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.336+0000 7ff9bd554640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff9b8153c80 0x7ff9b815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:57.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.336+0000 7ff9bcd53640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff9b80c4a10 0x7ff9b815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44176/0 (socket says 172.21.15.57:44176) 2023-10-30T14:42:57.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.336+0000 7ff9bd554640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff9b8153c80 0x7ff9b815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40266/0 (socket says 172.21.15.57:40266) 2023-10-30T14:42:57.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.336+0000 7ff9bcd53640 1 -- 172.21.15.57:0/1548359131 learned_addr learned my addr 172.21.15.57:0/1548359131 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:42:57.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.336+0000 7ff9bdd55640 1 --2- 172.21.15.57:0/1548359131 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9b80c42a0 0x7ff9b80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:57.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.336+0000 7ff9bd554640 1 -- 172.21.15.57:0/1548359131 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9b80c42a0 msgr2=0x7ff9b80ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:57.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.336+0000 7ff9bd554640 1 --2- 172.21.15.57:0/1548359131 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9b80c42a0 0x7ff9b80ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:57.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.336+0000 7ff9bd554640 1 -- 172.21.15.57:0/1548359131 >> v1:172.21.15.57:6789/0 conn(0x7ff9b80c4a10 legacy=0x7ff9b815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:57.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.336+0000 7ff9bd554640 1 -- 172.21.15.57:0/1548359131 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b815d620 con 0x7ff9b8153c80 2023-10-30T14:42:57.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.336+0000 7ff9affff640 1 -- 172.21.15.57:0/1548359131 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438099747 0 0) 0x7ff9b8159f20 con 0x7ff9b80c4a10 2023-10-30T14:42:57.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.340+0000 7ff9bd554640 1 --2- 172.21.15.57:0/1548359131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b8153c80 0x7ff9b815cf90 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7ff9b40097c0 tx=0x7ff9b401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8def667108a1cccc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:57.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.340+0000 7ff9affff640 1 -- 172.21.15.57:0/1548359131 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b40a8970 con 0x7ff9b8153c80 2023-10-30T14:42:57.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.340+0000 7ff9affff640 1 -- 172.21.15.57:0/1548359131 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b40a8ad0 con 0x7ff9b8153c80 2023-10-30T14:42:57.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.340+0000 7ff9affff640 1 -- 172.21.15.57:0/1548359131 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b40a8e60 con 0x7ff9b8153c80 2023-10-30T14:42:57.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.340+0000 7ff9bf7b8640 1 -- 172.21.15.57:0/1548359131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b8153c80 msgr2=0x7ff9b815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:57.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.340+0000 7ff9bf7b8640 1 --2- 172.21.15.57:0/1548359131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b8153c80 0x7ff9b815cf90 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7ff9b40097c0 tx=0x7ff9b401ec70 comp rx=0 tx=0).stop 2023-10-30T14:42:57.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.340+0000 7ff9bf7b8640 1 -- 172.21.15.57:0/1548359131 shutdown_connections 2023-10-30T14:42:57.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.340+0000 7ff9bf7b8640 1 --2- 172.21.15.57:0/1548359131 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9b80c42a0 0x7ff9b80ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:57.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.340+0000 7ff9bf7b8640 1 --2- 172.21.15.57:0/1548359131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b8153c80 0x7ff9b815cf90 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:57.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.340+0000 7ff9bf7b8640 1 -- 172.21.15.57:0/1548359131 >> 172.21.15.57:0/1548359131 conn(0x7ff9b80715c0 msgr2=0x7ff9b80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:57.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.340+0000 7ff9bf7b8640 1 -- 172.21.15.57:0/1548359131 shutdown_connections 2023-10-30T14:42:57.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.340+0000 7ff9bf7b8640 1 -- 172.21.15.57:0/1548359131 wait complete. 2023-10-30T14:42:57.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bf7b8640 1 Processor -- start 2023-10-30T14:42:57.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bf7b8640 1 -- start start 2023-10-30T14:42:57.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bf7b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9b80c42a0 0x7ff9b8154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:57.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bf7b8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b80c4a10 0x7ff9b8154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:57.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bf7b8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9b8153c80 0x7ff9b8155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:57.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bf7b8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b8156620 con 0x7ff9b80c4a10 2023-10-30T14:42:57.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bf7b8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b80ceac0 con 0x7ff9b8153c80 2023-10-30T14:42:57.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bf7b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b8154850 con 0x7ff9b80c42a0 2023-10-30T14:42:57.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bcd53640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b80c4a10 0x7ff9b8154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:57.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bcd53640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b80c4a10 0x7ff9b8154ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40280/0 (socket says 172.21.15.57:40280) 2023-10-30T14:42:57.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bcd53640 1 -- 172.21.15.57:0/1639903149 learned_addr learned my addr 172.21.15.57:0/1639903149 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:57.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bd554640 1 --2- 172.21.15.57:0/1639903149 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9b80c42a0 0x7ff9b8154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:57.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bdd55640 1 --2- 172.21.15.57:0/1639903149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9b8153c80 0x7ff9b8155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:57.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bcd53640 1 -- 172.21.15.57:0/1639903149 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9b80c42a0 msgr2=0x7ff9b8154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:57.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bcd53640 1 --2- 172.21.15.57:0/1639903149 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9b80c42a0 0x7ff9b8154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:57.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bcd53640 1 -- 172.21.15.57:0/1639903149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9b8153c80 msgr2=0x7ff9b8155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:57.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bcd53640 1 --2- 172.21.15.57:0/1639903149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9b8153c80 0x7ff9b8155220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:57.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bcd53640 1 -- 172.21.15.57:0/1639903149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b4009420 con 0x7ff9b80c4a10 2023-10-30T14:42:57.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bcd53640 1 --2- 172.21.15.57:0/1639903149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b80c4a10 0x7ff9b8154ce0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7ff9a800c4e0 tx=0x7ff9a800c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:57.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9adffb640 1 -- 172.21.15.57:0/1639903149 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a80074e0 con 0x7ff9b80c4a10 2023-10-30T14:42:57.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9adffb640 1 -- 172.21.15.57:0/1639903149 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a8007660 con 0x7ff9b80c4a10 2023-10-30T14:42:57.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bf7b8640 1 -- 172.21.15.57:0/1639903149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b81f29f0 con 0x7ff9b80c4a10 2023-10-30T14:42:57.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9adffb640 1 -- 172.21.15.57:0/1639903149 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8004a30 con 0x7ff9b80c4a10 2023-10-30T14:42:57.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.344+0000 7ff9bf7b8640 1 -- 172.21.15.57:0/1639903149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b81f2fa0 con 0x7ff9b80c4a10 2023-10-30T14:42:57.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.348+0000 7ff9bf7b8640 1 -- 172.21.15.57:0/1639903149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b804e170 con 0x7ff9b80c4a10 2023-10-30T14:42:57.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.348+0000 7ff9adffb640 1 -- 172.21.15.57:0/1639903149 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff9a8004b90 con 0x7ff9b80c4a10 2023-10-30T14:42:57.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.348+0000 7ff9adffb640 1 --2- 172.21.15.57:0/1639903149 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff99c0664a0 0x7ff99c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:57.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.348+0000 7ff9adffb640 1 -- 172.21.15.57:0/1639903149 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7ff9a8083980 con 0x7ff9b80c4a10 2023-10-30T14:42:57.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.348+0000 7ff9bd554640 1 --2- 172.21.15.57:0/1639903149 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff99c0664a0 0x7ff99c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:57.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.348+0000 7ff9bd554640 1 --2- 172.21.15.57:0/1639903149 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff99c0664a0 0x7ff99c068960 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff9b40b2000 tx=0x7ff9b4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:57.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.352+0000 7ff9adffb640 1 -- 172.21.15.57:0/1639903149 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff9a80508c0 con 0x7ff9b80c4a10 2023-10-30T14:42:57.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.544+0000 7ff9bf7b8640 1 -- 172.21.15.57:0/1639903149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9b814dae0 con 0x7ff9b80c4a10 2023-10-30T14:42:57.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.544+0000 7ff9adffb640 1 -- 172.21.15.57:0/1639903149 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+5744 (secure 0 0 0) 0x7ff9a8018020 con 0x7ff9b80c4a10 2023-10-30T14:42:57.549 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:57.549 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:42:53.290619+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:reconnect","state_seq":41,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:57.550 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 153 2023-10-30T14:42:57.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.548+0000 7ff9bf7b8640 1 -- 172.21.15.57:0/1639903149 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff99c0664a0 msgr2=0x7ff99c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:57.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.548+0000 7ff9bf7b8640 1 --2- 172.21.15.57:0/1639903149 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff99c0664a0 0x7ff99c068960 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff9b40b2000 tx=0x7ff9b4008040 comp rx=0 tx=0).stop 2023-10-30T14:42:57.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.548+0000 7ff9bf7b8640 1 -- 172.21.15.57:0/1639903149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b80c4a10 msgr2=0x7ff9b8154ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:57.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.548+0000 7ff9bf7b8640 1 --2- 172.21.15.57:0/1639903149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b80c4a10 0x7ff9b8154ce0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7ff9a800c4e0 tx=0x7ff9a800c9a0 comp rx=0 tx=0).stop 2023-10-30T14:42:57.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.548+0000 7ff9bf7b8640 1 -- 172.21.15.57:0/1639903149 shutdown_connections 2023-10-30T14:42:57.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.548+0000 7ff9bf7b8640 1 --2- 172.21.15.57:0/1639903149 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff99c0664a0 0x7ff99c068960 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:57.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.548+0000 7ff9bf7b8640 1 --2- 172.21.15.57:0/1639903149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9b8153c80 0x7ff9b8155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:57.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.548+0000 7ff9bf7b8640 1 --2- 172.21.15.57:0/1639903149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b80c4a10 0x7ff9b8154ce0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:57.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.548+0000 7ff9bf7b8640 1 --2- 172.21.15.57:0/1639903149 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9b80c42a0 0x7ff9b8154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:57.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.548+0000 7ff9bf7b8640 1 -- 172.21.15.57:0/1639903149 >> 172.21.15.57:0/1639903149 conn(0x7ff9b80715c0 msgr2=0x7ff9b8158450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:57.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.548+0000 7ff9bf7b8640 1 -- 172.21.15.57:0/1639903149 shutdown_connections 2023-10-30T14:42:57.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:57.548+0000 7ff9bf7b8640 1 -- 172.21.15.57:0/1639903149 wait complete. 2023-10-30T14:42:57.571 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:42:59.572 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:42:59.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.724+0000 7fd0b8bc5640 1 Processor -- start 2023-10-30T14:42:59.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.724+0000 7fd0b8bc5640 1 -- start start 2023-10-30T14:42:59.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.724+0000 7fd0b8bc5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd0b40c55d0 0x7fd0b406e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:59.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.724+0000 7fd0b8bc5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0b406eb70 0x7fd0b4151480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:59.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.724+0000 7fd0b8bc5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b41578e0 con 0x7fd0b40c59b0 2023-10-30T14:42:59.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.724+0000 7fd0b8bc5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b4158720 con 0x7fd0b40c55d0 2023-10-30T14:42:59.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.724+0000 7fd0b8bc5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b41576f0 con 0x7fd0b406eb70 2023-10-30T14:42:59.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.724+0000 7fd0b259c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd0b40c55d0 0x7fd0b406e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:59.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.724+0000 7fd0b259c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd0b40c55d0 0x7fd0b406e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40290/0 (socket says 172.21.15.57:40290) 2023-10-30T14:42:59.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.724+0000 7fd0b259c640 1 -- 172.21.15.57:0/3253460164 learned_addr learned my addr 172.21.15.57:0/3253460164 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:59.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.724+0000 7fd0abfff640 1 --2- 172.21.15.57:0/3253460164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0b406eb70 0x7fd0b4151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:59.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.724+0000 7fd0b259c640 1 -- 172.21.15.57:0/3253460164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0b406eb70 msgr2=0x7fd0b4151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:42:59.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.724+0000 7fd0b259c640 1 --2- 172.21.15.57:0/3253460164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0b406eb70 0x7fd0b4151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:59.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.724+0000 7fd0b259c640 1 -- 172.21.15.57:0/3253460164 >> v1:172.21.15.88:6789/0 conn(0x7fd0b40c59b0 legacy=0x7fd0b415b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:59.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.728+0000 7fd0b259c640 1 -- 172.21.15.57:0/3253460164 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b4157ba0 con 0x7fd0b40c55d0 2023-10-30T14:42:59.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.728+0000 7fd0b259c640 1 --2- 172.21.15.57:0/3253460164 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0b40c55d0 0x7fd0b406e630 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4009870 tx=0x7fd0a401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d0e5bd29f3dfb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:59.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.728+0000 7fd0b1d9b640 1 -- 172.21.15.57:0/3253460164 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a40a8a30 con 0x7fd0b40c55d0 2023-10-30T14:42:59.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.728+0000 7fd0b1d9b640 1 -- 172.21.15.57:0/3253460164 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a40a8b90 con 0x7fd0b40c55d0 2023-10-30T14:42:59.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.728+0000 7fd0b1d9b640 1 -- 172.21.15.57:0/3253460164 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a40b0440 con 0x7fd0b40c55d0 2023-10-30T14:42:59.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.728+0000 7fd0b8bc5640 1 -- 172.21.15.57:0/3253460164 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0b40c55d0 msgr2=0x7fd0b406e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:59.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.728+0000 7fd0b8bc5640 1 --2- 172.21.15.57:0/3253460164 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0b40c55d0 0x7fd0b406e630 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4009870 tx=0x7fd0a401ed30 comp rx=0 tx=0).stop 2023-10-30T14:42:59.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.728+0000 7fd0b8bc5640 1 -- 172.21.15.57:0/3253460164 shutdown_connections 2023-10-30T14:42:59.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.728+0000 7fd0b8bc5640 1 --2- 172.21.15.57:0/3253460164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0b406eb70 0x7fd0b4151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:59.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.728+0000 7fd0b8bc5640 1 --2- 172.21.15.57:0/3253460164 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0b40c55d0 0x7fd0b406e630 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:59.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.728+0000 7fd0b8bc5640 1 -- 172.21.15.57:0/3253460164 >> 172.21.15.57:0/3253460164 conn(0x7fd0b406afa0 msgr2=0x7fd0b406b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:59.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.728+0000 7fd0b8bc5640 1 -- 172.21.15.57:0/3253460164 shutdown_connections 2023-10-30T14:42:59.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.728+0000 7fd0b8bc5640 1 -- 172.21.15.57:0/3253460164 wait complete. 2023-10-30T14:42:59.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0b8bc5640 1 Processor -- start 2023-10-30T14:42:59.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0b8bc5640 1 -- start start 2023-10-30T14:42:59.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0b8bc5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0b406eb70 0x7fd0b40c3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:59.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0b8bc5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0b40c55d0 0x7fd0b40721e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:59.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0b8bc5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0b40c59b0 0x7fd0b4159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:59.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0b8bc5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b40c09d0 con 0x7fd0b40c55d0 2023-10-30T14:42:59.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0b8bc5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b41590a0 con 0x7fd0b406eb70 2023-10-30T14:42:59.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0b8bc5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b40c4200 con 0x7fd0b40c59b0 2023-10-30T14:42:59.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0abfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0b40c55d0 0x7fd0b40721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:59.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0abfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0b40c55d0 0x7fd0b40721e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40304/0 (socket says 172.21.15.57:40304) 2023-10-30T14:42:59.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0abfff640 1 -- 172.21.15.57:0/383730397 learned_addr learned my addr 172.21.15.57:0/383730397 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:42:59.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0b259c640 1 --2- 172.21.15.57:0/383730397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0b406eb70 0x7fd0b40c3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:59.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0b2d9d640 1 --2- 172.21.15.57:0/383730397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0b40c59b0 0x7fd0b4159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:59.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0abfff640 1 -- 172.21.15.57:0/383730397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0b40c59b0 msgr2=0x7fd0b4159af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:59.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0abfff640 1 --2- 172.21.15.57:0/383730397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0b40c59b0 0x7fd0b4159af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:59.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0abfff640 1 -- 172.21.15.57:0/383730397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0b406eb70 msgr2=0x7fd0b40c3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:59.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0abfff640 1 --2- 172.21.15.57:0/383730397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0b406eb70 0x7fd0b40c3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:59.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0abfff640 1 -- 172.21.15.57:0/383730397 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a40094d0 con 0x7fd0b40c55d0 2023-10-30T14:42:59.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0b2d9d640 1 --2- 172.21.15.57:0/383730397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0b40c59b0 0x7fd0b4159af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:42:59.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0abfff640 1 --2- 172.21.15.57:0/383730397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0b40c55d0 0x7fd0b40721e0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fd09c00e930 tx=0x7fd09c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:59.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0aaffd640 1 -- 172.21.15.57:0/383730397 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c009690 con 0x7fd0b40c55d0 2023-10-30T14:42:59.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.732+0000 7fd0aaffd640 1 -- 172.21.15.57:0/383730397 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09c00ca00 con 0x7fd0b40c55d0 2023-10-30T14:42:59.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.736+0000 7fd0aaffd640 1 -- 172.21.15.57:0/383730397 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c00cd70 con 0x7fd0b40c55d0 2023-10-30T14:42:59.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.736+0000 7fd0b8bc5640 1 -- 172.21.15.57:0/383730397 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b40c44e0 con 0x7fd0b40c55d0 2023-10-30T14:42:59.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.736+0000 7fd0b8bc5640 1 -- 172.21.15.57:0/383730397 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b41592f0 con 0x7fd0b40c55d0 2023-10-30T14:42:59.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.736+0000 7fd0b8bc5640 1 -- 172.21.15.57:0/383730397 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b404e170 con 0x7fd0b40c55d0 2023-10-30T14:42:59.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.736+0000 7fd0aaffd640 1 -- 172.21.15.57:0/383730397 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd09c013470 con 0x7fd0b40c55d0 2023-10-30T14:42:59.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.736+0000 7fd0aaffd640 1 --2- 172.21.15.57:0/383730397 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0940664a0 0x7fd094068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:42:59.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.736+0000 7fd0aaffd640 1 -- 172.21.15.57:0/383730397 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7fd09c084620 con 0x7fd0b40c55d0 2023-10-30T14:42:59.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.736+0000 7fd0b259c640 1 --2- 172.21.15.57:0/383730397 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0940664a0 0x7fd094068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:42:59.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.740+0000 7fd0b259c640 1 --2- 172.21.15.57:0/383730397 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0940664a0 0x7fd094068960 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd0a40b2000 tx=0x7fd0a4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:42:59.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.740+0000 7fd0aaffd640 1 -- 172.21.15.57:0/383730397 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd09c0515f0 con 0x7fd0b40c55d0 2023-10-30T14:42:59.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.932+0000 7fd0b8bc5640 1 -- 172.21.15.57:0/383730397 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0b40c9440 con 0x7fd0b40c55d0 2023-10-30T14:42:59.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.932+0000 7fd0aaffd640 1 -- 172.21.15.57:0/383730397 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+5741 (secure 0 0 0) 0x7fd09c054c10 con 0x7fd0b40c55d0 2023-10-30T14:42:59.937 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:42:59.938 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:42:59.252969+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:rejoin","state_seq":43,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:42:59.939 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 154 2023-10-30T14:42:59.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.936+0000 7fd0b8bc5640 1 -- 172.21.15.57:0/383730397 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0940664a0 msgr2=0x7fd094068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:59.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.936+0000 7fd0b8bc5640 1 --2- 172.21.15.57:0/383730397 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0940664a0 0x7fd094068960 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd0a40b2000 tx=0x7fd0a4009040 comp rx=0 tx=0).stop 2023-10-30T14:42:59.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.936+0000 7fd0b8bc5640 1 -- 172.21.15.57:0/383730397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0b40c55d0 msgr2=0x7fd0b40721e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:42:59.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.936+0000 7fd0b8bc5640 1 --2- 172.21.15.57:0/383730397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0b40c55d0 0x7fd0b40721e0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fd09c00e930 tx=0x7fd09c00edf0 comp rx=0 tx=0).stop 2023-10-30T14:42:59.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.936+0000 7fd0b8bc5640 1 -- 172.21.15.57:0/383730397 shutdown_connections 2023-10-30T14:42:59.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.936+0000 7fd0b8bc5640 1 --2- 172.21.15.57:0/383730397 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd0940664a0 0x7fd094068960 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:59.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.936+0000 7fd0b8bc5640 1 --2- 172.21.15.57:0/383730397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0b40c59b0 0x7fd0b4159af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:59.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.936+0000 7fd0b8bc5640 1 --2- 172.21.15.57:0/383730397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0b40c55d0 0x7fd0b40721e0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:59.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.936+0000 7fd0b8bc5640 1 --2- 172.21.15.57:0/383730397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0b406eb70 0x7fd0b40c3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:42:59.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.936+0000 7fd0b8bc5640 1 -- 172.21.15.57:0/383730397 >> 172.21.15.57:0/383730397 conn(0x7fd0b406afa0 msgr2=0x7fd0b406b380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:42:59.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.936+0000 7fd0b8bc5640 1 -- 172.21.15.57:0/383730397 shutdown_connections 2023-10-30T14:42:59.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:42:59.936+0000 7fd0b8bc5640 1 -- 172.21.15.57:0/383730397 wait complete. 2023-10-30T14:42:59.959 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 154, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 154, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6356": { "addr": "172.21.15.88:6839/1744808283", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1744808283, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1744808283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6356, "incarnation": 151, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:rejoin", "state_seq": 43 }, "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 44 }, "gid_7056": { "addr": "172.21.15.57:6837/2307041430", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 2307041430, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 2307041430, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 7056, "incarnation": 137, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:42:59.252969+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6356, "mds_1": 6747, "mds_2": 7056 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:43:01.960 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:02.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.112+0000 7f8c1ed01640 1 Processor -- start 2023-10-30T14:43:02.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.112+0000 7f8c1ed01640 1 -- start start 2023-10-30T14:43:02.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.112+0000 7f8c1ed01640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8c18069e90 0x7f8c1806f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:02.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.112+0000 7f8c1ed01640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8c180c9c40 0x7f8c1806a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:02.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.112+0000 7f8c1ed01640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c180c8440 con 0x7f8c181597d0 2023-10-30T14:43:02.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.112+0000 7f8c1ed01640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c1806a7b0 con 0x7f8c180c9c40 2023-10-30T14:43:02.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.112+0000 7f8c1ed01640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c18153f60 con 0x7f8c18069e90 2023-10-30T14:43:02.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.112+0000 7f8c17fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8c180c9c40 0x7f8c1806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:02.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.112+0000 7f8c17fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8c180c9c40 0x7f8c1806a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40312/0 (socket says 172.21.15.57:40312) 2023-10-30T14:43:02.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.112+0000 7f8c17fff640 1 -- 172.21.15.57:0/2662730163 learned_addr learned my addr 172.21.15.57:0/2662730163 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:02.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.112+0000 7f8c1d29e640 1 --1- 172.21.15.57:0/2662730163 >> v1:172.21.15.57:6789/0 conn(0x7f8c181597d0 0x7f8c180c8e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44178/0 (socket says 172.21.15.57:44178) 2023-10-30T14:43:02.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.112+0000 7f8c1ca9d640 1 --2- 172.21.15.57:0/2662730163 >> v2:172.21.15.88:3300/0 conn(0x7f8c18069e90 0x7f8c1806f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:02.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.116+0000 7f8c17fff640 1 -- 172.21.15.57:0/2662730163 >> v2:172.21.15.88:3300/0 conn(0x7f8c18069e90 msgr2=0x7f8c1806f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:02.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.116+0000 7f8c17fff640 1 --2- 172.21.15.57:0/2662730163 >> v2:172.21.15.88:3300/0 conn(0x7f8c18069e90 0x7f8c1806f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:02.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.116+0000 7f8c17fff640 1 -- 172.21.15.57:0/2662730163 >> v1:172.21.15.57:6789/0 conn(0x7f8c181597d0 legacy=0x7f8c180c8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:02.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.116+0000 7f8c17fff640 1 -- 172.21.15.57:0/2662730163 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c180c89f0 con 0x7f8c180c9c40 2023-10-30T14:43:02.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.116+0000 7f8c177fe640 1 -- 172.21.15.57:0/2662730163 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563765614 0 0) 0x7f8c100014a0 con 0x7f8c181597d0 2023-10-30T14:43:02.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.116+0000 7f8c17fff640 1 --2- 172.21.15.57:0/2662730163 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c180c9c40 0x7f8c1806a270 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f8c1806eec0 tx=0x7f8c0401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58260974f1d6fbd6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:02.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.116+0000 7f8c177fe640 1 -- 172.21.15.57:0/2662730163 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c040ab600 con 0x7f8c180c9c40 2023-10-30T14:43:02.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.116+0000 7f8c177fe640 1 -- 172.21.15.57:0/2662730163 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c040a7050 con 0x7f8c180c9c40 2023-10-30T14:43:02.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.116+0000 7f8c177fe640 1 -- 172.21.15.57:0/2662730163 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c040ab980 con 0x7f8c180c9c40 2023-10-30T14:43:02.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c1ed01640 1 -- 172.21.15.57:0/2662730163 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c180c9c40 msgr2=0x7f8c1806a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:02.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c1ed01640 1 --2- 172.21.15.57:0/2662730163 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c180c9c40 0x7f8c1806a270 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f8c1806eec0 tx=0x7f8c0401c500 comp rx=0 tx=0).stop 2023-10-30T14:43:02.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c1ed01640 1 -- 172.21.15.57:0/2662730163 shutdown_connections 2023-10-30T14:43:02.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c1ed01640 1 --2- 172.21.15.57:0/2662730163 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c180c9c40 0x7f8c1806a270 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:02.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c1ed01640 1 --2- 172.21.15.57:0/2662730163 >> v2:172.21.15.88:3300/0 conn(0x7f8c18069e90 0x7f8c1806f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:02.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c1ed01640 1 -- 172.21.15.57:0/2662730163 >> 172.21.15.57:0/2662730163 conn(0x7f8c1806afa0 msgr2=0x7f8c1806b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:02.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c1ed01640 1 -- 172.21.15.57:0/2662730163 shutdown_connections 2023-10-30T14:43:02.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c1ed01640 1 -- 172.21.15.57:0/2662730163 wait complete. 2023-10-30T14:43:02.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c1ed01640 1 Processor -- start 2023-10-30T14:43:02.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c1ed01640 1 -- start start 2023-10-30T14:43:02.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c1ed01640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c18069e90 0x7f8c181f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:02.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c1ed01640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c180c9c40 0x7f8c1814f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:02.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c1ed01640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c181597d0 0x7f8c180c3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:02.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c1ed01640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c181f24e0 con 0x7f8c180c9c40 2023-10-30T14:43:02.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c1ed01640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c18150040 con 0x7f8c181597d0 2023-10-30T14:43:02.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c1ed01640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c181f5f90 con 0x7f8c18069e90 2023-10-30T14:43:02.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c17fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c180c9c40 0x7f8c1814f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:02.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c17fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c180c9c40 0x7f8c1814f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40326/0 (socket says 172.21.15.57:40326) 2023-10-30T14:43:02.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c17fff640 1 -- 172.21.15.57:0/377946654 learned_addr learned my addr 172.21.15.57:0/377946654 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:02.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c1ca9d640 1 --2- 172.21.15.57:0/377946654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c18069e90 0x7f8c181f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:02.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c1d29e640 1 --2- 172.21.15.57:0/377946654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c181597d0 0x7f8c180c3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:02.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c17fff640 1 -- 172.21.15.57:0/377946654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c18069e90 msgr2=0x7f8c181f5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:02.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c17fff640 1 --2- 172.21.15.57:0/377946654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c18069e90 0x7f8c181f5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:02.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c17fff640 1 -- 172.21.15.57:0/377946654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c181597d0 msgr2=0x7f8c180c3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:02.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c17fff640 1 --2- 172.21.15.57:0/377946654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c181597d0 0x7f8c180c3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:02.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c17fff640 1 -- 172.21.15.57:0/377946654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c040094d0 con 0x7f8c180c9c40 2023-10-30T14:43:02.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c1d29e640 1 --2- 172.21.15.57:0/377946654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c181597d0 0x7f8c180c3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:43:02.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.120+0000 7f8c17fff640 1 --2- 172.21.15.57:0/377946654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c180c9c40 0x7f8c1814f6c0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f8c04019110 tx=0x7f8c040192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:02.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.124+0000 7f8c157fa640 1 -- 172.21.15.57:0/377946654 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c040b4620 con 0x7f8c180c9c40 2023-10-30T14:43:02.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.124+0000 7f8c157fa640 1 -- 172.21.15.57:0/377946654 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c040b4780 con 0x7f8c180c9c40 2023-10-30T14:43:02.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.124+0000 7f8c157fa640 1 -- 172.21.15.57:0/377946654 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c040b4b30 con 0x7f8c180c9c40 2023-10-30T14:43:02.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.124+0000 7f8c1ed01640 1 -- 172.21.15.57:0/377946654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c18151230 con 0x7f8c180c9c40 2023-10-30T14:43:02.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.124+0000 7f8c1ed01640 1 -- 172.21.15.57:0/377946654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c181f1e20 con 0x7f8c180c9c40 2023-10-30T14:43:02.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.124+0000 7f8c1ed01640 1 -- 172.21.15.57:0/377946654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c181f2620 con 0x7f8c180c9c40 2023-10-30T14:43:02.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.124+0000 7f8c157fa640 1 -- 172.21.15.57:0/377946654 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8c040b2070 con 0x7f8c180c9c40 2023-10-30T14:43:02.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.124+0000 7f8c157fa640 1 --2- 172.21.15.57:0/377946654 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8bfc0664d0 0x7f8bfc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:02.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.124+0000 7f8c157fa640 1 -- 172.21.15.57:0/377946654 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f8c0411d510 con 0x7f8c180c9c40 2023-10-30T14:43:02.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.124+0000 7f8c1ca9d640 1 --2- 172.21.15.57:0/377946654 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8bfc0664d0 0x7f8bfc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:02.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.128+0000 7f8c1ca9d640 1 --2- 172.21.15.57:0/377946654 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8bfc0664d0 0x7f8bfc068990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0079c0 tx=0x7f8c0c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:02.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.128+0000 7f8c157fa640 1 -- 172.21.15.57:0/377946654 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8c181f2620 con 0x7f8c180c9c40 2023-10-30T14:43:02.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.320+0000 7f8c1ed01640 1 -- 172.21.15.57:0/377946654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c181f2620 con 0x7f8c180c9c40 2023-10-30T14:43:02.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.320+0000 7f8c157fa640 1 -- 172.21.15.57:0/377946654 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+5741 (secure 0 0 0) 0x7f8c181f2620 con 0x7f8c180c9c40 2023-10-30T14:43:02.326 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:02.326 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:42:59.252969+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:rejoin","state_seq":43,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:02.327 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 154 2023-10-30T14:43:02.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.324+0000 7f8c1ed01640 1 -- 172.21.15.57:0/377946654 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8bfc0664d0 msgr2=0x7f8bfc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:02.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.324+0000 7f8c1ed01640 1 --2- 172.21.15.57:0/377946654 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8bfc0664d0 0x7f8bfc068990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0079c0 tx=0x7f8c0c008040 comp rx=0 tx=0).stop 2023-10-30T14:43:02.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.324+0000 7f8c1ed01640 1 -- 172.21.15.57:0/377946654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c180c9c40 msgr2=0x7f8c1814f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:02.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.324+0000 7f8c1ed01640 1 --2- 172.21.15.57:0/377946654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c180c9c40 0x7f8c1814f6c0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f8c04019110 tx=0x7f8c040192a0 comp rx=0 tx=0).stop 2023-10-30T14:43:02.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.324+0000 7f8c1ed01640 1 -- 172.21.15.57:0/377946654 shutdown_connections 2023-10-30T14:43:02.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.324+0000 7f8c1ed01640 1 --2- 172.21.15.57:0/377946654 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8bfc0664d0 0x7f8bfc068990 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:02.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.324+0000 7f8c1ed01640 1 --2- 172.21.15.57:0/377946654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c181597d0 0x7f8c180c3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:02.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.324+0000 7f8c1ed01640 1 --2- 172.21.15.57:0/377946654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c180c9c40 0x7f8c1814f6c0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:02.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.324+0000 7f8c1ed01640 1 --2- 172.21.15.57:0/377946654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c18069e90 0x7f8c181f5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:02.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.324+0000 7f8c1ed01640 1 -- 172.21.15.57:0/377946654 >> 172.21.15.57:0/377946654 conn(0x7f8c1806afa0 msgr2=0x7f8c1806b380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:02.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.324+0000 7f8c1ed01640 1 -- 172.21.15.57:0/377946654 shutdown_connections 2023-10-30T14:43:02.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:02.324+0000 7f8c1ed01640 1 -- 172.21.15.57:0/377946654 wait complete. 2023-10-30T14:43:02.348 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:43:04.349 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.500+0000 7fe59f994640 1 Processor -- start 2023-10-30T14:43:04.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.500+0000 7fe59f994640 1 -- start start 2023-10-30T14:43:04.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.504+0000 7fe59f994640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe598150830 0x7fe59806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:04.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.504+0000 7fe59f994640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe598070520 0x7fe5980709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:04.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.504+0000 7fe59f994640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5980c2310 con 0x7fe5980cb7d0 2023-10-30T14:43:04.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.504+0000 7fe59f994640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe598156710 con 0x7fe598150830 2023-10-30T14:43:04.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.504+0000 7fe59f994640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5980c4360 con 0x7fe598070520 2023-10-30T14:43:04.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.504+0000 7fe59df31640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe598070520 0x7fe5980709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:04.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.504+0000 7fe59d730640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe598150830 0x7fe59806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:04.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.504+0000 7fe59df31640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe598070520 0x7fe5980709b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:33548/0 (socket says 172.21.15.57:33548) 2023-10-30T14:43:04.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.504+0000 7fe59df31640 1 -- 172.21.15.57:0/725998141 learned_addr learned my addr 172.21.15.57:0/725998141 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:04.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.504+0000 7fe59df31640 1 -- 172.21.15.57:0/725998141 >> v2:172.21.15.88:3300/0 conn(0x7fe598150830 msgr2=0x7fe59806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:04.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.504+0000 7fe59df31640 1 --2- 172.21.15.57:0/725998141 >> v2:172.21.15.88:3300/0 conn(0x7fe598150830 0x7fe59806b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:04.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.504+0000 7fe59df31640 1 -- 172.21.15.57:0/725998141 >> v1:172.21.15.88:6789/0 conn(0x7fe5980cb7d0 legacy=0x7fe5980c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:04.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.504+0000 7fe59df31640 1 -- 172.21.15.57:0/725998141 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5981569b0 con 0x7fe598070520 2023-10-30T14:43:04.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.504+0000 7fe59d730640 1 --2- 172.21.15.57:0/725998141 >> v2:172.21.15.88:3300/0 conn(0x7fe598150830 0x7fe59806b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:43:04.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.508+0000 7fe59df31640 1 --2- 172.21.15.57:0/725998141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe598070520 0x7fe5980709b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe590009e50 tx=0x7fe59001f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=740511ba218b0d04 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:04.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.508+0000 7fe58ffff640 1 -- 172.21.15.57:0/725998141 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5900af400 con 0x7fe598070520 2023-10-30T14:43:04.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.508+0000 7fe58ffff640 1 -- 172.21.15.57:0/725998141 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5900ab050 con 0x7fe598070520 2023-10-30T14:43:04.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.508+0000 7fe58ffff640 1 -- 172.21.15.57:0/725998141 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5900af780 con 0x7fe598070520 2023-10-30T14:43:04.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.508+0000 7fe59f994640 1 -- 172.21.15.57:0/725998141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe598070520 msgr2=0x7fe5980709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:04.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.508+0000 7fe59f994640 1 --2- 172.21.15.57:0/725998141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe598070520 0x7fe5980709b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe590009e50 tx=0x7fe59001f1f0 comp rx=0 tx=0).stop 2023-10-30T14:43:04.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.508+0000 7fe59f994640 1 -- 172.21.15.57:0/725998141 shutdown_connections 2023-10-30T14:43:04.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.508+0000 7fe59f994640 1 --2- 172.21.15.57:0/725998141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe598070520 0x7fe5980709b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:04.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.508+0000 7fe59f994640 1 --2- 172.21.15.57:0/725998141 >> v2:172.21.15.88:3300/0 conn(0x7fe598150830 0x7fe59806b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:04.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.508+0000 7fe59f994640 1 -- 172.21.15.57:0/725998141 >> 172.21.15.57:0/725998141 conn(0x7fe5980cac70 msgr2=0x7fe5980cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:04.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.508+0000 7fe59f994640 1 -- 172.21.15.57:0/725998141 shutdown_connections 2023-10-30T14:43:04.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.508+0000 7fe59f994640 1 -- 172.21.15.57:0/725998141 wait complete. 2023-10-30T14:43:04.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.508+0000 7fe59f994640 1 Processor -- start 2023-10-30T14:43:04.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.508+0000 7fe59f994640 1 -- start start 2023-10-30T14:43:04.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.512+0000 7fe59f994640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe598070520 0x7fe5981f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:04.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.512+0000 7fe59f994640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5980cb7d0 0x7fe5981f27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:04.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.512+0000 7fe59f994640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe598150830 0x7fe5981f0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:04.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.512+0000 7fe59f994640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59815a680 con 0x7fe5980cb7d0 2023-10-30T14:43:04.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.512+0000 7fe59f994640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59814d020 con 0x7fe598070520 2023-10-30T14:43:04.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.512+0000 7fe59f994640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe598153c60 con 0x7fe598150830 2023-10-30T14:43:04.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.512+0000 7fe59cf2f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5980cb7d0 0x7fe5981f27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:04.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.512+0000 7fe59cf2f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5980cb7d0 0x7fe5981f27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44470/0 (socket says 172.21.15.57:44470) 2023-10-30T14:43:04.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.512+0000 7fe59cf2f640 1 -- 172.21.15.57:0/3623335219 learned_addr learned my addr 172.21.15.57:0/3623335219 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:04.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.512+0000 7fe59d730640 1 --2- 172.21.15.57:0/3623335219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe598070520 0x7fe5981f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:04.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.512+0000 7fe59df31640 1 --2- 172.21.15.57:0/3623335219 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe598150830 0x7fe5981f0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:04.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.512+0000 7fe59cf2f640 1 -- 172.21.15.57:0/3623335219 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe598150830 msgr2=0x7fe5981f0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:04.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.512+0000 7fe59cf2f640 1 --2- 172.21.15.57:0/3623335219 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe598150830 0x7fe5981f0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:04.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.512+0000 7fe59cf2f640 1 -- 172.21.15.57:0/3623335219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe598070520 msgr2=0x7fe5981f22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:04.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.512+0000 7fe59cf2f640 1 --2- 172.21.15.57:0/3623335219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe598070520 0x7fe5981f22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:04.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.512+0000 7fe59cf2f640 1 -- 172.21.15.57:0/3623335219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe594009420 con 0x7fe5980cb7d0 2023-10-30T14:43:04.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.512+0000 7fe59df31640 1 --2- 172.21.15.57:0/3623335219 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe598150830 0x7fe5981f0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:43:04.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.512+0000 7fe59cf2f640 1 --2- 172.21.15.57:0/3623335219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5980cb7d0 0x7fe5981f27f0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fe58800c500 tx=0x7fe58800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:04.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.516+0000 7fe58dffb640 1 -- 172.21.15.57:0/3623335219 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe588005220 con 0x7fe5980cb7d0 2023-10-30T14:43:04.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.516+0000 7fe58dffb640 1 -- 172.21.15.57:0/3623335219 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe588005380 con 0x7fe5980cb7d0 2023-10-30T14:43:04.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.516+0000 7fe58dffb640 1 -- 172.21.15.57:0/3623335219 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe588007270 con 0x7fe5980cb7d0 2023-10-30T14:43:04.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.516+0000 7fe59f994640 1 -- 172.21.15.57:0/3623335219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe590009b30 con 0x7fe5980cb7d0 2023-10-30T14:43:04.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.516+0000 7fe59f994640 1 -- 172.21.15.57:0/3623335219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5980c61e0 con 0x7fe5980cb7d0 2023-10-30T14:43:04.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.516+0000 7fe59f994640 1 -- 172.21.15.57:0/3623335219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe59804e170 con 0x7fe5980cb7d0 2023-10-30T14:43:04.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.516+0000 7fe58dffb640 1 -- 172.21.15.57:0/3623335219 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe5880073d0 con 0x7fe5980cb7d0 2023-10-30T14:43:04.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.520+0000 7fe58dffb640 1 --2- 172.21.15.57:0/3623335219 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe57c0664a0 0x7fe57c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:04.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.520+0000 7fe58dffb640 1 -- 172.21.15.57:0/3623335219 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7fe588083c30 con 0x7fe5980cb7d0 2023-10-30T14:43:04.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.520+0000 7fe59d730640 1 --2- 172.21.15.57:0/3623335219 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe57c0664a0 0x7fe57c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:04.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.520+0000 7fe59d730640 1 --2- 172.21.15.57:0/3623335219 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe57c0664a0 0x7fe57c068960 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe594009c50 tx=0x7fe594009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:04.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.524+0000 7fe58dffb640 1 -- 172.21.15.57:0/3623335219 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe58808a080 con 0x7fe5980cb7d0 2023-10-30T14:43:04.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.716+0000 7fe59f994640 1 -- 172.21.15.57:0/3623335219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe598070300 con 0x7fe5980cb7d0 2023-10-30T14:43:04.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.716+0000 7fe58dffb640 1 -- 172.21.15.57:0/3623335219 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5741 (secure 0 0 0) 0x7fe588050b20 con 0x7fe5980cb7d0 2023-10-30T14:43:04.721 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:04.721 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:04.464878+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:04.722 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 155 2023-10-30T14:43:04.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.720+0000 7fe59f994640 1 -- 172.21.15.57:0/3623335219 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe57c0664a0 msgr2=0x7fe57c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:04.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.720+0000 7fe59f994640 1 --2- 172.21.15.57:0/3623335219 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe57c0664a0 0x7fe57c068960 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe594009c50 tx=0x7fe594009190 comp rx=0 tx=0).stop 2023-10-30T14:43:04.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.720+0000 7fe59f994640 1 -- 172.21.15.57:0/3623335219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5980cb7d0 msgr2=0x7fe5981f27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:04.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.720+0000 7fe59f994640 1 --2- 172.21.15.57:0/3623335219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5980cb7d0 0x7fe5981f27f0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fe58800c500 tx=0x7fe58800c9c0 comp rx=0 tx=0).stop 2023-10-30T14:43:04.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.720+0000 7fe59f994640 1 -- 172.21.15.57:0/3623335219 shutdown_connections 2023-10-30T14:43:04.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.720+0000 7fe59f994640 1 --2- 172.21.15.57:0/3623335219 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe57c0664a0 0x7fe57c068960 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:04.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.720+0000 7fe59f994640 1 --2- 172.21.15.57:0/3623335219 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe598150830 0x7fe5981f0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:04.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.720+0000 7fe59f994640 1 --2- 172.21.15.57:0/3623335219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5980cb7d0 0x7fe5981f27f0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:04.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.720+0000 7fe59f994640 1 --2- 172.21.15.57:0/3623335219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe598070520 0x7fe5981f22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:04.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.720+0000 7fe59f994640 1 -- 172.21.15.57:0/3623335219 >> 172.21.15.57:0/3623335219 conn(0x7fe5980cac70 msgr2=0x7fe5980c3770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:04.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.720+0000 7fe59f994640 1 -- 172.21.15.57:0/3623335219 shutdown_connections 2023-10-30T14:43:04.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:04.720+0000 7fe59f994640 1 -- 172.21.15.57:0/3623335219 wait complete. 2023-10-30T14:43:04.742 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=0, replacing gid=6414 2023-10-30T14:43:04.743 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.a 2023-10-30T14:43:07.744 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T14:43:07.744 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T14:43:07.745 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T14:43:07.749 INFO:tasks.ceph.mds.a:Started 2023-10-30T14:43:07.750 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:07.849 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-30T14:43:07.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.904+0000 7fafd769c640 1 Processor -- start 2023-10-30T14:43:07.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.904+0000 7fafd769c640 1 -- start start 2023-10-30T14:43:07.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.908+0000 7fafd769c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafd0150830 0x7fafd006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:07.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.908+0000 7fafd769c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fafd0070520 0x7fafd00709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:07.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.908+0000 7fafd769c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd00c2310 con 0x7fafd00cb7d0 2023-10-30T14:43:07.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.908+0000 7fafd769c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafd0156710 con 0x7fafd0070520 2023-10-30T14:43:07.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.908+0000 7fafd769c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd00c4360 con 0x7fafd0150830 2023-10-30T14:43:07.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.908+0000 7fafd4c37640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fafd00cb7d0 0x7fafd00c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:57070/0 (socket says 172.21.15.57:57070) 2023-10-30T14:43:07.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.908+0000 7fafd4c37640 1 -- 172.21.15.57:0/4226352026 learned_addr learned my addr 172.21.15.57:0/4226352026 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:43:07.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.908+0000 7fafd5c39640 1 --2- 172.21.15.57:0/4226352026 >> v2:172.21.15.88:3300/0 conn(0x7fafd0070520 0x7fafd00709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:07.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.908+0000 7fafd5438640 1 --2- 172.21.15.57:0/4226352026 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafd0150830 0x7fafd006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:07.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.908+0000 7fafc7fff640 1 -- 172.21.15.57:0/4226352026 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054326040 0 0) 0x7fafd00c2310 con 0x7fafd00cb7d0 2023-10-30T14:43:07.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.908+0000 7fafc7fff640 1 -- 172.21.15.57:0/4226352026 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafbc003530 con 0x7fafd00cb7d0 2023-10-30T14:43:07.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.908+0000 7fafc7fff640 1 -- 172.21.15.57:0/4226352026 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 977560425 0 0) 0x7fafbc003530 con 0x7fafd00cb7d0 2023-10-30T14:43:07.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.908+0000 7fafd5c39640 1 -- 172.21.15.57:0/4226352026 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafd0150830 msgr2=0x7fafd006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:07.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.908+0000 7fafd5c39640 1 --2- 172.21.15.57:0/4226352026 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafd0150830 0x7fafd006b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:07.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.908+0000 7fafd5c39640 1 -- 172.21.15.57:0/4226352026 >> v1:172.21.15.57:6789/0 conn(0x7fafd00cb7d0 legacy=0x7fafd00c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:07.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.908+0000 7fafd5c39640 1 -- 172.21.15.57:0/4226352026 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd01569b0 con 0x7fafd0070520 2023-10-30T14:43:07.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.908+0000 7fafd5438640 1 --2- 172.21.15.57:0/4226352026 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafd0150830 0x7fafd006b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:43:07.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.912+0000 7fafd5c39640 1 --2- 172.21.15.57:0/4226352026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafd0070520 0x7fafd00709b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fafc800bfa0 tx=0x7fafc8021250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3638fbe5555ffbc2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:07.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.912+0000 7fafc7fff640 1 -- 172.21.15.57:0/4226352026 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc0001680 con 0x7fafd0070520 2023-10-30T14:43:07.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.912+0000 7fafc7fff640 1 -- 172.21.15.57:0/4226352026 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc80ad050 con 0x7fafd0070520 2023-10-30T14:43:07.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.912+0000 7fafc7fff640 1 -- 172.21.15.57:0/4226352026 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc800b2d0 con 0x7fafd0070520 2023-10-30T14:43:07.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.912+0000 7fafd769c640 1 -- 172.21.15.57:0/4226352026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafd0070520 msgr2=0x7fafd00709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:07.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.912+0000 7fafd769c640 1 --2- 172.21.15.57:0/4226352026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafd0070520 0x7fafd00709b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fafc800bfa0 tx=0x7fafc8021250 comp rx=0 tx=0).stop 2023-10-30T14:43:07.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.912+0000 7fafd769c640 1 -- 172.21.15.57:0/4226352026 shutdown_connections 2023-10-30T14:43:07.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.912+0000 7fafd769c640 1 --2- 172.21.15.57:0/4226352026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafd0070520 0x7fafd00709b0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:07.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.912+0000 7fafd769c640 1 --2- 172.21.15.57:0/4226352026 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafd0150830 0x7fafd006b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:07.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.912+0000 7fafd769c640 1 -- 172.21.15.57:0/4226352026 >> 172.21.15.57:0/4226352026 conn(0x7fafd00cac70 msgr2=0x7fafd00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:07.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.912+0000 7fafd769c640 1 -- 172.21.15.57:0/4226352026 shutdown_connections 2023-10-30T14:43:07.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.912+0000 7fafd769c640 1 -- 172.21.15.57:0/4226352026 wait complete. 2023-10-30T14:43:07.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.912+0000 7fafd769c640 1 Processor -- start 2023-10-30T14:43:07.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.912+0000 7fafd769c640 1 -- start start 2023-10-30T14:43:07.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd769c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafd0070520 0x7fafd0141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:07.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd769c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafd00cb7d0 0x7fafd0141fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:07.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd769c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafd0150830 0x7fafd014ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:07.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd769c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd0143e60 con 0x7fafd00cb7d0 2023-10-30T14:43:07.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd769c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd01efe20 con 0x7fafd0070520 2023-10-30T14:43:07.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd769c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd014ab50 con 0x7fafd0150830 2023-10-30T14:43:07.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd4c37640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafd00cb7d0 0x7fafd0141fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:07.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd4c37640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafd00cb7d0 0x7fafd0141fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44494/0 (socket says 172.21.15.57:44494) 2023-10-30T14:43:07.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd4c37640 1 -- 172.21.15.57:0/2251734287 learned_addr learned my addr 172.21.15.57:0/2251734287 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:07.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd5c39640 1 --2- 172.21.15.57:0/2251734287 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafd0150830 0x7fafd014ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:07.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd5438640 1 --2- 172.21.15.57:0/2251734287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafd0070520 0x7fafd0141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:07.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd4c37640 1 -- 172.21.15.57:0/2251734287 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafd0150830 msgr2=0x7fafd014ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:07.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd4c37640 1 --2- 172.21.15.57:0/2251734287 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafd0150830 0x7fafd014ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:07.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd4c37640 1 -- 172.21.15.57:0/2251734287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafd0070520 msgr2=0x7fafd0141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:07.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd4c37640 1 --2- 172.21.15.57:0/2251734287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafd0070520 0x7fafd0141760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:07.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd4c37640 1 -- 172.21.15.57:0/2251734287 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafcc009420 con 0x7fafd00cb7d0 2023-10-30T14:43:07.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd5438640 1 --2- 172.21.15.57:0/2251734287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafd0070520 0x7fafd0141760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:43:07.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd4c37640 1 --2- 172.21.15.57:0/2251734287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafd00cb7d0 0x7fafd0141fc0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fafc000c750 tx=0x7fafc000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:07.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafc5ffb640 1 -- 172.21.15.57:0/2251734287 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc0005ae0 con 0x7fafd00cb7d0 2023-10-30T14:43:07.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafc5ffb640 1 -- 172.21.15.57:0/2251734287 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc0005c40 con 0x7fafd00cb7d0 2023-10-30T14:43:07.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd769c640 1 -- 172.21.15.57:0/2251734287 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc800bc80 con 0x7fafd00cb7d0 2023-10-30T14:43:07.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafc5ffb640 1 -- 172.21.15.57:0/2251734287 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc0007070 con 0x7fafd00cb7d0 2023-10-30T14:43:07.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd769c640 1 -- 172.21.15.57:0/2251734287 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd01ef060 con 0x7fafd00cb7d0 2023-10-30T14:43:07.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafd769c640 1 -- 172.21.15.57:0/2251734287 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd004e170 con 0x7fafd00cb7d0 2023-10-30T14:43:07.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.916+0000 7fafc5ffb640 1 -- 172.21.15.57:0/2251734287 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fafc00071d0 con 0x7fafd00cb7d0 2023-10-30T14:43:07.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.920+0000 7fafc5ffb640 1 --2- 172.21.15.57:0/2251734287 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fafb40664a0 0x7fafb4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:07.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.920+0000 7fafc5ffb640 1 -- 172.21.15.57:0/2251734287 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7fafc0084250 con 0x7fafd00cb7d0 2023-10-30T14:43:07.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.920+0000 7fafd5438640 1 --2- 172.21.15.57:0/2251734287 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fafb40664a0 0x7fafb4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:07.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.920+0000 7fafd5438640 1 --2- 172.21.15.57:0/2251734287 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fafb40664a0 0x7fafb4068960 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009ca0 tx=0x7fafcc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:07.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:07.924+0000 7fafc5ffb640 1 -- 172.21.15.57:0/2251734287 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fafc00510b0 con 0x7fafd00cb7d0 2023-10-30T14:43:08.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:08.116+0000 7fafd769c640 1 -- 172.21.15.57:0/2251734287 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafd00c6980 con 0x7fafd00cb7d0 2023-10-30T14:43:08.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:08.116+0000 7fafc5ffb640 1 -- 172.21.15.57:0/2251734287 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5741 (secure 0 0 0) 0x7fafc00546d0 con 0x7fafd00cb7d0 2023-10-30T14:43:08.121 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:08.121 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:04.464878+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:08.122 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 155 2023-10-30T14:43:08.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:08.120+0000 7fafd769c640 1 -- 172.21.15.57:0/2251734287 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fafb40664a0 msgr2=0x7fafb4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:08.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:08.120+0000 7fafd769c640 1 --2- 172.21.15.57:0/2251734287 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fafb40664a0 0x7fafb4068960 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009ca0 tx=0x7fafcc008040 comp rx=0 tx=0).stop 2023-10-30T14:43:08.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:08.120+0000 7fafd769c640 1 -- 172.21.15.57:0/2251734287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafd00cb7d0 msgr2=0x7fafd0141fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:08.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:08.120+0000 7fafd769c640 1 --2- 172.21.15.57:0/2251734287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafd00cb7d0 0x7fafd0141fc0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fafc000c750 tx=0x7fafc000cc10 comp rx=0 tx=0).stop 2023-10-30T14:43:08.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:08.120+0000 7fafd769c640 1 -- 172.21.15.57:0/2251734287 shutdown_connections 2023-10-30T14:43:08.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:08.120+0000 7fafd769c640 1 --2- 172.21.15.57:0/2251734287 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fafb40664a0 0x7fafb4068960 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:08.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:08.120+0000 7fafd769c640 1 --2- 172.21.15.57:0/2251734287 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafd0150830 0x7fafd014ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:08.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:08.120+0000 7fafd769c640 1 --2- 172.21.15.57:0/2251734287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafd00cb7d0 0x7fafd0141fc0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:08.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:08.120+0000 7fafd769c640 1 --2- 172.21.15.57:0/2251734287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafd0070520 0x7fafd0141760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:08.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:08.120+0000 7fafd769c640 1 -- 172.21.15.57:0/2251734287 >> 172.21.15.57:0/2251734287 conn(0x7fafd00cac70 msgr2=0x7fafd00c6210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:08.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:08.120+0000 7fafd769c640 1 -- 172.21.15.57:0/2251734287 shutdown_connections 2023-10-30T14:43:08.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:08.120+0000 7fafd769c640 1 -- 172.21.15.57:0/2251734287 wait complete. 2023-10-30T14:43:08.142 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-30T14:43:08.737 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:43:08.742 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:43:10.144 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:10.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.288+0000 7fc406da0640 1 Processor -- start 2023-10-30T14:43:10.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.288+0000 7fc406da0640 1 -- start start 2023-10-30T14:43:10.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.292+0000 7fc406da0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc400157c80 0x7fc40015e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:10.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.292+0000 7fc406da0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4000c5460 0x7fc40006d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:10.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.292+0000 7fc406da0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40015eb00 con 0x7fc40006dff0 2023-10-30T14:43:10.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.292+0000 7fc406da0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4000c67d0 con 0x7fc400157c80 2023-10-30T14:43:10.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.292+0000 7fc406da0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4000c58e0 con 0x7fc4000c5460 2023-10-30T14:43:10.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.292+0000 7fc40559d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4000c5460 0x7fc40006d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:10.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.292+0000 7fc405d9e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc400157c80 0x7fc40015e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:10.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.292+0000 7fc40559d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4000c5460 0x7fc40006d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:33632/0 (socket says 172.21.15.57:33632) 2023-10-30T14:43:10.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.292+0000 7fc405d9e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc400157c80 0x7fc40015e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:33424/0 (socket says 172.21.15.57:33424) 2023-10-30T14:43:10.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.292+0000 7fc40559d640 1 -- 172.21.15.57:0/3248241808 learned_addr learned my addr 172.21.15.57:0/3248241808 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:10.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.292+0000 7fc40559d640 1 -- 172.21.15.57:0/3248241808 >> v2:172.21.15.88:3300/0 conn(0x7fc400157c80 msgr2=0x7fc40015e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:10.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.292+0000 7fc40559d640 1 --2- 172.21.15.57:0/3248241808 >> v2:172.21.15.88:3300/0 conn(0x7fc400157c80 0x7fc40015e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.292+0000 7fc40559d640 1 -- 172.21.15.57:0/3248241808 >> v1:172.21.15.88:6789/0 conn(0x7fc40006dff0 legacy=0x7fc40015f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:10.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.292+0000 7fc40559d640 1 -- 172.21.15.57:0/3248241808 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4000c5bc0 con 0x7fc4000c5460 2023-10-30T14:43:10.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.296+0000 7fc40559d640 1 --2- 172.21.15.57:0/3248241808 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4000c5460 0x7fc40006d4f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0009870 tx=0x7fc3f001c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1c3e216bf54353f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:10.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.296+0000 7fc404d9c640 1 -- 172.21.15.57:0/3248241808 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f00aa670 con 0x7fc4000c5460 2023-10-30T14:43:10.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.296+0000 7fc404d9c640 1 -- 172.21.15.57:0/3248241808 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f00aa7d0 con 0x7fc4000c5460 2023-10-30T14:43:10.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.296+0000 7fc404d9c640 1 -- 172.21.15.57:0/3248241808 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f00aab60 con 0x7fc4000c5460 2023-10-30T14:43:10.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.296+0000 7fc406da0640 1 -- 172.21.15.57:0/3248241808 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4000c5460 msgr2=0x7fc40006d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:10.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.296+0000 7fc406da0640 1 --2- 172.21.15.57:0/3248241808 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4000c5460 0x7fc40006d4f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0009870 tx=0x7fc3f001c510 comp rx=0 tx=0).stop 2023-10-30T14:43:10.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.296+0000 7fc406da0640 1 -- 172.21.15.57:0/3248241808 shutdown_connections 2023-10-30T14:43:10.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.296+0000 7fc406da0640 1 --2- 172.21.15.57:0/3248241808 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4000c5460 0x7fc40006d4f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.296+0000 7fc406da0640 1 --2- 172.21.15.57:0/3248241808 >> v2:172.21.15.88:3300/0 conn(0x7fc400157c80 0x7fc40015e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.296+0000 7fc406da0640 1 -- 172.21.15.57:0/3248241808 >> 172.21.15.57:0/3248241808 conn(0x7fc4000caae0 msgr2=0x7fc4000caec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:10.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.296+0000 7fc406da0640 1 -- 172.21.15.57:0/3248241808 shutdown_connections 2023-10-30T14:43:10.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.296+0000 7fc406da0640 1 -- 172.21.15.57:0/3248241808 wait complete. 2023-10-30T14:43:10.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.296+0000 7fc406da0640 1 Processor -- start 2023-10-30T14:43:10.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.296+0000 7fc406da0640 1 -- start start 2023-10-30T14:43:10.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc406da0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc40006dff0 0x7fc4000c2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:10.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc406da0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4000c5460 0x7fc4000c3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:10.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc406da0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc400157c80 0x7fc400145580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:10.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc406da0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc400147140 con 0x7fc400157c80 2023-10-30T14:43:10.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc406da0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4001477b0 con 0x7fc40006dff0 2023-10-30T14:43:10.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc406da0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4000c1de0 con 0x7fc4000c5460 2023-10-30T14:43:10.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc40659f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc400157c80 0x7fc400145580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:10.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc40659f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc400157c80 0x7fc400145580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44504/0 (socket says 172.21.15.57:44504) 2023-10-30T14:43:10.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc40659f640 1 -- 172.21.15.57:0/2888717777 learned_addr learned my addr 172.21.15.57:0/2888717777 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:10.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc40559d640 1 --2- 172.21.15.57:0/2888717777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4000c5460 0x7fc4000c3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:10.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc405d9e640 1 --2- 172.21.15.57:0/2888717777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc40006dff0 0x7fc4000c2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:10.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc40659f640 1 -- 172.21.15.57:0/2888717777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4000c5460 msgr2=0x7fc4000c3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:10.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc40659f640 1 --2- 172.21.15.57:0/2888717777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4000c5460 0x7fc4000c3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc40659f640 1 -- 172.21.15.57:0/2888717777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc40006dff0 msgr2=0x7fc4000c2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:10.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc40659f640 1 --2- 172.21.15.57:0/2888717777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc40006dff0 0x7fc4000c2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc40659f640 1 -- 172.21.15.57:0/2888717777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f00094d0 con 0x7fc400157c80 2023-10-30T14:43:10.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc40659f640 1 --2- 172.21.15.57:0/2888717777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc400157c80 0x7fc400145580 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fc3f800c7d0 tx=0x7fc3f800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:10.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc3f67fc640 1 -- 172.21.15.57:0/2888717777 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8003b80 con 0x7fc400157c80 2023-10-30T14:43:10.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc3f67fc640 1 -- 172.21.15.57:0/2888717777 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f8003ce0 con 0x7fc400157c80 2023-10-30T14:43:10.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc3f67fc640 1 -- 172.21.15.57:0/2888717777 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f800aa30 con 0x7fc400157c80 2023-10-30T14:43:10.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc406da0640 1 -- 172.21.15.57:0/2888717777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc40014be50 con 0x7fc400157c80 2023-10-30T14:43:10.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc406da0640 1 -- 172.21.15.57:0/2888717777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc40015c7b0 con 0x7fc400157c80 2023-10-30T14:43:10.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc406da0640 1 -- 172.21.15.57:0/2888717777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc40004e170 con 0x7fc400157c80 2023-10-30T14:43:10.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.300+0000 7fc3f67fc640 1 -- 172.21.15.57:0/2888717777 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc3f800ab90 con 0x7fc400157c80 2023-10-30T14:43:10.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.304+0000 7fc3f67fc640 1 --2- 172.21.15.57:0/2888717777 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc3e40664a0 0x7fc3e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:10.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.304+0000 7fc3f67fc640 1 -- 172.21.15.57:0/2888717777 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7fc3f8084100 con 0x7fc400157c80 2023-10-30T14:43:10.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.304+0000 7fc405d9e640 1 --2- 172.21.15.57:0/2888717777 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc3e40664a0 0x7fc3e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:10.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.304+0000 7fc405d9e640 1 --2- 172.21.15.57:0/2888717777 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc3e40664a0 0x7fc3e4068960 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc009ac0 tx=0x7fc3fc009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:10.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.308+0000 7fc3f67fc640 1 -- 172.21.15.57:0/2888717777 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc3f80510a0 con 0x7fc400157c80 2023-10-30T14:43:10.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.500+0000 7fc406da0640 1 -- 172.21.15.57:0/2888717777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc40006e4b0 con 0x7fc400157c80 2023-10-30T14:43:10.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.500+0000 7fc3f67fc640 1 -- 172.21.15.57:0/2888717777 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+6530 (secure 0 0 0) 0x7fc3f80546c0 con 0x7fc400157c80 2023-10-30T14:43:10.505 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:10.506 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":156,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:04.464878+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:10.506 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 156 2023-10-30T14:43:10.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.504+0000 7fc406da0640 1 -- 172.21.15.57:0/2888717777 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc3e40664a0 msgr2=0x7fc3e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:10.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.504+0000 7fc406da0640 1 --2- 172.21.15.57:0/2888717777 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc3e40664a0 0x7fc3e4068960 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc009ac0 tx=0x7fc3fc009190 comp rx=0 tx=0).stop 2023-10-30T14:43:10.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.504+0000 7fc406da0640 1 -- 172.21.15.57:0/2888717777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc400157c80 msgr2=0x7fc400145580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:10.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.504+0000 7fc406da0640 1 --2- 172.21.15.57:0/2888717777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc400157c80 0x7fc400145580 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fc3f800c7d0 tx=0x7fc3f800cc90 comp rx=0 tx=0).stop 2023-10-30T14:43:10.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.504+0000 7fc406da0640 1 -- 172.21.15.57:0/2888717777 shutdown_connections 2023-10-30T14:43:10.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.504+0000 7fc406da0640 1 --2- 172.21.15.57:0/2888717777 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc3e40664a0 0x7fc3e4068960 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.504+0000 7fc406da0640 1 --2- 172.21.15.57:0/2888717777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc400157c80 0x7fc400145580 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.504+0000 7fc406da0640 1 --2- 172.21.15.57:0/2888717777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4000c5460 0x7fc4000c3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.504+0000 7fc406da0640 1 --2- 172.21.15.57:0/2888717777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc40006dff0 0x7fc4000c2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.504+0000 7fc406da0640 1 -- 172.21.15.57:0/2888717777 >> 172.21.15.57:0/2888717777 conn(0x7fc4000caae0 msgr2=0x7fc40015f310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:10.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.504+0000 7fc406da0640 1 -- 172.21.15.57:0/2888717777 shutdown_connections 2023-10-30T14:43:10.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.504+0000 7fc406da0640 1 -- 172.21.15.57:0/2888717777 wait complete. 2023-10-30T14:43:10.528 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T14:43:10.528 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:10.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.672+0000 7f5446bc5640 1 Processor -- start 2023-10-30T14:43:10.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.672+0000 7f5446bc5640 1 -- start start 2023-10-30T14:43:10.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.672+0000 7f5446bc5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5440153aa0 0x7f544015d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:10.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.672+0000 7f5446bc5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f54400701e0 0x7f54400ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:10.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.672+0000 7f5446bc5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54400cd420 0x7f544006b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:10.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.672+0000 7f5446bc5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5440070a50 con 0x7f5440153aa0 2023-10-30T14:43:10.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.672+0000 7f5446bc5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f544015dd80 con 0x7f54400701e0 2023-10-30T14:43:10.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.672+0000 7f5446bc5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54400c7b00 con 0x7f54400cd420 2023-10-30T14:43:10.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.672+0000 7f5444961640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5440153aa0 0x7f544015d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:10.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.672+0000 7f5444961640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5440153aa0 0x7f544015d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44506/0 (socket says 172.21.15.57:44506) 2023-10-30T14:43:10.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.672+0000 7f5444961640 1 -- 172.21.15.57:0/987621953 learned_addr learned my addr 172.21.15.57:0/987621953 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:10.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.676+0000 7f5445162640 1 --2- 172.21.15.57:0/987621953 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54400cd420 0x7f544006b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:10.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.676+0000 7f543ffff640 1 --2- 172.21.15.57:0/987621953 >> v2:172.21.15.88:3300/0 conn(0x7f54400701e0 0x7f54400ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:10.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.676+0000 7f5444961640 1 -- 172.21.15.57:0/987621953 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54400cd420 msgr2=0x7f544006b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:10.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.676+0000 7f5444961640 1 --2- 172.21.15.57:0/987621953 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54400cd420 0x7f544006b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.676+0000 7f5444961640 1 -- 172.21.15.57:0/987621953 >> v2:172.21.15.88:3300/0 conn(0x7f54400701e0 msgr2=0x7f54400ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:10.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.676+0000 7f5444961640 1 --2- 172.21.15.57:0/987621953 >> v2:172.21.15.88:3300/0 conn(0x7f54400701e0 0x7f54400ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.676+0000 7f5444961640 1 -- 172.21.15.57:0/987621953 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54400c2050 con 0x7f5440153aa0 2023-10-30T14:43:10.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.676+0000 7f5444961640 1 --2- 172.21.15.57:0/987621953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5440153aa0 0x7f544015d840 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f54400caa90 tx=0x7f543401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0b5e800ca68812 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:10.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.676+0000 7f543f7fe640 1 -- 172.21.15.57:0/987621953 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54340a89b0 con 0x7f5440153aa0 2023-10-30T14:43:10.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.676+0000 7f543f7fe640 1 -- 172.21.15.57:0/987621953 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54340a8b10 con 0x7f5440153aa0 2023-10-30T14:43:10.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.676+0000 7f543f7fe640 1 -- 172.21.15.57:0/987621953 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54340a8ea0 con 0x7f5440153aa0 2023-10-30T14:43:10.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5446bc5640 1 -- 172.21.15.57:0/987621953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5440153aa0 msgr2=0x7f544015d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:10.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5446bc5640 1 --2- 172.21.15.57:0/987621953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5440153aa0 0x7f544015d840 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f54400caa90 tx=0x7f543401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5446bc5640 1 -- 172.21.15.57:0/987621953 shutdown_connections 2023-10-30T14:43:10.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5446bc5640 1 --2- 172.21.15.57:0/987621953 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54400cd420 0x7f544006b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5446bc5640 1 --2- 172.21.15.57:0/987621953 >> v2:172.21.15.88:3300/0 conn(0x7f54400701e0 0x7f54400ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5446bc5640 1 --2- 172.21.15.57:0/987621953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5440153aa0 0x7f544015d840 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5446bc5640 1 -- 172.21.15.57:0/987621953 >> 172.21.15.57:0/987621953 conn(0x7f5440071fb0 msgr2=0x7f54400723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:10.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5446bc5640 1 -- 172.21.15.57:0/987621953 shutdown_connections 2023-10-30T14:43:10.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5446bc5640 1 -- 172.21.15.57:0/987621953 wait complete. 2023-10-30T14:43:10.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5446bc5640 1 Processor -- start 2023-10-30T14:43:10.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5446bc5640 1 -- start start 2023-10-30T14:43:10.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5446bc5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54400701e0 0x7f54400c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:10.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5446bc5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54400cd420 0x7f54401f2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:10.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5446bc5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5440153aa0 0x7f544014ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:10.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5446bc5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54400c48b0 con 0x7f54400701e0 2023-10-30T14:43:10.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5446bc5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544014c850 con 0x7f54400cd420 2023-10-30T14:43:10.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5446bc5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54400cd0d0 con 0x7f5440153aa0 2023-10-30T14:43:10.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5444961640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54400701e0 0x7f54400c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:10.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5444961640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54400701e0 0x7f54400c42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44516/0 (socket says 172.21.15.57:44516) 2023-10-30T14:43:10.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5444961640 1 -- 172.21.15.57:0/1485617204 learned_addr learned my addr 172.21.15.57:0/1485617204 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:10.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f543ffff640 1 --2- 172.21.15.57:0/1485617204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54400cd420 0x7f54401f2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:10.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5445162640 1 --2- 172.21.15.57:0/1485617204 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5440153aa0 0x7f544014ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:10.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5444961640 1 -- 172.21.15.57:0/1485617204 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5440153aa0 msgr2=0x7f544014ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:10.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5444961640 1 --2- 172.21.15.57:0/1485617204 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5440153aa0 0x7f544014ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5444961640 1 -- 172.21.15.57:0/1485617204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54400cd420 msgr2=0x7f54401f2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:10.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.680+0000 7f5444961640 1 --2- 172.21.15.57:0/1485617204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54400cd420 0x7f54401f2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.684+0000 7f5444961640 1 -- 172.21.15.57:0/1485617204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54340094d0 con 0x7f54400701e0 2023-10-30T14:43:10.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.684+0000 7f543ffff640 1 --2- 172.21.15.57:0/1485617204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54400cd420 0x7f54401f2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:43:10.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.684+0000 7f5445162640 1 --2- 172.21.15.57:0/1485617204 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5440153aa0 0x7f544014ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:43:10.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.684+0000 7f5444961640 1 --2- 172.21.15.57:0/1485617204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54400701e0 0x7f54400c42d0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f54340b3040 tx=0x7f543401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:10.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.684+0000 7f543d7fa640 1 -- 172.21.15.57:0/1485617204 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54340ba600 con 0x7f54400701e0 2023-10-30T14:43:10.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.684+0000 7f543d7fa640 1 -- 172.21.15.57:0/1485617204 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54340ba760 con 0x7f54400701e0 2023-10-30T14:43:10.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.684+0000 7f543d7fa640 1 -- 172.21.15.57:0/1485617204 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54340bab10 con 0x7f54400701e0 2023-10-30T14:43:10.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.684+0000 7f5446bc5640 1 -- 172.21.15.57:0/1485617204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f544014ca40 con 0x7f54400701e0 2023-10-30T14:43:10.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.684+0000 7f5446bc5640 1 -- 172.21.15.57:0/1485617204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54401f2160 con 0x7f54400701e0 2023-10-30T14:43:10.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.684+0000 7f5446bc5640 1 -- 172.21.15.57:0/1485617204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f544004e170 con 0x7f54400701e0 2023-10-30T14:43:10.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.684+0000 7f543d7fa640 1 -- 172.21.15.57:0/1485617204 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f54340bac70 con 0x7f54400701e0 2023-10-30T14:43:10.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.684+0000 7f543d7fa640 1 --2- 172.21.15.57:0/1485617204 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f54240664a0 0x7f5424068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:10.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.688+0000 7f543d7fa640 1 -- 172.21.15.57:0/1485617204 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f54341229f0 con 0x7f54400701e0 2023-10-30T14:43:10.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.688+0000 7f543ffff640 1 --2- 172.21.15.57:0/1485617204 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f54240664a0 0x7f5424068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:10.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.688+0000 7f543ffff640 1 --2- 172.21.15.57:0/1485617204 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f54240664a0 0x7f5424068960 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f542c004660 tx=0x7f542c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:10.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.692+0000 7f543d7fa640 1 -- 172.21.15.57:0/1485617204 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f54340ef860 con 0x7f54400701e0 2023-10-30T14:43:10.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.880+0000 7f5446bc5640 1 -- 172.21.15.57:0/1485617204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5440150350 con 0x7f54400701e0 2023-10-30T14:43:10.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.880+0000 7f543d7fa640 1 -- 172.21.15.57:0/1485617204 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+6530 (secure 0 0 0) 0x7f54340b5020 con 0x7f54400701e0 2023-10-30T14:43:10.886 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:10.886 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":156,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:04.464878+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:active","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:10.887 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 156 2023-10-30T14:43:10.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.884+0000 7f5446bc5640 1 -- 172.21.15.57:0/1485617204 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f54240664a0 msgr2=0x7f5424068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:10.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.884+0000 7f5446bc5640 1 --2- 172.21.15.57:0/1485617204 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f54240664a0 0x7f5424068960 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f542c004660 tx=0x7f542c008040 comp rx=0 tx=0).stop 2023-10-30T14:43:10.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.884+0000 7f5446bc5640 1 -- 172.21.15.57:0/1485617204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54400701e0 msgr2=0x7f54400c42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:10.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.884+0000 7f5446bc5640 1 --2- 172.21.15.57:0/1485617204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54400701e0 0x7f54400c42d0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f54340b3040 tx=0x7f543401ba90 comp rx=0 tx=0).stop 2023-10-30T14:43:10.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.884+0000 7f5446bc5640 1 -- 172.21.15.57:0/1485617204 shutdown_connections 2023-10-30T14:43:10.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.884+0000 7f5446bc5640 1 --2- 172.21.15.57:0/1485617204 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f54240664a0 0x7f5424068960 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.884+0000 7f5446bc5640 1 --2- 172.21.15.57:0/1485617204 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5440153aa0 0x7f544014ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.884+0000 7f5446bc5640 1 --2- 172.21.15.57:0/1485617204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54400cd420 0x7f54401f2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.884+0000 7f5446bc5640 1 --2- 172.21.15.57:0/1485617204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54400701e0 0x7f54400c42d0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:10.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.884+0000 7f5446bc5640 1 -- 172.21.15.57:0/1485617204 >> 172.21.15.57:0/1485617204 conn(0x7f5440071fb0 msgr2=0x7f5440072390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:10.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.884+0000 7f5446bc5640 1 -- 172.21.15.57:0/1485617204 shutdown_connections 2023-10-30T14:43:10.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:10.884+0000 7f5446bc5640 1 -- 172.21.15.57:0/1485617204 wait complete. 2023-10-30T14:43:10.907 INFO:tasks.mds_thrash.fs.[cephfs]:thrashing max_mds: 3 -> 2 2023-10-30T14:43:10.908 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2023-10-30T14:43:11.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.092+0000 7f387dc88640 1 Processor -- start 2023-10-30T14:43:11.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.092+0000 7f387dc88640 1 -- start start 2023-10-30T14:43:11.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.092+0000 7f387dc88640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f38780718d0 0x7f38780c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:11.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.092+0000 7f387dc88640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3878071cb0 0x7f387815a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:11.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.092+0000 7f387dc88640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3878153300 0x7f3878153750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:11.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.092+0000 7f387dc88640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38780cca10 con 0x7f38780718d0 2023-10-30T14:43:11.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.092+0000 7f387dc88640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f387815a770 con 0x7f3878071cb0 2023-10-30T14:43:11.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.092+0000 7f387dc88640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3878153c90 con 0x7f3878153300 2023-10-30T14:43:11.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.092+0000 7f387cc86640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f38780718d0 0x7f38780c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:11.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.092+0000 7f387cc86640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f38780718d0 0x7f38780c9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44526/0 (socket says 172.21.15.57:44526) 2023-10-30T14:43:11.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.092+0000 7f387cc86640 1 -- 172.21.15.57:0/1268980515 learned_addr learned my addr 172.21.15.57:0/1268980515 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:11.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.092+0000 7f387d487640 1 --2- 172.21.15.57:0/1268980515 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3878153300 0x7f3878153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:11.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.092+0000 7f3877fff640 1 --2- 172.21.15.57:0/1268980515 >> v2:172.21.15.88:3300/0 conn(0x7f3878071cb0 0x7f387815a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:11.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.092+0000 7f387cc86640 1 -- 172.21.15.57:0/1268980515 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3878153300 msgr2=0x7f3878153750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:11.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.092+0000 7f387cc86640 1 --2- 172.21.15.57:0/1268980515 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3878153300 0x7f3878153750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:11.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.092+0000 7f387cc86640 1 -- 172.21.15.57:0/1268980515 >> v2:172.21.15.88:3300/0 conn(0x7f3878071cb0 msgr2=0x7f387815a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:11.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.092+0000 7f387cc86640 1 --2- 172.21.15.57:0/1268980515 >> v2:172.21.15.88:3300/0 conn(0x7f3878071cb0 0x7f387815a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:11.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.092+0000 7f387cc86640 1 -- 172.21.15.57:0/1268980515 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387815d0a0 con 0x7f38780718d0 2023-10-30T14:43:11.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.096+0000 7f387cc86640 1 --2- 172.21.15.57:0/1268980515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38780718d0 0x7f38780c9a10 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f3878072470 tx=0x7f386c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ed2d31e8b052b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:11.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.096+0000 7f38777fe640 1 -- 172.21.15.57:0/1268980515 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386c0a89b0 con 0x7f38780718d0 2023-10-30T14:43:11.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.096+0000 7f38777fe640 1 -- 172.21.15.57:0/1268980515 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f386c0a8b10 con 0x7f38780718d0 2023-10-30T14:43:11.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.096+0000 7f38777fe640 1 -- 172.21.15.57:0/1268980515 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386c0a8ea0 con 0x7f38780718d0 2023-10-30T14:43:11.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.096+0000 7f387dc88640 1 -- 172.21.15.57:0/1268980515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38780718d0 msgr2=0x7f38780c9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:11.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.096+0000 7f387dc88640 1 --2- 172.21.15.57:0/1268980515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38780718d0 0x7f38780c9a10 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f3878072470 tx=0x7f386c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:43:11.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.096+0000 7f387dc88640 1 -- 172.21.15.57:0/1268980515 shutdown_connections 2023-10-30T14:43:11.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.096+0000 7f387dc88640 1 --2- 172.21.15.57:0/1268980515 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3878153300 0x7f3878153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:11.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.096+0000 7f387dc88640 1 --2- 172.21.15.57:0/1268980515 >> v2:172.21.15.88:3300/0 conn(0x7f3878071cb0 0x7f387815a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:11.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.096+0000 7f387dc88640 1 --2- 172.21.15.57:0/1268980515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38780718d0 0x7f38780c9a10 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:11.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.096+0000 7f387dc88640 1 -- 172.21.15.57:0/1268980515 >> 172.21.15.57:0/1268980515 conn(0x7f387806ab00 msgr2=0x7f3878065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:11.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.096+0000 7f387dc88640 1 -- 172.21.15.57:0/1268980515 shutdown_connections 2023-10-30T14:43:11.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.096+0000 7f387dc88640 1 -- 172.21.15.57:0/1268980515 wait complete. 2023-10-30T14:43:11.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387dc88640 1 Processor -- start 2023-10-30T14:43:11.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387dc88640 1 -- start start 2023-10-30T14:43:11.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387dc88640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38780718d0 0x7f38781efc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:11.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387dc88640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3878071cb0 0x7f38781f0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:11.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387dc88640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3878153300 0x7f387814ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:11.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387dc88640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38781596b0 con 0x7f3878153300 2023-10-30T14:43:11.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387dc88640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3878155ee0 con 0x7f3878071cb0 2023-10-30T14:43:11.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387dc88640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38781561c0 con 0x7f38780718d0 2023-10-30T14:43:11.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387d487640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3878153300 0x7f387814ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:11.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387d487640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3878153300 0x7f387814ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44542/0 (socket says 172.21.15.57:44542) 2023-10-30T14:43:11.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387d487640 1 -- 172.21.15.57:0/729588197 learned_addr learned my addr 172.21.15.57:0/729588197 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:11.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387cc86640 1 --2- 172.21.15.57:0/729588197 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38780718d0 0x7f38781efc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:11.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f3877fff640 1 --2- 172.21.15.57:0/729588197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3878071cb0 0x7f38781f0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:11.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387d487640 1 -- 172.21.15.57:0/729588197 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38780718d0 msgr2=0x7f38781efc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:11.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387d487640 1 --2- 172.21.15.57:0/729588197 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38780718d0 0x7f38781efc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:11.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387d487640 1 -- 172.21.15.57:0/729588197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3878071cb0 msgr2=0x7f38781f0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:11.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387d487640 1 --2- 172.21.15.57:0/729588197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3878071cb0 0x7f38781f0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:11.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387d487640 1 -- 172.21.15.57:0/729588197 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f386c0094d0 con 0x7f3878153300 2023-10-30T14:43:11.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387d487640 1 --2- 172.21.15.57:0/729588197 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3878153300 0x7f387814ca00 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f3870007e60 tx=0x7f387000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:11.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f38757fa640 1 -- 172.21.15.57:0/729588197 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870014420 con 0x7f3878153300 2023-10-30T14:43:11.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f38757fa640 1 -- 172.21.15.57:0/729588197 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38700145a0 con 0x7f3878153300 2023-10-30T14:43:11.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387dc88640 1 -- 172.21.15.57:0/729588197 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38781564a0 con 0x7f3878153300 2023-10-30T14:43:11.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f38757fa640 1 -- 172.21.15.57:0/729588197 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870014950 con 0x7f3878153300 2023-10-30T14:43:11.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387dc88640 1 -- 172.21.15.57:0/729588197 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3878156a60 con 0x7f3878153300 2023-10-30T14:43:11.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.100+0000 7f387dc88640 1 -- 172.21.15.57:0/729588197 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f387804e170 con 0x7f3878153300 2023-10-30T14:43:11.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.104+0000 7f38757fa640 1 -- 172.21.15.57:0/729588197 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3870014ab0 con 0x7f3878153300 2023-10-30T14:43:11.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.104+0000 7f38757fa640 1 --2- 172.21.15.57:0/729588197 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f385c0664d0 0x7f385c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:11.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.104+0000 7f38757fa640 1 -- 172.21.15.57:0/729588197 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f3870084b10 con 0x7f3878153300 2023-10-30T14:43:11.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.104+0000 7f387cc86640 1 --2- 172.21.15.57:0/729588197 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f385c0664d0 0x7f385c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:11.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.104+0000 7f387cc86640 1 --2- 172.21.15.57:0/729588197 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f385c0664d0 0x7f385c068990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f386c0b2000 tx=0x7f386c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:11.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.108+0000 7f38757fa640 1 -- 172.21.15.57:0/729588197 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3870051ab0 con 0x7f3878153300 2023-10-30T14:43:11.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:11.300+0000 7f387dc88640 1 -- 172.21.15.57:0/729588197 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f38780c9f80 con 0x7f3878153300 2023-10-30T14:43:12.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.228+0000 7f38757fa640 1 -- 172.21.15.57:0/729588197 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v157) v1 ==== 105+0+0 (secure 0 0 0) 0x7f38700550d0 con 0x7f3878153300 2023-10-30T14:43:12.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.232+0000 7f387dc88640 1 -- 172.21.15.57:0/729588197 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f385c0664d0 msgr2=0x7f385c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:12.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.232+0000 7f387dc88640 1 --2- 172.21.15.57:0/729588197 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f385c0664d0 0x7f385c068990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f386c0b2000 tx=0x7f386c009040 comp rx=0 tx=0).stop 2023-10-30T14:43:12.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.232+0000 7f387dc88640 1 -- 172.21.15.57:0/729588197 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3878153300 msgr2=0x7f387814ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:12.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.232+0000 7f387dc88640 1 --2- 172.21.15.57:0/729588197 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3878153300 0x7f387814ca00 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f3870007e60 tx=0x7f387000e4b0 comp rx=0 tx=0).stop 2023-10-30T14:43:12.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.232+0000 7f387dc88640 1 -- 172.21.15.57:0/729588197 shutdown_connections 2023-10-30T14:43:12.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.232+0000 7f387dc88640 1 --2- 172.21.15.57:0/729588197 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f385c0664d0 0x7f385c068990 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:12.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.232+0000 7f387dc88640 1 --2- 172.21.15.57:0/729588197 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3878153300 0x7f387814ca00 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:12.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.232+0000 7f387dc88640 1 --2- 172.21.15.57:0/729588197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3878071cb0 0x7f38781f0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:12.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.232+0000 7f387dc88640 1 --2- 172.21.15.57:0/729588197 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38780718d0 0x7f38781efc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:12.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.232+0000 7f387dc88640 1 -- 172.21.15.57:0/729588197 >> 172.21.15.57:0/729588197 conn(0x7f387806ab00 msgr2=0x7f38780c4800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:12.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.232+0000 7f387dc88640 1 -- 172.21.15.57:0/729588197 shutdown_connections 2023-10-30T14:43:12.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.232+0000 7f387dc88640 1 -- 172.21.15.57:0/729588197 wait complete. 2023-10-30T14:43:12.263 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:43:12.263 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:12.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.404+0000 7f62f429e640 1 Processor -- start 2023-10-30T14:43:12.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.404+0000 7f62f429e640 1 -- start start 2023-10-30T14:43:12.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.404+0000 7f62f429e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f62ec151c20 0x7f62ec0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:12.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.404+0000 7f62f429e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62ec0c5030 con 0x7f62ec151840 2023-10-30T14:43:12.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.404+0000 7f62f429e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62ec157830 con 0x7f62ec0cb8b0 2023-10-30T14:43:12.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.404+0000 7f62f429e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62ec157b10 con 0x7f62ec151c20 2023-10-30T14:43:12.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.404+0000 7f62f203a640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f62ec151840 0x7f62ec072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:57072/0 (socket says 172.21.15.57:57072) 2023-10-30T14:43:12.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.404+0000 7f62f203a640 1 -- 172.21.15.57:0/4198467974 learned_addr learned my addr 172.21.15.57:0/4198467974 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:43:12.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.404+0000 7f62f1839640 1 --2- 172.21.15.57:0/4198467974 >> v2:172.21.15.88:3300/0 conn(0x7f62ec151c20 0x7f62ec0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:12.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.404+0000 7f62f1038640 1 -- 172.21.15.57:0/4198467974 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643209324 0 0) 0x7f62ec0c5030 con 0x7f62ec151840 2023-10-30T14:43:12.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.404+0000 7f62f1038640 1 -- 172.21.15.57:0/4198467974 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62d8003530 con 0x7f62ec151840 2023-10-30T14:43:12.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1038640 1 -- 172.21.15.57:0/4198467974 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1689037002 0 0) 0x7f62d8003530 con 0x7f62ec151840 2023-10-30T14:43:12.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1038640 1 -- 172.21.15.57:0/4198467974 >> v2:172.21.15.88:3300/0 conn(0x7f62ec151c20 msgr2=0x7f62ec0c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:12.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1038640 1 --2- 172.21.15.57:0/4198467974 >> v2:172.21.15.88:3300/0 conn(0x7f62ec151c20 0x7f62ec0c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:12.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1038640 1 -- 172.21.15.57:0/4198467974 >> v1:172.21.15.88:6789/0 conn(0x7f62ec0cb8b0 legacy=0x7f62ec06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:12.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1038640 1 -- 172.21.15.57:0/4198467974 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec157df0 con 0x7f62ec151840 2023-10-30T14:43:12.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1038640 1 -- 172.21.15.57:0/4198467974 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f62e8001680 con 0x7f62ec151840 2023-10-30T14:43:12.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1038640 1 -- 172.21.15.57:0/4198467974 >> v1:172.21.15.57:6789/0 conn(0x7f62ec151840 legacy=0x7f62ec072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:12.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1038640 1 --2- 172.21.15.57:0/4198467974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62d8003980 0x7f62d8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:12.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1038640 1 --2- 172.21.15.57:0/4198467974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62d80047d0 0x7f62d8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:12.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1038640 1 --2- 172.21.15.57:0/4198467974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62d8005190 0x7f62d800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:12.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1038640 1 -- 172.21.15.57:0/4198467974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec157b10 con 0x7f62d80047d0 2023-10-30T14:43:12.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1038640 1 -- 172.21.15.57:0/4198467974 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d800c040 con 0x7f62d8003980 2023-10-30T14:43:12.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1038640 1 -- 172.21.15.57:0/4198467974 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d800c200 con 0x7f62d8005190 2023-10-30T14:43:12.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1839640 1 --2- 172.21.15.57:0/4198467974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62d80047d0 0x7f62d8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:12.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f203a640 1 --2- 172.21.15.57:0/4198467974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62d8003980 0x7f62d8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:12.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f283b640 1 --2- 172.21.15.57:0/4198467974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62d8005190 0x7f62d800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:12.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1839640 1 -- 172.21.15.57:0/4198467974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62d8005190 msgr2=0x7f62d800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:12.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1839640 1 --2- 172.21.15.57:0/4198467974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62d8005190 0x7f62d800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:12.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1839640 1 -- 172.21.15.57:0/4198467974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62d8003980 msgr2=0x7f62d8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:12.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1839640 1 --2- 172.21.15.57:0/4198467974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62d8003980 0x7f62d8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:12.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1839640 1 -- 172.21.15.57:0/4198467974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d8003530 con 0x7f62d80047d0 2023-10-30T14:43:12.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f283b640 1 -- 172.21.15.57:0/4198467974 reap_dead start 2023-10-30T14:43:12.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.408+0000 7f62f1839640 1 --2- 172.21.15.57:0/4198467974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62d80047d0 0x7f62d8004c50 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f62dc009ae0 tx=0x7f62dc01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71750f0232861933 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:12.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.412+0000 7f62f1038640 1 -- 172.21.15.57:0/4198467974 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0a5660 con 0x7f62d80047d0 2023-10-30T14:43:12.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.412+0000 7f62f1038640 1 -- 172.21.15.57:0/4198467974 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62dc0a57c0 con 0x7f62d80047d0 2023-10-30T14:43:12.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.412+0000 7f62f1038640 1 -- 172.21.15.57:0/4198467974 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0a5b70 con 0x7f62d80047d0 2023-10-30T14:43:12.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.412+0000 7f62f429e640 1 -- 172.21.15.57:0/4198467974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62d80047d0 msgr2=0x7f62d8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:12.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.412+0000 7f62f429e640 1 --2- 172.21.15.57:0/4198467974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62d80047d0 0x7f62d8004c50 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f62dc009ae0 tx=0x7f62dc01ca70 comp rx=0 tx=0).stop 2023-10-30T14:43:12.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.412+0000 7f62f429e640 1 -- 172.21.15.57:0/4198467974 shutdown_connections 2023-10-30T14:43:12.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.412+0000 7f62f429e640 1 --2- 172.21.15.57:0/4198467974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62d80047d0 0x7f62d8004c50 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:12.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.412+0000 7f62f429e640 1 -- 172.21.15.57:0/4198467974 >> 172.21.15.57:0/4198467974 conn(0x7f62ec0cac70 msgr2=0x7f62ec0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:12.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.412+0000 7f62f429e640 1 -- 172.21.15.57:0/4198467974 shutdown_connections 2023-10-30T14:43:12.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.412+0000 7f62f429e640 1 -- 172.21.15.57:0/4198467974 wait complete. 2023-10-30T14:43:12.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.412+0000 7f62f429e640 1 Processor -- start 2023-10-30T14:43:12.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.412+0000 7f62f429e640 1 -- start start 2023-10-30T14:43:12.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.412+0000 7f62f429e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62d80047d0 0x7f62ec14fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:12.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.412+0000 7f62f429e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62ec154600 0x7f62ec155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:12.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.412+0000 7f62f429e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62ec154b50 0x7f62ec155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:12.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.412+0000 7f62f429e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec1eec00 con 0x7f62ec154600 2023-10-30T14:43:12.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.412+0000 7f62f429e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec1eedf0 con 0x7f62d80047d0 2023-10-30T14:43:12.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.412+0000 7f62f429e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec1ef0d0 con 0x7f62ec154b50 2023-10-30T14:43:12.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62f1839640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62ec154600 0x7f62ec155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:12.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62f1839640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62ec154600 0x7f62ec155950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44566/0 (socket says 172.21.15.57:44566) 2023-10-30T14:43:12.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62f1839640 1 -- 172.21.15.57:0/327565388 learned_addr learned my addr 172.21.15.57:0/327565388 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:12.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62f203a640 1 --2- 172.21.15.57:0/327565388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62d80047d0 0x7f62ec14fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:12.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62f283b640 1 --2- 172.21.15.57:0/327565388 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62ec154b50 0x7f62ec155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:12.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62f1839640 1 -- 172.21.15.57:0/327565388 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62ec154b50 msgr2=0x7f62ec155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:12.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62f1839640 1 --2- 172.21.15.57:0/327565388 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62ec154b50 0x7f62ec155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:12.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62f1839640 1 -- 172.21.15.57:0/327565388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62d80047d0 msgr2=0x7f62ec14fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:12.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62f1839640 1 --2- 172.21.15.57:0/327565388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62d80047d0 0x7f62ec14fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:12.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62f1839640 1 -- 172.21.15.57:0/327565388 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec1ef3b0 con 0x7f62ec154600 2023-10-30T14:43:12.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62f203a640 1 --2- 172.21.15.57:0/327565388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62d80047d0 0x7f62ec14fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:43:12.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62f1839640 1 --2- 172.21.15.57:0/327565388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62ec154600 0x7f62ec155950 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f62dc01c8a0 tx=0x7f62dc019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:12.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62e2ffd640 1 -- 172.21.15.57:0/327565388 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0bd070 con 0x7f62ec154600 2023-10-30T14:43:12.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62e2ffd640 1 -- 172.21.15.57:0/327565388 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62dc004020 con 0x7f62ec154600 2023-10-30T14:43:12.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62f429e640 1 -- 172.21.15.57:0/327565388 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62ec1ef690 con 0x7f62ec154600 2023-10-30T14:43:12.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62e2ffd640 1 -- 172.21.15.57:0/327565388 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0043d0 con 0x7f62ec154600 2023-10-30T14:43:12.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62f429e640 1 -- 172.21.15.57:0/327565388 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62ec150080 con 0x7f62ec154600 2023-10-30T14:43:12.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62f429e640 1 -- 172.21.15.57:0/327565388 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ec04e170 con 0x7f62ec154600 2023-10-30T14:43:12.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62e2ffd640 1 -- 172.21.15.57:0/327565388 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f62dc0b4070 con 0x7f62ec154600 2023-10-30T14:43:12.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62e2ffd640 1 --2- 172.21.15.57:0/327565388 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f62d00664a0 0x7f62d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:12.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.416+0000 7f62e2ffd640 1 -- 172.21.15.57:0/327565388 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f62dc11e4e0 con 0x7f62ec154600 2023-10-30T14:43:12.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.420+0000 7f62f203a640 1 --2- 172.21.15.57:0/327565388 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f62d00664a0 0x7f62d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:12.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.420+0000 7f62f203a640 1 --2- 172.21.15.57:0/327565388 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f62d00664a0 0x7f62d0068960 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f62e8007d50 tx=0x7f62e8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:12.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.420+0000 7f62e2ffd640 1 -- 172.21.15.57:0/327565388 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f62dc0eb3d0 con 0x7f62ec154600 2023-10-30T14:43:12.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.612+0000 7f62f429e640 1 -- 172.21.15.57:0/327565388 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62ec155060 con 0x7f62ec154600 2023-10-30T14:43:12.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.612+0000 7f62e2ffd640 1 -- 172.21.15.57:0/327565388 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+6532 (secure 0 0 0) 0x7f62dc0ee9f0 con 0x7f62ec154600 2023-10-30T14:43:12.617 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:12.617 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:12.237272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:12.619 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 158 2023-10-30T14:43:12.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.616+0000 7f62f429e640 1 -- 172.21.15.57:0/327565388 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f62d00664a0 msgr2=0x7f62d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:12.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.616+0000 7f62f429e640 1 --2- 172.21.15.57:0/327565388 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f62d00664a0 0x7f62d0068960 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f62e8007d50 tx=0x7f62e8009040 comp rx=0 tx=0).stop 2023-10-30T14:43:12.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.616+0000 7f62f429e640 1 -- 172.21.15.57:0/327565388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62ec154600 msgr2=0x7f62ec155950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:12.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.616+0000 7f62f429e640 1 --2- 172.21.15.57:0/327565388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62ec154600 0x7f62ec155950 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f62dc01c8a0 tx=0x7f62dc019810 comp rx=0 tx=0).stop 2023-10-30T14:43:12.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.616+0000 7f62f429e640 1 -- 172.21.15.57:0/327565388 shutdown_connections 2023-10-30T14:43:12.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.616+0000 7f62f429e640 1 --2- 172.21.15.57:0/327565388 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f62d00664a0 0x7f62d0068960 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:12.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.616+0000 7f62f429e640 1 --2- 172.21.15.57:0/327565388 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62ec154b50 0x7f62ec155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:12.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.616+0000 7f62f429e640 1 --2- 172.21.15.57:0/327565388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62ec154600 0x7f62ec155950 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:12.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.616+0000 7f62f429e640 1 --2- 172.21.15.57:0/327565388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62d80047d0 0x7f62ec14fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:12.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.616+0000 7f62f429e640 1 -- 172.21.15.57:0/327565388 >> 172.21.15.57:0/327565388 conn(0x7f62ec0cac70 msgr2=0x7f62ec1e3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:12.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.616+0000 7f62f429e640 1 -- 172.21.15.57:0/327565388 shutdown_connections 2023-10-30T14:43:12.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:12.616+0000 7f62f429e640 1 -- 172.21.15.57:0/327565388 wait complete. 2023-10-30T14:43:12.638 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T14:43:12.639 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 158, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 158, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6356": { "addr": "172.21.15.88:6839/1744808283", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1744808283, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1744808283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6356, "incarnation": 151, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 45 }, "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 44 }, "gid_7056": { "addr": "172.21.15.57:6837/2307041430", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 2307041430, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 2307041430, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 7056, "incarnation": 137, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:stopping", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:43:12.237272+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6356, "mds_1": 6747, "mds_2": 7056 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.57:6834/697507447", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 697507447, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 697507447, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 156, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7377, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:43:13.153 INFO:tasks.ceph.mds.f.smithi088.stderr:2023-10-30T14:43:13.151+0000 7fc6cb9f2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T14:43:13.153 INFO:tasks.ceph.mds.f.smithi088.stderr:2023-10-30T14:43:13.151+0000 7fc6cb9f2640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:43:14.641 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:14.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.799+0000 7fe28ad71640 1 Processor -- start 2023-10-30T14:43:14.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.799+0000 7fe28ad71640 1 -- start start 2023-10-30T14:43:14.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.799+0000 7fe28ad71640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe2840c8eb0 0x7fe28415cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:14.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.799+0000 7fe28ad71640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe28415ac70 0x7fe28415a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:14.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.799+0000 7fe28ad71640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe28406aeb0 0x7fe28406b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:14.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.799+0000 7fe28ad71640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe284159800 con 0x7fe2840c8eb0 2023-10-30T14:43:14.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.799+0000 7fe28ad71640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe28415dcc0 con 0x7fe28406aeb0 2023-10-30T14:43:14.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.799+0000 7fe28ad71640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe284159d80 con 0x7fe28415ac70 2023-10-30T14:43:14.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.799+0000 7fe289d6f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe2840c8eb0 0x7fe28415cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:14.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.799+0000 7fe289d6f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe2840c8eb0 0x7fe28415cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34772/0 (socket says 172.21.15.57:34772) 2023-10-30T14:43:14.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.799+0000 7fe289d6f640 1 -- 172.21.15.57:0/2947373969 learned_addr learned my addr 172.21.15.57:0/2947373969 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:14.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.799+0000 7fe28956e640 1 --2- 172.21.15.57:0/2947373969 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe28415ac70 0x7fe28415a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:14.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.799+0000 7fe28a570640 1 --2- 172.21.15.57:0/2947373969 >> v2:172.21.15.88:3300/0 conn(0x7fe28406aeb0 0x7fe28406b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:14.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.799+0000 7fe289d6f640 1 -- 172.21.15.57:0/2947373969 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe28415ac70 msgr2=0x7fe28415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:14.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.799+0000 7fe289d6f640 1 --2- 172.21.15.57:0/2947373969 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe28415ac70 0x7fe28415a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:14.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.799+0000 7fe289d6f640 1 -- 172.21.15.57:0/2947373969 >> v2:172.21.15.88:3300/0 conn(0x7fe28406aeb0 msgr2=0x7fe28406b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:14.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.799+0000 7fe289d6f640 1 --2- 172.21.15.57:0/2947373969 >> v2:172.21.15.88:3300/0 conn(0x7fe28406aeb0 0x7fe28406b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:14.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.799+0000 7fe289d6f640 1 -- 172.21.15.57:0/2947373969 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28415a750 con 0x7fe2840c8eb0 2023-10-30T14:43:14.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.799+0000 7fe28956e640 1 --2- 172.21.15.57:0/2947373969 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe28415ac70 0x7fe28415a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:43:14.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.803+0000 7fe289d6f640 1 --2- 172.21.15.57:0/2947373969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2840c8eb0 0x7fe28415cd70 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fe28408cfd0 tx=0x7fe28001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acb578aec938ec49 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:14.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.803+0000 7fe288d6d640 1 -- 172.21.15.57:0/2947373969 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2800a89b0 con 0x7fe2840c8eb0 2023-10-30T14:43:14.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.803+0000 7fe288d6d640 1 -- 172.21.15.57:0/2947373969 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2800a8b10 con 0x7fe2840c8eb0 2023-10-30T14:43:14.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.803+0000 7fe288d6d640 1 -- 172.21.15.57:0/2947373969 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2800a8ea0 con 0x7fe2840c8eb0 2023-10-30T14:43:14.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.803+0000 7fe28ad71640 1 -- 172.21.15.57:0/2947373969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2840c8eb0 msgr2=0x7fe28415cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:14.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.803+0000 7fe28ad71640 1 --2- 172.21.15.57:0/2947373969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2840c8eb0 0x7fe28415cd70 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fe28408cfd0 tx=0x7fe28001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:43:14.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.803+0000 7fe28ad71640 1 -- 172.21.15.57:0/2947373969 shutdown_connections 2023-10-30T14:43:14.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.803+0000 7fe28ad71640 1 --2- 172.21.15.57:0/2947373969 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe28415ac70 0x7fe28415a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:14.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.803+0000 7fe28ad71640 1 --2- 172.21.15.57:0/2947373969 >> v2:172.21.15.88:3300/0 conn(0x7fe28406aeb0 0x7fe28406b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:14.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.803+0000 7fe28ad71640 1 --2- 172.21.15.57:0/2947373969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2840c8eb0 0x7fe28415cd70 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:14.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.803+0000 7fe28ad71640 1 -- 172.21.15.57:0/2947373969 >> 172.21.15.57:0/2947373969 conn(0x7fe2840cef00 msgr2=0x7fe2840cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:14.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.803+0000 7fe28ad71640 1 -- 172.21.15.57:0/2947373969 shutdown_connections 2023-10-30T14:43:14.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.803+0000 7fe28ad71640 1 -- 172.21.15.57:0/2947373969 wait complete. 2023-10-30T14:43:14.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.803+0000 7fe28ad71640 1 Processor -- start 2023-10-30T14:43:14.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28ad71640 1 -- start start 2023-10-30T14:43:14.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28ad71640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe28406aeb0 0x7fe2841e40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:14.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28ad71640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2840c8eb0 0x7fe284157150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:14.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28ad71640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe28415ac70 0x7fe2840c47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:14.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28ad71640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2841e3580 con 0x7fe2840c8eb0 2023-10-30T14:43:14.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28ad71640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe284155b00 con 0x7fe28415ac70 2023-10-30T14:43:14.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28ad71640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe284156cc0 con 0x7fe28406aeb0 2023-10-30T14:43:14.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28956e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2840c8eb0 0x7fe284157150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:14.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28956e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2840c8eb0 0x7fe284157150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34784/0 (socket says 172.21.15.57:34784) 2023-10-30T14:43:14.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28956e640 1 -- 172.21.15.57:0/4121978155 learned_addr learned my addr 172.21.15.57:0/4121978155 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:14.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe289d6f640 1 --2- 172.21.15.57:0/4121978155 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe28406aeb0 0x7fe2841e40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:14.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28a570640 1 --2- 172.21.15.57:0/4121978155 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe28415ac70 0x7fe2840c47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:14.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28956e640 1 -- 172.21.15.57:0/4121978155 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe28406aeb0 msgr2=0x7fe2841e40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:14.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28956e640 1 --2- 172.21.15.57:0/4121978155 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe28406aeb0 0x7fe2841e40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:14.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28956e640 1 -- 172.21.15.57:0/4121978155 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe28415ac70 msgr2=0x7fe2840c47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:14.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28956e640 1 --2- 172.21.15.57:0/4121978155 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe28415ac70 0x7fe2840c47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:14.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28956e640 1 -- 172.21.15.57:0/4121978155 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2800094d0 con 0x7fe2840c8eb0 2023-10-30T14:43:14.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28956e640 1 --2- 172.21.15.57:0/4121978155 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2840c8eb0 0x7fe284157150 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fe27400c7f0 tx=0x7fe27400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:14.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe27a7fc640 1 -- 172.21.15.57:0/4121978155 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe274004220 con 0x7fe2840c8eb0 2023-10-30T14:43:14.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe27a7fc640 1 -- 172.21.15.57:0/4121978155 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe274004380 con 0x7fe2840c8eb0 2023-10-30T14:43:14.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28ad71640 1 -- 172.21.15.57:0/4121978155 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2841e4610 con 0x7fe2840c8eb0 2023-10-30T14:43:14.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe27a7fc640 1 -- 172.21.15.57:0/4121978155 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe274003840 con 0x7fe2840c8eb0 2023-10-30T14:43:14.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28ad71640 1 -- 172.21.15.57:0/4121978155 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe284156440 con 0x7fe2840c8eb0 2023-10-30T14:43:14.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.807+0000 7fe28ad71640 1 -- 172.21.15.57:0/4121978155 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28404e170 con 0x7fe2840c8eb0 2023-10-30T14:43:14.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.811+0000 7fe27a7fc640 1 -- 172.21.15.57:0/4121978155 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe274003bb0 con 0x7fe2840c8eb0 2023-10-30T14:43:14.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.811+0000 7fe27a7fc640 1 --2- 172.21.15.57:0/4121978155 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe2680664a0 0x7fe268068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:14.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.811+0000 7fe27a7fc640 1 -- 172.21.15.57:0/4121978155 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7fe274084190 con 0x7fe2840c8eb0 2023-10-30T14:43:14.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.811+0000 7fe289d6f640 1 --2- 172.21.15.57:0/4121978155 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe2680664a0 0x7fe268068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:14.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.811+0000 7fe289d6f640 1 --2- 172.21.15.57:0/4121978155 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe2680664a0 0x7fe268068960 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fe2800b2000 tx=0x7fe280009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:14.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:14.815+0000 7fe27a7fc640 1 -- 172.21.15.57:0/4121978155 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe274051290 con 0x7fe2840c8eb0 2023-10-30T14:43:15.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:15.003+0000 7fe28ad71640 1 -- 172.21.15.57:0/4121978155 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2840ca960 con 0x7fe2840c8eb0 2023-10-30T14:43:15.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:15.003+0000 7fe27a7fc640 1 -- 172.21.15.57:0/4121978155 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+6532 (secure 0 0 0) 0x7fe274003e60 con 0x7fe2840c8eb0 2023-10-30T14:43:15.008 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:15.008 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:12.237272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:15.009 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 158 2023-10-30T14:43:15.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:15.003+0000 7fe28ad71640 1 -- 172.21.15.57:0/4121978155 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe2680664a0 msgr2=0x7fe268068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:15.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:15.003+0000 7fe28ad71640 1 --2- 172.21.15.57:0/4121978155 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe2680664a0 0x7fe268068960 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fe2800b2000 tx=0x7fe280009040 comp rx=0 tx=0).stop 2023-10-30T14:43:15.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:15.003+0000 7fe28ad71640 1 -- 172.21.15.57:0/4121978155 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2840c8eb0 msgr2=0x7fe284157150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:15.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:15.003+0000 7fe28ad71640 1 --2- 172.21.15.57:0/4121978155 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2840c8eb0 0x7fe284157150 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fe27400c7f0 tx=0x7fe27400ccb0 comp rx=0 tx=0).stop 2023-10-30T14:43:15.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:15.003+0000 7fe28ad71640 1 -- 172.21.15.57:0/4121978155 shutdown_connections 2023-10-30T14:43:15.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:15.003+0000 7fe28ad71640 1 --2- 172.21.15.57:0/4121978155 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe2680664a0 0x7fe268068960 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:15.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:15.003+0000 7fe28ad71640 1 --2- 172.21.15.57:0/4121978155 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe28415ac70 0x7fe2840c47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:15.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:15.003+0000 7fe28ad71640 1 --2- 172.21.15.57:0/4121978155 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2840c8eb0 0x7fe284157150 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:15.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:15.003+0000 7fe28ad71640 1 --2- 172.21.15.57:0/4121978155 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe28406aeb0 0x7fe2841e40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:15.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:15.003+0000 7fe28ad71640 1 -- 172.21.15.57:0/4121978155 >> 172.21.15.57:0/4121978155 conn(0x7fe2840cef00 msgr2=0x7fe284158580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:15.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:15.003+0000 7fe28ad71640 1 -- 172.21.15.57:0/4121978155 shutdown_connections 2023-10-30T14:43:15.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:15.003+0000 7fe28ad71640 1 -- 172.21.15.57:0/4121978155 wait complete. 2023-10-30T14:43:15.028 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:43:17.030 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:17.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.179+0000 7f2ac42c9640 1 Processor -- start 2023-10-30T14:43:17.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.179+0000 7f2ac42c9640 1 -- start start 2023-10-30T14:43:17.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.183+0000 7f2ac42c9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2abc0cb7d0 0x7f2abc0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:17.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.183+0000 7f2ac42c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2abc0c3e40 0x7f2abc070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:17.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.183+0000 7f2ac42c9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2abc0c2310 con 0x7f2abc150830 2023-10-30T14:43:17.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.183+0000 7f2ac42c9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2abc06bc10 con 0x7f2abc0cb7d0 2023-10-30T14:43:17.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.183+0000 7f2ac42c9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2abc156710 con 0x7f2abc0c3e40 2023-10-30T14:43:17.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.183+0000 7f2ac1864640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2abc0cb7d0 0x7f2abc0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:17.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.183+0000 7f2ac1864640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2abc0cb7d0 0x7f2abc0c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34798/0 (socket says 172.21.15.57:34798) 2023-10-30T14:43:17.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.183+0000 7f2ac1864640 1 -- 172.21.15.57:0/331513549 learned_addr learned my addr 172.21.15.57:0/331513549 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:17.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.183+0000 7f2ac2866640 1 --2- 172.21.15.57:0/331513549 >> v2:172.21.15.88:3300/0 conn(0x7f2abc0c3e40 0x7f2abc070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:17.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.183+0000 7f2ac1864640 1 -- 172.21.15.57:0/331513549 >> v2:172.21.15.88:3300/0 conn(0x7f2abc0c3e40 msgr2=0x7f2abc070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:17.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.183+0000 7f2ac1864640 1 --2- 172.21.15.57:0/331513549 >> v2:172.21.15.88:3300/0 conn(0x7f2abc0c3e40 0x7f2abc070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:17.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.183+0000 7f2ac1864640 1 -- 172.21.15.57:0/331513549 >> v1:172.21.15.57:6789/0 conn(0x7f2abc150830 legacy=0x7f2abc06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:17.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.183+0000 7f2ac1864640 1 -- 172.21.15.57:0/331513549 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2abc0c4360 con 0x7f2abc0cb7d0 2023-10-30T14:43:17.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.183+0000 7f2ac1864640 1 --2- 172.21.15.57:0/331513549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2abc0cb7d0 0x7f2abc0c3900 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f2abc066e10 tx=0x7f2aac0213c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26350edc31f18a15 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:17.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.183+0000 7f2ac1063640 1 -- 172.21.15.57:0/331513549 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac0b1600 con 0x7f2abc0cb7d0 2023-10-30T14:43:17.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac1063640 1 -- 172.21.15.57:0/331513549 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aac0ad050 con 0x7f2abc0cb7d0 2023-10-30T14:43:17.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac1063640 1 -- 172.21.15.57:0/331513549 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac0b1980 con 0x7f2abc0cb7d0 2023-10-30T14:43:17.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac42c9640 1 -- 172.21.15.57:0/331513549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2abc0cb7d0 msgr2=0x7f2abc0c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:17.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac42c9640 1 --2- 172.21.15.57:0/331513549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2abc0cb7d0 0x7f2abc0c3900 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f2abc066e10 tx=0x7f2aac0213c0 comp rx=0 tx=0).stop 2023-10-30T14:43:17.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac42c9640 1 -- 172.21.15.57:0/331513549 shutdown_connections 2023-10-30T14:43:17.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac42c9640 1 --2- 172.21.15.57:0/331513549 >> v2:172.21.15.88:3300/0 conn(0x7f2abc0c3e40 0x7f2abc070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:17.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac42c9640 1 --2- 172.21.15.57:0/331513549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2abc0cb7d0 0x7f2abc0c3900 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:17.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac42c9640 1 -- 172.21.15.57:0/331513549 >> 172.21.15.57:0/331513549 conn(0x7f2abc0cac70 msgr2=0x7f2abc0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:17.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac42c9640 1 -- 172.21.15.57:0/331513549 shutdown_connections 2023-10-30T14:43:17.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac42c9640 1 -- 172.21.15.57:0/331513549 wait complete. 2023-10-30T14:43:17.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac42c9640 1 Processor -- start 2023-10-30T14:43:17.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac42c9640 1 -- start start 2023-10-30T14:43:17.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac42c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2abc0c3e40 0x7f2abc14b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:17.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac42c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2abc0cb7d0 0x7f2abc1559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:17.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac42c9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2abc150830 0x7f2abc155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:17.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac42c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc1f1f50 con 0x7f2abc0c3e40 2023-10-30T14:43:17.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac42c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc1f2230 con 0x7f2abc0cb7d0 2023-10-30T14:43:17.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac42c9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc1f2510 con 0x7f2abc150830 2023-10-30T14:43:17.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac2065640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2abc0c3e40 0x7f2abc14b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:17.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac2065640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2abc0c3e40 0x7f2abc14b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34810/0 (socket says 172.21.15.57:34810) 2023-10-30T14:43:17.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac2065640 1 -- 172.21.15.57:0/1006805554 learned_addr learned my addr 172.21.15.57:0/1006805554 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:17.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.187+0000 7f2ac1864640 1 --2- 172.21.15.57:0/1006805554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2abc0cb7d0 0x7f2abc1559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:17.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.191+0000 7f2ac2866640 1 --2- 172.21.15.57:0/1006805554 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2abc150830 0x7f2abc155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:17.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.191+0000 7f2ac2065640 1 -- 172.21.15.57:0/1006805554 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2abc150830 msgr2=0x7f2abc155ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:17.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.191+0000 7f2ac2065640 1 --2- 172.21.15.57:0/1006805554 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2abc150830 0x7f2abc155ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:17.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.191+0000 7f2ac2065640 1 -- 172.21.15.57:0/1006805554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2abc0cb7d0 msgr2=0x7f2abc1559a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:17.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.191+0000 7f2ac2065640 1 --2- 172.21.15.57:0/1006805554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2abc0cb7d0 0x7f2abc1559a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:17.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.191+0000 7f2ac2065640 1 -- 172.21.15.57:0/1006805554 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac00b770 con 0x7f2abc0c3e40 2023-10-30T14:43:17.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.191+0000 7f2ac2065640 1 --2- 172.21.15.57:0/1006805554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2abc0c3e40 0x7f2abc14b9c0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f2ab800c450 tx=0x7f2ab800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:17.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.191+0000 7f2ab2ffd640 1 -- 172.21.15.57:0/1006805554 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab80052d0 con 0x7f2abc0c3e40 2023-10-30T14:43:17.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.191+0000 7f2ab2ffd640 1 -- 172.21.15.57:0/1006805554 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab8005450 con 0x7f2abc0c3e40 2023-10-30T14:43:17.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.191+0000 7f2ac42c9640 1 -- 172.21.15.57:0/1006805554 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2abc1f27f0 con 0x7f2abc0c3e40 2023-10-30T14:43:17.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.191+0000 7f2ab2ffd640 1 -- 172.21.15.57:0/1006805554 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8012870 con 0x7f2abc0c3e40 2023-10-30T14:43:17.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.191+0000 7f2ac42c9640 1 -- 172.21.15.57:0/1006805554 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2abc14bf00 con 0x7f2abc0c3e40 2023-10-30T14:43:17.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.191+0000 7f2ac42c9640 1 -- 172.21.15.57:0/1006805554 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2abc04e170 con 0x7f2abc0c3e40 2023-10-30T14:43:17.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.191+0000 7f2ab2ffd640 1 -- 172.21.15.57:0/1006805554 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2ab80129d0 con 0x7f2abc0c3e40 2023-10-30T14:43:17.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.191+0000 7f2ab2ffd640 1 --2- 172.21.15.57:0/1006805554 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2aa00664d0 0x7f2aa0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:17.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.191+0000 7f2ab2ffd640 1 -- 172.21.15.57:0/1006805554 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f2ab8083ff0 con 0x7f2abc0c3e40 2023-10-30T14:43:17.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.191+0000 7f2ac1864640 1 --2- 172.21.15.57:0/1006805554 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2aa00664d0 0x7f2aa0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:17.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.195+0000 7f2ac1864640 1 --2- 172.21.15.57:0/1006805554 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2aa00664d0 0x7f2aa0068990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2aac0b3000 tx=0x7f2aac008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:17.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.195+0000 7f2ab2ffd640 1 -- 172.21.15.57:0/1006805554 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2ab80b5090 con 0x7f2abc0c3e40 2023-10-30T14:43:17.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.431+0000 7f2ac42c9640 1 -- 172.21.15.57:0/1006805554 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2abc156c60 con 0x7f2abc0c3e40 2023-10-30T14:43:17.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.435+0000 7f2ab2ffd640 1 -- 172.21.15.57:0/1006805554 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+6532 (secure 0 0 0) 0x7f2ab8050de0 con 0x7f2abc0c3e40 2023-10-30T14:43:17.438 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:17.439 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:12.237272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:17.439 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 158 2023-10-30T14:43:17.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.435+0000 7f2ac42c9640 1 -- 172.21.15.57:0/1006805554 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2aa00664d0 msgr2=0x7f2aa0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:17.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.435+0000 7f2ac42c9640 1 --2- 172.21.15.57:0/1006805554 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2aa00664d0 0x7f2aa0068990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2aac0b3000 tx=0x7f2aac008040 comp rx=0 tx=0).stop 2023-10-30T14:43:17.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.435+0000 7f2ac42c9640 1 -- 172.21.15.57:0/1006805554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2abc0c3e40 msgr2=0x7f2abc14b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:17.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.435+0000 7f2ac42c9640 1 --2- 172.21.15.57:0/1006805554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2abc0c3e40 0x7f2abc14b9c0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f2ab800c450 tx=0x7f2ab800c910 comp rx=0 tx=0).stop 2023-10-30T14:43:17.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.435+0000 7f2ac42c9640 1 -- 172.21.15.57:0/1006805554 shutdown_connections 2023-10-30T14:43:17.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.435+0000 7f2ac42c9640 1 --2- 172.21.15.57:0/1006805554 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2aa00664d0 0x7f2aa0068990 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:17.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.435+0000 7f2ac42c9640 1 --2- 172.21.15.57:0/1006805554 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2abc150830 0x7f2abc155ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:17.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.435+0000 7f2ac42c9640 1 --2- 172.21.15.57:0/1006805554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2abc0cb7d0 0x7f2abc1559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:17.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.435+0000 7f2ac42c9640 1 --2- 172.21.15.57:0/1006805554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2abc0c3e40 0x7f2abc14b9c0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:17.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.435+0000 7f2ac42c9640 1 -- 172.21.15.57:0/1006805554 >> 172.21.15.57:0/1006805554 conn(0x7f2abc0cac70 msgr2=0x7f2abc06bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:17.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.435+0000 7f2ac42c9640 1 -- 172.21.15.57:0/1006805554 shutdown_connections 2023-10-30T14:43:17.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:17.435+0000 7f2ac42c9640 1 -- 172.21.15.57:0/1006805554 wait complete. 2023-10-30T14:43:17.467 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:43:18.065 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 2% (6/258) Updating files: 3% (8/258) Updating files: 4% (11/258) Updating files: 5% (13/258) Updating files: 6% (16/258) Updating files: 7% (19/258) Updating files: 8% (21/258) Updating files: 9% (24/258) Updating files: 10% (26/258) Updating files: 11% (29/258) Updating files: 12% (31/258) Updating files: 13% (34/258) Updating files: 14% (37/258) Updating files: 15% (39/258) Updating files: 16% (42/258) Updating files: 17% (44/258) Updating files: 18% (47/258) Updating files: 19% (50/258) Updating files: 20% (52/258) Updating files: 21% (55/258) Updating files: 22% (57/258) Updating files: 23% (60/258) Updating files: 24% (62/258) Updating files: 25% (65/258) Updating files: 26% (68/258) Updating files: 27% (70/258) Updating files: 28% (73/258) Updating files: 29% (75/258) Updating files: 30% (78/258) Updating files: 31% (80/258) Updating files: 32% (83/258) Updating files: 33% (86/258) Updating files: 34% (88/258) Updating files: 35% (91/258) Updating files: 35% (92/258) Updating files: 36% (93/258) Updating files: 37% (96/258) Updating files: 38% (99/258) Updating files: 39% (101/258) Updating files: 40% (104/258) Updating files: 41% (106/258) Updating files: 42% (109/258) Updating files: 42% (110/258) Updating files: 43% (111/258) Updating files: 44% (114/258) Updating files: 45% (117/258) Updating files: 46% (119/258) Updating files: 47% (122/258) Updating files: 48% (124/258) Updating files: 48% (125/258) Updating files: 49% (127/258) Updating files: 50% (129/258) Updating files: 51% (132/258) Updating files: 52% (135/258) Updating files: 53% (137/258) Updating files: 54% (140/258) Updating files: 55% (142/258) Updating files: 56% (145/258) Updating files: 57% (148/258) Updating files: 58% (150/258) Updating files: 59% (153/258) Updating files: 60% (155/258) Updating files: 61% (158/258) Updating files: 62% (160/258) Updating files: 62% (162/258) Updating files: 63% (163/258) Updating files: 64% (166/258) Updating files: 65% (168/258) Updating files: 66% (171/258) Updating files: 67% (173/258) Updating files: 68% (176/258) Updating files: 69% (179/258) Updating files: 70% (181/258) Updating files: 71% (184/258) Updating files: 72% (186/258) Updating files: 73% (189/258) Updating files: 74% (191/258) Updating files: 75% (194/258) Updating files: 75% (196/258) Updating files: 76% (197/258) Updating files: 77% (199/258) Updating files: 78% (202/258) Updating files: 79% (204/258) Updating files: 80% (207/258) Updating files: 81% (209/258) Updating files: 82% (212/258) Updating files: 83% (215/258) Updating files: 84% (217/258) Updating files: 85% (220/258) Updating files: 86% (222/258) Updating files: 87% (225/258) Updating files: 88% (228/258) Updating files: 89% (230/258) Updating files: 90% (233/258) Updating files: 91% (235/258) Updating files: 92% (238/258) Updating files: 93% (240/258) Updating files: 94% (243/258) Updating files: 95% (246/258) Updating files: 96% (248/258) Updating files: 97% (251/258) Updating files: 98% (253/258) Updating files: 99% (256/258) Updating files: 100% (258/258) Updating files: 100% (258/258), done. 2023-10-30T14:43:19.097 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 321ecdaba2c v0.39 2023-10-30T14:43:19.100 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.39 2023-10-30T14:43:19.177 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 40 -eq 48 ] 2023-10-30T14:43:19.177 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.40 2023-10-30T14:43:19.177 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.40 2023-10-30T14:43:19.178 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.40 2023-10-30T14:43:19.178 INFO:tasks.workunit.client.0.smithi057.stdout:v0.40 2023-10-30T14:43:19.468 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:19.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.619+0000 7fc168446640 1 Processor -- start 2023-10-30T14:43:19.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.619+0000 7fc168446640 1 -- start start 2023-10-30T14:43:19.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.623+0000 7fc168446640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc160150830 0x7fc16006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:19.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.623+0000 7fc168446640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc1600cb7d0 0x7fc1600c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:19.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.623+0000 7fc168446640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1600c2310 con 0x7fc1600c3e40 2023-10-30T14:43:19.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.623+0000 7fc168446640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc160156710 con 0x7fc160150830 2023-10-30T14:43:19.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.623+0000 7fc168446640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1600c4360 con 0x7fc1600cb7d0 2023-10-30T14:43:19.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.623+0000 7fc1661e2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc160150830 0x7fc16006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:19.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.623+0000 7fc1661e2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc160150830 0x7fc16006b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34820/0 (socket says 172.21.15.57:34820) 2023-10-30T14:43:19.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.623+0000 7fc1661e2640 1 -- 172.21.15.57:0/2718442316 learned_addr learned my addr 172.21.15.57:0/2718442316 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:19.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.623+0000 7fc1659e1640 1 --2- 172.21.15.57:0/2718442316 >> v2:172.21.15.88:3300/0 conn(0x7fc1600cb7d0 0x7fc1600c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:19.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.623+0000 7fc1661e2640 1 -- 172.21.15.57:0/2718442316 >> v2:172.21.15.88:3300/0 conn(0x7fc1600cb7d0 msgr2=0x7fc1600c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:19.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.623+0000 7fc1661e2640 1 --2- 172.21.15.57:0/2718442316 >> v2:172.21.15.88:3300/0 conn(0x7fc1600cb7d0 0x7fc1600c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:19.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.623+0000 7fc1661e2640 1 -- 172.21.15.57:0/2718442316 >> v1:172.21.15.88:6789/0 conn(0x7fc1600c3e40 legacy=0x7fc160070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:19.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.623+0000 7fc1661e2640 1 -- 172.21.15.57:0/2718442316 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1601569b0 con 0x7fc160150830 2023-10-30T14:43:19.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.623+0000 7fc1659e1640 1 --2- 172.21.15.57:0/2718442316 >> v2:172.21.15.88:3300/0 conn(0x7fc1600cb7d0 0x7fc1600c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:43:19.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.623+0000 7fc1661e2640 1 --2- 172.21.15.57:0/2718442316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc160150830 0x7fc16006b500 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fc16006f9f0 tx=0x7fc15c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9f82841be5e5ca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:19.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.623+0000 7fc1651e0640 1 -- 172.21.15.57:0/2718442316 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c0a89f0 con 0x7fc160150830 2023-10-30T14:43:19.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc1651e0640 1 -- 172.21.15.57:0/2718442316 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15c0a8b50 con 0x7fc160150830 2023-10-30T14:43:19.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc1651e0640 1 -- 172.21.15.57:0/2718442316 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c0b0440 con 0x7fc160150830 2023-10-30T14:43:19.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc168446640 1 -- 172.21.15.57:0/2718442316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc160150830 msgr2=0x7fc16006b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:19.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc168446640 1 --2- 172.21.15.57:0/2718442316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc160150830 0x7fc16006b500 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fc16006f9f0 tx=0x7fc15c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:43:19.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc168446640 1 -- 172.21.15.57:0/2718442316 shutdown_connections 2023-10-30T14:43:19.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc168446640 1 --2- 172.21.15.57:0/2718442316 >> v2:172.21.15.88:3300/0 conn(0x7fc1600cb7d0 0x7fc1600c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:19.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc168446640 1 --2- 172.21.15.57:0/2718442316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc160150830 0x7fc16006b500 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:19.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc168446640 1 -- 172.21.15.57:0/2718442316 >> 172.21.15.57:0/2718442316 conn(0x7fc1600cac70 msgr2=0x7fc1600cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:19.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc168446640 1 -- 172.21.15.57:0/2718442316 shutdown_connections 2023-10-30T14:43:19.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc168446640 1 -- 172.21.15.57:0/2718442316 wait complete. 2023-10-30T14:43:19.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc168446640 1 Processor -- start 2023-10-30T14:43:19.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc168446640 1 -- start start 2023-10-30T14:43:19.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc168446640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1600c3e40 0x7fc160149a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:19.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc168446640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1600cb7d0 0x7fc1601ef980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:19.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc168446640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc160150830 0x7fc1601f2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:19.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc168446640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1601f32f0 con 0x7fc1600cb7d0 2023-10-30T14:43:19.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc168446640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1600c6dc0 con 0x7fc1600c3e40 2023-10-30T14:43:19.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc168446640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc160148a70 con 0x7fc160150830 2023-10-30T14:43:19.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc1659e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1600cb7d0 0x7fc1601ef980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:19.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc1659e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1600cb7d0 0x7fc1601ef980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34834/0 (socket says 172.21.15.57:34834) 2023-10-30T14:43:19.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.627+0000 7fc1659e1640 1 -- 172.21.15.57:0/1649590814 learned_addr learned my addr 172.21.15.57:0/1649590814 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:19.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.631+0000 7fc1661e2640 1 --2- 172.21.15.57:0/1649590814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1600c3e40 0x7fc160149a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:19.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.631+0000 7fc1669e3640 1 --2- 172.21.15.57:0/1649590814 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc160150830 0x7fc1601f2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:19.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.631+0000 7fc1659e1640 1 -- 172.21.15.57:0/1649590814 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc160150830 msgr2=0x7fc1601f2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:19.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.631+0000 7fc1659e1640 1 --2- 172.21.15.57:0/1649590814 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc160150830 0x7fc1601f2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:19.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.631+0000 7fc1659e1640 1 -- 172.21.15.57:0/1649590814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1600c3e40 msgr2=0x7fc160149a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:19.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.631+0000 7fc1659e1640 1 --2- 172.21.15.57:0/1649590814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1600c3e40 0x7fc160149a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:19.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.631+0000 7fc1659e1640 1 -- 172.21.15.57:0/1649590814 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15c0094d0 con 0x7fc1600cb7d0 2023-10-30T14:43:19.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.631+0000 7fc1661e2640 1 --2- 172.21.15.57:0/1649590814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1600c3e40 0x7fc160149a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:43:19.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.631+0000 7fc1659e1640 1 --2- 172.21.15.57:0/1649590814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1600cb7d0 0x7fc1601ef980 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fc15000e810 tx=0x7fc15000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:19.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.631+0000 7fc156ffd640 1 -- 172.21.15.57:0/1649590814 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15000c8d0 con 0x7fc1600cb7d0 2023-10-30T14:43:19.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.631+0000 7fc156ffd640 1 -- 172.21.15.57:0/1649590814 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15000ca30 con 0x7fc1600cb7d0 2023-10-30T14:43:19.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.631+0000 7fc156ffd640 1 -- 172.21.15.57:0/1649590814 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15000cdc0 con 0x7fc1600cb7d0 2023-10-30T14:43:19.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.631+0000 7fc168446640 1 -- 172.21.15.57:0/1649590814 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc160148d50 con 0x7fc1600cb7d0 2023-10-30T14:43:19.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.631+0000 7fc168446640 1 -- 172.21.15.57:0/1649590814 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1601ef180 con 0x7fc1600cb7d0 2023-10-30T14:43:19.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.631+0000 7fc168446640 1 -- 172.21.15.57:0/1649590814 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc16004e170 con 0x7fc1600cb7d0 2023-10-30T14:43:19.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.631+0000 7fc156ffd640 1 -- 172.21.15.57:0/1649590814 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc150013410 con 0x7fc1600cb7d0 2023-10-30T14:43:19.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.631+0000 7fc156ffd640 1 --2- 172.21.15.57:0/1649590814 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc1440664d0 0x7fc144068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:19.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.631+0000 7fc156ffd640 1 -- 172.21.15.57:0/1649590814 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7fc1500842d0 con 0x7fc1600cb7d0 2023-10-30T14:43:19.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.635+0000 7fc1661e2640 1 --2- 172.21.15.57:0/1649590814 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc1440664d0 0x7fc144068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:19.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.635+0000 7fc1661e2640 1 --2- 172.21.15.57:0/1649590814 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc1440664d0 0x7fc144068990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0b2000 tx=0x7fc15c0aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:19.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.635+0000 7fc156ffd640 1 -- 172.21.15.57:0/1649590814 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc150051140 con 0x7fc1600cb7d0 2023-10-30T14:43:19.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.827+0000 7fc168446640 1 -- 172.21.15.57:0/1649590814 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc160156c60 con 0x7fc1600cb7d0 2023-10-30T14:43:19.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.827+0000 7fc156ffd640 1 -- 172.21.15.57:0/1649590814 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+6532 (secure 0 0 0) 0x7fc150054760 con 0x7fc1600cb7d0 2023-10-30T14:43:19.834 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:19.834 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:12.237272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:19.836 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 158 2023-10-30T14:43:19.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.831+0000 7fc168446640 1 -- 172.21.15.57:0/1649590814 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc1440664d0 msgr2=0x7fc144068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:19.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.831+0000 7fc168446640 1 --2- 172.21.15.57:0/1649590814 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc1440664d0 0x7fc144068990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0b2000 tx=0x7fc15c0aa070 comp rx=0 tx=0).stop 2023-10-30T14:43:19.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.831+0000 7fc168446640 1 -- 172.21.15.57:0/1649590814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1600cb7d0 msgr2=0x7fc1601ef980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:19.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.831+0000 7fc168446640 1 --2- 172.21.15.57:0/1649590814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1600cb7d0 0x7fc1601ef980 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fc15000e810 tx=0x7fc15000ecd0 comp rx=0 tx=0).stop 2023-10-30T14:43:19.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.831+0000 7fc168446640 1 -- 172.21.15.57:0/1649590814 shutdown_connections 2023-10-30T14:43:19.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.831+0000 7fc168446640 1 --2- 172.21.15.57:0/1649590814 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc1440664d0 0x7fc144068990 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:19.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.831+0000 7fc168446640 1 --2- 172.21.15.57:0/1649590814 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc160150830 0x7fc1601f2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:19.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.831+0000 7fc168446640 1 --2- 172.21.15.57:0/1649590814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1600cb7d0 0x7fc1601ef980 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:19.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.831+0000 7fc168446640 1 --2- 172.21.15.57:0/1649590814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1600c3e40 0x7fc160149a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:19.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.831+0000 7fc168446640 1 -- 172.21.15.57:0/1649590814 >> 172.21.15.57:0/1649590814 conn(0x7fc1600cac70 msgr2=0x7fc1600c5f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:19.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.831+0000 7fc168446640 1 -- 172.21.15.57:0/1649590814 shutdown_connections 2023-10-30T14:43:19.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:19.831+0000 7fc168446640 1 -- 172.21.15.57:0/1649590814 wait complete. 2023-10-30T14:43:19.854 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:43:21.854 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:22.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.007+0000 7f3eb9fc5640 1 Processor -- start 2023-10-30T14:43:22.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.007+0000 7f3eb9fc5640 1 -- start start 2023-10-30T14:43:22.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.007+0000 7f3eb9fc5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3eb4153bb0 0x7f3eb40c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:22.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.007+0000 7f3eb9fc5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3eb406c420 0x7f3eb40c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:22.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.007+0000 7f3eb9fc5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb40c5f80 con 0x7f3eb40c7170 2023-10-30T14:43:22.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.007+0000 7f3eb9fc5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb40c7580 con 0x7f3eb406c420 2023-10-30T14:43:22.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.007+0000 7f3eb9fc5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb40c5690 con 0x7f3eb4153bb0 2023-10-30T14:43:22.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.011+0000 7f3eb2ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3eb406c420 0x7f3eb40c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:22.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.011+0000 7f3eb2ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3eb406c420 0x7f3eb40c6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34848/0 (socket says 172.21.15.57:34848) 2023-10-30T14:43:22.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.011+0000 7f3eb2ffd640 1 -- 172.21.15.57:0/1701627442 learned_addr learned my addr 172.21.15.57:0/1701627442 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:22.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.011+0000 7f3eb37fe640 1 --2- 172.21.15.57:0/1701627442 >> v2:172.21.15.88:3300/0 conn(0x7f3eb4153bb0 0x7f3eb40c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:22.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.011+0000 7f3eb2ffd640 1 -- 172.21.15.57:0/1701627442 >> v2:172.21.15.88:3300/0 conn(0x7f3eb4153bb0 msgr2=0x7f3eb40c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:22.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.011+0000 7f3eb2ffd640 1 --2- 172.21.15.57:0/1701627442 >> v2:172.21.15.88:3300/0 conn(0x7f3eb4153bb0 0x7f3eb40c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:22.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.011+0000 7f3eb2ffd640 1 -- 172.21.15.57:0/1701627442 >> v1:172.21.15.88:6789/0 conn(0x7f3eb40c7170 legacy=0x7f3eb4159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:22.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.011+0000 7f3eb2ffd640 1 -- 172.21.15.57:0/1701627442 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb415a580 con 0x7f3eb406c420 2023-10-30T14:43:22.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.011+0000 7f3eb2ffd640 1 --2- 172.21.15.57:0/1701627442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eb406c420 0x7f3eb40c6c30 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f3eb40cb840 tx=0x7f3ea801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e65be96ba17246 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:22.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.011+0000 7f3eb27fc640 1 -- 172.21.15.57:0/1701627442 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea80a89f0 con 0x7f3eb406c420 2023-10-30T14:43:22.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.011+0000 7f3eb27fc640 1 -- 172.21.15.57:0/1701627442 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea80a8b50 con 0x7f3eb406c420 2023-10-30T14:43:22.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.011+0000 7f3eb27fc640 1 -- 172.21.15.57:0/1701627442 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea80b0440 con 0x7f3eb406c420 2023-10-30T14:43:22.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb9fc5640 1 -- 172.21.15.57:0/1701627442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eb406c420 msgr2=0x7f3eb40c6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:22.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb9fc5640 1 --2- 172.21.15.57:0/1701627442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eb406c420 0x7f3eb40c6c30 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f3eb40cb840 tx=0x7f3ea801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:43:22.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb9fc5640 1 -- 172.21.15.57:0/1701627442 shutdown_connections 2023-10-30T14:43:22.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb9fc5640 1 --2- 172.21.15.57:0/1701627442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eb406c420 0x7f3eb40c6c30 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:22.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb9fc5640 1 --2- 172.21.15.57:0/1701627442 >> v2:172.21.15.88:3300/0 conn(0x7f3eb4153bb0 0x7f3eb40c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:22.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb9fc5640 1 -- 172.21.15.57:0/1701627442 >> 172.21.15.57:0/1701627442 conn(0x7f3eb40cac70 msgr2=0x7f3eb40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:22.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb9fc5640 1 -- 172.21.15.57:0/1701627442 shutdown_connections 2023-10-30T14:43:22.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb9fc5640 1 -- 172.21.15.57:0/1701627442 wait complete. 2023-10-30T14:43:22.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb9fc5640 1 Processor -- start 2023-10-30T14:43:22.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb9fc5640 1 -- start start 2023-10-30T14:43:22.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb9fc5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eb406c420 0x7f3eb41ef2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:22.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb9fc5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3eb40c7170 0x7f3eb40c4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:22.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb9fc5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3eb4153bb0 0x7f3eb414d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:22.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb9fc5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb40c0f00 con 0x7f3eb406c420 2023-10-30T14:43:22.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb9fc5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb406bb70 con 0x7f3eb4153bb0 2023-10-30T14:43:22.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb9fc5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb4152d10 con 0x7f3eb40c7170 2023-10-30T14:43:22.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eb406c420 0x7f3eb41ef2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:22.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eb406c420 0x7f3eb41ef2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34854/0 (socket says 172.21.15.57:34854) 2023-10-30T14:43:22.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb37fe640 1 -- 172.21.15.57:0/2979778851 learned_addr learned my addr 172.21.15.57:0/2979778851 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:22.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.015+0000 7f3eb2ffd640 1 --2- 172.21.15.57:0/2979778851 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3eb40c7170 0x7f3eb40c4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:22.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.019+0000 7f3eb3fff640 1 --2- 172.21.15.57:0/2979778851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3eb4153bb0 0x7f3eb414d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:22.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.019+0000 7f3eb37fe640 1 -- 172.21.15.57:0/2979778851 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3eb40c7170 msgr2=0x7f3eb40c4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:22.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.019+0000 7f3eb37fe640 1 --2- 172.21.15.57:0/2979778851 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3eb40c7170 0x7f3eb40c4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:22.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.019+0000 7f3eb37fe640 1 -- 172.21.15.57:0/2979778851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3eb4153bb0 msgr2=0x7f3eb414d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:22.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.019+0000 7f3eb37fe640 1 --2- 172.21.15.57:0/2979778851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3eb4153bb0 0x7f3eb414d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:22.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.019+0000 7f3eb37fe640 1 -- 172.21.15.57:0/2979778851 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea80094d0 con 0x7f3eb406c420 2023-10-30T14:43:22.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.019+0000 7f3eb3fff640 1 --2- 172.21.15.57:0/2979778851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3eb4153bb0 0x7f3eb414d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:43:22.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.019+0000 7f3eb2ffd640 1 --2- 172.21.15.57:0/2979778851 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3eb40c7170 0x7f3eb40c4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:43:22.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.019+0000 7f3eb37fe640 1 --2- 172.21.15.57:0/2979778851 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eb406c420 0x7f3eb41ef2b0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400e960 tx=0x7f3ea400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:22.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.019+0000 7f3e9bfff640 1 -- 172.21.15.57:0/2979778851 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4009690 con 0x7f3eb406c420 2023-10-30T14:43:22.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.019+0000 7f3e9bfff640 1 -- 172.21.15.57:0/2979778851 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea400ca00 con 0x7f3eb406c420 2023-10-30T14:43:22.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.019+0000 7f3e9bfff640 1 -- 172.21.15.57:0/2979778851 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea400cd70 con 0x7f3eb406c420 2023-10-30T14:43:22.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.019+0000 7f3eb9fc5640 1 -- 172.21.15.57:0/2979778851 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb40c1150 con 0x7f3eb406c420 2023-10-30T14:43:22.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.019+0000 7f3eb9fc5640 1 -- 172.21.15.57:0/2979778851 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb40c2810 con 0x7f3eb406c420 2023-10-30T14:43:22.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.019+0000 7f3eb9fc5640 1 -- 172.21.15.57:0/2979778851 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb404e170 con 0x7f3eb406c420 2023-10-30T14:43:22.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.019+0000 7f3e9bfff640 1 -- 172.21.15.57:0/2979778851 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3ea4013470 con 0x7f3eb406c420 2023-10-30T14:43:22.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.019+0000 7f3e9bfff640 1 --2- 172.21.15.57:0/2979778851 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3ea00664a0 0x7f3ea0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:22.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.019+0000 7f3e9bfff640 1 -- 172.21.15.57:0/2979778851 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f3ea4084aa0 con 0x7f3eb406c420 2023-10-30T14:43:22.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.023+0000 7f3eb2ffd640 1 --2- 172.21.15.57:0/2979778851 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3ea00664a0 0x7f3ea0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:22.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.023+0000 7f3eb2ffd640 1 --2- 172.21.15.57:0/2979778851 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3ea00664a0 0x7f3ea0068960 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80b2000 tx=0x7f3ea8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:22.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.023+0000 7f3e9bfff640 1 -- 172.21.15.57:0/2979778851 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3ea4051910 con 0x7f3eb406c420 2023-10-30T14:43:22.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.215+0000 7f3eb9fc5640 1 -- 172.21.15.57:0/2979778851 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3eb4152810 con 0x7f3eb406c420 2023-10-30T14:43:22.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.215+0000 7f3e9bfff640 1 -- 172.21.15.57:0/2979778851 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+6532 (secure 0 0 0) 0x7f3ea4054f30 con 0x7f3eb406c420 2023-10-30T14:43:22.222 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:22.222 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:12.237272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:22.222 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 158 2023-10-30T14:43:22.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.219+0000 7f3eb9fc5640 1 -- 172.21.15.57:0/2979778851 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3ea00664a0 msgr2=0x7f3ea0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:22.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.219+0000 7f3eb9fc5640 1 --2- 172.21.15.57:0/2979778851 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3ea00664a0 0x7f3ea0068960 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80b2000 tx=0x7f3ea8008040 comp rx=0 tx=0).stop 2023-10-30T14:43:22.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.219+0000 7f3eb9fc5640 1 -- 172.21.15.57:0/2979778851 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eb406c420 msgr2=0x7f3eb41ef2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:22.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.219+0000 7f3eb9fc5640 1 --2- 172.21.15.57:0/2979778851 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eb406c420 0x7f3eb41ef2b0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400e960 tx=0x7f3ea400ee20 comp rx=0 tx=0).stop 2023-10-30T14:43:22.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.219+0000 7f3eb9fc5640 1 -- 172.21.15.57:0/2979778851 shutdown_connections 2023-10-30T14:43:22.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.219+0000 7f3eb9fc5640 1 --2- 172.21.15.57:0/2979778851 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3ea00664a0 0x7f3ea0068960 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:22.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.219+0000 7f3eb9fc5640 1 --2- 172.21.15.57:0/2979778851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3eb4153bb0 0x7f3eb414d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:22.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.219+0000 7f3eb9fc5640 1 --2- 172.21.15.57:0/2979778851 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3eb40c7170 0x7f3eb40c4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:22.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.219+0000 7f3eb9fc5640 1 --2- 172.21.15.57:0/2979778851 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eb406c420 0x7f3eb41ef2b0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:22.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.219+0000 7f3eb9fc5640 1 -- 172.21.15.57:0/2979778851 >> 172.21.15.57:0/2979778851 conn(0x7f3eb40cac70 msgr2=0x7f3eb40c6410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:22.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.219+0000 7f3eb9fc5640 1 -- 172.21.15.57:0/2979778851 shutdown_connections 2023-10-30T14:43:22.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:22.219+0000 7f3eb9fc5640 1 -- 172.21.15.57:0/2979778851 wait complete. 2023-10-30T14:43:22.245 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:43:24.246 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:24.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.395+0000 7fdbecc17640 1 Processor -- start 2023-10-30T14:43:24.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.395+0000 7fdbecc17640 1 -- start start 2023-10-30T14:43:24.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.399+0000 7fdbecc17640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fdbe81537f0 0x7fdbe80701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:24.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.399+0000 7fdbecc17640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdbe815a2e0 0x7fdbe80ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:24.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.399+0000 7fdbecc17640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe80c2aa0 con 0x7fdbe80cd4a0 2023-10-30T14:43:24.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.399+0000 7fdbecc17640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe80cd8b0 con 0x7fdbe81537f0 2023-10-30T14:43:24.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.399+0000 7fdbecc17640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe80c3ab0 con 0x7fdbe815a2e0 2023-10-30T14:43:24.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.399+0000 7fdbe659c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fdbe81537f0 0x7fdbe80701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:24.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.399+0000 7fdbe659c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fdbe81537f0 0x7fdbe80701a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37000/0 (socket says 172.21.15.57:37000) 2023-10-30T14:43:24.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.399+0000 7fdbe659c640 1 -- 172.21.15.57:0/417513218 learned_addr learned my addr 172.21.15.57:0/417513218 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:24.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.399+0000 7fdbdffff640 1 --2- 172.21.15.57:0/417513218 >> v2:172.21.15.88:3300/0 conn(0x7fdbe815a2e0 0x7fdbe80ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:24.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.399+0000 7fdbe659c640 1 -- 172.21.15.57:0/417513218 >> v2:172.21.15.88:3300/0 conn(0x7fdbe815a2e0 msgr2=0x7fdbe80ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:24.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.399+0000 7fdbe659c640 1 --2- 172.21.15.57:0/417513218 >> v2:172.21.15.88:3300/0 conn(0x7fdbe815a2e0 0x7fdbe80ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:24.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.399+0000 7fdbe659c640 1 -- 172.21.15.57:0/417513218 >> v1:172.21.15.88:6789/0 conn(0x7fdbe80cd4a0 legacy=0x7fdbe806b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:24.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.399+0000 7fdbe659c640 1 -- 172.21.15.57:0/417513218 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe80708e0 con 0x7fdbe81537f0 2023-10-30T14:43:24.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.403+0000 7fdbe659c640 1 --2- 172.21.15.57:0/417513218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbe81537f0 0x7fdbe80701a0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fdbe806de40 tx=0x7fdbd801ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab4da777c1274bf2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:24.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.403+0000 7fdbe5d9b640 1 -- 172.21.15.57:0/417513218 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd80a89b0 con 0x7fdbe81537f0 2023-10-30T14:43:24.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.403+0000 7fdbe5d9b640 1 -- 172.21.15.57:0/417513218 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd80a8b10 con 0x7fdbe81537f0 2023-10-30T14:43:24.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.403+0000 7fdbe5d9b640 1 -- 172.21.15.57:0/417513218 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd80a8ea0 con 0x7fdbe81537f0 2023-10-30T14:43:24.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.403+0000 7fdbecc17640 1 -- 172.21.15.57:0/417513218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbe81537f0 msgr2=0x7fdbe80701a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:24.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.403+0000 7fdbecc17640 1 --2- 172.21.15.57:0/417513218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbe81537f0 0x7fdbe80701a0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fdbe806de40 tx=0x7fdbd801ecb0 comp rx=0 tx=0).stop 2023-10-30T14:43:24.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.403+0000 7fdbecc17640 1 -- 172.21.15.57:0/417513218 shutdown_connections 2023-10-30T14:43:24.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.403+0000 7fdbecc17640 1 --2- 172.21.15.57:0/417513218 >> v2:172.21.15.88:3300/0 conn(0x7fdbe815a2e0 0x7fdbe80ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:24.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.403+0000 7fdbecc17640 1 --2- 172.21.15.57:0/417513218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbe81537f0 0x7fdbe80701a0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:24.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.403+0000 7fdbecc17640 1 -- 172.21.15.57:0/417513218 >> 172.21.15.57:0/417513218 conn(0x7fdbe8071fb0 msgr2=0x7fdbe80723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:24.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.403+0000 7fdbecc17640 1 -- 172.21.15.57:0/417513218 shutdown_connections 2023-10-30T14:43:24.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.403+0000 7fdbecc17640 1 -- 172.21.15.57:0/417513218 wait complete. 2023-10-30T14:43:24.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.403+0000 7fdbecc17640 1 Processor -- start 2023-10-30T14:43:24.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.403+0000 7fdbecc17640 1 -- start start 2023-10-30T14:43:24.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbecc17640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbe80cd4a0 0x7fdbe815b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:24.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbecc17640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbe81537f0 0x7fdbe815d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:24.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbecc17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbe815a2e0 0x7fdbe815d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:24.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbecc17640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe80c4be0 con 0x7fdbe80cd4a0 2023-10-30T14:43:24.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbecc17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe80c4ec0 con 0x7fdbe815a2e0 2023-10-30T14:43:24.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbecc17640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe80c51a0 con 0x7fdbe81537f0 2023-10-30T14:43:24.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbe659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbe80cd4a0 0x7fdbe815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:24.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbe659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbe80cd4a0 0x7fdbe815b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37008/0 (socket says 172.21.15.57:37008) 2023-10-30T14:43:24.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbe659c640 1 -- 172.21.15.57:0/3880776213 learned_addr learned my addr 172.21.15.57:0/3880776213 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:24.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbdffff640 1 --2- 172.21.15.57:0/3880776213 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbe81537f0 0x7fdbe815d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:24.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbe6d9d640 1 --2- 172.21.15.57:0/3880776213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbe815a2e0 0x7fdbe815d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:24.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbe659c640 1 -- 172.21.15.57:0/3880776213 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbe81537f0 msgr2=0x7fdbe815d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:24.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbe659c640 1 --2- 172.21.15.57:0/3880776213 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbe81537f0 0x7fdbe815d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:24.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbe659c640 1 -- 172.21.15.57:0/3880776213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbe815a2e0 msgr2=0x7fdbe815d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:24.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbe659c640 1 --2- 172.21.15.57:0/3880776213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbe815a2e0 0x7fdbe815d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:24.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbe659c640 1 -- 172.21.15.57:0/3880776213 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd80094d0 con 0x7fdbe80cd4a0 2023-10-30T14:43:24.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbdffff640 1 --2- 172.21.15.57:0/3880776213 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbe81537f0 0x7fdbe815d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:43:24.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbe6d9d640 1 --2- 172.21.15.57:0/3880776213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbe815a2e0 0x7fdbe815d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:43:24.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbe659c640 1 --2- 172.21.15.57:0/3880776213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbe80cd4a0 0x7fdbe815b430 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fdbd80b3040 tx=0x7fdbd801ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:24.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbdeffd640 1 -- 172.21.15.57:0/3880776213 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd80b0e80 con 0x7fdbe80cd4a0 2023-10-30T14:43:24.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbdeffd640 1 -- 172.21.15.57:0/3880776213 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd80bb480 con 0x7fdbe80cd4a0 2023-10-30T14:43:24.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbecc17640 1 -- 172.21.15.57:0/3880776213 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe80c5480 con 0x7fdbe80cd4a0 2023-10-30T14:43:24.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbdeffd640 1 -- 172.21.15.57:0/3880776213 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd80bb810 con 0x7fdbe80cd4a0 2023-10-30T14:43:24.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbecc17640 1 -- 172.21.15.57:0/3880776213 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe815cd90 con 0x7fdbe80cd4a0 2023-10-30T14:43:24.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbecc17640 1 -- 172.21.15.57:0/3880776213 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe80c5ad0 con 0x7fdbe80cd4a0 2023-10-30T14:43:24.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbdeffd640 1 -- 172.21.15.57:0/3880776213 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdbd80bb970 con 0x7fdbe80cd4a0 2023-10-30T14:43:24.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.407+0000 7fdbdeffd640 1 --2- 172.21.15.57:0/3880776213 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdbc80664a0 0x7fdbc8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:24.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.411+0000 7fdbdeffd640 1 -- 172.21.15.57:0/3880776213 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7fdbd8122ab0 con 0x7fdbe80cd4a0 2023-10-30T14:43:24.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.411+0000 7fdbdffff640 1 --2- 172.21.15.57:0/3880776213 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdbc80664a0 0x7fdbc8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:24.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.411+0000 7fdbdffff640 1 --2- 172.21.15.57:0/3880776213 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdbc80664a0 0x7fdbc8068960 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdbe81f3360 tx=0x7fdbd0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:24.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.411+0000 7fdbdeffd640 1 -- 172.21.15.57:0/3880776213 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdbe80c5ad0 con 0x7fdbe80cd4a0 2023-10-30T14:43:24.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.611+0000 7fdbecc17640 1 -- 172.21.15.57:0/3880776213 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbe80c5ad0 con 0x7fdbe80cd4a0 2023-10-30T14:43:24.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.611+0000 7fdbdeffd640 1 -- 172.21.15.57:0/3880776213 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+6534 (secure 0 0 0) 0x7fdbe80c5ad0 con 0x7fdbe80cd4a0 2023-10-30T14:43:24.619 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:24.619 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:22.245048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:24.620 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 159 2023-10-30T14:43:24.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.615+0000 7fdbecc17640 1 -- 172.21.15.57:0/3880776213 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdbc80664a0 msgr2=0x7fdbc8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:24.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.615+0000 7fdbecc17640 1 --2- 172.21.15.57:0/3880776213 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdbc80664a0 0x7fdbc8068960 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdbe81f3360 tx=0x7fdbd0008040 comp rx=0 tx=0).stop 2023-10-30T14:43:24.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.615+0000 7fdbecc17640 1 -- 172.21.15.57:0/3880776213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbe80cd4a0 msgr2=0x7fdbe815b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:24.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.615+0000 7fdbecc17640 1 --2- 172.21.15.57:0/3880776213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbe80cd4a0 0x7fdbe815b430 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fdbd80b3040 tx=0x7fdbd801ba50 comp rx=0 tx=0).stop 2023-10-30T14:43:24.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.615+0000 7fdbecc17640 1 -- 172.21.15.57:0/3880776213 shutdown_connections 2023-10-30T14:43:24.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.615+0000 7fdbecc17640 1 --2- 172.21.15.57:0/3880776213 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdbc80664a0 0x7fdbc8068960 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:24.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.615+0000 7fdbecc17640 1 --2- 172.21.15.57:0/3880776213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbe815a2e0 0x7fdbe815d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:24.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.615+0000 7fdbecc17640 1 --2- 172.21.15.57:0/3880776213 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdbe81537f0 0x7fdbe815d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:24.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.615+0000 7fdbecc17640 1 --2- 172.21.15.57:0/3880776213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdbe80cd4a0 0x7fdbe815b430 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:24.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.615+0000 7fdbecc17640 1 -- 172.21.15.57:0/3880776213 >> 172.21.15.57:0/3880776213 conn(0x7fdbe8071fb0 msgr2=0x7fdbe81591c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:24.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.615+0000 7fdbecc17640 1 -- 172.21.15.57:0/3880776213 shutdown_connections 2023-10-30T14:43:24.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:24.615+0000 7fdbecc17640 1 -- 172.21.15.57:0/3880776213 wait complete. 2023-10-30T14:43:24.640 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T14:43:24.641 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 159, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 159, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6356": { "addr": "172.21.15.88:6839/1744808283", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1744808283, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1744808283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6356, "incarnation": 151, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 45 }, "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 44 }, "gid_7056": { "addr": "172.21.15.57:6837/2307041430", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 2307041430, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 2307041430, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 7056, "incarnation": 137, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:stopping", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:43:22.245048+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6356, "mds_1": 6747, "mds_2": 7056 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.57:6834/697507447", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 697507447, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 697507447, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 156, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7377, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:43:26.641 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:26.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.791+0000 7ff5f2c09640 1 Processor -- start 2023-10-30T14:43:26.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5f2c09640 1 -- start start 2023-10-30T14:43:26.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5f2c09640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff5ec06c420 0x7ff5ec0c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:26.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5f2c09640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5ec0c5f80 con 0x7ff5ec0c7170 2023-10-30T14:43:26.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5f2c09640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5ec0c5690 con 0x7ff5ec153bb0 2023-10-30T14:43:26.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5f2c09640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5ec0c7580 con 0x7ff5ec06c420 2023-10-30T14:43:26.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5f11a6640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff5ec0c7170 0x7ff5ec159980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46772/0 (socket says 172.21.15.57:46772) 2023-10-30T14:43:26.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5f11a6640 1 -- 172.21.15.57:0/1560450117 learned_addr learned my addr 172.21.15.57:0/1560450117 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:43:26.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5ebfff640 1 --2- 172.21.15.57:0/1560450117 >> v2:172.21.15.88:3300/0 conn(0x7ff5ec06c420 0x7ff5ec0c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:26.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5eb7fe640 1 -- 172.21.15.57:0/1560450117 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998919333 0 0) 0x7ff5ec0c5f80 con 0x7ff5ec0c7170 2023-10-30T14:43:26.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5eb7fe640 1 -- 172.21.15.57:0/1560450117 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5dc003530 con 0x7ff5ec0c7170 2023-10-30T14:43:26.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5eb7fe640 1 -- 172.21.15.57:0/1560450117 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427422156 0 0) 0x7ff5ec0c5690 con 0x7ff5ec153bb0 2023-10-30T14:43:26.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5eb7fe640 1 -- 172.21.15.57:0/1560450117 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5ec0c5f80 con 0x7ff5ec153bb0 2023-10-30T14:43:26.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5eb7fe640 1 -- 172.21.15.57:0/1560450117 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1236784361 0 0) 0x7ff5dc003530 con 0x7ff5ec0c7170 2023-10-30T14:43:26.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5eb7fe640 1 -- 172.21.15.57:0/1560450117 >> v2:172.21.15.88:3300/0 conn(0x7ff5ec06c420 msgr2=0x7ff5ec0c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:26.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5eb7fe640 1 --2- 172.21.15.57:0/1560450117 >> v2:172.21.15.88:3300/0 conn(0x7ff5ec06c420 0x7ff5ec0c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:26.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5eb7fe640 1 -- 172.21.15.57:0/1560450117 >> v1:172.21.15.88:6789/0 conn(0x7ff5ec153bb0 legacy=0x7ff5ec0c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:26.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5eb7fe640 1 -- 172.21.15.57:0/1560450117 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ec15a580 con 0x7ff5ec0c7170 2023-10-30T14:43:26.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5eb7fe640 1 -- 172.21.15.57:0/1560450117 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7ff5e4001a00 con 0x7ff5ec0c7170 2023-10-30T14:43:26.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5eb7fe640 1 -- 172.21.15.57:0/1560450117 >> v1:172.21.15.57:6789/0 conn(0x7ff5ec0c7170 legacy=0x7ff5ec159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:26.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5eb7fe640 1 --2- 172.21.15.57:0/1560450117 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5dc003b10 0x7ff5dc003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:26.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5eb7fe640 1 --2- 172.21.15.57:0/1560450117 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5dc0047d0 0x7ff5dc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:26.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5eb7fe640 1 --2- 172.21.15.57:0/1560450117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5dc005190 0x7ff5dc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:26.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5eb7fe640 1 -- 172.21.15.57:0/1560450117 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ec0c7580 con 0x7ff5dc003b10 2023-10-30T14:43:26.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5eb7fe640 1 -- 172.21.15.57:0/1560450117 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc00bf80 con 0x7ff5dc005190 2023-10-30T14:43:26.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5eb7fe640 1 -- 172.21.15.57:0/1560450117 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc00c140 con 0x7ff5dc0047d0 2023-10-30T14:43:26.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5f09a5640 1 --2- 172.21.15.57:0/1560450117 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5dc003b10 0x7ff5dc003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:26.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5f11a6640 1 --2- 172.21.15.57:0/1560450117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5dc005190 0x7ff5dc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:26.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5ebfff640 1 --2- 172.21.15.57:0/1560450117 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5dc0047d0 0x7ff5dc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:26.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5f09a5640 1 -- 172.21.15.57:0/1560450117 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5dc0047d0 msgr2=0x7ff5dc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:26.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5f09a5640 1 --2- 172.21.15.57:0/1560450117 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5dc0047d0 0x7ff5dc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:26.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5f09a5640 1 -- 172.21.15.57:0/1560450117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5dc005190 msgr2=0x7ff5dc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:26.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5f09a5640 1 --2- 172.21.15.57:0/1560450117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5dc005190 0x7ff5dc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:26.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5f09a5640 1 -- 172.21.15.57:0/1560450117 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5dc003530 con 0x7ff5dc003b10 2023-10-30T14:43:26.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5f11a6640 1 --2- 172.21.15.57:0/1560450117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5dc005190 0x7ff5dc00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:43:26.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.795+0000 7ff5f11a6640 1 -- 172.21.15.57:0/1560450117 reap_dead start 2023-10-30T14:43:26.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.799+0000 7ff5f09a5640 1 --2- 172.21.15.57:0/1560450117 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5dc003b10 0x7ff5dc003f10 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0007fd0 tx=0x7ff5e001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28aeece845cbe017 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:26.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.799+0000 7ff5eb7fe640 1 -- 172.21.15.57:0/1560450117 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e00052d0 con 0x7ff5dc003b10 2023-10-30T14:43:26.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.799+0000 7ff5eb7fe640 1 -- 172.21.15.57:0/1560450117 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e0005450 con 0x7ff5dc003b10 2023-10-30T14:43:26.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.799+0000 7ff5eb7fe640 1 -- 172.21.15.57:0/1560450117 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e00a88f0 con 0x7ff5dc003b10 2023-10-30T14:43:26.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.799+0000 7ff5f2c09640 1 -- 172.21.15.57:0/1560450117 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5dc003b10 msgr2=0x7ff5dc003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:26.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.799+0000 7ff5f2c09640 1 --2- 172.21.15.57:0/1560450117 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5dc003b10 0x7ff5dc003f10 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0007fd0 tx=0x7ff5e001ec90 comp rx=0 tx=0).stop 2023-10-30T14:43:26.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.799+0000 7ff5f2c09640 1 -- 172.21.15.57:0/1560450117 shutdown_connections 2023-10-30T14:43:26.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.799+0000 7ff5f2c09640 1 --2- 172.21.15.57:0/1560450117 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5dc003b10 0x7ff5dc003f10 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:26.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.799+0000 7ff5f2c09640 1 -- 172.21.15.57:0/1560450117 >> 172.21.15.57:0/1560450117 conn(0x7ff5ec0cac70 msgr2=0x7ff5ec0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:26.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.799+0000 7ff5f2c09640 1 -- 172.21.15.57:0/1560450117 shutdown_connections 2023-10-30T14:43:26.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.799+0000 7ff5f2c09640 1 -- 172.21.15.57:0/1560450117 wait complete. 2023-10-30T14:43:26.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5f2c09640 1 Processor -- start 2023-10-30T14:43:26.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5f2c09640 1 -- start start 2023-10-30T14:43:26.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5f2c09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5dc003b10 0x7ff5ec1f0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:26.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5f2c09640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5ec0c31f0 0x7ff5ec1f2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:26.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5f2c09640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5ec14fa40 0x7ff5ec14d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:26.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5f2c09640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ec0c3630 con 0x7ff5ec0c31f0 2023-10-30T14:43:26.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5f2c09640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ec1f3130 con 0x7ff5dc003b10 2023-10-30T14:43:26.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5f2c09640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ec0c41e0 con 0x7ff5ec14fa40 2023-10-30T14:43:26.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5ebfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5ec0c31f0 0x7ff5ec1f2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:26.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5ebfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5ec0c31f0 0x7ff5ec1f2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37040/0 (socket says 172.21.15.57:37040) 2023-10-30T14:43:26.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5ebfff640 1 -- 172.21.15.57:0/1437647905 learned_addr learned my addr 172.21.15.57:0/1437647905 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:26.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5f09a5640 1 --2- 172.21.15.57:0/1437647905 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5dc003b10 0x7ff5ec1f0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:26.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5f11a6640 1 --2- 172.21.15.57:0/1437647905 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5ec14fa40 0x7ff5ec14d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:26.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5ebfff640 1 -- 172.21.15.57:0/1437647905 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5ec14fa40 msgr2=0x7ff5ec14d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:26.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5ebfff640 1 --2- 172.21.15.57:0/1437647905 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5ec14fa40 0x7ff5ec14d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:26.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5ebfff640 1 -- 172.21.15.57:0/1437647905 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5dc003b10 msgr2=0x7ff5ec1f0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:26.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5ebfff640 1 --2- 172.21.15.57:0/1437647905 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5dc003b10 0x7ff5ec1f0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:26.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5ebfff640 1 -- 172.21.15.57:0/1437647905 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ec0c44c0 con 0x7ff5ec0c31f0 2023-10-30T14:43:26.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5ebfff640 1 --2- 172.21.15.57:0/1437647905 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5ec0c31f0 0x7ff5ec1f2bf0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8011500 tx=0x7ff5d80119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:26.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5e97fa640 1 -- 172.21.15.57:0/1437647905 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8015870 con 0x7ff5ec0c31f0 2023-10-30T14:43:26.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5e97fa640 1 -- 172.21.15.57:0/1437647905 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d80159d0 con 0x7ff5ec0c31f0 2023-10-30T14:43:26.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5f2c09640 1 -- 172.21.15.57:0/1437647905 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5ec0c2710 con 0x7ff5ec0c31f0 2023-10-30T14:43:26.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5e97fa640 1 -- 172.21.15.57:0/1437647905 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8015d40 con 0x7ff5ec0c31f0 2023-10-30T14:43:26.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.803+0000 7ff5f2c09640 1 -- 172.21.15.57:0/1437647905 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5ec0c2cd0 con 0x7ff5ec0c31f0 2023-10-30T14:43:26.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.807+0000 7ff5f2c09640 1 -- 172.21.15.57:0/1437647905 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5ec04e170 con 0x7ff5ec0c31f0 2023-10-30T14:43:26.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.807+0000 7ff5e97fa640 1 -- 172.21.15.57:0/1437647905 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff5d8013400 con 0x7ff5ec0c31f0 2023-10-30T14:43:26.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.807+0000 7ff5e97fa640 1 --2- 172.21.15.57:0/1437647905 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff5d00664d0 0x7ff5d0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:26.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.807+0000 7ff5e97fa640 1 -- 172.21.15.57:0/1437647905 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7ff5d8084c70 con 0x7ff5ec0c31f0 2023-10-30T14:43:26.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.807+0000 7ff5f09a5640 1 --2- 172.21.15.57:0/1437647905 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff5d00664d0 0x7ff5d0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:26.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.807+0000 7ff5f09a5640 1 --2- 172.21.15.57:0/1437647905 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff5d00664d0 0x7ff5d0068990 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0007fd0 tx=0x7ff5e00aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:26.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:26.811+0000 7ff5e97fa640 1 -- 172.21.15.57:0/1437647905 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff5d8051b90 con 0x7ff5ec0c31f0 2023-10-30T14:43:27.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:27.003+0000 7ff5f2c09640 1 -- 172.21.15.57:0/1437647905 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5ec06c190 con 0x7ff5ec0c31f0 2023-10-30T14:43:27.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:27.007+0000 7ff5e97fa640 1 -- 172.21.15.57:0/1437647905 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+6534 (secure 0 0 0) 0x7ff5d80551b0 con 0x7ff5ec0c31f0 2023-10-30T14:43:27.011 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:27.011 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:22.245048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:27.011 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 159 2023-10-30T14:43:27.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:27.007+0000 7ff5f2c09640 1 -- 172.21.15.57:0/1437647905 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff5d00664d0 msgr2=0x7ff5d0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:27.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:27.007+0000 7ff5f2c09640 1 --2- 172.21.15.57:0/1437647905 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff5d00664d0 0x7ff5d0068990 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0007fd0 tx=0x7ff5e00aa070 comp rx=0 tx=0).stop 2023-10-30T14:43:27.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:27.007+0000 7ff5f2c09640 1 -- 172.21.15.57:0/1437647905 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5ec0c31f0 msgr2=0x7ff5ec1f2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:27.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:27.007+0000 7ff5f2c09640 1 --2- 172.21.15.57:0/1437647905 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5ec0c31f0 0x7ff5ec1f2bf0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8011500 tx=0x7ff5d80119c0 comp rx=0 tx=0).stop 2023-10-30T14:43:27.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:27.007+0000 7ff5f2c09640 1 -- 172.21.15.57:0/1437647905 shutdown_connections 2023-10-30T14:43:27.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:27.007+0000 7ff5f2c09640 1 --2- 172.21.15.57:0/1437647905 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff5d00664d0 0x7ff5d0068990 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:27.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:27.007+0000 7ff5f2c09640 1 --2- 172.21.15.57:0/1437647905 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5ec14fa40 0x7ff5ec14d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:27.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:27.007+0000 7ff5f2c09640 1 --2- 172.21.15.57:0/1437647905 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5ec0c31f0 0x7ff5ec1f2bf0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:27.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:27.007+0000 7ff5f2c09640 1 --2- 172.21.15.57:0/1437647905 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5dc003b10 0x7ff5ec1f0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:27.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:27.007+0000 7ff5f2c09640 1 -- 172.21.15.57:0/1437647905 >> 172.21.15.57:0/1437647905 conn(0x7ff5ec0cac70 msgr2=0x7ff5ec152aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:27.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:27.007+0000 7ff5f2c09640 1 -- 172.21.15.57:0/1437647905 shutdown_connections 2023-10-30T14:43:27.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:27.007+0000 7ff5f2c09640 1 -- 172.21.15.57:0/1437647905 wait complete. 2023-10-30T14:43:27.031 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:43:29.033 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:29.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.187+0000 7ff42a6bb640 1 Processor -- start 2023-10-30T14:43:29.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.187+0000 7ff42a6bb640 1 -- start start 2023-10-30T14:43:29.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.187+0000 7ff42a6bb640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff424153b50 0x7ff4241597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:29.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.187+0000 7ff42a6bb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff4240cb180 0x7ff42406ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:29.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.187+0000 7ff42a6bb640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff42415a770 con 0x7ff4240c62b0 2023-10-30T14:43:29.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.187+0000 7ff42a6bb640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff424159cf0 con 0x7ff424153b50 2023-10-30T14:43:29.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.187+0000 7ff42a6bb640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff42406aa30 con 0x7ff4240cb180 2023-10-30T14:43:29.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.187+0000 7ff428eb8640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff4240c62b0 0x7ff4240ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46788/0 (socket says 172.21.15.57:46788) 2023-10-30T14:43:29.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.187+0000 7ff428eb8640 1 -- 172.21.15.57:0/2859397311 learned_addr learned my addr 172.21.15.57:0/2859397311 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:43:29.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.187+0000 7ff4296b9640 1 --2- 172.21.15.57:0/2859397311 >> v2:172.21.15.57:3300/0 conn(0x7ff424153b50 0x7ff4241597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:29.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.187+0000 7ff429eba640 1 --2- 172.21.15.57:0/2859397311 >> v2:172.21.15.88:3300/0 conn(0x7ff4240cb180 0x7ff42406ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:29.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.187+0000 7ff41bfff640 1 -- 172.21.15.57:0/2859397311 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111530725 0 0) 0x7ff414001120 con 0x7ff4240c62b0 2023-10-30T14:43:29.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.187+0000 7ff41bfff640 1 -- 172.21.15.57:0/2859397311 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff410003530 con 0x7ff4240c62b0 2023-10-30T14:43:29.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.187+0000 7ff4296b9640 1 -- 172.21.15.57:0/2859397311 >> v2:172.21.15.88:3300/0 conn(0x7ff4240cb180 msgr2=0x7ff42406ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:29.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.187+0000 7ff4296b9640 1 --2- 172.21.15.57:0/2859397311 >> v2:172.21.15.88:3300/0 conn(0x7ff4240cb180 0x7ff42406ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:29.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.187+0000 7ff4296b9640 1 -- 172.21.15.57:0/2859397311 >> v1:172.21.15.57:6789/0 conn(0x7ff4240c62b0 legacy=0x7ff4240ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:29.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.187+0000 7ff4296b9640 1 -- 172.21.15.57:0/2859397311 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4240c9d40 con 0x7ff424153b50 2023-10-30T14:43:29.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.191+0000 7ff4296b9640 1 --2- 172.21.15.57:0/2859397311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff424153b50 0x7ff4241597b0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7ff4240c5d20 tx=0x7ff42001ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8fc59c4fae58311 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:29.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.191+0000 7ff41bfff640 1 -- 172.21.15.57:0/2859397311 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4200a8930 con 0x7ff424153b50 2023-10-30T14:43:29.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.191+0000 7ff41bfff640 1 -- 172.21.15.57:0/2859397311 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4200a8a90 con 0x7ff424153b50 2023-10-30T14:43:29.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.191+0000 7ff41bfff640 1 -- 172.21.15.57:0/2859397311 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4200a8e20 con 0x7ff424153b50 2023-10-30T14:43:29.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.191+0000 7ff42a6bb640 1 -- 172.21.15.57:0/2859397311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff424153b50 msgr2=0x7ff4241597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:29.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.191+0000 7ff42a6bb640 1 --2- 172.21.15.57:0/2859397311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff424153b50 0x7ff4241597b0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7ff4240c5d20 tx=0x7ff42001ec30 comp rx=0 tx=0).stop 2023-10-30T14:43:29.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.191+0000 7ff42a6bb640 1 -- 172.21.15.57:0/2859397311 shutdown_connections 2023-10-30T14:43:29.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.191+0000 7ff42a6bb640 1 --2- 172.21.15.57:0/2859397311 >> v2:172.21.15.88:3300/0 conn(0x7ff4240cb180 0x7ff42406ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:29.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff42a6bb640 1 --2- 172.21.15.57:0/2859397311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff424153b50 0x7ff4241597b0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:29.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff42a6bb640 1 -- 172.21.15.57:0/2859397311 >> 172.21.15.57:0/2859397311 conn(0x7ff424065a80 msgr2=0x7ff424065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:29.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff42a6bb640 1 -- 172.21.15.57:0/2859397311 shutdown_connections 2023-10-30T14:43:29.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff42a6bb640 1 -- 172.21.15.57:0/2859397311 wait complete. 2023-10-30T14:43:29.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff42a6bb640 1 Processor -- start 2023-10-30T14:43:29.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff42a6bb640 1 -- start start 2023-10-30T14:43:29.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff42a6bb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4240c62b0 0x7ff4241494f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:29.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff42a6bb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4240cb180 0x7ff4241504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:29.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff42a6bb640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff424153b50 0x7ff424150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:29.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff42a6bb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4240c4840 con 0x7ff4240c62b0 2023-10-30T14:43:29.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff42a6bb640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4241ef5c0 con 0x7ff4240cb180 2023-10-30T14:43:29.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff42a6bb640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff424149a30 con 0x7ff424153b50 2023-10-30T14:43:29.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff4296b9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4240c62b0 0x7ff4241494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:29.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff4296b9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4240c62b0 0x7ff4241494f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37054/0 (socket says 172.21.15.57:37054) 2023-10-30T14:43:29.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff4296b9640 1 -- 172.21.15.57:0/274205316 learned_addr learned my addr 172.21.15.57:0/274205316 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:29.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff429eba640 1 --2- 172.21.15.57:0/274205316 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff424153b50 0x7ff424150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:29.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff4296b9640 1 -- 172.21.15.57:0/274205316 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff424153b50 msgr2=0x7ff424150a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:29.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff4296b9640 1 --2- 172.21.15.57:0/274205316 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff424153b50 0x7ff424150a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:29.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff4296b9640 1 -- 172.21.15.57:0/274205316 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4240cb180 msgr2=0x7ff4241504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:29.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff4296b9640 1 --2- 172.21.15.57:0/274205316 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4240cb180 0x7ff4241504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:29.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff4296b9640 1 -- 172.21.15.57:0/274205316 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff420009420 con 0x7ff4240c62b0 2023-10-30T14:43:29.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.195+0000 7ff4296b9640 1 --2- 172.21.15.57:0/274205316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4240c62b0 0x7ff4241494f0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7ff4200b3040 tx=0x7ff42001b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:29.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.199+0000 7ff419ffb640 1 -- 172.21.15.57:0/274205316 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4200b9630 con 0x7ff4240c62b0 2023-10-30T14:43:29.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.199+0000 7ff419ffb640 1 -- 172.21.15.57:0/274205316 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4200b97b0 con 0x7ff4240c62b0 2023-10-30T14:43:29.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.199+0000 7ff419ffb640 1 -- 172.21.15.57:0/274205316 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4200b9b60 con 0x7ff4240c62b0 2023-10-30T14:43:29.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.199+0000 7ff42a6bb640 1 -- 172.21.15.57:0/274205316 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4241ee6f0 con 0x7ff4240c62b0 2023-10-30T14:43:29.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.199+0000 7ff42a6bb640 1 -- 172.21.15.57:0/274205316 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4241eec90 con 0x7ff4240c62b0 2023-10-30T14:43:29.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.199+0000 7ff42a6bb640 1 -- 172.21.15.57:0/274205316 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff424067540 con 0x7ff4240c62b0 2023-10-30T14:43:29.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.199+0000 7ff419ffb640 1 -- 172.21.15.57:0/274205316 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff4200b7040 con 0x7ff4240c62b0 2023-10-30T14:43:29.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.199+0000 7ff419ffb640 1 --2- 172.21.15.57:0/274205316 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff4080664d0 0x7ff408068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:29.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.199+0000 7ff419ffb640 1 -- 172.21.15.57:0/274205316 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7ff420122280 con 0x7ff4240c62b0 2023-10-30T14:43:29.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.199+0000 7ff428eb8640 1 --2- 172.21.15.57:0/274205316 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff4080664d0 0x7ff408068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:29.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.203+0000 7ff428eb8640 1 --2- 172.21.15.57:0/274205316 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff4080664d0 0x7ff408068990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff414005890 tx=0x7ff414001160 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:29.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.203+0000 7ff419ffb640 1 -- 172.21.15.57:0/274205316 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff4200ef330 con 0x7ff4240c62b0 2023-10-30T14:43:29.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.395+0000 7ff42a6bb640 1 -- 172.21.15.57:0/274205316 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4240c7140 con 0x7ff4240c62b0 2023-10-30T14:43:29.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.395+0000 7ff419ffb640 1 -- 172.21.15.57:0/274205316 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+6534 (secure 0 0 0) 0x7ff4200f2950 con 0x7ff4240c62b0 2023-10-30T14:43:29.401 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:29.402 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:22.245048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:29.403 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 159 2023-10-30T14:43:29.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.395+0000 7ff42a6bb640 1 -- 172.21.15.57:0/274205316 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff4080664d0 msgr2=0x7ff408068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:29.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.395+0000 7ff42a6bb640 1 --2- 172.21.15.57:0/274205316 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff4080664d0 0x7ff408068990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff414005890 tx=0x7ff414001160 comp rx=0 tx=0).stop 2023-10-30T14:43:29.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.395+0000 7ff42a6bb640 1 -- 172.21.15.57:0/274205316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4240c62b0 msgr2=0x7ff4241494f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:29.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.395+0000 7ff42a6bb640 1 --2- 172.21.15.57:0/274205316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4240c62b0 0x7ff4241494f0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7ff4200b3040 tx=0x7ff42001b9d0 comp rx=0 tx=0).stop 2023-10-30T14:43:29.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.399+0000 7ff42a6bb640 1 -- 172.21.15.57:0/274205316 shutdown_connections 2023-10-30T14:43:29.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.399+0000 7ff42a6bb640 1 --2- 172.21.15.57:0/274205316 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff4080664d0 0x7ff408068990 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:29.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.399+0000 7ff42a6bb640 1 --2- 172.21.15.57:0/274205316 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff424153b50 0x7ff424150a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:29.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.399+0000 7ff42a6bb640 1 --2- 172.21.15.57:0/274205316 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4240cb180 0x7ff4241504c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:29.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.399+0000 7ff42a6bb640 1 --2- 172.21.15.57:0/274205316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4240c62b0 0x7ff4241494f0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:29.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.399+0000 7ff42a6bb640 1 -- 172.21.15.57:0/274205316 >> 172.21.15.57:0/274205316 conn(0x7ff424065a80 msgr2=0x7ff4240c7660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:29.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.399+0000 7ff42a6bb640 1 -- 172.21.15.57:0/274205316 shutdown_connections 2023-10-30T14:43:29.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:29.399+0000 7ff42a6bb640 1 -- 172.21.15.57:0/274205316 wait complete. 2023-10-30T14:43:29.421 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:43:31.423 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:31.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.571+0000 7fb3bc217640 1 Processor -- start 2023-10-30T14:43:31.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.575+0000 7fb3bc217640 1 -- start start 2023-10-30T14:43:31.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.575+0000 7fb3bc217640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb3b406a0f0 0x7fb3b4153830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:31.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.575+0000 7fb3bc217640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b40c1e60 con 0x7fb3b40cbea0 2023-10-30T14:43:31.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.575+0000 7fb3bc217640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b40cb2b0 con 0x7fb3b406a4d0 2023-10-30T14:43:31.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.575+0000 7fb3bc217640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b415a3f0 con 0x7fb3b406a0f0 2023-10-30T14:43:31.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.575+0000 7fb3b9fb3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb3b406a0f0 0x7fb3b4153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:31.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.575+0000 7fb3ba7b4640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb3b40cbea0 0x7fb3b4164320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46790/0 (socket says 172.21.15.57:46790) 2023-10-30T14:43:31.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.575+0000 7fb3ba7b4640 1 -- 172.21.15.57:0/4178540895 learned_addr learned my addr 172.21.15.57:0/4178540895 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:43:31.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.575+0000 7fb3b9fb3640 1 -- 172.21.15.57:0/4178540895 >> v1:172.21.15.88:6789/0 conn(0x7fb3b406a4d0 legacy=0x7fb3b40cb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:31.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.575+0000 7fb3b9fb3640 1 -- 172.21.15.57:0/4178540895 >> v1:172.21.15.57:6789/0 conn(0x7fb3b40cbea0 legacy=0x7fb3b4164320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:31.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.575+0000 7fb3b9fb3640 1 -- 172.21.15.57:0/4178540895 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b415a8c0 con 0x7fb3b406a0f0 2023-10-30T14:43:31.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.579+0000 7fb3b9fb3640 1 --2- 172.21.15.57:0/4178540895 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b406a0f0 0x7fb3b4153830 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fb3b00097c0 tx=0x7fb3b001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63f186517b90441 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:31.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.579+0000 7fb3b8fb1640 1 -- 172.21.15.57:0/4178540895 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b00ab650 con 0x7fb3b406a0f0 2023-10-30T14:43:31.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.579+0000 7fb3b8fb1640 1 -- 172.21.15.57:0/4178540895 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b00ab7b0 con 0x7fb3b406a0f0 2023-10-30T14:43:31.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.579+0000 7fb3b8fb1640 1 -- 172.21.15.57:0/4178540895 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b00abb40 con 0x7fb3b406a0f0 2023-10-30T14:43:31.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.579+0000 7fb3bc217640 1 -- 172.21.15.57:0/4178540895 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b406a0f0 msgr2=0x7fb3b4153830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:31.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.579+0000 7fb3bc217640 1 --2- 172.21.15.57:0/4178540895 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b406a0f0 0x7fb3b4153830 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fb3b00097c0 tx=0x7fb3b001c450 comp rx=0 tx=0).stop 2023-10-30T14:43:31.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.579+0000 7fb3bc217640 1 -- 172.21.15.57:0/4178540895 shutdown_connections 2023-10-30T14:43:31.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.579+0000 7fb3bc217640 1 --2- 172.21.15.57:0/4178540895 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b406a0f0 0x7fb3b4153830 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:31.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.579+0000 7fb3bc217640 1 -- 172.21.15.57:0/4178540895 >> 172.21.15.57:0/4178540895 conn(0x7fb3b406de90 msgr2=0x7fb3b406e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:31.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.579+0000 7fb3bc217640 1 -- 172.21.15.57:0/4178540895 shutdown_connections 2023-10-30T14:43:31.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.579+0000 7fb3bc217640 1 -- 172.21.15.57:0/4178540895 wait complete. 2023-10-30T14:43:31.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.579+0000 7fb3bc217640 1 Processor -- start 2023-10-30T14:43:31.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.579+0000 7fb3bc217640 1 -- start start 2023-10-30T14:43:31.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3bc217640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b406a0f0 0x7fb3b4156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:31.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3bc217640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3b406a4d0 0x7fb3b40c38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:31.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3bc217640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3b40cbea0 0x7fb3b40c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:31.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3bc217640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b40c4a70 con 0x7fb3b406a0f0 2023-10-30T14:43:31.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3bc217640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b41f2ec0 con 0x7fb3b40cbea0 2023-10-30T14:43:31.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3bc217640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b41f3350 con 0x7fb3b406a4d0 2023-10-30T14:43:31.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3b9fb3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b406a0f0 0x7fb3b4156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:31.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3b9fb3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b406a0f0 0x7fb3b4156c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37062/0 (socket says 172.21.15.57:37062) 2023-10-30T14:43:31.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3b9fb3640 1 -- 172.21.15.57:0/2013303885 learned_addr learned my addr 172.21.15.57:0/2013303885 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:31.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3ba7b4640 1 --2- 172.21.15.57:0/2013303885 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3b40cbea0 0x7fb3b40c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:31.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3b97b2640 1 --2- 172.21.15.57:0/2013303885 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3b406a4d0 0x7fb3b40c38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:31.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3b9fb3640 1 -- 172.21.15.57:0/2013303885 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3b406a4d0 msgr2=0x7fb3b40c38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:31.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3b9fb3640 1 --2- 172.21.15.57:0/2013303885 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3b406a4d0 0x7fb3b40c38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:31.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3b9fb3640 1 -- 172.21.15.57:0/2013303885 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3b40cbea0 msgr2=0x7fb3b40c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:31.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3b9fb3640 1 --2- 172.21.15.57:0/2013303885 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3b40cbea0 0x7fb3b40c6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:31.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3b9fb3640 1 -- 172.21.15.57:0/2013303885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b0009420 con 0x7fb3b406a0f0 2023-10-30T14:43:31.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3ba7b4640 1 --2- 172.21.15.57:0/2013303885 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3b40cbea0 0x7fb3b40c6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:43:31.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3b9fb3640 1 --2- 172.21.15.57:0/2013303885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b406a0f0 0x7fb3b4156c20 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fb3b00ad040 tx=0x7fb3b001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:31.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3aa7fc640 1 -- 172.21.15.57:0/2013303885 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b00b5810 con 0x7fb3b406a0f0 2023-10-30T14:43:31.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3aa7fc640 1 -- 172.21.15.57:0/2013303885 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b00b5970 con 0x7fb3b406a0f0 2023-10-30T14:43:31.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3aa7fc640 1 -- 172.21.15.57:0/2013303885 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b00b5d20 con 0x7fb3b406a0f0 2023-10-30T14:43:31.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3bc217640 1 -- 172.21.15.57:0/2013303885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b4157d60 con 0x7fb3b406a0f0 2023-10-30T14:43:31.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3bc217640 1 -- 172.21.15.57:0/2013303885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b40c2a60 con 0x7fb3b406a0f0 2023-10-30T14:43:31.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3bc217640 1 -- 172.21.15.57:0/2013303885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b404e170 con 0x7fb3b406a0f0 2023-10-30T14:43:31.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.583+0000 7fb3aa7fc640 1 -- 172.21.15.57:0/2013303885 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb3b00bf410 con 0x7fb3b406a0f0 2023-10-30T14:43:31.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.587+0000 7fb3aa7fc640 1 --2- 172.21.15.57:0/2013303885 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb3980664f0 0x7fb3980689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:31.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.587+0000 7fb3aa7fc640 1 -- 172.21.15.57:0/2013303885 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7fb3b011f540 con 0x7fb3b406a0f0 2023-10-30T14:43:31.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.587+0000 7fb3b97b2640 1 --2- 172.21.15.57:0/2013303885 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb3980664f0 0x7fb3980689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:31.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.587+0000 7fb3b97b2640 1 --2- 172.21.15.57:0/2013303885 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb3980664f0 0x7fb3980689b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb3b40c7b00 tx=0x7fb3a4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:31.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.591+0000 7fb3aa7fc640 1 -- 172.21.15.57:0/2013303885 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb3b00ec390 con 0x7fb3b406a0f0 2023-10-30T14:43:31.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.783+0000 7fb3bc217640 1 -- 172.21.15.57:0/2013303885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3b40b07d0 con 0x7fb3b406a0f0 2023-10-30T14:43:31.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.783+0000 7fb3aa7fc640 1 -- 172.21.15.57:0/2013303885 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+6534 (secure 0 0 0) 0x7fb3b00ef9b0 con 0x7fb3b406a0f0 2023-10-30T14:43:31.789 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:31.789 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:22.245048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:31.789 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 159 2023-10-30T14:43:31.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.787+0000 7fb3bc217640 1 -- 172.21.15.57:0/2013303885 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb3980664f0 msgr2=0x7fb3980689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:31.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.787+0000 7fb3bc217640 1 --2- 172.21.15.57:0/2013303885 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb3980664f0 0x7fb3980689b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb3b40c7b00 tx=0x7fb3a4009040 comp rx=0 tx=0).stop 2023-10-30T14:43:31.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.787+0000 7fb3bc217640 1 -- 172.21.15.57:0/2013303885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b406a0f0 msgr2=0x7fb3b4156c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:31.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.787+0000 7fb3bc217640 1 --2- 172.21.15.57:0/2013303885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b406a0f0 0x7fb3b4156c20 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fb3b00ad040 tx=0x7fb3b001ac90 comp rx=0 tx=0).stop 2023-10-30T14:43:31.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.787+0000 7fb3bc217640 1 -- 172.21.15.57:0/2013303885 shutdown_connections 2023-10-30T14:43:31.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.787+0000 7fb3bc217640 1 --2- 172.21.15.57:0/2013303885 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb3980664f0 0x7fb3980689b0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:31.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.787+0000 7fb3bc217640 1 --2- 172.21.15.57:0/2013303885 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3b40cbea0 0x7fb3b40c6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:31.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.787+0000 7fb3bc217640 1 --2- 172.21.15.57:0/2013303885 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3b406a4d0 0x7fb3b40c38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:31.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.787+0000 7fb3bc217640 1 --2- 172.21.15.57:0/2013303885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b406a0f0 0x7fb3b4156c20 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:31.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.787+0000 7fb3bc217640 1 -- 172.21.15.57:0/2013303885 >> 172.21.15.57:0/2013303885 conn(0x7fb3b406de90 msgr2=0x7fb3b4073580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:31.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.787+0000 7fb3bc217640 1 -- 172.21.15.57:0/2013303885 shutdown_connections 2023-10-30T14:43:31.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:31.787+0000 7fb3bc217640 1 -- 172.21.15.57:0/2013303885 wait complete. 2023-10-30T14:43:31.810 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:43:33.811 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:33.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.959+0000 7fb073446640 1 Processor -- start 2023-10-30T14:43:33.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.959+0000 7fb073446640 1 -- start start 2023-10-30T14:43:33.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.963+0000 7fb073446640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb06c151c20 0x7fb06c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:33.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.963+0000 7fb073446640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb06c0c5030 con 0x7fb06c0cb8b0 2023-10-30T14:43:33.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.963+0000 7fb073446640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb06c157830 con 0x7fb06c151840 2023-10-30T14:43:33.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.963+0000 7fb073446640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06c157b10 con 0x7fb06c151c20 2023-10-30T14:43:33.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.963+0000 7fb0719e3640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb06c0cb8b0 0x7fb06c06b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:40084/0 (socket says 172.21.15.57:40084) 2023-10-30T14:43:33.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.963+0000 7fb0719e3640 1 -- 172.21.15.57:0/3204262669 learned_addr learned my addr 172.21.15.57:0/3204262669 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:43:33.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.963+0000 7fb0709e1640 1 --2- 172.21.15.57:0/3204262669 >> v2:172.21.15.88:3300/0 conn(0x7fb06c151c20 0x7fb06c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:33.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.963+0000 7fb063fff640 1 -- 172.21.15.57:0/3204262669 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228116300 0 0) 0x7fb06c0c5030 con 0x7fb06c0cb8b0 2023-10-30T14:43:33.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.963+0000 7fb063fff640 1 -- 172.21.15.57:0/3204262669 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb058003530 con 0x7fb06c0cb8b0 2023-10-30T14:43:33.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.963+0000 7fb063fff640 1 -- 172.21.15.57:0/3204262669 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2480692001 0 0) 0x7fb058003530 con 0x7fb06c0cb8b0 2023-10-30T14:43:33.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.963+0000 7fb0709e1640 1 -- 172.21.15.57:0/3204262669 >> v1:172.21.15.88:6789/0 conn(0x7fb06c151840 legacy=0x7fb06c072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:33.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.963+0000 7fb0709e1640 1 -- 172.21.15.57:0/3204262669 >> v1:172.21.15.57:6789/0 conn(0x7fb06c0cb8b0 legacy=0x7fb06c06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:33.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.963+0000 7fb0709e1640 1 -- 172.21.15.57:0/3204262669 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06c157df0 con 0x7fb06c151c20 2023-10-30T14:43:33.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.967+0000 7fb0709e1640 1 --2- 172.21.15.57:0/3204262669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb06c151c20 0x7fb06c0c49a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb05c0097c0 tx=0x7fb05c01eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=101f3e96260f2b8c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:33.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.967+0000 7fb063fff640 1 -- 172.21.15.57:0/3204262669 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb064001a00 con 0x7fb06c151c20 2023-10-30T14:43:33.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.967+0000 7fb063fff640 1 -- 172.21.15.57:0/3204262669 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05c0a8610 con 0x7fb06c151c20 2023-10-30T14:43:33.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.967+0000 7fb063fff640 1 -- 172.21.15.57:0/3204262669 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c0a8980 con 0x7fb06c151c20 2023-10-30T14:43:33.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.967+0000 7fb073446640 1 -- 172.21.15.57:0/3204262669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb06c151c20 msgr2=0x7fb06c0c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:33.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.967+0000 7fb073446640 1 --2- 172.21.15.57:0/3204262669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb06c151c20 0x7fb06c0c49a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb05c0097c0 tx=0x7fb05c01eb90 comp rx=0 tx=0).stop 2023-10-30T14:43:33.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.967+0000 7fb073446640 1 -- 172.21.15.57:0/3204262669 shutdown_connections 2023-10-30T14:43:33.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.967+0000 7fb073446640 1 --2- 172.21.15.57:0/3204262669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb06c151c20 0x7fb06c0c49a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:33.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.967+0000 7fb073446640 1 -- 172.21.15.57:0/3204262669 >> 172.21.15.57:0/3204262669 conn(0x7fb06c0cac70 msgr2=0x7fb06c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:33.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.967+0000 7fb073446640 1 -- 172.21.15.57:0/3204262669 shutdown_connections 2023-10-30T14:43:33.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.967+0000 7fb073446640 1 -- 172.21.15.57:0/3204262669 wait complete. 2023-10-30T14:43:33.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.967+0000 7fb073446640 1 Processor -- start 2023-10-30T14:43:33.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb073446640 1 -- start start 2023-10-30T14:43:33.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb073446640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb06c0cb8b0 0x7fb06c14dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:33.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb073446640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb06c151840 0x7fb06c1f2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:33.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb073446640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb06c151c20 0x7fb06c1ef470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:33.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb073446640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c1e36c0 con 0x7fb06c151c20 2023-10-30T14:43:33.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb073446640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c1e3120 con 0x7fb06c151840 2023-10-30T14:43:33.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb073446640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c1e33e0 con 0x7fb06c0cb8b0 2023-10-30T14:43:33.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb0719e3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb06c151c20 0x7fb06c1ef470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:33.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb0719e3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb06c151c20 0x7fb06c1ef470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38154/0 (socket says 172.21.15.57:38154) 2023-10-30T14:43:33.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb0719e3640 1 -- 172.21.15.57:0/1779807820 learned_addr learned my addr 172.21.15.57:0/1779807820 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:33.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb0709e1640 1 --2- 172.21.15.57:0/1779807820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb06c151840 0x7fb06c1f2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:33.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb0711e2640 1 --2- 172.21.15.57:0/1779807820 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb06c0cb8b0 0x7fb06c14dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:33.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb0719e3640 1 -- 172.21.15.57:0/1779807820 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb06c0cb8b0 msgr2=0x7fb06c14dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:33.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb0719e3640 1 --2- 172.21.15.57:0/1779807820 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb06c0cb8b0 0x7fb06c14dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:33.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb0719e3640 1 -- 172.21.15.57:0/1779807820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb06c151840 msgr2=0x7fb06c1f2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:33.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb0719e3640 1 --2- 172.21.15.57:0/1779807820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb06c151840 0x7fb06c1f2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:33.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb0719e3640 1 -- 172.21.15.57:0/1779807820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05c009420 con 0x7fb06c151c20 2023-10-30T14:43:33.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb0709e1640 1 --2- 172.21.15.57:0/1779807820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb06c151840 0x7fb06c1f2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:43:33.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb0719e3640 1 --2- 172.21.15.57:0/1779807820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb06c151c20 0x7fb06c1ef470 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fb064001840 tx=0x7fb06400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:33.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb061ffb640 1 -- 172.21.15.57:0/1779807820 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb064012400 con 0x7fb06c151c20 2023-10-30T14:43:33.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb061ffb640 1 -- 172.21.15.57:0/1779807820 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb064012560 con 0x7fb06c151c20 2023-10-30T14:43:33.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb073446640 1 -- 172.21.15.57:0/1779807820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb06c1e1f50 con 0x7fb06c151c20 2023-10-30T14:43:33.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb061ffb640 1 -- 172.21.15.57:0/1779807820 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb064012910 con 0x7fb06c151c20 2023-10-30T14:43:33.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb073446640 1 -- 172.21.15.57:0/1779807820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb06c1f27c0 con 0x7fb06c151c20 2023-10-30T14:43:33.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.971+0000 7fb073446640 1 -- 172.21.15.57:0/1779807820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06c1f29b0 con 0x7fb06c151c20 2023-10-30T14:43:33.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.975+0000 7fb061ffb640 1 -- 172.21.15.57:0/1779807820 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb064012a70 con 0x7fb06c151c20 2023-10-30T14:43:33.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.975+0000 7fb061ffb640 1 --2- 172.21.15.57:0/1779807820 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb0500664d0 0x7fb050068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:33.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.975+0000 7fb061ffb640 1 -- 172.21.15.57:0/1779807820 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7fb064084c00 con 0x7fb06c151c20 2023-10-30T14:43:33.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.975+0000 7fb0711e2640 1 --2- 172.21.15.57:0/1779807820 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb0500664d0 0x7fb050068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:33.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.975+0000 7fb0711e2640 1 --2- 172.21.15.57:0/1779807820 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb0500664d0 0x7fb050068990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb068007fd0 tx=0x7fb068009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:33.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:33.979+0000 7fb061ffb640 1 -- 172.21.15.57:0/1779807820 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb064051a60 con 0x7fb06c151c20 2023-10-30T14:43:34.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:34.171+0000 7fb073446640 1 -- 172.21.15.57:0/1779807820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb06c04e170 con 0x7fb06c151c20 2023-10-30T14:43:34.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:34.171+0000 7fb061ffb640 1 -- 172.21.15.57:0/1779807820 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+6534 (secure 0 0 0) 0x7fb064055080 con 0x7fb06c151c20 2023-10-30T14:43:34.178 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:34.179 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:22.245048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:34.180 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 159 2023-10-30T14:43:34.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:34.175+0000 7fb073446640 1 -- 172.21.15.57:0/1779807820 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb0500664d0 msgr2=0x7fb050068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:34.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:34.175+0000 7fb073446640 1 --2- 172.21.15.57:0/1779807820 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb0500664d0 0x7fb050068990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb068007fd0 tx=0x7fb068009040 comp rx=0 tx=0).stop 2023-10-30T14:43:34.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:34.175+0000 7fb073446640 1 -- 172.21.15.57:0/1779807820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb06c151c20 msgr2=0x7fb06c1ef470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:34.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:34.175+0000 7fb073446640 1 --2- 172.21.15.57:0/1779807820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb06c151c20 0x7fb06c1ef470 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fb064001840 tx=0x7fb06400a430 comp rx=0 tx=0).stop 2023-10-30T14:43:34.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:34.175+0000 7fb073446640 1 -- 172.21.15.57:0/1779807820 shutdown_connections 2023-10-30T14:43:34.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:34.175+0000 7fb073446640 1 --2- 172.21.15.57:0/1779807820 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb0500664d0 0x7fb050068990 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:34.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:34.175+0000 7fb073446640 1 --2- 172.21.15.57:0/1779807820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb06c151c20 0x7fb06c1ef470 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:34.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:34.175+0000 7fb073446640 1 --2- 172.21.15.57:0/1779807820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb06c151840 0x7fb06c1f2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:34.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:34.175+0000 7fb073446640 1 --2- 172.21.15.57:0/1779807820 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb06c0cb8b0 0x7fb06c14dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:34.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:34.175+0000 7fb073446640 1 -- 172.21.15.57:0/1779807820 >> 172.21.15.57:0/1779807820 conn(0x7fb06c0cac70 msgr2=0x7fb06c1e47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:34.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:34.175+0000 7fb073446640 1 -- 172.21.15.57:0/1779807820 shutdown_connections 2023-10-30T14:43:34.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:34.175+0000 7fb073446640 1 -- 172.21.15.57:0/1779807820 wait complete. 2023-10-30T14:43:34.200 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:43:36.201 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:36.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.347+0000 7f068998d640 1 Processor -- start 2023-10-30T14:43:36.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.347+0000 7f068998d640 1 -- start start 2023-10-30T14:43:36.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f068998d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f068406c420 0x7f06840c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:36.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f068998d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f06840c7170 0x7f0684159980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:36.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f068998d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06840c5f80 con 0x7f0684153bb0 2023-10-30T14:43:36.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f068998d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06840c5690 con 0x7f06840c7170 2023-10-30T14:43:36.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f068998d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068415a580 con 0x7f068406c420 2023-10-30T14:43:36.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f0682ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f0684153bb0 0x7f06840c6410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:40088/0 (socket says 172.21.15.57:40088) 2023-10-30T14:43:36.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f0682ffd640 1 -- 172.21.15.57:0/765325498 learned_addr learned my addr 172.21.15.57:0/765325498 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:43:36.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f06837fe640 1 --2- 172.21.15.57:0/765325498 >> v2:172.21.15.88:3300/0 conn(0x7f06840c7170 0x7f0684159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:36.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f067bfff640 1 --2- 172.21.15.57:0/765325498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f068406c420 0x7f06840c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:36.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f06827fc640 1 -- 172.21.15.57:0/765325498 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238755204 0 0) 0x7f06840c5f80 con 0x7f0684153bb0 2023-10-30T14:43:36.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f06827fc640 1 -- 172.21.15.57:0/765325498 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0670003530 con 0x7f0684153bb0 2023-10-30T14:43:36.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f06827fc640 1 -- 172.21.15.57:0/765325498 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3292992744 0 0) 0x7f0670003530 con 0x7f0684153bb0 2023-10-30T14:43:36.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f06827fc640 1 -- 172.21.15.57:0/765325498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f068406c420 msgr2=0x7f06840c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:36.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f06827fc640 1 --2- 172.21.15.57:0/765325498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f068406c420 0x7f06840c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:36.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f06827fc640 1 -- 172.21.15.57:0/765325498 >> v2:172.21.15.88:3300/0 conn(0x7f06840c7170 msgr2=0x7f0684159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:36.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f06827fc640 1 --2- 172.21.15.57:0/765325498 >> v2:172.21.15.88:3300/0 conn(0x7f06840c7170 0x7f0684159980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:36.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f06827fc640 1 -- 172.21.15.57:0/765325498 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06840c5930 con 0x7f0684153bb0 2023-10-30T14:43:36.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f06827fc640 1 -- 172.21.15.57:0/765325498 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f0674001680 con 0x7f0684153bb0 2023-10-30T14:43:36.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f06827fc640 1 -- 172.21.15.57:0/765325498 >> v1:172.21.15.57:6789/0 conn(0x7f0684153bb0 legacy=0x7f06840c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:36.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f06827fc640 1 --2- 172.21.15.57:0/765325498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0670003980 0x7f0670003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:36.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f06827fc640 1 --2- 172.21.15.57:0/765325498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06700047d0 0x7f0670004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:36.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f06827fc640 1 --2- 172.21.15.57:0/765325498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f068406c420 0x7f0670005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:36.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f06827fc640 1 -- 172.21.15.57:0/765325498 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06840c5690 con 0x7f06700047d0 2023-10-30T14:43:36.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f06827fc640 1 -- 172.21.15.57:0/765325498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068415a580 con 0x7f0670003980 2023-10-30T14:43:36.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f06827fc640 1 -- 172.21.15.57:0/765325498 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067000b990 con 0x7f068406c420 2023-10-30T14:43:36.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f067bfff640 1 --2- 172.21.15.57:0/765325498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06700047d0 0x7f0670004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:36.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f0682ffd640 1 --2- 172.21.15.57:0/765325498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0670003980 0x7f0670003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:36.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f06837fe640 1 --2- 172.21.15.57:0/765325498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f068406c420 0x7f0670005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:36.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f067bfff640 1 -- 172.21.15.57:0/765325498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f068406c420 msgr2=0x7f0670005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:36.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f067bfff640 1 --2- 172.21.15.57:0/765325498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f068406c420 0x7f0670005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:36.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f067bfff640 1 -- 172.21.15.57:0/765325498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0670003980 msgr2=0x7f0670003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:36.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f067bfff640 1 --2- 172.21.15.57:0/765325498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0670003980 0x7f0670003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:36.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.351+0000 7f067bfff640 1 -- 172.21.15.57:0/765325498 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0670003530 con 0x7f06700047d0 2023-10-30T14:43:36.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.355+0000 7f067bfff640 1 --2- 172.21.15.57:0/765325498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06700047d0 0x7f0670004c50 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f066c009ae0 tx=0x7f066c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd09933c8b45067 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:36.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.355+0000 7f06827fc640 1 -- 172.21.15.57:0/765325498 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c0a5560 con 0x7f06700047d0 2023-10-30T14:43:36.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.355+0000 7f06827fc640 1 -- 172.21.15.57:0/765325498 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066c0a56c0 con 0x7f06700047d0 2023-10-30T14:43:36.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.355+0000 7f06827fc640 1 -- 172.21.15.57:0/765325498 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c0a5a70 con 0x7f06700047d0 2023-10-30T14:43:36.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.355+0000 7f068998d640 1 -- 172.21.15.57:0/765325498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06700047d0 msgr2=0x7f0670004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:36.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.355+0000 7f068998d640 1 --2- 172.21.15.57:0/765325498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06700047d0 0x7f0670004c50 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f066c009ae0 tx=0x7f066c01ca50 comp rx=0 tx=0).stop 2023-10-30T14:43:36.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.355+0000 7f06837fe640 1 -- 172.21.15.57:0/765325498 reap_dead start 2023-10-30T14:43:36.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.355+0000 7f068998d640 1 -- 172.21.15.57:0/765325498 shutdown_connections 2023-10-30T14:43:36.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.355+0000 7f068998d640 1 -- 172.21.15.57:0/765325498 >> 172.21.15.57:0/765325498 conn(0x7f06840cac70 msgr2=0x7f06840cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:36.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.355+0000 7f068998d640 1 -- 172.21.15.57:0/765325498 shutdown_connections 2023-10-30T14:43:36.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.355+0000 7f068998d640 1 -- 172.21.15.57:0/765325498 wait complete. 2023-10-30T14:43:36.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.355+0000 7f068998d640 1 Processor -- start 2023-10-30T14:43:36.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f068998d640 1 -- start start 2023-10-30T14:43:36.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f068998d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06700047d0 0x7f06841eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:36.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f068998d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06840c1200 0x7f06841f1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:36.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f068998d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06840c8b50 0x7f06841f15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:36.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f068998d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06841592c0 con 0x7f06840c8b50 2023-10-30T14:43:36.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f068998d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06841ef560 con 0x7f06700047d0 2023-10-30T14:43:36.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f068998d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06841597f0 con 0x7f06840c1200 2023-10-30T14:43:36.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f06837fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06840c8b50 0x7f06841f15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:36.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f06837fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06840c8b50 0x7f06841f15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38174/0 (socket says 172.21.15.57:38174) 2023-10-30T14:43:36.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f06837fe640 1 -- 172.21.15.57:0/483222843 learned_addr learned my addr 172.21.15.57:0/483222843 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:36.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f067bfff640 1 --2- 172.21.15.57:0/483222843 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06840c1200 0x7f06841f1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:36.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f0682ffd640 1 --2- 172.21.15.57:0/483222843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06700047d0 0x7f06841eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:36.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f06837fe640 1 -- 172.21.15.57:0/483222843 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06840c1200 msgr2=0x7f06841f1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:36.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f06837fe640 1 --2- 172.21.15.57:0/483222843 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06840c1200 0x7f06841f1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:36.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f06837fe640 1 -- 172.21.15.57:0/483222843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06700047d0 msgr2=0x7f06841eebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:36.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f06837fe640 1 --2- 172.21.15.57:0/483222843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06700047d0 0x7f06841eebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:36.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f06837fe640 1 -- 172.21.15.57:0/483222843 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0684159510 con 0x7f06840c8b50 2023-10-30T14:43:36.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f06837fe640 1 --2- 172.21.15.57:0/483222843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06840c8b50 0x7f06841f15a0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f068406fc60 tx=0x7f067c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:36.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f0680ff9640 1 -- 172.21.15.57:0/483222843 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c00e070 con 0x7f06840c8b50 2023-10-30T14:43:36.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f0680ff9640 1 -- 172.21.15.57:0/483222843 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067c011e10 con 0x7f06840c8b50 2023-10-30T14:43:36.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f068998d640 1 -- 172.21.15.57:0/483222843 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06841f0860 con 0x7f06840c8b50 2023-10-30T14:43:36.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f0680ff9640 1 -- 172.21.15.57:0/483222843 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c0039a0 con 0x7f06840c8b50 2023-10-30T14:43:36.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f068998d640 1 -- 172.21.15.57:0/483222843 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06840c6d40 con 0x7f06840c8b50 2023-10-30T14:43:36.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.359+0000 7f068998d640 1 -- 172.21.15.57:0/483222843 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f068404e170 con 0x7f06840c8b50 2023-10-30T14:43:36.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.363+0000 7f0680ff9640 1 -- 172.21.15.57:0/483222843 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f067c003b00 con 0x7f06840c8b50 2023-10-30T14:43:36.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.363+0000 7f0680ff9640 1 --2- 172.21.15.57:0/483222843 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06640664d0 0x7f0664068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:36.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.363+0000 7f0680ff9640 1 -- 172.21.15.57:0/483222843 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f067c084a80 con 0x7f06840c8b50 2023-10-30T14:43:36.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.363+0000 7f0682ffd640 1 --2- 172.21.15.57:0/483222843 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06640664d0 0x7f0664068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:36.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.363+0000 7f0682ffd640 1 --2- 172.21.15.57:0/483222843 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06640664d0 0x7f0664068990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f0674001060 tx=0x7f0674007040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:36.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.367+0000 7f0680ff9640 1 -- 172.21.15.57:0/483222843 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f067c051990 con 0x7f06840c8b50 2023-10-30T14:43:36.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.559+0000 7f068998d640 1 -- 172.21.15.57:0/483222843 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06840ca350 con 0x7f06840c8b50 2023-10-30T14:43:36.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.559+0000 7f0680ff9640 1 -- 172.21.15.57:0/483222843 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+6534 (secure 0 0 0) 0x7f067c054fb0 con 0x7f06840c8b50 2023-10-30T14:43:36.564 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:36.564 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:22.245048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:36.565 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 159 2023-10-30T14:43:36.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.559+0000 7f068998d640 1 -- 172.21.15.57:0/483222843 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06640664d0 msgr2=0x7f0664068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:36.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.559+0000 7f068998d640 1 --2- 172.21.15.57:0/483222843 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06640664d0 0x7f0664068990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f0674001060 tx=0x7f0674007040 comp rx=0 tx=0).stop 2023-10-30T14:43:36.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.559+0000 7f068998d640 1 -- 172.21.15.57:0/483222843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06840c8b50 msgr2=0x7f06841f15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:36.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.559+0000 7f068998d640 1 --2- 172.21.15.57:0/483222843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06840c8b50 0x7f06841f15a0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f068406fc60 tx=0x7f067c0115a0 comp rx=0 tx=0).stop 2023-10-30T14:43:36.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.559+0000 7f068998d640 1 -- 172.21.15.57:0/483222843 shutdown_connections 2023-10-30T14:43:36.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.559+0000 7f068998d640 1 --2- 172.21.15.57:0/483222843 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f06640664d0 0x7f0664068990 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:36.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.559+0000 7f068998d640 1 --2- 172.21.15.57:0/483222843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06840c8b50 0x7f06841f15a0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:36.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.559+0000 7f068998d640 1 --2- 172.21.15.57:0/483222843 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f06840c1200 0x7f06841f1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:36.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.559+0000 7f068998d640 1 --2- 172.21.15.57:0/483222843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06700047d0 0x7f06841eebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:36.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.559+0000 7f068998d640 1 -- 172.21.15.57:0/483222843 >> 172.21.15.57:0/483222843 conn(0x7f06840cac70 msgr2=0x7f06840719f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:36.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.559+0000 7f068998d640 1 -- 172.21.15.57:0/483222843 shutdown_connections 2023-10-30T14:43:36.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:36.559+0000 7f068998d640 1 -- 172.21.15.57:0/483222843 wait complete. 2023-10-30T14:43:36.585 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T14:43:36.586 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 159, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 159, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6356": { "addr": "172.21.15.88:6839/1744808283", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1744808283, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1744808283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6356, "incarnation": 151, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 45 }, "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 44 }, "gid_7056": { "addr": "172.21.15.57:6837/2307041430", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 2307041430, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 2307041430, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 7056, "incarnation": 137, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:stopping", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:43:22.245048+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6356, "mds_1": 6747, "mds_2": 7056 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.57:6834/697507447", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 697507447, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 697507447, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 156, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7377, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:43:38.587 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:38.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.735+0000 7f1c74a79640 1 Processor -- start 2023-10-30T14:43:38.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.739+0000 7f1c74a79640 1 -- start start 2023-10-30T14:43:38.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.739+0000 7f1c74a79640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1c70159760 0x7f1c7015d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:38.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.739+0000 7f1c74a79640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c7015da00 0x7f1c70069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:38.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.739+0000 7f1c74a79640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c7015cd50 con 0x7f1c700c5c10 2023-10-30T14:43:38.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.739+0000 7f1c74a79640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c7015d0a0 con 0x7f1c70159760 2023-10-30T14:43:38.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.739+0000 7f1c74a79640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7006f600 con 0x7f1c7015da00 2023-10-30T14:43:38.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.739+0000 7f1c6e59c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1c700c5c10 0x7f1c7006eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:40098/0 (socket says 172.21.15.57:40098) 2023-10-30T14:43:38.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.739+0000 7f1c6e59c640 1 -- 172.21.15.57:0/3961231224 learned_addr learned my addr 172.21.15.57:0/3961231224 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:43:38.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.739+0000 7f1c6ed9d640 1 --2- 172.21.15.57:0/3961231224 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c7015da00 0x7f1c70069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:38.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.739+0000 7f1c67fff640 1 --2- 172.21.15.57:0/3961231224 >> v2:172.21.15.88:3300/0 conn(0x7f1c70159760 0x7f1c7015d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:38.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.739+0000 7f1c6dd9b640 1 -- 172.21.15.57:0/3961231224 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082472861 0 0) 0x7f1c7015cd50 con 0x7f1c700c5c10 2023-10-30T14:43:38.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.739+0000 7f1c6dd9b640 1 -- 172.21.15.57:0/3961231224 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c5c003530 con 0x7f1c700c5c10 2023-10-30T14:43:38.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.739+0000 7f1c67fff640 1 -- 172.21.15.57:0/3961231224 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c7015da00 msgr2=0x7f1c70069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:38.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.739+0000 7f1c67fff640 1 --2- 172.21.15.57:0/3961231224 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c7015da00 0x7f1c70069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:38.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.739+0000 7f1c67fff640 1 -- 172.21.15.57:0/3961231224 >> v1:172.21.15.57:6789/0 conn(0x7f1c700c5c10 legacy=0x7f1c7006eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:38.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.739+0000 7f1c67fff640 1 -- 172.21.15.57:0/3961231224 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c700ca000 con 0x7f1c70159760 2023-10-30T14:43:38.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.743+0000 7f1c67fff640 1 --2- 172.21.15.57:0/3961231224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c70159760 0x7f1c7015d4c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1c580097c0 tx=0x7f1c5801c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6e8be2e989c2176 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:38.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.743+0000 7f1c6dd9b640 1 -- 172.21.15.57:0/3961231224 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c580ad660 con 0x7f1c70159760 2023-10-30T14:43:38.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.743+0000 7f1c6dd9b640 1 -- 172.21.15.57:0/3961231224 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c580ad7c0 con 0x7f1c70159760 2023-10-30T14:43:38.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.743+0000 7f1c6dd9b640 1 -- 172.21.15.57:0/3961231224 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c580adb30 con 0x7f1c70159760 2023-10-30T14:43:38.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.743+0000 7f1c74a79640 1 -- 172.21.15.57:0/3961231224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c70159760 msgr2=0x7f1c7015d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:38.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.743+0000 7f1c74a79640 1 --2- 172.21.15.57:0/3961231224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c70159760 0x7f1c7015d4c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1c580097c0 tx=0x7f1c5801c450 comp rx=0 tx=0).stop 2023-10-30T14:43:38.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.743+0000 7f1c74a79640 1 -- 172.21.15.57:0/3961231224 shutdown_connections 2023-10-30T14:43:38.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.743+0000 7f1c74a79640 1 --2- 172.21.15.57:0/3961231224 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c7015da00 0x7f1c70069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:38.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.743+0000 7f1c74a79640 1 --2- 172.21.15.57:0/3961231224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c70159760 0x7f1c7015d4c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:38.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.743+0000 7f1c74a79640 1 -- 172.21.15.57:0/3961231224 >> 172.21.15.57:0/3961231224 conn(0x7f1c7006afa0 msgr2=0x7f1c7006b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:38.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.743+0000 7f1c74a79640 1 -- 172.21.15.57:0/3961231224 shutdown_connections 2023-10-30T14:43:38.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.743+0000 7f1c74a79640 1 -- 172.21.15.57:0/3961231224 wait complete. 2023-10-30T14:43:38.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c74a79640 1 Processor -- start 2023-10-30T14:43:38.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c74a79640 1 -- start start 2023-10-30T14:43:38.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c74a79640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c700c5c10 0x7f1c701f2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:38.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c74a79640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c70159760 0x7f1c7014fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:38.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c74a79640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c7015da00 0x7f1c7015b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:38.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c74a79640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7015c8f0 con 0x7f1c7015da00 2023-10-30T14:43:38.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c74a79640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c701f1880 con 0x7f1c700c5c10 2023-10-30T14:43:38.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c74a79640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c701f27b0 con 0x7f1c70159760 2023-10-30T14:43:38.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c6ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c7015da00 0x7f1c7015b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:38.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c6ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c7015da00 0x7f1c7015b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38190/0 (socket says 172.21.15.57:38190) 2023-10-30T14:43:38.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c6ed9d640 1 -- 172.21.15.57:0/2598156036 learned_addr learned my addr 172.21.15.57:0/2598156036 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:38.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c6e59c640 1 --2- 172.21.15.57:0/2598156036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c700c5c10 0x7f1c701f2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:38.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c67fff640 1 --2- 172.21.15.57:0/2598156036 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c70159760 0x7f1c7014fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:38.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c6ed9d640 1 -- 172.21.15.57:0/2598156036 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c70159760 msgr2=0x7f1c7014fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:38.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c6ed9d640 1 --2- 172.21.15.57:0/2598156036 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c70159760 0x7f1c7014fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:38.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c6ed9d640 1 -- 172.21.15.57:0/2598156036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c700c5c10 msgr2=0x7f1c701f2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:38.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c6ed9d640 1 --2- 172.21.15.57:0/2598156036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c700c5c10 0x7f1c701f2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:38.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c6ed9d640 1 -- 172.21.15.57:0/2598156036 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c58009420 con 0x7f1c7015da00 2023-10-30T14:43:38.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c6ed9d640 1 --2- 172.21.15.57:0/2598156036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c7015da00 0x7f1c7015b410 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800ec10 tx=0x7f1c6800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:38.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c66ffd640 1 -- 172.21.15.57:0/2598156036 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c68009a70 con 0x7f1c7015da00 2023-10-30T14:43:38.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c66ffd640 1 -- 172.21.15.57:0/2598156036 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6800f040 con 0x7f1c7015da00 2023-10-30T14:43:38.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c66ffd640 1 -- 172.21.15.57:0/2598156036 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c68014480 con 0x7f1c7015da00 2023-10-30T14:43:38.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c74a79640 1 -- 172.21.15.57:0/2598156036 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c70152ac0 con 0x7f1c7015da00 2023-10-30T14:43:38.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c74a79640 1 -- 172.21.15.57:0/2598156036 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c701f1a70 con 0x7f1c7015da00 2023-10-30T14:43:38.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.747+0000 7f1c74a79640 1 -- 172.21.15.57:0/2598156036 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c701f5d20 con 0x7f1c7015da00 2023-10-30T14:43:38.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.751+0000 7f1c66ffd640 1 -- 172.21.15.57:0/2598156036 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1c680147f0 con 0x7f1c7015da00 2023-10-30T14:43:38.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.751+0000 7f1c66ffd640 1 --2- 172.21.15.57:0/2598156036 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1c500664d0 0x7f1c50068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:38.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.751+0000 7f1c66ffd640 1 -- 172.21.15.57:0/2598156036 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f1c68085340 con 0x7f1c7015da00 2023-10-30T14:43:38.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.751+0000 7f1c6e59c640 1 --2- 172.21.15.57:0/2598156036 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1c500664d0 0x7f1c50068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:38.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.751+0000 7f1c6e59c640 1 --2- 172.21.15.57:0/2598156036 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1c500664d0 0x7f1c50068990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f1c60007fd0 tx=0x7f1c60009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:38.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.755+0000 7f1c66ffd640 1 -- 172.21.15.57:0/2598156036 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1c680521c0 con 0x7f1c7015da00 2023-10-30T14:43:38.865 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:43:38.869 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:43:38.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.943+0000 7f1c74a79640 1 -- 172.21.15.57:0/2598156036 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c7004e170 con 0x7f1c7015da00 2023-10-30T14:43:38.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.943+0000 7f1c66ffd640 1 -- 172.21.15.57:0/2598156036 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+6534 (secure 0 0 0) 0x7f1c68018090 con 0x7f1c7015da00 2023-10-30T14:43:38.949 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:38.949 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:22.245048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:38.950 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 159 2023-10-30T14:43:38.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.947+0000 7f1c74a79640 1 -- 172.21.15.57:0/2598156036 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1c500664d0 msgr2=0x7f1c50068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:38.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.947+0000 7f1c74a79640 1 --2- 172.21.15.57:0/2598156036 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1c500664d0 0x7f1c50068990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f1c60007fd0 tx=0x7f1c60009040 comp rx=0 tx=0).stop 2023-10-30T14:43:38.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.947+0000 7f1c74a79640 1 -- 172.21.15.57:0/2598156036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c7015da00 msgr2=0x7f1c7015b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:38.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.947+0000 7f1c74a79640 1 --2- 172.21.15.57:0/2598156036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c7015da00 0x7f1c7015b410 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800ec10 tx=0x7f1c6800c490 comp rx=0 tx=0).stop 2023-10-30T14:43:38.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.947+0000 7f1c74a79640 1 -- 172.21.15.57:0/2598156036 shutdown_connections 2023-10-30T14:43:38.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.947+0000 7f1c74a79640 1 --2- 172.21.15.57:0/2598156036 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1c500664d0 0x7f1c50068990 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:38.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.947+0000 7f1c74a79640 1 --2- 172.21.15.57:0/2598156036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c7015da00 0x7f1c7015b410 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:38.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.947+0000 7f1c74a79640 1 --2- 172.21.15.57:0/2598156036 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c70159760 0x7f1c7014fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:38.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.947+0000 7f1c74a79640 1 --2- 172.21.15.57:0/2598156036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c700c5c10 0x7f1c701f2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:38.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.947+0000 7f1c74a79640 1 -- 172.21.15.57:0/2598156036 >> 172.21.15.57:0/2598156036 conn(0x7f1c7006afa0 msgr2=0x7f1c700c4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:38.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.947+0000 7f1c74a79640 1 -- 172.21.15.57:0/2598156036 shutdown_connections 2023-10-30T14:43:38.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:38.947+0000 7f1c74a79640 1 -- 172.21.15.57:0/2598156036 wait complete. 2023-10-30T14:43:38.971 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:43:40.972 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:41.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95e0468640 1 Processor -- start 2023-10-30T14:43:41.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95e0468640 1 -- start start 2023-10-30T14:43:41.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95e0468640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f95d8070520 0x7f95d8070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:41.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95e0468640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d80c2310 con 0x7f95d8150830 2023-10-30T14:43:41.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95e0468640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d8156710 con 0x7f95d80cb7d0 2023-10-30T14:43:41.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95e0468640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95d806bc10 con 0x7f95d8070520 2023-10-30T14:43:41.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95de204640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f95d8150830 0x7f95d806b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:40114/0 (socket says 172.21.15.57:40114) 2023-10-30T14:43:41.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95de204640 1 -- 172.21.15.57:0/2285124861 learned_addr learned my addr 172.21.15.57:0/2285124861 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:43:41.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95dea05640 1 --2- 172.21.15.57:0/2285124861 >> v2:172.21.15.88:3300/0 conn(0x7f95d8070520 0x7f95d8070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:41.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95dd202640 1 -- 172.21.15.57:0/2285124861 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168728003 0 0) 0x7f95d80c2310 con 0x7f95d8150830 2023-10-30T14:43:41.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95dd202640 1 -- 172.21.15.57:0/2285124861 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95c4003530 con 0x7f95d8150830 2023-10-30T14:43:41.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95dd202640 1 -- 172.21.15.57:0/2285124861 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1115540953 0 0) 0x7f95c4003530 con 0x7f95d8150830 2023-10-30T14:43:41.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95dd202640 1 -- 172.21.15.57:0/2285124861 >> v2:172.21.15.88:3300/0 conn(0x7f95d8070520 msgr2=0x7f95d8070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:41.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95dd202640 1 --2- 172.21.15.57:0/2285124861 >> v2:172.21.15.88:3300/0 conn(0x7f95d8070520 0x7f95d8070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:41.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95dd202640 1 -- 172.21.15.57:0/2285124861 >> v1:172.21.15.88:6789/0 conn(0x7f95d80cb7d0 legacy=0x7f95d80c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:41.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95dd202640 1 -- 172.21.15.57:0/2285124861 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d80c4360 con 0x7f95d8150830 2023-10-30T14:43:41.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95dd202640 1 -- 172.21.15.57:0/2285124861 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f95d40016a0 con 0x7f95d8150830 2023-10-30T14:43:41.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95dd202640 1 -- 172.21.15.57:0/2285124861 >> v1:172.21.15.57:6789/0 conn(0x7f95d8150830 legacy=0x7f95d806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:41.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95dd202640 1 --2- 172.21.15.57:0/2285124861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95c4003980 0x7f95c4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:41.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95dd202640 1 --2- 172.21.15.57:0/2285124861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95c40047d0 0x7f95c4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:41.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95dd202640 1 --2- 172.21.15.57:0/2285124861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c4005190 0x7f95c400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:41.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95dd202640 1 -- 172.21.15.57:0/2285124861 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d806bc10 con 0x7f95c4003980 2023-10-30T14:43:41.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95dd202640 1 -- 172.21.15.57:0/2285124861 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c400c040 con 0x7f95c4005190 2023-10-30T14:43:41.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.123+0000 7f95dd202640 1 -- 172.21.15.57:0/2285124861 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c400e410 con 0x7f95c40047d0 2023-10-30T14:43:41.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.127+0000 7f95de204640 1 --2- 172.21.15.57:0/2285124861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95c4003980 0x7f95c4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:41.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.127+0000 7f95dda03640 1 --2- 172.21.15.57:0/2285124861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95c40047d0 0x7f95c4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:41.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.127+0000 7f95dea05640 1 --2- 172.21.15.57:0/2285124861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c4005190 0x7f95c400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:41.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.127+0000 7f95de204640 1 -- 172.21.15.57:0/2285124861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95c40047d0 msgr2=0x7f95c4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:41.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.127+0000 7f95de204640 1 --2- 172.21.15.57:0/2285124861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95c40047d0 0x7f95c4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:41.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.127+0000 7f95de204640 1 -- 172.21.15.57:0/2285124861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c4005190 msgr2=0x7f95c400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:41.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.127+0000 7f95de204640 1 --2- 172.21.15.57:0/2285124861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c4005190 0x7f95c400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:41.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.127+0000 7f95de204640 1 -- 172.21.15.57:0/2285124861 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c4003530 con 0x7f95c4003980 2023-10-30T14:43:41.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.127+0000 7f95dea05640 1 --2- 172.21.15.57:0/2285124861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c4005190 0x7f95c400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:43:41.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.127+0000 7f95dea05640 1 -- 172.21.15.57:0/2285124861 reap_dead start 2023-10-30T14:43:41.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.127+0000 7f95de204640 1 --2- 172.21.15.57:0/2285124861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95c4003980 0x7f95c4003da0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f95d80c24a0 tx=0x7f95d401f2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=150a9522ff605c5a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:41.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.127+0000 7f95dd202640 1 -- 172.21.15.57:0/2285124861 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d4005ae0 con 0x7f95c4003980 2023-10-30T14:43:41.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.127+0000 7f95dd202640 1 -- 172.21.15.57:0/2285124861 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d40ab070 con 0x7f95c4003980 2023-10-30T14:43:41.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.127+0000 7f95dd202640 1 -- 172.21.15.57:0/2285124861 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d4005e80 con 0x7f95c4003980 2023-10-30T14:43:41.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95e0468640 1 -- 172.21.15.57:0/2285124861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95c4003980 msgr2=0x7f95c4003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:41.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95e0468640 1 --2- 172.21.15.57:0/2285124861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95c4003980 0x7f95c4003da0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f95d80c24a0 tx=0x7f95d401f2f0 comp rx=0 tx=0).stop 2023-10-30T14:43:41.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95e0468640 1 -- 172.21.15.57:0/2285124861 shutdown_connections 2023-10-30T14:43:41.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95e0468640 1 --2- 172.21.15.57:0/2285124861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95c4003980 0x7f95c4003da0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:41.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95e0468640 1 -- 172.21.15.57:0/2285124861 >> 172.21.15.57:0/2285124861 conn(0x7f95d80cac70 msgr2=0x7f95d80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:41.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95e0468640 1 -- 172.21.15.57:0/2285124861 shutdown_connections 2023-10-30T14:43:41.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95e0468640 1 -- 172.21.15.57:0/2285124861 wait complete. 2023-10-30T14:43:41.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95e0468640 1 Processor -- start 2023-10-30T14:43:41.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95e0468640 1 -- start start 2023-10-30T14:43:41.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95e0468640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c4003980 0x7f95d8153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:41.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95e0468640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95d8070930 0x7f95d815a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:41.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95e0468640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95d8157ce0 0x7f95d8158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:41.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95e0468640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d8157850 con 0x7f95d8070930 2023-10-30T14:43:41.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95e0468640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d814f7d0 con 0x7f95c4003980 2023-10-30T14:43:41.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95e0468640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d81586a0 con 0x7f95d8157ce0 2023-10-30T14:43:41.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95dda03640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95d8070930 0x7f95d815a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:41.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95dda03640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95d8070930 0x7f95d815a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38212/0 (socket says 172.21.15.57:38212) 2023-10-30T14:43:41.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95dda03640 1 -- 172.21.15.57:0/3372483862 learned_addr learned my addr 172.21.15.57:0/3372483862 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:41.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95de204640 1 --2- 172.21.15.57:0/3372483862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c4003980 0x7f95d8153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:41.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95dea05640 1 --2- 172.21.15.57:0/3372483862 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95d8157ce0 0x7f95d8158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:41.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95dda03640 1 -- 172.21.15.57:0/3372483862 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95d8157ce0 msgr2=0x7f95d8158160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:41.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95dda03640 1 --2- 172.21.15.57:0/3372483862 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95d8157ce0 0x7f95d8158160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:41.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95dda03640 1 -- 172.21.15.57:0/3372483862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c4003980 msgr2=0x7f95d8153a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:41.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95dda03640 1 --2- 172.21.15.57:0/3372483862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c4003980 0x7f95d8153a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:41.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95dda03640 1 -- 172.21.15.57:0/3372483862 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d8146020 con 0x7f95d8070930 2023-10-30T14:43:41.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.131+0000 7f95de204640 1 --2- 172.21.15.57:0/3372483862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c4003980 0x7f95d8153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:43:41.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.135+0000 7f95dda03640 1 --2- 172.21.15.57:0/3372483862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95d8070930 0x7f95d815a040 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f95c800e4e0 tx=0x7f95c800e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:41.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.135+0000 7f95ceffd640 1 -- 172.21.15.57:0/3372483862 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c80052e0 con 0x7f95d8070930 2023-10-30T14:43:41.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.135+0000 7f95ceffd640 1 -- 172.21.15.57:0/3372483862 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95c8005440 con 0x7f95d8070930 2023-10-30T14:43:41.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.135+0000 7f95ceffd640 1 -- 172.21.15.57:0/3372483862 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c8003840 con 0x7f95d8070930 2023-10-30T14:43:41.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.135+0000 7f95e0468640 1 -- 172.21.15.57:0/3372483862 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d8146300 con 0x7f95d8070930 2023-10-30T14:43:41.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.135+0000 7f95e0468640 1 -- 172.21.15.57:0/3372483862 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d80c6dd0 con 0x7f95d8070930 2023-10-30T14:43:41.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.135+0000 7f95e0468640 1 -- 172.21.15.57:0/3372483862 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d8157a00 con 0x7f95d8070930 2023-10-30T14:43:41.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.135+0000 7f95ceffd640 1 -- 172.21.15.57:0/3372483862 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f95c80039a0 con 0x7f95d8070930 2023-10-30T14:43:41.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.135+0000 7f95ceffd640 1 --2- 172.21.15.57:0/3372483862 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95bc0664d0 0x7f95bc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:41.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.135+0000 7f95ceffd640 1 -- 172.21.15.57:0/3372483862 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f95c8083ea0 con 0x7f95d8070930 2023-10-30T14:43:41.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.135+0000 7f95de204640 1 --2- 172.21.15.57:0/3372483862 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95bc0664d0 0x7f95bc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:41.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.139+0000 7f95de204640 1 --2- 172.21.15.57:0/3372483862 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95bc0664d0 0x7f95bc068990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f95d40af000 tx=0x7f95d40aa000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:41.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.139+0000 7f95ceffd640 1 -- 172.21.15.57:0/3372483862 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f95d8157a00 con 0x7f95d8070930 2023-10-30T14:43:41.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.331+0000 7f95e0468640 1 -- 172.21.15.57:0/3372483862 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95d8157a00 con 0x7f95d8070930 2023-10-30T14:43:41.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.331+0000 7f95ceffd640 1 -- 172.21.15.57:0/3372483862 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+6532 (secure 0 0 0) 0x7f95d8157a00 con 0x7f95d8070930 2023-10-30T14:43:41.337 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:41.337 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":160,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:39.163957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:41.337 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 160 2023-10-30T14:43:41.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.331+0000 7f95e0468640 1 -- 172.21.15.57:0/3372483862 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95bc0664d0 msgr2=0x7f95bc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:41.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.331+0000 7f95e0468640 1 --2- 172.21.15.57:0/3372483862 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95bc0664d0 0x7f95bc068990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f95d40af000 tx=0x7f95d40aa000 comp rx=0 tx=0).stop 2023-10-30T14:43:41.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.331+0000 7f95e0468640 1 -- 172.21.15.57:0/3372483862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95d8070930 msgr2=0x7f95d815a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:41.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.331+0000 7f95e0468640 1 --2- 172.21.15.57:0/3372483862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95d8070930 0x7f95d815a040 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f95c800e4e0 tx=0x7f95c800e9a0 comp rx=0 tx=0).stop 2023-10-30T14:43:41.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.331+0000 7f95e0468640 1 -- 172.21.15.57:0/3372483862 shutdown_connections 2023-10-30T14:43:41.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.331+0000 7f95e0468640 1 --2- 172.21.15.57:0/3372483862 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f95bc0664d0 0x7f95bc068990 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:41.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.331+0000 7f95e0468640 1 --2- 172.21.15.57:0/3372483862 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95d8157ce0 0x7f95d8158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:41.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.331+0000 7f95e0468640 1 --2- 172.21.15.57:0/3372483862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95d8070930 0x7f95d815a040 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:41.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.331+0000 7f95e0468640 1 --2- 172.21.15.57:0/3372483862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c4003980 0x7f95d8153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:41.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.331+0000 7f95e0468640 1 -- 172.21.15.57:0/3372483862 >> 172.21.15.57:0/3372483862 conn(0x7f95d80cac70 msgr2=0x7f95d814e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:41.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.335+0000 7f95e0468640 1 -- 172.21.15.57:0/3372483862 shutdown_connections 2023-10-30T14:43:41.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:41.335+0000 7f95e0468640 1 -- 172.21.15.57:0/3372483862 wait complete. 2023-10-30T14:43:41.357 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:43:43.359 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:43.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.507+0000 7f7234f65640 1 Processor -- start 2023-10-30T14:43:43.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.507+0000 7f7234f65640 1 -- start start 2023-10-30T14:43:43.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.511+0000 7f7234f65640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f72300cb8b0 0x7f723006f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:43.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.511+0000 7f7234f65640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f72300ca9b0 0x7f723015a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:43.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.511+0000 7f7234f65640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72300ca180 con 0x7f72300c8060 2023-10-30T14:43:43.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.511+0000 7f7234f65640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7230150d00 con 0x7f72300ca9b0 2023-10-30T14:43:43.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.511+0000 7f7234f65640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72300ca740 con 0x7f72300cb8b0 2023-10-30T14:43:43.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.511+0000 7f722ed9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f72300ca9b0 0x7f723015a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:43.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.511+0000 7f722ed9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f72300ca9b0 0x7f723015a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38218/0 (socket says 172.21.15.57:38218) 2023-10-30T14:43:43.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.511+0000 7f722ed9d640 1 -- 172.21.15.57:0/336170786 learned_addr learned my addr 172.21.15.57:0/336170786 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:43.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.511+0000 7f7227fff640 1 --2- 172.21.15.57:0/336170786 >> v2:172.21.15.88:3300/0 conn(0x7f72300cb8b0 0x7f723006f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:43.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.511+0000 7f722ed9d640 1 -- 172.21.15.57:0/336170786 >> v2:172.21.15.88:3300/0 conn(0x7f72300cb8b0 msgr2=0x7f723006f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:43.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.511+0000 7f722ed9d640 1 --2- 172.21.15.57:0/336170786 >> v2:172.21.15.88:3300/0 conn(0x7f72300cb8b0 0x7f723006f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:43.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.511+0000 7f722ed9d640 1 -- 172.21.15.57:0/336170786 >> v1:172.21.15.88:6789/0 conn(0x7f72300c8060 legacy=0x7f72300c8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:43.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.511+0000 7f722ed9d640 1 -- 172.21.15.57:0/336170786 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72300ca460 con 0x7f72300ca9b0 2023-10-30T14:43:43.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.511+0000 7f7227fff640 1 --2- 172.21.15.57:0/336170786 >> v2:172.21.15.88:3300/0 conn(0x7f72300cb8b0 0x7f723006f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:43:43.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f722ed9d640 1 --2- 172.21.15.57:0/336170786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72300ca9b0 0x7f723015a4d0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f723006e290 tx=0x7f722801f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=978ee366edadd7c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:43.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f722dd9b640 1 -- 172.21.15.57:0/336170786 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72280af400 con 0x7f72300ca9b0 2023-10-30T14:43:43.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f722dd9b640 1 -- 172.21.15.57:0/336170786 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72280ab050 con 0x7f72300ca9b0 2023-10-30T14:43:43.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f722dd9b640 1 -- 172.21.15.57:0/336170786 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72280af780 con 0x7f72300ca9b0 2023-10-30T14:43:43.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f7234f65640 1 -- 172.21.15.57:0/336170786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72300ca9b0 msgr2=0x7f723015a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:43.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f7234f65640 1 --2- 172.21.15.57:0/336170786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72300ca9b0 0x7f723015a4d0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f723006e290 tx=0x7f722801f180 comp rx=0 tx=0).stop 2023-10-30T14:43:43.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f7234f65640 1 -- 172.21.15.57:0/336170786 shutdown_connections 2023-10-30T14:43:43.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f7234f65640 1 --2- 172.21.15.57:0/336170786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72300ca9b0 0x7f723015a4d0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:43.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f7234f65640 1 --2- 172.21.15.57:0/336170786 >> v2:172.21.15.88:3300/0 conn(0x7f72300cb8b0 0x7f723006f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:43.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f7234f65640 1 -- 172.21.15.57:0/336170786 >> 172.21.15.57:0/336170786 conn(0x7f723006afa0 msgr2=0x7f723006b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:43.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f7234f65640 1 -- 172.21.15.57:0/336170786 shutdown_connections 2023-10-30T14:43:43.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f7234f65640 1 -- 172.21.15.57:0/336170786 wait complete. 2023-10-30T14:43:43.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f7234f65640 1 Processor -- start 2023-10-30T14:43:43.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f7234f65640 1 -- start start 2023-10-30T14:43:43.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f7234f65640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72300c8060 0x7f72300adaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:43.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f7234f65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72300ca9b0 0x7f72300ae030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:43.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f7234f65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72300cb8b0 0x7f7230153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:43.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f7234f65640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7230159880 con 0x7f72300c8060 2023-10-30T14:43:43.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f7234f65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723014fd50 con 0x7f72300ca9b0 2023-10-30T14:43:43.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.515+0000 7f7234f65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7230158990 con 0x7f72300cb8b0 2023-10-30T14:43:43.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.519+0000 7f722e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72300c8060 0x7f72300adaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:43.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.519+0000 7f722e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72300c8060 0x7f72300adaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38224/0 (socket says 172.21.15.57:38224) 2023-10-30T14:43:43.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.519+0000 7f722e59c640 1 -- 172.21.15.57:0/3723753256 learned_addr learned my addr 172.21.15.57:0/3723753256 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:43.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.519+0000 7f7227fff640 1 --2- 172.21.15.57:0/3723753256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72300ca9b0 0x7f72300ae030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:43.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.519+0000 7f722ed9d640 1 --2- 172.21.15.57:0/3723753256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72300cb8b0 0x7f7230153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:43.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.519+0000 7f722e59c640 1 -- 172.21.15.57:0/3723753256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72300cb8b0 msgr2=0x7f7230153430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:43.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.519+0000 7f722e59c640 1 --2- 172.21.15.57:0/3723753256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72300cb8b0 0x7f7230153430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:43.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.519+0000 7f722e59c640 1 -- 172.21.15.57:0/3723753256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72300ca9b0 msgr2=0x7f72300ae030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:43.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.519+0000 7f722e59c640 1 --2- 172.21.15.57:0/3723753256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72300ca9b0 0x7f72300ae030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:43.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.519+0000 7f722e59c640 1 -- 172.21.15.57:0/3723753256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7228009b30 con 0x7f72300c8060 2023-10-30T14:43:43.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.519+0000 7f722ed9d640 1 --2- 172.21.15.57:0/3723753256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72300cb8b0 0x7f7230153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:43:43.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.519+0000 7f722e59c640 1 --2- 172.21.15.57:0/3723753256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72300c8060 0x7f72300adaf0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f722000c450 tx=0x7f722000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:43.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.519+0000 7f7226ffd640 1 -- 172.21.15.57:0/3723753256 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7220012830 con 0x7f72300c8060 2023-10-30T14:43:43.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.519+0000 7f7226ffd640 1 -- 172.21.15.57:0/3723753256 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72200129b0 con 0x7f72300c8060 2023-10-30T14:43:43.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.519+0000 7f7234f65640 1 -- 172.21.15.57:0/3723753256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7230158c70 con 0x7f72300c8060 2023-10-30T14:43:43.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.519+0000 7f7226ffd640 1 -- 172.21.15.57:0/3723753256 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7220012d60 con 0x7f72300c8060 2023-10-30T14:43:43.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.519+0000 7f7234f65640 1 -- 172.21.15.57:0/3723753256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7230159230 con 0x7f72300c8060 2023-10-30T14:43:43.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.519+0000 7f7234f65640 1 -- 172.21.15.57:0/3723753256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f723004e170 con 0x7f72300c8060 2023-10-30T14:43:43.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.519+0000 7f7226ffd640 1 -- 172.21.15.57:0/3723753256 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f722001c460 con 0x7f72300c8060 2023-10-30T14:43:43.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.523+0000 7f7226ffd640 1 --2- 172.21.15.57:0/3723753256 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f72100664d0 0x7f7210068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:43.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.523+0000 7f7226ffd640 1 -- 172.21.15.57:0/3723753256 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f7220084130 con 0x7f72300c8060 2023-10-30T14:43:43.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.523+0000 7f7227fff640 1 --2- 172.21.15.57:0/3723753256 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f72100664d0 0x7f7210068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:43.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.523+0000 7f7227fff640 1 --2- 172.21.15.57:0/3723753256 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f72100664d0 0x7f7210068990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f72300bf620 tx=0x7f7218008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:43.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.527+0000 7f7226ffd640 1 -- 172.21.15.57:0/3723753256 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7220051020 con 0x7f72300c8060 2023-10-30T14:43:43.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.719+0000 7f7234f65640 1 -- 172.21.15.57:0/3723753256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72300b1aa0 con 0x7f72300c8060 2023-10-30T14:43:43.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.719+0000 7f7226ffd640 1 -- 172.21.15.57:0/3723753256 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+6532 (secure 0 0 0) 0x7f7220054640 con 0x7f72300c8060 2023-10-30T14:43:43.724 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:43.724 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":160,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:39.163957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:43.724 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 160 2023-10-30T14:43:43.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.719+0000 7f7234f65640 1 -- 172.21.15.57:0/3723753256 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f72100664d0 msgr2=0x7f7210068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:43.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.719+0000 7f7234f65640 1 --2- 172.21.15.57:0/3723753256 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f72100664d0 0x7f7210068990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f72300bf620 tx=0x7f7218008040 comp rx=0 tx=0).stop 2023-10-30T14:43:43.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.719+0000 7f7234f65640 1 -- 172.21.15.57:0/3723753256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72300c8060 msgr2=0x7f72300adaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:43.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.719+0000 7f7234f65640 1 --2- 172.21.15.57:0/3723753256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72300c8060 0x7f72300adaf0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f722000c450 tx=0x7f722000c910 comp rx=0 tx=0).stop 2023-10-30T14:43:43.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.719+0000 7f7234f65640 1 -- 172.21.15.57:0/3723753256 shutdown_connections 2023-10-30T14:43:43.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.719+0000 7f7234f65640 1 --2- 172.21.15.57:0/3723753256 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f72100664d0 0x7f7210068990 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:43.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.719+0000 7f7234f65640 1 --2- 172.21.15.57:0/3723753256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72300cb8b0 0x7f7230153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:43.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.719+0000 7f7234f65640 1 --2- 172.21.15.57:0/3723753256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72300ca9b0 0x7f72300ae030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:43.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.719+0000 7f7234f65640 1 --2- 172.21.15.57:0/3723753256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72300c8060 0x7f72300adaf0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:43.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.719+0000 7f7234f65640 1 -- 172.21.15.57:0/3723753256 >> 172.21.15.57:0/3723753256 conn(0x7f723006afa0 msgr2=0x7f72300c6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:43.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.719+0000 7f7234f65640 1 -- 172.21.15.57:0/3723753256 shutdown_connections 2023-10-30T14:43:43.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:43.719+0000 7f7234f65640 1 -- 172.21.15.57:0/3723753256 wait complete. 2023-10-30T14:43:43.744 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:43:45.745 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:45.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.895+0000 7f39a1965640 1 Processor -- start 2023-10-30T14:43:45.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.895+0000 7f39a1965640 1 -- start start 2023-10-30T14:43:45.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.895+0000 7f39a1965640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f399c06c420 0x7f399c0c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:45.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.895+0000 7f39a1965640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399c0c5f80 con 0x7f399c153bb0 2023-10-30T14:43:45.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.895+0000 7f39a1965640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399c0c5690 con 0x7f399c0c7170 2023-10-30T14:43:45.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.895+0000 7f39a1965640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f399c0c7580 con 0x7f399c06c420 2023-10-30T14:43:45.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.895+0000 7f3993fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f399c06c420 0x7f399c0c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:45.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.895+0000 7f399affd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f399c153bb0 0x7f399c0c6410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:59170/0 (socket says 172.21.15.57:59170) 2023-10-30T14:43:45.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.895+0000 7f399affd640 1 -- 172.21.15.57:0/3329324573 learned_addr learned my addr 172.21.15.57:0/3329324573 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:43:45.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.899+0000 7f3993fff640 1 -- 172.21.15.57:0/3329324573 >> v1:172.21.15.88:6789/0 conn(0x7f399c0c7170 legacy=0x7f399c159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:45.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.899+0000 7f3993fff640 1 -- 172.21.15.57:0/3329324573 >> v1:172.21.15.57:6789/0 conn(0x7f399c153bb0 legacy=0x7f399c0c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:45.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.899+0000 7f3993fff640 1 -- 172.21.15.57:0/3329324573 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399c15a580 con 0x7f399c06c420 2023-10-30T14:43:45.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.899+0000 7f399a7fc640 1 -- 172.21.15.57:0/3329324573 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698397242 0 0) 0x7f399c0c5f80 con 0x7f399c153bb0 2023-10-30T14:43:45.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.899+0000 7f3993fff640 1 --2- 172.21.15.57:0/3329324573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c06c420 0x7f399c0c6c30 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f39840097c0 tx=0x7f398401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f87959ef6c1d5ebd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:45.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.899+0000 7f399a7fc640 1 -- 172.21.15.57:0/3329324573 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39840a8970 con 0x7f399c06c420 2023-10-30T14:43:45.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.899+0000 7f399a7fc640 1 -- 172.21.15.57:0/3329324573 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39840a8ad0 con 0x7f399c06c420 2023-10-30T14:43:45.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.899+0000 7f399a7fc640 1 -- 172.21.15.57:0/3329324573 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39840a8e60 con 0x7f399c06c420 2023-10-30T14:43:45.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.899+0000 7f39a1965640 1 -- 172.21.15.57:0/3329324573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c06c420 msgr2=0x7f399c0c6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:45.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.899+0000 7f39a1965640 1 --2- 172.21.15.57:0/3329324573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c06c420 0x7f399c0c6c30 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f39840097c0 tx=0x7f398401ec70 comp rx=0 tx=0).stop 2023-10-30T14:43:45.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f39a1965640 1 -- 172.21.15.57:0/3329324573 shutdown_connections 2023-10-30T14:43:45.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f39a1965640 1 --2- 172.21.15.57:0/3329324573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c06c420 0x7f399c0c6c30 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:45.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f39a1965640 1 -- 172.21.15.57:0/3329324573 >> 172.21.15.57:0/3329324573 conn(0x7f399c0cac70 msgr2=0x7f399c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:45.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f39a1965640 1 -- 172.21.15.57:0/3329324573 shutdown_connections 2023-10-30T14:43:45.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f39a1965640 1 -- 172.21.15.57:0/3329324573 wait complete. 2023-10-30T14:43:45.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f39a1965640 1 Processor -- start 2023-10-30T14:43:45.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f39a1965640 1 -- start start 2023-10-30T14:43:45.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f39a1965640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f399c06c420 0x7f399c0be380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:45.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f39a1965640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c0c7170 0x7f399c154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:45.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f39a1965640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f399c153bb0 0x7f399c070e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:45.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f39a1965640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399c14d630 con 0x7f399c0c7170 2023-10-30T14:43:45.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f39a1965640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399c0bd2c0 con 0x7f399c06c420 2023-10-30T14:43:45.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f39a1965640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f399c0bcfe0 con 0x7f399c153bb0 2023-10-30T14:43:45.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f3993fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c0c7170 0x7f399c154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:45.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f3993fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c0c7170 0x7f399c154520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58728/0 (socket says 172.21.15.57:58728) 2023-10-30T14:43:45.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f3993fff640 1 -- 172.21.15.57:0/728913051 learned_addr learned my addr 172.21.15.57:0/728913051 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:45.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f399affd640 1 --2- 172.21.15.57:0/728913051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f399c06c420 0x7f399c0be380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:45.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f399b7fe640 1 --2- 172.21.15.57:0/728913051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f399c153bb0 0x7f399c070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:45.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f3993fff640 1 -- 172.21.15.57:0/728913051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f399c153bb0 msgr2=0x7f399c070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:45.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f3993fff640 1 --2- 172.21.15.57:0/728913051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f399c153bb0 0x7f399c070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:45.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f3993fff640 1 -- 172.21.15.57:0/728913051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f399c06c420 msgr2=0x7f399c0be380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:45.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f3993fff640 1 --2- 172.21.15.57:0/728913051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f399c06c420 0x7f399c0be380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:45.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f3993fff640 1 -- 172.21.15.57:0/728913051 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3984009420 con 0x7f399c0c7170 2023-10-30T14:43:45.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f399affd640 1 --2- 172.21.15.57:0/728913051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f399c06c420 0x7f399c0be380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:43:45.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.903+0000 7f3993fff640 1 --2- 172.21.15.57:0/728913051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c0c7170 0x7f399c154520 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f398401b880 tx=0x7f398401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:45.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.907+0000 7f3998ff9640 1 -- 172.21.15.57:0/728913051 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39840b0ea0 con 0x7f399c0c7170 2023-10-30T14:43:45.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.907+0000 7f3998ff9640 1 -- 172.21.15.57:0/728913051 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39840b9410 con 0x7f399c0c7170 2023-10-30T14:43:45.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.907+0000 7f3998ff9640 1 -- 172.21.15.57:0/728913051 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39840b97a0 con 0x7f399c0c7170 2023-10-30T14:43:45.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.907+0000 7f39a1965640 1 -- 172.21.15.57:0/728913051 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f399c0bd5c0 con 0x7f399c0c7170 2023-10-30T14:43:45.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.907+0000 7f39a1965640 1 -- 172.21.15.57:0/728913051 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f399c0bdb80 con 0x7f399c0c7170 2023-10-30T14:43:45.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.907+0000 7f39a1965640 1 -- 172.21.15.57:0/728913051 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f399c066510 con 0x7f399c0c7170 2023-10-30T14:43:45.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.907+0000 7f3998ff9640 1 -- 172.21.15.57:0/728913051 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f39840b7040 con 0x7f399c0c7170 2023-10-30T14:43:45.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.907+0000 7f3998ff9640 1 --2- 172.21.15.57:0/728913051 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f397c0664a0 0x7f397c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:45.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.907+0000 7f3998ff9640 1 -- 172.21.15.57:0/728913051 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f39841221f0 con 0x7f399c0c7170 2023-10-30T14:43:45.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.911+0000 7f399affd640 1 --2- 172.21.15.57:0/728913051 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f397c0664a0 0x7f397c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:45.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.911+0000 7f399affd640 1 --2- 172.21.15.57:0/728913051 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f397c0664a0 0x7f397c068960 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f398c000f70 tx=0x7f398c007c60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:45.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:45.915+0000 7f3998ff9640 1 -- 172.21.15.57:0/728913051 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f39840eefc0 con 0x7f399c0c7170 2023-10-30T14:43:46.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:46.103+0000 7f39a1965640 1 -- 172.21.15.57:0/728913051 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f399c04e170 con 0x7f399c0c7170 2023-10-30T14:43:46.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:46.103+0000 7f3998ff9640 1 -- 172.21.15.57:0/728913051 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+6531 (secure 0 0 0) 0x7f39840b5090 con 0x7f399c0c7170 2023-10-30T14:43:46.109 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:46.109 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:44.163921+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:46.109 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 161 2023-10-30T14:43:46.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:46.107+0000 7f39a1965640 1 -- 172.21.15.57:0/728913051 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f397c0664a0 msgr2=0x7f397c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:46.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:46.107+0000 7f39a1965640 1 --2- 172.21.15.57:0/728913051 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f397c0664a0 0x7f397c068960 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f398c000f70 tx=0x7f398c007c60 comp rx=0 tx=0).stop 2023-10-30T14:43:46.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:46.107+0000 7f39a1965640 1 -- 172.21.15.57:0/728913051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c0c7170 msgr2=0x7f399c154520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:46.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:46.107+0000 7f39a1965640 1 --2- 172.21.15.57:0/728913051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c0c7170 0x7f399c154520 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f398401b880 tx=0x7f398401ba10 comp rx=0 tx=0).stop 2023-10-30T14:43:46.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:46.107+0000 7f39a1965640 1 -- 172.21.15.57:0/728913051 shutdown_connections 2023-10-30T14:43:46.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:46.107+0000 7f39a1965640 1 --2- 172.21.15.57:0/728913051 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f397c0664a0 0x7f397c068960 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:46.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:46.107+0000 7f39a1965640 1 --2- 172.21.15.57:0/728913051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f399c153bb0 0x7f399c070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:46.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:46.107+0000 7f39a1965640 1 --2- 172.21.15.57:0/728913051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c0c7170 0x7f399c154520 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:46.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:46.107+0000 7f39a1965640 1 --2- 172.21.15.57:0/728913051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f399c06c420 0x7f399c0be380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:46.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:46.107+0000 7f39a1965640 1 -- 172.21.15.57:0/728913051 >> 172.21.15.57:0/728913051 conn(0x7f399c0cac70 msgr2=0x7f399c0c8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:46.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:46.107+0000 7f39a1965640 1 -- 172.21.15.57:0/728913051 shutdown_connections 2023-10-30T14:43:46.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:46.107+0000 7f39a1965640 1 -- 172.21.15.57:0/728913051 wait complete. 2023-10-30T14:43:46.129 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:43:48.130 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:48.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.279+0000 7f073482f640 1 Processor -- start 2023-10-30T14:43:48.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.279+0000 7f073482f640 1 -- start start 2023-10-30T14:43:48.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.283+0000 7f073482f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07300c8eb0 0x7f073015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:48.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.283+0000 7f073482f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f073015ac70 0x7f073015a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:48.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.283+0000 7f073482f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f073006aeb0 0x7f073006b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:48.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.283+0000 7f073482f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0730159800 con 0x7f073015ac70 2023-10-30T14:43:48.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.283+0000 7f073482f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f073015dcc0 con 0x7f073006aeb0 2023-10-30T14:43:48.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.283+0000 7f073482f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0730159d80 con 0x7f07300c8eb0 2023-10-30T14:43:48.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.283+0000 7f072effd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f073015ac70 0x7f073015a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:48.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.283+0000 7f072effd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f073015ac70 0x7f073015a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58742/0 (socket says 172.21.15.57:58742) 2023-10-30T14:43:48.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.283+0000 7f072effd640 1 -- 172.21.15.57:0/180931708 learned_addr learned my addr 172.21.15.57:0/180931708 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:48.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.283+0000 7f072ffff640 1 --2- 172.21.15.57:0/180931708 >> v2:172.21.15.88:3300/0 conn(0x7f073006aeb0 0x7f073006b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:48.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.283+0000 7f072f7fe640 1 --2- 172.21.15.57:0/180931708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07300c8eb0 0x7f073015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:48.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.283+0000 7f072effd640 1 -- 172.21.15.57:0/180931708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07300c8eb0 msgr2=0x7f073015cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:48.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.283+0000 7f072effd640 1 --2- 172.21.15.57:0/180931708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07300c8eb0 0x7f073015cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:48.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.283+0000 7f072effd640 1 -- 172.21.15.57:0/180931708 >> v2:172.21.15.88:3300/0 conn(0x7f073006aeb0 msgr2=0x7f073006b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:48.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.283+0000 7f072effd640 1 --2- 172.21.15.57:0/180931708 >> v2:172.21.15.88:3300/0 conn(0x7f073006aeb0 0x7f073006b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:48.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.283+0000 7f072effd640 1 -- 172.21.15.57:0/180931708 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073015a750 con 0x7f073015ac70 2023-10-30T14:43:48.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.283+0000 7f072f7fe640 1 --2- 172.21.15.57:0/180931708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07300c8eb0 0x7f073015cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:43:48.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.287+0000 7f072effd640 1 --2- 172.21.15.57:0/180931708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f073015ac70 0x7f073015a210 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f073008cfd0 tx=0x7f072401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e712d2a76d405a14 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:48.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.287+0000 7f072e7fc640 1 -- 172.21.15.57:0/180931708 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07240a89b0 con 0x7f073015ac70 2023-10-30T14:43:48.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.287+0000 7f072e7fc640 1 -- 172.21.15.57:0/180931708 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07240a8b10 con 0x7f073015ac70 2023-10-30T14:43:48.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.287+0000 7f072e7fc640 1 -- 172.21.15.57:0/180931708 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07240a8ea0 con 0x7f073015ac70 2023-10-30T14:43:48.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.287+0000 7f073482f640 1 -- 172.21.15.57:0/180931708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f073015ac70 msgr2=0x7f073015a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:48.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.287+0000 7f073482f640 1 --2- 172.21.15.57:0/180931708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f073015ac70 0x7f073015a210 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f073008cfd0 tx=0x7f072401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:43:48.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.287+0000 7f073482f640 1 -- 172.21.15.57:0/180931708 shutdown_connections 2023-10-30T14:43:48.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.287+0000 7f073482f640 1 --2- 172.21.15.57:0/180931708 >> v2:172.21.15.88:3300/0 conn(0x7f073006aeb0 0x7f073006b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:48.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.287+0000 7f073482f640 1 --2- 172.21.15.57:0/180931708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f073015ac70 0x7f073015a210 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:48.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.287+0000 7f073482f640 1 --2- 172.21.15.57:0/180931708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07300c8eb0 0x7f073015cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:48.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.287+0000 7f073482f640 1 -- 172.21.15.57:0/180931708 >> 172.21.15.57:0/180931708 conn(0x7f07300cef00 msgr2=0x7f07300cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:48.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.287+0000 7f073482f640 1 -- 172.21.15.57:0/180931708 shutdown_connections 2023-10-30T14:43:48.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.287+0000 7f073482f640 1 -- 172.21.15.57:0/180931708 wait complete. 2023-10-30T14:43:48.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.287+0000 7f073482f640 1 Processor -- start 2023-10-30T14:43:48.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.287+0000 7f073482f640 1 -- start start 2023-10-30T14:43:48.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f073482f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f073006aeb0 0x7f07301ef930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:48.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f073482f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07300c8eb0 0x7f0730154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:48.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f073482f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f073015ac70 0x7f07300cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:48.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f073482f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07301efe70 con 0x7f073015ac70 2023-10-30T14:43:48.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f073482f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07300c9b00 con 0x7f073006aeb0 2023-10-30T14:43:48.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f073482f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07301e63c0 con 0x7f07300c8eb0 2023-10-30T14:43:48.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f072ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f073015ac70 0x7f07300cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:48.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f072f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f073006aeb0 0x7f07301ef930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:48.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f072ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f073015ac70 0x7f07300cde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58744/0 (socket says 172.21.15.57:58744) 2023-10-30T14:43:48.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f072ffff640 1 -- 172.21.15.57:0/3641635100 learned_addr learned my addr 172.21.15.57:0/3641635100 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:48.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f072effd640 1 --2- 172.21.15.57:0/3641635100 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07300c8eb0 0x7f0730154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:48.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f072ffff640 1 -- 172.21.15.57:0/3641635100 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07300c8eb0 msgr2=0x7f0730154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:48.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f072ffff640 1 --2- 172.21.15.57:0/3641635100 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07300c8eb0 0x7f0730154170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:48.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f072ffff640 1 -- 172.21.15.57:0/3641635100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f073006aeb0 msgr2=0x7f07301ef930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:48.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f072ffff640 1 --2- 172.21.15.57:0/3641635100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f073006aeb0 0x7f07301ef930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:48.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f072ffff640 1 -- 172.21.15.57:0/3641635100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07240094d0 con 0x7f073015ac70 2023-10-30T14:43:48.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f072ffff640 1 --2- 172.21.15.57:0/3641635100 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f073015ac70 0x7f07300cde60 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f0728007e90 tx=0x7f072800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:48.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f0717fff640 1 -- 172.21.15.57:0/3641635100 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0728014420 con 0x7f073015ac70 2023-10-30T14:43:48.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f0717fff640 1 -- 172.21.15.57:0/3641635100 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07280145a0 con 0x7f073015ac70 2023-10-30T14:43:48.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f0717fff640 1 -- 172.21.15.57:0/3641635100 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0728014950 con 0x7f073015ac70 2023-10-30T14:43:48.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f073482f640 1 -- 172.21.15.57:0/3641635100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07300c9d50 con 0x7f073015ac70 2023-10-30T14:43:48.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f073482f640 1 -- 172.21.15.57:0/3641635100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07301e5d70 con 0x7f073015ac70 2023-10-30T14:43:48.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f073482f640 1 -- 172.21.15.57:0/3641635100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07301e6570 con 0x7f073015ac70 2023-10-30T14:43:48.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.291+0000 7f0717fff640 1 -- 172.21.15.57:0/3641635100 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0728014ab0 con 0x7f073015ac70 2023-10-30T14:43:48.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.295+0000 7f0717fff640 1 --2- 172.21.15.57:0/3641635100 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f071c0664a0 0x7f071c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:48.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.295+0000 7f0717fff640 1 -- 172.21.15.57:0/3641635100 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f0728084c20 con 0x7f073015ac70 2023-10-30T14:43:48.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.295+0000 7f072f7fe640 1 --2- 172.21.15.57:0/3641635100 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f071c0664a0 0x7f071c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:48.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.295+0000 7f072f7fe640 1 --2- 172.21.15.57:0/3641635100 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f071c0664a0 0x7f071c068960 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f0720009950 tx=0x7f0720008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:48.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.299+0000 7f0717fff640 1 -- 172.21.15.57:0/3641635100 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f07301e6570 con 0x7f073015ac70 2023-10-30T14:43:48.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.487+0000 7f073482f640 1 -- 172.21.15.57:0/3641635100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07301e6570 con 0x7f073015ac70 2023-10-30T14:43:48.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.487+0000 7f0717fff640 1 -- 172.21.15.57:0/3641635100 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+6531 (secure 0 0 0) 0x7f07301e6570 con 0x7f073015ac70 2023-10-30T14:43:48.493 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:48.494 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:44.163921+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:48.494 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 161 2023-10-30T14:43:48.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.491+0000 7f073482f640 1 -- 172.21.15.57:0/3641635100 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f071c0664a0 msgr2=0x7f071c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:48.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.491+0000 7f073482f640 1 --2- 172.21.15.57:0/3641635100 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f071c0664a0 0x7f071c068960 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f0720009950 tx=0x7f0720008040 comp rx=0 tx=0).stop 2023-10-30T14:43:48.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.491+0000 7f073482f640 1 -- 172.21.15.57:0/3641635100 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f073015ac70 msgr2=0x7f07300cde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:48.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.491+0000 7f073482f640 1 --2- 172.21.15.57:0/3641635100 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f073015ac70 0x7f07300cde60 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f0728007e90 tx=0x7f072800e4b0 comp rx=0 tx=0).stop 2023-10-30T14:43:48.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.491+0000 7f073482f640 1 -- 172.21.15.57:0/3641635100 shutdown_connections 2023-10-30T14:43:48.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.491+0000 7f073482f640 1 --2- 172.21.15.57:0/3641635100 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f071c0664a0 0x7f071c068960 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:48.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.491+0000 7f073482f640 1 --2- 172.21.15.57:0/3641635100 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f073015ac70 0x7f07300cde60 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:48.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.491+0000 7f073482f640 1 --2- 172.21.15.57:0/3641635100 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07300c8eb0 0x7f0730154170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:48.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.491+0000 7f073482f640 1 --2- 172.21.15.57:0/3641635100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f073006aeb0 0x7f07301ef930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:48.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.491+0000 7f073482f640 1 -- 172.21.15.57:0/3641635100 >> 172.21.15.57:0/3641635100 conn(0x7f07300cef00 msgr2=0x7f07300c6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:48.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.491+0000 7f073482f640 1 -- 172.21.15.57:0/3641635100 shutdown_connections 2023-10-30T14:43:48.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:48.491+0000 7f073482f640 1 -- 172.21.15.57:0/3641635100 wait complete. 2023-10-30T14:43:48.514 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T14:43:48.515 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 161, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 161, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6356": { "addr": "172.21.15.88:6839/1744808283", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1744808283, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1744808283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6356, "incarnation": 151, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 45 }, "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 44 }, "gid_7056": { "addr": "172.21.15.57:6837/2307041430", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 2307041430, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 2307041430, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 7056, "incarnation": 137, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:stopping", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:43:44.163921+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6356, "mds_1": 6747, "mds_2": 7056 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.57:6834/697507447", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 697507447, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 697507447, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 156, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7377, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:43:50.516 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:50.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.667+0000 7fa267054640 1 Processor -- start 2023-10-30T14:43:50.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.667+0000 7fa267054640 1 -- start start 2023-10-30T14:43:50.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.667+0000 7fa267054640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa260153c80 0x7fa26015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:50.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.667+0000 7fa267054640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa2600c4a10 0x7fa2600c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:50.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.667+0000 7fa267054640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa260159f20 con 0x7fa26015a3b0 2023-10-30T14:43:50.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.667+0000 7fa267054640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa26015dec0 con 0x7fa260153c80 2023-10-30T14:43:50.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.667+0000 7fa267054640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2601599a0 con 0x7fa2600c4a10 2023-10-30T14:43:50.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.667+0000 7fa264df0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa260153c80 0x7fa26015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:50.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.667+0000 7fa264df0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa260153c80 0x7fa26015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58752/0 (socket says 172.21.15.57:58752) 2023-10-30T14:43:50.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.667+0000 7fa264df0640 1 -- 172.21.15.57:0/2190301777 learned_addr learned my addr 172.21.15.57:0/2190301777 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:50.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.667+0000 7fa25ffff640 1 --2- 172.21.15.57:0/2190301777 >> v2:172.21.15.88:3300/0 conn(0x7fa2600c4a10 0x7fa2600c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:50.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.667+0000 7fa264df0640 1 -- 172.21.15.57:0/2190301777 >> v2:172.21.15.88:3300/0 conn(0x7fa2600c4a10 msgr2=0x7fa2600c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:50.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.667+0000 7fa264df0640 1 --2- 172.21.15.57:0/2190301777 >> v2:172.21.15.88:3300/0 conn(0x7fa2600c4a10 0x7fa2600c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:50.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.667+0000 7fa264df0640 1 -- 172.21.15.57:0/2190301777 >> v1:172.21.15.88:6789/0 conn(0x7fa26015a3b0 legacy=0x7fa2600ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:50.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.667+0000 7fa264df0640 1 -- 172.21.15.57:0/2190301777 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2600c47e0 con 0x7fa260153c80 2023-10-30T14:43:50.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.671+0000 7fa264df0640 1 --2- 172.21.15.57:0/2190301777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa260153c80 0x7fa26015cf90 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fa26006be50 tx=0x7fa25401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fb523635b788770 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:50.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.671+0000 7fa25f7fe640 1 -- 172.21.15.57:0/2190301777 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2540a89b0 con 0x7fa260153c80 2023-10-30T14:43:50.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.671+0000 7fa25f7fe640 1 -- 172.21.15.57:0/2190301777 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2540a8b10 con 0x7fa260153c80 2023-10-30T14:43:50.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.671+0000 7fa25f7fe640 1 -- 172.21.15.57:0/2190301777 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2540a8ea0 con 0x7fa260153c80 2023-10-30T14:43:50.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.671+0000 7fa267054640 1 -- 172.21.15.57:0/2190301777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa260153c80 msgr2=0x7fa26015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:50.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.671+0000 7fa267054640 1 --2- 172.21.15.57:0/2190301777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa260153c80 0x7fa26015cf90 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fa26006be50 tx=0x7fa25401ecb0 comp rx=0 tx=0).stop 2023-10-30T14:43:50.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa267054640 1 -- 172.21.15.57:0/2190301777 shutdown_connections 2023-10-30T14:43:50.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa267054640 1 --2- 172.21.15.57:0/2190301777 >> v2:172.21.15.88:3300/0 conn(0x7fa2600c4a10 0x7fa2600c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:50.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa267054640 1 --2- 172.21.15.57:0/2190301777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa260153c80 0x7fa26015cf90 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:50.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa267054640 1 -- 172.21.15.57:0/2190301777 >> 172.21.15.57:0/2190301777 conn(0x7fa2600715c0 msgr2=0x7fa2600719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:50.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa267054640 1 -- 172.21.15.57:0/2190301777 shutdown_connections 2023-10-30T14:43:50.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa267054640 1 -- 172.21.15.57:0/2190301777 wait complete. 2023-10-30T14:43:50.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa267054640 1 Processor -- start 2023-10-30T14:43:50.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa267054640 1 -- start start 2023-10-30T14:43:50.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa267054640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2600c4a10 0x7fa2601efc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:50.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa267054640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa260153c80 0x7fa26015ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:50.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa267054640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa26015a3b0 0x7fa26015bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:50.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa267054640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26014e4c0 con 0x7fa2600c4a10 2023-10-30T14:43:50.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa267054640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2601f0180 con 0x7fa260153c80 2023-10-30T14:43:50.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa267054640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa260152ea0 con 0x7fa26015a3b0 2023-10-30T14:43:50.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa264df0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2600c4a10 0x7fa2601efc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:50.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa264df0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2600c4a10 0x7fa2601efc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58758/0 (socket says 172.21.15.57:58758) 2023-10-30T14:43:50.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa264df0640 1 -- 172.21.15.57:0/865925915 learned_addr learned my addr 172.21.15.57:0/865925915 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:50.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa25ffff640 1 --2- 172.21.15.57:0/865925915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa260153c80 0x7fa26015ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:50.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa2655f1640 1 --2- 172.21.15.57:0/865925915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa26015a3b0 0x7fa26015bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:50.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa264df0640 1 -- 172.21.15.57:0/865925915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa26015a3b0 msgr2=0x7fa26015bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:50.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa264df0640 1 --2- 172.21.15.57:0/865925915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa26015a3b0 0x7fa26015bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:50.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa264df0640 1 -- 172.21.15.57:0/865925915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa260153c80 msgr2=0x7fa26015ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:50.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa264df0640 1 --2- 172.21.15.57:0/865925915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa260153c80 0x7fa26015ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:50.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa264df0640 1 -- 172.21.15.57:0/865925915 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2540094d0 con 0x7fa2600c4a10 2023-10-30T14:43:50.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa25ffff640 1 --2- 172.21.15.57:0/865925915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa260153c80 0x7fa26015ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:43:50.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa264df0640 1 --2- 172.21.15.57:0/865925915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2600c4a10 0x7fa2601efc40 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fa25401d360 tx=0x7fa25401ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:50.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.675+0000 7fa25d7fa640 1 -- 172.21.15.57:0/865925915 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2540b0e00 con 0x7fa2600c4a10 2023-10-30T14:43:50.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.679+0000 7fa25d7fa640 1 -- 172.21.15.57:0/865925915 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2540b9440 con 0x7fa2600c4a10 2023-10-30T14:43:50.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.679+0000 7fa267054640 1 -- 172.21.15.57:0/865925915 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa26015b6f0 con 0x7fa2600c4a10 2023-10-30T14:43:50.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.679+0000 7fa25d7fa640 1 -- 172.21.15.57:0/865925915 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2540b97f0 con 0x7fa2600c4a10 2023-10-30T14:43:50.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.679+0000 7fa267054640 1 -- 172.21.15.57:0/865925915 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa260152570 con 0x7fa2600c4a10 2023-10-30T14:43:50.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.679+0000 7fa267054640 1 -- 172.21.15.57:0/865925915 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa26004e170 con 0x7fa2600c4a10 2023-10-30T14:43:50.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.679+0000 7fa25d7fa640 1 -- 172.21.15.57:0/865925915 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa2540b7070 con 0x7fa2600c4a10 2023-10-30T14:43:50.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.679+0000 7fa25d7fa640 1 --2- 172.21.15.57:0/865925915 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2440664d0 0x7fa244068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:50.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.679+0000 7fa25d7fa640 1 -- 172.21.15.57:0/865925915 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7fa2541220c0 con 0x7fa2600c4a10 2023-10-30T14:43:50.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.679+0000 7fa25ffff640 1 --2- 172.21.15.57:0/865925915 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2440664d0 0x7fa244068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:50.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.683+0000 7fa25ffff640 1 --2- 172.21.15.57:0/865925915 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2440664d0 0x7fa244068990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa26015baa0 tx=0x7fa24c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:50.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.683+0000 7fa25d7fa640 1 -- 172.21.15.57:0/865925915 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa2540eef20 con 0x7fa2600c4a10 2023-10-30T14:43:50.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.875+0000 7fa267054640 1 -- 172.21.15.57:0/865925915 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa26004e170 con 0x7fa2600c4a10 2023-10-30T14:43:50.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.875+0000 7fa25d7fa640 1 -- 172.21.15.57:0/865925915 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+6531 (secure 0 0 0) 0x7fa2540b5090 con 0x7fa2600c4a10 2023-10-30T14:43:50.880 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:50.880 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:44.163921+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:50.882 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 161 2023-10-30T14:43:50.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.875+0000 7fa267054640 1 -- 172.21.15.57:0/865925915 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2440664d0 msgr2=0x7fa244068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:50.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.875+0000 7fa267054640 1 --2- 172.21.15.57:0/865925915 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2440664d0 0x7fa244068990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa26015baa0 tx=0x7fa24c008040 comp rx=0 tx=0).stop 2023-10-30T14:43:50.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.875+0000 7fa267054640 1 -- 172.21.15.57:0/865925915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2600c4a10 msgr2=0x7fa2601efc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:50.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.875+0000 7fa267054640 1 --2- 172.21.15.57:0/865925915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2600c4a10 0x7fa2601efc40 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fa25401d360 tx=0x7fa25401ba50 comp rx=0 tx=0).stop 2023-10-30T14:43:50.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.875+0000 7fa267054640 1 -- 172.21.15.57:0/865925915 shutdown_connections 2023-10-30T14:43:50.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.875+0000 7fa267054640 1 --2- 172.21.15.57:0/865925915 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2440664d0 0x7fa244068990 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:50.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.875+0000 7fa267054640 1 --2- 172.21.15.57:0/865925915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa26015a3b0 0x7fa26015bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:50.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.875+0000 7fa267054640 1 --2- 172.21.15.57:0/865925915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa260153c80 0x7fa26015ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:50.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.875+0000 7fa267054640 1 --2- 172.21.15.57:0/865925915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2600c4a10 0x7fa2601efc40 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:50.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.875+0000 7fa267054640 1 -- 172.21.15.57:0/865925915 >> 172.21.15.57:0/865925915 conn(0x7fa2600715c0 msgr2=0x7fa26006bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:50.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.875+0000 7fa267054640 1 -- 172.21.15.57:0/865925915 shutdown_connections 2023-10-30T14:43:50.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:50.875+0000 7fa267054640 1 -- 172.21.15.57:0/865925915 wait complete. 2023-10-30T14:43:50.901 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:43:52.902 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:53.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.051+0000 7f66593a1640 1 Processor -- start 2023-10-30T14:43:53.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.051+0000 7f66593a1640 1 -- start start 2023-10-30T14:43:53.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.055+0000 7f66593a1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6654150830 0x7f665406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:53.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.055+0000 7f66593a1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66540cb7d0 0x7f66540c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:53.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.055+0000 7f66593a1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66540c2310 con 0x7f66540c3e40 2023-10-30T14:43:53.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.055+0000 7f66593a1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6654156710 con 0x7f6654150830 2023-10-30T14:43:53.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.055+0000 7f66593a1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66540c4360 con 0x7f66540cb7d0 2023-10-30T14:43:53.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.055+0000 7f664bfff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66540cb7d0 0x7f66540c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:53.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.055+0000 7f6652ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6654150830 0x7f665406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:53.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.055+0000 7f664bfff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66540cb7d0 0x7f66540c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:60590/0 (socket says 172.21.15.57:60590) 2023-10-30T14:43:53.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.055+0000 7f664bfff640 1 -- 172.21.15.57:0/2685841678 learned_addr learned my addr 172.21.15.57:0/2685841678 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:53.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.055+0000 7f6652ffd640 1 -- 172.21.15.57:0/2685841678 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66540cb7d0 msgr2=0x7f66540c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:53.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.055+0000 7f6652ffd640 1 --2- 172.21.15.57:0/2685841678 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66540cb7d0 0x7f66540c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:53.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.055+0000 7f6652ffd640 1 -- 172.21.15.57:0/2685841678 >> v1:172.21.15.88:6789/0 conn(0x7f66540c3e40 legacy=0x7f6654070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:53.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.055+0000 7f6652ffd640 1 -- 172.21.15.57:0/2685841678 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66541569b0 con 0x7f6654150830 2023-10-30T14:43:53.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.055+0000 7f664bfff640 1 --2- 172.21.15.57:0/2685841678 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66540cb7d0 0x7f66540c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:43:53.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.059+0000 7f6652ffd640 1 --2- 172.21.15.57:0/2685841678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6654150830 0x7f665406b500 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f66440097c0 tx=0x7f664401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f56f7ebf001a2e1a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:53.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.059+0000 7f66527fc640 1 -- 172.21.15.57:0/2685841678 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66440a8970 con 0x7f6654150830 2023-10-30T14:43:53.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.059+0000 7f66527fc640 1 -- 172.21.15.57:0/2685841678 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66440a8ad0 con 0x7f6654150830 2023-10-30T14:43:53.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.059+0000 7f66527fc640 1 -- 172.21.15.57:0/2685841678 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66440a8e60 con 0x7f6654150830 2023-10-30T14:43:53.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.059+0000 7f66593a1640 1 -- 172.21.15.57:0/2685841678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6654150830 msgr2=0x7f665406b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:53.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.059+0000 7f66593a1640 1 --2- 172.21.15.57:0/2685841678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6654150830 0x7f665406b500 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f66440097c0 tx=0x7f664401ec70 comp rx=0 tx=0).stop 2023-10-30T14:43:53.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.059+0000 7f66593a1640 1 -- 172.21.15.57:0/2685841678 shutdown_connections 2023-10-30T14:43:53.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.059+0000 7f66593a1640 1 --2- 172.21.15.57:0/2685841678 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66540cb7d0 0x7f66540c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:53.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.059+0000 7f66593a1640 1 --2- 172.21.15.57:0/2685841678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6654150830 0x7f665406b500 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:53.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.059+0000 7f66593a1640 1 -- 172.21.15.57:0/2685841678 >> 172.21.15.57:0/2685841678 conn(0x7f66540cac70 msgr2=0x7f66540cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:53.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.059+0000 7f66593a1640 1 -- 172.21.15.57:0/2685841678 shutdown_connections 2023-10-30T14:43:53.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.059+0000 7f66593a1640 1 -- 172.21.15.57:0/2685841678 wait complete. 2023-10-30T14:43:53.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f66593a1640 1 Processor -- start 2023-10-30T14:43:53.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f66593a1640 1 -- start start 2023-10-30T14:43:53.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f66593a1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66540c3e40 0x7f6654145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:53.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f66593a1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66540cb7d0 0x7f66541577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:53.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f66593a1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6654150830 0x7f66541e3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:53.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f66593a1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6654159e60 con 0x7f66540cb7d0 2023-10-30T14:43:53.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f66593a1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6654145f80 con 0x7f6654150830 2023-10-30T14:43:53.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f66593a1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6654154ba0 con 0x7f66540c3e40 2023-10-30T14:43:53.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f664bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66540cb7d0 0x7f66541577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:53.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f664bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66540cb7d0 0x7f66541577c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58766/0 (socket says 172.21.15.57:58766) 2023-10-30T14:43:53.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f664bfff640 1 -- 172.21.15.57:0/2439453075 learned_addr learned my addr 172.21.15.57:0/2439453075 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:53.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f66537fe640 1 --2- 172.21.15.57:0/2439453075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6654150830 0x7f66541e3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:53.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f6652ffd640 1 --2- 172.21.15.57:0/2439453075 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66540c3e40 0x7f6654145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:53.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f664bfff640 1 -- 172.21.15.57:0/2439453075 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66540c3e40 msgr2=0x7f6654145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:53.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f664bfff640 1 --2- 172.21.15.57:0/2439453075 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66540c3e40 0x7f6654145a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:53.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f664bfff640 1 -- 172.21.15.57:0/2439453075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6654150830 msgr2=0x7f66541e3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:53.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f664bfff640 1 --2- 172.21.15.57:0/2439453075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6654150830 0x7f66541e3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:53.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f664bfff640 1 -- 172.21.15.57:0/2439453075 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c0094d0 con 0x7f66540cb7d0 2023-10-30T14:43:53.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f664bfff640 1 --2- 172.21.15.57:0/2439453075 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66540cb7d0 0x7f66541577c0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f663c00e9d0 tx=0x7f663c00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:53.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f664b7fe640 1 -- 172.21.15.57:0/2439453075 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c00ca60 con 0x7f66540cb7d0 2023-10-30T14:43:53.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f664b7fe640 1 -- 172.21.15.57:0/2439453075 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663c00cbc0 con 0x7f66540cb7d0 2023-10-30T14:43:53.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f664b7fe640 1 -- 172.21.15.57:0/2439453075 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c013470 con 0x7f66540cb7d0 2023-10-30T14:43:53.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f66593a1640 1 -- 172.21.15.57:0/2439453075 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6644009420 con 0x7f66540cb7d0 2023-10-30T14:43:53.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f66593a1640 1 -- 172.21.15.57:0/2439453075 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6654155160 con 0x7f66540cb7d0 2023-10-30T14:43:53.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.063+0000 7f66593a1640 1 -- 172.21.15.57:0/2439453075 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6654159fa0 con 0x7f66540cb7d0 2023-10-30T14:43:53.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.067+0000 7f664b7fe640 1 -- 172.21.15.57:0/2439453075 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f663c0135d0 con 0x7f66540cb7d0 2023-10-30T14:43:53.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.067+0000 7f664b7fe640 1 --2- 172.21.15.57:0/2439453075 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66340664a0 0x7f6634068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:53.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.067+0000 7f664b7fe640 1 -- 172.21.15.57:0/2439453075 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f663c084ac0 con 0x7f66540cb7d0 2023-10-30T14:43:53.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.067+0000 7f6652ffd640 1 --2- 172.21.15.57:0/2439453075 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66340664a0 0x7f6634068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:53.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.067+0000 7f6652ffd640 1 --2- 172.21.15.57:0/2439453075 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66340664a0 0x7f6634068960 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f66440b2000 tx=0x7f6644008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:53.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.071+0000 7f664b7fe640 1 -- 172.21.15.57:0/2439453075 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6654159fa0 con 0x7f66540cb7d0 2023-10-30T14:43:53.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.263+0000 7f66593a1640 1 -- 172.21.15.57:0/2439453075 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6654159fa0 con 0x7f66540cb7d0 2023-10-30T14:43:53.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.263+0000 7f664b7fe640 1 -- 172.21.15.57:0/2439453075 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+6531 (secure 0 0 0) 0x7f6654159fa0 con 0x7f66540cb7d0 2023-10-30T14:43:53.268 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:53.268 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:44.163921+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:53.269 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 161 2023-10-30T14:43:53.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.263+0000 7f66593a1640 1 -- 172.21.15.57:0/2439453075 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66340664a0 msgr2=0x7f6634068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:53.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.263+0000 7f66593a1640 1 --2- 172.21.15.57:0/2439453075 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66340664a0 0x7f6634068960 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f66440b2000 tx=0x7f6644008040 comp rx=0 tx=0).stop 2023-10-30T14:43:53.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.263+0000 7f66593a1640 1 -- 172.21.15.57:0/2439453075 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66540cb7d0 msgr2=0x7f66541577c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:53.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.263+0000 7f66593a1640 1 --2- 172.21.15.57:0/2439453075 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66540cb7d0 0x7f66541577c0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f663c00e9d0 tx=0x7f663c00ee90 comp rx=0 tx=0).stop 2023-10-30T14:43:53.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.263+0000 7f66593a1640 1 -- 172.21.15.57:0/2439453075 shutdown_connections 2023-10-30T14:43:53.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.263+0000 7f66593a1640 1 --2- 172.21.15.57:0/2439453075 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66340664a0 0x7f6634068960 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:53.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.263+0000 7f66593a1640 1 --2- 172.21.15.57:0/2439453075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6654150830 0x7f66541e3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:53.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.263+0000 7f66593a1640 1 --2- 172.21.15.57:0/2439453075 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66540cb7d0 0x7f66541577c0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:53.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.263+0000 7f66593a1640 1 --2- 172.21.15.57:0/2439453075 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66540c3e40 0x7f6654145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:53.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.263+0000 7f66593a1640 1 -- 172.21.15.57:0/2439453075 >> 172.21.15.57:0/2439453075 conn(0x7f66540cac70 msgr2=0x7f66540733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:53.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.263+0000 7f66593a1640 1 -- 172.21.15.57:0/2439453075 shutdown_connections 2023-10-30T14:43:53.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:53.263+0000 7f66593a1640 1 -- 172.21.15.57:0/2439453075 wait complete. 2023-10-30T14:43:53.288 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:43:55.290 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:55.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.439+0000 7f01f010e640 1 Processor -- start 2023-10-30T14:43:55.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.439+0000 7f01f010e640 1 -- start start 2023-10-30T14:43:55.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.439+0000 7f01f010e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f01e806dbe0 0x7f01e806e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:55.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.439+0000 7f01f010e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f01e8071f30 0x7f01e8156880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:55.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.439+0000 7f01f010e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e8156dc0 con 0x7f01e806dfc0 2023-10-30T14:43:55.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.439+0000 7f01f010e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01e806ea70 con 0x7f01e806dbe0 2023-10-30T14:43:55.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.439+0000 7f01f010e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01e8070c40 con 0x7f01e8071f30 2023-10-30T14:43:55.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.439+0000 7f01edeaa640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f01e806dbe0 0x7f01e806e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:55.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.439+0000 7f01ed6a9640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f01e806dfc0 0x7f01e8157270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:40178/0 (socket says 172.21.15.57:40178) 2023-10-30T14:43:55.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.439+0000 7f01ed6a9640 1 -- 172.21.15.57:0/1654867393 learned_addr learned my addr 172.21.15.57:0/1654867393 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:43:55.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.439+0000 7f01ee6ab640 1 --2- 172.21.15.57:0/1654867393 >> v2:172.21.15.88:3300/0 conn(0x7f01e8071f30 0x7f01e8156880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:55.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.443+0000 7f01edeaa640 1 -- 172.21.15.57:0/1654867393 >> v2:172.21.15.88:3300/0 conn(0x7f01e8071f30 msgr2=0x7f01e8156880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:55.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.443+0000 7f01edeaa640 1 --2- 172.21.15.57:0/1654867393 >> v2:172.21.15.88:3300/0 conn(0x7f01e8071f30 0x7f01e8156880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:55.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.443+0000 7f01edeaa640 1 -- 172.21.15.57:0/1654867393 >> v1:172.21.15.57:6789/0 conn(0x7f01e806dfc0 legacy=0x7f01e8157270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:55.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.443+0000 7f01ecea8640 1 -- 172.21.15.57:0/1654867393 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345589345 0 0) 0x7f01d8001120 con 0x7f01e806dfc0 2023-10-30T14:43:55.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.443+0000 7f01edeaa640 1 -- 172.21.15.57:0/1654867393 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e8070e90 con 0x7f01e806dbe0 2023-10-30T14:43:55.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.443+0000 7f01edeaa640 1 --2- 172.21.15.57:0/1654867393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e806dbe0 0x7f01e806e530 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f01e80c3260 tx=0x7f01e401d800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf4d2d454a7c7802 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:55.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.443+0000 7f01ecea8640 1 -- 172.21.15.57:0/1654867393 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e40a6830 con 0x7f01e806dbe0 2023-10-30T14:43:55.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.443+0000 7f01ecea8640 1 -- 172.21.15.57:0/1654867393 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e40a6990 con 0x7f01e806dbe0 2023-10-30T14:43:55.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.443+0000 7f01ecea8640 1 -- 172.21.15.57:0/1654867393 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e40a6d20 con 0x7f01e806dbe0 2023-10-30T14:43:55.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.443+0000 7f01f010e640 1 -- 172.21.15.57:0/1654867393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e806dbe0 msgr2=0x7f01e806e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:55.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.443+0000 7f01f010e640 1 --2- 172.21.15.57:0/1654867393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e806dbe0 0x7f01e806e530 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f01e80c3260 tx=0x7f01e401d800 comp rx=0 tx=0).stop 2023-10-30T14:43:55.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01f010e640 1 -- 172.21.15.57:0/1654867393 shutdown_connections 2023-10-30T14:43:55.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01f010e640 1 --2- 172.21.15.57:0/1654867393 >> v2:172.21.15.88:3300/0 conn(0x7f01e8071f30 0x7f01e8156880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:55.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01f010e640 1 --2- 172.21.15.57:0/1654867393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e806dbe0 0x7f01e806e530 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:55.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01f010e640 1 -- 172.21.15.57:0/1654867393 >> 172.21.15.57:0/1654867393 conn(0x7f01e80c7f60 msgr2=0x7f01e80c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:55.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01f010e640 1 -- 172.21.15.57:0/1654867393 shutdown_connections 2023-10-30T14:43:55.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01f010e640 1 -- 172.21.15.57:0/1654867393 wait complete. 2023-10-30T14:43:55.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01f010e640 1 Processor -- start 2023-10-30T14:43:55.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01f010e640 1 -- start start 2023-10-30T14:43:55.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01f010e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e806dbe0 0x7f01e81492f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:55.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01f010e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01e806dfc0 0x7f01e81f59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:55.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01f010e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f01e8071f30 0x7f01e8142e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:55.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01f010e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e8159770 con 0x7f01e806dbe0 2023-10-30T14:43:55.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01f010e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e80c4dc0 con 0x7f01e806dfc0 2023-10-30T14:43:55.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01f010e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e8153f60 con 0x7f01e8071f30 2023-10-30T14:43:55.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01edeaa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e806dbe0 0x7f01e81492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:55.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01edeaa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e806dbe0 0x7f01e81492f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50798/0 (socket says 172.21.15.57:50798) 2023-10-30T14:43:55.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01edeaa640 1 -- 172.21.15.57:0/2885659346 learned_addr learned my addr 172.21.15.57:0/2885659346 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:55.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01ed6a9640 1 --2- 172.21.15.57:0/2885659346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01e806dfc0 0x7f01e81f59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:55.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01ee6ab640 1 --2- 172.21.15.57:0/2885659346 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f01e8071f30 0x7f01e8142e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:55.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01edeaa640 1 -- 172.21.15.57:0/2885659346 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f01e8071f30 msgr2=0x7f01e8142e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:55.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01edeaa640 1 --2- 172.21.15.57:0/2885659346 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f01e8071f30 0x7f01e8142e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:55.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01edeaa640 1 -- 172.21.15.57:0/2885659346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01e806dfc0 msgr2=0x7f01e81f59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:55.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01edeaa640 1 --2- 172.21.15.57:0/2885659346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01e806dfc0 0x7f01e81f59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:55.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.447+0000 7f01edeaa640 1 -- 172.21.15.57:0/2885659346 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e400a5c0 con 0x7f01e806dbe0 2023-10-30T14:43:55.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.451+0000 7f01edeaa640 1 --2- 172.21.15.57:0/2885659346 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e806dbe0 0x7f01e81492f0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f01e40b1040 tx=0x7f01e401a5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:55.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.451+0000 7f01de7fc640 1 -- 172.21.15.57:0/2885659346 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e40ae850 con 0x7f01e806dbe0 2023-10-30T14:43:55.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.451+0000 7f01de7fc640 1 -- 172.21.15.57:0/2885659346 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e40ae9d0 con 0x7f01e806dbe0 2023-10-30T14:43:55.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.451+0000 7f01de7fc640 1 -- 172.21.15.57:0/2885659346 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e40aed80 con 0x7f01e806dbe0 2023-10-30T14:43:55.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.451+0000 7f01f010e640 1 -- 172.21.15.57:0/2885659346 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e80c5010 con 0x7f01e806dbe0 2023-10-30T14:43:55.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.451+0000 7f01f010e640 1 -- 172.21.15.57:0/2885659346 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e81f54a0 con 0x7f01e806dbe0 2023-10-30T14:43:55.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.451+0000 7f01f010e640 1 -- 172.21.15.57:0/2885659346 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e80740f0 con 0x7f01e806dbe0 2023-10-30T14:43:55.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.451+0000 7f01de7fc640 1 -- 172.21.15.57:0/2885659346 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f01e40b5040 con 0x7f01e806dbe0 2023-10-30T14:43:55.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.451+0000 7f01de7fc640 1 --2- 172.21.15.57:0/2885659346 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f01cc0664d0 0x7f01cc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:55.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.451+0000 7f01de7fc640 1 -- 172.21.15.57:0/2885659346 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f01e411e900 con 0x7f01e806dbe0 2023-10-30T14:43:55.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.455+0000 7f01ed6a9640 1 --2- 172.21.15.57:0/2885659346 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f01cc0664d0 0x7f01cc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:55.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.455+0000 7f01ed6a9640 1 --2- 172.21.15.57:0/2885659346 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f01cc0664d0 0x7f01cc068990 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f01e814a930 tx=0x7f01d8007ba0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:55.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.459+0000 7f01de7fc640 1 -- 172.21.15.57:0/2885659346 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f01e40eb760 con 0x7f01e806dbe0 2023-10-30T14:43:55.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.647+0000 7f01f010e640 1 -- 172.21.15.57:0/2885659346 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01e804e170 con 0x7f01e806dbe0 2023-10-30T14:43:55.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.647+0000 7f01de7fc640 1 -- 172.21.15.57:0/2885659346 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+6531 (secure 0 0 0) 0x7f01e40eed80 con 0x7f01e806dbe0 2023-10-30T14:43:55.795 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:55.795 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:44.163921+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:55.797 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 161 2023-10-30T14:43:55.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.647+0000 7f01f010e640 1 -- 172.21.15.57:0/2885659346 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f01cc0664d0 msgr2=0x7f01cc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:55.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.647+0000 7f01f010e640 1 --2- 172.21.15.57:0/2885659346 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f01cc0664d0 0x7f01cc068990 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f01e814a930 tx=0x7f01d8007ba0 comp rx=0 tx=0).stop 2023-10-30T14:43:55.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.647+0000 7f01f010e640 1 -- 172.21.15.57:0/2885659346 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e806dbe0 msgr2=0x7f01e81492f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:55.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.647+0000 7f01f010e640 1 --2- 172.21.15.57:0/2885659346 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e806dbe0 0x7f01e81492f0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f01e40b1040 tx=0x7f01e401a5a0 comp rx=0 tx=0).stop 2023-10-30T14:43:55.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.651+0000 7f01f010e640 1 -- 172.21.15.57:0/2885659346 shutdown_connections 2023-10-30T14:43:55.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.651+0000 7f01f010e640 1 --2- 172.21.15.57:0/2885659346 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f01cc0664d0 0x7f01cc068990 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:55.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.651+0000 7f01f010e640 1 --2- 172.21.15.57:0/2885659346 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f01e8071f30 0x7f01e8142e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:55.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.651+0000 7f01f010e640 1 --2- 172.21.15.57:0/2885659346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01e806dfc0 0x7f01e81f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:55.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.651+0000 7f01f010e640 1 --2- 172.21.15.57:0/2885659346 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e806dbe0 0x7f01e81492f0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:55.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.651+0000 7f01f010e640 1 -- 172.21.15.57:0/2885659346 >> 172.21.15.57:0/2885659346 conn(0x7f01e80c7f60 msgr2=0x7f01e814a980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:55.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.651+0000 7f01f010e640 1 -- 172.21.15.57:0/2885659346 shutdown_connections 2023-10-30T14:43:55.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:55.651+0000 7f01f010e640 1 -- 172.21.15.57:0/2885659346 wait complete. 2023-10-30T14:43:55.800 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:43:57.801 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:43:57.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.951+0000 7f16f3901640 1 Processor -- start 2023-10-30T14:43:57.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.951+0000 7f16f3901640 1 -- start start 2023-10-30T14:43:57.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.951+0000 7f16f3901640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f16ec06b760 0x7f16ec15e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:57.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.951+0000 7f16f3901640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16ec06c420 0x7f16ec0c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:57.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.951+0000 7f16f3901640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ec071e00 con 0x7f16ec06bb40 2023-10-30T14:43:57.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.951+0000 7f16f3901640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16ec070020 con 0x7f16ec06b760 2023-10-30T14:43:57.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.951+0000 7f16f3901640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec15eb90 con 0x7f16ec06c420 2023-10-30T14:43:57.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.951+0000 7f16f0e9c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f16ec06bb40 0x7f16ec157a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:40188/0 (socket says 172.21.15.57:40188) 2023-10-30T14:43:57.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.951+0000 7f16f0e9c640 1 -- 172.21.15.57:0/3578413845 learned_addr learned my addr 172.21.15.57:0/3578413845 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:43:57.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.955+0000 7f16f169d640 1 --2- 172.21.15.57:0/3578413845 >> v2:172.21.15.88:3300/0 conn(0x7f16ec06b760 0x7f16ec15e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:57.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.955+0000 7f16f1e9e640 1 --2- 172.21.15.57:0/3578413845 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16ec06c420 0x7f16ec0c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:57.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.955+0000 7f16e3fff640 1 -- 172.21.15.57:0/3578413845 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510825557 0 0) 0x7f16ec071e00 con 0x7f16ec06bb40 2023-10-30T14:43:57.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.955+0000 7f16e3fff640 1 -- 172.21.15.57:0/3578413845 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16d8003530 con 0x7f16ec06bb40 2023-10-30T14:43:57.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.955+0000 7f16e3fff640 1 -- 172.21.15.57:0/3578413845 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2198888878 0 0) 0x7f16d8003530 con 0x7f16ec06bb40 2023-10-30T14:43:57.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.955+0000 7f16f169d640 1 -- 172.21.15.57:0/3578413845 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16ec06c420 msgr2=0x7f16ec0c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:43:57.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.955+0000 7f16f169d640 1 --2- 172.21.15.57:0/3578413845 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16ec06c420 0x7f16ec0c57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:57.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.955+0000 7f16f169d640 1 -- 172.21.15.57:0/3578413845 >> v1:172.21.15.57:6789/0 conn(0x7f16ec06bb40 legacy=0x7f16ec157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:57.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.955+0000 7f16f169d640 1 -- 172.21.15.57:0/3578413845 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ec070f10 con 0x7f16ec06b760 2023-10-30T14:43:57.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.955+0000 7f16f1e9e640 1 --2- 172.21.15.57:0/3578413845 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16ec06c420 0x7f16ec0c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:43:57.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.955+0000 7f16f169d640 1 --2- 172.21.15.57:0/3578413845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16ec06b760 0x7f16ec15e650 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f16e80097c0 tx=0x7f16e801c370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b79cc1fc4a971205 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:57.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.955+0000 7f16e3fff640 1 -- 172.21.15.57:0/3578413845 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16dc001680 con 0x7f16ec06b760 2023-10-30T14:43:57.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16e3fff640 1 -- 172.21.15.57:0/3578413845 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e80a7050 con 0x7f16ec06b760 2023-10-30T14:43:57.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16e3fff640 1 -- 172.21.15.57:0/3578413845 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e80ab620 con 0x7f16ec06b760 2023-10-30T14:43:57.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f3901640 1 -- 172.21.15.57:0/3578413845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16ec06b760 msgr2=0x7f16ec15e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:57.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f3901640 1 --2- 172.21.15.57:0/3578413845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16ec06b760 0x7f16ec15e650 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f16e80097c0 tx=0x7f16e801c370 comp rx=0 tx=0).stop 2023-10-30T14:43:57.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f3901640 1 -- 172.21.15.57:0/3578413845 shutdown_connections 2023-10-30T14:43:57.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f3901640 1 --2- 172.21.15.57:0/3578413845 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16ec06c420 0x7f16ec0c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:57.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f3901640 1 --2- 172.21.15.57:0/3578413845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16ec06b760 0x7f16ec15e650 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:57.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f3901640 1 -- 172.21.15.57:0/3578413845 >> 172.21.15.57:0/3578413845 conn(0x7f16ec0cac70 msgr2=0x7f16ec0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:57.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f3901640 1 -- 172.21.15.57:0/3578413845 shutdown_connections 2023-10-30T14:43:57.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f3901640 1 -- 172.21.15.57:0/3578413845 wait complete. 2023-10-30T14:43:57.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f3901640 1 Processor -- start 2023-10-30T14:43:57.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f3901640 1 -- start start 2023-10-30T14:43:57.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f3901640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16ec06b760 0x7f16ec15b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:57.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f3901640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16ec06bb40 0x7f16ec0c6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:57.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f3901640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f16ec06c420 0x7f16ec1f6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:57.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f3901640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec15bdb0 con 0x7f16ec06c420 2023-10-30T14:43:57.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f3901640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec15bab0 con 0x7f16ec06bb40 2023-10-30T14:43:57.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f3901640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec0c4580 con 0x7f16ec06b760 2023-10-30T14:43:57.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f1e9e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f16ec06c420 0x7f16ec1f6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:57.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f1e9e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f16ec06c420 0x7f16ec1f6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50804/0 (socket says 172.21.15.57:50804) 2023-10-30T14:43:57.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f1e9e640 1 -- 172.21.15.57:0/4218329747 learned_addr learned my addr 172.21.15.57:0/4218329747 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:43:57.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f0e9c640 1 --2- 172.21.15.57:0/4218329747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16ec06bb40 0x7f16ec0c6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:57.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.959+0000 7f16f169d640 1 --2- 172.21.15.57:0/4218329747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16ec06b760 0x7f16ec15b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:57.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.963+0000 7f16f1e9e640 1 -- 172.21.15.57:0/4218329747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16ec06b760 msgr2=0x7f16ec15b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:57.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.963+0000 7f16f1e9e640 1 --2- 172.21.15.57:0/4218329747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16ec06b760 0x7f16ec15b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:57.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.963+0000 7f16f1e9e640 1 -- 172.21.15.57:0/4218329747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16ec06bb40 msgr2=0x7f16ec0c6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:57.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.963+0000 7f16f1e9e640 1 --2- 172.21.15.57:0/4218329747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16ec06bb40 0x7f16ec0c6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:57.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.963+0000 7f16f1e9e640 1 -- 172.21.15.57:0/4218329747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e4009a70 con 0x7f16ec06c420 2023-10-30T14:43:57.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.963+0000 7f16f1e9e640 1 --2- 172.21.15.57:0/4218329747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f16ec06c420 0x7f16ec1f6020 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f16e400efc0 tx=0x7f16e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:57.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.963+0000 7f16e1ffb640 1 -- 172.21.15.57:0/4218329747 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4009cd0 con 0x7f16ec06c420 2023-10-30T14:43:57.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.963+0000 7f16e1ffb640 1 -- 172.21.15.57:0/4218329747 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e400f040 con 0x7f16ec06c420 2023-10-30T14:43:57.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.963+0000 7f16e1ffb640 1 -- 172.21.15.57:0/4218329747 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e40039a0 con 0x7f16ec06c420 2023-10-30T14:43:57.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.963+0000 7f16f3901640 1 -- 172.21.15.57:0/4218329747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16e8009420 con 0x7f16ec06c420 2023-10-30T14:43:57.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.963+0000 7f16f3901640 1 -- 172.21.15.57:0/4218329747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ec0ca990 con 0x7f16ec06c420 2023-10-30T14:43:57.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.963+0000 7f16f3901640 1 -- 172.21.15.57:0/4218329747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ec15bf60 con 0x7f16ec06c420 2023-10-30T14:43:57.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.963+0000 7f16e1ffb640 1 -- 172.21.15.57:0/4218329747 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f16e4014480 con 0x7f16ec06c420 2023-10-30T14:43:57.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.967+0000 7f16e1ffb640 1 --2- 172.21.15.57:0/4218329747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f16d00664a0 0x7f16d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:43:57.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.967+0000 7f16e1ffb640 1 -- 172.21.15.57:0/4218329747 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f16e4084f60 con 0x7f16ec06c420 2023-10-30T14:43:57.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.967+0000 7f16f169d640 1 --2- 172.21.15.57:0/4218329747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f16d00664a0 0x7f16d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:43:57.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.967+0000 7f16f169d640 1 --2- 172.21.15.57:0/4218329747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f16d00664a0 0x7f16d0068960 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f16e80ad000 tx=0x7f16e80a6000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:43:57.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:57.971+0000 7f16e1ffb640 1 -- 172.21.15.57:0/4218329747 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f16ec15bf60 con 0x7f16ec06c420 2023-10-30T14:43:58.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:58.159+0000 7f16f3901640 1 -- 172.21.15.57:0/4218329747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16ec15bc20 con 0x7f16ec06c420 2023-10-30T14:43:58.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:58.159+0000 7f16e1ffb640 1 -- 172.21.15.57:0/4218329747 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+6531 (secure 0 0 0) 0x7f16e4018090 con 0x7f16ec06c420 2023-10-30T14:43:58.164 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:43:58.164 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:44.163921+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6356,"mds_1":6747,"mds_2":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7056":{"gid":7056,"name":"e","rank":2,"incarnation":137,"state":"up:stopping","state_seq":24,"addr":"172.21.15.57:6837/2307041430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2307041430},{"type":"v1","addr":"172.21.15.57:6837","nonce":2307041430}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:43:58.165 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 161 2023-10-30T14:43:58.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:58.159+0000 7f16f3901640 1 -- 172.21.15.57:0/4218329747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f16d00664a0 msgr2=0x7f16d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:58.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:58.159+0000 7f16f3901640 1 --2- 172.21.15.57:0/4218329747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f16d00664a0 0x7f16d0068960 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f16e80ad000 tx=0x7f16e80a6000 comp rx=0 tx=0).stop 2023-10-30T14:43:58.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:58.159+0000 7f16f3901640 1 -- 172.21.15.57:0/4218329747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f16ec06c420 msgr2=0x7f16ec1f6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:43:58.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:58.159+0000 7f16f3901640 1 --2- 172.21.15.57:0/4218329747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f16ec06c420 0x7f16ec1f6020 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f16e400efc0 tx=0x7f16e400c490 comp rx=0 tx=0).stop 2023-10-30T14:43:58.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:58.159+0000 7f16f3901640 1 -- 172.21.15.57:0/4218329747 shutdown_connections 2023-10-30T14:43:58.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:58.159+0000 7f16f3901640 1 --2- 172.21.15.57:0/4218329747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f16d00664a0 0x7f16d0068960 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:58.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:58.159+0000 7f16f3901640 1 --2- 172.21.15.57:0/4218329747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f16ec06c420 0x7f16ec1f6020 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:58.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:58.159+0000 7f16f3901640 1 --2- 172.21.15.57:0/4218329747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16ec06bb40 0x7f16ec0c6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:58.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:58.159+0000 7f16f3901640 1 --2- 172.21.15.57:0/4218329747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f16ec06b760 0x7f16ec15b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:43:58.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:58.159+0000 7f16f3901640 1 -- 172.21.15.57:0/4218329747 >> 172.21.15.57:0/4218329747 conn(0x7f16ec0cac70 msgr2=0x7f16ec154bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:43:58.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:58.159+0000 7f16f3901640 1 -- 172.21.15.57:0/4218329747 shutdown_connections 2023-10-30T14:43:58.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:43:58.159+0000 7f16f3901640 1 -- 172.21.15.57:0/4218329747 wait complete. 2023-10-30T14:43:58.185 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:43:58.302 INFO:tasks.ceph.mds.e.smithi057.stdout:starting mds.e at 2023-10-30T14:44:00.186 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:00.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.335+0000 7f38857f2640 1 Processor -- start 2023-10-30T14:44:00.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.335+0000 7f38857f2640 1 -- start start 2023-10-30T14:44:00.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f38857f2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38800c8eb0 0x7f388015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:00.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f38857f2640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388015dcc0 con 0x7f388015ac70 2023-10-30T14:44:00.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f38857f2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3880159d80 con 0x7f388006aeb0 2023-10-30T14:44:00.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f38857f2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3880159800 con 0x7f38800c8eb0 2023-10-30T14:44:00.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387f7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f388015ac70 0x7f388015a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:40200/0 (socket says 172.21.15.57:40200) 2023-10-30T14:44:00.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387f7fe640 1 -- 172.21.15.57:0/3564336157 learned_addr learned my addr 172.21.15.57:0/3564336157 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:44:00.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387ffff640 1 --2- 172.21.15.57:0/3564336157 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38800c8eb0 0x7f388015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:00.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387effd640 1 -- 172.21.15.57:0/3564336157 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974720467 0 0) 0x7f3874001120 con 0x7f388015ac70 2023-10-30T14:44:00.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387effd640 1 -- 172.21.15.57:0/3564336157 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3868003530 con 0x7f388015ac70 2023-10-30T14:44:00.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387effd640 1 -- 172.21.15.57:0/3564336157 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821040768 0 0) 0x7f3880159d80 con 0x7f388006aeb0 2023-10-30T14:44:00.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387effd640 1 -- 172.21.15.57:0/3564336157 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3874001120 con 0x7f388006aeb0 2023-10-30T14:44:00.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387effd640 1 -- 172.21.15.57:0/3564336157 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3824264470 0 0) 0x7f3868003530 con 0x7f388015ac70 2023-10-30T14:44:00.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387effd640 1 -- 172.21.15.57:0/3564336157 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38800c8eb0 msgr2=0x7f388015cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:00.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387effd640 1 --2- 172.21.15.57:0/3564336157 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38800c8eb0 0x7f388015cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387effd640 1 -- 172.21.15.57:0/3564336157 >> v1:172.21.15.88:6789/0 conn(0x7f388006aeb0 legacy=0x7f3880069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:00.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387effd640 1 -- 172.21.15.57:0/3564336157 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388015d420 con 0x7f388015ac70 2023-10-30T14:44:00.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387effd640 1 -- 172.21.15.57:0/3564336157 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f3874001810 con 0x7f388015ac70 2023-10-30T14:44:00.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387effd640 1 -- 172.21.15.57:0/3564336157 >> v1:172.21.15.57:6789/0 conn(0x7f388015ac70 legacy=0x7f388015a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:00.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387effd640 1 --2- 172.21.15.57:0/3564336157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3868003b40 0x7f3868003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:00.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387effd640 1 --2- 172.21.15.57:0/3564336157 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38800c8eb0 0x7f38680047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:00.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387effd640 1 --2- 172.21.15.57:0/3564336157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3868004d10 0x7f3868005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:00.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387effd640 1 -- 172.21.15.57:0/3564336157 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3880159800 con 0x7f3868004d10 2023-10-30T14:44:00.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387effd640 1 -- 172.21.15.57:0/3564336157 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386800ba40 con 0x7f3868003b40 2023-10-30T14:44:00.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387effd640 1 -- 172.21.15.57:0/3564336157 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386800bbb0 con 0x7f38800c8eb0 2023-10-30T14:44:00.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f3884ff1640 1 --2- 172.21.15.57:0/3564336157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3868004d10 0x7f3868005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:00.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387ffff640 1 --2- 172.21.15.57:0/3564336157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3868003b40 0x7f3868003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:00.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f387f7fe640 1 --2- 172.21.15.57:0/3564336157 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38800c8eb0 0x7f38680047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:00.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f3884ff1640 1 -- 172.21.15.57:0/3564336157 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38800c8eb0 msgr2=0x7f38680047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:00.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f3884ff1640 1 --2- 172.21.15.57:0/3564336157 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38800c8eb0 0x7f38680047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f3884ff1640 1 -- 172.21.15.57:0/3564336157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3868003b40 msgr2=0x7f3868003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:00.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f3884ff1640 1 --2- 172.21.15.57:0/3564336157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3868003b40 0x7f3868003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.339+0000 7f3884ff1640 1 -- 172.21.15.57:0/3564336157 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3868003530 con 0x7f3868004d10 2023-10-30T14:44:00.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.343+0000 7f3884ff1640 1 --2- 172.21.15.57:0/3564336157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3868004d10 0x7f3868005190 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f3878001fd0 tx=0x7f387801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f375f95f3eb9bff9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:00.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.343+0000 7f387effd640 1 -- 172.21.15.57:0/3564336157 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3878003b80 con 0x7f3868004d10 2023-10-30T14:44:00.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.343+0000 7f387effd640 1 -- 172.21.15.57:0/3564336157 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3878003d00 con 0x7f3868004d10 2023-10-30T14:44:00.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.343+0000 7f387effd640 1 -- 172.21.15.57:0/3564336157 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3878005310 con 0x7f3868004d10 2023-10-30T14:44:00.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.343+0000 7f38857f2640 1 -- 172.21.15.57:0/3564336157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3868004d10 msgr2=0x7f3868005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:00.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.343+0000 7f38857f2640 1 --2- 172.21.15.57:0/3564336157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3868004d10 0x7f3868005190 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f3878001fd0 tx=0x7f387801c8e0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.343+0000 7f3884ff1640 1 -- 172.21.15.57:0/3564336157 reap_dead start 2023-10-30T14:44:00.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.343+0000 7f38857f2640 1 -- 172.21.15.57:0/3564336157 shutdown_connections 2023-10-30T14:44:00.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.343+0000 7f38857f2640 1 -- 172.21.15.57:0/3564336157 >> 172.21.15.57:0/3564336157 conn(0x7f38800cef00 msgr2=0x7f38800cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:00.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.343+0000 7f38857f2640 1 -- 172.21.15.57:0/3564336157 shutdown_connections 2023-10-30T14:44:00.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.343+0000 7f38857f2640 1 -- 172.21.15.57:0/3564336157 wait complete. 2023-10-30T14:44:00.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.343+0000 7f38857f2640 1 Processor -- start 2023-10-30T14:44:00.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f38857f2640 1 -- start start 2023-10-30T14:44:00.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f38857f2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3868004d10 0x7f38800c5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:00.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f38857f2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38801f1db0 0x7f388014d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:00.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f38857f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3880156a70 0x7f3880152800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:00.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f38857f2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38801f21f0 con 0x7f3880156a70 2023-10-30T14:44:00.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f38857f2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3880154720 con 0x7f3868004d10 2023-10-30T14:44:00.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f38857f2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3880154a00 con 0x7f38801f1db0 2023-10-30T14:44:00.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f3884ff1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3880156a70 0x7f3880152800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:00.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f3884ff1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3880156a70 0x7f3880152800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50864/0 (socket says 172.21.15.57:50864) 2023-10-30T14:44:00.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f3884ff1640 1 -- 172.21.15.57:0/4267719398 learned_addr learned my addr 172.21.15.57:0/4267719398 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:00.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f387ffff640 1 --2- 172.21.15.57:0/4267719398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3868004d10 0x7f38800c5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:00.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f387f7fe640 1 --2- 172.21.15.57:0/4267719398 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38801f1db0 0x7f388014d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:00.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f3884ff1640 1 -- 172.21.15.57:0/4267719398 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38801f1db0 msgr2=0x7f388014d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:00.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f3884ff1640 1 --2- 172.21.15.57:0/4267719398 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38801f1db0 0x7f388014d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f3884ff1640 1 -- 172.21.15.57:0/4267719398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3868004d10 msgr2=0x7f38800c5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:00.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f3884ff1640 1 --2- 172.21.15.57:0/4267719398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3868004d10 0x7f38800c5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f3884ff1640 1 -- 172.21.15.57:0/4267719398 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3880154ce0 con 0x7f3880156a70 2023-10-30T14:44:00.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f387ffff640 1 --2- 172.21.15.57:0/4267719398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3868004d10 0x7f38800c5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:44:00.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f3884ff1640 1 --2- 172.21.15.57:0/4267719398 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3880156a70 0x7f3880152800 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f3878001fd0 tx=0x7f3868005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:00.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f387cff9640 1 -- 172.21.15.57:0/4267719398 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3878003b90 con 0x7f3880156a70 2023-10-30T14:44:00.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f387cff9640 1 -- 172.21.15.57:0/4267719398 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38780014e0 con 0x7f3880156a70 2023-10-30T14:44:00.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f387cff9640 1 -- 172.21.15.57:0/4267719398 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38780057f0 con 0x7f3880156a70 2023-10-30T14:44:00.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f38857f2640 1 -- 172.21.15.57:0/4267719398 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38800c4a00 con 0x7f3880156a70 2023-10-30T14:44:00.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f38857f2640 1 -- 172.21.15.57:0/4267719398 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f388014dd50 con 0x7f3880156a70 2023-10-30T14:44:00.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.347+0000 7f38857f2640 1 -- 172.21.15.57:0/4267719398 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f388014dd50 con 0x7f3880156a70 2023-10-30T14:44:00.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.351+0000 7f387cff9640 1 -- 172.21.15.57:0/4267719398 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f38780a55d0 con 0x7f3880156a70 2023-10-30T14:44:00.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.351+0000 7f387cff9640 1 --2- 172.21.15.57:0/4267719398 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f386c0664a0 0x7f386c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:00.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.351+0000 7f387cff9640 1 -- 172.21.15.57:0/4267719398 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f387811e730 con 0x7f3880156a70 2023-10-30T14:44:00.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.351+0000 7f387ffff640 1 --2- 172.21.15.57:0/4267719398 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f386c0664a0 0x7f386c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:00.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.351+0000 7f387ffff640 1 --2- 172.21.15.57:0/4267719398 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f386c0664a0 0x7f386c068960 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3870009be0 tx=0x7f3870008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:00.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.359+0000 7f387cff9640 1 -- 172.21.15.57:0/4267719398 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f38780eb620 con 0x7f3880156a70 2023-10-30T14:44:00.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.559+0000 7f38857f2640 1 -- 172.21.15.57:0/4267719398 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f388014dd50 con 0x7f3880156a70 2023-10-30T14:44:00.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.559+0000 7f387cff9640 1 -- 172.21.15.57:0/4267719398 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+6509 (secure 0 0 0) 0x7f38780eec40 con 0x7f3880156a70 2023-10-30T14:44:00.566 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:00.567 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7536,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:57.246176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6356,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:00.567 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 163 2023-10-30T14:44:00.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.563+0000 7f38857f2640 1 -- 172.21.15.57:0/4267719398 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f386c0664a0 msgr2=0x7f386c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:00.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.563+0000 7f38857f2640 1 --2- 172.21.15.57:0/4267719398 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f386c0664a0 0x7f386c068960 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3870009be0 tx=0x7f3870008040 comp rx=0 tx=0).stop 2023-10-30T14:44:00.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.563+0000 7f38857f2640 1 -- 172.21.15.57:0/4267719398 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3880156a70 msgr2=0x7f3880152800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:00.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.563+0000 7f38857f2640 1 --2- 172.21.15.57:0/4267719398 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3880156a70 0x7f3880152800 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f3878001fd0 tx=0x7f3868005800 comp rx=0 tx=0).stop 2023-10-30T14:44:00.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.563+0000 7f38857f2640 1 -- 172.21.15.57:0/4267719398 shutdown_connections 2023-10-30T14:44:00.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.563+0000 7f38857f2640 1 --2- 172.21.15.57:0/4267719398 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f386c0664a0 0x7f386c068960 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.563+0000 7f38857f2640 1 --2- 172.21.15.57:0/4267719398 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3880156a70 0x7f3880152800 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.563+0000 7f38857f2640 1 --2- 172.21.15.57:0/4267719398 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38801f1db0 0x7f388014d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.563+0000 7f38857f2640 1 --2- 172.21.15.57:0/4267719398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3868004d10 0x7f38800c5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.563+0000 7f38857f2640 1 -- 172.21.15.57:0/4267719398 >> 172.21.15.57:0/4267719398 conn(0x7f38800cef00 msgr2=0x7f38800cadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:00.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.563+0000 7f38857f2640 1 -- 172.21.15.57:0/4267719398 shutdown_connections 2023-10-30T14:44:00.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.563+0000 7f38857f2640 1 -- 172.21.15.57:0/4267719398 wait complete. 2023-10-30T14:44:00.588 INFO:tasks.mds_thrash.fs.[cephfs]:mds cluster has 2 alive and active, now stable! 2023-10-30T14:44:00.588 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2023-10-30T14:44:00.588 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T14:44:00.589 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:44:00.615 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T14:44:00.616 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:00.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668d1f2640 1 Processor -- start 2023-10-30T14:44:00.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668d1f2640 1 -- start start 2023-10-30T14:44:00.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668d1f2640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6688153b50 0x7f66881597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:00.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668d1f2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f66880c62b0 0x7f66880ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:00.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668d1f2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66880cb180 0x7f668806ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:00.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668d1f2640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6688159cf0 con 0x7f66880c62b0 2023-10-30T14:44:00.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668d1f2640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f668806aa30 con 0x7f6688153b50 2023-10-30T14:44:00.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668d1f2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f668815a770 con 0x7f66880cb180 2023-10-30T14:44:00.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f66877fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f66880c62b0 0x7f66880ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:00.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f66877fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f66880c62b0 0x7f66880ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50868/0 (socket says 172.21.15.57:50868) 2023-10-30T14:44:00.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f66877fe640 1 -- 172.21.15.57:0/526942022 learned_addr learned my addr 172.21.15.57:0/526942022 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:00.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f6687fff640 1 --2- 172.21.15.57:0/526942022 >> v2:172.21.15.88:3300/0 conn(0x7f6688153b50 0x7f66881597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:00.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668c9f1640 1 --2- 172.21.15.57:0/526942022 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66880cb180 0x7f668806ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:00.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f66877fe640 1 -- 172.21.15.57:0/526942022 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66880cb180 msgr2=0x7f668806ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:00.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f66877fe640 1 --2- 172.21.15.57:0/526942022 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66880cb180 0x7f668806ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f66877fe640 1 -- 172.21.15.57:0/526942022 >> v2:172.21.15.88:3300/0 conn(0x7f6688153b50 msgr2=0x7f66881597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:00.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f66877fe640 1 --2- 172.21.15.57:0/526942022 >> v2:172.21.15.88:3300/0 conn(0x7f6688153b50 0x7f66881597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f66877fe640 1 -- 172.21.15.57:0/526942022 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66880c9d40 con 0x7f66880c62b0 2023-10-30T14:44:00.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f66877fe640 1 --2- 172.21.15.57:0/526942022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66880c62b0 0x7f66880ca1d0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f6688066890 tx=0x7f667c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=935a8b5d6d7f8438 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:00.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f6686ffd640 1 -- 172.21.15.57:0/526942022 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c0a89b0 con 0x7f66880c62b0 2023-10-30T14:44:00.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f6686ffd640 1 -- 172.21.15.57:0/526942022 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667c0a8b10 con 0x7f66880c62b0 2023-10-30T14:44:00.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f6686ffd640 1 -- 172.21.15.57:0/526942022 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c0a8ea0 con 0x7f66880c62b0 2023-10-30T14:44:00.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668d1f2640 1 -- 172.21.15.57:0/526942022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66880c62b0 msgr2=0x7f66880ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:00.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668d1f2640 1 --2- 172.21.15.57:0/526942022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66880c62b0 0x7f66880ca1d0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f6688066890 tx=0x7f667c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668d1f2640 1 -- 172.21.15.57:0/526942022 shutdown_connections 2023-10-30T14:44:00.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668d1f2640 1 --2- 172.21.15.57:0/526942022 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66880cb180 0x7f668806ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668d1f2640 1 --2- 172.21.15.57:0/526942022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66880c62b0 0x7f66880ca1d0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668d1f2640 1 --2- 172.21.15.57:0/526942022 >> v2:172.21.15.88:3300/0 conn(0x7f6688153b50 0x7f66881597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668d1f2640 1 -- 172.21.15.57:0/526942022 >> 172.21.15.57:0/526942022 conn(0x7f6688065a80 msgr2=0x7f6688065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:00.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668d1f2640 1 -- 172.21.15.57:0/526942022 shutdown_connections 2023-10-30T14:44:00.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668d1f2640 1 -- 172.21.15.57:0/526942022 wait complete. 2023-10-30T14:44:00.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668d1f2640 1 Processor -- start 2023-10-30T14:44:00.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.767+0000 7f668d1f2640 1 -- start start 2023-10-30T14:44:00.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f668d1f2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66880c62b0 0x7f6688159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:00.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f668d1f2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66880cb180 0x7f66880c4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:00.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f668d1f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6688153b50 0x7f66880c5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:00.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f668d1f2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66880c5790 con 0x7f6688153b50 2023-10-30T14:44:00.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f668d1f2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6688157620 con 0x7f66880c62b0 2023-10-30T14:44:00.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f668d1f2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f668814e9a0 con 0x7f66880cb180 2023-10-30T14:44:00.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f668c9f1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6688153b50 0x7f66880c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:00.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f6687fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66880c62b0 0x7f6688159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:00.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f668c9f1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6688153b50 0x7f66880c5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50884/0 (socket says 172.21.15.57:50884) 2023-10-30T14:44:00.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f668c9f1640 1 -- 172.21.15.57:0/1718419948 learned_addr learned my addr 172.21.15.57:0/1718419948 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:00.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f66877fe640 1 --2- 172.21.15.57:0/1718419948 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66880cb180 0x7f66880c4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:00.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f668c9f1640 1 -- 172.21.15.57:0/1718419948 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66880cb180 msgr2=0x7f66880c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:00.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f668c9f1640 1 --2- 172.21.15.57:0/1718419948 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66880cb180 0x7f66880c4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f668c9f1640 1 -- 172.21.15.57:0/1718419948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66880c62b0 msgr2=0x7f6688159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:00.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f668c9f1640 1 --2- 172.21.15.57:0/1718419948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66880c62b0 0x7f6688159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f668c9f1640 1 -- 172.21.15.57:0/1718419948 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667c0094d0 con 0x7f6688153b50 2023-10-30T14:44:00.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f668c9f1640 1 --2- 172.21.15.57:0/1718419948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6688153b50 0x7f66880c5250 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f6680007e60 tx=0x7f668000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:00.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f6684ff9640 1 -- 172.21.15.57:0/1718419948 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6680014620 con 0x7f6688153b50 2023-10-30T14:44:00.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f6684ff9640 1 -- 172.21.15.57:0/1718419948 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668000ee90 con 0x7f6688153b50 2023-10-30T14:44:00.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f6684ff9640 1 -- 172.21.15.57:0/1718419948 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66800149e0 con 0x7f6688153b50 2023-10-30T14:44:00.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f668d1f2640 1 -- 172.21.15.57:0/1718419948 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66881581a0 con 0x7f6688153b50 2023-10-30T14:44:00.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f668d1f2640 1 -- 172.21.15.57:0/1718419948 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6688158760 con 0x7f6688153b50 2023-10-30T14:44:00.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.771+0000 7f668d1f2640 1 -- 172.21.15.57:0/1718419948 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6688067540 con 0x7f6688153b50 2023-10-30T14:44:00.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.775+0000 7f6684ff9640 1 -- 172.21.15.57:0/1718419948 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6680014b40 con 0x7f6688153b50 2023-10-30T14:44:00.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.775+0000 7f6684ff9640 1 --2- 172.21.15.57:0/1718419948 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66740664a0 0x7f6674068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:00.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.775+0000 7f6684ff9640 1 -- 172.21.15.57:0/1718419948 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f6680084cd0 con 0x7f6688153b50 2023-10-30T14:44:00.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.775+0000 7f6687fff640 1 --2- 172.21.15.57:0/1718419948 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66740664a0 0x7f6674068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:00.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.775+0000 7f6687fff640 1 --2- 172.21.15.57:0/1718419948 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66740664a0 0x7f6674068960 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f66780079c0 tx=0x7f6678008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:00.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.779+0000 7f6684ff9640 1 -- 172.21.15.57:0/1718419948 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6680051b50 con 0x7f6688153b50 2023-10-30T14:44:00.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.971+0000 7f668d1f2640 1 -- 172.21.15.57:0/1718419948 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6688066aa0 con 0x7f6688153b50 2023-10-30T14:44:00.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.971+0000 7f6684ff9640 1 -- 172.21.15.57:0/1718419948 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+6509 (secure 0 0 0) 0x7f6680055170 con 0x7f6688153b50 2023-10-30T14:44:00.975 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:00.976 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7536,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:57.246176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6356,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:00.977 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 163 2023-10-30T14:44:00.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.971+0000 7f668d1f2640 1 -- 172.21.15.57:0/1718419948 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66740664a0 msgr2=0x7f6674068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:00.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.971+0000 7f668d1f2640 1 --2- 172.21.15.57:0/1718419948 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66740664a0 0x7f6674068960 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f66780079c0 tx=0x7f6678008040 comp rx=0 tx=0).stop 2023-10-30T14:44:00.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.971+0000 7f668d1f2640 1 -- 172.21.15.57:0/1718419948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6688153b50 msgr2=0x7f66880c5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:00.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.971+0000 7f668d1f2640 1 --2- 172.21.15.57:0/1718419948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6688153b50 0x7f66880c5250 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f6680007e60 tx=0x7f668000e4b0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.971+0000 7f668d1f2640 1 -- 172.21.15.57:0/1718419948 shutdown_connections 2023-10-30T14:44:00.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.971+0000 7f668d1f2640 1 --2- 172.21.15.57:0/1718419948 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f66740664a0 0x7f6674068960 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.971+0000 7f668d1f2640 1 --2- 172.21.15.57:0/1718419948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6688153b50 0x7f66880c5250 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.971+0000 7f668d1f2640 1 --2- 172.21.15.57:0/1718419948 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66880cb180 0x7f66880c4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.971+0000 7f668d1f2640 1 --2- 172.21.15.57:0/1718419948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66880c62b0 0x7f6688159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:00.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.971+0000 7f668d1f2640 1 -- 172.21.15.57:0/1718419948 >> 172.21.15.57:0/1718419948 conn(0x7f6688065a80 msgr2=0x7f66880b4700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:00.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.971+0000 7f668d1f2640 1 -- 172.21.15.57:0/1718419948 shutdown_connections 2023-10-30T14:44:00.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:00.971+0000 7f668d1f2640 1 -- 172.21.15.57:0/1718419948 wait complete. 2023-10-30T14:44:00.997 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:44:03.000 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:03.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.151+0000 7f1a61a9e640 1 Processor -- start 2023-10-30T14:44:03.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.151+0000 7f1a61a9e640 1 -- start start 2023-10-30T14:44:03.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.155+0000 7f1a61a9e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a5c0c5ba0 0x7f1a5c0c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:03.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.155+0000 7f1a61a9e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c15cda0 con 0x7f1a5c0c9b40 2023-10-30T14:44:03.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.155+0000 7f1a61a9e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c06f490 con 0x7f1a5c0c5f80 2023-10-30T14:44:03.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.155+0000 7f1a61a9e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c15d990 con 0x7f1a5c0c5ba0 2023-10-30T14:44:03.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.155+0000 7f1a5bfff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1a5c0c9b40 0x7f1a5c1597b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:40204/0 (socket says 172.21.15.57:40204) 2023-10-30T14:44:03.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.155+0000 7f1a5bfff640 1 -- 172.21.15.57:0/3814469784 learned_addr learned my addr 172.21.15.57:0/3814469784 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:44:03.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.155+0000 7f1a5b7fe640 1 --2- 172.21.15.57:0/3814469784 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a5c0c5ba0 0x7f1a5c0c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:03.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.155+0000 7f1a5a7fc640 1 -- 172.21.15.57:0/3814469784 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470847547 0 0) 0x7f1a5c15cda0 con 0x7f1a5c0c9b40 2023-10-30T14:44:03.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.155+0000 7f1a5a7fc640 1 -- 172.21.15.57:0/3814469784 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a44003530 con 0x7f1a5c0c9b40 2023-10-30T14:44:03.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.155+0000 7f1a5a7fc640 1 -- 172.21.15.57:0/3814469784 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495784951 0 0) 0x7f1a5c06f490 con 0x7f1a5c0c5f80 2023-10-30T14:44:03.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.155+0000 7f1a5a7fc640 1 -- 172.21.15.57:0/3814469784 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a5c15cda0 con 0x7f1a5c0c5f80 2023-10-30T14:44:03.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.155+0000 7f1a5a7fc640 1 -- 172.21.15.57:0/3814469784 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1952701126 0 0) 0x7f1a44003530 con 0x7f1a5c0c9b40 2023-10-30T14:44:03.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.155+0000 7f1a5b7fe640 1 -- 172.21.15.57:0/3814469784 >> v1:172.21.15.88:6789/0 conn(0x7f1a5c0c5f80 legacy=0x7f1a5c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:03.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.155+0000 7f1a5b7fe640 1 -- 172.21.15.57:0/3814469784 >> v1:172.21.15.57:6789/0 conn(0x7f1a5c0c9b40 legacy=0x7f1a5c1597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:03.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.155+0000 7f1a5b7fe640 1 -- 172.21.15.57:0/3814469784 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c15cff0 con 0x7f1a5c0c5ba0 2023-10-30T14:44:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.159+0000 7f1a5b7fe640 1 --2- 172.21.15.57:0/3814469784 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a5c0c5ba0 0x7f1a5c0c70f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c0097f0 tx=0x7f1a4c01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb52371f97fcd11f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.159+0000 7f1a5a7fc640 1 -- 172.21.15.57:0/3814469784 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54001a00 con 0x7f1a5c0c5ba0 2023-10-30T14:44:03.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.159+0000 7f1a5a7fc640 1 -- 172.21.15.57:0/3814469784 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4c0a7050 con 0x7f1a5c0c5ba0 2023-10-30T14:44:03.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.159+0000 7f1a5a7fc640 1 -- 172.21.15.57:0/3814469784 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c0ab600 con 0x7f1a5c0c5ba0 2023-10-30T14:44:03.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.159+0000 7f1a61a9e640 1 -- 172.21.15.57:0/3814469784 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a5c0c5ba0 msgr2=0x7f1a5c0c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:03.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.159+0000 7f1a61a9e640 1 --2- 172.21.15.57:0/3814469784 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a5c0c5ba0 0x7f1a5c0c70f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c0097f0 tx=0x7f1a4c01c3e0 comp rx=0 tx=0).stop 2023-10-30T14:44:03.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.159+0000 7f1a61a9e640 1 -- 172.21.15.57:0/3814469784 shutdown_connections 2023-10-30T14:44:03.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.159+0000 7f1a61a9e640 1 --2- 172.21.15.57:0/3814469784 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a5c0c5ba0 0x7f1a5c0c70f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:03.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.159+0000 7f1a61a9e640 1 -- 172.21.15.57:0/3814469784 >> 172.21.15.57:0/3814469784 conn(0x7f1a5c06afa0 msgr2=0x7f1a5c06b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:03.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.159+0000 7f1a61a9e640 1 -- 172.21.15.57:0/3814469784 shutdown_connections 2023-10-30T14:44:03.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.159+0000 7f1a61a9e640 1 -- 172.21.15.57:0/3814469784 wait complete. 2023-10-30T14:44:03.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.159+0000 7f1a61a9e640 1 Processor -- start 2023-10-30T14:44:03.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.159+0000 7f1a61a9e640 1 -- start start 2023-10-30T14:44:03.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a61a9e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a5c0c5ba0 0x7f1a5c14c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:03.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a61a9e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c0c5f80 0x7f1a5c0c4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:03.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a61a9e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a5c0c9b40 0x7f1a5c0c5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:03.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a61a9e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c146ab0 con 0x7f1a5c0c5f80 2023-10-30T14:44:03.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a61a9e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0c37c0 con 0x7f1a5c0c5ba0 2023-10-30T14:44:03.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a61a9e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0c4870 con 0x7f1a5c0c9b40 2023-10-30T14:44:03.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a5affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c0c5f80 0x7f1a5c0c4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:03.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a5affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c0c5f80 0x7f1a5c0c4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50900/0 (socket says 172.21.15.57:50900) 2023-10-30T14:44:03.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a5affd640 1 -- 172.21.15.57:0/2309135347 learned_addr learned my addr 172.21.15.57:0/2309135347 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:03.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a5b7fe640 1 --2- 172.21.15.57:0/2309135347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a5c0c5ba0 0x7f1a5c14c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:03.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a5bfff640 1 --2- 172.21.15.57:0/2309135347 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a5c0c9b40 0x7f1a5c0c5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:03.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a5affd640 1 -- 172.21.15.57:0/2309135347 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a5c0c9b40 msgr2=0x7f1a5c0c5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:03.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a5affd640 1 --2- 172.21.15.57:0/2309135347 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a5c0c9b40 0x7f1a5c0c5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:03.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a5affd640 1 -- 172.21.15.57:0/2309135347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a5c0c5ba0 msgr2=0x7f1a5c14c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:03.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a5affd640 1 --2- 172.21.15.57:0/2309135347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a5c0c5ba0 0x7f1a5c14c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:03.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a5affd640 1 -- 172.21.15.57:0/2309135347 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c009420 con 0x7f1a5c0c5f80 2023-10-30T14:44:03.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a5affd640 1 --2- 172.21.15.57:0/2309135347 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c0c5f80 0x7f1a5c0c4d00 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000c460 tx=0x7f1a5000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:03.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a43fff640 1 -- 172.21.15.57:0/2309135347 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a500052d0 con 0x7f1a5c0c5f80 2023-10-30T14:44:03.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a43fff640 1 -- 172.21.15.57:0/2309135347 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a50005450 con 0x7f1a5c0c5f80 2023-10-30T14:44:03.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a43fff640 1 -- 172.21.15.57:0/2309135347 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50003840 con 0x7f1a5c0c5f80 2023-10-30T14:44:03.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a61a9e640 1 -- 172.21.15.57:0/2309135347 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5c151010 con 0x7f1a5c0c5f80 2023-10-30T14:44:03.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a61a9e640 1 -- 172.21.15.57:0/2309135347 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5c15a970 con 0x7f1a5c0c5f80 2023-10-30T14:44:03.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a61a9e640 1 -- 172.21.15.57:0/2309135347 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5c04e170 con 0x7f1a5c0c5f80 2023-10-30T14:44:03.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.163+0000 7f1a43fff640 1 -- 172.21.15.57:0/2309135347 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1a500039a0 con 0x7f1a5c0c5f80 2023-10-30T14:44:03.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.167+0000 7f1a43fff640 1 --2- 172.21.15.57:0/2309135347 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1a480664a0 0x7f1a48068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:03.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.167+0000 7f1a43fff640 1 -- 172.21.15.57:0/2309135347 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f1a50083980 con 0x7f1a5c0c5f80 2023-10-30T14:44:03.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.167+0000 7f1a5b7fe640 1 --2- 172.21.15.57:0/2309135347 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1a480664a0 0x7f1a48068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:03.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.167+0000 7f1a5b7fe640 1 --2- 172.21.15.57:0/2309135347 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1a480664a0 0x7f1a48068960 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c0ad000 tx=0x7f1a4c0a6000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:03.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.171+0000 7f1a43fff640 1 -- 172.21.15.57:0/2309135347 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1a50050930 con 0x7f1a5c0c5f80 2023-10-30T14:44:03.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.387+0000 7f1a61a9e640 1 -- 172.21.15.57:0/2309135347 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a5c0c6cd0 con 0x7f1a5c0c5f80 2023-10-30T14:44:03.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.387+0000 7f1a43fff640 1 -- 172.21.15.57:0/2309135347 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+6509 (secure 0 0 0) 0x7f1a50053f50 con 0x7f1a5c0c5f80 2023-10-30T14:44:03.391 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:03.391 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7536,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:57.246176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6356,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:03.392 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 163 2023-10-30T14:44:03.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.391+0000 7f1a61a9e640 1 -- 172.21.15.57:0/2309135347 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1a480664a0 msgr2=0x7f1a48068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:03.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.391+0000 7f1a61a9e640 1 --2- 172.21.15.57:0/2309135347 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1a480664a0 0x7f1a48068960 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c0ad000 tx=0x7f1a4c0a6000 comp rx=0 tx=0).stop 2023-10-30T14:44:03.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.391+0000 7f1a61a9e640 1 -- 172.21.15.57:0/2309135347 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c0c5f80 msgr2=0x7f1a5c0c4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:03.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.391+0000 7f1a61a9e640 1 --2- 172.21.15.57:0/2309135347 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c0c5f80 0x7f1a5c0c4d00 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000c460 tx=0x7f1a5000c920 comp rx=0 tx=0).stop 2023-10-30T14:44:03.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.391+0000 7f1a61a9e640 1 -- 172.21.15.57:0/2309135347 shutdown_connections 2023-10-30T14:44:03.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.391+0000 7f1a61a9e640 1 --2- 172.21.15.57:0/2309135347 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1a480664a0 0x7f1a48068960 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:03.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.391+0000 7f1a61a9e640 1 --2- 172.21.15.57:0/2309135347 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a5c0c9b40 0x7f1a5c0c5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:03.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.391+0000 7f1a61a9e640 1 --2- 172.21.15.57:0/2309135347 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1a5c0c5f80 0x7f1a5c0c4d00 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:03.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.391+0000 7f1a61a9e640 1 --2- 172.21.15.57:0/2309135347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a5c0c5ba0 0x7f1a5c14c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:03.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.391+0000 7f1a61a9e640 1 -- 172.21.15.57:0/2309135347 >> 172.21.15.57:0/2309135347 conn(0x7f1a5c06afa0 msgr2=0x7f1a5c071560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:03.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.391+0000 7f1a61a9e640 1 -- 172.21.15.57:0/2309135347 shutdown_connections 2023-10-30T14:44:03.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:03.391+0000 7f1a61a9e640 1 -- 172.21.15.57:0/2309135347 wait complete. 2023-10-30T14:44:03.411 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:44:05.412 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:05.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.567+0000 7ff3e5209640 1 Processor -- start 2023-10-30T14:44:05.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.567+0000 7ff3e5209640 1 -- start start 2023-10-30T14:44:05.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.567+0000 7ff3e5209640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff3e0153c80 0x7ff3e015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:05.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.567+0000 7ff3e5209640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff3e00c4a10 0x7ff3e00c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:05.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.567+0000 7ff3e5209640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e0159f20 con 0x7ff3e015a3b0 2023-10-30T14:44:05.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.567+0000 7ff3e5209640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e015dec0 con 0x7ff3e00c4a10 2023-10-30T14:44:05.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.567+0000 7ff3e5209640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e01599a0 con 0x7ff3e0153c80 2023-10-30T14:44:05.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.567+0000 7ff3d7fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff3e00c4a10 0x7ff3e00c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:05.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.567+0000 7ff3df59e640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff3e015a3b0 0x7ff3e00ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:33836/0 (socket says 172.21.15.57:33836) 2023-10-30T14:44:05.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.567+0000 7ff3df59e640 1 -- 172.21.15.57:0/852645249 learned_addr learned my addr 172.21.15.57:0/852645249 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:44:05.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.567+0000 7ff3ded9d640 1 --2- 172.21.15.57:0/852645249 >> v2:172.21.15.88:3300/0 conn(0x7ff3e0153c80 0x7ff3e015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:05.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.567+0000 7ff3d7fff640 1 -- 172.21.15.57:0/852645249 >> v2:172.21.15.88:3300/0 conn(0x7ff3e0153c80 msgr2=0x7ff3e015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:05.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.567+0000 7ff3d7fff640 1 --2- 172.21.15.57:0/852645249 >> v2:172.21.15.88:3300/0 conn(0x7ff3e0153c80 0x7ff3e015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:05.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.567+0000 7ff3d7fff640 1 -- 172.21.15.57:0/852645249 >> v1:172.21.15.57:6789/0 conn(0x7ff3e015a3b0 legacy=0x7ff3e00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:05.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.567+0000 7ff3d7fff640 1 -- 172.21.15.57:0/852645249 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e00c47e0 con 0x7ff3e00c4a10 2023-10-30T14:44:05.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.567+0000 7ff3de59c640 1 -- 172.21.15.57:0/852645249 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628235664 0 0) 0x7ff3e0159f20 con 0x7ff3e015a3b0 2023-10-30T14:44:05.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.571+0000 7ff3d7fff640 1 --2- 172.21.15.57:0/852645249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3e00c4a10 0x7ff3e00c42a0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00d00f0 tx=0x7ff3c801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f54c1920d97162 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:05.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.571+0000 7ff3de59c640 1 -- 172.21.15.57:0/852645249 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c80a8950 con 0x7ff3e00c4a10 2023-10-30T14:44:05.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.571+0000 7ff3de59c640 1 -- 172.21.15.57:0/852645249 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c80a8ab0 con 0x7ff3e00c4a10 2023-10-30T14:44:05.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.571+0000 7ff3de59c640 1 -- 172.21.15.57:0/852645249 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c80a8e40 con 0x7ff3e00c4a10 2023-10-30T14:44:05.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.571+0000 7ff3e5209640 1 -- 172.21.15.57:0/852645249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3e00c4a10 msgr2=0x7ff3e00c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:05.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.571+0000 7ff3e5209640 1 --2- 172.21.15.57:0/852645249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3e00c4a10 0x7ff3e00c42a0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00d00f0 tx=0x7ff3c801ec50 comp rx=0 tx=0).stop 2023-10-30T14:44:05.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.571+0000 7ff3e5209640 1 -- 172.21.15.57:0/852645249 shutdown_connections 2023-10-30T14:44:05.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.571+0000 7ff3e5209640 1 --2- 172.21.15.57:0/852645249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3e00c4a10 0x7ff3e00c42a0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:05.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.571+0000 7ff3e5209640 1 --2- 172.21.15.57:0/852645249 >> v2:172.21.15.88:3300/0 conn(0x7ff3e0153c80 0x7ff3e015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:05.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.571+0000 7ff3e5209640 1 -- 172.21.15.57:0/852645249 >> 172.21.15.57:0/852645249 conn(0x7ff3e00715c0 msgr2=0x7ff3e00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:05.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.571+0000 7ff3e5209640 1 -- 172.21.15.57:0/852645249 shutdown_connections 2023-10-30T14:44:05.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.571+0000 7ff3e5209640 1 -- 172.21.15.57:0/852645249 wait complete. 2023-10-30T14:44:05.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3e5209640 1 Processor -- start 2023-10-30T14:44:05.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3e5209640 1 -- start start 2023-10-30T14:44:05.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3e5209640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3e00c4a10 0x7ff3e014e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:05.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3e5209640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3e0153c80 0x7ff3e0152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:05.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3e5209640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3e015a3b0 0x7ff3e01eea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:05.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3e5209640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e0150ab0 con 0x7ff3e0153c80 2023-10-30T14:44:05.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3e5209640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e0150d70 con 0x7ff3e015a3b0 2023-10-30T14:44:05.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3e5209640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e01f1fe0 con 0x7ff3e00c4a10 2023-10-30T14:44:05.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3d7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3e0153c80 0x7ff3e0152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:05.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3d7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3e0153c80 0x7ff3e0152e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54558/0 (socket says 172.21.15.57:54558) 2023-10-30T14:44:05.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3d7fff640 1 -- 172.21.15.57:0/2718371020 learned_addr learned my addr 172.21.15.57:0/2718371020 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:05.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3ded9d640 1 --2- 172.21.15.57:0/2718371020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3e00c4a10 0x7ff3e014e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:05.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3df59e640 1 --2- 172.21.15.57:0/2718371020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3e015a3b0 0x7ff3e01eea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:05.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3d7fff640 1 -- 172.21.15.57:0/2718371020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3e00c4a10 msgr2=0x7ff3e014e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:05.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3d7fff640 1 --2- 172.21.15.57:0/2718371020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3e00c4a10 0x7ff3e014e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:05.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3d7fff640 1 -- 172.21.15.57:0/2718371020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3e015a3b0 msgr2=0x7ff3e01eea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:05.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3d7fff640 1 --2- 172.21.15.57:0/2718371020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3e015a3b0 0x7ff3e01eea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:05.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3d7fff640 1 -- 172.21.15.57:0/2718371020 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c8009420 con 0x7ff3e0153c80 2023-10-30T14:44:05.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3d7fff640 1 --2- 172.21.15.57:0/2718371020 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3e0153c80 0x7ff3e0152e90 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80b3040 tx=0x7ff3c801b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:05.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3d77fe640 1 -- 172.21.15.57:0/2718371020 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c80b0df0 con 0x7ff3e0153c80 2023-10-30T14:44:05.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3d77fe640 1 -- 172.21.15.57:0/2718371020 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c80b9410 con 0x7ff3e0153c80 2023-10-30T14:44:05.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3e5209640 1 -- 172.21.15.57:0/2718371020 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e01f22c0 con 0x7ff3e0153c80 2023-10-30T14:44:05.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3d77fe640 1 -- 172.21.15.57:0/2718371020 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c80b97c0 con 0x7ff3e0153c80 2023-10-30T14:44:05.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.575+0000 7ff3e5209640 1 -- 172.21.15.57:0/2718371020 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e01f2880 con 0x7ff3e0153c80 2023-10-30T14:44:05.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.579+0000 7ff3e5209640 1 -- 172.21.15.57:0/2718371020 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e004e170 con 0x7ff3e0153c80 2023-10-30T14:44:05.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.579+0000 7ff3d77fe640 1 -- 172.21.15.57:0/2718371020 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff3c80b7040 con 0x7ff3e0153c80 2023-10-30T14:44:05.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.579+0000 7ff3d77fe640 1 --2- 172.21.15.57:0/2718371020 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff3c00664a0 0x7ff3c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:05.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.579+0000 7ff3d77fe640 1 -- 172.21.15.57:0/2718371020 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7ff3c81220b0 con 0x7ff3e0153c80 2023-10-30T14:44:05.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.579+0000 7ff3ded9d640 1 --2- 172.21.15.57:0/2718371020 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff3c00664a0 0x7ff3c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:05.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.583+0000 7ff3ded9d640 1 --2- 172.21.15.57:0/2718371020 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff3c00664a0 0x7ff3c0068960 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0009af0 tx=0x7ff3d0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:05.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.583+0000 7ff3d77fe640 1 -- 172.21.15.57:0/2718371020 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff3c80eef10 con 0x7ff3e0153c80 2023-10-30T14:44:05.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.775+0000 7ff3e5209640 1 -- 172.21.15.57:0/2718371020 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3e01f2a70 con 0x7ff3e0153c80 2023-10-30T14:44:05.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.775+0000 7ff3d77fe640 1 -- 172.21.15.57:0/2718371020 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+6509 (secure 0 0 0) 0x7ff3c80b5020 con 0x7ff3e0153c80 2023-10-30T14:44:05.775 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:05.775 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7536,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:57.246176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6356,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:05.776 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 163 2023-10-30T14:44:05.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.775+0000 7ff3e5209640 1 -- 172.21.15.57:0/2718371020 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff3c00664a0 msgr2=0x7ff3c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:05.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.775+0000 7ff3e5209640 1 --2- 172.21.15.57:0/2718371020 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff3c00664a0 0x7ff3c0068960 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0009af0 tx=0x7ff3d0008040 comp rx=0 tx=0).stop 2023-10-30T14:44:05.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.775+0000 7ff3e5209640 1 -- 172.21.15.57:0/2718371020 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3e0153c80 msgr2=0x7ff3e0152e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:05.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.775+0000 7ff3e5209640 1 --2- 172.21.15.57:0/2718371020 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3e0153c80 0x7ff3e0152e90 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80b3040 tx=0x7ff3c801b9f0 comp rx=0 tx=0).stop 2023-10-30T14:44:05.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.775+0000 7ff3e5209640 1 -- 172.21.15.57:0/2718371020 shutdown_connections 2023-10-30T14:44:05.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.775+0000 7ff3e5209640 1 --2- 172.21.15.57:0/2718371020 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff3c00664a0 0x7ff3c0068960 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:05.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.775+0000 7ff3e5209640 1 --2- 172.21.15.57:0/2718371020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3e015a3b0 0x7ff3e01eea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:05.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.775+0000 7ff3e5209640 1 --2- 172.21.15.57:0/2718371020 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3e0153c80 0x7ff3e0152e90 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:05.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.775+0000 7ff3e5209640 1 --2- 172.21.15.57:0/2718371020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3e00c4a10 0x7ff3e014e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:05.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.775+0000 7ff3e5209640 1 -- 172.21.15.57:0/2718371020 >> 172.21.15.57:0/2718371020 conn(0x7ff3e00715c0 msgr2=0x7ff3e00cb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:05.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.775+0000 7ff3e5209640 1 -- 172.21.15.57:0/2718371020 shutdown_connections 2023-10-30T14:44:05.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:05.775+0000 7ff3e5209640 1 -- 172.21.15.57:0/2718371020 wait complete. 2023-10-30T14:44:05.796 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:44:07.797 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:07.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.951+0000 7faa36454640 1 Processor -- start 2023-10-30T14:44:07.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.951+0000 7faa36454640 1 -- start start 2023-10-30T14:44:07.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.951+0000 7faa36454640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7faa30070520 0x7faa30070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:07.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.951+0000 7faa36454640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa300c2310 con 0x7faa300cb7d0 2023-10-30T14:44:07.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.951+0000 7faa36454640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa30156710 con 0x7faa30150830 2023-10-30T14:44:07.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.951+0000 7faa36454640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa3006bc10 con 0x7faa30070520 2023-10-30T14:44:07.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.951+0000 7faa2f7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7faa300cb7d0 0x7faa300c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:33840/0 (socket says 172.21.15.57:33840) 2023-10-30T14:44:07.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.951+0000 7faa2f7fe640 1 -- 172.21.15.57:0/1685331564 learned_addr learned my addr 172.21.15.57:0/1685331564 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:44:07.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.955+0000 7faa349f1640 1 --2- 172.21.15.57:0/1685331564 >> v2:172.21.15.57:3300/0 conn(0x7faa30070520 0x7faa30070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:07.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.955+0000 7faa2effd640 1 -- 172.21.15.57:0/1685331564 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422964014 0 0) 0x7faa300c2310 con 0x7faa300cb7d0 2023-10-30T14:44:07.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.955+0000 7faa2effd640 1 -- 172.21.15.57:0/1685331564 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa18003530 con 0x7faa300cb7d0 2023-10-30T14:44:07.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.955+0000 7faa349f1640 1 -- 172.21.15.57:0/1685331564 >> v1:172.21.15.88:6789/0 conn(0x7faa30150830 legacy=0x7faa3006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:07.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.955+0000 7faa349f1640 1 -- 172.21.15.57:0/1685331564 >> v1:172.21.15.57:6789/0 conn(0x7faa300cb7d0 legacy=0x7faa300c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:07.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.955+0000 7faa349f1640 1 -- 172.21.15.57:0/1685331564 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa300c4360 con 0x7faa30070520 2023-10-30T14:44:07.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.955+0000 7faa349f1640 1 --2- 172.21.15.57:0/1685331564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa30070520 0x7faa30070990 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7faa28009d90 tx=0x7faa2801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3829a0ea89614031 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:07.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.955+0000 7faa2effd640 1 -- 172.21.15.57:0/1685331564 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa280af600 con 0x7faa30070520 2023-10-30T14:44:07.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.955+0000 7faa2effd640 1 -- 172.21.15.57:0/1685331564 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa280ab050 con 0x7faa30070520 2023-10-30T14:44:07.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.955+0000 7faa2effd640 1 -- 172.21.15.57:0/1685331564 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa280af980 con 0x7faa30070520 2023-10-30T14:44:07.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa36454640 1 -- 172.21.15.57:0/1685331564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa30070520 msgr2=0x7faa30070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:07.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa36454640 1 --2- 172.21.15.57:0/1685331564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa30070520 0x7faa30070990 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7faa28009d90 tx=0x7faa2801f130 comp rx=0 tx=0).stop 2023-10-30T14:44:07.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa36454640 1 -- 172.21.15.57:0/1685331564 shutdown_connections 2023-10-30T14:44:07.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa36454640 1 --2- 172.21.15.57:0/1685331564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa30070520 0x7faa30070990 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:07.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa36454640 1 -- 172.21.15.57:0/1685331564 >> 172.21.15.57:0/1685331564 conn(0x7faa300cac70 msgr2=0x7faa300cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:07.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa36454640 1 -- 172.21.15.57:0/1685331564 shutdown_connections 2023-10-30T14:44:07.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa36454640 1 -- 172.21.15.57:0/1685331564 wait complete. 2023-10-30T14:44:07.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa36454640 1 Processor -- start 2023-10-30T14:44:07.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa36454640 1 -- start start 2023-10-30T14:44:07.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa36454640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa30070520 0x7faa301f6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:07.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa36454640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa300cb7d0 0x7faa3014eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:07.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa36454640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa30150830 0x7faa300c1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:07.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa36454640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa300c0f40 con 0x7faa30150830 2023-10-30T14:44:07.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa36454640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa300c4790 con 0x7faa30070520 2023-10-30T14:44:07.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa36454640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa301f71b0 con 0x7faa300cb7d0 2023-10-30T14:44:07.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa349f1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa30150830 0x7faa300c1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:07.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa349f1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa30150830 0x7faa300c1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54582/0 (socket says 172.21.15.57:54582) 2023-10-30T14:44:07.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa349f1640 1 -- 172.21.15.57:0/2480050055 learned_addr learned my addr 172.21.15.57:0/2480050055 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:07.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa2ffff640 1 --2- 172.21.15.57:0/2480050055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa30070520 0x7faa301f6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:07.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.959+0000 7faa2f7fe640 1 --2- 172.21.15.57:0/2480050055 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa300cb7d0 0x7faa3014eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:07.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.963+0000 7faa349f1640 1 -- 172.21.15.57:0/2480050055 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa300cb7d0 msgr2=0x7faa3014eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:07.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.963+0000 7faa349f1640 1 --2- 172.21.15.57:0/2480050055 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa300cb7d0 0x7faa3014eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:07.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.963+0000 7faa349f1640 1 -- 172.21.15.57:0/2480050055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa30070520 msgr2=0x7faa301f6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:07.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.963+0000 7faa349f1640 1 --2- 172.21.15.57:0/2480050055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa30070520 0x7faa301f6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:07.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.963+0000 7faa349f1640 1 -- 172.21.15.57:0/2480050055 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa28009a70 con 0x7faa30150830 2023-10-30T14:44:07.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.963+0000 7faa2ffff640 1 --2- 172.21.15.57:0/2480050055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa30070520 0x7faa301f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:44:07.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.963+0000 7faa349f1640 1 --2- 172.21.15.57:0/2480050055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa30150830 0x7faa300c1390 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7faa280b2040 tx=0x7faa2801bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:07.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.963+0000 7faa2cff9640 1 -- 172.21.15.57:0/2480050055 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa28009450 con 0x7faa30150830 2023-10-30T14:44:07.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.963+0000 7faa2cff9640 1 -- 172.21.15.57:0/2480050055 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa280b9410 con 0x7faa30150830 2023-10-30T14:44:07.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.963+0000 7faa36454640 1 -- 172.21.15.57:0/2480050055 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa301f30a0 con 0x7faa30150830 2023-10-30T14:44:07.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.963+0000 7faa2cff9640 1 -- 172.21.15.57:0/2480050055 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa280b97c0 con 0x7faa30150830 2023-10-30T14:44:07.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.963+0000 7faa36454640 1 -- 172.21.15.57:0/2480050055 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa301f3660 con 0x7faa30150830 2023-10-30T14:44:07.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.963+0000 7faa36454640 1 -- 172.21.15.57:0/2480050055 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa300c10b0 con 0x7faa30150830 2023-10-30T14:44:07.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.963+0000 7faa2cff9640 1 -- 172.21.15.57:0/2480050055 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faa280b6050 con 0x7faa30150830 2023-10-30T14:44:07.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.967+0000 7faa2cff9640 1 --2- 172.21.15.57:0/2480050055 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa1c066470 0x7faa1c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:07.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.967+0000 7faa2cff9640 1 -- 172.21.15.57:0/2480050055 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7faa281228f0 con 0x7faa30150830 2023-10-30T14:44:07.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.967+0000 7faa2ffff640 1 --2- 172.21.15.57:0/2480050055 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa1c066470 0x7faa1c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:07.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.967+0000 7faa2ffff640 1 --2- 172.21.15.57:0/2480050055 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa1c066470 0x7faa1c068930 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7faa20000f70 tx=0x7faa20009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:07.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:07.971+0000 7faa2cff9640 1 -- 172.21.15.57:0/2480050055 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faa300c10b0 con 0x7faa30150830 2023-10-30T14:44:08.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:08.159+0000 7faa36454640 1 -- 172.21.15.57:0/2480050055 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa30066510 con 0x7faa30150830 2023-10-30T14:44:08.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:08.159+0000 7faa2cff9640 1 -- 172.21.15.57:0/2480050055 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+6509 (secure 0 0 0) 0x7faa280b4090 con 0x7faa30150830 2023-10-30T14:44:08.162 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:08.162 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7536,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:57.246176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6356,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:08.163 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 163 2023-10-30T14:44:08.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:08.163+0000 7faa36454640 1 -- 172.21.15.57:0/2480050055 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa1c066470 msgr2=0x7faa1c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:08.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:08.163+0000 7faa36454640 1 --2- 172.21.15.57:0/2480050055 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa1c066470 0x7faa1c068930 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7faa20000f70 tx=0x7faa20009040 comp rx=0 tx=0).stop 2023-10-30T14:44:08.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:08.163+0000 7faa36454640 1 -- 172.21.15.57:0/2480050055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa30150830 msgr2=0x7faa300c1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:08.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:08.163+0000 7faa36454640 1 --2- 172.21.15.57:0/2480050055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa30150830 0x7faa300c1390 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7faa280b2040 tx=0x7faa2801bed0 comp rx=0 tx=0).stop 2023-10-30T14:44:08.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:08.163+0000 7faa36454640 1 -- 172.21.15.57:0/2480050055 shutdown_connections 2023-10-30T14:44:08.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:08.163+0000 7faa36454640 1 --2- 172.21.15.57:0/2480050055 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa1c066470 0x7faa1c068930 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:08.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:08.163+0000 7faa36454640 1 --2- 172.21.15.57:0/2480050055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa30150830 0x7faa300c1390 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:08.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:08.163+0000 7faa36454640 1 --2- 172.21.15.57:0/2480050055 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa300cb7d0 0x7faa3014eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:08.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:08.163+0000 7faa36454640 1 --2- 172.21.15.57:0/2480050055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa30070520 0x7faa301f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:08.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:08.163+0000 7faa36454640 1 -- 172.21.15.57:0/2480050055 >> 172.21.15.57:0/2480050055 conn(0x7faa300cac70 msgr2=0x7faa3006c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:08.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:08.163+0000 7faa36454640 1 -- 172.21.15.57:0/2480050055 shutdown_connections 2023-10-30T14:44:08.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:08.163+0000 7faa36454640 1 -- 172.21.15.57:0/2480050055 wait complete. 2023-10-30T14:44:08.183 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:44:08.943 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:44:08.948 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:44:10.184 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:10.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.339+0000 7fd35b794640 1 Processor -- start 2023-10-30T14:44:10.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.339+0000 7fd35b794640 1 -- start start 2023-10-30T14:44:10.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.339+0000 7fd35b794640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd354150830 0x7fd35406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:10.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.339+0000 7fd35b794640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd3540cb7d0 0x7fd3540c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:10.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.339+0000 7fd35b794640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3540c2310 con 0x7fd3540c3e40 2023-10-30T14:44:10.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.339+0000 7fd35b794640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd354156710 con 0x7fd3540cb7d0 2023-10-30T14:44:10.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.339+0000 7fd35b794640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3540c4360 con 0x7fd354150830 2023-10-30T14:44:10.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.339+0000 7fd359d31640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fd3540c3e40 0x7fd354070520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:33856/0 (socket says 172.21.15.57:33856) 2023-10-30T14:44:10.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.339+0000 7fd359d31640 1 -- 172.21.15.57:0/1857150830 learned_addr learned my addr 172.21.15.57:0/1857150830 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:44:10.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd358d2f640 1 --2- 172.21.15.57:0/1857150830 >> v2:172.21.15.88:3300/0 conn(0x7fd3540cb7d0 0x7fd3540c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:10.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd359530640 1 --2- 172.21.15.57:0/1857150830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd354150830 0x7fd35406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:10.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd34bfff640 1 -- 172.21.15.57:0/1857150830 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874275207 0 0) 0x7fd3540c2310 con 0x7fd3540c3e40 2023-10-30T14:44:10.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd34bfff640 1 -- 172.21.15.57:0/1857150830 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd340003530 con 0x7fd3540c3e40 2023-10-30T14:44:10.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd34bfff640 1 -- 172.21.15.57:0/1857150830 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2635240012 0 0) 0x7fd340003530 con 0x7fd3540c3e40 2023-10-30T14:44:10.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd34bfff640 1 -- 172.21.15.57:0/1857150830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd354150830 msgr2=0x7fd35406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:10.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd34bfff640 1 --2- 172.21.15.57:0/1857150830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd354150830 0x7fd35406b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:10.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd34bfff640 1 -- 172.21.15.57:0/1857150830 >> v2:172.21.15.88:3300/0 conn(0x7fd3540cb7d0 msgr2=0x7fd3540c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:10.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd34bfff640 1 --2- 172.21.15.57:0/1857150830 >> v2:172.21.15.88:3300/0 conn(0x7fd3540cb7d0 0x7fd3540c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:10.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd34bfff640 1 -- 172.21.15.57:0/1857150830 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3541569b0 con 0x7fd3540c3e40 2023-10-30T14:44:10.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd34bfff640 1 -- 172.21.15.57:0/1857150830 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fd34c001a00 con 0x7fd3540c3e40 2023-10-30T14:44:10.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd34bfff640 1 -- 172.21.15.57:0/1857150830 >> v1:172.21.15.57:6789/0 conn(0x7fd3540c3e40 legacy=0x7fd354070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:10.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd34bfff640 1 --2- 172.21.15.57:0/1857150830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd354150830 0x7fd340003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:10.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd34bfff640 1 --2- 172.21.15.57:0/1857150830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd340003f00 0x7fd3400047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:10.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd34bfff640 1 --2- 172.21.15.57:0/1857150830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd340004d10 0x7fd340005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:10.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd34bfff640 1 -- 172.21.15.57:0/1857150830 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd354156710 con 0x7fd340003f00 2023-10-30T14:44:10.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd34bfff640 1 -- 172.21.15.57:0/1857150830 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3540c4360 con 0x7fd340004d10 2023-10-30T14:44:10.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd34bfff640 1 -- 172.21.15.57:0/1857150830 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34000b9c0 con 0x7fd354150830 2023-10-30T14:44:10.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd358d2f640 1 --2- 172.21.15.57:0/1857150830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd340003f00 0x7fd3400047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:10.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd359530640 1 --2- 172.21.15.57:0/1857150830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd354150830 0x7fd340003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:10.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd359d31640 1 --2- 172.21.15.57:0/1857150830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd340004d10 0x7fd340005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:10.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd358d2f640 1 -- 172.21.15.57:0/1857150830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd354150830 msgr2=0x7fd340003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:10.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd358d2f640 1 --2- 172.21.15.57:0/1857150830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd354150830 0x7fd340003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:10.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd358d2f640 1 -- 172.21.15.57:0/1857150830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd340004d10 msgr2=0x7fd340005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:10.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd358d2f640 1 --2- 172.21.15.57:0/1857150830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd340004d10 0x7fd340005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:10.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd358d2f640 1 -- 172.21.15.57:0/1857150830 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd340003530 con 0x7fd340003f00 2023-10-30T14:44:10.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.343+0000 7fd359530640 1 --2- 172.21.15.57:0/1857150830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd354150830 0x7fd340003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:44:10.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.347+0000 7fd358d2f640 1 --2- 172.21.15.57:0/1857150830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd340003f00 0x7fd3400047d0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fd344009ae0 tx=0x7fd34401ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deee71e77539d894 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:10.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.347+0000 7fd34bfff640 1 -- 172.21.15.57:0/1857150830 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3440a5680 con 0x7fd340003f00 2023-10-30T14:44:10.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.347+0000 7fd34bfff640 1 -- 172.21.15.57:0/1857150830 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3440a57e0 con 0x7fd340003f00 2023-10-30T14:44:10.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.347+0000 7fd34bfff640 1 -- 172.21.15.57:0/1857150830 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3440a5b90 con 0x7fd340003f00 2023-10-30T14:44:10.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.347+0000 7fd35b794640 1 -- 172.21.15.57:0/1857150830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd340003f00 msgr2=0x7fd3400047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:10.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.347+0000 7fd35b794640 1 --2- 172.21.15.57:0/1857150830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd340003f00 0x7fd3400047d0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fd344009ae0 tx=0x7fd34401ca90 comp rx=0 tx=0).stop 2023-10-30T14:44:10.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.347+0000 7fd359d31640 1 -- 172.21.15.57:0/1857150830 reap_dead start 2023-10-30T14:44:10.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.347+0000 7fd35b794640 1 -- 172.21.15.57:0/1857150830 shutdown_connections 2023-10-30T14:44:10.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.347+0000 7fd35b794640 1 -- 172.21.15.57:0/1857150830 >> 172.21.15.57:0/1857150830 conn(0x7fd3540cac70 msgr2=0x7fd3540cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:10.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.347+0000 7fd35b794640 1 -- 172.21.15.57:0/1857150830 shutdown_connections 2023-10-30T14:44:10.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.347+0000 7fd35b794640 1 -- 172.21.15.57:0/1857150830 wait complete. 2023-10-30T14:44:10.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.347+0000 7fd35b794640 1 Processor -- start 2023-10-30T14:44:10.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.347+0000 7fd35b794640 1 -- start start 2023-10-30T14:44:10.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.347+0000 7fd35b794640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd340003f00 0x7fd3540c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:10.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.347+0000 7fd35b794640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd354132cf0 0x7fd3540c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:10.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.347+0000 7fd35b794640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd35414fc90 0x7fd354138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:10.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.347+0000 7fd35b794640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd354134790 con 0x7fd354132cf0 2023-10-30T14:44:10.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.347+0000 7fd35b794640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd354141b60 con 0x7fd340003f00 2023-10-30T14:44:10.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.347+0000 7fd35b794640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3541344b0 con 0x7fd35414fc90 2023-10-30T14:44:10.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd358d2f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd354132cf0 0x7fd3540c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:10.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd358d2f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd354132cf0 0x7fd3540c8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54596/0 (socket says 172.21.15.57:54596) 2023-10-30T14:44:10.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd358d2f640 1 -- 172.21.15.57:0/1478432690 learned_addr learned my addr 172.21.15.57:0/1478432690 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:10.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd359530640 1 --2- 172.21.15.57:0/1478432690 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd340003f00 0x7fd3540c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:10.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd359d31640 1 --2- 172.21.15.57:0/1478432690 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd35414fc90 0x7fd354138fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:10.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd358d2f640 1 -- 172.21.15.57:0/1478432690 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd35414fc90 msgr2=0x7fd354138fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:10.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd358d2f640 1 --2- 172.21.15.57:0/1478432690 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd35414fc90 0x7fd354138fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:10.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd358d2f640 1 -- 172.21.15.57:0/1478432690 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd340003f00 msgr2=0x7fd3540c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:10.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd358d2f640 1 --2- 172.21.15.57:0/1478432690 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd340003f00 0x7fd3540c8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:10.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd358d2f640 1 -- 172.21.15.57:0/1478432690 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd354136a20 con 0x7fd354132cf0 2023-10-30T14:44:10.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd359530640 1 --2- 172.21.15.57:0/1478432690 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd340003f00 0x7fd3540c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:44:10.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd358d2f640 1 --2- 172.21.15.57:0/1478432690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd354132cf0 0x7fd3540c8f90 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fd344009ae0 tx=0x7fd340005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:10.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd349ffb640 1 -- 172.21.15.57:0/1478432690 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3440be070 con 0x7fd354132cf0 2023-10-30T14:44:10.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd349ffb640 1 -- 172.21.15.57:0/1478432690 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd344004020 con 0x7fd354132cf0 2023-10-30T14:44:10.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd349ffb640 1 -- 172.21.15.57:0/1478432690 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd344004390 con 0x7fd354132cf0 2023-10-30T14:44:10.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd35b794640 1 -- 172.21.15.57:0/1478432690 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd354136d00 con 0x7fd354132cf0 2023-10-30T14:44:10.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd35b794640 1 -- 172.21.15.57:0/1478432690 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3540739f0 con 0x7fd354132cf0 2023-10-30T14:44:10.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd35b794640 1 -- 172.21.15.57:0/1478432690 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35404e170 con 0x7fd354132cf0 2023-10-30T14:44:10.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd349ffb640 1 -- 172.21.15.57:0/1478432690 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd3440b4040 con 0x7fd354132cf0 2023-10-30T14:44:10.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.351+0000 7fd349ffb640 1 --2- 172.21.15.57:0/1478432690 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd3380664a0 0x7fd338068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:10.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.355+0000 7fd349ffb640 1 -- 172.21.15.57:0/1478432690 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7fd34411e350 con 0x7fd354132cf0 2023-10-30T14:44:10.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.355+0000 7fd359530640 1 --2- 172.21.15.57:0/1478432690 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd3380664a0 0x7fd338068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:10.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.355+0000 7fd359530640 1 --2- 172.21.15.57:0/1478432690 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd3380664a0 0x7fd338068960 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd350009a90 tx=0x7fd350008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:10.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.359+0000 7fd349ffb640 1 -- 172.21.15.57:0/1478432690 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd3440eb2d0 con 0x7fd354132cf0 2023-10-30T14:44:10.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.547+0000 7fd35b794640 1 -- 172.21.15.57:0/1478432690 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd354153fd0 con 0x7fd354132cf0 2023-10-30T14:44:10.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.547+0000 7fd349ffb640 1 -- 172.21.15.57:0/1478432690 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+6509 (secure 0 0 0) 0x7fd3440ee8f0 con 0x7fd354132cf0 2023-10-30T14:44:10.551 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:10.551 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7536,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:57.246176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6356,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:10.551 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 163 2023-10-30T14:44:10.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.551+0000 7fd35b794640 1 -- 172.21.15.57:0/1478432690 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd3380664a0 msgr2=0x7fd338068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:10.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.551+0000 7fd35b794640 1 --2- 172.21.15.57:0/1478432690 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd3380664a0 0x7fd338068960 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd350009a90 tx=0x7fd350008040 comp rx=0 tx=0).stop 2023-10-30T14:44:10.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.551+0000 7fd35b794640 1 -- 172.21.15.57:0/1478432690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd354132cf0 msgr2=0x7fd3540c8f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:10.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.551+0000 7fd35b794640 1 --2- 172.21.15.57:0/1478432690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd354132cf0 0x7fd3540c8f90 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fd344009ae0 tx=0x7fd340005800 comp rx=0 tx=0).stop 2023-10-30T14:44:10.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.551+0000 7fd35b794640 1 -- 172.21.15.57:0/1478432690 shutdown_connections 2023-10-30T14:44:10.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.551+0000 7fd35b794640 1 --2- 172.21.15.57:0/1478432690 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd3380664a0 0x7fd338068960 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:10.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.551+0000 7fd35b794640 1 --2- 172.21.15.57:0/1478432690 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd35414fc90 0x7fd354138fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:10.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.551+0000 7fd35b794640 1 --2- 172.21.15.57:0/1478432690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd354132cf0 0x7fd3540c8f90 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:10.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.551+0000 7fd35b794640 1 --2- 172.21.15.57:0/1478432690 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd340003f00 0x7fd3540c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:10.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.551+0000 7fd35b794640 1 -- 172.21.15.57:0/1478432690 >> 172.21.15.57:0/1478432690 conn(0x7fd3540cac70 msgr2=0x7fd3540c54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:10.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.551+0000 7fd35b794640 1 -- 172.21.15.57:0/1478432690 shutdown_connections 2023-10-30T14:44:10.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:10.551+0000 7fd35b794640 1 -- 172.21.15.57:0/1478432690 wait complete. 2023-10-30T14:44:10.571 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:44:12.573 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:12.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.727+0000 7f416af80640 1 Processor -- start 2023-10-30T14:44:12.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.727+0000 7f416af80640 1 -- start start 2023-10-30T14:44:12.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.731+0000 7f416af80640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4164153c80 0x7f416415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:12.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.731+0000 7f416af80640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f41640c42a0 0x7f41640ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:12.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.731+0000 7f416af80640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4164159f20 con 0x7f41640c4a10 2023-10-30T14:44:12.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.731+0000 7f416af80640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f416415dec0 con 0x7f41640c42a0 2023-10-30T14:44:12.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.731+0000 7f416af80640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41641599a0 con 0x7f4164153c80 2023-10-30T14:44:12.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.731+0000 7f416951d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f41640c42a0 0x7f41640ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:12.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.731+0000 7f4168d1c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4164153c80 0x7f416415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:12.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.731+0000 7f4163fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f41640c4a10 0x7f416415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:56914/0 (socket says 172.21.15.57:56914) 2023-10-30T14:44:12.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.731+0000 7f4163fff640 1 -- 172.21.15.57:0/800551060 learned_addr learned my addr 172.21.15.57:0/800551060 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:44:12.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.731+0000 7f4168d1c640 1 -- 172.21.15.57:0/800551060 >> v2:172.21.15.88:3300/0 conn(0x7f41640c42a0 msgr2=0x7f41640ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:12.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.731+0000 7f4168d1c640 1 --2- 172.21.15.57:0/800551060 >> v2:172.21.15.88:3300/0 conn(0x7f41640c42a0 0x7f41640ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:12.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.731+0000 7f4168d1c640 1 -- 172.21.15.57:0/800551060 >> v1:172.21.15.88:6789/0 conn(0x7f41640c4a10 legacy=0x7f416415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:12.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.731+0000 7f4168d1c640 1 -- 172.21.15.57:0/800551060 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416415d620 con 0x7f4164153c80 2023-10-30T14:44:12.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.735+0000 7f4168d1c640 1 --2- 172.21.15.57:0/800551060 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4164153c80 0x7f416415cf90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f41580097c0 tx=0x7f415801ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bbc8c0ea02c97da server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:12.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.735+0000 7f41637fe640 1 -- 172.21.15.57:0/800551060 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41580a8970 con 0x7f4164153c80 2023-10-30T14:44:12.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.735+0000 7f41637fe640 1 -- 172.21.15.57:0/800551060 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41580a8ad0 con 0x7f4164153c80 2023-10-30T14:44:12.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.735+0000 7f41637fe640 1 -- 172.21.15.57:0/800551060 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41580a8e60 con 0x7f4164153c80 2023-10-30T14:44:12.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.735+0000 7f416af80640 1 -- 172.21.15.57:0/800551060 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4164153c80 msgr2=0x7f416415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:12.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.735+0000 7f416af80640 1 --2- 172.21.15.57:0/800551060 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4164153c80 0x7f416415cf90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f41580097c0 tx=0x7f415801ec70 comp rx=0 tx=0).stop 2023-10-30T14:44:12.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.735+0000 7f416af80640 1 -- 172.21.15.57:0/800551060 shutdown_connections 2023-10-30T14:44:12.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.735+0000 7f416af80640 1 --2- 172.21.15.57:0/800551060 >> v2:172.21.15.88:3300/0 conn(0x7f41640c42a0 0x7f41640ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:12.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.735+0000 7f416af80640 1 --2- 172.21.15.57:0/800551060 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4164153c80 0x7f416415cf90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:12.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.735+0000 7f416af80640 1 -- 172.21.15.57:0/800551060 >> 172.21.15.57:0/800551060 conn(0x7f41640715c0 msgr2=0x7f41640719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:12.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.735+0000 7f416af80640 1 -- 172.21.15.57:0/800551060 shutdown_connections 2023-10-30T14:44:12.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.735+0000 7f416af80640 1 -- 172.21.15.57:0/800551060 wait complete. 2023-10-30T14:44:12.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.735+0000 7f416af80640 1 Processor -- start 2023-10-30T14:44:12.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f416af80640 1 -- start start 2023-10-30T14:44:12.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f416af80640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41640c42a0 0x7f41640c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:12.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f416af80640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41640c4a10 0x7f4164149ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:12.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f416af80640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4164153c80 0x7f416414a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:12.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f416af80640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4164150e80 con 0x7f41640c4a10 2023-10-30T14:44:12.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f416af80640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41641553e0 con 0x7f4164153c80 2023-10-30T14:44:12.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f416af80640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41641529b0 con 0x7f41640c42a0 2023-10-30T14:44:12.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f4163fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41640c4a10 0x7f4164149ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:12.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f4168d1c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41640c42a0 0x7f41640c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:12.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f4163fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41640c4a10 0x7f4164149ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54606/0 (socket says 172.21.15.57:54606) 2023-10-30T14:44:12.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f4163fff640 1 -- 172.21.15.57:0/4014226881 learned_addr learned my addr 172.21.15.57:0/4014226881 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:12.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f416951d640 1 --2- 172.21.15.57:0/4014226881 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4164153c80 0x7f416414a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:12.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f4163fff640 1 -- 172.21.15.57:0/4014226881 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41640c42a0 msgr2=0x7f41640c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:12.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f4163fff640 1 --2- 172.21.15.57:0/4014226881 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41640c42a0 0x7f41640c88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:12.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f4163fff640 1 -- 172.21.15.57:0/4014226881 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4164153c80 msgr2=0x7f416414a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:12.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f4163fff640 1 --2- 172.21.15.57:0/4014226881 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4164153c80 0x7f416414a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:12.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f4163fff640 1 -- 172.21.15.57:0/4014226881 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4158009420 con 0x7f41640c4a10 2023-10-30T14:44:12.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f4163fff640 1 --2- 172.21.15.57:0/4014226881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41640c4a10 0x7f4164149ba0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f415000c430 tx=0x7f415000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:12.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f41617fa640 1 -- 172.21.15.57:0/4014226881 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150005cf0 con 0x7f41640c4a10 2023-10-30T14:44:12.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f41617fa640 1 -- 172.21.15.57:0/4014226881 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4150005e50 con 0x7f41640c4a10 2023-10-30T14:44:12.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f41617fa640 1 -- 172.21.15.57:0/4014226881 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41500072f0 con 0x7f41640c4a10 2023-10-30T14:44:12.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f416af80640 1 -- 172.21.15.57:0/4014226881 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41641510b0 con 0x7f41640c4a10 2023-10-30T14:44:12.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f416af80640 1 -- 172.21.15.57:0/4014226881 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4164152360 con 0x7f41640c4a10 2023-10-30T14:44:12.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.739+0000 7f416af80640 1 -- 172.21.15.57:0/4014226881 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41641556c0 con 0x7f41640c4a10 2023-10-30T14:44:12.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.743+0000 7f41617fa640 1 -- 172.21.15.57:0/4014226881 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4150007510 con 0x7f41640c4a10 2023-10-30T14:44:12.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.743+0000 7f41617fa640 1 --2- 172.21.15.57:0/4014226881 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f41480664a0 0x7f4148068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:12.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.743+0000 7f41617fa640 1 -- 172.21.15.57:0/4014226881 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f41500839c0 con 0x7f41640c4a10 2023-10-30T14:44:12.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.743+0000 7f4168d1c640 1 --2- 172.21.15.57:0/4014226881 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f41480664a0 0x7f4148068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:12.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.743+0000 7f4168d1c640 1 --2- 172.21.15.57:0/4014226881 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f41480664a0 0x7f4148068960 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f41580b2000 tx=0x7f41580aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:12.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.747+0000 7f41617fa640 1 -- 172.21.15.57:0/4014226881 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f41500508e0 con 0x7f41640c4a10 2023-10-30T14:44:12.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.935+0000 7f416af80640 1 -- 172.21.15.57:0/4014226881 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f416404e170 con 0x7f41640c4a10 2023-10-30T14:44:12.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.935+0000 7f41617fa640 1 -- 172.21.15.57:0/4014226881 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+6509 (secure 0 0 0) 0x7f4150017020 con 0x7f41640c4a10 2023-10-30T14:44:12.938 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:12.938 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7536,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:57.246176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6356,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:12.938 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 163 2023-10-30T14:44:12.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.939+0000 7f416af80640 1 -- 172.21.15.57:0/4014226881 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f41480664a0 msgr2=0x7f4148068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:12.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.939+0000 7f416af80640 1 --2- 172.21.15.57:0/4014226881 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f41480664a0 0x7f4148068960 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f41580b2000 tx=0x7f41580aa070 comp rx=0 tx=0).stop 2023-10-30T14:44:12.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.939+0000 7f416af80640 1 -- 172.21.15.57:0/4014226881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41640c4a10 msgr2=0x7f4164149ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:12.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.939+0000 7f416af80640 1 --2- 172.21.15.57:0/4014226881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41640c4a10 0x7f4164149ba0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f415000c430 tx=0x7f415000c8f0 comp rx=0 tx=0).stop 2023-10-30T14:44:12.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.939+0000 7f416af80640 1 -- 172.21.15.57:0/4014226881 shutdown_connections 2023-10-30T14:44:12.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.939+0000 7f416af80640 1 --2- 172.21.15.57:0/4014226881 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f41480664a0 0x7f4148068960 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:12.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.939+0000 7f416af80640 1 --2- 172.21.15.57:0/4014226881 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4164153c80 0x7f416414a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:12.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.939+0000 7f416af80640 1 --2- 172.21.15.57:0/4014226881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41640c4a10 0x7f4164149ba0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:12.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.939+0000 7f416af80640 1 --2- 172.21.15.57:0/4014226881 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41640c42a0 0x7f41640c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:12.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.939+0000 7f416af80640 1 -- 172.21.15.57:0/4014226881 >> 172.21.15.57:0/4014226881 conn(0x7f41640715c0 msgr2=0x7f4164158640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:12.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.939+0000 7f416af80640 1 -- 172.21.15.57:0/4014226881 shutdown_connections 2023-10-30T14:44:12.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:12.939+0000 7f416af80640 1 -- 172.21.15.57:0/4014226881 wait complete. 2023-10-30T14:44:12.958 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:44:14.960 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:15.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.114+0000 7efd06fd3640 1 Processor -- start 2023-10-30T14:44:15.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.114+0000 7efd06fd3640 1 -- start start 2023-10-30T14:44:15.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.114+0000 7efd06fd3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efd000c6320 0x7efd0006fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:15.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.114+0000 7efd06fd3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7efd0014fc50 0x7efd00070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:15.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.114+0000 7efd06fd3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd000c55f0 con 0x7efd00155c40 2023-10-30T14:44:15.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.114+0000 7efd06fd3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd000c5f00 con 0x7efd0014fc50 2023-10-30T14:44:15.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.114+0000 7efd06fd3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd00071350 con 0x7efd000c6320 2023-10-30T14:44:15.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.114+0000 7efcfffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7efd0014fc50 0x7efd00070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:15.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.114+0000 7efcfffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7efd0014fc50 0x7efd00070e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57962/0 (socket says 172.21.15.57:57962) 2023-10-30T14:44:15.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.114+0000 7efcfffff640 1 -- 172.21.15.57:0/564112415 learned_addr learned my addr 172.21.15.57:0/564112415 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:15.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.118+0000 7efd04d6f640 1 --2- 172.21.15.57:0/564112415 >> v2:172.21.15.88:3300/0 conn(0x7efd000c6320 0x7efd0006fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:15.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.118+0000 7efcfffff640 1 -- 172.21.15.57:0/564112415 >> v2:172.21.15.88:3300/0 conn(0x7efd000c6320 msgr2=0x7efd0006fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:15.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.118+0000 7efcfffff640 1 --2- 172.21.15.57:0/564112415 >> v2:172.21.15.88:3300/0 conn(0x7efd000c6320 0x7efd0006fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:15.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.118+0000 7efcfffff640 1 -- 172.21.15.57:0/564112415 >> v1:172.21.15.88:6789/0 conn(0x7efd00155c40 legacy=0x7efd00159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:15.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.118+0000 7efcfffff640 1 -- 172.21.15.57:0/564112415 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd001567b0 con 0x7efd0014fc50 2023-10-30T14:44:15.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.118+0000 7efcfffff640 1 --2- 172.21.15.57:0/564112415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd0014fc50 0x7efd00070e10 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7efd0006f6e0 tx=0x7efcec01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39390f783d69e054 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:15.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.118+0000 7efcff7fe640 1 -- 172.21.15.57:0/564112415 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec0a7070 con 0x7efd0014fc50 2023-10-30T14:44:15.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.118+0000 7efcff7fe640 1 -- 172.21.15.57:0/564112415 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcec0ab430 con 0x7efd0014fc50 2023-10-30T14:44:15.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.118+0000 7efcff7fe640 1 -- 172.21.15.57:0/564112415 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec0ab7c0 con 0x7efd0014fc50 2023-10-30T14:44:15.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd06fd3640 1 -- 172.21.15.57:0/564112415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd0014fc50 msgr2=0x7efd00070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:15.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd06fd3640 1 --2- 172.21.15.57:0/564112415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd0014fc50 0x7efd00070e10 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7efd0006f6e0 tx=0x7efcec01c430 comp rx=0 tx=0).stop 2023-10-30T14:44:15.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd06fd3640 1 -- 172.21.15.57:0/564112415 shutdown_connections 2023-10-30T14:44:15.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd06fd3640 1 --2- 172.21.15.57:0/564112415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd0014fc50 0x7efd00070e10 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:15.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd06fd3640 1 --2- 172.21.15.57:0/564112415 >> v2:172.21.15.88:3300/0 conn(0x7efd000c6320 0x7efd0006fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:15.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd06fd3640 1 -- 172.21.15.57:0/564112415 >> 172.21.15.57:0/564112415 conn(0x7efd000cac70 msgr2=0x7efd000cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:15.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd06fd3640 1 -- 172.21.15.57:0/564112415 shutdown_connections 2023-10-30T14:44:15.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd06fd3640 1 -- 172.21.15.57:0/564112415 wait complete. 2023-10-30T14:44:15.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd06fd3640 1 Processor -- start 2023-10-30T14:44:15.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd06fd3640 1 -- start start 2023-10-30T14:44:15.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd06fd3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd000c6320 0x7efd001f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:15.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd06fd3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0014fc50 0x7efd001417b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:15.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd06fd3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd00155c40 0x7efd000c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:15.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd06fd3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd00140240 con 0x7efd000c6320 2023-10-30T14:44:15.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd06fd3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd001f3a20 con 0x7efd00155c40 2023-10-30T14:44:15.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd06fd3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd00141320 con 0x7efd0014fc50 2023-10-30T14:44:15.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd04d6f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd000c6320 0x7efd001f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:15.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd04d6f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd000c6320 0x7efd001f34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57974/0 (socket says 172.21.15.57:57974) 2023-10-30T14:44:15.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd04d6f640 1 -- 172.21.15.57:0/2769921908 learned_addr learned my addr 172.21.15.57:0/2769921908 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:15.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd05570640 1 --2- 172.21.15.57:0/2769921908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd00155c40 0x7efd000c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:15.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efcfffff640 1 --2- 172.21.15.57:0/2769921908 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0014fc50 0x7efd001417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:15.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd04d6f640 1 -- 172.21.15.57:0/2769921908 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0014fc50 msgr2=0x7efd001417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:15.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd04d6f640 1 --2- 172.21.15.57:0/2769921908 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0014fc50 0x7efd001417b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:15.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd04d6f640 1 -- 172.21.15.57:0/2769921908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd00155c40 msgr2=0x7efd000c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:15.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd04d6f640 1 --2- 172.21.15.57:0/2769921908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd00155c40 0x7efd000c9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:15.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.122+0000 7efd04d6f640 1 -- 172.21.15.57:0/2769921908 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec0094d0 con 0x7efd000c6320 2023-10-30T14:44:15.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.126+0000 7efd04d6f640 1 --2- 172.21.15.57:0/2769921908 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd000c6320 0x7efd001f34e0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7efcf400d800 tx=0x7efcf400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:15.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.126+0000 7efcfd7fa640 1 -- 172.21.15.57:0/2769921908 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf400b660 con 0x7efd000c6320 2023-10-30T14:44:15.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.126+0000 7efcfd7fa640 1 -- 172.21.15.57:0/2769921908 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf400b7c0 con 0x7efd000c6320 2023-10-30T14:44:15.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.126+0000 7efd06fd3640 1 -- 172.21.15.57:0/2769921908 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd00140a30 con 0x7efd000c6320 2023-10-30T14:44:15.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.126+0000 7efcfd7fa640 1 -- 172.21.15.57:0/2769921908 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf400bb30 con 0x7efd000c6320 2023-10-30T14:44:15.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.126+0000 7efd06fd3640 1 -- 172.21.15.57:0/2769921908 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd00140750 con 0x7efd000c6320 2023-10-30T14:44:15.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.126+0000 7efd06fd3640 1 -- 172.21.15.57:0/2769921908 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd0004e170 con 0x7efd000c6320 2023-10-30T14:44:15.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.126+0000 7efcfd7fa640 1 -- 172.21.15.57:0/2769921908 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efcf400bc90 con 0x7efd000c6320 2023-10-30T14:44:15.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.126+0000 7efcfd7fa640 1 --2- 172.21.15.57:0/2769921908 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efce40664d0 0x7efce4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:15.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.126+0000 7efcfd7fa640 1 -- 172.21.15.57:0/2769921908 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7efcf40847d0 con 0x7efd000c6320 2023-10-30T14:44:15.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.126+0000 7efcfffff640 1 --2- 172.21.15.57:0/2769921908 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efce40664d0 0x7efce4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:15.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.130+0000 7efcfffff640 1 --2- 172.21.15.57:0/2769921908 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efce40664d0 0x7efce4068990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7efcec0af000 tx=0x7efcec008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:15.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.130+0000 7efcfd7fa640 1 -- 172.21.15.57:0/2769921908 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efcf40516e0 con 0x7efd000c6320 2023-10-30T14:44:15.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.326+0000 7efd06fd3640 1 -- 172.21.15.57:0/2769921908 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd00070600 con 0x7efd000c6320 2023-10-30T14:44:15.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.326+0000 7efcfd7fa640 1 -- 172.21.15.57:0/2769921908 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+6509 (secure 0 0 0) 0x7efcf4054d00 con 0x7efd000c6320 2023-10-30T14:44:15.330 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:15.330 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7536,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:57.246176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6356,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:15.330 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 163 2023-10-30T14:44:15.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.330+0000 7efd06fd3640 1 -- 172.21.15.57:0/2769921908 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efce40664d0 msgr2=0x7efce4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:15.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.330+0000 7efd06fd3640 1 --2- 172.21.15.57:0/2769921908 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efce40664d0 0x7efce4068990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7efcec0af000 tx=0x7efcec008040 comp rx=0 tx=0).stop 2023-10-30T14:44:15.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.330+0000 7efd06fd3640 1 -- 172.21.15.57:0/2769921908 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd000c6320 msgr2=0x7efd001f34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:15.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.330+0000 7efd06fd3640 1 --2- 172.21.15.57:0/2769921908 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd000c6320 0x7efd001f34e0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7efcf400d800 tx=0x7efcf400dcc0 comp rx=0 tx=0).stop 2023-10-30T14:44:15.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.330+0000 7efd06fd3640 1 -- 172.21.15.57:0/2769921908 shutdown_connections 2023-10-30T14:44:15.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.330+0000 7efd06fd3640 1 --2- 172.21.15.57:0/2769921908 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efce40664d0 0x7efce4068990 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:15.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.330+0000 7efd06fd3640 1 --2- 172.21.15.57:0/2769921908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd00155c40 0x7efd000c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:15.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.330+0000 7efd06fd3640 1 --2- 172.21.15.57:0/2769921908 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0014fc50 0x7efd001417b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:15.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.330+0000 7efd06fd3640 1 --2- 172.21.15.57:0/2769921908 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd000c6320 0x7efd001f34e0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:15.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.330+0000 7efd06fd3640 1 -- 172.21.15.57:0/2769921908 >> 172.21.15.57:0/2769921908 conn(0x7efd000cac70 msgr2=0x7efd001552b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:15.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.330+0000 7efd06fd3640 1 -- 172.21.15.57:0/2769921908 shutdown_connections 2023-10-30T14:44:15.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:15.330+0000 7efd06fd3640 1 -- 172.21.15.57:0/2769921908 wait complete. 2023-10-30T14:44:15.351 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:44:17.352 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:17.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.506+0000 7fd07012a640 1 Processor -- start 2023-10-30T14:44:17.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.506+0000 7fd07012a640 1 -- start start 2023-10-30T14:44:17.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.506+0000 7fd07012a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd068151c20 0x7fd0680c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:17.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.506+0000 7fd07012a640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd0680cb8b0 0x7fd06806b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:17.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.506+0000 7fd07012a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0680c5030 con 0x7fd068151840 2023-10-30T14:44:17.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.506+0000 7fd07012a640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd068157830 con 0x7fd0680cb8b0 2023-10-30T14:44:17.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.506+0000 7fd07012a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd068157b10 con 0x7fd068151c20 2023-10-30T14:44:17.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.506+0000 7fd06e6c7640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd0680cb8b0 0x7fd06806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:17.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.506+0000 7fd06e6c7640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd0680cb8b0 0x7fd06806b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57984/0 (socket says 172.21.15.57:57984) 2023-10-30T14:44:17.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.506+0000 7fd06e6c7640 1 -- 172.21.15.57:0/4193031628 learned_addr learned my addr 172.21.15.57:0/4193031628 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:17.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.506+0000 7fd06d6c5640 1 --2- 172.21.15.57:0/4193031628 >> v2:172.21.15.88:3300/0 conn(0x7fd068151c20 0x7fd0680c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:17.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.506+0000 7fd06e6c7640 1 -- 172.21.15.57:0/4193031628 >> v2:172.21.15.88:3300/0 conn(0x7fd068151c20 msgr2=0x7fd0680c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:17.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.506+0000 7fd06e6c7640 1 --2- 172.21.15.57:0/4193031628 >> v2:172.21.15.88:3300/0 conn(0x7fd068151c20 0x7fd0680c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:17.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.506+0000 7fd06e6c7640 1 -- 172.21.15.57:0/4193031628 >> v1:172.21.15.88:6789/0 conn(0x7fd068151840 legacy=0x7fd068072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:17.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.506+0000 7fd06e6c7640 1 -- 172.21.15.57:0/4193031628 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd068157df0 con 0x7fd0680cb8b0 2023-10-30T14:44:17.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.510+0000 7fd06e6c7640 1 --2- 172.21.15.57:0/4193031628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0680cb8b0 0x7fd06806b600 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fd06806d4d0 tx=0x7fd06001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=213c3fe0f01afdc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:17.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.510+0000 7fd06cec4640 1 -- 172.21.15.57:0/4193031628 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0600af400 con 0x7fd0680cb8b0 2023-10-30T14:44:17.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.510+0000 7fd06cec4640 1 -- 172.21.15.57:0/4193031628 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0600ab050 con 0x7fd0680cb8b0 2023-10-30T14:44:17.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.510+0000 7fd06cec4640 1 -- 172.21.15.57:0/4193031628 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0600af780 con 0x7fd0680cb8b0 2023-10-30T14:44:17.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.510+0000 7fd07012a640 1 -- 172.21.15.57:0/4193031628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0680cb8b0 msgr2=0x7fd06806b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:17.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.510+0000 7fd07012a640 1 --2- 172.21.15.57:0/4193031628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0680cb8b0 0x7fd06806b600 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fd06806d4d0 tx=0x7fd06001f1c0 comp rx=0 tx=0).stop 2023-10-30T14:44:17.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.510+0000 7fd07012a640 1 -- 172.21.15.57:0/4193031628 shutdown_connections 2023-10-30T14:44:17.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.510+0000 7fd07012a640 1 --2- 172.21.15.57:0/4193031628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0680cb8b0 0x7fd06806b600 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:17.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.510+0000 7fd07012a640 1 --2- 172.21.15.57:0/4193031628 >> v2:172.21.15.88:3300/0 conn(0x7fd068151c20 0x7fd0680c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:17.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.510+0000 7fd07012a640 1 -- 172.21.15.57:0/4193031628 >> 172.21.15.57:0/4193031628 conn(0x7fd0680cac70 msgr2=0x7fd0680cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:17.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.510+0000 7fd07012a640 1 -- 172.21.15.57:0/4193031628 shutdown_connections 2023-10-30T14:44:17.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.510+0000 7fd07012a640 1 -- 172.21.15.57:0/4193031628 wait complete. 2023-10-30T14:44:17.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd07012a640 1 Processor -- start 2023-10-30T14:44:17.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd07012a640 1 -- start start 2023-10-30T14:44:17.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd07012a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0680cb8b0 0x7fd06814cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:17.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd07012a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd068151840 0x7fd0681f1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:17.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd07012a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd068151c20 0x7fd0681e4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:17.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd07012a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06814c7b0 con 0x7fd068151c20 2023-10-30T14:44:17.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd07012a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06814c090 con 0x7fd068151840 2023-10-30T14:44:17.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd07012a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd06814c370 con 0x7fd0680cb8b0 2023-10-30T14:44:17.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd06e6c7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd068151c20 0x7fd0681e4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:17.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd06e6c7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd068151c20 0x7fd0681e4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58000/0 (socket says 172.21.15.57:58000) 2023-10-30T14:44:17.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd06e6c7640 1 -- 172.21.15.57:0/2060870159 learned_addr learned my addr 172.21.15.57:0/2060870159 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:17.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd06d6c5640 1 --2- 172.21.15.57:0/2060870159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd068151840 0x7fd0681f1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:17.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd06dec6640 1 --2- 172.21.15.57:0/2060870159 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0680cb8b0 0x7fd06814cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:17.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd06e6c7640 1 -- 172.21.15.57:0/2060870159 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0680cb8b0 msgr2=0x7fd06814cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:17.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd06e6c7640 1 --2- 172.21.15.57:0/2060870159 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0680cb8b0 0x7fd06814cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:17.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd06e6c7640 1 -- 172.21.15.57:0/2060870159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd068151840 msgr2=0x7fd0681f1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:17.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd06e6c7640 1 --2- 172.21.15.57:0/2060870159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd068151840 0x7fd0681f1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:17.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd06e6c7640 1 -- 172.21.15.57:0/2060870159 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd060009b30 con 0x7fd068151c20 2023-10-30T14:44:17.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd06d6c5640 1 --2- 172.21.15.57:0/2060870159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd068151840 0x7fd0681f1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:44:17.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd06e6c7640 1 --2- 172.21.15.57:0/2060870159 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd068151c20 0x7fd0681e4940 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fd0600b2040 tx=0x7fd06001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:17.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd05e7fc640 1 -- 172.21.15.57:0/2060870159 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0600afd30 con 0x7fd068151c20 2023-10-30T14:44:17.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd05e7fc640 1 -- 172.21.15.57:0/2060870159 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd060009510 con 0x7fd068151c20 2023-10-30T14:44:17.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd07012a640 1 -- 172.21.15.57:0/2060870159 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0681e07b0 con 0x7fd068151c20 2023-10-30T14:44:17.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd05e7fc640 1 -- 172.21.15.57:0/2060870159 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0600ba660 con 0x7fd068151c20 2023-10-30T14:44:17.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.514+0000 7fd07012a640 1 -- 172.21.15.57:0/2060870159 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0681e04b0 con 0x7fd068151c20 2023-10-30T14:44:17.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.518+0000 7fd07012a640 1 -- 172.21.15.57:0/2060870159 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd06814c900 con 0x7fd068151c20 2023-10-30T14:44:17.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.518+0000 7fd05e7fc640 1 -- 172.21.15.57:0/2060870159 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd0600b6040 con 0x7fd068151c20 2023-10-30T14:44:17.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.518+0000 7fd05e7fc640 1 --2- 172.21.15.57:0/2060870159 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd04c0664a0 0x7fd04c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:17.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.518+0000 7fd05e7fc640 1 -- 172.21.15.57:0/2060870159 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7fd060122760 con 0x7fd068151c20 2023-10-30T14:44:17.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.518+0000 7fd06dec6640 1 --2- 172.21.15.57:0/2060870159 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd04c0664a0 0x7fd04c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:17.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.518+0000 7fd06dec6640 1 --2- 172.21.15.57:0/2060870159 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd04c0664a0 0x7fd04c068960 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd064000f70 tx=0x7fd064009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:17.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.522+0000 7fd05e7fc640 1 -- 172.21.15.57:0/2060870159 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd06814c900 con 0x7fd068151c20 2023-10-30T14:44:17.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.714+0000 7fd07012a640 1 -- 172.21.15.57:0/2060870159 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd06814c900 con 0x7fd068151c20 2023-10-30T14:44:17.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.714+0000 7fd05e7fc640 1 -- 172.21.15.57:0/2060870159 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+6509 (secure 0 0 0) 0x7fd06814c900 con 0x7fd068151c20 2023-10-30T14:44:17.717 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:17.717 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7536,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:57.246176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6356,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:17.718 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 163 2023-10-30T14:44:17.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.718+0000 7fd07012a640 1 -- 172.21.15.57:0/2060870159 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd04c0664a0 msgr2=0x7fd04c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:17.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.718+0000 7fd07012a640 1 --2- 172.21.15.57:0/2060870159 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd04c0664a0 0x7fd04c068960 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd064000f70 tx=0x7fd064009040 comp rx=0 tx=0).stop 2023-10-30T14:44:17.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.718+0000 7fd07012a640 1 -- 172.21.15.57:0/2060870159 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd068151c20 msgr2=0x7fd0681e4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:17.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.718+0000 7fd07012a640 1 --2- 172.21.15.57:0/2060870159 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd068151c20 0x7fd0681e4940 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fd0600b2040 tx=0x7fd06001bf60 comp rx=0 tx=0).stop 2023-10-30T14:44:17.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.718+0000 7fd07012a640 1 -- 172.21.15.57:0/2060870159 shutdown_connections 2023-10-30T14:44:17.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.718+0000 7fd07012a640 1 --2- 172.21.15.57:0/2060870159 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd04c0664a0 0x7fd04c068960 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:17.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.718+0000 7fd07012a640 1 --2- 172.21.15.57:0/2060870159 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd068151c20 0x7fd0681e4940 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:17.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.718+0000 7fd07012a640 1 --2- 172.21.15.57:0/2060870159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd068151840 0x7fd0681f1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:17.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.718+0000 7fd07012a640 1 --2- 172.21.15.57:0/2060870159 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0680cb8b0 0x7fd06814cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:17.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.718+0000 7fd07012a640 1 -- 172.21.15.57:0/2060870159 >> 172.21.15.57:0/2060870159 conn(0x7fd0680cac70 msgr2=0x7fd068145650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:17.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.718+0000 7fd07012a640 1 -- 172.21.15.57:0/2060870159 shutdown_connections 2023-10-30T14:44:17.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:17.718+0000 7fd07012a640 1 -- 172.21.15.57:0/2060870159 wait complete. 2023-10-30T14:44:17.738 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:44:19.740 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:19.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.894+0000 7f65bbe46640 1 Processor -- start 2023-10-30T14:44:19.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.894+0000 7f65bbe46640 1 -- start start 2023-10-30T14:44:19.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.894+0000 7f65bbe46640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f65b4153c10 0x7f65b4065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:19.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.894+0000 7f65bbe46640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f65b4070c30 0x7f65b4071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:19.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.894+0000 7f65bbe46640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65b4159970 0x7f65b4159e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:19.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.894+0000 7f65bbe46640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b4072390 con 0x7f65b4153c10 2023-10-30T14:44:19.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.894+0000 7f65bbe46640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b406e390 con 0x7f65b4070c30 2023-10-30T14:44:19.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.894+0000 7f65bbe46640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b40660e0 con 0x7f65b4159970 2023-10-30T14:44:19.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.894+0000 7f65b9be2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f65b4153c10 0x7f65b4065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:19.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.894+0000 7f65b9be2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f65b4153c10 0x7f65b4065ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58008/0 (socket says 172.21.15.57:58008) 2023-10-30T14:44:19.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.894+0000 7f65b9be2640 1 -- 172.21.15.57:0/2552018900 learned_addr learned my addr 172.21.15.57:0/2552018900 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:19.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.894+0000 7f65b93e1640 1 --2- 172.21.15.57:0/2552018900 >> v2:172.21.15.88:3300/0 conn(0x7f65b4070c30 0x7f65b4071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:19.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.894+0000 7f65ba3e3640 1 --2- 172.21.15.57:0/2552018900 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65b4159970 0x7f65b4159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:19.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.898+0000 7f65b9be2640 1 -- 172.21.15.57:0/2552018900 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65b4159970 msgr2=0x7f65b4159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:19.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.898+0000 7f65b9be2640 1 --2- 172.21.15.57:0/2552018900 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65b4159970 0x7f65b4159e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:19.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.898+0000 7f65b9be2640 1 -- 172.21.15.57:0/2552018900 >> v2:172.21.15.88:3300/0 conn(0x7f65b4070c30 msgr2=0x7f65b4071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:19.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.898+0000 7f65b9be2640 1 --2- 172.21.15.57:0/2552018900 >> v2:172.21.15.88:3300/0 conn(0x7f65b4070c30 0x7f65b4071070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:19.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.898+0000 7f65b9be2640 1 -- 172.21.15.57:0/2552018900 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b40715b0 con 0x7f65b4153c10 2023-10-30T14:44:19.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.898+0000 7f65ba3e3640 1 --2- 172.21.15.57:0/2552018900 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65b4159970 0x7f65b4159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:44:19.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.898+0000 7f65b9be2640 1 --2- 172.21.15.57:0/2552018900 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65b4153c10 0x7f65b4065ba0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f65b40717a0 tx=0x7f65b001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b76105fdae099 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:19.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.898+0000 7f65b8be0640 1 -- 172.21.15.57:0/2552018900 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b00ab600 con 0x7f65b4153c10 2023-10-30T14:44:19.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.898+0000 7f65b8be0640 1 -- 172.21.15.57:0/2552018900 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b00a7050 con 0x7f65b4153c10 2023-10-30T14:44:19.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.898+0000 7f65b8be0640 1 -- 172.21.15.57:0/2552018900 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b00ab980 con 0x7f65b4153c10 2023-10-30T14:44:19.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.898+0000 7f65bbe46640 1 -- 172.21.15.57:0/2552018900 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65b4153c10 msgr2=0x7f65b4065ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:19.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.898+0000 7f65bbe46640 1 --2- 172.21.15.57:0/2552018900 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65b4153c10 0x7f65b4065ba0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f65b40717a0 tx=0x7f65b001c470 comp rx=0 tx=0).stop 2023-10-30T14:44:19.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65bbe46640 1 -- 172.21.15.57:0/2552018900 shutdown_connections 2023-10-30T14:44:19.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65bbe46640 1 --2- 172.21.15.57:0/2552018900 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65b4159970 0x7f65b4159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:19.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65bbe46640 1 --2- 172.21.15.57:0/2552018900 >> v2:172.21.15.88:3300/0 conn(0x7f65b4070c30 0x7f65b4071070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:19.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65bbe46640 1 --2- 172.21.15.57:0/2552018900 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65b4153c10 0x7f65b4065ba0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:19.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65bbe46640 1 -- 172.21.15.57:0/2552018900 >> 172.21.15.57:0/2552018900 conn(0x7f65b40c7f60 msgr2=0x7f65b40c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:19.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65bbe46640 1 -- 172.21.15.57:0/2552018900 shutdown_connections 2023-10-30T14:44:19.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65bbe46640 1 -- 172.21.15.57:0/2552018900 wait complete. 2023-10-30T14:44:19.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65bbe46640 1 Processor -- start 2023-10-30T14:44:19.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65bbe46640 1 -- start start 2023-10-30T14:44:19.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65bbe46640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65b4070c30 0x7f65b415c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:19.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65bbe46640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65b4153c10 0x7f65b415b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:19.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65bbe46640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65b4159970 0x7f65b414f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:19.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65bbe46640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b414a1a0 con 0x7f65b4153c10 2023-10-30T14:44:19.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65bbe46640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b414b320 con 0x7f65b4159970 2023-10-30T14:44:19.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65bbe46640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b414aa80 con 0x7f65b4070c30 2023-10-30T14:44:19.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65b93e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65b4153c10 0x7f65b415b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:19.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65b93e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65b4153c10 0x7f65b415b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58014/0 (socket says 172.21.15.57:58014) 2023-10-30T14:44:19.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65b93e1640 1 -- 172.21.15.57:0/4113501320 learned_addr learned my addr 172.21.15.57:0/4113501320 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:19.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65ba3e3640 1 --2- 172.21.15.57:0/4113501320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65b4159970 0x7f65b414f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:19.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65b9be2640 1 --2- 172.21.15.57:0/4113501320 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65b4070c30 0x7f65b415c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:19.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65b93e1640 1 -- 172.21.15.57:0/4113501320 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65b4070c30 msgr2=0x7f65b415c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:19.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65b93e1640 1 --2- 172.21.15.57:0/4113501320 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65b4070c30 0x7f65b415c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:19.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65b93e1640 1 -- 172.21.15.57:0/4113501320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65b4159970 msgr2=0x7f65b414f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:19.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65b93e1640 1 --2- 172.21.15.57:0/4113501320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65b4159970 0x7f65b414f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:19.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.902+0000 7f65b93e1640 1 -- 172.21.15.57:0/4113501320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b00094d0 con 0x7f65b4153c10 2023-10-30T14:44:19.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.906+0000 7f65b93e1640 1 --2- 172.21.15.57:0/4113501320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65b4153c10 0x7f65b415b4e0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f65a400e8b0 tx=0x7f65a400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:19.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.906+0000 7f65aa7fc640 1 -- 172.21.15.57:0/4113501320 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a400c940 con 0x7f65b4153c10 2023-10-30T14:44:19.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.906+0000 7f65aa7fc640 1 -- 172.21.15.57:0/4113501320 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a400caa0 con 0x7f65b4153c10 2023-10-30T14:44:19.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.906+0000 7f65bbe46640 1 -- 172.21.15.57:0/4113501320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b414ad40 con 0x7f65b4153c10 2023-10-30T14:44:19.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.906+0000 7f65aa7fc640 1 -- 172.21.15.57:0/4113501320 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a400ce30 con 0x7f65b4153c10 2023-10-30T14:44:19.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.906+0000 7f65bbe46640 1 -- 172.21.15.57:0/4113501320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b415be90 con 0x7f65b4153c10 2023-10-30T14:44:19.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.906+0000 7f65bbe46640 1 -- 172.21.15.57:0/4113501320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b414b4d0 con 0x7f65b4153c10 2023-10-30T14:44:19.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.906+0000 7f65aa7fc640 1 -- 172.21.15.57:0/4113501320 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f65a4013470 con 0x7f65b4153c10 2023-10-30T14:44:19.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.906+0000 7f65aa7fc640 1 --2- 172.21.15.57:0/4113501320 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65980664d0 0x7f6598068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:19.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.906+0000 7f65aa7fc640 1 -- 172.21.15.57:0/4113501320 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f65a4084540 con 0x7f65b4153c10 2023-10-30T14:44:19.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.910+0000 7f65b9be2640 1 --2- 172.21.15.57:0/4113501320 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65980664d0 0x7f6598068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:19.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.910+0000 7f65b9be2640 1 --2- 172.21.15.57:0/4113501320 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65980664d0 0x7f6598068990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f65b00ad000 tx=0x7f65b00a6000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:19.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:19.914+0000 7f65aa7fc640 1 -- 172.21.15.57:0/4113501320 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f65a4051450 con 0x7f65b4153c10 2023-10-30T14:44:20.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:20.102+0000 7f65bbe46640 1 -- 172.21.15.57:0/4113501320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65b404e170 con 0x7f65b4153c10 2023-10-30T14:44:20.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:20.102+0000 7f65aa7fc640 1 -- 172.21.15.57:0/4113501320 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+6509 (secure 0 0 0) 0x7f65a4054a70 con 0x7f65b4153c10 2023-10-30T14:44:20.104 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:20.104 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7536,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:43:57.246176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6356,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6356":{"gid":6356,"name":"f","rank":0,"incarnation":151,"state":"up:active","state_seq":45,"addr":"172.21.15.88:6839/1744808283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1744808283},{"type":"v1","addr":"172.21.15.88:6839","nonce":1744808283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:20.105 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 163 2023-10-30T14:44:20.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:20.102+0000 7f65bbe46640 1 -- 172.21.15.57:0/4113501320 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65980664d0 msgr2=0x7f6598068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:20.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:20.102+0000 7f65bbe46640 1 --2- 172.21.15.57:0/4113501320 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65980664d0 0x7f6598068990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f65b00ad000 tx=0x7f65b00a6000 comp rx=0 tx=0).stop 2023-10-30T14:44:20.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:20.102+0000 7f65bbe46640 1 -- 172.21.15.57:0/4113501320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65b4153c10 msgr2=0x7f65b415b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:20.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:20.102+0000 7f65bbe46640 1 --2- 172.21.15.57:0/4113501320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65b4153c10 0x7f65b415b4e0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f65a400e8b0 tx=0x7f65a400ed70 comp rx=0 tx=0).stop 2023-10-30T14:44:20.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:20.102+0000 7f65bbe46640 1 -- 172.21.15.57:0/4113501320 shutdown_connections 2023-10-30T14:44:20.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:20.102+0000 7f65bbe46640 1 --2- 172.21.15.57:0/4113501320 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65980664d0 0x7f6598068990 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:20.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:20.102+0000 7f65bbe46640 1 --2- 172.21.15.57:0/4113501320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65b4159970 0x7f65b414f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:20.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:20.102+0000 7f65bbe46640 1 --2- 172.21.15.57:0/4113501320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65b4153c10 0x7f65b415b4e0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:20.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:20.102+0000 7f65bbe46640 1 --2- 172.21.15.57:0/4113501320 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65b4070c30 0x7f65b415c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:20.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:20.102+0000 7f65bbe46640 1 -- 172.21.15.57:0/4113501320 >> 172.21.15.57:0/4113501320 conn(0x7f65b40c7f60 msgr2=0x7f65b414cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:20.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:20.102+0000 7f65bbe46640 1 -- 172.21.15.57:0/4113501320 shutdown_connections 2023-10-30T14:44:20.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:20.102+0000 7f65bbe46640 1 -- 172.21.15.57:0/4113501320 wait complete. 2023-10-30T14:44:20.125 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T14:44:22.126 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:22.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.290+0000 7fafe4172640 1 Processor -- start 2023-10-30T14:44:22.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.290+0000 7fafe4172640 1 -- start start 2023-10-30T14:44:22.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.290+0000 7fafe4172640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fafdc151840 0x7fafdc0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:22.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.290+0000 7fafe4172640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fafdc0cb8b0 0x7fafdc06b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:22.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.290+0000 7fafe4172640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafdc0c5030 con 0x7fafdc151c20 2023-10-30T14:44:22.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.290+0000 7fafe4172640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafdc157830 con 0x7fafdc151840 2023-10-30T14:44:22.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.290+0000 7fafe4172640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafdc157b10 con 0x7fafdc0cb8b0 2023-10-30T14:44:22.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.294+0000 7fafe1f0e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fafdc151840 0x7fafdc0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:22.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.294+0000 7fafe1f0e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fafdc151840 0x7fafdc0c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58028/0 (socket says 172.21.15.57:58028) 2023-10-30T14:44:22.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.294+0000 7fafe1f0e640 1 -- 172.21.15.57:0/2839321549 learned_addr learned my addr 172.21.15.57:0/2839321549 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:22.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.294+0000 7fafe270f640 1 --2- 172.21.15.57:0/2839321549 >> v2:172.21.15.88:3300/0 conn(0x7fafdc0cb8b0 0x7fafdc06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:22.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.294+0000 7fafe1f0e640 1 -- 172.21.15.57:0/2839321549 >> v2:172.21.15.88:3300/0 conn(0x7fafdc0cb8b0 msgr2=0x7fafdc06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:22.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.294+0000 7fafe1f0e640 1 --2- 172.21.15.57:0/2839321549 >> v2:172.21.15.88:3300/0 conn(0x7fafdc0cb8b0 0x7fafdc06b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.294+0000 7fafe1f0e640 1 -- 172.21.15.57:0/2839321549 >> v1:172.21.15.88:6789/0 conn(0x7fafdc151c20 legacy=0x7fafdc072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:22.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.294+0000 7fafe1f0e640 1 -- 172.21.15.57:0/2839321549 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc157df0 con 0x7fafdc151840 2023-10-30T14:44:22.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.294+0000 7fafe1f0e640 1 --2- 172.21.15.57:0/2839321549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafdc151840 0x7fafdc0c49a0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fafdc0c6cb0 tx=0x7fafd801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2d204af091ea262 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:22.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.294+0000 7fafe0f0c640 1 -- 172.21.15.57:0/2839321549 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd80a89f0 con 0x7fafdc151840 2023-10-30T14:44:22.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.294+0000 7fafe0f0c640 1 -- 172.21.15.57:0/2839321549 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd80a8b50 con 0x7fafdc151840 2023-10-30T14:44:22.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.294+0000 7fafe0f0c640 1 -- 172.21.15.57:0/2839321549 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd80b0440 con 0x7fafdc151840 2023-10-30T14:44:22.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe4172640 1 -- 172.21.15.57:0/2839321549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafdc151840 msgr2=0x7fafdc0c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:22.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe4172640 1 --2- 172.21.15.57:0/2839321549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafdc151840 0x7fafdc0c49a0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fafdc0c6cb0 tx=0x7fafd801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe4172640 1 -- 172.21.15.57:0/2839321549 shutdown_connections 2023-10-30T14:44:22.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe4172640 1 --2- 172.21.15.57:0/2839321549 >> v2:172.21.15.88:3300/0 conn(0x7fafdc0cb8b0 0x7fafdc06b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe4172640 1 --2- 172.21.15.57:0/2839321549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafdc151840 0x7fafdc0c49a0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe4172640 1 -- 172.21.15.57:0/2839321549 >> 172.21.15.57:0/2839321549 conn(0x7fafdc0cac70 msgr2=0x7fafdc0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:22.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe4172640 1 -- 172.21.15.57:0/2839321549 shutdown_connections 2023-10-30T14:44:22.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe4172640 1 -- 172.21.15.57:0/2839321549 wait complete. 2023-10-30T14:44:22.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe4172640 1 Processor -- start 2023-10-30T14:44:22.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe4172640 1 -- start start 2023-10-30T14:44:22.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe4172640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafdc0cb8b0 0x7fafdc145630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:22.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe4172640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafdc151840 0x7fafdc14a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:22.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe4172640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafdc151c20 0x7fafdc1ef350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:22.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe4172640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc14ad90 con 0x7fafdc151840 2023-10-30T14:44:22.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe4172640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc1ee320 con 0x7fafdc151c20 2023-10-30T14:44:22.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe4172640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc148a30 con 0x7fafdc0cb8b0 2023-10-30T14:44:22.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe170d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafdc151840 0x7fafdc14a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:22.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe170d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafdc151840 0x7fafdc14a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58042/0 (socket says 172.21.15.57:58042) 2023-10-30T14:44:22.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe170d640 1 -- 172.21.15.57:0/2849345171 learned_addr learned my addr 172.21.15.57:0/2849345171 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:22.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe270f640 1 --2- 172.21.15.57:0/2849345171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafdc151c20 0x7fafdc1ef350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:22.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe1f0e640 1 --2- 172.21.15.57:0/2849345171 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafdc0cb8b0 0x7fafdc145630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:22.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe170d640 1 -- 172.21.15.57:0/2849345171 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafdc0cb8b0 msgr2=0x7fafdc145630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:22.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe170d640 1 --2- 172.21.15.57:0/2849345171 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafdc0cb8b0 0x7fafdc145630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe170d640 1 -- 172.21.15.57:0/2849345171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafdc151c20 msgr2=0x7fafdc1ef350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:22.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe170d640 1 --2- 172.21.15.57:0/2849345171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafdc151c20 0x7fafdc1ef350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe170d640 1 -- 172.21.15.57:0/2849345171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd80094d0 con 0x7fafdc151840 2023-10-30T14:44:22.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.298+0000 7fafe170d640 1 --2- 172.21.15.57:0/2849345171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafdc151840 0x7fafdc14a210 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fafcc00c3c0 tx=0x7fafcc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:22.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.302+0000 7fafd27fc640 1 -- 172.21.15.57:0/2849345171 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc0052d0 con 0x7fafdc151840 2023-10-30T14:44:22.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.302+0000 7fafd27fc640 1 -- 172.21.15.57:0/2849345171 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafcc005430 con 0x7fafdc151840 2023-10-30T14:44:22.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.302+0000 7fafd27fc640 1 -- 172.21.15.57:0/2849345171 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc003840 con 0x7fafdc151840 2023-10-30T14:44:22.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.302+0000 7fafe4172640 1 -- 172.21.15.57:0/2849345171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafdc1ee590 con 0x7fafdc151840 2023-10-30T14:44:22.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.302+0000 7fafe4172640 1 -- 172.21.15.57:0/2849345171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafdc1eeb50 con 0x7fafdc151840 2023-10-30T14:44:22.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.302+0000 7fafe4172640 1 -- 172.21.15.57:0/2849345171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafdc14fa00 con 0x7fafdc151840 2023-10-30T14:44:22.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.302+0000 7fafd27fc640 1 -- 172.21.15.57:0/2849345171 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fafcc0039a0 con 0x7fafdc151840 2023-10-30T14:44:22.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.302+0000 7fafd27fc640 1 --2- 172.21.15.57:0/2849345171 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fafc00664a0 0x7fafc0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:22.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.302+0000 7fafd27fc640 1 -- 172.21.15.57:0/2849345171 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7fafcc083900 con 0x7fafdc151840 2023-10-30T14:44:22.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.302+0000 7fafe1f0e640 1 --2- 172.21.15.57:0/2849345171 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fafc00664a0 0x7fafc0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:22.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.306+0000 7fafe1f0e640 1 --2- 172.21.15.57:0/2849345171 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fafc00664a0 0x7fafc0068960 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fafd80b2000 tx=0x7fafd8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:22.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.306+0000 7fafd27fc640 1 -- 172.21.15.57:0/2849345171 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fafcc050840 con 0x7fafdc151840 2023-10-30T14:44:22.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.502+0000 7fafe4172640 1 -- 172.21.15.57:0/2849345171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafdc04e170 con 0x7fafdc151840 2023-10-30T14:44:22.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.502+0000 7fafd27fc640 1 -- 172.21.15.57:0/2849345171 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5716 (secure 0 0 0) 0x7fafcc053e60 con 0x7fafdc151840 2023-10-30T14:44:22.503 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:22.504 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:44:20.161122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:22.504 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 164 2023-10-30T14:44:22.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.502+0000 7fafe4172640 1 -- 172.21.15.57:0/2849345171 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fafc00664a0 msgr2=0x7fafc0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:22.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.502+0000 7fafe4172640 1 --2- 172.21.15.57:0/2849345171 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fafc00664a0 0x7fafc0068960 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fafd80b2000 tx=0x7fafd8009040 comp rx=0 tx=0).stop 2023-10-30T14:44:22.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.502+0000 7fafe4172640 1 -- 172.21.15.57:0/2849345171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafdc151840 msgr2=0x7fafdc14a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:22.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.502+0000 7fafe4172640 1 --2- 172.21.15.57:0/2849345171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafdc151840 0x7fafdc14a210 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fafcc00c3c0 tx=0x7fafcc00c880 comp rx=0 tx=0).stop 2023-10-30T14:44:22.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.502+0000 7fafe4172640 1 -- 172.21.15.57:0/2849345171 shutdown_connections 2023-10-30T14:44:22.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.502+0000 7fafe4172640 1 --2- 172.21.15.57:0/2849345171 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fafc00664a0 0x7fafc0068960 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.502+0000 7fafe4172640 1 --2- 172.21.15.57:0/2849345171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafdc151c20 0x7fafdc1ef350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.502+0000 7fafe4172640 1 --2- 172.21.15.57:0/2849345171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafdc151840 0x7fafdc14a210 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.502+0000 7fafe4172640 1 --2- 172.21.15.57:0/2849345171 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafdc0cb8b0 0x7fafdc145630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.502+0000 7fafe4172640 1 -- 172.21.15.57:0/2849345171 >> 172.21.15.57:0/2849345171 conn(0x7fafdc0cac70 msgr2=0x7fafdc072000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.502+0000 7fafe4172640 1 -- 172.21.15.57:0/2849345171 shutdown_connections 2023-10-30T14:44:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.502+0000 7fafe4172640 1 -- 172.21.15.57:0/2849345171 wait complete. 2023-10-30T14:44:22.524 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-30T14:44:22.525 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:44:22.525 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:22.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.666+0000 7f189593e640 1 Processor -- start 2023-10-30T14:44:22.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.666+0000 7f189593e640 1 -- start start 2023-10-30T14:44:22.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.666+0000 7f189593e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1890150830 0x7f189006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:22.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.670+0000 7f189593e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f18900cb7d0 0x7f18900c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:22.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.670+0000 7f189593e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18900c2310 con 0x7f18900c3e40 2023-10-30T14:44:22.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.670+0000 7f189593e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1890156710 con 0x7f18900cb7d0 2023-10-30T14:44:22.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.670+0000 7f189593e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18900c4360 con 0x7f1890150830 2023-10-30T14:44:22.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.670+0000 7f1887fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f18900cb7d0 0x7f18900c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:22.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.670+0000 7f188f7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f18900c3e40 0x7f1890070520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:33494/0 (socket says 172.21.15.57:33494) 2023-10-30T14:44:22.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.670+0000 7f188f7fe640 1 -- 172.21.15.57:0/484518002 learned_addr learned my addr 172.21.15.57:0/484518002 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:44:22.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.670+0000 7f188effd640 1 --2- 172.21.15.57:0/484518002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1890150830 0x7f189006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:22.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.670+0000 7f1887fff640 1 -- 172.21.15.57:0/484518002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1890150830 msgr2=0x7f189006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:22.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.670+0000 7f1887fff640 1 --2- 172.21.15.57:0/484518002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1890150830 0x7f189006b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.670+0000 7f1887fff640 1 -- 172.21.15.57:0/484518002 >> v1:172.21.15.57:6789/0 conn(0x7f18900c3e40 legacy=0x7f1890070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:22.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.670+0000 7f188e7fc640 1 -- 172.21.15.57:0/484518002 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772767776 0 0) 0x7f18900c2310 con 0x7f18900c3e40 2023-10-30T14:44:22.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.670+0000 7f1887fff640 1 -- 172.21.15.57:0/484518002 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18901569b0 con 0x7f18900cb7d0 2023-10-30T14:44:22.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.670+0000 7f188effd640 1 --2- 172.21.15.57:0/484518002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1890150830 0x7f189006b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:44:22.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.670+0000 7f1887fff640 1 --2- 172.21.15.57:0/484518002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18900cb7d0 0x7f18900c3900 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f18780097c0 tx=0x7f187801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52486092d25a087 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:22.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.670+0000 7f188e7fc640 1 -- 172.21.15.57:0/484518002 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18780a8970 con 0x7f18900cb7d0 2023-10-30T14:44:22.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.670+0000 7f188e7fc640 1 -- 172.21.15.57:0/484518002 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18780a8ad0 con 0x7f18900cb7d0 2023-10-30T14:44:22.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f188e7fc640 1 -- 172.21.15.57:0/484518002 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18780a8e60 con 0x7f18900cb7d0 2023-10-30T14:44:22.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f189593e640 1 -- 172.21.15.57:0/484518002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18900cb7d0 msgr2=0x7f18900c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:22.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f189593e640 1 --2- 172.21.15.57:0/484518002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18900cb7d0 0x7f18900c3900 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f18780097c0 tx=0x7f187801ec70 comp rx=0 tx=0).stop 2023-10-30T14:44:22.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f189593e640 1 -- 172.21.15.57:0/484518002 shutdown_connections 2023-10-30T14:44:22.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f189593e640 1 --2- 172.21.15.57:0/484518002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18900cb7d0 0x7f18900c3900 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f189593e640 1 --2- 172.21.15.57:0/484518002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1890150830 0x7f189006b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f189593e640 1 -- 172.21.15.57:0/484518002 >> 172.21.15.57:0/484518002 conn(0x7f18900cac70 msgr2=0x7f18900cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:22.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f189593e640 1 -- 172.21.15.57:0/484518002 shutdown_connections 2023-10-30T14:44:22.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f189593e640 1 -- 172.21.15.57:0/484518002 wait complete. 2023-10-30T14:44:22.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f189593e640 1 Processor -- start 2023-10-30T14:44:22.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f189593e640 1 -- start start 2023-10-30T14:44:22.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f189593e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18900c3e40 0x7f189014c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:22.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f189593e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18900cb7d0 0x7f18901eea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:22.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f189593e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1890150830 0x7f1890154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:22.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f189593e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189015a3f0 con 0x7f1890150830 2023-10-30T14:44:22.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f189593e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18901561d0 con 0x7f18900c3e40 2023-10-30T14:44:22.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f189593e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1890156440 con 0x7f18900cb7d0 2023-10-30T14:44:22.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f188f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1890150830 0x7f1890154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:22.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f188f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1890150830 0x7f1890154e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58056/0 (socket says 172.21.15.57:58056) 2023-10-30T14:44:22.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f188f7fe640 1 -- 172.21.15.57:0/2615369969 learned_addr learned my addr 172.21.15.57:0/2615369969 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:22.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f1887fff640 1 --2- 172.21.15.57:0/2615369969 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18900cb7d0 0x7f18901eea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:22.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f188effd640 1 --2- 172.21.15.57:0/2615369969 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18900c3e40 0x7f189014c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:22.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f188f7fe640 1 -- 172.21.15.57:0/2615369969 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18900cb7d0 msgr2=0x7f18901eea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:22.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f188f7fe640 1 --2- 172.21.15.57:0/2615369969 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18900cb7d0 0x7f18901eea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f188f7fe640 1 -- 172.21.15.57:0/2615369969 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18900c3e40 msgr2=0x7f189014c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:22.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f188f7fe640 1 --2- 172.21.15.57:0/2615369969 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18900c3e40 0x7f189014c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f188f7fe640 1 -- 172.21.15.57:0/2615369969 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1878009420 con 0x7f1890150830 2023-10-30T14:44:22.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.674+0000 7f1887fff640 1 --2- 172.21.15.57:0/2615369969 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18900cb7d0 0x7f18901eea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:44:22.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.678+0000 7f188f7fe640 1 --2- 172.21.15.57:0/2615369969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1890150830 0x7f1890154e10 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f188800c7e0 tx=0x7f188800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:22.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.678+0000 7f188cff9640 1 -- 172.21.15.57:0/2615369969 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188800a640 con 0x7f1890150830 2023-10-30T14:44:22.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.678+0000 7f188cff9640 1 -- 172.21.15.57:0/2615369969 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f188800a7a0 con 0x7f1890150830 2023-10-30T14:44:22.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.678+0000 7f188cff9640 1 -- 172.21.15.57:0/2615369969 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188800ab50 con 0x7f1890150830 2023-10-30T14:44:22.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.678+0000 7f189593e640 1 -- 172.21.15.57:0/2615369969 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18900c60c0 con 0x7f1890150830 2023-10-30T14:44:22.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.678+0000 7f189593e640 1 -- 172.21.15.57:0/2615369969 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18901597c0 con 0x7f1890150830 2023-10-30T14:44:22.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.678+0000 7f189593e640 1 -- 172.21.15.57:0/2615369969 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f189015a560 con 0x7f1890150830 2023-10-30T14:44:22.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.678+0000 7f188cff9640 1 -- 172.21.15.57:0/2615369969 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f188800acb0 con 0x7f1890150830 2023-10-30T14:44:22.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.678+0000 7f188cff9640 1 --2- 172.21.15.57:0/2615369969 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f18700664a0 0x7f1870068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:22.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.678+0000 7f188cff9640 1 -- 172.21.15.57:0/2615369969 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f1888083a20 con 0x7f1890150830 2023-10-30T14:44:22.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.678+0000 7f188effd640 1 --2- 172.21.15.57:0/2615369969 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f18700664a0 0x7f1870068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:22.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.682+0000 7f188effd640 1 --2- 172.21.15.57:0/2615369969 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f18700664a0 0x7f1870068960 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f188000ba10 tx=0x7f1880008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:22.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.682+0000 7f188cff9640 1 -- 172.21.15.57:0/2615369969 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1888050840 con 0x7f1890150830 2023-10-30T14:44:22.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.878+0000 7f189593e640 1 -- 172.21.15.57:0/2615369969 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f189004e170 con 0x7f1890150830 2023-10-30T14:44:22.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.878+0000 7f188cff9640 1 -- 172.21.15.57:0/2615369969 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5716 (secure 0 0 0) 0x7f1888018090 con 0x7f1890150830 2023-10-30T14:44:22.881 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:22.881 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:44:20.161122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:22.881 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 164 2023-10-30T14:44:22.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.882+0000 7f189593e640 1 -- 172.21.15.57:0/2615369969 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f18700664a0 msgr2=0x7f1870068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:22.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.882+0000 7f189593e640 1 --2- 172.21.15.57:0/2615369969 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f18700664a0 0x7f1870068960 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f188000ba10 tx=0x7f1880008040 comp rx=0 tx=0).stop 2023-10-30T14:44:22.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.882+0000 7f189593e640 1 -- 172.21.15.57:0/2615369969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1890150830 msgr2=0x7f1890154e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:22.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.882+0000 7f189593e640 1 --2- 172.21.15.57:0/2615369969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1890150830 0x7f1890154e10 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f188800c7e0 tx=0x7f188800cca0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.882+0000 7f189593e640 1 -- 172.21.15.57:0/2615369969 shutdown_connections 2023-10-30T14:44:22.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.882+0000 7f189593e640 1 --2- 172.21.15.57:0/2615369969 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f18700664a0 0x7f1870068960 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.882+0000 7f189593e640 1 --2- 172.21.15.57:0/2615369969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1890150830 0x7f1890154e10 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.882+0000 7f189593e640 1 --2- 172.21.15.57:0/2615369969 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18900cb7d0 0x7f18901eea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.882+0000 7f189593e640 1 --2- 172.21.15.57:0/2615369969 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18900c3e40 0x7f189014c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:22.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.882+0000 7f189593e640 1 -- 172.21.15.57:0/2615369969 >> 172.21.15.57:0/2615369969 conn(0x7f18900cac70 msgr2=0x7f189014eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:22.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.882+0000 7f189593e640 1 -- 172.21.15.57:0/2615369969 shutdown_connections 2023-10-30T14:44:22.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:22.882+0000 7f189593e640 1 -- 172.21.15.57:0/2615369969 wait complete. 2023-10-30T14:44:22.902 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 164, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 164, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 44 }, "gid_7536": { "addr": "172.21.15.57:6837/816502245", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 816502245, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 816502245, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7536, "incarnation": 164, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:44:20.161122+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7536, "mds_1": 6747 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.57:6834/697507447", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 697507447, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 697507447, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 156, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7377, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:44:24.903 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:25.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.054+0000 7fadabc9c640 1 Processor -- start 2023-10-30T14:44:25.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.054+0000 7fadabc9c640 1 -- start start 2023-10-30T14:44:25.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fadabc9c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fada406bb40 0x7fada4157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:25.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fadabc9c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada4071e00 con 0x7fada406b760 2023-10-30T14:44:25.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fadabc9c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada4070020 con 0x7fada406c420 2023-10-30T14:44:25.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fadabc9c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada4070f10 con 0x7fada406bb40 2023-10-30T14:44:25.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada9a38640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fada406b760 0x7fada415e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:39282/0 (socket says 172.21.15.57:39282) 2023-10-30T14:44:25.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada9a38640 1 -- 172.21.15.57:0/2114240276 learned_addr learned my addr 172.21.15.57:0/2114240276 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:44:25.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada9237640 1 --2- 172.21.15.57:0/2114240276 >> v2:172.21.15.88:3300/0 conn(0x7fada406bb40 0x7fada4157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:25.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada8a36640 1 -- 172.21.15.57:0/2114240276 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403627924 0 0) 0x7fada4071e00 con 0x7fada406b760 2023-10-30T14:44:25.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada8a36640 1 -- 172.21.15.57:0/2114240276 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad90003530 con 0x7fada406b760 2023-10-30T14:44:25.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada8a36640 1 -- 172.21.15.57:0/2114240276 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3187477930 0 0) 0x7fad90003530 con 0x7fada406b760 2023-10-30T14:44:25.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada8a36640 1 -- 172.21.15.57:0/2114240276 >> v2:172.21.15.88:3300/0 conn(0x7fada406bb40 msgr2=0x7fada4157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:25.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada8a36640 1 --2- 172.21.15.57:0/2114240276 >> v2:172.21.15.88:3300/0 conn(0x7fada406bb40 0x7fada4157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:25.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada8a36640 1 -- 172.21.15.57:0/2114240276 >> v1:172.21.15.88:6789/0 conn(0x7fada406c420 legacy=0x7fada40c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:25.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada8a36640 1 -- 172.21.15.57:0/2114240276 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada40711f0 con 0x7fada406b760 2023-10-30T14:44:25.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada8a36640 1 -- 172.21.15.57:0/2114240276 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fada0001680 con 0x7fada406b760 2023-10-30T14:44:25.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada8a36640 1 -- 172.21.15.57:0/2114240276 >> v1:172.21.15.57:6789/0 conn(0x7fada406b760 legacy=0x7fada415e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:25.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada8a36640 1 --2- 172.21.15.57:0/2114240276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fad90003980 0x7fad90003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:25.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada8a36640 1 --2- 172.21.15.57:0/2114240276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad900047d0 0x7fad90004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:25.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada8a36640 1 --2- 172.21.15.57:0/2114240276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad90005190 0x7fad9000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:25.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada8a36640 1 -- 172.21.15.57:0/2114240276 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada4070f10 con 0x7fad900047d0 2023-10-30T14:44:25.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada8a36640 1 -- 172.21.15.57:0/2114240276 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9000c030 con 0x7fad90005190 2023-10-30T14:44:25.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada8a36640 1 -- 172.21.15.57:0/2114240276 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad9000c1f0 con 0x7fad90003980 2023-10-30T14:44:25.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada9237640 1 --2- 172.21.15.57:0/2114240276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad900047d0 0x7fad90004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:25.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fadaa239640 1 --2- 172.21.15.57:0/2114240276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad90005190 0x7fad9000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:25.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada9a38640 1 --2- 172.21.15.57:0/2114240276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fad90003980 0x7fad90003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:25.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada9237640 1 -- 172.21.15.57:0/2114240276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fad90003980 msgr2=0x7fad90003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:25.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada9237640 1 --2- 172.21.15.57:0/2114240276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fad90003980 0x7fad90003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:25.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada9237640 1 -- 172.21.15.57:0/2114240276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad90005190 msgr2=0x7fad9000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:25.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada9237640 1 --2- 172.21.15.57:0/2114240276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad90005190 0x7fad9000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:25.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fada9237640 1 -- 172.21.15.57:0/2114240276 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad90003530 con 0x7fad900047d0 2023-10-30T14:44:25.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.058+0000 7fadaa239640 1 -- 172.21.15.57:0/2114240276 reap_dead start 2023-10-30T14:44:25.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.062+0000 7fada9237640 1 --2- 172.21.15.57:0/2114240276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad900047d0 0x7fad90004c50 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fad94009ab0 tx=0x7fad9401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d17c9b6751a07ee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:25.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.062+0000 7fada8a36640 1 -- 172.21.15.57:0/2114240276 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad940a5510 con 0x7fad900047d0 2023-10-30T14:44:25.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.062+0000 7fada8a36640 1 -- 172.21.15.57:0/2114240276 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad940a5670 con 0x7fad900047d0 2023-10-30T14:44:25.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.062+0000 7fada8a36640 1 -- 172.21.15.57:0/2114240276 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad940a5a20 con 0x7fad900047d0 2023-10-30T14:44:25.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.062+0000 7fadabc9c640 1 -- 172.21.15.57:0/2114240276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad900047d0 msgr2=0x7fad90004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:25.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.062+0000 7fadabc9c640 1 --2- 172.21.15.57:0/2114240276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad900047d0 0x7fad90004c50 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fad94009ab0 tx=0x7fad9401ca30 comp rx=0 tx=0).stop 2023-10-30T14:44:25.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.062+0000 7fadabc9c640 1 -- 172.21.15.57:0/2114240276 shutdown_connections 2023-10-30T14:44:25.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.062+0000 7fadabc9c640 1 --2- 172.21.15.57:0/2114240276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad900047d0 0x7fad90004c50 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:25.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.062+0000 7fadabc9c640 1 -- 172.21.15.57:0/2114240276 >> 172.21.15.57:0/2114240276 conn(0x7fada40cac70 msgr2=0x7fada40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:25.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.062+0000 7fadabc9c640 1 -- 172.21.15.57:0/2114240276 shutdown_connections 2023-10-30T14:44:25.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.062+0000 7fadabc9c640 1 -- 172.21.15.57:0/2114240276 wait complete. 2023-10-30T14:44:25.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fadabc9c640 1 Processor -- start 2023-10-30T14:44:25.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fadabc9c640 1 -- start start 2023-10-30T14:44:25.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fadabc9c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad900047d0 0x7fada41594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:25.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fadabc9c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fada4161d30 0x7fada4152c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:25.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fadabc9c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fada4150e20 0x7fada41512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:25.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fadabc9c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada41517e0 con 0x7fada4161d30 2023-10-30T14:44:25.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fadabc9c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada4159a00 con 0x7fad900047d0 2023-10-30T14:44:25.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fadabc9c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada41591e0 con 0x7fada4150e20 2023-10-30T14:44:25.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fada9237640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fada4161d30 0x7fada4152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:25.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fada9237640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fada4161d30 0x7fada4152c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47468/0 (socket says 172.21.15.57:47468) 2023-10-30T14:44:25.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fada9237640 1 -- 172.21.15.57:0/882753312 learned_addr learned my addr 172.21.15.57:0/882753312 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:25.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fada9a38640 1 --2- 172.21.15.57:0/882753312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad900047d0 0x7fada41594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:25.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fadaa239640 1 --2- 172.21.15.57:0/882753312 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fada4150e20 0x7fada41512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:25.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fada9237640 1 -- 172.21.15.57:0/882753312 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fada4150e20 msgr2=0x7fada41512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:25.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fada9237640 1 --2- 172.21.15.57:0/882753312 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fada4150e20 0x7fada41512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:25.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fada9237640 1 -- 172.21.15.57:0/882753312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad900047d0 msgr2=0x7fada41594c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:25.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fada9237640 1 --2- 172.21.15.57:0/882753312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad900047d0 0x7fada41594c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:25.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fada9237640 1 -- 172.21.15.57:0/882753312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada415e470 con 0x7fada4161d30 2023-10-30T14:44:25.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fada9237640 1 --2- 172.21.15.57:0/882753312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fada4161d30 0x7fada4152c30 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fad94009ab0 tx=0x7fad940197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:25.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fad9a7fc640 1 -- 172.21.15.57:0/882753312 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad940b6870 con 0x7fada4161d30 2023-10-30T14:44:25.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fad9a7fc640 1 -- 172.21.15.57:0/882753312 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad940b69f0 con 0x7fada4161d30 2023-10-30T14:44:25.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fad9a7fc640 1 -- 172.21.15.57:0/882753312 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad940b6da0 con 0x7fada4161d30 2023-10-30T14:44:25.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fadabc9c640 1 -- 172.21.15.57:0/882753312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada415e750 con 0x7fada4161d30 2023-10-30T14:44:25.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fadabc9c640 1 -- 172.21.15.57:0/882753312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada41f2c60 con 0x7fada4161d30 2023-10-30T14:44:25.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.066+0000 7fadabc9c640 1 -- 172.21.15.57:0/882753312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada404e170 con 0x7fada4161d30 2023-10-30T14:44:25.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.070+0000 7fad9a7fc640 1 -- 172.21.15.57:0/882753312 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fad940b4070 con 0x7fada4161d30 2023-10-30T14:44:25.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.070+0000 7fad9a7fc640 1 --2- 172.21.15.57:0/882753312 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fad880664d0 0x7fad88068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:25.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.070+0000 7fad9a7fc640 1 -- 172.21.15.57:0/882753312 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7fad9411e6f0 con 0x7fada4161d30 2023-10-30T14:44:25.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.070+0000 7fada9a38640 1 --2- 172.21.15.57:0/882753312 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fad880664d0 0x7fad88068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:25.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.070+0000 7fada9a38640 1 --2- 172.21.15.57:0/882753312 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fad880664d0 0x7fad88068990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fada0001ec0 tx=0x7fada0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:25.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.074+0000 7fad9a7fc640 1 -- 172.21.15.57:0/882753312 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fad940eb5a0 con 0x7fada4161d30 2023-10-30T14:44:25.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.298+0000 7fadabc9c640 1 -- 172.21.15.57:0/882753312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fada4072a80 con 0x7fada4161d30 2023-10-30T14:44:25.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.298+0000 7fad9a7fc640 1 -- 172.21.15.57:0/882753312 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5716 (secure 0 0 0) 0x7fad940eebc0 con 0x7fada4161d30 2023-10-30T14:44:25.302 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:25.302 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:44:20.161122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:25.303 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 164 2023-10-30T14:44:25.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.302+0000 7fadabc9c640 1 -- 172.21.15.57:0/882753312 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fad880664d0 msgr2=0x7fad88068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:25.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.302+0000 7fadabc9c640 1 --2- 172.21.15.57:0/882753312 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fad880664d0 0x7fad88068990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fada0001ec0 tx=0x7fada0009040 comp rx=0 tx=0).stop 2023-10-30T14:44:25.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.302+0000 7fadabc9c640 1 -- 172.21.15.57:0/882753312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fada4161d30 msgr2=0x7fada4152c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:25.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.302+0000 7fadabc9c640 1 --2- 172.21.15.57:0/882753312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fada4161d30 0x7fada4152c30 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fad94009ab0 tx=0x7fad940197d0 comp rx=0 tx=0).stop 2023-10-30T14:44:25.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.302+0000 7fadabc9c640 1 -- 172.21.15.57:0/882753312 shutdown_connections 2023-10-30T14:44:25.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.302+0000 7fadabc9c640 1 --2- 172.21.15.57:0/882753312 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fad880664d0 0x7fad88068990 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:25.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.302+0000 7fadabc9c640 1 --2- 172.21.15.57:0/882753312 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fada4150e20 0x7fada41512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:25.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.302+0000 7fadabc9c640 1 --2- 172.21.15.57:0/882753312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fada4161d30 0x7fada4152c30 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:25.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.302+0000 7fadabc9c640 1 --2- 172.21.15.57:0/882753312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad900047d0 0x7fada41594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:25.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.302+0000 7fadabc9c640 1 -- 172.21.15.57:0/882753312 >> 172.21.15.57:0/882753312 conn(0x7fada40cac70 msgr2=0x7fada40c6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:25.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.302+0000 7fadabc9c640 1 -- 172.21.15.57:0/882753312 shutdown_connections 2023-10-30T14:44:25.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:25.302+0000 7fadabc9c640 1 -- 172.21.15.57:0/882753312 wait complete. 2023-10-30T14:44:25.324 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:44:27.325 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:27.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.474+0000 7f2a056c9640 1 Processor -- start 2023-10-30T14:44:27.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.474+0000 7f2a056c9640 1 -- start start 2023-10-30T14:44:27.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.478+0000 7f2a056c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2a00153c80 0x7f2a0015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:27.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.478+0000 7f2a056c9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2a000c4a10 0x7f2a000c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:27.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.478+0000 7f2a056c9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a00159f20 con 0x7f2a0015a3b0 2023-10-30T14:44:27.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.478+0000 7f2a056c9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a0015dec0 con 0x7f2a00153c80 2023-10-30T14:44:27.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.478+0000 7f2a056c9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a001599a0 con 0x7f2a000c4a10 2023-10-30T14:44:27.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.478+0000 7f29feffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2a00153c80 0x7f2a0015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:27.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.478+0000 7f29fe7fc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2a000c4a10 0x7f2a000c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:27.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.478+0000 7f29feffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2a00153c80 0x7f2a0015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:38968/0 (socket says 172.21.15.57:38968) 2023-10-30T14:44:27.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.478+0000 7f29feffd640 1 -- 172.21.15.57:0/4135216211 learned_addr learned my addr 172.21.15.57:0/4135216211 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:27.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.478+0000 7f29ff7fe640 1 --1- 172.21.15.57:0/4135216211 >> v1:172.21.15.88:6789/0 conn(0x7f2a0015a3b0 0x7f2a000ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:59724/0 (socket says 172.21.15.57:59724) 2023-10-30T14:44:27.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.478+0000 7f29feffd640 1 -- 172.21.15.57:0/4135216211 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2a000c4a10 msgr2=0x7f2a000c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:27.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.478+0000 7f29feffd640 1 --2- 172.21.15.57:0/4135216211 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2a000c4a10 0x7f2a000c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:27.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.478+0000 7f29feffd640 1 -- 172.21.15.57:0/4135216211 >> v1:172.21.15.88:6789/0 conn(0x7f2a0015a3b0 legacy=0x7f2a000ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:27.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.478+0000 7f29feffd640 1 -- 172.21.15.57:0/4135216211 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a000c47e0 con 0x7f2a00153c80 2023-10-30T14:44:27.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.482+0000 7f29feffd640 1 --2- 172.21.15.57:0/4135216211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a00153c80 0x7f2a0015cf90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f29f0009870 tx=0x7f29f001ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd34bf42cf21f559 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:27.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.482+0000 7f29fdffb640 1 -- 172.21.15.57:0/4135216211 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f00a8a30 con 0x7f2a00153c80 2023-10-30T14:44:27.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.482+0000 7f29fdffb640 1 -- 172.21.15.57:0/4135216211 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f00a8b90 con 0x7f2a00153c80 2023-10-30T14:44:27.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.482+0000 7f29fdffb640 1 -- 172.21.15.57:0/4135216211 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f00b0440 con 0x7f2a00153c80 2023-10-30T14:44:27.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.482+0000 7f2a056c9640 1 -- 172.21.15.57:0/4135216211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a00153c80 msgr2=0x7f2a0015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:27.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.482+0000 7f2a056c9640 1 --2- 172.21.15.57:0/4135216211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a00153c80 0x7f2a0015cf90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f29f0009870 tx=0x7f29f001ed30 comp rx=0 tx=0).stop 2023-10-30T14:44:27.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.482+0000 7f2a056c9640 1 -- 172.21.15.57:0/4135216211 shutdown_connections 2023-10-30T14:44:27.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.482+0000 7f2a056c9640 1 --2- 172.21.15.57:0/4135216211 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2a000c4a10 0x7f2a000c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:27.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.482+0000 7f2a056c9640 1 --2- 172.21.15.57:0/4135216211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a00153c80 0x7f2a0015cf90 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:27.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.482+0000 7f2a056c9640 1 -- 172.21.15.57:0/4135216211 >> 172.21.15.57:0/4135216211 conn(0x7f2a000715c0 msgr2=0x7f2a000719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:27.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.482+0000 7f2a056c9640 1 -- 172.21.15.57:0/4135216211 shutdown_connections 2023-10-30T14:44:27.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.482+0000 7f2a056c9640 1 -- 172.21.15.57:0/4135216211 wait complete. 2023-10-30T14:44:27.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f2a056c9640 1 Processor -- start 2023-10-30T14:44:27.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f2a056c9640 1 -- start start 2023-10-30T14:44:27.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f2a056c9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2a000c4a10 0x7f2a0014ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:27.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f2a056c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a00153c80 0x7f2a000b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:27.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f2a056c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a0015a3b0 0x7f2a000ce450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:27.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f2a056c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a000ce990 con 0x7f2a0015a3b0 2023-10-30T14:44:27.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f2a056c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a000b26b0 con 0x7f2a00153c80 2023-10-30T14:44:27.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f2a056c9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a000cdd20 con 0x7f2a000c4a10 2023-10-30T14:44:27.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f29ff7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a0015a3b0 0x7f2a000ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:27.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f29ff7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a0015a3b0 0x7f2a000ce450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47482/0 (socket says 172.21.15.57:47482) 2023-10-30T14:44:27.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f29ff7fe640 1 -- 172.21.15.57:0/840773438 learned_addr learned my addr 172.21.15.57:0/840773438 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:27.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f29fe7fc640 1 --2- 172.21.15.57:0/840773438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a00153c80 0x7f2a000b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:27.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f29feffd640 1 --2- 172.21.15.57:0/840773438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2a000c4a10 0x7f2a0014ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:27.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f29ff7fe640 1 -- 172.21.15.57:0/840773438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2a000c4a10 msgr2=0x7f2a0014ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:27.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f29ff7fe640 1 --2- 172.21.15.57:0/840773438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2a000c4a10 0x7f2a0014ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:27.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f29ff7fe640 1 -- 172.21.15.57:0/840773438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a00153c80 msgr2=0x7f2a000b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:27.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f29ff7fe640 1 --2- 172.21.15.57:0/840773438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a00153c80 0x7f2a000b2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:27.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f29ff7fe640 1 -- 172.21.15.57:0/840773438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f00094d0 con 0x7f2a0015a3b0 2023-10-30T14:44:27.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f29ff7fe640 1 --2- 172.21.15.57:0/840773438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a0015a3b0 0x7f2a000ce450 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f29f800c700 tx=0x7f29f800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:27.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f29e77fe640 1 -- 172.21.15.57:0/840773438 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f800ce10 con 0x7f2a0015a3b0 2023-10-30T14:44:27.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f29e77fe640 1 -- 172.21.15.57:0/840773438 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f800a5e0 con 0x7f2a0015a3b0 2023-10-30T14:44:27.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f2a056c9640 1 -- 172.21.15.57:0/840773438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a000cf400 con 0x7f2a0015a3b0 2023-10-30T14:44:27.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f29e77fe640 1 -- 172.21.15.57:0/840773438 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f800a950 con 0x7f2a0015a3b0 2023-10-30T14:44:27.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.486+0000 7f2a056c9640 1 -- 172.21.15.57:0/840773438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a000cf680 con 0x7f2a0015a3b0 2023-10-30T14:44:27.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.490+0000 7f2a056c9640 1 -- 172.21.15.57:0/840773438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0004e170 con 0x7f2a0015a3b0 2023-10-30T14:44:27.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.490+0000 7f29e77fe640 1 -- 172.21.15.57:0/840773438 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f29f800aab0 con 0x7f2a0015a3b0 2023-10-30T14:44:27.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.490+0000 7f29e77fe640 1 --2- 172.21.15.57:0/840773438 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29ec0664a0 0x7f29ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:27.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.490+0000 7f29e77fe640 1 -- 172.21.15.57:0/840773438 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f29f8083bb0 con 0x7f2a0015a3b0 2023-10-30T14:44:27.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.490+0000 7f29feffd640 1 --2- 172.21.15.57:0/840773438 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29ec0664a0 0x7f29ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:27.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.490+0000 7f29feffd640 1 --2- 172.21.15.57:0/840773438 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29ec0664a0 0x7f29ec068960 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f29f00b2000 tx=0x7f29f0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:27.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.494+0000 7f29e77fe640 1 -- 172.21.15.57:0/840773438 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f29f80509d0 con 0x7f2a0015a3b0 2023-10-30T14:44:27.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.698+0000 7f2a056c9640 1 -- 172.21.15.57:0/840773438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a000cad40 con 0x7f2a0015a3b0 2023-10-30T14:44:27.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.698+0000 7f29e77fe640 1 -- 172.21.15.57:0/840773438 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5716 (secure 0 0 0) 0x7f29f8017020 con 0x7f2a0015a3b0 2023-10-30T14:44:27.701 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:27.701 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:44:20.161122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:27.701 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 164 2023-10-30T14:44:27.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.702+0000 7f2a056c9640 1 -- 172.21.15.57:0/840773438 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29ec0664a0 msgr2=0x7f29ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:27.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.702+0000 7f2a056c9640 1 --2- 172.21.15.57:0/840773438 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29ec0664a0 0x7f29ec068960 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f29f00b2000 tx=0x7f29f0009040 comp rx=0 tx=0).stop 2023-10-30T14:44:27.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.702+0000 7f2a056c9640 1 -- 172.21.15.57:0/840773438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a0015a3b0 msgr2=0x7f2a000ce450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:27.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.702+0000 7f2a056c9640 1 --2- 172.21.15.57:0/840773438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a0015a3b0 0x7f2a000ce450 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f29f800c700 tx=0x7f29f800cbc0 comp rx=0 tx=0).stop 2023-10-30T14:44:27.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.702+0000 7f2a056c9640 1 -- 172.21.15.57:0/840773438 shutdown_connections 2023-10-30T14:44:27.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.702+0000 7f2a056c9640 1 --2- 172.21.15.57:0/840773438 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29ec0664a0 0x7f29ec068960 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:27.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.702+0000 7f2a056c9640 1 --2- 172.21.15.57:0/840773438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a0015a3b0 0x7f2a000ce450 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:27.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.702+0000 7f2a056c9640 1 --2- 172.21.15.57:0/840773438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a00153c80 0x7f2a000b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:27.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.702+0000 7f2a056c9640 1 --2- 172.21.15.57:0/840773438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2a000c4a10 0x7f2a0014ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:27.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.702+0000 7f2a056c9640 1 -- 172.21.15.57:0/840773438 >> 172.21.15.57:0/840773438 conn(0x7f2a000715c0 msgr2=0x7f2a00152fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:27.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.702+0000 7f2a056c9640 1 -- 172.21.15.57:0/840773438 shutdown_connections 2023-10-30T14:44:27.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:27.702+0000 7f2a056c9640 1 -- 172.21.15.57:0/840773438 wait complete. 2023-10-30T14:44:27.723 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:44:29.724 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:29.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.870+0000 7fa516a9f640 1 Processor -- start 2023-10-30T14:44:29.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.870+0000 7fa516a9f640 1 -- start start 2023-10-30T14:44:29.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.870+0000 7fa516a9f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa51015cd70 0x7fa510069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:29.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.870+0000 7fa516a9f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa51015dcc0 con 0x7fa51015ac70 2023-10-30T14:44:29.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.870+0000 7fa516a9f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa510159d80 con 0x7fa5100c8eb0 2023-10-30T14:44:29.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.870+0000 7fa516a9f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa510159800 con 0x7fa51015cd70 2023-10-30T14:44:29.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.870+0000 7fa51529c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa51015ac70 0x7fa51006aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:39288/0 (socket says 172.21.15.57:39288) 2023-10-30T14:44:29.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.870+0000 7fa51529c640 1 -- 172.21.15.57:0/1554557907 learned_addr learned my addr 172.21.15.57:0/1554557907 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:44:29.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.870+0000 7fa51629e640 1 --2- 172.21.15.57:0/1554557907 >> v2:172.21.15.57:3300/0 conn(0x7fa51015cd70 0x7fa510069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:29.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.870+0000 7fa514a9b640 1 -- 172.21.15.57:0/1554557907 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263008044 0 0) 0x7fa500001120 con 0x7fa51015ac70 2023-10-30T14:44:29.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.870+0000 7fa514a9b640 1 -- 172.21.15.57:0/1554557907 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4fc003530 con 0x7fa51015ac70 2023-10-30T14:44:29.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.870+0000 7fa51629e640 1 -- 172.21.15.57:0/1554557907 >> v1:172.21.15.88:6789/0 conn(0x7fa5100c8eb0 legacy=0x7fa51015a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:29.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.870+0000 7fa51629e640 1 -- 172.21.15.57:0/1554557907 >> v1:172.21.15.57:6789/0 conn(0x7fa51015ac70 legacy=0x7fa51006aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:29.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.870+0000 7fa51629e640 1 -- 172.21.15.57:0/1554557907 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51015d1c0 con 0x7fa51015cd70 2023-10-30T14:44:29.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.874+0000 7fa51629e640 1 --2- 172.21.15.57:0/1554557907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa51015cd70 0x7fa510069e90 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fa508009d90 tx=0x7fa50801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2a1d194d1094d07 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:29.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.874+0000 7fa514a9b640 1 -- 172.21.15.57:0/1554557907 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5080af600 con 0x7fa51015cd70 2023-10-30T14:44:29.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.874+0000 7fa514a9b640 1 -- 172.21.15.57:0/1554557907 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5080ab050 con 0x7fa51015cd70 2023-10-30T14:44:29.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.874+0000 7fa514a9b640 1 -- 172.21.15.57:0/1554557907 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5080af980 con 0x7fa51015cd70 2023-10-30T14:44:29.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa516a9f640 1 -- 172.21.15.57:0/1554557907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa51015cd70 msgr2=0x7fa510069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:29.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa516a9f640 1 --2- 172.21.15.57:0/1554557907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa51015cd70 0x7fa510069e90 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fa508009d90 tx=0x7fa50801f130 comp rx=0 tx=0).stop 2023-10-30T14:44:29.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa516a9f640 1 -- 172.21.15.57:0/1554557907 shutdown_connections 2023-10-30T14:44:29.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa516a9f640 1 --2- 172.21.15.57:0/1554557907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa51015cd70 0x7fa510069e90 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:29.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa516a9f640 1 -- 172.21.15.57:0/1554557907 >> 172.21.15.57:0/1554557907 conn(0x7fa5100cef00 msgr2=0x7fa5100cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:29.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa516a9f640 1 -- 172.21.15.57:0/1554557907 shutdown_connections 2023-10-30T14:44:29.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa516a9f640 1 -- 172.21.15.57:0/1554557907 wait complete. 2023-10-30T14:44:29.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa516a9f640 1 Processor -- start 2023-10-30T14:44:29.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa516a9f640 1 -- start start 2023-10-30T14:44:29.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa516a9f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5100c8eb0 0x7fa5100c5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:29.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa516a9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa51015ac70 0x7fa5101500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:29.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa516a9f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa51015cd70 0x7fa51006ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:29.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa516a9f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa510150620 con 0x7fa51015cd70 2023-10-30T14:44:29.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa516a9f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5101f2520 con 0x7fa51015ac70 2023-10-30T14:44:29.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa516a9f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5101f30a0 con 0x7fa5100c8eb0 2023-10-30T14:44:29.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa51629e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa51015cd70 0x7fa51006ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:29.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa51529c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa51015ac70 0x7fa5101500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:29.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa51629e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa51015cd70 0x7fa51006ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47500/0 (socket says 172.21.15.57:47500) 2023-10-30T14:44:29.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa51629e640 1 -- 172.21.15.57:0/1360470143 learned_addr learned my addr 172.21.15.57:0/1360470143 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:29.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.878+0000 7fa515a9d640 1 --2- 172.21.15.57:0/1360470143 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5100c8eb0 0x7fa5100c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:29.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.882+0000 7fa51629e640 1 -- 172.21.15.57:0/1360470143 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5100c8eb0 msgr2=0x7fa5100c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:29.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.882+0000 7fa51629e640 1 --2- 172.21.15.57:0/1360470143 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5100c8eb0 0x7fa5100c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:29.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.882+0000 7fa51629e640 1 -- 172.21.15.57:0/1360470143 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa51015ac70 msgr2=0x7fa5101500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:29.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.882+0000 7fa51629e640 1 --2- 172.21.15.57:0/1360470143 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa51015ac70 0x7fa5101500e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:29.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.882+0000 7fa51629e640 1 -- 172.21.15.57:0/1360470143 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa508009a70 con 0x7fa51015cd70 2023-10-30T14:44:29.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.882+0000 7fa51629e640 1 --2- 172.21.15.57:0/1360470143 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa51015cd70 0x7fa51006ae30 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fa5080b2040 tx=0x7fa50801bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:29.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.882+0000 7fa5067fc640 1 -- 172.21.15.57:0/1360470143 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa508009450 con 0x7fa51015cd70 2023-10-30T14:44:29.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.882+0000 7fa5067fc640 1 -- 172.21.15.57:0/1360470143 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5080b9410 con 0x7fa51015cd70 2023-10-30T14:44:29.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.882+0000 7fa5067fc640 1 -- 172.21.15.57:0/1360470143 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5080b97c0 con 0x7fa51015cd70 2023-10-30T14:44:29.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.882+0000 7fa516a9f640 1 -- 172.21.15.57:0/1360470143 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa510150e20 con 0x7fa51015cd70 2023-10-30T14:44:29.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.882+0000 7fa516a9f640 1 -- 172.21.15.57:0/1360470143 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5101f2a50 con 0x7fa51015cd70 2023-10-30T14:44:29.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.882+0000 7fa516a9f640 1 -- 172.21.15.57:0/1360470143 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5101f3210 con 0x7fa51015cd70 2023-10-30T14:44:29.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.886+0000 7fa5067fc640 1 -- 172.21.15.57:0/1360470143 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa5080b6050 con 0x7fa51015cd70 2023-10-30T14:44:29.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.886+0000 7fa5067fc640 1 --2- 172.21.15.57:0/1360470143 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa4f40664a0 0x7fa4f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:29.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.886+0000 7fa5067fc640 1 -- 172.21.15.57:0/1360470143 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7fa508122780 con 0x7fa51015cd70 2023-10-30T14:44:29.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.886+0000 7fa515a9d640 1 --2- 172.21.15.57:0/1360470143 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa4f40664a0 0x7fa4f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:29.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.886+0000 7fa515a9d640 1 --2- 172.21.15.57:0/1360470143 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa4f40664a0 0x7fa4f4068960 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa50c007fd0 tx=0x7fa50c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:29.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:29.890+0000 7fa5067fc640 1 -- 172.21.15.57:0/1360470143 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa5080ef6b0 con 0x7fa51015cd70 2023-10-30T14:44:30.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:30.082+0000 7fa516a9f640 1 -- 172.21.15.57:0/1360470143 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa51004e170 con 0x7fa51015cd70 2023-10-30T14:44:30.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:30.082+0000 7fa5067fc640 1 -- 172.21.15.57:0/1360470143 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5716 (secure 0 0 0) 0x7fa5080f2cd0 con 0x7fa51015cd70 2023-10-30T14:44:30.085 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:30.085 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:44:20.161122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:30.086 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 164 2023-10-30T14:44:30.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:30.086+0000 7fa516a9f640 1 -- 172.21.15.57:0/1360470143 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa4f40664a0 msgr2=0x7fa4f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:30.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:30.086+0000 7fa516a9f640 1 --2- 172.21.15.57:0/1360470143 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa4f40664a0 0x7fa4f4068960 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa50c007fd0 tx=0x7fa50c009040 comp rx=0 tx=0).stop 2023-10-30T14:44:30.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:30.086+0000 7fa516a9f640 1 -- 172.21.15.57:0/1360470143 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa51015cd70 msgr2=0x7fa51006ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:30.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:30.086+0000 7fa516a9f640 1 --2- 172.21.15.57:0/1360470143 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa51015cd70 0x7fa51006ae30 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fa5080b2040 tx=0x7fa50801bed0 comp rx=0 tx=0).stop 2023-10-30T14:44:30.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:30.086+0000 7fa516a9f640 1 -- 172.21.15.57:0/1360470143 shutdown_connections 2023-10-30T14:44:30.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:30.086+0000 7fa516a9f640 1 --2- 172.21.15.57:0/1360470143 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa4f40664a0 0x7fa4f4068960 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:30.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:30.086+0000 7fa516a9f640 1 --2- 172.21.15.57:0/1360470143 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa51015cd70 0x7fa51006ae30 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:30.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:30.086+0000 7fa516a9f640 1 --2- 172.21.15.57:0/1360470143 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa51015ac70 0x7fa5101500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:30.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:30.086+0000 7fa516a9f640 1 --2- 172.21.15.57:0/1360470143 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5100c8eb0 0x7fa5100c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:30.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:30.086+0000 7fa516a9f640 1 -- 172.21.15.57:0/1360470143 >> 172.21.15.57:0/1360470143 conn(0x7fa5100cef00 msgr2=0x7fa51015bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:30.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:30.086+0000 7fa516a9f640 1 -- 172.21.15.57:0/1360470143 shutdown_connections 2023-10-30T14:44:30.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:30.086+0000 7fa516a9f640 1 -- 172.21.15.57:0/1360470143 wait complete. 2023-10-30T14:44:30.108 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:44:32.109 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:32.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.250+0000 7f0d7a3f2640 1 Processor -- start 2023-10-30T14:44:32.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.250+0000 7f0d7a3f2640 1 -- start start 2023-10-30T14:44:32.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.254+0000 7f0d7a3f2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d740c34e0 0x7f0d7406dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:32.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.254+0000 7f0d7a3f2640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0d7406e230 0x7f0d74150240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:32.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.254+0000 7f0d7a3f2640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d740c39a0 con 0x7f0d740c3100 2023-10-30T14:44:32.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.254+0000 7f0d7a3f2640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d7406d8d0 con 0x7f0d7406e230 2023-10-30T14:44:32.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.254+0000 7f0d7a3f2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d74159f20 con 0x7f0d740c34e0 2023-10-30T14:44:32.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.254+0000 7f0d793f0640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f0d740c3100 0x7f0d740c1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:39290/0 (socket says 172.21.15.57:39290) 2023-10-30T14:44:32.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.254+0000 7f0d793f0640 1 -- 172.21.15.57:0/2655618280 learned_addr learned my addr 172.21.15.57:0/2655618280 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:44:32.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.254+0000 7f0d78bef640 1 --2- 172.21.15.57:0/2655618280 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d740c34e0 0x7f0d7406dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:32.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.254+0000 7f0d79bf1640 1 --2- 172.21.15.57:0/2655618280 >> v2:172.21.15.88:3300/0 conn(0x7f0d7406e230 0x7f0d74150240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:32.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.254+0000 7f0d6bfff640 1 -- 172.21.15.57:0/2655618280 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772398575 0 0) 0x7f0d740c39a0 con 0x7f0d740c3100 2023-10-30T14:44:32.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.254+0000 7f0d6bfff640 1 -- 172.21.15.57:0/2655618280 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d60003530 con 0x7f0d740c3100 2023-10-30T14:44:32.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.254+0000 7f0d78bef640 1 -- 172.21.15.57:0/2655618280 >> v2:172.21.15.88:3300/0 conn(0x7f0d7406e230 msgr2=0x7f0d74150240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:32.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.254+0000 7f0d78bef640 1 --2- 172.21.15.57:0/2655618280 >> v2:172.21.15.88:3300/0 conn(0x7f0d7406e230 0x7f0d74150240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:32.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.254+0000 7f0d78bef640 1 -- 172.21.15.57:0/2655618280 >> v1:172.21.15.57:6789/0 conn(0x7f0d740c3100 legacy=0x7f0d740c1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:32.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.254+0000 7f0d78bef640 1 -- 172.21.15.57:0/2655618280 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7415a200 con 0x7f0d740c34e0 2023-10-30T14:44:32.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.254+0000 7f0d79bf1640 1 --2- 172.21.15.57:0/2655618280 >> v2:172.21.15.88:3300/0 conn(0x7f0d7406e230 0x7f0d74150240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:44:32.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.258+0000 7f0d78bef640 1 --2- 172.21.15.57:0/2655618280 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d740c34e0 0x7f0d7406dcf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0d640097c0 tx=0x7f0d6401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1bd4cc43f048867 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:32.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.262+0000 7f0d6bfff640 1 -- 172.21.15.57:0/2655618280 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d640a8970 con 0x7f0d740c34e0 2023-10-30T14:44:32.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.262+0000 7f0d6bfff640 1 -- 172.21.15.57:0/2655618280 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d640a8ad0 con 0x7f0d740c34e0 2023-10-30T14:44:32.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.262+0000 7f0d6bfff640 1 -- 172.21.15.57:0/2655618280 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d640a8e60 con 0x7f0d740c34e0 2023-10-30T14:44:32.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.262+0000 7f0d7a3f2640 1 -- 172.21.15.57:0/2655618280 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d740c34e0 msgr2=0x7f0d7406dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:32.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.262+0000 7f0d7a3f2640 1 --2- 172.21.15.57:0/2655618280 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d740c34e0 0x7f0d7406dcf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0d640097c0 tx=0x7f0d6401ec70 comp rx=0 tx=0).stop 2023-10-30T14:44:32.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.262+0000 7f0d7a3f2640 1 -- 172.21.15.57:0/2655618280 shutdown_connections 2023-10-30T14:44:32.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.262+0000 7f0d7a3f2640 1 --2- 172.21.15.57:0/2655618280 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d740c34e0 0x7f0d7406dcf0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:32.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.262+0000 7f0d7a3f2640 1 --2- 172.21.15.57:0/2655618280 >> v2:172.21.15.88:3300/0 conn(0x7f0d7406e230 0x7f0d74150240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:32.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.262+0000 7f0d7a3f2640 1 -- 172.21.15.57:0/2655618280 >> 172.21.15.57:0/2655618280 conn(0x7f0d740caae0 msgr2=0x7f0d740caec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:32.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.262+0000 7f0d7a3f2640 1 -- 172.21.15.57:0/2655618280 shutdown_connections 2023-10-30T14:44:32.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.262+0000 7f0d7a3f2640 1 -- 172.21.15.57:0/2655618280 wait complete. 2023-10-30T14:44:32.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.262+0000 7f0d7a3f2640 1 Processor -- start 2023-10-30T14:44:32.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.262+0000 7f0d7a3f2640 1 -- start start 2023-10-30T14:44:32.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d7a3f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d7406e230 0x7f0d741f0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:32.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d7a3f2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d740c3100 0x7f0d741f1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:32.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d7a3f2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d740c34e0 0x7f0d7414d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:32.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d7a3f2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d741f1550 con 0x7f0d7406e230 2023-10-30T14:44:32.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d7a3f2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d74157330 con 0x7f0d740c3100 2023-10-30T14:44:32.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d7a3f2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d741ef7b0 con 0x7f0d740c34e0 2023-10-30T14:44:32.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d793f0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d7406e230 0x7f0d741f0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:32.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d793f0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d7406e230 0x7f0d741f0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47502/0 (socket says 172.21.15.57:47502) 2023-10-30T14:44:32.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d793f0640 1 -- 172.21.15.57:0/3859420483 learned_addr learned my addr 172.21.15.57:0/3859420483 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:32.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d78bef640 1 --2- 172.21.15.57:0/3859420483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d740c3100 0x7f0d741f1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:32.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d79bf1640 1 --2- 172.21.15.57:0/3859420483 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d740c34e0 0x7f0d7414d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:32.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d793f0640 1 -- 172.21.15.57:0/3859420483 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d740c34e0 msgr2=0x7f0d7414d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:32.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d793f0640 1 --2- 172.21.15.57:0/3859420483 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d740c34e0 0x7f0d7414d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:32.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d793f0640 1 -- 172.21.15.57:0/3859420483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d740c3100 msgr2=0x7f0d741f1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:32.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d793f0640 1 --2- 172.21.15.57:0/3859420483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d740c3100 0x7f0d741f1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:32.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d793f0640 1 -- 172.21.15.57:0/3859420483 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c009a70 con 0x7f0d7406e230 2023-10-30T14:44:32.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d78bef640 1 --2- 172.21.15.57:0/3859420483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d740c3100 0x7f0d741f1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:44:32.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d793f0640 1 --2- 172.21.15.57:0/3859420483 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d7406e230 0x7f0d741f0ad0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f0d7000c500 tx=0x7f0d7000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:32.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d69ffb640 1 -- 172.21.15.57:0/3859420483 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d70005cf0 con 0x7f0d7406e230 2023-10-30T14:44:32.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d69ffb640 1 -- 172.21.15.57:0/3859420483 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d70005e50 con 0x7f0d7406e230 2023-10-30T14:44:32.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d69ffb640 1 -- 172.21.15.57:0/3859420483 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d700072f0 con 0x7f0d7406e230 2023-10-30T14:44:32.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d7a3f2640 1 -- 172.21.15.57:0/3859420483 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d64009420 con 0x7f0d7406e230 2023-10-30T14:44:32.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d7a3f2640 1 -- 172.21.15.57:0/3859420483 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d741efd10 con 0x7f0d7406e230 2023-10-30T14:44:32.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d7a3f2640 1 -- 172.21.15.57:0/3859420483 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d7404e170 con 0x7f0d7406e230 2023-10-30T14:44:32.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.266+0000 7f0d69ffb640 1 -- 172.21.15.57:0/3859420483 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0d70007450 con 0x7f0d7406e230 2023-10-30T14:44:32.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.270+0000 7f0d69ffb640 1 --2- 172.21.15.57:0/3859420483 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0d580664a0 0x7f0d58068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:32.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.270+0000 7f0d69ffb640 1 -- 172.21.15.57:0/3859420483 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f0d70083a60 con 0x7f0d7406e230 2023-10-30T14:44:32.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.270+0000 7f0d78bef640 1 --2- 172.21.15.57:0/3859420483 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0d580664a0 0x7f0d58068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:32.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.270+0000 7f0d78bef640 1 --2- 172.21.15.57:0/3859420483 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0d580664a0 0x7f0d58068960 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f0d640b2000 tx=0x7f0d640aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:32.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.274+0000 7f0d69ffb640 1 -- 172.21.15.57:0/3859420483 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0d700509a0 con 0x7f0d7406e230 2023-10-30T14:44:32.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.462+0000 7f0d7a3f2640 1 -- 172.21.15.57:0/3859420483 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d74071130 con 0x7f0d7406e230 2023-10-30T14:44:32.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.462+0000 7f0d69ffb640 1 -- 172.21.15.57:0/3859420483 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+5718 (secure 0 0 0) 0x7f0d70017090 con 0x7f0d7406e230 2023-10-30T14:44:32.464 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:32.465 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:44:32.012064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:resolve","state_seq":10,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:32.465 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 165 2023-10-30T14:44:32.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.462+0000 7f0d7a3f2640 1 -- 172.21.15.57:0/3859420483 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0d580664a0 msgr2=0x7f0d58068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:32.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.462+0000 7f0d7a3f2640 1 --2- 172.21.15.57:0/3859420483 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0d580664a0 0x7f0d58068960 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f0d640b2000 tx=0x7f0d640aa070 comp rx=0 tx=0).stop 2023-10-30T14:44:32.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.462+0000 7f0d7a3f2640 1 -- 172.21.15.57:0/3859420483 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d7406e230 msgr2=0x7f0d741f0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:32.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.462+0000 7f0d7a3f2640 1 --2- 172.21.15.57:0/3859420483 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d7406e230 0x7f0d741f0ad0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f0d7000c500 tx=0x7f0d7000c9c0 comp rx=0 tx=0).stop 2023-10-30T14:44:32.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.462+0000 7f0d7a3f2640 1 -- 172.21.15.57:0/3859420483 shutdown_connections 2023-10-30T14:44:32.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.462+0000 7f0d7a3f2640 1 --2- 172.21.15.57:0/3859420483 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0d580664a0 0x7f0d58068960 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:32.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.462+0000 7f0d7a3f2640 1 --2- 172.21.15.57:0/3859420483 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d740c34e0 0x7f0d7414d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:32.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.462+0000 7f0d7a3f2640 1 --2- 172.21.15.57:0/3859420483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d740c3100 0x7f0d741f1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:32.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.462+0000 7f0d7a3f2640 1 --2- 172.21.15.57:0/3859420483 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d7406e230 0x7f0d741f0ad0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:32.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.462+0000 7f0d7a3f2640 1 -- 172.21.15.57:0/3859420483 >> 172.21.15.57:0/3859420483 conn(0x7f0d740caae0 msgr2=0x7f0d74159740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:32.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.466+0000 7f0d7a3f2640 1 -- 172.21.15.57:0/3859420483 shutdown_connections 2023-10-30T14:44:32.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:32.466+0000 7f0d7a3f2640 1 -- 172.21.15.57:0/3859420483 wait complete. 2023-10-30T14:44:32.485 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:44:34.487 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:34.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.642+0000 7f7d086ea640 1 Processor -- start 2023-10-30T14:44:34.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.642+0000 7f7d086ea640 1 -- start start 2023-10-30T14:44:34.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.642+0000 7f7d086ea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7d00159de0 0x7f7d000726b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:34.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.642+0000 7f7d086ea640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d00071c10 con 0x7f7d00065a80 2023-10-30T14:44:34.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.642+0000 7f7d086ea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0015a790 con 0x7f7d0006ee40 2023-10-30T14:44:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.642+0000 7f7d086ea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d00071eb0 con 0x7f7d00159de0 2023-10-30T14:44:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.642+0000 7f7d06c87640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f7d00065a80 0x7f7d00160f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41932/0 (socket says 172.21.15.57:41932) 2023-10-30T14:44:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.642+0000 7f7d06c87640 1 -- 172.21.15.57:0/3200447578 learned_addr learned my addr 172.21.15.57:0/3200447578 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:44:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.642+0000 7f7d06486640 1 --2- 172.21.15.57:0/3200447578 >> v2:172.21.15.88:3300/0 conn(0x7f7d00159de0 0x7f7d000726b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.642+0000 7f7d05484640 1 -- 172.21.15.57:0/3200447578 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3302308817 0 0) 0x7f7d00071c10 con 0x7f7d00065a80 2023-10-30T14:44:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.642+0000 7f7d05484640 1 -- 172.21.15.57:0/3200447578 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cec003530 con 0x7f7d00065a80 2023-10-30T14:44:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.642+0000 7f7d06486640 1 -- 172.21.15.57:0/3200447578 >> v1:172.21.15.88:6789/0 conn(0x7f7d0006ee40 legacy=0x7f7d00150280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.642+0000 7f7d05484640 1 -- 172.21.15.57:0/3200447578 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 186093363 0 0) 0x7f7cec003530 con 0x7f7d00065a80 2023-10-30T14:44:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.642+0000 7f7d06486640 1 -- 172.21.15.57:0/3200447578 >> v1:172.21.15.57:6789/0 conn(0x7f7d00065a80 legacy=0x7f7d00160f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.642+0000 7f7d06486640 1 -- 172.21.15.57:0/3200447578 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d00072190 con 0x7f7d00159de0 2023-10-30T14:44:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.646+0000 7f7d06486640 1 --2- 172.21.15.57:0/3200447578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d00159de0 0x7f7d000726b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc0097c0 tx=0x7f7cfc01c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df3ded86b8763381 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.646+0000 7f7d05484640 1 -- 172.21.15.57:0/3200447578 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8001a00 con 0x7f7d00159de0 2023-10-30T14:44:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.646+0000 7f7d05484640 1 -- 172.21.15.57:0/3200447578 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cfc0a7050 con 0x7f7d00159de0 2023-10-30T14:44:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.646+0000 7f7d05484640 1 -- 172.21.15.57:0/3200447578 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc0ab620 con 0x7f7d00159de0 2023-10-30T14:44:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.646+0000 7f7d086ea640 1 -- 172.21.15.57:0/3200447578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d00159de0 msgr2=0x7f7d000726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.646+0000 7f7d086ea640 1 --2- 172.21.15.57:0/3200447578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d00159de0 0x7f7d000726b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc0097c0 tx=0x7f7cfc01c3a0 comp rx=0 tx=0).stop 2023-10-30T14:44:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.646+0000 7f7d086ea640 1 -- 172.21.15.57:0/3200447578 shutdown_connections 2023-10-30T14:44:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.646+0000 7f7d086ea640 1 --2- 172.21.15.57:0/3200447578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d00159de0 0x7f7d000726b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:34.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.646+0000 7f7d086ea640 1 -- 172.21.15.57:0/3200447578 >> 172.21.15.57:0/3200447578 conn(0x7f7d000c7f60 msgr2=0x7f7d000c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:34.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.646+0000 7f7d086ea640 1 -- 172.21.15.57:0/3200447578 shutdown_connections 2023-10-30T14:44:34.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.646+0000 7f7d086ea640 1 -- 172.21.15.57:0/3200447578 wait complete. 2023-10-30T14:44:34.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d086ea640 1 Processor -- start 2023-10-30T14:44:34.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d086ea640 1 -- start start 2023-10-30T14:44:34.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d086ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d00065a80 0x7f7d00159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:34.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d086ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7d0006ee40 0x7f7d0014ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:34.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d086ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d00159de0 0x7f7d00157850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:34.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d086ea640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d00148210 con 0x7f7d00065a80 2023-10-30T14:44:34.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d086ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d00140ab0 con 0x7f7d00159de0 2023-10-30T14:44:34.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d086ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d001597b0 con 0x7f7d0006ee40 2023-10-30T14:44:34.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d06486640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d00065a80 0x7f7d00159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:34.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d06486640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d00065a80 0x7f7d00159170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50402/0 (socket says 172.21.15.57:50402) 2023-10-30T14:44:34.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d06486640 1 -- 172.21.15.57:0/3933745724 learned_addr learned my addr 172.21.15.57:0/3933745724 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:34.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d06c87640 1 --2- 172.21.15.57:0/3933745724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d00159de0 0x7f7d00157850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:34.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d05c85640 1 --2- 172.21.15.57:0/3933745724 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7d0006ee40 0x7f7d0014ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:34.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d06486640 1 -- 172.21.15.57:0/3933745724 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7d0006ee40 msgr2=0x7f7d0014ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:34.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d06486640 1 --2- 172.21.15.57:0/3933745724 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7d0006ee40 0x7f7d0014ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:34.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d06486640 1 -- 172.21.15.57:0/3933745724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d00159de0 msgr2=0x7f7d00157850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:34.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d06486640 1 --2- 172.21.15.57:0/3933745724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d00159de0 0x7f7d00157850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:34.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d06486640 1 -- 172.21.15.57:0/3933745724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cfc009420 con 0x7f7d00065a80 2023-10-30T14:44:34.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d06486640 1 --2- 172.21.15.57:0/3933745724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d00065a80 0x7f7d00159170 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc0097c0 tx=0x7f7cfc01abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:34.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7cf6ffd640 1 -- 172.21.15.57:0/3933745724 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc004020 con 0x7f7d00065a80 2023-10-30T14:44:34.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7cf6ffd640 1 -- 172.21.15.57:0/3933745724 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cfc004180 con 0x7f7d00065a80 2023-10-30T14:44:34.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7cf6ffd640 1 -- 172.21.15.57:0/3933745724 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc003620 con 0x7f7d00065a80 2023-10-30T14:44:34.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d086ea640 1 -- 172.21.15.57:0/3933745724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d00140d00 con 0x7f7d00065a80 2023-10-30T14:44:34.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.650+0000 7f7d086ea640 1 -- 172.21.15.57:0/3933745724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d001412c0 con 0x7f7d00065a80 2023-10-30T14:44:34.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.654+0000 7f7d086ea640 1 -- 172.21.15.57:0/3933745724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d0004e170 con 0x7f7d00065a80 2023-10-30T14:44:34.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.654+0000 7f7cf6ffd640 1 -- 172.21.15.57:0/3933745724 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7cfc003780 con 0x7f7d00065a80 2023-10-30T14:44:34.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.654+0000 7f7cf6ffd640 1 --2- 172.21.15.57:0/3933745724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7ce40664a0 0x7f7ce4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:34.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.654+0000 7f7cf6ffd640 1 -- 172.21.15.57:0/3933745724 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f7cfc11dfa0 con 0x7f7d00065a80 2023-10-30T14:44:34.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.654+0000 7f7d05c85640 1 --2- 172.21.15.57:0/3933745724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7ce40664a0 0x7f7ce4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:34.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.654+0000 7f7d05c85640 1 --2- 172.21.15.57:0/3933745724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7ce40664a0 0x7f7ce4068960 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7d001571b0 tx=0x7f7cf0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:34.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.658+0000 7f7cf6ffd640 1 -- 172.21.15.57:0/3933745724 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7cfc0ead10 con 0x7f7d00065a80 2023-10-30T14:44:34.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.850+0000 7f7d086ea640 1 -- 172.21.15.57:0/3933745724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d0006ea60 con 0x7f7d00065a80 2023-10-30T14:44:34.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.850+0000 7f7cf6ffd640 1 -- 172.21.15.57:0/3933745724 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+5718 (secure 0 0 0) 0x7f7cfc0bc090 con 0x7f7d00065a80 2023-10-30T14:44:34.853 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:34.853 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:44:32.012064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:resolve","state_seq":10,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:34.854 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 165 2023-10-30T14:44:34.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.854+0000 7f7d086ea640 1 -- 172.21.15.57:0/3933745724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7ce40664a0 msgr2=0x7f7ce4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:34.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.854+0000 7f7d086ea640 1 --2- 172.21.15.57:0/3933745724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7ce40664a0 0x7f7ce4068960 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7d001571b0 tx=0x7f7cf0009040 comp rx=0 tx=0).stop 2023-10-30T14:44:34.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.854+0000 7f7d086ea640 1 -- 172.21.15.57:0/3933745724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d00065a80 msgr2=0x7f7d00159170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:34.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.854+0000 7f7d086ea640 1 --2- 172.21.15.57:0/3933745724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d00065a80 0x7f7d00159170 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc0097c0 tx=0x7f7cfc01abe0 comp rx=0 tx=0).stop 2023-10-30T14:44:34.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.854+0000 7f7d086ea640 1 -- 172.21.15.57:0/3933745724 shutdown_connections 2023-10-30T14:44:34.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.854+0000 7f7d086ea640 1 --2- 172.21.15.57:0/3933745724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7ce40664a0 0x7f7ce4068960 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:34.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.854+0000 7f7d086ea640 1 --2- 172.21.15.57:0/3933745724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d00159de0 0x7f7d00157850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:34.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.854+0000 7f7d086ea640 1 --2- 172.21.15.57:0/3933745724 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7d0006ee40 0x7f7d0014ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:34.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.854+0000 7f7d086ea640 1 --2- 172.21.15.57:0/3933745724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d00065a80 0x7f7d00159170 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:34.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.854+0000 7f7d086ea640 1 -- 172.21.15.57:0/3933745724 >> 172.21.15.57:0/3933745724 conn(0x7f7d000c7f60 msgr2=0x7f7d00146b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:34.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.854+0000 7f7d086ea640 1 -- 172.21.15.57:0/3933745724 shutdown_connections 2023-10-30T14:44:34.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:34.854+0000 7f7d086ea640 1 -- 172.21.15.57:0/3933745724 wait complete. 2023-10-30T14:44:34.874 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 165, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 165, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 44 }, "gid_7536": { "addr": "172.21.15.57:6837/816502245", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 816502245, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 816502245, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7536, "incarnation": 164, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:resolve", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:44:32.012064+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7536, "mds_1": 6747 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.57:6834/697507447", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 697507447, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 697507447, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 156, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7377, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:44:36.875 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:37.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.026+0000 7f1390301640 1 Processor -- start 2023-10-30T14:44:37.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.026+0000 7f1390301640 1 -- start start 2023-10-30T14:44:37.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.030+0000 7f1390301640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1388151840 0x7f13880c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:37.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.030+0000 7f1390301640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1388151c20 0x7f1388072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:37.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.030+0000 7f1390301640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13880c5030 con 0x7f13880cb8b0 2023-10-30T14:44:37.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.030+0000 7f1390301640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1388157830 con 0x7f1388151c20 2023-10-30T14:44:37.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.030+0000 7f1390301640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1388157b10 con 0x7f1388151840 2023-10-30T14:44:37.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.030+0000 7f138e89e640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f13880cb8b0 0x7f138806b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41946/0 (socket says 172.21.15.57:41946) 2023-10-30T14:44:37.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.030+0000 7f138e89e640 1 -- 172.21.15.57:0/295536760 learned_addr learned my addr 172.21.15.57:0/295536760 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:44:37.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.030+0000 7f138e09d640 1 --2- 172.21.15.57:0/295536760 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1388151840 0x7f13880c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:37.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.030+0000 7f138d89c640 1 --2- 172.21.15.57:0/295536760 >> v2:172.21.15.88:3300/0 conn(0x7f1388151c20 0x7f1388072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:37.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.030+0000 7f138d09b640 1 -- 172.21.15.57:0/295536760 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3502189249 0 0) 0x7f13880c5030 con 0x7f13880cb8b0 2023-10-30T14:44:37.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.030+0000 7f138d09b640 1 -- 172.21.15.57:0/295536760 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1374003530 con 0x7f13880cb8b0 2023-10-30T14:44:37.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.030+0000 7f138e09d640 1 -- 172.21.15.57:0/295536760 >> v2:172.21.15.88:3300/0 conn(0x7f1388151c20 msgr2=0x7f1388072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:37.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.030+0000 7f138e09d640 1 --2- 172.21.15.57:0/295536760 >> v2:172.21.15.88:3300/0 conn(0x7f1388151c20 0x7f1388072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:37.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.030+0000 7f138e09d640 1 -- 172.21.15.57:0/295536760 >> v1:172.21.15.57:6789/0 conn(0x7f13880cb8b0 legacy=0x7f138806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:37.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.030+0000 7f138d09b640 1 -- 172.21.15.57:0/295536760 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 979854771 0 0) 0x7f1374003530 con 0x7f13880cb8b0 2023-10-30T14:44:37.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.030+0000 7f138e09d640 1 -- 172.21.15.57:0/295536760 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1388157df0 con 0x7f1388151840 2023-10-30T14:44:37.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.034+0000 7f138e09d640 1 --2- 172.21.15.57:0/295536760 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1388151840 0x7f13880c49a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f13840097c0 tx=0x7f138401eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af73becc721a8dc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:37.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.034+0000 7f138d09b640 1 -- 172.21.15.57:0/295536760 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1380001a00 con 0x7f1388151840 2023-10-30T14:44:37.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.034+0000 7f138d09b640 1 -- 172.21.15.57:0/295536760 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13840a85d0 con 0x7f1388151840 2023-10-30T14:44:37.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.034+0000 7f138d09b640 1 -- 172.21.15.57:0/295536760 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13840a8940 con 0x7f1388151840 2023-10-30T14:44:37.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.034+0000 7f1390301640 1 -- 172.21.15.57:0/295536760 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1388151840 msgr2=0x7f13880c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:37.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.034+0000 7f1390301640 1 --2- 172.21.15.57:0/295536760 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1388151840 0x7f13880c49a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f13840097c0 tx=0x7f138401eb90 comp rx=0 tx=0).stop 2023-10-30T14:44:37.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.034+0000 7f1390301640 1 -- 172.21.15.57:0/295536760 shutdown_connections 2023-10-30T14:44:37.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.034+0000 7f1390301640 1 --2- 172.21.15.57:0/295536760 >> v2:172.21.15.88:3300/0 conn(0x7f1388151c20 0x7f1388072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:37.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.034+0000 7f1390301640 1 --2- 172.21.15.57:0/295536760 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1388151840 0x7f13880c49a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:37.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.034+0000 7f1390301640 1 -- 172.21.15.57:0/295536760 >> 172.21.15.57:0/295536760 conn(0x7f13880cac70 msgr2=0x7f13880cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:37.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.034+0000 7f1390301640 1 -- 172.21.15.57:0/295536760 shutdown_connections 2023-10-30T14:44:37.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.034+0000 7f1390301640 1 -- 172.21.15.57:0/295536760 wait complete. 2023-10-30T14:44:37.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f1390301640 1 Processor -- start 2023-10-30T14:44:37.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f1390301640 1 -- start start 2023-10-30T14:44:37.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f1390301640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13880cb8b0 0x7f138814bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:37.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f1390301640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1388151840 0x7f138814c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:37.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f1390301640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1388151c20 0x7f13880c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:37.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f1390301640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13881e17d0 con 0x7f1388151c20 2023-10-30T14:44:37.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f1390301640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13881e0440 con 0x7f13880cb8b0 2023-10-30T14:44:37.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f1390301640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1388071010 con 0x7f1388151840 2023-10-30T14:44:37.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f138e89e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1388151c20 0x7f13880c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:37.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f138e89e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1388151c20 0x7f13880c8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50414/0 (socket says 172.21.15.57:50414) 2023-10-30T14:44:37.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f138e89e640 1 -- 172.21.15.57:0/543247204 learned_addr learned my addr 172.21.15.57:0/543247204 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:37.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f138e09d640 1 --2- 172.21.15.57:0/543247204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13880cb8b0 0x7f138814bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:37.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f138d89c640 1 --2- 172.21.15.57:0/543247204 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1388151840 0x7f138814c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:37.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f138e89e640 1 -- 172.21.15.57:0/543247204 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1388151840 msgr2=0x7f138814c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:37.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f138e89e640 1 --2- 172.21.15.57:0/543247204 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1388151840 0x7f138814c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:37.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f138e89e640 1 -- 172.21.15.57:0/543247204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13880cb8b0 msgr2=0x7f138814bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:37.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f138e89e640 1 --2- 172.21.15.57:0/543247204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13880cb8b0 0x7f138814bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:37.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f138e89e640 1 -- 172.21.15.57:0/543247204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1384009420 con 0x7f1388151c20 2023-10-30T14:44:37.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f138d89c640 1 --2- 172.21.15.57:0/543247204 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1388151840 0x7f138814c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:44:37.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f138e09d640 1 --2- 172.21.15.57:0/543247204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13880cb8b0 0x7f138814bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:44:37.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f138e89e640 1 --2- 172.21.15.57:0/543247204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1388151c20 0x7f13880c8ae0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f1380001f80 tx=0x7f138000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:37.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f137effd640 1 -- 172.21.15.57:0/543247204 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1380012400 con 0x7f1388151c20 2023-10-30T14:44:37.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f137effd640 1 -- 172.21.15.57:0/543247204 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1380012560 con 0x7f1388151c20 2023-10-30T14:44:37.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f1390301640 1 -- 172.21.15.57:0/543247204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13881e0700 con 0x7f1388151c20 2023-10-30T14:44:37.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f137effd640 1 -- 172.21.15.57:0/543247204 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13800128d0 con 0x7f1388151c20 2023-10-30T14:44:37.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f1390301640 1 -- 172.21.15.57:0/543247204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13880c3590 con 0x7f1388151c20 2023-10-30T14:44:37.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.038+0000 7f1390301640 1 -- 172.21.15.57:0/543247204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13881e1910 con 0x7f1388151c20 2023-10-30T14:44:37.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.042+0000 7f137effd640 1 -- 172.21.15.57:0/543247204 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1380012a30 con 0x7f1388151c20 2023-10-30T14:44:37.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.042+0000 7f137effd640 1 --2- 172.21.15.57:0/543247204 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f136c0664a0 0x7f136c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:37.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.042+0000 7f137effd640 1 -- 172.21.15.57:0/543247204 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f1380084d40 con 0x7f1388151c20 2023-10-30T14:44:37.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.042+0000 7f138e09d640 1 --2- 172.21.15.57:0/543247204 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f136c0664a0 0x7f136c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:37.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.042+0000 7f138e09d640 1 --2- 172.21.15.57:0/543247204 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f136c0664a0 0x7f136c068960 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f13840b2000 tx=0x7f13840aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:37.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.046+0000 7f137effd640 1 -- 172.21.15.57:0/543247204 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f13881e1910 con 0x7f1388151c20 2023-10-30T14:44:37.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.238+0000 7f1390301640 1 -- 172.21.15.57:0/543247204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13881e1910 con 0x7f1388151c20 2023-10-30T14:44:37.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.242+0000 7f137effd640 1 -- 172.21.15.57:0/543247204 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+5718 (secure 0 0 0) 0x7f13881e1910 con 0x7f1388151c20 2023-10-30T14:44:37.243 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:37.243 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:44:32.012064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:resolve","state_seq":10,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:37.244 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 165 2023-10-30T14:44:37.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.242+0000 7f1390301640 1 -- 172.21.15.57:0/543247204 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f136c0664a0 msgr2=0x7f136c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:37.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.242+0000 7f1390301640 1 --2- 172.21.15.57:0/543247204 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f136c0664a0 0x7f136c068960 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f13840b2000 tx=0x7f13840aa070 comp rx=0 tx=0).stop 2023-10-30T14:44:37.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.242+0000 7f1390301640 1 -- 172.21.15.57:0/543247204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1388151c20 msgr2=0x7f13880c8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:37.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.242+0000 7f1390301640 1 --2- 172.21.15.57:0/543247204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1388151c20 0x7f13880c8ae0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f1380001f80 tx=0x7f138000a430 comp rx=0 tx=0).stop 2023-10-30T14:44:37.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.242+0000 7f1390301640 1 -- 172.21.15.57:0/543247204 shutdown_connections 2023-10-30T14:44:37.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.242+0000 7f1390301640 1 --2- 172.21.15.57:0/543247204 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f136c0664a0 0x7f136c068960 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:37.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.242+0000 7f1390301640 1 --2- 172.21.15.57:0/543247204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1388151c20 0x7f13880c8ae0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:37.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.242+0000 7f1390301640 1 --2- 172.21.15.57:0/543247204 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1388151840 0x7f138814c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:37.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.242+0000 7f1390301640 1 --2- 172.21.15.57:0/543247204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13880cb8b0 0x7f138814bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:37.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.242+0000 7f1390301640 1 -- 172.21.15.57:0/543247204 >> 172.21.15.57:0/543247204 conn(0x7f13880cac70 msgr2=0x7f1388146720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:37.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.242+0000 7f1390301640 1 -- 172.21.15.57:0/543247204 shutdown_connections 2023-10-30T14:44:37.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:37.242+0000 7f1390301640 1 -- 172.21.15.57:0/543247204 wait complete. 2023-10-30T14:44:37.263 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:44:39.098 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:44:39.102 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:44:39.265 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:39.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.418+0000 7f3435501640 1 Processor -- start 2023-10-30T14:44:39.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.418+0000 7f3435501640 1 -- start start 2023-10-30T14:44:39.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.418+0000 7f3435501640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34300c4a10 0x7f343015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:39.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.418+0000 7f3435501640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f34300c42a0 0x7f34300ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:39.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.418+0000 7f3435501640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3430159f20 con 0x7f3430153c80 2023-10-30T14:44:39.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.418+0000 7f3435501640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343015dec0 con 0x7f34300c42a0 2023-10-30T14:44:39.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.418+0000 7f3435501640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34301599a0 con 0x7f34300c4a10 2023-10-30T14:44:39.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.422+0000 7f3427fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34300c4a10 0x7f343015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:39.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.422+0000 7f342f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f34300c42a0 0x7f34300ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:39.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.422+0000 7f342effd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3430153c80 0x7f343015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:44192/0 (socket says 172.21.15.57:44192) 2023-10-30T14:44:39.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.422+0000 7f342effd640 1 -- 172.21.15.57:0/2373695322 learned_addr learned my addr 172.21.15.57:0/2373695322 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:44:39.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.422+0000 7f3427fff640 1 -- 172.21.15.57:0/2373695322 >> v2:172.21.15.88:3300/0 conn(0x7f34300c42a0 msgr2=0x7f34300ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:39.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.422+0000 7f3427fff640 1 --2- 172.21.15.57:0/2373695322 >> v2:172.21.15.88:3300/0 conn(0x7f34300c42a0 0x7f34300ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:39.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.422+0000 7f3427fff640 1 -- 172.21.15.57:0/2373695322 >> v1:172.21.15.88:6789/0 conn(0x7f3430153c80 legacy=0x7f343015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:39.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.422+0000 7f3427fff640 1 -- 172.21.15.57:0/2373695322 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343015d620 con 0x7f34300c4a10 2023-10-30T14:44:39.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.422+0000 7f3427fff640 1 --2- 172.21.15.57:0/2373695322 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34300c4a10 0x7f343015cf90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f34180097c0 tx=0x7f341801ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7c362c77a0a6409 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:39.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.426+0000 7f342e7fc640 1 -- 172.21.15.57:0/2373695322 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34180a8970 con 0x7f34300c4a10 2023-10-30T14:44:39.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.426+0000 7f342e7fc640 1 -- 172.21.15.57:0/2373695322 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34180a8ad0 con 0x7f34300c4a10 2023-10-30T14:44:39.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.426+0000 7f342e7fc640 1 -- 172.21.15.57:0/2373695322 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34180a8e60 con 0x7f34300c4a10 2023-10-30T14:44:39.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.426+0000 7f3435501640 1 -- 172.21.15.57:0/2373695322 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34300c4a10 msgr2=0x7f343015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:39.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.426+0000 7f3435501640 1 --2- 172.21.15.57:0/2373695322 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34300c4a10 0x7f343015cf90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f34180097c0 tx=0x7f341801ec70 comp rx=0 tx=0).stop 2023-10-30T14:44:39.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.426+0000 7f3435501640 1 -- 172.21.15.57:0/2373695322 shutdown_connections 2023-10-30T14:44:39.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.426+0000 7f3435501640 1 --2- 172.21.15.57:0/2373695322 >> v2:172.21.15.88:3300/0 conn(0x7f34300c42a0 0x7f34300ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:39.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.426+0000 7f3435501640 1 --2- 172.21.15.57:0/2373695322 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34300c4a10 0x7f343015cf90 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:39.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.426+0000 7f3435501640 1 -- 172.21.15.57:0/2373695322 >> 172.21.15.57:0/2373695322 conn(0x7f34300715c0 msgr2=0x7f34300719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:39.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.426+0000 7f3435501640 1 -- 172.21.15.57:0/2373695322 shutdown_connections 2023-10-30T14:44:39.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.426+0000 7f3435501640 1 -- 172.21.15.57:0/2373695322 wait complete. 2023-10-30T14:44:39.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.426+0000 7f3435501640 1 Processor -- start 2023-10-30T14:44:39.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.426+0000 7f3435501640 1 -- start start 2023-10-30T14:44:39.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.426+0000 7f3435501640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34300c42a0 0x7f34300c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:39.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.426+0000 7f3435501640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34300c4a10 0x7f3430156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:39.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.426+0000 7f3435501640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3430153c80 0x7f3430156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:39.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.426+0000 7f3435501640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34300b3560 con 0x7f34300c4a10 2023-10-30T14:44:39.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.426+0000 7f3435501640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34300b28e0 con 0x7f34300c42a0 2023-10-30T14:44:39.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.426+0000 7f3435501640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34300b2bc0 con 0x7f3430153c80 2023-10-30T14:44:39.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.430+0000 7f3427fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34300c4a10 0x7f3430156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:39.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.430+0000 7f3427fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34300c4a10 0x7f3430156930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50416/0 (socket says 172.21.15.57:50416) 2023-10-30T14:44:39.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.430+0000 7f3427fff640 1 -- 172.21.15.57:0/608630986 learned_addr learned my addr 172.21.15.57:0/608630986 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:39.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.430+0000 7f342effd640 1 --2- 172.21.15.57:0/608630986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34300c42a0 0x7f34300c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:39.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.430+0000 7f342f7fe640 1 --2- 172.21.15.57:0/608630986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3430153c80 0x7f3430156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:39.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.430+0000 7f3427fff640 1 -- 172.21.15.57:0/608630986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3430153c80 msgr2=0x7f3430156e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:39.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.430+0000 7f3427fff640 1 --2- 172.21.15.57:0/608630986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3430153c80 0x7f3430156e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:39.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.430+0000 7f3427fff640 1 -- 172.21.15.57:0/608630986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34300c42a0 msgr2=0x7f34300c8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:39.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.430+0000 7f3427fff640 1 --2- 172.21.15.57:0/608630986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34300c42a0 0x7f34300c8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:39.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.430+0000 7f3427fff640 1 -- 172.21.15.57:0/608630986 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3418009420 con 0x7f34300c4a10 2023-10-30T14:44:39.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.430+0000 7f3427fff640 1 --2- 172.21.15.57:0/608630986 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34300c4a10 0x7f3430156930 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f341801d320 tx=0x7f341801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:39.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.430+0000 7f34277fe640 1 -- 172.21.15.57:0/608630986 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34180b1e00 con 0x7f34300c4a10 2023-10-30T14:44:39.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.430+0000 7f34277fe640 1 -- 172.21.15.57:0/608630986 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34180b9410 con 0x7f34300c4a10 2023-10-30T14:44:39.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.430+0000 7f34277fe640 1 -- 172.21.15.57:0/608630986 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34180b97c0 con 0x7f34300c4a10 2023-10-30T14:44:39.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.430+0000 7f3435501640 1 -- 172.21.15.57:0/608630986 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34300b2ea0 con 0x7f34300c4a10 2023-10-30T14:44:39.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.430+0000 7f3435501640 1 -- 172.21.15.57:0/608630986 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3430151e60 con 0x7f34300c4a10 2023-10-30T14:44:39.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.430+0000 7f3435501640 1 -- 172.21.15.57:0/608630986 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34300b36a0 con 0x7f34300c4a10 2023-10-30T14:44:39.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.430+0000 7f34277fe640 1 -- 172.21.15.57:0/608630986 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f34180b7040 con 0x7f34300c4a10 2023-10-30T14:44:39.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.434+0000 7f34277fe640 1 --2- 172.21.15.57:0/608630986 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f34100664a0 0x7f3410068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:39.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.434+0000 7f34277fe640 1 -- 172.21.15.57:0/608630986 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f3418121fb0 con 0x7f34300c4a10 2023-10-30T14:44:39.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.434+0000 7f342effd640 1 --2- 172.21.15.57:0/608630986 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f34100664a0 0x7f3410068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:39.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.434+0000 7f342effd640 1 --2- 172.21.15.57:0/608630986 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f34100664a0 0x7f3410068960 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3420000f70 tx=0x7f3420007e90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:39.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.438+0000 7f34277fe640 1 -- 172.21.15.57:0/608630986 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f34180eee60 con 0x7f34300c4a10 2023-10-30T14:44:39.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.626+0000 7f3435501640 1 -- 172.21.15.57:0/608630986 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f343004e170 con 0x7f34300c4a10 2023-10-30T14:44:39.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.626+0000 7f34277fe640 1 -- 172.21.15.57:0/608630986 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+5720 (secure 0 0 0) 0x7f34180b5020 con 0x7f34300c4a10 2023-10-30T14:44:39.628 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:39.629 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:44:37.830720+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:39.630 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 166 2023-10-30T14:44:39.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.626+0000 7f3435501640 1 -- 172.21.15.57:0/608630986 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f34100664a0 msgr2=0x7f3410068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:39.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.626+0000 7f3435501640 1 --2- 172.21.15.57:0/608630986 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f34100664a0 0x7f3410068960 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3420000f70 tx=0x7f3420007e90 comp rx=0 tx=0).stop 2023-10-30T14:44:39.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.626+0000 7f3435501640 1 -- 172.21.15.57:0/608630986 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34300c4a10 msgr2=0x7f3430156930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:39.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.626+0000 7f3435501640 1 --2- 172.21.15.57:0/608630986 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34300c4a10 0x7f3430156930 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f341801d320 tx=0x7f341801ba10 comp rx=0 tx=0).stop 2023-10-30T14:44:39.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.626+0000 7f3435501640 1 -- 172.21.15.57:0/608630986 shutdown_connections 2023-10-30T14:44:39.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.626+0000 7f3435501640 1 --2- 172.21.15.57:0/608630986 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f34100664a0 0x7f3410068960 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:39.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.626+0000 7f3435501640 1 --2- 172.21.15.57:0/608630986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3430153c80 0x7f3430156e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:39.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.626+0000 7f3435501640 1 --2- 172.21.15.57:0/608630986 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34300c4a10 0x7f3430156930 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:39.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.626+0000 7f3435501640 1 --2- 172.21.15.57:0/608630986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34300c42a0 0x7f34300c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:39.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.626+0000 7f3435501640 1 -- 172.21.15.57:0/608630986 >> 172.21.15.57:0/608630986 conn(0x7f34300715c0 msgr2=0x7f343006bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:39.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.626+0000 7f3435501640 1 -- 172.21.15.57:0/608630986 shutdown_connections 2023-10-30T14:44:39.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:39.626+0000 7f3435501640 1 -- 172.21.15.57:0/608630986 wait complete. 2023-10-30T14:44:39.649 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:44:41.650 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:41.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.802+0000 7f38c611d640 1 Processor -- start 2023-10-30T14:44:41.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.802+0000 7f38c611d640 1 -- start start 2023-10-30T14:44:41.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.806+0000 7f38c611d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f38c0151c20 0x7f38c00c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:41.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.806+0000 7f38c611d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f38c00cb8b0 0x7f38c006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:41.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.806+0000 7f38c611d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c00c5030 con 0x7f38c0151840 2023-10-30T14:44:41.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.806+0000 7f38c611d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c0157830 con 0x7f38c0151c20 2023-10-30T14:44:41.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.806+0000 7f38c611d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c0157b10 con 0x7f38c00cb8b0 2023-10-30T14:44:41.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.806+0000 7f38bf7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f38c0151840 0x7f38c0072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41952/0 (socket says 172.21.15.57:41952) 2023-10-30T14:44:41.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.806+0000 7f38bf7fe640 1 -- 172.21.15.57:0/1509959858 learned_addr learned my addr 172.21.15.57:0/1509959858 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:44:41.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.806+0000 7f38beffd640 1 --2- 172.21.15.57:0/1509959858 >> v2:172.21.15.57:3300/0 conn(0x7f38c0151c20 0x7f38c00c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:41.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.806+0000 7f38bffff640 1 --2- 172.21.15.57:0/1509959858 >> v2:172.21.15.88:3300/0 conn(0x7f38c00cb8b0 0x7f38c006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:41.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.806+0000 7f38be7fc640 1 -- 172.21.15.57:0/1509959858 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1987932628 0 0) 0x7f38c00c5030 con 0x7f38c0151840 2023-10-30T14:44:41.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.806+0000 7f38be7fc640 1 -- 172.21.15.57:0/1509959858 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a8003530 con 0x7f38c0151840 2023-10-30T14:44:41.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.806+0000 7f38beffd640 1 -- 172.21.15.57:0/1509959858 >> v2:172.21.15.88:3300/0 conn(0x7f38c00cb8b0 msgr2=0x7f38c006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:41.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.806+0000 7f38beffd640 1 --2- 172.21.15.57:0/1509959858 >> v2:172.21.15.88:3300/0 conn(0x7f38c00cb8b0 0x7f38c006b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:41.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.806+0000 7f38beffd640 1 -- 172.21.15.57:0/1509959858 >> v1:172.21.15.57:6789/0 conn(0x7f38c0151840 legacy=0x7f38c0072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:41.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.806+0000 7f38beffd640 1 -- 172.21.15.57:0/1509959858 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c0157df0 con 0x7f38c0151c20 2023-10-30T14:44:41.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.806+0000 7f38beffd640 1 --2- 172.21.15.57:0/1509959858 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38c0151c20 0x7f38c00c49a0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f38c00c6cb0 tx=0x7f38b401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f2c73afa20e9da2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:41.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38be7fc640 1 -- 172.21.15.57:0/1509959858 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b40a8950 con 0x7f38c0151c20 2023-10-30T14:44:41.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38be7fc640 1 -- 172.21.15.57:0/1509959858 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b40a8ab0 con 0x7f38c0151c20 2023-10-30T14:44:41.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38be7fc640 1 -- 172.21.15.57:0/1509959858 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b40a8e40 con 0x7f38c0151c20 2023-10-30T14:44:41.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38c611d640 1 -- 172.21.15.57:0/1509959858 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38c0151c20 msgr2=0x7f38c00c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:41.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38c611d640 1 --2- 172.21.15.57:0/1509959858 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38c0151c20 0x7f38c00c49a0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f38c00c6cb0 tx=0x7f38b401ec50 comp rx=0 tx=0).stop 2023-10-30T14:44:41.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38c611d640 1 -- 172.21.15.57:0/1509959858 shutdown_connections 2023-10-30T14:44:41.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38c611d640 1 --2- 172.21.15.57:0/1509959858 >> v2:172.21.15.88:3300/0 conn(0x7f38c00cb8b0 0x7f38c006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:41.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38c611d640 1 --2- 172.21.15.57:0/1509959858 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38c0151c20 0x7f38c00c49a0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:41.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38c611d640 1 -- 172.21.15.57:0/1509959858 >> 172.21.15.57:0/1509959858 conn(0x7f38c00cac70 msgr2=0x7f38c00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:41.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38c611d640 1 -- 172.21.15.57:0/1509959858 shutdown_connections 2023-10-30T14:44:41.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38c611d640 1 -- 172.21.15.57:0/1509959858 wait complete. 2023-10-30T14:44:41.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38c611d640 1 Processor -- start 2023-10-30T14:44:41.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38c611d640 1 -- start start 2023-10-30T14:44:41.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38c611d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38c00cb8b0 0x7f38c014baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:41.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38c611d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38c0151840 0x7f38c014c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:41.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38c611d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38c0151c20 0x7f38c01476a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:41.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38c611d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c0155b60 con 0x7f38c00cb8b0 2023-10-30T14:44:41.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38c611d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c014f710 con 0x7f38c0151840 2023-10-30T14:44:41.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38c611d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c01ec750 con 0x7f38c0151c20 2023-10-30T14:44:41.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38bf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38c00cb8b0 0x7f38c014baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:41.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38bf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38c00cb8b0 0x7f38c014baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50432/0 (socket says 172.21.15.57:50432) 2023-10-30T14:44:41.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.810+0000 7f38bf7fe640 1 -- 172.21.15.57:0/3112999603 learned_addr learned my addr 172.21.15.57:0/3112999603 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:41.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.814+0000 7f38beffd640 1 --2- 172.21.15.57:0/3112999603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38c0151840 0x7f38c014c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:41.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.814+0000 7f38bffff640 1 --2- 172.21.15.57:0/3112999603 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38c0151c20 0x7f38c01476a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:41.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.814+0000 7f38bf7fe640 1 -- 172.21.15.57:0/3112999603 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38c0151c20 msgr2=0x7f38c01476a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:41.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.814+0000 7f38bf7fe640 1 --2- 172.21.15.57:0/3112999603 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38c0151c20 0x7f38c01476a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:41.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.814+0000 7f38bf7fe640 1 -- 172.21.15.57:0/3112999603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38c0151840 msgr2=0x7f38c014c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:41.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.814+0000 7f38bf7fe640 1 --2- 172.21.15.57:0/3112999603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38c0151840 0x7f38c014c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:41.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.814+0000 7f38bf7fe640 1 -- 172.21.15.57:0/3112999603 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b4009420 con 0x7f38c00cb8b0 2023-10-30T14:44:41.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.814+0000 7f38bf7fe640 1 --2- 172.21.15.57:0/3112999603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38c00cb8b0 0x7f38c014baf0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f38b000c500 tx=0x7f38b000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.814+0000 7f38a7fff640 1 -- 172.21.15.57:0/3112999603 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b0005cf0 con 0x7f38c00cb8b0 2023-10-30T14:44:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.814+0000 7f38a7fff640 1 -- 172.21.15.57:0/3112999603 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b0005e50 con 0x7f38c00cb8b0 2023-10-30T14:44:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.814+0000 7f38c611d640 1 -- 172.21.15.57:0/3112999603 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c014c130 con 0x7f38c00cb8b0 2023-10-30T14:44:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.814+0000 7f38a7fff640 1 -- 172.21.15.57:0/3112999603 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b00072f0 con 0x7f38c00cb8b0 2023-10-30T14:44:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.814+0000 7f38c611d640 1 -- 172.21.15.57:0/3112999603 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c01eccb0 con 0x7f38c00cb8b0 2023-10-30T14:44:41.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.814+0000 7f38c611d640 1 -- 172.21.15.57:0/3112999603 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c00726f0 con 0x7f38c00cb8b0 2023-10-30T14:44:41.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.814+0000 7f38a7fff640 1 -- 172.21.15.57:0/3112999603 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f38b0007450 con 0x7f38c00cb8b0 2023-10-30T14:44:41.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.814+0000 7f38a7fff640 1 --2- 172.21.15.57:0/3112999603 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f38ac0664a0 0x7f38ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:41.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.818+0000 7f38a7fff640 1 -- 172.21.15.57:0/3112999603 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f38b00839c0 con 0x7f38c00cb8b0 2023-10-30T14:44:41.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.818+0000 7f38beffd640 1 --2- 172.21.15.57:0/3112999603 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f38ac0664a0 0x7f38ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:41.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.818+0000 7f38beffd640 1 --2- 172.21.15.57:0/3112999603 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f38ac0664a0 0x7f38ac068960 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f38b40b2000 tx=0x7f38b4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:41.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:41.822+0000 7f38a7fff640 1 -- 172.21.15.57:0/3112999603 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f38c00726f0 con 0x7f38c00cb8b0 2023-10-30T14:44:42.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:42.010+0000 7f38c611d640 1 -- 172.21.15.57:0/3112999603 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38c00726f0 con 0x7f38c00cb8b0 2023-10-30T14:44:42.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:42.010+0000 7f38a7fff640 1 -- 172.21.15.57:0/3112999603 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+5720 (secure 0 0 0) 0x7f38c00726f0 con 0x7f38c00cb8b0 2023-10-30T14:44:42.013 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:42.013 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:44:37.830720+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:42.014 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 166 2023-10-30T14:44:42.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:42.014+0000 7f38c611d640 1 -- 172.21.15.57:0/3112999603 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f38ac0664a0 msgr2=0x7f38ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:42.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:42.014+0000 7f38c611d640 1 --2- 172.21.15.57:0/3112999603 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f38ac0664a0 0x7f38ac068960 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f38b40b2000 tx=0x7f38b4008040 comp rx=0 tx=0).stop 2023-10-30T14:44:42.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:42.014+0000 7f38c611d640 1 -- 172.21.15.57:0/3112999603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38c00cb8b0 msgr2=0x7f38c014baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:42.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:42.014+0000 7f38c611d640 1 --2- 172.21.15.57:0/3112999603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38c00cb8b0 0x7f38c014baf0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f38b000c500 tx=0x7f38b000c9c0 comp rx=0 tx=0).stop 2023-10-30T14:44:42.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:42.014+0000 7f38c611d640 1 -- 172.21.15.57:0/3112999603 shutdown_connections 2023-10-30T14:44:42.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:42.014+0000 7f38c611d640 1 --2- 172.21.15.57:0/3112999603 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f38ac0664a0 0x7f38ac068960 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:42.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:42.014+0000 7f38c611d640 1 --2- 172.21.15.57:0/3112999603 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38c0151c20 0x7f38c01476a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:42.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:42.014+0000 7f38c611d640 1 --2- 172.21.15.57:0/3112999603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38c0151840 0x7f38c014c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:42.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:42.014+0000 7f38c611d640 1 --2- 172.21.15.57:0/3112999603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38c00cb8b0 0x7f38c014baf0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:42.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:42.014+0000 7f38c611d640 1 -- 172.21.15.57:0/3112999603 >> 172.21.15.57:0/3112999603 conn(0x7f38c00cac70 msgr2=0x7f38c00733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:42.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:42.014+0000 7f38c611d640 1 -- 172.21.15.57:0/3112999603 shutdown_connections 2023-10-30T14:44:42.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:42.014+0000 7f38c611d640 1 -- 172.21.15.57:0/3112999603 wait complete. 2023-10-30T14:44:42.034 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:44:44.035 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f78ab4ea640 1 Processor -- start 2023-10-30T14:44:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f78ab4ea640 1 -- start start 2023-10-30T14:44:44.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f78ab4ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a406b3b0 0x7f78a406b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:44.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f78ab4ea640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a40c7b00 con 0x7f78a4153aa0 2023-10-30T14:44:44.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f78ab4ea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a406bd60 con 0x7f78a415d840 2023-10-30T14:44:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f78ab4ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a4070a50 con 0x7f78a406b3b0 2023-10-30T14:44:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f78a9286640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f78a4153aa0 0x7f78a40701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46520/0 (socket says 172.21.15.57:46520) 2023-10-30T14:44:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f78a9286640 1 -- 172.21.15.57:0/3338362062 learned_addr learned my addr 172.21.15.57:0/3338362062 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:44:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f78a9a87640 1 --2- 172.21.15.57:0/3338362062 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a406b3b0 0x7f78a406b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f789bfff640 1 -- 172.21.15.57:0/3338362062 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316649194 0 0) 0x7f78a40c7b00 con 0x7f78a4153aa0 2023-10-30T14:44:44.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f789bfff640 1 -- 172.21.15.57:0/3338362062 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7890003530 con 0x7f78a4153aa0 2023-10-30T14:44:44.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f789bfff640 1 -- 172.21.15.57:0/3338362062 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4239975540 0 0) 0x7f7890003530 con 0x7f78a4153aa0 2023-10-30T14:44:44.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f789bfff640 1 -- 172.21.15.57:0/3338362062 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a406b3b0 msgr2=0x7f78a406b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:44.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f789bfff640 1 --2- 172.21.15.57:0/3338362062 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a406b3b0 0x7f78a406b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:44.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f789bfff640 1 -- 172.21.15.57:0/3338362062 >> v1:172.21.15.88:6789/0 conn(0x7f78a415d840 legacy=0x7f78a40ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:44.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f789bfff640 1 -- 172.21.15.57:0/3338362062 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a40c2050 con 0x7f78a4153aa0 2023-10-30T14:44:44.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f789bfff640 1 -- 172.21.15.57:0/3338362062 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f78a0001680 con 0x7f78a4153aa0 2023-10-30T14:44:44.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f789bfff640 1 -- 172.21.15.57:0/3338362062 >> v1:172.21.15.57:6789/0 conn(0x7f78a4153aa0 legacy=0x7f78a40701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:44.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f789bfff640 1 --2- 172.21.15.57:0/3338362062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78900039b0 0x7f7890003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:44.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f789bfff640 1 --2- 172.21.15.57:0/3338362062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7890004360 0x7f7890004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:44.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f789bfff640 1 --2- 172.21.15.57:0/3338362062 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a406b3b0 0x7f7890005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:44.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f789bfff640 1 -- 172.21.15.57:0/3338362062 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a4070a50 con 0x7f78900039b0 2023-10-30T14:44:44.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f789bfff640 1 -- 172.21.15.57:0/3338362062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789000b990 con 0x7f7890004360 2023-10-30T14:44:44.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f789bfff640 1 -- 172.21.15.57:0/3338362062 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f789000bb50 con 0x7f78a406b3b0 2023-10-30T14:44:44.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f78a9286640 1 --2- 172.21.15.57:0/3338362062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78900039b0 0x7f7890003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:44.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f78a8a85640 1 --2- 172.21.15.57:0/3338362062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7890004360 0x7f7890004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f78a9a87640 1 --2- 172.21.15.57:0/3338362062 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a406b3b0 0x7f7890005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f78a9286640 1 -- 172.21.15.57:0/3338362062 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a406b3b0 msgr2=0x7f7890005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f78a9286640 1 --2- 172.21.15.57:0/3338362062 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a406b3b0 0x7f7890005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f78a9286640 1 -- 172.21.15.57:0/3338362062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7890004360 msgr2=0x7f7890004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f78a9286640 1 --2- 172.21.15.57:0/3338362062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7890004360 0x7f7890004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:44.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f78a9286640 1 -- 172.21.15.57:0/3338362062 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7890003530 con 0x7f78900039b0 2023-10-30T14:44:44.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.190+0000 7f78a9a87640 1 --2- 172.21.15.57:0/3338362062 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78a406b3b0 0x7f7890005090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:44:44.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.194+0000 7f78a9286640 1 --2- 172.21.15.57:0/3338362062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78900039b0 0x7f7890003d90 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f78a000c190 tx=0x7f78a001cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74f7cc2e9d546b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:44.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.194+0000 7f789bfff640 1 -- 172.21.15.57:0/3338362062 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a0005ae0 con 0x7f78900039b0 2023-10-30T14:44:44.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.194+0000 7f789bfff640 1 -- 172.21.15.57:0/3338362062 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a0005c60 con 0x7f78900039b0 2023-10-30T14:44:44.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.194+0000 7f789bfff640 1 -- 172.21.15.57:0/3338362062 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a00a58e0 con 0x7f78900039b0 2023-10-30T14:44:44.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.194+0000 7f78ab4ea640 1 -- 172.21.15.57:0/3338362062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78900039b0 msgr2=0x7f7890003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:44.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.194+0000 7f78ab4ea640 1 --2- 172.21.15.57:0/3338362062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78900039b0 0x7f7890003d90 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f78a000c190 tx=0x7f78a001cbe0 comp rx=0 tx=0).stop 2023-10-30T14:44:44.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.194+0000 7f78a9a87640 1 -- 172.21.15.57:0/3338362062 reap_dead start 2023-10-30T14:44:44.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.194+0000 7f78ab4ea640 1 -- 172.21.15.57:0/3338362062 shutdown_connections 2023-10-30T14:44:44.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.194+0000 7f78ab4ea640 1 -- 172.21.15.57:0/3338362062 >> 172.21.15.57:0/3338362062 conn(0x7f78a4071fb0 msgr2=0x7f78a40723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:44.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.194+0000 7f78ab4ea640 1 -- 172.21.15.57:0/3338362062 shutdown_connections 2023-10-30T14:44:44.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.194+0000 7f78ab4ea640 1 -- 172.21.15.57:0/3338362062 wait complete. 2023-10-30T14:44:44.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78ab4ea640 1 Processor -- start 2023-10-30T14:44:44.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78ab4ea640 1 -- start start 2023-10-30T14:44:44.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78ab4ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78900039b0 0x7f78a414f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:44.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78ab4ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78a40cc3a0 0x7f78a41f2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:44.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78ab4ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78a41f2760 0x7f78a40c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:44.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78ab4ea640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a4158520 con 0x7f78a41f2760 2023-10-30T14:44:44.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78ab4ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a41efbd0 con 0x7f78a40cc3a0 2023-10-30T14:44:44.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78ab4ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a4157f80 con 0x7f78900039b0 2023-10-30T14:44:44.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78a9a87640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78a41f2760 0x7f78a40c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:44.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78a9a87640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78a41f2760 0x7f78a40c66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43180/0 (socket says 172.21.15.57:43180) 2023-10-30T14:44:44.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78a8a85640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78a40cc3a0 0x7f78a41f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:44.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78a9a87640 1 -- 172.21.15.57:0/1943756747 learned_addr learned my addr 172.21.15.57:0/1943756747 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:44.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78a9286640 1 --2- 172.21.15.57:0/1943756747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78900039b0 0x7f78a414f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:44.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78a9a87640 1 -- 172.21.15.57:0/1943756747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78900039b0 msgr2=0x7f78a414f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:44.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78a9a87640 1 --2- 172.21.15.57:0/1943756747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78900039b0 0x7f78a414f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:44.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78a9a87640 1 -- 172.21.15.57:0/1943756747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78a40cc3a0 msgr2=0x7f78a41f2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:44.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78a9a87640 1 --2- 172.21.15.57:0/1943756747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78a40cc3a0 0x7f78a41f2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:44.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78a9a87640 1 -- 172.21.15.57:0/1943756747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a4158240 con 0x7f78a41f2760 2023-10-30T14:44:44.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78a9a87640 1 --2- 172.21.15.57:0/1943756747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78a41f2760 0x7f78a40c66e0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f78a4071870 tx=0x7f789c010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:44.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f7899ffb640 1 -- 172.21.15.57:0/1943756747 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789c01b070 con 0x7f78a41f2760 2023-10-30T14:44:44.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f7899ffb640 1 -- 172.21.15.57:0/1943756747 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789c00f040 con 0x7f78a41f2760 2023-10-30T14:44:44.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f7899ffb640 1 -- 172.21.15.57:0/1943756747 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789c00e050 con 0x7f78a41f2760 2023-10-30T14:44:44.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.198+0000 7f78ab4ea640 1 -- 172.21.15.57:0/1943756747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a41eee10 con 0x7f78a41f2760 2023-10-30T14:44:44.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.202+0000 7f78ab4ea640 1 -- 172.21.15.57:0/1943756747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a41ef3d0 con 0x7f78a41f2760 2023-10-30T14:44:44.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.202+0000 7f78ab4ea640 1 -- 172.21.15.57:0/1943756747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a404e170 con 0x7f78a41f2760 2023-10-30T14:44:44.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.202+0000 7f7899ffb640 1 -- 172.21.15.57:0/1943756747 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f789c0039a0 con 0x7f78a41f2760 2023-10-30T14:44:44.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.202+0000 7f7899ffb640 1 --2- 172.21.15.57:0/1943756747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f78880664d0 0x7f7888068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:44.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.202+0000 7f7899ffb640 1 -- 172.21.15.57:0/1943756747 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f789c084830 con 0x7f78a41f2760 2023-10-30T14:44:44.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.202+0000 7f78a9286640 1 --2- 172.21.15.57:0/1943756747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f78880664d0 0x7f7888068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:44.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.206+0000 7f78a9286640 1 --2- 172.21.15.57:0/1943756747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f78880664d0 0x7f7888068990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f78a0005fd0 tx=0x7f78a00a7070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:44.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.206+0000 7f7899ffb640 1 -- 172.21.15.57:0/1943756747 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f789c0515a0 con 0x7f78a41f2760 2023-10-30T14:44:44.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.398+0000 7f78ab4ea640 1 -- 172.21.15.57:0/1943756747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78a414e540 con 0x7f78a41f2760 2023-10-30T14:44:44.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.398+0000 7f7899ffb640 1 -- 172.21.15.57:0/1943756747 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+5717 (secure 0 0 0) 0x7f789c054bc0 con 0x7f78a41f2760 2023-10-30T14:44:44.402 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:44.402 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:44:43.308225+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:44.402 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 167 2023-10-30T14:44:44.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.402+0000 7f78ab4ea640 1 -- 172.21.15.57:0/1943756747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f78880664d0 msgr2=0x7f7888068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:44.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.402+0000 7f78ab4ea640 1 --2- 172.21.15.57:0/1943756747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f78880664d0 0x7f7888068990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f78a0005fd0 tx=0x7f78a00a7070 comp rx=0 tx=0).stop 2023-10-30T14:44:44.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.402+0000 7f78ab4ea640 1 -- 172.21.15.57:0/1943756747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78a41f2760 msgr2=0x7f78a40c66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:44.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.402+0000 7f78ab4ea640 1 --2- 172.21.15.57:0/1943756747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78a41f2760 0x7f78a40c66e0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f78a4071870 tx=0x7f789c010590 comp rx=0 tx=0).stop 2023-10-30T14:44:44.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.402+0000 7f78ab4ea640 1 -- 172.21.15.57:0/1943756747 shutdown_connections 2023-10-30T14:44:44.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.402+0000 7f78ab4ea640 1 --2- 172.21.15.57:0/1943756747 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f78880664d0 0x7f7888068990 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:44.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.402+0000 7f78ab4ea640 1 --2- 172.21.15.57:0/1943756747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78a41f2760 0x7f78a40c66e0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:44.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.402+0000 7f78ab4ea640 1 --2- 172.21.15.57:0/1943756747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78a40cc3a0 0x7f78a41f2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:44.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.402+0000 7f78ab4ea640 1 --2- 172.21.15.57:0/1943756747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78900039b0 0x7f78a414f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:44.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.402+0000 7f78ab4ea640 1 -- 172.21.15.57:0/1943756747 >> 172.21.15.57:0/1943756747 conn(0x7f78a4071fb0 msgr2=0x7f78a40733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:44.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.402+0000 7f78ab4ea640 1 -- 172.21.15.57:0/1943756747 shutdown_connections 2023-10-30T14:44:44.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:44.402+0000 7f78ab4ea640 1 -- 172.21.15.57:0/1943756747 wait complete. 2023-10-30T14:44:44.423 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:44:46.423 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:46.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.578+0000 7f5c73446640 1 Processor -- start 2023-10-30T14:44:46.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.578+0000 7f5c73446640 1 -- start start 2023-10-30T14:44:46.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.578+0000 7f5c73446640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5c6c06bb40 0x7f5c6c157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:46.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.578+0000 7f5c73446640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c6c071e00 con 0x7f5c6c06b760 2023-10-30T14:44:46.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.578+0000 7f5c73446640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c6c070020 con 0x7f5c6c06c420 2023-10-30T14:44:46.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.578+0000 7f5c73446640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c6c070f10 con 0x7f5c6c06bb40 2023-10-30T14:44:46.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.578+0000 7f5c709e1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5c6c06bb40 0x7f5c6c157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:46.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.578+0000 7f5c711e2640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f5c6c06b760 0x7f5c6c15e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46526/0 (socket says 172.21.15.57:46526) 2023-10-30T14:44:46.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.578+0000 7f5c711e2640 1 -- 172.21.15.57:0/2433079953 learned_addr learned my addr 172.21.15.57:0/2433079953 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:44:46.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.578+0000 7f5c709e1640 1 -- 172.21.15.57:0/2433079953 >> v1:172.21.15.88:6789/0 conn(0x7f5c6c06c420 legacy=0x7f5c6c0c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:46.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.578+0000 7f5c709e1640 1 -- 172.21.15.57:0/2433079953 >> v1:172.21.15.57:6789/0 conn(0x7f5c6c06b760 legacy=0x7f5c6c15e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:46.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.578+0000 7f5c709e1640 1 -- 172.21.15.57:0/2433079953 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6c0711f0 con 0x7f5c6c06bb40 2023-10-30T14:44:46.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.582+0000 7f5c709e1640 1 --2- 172.21.15.57:0/2433079953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c6c06bb40 0x7f5c6c157a20 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c0097c0 tx=0x7f5c5c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=539e58ba33303d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:46.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.582+0000 7f5c63fff640 1 -- 172.21.15.57:0/2433079953 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c0ab650 con 0x7f5c6c06bb40 2023-10-30T14:44:46.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.582+0000 7f5c63fff640 1 -- 172.21.15.57:0/2433079953 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c5c0ab7b0 con 0x7f5c6c06bb40 2023-10-30T14:44:46.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.582+0000 7f5c63fff640 1 -- 172.21.15.57:0/2433079953 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c0abb40 con 0x7f5c6c06bb40 2023-10-30T14:44:46.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.582+0000 7f5c73446640 1 -- 172.21.15.57:0/2433079953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c6c06bb40 msgr2=0x7f5c6c157a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:46.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.582+0000 7f5c73446640 1 --2- 172.21.15.57:0/2433079953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c6c06bb40 0x7f5c6c157a20 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c0097c0 tx=0x7f5c5c01c450 comp rx=0 tx=0).stop 2023-10-30T14:44:46.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.582+0000 7f5c73446640 1 -- 172.21.15.57:0/2433079953 shutdown_connections 2023-10-30T14:44:46.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.582+0000 7f5c73446640 1 --2- 172.21.15.57:0/2433079953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c6c06bb40 0x7f5c6c157a20 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:46.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.582+0000 7f5c73446640 1 -- 172.21.15.57:0/2433079953 >> 172.21.15.57:0/2433079953 conn(0x7f5c6c0cac70 msgr2=0x7f5c6c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:46.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.582+0000 7f5c73446640 1 -- 172.21.15.57:0/2433079953 shutdown_connections 2023-10-30T14:44:46.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.582+0000 7f5c73446640 1 -- 172.21.15.57:0/2433079953 wait complete. 2023-10-30T14:44:46.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c73446640 1 Processor -- start 2023-10-30T14:44:46.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c73446640 1 -- start start 2023-10-30T14:44:46.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c73446640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c6c06b760 0x7f5c6c14f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:46.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c73446640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c6c06bb40 0x7f5c6c0c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:46.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c73446640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c6c06c420 0x7f5c6c1f9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:46.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c73446640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c0c3a20 con 0x7f5c6c06b760 2023-10-30T14:44:46.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c73446640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c0c3130 con 0x7f5c6c06bb40 2023-10-30T14:44:46.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c73446640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c152750 con 0x7f5c6c06c420 2023-10-30T14:44:46.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c711e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c6c06b760 0x7f5c6c14f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:46.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c711e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c6c06b760 0x7f5c6c14f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43202/0 (socket says 172.21.15.57:43202) 2023-10-30T14:44:46.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c711e2640 1 -- 172.21.15.57:0/1076281781 learned_addr learned my addr 172.21.15.57:0/1076281781 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:46.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c709e1640 1 --2- 172.21.15.57:0/1076281781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c6c06bb40 0x7f5c6c0c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:46.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c719e3640 1 --2- 172.21.15.57:0/1076281781 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c6c06c420 0x7f5c6c1f9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:46.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c711e2640 1 -- 172.21.15.57:0/1076281781 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c6c06c420 msgr2=0x7f5c6c1f9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:46.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c711e2640 1 --2- 172.21.15.57:0/1076281781 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c6c06c420 0x7f5c6c1f9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:46.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c711e2640 1 -- 172.21.15.57:0/1076281781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c6c06bb40 msgr2=0x7f5c6c0c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:46.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c711e2640 1 --2- 172.21.15.57:0/1076281781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c6c06bb40 0x7f5c6c0c3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:46.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c711e2640 1 -- 172.21.15.57:0/1076281781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5c009420 con 0x7f5c6c06b760 2023-10-30T14:44:46.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c709e1640 1 --2- 172.21.15.57:0/1076281781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c6c06bb40 0x7f5c6c0c3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:44:46.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c711e2640 1 --2- 172.21.15.57:0/1076281781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c6c06b760 0x7f5c6c14f0a0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f5c6800c500 tx=0x7f5c6800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:46.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c61ffb640 1 -- 172.21.15.57:0/1076281781 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c68005cf0 con 0x7f5c6c06b760 2023-10-30T14:44:46.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c61ffb640 1 -- 172.21.15.57:0/1076281781 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c68005e50 con 0x7f5c6c06b760 2023-10-30T14:44:46.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c61ffb640 1 -- 172.21.15.57:0/1076281781 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c680072f0 con 0x7f5c6c06b760 2023-10-30T14:44:46.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c73446640 1 -- 172.21.15.57:0/1076281781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c6c1faab0 con 0x7f5c6c06b760 2023-10-30T14:44:46.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c73446640 1 -- 172.21.15.57:0/1076281781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c6c0c36b0 con 0x7f5c6c06b760 2023-10-30T14:44:46.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.586+0000 7f5c73446640 1 -- 172.21.15.57:0/1076281781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c6c04e170 con 0x7f5c6c06b760 2023-10-30T14:44:46.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.590+0000 7f5c61ffb640 1 -- 172.21.15.57:0/1076281781 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5c68007450 con 0x7f5c6c06b760 2023-10-30T14:44:46.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.590+0000 7f5c61ffb640 1 --2- 172.21.15.57:0/1076281781 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5c500664a0 0x7f5c50068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:46.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.590+0000 7f5c61ffb640 1 -- 172.21.15.57:0/1076281781 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f5c680839c0 con 0x7f5c6c06b760 2023-10-30T14:44:46.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.590+0000 7f5c709e1640 1 --2- 172.21.15.57:0/1076281781 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5c500664a0 0x7f5c50068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:46.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.590+0000 7f5c709e1640 1 --2- 172.21.15.57:0/1076281781 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5c500664a0 0x7f5c50068960 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c0af000 tx=0x7f5c5c0a7000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:46.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.594+0000 7f5c61ffb640 1 -- 172.21.15.57:0/1076281781 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5c68050900 con 0x7f5c6c06b760 2023-10-30T14:44:46.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.786+0000 7f5c73446640 1 -- 172.21.15.57:0/1076281781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c6c0c5930 con 0x7f5c6c06b760 2023-10-30T14:44:46.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.786+0000 7f5c61ffb640 1 -- 172.21.15.57:0/1076281781 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+5717 (secure 0 0 0) 0x7f5c68017090 con 0x7f5c6c06b760 2023-10-30T14:44:46.788 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:46.789 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:44:43.308225+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:46.790 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 167 2023-10-30T14:44:46.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.786+0000 7f5c73446640 1 -- 172.21.15.57:0/1076281781 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5c500664a0 msgr2=0x7f5c50068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:46.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.786+0000 7f5c73446640 1 --2- 172.21.15.57:0/1076281781 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5c500664a0 0x7f5c50068960 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c0af000 tx=0x7f5c5c0a7000 comp rx=0 tx=0).stop 2023-10-30T14:44:46.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.786+0000 7f5c73446640 1 -- 172.21.15.57:0/1076281781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c6c06b760 msgr2=0x7f5c6c14f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:46.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.786+0000 7f5c73446640 1 --2- 172.21.15.57:0/1076281781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c6c06b760 0x7f5c6c14f0a0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f5c6800c500 tx=0x7f5c6800c9c0 comp rx=0 tx=0).stop 2023-10-30T14:44:46.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.786+0000 7f5c73446640 1 -- 172.21.15.57:0/1076281781 shutdown_connections 2023-10-30T14:44:46.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.786+0000 7f5c73446640 1 --2- 172.21.15.57:0/1076281781 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5c500664a0 0x7f5c50068960 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:46.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.786+0000 7f5c73446640 1 --2- 172.21.15.57:0/1076281781 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c6c06c420 0x7f5c6c1f9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:46.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.786+0000 7f5c73446640 1 --2- 172.21.15.57:0/1076281781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c6c06bb40 0x7f5c6c0c3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:46.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.786+0000 7f5c73446640 1 --2- 172.21.15.57:0/1076281781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c6c06b760 0x7f5c6c14f0a0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:46.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.786+0000 7f5c73446640 1 -- 172.21.15.57:0/1076281781 >> 172.21.15.57:0/1076281781 conn(0x7f5c6c0cac70 msgr2=0x7f5c6c1518f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:46.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.786+0000 7f5c73446640 1 -- 172.21.15.57:0/1076281781 shutdown_connections 2023-10-30T14:44:46.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:46.786+0000 7f5c73446640 1 -- 172.21.15.57:0/1076281781 wait complete. 2023-10-30T14:44:46.810 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 167, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 167, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 44 }, "gid_7536": { "addr": "172.21.15.57:6837/816502245", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 816502245, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 816502245, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7536, "incarnation": 164, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:rejoin", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:44:43.308225+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7536, "mds_1": 6747 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.57:6834/697507447", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 697507447, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 697507447, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 156, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7377, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:44:48.811 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:48.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.958+0000 7fe6aa8d6640 1 Processor -- start 2023-10-30T14:44:48.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.958+0000 7fe6aa8d6640 1 -- start start 2023-10-30T14:44:48.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.958+0000 7fe6aa8d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe6a4153aa0 0x7fe6a415d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:48.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.958+0000 7fe6aa8d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6a406b3b0 0x7fe6a406b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:48.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.958+0000 7fe6aa8d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a40c7b00 con 0x7fe6a40701e0 2023-10-30T14:44:48.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.958+0000 7fe6aa8d6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6a4070a50 con 0x7fe6a4153aa0 2023-10-30T14:44:48.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.958+0000 7fe6aa8d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a415dd80 con 0x7fe6a406b3b0 2023-10-30T14:44:48.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.958+0000 7fe6a8e73640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6a406b3b0 0x7fe6a406b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:48.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.958+0000 7fe6a3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe6a4153aa0 0x7fe6a415d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:48.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.958+0000 7fe6a8e73640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6a406b3b0 0x7fe6a406b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:57542/0 (socket says 172.21.15.57:57542) 2023-10-30T14:44:48.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.958+0000 7fe6a8e73640 1 -- 172.21.15.57:0/734464548 learned_addr learned my addr 172.21.15.57:0/734464548 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:48.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.958+0000 7fe6a8e73640 1 -- 172.21.15.57:0/734464548 >> v2:172.21.15.88:3300/0 conn(0x7fe6a4153aa0 msgr2=0x7fe6a415d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:48.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.958+0000 7fe6a8e73640 1 --2- 172.21.15.57:0/734464548 >> v2:172.21.15.88:3300/0 conn(0x7fe6a4153aa0 0x7fe6a415d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.958+0000 7fe6a8e73640 1 -- 172.21.15.57:0/734464548 >> v1:172.21.15.88:6789/0 conn(0x7fe6a40701e0 legacy=0x7fe6a40ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.958+0000 7fe6a8e73640 1 -- 172.21.15.57:0/734464548 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a406bd60 con 0x7fe6a406b3b0 2023-10-30T14:44:48.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.962+0000 7fe6a8e73640 1 --2- 172.21.15.57:0/734464548 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6a406b3b0 0x7fe6a406b820 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe69c009e50 tx=0x7fe69c01f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d4e10968b131e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:48.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.962+0000 7fe6a2ffd640 1 -- 172.21.15.57:0/734464548 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c0af400 con 0x7fe6a406b3b0 2023-10-30T14:44:48.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.962+0000 7fe6a2ffd640 1 -- 172.21.15.57:0/734464548 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69c0ab050 con 0x7fe6a406b3b0 2023-10-30T14:44:48.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.962+0000 7fe6a2ffd640 1 -- 172.21.15.57:0/734464548 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c0af780 con 0x7fe6a406b3b0 2023-10-30T14:44:48.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.962+0000 7fe6aa8d6640 1 -- 172.21.15.57:0/734464548 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6a406b3b0 msgr2=0x7fe6a406b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:48.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.962+0000 7fe6aa8d6640 1 --2- 172.21.15.57:0/734464548 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6a406b3b0 0x7fe6a406b820 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe69c009e50 tx=0x7fe69c01f1f0 comp rx=0 tx=0).stop 2023-10-30T14:44:48.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.962+0000 7fe6aa8d6640 1 -- 172.21.15.57:0/734464548 shutdown_connections 2023-10-30T14:44:48.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.962+0000 7fe6aa8d6640 1 --2- 172.21.15.57:0/734464548 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6a406b3b0 0x7fe6a406b820 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:48.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.962+0000 7fe6aa8d6640 1 --2- 172.21.15.57:0/734464548 >> v2:172.21.15.88:3300/0 conn(0x7fe6a4153aa0 0x7fe6a415d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:48.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.962+0000 7fe6aa8d6640 1 -- 172.21.15.57:0/734464548 >> 172.21.15.57:0/734464548 conn(0x7fe6a4071fb0 msgr2=0x7fe6a40723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:48.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.962+0000 7fe6aa8d6640 1 -- 172.21.15.57:0/734464548 shutdown_connections 2023-10-30T14:44:48.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.962+0000 7fe6aa8d6640 1 -- 172.21.15.57:0/734464548 wait complete. 2023-10-30T14:44:48.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6aa8d6640 1 Processor -- start 2023-10-30T14:44:48.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6aa8d6640 1 -- start start 2023-10-30T14:44:48.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6aa8d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6a406b3b0 0x7fe6a4157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:48.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6aa8d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6a40701e0 0x7fe6a4158410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:48.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6aa8d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6a4153aa0 0x7fe6a4149850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:48.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6aa8d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a4157a40 con 0x7fe6a406b3b0 2023-10-30T14:44:48.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6aa8d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a4159420 con 0x7fe6a4153aa0 2023-10-30T14:44:48.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6aa8d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a41596c0 con 0x7fe6a40701e0 2023-10-30T14:44:48.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6a3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6a406b3b0 0x7fe6a4157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:48.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6a3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6a406b3b0 0x7fe6a4157ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43218/0 (socket says 172.21.15.57:43218) 2023-10-30T14:44:48.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6a3fff640 1 -- 172.21.15.57:0/4077507777 learned_addr learned my addr 172.21.15.57:0/4077507777 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:48.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6a8e73640 1 --2- 172.21.15.57:0/4077507777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6a4153aa0 0x7fe6a4149850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:48.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6a37fe640 1 --2- 172.21.15.57:0/4077507777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6a40701e0 0x7fe6a4158410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:48.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6a3fff640 1 -- 172.21.15.57:0/4077507777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6a40701e0 msgr2=0x7fe6a4158410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:48.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6a3fff640 1 --2- 172.21.15.57:0/4077507777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6a40701e0 0x7fe6a4158410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:48.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6a3fff640 1 -- 172.21.15.57:0/4077507777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6a4153aa0 msgr2=0x7fe6a4149850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:48.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6a3fff640 1 --2- 172.21.15.57:0/4077507777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6a4153aa0 0x7fe6a4149850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:48.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6a3fff640 1 -- 172.21.15.57:0/4077507777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69c009b30 con 0x7fe6a406b3b0 2023-10-30T14:44:48.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6a37fe640 1 --2- 172.21.15.57:0/4077507777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6a40701e0 0x7fe6a4158410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:44:48.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6a3fff640 1 --2- 172.21.15.57:0/4077507777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6a406b3b0 0x7fe6a4157ed0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fe69400e870 tx=0x7fe69400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:48.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6a0ff9640 1 -- 172.21.15.57:0/4077507777 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe694009510 con 0x7fe6a406b3b0 2023-10-30T14:44:48.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6a0ff9640 1 -- 172.21.15.57:0/4077507777 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69400eec0 con 0x7fe6a406b3b0 2023-10-30T14:44:48.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6a0ff9640 1 -- 172.21.15.57:0/4077507777 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69400cb40 con 0x7fe6a406b3b0 2023-10-30T14:44:48.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6aa8d6640 1 -- 172.21.15.57:0/4077507777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a41599a0 con 0x7fe6a406b3b0 2023-10-30T14:44:48.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.966+0000 7fe6aa8d6640 1 -- 172.21.15.57:0/4077507777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a4159f60 con 0x7fe6a406b3b0 2023-10-30T14:44:48.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.970+0000 7fe6aa8d6640 1 -- 172.21.15.57:0/4077507777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a404e170 con 0x7fe6a406b3b0 2023-10-30T14:44:48.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.970+0000 7fe6a0ff9640 1 -- 172.21.15.57:0/4077507777 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe69400ccc0 con 0x7fe6a406b3b0 2023-10-30T14:44:48.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.970+0000 7fe6a0ff9640 1 --2- 172.21.15.57:0/4077507777 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe6900664a0 0x7fe690068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:48.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.970+0000 7fe6a0ff9640 1 -- 172.21.15.57:0/4077507777 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7fe6940844f0 con 0x7fe6a406b3b0 2023-10-30T14:44:48.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.970+0000 7fe6a37fe640 1 --2- 172.21.15.57:0/4077507777 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe6900664a0 0x7fe690068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:48.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.970+0000 7fe6a37fe640 1 --2- 172.21.15.57:0/4077507777 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe6900664a0 0x7fe690068960 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4156f20 tx=0x7fe698009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:48.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:48.974+0000 7fe6a0ff9640 1 -- 172.21.15.57:0/4077507777 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe694051440 con 0x7fe6a406b3b0 2023-10-30T14:44:49.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:49.162+0000 7fe6aa8d6640 1 -- 172.21.15.57:0/4077507777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6a415a120 con 0x7fe6a406b3b0 2023-10-30T14:44:49.167 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:49.167 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":168,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:44:48.803082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:clientreplay","state_seq":16,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:49.167 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 168 2023-10-30T14:44:49.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:49.166+0000 7fe6a0ff9640 1 -- 172.21.15.57:0/4077507777 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+5723 (secure 0 0 0) 0x7fe694054a60 con 0x7fe6a406b3b0 2023-10-30T14:44:49.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:49.166+0000 7fe6aa8d6640 1 -- 172.21.15.57:0/4077507777 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe6900664a0 msgr2=0x7fe690068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:49.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:49.166+0000 7fe6aa8d6640 1 --2- 172.21.15.57:0/4077507777 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe6900664a0 0x7fe690068960 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4156f20 tx=0x7fe698009040 comp rx=0 tx=0).stop 2023-10-30T14:44:49.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:49.166+0000 7fe6aa8d6640 1 -- 172.21.15.57:0/4077507777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6a406b3b0 msgr2=0x7fe6a4157ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:49.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:49.166+0000 7fe6aa8d6640 1 --2- 172.21.15.57:0/4077507777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6a406b3b0 0x7fe6a4157ed0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fe69400e870 tx=0x7fe69400ec70 comp rx=0 tx=0).stop 2023-10-30T14:44:49.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:49.166+0000 7fe6aa8d6640 1 -- 172.21.15.57:0/4077507777 shutdown_connections 2023-10-30T14:44:49.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:49.166+0000 7fe6aa8d6640 1 --2- 172.21.15.57:0/4077507777 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe6900664a0 0x7fe690068960 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:49.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:49.166+0000 7fe6aa8d6640 1 --2- 172.21.15.57:0/4077507777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6a4153aa0 0x7fe6a4149850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:49.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:49.166+0000 7fe6aa8d6640 1 --2- 172.21.15.57:0/4077507777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6a40701e0 0x7fe6a4158410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:49.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:49.166+0000 7fe6aa8d6640 1 --2- 172.21.15.57:0/4077507777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6a406b3b0 0x7fe6a4157ed0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:49.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:49.166+0000 7fe6aa8d6640 1 -- 172.21.15.57:0/4077507777 >> 172.21.15.57:0/4077507777 conn(0x7fe6a4071fb0 msgr2=0x7fe6a40ccfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:49.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:49.166+0000 7fe6aa8d6640 1 -- 172.21.15.57:0/4077507777 shutdown_connections 2023-10-30T14:44:49.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:49.166+0000 7fe6aa8d6640 1 -- 172.21.15.57:0/4077507777 wait complete. 2023-10-30T14:44:49.359 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:44:51.360 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:51.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.510+0000 7f970f7d3640 1 Processor -- start 2023-10-30T14:44:51.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.510+0000 7f970f7d3640 1 -- start start 2023-10-30T14:44:51.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.514+0000 7f970f7d3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f970806b760 0x7f970815e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:51.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.514+0000 7f970f7d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f970806bb40 0x7f9708157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:51.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.514+0000 7f970f7d3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9708071e00 con 0x7f970806c420 2023-10-30T14:44:51.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.514+0000 7f970f7d3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9708070020 con 0x7f970806b760 2023-10-30T14:44:51.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.514+0000 7f970f7d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970815eb90 con 0x7f970806bb40 2023-10-30T14:44:51.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.514+0000 7f970d56f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f970806b760 0x7f970815e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:51.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.514+0000 7f970dd70640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f970806c420 0x7f97080c57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46538/0 (socket says 172.21.15.57:46538) 2023-10-30T14:44:51.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.514+0000 7f970dd70640 1 -- 172.21.15.57:0/459808651 learned_addr learned my addr 172.21.15.57:0/459808651 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:44:51.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.514+0000 7f970cd6e640 1 --2- 172.21.15.57:0/459808651 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f970806bb40 0x7f9708157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:51.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.514+0000 7f970d56f640 1 -- 172.21.15.57:0/459808651 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f970806bb40 msgr2=0x7f9708157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:51.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.514+0000 7f970d56f640 1 --2- 172.21.15.57:0/459808651 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f970806bb40 0x7f9708157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:51.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.514+0000 7f970d56f640 1 -- 172.21.15.57:0/459808651 >> v1:172.21.15.57:6789/0 conn(0x7f970806c420 legacy=0x7f97080c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:51.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.514+0000 7f970d56f640 1 -- 172.21.15.57:0/459808651 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9708070f10 con 0x7f970806b760 2023-10-30T14:44:51.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.514+0000 7f96fffff640 1 -- 172.21.15.57:0/459808651 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839368073 0 0) 0x7f9708071e00 con 0x7f970806c420 2023-10-30T14:44:51.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.518+0000 7f970d56f640 1 --2- 172.21.15.57:0/459808651 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f970806b760 0x7f970815e650 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f97040097c0 tx=0x7f970401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e86294219e66cb1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:51.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.518+0000 7f96fffff640 1 -- 172.21.15.57:0/459808651 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97040ab650 con 0x7f970806b760 2023-10-30T14:44:51.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.518+0000 7f96fffff640 1 -- 172.21.15.57:0/459808651 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97040ab7b0 con 0x7f970806b760 2023-10-30T14:44:51.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.518+0000 7f96fffff640 1 -- 172.21.15.57:0/459808651 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97040abb40 con 0x7f970806b760 2023-10-30T14:44:51.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.518+0000 7f970f7d3640 1 -- 172.21.15.57:0/459808651 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f970806b760 msgr2=0x7f970815e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:51.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.518+0000 7f970f7d3640 1 --2- 172.21.15.57:0/459808651 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f970806b760 0x7f970815e650 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f97040097c0 tx=0x7f970401c450 comp rx=0 tx=0).stop 2023-10-30T14:44:51.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.518+0000 7f970f7d3640 1 -- 172.21.15.57:0/459808651 shutdown_connections 2023-10-30T14:44:51.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.518+0000 7f970f7d3640 1 --2- 172.21.15.57:0/459808651 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f970806bb40 0x7f9708157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:51.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.518+0000 7f970f7d3640 1 --2- 172.21.15.57:0/459808651 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f970806b760 0x7f970815e650 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:51.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.518+0000 7f970f7d3640 1 -- 172.21.15.57:0/459808651 >> 172.21.15.57:0/459808651 conn(0x7f97080cac70 msgr2=0x7f97080cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:51.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.518+0000 7f970f7d3640 1 -- 172.21.15.57:0/459808651 shutdown_connections 2023-10-30T14:44:51.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.518+0000 7f970f7d3640 1 -- 172.21.15.57:0/459808651 wait complete. 2023-10-30T14:44:51.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.518+0000 7f970f7d3640 1 Processor -- start 2023-10-30T14:44:51.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.518+0000 7f970f7d3640 1 -- start start 2023-10-30T14:44:51.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970f7d3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f970806b760 0x7f9708149fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:51.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970f7d3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f970806bb40 0x7f970814ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:51.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970f7d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f970806c420 0x7f970814e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:51.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970f7d3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970814f4f0 con 0x7f970806bb40 2023-10-30T14:44:51.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970f7d3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970814df00 con 0x7f970806b760 2023-10-30T14:44:51.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970f7d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970814f210 con 0x7f970806c420 2023-10-30T14:44:51.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970cd6e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f970806bb40 0x7f970814ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:51.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970cd6e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f970806bb40 0x7f970814ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43228/0 (socket says 172.21.15.57:43228) 2023-10-30T14:44:51.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970cd6e640 1 -- 172.21.15.57:0/806826676 learned_addr learned my addr 172.21.15.57:0/806826676 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:51.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970dd70640 1 --2- 172.21.15.57:0/806826676 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f970806c420 0x7f970814e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:51.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970d56f640 1 --2- 172.21.15.57:0/806826676 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f970806b760 0x7f9708149fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:51.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970cd6e640 1 -- 172.21.15.57:0/806826676 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f970806c420 msgr2=0x7f970814e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:51.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970cd6e640 1 --2- 172.21.15.57:0/806826676 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f970806c420 0x7f970814e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:51.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970cd6e640 1 -- 172.21.15.57:0/806826676 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f970806b760 msgr2=0x7f9708149fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:51.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970cd6e640 1 --2- 172.21.15.57:0/806826676 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f970806b760 0x7f9708149fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:51.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970cd6e640 1 -- 172.21.15.57:0/806826676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9704009420 con 0x7f970806bb40 2023-10-30T14:44:51.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970cd6e640 1 --2- 172.21.15.57:0/806826676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f970806bb40 0x7f970814ebd0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f96f800d800 tx=0x7f96f800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:51.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f96fdffb640 1 -- 172.21.15.57:0/806826676 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f8018070 con 0x7f970806bb40 2023-10-30T14:44:51.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f96fdffb640 1 -- 172.21.15.57:0/806826676 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f8004020 con 0x7f970806bb40 2023-10-30T14:44:51.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f96fdffb640 1 -- 172.21.15.57:0/806826676 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f80041d0 con 0x7f970806bb40 2023-10-30T14:44:51.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970f7d3640 1 -- 172.21.15.57:0/806826676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f970814d5d0 con 0x7f970806bb40 2023-10-30T14:44:51.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970f7d3640 1 -- 172.21.15.57:0/806826676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f970814db90 con 0x7f970806bb40 2023-10-30T14:44:51.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f970f7d3640 1 -- 172.21.15.57:0/806826676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f970806c190 con 0x7f970806bb40 2023-10-30T14:44:51.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.522+0000 7f96fdffb640 1 -- 172.21.15.57:0/806826676 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f96f8004330 con 0x7f970806bb40 2023-10-30T14:44:51.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.526+0000 7f96fdffb640 1 --2- 172.21.15.57:0/806826676 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f96ec0664d0 0x7f96ec068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:51.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.526+0000 7f96fdffb640 1 -- 172.21.15.57:0/806826676 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f96f8084360 con 0x7f970806bb40 2023-10-30T14:44:51.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.526+0000 7f970d56f640 1 --2- 172.21.15.57:0/806826676 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f96ec0664d0 0x7f96ec068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:51.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.526+0000 7f970d56f640 1 --2- 172.21.15.57:0/806826676 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f96ec0664d0 0x7f96ec068990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f97040af000 tx=0x7f97040a7000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:51.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.530+0000 7f96fdffb640 1 -- 172.21.15.57:0/806826676 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f96f8051160 con 0x7f970806bb40 2023-10-30T14:44:51.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.718+0000 7f970f7d3640 1 -- 172.21.15.57:0/806826676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f970804e170 con 0x7f970806bb40 2023-10-30T14:44:51.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.718+0000 7f96fdffb640 1 -- 172.21.15.57:0/806826676 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f96f8054780 con 0x7f970806bb40 2023-10-30T14:44:51.721 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:51.721 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":168,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:44:48.803082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:clientreplay","state_seq":16,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:51.722 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 168 2023-10-30T14:44:51.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.722+0000 7f970f7d3640 1 -- 172.21.15.57:0/806826676 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f96ec0664d0 msgr2=0x7f96ec068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:51.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.722+0000 7f970f7d3640 1 --2- 172.21.15.57:0/806826676 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f96ec0664d0 0x7f96ec068990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f97040af000 tx=0x7f97040a7000 comp rx=0 tx=0).stop 2023-10-30T14:44:51.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.722+0000 7f970f7d3640 1 -- 172.21.15.57:0/806826676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f970806bb40 msgr2=0x7f970814ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:51.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.722+0000 7f970f7d3640 1 --2- 172.21.15.57:0/806826676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f970806bb40 0x7f970814ebd0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f96f800d800 tx=0x7f96f800dcc0 comp rx=0 tx=0).stop 2023-10-30T14:44:51.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.722+0000 7f970f7d3640 1 -- 172.21.15.57:0/806826676 shutdown_connections 2023-10-30T14:44:51.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.722+0000 7f970f7d3640 1 --2- 172.21.15.57:0/806826676 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f96ec0664d0 0x7f96ec068990 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:51.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.722+0000 7f970f7d3640 1 --2- 172.21.15.57:0/806826676 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f970806c420 0x7f970814e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:51.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.722+0000 7f970f7d3640 1 --2- 172.21.15.57:0/806826676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f970806bb40 0x7f970814ebd0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:51.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.722+0000 7f970f7d3640 1 --2- 172.21.15.57:0/806826676 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f970806b760 0x7f9708149fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:51.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.722+0000 7f970f7d3640 1 -- 172.21.15.57:0/806826676 >> 172.21.15.57:0/806826676 conn(0x7f97080cac70 msgr2=0x7f97081520c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:51.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.722+0000 7f970f7d3640 1 -- 172.21.15.57:0/806826676 shutdown_connections 2023-10-30T14:44:51.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:51.722+0000 7f970f7d3640 1 -- 172.21.15.57:0/806826676 wait complete. 2023-10-30T14:44:51.742 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:44:53.744 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:53.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.906+0000 7f029b6d6640 1 Processor -- start 2023-10-30T14:44:53.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.906+0000 7f029b6d6640 1 -- start start 2023-10-30T14:44:53.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.906+0000 7f029b6d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0294151840 0x7f02940c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:53.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.906+0000 7f029b6d6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0294151c20 0x7f0294072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:53.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.906+0000 7f029b6d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f02940cb8b0 0x7f0294072540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:53.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.906+0000 7f029b6d6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02940c5030 con 0x7f0294151c20 2023-10-30T14:44:53.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.906+0000 7f029b6d6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0294157830 con 0x7f02940cb8b0 2023-10-30T14:44:53.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.906+0000 7f029b6d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0294157b10 con 0x7f0294151840 2023-10-30T14:44:53.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.906+0000 7f0298c71640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0294151c20 0x7f0294072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:53.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.906+0000 7f0298c71640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0294151c20 0x7f0294072000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53696/0 (socket says 172.21.15.57:53696) 2023-10-30T14:44:53.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.906+0000 7f0298c71640 1 -- 172.21.15.57:0/2398827936 learned_addr learned my addr 172.21.15.57:0/2398827936 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:53.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.906+0000 7f0299c73640 1 --2- 172.21.15.57:0/2398827936 >> v2:172.21.15.88:3300/0 conn(0x7f02940cb8b0 0x7f0294072540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:53.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.906+0000 7f0299472640 1 --2- 172.21.15.57:0/2398827936 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0294151840 0x7f02940c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:53.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.906+0000 7f0298c71640 1 -- 172.21.15.57:0/2398827936 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0294151840 msgr2=0x7f02940c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:53.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.906+0000 7f0298c71640 1 --2- 172.21.15.57:0/2398827936 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0294151840 0x7f02940c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:53.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.906+0000 7f0298c71640 1 -- 172.21.15.57:0/2398827936 >> v2:172.21.15.88:3300/0 conn(0x7f02940cb8b0 msgr2=0x7f0294072540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:53.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.906+0000 7f0298c71640 1 --2- 172.21.15.57:0/2398827936 >> v2:172.21.15.88:3300/0 conn(0x7f02940cb8b0 0x7f0294072540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:53.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.906+0000 7f0298c71640 1 -- 172.21.15.57:0/2398827936 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0294157df0 con 0x7f0294151c20 2023-10-30T14:44:53.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.910+0000 7f0298c71640 1 --2- 172.21.15.57:0/2398827936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0294151c20 0x7f0294072000 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f02940cb820 tx=0x7f028401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70bc41f4ec2d2bb4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:53.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.910+0000 7f028bfff640 1 -- 172.21.15.57:0/2398827936 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02840a89b0 con 0x7f0294151c20 2023-10-30T14:44:53.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.910+0000 7f028bfff640 1 -- 172.21.15.57:0/2398827936 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02840a8b10 con 0x7f0294151c20 2023-10-30T14:44:53.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.910+0000 7f028bfff640 1 -- 172.21.15.57:0/2398827936 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02840a8ea0 con 0x7f0294151c20 2023-10-30T14:44:53.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.910+0000 7f029b6d6640 1 -- 172.21.15.57:0/2398827936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0294151c20 msgr2=0x7f0294072000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:53.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.910+0000 7f029b6d6640 1 --2- 172.21.15.57:0/2398827936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0294151c20 0x7f0294072000 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f02940cb820 tx=0x7f028401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:44:53.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.910+0000 7f029b6d6640 1 -- 172.21.15.57:0/2398827936 shutdown_connections 2023-10-30T14:44:53.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.910+0000 7f029b6d6640 1 --2- 172.21.15.57:0/2398827936 >> v2:172.21.15.88:3300/0 conn(0x7f02940cb8b0 0x7f0294072540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:53.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.910+0000 7f029b6d6640 1 --2- 172.21.15.57:0/2398827936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0294151c20 0x7f0294072000 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:53.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.910+0000 7f029b6d6640 1 --2- 172.21.15.57:0/2398827936 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0294151840 0x7f02940c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:53.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.910+0000 7f029b6d6640 1 -- 172.21.15.57:0/2398827936 >> 172.21.15.57:0/2398827936 conn(0x7f02940cac70 msgr2=0x7f02940cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:53.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.910+0000 7f029b6d6640 1 -- 172.21.15.57:0/2398827936 shutdown_connections 2023-10-30T14:44:53.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.910+0000 7f029b6d6640 1 -- 172.21.15.57:0/2398827936 wait complete. 2023-10-30T14:44:53.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f029b6d6640 1 Processor -- start 2023-10-30T14:44:53.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f029b6d6640 1 -- start start 2023-10-30T14:44:53.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f029b6d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02940cb8b0 0x7f02941f1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:53.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f029b6d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0294151840 0x7f02941ef210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:53.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f029b6d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0294151c20 0x7f02940c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:53.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f029b6d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02941eedf0 con 0x7f0294151840 2023-10-30T14:44:53.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f029b6d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02941f2d10 con 0x7f02940cb8b0 2023-10-30T14:44:53.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f029b6d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02941f2470 con 0x7f0294151c20 2023-10-30T14:44:53.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f0298c71640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0294151840 0x7f02941ef210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:53.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f0298c71640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0294151840 0x7f02941ef210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53702/0 (socket says 172.21.15.57:53702) 2023-10-30T14:44:53.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f0298c71640 1 -- 172.21.15.57:0/3064209990 learned_addr learned my addr 172.21.15.57:0/3064209990 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:53.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f0299472640 1 --2- 172.21.15.57:0/3064209990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02940cb8b0 0x7f02941f1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:53.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f0299c73640 1 --2- 172.21.15.57:0/3064209990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0294151c20 0x7f02940c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:53.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f0298c71640 1 -- 172.21.15.57:0/3064209990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0294151c20 msgr2=0x7f02940c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:53.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f0298c71640 1 --2- 172.21.15.57:0/3064209990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0294151c20 0x7f02940c9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:53.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f0298c71640 1 -- 172.21.15.57:0/3064209990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02940cb8b0 msgr2=0x7f02941f1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:53.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f0298c71640 1 --2- 172.21.15.57:0/3064209990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02940cb8b0 0x7f02941f1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:53.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f0298c71640 1 -- 172.21.15.57:0/3064209990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02840094d0 con 0x7f0294151840 2023-10-30T14:44:53.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f0298c71640 1 --2- 172.21.15.57:0/3064209990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0294151840 0x7f02941ef210 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f028401d3a0 tx=0x7f028401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:53.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f0289ffb640 1 -- 172.21.15.57:0/3064209990 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02840b0d40 con 0x7f0294151840 2023-10-30T14:44:53.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f0289ffb640 1 -- 172.21.15.57:0/3064209990 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02840bb460 con 0x7f0294151840 2023-10-30T14:44:53.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f029b6d6640 1 -- 172.21.15.57:0/3064209990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02941f2750 con 0x7f0294151840 2023-10-30T14:44:53.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f0289ffb640 1 -- 172.21.15.57:0/3064209990 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02840b0ea0 con 0x7f0294151840 2023-10-30T14:44:53.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.914+0000 7f029b6d6640 1 -- 172.21.15.57:0/3064209990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02941eea10 con 0x7f0294151840 2023-10-30T14:44:53.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.918+0000 7f029b6d6640 1 -- 172.21.15.57:0/3064209990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02941ee870 con 0x7f0294151840 2023-10-30T14:44:53.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.918+0000 7f0289ffb640 1 -- 172.21.15.57:0/3064209990 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f02840b83f0 con 0x7f0294151840 2023-10-30T14:44:53.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.918+0000 7f0289ffb640 1 --2- 172.21.15.57:0/3064209990 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f02780664d0 0x7f0278068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:53.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.918+0000 7f0289ffb640 1 -- 172.21.15.57:0/3064209990 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f0284121ca0 con 0x7f0294151840 2023-10-30T14:44:53.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.918+0000 7f0299472640 1 --2- 172.21.15.57:0/3064209990 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f02780664d0 0x7f0278068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:53.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.922+0000 7f0299472640 1 --2- 172.21.15.57:0/3064209990 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f02780664d0 0x7f0278068990 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0290009b60 tx=0x7f0290009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:53.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:53.922+0000 7f0289ffb640 1 -- 172.21.15.57:0/3064209990 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f02941ee870 con 0x7f0294151840 2023-10-30T14:44:54.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:54.166+0000 7f029b6d6640 1 -- 172.21.15.57:0/3064209990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02941ee870 con 0x7f0294151840 2023-10-30T14:44:54.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:54.166+0000 7f0289ffb640 1 -- 172.21.15.57:0/3064209990 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f02941ee870 con 0x7f0294151840 2023-10-30T14:44:54.169 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:54.169 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":168,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:44:48.803082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:clientreplay","state_seq":16,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:54.169 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 168 2023-10-30T14:44:54.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:54.170+0000 7f029b6d6640 1 -- 172.21.15.57:0/3064209990 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f02780664d0 msgr2=0x7f0278068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:54.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:54.170+0000 7f029b6d6640 1 --2- 172.21.15.57:0/3064209990 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f02780664d0 0x7f0278068990 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0290009b60 tx=0x7f0290009190 comp rx=0 tx=0).stop 2023-10-30T14:44:54.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:54.170+0000 7f029b6d6640 1 -- 172.21.15.57:0/3064209990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0294151840 msgr2=0x7f02941ef210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:54.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:54.170+0000 7f029b6d6640 1 --2- 172.21.15.57:0/3064209990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0294151840 0x7f02941ef210 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f028401d3a0 tx=0x7f028401ba90 comp rx=0 tx=0).stop 2023-10-30T14:44:54.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:54.170+0000 7f029b6d6640 1 -- 172.21.15.57:0/3064209990 shutdown_connections 2023-10-30T14:44:54.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:54.170+0000 7f029b6d6640 1 --2- 172.21.15.57:0/3064209990 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f02780664d0 0x7f0278068990 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:54.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:54.170+0000 7f029b6d6640 1 --2- 172.21.15.57:0/3064209990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0294151c20 0x7f02940c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:54.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:54.170+0000 7f029b6d6640 1 --2- 172.21.15.57:0/3064209990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0294151840 0x7f02941ef210 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:54.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:54.170+0000 7f029b6d6640 1 --2- 172.21.15.57:0/3064209990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02940cb8b0 0x7f02941f1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:54.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:54.170+0000 7f029b6d6640 1 -- 172.21.15.57:0/3064209990 >> 172.21.15.57:0/3064209990 conn(0x7f02940cac70 msgr2=0x7f02941e3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:54.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:54.170+0000 7f029b6d6640 1 -- 172.21.15.57:0/3064209990 shutdown_connections 2023-10-30T14:44:54.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:54.170+0000 7f029b6d6640 1 -- 172.21.15.57:0/3064209990 wait complete. 2023-10-30T14:44:54.194 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:44:56.195 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:44:56.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.350+0000 7f865113f640 1 Processor -- start 2023-10-30T14:44:56.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.350+0000 7f865113f640 1 -- start start 2023-10-30T14:44:56.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.350+0000 7f865113f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f864c0c8eb0 0x7f864c15cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:56.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.350+0000 7f865113f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f864c15ac70 0x7f864c15a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:56.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.350+0000 7f865113f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f864c06aeb0 0x7f864c06b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:56.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.350+0000 7f865113f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f864c159800 con 0x7f864c06aeb0 2023-10-30T14:44:56.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.350+0000 7f865113f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f864c15dcc0 con 0x7f864c15ac70 2023-10-30T14:44:56.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.350+0000 7f865113f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864c159d80 con 0x7f864c0c8eb0 2023-10-30T14:44:56.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.350+0000 7f865093e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f864c06aeb0 0x7f864c06b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:56.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.350+0000 7f865093e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f864c06aeb0 0x7f864c06b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53704/0 (socket says 172.21.15.57:53704) 2023-10-30T14:44:56.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.350+0000 7f865093e640 1 -- 172.21.15.57:0/2227928103 learned_addr learned my addr 172.21.15.57:0/2227928103 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:56.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.350+0000 7f864b7fe640 1 --2- 172.21.15.57:0/2227928103 >> v2:172.21.15.88:3300/0 conn(0x7f864c15ac70 0x7f864c15a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:56.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.350+0000 7f864bfff640 1 --2- 172.21.15.57:0/2227928103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f864c0c8eb0 0x7f864c15cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:56.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.350+0000 7f865093e640 1 -- 172.21.15.57:0/2227928103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f864c0c8eb0 msgr2=0x7f864c15cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:56.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.350+0000 7f865093e640 1 --2- 172.21.15.57:0/2227928103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f864c0c8eb0 0x7f864c15cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:56.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.350+0000 7f865093e640 1 -- 172.21.15.57:0/2227928103 >> v2:172.21.15.88:3300/0 conn(0x7f864c15ac70 msgr2=0x7f864c15a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:44:56.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.350+0000 7f865093e640 1 --2- 172.21.15.57:0/2227928103 >> v2:172.21.15.88:3300/0 conn(0x7f864c15ac70 0x7f864c15a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:56.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.350+0000 7f865093e640 1 -- 172.21.15.57:0/2227928103 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c15a750 con 0x7f864c06aeb0 2023-10-30T14:44:56.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.350+0000 7f864bfff640 1 --2- 172.21.15.57:0/2227928103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f864c0c8eb0 0x7f864c15cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:44:56.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.354+0000 7f865093e640 1 --2- 172.21.15.57:0/2227928103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c06aeb0 0x7f864c06b300 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f864c073d00 tx=0x7f864401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f994a35734a2c36 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:56.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.354+0000 7f864affd640 1 -- 172.21.15.57:0/2227928103 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86440af400 con 0x7f864c06aeb0 2023-10-30T14:44:56.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.354+0000 7f864affd640 1 -- 172.21.15.57:0/2227928103 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86440ab050 con 0x7f864c06aeb0 2023-10-30T14:44:56.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.354+0000 7f864affd640 1 -- 172.21.15.57:0/2227928103 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86440af780 con 0x7f864c06aeb0 2023-10-30T14:44:56.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.354+0000 7f865113f640 1 -- 172.21.15.57:0/2227928103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c06aeb0 msgr2=0x7f864c06b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:56.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.354+0000 7f865113f640 1 --2- 172.21.15.57:0/2227928103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c06aeb0 0x7f864c06b300 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f864c073d00 tx=0x7f864401f1c0 comp rx=0 tx=0).stop 2023-10-30T14:44:56.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.354+0000 7f865113f640 1 -- 172.21.15.57:0/2227928103 shutdown_connections 2023-10-30T14:44:56.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.354+0000 7f865113f640 1 --2- 172.21.15.57:0/2227928103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c06aeb0 0x7f864c06b300 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:56.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.354+0000 7f865113f640 1 --2- 172.21.15.57:0/2227928103 >> v2:172.21.15.88:3300/0 conn(0x7f864c15ac70 0x7f864c15a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:56.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.354+0000 7f865113f640 1 --2- 172.21.15.57:0/2227928103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f864c0c8eb0 0x7f864c15cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:56.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.354+0000 7f865113f640 1 -- 172.21.15.57:0/2227928103 >> 172.21.15.57:0/2227928103 conn(0x7f864c0cef00 msgr2=0x7f864c0cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:56.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.354+0000 7f865113f640 1 -- 172.21.15.57:0/2227928103 shutdown_connections 2023-10-30T14:44:56.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.354+0000 7f865113f640 1 -- 172.21.15.57:0/2227928103 wait complete. 2023-10-30T14:44:56.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f865113f640 1 Processor -- start 2023-10-30T14:44:56.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f865113f640 1 -- start start 2023-10-30T14:44:56.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f865113f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f864c06aeb0 0x7f864c0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:56.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f865113f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c0c8eb0 0x7f864c154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:56.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f865113f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f864c15ac70 0x7f864c154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:56.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f865113f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c1f6100 con 0x7f864c0c8eb0 2023-10-30T14:44:56.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f865113f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c06a990 con 0x7f864c15ac70 2023-10-30T14:44:56.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f865113f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864c0b3550 con 0x7f864c06aeb0 2023-10-30T14:44:56.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f864b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c0c8eb0 0x7f864c154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:56.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f864b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c0c8eb0 0x7f864c154680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53706/0 (socket says 172.21.15.57:53706) 2023-10-30T14:44:56.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f864b7fe640 1 -- 172.21.15.57:0/89361816 learned_addr learned my addr 172.21.15.57:0/89361816 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:44:56.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f864bfff640 1 --2- 172.21.15.57:0/89361816 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f864c06aeb0 0x7f864c0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:56.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f865093e640 1 --2- 172.21.15.57:0/89361816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f864c15ac70 0x7f864c154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:56.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f864b7fe640 1 -- 172.21.15.57:0/89361816 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f864c06aeb0 msgr2=0x7f864c0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:56.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f864b7fe640 1 --2- 172.21.15.57:0/89361816 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f864c06aeb0 0x7f864c0ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:56.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f864b7fe640 1 -- 172.21.15.57:0/89361816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f864c15ac70 msgr2=0x7f864c154bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:56.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f864b7fe640 1 --2- 172.21.15.57:0/89361816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f864c15ac70 0x7f864c154bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:56.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f864b7fe640 1 -- 172.21.15.57:0/89361816 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8644009b30 con 0x7f864c0c8eb0 2023-10-30T14:44:56.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f864bfff640 1 --2- 172.21.15.57:0/89361816 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f864c06aeb0 0x7f864c0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:44:56.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f865093e640 1 --2- 172.21.15.57:0/89361816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f864c15ac70 0x7f864c154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:44:56.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f864b7fe640 1 --2- 172.21.15.57:0/89361816 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c0c8eb0 0x7f864c154680 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f864000e840 tx=0x7f864000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:56.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f8648ff9640 1 -- 172.21.15.57:0/89361816 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8640009510 con 0x7f864c0c8eb0 2023-10-30T14:44:56.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f8648ff9640 1 -- 172.21.15.57:0/89361816 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864000ee90 con 0x7f864c0c8eb0 2023-10-30T14:44:56.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f8648ff9640 1 -- 172.21.15.57:0/89361816 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864000cb80 con 0x7f864c0c8eb0 2023-10-30T14:44:56.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f865113f640 1 -- 172.21.15.57:0/89361816 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864c0b3270 con 0x7f864c0c8eb0 2023-10-30T14:44:56.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.358+0000 7f865113f640 1 -- 172.21.15.57:0/89361816 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f864c1f3990 con 0x7f864c0c8eb0 2023-10-30T14:44:56.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.362+0000 7f865113f640 1 -- 172.21.15.57:0/89361816 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f864c1f3b20 con 0x7f864c0c8eb0 2023-10-30T14:44:56.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.362+0000 7f8648ff9640 1 -- 172.21.15.57:0/89361816 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f864000cce0 con 0x7f864c0c8eb0 2023-10-30T14:44:56.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.362+0000 7f8648ff9640 1 --2- 172.21.15.57:0/89361816 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f86380664a0 0x7f8638068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:44:56.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.362+0000 7f8648ff9640 1 -- 172.21.15.57:0/89361816 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f8640084530 con 0x7f864c0c8eb0 2023-10-30T14:44:56.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.362+0000 7f864bfff640 1 --2- 172.21.15.57:0/89361816 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f86380664a0 0x7f8638068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:44:56.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.366+0000 7f864bfff640 1 --2- 172.21.15.57:0/89361816 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f86380664a0 0x7f8638068960 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f863c00c940 tx=0x7f863c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:44:56.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.366+0000 7f8648ff9640 1 -- 172.21.15.57:0/89361816 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f86400513c0 con 0x7f864c0c8eb0 2023-10-30T14:44:56.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.558+0000 7f865113f640 1 -- 172.21.15.57:0/89361816 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f864c04e170 con 0x7f864c0c8eb0 2023-10-30T14:44:56.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.558+0000 7f8648ff9640 1 -- 172.21.15.57:0/89361816 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5717 (secure 0 0 0) 0x7f86400549e0 con 0x7f864c0c8eb0 2023-10-30T14:44:56.560 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:44:56.561 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:44:54.212855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:44:56.561 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 169 2023-10-30T14:44:56.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.558+0000 7f865113f640 1 -- 172.21.15.57:0/89361816 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f86380664a0 msgr2=0x7f8638068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:56.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.558+0000 7f865113f640 1 --2- 172.21.15.57:0/89361816 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f86380664a0 0x7f8638068960 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f863c00c940 tx=0x7f863c008040 comp rx=0 tx=0).stop 2023-10-30T14:44:56.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.558+0000 7f865113f640 1 -- 172.21.15.57:0/89361816 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c0c8eb0 msgr2=0x7f864c154680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:44:56.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.558+0000 7f865113f640 1 --2- 172.21.15.57:0/89361816 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c0c8eb0 0x7f864c154680 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f864000e840 tx=0x7f864000ec40 comp rx=0 tx=0).stop 2023-10-30T14:44:56.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.558+0000 7f865113f640 1 -- 172.21.15.57:0/89361816 shutdown_connections 2023-10-30T14:44:56.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.558+0000 7f865113f640 1 --2- 172.21.15.57:0/89361816 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f86380664a0 0x7f8638068960 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:56.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.558+0000 7f865113f640 1 --2- 172.21.15.57:0/89361816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f864c15ac70 0x7f864c154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:56.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.558+0000 7f865113f640 1 --2- 172.21.15.57:0/89361816 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c0c8eb0 0x7f864c154680 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:56.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.558+0000 7f865113f640 1 --2- 172.21.15.57:0/89361816 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f864c06aeb0 0x7f864c0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:44:56.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.558+0000 7f865113f640 1 -- 172.21.15.57:0/89361816 >> 172.21.15.57:0/89361816 conn(0x7f864c0cef00 msgr2=0x7f864c15bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:44:56.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.558+0000 7f865113f640 1 -- 172.21.15.57:0/89361816 shutdown_connections 2023-10-30T14:44:56.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:44:56.562+0000 7f865113f640 1 -- 172.21.15.57:0/89361816 wait complete. 2023-10-30T14:44:56.581 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=6356 2023-10-30T14:44:56.582 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.f 2023-10-30T14:44:59.255 INFO:tasks.ceph.mds.e.smithi057.stderr:2023-10-30T14:44:59.254+0000 7f27adc3c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T14:44:59.256 INFO:tasks.ceph.mds.e.smithi057.stderr:2023-10-30T14:44:59.254+0000 7f27adc3c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:45:02.583 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-30T14:45:02.584 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T14:45:02.584 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T14:45:02.587 INFO:tasks.ceph.mds.f:Started 2023-10-30T14:45:02.588 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:45:02.665 INFO:tasks.ceph.mds.f.smithi088.stdout:starting mds.f at 2023-10-30T14:45:02.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.742+0000 7f3d9d2ea640 1 Processor -- start 2023-10-30T14:45:02.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.742+0000 7f3d9d2ea640 1 -- start start 2023-10-30T14:45:02.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.742+0000 7f3d9d2ea640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3d980c6320 0x7f3d9806fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:02.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.742+0000 7f3d9d2ea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3d9814fc50 0x7f3d98070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:02.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.742+0000 7f3d9d2ea640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d980c55f0 con 0x7f3d98155c40 2023-10-30T14:45:02.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.742+0000 7f3d9d2ea640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d980c5f00 con 0x7f3d980c6320 2023-10-30T14:45:02.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.742+0000 7f3d9d2ea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d98071350 con 0x7f3d9814fc50 2023-10-30T14:45:02.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.742+0000 7f3d977fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f3d98155c40 0x7f3d98159230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41780/0 (socket says 172.21.15.57:41780) 2023-10-30T14:45:02.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.742+0000 7f3d96ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3d980c6320 0x7f3d9806fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:02.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.742+0000 7f3d977fe640 1 -- 172.21.15.57:0/3534464603 learned_addr learned my addr 172.21.15.57:0/3534464603 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:45:02.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.742+0000 7f3d8ffff640 1 --2- 172.21.15.57:0/3534464603 >> v2:172.21.15.88:3300/0 conn(0x7f3d9814fc50 0x7f3d98070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:02.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.742+0000 7f3d967fc640 1 -- 172.21.15.57:0/3534464603 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936082596 0 0) 0x7f3d980c55f0 con 0x7f3d98155c40 2023-10-30T14:45:02.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.742+0000 7f3d967fc640 1 -- 172.21.15.57:0/3534464603 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d84003530 con 0x7f3d98155c40 2023-10-30T14:45:02.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.742+0000 7f3d96ffd640 1 -- 172.21.15.57:0/3534464603 >> v2:172.21.15.88:3300/0 conn(0x7f3d9814fc50 msgr2=0x7f3d98070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:45:02.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.742+0000 7f3d96ffd640 1 --2- 172.21.15.57:0/3534464603 >> v2:172.21.15.88:3300/0 conn(0x7f3d9814fc50 0x7f3d98070e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:02.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.742+0000 7f3d96ffd640 1 -- 172.21.15.57:0/3534464603 >> v1:172.21.15.57:6789/0 conn(0x7f3d98155c40 legacy=0x7f3d98159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:02.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.742+0000 7f3d96ffd640 1 -- 172.21.15.57:0/3534464603 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d981567b0 con 0x7f3d980c6320 2023-10-30T14:45:02.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.746+0000 7f3d96ffd640 1 --2- 172.21.15.57:0/3534464603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d980c6320 0x7f3d9806fdf0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f3d980667f0 tx=0x7f3d8801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36a63594d977776d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:45:02.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.746+0000 7f3d967fc640 1 -- 172.21.15.57:0/3534464603 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d880ac630 con 0x7f3d980c6320 2023-10-30T14:45:02.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.746+0000 7f3d967fc640 1 -- 172.21.15.57:0/3534464603 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d880ac790 con 0x7f3d980c6320 2023-10-30T14:45:02.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.746+0000 7f3d967fc640 1 -- 172.21.15.57:0/3534464603 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d880acb20 con 0x7f3d980c6320 2023-10-30T14:45:02.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.746+0000 7f3d9d2ea640 1 -- 172.21.15.57:0/3534464603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d980c6320 msgr2=0x7f3d9806fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:02.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.746+0000 7f3d9d2ea640 1 --2- 172.21.15.57:0/3534464603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d980c6320 0x7f3d9806fdf0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f3d980667f0 tx=0x7f3d8801c3d0 comp rx=0 tx=0).stop 2023-10-30T14:45:02.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.746+0000 7f3d9d2ea640 1 -- 172.21.15.57:0/3534464603 shutdown_connections 2023-10-30T14:45:02.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.746+0000 7f3d9d2ea640 1 --2- 172.21.15.57:0/3534464603 >> v2:172.21.15.88:3300/0 conn(0x7f3d9814fc50 0x7f3d98070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:02.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.746+0000 7f3d9d2ea640 1 --2- 172.21.15.57:0/3534464603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d980c6320 0x7f3d9806fdf0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:02.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.746+0000 7f3d9d2ea640 1 -- 172.21.15.57:0/3534464603 >> 172.21.15.57:0/3534464603 conn(0x7f3d980cac70 msgr2=0x7f3d980cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:45:02.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.746+0000 7f3d9d2ea640 1 -- 172.21.15.57:0/3534464603 shutdown_connections 2023-10-30T14:45:02.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.746+0000 7f3d9d2ea640 1 -- 172.21.15.57:0/3534464603 wait complete. 2023-10-30T14:45:02.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d9d2ea640 1 Processor -- start 2023-10-30T14:45:02.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d9d2ea640 1 -- start start 2023-10-30T14:45:02.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d9d2ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d980c6320 0x7f3d9813ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:02.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d9d2ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3d9814fc50 0x7f3d9813e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:02.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d9d2ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d98155c40 0x7f3d9813ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:02.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d9d2ea640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9813dd20 con 0x7f3d98155c40 2023-10-30T14:45:02.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d9d2ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d981610a0 con 0x7f3d980c6320 2023-10-30T14:45:02.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d9d2ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d981463c0 con 0x7f3d9814fc50 2023-10-30T14:45:02.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d977fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d98155c40 0x7f3d9813ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:02.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d8ffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3d9814fc50 0x7f3d9813e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:02.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d96ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d980c6320 0x7f3d9813ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:02.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d977fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d98155c40 0x7f3d9813ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53716/0 (socket says 172.21.15.57:53716) 2023-10-30T14:45:02.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d977fe640 1 -- 172.21.15.57:0/2344864524 learned_addr learned my addr 172.21.15.57:0/2344864524 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:45:02.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d977fe640 1 -- 172.21.15.57:0/2344864524 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3d9814fc50 msgr2=0x7f3d9813e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:02.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d977fe640 1 --2- 172.21.15.57:0/2344864524 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3d9814fc50 0x7f3d9813e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:02.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d977fe640 1 -- 172.21.15.57:0/2344864524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d980c6320 msgr2=0x7f3d9813ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:02.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d977fe640 1 --2- 172.21.15.57:0/2344864524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d980c6320 0x7f3d9813ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:02.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d977fe640 1 -- 172.21.15.57:0/2344864524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d88009420 con 0x7f3d98155c40 2023-10-30T14:45:02.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d977fe640 1 --2- 172.21.15.57:0/2344864524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d98155c40 0x7f3d9813ef20 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f3d9000caa0 tx=0x7f3d9000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:45:02.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d8f7fe640 1 -- 172.21.15.57:0/2344864524 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9000ac00 con 0x7f3d98155c40 2023-10-30T14:45:02.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d8f7fe640 1 -- 172.21.15.57:0/2344864524 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9000ad60 con 0x7f3d98155c40 2023-10-30T14:45:02.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d8f7fe640 1 -- 172.21.15.57:0/2344864524 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d90012620 con 0x7f3d98155c40 2023-10-30T14:45:02.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d9d2ea640 1 -- 172.21.15.57:0/2344864524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d9813d6d0 con 0x7f3d98155c40 2023-10-30T14:45:02.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.750+0000 7f3d9d2ea640 1 -- 172.21.15.57:0/2344864524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d980c8a50 con 0x7f3d98155c40 2023-10-30T14:45:02.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.754+0000 7f3d9d2ea640 1 -- 172.21.15.57:0/2344864524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d9813e700 con 0x7f3d98155c40 2023-10-30T14:45:02.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.754+0000 7f3d8f7fe640 1 -- 172.21.15.57:0/2344864524 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3d90007070 con 0x7f3d98155c40 2023-10-30T14:45:02.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.754+0000 7f3d8f7fe640 1 --2- 172.21.15.57:0/2344864524 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3d780664a0 0x7f3d78068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:02.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.754+0000 7f3d8f7fe640 1 -- 172.21.15.57:0/2344864524 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f3d900843c0 con 0x7f3d98155c40 2023-10-30T14:45:02.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.754+0000 7f3d96ffd640 1 --2- 172.21.15.57:0/2344864524 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3d780664a0 0x7f3d78068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:02.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.754+0000 7f3d96ffd640 1 --2- 172.21.15.57:0/2344864524 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3d780664a0 0x7f3d78068960 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f3d880ae000 tx=0x7f3d88008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:45:02.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.758+0000 7f3d8f7fe640 1 -- 172.21.15.57:0/2344864524 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3d9813e700 con 0x7f3d98155c40 2023-10-30T14:45:02.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.950+0000 7f3d9d2ea640 1 -- 172.21.15.57:0/2344864524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d9813e700 con 0x7f3d98155c40 2023-10-30T14:45:02.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.954+0000 7f3d8f7fe640 1 -- 172.21.15.57:0/2344864524 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5717 (secure 0 0 0) 0x7f3d9813e700 con 0x7f3d98155c40 2023-10-30T14:45:02.955 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:45:02.956 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:44:54.212855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:45:02.957 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 169 2023-10-30T14:45:02.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.954+0000 7f3d9d2ea640 1 -- 172.21.15.57:0/2344864524 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3d780664a0 msgr2=0x7f3d78068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:02.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.954+0000 7f3d9d2ea640 1 --2- 172.21.15.57:0/2344864524 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3d780664a0 0x7f3d78068960 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f3d880ae000 tx=0x7f3d88008040 comp rx=0 tx=0).stop 2023-10-30T14:45:02.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.954+0000 7f3d9d2ea640 1 -- 172.21.15.57:0/2344864524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d98155c40 msgr2=0x7f3d9813ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:02.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.954+0000 7f3d9d2ea640 1 --2- 172.21.15.57:0/2344864524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d98155c40 0x7f3d9813ef20 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f3d9000caa0 tx=0x7f3d9000cf60 comp rx=0 tx=0).stop 2023-10-30T14:45:02.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.954+0000 7f3d9d2ea640 1 -- 172.21.15.57:0/2344864524 shutdown_connections 2023-10-30T14:45:02.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.954+0000 7f3d9d2ea640 1 --2- 172.21.15.57:0/2344864524 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f3d780664a0 0x7f3d78068960 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:02.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.954+0000 7f3d9d2ea640 1 --2- 172.21.15.57:0/2344864524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d98155c40 0x7f3d9813ef20 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:02.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.954+0000 7f3d9d2ea640 1 --2- 172.21.15.57:0/2344864524 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3d9814fc50 0x7f3d9813e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:02.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.954+0000 7f3d9d2ea640 1 --2- 172.21.15.57:0/2344864524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d980c6320 0x7f3d9813ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:02.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.954+0000 7f3d9d2ea640 1 -- 172.21.15.57:0/2344864524 >> 172.21.15.57:0/2344864524 conn(0x7f3d980cac70 msgr2=0x7f3d980c6790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:45:02.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.954+0000 7f3d9d2ea640 1 -- 172.21.15.57:0/2344864524 shutdown_connections 2023-10-30T14:45:02.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:02.954+0000 7f3d9d2ea640 1 -- 172.21.15.57:0/2344864524 wait complete. 2023-10-30T14:45:02.976 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-30T14:45:04.092 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 17% (40/226) Updating files: 18% (41/226) Updating files: 18% (42/226) Updating files: 19% (43/226) Updating files: 20% (46/226) Updating files: 21% (48/226) Updating files: 22% (50/226) Updating files: 23% (52/226) Updating files: 24% (55/226) Updating files: 25% (57/226) Updating files: 26% (59/226) Updating files: 27% (62/226) Updating files: 28% (64/226) Updating files: 29% (66/226) Updating files: 30% (68/226) Updating files: 31% (71/226) Updating files: 32% (73/226) Updating files: 33% (75/226) Updating files: 34% (77/226) Updating files: 35% (80/226) Updating files: 36% (82/226) Updating files: 37% (84/226) Updating files: 38% (86/226) Updating files: 39% (89/226) Updating files: 40% (91/226) Updating files: 41% (93/226) Updating files: 42% (95/226) Updating files: 43% (98/226) Updating files: 44% (100/226) Updating files: 45% (102/226) Updating files: 46% (104/226) Updating files: 47% (107/226) Updating files: 48% (109/226) Updating files: 49% (111/226) Updating files: 50% (113/226) Updating files: 51% (116/226) Updating files: 52% (118/226) Updating files: 53% (120/226) Updating files: 54% (123/226) Updating files: 55% (125/226) Updating files: 56% (127/226) Updating files: 56% (128/226) Updating files: 57% (129/226) Updating files: 58% (132/226) Updating files: 59% (134/226) Updating files: 60% (136/226) Updating files: 61% (138/226) Updating files: 62% (141/226) Updating files: 63% (143/226) Updating files: 64% (145/226) Updating files: 65% (147/226) Updating files: 66% (150/226) Updating files: 67% (152/226) Updating files: 68% (154/226) Updating files: 69% (156/226) Updating files: 70% (159/226) Updating files: 71% (161/226) Updating files: 72% (163/226) Updating files: 73% (165/226) Updating files: 74% (168/226) Updating files: 75% (170/226) Updating files: 76% (172/226) Updating files: 77% (175/226) Updating files: 78% (177/226) Updating files: 79% (179/226) Updating files: 80% (181/226) Updating files: 80% (182/226) Updating files: 81% (184/226) Updating files: 82% (186/226) Updating files: 83% (188/226) Updating files: 83% (189/226) Updating files: 84% (190/226) Updating files: 85% (193/226) Updating files: 86% (195/226) Updating files: 87% (197/226) Updating files: 88% (199/226) Updating files: 89% (202/226) Updating files: 90% (204/226) Updating files: 91% (206/226) Updating files: 92% (208/226) Updating files: 93% (211/226) Updating files: 94% (213/226) Updating files: 95% (215/226) Updating files: 96% (217/226) Updating files: 97% (220/226) Updating files: 98% (222/226) Updating files: 99% (224/226) Updating files: 100% (226/226) Updating files: 100% (226/226), done. 2023-10-30T14:45:04.188 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 7eea40ea37f v0.40 2023-10-30T14:45:04.191 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.40 2023-10-30T14:45:04.207 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 41 -eq 48 ] 2023-10-30T14:45:04.207 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.41 2023-10-30T14:45:04.207 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.41 2023-10-30T14:45:04.208 INFO:tasks.workunit.client.0.smithi057.stdout:v0.41 2023-10-30T14:45:04.208 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.41 2023-10-30T14:45:04.977 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:45:05.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.122+0000 7f8dd5bb8640 1 Processor -- start 2023-10-30T14:45:05.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.122+0000 7f8dd5bb8640 1 -- start start 2023-10-30T14:45:05.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.122+0000 7f8dd5bb8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dd0150830 0x7f8dd006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:05.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.122+0000 7f8dd5bb8640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8dd00cb7d0 0x7f8dd00c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:05.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.122+0000 7f8dd5bb8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd00c2310 con 0x7f8dd00c3e40 2023-10-30T14:45:05.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.122+0000 7f8dd5bb8640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd0156710 con 0x7f8dd00cb7d0 2023-10-30T14:45:05.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.122+0000 7f8dd5bb8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd00c4360 con 0x7f8dd0150830 2023-10-30T14:45:05.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.122+0000 7f8dceffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8dd00cb7d0 0x7f8dd00c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:05.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.122+0000 7f8dceffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8dd00cb7d0 0x7f8dd00c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33280/0 (socket says 172.21.15.57:33280) 2023-10-30T14:45:05.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.122+0000 7f8dceffd640 1 -- 172.21.15.57:0/3288699456 learned_addr learned my addr 172.21.15.57:0/3288699456 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:45:05.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.122+0000 7f8dcf7fe640 1 --2- 172.21.15.57:0/3288699456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dd0150830 0x7f8dd006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:05.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.122+0000 7f8dceffd640 1 -- 172.21.15.57:0/3288699456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dd0150830 msgr2=0x7f8dd006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:45:05.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.122+0000 7f8dceffd640 1 --2- 172.21.15.57:0/3288699456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dd0150830 0x7f8dd006b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:05.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.122+0000 7f8dceffd640 1 -- 172.21.15.57:0/3288699456 >> v1:172.21.15.88:6789/0 conn(0x7f8dd00c3e40 legacy=0x7f8dd0070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:05.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.122+0000 7f8dceffd640 1 -- 172.21.15.57:0/3288699456 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd01569b0 con 0x7f8dd00cb7d0 2023-10-30T14:45:05.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.126+0000 7f8dceffd640 1 --2- 172.21.15.57:0/3288699456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dd00cb7d0 0x7f8dd00c3900 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4009870 tx=0x7f8dc401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f16194d0b56bcea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:45:05.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.126+0000 7f8dce7fc640 1 -- 172.21.15.57:0/3288699456 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc40a8a30 con 0x7f8dd00cb7d0 2023-10-30T14:45:05.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.126+0000 7f8dce7fc640 1 -- 172.21.15.57:0/3288699456 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc40a8b90 con 0x7f8dd00cb7d0 2023-10-30T14:45:05.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.126+0000 7f8dce7fc640 1 -- 172.21.15.57:0/3288699456 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc40b0440 con 0x7f8dd00cb7d0 2023-10-30T14:45:05.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.126+0000 7f8dd5bb8640 1 -- 172.21.15.57:0/3288699456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dd00cb7d0 msgr2=0x7f8dd00c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:05.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.126+0000 7f8dd5bb8640 1 --2- 172.21.15.57:0/3288699456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dd00cb7d0 0x7f8dd00c3900 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4009870 tx=0x7f8dc401ed30 comp rx=0 tx=0).stop 2023-10-30T14:45:05.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.126+0000 7f8dd5bb8640 1 -- 172.21.15.57:0/3288699456 shutdown_connections 2023-10-30T14:45:05.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.126+0000 7f8dd5bb8640 1 --2- 172.21.15.57:0/3288699456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dd00cb7d0 0x7f8dd00c3900 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:05.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.126+0000 7f8dd5bb8640 1 --2- 172.21.15.57:0/3288699456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dd0150830 0x7f8dd006b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:05.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.126+0000 7f8dd5bb8640 1 -- 172.21.15.57:0/3288699456 >> 172.21.15.57:0/3288699456 conn(0x7f8dd00cac70 msgr2=0x7f8dd00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:45:05.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.126+0000 7f8dd5bb8640 1 -- 172.21.15.57:0/3288699456 shutdown_connections 2023-10-30T14:45:05.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.126+0000 7f8dd5bb8640 1 -- 172.21.15.57:0/3288699456 wait complete. 2023-10-30T14:45:05.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dd5bb8640 1 Processor -- start 2023-10-30T14:45:05.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dd5bb8640 1 -- start start 2023-10-30T14:45:05.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dd5bb8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dd00c3e40 0x7f8dd01522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:05.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dd5bb8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dd00cb7d0 0x7f8dd0154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:05.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dd5bb8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dd0150830 0x7f8dd00c5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:05.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dd5bb8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd0150f10 con 0x7f8dd00cb7d0 2023-10-30T14:45:05.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dd5bb8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd0151730 con 0x7f8dd00c3e40 2023-10-30T14:45:05.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dd5bb8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd0154b80 con 0x7f8dd0150830 2023-10-30T14:45:05.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dceffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dd00cb7d0 0x7f8dd0154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:05.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dceffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dd00cb7d0 0x7f8dd0154640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33296/0 (socket says 172.21.15.57:33296) 2023-10-30T14:45:05.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dceffd640 1 -- 172.21.15.57:0/4005832724 learned_addr learned my addr 172.21.15.57:0/4005832724 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:45:05.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dcf7fe640 1 --2- 172.21.15.57:0/4005832724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dd00c3e40 0x7f8dd01522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:05.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dcffff640 1 --2- 172.21.15.57:0/4005832724 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dd0150830 0x7f8dd00c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:05.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dceffd640 1 -- 172.21.15.57:0/4005832724 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dd0150830 msgr2=0x7f8dd00c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:05.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dceffd640 1 --2- 172.21.15.57:0/4005832724 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dd0150830 0x7f8dd00c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:05.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dceffd640 1 -- 172.21.15.57:0/4005832724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dd00c3e40 msgr2=0x7f8dd01522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:05.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dceffd640 1 --2- 172.21.15.57:0/4005832724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dd00c3e40 0x7f8dd01522e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:05.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dceffd640 1 -- 172.21.15.57:0/4005832724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc40094d0 con 0x7f8dd00cb7d0 2023-10-30T14:45:05.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dcffff640 1 --2- 172.21.15.57:0/4005832724 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dd0150830 0x7f8dd00c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:45:05.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8dceffd640 1 --2- 172.21.15.57:0/4005832724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dd00cb7d0 0x7f8dd0154640 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f8dc40b3040 tx=0x7f8dc401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:45:05.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8db7fff640 1 -- 172.21.15.57:0/4005832724 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc40b0dd0 con 0x7f8dd00cb7d0 2023-10-30T14:45:05.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8db7fff640 1 -- 172.21.15.57:0/4005832724 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc40bb460 con 0x7f8dd00cb7d0 2023-10-30T14:45:05.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.130+0000 7f8db7fff640 1 -- 172.21.15.57:0/4005832724 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc40bb810 con 0x7f8dd00cb7d0 2023-10-30T14:45:05.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.134+0000 7f8dd5bb8640 1 -- 172.21.15.57:0/4005832724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd00c60f0 con 0x7f8dd00cb7d0 2023-10-30T14:45:05.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.134+0000 7f8dd5bb8640 1 -- 172.21.15.57:0/4005832724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd0151450 con 0x7f8dd00cb7d0 2023-10-30T14:45:05.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.134+0000 7f8dd5bb8640 1 -- 172.21.15.57:0/4005832724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd0066510 con 0x7f8dd00cb7d0 2023-10-30T14:45:05.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.134+0000 7f8db7fff640 1 -- 172.21.15.57:0/4005832724 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8dc40b83f0 con 0x7f8dd00cb7d0 2023-10-30T14:45:05.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.134+0000 7f8db7fff640 1 --2- 172.21.15.57:0/4005832724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dbc0664a0 0x7f8dbc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:05.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.134+0000 7f8db7fff640 1 -- 172.21.15.57:0/4005832724 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f8dc4121d30 con 0x7f8dd00cb7d0 2023-10-30T14:45:05.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.134+0000 7f8dcf7fe640 1 --2- 172.21.15.57:0/4005832724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dbc0664a0 0x7f8dbc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:05.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.138+0000 7f8dcf7fe640 1 --2- 172.21.15.57:0/4005832724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dbc0664a0 0x7f8dbc068960 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0009ac0 tx=0x7f8dc0009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:45:05.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.138+0000 7f8db7fff640 1 -- 172.21.15.57:0/4005832724 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8dc40eebe0 con 0x7f8dd00cb7d0 2023-10-30T14:45:05.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.334+0000 7f8dd5bb8640 1 -- 172.21.15.57:0/4005832724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8dd004e170 con 0x7f8dd00cb7d0 2023-10-30T14:45:05.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.334+0000 7f8db7fff640 1 -- 172.21.15.57:0/4005832724 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6510 (secure 0 0 0) 0x7f8dc40c0090 con 0x7f8dd00cb7d0 2023-10-30T14:45:05.336 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:45:05.337 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:45:04.252839+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:45:05.337 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 171 2023-10-30T14:45:05.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.334+0000 7f8dd5bb8640 1 -- 172.21.15.57:0/4005832724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dbc0664a0 msgr2=0x7f8dbc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:05.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.334+0000 7f8dd5bb8640 1 --2- 172.21.15.57:0/4005832724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dbc0664a0 0x7f8dbc068960 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0009ac0 tx=0x7f8dc0009190 comp rx=0 tx=0).stop 2023-10-30T14:45:05.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.334+0000 7f8dd5bb8640 1 -- 172.21.15.57:0/4005832724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dd00cb7d0 msgr2=0x7f8dd0154640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:05.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.334+0000 7f8dd5bb8640 1 --2- 172.21.15.57:0/4005832724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dd00cb7d0 0x7f8dd0154640 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f8dc40b3040 tx=0x7f8dc401bad0 comp rx=0 tx=0).stop 2023-10-30T14:45:05.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.334+0000 7f8dd5bb8640 1 -- 172.21.15.57:0/4005832724 shutdown_connections 2023-10-30T14:45:05.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.334+0000 7f8dd5bb8640 1 --2- 172.21.15.57:0/4005832724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8dbc0664a0 0x7f8dbc068960 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:05.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.334+0000 7f8dd5bb8640 1 --2- 172.21.15.57:0/4005832724 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8dd0150830 0x7f8dd00c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:05.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.334+0000 7f8dd5bb8640 1 --2- 172.21.15.57:0/4005832724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dd00cb7d0 0x7f8dd0154640 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:05.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.334+0000 7f8dd5bb8640 1 --2- 172.21.15.57:0/4005832724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dd00c3e40 0x7f8dd01522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:05.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.334+0000 7f8dd5bb8640 1 -- 172.21.15.57:0/4005832724 >> 172.21.15.57:0/4005832724 conn(0x7f8dd00cac70 msgr2=0x7f8dd014e080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:45:05.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.334+0000 7f8dd5bb8640 1 -- 172.21.15.57:0/4005832724 shutdown_connections 2023-10-30T14:45:05.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:05.334+0000 7f8dd5bb8640 1 -- 172.21.15.57:0/4005832724 wait complete. 2023-10-30T14:45:05.357 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-30T14:45:05.357 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 51 secs before thrashing 2023-10-30T14:45:09.163 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:45:09.169 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:45:12.586 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 49% (43/87) Updating files: 50% (44/87) Updating files: 51% (45/87) Updating files: 52% (46/87) Updating files: 54% (47/87) Updating files: 55% (48/87) Updating files: 56% (49/87) Updating files: 57% (50/87) Updating files: 58% (51/87) Updating files: 59% (52/87) Updating files: 60% (53/87) Updating files: 62% (54/87) Updating files: 63% (55/87) Updating files: 64% (56/87) Updating files: 65% (57/87) Updating files: 66% (58/87) Updating files: 67% (59/87) Updating files: 68% (60/87) Updating files: 70% (61/87) Updating files: 71% (62/87) Updating files: 72% (63/87) Updating files: 73% (64/87) Updating files: 74% (65/87) Updating files: 75% (66/87) Updating files: 77% (67/87) Updating files: 78% (68/87) Updating files: 79% (69/87) Updating files: 80% (70/87) Updating files: 81% (71/87) Updating files: 82% (72/87) Updating files: 83% (73/87) Updating files: 85% (74/87) Updating files: 86% (75/87) Updating files: 87% (76/87) Updating files: 88% (77/87) Updating files: 89% (78/87) Updating files: 90% (79/87) Updating files: 91% (80/87) Updating files: 93% (81/87) Updating files: 94% (82/87) Updating files: 95% (83/87) Updating files: 96% (84/87) Updating files: 97% (85/87) Updating files: 98% (86/87) Updating files: 100% (87/87) Updating files: 100% (87/87), done. 2023-10-30T14:45:13.634 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at c1345f7136a v0.41 2023-10-30T14:45:13.637 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.41 2023-10-30T14:45:14.203 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 42 -eq 48 ] 2023-10-30T14:45:14.204 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.42 2023-10-30T14:45:14.204 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.42 2023-10-30T14:45:14.205 INFO:tasks.workunit.client.0.smithi057.stdout:v0.42 2023-10-30T14:45:14.205 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.42 2023-10-30T14:45:21.999 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 11% (28/250) Updating files: 12% (30/250) Updating files: 13% (33/250) Updating files: 14% (35/250) Updating files: 15% (38/250) Updating files: 16% (40/250) Updating files: 17% (43/250) Updating files: 18% (45/250) Updating files: 19% (48/250) Updating files: 20% (50/250) Updating files: 21% (53/250) Updating files: 22% (55/250) Updating files: 23% (58/250) Updating files: 24% (60/250) Updating files: 25% (63/250) Updating files: 26% (65/250) Updating files: 27% (68/250) Updating files: 28% (70/250) Updating files: 29% (73/250) Updating files: 30% (75/250) Updating files: 31% (78/250) Updating files: 32% (80/250) Updating files: 33% (83/250) Updating files: 34% (85/250) Updating files: 35% (88/250) Updating files: 36% (90/250) Updating files: 37% (93/250) Updating files: 38% (95/250) Updating files: 39% (98/250) Updating files: 40% (100/250) Updating files: 41% (103/250) Updating files: 42% (105/250) Updating files: 43% (108/250) Updating files: 44% (110/250) Updating files: 45% (113/250) Updating files: 46% (115/250) Updating files: 46% (117/250) Updating files: 47% (118/250) Updating files: 48% (120/250) Updating files: 49% (123/250) Updating files: 50% (125/250) Updating files: 51% (128/250) Updating files: 52% (130/250) Updating files: 53% (133/250) Updating files: 54% (135/250) Updating files: 55% (138/250) Updating files: 56% (140/250) Updating files: 57% (143/250) Updating files: 58% (145/250) Updating files: 59% (148/250) Updating files: 60% (150/250) Updating files: 61% (153/250) Updating files: 62% (155/250) Updating files: 63% (158/250) Updating files: 64% (160/250) Updating files: 65% (163/250) Updating files: 66% (165/250) Updating files: 67% (168/250) Updating files: 68% (170/250) Updating files: 69% (173/250) Updating files: 70% (175/250) Updating files: 71% (178/250) Updating files: 72% (180/250) Updating files: 73% (183/250) Updating files: 74% (185/250) Updating files: 75% (188/250) Updating files: 76% (190/250) Updating files: 77% (193/250) Updating files: 78% (195/250) Updating files: 79% (198/250) Updating files: 80% (200/250) Updating files: 81% (203/250) Updating files: 82% (205/250) Updating files: 83% (208/250) Updating files: 84% (210/250) Updating files: 85% (213/250) Updating files: 86% (215/250) Updating files: 87% (218/250) Updating files: 88% (220/250) Updating files: 89% (223/250) Updating files: 90% (225/250) Updating files: 91% (228/250) Updating files: 92% (230/250) Updating files: 93% (233/250) Updating files: 94% (235/250) Updating files: 95% (238/250) Updating files: 96% (240/250) Updating files: 97% (243/250) Updating files: 98% (245/250) Updating files: 99% (248/250) Updating files: 100% (250/250) Updating files: 100% (250/250), done. 2023-10-30T14:45:22.550 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at b205c64c687 v0.42 2023-10-30T14:45:22.553 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.42 2023-10-30T14:45:22.653 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 43 -eq 48 ] 2023-10-30T14:45:22.653 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.43 2023-10-30T14:45:22.653 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.43 2023-10-30T14:45:22.653 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.43 2023-10-30T14:45:22.654 INFO:tasks.workunit.client.0.smithi057.stdout:v0.43 2023-10-30T14:45:25.714 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 68% (76/111) Updating files: 69% (77/111) Updating files: 70% (78/111) Updating files: 71% (79/111) Updating files: 72% (80/111) Updating files: 73% (82/111) Updating files: 74% (83/111) Updating files: 75% (84/111) Updating files: 76% (85/111) Updating files: 77% (86/111) Updating files: 78% (87/111) Updating files: 79% (88/111) Updating files: 80% (89/111) Updating files: 81% (90/111) Updating files: 82% (92/111) Updating files: 83% (93/111) Updating files: 84% (94/111) Updating files: 85% (95/111) Updating files: 86% (96/111) Updating files: 87% (97/111) Updating files: 88% (98/111) Updating files: 89% (99/111) Updating files: 90% (100/111) Updating files: 91% (102/111) Updating files: 92% (103/111) Updating files: 93% (104/111) Updating files: 94% (105/111) Updating files: 95% (106/111) Updating files: 96% (107/111) Updating files: 97% (108/111) Updating files: 98% (109/111) Updating files: 99% (110/111) Updating files: 100% (111/111) Updating files: 100% (111/111), done. 2023-10-30T14:45:25.799 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 9fa8781c014 v0.43 2023-10-30T14:45:25.802 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.43 2023-10-30T14:45:26.204 INFO:tasks.workunit.client.0.smithi057.stdout:v0.44 2023-10-30T14:45:26.205 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 44 -eq 48 ] 2023-10-30T14:45:26.205 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.44 2023-10-30T14:45:26.205 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.44 2023-10-30T14:45:26.205 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.44 2023-10-30T14:45:28.572 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 80% (149/186) Updating files: 81% (151/186) Updating files: 82% (153/186) Updating files: 83% (155/186) Updating files: 84% (157/186) Updating files: 85% (159/186) Updating files: 86% (160/186) Updating files: 87% (162/186) Updating files: 88% (164/186) Updating files: 89% (166/186) Updating files: 90% (168/186) Updating files: 91% (170/186) Updating files: 92% (172/186) Updating files: 93% (173/186) Updating files: 94% (175/186) Updating files: 95% (177/186) Updating files: 96% (179/186) Updating files: 97% (181/186) Updating files: 98% (183/186) Updating files: 99% (185/186) Updating files: 100% (186/186) Updating files: 100% (186/186), done. 2023-10-30T14:45:28.610 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at cdd5298d276 v0.44 2023-10-30T14:45:28.613 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.44 2023-10-30T14:45:28.745 INFO:tasks.workunit.client.0.smithi057.stdout:v0.45 2023-10-30T14:45:28.746 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 45 -eq 48 ] 2023-10-30T14:45:28.746 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.45 2023-10-30T14:45:28.746 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.45 2023-10-30T14:45:28.747 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.45 2023-10-30T14:45:39.237 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:45:39.241 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:45:45.890 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 6% (10/154) Updating files: 7% (11/154) Updating files: 8% (13/154) Updating files: 9% (14/154) Updating files: 10% (16/154) Updating files: 11% (17/154) Updating files: 12% (19/154) Updating files: 13% (21/154) Updating files: 14% (22/154) Updating files: 15% (24/154) Updating files: 16% (25/154) Updating files: 17% (27/154) Updating files: 18% (28/154) Updating files: 19% (30/154) Updating files: 20% (31/154) Updating files: 21% (33/154) Updating files: 22% (34/154) Updating files: 23% (36/154) Updating files: 24% (37/154) Updating files: 25% (39/154) Updating files: 26% (41/154) Updating files: 27% (42/154) Updating files: 28% (44/154) Updating files: 29% (45/154) Updating files: 30% (47/154) Updating files: 31% (48/154) Updating files: 32% (50/154) Updating files: 33% (51/154) Updating files: 34% (53/154) Updating files: 35% (54/154) Updating files: 36% (56/154) Updating files: 37% (57/154) Updating files: 38% (59/154) Updating files: 39% (61/154) Updating files: 40% (62/154) Updating files: 41% (64/154) Updating files: 42% (65/154) Updating files: 43% (67/154) Updating files: 44% (68/154) Updating files: 45% (70/154) Updating files: 46% (71/154) Updating files: 47% (73/154) Updating files: 48% (74/154) Updating files: 49% (76/154) Updating files: 50% (77/154) Updating files: 51% (79/154) Updating files: 52% (81/154) Updating files: 53% (82/154) Updating files: 54% (84/154) Updating files: 55% (85/154) Updating files: 56% (87/154) Updating files: 57% (88/154) Updating files: 58% (90/154) Updating files: 59% (91/154) Updating files: 60% (93/154) Updating files: 61% (94/154) Updating files: 62% (96/154) Updating files: 63% (98/154) Updating files: 64% (99/154) Updating files: 65% (101/154) Updating files: 66% (102/154) Updating files: 67% (104/154) Updating files: 68% (105/154) Updating files: 69% (107/154) Updating files: 70% (108/154) Updating files: 71% (110/154) Updating files: 72% (111/154) Updating files: 73% (113/154) Updating files: 74% (114/154) Updating files: 75% (116/154) Updating files: 76% (118/154) Updating files: 77% (119/154) Updating files: 78% (121/154) Updating files: 79% (122/154) Updating files: 80% (124/154) Updating files: 81% (125/154) Updating files: 82% (127/154) Updating files: 83% (128/154) Updating files: 84% (130/154) Updating files: 85% (131/154) Updating files: 86% (133/154) Updating files: 87% (134/154) Updating files: 88% (136/154) Updating files: 89% (138/154) Updating files: 90% (139/154) Updating files: 91% (141/154) Updating files: 92% (142/154) Updating files: 93% (144/154) Updating files: 94% (145/154) Updating files: 95% (147/154) Updating files: 96% (148/154) Updating files: 97% (150/154) Updating files: 98% (151/154) Updating files: 99% (153/154) Updating files: 100% (154/154) Updating files: 100% (154/154), done. 2023-10-30T14:45:45.928 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 0aea1cb1df5 v0.45 2023-10-30T14:45:45.931 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.45 2023-10-30T14:45:46.024 INFO:tasks.workunit.client.0.smithi057.stdout:v0.46 2023-10-30T14:45:46.024 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 46 -eq 48 ] 2023-10-30T14:45:46.025 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.46 2023-10-30T14:45:46.025 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.46 2023-10-30T14:45:46.025 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.46 2023-10-30T14:45:51.472 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 19% (59/309) Updating files: 20% (62/309) Updating files: 21% (65/309) Updating files: 22% (68/309) Updating files: 23% (72/309) Updating files: 24% (75/309) Updating files: 25% (78/309) Updating files: 26% (81/309) Updating files: 27% (84/309) Updating files: 28% (87/309) Updating files: 29% (90/309) Updating files: 30% (93/309) Updating files: 31% (96/309) Updating files: 32% (99/309) Updating files: 33% (102/309) Updating files: 34% (106/309) Updating files: 35% (109/309) Updating files: 36% (112/309) Updating files: 37% (115/309) Updating files: 38% (118/309) Updating files: 39% (121/309) Updating files: 40% (124/309) Updating files: 41% (127/309) Updating files: 42% (130/309) Updating files: 43% (133/309) Updating files: 44% (136/309) Updating files: 45% (140/309) Updating files: 46% (143/309) Updating files: 47% (146/309) Updating files: 48% (149/309) Updating files: 49% (152/309) Updating files: 50% (155/309) Updating files: 51% (158/309) Updating files: 52% (161/309) Updating files: 53% (164/309) Updating files: 54% (167/309) Updating files: 55% (170/309) Updating files: 55% (171/309) Updating files: 56% (174/309) Updating files: 57% (177/309) Updating files: 58% (180/309) Updating files: 59% (183/309) Updating files: 60% (186/309) Updating files: 61% (189/309) Updating files: 62% (192/309) Updating files: 63% (195/309) Updating files: 64% (198/309) Updating files: 65% (201/309) Updating files: 66% (204/309) Updating files: 67% (208/309) Updating files: 68% (211/309) Updating files: 69% (214/309) Updating files: 70% (217/309) Updating files: 71% (220/309) Updating files: 72% (223/309) Updating files: 73% (226/309) Updating files: 74% (229/309) Updating files: 75% (232/309) Updating files: 76% (235/309) Updating files: 77% (238/309) Updating files: 78% (242/309) Updating files: 79% (245/309) Updating files: 80% (248/309) Updating files: 81% (251/309) Updating files: 82% (254/309) Updating files: 83% (257/309) Updating files: 84% (260/309) Updating files: 85% (263/309) Updating files: 86% (266/309) Updating files: 87% (269/309) Updating files: 88% (272/309) Updating files: 88% (274/309) Updating files: 89% (276/309) Updating files: 90% (279/309) Updating files: 91% (282/309) Updating files: 92% (285/309) Updating files: 93% (288/309) Updating files: 94% (291/309) Updating files: 95% (294/309) Updating files: 96% (297/309) Updating files: 97% (300/309) Updating files: 98% (303/309) Updating files: 99% (306/309) Updating files: 100% (309/309) Updating files: 100% (309/309), done. 2023-10-30T14:45:55.493 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at cb7f1c9c752 v0.46 2023-10-30T14:45:55.496 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.46 2023-10-30T14:45:55.535 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 47 -eq 48 ] 2023-10-30T14:45:55.535 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.47 2023-10-30T14:45:55.536 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.47 2023-10-30T14:45:55.536 INFO:tasks.workunit.client.0.smithi057.stdout:v0.47 2023-10-30T14:45:55.536 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.47 2023-10-30T14:45:56.358 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:45:56.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.501+0000 7fe2a4617640 1 Processor -- start 2023-10-30T14:45:56.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.501+0000 7fe2a4617640 1 -- start start 2023-10-30T14:45:56.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.501+0000 7fe2a4617640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe29c153c10 0x7fe29c065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:56.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.501+0000 7fe2a4617640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe29c159970 0x7fe29c159e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:56.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.501+0000 7fe2a4617640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29c06e390 con 0x7fe29c070c30 2023-10-30T14:45:56.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.501+0000 7fe2a4617640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe29c072390 con 0x7fe29c159970 2023-10-30T14:45:56.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.501+0000 7fe2a4617640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c0660e0 con 0x7fe29c153c10 2023-10-30T14:45:56.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.505+0000 7fe2a23b3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe29c153c10 0x7fe29c065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:56.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.505+0000 7fe2a2bb4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe29c159970 0x7fe29c159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:56.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.505+0000 7fe2a1bb2640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe29c070c30 0x7fe29c071070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:46072/0 (socket says 172.21.15.57:46072) 2023-10-30T14:45:56.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.505+0000 7fe2a1bb2640 1 -- 172.21.15.57:0/3685607594 learned_addr learned my addr 172.21.15.57:0/3685607594 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:45:56.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.505+0000 7fe2a23b3640 1 -- 172.21.15.57:0/3685607594 >> v2:172.21.15.88:3300/0 conn(0x7fe29c159970 msgr2=0x7fe29c159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:45:56.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.505+0000 7fe2a23b3640 1 --2- 172.21.15.57:0/3685607594 >> v2:172.21.15.88:3300/0 conn(0x7fe29c159970 0x7fe29c159e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:56.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.505+0000 7fe2a23b3640 1 -- 172.21.15.57:0/3685607594 >> v1:172.21.15.88:6789/0 conn(0x7fe29c070c30 legacy=0x7fe29c071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:56.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.505+0000 7fe2a23b3640 1 -- 172.21.15.57:0/3685607594 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29c0bec10 con 0x7fe29c153c10 2023-10-30T14:45:56.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.505+0000 7fe2a23b3640 1 --2- 172.21.15.57:0/3685607594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe29c153c10 0x7fe29c065ba0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe2980097c0 tx=0x7fe29801c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c42a23cfc484e22f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:45:56.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a13b1640 1 -- 172.21.15.57:0/3685607594 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2980ab600 con 0x7fe29c153c10 2023-10-30T14:45:56.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a13b1640 1 -- 172.21.15.57:0/3685607594 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2980a7050 con 0x7fe29c153c10 2023-10-30T14:45:56.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a13b1640 1 -- 172.21.15.57:0/3685607594 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2980ab980 con 0x7fe29c153c10 2023-10-30T14:45:56.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a4617640 1 -- 172.21.15.57:0/3685607594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe29c153c10 msgr2=0x7fe29c065ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:56.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a4617640 1 --2- 172.21.15.57:0/3685607594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe29c153c10 0x7fe29c065ba0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe2980097c0 tx=0x7fe29801c3f0 comp rx=0 tx=0).stop 2023-10-30T14:45:56.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a4617640 1 -- 172.21.15.57:0/3685607594 shutdown_connections 2023-10-30T14:45:56.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a4617640 1 --2- 172.21.15.57:0/3685607594 >> v2:172.21.15.88:3300/0 conn(0x7fe29c159970 0x7fe29c159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:56.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a4617640 1 --2- 172.21.15.57:0/3685607594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe29c153c10 0x7fe29c065ba0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:56.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a4617640 1 -- 172.21.15.57:0/3685607594 >> 172.21.15.57:0/3685607594 conn(0x7fe29c0c7f60 msgr2=0x7fe29c0c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:45:56.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a4617640 1 -- 172.21.15.57:0/3685607594 shutdown_connections 2023-10-30T14:45:56.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a4617640 1 -- 172.21.15.57:0/3685607594 wait complete. 2023-10-30T14:45:56.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a4617640 1 Processor -- start 2023-10-30T14:45:56.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a4617640 1 -- start start 2023-10-30T14:45:56.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a4617640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe29c070c30 0x7fe29c1f2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:56.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a4617640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe29c153c10 0x7fe29c1eb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:56.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a4617640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe29c159970 0x7fe29c1ec300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:56.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a4617640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c15b650 con 0x7fe29c159970 2023-10-30T14:45:56.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a4617640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c149cf0 con 0x7fe29c070c30 2023-10-30T14:45:56.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a4617640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c1ec840 con 0x7fe29c153c10 2023-10-30T14:45:56.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a2bb4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe29c159970 0x7fe29c1ec300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:56.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a2bb4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe29c159970 0x7fe29c1ec300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33624/0 (socket says 172.21.15.57:33624) 2023-10-30T14:45:56.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a2bb4640 1 -- 172.21.15.57:0/3397212762 learned_addr learned my addr 172.21.15.57:0/3397212762 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:45:56.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a23b3640 1 --2- 172.21.15.57:0/3397212762 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe29c070c30 0x7fe29c1f2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:56.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.509+0000 7fe2a1bb2640 1 --2- 172.21.15.57:0/3397212762 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe29c153c10 0x7fe29c1eb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:56.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.513+0000 7fe2a2bb4640 1 -- 172.21.15.57:0/3397212762 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe29c153c10 msgr2=0x7fe29c1eb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:56.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.513+0000 7fe2a2bb4640 1 --2- 172.21.15.57:0/3397212762 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe29c153c10 0x7fe29c1eb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:56.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.513+0000 7fe2a2bb4640 1 -- 172.21.15.57:0/3397212762 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe29c070c30 msgr2=0x7fe29c1f2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:56.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.513+0000 7fe2a2bb4640 1 --2- 172.21.15.57:0/3397212762 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe29c070c30 0x7fe29c1f2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:56.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.513+0000 7fe2a2bb4640 1 -- 172.21.15.57:0/3397212762 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe298009420 con 0x7fe29c159970 2023-10-30T14:45:56.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.513+0000 7fe2a1bb2640 1 --2- 172.21.15.57:0/3397212762 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe29c153c10 0x7fe29c1eb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:45:56.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.513+0000 7fe2a2bb4640 1 --2- 172.21.15.57:0/3397212762 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe29c159970 0x7fe29c1ec300 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fe294009d30 tx=0x7fe29400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:45:56.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.513+0000 7fe292ffd640 1 -- 172.21.15.57:0/3397212762 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe294009ba0 con 0x7fe29c159970 2023-10-30T14:45:56.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.513+0000 7fe292ffd640 1 -- 172.21.15.57:0/3397212762 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe29400f040 con 0x7fe29c159970 2023-10-30T14:45:56.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.513+0000 7fe292ffd640 1 -- 172.21.15.57:0/3397212762 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2940146c0 con 0x7fe29c159970 2023-10-30T14:45:56.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.513+0000 7fe2a4617640 1 -- 172.21.15.57:0/3397212762 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe29c1f20d0 con 0x7fe29c159970 2023-10-30T14:45:56.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.513+0000 7fe2a4617640 1 -- 172.21.15.57:0/3397212762 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe29c14f0e0 con 0x7fe29c159970 2023-10-30T14:45:56.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.513+0000 7fe2a4617640 1 -- 172.21.15.57:0/3397212762 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe29c04e170 con 0x7fe29c159970 2023-10-30T14:45:56.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.513+0000 7fe292ffd640 1 -- 172.21.15.57:0/3397212762 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe294014820 con 0x7fe29c159970 2023-10-30T14:45:56.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.513+0000 7fe292ffd640 1 --2- 172.21.15.57:0/3397212762 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe280066470 0x7fe280068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:56.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.513+0000 7fe292ffd640 1 -- 172.21.15.57:0/3397212762 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7fe294084fc0 con 0x7fe29c159970 2023-10-30T14:45:56.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.517+0000 7fe2a23b3640 1 --2- 172.21.15.57:0/3397212762 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe280066470 0x7fe280068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:56.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.517+0000 7fe2a23b3640 1 --2- 172.21.15.57:0/3397212762 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe280066470 0x7fe280068930 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe2980ad000 tx=0x7fe2980a6000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:45:56.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.517+0000 7fe292ffd640 1 -- 172.21.15.57:0/3397212762 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe294051d30 con 0x7fe29c159970 2023-10-30T14:45:56.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.713+0000 7fe2a4617640 1 -- 172.21.15.57:0/3397212762 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe29c070150 con 0x7fe29c159970 2023-10-30T14:45:56.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.717+0000 7fe292ffd640 1 -- 172.21.15.57:0/3397212762 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+6511 (secure 0 0 0) 0x7fe294018070 con 0x7fe29c159970 2023-10-30T14:45:56.719 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:45:56.719 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:45:29.167395+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:active","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:45:56.719 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 172 2023-10-30T14:45:56.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.717+0000 7fe2a4617640 1 -- 172.21.15.57:0/3397212762 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe280066470 msgr2=0x7fe280068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:56.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.717+0000 7fe2a4617640 1 --2- 172.21.15.57:0/3397212762 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe280066470 0x7fe280068930 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe2980ad000 tx=0x7fe2980a6000 comp rx=0 tx=0).stop 2023-10-30T14:45:56.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.717+0000 7fe2a4617640 1 -- 172.21.15.57:0/3397212762 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe29c159970 msgr2=0x7fe29c1ec300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:56.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.717+0000 7fe2a4617640 1 --2- 172.21.15.57:0/3397212762 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe29c159970 0x7fe29c1ec300 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fe294009d30 tx=0x7fe29400c490 comp rx=0 tx=0).stop 2023-10-30T14:45:56.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.717+0000 7fe2a4617640 1 -- 172.21.15.57:0/3397212762 shutdown_connections 2023-10-30T14:45:56.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.717+0000 7fe2a4617640 1 --2- 172.21.15.57:0/3397212762 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe280066470 0x7fe280068930 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:56.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.717+0000 7fe2a4617640 1 --2- 172.21.15.57:0/3397212762 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe29c159970 0x7fe29c1ec300 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:56.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.717+0000 7fe2a4617640 1 --2- 172.21.15.57:0/3397212762 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe29c153c10 0x7fe29c1eb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:56.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.717+0000 7fe2a4617640 1 --2- 172.21.15.57:0/3397212762 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe29c070c30 0x7fe29c1f2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:56.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.717+0000 7fe2a4617640 1 -- 172.21.15.57:0/3397212762 >> 172.21.15.57:0/3397212762 conn(0x7fe29c0c7f60 msgr2=0x7fe29c14f830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:45:56.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.717+0000 7fe2a4617640 1 -- 172.21.15.57:0/3397212762 shutdown_connections 2023-10-30T14:45:56.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.717+0000 7fe2a4617640 1 -- 172.21.15.57:0/3397212762 wait complete. 2023-10-30T14:45:56.742 INFO:tasks.mds_thrash.fs.[cephfs]:thrashing max_mds: 2 -> 1 2023-10-30T14:45:56.742 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2023-10-30T14:45:56.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.897+0000 7f57744d6640 1 Processor -- start 2023-10-30T14:45:56.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.897+0000 7f57744d6640 1 -- start start 2023-10-30T14:45:56.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.897+0000 7f57744d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c150630 0x7f576c0c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:56.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.897+0000 7f57744d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f576c0c5be0 0x7f576c159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:56.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.897+0000 7f57744d6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576c157100 con 0x7f576c0c1430 2023-10-30T14:45:56.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.897+0000 7f57744d6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f576c150a10 con 0x7f576c0c5be0 2023-10-30T14:45:56.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.897+0000 7f57744d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576c072000 con 0x7f576c150630 2023-10-30T14:45:56.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.897+0000 7f5772a73640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f576c0c1430 0x7f576c0cacb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:53862/0 (socket says 172.21.15.57:53862) 2023-10-30T14:45:56.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.897+0000 7f5772a73640 1 -- 172.21.15.57:0/1793561337 learned_addr learned my addr 172.21.15.57:0/1793561337 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:45:56.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.897+0000 7f5772272640 1 --2- 172.21.15.57:0/1793561337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c150630 0x7f576c0c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:56.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.897+0000 7f5771a71640 1 --2- 172.21.15.57:0/1793561337 >> v2:172.21.15.88:3300/0 conn(0x7f576c0c5be0 0x7f576c159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:56.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.897+0000 7f5771270640 1 -- 172.21.15.57:0/1793561337 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923108121 0 0) 0x7f576c157100 con 0x7f576c0c1430 2023-10-30T14:45:56.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.897+0000 7f5771270640 1 -- 172.21.15.57:0/1793561337 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5758003530 con 0x7f576c0c1430 2023-10-30T14:45:56.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.897+0000 7f5771270640 1 -- 172.21.15.57:0/1793561337 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3968501788 0 0) 0x7f5758003530 con 0x7f576c0c1430 2023-10-30T14:45:56.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.897+0000 7f5772272640 1 -- 172.21.15.57:0/1793561337 >> v2:172.21.15.88:3300/0 conn(0x7f576c0c5be0 msgr2=0x7f576c159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:45:56.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.897+0000 7f5772272640 1 --2- 172.21.15.57:0/1793561337 >> v2:172.21.15.88:3300/0 conn(0x7f576c0c5be0 0x7f576c159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:56.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.897+0000 7f5772272640 1 -- 172.21.15.57:0/1793561337 >> v1:172.21.15.57:6789/0 conn(0x7f576c0c1430 legacy=0x7f576c0cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:56.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.897+0000 7f5772272640 1 -- 172.21.15.57:0/1793561337 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c156510 con 0x7f576c150630 2023-10-30T14:45:56.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.897+0000 7f5771a71640 1 --2- 172.21.15.57:0/1793561337 >> v2:172.21.15.88:3300/0 conn(0x7f576c0c5be0 0x7f576c159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:45:56.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.901+0000 7f5772272640 1 --2- 172.21.15.57:0/1793561337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c150630 0x7f576c0c56a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f57680097c0 tx=0x7f576801c310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6878b69ef859da5e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:45:56.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.901+0000 7f5771270640 1 -- 172.21.15.57:0/1793561337 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5764001a00 con 0x7f576c150630 2023-10-30T14:45:56.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.901+0000 7f5771270640 1 -- 172.21.15.57:0/1793561337 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57680a7050 con 0x7f576c150630 2023-10-30T14:45:56.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.901+0000 7f5771270640 1 -- 172.21.15.57:0/1793561337 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57680ab620 con 0x7f576c150630 2023-10-30T14:45:56.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.901+0000 7f57744d6640 1 -- 172.21.15.57:0/1793561337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c150630 msgr2=0x7f576c0c56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:56.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.901+0000 7f57744d6640 1 --2- 172.21.15.57:0/1793561337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c150630 0x7f576c0c56a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f57680097c0 tx=0x7f576801c310 comp rx=0 tx=0).stop 2023-10-30T14:45:56.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.901+0000 7f57744d6640 1 -- 172.21.15.57:0/1793561337 shutdown_connections 2023-10-30T14:45:56.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.901+0000 7f57744d6640 1 --2- 172.21.15.57:0/1793561337 >> v2:172.21.15.88:3300/0 conn(0x7f576c0c5be0 0x7f576c159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:56.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.901+0000 7f57744d6640 1 --2- 172.21.15.57:0/1793561337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c150630 0x7f576c0c56a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:56.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.901+0000 7f57744d6640 1 -- 172.21.15.57:0/1793561337 >> 172.21.15.57:0/1793561337 conn(0x7f576c06da40 msgr2=0x7f576c06de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:45:56.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.901+0000 7f57744d6640 1 -- 172.21.15.57:0/1793561337 shutdown_connections 2023-10-30T14:45:56.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.901+0000 7f57744d6640 1 -- 172.21.15.57:0/1793561337 wait complete. 2023-10-30T14:45:56.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f57744d6640 1 Processor -- start 2023-10-30T14:45:56.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f57744d6640 1 -- start start 2023-10-30T14:45:56.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f57744d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f576c0c1430 0x7f576c14f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:56.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f57744d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f576c0c5be0 0x7f576c1415b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:56.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f57744d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c150630 0x7f576c141af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:56.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f57744d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c1f2df0 con 0x7f576c0c5be0 2023-10-30T14:45:56.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f57744d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c1f1f00 con 0x7f576c0c1430 2023-10-30T14:45:56.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f57744d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576c1f21e0 con 0x7f576c150630 2023-10-30T14:45:56.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f5771a71640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f576c0c5be0 0x7f576c1415b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:56.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f5771a71640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f576c0c5be0 0x7f576c1415b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33638/0 (socket says 172.21.15.57:33638) 2023-10-30T14:45:56.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f5771a71640 1 -- 172.21.15.57:0/782513724 learned_addr learned my addr 172.21.15.57:0/782513724 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:45:56.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f5772a73640 1 --2- 172.21.15.57:0/782513724 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c150630 0x7f576c141af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:56.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f5772272640 1 --2- 172.21.15.57:0/782513724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f576c0c1430 0x7f576c14f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:56.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f5771a71640 1 -- 172.21.15.57:0/782513724 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c150630 msgr2=0x7f576c141af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:56.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f5771a71640 1 --2- 172.21.15.57:0/782513724 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c150630 0x7f576c141af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:56.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f5771a71640 1 -- 172.21.15.57:0/782513724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f576c0c1430 msgr2=0x7f576c14f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:56.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f5771a71640 1 --2- 172.21.15.57:0/782513724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f576c0c1430 0x7f576c14f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:56.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f5771a71640 1 -- 172.21.15.57:0/782513724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c009420 con 0x7f576c0c5be0 2023-10-30T14:45:56.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f5771a71640 1 --2- 172.21.15.57:0/782513724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f576c0c5be0 0x7f576c1415b0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f575c00ebe0 tx=0x7f575c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:45:56.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f5762ffd640 1 -- 172.21.15.57:0/782513724 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c00eee0 con 0x7f576c0c5be0 2023-10-30T14:45:56.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f5762ffd640 1 -- 172.21.15.57:0/782513724 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575c00cc30 con 0x7f576c0c5be0 2023-10-30T14:45:56.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f5762ffd640 1 -- 172.21.15.57:0/782513724 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c013470 con 0x7f576c0c5be0 2023-10-30T14:45:56.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f57744d6640 1 -- 172.21.15.57:0/782513724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5768009420 con 0x7f576c0c5be0 2023-10-30T14:45:56.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.905+0000 7f57744d6640 1 -- 172.21.15.57:0/782513724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f576c1f27a0 con 0x7f576c0c5be0 2023-10-30T14:45:56.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.909+0000 7f57744d6640 1 -- 172.21.15.57:0/782513724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f576c04e170 con 0x7f576c0c5be0 2023-10-30T14:45:56.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.909+0000 7f5762ffd640 1 -- 172.21.15.57:0/782513724 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f575c0135d0 con 0x7f576c0c5be0 2023-10-30T14:45:56.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.909+0000 7f5762ffd640 1 --2- 172.21.15.57:0/782513724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f57500664a0 0x7f5750068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:56.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.909+0000 7f5762ffd640 1 -- 172.21.15.57:0/782513724 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f575c0846d0 con 0x7f576c0c5be0 2023-10-30T14:45:56.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.909+0000 7f5772272640 1 --2- 172.21.15.57:0/782513724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f57500664a0 0x7f5750068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:56.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.913+0000 7f5772272640 1 --2- 172.21.15.57:0/782513724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f57500664a0 0x7f5750068960 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f57680ad000 tx=0x7f57680a6000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:45:56.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:56.917+0000 7f5762ffd640 1 -- 172.21.15.57:0/782513724 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f575c051580 con 0x7f576c0c5be0 2023-10-30T14:45:57.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.109+0000 7f57744d6640 1 -- 172.21.15.57:0/782513724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f576c1f5410 con 0x7f576c0c5be0 2023-10-30T14:45:57.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.249+0000 7f5762ffd640 1 -- 172.21.15.57:0/782513724 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v173) v1 ==== 105+0+0 (secure 0 0 0) 0x7f575c054ba0 con 0x7f576c0c5be0 2023-10-30T14:45:57.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.249+0000 7f57744d6640 1 -- 172.21.15.57:0/782513724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f57500664a0 msgr2=0x7f5750068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:57.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.249+0000 7f57744d6640 1 --2- 172.21.15.57:0/782513724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f57500664a0 0x7f5750068960 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f57680ad000 tx=0x7f57680a6000 comp rx=0 tx=0).stop 2023-10-30T14:45:57.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.249+0000 7f57744d6640 1 -- 172.21.15.57:0/782513724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f576c0c5be0 msgr2=0x7f576c1415b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:57.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.249+0000 7f57744d6640 1 --2- 172.21.15.57:0/782513724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f576c0c5be0 0x7f576c1415b0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f575c00ebe0 tx=0x7f575c00c520 comp rx=0 tx=0).stop 2023-10-30T14:45:57.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.249+0000 7f57744d6640 1 -- 172.21.15.57:0/782513724 shutdown_connections 2023-10-30T14:45:57.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.249+0000 7f57744d6640 1 --2- 172.21.15.57:0/782513724 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f57500664a0 0x7f5750068960 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:57.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.249+0000 7f57744d6640 1 --2- 172.21.15.57:0/782513724 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f576c150630 0x7f576c141af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:57.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.249+0000 7f57744d6640 1 --2- 172.21.15.57:0/782513724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f576c0c5be0 0x7f576c1415b0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:57.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.249+0000 7f57744d6640 1 --2- 172.21.15.57:0/782513724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f576c0c1430 0x7f576c14f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:57.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.249+0000 7f57744d6640 1 -- 172.21.15.57:0/782513724 >> 172.21.15.57:0/782513724 conn(0x7f576c06da40 msgr2=0x7f576c144e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:45:57.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.253+0000 7f57744d6640 1 -- 172.21.15.57:0/782513724 shutdown_connections 2023-10-30T14:45:57.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.253+0000 7f57744d6640 1 -- 172.21.15.57:0/782513724 wait complete. 2023-10-30T14:45:57.275 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:45:57.275 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:45:57.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.417+0000 7f1d98808640 1 Processor -- start 2023-10-30T14:45:57.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.417+0000 7f1d98808640 1 -- start start 2023-10-30T14:45:57.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.417+0000 7f1d98808640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1d9406d4f0 0x7f1d940706a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:57.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.417+0000 7f1d98808640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d94070be0 con 0x7f1d9406dfc0 2023-10-30T14:45:57.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.417+0000 7f1d98808640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9406fc30 con 0x7f1d940c5460 2023-10-30T14:45:57.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.417+0000 7f1d98808640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d9415b170 con 0x7f1d9406d4f0 2023-10-30T14:45:57.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.417+0000 7f1d92d9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1d9406dfc0 0x7f1d9415aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:53878/0 (socket says 172.21.15.57:53878) 2023-10-30T14:45:57.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.417+0000 7f1d92d9d640 1 -- 172.21.15.57:0/3456126186 learned_addr learned my addr 172.21.15.57:0/3456126186 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:45:57.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.417+0000 7f1d9359e640 1 --2- 172.21.15.57:0/3456126186 >> v2:172.21.15.57:3300/0 conn(0x7f1d9406d4f0 0x7f1d940706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:57.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.417+0000 7f1d9259c640 1 -- 172.21.15.57:0/3456126186 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598117133 0 0) 0x7f1d94070be0 con 0x7f1d9406dfc0 2023-10-30T14:45:57.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.417+0000 7f1d9259c640 1 -- 172.21.15.57:0/3456126186 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d80003530 con 0x7f1d9406dfc0 2023-10-30T14:45:57.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.417+0000 7f1d9359e640 1 -- 172.21.15.57:0/3456126186 >> v1:172.21.15.88:6789/0 conn(0x7f1d940c5460 legacy=0x7f1d940c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:57.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.417+0000 7f1d9359e640 1 -- 172.21.15.57:0/3456126186 >> v1:172.21.15.57:6789/0 conn(0x7f1d9406dfc0 legacy=0x7f1d9415aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:57.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.417+0000 7f1d9359e640 1 -- 172.21.15.57:0/3456126186 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9415b360 con 0x7f1d9406d4f0 2023-10-30T14:45:57.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.421+0000 7f1d9359e640 1 --2- 172.21.15.57:0/3456126186 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d9406d4f0 0x7f1d940706a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c009d90 tx=0x7f1d8c01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd8b2432bdc2037b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:45:57.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.421+0000 7f1d9259c640 1 -- 172.21.15.57:0/3456126186 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c0a5900 con 0x7f1d9406d4f0 2023-10-30T14:45:57.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.421+0000 7f1d9259c640 1 -- 172.21.15.57:0/3456126186 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d8c0a5a60 con 0x7f1d9406d4f0 2023-10-30T14:45:57.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.421+0000 7f1d9259c640 1 -- 172.21.15.57:0/3456126186 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c0a5df0 con 0x7f1d9406d4f0 2023-10-30T14:45:57.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.421+0000 7f1d98808640 1 -- 172.21.15.57:0/3456126186 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d9406d4f0 msgr2=0x7f1d940706a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:57.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.421+0000 7f1d98808640 1 --2- 172.21.15.57:0/3456126186 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d9406d4f0 0x7f1d940706a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c009d90 tx=0x7f1d8c01c950 comp rx=0 tx=0).stop 2023-10-30T14:45:57.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.421+0000 7f1d98808640 1 -- 172.21.15.57:0/3456126186 shutdown_connections 2023-10-30T14:45:57.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.421+0000 7f1d98808640 1 --2- 172.21.15.57:0/3456126186 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d9406d4f0 0x7f1d940706a0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:57.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.421+0000 7f1d98808640 1 -- 172.21.15.57:0/3456126186 >> 172.21.15.57:0/3456126186 conn(0x7f1d940caae0 msgr2=0x7f1d940caec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:45:57.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.421+0000 7f1d98808640 1 -- 172.21.15.57:0/3456126186 shutdown_connections 2023-10-30T14:45:57.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.421+0000 7f1d98808640 1 -- 172.21.15.57:0/3456126186 wait complete. 2023-10-30T14:45:57.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.421+0000 7f1d98808640 1 Processor -- start 2023-10-30T14:45:57.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d98808640 1 -- start start 2023-10-30T14:45:57.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d98808640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d9406d4f0 0x7f1d940c3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:57.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d98808640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d9406dfc0 0x7f1d94160800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:57.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d98808640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1d940c5460 0x7f1d941f1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:57.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d98808640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d940c2dd0 con 0x7f1d9406dfc0 2023-10-30T14:45:57.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d98808640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d940c4c60 con 0x7f1d9406d4f0 2023-10-30T14:45:57.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d98808640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d940c3ba0 con 0x7f1d940c5460 2023-10-30T14:45:57.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d8bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d9406dfc0 0x7f1d94160800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:57.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d92d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d9406d4f0 0x7f1d940c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:57.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d8bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d9406dfc0 0x7f1d94160800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33646/0 (socket says 172.21.15.57:33646) 2023-10-30T14:45:57.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d8bfff640 1 -- 172.21.15.57:0/1868031752 learned_addr learned my addr 172.21.15.57:0/1868031752 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:45:57.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d9359e640 1 --2- 172.21.15.57:0/1868031752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1d940c5460 0x7f1d941f1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:57.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d8bfff640 1 -- 172.21.15.57:0/1868031752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1d940c5460 msgr2=0x7f1d941f1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:57.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d8bfff640 1 --2- 172.21.15.57:0/1868031752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1d940c5460 0x7f1d941f1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:57.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d8bfff640 1 -- 172.21.15.57:0/1868031752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d9406d4f0 msgr2=0x7f1d940c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:57.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d8bfff640 1 --2- 172.21.15.57:0/1868031752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d9406d4f0 0x7f1d940c3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:57.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d8bfff640 1 -- 172.21.15.57:0/1868031752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c009a70 con 0x7f1d9406dfc0 2023-10-30T14:45:57.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d8bfff640 1 --2- 172.21.15.57:0/1868031752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d9406dfc0 0x7f1d94160800 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c00c3f0 tx=0x7f1d7c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:45:57.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d8b7fe640 1 -- 172.21.15.57:0/1868031752 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c0052d0 con 0x7f1d9406dfc0 2023-10-30T14:45:57.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d8b7fe640 1 -- 172.21.15.57:0/1868031752 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d7c005430 con 0x7f1d9406dfc0 2023-10-30T14:45:57.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d8b7fe640 1 -- 172.21.15.57:0/1868031752 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c003840 con 0x7f1d9406dfc0 2023-10-30T14:45:57.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d98808640 1 -- 172.21.15.57:0/1868031752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d94161d80 con 0x7f1d9406dfc0 2023-10-30T14:45:57.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d98808640 1 -- 172.21.15.57:0/1868031752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d940c4610 con 0x7f1d9406dfc0 2023-10-30T14:45:57.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.425+0000 7f1d98808640 1 -- 172.21.15.57:0/1868031752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d941f1460 con 0x7f1d9406dfc0 2023-10-30T14:45:57.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.429+0000 7f1d8b7fe640 1 -- 172.21.15.57:0/1868031752 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1d7c0039a0 con 0x7f1d9406dfc0 2023-10-30T14:45:57.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.429+0000 7f1d8b7fe640 1 --2- 172.21.15.57:0/1868031752 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1d740664a0 0x7f1d74068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:57.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.429+0000 7f1d8b7fe640 1 -- 172.21.15.57:0/1868031752 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f1d7c083990 con 0x7f1d9406dfc0 2023-10-30T14:45:57.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.429+0000 7f1d92d9d640 1 --2- 172.21.15.57:0/1868031752 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1d740664a0 0x7f1d74068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:57.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.429+0000 7f1d92d9d640 1 --2- 172.21.15.57:0/1868031752 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1d740664a0 0x7f1d74068960 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f1d840011f0 tx=0x7f1d84007d40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:45:57.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.433+0000 7f1d8b7fe640 1 -- 172.21.15.57:0/1868031752 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1d7c050840 con 0x7f1d9406dfc0 2023-10-30T14:45:57.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.625+0000 7f1d98808640 1 -- 172.21.15.57:0/1868031752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d94070150 con 0x7f1d9406dfc0 2023-10-30T14:45:57.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.625+0000 7f1d8b7fe640 1 -- 172.21.15.57:0/1868031752 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6513 (secure 0 0 0) 0x7f1d7c053e60 con 0x7f1d9406dfc0 2023-10-30T14:45:57.631 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:45:57.631 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:45:57.259750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:stopping","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:45:57.632 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 174 2023-10-30T14:45:57.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.629+0000 7f1d98808640 1 -- 172.21.15.57:0/1868031752 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1d740664a0 msgr2=0x7f1d74068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:57.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.629+0000 7f1d98808640 1 --2- 172.21.15.57:0/1868031752 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1d740664a0 0x7f1d74068960 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f1d840011f0 tx=0x7f1d84007d40 comp rx=0 tx=0).stop 2023-10-30T14:45:57.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.629+0000 7f1d98808640 1 -- 172.21.15.57:0/1868031752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d9406dfc0 msgr2=0x7f1d94160800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:57.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.629+0000 7f1d98808640 1 --2- 172.21.15.57:0/1868031752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d9406dfc0 0x7f1d94160800 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c00c3f0 tx=0x7f1d7c00c8b0 comp rx=0 tx=0).stop 2023-10-30T14:45:57.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.629+0000 7f1d98808640 1 -- 172.21.15.57:0/1868031752 shutdown_connections 2023-10-30T14:45:57.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.629+0000 7f1d98808640 1 --2- 172.21.15.57:0/1868031752 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f1d740664a0 0x7f1d74068960 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:57.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.629+0000 7f1d98808640 1 --2- 172.21.15.57:0/1868031752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1d940c5460 0x7f1d941f1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:57.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.629+0000 7f1d98808640 1 --2- 172.21.15.57:0/1868031752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d9406dfc0 0x7f1d94160800 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:57.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.629+0000 7f1d98808640 1 --2- 172.21.15.57:0/1868031752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d9406d4f0 0x7f1d940c3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:57.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.629+0000 7f1d98808640 1 -- 172.21.15.57:0/1868031752 >> 172.21.15.57:0/1868031752 conn(0x7f1d940caae0 msgr2=0x7f1d9415a330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:45:57.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.629+0000 7f1d98808640 1 -- 172.21.15.57:0/1868031752 shutdown_connections 2023-10-30T14:45:57.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:57.629+0000 7f1d98808640 1 -- 172.21.15.57:0/1868031752 wait complete. 2023-10-30T14:45:57.651 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T14:45:57.652 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 174, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 174, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:stopping", "state_seq": 44 }, "gid_7536": { "addr": "172.21.15.57:6837/816502245", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 816502245, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 816502245, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 7536, "incarnation": 164, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:45:57.259750+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7536, "mds_1": 6747 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6839/1078372885", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1078372885, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1078372885, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 170, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6716, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.57:6834/697507447", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 697507447, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 697507447, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 156, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7377, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:45:59.653 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:45:59.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.797+0000 7f24758d6640 1 Processor -- start 2023-10-30T14:45:59.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.797+0000 7f24758d6640 1 -- start start 2023-10-30T14:45:59.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.801+0000 7f24758d6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2470153c80 0x7f247015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:59.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.801+0000 7f24758d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24700c42a0 0x7f24700ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:59.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.801+0000 7f24758d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2470159f20 con 0x7f24700c4a10 2023-10-30T14:45:59.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.801+0000 7f24758d6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f247015dec0 con 0x7f2470153c80 2023-10-30T14:45:59.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.801+0000 7f24758d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24701599a0 con 0x7f24700c42a0 2023-10-30T14:45:59.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.801+0000 7f246effd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2470153c80 0x7f247015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:59.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.801+0000 7f246effd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2470153c80 0x7f247015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33648/0 (socket says 172.21.15.57:33648) 2023-10-30T14:45:59.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.801+0000 7f246effd640 1 -- 172.21.15.57:0/1613588201 learned_addr learned my addr 172.21.15.57:0/1613588201 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:45:59.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.801+0000 7f246f7fe640 1 --2- 172.21.15.57:0/1613588201 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24700c42a0 0x7f24700ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:59.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.801+0000 7f246effd640 1 -- 172.21.15.57:0/1613588201 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24700c42a0 msgr2=0x7f24700ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:45:59.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.801+0000 7f246effd640 1 --2- 172.21.15.57:0/1613588201 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24700c42a0 0x7f24700ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:59.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.801+0000 7f246effd640 1 -- 172.21.15.57:0/1613588201 >> v1:172.21.15.88:6789/0 conn(0x7f24700c4a10 legacy=0x7f247015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:45:59.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.801+0000 7f246effd640 1 -- 172.21.15.57:0/1613588201 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247015d620 con 0x7f2470153c80 2023-10-30T14:45:59.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.801+0000 7f246f7fe640 1 --2- 172.21.15.57:0/1613588201 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24700c42a0 0x7f24700ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:45:59.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.801+0000 7f246effd640 1 --2- 172.21.15.57:0/1613588201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2470153c80 0x7f247015cf90 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f2460009870 tx=0x7f246001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6ea8102ffd13065 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:45:59.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f246e7fc640 1 -- 172.21.15.57:0/1613588201 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24600a89b0 con 0x7f2470153c80 2023-10-30T14:45:59.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f246e7fc640 1 -- 172.21.15.57:0/1613588201 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24600a8b10 con 0x7f2470153c80 2023-10-30T14:45:59.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f246e7fc640 1 -- 172.21.15.57:0/1613588201 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24600a8ea0 con 0x7f2470153c80 2023-10-30T14:45:59.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f24758d6640 1 -- 172.21.15.57:0/1613588201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2470153c80 msgr2=0x7f247015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:59.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f24758d6640 1 --2- 172.21.15.57:0/1613588201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2470153c80 0x7f247015cf90 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f2460009870 tx=0x7f246001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:45:59.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f24758d6640 1 -- 172.21.15.57:0/1613588201 shutdown_connections 2023-10-30T14:45:59.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f24758d6640 1 --2- 172.21.15.57:0/1613588201 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24700c42a0 0x7f24700ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:59.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f24758d6640 1 --2- 172.21.15.57:0/1613588201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2470153c80 0x7f247015cf90 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:59.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f24758d6640 1 -- 172.21.15.57:0/1613588201 >> 172.21.15.57:0/1613588201 conn(0x7f24700715c0 msgr2=0x7f24700719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:45:59.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f24758d6640 1 -- 172.21.15.57:0/1613588201 shutdown_connections 2023-10-30T14:45:59.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f24758d6640 1 -- 172.21.15.57:0/1613588201 wait complete. 2023-10-30T14:45:59.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f24758d6640 1 Processor -- start 2023-10-30T14:45:59.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f24758d6640 1 -- start start 2023-10-30T14:45:59.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f24758d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24700c42a0 0x7f24701ee760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:59.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f24758d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24700c4a10 0x7f24700ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:59.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f24758d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2470153c80 0x7f24700ce830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:59.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f24758d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24701f2280 con 0x7f24700c4a10 2023-10-30T14:45:59.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f24758d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24701f2560 con 0x7f2470153c80 2023-10-30T14:45:59.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f24758d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24701f2840 con 0x7f24700c42a0 2023-10-30T14:45:59.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f24667fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24700c4a10 0x7f24700ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:59.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f24667fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24700c4a10 0x7f24700ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33664/0 (socket says 172.21.15.57:33664) 2023-10-30T14:45:59.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f24667fc640 1 -- 172.21.15.57:0/941180598 learned_addr learned my addr 172.21.15.57:0/941180598 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:45:59.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.805+0000 7f246effd640 1 --2- 172.21.15.57:0/941180598 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24700c42a0 0x7f24701ee760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:59.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.809+0000 7f246f7fe640 1 --2- 172.21.15.57:0/941180598 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2470153c80 0x7f24700ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:59.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.809+0000 7f24667fc640 1 -- 172.21.15.57:0/941180598 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24700c42a0 msgr2=0x7f24701ee760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:59.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.809+0000 7f24667fc640 1 --2- 172.21.15.57:0/941180598 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24700c42a0 0x7f24701ee760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:59.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.809+0000 7f24667fc640 1 -- 172.21.15.57:0/941180598 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2470153c80 msgr2=0x7f24700ce830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:45:59.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.809+0000 7f24667fc640 1 --2- 172.21.15.57:0/941180598 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2470153c80 0x7f24700ce830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:45:59.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.809+0000 7f24667fc640 1 -- 172.21.15.57:0/941180598 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24600094d0 con 0x7f24700c4a10 2023-10-30T14:45:59.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.809+0000 7f24667fc640 1 --2- 172.21.15.57:0/941180598 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24700c4a10 0x7f24700ce2f0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f245800c7a0 tx=0x7f245800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:45:59.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.809+0000 7f246cff9640 1 -- 172.21.15.57:0/941180598 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245800aa50 con 0x7f24700c4a10 2023-10-30T14:45:59.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.809+0000 7f246cff9640 1 -- 172.21.15.57:0/941180598 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245800abb0 con 0x7f24700c4a10 2023-10-30T14:45:59.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.809+0000 7f24758d6640 1 -- 172.21.15.57:0/941180598 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24701f2b20 con 0x7f24700c4a10 2023-10-30T14:45:59.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.809+0000 7f246cff9640 1 -- 172.21.15.57:0/941180598 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2458012420 con 0x7f24700c4a10 2023-10-30T14:45:59.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.809+0000 7f24758d6640 1 -- 172.21.15.57:0/941180598 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247014f370 con 0x7f24700c4a10 2023-10-30T14:45:59.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.809+0000 7f24758d6640 1 -- 172.21.15.57:0/941180598 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247004e170 con 0x7f24700c4a10 2023-10-30T14:45:59.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.813+0000 7f246cff9640 1 -- 172.21.15.57:0/941180598 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2458012580 con 0x7f24700c4a10 2023-10-30T14:45:59.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.813+0000 7f246cff9640 1 --2- 172.21.15.57:0/941180598 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f24500664a0 0x7f2450068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:45:59.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.813+0000 7f246cff9640 1 -- 172.21.15.57:0/941180598 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f2458084660 con 0x7f24700c4a10 2023-10-30T14:45:59.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.813+0000 7f246effd640 1 --2- 172.21.15.57:0/941180598 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f24500664a0 0x7f2450068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:45:59.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.813+0000 7f246effd640 1 --2- 172.21.15.57:0/941180598 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f24500664a0 0x7f2450068960 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f24600b2000 tx=0x7f2460009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:45:59.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:45:59.817+0000 7f246cff9640 1 -- 172.21.15.57:0/941180598 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2458051500 con 0x7f24700c4a10 2023-10-30T14:46:00.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:00.009+0000 7f24758d6640 1 -- 172.21.15.57:0/941180598 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24700cad40 con 0x7f24700c4a10 2023-10-30T14:46:00.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:00.013+0000 7f246cff9640 1 -- 172.21.15.57:0/941180598 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6513 (secure 0 0 0) 0x7f2458016020 con 0x7f24700c4a10 2023-10-30T14:46:00.015 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:00.015 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:45:57.259750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:stopping","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:00.016 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 174 2023-10-30T14:46:00.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:00.013+0000 7f24758d6640 1 -- 172.21.15.57:0/941180598 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f24500664a0 msgr2=0x7f2450068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:00.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:00.013+0000 7f24758d6640 1 --2- 172.21.15.57:0/941180598 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f24500664a0 0x7f2450068960 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f24600b2000 tx=0x7f2460009040 comp rx=0 tx=0).stop 2023-10-30T14:46:00.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:00.013+0000 7f24758d6640 1 -- 172.21.15.57:0/941180598 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24700c4a10 msgr2=0x7f24700ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:00.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:00.013+0000 7f24758d6640 1 --2- 172.21.15.57:0/941180598 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24700c4a10 0x7f24700ce2f0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f245800c7a0 tx=0x7f245800cc60 comp rx=0 tx=0).stop 2023-10-30T14:46:00.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:00.013+0000 7f24758d6640 1 -- 172.21.15.57:0/941180598 shutdown_connections 2023-10-30T14:46:00.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:00.013+0000 7f24758d6640 1 --2- 172.21.15.57:0/941180598 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f24500664a0 0x7f2450068960 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:00.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:00.013+0000 7f24758d6640 1 --2- 172.21.15.57:0/941180598 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2470153c80 0x7f24700ce830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:00.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:00.013+0000 7f24758d6640 1 --2- 172.21.15.57:0/941180598 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24700c4a10 0x7f24700ce2f0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:00.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:00.013+0000 7f24758d6640 1 --2- 172.21.15.57:0/941180598 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24700c42a0 0x7f24701ee760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:00.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:00.013+0000 7f24758d6640 1 -- 172.21.15.57:0/941180598 >> 172.21.15.57:0/941180598 conn(0x7f24700715c0 msgr2=0x7f2470157bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:00.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:00.013+0000 7f24758d6640 1 -- 172.21.15.57:0/941180598 shutdown_connections 2023-10-30T14:46:00.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:00.013+0000 7f24758d6640 1 -- 172.21.15.57:0/941180598 wait complete. 2023-10-30T14:46:00.036 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:46:02.037 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:02.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.193+0000 7faa92cd6640 1 Processor -- start 2023-10-30T14:46:02.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.193+0000 7faa92cd6640 1 -- start start 2023-10-30T14:46:02.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.193+0000 7faa92cd6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7faa8c151840 0x7faa8c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:02.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.193+0000 7faa92cd6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa8c151c20 0x7faa8c072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:02.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.193+0000 7faa92cd6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faa8c0cb8b0 0x7faa8c072540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:02.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.193+0000 7faa92cd6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa8c0c5030 con 0x7faa8c151840 2023-10-30T14:46:02.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.193+0000 7faa92cd6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa8c157830 con 0x7faa8c0cb8b0 2023-10-30T14:46:02.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.193+0000 7faa92cd6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c157b10 con 0x7faa8c151c20 2023-10-30T14:46:02.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.193+0000 7faa90a72640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7faa8c151840 0x7faa8c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:02.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.193+0000 7faa90a72640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7faa8c151840 0x7faa8c0c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33672/0 (socket says 172.21.15.57:33672) 2023-10-30T14:46:02.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.193+0000 7faa90a72640 1 -- 172.21.15.57:0/3693542993 learned_addr learned my addr 172.21.15.57:0/3693542993 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:02.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.197+0000 7faa8bfff640 1 --2- 172.21.15.57:0/3693542993 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa8c151c20 0x7faa8c072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:02.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.197+0000 7faa91273640 1 --2- 172.21.15.57:0/3693542993 >> v2:172.21.15.88:3300/0 conn(0x7faa8c0cb8b0 0x7faa8c072540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:02.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.197+0000 7faa90a72640 1 -- 172.21.15.57:0/3693542993 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa8c151c20 msgr2=0x7faa8c072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:02.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.197+0000 7faa90a72640 1 --2- 172.21.15.57:0/3693542993 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa8c151c20 0x7faa8c072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:02.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.197+0000 7faa90a72640 1 -- 172.21.15.57:0/3693542993 >> v2:172.21.15.88:3300/0 conn(0x7faa8c0cb8b0 msgr2=0x7faa8c072540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:02.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.197+0000 7faa90a72640 1 --2- 172.21.15.57:0/3693542993 >> v2:172.21.15.88:3300/0 conn(0x7faa8c0cb8b0 0x7faa8c072540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:02.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.197+0000 7faa90a72640 1 -- 172.21.15.57:0/3693542993 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c157df0 con 0x7faa8c151840 2023-10-30T14:46:02.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.197+0000 7faa90a72640 1 --2- 172.21.15.57:0/3693542993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa8c151840 0x7faa8c0c49a0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0cb820 tx=0x7faa8001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d51e484519bf9e0a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:02.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.197+0000 7faa8b7fe640 1 -- 172.21.15.57:0/3693542993 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa800a89b0 con 0x7faa8c151840 2023-10-30T14:46:02.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.197+0000 7faa8b7fe640 1 -- 172.21.15.57:0/3693542993 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa800a8b10 con 0x7faa8c151840 2023-10-30T14:46:02.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.197+0000 7faa8b7fe640 1 -- 172.21.15.57:0/3693542993 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa800a8ea0 con 0x7faa8c151840 2023-10-30T14:46:02.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa92cd6640 1 -- 172.21.15.57:0/3693542993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa8c151840 msgr2=0x7faa8c0c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:02.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa92cd6640 1 --2- 172.21.15.57:0/3693542993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa8c151840 0x7faa8c0c49a0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0cb820 tx=0x7faa8001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:46:02.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa92cd6640 1 -- 172.21.15.57:0/3693542993 shutdown_connections 2023-10-30T14:46:02.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa92cd6640 1 --2- 172.21.15.57:0/3693542993 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa8c151c20 0x7faa8c072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:02.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa92cd6640 1 --2- 172.21.15.57:0/3693542993 >> v2:172.21.15.88:3300/0 conn(0x7faa8c0cb8b0 0x7faa8c072540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:02.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa92cd6640 1 --2- 172.21.15.57:0/3693542993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa8c151840 0x7faa8c0c49a0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:02.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa92cd6640 1 -- 172.21.15.57:0/3693542993 >> 172.21.15.57:0/3693542993 conn(0x7faa8c0cac70 msgr2=0x7faa8c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:02.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa92cd6640 1 -- 172.21.15.57:0/3693542993 shutdown_connections 2023-10-30T14:46:02.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa92cd6640 1 -- 172.21.15.57:0/3693542993 wait complete. 2023-10-30T14:46:02.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa92cd6640 1 Processor -- start 2023-10-30T14:46:02.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa92cd6640 1 -- start start 2023-10-30T14:46:02.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa92cd6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa8c0cb8b0 0x7faa8c1f2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:02.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa92cd6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa8c151840 0x7faa8c1489f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:02.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa92cd6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa8c151c20 0x7faa8c0c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:02.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa92cd6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c152340 con 0x7faa8c151840 2023-10-30T14:46:02.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa92cd6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c1f2f70 con 0x7faa8c151c20 2023-10-30T14:46:02.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa92cd6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c153f00 con 0x7faa8c0cb8b0 2023-10-30T14:46:02.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa8bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa8c151840 0x7faa8c1489f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:02.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa8bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa8c151840 0x7faa8c1489f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33674/0 (socket says 172.21.15.57:33674) 2023-10-30T14:46:02.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa8bfff640 1 -- 172.21.15.57:0/2624503716 learned_addr learned my addr 172.21.15.57:0/2624503716 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:02.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa91273640 1 --2- 172.21.15.57:0/2624503716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa8c151c20 0x7faa8c0c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:02.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.201+0000 7faa90a72640 1 --2- 172.21.15.57:0/2624503716 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa8c0cb8b0 0x7faa8c1f2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:02.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.205+0000 7faa8bfff640 1 -- 172.21.15.57:0/2624503716 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa8c0cb8b0 msgr2=0x7faa8c1f2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:02.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.205+0000 7faa8bfff640 1 --2- 172.21.15.57:0/2624503716 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa8c0cb8b0 0x7faa8c1f2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:02.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.205+0000 7faa8bfff640 1 -- 172.21.15.57:0/2624503716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa8c151c20 msgr2=0x7faa8c0c9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:02.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.205+0000 7faa8bfff640 1 --2- 172.21.15.57:0/2624503716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa8c151c20 0x7faa8c0c9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:02.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.205+0000 7faa8bfff640 1 -- 172.21.15.57:0/2624503716 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa800094d0 con 0x7faa8c151840 2023-10-30T14:46:02.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.205+0000 7faa8bfff640 1 --2- 172.21.15.57:0/2624503716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa8c151840 0x7faa8c1489f0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7faa7800d7d0 tx=0x7faa7800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:02.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.205+0000 7faa897fa640 1 -- 172.21.15.57:0/2624503716 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7800b6c0 con 0x7faa8c151840 2023-10-30T14:46:02.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.205+0000 7faa897fa640 1 -- 172.21.15.57:0/2624503716 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7800b820 con 0x7faa8c151840 2023-10-30T14:46:02.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.205+0000 7faa897fa640 1 -- 172.21.15.57:0/2624503716 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7800bbb0 con 0x7faa8c151840 2023-10-30T14:46:02.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.205+0000 7faa92cd6640 1 -- 172.21.15.57:0/2624503716 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa8c156cb0 con 0x7faa8c151840 2023-10-30T14:46:02.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.205+0000 7faa92cd6640 1 -- 172.21.15.57:0/2624503716 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8c1571f0 con 0x7faa8c151840 2023-10-30T14:46:02.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.205+0000 7faa92cd6640 1 -- 172.21.15.57:0/2624503716 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa8c072c70 con 0x7faa8c151840 2023-10-30T14:46:02.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.205+0000 7faa897fa640 1 -- 172.21.15.57:0/2624503716 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faa7800bd10 con 0x7faa8c151840 2023-10-30T14:46:02.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.205+0000 7faa897fa640 1 --2- 172.21.15.57:0/2624503716 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa700664a0 0x7faa70068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:02.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.205+0000 7faa897fa640 1 -- 172.21.15.57:0/2624503716 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7faa780848f0 con 0x7faa8c151840 2023-10-30T14:46:02.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.209+0000 7faa90a72640 1 --2- 172.21.15.57:0/2624503716 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa700664a0 0x7faa70068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:02.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.209+0000 7faa90a72640 1 --2- 172.21.15.57:0/2624503716 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa700664a0 0x7faa70068960 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7faa800b2000 tx=0x7faa80009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:02.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.209+0000 7faa897fa640 1 -- 172.21.15.57:0/2624503716 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faa780517a0 con 0x7faa8c151840 2023-10-30T14:46:02.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.449+0000 7faa92cd6640 1 -- 172.21.15.57:0/2624503716 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa8c158380 con 0x7faa8c151840 2023-10-30T14:46:02.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.449+0000 7faa897fa640 1 -- 172.21.15.57:0/2624503716 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6513 (secure 0 0 0) 0x7faa78054dc0 con 0x7faa8c151840 2023-10-30T14:46:02.452 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:02.453 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:45:57.259750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:stopping","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:02.453 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 174 2023-10-30T14:46:02.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.453+0000 7faa92cd6640 1 -- 172.21.15.57:0/2624503716 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa700664a0 msgr2=0x7faa70068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:02.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.453+0000 7faa92cd6640 1 --2- 172.21.15.57:0/2624503716 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa700664a0 0x7faa70068960 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7faa800b2000 tx=0x7faa80009040 comp rx=0 tx=0).stop 2023-10-30T14:46:02.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.453+0000 7faa92cd6640 1 -- 172.21.15.57:0/2624503716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa8c151840 msgr2=0x7faa8c1489f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:02.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.453+0000 7faa92cd6640 1 --2- 172.21.15.57:0/2624503716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa8c151840 0x7faa8c1489f0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7faa7800d7d0 tx=0x7faa7800dc90 comp rx=0 tx=0).stop 2023-10-30T14:46:02.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.453+0000 7faa92cd6640 1 -- 172.21.15.57:0/2624503716 shutdown_connections 2023-10-30T14:46:02.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.453+0000 7faa92cd6640 1 --2- 172.21.15.57:0/2624503716 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa700664a0 0x7faa70068960 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:02.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.453+0000 7faa92cd6640 1 --2- 172.21.15.57:0/2624503716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa8c151c20 0x7faa8c0c9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:02.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.453+0000 7faa92cd6640 1 --2- 172.21.15.57:0/2624503716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa8c151840 0x7faa8c1489f0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:02.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.453+0000 7faa92cd6640 1 --2- 172.21.15.57:0/2624503716 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa8c0cb8b0 0x7faa8c1f2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:02.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.453+0000 7faa92cd6640 1 -- 172.21.15.57:0/2624503716 >> 172.21.15.57:0/2624503716 conn(0x7faa8c0cac70 msgr2=0x7faa8c0c59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:02.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.453+0000 7faa92cd6640 1 -- 172.21.15.57:0/2624503716 shutdown_connections 2023-10-30T14:46:02.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:02.453+0000 7faa92cd6640 1 -- 172.21.15.57:0/2624503716 wait complete. 2023-10-30T14:46:02.486 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:46:04.487 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:04.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db9b3e640 1 Processor -- start 2023-10-30T14:46:04.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db9b3e640 1 -- start start 2023-10-30T14:46:04.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db9b3e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2db40ca9b0 0x7f2db406f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:04.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db9b3e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db40ca180 con 0x7f2db40c8060 2023-10-30T14:46:04.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db9b3e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db40ca460 con 0x7f2db40cb8b0 2023-10-30T14:46:04.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db9b3e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db4150d00 con 0x7f2db40ca9b0 2023-10-30T14:46:04.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db37fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f2db40c8060 0x7f2db40c8730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48890/0 (socket says 172.21.15.57:48890) 2023-10-30T14:46:04.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db37fe640 1 -- 172.21.15.57:0/1691860861 learned_addr learned my addr 172.21.15.57:0/1691860861 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:46:04.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db3fff640 1 --2- 172.21.15.57:0/1691860861 >> v2:172.21.15.88:3300/0 conn(0x7f2db40ca9b0 0x7f2db406f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:04.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db27fc640 1 -- 172.21.15.57:0/1691860861 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481776585 0 0) 0x7f2db40ca180 con 0x7f2db40c8060 2023-10-30T14:46:04.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db27fc640 1 -- 172.21.15.57:0/1691860861 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d9c003530 con 0x7f2db40c8060 2023-10-30T14:46:04.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db27fc640 1 -- 172.21.15.57:0/1691860861 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3394957672 0 0) 0x7f2d9c003530 con 0x7f2db40c8060 2023-10-30T14:46:04.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db27fc640 1 -- 172.21.15.57:0/1691860861 >> v2:172.21.15.88:3300/0 conn(0x7f2db40ca9b0 msgr2=0x7f2db406f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:04.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db27fc640 1 --2- 172.21.15.57:0/1691860861 >> v2:172.21.15.88:3300/0 conn(0x7f2db40ca9b0 0x7f2db406f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:04.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db27fc640 1 -- 172.21.15.57:0/1691860861 >> v1:172.21.15.88:6789/0 conn(0x7f2db40cb8b0 legacy=0x7f2db415ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:04.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db27fc640 1 -- 172.21.15.57:0/1691860861 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db406e7b0 con 0x7f2db40c8060 2023-10-30T14:46:04.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db27fc640 1 -- 172.21.15.57:0/1691860861 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f2da4001680 con 0x7f2db40c8060 2023-10-30T14:46:04.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db27fc640 1 -- 172.21.15.57:0/1691860861 >> v1:172.21.15.57:6789/0 conn(0x7f2db40c8060 legacy=0x7f2db40c8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:04.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db27fc640 1 --2- 172.21.15.57:0/1691860861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d9c003980 0x7f2d9c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:04.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db27fc640 1 --2- 172.21.15.57:0/1691860861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d9c0047d0 0x7f2d9c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:04.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db27fc640 1 --2- 172.21.15.57:0/1691860861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d9c005190 0x7f2d9c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:04.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db27fc640 1 -- 172.21.15.57:0/1691860861 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db4150d00 con 0x7f2d9c003980 2023-10-30T14:46:04.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db27fc640 1 -- 172.21.15.57:0/1691860861 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9c00c040 con 0x7f2d9c005190 2023-10-30T14:46:04.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db27fc640 1 -- 172.21.15.57:0/1691860861 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9c00c200 con 0x7f2d9c0047d0 2023-10-30T14:46:04.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db37fe640 1 --2- 172.21.15.57:0/1691860861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d9c003980 0x7f2d9c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:04.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db2ffd640 1 --2- 172.21.15.57:0/1691860861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d9c0047d0 0x7f2d9c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:04.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.637+0000 7f2db3fff640 1 --2- 172.21.15.57:0/1691860861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d9c005190 0x7f2d9c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:04.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.641+0000 7f2db37fe640 1 -- 172.21.15.57:0/1691860861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d9c0047d0 msgr2=0x7f2d9c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:04.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.641+0000 7f2db37fe640 1 --2- 172.21.15.57:0/1691860861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d9c0047d0 0x7f2d9c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:04.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.641+0000 7f2db37fe640 1 -- 172.21.15.57:0/1691860861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d9c005190 msgr2=0x7f2d9c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:04.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.641+0000 7f2db37fe640 1 --2- 172.21.15.57:0/1691860861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d9c005190 0x7f2d9c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:04.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.641+0000 7f2db37fe640 1 -- 172.21.15.57:0/1691860861 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d9c003530 con 0x7f2d9c003980 2023-10-30T14:46:04.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.641+0000 7f2db3fff640 1 -- 172.21.15.57:0/1691860861 reap_dead start 2023-10-30T14:46:04.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.641+0000 7f2db2ffd640 1 --2- 172.21.15.57:0/1691860861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d9c0047d0 0x7f2d9c004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:46:04.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.641+0000 7f2db37fe640 1 --2- 172.21.15.57:0/1691860861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d9c003980 0x7f2d9c003da0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f2da400c1c0 tx=0x7f2da401f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb91d4bc7832abe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:04.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.641+0000 7f2db27fc640 1 -- 172.21.15.57:0/1691860861 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da4005ae0 con 0x7f2d9c003980 2023-10-30T14:46:04.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.641+0000 7f2db27fc640 1 -- 172.21.15.57:0/1691860861 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da40ab070 con 0x7f2d9c003980 2023-10-30T14:46:04.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.641+0000 7f2db27fc640 1 -- 172.21.15.57:0/1691860861 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da4007070 con 0x7f2d9c003980 2023-10-30T14:46:04.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db9b3e640 1 -- 172.21.15.57:0/1691860861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d9c003980 msgr2=0x7f2d9c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:04.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db9b3e640 1 --2- 172.21.15.57:0/1691860861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d9c003980 0x7f2d9c003da0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f2da400c1c0 tx=0x7f2da401f310 comp rx=0 tx=0).stop 2023-10-30T14:46:04.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db9b3e640 1 -- 172.21.15.57:0/1691860861 shutdown_connections 2023-10-30T14:46:04.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db9b3e640 1 --2- 172.21.15.57:0/1691860861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d9c003980 0x7f2d9c003da0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:04.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db9b3e640 1 -- 172.21.15.57:0/1691860861 >> 172.21.15.57:0/1691860861 conn(0x7f2db406afa0 msgr2=0x7f2db406b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:04.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db9b3e640 1 -- 172.21.15.57:0/1691860861 shutdown_connections 2023-10-30T14:46:04.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db9b3e640 1 -- 172.21.15.57:0/1691860861 wait complete. 2023-10-30T14:46:04.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db9b3e640 1 Processor -- start 2023-10-30T14:46:04.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db9b3e640 1 -- start start 2023-10-30T14:46:04.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db9b3e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d9c003980 0x7f2db4152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:04.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db9b3e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2db4156ec0 0x7f2db406a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:04.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db9b3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2db4153820 0x7f2db4150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:04.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db9b3e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db41ee460 con 0x7f2db4156ec0 2023-10-30T14:46:04.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db9b3e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db41ef410 con 0x7f2db4153820 2023-10-30T14:46:04.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db9b3e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db41eef80 con 0x7f2d9c003980 2023-10-30T14:46:04.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2db4156ec0 0x7f2db406a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:04.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2db4156ec0 0x7f2db406a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46028/0 (socket says 172.21.15.57:46028) 2023-10-30T14:46:04.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db2ffd640 1 -- 172.21.15.57:0/3020643665 learned_addr learned my addr 172.21.15.57:0/3020643665 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:04.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db3fff640 1 --2- 172.21.15.57:0/3020643665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2db4153820 0x7f2db4150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:04.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db37fe640 1 --2- 172.21.15.57:0/3020643665 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d9c003980 0x7f2db4152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:04.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db2ffd640 1 -- 172.21.15.57:0/3020643665 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d9c003980 msgr2=0x7f2db4152360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:04.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db2ffd640 1 --2- 172.21.15.57:0/3020643665 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d9c003980 0x7f2db4152360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:04.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db2ffd640 1 -- 172.21.15.57:0/3020643665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2db4153820 msgr2=0x7f2db4150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:04.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db2ffd640 1 --2- 172.21.15.57:0/3020643665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2db4153820 0x7f2db4150e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:04.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db2ffd640 1 -- 172.21.15.57:0/3020643665 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db4152a10 con 0x7f2db4156ec0 2023-10-30T14:46:04.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.645+0000 7f2db3fff640 1 --2- 172.21.15.57:0/3020643665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2db4153820 0x7f2db4150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:46:04.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.649+0000 7f2db2ffd640 1 --2- 172.21.15.57:0/3020643665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2db4156ec0 0x7f2db406a0f0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f2da800e480 tx=0x7f2da800e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:04.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.649+0000 7f2d9bfff640 1 -- 172.21.15.57:0/3020643665 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da8017070 con 0x7f2db4156ec0 2023-10-30T14:46:04.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.649+0000 7f2d9bfff640 1 -- 172.21.15.57:0/3020643665 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da80052e0 con 0x7f2db4156ec0 2023-10-30T14:46:04.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.649+0000 7f2d9bfff640 1 -- 172.21.15.57:0/3020643665 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da8005490 con 0x7f2db4156ec0 2023-10-30T14:46:04.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.649+0000 7f2db9b3e640 1 -- 172.21.15.57:0/3020643665 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db41ee650 con 0x7f2db4156ec0 2023-10-30T14:46:04.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.649+0000 7f2db9b3e640 1 -- 172.21.15.57:0/3020643665 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db41eec10 con 0x7f2db4156ec0 2023-10-30T14:46:04.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.649+0000 7f2db9b3e640 1 -- 172.21.15.57:0/3020643665 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db41eeda0 con 0x7f2db4156ec0 2023-10-30T14:46:04.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.649+0000 7f2d9bfff640 1 -- 172.21.15.57:0/3020643665 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2da8007020 con 0x7f2db4156ec0 2023-10-30T14:46:04.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.649+0000 7f2d9bfff640 1 --2- 172.21.15.57:0/3020643665 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2da00664a0 0x7f2da0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:04.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.649+0000 7f2d9bfff640 1 -- 172.21.15.57:0/3020643665 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f2da8083c20 con 0x7f2db4156ec0 2023-10-30T14:46:04.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.649+0000 7f2db37fe640 1 --2- 172.21.15.57:0/3020643665 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2da00664a0 0x7f2da0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:04.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.653+0000 7f2db37fe640 1 --2- 172.21.15.57:0/3020643665 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2da00664a0 0x7f2da0068960 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2da40af000 tx=0x7f2da40aa000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:04.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.653+0000 7f2d9bfff640 1 -- 172.21.15.57:0/3020643665 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2da8050a20 con 0x7f2db4156ec0 2023-10-30T14:46:04.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.845+0000 7f2db9b3e640 1 -- 172.21.15.57:0/3020643665 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2db41ef0f0 con 0x7f2db4156ec0 2023-10-30T14:46:04.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.845+0000 7f2d9bfff640 1 -- 172.21.15.57:0/3020643665 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6513 (secure 0 0 0) 0x7f2da8054040 con 0x7f2db4156ec0 2023-10-30T14:46:04.850 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:04.850 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:45:57.259750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:stopping","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:04.851 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 174 2023-10-30T14:46:04.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.849+0000 7f2db9b3e640 1 -- 172.21.15.57:0/3020643665 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2da00664a0 msgr2=0x7f2da0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:04.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.849+0000 7f2db9b3e640 1 --2- 172.21.15.57:0/3020643665 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2da00664a0 0x7f2da0068960 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2da40af000 tx=0x7f2da40aa000 comp rx=0 tx=0).stop 2023-10-30T14:46:04.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.849+0000 7f2db9b3e640 1 -- 172.21.15.57:0/3020643665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2db4156ec0 msgr2=0x7f2db406a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:04.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.849+0000 7f2db9b3e640 1 --2- 172.21.15.57:0/3020643665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2db4156ec0 0x7f2db406a0f0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f2da800e480 tx=0x7f2da800e940 comp rx=0 tx=0).stop 2023-10-30T14:46:04.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.849+0000 7f2db9b3e640 1 -- 172.21.15.57:0/3020643665 shutdown_connections 2023-10-30T14:46:04.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.849+0000 7f2db9b3e640 1 --2- 172.21.15.57:0/3020643665 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2da00664a0 0x7f2da0068960 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:04.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.849+0000 7f2db9b3e640 1 --2- 172.21.15.57:0/3020643665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2db4153820 0x7f2db4150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:04.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.849+0000 7f2db9b3e640 1 --2- 172.21.15.57:0/3020643665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2db4156ec0 0x7f2db406a0f0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:04.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.849+0000 7f2db9b3e640 1 --2- 172.21.15.57:0/3020643665 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d9c003980 0x7f2db4152360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:04.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.849+0000 7f2db9b3e640 1 -- 172.21.15.57:0/3020643665 >> 172.21.15.57:0/3020643665 conn(0x7f2db406afa0 msgr2=0x7f2db40c65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:04.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.849+0000 7f2db9b3e640 1 -- 172.21.15.57:0/3020643665 shutdown_connections 2023-10-30T14:46:04.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:04.849+0000 7f2db9b3e640 1 -- 172.21.15.57:0/3020643665 wait complete. 2023-10-30T14:46:04.872 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:46:06.874 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:07.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.029+0000 7fa096d0f640 1 Processor -- start 2023-10-30T14:46:07.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.029+0000 7fa096d0f640 1 -- start start 2023-10-30T14:46:07.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.029+0000 7fa096d0f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa09015cf90 0x7fa0900c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:07.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.029+0000 7fa096d0f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa090159f20 con 0x7fa0900c4a10 2023-10-30T14:46:07.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.029+0000 7fa096d0f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0900c47e0 con 0x7fa090153c80 2023-10-30T14:46:07.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.029+0000 7fa096d0f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa09015dec0 con 0x7fa09015cf90 2023-10-30T14:46:07.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.029+0000 7fa0952ac640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa09015cf90 0x7fa0900c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:07.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.029+0000 7fa08ffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa0900c4a10 0x7fa0900ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48896/0 (socket says 172.21.15.57:48896) 2023-10-30T14:46:07.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.029+0000 7fa08ffff640 1 -- 172.21.15.57:0/102246615 learned_addr learned my addr 172.21.15.57:0/102246615 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:46:07.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.029+0000 7fa0952ac640 1 -- 172.21.15.57:0/102246615 >> v1:172.21.15.88:6789/0 conn(0x7fa090153c80 legacy=0x7fa09015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:07.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.029+0000 7fa0952ac640 1 -- 172.21.15.57:0/102246615 >> v1:172.21.15.57:6789/0 conn(0x7fa0900c4a10 legacy=0x7fa0900ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:07.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.029+0000 7fa0952ac640 1 -- 172.21.15.57:0/102246615 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09015d620 con 0x7fa09015cf90 2023-10-30T14:46:07.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.029+0000 7fa08f7fe640 1 -- 172.21.15.57:0/102246615 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131426755 0 0) 0x7fa090159f20 con 0x7fa0900c4a10 2023-10-30T14:46:07.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.033+0000 7fa0952ac640 1 --2- 172.21.15.57:0/102246615 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa09015cf90 0x7fa0900c42a0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fa088009d90 tx=0x7fa08801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de4173253cdaedcf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:07.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.033+0000 7fa08f7fe640 1 -- 172.21.15.57:0/102246615 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0880af600 con 0x7fa09015cf90 2023-10-30T14:46:07.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.033+0000 7fa08f7fe640 1 -- 172.21.15.57:0/102246615 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0880ab050 con 0x7fa09015cf90 2023-10-30T14:46:07.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.033+0000 7fa08f7fe640 1 -- 172.21.15.57:0/102246615 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0880af980 con 0x7fa09015cf90 2023-10-30T14:46:07.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.033+0000 7fa096d0f640 1 -- 172.21.15.57:0/102246615 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa09015cf90 msgr2=0x7fa0900c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:07.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.033+0000 7fa096d0f640 1 --2- 172.21.15.57:0/102246615 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa09015cf90 0x7fa0900c42a0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fa088009d90 tx=0x7fa08801f130 comp rx=0 tx=0).stop 2023-10-30T14:46:07.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.033+0000 7fa096d0f640 1 -- 172.21.15.57:0/102246615 shutdown_connections 2023-10-30T14:46:07.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.033+0000 7fa096d0f640 1 --2- 172.21.15.57:0/102246615 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa09015cf90 0x7fa0900c42a0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:07.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.033+0000 7fa096d0f640 1 -- 172.21.15.57:0/102246615 >> 172.21.15.57:0/102246615 conn(0x7fa0900715c0 msgr2=0x7fa0900719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:07.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.033+0000 7fa096d0f640 1 -- 172.21.15.57:0/102246615 shutdown_connections 2023-10-30T14:46:07.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.033+0000 7fa096d0f640 1 -- 172.21.15.57:0/102246615 wait complete. 2023-10-30T14:46:07.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa096d0f640 1 Processor -- start 2023-10-30T14:46:07.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa096d0f640 1 -- start start 2023-10-30T14:46:07.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa096d0f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0900c4a10 0x7fa0900b3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:07.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa096d0f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa090153c80 0x7fa090148330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:07.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa096d0f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa09015cf90 0x7fa0901eea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:07.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa096d0f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0901efa50 con 0x7fa0900c4a10 2023-10-30T14:46:07.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa096d0f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0900b2020 con 0x7fa090153c80 2023-10-30T14:46:07.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa096d0f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0900b4df0 con 0x7fa09015cf90 2023-10-30T14:46:07.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa094aab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0900c4a10 0x7fa0900b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:07.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa094aab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0900c4a10 0x7fa0900b3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46050/0 (socket says 172.21.15.57:46050) 2023-10-30T14:46:07.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa094aab640 1 -- 172.21.15.57:0/884102350 learned_addr learned my addr 172.21.15.57:0/884102350 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:07.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa08ffff640 1 --2- 172.21.15.57:0/884102350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa090153c80 0x7fa090148330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:07.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa0952ac640 1 --2- 172.21.15.57:0/884102350 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa09015cf90 0x7fa0901eea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:07.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa094aab640 1 -- 172.21.15.57:0/884102350 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa09015cf90 msgr2=0x7fa0901eea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:07.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa094aab640 1 --2- 172.21.15.57:0/884102350 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa09015cf90 0x7fa0901eea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:07.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa094aab640 1 -- 172.21.15.57:0/884102350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa090153c80 msgr2=0x7fa090148330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:07.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa094aab640 1 --2- 172.21.15.57:0/884102350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa090153c80 0x7fa090148330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:07.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa094aab640 1 -- 172.21.15.57:0/884102350 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa088009a70 con 0x7fa0900c4a10 2023-10-30T14:46:07.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa08ffff640 1 --2- 172.21.15.57:0/884102350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa090153c80 0x7fa090148330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:46:07.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa094aab640 1 --2- 172.21.15.57:0/884102350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0900c4a10 0x7fa0900b3410 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fa08400c500 tx=0x7fa08400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:07.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa08d7fa640 1 -- 172.21.15.57:0/884102350 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa084005220 con 0x7fa0900c4a10 2023-10-30T14:46:07.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa08d7fa640 1 -- 172.21.15.57:0/884102350 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa084005380 con 0x7fa0900c4a10 2023-10-30T14:46:07.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa08d7fa640 1 -- 172.21.15.57:0/884102350 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa084007270 con 0x7fa0900c4a10 2023-10-30T14:46:07.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa096d0f640 1 -- 172.21.15.57:0/884102350 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0900c5cc0 con 0x7fa0900c4a10 2023-10-30T14:46:07.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa096d0f640 1 -- 172.21.15.57:0/884102350 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0900b2270 con 0x7fa0900c4a10 2023-10-30T14:46:07.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.037+0000 7fa096d0f640 1 -- 172.21.15.57:0/884102350 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa09004e170 con 0x7fa0900c4a10 2023-10-30T14:46:07.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.041+0000 7fa08d7fa640 1 -- 172.21.15.57:0/884102350 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa0840073d0 con 0x7fa0900c4a10 2023-10-30T14:46:07.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.041+0000 7fa08d7fa640 1 --2- 172.21.15.57:0/884102350 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa0740664a0 0x7fa074068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:07.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.041+0000 7fa08d7fa640 1 -- 172.21.15.57:0/884102350 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7fa0840837d0 con 0x7fa0900c4a10 2023-10-30T14:46:07.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.041+0000 7fa08ffff640 1 --2- 172.21.15.57:0/884102350 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa0740664a0 0x7fa074068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:07.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.041+0000 7fa08ffff640 1 --2- 172.21.15.57:0/884102350 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa0740664a0 0x7fa074068960 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa09014dcc0 tx=0x7fa07c007c60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:07.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.045+0000 7fa08d7fa640 1 -- 172.21.15.57:0/884102350 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa084050690 con 0x7fa0900c4a10 2023-10-30T14:46:07.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.237+0000 7fa096d0f640 1 -- 172.21.15.57:0/884102350 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0900cad40 con 0x7fa0900c4a10 2023-10-30T14:46:07.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.237+0000 7fa08d7fa640 1 -- 172.21.15.57:0/884102350 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6513 (secure 0 0 0) 0x7fa084053cb0 con 0x7fa0900c4a10 2023-10-30T14:46:07.243 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:07.243 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:45:57.259750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:stopping","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:07.243 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 174 2023-10-30T14:46:07.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.241+0000 7fa096d0f640 1 -- 172.21.15.57:0/884102350 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa0740664a0 msgr2=0x7fa074068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:07.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.241+0000 7fa096d0f640 1 --2- 172.21.15.57:0/884102350 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa0740664a0 0x7fa074068960 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa09014dcc0 tx=0x7fa07c007c60 comp rx=0 tx=0).stop 2023-10-30T14:46:07.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.241+0000 7fa096d0f640 1 -- 172.21.15.57:0/884102350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0900c4a10 msgr2=0x7fa0900b3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:07.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.241+0000 7fa096d0f640 1 --2- 172.21.15.57:0/884102350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0900c4a10 0x7fa0900b3410 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fa08400c500 tx=0x7fa08400c9c0 comp rx=0 tx=0).stop 2023-10-30T14:46:07.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.241+0000 7fa096d0f640 1 -- 172.21.15.57:0/884102350 shutdown_connections 2023-10-30T14:46:07.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.241+0000 7fa096d0f640 1 --2- 172.21.15.57:0/884102350 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa0740664a0 0x7fa074068960 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:07.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.241+0000 7fa096d0f640 1 --2- 172.21.15.57:0/884102350 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa09015cf90 0x7fa0901eea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:07.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.241+0000 7fa096d0f640 1 --2- 172.21.15.57:0/884102350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa090153c80 0x7fa090148330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:07.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.241+0000 7fa096d0f640 1 --2- 172.21.15.57:0/884102350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0900c4a10 0x7fa0900b3410 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:07.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.241+0000 7fa096d0f640 1 -- 172.21.15.57:0/884102350 >> 172.21.15.57:0/884102350 conn(0x7fa0900715c0 msgr2=0x7fa090151640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:07.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.241+0000 7fa096d0f640 1 -- 172.21.15.57:0/884102350 shutdown_connections 2023-10-30T14:46:07.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:07.241+0000 7fa096d0f640 1 -- 172.21.15.57:0/884102350 wait complete. 2023-10-30T14:46:07.263 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:46:09.156 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 11% (25/219) Updating files: 12% (27/219) Updating files: 13% (29/219) Updating files: 14% (31/219) Updating files: 15% (33/219) Updating files: 16% (36/219) Updating files: 17% (38/219) Updating files: 18% (40/219) Updating files: 19% (42/219) Updating files: 20% (44/219) Updating files: 21% (46/219) Updating files: 22% (49/219) Updating files: 23% (51/219) Updating files: 24% (53/219) Updating files: 25% (55/219) Updating files: 26% (57/219) Updating files: 27% (60/219) Updating files: 28% (62/219) Updating files: 29% (64/219) Updating files: 30% (66/219) Updating files: 31% (68/219) Updating files: 32% (71/219) Updating files: 33% (73/219) Updating files: 34% (75/219) Updating files: 35% (77/219) Updating files: 36% (79/219) Updating files: 37% (82/219) Updating files: 38% (84/219) Updating files: 39% (86/219) Updating files: 40% (88/219) Updating files: 41% (90/219) Updating files: 42% (92/219) Updating files: 43% (95/219) Updating files: 44% (97/219) Updating files: 45% (99/219) Updating files: 46% (101/219) Updating files: 47% (103/219) Updating files: 48% (106/219) Updating files: 49% (108/219) Updating files: 50% (110/219) Updating files: 50% (111/219) Updating files: 51% (112/219) Updating files: 52% (114/219) Updating files: 53% (117/219) Updating files: 54% (119/219) Updating files: 55% (121/219) Updating files: 56% (123/219) Updating files: 57% (125/219) Updating files: 58% (128/219) Updating files: 59% (130/219) Updating files: 60% (132/219) Updating files: 61% (134/219) Updating files: 62% (136/219) Updating files: 62% (137/219) Updating files: 63% (138/219) Updating files: 64% (141/219) Updating files: 65% (143/219) Updating files: 66% (145/219) Updating files: 67% (147/219) Updating files: 68% (149/219) Updating files: 69% (152/219) Updating files: 70% (154/219) Updating files: 71% (156/219) Updating files: 72% (158/219) Updating files: 73% (160/219) Updating files: 74% (163/219) Updating files: 75% (165/219) Updating files: 76% (167/219) Updating files: 77% (169/219) Updating files: 78% (171/219) Updating files: 79% (174/219) Updating files: 80% (176/219) Updating files: 81% (178/219) Updating files: 82% (180/219) Updating files: 83% (182/219) Updating files: 84% (184/219) Updating files: 85% (187/219) Updating files: 86% (189/219) Updating files: 87% (191/219) Updating files: 88% (193/219) Updating files: 89% (195/219) Updating files: 90% (198/219) Updating files: 91% (200/219) Updating files: 92% (202/219) Updating files: 93% (204/219) Updating files: 94% (206/219) Updating files: 95% (209/219) Updating files: 96% (211/219) Updating files: 97% (213/219) Updating files: 98% (215/219) Updating files: 99% (217/219) Updating files: 100% (219/219) Updating files: 100% (219/219), done. 2023-10-30T14:46:09.199 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at ad663d58c89 v0.47 2023-10-30T14:46:09.203 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.47 2023-10-30T14:46:09.265 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:09.309 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:46:09.321 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:46:09.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.425+0000 7faf43617640 1 Processor -- start 2023-10-30T14:46:09.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.425+0000 7faf43617640 1 -- start start 2023-10-30T14:46:09.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.429+0000 7faf43617640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7faf3c06b760 0x7faf3c15e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:09.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.429+0000 7faf43617640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf3c06c420 0x7faf3c0c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:09.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.429+0000 7faf43617640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf3c071e00 con 0x7faf3c06bb40 2023-10-30T14:46:09.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.429+0000 7faf43617640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf3c070020 con 0x7faf3c06b760 2023-10-30T14:46:09.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.429+0000 7faf43617640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c15eb90 con 0x7faf3c06c420 2023-10-30T14:46:09.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.429+0000 7faf413b3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7faf3c06b760 0x7faf3c15e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:09.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.429+0000 7faf413b3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7faf3c06b760 0x7faf3c15e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46052/0 (socket says 172.21.15.57:46052) 2023-10-30T14:46:09.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.429+0000 7faf413b3640 1 -- 172.21.15.57:0/2410829163 learned_addr learned my addr 172.21.15.57:0/2410829163 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:09.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.429+0000 7faf41bb4640 1 --2- 172.21.15.57:0/2410829163 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf3c06c420 0x7faf3c0c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:09.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.429+0000 7faf413b3640 1 -- 172.21.15.57:0/2410829163 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf3c06c420 msgr2=0x7faf3c0c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:09.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.429+0000 7faf413b3640 1 --2- 172.21.15.57:0/2410829163 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf3c06c420 0x7faf3c0c57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:09.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.429+0000 7faf413b3640 1 -- 172.21.15.57:0/2410829163 >> v1:172.21.15.88:6789/0 conn(0x7faf3c06bb40 legacy=0x7faf3c157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:09.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.429+0000 7faf413b3640 1 -- 172.21.15.57:0/2410829163 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3c070f10 con 0x7faf3c06b760 2023-10-30T14:46:09.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.429+0000 7faf413b3640 1 --2- 172.21.15.57:0/2410829163 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf3c06b760 0x7faf3c15e650 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7faf38009870 tx=0x7faf3801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95912b9c854a15c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:09.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf33fff640 1 -- 172.21.15.57:0/2410829163 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf380aa670 con 0x7faf3c06b760 2023-10-30T14:46:09.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf33fff640 1 -- 172.21.15.57:0/2410829163 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf380aa7d0 con 0x7faf3c06b760 2023-10-30T14:46:09.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf33fff640 1 -- 172.21.15.57:0/2410829163 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf380aab60 con 0x7faf3c06b760 2023-10-30T14:46:09.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf43617640 1 -- 172.21.15.57:0/2410829163 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf3c06b760 msgr2=0x7faf3c15e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:09.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf43617640 1 --2- 172.21.15.57:0/2410829163 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf3c06b760 0x7faf3c15e650 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7faf38009870 tx=0x7faf3801c510 comp rx=0 tx=0).stop 2023-10-30T14:46:09.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf43617640 1 -- 172.21.15.57:0/2410829163 shutdown_connections 2023-10-30T14:46:09.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf43617640 1 --2- 172.21.15.57:0/2410829163 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf3c06c420 0x7faf3c0c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:09.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf43617640 1 --2- 172.21.15.57:0/2410829163 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf3c06b760 0x7faf3c15e650 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:09.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf43617640 1 -- 172.21.15.57:0/2410829163 >> 172.21.15.57:0/2410829163 conn(0x7faf3c0cac70 msgr2=0x7faf3c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:09.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf43617640 1 -- 172.21.15.57:0/2410829163 shutdown_connections 2023-10-30T14:46:09.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf43617640 1 -- 172.21.15.57:0/2410829163 wait complete. 2023-10-30T14:46:09.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf43617640 1 Processor -- start 2023-10-30T14:46:09.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf43617640 1 -- start start 2023-10-30T14:46:09.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf43617640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf3c06b760 0x7faf3c1530f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:09.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf43617640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf3c06bb40 0x7faf3c15cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:09.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf43617640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf3c06c420 0x7faf3c14a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:09.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf43617640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c1f5cf0 con 0x7faf3c06c420 2023-10-30T14:46:09.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf43617640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c0c37c0 con 0x7faf3c06b760 2023-10-30T14:46:09.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf43617640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c1f5ee0 con 0x7faf3c06bb40 2023-10-30T14:46:09.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf41bb4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf3c06c420 0x7faf3c14a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:09.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.433+0000 7faf413b3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf3c06b760 0x7faf3c1530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:09.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.437+0000 7faf41bb4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf3c06c420 0x7faf3c14a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46058/0 (socket says 172.21.15.57:46058) 2023-10-30T14:46:09.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.437+0000 7faf41bb4640 1 -- 172.21.15.57:0/2526643419 learned_addr learned my addr 172.21.15.57:0/2526643419 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:09.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.437+0000 7faf40bb2640 1 --2- 172.21.15.57:0/2526643419 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf3c06bb40 0x7faf3c15cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:09.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.437+0000 7faf41bb4640 1 -- 172.21.15.57:0/2526643419 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf3c06bb40 msgr2=0x7faf3c15cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:09.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.437+0000 7faf41bb4640 1 --2- 172.21.15.57:0/2526643419 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf3c06bb40 0x7faf3c15cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:09.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.437+0000 7faf41bb4640 1 -- 172.21.15.57:0/2526643419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf3c06b760 msgr2=0x7faf3c1530f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:09.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.437+0000 7faf41bb4640 1 --2- 172.21.15.57:0/2526643419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf3c06b760 0x7faf3c1530f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:09.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.437+0000 7faf41bb4640 1 -- 172.21.15.57:0/2526643419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf380094d0 con 0x7faf3c06c420 2023-10-30T14:46:09.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.437+0000 7faf41bb4640 1 --2- 172.21.15.57:0/2526643419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf3c06c420 0x7faf3c14a490 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7faf34007e60 tx=0x7faf3400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:09.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.437+0000 7faf31ffb640 1 -- 172.21.15.57:0/2526643419 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf34014420 con 0x7faf3c06c420 2023-10-30T14:46:09.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.437+0000 7faf31ffb640 1 -- 172.21.15.57:0/2526643419 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf340145a0 con 0x7faf3c06c420 2023-10-30T14:46:09.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.437+0000 7faf31ffb640 1 -- 172.21.15.57:0/2526643419 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf34014950 con 0x7faf3c06c420 2023-10-30T14:46:09.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.437+0000 7faf43617640 1 -- 172.21.15.57:0/2526643419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf3c0c39d0 con 0x7faf3c06c420 2023-10-30T14:46:09.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.437+0000 7faf43617640 1 -- 172.21.15.57:0/2526643419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf3c0c3f90 con 0x7faf3c06c420 2023-10-30T14:46:09.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.437+0000 7faf43617640 1 -- 172.21.15.57:0/2526643419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf3c04e170 con 0x7faf3c06c420 2023-10-30T14:46:09.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.437+0000 7faf31ffb640 1 -- 172.21.15.57:0/2526643419 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faf34014ab0 con 0x7faf3c06c420 2023-10-30T14:46:09.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.437+0000 7faf31ffb640 1 --2- 172.21.15.57:0/2526643419 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faf200664a0 0x7faf20068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:09.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.437+0000 7faf31ffb640 1 -- 172.21.15.57:0/2526643419 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7faf34084cd0 con 0x7faf3c06c420 2023-10-30T14:46:09.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.441+0000 7faf413b3640 1 --2- 172.21.15.57:0/2526643419 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faf200664a0 0x7faf20068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:09.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.441+0000 7faf413b3640 1 --2- 172.21.15.57:0/2526643419 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faf200664a0 0x7faf20068960 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7faf380ae000 tx=0x7faf380b5040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:09.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.441+0000 7faf31ffb640 1 -- 172.21.15.57:0/2526643419 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faf34051b80 con 0x7faf3c06c420 2023-10-30T14:46:09.504 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 48 -eq 48 ] 2023-10-30T14:46:09.505 INFO:tasks.workunit.client.0.smithi057.stderr:+ continue 2023-10-30T14:46:09.505 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 49 -eq 48 ] 2023-10-30T14:46:09.505 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.49 2023-10-30T14:46:09.505 INFO:tasks.workunit.client.0.smithi057.stdout:v0.49 2023-10-30T14:46:09.506 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.49 2023-10-30T14:46:09.506 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.49 2023-10-30T14:46:09.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.661+0000 7faf43617640 1 -- 172.21.15.57:0/2526643419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf3c156ce0 con 0x7faf3c06c420 2023-10-30T14:46:09.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.661+0000 7faf31ffb640 1 -- 172.21.15.57:0/2526643419 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6513 (secure 0 0 0) 0x7faf340551a0 con 0x7faf3c06c420 2023-10-30T14:46:09.664 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:09.664 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:45:57.259750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:stopping","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:09.665 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 174 2023-10-30T14:46:09.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.661+0000 7faf43617640 1 -- 172.21.15.57:0/2526643419 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faf200664a0 msgr2=0x7faf20068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:09.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.661+0000 7faf43617640 1 --2- 172.21.15.57:0/2526643419 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faf200664a0 0x7faf20068960 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7faf380ae000 tx=0x7faf380b5040 comp rx=0 tx=0).stop 2023-10-30T14:46:09.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.661+0000 7faf43617640 1 -- 172.21.15.57:0/2526643419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf3c06c420 msgr2=0x7faf3c14a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:09.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.661+0000 7faf43617640 1 --2- 172.21.15.57:0/2526643419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf3c06c420 0x7faf3c14a490 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7faf34007e60 tx=0x7faf3400e4b0 comp rx=0 tx=0).stop 2023-10-30T14:46:09.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.661+0000 7faf43617640 1 -- 172.21.15.57:0/2526643419 shutdown_connections 2023-10-30T14:46:09.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.661+0000 7faf43617640 1 --2- 172.21.15.57:0/2526643419 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faf200664a0 0x7faf20068960 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:09.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.661+0000 7faf43617640 1 --2- 172.21.15.57:0/2526643419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf3c06c420 0x7faf3c14a490 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:09.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.661+0000 7faf43617640 1 --2- 172.21.15.57:0/2526643419 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf3c06bb40 0x7faf3c15cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:09.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.661+0000 7faf43617640 1 --2- 172.21.15.57:0/2526643419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf3c06b760 0x7faf3c1530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:09.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.661+0000 7faf43617640 1 -- 172.21.15.57:0/2526643419 >> 172.21.15.57:0/2526643419 conn(0x7faf3c0cac70 msgr2=0x7faf3c156fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:09.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.661+0000 7faf43617640 1 -- 172.21.15.57:0/2526643419 shutdown_connections 2023-10-30T14:46:09.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:09.661+0000 7faf43617640 1 -- 172.21.15.57:0/2526643419 wait complete. 2023-10-30T14:46:09.685 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T14:46:09.686 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 174, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 174, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:stopping", "state_seq": 44 }, "gid_7536": { "addr": "172.21.15.57:6837/816502245", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 816502245, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 816502245, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 7536, "incarnation": 164, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:45:57.259750+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7536, "mds_1": 6747 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6839/1078372885", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1078372885, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1078372885, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 170, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6716, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.57:6834/697507447", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 697507447, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 697507447, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 156, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7377, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:46:11.687 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:11.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.829+0000 7f29e72ea640 1 Processor -- start 2023-10-30T14:46:11.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.829+0000 7f29e72ea640 1 -- start start 2023-10-30T14:46:11.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.833+0000 7f29e72ea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f29e0153c80 0x7f29e015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:11.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.833+0000 7f29e72ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29e00c42a0 0x7f29e00ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:11.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.833+0000 7f29e72ea640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e0159f20 con 0x7f29e00c4a10 2023-10-30T14:46:11.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.833+0000 7f29e72ea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e015dec0 con 0x7f29e0153c80 2023-10-30T14:46:11.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.833+0000 7f29e72ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e01599a0 con 0x7f29e00c42a0 2023-10-30T14:46:11.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.833+0000 7f29e4885640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f29e00c4a10 0x7f29e015a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48906/0 (socket says 172.21.15.57:48906) 2023-10-30T14:46:11.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.833+0000 7f29e4885640 1 -- 172.21.15.57:0/1451436566 learned_addr learned my addr 172.21.15.57:0/1451436566 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:46:11.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.833+0000 7f29e5086640 1 --2- 172.21.15.57:0/1451436566 >> v2:172.21.15.88:3300/0 conn(0x7f29e0153c80 0x7f29e015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:11.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.833+0000 7f29e5887640 1 --2- 172.21.15.57:0/1451436566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29e00c42a0 0x7f29e00ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:11.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.833+0000 7f29d7fff640 1 -- 172.21.15.57:0/1451436566 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492487205 0 0) 0x7f29e0159f20 con 0x7f29e00c4a10 2023-10-30T14:46:11.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.833+0000 7f29d7fff640 1 -- 172.21.15.57:0/1451436566 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29cc003530 con 0x7f29e00c4a10 2023-10-30T14:46:11.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.833+0000 7f29e5086640 1 -- 172.21.15.57:0/1451436566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29e00c42a0 msgr2=0x7f29e00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:11.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.833+0000 7f29e5086640 1 --2- 172.21.15.57:0/1451436566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29e00c42a0 0x7f29e00ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:11.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.833+0000 7f29e5086640 1 -- 172.21.15.57:0/1451436566 >> v1:172.21.15.57:6789/0 conn(0x7f29e00c4a10 legacy=0x7f29e015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:11.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.833+0000 7f29e5086640 1 -- 172.21.15.57:0/1451436566 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e015d620 con 0x7f29e0153c80 2023-10-30T14:46:11.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.837+0000 7f29e5086640 1 --2- 172.21.15.57:0/1451436566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29e0153c80 0x7f29e015cf90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f29dc0097c0 tx=0x7f29dc01ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=484b75c28313e27a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:11.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.837+0000 7f29d7fff640 1 -- 172.21.15.57:0/1451436566 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29dc0a8970 con 0x7f29e0153c80 2023-10-30T14:46:11.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.837+0000 7f29d7fff640 1 -- 172.21.15.57:0/1451436566 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29dc0a8ad0 con 0x7f29e0153c80 2023-10-30T14:46:11.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.837+0000 7f29d7fff640 1 -- 172.21.15.57:0/1451436566 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29dc0a8e60 con 0x7f29e0153c80 2023-10-30T14:46:11.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.837+0000 7f29e72ea640 1 -- 172.21.15.57:0/1451436566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29e0153c80 msgr2=0x7f29e015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:11.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.837+0000 7f29e72ea640 1 --2- 172.21.15.57:0/1451436566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29e0153c80 0x7f29e015cf90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f29dc0097c0 tx=0x7f29dc01ec70 comp rx=0 tx=0).stop 2023-10-30T14:46:11.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.837+0000 7f29e72ea640 1 -- 172.21.15.57:0/1451436566 shutdown_connections 2023-10-30T14:46:11.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.837+0000 7f29e72ea640 1 --2- 172.21.15.57:0/1451436566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29e00c42a0 0x7f29e00ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:11.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.837+0000 7f29e72ea640 1 --2- 172.21.15.57:0/1451436566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29e0153c80 0x7f29e015cf90 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:11.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.837+0000 7f29e72ea640 1 -- 172.21.15.57:0/1451436566 >> 172.21.15.57:0/1451436566 conn(0x7f29e00715c0 msgr2=0x7f29e00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:11.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.837+0000 7f29e72ea640 1 -- 172.21.15.57:0/1451436566 shutdown_connections 2023-10-30T14:46:11.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.837+0000 7f29e72ea640 1 -- 172.21.15.57:0/1451436566 wait complete. 2023-10-30T14:46:11.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.837+0000 7f29e72ea640 1 Processor -- start 2023-10-30T14:46:11.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.837+0000 7f29e72ea640 1 -- start start 2023-10-30T14:46:11.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e72ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29e00c42a0 0x7f29e00c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:11.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e72ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29e00c4a10 0x7f29e013fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:11.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e72ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29e0153c80 0x7f29e013ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:11.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e72ea640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e01ee8d0 con 0x7f29e00c4a10 2023-10-30T14:46:11.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e72ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e01eeac0 con 0x7f29e0153c80 2023-10-30T14:46:11.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e72ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e01eeda0 con 0x7f29e00c42a0 2023-10-30T14:46:11.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e4885640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29e00c4a10 0x7f29e013fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:11.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e4885640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29e00c4a10 0x7f29e013fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46068/0 (socket says 172.21.15.57:46068) 2023-10-30T14:46:11.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e4885640 1 -- 172.21.15.57:0/3311482908 learned_addr learned my addr 172.21.15.57:0/3311482908 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:11.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e5086640 1 --2- 172.21.15.57:0/3311482908 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29e00c42a0 0x7f29e00c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:11.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e5887640 1 --2- 172.21.15.57:0/3311482908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29e0153c80 0x7f29e013ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:11.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e4885640 1 -- 172.21.15.57:0/3311482908 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29e00c42a0 msgr2=0x7f29e00c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:11.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e4885640 1 --2- 172.21.15.57:0/3311482908 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29e00c42a0 0x7f29e00c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:11.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e4885640 1 -- 172.21.15.57:0/3311482908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29e0153c80 msgr2=0x7f29e013ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:11.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e4885640 1 --2- 172.21.15.57:0/3311482908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29e0153c80 0x7f29e013ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:11.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e4885640 1 -- 172.21.15.57:0/3311482908 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29dc009420 con 0x7f29e00c4a10 2023-10-30T14:46:11.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e4885640 1 --2- 172.21.15.57:0/3311482908 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29e00c4a10 0x7f29e013fa90 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f29d0001ec0 tx=0x7f29d000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:11.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29d5ffb640 1 -- 172.21.15.57:0/3311482908 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d0005d50 con 0x7f29e00c4a10 2023-10-30T14:46:11.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29d5ffb640 1 -- 172.21.15.57:0/3311482908 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d0007070 con 0x7f29e00c4a10 2023-10-30T14:46:11.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29d5ffb640 1 -- 172.21.15.57:0/3311482908 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d00073e0 con 0x7f29e00c4a10 2023-10-30T14:46:11.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e72ea640 1 -- 172.21.15.57:0/3311482908 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e01ef080 con 0x7f29e00c4a10 2023-10-30T14:46:11.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e72ea640 1 -- 172.21.15.57:0/3311482908 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e01ef640 con 0x7f29e00c4a10 2023-10-30T14:46:11.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.841+0000 7f29e72ea640 1 -- 172.21.15.57:0/3311482908 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e01ef830 con 0x7f29e00c4a10 2023-10-30T14:46:11.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.845+0000 7f29d5ffb640 1 -- 172.21.15.57:0/3311482908 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f29d0007540 con 0x7f29e00c4a10 2023-10-30T14:46:11.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.845+0000 7f29d5ffb640 1 --2- 172.21.15.57:0/3311482908 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29c40664a0 0x7f29c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:11.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.845+0000 7f29d5ffb640 1 -- 172.21.15.57:0/3311482908 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f29d0083980 con 0x7f29e00c4a10 2023-10-30T14:46:11.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.845+0000 7f29e5086640 1 --2- 172.21.15.57:0/3311482908 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29c40664a0 0x7f29c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:11.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.845+0000 7f29e5086640 1 --2- 172.21.15.57:0/3311482908 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29c40664a0 0x7f29c4068960 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f29dc0b2000 tx=0x7f29dc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:11.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:11.849+0000 7f29d5ffb640 1 -- 172.21.15.57:0/3311482908 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f29d00508f0 con 0x7f29e00c4a10 2023-10-30T14:46:12.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:12.049+0000 7f29e72ea640 1 -- 172.21.15.57:0/3311482908 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29e004e170 con 0x7f29e00c4a10 2023-10-30T14:46:12.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:12.049+0000 7f29d5ffb640 1 -- 172.21.15.57:0/3311482908 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6513 (secure 0 0 0) 0x7f29d0017090 con 0x7f29e00c4a10 2023-10-30T14:46:12.054 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:12.054 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:45:57.259750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:stopping","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:12.055 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 174 2023-10-30T14:46:12.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:12.053+0000 7f29e72ea640 1 -- 172.21.15.57:0/3311482908 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29c40664a0 msgr2=0x7f29c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:12.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:12.053+0000 7f29e72ea640 1 --2- 172.21.15.57:0/3311482908 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29c40664a0 0x7f29c4068960 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f29dc0b2000 tx=0x7f29dc008040 comp rx=0 tx=0).stop 2023-10-30T14:46:12.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:12.053+0000 7f29e72ea640 1 -- 172.21.15.57:0/3311482908 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29e00c4a10 msgr2=0x7f29e013fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:12.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:12.053+0000 7f29e72ea640 1 --2- 172.21.15.57:0/3311482908 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29e00c4a10 0x7f29e013fa90 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f29d0001ec0 tx=0x7f29d000c960 comp rx=0 tx=0).stop 2023-10-30T14:46:12.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:12.053+0000 7f29e72ea640 1 -- 172.21.15.57:0/3311482908 shutdown_connections 2023-10-30T14:46:12.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:12.053+0000 7f29e72ea640 1 --2- 172.21.15.57:0/3311482908 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f29c40664a0 0x7f29c4068960 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:12.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:12.053+0000 7f29e72ea640 1 --2- 172.21.15.57:0/3311482908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29e0153c80 0x7f29e013ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:12.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:12.053+0000 7f29e72ea640 1 --2- 172.21.15.57:0/3311482908 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29e00c4a10 0x7f29e013fa90 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:12.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:12.053+0000 7f29e72ea640 1 --2- 172.21.15.57:0/3311482908 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29e00c42a0 0x7f29e00c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:12.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:12.053+0000 7f29e72ea640 1 -- 172.21.15.57:0/3311482908 >> 172.21.15.57:0/3311482908 conn(0x7f29e00715c0 msgr2=0x7f29e00719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:12.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:12.053+0000 7f29e72ea640 1 -- 172.21.15.57:0/3311482908 shutdown_connections 2023-10-30T14:46:12.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:12.053+0000 7f29e72ea640 1 -- 172.21.15.57:0/3311482908 wait complete. 2023-10-30T14:46:12.075 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:46:14.076 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:14.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.221+0000 7f8e67ed6640 1 Processor -- start 2023-10-30T14:46:14.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.225+0000 7f8e67ed6640 1 -- start start 2023-10-30T14:46:14.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.225+0000 7f8e67ed6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e60153c80 0x7f8e6015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:14.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.225+0000 7f8e67ed6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e60159f20 con 0x7f8e600c42a0 2023-10-30T14:46:14.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.225+0000 7f8e67ed6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e6015d620 con 0x7f8e600c4a10 2023-10-30T14:46:14.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.225+0000 7f8e67ed6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6015dec0 con 0x7f8e60153c80 2023-10-30T14:46:14.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.225+0000 7f8e66473640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f8e600c42a0 0x7f8e600ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44294/0 (socket says 172.21.15.57:44294) 2023-10-30T14:46:14.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.225+0000 7f8e66473640 1 -- 172.21.15.57:0/3842587885 learned_addr learned my addr 172.21.15.57:0/3842587885 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:46:14.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.225+0000 7f8e65c72640 1 --2- 172.21.15.57:0/3842587885 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e60153c80 0x7f8e6015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:14.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.225+0000 7f8e64c70640 1 -- 172.21.15.57:0/3842587885 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603284898 0 0) 0x7f8e60159f20 con 0x7f8e600c42a0 2023-10-30T14:46:14.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.225+0000 7f8e64c70640 1 -- 172.21.15.57:0/3842587885 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e4c003530 con 0x7f8e600c42a0 2023-10-30T14:46:14.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.225+0000 7f8e64c70640 1 -- 172.21.15.57:0/3842587885 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427748196 0 0) 0x7f8e6015d620 con 0x7f8e600c4a10 2023-10-30T14:46:14.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.225+0000 7f8e64c70640 1 -- 172.21.15.57:0/3842587885 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e60159f20 con 0x7f8e600c4a10 2023-10-30T14:46:14.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.225+0000 7f8e64c70640 1 -- 172.21.15.57:0/3842587885 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 367675881 0 0) 0x7f8e4c003530 con 0x7f8e600c42a0 2023-10-30T14:46:14.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.225+0000 7f8e65c72640 1 -- 172.21.15.57:0/3842587885 >> v1:172.21.15.88:6789/0 conn(0x7f8e600c4a10 legacy=0x7f8e6015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:14.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.225+0000 7f8e65c72640 1 -- 172.21.15.57:0/3842587885 >> v1:172.21.15.57:6789/0 conn(0x7f8e600c42a0 legacy=0x7f8e600ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:14.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.225+0000 7f8e65c72640 1 -- 172.21.15.57:0/3842587885 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6015d900 con 0x7f8e60153c80 2023-10-30T14:46:14.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.229+0000 7f8e65c72640 1 --2- 172.21.15.57:0/3842587885 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e60153c80 0x7f8e6015cf90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0097f0 tx=0x7f8e5c01ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a8eb4c313c26309 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:14.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.229+0000 7f8e64c70640 1 -- 172.21.15.57:0/3842587885 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e58001a00 con 0x7f8e60153c80 2023-10-30T14:46:14.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.229+0000 7f8e64c70640 1 -- 172.21.15.57:0/3842587885 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5c0a8650 con 0x7f8e60153c80 2023-10-30T14:46:14.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.229+0000 7f8e64c70640 1 -- 172.21.15.57:0/3842587885 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c0a89c0 con 0x7f8e60153c80 2023-10-30T14:46:14.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.229+0000 7f8e67ed6640 1 -- 172.21.15.57:0/3842587885 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e60153c80 msgr2=0x7f8e6015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:14.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.229+0000 7f8e67ed6640 1 --2- 172.21.15.57:0/3842587885 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e60153c80 0x7f8e6015cf90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0097f0 tx=0x7f8e5c01ebd0 comp rx=0 tx=0).stop 2023-10-30T14:46:14.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.229+0000 7f8e67ed6640 1 -- 172.21.15.57:0/3842587885 shutdown_connections 2023-10-30T14:46:14.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.229+0000 7f8e67ed6640 1 --2- 172.21.15.57:0/3842587885 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e60153c80 0x7f8e6015cf90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:14.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.229+0000 7f8e67ed6640 1 -- 172.21.15.57:0/3842587885 >> 172.21.15.57:0/3842587885 conn(0x7f8e600715c0 msgr2=0x7f8e600719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:14.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.229+0000 7f8e67ed6640 1 -- 172.21.15.57:0/3842587885 shutdown_connections 2023-10-30T14:46:14.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.229+0000 7f8e67ed6640 1 -- 172.21.15.57:0/3842587885 wait complete. 2023-10-30T14:46:14.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e67ed6640 1 Processor -- start 2023-10-30T14:46:14.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e67ed6640 1 -- start start 2023-10-30T14:46:14.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e67ed6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e600c42a0 0x7f8e60149ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:14.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e67ed6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e600c4a10 0x7f8e6015ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:14.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e67ed6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e60153c80 0x7f8e601e6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:14.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e67ed6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6014e410 con 0x7f8e600c4a10 2023-10-30T14:46:14.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e67ed6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e600c6ec0 con 0x7f8e600c42a0 2023-10-30T14:46:14.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e67ed6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6014f530 con 0x7f8e60153c80 2023-10-30T14:46:14.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e65471640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e600c4a10 0x7f8e6015ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:14.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e65471640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e600c4a10 0x7f8e6015ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43796/0 (socket says 172.21.15.57:43796) 2023-10-30T14:46:14.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e65471640 1 -- 172.21.15.57:0/1280969156 learned_addr learned my addr 172.21.15.57:0/1280969156 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:14.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e66473640 1 --2- 172.21.15.57:0/1280969156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e60153c80 0x7f8e601e6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:14.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e65c72640 1 --2- 172.21.15.57:0/1280969156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e600c42a0 0x7f8e60149ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:14.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e65471640 1 -- 172.21.15.57:0/1280969156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e60153c80 msgr2=0x7f8e601e6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:14.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e65471640 1 --2- 172.21.15.57:0/1280969156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e60153c80 0x7f8e601e6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:14.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e65471640 1 -- 172.21.15.57:0/1280969156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e600c42a0 msgr2=0x7f8e60149ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:14.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e65471640 1 --2- 172.21.15.57:0/1280969156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e600c42a0 0x7f8e60149ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:14.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e65471640 1 -- 172.21.15.57:0/1280969156 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c009420 con 0x7f8e600c4a10 2023-10-30T14:46:14.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e65c72640 1 --2- 172.21.15.57:0/1280969156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e600c42a0 0x7f8e60149ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:46:14.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e65471640 1 --2- 172.21.15.57:0/1280969156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e600c4a10 0x7f8e6015ab60 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000c530 tx=0x7f8e5000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:14.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e567fc640 1 -- 172.21.15.57:0/1280969156 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e500052d0 con 0x7f8e600c4a10 2023-10-30T14:46:14.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e567fc640 1 -- 172.21.15.57:0/1280969156 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e50005430 con 0x7f8e600c4a10 2023-10-30T14:46:14.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e67ed6640 1 -- 172.21.15.57:0/1280969156 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6014f9c0 con 0x7f8e600c4a10 2023-10-30T14:46:14.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e567fc640 1 -- 172.21.15.57:0/1280969156 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50012870 con 0x7f8e600c4a10 2023-10-30T14:46:14.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e67ed6640 1 -- 172.21.15.57:0/1280969156 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e6014e920 con 0x7f8e600c4a10 2023-10-30T14:46:14.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.233+0000 7f8e67ed6640 1 -- 172.21.15.57:0/1280969156 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e6004e170 con 0x7f8e600c4a10 2023-10-30T14:46:14.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.237+0000 7f8e567fc640 1 -- 172.21.15.57:0/1280969156 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8e500129d0 con 0x7f8e600c4a10 2023-10-30T14:46:14.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.237+0000 7f8e567fc640 1 --2- 172.21.15.57:0/1280969156 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8e440664a0 0x7f8e44068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:14.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.237+0000 7f8e567fc640 1 -- 172.21.15.57:0/1280969156 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f8e50084150 con 0x7f8e600c4a10 2023-10-30T14:46:14.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.237+0000 7f8e65c72640 1 --2- 172.21.15.57:0/1280969156 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8e440664a0 0x7f8e44068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:14.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.237+0000 7f8e65c72640 1 --2- 172.21.15.57:0/1280969156 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8e440664a0 0x7f8e44068960 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0b2000 tx=0x7f8e5c0aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:14.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.241+0000 7f8e567fc640 1 -- 172.21.15.57:0/1280969156 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8e50050ed0 con 0x7f8e600c4a10 2023-10-30T14:46:14.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.433+0000 7f8e67ed6640 1 -- 172.21.15.57:0/1280969156 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e6014f670 con 0x7f8e600c4a10 2023-10-30T14:46:14.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.433+0000 7f8e567fc640 1 -- 172.21.15.57:0/1280969156 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6513 (secure 0 0 0) 0x7f8e500544f0 con 0x7f8e600c4a10 2023-10-30T14:46:14.437 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:14.437 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:45:57.259750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:stopping","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:14.439 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 174 2023-10-30T14:46:14.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.433+0000 7f8e67ed6640 1 -- 172.21.15.57:0/1280969156 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8e440664a0 msgr2=0x7f8e44068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:14.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.433+0000 7f8e67ed6640 1 --2- 172.21.15.57:0/1280969156 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8e440664a0 0x7f8e44068960 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0b2000 tx=0x7f8e5c0aa070 comp rx=0 tx=0).stop 2023-10-30T14:46:14.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.433+0000 7f8e67ed6640 1 -- 172.21.15.57:0/1280969156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e600c4a10 msgr2=0x7f8e6015ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:14.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.433+0000 7f8e67ed6640 1 --2- 172.21.15.57:0/1280969156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e600c4a10 0x7f8e6015ab60 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000c530 tx=0x7f8e5000c9f0 comp rx=0 tx=0).stop 2023-10-30T14:46:14.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.433+0000 7f8e67ed6640 1 -- 172.21.15.57:0/1280969156 shutdown_connections 2023-10-30T14:46:14.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.433+0000 7f8e67ed6640 1 --2- 172.21.15.57:0/1280969156 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8e440664a0 0x7f8e44068960 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:14.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.433+0000 7f8e67ed6640 1 --2- 172.21.15.57:0/1280969156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8e60153c80 0x7f8e601e6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:14.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.433+0000 7f8e67ed6640 1 --2- 172.21.15.57:0/1280969156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e600c4a10 0x7f8e6015ab60 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:14.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.433+0000 7f8e67ed6640 1 --2- 172.21.15.57:0/1280969156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e600c42a0 0x7f8e60149ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:14.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.433+0000 7f8e67ed6640 1 -- 172.21.15.57:0/1280969156 >> 172.21.15.57:0/1280969156 conn(0x7f8e600715c0 msgr2=0x7f8e60151670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:14.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.437+0000 7f8e67ed6640 1 -- 172.21.15.57:0/1280969156 shutdown_connections 2023-10-30T14:46:14.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:14.437+0000 7f8e67ed6640 1 -- 172.21.15.57:0/1280969156 wait complete. 2023-10-30T14:46:14.457 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:46:16.458 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:16.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.617+0000 7f114fa91640 1 Processor -- start 2023-10-30T14:46:16.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.617+0000 7f114fa91640 1 -- start start 2023-10-30T14:46:16.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.617+0000 7f114fa91640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1148153c80 0x7f114815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:16.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.617+0000 7f114fa91640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f11480c42a0 0x7f11480ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:16.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.617+0000 7f114fa91640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1148159f20 con 0x7f11480c4a10 2023-10-30T14:46:16.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.617+0000 7f114fa91640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f114815dec0 con 0x7f1148153c80 2023-10-30T14:46:16.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.617+0000 7f114fa91640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11481599a0 con 0x7f11480c42a0 2023-10-30T14:46:16.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.617+0000 7f114d82d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1148153c80 0x7f114815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:16.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.617+0000 7f114d02c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f11480c4a10 0x7f114815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44302/0 (socket says 172.21.15.57:44302) 2023-10-30T14:46:16.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.617+0000 7f114d02c640 1 -- 172.21.15.57:0/675503902 learned_addr learned my addr 172.21.15.57:0/675503902 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:46:16.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.617+0000 7f114e02e640 1 --2- 172.21.15.57:0/675503902 >> v2:172.21.15.88:3300/0 conn(0x7f11480c42a0 0x7f11480ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:16.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.617+0000 7f114d82d640 1 -- 172.21.15.57:0/675503902 >> v2:172.21.15.88:3300/0 conn(0x7f11480c42a0 msgr2=0x7f11480ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:16.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.617+0000 7f114d82d640 1 --2- 172.21.15.57:0/675503902 >> v2:172.21.15.88:3300/0 conn(0x7f11480c42a0 0x7f11480ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:16.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.617+0000 7f114d82d640 1 -- 172.21.15.57:0/675503902 >> v1:172.21.15.57:6789/0 conn(0x7f11480c4a10 legacy=0x7f114815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:16.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.617+0000 7f114d82d640 1 -- 172.21.15.57:0/675503902 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114815d620 con 0x7f1148153c80 2023-10-30T14:46:16.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.617+0000 7f114c82b640 1 -- 172.21.15.57:0/675503902 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750335551 0 0) 0x7f1148159f20 con 0x7f11480c4a10 2023-10-30T14:46:16.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.621+0000 7f114d82d640 1 --2- 172.21.15.57:0/675503902 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1148153c80 0x7f114815cf90 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f11480b1d40 tx=0x7f114401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dbef0e24e76efbb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:16.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.621+0000 7f114c82b640 1 -- 172.21.15.57:0/675503902 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11440a8950 con 0x7f1148153c80 2023-10-30T14:46:16.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.621+0000 7f114c82b640 1 -- 172.21.15.57:0/675503902 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11440a8ab0 con 0x7f1148153c80 2023-10-30T14:46:16.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.621+0000 7f114c82b640 1 -- 172.21.15.57:0/675503902 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11440a8e40 con 0x7f1148153c80 2023-10-30T14:46:16.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.621+0000 7f114fa91640 1 -- 172.21.15.57:0/675503902 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1148153c80 msgr2=0x7f114815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:16.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.621+0000 7f114fa91640 1 --2- 172.21.15.57:0/675503902 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1148153c80 0x7f114815cf90 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f11480b1d40 tx=0x7f114401ec50 comp rx=0 tx=0).stop 2023-10-30T14:46:16.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.621+0000 7f114fa91640 1 -- 172.21.15.57:0/675503902 shutdown_connections 2023-10-30T14:46:16.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.621+0000 7f114fa91640 1 --2- 172.21.15.57:0/675503902 >> v2:172.21.15.88:3300/0 conn(0x7f11480c42a0 0x7f11480ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:16.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.621+0000 7f114fa91640 1 --2- 172.21.15.57:0/675503902 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1148153c80 0x7f114815cf90 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:16.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.621+0000 7f114fa91640 1 -- 172.21.15.57:0/675503902 >> 172.21.15.57:0/675503902 conn(0x7f11480715c0 msgr2=0x7f11480719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:16.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.621+0000 7f114fa91640 1 -- 172.21.15.57:0/675503902 shutdown_connections 2023-10-30T14:46:16.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.621+0000 7f114fa91640 1 -- 172.21.15.57:0/675503902 wait complete. 2023-10-30T14:46:16.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114fa91640 1 Processor -- start 2023-10-30T14:46:16.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114fa91640 1 -- start start 2023-10-30T14:46:16.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114fa91640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11480c42a0 0x7f114814df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:16.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114fa91640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11480c4a10 0x7f11481ef990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:16.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114fa91640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1148153c80 0x7f114814a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:16.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114fa91640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114814be40 con 0x7f11480c42a0 2023-10-30T14:46:16.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114fa91640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114815c7c0 con 0x7f11480c4a10 2023-10-30T14:46:16.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114fa91640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f114815caa0 con 0x7f1148153c80 2023-10-30T14:46:16.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114d82d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11480c42a0 0x7f114814df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:16.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114d82d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11480c42a0 0x7f114814df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43826/0 (socket says 172.21.15.57:43826) 2023-10-30T14:46:16.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114d82d640 1 -- 172.21.15.57:0/436054300 learned_addr learned my addr 172.21.15.57:0/436054300 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:16.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114e02e640 1 --2- 172.21.15.57:0/436054300 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1148153c80 0x7f114814a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:16.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114d02c640 1 --2- 172.21.15.57:0/436054300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11480c4a10 0x7f11481ef990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:16.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114d82d640 1 -- 172.21.15.57:0/436054300 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1148153c80 msgr2=0x7f114814a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:16.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114d82d640 1 --2- 172.21.15.57:0/436054300 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1148153c80 0x7f114814a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:16.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114d82d640 1 -- 172.21.15.57:0/436054300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11480c4a10 msgr2=0x7f11481ef990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:16.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114d82d640 1 --2- 172.21.15.57:0/436054300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11480c4a10 0x7f11481ef990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:16.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114d82d640 1 -- 172.21.15.57:0/436054300 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1144009420 con 0x7f11480c42a0 2023-10-30T14:46:16.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114e02e640 1 --2- 172.21.15.57:0/436054300 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1148153c80 0x7f114814a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:46:16.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114d02c640 1 --2- 172.21.15.57:0/436054300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11480c4a10 0x7f11481ef990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:46:16.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114d82d640 1 --2- 172.21.15.57:0/436054300 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11480c42a0 0x7f114814df30 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f11440b3040 tx=0x7f114401b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:16.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f113e7fc640 1 -- 172.21.15.57:0/436054300 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11440b0ea0 con 0x7f11480c42a0 2023-10-30T14:46:16.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f113e7fc640 1 -- 172.21.15.57:0/436054300 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11440b9410 con 0x7f11480c42a0 2023-10-30T14:46:16.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f113e7fc640 1 -- 172.21.15.57:0/436054300 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11440b97c0 con 0x7f11480c42a0 2023-10-30T14:46:16.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114fa91640 1 -- 172.21.15.57:0/436054300 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f114815cd80 con 0x7f11480c42a0 2023-10-30T14:46:16.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.625+0000 7f114fa91640 1 -- 172.21.15.57:0/436054300 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11481ef470 con 0x7f11480c42a0 2023-10-30T14:46:16.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.629+0000 7f114fa91640 1 -- 172.21.15.57:0/436054300 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11481ef660 con 0x7f11480c42a0 2023-10-30T14:46:16.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.629+0000 7f113e7fc640 1 -- 172.21.15.57:0/436054300 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f11440b7040 con 0x7f11480c42a0 2023-10-30T14:46:16.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.629+0000 7f113e7fc640 1 --2- 172.21.15.57:0/436054300 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f112c0664a0 0x7f112c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:16.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.629+0000 7f113e7fc640 1 -- 172.21.15.57:0/436054300 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f1144122110 con 0x7f11480c42a0 2023-10-30T14:46:16.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.629+0000 7f114d02c640 1 --2- 172.21.15.57:0/436054300 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f112c0664a0 0x7f112c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:16.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.629+0000 7f114d02c640 1 --2- 172.21.15.57:0/436054300 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f112c0664a0 0x7f112c068960 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f114814f370 tx=0x7f1138009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:16.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.633+0000 7f113e7fc640 1 -- 172.21.15.57:0/436054300 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f11440eef10 con 0x7f11480c42a0 2023-10-30T14:46:16.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.825+0000 7f114fa91640 1 -- 172.21.15.57:0/436054300 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f114804e170 con 0x7f11480c42a0 2023-10-30T14:46:16.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.825+0000 7f113e7fc640 1 -- 172.21.15.57:0/436054300 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6513 (secure 0 0 0) 0x7f11440b5020 con 0x7f11480c42a0 2023-10-30T14:46:16.829 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:16.830 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:45:57.259750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:stopping","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:16.830 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 174 2023-10-30T14:46:16.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.829+0000 7f114fa91640 1 -- 172.21.15.57:0/436054300 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f112c0664a0 msgr2=0x7f112c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:16.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.829+0000 7f114fa91640 1 --2- 172.21.15.57:0/436054300 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f112c0664a0 0x7f112c068960 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f114814f370 tx=0x7f1138009040 comp rx=0 tx=0).stop 2023-10-30T14:46:16.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.829+0000 7f114fa91640 1 -- 172.21.15.57:0/436054300 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11480c42a0 msgr2=0x7f114814df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:16.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.829+0000 7f114fa91640 1 --2- 172.21.15.57:0/436054300 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11480c42a0 0x7f114814df30 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f11440b3040 tx=0x7f114401b9f0 comp rx=0 tx=0).stop 2023-10-30T14:46:16.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.829+0000 7f114fa91640 1 -- 172.21.15.57:0/436054300 shutdown_connections 2023-10-30T14:46:16.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.829+0000 7f114fa91640 1 --2- 172.21.15.57:0/436054300 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f112c0664a0 0x7f112c068960 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:16.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.829+0000 7f114fa91640 1 --2- 172.21.15.57:0/436054300 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1148153c80 0x7f114814a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:16.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.829+0000 7f114fa91640 1 --2- 172.21.15.57:0/436054300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11480c4a10 0x7f11481ef990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:16.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.829+0000 7f114fa91640 1 --2- 172.21.15.57:0/436054300 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11480c42a0 0x7f114814df30 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:16.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.829+0000 7f114fa91640 1 -- 172.21.15.57:0/436054300 >> 172.21.15.57:0/436054300 conn(0x7f11480715c0 msgr2=0x7f1148157810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:16.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.829+0000 7f114fa91640 1 -- 172.21.15.57:0/436054300 shutdown_connections 2023-10-30T14:46:16.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:16.829+0000 7f114fa91640 1 -- 172.21.15.57:0/436054300 wait complete. 2023-10-30T14:46:16.850 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:46:18.852 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:19.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc48226b640 1 Processor -- start 2023-10-30T14:46:19.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc48226b640 1 -- start start 2023-10-30T14:46:19.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc48226b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc47c15cf90 0x7fc47c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:19.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc48226b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc47c159f20 con 0x7fc47c153c80 2023-10-30T14:46:19.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc48226b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc47c0c47e0 con 0x7fc47c0c4a10 2023-10-30T14:46:19.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc48226b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c15dec0 con 0x7fc47c15cf90 2023-10-30T14:46:19.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47bfff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc47c153c80 0x7fc47c15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44310/0 (socket says 172.21.15.57:44310) 2023-10-30T14:46:19.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47bfff640 1 -- 172.21.15.57:0/3986370566 learned_addr learned my addr 172.21.15.57:0/3986370566 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:46:19.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc480808640 1 --2- 172.21.15.57:0/3986370566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc47c15cf90 0x7fc47c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:19.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47affd640 1 -- 172.21.15.57:0/3986370566 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644573996 0 0) 0x7fc47c159f20 con 0x7fc47c153c80 2023-10-30T14:46:19.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47affd640 1 -- 172.21.15.57:0/3986370566 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc464003530 con 0x7fc47c153c80 2023-10-30T14:46:19.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47affd640 1 -- 172.21.15.57:0/3986370566 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3525132256 0 0) 0x7fc464003530 con 0x7fc47c153c80 2023-10-30T14:46:19.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47affd640 1 -- 172.21.15.57:0/3986370566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc47c15cf90 msgr2=0x7fc47c0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:19.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47affd640 1 --2- 172.21.15.57:0/3986370566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc47c15cf90 0x7fc47c0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:19.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47affd640 1 -- 172.21.15.57:0/3986370566 >> v1:172.21.15.88:6789/0 conn(0x7fc47c0c4a10 legacy=0x7fc47c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:19.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47affd640 1 -- 172.21.15.57:0/3986370566 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47c15d620 con 0x7fc47c153c80 2023-10-30T14:46:19.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47affd640 1 -- 172.21.15.57:0/3986370566 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fc46c001680 con 0x7fc47c153c80 2023-10-30T14:46:19.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47affd640 1 -- 172.21.15.57:0/3986370566 >> v1:172.21.15.57:6789/0 conn(0x7fc47c153c80 legacy=0x7fc47c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:19.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47affd640 1 --2- 172.21.15.57:0/3986370566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4640039b0 0x7fc464003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:19.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47affd640 1 --2- 172.21.15.57:0/3986370566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc464004360 0x7fc464004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:19.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47affd640 1 --2- 172.21.15.57:0/3986370566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc47c15cf90 0x7fc464005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:19.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47affd640 1 -- 172.21.15.57:0/3986370566 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c15dec0 con 0x7fc464004360 2023-10-30T14:46:19.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47affd640 1 -- 172.21.15.57:0/3986370566 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46400b9c0 con 0x7fc4640039b0 2023-10-30T14:46:19.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47affd640 1 -- 172.21.15.57:0/3986370566 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc46400bb80 con 0x7fc47c15cf90 2023-10-30T14:46:19.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47b7fe640 1 --2- 172.21.15.57:0/3986370566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc464004360 0x7fc464004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:19.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47bfff640 1 --2- 172.21.15.57:0/3986370566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4640039b0 0x7fc464003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:19.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc480808640 1 --2- 172.21.15.57:0/3986370566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc47c15cf90 0x7fc464005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:19.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47b7fe640 1 -- 172.21.15.57:0/3986370566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc47c15cf90 msgr2=0x7fc464005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:19.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47b7fe640 1 --2- 172.21.15.57:0/3986370566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc47c15cf90 0x7fc464005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:19.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47b7fe640 1 -- 172.21.15.57:0/3986370566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4640039b0 msgr2=0x7fc464003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:19.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47b7fe640 1 --2- 172.21.15.57:0/3986370566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4640039b0 0x7fc464003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:19.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.005+0000 7fc47b7fe640 1 -- 172.21.15.57:0/3986370566 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc464003530 con 0x7fc464004360 2023-10-30T14:46:19.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.009+0000 7fc47b7fe640 1 --2- 172.21.15.57:0/3986370566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc464004360 0x7fc464004830 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fc47c0c4e30 tx=0x7fc47001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a5b461ad3447eae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:19.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.009+0000 7fc47affd640 1 -- 172.21.15.57:0/3986370566 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c007ea0 con 0x7fc464004360 2023-10-30T14:46:19.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.009+0000 7fc47affd640 1 -- 172.21.15.57:0/3986370566 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4700ac640 con 0x7fc464004360 2023-10-30T14:46:19.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.009+0000 7fc47affd640 1 -- 172.21.15.57:0/3986370566 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4700ac9f0 con 0x7fc464004360 2023-10-30T14:46:19.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.009+0000 7fc48226b640 1 -- 172.21.15.57:0/3986370566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc464004360 msgr2=0x7fc464004830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:19.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.009+0000 7fc48226b640 1 --2- 172.21.15.57:0/3986370566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc464004360 0x7fc464004830 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fc47c0c4e30 tx=0x7fc47001c4a0 comp rx=0 tx=0).stop 2023-10-30T14:46:19.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.009+0000 7fc480808640 1 -- 172.21.15.57:0/3986370566 reap_dead start 2023-10-30T14:46:19.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.009+0000 7fc48226b640 1 -- 172.21.15.57:0/3986370566 shutdown_connections 2023-10-30T14:46:19.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.009+0000 7fc48226b640 1 -- 172.21.15.57:0/3986370566 >> 172.21.15.57:0/3986370566 conn(0x7fc47c0715c0 msgr2=0x7fc47c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:19.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.009+0000 7fc48226b640 1 -- 172.21.15.57:0/3986370566 shutdown_connections 2023-10-30T14:46:19.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.009+0000 7fc48226b640 1 -- 172.21.15.57:0/3986370566 wait complete. 2023-10-30T14:46:19.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc48226b640 1 Processor -- start 2023-10-30T14:46:19.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc48226b640 1 -- start start 2023-10-30T14:46:19.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc48226b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc464004360 0x7fc47c1eec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:19.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc48226b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc47c15c420 0x7fc47c0ce420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:19.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc48226b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc47c0c57d0 0x7fc47c15b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:19.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc48226b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c1f21a0 con 0x7fc47c15c420 2023-10-30T14:46:19.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc48226b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c15af70 con 0x7fc47c0c57d0 2023-10-30T14:46:19.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc48226b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c1f23d0 con 0x7fc464004360 2023-10-30T14:46:19.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc47b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc47c15c420 0x7fc47c0ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:19.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc47b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc47c15c420 0x7fc47c0ce420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43848/0 (socket says 172.21.15.57:43848) 2023-10-30T14:46:19.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc47b7fe640 1 -- 172.21.15.57:0/4008029966 learned_addr learned my addr 172.21.15.57:0/4008029966 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:19.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc47bfff640 1 --2- 172.21.15.57:0/4008029966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc464004360 0x7fc47c1eec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:19.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc480808640 1 --2- 172.21.15.57:0/4008029966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc47c0c57d0 0x7fc47c15b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:19.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc47b7fe640 1 -- 172.21.15.57:0/4008029966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc464004360 msgr2=0x7fc47c1eec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:19.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc47b7fe640 1 --2- 172.21.15.57:0/4008029966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc464004360 0x7fc47c1eec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:19.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc47b7fe640 1 -- 172.21.15.57:0/4008029966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc47c0c57d0 msgr2=0x7fc47c15b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:19.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc47b7fe640 1 --2- 172.21.15.57:0/4008029966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc47c0c57d0 0x7fc47c15b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:19.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc47b7fe640 1 -- 172.21.15.57:0/4008029966 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47c1f26b0 con 0x7fc47c15c420 2023-10-30T14:46:19.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc47b7fe640 1 --2- 172.21.15.57:0/4008029966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc47c15c420 0x7fc47c0ce420 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fc4700af040 tx=0x7fc464005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:19.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc478ff9640 1 -- 172.21.15.57:0/4008029966 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc470005320 con 0x7fc47c15c420 2023-10-30T14:46:19.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc478ff9640 1 -- 172.21.15.57:0/4008029966 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc470005480 con 0x7fc47c15c420 2023-10-30T14:46:19.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc478ff9640 1 -- 172.21.15.57:0/4008029966 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc470007280 con 0x7fc47c15c420 2023-10-30T14:46:19.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc48226b640 1 -- 172.21.15.57:0/4008029966 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc47c1f2990 con 0x7fc47c15c420 2023-10-30T14:46:19.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.013+0000 7fc48226b640 1 -- 172.21.15.57:0/4008029966 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47c1f2f50 con 0x7fc47c15c420 2023-10-30T14:46:19.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.017+0000 7fc48226b640 1 -- 172.21.15.57:0/4008029966 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc47c04e170 con 0x7fc47c15c420 2023-10-30T14:46:19.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.017+0000 7fc478ff9640 1 -- 172.21.15.57:0/4008029966 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc4700b3040 con 0x7fc47c15c420 2023-10-30T14:46:19.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.017+0000 7fc478ff9640 1 --2- 172.21.15.57:0/4008029966 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc4680664d0 0x7fc468068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:19.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.017+0000 7fc478ff9640 1 -- 172.21.15.57:0/4008029966 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7fc47011e250 con 0x7fc47c15c420 2023-10-30T14:46:19.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.017+0000 7fc47bfff640 1 --2- 172.21.15.57:0/4008029966 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc4680664d0 0x7fc468068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:19.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.021+0000 7fc47bfff640 1 --2- 172.21.15.57:0/4008029966 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc4680664d0 0x7fc468068990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fc46c005180 tx=0x7fc46c001bb0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:19.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.021+0000 7fc478ff9640 1 -- 172.21.15.57:0/4008029966 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc4700eb0e0 con 0x7fc47c15c420 2023-10-30T14:46:19.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.213+0000 7fc48226b640 1 -- 172.21.15.57:0/4008029966 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc47c0b1b20 con 0x7fc47c15c420 2023-10-30T14:46:19.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.213+0000 7fc478ff9640 1 -- 172.21.15.57:0/4008029966 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6513 (secure 0 0 0) 0x7fc4700ee700 con 0x7fc47c15c420 2023-10-30T14:46:19.217 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:19.217 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:45:57.259750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:stopping","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:19.218 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 174 2023-10-30T14:46:19.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.217+0000 7fc48226b640 1 -- 172.21.15.57:0/4008029966 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc4680664d0 msgr2=0x7fc468068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:19.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.217+0000 7fc48226b640 1 --2- 172.21.15.57:0/4008029966 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc4680664d0 0x7fc468068990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fc46c005180 tx=0x7fc46c001bb0 comp rx=0 tx=0).stop 2023-10-30T14:46:19.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.217+0000 7fc48226b640 1 -- 172.21.15.57:0/4008029966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc47c15c420 msgr2=0x7fc47c0ce420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:19.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.217+0000 7fc48226b640 1 --2- 172.21.15.57:0/4008029966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc47c15c420 0x7fc47c0ce420 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fc4700af040 tx=0x7fc464005800 comp rx=0 tx=0).stop 2023-10-30T14:46:19.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.217+0000 7fc48226b640 1 -- 172.21.15.57:0/4008029966 shutdown_connections 2023-10-30T14:46:19.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.217+0000 7fc48226b640 1 --2- 172.21.15.57:0/4008029966 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc4680664d0 0x7fc468068990 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:19.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.217+0000 7fc48226b640 1 --2- 172.21.15.57:0/4008029966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc47c0c57d0 0x7fc47c15b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:19.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.217+0000 7fc48226b640 1 --2- 172.21.15.57:0/4008029966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc47c15c420 0x7fc47c0ce420 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:19.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.217+0000 7fc48226b640 1 --2- 172.21.15.57:0/4008029966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc464004360 0x7fc47c1eec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:19.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.217+0000 7fc48226b640 1 -- 172.21.15.57:0/4008029966 >> 172.21.15.57:0/4008029966 conn(0x7fc47c0715c0 msgr2=0x7fc47c0cf490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:19.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.217+0000 7fc48226b640 1 -- 172.21.15.57:0/4008029966 shutdown_connections 2023-10-30T14:46:19.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:19.217+0000 7fc48226b640 1 -- 172.21.15.57:0/4008029966 wait complete. 2023-10-30T14:46:19.239 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:46:21.240 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:21.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.393+0000 7fd2ee5a0640 1 Processor -- start 2023-10-30T14:46:21.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.393+0000 7fd2ee5a0640 1 -- start start 2023-10-30T14:46:21.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.393+0000 7fd2ee5a0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd2e815ac70 0x7fd2e815cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:21.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.393+0000 7fd2ee5a0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd2e806aeb0 0x7fd2e806b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:21.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.393+0000 7fd2ee5a0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e815dcc0 con 0x7fd2e80c8eb0 2023-10-30T14:46:21.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.393+0000 7fd2ee5a0640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e8159800 con 0x7fd2e815ac70 2023-10-30T14:46:21.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.393+0000 7fd2ee5a0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e8159d80 con 0x7fd2e806aeb0 2023-10-30T14:46:21.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.393+0000 7fd2ecd9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd2e815ac70 0x7fd2e815cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:21.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.393+0000 7fd2ecd9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd2e815ac70 0x7fd2e815cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43858/0 (socket says 172.21.15.57:43858) 2023-10-30T14:46:21.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.393+0000 7fd2ecd9d640 1 -- 172.21.15.57:0/2520041156 learned_addr learned my addr 172.21.15.57:0/2520041156 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:21.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.393+0000 7fd2edd9f640 1 --2- 172.21.15.57:0/2520041156 >> v2:172.21.15.88:3300/0 conn(0x7fd2e806aeb0 0x7fd2e806b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:21.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.397+0000 7fd2ecd9d640 1 -- 172.21.15.57:0/2520041156 >> v2:172.21.15.88:3300/0 conn(0x7fd2e806aeb0 msgr2=0x7fd2e806b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:21.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.397+0000 7fd2ecd9d640 1 --2- 172.21.15.57:0/2520041156 >> v2:172.21.15.88:3300/0 conn(0x7fd2e806aeb0 0x7fd2e806b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:21.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.397+0000 7fd2ecd9d640 1 -- 172.21.15.57:0/2520041156 >> v1:172.21.15.88:6789/0 conn(0x7fd2e80c8eb0 legacy=0x7fd2e815a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:21.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.397+0000 7fd2ecd9d640 1 -- 172.21.15.57:0/2520041156 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e815d420 con 0x7fd2e815ac70 2023-10-30T14:46:21.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.397+0000 7fd2ecd9d640 1 --2- 172.21.15.57:0/2520041156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2e815ac70 0x7fd2e815cd70 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fd2e808cce0 tx=0x7fd2d801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd932c125c2c53d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:21.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.397+0000 7fd2dffff640 1 -- 172.21.15.57:0/2520041156 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d80a89f0 con 0x7fd2e815ac70 2023-10-30T14:46:21.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.397+0000 7fd2dffff640 1 -- 172.21.15.57:0/2520041156 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d80a8b50 con 0x7fd2e815ac70 2023-10-30T14:46:21.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.397+0000 7fd2dffff640 1 -- 172.21.15.57:0/2520041156 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d80b0440 con 0x7fd2e815ac70 2023-10-30T14:46:21.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.397+0000 7fd2ee5a0640 1 -- 172.21.15.57:0/2520041156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2e815ac70 msgr2=0x7fd2e815cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:21.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.397+0000 7fd2ee5a0640 1 --2- 172.21.15.57:0/2520041156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2e815ac70 0x7fd2e815cd70 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fd2e808cce0 tx=0x7fd2d801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:46:21.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.397+0000 7fd2ee5a0640 1 -- 172.21.15.57:0/2520041156 shutdown_connections 2023-10-30T14:46:21.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.397+0000 7fd2ee5a0640 1 --2- 172.21.15.57:0/2520041156 >> v2:172.21.15.88:3300/0 conn(0x7fd2e806aeb0 0x7fd2e806b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:21.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.397+0000 7fd2ee5a0640 1 --2- 172.21.15.57:0/2520041156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2e815ac70 0x7fd2e815cd70 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:21.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.397+0000 7fd2ee5a0640 1 -- 172.21.15.57:0/2520041156 >> 172.21.15.57:0/2520041156 conn(0x7fd2e80cef00 msgr2=0x7fd2e80cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:21.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2ee5a0640 1 -- 172.21.15.57:0/2520041156 shutdown_connections 2023-10-30T14:46:21.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2ee5a0640 1 -- 172.21.15.57:0/2520041156 wait complete. 2023-10-30T14:46:21.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2ee5a0640 1 Processor -- start 2023-10-30T14:46:21.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2ee5a0640 1 -- start start 2023-10-30T14:46:21.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2ee5a0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2e806aeb0 0x7fd2e8154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:21.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2ee5a0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd2e80c8eb0 0x7fd2e80ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:21.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2ee5a0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2e815ac70 0x7fd2e80cdd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:21.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2ee5a0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e81e69f0 con 0x7fd2e815ac70 2023-10-30T14:46:21.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2ee5a0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e81551a0 con 0x7fd2e806aeb0 2023-10-30T14:46:21.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2ee5a0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e81e5de0 con 0x7fd2e80c8eb0 2023-10-30T14:46:21.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2edd9f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2e815ac70 0x7fd2e80cdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:21.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2edd9f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2e815ac70 0x7fd2e80cdd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43870/0 (socket says 172.21.15.57:43870) 2023-10-30T14:46:21.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2edd9f640 1 -- 172.21.15.57:0/833971083 learned_addr learned my addr 172.21.15.57:0/833971083 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:21.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2ecd9d640 1 --2- 172.21.15.57:0/833971083 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd2e80c8eb0 0x7fd2e80ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:21.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2ed59e640 1 --2- 172.21.15.57:0/833971083 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2e806aeb0 0x7fd2e8154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:21.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2edd9f640 1 -- 172.21.15.57:0/833971083 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd2e80c8eb0 msgr2=0x7fd2e80ca0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:21.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2edd9f640 1 --2- 172.21.15.57:0/833971083 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd2e80c8eb0 0x7fd2e80ca0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:21.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2edd9f640 1 -- 172.21.15.57:0/833971083 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2e806aeb0 msgr2=0x7fd2e8154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:21.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2edd9f640 1 --2- 172.21.15.57:0/833971083 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2e806aeb0 0x7fd2e8154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:21.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2edd9f640 1 -- 172.21.15.57:0/833971083 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d80094d0 con 0x7fd2e815ac70 2023-10-30T14:46:21.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2ed59e640 1 --2- 172.21.15.57:0/833971083 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2e806aeb0 0x7fd2e8154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:46:21.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2edd9f640 1 --2- 172.21.15.57:0/833971083 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2e815ac70 0x7fd2e80cdd30 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000efc0 tx=0x7fd2e000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:21.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2ddffb640 1 -- 172.21.15.57:0/833971083 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0014480 con 0x7fd2e815ac70 2023-10-30T14:46:21.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2ddffb640 1 -- 172.21.15.57:0/833971083 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e000f040 con 0x7fd2e815ac70 2023-10-30T14:46:21.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.401+0000 7fd2ee5a0640 1 -- 172.21.15.57:0/833971083 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e81e60c0 con 0x7fd2e815ac70 2023-10-30T14:46:21.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.405+0000 7fd2ddffb640 1 -- 172.21.15.57:0/833971083 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0014820 con 0x7fd2e815ac70 2023-10-30T14:46:21.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.405+0000 7fd2ee5a0640 1 -- 172.21.15.57:0/833971083 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e81e6680 con 0x7fd2e815ac70 2023-10-30T14:46:21.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.405+0000 7fd2ee5a0640 1 -- 172.21.15.57:0/833971083 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e81e2b70 con 0x7fd2e815ac70 2023-10-30T14:46:21.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.405+0000 7fd2ddffb640 1 -- 172.21.15.57:0/833971083 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd2e0014980 con 0x7fd2e815ac70 2023-10-30T14:46:21.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.405+0000 7fd2ddffb640 1 --2- 172.21.15.57:0/833971083 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd2cc0664a0 0x7fd2cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:21.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.405+0000 7fd2ddffb640 1 -- 172.21.15.57:0/833971083 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7fd2e0084fe0 con 0x7fd2e815ac70 2023-10-30T14:46:21.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.405+0000 7fd2ed59e640 1 --2- 172.21.15.57:0/833971083 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd2cc0664a0 0x7fd2cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:21.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.409+0000 7fd2ed59e640 1 --2- 172.21.15.57:0/833971083 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd2cc0664a0 0x7fd2cc068960 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4000f70 tx=0x7fd2e4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:21.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.409+0000 7fd2ddffb640 1 -- 172.21.15.57:0/833971083 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd2e81e2b70 con 0x7fd2e815ac70 2023-10-30T14:46:21.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.601+0000 7fd2ee5a0640 1 -- 172.21.15.57:0/833971083 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2e81e2b70 con 0x7fd2e815ac70 2023-10-30T14:46:21.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.601+0000 7fd2ddffb640 1 -- 172.21.15.57:0/833971083 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6513 (secure 0 0 0) 0x7fd2e81e2b70 con 0x7fd2e815ac70 2023-10-30T14:46:21.606 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:21.607 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:45:57.259750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:stopping","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:21.607 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 174 2023-10-30T14:46:21.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.605+0000 7fd2ee5a0640 1 -- 172.21.15.57:0/833971083 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd2cc0664a0 msgr2=0x7fd2cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:21.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.605+0000 7fd2ee5a0640 1 --2- 172.21.15.57:0/833971083 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd2cc0664a0 0x7fd2cc068960 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4000f70 tx=0x7fd2e4009040 comp rx=0 tx=0).stop 2023-10-30T14:46:21.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.605+0000 7fd2ee5a0640 1 -- 172.21.15.57:0/833971083 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2e815ac70 msgr2=0x7fd2e80cdd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:21.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.605+0000 7fd2ee5a0640 1 --2- 172.21.15.57:0/833971083 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2e815ac70 0x7fd2e80cdd30 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000efc0 tx=0x7fd2e000c490 comp rx=0 tx=0).stop 2023-10-30T14:46:21.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.605+0000 7fd2ee5a0640 1 -- 172.21.15.57:0/833971083 shutdown_connections 2023-10-30T14:46:21.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.605+0000 7fd2ee5a0640 1 --2- 172.21.15.57:0/833971083 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd2cc0664a0 0x7fd2cc068960 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:21.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.605+0000 7fd2ee5a0640 1 --2- 172.21.15.57:0/833971083 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2e815ac70 0x7fd2e80cdd30 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:21.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.605+0000 7fd2ee5a0640 1 --2- 172.21.15.57:0/833971083 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd2e80c8eb0 0x7fd2e80ca0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:21.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.605+0000 7fd2ee5a0640 1 --2- 172.21.15.57:0/833971083 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2e806aeb0 0x7fd2e8154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:21.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.605+0000 7fd2ee5a0640 1 -- 172.21.15.57:0/833971083 >> 172.21.15.57:0/833971083 conn(0x7fd2e80cef00 msgr2=0x7fd2e806a300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:21.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.605+0000 7fd2ee5a0640 1 -- 172.21.15.57:0/833971083 shutdown_connections 2023-10-30T14:46:21.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:21.605+0000 7fd2ee5a0640 1 -- 172.21.15.57:0/833971083 wait complete. 2023-10-30T14:46:21.627 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T14:46:21.628 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 174, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 174, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:stopping", "state_seq": 44 }, "gid_7536": { "addr": "172.21.15.57:6837/816502245", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 816502245, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 816502245, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 7536, "incarnation": 164, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:45:57.259750+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7536, "mds_1": 6747 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6839/1078372885", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1078372885, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1078372885, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 170, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6716, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.57:6834/697507447", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 697507447, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 697507447, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 156, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7377, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:46:23.629 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:23.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.781+0000 7fea23d3e640 1 Processor -- start 2023-10-30T14:46:23.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.781+0000 7fea23d3e640 1 -- start start 2023-10-30T14:46:23.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.781+0000 7fea23d3e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea1c153c80 0x7fea1c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:23.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.781+0000 7fea23d3e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea1c159f20 con 0x7fea1c0c42a0 2023-10-30T14:46:23.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.781+0000 7fea23d3e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea1c15d620 con 0x7fea1c0c4a10 2023-10-30T14:46:23.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.781+0000 7fea23d3e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c15dec0 con 0x7fea1c153c80 2023-10-30T14:46:23.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.781+0000 7fea222db640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fea1c0c42a0 0x7fea1c0ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:53030/0 (socket says 172.21.15.57:53030) 2023-10-30T14:46:23.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.781+0000 7fea222db640 1 -- 172.21.15.57:0/2749784708 learned_addr learned my addr 172.21.15.57:0/2749784708 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:46:23.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.781+0000 7fea21ada640 1 --2- 172.21.15.57:0/2749784708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea1c153c80 0x7fea1c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:23.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.781+0000 7fea20ad8640 1 -- 172.21.15.57:0/2749784708 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 238013055 0 0) 0x7fea1c159f20 con 0x7fea1c0c42a0 2023-10-30T14:46:23.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.781+0000 7fea20ad8640 1 -- 172.21.15.57:0/2749784708 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea08003530 con 0x7fea1c0c42a0 2023-10-30T14:46:23.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea20ad8640 1 -- 172.21.15.57:0/2749784708 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2892099544 0 0) 0x7fea08003530 con 0x7fea1c0c42a0 2023-10-30T14:46:23.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea20ad8640 1 -- 172.21.15.57:0/2749784708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea1c153c80 msgr2=0x7fea1c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:23.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea20ad8640 1 --2- 172.21.15.57:0/2749784708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea1c153c80 0x7fea1c15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:23.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea20ad8640 1 -- 172.21.15.57:0/2749784708 >> v1:172.21.15.88:6789/0 conn(0x7fea1c0c4a10 legacy=0x7fea1c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:23.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea20ad8640 1 -- 172.21.15.57:0/2749784708 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1c15d900 con 0x7fea1c0c42a0 2023-10-30T14:46:23.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea20ad8640 1 -- 172.21.15.57:0/2749784708 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fea14001a00 con 0x7fea1c0c42a0 2023-10-30T14:46:23.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea20ad8640 1 -- 172.21.15.57:0/2749784708 >> v1:172.21.15.57:6789/0 conn(0x7fea1c0c42a0 legacy=0x7fea1c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:23.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea20ad8640 1 --2- 172.21.15.57:0/2749784708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea080039b0 0x7fea08003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:23.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea20ad8640 1 --2- 172.21.15.57:0/2749784708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea080047d0 0x7fea08004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:23.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea20ad8640 1 --2- 172.21.15.57:0/2749784708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea1c153c80 0x7fea08005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:23.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea20ad8640 1 -- 172.21.15.57:0/2749784708 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c15dec0 con 0x7fea080039b0 2023-10-30T14:46:23.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea20ad8640 1 -- 172.21.15.57:0/2749784708 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0800bad0 con 0x7fea080047d0 2023-10-30T14:46:23.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea20ad8640 1 -- 172.21.15.57:0/2749784708 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea0800bc90 con 0x7fea1c153c80 2023-10-30T14:46:23.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea21ada640 1 --2- 172.21.15.57:0/2749784708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea080039b0 0x7fea08003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:23.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea212d9640 1 --2- 172.21.15.57:0/2749784708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea080047d0 0x7fea08004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:23.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea222db640 1 --2- 172.21.15.57:0/2749784708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea1c153c80 0x7fea08005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:23.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea21ada640 1 -- 172.21.15.57:0/2749784708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea1c153c80 msgr2=0x7fea08005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:23.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea21ada640 1 --2- 172.21.15.57:0/2749784708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea1c153c80 0x7fea08005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:23.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea21ada640 1 -- 172.21.15.57:0/2749784708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea080047d0 msgr2=0x7fea08004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:23.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea21ada640 1 --2- 172.21.15.57:0/2749784708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea080047d0 0x7fea08004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:23.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea21ada640 1 -- 172.21.15.57:0/2749784708 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea08003530 con 0x7fea080039b0 2023-10-30T14:46:23.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea222db640 1 --2- 172.21.15.57:0/2749784708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea1c153c80 0x7fea08005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:46:23.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea21ada640 1 --2- 172.21.15.57:0/2749784708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea080039b0 0x7fea08003dd0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fea18009ab0 tx=0x7fea1801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d53838f5676982aa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:23.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea20ad8640 1 -- 172.21.15.57:0/2749784708 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea180a5600 con 0x7fea080039b0 2023-10-30T14:46:23.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea20ad8640 1 -- 172.21.15.57:0/2749784708 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea14002100 con 0x7fea080039b0 2023-10-30T14:46:23.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.785+0000 7fea20ad8640 1 -- 172.21.15.57:0/2749784708 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea180a59a0 con 0x7fea080039b0 2023-10-30T14:46:23.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea23d3e640 1 -- 172.21.15.57:0/2749784708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea080039b0 msgr2=0x7fea08003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:23.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea23d3e640 1 --2- 172.21.15.57:0/2749784708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea080039b0 0x7fea08003dd0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fea18009ab0 tx=0x7fea1801ca10 comp rx=0 tx=0).stop 2023-10-30T14:46:23.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea222db640 1 -- 172.21.15.57:0/2749784708 reap_dead start 2023-10-30T14:46:23.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea23d3e640 1 -- 172.21.15.57:0/2749784708 shutdown_connections 2023-10-30T14:46:23.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea23d3e640 1 -- 172.21.15.57:0/2749784708 >> 172.21.15.57:0/2749784708 conn(0x7fea1c0715c0 msgr2=0x7fea1c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:23.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea23d3e640 1 -- 172.21.15.57:0/2749784708 shutdown_connections 2023-10-30T14:46:23.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea23d3e640 1 -- 172.21.15.57:0/2749784708 wait complete. 2023-10-30T14:46:23.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea23d3e640 1 Processor -- start 2023-10-30T14:46:23.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea23d3e640 1 -- start start 2023-10-30T14:46:23.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea23d3e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea080039b0 0x7fea1c1569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:23.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea23d3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea1c15c490 0x7fea1c0cd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:23.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea23d3e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1c0cdcd0 0x7fea1c1582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:23.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea23d3e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c0cd520 con 0x7fea1c0cdcd0 2023-10-30T14:46:23.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea23d3e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c157320 con 0x7fea1c15c490 2023-10-30T14:46:23.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea23d3e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c158de0 con 0x7fea080039b0 2023-10-30T14:46:23.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea222db640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1c0cdcd0 0x7fea1c1582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:23.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea222db640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1c0cdcd0 0x7fea1c1582b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49794/0 (socket says 172.21.15.57:49794) 2023-10-30T14:46:23.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea222db640 1 -- 172.21.15.57:0/403837512 learned_addr learned my addr 172.21.15.57:0/403837512 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:23.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea21ada640 1 --2- 172.21.15.57:0/403837512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea080039b0 0x7fea1c1569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:23.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea212d9640 1 --2- 172.21.15.57:0/403837512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea1c15c490 0x7fea1c0cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:23.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea222db640 1 -- 172.21.15.57:0/403837512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea080039b0 msgr2=0x7fea1c1569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:23.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.789+0000 7fea222db640 1 --2- 172.21.15.57:0/403837512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea080039b0 0x7fea1c1569a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:23.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.793+0000 7fea222db640 1 -- 172.21.15.57:0/403837512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea1c15c490 msgr2=0x7fea1c0cd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:23.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.793+0000 7fea222db640 1 --2- 172.21.15.57:0/403837512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea1c15c490 0x7fea1c0cd790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:23.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.793+0000 7fea222db640 1 -- 172.21.15.57:0/403837512 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1c157940 con 0x7fea1c0cdcd0 2023-10-30T14:46:23.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.793+0000 7fea212d9640 1 --2- 172.21.15.57:0/403837512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea1c15c490 0x7fea1c0cd790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:46:23.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.793+0000 7fea21ada640 1 --2- 172.21.15.57:0/403837512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea080039b0 0x7fea1c1569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:46:23.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.793+0000 7fea222db640 1 --2- 172.21.15.57:0/403837512 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1c0cdcd0 0x7fea1c1582b0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fea14000f90 tx=0x7fea14010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:23.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.793+0000 7fea127fc640 1 -- 172.21.15.57:0/403837512 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1400eac0 con 0x7fea1c0cdcd0 2023-10-30T14:46:23.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.793+0000 7fea127fc640 1 -- 172.21.15.57:0/403837512 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea1400ec20 con 0x7fea1c0cdcd0 2023-10-30T14:46:23.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.793+0000 7fea127fc640 1 -- 172.21.15.57:0/403837512 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea14015400 con 0x7fea1c0cdcd0 2023-10-30T14:46:23.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.793+0000 7fea23d3e640 1 -- 172.21.15.57:0/403837512 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea1c1efb00 con 0x7fea1c0cdcd0 2023-10-30T14:46:23.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.793+0000 7fea23d3e640 1 -- 172.21.15.57:0/403837512 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea1c14fda0 con 0x7fea1c0cdcd0 2023-10-30T14:46:23.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.793+0000 7fea23d3e640 1 -- 172.21.15.57:0/403837512 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea1c1efd20 con 0x7fea1c0cdcd0 2023-10-30T14:46:23.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.793+0000 7fea127fc640 1 -- 172.21.15.57:0/403837512 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fea1401e3b0 con 0x7fea1c0cdcd0 2023-10-30T14:46:23.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.793+0000 7fea127fc640 1 --2- 172.21.15.57:0/403837512 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fea000664a0 0x7fea00068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:23.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.797+0000 7fea127fc640 1 -- 172.21.15.57:0/403837512 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7fea140870b0 con 0x7fea1c0cdcd0 2023-10-30T14:46:23.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.797+0000 7fea21ada640 1 --2- 172.21.15.57:0/403837512 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fea000664a0 0x7fea00068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:23.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.797+0000 7fea21ada640 1 --2- 172.21.15.57:0/403837512 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fea000664a0 0x7fea00068960 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fea180af000 tx=0x7fea180046d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:23.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.801+0000 7fea127fc640 1 -- 172.21.15.57:0/403837512 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fea140579e0 con 0x7fea1c0cdcd0 2023-10-30T14:46:23.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.989+0000 7fea23d3e640 1 -- 172.21.15.57:0/403837512 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea1c04e170 con 0x7fea1c0cdcd0 2023-10-30T14:46:23.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.989+0000 7fea127fc640 1 -- 172.21.15.57:0/403837512 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6513 (secure 0 0 0) 0x7fea1401a020 con 0x7fea1c0cdcd0 2023-10-30T14:46:23.992 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:23.992 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:45:57.259750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:stopping","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:23.992 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 174 2023-10-30T14:46:23.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.989+0000 7fea23d3e640 1 -- 172.21.15.57:0/403837512 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fea000664a0 msgr2=0x7fea00068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:23.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.989+0000 7fea23d3e640 1 --2- 172.21.15.57:0/403837512 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fea000664a0 0x7fea00068960 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fea180af000 tx=0x7fea180046d0 comp rx=0 tx=0).stop 2023-10-30T14:46:23.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.989+0000 7fea23d3e640 1 -- 172.21.15.57:0/403837512 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1c0cdcd0 msgr2=0x7fea1c1582b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:23.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.989+0000 7fea23d3e640 1 --2- 172.21.15.57:0/403837512 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1c0cdcd0 0x7fea1c1582b0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fea14000f90 tx=0x7fea14010d70 comp rx=0 tx=0).stop 2023-10-30T14:46:23.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.989+0000 7fea23d3e640 1 -- 172.21.15.57:0/403837512 shutdown_connections 2023-10-30T14:46:23.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.989+0000 7fea23d3e640 1 --2- 172.21.15.57:0/403837512 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fea000664a0 0x7fea00068960 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:23.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.989+0000 7fea23d3e640 1 --2- 172.21.15.57:0/403837512 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1c0cdcd0 0x7fea1c1582b0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:23.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.989+0000 7fea23d3e640 1 --2- 172.21.15.57:0/403837512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea1c15c490 0x7fea1c0cd790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:23.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.989+0000 7fea23d3e640 1 --2- 172.21.15.57:0/403837512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea080039b0 0x7fea1c1569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:23.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.989+0000 7fea23d3e640 1 -- 172.21.15.57:0/403837512 >> 172.21.15.57:0/403837512 conn(0x7fea1c0715c0 msgr2=0x7fea1c0ca140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:23.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.989+0000 7fea23d3e640 1 -- 172.21.15.57:0/403837512 shutdown_connections 2023-10-30T14:46:23.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:23.989+0000 7fea23d3e640 1 -- 172.21.15.57:0/403837512 wait complete. 2023-10-30T14:46:24.012 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:46:26.013 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:26.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.165+0000 7f4703c46640 1 Processor -- start 2023-10-30T14:46:26.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.165+0000 7f4703c46640 1 -- start start 2023-10-30T14:46:26.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.165+0000 7f4703c46640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f46fc06b760 0x7f46fc15e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:26.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.165+0000 7f4703c46640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f46fc06c420 0x7f46fc0c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:26.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.165+0000 7f4703c46640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46fc071e00 con 0x7f46fc06bb40 2023-10-30T14:46:26.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.165+0000 7f4703c46640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46fc070020 con 0x7f46fc06c420 2023-10-30T14:46:26.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.165+0000 7f4703c46640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46fc15eb90 con 0x7f46fc06b760 2023-10-30T14:46:26.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.169+0000 7f47011e1640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f46fc06bb40 0x7f46fc157a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:53044/0 (socket says 172.21.15.57:53044) 2023-10-30T14:46:26.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.169+0000 7f47011e1640 1 -- 172.21.15.57:0/3028745982 learned_addr learned my addr 172.21.15.57:0/3028745982 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:46:26.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.169+0000 7f47021e3640 1 --2- 172.21.15.57:0/3028745982 >> v2:172.21.15.57:3300/0 conn(0x7f46fc06c420 0x7f46fc0c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:26.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.169+0000 7f47019e2640 1 --2- 172.21.15.57:0/3028745982 >> v2:172.21.15.88:3300/0 conn(0x7f46fc06b760 0x7f46fc15e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:26.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.169+0000 7f47009e0640 1 -- 172.21.15.57:0/3028745982 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534727420 0 0) 0x7f46fc071e00 con 0x7f46fc06bb40 2023-10-30T14:46:26.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.169+0000 7f47009e0640 1 -- 172.21.15.57:0/3028745982 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e8003530 con 0x7f46fc06bb40 2023-10-30T14:46:26.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.169+0000 7f47021e3640 1 -- 172.21.15.57:0/3028745982 >> v2:172.21.15.88:3300/0 conn(0x7f46fc06b760 msgr2=0x7f46fc15e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:26.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.169+0000 7f47021e3640 1 --2- 172.21.15.57:0/3028745982 >> v2:172.21.15.88:3300/0 conn(0x7f46fc06b760 0x7f46fc15e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:26.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.169+0000 7f47021e3640 1 -- 172.21.15.57:0/3028745982 >> v1:172.21.15.57:6789/0 conn(0x7f46fc06bb40 legacy=0x7f46fc157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:26.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.169+0000 7f47021e3640 1 -- 172.21.15.57:0/3028745982 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc070f10 con 0x7f46fc06c420 2023-10-30T14:46:26.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.169+0000 7f47021e3640 1 --2- 172.21.15.57:0/3028745982 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46fc06c420 0x7f46fc0c57a0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0cb840 tx=0x7f46f401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30874e8dc4501420 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:26.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.169+0000 7f47009e0640 1 -- 172.21.15.57:0/3028745982 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f40a59c0 con 0x7f46fc06c420 2023-10-30T14:46:26.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.169+0000 7f47009e0640 1 -- 172.21.15.57:0/3028745982 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f40a5b20 con 0x7f46fc06c420 2023-10-30T14:46:26.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.169+0000 7f47009e0640 1 -- 172.21.15.57:0/3028745982 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f40ad440 con 0x7f46fc06c420 2023-10-30T14:46:26.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f4703c46640 1 -- 172.21.15.57:0/3028745982 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46fc06c420 msgr2=0x7f46fc0c57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:26.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f4703c46640 1 --2- 172.21.15.57:0/3028745982 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46fc06c420 0x7f46fc0c57a0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0cb840 tx=0x7f46f401c8e0 comp rx=0 tx=0).stop 2023-10-30T14:46:26.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f4703c46640 1 -- 172.21.15.57:0/3028745982 shutdown_connections 2023-10-30T14:46:26.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f4703c46640 1 --2- 172.21.15.57:0/3028745982 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46fc06c420 0x7f46fc0c57a0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:26.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f4703c46640 1 --2- 172.21.15.57:0/3028745982 >> v2:172.21.15.88:3300/0 conn(0x7f46fc06b760 0x7f46fc15e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:26.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f4703c46640 1 -- 172.21.15.57:0/3028745982 >> 172.21.15.57:0/3028745982 conn(0x7f46fc0cac70 msgr2=0x7f46fc0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:26.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f4703c46640 1 -- 172.21.15.57:0/3028745982 shutdown_connections 2023-10-30T14:46:26.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f4703c46640 1 -- 172.21.15.57:0/3028745982 wait complete. 2023-10-30T14:46:26.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f4703c46640 1 Processor -- start 2023-10-30T14:46:26.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f4703c46640 1 -- start start 2023-10-30T14:46:26.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f4703c46640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46fc06b760 0x7f46fc14ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:26.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f4703c46640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46fc06bb40 0x7f46fc14b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:26.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f4703c46640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46fc06c420 0x7f46fc148920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:26.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f4703c46640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc1f6ae0 con 0x7f46fc06b760 2023-10-30T14:46:26.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f4703c46640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc0c9c80 con 0x7f46fc06bb40 2023-10-30T14:46:26.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f4703c46640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc0c9eb0 con 0x7f46fc06c420 2023-10-30T14:46:26.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f47019e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46fc06b760 0x7f46fc14ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:26.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f47019e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46fc06b760 0x7f46fc14ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49812/0 (socket says 172.21.15.57:49812) 2023-10-30T14:46:26.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f47019e2640 1 -- 172.21.15.57:0/3288183445 learned_addr learned my addr 172.21.15.57:0/3288183445 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:26.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f47011e1640 1 --2- 172.21.15.57:0/3288183445 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46fc06bb40 0x7f46fc14b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:26.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.173+0000 7f47021e3640 1 --2- 172.21.15.57:0/3288183445 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46fc06c420 0x7f46fc148920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:26.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.177+0000 7f47019e2640 1 -- 172.21.15.57:0/3288183445 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46fc06c420 msgr2=0x7f46fc148920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:26.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.177+0000 7f47019e2640 1 --2- 172.21.15.57:0/3288183445 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46fc06c420 0x7f46fc148920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:26.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.177+0000 7f47019e2640 1 -- 172.21.15.57:0/3288183445 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46fc06bb40 msgr2=0x7f46fc14b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:26.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.177+0000 7f47019e2640 1 --2- 172.21.15.57:0/3288183445 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46fc06bb40 0x7f46fc14b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:26.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.177+0000 7f47019e2640 1 -- 172.21.15.57:0/3288183445 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f4009a70 con 0x7f46fc06b760 2023-10-30T14:46:26.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.177+0000 7f47019e2640 1 --2- 172.21.15.57:0/3288183445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46fc06b760 0x7f46fc14ae70 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f46f800e930 tx=0x7f46f800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:26.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.177+0000 7f46f27fc640 1 -- 172.21.15.57:0/3288183445 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f8009690 con 0x7f46fc06b760 2023-10-30T14:46:26.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.177+0000 7f46f27fc640 1 -- 172.21.15.57:0/3288183445 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f800ca00 con 0x7f46fc06b760 2023-10-30T14:46:26.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.177+0000 7f4703c46640 1 -- 172.21.15.57:0/3288183445 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46fc1f6140 con 0x7f46fc06b760 2023-10-30T14:46:26.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.177+0000 7f46f27fc640 1 -- 172.21.15.57:0/3288183445 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f800cd70 con 0x7f46fc06b760 2023-10-30T14:46:26.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.177+0000 7f4703c46640 1 -- 172.21.15.57:0/3288183445 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46fc1f6700 con 0x7f46fc06b760 2023-10-30T14:46:26.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.177+0000 7f4703c46640 1 -- 172.21.15.57:0/3288183445 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46fc04e170 con 0x7f46fc06b760 2023-10-30T14:46:26.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.177+0000 7f46f27fc640 1 -- 172.21.15.57:0/3288183445 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f46f8013470 con 0x7f46fc06b760 2023-10-30T14:46:26.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.177+0000 7f46f27fc640 1 --2- 172.21.15.57:0/3288183445 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f46e00664a0 0x7f46e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:26.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.177+0000 7f46f27fc640 1 -- 172.21.15.57:0/3288183445 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f46f8084730 con 0x7f46fc06b760 2023-10-30T14:46:26.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.181+0000 7f47011e1640 1 --2- 172.21.15.57:0/3288183445 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f46e00664a0 0x7f46e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:26.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.181+0000 7f47011e1640 1 --2- 172.21.15.57:0/3288183445 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f46e00664a0 0x7f46e0068960 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0c9300 tx=0x7f46ec009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:26.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.181+0000 7f46f27fc640 1 -- 172.21.15.57:0/3288183445 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f46f80515f0 con 0x7f46fc06b760 2023-10-30T14:46:26.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.373+0000 7f4703c46640 1 -- 172.21.15.57:0/3288183445 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46fc14e5b0 con 0x7f46fc06b760 2023-10-30T14:46:26.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.373+0000 7f46f27fc640 1 -- 172.21.15.57:0/3288183445 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6513 (secure 0 0 0) 0x7f46f8054c10 con 0x7f46fc06b760 2023-10-30T14:46:26.378 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:26.379 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:45:57.259750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:stopping","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:26.379 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 174 2023-10-30T14:46:26.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.377+0000 7f4703c46640 1 -- 172.21.15.57:0/3288183445 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f46e00664a0 msgr2=0x7f46e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:26.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.377+0000 7f4703c46640 1 --2- 172.21.15.57:0/3288183445 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f46e00664a0 0x7f46e0068960 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0c9300 tx=0x7f46ec009040 comp rx=0 tx=0).stop 2023-10-30T14:46:26.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.377+0000 7f4703c46640 1 -- 172.21.15.57:0/3288183445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46fc06b760 msgr2=0x7f46fc14ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:26.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.377+0000 7f4703c46640 1 --2- 172.21.15.57:0/3288183445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46fc06b760 0x7f46fc14ae70 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f46f800e930 tx=0x7f46f800edf0 comp rx=0 tx=0).stop 2023-10-30T14:46:26.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.377+0000 7f4703c46640 1 -- 172.21.15.57:0/3288183445 shutdown_connections 2023-10-30T14:46:26.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.377+0000 7f4703c46640 1 --2- 172.21.15.57:0/3288183445 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f46e00664a0 0x7f46e0068960 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:26.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.377+0000 7f4703c46640 1 --2- 172.21.15.57:0/3288183445 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46fc06c420 0x7f46fc148920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:26.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.377+0000 7f4703c46640 1 --2- 172.21.15.57:0/3288183445 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46fc06bb40 0x7f46fc14b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:26.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.377+0000 7f4703c46640 1 --2- 172.21.15.57:0/3288183445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46fc06b760 0x7f46fc14ae70 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:26.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.377+0000 7f4703c46640 1 -- 172.21.15.57:0/3288183445 >> 172.21.15.57:0/3288183445 conn(0x7f46fc0cac70 msgr2=0x7f46fc072550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:26.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.377+0000 7f4703c46640 1 -- 172.21.15.57:0/3288183445 shutdown_connections 2023-10-30T14:46:26.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:26.377+0000 7f4703c46640 1 -- 172.21.15.57:0/3288183445 wait complete. 2023-10-30T14:46:26.399 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:46:28.401 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:28.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.552+0000 7ff09286b640 1 Processor -- start 2023-10-30T14:46:28.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.552+0000 7ff09286b640 1 -- start start 2023-10-30T14:46:28.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.552+0000 7ff09286b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff08c06ba00 0x7ff08c15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:28.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.552+0000 7ff09286b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08c0c2aa0 con 0x7ff08c0701a0 2023-10-30T14:46:28.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.552+0000 7ff09286b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08c0708e0 con 0x7ff08c1537f0 2023-10-30T14:46:28.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.552+0000 7ff09286b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c0c3ab0 con 0x7ff08c06ba00 2023-10-30T14:46:28.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.552+0000 7ff08b7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff08c0701a0 0x7ff08c06b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:53048/0 (socket says 172.21.15.57:53048) 2023-10-30T14:46:28.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.552+0000 7ff08b7fe640 1 -- 172.21.15.57:0/3951926498 learned_addr learned my addr 172.21.15.57:0/3951926498 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:46:28.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.552+0000 7ff090e08640 1 --2- 172.21.15.57:0/3951926498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff08c06ba00 0x7ff08c15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:28.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.552+0000 7ff08affd640 1 -- 172.21.15.57:0/3951926498 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461806298 0 0) 0x7ff08c0c2aa0 con 0x7ff08c0701a0 2023-10-30T14:46:28.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.552+0000 7ff08affd640 1 -- 172.21.15.57:0/3951926498 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff074003530 con 0x7ff08c0701a0 2023-10-30T14:46:28.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.552+0000 7ff08affd640 1 -- 172.21.15.57:0/3951926498 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601687596 0 0) 0x7ff08c0708e0 con 0x7ff08c1537f0 2023-10-30T14:46:28.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.552+0000 7ff08affd640 1 -- 172.21.15.57:0/3951926498 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff08c0c2aa0 con 0x7ff08c1537f0 2023-10-30T14:46:28.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.552+0000 7ff08affd640 1 -- 172.21.15.57:0/3951926498 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1277295293 0 0) 0x7ff074003530 con 0x7ff08c0701a0 2023-10-30T14:46:28.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.552+0000 7ff090e08640 1 -- 172.21.15.57:0/3951926498 >> v1:172.21.15.88:6789/0 conn(0x7ff08c1537f0 legacy=0x7ff08c0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:28.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.552+0000 7ff090e08640 1 -- 172.21.15.57:0/3951926498 >> v1:172.21.15.57:6789/0 conn(0x7ff08c0701a0 legacy=0x7ff08c06b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:28.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.552+0000 7ff090e08640 1 -- 172.21.15.57:0/3951926498 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08c0c1eb0 con 0x7ff08c06ba00 2023-10-30T14:46:28.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.556+0000 7ff090e08640 1 --2- 172.21.15.57:0/3951926498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff08c06ba00 0x7ff08c15a2e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff084009dc0 tx=0x7ff08401f080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f36a6e5bd340c66a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:28.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.556+0000 7ff08affd640 1 -- 172.21.15.57:0/3951926498 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff080001680 con 0x7ff08c06ba00 2023-10-30T14:46:28.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.556+0000 7ff08affd640 1 -- 172.21.15.57:0/3951926498 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0840a8c10 con 0x7ff08c06ba00 2023-10-30T14:46:28.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.556+0000 7ff08affd640 1 -- 172.21.15.57:0/3951926498 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0840b0440 con 0x7ff08c06ba00 2023-10-30T14:46:28.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.556+0000 7ff09286b640 1 -- 172.21.15.57:0/3951926498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff08c06ba00 msgr2=0x7ff08c15a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:28.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.556+0000 7ff09286b640 1 --2- 172.21.15.57:0/3951926498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff08c06ba00 0x7ff08c15a2e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff084009dc0 tx=0x7ff08401f080 comp rx=0 tx=0).stop 2023-10-30T14:46:28.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.556+0000 7ff09286b640 1 -- 172.21.15.57:0/3951926498 shutdown_connections 2023-10-30T14:46:28.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.556+0000 7ff09286b640 1 --2- 172.21.15.57:0/3951926498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff08c06ba00 0x7ff08c15a2e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:28.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.556+0000 7ff09286b640 1 -- 172.21.15.57:0/3951926498 >> 172.21.15.57:0/3951926498 conn(0x7ff08c071fb0 msgr2=0x7ff08c0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:28.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.556+0000 7ff09286b640 1 -- 172.21.15.57:0/3951926498 shutdown_connections 2023-10-30T14:46:28.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.556+0000 7ff09286b640 1 -- 172.21.15.57:0/3951926498 wait complete. 2023-10-30T14:46:28.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff09286b640 1 Processor -- start 2023-10-30T14:46:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff09286b640 1 -- start start 2023-10-30T14:46:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff09286b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff08c06ba00 0x7ff08c0c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff09286b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff08c0701a0 0x7ff08c1f0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff09286b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff08c1537f0 0x7ff08c0cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff09286b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c0c89f0 con 0x7ff08c1537f0 2023-10-30T14:46:28.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff09286b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c1497a0 con 0x7ff08c0701a0 2023-10-30T14:46:28.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff09286b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c0c80e0 con 0x7ff08c06ba00 2023-10-30T14:46:28.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff090e08640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff08c1537f0 0x7ff08c0cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:28.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff090e08640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff08c1537f0 0x7ff08c0cc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49828/0 (socket says 172.21.15.57:49828) 2023-10-30T14:46:28.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff090e08640 1 -- 172.21.15.57:0/3270922718 learned_addr learned my addr 172.21.15.57:0/3270922718 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:28.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff08bfff640 1 --2- 172.21.15.57:0/3270922718 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff08c06ba00 0x7ff08c0c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:28.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff08b7fe640 1 --2- 172.21.15.57:0/3270922718 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff08c0701a0 0x7ff08c1f0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:28.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff090e08640 1 -- 172.21.15.57:0/3270922718 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff08c06ba00 msgr2=0x7ff08c0c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:28.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff090e08640 1 --2- 172.21.15.57:0/3270922718 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff08c06ba00 0x7ff08c0c8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:28.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff090e08640 1 -- 172.21.15.57:0/3270922718 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff08c0701a0 msgr2=0x7ff08c1f0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:28.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff090e08640 1 --2- 172.21.15.57:0/3270922718 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff08c0701a0 0x7ff08c1f0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:28.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff090e08640 1 -- 172.21.15.57:0/3270922718 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff084009a70 con 0x7ff08c1537f0 2023-10-30T14:46:28.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff08b7fe640 1 --2- 172.21.15.57:0/3270922718 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff08c0701a0 0x7ff08c1f0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:46:28.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff090e08640 1 --2- 172.21.15.57:0/3270922718 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff08c1537f0 0x7ff08c0cc060 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7ff08401d730 tx=0x7ff08401be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:28.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff088ff9640 1 -- 172.21.15.57:0/3270922718 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0840c0070 con 0x7ff08c1537f0 2023-10-30T14:46:28.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff088ff9640 1 -- 172.21.15.57:0/3270922718 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff084009450 con 0x7ff08c1537f0 2023-10-30T14:46:28.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff088ff9640 1 -- 172.21.15.57:0/3270922718 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084004450 con 0x7ff08c1537f0 2023-10-30T14:46:28.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.560+0000 7ff09286b640 1 -- 172.21.15.57:0/3270922718 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08c1ef6f0 con 0x7ff08c1537f0 2023-10-30T14:46:28.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.564+0000 7ff09286b640 1 -- 172.21.15.57:0/3270922718 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08c0c8330 con 0x7ff08c1537f0 2023-10-30T14:46:28.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.564+0000 7ff09286b640 1 -- 172.21.15.57:0/3270922718 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08c0c8b30 con 0x7ff08c1537f0 2023-10-30T14:46:28.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.564+0000 7ff088ff9640 1 -- 172.21.15.57:0/3270922718 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff0840b7030 con 0x7ff08c1537f0 2023-10-30T14:46:28.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.564+0000 7ff088ff9640 1 --2- 172.21.15.57:0/3270922718 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0780664a0 0x7ff078068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:28.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.564+0000 7ff088ff9640 1 -- 172.21.15.57:0/3270922718 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7ff084122110 con 0x7ff08c1537f0 2023-10-30T14:46:28.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.564+0000 7ff08bfff640 1 --2- 172.21.15.57:0/3270922718 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0780664a0 0x7ff078068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:28.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.568+0000 7ff08bfff640 1 --2- 172.21.15.57:0/3270922718 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0780664a0 0x7ff078068960 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7ff07c0011f0 tx=0x7ff07c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:28.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.568+0000 7ff088ff9640 1 -- 172.21.15.57:0/3270922718 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff08c0c8b30 con 0x7ff08c1537f0 2023-10-30T14:46:28.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.764+0000 7ff09286b640 1 -- 172.21.15.57:0/3270922718 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff08c0c8b30 con 0x7ff08c1537f0 2023-10-30T14:46:28.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.764+0000 7ff088ff9640 1 -- 172.21.15.57:0/3270922718 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6513 (secure 0 0 0) 0x7ff08c0c8b30 con 0x7ff08c1537f0 2023-10-30T14:46:28.768 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:28.768 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:45:57.259750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:stopping","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:28.770 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 174 2023-10-30T14:46:28.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.764+0000 7ff09286b640 1 -- 172.21.15.57:0/3270922718 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0780664a0 msgr2=0x7ff078068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:28.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.764+0000 7ff09286b640 1 --2- 172.21.15.57:0/3270922718 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0780664a0 0x7ff078068960 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7ff07c0011f0 tx=0x7ff07c009040 comp rx=0 tx=0).stop 2023-10-30T14:46:28.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.764+0000 7ff09286b640 1 -- 172.21.15.57:0/3270922718 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff08c1537f0 msgr2=0x7ff08c0cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:28.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.764+0000 7ff09286b640 1 --2- 172.21.15.57:0/3270922718 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff08c1537f0 0x7ff08c0cc060 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7ff08401d730 tx=0x7ff08401be20 comp rx=0 tx=0).stop 2023-10-30T14:46:28.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.764+0000 7ff09286b640 1 -- 172.21.15.57:0/3270922718 shutdown_connections 2023-10-30T14:46:28.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.764+0000 7ff09286b640 1 --2- 172.21.15.57:0/3270922718 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0780664a0 0x7ff078068960 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:28.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.764+0000 7ff09286b640 1 --2- 172.21.15.57:0/3270922718 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff08c1537f0 0x7ff08c0cc060 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:28.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.764+0000 7ff09286b640 1 --2- 172.21.15.57:0/3270922718 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff08c0701a0 0x7ff08c1f0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:28.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.764+0000 7ff09286b640 1 --2- 172.21.15.57:0/3270922718 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff08c06ba00 0x7ff08c0c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:28.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.764+0000 7ff09286b640 1 -- 172.21.15.57:0/3270922718 >> 172.21.15.57:0/3270922718 conn(0x7ff08c071fb0 msgr2=0x7ff08c152720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:28.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.764+0000 7ff09286b640 1 -- 172.21.15.57:0/3270922718 shutdown_connections 2023-10-30T14:46:28.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:28.764+0000 7ff09286b640 1 -- 172.21.15.57:0/3270922718 wait complete. 2023-10-30T14:46:28.788 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:46:30.789 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:30.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.940+0000 7f65c790f640 1 Processor -- start 2023-10-30T14:46:30.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.940+0000 7f65c790f640 1 -- start start 2023-10-30T14:46:30.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.940+0000 7f65c790f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f65c0153c80 0x7f65c015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:30.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.940+0000 7f65c790f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65c00c4a10 0x7f65c00c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:30.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.940+0000 7f65c790f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c0159f20 con 0x7f65c015a3b0 2023-10-30T14:46:30.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.940+0000 7f65c790f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65c015dec0 con 0x7f65c0153c80 2023-10-30T14:46:30.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.940+0000 7f65c790f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c01599a0 con 0x7f65c00c4a10 2023-10-30T14:46:30.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.940+0000 7f65c4eaa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65c00c4a10 0x7f65c00c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:30.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.940+0000 7f65c4eaa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65c00c4a10 0x7f65c00c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:41206/0 (socket says 172.21.15.57:41206) 2023-10-30T14:46:30.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.940+0000 7f65c56ab640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f65c0153c80 0x7f65c015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:30.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.940+0000 7f65c4eaa640 1 -- 172.21.15.57:0/1688703589 learned_addr learned my addr 172.21.15.57:0/1688703589 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:30.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.940+0000 7f65c4eaa640 1 -- 172.21.15.57:0/1688703589 >> v2:172.21.15.88:3300/0 conn(0x7f65c0153c80 msgr2=0x7f65c015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:30.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.940+0000 7f65c4eaa640 1 --2- 172.21.15.57:0/1688703589 >> v2:172.21.15.88:3300/0 conn(0x7f65c0153c80 0x7f65c015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:30.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.940+0000 7f65c4eaa640 1 -- 172.21.15.57:0/1688703589 >> v1:172.21.15.88:6789/0 conn(0x7f65c015a3b0 legacy=0x7f65c00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:30.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.940+0000 7f65c4eaa640 1 -- 172.21.15.57:0/1688703589 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c00c47e0 con 0x7f65c00c4a10 2023-10-30T14:46:30.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.944+0000 7f65c4eaa640 1 --2- 172.21.15.57:0/1688703589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65c00c4a10 0x7f65c00c42a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f65b0009870 tx=0x7f65b001ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30363a42af7d9379 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:30.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.944+0000 7f65b7fff640 1 -- 172.21.15.57:0/1688703589 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b00a8a30 con 0x7f65c00c4a10 2023-10-30T14:46:30.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.944+0000 7f65b7fff640 1 -- 172.21.15.57:0/1688703589 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b00a8b90 con 0x7f65c00c4a10 2023-10-30T14:46:30.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.944+0000 7f65b7fff640 1 -- 172.21.15.57:0/1688703589 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b00b0440 con 0x7f65c00c4a10 2023-10-30T14:46:30.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.944+0000 7f65c790f640 1 -- 172.21.15.57:0/1688703589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65c00c4a10 msgr2=0x7f65c00c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:30.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.944+0000 7f65c790f640 1 --2- 172.21.15.57:0/1688703589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65c00c4a10 0x7f65c00c42a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f65b0009870 tx=0x7f65b001ed30 comp rx=0 tx=0).stop 2023-10-30T14:46:30.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.944+0000 7f65c790f640 1 -- 172.21.15.57:0/1688703589 shutdown_connections 2023-10-30T14:46:30.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.944+0000 7f65c790f640 1 --2- 172.21.15.57:0/1688703589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65c00c4a10 0x7f65c00c42a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:30.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.944+0000 7f65c790f640 1 --2- 172.21.15.57:0/1688703589 >> v2:172.21.15.88:3300/0 conn(0x7f65c0153c80 0x7f65c015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:30.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.944+0000 7f65c790f640 1 -- 172.21.15.57:0/1688703589 >> 172.21.15.57:0/1688703589 conn(0x7f65c00715c0 msgr2=0x7f65c00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:30.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c790f640 1 -- 172.21.15.57:0/1688703589 shutdown_connections 2023-10-30T14:46:30.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c790f640 1 -- 172.21.15.57:0/1688703589 wait complete. 2023-10-30T14:46:30.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c790f640 1 Processor -- start 2023-10-30T14:46:30.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c790f640 1 -- start start 2023-10-30T14:46:30.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c790f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65c00c4a10 0x7f65c014bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:30.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c790f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c0153c80 0x7f65c014a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:30.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c790f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65c015a3b0 0x7f65c0158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:30.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c790f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c014c1c0 con 0x7f65c0153c80 2023-10-30T14:46:30.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c790f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c01f30f0 con 0x7f65c00c4a10 2023-10-30T14:46:30.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c790f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c01f2290 con 0x7f65c015a3b0 2023-10-30T14:46:30.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c4eaa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c0153c80 0x7f65c014a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:30.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c4eaa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c0153c80 0x7f65c014a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49842/0 (socket says 172.21.15.57:49842) 2023-10-30T14:46:30.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c4eaa640 1 -- 172.21.15.57:0/314811291 learned_addr learned my addr 172.21.15.57:0/314811291 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:30.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c56ab640 1 --2- 172.21.15.57:0/314811291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65c00c4a10 0x7f65c014bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:30.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c5eac640 1 --2- 172.21.15.57:0/314811291 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65c015a3b0 0x7f65c0158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:30.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c4eaa640 1 -- 172.21.15.57:0/314811291 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65c015a3b0 msgr2=0x7f65c0158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:30.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c4eaa640 1 --2- 172.21.15.57:0/314811291 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65c015a3b0 0x7f65c0158850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:30.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c4eaa640 1 -- 172.21.15.57:0/314811291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65c00c4a10 msgr2=0x7f65c014bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:30.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c4eaa640 1 --2- 172.21.15.57:0/314811291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65c00c4a10 0x7f65c014bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:30.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c4eaa640 1 -- 172.21.15.57:0/314811291 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b00094d0 con 0x7f65c0153c80 2023-10-30T14:46:30.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.948+0000 7f65c4eaa640 1 --2- 172.21.15.57:0/314811291 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c0153c80 0x7f65c014a9b0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f65b00b3040 tx=0x7f65b001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:30.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.952+0000 7f65b5ffb640 1 -- 172.21.15.57:0/314811291 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b00b0dd0 con 0x7f65c0153c80 2023-10-30T14:46:30.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.952+0000 7f65b5ffb640 1 -- 172.21.15.57:0/314811291 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b00bb460 con 0x7f65c0153c80 2023-10-30T14:46:30.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.952+0000 7f65b5ffb640 1 -- 172.21.15.57:0/314811291 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b00bb810 con 0x7f65c0153c80 2023-10-30T14:46:30.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.952+0000 7f65c790f640 1 -- 172.21.15.57:0/314811291 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c01f2570 con 0x7f65c0153c80 2023-10-30T14:46:30.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.952+0000 7f65c790f640 1 -- 172.21.15.57:0/314811291 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c01f2b30 con 0x7f65c0153c80 2023-10-30T14:46:30.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.952+0000 7f65c790f640 1 -- 172.21.15.57:0/314811291 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65c004e170 con 0x7f65c0153c80 2023-10-30T14:46:30.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.952+0000 7f65b5ffb640 1 -- 172.21.15.57:0/314811291 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f65b00b83f0 con 0x7f65c0153c80 2023-10-30T14:46:30.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.952+0000 7f65b5ffb640 1 --2- 172.21.15.57:0/314811291 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65a40664a0 0x7f65a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:30.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.952+0000 7f65b5ffb640 1 -- 172.21.15.57:0/314811291 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f65b0121cd0 con 0x7f65c0153c80 2023-10-30T14:46:30.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.952+0000 7f65c56ab640 1 --2- 172.21.15.57:0/314811291 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65a40664a0 0x7f65a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:30.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.956+0000 7f65c56ab640 1 --2- 172.21.15.57:0/314811291 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65a40664a0 0x7f65a4068960 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f65bc009b60 tx=0x7f65bc009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:30.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:30.960+0000 7f65b5ffb640 1 -- 172.21.15.57:0/314811291 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f65b00eec10 con 0x7f65c0153c80 2023-10-30T14:46:31.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:31.148+0000 7f65c790f640 1 -- 172.21.15.57:0/314811291 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65c014c630 con 0x7f65c0153c80 2023-10-30T14:46:31.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:31.148+0000 7f65b5ffb640 1 -- 172.21.15.57:0/314811291 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6513 (secure 0 0 0) 0x7f65b00c0090 con 0x7f65c0153c80 2023-10-30T14:46:31.152 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:31.152 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:45:57.259750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:stopping","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:31.153 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 174 2023-10-30T14:46:31.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:31.148+0000 7f65c790f640 1 -- 172.21.15.57:0/314811291 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65a40664a0 msgr2=0x7f65a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:31.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:31.148+0000 7f65c790f640 1 --2- 172.21.15.57:0/314811291 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65a40664a0 0x7f65a4068960 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f65bc009b60 tx=0x7f65bc009190 comp rx=0 tx=0).stop 2023-10-30T14:46:31.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:31.148+0000 7f65c790f640 1 -- 172.21.15.57:0/314811291 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c0153c80 msgr2=0x7f65c014a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:31.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:31.148+0000 7f65c790f640 1 --2- 172.21.15.57:0/314811291 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c0153c80 0x7f65c014a9b0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f65b00b3040 tx=0x7f65b001bad0 comp rx=0 tx=0).stop 2023-10-30T14:46:31.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:31.148+0000 7f65c790f640 1 -- 172.21.15.57:0/314811291 shutdown_connections 2023-10-30T14:46:31.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:31.148+0000 7f65c790f640 1 --2- 172.21.15.57:0/314811291 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f65a40664a0 0x7f65a4068960 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:31.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:31.148+0000 7f65c790f640 1 --2- 172.21.15.57:0/314811291 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65c015a3b0 0x7f65c0158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:31.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:31.148+0000 7f65c790f640 1 --2- 172.21.15.57:0/314811291 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65c0153c80 0x7f65c014a9b0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:31.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:31.148+0000 7f65c790f640 1 --2- 172.21.15.57:0/314811291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65c00c4a10 0x7f65c014bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:31.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:31.148+0000 7f65c790f640 1 -- 172.21.15.57:0/314811291 >> 172.21.15.57:0/314811291 conn(0x7f65c00715c0 msgr2=0x7f65c00719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:31.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:31.148+0000 7f65c790f640 1 -- 172.21.15.57:0/314811291 shutdown_connections 2023-10-30T14:46:31.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:31.148+0000 7f65c790f640 1 -- 172.21.15.57:0/314811291 wait complete. 2023-10-30T14:46:31.173 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:46:33.174 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:33.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.328+0000 7f2fb113e640 1 Processor -- start 2023-10-30T14:46:33.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.328+0000 7f2fb113e640 1 -- start start 2023-10-30T14:46:33.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.328+0000 7f2fb113e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2fac0c9c40 0x7f2fac06f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:33.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.328+0000 7f2fb113e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2fac06a270 0x7f2fac0c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:33.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.328+0000 7f2fb113e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fac0c8440 con 0x7f2fac069e90 2023-10-30T14:46:33.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.328+0000 7f2fb113e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fac153f60 con 0x7f2fac06a270 2023-10-30T14:46:33.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.328+0000 7f2fb113e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fac0c89f0 con 0x7f2fac0c9c40 2023-10-30T14:46:33.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.328+0000 7f2fab59e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2fac06a270 0x7f2fac0c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:33.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.328+0000 7f2fab59e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2fac06a270 0x7f2fac0c8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49848/0 (socket says 172.21.15.57:49848) 2023-10-30T14:46:33.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.328+0000 7f2faad9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f2fac069e90 0x7f2fac1597d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:53054/0 (socket says 172.21.15.57:53054) 2023-10-30T14:46:33.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.328+0000 7f2fab59e640 1 -- 172.21.15.57:0/4245806062 learned_addr learned my addr 172.21.15.57:0/4245806062 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:33.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.328+0000 7f2fa3fff640 1 --2- 172.21.15.57:0/4245806062 >> v2:172.21.15.88:3300/0 conn(0x7f2fac0c9c40 0x7f2fac06f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:33.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.328+0000 7f2fab59e640 1 -- 172.21.15.57:0/4245806062 >> v2:172.21.15.88:3300/0 conn(0x7f2fac0c9c40 msgr2=0x7f2fac06f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:33.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.328+0000 7f2fab59e640 1 --2- 172.21.15.57:0/4245806062 >> v2:172.21.15.88:3300/0 conn(0x7f2fac0c9c40 0x7f2fac06f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:33.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.328+0000 7f2fab59e640 1 -- 172.21.15.57:0/4245806062 >> v1:172.21.15.57:6789/0 conn(0x7f2fac069e90 legacy=0x7f2fac1597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:33.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.328+0000 7f2fab59e640 1 -- 172.21.15.57:0/4245806062 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac0ca290 con 0x7f2fac06a270 2023-10-30T14:46:33.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.328+0000 7f2faa59c640 1 -- 172.21.15.57:0/4245806062 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828788132 0 0) 0x7f2f9c001120 con 0x7f2fac069e90 2023-10-30T14:46:33.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.332+0000 7f2fab59e640 1 --2- 172.21.15.57:0/4245806062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fac06a270 0x7f2fac0c8e80 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f2fac06e9b0 tx=0x7f2fa401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dba020eaed2536b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:33.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.332+0000 7f2faa59c640 1 -- 172.21.15.57:0/4245806062 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa40a5780 con 0x7f2fac06a270 2023-10-30T14:46:33.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.332+0000 7f2faa59c640 1 -- 172.21.15.57:0/4245806062 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa40a58e0 con 0x7f2fac06a270 2023-10-30T14:46:33.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.332+0000 7f2faa59c640 1 -- 172.21.15.57:0/4245806062 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa40a5c70 con 0x7f2fac06a270 2023-10-30T14:46:33.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.332+0000 7f2fb113e640 1 -- 172.21.15.57:0/4245806062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fac06a270 msgr2=0x7f2fac0c8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:33.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.332+0000 7f2fb113e640 1 --2- 172.21.15.57:0/4245806062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fac06a270 0x7f2fac0c8e80 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f2fac06e9b0 tx=0x7f2fa401c9d0 comp rx=0 tx=0).stop 2023-10-30T14:46:33.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.332+0000 7f2fb113e640 1 -- 172.21.15.57:0/4245806062 shutdown_connections 2023-10-30T14:46:33.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.332+0000 7f2fb113e640 1 --2- 172.21.15.57:0/4245806062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fac06a270 0x7f2fac0c8e80 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:33.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.332+0000 7f2fb113e640 1 --2- 172.21.15.57:0/4245806062 >> v2:172.21.15.88:3300/0 conn(0x7f2fac0c9c40 0x7f2fac06f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:33.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.332+0000 7f2fb113e640 1 -- 172.21.15.57:0/4245806062 >> 172.21.15.57:0/4245806062 conn(0x7f2fac06afa0 msgr2=0x7f2fac06b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:33.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.332+0000 7f2fb113e640 1 -- 172.21.15.57:0/4245806062 shutdown_connections 2023-10-30T14:46:33.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.332+0000 7f2fb113e640 1 -- 172.21.15.57:0/4245806062 wait complete. 2023-10-30T14:46:33.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fb113e640 1 Processor -- start 2023-10-30T14:46:33.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fb113e640 1 -- start start 2023-10-30T14:46:33.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fb113e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fac069e90 0x7f2fac152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:33.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fb113e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fac06a270 0x7f2fac150050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:33.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fb113e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fac0c9c40 0x7f2fac0c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:33.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fb113e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac1f61b0 con 0x7f2fac0c9c40 2023-10-30T14:46:33.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fb113e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac153a90 con 0x7f2fac06a270 2023-10-30T14:46:33.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fb113e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac0c07b0 con 0x7f2fac069e90 2023-10-30T14:46:33.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fab59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fac0c9c40 0x7f2fac0c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:33.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fab59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fac0c9c40 0x7f2fac0c0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49854/0 (socket says 172.21.15.57:49854) 2023-10-30T14:46:33.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fab59e640 1 -- 172.21.15.57:0/2551896176 learned_addr learned my addr 172.21.15.57:0/2551896176 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:33.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fa3fff640 1 --2- 172.21.15.57:0/2551896176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fac06a270 0x7f2fac150050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:33.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2faad9d640 1 --2- 172.21.15.57:0/2551896176 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fac069e90 0x7f2fac152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:33.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fab59e640 1 -- 172.21.15.57:0/2551896176 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fac069e90 msgr2=0x7f2fac152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:33.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fab59e640 1 --2- 172.21.15.57:0/2551896176 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fac069e90 0x7f2fac152850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:33.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fab59e640 1 -- 172.21.15.57:0/2551896176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fac06a270 msgr2=0x7f2fac150050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:33.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fab59e640 1 --2- 172.21.15.57:0/2551896176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fac06a270 0x7f2fac150050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:33.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fab59e640 1 -- 172.21.15.57:0/2551896176 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa4009b30 con 0x7f2fac0c9c40 2023-10-30T14:46:33.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2faad9d640 1 --2- 172.21.15.57:0/2551896176 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fac069e90 0x7f2fac152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:46:33.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fab59e640 1 --2- 172.21.15.57:0/2551896176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fac0c9c40 0x7f2fac0c0ab0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f2fa401a600 tx=0x7f2fa4019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:33.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fa37fe640 1 -- 172.21.15.57:0/2551896176 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa40b8670 con 0x7f2fac0c9c40 2023-10-30T14:46:33.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fa37fe640 1 -- 172.21.15.57:0/2551896176 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa40b87d0 con 0x7f2fac0c9c40 2023-10-30T14:46:33.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fa37fe640 1 -- 172.21.15.57:0/2551896176 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa40b8b80 con 0x7f2fac0c9c40 2023-10-30T14:46:33.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fb113e640 1 -- 172.21.15.57:0/2551896176 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fac1f5430 con 0x7f2fac0c9c40 2023-10-30T14:46:33.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.336+0000 7f2fb113e640 1 -- 172.21.15.57:0/2551896176 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fac1f56d0 con 0x7f2fac0c9c40 2023-10-30T14:46:33.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.340+0000 7f2fb113e640 1 -- 172.21.15.57:0/2551896176 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fac04e170 con 0x7f2fac0c9c40 2023-10-30T14:46:33.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.340+0000 7f2fa37fe640 1 -- 172.21.15.57:0/2551896176 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2fa40b8ce0 con 0x7f2fac0c9c40 2023-10-30T14:46:33.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.340+0000 7f2fa37fe640 1 --2- 172.21.15.57:0/2551896176 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f8c0664a0 0x7f2f8c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:33.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.340+0000 7f2fa37fe640 1 -- 172.21.15.57:0/2551896176 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f2fa411f830 con 0x7f2fac0c9c40 2023-10-30T14:46:33.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.340+0000 7f2faad9d640 1 --2- 172.21.15.57:0/2551896176 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f8c0664a0 0x7f2f8c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:33.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.340+0000 7f2faad9d640 1 --2- 172.21.15.57:0/2551896176 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f8c0664a0 0x7f2f8c068960 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c000f70 tx=0x7f2f9c007cb0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:33.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.344+0000 7f2fa37fe640 1 -- 172.21.15.57:0/2551896176 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2fa40ec7e0 con 0x7f2fac0c9c40 2023-10-30T14:46:33.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.532+0000 7f2fb113e640 1 -- 172.21.15.57:0/2551896176 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fac0c7670 con 0x7f2fac0c9c40 2023-10-30T14:46:33.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.532+0000 7f2fa37fe640 1 -- 172.21.15.57:0/2551896176 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6513 (secure 0 0 0) 0x7f2fa40efe00 con 0x7f2fac0c9c40 2023-10-30T14:46:33.543 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:33.543 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:45:57.259750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:stopping","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:33.544 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 174 2023-10-30T14:46:33.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.536+0000 7f2fb113e640 1 -- 172.21.15.57:0/2551896176 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f8c0664a0 msgr2=0x7f2f8c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:33.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.536+0000 7f2fb113e640 1 --2- 172.21.15.57:0/2551896176 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f8c0664a0 0x7f2f8c068960 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c000f70 tx=0x7f2f9c007cb0 comp rx=0 tx=0).stop 2023-10-30T14:46:33.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.536+0000 7f2fb113e640 1 -- 172.21.15.57:0/2551896176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fac0c9c40 msgr2=0x7f2fac0c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:33.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.536+0000 7f2fb113e640 1 --2- 172.21.15.57:0/2551896176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fac0c9c40 0x7f2fac0c0ab0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f2fa401a600 tx=0x7f2fa4019770 comp rx=0 tx=0).stop 2023-10-30T14:46:33.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.536+0000 7f2fb113e640 1 -- 172.21.15.57:0/2551896176 shutdown_connections 2023-10-30T14:46:33.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.536+0000 7f2fb113e640 1 --2- 172.21.15.57:0/2551896176 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2f8c0664a0 0x7f2f8c068960 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:33.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.536+0000 7f2fb113e640 1 --2- 172.21.15.57:0/2551896176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fac0c9c40 0x7f2fac0c0ab0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:33.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.536+0000 7f2fb113e640 1 --2- 172.21.15.57:0/2551896176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fac06a270 0x7f2fac150050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:33.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.536+0000 7f2fb113e640 1 --2- 172.21.15.57:0/2551896176 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fac069e90 0x7f2fac152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:33.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.536+0000 7f2fb113e640 1 -- 172.21.15.57:0/2551896176 >> 172.21.15.57:0/2551896176 conn(0x7f2fac06afa0 msgr2=0x7f2fac0ca420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:33.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.536+0000 7f2fb113e640 1 -- 172.21.15.57:0/2551896176 shutdown_connections 2023-10-30T14:46:33.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:33.536+0000 7f2fb113e640 1 -- 172.21.15.57:0/2551896176 wait complete. 2023-10-30T14:46:33.558 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T14:46:33.559 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 174, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 174, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6747": { "addr": "172.21.15.57:6836/742151512", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 742151512, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 742151512, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6747, "incarnation": 120, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:stopping", "state_seq": 44 }, "gid_7536": { "addr": "172.21.15.57:6837/816502245", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 816502245, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 816502245, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 7536, "incarnation": 164, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:45:57.259750+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7536, "mds_1": 6747 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6839/1078372885", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1078372885, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1078372885, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 170, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6716, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.57:6834/697507447", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 697507447, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 697507447, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 156, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7377, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:46:35.560 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:35.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.712+0000 7fe0407c4640 1 Processor -- start 2023-10-30T14:46:35.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.712+0000 7fe0407c4640 1 -- start start 2023-10-30T14:46:35.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.712+0000 7fe0407c4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe0380c4a10 0x7fe03815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:35.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.712+0000 7fe0407c4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe0380c42a0 0x7fe0380ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:35.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.712+0000 7fe0407c4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe038159f20 con 0x7fe038153c80 2023-10-30T14:46:35.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.712+0000 7fe0407c4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe03815dec0 con 0x7fe0380c4a10 2023-10-30T14:46:35.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.712+0000 7fe0407c4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0381599a0 con 0x7fe0380c42a0 2023-10-30T14:46:35.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.712+0000 7fe03dd5f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe0380c4a10 0x7fe03815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:35.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.712+0000 7fe03dd5f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe0380c4a10 0x7fe03815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35784/0 (socket says 172.21.15.57:35784) 2023-10-30T14:46:35.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.712+0000 7fe03dd5f640 1 -- 172.21.15.57:0/855733154 learned_addr learned my addr 172.21.15.57:0/855733154 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:35.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.712+0000 7fe03ed61640 1 --2- 172.21.15.57:0/855733154 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe0380c42a0 0x7fe0380ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:35.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.712+0000 7fe03dd5f640 1 -- 172.21.15.57:0/855733154 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe0380c42a0 msgr2=0x7fe0380ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:35.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.712+0000 7fe03dd5f640 1 --2- 172.21.15.57:0/855733154 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe0380c42a0 0x7fe0380ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:35.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.712+0000 7fe03dd5f640 1 -- 172.21.15.57:0/855733154 >> v1:172.21.15.88:6789/0 conn(0x7fe038153c80 legacy=0x7fe03815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:35.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.712+0000 7fe03dd5f640 1 -- 172.21.15.57:0/855733154 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03815d620 con 0x7fe0380c4a10 2023-10-30T14:46:35.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.716+0000 7fe03dd5f640 1 --2- 172.21.15.57:0/855733154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0380c4a10 0x7fe03815cf90 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fe028009870 tx=0x7fe02801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5397656922a967b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:35.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.716+0000 7fe03d55e640 1 -- 172.21.15.57:0/855733154 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0280a8a30 con 0x7fe0380c4a10 2023-10-30T14:46:35.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.716+0000 7fe03d55e640 1 -- 172.21.15.57:0/855733154 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0280a8b90 con 0x7fe0380c4a10 2023-10-30T14:46:35.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.716+0000 7fe03d55e640 1 -- 172.21.15.57:0/855733154 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0280b0440 con 0x7fe0380c4a10 2023-10-30T14:46:35.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.716+0000 7fe0407c4640 1 -- 172.21.15.57:0/855733154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0380c4a10 msgr2=0x7fe03815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:35.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.716+0000 7fe0407c4640 1 --2- 172.21.15.57:0/855733154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0380c4a10 0x7fe03815cf90 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fe028009870 tx=0x7fe02801ed30 comp rx=0 tx=0).stop 2023-10-30T14:46:35.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.716+0000 7fe0407c4640 1 -- 172.21.15.57:0/855733154 shutdown_connections 2023-10-30T14:46:35.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.716+0000 7fe0407c4640 1 --2- 172.21.15.57:0/855733154 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe0380c42a0 0x7fe0380ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:35.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.716+0000 7fe0407c4640 1 --2- 172.21.15.57:0/855733154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0380c4a10 0x7fe03815cf90 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:35.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.716+0000 7fe0407c4640 1 -- 172.21.15.57:0/855733154 >> 172.21.15.57:0/855733154 conn(0x7fe0380715c0 msgr2=0x7fe0380719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:35.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.716+0000 7fe0407c4640 1 -- 172.21.15.57:0/855733154 shutdown_connections 2023-10-30T14:46:35.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.716+0000 7fe0407c4640 1 -- 172.21.15.57:0/855733154 wait complete. 2023-10-30T14:46:35.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe0407c4640 1 Processor -- start 2023-10-30T14:46:35.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe0407c4640 1 -- start start 2023-10-30T14:46:35.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe0407c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0380c42a0 0x7fe03814b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:35.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe0407c4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0380c4a10 0x7fe038149ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:35.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe0407c4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe038153c80 0x7fe03815c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:35.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe0407c4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0381550d0 con 0x7fe0380c4a10 2023-10-30T14:46:35.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe0407c4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03814ba50 con 0x7fe0380c42a0 2023-10-30T14:46:35.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe0407c4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe038152670 con 0x7fe038153c80 2023-10-30T14:46:35.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe03dd5f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0380c4a10 0x7fe038149ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:35.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe03dd5f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0380c4a10 0x7fe038149ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35796/0 (socket says 172.21.15.57:35796) 2023-10-30T14:46:35.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe03dd5f640 1 -- 172.21.15.57:0/1638777804 learned_addr learned my addr 172.21.15.57:0/1638777804 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:35.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe03ed61640 1 --2- 172.21.15.57:0/1638777804 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe038153c80 0x7fe03815c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:35.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe03e560640 1 --2- 172.21.15.57:0/1638777804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0380c42a0 0x7fe03814b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:35.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe03dd5f640 1 -- 172.21.15.57:0/1638777804 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe038153c80 msgr2=0x7fe03815c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:35.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe03dd5f640 1 --2- 172.21.15.57:0/1638777804 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe038153c80 0x7fe03815c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:35.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe03dd5f640 1 -- 172.21.15.57:0/1638777804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0380c42a0 msgr2=0x7fe03814b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:35.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe03dd5f640 1 --2- 172.21.15.57:0/1638777804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0380c42a0 0x7fe03814b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:35.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe03dd5f640 1 -- 172.21.15.57:0/1638777804 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0280094d0 con 0x7fe0380c4a10 2023-10-30T14:46:35.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe03ed61640 1 --2- 172.21.15.57:0/1638777804 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe038153c80 0x7fe03815c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:46:35.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe03dd5f640 1 --2- 172.21.15.57:0/1638777804 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0380c4a10 0x7fe038149ea0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fe0280b3040 tx=0x7fe02801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:35.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe02effd640 1 -- 172.21.15.57:0/1638777804 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0280be070 con 0x7fe0380c4a10 2023-10-30T14:46:35.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe02effd640 1 -- 172.21.15.57:0/1638777804 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0280ba400 con 0x7fe0380c4a10 2023-10-30T14:46:35.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe02effd640 1 -- 172.21.15.57:0/1638777804 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0280ba7d0 con 0x7fe0380c4a10 2023-10-30T14:46:35.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe0407c4640 1 -- 172.21.15.57:0/1638777804 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0381520b0 con 0x7fe0380c4a10 2023-10-30T14:46:35.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.720+0000 7fe0407c4640 1 -- 172.21.15.57:0/1638777804 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe03814aa30 con 0x7fe0380c4a10 2023-10-30T14:46:35.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.724+0000 7fe0407c4640 1 -- 172.21.15.57:0/1638777804 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03804e170 con 0x7fe0380c4a10 2023-10-30T14:46:35.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.724+0000 7fe02effd640 1 -- 172.21.15.57:0/1638777804 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe0280c3410 con 0x7fe0380c4a10 2023-10-30T14:46:35.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.724+0000 7fe02effd640 1 --2- 172.21.15.57:0/1638777804 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe01c0664a0 0x7fe01c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:35.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.724+0000 7fe02effd640 1 -- 172.21.15.57:0/1638777804 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7fe028122100 con 0x7fe0380c4a10 2023-10-30T14:46:35.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.724+0000 7fe03e560640 1 --2- 172.21.15.57:0/1638777804 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe01c0664a0 0x7fe01c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:35.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.724+0000 7fe03e560640 1 --2- 172.21.15.57:0/1638777804 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe01c0664a0 0x7fe01c068960 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fe034000f70 tx=0x7fe034009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:35.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.728+0000 7fe02effd640 1 -- 172.21.15.57:0/1638777804 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe0280eef00 con 0x7fe0380c4a10 2023-10-30T14:46:35.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.920+0000 7fe0407c4640 1 -- 172.21.15.57:0/1638777804 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0380cad40 con 0x7fe0380c4a10 2023-10-30T14:46:35.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.920+0000 7fe02effd640 1 -- 172.21.15.57:0/1638777804 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6513 (secure 0 0 0) 0x7fe0280f2520 con 0x7fe0380c4a10 2023-10-30T14:46:35.925 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:35.925 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:45:57.259750+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7536,"mds_1":6747},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6747":{"gid":6747,"name":"c","rank":1,"incarnation":120,"state":"up:stopping","state_seq":44,"addr":"172.21.15.57:6836/742151512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":742151512},{"type":"v1","addr":"172.21.15.57:6836","nonce":742151512}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:35.925 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 174 2023-10-30T14:46:35.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.924+0000 7fe0407c4640 1 -- 172.21.15.57:0/1638777804 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe01c0664a0 msgr2=0x7fe01c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:35.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.924+0000 7fe0407c4640 1 --2- 172.21.15.57:0/1638777804 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe01c0664a0 0x7fe01c068960 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fe034000f70 tx=0x7fe034009040 comp rx=0 tx=0).stop 2023-10-30T14:46:35.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.924+0000 7fe0407c4640 1 -- 172.21.15.57:0/1638777804 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0380c4a10 msgr2=0x7fe038149ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:35.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.924+0000 7fe0407c4640 1 --2- 172.21.15.57:0/1638777804 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0380c4a10 0x7fe038149ea0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fe0280b3040 tx=0x7fe02801bad0 comp rx=0 tx=0).stop 2023-10-30T14:46:35.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.924+0000 7fe0407c4640 1 -- 172.21.15.57:0/1638777804 shutdown_connections 2023-10-30T14:46:35.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.924+0000 7fe0407c4640 1 --2- 172.21.15.57:0/1638777804 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe01c0664a0 0x7fe01c068960 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:35.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.924+0000 7fe0407c4640 1 --2- 172.21.15.57:0/1638777804 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe038153c80 0x7fe03815c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:35.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.924+0000 7fe0407c4640 1 --2- 172.21.15.57:0/1638777804 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0380c4a10 0x7fe038149ea0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:35.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.924+0000 7fe0407c4640 1 --2- 172.21.15.57:0/1638777804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0380c42a0 0x7fe03814b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:35.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.924+0000 7fe0407c4640 1 -- 172.21.15.57:0/1638777804 >> 172.21.15.57:0/1638777804 conn(0x7fe0380715c0 msgr2=0x7fe0380c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:35.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.924+0000 7fe0407c4640 1 -- 172.21.15.57:0/1638777804 shutdown_connections 2023-10-30T14:46:35.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:35.924+0000 7fe0407c4640 1 -- 172.21.15.57:0/1638777804 wait complete. 2023-10-30T14:46:35.946 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:46:37.412 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-30T14:46:37.947 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:38.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.092+0000 7f491d33e640 1 Processor -- start 2023-10-30T14:46:38.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.092+0000 7f491d33e640 1 -- start start 2023-10-30T14:46:38.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.092+0000 7f491d33e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49180c4a10 0x7f491815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:38.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.092+0000 7f491d33e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f49180c42a0 0x7f49180ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:38.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.092+0000 7f491d33e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4918159f20 con 0x7f4918153c80 2023-10-30T14:46:38.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.092+0000 7f491d33e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f491815dec0 con 0x7f49180c42a0 2023-10-30T14:46:38.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.092+0000 7f491d33e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49181599a0 con 0x7f49180c4a10 2023-10-30T14:46:38.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.092+0000 7f49177fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f49180c42a0 0x7f49180ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:38.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.092+0000 7f49177fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f49180c42a0 0x7f49180ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35834/0 (socket says 172.21.15.57:35834) 2023-10-30T14:46:38.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.092+0000 7f49177fe640 1 -- 172.21.15.57:0/2518400968 learned_addr learned my addr 172.21.15.57:0/2518400968 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:38.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.092+0000 7f490ffff640 1 --2- 172.21.15.57:0/2518400968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49180c4a10 0x7f491815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:38.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.092+0000 7f49177fe640 1 -- 172.21.15.57:0/2518400968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49180c4a10 msgr2=0x7f491815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:38.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.092+0000 7f49177fe640 1 --2- 172.21.15.57:0/2518400968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49180c4a10 0x7f491815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:38.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.092+0000 7f49177fe640 1 -- 172.21.15.57:0/2518400968 >> v1:172.21.15.88:6789/0 conn(0x7f4918153c80 legacy=0x7f491815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:38.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.092+0000 7f49177fe640 1 -- 172.21.15.57:0/2518400968 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491815d620 con 0x7f49180c42a0 2023-10-30T14:46:38.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.096+0000 7f49177fe640 1 --2- 172.21.15.57:0/2518400968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49180c42a0 0x7f49180ca1e0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f4910009e50 tx=0x7f491001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5593d7e7e4397f71 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:38.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.096+0000 7f49167fc640 1 -- 172.21.15.57:0/2518400968 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49100af400 con 0x7f49180c42a0 2023-10-30T14:46:38.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.096+0000 7f49167fc640 1 -- 172.21.15.57:0/2518400968 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49100ab050 con 0x7f49180c42a0 2023-10-30T14:46:38.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.096+0000 7f49167fc640 1 -- 172.21.15.57:0/2518400968 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49100af780 con 0x7f49180c42a0 2023-10-30T14:46:38.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.096+0000 7f491d33e640 1 -- 172.21.15.57:0/2518400968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49180c42a0 msgr2=0x7f49180ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:38.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.096+0000 7f491d33e640 1 --2- 172.21.15.57:0/2518400968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49180c42a0 0x7f49180ca1e0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f4910009e50 tx=0x7f491001f1f0 comp rx=0 tx=0).stop 2023-10-30T14:46:38.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.096+0000 7f491d33e640 1 -- 172.21.15.57:0/2518400968 shutdown_connections 2023-10-30T14:46:38.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.096+0000 7f491d33e640 1 --2- 172.21.15.57:0/2518400968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49180c42a0 0x7f49180ca1e0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:38.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.096+0000 7f491d33e640 1 --2- 172.21.15.57:0/2518400968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49180c4a10 0x7f491815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:38.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.096+0000 7f491d33e640 1 -- 172.21.15.57:0/2518400968 >> 172.21.15.57:0/2518400968 conn(0x7f49180715c0 msgr2=0x7f49180719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:38.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.096+0000 7f491d33e640 1 -- 172.21.15.57:0/2518400968 shutdown_connections 2023-10-30T14:46:38.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.096+0000 7f491d33e640 1 -- 172.21.15.57:0/2518400968 wait complete. 2023-10-30T14:46:38.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f491d33e640 1 Processor -- start 2023-10-30T14:46:38.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f491d33e640 1 -- start start 2023-10-30T14:46:38.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f491d33e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49180c42a0 0x7f49181f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:38.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f491d33e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49180c4a10 0x7f49180b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:38.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f491d33e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4918153c80 0x7f491814f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:38.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f491d33e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49180b2610 con 0x7f49180c42a0 2023-10-30T14:46:38.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f491d33e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49180b3a60 con 0x7f4918153c80 2023-10-30T14:46:38.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f491d33e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49181f0630 con 0x7f49180c4a10 2023-10-30T14:46:38.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f4916ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49180c42a0 0x7f49181f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:38.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f4916ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49180c42a0 0x7f49181f00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35842/0 (socket says 172.21.15.57:35842) 2023-10-30T14:46:38.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f4916ffd640 1 -- 172.21.15.57:0/1186884811 learned_addr learned my addr 172.21.15.57:0/1186884811 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:38.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f490ffff640 1 --2- 172.21.15.57:0/1186884811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49180c4a10 0x7f49180b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:38.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f49177fe640 1 --2- 172.21.15.57:0/1186884811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4918153c80 0x7f491814f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:38.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f4916ffd640 1 -- 172.21.15.57:0/1186884811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49180c4a10 msgr2=0x7f49180b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:38.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f4916ffd640 1 --2- 172.21.15.57:0/1186884811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49180c4a10 0x7f49180b3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:38.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f4916ffd640 1 -- 172.21.15.57:0/1186884811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4918153c80 msgr2=0x7f491814f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:38.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f4916ffd640 1 --2- 172.21.15.57:0/1186884811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4918153c80 0x7f491814f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:38.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f4916ffd640 1 -- 172.21.15.57:0/1186884811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4910009b30 con 0x7f49180c42a0 2023-10-30T14:46:38.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f49177fe640 1 --2- 172.21.15.57:0/1186884811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4918153c80 0x7f491814f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:46:38.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f490ffff640 1 --2- 172.21.15.57:0/1186884811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49180c4a10 0x7f49180b3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:46:38.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f4916ffd640 1 --2- 172.21.15.57:0/1186884811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49180c42a0 0x7f49181f00f0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f490800c420 tx=0x7f490800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:38.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f490f7fe640 1 -- 172.21.15.57:0/1186884811 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49080052d0 con 0x7f49180c42a0 2023-10-30T14:46:38.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f490f7fe640 1 -- 172.21.15.57:0/1186884811 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4908005430 con 0x7f49180c42a0 2023-10-30T14:46:38.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f490f7fe640 1 -- 172.21.15.57:0/1186884811 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4908003840 con 0x7f49180c42a0 2023-10-30T14:46:38.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f491d33e640 1 -- 172.21.15.57:0/1186884811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49180b3cd0 con 0x7f49180c42a0 2023-10-30T14:46:38.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.100+0000 7f491d33e640 1 -- 172.21.15.57:0/1186884811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49180b4570 con 0x7f49180c42a0 2023-10-30T14:46:38.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.104+0000 7f491d33e640 1 -- 172.21.15.57:0/1186884811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49180b40d0 con 0x7f49180c42a0 2023-10-30T14:46:38.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.104+0000 7f490f7fe640 1 -- 172.21.15.57:0/1186884811 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f49080039a0 con 0x7f49180c42a0 2023-10-30T14:46:38.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.104+0000 7f490f7fe640 1 --2- 172.21.15.57:0/1186884811 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f48f80664a0 0x7f48f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:38.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.104+0000 7f490f7fe640 1 -- 172.21.15.57:0/1186884811 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f4908083960 con 0x7f49180c42a0 2023-10-30T14:46:38.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.104+0000 7f490ffff640 1 --2- 172.21.15.57:0/1186884811 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f48f80664a0 0x7f48f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:38.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.104+0000 7f490ffff640 1 --2- 172.21.15.57:0/1186884811 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f48f80664a0 0x7f48f8068960 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f49181f32b0 tx=0x7f4900008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:38.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.108+0000 7f490f7fe640 1 -- 172.21.15.57:0/1186884811 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f49080507f0 con 0x7f49180c42a0 2023-10-30T14:46:38.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.304+0000 7f491d33e640 1 -- 172.21.15.57:0/1186884811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f491804e170 con 0x7f49180c42a0 2023-10-30T14:46:38.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.304+0000 7f490f7fe640 1 -- 172.21.15.57:0/1186884811 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+5708 (secure 0 0 0) 0x7f4908053e10 con 0x7f49180c42a0 2023-10-30T14:46:38.310 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:38.310 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:46:37.266044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:38.310 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 175 2023-10-30T14:46:38.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.308+0000 7f491d33e640 1 -- 172.21.15.57:0/1186884811 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f48f80664a0 msgr2=0x7f48f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:38.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.308+0000 7f491d33e640 1 --2- 172.21.15.57:0/1186884811 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f48f80664a0 0x7f48f8068960 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f49181f32b0 tx=0x7f4900008040 comp rx=0 tx=0).stop 2023-10-30T14:46:38.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.308+0000 7f491d33e640 1 -- 172.21.15.57:0/1186884811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49180c42a0 msgr2=0x7f49181f00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:38.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.308+0000 7f491d33e640 1 --2- 172.21.15.57:0/1186884811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49180c42a0 0x7f49181f00f0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f490800c420 tx=0x7f490800c8e0 comp rx=0 tx=0).stop 2023-10-30T14:46:38.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.308+0000 7f491d33e640 1 -- 172.21.15.57:0/1186884811 shutdown_connections 2023-10-30T14:46:38.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.308+0000 7f491d33e640 1 --2- 172.21.15.57:0/1186884811 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f48f80664a0 0x7f48f8068960 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:38.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.308+0000 7f491d33e640 1 --2- 172.21.15.57:0/1186884811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4918153c80 0x7f491814f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:38.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.308+0000 7f491d33e640 1 --2- 172.21.15.57:0/1186884811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49180c4a10 0x7f49180b3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:38.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.308+0000 7f491d33e640 1 --2- 172.21.15.57:0/1186884811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49180c42a0 0x7f49181f00f0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:38.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.308+0000 7f491d33e640 1 -- 172.21.15.57:0/1186884811 >> 172.21.15.57:0/1186884811 conn(0x7f49180715c0 msgr2=0x7f49180719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:38.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.308+0000 7f491d33e640 1 -- 172.21.15.57:0/1186884811 shutdown_connections 2023-10-30T14:46:38.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.308+0000 7f491d33e640 1 -- 172.21.15.57:0/1186884811 wait complete. 2023-10-30T14:46:38.342 INFO:tasks.mds_thrash.fs.[cephfs]:mds cluster has 1 alive and active, now stable! 2023-10-30T14:46:38.342 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=1) 2023-10-30T14:46:38.343 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T14:46:38.343 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:46:38.356 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T14:46:38.357 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:38.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.500+0000 7f8063fc4640 1 Processor -- start 2023-10-30T14:46:38.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.500+0000 7f8063fc4640 1 -- start start 2023-10-30T14:46:38.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.500+0000 7f8063fc4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f805c0c4a10 0x7f805c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:38.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.500+0000 7f8063fc4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805c159f20 con 0x7f805c153c80 2023-10-30T14:46:38.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.500+0000 7f8063fc4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805c15d620 con 0x7f805c0c42a0 2023-10-30T14:46:38.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.500+0000 7f8063fc4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f805c15dec0 con 0x7f805c0c4a10 2023-10-30T14:46:38.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.500+0000 7f8061d60640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f805c153c80 0x7f805c15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:43742/0 (socket says 172.21.15.57:43742) 2023-10-30T14:46:38.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.500+0000 7f8061d60640 1 -- 172.21.15.57:0/192088180 learned_addr learned my addr 172.21.15.57:0/192088180 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:46:38.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.500+0000 7f806155f640 1 --2- 172.21.15.57:0/192088180 >> v2:172.21.15.57:3300/0 conn(0x7f805c0c4a10 0x7f805c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:38.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.500+0000 7f806155f640 1 -- 172.21.15.57:0/192088180 >> v1:172.21.15.88:6789/0 conn(0x7f805c0c42a0 legacy=0x7f805c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:38.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.500+0000 7f806155f640 1 -- 172.21.15.57:0/192088180 >> v1:172.21.15.57:6789/0 conn(0x7f805c153c80 legacy=0x7f805c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:38.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.500+0000 7f806155f640 1 -- 172.21.15.57:0/192088180 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805c15d900 con 0x7f805c0c4a10 2023-10-30T14:46:38.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.504+0000 7f806155f640 1 --2- 172.21.15.57:0/192088180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f805c0c4a10 0x7f805c15cf90 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f804c0097c0 tx=0x7f804c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6105f77019598224 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:38.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.504+0000 7f8060d5e640 1 -- 172.21.15.57:0/192088180 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c0a8970 con 0x7f805c0c4a10 2023-10-30T14:46:38.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.504+0000 7f8060d5e640 1 -- 172.21.15.57:0/192088180 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804c0a8ad0 con 0x7f805c0c4a10 2023-10-30T14:46:38.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.504+0000 7f8060d5e640 1 -- 172.21.15.57:0/192088180 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c0a8e60 con 0x7f805c0c4a10 2023-10-30T14:46:38.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.504+0000 7f8063fc4640 1 -- 172.21.15.57:0/192088180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f805c0c4a10 msgr2=0x7f805c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:38.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.504+0000 7f8063fc4640 1 --2- 172.21.15.57:0/192088180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f805c0c4a10 0x7f805c15cf90 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f804c0097c0 tx=0x7f804c01ec70 comp rx=0 tx=0).stop 2023-10-30T14:46:38.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.504+0000 7f8063fc4640 1 -- 172.21.15.57:0/192088180 shutdown_connections 2023-10-30T14:46:38.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.504+0000 7f8063fc4640 1 --2- 172.21.15.57:0/192088180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f805c0c4a10 0x7f805c15cf90 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:38.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.504+0000 7f8063fc4640 1 -- 172.21.15.57:0/192088180 >> 172.21.15.57:0/192088180 conn(0x7f805c0715c0 msgr2=0x7f805c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:38.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.504+0000 7f8063fc4640 1 -- 172.21.15.57:0/192088180 shutdown_connections 2023-10-30T14:46:38.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.504+0000 7f8063fc4640 1 -- 172.21.15.57:0/192088180 wait complete. 2023-10-30T14:46:38.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8063fc4640 1 Processor -- start 2023-10-30T14:46:38.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8063fc4640 1 -- start start 2023-10-30T14:46:38.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8063fc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f805c0c42a0 0x7f805c149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:38.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8063fc4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f805c0c4a10 0x7f805c14a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:38.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8063fc4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f805c153c80 0x7f805c0b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:38.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8063fc4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c15cc80 con 0x7f805c153c80 2023-10-30T14:46:38.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8063fc4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c0b3470 con 0x7f805c0c42a0 2023-10-30T14:46:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8063fc4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f805c0b3b60 con 0x7f805c0c4a10 2023-10-30T14:46:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8062561640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f805c153c80 0x7f805c0b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8062561640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f805c153c80 0x7f805c0b3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35860/0 (socket says 172.21.15.57:35860) 2023-10-30T14:46:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8062561640 1 -- 172.21.15.57:0/822813127 learned_addr learned my addr 172.21.15.57:0/822813127 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8061d60640 1 --2- 172.21.15.57:0/822813127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f805c0c42a0 0x7f805c149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f806155f640 1 --2- 172.21.15.57:0/822813127 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f805c0c4a10 0x7f805c14a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8062561640 1 -- 172.21.15.57:0/822813127 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f805c0c4a10 msgr2=0x7f805c14a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8062561640 1 --2- 172.21.15.57:0/822813127 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f805c0c4a10 0x7f805c14a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8062561640 1 -- 172.21.15.57:0/822813127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f805c0c42a0 msgr2=0x7f805c149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8062561640 1 --2- 172.21.15.57:0/822813127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f805c0c42a0 0x7f805c149b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:38.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8062561640 1 -- 172.21.15.57:0/822813127 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f804c009420 con 0x7f805c153c80 2023-10-30T14:46:38.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8062561640 1 --2- 172.21.15.57:0/822813127 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f805c153c80 0x7f805c0b3ff0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f805400c770 tx=0x7f805400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:38.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f80527fc640 1 -- 172.21.15.57:0/822813127 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8054003b80 con 0x7f805c153c80 2023-10-30T14:46:38.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f80527fc640 1 -- 172.21.15.57:0/822813127 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8054003ce0 con 0x7f805c153c80 2023-10-30T14:46:38.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f80527fc640 1 -- 172.21.15.57:0/822813127 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805400aa90 con 0x7f805c153c80 2023-10-30T14:46:38.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8063fc4640 1 -- 172.21.15.57:0/822813127 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f805c15d170 con 0x7f805c153c80 2023-10-30T14:46:38.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.508+0000 7f8063fc4640 1 -- 172.21.15.57:0/822813127 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f805c15ce90 con 0x7f805c153c80 2023-10-30T14:46:38.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.512+0000 7f8063fc4640 1 -- 172.21.15.57:0/822813127 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f805c04e170 con 0x7f805c153c80 2023-10-30T14:46:38.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.512+0000 7f80527fc640 1 -- 172.21.15.57:0/822813127 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f805400abf0 con 0x7f805c153c80 2023-10-30T14:46:38.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.512+0000 7f80527fc640 1 --2- 172.21.15.57:0/822813127 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80400664a0 0x7f8040068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:38.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.512+0000 7f80527fc640 1 -- 172.21.15.57:0/822813127 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f80540841f0 con 0x7f805c153c80 2023-10-30T14:46:38.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.512+0000 7f8061d60640 1 --2- 172.21.15.57:0/822813127 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80400664a0 0x7f8040068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:38.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.512+0000 7f8061d60640 1 --2- 172.21.15.57:0/822813127 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80400664a0 0x7f8040068960 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8058005890 tx=0x7f8058009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:38.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.516+0000 7f80527fc640 1 -- 172.21.15.57:0/822813127 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f80540510a0 con 0x7f805c153c80 2023-10-30T14:46:38.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.712+0000 7f8063fc4640 1 -- 172.21.15.57:0/822813127 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f805c0b3ca0 con 0x7f805c153c80 2023-10-30T14:46:38.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.712+0000 7f80527fc640 1 -- 172.21.15.57:0/822813127 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+6494 (secure 0 0 0) 0x7f80540546c0 con 0x7f805c153c80 2023-10-30T14:46:38.716 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:38.716 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7851,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/59168182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":59168182},{"type":"v1","addr":"172.21.15.57:6836","nonce":59168182}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:46:37.266044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:38.718 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 176 2023-10-30T14:46:38.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.712+0000 7f8063fc4640 1 -- 172.21.15.57:0/822813127 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80400664a0 msgr2=0x7f8040068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:38.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.712+0000 7f8063fc4640 1 --2- 172.21.15.57:0/822813127 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80400664a0 0x7f8040068960 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8058005890 tx=0x7f8058009040 comp rx=0 tx=0).stop 2023-10-30T14:46:38.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.712+0000 7f8063fc4640 1 -- 172.21.15.57:0/822813127 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f805c153c80 msgr2=0x7f805c0b3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:38.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.712+0000 7f8063fc4640 1 --2- 172.21.15.57:0/822813127 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f805c153c80 0x7f805c0b3ff0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f805400c770 tx=0x7f805400cc30 comp rx=0 tx=0).stop 2023-10-30T14:46:38.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.712+0000 7f8063fc4640 1 -- 172.21.15.57:0/822813127 shutdown_connections 2023-10-30T14:46:38.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.712+0000 7f8063fc4640 1 --2- 172.21.15.57:0/822813127 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f80400664a0 0x7f8040068960 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:38.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.712+0000 7f8063fc4640 1 --2- 172.21.15.57:0/822813127 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f805c153c80 0x7f805c0b3ff0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:38.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.712+0000 7f8063fc4640 1 --2- 172.21.15.57:0/822813127 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f805c0c4a10 0x7f805c14a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:38.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.712+0000 7f8063fc4640 1 --2- 172.21.15.57:0/822813127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f805c0c42a0 0x7f805c149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:38.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.712+0000 7f8063fc4640 1 -- 172.21.15.57:0/822813127 >> 172.21.15.57:0/822813127 conn(0x7f805c0715c0 msgr2=0x7f805c0c7720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:38.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.712+0000 7f8063fc4640 1 -- 172.21.15.57:0/822813127 shutdown_connections 2023-10-30T14:46:38.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:38.712+0000 7f8063fc4640 1 -- 172.21.15.57:0/822813127 wait complete. 2023-10-30T14:46:38.739 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:46:39.483 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:46:39.488 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:46:40.740 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:40.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.888+0000 7f4cd4e6b640 1 Processor -- start 2023-10-30T14:46:40.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.888+0000 7f4cd4e6b640 1 -- start start 2023-10-30T14:46:40.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.888+0000 7f4cd4e6b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4cd00c4a10 0x7f4cd015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:40.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.888+0000 7f4cd4e6b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cd00c42a0 0x7f4cd00ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:40.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.888+0000 7f4cd4e6b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd0159f20 con 0x7f4cd0153c80 2023-10-30T14:46:40.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.888+0000 7f4cd4e6b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd015dec0 con 0x7f4cd00c4a10 2023-10-30T14:46:40.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.888+0000 7f4cd4e6b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd01599a0 con 0x7f4cd00c42a0 2023-10-30T14:46:40.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.888+0000 7f4cced9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cd00c42a0 0x7f4cd00ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:40.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.892+0000 7f4cced9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cd00c42a0 0x7f4cd00ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:34106/0 (socket says 172.21.15.57:34106) 2023-10-30T14:46:40.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.892+0000 7f4cced9d640 1 -- 172.21.15.57:0/1303848823 learned_addr learned my addr 172.21.15.57:0/1303848823 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:40.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.892+0000 7f4cc7fff640 1 --2- 172.21.15.57:0/1303848823 >> v2:172.21.15.88:3300/0 conn(0x7f4cd00c4a10 0x7f4cd015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:40.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.892+0000 7f4cc7fff640 1 -- 172.21.15.57:0/1303848823 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cd00c42a0 msgr2=0x7f4cd00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:40.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.892+0000 7f4cc7fff640 1 --2- 172.21.15.57:0/1303848823 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cd00c42a0 0x7f4cd00ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:40.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.892+0000 7f4cc7fff640 1 -- 172.21.15.57:0/1303848823 >> v1:172.21.15.88:6789/0 conn(0x7f4cd0153c80 legacy=0x7f4cd015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:40.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.892+0000 7f4cc7fff640 1 -- 172.21.15.57:0/1303848823 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd015d620 con 0x7f4cd00c4a10 2023-10-30T14:46:40.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.892+0000 7f4cced9d640 1 --2- 172.21.15.57:0/1303848823 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cd00c42a0 0x7f4cd00ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:46:40.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.892+0000 7f4cc7fff640 1 --2- 172.21.15.57:0/1303848823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cd00c4a10 0x7f4cd015cf90 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f4cb80097c0 tx=0x7f4cb801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13248679b95fa28e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:40.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.896+0000 7f4ccdd9b640 1 -- 172.21.15.57:0/1303848823 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb80a8970 con 0x7f4cd00c4a10 2023-10-30T14:46:40.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.896+0000 7f4ccdd9b640 1 -- 172.21.15.57:0/1303848823 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb80a8ad0 con 0x7f4cd00c4a10 2023-10-30T14:46:40.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.896+0000 7f4ccdd9b640 1 -- 172.21.15.57:0/1303848823 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb80a8e60 con 0x7f4cd00c4a10 2023-10-30T14:46:40.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.896+0000 7f4cd4e6b640 1 -- 172.21.15.57:0/1303848823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cd00c4a10 msgr2=0x7f4cd015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:40.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.896+0000 7f4cd4e6b640 1 --2- 172.21.15.57:0/1303848823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cd00c4a10 0x7f4cd015cf90 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f4cb80097c0 tx=0x7f4cb801ec70 comp rx=0 tx=0).stop 2023-10-30T14:46:40.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.896+0000 7f4cd4e6b640 1 -- 172.21.15.57:0/1303848823 shutdown_connections 2023-10-30T14:46:40.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.896+0000 7f4cd4e6b640 1 --2- 172.21.15.57:0/1303848823 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cd00c42a0 0x7f4cd00ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:40.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.896+0000 7f4cd4e6b640 1 --2- 172.21.15.57:0/1303848823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cd00c4a10 0x7f4cd015cf90 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:40.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.896+0000 7f4cd4e6b640 1 -- 172.21.15.57:0/1303848823 >> 172.21.15.57:0/1303848823 conn(0x7f4cd00715c0 msgr2=0x7f4cd00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:40.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.896+0000 7f4cd4e6b640 1 -- 172.21.15.57:0/1303848823 shutdown_connections 2023-10-30T14:46:40.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.896+0000 7f4cd4e6b640 1 -- 172.21.15.57:0/1303848823 wait complete. 2023-10-30T14:46:40.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.896+0000 7f4cd4e6b640 1 Processor -- start 2023-10-30T14:46:40.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.896+0000 7f4cd4e6b640 1 -- start start 2023-10-30T14:46:40.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.896+0000 7f4cd4e6b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cd00c42a0 0x7f4cd014f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:40.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.896+0000 7f4cd4e6b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cd00c4a10 0x7f4cd014e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:40.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.896+0000 7f4cd4e6b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cd0153c80 0x7f4cd01f1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:40.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.896+0000 7f4cd4e6b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd0154c40 con 0x7f4cd0153c80 2023-10-30T14:46:40.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.896+0000 7f4cd4e6b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd014fc30 con 0x7f4cd00c4a10 2023-10-30T14:46:40.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.896+0000 7f4cd4e6b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd014ec10 con 0x7f4cd00c42a0 2023-10-30T14:46:40.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.900+0000 7f4cced9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cd0153c80 0x7f4cd01f1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:40.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.900+0000 7f4cced9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cd0153c80 0x7f4cd01f1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35868/0 (socket says 172.21.15.57:35868) 2023-10-30T14:46:40.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.900+0000 7f4cced9d640 1 -- 172.21.15.57:0/1042215068 learned_addr learned my addr 172.21.15.57:0/1042215068 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:40.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.900+0000 7f4cce59c640 1 --2- 172.21.15.57:0/1042215068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cd00c42a0 0x7f4cd014f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:40.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.900+0000 7f4cc7fff640 1 --2- 172.21.15.57:0/1042215068 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cd00c4a10 0x7f4cd014e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:40.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.900+0000 7f4cced9d640 1 -- 172.21.15.57:0/1042215068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cd00c42a0 msgr2=0x7f4cd014f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:40.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.900+0000 7f4cced9d640 1 --2- 172.21.15.57:0/1042215068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cd00c42a0 0x7f4cd014f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:40.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.900+0000 7f4cced9d640 1 -- 172.21.15.57:0/1042215068 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cd00c4a10 msgr2=0x7f4cd014e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:40.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.900+0000 7f4cced9d640 1 --2- 172.21.15.57:0/1042215068 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cd00c4a10 0x7f4cd014e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:40.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.900+0000 7f4cced9d640 1 -- 172.21.15.57:0/1042215068 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc8009b30 con 0x7f4cd0153c80 2023-10-30T14:46:40.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.900+0000 7f4cced9d640 1 --2- 172.21.15.57:0/1042215068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cd0153c80 0x7f4cd01f1aa0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f4cc800efc0 tx=0x7f4cc800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:40.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.900+0000 7f4cc6ffd640 1 -- 172.21.15.57:0/1042215068 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8018070 con 0x7f4cd0153c80 2023-10-30T14:46:40.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.900+0000 7f4cc6ffd640 1 -- 172.21.15.57:0/1042215068 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc800f040 con 0x7f4cd0153c80 2023-10-30T14:46:40.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.900+0000 7f4cc6ffd640 1 -- 172.21.15.57:0/1042215068 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc800ce00 con 0x7f4cd0153c80 2023-10-30T14:46:40.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.900+0000 7f4cd4e6b640 1 -- 172.21.15.57:0/1042215068 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb8009420 con 0x7f4cd0153c80 2023-10-30T14:46:40.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.900+0000 7f4cd4e6b640 1 -- 172.21.15.57:0/1042215068 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd01f28a0 con 0x7f4cd0153c80 2023-10-30T14:46:40.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.900+0000 7f4cd4e6b640 1 -- 172.21.15.57:0/1042215068 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd004e170 con 0x7f4cd0153c80 2023-10-30T14:46:40.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.904+0000 7f4cc6ffd640 1 -- 172.21.15.57:0/1042215068 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4cc8014480 con 0x7f4cd0153c80 2023-10-30T14:46:40.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.904+0000 7f4cc6ffd640 1 --2- 172.21.15.57:0/1042215068 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4cb00664a0 0x7f4cb0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:40.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.904+0000 7f4cc6ffd640 1 -- 172.21.15.57:0/1042215068 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f4cc8084640 con 0x7f4cd0153c80 2023-10-30T14:46:40.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.904+0000 7f4cce59c640 1 --2- 172.21.15.57:0/1042215068 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4cb00664a0 0x7f4cb0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:40.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.904+0000 7f4cce59c640 1 --2- 172.21.15.57:0/1042215068 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4cb00664a0 0x7f4cb0068960 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0007fd0 tx=0x7f4cc0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:40.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:40.908+0000 7f4cc6ffd640 1 -- 172.21.15.57:0/1042215068 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4cc8051590 con 0x7f4cd0153c80 2023-10-30T14:46:41.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:41.104+0000 7f4cd4e6b640 1 -- 172.21.15.57:0/1042215068 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cd00cad40 con 0x7f4cd0153c80 2023-10-30T14:46:41.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:41.104+0000 7f4cc6ffd640 1 -- 172.21.15.57:0/1042215068 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+6494 (secure 0 0 0) 0x7f4cc8054bb0 con 0x7f4cd0153c80 2023-10-30T14:46:41.109 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:41.109 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7851,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/59168182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":59168182},{"type":"v1","addr":"172.21.15.57:6836","nonce":59168182}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:46:37.266044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:41.110 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 176 2023-10-30T14:46:41.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:41.108+0000 7f4cd4e6b640 1 -- 172.21.15.57:0/1042215068 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4cb00664a0 msgr2=0x7f4cb0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:41.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:41.108+0000 7f4cd4e6b640 1 --2- 172.21.15.57:0/1042215068 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4cb00664a0 0x7f4cb0068960 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0007fd0 tx=0x7f4cc0009040 comp rx=0 tx=0).stop 2023-10-30T14:46:41.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:41.108+0000 7f4cd4e6b640 1 -- 172.21.15.57:0/1042215068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cd0153c80 msgr2=0x7f4cd01f1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:41.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:41.108+0000 7f4cd4e6b640 1 --2- 172.21.15.57:0/1042215068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cd0153c80 0x7f4cd01f1aa0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f4cc800efc0 tx=0x7f4cc800c490 comp rx=0 tx=0).stop 2023-10-30T14:46:41.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:41.108+0000 7f4cd4e6b640 1 -- 172.21.15.57:0/1042215068 shutdown_connections 2023-10-30T14:46:41.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:41.108+0000 7f4cd4e6b640 1 --2- 172.21.15.57:0/1042215068 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4cb00664a0 0x7f4cb0068960 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:41.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:41.108+0000 7f4cd4e6b640 1 --2- 172.21.15.57:0/1042215068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cd0153c80 0x7f4cd01f1aa0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:41.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:41.108+0000 7f4cd4e6b640 1 --2- 172.21.15.57:0/1042215068 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cd00c4a10 0x7f4cd014e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:41.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:41.108+0000 7f4cd4e6b640 1 --2- 172.21.15.57:0/1042215068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cd00c42a0 0x7f4cd014f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:41.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:41.108+0000 7f4cd4e6b640 1 -- 172.21.15.57:0/1042215068 >> 172.21.15.57:0/1042215068 conn(0x7f4cd00715c0 msgr2=0x7f4cd01591b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:41.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:41.108+0000 7f4cd4e6b640 1 -- 172.21.15.57:0/1042215068 shutdown_connections 2023-10-30T14:46:41.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:41.108+0000 7f4cd4e6b640 1 -- 172.21.15.57:0/1042215068 wait complete. 2023-10-30T14:46:41.130 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:46:43.131 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:43.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.272+0000 7f5690691640 1 Processor -- start 2023-10-30T14:46:43.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.272+0000 7f5690691640 1 -- start start 2023-10-30T14:46:43.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.276+0000 7f5690691640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56881516c0 0x7f56880c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:43.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.276+0000 7f5690691640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f56880c2500 0x7f56880c2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:43.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.276+0000 7f5690691640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5688151aa0 con 0x7f56880c82d0 2023-10-30T14:46:43.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.276+0000 7f5690691640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56880c74a0 con 0x7f56880c2500 2023-10-30T14:46:43.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.276+0000 7f5690691640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56880c2e90 con 0x7f56881516c0 2023-10-30T14:46:43.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.276+0000 7f568e42d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56881516c0 0x7f56880c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:43.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.276+0000 7f568ec2e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f56880c2500 0x7f56880c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:43.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.276+0000 7f568dc2c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f56880c82d0 0x7f56880ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:56398/0 (socket says 172.21.15.57:56398) 2023-10-30T14:46:43.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.276+0000 7f568ec2e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f56880c2500 0x7f56880c2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:58480/0 (socket says 172.21.15.57:58480) 2023-10-30T14:46:43.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.276+0000 7f568dc2c640 1 -- 172.21.15.57:0/2930913714 learned_addr learned my addr 172.21.15.57:0/2930913714 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:46:43.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.276+0000 7f568ec2e640 1 -- 172.21.15.57:0/2930913714 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56881516c0 msgr2=0x7f56880c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:43.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.276+0000 7f568ec2e640 1 --2- 172.21.15.57:0/2930913714 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56881516c0 0x7f56880c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:43.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.276+0000 7f568ec2e640 1 -- 172.21.15.57:0/2930913714 >> v1:172.21.15.88:6789/0 conn(0x7f56880c82d0 legacy=0x7f56880ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:43.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.276+0000 7f568ec2e640 1 -- 172.21.15.57:0/2930913714 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5688158af0 con 0x7f56880c2500 2023-10-30T14:46:43.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.280+0000 7f568ec2e640 1 --2- 172.21.15.57:0/2930913714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56880c2500 0x7f56880c2950 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5680009dc0 tx=0x7f568001c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32ad0fd43520c68f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:43.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.280+0000 7f568d42b640 1 -- 172.21.15.57:0/2930913714 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56800a57f0 con 0x7f56880c2500 2023-10-30T14:46:43.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.280+0000 7f568d42b640 1 -- 172.21.15.57:0/2930913714 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56800a5950 con 0x7f56880c2500 2023-10-30T14:46:43.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.280+0000 7f568d42b640 1 -- 172.21.15.57:0/2930913714 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56800a5ce0 con 0x7f56880c2500 2023-10-30T14:46:43.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.280+0000 7f5690691640 1 -- 172.21.15.57:0/2930913714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56880c2500 msgr2=0x7f56880c2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:43.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.280+0000 7f5690691640 1 --2- 172.21.15.57:0/2930913714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56880c2500 0x7f56880c2950 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5680009dc0 tx=0x7f568001c930 comp rx=0 tx=0).stop 2023-10-30T14:46:43.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.280+0000 7f5690691640 1 -- 172.21.15.57:0/2930913714 shutdown_connections 2023-10-30T14:46:43.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.280+0000 7f5690691640 1 --2- 172.21.15.57:0/2930913714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56880c2500 0x7f56880c2950 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:43.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.280+0000 7f5690691640 1 --2- 172.21.15.57:0/2930913714 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56881516c0 0x7f56880c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:43.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.280+0000 7f5690691640 1 -- 172.21.15.57:0/2930913714 >> 172.21.15.57:0/2930913714 conn(0x7f5688071fb0 msgr2=0x7f56880723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:43.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.280+0000 7f5690691640 1 -- 172.21.15.57:0/2930913714 shutdown_connections 2023-10-30T14:46:43.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.280+0000 7f5690691640 1 -- 172.21.15.57:0/2930913714 wait complete. 2023-10-30T14:46:43.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.280+0000 7f5690691640 1 Processor -- start 2023-10-30T14:46:43.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.280+0000 7f5690691640 1 -- start start 2023-10-30T14:46:43.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f5690691640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56880c2500 0x7f568814d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:43.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f5690691640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56880c82d0 0x7f5688160ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:43.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f5690691640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56881516c0 0x7f5688160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:43.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f5690691640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5688141340 con 0x7f56880c82d0 2023-10-30T14:46:43.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f5690691640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568813f700 con 0x7f56881516c0 2023-10-30T14:46:43.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f5690691640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5688140280 con 0x7f56880c2500 2023-10-30T14:46:43.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f568dc2c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56880c82d0 0x7f5688160ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:43.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f568dc2c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56880c82d0 0x7f5688160ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35876/0 (socket says 172.21.15.57:35876) 2023-10-30T14:46:43.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f568dc2c640 1 -- 172.21.15.57:0/3611210277 learned_addr learned my addr 172.21.15.57:0/3611210277 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:43.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f568e42d640 1 --2- 172.21.15.57:0/3611210277 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56880c2500 0x7f568814d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:43.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f568ec2e640 1 --2- 172.21.15.57:0/3611210277 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56881516c0 0x7f5688160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:43.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f568dc2c640 1 -- 172.21.15.57:0/3611210277 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56880c2500 msgr2=0x7f568814d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:43.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f568dc2c640 1 --2- 172.21.15.57:0/3611210277 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56880c2500 0x7f568814d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:43.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f568dc2c640 1 -- 172.21.15.57:0/3611210277 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56881516c0 msgr2=0x7f5688160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:43.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f568dc2c640 1 --2- 172.21.15.57:0/3611210277 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56881516c0 0x7f5688160ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:43.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f568dc2c640 1 -- 172.21.15.57:0/3611210277 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5680009a70 con 0x7f56880c82d0 2023-10-30T14:46:43.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f568e42d640 1 --2- 172.21.15.57:0/3611210277 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56880c2500 0x7f568814d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:46:43.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f568dc2c640 1 --2- 172.21.15.57:0/3611210277 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56880c82d0 0x7f5688160ab0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f567800c430 tx=0x7f567800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:43.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f567effd640 1 -- 172.21.15.57:0/3611210277 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5678005cf0 con 0x7f56880c82d0 2023-10-30T14:46:43.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f567effd640 1 -- 172.21.15.57:0/3611210277 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5678005e50 con 0x7f56880c82d0 2023-10-30T14:46:43.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f567effd640 1 -- 172.21.15.57:0/3611210277 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56780072d0 con 0x7f56880c82d0 2023-10-30T14:46:43.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f5690691640 1 -- 172.21.15.57:0/3611210277 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5688160620 con 0x7f56880c82d0 2023-10-30T14:46:43.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f5690691640 1 -- 172.21.15.57:0/3611210277 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5688140d80 con 0x7f56880c82d0 2023-10-30T14:46:43.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f5690691640 1 -- 172.21.15.57:0/3611210277 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56881607b0 con 0x7f56880c82d0 2023-10-30T14:46:43.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.284+0000 7f567effd640 1 -- 172.21.15.57:0/3611210277 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5678007430 con 0x7f56880c82d0 2023-10-30T14:46:43.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.288+0000 7f567effd640 1 --2- 172.21.15.57:0/3611210277 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f566c0664a0 0x7f566c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:43.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.288+0000 7f567effd640 1 -- 172.21.15.57:0/3611210277 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f5678083a70 con 0x7f56880c82d0 2023-10-30T14:46:43.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.288+0000 7f568e42d640 1 --2- 172.21.15.57:0/3611210277 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f566c0664a0 0x7f566c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:43.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.288+0000 7f568e42d640 1 --2- 172.21.15.57:0/3611210277 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f566c0664a0 0x7f566c068960 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f5684009bb0 tx=0x7f5684008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:43.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.292+0000 7f567effd640 1 -- 172.21.15.57:0/3611210277 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5678050890 con 0x7f56880c82d0 2023-10-30T14:46:43.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.512+0000 7f5690691640 1 -- 172.21.15.57:0/3611210277 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f568804e170 con 0x7f56880c82d0 2023-10-30T14:46:43.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.512+0000 7f567effd640 1 -- 172.21.15.57:0/3611210277 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+6494 (secure 0 0 0) 0x7f5678017090 con 0x7f56880c82d0 2023-10-30T14:46:43.518 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:43.518 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7851,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/59168182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":59168182},{"type":"v1","addr":"172.21.15.57:6836","nonce":59168182}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:46:37.266044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:43.518 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 176 2023-10-30T14:46:43.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.516+0000 7f5690691640 1 -- 172.21.15.57:0/3611210277 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f566c0664a0 msgr2=0x7f566c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:43.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.516+0000 7f5690691640 1 --2- 172.21.15.57:0/3611210277 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f566c0664a0 0x7f566c068960 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f5684009bb0 tx=0x7f5684008040 comp rx=0 tx=0).stop 2023-10-30T14:46:43.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.516+0000 7f5690691640 1 -- 172.21.15.57:0/3611210277 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56880c82d0 msgr2=0x7f5688160ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:43.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.516+0000 7f5690691640 1 --2- 172.21.15.57:0/3611210277 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56880c82d0 0x7f5688160ab0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f567800c430 tx=0x7f567800c8f0 comp rx=0 tx=0).stop 2023-10-30T14:46:43.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.516+0000 7f5690691640 1 -- 172.21.15.57:0/3611210277 shutdown_connections 2023-10-30T14:46:43.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.516+0000 7f5690691640 1 --2- 172.21.15.57:0/3611210277 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f566c0664a0 0x7f566c068960 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:43.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.516+0000 7f5690691640 1 --2- 172.21.15.57:0/3611210277 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56881516c0 0x7f5688160ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:43.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.516+0000 7f5690691640 1 --2- 172.21.15.57:0/3611210277 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56880c82d0 0x7f5688160ab0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:43.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.516+0000 7f5690691640 1 --2- 172.21.15.57:0/3611210277 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56880c2500 0x7f568814d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:43.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.516+0000 7f5690691640 1 -- 172.21.15.57:0/3611210277 >> 172.21.15.57:0/3611210277 conn(0x7f5688071fb0 msgr2=0x7f5688072390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:43.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.516+0000 7f5690691640 1 -- 172.21.15.57:0/3611210277 shutdown_connections 2023-10-30T14:46:43.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:43.516+0000 7f5690691640 1 -- 172.21.15.57:0/3611210277 wait complete. 2023-10-30T14:46:43.541 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:46:43.681 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 6% (29/481) Updating files: 7% (34/481) Updating files: 8% (39/481) Updating files: 9% (44/481) Updating files: 10% (49/481) Updating files: 11% (53/481) Updating files: 11% (56/481) Updating files: 12% (58/481) Updating files: 13% (63/481) Updating files: 14% (68/481) Updating files: 15% (73/481) Updating files: 16% (77/481) Updating files: 17% (82/481) Updating files: 18% (87/481) Updating files: 19% (92/481) Updating files: 20% (97/481) Updating files: 21% (102/481) Updating files: 22% (106/481) Updating files: 23% (111/481) Updating files: 24% (116/481) Updating files: 25% (121/481) Updating files: 26% (126/481) Updating files: 27% (130/481) Updating files: 28% (135/481) Updating files: 29% (140/481) Updating files: 30% (145/481) Updating files: 31% (150/481) Updating files: 32% (154/481) Updating files: 33% (159/481) Updating files: 34% (164/481) Updating files: 35% (169/481) Updating files: 36% (174/481) Updating files: 37% (178/481) Updating files: 38% (183/481) Updating files: 39% (188/481) Updating files: 40% (193/481) Updating files: 41% (198/481) Updating files: 42% (203/481) Updating files: 43% (207/481) Updating files: 43% (210/481) Updating files: 44% (212/481) Updating files: 45% (217/481) Updating files: 46% (222/481) Updating files: 47% (227/481) Updating files: 48% (231/481) Updating files: 49% (236/481) Updating files: 50% (241/481) Updating files: 51% (246/481) Updating files: 52% (251/481) Updating files: 53% (255/481) Updating files: 54% (260/481) Updating files: 55% (265/481) Updating files: 55% (266/481) Updating files: 56% (270/481) Updating files: 57% (275/481) Updating files: 58% (279/481) Updating files: 59% (284/481) Updating files: 60% (289/481) Updating files: 61% (294/481) Updating files: 61% (298/481) Updating files: 62% (299/481) Updating files: 63% (304/481) Updating files: 64% (308/481) Updating files: 65% (313/481) Updating files: 66% (318/481) Updating files: 67% (323/481) Updating files: 68% (328/481) Updating files: 69% (332/481) Updating files: 69% (334/481) Updating files: 70% (337/481) Updating files: 70% (341/481) Updating files: 71% (342/481) Updating files: 72% (347/481) Updating files: 73% (352/481) Updating files: 74% (356/481) Updating files: 75% (361/481) Updating files: 76% (366/481) Updating files: 77% (371/481) Updating files: 78% (376/481) Updating files: 79% (380/481) Updating files: 80% (385/481) Updating files: 81% (390/481) Updating files: 82% (395/481) Updating files: 83% (400/481) Updating files: 84% (405/481) Updating files: 85% (409/481) Updating files: 86% (414/481) Updating files: 87% (419/481) Updating files: 88% (424/481) Updating files: 88% (428/481) Updating files: 89% (429/481) Updating files: 90% (433/481) Updating files: 91% (438/481) Updating files: 92% (443/481) Updating files: 93% (448/481) Updating files: 94% (453/481) Updating files: 95% (457/481) Updating files: 96% (462/481) Updating files: 97% (467/481) Updating files: 98% (472/481) Updating files: 99% (477/481) Updating files: 100% (481/481) Updating files: 100% (481/481), done. 2023-10-30T14:46:43.816 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at ca6265d0f4d v0.49 2023-10-30T14:46:43.821 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.49 2023-10-30T14:46:44.724 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 50 -eq 48 ] 2023-10-30T14:46:44.724 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.50 2023-10-30T14:46:44.725 INFO:tasks.workunit.client.0.smithi057.stdout:v0.50 2023-10-30T14:46:44.725 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.50 2023-10-30T14:46:44.725 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.50 2023-10-30T14:46:45.542 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:45.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.680+0000 7f9bb4194640 1 Processor -- start 2023-10-30T14:46:45.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.680+0000 7f9bb4194640 1 -- start start 2023-10-30T14:46:45.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb4194640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9bac159760 0x7f9bac15d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:45.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb4194640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bac15cd50 con 0x7f9bac0c5c10 2023-10-30T14:46:45.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb4194640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bac06f600 con 0x7f9bac15da00 2023-10-30T14:46:45.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb4194640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bac15d0a0 con 0x7f9bac159760 2023-10-30T14:46:45.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb1f30640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f9bac0c5c10 0x7f9bac06eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:42728/0 (socket says 172.21.15.57:42728) 2023-10-30T14:46:45.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb1f30640 1 -- 172.21.15.57:0/2985434665 learned_addr learned my addr 172.21.15.57:0/2985434665 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:46:45.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb172f640 1 --2- 172.21.15.57:0/2985434665 >> v2:172.21.15.88:3300/0 conn(0x7f9bac159760 0x7f9bac15d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:45.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb0f2e640 1 -- 172.21.15.57:0/2985434665 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619428254 0 0) 0x7f9bac15cd50 con 0x7f9bac0c5c10 2023-10-30T14:46:45.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb0f2e640 1 -- 172.21.15.57:0/2985434665 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b98003530 con 0x7f9bac0c5c10 2023-10-30T14:46:45.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb0f2e640 1 -- 172.21.15.57:0/2985434665 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1129864353 0 0) 0x7f9b98003530 con 0x7f9bac0c5c10 2023-10-30T14:46:45.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb0f2e640 1 -- 172.21.15.57:0/2985434665 >> v2:172.21.15.88:3300/0 conn(0x7f9bac159760 msgr2=0x7f9bac15d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:45.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb0f2e640 1 --2- 172.21.15.57:0/2985434665 >> v2:172.21.15.88:3300/0 conn(0x7f9bac159760 0x7f9bac15d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:45.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb0f2e640 1 -- 172.21.15.57:0/2985434665 >> v1:172.21.15.88:6789/0 conn(0x7f9bac15da00 legacy=0x7f9bac069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:45.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb0f2e640 1 -- 172.21.15.57:0/2985434665 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac0ca000 con 0x7f9bac0c5c10 2023-10-30T14:46:45.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb0f2e640 1 -- 172.21.15.57:0/2985434665 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f9ba8001680 con 0x7f9bac0c5c10 2023-10-30T14:46:45.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb0f2e640 1 -- 172.21.15.57:0/2985434665 >> v1:172.21.15.57:6789/0 conn(0x7f9bac0c5c10 legacy=0x7f9bac06eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:45.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb0f2e640 1 --2- 172.21.15.57:0/2985434665 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9b98003980 0x7f9b98003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:45.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb0f2e640 1 --2- 172.21.15.57:0/2985434665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9b980047d0 0x7f9b98004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:45.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb0f2e640 1 --2- 172.21.15.57:0/2985434665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b98005190 0x7f9b9800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:45.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb0f2e640 1 -- 172.21.15.57:0/2985434665 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac15d0a0 con 0x7f9b98005190 2023-10-30T14:46:45.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb0f2e640 1 -- 172.21.15.57:0/2985434665 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9800c030 con 0x7f9b980047d0 2023-10-30T14:46:45.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb0f2e640 1 -- 172.21.15.57:0/2985434665 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9800c1f0 con 0x7f9b98003980 2023-10-30T14:46:45.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb2731640 1 --2- 172.21.15.57:0/2985434665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b98005190 0x7f9b9800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:45.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb1f30640 1 --2- 172.21.15.57:0/2985434665 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9b98003980 0x7f9b98003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:45.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb172f640 1 --2- 172.21.15.57:0/2985434665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9b980047d0 0x7f9b98004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:45.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb2731640 1 -- 172.21.15.57:0/2985434665 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9b98003980 msgr2=0x7f9b98003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:45.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb2731640 1 --2- 172.21.15.57:0/2985434665 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9b98003980 0x7f9b98003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:45.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb2731640 1 -- 172.21.15.57:0/2985434665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9b980047d0 msgr2=0x7f9b98004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:45.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb2731640 1 --2- 172.21.15.57:0/2985434665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9b980047d0 0x7f9b98004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:45.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.684+0000 7f9bb2731640 1 -- 172.21.15.57:0/2985434665 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b98003530 con 0x7f9b98005190 2023-10-30T14:46:45.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.688+0000 7f9bb2731640 1 -- 172.21.15.57:0/2985434665 reap_dead start 2023-10-30T14:46:45.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.688+0000 7f9bb2731640 1 --2- 172.21.15.57:0/2985434665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b98005190 0x7f9b9800b990 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4001e40 tx=0x7f9ba401c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c92831e9b73d775 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:45.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.688+0000 7f9bb0f2e640 1 -- 172.21.15.57:0/2985434665 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba4003b80 con 0x7f9b98005190 2023-10-30T14:46:45.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.688+0000 7f9bb0f2e640 1 -- 172.21.15.57:0/2985434665 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba4003ce0 con 0x7f9b98005190 2023-10-30T14:46:45.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.688+0000 7f9bb0f2e640 1 -- 172.21.15.57:0/2985434665 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba4005240 con 0x7f9b98005190 2023-10-30T14:46:45.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.688+0000 7f9bb4194640 1 -- 172.21.15.57:0/2985434665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b98005190 msgr2=0x7f9b9800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:45.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.688+0000 7f9bb4194640 1 --2- 172.21.15.57:0/2985434665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b98005190 0x7f9b9800b990 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4001e40 tx=0x7f9ba401c740 comp rx=0 tx=0).stop 2023-10-30T14:46:45.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.688+0000 7f9bb4194640 1 -- 172.21.15.57:0/2985434665 shutdown_connections 2023-10-30T14:46:45.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.688+0000 7f9bb4194640 1 --2- 172.21.15.57:0/2985434665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b98005190 0x7f9b9800b990 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:45.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.688+0000 7f9bb4194640 1 -- 172.21.15.57:0/2985434665 >> 172.21.15.57:0/2985434665 conn(0x7f9bac06afa0 msgr2=0x7f9bac06b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:45.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.688+0000 7f9bb4194640 1 -- 172.21.15.57:0/2985434665 shutdown_connections 2023-10-30T14:46:45.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.688+0000 7f9bb4194640 1 -- 172.21.15.57:0/2985434665 wait complete. 2023-10-30T14:46:45.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb4194640 1 Processor -- start 2023-10-30T14:46:45.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb4194640 1 -- start start 2023-10-30T14:46:45.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb4194640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b98005190 0x7f9bac14f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:45.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb4194640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bac073a10 0x7f9bac14fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:45.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb4194640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bac1f1e40 0x7f9bac1f22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:45.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb4194640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac152820 con 0x7f9b98005190 2023-10-30T14:46:45.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb4194640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac1f19b0 con 0x7f9bac073a10 2023-10-30T14:46:45.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb4194640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac1522f0 con 0x7f9bac1f1e40 2023-10-30T14:46:45.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb1f30640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b98005190 0x7f9bac14f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:45.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb1f30640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b98005190 0x7f9bac14f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59228/0 (socket says 172.21.15.57:59228) 2023-10-30T14:46:45.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb1f30640 1 -- 172.21.15.57:0/3195098247 learned_addr learned my addr 172.21.15.57:0/3195098247 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:45.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb172f640 1 --2- 172.21.15.57:0/3195098247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bac073a10 0x7f9bac14fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:45.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb2731640 1 --2- 172.21.15.57:0/3195098247 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bac1f1e40 0x7f9bac1f22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:45.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb1f30640 1 -- 172.21.15.57:0/3195098247 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bac1f1e40 msgr2=0x7f9bac1f22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:45.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb1f30640 1 --2- 172.21.15.57:0/3195098247 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bac1f1e40 0x7f9bac1f22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:45.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb1f30640 1 -- 172.21.15.57:0/3195098247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bac073a10 msgr2=0x7f9bac14fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:45.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb1f30640 1 --2- 172.21.15.57:0/3195098247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bac073a10 0x7f9bac14fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:45.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb1f30640 1 -- 172.21.15.57:0/3195098247 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac1f2800 con 0x7f9b98005190 2023-10-30T14:46:45.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb1f30640 1 --2- 172.21.15.57:0/3195098247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b98005190 0x7f9bac14f580 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f9ba80072c0 tx=0x7f9ba80076c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:45.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9ba27fc640 1 -- 172.21.15.57:0/3195098247 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba800f940 con 0x7f9b98005190 2023-10-30T14:46:45.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9ba27fc640 1 -- 172.21.15.57:0/3195098247 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba800faa0 con 0x7f9b98005190 2023-10-30T14:46:45.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9ba27fc640 1 -- 172.21.15.57:0/3195098247 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba800fe50 con 0x7f9b98005190 2023-10-30T14:46:45.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb4194640 1 -- 172.21.15.57:0/3195098247 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bac156e90 con 0x7f9b98005190 2023-10-30T14:46:45.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb4194640 1 -- 172.21.15.57:0/3195098247 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bac152540 con 0x7f9b98005190 2023-10-30T14:46:45.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.692+0000 7f9bb4194640 1 -- 172.21.15.57:0/3195098247 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bac156cd0 con 0x7f9b98005190 2023-10-30T14:46:45.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.696+0000 7f9ba27fc640 1 -- 172.21.15.57:0/3195098247 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9ba801d460 con 0x7f9b98005190 2023-10-30T14:46:45.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.696+0000 7f9ba27fc640 1 --2- 172.21.15.57:0/3195098247 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9b900664a0 0x7f9b90068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:45.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.696+0000 7f9ba27fc640 1 -- 172.21.15.57:0/3195098247 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f9ba80845f0 con 0x7f9b98005190 2023-10-30T14:46:45.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.696+0000 7f9bb172f640 1 --2- 172.21.15.57:0/3195098247 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9b900664a0 0x7f9b90068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:45.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.696+0000 7f9bb172f640 1 --2- 172.21.15.57:0/3195098247 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9b900664a0 0x7f9b90068960 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f9bac1ef570 tx=0x7f9b9c009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:45.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.700+0000 7f9ba27fc640 1 -- 172.21.15.57:0/3195098247 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9ba8051480 con 0x7f9b98005190 2023-10-30T14:46:45.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.896+0000 7f9bb4194640 1 -- 172.21.15.57:0/3195098247 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bac0c71e0 con 0x7f9b98005190 2023-10-30T14:46:45.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.896+0000 7f9ba27fc640 1 -- 172.21.15.57:0/3195098247 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+6494 (secure 0 0 0) 0x7f9ba8054aa0 con 0x7f9b98005190 2023-10-30T14:46:45.899 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:45.900 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7851,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/59168182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":59168182},{"type":"v1","addr":"172.21.15.57:6836","nonce":59168182}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:46:37.266044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:45.900 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 176 2023-10-30T14:46:45.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.896+0000 7f9bb4194640 1 -- 172.21.15.57:0/3195098247 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9b900664a0 msgr2=0x7f9b90068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:45.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.896+0000 7f9bb4194640 1 --2- 172.21.15.57:0/3195098247 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9b900664a0 0x7f9b90068960 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f9bac1ef570 tx=0x7f9b9c009190 comp rx=0 tx=0).stop 2023-10-30T14:46:45.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.896+0000 7f9bb4194640 1 -- 172.21.15.57:0/3195098247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b98005190 msgr2=0x7f9bac14f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:45.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.896+0000 7f9bb4194640 1 --2- 172.21.15.57:0/3195098247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b98005190 0x7f9bac14f580 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f9ba80072c0 tx=0x7f9ba80076c0 comp rx=0 tx=0).stop 2023-10-30T14:46:45.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.896+0000 7f9bb4194640 1 -- 172.21.15.57:0/3195098247 shutdown_connections 2023-10-30T14:46:45.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.896+0000 7f9bb4194640 1 --2- 172.21.15.57:0/3195098247 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9b900664a0 0x7f9b90068960 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:45.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.896+0000 7f9bb4194640 1 --2- 172.21.15.57:0/3195098247 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bac1f1e40 0x7f9bac1f22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:45.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.896+0000 7f9bb4194640 1 --2- 172.21.15.57:0/3195098247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bac073a10 0x7f9bac14fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:45.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.896+0000 7f9bb4194640 1 --2- 172.21.15.57:0/3195098247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b98005190 0x7f9bac14f580 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:45.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.896+0000 7f9bb4194640 1 -- 172.21.15.57:0/3195098247 >> 172.21.15.57:0/3195098247 conn(0x7f9bac06afa0 msgr2=0x7f9bac159050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:45.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.896+0000 7f9bb4194640 1 -- 172.21.15.57:0/3195098247 shutdown_connections 2023-10-30T14:46:45.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:45.896+0000 7f9bb4194640 1 -- 172.21.15.57:0/3195098247 wait complete. 2023-10-30T14:46:45.922 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:46:47.923 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:48.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.072+0000 7f0ad96a9640 1 Processor -- start 2023-10-30T14:46:48.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.072+0000 7f0ad96a9640 1 -- start start 2023-10-30T14:46:48.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.072+0000 7f0ad96a9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ad40718d0 0x7f0ad40c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:48.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.072+0000 7f0ad96a9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0ad4071cb0 0x7f0ad415a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:48.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.072+0000 7f0ad96a9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad415a770 con 0x7f0ad4153300 2023-10-30T14:46:48.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.072+0000 7f0ad96a9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad40cca10 con 0x7f0ad4071cb0 2023-10-30T14:46:48.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.072+0000 7f0ad96a9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad415d0a0 con 0x7f0ad40718d0 2023-10-30T14:46:48.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.072+0000 7f0ad2ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ad40718d0 0x7f0ad40c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:48.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.072+0000 7f0ad2ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ad40718d0 0x7f0ad40c9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:39138/0 (socket says 172.21.15.57:39138) 2023-10-30T14:46:48.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.072+0000 7f0ad2ffd640 1 -- 172.21.15.57:0/494402545 learned_addr learned my addr 172.21.15.57:0/494402545 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:48.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.072+0000 7f0ad27fc640 1 --2- 172.21.15.57:0/494402545 >> v2:172.21.15.88:3300/0 conn(0x7f0ad4071cb0 0x7f0ad415a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:48.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.076+0000 7f0ad2ffd640 1 -- 172.21.15.57:0/494402545 >> v2:172.21.15.88:3300/0 conn(0x7f0ad4071cb0 msgr2=0x7f0ad415a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:48.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.076+0000 7f0ad2ffd640 1 --2- 172.21.15.57:0/494402545 >> v2:172.21.15.88:3300/0 conn(0x7f0ad4071cb0 0x7f0ad415a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:48.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.076+0000 7f0ad2ffd640 1 -- 172.21.15.57:0/494402545 >> v1:172.21.15.88:6789/0 conn(0x7f0ad4153300 legacy=0x7f0ad40cade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:48.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.076+0000 7f0ad2ffd640 1 -- 172.21.15.57:0/494402545 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad415d380 con 0x7f0ad40718d0 2023-10-30T14:46:48.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.076+0000 7f0ad2ffd640 1 --2- 172.21.15.57:0/494402545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ad40718d0 0x7f0ad40c9a10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4009870 tx=0x7f0ac401ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8192cc9893237e09 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:48.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.080+0000 7f0ad1ffb640 1 -- 172.21.15.57:0/494402545 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac40a8a30 con 0x7f0ad40718d0 2023-10-30T14:46:48.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.080+0000 7f0ad1ffb640 1 -- 172.21.15.57:0/494402545 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac40a8b90 con 0x7f0ad40718d0 2023-10-30T14:46:48.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.080+0000 7f0ad1ffb640 1 -- 172.21.15.57:0/494402545 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac40b0440 con 0x7f0ad40718d0 2023-10-30T14:46:48.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.080+0000 7f0ad96a9640 1 -- 172.21.15.57:0/494402545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ad40718d0 msgr2=0x7f0ad40c9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:48.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.080+0000 7f0ad96a9640 1 --2- 172.21.15.57:0/494402545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ad40718d0 0x7f0ad40c9a10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4009870 tx=0x7f0ac401ed30 comp rx=0 tx=0).stop 2023-10-30T14:46:48.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.080+0000 7f0ad96a9640 1 -- 172.21.15.57:0/494402545 shutdown_connections 2023-10-30T14:46:48.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.080+0000 7f0ad96a9640 1 --2- 172.21.15.57:0/494402545 >> v2:172.21.15.88:3300/0 conn(0x7f0ad4071cb0 0x7f0ad415a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:48.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.080+0000 7f0ad96a9640 1 --2- 172.21.15.57:0/494402545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ad40718d0 0x7f0ad40c9a10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:48.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.080+0000 7f0ad96a9640 1 -- 172.21.15.57:0/494402545 >> 172.21.15.57:0/494402545 conn(0x7f0ad406ab00 msgr2=0x7f0ad4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:48.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.080+0000 7f0ad96a9640 1 -- 172.21.15.57:0/494402545 shutdown_connections 2023-10-30T14:46:48.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.080+0000 7f0ad96a9640 1 -- 172.21.15.57:0/494402545 wait complete. 2023-10-30T14:46:48.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad96a9640 1 Processor -- start 2023-10-30T14:46:48.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad96a9640 1 -- start start 2023-10-30T14:46:48.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad96a9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ad40718d0 0x7f0ad4148e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:48.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad96a9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ad4071cb0 0x7f0ad41f09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:48.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad96a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ad4153300 0x7f0ad41f0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:48.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad96a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad4149c50 con 0x7f0ad4153300 2023-10-30T14:46:48.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad96a9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad41f0540 con 0x7f0ad4071cb0 2023-10-30T14:46:48.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad96a9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad41efad0 con 0x7f0ad40718d0 2023-10-30T14:46:48.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ad4153300 0x7f0ad41f0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:48.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad2ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ad40718d0 0x7f0ad4148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:48.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad27fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ad4071cb0 0x7f0ad41f09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:48.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad2ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ad40718d0 0x7f0ad4148e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:39142/0 (socket says 172.21.15.57:39142) 2023-10-30T14:46:48.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad2ffd640 1 -- 172.21.15.57:0/3190946441 learned_addr learned my addr 172.21.15.57:0/3190946441 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:48.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad37fe640 1 -- 172.21.15.57:0/3190946441 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ad40718d0 msgr2=0x7f0ad4148e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:48.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad37fe640 1 --2- 172.21.15.57:0/3190946441 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ad40718d0 0x7f0ad4148e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:48.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad37fe640 1 -- 172.21.15.57:0/3190946441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ad4071cb0 msgr2=0x7f0ad41f09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:48.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad37fe640 1 --2- 172.21.15.57:0/3190946441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ad4071cb0 0x7f0ad41f09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:48.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad37fe640 1 -- 172.21.15.57:0/3190946441 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac40094d0 con 0x7f0ad4153300 2023-10-30T14:46:48.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad37fe640 1 --2- 172.21.15.57:0/3190946441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ad4153300 0x7f0ad41f0f10 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f0acc00c700 tx=0x7f0acc00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:48.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0abb7fe640 1 -- 172.21.15.57:0/3190946441 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc003bd0 con 0x7f0ad4153300 2023-10-30T14:46:48.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0abb7fe640 1 -- 172.21.15.57:0/3190946441 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0acc003d50 con 0x7f0ad4153300 2023-10-30T14:46:48.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0abb7fe640 1 -- 172.21.15.57:0/3190946441 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc005310 con 0x7f0ad4153300 2023-10-30T14:46:48.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad96a9640 1 -- 172.21.15.57:0/3190946441 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad4149970 con 0x7f0ad4153300 2023-10-30T14:46:48.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.084+0000 7f0ad96a9640 1 -- 172.21.15.57:0/3190946441 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad40b40e0 con 0x7f0ad4153300 2023-10-30T14:46:48.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.088+0000 7f0ad96a9640 1 -- 172.21.15.57:0/3190946441 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad41f06b0 con 0x7f0ad4153300 2023-10-30T14:46:48.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.088+0000 7f0abb7fe640 1 -- 172.21.15.57:0/3190946441 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0acc005470 con 0x7f0ad4153300 2023-10-30T14:46:48.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.088+0000 7f0abb7fe640 1 --2- 172.21.15.57:0/3190946441 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0ac00664d0 0x7f0ac0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:48.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.088+0000 7f0abb7fe640 1 -- 172.21.15.57:0/3190946441 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f0acc084a60 con 0x7f0ad4153300 2023-10-30T14:46:48.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.088+0000 7f0ad2ffd640 1 --2- 172.21.15.57:0/3190946441 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0ac00664d0 0x7f0ac0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:48.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.092+0000 7f0ad2ffd640 1 --2- 172.21.15.57:0/3190946441 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0ac00664d0 0x7f0ac0068990 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f0ac401ed70 tx=0x7f0ac4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:48.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.092+0000 7f0abb7fe640 1 -- 172.21.15.57:0/3190946441 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0acc051a90 con 0x7f0ad4153300 2023-10-30T14:46:48.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.352+0000 7f0ad96a9640 1 -- 172.21.15.57:0/3190946441 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ad404e170 con 0x7f0ad4153300 2023-10-30T14:46:48.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.352+0000 7f0abb7fe640 1 -- 172.21.15.57:0/3190946441 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+6494 (secure 0 0 0) 0x7f0acc0550b0 con 0x7f0ad4153300 2023-10-30T14:46:48.357 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:48.358 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7851,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/59168182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":59168182},{"type":"v1","addr":"172.21.15.57:6836","nonce":59168182}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:46:37.266044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:48.358 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 176 2023-10-30T14:46:48.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.356+0000 7f0ad96a9640 1 -- 172.21.15.57:0/3190946441 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0ac00664d0 msgr2=0x7f0ac0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:48.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.356+0000 7f0ad96a9640 1 --2- 172.21.15.57:0/3190946441 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0ac00664d0 0x7f0ac0068990 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f0ac401ed70 tx=0x7f0ac4009040 comp rx=0 tx=0).stop 2023-10-30T14:46:48.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.356+0000 7f0ad96a9640 1 -- 172.21.15.57:0/3190946441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ad4153300 msgr2=0x7f0ad41f0f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:48.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.356+0000 7f0ad96a9640 1 --2- 172.21.15.57:0/3190946441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ad4153300 0x7f0ad41f0f10 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f0acc00c700 tx=0x7f0acc00cbc0 comp rx=0 tx=0).stop 2023-10-30T14:46:48.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.356+0000 7f0ad96a9640 1 -- 172.21.15.57:0/3190946441 shutdown_connections 2023-10-30T14:46:48.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.356+0000 7f0ad96a9640 1 --2- 172.21.15.57:0/3190946441 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0ac00664d0 0x7f0ac0068990 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:48.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.356+0000 7f0ad96a9640 1 --2- 172.21.15.57:0/3190946441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ad4153300 0x7f0ad41f0f10 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:48.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.356+0000 7f0ad96a9640 1 --2- 172.21.15.57:0/3190946441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ad4071cb0 0x7f0ad41f09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:48.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.356+0000 7f0ad96a9640 1 --2- 172.21.15.57:0/3190946441 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ad40718d0 0x7f0ad4148e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:48.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.356+0000 7f0ad96a9640 1 -- 172.21.15.57:0/3190946441 >> 172.21.15.57:0/3190946441 conn(0x7f0ad406ab00 msgr2=0x7f0ad4150d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:48.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.356+0000 7f0ad96a9640 1 -- 172.21.15.57:0/3190946441 shutdown_connections 2023-10-30T14:46:48.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:48.356+0000 7f0ad96a9640 1 -- 172.21.15.57:0/3190946441 wait complete. 2023-10-30T14:46:48.379 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:46:48.787 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 42% (58/137) Updating files: 43% (59/137) Updating files: 44% (61/137) Updating files: 45% (62/137) Updating files: 46% (64/137) Updating files: 47% (65/137) Updating files: 48% (66/137) Updating files: 49% (68/137) Updating files: 50% (69/137) Updating files: 51% (70/137) Updating files: 52% (72/137) Updating files: 53% (73/137) Updating files: 54% (74/137) Updating files: 55% (76/137) Updating files: 56% (77/137) Updating files: 57% (79/137) Updating files: 58% (80/137) Updating files: 59% (81/137) Updating files: 60% (83/137) Updating files: 61% (84/137) Updating files: 62% (85/137) Updating files: 63% (87/137) Updating files: 64% (88/137) Updating files: 65% (90/137) Updating files: 66% (91/137) Updating files: 67% (92/137) Updating files: 68% (94/137) Updating files: 69% (95/137) Updating files: 70% (96/137) Updating files: 71% (98/137) Updating files: 72% (99/137) Updating files: 73% (101/137) Updating files: 74% (102/137) Updating files: 75% (103/137) Updating files: 76% (105/137) Updating files: 77% (106/137) Updating files: 78% (107/137) Updating files: 79% (109/137) Updating files: 80% (110/137) Updating files: 81% (111/137) Updating files: 82% (113/137) Updating files: 83% (114/137) Updating files: 84% (116/137) Updating files: 85% (117/137) Updating files: 86% (118/137) Updating files: 87% (120/137) Updating files: 88% (121/137) Updating files: 89% (122/137) Updating files: 90% (124/137) Updating files: 91% (125/137) Updating files: 92% (127/137) Updating files: 93% (128/137) Updating files: 94% (129/137) Updating files: 95% (131/137) Updating files: 96% (132/137) Updating files: 97% (133/137) Updating files: 98% (135/137) Updating files: 99% (136/137) Updating files: 100% (137/137) Updating files: 100% (137/137), done. 2023-10-30T14:46:48.817 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at a4428bd0e19 v0.50 2023-10-30T14:46:48.820 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.50 2023-10-30T14:46:48.886 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 51 -eq 48 ] 2023-10-30T14:46:48.886 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.51 2023-10-30T14:46:48.887 INFO:tasks.workunit.client.0.smithi057.stdout:v0.51 2023-10-30T14:46:48.887 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.51 2023-10-30T14:46:48.887 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.51 2023-10-30T14:46:50.000 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 70% (74/105) Updating files: 71% (75/105) Updating files: 72% (76/105) Updating files: 73% (77/105) Updating files: 74% (78/105) Updating files: 75% (79/105) Updating files: 76% (80/105) Updating files: 77% (81/105) Updating files: 78% (82/105) Updating files: 79% (83/105) Updating files: 80% (84/105) Updating files: 81% (86/105) Updating files: 82% (87/105) Updating files: 83% (88/105) Updating files: 84% (89/105) Updating files: 85% (90/105) Updating files: 86% (91/105) Updating files: 87% (92/105) Updating files: 88% (93/105) Updating files: 89% (94/105) Updating files: 90% (95/105) Updating files: 91% (96/105) Updating files: 92% (97/105) Updating files: 93% (98/105) Updating files: 94% (99/105) Updating files: 95% (100/105) Updating files: 96% (101/105) Updating files: 97% (102/105) Updating files: 98% (103/105) Updating files: 99% (104/105) Updating files: 100% (105/105) Updating files: 100% (105/105), done. 2023-10-30T14:46:50.062 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at c03ca95d235 v0.51 2023-10-30T14:46:50.066 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.51 2023-10-30T14:46:50.381 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:50.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.524+0000 7f2fd0805640 1 Processor -- start 2023-10-30T14:46:50.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.524+0000 7f2fd0805640 1 -- start start 2023-10-30T14:46:50.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.524+0000 7f2fd0805640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2fcc153c80 0x7f2fcc15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:50.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.524+0000 7f2fd0805640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2fcc0c4a10 0x7f2fcc0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:50.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.524+0000 7f2fd0805640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fcc15a3b0 0x7f2fcc0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:50.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.524+0000 7f2fd0805640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fcc15dec0 con 0x7f2fcc0c4a10 2023-10-30T14:46:50.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.524+0000 7f2fd0805640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fcc1599a0 con 0x7f2fcc153c80 2023-10-30T14:46:50.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.524+0000 7f2fd0805640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc159f20 con 0x7f2fcc15a3b0 2023-10-30T14:46:50.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.524+0000 7f2fc9d9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2fcc0c4a10 0x7f2fcc0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:50.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.524+0000 7f2fc9d9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2fcc0c4a10 0x7f2fcc0c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59244/0 (socket says 172.21.15.57:59244) 2023-10-30T14:46:50.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.524+0000 7f2fc9d9b640 1 -- 172.21.15.57:0/1365880508 learned_addr learned my addr 172.21.15.57:0/1365880508 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:50.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.524+0000 7f2fcad9d640 1 --2- 172.21.15.57:0/1365880508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fcc15a3b0 0x7f2fcc0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:50.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.524+0000 7f2fca59c640 1 --2- 172.21.15.57:0/1365880508 >> v2:172.21.15.88:3300/0 conn(0x7f2fcc153c80 0x7f2fcc15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:50.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.524+0000 7f2fc9d9b640 1 -- 172.21.15.57:0/1365880508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fcc15a3b0 msgr2=0x7f2fcc0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:50.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.524+0000 7f2fc9d9b640 1 --2- 172.21.15.57:0/1365880508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fcc15a3b0 0x7f2fcc0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:50.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.524+0000 7f2fc9d9b640 1 -- 172.21.15.57:0/1365880508 >> v2:172.21.15.88:3300/0 conn(0x7f2fcc153c80 msgr2=0x7f2fcc15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:50.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.524+0000 7f2fc9d9b640 1 --2- 172.21.15.57:0/1365880508 >> v2:172.21.15.88:3300/0 conn(0x7f2fcc153c80 0x7f2fcc15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:50.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.524+0000 7f2fc9d9b640 1 -- 172.21.15.57:0/1365880508 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc0c47e0 con 0x7f2fcc0c4a10 2023-10-30T14:46:50.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.528+0000 7f2fc9d9b640 1 --2- 172.21.15.57:0/1365880508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fcc0c4a10 0x7f2fcc0c42a0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0d00f0 tx=0x7f2fc001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=addd7d459923861f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:50.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.528+0000 7f2fc959a640 1 -- 172.21.15.57:0/1365880508 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc00a89b0 con 0x7f2fcc0c4a10 2023-10-30T14:46:50.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.528+0000 7f2fc959a640 1 -- 172.21.15.57:0/1365880508 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc00a8b10 con 0x7f2fcc0c4a10 2023-10-30T14:46:50.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.528+0000 7f2fc959a640 1 -- 172.21.15.57:0/1365880508 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc00a8ea0 con 0x7f2fcc0c4a10 2023-10-30T14:46:50.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.528+0000 7f2fd0805640 1 -- 172.21.15.57:0/1365880508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fcc0c4a10 msgr2=0x7f2fcc0c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:50.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.528+0000 7f2fd0805640 1 --2- 172.21.15.57:0/1365880508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fcc0c4a10 0x7f2fcc0c42a0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0d00f0 tx=0x7f2fc001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:46:50.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.528+0000 7f2fd0805640 1 -- 172.21.15.57:0/1365880508 shutdown_connections 2023-10-30T14:46:50.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.528+0000 7f2fd0805640 1 --2- 172.21.15.57:0/1365880508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fcc15a3b0 0x7f2fcc0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:50.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.528+0000 7f2fd0805640 1 --2- 172.21.15.57:0/1365880508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fcc0c4a10 0x7f2fcc0c42a0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:50.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.528+0000 7f2fd0805640 1 --2- 172.21.15.57:0/1365880508 >> v2:172.21.15.88:3300/0 conn(0x7f2fcc153c80 0x7f2fcc15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:50.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.528+0000 7f2fd0805640 1 -- 172.21.15.57:0/1365880508 >> 172.21.15.57:0/1365880508 conn(0x7f2fcc0715c0 msgr2=0x7f2fcc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:50.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.528+0000 7f2fd0805640 1 -- 172.21.15.57:0/1365880508 shutdown_connections 2023-10-30T14:46:50.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.528+0000 7f2fd0805640 1 -- 172.21.15.57:0/1365880508 wait complete. 2023-10-30T14:46:50.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fd0805640 1 Processor -- start 2023-10-30T14:46:50.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fd0805640 1 -- start start 2023-10-30T14:46:50.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fd0805640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fcc0c4a10 0x7f2fcc1f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:50.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fd0805640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fcc153c80 0x7f2fcc0b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:50.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fd0805640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fcc15a3b0 0x7f2fcc147c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:50.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fd0805640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1f2c20 con 0x7f2fcc153c80 2023-10-30T14:46:50.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fd0805640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc148620 con 0x7f2fcc0c4a10 2023-10-30T14:46:50.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fd0805640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1ee870 con 0x7f2fcc15a3b0 2023-10-30T14:46:50.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fc9d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fcc153c80 0x7f2fcc0b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:50.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fc9d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fcc153c80 0x7f2fcc0b4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59260/0 (socket says 172.21.15.57:59260) 2023-10-30T14:46:50.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fc9d9b640 1 -- 172.21.15.57:0/81643902 learned_addr learned my addr 172.21.15.57:0/81643902 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:50.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fca59c640 1 --2- 172.21.15.57:0/81643902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fcc0c4a10 0x7f2fcc1f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:50.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fcad9d640 1 --2- 172.21.15.57:0/81643902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fcc15a3b0 0x7f2fcc147c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:50.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fc9d9b640 1 -- 172.21.15.57:0/81643902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fcc15a3b0 msgr2=0x7f2fcc147c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:50.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fc9d9b640 1 --2- 172.21.15.57:0/81643902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fcc15a3b0 0x7f2fcc147c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:50.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fc9d9b640 1 -- 172.21.15.57:0/81643902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fcc0c4a10 msgr2=0x7f2fcc1f26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:50.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fc9d9b640 1 --2- 172.21.15.57:0/81643902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fcc0c4a10 0x7f2fcc1f26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:50.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fc9d9b640 1 -- 172.21.15.57:0/81643902 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc00094d0 con 0x7f2fcc153c80 2023-10-30T14:46:50.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fcad9d640 1 --2- 172.21.15.57:0/81643902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fcc15a3b0 0x7f2fcc147c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:46:50.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fca59c640 1 --2- 172.21.15.57:0/81643902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fcc0c4a10 0x7f2fcc1f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:46:50.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fc9d9b640 1 --2- 172.21.15.57:0/81643902 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fcc153c80 0x7f2fcc0b4d30 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f2fc001d3a0 tx=0x7f2fc001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:50.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fbaffd640 1 -- 172.21.15.57:0/81643902 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc00ba600 con 0x7f2fcc153c80 2023-10-30T14:46:50.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fbaffd640 1 -- 172.21.15.57:0/81643902 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc00ba760 con 0x7f2fcc153c80 2023-10-30T14:46:50.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fbaffd640 1 -- 172.21.15.57:0/81643902 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc00bab10 con 0x7f2fcc153c80 2023-10-30T14:46:50.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fd0805640 1 -- 172.21.15.57:0/81643902 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fcc1eeb50 con 0x7f2fcc153c80 2023-10-30T14:46:50.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fd0805640 1 -- 172.21.15.57:0/81643902 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fcc1ef110 con 0x7f2fcc153c80 2023-10-30T14:46:50.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.532+0000 7f2fd0805640 1 -- 172.21.15.57:0/81643902 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fcc04e170 con 0x7f2fcc153c80 2023-10-30T14:46:50.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.536+0000 7f2fbaffd640 1 -- 172.21.15.57:0/81643902 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2fc00bac70 con 0x7f2fcc153c80 2023-10-30T14:46:50.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.536+0000 7f2fbaffd640 1 --2- 172.21.15.57:0/81643902 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2fac0664a0 0x7f2fac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:50.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.536+0000 7f2fbaffd640 1 -- 172.21.15.57:0/81643902 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f2fc0122b00 con 0x7f2fcc153c80 2023-10-30T14:46:50.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.536+0000 7f2fca59c640 1 --2- 172.21.15.57:0/81643902 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2fac0664a0 0x7f2fac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:50.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.536+0000 7f2fca59c640 1 --2- 172.21.15.57:0/81643902 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2fac0664a0 0x7f2fac068960 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc004660 tx=0x7f2fbc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:50.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.540+0000 7f2fbaffd640 1 -- 172.21.15.57:0/81643902 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2fc0124e60 con 0x7f2fcc153c80 2023-10-30T14:46:50.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.728+0000 7f2fd0805640 1 -- 172.21.15.57:0/81643902 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fcc0cad40 con 0x7f2fcc153c80 2023-10-30T14:46:50.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.732+0000 7f2fbaffd640 1 -- 172.21.15.57:0/81643902 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+6494 (secure 0 0 0) 0x7f2fc00ef900 con 0x7f2fcc153c80 2023-10-30T14:46:50.735 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:50.735 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7851,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/59168182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":59168182},{"type":"v1","addr":"172.21.15.57:6836","nonce":59168182}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:46:37.266044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:50.736 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 176 2023-10-30T14:46:50.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.732+0000 7f2fd0805640 1 -- 172.21.15.57:0/81643902 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2fac0664a0 msgr2=0x7f2fac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:50.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.732+0000 7f2fd0805640 1 --2- 172.21.15.57:0/81643902 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2fac0664a0 0x7f2fac068960 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc004660 tx=0x7f2fbc008040 comp rx=0 tx=0).stop 2023-10-30T14:46:50.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.732+0000 7f2fd0805640 1 -- 172.21.15.57:0/81643902 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fcc153c80 msgr2=0x7f2fcc0b4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:50.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.732+0000 7f2fd0805640 1 --2- 172.21.15.57:0/81643902 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fcc153c80 0x7f2fcc0b4d30 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f2fc001d3a0 tx=0x7f2fc001ba90 comp rx=0 tx=0).stop 2023-10-30T14:46:50.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.732+0000 7f2fd0805640 1 -- 172.21.15.57:0/81643902 shutdown_connections 2023-10-30T14:46:50.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.732+0000 7f2fd0805640 1 --2- 172.21.15.57:0/81643902 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2fac0664a0 0x7f2fac068960 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:50.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.732+0000 7f2fd0805640 1 --2- 172.21.15.57:0/81643902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fcc15a3b0 0x7f2fcc147c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:50.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.732+0000 7f2fd0805640 1 --2- 172.21.15.57:0/81643902 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fcc153c80 0x7f2fcc0b4d30 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:50.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.732+0000 7f2fd0805640 1 --2- 172.21.15.57:0/81643902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fcc0c4a10 0x7f2fcc1f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:50.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.732+0000 7f2fd0805640 1 -- 172.21.15.57:0/81643902 >> 172.21.15.57:0/81643902 conn(0x7f2fcc0715c0 msgr2=0x7f2fcc0c9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:50.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.732+0000 7f2fd0805640 1 -- 172.21.15.57:0/81643902 shutdown_connections 2023-10-30T14:46:50.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:50.732+0000 7f2fd0805640 1 -- 172.21.15.57:0/81643902 wait complete. 2023-10-30T14:46:50.755 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:46:50.876 INFO:tasks.workunit.client.0.smithi057.stdout:v0.52 2023-10-30T14:46:50.877 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 52 -eq 48 ] 2023-10-30T14:46:50.877 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.52 2023-10-30T14:46:50.877 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.52 2023-10-30T14:46:50.877 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.52 2023-10-30T14:46:52.122 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at e48859474c4 v0.52 2023-10-30T14:46:52.126 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.52 2023-10-30T14:46:52.462 INFO:tasks.workunit.client.0.smithi057.stdout:v0.53 2023-10-30T14:46:52.462 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 53 -eq 48 ] 2023-10-30T14:46:52.463 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.53 2023-10-30T14:46:52.463 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.53 2023-10-30T14:46:52.463 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.53 2023-10-30T14:46:52.757 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:52.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.896+0000 7f20626ea640 1 Processor -- start 2023-10-30T14:46:52.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.896+0000 7f20626ea640 1 -- start start 2023-10-30T14:46:52.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.900+0000 7f20626ea640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f205c153c10 0x7f205c065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:52.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.900+0000 7f20626ea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f205c070c30 0x7f205c071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:52.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.900+0000 7f20626ea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f205c06e390 con 0x7f205c159970 2023-10-30T14:46:52.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.900+0000 7f20626ea640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f205c072390 con 0x7f205c153c10 2023-10-30T14:46:52.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.900+0000 7f20626ea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f205c0660e0 con 0x7f205c070c30 2023-10-30T14:46:52.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.900+0000 7f205bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f205c153c10 0x7f205c065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:52.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.900+0000 7f205bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f205c153c10 0x7f205c065ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59268/0 (socket says 172.21.15.57:59268) 2023-10-30T14:46:52.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.900+0000 7f205bfff640 1 -- 172.21.15.57:0/309124229 learned_addr learned my addr 172.21.15.57:0/309124229 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:52.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.900+0000 7f205b7fe640 1 --2- 172.21.15.57:0/309124229 >> v2:172.21.15.88:3300/0 conn(0x7f205c070c30 0x7f205c071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:52.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.900+0000 7f205bfff640 1 -- 172.21.15.57:0/309124229 >> v2:172.21.15.88:3300/0 conn(0x7f205c070c30 msgr2=0x7f205c071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:52.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.900+0000 7f205bfff640 1 --2- 172.21.15.57:0/309124229 >> v2:172.21.15.88:3300/0 conn(0x7f205c070c30 0x7f205c071070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:52.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.900+0000 7f205bfff640 1 -- 172.21.15.57:0/309124229 >> v1:172.21.15.88:6789/0 conn(0x7f205c159970 legacy=0x7f205c159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:52.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.900+0000 7f205bfff640 1 -- 172.21.15.57:0/309124229 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205c0715b0 con 0x7f205c153c10 2023-10-30T14:46:52.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.900+0000 7f205b7fe640 1 --2- 172.21.15.57:0/309124229 >> v2:172.21.15.88:3300/0 conn(0x7f205c070c30 0x7f205c071070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:46:52.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.904+0000 7f205bfff640 1 --2- 172.21.15.57:0/309124229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f205c153c10 0x7f205c065ba0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f205c06f200 tx=0x7f204c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae911710b4f4eb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:52.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.904+0000 7f205affd640 1 -- 172.21.15.57:0/309124229 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204c0ab600 con 0x7f205c153c10 2023-10-30T14:46:52.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.904+0000 7f205affd640 1 -- 172.21.15.57:0/309124229 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204c0a7050 con 0x7f205c153c10 2023-10-30T14:46:52.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.904+0000 7f205affd640 1 -- 172.21.15.57:0/309124229 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204c0ab980 con 0x7f205c153c10 2023-10-30T14:46:52.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.904+0000 7f20626ea640 1 -- 172.21.15.57:0/309124229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f205c153c10 msgr2=0x7f205c065ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:52.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.904+0000 7f20626ea640 1 --2- 172.21.15.57:0/309124229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f205c153c10 0x7f205c065ba0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f205c06f200 tx=0x7f204c01c430 comp rx=0 tx=0).stop 2023-10-30T14:46:52.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.904+0000 7f20626ea640 1 -- 172.21.15.57:0/309124229 shutdown_connections 2023-10-30T14:46:52.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.904+0000 7f20626ea640 1 --2- 172.21.15.57:0/309124229 >> v2:172.21.15.88:3300/0 conn(0x7f205c070c30 0x7f205c071070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:52.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.904+0000 7f20626ea640 1 --2- 172.21.15.57:0/309124229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f205c153c10 0x7f205c065ba0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:52.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.904+0000 7f20626ea640 1 -- 172.21.15.57:0/309124229 >> 172.21.15.57:0/309124229 conn(0x7f205c0c7f60 msgr2=0x7f205c0c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:52.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.904+0000 7f20626ea640 1 -- 172.21.15.57:0/309124229 shutdown_connections 2023-10-30T14:46:52.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.904+0000 7f20626ea640 1 -- 172.21.15.57:0/309124229 wait complete. 2023-10-30T14:46:52.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.904+0000 7f20626ea640 1 Processor -- start 2023-10-30T14:46:52.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.904+0000 7f20626ea640 1 -- start start 2023-10-30T14:46:52.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f20626ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f205c070c30 0x7f205c0c2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:52.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f20626ea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f205c153c10 0x7f205c06d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:52.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f20626ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f205c159970 0x7f205c0ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:52.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f20626ea640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c0c3050 con 0x7f205c070c30 2023-10-30T14:46:52.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f20626ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c0c33f0 con 0x7f205c159970 2023-10-30T14:46:52.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f20626ea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205c0c93e0 con 0x7f205c153c10 2023-10-30T14:46:52.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f205bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f205c070c30 0x7f205c0c2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:52.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f205bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f205c070c30 0x7f205c0c2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59284/0 (socket says 172.21.15.57:59284) 2023-10-30T14:46:52.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f205bfff640 1 -- 172.21.15.57:0/1436364429 learned_addr learned my addr 172.21.15.57:0/1436364429 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:52.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f205b7fe640 1 --2- 172.21.15.57:0/1436364429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f205c153c10 0x7f205c06d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:52.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f2060c87640 1 --2- 172.21.15.57:0/1436364429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f205c159970 0x7f205c0ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:52.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f205bfff640 1 -- 172.21.15.57:0/1436364429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f205c153c10 msgr2=0x7f205c06d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:52.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f205bfff640 1 --2- 172.21.15.57:0/1436364429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f205c153c10 0x7f205c06d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:52.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f205bfff640 1 -- 172.21.15.57:0/1436364429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f205c159970 msgr2=0x7f205c0ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:52.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f205bfff640 1 --2- 172.21.15.57:0/1436364429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f205c159970 0x7f205c0ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:52.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f205bfff640 1 -- 172.21.15.57:0/1436364429 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204c0094d0 con 0x7f205c070c30 2023-10-30T14:46:52.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f2060c87640 1 --2- 172.21.15.57:0/1436364429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f205c159970 0x7f205c0ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:46:52.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f205bfff640 1 --2- 172.21.15.57:0/1436364429 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f205c070c30 0x7f205c0c2b10 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f204c01a060 tx=0x7f204c0191d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:52.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f2058ff9640 1 -- 172.21.15.57:0/1436364429 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204c0b4640 con 0x7f205c070c30 2023-10-30T14:46:52.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f2058ff9640 1 -- 172.21.15.57:0/1436364429 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204c0b47a0 con 0x7f205c070c30 2023-10-30T14:46:52.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f2058ff9640 1 -- 172.21.15.57:0/1436364429 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204c0b4b30 con 0x7f205c070c30 2023-10-30T14:46:52.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f20626ea640 1 -- 172.21.15.57:0/1436364429 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f205c0c9c80 con 0x7f205c070c30 2023-10-30T14:46:52.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f20626ea640 1 -- 172.21.15.57:0/1436364429 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f205c1f6090 con 0x7f205c070c30 2023-10-30T14:46:52.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f20626ea640 1 -- 172.21.15.57:0/1436364429 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f205c04e170 con 0x7f205c070c30 2023-10-30T14:46:52.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.908+0000 7f2058ff9640 1 -- 172.21.15.57:0/1436364429 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f204c0b2070 con 0x7f205c070c30 2023-10-30T14:46:52.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.912+0000 7f2058ff9640 1 --2- 172.21.15.57:0/1436364429 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20480664a0 0x7f2048068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:52.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.912+0000 7f2058ff9640 1 -- 172.21.15.57:0/1436364429 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f204c11e470 con 0x7f205c070c30 2023-10-30T14:46:52.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.912+0000 7f205b7fe640 1 --2- 172.21.15.57:0/1436364429 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20480664a0 0x7f2048068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:52.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.912+0000 7f205b7fe640 1 --2- 172.21.15.57:0/1436364429 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20480664a0 0x7f2048068960 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f205c14c070 tx=0x7f2050008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:52.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:52.916+0000 7f2058ff9640 1 -- 172.21.15.57:0/1436364429 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f204c0eb270 con 0x7f205c070c30 2023-10-30T14:46:53.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:53.108+0000 7f20626ea640 1 -- 172.21.15.57:0/1436364429 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f205c071310 con 0x7f205c070c30 2023-10-30T14:46:53.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:53.108+0000 7f2058ff9640 1 -- 172.21.15.57:0/1436364429 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+6494 (secure 0 0 0) 0x7f204c0ee890 con 0x7f205c070c30 2023-10-30T14:46:53.113 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:53.113 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7851,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/59168182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":59168182},{"type":"v1","addr":"172.21.15.57:6836","nonce":59168182}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:46:37.266044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:53.113 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 176 2023-10-30T14:46:53.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:53.108+0000 7f20626ea640 1 -- 172.21.15.57:0/1436364429 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20480664a0 msgr2=0x7f2048068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:53.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:53.108+0000 7f20626ea640 1 --2- 172.21.15.57:0/1436364429 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20480664a0 0x7f2048068960 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f205c14c070 tx=0x7f2050008040 comp rx=0 tx=0).stop 2023-10-30T14:46:53.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:53.108+0000 7f20626ea640 1 -- 172.21.15.57:0/1436364429 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f205c070c30 msgr2=0x7f205c0c2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:53.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:53.108+0000 7f20626ea640 1 --2- 172.21.15.57:0/1436364429 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f205c070c30 0x7f205c0c2b10 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f204c01a060 tx=0x7f204c0191d0 comp rx=0 tx=0).stop 2023-10-30T14:46:53.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:53.108+0000 7f20626ea640 1 -- 172.21.15.57:0/1436364429 shutdown_connections 2023-10-30T14:46:53.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:53.108+0000 7f20626ea640 1 --2- 172.21.15.57:0/1436364429 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f20480664a0 0x7f2048068960 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:53.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:53.108+0000 7f20626ea640 1 --2- 172.21.15.57:0/1436364429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f205c159970 0x7f205c0ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:53.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:53.108+0000 7f20626ea640 1 --2- 172.21.15.57:0/1436364429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f205c153c10 0x7f205c06d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:53.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:53.108+0000 7f20626ea640 1 --2- 172.21.15.57:0/1436364429 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f205c070c30 0x7f205c0c2b10 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:53.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:53.108+0000 7f20626ea640 1 -- 172.21.15.57:0/1436364429 >> 172.21.15.57:0/1436364429 conn(0x7f205c0c7f60 msgr2=0x7f205c14f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:53.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:53.112+0000 7f20626ea640 1 -- 172.21.15.57:0/1436364429 shutdown_connections 2023-10-30T14:46:53.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:53.112+0000 7f20626ea640 1 -- 172.21.15.57:0/1436364429 wait complete. 2023-10-30T14:46:53.135 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:46:55.136 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:55.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.276+0000 7fc1a053e640 1 Processor -- start 2023-10-30T14:46:55.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.276+0000 7fc1a053e640 1 -- start start 2023-10-30T14:46:55.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.276+0000 7fc1a053e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc198153c80 0x7fc19815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:55.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.276+0000 7fc1a053e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc1980c42a0 0x7fc1980ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:55.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.276+0000 7fc1a053e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc198159f20 con 0x7fc1980c4a10 2023-10-30T14:46:55.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.276+0000 7fc1a053e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc19815dec0 con 0x7fc1980c42a0 2023-10-30T14:46:55.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.276+0000 7fc1a053e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1981599a0 con 0x7fc198153c80 2023-10-30T14:46:55.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.276+0000 7fc19eadb640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc1980c42a0 0x7fc1980ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:55.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.276+0000 7fc19eadb640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc1980c42a0 0x7fc1980ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45066/0 (socket says 172.21.15.57:45066) 2023-10-30T14:46:55.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.276+0000 7fc19eadb640 1 -- 172.21.15.57:0/1389681005 learned_addr learned my addr 172.21.15.57:0/1389681005 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:55.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.276+0000 7fc19e2da640 1 --2- 172.21.15.57:0/1389681005 >> v2:172.21.15.88:3300/0 conn(0x7fc198153c80 0x7fc19815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:55.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.276+0000 7fc19eadb640 1 -- 172.21.15.57:0/1389681005 >> v2:172.21.15.88:3300/0 conn(0x7fc198153c80 msgr2=0x7fc19815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:55.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.276+0000 7fc19eadb640 1 --2- 172.21.15.57:0/1389681005 >> v2:172.21.15.88:3300/0 conn(0x7fc198153c80 0x7fc19815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:55.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.276+0000 7fc19eadb640 1 -- 172.21.15.57:0/1389681005 >> v1:172.21.15.88:6789/0 conn(0x7fc1980c4a10 legacy=0x7fc19815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:55.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.276+0000 7fc19eadb640 1 -- 172.21.15.57:0/1389681005 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19815d620 con 0x7fc1980c42a0 2023-10-30T14:46:55.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.280+0000 7fc19eadb640 1 --2- 172.21.15.57:0/1389681005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1980c42a0 0x7fc1980ca1e0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fc1980d00f0 tx=0x7fc19001f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359e3d6b792c7ddf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:55.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.280+0000 7fc19d2d8640 1 -- 172.21.15.57:0/1389681005 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1900af400 con 0x7fc1980c42a0 2023-10-30T14:46:55.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.280+0000 7fc19d2d8640 1 -- 172.21.15.57:0/1389681005 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1900ab050 con 0x7fc1980c42a0 2023-10-30T14:46:55.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.280+0000 7fc19d2d8640 1 -- 172.21.15.57:0/1389681005 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1900af780 con 0x7fc1980c42a0 2023-10-30T14:46:55.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.280+0000 7fc1a053e640 1 -- 172.21.15.57:0/1389681005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1980c42a0 msgr2=0x7fc1980ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:55.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.280+0000 7fc1a053e640 1 --2- 172.21.15.57:0/1389681005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1980c42a0 0x7fc1980ca1e0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fc1980d00f0 tx=0x7fc19001f180 comp rx=0 tx=0).stop 2023-10-30T14:46:55.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.280+0000 7fc1a053e640 1 -- 172.21.15.57:0/1389681005 shutdown_connections 2023-10-30T14:46:55.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.280+0000 7fc1a053e640 1 --2- 172.21.15.57:0/1389681005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1980c42a0 0x7fc1980ca1e0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:55.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.280+0000 7fc1a053e640 1 --2- 172.21.15.57:0/1389681005 >> v2:172.21.15.88:3300/0 conn(0x7fc198153c80 0x7fc19815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:55.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.280+0000 7fc1a053e640 1 -- 172.21.15.57:0/1389681005 >> 172.21.15.57:0/1389681005 conn(0x7fc1980715c0 msgr2=0x7fc1980719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:55.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.280+0000 7fc1a053e640 1 -- 172.21.15.57:0/1389681005 shutdown_connections 2023-10-30T14:46:55.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.280+0000 7fc1a053e640 1 -- 172.21.15.57:0/1389681005 wait complete. 2023-10-30T14:46:55.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc1a053e640 1 Processor -- start 2023-10-30T14:46:55.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc1a053e640 1 -- start start 2023-10-30T14:46:55.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc1a053e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1980c42a0 0x7fc198152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:55.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc1a053e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1980c4a10 0x7fc19815ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:55.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc1a053e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc198153c80 0x7fc198149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:55.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc1a053e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1980b35b0 con 0x7fc198153c80 2023-10-30T14:46:55.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc1a053e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19815c420 con 0x7fc1980c4a10 2023-10-30T14:46:55.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc1a053e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1980b37c0 con 0x7fc1980c42a0 2023-10-30T14:46:55.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc19eadb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc198153c80 0x7fc198149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:55.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc19eadb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc198153c80 0x7fc198149b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45068/0 (socket says 172.21.15.57:45068) 2023-10-30T14:46:55.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc19eadb640 1 -- 172.21.15.57:0/2715550233 learned_addr learned my addr 172.21.15.57:0/2715550233 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:55.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc19dad9640 1 --2- 172.21.15.57:0/2715550233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1980c4a10 0x7fc19815ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:55.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc19e2da640 1 --2- 172.21.15.57:0/2715550233 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1980c42a0 0x7fc198152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:55.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc19eadb640 1 -- 172.21.15.57:0/2715550233 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1980c42a0 msgr2=0x7fc198152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:55.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc19eadb640 1 --2- 172.21.15.57:0/2715550233 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1980c42a0 0x7fc198152850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:55.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc19eadb640 1 -- 172.21.15.57:0/2715550233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1980c4a10 msgr2=0x7fc19815ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:55.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc19eadb640 1 --2- 172.21.15.57:0/2715550233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1980c4a10 0x7fc19815ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:55.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc19eadb640 1 -- 172.21.15.57:0/2715550233 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc190009b30 con 0x7fc198153c80 2023-10-30T14:46:55.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc19eadb640 1 --2- 172.21.15.57:0/2715550233 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc198153c80 0x7fc198149b30 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fc1900b2040 tx=0x7fc19001bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:55.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc18effd640 1 -- 172.21.15.57:0/2715550233 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1900afdc0 con 0x7fc198153c80 2023-10-30T14:46:55.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc18effd640 1 -- 172.21.15.57:0/2715550233 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc190009510 con 0x7fc198153c80 2023-10-30T14:46:55.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc18effd640 1 -- 172.21.15.57:0/2715550233 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1900b8640 con 0x7fc198153c80 2023-10-30T14:46:55.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc1a053e640 1 -- 172.21.15.57:0/2715550233 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1980b3aa0 con 0x7fc198153c80 2023-10-30T14:46:55.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.284+0000 7fc1a053e640 1 -- 172.21.15.57:0/2715550233 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1980b4060 con 0x7fc198153c80 2023-10-30T14:46:55.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.288+0000 7fc1a053e640 1 -- 172.21.15.57:0/2715550233 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc198073c30 con 0x7fc198153c80 2023-10-30T14:46:55.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.288+0000 7fc18effd640 1 -- 172.21.15.57:0/2715550233 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc1900b6040 con 0x7fc198153c80 2023-10-30T14:46:55.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.288+0000 7fc18effd640 1 --2- 172.21.15.57:0/2715550233 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc17c0664a0 0x7fc17c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:55.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.288+0000 7fc18effd640 1 -- 172.21.15.57:0/2715550233 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7fc190122370 con 0x7fc198153c80 2023-10-30T14:46:55.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.288+0000 7fc19e2da640 1 --2- 172.21.15.57:0/2715550233 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc17c0664a0 0x7fc17c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:55.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.288+0000 7fc19e2da640 1 --2- 172.21.15.57:0/2715550233 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc17c0664a0 0x7fc17c068960 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc1940079c0 tx=0x7fc194008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:55.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.292+0000 7fc18effd640 1 -- 172.21.15.57:0/2715550233 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc1900ef220 con 0x7fc198153c80 2023-10-30T14:46:55.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.496+0000 7fc1a053e640 1 -- 172.21.15.57:0/2715550233 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1980b4250 con 0x7fc198153c80 2023-10-30T14:46:55.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.496+0000 7fc18effd640 1 -- 172.21.15.57:0/2715550233 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+5708 (secure 0 0 0) 0x7fc1900f2840 con 0x7fc198153c80 2023-10-30T14:46:55.524 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:55.525 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:46:37.266044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:55.527 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 177 2023-10-30T14:46:55.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.496+0000 7fc1a053e640 1 -- 172.21.15.57:0/2715550233 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc17c0664a0 msgr2=0x7fc17c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:55.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.496+0000 7fc1a053e640 1 --2- 172.21.15.57:0/2715550233 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc17c0664a0 0x7fc17c068960 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc1940079c0 tx=0x7fc194008040 comp rx=0 tx=0).stop 2023-10-30T14:46:55.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.496+0000 7fc1a053e640 1 -- 172.21.15.57:0/2715550233 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc198153c80 msgr2=0x7fc198149b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:55.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.496+0000 7fc1a053e640 1 --2- 172.21.15.57:0/2715550233 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc198153c80 0x7fc198149b30 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fc1900b2040 tx=0x7fc19001bf20 comp rx=0 tx=0).stop 2023-10-30T14:46:55.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.496+0000 7fc1a053e640 1 -- 172.21.15.57:0/2715550233 shutdown_connections 2023-10-30T14:46:55.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.496+0000 7fc1a053e640 1 --2- 172.21.15.57:0/2715550233 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc17c0664a0 0x7fc17c068960 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:55.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.496+0000 7fc1a053e640 1 --2- 172.21.15.57:0/2715550233 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc198153c80 0x7fc198149b30 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:55.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.496+0000 7fc1a053e640 1 --2- 172.21.15.57:0/2715550233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1980c4a10 0x7fc19815ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:55.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.496+0000 7fc1a053e640 1 --2- 172.21.15.57:0/2715550233 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1980c42a0 0x7fc198152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:55.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.496+0000 7fc1a053e640 1 -- 172.21.15.57:0/2715550233 >> 172.21.15.57:0/2715550233 conn(0x7fc1980715c0 msgr2=0x7fc1981575b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:55.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.496+0000 7fc1a053e640 1 -- 172.21.15.57:0/2715550233 shutdown_connections 2023-10-30T14:46:55.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.496+0000 7fc1a053e640 1 -- 172.21.15.57:0/2715550233 wait complete. 2023-10-30T14:46:55.530 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T14:46:55.531 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:46:55.531 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:46:55.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.672+0000 7f64a48d6640 1 Processor -- start 2023-10-30T14:46:55.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.672+0000 7f64a48d6640 1 -- start start 2023-10-30T14:46:55.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.676+0000 7f64a48d6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f64a0153c80 0x7f64a015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:55.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.676+0000 7f64a48d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64a00c4a10 0x7f64a00c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:55.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.676+0000 7f64a48d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a0159f20 con 0x7f64a015a3b0 2023-10-30T14:46:55.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.676+0000 7f64a48d6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a015dec0 con 0x7f64a0153c80 2023-10-30T14:46:55.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.676+0000 7f64a48d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a01599a0 con 0x7f64a00c4a10 2023-10-30T14:46:55.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.676+0000 7f649e59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f64a0153c80 0x7f64a015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:55.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.676+0000 7f649e59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f64a0153c80 0x7f64a015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45080/0 (socket says 172.21.15.57:45080) 2023-10-30T14:46:55.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.676+0000 7f649e59c640 1 -- 172.21.15.57:0/1901012600 learned_addr learned my addr 172.21.15.57:0/1901012600 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:55.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.676+0000 7f649dd9b640 1 --2- 172.21.15.57:0/1901012600 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64a00c4a10 0x7f64a00c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:55.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.676+0000 7f649e59c640 1 -- 172.21.15.57:0/1901012600 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64a00c4a10 msgr2=0x7f64a00c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:46:55.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.676+0000 7f649e59c640 1 --2- 172.21.15.57:0/1901012600 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64a00c4a10 0x7f64a00c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:55.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.676+0000 7f649e59c640 1 -- 172.21.15.57:0/1901012600 >> v1:172.21.15.88:6789/0 conn(0x7f64a015a3b0 legacy=0x7f64a00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:55.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.676+0000 7f649e59c640 1 -- 172.21.15.57:0/1901012600 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a00c47e0 con 0x7f64a0153c80 2023-10-30T14:46:55.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.680+0000 7f649e59c640 1 --2- 172.21.15.57:0/1901012600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64a0153c80 0x7f64a015cf90 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f6490009870 tx=0x7f649001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d2701f163602f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:55.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.680+0000 7f649d59a640 1 -- 172.21.15.57:0/1901012600 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64900a8a30 con 0x7f64a0153c80 2023-10-30T14:46:55.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.680+0000 7f649d59a640 1 -- 172.21.15.57:0/1901012600 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64900a8b90 con 0x7f64a0153c80 2023-10-30T14:46:55.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.680+0000 7f649d59a640 1 -- 172.21.15.57:0/1901012600 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64900b0440 con 0x7f64a0153c80 2023-10-30T14:46:55.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.680+0000 7f64a48d6640 1 -- 172.21.15.57:0/1901012600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64a0153c80 msgr2=0x7f64a015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:55.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.680+0000 7f64a48d6640 1 --2- 172.21.15.57:0/1901012600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64a0153c80 0x7f64a015cf90 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f6490009870 tx=0x7f649001ed30 comp rx=0 tx=0).stop 2023-10-30T14:46:55.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.680+0000 7f64a48d6640 1 -- 172.21.15.57:0/1901012600 shutdown_connections 2023-10-30T14:46:55.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.680+0000 7f64a48d6640 1 --2- 172.21.15.57:0/1901012600 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64a00c4a10 0x7f64a00c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:55.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.680+0000 7f64a48d6640 1 --2- 172.21.15.57:0/1901012600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64a0153c80 0x7f64a015cf90 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:55.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.680+0000 7f64a48d6640 1 -- 172.21.15.57:0/1901012600 >> 172.21.15.57:0/1901012600 conn(0x7f64a00715c0 msgr2=0x7f64a00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:55.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.680+0000 7f64a48d6640 1 -- 172.21.15.57:0/1901012600 shutdown_connections 2023-10-30T14:46:55.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.680+0000 7f64a48d6640 1 -- 172.21.15.57:0/1901012600 wait complete. 2023-10-30T14:46:55.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.684+0000 7f64a48d6640 1 Processor -- start 2023-10-30T14:46:55.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.684+0000 7f64a48d6640 1 -- start start 2023-10-30T14:46:55.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.684+0000 7f64a48d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64a00c4a10 0x7f64a0149fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:55.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.684+0000 7f64a48d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a0153c80 0x7f64a00c8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:55.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.684+0000 7f64a48d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64a015a3b0 0x7f64a00cf270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:55.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.684+0000 7f64a48d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a00b2870 con 0x7f64a00c4a10 2023-10-30T14:46:55.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.684+0000 7f64a48d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a00b20b0 con 0x7f64a0153c80 2023-10-30T14:46:55.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.684+0000 7f64a48d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a01527a0 con 0x7f64a015a3b0 2023-10-30T14:46:55.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.684+0000 7f649e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64a00c4a10 0x7f64a0149fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:55.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.684+0000 7f649e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64a00c4a10 0x7f64a0149fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45088/0 (socket says 172.21.15.57:45088) 2023-10-30T14:46:55.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.684+0000 7f649e59c640 1 -- 172.21.15.57:0/4275901214 learned_addr learned my addr 172.21.15.57:0/4275901214 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:46:55.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.684+0000 7f649ed9d640 1 --2- 172.21.15.57:0/4275901214 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64a015a3b0 0x7f64a00cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:55.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.684+0000 7f649dd9b640 1 --2- 172.21.15.57:0/4275901214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a0153c80 0x7f64a00c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:55.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.684+0000 7f649e59c640 1 -- 172.21.15.57:0/4275901214 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64a015a3b0 msgr2=0x7f64a00cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:55.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.684+0000 7f649e59c640 1 --2- 172.21.15.57:0/4275901214 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64a015a3b0 0x7f64a00cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:55.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.688+0000 7f649e59c640 1 -- 172.21.15.57:0/4275901214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a0153c80 msgr2=0x7f64a00c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:55.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.688+0000 7f649e59c640 1 --2- 172.21.15.57:0/4275901214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a0153c80 0x7f64a00c8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:55.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.688+0000 7f649e59c640 1 -- 172.21.15.57:0/4275901214 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64900094d0 con 0x7f64a00c4a10 2023-10-30T14:46:55.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.688+0000 7f649dd9b640 1 --2- 172.21.15.57:0/4275901214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a0153c80 0x7f64a00c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:46:55.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.688+0000 7f649e59c640 1 --2- 172.21.15.57:0/4275901214 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64a00c4a10 0x7f64a0149fa0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f64900b3040 tx=0x7f649001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:55.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.688+0000 7f648effd640 1 -- 172.21.15.57:0/4275901214 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64900b0dd0 con 0x7f64a00c4a10 2023-10-30T14:46:55.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.688+0000 7f648effd640 1 -- 172.21.15.57:0/4275901214 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64900bb460 con 0x7f64a00c4a10 2023-10-30T14:46:55.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.688+0000 7f648effd640 1 -- 172.21.15.57:0/4275901214 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64900bb810 con 0x7f64a00c4a10 2023-10-30T14:46:55.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.688+0000 7f64a48d6640 1 -- 172.21.15.57:0/4275901214 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a0151b90 con 0x7f64a00c4a10 2023-10-30T14:46:55.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.688+0000 7f64a48d6640 1 -- 172.21.15.57:0/4275901214 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a0152150 con 0x7f64a00c4a10 2023-10-30T14:46:55.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.688+0000 7f64a48d6640 1 -- 172.21.15.57:0/4275901214 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a004e170 con 0x7f64a00c4a10 2023-10-30T14:46:55.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.688+0000 7f648effd640 1 -- 172.21.15.57:0/4275901214 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f64900b83f0 con 0x7f64a00c4a10 2023-10-30T14:46:55.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.688+0000 7f648effd640 1 --2- 172.21.15.57:0/4275901214 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f64800664a0 0x7f6480068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:46:55.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.688+0000 7f648effd640 1 -- 172.21.15.57:0/4275901214 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f6490121d40 con 0x7f64a00c4a10 2023-10-30T14:46:55.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.692+0000 7f649dd9b640 1 --2- 172.21.15.57:0/4275901214 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f64800664a0 0x7f6480068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:46:55.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.692+0000 7f649dd9b640 1 --2- 172.21.15.57:0/4275901214 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f64800664a0 0x7f6480068960 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f64a01519a0 tx=0x7f6494008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:46:55.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.696+0000 7f648effd640 1 -- 172.21.15.57:0/4275901214 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f64900eebe0 con 0x7f64a00c4a10 2023-10-30T14:46:55.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.884+0000 7f64a48d6640 1 -- 172.21.15.57:0/4275901214 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64a00cad40 con 0x7f64a00c4a10 2023-10-30T14:46:55.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.884+0000 7f648effd640 1 -- 172.21.15.57:0/4275901214 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+5708 (secure 0 0 0) 0x7f64900b5090 con 0x7f64a00c4a10 2023-10-30T14:46:55.891 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:46:55.891 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:46:37.266044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:46:55.892 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 177 2023-10-30T14:46:55.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.888+0000 7f64a48d6640 1 -- 172.21.15.57:0/4275901214 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f64800664a0 msgr2=0x7f6480068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:55.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.888+0000 7f64a48d6640 1 --2- 172.21.15.57:0/4275901214 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f64800664a0 0x7f6480068960 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f64a01519a0 tx=0x7f6494008040 comp rx=0 tx=0).stop 2023-10-30T14:46:55.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.888+0000 7f64a48d6640 1 -- 172.21.15.57:0/4275901214 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64a00c4a10 msgr2=0x7f64a0149fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:46:55.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.888+0000 7f64a48d6640 1 --2- 172.21.15.57:0/4275901214 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64a00c4a10 0x7f64a0149fa0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f64900b3040 tx=0x7f649001bad0 comp rx=0 tx=0).stop 2023-10-30T14:46:55.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.888+0000 7f64a48d6640 1 -- 172.21.15.57:0/4275901214 shutdown_connections 2023-10-30T14:46:55.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.888+0000 7f64a48d6640 1 --2- 172.21.15.57:0/4275901214 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f64800664a0 0x7f6480068960 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:55.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.888+0000 7f64a48d6640 1 --2- 172.21.15.57:0/4275901214 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64a015a3b0 0x7f64a00cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:55.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.888+0000 7f64a48d6640 1 --2- 172.21.15.57:0/4275901214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a0153c80 0x7f64a00c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:55.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.888+0000 7f64a48d6640 1 --2- 172.21.15.57:0/4275901214 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64a00c4a10 0x7f64a0149fa0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:46:55.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.888+0000 7f64a48d6640 1 -- 172.21.15.57:0/4275901214 >> 172.21.15.57:0/4275901214 conn(0x7f64a00715c0 msgr2=0x7f64a00cb460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:46:55.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.888+0000 7f64a48d6640 1 -- 172.21.15.57:0/4275901214 shutdown_connections 2023-10-30T14:46:55.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:46:55.888+0000 7f64a48d6640 1 -- 172.21.15.57:0/4275901214 wait complete. 2023-10-30T14:46:55.911 INFO:tasks.mds_thrash.fs.[cephfs]:cluster has 1 actives (max_mds is 1), no MDS can replace rank 1 2023-10-30T14:46:55.911 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.c 2023-10-30T14:47:02.911 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T14:47:02.912 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T14:47:02.913 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T14:47:02.917 INFO:tasks.ceph.mds.c:Started 2023-10-30T14:47:02.918 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:47:03.013 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-30T14:47:03.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.068+0000 7fa519fd3640 1 Processor -- start 2023-10-30T14:47:03.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.068+0000 7fa519fd3640 1 -- start start 2023-10-30T14:47:03.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.068+0000 7fa519fd3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa514153c10 0x7fa514065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:47:03.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.068+0000 7fa519fd3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa514070c30 0x7fa514071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:47:03.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.068+0000 7fa519fd3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa51406e390 con 0x7fa514159970 2023-10-30T14:47:03.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.068+0000 7fa519fd3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa514072390 con 0x7fa514153c10 2023-10-30T14:47:03.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.068+0000 7fa519fd3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5140660e0 con 0x7fa514070c30 2023-10-30T14:47:03.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.072+0000 7fa5137fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa514153c10 0x7fa514065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:47:03.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.072+0000 7fa513fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa514159970 0x7fa514159dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:56080/0 (socket says 172.21.15.57:56080) 2023-10-30T14:47:03.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.072+0000 7fa513fff640 1 -- 172.21.15.57:0/4241509634 learned_addr learned my addr 172.21.15.57:0/4241509634 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:47:03.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.072+0000 7fa512ffd640 1 --2- 172.21.15.57:0/4241509634 >> v2:172.21.15.88:3300/0 conn(0x7fa514070c30 0x7fa514071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:47:03.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.072+0000 7fa5137fe640 1 -- 172.21.15.57:0/4241509634 >> v2:172.21.15.88:3300/0 conn(0x7fa514070c30 msgr2=0x7fa514071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:47:03.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.072+0000 7fa5137fe640 1 --2- 172.21.15.57:0/4241509634 >> v2:172.21.15.88:3300/0 conn(0x7fa514070c30 0x7fa514071070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:47:03.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.072+0000 7fa5137fe640 1 -- 172.21.15.57:0/4241509634 >> v1:172.21.15.57:6789/0 conn(0x7fa514159970 legacy=0x7fa514159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:47:03.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.072+0000 7fa5137fe640 1 -- 172.21.15.57:0/4241509634 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5140715b0 con 0x7fa514153c10 2023-10-30T14:47:03.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.072+0000 7fa5127fc640 1 -- 172.21.15.57:0/4241509634 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975436887 0 0) 0x7fa51406e390 con 0x7fa514159970 2023-10-30T14:47:03.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.072+0000 7fa5137fe640 1 --2- 172.21.15.57:0/4241509634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa514153c10 0x7fa514065ba0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fa51406f200 tx=0x7fa50401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9510fea2c9396891 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:47:03.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.072+0000 7fa5127fc640 1 -- 172.21.15.57:0/4241509634 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5040ab600 con 0x7fa514153c10 2023-10-30T14:47:03.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.072+0000 7fa5127fc640 1 -- 172.21.15.57:0/4241509634 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5040a7050 con 0x7fa514153c10 2023-10-30T14:47:03.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa5127fc640 1 -- 172.21.15.57:0/4241509634 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5040ab980 con 0x7fa514153c10 2023-10-30T14:47:03.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa519fd3640 1 -- 172.21.15.57:0/4241509634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa514153c10 msgr2=0x7fa514065ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:47:03.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa519fd3640 1 --2- 172.21.15.57:0/4241509634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa514153c10 0x7fa514065ba0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fa51406f200 tx=0x7fa50401c3d0 comp rx=0 tx=0).stop 2023-10-30T14:47:03.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa519fd3640 1 -- 172.21.15.57:0/4241509634 shutdown_connections 2023-10-30T14:47:03.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa519fd3640 1 --2- 172.21.15.57:0/4241509634 >> v2:172.21.15.88:3300/0 conn(0x7fa514070c30 0x7fa514071070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:47:03.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa519fd3640 1 --2- 172.21.15.57:0/4241509634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa514153c10 0x7fa514065ba0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:47:03.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa519fd3640 1 -- 172.21.15.57:0/4241509634 >> 172.21.15.57:0/4241509634 conn(0x7fa5140c7f60 msgr2=0x7fa5140c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:47:03.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa519fd3640 1 -- 172.21.15.57:0/4241509634 shutdown_connections 2023-10-30T14:47:03.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa519fd3640 1 -- 172.21.15.57:0/4241509634 wait complete. 2023-10-30T14:47:03.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa519fd3640 1 Processor -- start 2023-10-30T14:47:03.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa519fd3640 1 -- start start 2023-10-30T14:47:03.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa519fd3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa514070c30 0x7fa5140ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:47:03.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa519fd3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa514153c10 0x7fa5140caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:47:03.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa519fd3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa514159970 0x7fa51414da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:47:03.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa519fd3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5141f2040 con 0x7fa514159970 2023-10-30T14:47:03.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa519fd3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5141f2e60 con 0x7fa514070c30 2023-10-30T14:47:03.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa519fd3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5141f22e0 con 0x7fa514153c10 2023-10-30T14:47:03.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa5137fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa514070c30 0x7fa5140ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:47:03.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa5137fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa514070c30 0x7fa5140ca560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:36328/0 (socket says 172.21.15.57:36328) 2023-10-30T14:47:03.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa513fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa514159970 0x7fa51414da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:47:03.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.076+0000 7fa5137fe640 1 -- 172.21.15.57:0/3274147200 learned_addr learned my addr 172.21.15.57:0/3274147200 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:47:03.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.080+0000 7fa512ffd640 1 --2- 172.21.15.57:0/3274147200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa514153c10 0x7fa5140caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:47:03.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.080+0000 7fa513fff640 1 -- 172.21.15.57:0/3274147200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa514153c10 msgr2=0x7fa5140caaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:47:03.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.080+0000 7fa513fff640 1 --2- 172.21.15.57:0/3274147200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa514153c10 0x7fa5140caaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:47:03.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.080+0000 7fa513fff640 1 -- 172.21.15.57:0/3274147200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa514070c30 msgr2=0x7fa5140ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:47:03.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.080+0000 7fa513fff640 1 --2- 172.21.15.57:0/3274147200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa514070c30 0x7fa5140ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:47:03.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.080+0000 7fa513fff640 1 -- 172.21.15.57:0/3274147200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa504009420 con 0x7fa514159970 2023-10-30T14:47:03.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.080+0000 7fa513fff640 1 --2- 172.21.15.57:0/3274147200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa514159970 0x7fa51414da60 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fa50c00c780 tx=0x7fa50c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:47:03.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.080+0000 7fa4fbfff640 1 -- 172.21.15.57:0/3274147200 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50c00ce30 con 0x7fa514159970 2023-10-30T14:47:03.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.080+0000 7fa4fbfff640 1 -- 172.21.15.57:0/3274147200 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50c00a640 con 0x7fa514159970 2023-10-30T14:47:03.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.080+0000 7fa4fbfff640 1 -- 172.21.15.57:0/3274147200 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50c00a9f0 con 0x7fa514159970 2023-10-30T14:47:03.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.080+0000 7fa519fd3640 1 -- 172.21.15.57:0/3274147200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5141f25c0 con 0x7fa514159970 2023-10-30T14:47:03.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.080+0000 7fa519fd3640 1 -- 172.21.15.57:0/3274147200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5141f2b80 con 0x7fa514159970 2023-10-30T14:47:03.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.080+0000 7fa519fd3640 1 -- 172.21.15.57:0/3274147200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa51404e170 con 0x7fa514159970 2023-10-30T14:47:03.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.080+0000 7fa4fbfff640 1 -- 172.21.15.57:0/3274147200 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa50c00ab50 con 0x7fa514159970 2023-10-30T14:47:03.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.084+0000 7fa4fbfff640 1 --2- 172.21.15.57:0/3274147200 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa5000664d0 0x7fa500068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:47:03.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.084+0000 7fa4fbfff640 1 -- 172.21.15.57:0/3274147200 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7fa50c083870 con 0x7fa514159970 2023-10-30T14:47:03.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.084+0000 7fa5137fe640 1 --2- 172.21.15.57:0/3274147200 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa5000664d0 0x7fa500068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:47:03.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.084+0000 7fa5137fe640 1 --2- 172.21.15.57:0/3274147200 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa5000664d0 0x7fa500068990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fa504018fe0 tx=0x7fa5040a6000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:47:03.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.088+0000 7fa4fbfff640 1 -- 172.21.15.57:0/3274147200 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa50c050690 con 0x7fa514159970 2023-10-30T14:47:03.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.276+0000 7fa519fd3640 1 -- 172.21.15.57:0/3274147200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa514070150 con 0x7fa514159970 2023-10-30T14:47:03.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.280+0000 7fa4fbfff640 1 -- 172.21.15.57:0/3274147200 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+5708 (secure 0 0 0) 0x7fa50c053cb0 con 0x7fa514159970 2023-10-30T14:47:03.284 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:47:03.284 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:46:37.266044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:47:03.285 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 177 2023-10-30T14:47:03.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.280+0000 7fa519fd3640 1 -- 172.21.15.57:0/3274147200 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa5000664d0 msgr2=0x7fa500068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:47:03.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.280+0000 7fa519fd3640 1 --2- 172.21.15.57:0/3274147200 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa5000664d0 0x7fa500068990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fa504018fe0 tx=0x7fa5040a6000 comp rx=0 tx=0).stop 2023-10-30T14:47:03.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.280+0000 7fa519fd3640 1 -- 172.21.15.57:0/3274147200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa514159970 msgr2=0x7fa51414da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:47:03.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.280+0000 7fa519fd3640 1 --2- 172.21.15.57:0/3274147200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa514159970 0x7fa51414da60 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fa50c00c780 tx=0x7fa50c00cc40 comp rx=0 tx=0).stop 2023-10-30T14:47:03.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.280+0000 7fa519fd3640 1 -- 172.21.15.57:0/3274147200 shutdown_connections 2023-10-30T14:47:03.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.280+0000 7fa519fd3640 1 --2- 172.21.15.57:0/3274147200 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa5000664d0 0x7fa500068990 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:47:03.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.280+0000 7fa519fd3640 1 --2- 172.21.15.57:0/3274147200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa514159970 0x7fa51414da60 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:47:03.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.280+0000 7fa519fd3640 1 --2- 172.21.15.57:0/3274147200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa514153c10 0x7fa5140caaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:47:03.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.280+0000 7fa519fd3640 1 --2- 172.21.15.57:0/3274147200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa514070c30 0x7fa5140ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:47:03.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.280+0000 7fa519fd3640 1 -- 172.21.15.57:0/3274147200 >> 172.21.15.57:0/3274147200 conn(0x7fa5140c7f60 msgr2=0x7fa5141532a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:47:03.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.284+0000 7fa519fd3640 1 -- 172.21.15.57:0/3274147200 shutdown_connections 2023-10-30T14:47:03.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:03.284+0000 7fa519fd3640 1 -- 172.21.15.57:0/3274147200 wait complete. 2023-10-30T14:47:03.308 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T14:47:05.309 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:47:05.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.464+0000 7f0c808c9640 1 Processor -- start 2023-10-30T14:47:05.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.464+0000 7f0c808c9640 1 -- start start 2023-10-30T14:47:05.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.464+0000 7f0c808c9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0c7c153c80 0x7f0c7c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:47:05.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.464+0000 7f0c808c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0c7c0c42a0 0x7f0c7c0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:47:05.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.464+0000 7f0c808c9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c7c159f20 con 0x7f0c7c0c4a10 2023-10-30T14:47:05.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.464+0000 7f0c808c9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c7c15dec0 con 0x7f0c7c153c80 2023-10-30T14:47:05.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.464+0000 7f0c808c9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c7c1599a0 con 0x7f0c7c0c42a0 2023-10-30T14:47:05.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.464+0000 7f0c7a59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0c7c153c80 0x7f0c7c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:47:05.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.464+0000 7f0c73fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f0c7c0c4a10 0x7f0c7c15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:35074/0 (socket says 172.21.15.57:35074) 2023-10-30T14:47:05.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.464+0000 7f0c73fff640 1 -- 172.21.15.57:0/3629417748 learned_addr learned my addr 172.21.15.57:0/3629417748 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:47:05.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.464+0000 7f0c7ad9d640 1 --2- 172.21.15.57:0/3629417748 >> v2:172.21.15.88:3300/0 conn(0x7f0c7c0c42a0 0x7f0c7c0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:47:05.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.464+0000 7f0c7a59c640 1 -- 172.21.15.57:0/3629417748 >> v2:172.21.15.88:3300/0 conn(0x7f0c7c0c42a0 msgr2=0x7f0c7c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:47:05.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.464+0000 7f0c7a59c640 1 --2- 172.21.15.57:0/3629417748 >> v2:172.21.15.88:3300/0 conn(0x7f0c7c0c42a0 0x7f0c7c0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:47:05.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.464+0000 7f0c7a59c640 1 -- 172.21.15.57:0/3629417748 >> v1:172.21.15.57:6789/0 conn(0x7f0c7c0c4a10 legacy=0x7f0c7c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:47:05.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.464+0000 7f0c7a59c640 1 -- 172.21.15.57:0/3629417748 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7c15d620 con 0x7f0c7c153c80 2023-10-30T14:47:05.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.464+0000 7f0c79d9b640 1 -- 172.21.15.57:0/3629417748 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245189217 0 0) 0x7f0c7c159f20 con 0x7f0c7c0c4a10 2023-10-30T14:47:05.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.464+0000 7f0c7ad9d640 1 --2- 172.21.15.57:0/3629417748 >> v2:172.21.15.88:3300/0 conn(0x7f0c7c0c42a0 0x7f0c7c0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:47:05.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.468+0000 7f0c7a59c640 1 --2- 172.21.15.57:0/3629417748 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c7c153c80 0x7f0c7c15cf90 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c0b1d40 tx=0x7f0c6c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c408a3ef182e80 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:47:05.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.468+0000 7f0c79d9b640 1 -- 172.21.15.57:0/3629417748 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c0a8950 con 0x7f0c7c153c80 2023-10-30T14:47:05.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.468+0000 7f0c79d9b640 1 -- 172.21.15.57:0/3629417748 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6c0a8ab0 con 0x7f0c7c153c80 2023-10-30T14:47:05.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.468+0000 7f0c79d9b640 1 -- 172.21.15.57:0/3629417748 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c0a8e40 con 0x7f0c7c153c80 2023-10-30T14:47:05.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.468+0000 7f0c808c9640 1 -- 172.21.15.57:0/3629417748 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c7c153c80 msgr2=0x7f0c7c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:47:05.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.468+0000 7f0c808c9640 1 --2- 172.21.15.57:0/3629417748 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c7c153c80 0x7f0c7c15cf90 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c0b1d40 tx=0x7f0c6c01ec50 comp rx=0 tx=0).stop 2023-10-30T14:47:05.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.468+0000 7f0c808c9640 1 -- 172.21.15.57:0/3629417748 shutdown_connections 2023-10-30T14:47:05.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.468+0000 7f0c808c9640 1 --2- 172.21.15.57:0/3629417748 >> v2:172.21.15.88:3300/0 conn(0x7f0c7c0c42a0 0x7f0c7c0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:47:05.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.468+0000 7f0c808c9640 1 --2- 172.21.15.57:0/3629417748 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c7c153c80 0x7f0c7c15cf90 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:47:05.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.468+0000 7f0c808c9640 1 -- 172.21.15.57:0/3629417748 >> 172.21.15.57:0/3629417748 conn(0x7f0c7c0715c0 msgr2=0x7f0c7c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:47:05.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.468+0000 7f0c808c9640 1 -- 172.21.15.57:0/3629417748 shutdown_connections 2023-10-30T14:47:05.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.468+0000 7f0c808c9640 1 -- 172.21.15.57:0/3629417748 wait complete. 2023-10-30T14:47:05.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c808c9640 1 Processor -- start 2023-10-30T14:47:05.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c808c9640 1 -- start start 2023-10-30T14:47:05.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c808c9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c7c0c42a0 0x7f0c7c1ef350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:47:05.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c808c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c7c0c4a10 0x7f0c7c0ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:47:05.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c808c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c7c153c80 0x7f0c7c14e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:47:05.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c808c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7c14fd40 con 0x7f0c7c0c4a10 2023-10-30T14:47:05.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c808c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7c1ef890 con 0x7f0c7c153c80 2023-10-30T14:47:05.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c808c9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7c14e400 con 0x7f0c7c0c42a0 2023-10-30T14:47:05.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c73fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c7c0c4a10 0x7f0c7c0ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:47:05.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c73fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c7c0c4a10 0x7f0c7c0ce800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47708/0 (socket says 172.21.15.57:47708) 2023-10-30T14:47:05.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c73fff640 1 -- 172.21.15.57:0/3045893845 learned_addr learned my addr 172.21.15.57:0/3045893845 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:47:05.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c7ad9d640 1 --2- 172.21.15.57:0/3045893845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c7c153c80 0x7f0c7c14e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:47:05.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c7a59c640 1 --2- 172.21.15.57:0/3045893845 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c7c0c42a0 0x7f0c7c1ef350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:47:05.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c73fff640 1 -- 172.21.15.57:0/3045893845 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c7c0c42a0 msgr2=0x7f0c7c1ef350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:47:05.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c73fff640 1 --2- 172.21.15.57:0/3045893845 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c7c0c42a0 0x7f0c7c1ef350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:47:05.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c73fff640 1 -- 172.21.15.57:0/3045893845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c7c153c80 msgr2=0x7f0c7c14e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:47:05.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c73fff640 1 --2- 172.21.15.57:0/3045893845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c7c153c80 0x7f0c7c14e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:47:05.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c73fff640 1 -- 172.21.15.57:0/3045893845 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c009420 con 0x7f0c7c0c4a10 2023-10-30T14:47:05.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c7ad9d640 1 --2- 172.21.15.57:0/3045893845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c7c153c80 0x7f0c7c14e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:47:05.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c7a59c640 1 --2- 172.21.15.57:0/3045893845 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c7c0c42a0 0x7f0c7c1ef350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:47:05.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c73fff640 1 --2- 172.21.15.57:0/3045893845 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c7c0c4a10 0x7f0c7c0ce800 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f0c6400c4e0 tx=0x7f0c6400c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:47:05.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c72ffd640 1 -- 172.21.15.57:0/3045893845 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c640074e0 con 0x7f0c7c0c4a10 2023-10-30T14:47:05.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c72ffd640 1 -- 172.21.15.57:0/3045893845 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c64007660 con 0x7f0c7c0c4a10 2023-10-30T14:47:05.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c72ffd640 1 -- 172.21.15.57:0/3045893845 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c64004a30 con 0x7f0c7c0c4a10 2023-10-30T14:47:05.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c808c9640 1 -- 172.21.15.57:0/3045893845 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c7c150290 con 0x7f0c7c0c4a10 2023-10-30T14:47:05.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.472+0000 7f0c808c9640 1 -- 172.21.15.57:0/3045893845 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c7c14f4a0 con 0x7f0c7c0c4a10 2023-10-30T14:47:05.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.476+0000 7f0c808c9640 1 -- 172.21.15.57:0/3045893845 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7c14df30 con 0x7f0c7c0c4a10 2023-10-30T14:47:05.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.476+0000 7f0c72ffd640 1 -- 172.21.15.57:0/3045893845 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0c64004b90 con 0x7f0c7c0c4a10 2023-10-30T14:47:05.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.476+0000 7f0c72ffd640 1 --2- 172.21.15.57:0/3045893845 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0c5c0664a0 0x7f0c5c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:47:05.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.476+0000 7f0c72ffd640 1 -- 172.21.15.57:0/3045893845 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f0c640837a0 con 0x7f0c7c0c4a10 2023-10-30T14:47:05.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.476+0000 7f0c7a59c640 1 --2- 172.21.15.57:0/3045893845 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0c5c0664a0 0x7f0c5c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:47:05.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.476+0000 7f0c7a59c640 1 --2- 172.21.15.57:0/3045893845 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0c5c0664a0 0x7f0c5c068960 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c0b2000 tx=0x7f0c6c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:47:05.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.480+0000 7f0c72ffd640 1 -- 172.21.15.57:0/3045893845 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0c7c14df30 con 0x7f0c7c0c4a10 2023-10-30T14:47:05.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.672+0000 7f0c808c9640 1 -- 172.21.15.57:0/3045893845 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c7c14df30 con 0x7f0c7c0c4a10 2023-10-30T14:47:05.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.672+0000 7f0c72ffd640 1 -- 172.21.15.57:0/3045893845 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+6500 (secure 0 0 0) 0x7f0c7c14df30 con 0x7f0c7c0c4a10 2023-10-30T14:47:05.675 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:47:05.676 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7911,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:46:37.266044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:47:05.676 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 178 2023-10-30T14:47:05.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.672+0000 7f0c808c9640 1 -- 172.21.15.57:0/3045893845 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0c5c0664a0 msgr2=0x7f0c5c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:47:05.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.672+0000 7f0c808c9640 1 --2- 172.21.15.57:0/3045893845 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0c5c0664a0 0x7f0c5c068960 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c0b2000 tx=0x7f0c6c008040 comp rx=0 tx=0).stop 2023-10-30T14:47:05.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.672+0000 7f0c808c9640 1 -- 172.21.15.57:0/3045893845 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c7c0c4a10 msgr2=0x7f0c7c0ce800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:47:05.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.672+0000 7f0c808c9640 1 --2- 172.21.15.57:0/3045893845 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c7c0c4a10 0x7f0c7c0ce800 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f0c6400c4e0 tx=0x7f0c6400c9a0 comp rx=0 tx=0).stop 2023-10-30T14:47:05.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.672+0000 7f0c808c9640 1 -- 172.21.15.57:0/3045893845 shutdown_connections 2023-10-30T14:47:05.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.672+0000 7f0c808c9640 1 --2- 172.21.15.57:0/3045893845 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0c5c0664a0 0x7f0c5c068960 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:47:05.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.672+0000 7f0c808c9640 1 --2- 172.21.15.57:0/3045893845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c7c153c80 0x7f0c7c14e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:47:05.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.672+0000 7f0c808c9640 1 --2- 172.21.15.57:0/3045893845 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c7c0c4a10 0x7f0c7c0ce800 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:47:05.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.672+0000 7f0c808c9640 1 --2- 172.21.15.57:0/3045893845 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c7c0c42a0 0x7f0c7c1ef350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:47:05.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.672+0000 7f0c808c9640 1 -- 172.21.15.57:0/3045893845 >> 172.21.15.57:0/3045893845 conn(0x7f0c7c0715c0 msgr2=0x7f0c7c158640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:47:05.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.672+0000 7f0c808c9640 1 -- 172.21.15.57:0/3045893845 shutdown_connections 2023-10-30T14:47:05.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:47:05.672+0000 7f0c808c9640 1 -- 172.21.15.57:0/3045893845 wait complete. 2023-10-30T14:47:05.696 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T14:47:05.696 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 69 secs before thrashing 2023-10-30T14:47:09.626 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:47:09.630 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:47:10.643 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 34% (132/386) Updating files: 35% (136/386) Updating files: 36% (139/386) Updating files: 37% (143/386) Updating files: 38% (147/386) Updating files: 39% (151/386) Updating files: 40% (155/386) Updating files: 41% (159/386) Updating files: 42% (163/386) Updating files: 43% (166/386) Updating files: 44% (170/386) Updating files: 45% (174/386) Updating files: 46% (178/386) Updating files: 47% (182/386) Updating files: 48% (186/386) Updating files: 49% (190/386) Updating files: 50% (193/386) Updating files: 51% (197/386) Updating files: 52% (201/386) Updating files: 53% (205/386) Updating files: 54% (209/386) Updating files: 55% (213/386) Updating files: 55% (216/386) Updating files: 56% (217/386) Updating files: 57% (221/386) Updating files: 58% (224/386) Updating files: 59% (228/386) Updating files: 60% (232/386) Updating files: 61% (236/386) Updating files: 62% (240/386) Updating files: 62% (243/386) Updating files: 63% (244/386) Updating files: 64% (248/386) Updating files: 65% (251/386) Updating files: 66% (255/386) Updating files: 67% (259/386) Updating files: 68% (263/386) Updating files: 69% (267/386) Updating files: 70% (271/386) Updating files: 71% (275/386) Updating files: 72% (278/386) Updating files: 73% (282/386) Updating files: 73% (285/386) Updating files: 74% (286/386) Updating files: 75% (290/386) Updating files: 76% (294/386) Updating files: 77% (298/386) Updating files: 78% (302/386) Updating files: 79% (305/386) Updating files: 80% (309/386) Updating files: 81% (313/386) Updating files: 81% (314/386) Updating files: 82% (317/386) Updating files: 83% (321/386) Updating files: 83% (324/386) Updating files: 84% (325/386) Updating files: 85% (329/386) Updating files: 86% (332/386) Updating files: 87% (336/386) Updating files: 88% (340/386) Updating files: 89% (344/386) Updating files: 90% (348/386) Updating files: 91% (352/386) Updating files: 92% (356/386) Updating files: 93% (359/386) Updating files: 94% (363/386) Updating files: 95% (367/386) Updating files: 96% (371/386) Updating files: 97% (375/386) Updating files: 98% (379/386) Updating files: 99% (383/386) Updating files: 100% (386/386) Updating files: 100% (386/386), done. 2023-10-30T14:47:11.242 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 2528b5ee105 v0.43 2023-10-30T14:47:11.247 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.53 2023-10-30T14:47:11.420 INFO:tasks.workunit.client.0.smithi057.stdout:v0.54 2023-10-30T14:47:11.421 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 54 -eq 48 ] 2023-10-30T14:47:11.421 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.54 2023-10-30T14:47:11.422 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.54 2023-10-30T14:47:11.422 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.54 2023-10-30T14:47:17.243 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 19% (42/212) Updating files: 20% (43/212) Updating files: 21% (45/212) Updating files: 22% (47/212) Updating files: 23% (49/212) Updating files: 24% (51/212) Updating files: 25% (53/212) Updating files: 26% (56/212) Updating files: 27% (58/212) Updating files: 28% (60/212) Updating files: 29% (62/212) Updating files: 30% (64/212) Updating files: 31% (66/212) Updating files: 32% (68/212) Updating files: 33% (70/212) Updating files: 34% (73/212) Updating files: 35% (75/212) Updating files: 36% (77/212) Updating files: 37% (79/212) Updating files: 38% (81/212) Updating files: 39% (83/212) Updating files: 40% (85/212) Updating files: 41% (87/212) Updating files: 42% (90/212) Updating files: 43% (92/212) Updating files: 44% (94/212) Updating files: 45% (96/212) Updating files: 46% (98/212) Updating files: 47% (100/212) Updating files: 48% (102/212) Updating files: 49% (104/212) Updating files: 50% (106/212) Updating files: 51% (109/212) Updating files: 52% (111/212) Updating files: 53% (113/212) Updating files: 54% (115/212) Updating files: 55% (117/212) Updating files: 56% (119/212) Updating files: 57% (121/212) Updating files: 58% (123/212) Updating files: 59% (126/212) Updating files: 60% (128/212) Updating files: 61% (130/212) Updating files: 62% (132/212) Updating files: 63% (134/212) Updating files: 64% (136/212) Updating files: 65% (138/212) Updating files: 66% (140/212) Updating files: 67% (143/212) Updating files: 67% (144/212) Updating files: 68% (145/212) Updating files: 69% (147/212) Updating files: 70% (149/212) Updating files: 71% (151/212) Updating files: 72% (153/212) Updating files: 73% (155/212) Updating files: 74% (157/212) Updating files: 75% (159/212) Updating files: 76% (162/212) Updating files: 77% (164/212) Updating files: 78% (166/212) Updating files: 79% (168/212) Updating files: 80% (170/212) Updating files: 81% (172/212) Updating files: 81% (173/212) Updating files: 82% (174/212) Updating files: 83% (176/212) Updating files: 84% (179/212) Updating files: 85% (181/212) Updating files: 86% (183/212) Updating files: 87% (185/212) Updating files: 88% (187/212) Updating files: 89% (189/212) Updating files: 90% (191/212) Updating files: 91% (193/212) Updating files: 92% (196/212) Updating files: 93% (198/212) Updating files: 94% (200/212) Updating files: 95% (202/212) Updating files: 96% (204/212) Updating files: 97% (206/212) Updating files: 98% (208/212) Updating files: 99% (210/212) Updating files: 100% (212/212) Updating files: 100% (212/212), done. 2023-10-30T14:47:17.287 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 60b84b095b1 v0.54 2023-10-30T14:47:17.292 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.54 2023-10-30T14:47:17.339 INFO:tasks.workunit.client.0.smithi057.stdout:v0.55 2023-10-30T14:47:17.339 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 55 -eq 48 ] 2023-10-30T14:47:17.339 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.55 2023-10-30T14:47:17.340 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.55 2023-10-30T14:47:17.340 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.55 2023-10-30T14:47:30.709 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 41% (225/540) Updating files: 42% (227/540) Updating files: 43% (233/540) Updating files: 44% (238/540) Updating files: 45% (243/540) Updating files: 46% (249/540) Updating files: 47% (254/540) Updating files: 47% (256/540) Updating files: 48% (260/540) Updating files: 49% (265/540) Updating files: 50% (270/540) Updating files: 50% (275/540) Updating files: 51% (276/540) Updating files: 52% (281/540) Updating files: 53% (287/540) Updating files: 54% (292/540) Updating files: 55% (297/540) Updating files: 56% (303/540) Updating files: 57% (308/540) Updating files: 58% (314/540) Updating files: 59% (319/540) Updating files: 60% (324/540) Updating files: 61% (330/540) Updating files: 62% (335/540) Updating files: 63% (341/540) Updating files: 64% (346/540) Updating files: 65% (351/540) Updating files: 65% (354/540) Updating files: 66% (357/540) Updating files: 67% (362/540) Updating files: 68% (368/540) Updating files: 69% (373/540) Updating files: 70% (378/540) Updating files: 71% (384/540) Updating files: 72% (389/540) Updating files: 73% (395/540) Updating files: 74% (400/540) Updating files: 75% (405/540) Updating files: 76% (411/540) Updating files: 77% (416/540) Updating files: 78% (422/540) Updating files: 79% (427/540) Updating files: 79% (428/540) Updating files: 80% (432/540) Updating files: 81% (438/540) Updating files: 82% (443/540) Updating files: 83% (449/540) Updating files: 84% (454/540) Updating files: 85% (459/540) Updating files: 86% (465/540) Updating files: 87% (470/540) Updating files: 88% (476/540) Updating files: 89% (481/540) Updating files: 90% (486/540) Updating files: 91% (492/540) Updating files: 92% (497/540) Updating files: 93% (503/540) Updating files: 94% (508/540) Updating files: 95% (513/540) Updating files: 96% (519/540) Updating files: 97% (524/540) Updating files: 98% (530/540) Updating files: 99% (535/540) Updating files: 100% (540/540) Updating files: 100% (540/540), done. 2023-10-30T14:47:31.066 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 690f8175606 v0.55 2023-10-30T14:47:31.077 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.55 2023-10-30T14:47:31.529 INFO:tasks.workunit.client.0.smithi057.stdout:v0.56 2023-10-30T14:47:31.530 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 56 -eq 48 ] 2023-10-30T14:47:31.530 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.56 2023-10-30T14:47:31.530 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.56 2023-10-30T14:47:31.531 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.56 2023-10-30T14:47:35.276 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 33% (83/246) Updating files: 34% (84/246) Updating files: 35% (87/246) Updating files: 36% (89/246) Updating files: 37% (92/246) Updating files: 38% (94/246) Updating files: 39% (96/246) Updating files: 40% (99/246) Updating files: 41% (101/246) Updating files: 42% (104/246) Updating files: 43% (106/246) Updating files: 44% (109/246) Updating files: 45% (111/246) Updating files: 46% (114/246) Updating files: 47% (116/246) Updating files: 48% (119/246) Updating files: 49% (121/246) Updating files: 50% (123/246) Updating files: 51% (126/246) Updating files: 52% (128/246) Updating files: 53% (131/246) Updating files: 54% (133/246) Updating files: 55% (136/246) Updating files: 56% (138/246) Updating files: 57% (141/246) Updating files: 58% (143/246) Updating files: 59% (146/246) Updating files: 60% (148/246) Updating files: 61% (151/246) Updating files: 62% (153/246) Updating files: 63% (155/246) Updating files: 64% (158/246) Updating files: 65% (160/246) Updating files: 66% (163/246) Updating files: 67% (165/246) Updating files: 68% (168/246) Updating files: 69% (170/246) Updating files: 70% (173/246) Updating files: 71% (175/246) Updating files: 72% (178/246) Updating files: 73% (180/246) Updating files: 74% (183/246) Updating files: 75% (185/246) Updating files: 76% (187/246) Updating files: 77% (190/246) Updating files: 78% (192/246) Updating files: 79% (195/246) Updating files: 80% (197/246) Updating files: 81% (200/246) Updating files: 82% (202/246) Updating files: 82% (204/246) Updating files: 83% (205/246) Updating files: 84% (207/246) Updating files: 85% (210/246) Updating files: 86% (212/246) Updating files: 87% (215/246) Updating files: 88% (217/246) Updating files: 89% (219/246) Updating files: 90% (222/246) Updating files: 91% (224/246) Updating files: 92% (227/246) Updating files: 93% (229/246) Updating files: 94% (232/246) Updating files: 95% (234/246) Updating files: 96% (237/246) Updating files: 97% (239/246) Updating files: 98% (242/246) Updating files: 99% (244/246) Updating files: 100% (246/246) Updating files: 100% (246/246), done. 2023-10-30T14:47:35.570 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 1a32f0a0b42 v0.56 2023-10-30T14:47:35.578 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.56 2023-10-30T14:47:35.646 INFO:tasks.workunit.client.0.smithi057.stdout:v0.57 2023-10-30T14:47:35.646 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 57 -eq 48 ] 2023-10-30T14:47:35.646 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.57 2023-10-30T14:47:35.647 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.57 2023-10-30T14:47:35.647 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.57 2023-10-30T14:47:39.910 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:47:39.914 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:47:41.126 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 51% (156/304) Updating files: 52% (159/304) Updating files: 53% (162/304) Updating files: 54% (165/304) Updating files: 55% (168/304) Updating files: 56% (171/304) Updating files: 57% (174/304) Updating files: 58% (177/304) Updating files: 59% (180/304) Updating files: 60% (183/304) Updating files: 61% (186/304) Updating files: 62% (189/304) Updating files: 63% (192/304) Updating files: 64% (195/304) Updating files: 65% (198/304) Updating files: 66% (201/304) Updating files: 67% (204/304) Updating files: 68% (207/304) Updating files: 69% (210/304) Updating files: 70% (213/304) Updating files: 71% (216/304) Updating files: 72% (219/304) Updating files: 73% (222/304) Updating files: 74% (225/304) Updating files: 75% (228/304) Updating files: 76% (232/304) Updating files: 77% (235/304) Updating files: 77% (237/304) Updating files: 78% (238/304) Updating files: 78% (240/304) Updating files: 79% (241/304) Updating files: 80% (244/304) Updating files: 81% (247/304) Updating files: 82% (250/304) Updating files: 83% (253/304) Updating files: 84% (256/304) Updating files: 84% (258/304) Updating files: 85% (259/304) Updating files: 86% (262/304) Updating files: 87% (265/304) Updating files: 88% (268/304) Updating files: 89% (271/304) Updating files: 90% (274/304) Updating files: 91% (277/304) Updating files: 92% (280/304) Updating files: 93% (283/304) Updating files: 94% (286/304) Updating files: 95% (289/304) Updating files: 96% (292/304) Updating files: 97% (295/304) Updating files: 98% (298/304) Updating files: 99% (301/304) Updating files: 100% (304/304) Updating files: 100% (304/304), done. 2023-10-30T14:47:41.160 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 9a7a9d06c06 v0.57 2023-10-30T14:47:41.167 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.57 2023-10-30T14:47:41.305 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 58 -eq 48 ] 2023-10-30T14:47:41.305 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.58 2023-10-30T14:47:41.306 INFO:tasks.workunit.client.0.smithi057.stdout:v0.58 2023-10-30T14:47:41.306 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.58 2023-10-30T14:47:41.306 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.58 2023-10-30T14:47:44.152 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 56% (151/268) Updating files: 57% (153/268) Updating files: 58% (156/268) Updating files: 59% (159/268) Updating files: 60% (161/268) Updating files: 61% (164/268) Updating files: 62% (167/268) Updating files: 63% (169/268) Updating files: 64% (172/268) Updating files: 65% (175/268) Updating files: 66% (177/268) Updating files: 67% (180/268) Updating files: 68% (183/268) Updating files: 69% (185/268) Updating files: 70% (188/268) Updating files: 71% (191/268) Updating files: 72% (193/268) Updating files: 73% (196/268) Updating files: 74% (199/268) Updating files: 75% (201/268) Updating files: 76% (204/268) Updating files: 77% (207/268) Updating files: 78% (210/268) Updating files: 79% (212/268) Updating files: 80% (215/268) Updating files: 81% (218/268) Updating files: 82% (220/268) Updating files: 83% (223/268) Updating files: 84% (226/268) Updating files: 85% (228/268) Updating files: 86% (231/268) Updating files: 87% (234/268) Updating files: 88% (236/268) Updating files: 89% (239/268) Updating files: 90% (242/268) Updating files: 91% (244/268) Updating files: 92% (247/268) Updating files: 93% (250/268) Updating files: 94% (252/268) Updating files: 94% (254/268) Updating files: 95% (255/268) Updating files: 96% (258/268) Updating files: 97% (260/268) Updating files: 98% (263/268) Updating files: 99% (266/268) Updating files: 100% (268/268) Updating files: 100% (268/268), done. 2023-10-30T14:47:44.199 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at ba3f91e7504 v0.58 2023-10-30T14:47:44.207 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.58 2023-10-30T14:47:44.544 INFO:tasks.workunit.client.0.smithi057.stdout:v0.59 2023-10-30T14:47:44.545 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 59 -eq 48 ] 2023-10-30T14:47:44.545 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.59 2023-10-30T14:47:44.545 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.59 2023-10-30T14:47:44.545 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.59 2023-10-30T14:48:00.141 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 83% (176/211) Updating files: 84% (178/211) Updating files: 85% (180/211) Updating files: 86% (182/211) Updating files: 87% (184/211) Updating files: 88% (186/211) Updating files: 88% (187/211) Updating files: 89% (188/211) Updating files: 90% (190/211) Updating files: 91% (193/211) Updating files: 92% (195/211) Updating files: 93% (197/211) Updating files: 94% (199/211) Updating files: 95% (201/211) Updating files: 96% (203/211) Updating files: 97% (205/211) Updating files: 98% (207/211) Updating files: 99% (209/211) Updating files: 100% (211/211) Updating files: 100% (211/211), done. 2023-10-30T14:48:00.184 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at cbae6a435c6 v0.59 2023-10-30T14:48:00.190 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.59 2023-10-30T14:48:00.293 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 60 -eq 48 ] 2023-10-30T14:48:00.293 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.60 2023-10-30T14:48:00.293 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.60 2023-10-30T14:48:00.294 INFO:tasks.workunit.client.0.smithi057.stdout:v0.60 2023-10-30T14:48:00.294 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.60 2023-10-30T14:48:06.835 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 10% (27/256) Updating files: 11% (29/256) Updating files: 12% (31/256) Updating files: 13% (34/256) Updating files: 14% (36/256) Updating files: 14% (37/256) Updating files: 15% (39/256) Updating files: 16% (41/256) Updating files: 17% (44/256) Updating files: 18% (47/256) Updating files: 18% (48/256) Updating files: 19% (49/256) Updating files: 20% (52/256) Updating files: 21% (54/256) Updating files: 22% (57/256) Updating files: 23% (59/256) Updating files: 24% (62/256) Updating files: 25% (64/256) Updating files: 26% (67/256) Updating files: 27% (70/256) Updating files: 28% (72/256) Updating files: 29% (75/256) Updating files: 30% (77/256) Updating files: 31% (80/256) Updating files: 32% (82/256) Updating files: 33% (85/256) Updating files: 34% (88/256) Updating files: 35% (90/256) Updating files: 36% (93/256) Updating files: 37% (95/256) Updating files: 38% (98/256) Updating files: 39% (100/256) Updating files: 40% (103/256) Updating files: 41% (105/256) Updating files: 42% (108/256) Updating files: 43% (111/256) Updating files: 44% (113/256) Updating files: 45% (116/256) Updating files: 46% (118/256) Updating files: 47% (121/256) Updating files: 48% (123/256) Updating files: 49% (126/256) Updating files: 50% (128/256) Updating files: 51% (131/256) Updating files: 52% (134/256) Updating files: 53% (136/256) Updating files: 54% (139/256) Updating files: 55% (141/256) Updating files: 56% (144/256) Updating files: 57% (146/256) Updating files: 57% (147/256) Updating files: 58% (149/256) Updating files: 59% (152/256) Updating files: 60% (154/256) Updating files: 61% (157/256) Updating files: 62% (159/256) Updating files: 63% (162/256) Updating files: 64% (164/256) Updating files: 65% (167/256) Updating files: 66% (169/256) Updating files: 67% (172/256) Updating files: 68% (175/256) Updating files: 69% (177/256) Updating files: 70% (180/256) Updating files: 71% (182/256) Updating files: 72% (185/256) Updating files: 73% (187/256) Updating files: 74% (190/256) Updating files: 75% (192/256) Updating files: 76% (195/256) Updating files: 77% (198/256) Updating files: 78% (200/256) Updating files: 79% (203/256) Updating files: 80% (205/256) Updating files: 81% (208/256) Updating files: 82% (210/256) Updating files: 83% (213/256) Updating files: 84% (216/256) Updating files: 85% (218/256) Updating files: 86% (221/256) Updating files: 87% (223/256) Updating files: 88% (226/256) Updating files: 89% (228/256) Updating files: 90% (231/256) Updating files: 91% (233/256) Updating files: 92% (236/256) Updating files: 93% (239/256) Updating files: 94% (241/256) Updating files: 95% (244/256) Updating files: 96% (246/256) Updating files: 97% (249/256) Updating files: 98% (251/256) Updating files: 99% (254/256) Updating files: 100% (256/256) Updating files: 100% (256/256), done. 2023-10-30T14:48:06.877 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at f26f7a39021 v0.60 2023-10-30T14:48:06.884 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.60 2023-10-30T14:48:07.182 INFO:tasks.workunit.client.0.smithi057.stdout:v0.61 2023-10-30T14:48:07.182 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 61 -eq 48 ] 2023-10-30T14:48:07.182 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.61 2023-10-30T14:48:07.182 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.61 2023-10-30T14:48:07.183 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.61 2023-10-30T14:48:09.971 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:48:09.975 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:48:12.865 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 14% (53/376) Updating files: 15% (57/376) Updating files: 16% (61/376) Updating files: 17% (64/376) Updating files: 18% (68/376) Updating files: 19% (72/376) Updating files: 20% (76/376) Updating files: 21% (79/376) Updating files: 22% (83/376) Updating files: 23% (87/376) Updating files: 24% (91/376) Updating files: 25% (94/376) Updating files: 26% (98/376) Updating files: 27% (102/376) Updating files: 28% (106/376) Updating files: 29% (110/376) Updating files: 30% (113/376) Updating files: 31% (117/376) Updating files: 32% (121/376) Updating files: 33% (125/376) Updating files: 34% (128/376) Updating files: 35% (132/376) Updating files: 36% (136/376) Updating files: 37% (140/376) Updating files: 38% (143/376) Updating files: 39% (147/376) Updating files: 40% (151/376) Updating files: 41% (155/376) Updating files: 42% (158/376) Updating files: 43% (162/376) Updating files: 44% (166/376) Updating files: 45% (170/376) Updating files: 46% (173/376) Updating files: 47% (177/376) Updating files: 48% (181/376) Updating files: 49% (185/376) Updating files: 50% (188/376) Updating files: 51% (192/376) Updating files: 52% (196/376) Updating files: 53% (200/376) Updating files: 54% (204/376) Updating files: 55% (207/376) Updating files: 56% (211/376) Updating files: 57% (215/376) Updating files: 58% (219/376) Updating files: 59% (222/376) Updating files: 60% (226/376) Updating files: 61% (230/376) Updating files: 62% (234/376) Updating files: 63% (237/376) Updating files: 64% (241/376) Updating files: 65% (245/376) Updating files: 66% (249/376) Updating files: 67% (252/376) Updating files: 68% (256/376) Updating files: 69% (260/376) Updating files: 70% (264/376) Updating files: 71% (267/376) Updating files: 72% (271/376) Updating files: 73% (275/376) Updating files: 74% (279/376) Updating files: 75% (282/376) Updating files: 76% (286/376) Updating files: 77% (290/376) Updating files: 78% (294/376) Updating files: 79% (298/376) Updating files: 80% (301/376) Updating files: 81% (305/376) Updating files: 82% (309/376) Updating files: 82% (310/376) Updating files: 83% (313/376) Updating files: 84% (316/376) Updating files: 85% (320/376) Updating files: 86% (324/376) Updating files: 87% (328/376) Updating files: 88% (331/376) Updating files: 89% (335/376) Updating files: 90% (339/376) Updating files: 91% (343/376) Updating files: 92% (346/376) Updating files: 92% (349/376) Updating files: 93% (350/376) Updating files: 94% (354/376) Updating files: 95% (358/376) Updating files: 96% (361/376) Updating files: 97% (365/376) Updating files: 98% (369/376) Updating files: 99% (373/376) Updating files: 100% (376/376) Updating files: 100% (376/376), done. 2023-10-30T14:48:13.161 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 237f3f1e8d8 v0.61 2023-10-30T14:48:13.170 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.61 2023-10-30T14:48:13.260 INFO:tasks.workunit.client.0.smithi057.stdout:v0.62 2023-10-30T14:48:13.260 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 62 -eq 48 ] 2023-10-30T14:48:13.260 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.62 2023-10-30T14:48:13.261 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.62 2023-10-30T14:48:13.261 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.62 2023-10-30T14:48:14.698 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:14.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.839+0000 7f054d13e640 1 Processor -- start 2023-10-30T14:48:14.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.839+0000 7f054d13e640 1 -- start start 2023-10-30T14:48:14.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.839+0000 7f054d13e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0548153c80 0x7f054815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:14.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.839+0000 7f054d13e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f05480c4a10 0x7f05480c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:14.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.839+0000 7f054d13e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0548159f20 con 0x7f054815a3b0 2023-10-30T14:48:14.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.839+0000 7f054d13e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f054815dec0 con 0x7f05480c4a10 2023-10-30T14:48:14.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.839+0000 7f054d13e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05481599a0 con 0x7f0548153c80 2023-10-30T14:48:14.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.839+0000 7f054659c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f05480c4a10 0x7f05480c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:14.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.839+0000 7f054659c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f05480c4a10 0x7f05480c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37830/0 (socket says 172.21.15.57:37830) 2023-10-30T14:48:14.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.839+0000 7f054659c640 1 -- 172.21.15.57:0/3162256946 learned_addr learned my addr 172.21.15.57:0/3162256946 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:14.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.839+0000 7f0546d9d640 1 --2- 172.21.15.57:0/3162256946 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0548153c80 0x7f054815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:14.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.839+0000 7f054659c640 1 -- 172.21.15.57:0/3162256946 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0548153c80 msgr2=0x7f054815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:14.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.839+0000 7f054659c640 1 --2- 172.21.15.57:0/3162256946 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0548153c80 0x7f054815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:14.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.839+0000 7f054659c640 1 -- 172.21.15.57:0/3162256946 >> v1:172.21.15.88:6789/0 conn(0x7f054815a3b0 legacy=0x7f05480ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:14.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.839+0000 7f054659c640 1 -- 172.21.15.57:0/3162256946 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05480c47e0 con 0x7f05480c4a10 2023-10-30T14:48:14.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.839+0000 7f0546d9d640 1 --2- 172.21.15.57:0/3162256946 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0548153c80 0x7f054815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:48:14.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.843+0000 7f054659c640 1 --2- 172.21.15.57:0/3162256946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05480c4a10 0x7f05480c42a0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f053c009870 tx=0x7f053c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb77497ddb1a199e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:14.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.843+0000 7f0545d9b640 1 -- 172.21.15.57:0/3162256946 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c0a8a30 con 0x7f05480c4a10 2023-10-30T14:48:14.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.843+0000 7f0545d9b640 1 -- 172.21.15.57:0/3162256946 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053c0a8b90 con 0x7f05480c4a10 2023-10-30T14:48:14.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.843+0000 7f0545d9b640 1 -- 172.21.15.57:0/3162256946 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c0b0440 con 0x7f05480c4a10 2023-10-30T14:48:14.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.843+0000 7f054d13e640 1 -- 172.21.15.57:0/3162256946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05480c4a10 msgr2=0x7f05480c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:14.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.843+0000 7f054d13e640 1 --2- 172.21.15.57:0/3162256946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05480c4a10 0x7f05480c42a0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f053c009870 tx=0x7f053c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:48:14.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.843+0000 7f054d13e640 1 -- 172.21.15.57:0/3162256946 shutdown_connections 2023-10-30T14:48:14.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.843+0000 7f054d13e640 1 --2- 172.21.15.57:0/3162256946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05480c4a10 0x7f05480c42a0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:14.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.843+0000 7f054d13e640 1 --2- 172.21.15.57:0/3162256946 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0548153c80 0x7f054815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:14.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.843+0000 7f054d13e640 1 -- 172.21.15.57:0/3162256946 >> 172.21.15.57:0/3162256946 conn(0x7f05480715c0 msgr2=0x7f05480719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:14.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.843+0000 7f054d13e640 1 -- 172.21.15.57:0/3162256946 shutdown_connections 2023-10-30T14:48:14.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.843+0000 7f054d13e640 1 -- 172.21.15.57:0/3162256946 wait complete. 2023-10-30T14:48:14.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054d13e640 1 Processor -- start 2023-10-30T14:48:14.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054d13e640 1 -- start start 2023-10-30T14:48:14.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054d13e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05480c4a10 0x7f05481f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:14.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054d13e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0548153c80 0x7f05480c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:14.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054d13e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f054815a3b0 0x7f054814eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:14.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054d13e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054815b330 con 0x7f0548153c80 2023-10-30T14:48:14.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054d13e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054815c930 con 0x7f05480c4a10 2023-10-30T14:48:14.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054d13e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05481f31b0 con 0x7f054815a3b0 2023-10-30T14:48:14.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0548153c80 0x7f05480c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:14.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0548153c80 0x7f05480c8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37834/0 (socket says 172.21.15.57:37834) 2023-10-30T14:48:14.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054659c640 1 -- 172.21.15.57:0/454690915 learned_addr learned my addr 172.21.15.57:0/454690915 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:14.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f0546d9d640 1 --2- 172.21.15.57:0/454690915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05480c4a10 0x7f05481f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:14.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054759e640 1 --2- 172.21.15.57:0/454690915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f054815a3b0 0x7f054814eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:14.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054659c640 1 -- 172.21.15.57:0/454690915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f054815a3b0 msgr2=0x7f054814eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:14.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054659c640 1 --2- 172.21.15.57:0/454690915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f054815a3b0 0x7f054814eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:14.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054659c640 1 -- 172.21.15.57:0/454690915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05480c4a10 msgr2=0x7f05481f2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:14.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054659c640 1 --2- 172.21.15.57:0/454690915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05480c4a10 0x7f05481f2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:14.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054659c640 1 -- 172.21.15.57:0/454690915 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c0094d0 con 0x7f0548153c80 2023-10-30T14:48:14.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054659c640 1 --2- 172.21.15.57:0/454690915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0548153c80 0x7f05480c8b00 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f053c01d3e0 tx=0x7f053c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:14.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f052f7fe640 1 -- 172.21.15.57:0/454690915 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c0ba600 con 0x7f0548153c80 2023-10-30T14:48:14.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f052f7fe640 1 -- 172.21.15.57:0/454690915 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053c0ba780 con 0x7f0548153c80 2023-10-30T14:48:14.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f052f7fe640 1 -- 172.21.15.57:0/454690915 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c0bab30 con 0x7f0548153c80 2023-10-30T14:48:14.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054d13e640 1 -- 172.21.15.57:0/454690915 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05480b1ae0 con 0x7f0548153c80 2023-10-30T14:48:14.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054d13e640 1 -- 172.21.15.57:0/454690915 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054815c000 con 0x7f0548153c80 2023-10-30T14:48:14.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.847+0000 7f054d13e640 1 -- 172.21.15.57:0/454690915 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054804e170 con 0x7f0548153c80 2023-10-30T14:48:14.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.851+0000 7f052f7fe640 1 -- 172.21.15.57:0/454690915 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f053c0bac90 con 0x7f0548153c80 2023-10-30T14:48:14.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.851+0000 7f052f7fe640 1 --2- 172.21.15.57:0/454690915 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05340664d0 0x7f0534068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:14.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.851+0000 7f052f7fe640 1 -- 172.21.15.57:0/454690915 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f053c1229e0 con 0x7f0548153c80 2023-10-30T14:48:14.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.851+0000 7f0546d9d640 1 --2- 172.21.15.57:0/454690915 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05340664d0 0x7f0534068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:14.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.851+0000 7f0546d9d640 1 --2- 172.21.15.57:0/454690915 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05340664d0 0x7f0534068990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f0538009950 tx=0x7f0538008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:14.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:14.855+0000 7f052f7fe640 1 -- 172.21.15.57:0/454690915 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f053c0ef810 con 0x7f0548153c80 2023-10-30T14:48:15.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.067+0000 7f054d13e640 1 -- 172.21.15.57:0/454690915 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f054815ca70 con 0x7f0548153c80 2023-10-30T14:48:15.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.067+0000 7f052f7fe640 1 -- 172.21.15.57:0/454690915 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6499 (secure 0 0 0) 0x7f053c0f2e30 con 0x7f0548153c80 2023-10-30T14:48:15.072 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:15.072 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7911,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:47:59.268639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:15.073 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 179 2023-10-30T14:48:15.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.067+0000 7f054d13e640 1 -- 172.21.15.57:0/454690915 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05340664d0 msgr2=0x7f0534068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:15.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.067+0000 7f054d13e640 1 --2- 172.21.15.57:0/454690915 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05340664d0 0x7f0534068990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f0538009950 tx=0x7f0538008040 comp rx=0 tx=0).stop 2023-10-30T14:48:15.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.067+0000 7f054d13e640 1 -- 172.21.15.57:0/454690915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0548153c80 msgr2=0x7f05480c8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:15.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.067+0000 7f054d13e640 1 --2- 172.21.15.57:0/454690915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0548153c80 0x7f05480c8b00 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f053c01d3e0 tx=0x7f053c01bad0 comp rx=0 tx=0).stop 2023-10-30T14:48:15.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.067+0000 7f054d13e640 1 -- 172.21.15.57:0/454690915 shutdown_connections 2023-10-30T14:48:15.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.067+0000 7f054d13e640 1 --2- 172.21.15.57:0/454690915 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05340664d0 0x7f0534068990 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:15.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.067+0000 7f054d13e640 1 --2- 172.21.15.57:0/454690915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f054815a3b0 0x7f054814eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:15.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.067+0000 7f054d13e640 1 --2- 172.21.15.57:0/454690915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0548153c80 0x7f05480c8b00 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:15.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.067+0000 7f054d13e640 1 --2- 172.21.15.57:0/454690915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05480c4a10 0x7f05481f2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:15.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.067+0000 7f054d13e640 1 -- 172.21.15.57:0/454690915 >> 172.21.15.57:0/454690915 conn(0x7f05480715c0 msgr2=0x7f05481586d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:15.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.067+0000 7f054d13e640 1 -- 172.21.15.57:0/454690915 shutdown_connections 2023-10-30T14:48:15.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.067+0000 7f054d13e640 1 -- 172.21.15.57:0/454690915 wait complete. 2023-10-30T14:48:15.094 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2023-10-30T14:48:15.094 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T14:48:15.095 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:48:15.129 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T14:48:15.129 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:15.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.311+0000 7f830c380640 1 Processor -- start 2023-10-30T14:48:15.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.311+0000 7f830c380640 1 -- start start 2023-10-30T14:48:15.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.311+0000 7f830c380640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8304153c80 0x7f830415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:15.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.311+0000 7f830c380640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83040c42a0 0x7f83040ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:15.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.311+0000 7f830c380640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8304159f20 con 0x7f83040c4a10 2023-10-30T14:48:15.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.315+0000 7f830c380640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f830415dec0 con 0x7f8304153c80 2023-10-30T14:48:15.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.315+0000 7f830c380640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83041599a0 con 0x7f83040c42a0 2023-10-30T14:48:15.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.315+0000 7f830a11c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8304153c80 0x7f830415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:15.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.315+0000 7f830a11c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8304153c80 0x7f830415cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37842/0 (socket says 172.21.15.57:37842) 2023-10-30T14:48:15.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.315+0000 7f830a11c640 1 -- 172.21.15.57:0/4046445191 learned_addr learned my addr 172.21.15.57:0/4046445191 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:15.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.315+0000 7f830a91d640 1 --2- 172.21.15.57:0/4046445191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83040c42a0 0x7f83040ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:15.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.315+0000 7f830a11c640 1 -- 172.21.15.57:0/4046445191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83040c42a0 msgr2=0x7f83040ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:15.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.315+0000 7f830a11c640 1 --2- 172.21.15.57:0/4046445191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83040c42a0 0x7f83040ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:15.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.315+0000 7f830a11c640 1 -- 172.21.15.57:0/4046445191 >> v1:172.21.15.57:6789/0 conn(0x7f83040c4a10 legacy=0x7f830415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:15.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.315+0000 7f830a11c640 1 -- 172.21.15.57:0/4046445191 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830415d620 con 0x7f8304153c80 2023-10-30T14:48:15.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.315+0000 7f830a11c640 1 --2- 172.21.15.57:0/4046445191 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8304153c80 0x7f830415cf90 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f8300009870 tx=0x7f830001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1c8af7135cce0ed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:15.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.319+0000 7f830911a640 1 -- 172.21.15.57:0/4046445191 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83000a8a30 con 0x7f8304153c80 2023-10-30T14:48:15.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.319+0000 7f830911a640 1 -- 172.21.15.57:0/4046445191 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83000a8b90 con 0x7f8304153c80 2023-10-30T14:48:15.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.319+0000 7f830911a640 1 -- 172.21.15.57:0/4046445191 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83000b0440 con 0x7f8304153c80 2023-10-30T14:48:15.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.319+0000 7f830c380640 1 -- 172.21.15.57:0/4046445191 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8304153c80 msgr2=0x7f830415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:15.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.319+0000 7f830c380640 1 --2- 172.21.15.57:0/4046445191 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8304153c80 0x7f830415cf90 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f8300009870 tx=0x7f830001ed30 comp rx=0 tx=0).stop 2023-10-30T14:48:15.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.319+0000 7f830c380640 1 -- 172.21.15.57:0/4046445191 shutdown_connections 2023-10-30T14:48:15.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.319+0000 7f830c380640 1 --2- 172.21.15.57:0/4046445191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83040c42a0 0x7f83040ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:15.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.319+0000 7f830c380640 1 --2- 172.21.15.57:0/4046445191 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8304153c80 0x7f830415cf90 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:15.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.319+0000 7f830c380640 1 -- 172.21.15.57:0/4046445191 >> 172.21.15.57:0/4046445191 conn(0x7f83040715c0 msgr2=0x7f83040719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:15.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.319+0000 7f830c380640 1 -- 172.21.15.57:0/4046445191 shutdown_connections 2023-10-30T14:48:15.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.319+0000 7f830c380640 1 -- 172.21.15.57:0/4046445191 wait complete. 2023-10-30T14:48:15.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830c380640 1 Processor -- start 2023-10-30T14:48:15.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830c380640 1 -- start start 2023-10-30T14:48:15.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830c380640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83040c42a0 0x7f8304155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:15.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830c380640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83040c4a10 0x7f830414f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:15.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830c380640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8304153c80 0x7f83041511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:15.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830c380640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83041f3320 con 0x7f8304153c80 2023-10-30T14:48:15.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830c380640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83041556d0 con 0x7f83040c42a0 2023-10-30T14:48:15.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830c380640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8304157ea0 con 0x7f83040c4a10 2023-10-30T14:48:15.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830a91d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8304153c80 0x7f83041511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:15.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830991b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83040c4a10 0x7f830414f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:15.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830a11c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83040c42a0 0x7f8304155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:15.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830a91d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8304153c80 0x7f83041511b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37854/0 (socket says 172.21.15.57:37854) 2023-10-30T14:48:15.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830a91d640 1 -- 172.21.15.57:0/2674165156 learned_addr learned my addr 172.21.15.57:0/2674165156 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:15.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830a91d640 1 -- 172.21.15.57:0/2674165156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83040c4a10 msgr2=0x7f830414f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:15.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830a91d640 1 --2- 172.21.15.57:0/2674165156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83040c4a10 0x7f830414f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:15.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830a91d640 1 -- 172.21.15.57:0/2674165156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83040c42a0 msgr2=0x7f8304155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:15.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830a91d640 1 --2- 172.21.15.57:0/2674165156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83040c42a0 0x7f8304155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:15.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830a91d640 1 -- 172.21.15.57:0/2674165156 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83000094d0 con 0x7f8304153c80 2023-10-30T14:48:15.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830a91d640 1 --2- 172.21.15.57:0/2674165156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8304153c80 0x7f83041511b0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f82fc007e40 tx=0x7f82fc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:15.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f82faffd640 1 -- 172.21.15.57:0/2674165156 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc014620 con 0x7f8304153c80 2023-10-30T14:48:15.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f82faffd640 1 -- 172.21.15.57:0/2674165156 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82fc00ee90 con 0x7f8304153c80 2023-10-30T14:48:15.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830c380640 1 -- 172.21.15.57:0/2674165156 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83041f2430 con 0x7f8304153c80 2023-10-30T14:48:15.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f82faffd640 1 -- 172.21.15.57:0/2674165156 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc0149e0 con 0x7f8304153c80 2023-10-30T14:48:15.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830c380640 1 -- 172.21.15.57:0/2674165156 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83041f29f0 con 0x7f8304153c80 2023-10-30T14:48:15.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.323+0000 7f830c380640 1 -- 172.21.15.57:0/2674165156 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830404e170 con 0x7f8304153c80 2023-10-30T14:48:15.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.327+0000 7f82faffd640 1 -- 172.21.15.57:0/2674165156 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f82fc014b40 con 0x7f8304153c80 2023-10-30T14:48:15.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.327+0000 7f82faffd640 1 --2- 172.21.15.57:0/2674165156 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f82e80664a0 0x7f82e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:15.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.327+0000 7f82faffd640 1 -- 172.21.15.57:0/2674165156 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f82fc084ca0 con 0x7f8304153c80 2023-10-30T14:48:15.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.327+0000 7f830a11c640 1 --2- 172.21.15.57:0/2674165156 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f82e80664a0 0x7f82e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:15.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.327+0000 7f830a11c640 1 --2- 172.21.15.57:0/2674165156 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f82e80664a0 0x7f82e8068960 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f83000b2000 tx=0x7f8300009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:15.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.339+0000 7f82faffd640 1 -- 172.21.15.57:0/2674165156 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f82fc051bf0 con 0x7f8304153c80 2023-10-30T14:48:15.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.539+0000 7f830c380640 1 -- 172.21.15.57:0/2674165156 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8304150f10 con 0x7f8304153c80 2023-10-30T14:48:15.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.539+0000 7f82faffd640 1 -- 172.21.15.57:0/2674165156 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6499 (secure 0 0 0) 0x7f82fc055210 con 0x7f8304153c80 2023-10-30T14:48:15.546 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:15.546 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7911,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:47:59.268639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:15.546 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 179 2023-10-30T14:48:15.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.543+0000 7f830c380640 1 -- 172.21.15.57:0/2674165156 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f82e80664a0 msgr2=0x7f82e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:15.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.543+0000 7f830c380640 1 --2- 172.21.15.57:0/2674165156 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f82e80664a0 0x7f82e8068960 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f83000b2000 tx=0x7f8300009040 comp rx=0 tx=0).stop 2023-10-30T14:48:15.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.543+0000 7f830c380640 1 -- 172.21.15.57:0/2674165156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8304153c80 msgr2=0x7f83041511b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:15.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.543+0000 7f830c380640 1 --2- 172.21.15.57:0/2674165156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8304153c80 0x7f83041511b0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f82fc007e40 tx=0x7f82fc00e4b0 comp rx=0 tx=0).stop 2023-10-30T14:48:15.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.543+0000 7f830c380640 1 -- 172.21.15.57:0/2674165156 shutdown_connections 2023-10-30T14:48:15.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.543+0000 7f830c380640 1 --2- 172.21.15.57:0/2674165156 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f82e80664a0 0x7f82e8068960 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:15.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.543+0000 7f830c380640 1 --2- 172.21.15.57:0/2674165156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8304153c80 0x7f83041511b0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:15.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.543+0000 7f830c380640 1 --2- 172.21.15.57:0/2674165156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83040c4a10 0x7f830414f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:15.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.543+0000 7f830c380640 1 --2- 172.21.15.57:0/2674165156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83040c42a0 0x7f8304155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:15.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.543+0000 7f830c380640 1 -- 172.21.15.57:0/2674165156 >> 172.21.15.57:0/2674165156 conn(0x7f83040715c0 msgr2=0x7f8304157670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:15.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.543+0000 7f830c380640 1 -- 172.21.15.57:0/2674165156 shutdown_connections 2023-10-30T14:48:15.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:15.543+0000 7f830c380640 1 -- 172.21.15.57:0/2674165156 wait complete. 2023-10-30T14:48:15.567 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:48:17.568 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:17.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.719+0000 7fb35cb01640 1 Processor -- start 2023-10-30T14:48:17.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.719+0000 7fb35cb01640 1 -- start start 2023-10-30T14:48:17.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.719+0000 7fb35cb01640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3580c4a10 0x7fb35815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:17.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.719+0000 7fb35cb01640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb3580c42a0 0x7fb3580ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:17.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.719+0000 7fb35cb01640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb358159f20 con 0x7fb358153c80 2023-10-30T14:48:17.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.719+0000 7fb35cb01640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb35815dec0 con 0x7fb3580c42a0 2023-10-30T14:48:17.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.719+0000 7fb35cb01640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3581599a0 con 0x7fb3580c4a10 2023-10-30T14:48:17.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.719+0000 7fb356d9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb3580c42a0 0x7fb3580ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:17.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.719+0000 7fb356d9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb3580c42a0 0x7fb3580ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37860/0 (socket says 172.21.15.57:37860) 2023-10-30T14:48:17.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.719+0000 7fb356d9d640 1 -- 172.21.15.57:0/1359669186 learned_addr learned my addr 172.21.15.57:0/1359669186 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:17.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.719+0000 7fb355d9b640 1 --2- 172.21.15.57:0/1359669186 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3580c4a10 0x7fb35815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:17.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.719+0000 7fb356d9d640 1 -- 172.21.15.57:0/1359669186 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3580c4a10 msgr2=0x7fb35815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:17.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.719+0000 7fb356d9d640 1 --2- 172.21.15.57:0/1359669186 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3580c4a10 0x7fb35815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:17.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.719+0000 7fb356d9d640 1 -- 172.21.15.57:0/1359669186 >> v1:172.21.15.88:6789/0 conn(0x7fb358153c80 legacy=0x7fb35815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:17.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.719+0000 7fb356d9d640 1 -- 172.21.15.57:0/1359669186 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35815d620 con 0x7fb3580c42a0 2023-10-30T14:48:17.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.723+0000 7fb356d9d640 1 --2- 172.21.15.57:0/1359669186 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3580c42a0 0x7fb3580ca1e0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fb350009e50 tx=0x7fb35001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70522241ff476889 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:17.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.723+0000 7fb35559a640 1 -- 172.21.15.57:0/1359669186 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3500af400 con 0x7fb3580c42a0 2023-10-30T14:48:17.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.723+0000 7fb35559a640 1 -- 172.21.15.57:0/1359669186 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3500ab050 con 0x7fb3580c42a0 2023-10-30T14:48:17.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.723+0000 7fb35559a640 1 -- 172.21.15.57:0/1359669186 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3500af780 con 0x7fb3580c42a0 2023-10-30T14:48:17.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.723+0000 7fb35cb01640 1 -- 172.21.15.57:0/1359669186 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3580c42a0 msgr2=0x7fb3580ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:17.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.723+0000 7fb35cb01640 1 --2- 172.21.15.57:0/1359669186 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3580c42a0 0x7fb3580ca1e0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fb350009e50 tx=0x7fb35001f1f0 comp rx=0 tx=0).stop 2023-10-30T14:48:17.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.723+0000 7fb35cb01640 1 -- 172.21.15.57:0/1359669186 shutdown_connections 2023-10-30T14:48:17.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.723+0000 7fb35cb01640 1 --2- 172.21.15.57:0/1359669186 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3580c42a0 0x7fb3580ca1e0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:17.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.723+0000 7fb35cb01640 1 --2- 172.21.15.57:0/1359669186 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3580c4a10 0x7fb35815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:17.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.723+0000 7fb35cb01640 1 -- 172.21.15.57:0/1359669186 >> 172.21.15.57:0/1359669186 conn(0x7fb3580715c0 msgr2=0x7fb3580719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:17.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.723+0000 7fb35cb01640 1 -- 172.21.15.57:0/1359669186 shutdown_connections 2023-10-30T14:48:17.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.723+0000 7fb35cb01640 1 -- 172.21.15.57:0/1359669186 wait complete. 2023-10-30T14:48:17.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb35cb01640 1 Processor -- start 2023-10-30T14:48:17.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb35cb01640 1 -- start start 2023-10-30T14:48:17.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb35cb01640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3580c42a0 0x7fb3581f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:17.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb35cb01640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3580c4a10 0x7fb3580b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:17.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb35cb01640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb358153c80 0x7fb35814f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:17.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb35cb01640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3580b2610 con 0x7fb3580c4a10 2023-10-30T14:48:17.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb35cb01640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3580b3a60 con 0x7fb358153c80 2023-10-30T14:48:17.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb35cb01640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3581f0630 con 0x7fb3580c42a0 2023-10-30T14:48:17.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb355d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3580c4a10 0x7fb3580b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:17.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb355d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3580c4a10 0x7fb3580b3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37874/0 (socket says 172.21.15.57:37874) 2023-10-30T14:48:17.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb355d9b640 1 -- 172.21.15.57:0/1836051109 learned_addr learned my addr 172.21.15.57:0/1836051109 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:17.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb35659c640 1 --2- 172.21.15.57:0/1836051109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3580c42a0 0x7fb3581f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:17.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb356d9d640 1 --2- 172.21.15.57:0/1836051109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb358153c80 0x7fb35814f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:17.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb355d9b640 1 -- 172.21.15.57:0/1836051109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3580c42a0 msgr2=0x7fb3581f00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:17.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb355d9b640 1 --2- 172.21.15.57:0/1836051109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3580c42a0 0x7fb3581f00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:17.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb355d9b640 1 -- 172.21.15.57:0/1836051109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb358153c80 msgr2=0x7fb35814f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:17.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb355d9b640 1 --2- 172.21.15.57:0/1836051109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb358153c80 0x7fb35814f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:17.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb355d9b640 1 -- 172.21.15.57:0/1836051109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb350009b30 con 0x7fb3580c4a10 2023-10-30T14:48:17.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb355d9b640 1 --2- 172.21.15.57:0/1836051109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3580c4a10 0x7fb3580b3080 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00e960 tx=0x7fb34c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:17.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb346ffd640 1 -- 172.21.15.57:0/1836051109 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c009690 con 0x7fb3580c4a10 2023-10-30T14:48:17.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb346ffd640 1 -- 172.21.15.57:0/1836051109 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34c00ca00 con 0x7fb3580c4a10 2023-10-30T14:48:17.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb346ffd640 1 -- 172.21.15.57:0/1836051109 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c00cdb0 con 0x7fb3580c4a10 2023-10-30T14:48:17.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb35cb01640 1 -- 172.21.15.57:0/1836051109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3580b3cd0 con 0x7fb3580c4a10 2023-10-30T14:48:17.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.727+0000 7fb35cb01640 1 -- 172.21.15.57:0/1836051109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3580b4570 con 0x7fb3580c4a10 2023-10-30T14:48:17.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.731+0000 7fb35cb01640 1 -- 172.21.15.57:0/1836051109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3580b40d0 con 0x7fb3580c4a10 2023-10-30T14:48:17.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.731+0000 7fb346ffd640 1 -- 172.21.15.57:0/1836051109 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb34c013470 con 0x7fb3580c4a10 2023-10-30T14:48:17.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.731+0000 7fb346ffd640 1 --2- 172.21.15.57:0/1836051109 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb3380664a0 0x7fb338068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:17.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.731+0000 7fb346ffd640 1 -- 172.21.15.57:0/1836051109 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7fb34c0845e0 con 0x7fb3580c4a10 2023-10-30T14:48:17.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.731+0000 7fb35659c640 1 --2- 172.21.15.57:0/1836051109 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb3380664a0 0x7fb338068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:17.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.731+0000 7fb35659c640 1 --2- 172.21.15.57:0/1836051109 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb3380664a0 0x7fb338068960 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fb348007fd0 tx=0x7fb348009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:17.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.735+0000 7fb346ffd640 1 -- 172.21.15.57:0/1836051109 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb34c051520 con 0x7fb3580c4a10 2023-10-30T14:48:17.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.923+0000 7fb35cb01640 1 -- 172.21.15.57:0/1836051109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb35804e170 con 0x7fb3580c4a10 2023-10-30T14:48:17.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.923+0000 7fb346ffd640 1 -- 172.21.15.57:0/1836051109 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6499 (secure 0 0 0) 0x7fb34c054b40 con 0x7fb3580c4a10 2023-10-30T14:48:17.929 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:17.929 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7911,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:47:59.268639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:17.930 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 179 2023-10-30T14:48:17.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.923+0000 7fb35cb01640 1 -- 172.21.15.57:0/1836051109 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb3380664a0 msgr2=0x7fb338068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:17.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.923+0000 7fb35cb01640 1 --2- 172.21.15.57:0/1836051109 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb3380664a0 0x7fb338068960 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fb348007fd0 tx=0x7fb348009040 comp rx=0 tx=0).stop 2023-10-30T14:48:17.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.923+0000 7fb35cb01640 1 -- 172.21.15.57:0/1836051109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3580c4a10 msgr2=0x7fb3580b3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:17.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.923+0000 7fb35cb01640 1 --2- 172.21.15.57:0/1836051109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3580c4a10 0x7fb3580b3080 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00e960 tx=0x7fb34c00ee20 comp rx=0 tx=0).stop 2023-10-30T14:48:17.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.923+0000 7fb35cb01640 1 -- 172.21.15.57:0/1836051109 shutdown_connections 2023-10-30T14:48:17.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.923+0000 7fb35cb01640 1 --2- 172.21.15.57:0/1836051109 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb3380664a0 0x7fb338068960 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:17.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.923+0000 7fb35cb01640 1 --2- 172.21.15.57:0/1836051109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb358153c80 0x7fb35814f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:17.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.923+0000 7fb35cb01640 1 --2- 172.21.15.57:0/1836051109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3580c4a10 0x7fb3580b3080 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:17.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.923+0000 7fb35cb01640 1 --2- 172.21.15.57:0/1836051109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3580c42a0 0x7fb3581f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:17.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.923+0000 7fb35cb01640 1 -- 172.21.15.57:0/1836051109 >> 172.21.15.57:0/1836051109 conn(0x7fb3580715c0 msgr2=0x7fb3580719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:17.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.927+0000 7fb35cb01640 1 -- 172.21.15.57:0/1836051109 shutdown_connections 2023-10-30T14:48:17.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:17.927+0000 7fb35cb01640 1 -- 172.21.15.57:0/1836051109 wait complete. 2023-10-30T14:48:17.950 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:48:19.951 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:20.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.103+0000 7fa9af4d6640 1 Processor -- start 2023-10-30T14:48:20.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.103+0000 7fa9af4d6640 1 -- start start 2023-10-30T14:48:20.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.103+0000 7fa9af4d6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa9a80c4a10 0x7fa9a815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:20.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.103+0000 7fa9af4d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa9a80c42a0 0x7fa9a80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:20.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.103+0000 7fa9af4d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a8159f20 con 0x7fa9a8153c80 2023-10-30T14:48:20.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.103+0000 7fa9af4d6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9a815dec0 con 0x7fa9a80c4a10 2023-10-30T14:48:20.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.103+0000 7fa9af4d6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9a81599a0 con 0x7fa9a80c42a0 2023-10-30T14:48:20.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.103+0000 7fa9aca71640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa9a80c4a10 0x7fa9a815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:20.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.103+0000 7fa9aca71640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa9a80c4a10 0x7fa9a815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37890/0 (socket says 172.21.15.57:37890) 2023-10-30T14:48:20.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.103+0000 7fa9aca71640 1 -- 172.21.15.57:0/438932146 learned_addr learned my addr 172.21.15.57:0/438932146 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:20.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.103+0000 7fa9ada73640 1 --2- 172.21.15.57:0/438932146 >> v2:172.21.15.88:3300/0 conn(0x7fa9a80c42a0 0x7fa9a80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:20.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.103+0000 7fa9aca71640 1 -- 172.21.15.57:0/438932146 >> v2:172.21.15.88:3300/0 conn(0x7fa9a80c42a0 msgr2=0x7fa9a80ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:20.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.103+0000 7fa9aca71640 1 --2- 172.21.15.57:0/438932146 >> v2:172.21.15.88:3300/0 conn(0x7fa9a80c42a0 0x7fa9a80ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:20.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.103+0000 7fa9aca71640 1 -- 172.21.15.57:0/438932146 >> v1:172.21.15.88:6789/0 conn(0x7fa9a8153c80 legacy=0x7fa9a815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:20.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.103+0000 7fa9aca71640 1 -- 172.21.15.57:0/438932146 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a815d620 con 0x7fa9a80c4a10 2023-10-30T14:48:20.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.107+0000 7fa9aca71640 1 --2- 172.21.15.57:0/438932146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9a80c4a10 0x7fa9a815cf90 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80b1d40 tx=0x7fa99801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36be4ddb34dc5b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:20.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.107+0000 7fa99ffff640 1 -- 172.21.15.57:0/438932146 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9980a89f0 con 0x7fa9a80c4a10 2023-10-30T14:48:20.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.107+0000 7fa99ffff640 1 -- 172.21.15.57:0/438932146 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9980a8b50 con 0x7fa9a80c4a10 2023-10-30T14:48:20.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.107+0000 7fa99ffff640 1 -- 172.21.15.57:0/438932146 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9980b0440 con 0x7fa9a80c4a10 2023-10-30T14:48:20.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.107+0000 7fa9af4d6640 1 -- 172.21.15.57:0/438932146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9a80c4a10 msgr2=0x7fa9a815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:20.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.107+0000 7fa9af4d6640 1 --2- 172.21.15.57:0/438932146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9a80c4a10 0x7fa9a815cf90 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80b1d40 tx=0x7fa99801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:48:20.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.107+0000 7fa9af4d6640 1 -- 172.21.15.57:0/438932146 shutdown_connections 2023-10-30T14:48:20.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.107+0000 7fa9af4d6640 1 --2- 172.21.15.57:0/438932146 >> v2:172.21.15.88:3300/0 conn(0x7fa9a80c42a0 0x7fa9a80ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:20.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.107+0000 7fa9af4d6640 1 --2- 172.21.15.57:0/438932146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9a80c4a10 0x7fa9a815cf90 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:20.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.107+0000 7fa9af4d6640 1 -- 172.21.15.57:0/438932146 >> 172.21.15.57:0/438932146 conn(0x7fa9a80715c0 msgr2=0x7fa9a80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:20.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.107+0000 7fa9af4d6640 1 -- 172.21.15.57:0/438932146 shutdown_connections 2023-10-30T14:48:20.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.107+0000 7fa9af4d6640 1 -- 172.21.15.57:0/438932146 wait complete. 2023-10-30T14:48:20.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9af4d6640 1 Processor -- start 2023-10-30T14:48:20.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9af4d6640 1 -- start start 2023-10-30T14:48:20.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9af4d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9a80c42a0 0x7fa9a8150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:20.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9af4d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9a80c4a10 0x7fa9a80cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:20.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9af4d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa9a8153c80 0x7fa9a8149e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:20.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9af4d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a8151940 con 0x7fa9a80c4a10 2023-10-30T14:48:20.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9af4d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a814fd40 con 0x7fa9a80c42a0 2023-10-30T14:48:20.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9af4d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a8150020 con 0x7fa9a8153c80 2023-10-30T14:48:20.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9aca71640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9a80c4a10 0x7fa9a80cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:20.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9aca71640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9a80c4a10 0x7fa9a80cd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37900/0 (socket says 172.21.15.57:37900) 2023-10-30T14:48:20.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9aca71640 1 -- 172.21.15.57:0/2815029180 learned_addr learned my addr 172.21.15.57:0/2815029180 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:20.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9ad272640 1 --2- 172.21.15.57:0/2815029180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9a80c42a0 0x7fa9a8150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:20.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9ada73640 1 --2- 172.21.15.57:0/2815029180 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa9a8153c80 0x7fa9a8149e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:20.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9aca71640 1 -- 172.21.15.57:0/2815029180 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa9a8153c80 msgr2=0x7fa9a8149e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:20.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9aca71640 1 --2- 172.21.15.57:0/2815029180 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa9a8153c80 0x7fa9a8149e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:20.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9aca71640 1 -- 172.21.15.57:0/2815029180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9a80c42a0 msgr2=0x7fa9a8150ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:20.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9aca71640 1 --2- 172.21.15.57:0/2815029180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9a80c42a0 0x7fa9a8150ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:20.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9aca71640 1 -- 172.21.15.57:0/2815029180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9980094d0 con 0x7fa9a80c4a10 2023-10-30T14:48:20.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9aca71640 1 --2- 172.21.15.57:0/2815029180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9a80c4a10 0x7fa9a80cd040 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fa9980b3040 tx=0x7fa99801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:20.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa99dffb640 1 -- 172.21.15.57:0/2815029180 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9980b0e00 con 0x7fa9a80c4a10 2023-10-30T14:48:20.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa99dffb640 1 -- 172.21.15.57:0/2815029180 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9980bb470 con 0x7fa9a80c4a10 2023-10-30T14:48:20.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9af4d6640 1 -- 172.21.15.57:0/2815029180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a81f1f10 con 0x7fa9a80c4a10 2023-10-30T14:48:20.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa99dffb640 1 -- 172.21.15.57:0/2815029180 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9980bb820 con 0x7fa9a80c4a10 2023-10-30T14:48:20.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.111+0000 7fa9af4d6640 1 -- 172.21.15.57:0/2815029180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a81f24d0 con 0x7fa9a80c4a10 2023-10-30T14:48:20.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.115+0000 7fa9af4d6640 1 -- 172.21.15.57:0/2815029180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a804e170 con 0x7fa9a80c4a10 2023-10-30T14:48:20.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.115+0000 7fa99dffb640 1 -- 172.21.15.57:0/2815029180 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa9980b7070 con 0x7fa9a80c4a10 2023-10-30T14:48:20.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.115+0000 7fa99dffb640 1 --2- 172.21.15.57:0/2815029180 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa98c0664a0 0x7fa98c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:20.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.115+0000 7fa99dffb640 1 -- 172.21.15.57:0/2815029180 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7fa9981228d0 con 0x7fa9a80c4a10 2023-10-30T14:48:20.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.115+0000 7fa9ad272640 1 --2- 172.21.15.57:0/2815029180 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa98c0664a0 0x7fa98c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:20.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.115+0000 7fa9ad272640 1 --2- 172.21.15.57:0/2815029180 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa98c0664a0 0x7fa98c068960 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4000f70 tx=0x7fa9a4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:20.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.119+0000 7fa99dffb640 1 -- 172.21.15.57:0/2815029180 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa9980ef640 con 0x7fa9a80c4a10 2023-10-30T14:48:20.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.307+0000 7fa9af4d6640 1 -- 172.21.15.57:0/2815029180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9a80cad40 con 0x7fa9a80c4a10 2023-10-30T14:48:20.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.307+0000 7fa99dffb640 1 -- 172.21.15.57:0/2815029180 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6499 (secure 0 0 0) 0x7fa9980b5020 con 0x7fa9a80c4a10 2023-10-30T14:48:20.314 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:20.314 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7911,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:47:59.268639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:20.315 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 179 2023-10-30T14:48:20.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.311+0000 7fa9af4d6640 1 -- 172.21.15.57:0/2815029180 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa98c0664a0 msgr2=0x7fa98c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:20.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.311+0000 7fa9af4d6640 1 --2- 172.21.15.57:0/2815029180 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa98c0664a0 0x7fa98c068960 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4000f70 tx=0x7fa9a4009040 comp rx=0 tx=0).stop 2023-10-30T14:48:20.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.311+0000 7fa9af4d6640 1 -- 172.21.15.57:0/2815029180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9a80c4a10 msgr2=0x7fa9a80cd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:20.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.311+0000 7fa9af4d6640 1 --2- 172.21.15.57:0/2815029180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9a80c4a10 0x7fa9a80cd040 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fa9980b3040 tx=0x7fa99801ba90 comp rx=0 tx=0).stop 2023-10-30T14:48:20.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.311+0000 7fa9af4d6640 1 -- 172.21.15.57:0/2815029180 shutdown_connections 2023-10-30T14:48:20.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.311+0000 7fa9af4d6640 1 --2- 172.21.15.57:0/2815029180 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa98c0664a0 0x7fa98c068960 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:20.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.311+0000 7fa9af4d6640 1 --2- 172.21.15.57:0/2815029180 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa9a8153c80 0x7fa9a8149e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:20.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.311+0000 7fa9af4d6640 1 --2- 172.21.15.57:0/2815029180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9a80c4a10 0x7fa9a80cd040 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:20.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.311+0000 7fa9af4d6640 1 --2- 172.21.15.57:0/2815029180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9a80c42a0 0x7fa9a8150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:20.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.311+0000 7fa9af4d6640 1 -- 172.21.15.57:0/2815029180 >> 172.21.15.57:0/2815029180 conn(0x7fa9a80715c0 msgr2=0x7fa9a80c4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:20.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.311+0000 7fa9af4d6640 1 -- 172.21.15.57:0/2815029180 shutdown_connections 2023-10-30T14:48:20.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:20.311+0000 7fa9af4d6640 1 -- 172.21.15.57:0/2815029180 wait complete. 2023-10-30T14:48:20.335 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:48:22.336 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:22.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.487+0000 7fba14765640 1 Processor -- start 2023-10-30T14:48:22.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.487+0000 7fba14765640 1 -- start start 2023-10-30T14:48:22.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.487+0000 7fba14765640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba0c0cb7d0 0x7fba0c0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:22.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.487+0000 7fba14765640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba0c0c2310 con 0x7fba0c150830 2023-10-30T14:48:22.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.487+0000 7fba14765640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba0c156710 con 0x7fba0c0c3e40 2023-10-30T14:48:22.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.487+0000 7fba14765640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c06bc10 con 0x7fba0c0cb7d0 2023-10-30T14:48:22.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.487+0000 7fba12501640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fba0c150830 0x7fba0c06b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:51226/0 (socket says 172.21.15.57:51226) 2023-10-30T14:48:22.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.487+0000 7fba12501640 1 -- 172.21.15.57:0/4262115499 learned_addr learned my addr 172.21.15.57:0/4262115499 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:48:22.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.487+0000 7fba11d00640 1 --2- 172.21.15.57:0/4262115499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba0c0cb7d0 0x7fba0c0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:22.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.491+0000 7fba114ff640 1 -- 172.21.15.57:0/4262115499 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700826270 0 0) 0x7fba0c0c2310 con 0x7fba0c150830 2023-10-30T14:48:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.491+0000 7fba114ff640 1 -- 172.21.15.57:0/4262115499 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9f8003530 con 0x7fba0c150830 2023-10-30T14:48:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.491+0000 7fba114ff640 1 -- 172.21.15.57:0/4262115499 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422873732 0 0) 0x7fba0c156710 con 0x7fba0c0c3e40 2023-10-30T14:48:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.491+0000 7fba11d00640 1 -- 172.21.15.57:0/4262115499 >> v1:172.21.15.88:6789/0 conn(0x7fba0c0c3e40 legacy=0x7fba0c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.491+0000 7fba11d00640 1 -- 172.21.15.57:0/4262115499 >> v1:172.21.15.57:6789/0 conn(0x7fba0c150830 legacy=0x7fba0c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.491+0000 7fba11d00640 1 -- 172.21.15.57:0/4262115499 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0c0c4360 con 0x7fba0c0cb7d0 2023-10-30T14:48:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.491+0000 7fba11d00640 1 --2- 172.21.15.57:0/4262115499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba0c0cb7d0 0x7fba0c0c3900 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc0097f0 tx=0x7fb9fc01eaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb0e0b5c41e26b4d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:22.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.491+0000 7fba114ff640 1 -- 172.21.15.57:0/4262115499 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba08001600 con 0x7fba0c0cb7d0 2023-10-30T14:48:22.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.491+0000 7fba114ff640 1 -- 172.21.15.57:0/4262115499 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9fc0a85b0 con 0x7fba0c0cb7d0 2023-10-30T14:48:22.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.491+0000 7fba114ff640 1 -- 172.21.15.57:0/4262115499 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc0a8920 con 0x7fba0c0cb7d0 2023-10-30T14:48:22.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba14765640 1 -- 172.21.15.57:0/4262115499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba0c0cb7d0 msgr2=0x7fba0c0c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:22.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba14765640 1 --2- 172.21.15.57:0/4262115499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba0c0cb7d0 0x7fba0c0c3900 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc0097f0 tx=0x7fb9fc01eaf0 comp rx=0 tx=0).stop 2023-10-30T14:48:22.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba14765640 1 -- 172.21.15.57:0/4262115499 shutdown_connections 2023-10-30T14:48:22.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba14765640 1 --2- 172.21.15.57:0/4262115499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba0c0cb7d0 0x7fba0c0c3900 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:22.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba14765640 1 -- 172.21.15.57:0/4262115499 >> 172.21.15.57:0/4262115499 conn(0x7fba0c0cac70 msgr2=0x7fba0c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:22.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba14765640 1 -- 172.21.15.57:0/4262115499 shutdown_connections 2023-10-30T14:48:22.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba14765640 1 -- 172.21.15.57:0/4262115499 wait complete. 2023-10-30T14:48:22.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba14765640 1 Processor -- start 2023-10-30T14:48:22.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba14765640 1 -- start start 2023-10-30T14:48:22.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba14765640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba0c0c3e40 0x7fba0c1428f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:22.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba14765640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba0c0cb7d0 0x7fba0c070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:22.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba14765640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba0c150830 0x7fba0c071000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:22.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba14765640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c0c1d90 con 0x7fba0c0cb7d0 2023-10-30T14:48:22.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba14765640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c154470 con 0x7fba0c150830 2023-10-30T14:48:22.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba14765640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c157990 con 0x7fba0c0c3e40 2023-10-30T14:48:22.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba11d00640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba0c0cb7d0 0x7fba0c070ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:22.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba11d00640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba0c0cb7d0 0x7fba0c070ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37912/0 (socket says 172.21.15.57:37912) 2023-10-30T14:48:22.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba11d00640 1 -- 172.21.15.57:0/1690232750 learned_addr learned my addr 172.21.15.57:0/1690232750 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:22.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba12501640 1 --2- 172.21.15.57:0/1690232750 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba0c0c3e40 0x7fba0c1428f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:22.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.495+0000 7fba12d02640 1 --2- 172.21.15.57:0/1690232750 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba0c150830 0x7fba0c071000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:22.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.499+0000 7fba11d00640 1 -- 172.21.15.57:0/1690232750 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba0c0c3e40 msgr2=0x7fba0c1428f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:22.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.499+0000 7fba11d00640 1 --2- 172.21.15.57:0/1690232750 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba0c0c3e40 0x7fba0c1428f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:22.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.499+0000 7fba11d00640 1 -- 172.21.15.57:0/1690232750 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba0c150830 msgr2=0x7fba0c071000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:22.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.499+0000 7fba11d00640 1 --2- 172.21.15.57:0/1690232750 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba0c150830 0x7fba0c071000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:22.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.499+0000 7fba11d00640 1 -- 172.21.15.57:0/1690232750 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9fc009420 con 0x7fba0c0cb7d0 2023-10-30T14:48:22.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.499+0000 7fba12501640 1 --2- 172.21.15.57:0/1690232750 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba0c0c3e40 0x7fba0c1428f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:48:22.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.499+0000 7fba12d02640 1 --2- 172.21.15.57:0/1690232750 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba0c150830 0x7fba0c071000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:48:22.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.499+0000 7fba11d00640 1 --2- 172.21.15.57:0/1690232750 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba0c0cb7d0 0x7fba0c070ac0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc000c10 tx=0x7fb9fc01b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:22.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.499+0000 7fba02ffd640 1 -- 172.21.15.57:0/1690232750 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc0a9c40 con 0x7fba0c0cb7d0 2023-10-30T14:48:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.499+0000 7fba02ffd640 1 -- 172.21.15.57:0/1690232750 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9fc0a9da0 con 0x7fba0c0cb7d0 2023-10-30T14:48:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.499+0000 7fba02ffd640 1 -- 172.21.15.57:0/1690232750 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc003820 con 0x7fba0c0cb7d0 2023-10-30T14:48:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.499+0000 7fba14765640 1 -- 172.21.15.57:0/1690232750 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba0c1f2f20 con 0x7fba0c0cb7d0 2023-10-30T14:48:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.499+0000 7fba14765640 1 -- 172.21.15.57:0/1690232750 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0c147ea0 con 0x7fba0c0cb7d0 2023-10-30T14:48:22.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.499+0000 7fba14765640 1 -- 172.21.15.57:0/1690232750 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba0c04e170 con 0x7fba0c0cb7d0 2023-10-30T14:48:22.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.499+0000 7fba02ffd640 1 -- 172.21.15.57:0/1690232750 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb9fc0b7040 con 0x7fba0c0cb7d0 2023-10-30T14:48:22.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.499+0000 7fba02ffd640 1 --2- 172.21.15.57:0/1690232750 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb9f00664a0 0x7fb9f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:22.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.499+0000 7fba02ffd640 1 -- 172.21.15.57:0/1690232750 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7fb9fc121c90 con 0x7fba0c0cb7d0 2023-10-30T14:48:22.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.499+0000 7fba12501640 1 --2- 172.21.15.57:0/1690232750 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb9f00664a0 0x7fb9f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:22.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.503+0000 7fba12501640 1 --2- 172.21.15.57:0/1690232750 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb9f00664a0 0x7fb9f0068960 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fba08000fc0 tx=0x7fba08001940 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:22.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.503+0000 7fba02ffd640 1 -- 172.21.15.57:0/1690232750 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb9fc0eebb0 con 0x7fba0c0cb7d0 2023-10-30T14:48:22.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.695+0000 7fba14765640 1 -- 172.21.15.57:0/1690232750 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba0c04e170 con 0x7fba0c0cb7d0 2023-10-30T14:48:22.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.695+0000 7fba02ffd640 1 -- 172.21.15.57:0/1690232750 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6499 (secure 0 0 0) 0x7fb9fc0b5090 con 0x7fba0c0cb7d0 2023-10-30T14:48:22.702 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:22.702 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7911,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:47:59.268639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:22.703 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 179 2023-10-30T14:48:22.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.699+0000 7fba14765640 1 -- 172.21.15.57:0/1690232750 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb9f00664a0 msgr2=0x7fb9f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:22.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.699+0000 7fba14765640 1 --2- 172.21.15.57:0/1690232750 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb9f00664a0 0x7fb9f0068960 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fba08000fc0 tx=0x7fba08001940 comp rx=0 tx=0).stop 2023-10-30T14:48:22.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.699+0000 7fba14765640 1 -- 172.21.15.57:0/1690232750 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba0c0cb7d0 msgr2=0x7fba0c070ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:22.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.699+0000 7fba14765640 1 --2- 172.21.15.57:0/1690232750 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba0c0cb7d0 0x7fba0c070ac0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc000c10 tx=0x7fb9fc01b890 comp rx=0 tx=0).stop 2023-10-30T14:48:22.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.699+0000 7fba14765640 1 -- 172.21.15.57:0/1690232750 shutdown_connections 2023-10-30T14:48:22.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.699+0000 7fba14765640 1 --2- 172.21.15.57:0/1690232750 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb9f00664a0 0x7fb9f0068960 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:22.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.699+0000 7fba14765640 1 --2- 172.21.15.57:0/1690232750 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba0c150830 0x7fba0c071000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:22.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.699+0000 7fba14765640 1 --2- 172.21.15.57:0/1690232750 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba0c0cb7d0 0x7fba0c070ac0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:22.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.699+0000 7fba14765640 1 --2- 172.21.15.57:0/1690232750 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fba0c0c3e40 0x7fba0c1428f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:22.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.699+0000 7fba14765640 1 -- 172.21.15.57:0/1690232750 >> 172.21.15.57:0/1690232750 conn(0x7fba0c0cac70 msgr2=0x7fba0c06bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:22.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.699+0000 7fba14765640 1 -- 172.21.15.57:0/1690232750 shutdown_connections 2023-10-30T14:48:22.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:22.699+0000 7fba14765640 1 -- 172.21.15.57:0/1690232750 wait complete. 2023-10-30T14:48:22.723 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:48:24.725 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:24.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe3929d3640 1 Processor -- start 2023-10-30T14:48:24.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe3929d3640 1 -- start start 2023-10-30T14:48:24.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe3929d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe38c0cb7d0 0x7fe38c0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:24.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe3929d3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe38c0c2310 con 0x7fe38c150830 2023-10-30T14:48:24.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe3929d3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe38c156710 con 0x7fe38c0c3e40 2023-10-30T14:48:24.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe3929d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c06bc10 con 0x7fe38c0cb7d0 2023-10-30T14:48:24.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38bfff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fe38c150830 0x7fe38c06b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:39926/0 (socket says 172.21.15.57:39926) 2023-10-30T14:48:24.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38bfff640 1 -- 172.21.15.57:0/2631995790 learned_addr learned my addr 172.21.15.57:0/2631995790 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:48:24.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38b7fe640 1 --2- 172.21.15.57:0/2631995790 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe38c0cb7d0 0x7fe38c0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:24.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38affd640 1 -- 172.21.15.57:0/2631995790 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242926647 0 0) 0x7fe38c0c2310 con 0x7fe38c150830 2023-10-30T14:48:24.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38affd640 1 -- 172.21.15.57:0/2631995790 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe374003530 con 0x7fe38c150830 2023-10-30T14:48:24.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38affd640 1 -- 172.21.15.57:0/2631995790 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1881916837 0 0) 0x7fe374003530 con 0x7fe38c150830 2023-10-30T14:48:24.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38affd640 1 -- 172.21.15.57:0/2631995790 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe38c0cb7d0 msgr2=0x7fe38c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:24.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38affd640 1 --2- 172.21.15.57:0/2631995790 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe38c0cb7d0 0x7fe38c0c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:24.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38affd640 1 -- 172.21.15.57:0/2631995790 >> v1:172.21.15.88:6789/0 conn(0x7fe38c0c3e40 legacy=0x7fe38c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:24.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38affd640 1 -- 172.21.15.57:0/2631995790 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38c0c4360 con 0x7fe38c150830 2023-10-30T14:48:24.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38affd640 1 -- 172.21.15.57:0/2631995790 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fe37c001680 con 0x7fe38c150830 2023-10-30T14:48:24.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38affd640 1 -- 172.21.15.57:0/2631995790 >> v1:172.21.15.57:6789/0 conn(0x7fe38c150830 legacy=0x7fe38c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:24.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38affd640 1 --2- 172.21.15.57:0/2631995790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3740039b0 0x7fe374003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:24.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38affd640 1 --2- 172.21.15.57:0/2631995790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3740047d0 0x7fe374004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:24.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38affd640 1 --2- 172.21.15.57:0/2631995790 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe38c0cb7d0 0x7fe374005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:24.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38affd640 1 -- 172.21.15.57:0/2631995790 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c06bc10 con 0x7fe3740047d0 2023-10-30T14:48:24.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38affd640 1 -- 172.21.15.57:0/2631995790 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37400dce0 con 0x7fe3740039b0 2023-10-30T14:48:24.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38affd640 1 -- 172.21.15.57:0/2631995790 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe37400dea0 con 0x7fe38c0cb7d0 2023-10-30T14:48:24.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38b7fe640 1 --2- 172.21.15.57:0/2631995790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3740047d0 0x7fe374004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:24.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe38bfff640 1 --2- 172.21.15.57:0/2631995790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3740039b0 0x7fe374003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:24.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.875+0000 7fe390f70640 1 --2- 172.21.15.57:0/2631995790 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe38c0cb7d0 0x7fe374005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:24.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.879+0000 7fe38b7fe640 1 -- 172.21.15.57:0/2631995790 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe38c0cb7d0 msgr2=0x7fe374005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:24.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.879+0000 7fe38b7fe640 1 --2- 172.21.15.57:0/2631995790 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe38c0cb7d0 0x7fe374005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:24.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.879+0000 7fe38b7fe640 1 -- 172.21.15.57:0/2631995790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3740039b0 msgr2=0x7fe374003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:24.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.879+0000 7fe38b7fe640 1 --2- 172.21.15.57:0/2631995790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3740039b0 0x7fe374003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:24.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.879+0000 7fe38b7fe640 1 -- 172.21.15.57:0/2631995790 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe374003530 con 0x7fe3740047d0 2023-10-30T14:48:24.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.879+0000 7fe38b7fe640 1 --2- 172.21.15.57:0/2631995790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3740047d0 0x7fe374004c50 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fe380009ab0 tx=0x7fe38001caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1b4483d1f4bf16 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:24.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.879+0000 7fe38affd640 1 -- 172.21.15.57:0/2631995790 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3800a5590 con 0x7fe3740047d0 2023-10-30T14:48:24.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.879+0000 7fe38affd640 1 -- 172.21.15.57:0/2631995790 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3800a56f0 con 0x7fe3740047d0 2023-10-30T14:48:24.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.879+0000 7fe38affd640 1 -- 172.21.15.57:0/2631995790 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3800a5aa0 con 0x7fe3740047d0 2023-10-30T14:48:24.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.879+0000 7fe3929d3640 1 -- 172.21.15.57:0/2631995790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3740047d0 msgr2=0x7fe374004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:24.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.879+0000 7fe3929d3640 1 --2- 172.21.15.57:0/2631995790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3740047d0 0x7fe374004c50 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fe380009ab0 tx=0x7fe38001caa0 comp rx=0 tx=0).stop 2023-10-30T14:48:24.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.879+0000 7fe390f70640 1 -- 172.21.15.57:0/2631995790 reap_dead start 2023-10-30T14:48:24.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.879+0000 7fe3929d3640 1 -- 172.21.15.57:0/2631995790 shutdown_connections 2023-10-30T14:48:24.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.879+0000 7fe3929d3640 1 -- 172.21.15.57:0/2631995790 >> 172.21.15.57:0/2631995790 conn(0x7fe38c0cac70 msgr2=0x7fe38c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:24.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.879+0000 7fe3929d3640 1 -- 172.21.15.57:0/2631995790 shutdown_connections 2023-10-30T14:48:24.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.879+0000 7fe3929d3640 1 -- 172.21.15.57:0/2631995790 wait complete. 2023-10-30T14:48:24.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe3929d3640 1 Processor -- start 2023-10-30T14:48:24.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe3929d3640 1 -- start start 2023-10-30T14:48:24.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe3929d3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3740047d0 0x7fe38c0c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:24.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe3929d3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe38c148660 0x7fe38c1ee260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:24.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe3929d3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe38c154400 0x7fe38c073620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:24.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe3929d3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c0ca720 con 0x7fe3740047d0 2023-10-30T14:48:24.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe3929d3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c0c90a0 con 0x7fe38c148660 2023-10-30T14:48:24.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe3929d3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c144660 con 0x7fe38c154400 2023-10-30T14:48:24.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe38bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3740047d0 0x7fe38c0c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:24.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe38bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3740047d0 0x7fe38c0c9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47482/0 (socket says 172.21.15.57:47482) 2023-10-30T14:48:24.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe38bfff640 1 -- 172.21.15.57:0/875551364 learned_addr learned my addr 172.21.15.57:0/875551364 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:24.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe38b7fe640 1 --2- 172.21.15.57:0/875551364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe38c148660 0x7fe38c1ee260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:24.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe390f70640 1 --2- 172.21.15.57:0/875551364 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe38c154400 0x7fe38c073620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:24.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe38bfff640 1 -- 172.21.15.57:0/875551364 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe38c154400 msgr2=0x7fe38c073620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:24.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe38bfff640 1 --2- 172.21.15.57:0/875551364 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe38c154400 0x7fe38c073620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:24.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe38bfff640 1 -- 172.21.15.57:0/875551364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe38c148660 msgr2=0x7fe38c1ee260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:24.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe38bfff640 1 --2- 172.21.15.57:0/875551364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe38c148660 0x7fe38c1ee260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:24.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe38bfff640 1 -- 172.21.15.57:0/875551364 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38c0ca930 con 0x7fe3740047d0 2023-10-30T14:48:24.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe38bfff640 1 --2- 172.21.15.57:0/875551364 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3740047d0 0x7fe38c0c9530 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0078c0 tx=0x7fe37c010670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:24.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe388ff9640 1 -- 172.21.15.57:0/875551364 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c019030 con 0x7fe3740047d0 2023-10-30T14:48:24.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe388ff9640 1 -- 172.21.15.57:0/875551364 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe37c014600 con 0x7fe3740047d0 2023-10-30T14:48:24.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe388ff9640 1 -- 172.21.15.57:0/875551364 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c0149b0 con 0x7fe3740047d0 2023-10-30T14:48:24.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.883+0000 7fe3929d3640 1 -- 172.21.15.57:0/875551364 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe38c0c8a50 con 0x7fe3740047d0 2023-10-30T14:48:24.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.887+0000 7fe3929d3640 1 -- 172.21.15.57:0/875551364 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe38c1477a0 con 0x7fe3740047d0 2023-10-30T14:48:24.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.887+0000 7fe3929d3640 1 -- 172.21.15.57:0/875551364 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe38c04e170 con 0x7fe3740047d0 2023-10-30T14:48:24.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.887+0000 7fe388ff9640 1 -- 172.21.15.57:0/875551364 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe37c01c070 con 0x7fe3740047d0 2023-10-30T14:48:24.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.887+0000 7fe388ff9640 1 --2- 172.21.15.57:0/875551364 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3780664d0 0x7fe378068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:24.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.887+0000 7fe388ff9640 1 -- 172.21.15.57:0/875551364 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7fe37c01a070 con 0x7fe3740047d0 2023-10-30T14:48:24.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.887+0000 7fe38b7fe640 1 --2- 172.21.15.57:0/875551364 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3780664d0 0x7fe378068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:24.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.891+0000 7fe38b7fe640 1 --2- 172.21.15.57:0/875551364 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3780664d0 0x7fe378068990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe3800af000 tx=0x7fe3800046d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:24.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:24.891+0000 7fe388ff9640 1 -- 172.21.15.57:0/875551364 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe37c051d90 con 0x7fe3740047d0 2023-10-30T14:48:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:25.083+0000 7fe3929d3640 1 -- 172.21.15.57:0/875551364 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe38c066510 con 0x7fe3740047d0 2023-10-30T14:48:25.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:25.083+0000 7fe388ff9640 1 -- 172.21.15.57:0/875551364 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6499 (secure 0 0 0) 0x7fe37c0553b0 con 0x7fe3740047d0 2023-10-30T14:48:25.090 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:25.090 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7911,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:47:59.268639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:25.090 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 179 2023-10-30T14:48:25.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:25.087+0000 7fe3929d3640 1 -- 172.21.15.57:0/875551364 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3780664d0 msgr2=0x7fe378068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:25.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:25.087+0000 7fe3929d3640 1 --2- 172.21.15.57:0/875551364 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3780664d0 0x7fe378068990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe3800af000 tx=0x7fe3800046d0 comp rx=0 tx=0).stop 2023-10-30T14:48:25.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:25.087+0000 7fe3929d3640 1 -- 172.21.15.57:0/875551364 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3740047d0 msgr2=0x7fe38c0c9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:25.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:25.087+0000 7fe3929d3640 1 --2- 172.21.15.57:0/875551364 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3740047d0 0x7fe38c0c9530 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0078c0 tx=0x7fe37c010670 comp rx=0 tx=0).stop 2023-10-30T14:48:25.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:25.087+0000 7fe3929d3640 1 -- 172.21.15.57:0/875551364 shutdown_connections 2023-10-30T14:48:25.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:25.087+0000 7fe3929d3640 1 --2- 172.21.15.57:0/875551364 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe3780664d0 0x7fe378068990 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:25.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:25.087+0000 7fe3929d3640 1 --2- 172.21.15.57:0/875551364 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe38c154400 0x7fe38c073620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:25.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:25.087+0000 7fe3929d3640 1 --2- 172.21.15.57:0/875551364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe38c148660 0x7fe38c1ee260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:25.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:25.087+0000 7fe3929d3640 1 --2- 172.21.15.57:0/875551364 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3740047d0 0x7fe38c0c9530 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:25.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:25.087+0000 7fe3929d3640 1 -- 172.21.15.57:0/875551364 >> 172.21.15.57:0/875551364 conn(0x7fe38c0cac70 msgr2=0x7fe38c14b760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:25.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:25.087+0000 7fe3929d3640 1 -- 172.21.15.57:0/875551364 shutdown_connections 2023-10-30T14:48:25.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:25.087+0000 7fe3929d3640 1 -- 172.21.15.57:0/875551364 wait complete. 2023-10-30T14:48:25.111 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:48:27.112 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:27.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.259+0000 7f6cea409640 1 Processor -- start 2023-10-30T14:48:27.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.263+0000 7f6cea409640 1 -- start start 2023-10-30T14:48:27.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.263+0000 7f6cea409640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6ce406dfc0 0x7f6ce406d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:27.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.263+0000 7f6cea409640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce406fc30 con 0x7f6ce40c5460 2023-10-30T14:48:27.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.263+0000 7f6cea409640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce415b360 con 0x7f6ce40706a0 2023-10-30T14:48:27.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.263+0000 7f6cea409640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce415b170 con 0x7f6ce406dfc0 2023-10-30T14:48:27.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.263+0000 7f6ce9407640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6ce406dfc0 0x7f6ce406d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:27.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.263+0000 7f6ce8c06640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f6ce40c5460 0x7f6ce415aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:39934/0 (socket says 172.21.15.57:39934) 2023-10-30T14:48:27.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.263+0000 7f6ce8c06640 1 -- 172.21.15.57:0/2917617567 learned_addr learned my addr 172.21.15.57:0/2917617567 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:48:27.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.263+0000 7f6ce9407640 1 -- 172.21.15.57:0/2917617567 >> v1:172.21.15.88:6789/0 conn(0x7f6ce40706a0 legacy=0x7f6ce40c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:27.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.263+0000 7f6ce9407640 1 -- 172.21.15.57:0/2917617567 >> v1:172.21.15.57:6789/0 conn(0x7f6ce40c5460 legacy=0x7f6ce415aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:27.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.263+0000 7f6cdbfff640 1 -- 172.21.15.57:0/2917617567 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4086700077 0 0) 0x7f6ce406fc30 con 0x7f6ce40c5460 2023-10-30T14:48:27.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.263+0000 7f6ce9407640 1 -- 172.21.15.57:0/2917617567 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce415b620 con 0x7f6ce406dfc0 2023-10-30T14:48:27.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.267+0000 7f6ce9407640 1 --2- 172.21.15.57:0/2917617567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce406dfc0 0x7f6ce406d4f0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00097c0 tx=0x7f6ce001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32ca1eeb9c4029f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:27.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.267+0000 7f6cdbfff640 1 -- 172.21.15.57:0/2917617567 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce00ab650 con 0x7f6ce406dfc0 2023-10-30T14:48:27.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.267+0000 7f6cdbfff640 1 -- 172.21.15.57:0/2917617567 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce00ab7b0 con 0x7f6ce406dfc0 2023-10-30T14:48:27.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.267+0000 7f6cdbfff640 1 -- 172.21.15.57:0/2917617567 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce00abb40 con 0x7f6ce406dfc0 2023-10-30T14:48:27.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.267+0000 7f6cea409640 1 -- 172.21.15.57:0/2917617567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce406dfc0 msgr2=0x7f6ce406d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:27.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.267+0000 7f6cea409640 1 --2- 172.21.15.57:0/2917617567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce406dfc0 0x7f6ce406d4f0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00097c0 tx=0x7f6ce001c450 comp rx=0 tx=0).stop 2023-10-30T14:48:27.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.267+0000 7f6cea409640 1 -- 172.21.15.57:0/2917617567 shutdown_connections 2023-10-30T14:48:27.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.267+0000 7f6cea409640 1 --2- 172.21.15.57:0/2917617567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce406dfc0 0x7f6ce406d4f0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:27.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.267+0000 7f6cea409640 1 -- 172.21.15.57:0/2917617567 >> 172.21.15.57:0/2917617567 conn(0x7f6ce40caae0 msgr2=0x7f6ce40caec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:27.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.267+0000 7f6cea409640 1 -- 172.21.15.57:0/2917617567 shutdown_connections 2023-10-30T14:48:27.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.267+0000 7f6cea409640 1 -- 172.21.15.57:0/2917617567 wait complete. 2023-10-30T14:48:27.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.267+0000 7f6cea409640 1 Processor -- start 2023-10-30T14:48:27.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.267+0000 7f6cea409640 1 -- start start 2023-10-30T14:48:27.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6cea409640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ce406dfc0 0x7f6ce4153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:27.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6cea409640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce40706a0 0x7f6ce41f5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:27.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6cea409640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ce40c5460 0x7f6ce4148b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:27.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6cea409640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce4073a60 con 0x7f6ce40706a0 2023-10-30T14:48:27.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6cea409640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce40c14a0 con 0x7f6ce406dfc0 2023-10-30T14:48:27.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6cea409640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce41f4cd0 con 0x7f6ce40c5460 2023-10-30T14:48:27.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6ce8c06640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce40706a0 0x7f6ce41f5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:27.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6ce8c06640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce40706a0 0x7f6ce41f5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47502/0 (socket says 172.21.15.57:47502) 2023-10-30T14:48:27.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6ce8c06640 1 -- 172.21.15.57:0/2670853974 learned_addr learned my addr 172.21.15.57:0/2670853974 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:27.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6ce9407640 1 --2- 172.21.15.57:0/2670853974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ce406dfc0 0x7f6ce4153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:27.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6ce9c08640 1 --2- 172.21.15.57:0/2670853974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ce40c5460 0x7f6ce4148b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:27.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6ce8c06640 1 -- 172.21.15.57:0/2670853974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ce40c5460 msgr2=0x7f6ce4148b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:27.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6ce8c06640 1 --2- 172.21.15.57:0/2670853974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ce40c5460 0x7f6ce4148b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:27.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6ce8c06640 1 -- 172.21.15.57:0/2670853974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ce406dfc0 msgr2=0x7f6ce4153590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:27.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6ce8c06640 1 --2- 172.21.15.57:0/2670853974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ce406dfc0 0x7f6ce4153590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:27.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6ce8c06640 1 -- 172.21.15.57:0/2670853974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce0009420 con 0x7f6ce40706a0 2023-10-30T14:48:27.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6ce8c06640 1 --2- 172.21.15.57:0/2670853974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce40706a0 0x7f6ce41f5160 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f6cd400c440 tx=0x7f6cd400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:27.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6cd9ffb640 1 -- 172.21.15.57:0/2670853974 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd40074e0 con 0x7f6ce40706a0 2023-10-30T14:48:27.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6cd9ffb640 1 -- 172.21.15.57:0/2670853974 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd4007640 con 0x7f6ce40706a0 2023-10-30T14:48:27.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6cd9ffb640 1 -- 172.21.15.57:0/2670853974 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4004a10 con 0x7f6ce40706a0 2023-10-30T14:48:27.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6cea409640 1 -- 172.21.15.57:0/2670853974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce41f56a0 con 0x7f6ce40706a0 2023-10-30T14:48:27.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6cea409640 1 -- 172.21.15.57:0/2670853974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce4073780 con 0x7f6ce40706a0 2023-10-30T14:48:27.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.271+0000 7f6cea409640 1 -- 172.21.15.57:0/2670853974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce404e170 con 0x7f6ce40706a0 2023-10-30T14:48:27.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.275+0000 7f6cd9ffb640 1 -- 172.21.15.57:0/2670853974 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6cd4004b70 con 0x7f6ce40706a0 2023-10-30T14:48:27.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.275+0000 7f6cd9ffb640 1 --2- 172.21.15.57:0/2670853974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6cc80664d0 0x7f6cc8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:27.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.275+0000 7f6cd9ffb640 1 -- 172.21.15.57:0/2670853974 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f6cd4083500 con 0x7f6ce40706a0 2023-10-30T14:48:27.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.275+0000 7f6ce9407640 1 --2- 172.21.15.57:0/2670853974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6cc80664d0 0x7f6cc8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:27.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.275+0000 7f6ce9407640 1 --2- 172.21.15.57:0/2670853974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6cc80664d0 0x7f6cc8068990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00af000 tx=0x7f6ce00a7000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:27.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.279+0000 7f6cd9ffb640 1 -- 172.21.15.57:0/2670853974 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6cd40503b0 con 0x7f6ce40706a0 2023-10-30T14:48:27.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.467+0000 7f6cea409640 1 -- 172.21.15.57:0/2670853974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ce41f4e10 con 0x7f6ce40706a0 2023-10-30T14:48:27.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.467+0000 7f6cd9ffb640 1 -- 172.21.15.57:0/2670853974 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6499 (secure 0 0 0) 0x7f6cd40539d0 con 0x7f6ce40706a0 2023-10-30T14:48:27.474 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:27.474 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7911,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:47:59.268639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:27.475 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 179 2023-10-30T14:48:27.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.471+0000 7f6cea409640 1 -- 172.21.15.57:0/2670853974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6cc80664d0 msgr2=0x7f6cc8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:27.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.471+0000 7f6cea409640 1 --2- 172.21.15.57:0/2670853974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6cc80664d0 0x7f6cc8068990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00af000 tx=0x7f6ce00a7000 comp rx=0 tx=0).stop 2023-10-30T14:48:27.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.471+0000 7f6cea409640 1 -- 172.21.15.57:0/2670853974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce40706a0 msgr2=0x7f6ce41f5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:27.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.471+0000 7f6cea409640 1 --2- 172.21.15.57:0/2670853974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce40706a0 0x7f6ce41f5160 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f6cd400c440 tx=0x7f6cd400c900 comp rx=0 tx=0).stop 2023-10-30T14:48:27.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.471+0000 7f6cea409640 1 -- 172.21.15.57:0/2670853974 shutdown_connections 2023-10-30T14:48:27.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.471+0000 7f6cea409640 1 --2- 172.21.15.57:0/2670853974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6cc80664d0 0x7f6cc8068990 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:27.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.471+0000 7f6cea409640 1 --2- 172.21.15.57:0/2670853974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ce40c5460 0x7f6ce4148b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:27.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.471+0000 7f6cea409640 1 --2- 172.21.15.57:0/2670853974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce40706a0 0x7f6ce41f5160 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:27.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.471+0000 7f6cea409640 1 --2- 172.21.15.57:0/2670853974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ce406dfc0 0x7f6ce4153590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:27.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.471+0000 7f6cea409640 1 -- 172.21.15.57:0/2670853974 >> 172.21.15.57:0/2670853974 conn(0x7f6ce40caae0 msgr2=0x7f6ce414b940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:27.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.471+0000 7f6cea409640 1 -- 172.21.15.57:0/2670853974 shutdown_connections 2023-10-30T14:48:27.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:27.471+0000 7f6cea409640 1 -- 172.21.15.57:0/2670853974 wait complete. 2023-10-30T14:48:27.495 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:48:29.497 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:29.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd8fd3640 1 Processor -- start 2023-10-30T14:48:29.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd8fd3640 1 -- start start 2023-10-30T14:48:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd8fd3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8cd4070520 0x7f8cd4070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd8fd3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd40c2310 con 0x7f8cd40cb7d0 2023-10-30T14:48:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd8fd3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd4156710 con 0x7f8cd4150830 2023-10-30T14:48:29.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd8fd3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd406bc10 con 0x7f8cd4070520 2023-10-30T14:48:29.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd1d9b640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f8cd40cb7d0 0x7f8cd40c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:39940/0 (socket says 172.21.15.57:39940) 2023-10-30T14:48:29.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd1d9b640 1 -- 172.21.15.57:0/2250624476 learned_addr learned my addr 172.21.15.57:0/2250624476 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:48:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd2d9d640 1 --2- 172.21.15.57:0/2250624476 >> v2:172.21.15.88:3300/0 conn(0x7f8cd4070520 0x7f8cd4070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd159a640 1 -- 172.21.15.57:0/2250624476 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529562747 0 0) 0x7f8cd40c2310 con 0x7f8cd40cb7d0 2023-10-30T14:48:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd159a640 1 -- 172.21.15.57:0/2250624476 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cbc005740 con 0x7f8cd40cb7d0 2023-10-30T14:48:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd159a640 1 -- 172.21.15.57:0/2250624476 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821745926 0 0) 0x7f8cd4156710 con 0x7f8cd4150830 2023-10-30T14:48:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd159a640 1 -- 172.21.15.57:0/2250624476 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd40c2310 con 0x7f8cd4150830 2023-10-30T14:48:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd159a640 1 -- 172.21.15.57:0/2250624476 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1908374607 0 0) 0x7f8cbc005740 con 0x7f8cd40cb7d0 2023-10-30T14:48:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd159a640 1 -- 172.21.15.57:0/2250624476 >> v2:172.21.15.88:3300/0 conn(0x7f8cd4070520 msgr2=0x7f8cd4070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:29.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd159a640 1 --2- 172.21.15.57:0/2250624476 >> v2:172.21.15.88:3300/0 conn(0x7f8cd4070520 0x7f8cd4070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:29.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd159a640 1 -- 172.21.15.57:0/2250624476 >> v1:172.21.15.88:6789/0 conn(0x7f8cd4150830 legacy=0x7f8cd406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:29.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd159a640 1 -- 172.21.15.57:0/2250624476 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd40c4360 con 0x7f8cd40cb7d0 2023-10-30T14:48:29.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd159a640 1 -- 172.21.15.57:0/2250624476 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f8cc80016a0 con 0x7f8cd40cb7d0 2023-10-30T14:48:29.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd159a640 1 -- 172.21.15.57:0/2250624476 >> v1:172.21.15.57:6789/0 conn(0x7f8cd40cb7d0 legacy=0x7f8cd40c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:29.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd159a640 1 --2- 172.21.15.57:0/2250624476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cbc005d20 0x7f8cbc006120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:29.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd159a640 1 --2- 172.21.15.57:0/2250624476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cbc0069e0 0x7f8cbc006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:29.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd159a640 1 --2- 172.21.15.57:0/2250624476 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8cbc0073a0 0x7f8cbc00dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:29.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd159a640 1 -- 172.21.15.57:0/2250624476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd406bc10 con 0x7f8cbc005d20 2023-10-30T14:48:29.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd159a640 1 -- 172.21.15.57:0/2250624476 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc00e190 con 0x7f8cbc0069e0 2023-10-30T14:48:29.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd159a640 1 -- 172.21.15.57:0/2250624476 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc00e350 con 0x7f8cbc0073a0 2023-10-30T14:48:29.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd259c640 1 --2- 172.21.15.57:0/2250624476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cbc005d20 0x7f8cbc006120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:29.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd1d9b640 1 --2- 172.21.15.57:0/2250624476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cbc0069e0 0x7f8cbc006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:29.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.647+0000 7f8cd2d9d640 1 --2- 172.21.15.57:0/2250624476 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8cbc0073a0 0x7f8cbc00dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:29.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.651+0000 7f8cd259c640 1 -- 172.21.15.57:0/2250624476 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8cbc0073a0 msgr2=0x7f8cbc00dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:29.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.651+0000 7f8cd259c640 1 --2- 172.21.15.57:0/2250624476 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8cbc0073a0 0x7f8cbc00dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:29.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.651+0000 7f8cd259c640 1 -- 172.21.15.57:0/2250624476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cbc0069e0 msgr2=0x7f8cbc006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:29.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.651+0000 7f8cd259c640 1 --2- 172.21.15.57:0/2250624476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cbc0069e0 0x7f8cbc006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:29.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.651+0000 7f8cd259c640 1 -- 172.21.15.57:0/2250624476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc005740 con 0x7f8cbc005d20 2023-10-30T14:48:29.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.651+0000 7f8cd2d9d640 1 --2- 172.21.15.57:0/2250624476 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8cbc0073a0 0x7f8cbc00dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:48:29.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.651+0000 7f8cd2d9d640 1 -- 172.21.15.57:0/2250624476 reap_dead start 2023-10-30T14:48:29.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.651+0000 7f8cd259c640 1 --2- 172.21.15.57:0/2250624476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cbc005d20 0x7f8cbc006120 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4007fd0 tx=0x7f8cc401ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8576ac6b4e8aec5f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:29.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.651+0000 7f8cd159a640 1 -- 172.21.15.57:0/2250624476 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc40052d0 con 0x7f8cbc005d20 2023-10-30T14:48:29.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.651+0000 7f8cd159a640 1 -- 172.21.15.57:0/2250624476 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc4005450 con 0x7f8cbc005d20 2023-10-30T14:48:29.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.651+0000 7f8cd159a640 1 -- 172.21.15.57:0/2250624476 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc40a8890 con 0x7f8cbc005d20 2023-10-30T14:48:29.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.651+0000 7f8cd8fd3640 1 -- 172.21.15.57:0/2250624476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cbc005d20 msgr2=0x7f8cbc006120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:29.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.651+0000 7f8cd8fd3640 1 --2- 172.21.15.57:0/2250624476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cbc005d20 0x7f8cbc006120 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4007fd0 tx=0x7f8cc401ec30 comp rx=0 tx=0).stop 2023-10-30T14:48:29.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.651+0000 7f8cd8fd3640 1 -- 172.21.15.57:0/2250624476 shutdown_connections 2023-10-30T14:48:29.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.651+0000 7f8cd8fd3640 1 --2- 172.21.15.57:0/2250624476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cbc005d20 0x7f8cbc006120 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:29.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.651+0000 7f8cd8fd3640 1 -- 172.21.15.57:0/2250624476 >> 172.21.15.57:0/2250624476 conn(0x7f8cd40cac70 msgr2=0x7f8cd40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:29.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd8fd3640 1 -- 172.21.15.57:0/2250624476 shutdown_connections 2023-10-30T14:48:29.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd8fd3640 1 -- 172.21.15.57:0/2250624476 wait complete. 2023-10-30T14:48:29.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd8fd3640 1 Processor -- start 2023-10-30T14:48:29.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd8fd3640 1 -- start start 2023-10-30T14:48:29.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd8fd3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8cbc005d20 0x7f8cd41e3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:29.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd8fd3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cd40ca720 0x7f8cd41e2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:29.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd8fd3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cd41561b0 0x7f8cd40c48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:29.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd8fd3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd41e2860 con 0x7f8cd41561b0 2023-10-30T14:48:29.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd8fd3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd41e3a60 con 0x7f8cd40ca720 2023-10-30T14:48:29.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd8fd3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd4154550 con 0x7f8cbc005d20 2023-10-30T14:48:29.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cd41561b0 0x7f8cd40c48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:29.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cd41561b0 0x7f8cd40c48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47516/0 (socket says 172.21.15.57:47516) 2023-10-30T14:48:29.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd2d9d640 1 -- 172.21.15.57:0/1609610549 learned_addr learned my addr 172.21.15.57:0/1609610549 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:29.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd259c640 1 --2- 172.21.15.57:0/1609610549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8cbc005d20 0x7f8cd41e3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:29.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd1d9b640 1 --2- 172.21.15.57:0/1609610549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cd40ca720 0x7f8cd41e2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:29.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd2d9d640 1 -- 172.21.15.57:0/1609610549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8cbc005d20 msgr2=0x7f8cd41e3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:29.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd2d9d640 1 --2- 172.21.15.57:0/1609610549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8cbc005d20 0x7f8cd41e3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:29.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd2d9d640 1 -- 172.21.15.57:0/1609610549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cd40ca720 msgr2=0x7f8cd41e2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:29.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd2d9d640 1 --2- 172.21.15.57:0/1609610549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cd40ca720 0x7f8cd41e2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:29.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd2d9d640 1 -- 172.21.15.57:0/1609610549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd4154830 con 0x7f8cd41561b0 2023-10-30T14:48:29.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd1d9b640 1 --2- 172.21.15.57:0/1609610549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cd40ca720 0x7f8cd41e2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:48:29.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.655+0000 7f8cd259c640 1 --2- 172.21.15.57:0/1609610549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8cbc005d20 0x7f8cd41e3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:48:29.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.659+0000 7f8cd2d9d640 1 --2- 172.21.15.57:0/1609610549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cd41561b0 0x7f8cd40c48a0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f8cd41541f0 tx=0x7f8ccc010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:29.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.659+0000 7f8cc2ffd640 1 -- 172.21.15.57:0/1609610549 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc01b070 con 0x7f8cd41561b0 2023-10-30T14:48:29.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.659+0000 7f8cc2ffd640 1 -- 172.21.15.57:0/1609610549 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ccc00f040 con 0x7f8cd41561b0 2023-10-30T14:48:29.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.659+0000 7f8cc2ffd640 1 -- 172.21.15.57:0/1609610549 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc00e050 con 0x7f8cd41561b0 2023-10-30T14:48:29.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.659+0000 7f8cd8fd3640 1 -- 172.21.15.57:0/1609610549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd4154b10 con 0x7f8cd41561b0 2023-10-30T14:48:29.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.659+0000 7f8cd8fd3640 1 -- 172.21.15.57:0/1609610549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd41e3230 con 0x7f8cd41561b0 2023-10-30T14:48:29.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.659+0000 7f8cd8fd3640 1 -- 172.21.15.57:0/1609610549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd41e2a10 con 0x7f8cd41561b0 2023-10-30T14:48:29.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.659+0000 7f8cc2ffd640 1 -- 172.21.15.57:0/1609610549 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8ccc003be0 con 0x7f8cd41561b0 2023-10-30T14:48:29.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.659+0000 7f8cc2ffd640 1 --2- 172.21.15.57:0/1609610549 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8cb40664d0 0x7f8cb4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:29.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.659+0000 7f8cc2ffd640 1 -- 172.21.15.57:0/1609610549 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f8ccc084b50 con 0x7f8cd41561b0 2023-10-30T14:48:29.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.659+0000 7f8cd259c640 1 --2- 172.21.15.57:0/1609610549 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8cb40664d0 0x7f8cb4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:29.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.663+0000 7f8cd259c640 1 --2- 172.21.15.57:0/1609610549 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8cb40664d0 0x7f8cb4068990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4007fd0 tx=0x7f8cc40aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:29.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.663+0000 7f8cc2ffd640 1 -- 172.21.15.57:0/1609610549 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8cd41e2a10 con 0x7f8cd41561b0 2023-10-30T14:48:29.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.859+0000 7f8cd8fd3640 1 -- 172.21.15.57:0/1609610549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cd404e170 con 0x7f8cd41561b0 2023-10-30T14:48:29.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.859+0000 7f8cc2ffd640 1 -- 172.21.15.57:0/1609610549 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6499 (secure 0 0 0) 0x7f8ccc0519e0 con 0x7f8cd41561b0 2023-10-30T14:48:29.865 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:29.866 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":7911,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:47:59.268639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7536},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7536":{"gid":7536,"name":"e","rank":0,"incarnation":164,"state":"up:active","state_seq":18,"addr":"172.21.15.57:6837/816502245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":816502245},{"type":"v1","addr":"172.21.15.57:6837","nonce":816502245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:29.866 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 179 2023-10-30T14:48:29.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.863+0000 7f8cd8fd3640 1 -- 172.21.15.57:0/1609610549 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8cb40664d0 msgr2=0x7f8cb4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:29.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.863+0000 7f8cd8fd3640 1 --2- 172.21.15.57:0/1609610549 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8cb40664d0 0x7f8cb4068990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4007fd0 tx=0x7f8cc40aa070 comp rx=0 tx=0).stop 2023-10-30T14:48:29.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.863+0000 7f8cd8fd3640 1 -- 172.21.15.57:0/1609610549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cd41561b0 msgr2=0x7f8cd40c48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:29.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.863+0000 7f8cd8fd3640 1 --2- 172.21.15.57:0/1609610549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cd41561b0 0x7f8cd40c48a0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f8cd41541f0 tx=0x7f8ccc010590 comp rx=0 tx=0).stop 2023-10-30T14:48:29.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.863+0000 7f8cd8fd3640 1 -- 172.21.15.57:0/1609610549 shutdown_connections 2023-10-30T14:48:29.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.863+0000 7f8cd8fd3640 1 --2- 172.21.15.57:0/1609610549 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8cb40664d0 0x7f8cb4068990 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:29.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.863+0000 7f8cd8fd3640 1 --2- 172.21.15.57:0/1609610549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cd41561b0 0x7f8cd40c48a0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:29.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.863+0000 7f8cd8fd3640 1 --2- 172.21.15.57:0/1609610549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cd40ca720 0x7f8cd41e2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:29.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.863+0000 7f8cd8fd3640 1 --2- 172.21.15.57:0/1609610549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8cbc005d20 0x7f8cd41e3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:29.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.863+0000 7f8cd8fd3640 1 -- 172.21.15.57:0/1609610549 >> 172.21.15.57:0/1609610549 conn(0x7f8cd40cac70 msgr2=0x7f8cd40c5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:29.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.863+0000 7f8cd8fd3640 1 -- 172.21.15.57:0/1609610549 shutdown_connections 2023-10-30T14:48:29.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:29.863+0000 7f8cd8fd3640 1 -- 172.21.15.57:0/1609610549 wait complete. 2023-10-30T14:48:29.887 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:48:31.889 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:32.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.043+0000 7fbd464d6640 1 Processor -- start 2023-10-30T14:48:32.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.043+0000 7fbd464d6640 1 -- start start 2023-10-30T14:48:32.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.043+0000 7fbd464d6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbd40151c20 0x7fbd400c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:32.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.043+0000 7fbd464d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbd400cb8b0 0x7fbd4006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:32.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.043+0000 7fbd464d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd400c5030 con 0x7fbd40151840 2023-10-30T14:48:32.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.043+0000 7fbd464d6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd40157830 con 0x7fbd40151c20 2023-10-30T14:48:32.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.043+0000 7fbd464d6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd40157b10 con 0x7fbd400cb8b0 2023-10-30T14:48:32.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.043+0000 7fbd3f7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbd40151c20 0x7fbd400c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:32.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.043+0000 7fbd3f7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbd40151c20 0x7fbd400c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47532/0 (socket says 172.21.15.57:47532) 2023-10-30T14:48:32.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.043+0000 7fbd3f7fe640 1 -- 172.21.15.57:0/2675363017 learned_addr learned my addr 172.21.15.57:0/2675363017 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:32.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.043+0000 7fbd44a73640 1 --2- 172.21.15.57:0/2675363017 >> v2:172.21.15.88:3300/0 conn(0x7fbd400cb8b0 0x7fbd4006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:32.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.043+0000 7fbd3f7fe640 1 -- 172.21.15.57:0/2675363017 >> v2:172.21.15.88:3300/0 conn(0x7fbd400cb8b0 msgr2=0x7fbd4006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:32.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.043+0000 7fbd3f7fe640 1 --2- 172.21.15.57:0/2675363017 >> v2:172.21.15.88:3300/0 conn(0x7fbd400cb8b0 0x7fbd4006b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.043+0000 7fbd3f7fe640 1 -- 172.21.15.57:0/2675363017 >> v1:172.21.15.88:6789/0 conn(0x7fbd40151840 legacy=0x7fbd40072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:32.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.043+0000 7fbd3f7fe640 1 -- 172.21.15.57:0/2675363017 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd40157df0 con 0x7fbd40151c20 2023-10-30T14:48:32.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.047+0000 7fbd3f7fe640 1 --2- 172.21.15.57:0/2675363017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd40151c20 0x7fbd400c49a0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fbd400c6cb0 tx=0x7fbd3401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc34f1907eed9383 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:32.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.047+0000 7fbd3effd640 1 -- 172.21.15.57:0/2675363017 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd340a89b0 con 0x7fbd40151c20 2023-10-30T14:48:32.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.047+0000 7fbd3effd640 1 -- 172.21.15.57:0/2675363017 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd340a8b10 con 0x7fbd40151c20 2023-10-30T14:48:32.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.047+0000 7fbd3effd640 1 -- 172.21.15.57:0/2675363017 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd340a8ea0 con 0x7fbd40151c20 2023-10-30T14:48:32.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.047+0000 7fbd464d6640 1 -- 172.21.15.57:0/2675363017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd40151c20 msgr2=0x7fbd400c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:32.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.047+0000 7fbd464d6640 1 --2- 172.21.15.57:0/2675363017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd40151c20 0x7fbd400c49a0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fbd400c6cb0 tx=0x7fbd3401ecb0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.047+0000 7fbd464d6640 1 -- 172.21.15.57:0/2675363017 shutdown_connections 2023-10-30T14:48:32.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.047+0000 7fbd464d6640 1 --2- 172.21.15.57:0/2675363017 >> v2:172.21.15.88:3300/0 conn(0x7fbd400cb8b0 0x7fbd4006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.047+0000 7fbd464d6640 1 --2- 172.21.15.57:0/2675363017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd40151c20 0x7fbd400c49a0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.047+0000 7fbd464d6640 1 -- 172.21.15.57:0/2675363017 >> 172.21.15.57:0/2675363017 conn(0x7fbd400cac70 msgr2=0x7fbd400cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:32.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.047+0000 7fbd464d6640 1 -- 172.21.15.57:0/2675363017 shutdown_connections 2023-10-30T14:48:32.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.047+0000 7fbd464d6640 1 -- 172.21.15.57:0/2675363017 wait complete. 2023-10-30T14:48:32.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.047+0000 7fbd464d6640 1 Processor -- start 2023-10-30T14:48:32.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd464d6640 1 -- start start 2023-10-30T14:48:32.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd464d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd400cb8b0 0x7fbd401f0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:32.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd464d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd40151840 0x7fbd400c7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:32.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd464d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd40151c20 0x7fbd400c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:32.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd464d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd401546e0 con 0x7fbd40151c20 2023-10-30T14:48:32.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd464d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd401487b0 con 0x7fbd400cb8b0 2023-10-30T14:48:32.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd464d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd401f1130 con 0x7fbd40151840 2023-10-30T14:48:32.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd44a73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd40151c20 0x7fbd400c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:32.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd44a73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd40151c20 0x7fbd400c7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47546/0 (socket says 172.21.15.57:47546) 2023-10-30T14:48:32.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd44a73640 1 -- 172.21.15.57:0/246705771 learned_addr learned my addr 172.21.15.57:0/246705771 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:32.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd3ffff640 1 --2- 172.21.15.57:0/246705771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd400cb8b0 0x7fbd401f0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:32.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd3f7fe640 1 --2- 172.21.15.57:0/246705771 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd40151840 0x7fbd400c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:32.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd44a73640 1 -- 172.21.15.57:0/246705771 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd40151840 msgr2=0x7fbd400c7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:32.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd44a73640 1 --2- 172.21.15.57:0/246705771 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd40151840 0x7fbd400c7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd44a73640 1 -- 172.21.15.57:0/246705771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd400cb8b0 msgr2=0x7fbd401f0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:32.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd44a73640 1 --2- 172.21.15.57:0/246705771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd400cb8b0 0x7fbd401f0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd44a73640 1 -- 172.21.15.57:0/246705771 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd340094d0 con 0x7fbd40151c20 2023-10-30T14:48:32.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd3f7fe640 1 --2- 172.21.15.57:0/246705771 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd40151840 0x7fbd400c7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:48:32.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd44a73640 1 --2- 172.21.15.57:0/246705771 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd40151c20 0x7fbd400c7bc0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fbd38007e40 tx=0x7fbd3800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:32.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd3cff9640 1 -- 172.21.15.57:0/246705771 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd38014620 con 0x7fbd40151c20 2023-10-30T14:48:32.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd3cff9640 1 -- 172.21.15.57:0/246705771 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd380147a0 con 0x7fbd40151c20 2023-10-30T14:48:32.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd3cff9640 1 -- 172.21.15.57:0/246705771 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd38014b50 con 0x7fbd40151c20 2023-10-30T14:48:32.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd464d6640 1 -- 172.21.15.57:0/246705771 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd40144d70 con 0x7fbd40151c20 2023-10-30T14:48:32.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd464d6640 1 -- 172.21.15.57:0/246705771 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd401452a0 con 0x7fbd40151c20 2023-10-30T14:48:32.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.051+0000 7fbd464d6640 1 -- 172.21.15.57:0/246705771 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd4004e170 con 0x7fbd40151c20 2023-10-30T14:48:32.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.055+0000 7fbd3cff9640 1 -- 172.21.15.57:0/246705771 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbd38014cb0 con 0x7fbd40151c20 2023-10-30T14:48:32.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.055+0000 7fbd3cff9640 1 --2- 172.21.15.57:0/246705771 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd2c0664d0 0x7fbd2c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:32.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.055+0000 7fbd3cff9640 1 -- 172.21.15.57:0/246705771 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7fbd38084e30 con 0x7fbd40151c20 2023-10-30T14:48:32.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.055+0000 7fbd3ffff640 1 --2- 172.21.15.57:0/246705771 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd2c0664d0 0x7fbd2c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:32.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.055+0000 7fbd3ffff640 1 --2- 172.21.15.57:0/246705771 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd2c0664d0 0x7fbd2c068990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fbd30007fd0 tx=0x7fbd30007e40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:32.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.059+0000 7fbd3cff9640 1 -- 172.21.15.57:0/246705771 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbd38051be0 con 0x7fbd40151c20 2023-10-30T14:48:32.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.247+0000 7fbd464d6640 1 -- 172.21.15.57:0/246705771 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd400c41c0 con 0x7fbd40151c20 2023-10-30T14:48:32.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.247+0000 7fbd3cff9640 1 -- 172.21.15.57:0/246705771 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5709 (secure 0 0 0) 0x7fbd38055200 con 0x7fbd40151c20 2023-10-30T14:48:32.254 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:32.254 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:30.252464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:32.255 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 180 2023-10-30T14:48:32.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.251+0000 7fbd464d6640 1 -- 172.21.15.57:0/246705771 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd2c0664d0 msgr2=0x7fbd2c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:32.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.251+0000 7fbd464d6640 1 --2- 172.21.15.57:0/246705771 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd2c0664d0 0x7fbd2c068990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fbd30007fd0 tx=0x7fbd30007e40 comp rx=0 tx=0).stop 2023-10-30T14:48:32.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.251+0000 7fbd464d6640 1 -- 172.21.15.57:0/246705771 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd40151c20 msgr2=0x7fbd400c7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:32.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.251+0000 7fbd464d6640 1 --2- 172.21.15.57:0/246705771 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd40151c20 0x7fbd400c7bc0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fbd38007e40 tx=0x7fbd3800e4b0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.251+0000 7fbd464d6640 1 -- 172.21.15.57:0/246705771 shutdown_connections 2023-10-30T14:48:32.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.251+0000 7fbd464d6640 1 --2- 172.21.15.57:0/246705771 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd2c0664d0 0x7fbd2c068990 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.251+0000 7fbd464d6640 1 --2- 172.21.15.57:0/246705771 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd40151c20 0x7fbd400c7bc0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.251+0000 7fbd464d6640 1 --2- 172.21.15.57:0/246705771 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd40151840 0x7fbd400c7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.251+0000 7fbd464d6640 1 --2- 172.21.15.57:0/246705771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd400cb8b0 0x7fbd401f0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.251+0000 7fbd464d6640 1 -- 172.21.15.57:0/246705771 >> 172.21.15.57:0/246705771 conn(0x7fbd400cac70 msgr2=0x7fbd40073700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:32.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.251+0000 7fbd464d6640 1 -- 172.21.15.57:0/246705771 shutdown_connections 2023-10-30T14:48:32.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.251+0000 7fbd464d6640 1 -- 172.21.15.57:0/246705771 wait complete. 2023-10-30T14:48:32.276 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T14:48:32.276 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:48:32.276 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:32.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.415+0000 7ff260a46640 1 Processor -- start 2023-10-30T14:48:32.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.415+0000 7ff260a46640 1 -- start start 2023-10-30T14:48:32.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.415+0000 7ff260a46640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff25c1537f0 0x7ff25c0701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:32.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.415+0000 7ff260a46640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff25c15a2e0 0x7ff25c0ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:32.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.415+0000 7ff260a46640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25c0c2aa0 con 0x7ff25c0cd4a0 2023-10-30T14:48:32.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.415+0000 7ff260a46640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff25c0cd8b0 con 0x7ff25c15a2e0 2023-10-30T14:48:32.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.415+0000 7ff260a46640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff25c0c3ab0 con 0x7ff25c1537f0 2023-10-30T14:48:32.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.415+0000 7ff253fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff25c15a2e0 0x7ff25c0ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:32.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.419+0000 7ff253fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff25c15a2e0 0x7ff25c0ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47552/0 (socket says 172.21.15.57:47552) 2023-10-30T14:48:32.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.419+0000 7ff253fff640 1 -- 172.21.15.57:0/2746134698 learned_addr learned my addr 172.21.15.57:0/2746134698 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:32.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.419+0000 7ff25a59c640 1 --2- 172.21.15.57:0/2746134698 >> v2:172.21.15.88:3300/0 conn(0x7ff25c1537f0 0x7ff25c0701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:32.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.419+0000 7ff253fff640 1 -- 172.21.15.57:0/2746134698 >> v2:172.21.15.88:3300/0 conn(0x7ff25c1537f0 msgr2=0x7ff25c0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:32.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.419+0000 7ff253fff640 1 --2- 172.21.15.57:0/2746134698 >> v2:172.21.15.88:3300/0 conn(0x7ff25c1537f0 0x7ff25c0701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.419+0000 7ff253fff640 1 -- 172.21.15.57:0/2746134698 >> v1:172.21.15.88:6789/0 conn(0x7ff25c0cd4a0 legacy=0x7ff25c06b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:32.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.419+0000 7ff253fff640 1 -- 172.21.15.57:0/2746134698 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25c0708e0 con 0x7ff25c15a2e0 2023-10-30T14:48:32.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.419+0000 7ff253fff640 1 --2- 172.21.15.57:0/2746134698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff25c15a2e0 0x7ff25c0ccf60 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0cdb70 tx=0x7ff24401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12365d98159c2a45 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:32.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.419+0000 7ff259d9b640 1 -- 172.21.15.57:0/2746134698 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2440a89f0 con 0x7ff25c15a2e0 2023-10-30T14:48:32.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.419+0000 7ff259d9b640 1 -- 172.21.15.57:0/2746134698 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2440a8b50 con 0x7ff25c15a2e0 2023-10-30T14:48:32.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.419+0000 7ff259d9b640 1 -- 172.21.15.57:0/2746134698 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2440b0440 con 0x7ff25c15a2e0 2023-10-30T14:48:32.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff260a46640 1 -- 172.21.15.57:0/2746134698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff25c15a2e0 msgr2=0x7ff25c0ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:32.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff260a46640 1 --2- 172.21.15.57:0/2746134698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff25c15a2e0 0x7ff25c0ccf60 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0cdb70 tx=0x7ff24401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff260a46640 1 -- 172.21.15.57:0/2746134698 shutdown_connections 2023-10-30T14:48:32.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff260a46640 1 --2- 172.21.15.57:0/2746134698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff25c15a2e0 0x7ff25c0ccf60 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff260a46640 1 --2- 172.21.15.57:0/2746134698 >> v2:172.21.15.88:3300/0 conn(0x7ff25c1537f0 0x7ff25c0701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff260a46640 1 -- 172.21.15.57:0/2746134698 >> 172.21.15.57:0/2746134698 conn(0x7ff25c071fb0 msgr2=0x7ff25c0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:32.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff260a46640 1 -- 172.21.15.57:0/2746134698 shutdown_connections 2023-10-30T14:48:32.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff260a46640 1 -- 172.21.15.57:0/2746134698 wait complete. 2023-10-30T14:48:32.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff260a46640 1 Processor -- start 2023-10-30T14:48:32.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff260a46640 1 -- start start 2023-10-30T14:48:32.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff260a46640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff25c0cd4a0 0x7ff25c14cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:32.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff260a46640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff25c1537f0 0x7ff25c1f1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:32.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff260a46640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff25c15a2e0 0x7ff25c06b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:32.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff260a46640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c0c82e0 con 0x7ff25c0cd4a0 2023-10-30T14:48:32.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff260a46640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c155610 con 0x7ff25c15a2e0 2023-10-30T14:48:32.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff260a46640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c0c5e20 con 0x7ff25c1537f0 2023-10-30T14:48:32.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff25a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff25c0cd4a0 0x7ff25c14cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:32.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff25a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff25c0cd4a0 0x7ff25c14cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47560/0 (socket says 172.21.15.57:47560) 2023-10-30T14:48:32.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff25a59c640 1 -- 172.21.15.57:0/4049705261 learned_addr learned my addr 172.21.15.57:0/4049705261 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:32.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff253fff640 1 --2- 172.21.15.57:0/4049705261 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff25c1537f0 0x7ff25c1f1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:32.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff25ad9d640 1 --2- 172.21.15.57:0/4049705261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff25c15a2e0 0x7ff25c06b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:32.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff25a59c640 1 -- 172.21.15.57:0/4049705261 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff25c1537f0 msgr2=0x7ff25c1f1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:32.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff25a59c640 1 --2- 172.21.15.57:0/4049705261 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff25c1537f0 0x7ff25c1f1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff25a59c640 1 -- 172.21.15.57:0/4049705261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff25c15a2e0 msgr2=0x7ff25c06b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:32.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff25a59c640 1 --2- 172.21.15.57:0/4049705261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff25c15a2e0 0x7ff25c06b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.423+0000 7ff25a59c640 1 -- 172.21.15.57:0/4049705261 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2440094d0 con 0x7ff25c0cd4a0 2023-10-30T14:48:32.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.427+0000 7ff25a59c640 1 --2- 172.21.15.57:0/4049705261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff25c0cd4a0 0x7ff25c14cd80 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7ff24c00d830 tx=0x7ff24c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:32.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.427+0000 7ff252ffd640 1 -- 172.21.15.57:0/4049705261 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c00b6c0 con 0x7ff25c0cd4a0 2023-10-30T14:48:32.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.427+0000 7ff252ffd640 1 -- 172.21.15.57:0/4049705261 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24c00b820 con 0x7ff25c0cd4a0 2023-10-30T14:48:32.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.427+0000 7ff252ffd640 1 -- 172.21.15.57:0/4049705261 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c00bb90 con 0x7ff25c0cd4a0 2023-10-30T14:48:32.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.427+0000 7ff260a46640 1 -- 172.21.15.57:0/4049705261 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff25c14d2c0 con 0x7ff25c0cd4a0 2023-10-30T14:48:32.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.427+0000 7ff260a46640 1 -- 172.21.15.57:0/4049705261 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff25c0c6050 con 0x7ff25c0cd4a0 2023-10-30T14:48:32.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.427+0000 7ff260a46640 1 -- 172.21.15.57:0/4049705261 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff25c04e170 con 0x7ff25c0cd4a0 2023-10-30T14:48:32.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.427+0000 7ff252ffd640 1 -- 172.21.15.57:0/4049705261 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff24c01e460 con 0x7ff25c0cd4a0 2023-10-30T14:48:32.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.427+0000 7ff252ffd640 1 --2- 172.21.15.57:0/4049705261 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff23c0664a0 0x7ff23c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:32.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.427+0000 7ff252ffd640 1 -- 172.21.15.57:0/4049705261 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7ff24c084c70 con 0x7ff25c0cd4a0 2023-10-30T14:48:32.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.427+0000 7ff253fff640 1 --2- 172.21.15.57:0/4049705261 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff23c0664a0 0x7ff23c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:32.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.431+0000 7ff253fff640 1 --2- 172.21.15.57:0/4049705261 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff23c0664a0 0x7ff23c068960 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ff2440b2000 tx=0x7ff244009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:32.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.431+0000 7ff252ffd640 1 -- 172.21.15.57:0/4049705261 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff24c051a40 con 0x7ff25c0cd4a0 2023-10-30T14:48:32.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.623+0000 7ff260a46640 1 -- 172.21.15.57:0/4049705261 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff25c073de0 con 0x7ff25c0cd4a0 2023-10-30T14:48:32.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.623+0000 7ff252ffd640 1 -- 172.21.15.57:0/4049705261 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5709 (secure 0 0 0) 0x7ff24c055060 con 0x7ff25c0cd4a0 2023-10-30T14:48:32.629 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:32.629 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:30.252464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:32.631 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 180 2023-10-30T14:48:32.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.627+0000 7ff260a46640 1 -- 172.21.15.57:0/4049705261 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff23c0664a0 msgr2=0x7ff23c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:32.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.627+0000 7ff260a46640 1 --2- 172.21.15.57:0/4049705261 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff23c0664a0 0x7ff23c068960 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ff2440b2000 tx=0x7ff244009040 comp rx=0 tx=0).stop 2023-10-30T14:48:32.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.627+0000 7ff260a46640 1 -- 172.21.15.57:0/4049705261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff25c0cd4a0 msgr2=0x7ff25c14cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:32.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.627+0000 7ff260a46640 1 --2- 172.21.15.57:0/4049705261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff25c0cd4a0 0x7ff25c14cd80 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7ff24c00d830 tx=0x7ff24c00dcf0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.627+0000 7ff260a46640 1 -- 172.21.15.57:0/4049705261 shutdown_connections 2023-10-30T14:48:32.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.627+0000 7ff260a46640 1 --2- 172.21.15.57:0/4049705261 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff23c0664a0 0x7ff23c068960 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.627+0000 7ff260a46640 1 --2- 172.21.15.57:0/4049705261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff25c15a2e0 0x7ff25c06b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.627+0000 7ff260a46640 1 --2- 172.21.15.57:0/4049705261 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff25c1537f0 0x7ff25c1f1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.627+0000 7ff260a46640 1 --2- 172.21.15.57:0/4049705261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff25c0cd4a0 0x7ff25c14cd80 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:32.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.627+0000 7ff260a46640 1 -- 172.21.15.57:0/4049705261 >> 172.21.15.57:0/4049705261 conn(0x7ff25c071fb0 msgr2=0x7ff25c072390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:32.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.627+0000 7ff260a46640 1 -- 172.21.15.57:0/4049705261 shutdown_connections 2023-10-30T14:48:32.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:32.627+0000 7ff260a46640 1 -- 172.21.15.57:0/4049705261 wait complete. 2023-10-30T14:48:32.651 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 180, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 180, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7911": { "addr": "172.21.15.57:6836/2613636524", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2613636524, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2613636524, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7911, "incarnation": 180, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:48:30.252464+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7911 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6839/1078372885", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1078372885, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1078372885, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 170, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6716, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.57:6834/697507447", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 697507447, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 697507447, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 156, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7377, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:48:34.653 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:34.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.803+0000 7f8fcd2d6640 1 Processor -- start 2023-10-30T14:48:34.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.803+0000 7f8fcd2d6640 1 -- start start 2023-10-30T14:48:34.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.803+0000 7f8fcd2d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fc8153aa0 0x7f8fc815d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:34.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.803+0000 7f8fcd2d6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8fc806b3b0 0x7f8fc806b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:34.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.803+0000 7f8fcd2d6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc806bd80 con 0x7f8fc80701e0 2023-10-30T14:48:34.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.803+0000 7f8fcd2d6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fc8070a50 con 0x7f8fc806b3b0 2023-10-30T14:48:34.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.803+0000 7f8fcd2d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc815dd80 con 0x7f8fc8153aa0 2023-10-30T14:48:34.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.803+0000 7f8fc77fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8fc806b3b0 0x7f8fc806b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:34.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.807+0000 7f8fbffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f8fc80701e0 0x7f8fc80ccee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:58550/0 (socket says 172.21.15.57:58550) 2023-10-30T14:48:34.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.807+0000 7f8fbffff640 1 -- 172.21.15.57:0/761997097 learned_addr learned my addr 172.21.15.57:0/761997097 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:48:34.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.807+0000 7f8fc6ffd640 1 --2- 172.21.15.57:0/761997097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fc8153aa0 0x7f8fc815d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:34.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.807+0000 7f8fc77fe640 1 -- 172.21.15.57:0/761997097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fc8153aa0 msgr2=0x7f8fc815d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:34.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.807+0000 7f8fc77fe640 1 --2- 172.21.15.57:0/761997097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fc8153aa0 0x7f8fc815d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:34.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.807+0000 7f8fc77fe640 1 -- 172.21.15.57:0/761997097 >> v1:172.21.15.57:6789/0 conn(0x7f8fc80701e0 legacy=0x7f8fc80ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:34.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.807+0000 7f8fc77fe640 1 -- 172.21.15.57:0/761997097 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc80c7b00 con 0x7f8fc806b3b0 2023-10-30T14:48:34.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.807+0000 7f8fc67fc640 1 -- 172.21.15.57:0/761997097 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335601265 0 0) 0x7f8fc806bd80 con 0x7f8fc80701e0 2023-10-30T14:48:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.807+0000 7f8fc77fe640 1 --2- 172.21.15.57:0/761997097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fc806b3b0 0x7f8fc806b840 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0009d90 tx=0x7f8fc001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1f269f1dc4be79d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.807+0000 7f8fc67fc640 1 -- 172.21.15.57:0/761997097 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc00af600 con 0x7f8fc806b3b0 2023-10-30T14:48:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.807+0000 7f8fc67fc640 1 -- 172.21.15.57:0/761997097 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc00ab050 con 0x7f8fc806b3b0 2023-10-30T14:48:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.807+0000 7f8fc67fc640 1 -- 172.21.15.57:0/761997097 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc00af980 con 0x7f8fc806b3b0 2023-10-30T14:48:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fcd2d6640 1 -- 172.21.15.57:0/761997097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fc806b3b0 msgr2=0x7f8fc806b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fcd2d6640 1 --2- 172.21.15.57:0/761997097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fc806b3b0 0x7f8fc806b840 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0009d90 tx=0x7f8fc001f130 comp rx=0 tx=0).stop 2023-10-30T14:48:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fcd2d6640 1 -- 172.21.15.57:0/761997097 shutdown_connections 2023-10-30T14:48:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fcd2d6640 1 --2- 172.21.15.57:0/761997097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fc806b3b0 0x7f8fc806b840 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fcd2d6640 1 --2- 172.21.15.57:0/761997097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fc8153aa0 0x7f8fc815d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fcd2d6640 1 -- 172.21.15.57:0/761997097 >> 172.21.15.57:0/761997097 conn(0x7f8fc8071fb0 msgr2=0x7f8fc80723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:34.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fcd2d6640 1 -- 172.21.15.57:0/761997097 shutdown_connections 2023-10-30T14:48:34.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fcd2d6640 1 -- 172.21.15.57:0/761997097 wait complete. 2023-10-30T14:48:34.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fcd2d6640 1 Processor -- start 2023-10-30T14:48:34.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fcd2d6640 1 -- start start 2023-10-30T14:48:34.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fcd2d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fc806b3b0 0x7f8fc8149cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:34.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fcd2d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fc80701e0 0x7f8fc81efa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:34.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fcd2d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fc8153aa0 0x7f8fc81f1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:34.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fcd2d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc815a420 con 0x7f8fc8153aa0 2023-10-30T14:48:34.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fcd2d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc81eeed0 con 0x7f8fc806b3b0 2023-10-30T14:48:34.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fcd2d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc814e2a0 con 0x7f8fc80701e0 2023-10-30T14:48:34.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fc77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fc8153aa0 0x7f8fc81f1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:34.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fc77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fc8153aa0 0x7f8fc81f1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54926/0 (socket says 172.21.15.57:54926) 2023-10-30T14:48:34.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fc77fe640 1 -- 172.21.15.57:0/2507178149 learned_addr learned my addr 172.21.15.57:0/2507178149 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:34.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fc6ffd640 1 --2- 172.21.15.57:0/2507178149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fc806b3b0 0x7f8fc8149cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:34.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fbffff640 1 --2- 172.21.15.57:0/2507178149 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fc80701e0 0x7f8fc81efa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:34.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fc77fe640 1 -- 172.21.15.57:0/2507178149 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fc80701e0 msgr2=0x7f8fc81efa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:34.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fc77fe640 1 --2- 172.21.15.57:0/2507178149 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fc80701e0 0x7f8fc81efa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:34.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fc77fe640 1 -- 172.21.15.57:0/2507178149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fc806b3b0 msgr2=0x7f8fc8149cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:34.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fc77fe640 1 --2- 172.21.15.57:0/2507178149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fc806b3b0 0x7f8fc8149cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:34.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.811+0000 7f8fc77fe640 1 -- 172.21.15.57:0/2507178149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc0009a70 con 0x7f8fc8153aa0 2023-10-30T14:48:34.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.815+0000 7f8fc77fe640 1 --2- 172.21.15.57:0/2507178149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fc8153aa0 0x7f8fc81f1b30 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00b2040 tx=0x7f8fc001bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:34.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.815+0000 7f8fbf7fe640 1 -- 172.21.15.57:0/2507178149 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc00b9610 con 0x7f8fc8153aa0 2023-10-30T14:48:34.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.815+0000 7f8fbf7fe640 1 -- 172.21.15.57:0/2507178149 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc00b9770 con 0x7f8fc8153aa0 2023-10-30T14:48:34.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.815+0000 7f8fbf7fe640 1 -- 172.21.15.57:0/2507178149 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc00b9b20 con 0x7f8fc8153aa0 2023-10-30T14:48:34.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.815+0000 7f8fcd2d6640 1 -- 172.21.15.57:0/2507178149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc814a200 con 0x7f8fc8153aa0 2023-10-30T14:48:34.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.815+0000 7f8fcd2d6640 1 -- 172.21.15.57:0/2507178149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fc81f25e0 con 0x7f8fc8153aa0 2023-10-30T14:48:34.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.815+0000 7f8fcd2d6640 1 -- 172.21.15.57:0/2507178149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc804e170 con 0x7f8fc8153aa0 2023-10-30T14:48:34.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.815+0000 7f8fbf7fe640 1 -- 172.21.15.57:0/2507178149 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8fc00b6050 con 0x7f8fc8153aa0 2023-10-30T14:48:34.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.815+0000 7f8fbf7fe640 1 --2- 172.21.15.57:0/2507178149 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8fa80664a0 0x7f8fa8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:34.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.815+0000 7f8fbf7fe640 1 -- 172.21.15.57:0/2507178149 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7f8fc0122970 con 0x7f8fc8153aa0 2023-10-30T14:48:34.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.815+0000 7f8fc6ffd640 1 --2- 172.21.15.57:0/2507178149 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8fa80664a0 0x7f8fa8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:34.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.819+0000 7f8fc6ffd640 1 --2- 172.21.15.57:0/2507178149 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8fa80664a0 0x7f8fa8068960 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f8fb80079c0 tx=0x7f8fb8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:34.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:34.819+0000 7f8fbf7fe640 1 -- 172.21.15.57:0/2507178149 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8fc00ef7d0 con 0x7f8fc8153aa0 2023-10-30T14:48:35.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:35.015+0000 7f8fcd2d6640 1 -- 172.21.15.57:0/2507178149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fc81ef010 con 0x7f8fc8153aa0 2023-10-30T14:48:35.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:35.015+0000 7f8fbf7fe640 1 -- 172.21.15.57:0/2507178149 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5709 (secure 0 0 0) 0x7f8fc00b4090 con 0x7f8fc8153aa0 2023-10-30T14:48:35.019 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:35.020 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:30.252464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:35.020 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 180 2023-10-30T14:48:35.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:35.015+0000 7f8fcd2d6640 1 -- 172.21.15.57:0/2507178149 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8fa80664a0 msgr2=0x7f8fa8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:35.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:35.015+0000 7f8fcd2d6640 1 --2- 172.21.15.57:0/2507178149 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8fa80664a0 0x7f8fa8068960 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f8fb80079c0 tx=0x7f8fb8008040 comp rx=0 tx=0).stop 2023-10-30T14:48:35.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:35.015+0000 7f8fcd2d6640 1 -- 172.21.15.57:0/2507178149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fc8153aa0 msgr2=0x7f8fc81f1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:35.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:35.015+0000 7f8fcd2d6640 1 --2- 172.21.15.57:0/2507178149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fc8153aa0 0x7f8fc81f1b30 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00b2040 tx=0x7f8fc001bed0 comp rx=0 tx=0).stop 2023-10-30T14:48:35.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:35.015+0000 7f8fcd2d6640 1 -- 172.21.15.57:0/2507178149 shutdown_connections 2023-10-30T14:48:35.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:35.015+0000 7f8fcd2d6640 1 --2- 172.21.15.57:0/2507178149 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8fa80664a0 0x7f8fa8068960 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:35.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:35.015+0000 7f8fcd2d6640 1 --2- 172.21.15.57:0/2507178149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fc8153aa0 0x7f8fc81f1b30 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:35.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:35.015+0000 7f8fcd2d6640 1 --2- 172.21.15.57:0/2507178149 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8fc80701e0 0x7f8fc81efa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:35.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:35.015+0000 7f8fcd2d6640 1 --2- 172.21.15.57:0/2507178149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fc806b3b0 0x7f8fc8149cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:35.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:35.015+0000 7f8fcd2d6640 1 -- 172.21.15.57:0/2507178149 >> 172.21.15.57:0/2507178149 conn(0x7f8fc8071fb0 msgr2=0x7f8fc8152dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:35.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:35.015+0000 7f8fcd2d6640 1 -- 172.21.15.57:0/2507178149 shutdown_connections 2023-10-30T14:48:35.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:35.015+0000 7f8fcd2d6640 1 -- 172.21.15.57:0/2507178149 wait complete. 2023-10-30T14:48:35.041 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:48:37.042 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:37.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.191+0000 7f7b3511d640 1 Processor -- start 2023-10-30T14:48:37.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.191+0000 7f7b3511d640 1 -- start start 2023-10-30T14:48:37.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.191+0000 7f7b3511d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7b30153c80 0x7f7b3015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:37.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.191+0000 7f7b3511d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7b300c4a10 0x7f7b300c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:37.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.191+0000 7f7b3511d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b30159f20 con 0x7f7b3015a3b0 2023-10-30T14:48:37.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.191+0000 7f7b3511d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b3015dec0 con 0x7f7b30153c80 2023-10-30T14:48:37.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.191+0000 7f7b3511d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b301599a0 con 0x7f7b300c4a10 2023-10-30T14:48:37.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.191+0000 7f7b2ed9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7b30153c80 0x7f7b3015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:37.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.191+0000 7f7b2ed9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7b30153c80 0x7f7b3015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54940/0 (socket says 172.21.15.57:54940) 2023-10-30T14:48:37.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.191+0000 7f7b2ed9d640 1 -- 172.21.15.57:0/2511819181 learned_addr learned my addr 172.21.15.57:0/2511819181 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:37.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.191+0000 7f7b2e59c640 1 --2- 172.21.15.57:0/2511819181 >> v2:172.21.15.88:3300/0 conn(0x7f7b300c4a10 0x7f7b300c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:37.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.191+0000 7f7b2ed9d640 1 -- 172.21.15.57:0/2511819181 >> v2:172.21.15.88:3300/0 conn(0x7f7b300c4a10 msgr2=0x7f7b300c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:37.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.191+0000 7f7b2ed9d640 1 --2- 172.21.15.57:0/2511819181 >> v2:172.21.15.88:3300/0 conn(0x7f7b300c4a10 0x7f7b300c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:37.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.191+0000 7f7b2ed9d640 1 -- 172.21.15.57:0/2511819181 >> v1:172.21.15.88:6789/0 conn(0x7f7b3015a3b0 legacy=0x7f7b300ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:37.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.191+0000 7f7b2ed9d640 1 -- 172.21.15.57:0/2511819181 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b300c47e0 con 0x7f7b30153c80 2023-10-30T14:48:37.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.199+0000 7f7b2ed9d640 1 --2- 172.21.15.57:0/2511819181 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b30153c80 0x7f7b3015cf90 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f7b3006be50 tx=0x7f7b2001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce05fa011cac2b1b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:37.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.199+0000 7f7b2dd9b640 1 -- 172.21.15.57:0/2511819181 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b200a89f0 con 0x7f7b30153c80 2023-10-30T14:48:37.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.199+0000 7f7b2dd9b640 1 -- 172.21.15.57:0/2511819181 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b200a8b50 con 0x7f7b30153c80 2023-10-30T14:48:37.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.199+0000 7f7b2dd9b640 1 -- 172.21.15.57:0/2511819181 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b200b0440 con 0x7f7b30153c80 2023-10-30T14:48:37.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.199+0000 7f7b3511d640 1 -- 172.21.15.57:0/2511819181 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b30153c80 msgr2=0x7f7b3015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:37.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.199+0000 7f7b3511d640 1 --2- 172.21.15.57:0/2511819181 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b30153c80 0x7f7b3015cf90 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f7b3006be50 tx=0x7f7b2001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:48:37.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.199+0000 7f7b3511d640 1 -- 172.21.15.57:0/2511819181 shutdown_connections 2023-10-30T14:48:37.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.199+0000 7f7b3511d640 1 --2- 172.21.15.57:0/2511819181 >> v2:172.21.15.88:3300/0 conn(0x7f7b300c4a10 0x7f7b300c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:37.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.199+0000 7f7b3511d640 1 --2- 172.21.15.57:0/2511819181 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b30153c80 0x7f7b3015cf90 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:37.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.199+0000 7f7b3511d640 1 -- 172.21.15.57:0/2511819181 >> 172.21.15.57:0/2511819181 conn(0x7f7b300715c0 msgr2=0x7f7b300719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:37.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.199+0000 7f7b3511d640 1 -- 172.21.15.57:0/2511819181 shutdown_connections 2023-10-30T14:48:37.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.199+0000 7f7b3511d640 1 -- 172.21.15.57:0/2511819181 wait complete. 2023-10-30T14:48:37.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b3511d640 1 Processor -- start 2023-10-30T14:48:37.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b3511d640 1 -- start start 2023-10-30T14:48:37.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b3511d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b300c4a10 0x7f7b300ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:37.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b3511d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b30153c80 0x7f7b301ef1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:37.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b3511d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b3015a3b0 0x7f7b3014f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:37.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b3511d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b300cee00 con 0x7f7b30153c80 2023-10-30T14:48:37.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b3511d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b301ef720 con 0x7f7b3015a3b0 2023-10-30T14:48:37.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b3511d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b300cefe0 con 0x7f7b300c4a10 2023-10-30T14:48:37.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b2e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b30153c80 0x7f7b301ef1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:37.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b2e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b30153c80 0x7f7b301ef1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54952/0 (socket says 172.21.15.57:54952) 2023-10-30T14:48:37.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b2e59c640 1 -- 172.21.15.57:0/2097436492 learned_addr learned my addr 172.21.15.57:0/2097436492 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:37.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b2f59e640 1 --2- 172.21.15.57:0/2097436492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b3015a3b0 0x7f7b3014f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:37.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b2ed9d640 1 --2- 172.21.15.57:0/2097436492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b300c4a10 0x7f7b300ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:37.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b2e59c640 1 -- 172.21.15.57:0/2097436492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b300c4a10 msgr2=0x7f7b300ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:37.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b2e59c640 1 --2- 172.21.15.57:0/2097436492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b300c4a10 0x7f7b300ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:37.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b2e59c640 1 -- 172.21.15.57:0/2097436492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b3015a3b0 msgr2=0x7f7b3014f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:37.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b2e59c640 1 --2- 172.21.15.57:0/2097436492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b3015a3b0 0x7f7b3014f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:37.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b2e59c640 1 -- 172.21.15.57:0/2097436492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b200094d0 con 0x7f7b30153c80 2023-10-30T14:48:37.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b2f59e640 1 --2- 172.21.15.57:0/2097436492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b3015a3b0 0x7f7b3014f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:48:37.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b2ed9d640 1 --2- 172.21.15.57:0/2097436492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b300c4a10 0x7f7b300ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:48:37.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b2e59c640 1 --2- 172.21.15.57:0/2097436492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b30153c80 0x7f7b301ef1e0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f7b2400d800 tx=0x7f7b2400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:37.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b177fe640 1 -- 172.21.15.57:0/2097436492 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2400b660 con 0x7f7b30153c80 2023-10-30T14:48:37.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b177fe640 1 -- 172.21.15.57:0/2097436492 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2400b7c0 con 0x7f7b30153c80 2023-10-30T14:48:37.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b3511d640 1 -- 172.21.15.57:0/2097436492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b301f2a10 con 0x7f7b30153c80 2023-10-30T14:48:37.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b177fe640 1 -- 172.21.15.57:0/2097436492 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2400bb70 con 0x7f7b30153c80 2023-10-30T14:48:37.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.203+0000 7f7b3511d640 1 -- 172.21.15.57:0/2097436492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b301f20e0 con 0x7f7b30153c80 2023-10-30T14:48:37.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.207+0000 7f7b3511d640 1 -- 172.21.15.57:0/2097436492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b3004e170 con 0x7f7b30153c80 2023-10-30T14:48:37.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.207+0000 7f7b177fe640 1 -- 172.21.15.57:0/2097436492 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7b2400bcd0 con 0x7f7b30153c80 2023-10-30T14:48:37.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.207+0000 7f7b177fe640 1 --2- 172.21.15.57:0/2097436492 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7b1c0664a0 0x7f7b1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:37.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.207+0000 7f7b177fe640 1 -- 172.21.15.57:0/2097436492 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7f7b24084a90 con 0x7f7b30153c80 2023-10-30T14:48:37.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.207+0000 7f7b2ed9d640 1 --2- 172.21.15.57:0/2097436492 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7b1c0664a0 0x7f7b1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:37.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.207+0000 7f7b2ed9d640 1 --2- 172.21.15.57:0/2097436492 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7b1c0664a0 0x7f7b1c068960 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f7b200b2000 tx=0x7f7b200aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:37.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.215+0000 7f7b177fe640 1 -- 172.21.15.57:0/2097436492 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7b240517b0 con 0x7f7b30153c80 2023-10-30T14:48:37.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.415+0000 7f7b3511d640 1 -- 172.21.15.57:0/2097436492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b300cad40 con 0x7f7b30153c80 2023-10-30T14:48:37.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.415+0000 7f7b177fe640 1 -- 172.21.15.57:0/2097436492 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5709 (secure 0 0 0) 0x7f7b24054dd0 con 0x7f7b30153c80 2023-10-30T14:48:37.415 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:37.416 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:30.252464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:37.416 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 180 2023-10-30T14:48:37.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.415+0000 7f7b3511d640 1 -- 172.21.15.57:0/2097436492 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7b1c0664a0 msgr2=0x7f7b1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:37.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.415+0000 7f7b3511d640 1 --2- 172.21.15.57:0/2097436492 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7b1c0664a0 0x7f7b1c068960 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f7b200b2000 tx=0x7f7b200aa070 comp rx=0 tx=0).stop 2023-10-30T14:48:37.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.415+0000 7f7b3511d640 1 -- 172.21.15.57:0/2097436492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b30153c80 msgr2=0x7f7b301ef1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:37.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.415+0000 7f7b3511d640 1 --2- 172.21.15.57:0/2097436492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b30153c80 0x7f7b301ef1e0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f7b2400d800 tx=0x7f7b2400dcc0 comp rx=0 tx=0).stop 2023-10-30T14:48:37.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.415+0000 7f7b3511d640 1 -- 172.21.15.57:0/2097436492 shutdown_connections 2023-10-30T14:48:37.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.415+0000 7f7b3511d640 1 --2- 172.21.15.57:0/2097436492 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7b1c0664a0 0x7f7b1c068960 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:37.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.415+0000 7f7b3511d640 1 --2- 172.21.15.57:0/2097436492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b3015a3b0 0x7f7b3014f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:37.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.415+0000 7f7b3511d640 1 --2- 172.21.15.57:0/2097436492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b30153c80 0x7f7b301ef1e0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:37.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.415+0000 7f7b3511d640 1 --2- 172.21.15.57:0/2097436492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b300c4a10 0x7f7b300ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:37.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.415+0000 7f7b3511d640 1 -- 172.21.15.57:0/2097436492 >> 172.21.15.57:0/2097436492 conn(0x7f7b300715c0 msgr2=0x7f7b300719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:37.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.415+0000 7f7b3511d640 1 -- 172.21.15.57:0/2097436492 shutdown_connections 2023-10-30T14:48:37.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:37.415+0000 7f7b3511d640 1 -- 172.21.15.57:0/2097436492 wait complete. 2023-10-30T14:48:37.438 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:48:39.439 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:39.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.599+0000 7f7692a74640 1 Processor -- start 2023-10-30T14:48:39.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.599+0000 7f7692a74640 1 -- start start 2023-10-30T14:48:39.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.599+0000 7f7692a74640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f768c0cb180 0x7f768c1597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:39.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.599+0000 7f7692a74640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c15a770 con 0x7f768c153b50 2023-10-30T14:48:39.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.599+0000 7f7692a74640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c0c9d40 con 0x7f768c0c62b0 2023-10-30T14:48:39.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.599+0000 7f7692a74640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f768c159cf0 con 0x7f768c0cb180 2023-10-30T14:48:39.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.599+0000 7f7691a72640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f768c153b50 0x7f768c0ca1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:58556/0 (socket says 172.21.15.57:58556) 2023-10-30T14:48:39.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.599+0000 7f7691a72640 1 -- 172.21.15.57:0/262942093 learned_addr learned my addr 172.21.15.57:0/262942093 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:48:39.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.599+0000 7f7692273640 1 --2- 172.21.15.57:0/262942093 >> v2:172.21.15.57:3300/0 conn(0x7f768c0cb180 0x7f768c1597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:39.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.599+0000 7f7690a70640 1 -- 172.21.15.57:0/262942093 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3595642190 0 0) 0x7f7688001120 con 0x7f768c153b50 2023-10-30T14:48:39.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.599+0000 7f7690a70640 1 -- 172.21.15.57:0/262942093 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7678003530 con 0x7f768c153b50 2023-10-30T14:48:39.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.599+0000 7f7692273640 1 -- 172.21.15.57:0/262942093 >> v1:172.21.15.88:6789/0 conn(0x7f768c0c62b0 legacy=0x7f768c06ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:39.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.599+0000 7f7692273640 1 -- 172.21.15.57:0/262942093 >> v1:172.21.15.57:6789/0 conn(0x7f768c153b50 legacy=0x7f768c0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:39.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.599+0000 7f7692273640 1 -- 172.21.15.57:0/262942093 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c153f30 con 0x7f768c0cb180 2023-10-30T14:48:39.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.603+0000 7f7692273640 1 --2- 172.21.15.57:0/262942093 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f768c0cb180 0x7f768c1597b0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f7684009d90 tx=0x7f768401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b71f884c1205070 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:39.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.603+0000 7f7690a70640 1 -- 172.21.15.57:0/262942093 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76840af600 con 0x7f768c0cb180 2023-10-30T14:48:39.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.603+0000 7f7690a70640 1 -- 172.21.15.57:0/262942093 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76840ab050 con 0x7f768c0cb180 2023-10-30T14:48:39.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.603+0000 7f7690a70640 1 -- 172.21.15.57:0/262942093 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76840af980 con 0x7f768c0cb180 2023-10-30T14:48:39.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.603+0000 7f7692a74640 1 -- 172.21.15.57:0/262942093 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f768c0cb180 msgr2=0x7f768c1597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:39.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.603+0000 7f7692a74640 1 --2- 172.21.15.57:0/262942093 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f768c0cb180 0x7f768c1597b0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f7684009d90 tx=0x7f768401f130 comp rx=0 tx=0).stop 2023-10-30T14:48:39.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.603+0000 7f7692a74640 1 -- 172.21.15.57:0/262942093 shutdown_connections 2023-10-30T14:48:39.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.603+0000 7f7692a74640 1 --2- 172.21.15.57:0/262942093 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f768c0cb180 0x7f768c1597b0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:39.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.603+0000 7f7692a74640 1 -- 172.21.15.57:0/262942093 >> 172.21.15.57:0/262942093 conn(0x7f768c065a80 msgr2=0x7f768c065e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:39.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.603+0000 7f7692a74640 1 -- 172.21.15.57:0/262942093 shutdown_connections 2023-10-30T14:48:39.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.603+0000 7f7692a74640 1 -- 172.21.15.57:0/262942093 wait complete. 2023-10-30T14:48:39.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.607+0000 7f7692a74640 1 Processor -- start 2023-10-30T14:48:39.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.607+0000 7f7692a74640 1 -- start start 2023-10-30T14:48:39.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.607+0000 7f7692a74640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f768c0c62b0 0x7f768c14c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:39.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.607+0000 7f7692a74640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f768c0cb180 0x7f768c0c3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:39.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.607+0000 7f7692a74640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f768c153b50 0x7f768c1f1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:39.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.607+0000 7f7692a74640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c159b00 con 0x7f768c0c62b0 2023-10-30T14:48:39.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.607+0000 7f7692a74640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c14a6a0 con 0x7f768c0cb180 2023-10-30T14:48:39.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.607+0000 7f7692a74640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c1592e0 con 0x7f768c153b50 2023-10-30T14:48:39.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.607+0000 7f7691a72640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f768c0c62b0 0x7f768c14c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:39.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.611+0000 7f7691a72640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f768c0c62b0 0x7f768c14c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54966/0 (socket says 172.21.15.57:54966) 2023-10-30T14:48:39.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.611+0000 7f7691a72640 1 -- 172.21.15.57:0/786557681 learned_addr learned my addr 172.21.15.57:0/786557681 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:39.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.611+0000 7f7692273640 1 --2- 172.21.15.57:0/786557681 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f768c153b50 0x7f768c1f1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:39.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.611+0000 7f7691271640 1 --2- 172.21.15.57:0/786557681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f768c0cb180 0x7f768c0c3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:39.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.611+0000 7f7691a72640 1 -- 172.21.15.57:0/786557681 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f768c153b50 msgr2=0x7f768c1f1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:39.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.611+0000 7f7691a72640 1 --2- 172.21.15.57:0/786557681 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f768c153b50 0x7f768c1f1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:39.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.611+0000 7f7691a72640 1 -- 172.21.15.57:0/786557681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f768c0cb180 msgr2=0x7f768c0c3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:39.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.611+0000 7f7691a72640 1 --2- 172.21.15.57:0/786557681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f768c0cb180 0x7f768c0c3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:39.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.611+0000 7f7691a72640 1 -- 172.21.15.57:0/786557681 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7684009a70 con 0x7f768c0c62b0 2023-10-30T14:48:39.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.611+0000 7f7691a72640 1 --2- 172.21.15.57:0/786557681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f768c0c62b0 0x7f768c14c210 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f768800c420 tx=0x7f768800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:39.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.611+0000 7f76827fc640 1 -- 172.21.15.57:0/786557681 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7688005cf0 con 0x7f768c0c62b0 2023-10-30T14:48:39.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.611+0000 7f76827fc640 1 -- 172.21.15.57:0/786557681 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7688005e50 con 0x7f768c0c62b0 2023-10-30T14:48:39.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.611+0000 7f76827fc640 1 -- 172.21.15.57:0/786557681 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76880072f0 con 0x7f768c0c62b0 2023-10-30T14:48:39.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.611+0000 7f7692a74640 1 -- 172.21.15.57:0/786557681 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f768c0c2de0 con 0x7f768c0c62b0 2023-10-30T14:48:39.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.611+0000 7f7692a74640 1 -- 172.21.15.57:0/786557681 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768c1597e0 con 0x7f768c0c62b0 2023-10-30T14:48:39.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.611+0000 7f7692a74640 1 -- 172.21.15.57:0/786557681 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f768c0c2c20 con 0x7f768c0c62b0 2023-10-30T14:48:39.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.615+0000 7f76827fc640 1 -- 172.21.15.57:0/786557681 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7688007450 con 0x7f768c0c62b0 2023-10-30T14:48:39.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.615+0000 7f76827fc640 1 --2- 172.21.15.57:0/786557681 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76700664a0 0x7f7670068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:39.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.615+0000 7f76827fc640 1 -- 172.21.15.57:0/786557681 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7f76880837c0 con 0x7f768c0c62b0 2023-10-30T14:48:39.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.615+0000 7f7691271640 1 --2- 172.21.15.57:0/786557681 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76700664a0 0x7f7670068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:39.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.615+0000 7f7691271640 1 --2- 172.21.15.57:0/786557681 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76700664a0 0x7f7670068960 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f768c14ee10 tx=0x7f767c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:39.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.623+0000 7f76827fc640 1 -- 172.21.15.57:0/786557681 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7688050750 con 0x7f768c0c62b0 2023-10-30T14:48:39.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.835+0000 7f7692a74640 1 -- 172.21.15.57:0/786557681 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f768c15a770 con 0x7f768c0c62b0 2023-10-30T14:48:39.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.835+0000 7f76827fc640 1 -- 172.21.15.57:0/786557681 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5709 (secure 0 0 0) 0x7f768c15a770 con 0x7f768c0c62b0 2023-10-30T14:48:39.835 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:39.836 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:30.252464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:39.836 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 180 2023-10-30T14:48:39.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.835+0000 7f7692a74640 1 -- 172.21.15.57:0/786557681 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76700664a0 msgr2=0x7f7670068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:39.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.835+0000 7f7692a74640 1 --2- 172.21.15.57:0/786557681 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76700664a0 0x7f7670068960 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f768c14ee10 tx=0x7f767c009040 comp rx=0 tx=0).stop 2023-10-30T14:48:39.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.835+0000 7f7692a74640 1 -- 172.21.15.57:0/786557681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f768c0c62b0 msgr2=0x7f768c14c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:39.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.835+0000 7f7692a74640 1 --2- 172.21.15.57:0/786557681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f768c0c62b0 0x7f768c14c210 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f768800c420 tx=0x7f768800c8e0 comp rx=0 tx=0).stop 2023-10-30T14:48:39.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.835+0000 7f7692a74640 1 -- 172.21.15.57:0/786557681 shutdown_connections 2023-10-30T14:48:39.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.835+0000 7f7692a74640 1 --2- 172.21.15.57:0/786557681 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76700664a0 0x7f7670068960 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:39.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.835+0000 7f7692a74640 1 --2- 172.21.15.57:0/786557681 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f768c153b50 0x7f768c1f1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:39.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.835+0000 7f7692a74640 1 --2- 172.21.15.57:0/786557681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f768c0cb180 0x7f768c0c3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:39.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.835+0000 7f7692a74640 1 --2- 172.21.15.57:0/786557681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f768c0c62b0 0x7f768c14c210 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:39.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.835+0000 7f7692a74640 1 -- 172.21.15.57:0/786557681 >> 172.21.15.57:0/786557681 conn(0x7f768c065a80 msgr2=0x7f768c14e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:39.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.835+0000 7f7692a74640 1 -- 172.21.15.57:0/786557681 shutdown_connections 2023-10-30T14:48:39.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:39.835+0000 7f7692a74640 1 -- 172.21.15.57:0/786557681 wait complete. 2023-10-30T14:48:39.858 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:48:41.471 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:48:41.477 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:48:41.858 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:42.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.011+0000 7f2e909c5640 1 Processor -- start 2023-10-30T14:48:42.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.011+0000 7f2e909c5640 1 -- start start 2023-10-30T14:48:42.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.011+0000 7f2e909c5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2e8c150830 0x7f2e8c06b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:42.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.011+0000 7f2e909c5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8c0c2310 con 0x7f2e8c070520 2023-10-30T14:48:42.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.011+0000 7f2e909c5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8c156710 con 0x7f2e8c0cb7d0 2023-10-30T14:48:42.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.011+0000 7f2e909c5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e8c0c4360 con 0x7f2e8c150830 2023-10-30T14:48:42.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.011+0000 7f2e8ad9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f2e8c070520 0x7f2e8c160f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:58562/0 (socket says 172.21.15.57:58562) 2023-10-30T14:48:42.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.011+0000 7f2e8ad9d640 1 -- 172.21.15.57:0/1283032586 learned_addr learned my addr 172.21.15.57:0/1283032586 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:48:42.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.011+0000 7f2e8a59c640 1 --2- 172.21.15.57:0/1283032586 >> v2:172.21.15.88:3300/0 conn(0x7f2e8c150830 0x7f2e8c06b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:42.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8959a640 1 -- 172.21.15.57:0/1283032586 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605124658 0 0) 0x7f2e8c0c2310 con 0x7f2e8c070520 2023-10-30T14:48:42.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8959a640 1 -- 172.21.15.57:0/1283032586 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e74003530 con 0x7f2e8c070520 2023-10-30T14:48:42.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8959a640 1 -- 172.21.15.57:0/1283032586 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2596440876 0 0) 0x7f2e74003530 con 0x7f2e8c070520 2023-10-30T14:48:42.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8959a640 1 -- 172.21.15.57:0/1283032586 >> v2:172.21.15.88:3300/0 conn(0x7f2e8c150830 msgr2=0x7f2e8c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:42.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8959a640 1 --2- 172.21.15.57:0/1283032586 >> v2:172.21.15.88:3300/0 conn(0x7f2e8c150830 0x7f2e8c06b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:42.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8959a640 1 -- 172.21.15.57:0/1283032586 >> v1:172.21.15.88:6789/0 conn(0x7f2e8c0cb7d0 legacy=0x7f2e8c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:42.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8959a640 1 -- 172.21.15.57:0/1283032586 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8c1569b0 con 0x7f2e8c070520 2023-10-30T14:48:42.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8959a640 1 -- 172.21.15.57:0/1283032586 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f2e84001a20 con 0x7f2e8c070520 2023-10-30T14:48:42.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8959a640 1 -- 172.21.15.57:0/1283032586 >> v1:172.21.15.57:6789/0 conn(0x7f2e8c070520 legacy=0x7f2e8c160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:42.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8959a640 1 --2- 172.21.15.57:0/1283032586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e74003980 0x7f2e74003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:42.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8959a640 1 --2- 172.21.15.57:0/1283032586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e740047d0 0x7f2e74004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:42.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8959a640 1 --2- 172.21.15.57:0/1283032586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e74005190 0x7f2e7400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:42.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8959a640 1 -- 172.21.15.57:0/1283032586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8c0c4360 con 0x7f2e74005190 2023-10-30T14:48:42.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8959a640 1 -- 172.21.15.57:0/1283032586 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7400c040 con 0x7f2e740047d0 2023-10-30T14:48:42.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8959a640 1 -- 172.21.15.57:0/1283032586 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7400c200 con 0x7f2e74003980 2023-10-30T14:48:42.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8ad9d640 1 --2- 172.21.15.57:0/1283032586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e74005190 0x7f2e7400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:42.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8a59c640 1 --2- 172.21.15.57:0/1283032586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e74003980 0x7f2e74003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:42.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e89d9b640 1 --2- 172.21.15.57:0/1283032586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e740047d0 0x7f2e74004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:42.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8ad9d640 1 -- 172.21.15.57:0/1283032586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e74003980 msgr2=0x7f2e74003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:42.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8ad9d640 1 --2- 172.21.15.57:0/1283032586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e74003980 0x7f2e74003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:42.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8ad9d640 1 -- 172.21.15.57:0/1283032586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e740047d0 msgr2=0x7f2e74004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:42.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8ad9d640 1 --2- 172.21.15.57:0/1283032586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e740047d0 0x7f2e74004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:42.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.015+0000 7f2e8ad9d640 1 -- 172.21.15.57:0/1283032586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e74003530 con 0x7f2e74005190 2023-10-30T14:48:42.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.019+0000 7f2e8ad9d640 1 -- 172.21.15.57:0/1283032586 reap_dead start 2023-10-30T14:48:42.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.019+0000 7f2e8ad9d640 1 --2- 172.21.15.57:0/1283032586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e74005190 0x7f2e7400b990 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c0c24a0 tx=0x7f2e8401d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1474d85db47fc0d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:42.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.019+0000 7f2e8959a640 1 -- 172.21.15.57:0/1283032586 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e840ad630 con 0x7f2e74005190 2023-10-30T14:48:42.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.019+0000 7f2e8959a640 1 -- 172.21.15.57:0/1283032586 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e840a8070 con 0x7f2e74005190 2023-10-30T14:48:42.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.019+0000 7f2e8959a640 1 -- 172.21.15.57:0/1283032586 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e840ad990 con 0x7f2e74005190 2023-10-30T14:48:42.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.019+0000 7f2e909c5640 1 -- 172.21.15.57:0/1283032586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e74005190 msgr2=0x7f2e7400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:42.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.019+0000 7f2e909c5640 1 --2- 172.21.15.57:0/1283032586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e74005190 0x7f2e7400b990 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c0c24a0 tx=0x7f2e8401d300 comp rx=0 tx=0).stop 2023-10-30T14:48:42.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.019+0000 7f2e909c5640 1 -- 172.21.15.57:0/1283032586 shutdown_connections 2023-10-30T14:48:42.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.019+0000 7f2e909c5640 1 --2- 172.21.15.57:0/1283032586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e74005190 0x7f2e7400b990 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:42.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.019+0000 7f2e909c5640 1 -- 172.21.15.57:0/1283032586 >> 172.21.15.57:0/1283032586 conn(0x7f2e8c0cac70 msgr2=0x7f2e8c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:42.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.019+0000 7f2e909c5640 1 -- 172.21.15.57:0/1283032586 shutdown_connections 2023-10-30T14:48:42.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.019+0000 7f2e909c5640 1 -- 172.21.15.57:0/1283032586 wait complete. 2023-10-30T14:48:42.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.019+0000 7f2e909c5640 1 Processor -- start 2023-10-30T14:48:42.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.019+0000 7f2e909c5640 1 -- start start 2023-10-30T14:48:42.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e909c5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e74005190 0x7f2e8c13f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:42.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e909c5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e8c144d60 0x7f2e8c0c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:42.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e909c5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e8c13f540 0x7f2e8c0ca490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:42.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e909c5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8c0c2050 con 0x7f2e8c13f540 2023-10-30T14:48:42.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e909c5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8c0c9c90 con 0x7f2e8c144d60 2023-10-30T14:48:42.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e909c5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8c143bb0 con 0x7f2e74005190 2023-10-30T14:48:42.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e8ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e8c13f540 0x7f2e8c0ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:42.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e8ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e8c13f540 0x7f2e8c0ca490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54982/0 (socket says 172.21.15.57:54982) 2023-10-30T14:48:42.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e8ad9d640 1 -- 172.21.15.57:0/2142382835 learned_addr learned my addr 172.21.15.57:0/2142382835 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:42.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e8a59c640 1 --2- 172.21.15.57:0/2142382835 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e74005190 0x7f2e8c13f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:42.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e89d9b640 1 --2- 172.21.15.57:0/2142382835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e8c144d60 0x7f2e8c0c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:42.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e8ad9d640 1 -- 172.21.15.57:0/2142382835 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e74005190 msgr2=0x7f2e8c13f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:42.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e8ad9d640 1 --2- 172.21.15.57:0/2142382835 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e74005190 0x7f2e8c13f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:42.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e8ad9d640 1 -- 172.21.15.57:0/2142382835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e8c144d60 msgr2=0x7f2e8c0c9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:42.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e8ad9d640 1 --2- 172.21.15.57:0/2142382835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e8c144d60 0x7f2e8c0c9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:42.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e8ad9d640 1 -- 172.21.15.57:0/2142382835 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8c143e70 con 0x7f2e8c13f540 2023-10-30T14:48:42.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e8a59c640 1 --2- 172.21.15.57:0/2142382835 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e74005190 0x7f2e8c13f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:48:42.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e89d9b640 1 --2- 172.21.15.57:0/2142382835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e8c144d60 0x7f2e8c0c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:48:42.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e8ad9d640 1 --2- 172.21.15.57:0/2142382835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e8c13f540 0x7f2e8c0ca490 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f2e8401d130 tx=0x7f2e74005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:42.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e7affd640 1 -- 172.21.15.57:0/2142382835 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e840ad630 con 0x7f2e8c13f540 2023-10-30T14:48:42.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e7affd640 1 -- 172.21.15.57:0/2142382835 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e840ad790 con 0x7f2e8c13f540 2023-10-30T14:48:42.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e909c5640 1 -- 172.21.15.57:0/2142382835 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e8c155550 con 0x7f2e8c13f540 2023-10-30T14:48:42.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e7affd640 1 -- 172.21.15.57:0/2142382835 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e840adb40 con 0x7f2e8c13f540 2023-10-30T14:48:42.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e909c5640 1 -- 172.21.15.57:0/2142382835 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e8c155b10 con 0x7f2e8c13f540 2023-10-30T14:48:42.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e909c5640 1 -- 172.21.15.57:0/2142382835 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e8c0c2050 con 0x7f2e8c13f540 2023-10-30T14:48:42.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.023+0000 7f2e7affd640 1 -- 172.21.15.57:0/2142382835 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2e840aa030 con 0x7f2e8c13f540 2023-10-30T14:48:42.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.027+0000 7f2e7affd640 1 --2- 172.21.15.57:0/2142382835 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2e6c0664a0 0x7f2e6c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:42.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.027+0000 7f2e7affd640 1 -- 172.21.15.57:0/2142382835 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7f2e8411e590 con 0x7f2e8c13f540 2023-10-30T14:48:42.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.027+0000 7f2e8a59c640 1 --2- 172.21.15.57:0/2142382835 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2e6c0664a0 0x7f2e6c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:42.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.027+0000 7f2e8a59c640 1 --2- 172.21.15.57:0/2142382835 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2e6c0664a0 0x7f2e6c068960 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009b60 tx=0x7f2e7c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:42.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.031+0000 7f2e7affd640 1 -- 172.21.15.57:0/2142382835 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2e840eb350 con 0x7f2e8c13f540 2023-10-30T14:48:42.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.215+0000 7f2e909c5640 1 -- 172.21.15.57:0/2142382835 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e8c066510 con 0x7f2e8c13f540 2023-10-30T14:48:42.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.219+0000 7f2e7affd640 1 -- 172.21.15.57:0/2142382835 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+5713 (secure 0 0 0) 0x7f2e840ee970 con 0x7f2e8c13f540 2023-10-30T14:48:42.218 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:42.219 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:39.883291+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:42.219 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 181 2023-10-30T14:48:42.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.219+0000 7f2e909c5640 1 -- 172.21.15.57:0/2142382835 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2e6c0664a0 msgr2=0x7f2e6c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:42.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.219+0000 7f2e909c5640 1 --2- 172.21.15.57:0/2142382835 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2e6c0664a0 0x7f2e6c068960 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009b60 tx=0x7f2e7c008040 comp rx=0 tx=0).stop 2023-10-30T14:48:42.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.219+0000 7f2e909c5640 1 -- 172.21.15.57:0/2142382835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e8c13f540 msgr2=0x7f2e8c0ca490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:42.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.219+0000 7f2e909c5640 1 --2- 172.21.15.57:0/2142382835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e8c13f540 0x7f2e8c0ca490 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f2e8401d130 tx=0x7f2e74005800 comp rx=0 tx=0).stop 2023-10-30T14:48:42.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.219+0000 7f2e909c5640 1 -- 172.21.15.57:0/2142382835 shutdown_connections 2023-10-30T14:48:42.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.219+0000 7f2e909c5640 1 --2- 172.21.15.57:0/2142382835 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2e6c0664a0 0x7f2e6c068960 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:42.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.219+0000 7f2e909c5640 1 --2- 172.21.15.57:0/2142382835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e8c13f540 0x7f2e8c0ca490 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:42.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.219+0000 7f2e909c5640 1 --2- 172.21.15.57:0/2142382835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e8c144d60 0x7f2e8c0c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:42.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.219+0000 7f2e909c5640 1 --2- 172.21.15.57:0/2142382835 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e74005190 0x7f2e8c13f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:42.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.219+0000 7f2e909c5640 1 -- 172.21.15.57:0/2142382835 >> 172.21.15.57:0/2142382835 conn(0x7f2e8c0cac70 msgr2=0x7f2e8c0c3890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:42.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.219+0000 7f2e909c5640 1 -- 172.21.15.57:0/2142382835 shutdown_connections 2023-10-30T14:48:42.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:42.219+0000 7f2e909c5640 1 -- 172.21.15.57:0/2142382835 wait complete. 2023-10-30T14:48:42.239 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:48:44.241 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:44.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.403+0000 7fdfdfc6b640 1 Processor -- start 2023-10-30T14:48:44.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.403+0000 7fdfdfc6b640 1 -- start start 2023-10-30T14:48:44.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.403+0000 7fdfdfc6b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fdfd8150830 0x7fdfd806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:44.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.403+0000 7fdfdfc6b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdfd8070520 0x7fdfd8070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:44.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.403+0000 7fdfdfc6b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd80c2310 con 0x7fdfd80cb7d0 2023-10-30T14:48:44.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.403+0000 7fdfdfc6b640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfd8156710 con 0x7fdfd8150830 2023-10-30T14:48:44.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.403+0000 7fdfdfc6b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfd80c4360 con 0x7fdfd8070520 2023-10-30T14:48:44.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.403+0000 7fdfdda07640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fdfd8150830 0x7fdfd806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:44.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.403+0000 7fdfdda07640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fdfd8150830 0x7fdfd806b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55666/0 (socket says 172.21.15.57:55666) 2023-10-30T14:48:44.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.403+0000 7fdfdda07640 1 -- 172.21.15.57:0/2926681496 learned_addr learned my addr 172.21.15.57:0/2926681496 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:44.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.403+0000 7fdfde208640 1 --2- 172.21.15.57:0/2926681496 >> v2:172.21.15.88:3300/0 conn(0x7fdfd8070520 0x7fdfd8070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:44.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.403+0000 7fdfdda07640 1 -- 172.21.15.57:0/2926681496 >> v2:172.21.15.88:3300/0 conn(0x7fdfd8070520 msgr2=0x7fdfd8070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:44.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.403+0000 7fdfdda07640 1 --2- 172.21.15.57:0/2926681496 >> v2:172.21.15.88:3300/0 conn(0x7fdfd8070520 0x7fdfd8070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:44.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.403+0000 7fdfdda07640 1 -- 172.21.15.57:0/2926681496 >> v1:172.21.15.88:6789/0 conn(0x7fdfd80cb7d0 legacy=0x7fdfd80c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:44.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.403+0000 7fdfdda07640 1 -- 172.21.15.57:0/2926681496 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd81569b0 con 0x7fdfd8150830 2023-10-30T14:48:44.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.407+0000 7fdfdda07640 1 --2- 172.21.15.57:0/2926681496 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfd8150830 0x7fdfd806b500 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8066e10 tx=0x7fdfd401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596a8c70deabcdfb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:44.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.407+0000 7fdfdca05640 1 -- 172.21.15.57:0/2926681496 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd40a89f0 con 0x7fdfd8150830 2023-10-30T14:48:44.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.407+0000 7fdfdca05640 1 -- 172.21.15.57:0/2926681496 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd40a8b50 con 0x7fdfd8150830 2023-10-30T14:48:44.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.407+0000 7fdfdca05640 1 -- 172.21.15.57:0/2926681496 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd40b0440 con 0x7fdfd8150830 2023-10-30T14:48:44.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.407+0000 7fdfdfc6b640 1 -- 172.21.15.57:0/2926681496 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfd8150830 msgr2=0x7fdfd806b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:44.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.407+0000 7fdfdfc6b640 1 --2- 172.21.15.57:0/2926681496 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfd8150830 0x7fdfd806b500 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8066e10 tx=0x7fdfd401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:48:44.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.407+0000 7fdfdfc6b640 1 -- 172.21.15.57:0/2926681496 shutdown_connections 2023-10-30T14:48:44.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.407+0000 7fdfdfc6b640 1 --2- 172.21.15.57:0/2926681496 >> v2:172.21.15.88:3300/0 conn(0x7fdfd8070520 0x7fdfd8070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:44.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.407+0000 7fdfdfc6b640 1 --2- 172.21.15.57:0/2926681496 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfd8150830 0x7fdfd806b500 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:44.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.407+0000 7fdfdfc6b640 1 -- 172.21.15.57:0/2926681496 >> 172.21.15.57:0/2926681496 conn(0x7fdfd80cac70 msgr2=0x7fdfd80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:44.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdfc6b640 1 -- 172.21.15.57:0/2926681496 shutdown_connections 2023-10-30T14:48:44.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdfc6b640 1 -- 172.21.15.57:0/2926681496 wait complete. 2023-10-30T14:48:44.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdfc6b640 1 Processor -- start 2023-10-30T14:48:44.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdfc6b640 1 -- start start 2023-10-30T14:48:44.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdfc6b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfd8070520 0x7fdfd81f2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:44.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdfc6b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfd80cb7d0 0x7fdfd80c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:44.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdfc6b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdfd8150830 0x7fdfd81473d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:44.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdfc6b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd80cab10 con 0x7fdfd8070520 2023-10-30T14:48:44.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdfc6b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd81f34d0 con 0x7fdfd80cb7d0 2023-10-30T14:48:44.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdfc6b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd81492b0 con 0x7fdfd8150830 2023-10-30T14:48:44.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdda07640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfd8070520 0x7fdfd81f2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:44.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdda07640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfd8070520 0x7fdfd81f2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55676/0 (socket says 172.21.15.57:55676) 2023-10-30T14:48:44.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdda07640 1 -- 172.21.15.57:0/3179362179 learned_addr learned my addr 172.21.15.57:0/3179362179 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:44.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdd206640 1 --2- 172.21.15.57:0/3179362179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfd80cb7d0 0x7fdfd80c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:44.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfde208640 1 --2- 172.21.15.57:0/3179362179 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdfd8150830 0x7fdfd81473d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:44.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdda07640 1 -- 172.21.15.57:0/3179362179 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdfd8150830 msgr2=0x7fdfd81473d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:44.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdda07640 1 --2- 172.21.15.57:0/3179362179 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdfd8150830 0x7fdfd81473d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:44.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdda07640 1 -- 172.21.15.57:0/3179362179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfd80cb7d0 msgr2=0x7fdfd80c8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:44.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdda07640 1 --2- 172.21.15.57:0/3179362179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfd80cb7d0 0x7fdfd80c8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:44.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdda07640 1 -- 172.21.15.57:0/3179362179 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd40094d0 con 0x7fdfd8070520 2023-10-30T14:48:44.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfde208640 1 --2- 172.21.15.57:0/3179362179 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdfd8150830 0x7fdfd81473d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:48:44.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfdda07640 1 --2- 172.21.15.57:0/3179362179 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfd8070520 0x7fdfd81f2f90 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fdfd401b900 tx=0x7fdfd401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:44.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.411+0000 7fdfce7fc640 1 -- 172.21.15.57:0/3179362179 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd40ba600 con 0x7fdfd8070520 2023-10-30T14:48:44.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.415+0000 7fdfce7fc640 1 -- 172.21.15.57:0/3179362179 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd40ba760 con 0x7fdfd8070520 2023-10-30T14:48:44.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.415+0000 7fdfdfc6b640 1 -- 172.21.15.57:0/3179362179 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd81ef120 con 0x7fdfd8070520 2023-10-30T14:48:44.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.415+0000 7fdfce7fc640 1 -- 172.21.15.57:0/3179362179 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd40bab10 con 0x7fdfd8070520 2023-10-30T14:48:44.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.415+0000 7fdfdfc6b640 1 -- 172.21.15.57:0/3179362179 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd81ef6e0 con 0x7fdfd8070520 2023-10-30T14:48:44.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.415+0000 7fdfdfc6b640 1 -- 172.21.15.57:0/3179362179 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd8066510 con 0x7fdfd8070520 2023-10-30T14:48:44.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.415+0000 7fdfce7fc640 1 -- 172.21.15.57:0/3179362179 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdfd40bac70 con 0x7fdfd8070520 2023-10-30T14:48:44.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.415+0000 7fdfce7fc640 1 --2- 172.21.15.57:0/3179362179 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdfbc0664a0 0x7fdfbc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:44.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.415+0000 7fdfce7fc640 1 -- 172.21.15.57:0/3179362179 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7fdfd4122a30 con 0x7fdfd8070520 2023-10-30T14:48:44.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.415+0000 7fdfdd206640 1 --2- 172.21.15.57:0/3179362179 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdfbc0664a0 0x7fdfbc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:44.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.419+0000 7fdfdd206640 1 --2- 172.21.15.57:0/3179362179 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdfbc0664a0 0x7fdfbc068960 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fdfd815acb0 tx=0x7fdfc8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:44.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.419+0000 7fdfce7fc640 1 -- 172.21.15.57:0/3179362179 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdfd40ef7f0 con 0x7fdfd8070520 2023-10-30T14:48:44.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.615+0000 7fdfdfc6b640 1 -- 172.21.15.57:0/3179362179 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfd804e170 con 0x7fdfd8070520 2023-10-30T14:48:44.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.615+0000 7fdfce7fc640 1 -- 172.21.15.57:0/3179362179 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+5713 (secure 0 0 0) 0x7fdfd40b5020 con 0x7fdfd8070520 2023-10-30T14:48:44.616 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:44.617 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:39.883291+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:44.617 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 181 2023-10-30T14:48:44.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.615+0000 7fdfdfc6b640 1 -- 172.21.15.57:0/3179362179 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdfbc0664a0 msgr2=0x7fdfbc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:44.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.615+0000 7fdfdfc6b640 1 --2- 172.21.15.57:0/3179362179 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdfbc0664a0 0x7fdfbc068960 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fdfd815acb0 tx=0x7fdfc8009040 comp rx=0 tx=0).stop 2023-10-30T14:48:44.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.615+0000 7fdfdfc6b640 1 -- 172.21.15.57:0/3179362179 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfd8070520 msgr2=0x7fdfd81f2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:44.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.615+0000 7fdfdfc6b640 1 --2- 172.21.15.57:0/3179362179 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfd8070520 0x7fdfd81f2f90 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fdfd401b900 tx=0x7fdfd401ba90 comp rx=0 tx=0).stop 2023-10-30T14:48:44.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.615+0000 7fdfdfc6b640 1 -- 172.21.15.57:0/3179362179 shutdown_connections 2023-10-30T14:48:44.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.615+0000 7fdfdfc6b640 1 --2- 172.21.15.57:0/3179362179 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdfbc0664a0 0x7fdfbc068960 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:44.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.615+0000 7fdfdfc6b640 1 --2- 172.21.15.57:0/3179362179 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdfd8150830 0x7fdfd81473d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:44.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.615+0000 7fdfdfc6b640 1 --2- 172.21.15.57:0/3179362179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfd80cb7d0 0x7fdfd80c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:44.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.615+0000 7fdfdfc6b640 1 --2- 172.21.15.57:0/3179362179 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfd8070520 0x7fdfd81f2f90 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:44.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.615+0000 7fdfdfc6b640 1 -- 172.21.15.57:0/3179362179 >> 172.21.15.57:0/3179362179 conn(0x7fdfd80cac70 msgr2=0x7fdfd80c7680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:44.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.619+0000 7fdfdfc6b640 1 -- 172.21.15.57:0/3179362179 shutdown_connections 2023-10-30T14:48:44.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:44.619+0000 7fdfdfc6b640 1 -- 172.21.15.57:0/3179362179 wait complete. 2023-10-30T14:48:44.639 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 181, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 181, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7911": { "addr": "172.21.15.57:6836/2613636524", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6835", "nonce": 2613636524, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 2613636524, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7911, "incarnation": 180, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 26 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:48:39.883291+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7911 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6839/1078372885", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1078372885, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1078372885, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 170, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6716, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.57:6834/697507447", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 697507447, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 697507447, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 156, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7377, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:48:46.640 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:46.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.795+0000 7f57684d6640 1 Processor -- start 2023-10-30T14:48:46.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.795+0000 7f57684d6640 1 -- start start 2023-10-30T14:48:46.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.795+0000 7f57684d6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5760071340 0x7f5760065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:46.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.795+0000 7f57684d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5760159970 0x7f5760159e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:46.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.795+0000 7f57684d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576006e390 con 0x7f5760153c10 2023-10-30T14:48:46.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.795+0000 7f57684d6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5760072390 con 0x7f5760071340 2023-10-30T14:48:46.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.795+0000 7f57684d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57600660e0 con 0x7f5760159970 2023-10-30T14:48:46.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.795+0000 7f5765a71640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5760071340 0x7f5760065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:46.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.795+0000 7f5765a71640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5760071340 0x7f5760065ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55690/0 (socket says 172.21.15.57:55690) 2023-10-30T14:48:46.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.795+0000 7f5765a71640 1 -- 172.21.15.57:0/4227576907 learned_addr learned my addr 172.21.15.57:0/4227576907 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:46.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.795+0000 7f5766a73640 1 --2- 172.21.15.57:0/4227576907 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5760159970 0x7f5760159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:46.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.799+0000 7f5765a71640 1 -- 172.21.15.57:0/4227576907 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5760159970 msgr2=0x7f5760159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:46.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.799+0000 7f5765a71640 1 --2- 172.21.15.57:0/4227576907 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5760159970 0x7f5760159e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:46.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.799+0000 7f5765a71640 1 -- 172.21.15.57:0/4227576907 >> v1:172.21.15.88:6789/0 conn(0x7f5760153c10 legacy=0x7f5760070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:46.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.799+0000 7f5765a71640 1 -- 172.21.15.57:0/4227576907 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57600bec10 con 0x7f5760071340 2023-10-30T14:48:46.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.799+0000 7f5765a71640 1 --2- 172.21.15.57:0/4227576907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5760071340 0x7f5760065ba0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f5750009870 tx=0x7f575001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23f07073ed0c440c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:46.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.799+0000 7f5765270640 1 -- 172.21.15.57:0/4227576907 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57500ab600 con 0x7f5760071340 2023-10-30T14:48:46.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.799+0000 7f5765270640 1 -- 172.21.15.57:0/4227576907 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57500a7050 con 0x7f5760071340 2023-10-30T14:48:46.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.799+0000 7f5765270640 1 -- 172.21.15.57:0/4227576907 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57500ab980 con 0x7f5760071340 2023-10-30T14:48:46.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.799+0000 7f57684d6640 1 -- 172.21.15.57:0/4227576907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5760071340 msgr2=0x7f5760065ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:46.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.799+0000 7f57684d6640 1 --2- 172.21.15.57:0/4227576907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5760071340 0x7f5760065ba0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f5750009870 tx=0x7f575001c4b0 comp rx=0 tx=0).stop 2023-10-30T14:48:46.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f57684d6640 1 -- 172.21.15.57:0/4227576907 shutdown_connections 2023-10-30T14:48:46.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f57684d6640 1 --2- 172.21.15.57:0/4227576907 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5760159970 0x7f5760159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:46.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f57684d6640 1 --2- 172.21.15.57:0/4227576907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5760071340 0x7f5760065ba0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:46.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f57684d6640 1 -- 172.21.15.57:0/4227576907 >> 172.21.15.57:0/4227576907 conn(0x7f57600c7f60 msgr2=0x7f57600c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:46.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f57684d6640 1 -- 172.21.15.57:0/4227576907 shutdown_connections 2023-10-30T14:48:46.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f57684d6640 1 -- 172.21.15.57:0/4227576907 wait complete. 2023-10-30T14:48:46.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f57684d6640 1 Processor -- start 2023-10-30T14:48:46.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f57684d6640 1 -- start start 2023-10-30T14:48:46.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f57684d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5760071340 0x7f576014b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:46.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f57684d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5760153c10 0x7f57600c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:46.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f57684d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5760159970 0x7f57600c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:46.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f57684d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57600ca520 con 0x7f5760071340 2023-10-30T14:48:46.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f57684d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5760146950 con 0x7f5760153c10 2023-10-30T14:48:46.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f57684d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576014e0d0 con 0x7f5760159970 2023-10-30T14:48:46.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f5766272640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5760071340 0x7f576014b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:46.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f5766272640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5760071340 0x7f576014b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55702/0 (socket says 172.21.15.57:55702) 2023-10-30T14:48:46.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f5766272640 1 -- 172.21.15.57:0/4070760852 learned_addr learned my addr 172.21.15.57:0/4070760852 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:46.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f5766a73640 1 --2- 172.21.15.57:0/4070760852 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5760159970 0x7f57600c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:46.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f5765a71640 1 --2- 172.21.15.57:0/4070760852 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5760153c10 0x7f57600c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:46.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f5766272640 1 -- 172.21.15.57:0/4070760852 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5760159970 msgr2=0x7f57600c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:46.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f5766272640 1 --2- 172.21.15.57:0/4070760852 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5760159970 0x7f57600c9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:46.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f5766272640 1 -- 172.21.15.57:0/4070760852 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5760153c10 msgr2=0x7f57600c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:46.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f5766272640 1 --2- 172.21.15.57:0/4070760852 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5760153c10 0x7f57600c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:46.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f5766272640 1 -- 172.21.15.57:0/4070760852 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57500094d0 con 0x7f5760071340 2023-10-30T14:48:46.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.803+0000 7f5766272640 1 --2- 172.21.15.57:0/4070760852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5760071340 0x7f576014b820 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f575c00c530 tx=0x7f575c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:46.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.807+0000 7f5756ffd640 1 -- 172.21.15.57:0/4070760852 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c005220 con 0x7f5760071340 2023-10-30T14:48:46.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.807+0000 7f5756ffd640 1 -- 172.21.15.57:0/4070760852 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575c005380 con 0x7f5760071340 2023-10-30T14:48:46.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.807+0000 7f5756ffd640 1 -- 172.21.15.57:0/4070760852 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c007270 con 0x7f5760071340 2023-10-30T14:48:46.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.807+0000 7f57684d6640 1 -- 172.21.15.57:0/4070760852 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5760149b30 con 0x7f5760071340 2023-10-30T14:48:46.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.807+0000 7f57684d6640 1 -- 172.21.15.57:0/4070760852 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57600c92c0 con 0x7f5760071340 2023-10-30T14:48:46.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.807+0000 7f57684d6640 1 -- 172.21.15.57:0/4070760852 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f576004e170 con 0x7f5760071340 2023-10-30T14:48:46.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.807+0000 7f5756ffd640 1 -- 172.21.15.57:0/4070760852 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f575c0073d0 con 0x7f5760071340 2023-10-30T14:48:46.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.807+0000 7f5756ffd640 1 --2- 172.21.15.57:0/4070760852 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5744066520 0x7f57440689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:46.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.807+0000 7f5756ffd640 1 -- 172.21.15.57:0/4070760852 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7f575c083cb0 con 0x7f5760071340 2023-10-30T14:48:46.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.807+0000 7f5765a71640 1 --2- 172.21.15.57:0/4070760852 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5744066520 0x7f57440689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:46.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.811+0000 7f5765a71640 1 --2- 172.21.15.57:0/4070760852 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5744066520 0x7f57440689e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f57500ad000 tx=0x7f5750008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:46.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:46.811+0000 7f5756ffd640 1 -- 172.21.15.57:0/4070760852 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f575c050b90 con 0x7f5760071340 2023-10-30T14:48:47.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:47.003+0000 7f57684d6640 1 -- 172.21.15.57:0/4070760852 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5760158220 con 0x7f5760071340 2023-10-30T14:48:47.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:47.003+0000 7f5756ffd640 1 -- 172.21.15.57:0/4070760852 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+5710 (secure 0 0 0) 0x7f575c0541b0 con 0x7f5760071340 2023-10-30T14:48:47.005 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:47.006 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:45.445730+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:47.007 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 182 2023-10-30T14:48:47.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:47.007+0000 7f57684d6640 1 -- 172.21.15.57:0/4070760852 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5744066520 msgr2=0x7f57440689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:47.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:47.007+0000 7f57684d6640 1 --2- 172.21.15.57:0/4070760852 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5744066520 0x7f57440689e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f57500ad000 tx=0x7f5750008040 comp rx=0 tx=0).stop 2023-10-30T14:48:47.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:47.007+0000 7f57684d6640 1 -- 172.21.15.57:0/4070760852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5760071340 msgr2=0x7f576014b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:47.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:47.007+0000 7f57684d6640 1 --2- 172.21.15.57:0/4070760852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5760071340 0x7f576014b820 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f575c00c530 tx=0x7f575c00c9f0 comp rx=0 tx=0).stop 2023-10-30T14:48:47.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:47.007+0000 7f57684d6640 1 -- 172.21.15.57:0/4070760852 shutdown_connections 2023-10-30T14:48:47.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:47.007+0000 7f57684d6640 1 --2- 172.21.15.57:0/4070760852 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5744066520 0x7f57440689e0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:47.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:47.007+0000 7f57684d6640 1 --2- 172.21.15.57:0/4070760852 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5760159970 0x7f57600c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:47.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:47.007+0000 7f57684d6640 1 --2- 172.21.15.57:0/4070760852 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5760153c10 0x7f57600c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:47.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:47.007+0000 7f57684d6640 1 --2- 172.21.15.57:0/4070760852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5760071340 0x7f576014b820 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:47.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:47.007+0000 7f57684d6640 1 -- 172.21.15.57:0/4070760852 >> 172.21.15.57:0/4070760852 conn(0x7f57600c7f60 msgr2=0x7f576006d750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:47.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:47.007+0000 7f57684d6640 1 -- 172.21.15.57:0/4070760852 shutdown_connections 2023-10-30T14:48:47.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:47.007+0000 7f57684d6640 1 -- 172.21.15.57:0/4070760852 wait complete. 2023-10-30T14:48:47.026 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:48:49.028 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:49.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.183+0000 7f5d011c4640 1 Processor -- start 2023-10-30T14:48:49.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.183+0000 7f5d011c4640 1 -- start start 2023-10-30T14:48:49.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.183+0000 7f5d011c4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cfc150830 0x7f5cfc06b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:49.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.183+0000 7f5d011c4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5cfc0cb7d0 0x7f5cfc0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:49.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.183+0000 7f5d011c4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cfc0c2310 con 0x7f5cfc0c3e40 2023-10-30T14:48:49.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.183+0000 7f5d011c4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cfc156710 con 0x7f5cfc0cb7d0 2023-10-30T14:48:49.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.183+0000 7f5d011c4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc0c4360 con 0x7f5cfc150830 2023-10-30T14:48:49.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.183+0000 7f5cfad9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cfc150830 0x7f5cfc06b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:49.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.183+0000 7f5cfa59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5cfc0cb7d0 0x7f5cfc0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:49.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.183+0000 7f5cfb59e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5cfc0c3e40 0x7f5cfc070520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:36072/0 (socket says 172.21.15.57:36072) 2023-10-30T14:48:49.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.183+0000 7f5cfb59e640 1 -- 172.21.15.57:0/2498695833 learned_addr learned my addr 172.21.15.57:0/2498695833 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:48:49.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.183+0000 7f5cf9d9b640 1 -- 172.21.15.57:0/2498695833 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470161505 0 0) 0x7f5cfc0c2310 con 0x7f5cfc0c3e40 2023-10-30T14:48:49.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.183+0000 7f5cf9d9b640 1 -- 172.21.15.57:0/2498695833 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ce4003530 con 0x7f5cfc0c3e40 2023-10-30T14:48:49.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.183+0000 7f5cfad9d640 1 -- 172.21.15.57:0/2498695833 >> v2:172.21.15.88:3300/0 conn(0x7f5cfc0cb7d0 msgr2=0x7f5cfc0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:49.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.183+0000 7f5cfad9d640 1 --2- 172.21.15.57:0/2498695833 >> v2:172.21.15.88:3300/0 conn(0x7f5cfc0cb7d0 0x7f5cfc0c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:49.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.183+0000 7f5cfad9d640 1 -- 172.21.15.57:0/2498695833 >> v1:172.21.15.88:6789/0 conn(0x7f5cfc0c3e40 legacy=0x7f5cfc070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:49.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.183+0000 7f5cfad9d640 1 -- 172.21.15.57:0/2498695833 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cfc1569b0 con 0x7f5cfc150830 2023-10-30T14:48:49.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.183+0000 7f5cfa59c640 1 --2- 172.21.15.57:0/2498695833 >> v2:172.21.15.88:3300/0 conn(0x7f5cfc0cb7d0 0x7f5cfc0c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:48:49.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.187+0000 7f5cfad9d640 1 --2- 172.21.15.57:0/2498695833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cfc150830 0x7f5cfc06b500 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5cec0097c0 tx=0x7f5cec01ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=170a6111f8ac9e7d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:49.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.187+0000 7f5cf9d9b640 1 -- 172.21.15.57:0/2498695833 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec0a8970 con 0x7f5cfc150830 2023-10-30T14:48:49.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.187+0000 7f5cf9d9b640 1 -- 172.21.15.57:0/2498695833 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cec0a8ad0 con 0x7f5cfc150830 2023-10-30T14:48:49.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.187+0000 7f5cf9d9b640 1 -- 172.21.15.57:0/2498695833 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec0a8e60 con 0x7f5cfc150830 2023-10-30T14:48:49.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.187+0000 7f5d011c4640 1 -- 172.21.15.57:0/2498695833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cfc150830 msgr2=0x7f5cfc06b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:49.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.187+0000 7f5d011c4640 1 --2- 172.21.15.57:0/2498695833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cfc150830 0x7f5cfc06b500 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5cec0097c0 tx=0x7f5cec01ec70 comp rx=0 tx=0).stop 2023-10-30T14:48:49.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.187+0000 7f5d011c4640 1 -- 172.21.15.57:0/2498695833 shutdown_connections 2023-10-30T14:48:49.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.187+0000 7f5d011c4640 1 --2- 172.21.15.57:0/2498695833 >> v2:172.21.15.88:3300/0 conn(0x7f5cfc0cb7d0 0x7f5cfc0c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:49.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.187+0000 7f5d011c4640 1 --2- 172.21.15.57:0/2498695833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cfc150830 0x7f5cfc06b500 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:49.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.187+0000 7f5d011c4640 1 -- 172.21.15.57:0/2498695833 >> 172.21.15.57:0/2498695833 conn(0x7f5cfc0cac70 msgr2=0x7f5cfc0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:49.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.187+0000 7f5d011c4640 1 -- 172.21.15.57:0/2498695833 shutdown_connections 2023-10-30T14:48:49.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.187+0000 7f5d011c4640 1 -- 172.21.15.57:0/2498695833 wait complete. 2023-10-30T14:48:49.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5d011c4640 1 Processor -- start 2023-10-30T14:48:49.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5d011c4640 1 -- start start 2023-10-30T14:48:49.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5d011c4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cfc0c3e40 0x7f5cfc1f18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:49.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5d011c4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5cfc0cb7d0 0x7f5cfc14cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:49.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5d011c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cfc150830 0x7f5cfc14e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:49.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5d011c4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc1f30c0 con 0x7f5cfc0cb7d0 2023-10-30T14:48:49.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5d011c4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc1f2540 con 0x7f5cfc150830 2023-10-30T14:48:49.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5d011c4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc1f2820 con 0x7f5cfc0c3e40 2023-10-30T14:48:49.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5cfa59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5cfc0cb7d0 0x7f5cfc14cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:49.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5cfa59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5cfc0cb7d0 0x7f5cfc14cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55716/0 (socket says 172.21.15.57:55716) 2023-10-30T14:48:49.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5cfa59c640 1 -- 172.21.15.57:0/1347056598 learned_addr learned my addr 172.21.15.57:0/1347056598 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:49.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5cfad9d640 1 --2- 172.21.15.57:0/1347056598 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cfc0c3e40 0x7f5cfc1f18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:49.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5cfb59e640 1 --2- 172.21.15.57:0/1347056598 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cfc150830 0x7f5cfc14e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:49.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5cfa59c640 1 -- 172.21.15.57:0/1347056598 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cfc0c3e40 msgr2=0x7f5cfc1f18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:49.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5cfa59c640 1 --2- 172.21.15.57:0/1347056598 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cfc0c3e40 0x7f5cfc1f18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:49.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5cfa59c640 1 -- 172.21.15.57:0/1347056598 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cfc150830 msgr2=0x7f5cfc14e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:49.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5cfa59c640 1 --2- 172.21.15.57:0/1347056598 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cfc150830 0x7f5cfc14e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:49.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5cfa59c640 1 -- 172.21.15.57:0/1347056598 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf0009420 con 0x7f5cfc0cb7d0 2023-10-30T14:48:49.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5cfa59c640 1 --2- 172.21.15.57:0/1347056598 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5cfc0cb7d0 0x7f5cfc14cb40 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000ebe0 tx=0x7f5cf000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:49.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5ce37fe640 1 -- 172.21.15.57:0/1347056598 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf000eee0 con 0x7f5cfc0cb7d0 2023-10-30T14:48:49.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5ce37fe640 1 -- 172.21.15.57:0/1347056598 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf000cc30 con 0x7f5cfc0cb7d0 2023-10-30T14:48:49.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5ce37fe640 1 -- 172.21.15.57:0/1347056598 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0013470 con 0x7f5cfc0cb7d0 2023-10-30T14:48:49.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5d011c4640 1 -- 172.21.15.57:0/1347056598 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cec009420 con 0x7f5cfc0cb7d0 2023-10-30T14:48:49.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.191+0000 7f5d011c4640 1 -- 172.21.15.57:0/1347056598 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cfc1f2de0 con 0x7f5cfc0cb7d0 2023-10-30T14:48:49.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.195+0000 7f5d011c4640 1 -- 172.21.15.57:0/1347056598 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cfc0c6c00 con 0x7f5cfc0cb7d0 2023-10-30T14:48:49.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.195+0000 7f5ce37fe640 1 -- 172.21.15.57:0/1347056598 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5cf00135d0 con 0x7f5cfc0cb7d0 2023-10-30T14:48:49.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.195+0000 7f5ce37fe640 1 --2- 172.21.15.57:0/1347056598 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5ce80664a0 0x7f5ce8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:49.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.195+0000 7f5ce37fe640 1 -- 172.21.15.57:0/1347056598 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7f5cf0084710 con 0x7f5cfc0cb7d0 2023-10-30T14:48:49.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.195+0000 7f5cfad9d640 1 --2- 172.21.15.57:0/1347056598 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5ce80664a0 0x7f5ce8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:49.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.195+0000 7f5cfad9d640 1 --2- 172.21.15.57:0/1347056598 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5ce80664a0 0x7f5ce8068960 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f5cec0b2000 tx=0x7f5cec0aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:49.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.199+0000 7f5ce37fe640 1 -- 172.21.15.57:0/1347056598 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5cf0051600 con 0x7f5cfc0cb7d0 2023-10-30T14:48:49.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.387+0000 7f5d011c4640 1 -- 172.21.15.57:0/1347056598 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5cfc04e170 con 0x7f5cfc0cb7d0 2023-10-30T14:48:49.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.391+0000 7f5ce37fe640 1 -- 172.21.15.57:0/1347056598 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+5710 (secure 0 0 0) 0x7f5cf0054c20 con 0x7f5cfc0cb7d0 2023-10-30T14:48:49.391 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:49.391 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:45.445730+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:49.392 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 182 2023-10-30T14:48:49.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.391+0000 7f5d011c4640 1 -- 172.21.15.57:0/1347056598 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5ce80664a0 msgr2=0x7f5ce8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:49.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.391+0000 7f5d011c4640 1 --2- 172.21.15.57:0/1347056598 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5ce80664a0 0x7f5ce8068960 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f5cec0b2000 tx=0x7f5cec0aa070 comp rx=0 tx=0).stop 2023-10-30T14:48:49.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.391+0000 7f5d011c4640 1 -- 172.21.15.57:0/1347056598 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5cfc0cb7d0 msgr2=0x7f5cfc14cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:49.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.391+0000 7f5d011c4640 1 --2- 172.21.15.57:0/1347056598 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5cfc0cb7d0 0x7f5cfc14cb40 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000ebe0 tx=0x7f5cf000c520 comp rx=0 tx=0).stop 2023-10-30T14:48:49.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.391+0000 7f5d011c4640 1 -- 172.21.15.57:0/1347056598 shutdown_connections 2023-10-30T14:48:49.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.391+0000 7f5d011c4640 1 --2- 172.21.15.57:0/1347056598 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f5ce80664a0 0x7f5ce8068960 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:49.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.391+0000 7f5d011c4640 1 --2- 172.21.15.57:0/1347056598 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cfc150830 0x7f5cfc14e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:49.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.391+0000 7f5d011c4640 1 --2- 172.21.15.57:0/1347056598 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5cfc0cb7d0 0x7f5cfc14cb40 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:49.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.391+0000 7f5d011c4640 1 --2- 172.21.15.57:0/1347056598 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cfc0c3e40 0x7f5cfc1f18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:49.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.391+0000 7f5d011c4640 1 -- 172.21.15.57:0/1347056598 >> 172.21.15.57:0/1347056598 conn(0x7f5cfc0cac70 msgr2=0x7f5cfc0c6210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:49.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.391+0000 7f5d011c4640 1 -- 172.21.15.57:0/1347056598 shutdown_connections 2023-10-30T14:48:49.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:49.391+0000 7f5d011c4640 1 -- 172.21.15.57:0/1347056598 wait complete. 2023-10-30T14:48:49.412 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:48:51.414 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:51.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.555+0000 7f18e530f640 1 Processor -- start 2023-10-30T14:48:51.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.555+0000 7f18e530f640 1 -- start start 2023-10-30T14:48:51.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.559+0000 7f18e530f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18e015d840 0x7f18e00ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:51.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.559+0000 7f18e530f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f18e00cd420 0x7f18e006b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:51.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.559+0000 7f18e530f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e00c7b00 con 0x7f18e0153aa0 2023-10-30T14:48:51.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.559+0000 7f18e530f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e0070a50 con 0x7f18e00cd420 2023-10-30T14:48:51.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.559+0000 7f18e530f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e00c2050 con 0x7f18e015d840 2023-10-30T14:48:51.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.559+0000 7f18df7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f18e00cd420 0x7f18e006b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:51.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.559+0000 7f18df7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f18e00cd420 0x7f18e006b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55730/0 (socket says 172.21.15.57:55730) 2023-10-30T14:48:51.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.559+0000 7f18df7fe640 1 -- 172.21.15.57:0/1583741755 learned_addr learned my addr 172.21.15.57:0/1583741755 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:51.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.559+0000 7f18de7fc640 1 --2- 172.21.15.57:0/1583741755 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18e015d840 0x7f18e00ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:51.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.559+0000 7f18df7fe640 1 -- 172.21.15.57:0/1583741755 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18e015d840 msgr2=0x7f18e00ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:51.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.559+0000 7f18df7fe640 1 --2- 172.21.15.57:0/1583741755 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18e015d840 0x7f18e00ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:51.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.559+0000 7f18df7fe640 1 -- 172.21.15.57:0/1583741755 >> v1:172.21.15.88:6789/0 conn(0x7f18e0153aa0 legacy=0x7f18e00701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:51.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.559+0000 7f18df7fe640 1 -- 172.21.15.57:0/1583741755 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e00c2330 con 0x7f18e00cd420 2023-10-30T14:48:51.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.559+0000 7f18de7fc640 1 --2- 172.21.15.57:0/1583741755 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18e015d840 0x7f18e00ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:48:51.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18df7fe640 1 --2- 172.21.15.57:0/1583741755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18e00cd420 0x7f18e006b3b0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f18d8009e50 tx=0x7f18d801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=571cf76480f95081 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:51.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18ddffb640 1 -- 172.21.15.57:0/1583741755 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d80af400 con 0x7f18e00cd420 2023-10-30T14:48:51.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18ddffb640 1 -- 172.21.15.57:0/1583741755 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d80ab050 con 0x7f18e00cd420 2023-10-30T14:48:51.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18ddffb640 1 -- 172.21.15.57:0/1583741755 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d80af780 con 0x7f18e00cd420 2023-10-30T14:48:51.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18e530f640 1 -- 172.21.15.57:0/1583741755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18e00cd420 msgr2=0x7f18e006b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:51.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18e530f640 1 --2- 172.21.15.57:0/1583741755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18e00cd420 0x7f18e006b3b0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f18d8009e50 tx=0x7f18d801f1f0 comp rx=0 tx=0).stop 2023-10-30T14:48:51.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18e530f640 1 -- 172.21.15.57:0/1583741755 shutdown_connections 2023-10-30T14:48:51.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18e530f640 1 --2- 172.21.15.57:0/1583741755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18e00cd420 0x7f18e006b3b0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:51.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18e530f640 1 --2- 172.21.15.57:0/1583741755 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18e015d840 0x7f18e00ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:51.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18e530f640 1 -- 172.21.15.57:0/1583741755 >> 172.21.15.57:0/1583741755 conn(0x7f18e0071fb0 msgr2=0x7f18e00723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:51.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18e530f640 1 -- 172.21.15.57:0/1583741755 shutdown_connections 2023-10-30T14:48:51.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18e530f640 1 -- 172.21.15.57:0/1583741755 wait complete. 2023-10-30T14:48:51.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18e530f640 1 Processor -- start 2023-10-30T14:48:51.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18e530f640 1 -- start start 2023-10-30T14:48:51.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18e530f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18e00cd420 0x7f18e01504d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:51.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18e530f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18e0153aa0 0x7f18e0150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:51.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18e530f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18e015d840 0x7f18e00cb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:51.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18e530f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e014ee90 con 0x7f18e0153aa0 2023-10-30T14:48:51.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18e530f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e00ccb00 con 0x7f18e00cd420 2023-10-30T14:48:51.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.563+0000 7f18e530f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e00c6a90 con 0x7f18e015d840 2023-10-30T14:48:51.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.567+0000 7f18de7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18e0153aa0 0x7f18e0150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:51.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.567+0000 7f18de7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18e0153aa0 0x7f18e0150a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55738/0 (socket says 172.21.15.57:55738) 2023-10-30T14:48:51.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.567+0000 7f18de7fc640 1 -- 172.21.15.57:0/3755613544 learned_addr learned my addr 172.21.15.57:0/3755613544 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:51.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.567+0000 7f18df7fe640 1 --2- 172.21.15.57:0/3755613544 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18e015d840 0x7f18e00cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:51.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.567+0000 7f18deffd640 1 --2- 172.21.15.57:0/3755613544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18e00cd420 0x7f18e01504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:51.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.567+0000 7f18de7fc640 1 -- 172.21.15.57:0/3755613544 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18e015d840 msgr2=0x7f18e00cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:51.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.567+0000 7f18de7fc640 1 --2- 172.21.15.57:0/3755613544 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18e015d840 0x7f18e00cb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:51.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.567+0000 7f18de7fc640 1 -- 172.21.15.57:0/3755613544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18e00cd420 msgr2=0x7f18e01504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:51.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.567+0000 7f18de7fc640 1 --2- 172.21.15.57:0/3755613544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18e00cd420 0x7f18e01504d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:51.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.567+0000 7f18de7fc640 1 -- 172.21.15.57:0/3755613544 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d8009b30 con 0x7f18e0153aa0 2023-10-30T14:48:51.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.567+0000 7f18deffd640 1 --2- 172.21.15.57:0/3755613544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18e00cd420 0x7f18e01504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:48:51.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.567+0000 7f18de7fc640 1 --2- 172.21.15.57:0/3755613544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18e0153aa0 0x7f18e0150a10 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f18d400d800 tx=0x7f18d400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:51.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.567+0000 7f18c77fe640 1 -- 172.21.15.57:0/3755613544 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d4009660 con 0x7f18e0153aa0 2023-10-30T14:48:51.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.567+0000 7f18c77fe640 1 -- 172.21.15.57:0/3755613544 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d40097c0 con 0x7f18e0153aa0 2023-10-30T14:48:51.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.567+0000 7f18e530f640 1 -- 172.21.15.57:0/3755613544 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e00c6d30 con 0x7f18e0153aa0 2023-10-30T14:48:51.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.567+0000 7f18c77fe640 1 -- 172.21.15.57:0/3755613544 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d4009b70 con 0x7f18e0153aa0 2023-10-30T14:48:51.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.567+0000 7f18e530f640 1 -- 172.21.15.57:0/3755613544 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e00ccd50 con 0x7f18e0153aa0 2023-10-30T14:48:51.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.567+0000 7f18e530f640 1 -- 172.21.15.57:0/3755613544 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e004e170 con 0x7f18e0153aa0 2023-10-30T14:48:51.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.567+0000 7f18c77fe640 1 -- 172.21.15.57:0/3755613544 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f18d4009cd0 con 0x7f18e0153aa0 2023-10-30T14:48:51.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.571+0000 7f18c77fe640 1 --2- 172.21.15.57:0/3755613544 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f18cc0664a0 0x7f18cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:51.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.571+0000 7f18c77fe640 1 -- 172.21.15.57:0/3755613544 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7f18d4084b40 con 0x7f18e0153aa0 2023-10-30T14:48:51.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.571+0000 7f18deffd640 1 --2- 172.21.15.57:0/3755613544 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f18cc0664a0 0x7f18cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:51.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.571+0000 7f18deffd640 1 --2- 172.21.15.57:0/3755613544 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f18cc0664a0 0x7f18cc068960 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f18d0000f70 tx=0x7f18d0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:51.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.575+0000 7f18c77fe640 1 -- 172.21.15.57:0/3755613544 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f18d4051990 con 0x7f18e0153aa0 2023-10-30T14:48:51.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.767+0000 7f18e530f640 1 -- 172.21.15.57:0/3755613544 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18e00cadb0 con 0x7f18e0153aa0 2023-10-30T14:48:51.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.767+0000 7f18c77fe640 1 -- 172.21.15.57:0/3755613544 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+5710 (secure 0 0 0) 0x7f18d4082e60 con 0x7f18e0153aa0 2023-10-30T14:48:51.767 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:51.768 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:45.445730+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:51.768 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 182 2023-10-30T14:48:51.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.767+0000 7f18e530f640 1 -- 172.21.15.57:0/3755613544 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f18cc0664a0 msgr2=0x7f18cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:51.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.767+0000 7f18e530f640 1 --2- 172.21.15.57:0/3755613544 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f18cc0664a0 0x7f18cc068960 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f18d0000f70 tx=0x7f18d0009040 comp rx=0 tx=0).stop 2023-10-30T14:48:51.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.767+0000 7f18e530f640 1 -- 172.21.15.57:0/3755613544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18e0153aa0 msgr2=0x7f18e0150a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:51.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.767+0000 7f18e530f640 1 --2- 172.21.15.57:0/3755613544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18e0153aa0 0x7f18e0150a10 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f18d400d800 tx=0x7f18d400dcc0 comp rx=0 tx=0).stop 2023-10-30T14:48:51.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.767+0000 7f18e530f640 1 -- 172.21.15.57:0/3755613544 shutdown_connections 2023-10-30T14:48:51.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.767+0000 7f18e530f640 1 --2- 172.21.15.57:0/3755613544 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f18cc0664a0 0x7f18cc068960 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:51.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.767+0000 7f18e530f640 1 --2- 172.21.15.57:0/3755613544 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18e015d840 0x7f18e00cb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:51.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.767+0000 7f18e530f640 1 --2- 172.21.15.57:0/3755613544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18e0153aa0 0x7f18e0150a10 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:51.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.767+0000 7f18e530f640 1 --2- 172.21.15.57:0/3755613544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18e00cd420 0x7f18e01504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:51.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.767+0000 7f18e530f640 1 -- 172.21.15.57:0/3755613544 >> 172.21.15.57:0/3755613544 conn(0x7f18e0071fb0 msgr2=0x7f18e015c230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:51.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.767+0000 7f18e530f640 1 -- 172.21.15.57:0/3755613544 shutdown_connections 2023-10-30T14:48:51.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:51.767+0000 7f18e530f640 1 -- 172.21.15.57:0/3755613544 wait complete. 2023-10-30T14:48:51.789 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:48:53.790 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:53.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.942+0000 7f907f594640 1 Processor -- start 2023-10-30T14:48:53.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.942+0000 7f907f594640 1 -- start start 2023-10-30T14:48:53.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.942+0000 7f907f594640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9078151840 0x7f90780c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:53.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.946+0000 7f907f594640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f90780cb8b0 0x7f907806b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:53.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.946+0000 7f907f594640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90780c5030 con 0x7f9078151c20 2023-10-30T14:48:53.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.946+0000 7f907f594640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9078157830 con 0x7f9078151840 2023-10-30T14:48:53.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.946+0000 7f907f594640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9078157b10 con 0x7f90780cb8b0 2023-10-30T14:48:53.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.946+0000 7f907d330640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9078151840 0x7f90780c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:53.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.946+0000 7f907d330640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9078151840 0x7f90780c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35326/0 (socket says 172.21.15.57:35326) 2023-10-30T14:48:53.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.946+0000 7f907d330640 1 -- 172.21.15.57:0/3113092814 learned_addr learned my addr 172.21.15.57:0/3113092814 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:53.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.946+0000 7f907db31640 1 --2- 172.21.15.57:0/3113092814 >> v2:172.21.15.88:3300/0 conn(0x7f90780cb8b0 0x7f907806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:53.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.946+0000 7f907d330640 1 -- 172.21.15.57:0/3113092814 >> v2:172.21.15.88:3300/0 conn(0x7f90780cb8b0 msgr2=0x7f907806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:53.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.946+0000 7f907d330640 1 --2- 172.21.15.57:0/3113092814 >> v2:172.21.15.88:3300/0 conn(0x7f90780cb8b0 0x7f907806b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:53.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.946+0000 7f907d330640 1 -- 172.21.15.57:0/3113092814 >> v1:172.21.15.88:6789/0 conn(0x7f9078151c20 legacy=0x7f9078072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:48:53.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.946+0000 7f907d330640 1 -- 172.21.15.57:0/3113092814 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9078157df0 con 0x7f9078151840 2023-10-30T14:48:53.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.946+0000 7f907d330640 1 --2- 172.21.15.57:0/3113092814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9078151840 0x7f90780c49a0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f90780c6cb0 tx=0x7f907401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376976b0fc399e4d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:53.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.946+0000 7f906ffff640 1 -- 172.21.15.57:0/3113092814 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90740a89b0 con 0x7f9078151840 2023-10-30T14:48:53.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.946+0000 7f906ffff640 1 -- 172.21.15.57:0/3113092814 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90740a8b10 con 0x7f9078151840 2023-10-30T14:48:53.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.946+0000 7f906ffff640 1 -- 172.21.15.57:0/3113092814 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90740a8ea0 con 0x7f9078151840 2023-10-30T14:48:53.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907f594640 1 -- 172.21.15.57:0/3113092814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9078151840 msgr2=0x7f90780c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:53.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907f594640 1 --2- 172.21.15.57:0/3113092814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9078151840 0x7f90780c49a0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f90780c6cb0 tx=0x7f907401ecb0 comp rx=0 tx=0).stop 2023-10-30T14:48:53.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907f594640 1 -- 172.21.15.57:0/3113092814 shutdown_connections 2023-10-30T14:48:53.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907f594640 1 --2- 172.21.15.57:0/3113092814 >> v2:172.21.15.88:3300/0 conn(0x7f90780cb8b0 0x7f907806b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:53.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907f594640 1 --2- 172.21.15.57:0/3113092814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9078151840 0x7f90780c49a0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:53.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907f594640 1 -- 172.21.15.57:0/3113092814 >> 172.21.15.57:0/3113092814 conn(0x7f90780cac70 msgr2=0x7f90780cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:53.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907f594640 1 -- 172.21.15.57:0/3113092814 shutdown_connections 2023-10-30T14:48:53.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907f594640 1 -- 172.21.15.57:0/3113092814 wait complete. 2023-10-30T14:48:53.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907f594640 1 Processor -- start 2023-10-30T14:48:53.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907f594640 1 -- start start 2023-10-30T14:48:53.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907f594640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90780cb8b0 0x7f907814a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:53.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907f594640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9078151840 0x7f907814e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:53.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907f594640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9078151c20 0x7f9078145c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:53.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907f594640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907814e1c0 con 0x7f9078151840 2023-10-30T14:48:53.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907f594640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907814eb70 con 0x7f9078151c20 2023-10-30T14:48:53.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907f594640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90781486a0 con 0x7f90780cb8b0 2023-10-30T14:48:53.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907cb2f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9078151840 0x7f907814e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:53.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907cb2f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9078151840 0x7f907814e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35336/0 (socket says 172.21.15.57:35336) 2023-10-30T14:48:53.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907cb2f640 1 -- 172.21.15.57:0/1273640974 learned_addr learned my addr 172.21.15.57:0/1273640974 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:53.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907db31640 1 --2- 172.21.15.57:0/1273640974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9078151c20 0x7f9078145c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:53.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.950+0000 7f907d330640 1 --2- 172.21.15.57:0/1273640974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90780cb8b0 0x7f907814a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:53.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.954+0000 7f907cb2f640 1 -- 172.21.15.57:0/1273640974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90780cb8b0 msgr2=0x7f907814a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:53.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.954+0000 7f907cb2f640 1 --2- 172.21.15.57:0/1273640974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90780cb8b0 0x7f907814a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:53.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.954+0000 7f907cb2f640 1 -- 172.21.15.57:0/1273640974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9078151c20 msgr2=0x7f9078145c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:53.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.954+0000 7f907cb2f640 1 --2- 172.21.15.57:0/1273640974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9078151c20 0x7f9078145c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:53.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.954+0000 7f907cb2f640 1 -- 172.21.15.57:0/1273640974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90740094d0 con 0x7f9078151840 2023-10-30T14:48:53.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.954+0000 7f907cb2f640 1 --2- 172.21.15.57:0/1273640974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9078151840 0x7f907814e480 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f906800b410 tx=0x7f906800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:53.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.954+0000 7f906dffb640 1 -- 172.21.15.57:0/1273640974 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9068004020 con 0x7f9078151840 2023-10-30T14:48:53.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.954+0000 7f906dffb640 1 -- 172.21.15.57:0/1273640974 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9068004180 con 0x7f9078151840 2023-10-30T14:48:53.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.954+0000 7f906dffb640 1 -- 172.21.15.57:0/1273640974 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9068011630 con 0x7f9078151840 2023-10-30T14:48:53.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.954+0000 7f907f594640 1 -- 172.21.15.57:0/1273640974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9078148980 con 0x7f9078151840 2023-10-30T14:48:53.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.954+0000 7f907f594640 1 -- 172.21.15.57:0/1273640974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9078148f40 con 0x7f9078151840 2023-10-30T14:48:53.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.954+0000 7f907f594640 1 -- 172.21.15.57:0/1273640974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f907804e170 con 0x7f9078151840 2023-10-30T14:48:53.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.954+0000 7f906dffb640 1 -- 172.21.15.57:0/1273640974 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9068011790 con 0x7f9078151840 2023-10-30T14:48:53.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.954+0000 7f906dffb640 1 --2- 172.21.15.57:0/1273640974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f905c0664d0 0x7f905c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:53.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.958+0000 7f906dffb640 1 -- 172.21.15.57:0/1273640974 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7f90680840b0 con 0x7f9078151840 2023-10-30T14:48:53.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.958+0000 7f907d330640 1 --2- 172.21.15.57:0/1273640974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f905c0664d0 0x7f905c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:53.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.958+0000 7f907d330640 1 --2- 172.21.15.57:0/1273640974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f905c0664d0 0x7f905c068990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f90740b2000 tx=0x7f9074009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:53.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:53.962+0000 7f906dffb640 1 -- 172.21.15.57:0/1273640974 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9068050fa0 con 0x7f9078151840 2023-10-30T14:48:54.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:54.150+0000 7f907f594640 1 -- 172.21.15.57:0/1273640974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9078155060 con 0x7f9078151840 2023-10-30T14:48:54.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:54.150+0000 7f906dffb640 1 -- 172.21.15.57:0/1273640974 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+5710 (secure 0 0 0) 0x7f9068015090 con 0x7f9078151840 2023-10-30T14:48:54.155 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:54.155 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:52.300180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:active","state_seq":30,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:54.157 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 183 2023-10-30T14:48:54.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:54.154+0000 7f907f594640 1 -- 172.21.15.57:0/1273640974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f905c0664d0 msgr2=0x7f905c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:54.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:54.154+0000 7f907f594640 1 --2- 172.21.15.57:0/1273640974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f905c0664d0 0x7f905c068990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f90740b2000 tx=0x7f9074009040 comp rx=0 tx=0).stop 2023-10-30T14:48:54.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:54.154+0000 7f907f594640 1 -- 172.21.15.57:0/1273640974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9078151840 msgr2=0x7f907814e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:54.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:54.154+0000 7f907f594640 1 --2- 172.21.15.57:0/1273640974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9078151840 0x7f907814e480 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f906800b410 tx=0x7f906800b8d0 comp rx=0 tx=0).stop 2023-10-30T14:48:54.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:54.154+0000 7f907f594640 1 -- 172.21.15.57:0/1273640974 shutdown_connections 2023-10-30T14:48:54.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:54.154+0000 7f907f594640 1 --2- 172.21.15.57:0/1273640974 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f905c0664d0 0x7f905c068990 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:54.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:54.154+0000 7f907f594640 1 --2- 172.21.15.57:0/1273640974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9078151c20 0x7f9078145c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:54.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:54.154+0000 7f907f594640 1 --2- 172.21.15.57:0/1273640974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9078151840 0x7f907814e480 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:54.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:54.154+0000 7f907f594640 1 --2- 172.21.15.57:0/1273640974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90780cb8b0 0x7f907814a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:54.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:54.154+0000 7f907f594640 1 -- 172.21.15.57:0/1273640974 >> 172.21.15.57:0/1273640974 conn(0x7f90780cac70 msgr2=0x7f90780733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:54.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:54.154+0000 7f907f594640 1 -- 172.21.15.57:0/1273640974 shutdown_connections 2023-10-30T14:48:54.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:54.154+0000 7f907f594640 1 -- 172.21.15.57:0/1273640974 wait complete. 2023-10-30T14:48:54.175 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=7536 2023-10-30T14:48:54.175 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.e 2023-10-30T14:48:58.302 INFO:tasks.ceph.mds.c.smithi057.stderr:2023-10-30T14:48:58.298+0000 7ff1cb4c7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T14:48:58.892 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 77% (46/59) Updating files: 79% (47/59) Updating files: 81% (48/59) Updating files: 83% (49/59) Updating files: 84% (50/59) Updating files: 86% (51/59) Updating files: 88% (52/59) Updating files: 89% (53/59) Updating files: 91% (54/59) Updating files: 93% (55/59) Updating files: 94% (56/59) Updating files: 96% (57/59) Updating files: 98% (58/59) Updating files: 100% (59/59) Updating files: 100% (59/59), done. 2023-10-30T14:48:59.176 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T14:48:59.177 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T14:48:59.177 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T14:48:59.181 INFO:tasks.ceph.mds.e:Started 2023-10-30T14:48:59.182 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:48:59.270 INFO:tasks.ceph.mds.e.smithi057.stdout:starting mds.e at 2023-10-30T14:48:59.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.326+0000 7fe63b446640 1 Processor -- start 2023-10-30T14:48:59.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.326+0000 7fe63b446640 1 -- start start 2023-10-30T14:48:59.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.326+0000 7fe63b446640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe63415cf90 0x7fe6340c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:59.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.326+0000 7fe63b446640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe634159f20 con 0x7fe634153c80 2023-10-30T14:48:59.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.326+0000 7fe63b446640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6340c47e0 con 0x7fe6340c4a10 2023-10-30T14:48:59.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.326+0000 7fe63b446640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe63415dec0 con 0x7fe63415cf90 2023-10-30T14:48:59.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.326+0000 7fe6391e2640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fe634153c80 0x7fe63415a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:33350/0 (socket says 172.21.15.57:33350) 2023-10-30T14:48:59.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.326+0000 7fe6391e2640 1 -- 172.21.15.57:0/2634995388 learned_addr learned my addr 172.21.15.57:0/2634995388 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:48:59.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.326+0000 7fe6399e3640 1 --2- 172.21.15.57:0/2634995388 >> v2:172.21.15.57:3300/0 conn(0x7fe63415cf90 0x7fe6340c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:59.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.326+0000 7fe62bfff640 1 -- 172.21.15.57:0/2634995388 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572921020 0 0) 0x7fe634159f20 con 0x7fe634153c80 2023-10-30T14:48:59.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.326+0000 7fe62bfff640 1 -- 172.21.15.57:0/2634995388 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe620003530 con 0x7fe634153c80 2023-10-30T14:48:59.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.326+0000 7fe6399e3640 1 -- 172.21.15.57:0/2634995388 >> v1:172.21.15.88:6789/0 conn(0x7fe6340c4a10 legacy=0x7fe6340ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:59.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.326+0000 7fe62bfff640 1 -- 172.21.15.57:0/2634995388 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2580071167 0 0) 0x7fe6340c47e0 con 0x7fe6340c4a10 2023-10-30T14:48:59.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.326+0000 7fe6399e3640 1 -- 172.21.15.57:0/2634995388 >> v1:172.21.15.57:6789/0 conn(0x7fe634153c80 legacy=0x7fe63415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:59.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.326+0000 7fe6399e3640 1 -- 172.21.15.57:0/2634995388 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63415d620 con 0x7fe63415cf90 2023-10-30T14:48:59.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.330+0000 7fe6399e3640 1 --2- 172.21.15.57:0/2634995388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe63415cf90 0x7fe6340c42a0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fe62c009d90 tx=0x7fe62c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b04d0a3762821c93 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:59.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.330+0000 7fe62bfff640 1 -- 172.21.15.57:0/2634995388 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62c0af600 con 0x7fe63415cf90 2023-10-30T14:48:59.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.330+0000 7fe62bfff640 1 -- 172.21.15.57:0/2634995388 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62c0ab050 con 0x7fe63415cf90 2023-10-30T14:48:59.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.330+0000 7fe62bfff640 1 -- 172.21.15.57:0/2634995388 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62c0af980 con 0x7fe63415cf90 2023-10-30T14:48:59.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.330+0000 7fe63b446640 1 -- 172.21.15.57:0/2634995388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe63415cf90 msgr2=0x7fe6340c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:59.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.330+0000 7fe63b446640 1 --2- 172.21.15.57:0/2634995388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe63415cf90 0x7fe6340c42a0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fe62c009d90 tx=0x7fe62c01f130 comp rx=0 tx=0).stop 2023-10-30T14:48:59.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.330+0000 7fe63b446640 1 -- 172.21.15.57:0/2634995388 shutdown_connections 2023-10-30T14:48:59.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.330+0000 7fe63b446640 1 --2- 172.21.15.57:0/2634995388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe63415cf90 0x7fe6340c42a0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:59.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.330+0000 7fe63b446640 1 -- 172.21.15.57:0/2634995388 >> 172.21.15.57:0/2634995388 conn(0x7fe6340715c0 msgr2=0x7fe6340719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:59.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.330+0000 7fe63b446640 1 -- 172.21.15.57:0/2634995388 shutdown_connections 2023-10-30T14:48:59.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.330+0000 7fe63b446640 1 -- 172.21.15.57:0/2634995388 wait complete. 2023-10-30T14:48:59.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe63b446640 1 Processor -- start 2023-10-30T14:48:59.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe63b446640 1 -- start start 2023-10-30T14:48:59.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe63b446640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6340c4a10 0x7fe63415c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:59.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe63b446640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe634153c80 0x7fe6341f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:59.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe63b446640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe63415cf90 0x7fe6341f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:59.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe63b446640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe634149940 con 0x7fe63415cf90 2023-10-30T14:48:59.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe63b446640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe634149160 con 0x7fe634153c80 2023-10-30T14:48:59.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe63b446640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe634157f70 con 0x7fe6340c4a10 2023-10-30T14:48:59.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe6399e3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe63415cf90 0x7fe6341f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:59.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe6399e3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe63415cf90 0x7fe6341f2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35372/0 (socket says 172.21.15.57:35372) 2023-10-30T14:48:59.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe6399e3640 1 -- 172.21.15.57:0/139377551 learned_addr learned my addr 172.21.15.57:0/139377551 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:48:59.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe6391e2640 1 --2- 172.21.15.57:0/139377551 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6340c4a10 0x7fe63415c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:59.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe6389e1640 1 --2- 172.21.15.57:0/139377551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe634153c80 0x7fe6341f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:59.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe6399e3640 1 -- 172.21.15.57:0/139377551 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6340c4a10 msgr2=0x7fe63415c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:59.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe6399e3640 1 --2- 172.21.15.57:0/139377551 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6340c4a10 0x7fe63415c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:59.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe6399e3640 1 -- 172.21.15.57:0/139377551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe634153c80 msgr2=0x7fe6341f24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:59.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe6399e3640 1 --2- 172.21.15.57:0/139377551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe634153c80 0x7fe6341f24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:59.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe6399e3640 1 -- 172.21.15.57:0/139377551 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe62c009a70 con 0x7fe63415cf90 2023-10-30T14:48:59.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe6391e2640 1 --2- 172.21.15.57:0/139377551 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6340c4a10 0x7fe63415c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:48:59.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe6389e1640 1 --2- 172.21.15.57:0/139377551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe634153c80 0x7fe6341f24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:48:59.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe6399e3640 1 --2- 172.21.15.57:0/139377551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe63415cf90 0x7fe6341f2a10 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fe62c009e80 tx=0x7fe62c01bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:59.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe629ffb640 1 -- 172.21.15.57:0/139377551 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62c009450 con 0x7fe63415cf90 2023-10-30T14:48:59.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe629ffb640 1 -- 172.21.15.57:0/139377551 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62c0039a0 con 0x7fe63415cf90 2023-10-30T14:48:59.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe629ffb640 1 -- 172.21.15.57:0/139377551 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62c003d10 con 0x7fe63415cf90 2023-10-30T14:48:59.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.334+0000 7fe63b446640 1 -- 172.21.15.57:0/139377551 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe634149380 con 0x7fe63415cf90 2023-10-30T14:48:59.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.338+0000 7fe63b446640 1 -- 172.21.15.57:0/139377551 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6341579b0 con 0x7fe63415cf90 2023-10-30T14:48:59.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.338+0000 7fe63b446640 1 -- 172.21.15.57:0/139377551 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe634157810 con 0x7fe63415cf90 2023-10-30T14:48:59.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.338+0000 7fe629ffb640 1 -- 172.21.15.57:0/139377551 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe62c0b6040 con 0x7fe63415cf90 2023-10-30T14:48:59.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.338+0000 7fe629ffb640 1 --2- 172.21.15.57:0/139377551 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe6180664a0 0x7fe618068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:48:59.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.338+0000 7fe629ffb640 1 -- 172.21.15.57:0/139377551 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7fe62c1220f0 con 0x7fe63415cf90 2023-10-30T14:48:59.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.342+0000 7fe6391e2640 1 --2- 172.21.15.57:0/139377551 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe6180664a0 0x7fe618068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:48:59.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.342+0000 7fe6391e2640 1 --2- 172.21.15.57:0/139377551 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe6180664a0 0x7fe618068960 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe630005890 tx=0x7fe630009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:48:59.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.342+0000 7fe629ffb640 1 -- 172.21.15.57:0/139377551 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe634157810 con 0x7fe63415cf90 2023-10-30T14:48:59.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.538+0000 7fe63b446640 1 -- 172.21.15.57:0/139377551 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe634157810 con 0x7fe63415cf90 2023-10-30T14:48:59.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.538+0000 7fe629ffb640 1 -- 172.21.15.57:0/139377551 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+5710 (secure 0 0 0) 0x7fe634157810 con 0x7fe63415cf90 2023-10-30T14:48:59.541 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:48:59.541 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:52.300180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:active","state_seq":30,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:48:59.542 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 183 2023-10-30T14:48:59.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.538+0000 7fe63b446640 1 -- 172.21.15.57:0/139377551 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe6180664a0 msgr2=0x7fe618068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:59.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.538+0000 7fe63b446640 1 --2- 172.21.15.57:0/139377551 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe6180664a0 0x7fe618068960 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe630005890 tx=0x7fe630009040 comp rx=0 tx=0).stop 2023-10-30T14:48:59.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.538+0000 7fe63b446640 1 -- 172.21.15.57:0/139377551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe63415cf90 msgr2=0x7fe6341f2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:48:59.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.538+0000 7fe63b446640 1 --2- 172.21.15.57:0/139377551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe63415cf90 0x7fe6341f2a10 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fe62c009e80 tx=0x7fe62c01bed0 comp rx=0 tx=0).stop 2023-10-30T14:48:59.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.542+0000 7fe63b446640 1 -- 172.21.15.57:0/139377551 shutdown_connections 2023-10-30T14:48:59.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.542+0000 7fe63b446640 1 --2- 172.21.15.57:0/139377551 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe6180664a0 0x7fe618068960 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:59.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.542+0000 7fe63b446640 1 --2- 172.21.15.57:0/139377551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe63415cf90 0x7fe6341f2a10 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:59.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.542+0000 7fe63b446640 1 --2- 172.21.15.57:0/139377551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe634153c80 0x7fe6341f24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:59.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.542+0000 7fe63b446640 1 --2- 172.21.15.57:0/139377551 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6340c4a10 0x7fe63415c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:48:59.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.542+0000 7fe63b446640 1 -- 172.21.15.57:0/139377551 >> 172.21.15.57:0/139377551 conn(0x7fe6340715c0 msgr2=0x7fe6340719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:48:59.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.542+0000 7fe63b446640 1 -- 172.21.15.57:0/139377551 shutdown_connections 2023-10-30T14:48:59.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:48:59.542+0000 7fe63b446640 1 -- 172.21.15.57:0/139377551 wait complete. 2023-10-30T14:48:59.561 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-30T14:49:00.241 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 9382379cc6b v0.62 2023-10-30T14:49:00.243 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.62 2023-10-30T14:49:00.258 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 63 -eq 48 ] 2023-10-30T14:49:00.258 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.63 2023-10-30T14:49:00.259 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.63 2023-10-30T14:49:00.259 INFO:tasks.workunit.client.0.smithi057.stdout:v0.63 2023-10-30T14:49:00.259 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.63 2023-10-30T14:49:01.563 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:49:01.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.710+0000 7f2c20a9e640 1 Processor -- start 2023-10-30T14:49:01.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.710+0000 7f2c20a9e640 1 -- start start 2023-10-30T14:49:01.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.710+0000 7f2c20a9e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2c1c153c80 0x7f2c1c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:49:01.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.710+0000 7f2c20a9e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2c1c0c4a10 0x7f2c1c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:49:01.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.710+0000 7f2c20a9e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c1c15a3b0 0x7f2c1c0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:49:01.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.710+0000 7f2c20a9e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c1c15dec0 con 0x7f2c1c153c80 2023-10-30T14:49:01.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.710+0000 7f2c20a9e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c1c1599a0 con 0x7f2c1c0c4a10 2023-10-30T14:49:01.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.710+0000 7f2c20a9e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c159f20 con 0x7f2c1c15a3b0 2023-10-30T14:49:01.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.710+0000 7f2c1a59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2c1c153c80 0x7f2c1c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:49:01.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.710+0000 7f2c1a59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2c1c153c80 0x7f2c1c15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35380/0 (socket says 172.21.15.57:35380) 2023-10-30T14:49:01.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.710+0000 7f2c1a59c640 1 -- 172.21.15.57:0/4175175809 learned_addr learned my addr 172.21.15.57:0/4175175809 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:49:01.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.710+0000 7f2c19d9b640 1 --2- 172.21.15.57:0/4175175809 >> v2:172.21.15.88:3300/0 conn(0x7f2c1c0c4a10 0x7f2c1c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:49:01.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.710+0000 7f2c1ad9d640 1 --2- 172.21.15.57:0/4175175809 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c1c15a3b0 0x7f2c1c0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:49:01.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.710+0000 7f2c1a59c640 1 -- 172.21.15.57:0/4175175809 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c1c15a3b0 msgr2=0x7f2c1c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:49:01.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.710+0000 7f2c1a59c640 1 --2- 172.21.15.57:0/4175175809 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c1c15a3b0 0x7f2c1c0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:49:01.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.710+0000 7f2c1a59c640 1 -- 172.21.15.57:0/4175175809 >> v2:172.21.15.88:3300/0 conn(0x7f2c1c0c4a10 msgr2=0x7f2c1c0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:49:01.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.710+0000 7f2c1a59c640 1 --2- 172.21.15.57:0/4175175809 >> v2:172.21.15.88:3300/0 conn(0x7f2c1c0c4a10 0x7f2c1c0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:49:01.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.710+0000 7f2c1a59c640 1 -- 172.21.15.57:0/4175175809 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1c0c47e0 con 0x7f2c1c153c80 2023-10-30T14:49:01.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.714+0000 7f2c1a59c640 1 --2- 172.21.15.57:0/4175175809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c1c153c80 0x7f2c1c15cf90 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c06be50 tx=0x7f2c0c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac4905192af3f6b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:49:01.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.714+0000 7f2c1959a640 1 -- 172.21.15.57:0/4175175809 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c0a89b0 con 0x7f2c1c153c80 2023-10-30T14:49:01.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.714+0000 7f2c1959a640 1 -- 172.21.15.57:0/4175175809 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0c0a8b10 con 0x7f2c1c153c80 2023-10-30T14:49:01.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.714+0000 7f2c1959a640 1 -- 172.21.15.57:0/4175175809 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c0a8ea0 con 0x7f2c1c153c80 2023-10-30T14:49:01.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.714+0000 7f2c20a9e640 1 -- 172.21.15.57:0/4175175809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c1c153c80 msgr2=0x7f2c1c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:49:01.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.714+0000 7f2c20a9e640 1 --2- 172.21.15.57:0/4175175809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c1c153c80 0x7f2c1c15cf90 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c06be50 tx=0x7f2c0c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:49:01.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.714+0000 7f2c20a9e640 1 -- 172.21.15.57:0/4175175809 shutdown_connections 2023-10-30T14:49:01.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.714+0000 7f2c20a9e640 1 --2- 172.21.15.57:0/4175175809 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c1c15a3b0 0x7f2c1c0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:49:01.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.714+0000 7f2c20a9e640 1 --2- 172.21.15.57:0/4175175809 >> v2:172.21.15.88:3300/0 conn(0x7f2c1c0c4a10 0x7f2c1c0c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:49:01.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.714+0000 7f2c20a9e640 1 --2- 172.21.15.57:0/4175175809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c1c153c80 0x7f2c1c15cf90 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:49:01.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.714+0000 7f2c20a9e640 1 -- 172.21.15.57:0/4175175809 >> 172.21.15.57:0/4175175809 conn(0x7f2c1c0715c0 msgr2=0x7f2c1c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:49:01.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.714+0000 7f2c20a9e640 1 -- 172.21.15.57:0/4175175809 shutdown_connections 2023-10-30T14:49:01.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.714+0000 7f2c20a9e640 1 -- 172.21.15.57:0/4175175809 wait complete. 2023-10-30T14:49:01.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c20a9e640 1 Processor -- start 2023-10-30T14:49:01.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c20a9e640 1 -- start start 2023-10-30T14:49:01.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c20a9e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c1c0c4a10 0x7f2c1c0c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:49:01.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c20a9e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c1c153c80 0x7f2c1c151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:49:01.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c20a9e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c1c15a3b0 0x7f2c1c1f24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:49:01.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c20a9e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0c9110 con 0x7f2c1c0c4a10 2023-10-30T14:49:01.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c20a9e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0c93d0 con 0x7f2c1c153c80 2023-10-30T14:49:01.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c20a9e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c15b1b0 con 0x7f2c1c15a3b0 2023-10-30T14:49:01.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c1a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c1c0c4a10 0x7f2c1c0c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:49:01.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c1a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c1c0c4a10 0x7f2c1c0c8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35392/0 (socket says 172.21.15.57:35392) 2023-10-30T14:49:01.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c1a59c640 1 -- 172.21.15.57:0/3177288250 learned_addr learned my addr 172.21.15.57:0/3177288250 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:49:01.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c19d9b640 1 --2- 172.21.15.57:0/3177288250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c1c153c80 0x7f2c1c151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:49:01.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c1ad9d640 1 --2- 172.21.15.57:0/3177288250 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c1c15a3b0 0x7f2c1c1f24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:49:01.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c1a59c640 1 -- 172.21.15.57:0/3177288250 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c1c15a3b0 msgr2=0x7f2c1c1f24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:49:01.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c1a59c640 1 --2- 172.21.15.57:0/3177288250 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c1c15a3b0 0x7f2c1c1f24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:49:01.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c1a59c640 1 -- 172.21.15.57:0/3177288250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c1c153c80 msgr2=0x7f2c1c151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:49:01.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c1a59c640 1 --2- 172.21.15.57:0/3177288250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c1c153c80 0x7f2c1c151e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:49:01.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c1a59c640 1 -- 172.21.15.57:0/3177288250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0c0094d0 con 0x7f2c1c0c4a10 2023-10-30T14:49:01.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c1a59c640 1 --2- 172.21.15.57:0/3177288250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c1c0c4a10 0x7f2c1c0c8af0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c01d3a0 tx=0x7f2c0c01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:49:01.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c0affd640 1 -- 172.21.15.57:0/3177288250 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c0b0d40 con 0x7f2c1c0c4a10 2023-10-30T14:49:01.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c0affd640 1 -- 172.21.15.57:0/3177288250 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0c0bb460 con 0x7f2c1c0c4a10 2023-10-30T14:49:01.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c0affd640 1 -- 172.21.15.57:0/3177288250 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c0b0ea0 con 0x7f2c1c0c4a10 2023-10-30T14:49:01.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c20a9e640 1 -- 172.21.15.57:0/3177288250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1c15b490 con 0x7f2c1c0c4a10 2023-10-30T14:49:01.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c20a9e640 1 -- 172.21.15.57:0/3177288250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c1c15bc90 con 0x7f2c1c0c4a10 2023-10-30T14:49:01.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.718+0000 7f2c20a9e640 1 -- 172.21.15.57:0/3177288250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c1c15b010 con 0x7f2c1c0c4a10 2023-10-30T14:49:01.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.722+0000 7f2c0affd640 1 -- 172.21.15.57:0/3177288250 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2c0c0b83f0 con 0x7f2c1c0c4a10 2023-10-30T14:49:01.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.722+0000 7f2c0affd640 1 --2- 172.21.15.57:0/3177288250 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2bfc0664a0 0x7f2bfc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:49:01.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.722+0000 7f2c0affd640 1 -- 172.21.15.57:0/3177288250 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7f2c0c121dc0 con 0x7f2c1c0c4a10 2023-10-30T14:49:01.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.722+0000 7f2c19d9b640 1 --2- 172.21.15.57:0/3177288250 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2bfc0664a0 0x7f2bfc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:49:01.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.722+0000 7f2c19d9b640 1 --2- 172.21.15.57:0/3177288250 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2bfc0664a0 0x7f2bfc068960 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c158310 tx=0x7f2c10009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:49:01.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.726+0000 7f2c0affd640 1 -- 172.21.15.57:0/3177288250 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2c1c15b010 con 0x7f2c1c0c4a10 2023-10-30T14:49:01.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.922+0000 7f2c20a9e640 1 -- 172.21.15.57:0/3177288250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c1c15b010 con 0x7f2c1c0c4a10 2023-10-30T14:49:01.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.922+0000 7f2c0affd640 1 -- 172.21.15.57:0/3177288250 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+6502 (secure 0 0 0) 0x7f2c1c15b010 con 0x7f2c1c0c4a10 2023-10-30T14:49:01.925 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:49:01.926 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":8058,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:52.300180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:active","state_seq":30,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:49:01.926 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 184 2023-10-30T14:49:01.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.926+0000 7f2c20a9e640 1 -- 172.21.15.57:0/3177288250 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2bfc0664a0 msgr2=0x7f2bfc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:49:01.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.926+0000 7f2c20a9e640 1 --2- 172.21.15.57:0/3177288250 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2bfc0664a0 0x7f2bfc068960 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c158310 tx=0x7f2c10009190 comp rx=0 tx=0).stop 2023-10-30T14:49:01.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.926+0000 7f2c20a9e640 1 -- 172.21.15.57:0/3177288250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c1c0c4a10 msgr2=0x7f2c1c0c8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:49:01.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.926+0000 7f2c20a9e640 1 --2- 172.21.15.57:0/3177288250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c1c0c4a10 0x7f2c1c0c8af0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c01d3a0 tx=0x7f2c0c01ba90 comp rx=0 tx=0).stop 2023-10-30T14:49:01.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.926+0000 7f2c20a9e640 1 -- 172.21.15.57:0/3177288250 shutdown_connections 2023-10-30T14:49:01.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.926+0000 7f2c20a9e640 1 --2- 172.21.15.57:0/3177288250 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f2bfc0664a0 0x7f2bfc068960 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:49:01.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.926+0000 7f2c20a9e640 1 --2- 172.21.15.57:0/3177288250 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c1c15a3b0 0x7f2c1c1f24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:49:01.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.926+0000 7f2c20a9e640 1 --2- 172.21.15.57:0/3177288250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c1c153c80 0x7f2c1c151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:49:01.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.926+0000 7f2c20a9e640 1 --2- 172.21.15.57:0/3177288250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c1c0c4a10 0x7f2c1c0c8af0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:49:01.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.926+0000 7f2c20a9e640 1 -- 172.21.15.57:0/3177288250 >> 172.21.15.57:0/3177288250 conn(0x7f2c1c0715c0 msgr2=0x7f2c1c06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:49:01.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.926+0000 7f2c20a9e640 1 -- 172.21.15.57:0/3177288250 shutdown_connections 2023-10-30T14:49:01.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:49:01.926+0000 7f2c20a9e640 1 -- 172.21.15.57:0/3177288250 wait complete. 2023-10-30T14:49:01.951 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T14:49:01.952 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 91 secs before thrashing 2023-10-30T14:49:06.552 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 65% (95/146) Updating files: 66% (97/146) Updating files: 67% (98/146) Updating files: 68% (100/146) Updating files: 69% (101/146) Updating files: 70% (103/146) Updating files: 71% (104/146) Updating files: 72% (106/146) Updating files: 73% (107/146) Updating files: 74% (109/146) Updating files: 75% (110/146) Updating files: 76% (111/146) Updating files: 77% (113/146) Updating files: 78% (114/146) Updating files: 79% (116/146) Updating files: 80% (117/146) Updating files: 81% (119/146) Updating files: 82% (120/146) Updating files: 83% (122/146) Updating files: 84% (123/146) Updating files: 85% (125/146) Updating files: 86% (126/146) Updating files: 87% (128/146) Updating files: 88% (129/146) Updating files: 89% (130/146) Updating files: 90% (132/146) Updating files: 91% (133/146) Updating files: 92% (135/146) Updating files: 93% (136/146) Updating files: 94% (138/146) Updating files: 95% (139/146) Updating files: 96% (141/146) Updating files: 97% (142/146) Updating files: 98% (144/146) Updating files: 99% (145/146) Updating files: 100% (146/146) Updating files: 100% (146/146), done. 2023-10-30T14:49:06.594 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 054e96cf79e v0.63 2023-10-30T14:49:06.601 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.63 2023-10-30T14:49:06.654 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 64 -eq 48 ] 2023-10-30T14:49:06.654 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.64 2023-10-30T14:49:06.654 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.64 2023-10-30T14:49:06.655 INFO:tasks.workunit.client.0.smithi057.stdout:v0.64 2023-10-30T14:49:06.655 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.64 2023-10-30T14:49:11.695 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:49:11.699 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:49:12.839 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 42% (61/143) Updating files: 43% (62/143) Updating files: 44% (63/143) Updating files: 45% (65/143) Updating files: 46% (66/143) Updating files: 47% (68/143) Updating files: 48% (69/143) Updating files: 49% (71/143) Updating files: 50% (72/143) Updating files: 51% (73/143) Updating files: 52% (75/143) Updating files: 53% (76/143) Updating files: 54% (78/143) Updating files: 55% (79/143) Updating files: 56% (81/143) Updating files: 57% (82/143) Updating files: 58% (83/143) Updating files: 59% (85/143) Updating files: 60% (86/143) Updating files: 61% (88/143) Updating files: 62% (89/143) Updating files: 63% (91/143) Updating files: 64% (92/143) Updating files: 65% (93/143) Updating files: 66% (95/143) Updating files: 67% (96/143) Updating files: 68% (98/143) Updating files: 69% (99/143) Updating files: 70% (101/143) Updating files: 71% (102/143) Updating files: 72% (103/143) Updating files: 73% (105/143) Updating files: 74% (106/143) Updating files: 75% (108/143) Updating files: 76% (109/143) Updating files: 77% (111/143) Updating files: 78% (112/143) Updating files: 79% (113/143) Updating files: 80% (115/143) Updating files: 81% (116/143) Updating files: 82% (118/143) Updating files: 83% (119/143) Updating files: 84% (121/143) Updating files: 85% (122/143) Updating files: 86% (123/143) Updating files: 87% (125/143) Updating files: 88% (126/143) Updating files: 89% (128/143) Updating files: 90% (129/143) Updating files: 91% (131/143) Updating files: 92% (132/143) Updating files: 93% (133/143) Updating files: 94% (135/143) Updating files: 95% (136/143) Updating files: 96% (138/143) Updating files: 97% (139/143) Updating files: 98% (141/143) Updating files: 99% (142/143) Updating files: 100% (143/143) Updating files: 100% (143/143), done. 2023-10-30T14:49:12.870 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 42e06c12db6 v0.64 2023-10-30T14:49:12.876 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.64 2023-10-30T14:49:12.913 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 65 -eq 48 ] 2023-10-30T14:49:12.914 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.65 2023-10-30T14:49:12.914 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.65 2023-10-30T14:49:12.914 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.65 2023-10-30T14:49:12.914 INFO:tasks.workunit.client.0.smithi057.stdout:v0.65 2023-10-30T14:49:17.477 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 21% (43/203) Updating files: 22% (45/203) Updating files: 23% (47/203) Updating files: 24% (49/203) Updating files: 25% (51/203) Updating files: 26% (53/203) Updating files: 27% (55/203) Updating files: 28% (57/203) Updating files: 29% (59/203) Updating files: 30% (61/203) Updating files: 31% (63/203) Updating files: 32% (65/203) Updating files: 33% (67/203) Updating files: 34% (70/203) Updating files: 35% (72/203) Updating files: 36% (74/203) Updating files: 37% (76/203) Updating files: 38% (78/203) Updating files: 39% (80/203) Updating files: 40% (82/203) Updating files: 41% (84/203) Updating files: 42% (86/203) Updating files: 43% (88/203) Updating files: 44% (90/203) Updating files: 45% (92/203) Updating files: 46% (94/203) Updating files: 47% (96/203) Updating files: 48% (98/203) Updating files: 49% (100/203) Updating files: 50% (102/203) Updating files: 51% (104/203) Updating files: 52% (106/203) Updating files: 53% (108/203) Updating files: 54% (110/203) Updating files: 55% (112/203) Updating files: 56% (114/203) Updating files: 57% (116/203) Updating files: 58% (118/203) Updating files: 59% (120/203) Updating files: 60% (122/203) Updating files: 61% (124/203) Updating files: 62% (126/203) Updating files: 63% (128/203) Updating files: 64% (130/203) Updating files: 65% (132/203) Updating files: 66% (134/203) Updating files: 67% (137/203) Updating files: 67% (138/203) Updating files: 68% (139/203) Updating files: 69% (141/203) Updating files: 70% (143/203) Updating files: 71% (145/203) Updating files: 72% (147/203) Updating files: 73% (149/203) Updating files: 74% (151/203) Updating files: 75% (153/203) Updating files: 76% (155/203) Updating files: 77% (157/203) Updating files: 78% (159/203) Updating files: 79% (161/203) Updating files: 80% (163/203) Updating files: 81% (165/203) Updating files: 82% (167/203) Updating files: 83% (169/203) Updating files: 84% (171/203) Updating files: 85% (173/203) Updating files: 86% (175/203) Updating files: 87% (177/203) Updating files: 88% (179/203) Updating files: 89% (181/203) Updating files: 90% (183/203) Updating files: 91% (185/203) Updating files: 92% (187/203) Updating files: 93% (189/203) Updating files: 94% (191/203) Updating files: 95% (193/203) Updating files: 96% (195/203) Updating files: 97% (197/203) Updating files: 98% (199/203) Updating files: 99% (201/203) Updating files: 99% (202/203) Updating files: 100% (203/203) Updating files: 100% (203/203), done. 2023-10-30T14:49:17.529 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at c2d517ef966 v0.65 2023-10-30T14:49:17.538 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.65 2023-10-30T14:49:17.765 INFO:tasks.workunit.client.0.smithi057.stdout:v0.66 2023-10-30T14:49:17.766 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 66 -eq 48 ] 2023-10-30T14:49:17.766 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.66 2023-10-30T14:49:17.766 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.66 2023-10-30T14:49:17.766 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.66 2023-10-30T14:49:37.760 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 63% (96/152) Updating files: 64% (98/152) Updating files: 65% (99/152) Updating files: 66% (101/152) Updating files: 67% (102/152) Updating files: 68% (104/152) Updating files: 69% (105/152) Updating files: 70% (107/152) Updating files: 71% (108/152) Updating files: 72% (110/152) Updating files: 73% (111/152) Updating files: 74% (113/152) Updating files: 75% (114/152) Updating files: 76% (116/152) Updating files: 77% (118/152) Updating files: 78% (119/152) Updating files: 79% (121/152) Updating files: 80% (122/152) Updating files: 81% (124/152) Updating files: 82% (125/152) Updating files: 83% (127/152) Updating files: 84% (128/152) Updating files: 85% (130/152) Updating files: 86% (131/152) Updating files: 87% (133/152) Updating files: 88% (134/152) Updating files: 89% (136/152) Updating files: 90% (137/152) Updating files: 91% (139/152) Updating files: 92% (140/152) Updating files: 93% (142/152) Updating files: 94% (143/152) Updating files: 95% (145/152) Updating files: 96% (146/152) Updating files: 97% (148/152) Updating files: 98% (149/152) Updating files: 99% (151/152) Updating files: 100% (152/152) Updating files: 100% (152/152), done. 2023-10-30T14:49:37.806 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at b6b48dbefad v0.66 2023-10-30T14:49:37.812 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.66 2023-10-30T14:49:38.690 INFO:tasks.workunit.client.0.smithi057.stdout:v0.67 2023-10-30T14:49:38.691 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 67 -eq 48 ] 2023-10-30T14:49:38.691 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.67 2023-10-30T14:49:38.691 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.67 2023-10-30T14:49:38.691 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.67 2023-10-30T14:49:41.782 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:49:41.786 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:49:46.417 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 13% (49/355) Updating files: 14% (50/355) Updating files: 15% (54/355) Updating files: 16% (57/355) Updating files: 17% (61/355) Updating files: 18% (64/355) Updating files: 19% (68/355) Updating files: 20% (71/355) Updating files: 21% (75/355) Updating files: 22% (79/355) Updating files: 23% (82/355) Updating files: 24% (86/355) Updating files: 25% (89/355) Updating files: 26% (93/355) Updating files: 27% (96/355) Updating files: 28% (100/355) Updating files: 29% (103/355) Updating files: 30% (107/355) Updating files: 31% (111/355) Updating files: 32% (114/355) Updating files: 33% (118/355) Updating files: 34% (121/355) Updating files: 35% (125/355) Updating files: 36% (128/355) Updating files: 37% (132/355) Updating files: 38% (135/355) Updating files: 39% (139/355) Updating files: 40% (142/355) Updating files: 41% (146/355) Updating files: 41% (147/355) Updating files: 42% (150/355) Updating files: 43% (153/355) Updating files: 44% (157/355) Updating files: 45% (160/355) Updating files: 46% (164/355) Updating files: 47% (167/355) Updating files: 48% (171/355) Updating files: 49% (174/355) Updating files: 50% (178/355) Updating files: 51% (182/355) Updating files: 51% (184/355) Updating files: 52% (185/355) Updating files: 53% (189/355) Updating files: 54% (192/355) Updating files: 55% (196/355) Updating files: 56% (199/355) Updating files: 57% (203/355) Updating files: 58% (206/355) Updating files: 59% (210/355) Updating files: 60% (213/355) Updating files: 61% (217/355) Updating files: 62% (221/355) Updating files: 63% (224/355) Updating files: 64% (228/355) Updating files: 65% (231/355) Updating files: 66% (235/355) Updating files: 67% (238/355) Updating files: 68% (242/355) Updating files: 69% (245/355) Updating files: 70% (249/355) Updating files: 71% (253/355) Updating files: 72% (256/355) Updating files: 73% (260/355) Updating files: 74% (263/355) Updating files: 75% (267/355) Updating files: 75% (268/355) Updating files: 76% (270/355) Updating files: 77% (274/355) Updating files: 78% (277/355) Updating files: 79% (281/355) Updating files: 80% (284/355) Updating files: 81% (288/355) Updating files: 82% (292/355) Updating files: 83% (295/355) Updating files: 84% (299/355) Updating files: 85% (302/355) Updating files: 86% (306/355) Updating files: 87% (309/355) Updating files: 88% (313/355) Updating files: 89% (316/355) Updating files: 90% (320/355) Updating files: 91% (324/355) Updating files: 92% (327/355) Updating files: 93% (331/355) Updating files: 94% (334/355) Updating files: 95% (338/355) Updating files: 96% (341/355) Updating files: 97% (345/355) Updating files: 98% (348/355) Updating files: 99% (352/355) Updating files: 100% (355/355) Updating files: 100% (355/355), done. 2023-10-30T14:49:46.468 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at e3b7bc5bce8 v0.67 2023-10-30T14:49:46.475 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.67 2023-10-30T14:49:46.786 INFO:tasks.workunit.client.0.smithi057.stdout:v0.68 2023-10-30T14:49:46.786 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 68 -eq 48 ] 2023-10-30T14:49:46.786 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.68 2023-10-30T14:49:46.787 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.68 2023-10-30T14:49:46.787 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.68 2023-10-30T14:49:50.858 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 2% (7/241) Updating files: 3% (8/241) Updating files: 4% (10/241) Updating files: 5% (13/241) Updating files: 6% (15/241) Updating files: 7% (17/241) Updating files: 8% (20/241) Updating files: 9% (22/241) Updating files: 10% (25/241) Updating files: 11% (27/241) Updating files: 12% (29/241) Updating files: 13% (32/241) Updating files: 14% (34/241) Updating files: 15% (37/241) Updating files: 16% (39/241) Updating files: 17% (41/241) Updating files: 18% (44/241) Updating files: 19% (46/241) Updating files: 20% (49/241) Updating files: 21% (51/241) Updating files: 22% (54/241) Updating files: 23% (56/241) Updating files: 24% (58/241) Updating files: 25% (61/241) Updating files: 26% (63/241) Updating files: 27% (66/241) Updating files: 28% (68/241) Updating files: 29% (70/241) Updating files: 30% (73/241) Updating files: 31% (75/241) Updating files: 32% (78/241) Updating files: 33% (80/241) Updating files: 34% (82/241) Updating files: 35% (85/241) Updating files: 36% (87/241) Updating files: 37% (90/241) Updating files: 38% (92/241) Updating files: 39% (94/241) Updating files: 40% (97/241) Updating files: 41% (99/241) Updating files: 42% (102/241) Updating files: 43% (104/241) Updating files: 44% (107/241) Updating files: 45% (109/241) Updating files: 46% (111/241) Updating files: 47% (114/241) Updating files: 48% (116/241) Updating files: 49% (119/241) Updating files: 50% (121/241) Updating files: 51% (123/241) Updating files: 52% (126/241) Updating files: 53% (128/241) Updating files: 54% (131/241) Updating files: 55% (133/241) Updating files: 56% (135/241) Updating files: 57% (138/241) Updating files: 58% (140/241) Updating files: 59% (143/241) Updating files: 60% (145/241) Updating files: 61% (148/241) Updating files: 62% (150/241) Updating files: 63% (152/241) Updating files: 64% (155/241) Updating files: 65% (157/241) Updating files: 66% (160/241) Updating files: 67% (162/241) Updating files: 68% (164/241) Updating files: 69% (167/241) Updating files: 70% (169/241) Updating files: 70% (170/241) Updating files: 71% (172/241) Updating files: 72% (174/241) Updating files: 73% (176/241) Updating files: 74% (179/241) Updating files: 75% (181/241) Updating files: 76% (184/241) Updating files: 77% (186/241) Updating files: 78% (188/241) Updating files: 79% (191/241) Updating files: 80% (193/241) Updating files: 81% (196/241) Updating files: 82% (198/241) Updating files: 83% (201/241) Updating files: 84% (203/241) Updating files: 85% (205/241) Updating files: 86% (208/241) Updating files: 87% (210/241) Updating files: 87% (211/241) Updating files: 88% (213/241) Updating files: 89% (215/241) Updating files: 90% (217/241) Updating files: 91% (220/241) Updating files: 92% (222/241) Updating files: 93% (225/241) Updating files: 94% (227/241) Updating files: 95% (229/241) Updating files: 96% (232/241) Updating files: 97% (234/241) Updating files: 98% (237/241) Updating files: 99% (239/241) Updating files: 100% (241/241) Updating files: 100% (241/241), done. 2023-10-30T14:49:50.900 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at b4cf0f2574b v0.68 2023-10-30T14:49:50.907 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.68 2023-10-30T14:49:51.001 INFO:tasks.workunit.client.0.smithi057.stdout:v0.69 2023-10-30T14:49:51.001 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 69 -eq 48 ] 2023-10-30T14:49:51.001 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.69 2023-10-30T14:49:51.002 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.69 2023-10-30T14:49:51.002 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.69 2023-10-30T14:50:04.118 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 14% (23/155) Updating files: 15% (24/155) Updating files: 16% (25/155) Updating files: 17% (27/155) Updating files: 18% (28/155) Updating files: 19% (30/155) Updating files: 20% (31/155) Updating files: 21% (33/155) Updating files: 22% (35/155) Updating files: 23% (36/155) Updating files: 24% (38/155) Updating files: 25% (39/155) Updating files: 26% (41/155) Updating files: 27% (42/155) Updating files: 28% (44/155) Updating files: 29% (45/155) Updating files: 30% (47/155) Updating files: 31% (49/155) Updating files: 32% (50/155) Updating files: 33% (52/155) Updating files: 34% (53/155) Updating files: 35% (55/155) Updating files: 36% (56/155) Updating files: 37% (58/155) Updating files: 38% (59/155) Updating files: 39% (61/155) Updating files: 40% (62/155) Updating files: 41% (64/155) Updating files: 42% (66/155) Updating files: 43% (67/155) Updating files: 44% (69/155) Updating files: 45% (70/155) Updating files: 46% (72/155) Updating files: 47% (73/155) Updating files: 48% (75/155) Updating files: 49% (76/155) Updating files: 50% (78/155) Updating files: 51% (80/155) Updating files: 52% (81/155) Updating files: 53% (83/155) Updating files: 54% (84/155) Updating files: 55% (86/155) Updating files: 56% (87/155) Updating files: 57% (89/155) Updating files: 58% (90/155) Updating files: 59% (92/155) Updating files: 60% (93/155) Updating files: 61% (95/155) Updating files: 61% (96/155) Updating files: 62% (97/155) Updating files: 63% (98/155) Updating files: 64% (100/155) Updating files: 65% (101/155) Updating files: 66% (103/155) Updating files: 67% (104/155) Updating files: 68% (106/155) Updating files: 69% (107/155) Updating files: 70% (109/155) Updating files: 71% (111/155) Updating files: 72% (112/155) Updating files: 73% (114/155) Updating files: 74% (115/155) Updating files: 75% (117/155) Updating files: 76% (118/155) Updating files: 77% (120/155) Updating files: 78% (121/155) Updating files: 79% (123/155) Updating files: 80% (124/155) Updating files: 81% (126/155) Updating files: 82% (128/155) Updating files: 83% (129/155) Updating files: 84% (131/155) Updating files: 85% (132/155) Updating files: 86% (134/155) Updating files: 87% (135/155) Updating files: 88% (137/155) Updating files: 89% (138/155) Updating files: 90% (140/155) Updating files: 91% (142/155) Updating files: 92% (143/155) Updating files: 93% (145/155) Updating files: 94% (146/155) Updating files: 95% (148/155) Updating files: 96% (149/155) Updating files: 97% (151/155) Updating files: 98% (152/155) Updating files: 99% (154/155) Updating files: 100% (155/155) Updating files: 100% (155/155), done. 2023-10-30T14:50:04.166 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 6ca6f2f9f75 v0.69 2023-10-30T14:50:04.172 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.69 2023-10-30T14:50:04.273 INFO:tasks.workunit.client.0.smithi057.stdout:v0.70 2023-10-30T14:50:04.273 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 70 -eq 48 ] 2023-10-30T14:50:04.273 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.70 2023-10-30T14:50:04.274 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.70 2023-10-30T14:50:04.274 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.70 2023-10-30T14:50:05.892 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at e3bb0656d92 v0.70 2023-10-30T14:50:05.894 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.70 2023-10-30T14:50:06.341 INFO:tasks.workunit.client.0.smithi057.stdout:v0.71 2023-10-30T14:50:06.342 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 71 -eq 48 ] 2023-10-30T14:50:06.342 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.71 2023-10-30T14:50:06.342 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.71 2023-10-30T14:50:06.342 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.71 2023-10-30T14:50:11.523 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 6% (25/405) Updating files: 7% (29/405) Updating files: 8% (33/405) Updating files: 9% (37/405) Updating files: 10% (41/405) Updating files: 11% (45/405) Updating files: 12% (49/405) Updating files: 13% (53/405) Updating files: 14% (57/405) Updating files: 15% (61/405) Updating files: 16% (65/405) Updating files: 17% (69/405) Updating files: 18% (73/405) Updating files: 19% (77/405) Updating files: 20% (81/405) Updating files: 21% (86/405) Updating files: 22% (90/405) Updating files: 23% (94/405) Updating files: 24% (98/405) Updating files: 25% (102/405) Updating files: 26% (106/405) Updating files: 27% (110/405) Updating files: 28% (114/405) Updating files: 29% (118/405) Updating files: 30% (122/405) Updating files: 30% (124/405) Updating files: 31% (126/405) Updating files: 32% (130/405) Updating files: 33% (134/405) Updating files: 34% (138/405) Updating files: 34% (141/405) Updating files: 35% (142/405) Updating files: 36% (146/405) Updating files: 37% (150/405) Updating files: 38% (154/405) Updating files: 39% (158/405) Updating files: 40% (162/405) Updating files: 41% (167/405) Updating files: 42% (171/405) Updating files: 43% (175/405) Updating files: 44% (179/405) Updating files: 45% (183/405) Updating files: 46% (187/405) Updating files: 47% (191/405) Updating files: 48% (195/405) Updating files: 49% (199/405) Updating files: 50% (203/405) Updating files: 51% (207/405) Updating files: 52% (211/405) Updating files: 53% (215/405) Updating files: 54% (219/405) Updating files: 55% (223/405) Updating files: 56% (227/405) Updating files: 56% (228/405) Updating files: 57% (231/405) Updating files: 58% (235/405) Updating files: 59% (239/405) Updating files: 60% (243/405) Updating files: 61% (248/405) Updating files: 62% (252/405) Updating files: 63% (256/405) Updating files: 64% (260/405) Updating files: 65% (264/405) Updating files: 66% (268/405) Updating files: 67% (272/405) Updating files: 68% (276/405) Updating files: 69% (280/405) Updating files: 70% (284/405) Updating files: 71% (288/405) Updating files: 72% (292/405) Updating files: 73% (296/405) Updating files: 74% (300/405) Updating files: 75% (304/405) Updating files: 76% (308/405) Updating files: 77% (312/405) Updating files: 78% (316/405) Updating files: 79% (320/405) Updating files: 80% (324/405) Updating files: 81% (329/405) Updating files: 82% (333/405) Updating files: 83% (337/405) Updating files: 84% (341/405) Updating files: 85% (345/405) Updating files: 86% (349/405) Updating files: 87% (353/405) Updating files: 88% (357/405) Updating files: 89% (361/405) Updating files: 90% (365/405) Updating files: 91% (369/405) Updating files: 92% (373/405) Updating files: 93% (377/405) Updating files: 94% (381/405) Updating files: 95% (385/405) Updating files: 96% (389/405) Updating files: 97% (393/405) Updating files: 98% (397/405) Updating files: 99% (401/405) Updating files: 100% (405/405) Updating files: 100% (405/405), done. 2023-10-30T14:50:11.570 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at e509cb1e69c v0.71 2023-10-30T14:50:11.577 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.71 2023-10-30T14:50:11.656 INFO:tasks.workunit.client.0.smithi057.stdout:v0.72 2023-10-30T14:50:11.657 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 72 -eq 48 ] 2023-10-30T14:50:11.657 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.72 2023-10-30T14:50:11.657 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.72 2023-10-30T14:50:11.657 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.72 2023-10-30T14:50:11.860 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:50:11.865 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:50:14.734 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 28% (55/191) Updating files: 29% (56/191) Updating files: 30% (58/191) Updating files: 31% (60/191) Updating files: 32% (62/191) Updating files: 33% (64/191) Updating files: 34% (65/191) Updating files: 35% (67/191) Updating files: 36% (69/191) Updating files: 37% (71/191) Updating files: 38% (73/191) Updating files: 39% (75/191) Updating files: 40% (77/191) Updating files: 41% (79/191) Updating files: 42% (81/191) Updating files: 43% (83/191) Updating files: 44% (85/191) Updating files: 45% (86/191) Updating files: 46% (88/191) Updating files: 47% (90/191) Updating files: 48% (92/191) Updating files: 49% (94/191) Updating files: 50% (96/191) Updating files: 51% (98/191) Updating files: 52% (100/191) Updating files: 53% (102/191) Updating files: 54% (104/191) Updating files: 55% (106/191) Updating files: 56% (107/191) Updating files: 57% (109/191) Updating files: 58% (111/191) Updating files: 59% (113/191) Updating files: 60% (115/191) Updating files: 61% (117/191) Updating files: 62% (119/191) Updating files: 63% (121/191) Updating files: 64% (123/191) Updating files: 65% (125/191) Updating files: 66% (127/191) Updating files: 67% (128/191) Updating files: 68% (130/191) Updating files: 69% (132/191) Updating files: 70% (134/191) Updating files: 71% (136/191) Updating files: 72% (138/191) Updating files: 73% (140/191) Updating files: 74% (142/191) Updating files: 75% (144/191) Updating files: 76% (146/191) Updating files: 77% (148/191) Updating files: 78% (149/191) Updating files: 79% (151/191) Updating files: 80% (153/191) Updating files: 81% (155/191) Updating files: 82% (157/191) Updating files: 83% (159/191) Updating files: 84% (161/191) Updating files: 85% (163/191) Updating files: 86% (165/191) Updating files: 87% (167/191) Updating files: 88% (169/191) Updating files: 89% (170/191) Updating files: 90% (172/191) Updating files: 91% (174/191) Updating files: 92% (176/191) Updating files: 93% (178/191) Updating files: 94% (180/191) Updating files: 95% (182/191) Updating files: 96% (184/191) Updating files: 97% (186/191) Updating files: 98% (188/191) Updating files: 99% (190/191) Updating files: 100% (191/191) Updating files: 100% (191/191), done. 2023-10-30T14:50:14.783 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at 5832e2603c7 v0.72 2023-10-30T14:50:14.790 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.72 2023-10-30T14:50:14.905 INFO:tasks.workunit.client.0.smithi057.stdout:v0.73 2023-10-30T14:50:14.905 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 73 -eq 48 ] 2023-10-30T14:50:14.905 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.73 2023-10-30T14:50:14.906 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.73 2023-10-30T14:50:14.906 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.73 2023-10-30T14:50:16.992 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 47% (70/148) Updating files: 48% (72/148) Updating files: 49% (73/148) Updating files: 50% (74/148) Updating files: 51% (76/148) Updating files: 52% (77/148) Updating files: 53% (79/148) Updating files: 54% (80/148) Updating files: 55% (82/148) Updating files: 56% (83/148) Updating files: 57% (85/148) Updating files: 58% (86/148) Updating files: 59% (88/148) Updating files: 60% (89/148) Updating files: 61% (91/148) Updating files: 62% (92/148) Updating files: 63% (94/148) Updating files: 64% (95/148) Updating files: 65% (97/148) Updating files: 66% (98/148) Updating files: 67% (100/148) Updating files: 68% (101/148) Updating files: 69% (103/148) Updating files: 70% (104/148) Updating files: 71% (106/148) Updating files: 72% (107/148) Updating files: 73% (109/148) Updating files: 74% (110/148) Updating files: 75% (111/148) Updating files: 76% (113/148) Updating files: 77% (114/148) Updating files: 78% (116/148) Updating files: 79% (117/148) Updating files: 80% (119/148) Updating files: 81% (120/148) Updating files: 82% (122/148) Updating files: 83% (123/148) Updating files: 84% (125/148) Updating files: 85% (126/148) Updating files: 86% (128/148) Updating files: 87% (129/148) Updating files: 88% (131/148) Updating files: 89% (132/148) Updating files: 90% (134/148) Updating files: 91% (135/148) Updating files: 92% (137/148) Updating files: 93% (138/148) Updating files: 94% (140/148) Updating files: 95% (141/148) Updating files: 96% (143/148) Updating files: 97% (144/148) Updating files: 98% (146/148) Updating files: 99% (147/148) Updating files: 100% (148/148) Updating files: 100% (148/148), done. 2023-10-30T14:50:17.911 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at d8ad51ee8ab v0.73 2023-10-30T14:50:17.920 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.73 2023-10-30T14:50:18.238 INFO:tasks.workunit.client.0.smithi057.stdout:v0.74 2023-10-30T14:50:18.238 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 74 -eq 48 ] 2023-10-30T14:50:18.238 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.74 2023-10-30T14:50:18.239 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.74 2023-10-30T14:50:18.239 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.74 2023-10-30T14:50:31.731 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 76% (146/192) Updating files: 77% (148/192) Updating files: 78% (150/192) Updating files: 79% (152/192) Updating files: 79% (153/192) Updating files: 80% (154/192) Updating files: 81% (156/192) Updating files: 82% (158/192) Updating files: 83% (160/192) Updating files: 84% (162/192) Updating files: 85% (164/192) Updating files: 86% (166/192) Updating files: 87% (168/192) Updating files: 88% (169/192) Updating files: 89% (171/192) Updating files: 90% (173/192) Updating files: 91% (175/192) Updating files: 92% (177/192) Updating files: 93% (179/192) Updating files: 94% (181/192) Updating files: 95% (183/192) Updating files: 96% (185/192) Updating files: 97% (187/192) Updating files: 98% (189/192) Updating files: 99% (191/192) Updating files: 100% (192/192) Updating files: 100% (192/192), done. 2023-10-30T14:50:32.125 INFO:tasks.workunit.client.0.smithi057.stdout:HEAD is now at c165483bc72 v0.74 2023-10-30T14:50:32.131 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir .snap/v0.74 2023-10-30T14:50:32.169 INFO:tasks.workunit.client.0.smithi057.stdout:v0.75 2023-10-30T14:50:32.170 INFO:tasks.workunit.client.0.smithi057.stderr:+ [ 75 -eq 48 ] 2023-10-30T14:50:32.170 INFO:tasks.workunit.client.0.smithi057.stderr:+ ver=v0.75 2023-10-30T14:50:32.170 INFO:tasks.workunit.client.0.smithi057.stderr:+ echo v0.75 2023-10-30T14:50:32.171 INFO:tasks.workunit.client.0.smithi057.stderr:+ git reset --hard v0.75 2023-10-30T14:50:32.953 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:50:33.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.101+0000 7f9342409640 1 Processor -- start 2023-10-30T14:50:33.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.101+0000 7f9342409640 1 -- start start 2023-10-30T14:50:33.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.101+0000 7f9342409640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f933c151840 0x7f933c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:33.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.101+0000 7f9342409640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f933c151c20 0x7f933c072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:33.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.101+0000 7f9342409640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c0c5030 con 0x7f933c0cb8b0 2023-10-30T14:50:33.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.105+0000 7f9342409640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f933c157830 con 0x7f933c151840 2023-10-30T14:50:33.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.105+0000 7f9342409640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933c157b10 con 0x7f933c151c20 2023-10-30T14:50:33.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.105+0000 7f933b7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f933c151c20 0x7f933c072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:33.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.105+0000 7f933b7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f933c151c20 0x7f933c072000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:41502/0 (socket says 172.21.15.57:41502) 2023-10-30T14:50:33.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.105+0000 7f933b7fe640 1 -- 172.21.15.57:0/3193609136 learned_addr learned my addr 172.21.15.57:0/3193609136 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:33.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.105+0000 7f933bfff640 1 --2- 172.21.15.57:0/3193609136 >> v2:172.21.15.88:3300/0 conn(0x7f933c151840 0x7f933c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:33.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.105+0000 7f933bfff640 1 -- 172.21.15.57:0/3193609136 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f933c151c20 msgr2=0x7f933c072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:50:33.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.105+0000 7f933bfff640 1 --2- 172.21.15.57:0/3193609136 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f933c151c20 0x7f933c072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.105+0000 7f933bfff640 1 -- 172.21.15.57:0/3193609136 >> v1:172.21.15.88:6789/0 conn(0x7f933c0cb8b0 legacy=0x7f933c06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:33.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.105+0000 7f933bfff640 1 -- 172.21.15.57:0/3193609136 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c157df0 con 0x7f933c151840 2023-10-30T14:50:33.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.105+0000 7f933b7fe640 1 --2- 172.21.15.57:0/3193609136 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f933c151c20 0x7f933c072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:50:33.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.109+0000 7f933bfff640 1 --2- 172.21.15.57:0/3193609136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f933c151840 0x7f933c0c49a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f932c0097c0 tx=0x7f932c01ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63c937fbcfd18fcc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:33.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.109+0000 7f933affd640 1 -- 172.21.15.57:0/3193609136 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c0a8970 con 0x7f933c151840 2023-10-30T14:50:33.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.109+0000 7f933affd640 1 -- 172.21.15.57:0/3193609136 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932c0a8ad0 con 0x7f933c151840 2023-10-30T14:50:33.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.109+0000 7f933affd640 1 -- 172.21.15.57:0/3193609136 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c0a8e60 con 0x7f933c151840 2023-10-30T14:50:33.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.109+0000 7f9342409640 1 -- 172.21.15.57:0/3193609136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f933c151840 msgr2=0x7f933c0c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:33.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.109+0000 7f9342409640 1 --2- 172.21.15.57:0/3193609136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f933c151840 0x7f933c0c49a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f932c0097c0 tx=0x7f932c01ec70 comp rx=0 tx=0).stop 2023-10-30T14:50:33.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.109+0000 7f9342409640 1 -- 172.21.15.57:0/3193609136 shutdown_connections 2023-10-30T14:50:33.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.109+0000 7f9342409640 1 --2- 172.21.15.57:0/3193609136 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f933c151c20 0x7f933c072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.109+0000 7f9342409640 1 --2- 172.21.15.57:0/3193609136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f933c151840 0x7f933c0c49a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.109+0000 7f9342409640 1 -- 172.21.15.57:0/3193609136 >> 172.21.15.57:0/3193609136 conn(0x7f933c0cac70 msgr2=0x7f933c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:33.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.109+0000 7f9342409640 1 -- 172.21.15.57:0/3193609136 shutdown_connections 2023-10-30T14:50:33.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.109+0000 7f9342409640 1 -- 172.21.15.57:0/3193609136 wait complete. 2023-10-30T14:50:33.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.109+0000 7f9342409640 1 Processor -- start 2023-10-30T14:50:33.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.109+0000 7f9342409640 1 -- start start 2023-10-30T14:50:33.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f9342409640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f933c0cb8b0 0x7f933c1ef300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:33.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f9342409640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f933c151840 0x7f933c155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:33.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f9342409640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f933c151c20 0x7f933c1566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:33.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f9342409640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c152650 con 0x7f933c151c20 2023-10-30T14:50:33.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f9342409640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c146e20 con 0x7f933c0cb8b0 2023-10-30T14:50:33.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f9342409640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933c1ef840 con 0x7f933c151840 2023-10-30T14:50:33.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f93409a6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f933c151c20 0x7f933c1566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:33.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f93409a6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f933c151c20 0x7f933c1566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33364/0 (socket says 172.21.15.57:33364) 2023-10-30T14:50:33.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f93409a6640 1 -- 172.21.15.57:0/2373007717 learned_addr learned my addr 172.21.15.57:0/2373007717 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:33.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f933b7fe640 1 --2- 172.21.15.57:0/2373007717 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f933c151840 0x7f933c155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:33.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f933bfff640 1 --2- 172.21.15.57:0/2373007717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f933c0cb8b0 0x7f933c1ef300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:33.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f93409a6640 1 -- 172.21.15.57:0/2373007717 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f933c151840 msgr2=0x7f933c155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:33.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f93409a6640 1 --2- 172.21.15.57:0/2373007717 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f933c151840 0x7f933c155c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f93409a6640 1 -- 172.21.15.57:0/2373007717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f933c0cb8b0 msgr2=0x7f933c1ef300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:33.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f93409a6640 1 --2- 172.21.15.57:0/2373007717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f933c0cb8b0 0x7f933c1ef300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f93409a6640 1 -- 172.21.15.57:0/2373007717 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93300094d0 con 0x7f933c151c20 2023-10-30T14:50:33.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f93409a6640 1 --2- 172.21.15.57:0/2373007717 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f933c151c20 0x7f933c1566b0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f933400c7a0 tx=0x7f933400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:33.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f9338ff9640 1 -- 172.21.15.57:0/2373007717 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9334003b80 con 0x7f933c151c20 2023-10-30T14:50:33.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f9338ff9640 1 -- 172.21.15.57:0/2373007717 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9334003ce0 con 0x7f933c151c20 2023-10-30T14:50:33.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f9338ff9640 1 -- 172.21.15.57:0/2373007717 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933400aa30 con 0x7f933c151c20 2023-10-30T14:50:33.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f9342409640 1 -- 172.21.15.57:0/2373007717 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f932c009420 con 0x7f933c151c20 2023-10-30T14:50:33.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f9342409640 1 -- 172.21.15.57:0/2373007717 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f933c146330 con 0x7f933c151c20 2023-10-30T14:50:33.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f9342409640 1 -- 172.21.15.57:0/2373007717 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f933c06c1c0 con 0x7f933c151c20 2023-10-30T14:50:33.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.113+0000 7f9338ff9640 1 -- 172.21.15.57:0/2373007717 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f933400ab90 con 0x7f933c151c20 2023-10-30T14:50:33.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.117+0000 7f9338ff9640 1 --2- 172.21.15.57:0/2373007717 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f93280664a0 0x7f9328068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:33.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.117+0000 7f9338ff9640 1 -- 172.21.15.57:0/2373007717 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7f9334084180 con 0x7f933c151c20 2023-10-30T14:50:33.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.117+0000 7f933bfff640 1 --2- 172.21.15.57:0/2373007717 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f93280664a0 0x7f9328068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:33.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.117+0000 7f933bfff640 1 --2- 172.21.15.57:0/2373007717 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f93280664a0 0x7f9328068960 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f932c0b2000 tx=0x7f932c0aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:33.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.121+0000 7f9338ff9640 1 -- 172.21.15.57:0/2373007717 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9334050fe0 con 0x7f933c151c20 2023-10-30T14:50:33.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.309+0000 7f9342409640 1 -- 172.21.15.57:0/2373007717 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f933c04e170 con 0x7f933c151c20 2023-10-30T14:50:33.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.309+0000 7f9338ff9640 1 -- 172.21.15.57:0/2373007717 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+6502 (secure 0 0 0) 0x7f9334054600 con 0x7f933c151c20 2023-10-30T14:50:33.313 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:50:33.313 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":8058,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:52.300180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:active","state_seq":30,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:50:33.314 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 184 2023-10-30T14:50:33.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.313+0000 7f9342409640 1 -- 172.21.15.57:0/2373007717 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f93280664a0 msgr2=0x7f9328068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:33.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.313+0000 7f9342409640 1 --2- 172.21.15.57:0/2373007717 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f93280664a0 0x7f9328068960 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f932c0b2000 tx=0x7f932c0aa070 comp rx=0 tx=0).stop 2023-10-30T14:50:33.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.313+0000 7f9342409640 1 -- 172.21.15.57:0/2373007717 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f933c151c20 msgr2=0x7f933c1566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:33.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.313+0000 7f9342409640 1 --2- 172.21.15.57:0/2373007717 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f933c151c20 0x7f933c1566b0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f933400c7a0 tx=0x7f933400cc60 comp rx=0 tx=0).stop 2023-10-30T14:50:33.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.313+0000 7f9342409640 1 -- 172.21.15.57:0/2373007717 shutdown_connections 2023-10-30T14:50:33.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.313+0000 7f9342409640 1 --2- 172.21.15.57:0/2373007717 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f93280664a0 0x7f9328068960 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.313+0000 7f9342409640 1 --2- 172.21.15.57:0/2373007717 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f933c151c20 0x7f933c1566b0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.313+0000 7f9342409640 1 --2- 172.21.15.57:0/2373007717 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f933c151840 0x7f933c155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.313+0000 7f9342409640 1 --2- 172.21.15.57:0/2373007717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f933c0cb8b0 0x7f933c1ef300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.313+0000 7f9342409640 1 -- 172.21.15.57:0/2373007717 >> 172.21.15.57:0/2373007717 conn(0x7f933c0cac70 msgr2=0x7f933c073730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:33.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.313+0000 7f9342409640 1 -- 172.21.15.57:0/2373007717 shutdown_connections 2023-10-30T14:50:33.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.313+0000 7f9342409640 1 -- 172.21.15.57:0/2373007717 wait complete. 2023-10-30T14:50:33.340 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T14:50:33.340 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T14:50:33.341 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:50:33.375 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T14:50:33.375 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:50:33.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68b2668640 1 Processor -- start 2023-10-30T14:50:33.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68b2668640 1 -- start start 2023-10-30T14:50:33.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68b2668640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f68ac151840 0x7f68ac0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:33.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68b2668640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68ac0cb8b0 0x7f68ac06b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:33.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68b2668640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68ac0c5030 con 0x7f68ac151c20 2023-10-30T14:50:33.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68b2668640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68ac157830 con 0x7f68ac151840 2023-10-30T14:50:33.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68b2668640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac157b10 con 0x7f68ac0cb8b0 2023-10-30T14:50:33.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68ab7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f68ac151c20 0x7f68ac072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:60762/0 (socket says 172.21.15.57:60762) 2023-10-30T14:50:33.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68ab7fe640 1 -- 172.21.15.57:0/1977328389 learned_addr learned my addr 172.21.15.57:0/1977328389 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:50:33.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68b0c05640 1 --2- 172.21.15.57:0/1977328389 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68ac0cb8b0 0x7f68ac06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:33.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68abfff640 1 --2- 172.21.15.57:0/1977328389 >> v2:172.21.15.88:3300/0 conn(0x7f68ac151840 0x7f68ac0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:33.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68aaffd640 1 -- 172.21.15.57:0/1977328389 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 889712733 0 0) 0x7f68ac0c5030 con 0x7f68ac151c20 2023-10-30T14:50:33.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68aaffd640 1 -- 172.21.15.57:0/1977328389 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6894003530 con 0x7f68ac151c20 2023-10-30T14:50:33.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68aaffd640 1 -- 172.21.15.57:0/1977328389 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1046090054 0 0) 0x7f6894003530 con 0x7f68ac151c20 2023-10-30T14:50:33.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68aaffd640 1 -- 172.21.15.57:0/1977328389 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68ac0cb8b0 msgr2=0x7f68ac06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:50:33.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68aaffd640 1 --2- 172.21.15.57:0/1977328389 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68ac0cb8b0 0x7f68ac06b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68aaffd640 1 -- 172.21.15.57:0/1977328389 >> v2:172.21.15.88:3300/0 conn(0x7f68ac151840 msgr2=0x7f68ac0c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:50:33.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68aaffd640 1 --2- 172.21.15.57:0/1977328389 >> v2:172.21.15.88:3300/0 conn(0x7f68ac151840 0x7f68ac0c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68aaffd640 1 -- 172.21.15.57:0/1977328389 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac157df0 con 0x7f68ac151c20 2023-10-30T14:50:33.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.561+0000 7f68aaffd640 1 -- 172.21.15.57:0/1977328389 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f68a0001680 con 0x7f68ac151c20 2023-10-30T14:50:33.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.565+0000 7f68aaffd640 1 -- 172.21.15.57:0/1977328389 >> v1:172.21.15.57:6789/0 conn(0x7f68ac151c20 legacy=0x7f68ac072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:33.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.565+0000 7f68aaffd640 1 --2- 172.21.15.57:0/1977328389 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68ac0cb8b0 0x7f6894003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:33.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.565+0000 7f68aaffd640 1 --2- 172.21.15.57:0/1977328389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6894003f00 0x7f68940047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:33.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.565+0000 7f68aaffd640 1 --2- 172.21.15.57:0/1977328389 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6894004d10 0x7f6894005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:33.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.565+0000 7f68aaffd640 1 -- 172.21.15.57:0/1977328389 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac157830 con 0x7f6894004d10 2023-10-30T14:50:33.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.565+0000 7f68aaffd640 1 -- 172.21.15.57:0/1977328389 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac157b10 con 0x7f6894003f00 2023-10-30T14:50:33.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.565+0000 7f68aaffd640 1 -- 172.21.15.57:0/1977328389 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f689400b9c0 con 0x7f68ac0cb8b0 2023-10-30T14:50:33.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.565+0000 7f68b0c05640 1 --2- 172.21.15.57:0/1977328389 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6894004d10 0x7f6894005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:33.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.565+0000 7f68abfff640 1 --2- 172.21.15.57:0/1977328389 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68ac0cb8b0 0x7f6894003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:33.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.565+0000 7f68ab7fe640 1 --2- 172.21.15.57:0/1977328389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6894003f00 0x7f68940047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:33.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.565+0000 7f68b0c05640 1 -- 172.21.15.57:0/1977328389 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68ac0cb8b0 msgr2=0x7f6894003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:50:33.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.565+0000 7f68b0c05640 1 --2- 172.21.15.57:0/1977328389 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68ac0cb8b0 0x7f6894003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.565+0000 7f68b0c05640 1 -- 172.21.15.57:0/1977328389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6894003f00 msgr2=0x7f68940047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:50:33.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.565+0000 7f68b0c05640 1 --2- 172.21.15.57:0/1977328389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6894003f00 0x7f68940047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.565+0000 7f68b0c05640 1 -- 172.21.15.57:0/1977328389 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6894003530 con 0x7f6894004d10 2023-10-30T14:50:33.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.565+0000 7f68b0c05640 1 --2- 172.21.15.57:0/1977328389 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6894004d10 0x7f6894005190 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f68a4009fc0 tx=0x7f68a401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d26244f719b5bcb0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:33.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.565+0000 7f68aaffd640 1 -- 172.21.15.57:0/1977328389 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a40a5c70 con 0x7f6894004d10 2023-10-30T14:50:33.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.565+0000 7f68aaffd640 1 -- 172.21.15.57:0/1977328389 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a400f040 con 0x7f6894004d10 2023-10-30T14:50:33.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.565+0000 7f68aaffd640 1 -- 172.21.15.57:0/1977328389 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a40ad440 con 0x7f6894004d10 2023-10-30T14:50:33.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68b2668640 1 -- 172.21.15.57:0/1977328389 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6894004d10 msgr2=0x7f6894005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:33.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68b2668640 1 --2- 172.21.15.57:0/1977328389 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6894004d10 0x7f6894005190 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f68a4009fc0 tx=0x7f68a401cc60 comp rx=0 tx=0).stop 2023-10-30T14:50:33.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68b0c05640 1 -- 172.21.15.57:0/1977328389 reap_dead start 2023-10-30T14:50:33.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68b2668640 1 -- 172.21.15.57:0/1977328389 shutdown_connections 2023-10-30T14:50:33.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68b2668640 1 -- 172.21.15.57:0/1977328389 >> 172.21.15.57:0/1977328389 conn(0x7f68ac0cac70 msgr2=0x7f68ac0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:33.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68b2668640 1 -- 172.21.15.57:0/1977328389 shutdown_connections 2023-10-30T14:50:33.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68b2668640 1 -- 172.21.15.57:0/1977328389 wait complete. 2023-10-30T14:50:33.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68b2668640 1 Processor -- start 2023-10-30T14:50:33.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68b2668640 1 -- start start 2023-10-30T14:50:33.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68b2668640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6894004d10 0x7f68ac1560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:33.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68b2668640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68ac1569f0 0x7f68ac155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:33.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68b2668640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68ac155a40 0x7f68ac0c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:33.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68b2668640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac0705f0 con 0x7f6894004d10 2023-10-30T14:50:33.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68b2668640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac0c8af0 con 0x7f68ac1569f0 2023-10-30T14:50:33.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68b2668640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac071290 con 0x7f68ac155a40 2023-10-30T14:50:33.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68abfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6894004d10 0x7f68ac1560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:33.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68abfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6894004d10 0x7f68ac1560c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33384/0 (socket says 172.21.15.57:33384) 2023-10-30T14:50:33.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68abfff640 1 -- 172.21.15.57:0/3877160984 learned_addr learned my addr 172.21.15.57:0/3877160984 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:33.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68b0c05640 1 --2- 172.21.15.57:0/3877160984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68ac155a40 0x7f68ac0c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:33.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.569+0000 7f68ab7fe640 1 --2- 172.21.15.57:0/3877160984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68ac1569f0 0x7f68ac155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:33.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.573+0000 7f68abfff640 1 -- 172.21.15.57:0/3877160984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68ac155a40 msgr2=0x7f68ac0c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:33.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.573+0000 7f68abfff640 1 --2- 172.21.15.57:0/3877160984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68ac155a40 0x7f68ac0c9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.573+0000 7f68abfff640 1 -- 172.21.15.57:0/3877160984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68ac1569f0 msgr2=0x7f68ac155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:33.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.573+0000 7f68abfff640 1 --2- 172.21.15.57:0/3877160984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68ac1569f0 0x7f68ac155500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.573+0000 7f68abfff640 1 -- 172.21.15.57:0/3877160984 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac071510 con 0x7f6894004d10 2023-10-30T14:50:33.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.573+0000 7f68abfff640 1 --2- 172.21.15.57:0/3877160984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6894004d10 0x7f68ac1560c0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f689c011aa0 tx=0x7f689c011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:33.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.573+0000 7f68a8ff9640 1 -- 172.21.15.57:0/3877160984 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c00fa60 con 0x7f6894004d10 2023-10-30T14:50:33.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.573+0000 7f68a8ff9640 1 -- 172.21.15.57:0/3877160984 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689c00fbc0 con 0x7f6894004d10 2023-10-30T14:50:33.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.573+0000 7f68a8ff9640 1 -- 172.21.15.57:0/3877160984 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c014440 con 0x7f6894004d10 2023-10-30T14:50:33.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.573+0000 7f68b2668640 1 -- 172.21.15.57:0/3877160984 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ac0717f0 con 0x7f6894004d10 2023-10-30T14:50:33.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.573+0000 7f68b2668640 1 -- 172.21.15.57:0/3877160984 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ac0c9040 con 0x7f6894004d10 2023-10-30T14:50:33.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.573+0000 7f68b2668640 1 -- 172.21.15.57:0/3877160984 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ac04e170 con 0x7f6894004d10 2023-10-30T14:50:33.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.573+0000 7f68a8ff9640 1 -- 172.21.15.57:0/3877160984 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f689c0145a0 con 0x7f6894004d10 2023-10-30T14:50:33.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.573+0000 7f68a8ff9640 1 --2- 172.21.15.57:0/3877160984 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f68980664a0 0x7f6898068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:33.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.577+0000 7f68a8ff9640 1 -- 172.21.15.57:0/3877160984 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7f689c084cf0 con 0x7f6894004d10 2023-10-30T14:50:33.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.577+0000 7f68ab7fe640 1 --2- 172.21.15.57:0/3877160984 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f68980664a0 0x7f6898068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:33.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.577+0000 7f68ab7fe640 1 --2- 172.21.15.57:0/3877160984 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f68980664a0 0x7f6898068960 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f68ac150c90 tx=0x7f68a0007900 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:33.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.577+0000 7f68a8ff9640 1 -- 172.21.15.57:0/3877160984 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f689c051aa0 con 0x7f6894004d10 2023-10-30T14:50:33.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.769+0000 7f68b2668640 1 -- 172.21.15.57:0/3877160984 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68ac0c78d0 con 0x7f6894004d10 2023-10-30T14:50:33.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.769+0000 7f68a8ff9640 1 -- 172.21.15.57:0/3877160984 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+6502 (secure 0 0 0) 0x7f689c022090 con 0x7f6894004d10 2023-10-30T14:50:33.772 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:50:33.772 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":8058,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:52.300180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:active","state_seq":30,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:50:33.773 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 184 2023-10-30T14:50:33.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.769+0000 7f68b2668640 1 -- 172.21.15.57:0/3877160984 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f68980664a0 msgr2=0x7f6898068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:33.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.769+0000 7f68b2668640 1 --2- 172.21.15.57:0/3877160984 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f68980664a0 0x7f6898068960 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f68ac150c90 tx=0x7f68a0007900 comp rx=0 tx=0).stop 2023-10-30T14:50:33.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.769+0000 7f68b2668640 1 -- 172.21.15.57:0/3877160984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6894004d10 msgr2=0x7f68ac1560c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:33.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.769+0000 7f68b2668640 1 --2- 172.21.15.57:0/3877160984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6894004d10 0x7f68ac1560c0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f689c011aa0 tx=0x7f689c011ea0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.769+0000 7f68b2668640 1 -- 172.21.15.57:0/3877160984 shutdown_connections 2023-10-30T14:50:33.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.769+0000 7f68b2668640 1 --2- 172.21.15.57:0/3877160984 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f68980664a0 0x7f6898068960 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.769+0000 7f68b2668640 1 --2- 172.21.15.57:0/3877160984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f68ac155a40 0x7f68ac0c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.769+0000 7f68b2668640 1 --2- 172.21.15.57:0/3877160984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68ac1569f0 0x7f68ac155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.769+0000 7f68b2668640 1 --2- 172.21.15.57:0/3877160984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6894004d10 0x7f68ac1560c0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:33.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.769+0000 7f68b2668640 1 -- 172.21.15.57:0/3877160984 >> 172.21.15.57:0/3877160984 conn(0x7f68ac0cac70 msgr2=0x7f68ac0c7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:33.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.769+0000 7f68b2668640 1 -- 172.21.15.57:0/3877160984 shutdown_connections 2023-10-30T14:50:33.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:33.769+0000 7f68b2668640 1 -- 172.21.15.57:0/3877160984 wait complete. 2023-10-30T14:50:33.793 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:50:35.794 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:50:35.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.945+0000 7fdb5a580640 1 Processor -- start 2023-10-30T14:50:35.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.945+0000 7fdb5a580640 1 -- start start 2023-10-30T14:50:35.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.949+0000 7fdb5a580640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb54151840 0x7fdb540c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:35.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.949+0000 7fdb5a580640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdb54151c20 0x7fdb54072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:35.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.949+0000 7fdb5a580640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb540c5030 con 0x7fdb540cb8b0 2023-10-30T14:50:35.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.949+0000 7fdb5a580640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb54157830 con 0x7fdb54151c20 2023-10-30T14:50:35.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.949+0000 7fdb5a580640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb54157b10 con 0x7fdb54151840 2023-10-30T14:50:35.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.949+0000 7fdb58b1d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fdb540cb8b0 0x7fdb5406b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:59486/0 (socket says 172.21.15.57:59486) 2023-10-30T14:50:35.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.949+0000 7fdb58b1d640 1 -- 172.21.15.57:0/1578223430 learned_addr learned my addr 172.21.15.57:0/1578223430 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:50:35.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.949+0000 7fdb537fe640 1 --2- 172.21.15.57:0/1578223430 >> v2:172.21.15.88:3300/0 conn(0x7fdb54151c20 0x7fdb54072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:35.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.949+0000 7fdb53fff640 1 --2- 172.21.15.57:0/1578223430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb54151840 0x7fdb540c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:35.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.949+0000 7fdb52ffd640 1 -- 172.21.15.57:0/1578223430 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027311386 0 0) 0x7fdb540c5030 con 0x7fdb540cb8b0 2023-10-30T14:50:35.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.949+0000 7fdb52ffd640 1 -- 172.21.15.57:0/1578223430 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb3c003530 con 0x7fdb540cb8b0 2023-10-30T14:50:35.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.949+0000 7fdb53fff640 1 -- 172.21.15.57:0/1578223430 >> v2:172.21.15.88:3300/0 conn(0x7fdb54151c20 msgr2=0x7fdb54072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:50:35.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.949+0000 7fdb53fff640 1 --2- 172.21.15.57:0/1578223430 >> v2:172.21.15.88:3300/0 conn(0x7fdb54151c20 0x7fdb54072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:35.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.949+0000 7fdb53fff640 1 -- 172.21.15.57:0/1578223430 >> v1:172.21.15.57:6789/0 conn(0x7fdb540cb8b0 legacy=0x7fdb5406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:35.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.949+0000 7fdb52ffd640 1 -- 172.21.15.57:0/1578223430 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3659091923 0 0) 0x7fdb3c003530 con 0x7fdb540cb8b0 2023-10-30T14:50:35.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.949+0000 7fdb53fff640 1 -- 172.21.15.57:0/1578223430 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb54157df0 con 0x7fdb54151840 2023-10-30T14:50:35.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.953+0000 7fdb53fff640 1 --2- 172.21.15.57:0/1578223430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb54151840 0x7fdb540c49a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fdb440097c0 tx=0x7fdb4401eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87e311acf9b0f9f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:35.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.953+0000 7fdb52ffd640 1 -- 172.21.15.57:0/1578223430 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c001a00 con 0x7fdb54151840 2023-10-30T14:50:35.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.953+0000 7fdb52ffd640 1 -- 172.21.15.57:0/1578223430 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb440a85d0 con 0x7fdb54151840 2023-10-30T14:50:35.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.953+0000 7fdb52ffd640 1 -- 172.21.15.57:0/1578223430 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb440a8940 con 0x7fdb54151840 2023-10-30T14:50:35.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.953+0000 7fdb5a580640 1 -- 172.21.15.57:0/1578223430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb54151840 msgr2=0x7fdb540c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:35.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.953+0000 7fdb5a580640 1 --2- 172.21.15.57:0/1578223430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb54151840 0x7fdb540c49a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fdb440097c0 tx=0x7fdb4401eb90 comp rx=0 tx=0).stop 2023-10-30T14:50:35.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.953+0000 7fdb5a580640 1 -- 172.21.15.57:0/1578223430 shutdown_connections 2023-10-30T14:50:35.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.953+0000 7fdb5a580640 1 --2- 172.21.15.57:0/1578223430 >> v2:172.21.15.88:3300/0 conn(0x7fdb54151c20 0x7fdb54072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:35.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.953+0000 7fdb5a580640 1 --2- 172.21.15.57:0/1578223430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb54151840 0x7fdb540c49a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:35.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.953+0000 7fdb5a580640 1 -- 172.21.15.57:0/1578223430 >> 172.21.15.57:0/1578223430 conn(0x7fdb540cac70 msgr2=0x7fdb540cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:35.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.953+0000 7fdb5a580640 1 -- 172.21.15.57:0/1578223430 shutdown_connections 2023-10-30T14:50:35.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.953+0000 7fdb5a580640 1 -- 172.21.15.57:0/1578223430 wait complete. 2023-10-30T14:50:35.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb5a580640 1 Processor -- start 2023-10-30T14:50:35.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb5a580640 1 -- start start 2023-10-30T14:50:35.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb5a580640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb540cb8b0 0x7fdb54156450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:35.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb5a580640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb54151840 0x7fdb54156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:35.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb5a580640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb54151c20 0x7fdb54145440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:35.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb5a580640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5414d770 con 0x7fdb54151840 2023-10-30T14:50:35.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb5a580640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb54155fc0 con 0x7fdb54151c20 2023-10-30T14:50:35.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb5a580640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5414d1f0 con 0x7fdb540cb8b0 2023-10-30T14:50:35.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb537fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb54151840 0x7fdb54156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:35.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb537fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb54151840 0x7fdb54156990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35690/0 (socket says 172.21.15.57:35690) 2023-10-30T14:50:35.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb537fe640 1 -- 172.21.15.57:0/1069514035 learned_addr learned my addr 172.21.15.57:0/1069514035 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:35.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb53fff640 1 --2- 172.21.15.57:0/1069514035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb540cb8b0 0x7fdb54156450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:35.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb58b1d640 1 --2- 172.21.15.57:0/1069514035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb54151c20 0x7fdb54145440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:35.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb537fe640 1 -- 172.21.15.57:0/1069514035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb540cb8b0 msgr2=0x7fdb54156450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:35.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb537fe640 1 --2- 172.21.15.57:0/1069514035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb540cb8b0 0x7fdb54156450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:35.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb537fe640 1 -- 172.21.15.57:0/1069514035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb54151c20 msgr2=0x7fdb54145440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:35.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb537fe640 1 --2- 172.21.15.57:0/1069514035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb54151c20 0x7fdb54145440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:35.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb537fe640 1 -- 172.21.15.57:0/1069514035 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb44009420 con 0x7fdb54151840 2023-10-30T14:50:35.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb537fe640 1 --2- 172.21.15.57:0/1069514035 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb54151840 0x7fdb54156990 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fdb4800e840 tx=0x7fdb4800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:35.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb50ff9640 1 -- 172.21.15.57:0/1069514035 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb48009510 con 0x7fdb54151840 2023-10-30T14:50:35.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb50ff9640 1 -- 172.21.15.57:0/1069514035 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4800ee90 con 0x7fdb54151840 2023-10-30T14:50:35.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb50ff9640 1 -- 172.21.15.57:0/1069514035 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4800cb80 con 0x7fdb54151840 2023-10-30T14:50:35.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb5a580640 1 -- 172.21.15.57:0/1069514035 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb541e3b10 con 0x7fdb54151840 2023-10-30T14:50:35.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.957+0000 7fdb5a580640 1 -- 172.21.15.57:0/1069514035 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5414bc20 con 0x7fdb54151840 2023-10-30T14:50:35.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.961+0000 7fdb5a580640 1 -- 172.21.15.57:0/1069514035 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb54156130 con 0x7fdb54151840 2023-10-30T14:50:35.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.961+0000 7fdb50ff9640 1 -- 172.21.15.57:0/1069514035 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdb4800cce0 con 0x7fdb54151840 2023-10-30T14:50:35.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.961+0000 7fdb50ff9640 1 --2- 172.21.15.57:0/1069514035 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdb400664d0 0x7fdb40068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:35.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.961+0000 7fdb50ff9640 1 -- 172.21.15.57:0/1069514035 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7fdb480843e0 con 0x7fdb54151840 2023-10-30T14:50:35.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.961+0000 7fdb53fff640 1 --2- 172.21.15.57:0/1069514035 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdb400664d0 0x7fdb40068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:35.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.961+0000 7fdb53fff640 1 --2- 172.21.15.57:0/1069514035 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdb400664d0 0x7fdb40068990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fdb440b2000 tx=0x7fdb440aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:35.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:35.965+0000 7fdb50ff9640 1 -- 172.21.15.57:0/1069514035 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdb48051370 con 0x7fdb54151840 2023-10-30T14:50:36.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:36.157+0000 7fdb5a580640 1 -- 172.21.15.57:0/1069514035 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb5404e170 con 0x7fdb54151840 2023-10-30T14:50:36.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:36.161+0000 7fdb50ff9640 1 -- 172.21.15.57:0/1069514035 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+6502 (secure 0 0 0) 0x7fdb48054990 con 0x7fdb54151840 2023-10-30T14:50:36.163 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:50:36.163 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":8058,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:52.300180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:active","state_seq":30,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:50:36.164 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 184 2023-10-30T14:50:36.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:36.161+0000 7fdb5a580640 1 -- 172.21.15.57:0/1069514035 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdb400664d0 msgr2=0x7fdb40068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:36.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:36.161+0000 7fdb5a580640 1 --2- 172.21.15.57:0/1069514035 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdb400664d0 0x7fdb40068990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fdb440b2000 tx=0x7fdb440aa070 comp rx=0 tx=0).stop 2023-10-30T14:50:36.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:36.161+0000 7fdb5a580640 1 -- 172.21.15.57:0/1069514035 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb54151840 msgr2=0x7fdb54156990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:36.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:36.161+0000 7fdb5a580640 1 --2- 172.21.15.57:0/1069514035 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb54151840 0x7fdb54156990 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fdb4800e840 tx=0x7fdb4800ec40 comp rx=0 tx=0).stop 2023-10-30T14:50:36.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:36.161+0000 7fdb5a580640 1 -- 172.21.15.57:0/1069514035 shutdown_connections 2023-10-30T14:50:36.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:36.161+0000 7fdb5a580640 1 --2- 172.21.15.57:0/1069514035 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fdb400664d0 0x7fdb40068990 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:36.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:36.161+0000 7fdb5a580640 1 --2- 172.21.15.57:0/1069514035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb54151c20 0x7fdb54145440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:36.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:36.161+0000 7fdb5a580640 1 --2- 172.21.15.57:0/1069514035 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb54151840 0x7fdb54156990 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:36.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:36.161+0000 7fdb5a580640 1 --2- 172.21.15.57:0/1069514035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb540cb8b0 0x7fdb54156450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:36.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:36.161+0000 7fdb5a580640 1 -- 172.21.15.57:0/1069514035 >> 172.21.15.57:0/1069514035 conn(0x7fdb540cac70 msgr2=0x7fdb540c7680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:36.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:36.161+0000 7fdb5a580640 1 -- 172.21.15.57:0/1069514035 shutdown_connections 2023-10-30T14:50:36.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:36.161+0000 7fdb5a580640 1 -- 172.21.15.57:0/1069514035 wait complete. 2023-10-30T14:50:36.184 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:50:38.185 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:50:38.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.341+0000 7ff053bb8640 1 Processor -- start 2023-10-30T14:50:38.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.341+0000 7ff053bb8640 1 -- start start 2023-10-30T14:50:38.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.341+0000 7ff053bb8640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff04c151c20 0x7ff04c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:38.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.341+0000 7ff053bb8640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04c0c5030 con 0x7ff04c151840 2023-10-30T14:50:38.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.341+0000 7ff053bb8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04c157830 con 0x7ff04c0cb8b0 2023-10-30T14:50:38.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.341+0000 7ff053bb8640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff04c157b10 con 0x7ff04c151c20 2023-10-30T14:50:38.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.341+0000 7ff051153640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff04c151c20 0x7ff04c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:38.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.341+0000 7ff051153640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff04c151c20 0x7ff04c0c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35704/0 (socket says 172.21.15.57:35704) 2023-10-30T14:50:38.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.341+0000 7ff051153640 1 -- 172.21.15.57:0/2433631333 learned_addr learned my addr 172.21.15.57:0/2433631333 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:38.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.341+0000 7ff051153640 1 -- 172.21.15.57:0/2433631333 >> v1:172.21.15.88:6789/0 conn(0x7ff04c0cb8b0 legacy=0x7ff04c06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:38.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.341+0000 7ff051153640 1 -- 172.21.15.57:0/2433631333 >> v1:172.21.15.57:6789/0 conn(0x7ff04c151840 legacy=0x7ff04c072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:38.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.341+0000 7ff051153640 1 -- 172.21.15.57:0/2433631333 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04c157df0 con 0x7ff04c151c20 2023-10-30T14:50:38.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.345+0000 7ff051153640 1 --2- 172.21.15.57:0/2433631333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff04c151c20 0x7ff04c0c49a0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7ff03c009870 tx=0x7ff03c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7dc2c24d03e7b7b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:38.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.345+0000 7ff050952640 1 -- 172.21.15.57:0/2433631333 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0a8a30 con 0x7ff04c151c20 2023-10-30T14:50:38.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.345+0000 7ff050952640 1 -- 172.21.15.57:0/2433631333 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03c0a8b90 con 0x7ff04c151c20 2023-10-30T14:50:38.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.345+0000 7ff050952640 1 -- 172.21.15.57:0/2433631333 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0b0440 con 0x7ff04c151c20 2023-10-30T14:50:38.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.345+0000 7ff053bb8640 1 -- 172.21.15.57:0/2433631333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff04c151c20 msgr2=0x7ff04c0c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:38.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.345+0000 7ff053bb8640 1 --2- 172.21.15.57:0/2433631333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff04c151c20 0x7ff04c0c49a0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7ff03c009870 tx=0x7ff03c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:50:38.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.345+0000 7ff053bb8640 1 -- 172.21.15.57:0/2433631333 shutdown_connections 2023-10-30T14:50:38.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.345+0000 7ff053bb8640 1 --2- 172.21.15.57:0/2433631333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff04c151c20 0x7ff04c0c49a0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:38.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.345+0000 7ff053bb8640 1 -- 172.21.15.57:0/2433631333 >> 172.21.15.57:0/2433631333 conn(0x7ff04c0cac70 msgr2=0x7ff04c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:38.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.345+0000 7ff053bb8640 1 -- 172.21.15.57:0/2433631333 shutdown_connections 2023-10-30T14:50:38.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.345+0000 7ff053bb8640 1 -- 172.21.15.57:0/2433631333 wait complete. 2023-10-30T14:50:38.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff053bb8640 1 Processor -- start 2023-10-30T14:50:38.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff053bb8640 1 -- start start 2023-10-30T14:50:38.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff053bb8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff04c0cb8b0 0x7ff04c073620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:38.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff053bb8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff04c151840 0x7ff04c1ee550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:38.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff053bb8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff04c151c20 0x7ff04c147080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:38.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff053bb8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c1e3740 con 0x7ff04c0cb8b0 2023-10-30T14:50:38.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff053bb8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c1f1ac0 con 0x7ff04c151840 2023-10-30T14:50:38.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff053bb8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c1e3040 con 0x7ff04c151c20 2023-10-30T14:50:38.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff051954640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff04c0cb8b0 0x7ff04c073620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:38.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff051954640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff04c0cb8b0 0x7ff04c073620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35712/0 (socket says 172.21.15.57:35712) 2023-10-30T14:50:38.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff051954640 1 -- 172.21.15.57:0/1926118217 learned_addr learned my addr 172.21.15.57:0/1926118217 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:38.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff052155640 1 --2- 172.21.15.57:0/1926118217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff04c151c20 0x7ff04c147080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:38.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff051153640 1 --2- 172.21.15.57:0/1926118217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff04c151840 0x7ff04c1ee550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:38.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff051954640 1 -- 172.21.15.57:0/1926118217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff04c151c20 msgr2=0x7ff04c147080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:38.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff051954640 1 --2- 172.21.15.57:0/1926118217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff04c151c20 0x7ff04c147080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:38.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff051954640 1 -- 172.21.15.57:0/1926118217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff04c151840 msgr2=0x7ff04c1ee550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:38.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff051954640 1 --2- 172.21.15.57:0/1926118217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff04c151840 0x7ff04c1ee550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:38.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff051954640 1 -- 172.21.15.57:0/1926118217 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03c0094d0 con 0x7ff04c0cb8b0 2023-10-30T14:50:38.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff051954640 1 --2- 172.21.15.57:0/1926118217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff04c0cb8b0 0x7ff04c073620 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7ff04800c500 tx=0x7ff04800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:38.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff0427fc640 1 -- 172.21.15.57:0/1926118217 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff048005220 con 0x7ff04c0cb8b0 2023-10-30T14:50:38.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff0427fc640 1 -- 172.21.15.57:0/1926118217 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff048005380 con 0x7ff04c0cb8b0 2023-10-30T14:50:38.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff053bb8640 1 -- 172.21.15.57:0/1926118217 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff04c1e3920 con 0x7ff04c0cb8b0 2023-10-30T14:50:38.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff0427fc640 1 -- 172.21.15.57:0/1926118217 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff048007270 con 0x7ff04c0cb8b0 2023-10-30T14:50:38.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.349+0000 7ff053bb8640 1 -- 172.21.15.57:0/1926118217 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04c1f1d10 con 0x7ff04c0cb8b0 2023-10-30T14:50:38.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.353+0000 7ff053bb8640 1 -- 172.21.15.57:0/1926118217 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff04c04e170 con 0x7ff04c0cb8b0 2023-10-30T14:50:38.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.353+0000 7ff0427fc640 1 -- 172.21.15.57:0/1926118217 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff0480073d0 con 0x7ff04c0cb8b0 2023-10-30T14:50:38.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.353+0000 7ff0427fc640 1 --2- 172.21.15.57:0/1926118217 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0300664a0 0x7ff030068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:38.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.353+0000 7ff0427fc640 1 -- 172.21.15.57:0/1926118217 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7ff048083a60 con 0x7ff04c0cb8b0 2023-10-30T14:50:38.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.353+0000 7ff051153640 1 --2- 172.21.15.57:0/1926118217 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0300664a0 0x7ff030068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:38.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.353+0000 7ff051153640 1 --2- 172.21.15.57:0/1926118217 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0300664a0 0x7ff030068960 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0b2000 tx=0x7ff03c0aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:38.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.357+0000 7ff0427fc640 1 -- 172.21.15.57:0/1926118217 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff048050840 con 0x7ff04c0cb8b0 2023-10-30T14:50:38.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.553+0000 7ff053bb8640 1 -- 172.21.15.57:0/1926118217 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff04c06c680 con 0x7ff04c0cb8b0 2023-10-30T14:50:38.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.553+0000 7ff0427fc640 1 -- 172.21.15.57:0/1926118217 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+6502 (secure 0 0 0) 0x7ff048053e60 con 0x7ff04c0cb8b0 2023-10-30T14:50:38.556 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:50:38.556 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":8058,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:52.300180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:active","state_seq":30,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:50:38.557 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 184 2023-10-30T14:50:38.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.553+0000 7ff053bb8640 1 -- 172.21.15.57:0/1926118217 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0300664a0 msgr2=0x7ff030068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:38.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.553+0000 7ff053bb8640 1 --2- 172.21.15.57:0/1926118217 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0300664a0 0x7ff030068960 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0b2000 tx=0x7ff03c0aa070 comp rx=0 tx=0).stop 2023-10-30T14:50:38.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.553+0000 7ff053bb8640 1 -- 172.21.15.57:0/1926118217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff04c0cb8b0 msgr2=0x7ff04c073620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:38.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.553+0000 7ff053bb8640 1 --2- 172.21.15.57:0/1926118217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff04c0cb8b0 0x7ff04c073620 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7ff04800c500 tx=0x7ff04800c9c0 comp rx=0 tx=0).stop 2023-10-30T14:50:38.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.553+0000 7ff053bb8640 1 -- 172.21.15.57:0/1926118217 shutdown_connections 2023-10-30T14:50:38.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.553+0000 7ff053bb8640 1 --2- 172.21.15.57:0/1926118217 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ff0300664a0 0x7ff030068960 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:38.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.553+0000 7ff053bb8640 1 --2- 172.21.15.57:0/1926118217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff04c151c20 0x7ff04c147080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:38.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.553+0000 7ff053bb8640 1 --2- 172.21.15.57:0/1926118217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff04c151840 0x7ff04c1ee550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:38.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.553+0000 7ff053bb8640 1 --2- 172.21.15.57:0/1926118217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff04c0cb8b0 0x7ff04c073620 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:38.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.553+0000 7ff053bb8640 1 -- 172.21.15.57:0/1926118217 >> 172.21.15.57:0/1926118217 conn(0x7ff04c0cac70 msgr2=0x7ff04c0cb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:38.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.553+0000 7ff053bb8640 1 -- 172.21.15.57:0/1926118217 shutdown_connections 2023-10-30T14:50:38.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:38.553+0000 7ff053bb8640 1 -- 172.21.15.57:0/1926118217 wait complete. 2023-10-30T14:50:38.578 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:50:40.578 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:50:40.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.729+0000 7f76cc09c640 1 Processor -- start 2023-10-30T14:50:40.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.729+0000 7f76cc09c640 1 -- start start 2023-10-30T14:50:40.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.729+0000 7f76cc09c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f76c4150830 0x7f76c406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:40.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.729+0000 7f76cc09c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76c4070520 0x7f76c40709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:40.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.729+0000 7f76cc09c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76c40c2310 con 0x7f76c40cb7d0 2023-10-30T14:50:40.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.729+0000 7f76cc09c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76c4156710 con 0x7f76c4150830 2023-10-30T14:50:40.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.729+0000 7f76cc09c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76c40c4360 con 0x7f76c4070520 2023-10-30T14:50:40.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.733+0000 7f76ca639640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76c4070520 0x7f76c40709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:40.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.733+0000 7f76c9e38640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f76c4150830 0x7f76c406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:40.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.733+0000 7f76ca639640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76c4070520 0x7f76c40709b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:38156/0 (socket says 172.21.15.57:38156) 2023-10-30T14:50:40.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.733+0000 7f76ca639640 1 -- 172.21.15.57:0/4119356355 learned_addr learned my addr 172.21.15.57:0/4119356355 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:40.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.733+0000 7f76ca639640 1 -- 172.21.15.57:0/4119356355 >> v2:172.21.15.88:3300/0 conn(0x7f76c4150830 msgr2=0x7f76c406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:50:40.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.733+0000 7f76ca639640 1 --2- 172.21.15.57:0/4119356355 >> v2:172.21.15.88:3300/0 conn(0x7f76c4150830 0x7f76c406b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:40.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.733+0000 7f76ca639640 1 -- 172.21.15.57:0/4119356355 >> v1:172.21.15.88:6789/0 conn(0x7f76c40cb7d0 legacy=0x7f76c40c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:40.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.733+0000 7f76ca639640 1 -- 172.21.15.57:0/4119356355 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c41569b0 con 0x7f76c4070520 2023-10-30T14:50:40.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.733+0000 7f76c9e38640 1 --2- 172.21.15.57:0/4119356355 >> v2:172.21.15.88:3300/0 conn(0x7f76c4150830 0x7f76c406b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:50:40.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.733+0000 7f76ca639640 1 --2- 172.21.15.57:0/4119356355 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76c4070520 0x7f76c40709b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f76bc009e50 tx=0x7f76bc01f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98458fb41ca70dc4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:40.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76c8e36640 1 -- 172.21.15.57:0/4119356355 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76bc0af400 con 0x7f76c4070520 2023-10-30T14:50:40.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76c8e36640 1 -- 172.21.15.57:0/4119356355 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76bc0ab050 con 0x7f76c4070520 2023-10-30T14:50:40.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76c8e36640 1 -- 172.21.15.57:0/4119356355 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76bc0af780 con 0x7f76c4070520 2023-10-30T14:50:40.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76cc09c640 1 -- 172.21.15.57:0/4119356355 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76c4070520 msgr2=0x7f76c40709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:40.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76cc09c640 1 --2- 172.21.15.57:0/4119356355 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76c4070520 0x7f76c40709b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f76bc009e50 tx=0x7f76bc01f1f0 comp rx=0 tx=0).stop 2023-10-30T14:50:40.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76cc09c640 1 -- 172.21.15.57:0/4119356355 shutdown_connections 2023-10-30T14:50:40.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76cc09c640 1 --2- 172.21.15.57:0/4119356355 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76c4070520 0x7f76c40709b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:40.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76cc09c640 1 --2- 172.21.15.57:0/4119356355 >> v2:172.21.15.88:3300/0 conn(0x7f76c4150830 0x7f76c406b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:40.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76cc09c640 1 -- 172.21.15.57:0/4119356355 >> 172.21.15.57:0/4119356355 conn(0x7f76c40cac70 msgr2=0x7f76c40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:40.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76cc09c640 1 -- 172.21.15.57:0/4119356355 shutdown_connections 2023-10-30T14:50:40.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76cc09c640 1 -- 172.21.15.57:0/4119356355 wait complete. 2023-10-30T14:50:40.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76cc09c640 1 Processor -- start 2023-10-30T14:50:40.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76cc09c640 1 -- start start 2023-10-30T14:50:40.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76cc09c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76c4070520 0x7f76c40c6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:40.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76cc09c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76c40cb7d0 0x7f76c4155d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:40.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76cc09c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76c4150830 0x7f76c41440d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:40.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76cc09c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c4142f80 con 0x7f76c40cb7d0 2023-10-30T14:50:40.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76cc09c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c4155160 con 0x7f76c4070520 2023-10-30T14:50:40.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76cc09c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76c4155580 con 0x7f76c4150830 2023-10-30T14:50:40.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76c9637640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76c40cb7d0 0x7f76c4155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:40.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76c9e38640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76c4070520 0x7f76c40c6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:40.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76c9637640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76c40cb7d0 0x7f76c4155d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35726/0 (socket says 172.21.15.57:35726) 2023-10-30T14:50:40.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76c9637640 1 -- 172.21.15.57:0/3316303452 learned_addr learned my addr 172.21.15.57:0/3316303452 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:40.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.737+0000 7f76ca639640 1 --2- 172.21.15.57:0/3316303452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76c4150830 0x7f76c41440d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:40.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.741+0000 7f76c9637640 1 -- 172.21.15.57:0/3316303452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76c4150830 msgr2=0x7f76c41440d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:40.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.741+0000 7f76c9637640 1 --2- 172.21.15.57:0/3316303452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76c4150830 0x7f76c41440d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:40.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.741+0000 7f76c9637640 1 -- 172.21.15.57:0/3316303452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76c4070520 msgr2=0x7f76c40c6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:40.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.741+0000 7f76c9637640 1 --2- 172.21.15.57:0/3316303452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76c4070520 0x7f76c40c6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:40.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.741+0000 7f76c9637640 1 -- 172.21.15.57:0/3316303452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c0009420 con 0x7f76c40cb7d0 2023-10-30T14:50:40.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.741+0000 7f76c9637640 1 --2- 172.21.15.57:0/3316303452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76c40cb7d0 0x7f76c4155d90 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f76b400c420 tx=0x7f76b400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:40.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.741+0000 7f76ba7fc640 1 -- 172.21.15.57:0/3316303452 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b40052d0 con 0x7f76c40cb7d0 2023-10-30T14:50:40.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.741+0000 7f76ba7fc640 1 -- 172.21.15.57:0/3316303452 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b4005430 con 0x7f76c40cb7d0 2023-10-30T14:50:40.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.741+0000 7f76ba7fc640 1 -- 172.21.15.57:0/3316303452 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b4007260 con 0x7f76c40cb7d0 2023-10-30T14:50:40.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.741+0000 7f76cc09c640 1 -- 172.21.15.57:0/3316303452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76bc009b30 con 0x7f76c40cb7d0 2023-10-30T14:50:40.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.741+0000 7f76cc09c640 1 -- 172.21.15.57:0/3316303452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76c4142bd0 con 0x7f76c40cb7d0 2023-10-30T14:50:40.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.741+0000 7f76cc09c640 1 -- 172.21.15.57:0/3316303452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76c41552a0 con 0x7f76c40cb7d0 2023-10-30T14:50:40.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.741+0000 7f76ba7fc640 1 -- 172.21.15.57:0/3316303452 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f76b40073c0 con 0x7f76c40cb7d0 2023-10-30T14:50:40.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.741+0000 7f76ba7fc640 1 --2- 172.21.15.57:0/3316303452 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76a80664d0 0x7f76a8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:40.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.741+0000 7f76ba7fc640 1 -- 172.21.15.57:0/3316303452 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7f76b40836a0 con 0x7f76c40cb7d0 2023-10-30T14:50:40.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.745+0000 7f76c9e38640 1 --2- 172.21.15.57:0/3316303452 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76a80664d0 0x7f76a8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:40.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.745+0000 7f76c9e38640 1 --2- 172.21.15.57:0/3316303452 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76a80664d0 0x7f76a8068990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f76c0009c50 tx=0x7f76c0009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:40.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.745+0000 7f76ba7fc640 1 -- 172.21.15.57:0/3316303452 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f76c41552a0 con 0x7f76c40cb7d0 2023-10-30T14:50:40.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.937+0000 7f76cc09c640 1 -- 172.21.15.57:0/3316303452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76c4142bd0 con 0x7f76c40cb7d0 2023-10-30T14:50:40.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.937+0000 7f76ba7fc640 1 -- 172.21.15.57:0/3316303452 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+6502 (secure 0 0 0) 0x7f76c4142bd0 con 0x7f76c40cb7d0 2023-10-30T14:50:40.940 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:50:40.941 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":8058,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:52.300180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:active","state_seq":30,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:50:40.941 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 184 2023-10-30T14:50:40.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.937+0000 7f76cc09c640 1 -- 172.21.15.57:0/3316303452 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76a80664d0 msgr2=0x7f76a8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:40.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.937+0000 7f76cc09c640 1 --2- 172.21.15.57:0/3316303452 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76a80664d0 0x7f76a8068990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f76c0009c50 tx=0x7f76c0009190 comp rx=0 tx=0).stop 2023-10-30T14:50:40.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.937+0000 7f76cc09c640 1 -- 172.21.15.57:0/3316303452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76c40cb7d0 msgr2=0x7f76c4155d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:40.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.937+0000 7f76cc09c640 1 --2- 172.21.15.57:0/3316303452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76c40cb7d0 0x7f76c4155d90 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f76b400c420 tx=0x7f76b400c8e0 comp rx=0 tx=0).stop 2023-10-30T14:50:40.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.937+0000 7f76cc09c640 1 -- 172.21.15.57:0/3316303452 shutdown_connections 2023-10-30T14:50:40.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.937+0000 7f76cc09c640 1 --2- 172.21.15.57:0/3316303452 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f76a80664d0 0x7f76a8068990 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:40.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.937+0000 7f76cc09c640 1 --2- 172.21.15.57:0/3316303452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76c4150830 0x7f76c41440d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:40.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.937+0000 7f76cc09c640 1 --2- 172.21.15.57:0/3316303452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76c40cb7d0 0x7f76c4155d90 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:40.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.937+0000 7f76cc09c640 1 --2- 172.21.15.57:0/3316303452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76c4070520 0x7f76c40c6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:40.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.937+0000 7f76cc09c640 1 -- 172.21.15.57:0/3316303452 >> 172.21.15.57:0/3316303452 conn(0x7f76c40cac70 msgr2=0x7f76c40c3770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:40.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.937+0000 7f76cc09c640 1 -- 172.21.15.57:0/3316303452 shutdown_connections 2023-10-30T14:50:40.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:40.941+0000 7f76cc09c640 1 -- 172.21.15.57:0/3316303452 wait complete. 2023-10-30T14:50:40.962 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:50:41.972 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:50:41.977 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:50:42.963 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:50:43.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.113+0000 7f057df3e640 1 Processor -- start 2023-10-30T14:50:43.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.113+0000 7f057df3e640 1 -- start start 2023-10-30T14:50:43.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.117+0000 7f057df3e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0578153c10 0x7f0578065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:43.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.117+0000 7f057df3e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0578159970 0x7f0578159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:43.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.117+0000 7f057df3e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057806e390 con 0x7f0578070c30 2023-10-30T14:50:43.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.117+0000 7f057df3e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0578072390 con 0x7f0578159970 2023-10-30T14:50:43.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.117+0000 7f057df3e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05780660e0 con 0x7f0578153c10 2023-10-30T14:50:43.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.117+0000 7f0577fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0578159970 0x7f0578159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:43.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.117+0000 7f0577fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0578159970 0x7f0578159dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35738/0 (socket says 172.21.15.57:35738) 2023-10-30T14:50:43.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.117+0000 7f0577fff640 1 -- 172.21.15.57:0/1391282309 learned_addr learned my addr 172.21.15.57:0/1391282309 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:43.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.117+0000 7f05777fe640 1 --2- 172.21.15.57:0/1391282309 >> v2:172.21.15.88:3300/0 conn(0x7f0578153c10 0x7f0578065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:43.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.117+0000 7f0577fff640 1 -- 172.21.15.57:0/1391282309 >> v2:172.21.15.88:3300/0 conn(0x7f0578153c10 msgr2=0x7f0578065ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:50:43.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.117+0000 7f0577fff640 1 --2- 172.21.15.57:0/1391282309 >> v2:172.21.15.88:3300/0 conn(0x7f0578153c10 0x7f0578065ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:43.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.117+0000 7f0577fff640 1 -- 172.21.15.57:0/1391282309 >> v1:172.21.15.57:6789/0 conn(0x7f0578070c30 legacy=0x7f0578071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:43.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.117+0000 7f0577fff640 1 -- 172.21.15.57:0/1391282309 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05780bec10 con 0x7f0578159970 2023-10-30T14:50:43.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f0577fff640 1 --2- 172.21.15.57:0/1391282309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0578159970 0x7f0578159dc0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f05780c8e10 tx=0x7f057001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f2a7880ddf29290 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:43.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f05767fc640 1 -- 172.21.15.57:0/1391282309 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05700a64e0 con 0x7f0578159970 2023-10-30T14:50:43.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f05767fc640 1 -- 172.21.15.57:0/1391282309 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05700a6640 con 0x7f0578159970 2023-10-30T14:50:43.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f05767fc640 1 -- 172.21.15.57:0/1391282309 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05700a69d0 con 0x7f0578159970 2023-10-30T14:50:43.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f057df3e640 1 -- 172.21.15.57:0/1391282309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0578159970 msgr2=0x7f0578159dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:43.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f057df3e640 1 --2- 172.21.15.57:0/1391282309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0578159970 0x7f0578159dc0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f05780c8e10 tx=0x7f057001c940 comp rx=0 tx=0).stop 2023-10-30T14:50:43.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f057df3e640 1 -- 172.21.15.57:0/1391282309 shutdown_connections 2023-10-30T14:50:43.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f057df3e640 1 --2- 172.21.15.57:0/1391282309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0578159970 0x7f0578159dc0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:43.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f057df3e640 1 --2- 172.21.15.57:0/1391282309 >> v2:172.21.15.88:3300/0 conn(0x7f0578153c10 0x7f0578065ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:43.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f057df3e640 1 -- 172.21.15.57:0/1391282309 >> 172.21.15.57:0/1391282309 conn(0x7f05780c7f60 msgr2=0x7f05780c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:43.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f057df3e640 1 -- 172.21.15.57:0/1391282309 shutdown_connections 2023-10-30T14:50:43.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f057df3e640 1 -- 172.21.15.57:0/1391282309 wait complete. 2023-10-30T14:50:43.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f057df3e640 1 Processor -- start 2023-10-30T14:50:43.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f057df3e640 1 -- start start 2023-10-30T14:50:43.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f057df3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0578070c30 0x7f05780c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:43.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f057df3e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0578153c10 0x7f057814f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:43.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f057df3e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0578159970 0x7f057814fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:43.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f057df3e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0578143260 con 0x7f0578159970 2023-10-30T14:50:43.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f057df3e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0578141350 con 0x7f0578070c30 2023-10-30T14:50:43.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.121+0000 7f057df3e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0578141630 con 0x7f0578153c10 2023-10-30T14:50:43.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.125+0000 7f0577fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0578159970 0x7f057814fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:43.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.125+0000 7f0577fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0578159970 0x7f057814fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35742/0 (socket says 172.21.15.57:35742) 2023-10-30T14:50:43.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.125+0000 7f0577fff640 1 -- 172.21.15.57:0/191939540 learned_addr learned my addr 172.21.15.57:0/191939540 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:43.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.125+0000 7f05777fe640 1 --2- 172.21.15.57:0/191939540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0578070c30 0x7f05780c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:43.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.125+0000 7f0576ffd640 1 --2- 172.21.15.57:0/191939540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0578153c10 0x7f057814f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:43.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.125+0000 7f0577fff640 1 -- 172.21.15.57:0/191939540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0578153c10 msgr2=0x7f057814f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:43.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.125+0000 7f0577fff640 1 --2- 172.21.15.57:0/191939540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0578153c10 0x7f057814f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:43.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.125+0000 7f0577fff640 1 -- 172.21.15.57:0/191939540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0578070c30 msgr2=0x7f05780c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:43.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.125+0000 7f0577fff640 1 --2- 172.21.15.57:0/191939540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0578070c30 0x7f05780c9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:43.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.125+0000 7f0577fff640 1 -- 172.21.15.57:0/191939540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0570009b30 con 0x7f0578159970 2023-10-30T14:50:43.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.125+0000 7f0577fff640 1 --2- 172.21.15.57:0/191939540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0578159970 0x7f057814fb40 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f0570009e20 tx=0x7f05700196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:43.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.125+0000 7f055ffff640 1 -- 172.21.15.57:0/191939540 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05700b4070 con 0x7f0578159970 2023-10-30T14:50:43.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.125+0000 7f055ffff640 1 -- 172.21.15.57:0/191939540 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0570009150 con 0x7f0578159970 2023-10-30T14:50:43.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.125+0000 7f057df3e640 1 -- 172.21.15.57:0/191939540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0578142ba0 con 0x7f0578159970 2023-10-30T14:50:43.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.125+0000 7f055ffff640 1 -- 172.21.15.57:0/191939540 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05700a69f0 con 0x7f0578159970 2023-10-30T14:50:43.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.125+0000 7f057df3e640 1 -- 172.21.15.57:0/191939540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0578140d80 con 0x7f0578159970 2023-10-30T14:50:43.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.125+0000 7f057df3e640 1 -- 172.21.15.57:0/191939540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05781433a0 con 0x7f0578159970 2023-10-30T14:50:43.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.125+0000 7f055ffff640 1 -- 172.21.15.57:0/191939540 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f05700b8070 con 0x7f0578159970 2023-10-30T14:50:43.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.129+0000 7f055ffff640 1 --2- 172.21.15.57:0/191939540 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05640664a0 0x7f0564068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:43.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.129+0000 7f055ffff640 1 -- 172.21.15.57:0/191939540 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7f05700aa070 con 0x7f0578159970 2023-10-30T14:50:43.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.129+0000 7f05777fe640 1 --2- 172.21.15.57:0/191939540 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05640664a0 0x7f0564068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:43.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.129+0000 7f05777fe640 1 --2- 172.21.15.57:0/191939540 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05640664a0 0x7f0564068960 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f0568009b60 tx=0x7f0568009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:43.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.133+0000 7f055ffff640 1 -- 172.21.15.57:0/191939540 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f05781433a0 con 0x7f0578159970 2023-10-30T14:50:43.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.337+0000 7f057df3e640 1 -- 172.21.15.57:0/191939540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05781433a0 con 0x7f0578159970 2023-10-30T14:50:43.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.337+0000 7f055ffff640 1 -- 172.21.15.57:0/191939540 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+6502 (secure 0 0 0) 0x7f05781433a0 con 0x7f0578159970 2023-10-30T14:50:43.342 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:50:43.343 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":8058,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:52.300180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:active","state_seq":30,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:50:43.343 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 184 2023-10-30T14:50:43.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.341+0000 7f057df3e640 1 -- 172.21.15.57:0/191939540 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05640664a0 msgr2=0x7f0564068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:43.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.341+0000 7f057df3e640 1 --2- 172.21.15.57:0/191939540 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05640664a0 0x7f0564068960 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f0568009b60 tx=0x7f0568009190 comp rx=0 tx=0).stop 2023-10-30T14:50:43.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.341+0000 7f057df3e640 1 -- 172.21.15.57:0/191939540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0578159970 msgr2=0x7f057814fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:43.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.341+0000 7f057df3e640 1 --2- 172.21.15.57:0/191939540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0578159970 0x7f057814fb40 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f0570009e20 tx=0x7f05700196e0 comp rx=0 tx=0).stop 2023-10-30T14:50:43.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.341+0000 7f057df3e640 1 -- 172.21.15.57:0/191939540 shutdown_connections 2023-10-30T14:50:43.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.341+0000 7f057df3e640 1 --2- 172.21.15.57:0/191939540 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f05640664a0 0x7f0564068960 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:43.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.341+0000 7f057df3e640 1 --2- 172.21.15.57:0/191939540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0578159970 0x7f057814fb40 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:43.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.341+0000 7f057df3e640 1 --2- 172.21.15.57:0/191939540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0578153c10 0x7f057814f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:43.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.341+0000 7f057df3e640 1 --2- 172.21.15.57:0/191939540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0578070c30 0x7f05780c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:43.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.341+0000 7f057df3e640 1 -- 172.21.15.57:0/191939540 >> 172.21.15.57:0/191939540 conn(0x7f05780c7f60 msgr2=0x7f05780c6920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:43.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.341+0000 7f057df3e640 1 -- 172.21.15.57:0/191939540 shutdown_connections 2023-10-30T14:50:43.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:43.341+0000 7f057df3e640 1 -- 172.21.15.57:0/191939540 wait complete. 2023-10-30T14:50:43.364 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:50:45.366 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:50:45.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.517+0000 7fc3573ab640 1 Processor -- start 2023-10-30T14:50:45.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.517+0000 7fc3573ab640 1 -- start start 2023-10-30T14:50:45.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.517+0000 7fc3573ab640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc35006bb40 0x7fc350157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:45.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.517+0000 7fc3573ab640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc35006c420 0x7fc350157f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:45.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.517+0000 7fc3573ab640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc350071e00 con 0x7fc35006b760 2023-10-30T14:50:45.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.517+0000 7fc3573ab640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc350070020 con 0x7fc35006c420 2023-10-30T14:50:45.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.517+0000 7fc3573ab640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc350070f10 con 0x7fc35006bb40 2023-10-30T14:50:45.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.521+0000 7fc355948640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc35006c420 0x7fc350157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:45.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.521+0000 7fc355948640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc35006c420 0x7fc350157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34016/0 (socket says 172.21.15.57:34016) 2023-10-30T14:50:45.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.521+0000 7fc355948640 1 -- 172.21.15.57:0/2361589925 learned_addr learned my addr 172.21.15.57:0/2361589925 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:45.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.521+0000 7fc354946640 1 --2- 172.21.15.57:0/2361589925 >> v2:172.21.15.88:3300/0 conn(0x7fc35006bb40 0x7fc350157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:45.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.521+0000 7fc355948640 1 -- 172.21.15.57:0/2361589925 >> v2:172.21.15.88:3300/0 conn(0x7fc35006bb40 msgr2=0x7fc350157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:50:45.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.521+0000 7fc355948640 1 --2- 172.21.15.57:0/2361589925 >> v2:172.21.15.88:3300/0 conn(0x7fc35006bb40 0x7fc350157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:45.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.521+0000 7fc355948640 1 -- 172.21.15.57:0/2361589925 >> v1:172.21.15.88:6789/0 conn(0x7fc35006b760 legacy=0x7fc35015e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:45.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.521+0000 7fc355948640 1 -- 172.21.15.57:0/2361589925 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3500711f0 con 0x7fc35006c420 2023-10-30T14:50:45.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.521+0000 7fc355948640 1 --2- 172.21.15.57:0/2361589925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc35006c420 0x7fc350157f60 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fc35006fa40 tx=0x7fc34801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11337c1d51654c53 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:45.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.521+0000 7fc347fff640 1 -- 172.21.15.57:0/2361589925 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3480a5860 con 0x7fc35006c420 2023-10-30T14:50:45.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.521+0000 7fc347fff640 1 -- 172.21.15.57:0/2361589925 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3480a59c0 con 0x7fc35006c420 2023-10-30T14:50:45.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.521+0000 7fc347fff640 1 -- 172.21.15.57:0/2361589925 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3480a5d50 con 0x7fc35006c420 2023-10-30T14:50:45.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc3573ab640 1 -- 172.21.15.57:0/2361589925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc35006c420 msgr2=0x7fc350157f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:45.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc3573ab640 1 --2- 172.21.15.57:0/2361589925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc35006c420 0x7fc350157f60 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fc35006fa40 tx=0x7fc34801c9a0 comp rx=0 tx=0).stop 2023-10-30T14:50:45.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc3573ab640 1 -- 172.21.15.57:0/2361589925 shutdown_connections 2023-10-30T14:50:45.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc3573ab640 1 --2- 172.21.15.57:0/2361589925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc35006c420 0x7fc350157f60 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:45.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc3573ab640 1 --2- 172.21.15.57:0/2361589925 >> v2:172.21.15.88:3300/0 conn(0x7fc35006bb40 0x7fc350157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:45.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc3573ab640 1 -- 172.21.15.57:0/2361589925 >> 172.21.15.57:0/2361589925 conn(0x7fc3500cac70 msgr2=0x7fc3500cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:45.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc3573ab640 1 -- 172.21.15.57:0/2361589925 shutdown_connections 2023-10-30T14:50:45.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc3573ab640 1 -- 172.21.15.57:0/2361589925 wait complete. 2023-10-30T14:50:45.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc3573ab640 1 Processor -- start 2023-10-30T14:50:45.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc3573ab640 1 -- start start 2023-10-30T14:50:45.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc3573ab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc35006b760 0x7fc3500c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:45.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc3573ab640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc35006bb40 0x7fc350159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:45.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc3573ab640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc35006c420 0x7fc35015a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:45.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc3573ab640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc350159400 con 0x7fc35006b760 2023-10-30T14:50:45.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc3573ab640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3501f9e50 con 0x7fc35006c420 2023-10-30T14:50:45.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc3573ab640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3501fa130 con 0x7fc35006bb40 2023-10-30T14:50:45.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc355147640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc35006b760 0x7fc3500c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:45.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc355147640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc35006b760 0x7fc3500c3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34020/0 (socket says 172.21.15.57:34020) 2023-10-30T14:50:45.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc355147640 1 -- 172.21.15.57:0/3995416555 learned_addr learned my addr 172.21.15.57:0/3995416555 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:45.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc355948640 1 --2- 172.21.15.57:0/3995416555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc35006c420 0x7fc35015a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:45.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.525+0000 7fc354946640 1 --2- 172.21.15.57:0/3995416555 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc35006bb40 0x7fc350159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:45.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.529+0000 7fc355147640 1 -- 172.21.15.57:0/3995416555 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc35006bb40 msgr2=0x7fc350159b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:45.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.529+0000 7fc355147640 1 --2- 172.21.15.57:0/3995416555 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc35006bb40 0x7fc350159b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:45.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.529+0000 7fc355147640 1 -- 172.21.15.57:0/3995416555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc35006c420 msgr2=0x7fc35015a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:45.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.529+0000 7fc355147640 1 --2- 172.21.15.57:0/3995416555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc35006c420 0x7fc35015a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:45.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.529+0000 7fc355147640 1 -- 172.21.15.57:0/3995416555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc348009b30 con 0x7fc35006b760 2023-10-30T14:50:45.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.529+0000 7fc355147640 1 --2- 172.21.15.57:0/3995416555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc35006b760 0x7fc3500c3a20 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00c500 tx=0x7fc34c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:45.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.529+0000 7fc345ffb640 1 -- 172.21.15.57:0/3995416555 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c005220 con 0x7fc35006b760 2023-10-30T14:50:45.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.529+0000 7fc345ffb640 1 -- 172.21.15.57:0/3995416555 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc34c005380 con 0x7fc35006b760 2023-10-30T14:50:45.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.529+0000 7fc3573ab640 1 -- 172.21.15.57:0/3995416555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3501fa410 con 0x7fc35006b760 2023-10-30T14:50:45.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.529+0000 7fc345ffb640 1 -- 172.21.15.57:0/3995416555 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c007270 con 0x7fc35006b760 2023-10-30T14:50:45.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.529+0000 7fc3573ab640 1 -- 172.21.15.57:0/3995416555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35015ed90 con 0x7fc35006b760 2023-10-30T14:50:45.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.529+0000 7fc3573ab640 1 -- 172.21.15.57:0/3995416555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3501599e0 con 0x7fc35006b760 2023-10-30T14:50:45.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.529+0000 7fc345ffb640 1 -- 172.21.15.57:0/3995416555 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc34c0073d0 con 0x7fc35006b760 2023-10-30T14:50:45.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.529+0000 7fc345ffb640 1 --2- 172.21.15.57:0/3995416555 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc3340664a0 0x7fc334068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:45.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.529+0000 7fc345ffb640 1 -- 172.21.15.57:0/3995416555 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7fc34c083d00 con 0x7fc35006b760 2023-10-30T14:50:45.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.533+0000 7fc354946640 1 --2- 172.21.15.57:0/3995416555 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc3340664a0 0x7fc334068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:45.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.533+0000 7fc354946640 1 --2- 172.21.15.57:0/3995416555 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc3340664a0 0x7fc334068960 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc350073c70 tx=0x7fc340009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:45.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.533+0000 7fc345ffb640 1 -- 172.21.15.57:0/3995416555 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc34c050b70 con 0x7fc35006b760 2023-10-30T14:50:45.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.725+0000 7fc3573ab640 1 -- 172.21.15.57:0/3995416555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc350159540 con 0x7fc35006b760 2023-10-30T14:50:45.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.725+0000 7fc345ffb640 1 -- 172.21.15.57:0/3995416555 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+6502 (secure 0 0 0) 0x7fc34c054190 con 0x7fc35006b760 2023-10-30T14:50:45.730 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:50:45.730 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":8058,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:52.300180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:active","state_seq":30,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:50:45.730 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 184 2023-10-30T14:50:45.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.729+0000 7fc3573ab640 1 -- 172.21.15.57:0/3995416555 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc3340664a0 msgr2=0x7fc334068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:45.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.729+0000 7fc3573ab640 1 --2- 172.21.15.57:0/3995416555 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc3340664a0 0x7fc334068960 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc350073c70 tx=0x7fc340009190 comp rx=0 tx=0).stop 2023-10-30T14:50:45.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.729+0000 7fc3573ab640 1 -- 172.21.15.57:0/3995416555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc35006b760 msgr2=0x7fc3500c3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:45.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.729+0000 7fc3573ab640 1 --2- 172.21.15.57:0/3995416555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc35006b760 0x7fc3500c3a20 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00c500 tx=0x7fc34c00c9c0 comp rx=0 tx=0).stop 2023-10-30T14:50:45.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.729+0000 7fc3573ab640 1 -- 172.21.15.57:0/3995416555 shutdown_connections 2023-10-30T14:50:45.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.729+0000 7fc3573ab640 1 --2- 172.21.15.57:0/3995416555 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc3340664a0 0x7fc334068960 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:45.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.729+0000 7fc3573ab640 1 --2- 172.21.15.57:0/3995416555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc35006c420 0x7fc35015a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:45.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.729+0000 7fc3573ab640 1 --2- 172.21.15.57:0/3995416555 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc35006bb40 0x7fc350159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:45.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.729+0000 7fc3573ab640 1 --2- 172.21.15.57:0/3995416555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc35006b760 0x7fc3500c3a20 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:45.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.729+0000 7fc3573ab640 1 -- 172.21.15.57:0/3995416555 >> 172.21.15.57:0/3995416555 conn(0x7fc3500cac70 msgr2=0x7fc350072550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:45.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.729+0000 7fc3573ab640 1 -- 172.21.15.57:0/3995416555 shutdown_connections 2023-10-30T14:50:45.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:45.729+0000 7fc3573ab640 1 -- 172.21.15.57:0/3995416555 wait complete. 2023-10-30T14:50:45.751 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:50:47.752 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:50:47.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.905+0000 7fb9b2217640 1 Processor -- start 2023-10-30T14:50:47.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.905+0000 7fb9b2217640 1 -- start start 2023-10-30T14:50:47.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.905+0000 7fb9b2217640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb9ac1537f0 0x7fb9ac0701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:47.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.905+0000 7fb9b2217640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9ac15a2e0 0x7fb9ac0ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:47.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.905+0000 7fb9b2217640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac0c2aa0 con 0x7fb9ac0cd4a0 2023-10-30T14:50:47.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.905+0000 7fb9b2217640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9ac0c3ab0 con 0x7fb9ac1537f0 2023-10-30T14:50:47.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.905+0000 7fb9b2217640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ac0706e0 con 0x7fb9ac15a2e0 2023-10-30T14:50:47.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.905+0000 7fb9ab7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb9ac1537f0 0x7fb9ac0701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:47.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.905+0000 7fb9ab7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb9ac1537f0 0x7fb9ac0701a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34034/0 (socket says 172.21.15.57:34034) 2023-10-30T14:50:47.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.905+0000 7fb9ab7fe640 1 -- 172.21.15.57:0/2107939484 learned_addr learned my addr 172.21.15.57:0/2107939484 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:47.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.905+0000 7fb9aaffd640 1 --2- 172.21.15.57:0/2107939484 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9ac15a2e0 0x7fb9ac0ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:47.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.905+0000 7fb9ab7fe640 1 -- 172.21.15.57:0/2107939484 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9ac15a2e0 msgr2=0x7fb9ac0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:50:47.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.905+0000 7fb9ab7fe640 1 --2- 172.21.15.57:0/2107939484 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9ac15a2e0 0x7fb9ac0ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:47.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.905+0000 7fb9ab7fe640 1 -- 172.21.15.57:0/2107939484 >> v1:172.21.15.57:6789/0 conn(0x7fb9ac0cd4a0 legacy=0x7fb9ac06b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:47.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.905+0000 7fb9ab7fe640 1 -- 172.21.15.57:0/2107939484 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ac0708e0 con 0x7fb9ac1537f0 2023-10-30T14:50:47.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.909+0000 7fb9ab7fe640 1 --2- 172.21.15.57:0/2107939484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9ac1537f0 0x7fb9ac0701a0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fb99c009870 tx=0x7fb99c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e267f9cc26e55357 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:47.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.909+0000 7fb9aa7fc640 1 -- 172.21.15.57:0/2107939484 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99c0a8a30 con 0x7fb9ac1537f0 2023-10-30T14:50:47.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.909+0000 7fb9aa7fc640 1 -- 172.21.15.57:0/2107939484 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb99c0a8b90 con 0x7fb9ac1537f0 2023-10-30T14:50:47.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.909+0000 7fb9aa7fc640 1 -- 172.21.15.57:0/2107939484 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99c0b0440 con 0x7fb9ac1537f0 2023-10-30T14:50:47.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.909+0000 7fb9b2217640 1 -- 172.21.15.57:0/2107939484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9ac1537f0 msgr2=0x7fb9ac0701a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:47.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.909+0000 7fb9b2217640 1 --2- 172.21.15.57:0/2107939484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9ac1537f0 0x7fb9ac0701a0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fb99c009870 tx=0x7fb99c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:50:47.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.909+0000 7fb9b2217640 1 -- 172.21.15.57:0/2107939484 shutdown_connections 2023-10-30T14:50:47.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.909+0000 7fb9b2217640 1 --2- 172.21.15.57:0/2107939484 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9ac15a2e0 0x7fb9ac0ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:47.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.909+0000 7fb9b2217640 1 --2- 172.21.15.57:0/2107939484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9ac1537f0 0x7fb9ac0701a0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:47.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.909+0000 7fb9b2217640 1 -- 172.21.15.57:0/2107939484 >> 172.21.15.57:0/2107939484 conn(0x7fb9ac071fb0 msgr2=0x7fb9ac0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:47.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.909+0000 7fb9b2217640 1 -- 172.21.15.57:0/2107939484 shutdown_connections 2023-10-30T14:50:47.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.909+0000 7fb9b2217640 1 -- 172.21.15.57:0/2107939484 wait complete. 2023-10-30T14:50:47.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9b2217640 1 Processor -- start 2023-10-30T14:50:47.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9b2217640 1 -- start start 2023-10-30T14:50:47.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9b2217640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9ac0cd4a0 0x7fb9ac158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:47.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9b2217640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9ac1537f0 0x7fb9ac14c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:47.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9b2217640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9ac15a2e0 0x7fb9ac14c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:47.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9b2217640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ac0c7120 con 0x7fb9ac1537f0 2023-10-30T14:50:47.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9b2217640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ac158730 con 0x7fb9ac0cd4a0 2023-10-30T14:50:47.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9b2217640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ac159a30 con 0x7fb9ac15a2e0 2023-10-30T14:50:47.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9aaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9ac1537f0 0x7fb9ac14c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:47.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9aaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9ac1537f0 0x7fb9ac14c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34042/0 (socket says 172.21.15.57:34042) 2023-10-30T14:50:47.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9aaffd640 1 -- 172.21.15.57:0/137379173 learned_addr learned my addr 172.21.15.57:0/137379173 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:47.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9abfff640 1 --2- 172.21.15.57:0/137379173 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9ac15a2e0 0x7fb9ac14c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:47.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9ab7fe640 1 --2- 172.21.15.57:0/137379173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9ac0cd4a0 0x7fb9ac158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:47.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9aaffd640 1 -- 172.21.15.57:0/137379173 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9ac15a2e0 msgr2=0x7fb9ac14c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:47.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9aaffd640 1 --2- 172.21.15.57:0/137379173 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9ac15a2e0 0x7fb9ac14c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:47.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9aaffd640 1 -- 172.21.15.57:0/137379173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9ac0cd4a0 msgr2=0x7fb9ac158bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:47.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9aaffd640 1 --2- 172.21.15.57:0/137379173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9ac0cd4a0 0x7fb9ac158bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:47.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9aaffd640 1 -- 172.21.15.57:0/137379173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99c0094d0 con 0x7fb9ac1537f0 2023-10-30T14:50:47.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9ab7fe640 1 --2- 172.21.15.57:0/137379173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9ac0cd4a0 0x7fb9ac158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:50:47.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9aaffd640 1 --2- 172.21.15.57:0/137379173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9ac1537f0 0x7fb9ac14c0d0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fb9a000e930 tx=0x7fb9a000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:47.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb993fff640 1 -- 172.21.15.57:0/137379173 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a0009690 con 0x7fb9ac1537f0 2023-10-30T14:50:47.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb993fff640 1 -- 172.21.15.57:0/137379173 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a000ca00 con 0x7fb9ac1537f0 2023-10-30T14:50:47.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9b2217640 1 -- 172.21.15.57:0/137379173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9ac159100 con 0x7fb9ac1537f0 2023-10-30T14:50:47.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb993fff640 1 -- 172.21.15.57:0/137379173 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a000cd70 con 0x7fb9ac1537f0 2023-10-30T14:50:47.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9b2217640 1 -- 172.21.15.57:0/137379173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9ac159f70 con 0x7fb9ac1537f0 2023-10-30T14:50:47.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.913+0000 7fb9b2217640 1 -- 172.21.15.57:0/137379173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9ac04e170 con 0x7fb9ac1537f0 2023-10-30T14:50:47.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.917+0000 7fb993fff640 1 -- 172.21.15.57:0/137379173 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb9a0013470 con 0x7fb9ac1537f0 2023-10-30T14:50:47.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.917+0000 7fb993fff640 1 --2- 172.21.15.57:0/137379173 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb9980664a0 0x7fb998068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:47.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.917+0000 7fb993fff640 1 -- 172.21.15.57:0/137379173 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7287+0+0 (secure 0 0 0) 0x7fb9a0084aa0 con 0x7fb9ac1537f0 2023-10-30T14:50:47.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.917+0000 7fb9ab7fe640 1 --2- 172.21.15.57:0/137379173 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb9980664a0 0x7fb998068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:47.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.917+0000 7fb9ab7fe640 1 --2- 172.21.15.57:0/137379173 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb9980664a0 0x7fb998068960 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb99c0b2000 tx=0x7fb99c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:47.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:47.921+0000 7fb993fff640 1 -- 172.21.15.57:0/137379173 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb9a00519c0 con 0x7fb9ac1537f0 2023-10-30T14:50:48.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:48.113+0000 7fb9b2217640 1 -- 172.21.15.57:0/137379173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9ac152a00 con 0x7fb9ac1537f0 2023-10-30T14:50:48.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:48.113+0000 7fb993fff640 1 -- 172.21.15.57:0/137379173 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+6502 (secure 0 0 0) 0x7fb9a0054fe0 con 0x7fb9ac1537f0 2023-10-30T14:50:48.246 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:50:48.246 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":8058,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:48:52.300180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7911":{"gid":7911,"name":"c","rank":0,"incarnation":180,"state":"up:active","state_seq":30,"addr":"172.21.15.57:6836/2613636524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":2613636524},{"type":"v1","addr":"172.21.15.57:6836","nonce":2613636524}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:50:48.248 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 184 2023-10-30T14:50:48.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:48.113+0000 7fb9b2217640 1 -- 172.21.15.57:0/137379173 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb9980664a0 msgr2=0x7fb998068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:48.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:48.113+0000 7fb9b2217640 1 --2- 172.21.15.57:0/137379173 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb9980664a0 0x7fb998068960 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb99c0b2000 tx=0x7fb99c008040 comp rx=0 tx=0).stop 2023-10-30T14:50:48.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:48.113+0000 7fb9b2217640 1 -- 172.21.15.57:0/137379173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9ac1537f0 msgr2=0x7fb9ac14c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:48.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:48.113+0000 7fb9b2217640 1 --2- 172.21.15.57:0/137379173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9ac1537f0 0x7fb9ac14c0d0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fb9a000e930 tx=0x7fb9a000edf0 comp rx=0 tx=0).stop 2023-10-30T14:50:48.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:48.117+0000 7fb9b2217640 1 -- 172.21.15.57:0/137379173 shutdown_connections 2023-10-30T14:50:48.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:48.117+0000 7fb9b2217640 1 --2- 172.21.15.57:0/137379173 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb9980664a0 0x7fb998068960 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:48.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:48.117+0000 7fb9b2217640 1 --2- 172.21.15.57:0/137379173 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9ac15a2e0 0x7fb9ac14c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:48.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:48.117+0000 7fb9b2217640 1 --2- 172.21.15.57:0/137379173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9ac1537f0 0x7fb9ac14c0d0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:48.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:48.117+0000 7fb9b2217640 1 --2- 172.21.15.57:0/137379173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9ac0cd4a0 0x7fb9ac158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:48.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:48.117+0000 7fb9b2217640 1 -- 172.21.15.57:0/137379173 >> 172.21.15.57:0/137379173 conn(0x7fb9ac071fb0 msgr2=0x7fb9ac06c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:48.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:48.117+0000 7fb9b2217640 1 -- 172.21.15.57:0/137379173 shutdown_connections 2023-10-30T14:50:48.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:48.117+0000 7fb9b2217640 1 -- 172.21.15.57:0/137379173 wait complete. 2023-10-30T14:50:48.251 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:50:50.251 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:50:50.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.401+0000 7f99e4bd3640 1 Processor -- start 2023-10-30T14:50:50.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.401+0000 7f99e4bd3640 1 -- start start 2023-10-30T14:50:50.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.401+0000 7f99e4bd3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f99e0150830 0x7f99e006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:50.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.401+0000 7f99e4bd3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99e0070520 0x7f99e00709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:50.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.401+0000 7f99e4bd3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e00c2310 con 0x7f99e00cb7d0 2023-10-30T14:50:50.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.401+0000 7f99e4bd3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e0156710 con 0x7f99e0150830 2023-10-30T14:50:50.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.401+0000 7f99e4bd3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e00c4360 con 0x7f99e0070520 2023-10-30T14:50:50.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.401+0000 7f99de59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f99e0150830 0x7f99e006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:50.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.401+0000 7f99de59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f99e0150830 0x7f99e006b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34046/0 (socket says 172.21.15.57:34046) 2023-10-30T14:50:50.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.401+0000 7f99de59c640 1 -- 172.21.15.57:0/1610703924 learned_addr learned my addr 172.21.15.57:0/1610703924 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:50.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.401+0000 7f99ded9d640 1 --2- 172.21.15.57:0/1610703924 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99e0070520 0x7f99e00709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:50.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.401+0000 7f99de59c640 1 -- 172.21.15.57:0/1610703924 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99e0070520 msgr2=0x7f99e00709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:50:50.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.401+0000 7f99de59c640 1 --2- 172.21.15.57:0/1610703924 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99e0070520 0x7f99e00709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:50.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.401+0000 7f99de59c640 1 -- 172.21.15.57:0/1610703924 >> v1:172.21.15.57:6789/0 conn(0x7f99e00cb7d0 legacy=0x7f99e00c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:50.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.401+0000 7f99de59c640 1 -- 172.21.15.57:0/1610703924 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e01569b0 con 0x7f99e0150830 2023-10-30T14:50:50.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.405+0000 7f99de59c640 1 --2- 172.21.15.57:0/1610703924 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99e0150830 0x7f99e006b500 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f99d000bb10 tx=0x7f99d0021400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b60c45dce74f10cf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:50.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.405+0000 7f99ddd9b640 1 -- 172.21.15.57:0/1610703924 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d00b1600 con 0x7f99e0150830 2023-10-30T14:50:50.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.405+0000 7f99ddd9b640 1 -- 172.21.15.57:0/1610703924 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d00ad050 con 0x7f99e0150830 2023-10-30T14:50:50.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.405+0000 7f99ddd9b640 1 -- 172.21.15.57:0/1610703924 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d00b1980 con 0x7f99e0150830 2023-10-30T14:50:50.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.405+0000 7f99e4bd3640 1 -- 172.21.15.57:0/1610703924 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99e0150830 msgr2=0x7f99e006b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:50.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.405+0000 7f99e4bd3640 1 --2- 172.21.15.57:0/1610703924 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99e0150830 0x7f99e006b500 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f99d000bb10 tx=0x7f99d0021400 comp rx=0 tx=0).stop 2023-10-30T14:50:50.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.405+0000 7f99e4bd3640 1 -- 172.21.15.57:0/1610703924 shutdown_connections 2023-10-30T14:50:50.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.405+0000 7f99e4bd3640 1 --2- 172.21.15.57:0/1610703924 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99e0070520 0x7f99e00709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:50.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.405+0000 7f99e4bd3640 1 --2- 172.21.15.57:0/1610703924 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99e0150830 0x7f99e006b500 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:50.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.405+0000 7f99e4bd3640 1 -- 172.21.15.57:0/1610703924 >> 172.21.15.57:0/1610703924 conn(0x7f99e00cac70 msgr2=0x7f99e00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:50.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.405+0000 7f99e4bd3640 1 -- 172.21.15.57:0/1610703924 shutdown_connections 2023-10-30T14:50:50.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.405+0000 7f99e4bd3640 1 -- 172.21.15.57:0/1610703924 wait complete. 2023-10-30T14:50:50.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99e4bd3640 1 Processor -- start 2023-10-30T14:50:50.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99e4bd3640 1 -- start start 2023-10-30T14:50:50.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99e4bd3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99e0070520 0x7f99e01512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:50.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99e4bd3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99e00cb7d0 0x7f99e01e1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:50.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99e4bd3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99e0150830 0x7f99e01e1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:50.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99e4bd3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e0152500 con 0x7f99e0150830 2023-10-30T14:50:50.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99e4bd3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e0152da0 con 0x7f99e0070520 2023-10-30T14:50:50.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99e4bd3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e01e1680 con 0x7f99e00cb7d0 2023-10-30T14:50:50.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99ded9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99e0150830 0x7f99e01e1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:50.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99ded9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99e0150830 0x7f99e01e1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34054/0 (socket says 172.21.15.57:34054) 2023-10-30T14:50:50.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99ded9d640 1 -- 172.21.15.57:0/2454929800 learned_addr learned my addr 172.21.15.57:0/2454929800 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:50.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99de59c640 1 --2- 172.21.15.57:0/2454929800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99e0070520 0x7f99e01512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:50.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99d7fff640 1 --2- 172.21.15.57:0/2454929800 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99e00cb7d0 0x7f99e01e1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:50.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99ded9d640 1 -- 172.21.15.57:0/2454929800 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99e00cb7d0 msgr2=0x7f99e01e1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:50.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99ded9d640 1 --2- 172.21.15.57:0/2454929800 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99e00cb7d0 0x7f99e01e1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:50.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99ded9d640 1 -- 172.21.15.57:0/2454929800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99e0070520 msgr2=0x7f99e01512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:50.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99ded9d640 1 --2- 172.21.15.57:0/2454929800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99e0070520 0x7f99e01512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:50.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99ded9d640 1 -- 172.21.15.57:0/2454929800 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d000b770 con 0x7f99e0150830 2023-10-30T14:50:50.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99ded9d640 1 --2- 172.21.15.57:0/2454929800 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99e0150830 0x7f99e01e1cd0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009dd0 tx=0x7f99d800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:50.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99d6ffd640 1 -- 172.21.15.57:0/2454929800 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d8014480 con 0x7f99e0150830 2023-10-30T14:50:50.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99d6ffd640 1 -- 172.21.15.57:0/2454929800 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d800f040 con 0x7f99e0150830 2023-10-30T14:50:50.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99e4bd3640 1 -- 172.21.15.57:0/2454929800 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e0151950 con 0x7f99e0150830 2023-10-30T14:50:50.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99d6ffd640 1 -- 172.21.15.57:0/2454929800 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d8014840 con 0x7f99e0150830 2023-10-30T14:50:50.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.409+0000 7f99e4bd3640 1 -- 172.21.15.57:0/2454929800 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e0152750 con 0x7f99e0150830 2023-10-30T14:50:50.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.413+0000 7f99e4bd3640 1 -- 172.21.15.57:0/2454929800 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e0152f10 con 0x7f99e0150830 2023-10-30T14:50:50.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.413+0000 7f99d6ffd640 1 -- 172.21.15.57:0/2454929800 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f99d80149a0 con 0x7f99e0150830 2023-10-30T14:50:50.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.413+0000 7f99d6ffd640 1 --2- 172.21.15.57:0/2454929800 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f99c00664a0 0x7f99c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:50.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.413+0000 7f99d6ffd640 1 -- 172.21.15.57:0/2454929800 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f99d8085330 con 0x7f99e0150830 2023-10-30T14:50:50.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.413+0000 7f99de59c640 1 --2- 172.21.15.57:0/2454929800 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f99c00664a0 0x7f99c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:50.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.413+0000 7f99de59c640 1 --2- 172.21.15.57:0/2454929800 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f99c00664a0 0x7f99c0068960 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f99d00b3000 tx=0x7f99d00ac000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:50.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.417+0000 7f99d6ffd640 1 -- 172.21.15.57:0/2454929800 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f99d80521c0 con 0x7f99e0150830 2023-10-30T14:50:50.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.613+0000 7f99e4bd3640 1 -- 172.21.15.57:0/2454929800 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99e004e170 con 0x7f99e0150830 2023-10-30T14:50:50.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.613+0000 7f99d6ffd640 1 -- 172.21.15.57:0/2454929800 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+5709 (secure 0 0 0) 0x7f99d80557e0 con 0x7f99e0150830 2023-10-30T14:50:50.617 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:50:50.617 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:50:50.290551+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8058":{"gid":8058,"name":"e","rank":0,"incarnation":185,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:50:50.617 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 185 2023-10-30T14:50:50.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.613+0000 7f99e4bd3640 1 -- 172.21.15.57:0/2454929800 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f99c00664a0 msgr2=0x7f99c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:50.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.613+0000 7f99e4bd3640 1 --2- 172.21.15.57:0/2454929800 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f99c00664a0 0x7f99c0068960 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f99d00b3000 tx=0x7f99d00ac000 comp rx=0 tx=0).stop 2023-10-30T14:50:50.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.613+0000 7f99e4bd3640 1 -- 172.21.15.57:0/2454929800 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99e0150830 msgr2=0x7f99e01e1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:50.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.613+0000 7f99e4bd3640 1 --2- 172.21.15.57:0/2454929800 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99e0150830 0x7f99e01e1cd0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009dd0 tx=0x7f99d800c490 comp rx=0 tx=0).stop 2023-10-30T14:50:50.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.613+0000 7f99e4bd3640 1 -- 172.21.15.57:0/2454929800 shutdown_connections 2023-10-30T14:50:50.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.613+0000 7f99e4bd3640 1 --2- 172.21.15.57:0/2454929800 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f99c00664a0 0x7f99c0068960 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:50.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.613+0000 7f99e4bd3640 1 --2- 172.21.15.57:0/2454929800 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99e0150830 0x7f99e01e1cd0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:50.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.617+0000 7f99e4bd3640 1 --2- 172.21.15.57:0/2454929800 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99e00cb7d0 0x7f99e01e1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:50.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.617+0000 7f99e4bd3640 1 --2- 172.21.15.57:0/2454929800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99e0070520 0x7f99e01512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:50.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.617+0000 7f99e4bd3640 1 -- 172.21.15.57:0/2454929800 >> 172.21.15.57:0/2454929800 conn(0x7f99e00cac70 msgr2=0x7f99e00c6a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:50.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.617+0000 7f99e4bd3640 1 -- 172.21.15.57:0/2454929800 shutdown_connections 2023-10-30T14:50:50.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.617+0000 7f99e4bd3640 1 -- 172.21.15.57:0/2454929800 wait complete. 2023-10-30T14:50:50.638 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T14:50:50.638 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:50:50.639 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:50:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.777+0000 7efe82046640 1 Processor -- start 2023-10-30T14:50:50.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.777+0000 7efe82046640 1 -- start start 2023-10-30T14:50:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.781+0000 7efe82046640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7efe7c151840 0x7efe7c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.781+0000 7efe82046640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efe7c151c20 0x7efe7c072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.781+0000 7efe82046640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe7c0c5030 con 0x7efe7c0cb8b0 2023-10-30T14:50:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.781+0000 7efe82046640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe7c157830 con 0x7efe7c151840 2023-10-30T14:50:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.781+0000 7efe82046640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe7c157b10 con 0x7efe7c151c20 2023-10-30T14:50:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.781+0000 7efe7b7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7efe7c151840 0x7efe7c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.781+0000 7efe7b7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7efe7c151840 0x7efe7c0c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34062/0 (socket says 172.21.15.57:34062) 2023-10-30T14:50:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.781+0000 7efe7b7fe640 1 -- 172.21.15.57:0/1152846238 learned_addr learned my addr 172.21.15.57:0/1152846238 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.781+0000 7efe7bfff640 1 --1- 172.21.15.57:0/1152846238 >> v1:172.21.15.57:6789/0 conn(0x7efe7c0cb8b0 0x7efe7c06b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50410/0 (socket says 172.21.15.57:50410) 2023-10-30T14:50:50.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.781+0000 7efe7affd640 1 --2- 172.21.15.57:0/1152846238 >> v2:172.21.15.88:3300/0 conn(0x7efe7c151c20 0x7efe7c072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:50.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.781+0000 7efe7b7fe640 1 -- 172.21.15.57:0/1152846238 >> v2:172.21.15.88:3300/0 conn(0x7efe7c151c20 msgr2=0x7efe7c072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:50:50.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.781+0000 7efe7b7fe640 1 --2- 172.21.15.57:0/1152846238 >> v2:172.21.15.88:3300/0 conn(0x7efe7c151c20 0x7efe7c072000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:50.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.781+0000 7efe7b7fe640 1 -- 172.21.15.57:0/1152846238 >> v1:172.21.15.57:6789/0 conn(0x7efe7c0cb8b0 legacy=0x7efe7c06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:50.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.781+0000 7efe7b7fe640 1 -- 172.21.15.57:0/1152846238 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe7c157df0 con 0x7efe7c151840 2023-10-30T14:50:50.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe7b7fe640 1 --2- 172.21.15.57:0/1152846238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe7c151840 0x7efe7c0c49a0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7efe7c06d4d0 tx=0x7efe6c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71af27bc7df8aed8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:50.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe7a7fc640 1 -- 172.21.15.57:0/1152846238 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c0a89f0 con 0x7efe7c151840 2023-10-30T14:50:50.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe7a7fc640 1 -- 172.21.15.57:0/1152846238 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c0a8b50 con 0x7efe7c151840 2023-10-30T14:50:50.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe7a7fc640 1 -- 172.21.15.57:0/1152846238 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c0b0440 con 0x7efe7c151840 2023-10-30T14:50:50.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe82046640 1 -- 172.21.15.57:0/1152846238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe7c151840 msgr2=0x7efe7c0c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:50.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe82046640 1 --2- 172.21.15.57:0/1152846238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe7c151840 0x7efe7c0c49a0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7efe7c06d4d0 tx=0x7efe6c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:50:50.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe82046640 1 -- 172.21.15.57:0/1152846238 shutdown_connections 2023-10-30T14:50:50.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe82046640 1 --2- 172.21.15.57:0/1152846238 >> v2:172.21.15.88:3300/0 conn(0x7efe7c151c20 0x7efe7c072000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:50.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe82046640 1 --2- 172.21.15.57:0/1152846238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe7c151840 0x7efe7c0c49a0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:50.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe82046640 1 -- 172.21.15.57:0/1152846238 >> 172.21.15.57:0/1152846238 conn(0x7efe7c0cac70 msgr2=0x7efe7c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:50.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe82046640 1 -- 172.21.15.57:0/1152846238 shutdown_connections 2023-10-30T14:50:50.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe82046640 1 -- 172.21.15.57:0/1152846238 wait complete. 2023-10-30T14:50:50.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe82046640 1 Processor -- start 2023-10-30T14:50:50.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe82046640 1 -- start start 2023-10-30T14:50:50.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe82046640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe7c0cb8b0 0x7efe7c152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:50.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe82046640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efe7c151840 0x7efe7c06fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:50.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe82046640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe7c151c20 0x7efe7c0c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:50.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe82046640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c071ec0 con 0x7efe7c151c20 2023-10-30T14:50:50.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe82046640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c153010 con 0x7efe7c0cb8b0 2023-10-30T14:50:50.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.785+0000 7efe82046640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c071900 con 0x7efe7c151840 2023-10-30T14:50:50.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.789+0000 7efe7bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe7c151c20 0x7efe7c0c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:50.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.789+0000 7efe7bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe7c151c20 0x7efe7c0c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34070/0 (socket says 172.21.15.57:34070) 2023-10-30T14:50:50.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.789+0000 7efe7bfff640 1 -- 172.21.15.57:0/675893994 learned_addr learned my addr 172.21.15.57:0/675893994 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:50.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.789+0000 7efe7b7fe640 1 --2- 172.21.15.57:0/675893994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe7c0cb8b0 0x7efe7c152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:50.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.789+0000 7efe7affd640 1 --2- 172.21.15.57:0/675893994 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efe7c151840 0x7efe7c06fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:50.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.789+0000 7efe7bfff640 1 -- 172.21.15.57:0/675893994 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efe7c151840 msgr2=0x7efe7c06fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:50.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.789+0000 7efe7bfff640 1 --2- 172.21.15.57:0/675893994 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efe7c151840 0x7efe7c06fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:50.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.789+0000 7efe7bfff640 1 -- 172.21.15.57:0/675893994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe7c0cb8b0 msgr2=0x7efe7c152ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:50.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.789+0000 7efe7bfff640 1 --2- 172.21.15.57:0/675893994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe7c0cb8b0 0x7efe7c152ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:50.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.789+0000 7efe7bfff640 1 -- 172.21.15.57:0/675893994 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c0094d0 con 0x7efe7c151c20 2023-10-30T14:50:50.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.789+0000 7efe7bfff640 1 --2- 172.21.15.57:0/675893994 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe7c151c20 0x7efe7c0c7850 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7efe7400c7a0 tx=0x7efe7400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:50.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.789+0000 7efe63fff640 1 -- 172.21.15.57:0/675893994 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7400a640 con 0x7efe7c151c20 2023-10-30T14:50:50.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.789+0000 7efe63fff640 1 -- 172.21.15.57:0/675893994 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe7400a7a0 con 0x7efe7c151c20 2023-10-30T14:50:50.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.789+0000 7efe82046640 1 -- 172.21.15.57:0/675893994 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7c071be0 con 0x7efe7c151c20 2023-10-30T14:50:50.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.789+0000 7efe63fff640 1 -- 172.21.15.57:0/675893994 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7400ab10 con 0x7efe7c151c20 2023-10-30T14:50:50.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.789+0000 7efe82046640 1 -- 172.21.15.57:0/675893994 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7c1f24a0 con 0x7efe7c151c20 2023-10-30T14:50:50.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.789+0000 7efe82046640 1 -- 172.21.15.57:0/675893994 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe7c04e170 con 0x7efe7c151c20 2023-10-30T14:50:50.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.789+0000 7efe63fff640 1 -- 172.21.15.57:0/675893994 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efe7400ac70 con 0x7efe7c151c20 2023-10-30T14:50:50.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.793+0000 7efe63fff640 1 --2- 172.21.15.57:0/675893994 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efe680664d0 0x7efe68068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:50.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.793+0000 7efe63fff640 1 -- 172.21.15.57:0/675893994 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7efe74083cf0 con 0x7efe7c151c20 2023-10-30T14:50:50.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.793+0000 7efe7b7fe640 1 --2- 172.21.15.57:0/675893994 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efe680664d0 0x7efe68068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:50.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.793+0000 7efe7b7fe640 1 --2- 172.21.15.57:0/675893994 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efe680664d0 0x7efe68068990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7efe6c0b2000 tx=0x7efe6c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:50.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.797+0000 7efe63fff640 1 -- 172.21.15.57:0/675893994 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efe74050b00 con 0x7efe7c151c20 2023-10-30T14:50:50.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.997+0000 7efe82046640 1 -- 172.21.15.57:0/675893994 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe7c155060 con 0x7efe7c151c20 2023-10-30T14:50:50.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.997+0000 7efe63fff640 1 -- 172.21.15.57:0/675893994 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+5709 (secure 0 0 0) 0x7efe74017090 con 0x7efe7c151c20 2023-10-30T14:50:51.000 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:50:51.000 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:50:50.290551+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8058":{"gid":8058,"name":"e","rank":0,"incarnation":185,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:50:51.000 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 185 2023-10-30T14:50:51.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.997+0000 7efe82046640 1 -- 172.21.15.57:0/675893994 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efe680664d0 msgr2=0x7efe68068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:51.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.997+0000 7efe82046640 1 --2- 172.21.15.57:0/675893994 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efe680664d0 0x7efe68068990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7efe6c0b2000 tx=0x7efe6c009040 comp rx=0 tx=0).stop 2023-10-30T14:50:51.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.997+0000 7efe82046640 1 -- 172.21.15.57:0/675893994 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe7c151c20 msgr2=0x7efe7c0c7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:51.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.997+0000 7efe82046640 1 --2- 172.21.15.57:0/675893994 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe7c151c20 0x7efe7c0c7850 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7efe7400c7a0 tx=0x7efe7400cc60 comp rx=0 tx=0).stop 2023-10-30T14:50:51.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.997+0000 7efe82046640 1 -- 172.21.15.57:0/675893994 shutdown_connections 2023-10-30T14:50:51.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.997+0000 7efe82046640 1 --2- 172.21.15.57:0/675893994 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7efe680664d0 0x7efe68068990 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:51.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.997+0000 7efe82046640 1 --2- 172.21.15.57:0/675893994 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe7c151c20 0x7efe7c0c7850 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:51.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.997+0000 7efe82046640 1 --2- 172.21.15.57:0/675893994 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efe7c151840 0x7efe7c06fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:51.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.997+0000 7efe82046640 1 --2- 172.21.15.57:0/675893994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe7c0cb8b0 0x7efe7c152ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:51.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.997+0000 7efe82046640 1 -- 172.21.15.57:0/675893994 >> 172.21.15.57:0/675893994 conn(0x7efe7c0cac70 msgr2=0x7efe7c0cb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:51.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.997+0000 7efe82046640 1 -- 172.21.15.57:0/675893994 shutdown_connections 2023-10-30T14:50:51.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:50.997+0000 7efe82046640 1 -- 172.21.15.57:0/675893994 wait complete. 2023-10-30T14:50:51.022 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 185, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 185, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8058": { "addr": "172.21.15.57:6837/2594912006", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 2594912006, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 2594912006, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8058, "incarnation": 185, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:50:50.290551+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 8058 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6839/1078372885", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1078372885, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1078372885, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 170, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6716, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.57:6834/697507447", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 697507447, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 697507447, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 156, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7377, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:50:53.023 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:50:53.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.173+0000 7f33c6dd3640 1 Processor -- start 2023-10-30T14:50:53.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.173+0000 7f33c6dd3640 1 -- start start 2023-10-30T14:50:53.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.173+0000 7f33c6dd3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f33c0153c10 0x7f33c0065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:53.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.173+0000 7f33c6dd3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f33c0070c30 0x7f33c0071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:53.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.173+0000 7f33c6dd3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c006e390 con 0x7f33c0159970 2023-10-30T14:50:53.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.173+0000 7f33c6dd3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c0072390 con 0x7f33c0070c30 2023-10-30T14:50:53.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.173+0000 7f33c6dd3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c00660e0 con 0x7f33c0153c10 2023-10-30T14:50:53.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.173+0000 7f33bffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f33c0070c30 0x7f33c0071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:53.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.177+0000 7f33bffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f33c0070c30 0x7f33c0071070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34086/0 (socket says 172.21.15.57:34086) 2023-10-30T14:50:53.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.177+0000 7f33bffff640 1 -- 172.21.15.57:0/2080276892 learned_addr learned my addr 172.21.15.57:0/2080276892 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:53.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.177+0000 7f33c4b6f640 1 --2- 172.21.15.57:0/2080276892 >> v2:172.21.15.88:3300/0 conn(0x7f33c0153c10 0x7f33c0065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:53.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.177+0000 7f33bffff640 1 -- 172.21.15.57:0/2080276892 >> v2:172.21.15.88:3300/0 conn(0x7f33c0153c10 msgr2=0x7f33c0065ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:50:53.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.177+0000 7f33bffff640 1 --2- 172.21.15.57:0/2080276892 >> v2:172.21.15.88:3300/0 conn(0x7f33c0153c10 0x7f33c0065ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:53.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.177+0000 7f33bffff640 1 -- 172.21.15.57:0/2080276892 >> v1:172.21.15.88:6789/0 conn(0x7f33c0159970 legacy=0x7f33c0159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:53.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.177+0000 7f33bffff640 1 -- 172.21.15.57:0/2080276892 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c00715b0 con 0x7f33c0070c30 2023-10-30T14:50:53.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.177+0000 7f33bffff640 1 --2- 172.21.15.57:0/2080276892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33c0070c30 0x7f33c0071070 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f33c00c8e10 tx=0x7f33ac01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e72eaaa0b4d1332a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:53.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.177+0000 7f33bf7fe640 1 -- 172.21.15.57:0/2080276892 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac0ab600 con 0x7f33c0070c30 2023-10-30T14:50:53.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.177+0000 7f33bf7fe640 1 -- 172.21.15.57:0/2080276892 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ac0a7050 con 0x7f33c0070c30 2023-10-30T14:50:53.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.177+0000 7f33bf7fe640 1 -- 172.21.15.57:0/2080276892 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac0ab980 con 0x7f33c0070c30 2023-10-30T14:50:53.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c6dd3640 1 -- 172.21.15.57:0/2080276892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33c0070c30 msgr2=0x7f33c0071070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:53.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c6dd3640 1 --2- 172.21.15.57:0/2080276892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33c0070c30 0x7f33c0071070 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f33c00c8e10 tx=0x7f33ac01c470 comp rx=0 tx=0).stop 2023-10-30T14:50:53.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c6dd3640 1 -- 172.21.15.57:0/2080276892 shutdown_connections 2023-10-30T14:50:53.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c6dd3640 1 --2- 172.21.15.57:0/2080276892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33c0070c30 0x7f33c0071070 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:53.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c6dd3640 1 --2- 172.21.15.57:0/2080276892 >> v2:172.21.15.88:3300/0 conn(0x7f33c0153c10 0x7f33c0065ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:53.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c6dd3640 1 -- 172.21.15.57:0/2080276892 >> 172.21.15.57:0/2080276892 conn(0x7f33c00c7f60 msgr2=0x7f33c00c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:53.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c6dd3640 1 -- 172.21.15.57:0/2080276892 shutdown_connections 2023-10-30T14:50:53.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c6dd3640 1 -- 172.21.15.57:0/2080276892 wait complete. 2023-10-30T14:50:53.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c6dd3640 1 Processor -- start 2023-10-30T14:50:53.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c6dd3640 1 -- start start 2023-10-30T14:50:53.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c6dd3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33c0070c30 0x7f33c0150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:53.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c6dd3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33c0153c10 0x7f33c014e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:53.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c6dd3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33c0159970 0x7f33c0145e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:53.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c6dd3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c014f5c0 con 0x7f33c0070c30 2023-10-30T14:50:53.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c6dd3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c01f2630 con 0x7f33c0159970 2023-10-30T14:50:53.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c6dd3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c0157da0 con 0x7f33c0153c10 2023-10-30T14:50:53.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c4b6f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33c0070c30 0x7f33c0150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:53.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c4b6f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33c0070c30 0x7f33c0150b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34092/0 (socket says 172.21.15.57:34092) 2023-10-30T14:50:53.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c4b6f640 1 -- 172.21.15.57:0/4129361510 learned_addr learned my addr 172.21.15.57:0/4129361510 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:53.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33bffff640 1 --2- 172.21.15.57:0/4129361510 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33c0153c10 0x7f33c014e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:53.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c5370640 1 --2- 172.21.15.57:0/4129361510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33c0159970 0x7f33c0145e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:53.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c4b6f640 1 -- 172.21.15.57:0/4129361510 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33c0153c10 msgr2=0x7f33c014e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:53.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c4b6f640 1 --2- 172.21.15.57:0/4129361510 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33c0153c10 0x7f33c014e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:53.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c4b6f640 1 -- 172.21.15.57:0/4129361510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33c0159970 msgr2=0x7f33c0145e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:53.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c4b6f640 1 --2- 172.21.15.57:0/4129361510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33c0159970 0x7f33c0145e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:53.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33c4b6f640 1 -- 172.21.15.57:0/4129361510 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ac0094d0 con 0x7f33c0070c30 2023-10-30T14:50:53.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.181+0000 7f33bffff640 1 --2- 172.21.15.57:0/4129361510 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33c0153c10 0x7f33c014e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:50:53.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.185+0000 7f33c4b6f640 1 --2- 172.21.15.57:0/4129361510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33c0070c30 0x7f33c0150b00 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f33b400d800 tx=0x7f33b400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:53.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.185+0000 7f33bd7fa640 1 -- 172.21.15.57:0/4129361510 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b400b6c0 con 0x7f33c0070c30 2023-10-30T14:50:53.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.185+0000 7f33bd7fa640 1 -- 172.21.15.57:0/4129361510 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b400b820 con 0x7f33c0070c30 2023-10-30T14:50:53.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.185+0000 7f33c6dd3640 1 -- 172.21.15.57:0/4129361510 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c01f1cb0 con 0x7f33c0070c30 2023-10-30T14:50:53.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.185+0000 7f33bd7fa640 1 -- 172.21.15.57:0/4129361510 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b400bb90 con 0x7f33c0070c30 2023-10-30T14:50:53.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.185+0000 7f33c6dd3640 1 -- 172.21.15.57:0/4129361510 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c01f1f70 con 0x7f33c0070c30 2023-10-30T14:50:53.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.185+0000 7f33c6dd3640 1 -- 172.21.15.57:0/4129361510 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c01f28a0 con 0x7f33c0070c30 2023-10-30T14:50:53.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.185+0000 7f33bd7fa640 1 -- 172.21.15.57:0/4129361510 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f33b400bcf0 con 0x7f33c0070c30 2023-10-30T14:50:53.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.185+0000 7f33bd7fa640 1 --2- 172.21.15.57:0/4129361510 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f33a40664d0 0x7f33a4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:53.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.185+0000 7f33bd7fa640 1 -- 172.21.15.57:0/4129361510 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f33b4084a30 con 0x7f33c0070c30 2023-10-30T14:50:53.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.189+0000 7f33bffff640 1 --2- 172.21.15.57:0/4129361510 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f33a40664d0 0x7f33a4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:53.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.189+0000 7f33bffff640 1 --2- 172.21.15.57:0/4129361510 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f33a40664d0 0x7f33a4068990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f33ac0ad000 tx=0x7f33ac008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:53.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.189+0000 7f33bd7fa640 1 -- 172.21.15.57:0/4129361510 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f33b4051820 con 0x7f33c0070c30 2023-10-30T14:50:53.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.381+0000 7f33c6dd3640 1 -- 172.21.15.57:0/4129361510 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33c004e170 con 0x7f33c0070c30 2023-10-30T14:50:53.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.381+0000 7f33bd7fa640 1 -- 172.21.15.57:0/4129361510 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+5709 (secure 0 0 0) 0x7f33b4054e40 con 0x7f33c0070c30 2023-10-30T14:50:53.385 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:50:53.386 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:50:50.290551+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8058":{"gid":8058,"name":"e","rank":0,"incarnation":185,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:50:53.386 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 185 2023-10-30T14:50:53.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.385+0000 7f33c6dd3640 1 -- 172.21.15.57:0/4129361510 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f33a40664d0 msgr2=0x7f33a4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:53.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.385+0000 7f33c6dd3640 1 --2- 172.21.15.57:0/4129361510 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f33a40664d0 0x7f33a4068990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f33ac0ad000 tx=0x7f33ac008040 comp rx=0 tx=0).stop 2023-10-30T14:50:53.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.385+0000 7f33c6dd3640 1 -- 172.21.15.57:0/4129361510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33c0070c30 msgr2=0x7f33c0150b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:53.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.385+0000 7f33c6dd3640 1 --2- 172.21.15.57:0/4129361510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33c0070c30 0x7f33c0150b00 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f33b400d800 tx=0x7f33b400dcc0 comp rx=0 tx=0).stop 2023-10-30T14:50:53.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.385+0000 7f33c6dd3640 1 -- 172.21.15.57:0/4129361510 shutdown_connections 2023-10-30T14:50:53.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.385+0000 7f33c6dd3640 1 --2- 172.21.15.57:0/4129361510 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f33a40664d0 0x7f33a4068990 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:53.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.385+0000 7f33c6dd3640 1 --2- 172.21.15.57:0/4129361510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33c0159970 0x7f33c0145e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:53.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.385+0000 7f33c6dd3640 1 --2- 172.21.15.57:0/4129361510 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33c0153c10 0x7f33c014e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:53.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.385+0000 7f33c6dd3640 1 --2- 172.21.15.57:0/4129361510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33c0070c30 0x7f33c0150b00 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:53.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.385+0000 7f33c6dd3640 1 -- 172.21.15.57:0/4129361510 >> 172.21.15.57:0/4129361510 conn(0x7f33c00c7f60 msgr2=0x7f33c0159d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:53.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.385+0000 7f33c6dd3640 1 -- 172.21.15.57:0/4129361510 shutdown_connections 2023-10-30T14:50:53.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:53.385+0000 7f33c6dd3640 1 -- 172.21.15.57:0/4129361510 wait complete. 2023-10-30T14:50:53.419 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:50:55.419 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:50:55.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.569+0000 7f9545fd3640 1 Processor -- start 2023-10-30T14:50:55.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.569+0000 7f9545fd3640 1 -- start start 2023-10-30T14:50:55.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.569+0000 7f9545fd3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f95400c9b90 0x7f954006bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:55.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.569+0000 7f9545fd3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f954006c480 0x7f9540159d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:55.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.569+0000 7f9545fd3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f954015d070 con 0x7f9540153b10 2023-10-30T14:50:55.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.569+0000 7f9545fd3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9540159900 con 0x7f95400c9b90 2023-10-30T14:50:55.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.569+0000 7f9545fd3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95400ca000 con 0x7f954006c480 2023-10-30T14:50:55.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.569+0000 7f953effd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f95400c9b90 0x7f954006bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:55.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.569+0000 7f953f7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9540153b10 0x7f95400ca8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:56084/0 (socket says 172.21.15.57:56084) 2023-10-30T14:50:55.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.569+0000 7f953f7fe640 1 -- 172.21.15.57:0/934427619 learned_addr learned my addr 172.21.15.57:0/934427619 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:50:55.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.569+0000 7f953ffff640 1 --2- 172.21.15.57:0/934427619 >> v2:172.21.15.88:3300/0 conn(0x7f954006c480 0x7f9540159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:55.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.569+0000 7f953effd640 1 -- 172.21.15.57:0/934427619 >> v2:172.21.15.88:3300/0 conn(0x7f954006c480 msgr2=0x7f9540159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:50:55.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.569+0000 7f953effd640 1 --2- 172.21.15.57:0/934427619 >> v2:172.21.15.88:3300/0 conn(0x7f954006c480 0x7f9540159d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:55.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.569+0000 7f953effd640 1 -- 172.21.15.57:0/934427619 >> v1:172.21.15.88:6789/0 conn(0x7f9540153b10 legacy=0x7f95400ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:55.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.569+0000 7f953effd640 1 -- 172.21.15.57:0/934427619 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954015d350 con 0x7f95400c9b90 2023-10-30T14:50:55.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.569+0000 7f953e7fc640 1 -- 172.21.15.57:0/934427619 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387185114 0 0) 0x7f954015d070 con 0x7f9540153b10 2023-10-30T14:50:55.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.573+0000 7f953effd640 1 --2- 172.21.15.57:0/934427619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95400c9b90 0x7f954006bf40 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f954006bdd0 tx=0x7f953401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf00d24641d9ff0e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:55.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.573+0000 7f953e7fc640 1 -- 172.21.15.57:0/934427619 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95340ab600 con 0x7f95400c9b90 2023-10-30T14:50:55.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.573+0000 7f953e7fc640 1 -- 172.21.15.57:0/934427619 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95340a7050 con 0x7f95400c9b90 2023-10-30T14:50:55.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.573+0000 7f953e7fc640 1 -- 172.21.15.57:0/934427619 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95340ab980 con 0x7f95400c9b90 2023-10-30T14:50:55.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.573+0000 7f9545fd3640 1 -- 172.21.15.57:0/934427619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95400c9b90 msgr2=0x7f954006bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:55.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.573+0000 7f9545fd3640 1 --2- 172.21.15.57:0/934427619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95400c9b90 0x7f954006bf40 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f954006bdd0 tx=0x7f953401c3d0 comp rx=0 tx=0).stop 2023-10-30T14:50:55.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.573+0000 7f9545fd3640 1 -- 172.21.15.57:0/934427619 shutdown_connections 2023-10-30T14:50:55.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.573+0000 7f9545fd3640 1 --2- 172.21.15.57:0/934427619 >> v2:172.21.15.88:3300/0 conn(0x7f954006c480 0x7f9540159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:55.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.573+0000 7f9545fd3640 1 --2- 172.21.15.57:0/934427619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95400c9b90 0x7f954006bf40 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:55.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.573+0000 7f9545fd3640 1 -- 172.21.15.57:0/934427619 >> 172.21.15.57:0/934427619 conn(0x7f954006a240 msgr2=0x7f954006a650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:55.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.573+0000 7f9545fd3640 1 -- 172.21.15.57:0/934427619 shutdown_connections 2023-10-30T14:50:55.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.573+0000 7f9545fd3640 1 -- 172.21.15.57:0/934427619 wait complete. 2023-10-30T14:50:55.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f9545fd3640 1 Processor -- start 2023-10-30T14:50:55.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f9545fd3640 1 -- start start 2023-10-30T14:50:55.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f9545fd3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f954006c480 0x7f9540149320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:55.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f9545fd3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95400c9b90 0x7f9540147b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:55.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f9545fd3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9540153b10 0x7f95401480b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:55.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f9545fd3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95401f2c20 con 0x7f95400c9b90 2023-10-30T14:50:55.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f9545fd3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95401f2680 con 0x7f9540153b10 2023-10-30T14:50:55.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f9545fd3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95401f2940 con 0x7f954006c480 2023-10-30T14:50:55.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f953effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95400c9b90 0x7f9540147b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:55.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f953effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95400c9b90 0x7f9540147b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50046/0 (socket says 172.21.15.57:50046) 2023-10-30T14:50:55.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f953effd640 1 -- 172.21.15.57:0/3460073138 learned_addr learned my addr 172.21.15.57:0/3460073138 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:55.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f953f7fe640 1 --2- 172.21.15.57:0/3460073138 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f954006c480 0x7f9540149320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:55.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f953ffff640 1 --2- 172.21.15.57:0/3460073138 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9540153b10 0x7f95401480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:55.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f953effd640 1 -- 172.21.15.57:0/3460073138 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f954006c480 msgr2=0x7f9540149320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:55.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f953effd640 1 --2- 172.21.15.57:0/3460073138 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f954006c480 0x7f9540149320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:55.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f953effd640 1 -- 172.21.15.57:0/3460073138 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9540153b10 msgr2=0x7f95401480b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:55.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f953effd640 1 --2- 172.21.15.57:0/3460073138 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9540153b10 0x7f95401480b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:55.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f953effd640 1 -- 172.21.15.57:0/3460073138 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9534009420 con 0x7f95400c9b90 2023-10-30T14:50:55.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f953effd640 1 --2- 172.21.15.57:0/3460073138 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95400c9b90 0x7f9540147b70 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f95340ae040 tx=0x7f9534019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:55.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f9527fff640 1 -- 172.21.15.57:0/3460073138 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95340abe90 con 0x7f95400c9b90 2023-10-30T14:50:55.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f9527fff640 1 -- 172.21.15.57:0/3460073138 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95340b4430 con 0x7f95400c9b90 2023-10-30T14:50:55.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f9527fff640 1 -- 172.21.15.57:0/3460073138 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95340b4800 con 0x7f95400c9b90 2023-10-30T14:50:55.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.577+0000 7f9545fd3640 1 -- 172.21.15.57:0/3460073138 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95401470d0 con 0x7f95400c9b90 2023-10-30T14:50:55.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.581+0000 7f9545fd3640 1 -- 172.21.15.57:0/3460073138 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95400b27a0 con 0x7f95400c9b90 2023-10-30T14:50:55.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.581+0000 7f9545fd3640 1 -- 172.21.15.57:0/3460073138 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9540146f30 con 0x7f95400c9b90 2023-10-30T14:50:55.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.581+0000 7f9527fff640 1 -- 172.21.15.57:0/3460073138 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f95340b2040 con 0x7f95400c9b90 2023-10-30T14:50:55.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.581+0000 7f9527fff640 1 --2- 172.21.15.57:0/3460073138 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f952c0664a0 0x7f952c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:55.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.581+0000 7f9527fff640 1 -- 172.21.15.57:0/3460073138 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f953411e160 con 0x7f95400c9b90 2023-10-30T14:50:55.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.581+0000 7f953f7fe640 1 --2- 172.21.15.57:0/3460073138 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f952c0664a0 0x7f952c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:55.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.585+0000 7f953f7fe640 1 --2- 172.21.15.57:0/3460073138 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f952c0664a0 0x7f952c068960 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f9530000f70 tx=0x7f9530007df0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:55.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.585+0000 7f9527fff640 1 -- 172.21.15.57:0/3460073138 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9540146f30 con 0x7f95400c9b90 2023-10-30T14:50:55.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.793+0000 7f9545fd3640 1 -- 172.21.15.57:0/3460073138 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9540146f30 con 0x7f95400c9b90 2023-10-30T14:50:55.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.793+0000 7f9527fff640 1 -- 172.21.15.57:0/3460073138 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+5709 (secure 0 0 0) 0x7f9540146f30 con 0x7f95400c9b90 2023-10-30T14:50:55.798 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:50:55.798 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:50:50.290551+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8058":{"gid":8058,"name":"e","rank":0,"incarnation":185,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:50:55.799 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 185 2023-10-30T14:50:55.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.797+0000 7f9545fd3640 1 -- 172.21.15.57:0/3460073138 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f952c0664a0 msgr2=0x7f952c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:55.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.797+0000 7f9545fd3640 1 --2- 172.21.15.57:0/3460073138 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f952c0664a0 0x7f952c068960 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f9530000f70 tx=0x7f9530007df0 comp rx=0 tx=0).stop 2023-10-30T14:50:55.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.797+0000 7f9545fd3640 1 -- 172.21.15.57:0/3460073138 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95400c9b90 msgr2=0x7f9540147b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:55.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.797+0000 7f9545fd3640 1 --2- 172.21.15.57:0/3460073138 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95400c9b90 0x7f9540147b70 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f95340ae040 tx=0x7f9534019170 comp rx=0 tx=0).stop 2023-10-30T14:50:55.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.797+0000 7f9545fd3640 1 -- 172.21.15.57:0/3460073138 shutdown_connections 2023-10-30T14:50:55.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.797+0000 7f9545fd3640 1 --2- 172.21.15.57:0/3460073138 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f952c0664a0 0x7f952c068960 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:55.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.797+0000 7f9545fd3640 1 --2- 172.21.15.57:0/3460073138 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9540153b10 0x7f95401480b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:55.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.797+0000 7f9545fd3640 1 --2- 172.21.15.57:0/3460073138 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95400c9b90 0x7f9540147b70 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:55.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.797+0000 7f9545fd3640 1 --2- 172.21.15.57:0/3460073138 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f954006c480 0x7f9540149320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:55.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.797+0000 7f9545fd3640 1 -- 172.21.15.57:0/3460073138 >> 172.21.15.57:0/3460073138 conn(0x7f954006a240 msgr2=0x7f954006a620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:55.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.797+0000 7f9545fd3640 1 -- 172.21.15.57:0/3460073138 shutdown_connections 2023-10-30T14:50:55.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:55.797+0000 7f9545fd3640 1 -- 172.21.15.57:0/3460073138 wait complete. 2023-10-30T14:50:55.820 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:50:57.821 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:50:57.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.965+0000 7f4b2709a640 1 Processor -- start 2023-10-30T14:50:57.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.965+0000 7f4b2709a640 1 -- start start 2023-10-30T14:50:57.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b2709a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b2015cf90 0x7f4b200c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:57.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b2709a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b20159f20 con 0x7f4b200c4a10 2023-10-30T14:50:57.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b1ffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f4b200c4a10 0x7f4b200ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:33836/0 (socket says 172.21.15.57:33836) 2023-10-30T14:50:57.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b1ffff640 1 -- 172.21.15.57:0/1242268683 learned_addr learned my addr 172.21.15.57:0/1242268683 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:50:57.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b2709a640 1 -- 172.21.15.57:0/1242268683 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b200c47e0 con 0x7f4b20153c80 2023-10-30T14:50:57.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b2709a640 1 -- 172.21.15.57:0/1242268683 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2015dec0 con 0x7f4b2015cf90 2023-10-30T14:50:57.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b25637640 1 --2- 172.21.15.57:0/1242268683 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b2015cf90 0x7f4b200c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:57.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b1f7fe640 1 -- 172.21.15.57:0/1242268683 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598327847 0 0) 0x7f4b20159f20 con 0x7f4b200c4a10 2023-10-30T14:50:57.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b1f7fe640 1 -- 172.21.15.57:0/1242268683 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b10003530 con 0x7f4b200c4a10 2023-10-30T14:50:57.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b1f7fe640 1 -- 172.21.15.57:0/1242268683 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 353998439 0 0) 0x7f4b10003530 con 0x7f4b200c4a10 2023-10-30T14:50:57.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b1f7fe640 1 -- 172.21.15.57:0/1242268683 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b2015cf90 msgr2=0x7f4b200c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:50:57.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b1f7fe640 1 --2- 172.21.15.57:0/1242268683 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b2015cf90 0x7f4b200c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:57.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b1f7fe640 1 -- 172.21.15.57:0/1242268683 >> v1:172.21.15.88:6789/0 conn(0x7f4b20153c80 legacy=0x7f4b2015a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:57.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b1f7fe640 1 -- 172.21.15.57:0/1242268683 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2015d620 con 0x7f4b200c4a10 2023-10-30T14:50:57.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b1f7fe640 1 -- 172.21.15.57:0/1242268683 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f4b0c001680 con 0x7f4b200c4a10 2023-10-30T14:50:57.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b1f7fe640 1 -- 172.21.15.57:0/1242268683 >> v1:172.21.15.57:6789/0 conn(0x7f4b200c4a10 legacy=0x7f4b200ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:57.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b1f7fe640 1 --2- 172.21.15.57:0/1242268683 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b2015cf90 0x7f4b100039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:57.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b1f7fe640 1 --2- 172.21.15.57:0/1242268683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b10003ef0 0x7f4b100047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:57.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b1f7fe640 1 --2- 172.21.15.57:0/1242268683 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b10004d10 0x7f4b10005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:57.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b1f7fe640 1 -- 172.21.15.57:0/1242268683 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2015dec0 con 0x7f4b10004d10 2023-10-30T14:50:57.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b1f7fe640 1 -- 172.21.15.57:0/1242268683 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1000b9f0 con 0x7f4b10003ef0 2023-10-30T14:50:57.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b1f7fe640 1 -- 172.21.15.57:0/1242268683 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1000bbb0 con 0x7f4b2015cf90 2023-10-30T14:50:57.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b24e36640 1 --2- 172.21.15.57:0/1242268683 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b2015cf90 0x7f4b100039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:57.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b25637640 1 --2- 172.21.15.57:0/1242268683 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b10004d10 0x7f4b10005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:57.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b1ffff640 1 --2- 172.21.15.57:0/1242268683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b10003ef0 0x7f4b100047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:57.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b25637640 1 -- 172.21.15.57:0/1242268683 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b2015cf90 msgr2=0x7f4b100039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:50:57.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b25637640 1 --2- 172.21.15.57:0/1242268683 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b2015cf90 0x7f4b100039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:57.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b25637640 1 -- 172.21.15.57:0/1242268683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b10003ef0 msgr2=0x7f4b100047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:50:57.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b25637640 1 --2- 172.21.15.57:0/1242268683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b10003ef0 0x7f4b100047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:57.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.969+0000 7f4b25637640 1 -- 172.21.15.57:0/1242268683 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b10003530 con 0x7f4b10004d10 2023-10-30T14:50:57.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.973+0000 7f4b25637640 1 --2- 172.21.15.57:0/1242268683 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b10004d10 0x7f4b10005190 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f4b18009fc0 tx=0x7f4b1801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f59d799894c8254 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:57.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.973+0000 7f4b1f7fe640 1 -- 172.21.15.57:0/1242268683 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b180a5c70 con 0x7f4b10004d10 2023-10-30T14:50:57.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.973+0000 7f4b1f7fe640 1 -- 172.21.15.57:0/1242268683 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1800f040 con 0x7f4b10004d10 2023-10-30T14:50:57.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.973+0000 7f4b1f7fe640 1 -- 172.21.15.57:0/1242268683 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b180ad440 con 0x7f4b10004d10 2023-10-30T14:50:57.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.973+0000 7f4b2709a640 1 -- 172.21.15.57:0/1242268683 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b10004d10 msgr2=0x7f4b10005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:57.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.973+0000 7f4b2709a640 1 --2- 172.21.15.57:0/1242268683 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b10004d10 0x7f4b10005190 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f4b18009fc0 tx=0x7f4b1801cc60 comp rx=0 tx=0).stop 2023-10-30T14:50:57.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.973+0000 7f4b25637640 1 -- 172.21.15.57:0/1242268683 reap_dead start 2023-10-30T14:50:57.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.973+0000 7f4b2709a640 1 -- 172.21.15.57:0/1242268683 shutdown_connections 2023-10-30T14:50:57.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.973+0000 7f4b2709a640 1 -- 172.21.15.57:0/1242268683 >> 172.21.15.57:0/1242268683 conn(0x7f4b200715c0 msgr2=0x7f4b200719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:57.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.973+0000 7f4b2709a640 1 -- 172.21.15.57:0/1242268683 shutdown_connections 2023-10-30T14:50:57.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.973+0000 7f4b2709a640 1 -- 172.21.15.57:0/1242268683 wait complete. 2023-10-30T14:50:57.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.977+0000 7f4b2709a640 1 Processor -- start 2023-10-30T14:50:57.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.977+0000 7f4b2709a640 1 -- start start 2023-10-30T14:50:57.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.977+0000 7f4b2709a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b10004d10 0x7f4b200c9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:57.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.977+0000 7f4b2709a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b200c5c50 0x7f4b200ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:57.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.977+0000 7f4b2709a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b201f1dd0 0x7f4b200ce700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:57.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.977+0000 7f4b2709a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b201ee860 con 0x7f4b10004d10 2023-10-30T14:50:57.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.977+0000 7f4b2709a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b20147d30 con 0x7f4b201f1dd0 2023-10-30T14:50:57.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.977+0000 7f4b2709a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b201eea70 con 0x7f4b200c5c50 2023-10-30T14:50:57.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.977+0000 7f4b24e36640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b10004d10 0x7f4b200c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:57.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.981+0000 7f4b24e36640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b10004d10 0x7f4b200c9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50070/0 (socket says 172.21.15.57:50070) 2023-10-30T14:50:57.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.981+0000 7f4b24e36640 1 -- 172.21.15.57:0/1488180375 learned_addr learned my addr 172.21.15.57:0/1488180375 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:50:57.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.981+0000 7f4b1ffff640 1 --2- 172.21.15.57:0/1488180375 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b200c5c50 0x7f4b200ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:57.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.981+0000 7f4b25637640 1 --2- 172.21.15.57:0/1488180375 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b201f1dd0 0x7f4b200ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:57.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.981+0000 7f4b24e36640 1 -- 172.21.15.57:0/1488180375 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b200c5c50 msgr2=0x7f4b200ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:57.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.981+0000 7f4b24e36640 1 --2- 172.21.15.57:0/1488180375 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b200c5c50 0x7f4b200ce1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:57.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.981+0000 7f4b24e36640 1 -- 172.21.15.57:0/1488180375 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b201f1dd0 msgr2=0x7f4b200ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:57.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.981+0000 7f4b24e36640 1 --2- 172.21.15.57:0/1488180375 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b201f1dd0 0x7f4b200ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:57.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.981+0000 7f4b24e36640 1 -- 172.21.15.57:0/1488180375 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b201eed50 con 0x7f4b10004d10 2023-10-30T14:50:57.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.981+0000 7f4b24e36640 1 --2- 172.21.15.57:0/1488180375 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b10004d10 0x7f4b200c9010 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f4b1400f7a0 tx=0x7f4b1400fba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:57.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.981+0000 7f4b1d7fa640 1 -- 172.21.15.57:0/1488180375 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b140050d0 con 0x7f4b10004d10 2023-10-30T14:50:57.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.981+0000 7f4b1d7fa640 1 -- 172.21.15.57:0/1488180375 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b14005230 con 0x7f4b10004d10 2023-10-30T14:50:57.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.981+0000 7f4b2709a640 1 -- 172.21.15.57:0/1488180375 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b201ef030 con 0x7f4b10004d10 2023-10-30T14:50:57.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.981+0000 7f4b1d7fa640 1 -- 172.21.15.57:0/1488180375 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1400d730 con 0x7f4b10004d10 2023-10-30T14:50:57.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.981+0000 7f4b2709a640 1 -- 172.21.15.57:0/1488180375 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b201ef5f0 con 0x7f4b10004d10 2023-10-30T14:50:57.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.981+0000 7f4b2709a640 1 -- 172.21.15.57:0/1488180375 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b201ef7b0 con 0x7f4b10004d10 2023-10-30T14:50:57.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.985+0000 7f4b1d7fa640 1 -- 172.21.15.57:0/1488180375 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4b1400d890 con 0x7f4b10004d10 2023-10-30T14:50:57.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.985+0000 7f4b1d7fa640 1 --2- 172.21.15.57:0/1488180375 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4b040664d0 0x7f4b04068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:50:57.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.985+0000 7f4b1d7fa640 1 -- 172.21.15.57:0/1488180375 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f4b140843d0 con 0x7f4b10004d10 2023-10-30T14:50:57.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.985+0000 7f4b1ffff640 1 --2- 172.21.15.57:0/1488180375 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4b040664d0 0x7f4b04068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:50:57.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.985+0000 7f4b1ffff640 1 --2- 172.21.15.57:0/1488180375 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4b040664d0 0x7f4b04068990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4b201597e0 tx=0x7f4b0c000ee0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:50:57.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:57.993+0000 7f4b1d7fa640 1 -- 172.21.15.57:0/1488180375 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4b140511d0 con 0x7f4b10004d10 2023-10-30T14:50:58.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:58.193+0000 7f4b2709a640 1 -- 172.21.15.57:0/1488180375 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b2004e170 con 0x7f4b10004d10 2023-10-30T14:50:58.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:58.193+0000 7f4b1d7fa640 1 -- 172.21.15.57:0/1488180375 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+5709 (secure 0 0 0) 0x7f4b140547f0 con 0x7f4b10004d10 2023-10-30T14:50:58.195 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:50:58.196 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:50:50.290551+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8058":{"gid":8058,"name":"e","rank":0,"incarnation":185,"state":"up:replay","state_seq":1,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:50:58.196 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 185 2023-10-30T14:50:58.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:58.193+0000 7f4b2709a640 1 -- 172.21.15.57:0/1488180375 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4b040664d0 msgr2=0x7f4b04068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:58.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:58.193+0000 7f4b2709a640 1 --2- 172.21.15.57:0/1488180375 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4b040664d0 0x7f4b04068990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4b201597e0 tx=0x7f4b0c000ee0 comp rx=0 tx=0).stop 2023-10-30T14:50:58.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:58.193+0000 7f4b2709a640 1 -- 172.21.15.57:0/1488180375 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b10004d10 msgr2=0x7f4b200c9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:50:58.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:58.193+0000 7f4b2709a640 1 --2- 172.21.15.57:0/1488180375 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b10004d10 0x7f4b200c9010 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f4b1400f7a0 tx=0x7f4b1400fba0 comp rx=0 tx=0).stop 2023-10-30T14:50:58.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:58.193+0000 7f4b2709a640 1 -- 172.21.15.57:0/1488180375 shutdown_connections 2023-10-30T14:50:58.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:58.193+0000 7f4b2709a640 1 --2- 172.21.15.57:0/1488180375 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4b040664d0 0x7f4b04068990 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:58.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:58.193+0000 7f4b2709a640 1 --2- 172.21.15.57:0/1488180375 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b201f1dd0 0x7f4b200ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:58.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:58.193+0000 7f4b2709a640 1 --2- 172.21.15.57:0/1488180375 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b200c5c50 0x7f4b200ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:58.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:58.193+0000 7f4b2709a640 1 --2- 172.21.15.57:0/1488180375 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b10004d10 0x7f4b200c9010 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:50:58.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:58.193+0000 7f4b2709a640 1 -- 172.21.15.57:0/1488180375 >> 172.21.15.57:0/1488180375 conn(0x7f4b200715c0 msgr2=0x7f4b200c9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:50:58.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:58.193+0000 7f4b2709a640 1 -- 172.21.15.57:0/1488180375 shutdown_connections 2023-10-30T14:50:58.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:50:58.193+0000 7f4b2709a640 1 -- 172.21.15.57:0/1488180375 wait complete. 2023-10-30T14:50:58.219 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:51:00.220 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:51:00.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.368+0000 7f747dc17640 1 Processor -- start 2023-10-30T14:51:00.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.368+0000 7f747dc17640 1 -- start start 2023-10-30T14:51:00.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.368+0000 7f747dc17640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f747815cf90 0x7f74780c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:00.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.368+0000 7f747dc17640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7478159f20 con 0x7f74780c4a10 2023-10-30T14:51:00.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.368+0000 7f747dc17640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74780c47e0 con 0x7f7478153c80 2023-10-30T14:51:00.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.368+0000 7f747dc17640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f747815dec0 con 0x7f747815cf90 2023-10-30T14:51:00.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.368+0000 7f7477fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f747815cf90 0x7f74780c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:00.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.368+0000 7f7477fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f747815cf90 0x7f74780c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50074/0 (socket says 172.21.15.57:50074) 2023-10-30T14:51:00.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.368+0000 7f7477fff640 1 -- 172.21.15.57:0/1922066499 learned_addr learned my addr 172.21.15.57:0/1922066499 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:00.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.368+0000 7f7477fff640 1 -- 172.21.15.57:0/1922066499 >> v1:172.21.15.88:6789/0 conn(0x7f7478153c80 legacy=0x7f747815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:00.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.368+0000 7f7477fff640 1 -- 172.21.15.57:0/1922066499 >> v1:172.21.15.57:6789/0 conn(0x7f74780c4a10 legacy=0x7f74780ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:00.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.368+0000 7f7477fff640 1 -- 172.21.15.57:0/1922066499 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747815d620 con 0x7f747815cf90 2023-10-30T14:51:00.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.372+0000 7f7477fff640 1 --2- 172.21.15.57:0/1922066499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f747815cf90 0x7f74780c42a0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f7470009e50 tx=0x7f747001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e54399abdd513b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:00.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.372+0000 7f74767fc640 1 -- 172.21.15.57:0/1922066499 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74700af400 con 0x7f747815cf90 2023-10-30T14:51:00.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.372+0000 7f74767fc640 1 -- 172.21.15.57:0/1922066499 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74700ab050 con 0x7f747815cf90 2023-10-30T14:51:00.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.372+0000 7f74767fc640 1 -- 172.21.15.57:0/1922066499 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74700af780 con 0x7f747815cf90 2023-10-30T14:51:00.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.372+0000 7f747dc17640 1 -- 172.21.15.57:0/1922066499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f747815cf90 msgr2=0x7f74780c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:00.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.372+0000 7f747dc17640 1 --2- 172.21.15.57:0/1922066499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f747815cf90 0x7f74780c42a0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f7470009e50 tx=0x7f747001f1f0 comp rx=0 tx=0).stop 2023-10-30T14:51:00.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.372+0000 7f747dc17640 1 -- 172.21.15.57:0/1922066499 shutdown_connections 2023-10-30T14:51:00.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.372+0000 7f747dc17640 1 --2- 172.21.15.57:0/1922066499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f747815cf90 0x7f74780c42a0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:00.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.372+0000 7f747dc17640 1 -- 172.21.15.57:0/1922066499 >> 172.21.15.57:0/1922066499 conn(0x7f74780715c0 msgr2=0x7f74780719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:00.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.372+0000 7f747dc17640 1 -- 172.21.15.57:0/1922066499 shutdown_connections 2023-10-30T14:51:00.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.372+0000 7f747dc17640 1 -- 172.21.15.57:0/1922066499 wait complete. 2023-10-30T14:51:00.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f747dc17640 1 Processor -- start 2023-10-30T14:51:00.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f747dc17640 1 -- start start 2023-10-30T14:51:00.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f747dc17640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74780c4a10 0x7f74780ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:00.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f747dc17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7478153c80 0x7f74780c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:00.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f747dc17640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f747815cf90 0x7f74780c8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:00.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f747dc17640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74781574a0 con 0x7f74780c4a10 2023-10-30T14:51:00.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f747dc17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74780c7ac0 con 0x7f7478153c80 2023-10-30T14:51:00.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f747dc17640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7478156980 con 0x7f747815cf90 2023-10-30T14:51:00.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f74777fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74780c4a10 0x7f74780ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:00.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f74777fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74780c4a10 0x7f74780ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50080/0 (socket says 172.21.15.57:50080) 2023-10-30T14:51:00.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f74777fe640 1 -- 172.21.15.57:0/421783414 learned_addr learned my addr 172.21.15.57:0/421783414 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:00.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f7477fff640 1 --2- 172.21.15.57:0/421783414 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f747815cf90 0x7f74780c8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:00.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f7476ffd640 1 --2- 172.21.15.57:0/421783414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7478153c80 0x7f74780c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:00.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f74777fe640 1 -- 172.21.15.57:0/421783414 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f747815cf90 msgr2=0x7f74780c8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:00.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f74777fe640 1 --2- 172.21.15.57:0/421783414 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f747815cf90 0x7f74780c8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:00.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f74777fe640 1 -- 172.21.15.57:0/421783414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7478153c80 msgr2=0x7f74780c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:00.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f74777fe640 1 --2- 172.21.15.57:0/421783414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7478153c80 0x7f74780c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:00.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f74777fe640 1 -- 172.21.15.57:0/421783414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7470009b30 con 0x7f74780c4a10 2023-10-30T14:51:00.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f7477fff640 1 --2- 172.21.15.57:0/421783414 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f747815cf90 0x7f74780c8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:00.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f7476ffd640 1 --2- 172.21.15.57:0/421783414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7478153c80 0x7f74780c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:00.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f74777fe640 1 --2- 172.21.15.57:0/421783414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74780c4a10 0x7f74780ce2f0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f746800c560 tx=0x7f746800ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:00.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f745ffff640 1 -- 172.21.15.57:0/421783414 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7468005220 con 0x7f74780c4a10 2023-10-30T14:51:00.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f745ffff640 1 -- 172.21.15.57:0/421783414 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7468005380 con 0x7f74780c4a10 2023-10-30T14:51:00.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f747dc17640 1 -- 172.21.15.57:0/421783414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7478156c00 con 0x7f74780c4a10 2023-10-30T14:51:00.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f745ffff640 1 -- 172.21.15.57:0/421783414 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7468007270 con 0x7f74780c4a10 2023-10-30T14:51:00.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.376+0000 7f747dc17640 1 -- 172.21.15.57:0/421783414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74781571c0 con 0x7f74780c4a10 2023-10-30T14:51:00.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.380+0000 7f747dc17640 1 -- 172.21.15.57:0/421783414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f747804e170 con 0x7f74780c4a10 2023-10-30T14:51:00.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.380+0000 7f745ffff640 1 -- 172.21.15.57:0/421783414 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f74680073d0 con 0x7f74780c4a10 2023-10-30T14:51:00.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.380+0000 7f745ffff640 1 --2- 172.21.15.57:0/421783414 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f74640664a0 0x7f7464068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:00.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.380+0000 7f745ffff640 1 -- 172.21.15.57:0/421783414 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f7468083da0 con 0x7f74780c4a10 2023-10-30T14:51:00.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.380+0000 7f7476ffd640 1 --2- 172.21.15.57:0/421783414 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f74640664a0 0x7f7464068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:00.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.380+0000 7f7476ffd640 1 --2- 172.21.15.57:0/421783414 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f74640664a0 0x7f7464068960 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f74780c92d0 tx=0x7f746c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:00.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.384+0000 7f745ffff640 1 -- 172.21.15.57:0/421783414 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7468050c30 con 0x7f74780c4a10 2023-10-30T14:51:00.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.572+0000 7f747dc17640 1 -- 172.21.15.57:0/421783414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74780c41a0 con 0x7f74780c4a10 2023-10-30T14:51:00.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.576+0000 7f745ffff640 1 -- 172.21.15.57:0/421783414 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+5713 (secure 0 0 0) 0x7f7468054250 con 0x7f74780c4a10 2023-10-30T14:51:00.579 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:51:00.579 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:50:59.613314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8058":{"gid":8058,"name":"e","rank":0,"incarnation":185,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:51:00.580 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 186 2023-10-30T14:51:00.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.576+0000 7f747dc17640 1 -- 172.21.15.57:0/421783414 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f74640664a0 msgr2=0x7f7464068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:00.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.576+0000 7f747dc17640 1 --2- 172.21.15.57:0/421783414 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f74640664a0 0x7f7464068960 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f74780c92d0 tx=0x7f746c009040 comp rx=0 tx=0).stop 2023-10-30T14:51:00.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.576+0000 7f747dc17640 1 -- 172.21.15.57:0/421783414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74780c4a10 msgr2=0x7f74780ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:00.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.576+0000 7f747dc17640 1 --2- 172.21.15.57:0/421783414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74780c4a10 0x7f74780ce2f0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f746800c560 tx=0x7f746800ca20 comp rx=0 tx=0).stop 2023-10-30T14:51:00.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.576+0000 7f747dc17640 1 -- 172.21.15.57:0/421783414 shutdown_connections 2023-10-30T14:51:00.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.576+0000 7f747dc17640 1 --2- 172.21.15.57:0/421783414 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f74640664a0 0x7f7464068960 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:00.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.576+0000 7f747dc17640 1 --2- 172.21.15.57:0/421783414 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f747815cf90 0x7f74780c8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:00.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.576+0000 7f747dc17640 1 --2- 172.21.15.57:0/421783414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7478153c80 0x7f74780c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:00.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.576+0000 7f747dc17640 1 --2- 172.21.15.57:0/421783414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74780c4a10 0x7f74780ce2f0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:00.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.576+0000 7f747dc17640 1 -- 172.21.15.57:0/421783414 >> 172.21.15.57:0/421783414 conn(0x7f74780715c0 msgr2=0x7f74780c5750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:00.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.576+0000 7f747dc17640 1 -- 172.21.15.57:0/421783414 shutdown_connections 2023-10-30T14:51:00.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:00.576+0000 7f747dc17640 1 -- 172.21.15.57:0/421783414 wait complete. 2023-10-30T14:51:00.603 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:51:02.604 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:51:02.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.756+0000 7f4984246640 1 Processor -- start 2023-10-30T14:51:02.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.756+0000 7f4984246640 1 -- start start 2023-10-30T14:51:02.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.756+0000 7f4984246640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f497c1567b0 0x7f497c159a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:02.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.756+0000 7f4984246640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f497c0c6400 0x7f497c152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:02.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.756+0000 7f4984246640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f497c072d70 con 0x7f497c0670f0 2023-10-30T14:51:02.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.756+0000 7f4984246640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f497c155ac0 con 0x7f497c1567b0 2023-10-30T14:51:02.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.756+0000 7f4984246640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f497c159f80 con 0x7f497c0c6400 2023-10-30T14:51:02.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.756+0000 7f4981fe2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f497c1567b0 0x7f497c159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:02.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.756+0000 7f4981fe2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f497c1567b0 0x7f497c159a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50082/0 (socket says 172.21.15.57:50082) 2023-10-30T14:51:02.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.756+0000 7f4981fe2640 1 -- 172.21.15.57:0/2668218829 learned_addr learned my addr 172.21.15.57:0/2668218829 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:02.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.756+0000 7f49827e3640 1 --2- 172.21.15.57:0/2668218829 >> v2:172.21.15.88:3300/0 conn(0x7f497c0c6400 0x7f497c152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:02.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.756+0000 7f4981fe2640 1 -- 172.21.15.57:0/2668218829 >> v2:172.21.15.88:3300/0 conn(0x7f497c0c6400 msgr2=0x7f497c152ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:02.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.756+0000 7f4981fe2640 1 --2- 172.21.15.57:0/2668218829 >> v2:172.21.15.88:3300/0 conn(0x7f497c0c6400 0x7f497c152ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:02.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.756+0000 7f4981fe2640 1 -- 172.21.15.57:0/2668218829 >> v1:172.21.15.88:6789/0 conn(0x7f497c0670f0 legacy=0x7f497c0c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:02.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.756+0000 7f4981fe2640 1 -- 172.21.15.57:0/2668218829 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497c0727f0 con 0x7f497c1567b0 2023-10-30T14:51:02.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.760+0000 7f4981fe2640 1 --2- 172.21.15.57:0/2668218829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f497c1567b0 0x7f497c159a40 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f497c06ab50 tx=0x7f497801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dee303b9988ab6e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:02.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.760+0000 7f4980fe0640 1 -- 172.21.15.57:0/2668218829 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49780aa430 con 0x7f497c1567b0 2023-10-30T14:51:02.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.760+0000 7f4980fe0640 1 -- 172.21.15.57:0/2668218829 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49780aa590 con 0x7f497c1567b0 2023-10-30T14:51:02.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.760+0000 7f4980fe0640 1 -- 172.21.15.57:0/2668218829 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49780aa920 con 0x7f497c1567b0 2023-10-30T14:51:02.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.760+0000 7f4984246640 1 -- 172.21.15.57:0/2668218829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f497c1567b0 msgr2=0x7f497c159a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:02.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.760+0000 7f4984246640 1 --2- 172.21.15.57:0/2668218829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f497c1567b0 0x7f497c159a40 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f497c06ab50 tx=0x7f497801c4d0 comp rx=0 tx=0).stop 2023-10-30T14:51:02.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.760+0000 7f4984246640 1 -- 172.21.15.57:0/2668218829 shutdown_connections 2023-10-30T14:51:02.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.760+0000 7f4984246640 1 --2- 172.21.15.57:0/2668218829 >> v2:172.21.15.88:3300/0 conn(0x7f497c0c6400 0x7f497c152ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:02.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.760+0000 7f4984246640 1 --2- 172.21.15.57:0/2668218829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f497c1567b0 0x7f497c159a40 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:02.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.760+0000 7f4984246640 1 -- 172.21.15.57:0/2668218829 >> 172.21.15.57:0/2668218829 conn(0x7f497c06de90 msgr2=0x7f497c06e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:02.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.760+0000 7f4984246640 1 -- 172.21.15.57:0/2668218829 shutdown_connections 2023-10-30T14:51:02.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.760+0000 7f4984246640 1 -- 172.21.15.57:0/2668218829 wait complete. 2023-10-30T14:51:02.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f4984246640 1 Processor -- start 2023-10-30T14:51:02.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f4984246640 1 -- start start 2023-10-30T14:51:02.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f4984246640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f497c0670f0 0x7f497c154d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:02.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f4984246640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f497c0c6400 0x7f497c1f5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:02.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f4984246640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f497c1567b0 0x7f497c1f6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:02.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f4984246640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c150630 con 0x7f497c0670f0 2023-10-30T14:51:02.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f4984246640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c0cb080 con 0x7f497c1567b0 2023-10-30T14:51:02.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f4984246640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f497c150370 con 0x7f497c0c6400 2023-10-30T14:51:02.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f4981fe2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f497c0670f0 0x7f497c154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:02.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f4981fe2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f497c0670f0 0x7f497c154d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50088/0 (socket says 172.21.15.57:50088) 2023-10-30T14:51:02.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f4981fe2640 1 -- 172.21.15.57:0/1421077387 learned_addr learned my addr 172.21.15.57:0/1421077387 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:02.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f49817e1640 1 --2- 172.21.15.57:0/1421077387 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f497c0c6400 0x7f497c1f5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:02.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f49827e3640 1 --2- 172.21.15.57:0/1421077387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f497c1567b0 0x7f497c1f6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:02.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f4981fe2640 1 -- 172.21.15.57:0/1421077387 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f497c0c6400 msgr2=0x7f497c1f5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:02.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f4981fe2640 1 --2- 172.21.15.57:0/1421077387 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f497c0c6400 0x7f497c1f5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:02.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f4981fe2640 1 -- 172.21.15.57:0/1421077387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f497c1567b0 msgr2=0x7f497c1f6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:02.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f4981fe2640 1 --2- 172.21.15.57:0/1421077387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f497c1567b0 0x7f497c1f6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:02.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f4981fe2640 1 -- 172.21.15.57:0/1421077387 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49780094d0 con 0x7f497c0670f0 2023-10-30T14:51:02.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f4981fe2640 1 --2- 172.21.15.57:0/1421077387 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f497c0670f0 0x7f497c154d80 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f49780190e0 tx=0x7f497801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:02.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f49727fc640 1 -- 172.21.15.57:0/1421077387 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49780aa430 con 0x7f497c0670f0 2023-10-30T14:51:02.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f49727fc640 1 -- 172.21.15.57:0/1421077387 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49780b3820 con 0x7f497c0670f0 2023-10-30T14:51:02.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f49727fc640 1 -- 172.21.15.57:0/1421077387 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49780b3bd0 con 0x7f497c0670f0 2023-10-30T14:51:02.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f4984246640 1 -- 172.21.15.57:0/1421077387 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f497c0cb320 con 0x7f497c0670f0 2023-10-30T14:51:02.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.764+0000 7f4984246640 1 -- 172.21.15.57:0/1421077387 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f497c0cb8e0 con 0x7f497c0670f0 2023-10-30T14:51:02.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.768+0000 7f4984246640 1 -- 172.21.15.57:0/1421077387 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f497c0c9aa0 con 0x7f497c0670f0 2023-10-30T14:51:02.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.768+0000 7f49727fc640 1 -- 172.21.15.57:0/1421077387 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f49780b3d30 con 0x7f497c0670f0 2023-10-30T14:51:02.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.768+0000 7f49727fc640 1 --2- 172.21.15.57:0/1421077387 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f49600664a0 0x7f4960068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:02.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.768+0000 7f49727fc640 1 -- 172.21.15.57:0/1421077387 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f497811e9e0 con 0x7f497c0670f0 2023-10-30T14:51:02.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.768+0000 7f49817e1640 1 --2- 172.21.15.57:0/1421077387 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f49600664a0 0x7f4960068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:02.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.768+0000 7f49817e1640 1 --2- 172.21.15.57:0/1421077387 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f49600664a0 0x7f4960068960 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f497c14c9a0 tx=0x7f496c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:02.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.772+0000 7f49727fc640 1 -- 172.21.15.57:0/1421077387 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f497c0c9aa0 con 0x7f497c0670f0 2023-10-30T14:51:02.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.960+0000 7f4984246640 1 -- 172.21.15.57:0/1421077387 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f497c0c9aa0 con 0x7f497c0670f0 2023-10-30T14:51:02.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.960+0000 7f49727fc640 1 -- 172.21.15.57:0/1421077387 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+5713 (secure 0 0 0) 0x7f497c0c9aa0 con 0x7f497c0670f0 2023-10-30T14:51:02.966 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:51:02.966 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:50:59.613314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8058":{"gid":8058,"name":"e","rank":0,"incarnation":185,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:51:02.967 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 186 2023-10-30T14:51:02.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.964+0000 7f4984246640 1 -- 172.21.15.57:0/1421077387 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f49600664a0 msgr2=0x7f4960068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:02.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.964+0000 7f4984246640 1 --2- 172.21.15.57:0/1421077387 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f49600664a0 0x7f4960068960 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f497c14c9a0 tx=0x7f496c009040 comp rx=0 tx=0).stop 2023-10-30T14:51:02.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.964+0000 7f4984246640 1 -- 172.21.15.57:0/1421077387 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f497c0670f0 msgr2=0x7f497c154d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:02.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.964+0000 7f4984246640 1 --2- 172.21.15.57:0/1421077387 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f497c0670f0 0x7f497c154d80 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f49780190e0 tx=0x7f497801ad10 comp rx=0 tx=0).stop 2023-10-30T14:51:02.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.964+0000 7f4984246640 1 -- 172.21.15.57:0/1421077387 shutdown_connections 2023-10-30T14:51:02.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.964+0000 7f4984246640 1 --2- 172.21.15.57:0/1421077387 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f49600664a0 0x7f4960068960 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:02.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.964+0000 7f4984246640 1 --2- 172.21.15.57:0/1421077387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f497c1567b0 0x7f497c1f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:02.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.964+0000 7f4984246640 1 --2- 172.21.15.57:0/1421077387 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f497c0c6400 0x7f497c1f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:02.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.964+0000 7f4984246640 1 --2- 172.21.15.57:0/1421077387 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f497c0670f0 0x7f497c154d80 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:02.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.964+0000 7f4984246640 1 -- 172.21.15.57:0/1421077387 >> 172.21.15.57:0/1421077387 conn(0x7f497c06de90 msgr2=0x7f497c0aee10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:02.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.964+0000 7f4984246640 1 -- 172.21.15.57:0/1421077387 shutdown_connections 2023-10-30T14:51:02.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:02.964+0000 7f4984246640 1 -- 172.21.15.57:0/1421077387 wait complete. 2023-10-30T14:51:02.987 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 186, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 186, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8058": { "addr": "172.21.15.57:6837/2594912006", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 2594912006, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 2594912006, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8058, "incarnation": 185, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:reconnect", "state_seq": 32 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:50:59.613314+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 8058 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6839/1078372885", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1078372885, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1078372885, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 170, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6716, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.57:6834/697507447", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 697507447, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 697507447, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 156, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7377, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:51:04.988 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:51:05.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.132+0000 7faa4b5f8640 1 Processor -- start 2023-10-30T14:51:05.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.132+0000 7faa4b5f8640 1 -- start start 2023-10-30T14:51:05.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.132+0000 7faa4b5f8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faa4414fc50 0x7faa4406fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:05.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.132+0000 7faa4b5f8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa440c6320 0x7faa44070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:05.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.132+0000 7faa4b5f8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa440c55f0 con 0x7faa4406be90 2023-10-30T14:51:05.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.132+0000 7faa4b5f8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa440c5f00 con 0x7faa4414fc50 2023-10-30T14:51:05.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.132+0000 7faa4b5f8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa44071350 con 0x7faa440c6320 2023-10-30T14:51:05.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.132+0000 7faa48b93640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa440c6320 0x7faa44070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:05.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.132+0000 7faa48b93640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa440c6320 0x7faa44070e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:56388/0 (socket says 172.21.15.57:56388) 2023-10-30T14:51:05.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.132+0000 7faa48b93640 1 -- 172.21.15.57:0/4216124101 learned_addr learned my addr 172.21.15.57:0/4216124101 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:05.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.132+0000 7faa49394640 1 --2- 172.21.15.57:0/4216124101 >> v2:172.21.15.88:3300/0 conn(0x7faa4414fc50 0x7faa4406fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:05.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.136+0000 7faa48b93640 1 -- 172.21.15.57:0/4216124101 >> v2:172.21.15.88:3300/0 conn(0x7faa4414fc50 msgr2=0x7faa4406fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:05.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.136+0000 7faa48b93640 1 --2- 172.21.15.57:0/4216124101 >> v2:172.21.15.88:3300/0 conn(0x7faa4414fc50 0x7faa4406fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:05.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.136+0000 7faa48b93640 1 -- 172.21.15.57:0/4216124101 >> v1:172.21.15.88:6789/0 conn(0x7faa4406be90 legacy=0x7faa44155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:05.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.136+0000 7faa48b93640 1 -- 172.21.15.57:0/4216124101 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa441567b0 con 0x7faa440c6320 2023-10-30T14:51:05.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.136+0000 7faa48b93640 1 --2- 172.21.15.57:0/4216124101 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa440c6320 0x7faa44070e10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7faa34006fd0 tx=0x7faa3401e940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99ce41a50138d201 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:05.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.136+0000 7faa3bfff640 1 -- 172.21.15.57:0/4216124101 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa34004230 con 0x7faa440c6320 2023-10-30T14:51:05.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.136+0000 7faa3bfff640 1 -- 172.21.15.57:0/4216124101 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa340ab070 con 0x7faa440c6320 2023-10-30T14:51:05.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.136+0000 7faa3bfff640 1 -- 172.21.15.57:0/4216124101 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa340a7790 con 0x7faa440c6320 2023-10-30T14:51:05.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa4b5f8640 1 -- 172.21.15.57:0/4216124101 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa440c6320 msgr2=0x7faa44070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:05.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa4b5f8640 1 --2- 172.21.15.57:0/4216124101 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa440c6320 0x7faa44070e10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7faa34006fd0 tx=0x7faa3401e940 comp rx=0 tx=0).stop 2023-10-30T14:51:05.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa4b5f8640 1 -- 172.21.15.57:0/4216124101 shutdown_connections 2023-10-30T14:51:05.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa4b5f8640 1 --2- 172.21.15.57:0/4216124101 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa440c6320 0x7faa44070e10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:05.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa4b5f8640 1 --2- 172.21.15.57:0/4216124101 >> v2:172.21.15.88:3300/0 conn(0x7faa4414fc50 0x7faa4406fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:05.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa4b5f8640 1 -- 172.21.15.57:0/4216124101 >> 172.21.15.57:0/4216124101 conn(0x7faa440cac70 msgr2=0x7faa440cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:05.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa4b5f8640 1 -- 172.21.15.57:0/4216124101 shutdown_connections 2023-10-30T14:51:05.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa4b5f8640 1 -- 172.21.15.57:0/4216124101 wait complete. 2023-10-30T14:51:05.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa4b5f8640 1 Processor -- start 2023-10-30T14:51:05.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa4b5f8640 1 -- start start 2023-10-30T14:51:05.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa4b5f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa4406be90 0x7faa4415f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:05.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa4b5f8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa440c6320 0x7faa44158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:05.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa4b5f8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa4414fc50 0x7faa441590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:05.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa4b5f8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4414a080 con 0x7faa4414fc50 2023-10-30T14:51:05.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa4b5f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4415e8a0 con 0x7faa4406be90 2023-10-30T14:51:05.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa4b5f8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa4415eb80 con 0x7faa440c6320 2023-10-30T14:51:05.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa49b95640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa4414fc50 0x7faa441590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:05.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa48b93640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa440c6320 0x7faa44158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:05.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa49394640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa4406be90 0x7faa4415f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:05.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa49b95640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa4414fc50 0x7faa441590f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51806/0 (socket says 172.21.15.57:51806) 2023-10-30T14:51:05.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.140+0000 7faa49b95640 1 -- 172.21.15.57:0/316248891 learned_addr learned my addr 172.21.15.57:0/316248891 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:05.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.144+0000 7faa49b95640 1 -- 172.21.15.57:0/316248891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa440c6320 msgr2=0x7faa44158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:05.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.144+0000 7faa49b95640 1 --2- 172.21.15.57:0/316248891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa440c6320 0x7faa44158bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:05.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.144+0000 7faa49b95640 1 -- 172.21.15.57:0/316248891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa4406be90 msgr2=0x7faa4415f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:05.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.144+0000 7faa49b95640 1 --2- 172.21.15.57:0/316248891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa4406be90 0x7faa4415f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:05.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.144+0000 7faa49b95640 1 -- 172.21.15.57:0/316248891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3400b780 con 0x7faa4414fc50 2023-10-30T14:51:05.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.144+0000 7faa49b95640 1 --2- 172.21.15.57:0/316248891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa4414fc50 0x7faa441590f0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00c770 tx=0x7faa3c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:05.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.144+0000 7faa39ffb640 1 -- 172.21.15.57:0/316248891 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c003b80 con 0x7faa4414fc50 2023-10-30T14:51:05.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.144+0000 7faa39ffb640 1 -- 172.21.15.57:0/316248891 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa3c003ce0 con 0x7faa4414fc50 2023-10-30T14:51:05.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.144+0000 7faa4b5f8640 1 -- 172.21.15.57:0/316248891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa4415ee60 con 0x7faa4414fc50 2023-10-30T14:51:05.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.144+0000 7faa39ffb640 1 -- 172.21.15.57:0/316248891 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c00aa90 con 0x7faa4414fc50 2023-10-30T14:51:05.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.144+0000 7faa4b5f8640 1 -- 172.21.15.57:0/316248891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa4415f420 con 0x7faa4414fc50 2023-10-30T14:51:05.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.144+0000 7faa4b5f8640 1 -- 172.21.15.57:0/316248891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa4415f610 con 0x7faa4414fc50 2023-10-30T14:51:05.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.144+0000 7faa39ffb640 1 -- 172.21.15.57:0/316248891 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faa3c00abf0 con 0x7faa4414fc50 2023-10-30T14:51:05.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.144+0000 7faa39ffb640 1 --2- 172.21.15.57:0/316248891 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa280664a0 0x7faa28068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:05.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.144+0000 7faa39ffb640 1 -- 172.21.15.57:0/316248891 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7faa3c084320 con 0x7faa4414fc50 2023-10-30T14:51:05.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.148+0000 7faa49394640 1 --2- 172.21.15.57:0/316248891 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa280664a0 0x7faa28068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:05.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.148+0000 7faa49394640 1 --2- 172.21.15.57:0/316248891 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa280664a0 0x7faa28068960 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7faa40009ac0 tx=0x7faa40009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:05.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.148+0000 7faa39ffb640 1 -- 172.21.15.57:0/316248891 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faa3c0510a0 con 0x7faa4414fc50 2023-10-30T14:51:05.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.340+0000 7faa4b5f8640 1 -- 172.21.15.57:0/316248891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa4404e170 con 0x7faa4414fc50 2023-10-30T14:51:05.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.340+0000 7faa39ffb640 1 -- 172.21.15.57:0/316248891 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+5713 (secure 0 0 0) 0x7faa3c0546c0 con 0x7faa4414fc50 2023-10-30T14:51:05.345 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:51:05.345 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:50:59.613314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8058":{"gid":8058,"name":"e","rank":0,"incarnation":185,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:51:05.345 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 186 2023-10-30T14:51:05.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.340+0000 7faa4b5f8640 1 -- 172.21.15.57:0/316248891 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa280664a0 msgr2=0x7faa28068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:05.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.340+0000 7faa4b5f8640 1 --2- 172.21.15.57:0/316248891 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa280664a0 0x7faa28068960 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7faa40009ac0 tx=0x7faa40009190 comp rx=0 tx=0).stop 2023-10-30T14:51:05.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.344+0000 7faa4b5f8640 1 -- 172.21.15.57:0/316248891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa4414fc50 msgr2=0x7faa441590f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:05.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.344+0000 7faa4b5f8640 1 --2- 172.21.15.57:0/316248891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa4414fc50 0x7faa441590f0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00c770 tx=0x7faa3c00cc30 comp rx=0 tx=0).stop 2023-10-30T14:51:05.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.344+0000 7faa4b5f8640 1 -- 172.21.15.57:0/316248891 shutdown_connections 2023-10-30T14:51:05.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.344+0000 7faa4b5f8640 1 --2- 172.21.15.57:0/316248891 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7faa280664a0 0x7faa28068960 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:05.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.344+0000 7faa4b5f8640 1 --2- 172.21.15.57:0/316248891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa4414fc50 0x7faa441590f0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:05.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.344+0000 7faa4b5f8640 1 --2- 172.21.15.57:0/316248891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa440c6320 0x7faa44158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:05.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.344+0000 7faa4b5f8640 1 --2- 172.21.15.57:0/316248891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa4406be90 0x7faa4415f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:05.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.344+0000 7faa4b5f8640 1 -- 172.21.15.57:0/316248891 >> 172.21.15.57:0/316248891 conn(0x7faa440cac70 msgr2=0x7faa440c9540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:05.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.344+0000 7faa4b5f8640 1 -- 172.21.15.57:0/316248891 shutdown_connections 2023-10-30T14:51:05.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:05.344+0000 7faa4b5f8640 1 -- 172.21.15.57:0/316248891 wait complete. 2023-10-30T14:51:05.367 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:51:07.368 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:51:07.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.516+0000 7fda16fe4640 1 Processor -- start 2023-10-30T14:51:07.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.520+0000 7fda16fe4640 1 -- start start 2023-10-30T14:51:07.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.520+0000 7fda16fe4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fda100c8eb0 0x7fda1015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:07.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.520+0000 7fda16fe4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fda1015ac70 0x7fda1015a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:07.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.520+0000 7fda16fe4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda1006aeb0 0x7fda10069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:07.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.520+0000 7fda16fe4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda10159800 con 0x7fda1015ac70 2023-10-30T14:51:07.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.520+0000 7fda16fe4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda1015dcc0 con 0x7fda100c8eb0 2023-10-30T14:51:07.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.520+0000 7fda16fe4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda10159d80 con 0x7fda1006aeb0 2023-10-30T14:51:07.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.520+0000 7fda157e1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fda1015ac70 0x7fda1015a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:07.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.520+0000 7fda157e1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fda1015ac70 0x7fda1015a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51810/0 (socket says 172.21.15.57:51810) 2023-10-30T14:51:07.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.520+0000 7fda157e1640 1 -- 172.21.15.57:0/1737932458 learned_addr learned my addr 172.21.15.57:0/1737932458 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:07.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.520+0000 7fda15fe2640 1 --2- 172.21.15.57:0/1737932458 >> v2:172.21.15.88:3300/0 conn(0x7fda100c8eb0 0x7fda1015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:07.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.520+0000 7fda167e3640 1 --2- 172.21.15.57:0/1737932458 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda1006aeb0 0x7fda10069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:07.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.520+0000 7fda157e1640 1 -- 172.21.15.57:0/1737932458 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda1006aeb0 msgr2=0x7fda10069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:07.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.520+0000 7fda157e1640 1 --2- 172.21.15.57:0/1737932458 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda1006aeb0 0x7fda10069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:07.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.520+0000 7fda157e1640 1 -- 172.21.15.57:0/1737932458 >> v2:172.21.15.88:3300/0 conn(0x7fda100c8eb0 msgr2=0x7fda1015cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:07.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.520+0000 7fda157e1640 1 --2- 172.21.15.57:0/1737932458 >> v2:172.21.15.88:3300/0 conn(0x7fda100c8eb0 0x7fda1015cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:07.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.520+0000 7fda157e1640 1 -- 172.21.15.57:0/1737932458 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1015a750 con 0x7fda1015ac70 2023-10-30T14:51:07.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.524+0000 7fda157e1640 1 --2- 172.21.15.57:0/1737932458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda1015ac70 0x7fda1015a210 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fda1006aa40 tx=0x7fda0001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18e8dfa65dc3b83 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:07.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.524+0000 7fda14fe0640 1 -- 172.21.15.57:0/1737932458 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda000a89b0 con 0x7fda1015ac70 2023-10-30T14:51:07.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.524+0000 7fda14fe0640 1 -- 172.21.15.57:0/1737932458 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda000a8b10 con 0x7fda1015ac70 2023-10-30T14:51:07.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.524+0000 7fda14fe0640 1 -- 172.21.15.57:0/1737932458 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda000a8ea0 con 0x7fda1015ac70 2023-10-30T14:51:07.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.524+0000 7fda16fe4640 1 -- 172.21.15.57:0/1737932458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda1015ac70 msgr2=0x7fda1015a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:07.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.524+0000 7fda16fe4640 1 --2- 172.21.15.57:0/1737932458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda1015ac70 0x7fda1015a210 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fda1006aa40 tx=0x7fda0001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:51:07.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.524+0000 7fda16fe4640 1 -- 172.21.15.57:0/1737932458 shutdown_connections 2023-10-30T14:51:07.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.524+0000 7fda16fe4640 1 --2- 172.21.15.57:0/1737932458 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda1006aeb0 0x7fda10069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:07.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.524+0000 7fda16fe4640 1 --2- 172.21.15.57:0/1737932458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda1015ac70 0x7fda1015a210 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:07.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.524+0000 7fda16fe4640 1 --2- 172.21.15.57:0/1737932458 >> v2:172.21.15.88:3300/0 conn(0x7fda100c8eb0 0x7fda1015cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:07.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.524+0000 7fda16fe4640 1 -- 172.21.15.57:0/1737932458 >> 172.21.15.57:0/1737932458 conn(0x7fda100cef00 msgr2=0x7fda100cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:07.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.524+0000 7fda16fe4640 1 -- 172.21.15.57:0/1737932458 shutdown_connections 2023-10-30T14:51:07.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.524+0000 7fda16fe4640 1 -- 172.21.15.57:0/1737932458 wait complete. 2023-10-30T14:51:07.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.524+0000 7fda16fe4640 1 Processor -- start 2023-10-30T14:51:07.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.524+0000 7fda16fe4640 1 -- start start 2023-10-30T14:51:07.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda16fe4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda1006aeb0 0x7fda100cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:07.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda16fe4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda100c8eb0 0x7fda101586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:07.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda16fe4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda1015ac70 0x7fda101eee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:07.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda16fe4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda101eea00 con 0x7fda1015ac70 2023-10-30T14:51:07.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda16fe4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda101f2c70 con 0x7fda100c8eb0 2023-10-30T14:51:07.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda16fe4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda10159430 con 0x7fda1006aeb0 2023-10-30T14:51:07.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda167e3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda1015ac70 0x7fda101eee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:07.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda167e3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda1015ac70 0x7fda101eee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51818/0 (socket says 172.21.15.57:51818) 2023-10-30T14:51:07.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda167e3640 1 -- 172.21.15.57:0/792604878 learned_addr learned my addr 172.21.15.57:0/792604878 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:07.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda15fe2640 1 --2- 172.21.15.57:0/792604878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda1006aeb0 0x7fda100cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:07.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda157e1640 1 --2- 172.21.15.57:0/792604878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda100c8eb0 0x7fda101586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:07.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda167e3640 1 -- 172.21.15.57:0/792604878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda1006aeb0 msgr2=0x7fda100cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:07.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda167e3640 1 --2- 172.21.15.57:0/792604878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda1006aeb0 0x7fda100cc230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:07.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda167e3640 1 -- 172.21.15.57:0/792604878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda100c8eb0 msgr2=0x7fda101586f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:07.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda167e3640 1 --2- 172.21.15.57:0/792604878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda100c8eb0 0x7fda101586f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:07.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda167e3640 1 -- 172.21.15.57:0/792604878 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda000094d0 con 0x7fda1015ac70 2023-10-30T14:51:07.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda157e1640 1 --2- 172.21.15.57:0/792604878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda100c8eb0 0x7fda101586f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:07.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda167e3640 1 --2- 172.21.15.57:0/792604878 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda1015ac70 0x7fda101eee90 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fda08007d20 tx=0x7fda0800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:07.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda067fc640 1 -- 172.21.15.57:0/792604878 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda08014420 con 0x7fda1015ac70 2023-10-30T14:51:07.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda067fc640 1 -- 172.21.15.57:0/792604878 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda080145a0 con 0x7fda1015ac70 2023-10-30T14:51:07.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda067fc640 1 -- 172.21.15.57:0/792604878 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda08014950 con 0x7fda1015ac70 2023-10-30T14:51:07.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda16fe4640 1 -- 172.21.15.57:0/792604878 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda100cc770 con 0x7fda1015ac70 2023-10-30T14:51:07.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda16fe4640 1 -- 172.21.15.57:0/792604878 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda101f23d0 con 0x7fda1015ac70 2023-10-30T14:51:07.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda16fe4640 1 -- 172.21.15.57:0/792604878 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda101eeb70 con 0x7fda1015ac70 2023-10-30T14:51:07.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.528+0000 7fda067fc640 1 -- 172.21.15.57:0/792604878 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fda08014ab0 con 0x7fda1015ac70 2023-10-30T14:51:07.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.532+0000 7fda067fc640 1 --2- 172.21.15.57:0/792604878 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd9f40664a0 0x7fd9f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:07.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.532+0000 7fda067fc640 1 -- 172.21.15.57:0/792604878 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fda08084eb0 con 0x7fda1015ac70 2023-10-30T14:51:07.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.532+0000 7fda15fe2640 1 --2- 172.21.15.57:0/792604878 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd9f40664a0 0x7fd9f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:07.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.532+0000 7fda15fe2640 1 --2- 172.21.15.57:0/792604878 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd9f40664a0 0x7fd9f4068960 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fda0c007a60 tx=0x7fda0c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:07.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.536+0000 7fda067fc640 1 -- 172.21.15.57:0/792604878 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fda08051b80 con 0x7fda1015ac70 2023-10-30T14:51:07.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.728+0000 7fda16fe4640 1 -- 172.21.15.57:0/792604878 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda1004e170 con 0x7fda1015ac70 2023-10-30T14:51:07.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.728+0000 7fda067fc640 1 -- 172.21.15.57:0/792604878 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+5710 (secure 0 0 0) 0x7fda080551a0 con 0x7fda1015ac70 2023-10-30T14:51:07.733 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:51:07.733 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:51:05.354063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8058":{"gid":8058,"name":"e","rank":0,"incarnation":185,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:51:07.734 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 187 2023-10-30T14:51:07.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.732+0000 7fda16fe4640 1 -- 172.21.15.57:0/792604878 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd9f40664a0 msgr2=0x7fd9f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:07.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.732+0000 7fda16fe4640 1 --2- 172.21.15.57:0/792604878 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd9f40664a0 0x7fd9f4068960 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fda0c007a60 tx=0x7fda0c008040 comp rx=0 tx=0).stop 2023-10-30T14:51:07.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.732+0000 7fda16fe4640 1 -- 172.21.15.57:0/792604878 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda1015ac70 msgr2=0x7fda101eee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:07.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.732+0000 7fda16fe4640 1 --2- 172.21.15.57:0/792604878 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda1015ac70 0x7fda101eee90 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fda08007d20 tx=0x7fda0800e4b0 comp rx=0 tx=0).stop 2023-10-30T14:51:07.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.732+0000 7fda16fe4640 1 -- 172.21.15.57:0/792604878 shutdown_connections 2023-10-30T14:51:07.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.732+0000 7fda16fe4640 1 --2- 172.21.15.57:0/792604878 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd9f40664a0 0x7fd9f4068960 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:07.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.732+0000 7fda16fe4640 1 --2- 172.21.15.57:0/792604878 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda1015ac70 0x7fda101eee90 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:07.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.732+0000 7fda16fe4640 1 --2- 172.21.15.57:0/792604878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda100c8eb0 0x7fda101586f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:07.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.732+0000 7fda16fe4640 1 --2- 172.21.15.57:0/792604878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda1006aeb0 0x7fda100cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:07.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.732+0000 7fda16fe4640 1 -- 172.21.15.57:0/792604878 >> 172.21.15.57:0/792604878 conn(0x7fda100cef00 msgr2=0x7fda100c7690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:07.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.732+0000 7fda16fe4640 1 -- 172.21.15.57:0/792604878 shutdown_connections 2023-10-30T14:51:07.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:07.732+0000 7fda16fe4640 1 -- 172.21.15.57:0/792604878 wait complete. 2023-10-30T14:51:07.754 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:51:09.755 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:51:09.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.904+0000 7f07ff765640 1 Processor -- start 2023-10-30T14:51:09.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.904+0000 7f07ff765640 1 -- start start 2023-10-30T14:51:09.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.908+0000 7f07ff765640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f07f8071340 0x7f07f8065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:09.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.908+0000 7f07ff765640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f07f8159970 0x7f07f8159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:09.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.908+0000 7f07ff765640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f806e390 con 0x7f07f8153c10 2023-10-30T14:51:09.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.908+0000 7f07ff765640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07f8072390 con 0x7f07f8159970 2023-10-30T14:51:09.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.908+0000 7f07ff765640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07f80660e0 con 0x7f07f8071340 2023-10-30T14:51:09.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.908+0000 7f07fdd02640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f07f8159970 0x7f07f8159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:09.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.908+0000 7f07fdd02640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f07f8159970 0x7f07f8159dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51832/0 (socket says 172.21.15.57:51832) 2023-10-30T14:51:09.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.908+0000 7f07fdd02640 1 -- 172.21.15.57:0/2764517721 learned_addr learned my addr 172.21.15.57:0/2764517721 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:09.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.908+0000 7f07fcd00640 1 --2- 172.21.15.57:0/2764517721 >> v2:172.21.15.88:3300/0 conn(0x7f07f8071340 0x7f07f8065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:09.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.908+0000 7f07fdd02640 1 -- 172.21.15.57:0/2764517721 >> v2:172.21.15.88:3300/0 conn(0x7f07f8071340 msgr2=0x7f07f8065ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:09.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.908+0000 7f07fdd02640 1 --2- 172.21.15.57:0/2764517721 >> v2:172.21.15.88:3300/0 conn(0x7f07f8071340 0x7f07f8065ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:09.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.908+0000 7f07fdd02640 1 -- 172.21.15.57:0/2764517721 >> v1:172.21.15.88:6789/0 conn(0x7f07f8153c10 legacy=0x7f07f8070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:09.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.908+0000 7f07fdd02640 1 -- 172.21.15.57:0/2764517721 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f80bec10 con 0x7f07f8159970 2023-10-30T14:51:09.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.912+0000 7f07fdd02640 1 --2- 172.21.15.57:0/2764517721 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07f8159970 0x7f07f8159dc0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f07f806f200 tx=0x7f07f001c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd1baeb9d46d8587 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:09.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.912+0000 7f07effff640 1 -- 172.21.15.57:0/2764517721 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f00a64a0 con 0x7f07f8159970 2023-10-30T14:51:09.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.912+0000 7f07effff640 1 -- 172.21.15.57:0/2764517721 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f00a6600 con 0x7f07f8159970 2023-10-30T14:51:09.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.912+0000 7f07effff640 1 -- 172.21.15.57:0/2764517721 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f00a6990 con 0x7f07f8159970 2023-10-30T14:51:09.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.912+0000 7f07ff765640 1 -- 172.21.15.57:0/2764517721 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07f8159970 msgr2=0x7f07f8159dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:09.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.912+0000 7f07ff765640 1 --2- 172.21.15.57:0/2764517721 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07f8159970 0x7f07f8159dc0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f07f806f200 tx=0x7f07f001c900 comp rx=0 tx=0).stop 2023-10-30T14:51:09.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.912+0000 7f07ff765640 1 -- 172.21.15.57:0/2764517721 shutdown_connections 2023-10-30T14:51:09.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.912+0000 7f07ff765640 1 --2- 172.21.15.57:0/2764517721 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07f8159970 0x7f07f8159dc0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:09.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.912+0000 7f07ff765640 1 --2- 172.21.15.57:0/2764517721 >> v2:172.21.15.88:3300/0 conn(0x7f07f8071340 0x7f07f8065ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:09.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.912+0000 7f07ff765640 1 -- 172.21.15.57:0/2764517721 >> 172.21.15.57:0/2764517721 conn(0x7f07f80c7f60 msgr2=0x7f07f80c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:09.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.912+0000 7f07ff765640 1 -- 172.21.15.57:0/2764517721 shutdown_connections 2023-10-30T14:51:09.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.912+0000 7f07ff765640 1 -- 172.21.15.57:0/2764517721 wait complete. 2023-10-30T14:51:09.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.912+0000 7f07ff765640 1 Processor -- start 2023-10-30T14:51:09.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.912+0000 7f07ff765640 1 -- start start 2023-10-30T14:51:09.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07ff765640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07f8071340 0x7f07f80c37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:09.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07ff765640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07f8153c10 0x7f07f8151e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:09.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07ff765640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07f8159970 0x7f07f8148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:09.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07ff765640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f81f2730 con 0x7f07f8159970 2023-10-30T14:51:09.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07ff765640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f81f1db0 con 0x7f07f8153c10 2023-10-30T14:51:09.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07ff765640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f81f5dd0 con 0x7f07f8071340 2023-10-30T14:51:09.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07fdd02640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07f8159970 0x7f07f8148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:09.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07fdd02640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07f8159970 0x7f07f8148ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51836/0 (socket says 172.21.15.57:51836) 2023-10-30T14:51:09.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07fdd02640 1 -- 172.21.15.57:0/2149301970 learned_addr learned my addr 172.21.15.57:0/2149301970 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:09.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07fcd00640 1 --2- 172.21.15.57:0/2149301970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07f8153c10 0x7f07f8151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:09.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07fd501640 1 --2- 172.21.15.57:0/2149301970 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07f8071340 0x7f07f80c37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:09.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07fdd02640 1 -- 172.21.15.57:0/2149301970 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07f8071340 msgr2=0x7f07f80c37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:09.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07fdd02640 1 --2- 172.21.15.57:0/2149301970 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07f8071340 0x7f07f80c37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:09.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07fdd02640 1 -- 172.21.15.57:0/2149301970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07f8153c10 msgr2=0x7f07f8151e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:09.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07fdd02640 1 --2- 172.21.15.57:0/2149301970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07f8153c10 0x7f07f8151e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:09.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07fdd02640 1 -- 172.21.15.57:0/2149301970 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f0009b30 con 0x7f07f8159970 2023-10-30T14:51:09.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07fdd02640 1 --2- 172.21.15.57:0/2149301970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07f8159970 0x7f07f8148ef0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f07f001a530 tx=0x7f07f00196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:09.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07edffb640 1 -- 172.21.15.57:0/2149301970 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f00a64a0 con 0x7f07f8159970 2023-10-30T14:51:09.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07edffb640 1 -- 172.21.15.57:0/2149301970 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f00b5070 con 0x7f07f8159970 2023-10-30T14:51:09.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07ff765640 1 -- 172.21.15.57:0/2149301970 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f81f60b0 con 0x7f07f8159970 2023-10-30T14:51:09.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07edffb640 1 -- 172.21.15.57:0/2149301970 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f00a78e0 con 0x7f07f8159970 2023-10-30T14:51:09.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07ff765640 1 -- 172.21.15.57:0/2149301970 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f81f2350 con 0x7f07f8159970 2023-10-30T14:51:09.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.916+0000 7f07ff765640 1 -- 172.21.15.57:0/2149301970 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f81f2870 con 0x7f07f8159970 2023-10-30T14:51:09.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.920+0000 7f07edffb640 1 -- 172.21.15.57:0/2149301970 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f07f00a7a40 con 0x7f07f8159970 2023-10-30T14:51:09.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.920+0000 7f07edffb640 1 --2- 172.21.15.57:0/2149301970 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07dc0664d0 0x7f07dc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:09.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.920+0000 7f07edffb640 1 -- 172.21.15.57:0/2149301970 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f07f011e8c0 con 0x7f07f8159970 2023-10-30T14:51:09.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.920+0000 7f07fd501640 1 --2- 172.21.15.57:0/2149301970 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07dc0664d0 0x7f07dc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:09.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.920+0000 7f07fd501640 1 --2- 172.21.15.57:0/2149301970 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07dc0664d0 0x7f07dc068990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f07f4007fd0 tx=0x7f07f4007e40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:09.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:09.924+0000 7f07edffb640 1 -- 172.21.15.57:0/2149301970 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f07f81f2870 con 0x7f07f8159970 2023-10-30T14:51:10.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:10.112+0000 7f07ff765640 1 -- 172.21.15.57:0/2149301970 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07f81f2870 con 0x7f07f8159970 2023-10-30T14:51:10.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:10.112+0000 7f07edffb640 1 -- 172.21.15.57:0/2149301970 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+5710 (secure 0 0 0) 0x7f07f81f2870 con 0x7f07f8159970 2023-10-30T14:51:10.118 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:51:10.118 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:51:05.354063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8058":{"gid":8058,"name":"e","rank":0,"incarnation":185,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:51:10.119 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 187 2023-10-30T14:51:10.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:10.116+0000 7f07ff765640 1 -- 172.21.15.57:0/2149301970 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07dc0664d0 msgr2=0x7f07dc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:10.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:10.116+0000 7f07ff765640 1 --2- 172.21.15.57:0/2149301970 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07dc0664d0 0x7f07dc068990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f07f4007fd0 tx=0x7f07f4007e40 comp rx=0 tx=0).stop 2023-10-30T14:51:10.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:10.116+0000 7f07ff765640 1 -- 172.21.15.57:0/2149301970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07f8159970 msgr2=0x7f07f8148ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:10.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:10.116+0000 7f07ff765640 1 --2- 172.21.15.57:0/2149301970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07f8159970 0x7f07f8148ef0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f07f001a530 tx=0x7f07f00196a0 comp rx=0 tx=0).stop 2023-10-30T14:51:10.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:10.116+0000 7f07ff765640 1 -- 172.21.15.57:0/2149301970 shutdown_connections 2023-10-30T14:51:10.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:10.116+0000 7f07ff765640 1 --2- 172.21.15.57:0/2149301970 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f07dc0664d0 0x7f07dc068990 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:10.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:10.116+0000 7f07ff765640 1 --2- 172.21.15.57:0/2149301970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07f8159970 0x7f07f8148ef0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:10.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:10.116+0000 7f07ff765640 1 --2- 172.21.15.57:0/2149301970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07f8153c10 0x7f07f8151e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:10.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:10.116+0000 7f07ff765640 1 --2- 172.21.15.57:0/2149301970 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07f8071340 0x7f07f80c37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:10.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:10.116+0000 7f07ff765640 1 -- 172.21.15.57:0/2149301970 >> 172.21.15.57:0/2149301970 conn(0x7f07f80c7f60 msgr2=0x7f07f8070c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:10.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:10.116+0000 7f07ff765640 1 -- 172.21.15.57:0/2149301970 shutdown_connections 2023-10-30T14:51:10.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:10.116+0000 7f07ff765640 1 -- 172.21.15.57:0/2149301970 wait complete. 2023-10-30T14:51:10.138 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:51:12.035 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:51:12.039 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:51:12.139 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:51:12.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.280+0000 7f9b8559d640 1 Processor -- start 2023-10-30T14:51:12.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.280+0000 7f9b8559d640 1 -- start start 2023-10-30T14:51:12.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.280+0000 7f9b8559d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9b80153aa0 0x7f9b8015d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:12.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.280+0000 7f9b8559d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9b800701e0 0x7f9b800ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:12.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.280+0000 7f9b8559d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9b800cd420 0x7f9b8006b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:12.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.280+0000 7f9b8559d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b80070a50 con 0x7f9b800701e0 2023-10-30T14:51:12.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.280+0000 7f9b8559d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b8015dd80 con 0x7f9b80153aa0 2023-10-30T14:51:12.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.280+0000 7f9b8559d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b800c7b00 con 0x7f9b800cd420 2023-10-30T14:51:12.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.280+0000 7f9b77fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9b800701e0 0x7f9b800ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:12.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.280+0000 7f9b77fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9b800701e0 0x7f9b800ccee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51840/0 (socket says 172.21.15.57:51840) 2023-10-30T14:51:12.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.280+0000 7f9b77fff640 1 -- 172.21.15.57:0/670228133 learned_addr learned my addr 172.21.15.57:0/670228133 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:12.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.280+0000 7f9b7f7fe640 1 --2- 172.21.15.57:0/670228133 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9b800cd420 0x7f9b8006b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:12.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.280+0000 7f9b7effd640 1 --2- 172.21.15.57:0/670228133 >> v2:172.21.15.88:3300/0 conn(0x7f9b80153aa0 0x7f9b8015d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:12.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.280+0000 7f9b77fff640 1 -- 172.21.15.57:0/670228133 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9b800cd420 msgr2=0x7f9b8006b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:12.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.280+0000 7f9b77fff640 1 --2- 172.21.15.57:0/670228133 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9b800cd420 0x7f9b8006b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:12.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.280+0000 7f9b77fff640 1 -- 172.21.15.57:0/670228133 >> v2:172.21.15.88:3300/0 conn(0x7f9b80153aa0 msgr2=0x7f9b8015d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:12.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.280+0000 7f9b77fff640 1 --2- 172.21.15.57:0/670228133 >> v2:172.21.15.88:3300/0 conn(0x7f9b80153aa0 0x7f9b8015d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:12.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.280+0000 7f9b77fff640 1 -- 172.21.15.57:0/670228133 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b800c2050 con 0x7f9b800701e0 2023-10-30T14:51:12.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.284+0000 7f9b77fff640 1 --2- 172.21.15.57:0/670228133 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b800701e0 0x7f9b800ccee0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f9b800caae0 tx=0x7f9b6801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4f33c4aa62d4a04 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:12.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.284+0000 7f9b7e7fc640 1 -- 172.21.15.57:0/670228133 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b680a89b0 con 0x7f9b800701e0 2023-10-30T14:51:12.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.284+0000 7f9b7e7fc640 1 -- 172.21.15.57:0/670228133 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b680a8b10 con 0x7f9b800701e0 2023-10-30T14:51:12.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.284+0000 7f9b7e7fc640 1 -- 172.21.15.57:0/670228133 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b680a8ea0 con 0x7f9b800701e0 2023-10-30T14:51:12.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.284+0000 7f9b8559d640 1 -- 172.21.15.57:0/670228133 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b800701e0 msgr2=0x7f9b800ccee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:12.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.284+0000 7f9b8559d640 1 --2- 172.21.15.57:0/670228133 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b800701e0 0x7f9b800ccee0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f9b800caae0 tx=0x7f9b6801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:51:12.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.284+0000 7f9b8559d640 1 -- 172.21.15.57:0/670228133 shutdown_connections 2023-10-30T14:51:12.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.284+0000 7f9b8559d640 1 --2- 172.21.15.57:0/670228133 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9b800cd420 0x7f9b8006b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:12.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.284+0000 7f9b8559d640 1 --2- 172.21.15.57:0/670228133 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b800701e0 0x7f9b800ccee0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:12.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.284+0000 7f9b8559d640 1 --2- 172.21.15.57:0/670228133 >> v2:172.21.15.88:3300/0 conn(0x7f9b80153aa0 0x7f9b8015d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:12.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.284+0000 7f9b8559d640 1 -- 172.21.15.57:0/670228133 >> 172.21.15.57:0/670228133 conn(0x7f9b80071fb0 msgr2=0x7f9b800723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:12.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.284+0000 7f9b8559d640 1 -- 172.21.15.57:0/670228133 shutdown_connections 2023-10-30T14:51:12.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.284+0000 7f9b8559d640 1 -- 172.21.15.57:0/670228133 wait complete. 2023-10-30T14:51:12.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b8559d640 1 Processor -- start 2023-10-30T14:51:12.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b8559d640 1 -- start start 2023-10-30T14:51:12.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b8559d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9b800701e0 0x7f9b801564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:12.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b8559d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b800cd420 0x7f9b801eecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:12.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b8559d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9b80153aa0 0x7f9b801497c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:12.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b8559d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8006cf80 con 0x7f9b800cd420 2023-10-30T14:51:12.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b8559d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b801580b0 con 0x7f9b80153aa0 2023-10-30T14:51:12.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b8559d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b801ef7e0 con 0x7f9b800701e0 2023-10-30T14:51:12.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b77fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b800cd420 0x7f9b801eecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:12.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b77fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b800cd420 0x7f9b801eecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51854/0 (socket says 172.21.15.57:51854) 2023-10-30T14:51:12.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b77fff640 1 -- 172.21.15.57:0/242289029 learned_addr learned my addr 172.21.15.57:0/242289029 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:12.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b7effd640 1 --2- 172.21.15.57:0/242289029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9b800701e0 0x7f9b801564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:12.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b7f7fe640 1 --2- 172.21.15.57:0/242289029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9b80153aa0 0x7f9b801497c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:12.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b77fff640 1 -- 172.21.15.57:0/242289029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9b800701e0 msgr2=0x7f9b801564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:12.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b77fff640 1 --2- 172.21.15.57:0/242289029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9b800701e0 0x7f9b801564d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:12.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b77fff640 1 -- 172.21.15.57:0/242289029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9b80153aa0 msgr2=0x7f9b801497c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:12.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b77fff640 1 --2- 172.21.15.57:0/242289029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9b80153aa0 0x7f9b801497c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:12.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b77fff640 1 -- 172.21.15.57:0/242289029 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b680094d0 con 0x7f9b800cd420 2023-10-30T14:51:12.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b77fff640 1 --2- 172.21.15.57:0/242289029 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b800cd420 0x7f9b801eecd0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f9b680b3040 tx=0x7f9b6801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:12.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b777fe640 1 -- 172.21.15.57:0/242289029 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b680ba600 con 0x7f9b800cd420 2023-10-30T14:51:12.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b777fe640 1 -- 172.21.15.57:0/242289029 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b680ba760 con 0x7f9b800cd420 2023-10-30T14:51:12.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b8559d640 1 -- 172.21.15.57:0/242289029 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b80155cd0 con 0x7f9b800cd420 2023-10-30T14:51:12.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b777fe640 1 -- 172.21.15.57:0/242289029 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b680bab10 con 0x7f9b800cd420 2023-10-30T14:51:12.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.288+0000 7f9b8559d640 1 -- 172.21.15.57:0/242289029 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b8006cc90 con 0x7f9b800cd420 2023-10-30T14:51:12.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.292+0000 7f9b8559d640 1 -- 172.21.15.57:0/242289029 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b801561f0 con 0x7f9b800cd420 2023-10-30T14:51:12.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.292+0000 7f9b777fe640 1 -- 172.21.15.57:0/242289029 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9b680bac70 con 0x7f9b800cd420 2023-10-30T14:51:12.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.292+0000 7f9b777fe640 1 --2- 172.21.15.57:0/242289029 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9b600664a0 0x7f9b60068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:12.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.292+0000 7f9b777fe640 1 -- 172.21.15.57:0/242289029 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f9b68122b20 con 0x7f9b800cd420 2023-10-30T14:51:12.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.292+0000 7f9b7effd640 1 --2- 172.21.15.57:0/242289029 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9b600664a0 0x7f9b60068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:12.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.292+0000 7f9b7effd640 1 --2- 172.21.15.57:0/242289029 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9b600664a0 0x7f9b60068960 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9b7000b930 tx=0x7f9b70008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:12.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.296+0000 7f9b777fe640 1 -- 172.21.15.57:0/242289029 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9b801561f0 con 0x7f9b800cd420 2023-10-30T14:51:12.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.532+0000 7f9b8559d640 1 -- 172.21.15.57:0/242289029 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b801561f0 con 0x7f9b800cd420 2023-10-30T14:51:12.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.532+0000 7f9b777fe640 1 -- 172.21.15.57:0/242289029 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+5710 (secure 0 0 0) 0x7f9b801561f0 con 0x7f9b800cd420 2023-10-30T14:51:12.537 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:51:12.538 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:51:05.354063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8058":{"gid":8058,"name":"e","rank":0,"incarnation":185,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:51:12.538 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 187 2023-10-30T14:51:12.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.536+0000 7f9b8559d640 1 -- 172.21.15.57:0/242289029 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9b600664a0 msgr2=0x7f9b60068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:12.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.536+0000 7f9b8559d640 1 --2- 172.21.15.57:0/242289029 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9b600664a0 0x7f9b60068960 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9b7000b930 tx=0x7f9b70008040 comp rx=0 tx=0).stop 2023-10-30T14:51:12.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.536+0000 7f9b8559d640 1 -- 172.21.15.57:0/242289029 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b800cd420 msgr2=0x7f9b801eecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:12.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.536+0000 7f9b8559d640 1 --2- 172.21.15.57:0/242289029 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b800cd420 0x7f9b801eecd0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f9b680b3040 tx=0x7f9b6801ba90 comp rx=0 tx=0).stop 2023-10-30T14:51:12.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.536+0000 7f9b8559d640 1 -- 172.21.15.57:0/242289029 shutdown_connections 2023-10-30T14:51:12.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.536+0000 7f9b8559d640 1 --2- 172.21.15.57:0/242289029 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f9b600664a0 0x7f9b60068960 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:12.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.536+0000 7f9b8559d640 1 --2- 172.21.15.57:0/242289029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9b80153aa0 0x7f9b801497c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:12.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.536+0000 7f9b8559d640 1 --2- 172.21.15.57:0/242289029 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b800cd420 0x7f9b801eecd0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:12.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.536+0000 7f9b8559d640 1 --2- 172.21.15.57:0/242289029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9b800701e0 0x7f9b801564d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:12.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.536+0000 7f9b8559d640 1 -- 172.21.15.57:0/242289029 >> 172.21.15.57:0/242289029 conn(0x7f9b80071fb0 msgr2=0x7f9b80152600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:12.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.536+0000 7f9b8559d640 1 -- 172.21.15.57:0/242289029 shutdown_connections 2023-10-30T14:51:12.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:12.536+0000 7f9b8559d640 1 -- 172.21.15.57:0/242289029 wait complete. 2023-10-30T14:51:12.558 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:51:14.560 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:51:14.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.712+0000 7fbb054dc640 1 Processor -- start 2023-10-30T14:51:14.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.712+0000 7fbb054dc640 1 -- start start 2023-10-30T14:51:14.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.712+0000 7fbb054dc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbb000c8eb0 0x7fbb0015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:14.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.712+0000 7fbb054dc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbb0015ac70 0x7fbb0015a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:14.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.712+0000 7fbb054dc640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0015dcc0 con 0x7fbb0006aeb0 2023-10-30T14:51:14.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.712+0000 7fbb054dc640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb00159800 con 0x7fbb000c8eb0 2023-10-30T14:51:14.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.712+0000 7fbb054dc640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb00159d80 con 0x7fbb0015ac70 2023-10-30T14:51:14.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.712+0000 7fbafffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbb000c8eb0 0x7fbb0015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:14.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.712+0000 7fbafffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbb000c8eb0 0x7fbb0015cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42054/0 (socket says 172.21.15.57:42054) 2023-10-30T14:51:14.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.712+0000 7fbafffff640 1 -- 172.21.15.57:0/2617722472 learned_addr learned my addr 172.21.15.57:0/2617722472 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:14.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.712+0000 7fbaff7fe640 1 --2- 172.21.15.57:0/2617722472 >> v2:172.21.15.88:3300/0 conn(0x7fbb0015ac70 0x7fbb0015a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:14.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.712+0000 7fbafffff640 1 -- 172.21.15.57:0/2617722472 >> v2:172.21.15.88:3300/0 conn(0x7fbb0015ac70 msgr2=0x7fbb0015a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:14.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.712+0000 7fbafffff640 1 --2- 172.21.15.57:0/2617722472 >> v2:172.21.15.88:3300/0 conn(0x7fbb0015ac70 0x7fbb0015a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:14.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.712+0000 7fbafffff640 1 -- 172.21.15.57:0/2617722472 >> v1:172.21.15.88:6789/0 conn(0x7fbb0006aeb0 legacy=0x7fbb00069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:14.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.712+0000 7fbafffff640 1 -- 172.21.15.57:0/2617722472 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0015a750 con 0x7fbb000c8eb0 2023-10-30T14:51:14.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.716+0000 7fbafffff640 1 --2- 172.21.15.57:0/2617722472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb000c8eb0 0x7fbb0015cd70 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fbb000b4130 tx=0x7fbaf001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2afa9186a4e693c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:14.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.716+0000 7fbafeffd640 1 -- 172.21.15.57:0/2617722472 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf00a89f0 con 0x7fbb000c8eb0 2023-10-30T14:51:14.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.716+0000 7fbafeffd640 1 -- 172.21.15.57:0/2617722472 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf00a8b50 con 0x7fbb000c8eb0 2023-10-30T14:51:14.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.716+0000 7fbafeffd640 1 -- 172.21.15.57:0/2617722472 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf00b0440 con 0x7fbb000c8eb0 2023-10-30T14:51:14.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.716+0000 7fbb054dc640 1 -- 172.21.15.57:0/2617722472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb000c8eb0 msgr2=0x7fbb0015cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:14.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.716+0000 7fbb054dc640 1 --2- 172.21.15.57:0/2617722472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb000c8eb0 0x7fbb0015cd70 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fbb000b4130 tx=0x7fbaf001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:51:14.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.716+0000 7fbb054dc640 1 -- 172.21.15.57:0/2617722472 shutdown_connections 2023-10-30T14:51:14.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.716+0000 7fbb054dc640 1 --2- 172.21.15.57:0/2617722472 >> v2:172.21.15.88:3300/0 conn(0x7fbb0015ac70 0x7fbb0015a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:14.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.716+0000 7fbb054dc640 1 --2- 172.21.15.57:0/2617722472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb000c8eb0 0x7fbb0015cd70 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:14.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.716+0000 7fbb054dc640 1 -- 172.21.15.57:0/2617722472 >> 172.21.15.57:0/2617722472 conn(0x7fbb000cef00 msgr2=0x7fbb000cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:14.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.716+0000 7fbb054dc640 1 -- 172.21.15.57:0/2617722472 shutdown_connections 2023-10-30T14:51:14.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.716+0000 7fbb054dc640 1 -- 172.21.15.57:0/2617722472 wait complete. 2023-10-30T14:51:14.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbb054dc640 1 Processor -- start 2023-10-30T14:51:14.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbb054dc640 1 -- start start 2023-10-30T14:51:14.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbb054dc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb0006aeb0 0x7fbb00158a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:14.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbb054dc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb000c8eb0 0x7fbb000c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:14.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbb054dc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb0015ac70 0x7fbb0014f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:14.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbb054dc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb000c7850 con 0x7fbb0015ac70 2023-10-30T14:51:14.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbb054dc640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb000b3390 con 0x7fbb0006aeb0 2023-10-30T14:51:14.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbb054dc640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb000c7520 con 0x7fbb000c8eb0 2023-10-30T14:51:14.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbb04cdb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb0015ac70 0x7fbb0014f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:14.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbafffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb0006aeb0 0x7fbb00158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:14.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbaff7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb000c8eb0 0x7fbb000c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:14.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbafffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb0006aeb0 0x7fbb00158a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:52726/0 (socket says 172.21.15.57:52726) 2023-10-30T14:51:14.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbb04cdb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb0015ac70 0x7fbb0014f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42060/0 (socket says 172.21.15.57:42060) 2023-10-30T14:51:14.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbaff7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb000c8eb0 0x7fbb000c7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:43078/0 (socket says 172.21.15.57:43078) 2023-10-30T14:51:14.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbafffff640 1 -- 172.21.15.57:0/840787357 learned_addr learned my addr 172.21.15.57:0/840787357 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:14.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbb04cdb640 1 -- 172.21.15.57:0/840787357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb000c8eb0 msgr2=0x7fbb000c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:14.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbb04cdb640 1 --2- 172.21.15.57:0/840787357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb000c8eb0 0x7fbb000c7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:14.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbb04cdb640 1 -- 172.21.15.57:0/840787357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb0006aeb0 msgr2=0x7fbb00158a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:14.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbb04cdb640 1 --2- 172.21.15.57:0/840787357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb0006aeb0 0x7fbb00158a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:14.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbb04cdb640 1 -- 172.21.15.57:0/840787357 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf00094d0 con 0x7fbb0015ac70 2023-10-30T14:51:14.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbb04cdb640 1 --2- 172.21.15.57:0/840787357 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb0015ac70 0x7fbb0014f110 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fbaf800c700 tx=0x7fbaf800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:14.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbafcff9640 1 -- 172.21.15.57:0/840787357 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8003b80 con 0x7fbb0015ac70 2023-10-30T14:51:14.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbafcff9640 1 -- 172.21.15.57:0/840787357 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf8003ce0 con 0x7fbb0015ac70 2023-10-30T14:51:14.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbb054dc640 1 -- 172.21.15.57:0/840787357 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb000b2dd0 con 0x7fbb0015ac70 2023-10-30T14:51:14.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbafcff9640 1 -- 172.21.15.57:0/840787357 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8005310 con 0x7fbb0015ac70 2023-10-30T14:51:14.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbb054dc640 1 -- 172.21.15.57:0/840787357 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb001f11e0 con 0x7fbb0015ac70 2023-10-30T14:51:14.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.720+0000 7fbb054dc640 1 -- 172.21.15.57:0/840787357 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb000c7a00 con 0x7fbb0015ac70 2023-10-30T14:51:14.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.724+0000 7fbafcff9640 1 -- 172.21.15.57:0/840787357 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbaf8005470 con 0x7fbb0015ac70 2023-10-30T14:51:14.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.724+0000 7fbafcff9640 1 --2- 172.21.15.57:0/840787357 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbaec0664a0 0x7fbaec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:14.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.724+0000 7fbafcff9640 1 -- 172.21.15.57:0/840787357 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fbaf8083f60 con 0x7fbb0015ac70 2023-10-30T14:51:14.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.724+0000 7fbafffff640 1 --2- 172.21.15.57:0/840787357 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbaec0664a0 0x7fbaec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:14.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.724+0000 7fbafffff640 1 --2- 172.21.15.57:0/840787357 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbaec0664a0 0x7fbaec068960 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fbaf001ed30 tx=0x7fbaf0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:14.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.728+0000 7fbafcff9640 1 -- 172.21.15.57:0/840787357 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbb000c7a00 con 0x7fbb0015ac70 2023-10-30T14:51:14.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.916+0000 7fbb054dc640 1 -- 172.21.15.57:0/840787357 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb0015dcc0 con 0x7fbb0015ac70 2023-10-30T14:51:14.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.920+0000 7fbafcff9640 1 -- 172.21.15.57:0/840787357 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+5716 (secure 0 0 0) 0x7fbb0015dcc0 con 0x7fbb0015ac70 2023-10-30T14:51:14.923 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:51:14.923 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:51:12.479933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8058":{"gid":8058,"name":"e","rank":0,"incarnation":185,"state":"up:clientreplay","state_seq":36,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:51:14.923 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 188 2023-10-30T14:51:14.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.920+0000 7fbb054dc640 1 -- 172.21.15.57:0/840787357 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbaec0664a0 msgr2=0x7fbaec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:14.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.920+0000 7fbb054dc640 1 --2- 172.21.15.57:0/840787357 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbaec0664a0 0x7fbaec068960 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fbaf001ed30 tx=0x7fbaf0009040 comp rx=0 tx=0).stop 2023-10-30T14:51:14.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.920+0000 7fbb054dc640 1 -- 172.21.15.57:0/840787357 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb0015ac70 msgr2=0x7fbb0014f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:14.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.920+0000 7fbb054dc640 1 --2- 172.21.15.57:0/840787357 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb0015ac70 0x7fbb0014f110 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fbaf800c700 tx=0x7fbaf800cbc0 comp rx=0 tx=0).stop 2023-10-30T14:51:14.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.920+0000 7fbb054dc640 1 -- 172.21.15.57:0/840787357 shutdown_connections 2023-10-30T14:51:14.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.920+0000 7fbb054dc640 1 --2- 172.21.15.57:0/840787357 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbaec0664a0 0x7fbaec068960 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:14.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.920+0000 7fbb054dc640 1 --2- 172.21.15.57:0/840787357 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb0015ac70 0x7fbb0014f110 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:14.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.920+0000 7fbb054dc640 1 --2- 172.21.15.57:0/840787357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb000c8eb0 0x7fbb000c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:14.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.920+0000 7fbb054dc640 1 --2- 172.21.15.57:0/840787357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb0006aeb0 0x7fbb00158a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:14.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.920+0000 7fbb054dc640 1 -- 172.21.15.57:0/840787357 >> 172.21.15.57:0/840787357 conn(0x7fbb000cef00 msgr2=0x7fbb000ccce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:14.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.920+0000 7fbb054dc640 1 -- 172.21.15.57:0/840787357 shutdown_connections 2023-10-30T14:51:14.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:14.920+0000 7fbb054dc640 1 -- 172.21.15.57:0/840787357 wait complete. 2023-10-30T14:51:14.944 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 188, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T14:14:04.087067+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 188, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8058": { "addr": "172.21.15.57:6837/2594912006", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6833", "nonce": 2594912006, "type": "v2" }, { "addr": "172.21.15.57:6837", "nonce": 2594912006, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8058, "incarnation": 185, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:clientreplay", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:51:12.479933+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 8058 } } } ], "standbys": [ { "addr": "172.21.15.88:6835/3394468178", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3394468178, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3394468178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6837/1227890752", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 1227890752, "type": "v2" }, { "addr": "172.21.15.88:6837", "nonce": 1227890752, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.88:6839/1078372885", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6838", "nonce": 1078372885, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1078372885, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 170, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6716, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.57:6834/697507447", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 697507447, "type": "v2" }, { "addr": "172.21.15.57:6834", "nonce": 697507447, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 156, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7377, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:51:16.945 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:51:17.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.096+0000 7fad9330f640 1 Processor -- start 2023-10-30T14:51:17.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.096+0000 7fad9330f640 1 -- start start 2023-10-30T14:51:17.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.096+0000 7fad9330f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fad8c163a30 0x7fad8c166f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:17.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.096+0000 7fad9330f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fad8c06fdf0 0x7fad8c0c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:17.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.096+0000 7fad9330f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8c070c10 con 0x7fad8c0c5b30 2023-10-30T14:51:17.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.096+0000 7fad9330f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad8c071b70 con 0x7fad8c06fdf0 2023-10-30T14:51:17.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.096+0000 7fad9330f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c071f90 con 0x7fad8c163a30 2023-10-30T14:51:17.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.096+0000 7fad908aa640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fad8c06fdf0 0x7fad8c0c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:17.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.096+0000 7fad908aa640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fad8c06fdf0 0x7fad8c0c55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42074/0 (socket says 172.21.15.57:42074) 2023-10-30T14:51:17.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.096+0000 7fad908aa640 1 -- 172.21.15.57:0/1515986871 learned_addr learned my addr 172.21.15.57:0/1515986871 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:17.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.096+0000 7fad910ab640 1 --2- 172.21.15.57:0/1515986871 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fad8c163a30 0x7fad8c166f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:17.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.096+0000 7fad83fff640 1 -- 172.21.15.57:0/1515986871 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226214200 0 0) 0x7fad8c070c10 con 0x7fad8c0c5b30 2023-10-30T14:51:17.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.316+0000 7fad83fff640 1 -- 172.21.15.57:0/1515986871 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad78003510 con 0x7fad8c0c5b30 2023-10-30T14:51:17.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.316+0000 7fad908aa640 1 -- 172.21.15.57:0/1515986871 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fad8c163a30 msgr2=0x7fad8c166f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:17.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.316+0000 7fad908aa640 1 --2- 172.21.15.57:0/1515986871 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fad8c163a30 0x7fad8c166f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:17.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.316+0000 7fad908aa640 1 -- 172.21.15.57:0/1515986871 >> v1:172.21.15.57:6789/0 conn(0x7fad8c0c5b30 legacy=0x7fad8c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:17.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.316+0000 7fad908aa640 1 -- 172.21.15.57:0/1515986871 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8c070ef0 con 0x7fad8c06fdf0 2023-10-30T14:51:17.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.320+0000 7fad908aa640 1 --2- 172.21.15.57:0/1515986871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad8c06fdf0 0x7fad8c0c55f0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fad7c008910 tx=0x7fad7c027660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77e75ac40c544be4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:17.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.320+0000 7fad83fff640 1 -- 172.21.15.57:0/1515986871 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c0089e0 con 0x7fad8c06fdf0 2023-10-30T14:51:17.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.320+0000 7fad83fff640 1 -- 172.21.15.57:0/1515986871 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7c008b40 con 0x7fad8c06fdf0 2023-10-30T14:51:17.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.320+0000 7fad83fff640 1 -- 172.21.15.57:0/1515986871 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c00b170 con 0x7fad8c06fdf0 2023-10-30T14:51:17.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.320+0000 7fad9330f640 1 -- 172.21.15.57:0/1515986871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad8c06fdf0 msgr2=0x7fad8c0c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:17.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.320+0000 7fad9330f640 1 --2- 172.21.15.57:0/1515986871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad8c06fdf0 0x7fad8c0c55f0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fad7c008910 tx=0x7fad7c027660 comp rx=0 tx=0).stop 2023-10-30T14:51:17.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.320+0000 7fad9330f640 1 -- 172.21.15.57:0/1515986871 shutdown_connections 2023-10-30T14:51:17.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.320+0000 7fad9330f640 1 --2- 172.21.15.57:0/1515986871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad8c06fdf0 0x7fad8c0c55f0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:17.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.320+0000 7fad9330f640 1 --2- 172.21.15.57:0/1515986871 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fad8c163a30 0x7fad8c166f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:17.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.320+0000 7fad9330f640 1 -- 172.21.15.57:0/1515986871 >> 172.21.15.57:0/1515986871 conn(0x7fad8c0cac70 msgr2=0x7fad8c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:17.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.320+0000 7fad9330f640 1 -- 172.21.15.57:0/1515986871 shutdown_connections 2023-10-30T14:51:17.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.320+0000 7fad9330f640 1 -- 172.21.15.57:0/1515986871 wait complete. 2023-10-30T14:51:17.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.324+0000 7fad9330f640 1 Processor -- start 2023-10-30T14:51:17.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.324+0000 7fad9330f640 1 -- start start 2023-10-30T14:51:17.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.324+0000 7fad9330f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad8c06fdf0 0x7fad8c1fed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:17.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.324+0000 7fad9330f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad8c0c5b30 0x7fad8c160470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:17.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.324+0000 7fad9330f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fad8c163a30 0x7fad8c1fce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:17.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.324+0000 7fad9330f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c167cf0 con 0x7fad8c06fdf0 2023-10-30T14:51:17.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.324+0000 7fad9330f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c164fe0 con 0x7fad8c0c5b30 2023-10-30T14:51:17.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.324+0000 7fad9330f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c1ff240 con 0x7fad8c163a30 2023-10-30T14:51:17.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.324+0000 7fad910ab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad8c06fdf0 0x7fad8c1fed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:17.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.324+0000 7fad910ab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad8c06fdf0 0x7fad8c1fed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42084/0 (socket says 172.21.15.57:42084) 2023-10-30T14:51:17.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.324+0000 7fad910ab640 1 -- 172.21.15.57:0/3423485436 learned_addr learned my addr 172.21.15.57:0/3423485436 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:17.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.324+0000 7fad918ac640 1 --2- 172.21.15.57:0/3423485436 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fad8c163a30 0x7fad8c1fce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:17.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.324+0000 7fad908aa640 1 --2- 172.21.15.57:0/3423485436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad8c0c5b30 0x7fad8c160470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:17.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.324+0000 7fad910ab640 1 -- 172.21.15.57:0/3423485436 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fad8c163a30 msgr2=0x7fad8c1fce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:17.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.324+0000 7fad910ab640 1 --2- 172.21.15.57:0/3423485436 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fad8c163a30 0x7fad8c1fce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:17.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.324+0000 7fad910ab640 1 -- 172.21.15.57:0/3423485436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad8c0c5b30 msgr2=0x7fad8c160470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:17.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.324+0000 7fad910ab640 1 --2- 172.21.15.57:0/3423485436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad8c0c5b30 0x7fad8c160470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:17.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.324+0000 7fad910ab640 1 -- 172.21.15.57:0/3423485436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7c008540 con 0x7fad8c06fdf0 2023-10-30T14:51:17.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.328+0000 7fad910ab640 1 --2- 172.21.15.57:0/3423485436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad8c06fdf0 0x7fad8c1fed00 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fad8800ba20 tx=0x7fad8800bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:17.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.328+0000 7fad81ffb640 1 -- 172.21.15.57:0/3423485436 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8800d220 con 0x7fad8c06fdf0 2023-10-30T14:51:17.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.328+0000 7fad81ffb640 1 -- 172.21.15.57:0/3423485436 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad88008c20 con 0x7fad8c06fdf0 2023-10-30T14:51:17.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.328+0000 7fad9330f640 1 -- 172.21.15.57:0/3423485436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad8c164b70 con 0x7fad8c06fdf0 2023-10-30T14:51:17.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.328+0000 7fad81ffb640 1 -- 172.21.15.57:0/3423485436 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad88015420 con 0x7fad8c06fdf0 2023-10-30T14:51:17.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.328+0000 7fad9330f640 1 -- 172.21.15.57:0/3423485436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad8c167a10 con 0x7fad8c06fdf0 2023-10-30T14:51:17.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.328+0000 7fad9330f640 1 -- 172.21.15.57:0/3423485436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad8c04e170 con 0x7fad8c06fdf0 2023-10-30T14:51:17.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.328+0000 7fad81ffb640 1 -- 172.21.15.57:0/3423485436 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fad88015580 con 0x7fad8c06fdf0 2023-10-30T14:51:17.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.332+0000 7fad81ffb640 1 --2- 172.21.15.57:0/3423485436 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fad700664d0 0x7fad70068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:17.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.332+0000 7fad81ffb640 1 -- 172.21.15.57:0/3423485436 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fad880847b0 con 0x7fad8c06fdf0 2023-10-30T14:51:17.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.332+0000 7fad908aa640 1 --2- 172.21.15.57:0/3423485436 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fad700664d0 0x7fad70068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:17.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.332+0000 7fad908aa640 1 --2- 172.21.15.57:0/3423485436 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fad700664d0 0x7fad70068990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fad7c0b8000 tx=0x7fad7c0b3000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:17.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.336+0000 7fad81ffb640 1 -- 172.21.15.57:0/3423485436 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fad88051660 con 0x7fad8c06fdf0 2023-10-30T14:51:17.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.532+0000 7fad9330f640 1 -- 172.21.15.57:0/3423485436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad8c073d50 con 0x7fad8c06fdf0 2023-10-30T14:51:17.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.532+0000 7fad81ffb640 1 -- 172.21.15.57:0/3423485436 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+5716 (secure 0 0 0) 0x7fad88054c80 con 0x7fad8c06fdf0 2023-10-30T14:51:17.537 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:51:17.537 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:51:12.479933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8058":{"gid":8058,"name":"e","rank":0,"incarnation":185,"state":"up:clientreplay","state_seq":36,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:51:17.538 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 188 2023-10-30T14:51:17.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.536+0000 7fad9330f640 1 -- 172.21.15.57:0/3423485436 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fad700664d0 msgr2=0x7fad70068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:17.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.536+0000 7fad9330f640 1 --2- 172.21.15.57:0/3423485436 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fad700664d0 0x7fad70068990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fad7c0b8000 tx=0x7fad7c0b3000 comp rx=0 tx=0).stop 2023-10-30T14:51:17.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.536+0000 7fad9330f640 1 -- 172.21.15.57:0/3423485436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad8c06fdf0 msgr2=0x7fad8c1fed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:17.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.536+0000 7fad9330f640 1 --2- 172.21.15.57:0/3423485436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad8c06fdf0 0x7fad8c1fed00 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fad8800ba20 tx=0x7fad8800bee0 comp rx=0 tx=0).stop 2023-10-30T14:51:17.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.536+0000 7fad9330f640 1 -- 172.21.15.57:0/3423485436 shutdown_connections 2023-10-30T14:51:17.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.536+0000 7fad9330f640 1 --2- 172.21.15.57:0/3423485436 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fad700664d0 0x7fad70068990 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:17.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.536+0000 7fad9330f640 1 --2- 172.21.15.57:0/3423485436 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fad8c163a30 0x7fad8c1fce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:17.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.536+0000 7fad9330f640 1 --2- 172.21.15.57:0/3423485436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad8c0c5b30 0x7fad8c160470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:17.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.536+0000 7fad9330f640 1 --2- 172.21.15.57:0/3423485436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad8c06fdf0 0x7fad8c1fed00 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:17.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.536+0000 7fad9330f640 1 -- 172.21.15.57:0/3423485436 >> 172.21.15.57:0/3423485436 conn(0x7fad8c0cac70 msgr2=0x7fad8c08b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:17.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.536+0000 7fad9330f640 1 -- 172.21.15.57:0/3423485436 shutdown_connections 2023-10-30T14:51:17.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:17.536+0000 7fad9330f640 1 -- 172.21.15.57:0/3423485436 wait complete. 2023-10-30T14:51:17.558 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:51:19.559 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:51:19.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.700+0000 7fbd5ce9c640 1 Processor -- start 2023-10-30T14:51:19.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.700+0000 7fbd5ce9c640 1 -- start start 2023-10-30T14:51:19.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.700+0000 7fbd5ce9c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbd58151c20 0x7fbd580c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:19.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.700+0000 7fbd5ce9c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd580cb8b0 0x7fbd5806b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:19.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.700+0000 7fbd5ce9c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd580c5030 con 0x7fbd58151840 2023-10-30T14:51:19.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.700+0000 7fbd5ce9c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd58157830 con 0x7fbd58151c20 2023-10-30T14:51:19.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.700+0000 7fbd5ce9c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd58157b10 con 0x7fbd580cb8b0 2023-10-30T14:51:19.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.700+0000 7fbd55d9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbd58151c20 0x7fbd580c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:19.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.700+0000 7fbd55d9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbd58151c20 0x7fbd580c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42092/0 (socket says 172.21.15.57:42092) 2023-10-30T14:51:19.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.700+0000 7fbd55d9b640 1 -- 172.21.15.57:0/3128544846 learned_addr learned my addr 172.21.15.57:0/3128544846 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:19.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.700+0000 7fbd56d9d640 1 --2- 172.21.15.57:0/3128544846 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd580cb8b0 0x7fbd5806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:19.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.700+0000 7fbd55d9b640 1 -- 172.21.15.57:0/3128544846 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd580cb8b0 msgr2=0x7fbd5806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:19.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.700+0000 7fbd55d9b640 1 --2- 172.21.15.57:0/3128544846 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd580cb8b0 0x7fbd5806b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:19.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.700+0000 7fbd55d9b640 1 -- 172.21.15.57:0/3128544846 >> v1:172.21.15.88:6789/0 conn(0x7fbd58151840 legacy=0x7fbd58072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:19.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.700+0000 7fbd55d9b640 1 -- 172.21.15.57:0/3128544846 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd58157df0 con 0x7fbd58151c20 2023-10-30T14:51:19.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.700+0000 7fbd56d9d640 1 --2- 172.21.15.57:0/3128544846 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd580cb8b0 0x7fbd5806b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:19.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.704+0000 7fbd55d9b640 1 --2- 172.21.15.57:0/3128544846 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd58151c20 0x7fbd580c49a0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c009870 tx=0x7fbd4c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c882dcfaed0761c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:19.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.704+0000 7fbd5559a640 1 -- 172.21.15.57:0/3128544846 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c0a8a30 con 0x7fbd58151c20 2023-10-30T14:51:19.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.704+0000 7fbd5559a640 1 -- 172.21.15.57:0/3128544846 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4c0a8b90 con 0x7fbd58151c20 2023-10-30T14:51:19.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.704+0000 7fbd5559a640 1 -- 172.21.15.57:0/3128544846 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c0b0440 con 0x7fbd58151c20 2023-10-30T14:51:19.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.704+0000 7fbd5ce9c640 1 -- 172.21.15.57:0/3128544846 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd58151c20 msgr2=0x7fbd580c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:19.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.704+0000 7fbd5ce9c640 1 --2- 172.21.15.57:0/3128544846 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd58151c20 0x7fbd580c49a0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c009870 tx=0x7fbd4c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:51:19.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.704+0000 7fbd5ce9c640 1 -- 172.21.15.57:0/3128544846 shutdown_connections 2023-10-30T14:51:19.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.704+0000 7fbd5ce9c640 1 --2- 172.21.15.57:0/3128544846 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd580cb8b0 0x7fbd5806b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:19.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.704+0000 7fbd5ce9c640 1 --2- 172.21.15.57:0/3128544846 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd58151c20 0x7fbd580c49a0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:19.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.704+0000 7fbd5ce9c640 1 -- 172.21.15.57:0/3128544846 >> 172.21.15.57:0/3128544846 conn(0x7fbd580cac70 msgr2=0x7fbd580cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:19.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.704+0000 7fbd5ce9c640 1 -- 172.21.15.57:0/3128544846 shutdown_connections 2023-10-30T14:51:19.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.704+0000 7fbd5ce9c640 1 -- 172.21.15.57:0/3128544846 wait complete. 2023-10-30T14:51:19.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd5ce9c640 1 Processor -- start 2023-10-30T14:51:19.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd5ce9c640 1 -- start start 2023-10-30T14:51:19.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd5ce9c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd580cb8b0 0x7fbd5806fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:19.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd5ce9c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd58151840 0x7fbd58070330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:19.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd5ce9c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd58151c20 0x7fbd58153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:19.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd5ce9c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd581efb50 con 0x7fbd58151840 2023-10-30T14:51:19.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd5ce9c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd581536a0 con 0x7fbd58151c20 2023-10-30T14:51:19.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd5ce9c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd581e4730 con 0x7fbd580cb8b0 2023-10-30T14:51:19.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd55d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd58151840 0x7fbd58070330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:19.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd55d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd58151840 0x7fbd58070330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42098/0 (socket says 172.21.15.57:42098) 2023-10-30T14:51:19.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd55d9b640 1 -- 172.21.15.57:0/2306834147 learned_addr learned my addr 172.21.15.57:0/2306834147 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:19.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd5659c640 1 --2- 172.21.15.57:0/2306834147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd580cb8b0 0x7fbd5806fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:19.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd56d9d640 1 --2- 172.21.15.57:0/2306834147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd58151c20 0x7fbd58153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:19.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd55d9b640 1 -- 172.21.15.57:0/2306834147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd580cb8b0 msgr2=0x7fbd5806fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:19.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd55d9b640 1 --2- 172.21.15.57:0/2306834147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd580cb8b0 0x7fbd5806fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:19.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd55d9b640 1 -- 172.21.15.57:0/2306834147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd58151c20 msgr2=0x7fbd58153d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:19.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd55d9b640 1 --2- 172.21.15.57:0/2306834147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd58151c20 0x7fbd58153d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:19.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd55d9b640 1 -- 172.21.15.57:0/2306834147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4c0094d0 con 0x7fbd58151840 2023-10-30T14:51:19.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd55d9b640 1 --2- 172.21.15.57:0/2306834147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd58151840 0x7fbd58070330 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c01d3e0 tx=0x7fbd4c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:19.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd46ffd640 1 -- 172.21.15.57:0/2306834147 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c0ba600 con 0x7fbd58151840 2023-10-30T14:51:19.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd46ffd640 1 -- 172.21.15.57:0/2306834147 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4c0ba780 con 0x7fbd58151840 2023-10-30T14:51:19.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd5ce9c640 1 -- 172.21.15.57:0/2306834147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd581e4170 con 0x7fbd58151840 2023-10-30T14:51:19.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd46ffd640 1 -- 172.21.15.57:0/2306834147 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c0bab30 con 0x7fbd58151840 2023-10-30T14:51:19.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.708+0000 7fbd5ce9c640 1 -- 172.21.15.57:0/2306834147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd581520d0 con 0x7fbd58151840 2023-10-30T14:51:19.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.712+0000 7fbd5ce9c640 1 -- 172.21.15.57:0/2306834147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5804e170 con 0x7fbd58151840 2023-10-30T14:51:19.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.712+0000 7fbd46ffd640 1 -- 172.21.15.57:0/2306834147 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbd4c0bac90 con 0x7fbd58151840 2023-10-30T14:51:19.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.712+0000 7fbd46ffd640 1 --2- 172.21.15.57:0/2306834147 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd380664a0 0x7fbd38068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:19.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.712+0000 7fbd46ffd640 1 -- 172.21.15.57:0/2306834147 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fbd4c1229f0 con 0x7fbd58151840 2023-10-30T14:51:19.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.712+0000 7fbd5659c640 1 --2- 172.21.15.57:0/2306834147 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd380664a0 0x7fbd38068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:19.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.712+0000 7fbd5659c640 1 --2- 172.21.15.57:0/2306834147 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd380664a0 0x7fbd38068960 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fbd48007fd0 tx=0x7fbd48009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:19.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.716+0000 7fbd46ffd640 1 -- 172.21.15.57:0/2306834147 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbd4c0ef870 con 0x7fbd58151840 2023-10-30T14:51:19.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.908+0000 7fbd5ce9c640 1 -- 172.21.15.57:0/2306834147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd58155060 con 0x7fbd58151840 2023-10-30T14:51:19.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.908+0000 7fbd46ffd640 1 -- 172.21.15.57:0/2306834147 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) v1 ==== 95+0+5710 (secure 0 0 0) 0x7fbd4c0f2e90 con 0x7fbd58151840 2023-10-30T14:51:19.914 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:51:19.915 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":189,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:51:19.061797+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8058":{"gid":8058,"name":"e","rank":0,"incarnation":185,"state":"up:active","state_seq":38,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:51:19.915 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 189 2023-10-30T14:51:19.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.912+0000 7fbd5ce9c640 1 -- 172.21.15.57:0/2306834147 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd380664a0 msgr2=0x7fbd38068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:19.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.912+0000 7fbd5ce9c640 1 --2- 172.21.15.57:0/2306834147 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd380664a0 0x7fbd38068960 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fbd48007fd0 tx=0x7fbd48009040 comp rx=0 tx=0).stop 2023-10-30T14:51:19.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.912+0000 7fbd5ce9c640 1 -- 172.21.15.57:0/2306834147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd58151840 msgr2=0x7fbd58070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:19.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.912+0000 7fbd5ce9c640 1 --2- 172.21.15.57:0/2306834147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd58151840 0x7fbd58070330 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c01d3e0 tx=0x7fbd4c01bad0 comp rx=0 tx=0).stop 2023-10-30T14:51:19.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.912+0000 7fbd5ce9c640 1 -- 172.21.15.57:0/2306834147 shutdown_connections 2023-10-30T14:51:19.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.912+0000 7fbd5ce9c640 1 --2- 172.21.15.57:0/2306834147 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbd380664a0 0x7fbd38068960 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:19.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.912+0000 7fbd5ce9c640 1 --2- 172.21.15.57:0/2306834147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd58151c20 0x7fbd58153d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:19.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.912+0000 7fbd5ce9c640 1 --2- 172.21.15.57:0/2306834147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd58151840 0x7fbd58070330 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:19.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.912+0000 7fbd5ce9c640 1 --2- 172.21.15.57:0/2306834147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd580cb8b0 0x7fbd5806fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:19.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.912+0000 7fbd5ce9c640 1 -- 172.21.15.57:0/2306834147 >> 172.21.15.57:0/2306834147 conn(0x7fbd580cac70 msgr2=0x7fbd580c63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:19.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.912+0000 7fbd5ce9c640 1 -- 172.21.15.57:0/2306834147 shutdown_connections 2023-10-30T14:51:19.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:19.912+0000 7fbd5ce9c640 1 -- 172.21.15.57:0/2306834147 wait complete. 2023-10-30T14:51:19.935 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=7911 2023-10-30T14:51:19.936 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.c 2023-10-30T14:51:24.658 INFO:tasks.workunit.client.0.smithi057.stderr:error: unable to create file src/common/ceph_argparse.h: File exists 2023-10-30T14:51:25.291 INFO:tasks.ceph.mds.e.smithi057.stderr:2023-10-30T14:51:25.288+0000 7fbbfd486640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T14:51:28.936 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T14:51:28.937 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T14:51:28.937 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T14:51:28.941 INFO:tasks.ceph.mds.c:Started 2023-10-30T14:51:28.941 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:51:29.036 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-30T14:51:29.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.084+0000 7f254ff80640 1 Processor -- start 2023-10-30T14:51:29.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.084+0000 7f254ff80640 1 -- start start 2023-10-30T14:51:29.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.088+0000 7f254ff80640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2548153c80 0x7f254815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:29.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.088+0000 7f254ff80640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f25480c4a10 0x7f25480c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:29.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.088+0000 7f254ff80640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f254815a3b0 0x7f25480ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:29.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.088+0000 7f254ff80640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f254815dec0 con 0x7f25480c4a10 2023-10-30T14:51:29.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.088+0000 7f254ff80640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25481599a0 con 0x7f254815a3b0 2023-10-30T14:51:29.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.088+0000 7f254ff80640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2548159f20 con 0x7f2548153c80 2023-10-30T14:51:29.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.088+0000 7f254d51b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f25480c4a10 0x7f25480c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:29.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.088+0000 7f254d51b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f25480c4a10 0x7f25480c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52558/0 (socket says 172.21.15.57:52558) 2023-10-30T14:51:29.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.088+0000 7f254d51b640 1 -- 172.21.15.57:0/1158519316 learned_addr learned my addr 172.21.15.57:0/1158519316 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:29.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.088+0000 7f254dd1c640 1 --2- 172.21.15.57:0/1158519316 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2548153c80 0x7f254815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:29.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.088+0000 7f254e51d640 1 --2- 172.21.15.57:0/1158519316 >> v2:172.21.15.88:3300/0 conn(0x7f254815a3b0 0x7f25480ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:29.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.088+0000 7f254d51b640 1 -- 172.21.15.57:0/1158519316 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2548153c80 msgr2=0x7f254815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:29.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.088+0000 7f254d51b640 1 --2- 172.21.15.57:0/1158519316 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2548153c80 0x7f254815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:29.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.088+0000 7f254d51b640 1 -- 172.21.15.57:0/1158519316 >> v2:172.21.15.88:3300/0 conn(0x7f254815a3b0 msgr2=0x7f25480ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:29.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.088+0000 7f254d51b640 1 --2- 172.21.15.57:0/1158519316 >> v2:172.21.15.88:3300/0 conn(0x7f254815a3b0 0x7f25480ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:29.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.088+0000 7f254d51b640 1 -- 172.21.15.57:0/1158519316 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25480c47e0 con 0x7f25480c4a10 2023-10-30T14:51:29.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.088+0000 7f254d51b640 1 --2- 172.21.15.57:0/1158519316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25480c4a10 0x7f25480c42a0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f25480cbbc0 tx=0x7f253801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=488f48d9cbb3ac65 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:29.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.088+0000 7f254cd1a640 1 -- 172.21.15.57:0/1158519316 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25380a89b0 con 0x7f25480c4a10 2023-10-30T14:51:29.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254cd1a640 1 -- 172.21.15.57:0/1158519316 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25380a8b10 con 0x7f25480c4a10 2023-10-30T14:51:29.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254cd1a640 1 -- 172.21.15.57:0/1158519316 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25380a8ea0 con 0x7f25480c4a10 2023-10-30T14:51:29.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254ff80640 1 -- 172.21.15.57:0/1158519316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25480c4a10 msgr2=0x7f25480c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:29.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254ff80640 1 --2- 172.21.15.57:0/1158519316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25480c4a10 0x7f25480c42a0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f25480cbbc0 tx=0x7f253801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:51:29.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254ff80640 1 -- 172.21.15.57:0/1158519316 shutdown_connections 2023-10-30T14:51:29.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254ff80640 1 --2- 172.21.15.57:0/1158519316 >> v2:172.21.15.88:3300/0 conn(0x7f254815a3b0 0x7f25480ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:29.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254ff80640 1 --2- 172.21.15.57:0/1158519316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25480c4a10 0x7f25480c42a0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:29.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254ff80640 1 --2- 172.21.15.57:0/1158519316 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2548153c80 0x7f254815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:29.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254ff80640 1 -- 172.21.15.57:0/1158519316 >> 172.21.15.57:0/1158519316 conn(0x7f25480715c0 msgr2=0x7f25480719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:29.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254ff80640 1 -- 172.21.15.57:0/1158519316 shutdown_connections 2023-10-30T14:51:29.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254ff80640 1 -- 172.21.15.57:0/1158519316 wait complete. 2023-10-30T14:51:29.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254ff80640 1 Processor -- start 2023-10-30T14:51:29.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254ff80640 1 -- start start 2023-10-30T14:51:29.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254ff80640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25480c4a10 0x7f2548151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:29.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254ff80640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2548153c80 0x7f25480b3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:29.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254ff80640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f254815a3b0 0x7f2548149b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:29.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254ff80640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25480b2b70 con 0x7f2548153c80 2023-10-30T14:51:29.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254ff80640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25481eea50 con 0x7f254815a3b0 2023-10-30T14:51:29.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254ff80640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25480b2e30 con 0x7f25480c4a10 2023-10-30T14:51:29.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254d51b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2548153c80 0x7f25480b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:29.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254d51b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2548153c80 0x7f25480b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52560/0 (socket says 172.21.15.57:52560) 2023-10-30T14:51:29.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.092+0000 7f254d51b640 1 -- 172.21.15.57:0/1029280958 learned_addr learned my addr 172.21.15.57:0/1029280958 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:29.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.096+0000 7f254dd1c640 1 --2- 172.21.15.57:0/1029280958 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25480c4a10 0x7f2548151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:29.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.096+0000 7f254e51d640 1 --2- 172.21.15.57:0/1029280958 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f254815a3b0 0x7f2548149b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:29.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.096+0000 7f254d51b640 1 -- 172.21.15.57:0/1029280958 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25480c4a10 msgr2=0x7f2548151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:29.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.096+0000 7f254d51b640 1 --2- 172.21.15.57:0/1029280958 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25480c4a10 0x7f2548151fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:29.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.096+0000 7f254d51b640 1 -- 172.21.15.57:0/1029280958 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f254815a3b0 msgr2=0x7f2548149b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:29.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.096+0000 7f254d51b640 1 --2- 172.21.15.57:0/1029280958 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f254815a3b0 0x7f2548149b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:29.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.096+0000 7f254d51b640 1 -- 172.21.15.57:0/1029280958 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25380094d0 con 0x7f2548153c80 2023-10-30T14:51:29.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.096+0000 7f254e51d640 1 --2- 172.21.15.57:0/1029280958 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f254815a3b0 0x7f2548149b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:29.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.096+0000 7f254d51b640 1 --2- 172.21.15.57:0/1029280958 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2548153c80 0x7f25480b3470 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f253801d3a0 tx=0x7f253801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:29.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.096+0000 7f253e7fc640 1 -- 172.21.15.57:0/1029280958 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25380ba600 con 0x7f2548153c80 2023-10-30T14:51:29.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.096+0000 7f253e7fc640 1 -- 172.21.15.57:0/1029280958 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25380ba760 con 0x7f2548153c80 2023-10-30T14:51:29.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.096+0000 7f253e7fc640 1 -- 172.21.15.57:0/1029280958 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25380bab10 con 0x7f2548153c80 2023-10-30T14:51:29.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.096+0000 7f254ff80640 1 -- 172.21.15.57:0/1029280958 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25481eed10 con 0x7f2548153c80 2023-10-30T14:51:29.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.096+0000 7f254ff80640 1 -- 172.21.15.57:0/1029280958 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25481ef2d0 con 0x7f2548153c80 2023-10-30T14:51:29.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.096+0000 7f254ff80640 1 -- 172.21.15.57:0/1029280958 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f254804e170 con 0x7f2548153c80 2023-10-30T14:51:29.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.096+0000 7f253e7fc640 1 -- 172.21.15.57:0/1029280958 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f25380bac70 con 0x7f2548153c80 2023-10-30T14:51:29.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.096+0000 7f253e7fc640 1 --2- 172.21.15.57:0/1029280958 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f252c066520 0x7f252c0689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:29.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.096+0000 7f253e7fc640 1 -- 172.21.15.57:0/1029280958 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f2538122a80 con 0x7f2548153c80 2023-10-30T14:51:29.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.100+0000 7f254dd1c640 1 --2- 172.21.15.57:0/1029280958 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f252c066520 0x7f252c0689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:29.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.100+0000 7f254dd1c640 1 --2- 172.21.15.57:0/1029280958 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f252c066520 0x7f252c0689e0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f2544009ae0 tx=0x7f2544009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:29.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.104+0000 7f253e7fc640 1 -- 172.21.15.57:0/1029280958 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f25380ef7d0 con 0x7f2548153c80 2023-10-30T14:51:29.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.296+0000 7f254ff80640 1 -- 172.21.15.57:0/1029280958 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25481ef4c0 con 0x7f2548153c80 2023-10-30T14:51:29.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.296+0000 7f253e7fc640 1 -- 172.21.15.57:0/1029280958 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) v1 ==== 95+0+5710 (secure 0 0 0) 0x7f25380b5030 con 0x7f2548153c80 2023-10-30T14:51:29.300 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:51:29.300 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":189,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:51:19.061797+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8058":{"gid":8058,"name":"e","rank":0,"incarnation":185,"state":"up:active","state_seq":38,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:51:29.301 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 189 2023-10-30T14:51:29.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.296+0000 7f254ff80640 1 -- 172.21.15.57:0/1029280958 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f252c066520 msgr2=0x7f252c0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:29.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.296+0000 7f254ff80640 1 --2- 172.21.15.57:0/1029280958 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f252c066520 0x7f252c0689e0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f2544009ae0 tx=0x7f2544009190 comp rx=0 tx=0).stop 2023-10-30T14:51:29.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.296+0000 7f254ff80640 1 -- 172.21.15.57:0/1029280958 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2548153c80 msgr2=0x7f25480b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:29.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.296+0000 7f254ff80640 1 --2- 172.21.15.57:0/1029280958 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2548153c80 0x7f25480b3470 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f253801d3a0 tx=0x7f253801ba90 comp rx=0 tx=0).stop 2023-10-30T14:51:29.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.296+0000 7f254ff80640 1 -- 172.21.15.57:0/1029280958 shutdown_connections 2023-10-30T14:51:29.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.296+0000 7f254ff80640 1 --2- 172.21.15.57:0/1029280958 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f252c066520 0x7f252c0689e0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:29.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.296+0000 7f254ff80640 1 --2- 172.21.15.57:0/1029280958 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f254815a3b0 0x7f2548149b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:29.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.296+0000 7f254ff80640 1 --2- 172.21.15.57:0/1029280958 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2548153c80 0x7f25480b3470 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:29.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.296+0000 7f254ff80640 1 --2- 172.21.15.57:0/1029280958 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25480c4a10 0x7f2548151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:29.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.296+0000 7f254ff80640 1 -- 172.21.15.57:0/1029280958 >> 172.21.15.57:0/1029280958 conn(0x7f25480715c0 msgr2=0x7f2548158450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:29.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.296+0000 7f254ff80640 1 -- 172.21.15.57:0/1029280958 shutdown_connections 2023-10-30T14:51:29.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:29.296+0000 7f254ff80640 1 -- 172.21.15.57:0/1029280958 wait complete. 2023-10-30T14:51:29.321 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T14:51:29.593 INFO:tasks.workunit.client.0.smithi057.stderr:Updating files: 19% (56/291) Updating files: 20% (59/291) Updating files: 21% (62/291) Updating files: 22% (65/291) Updating files: 23% (67/291) Updating files: 24% (70/291) Updating files: 25% (73/291) Updating files: 26% (76/291) Updating files: 27% (79/291) Updating files: 27% (80/291) Updating files: 28% (82/291) Updating files: 29% (85/291) Updating files: 30% (88/291) Updating files: 31% (91/291) Updating files: 32% (94/291) Updating files: 33% (97/291) Updating files: 34% (99/291) Updating files: 35% (102/291) Updating files: 36% (105/291) Updating files: 37% (108/291) Updating files: 38% (111/291) Updating files: 39% (114/291) Updating files: 40% (117/291) Updating files: 41% (120/291) Updating files: 42% (123/291) Updating files: 43% (126/291) Updating files: 44% (129/291) Updating files: 45% (131/291) Updating files: 46% (134/291) Updating files: 47% (137/291) Updating files: 48% (140/291) Updating files: 49% (143/291) Updating files: 50% (146/291) Updating files: 51% (149/291) Updating files: 52% (152/291) Updating files: 53% (155/291) Updating files: 54% (158/291) Updating files: 55% (161/291) Updating files: 56% (163/291) Updating files: 56% (164/291) Updating files: 57% (166/291) Updating files: 58% (169/291) Updating files: 59% (172/291) Updating files: 60% (175/291) Updating files: 61% (178/291) Updating files: 62% (181/291) Updating files: 63% (184/291) Updating files: 64% (187/291) Updating files: 65% (190/291) Updating files: 66% (193/291) Updating files: 67% (195/291) Updating files: 68% (198/291) Updating files: 69% (201/291) Updating files: 70% (204/291) Updating files: 71% (207/291) Updating files: 72% (210/291) Updating files: 73% (213/291) Updating files: 74% (216/291) Updating files: 75% (219/291) Updating files: 76% (222/291) Updating files: 77% (225/291) Updating files: 78% (227/291) Updating files: 79% (230/291) Updating files: 80% (233/291) Updating files: 80% (235/291) Updating files: 81% (236/291) Updating files: 82% (239/291) Updating files: 83% (242/291) Updating files: 84% (245/291) Updating files: 85% (248/291) Updating files: 86% (251/291) Updating files: 87% (254/291) Updating files: 88% (257/291) Updating files: 89% (259/291) Updating files: 90% (262/291) Updating files: 91% (265/291) Updating files: 92% (268/291) Updating files: 92% (269/291) Updating files: 93% (271/291) Updating files: 94% (274/291) Updating files: 95% (277/291) Updating files: 96% (280/291) Updating files: 97% (283/291) Updating files: 98% (286/291) Updating files: 99% (289/291) Updating files: 100% (291/291) Updating files: 100% (291/291), done. 2023-10-30T14:51:29.594 INFO:tasks.workunit.client.0.smithi057.stderr:fatal: Could not reset index file to revision 'v0.75'. 2023-10-30T14:51:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 128 2023-10-30T14:51:29.603 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2023-10-30T14:51:29.603 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-30T14:51:30.497 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi057 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-10-30T14:51:30.667 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=df3faa8d3a3b47ceb97651a5cf1753a5 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi057 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-10-30T14:51:30.670 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2023-10-30T14:51:30.682 INFO:tasks.mds_thrash:joining mds_thrasher 2023-10-30T14:51:31.322 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:51:31.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.472+0000 7f354f046640 1 Processor -- start 2023-10-30T14:51:31.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.472+0000 7f354f046640 1 -- start start 2023-10-30T14:51:31.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.476+0000 7f354f046640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f354815d840 0x7f35480ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:31.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.476+0000 7f354f046640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f35480cd420 0x7f354806b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:31.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.476+0000 7f354f046640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35480c7b00 con 0x7f3548153aa0 2023-10-30T14:51:31.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.476+0000 7f354f046640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3548070a50 con 0x7f35480cd420 2023-10-30T14:51:31.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.476+0000 7f354f046640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35480c2050 con 0x7f354815d840 2023-10-30T14:51:31.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.476+0000 7f354d5e3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f35480cd420 0x7f354806b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:31.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.476+0000 7f354cde2640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f3548153aa0 0x7f35480701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:56444/0 (socket says 172.21.15.57:56444) 2023-10-30T14:51:31.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.476+0000 7f354cde2640 1 -- 172.21.15.57:0/989078720 learned_addr learned my addr 172.21.15.57:0/989078720 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:51:31.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.476+0000 7f3547fff640 1 --2- 172.21.15.57:0/989078720 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f354815d840 0x7f35480ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:31.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.476+0000 7f354d5e3640 1 -- 172.21.15.57:0/989078720 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f354815d840 msgr2=0x7f35480ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:31.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.476+0000 7f354d5e3640 1 --2- 172.21.15.57:0/989078720 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f354815d840 0x7f35480ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:31.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.476+0000 7f354d5e3640 1 -- 172.21.15.57:0/989078720 >> v1:172.21.15.57:6789/0 conn(0x7f3548153aa0 legacy=0x7f35480701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:31.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.476+0000 7f354d5e3640 1 -- 172.21.15.57:0/989078720 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35480c2330 con 0x7f35480cd420 2023-10-30T14:51:31.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.476+0000 7f35477fe640 1 -- 172.21.15.57:0/989078720 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650071931 0 0) 0x7f35480c7b00 con 0x7f3548153aa0 2023-10-30T14:51:31.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.480+0000 7f354d5e3640 1 --2- 172.21.15.57:0/989078720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35480cd420 0x7f354806b3b0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f3540009d90 tx=0x7f354001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467b809492807752 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:31.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.480+0000 7f35477fe640 1 -- 172.21.15.57:0/989078720 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35400af600 con 0x7f35480cd420 2023-10-30T14:51:31.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.480+0000 7f35477fe640 1 -- 172.21.15.57:0/989078720 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35400ab050 con 0x7f35480cd420 2023-10-30T14:51:31.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.480+0000 7f35477fe640 1 -- 172.21.15.57:0/989078720 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35400af980 con 0x7f35480cd420 2023-10-30T14:51:31.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.480+0000 7f354f046640 1 -- 172.21.15.57:0/989078720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35480cd420 msgr2=0x7f354806b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:31.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.480+0000 7f354f046640 1 --2- 172.21.15.57:0/989078720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35480cd420 0x7f354806b3b0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f3540009d90 tx=0x7f354001f130 comp rx=0 tx=0).stop 2023-10-30T14:51:31.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.480+0000 7f354f046640 1 -- 172.21.15.57:0/989078720 shutdown_connections 2023-10-30T14:51:31.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.480+0000 7f354f046640 1 --2- 172.21.15.57:0/989078720 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f354815d840 0x7f35480ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:31.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.480+0000 7f354f046640 1 --2- 172.21.15.57:0/989078720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35480cd420 0x7f354806b3b0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:31.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.480+0000 7f354f046640 1 -- 172.21.15.57:0/989078720 >> 172.21.15.57:0/989078720 conn(0x7f3548071fb0 msgr2=0x7f35480723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:31.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.480+0000 7f354f046640 1 -- 172.21.15.57:0/989078720 shutdown_connections 2023-10-30T14:51:31.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.480+0000 7f354f046640 1 -- 172.21.15.57:0/989078720 wait complete. 2023-10-30T14:51:31.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.480+0000 7f354f046640 1 Processor -- start 2023-10-30T14:51:31.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.480+0000 7f354f046640 1 -- start start 2023-10-30T14:51:31.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f354f046640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35480cd420 0x7f354814b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:31.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f354f046640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3548153aa0 0x7f35481ef5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:31.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f354f046640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f354815d840 0x7f35481514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:31.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f354f046640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35480ca070 con 0x7f35480cd420 2023-10-30T14:51:31.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f354f046640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35481f28b0 con 0x7f354815d840 2023-10-30T14:51:31.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f354f046640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f354814bf30 con 0x7f3548153aa0 2023-10-30T14:51:31.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f354cde2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35480cd420 0x7f354814b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:31.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f354cde2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35480cd420 0x7f354814b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52572/0 (socket says 172.21.15.57:52572) 2023-10-30T14:51:31.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f354cde2640 1 -- 172.21.15.57:0/2413394622 learned_addr learned my addr 172.21.15.57:0/2413394622 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:31.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f3547fff640 1 --2- 172.21.15.57:0/2413394622 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3548153aa0 0x7f35481ef5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:31.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f354d5e3640 1 --2- 172.21.15.57:0/2413394622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f354815d840 0x7f35481514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:31.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f354cde2640 1 -- 172.21.15.57:0/2413394622 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3548153aa0 msgr2=0x7f35481ef5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:31.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f354cde2640 1 --2- 172.21.15.57:0/2413394622 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3548153aa0 0x7f35481ef5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:31.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f354cde2640 1 -- 172.21.15.57:0/2413394622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f354815d840 msgr2=0x7f35481514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:31.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f354cde2640 1 --2- 172.21.15.57:0/2413394622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f354815d840 0x7f35481514f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:31.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f354cde2640 1 -- 172.21.15.57:0/2413394622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3540009a70 con 0x7f35480cd420 2023-10-30T14:51:31.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f354cde2640 1 --2- 172.21.15.57:0/2413394622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35480cd420 0x7f354814b9f0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f353c00c520 tx=0x7f353c00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:31.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f35457fa640 1 -- 172.21.15.57:0/2413394622 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c0074e0 con 0x7f35480cd420 2023-10-30T14:51:31.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f35457fa640 1 -- 172.21.15.57:0/2413394622 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353c007640 con 0x7f35480cd420 2023-10-30T14:51:31.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f35457fa640 1 -- 172.21.15.57:0/2413394622 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c0049f0 con 0x7f35480cd420 2023-10-30T14:51:31.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f354f046640 1 -- 172.21.15.57:0/2413394622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35481efae0 con 0x7f35480cd420 2023-10-30T14:51:31.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f354f046640 1 -- 172.21.15.57:0/2413394622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35480ca5a0 con 0x7f35480cd420 2023-10-30T14:51:31.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f354f046640 1 -- 172.21.15.57:0/2413394622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f354804e170 con 0x7f35480cd420 2023-10-30T14:51:31.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.484+0000 7f35457fa640 1 -- 172.21.15.57:0/2413394622 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f353c004b50 con 0x7f35480cd420 2023-10-30T14:51:31.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.488+0000 7f35457fa640 1 --2- 172.21.15.57:0/2413394622 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f352c0664d0 0x7f352c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:31.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.488+0000 7f35457fa640 1 -- 172.21.15.57:0/2413394622 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f353c0836b0 con 0x7f35480cd420 2023-10-30T14:51:31.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.488+0000 7f3547fff640 1 --2- 172.21.15.57:0/2413394622 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f352c0664d0 0x7f352c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:31.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.488+0000 7f3547fff640 1 --2- 172.21.15.57:0/2413394622 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f352c0664d0 0x7f352c068990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f35481ec7a0 tx=0x7f3534008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:31.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.492+0000 7f35457fa640 1 -- 172.21.15.57:0/2413394622 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f353c050550 con 0x7f35480cd420 2023-10-30T14:51:31.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.680+0000 7f354f046640 1 -- 172.21.15.57:0/2413394622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35480b2fd0 con 0x7f35480cd420 2023-10-30T14:51:31.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.680+0000 7f35457fa640 1 -- 172.21.15.57:0/2413394622 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+6502 (secure 0 0 0) 0x7f353c018090 con 0x7f35480cd420 2023-10-30T14:51:31.684 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:51:31.685 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":8211,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/3751121680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":3751121680},{"type":"v1","addr":"172.21.15.57:6836","nonce":3751121680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:51:19.061797+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8058":{"gid":8058,"name":"e","rank":0,"incarnation":185,"state":"up:active","state_seq":38,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:51:31.686 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 190 2023-10-30T14:51:31.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.680+0000 7f354f046640 1 -- 172.21.15.57:0/2413394622 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f352c0664d0 msgr2=0x7f352c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:31.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.680+0000 7f354f046640 1 --2- 172.21.15.57:0/2413394622 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f352c0664d0 0x7f352c068990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f35481ec7a0 tx=0x7f3534008040 comp rx=0 tx=0).stop 2023-10-30T14:51:31.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.680+0000 7f354f046640 1 -- 172.21.15.57:0/2413394622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35480cd420 msgr2=0x7f354814b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:31.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.680+0000 7f354f046640 1 --2- 172.21.15.57:0/2413394622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35480cd420 0x7f354814b9f0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f353c00c520 tx=0x7f353c00c9e0 comp rx=0 tx=0).stop 2023-10-30T14:51:31.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.680+0000 7f354f046640 1 -- 172.21.15.57:0/2413394622 shutdown_connections 2023-10-30T14:51:31.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.680+0000 7f354f046640 1 --2- 172.21.15.57:0/2413394622 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f352c0664d0 0x7f352c068990 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:31.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.680+0000 7f354f046640 1 --2- 172.21.15.57:0/2413394622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f354815d840 0x7f35481514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:31.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.680+0000 7f354f046640 1 --2- 172.21.15.57:0/2413394622 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3548153aa0 0x7f35481ef5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:31.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.680+0000 7f354f046640 1 --2- 172.21.15.57:0/2413394622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35480cd420 0x7f354814b9f0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:31.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.680+0000 7f354f046640 1 -- 172.21.15.57:0/2413394622 >> 172.21.15.57:0/2413394622 conn(0x7f3548071fb0 msgr2=0x7f354814f750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:31.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.680+0000 7f354f046640 1 -- 172.21.15.57:0/2413394622 shutdown_connections 2023-10-30T14:51:31.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.680+0000 7f354f046640 1 -- 172.21.15.57:0/2413394622 wait complete. 2023-10-30T14:51:31.706 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T14:51:31.706 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 4 2023-10-30T14:51:31.707 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2023-10-30T14:51:31.707 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 20 2023-10-30T14:51:31.707 INFO:tasks.mds_thrash:done joining 2023-10-30T14:51:31.708 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-10-30T14:51:31.720 INFO:tasks.kclient:Unmounting kernel clients... 2023-10-30T14:51:31.720 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:51:31.720 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-30T14:51:31.730 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-30T14:51:31.730 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-30T14:51:31.783 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:51:31.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.968+0000 7f44a4adc640 1 Processor -- start 2023-10-30T14:51:31.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.968+0000 7f44a4adc640 1 -- start start 2023-10-30T14:51:31.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f44a4adc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f44a00c3100 0x7f44a00c1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:31.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f44a4adc640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a00c39a0 con 0x7f44a00c34e0 2023-10-30T14:51:31.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f44a4adc640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a0159f20 con 0x7f44a0159950 2023-10-30T14:51:31.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f44a4adc640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44a006d8d0 con 0x7f44a00c3100 2023-10-30T14:51:31.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449effd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f44a00c34e0 0x7f44a006dcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:56454/0 (socket says 172.21.15.57:56454) 2023-10-30T14:51:31.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449effd640 1 -- 172.21.15.57:0/609302387 learned_addr learned my addr 172.21.15.57:0/609302387 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:51:31.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449f7fe640 1 --2- 172.21.15.57:0/609302387 >> v2:172.21.15.88:3300/0 conn(0x7f44a00c3100 0x7f44a00c1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:31.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449e7fc640 1 -- 172.21.15.57:0/609302387 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715249100 0 0) 0x7f44a00c39a0 con 0x7f44a00c34e0 2023-10-30T14:51:31.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449e7fc640 1 -- 172.21.15.57:0/609302387 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4488005740 con 0x7f44a00c34e0 2023-10-30T14:51:31.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449e7fc640 1 -- 172.21.15.57:0/609302387 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731267629 0 0) 0x7f44a0159f20 con 0x7f44a0159950 2023-10-30T14:51:31.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449e7fc640 1 -- 172.21.15.57:0/609302387 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44a00c39a0 con 0x7f44a0159950 2023-10-30T14:51:31.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449e7fc640 1 -- 172.21.15.57:0/609302387 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1274079163 0 0) 0x7f4488005740 con 0x7f44a00c34e0 2023-10-30T14:51:31.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449e7fc640 1 -- 172.21.15.57:0/609302387 >> v2:172.21.15.88:3300/0 conn(0x7f44a00c3100 msgr2=0x7f44a00c1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:31.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449e7fc640 1 --2- 172.21.15.57:0/609302387 >> v2:172.21.15.88:3300/0 conn(0x7f44a00c3100 0x7f44a00c1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:31.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449e7fc640 1 -- 172.21.15.57:0/609302387 >> v1:172.21.15.88:6789/0 conn(0x7f44a0159950 legacy=0x7f44a0150240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:31.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449e7fc640 1 -- 172.21.15.57:0/609302387 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a015a200 con 0x7f44a00c34e0 2023-10-30T14:51:31.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449e7fc640 1 -- 172.21.15.57:0/609302387 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f4494001680 con 0x7f44a00c34e0 2023-10-30T14:51:31.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449e7fc640 1 -- 172.21.15.57:0/609302387 >> v1:172.21.15.57:6789/0 conn(0x7f44a00c34e0 legacy=0x7f44a006dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:31.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449e7fc640 1 --2- 172.21.15.57:0/609302387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4488005d20 0x7f4488006120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:31.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449e7fc640 1 --2- 172.21.15.57:0/609302387 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44880069e0 0x7f4488006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:31.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449e7fc640 1 --2- 172.21.15.57:0/609302387 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44880073a0 0x7f448800dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:31.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449e7fc640 1 -- 172.21.15.57:0/609302387 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a006d8d0 con 0x7f44880069e0 2023-10-30T14:51:31.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449e7fc640 1 -- 172.21.15.57:0/609302387 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448800e190 con 0x7f4488005d20 2023-10-30T14:51:31.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449e7fc640 1 -- 172.21.15.57:0/609302387 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f448800e350 con 0x7f44880073a0 2023-10-30T14:51:31.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449effd640 1 --2- 172.21.15.57:0/609302387 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44880069e0 0x7f4488006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:31.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449f7fe640 1 --2- 172.21.15.57:0/609302387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4488005d20 0x7f4488006120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:31.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449ffff640 1 --2- 172.21.15.57:0/609302387 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44880073a0 0x7f448800dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:31.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449effd640 1 -- 172.21.15.57:0/609302387 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44880073a0 msgr2=0x7f448800dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:31.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449effd640 1 --2- 172.21.15.57:0/609302387 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44880073a0 0x7f448800dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:31.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449effd640 1 -- 172.21.15.57:0/609302387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4488005d20 msgr2=0x7f4488006120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:31.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449effd640 1 --2- 172.21.15.57:0/609302387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4488005d20 0x7f4488006120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:31.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449effd640 1 -- 172.21.15.57:0/609302387 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4488005740 con 0x7f44880069e0 2023-10-30T14:51:32.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.972+0000 7f449ffff640 1 -- 172.21.15.57:0/609302387 reap_dead start 2023-10-30T14:51:32.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.976+0000 7f449effd640 1 --2- 172.21.15.57:0/609302387 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44880069e0 0x7f4488006e60 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f4494001ec0 tx=0x7f449401f2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b654af740c9dee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:32.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.976+0000 7f449e7fc640 1 -- 172.21.15.57:0/609302387 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44940a8ce0 con 0x7f44880069e0 2023-10-30T14:51:32.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.976+0000 7f449e7fc640 1 -- 172.21.15.57:0/609302387 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44940a8e60 con 0x7f44880069e0 2023-10-30T14:51:32.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.976+0000 7f449e7fc640 1 -- 172.21.15.57:0/609302387 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4494005b20 con 0x7f44880069e0 2023-10-30T14:51:32.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.976+0000 7f44a4adc640 1 -- 172.21.15.57:0/609302387 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44880069e0 msgr2=0x7f4488006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:32.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.976+0000 7f44a4adc640 1 --2- 172.21.15.57:0/609302387 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44880069e0 0x7f4488006e60 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f4494001ec0 tx=0x7f449401f2b0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.976+0000 7f44a4adc640 1 -- 172.21.15.57:0/609302387 shutdown_connections 2023-10-30T14:51:32.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.976+0000 7f44a4adc640 1 --2- 172.21.15.57:0/609302387 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44880069e0 0x7f4488006e60 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.976+0000 7f44a4adc640 1 -- 172.21.15.57:0/609302387 >> 172.21.15.57:0/609302387 conn(0x7f44a00caae0 msgr2=0x7f44a00caec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:32.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.976+0000 7f44a4adc640 1 -- 172.21.15.57:0/609302387 shutdown_connections 2023-10-30T14:51:32.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.976+0000 7f44a4adc640 1 -- 172.21.15.57:0/609302387 wait complete. 2023-10-30T14:51:32.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f44a4adc640 1 Processor -- start 2023-10-30T14:51:32.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f44a4adc640 1 -- start start 2023-10-30T14:51:32.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f44a4adc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44880069e0 0x7f44a01533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:32.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f44a4adc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44a00bcf60 0x7f44a00c6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:32.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f44a4adc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44a00c6480 0x7f44a01f1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:32.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f44a4adc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a00c0650 con 0x7f44880069e0 2023-10-30T14:51:32.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f44a4adc640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a0157540 con 0x7f44a00bcf60 2023-10-30T14:51:32.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f44a4adc640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a00bd8a0 con 0x7f44a00c6480 2023-10-30T14:51:32.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f449f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44880069e0 0x7f44a01533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:32.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f449f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44880069e0 0x7f44a01533d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52598/0 (socket says 172.21.15.57:52598) 2023-10-30T14:51:32.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f449f7fe640 1 -- 172.21.15.57:0/2001727891 learned_addr learned my addr 172.21.15.57:0/2001727891 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:32.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f449effd640 1 --2- 172.21.15.57:0/2001727891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44a00bcf60 0x7f44a00c6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:32.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f449ffff640 1 --2- 172.21.15.57:0/2001727891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44a00c6480 0x7f44a01f1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:32.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f449f7fe640 1 -- 172.21.15.57:0/2001727891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44a00c6480 msgr2=0x7f44a01f1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:32.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f449f7fe640 1 --2- 172.21.15.57:0/2001727891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44a00c6480 0x7f44a01f1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f449f7fe640 1 -- 172.21.15.57:0/2001727891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44a00bcf60 msgr2=0x7f44a00c6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:32.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f449f7fe640 1 --2- 172.21.15.57:0/2001727891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44a00bcf60 0x7f44a00c6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f449f7fe640 1 -- 172.21.15.57:0/2001727891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a01f23f0 con 0x7f44880069e0 2023-10-30T14:51:32.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f449effd640 1 --2- 172.21.15.57:0/2001727891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44a00bcf60 0x7f44a00c6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:32.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f449f7fe640 1 --2- 172.21.15.57:0/2001727891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44880069e0 0x7f44a01533d0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f44900119d0 tx=0x7f4490011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:32.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f4487fff640 1 -- 172.21.15.57:0/2001727891 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449000fb50 con 0x7f44880069e0 2023-10-30T14:51:32.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f4487fff640 1 -- 172.21.15.57:0/2001727891 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449000fcb0 con 0x7f44880069e0 2023-10-30T14:51:32.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f4487fff640 1 -- 172.21.15.57:0/2001727891 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4490014440 con 0x7f44880069e0 2023-10-30T14:51:32.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f44a4adc640 1 -- 172.21.15.57:0/2001727891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a0157260 con 0x7f44880069e0 2023-10-30T14:51:32.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f44a4adc640 1 -- 172.21.15.57:0/2001727891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a00c6900 con 0x7f44880069e0 2023-10-30T14:51:32.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.980+0000 7f44a4adc640 1 -- 172.21.15.57:0/2001727891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a004e170 con 0x7f44880069e0 2023-10-30T14:51:32.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.984+0000 7f4487fff640 1 -- 172.21.15.57:0/2001727891 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f44900145a0 con 0x7f44880069e0 2023-10-30T14:51:32.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.984+0000 7f4487fff640 1 --2- 172.21.15.57:0/2001727891 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f448c0664a0 0x7f448c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:32.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.984+0000 7f4487fff640 1 -- 172.21.15.57:0/2001727891 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f4490084cf0 con 0x7f44880069e0 2023-10-30T14:51:32.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.984+0000 7f449effd640 1 --2- 172.21.15.57:0/2001727891 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f448c0664a0 0x7f448c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:32.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.984+0000 7f449effd640 1 --2- 172.21.15.57:0/2001727891 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f448c0664a0 0x7f448c068960 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f4494001ec0 tx=0x7f4494001a00 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:32.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:31.988+0000 7f4487fff640 1 -- 172.21.15.57:0/2001727891 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4490051ba0 con 0x7f44880069e0 2023-10-30T14:51:32.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.148+0000 7f44a4adc640 1 -- 172.21.15.57:0/2001727891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44a006e200 con 0x7f44880069e0 2023-10-30T14:51:32.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.152+0000 7f4487fff640 1 -- 172.21.15.57:0/2001727891 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+19036 (secure 0 0 0) 0x7f4490022090 con 0x7f44880069e0 2023-10-30T14:51:32.155 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:51:32.156 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":67,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:50:50.290577+0000","last_up_change":"2023-10-30T14:13:51.171287+0000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T14:13:52.574639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T14:14:02.600122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T14:14:03.069429+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":53,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6801","nonce":3287367848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6803","nonce":3287367848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6807","nonce":3287367848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6805","nonce":3287367848}]},"public_addr":"172.21.15.57:6801/3287367848","cluster_addr":"172.21.15.57:6803/3287367848","heartbeat_back_addr":"172.21.15.57:6807/3287367848","heartbeat_front_addr":"172.21.15.57:6805/3287367848","state":["exists","up"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6809","nonce":3340797836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6811","nonce":3340797836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6815","nonce":3340797836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6813","nonce":3340797836}]},"public_addr":"172.21.15.57:6809/3340797836","cluster_addr":"172.21.15.57:6811/3340797836","heartbeat_back_addr":"172.21.15.57:6815/3340797836","heartbeat_front_addr":"172.21.15.57:6813/3340797836","state":["exists","up"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6825","nonce":1700292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6827","nonce":1700292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6831","nonce":1700292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6829","nonce":1700292}]},"public_addr":"172.21.15.57:6825/1700292","cluster_addr":"172.21.15.57:6827/1700292","heartbeat_back_addr":"172.21.15.57:6831/1700292","heartbeat_front_addr":"172.21.15.57:6829/1700292","state":["exists","up"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6817","nonce":2893420631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6819","nonce":2893420631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6823","nonce":2893420631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6821","nonce":2893420631}]},"public_addr":"172.21.15.57:6817/2893420631","cluster_addr":"172.21.15.57:6819/2893420631","heartbeat_back_addr":"172.21.15.57:6823/2893420631","heartbeat_front_addr":"172.21.15.57:6821/2893420631","state":["exists","up"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6807","nonce":2784816485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6811","nonce":2784816485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6815","nonce":2784816485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6813","nonce":2784816485}]},"public_addr":"172.21.15.88:6807/2784816485","cluster_addr":"172.21.15.88:6811/2784816485","heartbeat_back_addr":"172.21.15.88:6815/2784816485","heartbeat_front_addr":"172.21.15.88:6813/2784816485","state":["exists","up"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6801","nonce":2682449812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6803","nonce":2682449812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6810","nonce":2682449812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6806","nonce":2682449812}]},"public_addr":"172.21.15.88:6801/2682449812","cluster_addr":"172.21.15.88:6803/2682449812","heartbeat_back_addr":"172.21.15.88:6810/2682449812","heartbeat_front_addr":"172.21.15.88:6806/2682449812","state":["exists","up"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6822","nonce":3340693897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6827","nonce":3340693897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6831","nonce":3340693897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6829","nonce":3340693897}]},"public_addr":"172.21.15.88:6822/3340693897","cluster_addr":"172.21.15.88:6827/3340693897","heartbeat_back_addr":"172.21.15.88:6831/3340693897","heartbeat_front_addr":"172.21.15.88:6829/3340693897","state":["exists","up"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6817","nonce":2862111199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6820","nonce":2862111199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6825","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6826","nonce":2862111199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6821","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6823","nonce":2862111199}]},"public_addr":"172.21.15.88:6817/2862111199","cluster_addr":"172.21.15.88:6820/2862111199","heartbeat_back_addr":"172.21.15.88:6826/2862111199","heartbeat_front_addr":"172.21.15.88:6823/2862111199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.634920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.773993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.725825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.761452+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.694274+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.650718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6835/2613636524":"2023-10-31T14:50:50.290535+0000","172.21.15.57:6837/816502245":"2023-10-31T14:48:30.252448+0000","172.21.15.88:6838/1744808283":"2023-10-31T14:44:20.161100+0000","172.21.15.57:6834/2321408149":"2023-10-31T14:42:35.131496+0000","172.21.15.57:6833/816502245":"2023-10-31T14:48:30.252448+0000","172.21.15.57:6834/2912050965":"2023-10-31T14:23:54.764165+0000","172.21.15.57:6837/1988100930":"2023-10-31T14:20:34.693623+0000","172.21.15.88:6839/2945854794":"2023-10-31T14:18:54.654873+0000","172.21.15.57:6835/2427081813":"2023-10-31T14:26:24.805473+0000","172.21.15.57:6833/1988100930":"2023-10-31T14:20:34.693623+0000","172.21.15.88:6839/358521496":"2023-10-31T14:30:09.871507+0000","172.21.15.57:6836/2693548867":"2023-10-31T14:17:44.635085+0000","172.21.15.57:6832/2912050965":"2023-10-31T14:23:54.764165+0000","172.21.15.57:6832/3701655528":"2023-10-31T14:16:49.618077+0000","172.21.15.88:6838/358521496":"2023-10-31T14:30:09.871507+0000","172.21.15.57:6837/901968115":"2023-10-31T14:14:59.588353+0000","172.21.15.88:6839/1744808283":"2023-10-31T14:44:20.161100+0000","172.21.15.88:6838/2945854794":"2023-10-31T14:18:54.654873+0000","172.21.15.57:6834/3701655528":"2023-10-31T14:16:49.618077+0000","172.21.15.57:6835/2733323524":"2023-10-31T14:32:54.940701+0000","172.21.15.57:6836/2733323524":"2023-10-31T14:32:54.940701+0000","172.21.15.57:6835/2693548867":"2023-10-31T14:17:44.635085+0000","172.21.15.57:6832/2321408149":"2023-10-31T14:42:35.131496+0000","172.21.15.57:6833/862088802":"2023-10-31T14:27:49.829937+0000","172.21.15.88:6838/3604033338":"2023-10-31T14:35:29.980317+0000","172.21.15.57:6836/2613636524":"2023-10-31T14:50:50.290535+0000","172.21.15.88:6839/2390289388":"2023-10-31T14:39:45.086742+0000","172.21.15.57:6837/862088802":"2023-10-31T14:27:49.829937+0000","172.21.15.88:6838/2390289388":"2023-10-31T14:39:45.086742+0000","172.21.15.57:6832/2940934016":"2023-10-31T14:29:09.852381+0000","172.21.15.57:6834/2940934016":"2023-10-31T14:29:09.852381+0000","172.21.15.88:6839/3604033338":"2023-10-31T14:35:29.980317+0000","172.21.15.57:6833/901968115":"2023-10-31T14:14:59.588353+0000","172.21.15.57:6836/2427081813":"2023-10-31T14:26:24.805473+0000","172.21.15.57:6833/1843528309":"2023-10-31T14:38:15.025313+0000","172.21.15.57:6837/1843528309":"2023-10-31T14:38:15.025313+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:51:32.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.152+0000 7f44a4adc640 1 -- 172.21.15.57:0/2001727891 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f448c0664a0 msgr2=0x7f448c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:32.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.152+0000 7f44a4adc640 1 --2- 172.21.15.57:0/2001727891 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f448c0664a0 0x7f448c068960 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f4494001ec0 tx=0x7f4494001a00 comp rx=0 tx=0).stop 2023-10-30T14:51:32.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.152+0000 7f44a4adc640 1 -- 172.21.15.57:0/2001727891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44880069e0 msgr2=0x7f44a01533d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:32.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.152+0000 7f44a4adc640 1 --2- 172.21.15.57:0/2001727891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44880069e0 0x7f44a01533d0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f44900119d0 tx=0x7f4490011e90 comp rx=0 tx=0).stop 2023-10-30T14:51:32.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.152+0000 7f44a4adc640 1 -- 172.21.15.57:0/2001727891 shutdown_connections 2023-10-30T14:51:32.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.152+0000 7f44a4adc640 1 --2- 172.21.15.57:0/2001727891 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f448c0664a0 0x7f448c068960 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.152+0000 7f44a4adc640 1 --2- 172.21.15.57:0/2001727891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44a00c6480 0x7f44a01f1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.152+0000 7f44a4adc640 1 --2- 172.21.15.57:0/2001727891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44a00bcf60 0x7f44a00c6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.152+0000 7f44a4adc640 1 --2- 172.21.15.57:0/2001727891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44880069e0 0x7f44a01533d0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.152+0000 7f44a4adc640 1 -- 172.21.15.57:0/2001727891 >> 172.21.15.57:0/2001727891 conn(0x7f44a00caae0 msgr2=0x7f44a00c56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:32.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.152+0000 7f44a4adc640 1 -- 172.21.15.57:0/2001727891 shutdown_connections 2023-10-30T14:51:32.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.152+0000 7f44a4adc640 1 -- 172.21.15.57:0/2001727891 wait complete. 2023-10-30T14:51:32.177 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T14:13:52.574639+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T14:14:02.600122+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T14:14:03.069429+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 53, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T14:51:32.178 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T14:51:32.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.320+0000 7f0dc990f640 1 Processor -- start 2023-10-30T14:51:32.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.320+0000 7f0dc990f640 1 -- start start 2023-10-30T14:51:32.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.320+0000 7f0dc990f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0dc41500e0 0x7f0dc40b0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:32.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.320+0000 7f0dc990f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0dc4072520 0x7f0dc40b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:32.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.320+0000 7f0dc990f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc40c56a0 0x7f0dc4156350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:32.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.320+0000 7f0dc990f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dc40c5ca0 con 0x7f0dc41500e0 2023-10-30T14:51:32.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.320+0000 7f0dc990f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dc41504c0 con 0x7f0dc4072520 2023-10-30T14:51:32.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.320+0000 7f0dc990f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc40c5e90 con 0x7f0dc40c56a0 2023-10-30T14:51:32.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.320+0000 7f0dc2ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0dc41500e0 0x7f0dc40b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:32.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.320+0000 7f0dc2ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0dc41500e0 0x7f0dc40b0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52606/0 (socket says 172.21.15.57:52606) 2023-10-30T14:51:32.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.320+0000 7f0dc2ffd640 1 -- 172.21.15.57:0/2094634323 learned_addr learned my addr 172.21.15.57:0/2094634323 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:32.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.320+0000 7f0dc37fe640 1 --2- 172.21.15.57:0/2094634323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc40c56a0 0x7f0dc4156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:32.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.320+0000 7f0dbbfff640 1 --2- 172.21.15.57:0/2094634323 >> v2:172.21.15.88:3300/0 conn(0x7f0dc4072520 0x7f0dc40b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:32.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.320+0000 7f0dc2ffd640 1 -- 172.21.15.57:0/2094634323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc40c56a0 msgr2=0x7f0dc4156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:32.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.320+0000 7f0dc2ffd640 1 --2- 172.21.15.57:0/2094634323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc40c56a0 0x7f0dc4156350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.320+0000 7f0dc2ffd640 1 -- 172.21.15.57:0/2094634323 >> v2:172.21.15.88:3300/0 conn(0x7f0dc4072520 msgr2=0x7f0dc40b0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:32.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.320+0000 7f0dc2ffd640 1 --2- 172.21.15.57:0/2094634323 >> v2:172.21.15.88:3300/0 conn(0x7f0dc4072520 0x7f0dc40b0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.320+0000 7f0dc2ffd640 1 -- 172.21.15.57:0/2094634323 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc40c6170 con 0x7f0dc41500e0 2023-10-30T14:51:32.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.324+0000 7f0dc2ffd640 1 --2- 172.21.15.57:0/2094634323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dc41500e0 0x7f0dc40b0710 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f0dc40cbdd0 tx=0x7f0db401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75fe6d6ecd8d305d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:32.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.324+0000 7f0dc27fc640 1 -- 172.21.15.57:0/2094634323 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db40a89b0 con 0x7f0dc41500e0 2023-10-30T14:51:32.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.324+0000 7f0dc27fc640 1 -- 172.21.15.57:0/2094634323 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db40a8b10 con 0x7f0dc41500e0 2023-10-30T14:51:32.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.324+0000 7f0dc27fc640 1 -- 172.21.15.57:0/2094634323 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db40a8ea0 con 0x7f0dc41500e0 2023-10-30T14:51:32.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.324+0000 7f0dc990f640 1 -- 172.21.15.57:0/2094634323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dc41500e0 msgr2=0x7f0dc40b0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:32.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.324+0000 7f0dc990f640 1 --2- 172.21.15.57:0/2094634323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dc41500e0 0x7f0dc40b0710 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f0dc40cbdd0 tx=0x7f0db401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.324+0000 7f0dc990f640 1 -- 172.21.15.57:0/2094634323 shutdown_connections 2023-10-30T14:51:32.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.324+0000 7f0dc990f640 1 --2- 172.21.15.57:0/2094634323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc40c56a0 0x7f0dc4156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.324+0000 7f0dc990f640 1 --2- 172.21.15.57:0/2094634323 >> v2:172.21.15.88:3300/0 conn(0x7f0dc4072520 0x7f0dc40b0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.324+0000 7f0dc990f640 1 --2- 172.21.15.57:0/2094634323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dc41500e0 0x7f0dc40b0710 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.324+0000 7f0dc990f640 1 -- 172.21.15.57:0/2094634323 >> 172.21.15.57:0/2094634323 conn(0x7f0dc406da40 msgr2=0x7f0dc406de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:32.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.324+0000 7f0dc990f640 1 -- 172.21.15.57:0/2094634323 shutdown_connections 2023-10-30T14:51:32.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.324+0000 7f0dc990f640 1 -- 172.21.15.57:0/2094634323 wait complete. 2023-10-30T14:51:32.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dc990f640 1 Processor -- start 2023-10-30T14:51:32.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dc990f640 1 -- start start 2023-10-30T14:51:32.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dc990f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc4072520 0x7f0dc40c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:32.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dc990f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dc40c56a0 0x7f0dc40c3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:32.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dc990f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dc41500e0 0x7f0dc4144e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:32.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dc990f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc40c7430 con 0x7f0dc40c56a0 2023-10-30T14:51:32.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dc990f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc40c26a0 con 0x7f0dc41500e0 2023-10-30T14:51:32.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dc990f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc40c6b20 con 0x7f0dc4072520 2023-10-30T14:51:32.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dbbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dc40c56a0 0x7f0dc40c3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:32.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dbbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dc40c56a0 0x7f0dc40c3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52608/0 (socket says 172.21.15.57:52608) 2023-10-30T14:51:32.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dbbfff640 1 -- 172.21.15.57:0/476640530 learned_addr learned my addr 172.21.15.57:0/476640530 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:32.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dc37fe640 1 --2- 172.21.15.57:0/476640530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dc41500e0 0x7f0dc4144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:32.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dc2ffd640 1 --2- 172.21.15.57:0/476640530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc4072520 0x7f0dc40c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:32.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dbbfff640 1 -- 172.21.15.57:0/476640530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc4072520 msgr2=0x7f0dc40c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:32.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dbbfff640 1 --2- 172.21.15.57:0/476640530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc4072520 0x7f0dc40c7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dbbfff640 1 -- 172.21.15.57:0/476640530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dc41500e0 msgr2=0x7f0dc4144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:32.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dbbfff640 1 --2- 172.21.15.57:0/476640530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dc41500e0 0x7f0dc4144e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dbbfff640 1 -- 172.21.15.57:0/476640530 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db40094d0 con 0x7f0dc40c56a0 2023-10-30T14:51:32.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dbbfff640 1 --2- 172.21.15.57:0/476640530 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dc40c56a0 0x7f0dc40c3240 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f0dac00d830 tx=0x7f0dac00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:32.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dc0ff9640 1 -- 172.21.15.57:0/476640530 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac00be10 con 0x7f0dc40c56a0 2023-10-30T14:51:32.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dc0ff9640 1 -- 172.21.15.57:0/476640530 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dac0098b0 con 0x7f0dc40c56a0 2023-10-30T14:51:32.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dc0ff9640 1 -- 172.21.15.57:0/476640530 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac009a40 con 0x7f0dc40c56a0 2023-10-30T14:51:32.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dc990f640 1 -- 172.21.15.57:0/476640530 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc41ef110 con 0x7f0dc40c56a0 2023-10-30T14:51:32.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.328+0000 7f0dc990f640 1 -- 172.21.15.57:0/476640530 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc40c6d70 con 0x7f0dc40c56a0 2023-10-30T14:51:32.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.332+0000 7f0dc990f640 1 -- 172.21.15.57:0/476640530 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc40c7570 con 0x7f0dc40c56a0 2023-10-30T14:51:32.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.332+0000 7f0dc0ff9640 1 -- 172.21.15.57:0/476640530 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0dac009ba0 con 0x7f0dc40c56a0 2023-10-30T14:51:32.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.332+0000 7f0dc0ff9640 1 --2- 172.21.15.57:0/476640530 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0da40664d0 0x7f0da4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:32.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.332+0000 7f0dc0ff9640 1 -- 172.21.15.57:0/476640530 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f0dac084730 con 0x7f0dc40c56a0 2023-10-30T14:51:32.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.332+0000 7f0dc2ffd640 1 --2- 172.21.15.57:0/476640530 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0da40664d0 0x7f0da4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:32.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.332+0000 7f0dc2ffd640 1 --2- 172.21.15.57:0/476640530 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0da40664d0 0x7f0da4068990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f0db40b2000 tx=0x7f0db4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:32.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.336+0000 7f0dc0ff9640 1 -- 172.21.15.57:0/476640530 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0dc40c7570 con 0x7f0dc40c56a0 2023-10-30T14:51:32.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.500+0000 7f0dc990f640 1 -- 172.21.15.57:0/476640530 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0dc40c7570 con 0x7f0dc40c56a0 2023-10-30T14:51:32.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.500+0000 7f0dc0ff9640 1 -- 172.21.15.57:0/476640530 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0dc40c7570 con 0x7f0dc40c56a0 2023-10-30T14:51:32.506 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-30T14:51:32.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.504+0000 7f0dc990f640 1 -- 172.21.15.57:0/476640530 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0da40664d0 msgr2=0x7f0da4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:32.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.504+0000 7f0dc990f640 1 --2- 172.21.15.57:0/476640530 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0da40664d0 0x7f0da4068990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f0db40b2000 tx=0x7f0db4009040 comp rx=0 tx=0).stop 2023-10-30T14:51:32.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.504+0000 7f0dc990f640 1 -- 172.21.15.57:0/476640530 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dc40c56a0 msgr2=0x7f0dc40c3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:32.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.504+0000 7f0dc990f640 1 --2- 172.21.15.57:0/476640530 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dc40c56a0 0x7f0dc40c3240 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f0dac00d830 tx=0x7f0dac00dcf0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.504+0000 7f0dc990f640 1 -- 172.21.15.57:0/476640530 shutdown_connections 2023-10-30T14:51:32.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.504+0000 7f0dc990f640 1 --2- 172.21.15.57:0/476640530 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0da40664d0 0x7f0da4068990 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.504+0000 7f0dc990f640 1 --2- 172.21.15.57:0/476640530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dc41500e0 0x7f0dc4144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.504+0000 7f0dc990f640 1 --2- 172.21.15.57:0/476640530 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dc40c56a0 0x7f0dc40c3240 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.504+0000 7f0dc990f640 1 --2- 172.21.15.57:0/476640530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc4072520 0x7f0dc40c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.504+0000 7f0dc990f640 1 -- 172.21.15.57:0/476640530 >> 172.21.15.57:0/476640530 conn(0x7f0dc406da40 msgr2=0x7f0dc41553b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:32.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.504+0000 7f0dc990f640 1 -- 172.21.15.57:0/476640530 shutdown_connections 2023-10-30T14:51:32.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.504+0000 7f0dc990f640 1 -- 172.21.15.57:0/476640530 wait complete. 2023-10-30T14:51:32.526 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T14:51:32.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.668+0000 7fa41b59e640 1 Processor -- start 2023-10-30T14:51:32.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.668+0000 7fa41b59e640 1 -- start start 2023-10-30T14:51:32.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.672+0000 7fa41b59e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa41c153a10 0x7fa41c0c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:32.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.672+0000 7fa41b59e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41c15d0a0 con 0x7fa41c071cb0 2023-10-30T14:51:32.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.672+0000 7fa41b59e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41c15d380 con 0x7fa41c0718d0 2023-10-30T14:51:32.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.672+0000 7fa41b59e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa41c0cca10 con 0x7fa41c153a10 2023-10-30T14:51:32.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.672+0000 7fa419d9b640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa41c071cb0 0x7fa41c153300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:56464/0 (socket says 172.21.15.57:56464) 2023-10-30T14:51:32.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.672+0000 7fa419d9b640 1 -- 172.21.15.57:0/2064859595 learned_addr learned my addr 172.21.15.57:0/2064859595 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:51:32.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.672+0000 7fa41ad9d640 1 --2- 172.21.15.57:0/2064859595 >> v2:172.21.15.88:3300/0 conn(0x7fa41c153a10 0x7fa41c0c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:32.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.672+0000 7fa41959a640 1 -- 172.21.15.57:0/2064859595 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484493046 0 0) 0x7fa41c15d0a0 con 0x7fa41c071cb0 2023-10-30T14:51:32.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.672+0000 7fa41959a640 1 -- 172.21.15.57:0/2064859595 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa404003530 con 0x7fa41c071cb0 2023-10-30T14:51:32.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.672+0000 7fa41959a640 1 -- 172.21.15.57:0/2064859595 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2316260763 0 0) 0x7fa404003530 con 0x7fa41c071cb0 2023-10-30T14:51:32.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.672+0000 7fa41ad9d640 1 -- 172.21.15.57:0/2064859595 >> v1:172.21.15.88:6789/0 conn(0x7fa41c0718d0 legacy=0x7fa41c15a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:32.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.672+0000 7fa41ad9d640 1 -- 172.21.15.57:0/2064859595 >> v1:172.21.15.57:6789/0 conn(0x7fa41c071cb0 legacy=0x7fa41c153300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:32.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.672+0000 7fa41ad9d640 1 -- 172.21.15.57:0/2064859595 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c15d660 con 0x7fa41c153a10 2023-10-30T14:51:32.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.676+0000 7fa41ad9d640 1 --2- 172.21.15.57:0/2064859595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41c153a10 0x7fa41c0c9a10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa414009d90 tx=0x7fa41401f060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6de1d90655835a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:32.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.676+0000 7fa41959a640 1 -- 172.21.15.57:0/2064859595 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa410001680 con 0x7fa41c153a10 2023-10-30T14:51:32.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.676+0000 7fa41959a640 1 -- 172.21.15.57:0/2064859595 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4140a8bf0 con 0x7fa41c153a10 2023-10-30T14:51:32.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.676+0000 7fa41959a640 1 -- 172.21.15.57:0/2064859595 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4140b0440 con 0x7fa41c153a10 2023-10-30T14:51:32.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.676+0000 7fa41b59e640 1 -- 172.21.15.57:0/2064859595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41c153a10 msgr2=0x7fa41c0c9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:32.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.676+0000 7fa41b59e640 1 --2- 172.21.15.57:0/2064859595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41c153a10 0x7fa41c0c9a10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa414009d90 tx=0x7fa41401f060 comp rx=0 tx=0).stop 2023-10-30T14:51:32.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.676+0000 7fa41b59e640 1 -- 172.21.15.57:0/2064859595 shutdown_connections 2023-10-30T14:51:32.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.676+0000 7fa41b59e640 1 --2- 172.21.15.57:0/2064859595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41c153a10 0x7fa41c0c9a10 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.676+0000 7fa41b59e640 1 -- 172.21.15.57:0/2064859595 >> 172.21.15.57:0/2064859595 conn(0x7fa41c06ab00 msgr2=0x7fa41c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:32.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.676+0000 7fa41b59e640 1 -- 172.21.15.57:0/2064859595 shutdown_connections 2023-10-30T14:51:32.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.676+0000 7fa41b59e640 1 -- 172.21.15.57:0/2064859595 wait complete. 2023-10-30T14:51:32.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.676+0000 7fa41b59e640 1 Processor -- start 2023-10-30T14:51:32.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.676+0000 7fa41b59e640 1 -- start start 2023-10-30T14:51:32.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa41b59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41c0718d0 0x7fa41c0b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:32.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa41b59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa41c071cb0 0x7fa41c14f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:32.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa41b59e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa41c153a10 0x7fa41c147c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:32.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa41b59e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c14ede0 con 0x7fa41c071cb0 2023-10-30T14:51:32.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa41b59e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c0b45a0 con 0x7fa41c0718d0 2023-10-30T14:51:32.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa41b59e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c0b33a0 con 0x7fa41c153a10 2023-10-30T14:51:32.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa419d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa41c071cb0 0x7fa41c14f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:32.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa419d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa41c071cb0 0x7fa41c14f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52620/0 (socket says 172.21.15.57:52620) 2023-10-30T14:51:32.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa419d9b640 1 -- 172.21.15.57:0/72689938 learned_addr learned my addr 172.21.15.57:0/72689938 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:32.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa41a59c640 1 --2- 172.21.15.57:0/72689938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41c0718d0 0x7fa41c0b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:32.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa41ad9d640 1 --2- 172.21.15.57:0/72689938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa41c153a10 0x7fa41c147c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:32.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa419d9b640 1 -- 172.21.15.57:0/72689938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa41c153a10 msgr2=0x7fa41c147c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:32.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa419d9b640 1 --2- 172.21.15.57:0/72689938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa41c153a10 0x7fa41c147c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa419d9b640 1 -- 172.21.15.57:0/72689938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41c0718d0 msgr2=0x7fa41c0b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:32.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa419d9b640 1 --2- 172.21.15.57:0/72689938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41c0718d0 0x7fa41c0b3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa419d9b640 1 -- 172.21.15.57:0/72689938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa414009a70 con 0x7fa41c071cb0 2023-10-30T14:51:32.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa419d9b640 1 --2- 172.21.15.57:0/72689938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa41c071cb0 0x7fa41c14f200 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fa4100011f0 tx=0x7fa41000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:32.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa40affd640 1 -- 172.21.15.57:0/72689938 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa410005ae0 con 0x7fa41c071cb0 2023-10-30T14:51:32.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa40affd640 1 -- 172.21.15.57:0/72689938 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa410005c40 con 0x7fa41c071cb0 2023-10-30T14:51:32.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa40affd640 1 -- 172.21.15.57:0/72689938 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa410007070 con 0x7fa41c071cb0 2023-10-30T14:51:32.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa41b59e640 1 -- 172.21.15.57:0/72689938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa41c14f740 con 0x7fa41c071cb0 2023-10-30T14:51:32.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa41b59e640 1 -- 172.21.15.57:0/72689938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa41c0b2de0 con 0x7fa41c071cb0 2023-10-30T14:51:32.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.680+0000 7fa41b59e640 1 -- 172.21.15.57:0/72689938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa41c04e170 con 0x7fa41c071cb0 2023-10-30T14:51:32.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.684+0000 7fa40affd640 1 -- 172.21.15.57:0/72689938 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa4100071d0 con 0x7fa41c071cb0 2023-10-30T14:51:32.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.684+0000 7fa40affd640 1 --2- 172.21.15.57:0/72689938 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa3fc0664a0 0x7fa3fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:32.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.684+0000 7fa40affd640 1 -- 172.21.15.57:0/72689938 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fa410083b40 con 0x7fa41c071cb0 2023-10-30T14:51:32.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.684+0000 7fa41a59c640 1 --2- 172.21.15.57:0/72689938 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa3fc0664a0 0x7fa3fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:32.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.684+0000 7fa41a59c640 1 --2- 172.21.15.57:0/72689938 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa3fc0664a0 0x7fa3fc068960 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa40c007fd0 tx=0x7fa40c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:32.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.688+0000 7fa40affd640 1 -- 172.21.15.57:0/72689938 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa4100509f0 con 0x7fa41c071cb0 2023-10-30T14:51:32.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.848+0000 7fa41b59e640 1 -- 172.21.15.57:0/72689938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa41c14d210 con 0x7fa41c071cb0 2023-10-30T14:51:32.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.848+0000 7fa40affd640 1 -- 172.21.15.57:0/72689938 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v67) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa410017020 con 0x7fa41c071cb0 2023-10-30T14:51:32.852 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-30T14:51:32.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.848+0000 7fa41b59e640 1 -- 172.21.15.57:0/72689938 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa3fc0664a0 msgr2=0x7fa3fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:32.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.848+0000 7fa41b59e640 1 --2- 172.21.15.57:0/72689938 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa3fc0664a0 0x7fa3fc068960 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa40c007fd0 tx=0x7fa40c009040 comp rx=0 tx=0).stop 2023-10-30T14:51:32.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.848+0000 7fa41b59e640 1 -- 172.21.15.57:0/72689938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa41c071cb0 msgr2=0x7fa41c14f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:32.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.848+0000 7fa41b59e640 1 --2- 172.21.15.57:0/72689938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa41c071cb0 0x7fa41c14f200 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fa4100011f0 tx=0x7fa41000cbe0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.848+0000 7fa41b59e640 1 -- 172.21.15.57:0/72689938 shutdown_connections 2023-10-30T14:51:32.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.848+0000 7fa41b59e640 1 --2- 172.21.15.57:0/72689938 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa3fc0664a0 0x7fa3fc068960 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.848+0000 7fa41b59e640 1 --2- 172.21.15.57:0/72689938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa41c153a10 0x7fa41c147c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.848+0000 7fa41b59e640 1 --2- 172.21.15.57:0/72689938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa41c071cb0 0x7fa41c14f200 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.848+0000 7fa41b59e640 1 --2- 172.21.15.57:0/72689938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41c0718d0 0x7fa41c0b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:32.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.848+0000 7fa41b59e640 1 -- 172.21.15.57:0/72689938 >> 172.21.15.57:0/72689938 conn(0x7fa41c06ab00 msgr2=0x7fa41c0c4880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:32.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.848+0000 7fa41b59e640 1 -- 172.21.15.57:0/72689938 shutdown_connections 2023-10-30T14:51:32.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:32.848+0000 7fa41b59e640 1 -- 172.21.15.57:0/72689938 wait complete. 2023-10-30T14:51:32.872 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T14:51:33.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.016+0000 7f212c19d640 1 Processor -- start 2023-10-30T14:51:33.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.016+0000 7f212c19d640 1 -- start start 2023-10-30T14:51:33.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.016+0000 7f212c19d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21240b2000 0x7f212406cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:33.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.016+0000 7f212c19d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f212406efb0 0x7f212406e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:33.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.016+0000 7f212c19d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f212406b040 con 0x7f212415dcc0 2023-10-30T14:51:33.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.016+0000 7f212c19d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f212406b600 con 0x7f212406efb0 2023-10-30T14:51:33.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.016+0000 7f212c19d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f212406c520 con 0x7f21240b2000 2023-10-30T14:51:33.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.016+0000 7f2129f39640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21240b2000 0x7f212406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:33.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.016+0000 7f212a73a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f212406efb0 0x7f212406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:33.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.016+0000 7f2129f39640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21240b2000 0x7f212406cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:36354/0 (socket says 172.21.15.57:36354) 2023-10-30T14:51:33.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.016+0000 7f2129f39640 1 -- 172.21.15.57:0/1717824495 learned_addr learned my addr 172.21.15.57:0/1717824495 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:33.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.016+0000 7f212a73a640 1 -- 172.21.15.57:0/1717824495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21240b2000 msgr2=0x7f212406cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:33.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.016+0000 7f212a73a640 1 --2- 172.21.15.57:0/1717824495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21240b2000 0x7f212406cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:33.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.016+0000 7f212a73a640 1 -- 172.21.15.57:0/1717824495 >> v1:172.21.15.88:6789/0 conn(0x7f212415dcc0 legacy=0x7f2124159ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:33.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.016+0000 7f212a73a640 1 -- 172.21.15.57:0/1717824495 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212406b320 con 0x7f212406efb0 2023-10-30T14:51:33.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.020+0000 7f212a73a640 1 --2- 172.21.15.57:0/1717824495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f212406efb0 0x7f212406e230 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f211c009d90 tx=0x7f211c01f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bac600a94fb7d59 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:33.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.020+0000 7f2128f37640 1 -- 172.21.15.57:0/1717824495 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211c0af600 con 0x7f212406efb0 2023-10-30T14:51:33.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.020+0000 7f2128f37640 1 -- 172.21.15.57:0/1717824495 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f211c0ab050 con 0x7f212406efb0 2023-10-30T14:51:33.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.020+0000 7f2128f37640 1 -- 172.21.15.57:0/1717824495 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211c0af980 con 0x7f212406efb0 2023-10-30T14:51:33.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.020+0000 7f212c19d640 1 -- 172.21.15.57:0/1717824495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f212406efb0 msgr2=0x7f212406e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:33.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.020+0000 7f212c19d640 1 --2- 172.21.15.57:0/1717824495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f212406efb0 0x7f212406e230 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f211c009d90 tx=0x7f211c01f130 comp rx=0 tx=0).stop 2023-10-30T14:51:33.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.020+0000 7f212c19d640 1 -- 172.21.15.57:0/1717824495 shutdown_connections 2023-10-30T14:51:33.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.020+0000 7f212c19d640 1 --2- 172.21.15.57:0/1717824495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f212406efb0 0x7f212406e230 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:33.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.020+0000 7f212c19d640 1 --2- 172.21.15.57:0/1717824495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21240b2000 0x7f212406cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:33.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.020+0000 7f212c19d640 1 -- 172.21.15.57:0/1717824495 >> 172.21.15.57:0/1717824495 conn(0x7f21240723f0 msgr2=0x7f2124072800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:33.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.020+0000 7f212c19d640 1 -- 172.21.15.57:0/1717824495 shutdown_connections 2023-10-30T14:51:33.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.020+0000 7f212c19d640 1 -- 172.21.15.57:0/1717824495 wait complete. 2023-10-30T14:51:33.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f212c19d640 1 Processor -- start 2023-10-30T14:51:33.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f212c19d640 1 -- start start 2023-10-30T14:51:33.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f212c19d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f212406efb0 0x7f21240cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:33.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f212c19d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21240b2000 0x7f21240c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:33.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f212c19d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f212415dcc0 0x7f21240c6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:33.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f212c19d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21240c56b0 con 0x7f212406efb0 2023-10-30T14:51:33.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f212c19d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21240c5c00 con 0x7f212415dcc0 2023-10-30T14:51:33.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f212c19d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21240c5920 con 0x7f21240b2000 2023-10-30T14:51:33.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f2129f39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f212406efb0 0x7f21240cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:33.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f2129f39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f212406efb0 0x7f21240cd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52630/0 (socket says 172.21.15.57:52630) 2023-10-30T14:51:33.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f2129f39640 1 -- 172.21.15.57:0/946011079 learned_addr learned my addr 172.21.15.57:0/946011079 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:33.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f2129738640 1 --2- 172.21.15.57:0/946011079 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21240b2000 0x7f21240c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:33.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f212a73a640 1 --2- 172.21.15.57:0/946011079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f212415dcc0 0x7f21240c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:33.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f2129f39640 1 -- 172.21.15.57:0/946011079 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21240b2000 msgr2=0x7f21240c62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:33.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f2129f39640 1 --2- 172.21.15.57:0/946011079 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21240b2000 0x7f21240c62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:33.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f2129f39640 1 -- 172.21.15.57:0/946011079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f212415dcc0 msgr2=0x7f21240c6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:33.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f2129f39640 1 --2- 172.21.15.57:0/946011079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f212415dcc0 0x7f21240c6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:33.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f2129f39640 1 -- 172.21.15.57:0/946011079 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211c009a70 con 0x7f212406efb0 2023-10-30T14:51:33.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f2129f39640 1 --2- 172.21.15.57:0/946011079 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f212406efb0 0x7f21240cd810 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f212000e890 tx=0x7f212000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:33.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f211a7fc640 1 -- 172.21.15.57:0/946011079 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2120009620 con 0x7f212406efb0 2023-10-30T14:51:33.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f211a7fc640 1 -- 172.21.15.57:0/946011079 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f212000c940 con 0x7f212406efb0 2023-10-30T14:51:33.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f211a7fc640 1 -- 172.21.15.57:0/946011079 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212000ccf0 con 0x7f212406efb0 2023-10-30T14:51:33.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f212c19d640 1 -- 172.21.15.57:0/946011079 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2124159670 con 0x7f212406efb0 2023-10-30T14:51:33.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.024+0000 7f212c19d640 1 -- 172.21.15.57:0/946011079 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f212415afb0 con 0x7f212406efb0 2023-10-30T14:51:33.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.028+0000 7f212c19d640 1 -- 172.21.15.57:0/946011079 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2124159cc0 con 0x7f212406efb0 2023-10-30T14:51:33.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.028+0000 7f211a7fc640 1 -- 172.21.15.57:0/946011079 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2120013470 con 0x7f212406efb0 2023-10-30T14:51:33.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.028+0000 7f211a7fc640 1 --2- 172.21.15.57:0/946011079 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f21080664d0 0x7f2108068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:33.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.028+0000 7f211a7fc640 1 -- 172.21.15.57:0/946011079 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f21200845b0 con 0x7f212406efb0 2023-10-30T14:51:33.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.028+0000 7f2129738640 1 --2- 172.21.15.57:0/946011079 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f21080664d0 0x7f2108068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:33.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.028+0000 7f2129738640 1 --2- 172.21.15.57:0/946011079 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f21080664d0 0x7f2108068990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f21240c8310 tx=0x7f2114009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:33.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.032+0000 7f211a7fc640 1 -- 172.21.15.57:0/946011079 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2120051450 con 0x7f212406efb0 2023-10-30T14:51:33.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.188+0000 7f212c19d640 1 -- 172.21.15.57:0/946011079 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f212404e170 con 0x7f212406efb0 2023-10-30T14:51:33.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.188+0000 7f211a7fc640 1 -- 172.21.15.57:0/946011079 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v67) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2120054a70 con 0x7f212406efb0 2023-10-30T14:51:33.194 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-30T14:51:33.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.192+0000 7f212c19d640 1 -- 172.21.15.57:0/946011079 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f21080664d0 msgr2=0x7f2108068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:33.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.192+0000 7f212c19d640 1 --2- 172.21.15.57:0/946011079 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f21080664d0 0x7f2108068990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f21240c8310 tx=0x7f2114009040 comp rx=0 tx=0).stop 2023-10-30T14:51:33.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.192+0000 7f212c19d640 1 -- 172.21.15.57:0/946011079 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f212406efb0 msgr2=0x7f21240cd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:33.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.192+0000 7f212c19d640 1 --2- 172.21.15.57:0/946011079 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f212406efb0 0x7f21240cd810 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f212000e890 tx=0x7f212000ed50 comp rx=0 tx=0).stop 2023-10-30T14:51:33.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.192+0000 7f212c19d640 1 -- 172.21.15.57:0/946011079 shutdown_connections 2023-10-30T14:51:33.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.192+0000 7f212c19d640 1 --2- 172.21.15.57:0/946011079 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f21080664d0 0x7f2108068990 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:33.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.192+0000 7f212c19d640 1 --2- 172.21.15.57:0/946011079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f212415dcc0 0x7f21240c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:33.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.192+0000 7f212c19d640 1 --2- 172.21.15.57:0/946011079 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21240b2000 0x7f21240c62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:33.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.192+0000 7f212c19d640 1 --2- 172.21.15.57:0/946011079 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f212406efb0 0x7f21240cd810 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:33.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.192+0000 7f212c19d640 1 -- 172.21.15.57:0/946011079 >> 172.21.15.57:0/946011079 conn(0x7f21240723f0 msgr2=0x7f212415b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:33.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.192+0000 7f212c19d640 1 -- 172.21.15.57:0/946011079 shutdown_connections 2023-10-30T14:51:33.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.192+0000 7f212c19d640 1 -- 172.21.15.57:0/946011079 wait complete. 2023-10-30T14:51:33.215 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-30T14:51:33.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.356+0000 7f8f3c009640 1 Processor -- start 2023-10-30T14:51:33.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.356+0000 7f8f3c009640 1 -- start start 2023-10-30T14:51:33.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.356+0000 7f8f3c009640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8f340c4a10 0x7f8f3415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:33.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.356+0000 7f8f3c009640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f340c42a0 0x7f8f340ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:33.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.356+0000 7f8f3c009640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f34159f20 con 0x7f8f34153c80 2023-10-30T14:51:33.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.356+0000 7f8f3c009640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f3415dec0 con 0x7f8f340c4a10 2023-10-30T14:51:33.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.356+0000 7f8f3c009640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f341599a0 con 0x7f8f340c42a0 2023-10-30T14:51:33.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.356+0000 7f8f395a4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8f340c4a10 0x7f8f3415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:33.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.356+0000 7f8f395a4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8f340c4a10 0x7f8f3415cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52634/0 (socket says 172.21.15.57:52634) 2023-10-30T14:51:33.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.356+0000 7f8f395a4640 1 -- 172.21.15.57:0/609536454 learned_addr learned my addr 172.21.15.57:0/609536454 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:33.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.356+0000 7f8f3a5a6640 1 --2- 172.21.15.57:0/609536454 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f340c42a0 0x7f8f340ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:33.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.356+0000 7f8f395a4640 1 -- 172.21.15.57:0/609536454 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f340c42a0 msgr2=0x7f8f340ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:33.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.356+0000 7f8f395a4640 1 --2- 172.21.15.57:0/609536454 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f340c42a0 0x7f8f340ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:33.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.356+0000 7f8f395a4640 1 -- 172.21.15.57:0/609536454 >> v1:172.21.15.88:6789/0 conn(0x7f8f34153c80 legacy=0x7f8f3415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:33.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.356+0000 7f8f395a4640 1 -- 172.21.15.57:0/609536454 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3415d620 con 0x7f8f340c4a10 2023-10-30T14:51:33.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.360+0000 7f8f395a4640 1 --2- 172.21.15.57:0/609536454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f340c4a10 0x7f8f3415cf90 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f8f24009870 tx=0x7f8f2401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf4950aa6f7bf5e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:33.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.360+0000 7f8f38da3640 1 -- 172.21.15.57:0/609536454 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f240a8a30 con 0x7f8f340c4a10 2023-10-30T14:51:33.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.360+0000 7f8f38da3640 1 -- 172.21.15.57:0/609536454 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f240a8b90 con 0x7f8f340c4a10 2023-10-30T14:51:33.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.360+0000 7f8f38da3640 1 -- 172.21.15.57:0/609536454 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f240b0440 con 0x7f8f340c4a10 2023-10-30T14:51:33.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.360+0000 7f8f3c009640 1 -- 172.21.15.57:0/609536454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f340c4a10 msgr2=0x7f8f3415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:33.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.360+0000 7f8f3c009640 1 --2- 172.21.15.57:0/609536454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f340c4a10 0x7f8f3415cf90 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f8f24009870 tx=0x7f8f2401ed30 comp rx=0 tx=0).stop 2023-10-30T14:51:33.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.360+0000 7f8f3c009640 1 -- 172.21.15.57:0/609536454 shutdown_connections 2023-10-30T14:51:33.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.360+0000 7f8f3c009640 1 --2- 172.21.15.57:0/609536454 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f340c42a0 0x7f8f340ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:33.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.360+0000 7f8f3c009640 1 --2- 172.21.15.57:0/609536454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f340c4a10 0x7f8f3415cf90 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:33.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.360+0000 7f8f3c009640 1 -- 172.21.15.57:0/609536454 >> 172.21.15.57:0/609536454 conn(0x7f8f340715c0 msgr2=0x7f8f340719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:33.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.360+0000 7f8f3c009640 1 -- 172.21.15.57:0/609536454 shutdown_connections 2023-10-30T14:51:33.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.360+0000 7f8f3c009640 1 -- 172.21.15.57:0/609536454 wait complete. 2023-10-30T14:51:33.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f3c009640 1 Processor -- start 2023-10-30T14:51:33.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f3c009640 1 -- start start 2023-10-30T14:51:33.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f3c009640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f340c42a0 0x7f8f3414b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:33.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f3c009640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f340c4a10 0x7f8f34149ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:33.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f3c009640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f34153c80 0x7f8f3415c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:33.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f3c009640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f341550d0 con 0x7f8f340c42a0 2023-10-30T14:51:33.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f3c009640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3414ba50 con 0x7f8f340c4a10 2023-10-30T14:51:33.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f3c009640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f34152670 con 0x7f8f34153c80 2023-10-30T14:51:33.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f39da5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f340c42a0 0x7f8f3414b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:33.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f39da5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f340c42a0 0x7f8f3414b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52650/0 (socket says 172.21.15.57:52650) 2023-10-30T14:51:33.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f39da5640 1 -- 172.21.15.57:0/2439141008 learned_addr learned my addr 172.21.15.57:0/2439141008 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:33.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f395a4640 1 --2- 172.21.15.57:0/2439141008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f340c4a10 0x7f8f34149ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:33.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f3a5a6640 1 --2- 172.21.15.57:0/2439141008 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f34153c80 0x7f8f3415c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:33.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f39da5640 1 -- 172.21.15.57:0/2439141008 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f34153c80 msgr2=0x7f8f3415c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:33.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f39da5640 1 --2- 172.21.15.57:0/2439141008 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f34153c80 0x7f8f3415c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:33.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f39da5640 1 -- 172.21.15.57:0/2439141008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f340c4a10 msgr2=0x7f8f34149ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:33.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f39da5640 1 --2- 172.21.15.57:0/2439141008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f340c4a10 0x7f8f34149ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:33.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f39da5640 1 -- 172.21.15.57:0/2439141008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f240094d0 con 0x7f8f340c42a0 2023-10-30T14:51:33.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f39da5640 1 --2- 172.21.15.57:0/2439141008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f340c42a0 0x7f8f3414b510 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f8f3000c3c0 tx=0x7f8f3000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:33.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f2a7fc640 1 -- 172.21.15.57:0/2439141008 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f300052d0 con 0x7f8f340c42a0 2023-10-30T14:51:33.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f2a7fc640 1 -- 172.21.15.57:0/2439141008 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f30005430 con 0x7f8f340c42a0 2023-10-30T14:51:33.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f2a7fc640 1 -- 172.21.15.57:0/2439141008 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f30003840 con 0x7f8f340c42a0 2023-10-30T14:51:33.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f3c009640 1 -- 172.21.15.57:0/2439141008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f341520b0 con 0x7f8f340c42a0 2023-10-30T14:51:33.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.364+0000 7f8f3c009640 1 -- 172.21.15.57:0/2439141008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f3414aa30 con 0x7f8f340c42a0 2023-10-30T14:51:33.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.368+0000 7f8f3c009640 1 -- 172.21.15.57:0/2439141008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f3404e170 con 0x7f8f340c42a0 2023-10-30T14:51:33.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.368+0000 7f8f2a7fc640 1 -- 172.21.15.57:0/2439141008 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8f300039a0 con 0x7f8f340c42a0 2023-10-30T14:51:33.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.368+0000 7f8f2a7fc640 1 --2- 172.21.15.57:0/2439141008 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f180664a0 0x7f8f18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:33.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.368+0000 7f8f2a7fc640 1 -- 172.21.15.57:0/2439141008 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f8f30083ad0 con 0x7f8f340c42a0 2023-10-30T14:51:33.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.368+0000 7f8f395a4640 1 --2- 172.21.15.57:0/2439141008 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f180664a0 0x7f8f18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:33.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.368+0000 7f8f395a4640 1 --2- 172.21.15.57:0/2439141008 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f180664a0 0x7f8f18068960 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f8f240b2000 tx=0x7f8f24009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:33.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.372+0000 7f8f2a7fc640 1 -- 172.21.15.57:0/2439141008 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8f30050850 con 0x7f8f340c42a0 2023-10-30T14:51:33.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.544+0000 7f8f3c009640 1 -- 172.21.15.57:0/2439141008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8f340cad40 con 0x7f8f340c42a0 2023-10-30T14:51:33.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.544+0000 7f8f2a7fc640 1 -- 172.21.15.57:0/2439141008 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 36 entries v67) v1 ==== 81+0+2188 (secure 0 0 0) 0x7f8f30017080 con 0x7f8f340c42a0 2023-10-30T14:51:33.549 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6835/2613636524 2023-10-31T14:50:50.290535+0000 2023-10-30T14:51:33.549 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6837/816502245 2023-10-31T14:48:30.252448+0000 2023-10-30T14:51:33.549 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.88:6838/1744808283 2023-10-31T14:44:20.161100+0000 2023-10-30T14:51:33.550 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6834/2321408149 2023-10-31T14:42:35.131496+0000 2023-10-30T14:51:33.550 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6833/816502245 2023-10-31T14:48:30.252448+0000 2023-10-30T14:51:33.550 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6834/2912050965 2023-10-31T14:23:54.764165+0000 2023-10-30T14:51:33.550 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6837/1988100930 2023-10-31T14:20:34.693623+0000 2023-10-30T14:51:33.551 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.88:6839/2945854794 2023-10-31T14:18:54.654873+0000 2023-10-30T14:51:33.551 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6835/2427081813 2023-10-31T14:26:24.805473+0000 2023-10-30T14:51:33.551 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6833/1988100930 2023-10-31T14:20:34.693623+0000 2023-10-30T14:51:33.551 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.88:6839/358521496 2023-10-31T14:30:09.871507+0000 2023-10-30T14:51:33.552 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6836/2693548867 2023-10-31T14:17:44.635085+0000 2023-10-30T14:51:33.552 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6832/2912050965 2023-10-31T14:23:54.764165+0000 2023-10-30T14:51:33.552 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6832/3701655528 2023-10-31T14:16:49.618077+0000 2023-10-30T14:51:33.553 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.88:6838/358521496 2023-10-31T14:30:09.871507+0000 2023-10-30T14:51:33.553 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6837/901968115 2023-10-31T14:14:59.588353+0000 2023-10-30T14:51:33.553 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.88:6839/1744808283 2023-10-31T14:44:20.161100+0000 2023-10-30T14:51:33.553 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.88:6838/2945854794 2023-10-31T14:18:54.654873+0000 2023-10-30T14:51:33.554 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6834/3701655528 2023-10-31T14:16:49.618077+0000 2023-10-30T14:51:33.554 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6835/2733323524 2023-10-31T14:32:54.940701+0000 2023-10-30T14:51:33.554 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6836/2733323524 2023-10-31T14:32:54.940701+0000 2023-10-30T14:51:33.554 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6835/2693548867 2023-10-31T14:17:44.635085+0000 2023-10-30T14:51:33.555 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6832/2321408149 2023-10-31T14:42:35.131496+0000 2023-10-30T14:51:33.555 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6833/862088802 2023-10-31T14:27:49.829937+0000 2023-10-30T14:51:33.555 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.88:6838/3604033338 2023-10-31T14:35:29.980317+0000 2023-10-30T14:51:33.556 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6836/2613636524 2023-10-31T14:50:50.290535+0000 2023-10-30T14:51:33.556 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.88:6839/2390289388 2023-10-31T14:39:45.086742+0000 2023-10-30T14:51:33.556 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6837/862088802 2023-10-31T14:27:49.829937+0000 2023-10-30T14:51:33.556 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.88:6838/2390289388 2023-10-31T14:39:45.086742+0000 2023-10-30T14:51:33.557 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6832/2940934016 2023-10-31T14:29:09.852381+0000 2023-10-30T14:51:33.557 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6834/2940934016 2023-10-31T14:29:09.852381+0000 2023-10-30T14:51:33.557 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.88:6839/3604033338 2023-10-31T14:35:29.980317+0000 2023-10-30T14:51:33.558 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6833/901968115 2023-10-31T14:14:59.588353+0000 2023-10-30T14:51:33.558 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6836/2427081813 2023-10-31T14:26:24.805473+0000 2023-10-30T14:51:33.558 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6833/1843528309 2023-10-31T14:38:15.025313+0000 2023-10-30T14:51:33.558 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6837/1843528309 2023-10-31T14:38:15.025313+0000 2023-10-30T14:51:33.559 INFO:teuthology.orchestra.run.smithi057.stderr:listed 36 entries 2023-10-30T14:51:33.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.544+0000 7f8f3c009640 1 -- 172.21.15.57:0/2439141008 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f180664a0 msgr2=0x7f8f18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:33.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.544+0000 7f8f3c009640 1 --2- 172.21.15.57:0/2439141008 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f180664a0 0x7f8f18068960 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f8f240b2000 tx=0x7f8f24009040 comp rx=0 tx=0).stop 2023-10-30T14:51:33.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.544+0000 7f8f3c009640 1 -- 172.21.15.57:0/2439141008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f340c42a0 msgr2=0x7f8f3414b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:33.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.544+0000 7f8f3c009640 1 --2- 172.21.15.57:0/2439141008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f340c42a0 0x7f8f3414b510 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f8f3000c3c0 tx=0x7f8f3000c880 comp rx=0 tx=0).stop 2023-10-30T14:51:33.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.544+0000 7f8f3c009640 1 -- 172.21.15.57:0/2439141008 shutdown_connections 2023-10-30T14:51:33.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.544+0000 7f8f3c009640 1 --2- 172.21.15.57:0/2439141008 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8f180664a0 0x7f8f18068960 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:33.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.544+0000 7f8f3c009640 1 --2- 172.21.15.57:0/2439141008 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f34153c80 0x7f8f3415c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:33.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.544+0000 7f8f3c009640 1 --2- 172.21.15.57:0/2439141008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f340c4a10 0x7f8f34149ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:33.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.544+0000 7f8f3c009640 1 --2- 172.21.15.57:0/2439141008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f340c42a0 0x7f8f3414b510 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:33.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.544+0000 7f8f3c009640 1 -- 172.21.15.57:0/2439141008 >> 172.21.15.57:0/2439141008 conn(0x7f8f340715c0 msgr2=0x7f8f340c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:33.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.544+0000 7f8f3c009640 1 -- 172.21.15.57:0/2439141008 shutdown_connections 2023-10-30T14:51:33.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:33.544+0000 7f8f3c009640 1 -- 172.21.15.57:0/2439141008 wait complete. 2023-10-30T14:51:33.569 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-30T14:51:33.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:51:33.570 DEBUG:teuthology.orchestra.run.smithi057:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-30T14:51:34.407 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-30T14:51:34.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:51:34.408 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-30T14:51:34.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:51:34.428 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-30T14:51:34.483 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-30T14:51:34.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:51:34.486 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-30T14:51:34.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:51:34.555 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link delete ceph-brx 2023-10-30T14:51:34.805 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-30T14:51:34.817 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T14:51:34.818 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:51:34.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.956+0000 7f547a8c9640 1 Processor -- start 2023-10-30T14:51:34.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.956+0000 7f547a8c9640 1 -- start start 2023-10-30T14:51:34.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.960+0000 7f547a8c9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5474153aa0 0x7f547415d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:34.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.960+0000 7f547a8c9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f547406b3b0 0x7f547406b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:34.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.960+0000 7f547a8c9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54740c7b00 con 0x7f54740701e0 2023-10-30T14:51:34.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.960+0000 7f547a8c9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5474070a50 con 0x7f5474153aa0 2023-10-30T14:51:34.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.960+0000 7f547a8c9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f547415dd80 con 0x7f547406b3b0 2023-10-30T14:51:34.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.960+0000 7f546bfff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f54740701e0 0x7f54740ccee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41796/0 (socket says 172.21.15.57:41796) 2023-10-30T14:51:34.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.960+0000 7f546bfff640 1 -- 172.21.15.57:0/2443175078 learned_addr learned my addr 172.21.15.57:0/2443175078 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:51:34.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.960+0000 7f5473fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5474153aa0 0x7f547415d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:34.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.960+0000 7f5478e66640 1 --2- 172.21.15.57:0/2443175078 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f547406b3b0 0x7f547406b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:34.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.960+0000 7f5473fff640 1 -- 172.21.15.57:0/2443175078 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f547406b3b0 msgr2=0x7f547406b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:34.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.960+0000 7f5473fff640 1 --2- 172.21.15.57:0/2443175078 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f547406b3b0 0x7f547406b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:34.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.960+0000 7f5473fff640 1 -- 172.21.15.57:0/2443175078 >> v1:172.21.15.57:6789/0 conn(0x7f54740701e0 legacy=0x7f54740ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:34.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.960+0000 7f5473fff640 1 -- 172.21.15.57:0/2443175078 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547406bd60 con 0x7f5474153aa0 2023-10-30T14:51:34.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.960+0000 7f54737fe640 1 -- 172.21.15.57:0/2443175078 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463422714 0 0) 0x7f54740c7b00 con 0x7f54740701e0 2023-10-30T14:51:34.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.960+0000 7f5473fff640 1 --2- 172.21.15.57:0/2443175078 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5474153aa0 0x7f547415d840 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f54640097c0 tx=0x7f546401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47db35fbf1fedf26 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:34.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.960+0000 7f54737fe640 1 -- 172.21.15.57:0/2443175078 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54640a8970 con 0x7f5474153aa0 2023-10-30T14:51:34.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.960+0000 7f54737fe640 1 -- 172.21.15.57:0/2443175078 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54640a8ad0 con 0x7f5474153aa0 2023-10-30T14:51:34.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.960+0000 7f54737fe640 1 -- 172.21.15.57:0/2443175078 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54640a8e60 con 0x7f5474153aa0 2023-10-30T14:51:34.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f547a8c9640 1 -- 172.21.15.57:0/2443175078 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5474153aa0 msgr2=0x7f547415d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:34.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f547a8c9640 1 --2- 172.21.15.57:0/2443175078 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5474153aa0 0x7f547415d840 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f54640097c0 tx=0x7f546401ec70 comp rx=0 tx=0).stop 2023-10-30T14:51:34.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f547a8c9640 1 -- 172.21.15.57:0/2443175078 shutdown_connections 2023-10-30T14:51:34.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f547a8c9640 1 --2- 172.21.15.57:0/2443175078 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f547406b3b0 0x7f547406b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:34.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f547a8c9640 1 --2- 172.21.15.57:0/2443175078 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5474153aa0 0x7f547415d840 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:34.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f547a8c9640 1 -- 172.21.15.57:0/2443175078 >> 172.21.15.57:0/2443175078 conn(0x7f5474071fb0 msgr2=0x7f54740723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:34.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f547a8c9640 1 -- 172.21.15.57:0/2443175078 shutdown_connections 2023-10-30T14:51:34.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f547a8c9640 1 -- 172.21.15.57:0/2443175078 wait complete. 2023-10-30T14:51:34.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f547a8c9640 1 Processor -- start 2023-10-30T14:51:34.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f547a8c9640 1 -- start start 2023-10-30T14:51:34.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f547a8c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f547406b3b0 0x7f54741553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:34.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f547a8c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54740701e0 0x7f54741499f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:34.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f547a8c9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5474153aa0 0x7f5474149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:34.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f547a8c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547415d0b0 con 0x7f547406b3b0 2023-10-30T14:51:34.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f547a8c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5474157850 con 0x7f54740701e0 2023-10-30T14:51:34.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f547a8c9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5474157b30 con 0x7f5474153aa0 2023-10-30T14:51:34.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f5473fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f547406b3b0 0x7f54741553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:34.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f5473fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f547406b3b0 0x7f54741553c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56602/0 (socket says 172.21.15.57:56602) 2023-10-30T14:51:34.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f5473fff640 1 -- 172.21.15.57:0/3129814617 learned_addr learned my addr 172.21.15.57:0/3129814617 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:34.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f5478e66640 1 --2- 172.21.15.57:0/3129814617 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5474153aa0 0x7f5474149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:34.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.964+0000 7f546bfff640 1 --2- 172.21.15.57:0/3129814617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54740701e0 0x7f54741499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:34.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.968+0000 7f5473fff640 1 -- 172.21.15.57:0/3129814617 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5474153aa0 msgr2=0x7f5474149f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:34.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.968+0000 7f5473fff640 1 --2- 172.21.15.57:0/3129814617 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5474153aa0 0x7f5474149f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:34.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.968+0000 7f5473fff640 1 -- 172.21.15.57:0/3129814617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54740701e0 msgr2=0x7f54741499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:34.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.968+0000 7f5473fff640 1 --2- 172.21.15.57:0/3129814617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54740701e0 0x7f54741499f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:34.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.968+0000 7f5473fff640 1 -- 172.21.15.57:0/3129814617 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5464009420 con 0x7f547406b3b0 2023-10-30T14:51:34.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.968+0000 7f5473fff640 1 --2- 172.21.15.57:0/3129814617 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f547406b3b0 0x7f54741553c0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f54640b3040 tx=0x7f546401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:34.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.968+0000 7f54717fa640 1 -- 172.21.15.57:0/3129814617 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54640b9630 con 0x7f547406b3b0 2023-10-30T14:51:34.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.968+0000 7f54717fa640 1 -- 172.21.15.57:0/3129814617 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54640b9790 con 0x7f547406b3b0 2023-10-30T14:51:34.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.968+0000 7f54717fa640 1 -- 172.21.15.57:0/3129814617 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54640b9b40 con 0x7f547406b3b0 2023-10-30T14:51:34.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.968+0000 7f547a8c9640 1 -- 172.21.15.57:0/3129814617 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f547415c710 con 0x7f547406b3b0 2023-10-30T14:51:34.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.968+0000 7f547a8c9640 1 -- 172.21.15.57:0/3129814617 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f547415ccd0 con 0x7f547406b3b0 2023-10-30T14:51:34.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.968+0000 7f547a8c9640 1 -- 172.21.15.57:0/3129814617 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54740ca4f0 con 0x7f547406b3b0 2023-10-30T14:51:34.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.968+0000 7f54717fa640 1 -- 172.21.15.57:0/3129814617 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f54640b7040 con 0x7f547406b3b0 2023-10-30T14:51:34.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.968+0000 7f54717fa640 1 --2- 172.21.15.57:0/3129814617 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f54540664a0 0x7f5454068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:34.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.968+0000 7f54717fa640 1 -- 172.21.15.57:0/3129814617 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f54641223e0 con 0x7f547406b3b0 2023-10-30T14:51:34.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.968+0000 7f546bfff640 1 --2- 172.21.15.57:0/3129814617 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f54540664a0 0x7f5454068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:34.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.972+0000 7f546bfff640 1 --2- 172.21.15.57:0/3129814617 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f54540664a0 0x7f5454068960 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f547415c640 tx=0x7f545c007b60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:34.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:34.972+0000 7f54717fa640 1 -- 172.21.15.57:0/3129814617 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f54640ef150 con 0x7f547406b3b0 2023-10-30T14:51:35.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.128+0000 7f547a8c9640 1 -- 172.21.15.57:0/3129814617 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f54740c2330 con 0x7f54540664a0 2023-10-30T14:51:35.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.152+0000 7f54717fa640 1 -- 172.21.15.57:0/3129814617 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+293482 (secure 0 0 0) 0x7f545c0071f0 con 0x7f54540664a0 2023-10-30T14:51:35.161 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:51:35.163 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-30T14:51:35.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.156+0000 7f547a8c9640 1 -- 172.21.15.57:0/3129814617 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f54540664a0 msgr2=0x7f5454068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:35.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.156+0000 7f547a8c9640 1 --2- 172.21.15.57:0/3129814617 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f54540664a0 0x7f5454068960 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f547415c640 tx=0x7f545c007b60 comp rx=0 tx=0).stop 2023-10-30T14:51:35.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.156+0000 7f547a8c9640 1 -- 172.21.15.57:0/3129814617 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f547406b3b0 msgr2=0x7f54741553c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:35.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.156+0000 7f547a8c9640 1 --2- 172.21.15.57:0/3129814617 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f547406b3b0 0x7f54741553c0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f54640b3040 tx=0x7f546401ba10 comp rx=0 tx=0).stop 2023-10-30T14:51:35.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.156+0000 7f547a8c9640 1 -- 172.21.15.57:0/3129814617 shutdown_connections 2023-10-30T14:51:35.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.156+0000 7f547a8c9640 1 --2- 172.21.15.57:0/3129814617 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f54540664a0 0x7f5454068960 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.156+0000 7f547a8c9640 1 --2- 172.21.15.57:0/3129814617 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5474153aa0 0x7f5474149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.156+0000 7f547a8c9640 1 --2- 172.21.15.57:0/3129814617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54740701e0 0x7f54741499f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.156+0000 7f547a8c9640 1 --2- 172.21.15.57:0/3129814617 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f547406b3b0 0x7f54741553c0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.156+0000 7f547a8c9640 1 -- 172.21.15.57:0/3129814617 >> 172.21.15.57:0/3129814617 conn(0x7f5474071fb0 msgr2=0x7f5474151f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:35.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.156+0000 7f547a8c9640 1 -- 172.21.15.57:0/3129814617 shutdown_connections 2023-10-30T14:51:35.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.156+0000 7f547a8c9640 1 -- 172.21.15.57:0/3129814617 wait complete. 2023-10-30T14:51:35.180 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":1191,"stamp":"2023-10-30T14:51:33.316877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1721022653,"num_objects":32709,"num_object_clones":14593,"num_object_copies":65418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32709,"num_whiteouts":12799,"num_read":10747,"num_read_kb":3497812,"num_write":97664,"num_write_kb":2851506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1111,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109301,"ondisk_log_size":109301,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":4473736,"kb_used_data":3440456,"kb_used_omap":12,"kb_used_meta":1033203,"kb_avail":750500984,"statfs":{"total":773094113280,"available":768513007616,"internally_reserved":0,"allocated":3523026944,"data_stored":3445826398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":1058000441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":75,"apply_latency_ms":75,"commit_latency_ns":75000000,"apply_latency_ns":75000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36042696,"num_objects":236,"num_object_clones":182,"num_object_copies":472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":236,"num_whiteouts":24,"num_read":168,"num_read_kb":5180,"num_write":5012,"num_write_kb":42498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004406"},"pg_stats":[{"pgid":"3.3f","version":"67'1276","reported_seq":3239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.482005+0000","last_change":"2023-10-30T14:23:31.082534+0000","last_active":"2023-10-30T14:51:28.482005+0000","last_peered":"2023-10-30T14:51:28.482005+0000","last_clean":"2023-10-30T14:51:28.482005+0000","last_became_active":"2023-10-30T14:14:04.650882+0000","last_became_peered":"2023-10-30T14:14:04.650882+0000","last_unstale":"2023-10-30T14:51:28.482005+0000","last_undegraded":"2023-10-30T14:51:28.482005+0000","last_fullsized":"2023-10-30T14:51:28.482005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1276,"log_dups_size":0,"ondisk_log_size":1276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:46:41.132286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010330941,"stat_sum":{"num_bytes":16875912,"num_objects":459,"num_object_clones":215,"num_object_copies":918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":459,"num_whiteouts":184,"num_read":62,"num_read_kb":8246,"num_write":1073,"num_write_kb":18488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"67'1268","reported_seq":3223,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.429379+0000","last_change":"2023-10-30T14:23:30.771600+0000","last_active":"2023-10-30T14:51:29.429379+0000","last_peered":"2023-10-30T14:51:29.429379+0000","last_clean":"2023-10-30T14:51:29.429379+0000","last_became_active":"2023-10-30T14:14:04.587942+0000","last_became_peered":"2023-10-30T14:14:04.587942+0000","last_unstale":"2023-10-30T14:51:29.429379+0000","last_undegraded":"2023-10-30T14:51:29.429379+0000","last_fullsized":"2023-10-30T14:51:29.429379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1268,"log_dups_size":0,"ondisk_log_size":1268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:54:24.389508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002850184,"stat_sum":{"num_bytes":12592181,"num_objects":480,"num_object_clones":220,"num_object_copies":960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":480,"num_whiteouts":195,"num_read":59,"num_read_kb":8220,"num_write":1065,"num_write_kb":14470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"67'1295","reported_seq":3258,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.481286+0000","last_change":"2023-10-30T14:23:30.776754+0000","last_active":"2023-10-30T14:51:28.481286+0000","last_peered":"2023-10-30T14:51:28.481286+0000","last_clean":"2023-10-30T14:51:28.481286+0000","last_became_active":"2023-10-30T14:14:04.590700+0000","last_became_peered":"2023-10-30T14:14:04.590700+0000","last_unstale":"2023-10-30T14:51:28.481286+0000","last_undegraded":"2023-10-30T14:51:28.481286+0000","last_fullsized":"2023-10-30T14:51:28.481286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:11:25.956171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042861139999999997,"stat_sum":{"num_bytes":12573829,"num_objects":487,"num_object_clones":226,"num_object_copies":974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":204,"num_read":73,"num_read_kb":16409,"num_write":1057,"num_write_kb":22081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"67'1290","reported_seq":3252,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.429904+0000","last_change":"2023-10-30T14:23:31.067598+0000","last_active":"2023-10-30T14:51:29.429904+0000","last_peered":"2023-10-30T14:51:29.429904+0000","last_clean":"2023-10-30T14:51:29.429904+0000","last_became_active":"2023-10-30T14:14:45.661677+0000","last_became_peered":"2023-10-30T14:14:45.661677+0000","last_unstale":"2023-10-30T14:51:29.429904+0000","last_undegraded":"2023-10-30T14:51:29.429904+0000","last_fullsized":"2023-10-30T14:51:29.429904+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1290,"log_dups_size":0,"ondisk_log_size":1290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:33.531763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003131572,"stat_sum":{"num_bytes":8739581,"num_objects":477,"num_object_clones":215,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":188,"num_read":63,"num_read_kb":69,"num_write":1083,"num_write_kb":14952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"67'1295","reported_seq":3255,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.455528+0000","last_change":"2023-10-30T14:23:31.072132+0000","last_active":"2023-10-30T14:51:28.455528+0000","last_peered":"2023-10-30T14:51:28.455528+0000","last_clean":"2023-10-30T14:51:28.455528+0000","last_became_active":"2023-10-30T14:14:04.644683+0000","last_became_peered":"2023-10-30T14:14:04.644683+0000","last_unstale":"2023-10-30T14:51:28.455528+0000","last_undegraded":"2023-10-30T14:51:28.455528+0000","last_fullsized":"2023-10-30T14:51:28.455528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:39.271356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052462430000000003,"stat_sum":{"num_bytes":8813426,"num_objects":481,"num_object_clones":223,"num_object_copies":962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":202,"num_read":70,"num_read_kb":4116,"num_write":1073,"num_write_kb":10060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'1284","reported_seq":3252,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.493921+0000","last_change":"2023-10-30T14:23:30.777057+0000","last_active":"2023-10-30T14:51:28.493921+0000","last_peered":"2023-10-30T14:51:28.493921+0000","last_clean":"2023-10-30T14:51:28.493921+0000","last_became_active":"2023-10-30T14:14:04.619335+0000","last_became_peered":"2023-10-30T14:14:04.619335+0000","last_unstale":"2023-10-30T14:51:28.493921+0000","last_undegraded":"2023-10-30T14:51:28.493921+0000","last_fullsized":"2023-10-30T14:51:28.493921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1284,"log_dups_size":0,"ondisk_log_size":1284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:16:48.356144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062283859999999998,"stat_sum":{"num_bytes":20637019,"num_objects":477,"num_object_clones":220,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":194,"num_read":75,"num_read_kb":12254,"num_write":1076,"num_write_kb":22206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"67'1281","reported_seq":3231,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.464417+0000","last_change":"2023-10-30T14:23:31.080820+0000","last_active":"2023-10-30T14:51:29.464417+0000","last_peered":"2023-10-30T14:51:29.464417+0000","last_clean":"2023-10-30T14:51:29.464417+0000","last_became_active":"2023-10-30T14:14:04.641323+0000","last_became_peered":"2023-10-30T14:14:04.641323+0000","last_unstale":"2023-10-30T14:51:29.464417+0000","last_undegraded":"2023-10-30T14:51:29.464417+0000","last_fullsized":"2023-10-30T14:51:29.464417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1281,"log_dups_size":0,"ondisk_log_size":1281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:11.764013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011228897,"stat_sum":{"num_bytes":14656224,"num_objects":488,"num_object_clones":222,"num_object_copies":976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":202,"num_read":59,"num_read_kb":8203,"num_write":1074,"num_write_kb":18148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'1241","reported_seq":3196,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.459518+0000","last_change":"2023-10-30T14:23:31.067811+0000","last_active":"2023-10-30T14:51:29.459518+0000","last_peered":"2023-10-30T14:51:29.459518+0000","last_clean":"2023-10-30T14:51:29.459518+0000","last_became_active":"2023-10-30T14:14:04.618312+0000","last_became_peered":"2023-10-30T14:14:04.618312+0000","last_unstale":"2023-10-30T14:51:29.459518+0000","last_undegraded":"2023-10-30T14:51:29.459518+0000","last_fullsized":"2023-10-30T14:51:29.459518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1241,"log_dups_size":0,"ondisk_log_size":1241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:57:22.752126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002849426,"stat_sum":{"num_bytes":12868892,"num_objects":473,"num_object_clones":218,"num_object_copies":946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":473,"num_whiteouts":195,"num_read":60,"num_read_kb":8209,"num_write":1023,"num_write_kb":15138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"67'1328","reported_seq":3278,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.473222+0000","last_change":"2023-10-30T14:23:31.083143+0000","last_active":"2023-10-30T14:51:29.473222+0000","last_peered":"2023-10-30T14:51:29.473222+0000","last_clean":"2023-10-30T14:51:29.473222+0000","last_became_active":"2023-10-30T14:14:04.647361+0000","last_became_peered":"2023-10-30T14:14:04.647361+0000","last_unstale":"2023-10-30T14:51:29.473222+0000","last_undegraded":"2023-10-30T14:51:29.473222+0000","last_fullsized":"2023-10-30T14:51:29.473222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1328,"log_dups_size":0,"ondisk_log_size":1328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:06:53.616905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010459703000000001,"stat_sum":{"num_bytes":8468640,"num_objects":507,"num_object_clones":234,"num_object_copies":1014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":507,"num_whiteouts":206,"num_read":70,"num_read_kb":8267,"num_write":1118,"num_write_kb":13882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"67'1366","reported_seq":3270,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.468701+0000","last_change":"2023-10-30T14:23:31.075673+0000","last_active":"2023-10-30T14:51:29.468701+0000","last_peered":"2023-10-30T14:51:29.468701+0000","last_clean":"2023-10-30T14:51:29.468701+0000","last_became_active":"2023-10-30T14:14:04.630959+0000","last_became_peered":"2023-10-30T14:14:04.630959+0000","last_unstale":"2023-10-30T14:51:29.468701+0000","last_undegraded":"2023-10-30T14:51:29.468701+0000","last_fullsized":"2023-10-30T14:51:29.468701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1366,"log_dups_size":0,"ondisk_log_size":1366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:36.246017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079325130000000004,"stat_sum":{"num_bytes":12335215,"num_objects":539,"num_object_clones":286,"num_object_copies":1078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":191,"num_read":76,"num_read_kb":4189,"num_write":1090,"num_write_kb":13918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"67'1287","reported_seq":3229,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469676+0000","last_change":"2023-10-30T14:23:31.075101+0000","last_active":"2023-10-30T14:51:29.469676+0000","last_peered":"2023-10-30T14:51:29.469676+0000","last_clean":"2023-10-30T14:51:29.469676+0000","last_became_active":"2023-10-30T14:14:04.612653+0000","last_became_peered":"2023-10-30T14:14:04.612653+0000","last_unstale":"2023-10-30T14:51:29.469676+0000","last_undegraded":"2023-10-30T14:51:29.469676+0000","last_fullsized":"2023-10-30T14:51:29.469676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1287,"log_dups_size":0,"ondisk_log_size":1287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:34:44.855573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079098320000000003,"stat_sum":{"num_bytes":17654064,"num_objects":517,"num_object_clones":235,"num_object_copies":1034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":517,"num_whiteouts":205,"num_read":63,"num_read_kb":9809,"num_write":1073,"num_write_kb":19026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"67'1267","reported_seq":3224,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469272+0000","last_change":"2023-10-30T14:23:30.957983+0000","last_active":"2023-10-30T14:51:29.469272+0000","last_peered":"2023-10-30T14:51:29.469272+0000","last_clean":"2023-10-30T14:51:29.469272+0000","last_became_active":"2023-10-30T14:14:04.621135+0000","last_became_peered":"2023-10-30T14:14:04.621135+0000","last_unstale":"2023-10-30T14:51:29.469272+0000","last_undegraded":"2023-10-30T14:51:29.469272+0000","last_fullsized":"2023-10-30T14:51:29.469272+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1267,"log_dups_size":0,"ondisk_log_size":1267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:04:43.303589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025042409999999999,"stat_sum":{"num_bytes":14287686,"num_objects":497,"num_object_clones":224,"num_object_copies":994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":497,"num_whiteouts":193,"num_read":67,"num_read_kb":6339,"num_write":1071,"num_write_kb":19938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"67'1268","reported_seq":3251,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.468243+0000","last_change":"2023-10-30T14:23:31.070450+0000","last_active":"2023-10-30T14:51:29.468243+0000","last_peered":"2023-10-30T14:51:29.468243+0000","last_clean":"2023-10-30T14:51:29.468243+0000","last_became_active":"2023-10-30T14:14:04.616142+0000","last_became_peered":"2023-10-30T14:14:04.616142+0000","last_unstale":"2023-10-30T14:51:29.468243+0000","last_undegraded":"2023-10-30T14:51:29.468243+0000","last_fullsized":"2023-10-30T14:51:29.468243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1268,"log_dups_size":0,"ondisk_log_size":1268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:46:20.636160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049623779999999999,"stat_sum":{"num_bytes":9235702,"num_objects":480,"num_object_clones":219,"num_object_copies":960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":480,"num_whiteouts":189,"num_read":88,"num_read_kb":8286,"num_write":1065,"num_write_kb":19474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"67'1307","reported_seq":3281,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.633026+0000","last_change":"2023-10-30T14:23:30.775687+0000","last_active":"2023-10-30T14:51:29.633026+0000","last_peered":"2023-10-30T14:51:29.633026+0000","last_clean":"2023-10-30T14:51:29.633026+0000","last_became_active":"2023-10-30T14:14:04.588452+0000","last_became_peered":"2023-10-30T14:14:04.588452+0000","last_unstale":"2023-10-30T14:51:29.633026+0000","last_undegraded":"2023-10-30T14:51:29.633026+0000","last_fullsized":"2023-10-30T14:51:29.633026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1307,"log_dups_size":0,"ondisk_log_size":1307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:40:26.637820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046351760000000004,"stat_sum":{"num_bytes":12783229,"num_objects":466,"num_object_clones":216,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":195,"num_read":76,"num_read_kb":7792,"num_write":1098,"num_write_kb":17084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"67'1312","reported_seq":3276,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.502566+0000","last_change":"2023-10-30T14:23:30.770743+0000","last_active":"2023-10-30T14:51:28.502566+0000","last_peered":"2023-10-30T14:51:28.502566+0000","last_clean":"2023-10-30T14:51:28.502566+0000","last_became_active":"2023-10-30T14:14:04.620781+0000","last_became_peered":"2023-10-30T14:14:04.620781+0000","last_unstale":"2023-10-30T14:51:28.502566+0000","last_undegraded":"2023-10-30T14:51:28.502566+0000","last_fullsized":"2023-10-30T14:51:28.502566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1312,"log_dups_size":0,"ondisk_log_size":1312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:09.260750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002316706,"stat_sum":{"num_bytes":21325663,"num_objects":484,"num_object_clones":220,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":196,"num_read":69,"num_read_kb":13296,"num_write":1104,"num_write_kb":27083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"67'1296","reported_seq":3281,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.502845+0000","last_change":"2023-10-30T14:23:30.774013+0000","last_active":"2023-10-30T14:51:28.502845+0000","last_peered":"2023-10-30T14:51:28.502845+0000","last_clean":"2023-10-30T14:51:28.502845+0000","last_became_active":"2023-10-30T14:14:04.608190+0000","last_became_peered":"2023-10-30T14:14:04.608190+0000","last_unstale":"2023-10-30T14:51:28.502845+0000","last_undegraded":"2023-10-30T14:51:28.502845+0000","last_fullsized":"2023-10-30T14:51:28.502845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1296,"log_dups_size":0,"ondisk_log_size":1296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:44:40.113536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035618450000000001,"stat_sum":{"num_bytes":31744659,"num_objects":477,"num_object_clones":215,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":193,"num_read":85,"num_read_kb":23305,"num_write":1082,"num_write_kb":37194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"67'1351","reported_seq":3310,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469699+0000","last_change":"2023-10-30T14:23:30.779367+0000","last_active":"2023-10-30T14:51:29.469699+0000","last_peered":"2023-10-30T14:51:29.469699+0000","last_clean":"2023-10-30T14:51:29.469699+0000","last_became_active":"2023-10-30T14:14:04.641306+0000","last_became_peered":"2023-10-30T14:14:04.641306+0000","last_unstale":"2023-10-30T14:51:29.469699+0000","last_undegraded":"2023-10-30T14:51:29.469699+0000","last_fullsized":"2023-10-30T14:51:29.469699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1351,"log_dups_size":0,"ondisk_log_size":1351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:53:12.985530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066189990000000004,"stat_sum":{"num_bytes":3981038,"num_objects":511,"num_object_clones":236,"num_object_copies":1022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":206,"num_read":80,"num_read_kb":4151,"num_write":1127,"num_write_kb":9503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"67'1221","reported_seq":3204,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469982+0000","last_change":"2023-10-30T14:23:31.067340+0000","last_active":"2023-10-30T14:51:29.469982+0000","last_peered":"2023-10-30T14:51:29.469982+0000","last_clean":"2023-10-30T14:51:29.469982+0000","last_became_active":"2023-10-30T14:14:04.613644+0000","last_became_peered":"2023-10-30T14:14:04.613644+0000","last_unstale":"2023-10-30T14:51:29.469982+0000","last_undegraded":"2023-10-30T14:51:29.469982+0000","last_fullsized":"2023-10-30T14:51:29.469982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1221,"log_dups_size":0,"ondisk_log_size":1221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:50:40.405415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030699740000000001,"stat_sum":{"num_bytes":12387639,"num_objects":435,"num_object_clones":203,"num_object_copies":870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":435,"num_whiteouts":177,"num_read":71,"num_read_kb":8233,"num_write":1033,"num_write_kb":14118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"67'1285","reported_seq":3258,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.502929+0000","last_change":"2023-10-30T14:23:31.082413+0000","last_active":"2023-10-30T14:51:28.502929+0000","last_peered":"2023-10-30T14:51:28.502929+0000","last_clean":"2023-10-30T14:51:28.502929+0000","last_became_active":"2023-10-30T14:14:04.649850+0000","last_became_peered":"2023-10-30T14:14:04.649850+0000","last_unstale":"2023-10-30T14:51:28.502929+0000","last_undegraded":"2023-10-30T14:51:28.502929+0000","last_fullsized":"2023-10-30T14:51:28.502929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1285,"log_dups_size":0,"ondisk_log_size":1285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:27:56.738513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012599246,"stat_sum":{"num_bytes":25948792,"num_objects":470,"num_object_clones":219,"num_object_copies":940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":192,"num_read":77,"num_read_kb":24658,"num_write":1071,"num_write_kb":31961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"67'1335","reported_seq":3270,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.012858+0000","last_change":"2023-10-30T14:23:31.074724+0000","last_active":"2023-10-30T14:51:25.012858+0000","last_peered":"2023-10-30T14:51:25.012858+0000","last_clean":"2023-10-30T14:51:25.012858+0000","last_became_active":"2023-10-30T14:14:04.625042+0000","last_became_peered":"2023-10-30T14:14:04.625042+0000","last_unstale":"2023-10-30T14:51:25.012858+0000","last_undegraded":"2023-10-30T14:51:25.012858+0000","last_fullsized":"2023-10-30T14:51:25.012858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1335,"log_dups_size":0,"ondisk_log_size":1335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:45.303841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082351539999999997,"stat_sum":{"num_bytes":12700226,"num_objects":511,"num_object_clones":241,"num_object_copies":1022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":214,"num_read":61,"num_read_kb":10278,"num_write":1102,"num_write_kb":27066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"67'1264","reported_seq":3228,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.049982+0000","last_change":"2023-10-30T14:23:30.775419+0000","last_active":"2023-10-30T14:51:25.049982+0000","last_peered":"2023-10-30T14:51:25.049982+0000","last_clean":"2023-10-30T14:51:25.049982+0000","last_became_active":"2023-10-30T14:14:04.633586+0000","last_became_peered":"2023-10-30T14:14:04.633586+0000","last_unstale":"2023-10-30T14:51:25.049982+0000","last_undegraded":"2023-10-30T14:51:25.049982+0000","last_fullsized":"2023-10-30T14:51:25.049982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1264,"log_dups_size":0,"ondisk_log_size":1264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:19:06.449779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043616920000000003,"stat_sum":{"num_bytes":12325132,"num_objects":460,"num_object_clones":214,"num_object_copies":920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":192,"num_read":63,"num_read_kb":8213,"num_write":1055,"num_write_kb":14000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"67'1368","reported_seq":3340,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.046013+0000","last_change":"2023-10-30T14:23:30.777885+0000","last_active":"2023-10-30T14:51:25.046013+0000","last_peered":"2023-10-30T14:51:25.046013+0000","last_clean":"2023-10-30T14:51:25.046013+0000","last_became_active":"2023-10-30T14:14:04.642244+0000","last_became_peered":"2023-10-30T14:14:04.642244+0000","last_unstale":"2023-10-30T14:51:25.046013+0000","last_undegraded":"2023-10-30T14:51:25.046013+0000","last_fullsized":"2023-10-30T14:51:25.046013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1368,"log_dups_size":0,"ondisk_log_size":1368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:25.186063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065088710000000003,"stat_sum":{"num_bytes":25821219,"num_objects":511,"num_object_clones":233,"num_object_copies":1022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":210,"num_read":90,"num_read_kb":18405,"num_write":1129,"num_write_kb":31180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'1297","reported_seq":3278,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.499778+0000","last_change":"2023-10-30T14:23:31.070902+0000","last_active":"2023-10-30T14:51:28.499778+0000","last_peered":"2023-10-30T14:51:28.499778+0000","last_clean":"2023-10-30T14:51:28.499778+0000","last_became_active":"2023-10-30T14:14:04.620343+0000","last_became_peered":"2023-10-30T14:14:04.620343+0000","last_unstale":"2023-10-30T14:51:28.499778+0000","last_undegraded":"2023-10-30T14:51:28.499778+0000","last_fullsized":"2023-10-30T14:51:28.499778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1297,"log_dups_size":0,"ondisk_log_size":1297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:45:58.152823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048974149999999996,"stat_sum":{"num_bytes":12397335,"num_objects":479,"num_object_clones":220,"num_object_copies":958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":200,"num_read":86,"num_read_kb":8232,"num_write":1076,"num_write_kb":17925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'1353","reported_seq":3298,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.461566+0000","last_change":"2023-10-30T14:23:30.774642+0000","last_active":"2023-10-30T14:51:29.461566+0000","last_peered":"2023-10-30T14:51:29.461566+0000","last_clean":"2023-10-30T14:51:29.461566+0000","last_became_active":"2023-10-30T14:14:04.609720+0000","last_became_peered":"2023-10-30T14:14:04.609720+0000","last_unstale":"2023-10-30T14:51:29.461566+0000","last_undegraded":"2023-10-30T14:51:29.461566+0000","last_fullsized":"2023-10-30T14:51:29.461566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1353,"log_dups_size":0,"ondisk_log_size":1353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:10:58.430266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040008739999999997,"stat_sum":{"num_bytes":31017079,"num_objects":532,"num_object_clones":246,"num_object_copies":1064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":532,"num_whiteouts":228,"num_read":75,"num_read_kb":20512,"num_write":1096,"num_write_kb":31998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"67'1204","reported_seq":3182,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.455721+0000","last_change":"2023-10-30T14:23:31.069745+0000","last_active":"2023-10-30T14:51:29.455721+0000","last_peered":"2023-10-30T14:51:29.455721+0000","last_clean":"2023-10-30T14:51:29.455721+0000","last_became_active":"2023-10-30T14:14:04.620217+0000","last_became_peered":"2023-10-30T14:14:04.620217+0000","last_unstale":"2023-10-30T14:51:29.455721+0000","last_undegraded":"2023-10-30T14:51:29.455721+0000","last_fullsized":"2023-10-30T14:51:29.455721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:36:29.759579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044491130000000002,"stat_sum":{"num_bytes":20607496,"num_objects":451,"num_object_clones":209,"num_object_copies":902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":451,"num_whiteouts":184,"num_read":71,"num_read_kb":15956,"num_write":987,"num_write_kb":22411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"67'1350","reported_seq":3335,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.475769+0000","last_change":"2023-10-30T14:23:31.075832+0000","last_active":"2023-10-30T14:51:29.475769+0000","last_peered":"2023-10-30T14:51:29.475769+0000","last_clean":"2023-10-30T14:51:29.475769+0000","last_became_active":"2023-10-30T14:14:45.669878+0000","last_became_peered":"2023-10-30T14:14:45.669878+0000","last_unstale":"2023-10-30T14:51:29.475769+0000","last_undegraded":"2023-10-30T14:51:29.475769+0000","last_fullsized":"2023-10-30T14:51:29.475769+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1350,"log_dups_size":0,"ondisk_log_size":1350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:53:43.313641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0077997409999999998,"stat_sum":{"num_bytes":25371605,"num_objects":475,"num_object_clones":218,"num_object_copies":950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":193,"num_read":80,"num_read_kb":18178,"num_write":1139,"num_write_kb":27403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"67'1292","reported_seq":3264,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.057724+0000","last_change":"2023-10-30T14:23:30.773367+0000","last_active":"2023-10-30T14:51:25.057724+0000","last_peered":"2023-10-30T14:51:25.057724+0000","last_clean":"2023-10-30T14:51:25.057724+0000","last_became_active":"2023-10-30T14:14:04.597590+0000","last_became_peered":"2023-10-30T14:14:04.597590+0000","last_unstale":"2023-10-30T14:51:25.057724+0000","last_undegraded":"2023-10-30T14:51:25.057724+0000","last_fullsized":"2023-10-30T14:51:25.057724+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1292,"log_dups_size":0,"ondisk_log_size":1292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:28:58.351077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044830970000000001,"stat_sum":{"num_bytes":8698815,"num_objects":476,"num_object_clones":219,"num_object_copies":952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":476,"num_whiteouts":192,"num_read":75,"num_read_kb":4156,"num_write":1089,"num_write_kb":11768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"67'1181","reported_seq":3142,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.452118+0000","last_change":"2023-10-30T14:23:31.070930+0000","last_active":"2023-10-30T14:51:29.452118+0000","last_peered":"2023-10-30T14:51:29.452118+0000","last_clean":"2023-10-30T14:51:29.452118+0000","last_became_active":"2023-10-30T14:14:04.623902+0000","last_became_peered":"2023-10-30T14:14:04.623902+0000","last_unstale":"2023-10-30T14:51:29.452118+0000","last_undegraded":"2023-10-30T14:51:29.452118+0000","last_fullsized":"2023-10-30T14:51:29.452118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1181,"log_dups_size":0,"ondisk_log_size":1181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:13:07.040210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051343489999999999,"stat_sum":{"num_bytes":21106575,"num_objects":457,"num_object_clones":213,"num_object_copies":914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":457,"num_whiteouts":188,"num_read":58,"num_read_kb":12498,"num_write":966,"num_write_kb":25422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"67'1325","reported_seq":3275,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.632286+0000","last_change":"2023-10-30T14:23:30.775060+0000","last_active":"2023-10-30T14:51:29.632286+0000","last_peered":"2023-10-30T14:51:29.632286+0000","last_clean":"2023-10-30T14:51:29.632286+0000","last_became_active":"2023-10-30T14:14:04.597556+0000","last_became_peered":"2023-10-30T14:14:04.597556+0000","last_unstale":"2023-10-30T14:51:29.632286+0000","last_undegraded":"2023-10-30T14:51:29.632286+0000","last_fullsized":"2023-10-30T14:51:29.632286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1325,"log_dups_size":0,"ondisk_log_size":1325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:42.236111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040069729999999996,"stat_sum":{"num_bytes":25662090,"num_objects":516,"num_object_clones":239,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":516,"num_whiteouts":210,"num_read":73,"num_read_kb":17618,"num_write":1095,"num_write_kb":27367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"67'1302","reported_seq":3272,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.461498+0000","last_change":"2023-10-30T14:23:30.779406+0000","last_active":"2023-10-30T14:51:29.461498+0000","last_peered":"2023-10-30T14:51:29.461498+0000","last_clean":"2023-10-30T14:51:29.461498+0000","last_became_active":"2023-10-30T14:14:04.618430+0000","last_became_peered":"2023-10-30T14:14:04.618430+0000","last_unstale":"2023-10-30T14:51:29.461498+0000","last_undegraded":"2023-10-30T14:51:29.461498+0000","last_fullsized":"2023-10-30T14:51:29.461498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1302,"log_dups_size":0,"ondisk_log_size":1302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:44:00.464702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074984149999999996,"stat_sum":{"num_bytes":16389310,"num_objects":474,"num_object_clones":221,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":474,"num_whiteouts":197,"num_read":75,"num_read_kb":12323,"num_write":1094,"num_write_kb":17656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"67'1414","reported_seq":3372,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.458611+0000","last_change":"2023-10-30T14:23:31.067667+0000","last_active":"2023-10-30T14:51:29.458611+0000","last_peered":"2023-10-30T14:51:29.458611+0000","last_clean":"2023-10-30T14:51:29.458611+0000","last_became_active":"2023-10-30T14:14:04.606340+0000","last_became_peered":"2023-10-30T14:14:04.606340+0000","last_unstale":"2023-10-30T14:51:29.458611+0000","last_undegraded":"2023-10-30T14:51:29.458611+0000","last_fullsized":"2023-10-30T14:51:29.458611+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1414,"log_dups_size":0,"ondisk_log_size":1414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:41:14.948036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035094890000000002,"stat_sum":{"num_bytes":22345046,"num_objects":524,"num_object_clones":240,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":211,"num_read":82,"num_read_kb":9614,"num_write":1184,"num_write_kb":32857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"67'1333","reported_seq":3309,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.486748+0000","last_change":"2023-10-30T14:23:31.074353+0000","last_active":"2023-10-30T14:51:28.486748+0000","last_peered":"2023-10-30T14:51:28.486748+0000","last_clean":"2023-10-30T14:51:28.486748+0000","last_became_active":"2023-10-30T14:14:04.612588+0000","last_became_peered":"2023-10-30T14:14:04.612588+0000","last_unstale":"2023-10-30T14:51:28.486748+0000","last_undegraded":"2023-10-30T14:51:28.486748+0000","last_fullsized":"2023-10-30T14:51:28.486748+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1333,"log_dups_size":0,"ondisk_log_size":1333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:56:59.526944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084570800000000005,"stat_sum":{"num_bytes":13192668,"num_objects":479,"num_object_clones":223,"num_object_copies":958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":199,"num_read":83,"num_read_kb":8248,"num_write":1116,"num_write_kb":15001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"67'1277","reported_seq":3244,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.457130+0000","last_change":"2023-10-30T14:23:30.955845+0000","last_active":"2023-10-30T14:51:29.457130+0000","last_peered":"2023-10-30T14:51:29.457130+0000","last_clean":"2023-10-30T14:51:29.457130+0000","last_became_active":"2023-10-30T14:14:04.607683+0000","last_became_peered":"2023-10-30T14:14:04.607683+0000","last_unstale":"2023-10-30T14:51:29.457130+0000","last_undegraded":"2023-10-30T14:51:29.457130+0000","last_fullsized":"2023-10-30T14:51:29.457130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1277,"log_dups_size":0,"ondisk_log_size":1277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:41.822862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001856688,"stat_sum":{"num_bytes":21370110,"num_objects":478,"num_object_clones":220,"num_object_copies":956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":200,"num_read":74,"num_read_kb":12305,"num_write":1050,"num_write_kb":24643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'640","reported_seq":808,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.464421+0000","last_change":"2023-10-30T14:14:03.732084+0000","last_active":"2023-10-30T14:51:29.464421+0000","last_peered":"2023-10-30T14:51:29.464421+0000","last_clean":"2023-10-30T14:51:29.464421+0000","last_became_active":"2023-10-30T14:14:03.731740+0000","last_became_peered":"2023-10-30T14:14:03.731740+0000","last_unstale":"2023-10-30T14:51:29.464421+0000","last_undegraded":"2023-10-30T14:51:29.464421+0000","last_fullsized":"2023-10-30T14:51:29.464421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:34:27.462477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137426,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":149,"num_read_kb":143937,"num_write":662,"num_write_kb":70557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"67'1312","reported_seq":3279,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.631540+0000","last_change":"2023-10-30T14:23:30.776069+0000","last_active":"2023-10-30T14:51:29.631540+0000","last_peered":"2023-10-30T14:51:29.631540+0000","last_clean":"2023-10-30T14:51:29.631540+0000","last_became_active":"2023-10-30T14:14:04.621526+0000","last_became_peered":"2023-10-30T14:14:04.621526+0000","last_unstale":"2023-10-30T14:51:29.631540+0000","last_undegraded":"2023-10-30T14:51:29.631540+0000","last_fullsized":"2023-10-30T14:51:29.631540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1312,"log_dups_size":0,"ondisk_log_size":1312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:59:59.926386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047814620000000002,"stat_sum":{"num_bytes":12410003,"num_objects":492,"num_object_clones":225,"num_object_copies":984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":492,"num_whiteouts":188,"num_read":79,"num_read_kb":8245,"num_write":1121,"num_write_kb":15268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"67'1480","reported_seq":1960,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.231740+0000","last_change":"2023-10-30T14:14:03.730789+0000","last_active":"2023-10-30T14:51:30.231740+0000","last_peered":"2023-10-30T14:51:30.231740+0000","last_clean":"2023-10-30T14:51:30.231740+0000","last_became_active":"2023-10-30T14:14:03.729386+0000","last_became_peered":"2023-10-30T14:14:03.729386+0000","last_unstale":"2023-10-30T14:51:30.231740+0000","last_undegraded":"2023-10-30T14:51:30.231740+0000","last_fullsized":"2023-10-30T14:51:30.231740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1480,"log_dups_size":0,"ondisk_log_size":1480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29761637,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":457,"num_read_kb":164028,"num_write":1504,"num_write_kb":71675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'761","reported_seq":1325,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.022393+0000","last_change":"2023-10-30T14:14:03.724379+0000","last_active":"2023-10-30T14:51:25.022393+0000","last_peered":"2023-10-30T14:51:25.022393+0000","last_clean":"2023-10-30T14:51:25.022393+0000","last_became_active":"2023-10-30T14:14:03.724078+0000","last_became_peered":"2023-10-30T14:14:03.724078+0000","last_unstale":"2023-10-30T14:51:25.022393+0000","last_undegraded":"2023-10-30T14:51:25.022393+0000","last_fullsized":"2023-10-30T14:51:25.022393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":761,"log_dups_size":0,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:10:17.829899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30316903,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":547,"num_read_kb":63175,"num_write":793,"num_write_kb":47477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"67'1430","reported_seq":3365,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.433623+0000","last_change":"2023-10-30T14:23:31.068462+0000","last_active":"2023-10-30T14:51:29.433623+0000","last_peered":"2023-10-30T14:51:29.433623+0000","last_clean":"2023-10-30T14:51:29.433623+0000","last_became_active":"2023-10-30T14:14:04.597099+0000","last_became_peered":"2023-10-30T14:14:04.597099+0000","last_unstale":"2023-10-30T14:51:29.433623+0000","last_undegraded":"2023-10-30T14:51:29.433623+0000","last_fullsized":"2023-10-30T14:51:29.433623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1430,"log_dups_size":0,"ondisk_log_size":1430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:04:00.818307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033627510000000002,"stat_sum":{"num_bytes":14915943,"num_objects":541,"num_object_clones":248,"num_object_copies":1082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":541,"num_whiteouts":226,"num_read":69,"num_read_kb":8270,"num_write":1175,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"67'1579","reported_seq":1742,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.595991+0000","last_change":"2023-10-30T14:14:03.721030+0000","last_active":"2023-10-30T14:51:25.595991+0000","last_peered":"2023-10-30T14:51:25.595991+0000","last_clean":"2023-10-30T14:51:25.595991+0000","last_became_active":"2023-10-30T14:14:03.719816+0000","last_became_peered":"2023-10-30T14:14:03.719816+0000","last_unstale":"2023-10-30T14:51:25.595991+0000","last_undegraded":"2023-10-30T14:51:25.595991+0000","last_fullsized":"2023-10-30T14:51:25.595991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1579,"log_dups_size":0,"ondisk_log_size":1579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:43:50.150156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35358900,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":164,"num_read_kb":135701,"num_write":1591,"num_write_kb":77227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"67'1277","reported_seq":3242,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.437783+0000","last_change":"2023-10-30T14:23:30.771109+0000","last_active":"2023-10-30T14:51:29.437783+0000","last_peered":"2023-10-30T14:51:29.437783+0000","last_clean":"2023-10-30T14:51:29.437783+0000","last_became_active":"2023-10-30T14:14:04.599323+0000","last_became_peered":"2023-10-30T14:14:04.599323+0000","last_unstale":"2023-10-30T14:51:29.437783+0000","last_undegraded":"2023-10-30T14:51:29.437783+0000","last_fullsized":"2023-10-30T14:51:29.437783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1277,"log_dups_size":0,"ondisk_log_size":1277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:27.656629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025427000000000002,"stat_sum":{"num_bytes":25763183,"num_objects":478,"num_object_clones":224,"num_object_copies":956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":194,"num_read":75,"num_read_kb":21385,"num_write":1050,"num_write_kb":37074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"67'640","reported_seq":767,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.998751+0000","last_change":"2023-10-30T14:14:03.738694+0000","last_active":"2023-10-30T14:51:30.998751+0000","last_peered":"2023-10-30T14:51:30.998751+0000","last_clean":"2023-10-30T14:51:30.998751+0000","last_became_active":"2023-10-30T14:14:03.738132+0000","last_became_peered":"2023-10-30T14:14:03.738132+0000","last_unstale":"2023-10-30T14:51:30.998751+0000","last_undegraded":"2023-10-30T14:51:30.998751+0000","last_fullsized":"2023-10-30T14:51:30.998751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:46:59.692900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360218,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":99,"num_read_kb":53823,"num_write":691,"num_write_kb":33991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"67'1295","reported_seq":3273,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.505595+0000","last_change":"2023-10-30T14:23:30.778832+0000","last_active":"2023-10-30T14:51:28.505595+0000","last_peered":"2023-10-30T14:51:28.505595+0000","last_clean":"2023-10-30T14:51:28.505595+0000","last_became_active":"2023-10-30T14:14:04.648816+0000","last_became_peered":"2023-10-30T14:14:04.648816+0000","last_unstale":"2023-10-30T14:51:28.505595+0000","last_undegraded":"2023-10-30T14:51:28.505595+0000","last_fullsized":"2023-10-30T14:51:28.505595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:20.078590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069848840000000002,"stat_sum":{"num_bytes":12063580,"num_objects":456,"num_object_clones":213,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":456,"num_whiteouts":185,"num_read":77,"num_read_kb":8211,"num_write":1082,"num_write_kb":14557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"67'1049","reported_seq":1186,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469669+0000","last_change":"2023-10-30T14:14:03.726112+0000","last_active":"2023-10-30T14:51:29.469669+0000","last_peered":"2023-10-30T14:51:29.469669+0000","last_clean":"2023-10-30T14:51:29.469669+0000","last_became_active":"2023-10-30T14:14:03.725272+0000","last_became_peered":"2023-10-30T14:14:03.725272+0000","last_unstale":"2023-10-30T14:51:29.469669+0000","last_undegraded":"2023-10-30T14:51:29.469669+0000","last_fullsized":"2023-10-30T14:51:29.469669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1049,"log_dups_size":0,"ondisk_log_size":1049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:37:20.764294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":123,"num_read_kb":86587,"num_write":1102,"num_write_kb":54778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"67'1384","reported_seq":3308,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.501832+0000","last_change":"2023-10-30T14:23:30.776486+0000","last_active":"2023-10-30T14:51:28.501832+0000","last_peered":"2023-10-30T14:51:28.501832+0000","last_clean":"2023-10-30T14:51:28.501832+0000","last_became_active":"2023-10-30T14:14:04.609697+0000","last_became_peered":"2023-10-30T14:14:04.609697+0000","last_unstale":"2023-10-30T14:51:28.501832+0000","last_undegraded":"2023-10-30T14:51:28.501832+0000","last_fullsized":"2023-10-30T14:51:28.501832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1384,"log_dups_size":0,"ondisk_log_size":1384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:45:56.464708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043395700000000001,"stat_sum":{"num_bytes":12432092,"num_objects":547,"num_object_clones":258,"num_object_copies":1094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":547,"num_whiteouts":233,"num_read":68,"num_read_kb":4261,"num_write":1136,"num_write_kb":13809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"67'1004","reported_seq":1165,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.048045+0000","last_change":"2023-10-30T14:14:03.727871+0000","last_active":"2023-10-30T14:51:25.048045+0000","last_peered":"2023-10-30T14:51:25.048045+0000","last_clean":"2023-10-30T14:51:25.048045+0000","last_became_active":"2023-10-30T14:14:03.727422+0000","last_became_peered":"2023-10-30T14:14:03.727422+0000","last_unstale":"2023-10-30T14:51:25.048045+0000","last_undegraded":"2023-10-30T14:51:25.048045+0000","last_fullsized":"2023-10-30T14:51:25.048045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1004,"log_dups_size":0,"ondisk_log_size":1004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:54.782533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389280,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":161,"num_read_kb":61498,"num_write":1080,"num_write_kb":30288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1480","reported_seq":3358,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.465294+0000","last_change":"2023-10-30T14:23:30.957793+0000","last_active":"2023-10-30T14:51:29.465294+0000","last_peered":"2023-10-30T14:51:29.465294+0000","last_clean":"2023-10-30T14:51:29.465294+0000","last_became_active":"2023-10-30T14:14:04.645333+0000","last_became_peered":"2023-10-30T14:14:04.645333+0000","last_unstale":"2023-10-30T14:51:29.465294+0000","last_undegraded":"2023-10-30T14:51:29.465294+0000","last_fullsized":"2023-10-30T14:51:29.465294+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1480,"log_dups_size":0,"ondisk_log_size":1480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:13:02.147712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002301474,"stat_sum":{"num_bytes":21687884,"num_objects":594,"num_object_clones":315,"num_object_copies":1188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":594,"num_whiteouts":221,"num_read":78,"num_read_kb":5632,"num_write":1148,"num_write_kb":28133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"67'579","reported_seq":766,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:27.713419+0000","last_change":"2023-10-30T14:14:03.725554+0000","last_active":"2023-10-30T14:51:27.713419+0000","last_peered":"2023-10-30T14:51:27.713419+0000","last_clean":"2023-10-30T14:51:27.713419+0000","last_became_active":"2023-10-30T14:14:03.724806+0000","last_became_peered":"2023-10-30T14:14:03.724806+0000","last_unstale":"2023-10-30T14:51:27.713419+0000","last_undegraded":"2023-10-30T14:51:27.713419+0000","last_fullsized":"2023-10-30T14:51:27.713419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:37.823561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360800,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":192,"num_read_kb":131840,"num_write":657,"num_write_kb":54408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"67'1313","reported_seq":3260,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.464551+0000","last_change":"2023-10-30T14:23:31.073841+0000","last_active":"2023-10-30T14:51:29.464551+0000","last_peered":"2023-10-30T14:51:29.464551+0000","last_clean":"2023-10-30T14:51:29.464551+0000","last_became_active":"2023-10-30T14:14:04.646543+0000","last_became_peered":"2023-10-30T14:14:04.646543+0000","last_unstale":"2023-10-30T14:51:29.464551+0000","last_undegraded":"2023-10-30T14:51:29.464551+0000","last_fullsized":"2023-10-30T14:51:29.464551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1313,"log_dups_size":0,"ondisk_log_size":1313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:30:29.655398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00744474,"stat_sum":{"num_bytes":21739186,"num_objects":502,"num_object_clones":232,"num_object_copies":1004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":502,"num_whiteouts":205,"num_read":65,"num_read_kb":16406,"num_write":1092,"num_write_kb":23651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"67'683","reported_seq":858,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.629444+0000","last_change":"2023-10-30T14:14:03.717903+0000","last_active":"2023-10-30T14:51:29.629444+0000","last_peered":"2023-10-30T14:51:29.629444+0000","last_clean":"2023-10-30T14:51:29.629444+0000","last_became_active":"2023-10-30T14:14:03.716941+0000","last_became_peered":"2023-10-30T14:14:03.716941+0000","last_unstale":"2023-10-30T14:51:29.629444+0000","last_undegraded":"2023-10-30T14:51:29.629444+0000","last_fullsized":"2023-10-30T14:51:29.629444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":683,"log_dups_size":0,"ondisk_log_size":683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:58.178166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777250,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":201,"num_read_kb":82705,"num_write":761,"num_write_kb":38720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"67'1396","reported_seq":3336,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:26.996443+0000","last_change":"2023-10-30T14:23:30.775738+0000","last_active":"2023-10-30T14:51:26.996443+0000","last_peered":"2023-10-30T14:51:26.996443+0000","last_clean":"2023-10-30T14:51:26.996443+0000","last_became_active":"2023-10-30T14:14:04.641494+0000","last_became_peered":"2023-10-30T14:14:04.641494+0000","last_unstale":"2023-10-30T14:51:26.996443+0000","last_undegraded":"2023-10-30T14:51:26.996443+0000","last_fullsized":"2023-10-30T14:51:26.996443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1396,"log_dups_size":0,"ondisk_log_size":1396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:44.138968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039664019999999999,"stat_sum":{"num_bytes":26403621,"num_objects":532,"num_object_clones":244,"num_object_copies":1064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":532,"num_whiteouts":206,"num_read":69,"num_read_kb":14198,"num_write":1171,"num_write_kb":36700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"67'300","reported_seq":421,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.447336+0000","last_change":"2023-10-30T14:14:03.729626+0000","last_active":"2023-10-30T14:51:29.447336+0000","last_peered":"2023-10-30T14:51:29.447336+0000","last_clean":"2023-10-30T14:51:29.447336+0000","last_became_active":"2023-10-30T14:14:03.728053+0000","last_became_peered":"2023-10-30T14:14:03.728053+0000","last_unstale":"2023-10-30T14:51:29.447336+0000","last_undegraded":"2023-10-30T14:51:29.447336+0000","last_fullsized":"2023-10-30T14:51:29.447336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:44:53.362066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":90,"num_read_kb":49506,"num_write":340,"num_write_kb":25387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1391","reported_seq":3353,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:27.468452+0000","last_change":"2023-10-30T14:23:31.078295+0000","last_active":"2023-10-30T14:51:27.468452+0000","last_peered":"2023-10-30T14:51:27.468452+0000","last_clean":"2023-10-30T14:51:27.468452+0000","last_became_active":"2023-10-30T14:14:04.649896+0000","last_became_peered":"2023-10-30T14:14:04.649896+0000","last_unstale":"2023-10-30T14:51:27.468452+0000","last_undegraded":"2023-10-30T14:51:27.468452+0000","last_fullsized":"2023-10-30T14:51:27.468452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1391,"log_dups_size":0,"ondisk_log_size":1391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:07:44.011495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010792094,"stat_sum":{"num_bytes":9382763,"num_objects":511,"num_object_clones":240,"num_object_copies":1022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":210,"num_read":88,"num_read_kb":2154,"num_write":1160,"num_write_kb":10817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"67'712","reported_seq":848,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.468850+0000","last_change":"2023-10-30T14:14:03.736824+0000","last_active":"2023-10-30T14:51:29.468850+0000","last_peered":"2023-10-30T14:51:29.468850+0000","last_clean":"2023-10-30T14:51:29.468850+0000","last_became_active":"2023-10-30T14:14:03.736155+0000","last_became_peered":"2023-10-30T14:14:03.736155+0000","last_unstale":"2023-10-30T14:51:29.468850+0000","last_undegraded":"2023-10-30T14:51:29.468850+0000","last_fullsized":"2023-10-30T14:51:29.468850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:35.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":104,"num_read_kb":67382,"num_write":738,"num_write_kb":42054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"67'1257","reported_seq":3230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.475777+0000","last_change":"2023-10-30T14:23:31.076487+0000","last_active":"2023-10-30T14:51:28.475777+0000","last_peered":"2023-10-30T14:51:28.475777+0000","last_clean":"2023-10-30T14:51:28.475777+0000","last_became_active":"2023-10-30T14:14:04.614328+0000","last_became_peered":"2023-10-30T14:14:04.614328+0000","last_unstale":"2023-10-30T14:51:28.475777+0000","last_undegraded":"2023-10-30T14:51:28.475777+0000","last_fullsized":"2023-10-30T14:51:28.475777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1257,"log_dups_size":0,"ondisk_log_size":1257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:04:54.384597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010029286,"stat_sum":{"num_bytes":9023665,"num_objects":474,"num_object_clones":219,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":474,"num_whiteouts":189,"num_read":77,"num_read_kb":4169,"num_write":1052,"num_write_kb":11363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"67'1214","reported_seq":1398,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.435186+0000","last_change":"2023-10-30T14:14:03.700690+0000","last_active":"2023-10-30T14:51:29.435186+0000","last_peered":"2023-10-30T14:51:29.435186+0000","last_clean":"2023-10-30T14:51:29.435186+0000","last_became_active":"2023-10-30T14:14:03.700352+0000","last_became_peered":"2023-10-30T14:14:03.700352+0000","last_unstale":"2023-10-30T14:51:29.435186+0000","last_undegraded":"2023-10-30T14:51:29.435186+0000","last_fullsized":"2023-10-30T14:51:29.435186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1214,"log_dups_size":0,"ondisk_log_size":1214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:03:01.103359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":252,"num_read_kb":55463,"num_write":1288,"num_write_kb":26279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"67'1354","reported_seq":3315,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.447744+0000","last_change":"2023-10-30T14:23:31.079218+0000","last_active":"2023-10-30T14:51:28.447744+0000","last_peered":"2023-10-30T14:51:28.447744+0000","last_clean":"2023-10-30T14:51:28.447744+0000","last_became_active":"2023-10-30T14:14:04.646639+0000","last_became_peered":"2023-10-30T14:14:04.646639+0000","last_unstale":"2023-10-30T14:51:28.447744+0000","last_undegraded":"2023-10-30T14:51:28.447744+0000","last_fullsized":"2023-10-30T14:51:28.447744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1354,"log_dups_size":0,"ondisk_log_size":1354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:32.477580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010379724,"stat_sum":{"num_bytes":13446392,"num_objects":502,"num_object_clones":231,"num_object_copies":1004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":502,"num_whiteouts":206,"num_read":77,"num_read_kb":12340,"num_write":1116,"num_write_kb":24186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'643","reported_seq":768,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.271430+0000","last_change":"2023-10-30T14:14:03.710717+0000","last_active":"2023-10-30T14:51:25.271430+0000","last_peered":"2023-10-30T14:51:25.271430+0000","last_clean":"2023-10-30T14:51:25.271430+0000","last_became_active":"2023-10-30T14:14:03.707950+0000","last_became_peered":"2023-10-30T14:14:03.707950+0000","last_unstale":"2023-10-30T14:51:25.271430+0000","last_undegraded":"2023-10-30T14:51:25.271430+0000","last_fullsized":"2023-10-30T14:51:25.271430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:29:41.232902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971542,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":82,"num_read_kb":57693,"num_write":882,"num_write_kb":42351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"67'1239","reported_seq":3190,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.012771+0000","last_change":"2023-10-30T14:23:31.080966+0000","last_active":"2023-10-30T14:51:25.012771+0000","last_peered":"2023-10-30T14:51:25.012771+0000","last_clean":"2023-10-30T14:51:25.012771+0000","last_became_active":"2023-10-30T14:14:04.644761+0000","last_became_peered":"2023-10-30T14:14:04.644761+0000","last_unstale":"2023-10-30T14:51:25.012771+0000","last_undegraded":"2023-10-30T14:51:25.012771+0000","last_fullsized":"2023-10-30T14:51:25.012771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1239,"log_dups_size":0,"ondisk_log_size":1239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:55:03.102431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011171970999999999,"stat_sum":{"num_bytes":12914479,"num_objects":495,"num_object_clones":226,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":208,"num_read":62,"num_read_kb":4133,"num_write":998,"num_write_kb":18863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'535","reported_seq":666,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.248852+0000","last_change":"2023-10-30T14:14:03.705567+0000","last_active":"2023-10-30T14:51:30.248852+0000","last_peered":"2023-10-30T14:51:30.248852+0000","last_clean":"2023-10-30T14:51:30.248852+0000","last_became_active":"2023-10-30T14:14:03.705186+0000","last_became_peered":"2023-10-30T14:14:03.705186+0000","last_unstale":"2023-10-30T14:51:30.248852+0000","last_undegraded":"2023-10-30T14:51:30.248852+0000","last_fullsized":"2023-10-30T14:51:30.248852+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":535,"log_dups_size":0,"ondisk_log_size":535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:14.609243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":102,"num_read_kb":90592,"num_write":573,"num_write_kb":50573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"67'1299","reported_seq":3247,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.473089+0000","last_change":"2023-10-30T14:23:31.083260+0000","last_active":"2023-10-30T14:51:29.473089+0000","last_peered":"2023-10-30T14:51:29.473089+0000","last_clean":"2023-10-30T14:51:29.473089+0000","last_became_active":"2023-10-30T14:14:04.646785+0000","last_became_peered":"2023-10-30T14:14:04.646785+0000","last_unstale":"2023-10-30T14:51:29.473089+0000","last_undegraded":"2023-10-30T14:51:29.473089+0000","last_fullsized":"2023-10-30T14:51:29.473089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1299,"log_dups_size":0,"ondisk_log_size":1299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:52:21.536104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010111184,"stat_sum":{"num_bytes":17345938,"num_objects":504,"num_object_clones":239,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":504,"num_whiteouts":216,"num_read":72,"num_read_kb":10698,"num_write":1034,"num_write_kb":23411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"67'598","reported_seq":748,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.425928+0000","last_change":"2023-10-30T14:14:03.693349+0000","last_active":"2023-10-30T14:51:29.425928+0000","last_peered":"2023-10-30T14:51:29.425928+0000","last_clean":"2023-10-30T14:51:29.425928+0000","last_became_active":"2023-10-30T14:14:03.692966+0000","last_became_peered":"2023-10-30T14:14:03.692966+0000","last_unstale":"2023-10-30T14:51:29.425928+0000","last_undegraded":"2023-10-30T14:51:29.425928+0000","last_fullsized":"2023-10-30T14:51:29.425928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":598,"log_dups_size":0,"ondisk_log_size":598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:30:28.800214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16780852,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":126,"num_read_kb":110211,"num_write":638,"num_write_kb":50143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'1328","reported_seq":3304,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.461147+0000","last_change":"2023-10-30T14:23:30.778622+0000","last_active":"2023-10-30T14:51:29.461147+0000","last_peered":"2023-10-30T14:51:29.461147+0000","last_clean":"2023-10-30T14:51:29.461147+0000","last_became_active":"2023-10-30T14:14:04.615925+0000","last_became_peered":"2023-10-30T14:14:04.615925+0000","last_unstale":"2023-10-30T14:51:29.461147+0000","last_undegraded":"2023-10-30T14:51:29.461147+0000","last_fullsized":"2023-10-30T14:51:29.461147+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1328,"log_dups_size":0,"ondisk_log_size":1328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:54.760495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068317569999999999,"stat_sum":{"num_bytes":4298895,"num_objects":483,"num_object_clones":223,"num_object_copies":966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":483,"num_whiteouts":204,"num_read":84,"num_read_kb":56,"num_write":1103,"num_write_kb":8864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"67'667","reported_seq":805,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.244959+0000","last_change":"2023-10-30T14:14:03.727659+0000","last_active":"2023-10-30T14:51:25.244959+0000","last_peered":"2023-10-30T14:51:25.244959+0000","last_clean":"2023-10-30T14:51:25.244959+0000","last_became_active":"2023-10-30T14:14:03.727409+0000","last_became_peered":"2023-10-30T14:14:03.727409+0000","last_unstale":"2023-10-30T14:51:25.244959+0000","last_undegraded":"2023-10-30T14:51:25.244959+0000","last_fullsized":"2023-10-30T14:51:25.244959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:24.210635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166406,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":118,"num_read_kb":66012,"num_write":697,"num_write_kb":50302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1282","reported_seq":3257,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.467914+0000","last_change":"2023-10-30T14:23:31.077576+0000","last_active":"2023-10-30T14:51:29.467914+0000","last_peered":"2023-10-30T14:51:29.467914+0000","last_clean":"2023-10-30T14:51:29.467914+0000","last_became_active":"2023-10-30T14:14:45.818032+0000","last_became_peered":"2023-10-30T14:14:45.818032+0000","last_unstale":"2023-10-30T14:51:29.467914+0000","last_undegraded":"2023-10-30T14:51:29.467914+0000","last_fullsized":"2023-10-30T14:51:29.467914+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1282,"log_dups_size":0,"ondisk_log_size":1282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:54:03.288491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010655906,"stat_sum":{"num_bytes":23795405,"num_objects":481,"num_object_clones":221,"num_object_copies":962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":199,"num_read":74,"num_read_kb":13429,"num_write":1070,"num_write_kb":29341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":229,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T14:49:57.449030+0000","last_change":"2023-10-30T14:13:54.251572+0000","last_active":"2023-10-30T14:49:57.449030+0000","last_peered":"2023-10-30T14:49:57.449030+0000","last_clean":"2023-10-30T14:49:57.449030+0000","last_became_active":"2023-10-30T14:13:54.250907+0000","last_became_peered":"2023-10-30T14:13:54.250907+0000","last_unstale":"2023-10-30T14:49:57.449030+0000","last_undegraded":"2023-10-30T14:49:57.449030+0000","last_fullsized":"2023-10-30T14:49:57.449030+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_clean_scrub_stamp":"2023-10-30T14:13:53.220267+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:44.590852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"67'724","reported_seq":867,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.024538+0000","last_change":"2023-10-30T14:14:03.729034+0000","last_active":"2023-10-30T14:51:25.024538+0000","last_peered":"2023-10-30T14:51:25.024538+0000","last_clean":"2023-10-30T14:51:25.024538+0000","last_became_active":"2023-10-30T14:14:03.728293+0000","last_became_peered":"2023-10-30T14:14:03.728293+0000","last_unstale":"2023-10-30T14:51:25.024538+0000","last_undegraded":"2023-10-30T14:51:25.024538+0000","last_fullsized":"2023-10-30T14:51:25.024538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":724,"log_dups_size":0,"ondisk_log_size":724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165954,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":121,"num_read_kb":78313,"num_write":794,"num_write_kb":42213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"67'1244","reported_seq":3209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.448701+0000","last_change":"2023-10-30T14:23:31.067976+0000","last_active":"2023-10-30T14:51:29.448701+0000","last_peered":"2023-10-30T14:51:29.448701+0000","last_clean":"2023-10-30T14:51:29.448701+0000","last_became_active":"2023-10-30T14:14:46.451620+0000","last_became_peered":"2023-10-30T14:14:46.451620+0000","last_unstale":"2023-10-30T14:51:29.448701+0000","last_undegraded":"2023-10-30T14:51:29.448701+0000","last_fullsized":"2023-10-30T14:51:29.448701+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1244,"log_dups_size":0,"ondisk_log_size":1244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:10.312892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034002070000000001,"stat_sum":{"num_bytes":12494676,"num_objects":466,"num_object_clones":213,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":182,"num_read":65,"num_read_kb":8203,"num_write":1045,"num_write_kb":18288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"67'823","reported_seq":1009,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.517181+0000","last_change":"2023-10-30T14:14:03.732909+0000","last_active":"2023-10-30T14:51:29.517181+0000","last_peered":"2023-10-30T14:51:29.517181+0000","last_clean":"2023-10-30T14:51:29.517181+0000","last_became_active":"2023-10-30T14:14:03.732199+0000","last_became_peered":"2023-10-30T14:14:03.732199+0000","last_unstale":"2023-10-30T14:51:29.517181+0000","last_undegraded":"2023-10-30T14:51:29.517181+0000","last_fullsized":"2023-10-30T14:51:29.517181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":823,"log_dups_size":0,"ondisk_log_size":823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:38.117750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":198,"num_read_kb":136195,"num_write":895,"num_write_kb":59080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"67'1255","reported_seq":3217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.505461+0000","last_change":"2023-10-30T14:23:30.775385+0000","last_active":"2023-10-30T14:51:28.505461+0000","last_peered":"2023-10-30T14:51:28.505461+0000","last_clean":"2023-10-30T14:51:28.505461+0000","last_became_active":"2023-10-30T14:14:04.619990+0000","last_became_peered":"2023-10-30T14:14:04.619990+0000","last_unstale":"2023-10-30T14:51:28.505461+0000","last_undegraded":"2023-10-30T14:51:28.505461+0000","last_fullsized":"2023-10-30T14:51:28.505461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1255,"log_dups_size":0,"ondisk_log_size":1255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:42.711461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045963250000000001,"stat_sum":{"num_bytes":8263423,"num_objects":470,"num_object_clones":217,"num_object_copies":940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":194,"num_read":63,"num_read_kb":1576,"num_write":1033,"num_write_kb":10839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"67'1750","reported_seq":2853,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.563109+0000","last_change":"2023-10-30T14:14:03.702862+0000","last_active":"2023-10-30T14:51:29.563109+0000","last_peered":"2023-10-30T14:51:29.563109+0000","last_clean":"2023-10-30T14:51:29.563109+0000","last_became_active":"2023-10-30T14:14:03.702513+0000","last_became_peered":"2023-10-30T14:14:03.702513+0000","last_unstale":"2023-10-30T14:51:29.563109+0000","last_undegraded":"2023-10-30T14:51:29.563109+0000","last_fullsized":"2023-10-30T14:51:29.563109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1750,"log_dups_size":0,"ondisk_log_size":1750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:25.499390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27766724,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":1109,"num_read_kb":113686,"num_write":1839,"num_write_kb":62085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"67'1316","reported_seq":3268,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.468492+0000","last_change":"2023-10-30T14:23:30.770719+0000","last_active":"2023-10-30T14:51:28.468492+0000","last_peered":"2023-10-30T14:51:28.468492+0000","last_clean":"2023-10-30T14:51:28.468492+0000","last_became_active":"2023-10-30T14:14:04.589087+0000","last_became_peered":"2023-10-30T14:14:04.589087+0000","last_unstale":"2023-10-30T14:51:28.468492+0000","last_undegraded":"2023-10-30T14:51:28.468492+0000","last_fullsized":"2023-10-30T14:51:28.468492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1316,"log_dups_size":0,"ondisk_log_size":1316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:44:54.038742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022871390000000001,"stat_sum":{"num_bytes":30600386,"num_objects":508,"num_object_clones":243,"num_object_copies":1016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":508,"num_whiteouts":221,"num_read":78,"num_read_kb":20511,"num_write":1060,"num_write_kb":36437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"67'633","reported_seq":782,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.234880+0000","last_change":"2023-10-30T14:14:03.730048+0000","last_active":"2023-10-30T14:51:30.234880+0000","last_peered":"2023-10-30T14:51:30.234880+0000","last_clean":"2023-10-30T14:51:30.234880+0000","last_became_active":"2023-10-30T14:14:03.728925+0000","last_became_peered":"2023-10-30T14:14:03.728925+0000","last_unstale":"2023-10-30T14:51:30.234880+0000","last_undegraded":"2023-10-30T14:51:30.234880+0000","last_fullsized":"2023-10-30T14:51:30.234880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":633,"log_dups_size":0,"ondisk_log_size":633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:32:05.056039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":120,"num_read_kb":131794,"num_write":657,"num_write_kb":62419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"67'1355","reported_seq":3305,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.495764+0000","last_change":"2023-10-30T14:23:30.956956+0000","last_active":"2023-10-30T14:51:28.495764+0000","last_peered":"2023-10-30T14:51:28.495764+0000","last_clean":"2023-10-30T14:51:28.495764+0000","last_became_active":"2023-10-30T14:14:04.607377+0000","last_became_peered":"2023-10-30T14:14:04.607377+0000","last_unstale":"2023-10-30T14:51:28.495764+0000","last_undegraded":"2023-10-30T14:51:28.495764+0000","last_fullsized":"2023-10-30T14:51:28.495764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1355,"log_dups_size":0,"ondisk_log_size":1355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:58:23.634836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022021179999999999,"stat_sum":{"num_bytes":18058258,"num_objects":500,"num_object_clones":231,"num_object_copies":1000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":500,"num_whiteouts":206,"num_read":65,"num_read_kb":9172,"num_write":1119,"num_write_kb":20820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"67'880","reported_seq":1017,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.230091+0000","last_change":"2023-10-30T14:14:03.693753+0000","last_active":"2023-10-30T14:51:30.230091+0000","last_peered":"2023-10-30T14:51:30.230091+0000","last_clean":"2023-10-30T14:51:30.230091+0000","last_became_active":"2023-10-30T14:14:03.692621+0000","last_became_peered":"2023-10-30T14:14:03.692621+0000","last_unstale":"2023-10-30T14:51:30.230091+0000","last_undegraded":"2023-10-30T14:51:30.230091+0000","last_fullsized":"2023-10-30T14:51:30.230091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:32:28.577584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":107,"num_read_kb":66236,"num_write":920,"num_write_kb":58565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'1338","reported_seq":3267,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.493889+0000","last_change":"2023-10-30T14:23:30.780258+0000","last_active":"2023-10-30T14:51:28.493889+0000","last_peered":"2023-10-30T14:51:28.493889+0000","last_clean":"2023-10-30T14:51:28.493889+0000","last_became_active":"2023-10-30T14:14:04.613780+0000","last_became_peered":"2023-10-30T14:14:04.613780+0000","last_unstale":"2023-10-30T14:51:28.493889+0000","last_undegraded":"2023-10-30T14:51:28.493889+0000","last_fullsized":"2023-10-30T14:51:28.493889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1338,"log_dups_size":0,"ondisk_log_size":1338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:07:05.556883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065285270000000001,"stat_sum":{"num_bytes":12665534,"num_objects":519,"num_object_clones":250,"num_object_copies":1038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":519,"num_whiteouts":228,"num_read":63,"num_read_kb":8216,"num_write":1076,"num_write_kb":17979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"67'721","reported_seq":883,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.249637+0000","last_change":"2023-10-30T14:14:03.740213+0000","last_active":"2023-10-30T14:51:30.249637+0000","last_peered":"2023-10-30T14:51:30.249637+0000","last_clean":"2023-10-30T14:51:30.249637+0000","last_became_active":"2023-10-30T14:14:03.739650+0000","last_became_peered":"2023-10-30T14:14:03.739650+0000","last_unstale":"2023-10-30T14:51:30.249637+0000","last_undegraded":"2023-10-30T14:51:30.249637+0000","last_fullsized":"2023-10-30T14:51:30.249637+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":721,"log_dups_size":0,"ondisk_log_size":721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:54.120757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777238,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":143,"num_read_kb":109192,"num_write":780,"num_write_kb":46441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"67'1318","reported_seq":3274,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.425113+0000","last_change":"2023-10-30T14:23:30.771719+0000","last_active":"2023-10-30T14:51:29.425113+0000","last_peered":"2023-10-30T14:51:29.425113+0000","last_clean":"2023-10-30T14:51:29.425113+0000","last_became_active":"2023-10-30T14:14:04.590138+0000","last_became_peered":"2023-10-30T14:14:04.590138+0000","last_unstale":"2023-10-30T14:51:29.425113+0000","last_undegraded":"2023-10-30T14:51:29.425113+0000","last_fullsized":"2023-10-30T14:51:29.425113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1318,"log_dups_size":0,"ondisk_log_size":1318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:49.779956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031364370000000002,"stat_sum":{"num_bytes":9073572,"num_objects":494,"num_object_clones":232,"num_object_copies":988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":494,"num_whiteouts":206,"num_read":72,"num_read_kb":4145,"num_write":1090,"num_write_kb":19754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"67'771","reported_seq":913,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469946+0000","last_change":"2023-10-30T14:14:03.721450+0000","last_active":"2023-10-30T14:51:29.469946+0000","last_peered":"2023-10-30T14:51:29.469946+0000","last_clean":"2023-10-30T14:51:29.469946+0000","last_became_active":"2023-10-30T14:14:03.721234+0000","last_became_peered":"2023-10-30T14:14:03.721234+0000","last_unstale":"2023-10-30T14:51:29.469946+0000","last_undegraded":"2023-10-30T14:51:29.469946+0000","last_fullsized":"2023-10-30T14:51:29.469946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":771,"log_dups_size":0,"ondisk_log_size":771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:26.959676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":136,"num_read_kb":37589,"num_write":1311,"num_write_kb":40350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"67'1259","reported_seq":3214,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.476582+0000","last_change":"2023-10-30T14:23:31.070447+0000","last_active":"2023-10-30T14:51:28.476582+0000","last_peered":"2023-10-30T14:51:28.476582+0000","last_clean":"2023-10-30T14:51:28.476582+0000","last_became_active":"2023-10-30T14:14:04.633643+0000","last_became_peered":"2023-10-30T14:14:04.633643+0000","last_unstale":"2023-10-30T14:51:28.476582+0000","last_undegraded":"2023-10-30T14:51:28.476582+0000","last_fullsized":"2023-10-30T14:51:28.476582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1259,"log_dups_size":0,"ondisk_log_size":1259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:53:41.463549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048916599999999999,"stat_sum":{"num_bytes":9340734,"num_objects":487,"num_object_clones":228,"num_object_copies":974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":205,"num_read":67,"num_read_kb":4121,"num_write":1045,"num_write_kb":15590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"67'440","reported_seq":577,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.705366+0000","last_change":"2023-10-30T14:14:03.738557+0000","last_active":"2023-10-30T14:51:29.705366+0000","last_peered":"2023-10-30T14:51:29.705366+0000","last_clean":"2023-10-30T14:51:29.705366+0000","last_became_active":"2023-10-30T14:14:03.737801+0000","last_became_peered":"2023-10-30T14:14:03.737801+0000","last_unstale":"2023-10-30T14:51:29.705366+0000","last_undegraded":"2023-10-30T14:51:29.705366+0000","last_fullsized":"2023-10-30T14:51:29.705366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":440,"log_dups_size":0,"ondisk_log_size":440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:26.938343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":132,"num_read_kb":45549,"num_write":530,"num_write_kb":26044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"67'1244","reported_seq":3217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.425279+0000","last_change":"2023-10-30T14:23:30.776079+0000","last_active":"2023-10-30T14:51:28.425279+0000","last_peered":"2023-10-30T14:51:28.425279+0000","last_clean":"2023-10-30T14:51:28.425279+0000","last_became_active":"2023-10-30T14:14:04.588826+0000","last_became_peered":"2023-10-30T14:14:04.588826+0000","last_unstale":"2023-10-30T14:51:28.425279+0000","last_undegraded":"2023-10-30T14:51:28.425279+0000","last_fullsized":"2023-10-30T14:51:28.425279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1244,"log_dups_size":0,"ondisk_log_size":1244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:27.236524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040592670000000001,"stat_sum":{"num_bytes":8246712,"num_objects":450,"num_object_clones":210,"num_object_copies":900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":450,"num_whiteouts":181,"num_read":67,"num_read_kb":4125,"num_write":1046,"num_write_kb":10688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"67'429","reported_seq":542,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.467848+0000","last_change":"2023-10-30T14:14:03.733836+0000","last_active":"2023-10-30T14:51:29.467848+0000","last_peered":"2023-10-30T14:51:29.467848+0000","last_clean":"2023-10-30T14:51:29.467848+0000","last_became_active":"2023-10-30T14:14:03.732912+0000","last_became_peered":"2023-10-30T14:14:03.732912+0000","last_unstale":"2023-10-30T14:51:29.467848+0000","last_undegraded":"2023-10-30T14:51:29.467848+0000","last_fullsized":"2023-10-30T14:51:29.467848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:50.332738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":78,"num_read_kb":41242,"num_write":426,"num_write_kb":21091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"67'1279","reported_seq":3248,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.450800+0000","last_change":"2023-10-30T14:23:30.957473+0000","last_active":"2023-10-30T14:51:29.450800+0000","last_peered":"2023-10-30T14:51:29.450800+0000","last_clean":"2023-10-30T14:51:29.450800+0000","last_became_active":"2023-10-30T14:14:04.609835+0000","last_became_peered":"2023-10-30T14:14:04.609835+0000","last_unstale":"2023-10-30T14:51:29.450800+0000","last_undegraded":"2023-10-30T14:51:29.450800+0000","last_fullsized":"2023-10-30T14:51:29.450800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1279,"log_dups_size":0,"ondisk_log_size":1279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:29:36.272312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023958389999999999,"stat_sum":{"num_bytes":25809405,"num_objects":464,"num_object_clones":214,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":464,"num_whiteouts":191,"num_read":67,"num_read_kb":20495,"num_write":1061,"num_write_kb":27581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"67'867","reported_seq":1008,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.048969+0000","last_change":"2023-10-30T14:14:03.732539+0000","last_active":"2023-10-30T14:51:25.048969+0000","last_peered":"2023-10-30T14:51:25.048969+0000","last_clean":"2023-10-30T14:51:25.048969+0000","last_became_active":"2023-10-30T14:14:03.732265+0000","last_became_peered":"2023-10-30T14:14:03.732265+0000","last_unstale":"2023-10-30T14:51:25.048969+0000","last_undegraded":"2023-10-30T14:51:25.048969+0000","last_fullsized":"2023-10-30T14:51:25.048969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":867,"log_dups_size":0,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:02:00.929612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":128,"num_read_kb":49868,"num_write":925,"num_write_kb":26024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"67'1338","reported_seq":3317,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.437725+0000","last_change":"2023-10-30T14:23:31.069537+0000","last_active":"2023-10-30T14:51:29.437725+0000","last_peered":"2023-10-30T14:51:29.437725+0000","last_clean":"2023-10-30T14:51:29.437725+0000","last_became_active":"2023-10-30T14:14:04.594998+0000","last_became_peered":"2023-10-30T14:14:04.594998+0000","last_unstale":"2023-10-30T14:51:29.437725+0000","last_undegraded":"2023-10-30T14:51:29.437725+0000","last_fullsized":"2023-10-30T14:51:29.437725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1338,"log_dups_size":0,"ondisk_log_size":1338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:29:28.794439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00432118,"stat_sum":{"num_bytes":12291745,"num_objects":459,"num_object_clones":215,"num_object_copies":918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":459,"num_whiteouts":187,"num_read":78,"num_read_kb":8212,"num_write":1121,"num_write_kb":18507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'816","reported_seq":962,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.428311+0000","last_change":"2023-10-30T14:14:03.703744+0000","last_active":"2023-10-30T14:51:29.428311+0000","last_peered":"2023-10-30T14:51:29.428311+0000","last_clean":"2023-10-30T14:51:29.428311+0000","last_became_active":"2023-10-30T14:14:03.701759+0000","last_became_peered":"2023-10-30T14:14:03.701759+0000","last_unstale":"2023-10-30T14:51:29.428311+0000","last_undegraded":"2023-10-30T14:51:29.428311+0000","last_fullsized":"2023-10-30T14:51:29.428311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":816,"log_dups_size":0,"ondisk_log_size":816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:22:27.948007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":122,"num_read_kb":94942,"num_write":1181,"num_write_kb":51316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"67'1224","reported_seq":3205,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.415105+0000","last_change":"2023-10-30T14:23:31.071315+0000","last_active":"2023-10-30T14:51:29.415105+0000","last_peered":"2023-10-30T14:51:29.415105+0000","last_clean":"2023-10-30T14:51:29.415105+0000","last_became_active":"2023-10-30T14:14:04.600100+0000","last_became_peered":"2023-10-30T14:14:04.600100+0000","last_unstale":"2023-10-30T14:51:29.415105+0000","last_undegraded":"2023-10-30T14:51:29.415105+0000","last_fullsized":"2023-10-30T14:51:29.415105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1224,"log_dups_size":0,"ondisk_log_size":1224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:30.476350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052227630000000001,"stat_sum":{"num_bytes":16868959,"num_objects":442,"num_object_clones":207,"num_object_copies":884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":442,"num_whiteouts":180,"num_read":73,"num_read_kb":11653,"num_write":1014,"num_write_kb":18882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"67'773","reported_seq":906,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.997876+0000","last_change":"2023-10-30T14:14:03.741268+0000","last_active":"2023-10-30T14:51:30.997876+0000","last_peered":"2023-10-30T14:51:30.997876+0000","last_clean":"2023-10-30T14:51:30.997876+0000","last_became_active":"2023-10-30T14:14:03.740648+0000","last_became_peered":"2023-10-30T14:14:03.740648+0000","last_unstale":"2023-10-30T14:51:30.997876+0000","last_undegraded":"2023-10-30T14:51:30.997876+0000","last_fullsized":"2023-10-30T14:51:30.997876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:30.799148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":98,"num_read_kb":107365,"num_write":798,"num_write_kb":50458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"67'1285","reported_seq":3266,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469362+0000","last_change":"2023-10-30T14:23:31.073589+0000","last_active":"2023-10-30T14:51:29.469362+0000","last_peered":"2023-10-30T14:51:29.469362+0000","last_clean":"2023-10-30T14:51:29.469362+0000","last_became_active":"2023-10-30T14:14:04.622921+0000","last_became_peered":"2023-10-30T14:14:04.622921+0000","last_unstale":"2023-10-30T14:51:29.469362+0000","last_undegraded":"2023-10-30T14:51:29.469362+0000","last_fullsized":"2023-10-30T14:51:29.469362+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1285,"log_dups_size":0,"ondisk_log_size":1285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:04:46.396711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0071219910000000003,"stat_sum":{"num_bytes":20705836,"num_objects":462,"num_object_clones":210,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":462,"num_whiteouts":184,"num_read":75,"num_read_kb":12323,"num_write":1096,"num_write_kb":26452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"67'1315","reported_seq":3269,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.461591+0000","last_change":"2023-10-30T14:23:30.779989+0000","last_active":"2023-10-30T14:51:29.461591+0000","last_peered":"2023-10-30T14:51:29.461591+0000","last_clean":"2023-10-30T14:51:29.461591+0000","last_became_active":"2023-10-30T14:14:04.621885+0000","last_became_peered":"2023-10-30T14:14:04.621885+0000","last_unstale":"2023-10-30T14:51:29.461591+0000","last_undegraded":"2023-10-30T14:51:29.461591+0000","last_fullsized":"2023-10-30T14:51:29.461591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1315,"log_dups_size":0,"ondisk_log_size":1315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:59:52.839990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067851329999999996,"stat_sum":{"num_bytes":13415563,"num_objects":482,"num_object_clones":227,"num_object_copies":964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":190,"num_read":68,"num_read_kb":8221,"num_write":1118,"num_write_kb":20704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"67'466","reported_seq":615,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:32.456573+0000","last_change":"2023-10-30T14:14:03.740897+0000","last_active":"2023-10-30T14:51:32.456573+0000","last_peered":"2023-10-30T14:51:32.456573+0000","last_clean":"2023-10-30T14:51:32.456573+0000","last_became_active":"2023-10-30T14:14:03.740247+0000","last_became_peered":"2023-10-30T14:14:03.740247+0000","last_unstale":"2023-10-30T14:51:32.456573+0000","last_undegraded":"2023-10-30T14:51:32.456573+0000","last_fullsized":"2023-10-30T14:51:32.456573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:29.601978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30406549,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":149,"num_read_kb":82851,"num_write":524,"num_write_kb":39214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'1397","reported_seq":3340,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.460907+0000","last_change":"2023-10-30T14:23:30.776236+0000","last_active":"2023-10-30T14:51:29.460907+0000","last_peered":"2023-10-30T14:51:29.460907+0000","last_clean":"2023-10-30T14:51:29.460907+0000","last_became_active":"2023-10-30T14:14:04.623638+0000","last_became_peered":"2023-10-30T14:14:04.623638+0000","last_unstale":"2023-10-30T14:51:29.460907+0000","last_undegraded":"2023-10-30T14:51:29.460907+0000","last_fullsized":"2023-10-30T14:51:29.460907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1397,"log_dups_size":0,"ondisk_log_size":1397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:37:06.313676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049828509999999999,"stat_sum":{"num_bytes":25425754,"num_objects":536,"num_object_clones":247,"num_object_copies":1072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":221,"num_read":77,"num_read_kb":19614,"num_write":1171,"num_write_kb":31740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"67'1024","reported_seq":1176,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.475951+0000","last_change":"2023-10-30T14:14:03.733233+0000","last_active":"2023-10-30T14:51:29.475951+0000","last_peered":"2023-10-30T14:51:29.475951+0000","last_clean":"2023-10-30T14:51:29.475951+0000","last_became_active":"2023-10-30T14:14:03.732538+0000","last_became_peered":"2023-10-30T14:14:03.732538+0000","last_unstale":"2023-10-30T14:51:29.475951+0000","last_undegraded":"2023-10-30T14:51:29.475951+0000","last_fullsized":"2023-10-30T14:51:29.475951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1024,"log_dups_size":0,"ondisk_log_size":1024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:25.696173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":137,"num_read_kb":150323,"num_write":1064,"num_write_kb":62931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1272","reported_seq":3223,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.627078+0000","last_change":"2023-10-30T14:23:30.776451+0000","last_active":"2023-10-30T14:51:29.627078+0000","last_peered":"2023-10-30T14:51:29.627078+0000","last_clean":"2023-10-30T14:51:29.627078+0000","last_became_active":"2023-10-30T14:14:04.611773+0000","last_became_peered":"2023-10-30T14:14:04.611773+0000","last_unstale":"2023-10-30T14:51:29.627078+0000","last_undegraded":"2023-10-30T14:51:29.627078+0000","last_fullsized":"2023-10-30T14:51:29.627078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1272,"log_dups_size":0,"ondisk_log_size":1272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:55.244396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044924040000000002,"stat_sum":{"num_bytes":26143587,"num_objects":507,"num_object_clones":231,"num_object_copies":1014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":507,"num_whiteouts":205,"num_read":66,"num_read_kb":19602,"num_write":1063,"num_write_kb":27312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"67'731","reported_seq":872,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.995935+0000","last_change":"2023-10-30T14:14:03.703988+0000","last_active":"2023-10-30T14:51:30.995935+0000","last_peered":"2023-10-30T14:51:30.995935+0000","last_clean":"2023-10-30T14:51:30.995935+0000","last_became_active":"2023-10-30T14:14:03.703691+0000","last_became_peered":"2023-10-30T14:14:03.703691+0000","last_unstale":"2023-10-30T14:51:30.995935+0000","last_undegraded":"2023-10-30T14:51:30.995935+0000","last_fullsized":"2023-10-30T14:51:30.995935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":731,"log_dups_size":0,"ondisk_log_size":731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:34.538541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":102,"num_read_kb":111696,"num_write":760,"num_write_kb":58649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"67'1327","reported_seq":3261,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.468906+0000","last_change":"2023-10-30T14:23:31.069185+0000","last_active":"2023-10-30T14:51:29.468906+0000","last_peered":"2023-10-30T14:51:29.468906+0000","last_clean":"2023-10-30T14:51:29.468906+0000","last_became_active":"2023-10-30T14:14:04.613420+0000","last_became_peered":"2023-10-30T14:14:04.613420+0000","last_unstale":"2023-10-30T14:51:29.468906+0000","last_undegraded":"2023-10-30T14:51:29.468906+0000","last_fullsized":"2023-10-30T14:51:29.468906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1327,"log_dups_size":0,"ondisk_log_size":1327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:51:35.517831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004010128,"stat_sum":{"num_bytes":12779880,"num_objects":504,"num_object_clones":238,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":504,"num_whiteouts":203,"num_read":55,"num_read_kb":8221,"num_write":1111,"num_write_kb":14267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"67'636","reported_seq":796,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.650555+0000","last_change":"2023-10-30T14:14:03.737319+0000","last_active":"2023-10-30T14:51:29.650555+0000","last_peered":"2023-10-30T14:51:29.650555+0000","last_clean":"2023-10-30T14:51:29.650555+0000","last_became_active":"2023-10-30T14:14:03.736964+0000","last_became_peered":"2023-10-30T14:14:03.736964+0000","last_unstale":"2023-10-30T14:51:29.650555+0000","last_undegraded":"2023-10-30T14:51:29.650555+0000","last_fullsized":"2023-10-30T14:51:29.650555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:41.469809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":182,"num_read_kb":58134,"num_write":693,"num_write_kb":34123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1046721150,"num_objects":31370,"num_object_clones":14593,"num_object_copies":62740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31370,"num_whiteouts":12799,"num_read":4602,"num_read_kb":662584,"num_write":69411,"num_write_kb":1341877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2168676352,"data_stored":2094671100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83616,"ondisk_log_size":83616,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":673711135,"num_objects":1337,"num_object_clones":0,"num_object_copies":2674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1337,"num_whiteouts":0,"num_read":6029,"num_read_kb":2835128,"num_write":28094,"num_write_kb":1507213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1111,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1347235840,"data_stored":1347090008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25568,"ondisk_log_size":25568,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575305,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614432,"kb_used_data":479192,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":93757408,"statfs":{"total":96636764160,"available":96007585792,"internally_reserved":0,"allocated":490692608,"data_stored":479962954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":138475979},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.077,"15min":0.997},"min":{"1min":0.51700000000000002,"5min":0.46300000000000002,"15min":0.45500000000000002},"max":{"1min":6.3040000000000003,"5min":6.399,"15min":6.399},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.2370000000000001,"5min":0.97799999999999998,"15min":0.94399999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":6.4969999999999999,"5min":6.4969999999999999,"15min":6.4969999999999999},"last":0.83199999999999996}]},{"osd":1,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":0.92500000000000004,"15min":0.93000000000000005},"min":{"1min":0.61599999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":6.1390000000000002,"5min":6.1390000000000002,"15min":6.1390000000000002},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.96599999999999997,"15min":0.99399999999999999},"min":{"1min":0.58699999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":3.8460000000000001,"5min":6.1920000000000002,"15min":6.1920000000000002},"last":0.63700000000000001}]},{"osd":2,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.0449999999999999,"15min":0.96899999999999997},"min":{"1min":0.70099999999999996,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.5739999999999998,"5min":6.5739999999999998,"15min":6.5739999999999998},"last":1.125},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.115,"15min":1.01},"min":{"1min":0.81599999999999995,"5min":0.54500000000000004,"15min":0.5},"max":{"1min":6.2350000000000003,"5min":6.5149999999999997,"15min":6.5149999999999997},"last":1.2450000000000001}]},{"osd":3,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.86199999999999999,"15min":0.85599999999999998},"min":{"1min":0.56200000000000006,"5min":0.54800000000000004,"15min":0.53800000000000003},"max":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":4.3230000000000004},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.121,"15min":1.056},"min":{"1min":0.66800000000000004,"5min":0.64800000000000002,"15min":0.54300000000000004},"max":{"1min":3.157,"5min":6.2670000000000003,"15min":6.2670000000000003},"last":0.91500000000000004}]},{"osd":4,"last update":"Mon Oct 30 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.95599999999999996,"15min":0.95699999999999996},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.54800000000000004},"max":{"1min":4.7409999999999997,"5min":4.7409999999999997,"15min":4.7409999999999997},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.036,"15min":0.97399999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":8.0719999999999992,"5min":8.0719999999999992,"15min":8.0719999999999992},"last":0.873}]},{"osd":5,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3280000000000001,"15min":1.1759999999999999},"min":{"1min":0.78100000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":10.692,"5min":10.692,"15min":10.692},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.3939999999999999,"15min":1.2869999999999999},"min":{"1min":0.72099999999999997,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":6.6429999999999998,"5min":6.6429999999999998,"15min":6.6429999999999998},"last":1.3520000000000001}]},{"osd":6,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.135,"15min":1.129},"min":{"1min":0.748,"5min":0.71599999999999997,"15min":0.62},"max":{"1min":5.2160000000000002,"5min":5.2160000000000002,"15min":5.2160000000000002},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.1639999999999999,"15min":1.107},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":4.367,"5min":4.367,"15min":4.367},"last":1.2010000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575305,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511764,"kb_used_data":386252,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":93860076,"statfs":{"total":96636764160,"available":96112717824,"internally_reserved":0,"allocated":395522048,"data_stored":388333646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":128514503},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.82899999999999996,"15min":0.83199999999999996},"min":{"1min":0.503,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.216,"5min":1.288,"15min":1.2929999999999999},"last":1.5129999999999999},{"interface":"front","average":{"1min":0.997,"5min":0.90500000000000003,"15min":0.88800000000000001},"min":{"1min":0.68600000000000005,"5min":0.435,"15min":0.435},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.391},"last":1.726}]},{"osd":1,"last update":"Mon Oct 30 14:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.89700000000000002,"15min":0.85599999999999998},"min":{"1min":0.66000000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.135,"5min":8,"15min":8},"last":1.468},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.94199999999999995,"15min":0.96599999999999997},"min":{"1min":0.67300000000000004,"5min":0.53400000000000003,"15min":0.318},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.514},"last":1.2969999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.89300000000000002,"15min":0.89000000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.40999999999999998},"max":{"1min":1.232,"5min":1.2410000000000001,"15min":1.617},"last":0.876},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.92900000000000005,"15min":0.94299999999999995},"min":{"1min":0.63400000000000001,"5min":0.59999999999999998,"15min":0.49299999999999999},"max":{"1min":1.181,"5min":1.3149999999999999,"15min":1.4119999999999999},"last":2.9609999999999999}]},{"osd":3,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":0.997,"15min":0.98099999999999998},"min":{"1min":0.73999999999999999,"5min":0.57599999999999996,"15min":0.47799999999999998},"max":{"1min":1.3700000000000001,"5min":1.3720000000000001,"15min":1.6890000000000001},"last":1.6200000000000001},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.95499999999999996,"15min":0.96099999999999997},"min":{"1min":0.61499999999999999,"5min":0.59199999999999997,"15min":0.48899999999999999},"max":{"1min":1.286,"5min":1.286,"15min":1.472},"last":1.206}]},{"osd":4,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.0309999999999999,"15min":1.032},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.59899999999999998},"max":{"1min":1.1830000000000001,"5min":3.8399999999999999,"15min":3.8399999999999999},"last":3.7549999999999999},{"interface":"front","average":{"1min":1.016,"5min":1.0269999999999999,"15min":1.04},"min":{"1min":0.69999999999999996,"5min":0.62,"15min":0.435},"max":{"1min":1.341,"5min":1.4730000000000001,"15min":1.645},"last":3.0790000000000002}]},{"osd":5,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":1.0149999999999999,"15min":0.97299999999999998},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.51100000000000001},"max":{"1min":1.1659999999999999,"5min":11.728,"15min":11.728},"last":3.2120000000000002},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.1000000000000001,"15min":1.0920000000000001},"min":{"1min":0.72899999999999998,"5min":0.51700000000000002,"15min":0.51400000000000001},"max":{"1min":1.2310000000000001,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":4.6779999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.224,"15min":1.137},"min":{"1min":0.76400000000000001,"5min":0.755,"15min":0.65200000000000002},"max":{"1min":1.268,"5min":18.041,"15min":18.041},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.345,"15min":1.228},"min":{"1min":0.98699999999999999,"5min":0.79700000000000004,"15min":0.56000000000000005},"max":{"1min":1.579,"5min":20.375,"15min":20.375},"last":3.3319999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575305,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":665052,"kb_used_data":529812,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":93706788,"statfs":{"total":96636764160,"available":95955750912,"internally_reserved":0,"allocated":542527488,"data_stored":530952161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":138475974},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":0.83699999999999997,"15min":0.82099999999999995},"min":{"1min":0.54400000000000004,"5min":0.43099999999999999,"15min":0.40100000000000002},"max":{"1min":7.04,"5min":7.04,"15min":7.04},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.131,"5min":0.876,"15min":0.85899999999999999},"min":{"1min":0.53500000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":5.4720000000000004,"5min":5.4720000000000004,"15min":5.4720000000000004},"last":1.0680000000000001}]},{"osd":1,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83199999999999996,"15min":0.88},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.284,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":1.151},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.83699999999999997,"15min":0.85099999999999998},"min":{"1min":0.57999999999999996,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":4.1630000000000003,"5min":4.1630000000000003,"15min":4.1630000000000003},"last":0.91700000000000004}]},{"osd":2,"last update":"Mon Oct 30 14:51:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.92300000000000004,"15min":0.95999999999999996},"min":{"1min":0.61299999999999999,"5min":0.503,"15min":0.44800000000000001},"max":{"1min":1.3320000000000001,"5min":1.3640000000000001,"15min":2.044},"last":1.224},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.087,"15min":1.024},"min":{"1min":0.81899999999999995,"5min":0.69099999999999995,"15min":0.51600000000000001},"max":{"1min":1.425,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":0.98699999999999999}]},{"osd":3,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.85599999999999998,"15min":0.93899999999999995},"min":{"1min":0.623,"5min":0.58399999999999996,"15min":0.56499999999999995},"max":{"1min":1.1000000000000001,"5min":1.2010000000000001,"15min":1.4550000000000001},"last":0.877},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.03,"15min":0.97199999999999998},"min":{"1min":0.75,"5min":0.623,"15min":0.52100000000000002},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.4059999999999999},"last":1.294}]},{"osd":4,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.149,"15min":1.0469999999999999},"min":{"1min":0.622,"5min":0.62,"15min":0.59099999999999997},"max":{"1min":8.7360000000000007,"5min":8.7360000000000007,"15min":8.7360000000000007},"last":0.77800000000000002},{"interface":"front","average":{"1min":2.335,"5min":1.244,"15min":1.0640000000000001},"min":{"1min":0.621,"5min":0.61199999999999999,"15min":0.59399999999999997},"max":{"1min":23.640000000000001,"5min":23.640000000000001,"15min":23.640000000000001},"last":0.73199999999999998}]},{"osd":6,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1339999999999999,"15min":1.0700000000000001},"min":{"1min":0.80100000000000005,"5min":0.73399999999999999,"15min":0.55500000000000005},"max":{"1min":1.546,"5min":1.7470000000000001,"15min":2.306},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.194,"15min":1.127},"min":{"1min":0.878,"5min":0.875,"15min":0.57899999999999996},"max":{"1min":1.575,"5min":1.7569999999999999,"15min":2.774},"last":1.367}]},{"osd":7,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.99399999999999999,"15min":1.0629999999999999},"min":{"1min":0.71899999999999997,"5min":0.63,"15min":0.40799999999999997},"max":{"1min":1.198,"5min":1.3100000000000001,"15min":3.0739999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.036,"5min":1.129,"15min":1.1739999999999999},"min":{"1min":0.87,"5min":0.87,"15min":0.77600000000000002},"max":{"1min":1.244,"5min":1.53,"15min":2.1339999999999999},"last":0.95299999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575305,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611736,"kb_used_data":476496,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":93760104,"statfs":{"total":96636764160,"available":96010346496,"internally_reserved":0,"allocated":487931904,"data_stored":476574704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":138475974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.0109999999999999,"15min":0.876},"min":{"1min":0.53300000000000003,"5min":0.48199999999999998,"15min":0.27900000000000003},"max":{"1min":3.222,"5min":7.1870000000000003,"15min":7.1870000000000003},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.099,"5min":1.107,"15min":0.95099999999999996},"min":{"1min":0.68899999999999995,"5min":0.48199999999999998,"15min":0.33900000000000002},"max":{"1min":3.3010000000000002,"5min":10.864000000000001,"15min":10.864000000000001},"last":0.99299999999999999}]},{"osd":1,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.04,"15min":0.95199999999999996},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.50700000000000001},"max":{"1min":3.3780000000000001,"5min":7.6980000000000004,"15min":7.6980000000000004},"last":1.101},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.91300000000000003,"15min":0.83999999999999997},"min":{"1min":0.58699999999999997,"5min":0.48399999999999999,"15min":0.36599999999999999},"max":{"1min":2.1789999999999998,"5min":5.5259999999999998,"15min":5.5259999999999998},"last":0.63200000000000001}]},{"osd":2,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0229999999999999,"15min":0.96199999999999997},"min":{"1min":0.64300000000000002,"5min":0.57499999999999996,"15min":0.40899999999999997},"max":{"1min":2.2839999999999998,"5min":5.423,"15min":5.423},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.254,"15min":1.0940000000000001},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.54400000000000004},"max":{"1min":3.4500000000000002,"5min":13.221,"15min":13.221},"last":1.181}]},{"osd":3,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.96099999999999997,"15min":0.86199999999999999},"min":{"1min":0.626,"5min":0.54800000000000004,"15min":0.46899999999999997},"max":{"1min":7.5620000000000003,"5min":7.5620000000000003,"15min":7.5620000000000003},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.0860000000000001,"15min":1.0029999999999999},"min":{"1min":0.71199999999999997,"5min":0.65500000000000003,"15min":0.497},"max":{"1min":7.7370000000000001,"5min":7.7370000000000001,"15min":7.7370000000000001},"last":1.056}]},{"osd":5,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.091,"15min":0.97599999999999998},"min":{"1min":0.72499999999999998,"5min":0.64200000000000002,"15min":0.317},"max":{"1min":7.6360000000000001,"5min":7.6360000000000001,"15min":7.6360000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.159,"5min":1.2609999999999999,"15min":1.145},"min":{"1min":0.628,"5min":0.628,"15min":0.442},"max":{"1min":3.6389999999999998,"5min":7.2599999999999998,"15min":7.2599999999999998},"last":1.2749999999999999}]},{"osd":6,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1910000000000001,"15min":1.1020000000000001},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.39200000000000002},"max":{"1min":2.379,"5min":5.4729999999999999,"15min":5.4729999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.2969999999999999,"15min":1.1830000000000001},"min":{"1min":0.84299999999999997,"5min":0.73999999999999999,"15min":0.47899999999999998},"max":{"1min":6.5270000000000001,"5min":10.102,"15min":10.102},"last":1.23}]},{"osd":7,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.1120000000000001,"15min":1.042},"min":{"1min":0.75700000000000001,"5min":0.69099999999999995,"15min":0.40100000000000002},"max":{"1min":9.1240000000000006,"5min":9.1240000000000006,"15min":9.1240000000000006},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.423,"5min":1.2809999999999999,"15min":1.2130000000000001},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.52900000000000003},"max":{"1min":7.0490000000000004,"5min":7.0490000000000004,"15min":7.0490000000000004},"last":1.1299999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575303,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645632,"kb_used_data":497912,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":93726208,"statfs":{"total":96636764160,"available":95975636992,"internally_reserved":0,"allocated":509861888,"data_stored":498479753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":151255496},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.53600000000000003,"15min":0.48299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.1130000000000004,"5min":4.1130000000000004,"15min":4.1130000000000004},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.54200000000000004,"15min":0.51200000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.91100000000000003,"5min":4.5190000000000001,"15min":4.5190000000000001},"last":0.376}]},{"osd":1,"last update":"Mon Oct 30 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.58999999999999997,"15min":0.54400000000000004},"min":{"1min":0.33100000000000002,"5min":0.317,"15min":0.314},"max":{"1min":3.988,"5min":3.988,"15min":3.988},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.56399999999999995,"15min":0.54900000000000004},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.29999999999999999},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.432}]},{"osd":2,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.65600000000000003,"15min":0.56100000000000005},"min":{"1min":0.31,"5min":0.29199999999999998,"15min":0.27600000000000002},"max":{"1min":1.0740000000000001,"5min":8.6519999999999992,"15min":8.6519999999999992},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.73999999999999999,"15min":0.63700000000000001},"min":{"1min":0.42199999999999999,"5min":0.41999999999999998,"15min":0.313},"max":{"1min":1.1040000000000001,"5min":7.0949999999999998,"15min":7.0949999999999998},"last":0.63300000000000001}]},{"osd":4,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.83399999999999996,"15min":0.82099999999999995},"min":{"1min":0.54600000000000004,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.1220000000000001,"5min":4.5730000000000004,"15min":4.5730000000000004},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.90200000000000002,"15min":0.85299999999999998},"min":{"1min":0.51500000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":4.0910000000000002,"5min":4.4720000000000004,"15min":4.4720000000000004},"last":0.78400000000000003}]},{"osd":5,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.0029999999999999,"15min":0.90000000000000002},"min":{"1min":0.72799999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.296,"5min":14.974,"15min":14.974},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.042,"5min":0.96699999999999997,"15min":0.873},"min":{"1min":0.746,"5min":0.52600000000000002,"15min":0.46600000000000003},"max":{"1min":4.0629999999999997,"5min":5.907,"15min":5.907},"last":0.88400000000000001}]},{"osd":6,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.01,"15min":0.91900000000000004},"min":{"1min":0.70899999999999996,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":5.8140000000000001,"5min":5.8300000000000001,"15min":5.8300000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.159,"5min":1.073,"15min":0.96299999999999997},"min":{"1min":0.70799999999999996,"5min":0.60199999999999998,"15min":0.55100000000000005},"max":{"1min":6.0300000000000002,"5min":6.0300000000000002,"15min":6.0300000000000002},"last":0.99099999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":1.115,"15min":0.995},"min":{"1min":0.67600000000000005,"5min":0.66800000000000004,"15min":0.61699999999999999},"max":{"1min":1.2549999999999999,"5min":16.282,"15min":16.282},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.123,"15min":0.97199999999999998},"min":{"1min":0.70999999999999996,"5min":0.627,"15min":0.58099999999999996},"max":{"1min":1.6519999999999999,"5min":16.096,"15min":16.096},"last":0.83599999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575305,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":417172,"kb_used_data":309708,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":93954668,"statfs":{"total":96636764160,"available":96209580032,"internally_reserved":0,"allocated":317140992,"data_stored":310407991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.45300000000000001,"15min":0.48699999999999999},"min":{"1min":0.30099999999999999,"5min":0.27400000000000002,"15min":0.26300000000000001},"max":{"1min":0.67800000000000005,"5min":0.79100000000000004,"15min":13.526},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.50900000000000001,"15min":0.51300000000000001},"min":{"1min":0.38300000000000001,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":1.2190000000000001},"last":0.434}]},{"osd":1,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46300000000000002,"15min":0.47199999999999998},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65500000000000003,"5min":0.67200000000000004,"15min":1.5469999999999999},"last":0.314},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54400000000000004,"15min":0.56000000000000005},"min":{"1min":0.38700000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73499999999999999,"5min":0.79300000000000004,"15min":1.452},"last":0.46000000000000002}]},{"osd":3,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58799999999999997,"15min":0.60699999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81299999999999994,"5min":0.95399999999999996,"15min":1.9390000000000001},"last":0.497},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.53900000000000003},"min":{"1min":0.35899999999999999,"5min":0.35499999999999998,"15min":0.33100000000000002},"max":{"1min":0.78500000000000003,"5min":0.79300000000000004,"15min":1.913},"last":0.47599999999999998}]},{"osd":4,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.81699999999999995,"15min":0.81499999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.44900000000000001},"max":{"1min":1.0309999999999999,"5min":4.2229999999999999,"15min":4.2229999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.91600000000000004,"15min":0.91000000000000003},"min":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.59599999999999997},"max":{"1min":1.081,"5min":4.2439999999999998,"15min":4.2439999999999998},"last":0.82799999999999996}]},{"osd":5,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.98999999999999999,"15min":0.86399999999999999},"min":{"1min":0.51900000000000002,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0620000000000001,"5min":22.285,"15min":22.285},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.82899999999999996,"15min":0.83399999999999996},"min":{"1min":0.58199999999999996,"5min":0.56000000000000005,"15min":0.51900000000000002},"max":{"1min":1.1319999999999999,"5min":2.3999999999999999,"15min":2.742},"last":0.66900000000000004}]},{"osd":6,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.86099999999999999,"15min":0.91500000000000004},"min":{"1min":0.65000000000000002,"5min":0.58799999999999997,"15min":0.54500000000000004},"max":{"1min":1.0489999999999999,"5min":2.911,"15min":27.731000000000002},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.872,"15min":0.874},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.1679999999999999,"5min":1.9099999999999999,"15min":2.0009999999999999},"last":0.81699999999999995}]},{"osd":7,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84299999999999997,"15min":0.91600000000000004},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.56499999999999995},"max":{"1min":1.105,"5min":1.2390000000000001,"15min":18.870999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.90400000000000003,"15min":0.96399999999999997},"min":{"1min":0.66600000000000004,"5min":0.61499999999999999,"15min":0.55500000000000005},"max":{"1min":1.0860000000000001,"5min":3.3809999999999998,"15min":18.356999999999999},"last":0.99099999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575304,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461964,"kb_used_data":344772,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":93909876,"statfs":{"total":96636764160,"available":96163713024,"internally_reserved":0,"allocated":353046528,"data_stored":346331165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":119994823},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.47399999999999998,"15min":0.44800000000000001},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.68200000000000005,"5min":2.4889999999999999,"15min":2.4889999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.51600000000000001,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71199999999999997,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.55800000000000005}]},{"osd":2,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.64500000000000002,"15min":0.56699999999999995},"min":{"1min":0.318,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79000000000000004,"5min":8.3160000000000007,"15min":8.3160000000000007},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.54000000000000004,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.27200000000000002,"15min":0.26800000000000002},"max":{"1min":0.61399999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.58099999999999996}]},{"osd":3,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.61099999999999999,"15min":0.59599999999999997},"min":{"1min":0.38800000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84399999999999997,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.623,"5min":0.64900000000000002,"15min":0.63600000000000001},"min":{"1min":0.439,"5min":0.38100000000000001,"15min":0.36499999999999999},"max":{"1min":0.89200000000000002,"5min":3.54,"15min":3.54},"last":0.66000000000000003}]},{"osd":4,"last update":"Mon Oct 30 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.92000000000000004,"15min":0.83999999999999997},"min":{"1min":0.505,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.92600000000000005,"5min":10.525,"15min":10.525},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.94899999999999995,"15min":0.83899999999999997},"min":{"1min":0.68400000000000005,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.997,"5min":10.571,"15min":10.571},"last":0.84799999999999998}]},{"osd":5,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.88900000000000001,"15min":0.89100000000000001},"min":{"1min":0.628,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.0289999999999999,"5min":3.4969999999999999,"15min":3.4969999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.876,"5min":0.84399999999999997,"15min":0.79200000000000004},"min":{"1min":0.72399999999999998,"5min":0.56000000000000005,"15min":0.49099999999999999},"max":{"1min":1.1739999999999999,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.96499999999999997}]},{"osd":6,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.80900000000000005},"min":{"1min":0.58399999999999996,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.1040000000000001,"5min":1.875,"15min":2.4710000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.88900000000000001,"15min":0.83399999999999996},"min":{"1min":0.57399999999999995,"5min":0.53100000000000003,"15min":0.51700000000000002},"max":{"1min":1.2130000000000001,"5min":5.3680000000000003,"15min":5.3680000000000003},"last":0.995}]},{"osd":7,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":1.081,"15min":0.91600000000000004},"min":{"1min":0.59599999999999997,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0249999999999999,"5min":19.885000000000002,"15min":19.885000000000002},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":1.054,"15min":0.93200000000000005},"min":{"1min":0.60099999999999998,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.127,"5min":13.401999999999999,"15min":13.401999999999999},"last":1.117}]}]},{"osd":0,"up_from":13,"seq":55834575306,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545984,"kb_used_data":416312,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":93825856,"statfs":{"total":96636764160,"available":96077676544,"internally_reserved":0,"allocated":426303488,"data_stored":414784024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132774342},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45900000000000002,"15min":0.44800000000000001},"min":{"1min":0.32200000000000001,"5min":0.26800000000000002,"15min":0.22500000000000001},"max":{"1min":0.749,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.53300000000000003,"15min":0.503},"min":{"1min":0.36699999999999999,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":0.68999999999999995,"5min":3.367,"15min":3.367},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58199999999999996,"15min":0.53600000000000003},"min":{"1min":0.44400000000000001,"5min":0.378,"15min":0.30499999999999999},"max":{"1min":0.751,"5min":3.222,"15min":3.222},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53200000000000003,"15min":0.56000000000000005},"min":{"1min":0.40000000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":1.095},"last":0.434}]},{"osd":3,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.56599999999999995,"15min":0.60199999999999998},"min":{"1min":0.34000000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.63300000000000001,"5min":6.4980000000000002,"15min":6.4980000000000002},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57499999999999996,"15min":0.52900000000000003},"min":{"1min":0.40400000000000003,"5min":0.40100000000000002,"15min":0.28000000000000003},"max":{"1min":0.71699999999999997,"5min":2.024,"15min":2.024},"last":0.497}]},{"osd":4,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.78800000000000003,"15min":0.82599999999999996},"min":{"1min":0.57599999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.95399999999999996,"5min":2.0659999999999998,"15min":2.8380000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81399999999999995,"15min":0.85999999999999999},"min":{"1min":0.65100000000000002,"5min":0.56899999999999995,"15min":0.53900000000000003},"max":{"1min":0.97699999999999998,"5min":2.1000000000000001,"15min":5.8959999999999999},"last":0.75700000000000001}]},{"osd":5,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.872,"15min":0.86099999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.46300000000000002},"max":{"1min":1.212,"5min":1.278,"15min":6.4480000000000004},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.92000000000000004,"15min":0.85599999999999998},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.54000000000000004},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":2.6869999999999998},"last":0.84399999999999997}]},{"osd":6,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.83899999999999997,"15min":0.84699999999999998},"min":{"1min":0.64800000000000002,"5min":0.55300000000000005,"15min":0.54400000000000004},"max":{"1min":1.133,"5min":1.2150000000000001,"15min":4.9980000000000002},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.877,"5min":0.85499999999999998,"15min":0.86099999999999999},"min":{"1min":0.65300000000000002,"5min":0.55100000000000005,"15min":0.54000000000000004},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.5980000000000001},"last":0.74099999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.86799999999999999,"15min":0.84699999999999998},"min":{"1min":0.63700000000000001,"5min":0.57899999999999996,"15min":0.499},"max":{"1min":1.1120000000000001,"5min":3.6589999999999998,"15min":3.6589999999999998},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.93100000000000005,"15min":0.88400000000000001},"min":{"1min":0.76300000000000001,"5min":0.625,"15min":0.54100000000000004},"max":{"1min":1.1579999999999999,"5min":1.262,"15min":2.6429999999999998},"last":1.02}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":104873984,"data_stored":104858260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159408128,"data_stored":159388062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":115867648,"data_stored":115847932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":200708096,"data_stored":200690717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":153219072,"data_stored":153204613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":208064512,"data_stored":208040997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":211091456,"data_stored":211074246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":194002944,"data_stored":193985181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":320053248,"data_stored":308933294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192856064,"data_stored":186541001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200491008,"data_stored":194157957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":308371456,"data_stored":297386934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":333930496,"data_stored":322967989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":333680640,"data_stored":322509062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":183648256,"data_stored":176857298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":295645184,"data_stored":285317565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:51:35.187 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:51:35.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.360+0000 7f517d1c5640 1 Processor -- start 2023-10-30T14:51:35.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.360+0000 7f517d1c5640 1 -- start start 2023-10-30T14:51:35.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f517d1c5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f51780cb7d0 0x7f51780c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:35.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f517d1c5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51780c2310 con 0x7f51780c3e40 2023-10-30T14:51:35.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f517d1c5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5178156710 con 0x7f5178150830 2023-10-30T14:51:35.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f517d1c5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f517806bc10 con 0x7f51780cb7d0 2023-10-30T14:51:35.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f517759e640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f51780c3e40 0x7f5178070520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41810/0 (socket says 172.21.15.57:41810) 2023-10-30T14:51:35.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f517759e640 1 -- 172.21.15.57:0/3961777402 learned_addr learned my addr 172.21.15.57:0/3961777402 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:51:35.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f517659c640 1 --2- 172.21.15.57:0/3961777402 >> v2:172.21.15.88:3300/0 conn(0x7f51780cb7d0 0x7f51780c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:35.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f5175d9b640 1 -- 172.21.15.57:0/3961777402 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300489058 0 0) 0x7f51780c2310 con 0x7f51780c3e40 2023-10-30T14:51:35.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f5175d9b640 1 -- 172.21.15.57:0/3961777402 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5160003530 con 0x7f51780c3e40 2023-10-30T14:51:35.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f5175d9b640 1 -- 172.21.15.57:0/3961777402 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2354082780 0 0) 0x7f5160003530 con 0x7f51780c3e40 2023-10-30T14:51:35.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f5175d9b640 1 -- 172.21.15.57:0/3961777402 >> v2:172.21.15.88:3300/0 conn(0x7f51780cb7d0 msgr2=0x7f51780c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:35.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f5175d9b640 1 --2- 172.21.15.57:0/3961777402 >> v2:172.21.15.88:3300/0 conn(0x7f51780cb7d0 0x7f51780c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f5175d9b640 1 -- 172.21.15.57:0/3961777402 >> v1:172.21.15.88:6789/0 conn(0x7f5178150830 legacy=0x7f517806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:35.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f5175d9b640 1 -- 172.21.15.57:0/3961777402 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51780c4360 con 0x7f51780c3e40 2023-10-30T14:51:35.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f5175d9b640 1 -- 172.21.15.57:0/3961777402 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f5170001a20 con 0x7f51780c3e40 2023-10-30T14:51:35.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f5175d9b640 1 -- 172.21.15.57:0/3961777402 >> v1:172.21.15.57:6789/0 conn(0x7f51780c3e40 legacy=0x7f5178070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:35.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f5175d9b640 1 --2- 172.21.15.57:0/3961777402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5160003980 0x7f5160003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:35.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f5175d9b640 1 --2- 172.21.15.57:0/3961777402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51600047d0 0x7f5160004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:35.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f5175d9b640 1 --2- 172.21.15.57:0/3961777402 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5160005190 0x7f516000dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:35.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f5175d9b640 1 -- 172.21.15.57:0/3961777402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517806bc10 con 0x7f51600047d0 2023-10-30T14:51:35.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f5175d9b640 1 -- 172.21.15.57:0/3961777402 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516000e250 con 0x7f5160003980 2023-10-30T14:51:35.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f5175d9b640 1 -- 172.21.15.57:0/3961777402 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516000e410 con 0x7f5160005190 2023-10-30T14:51:35.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f517659c640 1 --2- 172.21.15.57:0/3961777402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51600047d0 0x7f5160004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:35.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f5176d9d640 1 --2- 172.21.15.57:0/3961777402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5160003980 0x7f5160003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:35.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f517759e640 1 --2- 172.21.15.57:0/3961777402 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5160005190 0x7f516000dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:35.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f517659c640 1 -- 172.21.15.57:0/3961777402 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5160005190 msgr2=0x7f516000dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:35.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f517659c640 1 --2- 172.21.15.57:0/3961777402 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5160005190 0x7f516000dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f517659c640 1 -- 172.21.15.57:0/3961777402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5160003980 msgr2=0x7f5160003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:35.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f517659c640 1 --2- 172.21.15.57:0/3961777402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5160003980 0x7f5160003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f517659c640 1 -- 172.21.15.57:0/3961777402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5160003530 con 0x7f51600047d0 2023-10-30T14:51:35.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.364+0000 7f517759e640 1 -- 172.21.15.57:0/3961777402 reap_dead start 2023-10-30T14:51:35.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.368+0000 7f517659c640 1 --2- 172.21.15.57:0/3961777402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51600047d0 0x7f5160004c50 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f516c009ae0 tx=0x7f516c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d856f585091b86 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:35.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.368+0000 7f5175d9b640 1 -- 172.21.15.57:0/3961777402 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c0a5640 con 0x7f51600047d0 2023-10-30T14:51:35.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.368+0000 7f5175d9b640 1 -- 172.21.15.57:0/3961777402 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516c0a57a0 con 0x7f51600047d0 2023-10-30T14:51:35.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.368+0000 7f5175d9b640 1 -- 172.21.15.57:0/3961777402 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c0a5b50 con 0x7f51600047d0 2023-10-30T14:51:35.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.368+0000 7f517d1c5640 1 -- 172.21.15.57:0/3961777402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51600047d0 msgr2=0x7f5160004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:35.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.368+0000 7f517d1c5640 1 --2- 172.21.15.57:0/3961777402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51600047d0 0x7f5160004c50 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f516c009ae0 tx=0x7f516c01ca50 comp rx=0 tx=0).stop 2023-10-30T14:51:35.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.368+0000 7f517d1c5640 1 -- 172.21.15.57:0/3961777402 shutdown_connections 2023-10-30T14:51:35.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.368+0000 7f517d1c5640 1 --2- 172.21.15.57:0/3961777402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51600047d0 0x7f5160004c50 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.368+0000 7f517d1c5640 1 -- 172.21.15.57:0/3961777402 >> 172.21.15.57:0/3961777402 conn(0x7f51780cac70 msgr2=0x7f51780cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:35.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.368+0000 7f517d1c5640 1 -- 172.21.15.57:0/3961777402 shutdown_connections 2023-10-30T14:51:35.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.368+0000 7f517d1c5640 1 -- 172.21.15.57:0/3961777402 wait complete. 2023-10-30T14:51:35.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517d1c5640 1 Processor -- start 2023-10-30T14:51:35.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517d1c5640 1 -- start start 2023-10-30T14:51:35.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517d1c5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51600047d0 0x7f5178155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:35.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517d1c5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51781ed9e0 0x7f5178154eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:35.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517d1c5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51781577c0 0x7f51781473d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:35.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517d1c5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5178155aa0 con 0x7f51781ed9e0 2023-10-30T14:51:35.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517d1c5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5178155750 con 0x7f51600047d0 2023-10-30T14:51:35.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517d1c5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5178156400 con 0x7f51781577c0 2023-10-30T14:51:35.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51781ed9e0 0x7f5178154eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:35.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51781ed9e0 0x7f5178154eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56628/0 (socket says 172.21.15.57:56628) 2023-10-30T14:51:35.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517659c640 1 -- 172.21.15.57:0/4228678448 learned_addr learned my addr 172.21.15.57:0/4228678448 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:35.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f5176d9d640 1 --2- 172.21.15.57:0/4228678448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51600047d0 0x7f5178155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:35.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517759e640 1 --2- 172.21.15.57:0/4228678448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51781577c0 0x7f51781473d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:35.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517659c640 1 -- 172.21.15.57:0/4228678448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51781577c0 msgr2=0x7f51781473d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:35.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517659c640 1 --2- 172.21.15.57:0/4228678448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51781577c0 0x7f51781473d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517659c640 1 -- 172.21.15.57:0/4228678448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51600047d0 msgr2=0x7f5178155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:35.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517659c640 1 --2- 172.21.15.57:0/4228678448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51600047d0 0x7f5178155ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517659c640 1 -- 172.21.15.57:0/4228678448 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51780c9c70 con 0x7f51781ed9e0 2023-10-30T14:51:35.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f5176d9d640 1 --2- 172.21.15.57:0/4228678448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51600047d0 0x7f5178155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:35.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517659c640 1 --2- 172.21.15.57:0/4228678448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51781ed9e0 0x7f5178154eb0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f516c0b0040 tx=0x7f516c0197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:35.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f515f7fe640 1 -- 172.21.15.57:0/4228678448 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c004220 con 0x7f51781ed9e0 2023-10-30T14:51:35.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f515f7fe640 1 -- 172.21.15.57:0/4228678448 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516c0043a0 con 0x7f51781ed9e0 2023-10-30T14:51:35.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517d1c5640 1 -- 172.21.15.57:0/4228678448 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51780c9f30 con 0x7f51781ed9e0 2023-10-30T14:51:35.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f515f7fe640 1 -- 172.21.15.57:0/4228678448 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c004510 con 0x7f51781ed9e0 2023-10-30T14:51:35.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517d1c5640 1 -- 172.21.15.57:0/4228678448 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51780ca4f0 con 0x7f51781ed9e0 2023-10-30T14:51:35.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.372+0000 7f517d1c5640 1 -- 172.21.15.57:0/4228678448 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f517804e170 con 0x7f51781ed9e0 2023-10-30T14:51:35.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.376+0000 7f515f7fe640 1 -- 172.21.15.57:0/4228678448 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f516c0b4070 con 0x7f51781ed9e0 2023-10-30T14:51:35.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.376+0000 7f515f7fe640 1 --2- 172.21.15.57:0/4228678448 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51640664d0 0x7f5164068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:35.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.376+0000 7f515f7fe640 1 -- 172.21.15.57:0/4228678448 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f516c11e430 con 0x7f51781ed9e0 2023-10-30T14:51:35.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.376+0000 7f5176d9d640 1 --2- 172.21.15.57:0/4228678448 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51640664d0 0x7f5164068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:35.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.376+0000 7f5176d9d640 1 --2- 172.21.15.57:0/4228678448 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51640664d0 0x7f5164068990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f5168005630 tx=0x7f5168009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:35.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.380+0000 7f515f7fe640 1 -- 172.21.15.57:0/4228678448 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f516c0eb2d0 con 0x7f51781ed9e0 2023-10-30T14:51:35.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.536+0000 7f517d1c5640 1 -- 172.21.15.57:0/4228678448 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5178156c90 con 0x7f51640664d0 2023-10-30T14:51:35.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.556+0000 7f515f7fe640 1 -- 172.21.15.57:0/4228678448 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+293482 (secure 0 0 0) 0x7f5168005320 con 0x7f51640664d0 2023-10-30T14:51:35.563 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:51:35.565 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-30T14:51:35.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.560+0000 7f517d1c5640 1 -- 172.21.15.57:0/4228678448 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51640664d0 msgr2=0x7f5164068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:35.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.560+0000 7f517d1c5640 1 --2- 172.21.15.57:0/4228678448 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51640664d0 0x7f5164068990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f5168005630 tx=0x7f5168009040 comp rx=0 tx=0).stop 2023-10-30T14:51:35.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.560+0000 7f517d1c5640 1 -- 172.21.15.57:0/4228678448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51781ed9e0 msgr2=0x7f5178154eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:35.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.560+0000 7f517d1c5640 1 --2- 172.21.15.57:0/4228678448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51781ed9e0 0x7f5178154eb0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f516c0b0040 tx=0x7f516c0197f0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.560+0000 7f517d1c5640 1 -- 172.21.15.57:0/4228678448 shutdown_connections 2023-10-30T14:51:35.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.560+0000 7f517d1c5640 1 --2- 172.21.15.57:0/4228678448 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f51640664d0 0x7f5164068990 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.560+0000 7f517d1c5640 1 --2- 172.21.15.57:0/4228678448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51781577c0 0x7f51781473d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.560+0000 7f517d1c5640 1 --2- 172.21.15.57:0/4228678448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51781ed9e0 0x7f5178154eb0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.560+0000 7f517d1c5640 1 --2- 172.21.15.57:0/4228678448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51600047d0 0x7f5178155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.560+0000 7f517d1c5640 1 -- 172.21.15.57:0/4228678448 >> 172.21.15.57:0/4228678448 conn(0x7f51780cac70 msgr2=0x7f51780733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:35.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.560+0000 7f517d1c5640 1 -- 172.21.15.57:0/4228678448 shutdown_connections 2023-10-30T14:51:35.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.560+0000 7f517d1c5640 1 -- 172.21.15.57:0/4228678448 wait complete. 2023-10-30T14:51:35.584 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":1192,"stamp":"2023-10-30T14:51:35.317764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1721022653,"num_objects":32709,"num_object_clones":14593,"num_object_copies":65418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32709,"num_whiteouts":12799,"num_read":10747,"num_read_kb":3497812,"num_write":97664,"num_write_kb":2851506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1111,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109301,"ondisk_log_size":109301,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":4473736,"kb_used_data":3440456,"kb_used_omap":12,"kb_used_meta":1033203,"kb_avail":750500984,"statfs":{"total":773094113280,"available":768513007616,"internally_reserved":0,"allocated":3523026944,"data_stored":3445826398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":1058000441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":75,"apply_latency_ms":75,"commit_latency_ns":75000000,"apply_latency_ns":75000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20857760,"num_objects":198,"num_object_clones":148,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":-10,"num_read":54,"num_read_kb":4921,"num_write":4539,"num_write_kb":27455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004603"},"pg_stats":[{"pgid":"3.3f","version":"67'1276","reported_seq":3239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.482005+0000","last_change":"2023-10-30T14:23:31.082534+0000","last_active":"2023-10-30T14:51:28.482005+0000","last_peered":"2023-10-30T14:51:28.482005+0000","last_clean":"2023-10-30T14:51:28.482005+0000","last_became_active":"2023-10-30T14:14:04.650882+0000","last_became_peered":"2023-10-30T14:14:04.650882+0000","last_unstale":"2023-10-30T14:51:28.482005+0000","last_undegraded":"2023-10-30T14:51:28.482005+0000","last_fullsized":"2023-10-30T14:51:28.482005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1276,"log_dups_size":0,"ondisk_log_size":1276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:46:41.132286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010330941,"stat_sum":{"num_bytes":16875912,"num_objects":459,"num_object_clones":215,"num_object_copies":918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":459,"num_whiteouts":184,"num_read":62,"num_read_kb":8246,"num_write":1073,"num_write_kb":18488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"67'1268","reported_seq":3223,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.429379+0000","last_change":"2023-10-30T14:23:30.771600+0000","last_active":"2023-10-30T14:51:29.429379+0000","last_peered":"2023-10-30T14:51:29.429379+0000","last_clean":"2023-10-30T14:51:29.429379+0000","last_became_active":"2023-10-30T14:14:04.587942+0000","last_became_peered":"2023-10-30T14:14:04.587942+0000","last_unstale":"2023-10-30T14:51:29.429379+0000","last_undegraded":"2023-10-30T14:51:29.429379+0000","last_fullsized":"2023-10-30T14:51:29.429379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1268,"log_dups_size":0,"ondisk_log_size":1268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:54:24.389508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002850184,"stat_sum":{"num_bytes":12592181,"num_objects":480,"num_object_clones":220,"num_object_copies":960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":480,"num_whiteouts":195,"num_read":59,"num_read_kb":8220,"num_write":1065,"num_write_kb":14470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"67'1295","reported_seq":3258,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.481286+0000","last_change":"2023-10-30T14:23:30.776754+0000","last_active":"2023-10-30T14:51:28.481286+0000","last_peered":"2023-10-30T14:51:28.481286+0000","last_clean":"2023-10-30T14:51:28.481286+0000","last_became_active":"2023-10-30T14:14:04.590700+0000","last_became_peered":"2023-10-30T14:14:04.590700+0000","last_unstale":"2023-10-30T14:51:28.481286+0000","last_undegraded":"2023-10-30T14:51:28.481286+0000","last_fullsized":"2023-10-30T14:51:28.481286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:11:25.956171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042861139999999997,"stat_sum":{"num_bytes":12573829,"num_objects":487,"num_object_clones":226,"num_object_copies":974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":204,"num_read":73,"num_read_kb":16409,"num_write":1057,"num_write_kb":22081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"67'1290","reported_seq":3252,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.429904+0000","last_change":"2023-10-30T14:23:31.067598+0000","last_active":"2023-10-30T14:51:29.429904+0000","last_peered":"2023-10-30T14:51:29.429904+0000","last_clean":"2023-10-30T14:51:29.429904+0000","last_became_active":"2023-10-30T14:14:45.661677+0000","last_became_peered":"2023-10-30T14:14:45.661677+0000","last_unstale":"2023-10-30T14:51:29.429904+0000","last_undegraded":"2023-10-30T14:51:29.429904+0000","last_fullsized":"2023-10-30T14:51:29.429904+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1290,"log_dups_size":0,"ondisk_log_size":1290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:33.531763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003131572,"stat_sum":{"num_bytes":8739581,"num_objects":477,"num_object_clones":215,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":188,"num_read":63,"num_read_kb":69,"num_write":1083,"num_write_kb":14952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"67'1295","reported_seq":3255,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.455528+0000","last_change":"2023-10-30T14:23:31.072132+0000","last_active":"2023-10-30T14:51:28.455528+0000","last_peered":"2023-10-30T14:51:28.455528+0000","last_clean":"2023-10-30T14:51:28.455528+0000","last_became_active":"2023-10-30T14:14:04.644683+0000","last_became_peered":"2023-10-30T14:14:04.644683+0000","last_unstale":"2023-10-30T14:51:28.455528+0000","last_undegraded":"2023-10-30T14:51:28.455528+0000","last_fullsized":"2023-10-30T14:51:28.455528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:39.271356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052462430000000003,"stat_sum":{"num_bytes":8813426,"num_objects":481,"num_object_clones":223,"num_object_copies":962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":202,"num_read":70,"num_read_kb":4116,"num_write":1073,"num_write_kb":10060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'1284","reported_seq":3252,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.493921+0000","last_change":"2023-10-30T14:23:30.777057+0000","last_active":"2023-10-30T14:51:28.493921+0000","last_peered":"2023-10-30T14:51:28.493921+0000","last_clean":"2023-10-30T14:51:28.493921+0000","last_became_active":"2023-10-30T14:14:04.619335+0000","last_became_peered":"2023-10-30T14:14:04.619335+0000","last_unstale":"2023-10-30T14:51:28.493921+0000","last_undegraded":"2023-10-30T14:51:28.493921+0000","last_fullsized":"2023-10-30T14:51:28.493921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1284,"log_dups_size":0,"ondisk_log_size":1284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:16:48.356144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062283859999999998,"stat_sum":{"num_bytes":20637019,"num_objects":477,"num_object_clones":220,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":194,"num_read":75,"num_read_kb":12254,"num_write":1076,"num_write_kb":22206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"67'1281","reported_seq":3231,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.464417+0000","last_change":"2023-10-30T14:23:31.080820+0000","last_active":"2023-10-30T14:51:29.464417+0000","last_peered":"2023-10-30T14:51:29.464417+0000","last_clean":"2023-10-30T14:51:29.464417+0000","last_became_active":"2023-10-30T14:14:04.641323+0000","last_became_peered":"2023-10-30T14:14:04.641323+0000","last_unstale":"2023-10-30T14:51:29.464417+0000","last_undegraded":"2023-10-30T14:51:29.464417+0000","last_fullsized":"2023-10-30T14:51:29.464417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1281,"log_dups_size":0,"ondisk_log_size":1281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:11.764013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011228897,"stat_sum":{"num_bytes":14656224,"num_objects":488,"num_object_clones":222,"num_object_copies":976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":202,"num_read":59,"num_read_kb":8203,"num_write":1074,"num_write_kb":18148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'1241","reported_seq":3196,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.459518+0000","last_change":"2023-10-30T14:23:31.067811+0000","last_active":"2023-10-30T14:51:29.459518+0000","last_peered":"2023-10-30T14:51:29.459518+0000","last_clean":"2023-10-30T14:51:29.459518+0000","last_became_active":"2023-10-30T14:14:04.618312+0000","last_became_peered":"2023-10-30T14:14:04.618312+0000","last_unstale":"2023-10-30T14:51:29.459518+0000","last_undegraded":"2023-10-30T14:51:29.459518+0000","last_fullsized":"2023-10-30T14:51:29.459518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1241,"log_dups_size":0,"ondisk_log_size":1241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:57:22.752126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002849426,"stat_sum":{"num_bytes":12868892,"num_objects":473,"num_object_clones":218,"num_object_copies":946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":473,"num_whiteouts":195,"num_read":60,"num_read_kb":8209,"num_write":1023,"num_write_kb":15138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"67'1328","reported_seq":3278,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.473222+0000","last_change":"2023-10-30T14:23:31.083143+0000","last_active":"2023-10-30T14:51:29.473222+0000","last_peered":"2023-10-30T14:51:29.473222+0000","last_clean":"2023-10-30T14:51:29.473222+0000","last_became_active":"2023-10-30T14:14:04.647361+0000","last_became_peered":"2023-10-30T14:14:04.647361+0000","last_unstale":"2023-10-30T14:51:29.473222+0000","last_undegraded":"2023-10-30T14:51:29.473222+0000","last_fullsized":"2023-10-30T14:51:29.473222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1328,"log_dups_size":0,"ondisk_log_size":1328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:06:53.616905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010459703000000001,"stat_sum":{"num_bytes":8468640,"num_objects":507,"num_object_clones":234,"num_object_copies":1014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":507,"num_whiteouts":206,"num_read":70,"num_read_kb":8267,"num_write":1118,"num_write_kb":13882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"67'1366","reported_seq":3270,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.468701+0000","last_change":"2023-10-30T14:23:31.075673+0000","last_active":"2023-10-30T14:51:29.468701+0000","last_peered":"2023-10-30T14:51:29.468701+0000","last_clean":"2023-10-30T14:51:29.468701+0000","last_became_active":"2023-10-30T14:14:04.630959+0000","last_became_peered":"2023-10-30T14:14:04.630959+0000","last_unstale":"2023-10-30T14:51:29.468701+0000","last_undegraded":"2023-10-30T14:51:29.468701+0000","last_fullsized":"2023-10-30T14:51:29.468701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1366,"log_dups_size":0,"ondisk_log_size":1366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:36.246017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079325130000000004,"stat_sum":{"num_bytes":12335215,"num_objects":539,"num_object_clones":286,"num_object_copies":1078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":191,"num_read":76,"num_read_kb":4189,"num_write":1090,"num_write_kb":13918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"67'1287","reported_seq":3229,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469676+0000","last_change":"2023-10-30T14:23:31.075101+0000","last_active":"2023-10-30T14:51:29.469676+0000","last_peered":"2023-10-30T14:51:29.469676+0000","last_clean":"2023-10-30T14:51:29.469676+0000","last_became_active":"2023-10-30T14:14:04.612653+0000","last_became_peered":"2023-10-30T14:14:04.612653+0000","last_unstale":"2023-10-30T14:51:29.469676+0000","last_undegraded":"2023-10-30T14:51:29.469676+0000","last_fullsized":"2023-10-30T14:51:29.469676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1287,"log_dups_size":0,"ondisk_log_size":1287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:34:44.855573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079098320000000003,"stat_sum":{"num_bytes":17654064,"num_objects":517,"num_object_clones":235,"num_object_copies":1034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":517,"num_whiteouts":205,"num_read":63,"num_read_kb":9809,"num_write":1073,"num_write_kb":19026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"67'1267","reported_seq":3224,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469272+0000","last_change":"2023-10-30T14:23:30.957983+0000","last_active":"2023-10-30T14:51:29.469272+0000","last_peered":"2023-10-30T14:51:29.469272+0000","last_clean":"2023-10-30T14:51:29.469272+0000","last_became_active":"2023-10-30T14:14:04.621135+0000","last_became_peered":"2023-10-30T14:14:04.621135+0000","last_unstale":"2023-10-30T14:51:29.469272+0000","last_undegraded":"2023-10-30T14:51:29.469272+0000","last_fullsized":"2023-10-30T14:51:29.469272+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1267,"log_dups_size":0,"ondisk_log_size":1267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:04:43.303589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025042409999999999,"stat_sum":{"num_bytes":14287686,"num_objects":497,"num_object_clones":224,"num_object_copies":994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":497,"num_whiteouts":193,"num_read":67,"num_read_kb":6339,"num_write":1071,"num_write_kb":19938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"67'1268","reported_seq":3251,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.468243+0000","last_change":"2023-10-30T14:23:31.070450+0000","last_active":"2023-10-30T14:51:29.468243+0000","last_peered":"2023-10-30T14:51:29.468243+0000","last_clean":"2023-10-30T14:51:29.468243+0000","last_became_active":"2023-10-30T14:14:04.616142+0000","last_became_peered":"2023-10-30T14:14:04.616142+0000","last_unstale":"2023-10-30T14:51:29.468243+0000","last_undegraded":"2023-10-30T14:51:29.468243+0000","last_fullsized":"2023-10-30T14:51:29.468243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1268,"log_dups_size":0,"ondisk_log_size":1268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:46:20.636160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049623779999999999,"stat_sum":{"num_bytes":9235702,"num_objects":480,"num_object_clones":219,"num_object_copies":960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":480,"num_whiteouts":189,"num_read":88,"num_read_kb":8286,"num_write":1065,"num_write_kb":19474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"67'1307","reported_seq":3281,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.633026+0000","last_change":"2023-10-30T14:23:30.775687+0000","last_active":"2023-10-30T14:51:29.633026+0000","last_peered":"2023-10-30T14:51:29.633026+0000","last_clean":"2023-10-30T14:51:29.633026+0000","last_became_active":"2023-10-30T14:14:04.588452+0000","last_became_peered":"2023-10-30T14:14:04.588452+0000","last_unstale":"2023-10-30T14:51:29.633026+0000","last_undegraded":"2023-10-30T14:51:29.633026+0000","last_fullsized":"2023-10-30T14:51:29.633026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1307,"log_dups_size":0,"ondisk_log_size":1307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:40:26.637820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046351760000000004,"stat_sum":{"num_bytes":12783229,"num_objects":466,"num_object_clones":216,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":195,"num_read":76,"num_read_kb":7792,"num_write":1098,"num_write_kb":17084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"67'1312","reported_seq":3276,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.502566+0000","last_change":"2023-10-30T14:23:30.770743+0000","last_active":"2023-10-30T14:51:28.502566+0000","last_peered":"2023-10-30T14:51:28.502566+0000","last_clean":"2023-10-30T14:51:28.502566+0000","last_became_active":"2023-10-30T14:14:04.620781+0000","last_became_peered":"2023-10-30T14:14:04.620781+0000","last_unstale":"2023-10-30T14:51:28.502566+0000","last_undegraded":"2023-10-30T14:51:28.502566+0000","last_fullsized":"2023-10-30T14:51:28.502566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1312,"log_dups_size":0,"ondisk_log_size":1312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:09.260750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002316706,"stat_sum":{"num_bytes":21325663,"num_objects":484,"num_object_clones":220,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":196,"num_read":69,"num_read_kb":13296,"num_write":1104,"num_write_kb":27083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"67'1296","reported_seq":3281,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.502845+0000","last_change":"2023-10-30T14:23:30.774013+0000","last_active":"2023-10-30T14:51:28.502845+0000","last_peered":"2023-10-30T14:51:28.502845+0000","last_clean":"2023-10-30T14:51:28.502845+0000","last_became_active":"2023-10-30T14:14:04.608190+0000","last_became_peered":"2023-10-30T14:14:04.608190+0000","last_unstale":"2023-10-30T14:51:28.502845+0000","last_undegraded":"2023-10-30T14:51:28.502845+0000","last_fullsized":"2023-10-30T14:51:28.502845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1296,"log_dups_size":0,"ondisk_log_size":1296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:44:40.113536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035618450000000001,"stat_sum":{"num_bytes":31744659,"num_objects":477,"num_object_clones":215,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":193,"num_read":85,"num_read_kb":23305,"num_write":1082,"num_write_kb":37194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"67'1351","reported_seq":3310,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469699+0000","last_change":"2023-10-30T14:23:30.779367+0000","last_active":"2023-10-30T14:51:29.469699+0000","last_peered":"2023-10-30T14:51:29.469699+0000","last_clean":"2023-10-30T14:51:29.469699+0000","last_became_active":"2023-10-30T14:14:04.641306+0000","last_became_peered":"2023-10-30T14:14:04.641306+0000","last_unstale":"2023-10-30T14:51:29.469699+0000","last_undegraded":"2023-10-30T14:51:29.469699+0000","last_fullsized":"2023-10-30T14:51:29.469699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1351,"log_dups_size":0,"ondisk_log_size":1351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:53:12.985530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066189990000000004,"stat_sum":{"num_bytes":3981038,"num_objects":511,"num_object_clones":236,"num_object_copies":1022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":206,"num_read":80,"num_read_kb":4151,"num_write":1127,"num_write_kb":9503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"67'1221","reported_seq":3204,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469982+0000","last_change":"2023-10-30T14:23:31.067340+0000","last_active":"2023-10-30T14:51:29.469982+0000","last_peered":"2023-10-30T14:51:29.469982+0000","last_clean":"2023-10-30T14:51:29.469982+0000","last_became_active":"2023-10-30T14:14:04.613644+0000","last_became_peered":"2023-10-30T14:14:04.613644+0000","last_unstale":"2023-10-30T14:51:29.469982+0000","last_undegraded":"2023-10-30T14:51:29.469982+0000","last_fullsized":"2023-10-30T14:51:29.469982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1221,"log_dups_size":0,"ondisk_log_size":1221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:50:40.405415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030699740000000001,"stat_sum":{"num_bytes":12387639,"num_objects":435,"num_object_clones":203,"num_object_copies":870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":435,"num_whiteouts":177,"num_read":71,"num_read_kb":8233,"num_write":1033,"num_write_kb":14118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"67'1285","reported_seq":3258,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.502929+0000","last_change":"2023-10-30T14:23:31.082413+0000","last_active":"2023-10-30T14:51:28.502929+0000","last_peered":"2023-10-30T14:51:28.502929+0000","last_clean":"2023-10-30T14:51:28.502929+0000","last_became_active":"2023-10-30T14:14:04.649850+0000","last_became_peered":"2023-10-30T14:14:04.649850+0000","last_unstale":"2023-10-30T14:51:28.502929+0000","last_undegraded":"2023-10-30T14:51:28.502929+0000","last_fullsized":"2023-10-30T14:51:28.502929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1285,"log_dups_size":0,"ondisk_log_size":1285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:27:56.738513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012599246,"stat_sum":{"num_bytes":25948792,"num_objects":470,"num_object_clones":219,"num_object_copies":940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":192,"num_read":77,"num_read_kb":24658,"num_write":1071,"num_write_kb":31961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"67'1335","reported_seq":3270,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.012858+0000","last_change":"2023-10-30T14:23:31.074724+0000","last_active":"2023-10-30T14:51:25.012858+0000","last_peered":"2023-10-30T14:51:25.012858+0000","last_clean":"2023-10-30T14:51:25.012858+0000","last_became_active":"2023-10-30T14:14:04.625042+0000","last_became_peered":"2023-10-30T14:14:04.625042+0000","last_unstale":"2023-10-30T14:51:25.012858+0000","last_undegraded":"2023-10-30T14:51:25.012858+0000","last_fullsized":"2023-10-30T14:51:25.012858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1335,"log_dups_size":0,"ondisk_log_size":1335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:45.303841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082351539999999997,"stat_sum":{"num_bytes":12700226,"num_objects":511,"num_object_clones":241,"num_object_copies":1022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":214,"num_read":61,"num_read_kb":10278,"num_write":1102,"num_write_kb":27066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"67'1264","reported_seq":3228,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.049982+0000","last_change":"2023-10-30T14:23:30.775419+0000","last_active":"2023-10-30T14:51:25.049982+0000","last_peered":"2023-10-30T14:51:25.049982+0000","last_clean":"2023-10-30T14:51:25.049982+0000","last_became_active":"2023-10-30T14:14:04.633586+0000","last_became_peered":"2023-10-30T14:14:04.633586+0000","last_unstale":"2023-10-30T14:51:25.049982+0000","last_undegraded":"2023-10-30T14:51:25.049982+0000","last_fullsized":"2023-10-30T14:51:25.049982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1264,"log_dups_size":0,"ondisk_log_size":1264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:19:06.449779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043616920000000003,"stat_sum":{"num_bytes":12325132,"num_objects":460,"num_object_clones":214,"num_object_copies":920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":192,"num_read":63,"num_read_kb":8213,"num_write":1055,"num_write_kb":14000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"67'1368","reported_seq":3340,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.046013+0000","last_change":"2023-10-30T14:23:30.777885+0000","last_active":"2023-10-30T14:51:25.046013+0000","last_peered":"2023-10-30T14:51:25.046013+0000","last_clean":"2023-10-30T14:51:25.046013+0000","last_became_active":"2023-10-30T14:14:04.642244+0000","last_became_peered":"2023-10-30T14:14:04.642244+0000","last_unstale":"2023-10-30T14:51:25.046013+0000","last_undegraded":"2023-10-30T14:51:25.046013+0000","last_fullsized":"2023-10-30T14:51:25.046013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1368,"log_dups_size":0,"ondisk_log_size":1368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:25.186063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065088710000000003,"stat_sum":{"num_bytes":25821219,"num_objects":511,"num_object_clones":233,"num_object_copies":1022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":210,"num_read":90,"num_read_kb":18405,"num_write":1129,"num_write_kb":31180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'1297","reported_seq":3278,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.499778+0000","last_change":"2023-10-30T14:23:31.070902+0000","last_active":"2023-10-30T14:51:28.499778+0000","last_peered":"2023-10-30T14:51:28.499778+0000","last_clean":"2023-10-30T14:51:28.499778+0000","last_became_active":"2023-10-30T14:14:04.620343+0000","last_became_peered":"2023-10-30T14:14:04.620343+0000","last_unstale":"2023-10-30T14:51:28.499778+0000","last_undegraded":"2023-10-30T14:51:28.499778+0000","last_fullsized":"2023-10-30T14:51:28.499778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1297,"log_dups_size":0,"ondisk_log_size":1297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:45:58.152823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048974149999999996,"stat_sum":{"num_bytes":12397335,"num_objects":479,"num_object_clones":220,"num_object_copies":958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":200,"num_read":86,"num_read_kb":8232,"num_write":1076,"num_write_kb":17925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'1353","reported_seq":3298,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.461566+0000","last_change":"2023-10-30T14:23:30.774642+0000","last_active":"2023-10-30T14:51:29.461566+0000","last_peered":"2023-10-30T14:51:29.461566+0000","last_clean":"2023-10-30T14:51:29.461566+0000","last_became_active":"2023-10-30T14:14:04.609720+0000","last_became_peered":"2023-10-30T14:14:04.609720+0000","last_unstale":"2023-10-30T14:51:29.461566+0000","last_undegraded":"2023-10-30T14:51:29.461566+0000","last_fullsized":"2023-10-30T14:51:29.461566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1353,"log_dups_size":0,"ondisk_log_size":1353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:10:58.430266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040008739999999997,"stat_sum":{"num_bytes":31017079,"num_objects":532,"num_object_clones":246,"num_object_copies":1064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":532,"num_whiteouts":228,"num_read":75,"num_read_kb":20512,"num_write":1096,"num_write_kb":31998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"67'1204","reported_seq":3182,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.455721+0000","last_change":"2023-10-30T14:23:31.069745+0000","last_active":"2023-10-30T14:51:29.455721+0000","last_peered":"2023-10-30T14:51:29.455721+0000","last_clean":"2023-10-30T14:51:29.455721+0000","last_became_active":"2023-10-30T14:14:04.620217+0000","last_became_peered":"2023-10-30T14:14:04.620217+0000","last_unstale":"2023-10-30T14:51:29.455721+0000","last_undegraded":"2023-10-30T14:51:29.455721+0000","last_fullsized":"2023-10-30T14:51:29.455721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:36:29.759579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044491130000000002,"stat_sum":{"num_bytes":20607496,"num_objects":451,"num_object_clones":209,"num_object_copies":902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":451,"num_whiteouts":184,"num_read":71,"num_read_kb":15956,"num_write":987,"num_write_kb":22411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"67'1350","reported_seq":3335,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.475769+0000","last_change":"2023-10-30T14:23:31.075832+0000","last_active":"2023-10-30T14:51:29.475769+0000","last_peered":"2023-10-30T14:51:29.475769+0000","last_clean":"2023-10-30T14:51:29.475769+0000","last_became_active":"2023-10-30T14:14:45.669878+0000","last_became_peered":"2023-10-30T14:14:45.669878+0000","last_unstale":"2023-10-30T14:51:29.475769+0000","last_undegraded":"2023-10-30T14:51:29.475769+0000","last_fullsized":"2023-10-30T14:51:29.475769+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1350,"log_dups_size":0,"ondisk_log_size":1350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:53:43.313641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0077997409999999998,"stat_sum":{"num_bytes":25371605,"num_objects":475,"num_object_clones":218,"num_object_copies":950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":193,"num_read":80,"num_read_kb":18178,"num_write":1139,"num_write_kb":27403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"67'1292","reported_seq":3264,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.057724+0000","last_change":"2023-10-30T14:23:30.773367+0000","last_active":"2023-10-30T14:51:25.057724+0000","last_peered":"2023-10-30T14:51:25.057724+0000","last_clean":"2023-10-30T14:51:25.057724+0000","last_became_active":"2023-10-30T14:14:04.597590+0000","last_became_peered":"2023-10-30T14:14:04.597590+0000","last_unstale":"2023-10-30T14:51:25.057724+0000","last_undegraded":"2023-10-30T14:51:25.057724+0000","last_fullsized":"2023-10-30T14:51:25.057724+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1292,"log_dups_size":0,"ondisk_log_size":1292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:28:58.351077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044830970000000001,"stat_sum":{"num_bytes":8698815,"num_objects":476,"num_object_clones":219,"num_object_copies":952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":476,"num_whiteouts":192,"num_read":75,"num_read_kb":4156,"num_write":1089,"num_write_kb":11768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"67'1181","reported_seq":3142,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.452118+0000","last_change":"2023-10-30T14:23:31.070930+0000","last_active":"2023-10-30T14:51:29.452118+0000","last_peered":"2023-10-30T14:51:29.452118+0000","last_clean":"2023-10-30T14:51:29.452118+0000","last_became_active":"2023-10-30T14:14:04.623902+0000","last_became_peered":"2023-10-30T14:14:04.623902+0000","last_unstale":"2023-10-30T14:51:29.452118+0000","last_undegraded":"2023-10-30T14:51:29.452118+0000","last_fullsized":"2023-10-30T14:51:29.452118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1181,"log_dups_size":0,"ondisk_log_size":1181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:13:07.040210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051343489999999999,"stat_sum":{"num_bytes":21106575,"num_objects":457,"num_object_clones":213,"num_object_copies":914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":457,"num_whiteouts":188,"num_read":58,"num_read_kb":12498,"num_write":966,"num_write_kb":25422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"67'1325","reported_seq":3275,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.632286+0000","last_change":"2023-10-30T14:23:30.775060+0000","last_active":"2023-10-30T14:51:29.632286+0000","last_peered":"2023-10-30T14:51:29.632286+0000","last_clean":"2023-10-30T14:51:29.632286+0000","last_became_active":"2023-10-30T14:14:04.597556+0000","last_became_peered":"2023-10-30T14:14:04.597556+0000","last_unstale":"2023-10-30T14:51:29.632286+0000","last_undegraded":"2023-10-30T14:51:29.632286+0000","last_fullsized":"2023-10-30T14:51:29.632286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1325,"log_dups_size":0,"ondisk_log_size":1325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:42.236111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040069729999999996,"stat_sum":{"num_bytes":25662090,"num_objects":516,"num_object_clones":239,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":516,"num_whiteouts":210,"num_read":73,"num_read_kb":17618,"num_write":1095,"num_write_kb":27367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"67'1302","reported_seq":3272,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.461498+0000","last_change":"2023-10-30T14:23:30.779406+0000","last_active":"2023-10-30T14:51:29.461498+0000","last_peered":"2023-10-30T14:51:29.461498+0000","last_clean":"2023-10-30T14:51:29.461498+0000","last_became_active":"2023-10-30T14:14:04.618430+0000","last_became_peered":"2023-10-30T14:14:04.618430+0000","last_unstale":"2023-10-30T14:51:29.461498+0000","last_undegraded":"2023-10-30T14:51:29.461498+0000","last_fullsized":"2023-10-30T14:51:29.461498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1302,"log_dups_size":0,"ondisk_log_size":1302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:44:00.464702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074984149999999996,"stat_sum":{"num_bytes":16389310,"num_objects":474,"num_object_clones":221,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":474,"num_whiteouts":197,"num_read":75,"num_read_kb":12323,"num_write":1094,"num_write_kb":17656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"67'1414","reported_seq":3372,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.458611+0000","last_change":"2023-10-30T14:23:31.067667+0000","last_active":"2023-10-30T14:51:29.458611+0000","last_peered":"2023-10-30T14:51:29.458611+0000","last_clean":"2023-10-30T14:51:29.458611+0000","last_became_active":"2023-10-30T14:14:04.606340+0000","last_became_peered":"2023-10-30T14:14:04.606340+0000","last_unstale":"2023-10-30T14:51:29.458611+0000","last_undegraded":"2023-10-30T14:51:29.458611+0000","last_fullsized":"2023-10-30T14:51:29.458611+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1414,"log_dups_size":0,"ondisk_log_size":1414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:41:14.948036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035094890000000002,"stat_sum":{"num_bytes":22345046,"num_objects":524,"num_object_clones":240,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":211,"num_read":82,"num_read_kb":9614,"num_write":1184,"num_write_kb":32857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"67'1333","reported_seq":3309,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.486748+0000","last_change":"2023-10-30T14:23:31.074353+0000","last_active":"2023-10-30T14:51:28.486748+0000","last_peered":"2023-10-30T14:51:28.486748+0000","last_clean":"2023-10-30T14:51:28.486748+0000","last_became_active":"2023-10-30T14:14:04.612588+0000","last_became_peered":"2023-10-30T14:14:04.612588+0000","last_unstale":"2023-10-30T14:51:28.486748+0000","last_undegraded":"2023-10-30T14:51:28.486748+0000","last_fullsized":"2023-10-30T14:51:28.486748+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1333,"log_dups_size":0,"ondisk_log_size":1333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:56:59.526944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084570800000000005,"stat_sum":{"num_bytes":13192668,"num_objects":479,"num_object_clones":223,"num_object_copies":958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":199,"num_read":83,"num_read_kb":8248,"num_write":1116,"num_write_kb":15001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"67'1277","reported_seq":3244,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.457130+0000","last_change":"2023-10-30T14:23:30.955845+0000","last_active":"2023-10-30T14:51:29.457130+0000","last_peered":"2023-10-30T14:51:29.457130+0000","last_clean":"2023-10-30T14:51:29.457130+0000","last_became_active":"2023-10-30T14:14:04.607683+0000","last_became_peered":"2023-10-30T14:14:04.607683+0000","last_unstale":"2023-10-30T14:51:29.457130+0000","last_undegraded":"2023-10-30T14:51:29.457130+0000","last_fullsized":"2023-10-30T14:51:29.457130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1277,"log_dups_size":0,"ondisk_log_size":1277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:41.822862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001856688,"stat_sum":{"num_bytes":21370110,"num_objects":478,"num_object_clones":220,"num_object_copies":956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":200,"num_read":74,"num_read_kb":12305,"num_write":1050,"num_write_kb":24643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'640","reported_seq":808,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.464421+0000","last_change":"2023-10-30T14:14:03.732084+0000","last_active":"2023-10-30T14:51:29.464421+0000","last_peered":"2023-10-30T14:51:29.464421+0000","last_clean":"2023-10-30T14:51:29.464421+0000","last_became_active":"2023-10-30T14:14:03.731740+0000","last_became_peered":"2023-10-30T14:14:03.731740+0000","last_unstale":"2023-10-30T14:51:29.464421+0000","last_undegraded":"2023-10-30T14:51:29.464421+0000","last_fullsized":"2023-10-30T14:51:29.464421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:34:27.462477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137426,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":149,"num_read_kb":143937,"num_write":662,"num_write_kb":70557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"67'1312","reported_seq":3279,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.631540+0000","last_change":"2023-10-30T14:23:30.776069+0000","last_active":"2023-10-30T14:51:29.631540+0000","last_peered":"2023-10-30T14:51:29.631540+0000","last_clean":"2023-10-30T14:51:29.631540+0000","last_became_active":"2023-10-30T14:14:04.621526+0000","last_became_peered":"2023-10-30T14:14:04.621526+0000","last_unstale":"2023-10-30T14:51:29.631540+0000","last_undegraded":"2023-10-30T14:51:29.631540+0000","last_fullsized":"2023-10-30T14:51:29.631540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1312,"log_dups_size":0,"ondisk_log_size":1312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:59:59.926386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047814620000000002,"stat_sum":{"num_bytes":12410003,"num_objects":492,"num_object_clones":225,"num_object_copies":984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":492,"num_whiteouts":188,"num_read":79,"num_read_kb":8245,"num_write":1121,"num_write_kb":15268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"67'1480","reported_seq":1960,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.231740+0000","last_change":"2023-10-30T14:14:03.730789+0000","last_active":"2023-10-30T14:51:30.231740+0000","last_peered":"2023-10-30T14:51:30.231740+0000","last_clean":"2023-10-30T14:51:30.231740+0000","last_became_active":"2023-10-30T14:14:03.729386+0000","last_became_peered":"2023-10-30T14:14:03.729386+0000","last_unstale":"2023-10-30T14:51:30.231740+0000","last_undegraded":"2023-10-30T14:51:30.231740+0000","last_fullsized":"2023-10-30T14:51:30.231740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1480,"log_dups_size":0,"ondisk_log_size":1480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29761637,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":457,"num_read_kb":164028,"num_write":1504,"num_write_kb":71675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'761","reported_seq":1325,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.022393+0000","last_change":"2023-10-30T14:14:03.724379+0000","last_active":"2023-10-30T14:51:25.022393+0000","last_peered":"2023-10-30T14:51:25.022393+0000","last_clean":"2023-10-30T14:51:25.022393+0000","last_became_active":"2023-10-30T14:14:03.724078+0000","last_became_peered":"2023-10-30T14:14:03.724078+0000","last_unstale":"2023-10-30T14:51:25.022393+0000","last_undegraded":"2023-10-30T14:51:25.022393+0000","last_fullsized":"2023-10-30T14:51:25.022393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":761,"log_dups_size":0,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:10:17.829899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30316903,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":547,"num_read_kb":63175,"num_write":793,"num_write_kb":47477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"67'1430","reported_seq":3365,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.433623+0000","last_change":"2023-10-30T14:23:31.068462+0000","last_active":"2023-10-30T14:51:29.433623+0000","last_peered":"2023-10-30T14:51:29.433623+0000","last_clean":"2023-10-30T14:51:29.433623+0000","last_became_active":"2023-10-30T14:14:04.597099+0000","last_became_peered":"2023-10-30T14:14:04.597099+0000","last_unstale":"2023-10-30T14:51:29.433623+0000","last_undegraded":"2023-10-30T14:51:29.433623+0000","last_fullsized":"2023-10-30T14:51:29.433623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1430,"log_dups_size":0,"ondisk_log_size":1430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:04:00.818307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033627510000000002,"stat_sum":{"num_bytes":14915943,"num_objects":541,"num_object_clones":248,"num_object_copies":1082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":541,"num_whiteouts":226,"num_read":69,"num_read_kb":8270,"num_write":1175,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"67'1579","reported_seq":1742,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.595991+0000","last_change":"2023-10-30T14:14:03.721030+0000","last_active":"2023-10-30T14:51:25.595991+0000","last_peered":"2023-10-30T14:51:25.595991+0000","last_clean":"2023-10-30T14:51:25.595991+0000","last_became_active":"2023-10-30T14:14:03.719816+0000","last_became_peered":"2023-10-30T14:14:03.719816+0000","last_unstale":"2023-10-30T14:51:25.595991+0000","last_undegraded":"2023-10-30T14:51:25.595991+0000","last_fullsized":"2023-10-30T14:51:25.595991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1579,"log_dups_size":0,"ondisk_log_size":1579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:43:50.150156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35358900,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":164,"num_read_kb":135701,"num_write":1591,"num_write_kb":77227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"67'1277","reported_seq":3242,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.437783+0000","last_change":"2023-10-30T14:23:30.771109+0000","last_active":"2023-10-30T14:51:29.437783+0000","last_peered":"2023-10-30T14:51:29.437783+0000","last_clean":"2023-10-30T14:51:29.437783+0000","last_became_active":"2023-10-30T14:14:04.599323+0000","last_became_peered":"2023-10-30T14:14:04.599323+0000","last_unstale":"2023-10-30T14:51:29.437783+0000","last_undegraded":"2023-10-30T14:51:29.437783+0000","last_fullsized":"2023-10-30T14:51:29.437783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1277,"log_dups_size":0,"ondisk_log_size":1277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:27.656629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025427000000000002,"stat_sum":{"num_bytes":25763183,"num_objects":478,"num_object_clones":224,"num_object_copies":956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":194,"num_read":75,"num_read_kb":21385,"num_write":1050,"num_write_kb":37074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"67'640","reported_seq":767,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.998751+0000","last_change":"2023-10-30T14:14:03.738694+0000","last_active":"2023-10-30T14:51:30.998751+0000","last_peered":"2023-10-30T14:51:30.998751+0000","last_clean":"2023-10-30T14:51:30.998751+0000","last_became_active":"2023-10-30T14:14:03.738132+0000","last_became_peered":"2023-10-30T14:14:03.738132+0000","last_unstale":"2023-10-30T14:51:30.998751+0000","last_undegraded":"2023-10-30T14:51:30.998751+0000","last_fullsized":"2023-10-30T14:51:30.998751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:46:59.692900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360218,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":99,"num_read_kb":53823,"num_write":691,"num_write_kb":33991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"67'1295","reported_seq":3273,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.505595+0000","last_change":"2023-10-30T14:23:30.778832+0000","last_active":"2023-10-30T14:51:28.505595+0000","last_peered":"2023-10-30T14:51:28.505595+0000","last_clean":"2023-10-30T14:51:28.505595+0000","last_became_active":"2023-10-30T14:14:04.648816+0000","last_became_peered":"2023-10-30T14:14:04.648816+0000","last_unstale":"2023-10-30T14:51:28.505595+0000","last_undegraded":"2023-10-30T14:51:28.505595+0000","last_fullsized":"2023-10-30T14:51:28.505595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:20.078590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069848840000000002,"stat_sum":{"num_bytes":12063580,"num_objects":456,"num_object_clones":213,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":456,"num_whiteouts":185,"num_read":77,"num_read_kb":8211,"num_write":1082,"num_write_kb":14557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"67'1049","reported_seq":1186,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469669+0000","last_change":"2023-10-30T14:14:03.726112+0000","last_active":"2023-10-30T14:51:29.469669+0000","last_peered":"2023-10-30T14:51:29.469669+0000","last_clean":"2023-10-30T14:51:29.469669+0000","last_became_active":"2023-10-30T14:14:03.725272+0000","last_became_peered":"2023-10-30T14:14:03.725272+0000","last_unstale":"2023-10-30T14:51:29.469669+0000","last_undegraded":"2023-10-30T14:51:29.469669+0000","last_fullsized":"2023-10-30T14:51:29.469669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1049,"log_dups_size":0,"ondisk_log_size":1049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:37:20.764294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":123,"num_read_kb":86587,"num_write":1102,"num_write_kb":54778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"67'1384","reported_seq":3308,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.501832+0000","last_change":"2023-10-30T14:23:30.776486+0000","last_active":"2023-10-30T14:51:28.501832+0000","last_peered":"2023-10-30T14:51:28.501832+0000","last_clean":"2023-10-30T14:51:28.501832+0000","last_became_active":"2023-10-30T14:14:04.609697+0000","last_became_peered":"2023-10-30T14:14:04.609697+0000","last_unstale":"2023-10-30T14:51:28.501832+0000","last_undegraded":"2023-10-30T14:51:28.501832+0000","last_fullsized":"2023-10-30T14:51:28.501832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1384,"log_dups_size":0,"ondisk_log_size":1384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:45:56.464708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043395700000000001,"stat_sum":{"num_bytes":12432092,"num_objects":547,"num_object_clones":258,"num_object_copies":1094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":547,"num_whiteouts":233,"num_read":68,"num_read_kb":4261,"num_write":1136,"num_write_kb":13809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"67'1004","reported_seq":1165,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.048045+0000","last_change":"2023-10-30T14:14:03.727871+0000","last_active":"2023-10-30T14:51:25.048045+0000","last_peered":"2023-10-30T14:51:25.048045+0000","last_clean":"2023-10-30T14:51:25.048045+0000","last_became_active":"2023-10-30T14:14:03.727422+0000","last_became_peered":"2023-10-30T14:14:03.727422+0000","last_unstale":"2023-10-30T14:51:25.048045+0000","last_undegraded":"2023-10-30T14:51:25.048045+0000","last_fullsized":"2023-10-30T14:51:25.048045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1004,"log_dups_size":0,"ondisk_log_size":1004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:54.782533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389280,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":161,"num_read_kb":61498,"num_write":1080,"num_write_kb":30288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1480","reported_seq":3358,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.465294+0000","last_change":"2023-10-30T14:23:30.957793+0000","last_active":"2023-10-30T14:51:29.465294+0000","last_peered":"2023-10-30T14:51:29.465294+0000","last_clean":"2023-10-30T14:51:29.465294+0000","last_became_active":"2023-10-30T14:14:04.645333+0000","last_became_peered":"2023-10-30T14:14:04.645333+0000","last_unstale":"2023-10-30T14:51:29.465294+0000","last_undegraded":"2023-10-30T14:51:29.465294+0000","last_fullsized":"2023-10-30T14:51:29.465294+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1480,"log_dups_size":0,"ondisk_log_size":1480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:13:02.147712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002301474,"stat_sum":{"num_bytes":21687884,"num_objects":594,"num_object_clones":315,"num_object_copies":1188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":594,"num_whiteouts":221,"num_read":78,"num_read_kb":5632,"num_write":1148,"num_write_kb":28133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"67'579","reported_seq":766,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:27.713419+0000","last_change":"2023-10-30T14:14:03.725554+0000","last_active":"2023-10-30T14:51:27.713419+0000","last_peered":"2023-10-30T14:51:27.713419+0000","last_clean":"2023-10-30T14:51:27.713419+0000","last_became_active":"2023-10-30T14:14:03.724806+0000","last_became_peered":"2023-10-30T14:14:03.724806+0000","last_unstale":"2023-10-30T14:51:27.713419+0000","last_undegraded":"2023-10-30T14:51:27.713419+0000","last_fullsized":"2023-10-30T14:51:27.713419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:37.823561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360800,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":192,"num_read_kb":131840,"num_write":657,"num_write_kb":54408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"67'1313","reported_seq":3260,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.464551+0000","last_change":"2023-10-30T14:23:31.073841+0000","last_active":"2023-10-30T14:51:29.464551+0000","last_peered":"2023-10-30T14:51:29.464551+0000","last_clean":"2023-10-30T14:51:29.464551+0000","last_became_active":"2023-10-30T14:14:04.646543+0000","last_became_peered":"2023-10-30T14:14:04.646543+0000","last_unstale":"2023-10-30T14:51:29.464551+0000","last_undegraded":"2023-10-30T14:51:29.464551+0000","last_fullsized":"2023-10-30T14:51:29.464551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1313,"log_dups_size":0,"ondisk_log_size":1313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:30:29.655398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00744474,"stat_sum":{"num_bytes":21739186,"num_objects":502,"num_object_clones":232,"num_object_copies":1004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":502,"num_whiteouts":205,"num_read":65,"num_read_kb":16406,"num_write":1092,"num_write_kb":23651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"67'683","reported_seq":858,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.629444+0000","last_change":"2023-10-30T14:14:03.717903+0000","last_active":"2023-10-30T14:51:29.629444+0000","last_peered":"2023-10-30T14:51:29.629444+0000","last_clean":"2023-10-30T14:51:29.629444+0000","last_became_active":"2023-10-30T14:14:03.716941+0000","last_became_peered":"2023-10-30T14:14:03.716941+0000","last_unstale":"2023-10-30T14:51:29.629444+0000","last_undegraded":"2023-10-30T14:51:29.629444+0000","last_fullsized":"2023-10-30T14:51:29.629444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":683,"log_dups_size":0,"ondisk_log_size":683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:58.178166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777250,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":201,"num_read_kb":82705,"num_write":761,"num_write_kb":38720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"67'1396","reported_seq":3336,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:26.996443+0000","last_change":"2023-10-30T14:23:30.775738+0000","last_active":"2023-10-30T14:51:26.996443+0000","last_peered":"2023-10-30T14:51:26.996443+0000","last_clean":"2023-10-30T14:51:26.996443+0000","last_became_active":"2023-10-30T14:14:04.641494+0000","last_became_peered":"2023-10-30T14:14:04.641494+0000","last_unstale":"2023-10-30T14:51:26.996443+0000","last_undegraded":"2023-10-30T14:51:26.996443+0000","last_fullsized":"2023-10-30T14:51:26.996443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1396,"log_dups_size":0,"ondisk_log_size":1396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:44.138968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039664019999999999,"stat_sum":{"num_bytes":26403621,"num_objects":532,"num_object_clones":244,"num_object_copies":1064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":532,"num_whiteouts":206,"num_read":69,"num_read_kb":14198,"num_write":1171,"num_write_kb":36700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"67'300","reported_seq":421,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.447336+0000","last_change":"2023-10-30T14:14:03.729626+0000","last_active":"2023-10-30T14:51:29.447336+0000","last_peered":"2023-10-30T14:51:29.447336+0000","last_clean":"2023-10-30T14:51:29.447336+0000","last_became_active":"2023-10-30T14:14:03.728053+0000","last_became_peered":"2023-10-30T14:14:03.728053+0000","last_unstale":"2023-10-30T14:51:29.447336+0000","last_undegraded":"2023-10-30T14:51:29.447336+0000","last_fullsized":"2023-10-30T14:51:29.447336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:44:53.362066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":90,"num_read_kb":49506,"num_write":340,"num_write_kb":25387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1391","reported_seq":3353,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:27.468452+0000","last_change":"2023-10-30T14:23:31.078295+0000","last_active":"2023-10-30T14:51:27.468452+0000","last_peered":"2023-10-30T14:51:27.468452+0000","last_clean":"2023-10-30T14:51:27.468452+0000","last_became_active":"2023-10-30T14:14:04.649896+0000","last_became_peered":"2023-10-30T14:14:04.649896+0000","last_unstale":"2023-10-30T14:51:27.468452+0000","last_undegraded":"2023-10-30T14:51:27.468452+0000","last_fullsized":"2023-10-30T14:51:27.468452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1391,"log_dups_size":0,"ondisk_log_size":1391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:07:44.011495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010792094,"stat_sum":{"num_bytes":9382763,"num_objects":511,"num_object_clones":240,"num_object_copies":1022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":210,"num_read":88,"num_read_kb":2154,"num_write":1160,"num_write_kb":10817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"67'712","reported_seq":848,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.468850+0000","last_change":"2023-10-30T14:14:03.736824+0000","last_active":"2023-10-30T14:51:29.468850+0000","last_peered":"2023-10-30T14:51:29.468850+0000","last_clean":"2023-10-30T14:51:29.468850+0000","last_became_active":"2023-10-30T14:14:03.736155+0000","last_became_peered":"2023-10-30T14:14:03.736155+0000","last_unstale":"2023-10-30T14:51:29.468850+0000","last_undegraded":"2023-10-30T14:51:29.468850+0000","last_fullsized":"2023-10-30T14:51:29.468850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:35.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":104,"num_read_kb":67382,"num_write":738,"num_write_kb":42054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"67'1257","reported_seq":3230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.475777+0000","last_change":"2023-10-30T14:23:31.076487+0000","last_active":"2023-10-30T14:51:28.475777+0000","last_peered":"2023-10-30T14:51:28.475777+0000","last_clean":"2023-10-30T14:51:28.475777+0000","last_became_active":"2023-10-30T14:14:04.614328+0000","last_became_peered":"2023-10-30T14:14:04.614328+0000","last_unstale":"2023-10-30T14:51:28.475777+0000","last_undegraded":"2023-10-30T14:51:28.475777+0000","last_fullsized":"2023-10-30T14:51:28.475777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1257,"log_dups_size":0,"ondisk_log_size":1257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:04:54.384597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010029286,"stat_sum":{"num_bytes":9023665,"num_objects":474,"num_object_clones":219,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":474,"num_whiteouts":189,"num_read":77,"num_read_kb":4169,"num_write":1052,"num_write_kb":11363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"67'1214","reported_seq":1398,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.435186+0000","last_change":"2023-10-30T14:14:03.700690+0000","last_active":"2023-10-30T14:51:29.435186+0000","last_peered":"2023-10-30T14:51:29.435186+0000","last_clean":"2023-10-30T14:51:29.435186+0000","last_became_active":"2023-10-30T14:14:03.700352+0000","last_became_peered":"2023-10-30T14:14:03.700352+0000","last_unstale":"2023-10-30T14:51:29.435186+0000","last_undegraded":"2023-10-30T14:51:29.435186+0000","last_fullsized":"2023-10-30T14:51:29.435186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1214,"log_dups_size":0,"ondisk_log_size":1214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:03:01.103359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":252,"num_read_kb":55463,"num_write":1288,"num_write_kb":26279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"67'1354","reported_seq":3315,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.447744+0000","last_change":"2023-10-30T14:23:31.079218+0000","last_active":"2023-10-30T14:51:28.447744+0000","last_peered":"2023-10-30T14:51:28.447744+0000","last_clean":"2023-10-30T14:51:28.447744+0000","last_became_active":"2023-10-30T14:14:04.646639+0000","last_became_peered":"2023-10-30T14:14:04.646639+0000","last_unstale":"2023-10-30T14:51:28.447744+0000","last_undegraded":"2023-10-30T14:51:28.447744+0000","last_fullsized":"2023-10-30T14:51:28.447744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1354,"log_dups_size":0,"ondisk_log_size":1354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:32.477580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010379724,"stat_sum":{"num_bytes":13446392,"num_objects":502,"num_object_clones":231,"num_object_copies":1004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":502,"num_whiteouts":206,"num_read":77,"num_read_kb":12340,"num_write":1116,"num_write_kb":24186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'643","reported_seq":768,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.271430+0000","last_change":"2023-10-30T14:14:03.710717+0000","last_active":"2023-10-30T14:51:25.271430+0000","last_peered":"2023-10-30T14:51:25.271430+0000","last_clean":"2023-10-30T14:51:25.271430+0000","last_became_active":"2023-10-30T14:14:03.707950+0000","last_became_peered":"2023-10-30T14:14:03.707950+0000","last_unstale":"2023-10-30T14:51:25.271430+0000","last_undegraded":"2023-10-30T14:51:25.271430+0000","last_fullsized":"2023-10-30T14:51:25.271430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:29:41.232902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971542,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":82,"num_read_kb":57693,"num_write":882,"num_write_kb":42351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"67'1239","reported_seq":3190,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.012771+0000","last_change":"2023-10-30T14:23:31.080966+0000","last_active":"2023-10-30T14:51:25.012771+0000","last_peered":"2023-10-30T14:51:25.012771+0000","last_clean":"2023-10-30T14:51:25.012771+0000","last_became_active":"2023-10-30T14:14:04.644761+0000","last_became_peered":"2023-10-30T14:14:04.644761+0000","last_unstale":"2023-10-30T14:51:25.012771+0000","last_undegraded":"2023-10-30T14:51:25.012771+0000","last_fullsized":"2023-10-30T14:51:25.012771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1239,"log_dups_size":0,"ondisk_log_size":1239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:55:03.102431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011171970999999999,"stat_sum":{"num_bytes":12914479,"num_objects":495,"num_object_clones":226,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":208,"num_read":62,"num_read_kb":4133,"num_write":998,"num_write_kb":18863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'535","reported_seq":666,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.248852+0000","last_change":"2023-10-30T14:14:03.705567+0000","last_active":"2023-10-30T14:51:30.248852+0000","last_peered":"2023-10-30T14:51:30.248852+0000","last_clean":"2023-10-30T14:51:30.248852+0000","last_became_active":"2023-10-30T14:14:03.705186+0000","last_became_peered":"2023-10-30T14:14:03.705186+0000","last_unstale":"2023-10-30T14:51:30.248852+0000","last_undegraded":"2023-10-30T14:51:30.248852+0000","last_fullsized":"2023-10-30T14:51:30.248852+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":535,"log_dups_size":0,"ondisk_log_size":535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:14.609243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":102,"num_read_kb":90592,"num_write":573,"num_write_kb":50573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"67'1299","reported_seq":3247,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.473089+0000","last_change":"2023-10-30T14:23:31.083260+0000","last_active":"2023-10-30T14:51:29.473089+0000","last_peered":"2023-10-30T14:51:29.473089+0000","last_clean":"2023-10-30T14:51:29.473089+0000","last_became_active":"2023-10-30T14:14:04.646785+0000","last_became_peered":"2023-10-30T14:14:04.646785+0000","last_unstale":"2023-10-30T14:51:29.473089+0000","last_undegraded":"2023-10-30T14:51:29.473089+0000","last_fullsized":"2023-10-30T14:51:29.473089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1299,"log_dups_size":0,"ondisk_log_size":1299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:52:21.536104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010111184,"stat_sum":{"num_bytes":17345938,"num_objects":504,"num_object_clones":239,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":504,"num_whiteouts":216,"num_read":72,"num_read_kb":10698,"num_write":1034,"num_write_kb":23411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"67'598","reported_seq":748,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.425928+0000","last_change":"2023-10-30T14:14:03.693349+0000","last_active":"2023-10-30T14:51:29.425928+0000","last_peered":"2023-10-30T14:51:29.425928+0000","last_clean":"2023-10-30T14:51:29.425928+0000","last_became_active":"2023-10-30T14:14:03.692966+0000","last_became_peered":"2023-10-30T14:14:03.692966+0000","last_unstale":"2023-10-30T14:51:29.425928+0000","last_undegraded":"2023-10-30T14:51:29.425928+0000","last_fullsized":"2023-10-30T14:51:29.425928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":598,"log_dups_size":0,"ondisk_log_size":598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:30:28.800214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16780852,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":126,"num_read_kb":110211,"num_write":638,"num_write_kb":50143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'1328","reported_seq":3304,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.461147+0000","last_change":"2023-10-30T14:23:30.778622+0000","last_active":"2023-10-30T14:51:29.461147+0000","last_peered":"2023-10-30T14:51:29.461147+0000","last_clean":"2023-10-30T14:51:29.461147+0000","last_became_active":"2023-10-30T14:14:04.615925+0000","last_became_peered":"2023-10-30T14:14:04.615925+0000","last_unstale":"2023-10-30T14:51:29.461147+0000","last_undegraded":"2023-10-30T14:51:29.461147+0000","last_fullsized":"2023-10-30T14:51:29.461147+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1328,"log_dups_size":0,"ondisk_log_size":1328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:54.760495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068317569999999999,"stat_sum":{"num_bytes":4298895,"num_objects":483,"num_object_clones":223,"num_object_copies":966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":483,"num_whiteouts":204,"num_read":84,"num_read_kb":56,"num_write":1103,"num_write_kb":8864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"67'667","reported_seq":805,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.244959+0000","last_change":"2023-10-30T14:14:03.727659+0000","last_active":"2023-10-30T14:51:25.244959+0000","last_peered":"2023-10-30T14:51:25.244959+0000","last_clean":"2023-10-30T14:51:25.244959+0000","last_became_active":"2023-10-30T14:14:03.727409+0000","last_became_peered":"2023-10-30T14:14:03.727409+0000","last_unstale":"2023-10-30T14:51:25.244959+0000","last_undegraded":"2023-10-30T14:51:25.244959+0000","last_fullsized":"2023-10-30T14:51:25.244959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:24.210635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166406,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":118,"num_read_kb":66012,"num_write":697,"num_write_kb":50302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1282","reported_seq":3257,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.467914+0000","last_change":"2023-10-30T14:23:31.077576+0000","last_active":"2023-10-30T14:51:29.467914+0000","last_peered":"2023-10-30T14:51:29.467914+0000","last_clean":"2023-10-30T14:51:29.467914+0000","last_became_active":"2023-10-30T14:14:45.818032+0000","last_became_peered":"2023-10-30T14:14:45.818032+0000","last_unstale":"2023-10-30T14:51:29.467914+0000","last_undegraded":"2023-10-30T14:51:29.467914+0000","last_fullsized":"2023-10-30T14:51:29.467914+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1282,"log_dups_size":0,"ondisk_log_size":1282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:54:03.288491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010655906,"stat_sum":{"num_bytes":23795405,"num_objects":481,"num_object_clones":221,"num_object_copies":962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":199,"num_read":74,"num_read_kb":13429,"num_write":1070,"num_write_kb":29341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":229,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T14:49:57.449030+0000","last_change":"2023-10-30T14:13:54.251572+0000","last_active":"2023-10-30T14:49:57.449030+0000","last_peered":"2023-10-30T14:49:57.449030+0000","last_clean":"2023-10-30T14:49:57.449030+0000","last_became_active":"2023-10-30T14:13:54.250907+0000","last_became_peered":"2023-10-30T14:13:54.250907+0000","last_unstale":"2023-10-30T14:49:57.449030+0000","last_undegraded":"2023-10-30T14:49:57.449030+0000","last_fullsized":"2023-10-30T14:49:57.449030+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_clean_scrub_stamp":"2023-10-30T14:13:53.220267+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:44.590852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"67'724","reported_seq":867,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.024538+0000","last_change":"2023-10-30T14:14:03.729034+0000","last_active":"2023-10-30T14:51:25.024538+0000","last_peered":"2023-10-30T14:51:25.024538+0000","last_clean":"2023-10-30T14:51:25.024538+0000","last_became_active":"2023-10-30T14:14:03.728293+0000","last_became_peered":"2023-10-30T14:14:03.728293+0000","last_unstale":"2023-10-30T14:51:25.024538+0000","last_undegraded":"2023-10-30T14:51:25.024538+0000","last_fullsized":"2023-10-30T14:51:25.024538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":724,"log_dups_size":0,"ondisk_log_size":724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165954,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":121,"num_read_kb":78313,"num_write":794,"num_write_kb":42213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"67'1244","reported_seq":3209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.448701+0000","last_change":"2023-10-30T14:23:31.067976+0000","last_active":"2023-10-30T14:51:29.448701+0000","last_peered":"2023-10-30T14:51:29.448701+0000","last_clean":"2023-10-30T14:51:29.448701+0000","last_became_active":"2023-10-30T14:14:46.451620+0000","last_became_peered":"2023-10-30T14:14:46.451620+0000","last_unstale":"2023-10-30T14:51:29.448701+0000","last_undegraded":"2023-10-30T14:51:29.448701+0000","last_fullsized":"2023-10-30T14:51:29.448701+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1244,"log_dups_size":0,"ondisk_log_size":1244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:10.312892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034002070000000001,"stat_sum":{"num_bytes":12494676,"num_objects":466,"num_object_clones":213,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":182,"num_read":65,"num_read_kb":8203,"num_write":1045,"num_write_kb":18288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"67'823","reported_seq":1009,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.517181+0000","last_change":"2023-10-30T14:14:03.732909+0000","last_active":"2023-10-30T14:51:29.517181+0000","last_peered":"2023-10-30T14:51:29.517181+0000","last_clean":"2023-10-30T14:51:29.517181+0000","last_became_active":"2023-10-30T14:14:03.732199+0000","last_became_peered":"2023-10-30T14:14:03.732199+0000","last_unstale":"2023-10-30T14:51:29.517181+0000","last_undegraded":"2023-10-30T14:51:29.517181+0000","last_fullsized":"2023-10-30T14:51:29.517181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":823,"log_dups_size":0,"ondisk_log_size":823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:38.117750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":198,"num_read_kb":136195,"num_write":895,"num_write_kb":59080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"67'1255","reported_seq":3217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.505461+0000","last_change":"2023-10-30T14:23:30.775385+0000","last_active":"2023-10-30T14:51:28.505461+0000","last_peered":"2023-10-30T14:51:28.505461+0000","last_clean":"2023-10-30T14:51:28.505461+0000","last_became_active":"2023-10-30T14:14:04.619990+0000","last_became_peered":"2023-10-30T14:14:04.619990+0000","last_unstale":"2023-10-30T14:51:28.505461+0000","last_undegraded":"2023-10-30T14:51:28.505461+0000","last_fullsized":"2023-10-30T14:51:28.505461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1255,"log_dups_size":0,"ondisk_log_size":1255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:42.711461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045963250000000001,"stat_sum":{"num_bytes":8263423,"num_objects":470,"num_object_clones":217,"num_object_copies":940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":194,"num_read":63,"num_read_kb":1576,"num_write":1033,"num_write_kb":10839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"67'1750","reported_seq":2853,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.563109+0000","last_change":"2023-10-30T14:14:03.702862+0000","last_active":"2023-10-30T14:51:29.563109+0000","last_peered":"2023-10-30T14:51:29.563109+0000","last_clean":"2023-10-30T14:51:29.563109+0000","last_became_active":"2023-10-30T14:14:03.702513+0000","last_became_peered":"2023-10-30T14:14:03.702513+0000","last_unstale":"2023-10-30T14:51:29.563109+0000","last_undegraded":"2023-10-30T14:51:29.563109+0000","last_fullsized":"2023-10-30T14:51:29.563109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1750,"log_dups_size":0,"ondisk_log_size":1750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:25.499390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27766724,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":1109,"num_read_kb":113686,"num_write":1839,"num_write_kb":62085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"67'1316","reported_seq":3268,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.468492+0000","last_change":"2023-10-30T14:23:30.770719+0000","last_active":"2023-10-30T14:51:28.468492+0000","last_peered":"2023-10-30T14:51:28.468492+0000","last_clean":"2023-10-30T14:51:28.468492+0000","last_became_active":"2023-10-30T14:14:04.589087+0000","last_became_peered":"2023-10-30T14:14:04.589087+0000","last_unstale":"2023-10-30T14:51:28.468492+0000","last_undegraded":"2023-10-30T14:51:28.468492+0000","last_fullsized":"2023-10-30T14:51:28.468492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1316,"log_dups_size":0,"ondisk_log_size":1316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:44:54.038742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022871390000000001,"stat_sum":{"num_bytes":30600386,"num_objects":508,"num_object_clones":243,"num_object_copies":1016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":508,"num_whiteouts":221,"num_read":78,"num_read_kb":20511,"num_write":1060,"num_write_kb":36437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"67'633","reported_seq":782,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.234880+0000","last_change":"2023-10-30T14:14:03.730048+0000","last_active":"2023-10-30T14:51:30.234880+0000","last_peered":"2023-10-30T14:51:30.234880+0000","last_clean":"2023-10-30T14:51:30.234880+0000","last_became_active":"2023-10-30T14:14:03.728925+0000","last_became_peered":"2023-10-30T14:14:03.728925+0000","last_unstale":"2023-10-30T14:51:30.234880+0000","last_undegraded":"2023-10-30T14:51:30.234880+0000","last_fullsized":"2023-10-30T14:51:30.234880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":633,"log_dups_size":0,"ondisk_log_size":633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:32:05.056039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":120,"num_read_kb":131794,"num_write":657,"num_write_kb":62419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"67'1355","reported_seq":3305,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.495764+0000","last_change":"2023-10-30T14:23:30.956956+0000","last_active":"2023-10-30T14:51:28.495764+0000","last_peered":"2023-10-30T14:51:28.495764+0000","last_clean":"2023-10-30T14:51:28.495764+0000","last_became_active":"2023-10-30T14:14:04.607377+0000","last_became_peered":"2023-10-30T14:14:04.607377+0000","last_unstale":"2023-10-30T14:51:28.495764+0000","last_undegraded":"2023-10-30T14:51:28.495764+0000","last_fullsized":"2023-10-30T14:51:28.495764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1355,"log_dups_size":0,"ondisk_log_size":1355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:58:23.634836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022021179999999999,"stat_sum":{"num_bytes":18058258,"num_objects":500,"num_object_clones":231,"num_object_copies":1000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":500,"num_whiteouts":206,"num_read":65,"num_read_kb":9172,"num_write":1119,"num_write_kb":20820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"67'880","reported_seq":1017,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.230091+0000","last_change":"2023-10-30T14:14:03.693753+0000","last_active":"2023-10-30T14:51:30.230091+0000","last_peered":"2023-10-30T14:51:30.230091+0000","last_clean":"2023-10-30T14:51:30.230091+0000","last_became_active":"2023-10-30T14:14:03.692621+0000","last_became_peered":"2023-10-30T14:14:03.692621+0000","last_unstale":"2023-10-30T14:51:30.230091+0000","last_undegraded":"2023-10-30T14:51:30.230091+0000","last_fullsized":"2023-10-30T14:51:30.230091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:32:28.577584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":107,"num_read_kb":66236,"num_write":920,"num_write_kb":58565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'1338","reported_seq":3267,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.493889+0000","last_change":"2023-10-30T14:23:30.780258+0000","last_active":"2023-10-30T14:51:28.493889+0000","last_peered":"2023-10-30T14:51:28.493889+0000","last_clean":"2023-10-30T14:51:28.493889+0000","last_became_active":"2023-10-30T14:14:04.613780+0000","last_became_peered":"2023-10-30T14:14:04.613780+0000","last_unstale":"2023-10-30T14:51:28.493889+0000","last_undegraded":"2023-10-30T14:51:28.493889+0000","last_fullsized":"2023-10-30T14:51:28.493889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1338,"log_dups_size":0,"ondisk_log_size":1338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:07:05.556883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065285270000000001,"stat_sum":{"num_bytes":12665534,"num_objects":519,"num_object_clones":250,"num_object_copies":1038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":519,"num_whiteouts":228,"num_read":63,"num_read_kb":8216,"num_write":1076,"num_write_kb":17979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"67'721","reported_seq":883,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.249637+0000","last_change":"2023-10-30T14:14:03.740213+0000","last_active":"2023-10-30T14:51:30.249637+0000","last_peered":"2023-10-30T14:51:30.249637+0000","last_clean":"2023-10-30T14:51:30.249637+0000","last_became_active":"2023-10-30T14:14:03.739650+0000","last_became_peered":"2023-10-30T14:14:03.739650+0000","last_unstale":"2023-10-30T14:51:30.249637+0000","last_undegraded":"2023-10-30T14:51:30.249637+0000","last_fullsized":"2023-10-30T14:51:30.249637+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":721,"log_dups_size":0,"ondisk_log_size":721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:54.120757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777238,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":143,"num_read_kb":109192,"num_write":780,"num_write_kb":46441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"67'1318","reported_seq":3274,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.425113+0000","last_change":"2023-10-30T14:23:30.771719+0000","last_active":"2023-10-30T14:51:29.425113+0000","last_peered":"2023-10-30T14:51:29.425113+0000","last_clean":"2023-10-30T14:51:29.425113+0000","last_became_active":"2023-10-30T14:14:04.590138+0000","last_became_peered":"2023-10-30T14:14:04.590138+0000","last_unstale":"2023-10-30T14:51:29.425113+0000","last_undegraded":"2023-10-30T14:51:29.425113+0000","last_fullsized":"2023-10-30T14:51:29.425113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1318,"log_dups_size":0,"ondisk_log_size":1318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:49.779956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031364370000000002,"stat_sum":{"num_bytes":9073572,"num_objects":494,"num_object_clones":232,"num_object_copies":988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":494,"num_whiteouts":206,"num_read":72,"num_read_kb":4145,"num_write":1090,"num_write_kb":19754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"67'771","reported_seq":913,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469946+0000","last_change":"2023-10-30T14:14:03.721450+0000","last_active":"2023-10-30T14:51:29.469946+0000","last_peered":"2023-10-30T14:51:29.469946+0000","last_clean":"2023-10-30T14:51:29.469946+0000","last_became_active":"2023-10-30T14:14:03.721234+0000","last_became_peered":"2023-10-30T14:14:03.721234+0000","last_unstale":"2023-10-30T14:51:29.469946+0000","last_undegraded":"2023-10-30T14:51:29.469946+0000","last_fullsized":"2023-10-30T14:51:29.469946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":771,"log_dups_size":0,"ondisk_log_size":771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:26.959676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":136,"num_read_kb":37589,"num_write":1311,"num_write_kb":40350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"67'1259","reported_seq":3214,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.476582+0000","last_change":"2023-10-30T14:23:31.070447+0000","last_active":"2023-10-30T14:51:28.476582+0000","last_peered":"2023-10-30T14:51:28.476582+0000","last_clean":"2023-10-30T14:51:28.476582+0000","last_became_active":"2023-10-30T14:14:04.633643+0000","last_became_peered":"2023-10-30T14:14:04.633643+0000","last_unstale":"2023-10-30T14:51:28.476582+0000","last_undegraded":"2023-10-30T14:51:28.476582+0000","last_fullsized":"2023-10-30T14:51:28.476582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1259,"log_dups_size":0,"ondisk_log_size":1259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:53:41.463549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048916599999999999,"stat_sum":{"num_bytes":9340734,"num_objects":487,"num_object_clones":228,"num_object_copies":974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":205,"num_read":67,"num_read_kb":4121,"num_write":1045,"num_write_kb":15590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"67'440","reported_seq":577,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.705366+0000","last_change":"2023-10-30T14:14:03.738557+0000","last_active":"2023-10-30T14:51:29.705366+0000","last_peered":"2023-10-30T14:51:29.705366+0000","last_clean":"2023-10-30T14:51:29.705366+0000","last_became_active":"2023-10-30T14:14:03.737801+0000","last_became_peered":"2023-10-30T14:14:03.737801+0000","last_unstale":"2023-10-30T14:51:29.705366+0000","last_undegraded":"2023-10-30T14:51:29.705366+0000","last_fullsized":"2023-10-30T14:51:29.705366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":440,"log_dups_size":0,"ondisk_log_size":440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:26.938343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":132,"num_read_kb":45549,"num_write":530,"num_write_kb":26044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"67'1244","reported_seq":3217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.425279+0000","last_change":"2023-10-30T14:23:30.776079+0000","last_active":"2023-10-30T14:51:28.425279+0000","last_peered":"2023-10-30T14:51:28.425279+0000","last_clean":"2023-10-30T14:51:28.425279+0000","last_became_active":"2023-10-30T14:14:04.588826+0000","last_became_peered":"2023-10-30T14:14:04.588826+0000","last_unstale":"2023-10-30T14:51:28.425279+0000","last_undegraded":"2023-10-30T14:51:28.425279+0000","last_fullsized":"2023-10-30T14:51:28.425279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1244,"log_dups_size":0,"ondisk_log_size":1244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:27.236524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040592670000000001,"stat_sum":{"num_bytes":8246712,"num_objects":450,"num_object_clones":210,"num_object_copies":900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":450,"num_whiteouts":181,"num_read":67,"num_read_kb":4125,"num_write":1046,"num_write_kb":10688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"67'429","reported_seq":542,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.467848+0000","last_change":"2023-10-30T14:14:03.733836+0000","last_active":"2023-10-30T14:51:29.467848+0000","last_peered":"2023-10-30T14:51:29.467848+0000","last_clean":"2023-10-30T14:51:29.467848+0000","last_became_active":"2023-10-30T14:14:03.732912+0000","last_became_peered":"2023-10-30T14:14:03.732912+0000","last_unstale":"2023-10-30T14:51:29.467848+0000","last_undegraded":"2023-10-30T14:51:29.467848+0000","last_fullsized":"2023-10-30T14:51:29.467848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:50.332738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":78,"num_read_kb":41242,"num_write":426,"num_write_kb":21091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"67'1279","reported_seq":3248,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.450800+0000","last_change":"2023-10-30T14:23:30.957473+0000","last_active":"2023-10-30T14:51:29.450800+0000","last_peered":"2023-10-30T14:51:29.450800+0000","last_clean":"2023-10-30T14:51:29.450800+0000","last_became_active":"2023-10-30T14:14:04.609835+0000","last_became_peered":"2023-10-30T14:14:04.609835+0000","last_unstale":"2023-10-30T14:51:29.450800+0000","last_undegraded":"2023-10-30T14:51:29.450800+0000","last_fullsized":"2023-10-30T14:51:29.450800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1279,"log_dups_size":0,"ondisk_log_size":1279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:29:36.272312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023958389999999999,"stat_sum":{"num_bytes":25809405,"num_objects":464,"num_object_clones":214,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":464,"num_whiteouts":191,"num_read":67,"num_read_kb":20495,"num_write":1061,"num_write_kb":27581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"67'867","reported_seq":1008,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.048969+0000","last_change":"2023-10-30T14:14:03.732539+0000","last_active":"2023-10-30T14:51:25.048969+0000","last_peered":"2023-10-30T14:51:25.048969+0000","last_clean":"2023-10-30T14:51:25.048969+0000","last_became_active":"2023-10-30T14:14:03.732265+0000","last_became_peered":"2023-10-30T14:14:03.732265+0000","last_unstale":"2023-10-30T14:51:25.048969+0000","last_undegraded":"2023-10-30T14:51:25.048969+0000","last_fullsized":"2023-10-30T14:51:25.048969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":867,"log_dups_size":0,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:02:00.929612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":128,"num_read_kb":49868,"num_write":925,"num_write_kb":26024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"67'1338","reported_seq":3317,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.437725+0000","last_change":"2023-10-30T14:23:31.069537+0000","last_active":"2023-10-30T14:51:29.437725+0000","last_peered":"2023-10-30T14:51:29.437725+0000","last_clean":"2023-10-30T14:51:29.437725+0000","last_became_active":"2023-10-30T14:14:04.594998+0000","last_became_peered":"2023-10-30T14:14:04.594998+0000","last_unstale":"2023-10-30T14:51:29.437725+0000","last_undegraded":"2023-10-30T14:51:29.437725+0000","last_fullsized":"2023-10-30T14:51:29.437725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1338,"log_dups_size":0,"ondisk_log_size":1338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:29:28.794439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00432118,"stat_sum":{"num_bytes":12291745,"num_objects":459,"num_object_clones":215,"num_object_copies":918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":459,"num_whiteouts":187,"num_read":78,"num_read_kb":8212,"num_write":1121,"num_write_kb":18507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'816","reported_seq":962,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.428311+0000","last_change":"2023-10-30T14:14:03.703744+0000","last_active":"2023-10-30T14:51:29.428311+0000","last_peered":"2023-10-30T14:51:29.428311+0000","last_clean":"2023-10-30T14:51:29.428311+0000","last_became_active":"2023-10-30T14:14:03.701759+0000","last_became_peered":"2023-10-30T14:14:03.701759+0000","last_unstale":"2023-10-30T14:51:29.428311+0000","last_undegraded":"2023-10-30T14:51:29.428311+0000","last_fullsized":"2023-10-30T14:51:29.428311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":816,"log_dups_size":0,"ondisk_log_size":816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:22:27.948007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":122,"num_read_kb":94942,"num_write":1181,"num_write_kb":51316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"67'1224","reported_seq":3205,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.415105+0000","last_change":"2023-10-30T14:23:31.071315+0000","last_active":"2023-10-30T14:51:29.415105+0000","last_peered":"2023-10-30T14:51:29.415105+0000","last_clean":"2023-10-30T14:51:29.415105+0000","last_became_active":"2023-10-30T14:14:04.600100+0000","last_became_peered":"2023-10-30T14:14:04.600100+0000","last_unstale":"2023-10-30T14:51:29.415105+0000","last_undegraded":"2023-10-30T14:51:29.415105+0000","last_fullsized":"2023-10-30T14:51:29.415105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1224,"log_dups_size":0,"ondisk_log_size":1224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:30.476350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052227630000000001,"stat_sum":{"num_bytes":16868959,"num_objects":442,"num_object_clones":207,"num_object_copies":884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":442,"num_whiteouts":180,"num_read":73,"num_read_kb":11653,"num_write":1014,"num_write_kb":18882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"67'773","reported_seq":906,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.997876+0000","last_change":"2023-10-30T14:14:03.741268+0000","last_active":"2023-10-30T14:51:30.997876+0000","last_peered":"2023-10-30T14:51:30.997876+0000","last_clean":"2023-10-30T14:51:30.997876+0000","last_became_active":"2023-10-30T14:14:03.740648+0000","last_became_peered":"2023-10-30T14:14:03.740648+0000","last_unstale":"2023-10-30T14:51:30.997876+0000","last_undegraded":"2023-10-30T14:51:30.997876+0000","last_fullsized":"2023-10-30T14:51:30.997876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:30.799148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":98,"num_read_kb":107365,"num_write":798,"num_write_kb":50458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"67'1285","reported_seq":3266,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469362+0000","last_change":"2023-10-30T14:23:31.073589+0000","last_active":"2023-10-30T14:51:29.469362+0000","last_peered":"2023-10-30T14:51:29.469362+0000","last_clean":"2023-10-30T14:51:29.469362+0000","last_became_active":"2023-10-30T14:14:04.622921+0000","last_became_peered":"2023-10-30T14:14:04.622921+0000","last_unstale":"2023-10-30T14:51:29.469362+0000","last_undegraded":"2023-10-30T14:51:29.469362+0000","last_fullsized":"2023-10-30T14:51:29.469362+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1285,"log_dups_size":0,"ondisk_log_size":1285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:04:46.396711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0071219910000000003,"stat_sum":{"num_bytes":20705836,"num_objects":462,"num_object_clones":210,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":462,"num_whiteouts":184,"num_read":75,"num_read_kb":12323,"num_write":1096,"num_write_kb":26452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"67'1315","reported_seq":3269,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.461591+0000","last_change":"2023-10-30T14:23:30.779989+0000","last_active":"2023-10-30T14:51:29.461591+0000","last_peered":"2023-10-30T14:51:29.461591+0000","last_clean":"2023-10-30T14:51:29.461591+0000","last_became_active":"2023-10-30T14:14:04.621885+0000","last_became_peered":"2023-10-30T14:14:04.621885+0000","last_unstale":"2023-10-30T14:51:29.461591+0000","last_undegraded":"2023-10-30T14:51:29.461591+0000","last_fullsized":"2023-10-30T14:51:29.461591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1315,"log_dups_size":0,"ondisk_log_size":1315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:59:52.839990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067851329999999996,"stat_sum":{"num_bytes":13415563,"num_objects":482,"num_object_clones":227,"num_object_copies":964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":190,"num_read":68,"num_read_kb":8221,"num_write":1118,"num_write_kb":20704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"67'466","reported_seq":615,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:32.456573+0000","last_change":"2023-10-30T14:14:03.740897+0000","last_active":"2023-10-30T14:51:32.456573+0000","last_peered":"2023-10-30T14:51:32.456573+0000","last_clean":"2023-10-30T14:51:32.456573+0000","last_became_active":"2023-10-30T14:14:03.740247+0000","last_became_peered":"2023-10-30T14:14:03.740247+0000","last_unstale":"2023-10-30T14:51:32.456573+0000","last_undegraded":"2023-10-30T14:51:32.456573+0000","last_fullsized":"2023-10-30T14:51:32.456573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:29.601978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30406549,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":149,"num_read_kb":82851,"num_write":524,"num_write_kb":39214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'1397","reported_seq":3340,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.460907+0000","last_change":"2023-10-30T14:23:30.776236+0000","last_active":"2023-10-30T14:51:29.460907+0000","last_peered":"2023-10-30T14:51:29.460907+0000","last_clean":"2023-10-30T14:51:29.460907+0000","last_became_active":"2023-10-30T14:14:04.623638+0000","last_became_peered":"2023-10-30T14:14:04.623638+0000","last_unstale":"2023-10-30T14:51:29.460907+0000","last_undegraded":"2023-10-30T14:51:29.460907+0000","last_fullsized":"2023-10-30T14:51:29.460907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1397,"log_dups_size":0,"ondisk_log_size":1397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:37:06.313676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049828509999999999,"stat_sum":{"num_bytes":25425754,"num_objects":536,"num_object_clones":247,"num_object_copies":1072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":221,"num_read":77,"num_read_kb":19614,"num_write":1171,"num_write_kb":31740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"67'1024","reported_seq":1176,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.475951+0000","last_change":"2023-10-30T14:14:03.733233+0000","last_active":"2023-10-30T14:51:29.475951+0000","last_peered":"2023-10-30T14:51:29.475951+0000","last_clean":"2023-10-30T14:51:29.475951+0000","last_became_active":"2023-10-30T14:14:03.732538+0000","last_became_peered":"2023-10-30T14:14:03.732538+0000","last_unstale":"2023-10-30T14:51:29.475951+0000","last_undegraded":"2023-10-30T14:51:29.475951+0000","last_fullsized":"2023-10-30T14:51:29.475951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1024,"log_dups_size":0,"ondisk_log_size":1024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:25.696173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":137,"num_read_kb":150323,"num_write":1064,"num_write_kb":62931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1272","reported_seq":3223,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.627078+0000","last_change":"2023-10-30T14:23:30.776451+0000","last_active":"2023-10-30T14:51:29.627078+0000","last_peered":"2023-10-30T14:51:29.627078+0000","last_clean":"2023-10-30T14:51:29.627078+0000","last_became_active":"2023-10-30T14:14:04.611773+0000","last_became_peered":"2023-10-30T14:14:04.611773+0000","last_unstale":"2023-10-30T14:51:29.627078+0000","last_undegraded":"2023-10-30T14:51:29.627078+0000","last_fullsized":"2023-10-30T14:51:29.627078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1272,"log_dups_size":0,"ondisk_log_size":1272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:55.244396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044924040000000002,"stat_sum":{"num_bytes":26143587,"num_objects":507,"num_object_clones":231,"num_object_copies":1014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":507,"num_whiteouts":205,"num_read":66,"num_read_kb":19602,"num_write":1063,"num_write_kb":27312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"67'731","reported_seq":872,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.995935+0000","last_change":"2023-10-30T14:14:03.703988+0000","last_active":"2023-10-30T14:51:30.995935+0000","last_peered":"2023-10-30T14:51:30.995935+0000","last_clean":"2023-10-30T14:51:30.995935+0000","last_became_active":"2023-10-30T14:14:03.703691+0000","last_became_peered":"2023-10-30T14:14:03.703691+0000","last_unstale":"2023-10-30T14:51:30.995935+0000","last_undegraded":"2023-10-30T14:51:30.995935+0000","last_fullsized":"2023-10-30T14:51:30.995935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":731,"log_dups_size":0,"ondisk_log_size":731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:34.538541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":102,"num_read_kb":111696,"num_write":760,"num_write_kb":58649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"67'1327","reported_seq":3261,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.468906+0000","last_change":"2023-10-30T14:23:31.069185+0000","last_active":"2023-10-30T14:51:29.468906+0000","last_peered":"2023-10-30T14:51:29.468906+0000","last_clean":"2023-10-30T14:51:29.468906+0000","last_became_active":"2023-10-30T14:14:04.613420+0000","last_became_peered":"2023-10-30T14:14:04.613420+0000","last_unstale":"2023-10-30T14:51:29.468906+0000","last_undegraded":"2023-10-30T14:51:29.468906+0000","last_fullsized":"2023-10-30T14:51:29.468906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1327,"log_dups_size":0,"ondisk_log_size":1327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:51:35.517831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004010128,"stat_sum":{"num_bytes":12779880,"num_objects":504,"num_object_clones":238,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":504,"num_whiteouts":203,"num_read":55,"num_read_kb":8221,"num_write":1111,"num_write_kb":14267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"67'636","reported_seq":796,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.650555+0000","last_change":"2023-10-30T14:14:03.737319+0000","last_active":"2023-10-30T14:51:29.650555+0000","last_peered":"2023-10-30T14:51:29.650555+0000","last_clean":"2023-10-30T14:51:29.650555+0000","last_became_active":"2023-10-30T14:14:03.736964+0000","last_became_peered":"2023-10-30T14:14:03.736964+0000","last_unstale":"2023-10-30T14:51:29.650555+0000","last_undegraded":"2023-10-30T14:51:29.650555+0000","last_fullsized":"2023-10-30T14:51:29.650555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:41.469809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":182,"num_read_kb":58134,"num_write":693,"num_write_kb":34123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1046721150,"num_objects":31370,"num_object_clones":14593,"num_object_copies":62740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31370,"num_whiteouts":12799,"num_read":4602,"num_read_kb":662584,"num_write":69411,"num_write_kb":1341877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2168676352,"data_stored":2094671100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83616,"ondisk_log_size":83616,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":673711135,"num_objects":1337,"num_object_clones":0,"num_object_copies":2674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1337,"num_whiteouts":0,"num_read":6029,"num_read_kb":2835128,"num_write":28094,"num_write_kb":1507213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1111,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1347235840,"data_stored":1347090008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25568,"ondisk_log_size":25568,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575305,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614432,"kb_used_data":479192,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":93757408,"statfs":{"total":96636764160,"available":96007585792,"internally_reserved":0,"allocated":490692608,"data_stored":479962954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":138475979},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.077,"15min":0.997},"min":{"1min":0.51700000000000002,"5min":0.46300000000000002,"15min":0.45500000000000002},"max":{"1min":6.3040000000000003,"5min":6.399,"15min":6.399},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.2370000000000001,"5min":0.97799999999999998,"15min":0.94399999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":6.4969999999999999,"5min":6.4969999999999999,"15min":6.4969999999999999},"last":0.83199999999999996}]},{"osd":1,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":0.92500000000000004,"15min":0.93000000000000005},"min":{"1min":0.61599999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":6.1390000000000002,"5min":6.1390000000000002,"15min":6.1390000000000002},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.96599999999999997,"15min":0.99399999999999999},"min":{"1min":0.58699999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":3.8460000000000001,"5min":6.1920000000000002,"15min":6.1920000000000002},"last":0.63700000000000001}]},{"osd":2,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.0449999999999999,"15min":0.96899999999999997},"min":{"1min":0.70099999999999996,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.5739999999999998,"5min":6.5739999999999998,"15min":6.5739999999999998},"last":1.125},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.115,"15min":1.01},"min":{"1min":0.81599999999999995,"5min":0.54500000000000004,"15min":0.5},"max":{"1min":6.2350000000000003,"5min":6.5149999999999997,"15min":6.5149999999999997},"last":1.2450000000000001}]},{"osd":3,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.86199999999999999,"15min":0.85599999999999998},"min":{"1min":0.56200000000000006,"5min":0.54800000000000004,"15min":0.53800000000000003},"max":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":4.3230000000000004},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.121,"15min":1.056},"min":{"1min":0.66800000000000004,"5min":0.64800000000000002,"15min":0.54300000000000004},"max":{"1min":3.157,"5min":6.2670000000000003,"15min":6.2670000000000003},"last":0.91500000000000004}]},{"osd":4,"last update":"Mon Oct 30 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.95599999999999996,"15min":0.95699999999999996},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.54800000000000004},"max":{"1min":4.7409999999999997,"5min":4.7409999999999997,"15min":4.7409999999999997},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.036,"15min":0.97399999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":8.0719999999999992,"5min":8.0719999999999992,"15min":8.0719999999999992},"last":0.873}]},{"osd":5,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3280000000000001,"15min":1.1759999999999999},"min":{"1min":0.78100000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":10.692,"5min":10.692,"15min":10.692},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.3939999999999999,"15min":1.2869999999999999},"min":{"1min":0.72099999999999997,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":6.6429999999999998,"5min":6.6429999999999998,"15min":6.6429999999999998},"last":1.3520000000000001}]},{"osd":6,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.135,"15min":1.129},"min":{"1min":0.748,"5min":0.71599999999999997,"15min":0.62},"max":{"1min":5.2160000000000002,"5min":5.2160000000000002,"15min":5.2160000000000002},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.1639999999999999,"15min":1.107},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":4.367,"5min":4.367,"15min":4.367},"last":1.2010000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575305,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511764,"kb_used_data":386252,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":93860076,"statfs":{"total":96636764160,"available":96112717824,"internally_reserved":0,"allocated":395522048,"data_stored":388333646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":128514503},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.82899999999999996,"15min":0.83199999999999996},"min":{"1min":0.503,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.216,"5min":1.288,"15min":1.2929999999999999},"last":1.5129999999999999},{"interface":"front","average":{"1min":0.997,"5min":0.90500000000000003,"15min":0.88800000000000001},"min":{"1min":0.68600000000000005,"5min":0.435,"15min":0.435},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.391},"last":1.726}]},{"osd":1,"last update":"Mon Oct 30 14:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.89700000000000002,"15min":0.85599999999999998},"min":{"1min":0.66000000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.135,"5min":8,"15min":8},"last":1.468},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.94199999999999995,"15min":0.96599999999999997},"min":{"1min":0.67300000000000004,"5min":0.53400000000000003,"15min":0.318},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.514},"last":1.2969999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.89300000000000002,"15min":0.89000000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.40999999999999998},"max":{"1min":1.232,"5min":1.2410000000000001,"15min":1.617},"last":0.876},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.92900000000000005,"15min":0.94299999999999995},"min":{"1min":0.63400000000000001,"5min":0.59999999999999998,"15min":0.49299999999999999},"max":{"1min":1.181,"5min":1.3149999999999999,"15min":1.4119999999999999},"last":2.9609999999999999}]},{"osd":3,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":0.997,"15min":0.98099999999999998},"min":{"1min":0.73999999999999999,"5min":0.57599999999999996,"15min":0.47799999999999998},"max":{"1min":1.3700000000000001,"5min":1.3720000000000001,"15min":1.6890000000000001},"last":1.6200000000000001},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.95499999999999996,"15min":0.96099999999999997},"min":{"1min":0.61499999999999999,"5min":0.59199999999999997,"15min":0.48899999999999999},"max":{"1min":1.286,"5min":1.286,"15min":1.472},"last":1.206}]},{"osd":4,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.0309999999999999,"15min":1.032},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.59899999999999998},"max":{"1min":1.1830000000000001,"5min":3.8399999999999999,"15min":3.8399999999999999},"last":3.7549999999999999},{"interface":"front","average":{"1min":1.016,"5min":1.0269999999999999,"15min":1.04},"min":{"1min":0.69999999999999996,"5min":0.62,"15min":0.435},"max":{"1min":1.341,"5min":1.4730000000000001,"15min":1.645},"last":3.0790000000000002}]},{"osd":5,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":1.0149999999999999,"15min":0.97299999999999998},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.51100000000000001},"max":{"1min":1.1659999999999999,"5min":11.728,"15min":11.728},"last":3.2120000000000002},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.1000000000000001,"15min":1.0920000000000001},"min":{"1min":0.72899999999999998,"5min":0.51700000000000002,"15min":0.51400000000000001},"max":{"1min":1.2310000000000001,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":4.6779999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.224,"15min":1.137},"min":{"1min":0.76400000000000001,"5min":0.755,"15min":0.65200000000000002},"max":{"1min":1.268,"5min":18.041,"15min":18.041},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.345,"15min":1.228},"min":{"1min":0.98699999999999999,"5min":0.79700000000000004,"15min":0.56000000000000005},"max":{"1min":1.579,"5min":20.375,"15min":20.375},"last":3.3319999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575305,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":665052,"kb_used_data":529812,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":93706788,"statfs":{"total":96636764160,"available":95955750912,"internally_reserved":0,"allocated":542527488,"data_stored":530952161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":138475974},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":0.83699999999999997,"15min":0.82099999999999995},"min":{"1min":0.54400000000000004,"5min":0.43099999999999999,"15min":0.40100000000000002},"max":{"1min":7.04,"5min":7.04,"15min":7.04},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.131,"5min":0.876,"15min":0.85899999999999999},"min":{"1min":0.53500000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":5.4720000000000004,"5min":5.4720000000000004,"15min":5.4720000000000004},"last":1.0680000000000001}]},{"osd":1,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83199999999999996,"15min":0.88},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.284,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":1.151},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.83699999999999997,"15min":0.85099999999999998},"min":{"1min":0.57999999999999996,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":4.1630000000000003,"5min":4.1630000000000003,"15min":4.1630000000000003},"last":0.91700000000000004}]},{"osd":2,"last update":"Mon Oct 30 14:51:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.92300000000000004,"15min":0.95999999999999996},"min":{"1min":0.61299999999999999,"5min":0.503,"15min":0.44800000000000001},"max":{"1min":1.3320000000000001,"5min":1.3640000000000001,"15min":2.044},"last":1.224},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.087,"15min":1.024},"min":{"1min":0.81899999999999995,"5min":0.69099999999999995,"15min":0.51600000000000001},"max":{"1min":1.425,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":0.98699999999999999}]},{"osd":3,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.85599999999999998,"15min":0.93899999999999995},"min":{"1min":0.623,"5min":0.58399999999999996,"15min":0.56499999999999995},"max":{"1min":1.1000000000000001,"5min":1.2010000000000001,"15min":1.4550000000000001},"last":0.877},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.03,"15min":0.97199999999999998},"min":{"1min":0.75,"5min":0.623,"15min":0.52100000000000002},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.4059999999999999},"last":1.294}]},{"osd":4,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.149,"15min":1.0469999999999999},"min":{"1min":0.622,"5min":0.62,"15min":0.59099999999999997},"max":{"1min":8.7360000000000007,"5min":8.7360000000000007,"15min":8.7360000000000007},"last":0.77800000000000002},{"interface":"front","average":{"1min":2.335,"5min":1.244,"15min":1.0640000000000001},"min":{"1min":0.621,"5min":0.61199999999999999,"15min":0.59399999999999997},"max":{"1min":23.640000000000001,"5min":23.640000000000001,"15min":23.640000000000001},"last":0.73199999999999998}]},{"osd":6,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1339999999999999,"15min":1.0700000000000001},"min":{"1min":0.80100000000000005,"5min":0.73399999999999999,"15min":0.55500000000000005},"max":{"1min":1.546,"5min":1.7470000000000001,"15min":2.306},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.194,"15min":1.127},"min":{"1min":0.878,"5min":0.875,"15min":0.57899999999999996},"max":{"1min":1.575,"5min":1.7569999999999999,"15min":2.774},"last":1.367}]},{"osd":7,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.99399999999999999,"15min":1.0629999999999999},"min":{"1min":0.71899999999999997,"5min":0.63,"15min":0.40799999999999997},"max":{"1min":1.198,"5min":1.3100000000000001,"15min":3.0739999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.036,"5min":1.129,"15min":1.1739999999999999},"min":{"1min":0.87,"5min":0.87,"15min":0.77600000000000002},"max":{"1min":1.244,"5min":1.53,"15min":2.1339999999999999},"last":0.95299999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575305,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611736,"kb_used_data":476496,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":93760104,"statfs":{"total":96636764160,"available":96010346496,"internally_reserved":0,"allocated":487931904,"data_stored":476574704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":138475974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.0109999999999999,"15min":0.876},"min":{"1min":0.53300000000000003,"5min":0.48199999999999998,"15min":0.27900000000000003},"max":{"1min":3.222,"5min":7.1870000000000003,"15min":7.1870000000000003},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.099,"5min":1.107,"15min":0.95099999999999996},"min":{"1min":0.68899999999999995,"5min":0.48199999999999998,"15min":0.33900000000000002},"max":{"1min":3.3010000000000002,"5min":10.864000000000001,"15min":10.864000000000001},"last":0.99299999999999999}]},{"osd":1,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.04,"15min":0.95199999999999996},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.50700000000000001},"max":{"1min":3.3780000000000001,"5min":7.6980000000000004,"15min":7.6980000000000004},"last":1.101},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.91300000000000003,"15min":0.83999999999999997},"min":{"1min":0.58699999999999997,"5min":0.48399999999999999,"15min":0.36599999999999999},"max":{"1min":2.1789999999999998,"5min":5.5259999999999998,"15min":5.5259999999999998},"last":0.63200000000000001}]},{"osd":2,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0229999999999999,"15min":0.96199999999999997},"min":{"1min":0.64300000000000002,"5min":0.57499999999999996,"15min":0.40899999999999997},"max":{"1min":2.2839999999999998,"5min":5.423,"15min":5.423},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.254,"15min":1.0940000000000001},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.54400000000000004},"max":{"1min":3.4500000000000002,"5min":13.221,"15min":13.221},"last":1.181}]},{"osd":3,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.96099999999999997,"15min":0.86199999999999999},"min":{"1min":0.626,"5min":0.54800000000000004,"15min":0.46899999999999997},"max":{"1min":7.5620000000000003,"5min":7.5620000000000003,"15min":7.5620000000000003},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.0860000000000001,"15min":1.0029999999999999},"min":{"1min":0.71199999999999997,"5min":0.65500000000000003,"15min":0.497},"max":{"1min":7.7370000000000001,"5min":7.7370000000000001,"15min":7.7370000000000001},"last":1.056}]},{"osd":5,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.091,"15min":0.97599999999999998},"min":{"1min":0.72499999999999998,"5min":0.64200000000000002,"15min":0.317},"max":{"1min":7.6360000000000001,"5min":7.6360000000000001,"15min":7.6360000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.159,"5min":1.2609999999999999,"15min":1.145},"min":{"1min":0.628,"5min":0.628,"15min":0.442},"max":{"1min":3.6389999999999998,"5min":7.2599999999999998,"15min":7.2599999999999998},"last":1.2749999999999999}]},{"osd":6,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1910000000000001,"15min":1.1020000000000001},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.39200000000000002},"max":{"1min":2.379,"5min":5.4729999999999999,"15min":5.4729999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.2969999999999999,"15min":1.1830000000000001},"min":{"1min":0.84299999999999997,"5min":0.73999999999999999,"15min":0.47899999999999998},"max":{"1min":6.5270000000000001,"5min":10.102,"15min":10.102},"last":1.23}]},{"osd":7,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.1120000000000001,"15min":1.042},"min":{"1min":0.75700000000000001,"5min":0.69099999999999995,"15min":0.40100000000000002},"max":{"1min":9.1240000000000006,"5min":9.1240000000000006,"15min":9.1240000000000006},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.423,"5min":1.2809999999999999,"15min":1.2130000000000001},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.52900000000000003},"max":{"1min":7.0490000000000004,"5min":7.0490000000000004,"15min":7.0490000000000004},"last":1.1299999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575303,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645632,"kb_used_data":497912,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":93726208,"statfs":{"total":96636764160,"available":95975636992,"internally_reserved":0,"allocated":509861888,"data_stored":498479753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":151255496},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.53600000000000003,"15min":0.48299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.1130000000000004,"5min":4.1130000000000004,"15min":4.1130000000000004},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.54200000000000004,"15min":0.51200000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.91100000000000003,"5min":4.5190000000000001,"15min":4.5190000000000001},"last":0.376}]},{"osd":1,"last update":"Mon Oct 30 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.58999999999999997,"15min":0.54400000000000004},"min":{"1min":0.33100000000000002,"5min":0.317,"15min":0.314},"max":{"1min":3.988,"5min":3.988,"15min":3.988},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.56399999999999995,"15min":0.54900000000000004},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.29999999999999999},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.432}]},{"osd":2,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.65600000000000003,"15min":0.56100000000000005},"min":{"1min":0.31,"5min":0.29199999999999998,"15min":0.27600000000000002},"max":{"1min":1.0740000000000001,"5min":8.6519999999999992,"15min":8.6519999999999992},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.73999999999999999,"15min":0.63700000000000001},"min":{"1min":0.42199999999999999,"5min":0.41999999999999998,"15min":0.313},"max":{"1min":1.1040000000000001,"5min":7.0949999999999998,"15min":7.0949999999999998},"last":0.63300000000000001}]},{"osd":4,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.83399999999999996,"15min":0.82099999999999995},"min":{"1min":0.54600000000000004,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.1220000000000001,"5min":4.5730000000000004,"15min":4.5730000000000004},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.90200000000000002,"15min":0.85299999999999998},"min":{"1min":0.51500000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":4.0910000000000002,"5min":4.4720000000000004,"15min":4.4720000000000004},"last":0.78400000000000003}]},{"osd":5,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.0029999999999999,"15min":0.90000000000000002},"min":{"1min":0.72799999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.296,"5min":14.974,"15min":14.974},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.042,"5min":0.96699999999999997,"15min":0.873},"min":{"1min":0.746,"5min":0.52600000000000002,"15min":0.46600000000000003},"max":{"1min":4.0629999999999997,"5min":5.907,"15min":5.907},"last":0.88400000000000001}]},{"osd":6,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.01,"15min":0.91900000000000004},"min":{"1min":0.70899999999999996,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":5.8140000000000001,"5min":5.8300000000000001,"15min":5.8300000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.159,"5min":1.073,"15min":0.96299999999999997},"min":{"1min":0.70799999999999996,"5min":0.60199999999999998,"15min":0.55100000000000005},"max":{"1min":6.0300000000000002,"5min":6.0300000000000002,"15min":6.0300000000000002},"last":0.99099999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":1.115,"15min":0.995},"min":{"1min":0.67600000000000005,"5min":0.66800000000000004,"15min":0.61699999999999999},"max":{"1min":1.2549999999999999,"5min":16.282,"15min":16.282},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.123,"15min":0.97199999999999998},"min":{"1min":0.70999999999999996,"5min":0.627,"15min":0.58099999999999996},"max":{"1min":1.6519999999999999,"5min":16.096,"15min":16.096},"last":0.83599999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575305,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":417172,"kb_used_data":309708,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":93954668,"statfs":{"total":96636764160,"available":96209580032,"internally_reserved":0,"allocated":317140992,"data_stored":310407991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.45300000000000001,"15min":0.48699999999999999},"min":{"1min":0.30099999999999999,"5min":0.27400000000000002,"15min":0.26300000000000001},"max":{"1min":0.67800000000000005,"5min":0.79100000000000004,"15min":13.526},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.50900000000000001,"15min":0.51300000000000001},"min":{"1min":0.38300000000000001,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":1.2190000000000001},"last":0.434}]},{"osd":1,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46300000000000002,"15min":0.47199999999999998},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65500000000000003,"5min":0.67200000000000004,"15min":1.5469999999999999},"last":0.314},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54400000000000004,"15min":0.56000000000000005},"min":{"1min":0.38700000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73499999999999999,"5min":0.79300000000000004,"15min":1.452},"last":0.46000000000000002}]},{"osd":3,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58799999999999997,"15min":0.60699999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81299999999999994,"5min":0.95399999999999996,"15min":1.9390000000000001},"last":0.497},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.53900000000000003},"min":{"1min":0.35899999999999999,"5min":0.35499999999999998,"15min":0.33100000000000002},"max":{"1min":0.78500000000000003,"5min":0.79300000000000004,"15min":1.913},"last":0.47599999999999998}]},{"osd":4,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.81699999999999995,"15min":0.81499999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.44900000000000001},"max":{"1min":1.0309999999999999,"5min":4.2229999999999999,"15min":4.2229999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.91600000000000004,"15min":0.91000000000000003},"min":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.59599999999999997},"max":{"1min":1.081,"5min":4.2439999999999998,"15min":4.2439999999999998},"last":0.82799999999999996}]},{"osd":5,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.98999999999999999,"15min":0.86399999999999999},"min":{"1min":0.51900000000000002,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0620000000000001,"5min":22.285,"15min":22.285},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.82899999999999996,"15min":0.83399999999999996},"min":{"1min":0.58199999999999996,"5min":0.56000000000000005,"15min":0.51900000000000002},"max":{"1min":1.1319999999999999,"5min":2.3999999999999999,"15min":2.742},"last":0.66900000000000004}]},{"osd":6,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.86099999999999999,"15min":0.91500000000000004},"min":{"1min":0.65000000000000002,"5min":0.58799999999999997,"15min":0.54500000000000004},"max":{"1min":1.0489999999999999,"5min":2.911,"15min":27.731000000000002},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.872,"15min":0.874},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.1679999999999999,"5min":1.9099999999999999,"15min":2.0009999999999999},"last":0.81699999999999995}]},{"osd":7,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84299999999999997,"15min":0.91600000000000004},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.56499999999999995},"max":{"1min":1.105,"5min":1.2390000000000001,"15min":18.870999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.90400000000000003,"15min":0.96399999999999997},"min":{"1min":0.66600000000000004,"5min":0.61499999999999999,"15min":0.55500000000000005},"max":{"1min":1.0860000000000001,"5min":3.3809999999999998,"15min":18.356999999999999},"last":0.99099999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575304,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461964,"kb_used_data":344772,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":93909876,"statfs":{"total":96636764160,"available":96163713024,"internally_reserved":0,"allocated":353046528,"data_stored":346331165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":119994823},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.47399999999999998,"15min":0.44800000000000001},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.68200000000000005,"5min":2.4889999999999999,"15min":2.4889999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.51600000000000001,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71199999999999997,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.55800000000000005}]},{"osd":2,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.64500000000000002,"15min":0.56699999999999995},"min":{"1min":0.318,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79000000000000004,"5min":8.3160000000000007,"15min":8.3160000000000007},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.54000000000000004,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.27200000000000002,"15min":0.26800000000000002},"max":{"1min":0.61399999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.58099999999999996}]},{"osd":3,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.61099999999999999,"15min":0.59599999999999997},"min":{"1min":0.38800000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84399999999999997,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.623,"5min":0.64900000000000002,"15min":0.63600000000000001},"min":{"1min":0.439,"5min":0.38100000000000001,"15min":0.36499999999999999},"max":{"1min":0.89200000000000002,"5min":3.54,"15min":3.54},"last":0.66000000000000003}]},{"osd":4,"last update":"Mon Oct 30 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.92000000000000004,"15min":0.83999999999999997},"min":{"1min":0.505,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.92600000000000005,"5min":10.525,"15min":10.525},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.94899999999999995,"15min":0.83899999999999997},"min":{"1min":0.68400000000000005,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.997,"5min":10.571,"15min":10.571},"last":0.84799999999999998}]},{"osd":5,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.88900000000000001,"15min":0.89100000000000001},"min":{"1min":0.628,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.0289999999999999,"5min":3.4969999999999999,"15min":3.4969999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.876,"5min":0.84399999999999997,"15min":0.79200000000000004},"min":{"1min":0.72399999999999998,"5min":0.56000000000000005,"15min":0.49099999999999999},"max":{"1min":1.1739999999999999,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.96499999999999997}]},{"osd":6,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.80900000000000005},"min":{"1min":0.58399999999999996,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.1040000000000001,"5min":1.875,"15min":2.4710000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.88900000000000001,"15min":0.83399999999999996},"min":{"1min":0.57399999999999995,"5min":0.53100000000000003,"15min":0.51700000000000002},"max":{"1min":1.2130000000000001,"5min":5.3680000000000003,"15min":5.3680000000000003},"last":0.995}]},{"osd":7,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":1.081,"15min":0.91600000000000004},"min":{"1min":0.59599999999999997,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0249999999999999,"5min":19.885000000000002,"15min":19.885000000000002},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":1.054,"15min":0.93200000000000005},"min":{"1min":0.60099999999999998,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.127,"5min":13.401999999999999,"15min":13.401999999999999},"last":1.117}]}]},{"osd":0,"up_from":13,"seq":55834575306,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545984,"kb_used_data":416312,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":93825856,"statfs":{"total":96636764160,"available":96077676544,"internally_reserved":0,"allocated":426303488,"data_stored":414784024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132774342},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45900000000000002,"15min":0.44800000000000001},"min":{"1min":0.32200000000000001,"5min":0.26800000000000002,"15min":0.22500000000000001},"max":{"1min":0.749,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.53300000000000003,"15min":0.503},"min":{"1min":0.36699999999999999,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":0.68999999999999995,"5min":3.367,"15min":3.367},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58199999999999996,"15min":0.53600000000000003},"min":{"1min":0.44400000000000001,"5min":0.378,"15min":0.30499999999999999},"max":{"1min":0.751,"5min":3.222,"15min":3.222},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53200000000000003,"15min":0.56000000000000005},"min":{"1min":0.40000000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":1.095},"last":0.434}]},{"osd":3,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.56599999999999995,"15min":0.60199999999999998},"min":{"1min":0.34000000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.63300000000000001,"5min":6.4980000000000002,"15min":6.4980000000000002},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57499999999999996,"15min":0.52900000000000003},"min":{"1min":0.40400000000000003,"5min":0.40100000000000002,"15min":0.28000000000000003},"max":{"1min":0.71699999999999997,"5min":2.024,"15min":2.024},"last":0.497}]},{"osd":4,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.78800000000000003,"15min":0.82599999999999996},"min":{"1min":0.57599999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.95399999999999996,"5min":2.0659999999999998,"15min":2.8380000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81399999999999995,"15min":0.85999999999999999},"min":{"1min":0.65100000000000002,"5min":0.56899999999999995,"15min":0.53900000000000003},"max":{"1min":0.97699999999999998,"5min":2.1000000000000001,"15min":5.8959999999999999},"last":0.75700000000000001}]},{"osd":5,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.872,"15min":0.86099999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.46300000000000002},"max":{"1min":1.212,"5min":1.278,"15min":6.4480000000000004},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.92000000000000004,"15min":0.85599999999999998},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.54000000000000004},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":2.6869999999999998},"last":0.84399999999999997}]},{"osd":6,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.83899999999999997,"15min":0.84699999999999998},"min":{"1min":0.64800000000000002,"5min":0.55300000000000005,"15min":0.54400000000000004},"max":{"1min":1.133,"5min":1.2150000000000001,"15min":4.9980000000000002},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.877,"5min":0.85499999999999998,"15min":0.86099999999999999},"min":{"1min":0.65300000000000002,"5min":0.55100000000000005,"15min":0.54000000000000004},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.5980000000000001},"last":0.74099999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.86799999999999999,"15min":0.84699999999999998},"min":{"1min":0.63700000000000001,"5min":0.57899999999999996,"15min":0.499},"max":{"1min":1.1120000000000001,"5min":3.6589999999999998,"15min":3.6589999999999998},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.93100000000000005,"15min":0.88400000000000001},"min":{"1min":0.76300000000000001,"5min":0.625,"15min":0.54100000000000004},"max":{"1min":1.1579999999999999,"5min":1.262,"15min":2.6429999999999998},"last":1.02}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":104873984,"data_stored":104858260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159408128,"data_stored":159388062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":115867648,"data_stored":115847932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":200708096,"data_stored":200690717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":153219072,"data_stored":153204613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":208064512,"data_stored":208040997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":211091456,"data_stored":211074246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":194002944,"data_stored":193985181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":320053248,"data_stored":308933294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192856064,"data_stored":186541001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200491008,"data_stored":194157957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":308371456,"data_stored":297386934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":333930496,"data_stored":322967989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":333680640,"data_stored":322509062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":183648256,"data_stored":176857298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":295645184,"data_stored":285317565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:51:35.590 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T14:51:35.592 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:51:35.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.764+0000 7f35aa654640 1 Processor -- start 2023-10-30T14:51:35.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.764+0000 7f35aa654640 1 -- start start 2023-10-30T14:51:35.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.764+0000 7f35aa654640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35a41537f0 0x7f35a40701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:35.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.764+0000 7f35aa654640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f35a415a2e0 0x7f35a40ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:35.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.764+0000 7f35aa654640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f35a40cd4a0 0x7f35a406b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:35.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.764+0000 7f35aa654640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a40c3ab0 con 0x7f35a40cd4a0 2023-10-30T14:51:35.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.764+0000 7f35aa654640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a40706e0 con 0x7f35a415a2e0 2023-10-30T14:51:35.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.764+0000 7f35aa654640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a40c2aa0 con 0x7f35a41537f0 2023-10-30T14:51:35.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.764+0000 7f35a8bf1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f35a40cd4a0 0x7f35a406b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:35.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.764+0000 7f35a8bf1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f35a40cd4a0 0x7f35a406b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56634/0 (socket says 172.21.15.57:56634) 2023-10-30T14:51:35.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.764+0000 7f35a8bf1640 1 -- 172.21.15.57:0/907529914 learned_addr learned my addr 172.21.15.57:0/907529914 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:35.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.764+0000 7f35a3fff640 1 --2- 172.21.15.57:0/907529914 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35a41537f0 0x7f35a40701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:35.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.764+0000 7f35a37fe640 1 --2- 172.21.15.57:0/907529914 >> v2:172.21.15.88:3300/0 conn(0x7f35a415a2e0 0x7f35a40ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:35.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.768+0000 7f35a8bf1640 1 -- 172.21.15.57:0/907529914 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35a41537f0 msgr2=0x7f35a40701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:35.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.768+0000 7f35a8bf1640 1 --2- 172.21.15.57:0/907529914 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35a41537f0 0x7f35a40701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.768+0000 7f35a8bf1640 1 -- 172.21.15.57:0/907529914 >> v2:172.21.15.88:3300/0 conn(0x7f35a415a2e0 msgr2=0x7f35a40ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:35.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.768+0000 7f35a8bf1640 1 --2- 172.21.15.57:0/907529914 >> v2:172.21.15.88:3300/0 conn(0x7f35a415a2e0 0x7f35a40ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.768+0000 7f35a8bf1640 1 -- 172.21.15.57:0/907529914 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a40708e0 con 0x7f35a40cd4a0 2023-10-30T14:51:35.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.768+0000 7f35a8bf1640 1 --2- 172.21.15.57:0/907529914 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35a40cd4a0 0x7f35a406b2f0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f35a40ccf30 tx=0x7f359c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7869f7958854d579 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:35.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.768+0000 7f35a2ffd640 1 -- 172.21.15.57:0/907529914 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c0af400 con 0x7f35a40cd4a0 2023-10-30T14:51:35.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.768+0000 7f35a2ffd640 1 -- 172.21.15.57:0/907529914 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359c0ab050 con 0x7f35a40cd4a0 2023-10-30T14:51:35.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.768+0000 7f35a2ffd640 1 -- 172.21.15.57:0/907529914 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c0af780 con 0x7f35a40cd4a0 2023-10-30T14:51:35.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.768+0000 7f35aa654640 1 -- 172.21.15.57:0/907529914 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35a40cd4a0 msgr2=0x7f35a406b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:35.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.768+0000 7f35aa654640 1 --2- 172.21.15.57:0/907529914 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35a40cd4a0 0x7f35a406b2f0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f35a40ccf30 tx=0x7f359c01f1c0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.768+0000 7f35aa654640 1 -- 172.21.15.57:0/907529914 shutdown_connections 2023-10-30T14:51:35.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.768+0000 7f35aa654640 1 --2- 172.21.15.57:0/907529914 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35a40cd4a0 0x7f35a406b2f0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.768+0000 7f35aa654640 1 --2- 172.21.15.57:0/907529914 >> v2:172.21.15.88:3300/0 conn(0x7f35a415a2e0 0x7f35a40ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.768+0000 7f35aa654640 1 --2- 172.21.15.57:0/907529914 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35a41537f0 0x7f35a40701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35aa654640 1 -- 172.21.15.57:0/907529914 >> 172.21.15.57:0/907529914 conn(0x7f35a4071fb0 msgr2=0x7f35a40723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:35.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35aa654640 1 -- 172.21.15.57:0/907529914 shutdown_connections 2023-10-30T14:51:35.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35aa654640 1 -- 172.21.15.57:0/907529914 wait complete. 2023-10-30T14:51:35.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35aa654640 1 Processor -- start 2023-10-30T14:51:35.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35aa654640 1 -- start start 2023-10-30T14:51:35.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35aa654640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35a40cd4a0 0x7f35a415b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:35.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35aa654640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35a41537f0 0x7f35a41ef770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:35.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35aa654640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35a415a2e0 0x7f35a41efcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:35.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35aa654640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a415bbe0 con 0x7f35a415a2e0 2023-10-30T14:51:35.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35aa654640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a41ef2e0 con 0x7f35a41537f0 2023-10-30T14:51:35.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35aa654640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a40c4a30 con 0x7f35a40cd4a0 2023-10-30T14:51:35.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35a8bf1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35a415a2e0 0x7f35a41efcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:35.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35a8bf1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35a415a2e0 0x7f35a41efcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56638/0 (socket says 172.21.15.57:56638) 2023-10-30T14:51:35.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35a8bf1640 1 -- 172.21.15.57:0/1723967295 learned_addr learned my addr 172.21.15.57:0/1723967295 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:35.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35a37fe640 1 --2- 172.21.15.57:0/1723967295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35a41537f0 0x7f35a41ef770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:35.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35a3fff640 1 --2- 172.21.15.57:0/1723967295 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35a40cd4a0 0x7f35a415b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:35.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35a8bf1640 1 -- 172.21.15.57:0/1723967295 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35a40cd4a0 msgr2=0x7f35a415b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:35.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35a8bf1640 1 --2- 172.21.15.57:0/1723967295 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35a40cd4a0 0x7f35a415b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35a8bf1640 1 -- 172.21.15.57:0/1723967295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35a41537f0 msgr2=0x7f35a41ef770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:35.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35a8bf1640 1 --2- 172.21.15.57:0/1723967295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35a41537f0 0x7f35a41ef770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35a8bf1640 1 -- 172.21.15.57:0/1723967295 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c009b30 con 0x7f35a415a2e0 2023-10-30T14:51:35.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.772+0000 7f35a8bf1640 1 --2- 172.21.15.57:0/1723967295 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35a415a2e0 0x7f35a41efcb0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f359c0b2040 tx=0x7f359c01bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:35.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.776+0000 7f35a0ff9640 1 -- 172.21.15.57:0/1723967295 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c0c0070 con 0x7f35a415a2e0 2023-10-30T14:51:35.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.776+0000 7f35a0ff9640 1 -- 172.21.15.57:0/1723967295 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359c009510 con 0x7f35a415a2e0 2023-10-30T14:51:35.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.776+0000 7f35a0ff9640 1 -- 172.21.15.57:0/1723967295 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c0afcc0 con 0x7f35a415a2e0 2023-10-30T14:51:35.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.776+0000 7f35aa654640 1 -- 172.21.15.57:0/1723967295 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a4152d30 con 0x7f35a415a2e0 2023-10-30T14:51:35.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.776+0000 7f35aa654640 1 -- 172.21.15.57:0/1723967295 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a40c4750 con 0x7f35a415a2e0 2023-10-30T14:51:35.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.776+0000 7f35aa654640 1 -- 172.21.15.57:0/1723967295 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a41ef450 con 0x7f35a415a2e0 2023-10-30T14:51:35.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.776+0000 7f35a0ff9640 1 -- 172.21.15.57:0/1723967295 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f359c0b6040 con 0x7f35a415a2e0 2023-10-30T14:51:35.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.776+0000 7f35a0ff9640 1 --2- 172.21.15.57:0/1723967295 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f35900664a0 0x7f3590068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:35.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.776+0000 7f35a0ff9640 1 -- 172.21.15.57:0/1723967295 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f359c121a00 con 0x7f35a415a2e0 2023-10-30T14:51:35.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.776+0000 7f35a3fff640 1 --2- 172.21.15.57:0/1723967295 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f35900664a0 0x7f3590068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:35.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.780+0000 7f35a3fff640 1 --2- 172.21.15.57:0/1723967295 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f35900664a0 0x7f3590068960 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f3594009ae0 tx=0x7f3594009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:35.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.780+0000 7f35a0ff9640 1 -- 172.21.15.57:0/1723967295 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f359c0ee960 con 0x7f35a415a2e0 2023-10-30T14:51:35.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.936+0000 7f35aa654640 1 -- 172.21.15.57:0/1723967295 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f35a40c2170 con 0x7f35900664a0 2023-10-30T14:51:35.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.956+0000 7f35a0ff9640 1 -- 172.21.15.57:0/1723967295 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+293482 (secure 0 0 0) 0x7f3594011a00 con 0x7f35900664a0 2023-10-30T14:51:35.964 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:51:35.966 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-30T14:51:35.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.960+0000 7f35aa654640 1 -- 172.21.15.57:0/1723967295 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f35900664a0 msgr2=0x7f3590068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:35.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.960+0000 7f35aa654640 1 --2- 172.21.15.57:0/1723967295 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f35900664a0 0x7f3590068960 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f3594009ae0 tx=0x7f3594009190 comp rx=0 tx=0).stop 2023-10-30T14:51:35.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.960+0000 7f35aa654640 1 -- 172.21.15.57:0/1723967295 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35a415a2e0 msgr2=0x7f35a41efcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:35.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.960+0000 7f35aa654640 1 --2- 172.21.15.57:0/1723967295 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35a415a2e0 0x7f35a41efcb0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f359c0b2040 tx=0x7f359c01bf60 comp rx=0 tx=0).stop 2023-10-30T14:51:35.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.960+0000 7f35aa654640 1 -- 172.21.15.57:0/1723967295 shutdown_connections 2023-10-30T14:51:35.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.960+0000 7f35aa654640 1 --2- 172.21.15.57:0/1723967295 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f35900664a0 0x7f3590068960 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.960+0000 7f35aa654640 1 --2- 172.21.15.57:0/1723967295 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35a415a2e0 0x7f35a41efcb0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.960+0000 7f35aa654640 1 --2- 172.21.15.57:0/1723967295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35a41537f0 0x7f35a41ef770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.960+0000 7f35aa654640 1 --2- 172.21.15.57:0/1723967295 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f35a40cd4a0 0x7f35a415b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:35.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.960+0000 7f35aa654640 1 -- 172.21.15.57:0/1723967295 >> 172.21.15.57:0/1723967295 conn(0x7f35a4071fb0 msgr2=0x7f35a40cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:35.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.960+0000 7f35aa654640 1 -- 172.21.15.57:0/1723967295 shutdown_connections 2023-10-30T14:51:35.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:35.960+0000 7f35aa654640 1 -- 172.21.15.57:0/1723967295 wait complete. 2023-10-30T14:51:35.985 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":1192,"stamp":"2023-10-30T14:51:35.317764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1721022653,"num_objects":32709,"num_object_clones":14593,"num_object_copies":65418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32709,"num_whiteouts":12799,"num_read":10747,"num_read_kb":3497812,"num_write":97664,"num_write_kb":2851506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1111,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109301,"ondisk_log_size":109301,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":4473736,"kb_used_data":3440456,"kb_used_omap":12,"kb_used_meta":1033203,"kb_avail":750500984,"statfs":{"total":773094113280,"available":768513007616,"internally_reserved":0,"allocated":3523026944,"data_stored":3445826398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":1058000441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":75,"apply_latency_ms":75,"commit_latency_ns":75000000,"apply_latency_ns":75000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20857760,"num_objects":198,"num_object_clones":148,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":-10,"num_read":54,"num_read_kb":4921,"num_write":4539,"num_write_kb":27455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004603"},"pg_stats":[{"pgid":"3.3f","version":"67'1276","reported_seq":3239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.482005+0000","last_change":"2023-10-30T14:23:31.082534+0000","last_active":"2023-10-30T14:51:28.482005+0000","last_peered":"2023-10-30T14:51:28.482005+0000","last_clean":"2023-10-30T14:51:28.482005+0000","last_became_active":"2023-10-30T14:14:04.650882+0000","last_became_peered":"2023-10-30T14:14:04.650882+0000","last_unstale":"2023-10-30T14:51:28.482005+0000","last_undegraded":"2023-10-30T14:51:28.482005+0000","last_fullsized":"2023-10-30T14:51:28.482005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1276,"log_dups_size":0,"ondisk_log_size":1276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:46:41.132286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010330941,"stat_sum":{"num_bytes":16875912,"num_objects":459,"num_object_clones":215,"num_object_copies":918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":459,"num_whiteouts":184,"num_read":62,"num_read_kb":8246,"num_write":1073,"num_write_kb":18488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"67'1268","reported_seq":3223,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.429379+0000","last_change":"2023-10-30T14:23:30.771600+0000","last_active":"2023-10-30T14:51:29.429379+0000","last_peered":"2023-10-30T14:51:29.429379+0000","last_clean":"2023-10-30T14:51:29.429379+0000","last_became_active":"2023-10-30T14:14:04.587942+0000","last_became_peered":"2023-10-30T14:14:04.587942+0000","last_unstale":"2023-10-30T14:51:29.429379+0000","last_undegraded":"2023-10-30T14:51:29.429379+0000","last_fullsized":"2023-10-30T14:51:29.429379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1268,"log_dups_size":0,"ondisk_log_size":1268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:54:24.389508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002850184,"stat_sum":{"num_bytes":12592181,"num_objects":480,"num_object_clones":220,"num_object_copies":960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":480,"num_whiteouts":195,"num_read":59,"num_read_kb":8220,"num_write":1065,"num_write_kb":14470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"67'1295","reported_seq":3258,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.481286+0000","last_change":"2023-10-30T14:23:30.776754+0000","last_active":"2023-10-30T14:51:28.481286+0000","last_peered":"2023-10-30T14:51:28.481286+0000","last_clean":"2023-10-30T14:51:28.481286+0000","last_became_active":"2023-10-30T14:14:04.590700+0000","last_became_peered":"2023-10-30T14:14:04.590700+0000","last_unstale":"2023-10-30T14:51:28.481286+0000","last_undegraded":"2023-10-30T14:51:28.481286+0000","last_fullsized":"2023-10-30T14:51:28.481286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:11:25.956171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042861139999999997,"stat_sum":{"num_bytes":12573829,"num_objects":487,"num_object_clones":226,"num_object_copies":974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":204,"num_read":73,"num_read_kb":16409,"num_write":1057,"num_write_kb":22081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"67'1290","reported_seq":3252,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.429904+0000","last_change":"2023-10-30T14:23:31.067598+0000","last_active":"2023-10-30T14:51:29.429904+0000","last_peered":"2023-10-30T14:51:29.429904+0000","last_clean":"2023-10-30T14:51:29.429904+0000","last_became_active":"2023-10-30T14:14:45.661677+0000","last_became_peered":"2023-10-30T14:14:45.661677+0000","last_unstale":"2023-10-30T14:51:29.429904+0000","last_undegraded":"2023-10-30T14:51:29.429904+0000","last_fullsized":"2023-10-30T14:51:29.429904+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1290,"log_dups_size":0,"ondisk_log_size":1290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:33.531763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003131572,"stat_sum":{"num_bytes":8739581,"num_objects":477,"num_object_clones":215,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":188,"num_read":63,"num_read_kb":69,"num_write":1083,"num_write_kb":14952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"67'1295","reported_seq":3255,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.455528+0000","last_change":"2023-10-30T14:23:31.072132+0000","last_active":"2023-10-30T14:51:28.455528+0000","last_peered":"2023-10-30T14:51:28.455528+0000","last_clean":"2023-10-30T14:51:28.455528+0000","last_became_active":"2023-10-30T14:14:04.644683+0000","last_became_peered":"2023-10-30T14:14:04.644683+0000","last_unstale":"2023-10-30T14:51:28.455528+0000","last_undegraded":"2023-10-30T14:51:28.455528+0000","last_fullsized":"2023-10-30T14:51:28.455528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:39.271356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052462430000000003,"stat_sum":{"num_bytes":8813426,"num_objects":481,"num_object_clones":223,"num_object_copies":962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":202,"num_read":70,"num_read_kb":4116,"num_write":1073,"num_write_kb":10060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'1284","reported_seq":3252,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.493921+0000","last_change":"2023-10-30T14:23:30.777057+0000","last_active":"2023-10-30T14:51:28.493921+0000","last_peered":"2023-10-30T14:51:28.493921+0000","last_clean":"2023-10-30T14:51:28.493921+0000","last_became_active":"2023-10-30T14:14:04.619335+0000","last_became_peered":"2023-10-30T14:14:04.619335+0000","last_unstale":"2023-10-30T14:51:28.493921+0000","last_undegraded":"2023-10-30T14:51:28.493921+0000","last_fullsized":"2023-10-30T14:51:28.493921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1284,"log_dups_size":0,"ondisk_log_size":1284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:16:48.356144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062283859999999998,"stat_sum":{"num_bytes":20637019,"num_objects":477,"num_object_clones":220,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":194,"num_read":75,"num_read_kb":12254,"num_write":1076,"num_write_kb":22206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"67'1281","reported_seq":3231,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.464417+0000","last_change":"2023-10-30T14:23:31.080820+0000","last_active":"2023-10-30T14:51:29.464417+0000","last_peered":"2023-10-30T14:51:29.464417+0000","last_clean":"2023-10-30T14:51:29.464417+0000","last_became_active":"2023-10-30T14:14:04.641323+0000","last_became_peered":"2023-10-30T14:14:04.641323+0000","last_unstale":"2023-10-30T14:51:29.464417+0000","last_undegraded":"2023-10-30T14:51:29.464417+0000","last_fullsized":"2023-10-30T14:51:29.464417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1281,"log_dups_size":0,"ondisk_log_size":1281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:11.764013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011228897,"stat_sum":{"num_bytes":14656224,"num_objects":488,"num_object_clones":222,"num_object_copies":976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":202,"num_read":59,"num_read_kb":8203,"num_write":1074,"num_write_kb":18148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'1241","reported_seq":3196,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.459518+0000","last_change":"2023-10-30T14:23:31.067811+0000","last_active":"2023-10-30T14:51:29.459518+0000","last_peered":"2023-10-30T14:51:29.459518+0000","last_clean":"2023-10-30T14:51:29.459518+0000","last_became_active":"2023-10-30T14:14:04.618312+0000","last_became_peered":"2023-10-30T14:14:04.618312+0000","last_unstale":"2023-10-30T14:51:29.459518+0000","last_undegraded":"2023-10-30T14:51:29.459518+0000","last_fullsized":"2023-10-30T14:51:29.459518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1241,"log_dups_size":0,"ondisk_log_size":1241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:57:22.752126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002849426,"stat_sum":{"num_bytes":12868892,"num_objects":473,"num_object_clones":218,"num_object_copies":946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":473,"num_whiteouts":195,"num_read":60,"num_read_kb":8209,"num_write":1023,"num_write_kb":15138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"67'1328","reported_seq":3278,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.473222+0000","last_change":"2023-10-30T14:23:31.083143+0000","last_active":"2023-10-30T14:51:29.473222+0000","last_peered":"2023-10-30T14:51:29.473222+0000","last_clean":"2023-10-30T14:51:29.473222+0000","last_became_active":"2023-10-30T14:14:04.647361+0000","last_became_peered":"2023-10-30T14:14:04.647361+0000","last_unstale":"2023-10-30T14:51:29.473222+0000","last_undegraded":"2023-10-30T14:51:29.473222+0000","last_fullsized":"2023-10-30T14:51:29.473222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1328,"log_dups_size":0,"ondisk_log_size":1328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:06:53.616905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010459703000000001,"stat_sum":{"num_bytes":8468640,"num_objects":507,"num_object_clones":234,"num_object_copies":1014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":507,"num_whiteouts":206,"num_read":70,"num_read_kb":8267,"num_write":1118,"num_write_kb":13882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"67'1366","reported_seq":3270,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.468701+0000","last_change":"2023-10-30T14:23:31.075673+0000","last_active":"2023-10-30T14:51:29.468701+0000","last_peered":"2023-10-30T14:51:29.468701+0000","last_clean":"2023-10-30T14:51:29.468701+0000","last_became_active":"2023-10-30T14:14:04.630959+0000","last_became_peered":"2023-10-30T14:14:04.630959+0000","last_unstale":"2023-10-30T14:51:29.468701+0000","last_undegraded":"2023-10-30T14:51:29.468701+0000","last_fullsized":"2023-10-30T14:51:29.468701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1366,"log_dups_size":0,"ondisk_log_size":1366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:36.246017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079325130000000004,"stat_sum":{"num_bytes":12335215,"num_objects":539,"num_object_clones":286,"num_object_copies":1078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":191,"num_read":76,"num_read_kb":4189,"num_write":1090,"num_write_kb":13918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"67'1287","reported_seq":3229,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469676+0000","last_change":"2023-10-30T14:23:31.075101+0000","last_active":"2023-10-30T14:51:29.469676+0000","last_peered":"2023-10-30T14:51:29.469676+0000","last_clean":"2023-10-30T14:51:29.469676+0000","last_became_active":"2023-10-30T14:14:04.612653+0000","last_became_peered":"2023-10-30T14:14:04.612653+0000","last_unstale":"2023-10-30T14:51:29.469676+0000","last_undegraded":"2023-10-30T14:51:29.469676+0000","last_fullsized":"2023-10-30T14:51:29.469676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1287,"log_dups_size":0,"ondisk_log_size":1287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:34:44.855573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079098320000000003,"stat_sum":{"num_bytes":17654064,"num_objects":517,"num_object_clones":235,"num_object_copies":1034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":517,"num_whiteouts":205,"num_read":63,"num_read_kb":9809,"num_write":1073,"num_write_kb":19026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"67'1267","reported_seq":3224,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469272+0000","last_change":"2023-10-30T14:23:30.957983+0000","last_active":"2023-10-30T14:51:29.469272+0000","last_peered":"2023-10-30T14:51:29.469272+0000","last_clean":"2023-10-30T14:51:29.469272+0000","last_became_active":"2023-10-30T14:14:04.621135+0000","last_became_peered":"2023-10-30T14:14:04.621135+0000","last_unstale":"2023-10-30T14:51:29.469272+0000","last_undegraded":"2023-10-30T14:51:29.469272+0000","last_fullsized":"2023-10-30T14:51:29.469272+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1267,"log_dups_size":0,"ondisk_log_size":1267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:04:43.303589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025042409999999999,"stat_sum":{"num_bytes":14287686,"num_objects":497,"num_object_clones":224,"num_object_copies":994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":497,"num_whiteouts":193,"num_read":67,"num_read_kb":6339,"num_write":1071,"num_write_kb":19938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"67'1268","reported_seq":3251,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.468243+0000","last_change":"2023-10-30T14:23:31.070450+0000","last_active":"2023-10-30T14:51:29.468243+0000","last_peered":"2023-10-30T14:51:29.468243+0000","last_clean":"2023-10-30T14:51:29.468243+0000","last_became_active":"2023-10-30T14:14:04.616142+0000","last_became_peered":"2023-10-30T14:14:04.616142+0000","last_unstale":"2023-10-30T14:51:29.468243+0000","last_undegraded":"2023-10-30T14:51:29.468243+0000","last_fullsized":"2023-10-30T14:51:29.468243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1268,"log_dups_size":0,"ondisk_log_size":1268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:46:20.636160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049623779999999999,"stat_sum":{"num_bytes":9235702,"num_objects":480,"num_object_clones":219,"num_object_copies":960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":480,"num_whiteouts":189,"num_read":88,"num_read_kb":8286,"num_write":1065,"num_write_kb":19474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"67'1307","reported_seq":3281,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.633026+0000","last_change":"2023-10-30T14:23:30.775687+0000","last_active":"2023-10-30T14:51:29.633026+0000","last_peered":"2023-10-30T14:51:29.633026+0000","last_clean":"2023-10-30T14:51:29.633026+0000","last_became_active":"2023-10-30T14:14:04.588452+0000","last_became_peered":"2023-10-30T14:14:04.588452+0000","last_unstale":"2023-10-30T14:51:29.633026+0000","last_undegraded":"2023-10-30T14:51:29.633026+0000","last_fullsized":"2023-10-30T14:51:29.633026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1307,"log_dups_size":0,"ondisk_log_size":1307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:40:26.637820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046351760000000004,"stat_sum":{"num_bytes":12783229,"num_objects":466,"num_object_clones":216,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":195,"num_read":76,"num_read_kb":7792,"num_write":1098,"num_write_kb":17084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"67'1312","reported_seq":3276,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.502566+0000","last_change":"2023-10-30T14:23:30.770743+0000","last_active":"2023-10-30T14:51:28.502566+0000","last_peered":"2023-10-30T14:51:28.502566+0000","last_clean":"2023-10-30T14:51:28.502566+0000","last_became_active":"2023-10-30T14:14:04.620781+0000","last_became_peered":"2023-10-30T14:14:04.620781+0000","last_unstale":"2023-10-30T14:51:28.502566+0000","last_undegraded":"2023-10-30T14:51:28.502566+0000","last_fullsized":"2023-10-30T14:51:28.502566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1312,"log_dups_size":0,"ondisk_log_size":1312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:09.260750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002316706,"stat_sum":{"num_bytes":21325663,"num_objects":484,"num_object_clones":220,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":196,"num_read":69,"num_read_kb":13296,"num_write":1104,"num_write_kb":27083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"67'1296","reported_seq":3281,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.502845+0000","last_change":"2023-10-30T14:23:30.774013+0000","last_active":"2023-10-30T14:51:28.502845+0000","last_peered":"2023-10-30T14:51:28.502845+0000","last_clean":"2023-10-30T14:51:28.502845+0000","last_became_active":"2023-10-30T14:14:04.608190+0000","last_became_peered":"2023-10-30T14:14:04.608190+0000","last_unstale":"2023-10-30T14:51:28.502845+0000","last_undegraded":"2023-10-30T14:51:28.502845+0000","last_fullsized":"2023-10-30T14:51:28.502845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1296,"log_dups_size":0,"ondisk_log_size":1296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:44:40.113536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035618450000000001,"stat_sum":{"num_bytes":31744659,"num_objects":477,"num_object_clones":215,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":193,"num_read":85,"num_read_kb":23305,"num_write":1082,"num_write_kb":37194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"67'1351","reported_seq":3310,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469699+0000","last_change":"2023-10-30T14:23:30.779367+0000","last_active":"2023-10-30T14:51:29.469699+0000","last_peered":"2023-10-30T14:51:29.469699+0000","last_clean":"2023-10-30T14:51:29.469699+0000","last_became_active":"2023-10-30T14:14:04.641306+0000","last_became_peered":"2023-10-30T14:14:04.641306+0000","last_unstale":"2023-10-30T14:51:29.469699+0000","last_undegraded":"2023-10-30T14:51:29.469699+0000","last_fullsized":"2023-10-30T14:51:29.469699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1351,"log_dups_size":0,"ondisk_log_size":1351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:53:12.985530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066189990000000004,"stat_sum":{"num_bytes":3981038,"num_objects":511,"num_object_clones":236,"num_object_copies":1022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":206,"num_read":80,"num_read_kb":4151,"num_write":1127,"num_write_kb":9503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"67'1221","reported_seq":3204,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469982+0000","last_change":"2023-10-30T14:23:31.067340+0000","last_active":"2023-10-30T14:51:29.469982+0000","last_peered":"2023-10-30T14:51:29.469982+0000","last_clean":"2023-10-30T14:51:29.469982+0000","last_became_active":"2023-10-30T14:14:04.613644+0000","last_became_peered":"2023-10-30T14:14:04.613644+0000","last_unstale":"2023-10-30T14:51:29.469982+0000","last_undegraded":"2023-10-30T14:51:29.469982+0000","last_fullsized":"2023-10-30T14:51:29.469982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1221,"log_dups_size":0,"ondisk_log_size":1221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:50:40.405415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030699740000000001,"stat_sum":{"num_bytes":12387639,"num_objects":435,"num_object_clones":203,"num_object_copies":870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":435,"num_whiteouts":177,"num_read":71,"num_read_kb":8233,"num_write":1033,"num_write_kb":14118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"67'1285","reported_seq":3258,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.502929+0000","last_change":"2023-10-30T14:23:31.082413+0000","last_active":"2023-10-30T14:51:28.502929+0000","last_peered":"2023-10-30T14:51:28.502929+0000","last_clean":"2023-10-30T14:51:28.502929+0000","last_became_active":"2023-10-30T14:14:04.649850+0000","last_became_peered":"2023-10-30T14:14:04.649850+0000","last_unstale":"2023-10-30T14:51:28.502929+0000","last_undegraded":"2023-10-30T14:51:28.502929+0000","last_fullsized":"2023-10-30T14:51:28.502929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1285,"log_dups_size":0,"ondisk_log_size":1285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:27:56.738513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012599246,"stat_sum":{"num_bytes":25948792,"num_objects":470,"num_object_clones":219,"num_object_copies":940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":192,"num_read":77,"num_read_kb":24658,"num_write":1071,"num_write_kb":31961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"67'1335","reported_seq":3270,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.012858+0000","last_change":"2023-10-30T14:23:31.074724+0000","last_active":"2023-10-30T14:51:25.012858+0000","last_peered":"2023-10-30T14:51:25.012858+0000","last_clean":"2023-10-30T14:51:25.012858+0000","last_became_active":"2023-10-30T14:14:04.625042+0000","last_became_peered":"2023-10-30T14:14:04.625042+0000","last_unstale":"2023-10-30T14:51:25.012858+0000","last_undegraded":"2023-10-30T14:51:25.012858+0000","last_fullsized":"2023-10-30T14:51:25.012858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1335,"log_dups_size":0,"ondisk_log_size":1335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:45.303841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082351539999999997,"stat_sum":{"num_bytes":12700226,"num_objects":511,"num_object_clones":241,"num_object_copies":1022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":214,"num_read":61,"num_read_kb":10278,"num_write":1102,"num_write_kb":27066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"67'1264","reported_seq":3228,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.049982+0000","last_change":"2023-10-30T14:23:30.775419+0000","last_active":"2023-10-30T14:51:25.049982+0000","last_peered":"2023-10-30T14:51:25.049982+0000","last_clean":"2023-10-30T14:51:25.049982+0000","last_became_active":"2023-10-30T14:14:04.633586+0000","last_became_peered":"2023-10-30T14:14:04.633586+0000","last_unstale":"2023-10-30T14:51:25.049982+0000","last_undegraded":"2023-10-30T14:51:25.049982+0000","last_fullsized":"2023-10-30T14:51:25.049982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1264,"log_dups_size":0,"ondisk_log_size":1264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:19:06.449779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043616920000000003,"stat_sum":{"num_bytes":12325132,"num_objects":460,"num_object_clones":214,"num_object_copies":920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":192,"num_read":63,"num_read_kb":8213,"num_write":1055,"num_write_kb":14000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"67'1368","reported_seq":3340,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.046013+0000","last_change":"2023-10-30T14:23:30.777885+0000","last_active":"2023-10-30T14:51:25.046013+0000","last_peered":"2023-10-30T14:51:25.046013+0000","last_clean":"2023-10-30T14:51:25.046013+0000","last_became_active":"2023-10-30T14:14:04.642244+0000","last_became_peered":"2023-10-30T14:14:04.642244+0000","last_unstale":"2023-10-30T14:51:25.046013+0000","last_undegraded":"2023-10-30T14:51:25.046013+0000","last_fullsized":"2023-10-30T14:51:25.046013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1368,"log_dups_size":0,"ondisk_log_size":1368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:25.186063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065088710000000003,"stat_sum":{"num_bytes":25821219,"num_objects":511,"num_object_clones":233,"num_object_copies":1022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":210,"num_read":90,"num_read_kb":18405,"num_write":1129,"num_write_kb":31180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'1297","reported_seq":3278,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.499778+0000","last_change":"2023-10-30T14:23:31.070902+0000","last_active":"2023-10-30T14:51:28.499778+0000","last_peered":"2023-10-30T14:51:28.499778+0000","last_clean":"2023-10-30T14:51:28.499778+0000","last_became_active":"2023-10-30T14:14:04.620343+0000","last_became_peered":"2023-10-30T14:14:04.620343+0000","last_unstale":"2023-10-30T14:51:28.499778+0000","last_undegraded":"2023-10-30T14:51:28.499778+0000","last_fullsized":"2023-10-30T14:51:28.499778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1297,"log_dups_size":0,"ondisk_log_size":1297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:45:58.152823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048974149999999996,"stat_sum":{"num_bytes":12397335,"num_objects":479,"num_object_clones":220,"num_object_copies":958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":200,"num_read":86,"num_read_kb":8232,"num_write":1076,"num_write_kb":17925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'1353","reported_seq":3298,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.461566+0000","last_change":"2023-10-30T14:23:30.774642+0000","last_active":"2023-10-30T14:51:29.461566+0000","last_peered":"2023-10-30T14:51:29.461566+0000","last_clean":"2023-10-30T14:51:29.461566+0000","last_became_active":"2023-10-30T14:14:04.609720+0000","last_became_peered":"2023-10-30T14:14:04.609720+0000","last_unstale":"2023-10-30T14:51:29.461566+0000","last_undegraded":"2023-10-30T14:51:29.461566+0000","last_fullsized":"2023-10-30T14:51:29.461566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1353,"log_dups_size":0,"ondisk_log_size":1353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:10:58.430266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040008739999999997,"stat_sum":{"num_bytes":31017079,"num_objects":532,"num_object_clones":246,"num_object_copies":1064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":532,"num_whiteouts":228,"num_read":75,"num_read_kb":20512,"num_write":1096,"num_write_kb":31998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"67'1204","reported_seq":3182,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.455721+0000","last_change":"2023-10-30T14:23:31.069745+0000","last_active":"2023-10-30T14:51:29.455721+0000","last_peered":"2023-10-30T14:51:29.455721+0000","last_clean":"2023-10-30T14:51:29.455721+0000","last_became_active":"2023-10-30T14:14:04.620217+0000","last_became_peered":"2023-10-30T14:14:04.620217+0000","last_unstale":"2023-10-30T14:51:29.455721+0000","last_undegraded":"2023-10-30T14:51:29.455721+0000","last_fullsized":"2023-10-30T14:51:29.455721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:36:29.759579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044491130000000002,"stat_sum":{"num_bytes":20607496,"num_objects":451,"num_object_clones":209,"num_object_copies":902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":451,"num_whiteouts":184,"num_read":71,"num_read_kb":15956,"num_write":987,"num_write_kb":22411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"67'1350","reported_seq":3335,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.475769+0000","last_change":"2023-10-30T14:23:31.075832+0000","last_active":"2023-10-30T14:51:29.475769+0000","last_peered":"2023-10-30T14:51:29.475769+0000","last_clean":"2023-10-30T14:51:29.475769+0000","last_became_active":"2023-10-30T14:14:45.669878+0000","last_became_peered":"2023-10-30T14:14:45.669878+0000","last_unstale":"2023-10-30T14:51:29.475769+0000","last_undegraded":"2023-10-30T14:51:29.475769+0000","last_fullsized":"2023-10-30T14:51:29.475769+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1350,"log_dups_size":0,"ondisk_log_size":1350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:53:43.313641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0077997409999999998,"stat_sum":{"num_bytes":25371605,"num_objects":475,"num_object_clones":218,"num_object_copies":950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":193,"num_read":80,"num_read_kb":18178,"num_write":1139,"num_write_kb":27403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"67'1292","reported_seq":3264,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.057724+0000","last_change":"2023-10-30T14:23:30.773367+0000","last_active":"2023-10-30T14:51:25.057724+0000","last_peered":"2023-10-30T14:51:25.057724+0000","last_clean":"2023-10-30T14:51:25.057724+0000","last_became_active":"2023-10-30T14:14:04.597590+0000","last_became_peered":"2023-10-30T14:14:04.597590+0000","last_unstale":"2023-10-30T14:51:25.057724+0000","last_undegraded":"2023-10-30T14:51:25.057724+0000","last_fullsized":"2023-10-30T14:51:25.057724+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1292,"log_dups_size":0,"ondisk_log_size":1292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:28:58.351077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044830970000000001,"stat_sum":{"num_bytes":8698815,"num_objects":476,"num_object_clones":219,"num_object_copies":952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":476,"num_whiteouts":192,"num_read":75,"num_read_kb":4156,"num_write":1089,"num_write_kb":11768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"67'1181","reported_seq":3142,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.452118+0000","last_change":"2023-10-30T14:23:31.070930+0000","last_active":"2023-10-30T14:51:29.452118+0000","last_peered":"2023-10-30T14:51:29.452118+0000","last_clean":"2023-10-30T14:51:29.452118+0000","last_became_active":"2023-10-30T14:14:04.623902+0000","last_became_peered":"2023-10-30T14:14:04.623902+0000","last_unstale":"2023-10-30T14:51:29.452118+0000","last_undegraded":"2023-10-30T14:51:29.452118+0000","last_fullsized":"2023-10-30T14:51:29.452118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1181,"log_dups_size":0,"ondisk_log_size":1181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:13:07.040210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051343489999999999,"stat_sum":{"num_bytes":21106575,"num_objects":457,"num_object_clones":213,"num_object_copies":914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":457,"num_whiteouts":188,"num_read":58,"num_read_kb":12498,"num_write":966,"num_write_kb":25422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"67'1325","reported_seq":3275,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.632286+0000","last_change":"2023-10-30T14:23:30.775060+0000","last_active":"2023-10-30T14:51:29.632286+0000","last_peered":"2023-10-30T14:51:29.632286+0000","last_clean":"2023-10-30T14:51:29.632286+0000","last_became_active":"2023-10-30T14:14:04.597556+0000","last_became_peered":"2023-10-30T14:14:04.597556+0000","last_unstale":"2023-10-30T14:51:29.632286+0000","last_undegraded":"2023-10-30T14:51:29.632286+0000","last_fullsized":"2023-10-30T14:51:29.632286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1325,"log_dups_size":0,"ondisk_log_size":1325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:42.236111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040069729999999996,"stat_sum":{"num_bytes":25662090,"num_objects":516,"num_object_clones":239,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":516,"num_whiteouts":210,"num_read":73,"num_read_kb":17618,"num_write":1095,"num_write_kb":27367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"67'1302","reported_seq":3272,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.461498+0000","last_change":"2023-10-30T14:23:30.779406+0000","last_active":"2023-10-30T14:51:29.461498+0000","last_peered":"2023-10-30T14:51:29.461498+0000","last_clean":"2023-10-30T14:51:29.461498+0000","last_became_active":"2023-10-30T14:14:04.618430+0000","last_became_peered":"2023-10-30T14:14:04.618430+0000","last_unstale":"2023-10-30T14:51:29.461498+0000","last_undegraded":"2023-10-30T14:51:29.461498+0000","last_fullsized":"2023-10-30T14:51:29.461498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1302,"log_dups_size":0,"ondisk_log_size":1302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:44:00.464702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074984149999999996,"stat_sum":{"num_bytes":16389310,"num_objects":474,"num_object_clones":221,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":474,"num_whiteouts":197,"num_read":75,"num_read_kb":12323,"num_write":1094,"num_write_kb":17656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"67'1414","reported_seq":3372,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.458611+0000","last_change":"2023-10-30T14:23:31.067667+0000","last_active":"2023-10-30T14:51:29.458611+0000","last_peered":"2023-10-30T14:51:29.458611+0000","last_clean":"2023-10-30T14:51:29.458611+0000","last_became_active":"2023-10-30T14:14:04.606340+0000","last_became_peered":"2023-10-30T14:14:04.606340+0000","last_unstale":"2023-10-30T14:51:29.458611+0000","last_undegraded":"2023-10-30T14:51:29.458611+0000","last_fullsized":"2023-10-30T14:51:29.458611+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1414,"log_dups_size":0,"ondisk_log_size":1414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:41:14.948036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035094890000000002,"stat_sum":{"num_bytes":22345046,"num_objects":524,"num_object_clones":240,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":211,"num_read":82,"num_read_kb":9614,"num_write":1184,"num_write_kb":32857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"67'1333","reported_seq":3309,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.486748+0000","last_change":"2023-10-30T14:23:31.074353+0000","last_active":"2023-10-30T14:51:28.486748+0000","last_peered":"2023-10-30T14:51:28.486748+0000","last_clean":"2023-10-30T14:51:28.486748+0000","last_became_active":"2023-10-30T14:14:04.612588+0000","last_became_peered":"2023-10-30T14:14:04.612588+0000","last_unstale":"2023-10-30T14:51:28.486748+0000","last_undegraded":"2023-10-30T14:51:28.486748+0000","last_fullsized":"2023-10-30T14:51:28.486748+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1333,"log_dups_size":0,"ondisk_log_size":1333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:56:59.526944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084570800000000005,"stat_sum":{"num_bytes":13192668,"num_objects":479,"num_object_clones":223,"num_object_copies":958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":199,"num_read":83,"num_read_kb":8248,"num_write":1116,"num_write_kb":15001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"67'1277","reported_seq":3244,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.457130+0000","last_change":"2023-10-30T14:23:30.955845+0000","last_active":"2023-10-30T14:51:29.457130+0000","last_peered":"2023-10-30T14:51:29.457130+0000","last_clean":"2023-10-30T14:51:29.457130+0000","last_became_active":"2023-10-30T14:14:04.607683+0000","last_became_peered":"2023-10-30T14:14:04.607683+0000","last_unstale":"2023-10-30T14:51:29.457130+0000","last_undegraded":"2023-10-30T14:51:29.457130+0000","last_fullsized":"2023-10-30T14:51:29.457130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1277,"log_dups_size":0,"ondisk_log_size":1277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:41.822862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001856688,"stat_sum":{"num_bytes":21370110,"num_objects":478,"num_object_clones":220,"num_object_copies":956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":200,"num_read":74,"num_read_kb":12305,"num_write":1050,"num_write_kb":24643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'640","reported_seq":808,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.464421+0000","last_change":"2023-10-30T14:14:03.732084+0000","last_active":"2023-10-30T14:51:29.464421+0000","last_peered":"2023-10-30T14:51:29.464421+0000","last_clean":"2023-10-30T14:51:29.464421+0000","last_became_active":"2023-10-30T14:14:03.731740+0000","last_became_peered":"2023-10-30T14:14:03.731740+0000","last_unstale":"2023-10-30T14:51:29.464421+0000","last_undegraded":"2023-10-30T14:51:29.464421+0000","last_fullsized":"2023-10-30T14:51:29.464421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:34:27.462477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137426,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":149,"num_read_kb":143937,"num_write":662,"num_write_kb":70557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"67'1312","reported_seq":3279,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.631540+0000","last_change":"2023-10-30T14:23:30.776069+0000","last_active":"2023-10-30T14:51:29.631540+0000","last_peered":"2023-10-30T14:51:29.631540+0000","last_clean":"2023-10-30T14:51:29.631540+0000","last_became_active":"2023-10-30T14:14:04.621526+0000","last_became_peered":"2023-10-30T14:14:04.621526+0000","last_unstale":"2023-10-30T14:51:29.631540+0000","last_undegraded":"2023-10-30T14:51:29.631540+0000","last_fullsized":"2023-10-30T14:51:29.631540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1312,"log_dups_size":0,"ondisk_log_size":1312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:59:59.926386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047814620000000002,"stat_sum":{"num_bytes":12410003,"num_objects":492,"num_object_clones":225,"num_object_copies":984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":492,"num_whiteouts":188,"num_read":79,"num_read_kb":8245,"num_write":1121,"num_write_kb":15268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"67'1480","reported_seq":1960,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.231740+0000","last_change":"2023-10-30T14:14:03.730789+0000","last_active":"2023-10-30T14:51:30.231740+0000","last_peered":"2023-10-30T14:51:30.231740+0000","last_clean":"2023-10-30T14:51:30.231740+0000","last_became_active":"2023-10-30T14:14:03.729386+0000","last_became_peered":"2023-10-30T14:14:03.729386+0000","last_unstale":"2023-10-30T14:51:30.231740+0000","last_undegraded":"2023-10-30T14:51:30.231740+0000","last_fullsized":"2023-10-30T14:51:30.231740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1480,"log_dups_size":0,"ondisk_log_size":1480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29761637,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":457,"num_read_kb":164028,"num_write":1504,"num_write_kb":71675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'761","reported_seq":1325,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.022393+0000","last_change":"2023-10-30T14:14:03.724379+0000","last_active":"2023-10-30T14:51:25.022393+0000","last_peered":"2023-10-30T14:51:25.022393+0000","last_clean":"2023-10-30T14:51:25.022393+0000","last_became_active":"2023-10-30T14:14:03.724078+0000","last_became_peered":"2023-10-30T14:14:03.724078+0000","last_unstale":"2023-10-30T14:51:25.022393+0000","last_undegraded":"2023-10-30T14:51:25.022393+0000","last_fullsized":"2023-10-30T14:51:25.022393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":761,"log_dups_size":0,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:10:17.829899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30316903,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":547,"num_read_kb":63175,"num_write":793,"num_write_kb":47477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"67'1430","reported_seq":3365,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.433623+0000","last_change":"2023-10-30T14:23:31.068462+0000","last_active":"2023-10-30T14:51:29.433623+0000","last_peered":"2023-10-30T14:51:29.433623+0000","last_clean":"2023-10-30T14:51:29.433623+0000","last_became_active":"2023-10-30T14:14:04.597099+0000","last_became_peered":"2023-10-30T14:14:04.597099+0000","last_unstale":"2023-10-30T14:51:29.433623+0000","last_undegraded":"2023-10-30T14:51:29.433623+0000","last_fullsized":"2023-10-30T14:51:29.433623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1430,"log_dups_size":0,"ondisk_log_size":1430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:04:00.818307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033627510000000002,"stat_sum":{"num_bytes":14915943,"num_objects":541,"num_object_clones":248,"num_object_copies":1082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":541,"num_whiteouts":226,"num_read":69,"num_read_kb":8270,"num_write":1175,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"67'1579","reported_seq":1742,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.595991+0000","last_change":"2023-10-30T14:14:03.721030+0000","last_active":"2023-10-30T14:51:25.595991+0000","last_peered":"2023-10-30T14:51:25.595991+0000","last_clean":"2023-10-30T14:51:25.595991+0000","last_became_active":"2023-10-30T14:14:03.719816+0000","last_became_peered":"2023-10-30T14:14:03.719816+0000","last_unstale":"2023-10-30T14:51:25.595991+0000","last_undegraded":"2023-10-30T14:51:25.595991+0000","last_fullsized":"2023-10-30T14:51:25.595991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1579,"log_dups_size":0,"ondisk_log_size":1579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:43:50.150156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35358900,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":164,"num_read_kb":135701,"num_write":1591,"num_write_kb":77227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"67'1277","reported_seq":3242,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.437783+0000","last_change":"2023-10-30T14:23:30.771109+0000","last_active":"2023-10-30T14:51:29.437783+0000","last_peered":"2023-10-30T14:51:29.437783+0000","last_clean":"2023-10-30T14:51:29.437783+0000","last_became_active":"2023-10-30T14:14:04.599323+0000","last_became_peered":"2023-10-30T14:14:04.599323+0000","last_unstale":"2023-10-30T14:51:29.437783+0000","last_undegraded":"2023-10-30T14:51:29.437783+0000","last_fullsized":"2023-10-30T14:51:29.437783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1277,"log_dups_size":0,"ondisk_log_size":1277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:27.656629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025427000000000002,"stat_sum":{"num_bytes":25763183,"num_objects":478,"num_object_clones":224,"num_object_copies":956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":194,"num_read":75,"num_read_kb":21385,"num_write":1050,"num_write_kb":37074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"67'640","reported_seq":767,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.998751+0000","last_change":"2023-10-30T14:14:03.738694+0000","last_active":"2023-10-30T14:51:30.998751+0000","last_peered":"2023-10-30T14:51:30.998751+0000","last_clean":"2023-10-30T14:51:30.998751+0000","last_became_active":"2023-10-30T14:14:03.738132+0000","last_became_peered":"2023-10-30T14:14:03.738132+0000","last_unstale":"2023-10-30T14:51:30.998751+0000","last_undegraded":"2023-10-30T14:51:30.998751+0000","last_fullsized":"2023-10-30T14:51:30.998751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:46:59.692900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360218,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":99,"num_read_kb":53823,"num_write":691,"num_write_kb":33991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"67'1295","reported_seq":3273,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.505595+0000","last_change":"2023-10-30T14:23:30.778832+0000","last_active":"2023-10-30T14:51:28.505595+0000","last_peered":"2023-10-30T14:51:28.505595+0000","last_clean":"2023-10-30T14:51:28.505595+0000","last_became_active":"2023-10-30T14:14:04.648816+0000","last_became_peered":"2023-10-30T14:14:04.648816+0000","last_unstale":"2023-10-30T14:51:28.505595+0000","last_undegraded":"2023-10-30T14:51:28.505595+0000","last_fullsized":"2023-10-30T14:51:28.505595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:20.078590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069848840000000002,"stat_sum":{"num_bytes":12063580,"num_objects":456,"num_object_clones":213,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":456,"num_whiteouts":185,"num_read":77,"num_read_kb":8211,"num_write":1082,"num_write_kb":14557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"67'1049","reported_seq":1186,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469669+0000","last_change":"2023-10-30T14:14:03.726112+0000","last_active":"2023-10-30T14:51:29.469669+0000","last_peered":"2023-10-30T14:51:29.469669+0000","last_clean":"2023-10-30T14:51:29.469669+0000","last_became_active":"2023-10-30T14:14:03.725272+0000","last_became_peered":"2023-10-30T14:14:03.725272+0000","last_unstale":"2023-10-30T14:51:29.469669+0000","last_undegraded":"2023-10-30T14:51:29.469669+0000","last_fullsized":"2023-10-30T14:51:29.469669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1049,"log_dups_size":0,"ondisk_log_size":1049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:37:20.764294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":123,"num_read_kb":86587,"num_write":1102,"num_write_kb":54778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"67'1384","reported_seq":3308,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.501832+0000","last_change":"2023-10-30T14:23:30.776486+0000","last_active":"2023-10-30T14:51:28.501832+0000","last_peered":"2023-10-30T14:51:28.501832+0000","last_clean":"2023-10-30T14:51:28.501832+0000","last_became_active":"2023-10-30T14:14:04.609697+0000","last_became_peered":"2023-10-30T14:14:04.609697+0000","last_unstale":"2023-10-30T14:51:28.501832+0000","last_undegraded":"2023-10-30T14:51:28.501832+0000","last_fullsized":"2023-10-30T14:51:28.501832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1384,"log_dups_size":0,"ondisk_log_size":1384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:45:56.464708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043395700000000001,"stat_sum":{"num_bytes":12432092,"num_objects":547,"num_object_clones":258,"num_object_copies":1094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":547,"num_whiteouts":233,"num_read":68,"num_read_kb":4261,"num_write":1136,"num_write_kb":13809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"67'1004","reported_seq":1165,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.048045+0000","last_change":"2023-10-30T14:14:03.727871+0000","last_active":"2023-10-30T14:51:25.048045+0000","last_peered":"2023-10-30T14:51:25.048045+0000","last_clean":"2023-10-30T14:51:25.048045+0000","last_became_active":"2023-10-30T14:14:03.727422+0000","last_became_peered":"2023-10-30T14:14:03.727422+0000","last_unstale":"2023-10-30T14:51:25.048045+0000","last_undegraded":"2023-10-30T14:51:25.048045+0000","last_fullsized":"2023-10-30T14:51:25.048045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1004,"log_dups_size":0,"ondisk_log_size":1004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:54.782533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389280,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":161,"num_read_kb":61498,"num_write":1080,"num_write_kb":30288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1480","reported_seq":3358,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.465294+0000","last_change":"2023-10-30T14:23:30.957793+0000","last_active":"2023-10-30T14:51:29.465294+0000","last_peered":"2023-10-30T14:51:29.465294+0000","last_clean":"2023-10-30T14:51:29.465294+0000","last_became_active":"2023-10-30T14:14:04.645333+0000","last_became_peered":"2023-10-30T14:14:04.645333+0000","last_unstale":"2023-10-30T14:51:29.465294+0000","last_undegraded":"2023-10-30T14:51:29.465294+0000","last_fullsized":"2023-10-30T14:51:29.465294+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1480,"log_dups_size":0,"ondisk_log_size":1480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:13:02.147712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002301474,"stat_sum":{"num_bytes":21687884,"num_objects":594,"num_object_clones":315,"num_object_copies":1188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":594,"num_whiteouts":221,"num_read":78,"num_read_kb":5632,"num_write":1148,"num_write_kb":28133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"67'579","reported_seq":766,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:27.713419+0000","last_change":"2023-10-30T14:14:03.725554+0000","last_active":"2023-10-30T14:51:27.713419+0000","last_peered":"2023-10-30T14:51:27.713419+0000","last_clean":"2023-10-30T14:51:27.713419+0000","last_became_active":"2023-10-30T14:14:03.724806+0000","last_became_peered":"2023-10-30T14:14:03.724806+0000","last_unstale":"2023-10-30T14:51:27.713419+0000","last_undegraded":"2023-10-30T14:51:27.713419+0000","last_fullsized":"2023-10-30T14:51:27.713419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:37.823561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360800,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":192,"num_read_kb":131840,"num_write":657,"num_write_kb":54408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"67'1313","reported_seq":3260,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.464551+0000","last_change":"2023-10-30T14:23:31.073841+0000","last_active":"2023-10-30T14:51:29.464551+0000","last_peered":"2023-10-30T14:51:29.464551+0000","last_clean":"2023-10-30T14:51:29.464551+0000","last_became_active":"2023-10-30T14:14:04.646543+0000","last_became_peered":"2023-10-30T14:14:04.646543+0000","last_unstale":"2023-10-30T14:51:29.464551+0000","last_undegraded":"2023-10-30T14:51:29.464551+0000","last_fullsized":"2023-10-30T14:51:29.464551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1313,"log_dups_size":0,"ondisk_log_size":1313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:30:29.655398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00744474,"stat_sum":{"num_bytes":21739186,"num_objects":502,"num_object_clones":232,"num_object_copies":1004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":502,"num_whiteouts":205,"num_read":65,"num_read_kb":16406,"num_write":1092,"num_write_kb":23651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"67'683","reported_seq":858,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.629444+0000","last_change":"2023-10-30T14:14:03.717903+0000","last_active":"2023-10-30T14:51:29.629444+0000","last_peered":"2023-10-30T14:51:29.629444+0000","last_clean":"2023-10-30T14:51:29.629444+0000","last_became_active":"2023-10-30T14:14:03.716941+0000","last_became_peered":"2023-10-30T14:14:03.716941+0000","last_unstale":"2023-10-30T14:51:29.629444+0000","last_undegraded":"2023-10-30T14:51:29.629444+0000","last_fullsized":"2023-10-30T14:51:29.629444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":683,"log_dups_size":0,"ondisk_log_size":683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:58.178166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777250,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":201,"num_read_kb":82705,"num_write":761,"num_write_kb":38720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"67'1396","reported_seq":3336,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:26.996443+0000","last_change":"2023-10-30T14:23:30.775738+0000","last_active":"2023-10-30T14:51:26.996443+0000","last_peered":"2023-10-30T14:51:26.996443+0000","last_clean":"2023-10-30T14:51:26.996443+0000","last_became_active":"2023-10-30T14:14:04.641494+0000","last_became_peered":"2023-10-30T14:14:04.641494+0000","last_unstale":"2023-10-30T14:51:26.996443+0000","last_undegraded":"2023-10-30T14:51:26.996443+0000","last_fullsized":"2023-10-30T14:51:26.996443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1396,"log_dups_size":0,"ondisk_log_size":1396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:44.138968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039664019999999999,"stat_sum":{"num_bytes":26403621,"num_objects":532,"num_object_clones":244,"num_object_copies":1064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":532,"num_whiteouts":206,"num_read":69,"num_read_kb":14198,"num_write":1171,"num_write_kb":36700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"67'300","reported_seq":421,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.447336+0000","last_change":"2023-10-30T14:14:03.729626+0000","last_active":"2023-10-30T14:51:29.447336+0000","last_peered":"2023-10-30T14:51:29.447336+0000","last_clean":"2023-10-30T14:51:29.447336+0000","last_became_active":"2023-10-30T14:14:03.728053+0000","last_became_peered":"2023-10-30T14:14:03.728053+0000","last_unstale":"2023-10-30T14:51:29.447336+0000","last_undegraded":"2023-10-30T14:51:29.447336+0000","last_fullsized":"2023-10-30T14:51:29.447336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:44:53.362066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":90,"num_read_kb":49506,"num_write":340,"num_write_kb":25387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1391","reported_seq":3353,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:27.468452+0000","last_change":"2023-10-30T14:23:31.078295+0000","last_active":"2023-10-30T14:51:27.468452+0000","last_peered":"2023-10-30T14:51:27.468452+0000","last_clean":"2023-10-30T14:51:27.468452+0000","last_became_active":"2023-10-30T14:14:04.649896+0000","last_became_peered":"2023-10-30T14:14:04.649896+0000","last_unstale":"2023-10-30T14:51:27.468452+0000","last_undegraded":"2023-10-30T14:51:27.468452+0000","last_fullsized":"2023-10-30T14:51:27.468452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1391,"log_dups_size":0,"ondisk_log_size":1391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:07:44.011495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010792094,"stat_sum":{"num_bytes":9382763,"num_objects":511,"num_object_clones":240,"num_object_copies":1022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":210,"num_read":88,"num_read_kb":2154,"num_write":1160,"num_write_kb":10817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"67'712","reported_seq":848,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.468850+0000","last_change":"2023-10-30T14:14:03.736824+0000","last_active":"2023-10-30T14:51:29.468850+0000","last_peered":"2023-10-30T14:51:29.468850+0000","last_clean":"2023-10-30T14:51:29.468850+0000","last_became_active":"2023-10-30T14:14:03.736155+0000","last_became_peered":"2023-10-30T14:14:03.736155+0000","last_unstale":"2023-10-30T14:51:29.468850+0000","last_undegraded":"2023-10-30T14:51:29.468850+0000","last_fullsized":"2023-10-30T14:51:29.468850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:35.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":104,"num_read_kb":67382,"num_write":738,"num_write_kb":42054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"67'1257","reported_seq":3230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.475777+0000","last_change":"2023-10-30T14:23:31.076487+0000","last_active":"2023-10-30T14:51:28.475777+0000","last_peered":"2023-10-30T14:51:28.475777+0000","last_clean":"2023-10-30T14:51:28.475777+0000","last_became_active":"2023-10-30T14:14:04.614328+0000","last_became_peered":"2023-10-30T14:14:04.614328+0000","last_unstale":"2023-10-30T14:51:28.475777+0000","last_undegraded":"2023-10-30T14:51:28.475777+0000","last_fullsized":"2023-10-30T14:51:28.475777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1257,"log_dups_size":0,"ondisk_log_size":1257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:04:54.384597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010029286,"stat_sum":{"num_bytes":9023665,"num_objects":474,"num_object_clones":219,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":474,"num_whiteouts":189,"num_read":77,"num_read_kb":4169,"num_write":1052,"num_write_kb":11363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"67'1214","reported_seq":1398,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.435186+0000","last_change":"2023-10-30T14:14:03.700690+0000","last_active":"2023-10-30T14:51:29.435186+0000","last_peered":"2023-10-30T14:51:29.435186+0000","last_clean":"2023-10-30T14:51:29.435186+0000","last_became_active":"2023-10-30T14:14:03.700352+0000","last_became_peered":"2023-10-30T14:14:03.700352+0000","last_unstale":"2023-10-30T14:51:29.435186+0000","last_undegraded":"2023-10-30T14:51:29.435186+0000","last_fullsized":"2023-10-30T14:51:29.435186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1214,"log_dups_size":0,"ondisk_log_size":1214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:03:01.103359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":252,"num_read_kb":55463,"num_write":1288,"num_write_kb":26279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"67'1354","reported_seq":3315,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.447744+0000","last_change":"2023-10-30T14:23:31.079218+0000","last_active":"2023-10-30T14:51:28.447744+0000","last_peered":"2023-10-30T14:51:28.447744+0000","last_clean":"2023-10-30T14:51:28.447744+0000","last_became_active":"2023-10-30T14:14:04.646639+0000","last_became_peered":"2023-10-30T14:14:04.646639+0000","last_unstale":"2023-10-30T14:51:28.447744+0000","last_undegraded":"2023-10-30T14:51:28.447744+0000","last_fullsized":"2023-10-30T14:51:28.447744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1354,"log_dups_size":0,"ondisk_log_size":1354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:32.477580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010379724,"stat_sum":{"num_bytes":13446392,"num_objects":502,"num_object_clones":231,"num_object_copies":1004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":502,"num_whiteouts":206,"num_read":77,"num_read_kb":12340,"num_write":1116,"num_write_kb":24186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'643","reported_seq":768,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.271430+0000","last_change":"2023-10-30T14:14:03.710717+0000","last_active":"2023-10-30T14:51:25.271430+0000","last_peered":"2023-10-30T14:51:25.271430+0000","last_clean":"2023-10-30T14:51:25.271430+0000","last_became_active":"2023-10-30T14:14:03.707950+0000","last_became_peered":"2023-10-30T14:14:03.707950+0000","last_unstale":"2023-10-30T14:51:25.271430+0000","last_undegraded":"2023-10-30T14:51:25.271430+0000","last_fullsized":"2023-10-30T14:51:25.271430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:29:41.232902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971542,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":82,"num_read_kb":57693,"num_write":882,"num_write_kb":42351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"67'1239","reported_seq":3190,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.012771+0000","last_change":"2023-10-30T14:23:31.080966+0000","last_active":"2023-10-30T14:51:25.012771+0000","last_peered":"2023-10-30T14:51:25.012771+0000","last_clean":"2023-10-30T14:51:25.012771+0000","last_became_active":"2023-10-30T14:14:04.644761+0000","last_became_peered":"2023-10-30T14:14:04.644761+0000","last_unstale":"2023-10-30T14:51:25.012771+0000","last_undegraded":"2023-10-30T14:51:25.012771+0000","last_fullsized":"2023-10-30T14:51:25.012771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1239,"log_dups_size":0,"ondisk_log_size":1239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:55:03.102431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011171970999999999,"stat_sum":{"num_bytes":12914479,"num_objects":495,"num_object_clones":226,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":208,"num_read":62,"num_read_kb":4133,"num_write":998,"num_write_kb":18863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'535","reported_seq":666,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.248852+0000","last_change":"2023-10-30T14:14:03.705567+0000","last_active":"2023-10-30T14:51:30.248852+0000","last_peered":"2023-10-30T14:51:30.248852+0000","last_clean":"2023-10-30T14:51:30.248852+0000","last_became_active":"2023-10-30T14:14:03.705186+0000","last_became_peered":"2023-10-30T14:14:03.705186+0000","last_unstale":"2023-10-30T14:51:30.248852+0000","last_undegraded":"2023-10-30T14:51:30.248852+0000","last_fullsized":"2023-10-30T14:51:30.248852+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":535,"log_dups_size":0,"ondisk_log_size":535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:14.609243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":102,"num_read_kb":90592,"num_write":573,"num_write_kb":50573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"67'1299","reported_seq":3247,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.473089+0000","last_change":"2023-10-30T14:23:31.083260+0000","last_active":"2023-10-30T14:51:29.473089+0000","last_peered":"2023-10-30T14:51:29.473089+0000","last_clean":"2023-10-30T14:51:29.473089+0000","last_became_active":"2023-10-30T14:14:04.646785+0000","last_became_peered":"2023-10-30T14:14:04.646785+0000","last_unstale":"2023-10-30T14:51:29.473089+0000","last_undegraded":"2023-10-30T14:51:29.473089+0000","last_fullsized":"2023-10-30T14:51:29.473089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1299,"log_dups_size":0,"ondisk_log_size":1299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:52:21.536104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010111184,"stat_sum":{"num_bytes":17345938,"num_objects":504,"num_object_clones":239,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":504,"num_whiteouts":216,"num_read":72,"num_read_kb":10698,"num_write":1034,"num_write_kb":23411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"67'598","reported_seq":748,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.425928+0000","last_change":"2023-10-30T14:14:03.693349+0000","last_active":"2023-10-30T14:51:29.425928+0000","last_peered":"2023-10-30T14:51:29.425928+0000","last_clean":"2023-10-30T14:51:29.425928+0000","last_became_active":"2023-10-30T14:14:03.692966+0000","last_became_peered":"2023-10-30T14:14:03.692966+0000","last_unstale":"2023-10-30T14:51:29.425928+0000","last_undegraded":"2023-10-30T14:51:29.425928+0000","last_fullsized":"2023-10-30T14:51:29.425928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":598,"log_dups_size":0,"ondisk_log_size":598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:30:28.800214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16780852,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":126,"num_read_kb":110211,"num_write":638,"num_write_kb":50143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'1328","reported_seq":3304,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.461147+0000","last_change":"2023-10-30T14:23:30.778622+0000","last_active":"2023-10-30T14:51:29.461147+0000","last_peered":"2023-10-30T14:51:29.461147+0000","last_clean":"2023-10-30T14:51:29.461147+0000","last_became_active":"2023-10-30T14:14:04.615925+0000","last_became_peered":"2023-10-30T14:14:04.615925+0000","last_unstale":"2023-10-30T14:51:29.461147+0000","last_undegraded":"2023-10-30T14:51:29.461147+0000","last_fullsized":"2023-10-30T14:51:29.461147+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1328,"log_dups_size":0,"ondisk_log_size":1328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:54.760495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068317569999999999,"stat_sum":{"num_bytes":4298895,"num_objects":483,"num_object_clones":223,"num_object_copies":966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":483,"num_whiteouts":204,"num_read":84,"num_read_kb":56,"num_write":1103,"num_write_kb":8864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"67'667","reported_seq":805,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.244959+0000","last_change":"2023-10-30T14:14:03.727659+0000","last_active":"2023-10-30T14:51:25.244959+0000","last_peered":"2023-10-30T14:51:25.244959+0000","last_clean":"2023-10-30T14:51:25.244959+0000","last_became_active":"2023-10-30T14:14:03.727409+0000","last_became_peered":"2023-10-30T14:14:03.727409+0000","last_unstale":"2023-10-30T14:51:25.244959+0000","last_undegraded":"2023-10-30T14:51:25.244959+0000","last_fullsized":"2023-10-30T14:51:25.244959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:24.210635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166406,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":118,"num_read_kb":66012,"num_write":697,"num_write_kb":50302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1282","reported_seq":3257,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.467914+0000","last_change":"2023-10-30T14:23:31.077576+0000","last_active":"2023-10-30T14:51:29.467914+0000","last_peered":"2023-10-30T14:51:29.467914+0000","last_clean":"2023-10-30T14:51:29.467914+0000","last_became_active":"2023-10-30T14:14:45.818032+0000","last_became_peered":"2023-10-30T14:14:45.818032+0000","last_unstale":"2023-10-30T14:51:29.467914+0000","last_undegraded":"2023-10-30T14:51:29.467914+0000","last_fullsized":"2023-10-30T14:51:29.467914+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1282,"log_dups_size":0,"ondisk_log_size":1282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:54:03.288491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010655906,"stat_sum":{"num_bytes":23795405,"num_objects":481,"num_object_clones":221,"num_object_copies":962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":199,"num_read":74,"num_read_kb":13429,"num_write":1070,"num_write_kb":29341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":229,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T14:49:57.449030+0000","last_change":"2023-10-30T14:13:54.251572+0000","last_active":"2023-10-30T14:49:57.449030+0000","last_peered":"2023-10-30T14:49:57.449030+0000","last_clean":"2023-10-30T14:49:57.449030+0000","last_became_active":"2023-10-30T14:13:54.250907+0000","last_became_peered":"2023-10-30T14:13:54.250907+0000","last_unstale":"2023-10-30T14:49:57.449030+0000","last_undegraded":"2023-10-30T14:49:57.449030+0000","last_fullsized":"2023-10-30T14:49:57.449030+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_clean_scrub_stamp":"2023-10-30T14:13:53.220267+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:44.590852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"67'724","reported_seq":867,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.024538+0000","last_change":"2023-10-30T14:14:03.729034+0000","last_active":"2023-10-30T14:51:25.024538+0000","last_peered":"2023-10-30T14:51:25.024538+0000","last_clean":"2023-10-30T14:51:25.024538+0000","last_became_active":"2023-10-30T14:14:03.728293+0000","last_became_peered":"2023-10-30T14:14:03.728293+0000","last_unstale":"2023-10-30T14:51:25.024538+0000","last_undegraded":"2023-10-30T14:51:25.024538+0000","last_fullsized":"2023-10-30T14:51:25.024538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":724,"log_dups_size":0,"ondisk_log_size":724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165954,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":121,"num_read_kb":78313,"num_write":794,"num_write_kb":42213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"67'1244","reported_seq":3209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.448701+0000","last_change":"2023-10-30T14:23:31.067976+0000","last_active":"2023-10-30T14:51:29.448701+0000","last_peered":"2023-10-30T14:51:29.448701+0000","last_clean":"2023-10-30T14:51:29.448701+0000","last_became_active":"2023-10-30T14:14:46.451620+0000","last_became_peered":"2023-10-30T14:14:46.451620+0000","last_unstale":"2023-10-30T14:51:29.448701+0000","last_undegraded":"2023-10-30T14:51:29.448701+0000","last_fullsized":"2023-10-30T14:51:29.448701+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1244,"log_dups_size":0,"ondisk_log_size":1244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:10.312892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034002070000000001,"stat_sum":{"num_bytes":12494676,"num_objects":466,"num_object_clones":213,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":182,"num_read":65,"num_read_kb":8203,"num_write":1045,"num_write_kb":18288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"67'823","reported_seq":1009,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.517181+0000","last_change":"2023-10-30T14:14:03.732909+0000","last_active":"2023-10-30T14:51:29.517181+0000","last_peered":"2023-10-30T14:51:29.517181+0000","last_clean":"2023-10-30T14:51:29.517181+0000","last_became_active":"2023-10-30T14:14:03.732199+0000","last_became_peered":"2023-10-30T14:14:03.732199+0000","last_unstale":"2023-10-30T14:51:29.517181+0000","last_undegraded":"2023-10-30T14:51:29.517181+0000","last_fullsized":"2023-10-30T14:51:29.517181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":823,"log_dups_size":0,"ondisk_log_size":823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:38.117750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":198,"num_read_kb":136195,"num_write":895,"num_write_kb":59080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"67'1255","reported_seq":3217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.505461+0000","last_change":"2023-10-30T14:23:30.775385+0000","last_active":"2023-10-30T14:51:28.505461+0000","last_peered":"2023-10-30T14:51:28.505461+0000","last_clean":"2023-10-30T14:51:28.505461+0000","last_became_active":"2023-10-30T14:14:04.619990+0000","last_became_peered":"2023-10-30T14:14:04.619990+0000","last_unstale":"2023-10-30T14:51:28.505461+0000","last_undegraded":"2023-10-30T14:51:28.505461+0000","last_fullsized":"2023-10-30T14:51:28.505461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1255,"log_dups_size":0,"ondisk_log_size":1255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:42.711461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045963250000000001,"stat_sum":{"num_bytes":8263423,"num_objects":470,"num_object_clones":217,"num_object_copies":940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":194,"num_read":63,"num_read_kb":1576,"num_write":1033,"num_write_kb":10839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"67'1750","reported_seq":2853,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.563109+0000","last_change":"2023-10-30T14:14:03.702862+0000","last_active":"2023-10-30T14:51:29.563109+0000","last_peered":"2023-10-30T14:51:29.563109+0000","last_clean":"2023-10-30T14:51:29.563109+0000","last_became_active":"2023-10-30T14:14:03.702513+0000","last_became_peered":"2023-10-30T14:14:03.702513+0000","last_unstale":"2023-10-30T14:51:29.563109+0000","last_undegraded":"2023-10-30T14:51:29.563109+0000","last_fullsized":"2023-10-30T14:51:29.563109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1750,"log_dups_size":0,"ondisk_log_size":1750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:25.499390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27766724,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":1109,"num_read_kb":113686,"num_write":1839,"num_write_kb":62085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"67'1316","reported_seq":3268,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.468492+0000","last_change":"2023-10-30T14:23:30.770719+0000","last_active":"2023-10-30T14:51:28.468492+0000","last_peered":"2023-10-30T14:51:28.468492+0000","last_clean":"2023-10-30T14:51:28.468492+0000","last_became_active":"2023-10-30T14:14:04.589087+0000","last_became_peered":"2023-10-30T14:14:04.589087+0000","last_unstale":"2023-10-30T14:51:28.468492+0000","last_undegraded":"2023-10-30T14:51:28.468492+0000","last_fullsized":"2023-10-30T14:51:28.468492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1316,"log_dups_size":0,"ondisk_log_size":1316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:44:54.038742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022871390000000001,"stat_sum":{"num_bytes":30600386,"num_objects":508,"num_object_clones":243,"num_object_copies":1016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":508,"num_whiteouts":221,"num_read":78,"num_read_kb":20511,"num_write":1060,"num_write_kb":36437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"67'633","reported_seq":782,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.234880+0000","last_change":"2023-10-30T14:14:03.730048+0000","last_active":"2023-10-30T14:51:30.234880+0000","last_peered":"2023-10-30T14:51:30.234880+0000","last_clean":"2023-10-30T14:51:30.234880+0000","last_became_active":"2023-10-30T14:14:03.728925+0000","last_became_peered":"2023-10-30T14:14:03.728925+0000","last_unstale":"2023-10-30T14:51:30.234880+0000","last_undegraded":"2023-10-30T14:51:30.234880+0000","last_fullsized":"2023-10-30T14:51:30.234880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":633,"log_dups_size":0,"ondisk_log_size":633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:32:05.056039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":120,"num_read_kb":131794,"num_write":657,"num_write_kb":62419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"67'1355","reported_seq":3305,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.495764+0000","last_change":"2023-10-30T14:23:30.956956+0000","last_active":"2023-10-30T14:51:28.495764+0000","last_peered":"2023-10-30T14:51:28.495764+0000","last_clean":"2023-10-30T14:51:28.495764+0000","last_became_active":"2023-10-30T14:14:04.607377+0000","last_became_peered":"2023-10-30T14:14:04.607377+0000","last_unstale":"2023-10-30T14:51:28.495764+0000","last_undegraded":"2023-10-30T14:51:28.495764+0000","last_fullsized":"2023-10-30T14:51:28.495764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1355,"log_dups_size":0,"ondisk_log_size":1355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:58:23.634836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022021179999999999,"stat_sum":{"num_bytes":18058258,"num_objects":500,"num_object_clones":231,"num_object_copies":1000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":500,"num_whiteouts":206,"num_read":65,"num_read_kb":9172,"num_write":1119,"num_write_kb":20820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"67'880","reported_seq":1017,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.230091+0000","last_change":"2023-10-30T14:14:03.693753+0000","last_active":"2023-10-30T14:51:30.230091+0000","last_peered":"2023-10-30T14:51:30.230091+0000","last_clean":"2023-10-30T14:51:30.230091+0000","last_became_active":"2023-10-30T14:14:03.692621+0000","last_became_peered":"2023-10-30T14:14:03.692621+0000","last_unstale":"2023-10-30T14:51:30.230091+0000","last_undegraded":"2023-10-30T14:51:30.230091+0000","last_fullsized":"2023-10-30T14:51:30.230091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:32:28.577584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":107,"num_read_kb":66236,"num_write":920,"num_write_kb":58565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'1338","reported_seq":3267,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.493889+0000","last_change":"2023-10-30T14:23:30.780258+0000","last_active":"2023-10-30T14:51:28.493889+0000","last_peered":"2023-10-30T14:51:28.493889+0000","last_clean":"2023-10-30T14:51:28.493889+0000","last_became_active":"2023-10-30T14:14:04.613780+0000","last_became_peered":"2023-10-30T14:14:04.613780+0000","last_unstale":"2023-10-30T14:51:28.493889+0000","last_undegraded":"2023-10-30T14:51:28.493889+0000","last_fullsized":"2023-10-30T14:51:28.493889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1338,"log_dups_size":0,"ondisk_log_size":1338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:07:05.556883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065285270000000001,"stat_sum":{"num_bytes":12665534,"num_objects":519,"num_object_clones":250,"num_object_copies":1038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":519,"num_whiteouts":228,"num_read":63,"num_read_kb":8216,"num_write":1076,"num_write_kb":17979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"67'721","reported_seq":883,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.249637+0000","last_change":"2023-10-30T14:14:03.740213+0000","last_active":"2023-10-30T14:51:30.249637+0000","last_peered":"2023-10-30T14:51:30.249637+0000","last_clean":"2023-10-30T14:51:30.249637+0000","last_became_active":"2023-10-30T14:14:03.739650+0000","last_became_peered":"2023-10-30T14:14:03.739650+0000","last_unstale":"2023-10-30T14:51:30.249637+0000","last_undegraded":"2023-10-30T14:51:30.249637+0000","last_fullsized":"2023-10-30T14:51:30.249637+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":721,"log_dups_size":0,"ondisk_log_size":721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:54.120757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777238,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":143,"num_read_kb":109192,"num_write":780,"num_write_kb":46441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"67'1318","reported_seq":3274,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.425113+0000","last_change":"2023-10-30T14:23:30.771719+0000","last_active":"2023-10-30T14:51:29.425113+0000","last_peered":"2023-10-30T14:51:29.425113+0000","last_clean":"2023-10-30T14:51:29.425113+0000","last_became_active":"2023-10-30T14:14:04.590138+0000","last_became_peered":"2023-10-30T14:14:04.590138+0000","last_unstale":"2023-10-30T14:51:29.425113+0000","last_undegraded":"2023-10-30T14:51:29.425113+0000","last_fullsized":"2023-10-30T14:51:29.425113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1318,"log_dups_size":0,"ondisk_log_size":1318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:49.779956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031364370000000002,"stat_sum":{"num_bytes":9073572,"num_objects":494,"num_object_clones":232,"num_object_copies":988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":494,"num_whiteouts":206,"num_read":72,"num_read_kb":4145,"num_write":1090,"num_write_kb":19754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"67'771","reported_seq":913,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469946+0000","last_change":"2023-10-30T14:14:03.721450+0000","last_active":"2023-10-30T14:51:29.469946+0000","last_peered":"2023-10-30T14:51:29.469946+0000","last_clean":"2023-10-30T14:51:29.469946+0000","last_became_active":"2023-10-30T14:14:03.721234+0000","last_became_peered":"2023-10-30T14:14:03.721234+0000","last_unstale":"2023-10-30T14:51:29.469946+0000","last_undegraded":"2023-10-30T14:51:29.469946+0000","last_fullsized":"2023-10-30T14:51:29.469946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":771,"log_dups_size":0,"ondisk_log_size":771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:26.959676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":136,"num_read_kb":37589,"num_write":1311,"num_write_kb":40350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"67'1259","reported_seq":3214,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.476582+0000","last_change":"2023-10-30T14:23:31.070447+0000","last_active":"2023-10-30T14:51:28.476582+0000","last_peered":"2023-10-30T14:51:28.476582+0000","last_clean":"2023-10-30T14:51:28.476582+0000","last_became_active":"2023-10-30T14:14:04.633643+0000","last_became_peered":"2023-10-30T14:14:04.633643+0000","last_unstale":"2023-10-30T14:51:28.476582+0000","last_undegraded":"2023-10-30T14:51:28.476582+0000","last_fullsized":"2023-10-30T14:51:28.476582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1259,"log_dups_size":0,"ondisk_log_size":1259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:53:41.463549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048916599999999999,"stat_sum":{"num_bytes":9340734,"num_objects":487,"num_object_clones":228,"num_object_copies":974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":205,"num_read":67,"num_read_kb":4121,"num_write":1045,"num_write_kb":15590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"67'440","reported_seq":577,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.705366+0000","last_change":"2023-10-30T14:14:03.738557+0000","last_active":"2023-10-30T14:51:29.705366+0000","last_peered":"2023-10-30T14:51:29.705366+0000","last_clean":"2023-10-30T14:51:29.705366+0000","last_became_active":"2023-10-30T14:14:03.737801+0000","last_became_peered":"2023-10-30T14:14:03.737801+0000","last_unstale":"2023-10-30T14:51:29.705366+0000","last_undegraded":"2023-10-30T14:51:29.705366+0000","last_fullsized":"2023-10-30T14:51:29.705366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":440,"log_dups_size":0,"ondisk_log_size":440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:26.938343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":132,"num_read_kb":45549,"num_write":530,"num_write_kb":26044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"67'1244","reported_seq":3217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:28.425279+0000","last_change":"2023-10-30T14:23:30.776079+0000","last_active":"2023-10-30T14:51:28.425279+0000","last_peered":"2023-10-30T14:51:28.425279+0000","last_clean":"2023-10-30T14:51:28.425279+0000","last_became_active":"2023-10-30T14:14:04.588826+0000","last_became_peered":"2023-10-30T14:14:04.588826+0000","last_unstale":"2023-10-30T14:51:28.425279+0000","last_undegraded":"2023-10-30T14:51:28.425279+0000","last_fullsized":"2023-10-30T14:51:28.425279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1244,"log_dups_size":0,"ondisk_log_size":1244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:27.236524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040592670000000001,"stat_sum":{"num_bytes":8246712,"num_objects":450,"num_object_clones":210,"num_object_copies":900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":450,"num_whiteouts":181,"num_read":67,"num_read_kb":4125,"num_write":1046,"num_write_kb":10688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"67'429","reported_seq":542,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.467848+0000","last_change":"2023-10-30T14:14:03.733836+0000","last_active":"2023-10-30T14:51:29.467848+0000","last_peered":"2023-10-30T14:51:29.467848+0000","last_clean":"2023-10-30T14:51:29.467848+0000","last_became_active":"2023-10-30T14:14:03.732912+0000","last_became_peered":"2023-10-30T14:14:03.732912+0000","last_unstale":"2023-10-30T14:51:29.467848+0000","last_undegraded":"2023-10-30T14:51:29.467848+0000","last_fullsized":"2023-10-30T14:51:29.467848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:50.332738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":78,"num_read_kb":41242,"num_write":426,"num_write_kb":21091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"67'1279","reported_seq":3248,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.450800+0000","last_change":"2023-10-30T14:23:30.957473+0000","last_active":"2023-10-30T14:51:29.450800+0000","last_peered":"2023-10-30T14:51:29.450800+0000","last_clean":"2023-10-30T14:51:29.450800+0000","last_became_active":"2023-10-30T14:14:04.609835+0000","last_became_peered":"2023-10-30T14:14:04.609835+0000","last_unstale":"2023-10-30T14:51:29.450800+0000","last_undegraded":"2023-10-30T14:51:29.450800+0000","last_fullsized":"2023-10-30T14:51:29.450800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1279,"log_dups_size":0,"ondisk_log_size":1279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:29:36.272312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023958389999999999,"stat_sum":{"num_bytes":25809405,"num_objects":464,"num_object_clones":214,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":464,"num_whiteouts":191,"num_read":67,"num_read_kb":20495,"num_write":1061,"num_write_kb":27581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"67'867","reported_seq":1008,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.048969+0000","last_change":"2023-10-30T14:14:03.732539+0000","last_active":"2023-10-30T14:51:25.048969+0000","last_peered":"2023-10-30T14:51:25.048969+0000","last_clean":"2023-10-30T14:51:25.048969+0000","last_became_active":"2023-10-30T14:14:03.732265+0000","last_became_peered":"2023-10-30T14:14:03.732265+0000","last_unstale":"2023-10-30T14:51:25.048969+0000","last_undegraded":"2023-10-30T14:51:25.048969+0000","last_fullsized":"2023-10-30T14:51:25.048969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":867,"log_dups_size":0,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:02:00.929612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":128,"num_read_kb":49868,"num_write":925,"num_write_kb":26024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"67'1338","reported_seq":3317,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.437725+0000","last_change":"2023-10-30T14:23:31.069537+0000","last_active":"2023-10-30T14:51:29.437725+0000","last_peered":"2023-10-30T14:51:29.437725+0000","last_clean":"2023-10-30T14:51:29.437725+0000","last_became_active":"2023-10-30T14:14:04.594998+0000","last_became_peered":"2023-10-30T14:14:04.594998+0000","last_unstale":"2023-10-30T14:51:29.437725+0000","last_undegraded":"2023-10-30T14:51:29.437725+0000","last_fullsized":"2023-10-30T14:51:29.437725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1338,"log_dups_size":0,"ondisk_log_size":1338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:29:28.794439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00432118,"stat_sum":{"num_bytes":12291745,"num_objects":459,"num_object_clones":215,"num_object_copies":918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":459,"num_whiteouts":187,"num_read":78,"num_read_kb":8212,"num_write":1121,"num_write_kb":18507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'816","reported_seq":962,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.428311+0000","last_change":"2023-10-30T14:14:03.703744+0000","last_active":"2023-10-30T14:51:29.428311+0000","last_peered":"2023-10-30T14:51:29.428311+0000","last_clean":"2023-10-30T14:51:29.428311+0000","last_became_active":"2023-10-30T14:14:03.701759+0000","last_became_peered":"2023-10-30T14:14:03.701759+0000","last_unstale":"2023-10-30T14:51:29.428311+0000","last_undegraded":"2023-10-30T14:51:29.428311+0000","last_fullsized":"2023-10-30T14:51:29.428311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":816,"log_dups_size":0,"ondisk_log_size":816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:22:27.948007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":122,"num_read_kb":94942,"num_write":1181,"num_write_kb":51316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"67'1224","reported_seq":3205,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.415105+0000","last_change":"2023-10-30T14:23:31.071315+0000","last_active":"2023-10-30T14:51:29.415105+0000","last_peered":"2023-10-30T14:51:29.415105+0000","last_clean":"2023-10-30T14:51:29.415105+0000","last_became_active":"2023-10-30T14:14:04.600100+0000","last_became_peered":"2023-10-30T14:14:04.600100+0000","last_unstale":"2023-10-30T14:51:29.415105+0000","last_undegraded":"2023-10-30T14:51:29.415105+0000","last_fullsized":"2023-10-30T14:51:29.415105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1224,"log_dups_size":0,"ondisk_log_size":1224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:30.476350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052227630000000001,"stat_sum":{"num_bytes":16868959,"num_objects":442,"num_object_clones":207,"num_object_copies":884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":442,"num_whiteouts":180,"num_read":73,"num_read_kb":11653,"num_write":1014,"num_write_kb":18882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"67'773","reported_seq":906,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.997876+0000","last_change":"2023-10-30T14:14:03.741268+0000","last_active":"2023-10-30T14:51:30.997876+0000","last_peered":"2023-10-30T14:51:30.997876+0000","last_clean":"2023-10-30T14:51:30.997876+0000","last_became_active":"2023-10-30T14:14:03.740648+0000","last_became_peered":"2023-10-30T14:14:03.740648+0000","last_unstale":"2023-10-30T14:51:30.997876+0000","last_undegraded":"2023-10-30T14:51:30.997876+0000","last_fullsized":"2023-10-30T14:51:30.997876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:30.799148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":98,"num_read_kb":107365,"num_write":798,"num_write_kb":50458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"67'1285","reported_seq":3266,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.469362+0000","last_change":"2023-10-30T14:23:31.073589+0000","last_active":"2023-10-30T14:51:29.469362+0000","last_peered":"2023-10-30T14:51:29.469362+0000","last_clean":"2023-10-30T14:51:29.469362+0000","last_became_active":"2023-10-30T14:14:04.622921+0000","last_became_peered":"2023-10-30T14:14:04.622921+0000","last_unstale":"2023-10-30T14:51:29.469362+0000","last_undegraded":"2023-10-30T14:51:29.469362+0000","last_fullsized":"2023-10-30T14:51:29.469362+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1285,"log_dups_size":0,"ondisk_log_size":1285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:04:46.396711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0071219910000000003,"stat_sum":{"num_bytes":20705836,"num_objects":462,"num_object_clones":210,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":462,"num_whiteouts":184,"num_read":75,"num_read_kb":12323,"num_write":1096,"num_write_kb":26452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"67'1315","reported_seq":3269,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.461591+0000","last_change":"2023-10-30T14:23:30.779989+0000","last_active":"2023-10-30T14:51:29.461591+0000","last_peered":"2023-10-30T14:51:29.461591+0000","last_clean":"2023-10-30T14:51:29.461591+0000","last_became_active":"2023-10-30T14:14:04.621885+0000","last_became_peered":"2023-10-30T14:14:04.621885+0000","last_unstale":"2023-10-30T14:51:29.461591+0000","last_undegraded":"2023-10-30T14:51:29.461591+0000","last_fullsized":"2023-10-30T14:51:29.461591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1315,"log_dups_size":0,"ondisk_log_size":1315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:59:52.839990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067851329999999996,"stat_sum":{"num_bytes":13415563,"num_objects":482,"num_object_clones":227,"num_object_copies":964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":190,"num_read":68,"num_read_kb":8221,"num_write":1118,"num_write_kb":20704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"67'466","reported_seq":615,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:32.456573+0000","last_change":"2023-10-30T14:14:03.740897+0000","last_active":"2023-10-30T14:51:32.456573+0000","last_peered":"2023-10-30T14:51:32.456573+0000","last_clean":"2023-10-30T14:51:32.456573+0000","last_became_active":"2023-10-30T14:14:03.740247+0000","last_became_peered":"2023-10-30T14:14:03.740247+0000","last_unstale":"2023-10-30T14:51:32.456573+0000","last_undegraded":"2023-10-30T14:51:32.456573+0000","last_fullsized":"2023-10-30T14:51:32.456573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:29.601978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30406549,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":149,"num_read_kb":82851,"num_write":524,"num_write_kb":39214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'1397","reported_seq":3340,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.460907+0000","last_change":"2023-10-30T14:23:30.776236+0000","last_active":"2023-10-30T14:51:29.460907+0000","last_peered":"2023-10-30T14:51:29.460907+0000","last_clean":"2023-10-30T14:51:29.460907+0000","last_became_active":"2023-10-30T14:14:04.623638+0000","last_became_peered":"2023-10-30T14:14:04.623638+0000","last_unstale":"2023-10-30T14:51:29.460907+0000","last_undegraded":"2023-10-30T14:51:29.460907+0000","last_fullsized":"2023-10-30T14:51:29.460907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1397,"log_dups_size":0,"ondisk_log_size":1397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:37:06.313676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049828509999999999,"stat_sum":{"num_bytes":25425754,"num_objects":536,"num_object_clones":247,"num_object_copies":1072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":221,"num_read":77,"num_read_kb":19614,"num_write":1171,"num_write_kb":31740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"67'1024","reported_seq":1176,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.475951+0000","last_change":"2023-10-30T14:14:03.733233+0000","last_active":"2023-10-30T14:51:29.475951+0000","last_peered":"2023-10-30T14:51:29.475951+0000","last_clean":"2023-10-30T14:51:29.475951+0000","last_became_active":"2023-10-30T14:14:03.732538+0000","last_became_peered":"2023-10-30T14:14:03.732538+0000","last_unstale":"2023-10-30T14:51:29.475951+0000","last_undegraded":"2023-10-30T14:51:29.475951+0000","last_fullsized":"2023-10-30T14:51:29.475951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1024,"log_dups_size":0,"ondisk_log_size":1024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:25.696173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":137,"num_read_kb":150323,"num_write":1064,"num_write_kb":62931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1272","reported_seq":3223,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.627078+0000","last_change":"2023-10-30T14:23:30.776451+0000","last_active":"2023-10-30T14:51:29.627078+0000","last_peered":"2023-10-30T14:51:29.627078+0000","last_clean":"2023-10-30T14:51:29.627078+0000","last_became_active":"2023-10-30T14:14:04.611773+0000","last_became_peered":"2023-10-30T14:14:04.611773+0000","last_unstale":"2023-10-30T14:51:29.627078+0000","last_undegraded":"2023-10-30T14:51:29.627078+0000","last_fullsized":"2023-10-30T14:51:29.627078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1272,"log_dups_size":0,"ondisk_log_size":1272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:55.244396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044924040000000002,"stat_sum":{"num_bytes":26143587,"num_objects":507,"num_object_clones":231,"num_object_copies":1014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":507,"num_whiteouts":205,"num_read":66,"num_read_kb":19602,"num_write":1063,"num_write_kb":27312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"67'731","reported_seq":872,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.995935+0000","last_change":"2023-10-30T14:14:03.703988+0000","last_active":"2023-10-30T14:51:30.995935+0000","last_peered":"2023-10-30T14:51:30.995935+0000","last_clean":"2023-10-30T14:51:30.995935+0000","last_became_active":"2023-10-30T14:14:03.703691+0000","last_became_peered":"2023-10-30T14:14:03.703691+0000","last_unstale":"2023-10-30T14:51:30.995935+0000","last_undegraded":"2023-10-30T14:51:30.995935+0000","last_fullsized":"2023-10-30T14:51:30.995935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":731,"log_dups_size":0,"ondisk_log_size":731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:34.538541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":102,"num_read_kb":111696,"num_write":760,"num_write_kb":58649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"67'1327","reported_seq":3261,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.468906+0000","last_change":"2023-10-30T14:23:31.069185+0000","last_active":"2023-10-30T14:51:29.468906+0000","last_peered":"2023-10-30T14:51:29.468906+0000","last_clean":"2023-10-30T14:51:29.468906+0000","last_became_active":"2023-10-30T14:14:04.613420+0000","last_became_peered":"2023-10-30T14:14:04.613420+0000","last_unstale":"2023-10-30T14:51:29.468906+0000","last_undegraded":"2023-10-30T14:51:29.468906+0000","last_fullsized":"2023-10-30T14:51:29.468906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1327,"log_dups_size":0,"ondisk_log_size":1327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:51:35.517831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004010128,"stat_sum":{"num_bytes":12779880,"num_objects":504,"num_object_clones":238,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":504,"num_whiteouts":203,"num_read":55,"num_read_kb":8221,"num_write":1111,"num_write_kb":14267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"67'636","reported_seq":796,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.650555+0000","last_change":"2023-10-30T14:14:03.737319+0000","last_active":"2023-10-30T14:51:29.650555+0000","last_peered":"2023-10-30T14:51:29.650555+0000","last_clean":"2023-10-30T14:51:29.650555+0000","last_became_active":"2023-10-30T14:14:03.736964+0000","last_became_peered":"2023-10-30T14:14:03.736964+0000","last_unstale":"2023-10-30T14:51:29.650555+0000","last_undegraded":"2023-10-30T14:51:29.650555+0000","last_fullsized":"2023-10-30T14:51:29.650555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:41.469809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":182,"num_read_kb":58134,"num_write":693,"num_write_kb":34123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1046721150,"num_objects":31370,"num_object_clones":14593,"num_object_copies":62740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31370,"num_whiteouts":12799,"num_read":4602,"num_read_kb":662584,"num_write":69411,"num_write_kb":1341877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2168676352,"data_stored":2094671100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83616,"ondisk_log_size":83616,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":673711135,"num_objects":1337,"num_object_clones":0,"num_object_copies":2674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1337,"num_whiteouts":0,"num_read":6029,"num_read_kb":2835128,"num_write":28094,"num_write_kb":1507213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1111,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1347235840,"data_stored":1347090008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25568,"ondisk_log_size":25568,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575305,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614432,"kb_used_data":479192,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":93757408,"statfs":{"total":96636764160,"available":96007585792,"internally_reserved":0,"allocated":490692608,"data_stored":479962954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":138475979},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.077,"15min":0.997},"min":{"1min":0.51700000000000002,"5min":0.46300000000000002,"15min":0.45500000000000002},"max":{"1min":6.3040000000000003,"5min":6.399,"15min":6.399},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.2370000000000001,"5min":0.97799999999999998,"15min":0.94399999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":6.4969999999999999,"5min":6.4969999999999999,"15min":6.4969999999999999},"last":0.83199999999999996}]},{"osd":1,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":0.92500000000000004,"15min":0.93000000000000005},"min":{"1min":0.61599999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":6.1390000000000002,"5min":6.1390000000000002,"15min":6.1390000000000002},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.96599999999999997,"15min":0.99399999999999999},"min":{"1min":0.58699999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":3.8460000000000001,"5min":6.1920000000000002,"15min":6.1920000000000002},"last":0.63700000000000001}]},{"osd":2,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.0449999999999999,"15min":0.96899999999999997},"min":{"1min":0.70099999999999996,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.5739999999999998,"5min":6.5739999999999998,"15min":6.5739999999999998},"last":1.125},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.115,"15min":1.01},"min":{"1min":0.81599999999999995,"5min":0.54500000000000004,"15min":0.5},"max":{"1min":6.2350000000000003,"5min":6.5149999999999997,"15min":6.5149999999999997},"last":1.2450000000000001}]},{"osd":3,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.86199999999999999,"15min":0.85599999999999998},"min":{"1min":0.56200000000000006,"5min":0.54800000000000004,"15min":0.53800000000000003},"max":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":4.3230000000000004},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.121,"15min":1.056},"min":{"1min":0.66800000000000004,"5min":0.64800000000000002,"15min":0.54300000000000004},"max":{"1min":3.157,"5min":6.2670000000000003,"15min":6.2670000000000003},"last":0.91500000000000004}]},{"osd":4,"last update":"Mon Oct 30 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.95599999999999996,"15min":0.95699999999999996},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.54800000000000004},"max":{"1min":4.7409999999999997,"5min":4.7409999999999997,"15min":4.7409999999999997},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.036,"15min":0.97399999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":8.0719999999999992,"5min":8.0719999999999992,"15min":8.0719999999999992},"last":0.873}]},{"osd":5,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3280000000000001,"15min":1.1759999999999999},"min":{"1min":0.78100000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":10.692,"5min":10.692,"15min":10.692},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.3939999999999999,"15min":1.2869999999999999},"min":{"1min":0.72099999999999997,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":6.6429999999999998,"5min":6.6429999999999998,"15min":6.6429999999999998},"last":1.3520000000000001}]},{"osd":6,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.135,"15min":1.129},"min":{"1min":0.748,"5min":0.71599999999999997,"15min":0.62},"max":{"1min":5.2160000000000002,"5min":5.2160000000000002,"15min":5.2160000000000002},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.1639999999999999,"15min":1.107},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":4.367,"5min":4.367,"15min":4.367},"last":1.2010000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575305,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511764,"kb_used_data":386252,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":93860076,"statfs":{"total":96636764160,"available":96112717824,"internally_reserved":0,"allocated":395522048,"data_stored":388333646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":128514503},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.82899999999999996,"15min":0.83199999999999996},"min":{"1min":0.503,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.216,"5min":1.288,"15min":1.2929999999999999},"last":1.5129999999999999},{"interface":"front","average":{"1min":0.997,"5min":0.90500000000000003,"15min":0.88800000000000001},"min":{"1min":0.68600000000000005,"5min":0.435,"15min":0.435},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.391},"last":1.726}]},{"osd":1,"last update":"Mon Oct 30 14:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.89700000000000002,"15min":0.85599999999999998},"min":{"1min":0.66000000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.135,"5min":8,"15min":8},"last":1.468},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.94199999999999995,"15min":0.96599999999999997},"min":{"1min":0.67300000000000004,"5min":0.53400000000000003,"15min":0.318},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.514},"last":1.2969999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.89300000000000002,"15min":0.89000000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.40999999999999998},"max":{"1min":1.232,"5min":1.2410000000000001,"15min":1.617},"last":0.876},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.92900000000000005,"15min":0.94299999999999995},"min":{"1min":0.63400000000000001,"5min":0.59999999999999998,"15min":0.49299999999999999},"max":{"1min":1.181,"5min":1.3149999999999999,"15min":1.4119999999999999},"last":2.9609999999999999}]},{"osd":3,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":0.997,"15min":0.98099999999999998},"min":{"1min":0.73999999999999999,"5min":0.57599999999999996,"15min":0.47799999999999998},"max":{"1min":1.3700000000000001,"5min":1.3720000000000001,"15min":1.6890000000000001},"last":1.6200000000000001},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.95499999999999996,"15min":0.96099999999999997},"min":{"1min":0.61499999999999999,"5min":0.59199999999999997,"15min":0.48899999999999999},"max":{"1min":1.286,"5min":1.286,"15min":1.472},"last":1.206}]},{"osd":4,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.0309999999999999,"15min":1.032},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.59899999999999998},"max":{"1min":1.1830000000000001,"5min":3.8399999999999999,"15min":3.8399999999999999},"last":3.7549999999999999},{"interface":"front","average":{"1min":1.016,"5min":1.0269999999999999,"15min":1.04},"min":{"1min":0.69999999999999996,"5min":0.62,"15min":0.435},"max":{"1min":1.341,"5min":1.4730000000000001,"15min":1.645},"last":3.0790000000000002}]},{"osd":5,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":1.0149999999999999,"15min":0.97299999999999998},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.51100000000000001},"max":{"1min":1.1659999999999999,"5min":11.728,"15min":11.728},"last":3.2120000000000002},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.1000000000000001,"15min":1.0920000000000001},"min":{"1min":0.72899999999999998,"5min":0.51700000000000002,"15min":0.51400000000000001},"max":{"1min":1.2310000000000001,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":4.6779999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.224,"15min":1.137},"min":{"1min":0.76400000000000001,"5min":0.755,"15min":0.65200000000000002},"max":{"1min":1.268,"5min":18.041,"15min":18.041},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.345,"15min":1.228},"min":{"1min":0.98699999999999999,"5min":0.79700000000000004,"15min":0.56000000000000005},"max":{"1min":1.579,"5min":20.375,"15min":20.375},"last":3.3319999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575305,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":665052,"kb_used_data":529812,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":93706788,"statfs":{"total":96636764160,"available":95955750912,"internally_reserved":0,"allocated":542527488,"data_stored":530952161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":138475974},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":0.83699999999999997,"15min":0.82099999999999995},"min":{"1min":0.54400000000000004,"5min":0.43099999999999999,"15min":0.40100000000000002},"max":{"1min":7.04,"5min":7.04,"15min":7.04},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.131,"5min":0.876,"15min":0.85899999999999999},"min":{"1min":0.53500000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":5.4720000000000004,"5min":5.4720000000000004,"15min":5.4720000000000004},"last":1.0680000000000001}]},{"osd":1,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83199999999999996,"15min":0.88},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.284,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":1.151},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.83699999999999997,"15min":0.85099999999999998},"min":{"1min":0.57999999999999996,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":4.1630000000000003,"5min":4.1630000000000003,"15min":4.1630000000000003},"last":0.91700000000000004}]},{"osd":2,"last update":"Mon Oct 30 14:51:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.92300000000000004,"15min":0.95999999999999996},"min":{"1min":0.61299999999999999,"5min":0.503,"15min":0.44800000000000001},"max":{"1min":1.3320000000000001,"5min":1.3640000000000001,"15min":2.044},"last":1.224},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.087,"15min":1.024},"min":{"1min":0.81899999999999995,"5min":0.69099999999999995,"15min":0.51600000000000001},"max":{"1min":1.425,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":0.98699999999999999}]},{"osd":3,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.85599999999999998,"15min":0.93899999999999995},"min":{"1min":0.623,"5min":0.58399999999999996,"15min":0.56499999999999995},"max":{"1min":1.1000000000000001,"5min":1.2010000000000001,"15min":1.4550000000000001},"last":0.877},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.03,"15min":0.97199999999999998},"min":{"1min":0.75,"5min":0.623,"15min":0.52100000000000002},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.4059999999999999},"last":1.294}]},{"osd":4,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.149,"15min":1.0469999999999999},"min":{"1min":0.622,"5min":0.62,"15min":0.59099999999999997},"max":{"1min":8.7360000000000007,"5min":8.7360000000000007,"15min":8.7360000000000007},"last":0.77800000000000002},{"interface":"front","average":{"1min":2.335,"5min":1.244,"15min":1.0640000000000001},"min":{"1min":0.621,"5min":0.61199999999999999,"15min":0.59399999999999997},"max":{"1min":23.640000000000001,"5min":23.640000000000001,"15min":23.640000000000001},"last":0.73199999999999998}]},{"osd":6,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1339999999999999,"15min":1.0700000000000001},"min":{"1min":0.80100000000000005,"5min":0.73399999999999999,"15min":0.55500000000000005},"max":{"1min":1.546,"5min":1.7470000000000001,"15min":2.306},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.194,"15min":1.127},"min":{"1min":0.878,"5min":0.875,"15min":0.57899999999999996},"max":{"1min":1.575,"5min":1.7569999999999999,"15min":2.774},"last":1.367}]},{"osd":7,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.99399999999999999,"15min":1.0629999999999999},"min":{"1min":0.71899999999999997,"5min":0.63,"15min":0.40799999999999997},"max":{"1min":1.198,"5min":1.3100000000000001,"15min":3.0739999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.036,"5min":1.129,"15min":1.1739999999999999},"min":{"1min":0.87,"5min":0.87,"15min":0.77600000000000002},"max":{"1min":1.244,"5min":1.53,"15min":2.1339999999999999},"last":0.95299999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575305,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611736,"kb_used_data":476496,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":93760104,"statfs":{"total":96636764160,"available":96010346496,"internally_reserved":0,"allocated":487931904,"data_stored":476574704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":138475974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.0109999999999999,"15min":0.876},"min":{"1min":0.53300000000000003,"5min":0.48199999999999998,"15min":0.27900000000000003},"max":{"1min":3.222,"5min":7.1870000000000003,"15min":7.1870000000000003},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.099,"5min":1.107,"15min":0.95099999999999996},"min":{"1min":0.68899999999999995,"5min":0.48199999999999998,"15min":0.33900000000000002},"max":{"1min":3.3010000000000002,"5min":10.864000000000001,"15min":10.864000000000001},"last":0.99299999999999999}]},{"osd":1,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.04,"15min":0.95199999999999996},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.50700000000000001},"max":{"1min":3.3780000000000001,"5min":7.6980000000000004,"15min":7.6980000000000004},"last":1.101},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.91300000000000003,"15min":0.83999999999999997},"min":{"1min":0.58699999999999997,"5min":0.48399999999999999,"15min":0.36599999999999999},"max":{"1min":2.1789999999999998,"5min":5.5259999999999998,"15min":5.5259999999999998},"last":0.63200000000000001}]},{"osd":2,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0229999999999999,"15min":0.96199999999999997},"min":{"1min":0.64300000000000002,"5min":0.57499999999999996,"15min":0.40899999999999997},"max":{"1min":2.2839999999999998,"5min":5.423,"15min":5.423},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.254,"15min":1.0940000000000001},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.54400000000000004},"max":{"1min":3.4500000000000002,"5min":13.221,"15min":13.221},"last":1.181}]},{"osd":3,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.96099999999999997,"15min":0.86199999999999999},"min":{"1min":0.626,"5min":0.54800000000000004,"15min":0.46899999999999997},"max":{"1min":7.5620000000000003,"5min":7.5620000000000003,"15min":7.5620000000000003},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.0860000000000001,"15min":1.0029999999999999},"min":{"1min":0.71199999999999997,"5min":0.65500000000000003,"15min":0.497},"max":{"1min":7.7370000000000001,"5min":7.7370000000000001,"15min":7.7370000000000001},"last":1.056}]},{"osd":5,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.091,"15min":0.97599999999999998},"min":{"1min":0.72499999999999998,"5min":0.64200000000000002,"15min":0.317},"max":{"1min":7.6360000000000001,"5min":7.6360000000000001,"15min":7.6360000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.159,"5min":1.2609999999999999,"15min":1.145},"min":{"1min":0.628,"5min":0.628,"15min":0.442},"max":{"1min":3.6389999999999998,"5min":7.2599999999999998,"15min":7.2599999999999998},"last":1.2749999999999999}]},{"osd":6,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1910000000000001,"15min":1.1020000000000001},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.39200000000000002},"max":{"1min":2.379,"5min":5.4729999999999999,"15min":5.4729999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.2969999999999999,"15min":1.1830000000000001},"min":{"1min":0.84299999999999997,"5min":0.73999999999999999,"15min":0.47899999999999998},"max":{"1min":6.5270000000000001,"5min":10.102,"15min":10.102},"last":1.23}]},{"osd":7,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.1120000000000001,"15min":1.042},"min":{"1min":0.75700000000000001,"5min":0.69099999999999995,"15min":0.40100000000000002},"max":{"1min":9.1240000000000006,"5min":9.1240000000000006,"15min":9.1240000000000006},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.423,"5min":1.2809999999999999,"15min":1.2130000000000001},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.52900000000000003},"max":{"1min":7.0490000000000004,"5min":7.0490000000000004,"15min":7.0490000000000004},"last":1.1299999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575303,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645632,"kb_used_data":497912,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":93726208,"statfs":{"total":96636764160,"available":95975636992,"internally_reserved":0,"allocated":509861888,"data_stored":498479753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":151255496},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.53600000000000003,"15min":0.48299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.1130000000000004,"5min":4.1130000000000004,"15min":4.1130000000000004},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.54200000000000004,"15min":0.51200000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.91100000000000003,"5min":4.5190000000000001,"15min":4.5190000000000001},"last":0.376}]},{"osd":1,"last update":"Mon Oct 30 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.58999999999999997,"15min":0.54400000000000004},"min":{"1min":0.33100000000000002,"5min":0.317,"15min":0.314},"max":{"1min":3.988,"5min":3.988,"15min":3.988},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.56399999999999995,"15min":0.54900000000000004},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.29999999999999999},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.432}]},{"osd":2,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.65600000000000003,"15min":0.56100000000000005},"min":{"1min":0.31,"5min":0.29199999999999998,"15min":0.27600000000000002},"max":{"1min":1.0740000000000001,"5min":8.6519999999999992,"15min":8.6519999999999992},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.73999999999999999,"15min":0.63700000000000001},"min":{"1min":0.42199999999999999,"5min":0.41999999999999998,"15min":0.313},"max":{"1min":1.1040000000000001,"5min":7.0949999999999998,"15min":7.0949999999999998},"last":0.63300000000000001}]},{"osd":4,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.83399999999999996,"15min":0.82099999999999995},"min":{"1min":0.54600000000000004,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.1220000000000001,"5min":4.5730000000000004,"15min":4.5730000000000004},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.90200000000000002,"15min":0.85299999999999998},"min":{"1min":0.51500000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":4.0910000000000002,"5min":4.4720000000000004,"15min":4.4720000000000004},"last":0.78400000000000003}]},{"osd":5,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.0029999999999999,"15min":0.90000000000000002},"min":{"1min":0.72799999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.296,"5min":14.974,"15min":14.974},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.042,"5min":0.96699999999999997,"15min":0.873},"min":{"1min":0.746,"5min":0.52600000000000002,"15min":0.46600000000000003},"max":{"1min":4.0629999999999997,"5min":5.907,"15min":5.907},"last":0.88400000000000001}]},{"osd":6,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.01,"15min":0.91900000000000004},"min":{"1min":0.70899999999999996,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":5.8140000000000001,"5min":5.8300000000000001,"15min":5.8300000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.159,"5min":1.073,"15min":0.96299999999999997},"min":{"1min":0.70799999999999996,"5min":0.60199999999999998,"15min":0.55100000000000005},"max":{"1min":6.0300000000000002,"5min":6.0300000000000002,"15min":6.0300000000000002},"last":0.99099999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":1.115,"15min":0.995},"min":{"1min":0.67600000000000005,"5min":0.66800000000000004,"15min":0.61699999999999999},"max":{"1min":1.2549999999999999,"5min":16.282,"15min":16.282},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.123,"15min":0.97199999999999998},"min":{"1min":0.70999999999999996,"5min":0.627,"15min":0.58099999999999996},"max":{"1min":1.6519999999999999,"5min":16.096,"15min":16.096},"last":0.83599999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575305,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":417172,"kb_used_data":309708,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":93954668,"statfs":{"total":96636764160,"available":96209580032,"internally_reserved":0,"allocated":317140992,"data_stored":310407991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.45300000000000001,"15min":0.48699999999999999},"min":{"1min":0.30099999999999999,"5min":0.27400000000000002,"15min":0.26300000000000001},"max":{"1min":0.67800000000000005,"5min":0.79100000000000004,"15min":13.526},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.50900000000000001,"15min":0.51300000000000001},"min":{"1min":0.38300000000000001,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":1.2190000000000001},"last":0.434}]},{"osd":1,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46300000000000002,"15min":0.47199999999999998},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65500000000000003,"5min":0.67200000000000004,"15min":1.5469999999999999},"last":0.314},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54400000000000004,"15min":0.56000000000000005},"min":{"1min":0.38700000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73499999999999999,"5min":0.79300000000000004,"15min":1.452},"last":0.46000000000000002}]},{"osd":3,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58799999999999997,"15min":0.60699999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81299999999999994,"5min":0.95399999999999996,"15min":1.9390000000000001},"last":0.497},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.53900000000000003},"min":{"1min":0.35899999999999999,"5min":0.35499999999999998,"15min":0.33100000000000002},"max":{"1min":0.78500000000000003,"5min":0.79300000000000004,"15min":1.913},"last":0.47599999999999998}]},{"osd":4,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.81699999999999995,"15min":0.81499999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.44900000000000001},"max":{"1min":1.0309999999999999,"5min":4.2229999999999999,"15min":4.2229999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.91600000000000004,"15min":0.91000000000000003},"min":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.59599999999999997},"max":{"1min":1.081,"5min":4.2439999999999998,"15min":4.2439999999999998},"last":0.82799999999999996}]},{"osd":5,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.98999999999999999,"15min":0.86399999999999999},"min":{"1min":0.51900000000000002,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0620000000000001,"5min":22.285,"15min":22.285},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.82899999999999996,"15min":0.83399999999999996},"min":{"1min":0.58199999999999996,"5min":0.56000000000000005,"15min":0.51900000000000002},"max":{"1min":1.1319999999999999,"5min":2.3999999999999999,"15min":2.742},"last":0.66900000000000004}]},{"osd":6,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.86099999999999999,"15min":0.91500000000000004},"min":{"1min":0.65000000000000002,"5min":0.58799999999999997,"15min":0.54500000000000004},"max":{"1min":1.0489999999999999,"5min":2.911,"15min":27.731000000000002},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.872,"15min":0.874},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.1679999999999999,"5min":1.9099999999999999,"15min":2.0009999999999999},"last":0.81699999999999995}]},{"osd":7,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84299999999999997,"15min":0.91600000000000004},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.56499999999999995},"max":{"1min":1.105,"5min":1.2390000000000001,"15min":18.870999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.90400000000000003,"15min":0.96399999999999997},"min":{"1min":0.66600000000000004,"5min":0.61499999999999999,"15min":0.55500000000000005},"max":{"1min":1.0860000000000001,"5min":3.3809999999999998,"15min":18.356999999999999},"last":0.99099999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575304,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461964,"kb_used_data":344772,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":93909876,"statfs":{"total":96636764160,"available":96163713024,"internally_reserved":0,"allocated":353046528,"data_stored":346331165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":119994823},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.47399999999999998,"15min":0.44800000000000001},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.68200000000000005,"5min":2.4889999999999999,"15min":2.4889999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.51600000000000001,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71199999999999997,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.55800000000000005}]},{"osd":2,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.64500000000000002,"15min":0.56699999999999995},"min":{"1min":0.318,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79000000000000004,"5min":8.3160000000000007,"15min":8.3160000000000007},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.54000000000000004,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.27200000000000002,"15min":0.26800000000000002},"max":{"1min":0.61399999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.58099999999999996}]},{"osd":3,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.61099999999999999,"15min":0.59599999999999997},"min":{"1min":0.38800000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84399999999999997,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.623,"5min":0.64900000000000002,"15min":0.63600000000000001},"min":{"1min":0.439,"5min":0.38100000000000001,"15min":0.36499999999999999},"max":{"1min":0.89200000000000002,"5min":3.54,"15min":3.54},"last":0.66000000000000003}]},{"osd":4,"last update":"Mon Oct 30 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.92000000000000004,"15min":0.83999999999999997},"min":{"1min":0.505,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.92600000000000005,"5min":10.525,"15min":10.525},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.94899999999999995,"15min":0.83899999999999997},"min":{"1min":0.68400000000000005,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.997,"5min":10.571,"15min":10.571},"last":0.84799999999999998}]},{"osd":5,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.88900000000000001,"15min":0.89100000000000001},"min":{"1min":0.628,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.0289999999999999,"5min":3.4969999999999999,"15min":3.4969999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.876,"5min":0.84399999999999997,"15min":0.79200000000000004},"min":{"1min":0.72399999999999998,"5min":0.56000000000000005,"15min":0.49099999999999999},"max":{"1min":1.1739999999999999,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.96499999999999997}]},{"osd":6,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.80900000000000005},"min":{"1min":0.58399999999999996,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.1040000000000001,"5min":1.875,"15min":2.4710000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.88900000000000001,"15min":0.83399999999999996},"min":{"1min":0.57399999999999995,"5min":0.53100000000000003,"15min":0.51700000000000002},"max":{"1min":1.2130000000000001,"5min":5.3680000000000003,"15min":5.3680000000000003},"last":0.995}]},{"osd":7,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":1.081,"15min":0.91600000000000004},"min":{"1min":0.59599999999999997,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0249999999999999,"5min":19.885000000000002,"15min":19.885000000000002},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":1.054,"15min":0.93200000000000005},"min":{"1min":0.60099999999999998,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.127,"5min":13.401999999999999,"15min":13.401999999999999},"last":1.117}]}]},{"osd":0,"up_from":13,"seq":55834575306,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545984,"kb_used_data":416312,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":93825856,"statfs":{"total":96636764160,"available":96077676544,"internally_reserved":0,"allocated":426303488,"data_stored":414784024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132774342},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45900000000000002,"15min":0.44800000000000001},"min":{"1min":0.32200000000000001,"5min":0.26800000000000002,"15min":0.22500000000000001},"max":{"1min":0.749,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.53300000000000003,"15min":0.503},"min":{"1min":0.36699999999999999,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":0.68999999999999995,"5min":3.367,"15min":3.367},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58199999999999996,"15min":0.53600000000000003},"min":{"1min":0.44400000000000001,"5min":0.378,"15min":0.30499999999999999},"max":{"1min":0.751,"5min":3.222,"15min":3.222},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53200000000000003,"15min":0.56000000000000005},"min":{"1min":0.40000000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":1.095},"last":0.434}]},{"osd":3,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.56599999999999995,"15min":0.60199999999999998},"min":{"1min":0.34000000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.63300000000000001,"5min":6.4980000000000002,"15min":6.4980000000000002},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57499999999999996,"15min":0.52900000000000003},"min":{"1min":0.40400000000000003,"5min":0.40100000000000002,"15min":0.28000000000000003},"max":{"1min":0.71699999999999997,"5min":2.024,"15min":2.024},"last":0.497}]},{"osd":4,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.78800000000000003,"15min":0.82599999999999996},"min":{"1min":0.57599999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.95399999999999996,"5min":2.0659999999999998,"15min":2.8380000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81399999999999995,"15min":0.85999999999999999},"min":{"1min":0.65100000000000002,"5min":0.56899999999999995,"15min":0.53900000000000003},"max":{"1min":0.97699999999999998,"5min":2.1000000000000001,"15min":5.8959999999999999},"last":0.75700000000000001}]},{"osd":5,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.872,"15min":0.86099999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.46300000000000002},"max":{"1min":1.212,"5min":1.278,"15min":6.4480000000000004},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.92000000000000004,"15min":0.85599999999999998},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.54000000000000004},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":2.6869999999999998},"last":0.84399999999999997}]},{"osd":6,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.83899999999999997,"15min":0.84699999999999998},"min":{"1min":0.64800000000000002,"5min":0.55300000000000005,"15min":0.54400000000000004},"max":{"1min":1.133,"5min":1.2150000000000001,"15min":4.9980000000000002},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.877,"5min":0.85499999999999998,"15min":0.86099999999999999},"min":{"1min":0.65300000000000002,"5min":0.55100000000000005,"15min":0.54000000000000004},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.5980000000000001},"last":0.74099999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.86799999999999999,"15min":0.84699999999999998},"min":{"1min":0.63700000000000001,"5min":0.57899999999999996,"15min":0.499},"max":{"1min":1.1120000000000001,"5min":3.6589999999999998,"15min":3.6589999999999998},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.93100000000000005,"15min":0.88400000000000001},"min":{"1min":0.76300000000000001,"5min":0.625,"15min":0.54100000000000004},"max":{"1min":1.1579999999999999,"5min":1.262,"15min":2.6429999999999998},"last":1.02}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":104873984,"data_stored":104858260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159408128,"data_stored":159388062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":115867648,"data_stored":115847932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":200708096,"data_stored":200690717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":153219072,"data_stored":153204613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":208064512,"data_stored":208040997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":211091456,"data_stored":211074246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":194002944,"data_stored":193985181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":320053248,"data_stored":308933294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192856064,"data_stored":186541001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200491008,"data_stored":194157957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":308371456,"data_stored":297386934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":333930496,"data_stored":322967989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":333680640,"data_stored":322509062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":183648256,"data_stored":176857298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":295645184,"data_stored":285317565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:51:35.991 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:51:36.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.164+0000 7f8276b72640 1 Processor -- start 2023-10-30T14:51:36.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.164+0000 7f8276b72640 1 -- start start 2023-10-30T14:51:36.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.164+0000 7f8276b72640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8270153c80 0x7f827015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:36.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.164+0000 7f8276b72640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82700c4a10 0x7f82700c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:36.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.164+0000 7f8276b72640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8270159f20 con 0x7f827015a3b0 2023-10-30T14:51:36.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.164+0000 7f8276b72640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f827015dec0 con 0x7f8270153c80 2023-10-30T14:51:36.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.164+0000 7f8276b72640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82701599a0 con 0x7f82700c4a10 2023-10-30T14:51:36.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.164+0000 7f827490e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8270153c80 0x7f827015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:36.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.164+0000 7f826ffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82700c4a10 0x7f82700c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:36.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.164+0000 7f826ffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82700c4a10 0x7f82700c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:44212/0 (socket says 172.21.15.57:44212) 2023-10-30T14:51:36.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.164+0000 7f826ffff640 1 -- 172.21.15.57:0/526719267 learned_addr learned my addr 172.21.15.57:0/526719267 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:36.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.164+0000 7f827490e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8270153c80 0x7f827015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:57346/0 (socket says 172.21.15.57:57346) 2023-10-30T14:51:36.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.164+0000 7f826ffff640 1 -- 172.21.15.57:0/526719267 >> v2:172.21.15.88:3300/0 conn(0x7f8270153c80 msgr2=0x7f827015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:36.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.164+0000 7f826ffff640 1 --2- 172.21.15.57:0/526719267 >> v2:172.21.15.88:3300/0 conn(0x7f8270153c80 0x7f827015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:36.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.164+0000 7f826ffff640 1 -- 172.21.15.57:0/526719267 >> v1:172.21.15.88:6789/0 conn(0x7f827015a3b0 legacy=0x7f82700ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:36.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.164+0000 7f826ffff640 1 -- 172.21.15.57:0/526719267 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82700c47e0 con 0x7f82700c4a10 2023-10-30T14:51:36.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.168+0000 7f826ffff640 1 --2- 172.21.15.57:0/526719267 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82700c4a10 0x7f82700c42a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f825c009870 tx=0x7f825c01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92e6e4689da1e1ef server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:36.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.168+0000 7f826f7fe640 1 -- 172.21.15.57:0/526719267 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c0a8a30 con 0x7f82700c4a10 2023-10-30T14:51:36.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.168+0000 7f826f7fe640 1 -- 172.21.15.57:0/526719267 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825c0a8b90 con 0x7f82700c4a10 2023-10-30T14:51:36.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.168+0000 7f826f7fe640 1 -- 172.21.15.57:0/526719267 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c0b0440 con 0x7f82700c4a10 2023-10-30T14:51:36.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.168+0000 7f8276b72640 1 -- 172.21.15.57:0/526719267 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82700c4a10 msgr2=0x7f82700c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:36.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.168+0000 7f8276b72640 1 --2- 172.21.15.57:0/526719267 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82700c4a10 0x7f82700c42a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f825c009870 tx=0x7f825c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:51:36.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.168+0000 7f8276b72640 1 -- 172.21.15.57:0/526719267 shutdown_connections 2023-10-30T14:51:36.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.168+0000 7f8276b72640 1 --2- 172.21.15.57:0/526719267 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82700c4a10 0x7f82700c42a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:36.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.168+0000 7f8276b72640 1 --2- 172.21.15.57:0/526719267 >> v2:172.21.15.88:3300/0 conn(0x7f8270153c80 0x7f827015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:36.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.168+0000 7f8276b72640 1 -- 172.21.15.57:0/526719267 >> 172.21.15.57:0/526719267 conn(0x7f82700715c0 msgr2=0x7f82700719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:36.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.168+0000 7f8276b72640 1 -- 172.21.15.57:0/526719267 shutdown_connections 2023-10-30T14:51:36.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.168+0000 7f8276b72640 1 -- 172.21.15.57:0/526719267 wait complete. 2023-10-30T14:51:36.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f8276b72640 1 Processor -- start 2023-10-30T14:51:36.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f8276b72640 1 -- start start 2023-10-30T14:51:36.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f8276b72640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82700c4a10 0x7f827014bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:36.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f8276b72640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8270153c80 0x7f827014a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:36.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f8276b72640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f827015a3b0 0x7f8270158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:36.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f8276b72640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827014c1c0 con 0x7f82700c4a10 2023-10-30T14:51:36.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f8276b72640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82701f30f0 con 0x7f827015a3b0 2023-10-30T14:51:36.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f8276b72640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82701f2290 con 0x7f8270153c80 2023-10-30T14:51:36.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f827490e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82700c4a10 0x7f827014bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:36.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f827490e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82700c4a10 0x7f827014bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56650/0 (socket says 172.21.15.57:56650) 2023-10-30T14:51:36.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f827490e640 1 -- 172.21.15.57:0/3537805018 learned_addr learned my addr 172.21.15.57:0/3537805018 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:36.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f826ffff640 1 --2- 172.21.15.57:0/3537805018 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8270153c80 0x7f827014a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:36.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f827510f640 1 --2- 172.21.15.57:0/3537805018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f827015a3b0 0x7f8270158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:36.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f827490e640 1 -- 172.21.15.57:0/3537805018 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8270153c80 msgr2=0x7f827014a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:36.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f827490e640 1 --2- 172.21.15.57:0/3537805018 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8270153c80 0x7f827014a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:36.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f827490e640 1 -- 172.21.15.57:0/3537805018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f827015a3b0 msgr2=0x7f8270158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:36.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f827490e640 1 --2- 172.21.15.57:0/3537805018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f827015a3b0 0x7f8270158850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:36.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f827490e640 1 -- 172.21.15.57:0/3537805018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c0094d0 con 0x7f82700c4a10 2023-10-30T14:51:36.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f827510f640 1 --2- 172.21.15.57:0/3537805018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f827015a3b0 0x7f8270158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:36.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f827490e640 1 --2- 172.21.15.57:0/3537805018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82700c4a10 0x7f827014bc80 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f826400e930 tx=0x7f826400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:36.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f826d7fa640 1 -- 172.21.15.57:0/3537805018 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264009690 con 0x7f82700c4a10 2023-10-30T14:51:36.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f826d7fa640 1 -- 172.21.15.57:0/3537805018 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826400ca00 con 0x7f82700c4a10 2023-10-30T14:51:36.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f826d7fa640 1 -- 172.21.15.57:0/3537805018 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826400cd70 con 0x7f82700c4a10 2023-10-30T14:51:36.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f8276b72640 1 -- 172.21.15.57:0/3537805018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82701f2570 con 0x7f82700c4a10 2023-10-30T14:51:36.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.172+0000 7f8276b72640 1 -- 172.21.15.57:0/3537805018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82701f2b30 con 0x7f82700c4a10 2023-10-30T14:51:36.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.176+0000 7f8276b72640 1 -- 172.21.15.57:0/3537805018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f827004e170 con 0x7f82700c4a10 2023-10-30T14:51:36.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.176+0000 7f826d7fa640 1 -- 172.21.15.57:0/3537805018 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8264013470 con 0x7f82700c4a10 2023-10-30T14:51:36.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.176+0000 7f826d7fa640 1 --2- 172.21.15.57:0/3537805018 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f82540664a0 0x7f8254068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:36.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.176+0000 7f826d7fa640 1 -- 172.21.15.57:0/3537805018 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f82640846b0 con 0x7f82700c4a10 2023-10-30T14:51:36.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.176+0000 7f826ffff640 1 --2- 172.21.15.57:0/3537805018 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f82540664a0 0x7f8254068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:36.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.176+0000 7f826ffff640 1 --2- 172.21.15.57:0/3537805018 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f82540664a0 0x7f8254068960 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f825c0b2000 tx=0x7f825c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:36.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.180+0000 7f826d7fa640 1 -- 172.21.15.57:0/3537805018 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8264051610 con 0x7f82700c4a10 2023-10-30T14:51:36.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.340+0000 7f8276b72640 1 -- 172.21.15.57:0/3537805018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8270157550 con 0x7f82700c4a10 2023-10-30T14:51:36.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.340+0000 7f826d7fa640 1 -- 172.21.15.57:0/3537805018 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+19036 (secure 0 0 0) 0x7f8264054c30 con 0x7f82700c4a10 2023-10-30T14:51:36.345 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:51:36.346 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":67,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:50:50.290577+0000","last_up_change":"2023-10-30T14:13:51.171287+0000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T14:13:52.574639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T14:14:02.600122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T14:14:03.069429+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":53,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6801","nonce":3287367848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6803","nonce":3287367848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6807","nonce":3287367848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6805","nonce":3287367848}]},"public_addr":"172.21.15.57:6801/3287367848","cluster_addr":"172.21.15.57:6803/3287367848","heartbeat_back_addr":"172.21.15.57:6807/3287367848","heartbeat_front_addr":"172.21.15.57:6805/3287367848","state":["exists","up"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6809","nonce":3340797836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6811","nonce":3340797836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6815","nonce":3340797836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6813","nonce":3340797836}]},"public_addr":"172.21.15.57:6809/3340797836","cluster_addr":"172.21.15.57:6811/3340797836","heartbeat_back_addr":"172.21.15.57:6815/3340797836","heartbeat_front_addr":"172.21.15.57:6813/3340797836","state":["exists","up"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6825","nonce":1700292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6827","nonce":1700292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6831","nonce":1700292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6829","nonce":1700292}]},"public_addr":"172.21.15.57:6825/1700292","cluster_addr":"172.21.15.57:6827/1700292","heartbeat_back_addr":"172.21.15.57:6831/1700292","heartbeat_front_addr":"172.21.15.57:6829/1700292","state":["exists","up"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6817","nonce":2893420631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6819","nonce":2893420631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6823","nonce":2893420631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6821","nonce":2893420631}]},"public_addr":"172.21.15.57:6817/2893420631","cluster_addr":"172.21.15.57:6819/2893420631","heartbeat_back_addr":"172.21.15.57:6823/2893420631","heartbeat_front_addr":"172.21.15.57:6821/2893420631","state":["exists","up"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6807","nonce":2784816485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6811","nonce":2784816485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6815","nonce":2784816485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6813","nonce":2784816485}]},"public_addr":"172.21.15.88:6807/2784816485","cluster_addr":"172.21.15.88:6811/2784816485","heartbeat_back_addr":"172.21.15.88:6815/2784816485","heartbeat_front_addr":"172.21.15.88:6813/2784816485","state":["exists","up"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6801","nonce":2682449812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6803","nonce":2682449812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6810","nonce":2682449812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6806","nonce":2682449812}]},"public_addr":"172.21.15.88:6801/2682449812","cluster_addr":"172.21.15.88:6803/2682449812","heartbeat_back_addr":"172.21.15.88:6810/2682449812","heartbeat_front_addr":"172.21.15.88:6806/2682449812","state":["exists","up"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6822","nonce":3340693897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6827","nonce":3340693897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6831","nonce":3340693897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6829","nonce":3340693897}]},"public_addr":"172.21.15.88:6822/3340693897","cluster_addr":"172.21.15.88:6827/3340693897","heartbeat_back_addr":"172.21.15.88:6831/3340693897","heartbeat_front_addr":"172.21.15.88:6829/3340693897","state":["exists","up"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6817","nonce":2862111199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6820","nonce":2862111199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6825","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6826","nonce":2862111199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6821","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6823","nonce":2862111199}]},"public_addr":"172.21.15.88:6817/2862111199","cluster_addr":"172.21.15.88:6820/2862111199","heartbeat_back_addr":"172.21.15.88:6826/2862111199","heartbeat_front_addr":"172.21.15.88:6823/2862111199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.634920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.773993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.725825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.761452+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.694274+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.650718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6835/2613636524":"2023-10-31T14:50:50.290535+0000","172.21.15.57:6837/816502245":"2023-10-31T14:48:30.252448+0000","172.21.15.88:6838/1744808283":"2023-10-31T14:44:20.161100+0000","172.21.15.57:6834/2321408149":"2023-10-31T14:42:35.131496+0000","172.21.15.57:6833/816502245":"2023-10-31T14:48:30.252448+0000","172.21.15.57:6834/2912050965":"2023-10-31T14:23:54.764165+0000","172.21.15.57:6837/1988100930":"2023-10-31T14:20:34.693623+0000","172.21.15.88:6839/2945854794":"2023-10-31T14:18:54.654873+0000","172.21.15.57:6835/2427081813":"2023-10-31T14:26:24.805473+0000","172.21.15.57:6833/1988100930":"2023-10-31T14:20:34.693623+0000","172.21.15.88:6839/358521496":"2023-10-31T14:30:09.871507+0000","172.21.15.57:6836/2693548867":"2023-10-31T14:17:44.635085+0000","172.21.15.57:6832/2912050965":"2023-10-31T14:23:54.764165+0000","172.21.15.57:6832/3701655528":"2023-10-31T14:16:49.618077+0000","172.21.15.88:6838/358521496":"2023-10-31T14:30:09.871507+0000","172.21.15.57:6837/901968115":"2023-10-31T14:14:59.588353+0000","172.21.15.88:6839/1744808283":"2023-10-31T14:44:20.161100+0000","172.21.15.88:6838/2945854794":"2023-10-31T14:18:54.654873+0000","172.21.15.57:6834/3701655528":"2023-10-31T14:16:49.618077+0000","172.21.15.57:6835/2733323524":"2023-10-31T14:32:54.940701+0000","172.21.15.57:6836/2733323524":"2023-10-31T14:32:54.940701+0000","172.21.15.57:6835/2693548867":"2023-10-31T14:17:44.635085+0000","172.21.15.57:6832/2321408149":"2023-10-31T14:42:35.131496+0000","172.21.15.57:6833/862088802":"2023-10-31T14:27:49.829937+0000","172.21.15.88:6838/3604033338":"2023-10-31T14:35:29.980317+0000","172.21.15.57:6836/2613636524":"2023-10-31T14:50:50.290535+0000","172.21.15.88:6839/2390289388":"2023-10-31T14:39:45.086742+0000","172.21.15.57:6837/862088802":"2023-10-31T14:27:49.829937+0000","172.21.15.88:6838/2390289388":"2023-10-31T14:39:45.086742+0000","172.21.15.57:6832/2940934016":"2023-10-31T14:29:09.852381+0000","172.21.15.57:6834/2940934016":"2023-10-31T14:29:09.852381+0000","172.21.15.88:6839/3604033338":"2023-10-31T14:35:29.980317+0000","172.21.15.57:6833/901968115":"2023-10-31T14:14:59.588353+0000","172.21.15.57:6836/2427081813":"2023-10-31T14:26:24.805473+0000","172.21.15.57:6833/1843528309":"2023-10-31T14:38:15.025313+0000","172.21.15.57:6837/1843528309":"2023-10-31T14:38:15.025313+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:51:36.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.344+0000 7f8276b72640 1 -- 172.21.15.57:0/3537805018 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f82540664a0 msgr2=0x7f8254068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:36.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.344+0000 7f8276b72640 1 --2- 172.21.15.57:0/3537805018 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f82540664a0 0x7f8254068960 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f825c0b2000 tx=0x7f825c009040 comp rx=0 tx=0).stop 2023-10-30T14:51:36.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.344+0000 7f8276b72640 1 -- 172.21.15.57:0/3537805018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82700c4a10 msgr2=0x7f827014bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:36.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.344+0000 7f8276b72640 1 --2- 172.21.15.57:0/3537805018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82700c4a10 0x7f827014bc80 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f826400e930 tx=0x7f826400edf0 comp rx=0 tx=0).stop 2023-10-30T14:51:36.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.344+0000 7f8276b72640 1 -- 172.21.15.57:0/3537805018 shutdown_connections 2023-10-30T14:51:36.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.344+0000 7f8276b72640 1 --2- 172.21.15.57:0/3537805018 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f82540664a0 0x7f8254068960 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:36.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.344+0000 7f8276b72640 1 --2- 172.21.15.57:0/3537805018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f827015a3b0 0x7f8270158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:36.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.344+0000 7f8276b72640 1 --2- 172.21.15.57:0/3537805018 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8270153c80 0x7f827014a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:36.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.344+0000 7f8276b72640 1 --2- 172.21.15.57:0/3537805018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82700c4a10 0x7f827014bc80 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:36.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.344+0000 7f8276b72640 1 -- 172.21.15.57:0/3537805018 >> 172.21.15.57:0/3537805018 conn(0x7f82700715c0 msgr2=0x7f82700719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:36.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.344+0000 7f8276b72640 1 -- 172.21.15.57:0/3537805018 shutdown_connections 2023-10-30T14:51:36.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:36.344+0000 7f8276b72640 1 -- 172.21.15.57:0/3537805018 wait complete. 2023-10-30T14:51:37.368 INFO:tasks.ceph:Scrubbing osd.0 2023-10-30T14:51:37.369 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-30T14:51:37.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.520+0000 7fa219cd6640 1 Processor -- start 2023-10-30T14:51:37.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.520+0000 7fa219cd6640 1 -- start start 2023-10-30T14:51:37.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.520+0000 7fa219cd6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa2140c4340 0x7fa2141599e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:37.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.520+0000 7fa219cd6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21406af00 con 0x7fa2140c4720 2023-10-30T14:51:37.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.520+0000 7fa219cd6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21406a660 con 0x7fa2140ca470 2023-10-30T14:51:37.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.520+0000 7fa219cd6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2140c4cd0 con 0x7fa2140c4340 2023-10-30T14:51:37.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.520+0000 7fa212ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa2140c4720 0x7fa214069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41816/0 (socket says 172.21.15.57:41816) 2023-10-30T14:51:37.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.520+0000 7fa212ffd640 1 -- 172.21.15.57:0/362098521 learned_addr learned my addr 172.21.15.57:0/362098521 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:51:37.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa2137fe640 1 --2- 172.21.15.57:0/362098521 >> v2:172.21.15.88:3300/0 conn(0x7fa2140c4340 0x7fa2141599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:37.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa2127fc640 1 -- 172.21.15.57:0/362098521 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932290221 0 0) 0x7fa21406af00 con 0x7fa2140c4720 2023-10-30T14:51:37.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa2127fc640 1 -- 172.21.15.57:0/362098521 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1fc003530 con 0x7fa2140c4720 2023-10-30T14:51:37.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa2127fc640 1 -- 172.21.15.57:0/362098521 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2097891932 0 0) 0x7fa1fc003530 con 0x7fa2140c4720 2023-10-30T14:51:37.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa2127fc640 1 -- 172.21.15.57:0/362098521 >> v2:172.21.15.88:3300/0 conn(0x7fa2140c4340 msgr2=0x7fa2141599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:37.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa2127fc640 1 --2- 172.21.15.57:0/362098521 >> v2:172.21.15.88:3300/0 conn(0x7fa2140c4340 0x7fa2141599e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa2127fc640 1 -- 172.21.15.57:0/362098521 >> v1:172.21.15.88:6789/0 conn(0x7fa2140ca470 legacy=0x7fa214153880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:37.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa2127fc640 1 -- 172.21.15.57:0/362098521 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21406a940 con 0x7fa2140c4720 2023-10-30T14:51:37.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa2127fc640 1 -- 172.21.15.57:0/362098521 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fa2080016a0 con 0x7fa2140c4720 2023-10-30T14:51:37.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa2127fc640 1 -- 172.21.15.57:0/362098521 >> v1:172.21.15.57:6789/0 conn(0x7fa2140c4720 legacy=0x7fa214069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:37.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa2127fc640 1 --2- 172.21.15.57:0/362098521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1fc003980 0x7fa1fc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:37.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa2127fc640 1 --2- 172.21.15.57:0/362098521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1fc0047d0 0x7fa1fc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:37.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa2127fc640 1 --2- 172.21.15.57:0/362098521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1fc005190 0x7fa1fc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:37.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa2127fc640 1 -- 172.21.15.57:0/362098521 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2140c4cd0 con 0x7fa1fc0047d0 2023-10-30T14:51:37.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa2127fc640 1 -- 172.21.15.57:0/362098521 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc00c130 con 0x7fa1fc003980 2023-10-30T14:51:37.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa2127fc640 1 -- 172.21.15.57:0/362098521 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc00c2f0 con 0x7fa1fc005190 2023-10-30T14:51:37.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa212ffd640 1 --2- 172.21.15.57:0/362098521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1fc0047d0 0x7fa1fc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:37.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa213fff640 1 --2- 172.21.15.57:0/362098521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1fc005190 0x7fa1fc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:37.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa2137fe640 1 --2- 172.21.15.57:0/362098521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1fc003980 0x7fa1fc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:37.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa212ffd640 1 -- 172.21.15.57:0/362098521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1fc005190 msgr2=0x7fa1fc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:37.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa212ffd640 1 --2- 172.21.15.57:0/362098521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1fc005190 0x7fa1fc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa212ffd640 1 -- 172.21.15.57:0/362098521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1fc003980 msgr2=0x7fa1fc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:37.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa212ffd640 1 --2- 172.21.15.57:0/362098521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1fc003980 0x7fa1fc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa212ffd640 1 -- 172.21.15.57:0/362098521 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1fc003530 con 0x7fa1fc0047d0 2023-10-30T14:51:37.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.524+0000 7fa213fff640 1 -- 172.21.15.57:0/362098521 reap_dead start 2023-10-30T14:51:37.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.528+0000 7fa212ffd640 1 --2- 172.21.15.57:0/362098521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1fc0047d0 0x7fa1fc004c50 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fa2140c4b40 tx=0x7fa20801cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd74634352730b89 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:37.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.528+0000 7fa2127fc640 1 -- 172.21.15.57:0/362098521 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa208005ae0 con 0x7fa1fc0047d0 2023-10-30T14:51:37.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.528+0000 7fa2127fc640 1 -- 172.21.15.57:0/362098521 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa208005c40 con 0x7fa1fc0047d0 2023-10-30T14:51:37.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.528+0000 7fa2127fc640 1 -- 172.21.15.57:0/362098521 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa208007070 con 0x7fa1fc0047d0 2023-10-30T14:51:37.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.528+0000 7fa219cd6640 1 -- 172.21.15.57:0/362098521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1fc0047d0 msgr2=0x7fa1fc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:37.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.528+0000 7fa219cd6640 1 --2- 172.21.15.57:0/362098521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1fc0047d0 0x7fa1fc004c50 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fa2140c4b40 tx=0x7fa20801cb70 comp rx=0 tx=0).stop 2023-10-30T14:51:37.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.528+0000 7fa219cd6640 1 -- 172.21.15.57:0/362098521 shutdown_connections 2023-10-30T14:51:37.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.528+0000 7fa219cd6640 1 --2- 172.21.15.57:0/362098521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1fc0047d0 0x7fa1fc004c50 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.528+0000 7fa219cd6640 1 -- 172.21.15.57:0/362098521 >> 172.21.15.57:0/362098521 conn(0x7fa2140cf1c0 msgr2=0x7fa2140cf5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:37.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.528+0000 7fa219cd6640 1 -- 172.21.15.57:0/362098521 shutdown_connections 2023-10-30T14:51:37.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.528+0000 7fa219cd6640 1 -- 172.21.15.57:0/362098521 wait complete. 2023-10-30T14:51:37.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.528+0000 7fa219cd6640 1 Processor -- start 2023-10-30T14:51:37.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.528+0000 7fa219cd6640 1 -- start start 2023-10-30T14:51:37.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.528+0000 7fa219cd6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1fc0047d0 0x7fa2141f19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:37.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa219cd6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2141556e0 0x7fa2140ce740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:37.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa219cd6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa214156ff0 0x7fa21414bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:37.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa219cd6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21414caa0 con 0x7fa1fc0047d0 2023-10-30T14:51:37.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa219cd6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21414baa0 con 0x7fa214156ff0 2023-10-30T14:51:37.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa219cd6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa21414b770 con 0x7fa2141556e0 2023-10-30T14:51:37.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa2137fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1fc0047d0 0x7fa2141f19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:37.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa2137fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1fc0047d0 0x7fa2141f19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56656/0 (socket says 172.21.15.57:56656) 2023-10-30T14:51:37.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa2137fe640 1 -- 172.21.15.57:0/1774040769 learned_addr learned my addr 172.21.15.57:0/1774040769 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:37.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa213fff640 1 --2- 172.21.15.57:0/1774040769 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa214156ff0 0x7fa21414bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:37.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa212ffd640 1 --2- 172.21.15.57:0/1774040769 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2141556e0 0x7fa2140ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:37.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa2137fe640 1 -- 172.21.15.57:0/1774040769 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2141556e0 msgr2=0x7fa2140ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:37.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa2137fe640 1 --2- 172.21.15.57:0/1774040769 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2141556e0 0x7fa2140ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa2137fe640 1 -- 172.21.15.57:0/1774040769 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa214156ff0 msgr2=0x7fa21414bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:37.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa2137fe640 1 --2- 172.21.15.57:0/1774040769 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa214156ff0 0x7fa21414bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa2137fe640 1 -- 172.21.15.57:0/1774040769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2141f2000 con 0x7fa1fc0047d0 2023-10-30T14:51:37.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa213fff640 1 --2- 172.21.15.57:0/1774040769 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa214156ff0 0x7fa21414bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:37.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa2137fe640 1 --2- 172.21.15.57:0/1774040769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1fc0047d0 0x7fa2141f19c0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fa204011500 tx=0x7fa2040119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:37.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa1fbfff640 1 -- 172.21.15.57:0/1774040769 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa204015870 con 0x7fa1fc0047d0 2023-10-30T14:51:37.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa1fbfff640 1 -- 172.21.15.57:0/1774040769 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2040159d0 con 0x7fa1fc0047d0 2023-10-30T14:51:37.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa219cd6640 1 -- 172.21.15.57:0/1774040769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2141f22e0 con 0x7fa1fc0047d0 2023-10-30T14:51:37.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa1fbfff640 1 -- 172.21.15.57:0/1774040769 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa204015d40 con 0x7fa1fc0047d0 2023-10-30T14:51:37.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa219cd6640 1 -- 172.21.15.57:0/1774040769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2141f28a0 con 0x7fa1fc0047d0 2023-10-30T14:51:37.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa219cd6640 1 -- 172.21.15.57:0/1774040769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa2140cac90 con 0x7fa1fc0047d0 2023-10-30T14:51:37.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.532+0000 7fa1fbfff640 1 -- 172.21.15.57:0/1774040769 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa204013400 con 0x7fa1fc0047d0 2023-10-30T14:51:37.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.536+0000 7fa1fbfff640 1 --2- 172.21.15.57:0/1774040769 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2000664a0 0x7fa200068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:37.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.536+0000 7fa1fbfff640 1 -- 172.21.15.57:0/1774040769 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fa204084a50 con 0x7fa1fc0047d0 2023-10-30T14:51:37.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.536+0000 7fa1fbfff640 1 --2- 172.21.15.57:0/1774040769 >> [v2:172.21.15.57:6800/3287367848,v1:172.21.15.57:6801/3287367848] conn(0x7fa20006f640 0x7fa200071ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:37.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.536+0000 7fa1fbfff640 1 -- 172.21.15.57:0/1774040769 --> [v2:172.21.15.57:6800/3287367848,v1:172.21.15.57:6801/3287367848] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa200072190 con 0x7fa20006f640 2023-10-30T14:51:37.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.536+0000 7fa1fbfff640 1 -- 172.21.15.57:0/1774040769 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2040892a0 con 0x7fa1fc0047d0 2023-10-30T14:51:37.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.536+0000 7fa212ffd640 1 --2- 172.21.15.57:0/1774040769 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2000664a0 0x7fa200068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:37.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.536+0000 7fa213fff640 1 --2- 172.21.15.57:0/1774040769 >> [v2:172.21.15.57:6800/3287367848,v1:172.21.15.57:6801/3287367848] conn(0x7fa20006f640 0x7fa200071ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:37.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.536+0000 7fa213fff640 1 --2- 172.21.15.57:0/1774040769 >> [v2:172.21.15.57:6800/3287367848,v1:172.21.15.57:6801/3287367848] conn(0x7fa20006f640 0x7fa200071ac0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:37.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.536+0000 7fa212ffd640 1 --2- 172.21.15.57:0/1774040769 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2000664a0 0x7fa200068960 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fa2080ab000 tx=0x7fa2080b1040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:37.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.536+0000 7fa1fbfff640 1 -- 172.21.15.57:0/1774040769 <== osd.0 v2:172.21.15.57:6800/3287367848 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fa200072190 con 0x7fa20006f640 2023-10-30T14:51:37.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.548+0000 7fa219cd6640 1 -- 172.21.15.57:0/1774040769 --> [v2:172.21.15.57:6800/3287367848,v1:172.21.15.57:6801/3287367848] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa2140c9f50 con 0x7fa20006f640 2023-10-30T14:51:37.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.548+0000 7fa1fbfff640 1 -- 172.21.15.57:0/1774040769 <== osd.0 v2:172.21.15.57:6800/3287367848 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fa20c003c80 con 0x7fa20006f640 2023-10-30T14:51:37.555 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-30T14:51:37.555 INFO:teuthology.orchestra.run.smithi057.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T14:51:37.555 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-30T14:51:37.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.548+0000 7fa219cd6640 1 -- 172.21.15.57:0/1774040769 >> [v2:172.21.15.57:6800/3287367848,v1:172.21.15.57:6801/3287367848] conn(0x7fa20006f640 msgr2=0x7fa200071ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:37.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.548+0000 7fa219cd6640 1 --2- 172.21.15.57:0/1774040769 >> [v2:172.21.15.57:6800/3287367848,v1:172.21.15.57:6801/3287367848] conn(0x7fa20006f640 0x7fa200071ac0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.548+0000 7fa219cd6640 1 -- 172.21.15.57:0/1774040769 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2000664a0 msgr2=0x7fa200068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:37.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.548+0000 7fa219cd6640 1 --2- 172.21.15.57:0/1774040769 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa2000664a0 0x7fa200068960 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fa2080ab000 tx=0x7fa2080b1040 comp rx=0 tx=0).stop 2023-10-30T14:51:37.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.548+0000 7fa219cd6640 1 -- 172.21.15.57:0/1774040769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1fc0047d0 msgr2=0x7fa2141f19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:37.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.548+0000 7fa219cd6640 1 --2- 172.21.15.57:0/1774040769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1fc0047d0 0x7fa2141f19c0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fa204011500 tx=0x7fa2040119c0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.548+0000 7fa213fff640 1 -- 172.21.15.57:0/1774040769 reap_dead start 2023-10-30T14:51:37.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.548+0000 7fa219cd6640 1 -- 172.21.15.57:0/1774040769 shutdown_connections 2023-10-30T14:51:37.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.548+0000 7fa219cd6640 1 -- 172.21.15.57:0/1774040769 >> 172.21.15.57:0/1774040769 conn(0x7fa2140cf1c0 msgr2=0x7fa214159070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:37.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.548+0000 7fa219cd6640 1 -- 172.21.15.57:0/1774040769 shutdown_connections 2023-10-30T14:51:37.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.548+0000 7fa219cd6640 1 -- 172.21.15.57:0/1774040769 wait complete. 2023-10-30T14:51:37.572 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-30T14:51:37.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.712+0000 7ffbd5965640 1 Processor -- start 2023-10-30T14:51:37.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.712+0000 7ffbd5965640 1 -- start start 2023-10-30T14:51:37.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.712+0000 7ffbd5965640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ffbd0153b10 0x7ffbd00c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:37.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.712+0000 7ffbd5965640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ffbd00ca8e0 0x7ffbd006bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:37.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.712+0000 7ffbd5965640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffbd006c480 0x7ffbd0159d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:37.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.712+0000 7ffbd5965640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd0159900 con 0x7ffbd00ca8e0 2023-10-30T14:51:37.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.712+0000 7ffbd5965640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd015d070 con 0x7ffbd0153b10 2023-10-30T14:51:37.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.712+0000 7ffbd5965640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd015d350 con 0x7ffbd006c480 2023-10-30T14:51:37.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.712+0000 7ffbc7fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ffbd00ca8e0 0x7ffbd006bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:37.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.712+0000 7ffbc7fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ffbd00ca8e0 0x7ffbd006bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56672/0 (socket says 172.21.15.57:56672) 2023-10-30T14:51:37.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.712+0000 7ffbc7fff640 1 -- 172.21.15.57:0/1031088294 learned_addr learned my addr 172.21.15.57:0/1031088294 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:37.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.712+0000 7ffbcf7fe640 1 --2- 172.21.15.57:0/1031088294 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffbd006c480 0x7ffbd0159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:37.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.712+0000 7ffbceffd640 1 --2- 172.21.15.57:0/1031088294 >> v2:172.21.15.88:3300/0 conn(0x7ffbd0153b10 0x7ffbd00c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:37.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.712+0000 7ffbc7fff640 1 -- 172.21.15.57:0/1031088294 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffbd006c480 msgr2=0x7ffbd0159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:37.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.712+0000 7ffbc7fff640 1 --2- 172.21.15.57:0/1031088294 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffbd006c480 0x7ffbd0159d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.712+0000 7ffbc7fff640 1 -- 172.21.15.57:0/1031088294 >> v2:172.21.15.88:3300/0 conn(0x7ffbd0153b10 msgr2=0x7ffbd00c9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:37.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.712+0000 7ffbc7fff640 1 --2- 172.21.15.57:0/1031088294 >> v2:172.21.15.88:3300/0 conn(0x7ffbd0153b10 0x7ffbd00c9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.712+0000 7ffbc7fff640 1 -- 172.21.15.57:0/1031088294 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd015d630 con 0x7ffbd00ca8e0 2023-10-30T14:51:37.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.712+0000 7ffbceffd640 1 --2- 172.21.15.57:0/1031088294 >> v2:172.21.15.88:3300/0 conn(0x7ffbd0153b10 0x7ffbd00c9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:37.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.716+0000 7ffbc7fff640 1 --2- 172.21.15.57:0/1031088294 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffbd00ca8e0 0x7ffbd006bf40 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7ffbd00cc7e0 tx=0x7ffbb801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ead3ef6854e001d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:37.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.716+0000 7ffbce7fc640 1 -- 172.21.15.57:0/1031088294 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb80ac630 con 0x7ffbd00ca8e0 2023-10-30T14:51:37.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.716+0000 7ffbce7fc640 1 -- 172.21.15.57:0/1031088294 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb80ac790 con 0x7ffbd00ca8e0 2023-10-30T14:51:37.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.716+0000 7ffbce7fc640 1 -- 172.21.15.57:0/1031088294 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb80acb20 con 0x7ffbd00ca8e0 2023-10-30T14:51:37.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.716+0000 7ffbd5965640 1 -- 172.21.15.57:0/1031088294 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffbd00ca8e0 msgr2=0x7ffbd006bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:37.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.716+0000 7ffbd5965640 1 --2- 172.21.15.57:0/1031088294 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffbd00ca8e0 0x7ffbd006bf40 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7ffbd00cc7e0 tx=0x7ffbb801c500 comp rx=0 tx=0).stop 2023-10-30T14:51:37.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.716+0000 7ffbd5965640 1 -- 172.21.15.57:0/1031088294 shutdown_connections 2023-10-30T14:51:37.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.716+0000 7ffbd5965640 1 --2- 172.21.15.57:0/1031088294 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffbd006c480 0x7ffbd0159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.716+0000 7ffbd5965640 1 --2- 172.21.15.57:0/1031088294 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffbd00ca8e0 0x7ffbd006bf40 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.716+0000 7ffbd5965640 1 --2- 172.21.15.57:0/1031088294 >> v2:172.21.15.88:3300/0 conn(0x7ffbd0153b10 0x7ffbd00c9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.716+0000 7ffbd5965640 1 -- 172.21.15.57:0/1031088294 >> 172.21.15.57:0/1031088294 conn(0x7ffbd006a240 msgr2=0x7ffbd006a650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:37.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.716+0000 7ffbd5965640 1 -- 172.21.15.57:0/1031088294 shutdown_connections 2023-10-30T14:51:37.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.716+0000 7ffbd5965640 1 -- 172.21.15.57:0/1031088294 wait complete. 2023-10-30T14:51:37.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbd5965640 1 Processor -- start 2023-10-30T14:51:37.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbd5965640 1 -- start start 2023-10-30T14:51:37.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbd5965640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffbd006c480 0x7ffbd0152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:37.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbd5965640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffbd00ca8e0 0x7ffbd01523c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:37.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbd5965640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffbd0153b10 0x7ffbd0145f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:37.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbd5965640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd01417b0 con 0x7ffbd00ca8e0 2023-10-30T14:51:37.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbd5965640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd0141ce0 con 0x7ffbd006c480 2023-10-30T14:51:37.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbd5965640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd015ca90 con 0x7ffbd0153b10 2023-10-30T14:51:37.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbc7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffbd00ca8e0 0x7ffbd01523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:37.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbc7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffbd00ca8e0 0x7ffbd01523c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56674/0 (socket says 172.21.15.57:56674) 2023-10-30T14:51:37.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbc7fff640 1 -- 172.21.15.57:0/3815064746 learned_addr learned my addr 172.21.15.57:0/3815064746 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:37.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbceffd640 1 --2- 172.21.15.57:0/3815064746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffbd006c480 0x7ffbd0152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:37.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbcf7fe640 1 --2- 172.21.15.57:0/3815064746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffbd0153b10 0x7ffbd0145f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:37.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbc7fff640 1 -- 172.21.15.57:0/3815064746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffbd0153b10 msgr2=0x7ffbd0145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:37.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbc7fff640 1 --2- 172.21.15.57:0/3815064746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffbd0153b10 0x7ffbd0145f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbc7fff640 1 -- 172.21.15.57:0/3815064746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffbd006c480 msgr2=0x7ffbd0152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:37.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbc7fff640 1 --2- 172.21.15.57:0/3815064746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffbd006c480 0x7ffbd0152a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbc7fff640 1 -- 172.21.15.57:0/3815064746 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb80094d0 con 0x7ffbd00ca8e0 2023-10-30T14:51:37.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbc7fff640 1 --2- 172.21.15.57:0/3815064746 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffbd00ca8e0 0x7ffbd01523c0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7ffbb801cda0 tx=0x7ffbb80192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:37.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbccff9640 1 -- 172.21.15.57:0/3815064746 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb80b5640 con 0x7ffbd00ca8e0 2023-10-30T14:51:37.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbccff9640 1 -- 172.21.15.57:0/3815064746 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb80b57a0 con 0x7ffbd00ca8e0 2023-10-30T14:51:37.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbccff9640 1 -- 172.21.15.57:0/3815064746 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb80b5b50 con 0x7ffbd00ca8e0 2023-10-30T14:51:37.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbd5965640 1 -- 172.21.15.57:0/3815064746 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd0141a00 con 0x7ffbd00ca8e0 2023-10-30T14:51:37.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.720+0000 7ffbd5965640 1 -- 172.21.15.57:0/3815064746 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd015c720 con 0x7ffbd00ca8e0 2023-10-30T14:51:37.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.724+0000 7ffbd5965640 1 -- 172.21.15.57:0/3815064746 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd0067540 con 0x7ffbd00ca8e0 2023-10-30T14:51:37.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.724+0000 7ffbccff9640 1 -- 172.21.15.57:0/3815064746 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffbb80b3070 con 0x7ffbd00ca8e0 2023-10-30T14:51:37.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.724+0000 7ffbccff9640 1 --2- 172.21.15.57:0/3815064746 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffbb00664a0 0x7ffbb0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:37.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.724+0000 7ffbccff9640 1 -- 172.21.15.57:0/3815064746 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7ffbb811e3a0 con 0x7ffbd00ca8e0 2023-10-30T14:51:37.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.724+0000 7ffbceffd640 1 --2- 172.21.15.57:0/3815064746 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffbb00664a0 0x7ffbb0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:37.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.724+0000 7ffbceffd640 1 --2- 172.21.15.57:0/3815064746 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffbb00664a0 0x7ffbb0068960 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00045f0 tx=0x7ffbc0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:37.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.728+0000 7ffbccff9640 1 -- 172.21.15.57:0/3815064746 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffbb80eb360 con 0x7ffbd00ca8e0 2023-10-30T14:51:37.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.884+0000 7ffbd5965640 1 -- 172.21.15.57:0/3815064746 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7ffbd00ccaf0 con 0x7ffbb00664a0 2023-10-30T14:51:37.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.888+0000 7ffbccff9640 1 -- 172.21.15.57:0/3815064746 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ffbc000b720 con 0x7ffbb00664a0 2023-10-30T14:51:37.891 INFO:teuthology.orchestra.run.smithi057.stderr:instructed osd(s) 0 to deep-scrub 2023-10-30T14:51:37.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.888+0000 7ffbd5965640 1 -- 172.21.15.57:0/3815064746 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffbb00664a0 msgr2=0x7ffbb0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:37.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.888+0000 7ffbd5965640 1 --2- 172.21.15.57:0/3815064746 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffbb00664a0 0x7ffbb0068960 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00045f0 tx=0x7ffbc0008040 comp rx=0 tx=0).stop 2023-10-30T14:51:37.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.888+0000 7ffbd5965640 1 -- 172.21.15.57:0/3815064746 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffbd00ca8e0 msgr2=0x7ffbd01523c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:37.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.888+0000 7ffbd5965640 1 --2- 172.21.15.57:0/3815064746 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffbd00ca8e0 0x7ffbd01523c0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7ffbb801cda0 tx=0x7ffbb80192a0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.888+0000 7ffbd5965640 1 -- 172.21.15.57:0/3815064746 shutdown_connections 2023-10-30T14:51:37.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.888+0000 7ffbd5965640 1 --2- 172.21.15.57:0/3815064746 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7ffbb00664a0 0x7ffbb0068960 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.888+0000 7ffbd5965640 1 --2- 172.21.15.57:0/3815064746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffbd0153b10 0x7ffbd0145f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.888+0000 7ffbd5965640 1 --2- 172.21.15.57:0/3815064746 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffbd00ca8e0 0x7ffbd01523c0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.888+0000 7ffbd5965640 1 --2- 172.21.15.57:0/3815064746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffbd006c480 0x7ffbd0152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:37.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.888+0000 7ffbd5965640 1 -- 172.21.15.57:0/3815064746 >> 172.21.15.57:0/3815064746 conn(0x7ffbd006a240 msgr2=0x7ffbd0158b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:37.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.888+0000 7ffbd5965640 1 -- 172.21.15.57:0/3815064746 shutdown_connections 2023-10-30T14:51:37.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:37.888+0000 7ffbd5965640 1 -- 172.21.15.57:0/3815064746 wait complete. 2023-10-30T14:51:37.912 INFO:tasks.ceph:Scrubbing osd.1 2023-10-30T14:51:37.912 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-30T14:51:38.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5b293e640 1 Processor -- start 2023-10-30T14:51:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5b293e640 1 -- start start 2023-10-30T14:51:38.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5b293e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa5ac0c0930 0x7fa5ac06fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5b293e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac0c74d0 con 0x7fa5ac150690 2023-10-30T14:51:38.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5b293e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac0718a0 con 0x7fa5ac0c7770 2023-10-30T14:51:38.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5b293e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5ac072420 con 0x7fa5ac0c0930 2023-10-30T14:51:38.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5abfff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa5ac150690 0x7fa5ac156c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41828/0 (socket says 172.21.15.57:41828) 2023-10-30T14:51:38.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5abfff640 1 -- 172.21.15.57:0/1761555201 learned_addr learned my addr 172.21.15.57:0/1761555201 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:51:38.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5ab7fe640 1 --2- 172.21.15.57:0/1761555201 >> v2:172.21.15.88:3300/0 conn(0x7fa5ac0c0930 0x7fa5ac06fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5aaffd640 1 -- 172.21.15.57:0/1761555201 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033035217 0 0) 0x7fa59c001120 con 0x7fa5ac150690 2023-10-30T14:51:38.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5aaffd640 1 -- 172.21.15.57:0/1761555201 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa594003530 con 0x7fa5ac150690 2023-10-30T14:51:38.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5aaffd640 1 -- 172.21.15.57:0/1761555201 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1213195403 0 0) 0x7fa594003530 con 0x7fa5ac150690 2023-10-30T14:51:38.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5aaffd640 1 -- 172.21.15.57:0/1761555201 >> v2:172.21.15.88:3300/0 conn(0x7fa5ac0c0930 msgr2=0x7fa5ac06fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:38.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5aaffd640 1 --2- 172.21.15.57:0/1761555201 >> v2:172.21.15.88:3300/0 conn(0x7fa5ac0c0930 0x7fa5ac06fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5aaffd640 1 -- 172.21.15.57:0/1761555201 >> v1:172.21.15.88:6789/0 conn(0x7fa5ac0c7770 legacy=0x7fa5ac160e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5aaffd640 1 -- 172.21.15.57:0/1761555201 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ac071b80 con 0x7fa5ac150690 2023-10-30T14:51:38.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5aaffd640 1 -- 172.21.15.57:0/1761555201 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fa59c001810 con 0x7fa5ac150690 2023-10-30T14:51:38.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5aaffd640 1 -- 172.21.15.57:0/1761555201 >> v1:172.21.15.57:6789/0 conn(0x7fa5ac150690 legacy=0x7fa5ac156c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5aaffd640 1 --2- 172.21.15.57:0/1761555201 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa594003980 0x7fa594003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5aaffd640 1 --2- 172.21.15.57:0/1761555201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5940047d0 0x7fa594004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5aaffd640 1 --2- 172.21.15.57:0/1761555201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa594005190 0x7fa59400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5aaffd640 1 -- 172.21.15.57:0/1761555201 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ac072420 con 0x7fa594005190 2023-10-30T14:51:38.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5aaffd640 1 -- 172.21.15.57:0/1761555201 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59400c040 con 0x7fa5940047d0 2023-10-30T14:51:38.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.056+0000 7fa5aaffd640 1 -- 172.21.15.57:0/1761555201 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa59400c200 con 0x7fa594003980 2023-10-30T14:51:38.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.060+0000 7fa5b0edb640 1 --2- 172.21.15.57:0/1761555201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa594005190 0x7fa59400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.060+0000 7fa5abfff640 1 --2- 172.21.15.57:0/1761555201 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa594003980 0x7fa594003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.060+0000 7fa5ab7fe640 1 --2- 172.21.15.57:0/1761555201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5940047d0 0x7fa594004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.060+0000 7fa5b0edb640 1 -- 172.21.15.57:0/1761555201 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa594003980 msgr2=0x7fa594003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:38.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.060+0000 7fa5b0edb640 1 --2- 172.21.15.57:0/1761555201 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa594003980 0x7fa594003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.060+0000 7fa5b0edb640 1 -- 172.21.15.57:0/1761555201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5940047d0 msgr2=0x7fa594004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:38.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.060+0000 7fa5b0edb640 1 --2- 172.21.15.57:0/1761555201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5940047d0 0x7fa594004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.060+0000 7fa5b0edb640 1 -- 172.21.15.57:0/1761555201 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa594003530 con 0x7fa594005190 2023-10-30T14:51:38.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.060+0000 7fa5ab7fe640 1 --2- 172.21.15.57:0/1761555201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5940047d0 0x7fa594004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:38.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.060+0000 7fa5b0edb640 1 -- 172.21.15.57:0/1761555201 reap_dead start 2023-10-30T14:51:38.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.060+0000 7fa5b0edb640 1 --2- 172.21.15.57:0/1761555201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa594005190 0x7fa59400b990 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4001e70 tx=0x7fa5a401c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9f391de440f7fb0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:38.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.060+0000 7fa5aaffd640 1 -- 172.21.15.57:0/1761555201 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4003b80 con 0x7fa594005190 2023-10-30T14:51:38.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.060+0000 7fa5aaffd640 1 -- 172.21.15.57:0/1761555201 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a4003d20 con 0x7fa594005190 2023-10-30T14:51:38.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.060+0000 7fa5aaffd640 1 -- 172.21.15.57:0/1761555201 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4005260 con 0x7fa594005190 2023-10-30T14:51:38.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.060+0000 7fa5b293e640 1 -- 172.21.15.57:0/1761555201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa594005190 msgr2=0x7fa59400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.060+0000 7fa5b293e640 1 --2- 172.21.15.57:0/1761555201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa594005190 0x7fa59400b990 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4001e70 tx=0x7fa5a401c760 comp rx=0 tx=0).stop 2023-10-30T14:51:38.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b293e640 1 -- 172.21.15.57:0/1761555201 shutdown_connections 2023-10-30T14:51:38.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b293e640 1 --2- 172.21.15.57:0/1761555201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa594005190 0x7fa59400b990 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b293e640 1 -- 172.21.15.57:0/1761555201 >> 172.21.15.57:0/1761555201 conn(0x7fa5ac06e150 msgr2=0x7fa5ac069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:38.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b293e640 1 -- 172.21.15.57:0/1761555201 shutdown_connections 2023-10-30T14:51:38.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b293e640 1 -- 172.21.15.57:0/1761555201 wait complete. 2023-10-30T14:51:38.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b293e640 1 Processor -- start 2023-10-30T14:51:38.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b293e640 1 -- start start 2023-10-30T14:51:38.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b293e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa594005190 0x7fa5ac1f2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b293e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5ac14d570 0x7fa5ac0c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b293e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5ac158a00 0x7fa5ac158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b293e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ac1593f0 con 0x7fa5ac158a00 2023-10-30T14:51:38.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b293e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ac1eee20 con 0x7fa594005190 2023-10-30T14:51:38.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b293e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ac1ef100 con 0x7fa5ac14d570 2023-10-30T14:51:38.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b0edb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5ac158a00 0x7fa5ac158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b0edb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5ac158a00 0x7fa5ac158eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56688/0 (socket says 172.21.15.57:56688) 2023-10-30T14:51:38.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b0edb640 1 -- 172.21.15.57:0/4185715251 learned_addr learned my addr 172.21.15.57:0/4185715251 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:38.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5abfff640 1 --2- 172.21.15.57:0/4185715251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa594005190 0x7fa5ac1f2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5ab7fe640 1 --2- 172.21.15.57:0/4185715251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5ac14d570 0x7fa5ac0c9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b0edb640 1 -- 172.21.15.57:0/4185715251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5ac14d570 msgr2=0x7fa5ac0c9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b0edb640 1 --2- 172.21.15.57:0/4185715251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5ac14d570 0x7fa5ac0c9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b0edb640 1 -- 172.21.15.57:0/4185715251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa594005190 msgr2=0x7fa5ac1f2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b0edb640 1 --2- 172.21.15.57:0/4185715251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa594005190 0x7fa5ac1f2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b0edb640 1 -- 172.21.15.57:0/4185715251 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ac1ef3e0 con 0x7fa5ac158a00 2023-10-30T14:51:38.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.064+0000 7fa5b0edb640 1 --2- 172.21.15.57:0/4185715251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5ac158a00 0x7fa5ac158eb0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fa5a401cc40 tx=0x7fa594005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:38.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.068+0000 7fa5a8ff9640 1 -- 172.21.15.57:0/4185715251 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a40ac070 con 0x7fa5ac158a00 2023-10-30T14:51:38.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.068+0000 7fa5a8ff9640 1 -- 172.21.15.57:0/4185715251 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a4003e90 con 0x7fa5ac158a00 2023-10-30T14:51:38.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.068+0000 7fa5a8ff9640 1 -- 172.21.15.57:0/4185715251 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4006d60 con 0x7fa5ac158a00 2023-10-30T14:51:38.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.068+0000 7fa5b293e640 1 -- 172.21.15.57:0/4185715251 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ac1ef6c0 con 0x7fa5ac158a00 2023-10-30T14:51:38.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.068+0000 7fa5b293e640 1 -- 172.21.15.57:0/4185715251 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ac0c61d0 con 0x7fa5ac158a00 2023-10-30T14:51:38.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.068+0000 7fa5b293e640 1 -- 172.21.15.57:0/4185715251 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa5ac071e60 con 0x7fa5ac158a00 2023-10-30T14:51:38.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.068+0000 7fa5a8ff9640 1 -- 172.21.15.57:0/4185715251 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa5a4007030 con 0x7fa5ac158a00 2023-10-30T14:51:38.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.068+0000 7fa5a8ff9640 1 --2- 172.21.15.57:0/4185715251 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa5980664d0 0x7fa598068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.068+0000 7fa5a8ff9640 1 -- 172.21.15.57:0/4185715251 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fa5a411dcb0 con 0x7fa5ac158a00 2023-10-30T14:51:38.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.068+0000 7fa5a8ff9640 1 --2- 172.21.15.57:0/4185715251 >> [v2:172.21.15.57:6808/3340797836,v1:172.21.15.57:6809/3340797836] conn(0x7fa59806f670 0x7fa598071af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.068+0000 7fa5a8ff9640 1 -- 172.21.15.57:0/4185715251 --> [v2:172.21.15.57:6808/3340797836,v1:172.21.15.57:6809/3340797836] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa5980721c0 con 0x7fa59806f670 2023-10-30T14:51:38.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.068+0000 7fa5a8ff9640 1 -- 172.21.15.57:0/4185715251 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5a411e020 con 0x7fa5ac158a00 2023-10-30T14:51:38.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.068+0000 7fa5abfff640 1 --2- 172.21.15.57:0/4185715251 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa5980664d0 0x7fa598068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.068+0000 7fa5ab7fe640 1 --2- 172.21.15.57:0/4185715251 >> [v2:172.21.15.57:6808/3340797836,v1:172.21.15.57:6809/3340797836] conn(0x7fa59806f670 0x7fa598071af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.072+0000 7fa5ab7fe640 1 --2- 172.21.15.57:0/4185715251 >> [v2:172.21.15.57:6808/3340797836,v1:172.21.15.57:6809/3340797836] conn(0x7fa59806f670 0x7fa598071af0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:38.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.072+0000 7fa5abfff640 1 --2- 172.21.15.57:0/4185715251 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa5980664d0 0x7fa598068990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fa59c001060 tx=0x7fa59c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:38.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.072+0000 7fa5a8ff9640 1 -- 172.21.15.57:0/4185715251 <== osd.1 v2:172.21.15.57:6808/3340797836 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fa5980721c0 con 0x7fa59806f670 2023-10-30T14:51:38.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.084+0000 7fa5b293e640 1 -- 172.21.15.57:0/4185715251 --> [v2:172.21.15.57:6808/3340797836,v1:172.21.15.57:6809/3340797836] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa5ac0716e0 con 0x7fa59806f670 2023-10-30T14:51:38.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.084+0000 7fa5a8ff9640 1 -- 172.21.15.57:0/4185715251 <== osd.1 v2:172.21.15.57:6808/3340797836 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fa5a0009a40 con 0x7fa59806f670 2023-10-30T14:51:38.090 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-30T14:51:38.091 INFO:teuthology.orchestra.run.smithi057.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T14:51:38.091 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-30T14:51:38.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.084+0000 7fa5b293e640 1 -- 172.21.15.57:0/4185715251 >> [v2:172.21.15.57:6808/3340797836,v1:172.21.15.57:6809/3340797836] conn(0x7fa59806f670 msgr2=0x7fa598071af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.084+0000 7fa5b293e640 1 --2- 172.21.15.57:0/4185715251 >> [v2:172.21.15.57:6808/3340797836,v1:172.21.15.57:6809/3340797836] conn(0x7fa59806f670 0x7fa598071af0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.084+0000 7fa5b293e640 1 -- 172.21.15.57:0/4185715251 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa5980664d0 msgr2=0x7fa598068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.084+0000 7fa5b293e640 1 --2- 172.21.15.57:0/4185715251 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fa5980664d0 0x7fa598068990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fa59c001060 tx=0x7fa59c009040 comp rx=0 tx=0).stop 2023-10-30T14:51:38.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.084+0000 7fa5b293e640 1 -- 172.21.15.57:0/4185715251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5ac158a00 msgr2=0x7fa5ac158eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.084+0000 7fa5b293e640 1 --2- 172.21.15.57:0/4185715251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5ac158a00 0x7fa5ac158eb0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fa5a401cc40 tx=0x7fa594005800 comp rx=0 tx=0).stop 2023-10-30T14:51:38.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.084+0000 7fa5b0edb640 1 -- 172.21.15.57:0/4185715251 reap_dead start 2023-10-30T14:51:38.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.084+0000 7fa5b293e640 1 -- 172.21.15.57:0/4185715251 shutdown_connections 2023-10-30T14:51:38.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.084+0000 7fa5b293e640 1 -- 172.21.15.57:0/4185715251 >> 172.21.15.57:0/4185715251 conn(0x7fa5ac06e150 msgr2=0x7fa5ac069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:38.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.084+0000 7fa5b293e640 1 -- 172.21.15.57:0/4185715251 shutdown_connections 2023-10-30T14:51:38.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.084+0000 7fa5b293e640 1 -- 172.21.15.57:0/4185715251 wait complete. 2023-10-30T14:51:38.108 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-30T14:51:38.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.248+0000 7fb45b3ee640 1 Processor -- start 2023-10-30T14:51:38.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.248+0000 7fb45b3ee640 1 -- start start 2023-10-30T14:51:38.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.248+0000 7fb45b3ee640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb454153c80 0x7fb45415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.248+0000 7fb45b3ee640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb4540c4a10 0x7fb4540c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.248+0000 7fb45b3ee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb45415a3b0 0x7fb4540ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.248+0000 7fb45b3ee640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb45415dec0 con 0x7fb4540c4a10 2023-10-30T14:51:38.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.248+0000 7fb45b3ee640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4541599a0 con 0x7fb454153c80 2023-10-30T14:51:38.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.248+0000 7fb45b3ee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb454159f20 con 0x7fb45415a3b0 2023-10-30T14:51:38.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.248+0000 7fb458989640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb4540c4a10 0x7fb4540c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.248+0000 7fb458989640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb4540c4a10 0x7fb4540c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56698/0 (socket says 172.21.15.57:56698) 2023-10-30T14:51:38.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.248+0000 7fb458989640 1 -- 172.21.15.57:0/908363619 learned_addr learned my addr 172.21.15.57:0/908363619 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:38.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.248+0000 7fb45998b640 1 --2- 172.21.15.57:0/908363619 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb45415a3b0 0x7fb4540ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.248+0000 7fb45918a640 1 --2- 172.21.15.57:0/908363619 >> v2:172.21.15.88:3300/0 conn(0x7fb454153c80 0x7fb45415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.248+0000 7fb458989640 1 -- 172.21.15.57:0/908363619 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb45415a3b0 msgr2=0x7fb4540ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:38.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.248+0000 7fb458989640 1 --2- 172.21.15.57:0/908363619 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb45415a3b0 0x7fb4540ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.248+0000 7fb458989640 1 -- 172.21.15.57:0/908363619 >> v2:172.21.15.88:3300/0 conn(0x7fb454153c80 msgr2=0x7fb45415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:38.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.248+0000 7fb458989640 1 --2- 172.21.15.57:0/908363619 >> v2:172.21.15.88:3300/0 conn(0x7fb454153c80 0x7fb45415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.248+0000 7fb458989640 1 -- 172.21.15.57:0/908363619 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4540c47e0 con 0x7fb4540c4a10 2023-10-30T14:51:38.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.252+0000 7fb458989640 1 --2- 172.21.15.57:0/908363619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4540c4a10 0x7fb4540c42a0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fb4540d00f0 tx=0x7fb44401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7028719d1e0767e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:38.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.252+0000 7fb44bfff640 1 -- 172.21.15.57:0/908363619 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4440a89b0 con 0x7fb4540c4a10 2023-10-30T14:51:38.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.252+0000 7fb44bfff640 1 -- 172.21.15.57:0/908363619 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4440a8b10 con 0x7fb4540c4a10 2023-10-30T14:51:38.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.252+0000 7fb44bfff640 1 -- 172.21.15.57:0/908363619 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4440a8ea0 con 0x7fb4540c4a10 2023-10-30T14:51:38.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.252+0000 7fb45b3ee640 1 -- 172.21.15.57:0/908363619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4540c4a10 msgr2=0x7fb4540c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.252+0000 7fb45b3ee640 1 --2- 172.21.15.57:0/908363619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4540c4a10 0x7fb4540c42a0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fb4540d00f0 tx=0x7fb44401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.252+0000 7fb45b3ee640 1 -- 172.21.15.57:0/908363619 shutdown_connections 2023-10-30T14:51:38.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.252+0000 7fb45b3ee640 1 --2- 172.21.15.57:0/908363619 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb45415a3b0 0x7fb4540ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.252+0000 7fb45b3ee640 1 --2- 172.21.15.57:0/908363619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4540c4a10 0x7fb4540c42a0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.252+0000 7fb45b3ee640 1 --2- 172.21.15.57:0/908363619 >> v2:172.21.15.88:3300/0 conn(0x7fb454153c80 0x7fb45415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.252+0000 7fb45b3ee640 1 -- 172.21.15.57:0/908363619 >> 172.21.15.57:0/908363619 conn(0x7fb4540715c0 msgr2=0x7fb4540719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:38.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.252+0000 7fb45b3ee640 1 -- 172.21.15.57:0/908363619 shutdown_connections 2023-10-30T14:51:38.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.252+0000 7fb45b3ee640 1 -- 172.21.15.57:0/908363619 wait complete. 2023-10-30T14:51:38.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45b3ee640 1 Processor -- start 2023-10-30T14:51:38.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45b3ee640 1 -- start start 2023-10-30T14:51:38.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45b3ee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4540c4a10 0x7fb4541f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45b3ee640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb454153c80 0x7fb4540b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45b3ee640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb45415a3b0 0x7fb454147c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45b3ee640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4541f2c20 con 0x7fb45415a3b0 2023-10-30T14:51:38.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45b3ee640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb454148620 con 0x7fb454153c80 2023-10-30T14:51:38.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45b3ee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4541ee870 con 0x7fb4540c4a10 2023-10-30T14:51:38.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45998b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb45415a3b0 0x7fb454147c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45998b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb45415a3b0 0x7fb454147c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56702/0 (socket says 172.21.15.57:56702) 2023-10-30T14:51:38.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45998b640 1 -- 172.21.15.57:0/2634349966 learned_addr learned my addr 172.21.15.57:0/2634349966 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:38.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45918a640 1 --2- 172.21.15.57:0/2634349966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4540c4a10 0x7fb4541f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb458989640 1 --2- 172.21.15.57:0/2634349966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb454153c80 0x7fb4540b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45998b640 1 -- 172.21.15.57:0/2634349966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4540c4a10 msgr2=0x7fb4541f26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45998b640 1 --2- 172.21.15.57:0/2634349966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4540c4a10 0x7fb4541f26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45998b640 1 -- 172.21.15.57:0/2634349966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb454153c80 msgr2=0x7fb4540b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45998b640 1 --2- 172.21.15.57:0/2634349966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb454153c80 0x7fb4540b4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45998b640 1 -- 172.21.15.57:0/2634349966 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4440094d0 con 0x7fb45415a3b0 2023-10-30T14:51:38.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb458989640 1 --2- 172.21.15.57:0/2634349966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb454153c80 0x7fb4540b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:38.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45998b640 1 --2- 172.21.15.57:0/2634349966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb45415a3b0 0x7fb454147c10 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00efc0 tx=0x7fb44c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:38.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb449ffb640 1 -- 172.21.15.57:0/2634349966 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c009ba0 con 0x7fb45415a3b0 2023-10-30T14:51:38.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb449ffb640 1 -- 172.21.15.57:0/2634349966 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44c00f040 con 0x7fb45415a3b0 2023-10-30T14:51:38.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb449ffb640 1 -- 172.21.15.57:0/2634349966 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c0146c0 con 0x7fb45415a3b0 2023-10-30T14:51:38.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45b3ee640 1 -- 172.21.15.57:0/2634349966 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4541eeb50 con 0x7fb45415a3b0 2023-10-30T14:51:38.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.256+0000 7fb45b3ee640 1 -- 172.21.15.57:0/2634349966 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4541ef110 con 0x7fb45415a3b0 2023-10-30T14:51:38.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.260+0000 7fb45b3ee640 1 -- 172.21.15.57:0/2634349966 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb45404e170 con 0x7fb45415a3b0 2023-10-30T14:51:38.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.260+0000 7fb449ffb640 1 -- 172.21.15.57:0/2634349966 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb44c014820 con 0x7fb45415a3b0 2023-10-30T14:51:38.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.260+0000 7fb449ffb640 1 --2- 172.21.15.57:0/2634349966 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4380664a0 0x7fb438068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.260+0000 7fb449ffb640 1 -- 172.21.15.57:0/2634349966 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fb44c085090 con 0x7fb45415a3b0 2023-10-30T14:51:38.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.260+0000 7fb45918a640 1 --2- 172.21.15.57:0/2634349966 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4380664a0 0x7fb438068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.260+0000 7fb45918a640 1 --2- 172.21.15.57:0/2634349966 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4380664a0 0x7fb438068960 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fb450009ae0 tx=0x7fb450009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:38.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.264+0000 7fb449ffb640 1 -- 172.21.15.57:0/2634349966 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb44c051df0 con 0x7fb45415a3b0 2023-10-30T14:51:38.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.424+0000 7fb45b3ee640 1 -- 172.21.15.57:0/2634349966 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb45415d900 con 0x7fb4380664a0 2023-10-30T14:51:38.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.424+0000 7fb449ffb640 1 -- 172.21.15.57:0/2634349966 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb450011a00 con 0x7fb4380664a0 2023-10-30T14:51:38.430 INFO:teuthology.orchestra.run.smithi057.stderr:instructed osd(s) 1 to deep-scrub 2023-10-30T14:51:38.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.428+0000 7fb45b3ee640 1 -- 172.21.15.57:0/2634349966 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4380664a0 msgr2=0x7fb438068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.428+0000 7fb45b3ee640 1 --2- 172.21.15.57:0/2634349966 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4380664a0 0x7fb438068960 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fb450009ae0 tx=0x7fb450009190 comp rx=0 tx=0).stop 2023-10-30T14:51:38.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.428+0000 7fb45b3ee640 1 -- 172.21.15.57:0/2634349966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb45415a3b0 msgr2=0x7fb454147c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.428+0000 7fb45b3ee640 1 --2- 172.21.15.57:0/2634349966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb45415a3b0 0x7fb454147c10 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00efc0 tx=0x7fb44c00c490 comp rx=0 tx=0).stop 2023-10-30T14:51:38.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.428+0000 7fb45b3ee640 1 -- 172.21.15.57:0/2634349966 shutdown_connections 2023-10-30T14:51:38.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.428+0000 7fb45b3ee640 1 --2- 172.21.15.57:0/2634349966 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fb4380664a0 0x7fb438068960 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.428+0000 7fb45b3ee640 1 --2- 172.21.15.57:0/2634349966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb45415a3b0 0x7fb454147c10 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.428+0000 7fb45b3ee640 1 --2- 172.21.15.57:0/2634349966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb454153c80 0x7fb4540b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.428+0000 7fb45b3ee640 1 --2- 172.21.15.57:0/2634349966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4540c4a10 0x7fb4541f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.428+0000 7fb45b3ee640 1 -- 172.21.15.57:0/2634349966 >> 172.21.15.57:0/2634349966 conn(0x7fb4540715c0 msgr2=0x7fb4540c9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:38.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.428+0000 7fb45b3ee640 1 -- 172.21.15.57:0/2634349966 shutdown_connections 2023-10-30T14:51:38.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.428+0000 7fb45b3ee640 1 -- 172.21.15.57:0/2634349966 wait complete. 2023-10-30T14:51:38.451 INFO:tasks.ceph:Scrubbing osd.2 2023-10-30T14:51:38.451 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-30T14:51:38.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.600+0000 7f615013e640 1 Processor -- start 2023-10-30T14:51:38.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.600+0000 7f615013e640 1 -- start start 2023-10-30T14:51:38.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.600+0000 7f615013e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f614814d430 0x7f61480724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.600+0000 7f615013e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f61480729f0 0x7f61480c88f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.600+0000 7f615013e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61480c6f70 con 0x7f61480c8e30 2023-10-30T14:51:38.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.600+0000 7f615013e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61480c2980 con 0x7f61480729f0 2023-10-30T14:51:38.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.600+0000 7f615013e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61480c2c60 con 0x7f614814d430 2023-10-30T14:51:38.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.600+0000 7f614e6db640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f61480c8e30 0x7f61480c7400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41842/0 (socket says 172.21.15.57:41842) 2023-10-30T14:51:38.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.600+0000 7f614e6db640 1 -- 172.21.15.57:0/1740496776 learned_addr learned my addr 172.21.15.57:0/1740496776 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:51:38.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.600+0000 7f614deda640 1 --2- 172.21.15.57:0/1740496776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f614814d430 0x7f61480724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.600+0000 7f614d6d9640 1 --2- 172.21.15.57:0/1740496776 >> v2:172.21.15.88:3300/0 conn(0x7f61480729f0 0x7f61480c88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.600+0000 7f614ced8640 1 -- 172.21.15.57:0/1740496776 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170647621 0 0) 0x7f61480c6f70 con 0x7f61480c8e30 2023-10-30T14:51:38.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.600+0000 7f614ced8640 1 -- 172.21.15.57:0/1740496776 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6134003530 con 0x7f61480c8e30 2023-10-30T14:51:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.600+0000 7f614deda640 1 -- 172.21.15.57:0/1740496776 >> v2:172.21.15.88:3300/0 conn(0x7f61480729f0 msgr2=0x7f61480c88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.600+0000 7f614deda640 1 --2- 172.21.15.57:0/1740496776 >> v2:172.21.15.88:3300/0 conn(0x7f61480729f0 0x7f61480c88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.600+0000 7f614deda640 1 -- 172.21.15.57:0/1740496776 >> v1:172.21.15.57:6789/0 conn(0x7f61480c8e30 legacy=0x7f61480c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.600+0000 7f614deda640 1 -- 172.21.15.57:0/1740496776 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61480c2f40 con 0x7f614814d430 2023-10-30T14:51:38.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.604+0000 7f614deda640 1 --2- 172.21.15.57:0/1740496776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f614814d430 0x7f61480724b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f61440097c0 tx=0x7f614401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d90752577b4402e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:38.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.604+0000 7f614ced8640 1 -- 172.21.15.57:0/1740496776 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61440a8970 con 0x7f614814d430 2023-10-30T14:51:38.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.604+0000 7f614ced8640 1 -- 172.21.15.57:0/1740496776 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61440a8ad0 con 0x7f614814d430 2023-10-30T14:51:38.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.604+0000 7f614ced8640 1 -- 172.21.15.57:0/1740496776 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61440a8e40 con 0x7f614814d430 2023-10-30T14:51:38.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.604+0000 7f615013e640 1 -- 172.21.15.57:0/1740496776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f614814d430 msgr2=0x7f61480724b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.604+0000 7f615013e640 1 --2- 172.21.15.57:0/1740496776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f614814d430 0x7f61480724b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f61440097c0 tx=0x7f614401ec70 comp rx=0 tx=0).stop 2023-10-30T14:51:38.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.604+0000 7f615013e640 1 -- 172.21.15.57:0/1740496776 shutdown_connections 2023-10-30T14:51:38.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.604+0000 7f615013e640 1 --2- 172.21.15.57:0/1740496776 >> v2:172.21.15.88:3300/0 conn(0x7f61480729f0 0x7f61480c88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.604+0000 7f615013e640 1 --2- 172.21.15.57:0/1740496776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f614814d430 0x7f61480724b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.604+0000 7f615013e640 1 -- 172.21.15.57:0/1740496776 >> 172.21.15.57:0/1740496776 conn(0x7f614806afa0 msgr2=0x7f614806b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:38.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.604+0000 7f615013e640 1 -- 172.21.15.57:0/1740496776 shutdown_connections 2023-10-30T14:51:38.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.604+0000 7f615013e640 1 -- 172.21.15.57:0/1740496776 wait complete. 2023-10-30T14:51:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f615013e640 1 Processor -- start 2023-10-30T14:51:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f615013e640 1 -- start start 2023-10-30T14:51:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f615013e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61480729f0 0x7f61481f2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f615013e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61480c8e30 0x7f61481ef950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f615013e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f614814d430 0x7f61480cc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f615013e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614813b5b0 con 0x7f61480c8e30 2023-10-30T14:51:38.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f615013e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61481efe90 con 0x7f614814d430 2023-10-30T14:51:38.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f615013e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614813d270 con 0x7f61480729f0 2023-10-30T14:51:38.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f614d6d9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61480c8e30 0x7f61481ef950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f614d6d9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61480c8e30 0x7f61481ef950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56706/0 (socket says 172.21.15.57:56706) 2023-10-30T14:51:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f614d6d9640 1 -- 172.21.15.57:0/4170586351 learned_addr learned my addr 172.21.15.57:0/4170586351 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f614deda640 1 --2- 172.21.15.57:0/4170586351 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61480729f0 0x7f61481f2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f614e6db640 1 --2- 172.21.15.57:0/4170586351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f614814d430 0x7f61480cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f614d6d9640 1 -- 172.21.15.57:0/4170586351 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61480729f0 msgr2=0x7f61481f2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f614d6d9640 1 --2- 172.21.15.57:0/4170586351 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61480729f0 0x7f61481f2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f614d6d9640 1 -- 172.21.15.57:0/4170586351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f614814d430 msgr2=0x7f61480cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f614d6d9640 1 --2- 172.21.15.57:0/4170586351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f614814d430 0x7f61480cc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f614d6d9640 1 -- 172.21.15.57:0/4170586351 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6144009420 con 0x7f61480c8e30 2023-10-30T14:51:38.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f614e6db640 1 --2- 172.21.15.57:0/4170586351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f614814d430 0x7f61480cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:38.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f614d6d9640 1 --2- 172.21.15.57:0/4170586351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61480c8e30 0x7f61481ef950 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f613800e840 tx=0x7f613800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:38.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f613e7fc640 1 -- 172.21.15.57:0/4170586351 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138009510 con 0x7f61480c8e30 2023-10-30T14:51:38.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f613e7fc640 1 -- 172.21.15.57:0/4170586351 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613800ee90 con 0x7f61480c8e30 2023-10-30T14:51:38.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f613e7fc640 1 -- 172.21.15.57:0/4170586351 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613800cb20 con 0x7f61480c8e30 2023-10-30T14:51:38.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f615013e640 1 -- 172.21.15.57:0/4170586351 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61481f3320 con 0x7f61480c8e30 2023-10-30T14:51:38.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.608+0000 7f615013e640 1 -- 172.21.15.57:0/4170586351 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61481f2d60 con 0x7f61480c8e30 2023-10-30T14:51:38.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.612+0000 7f615013e640 1 -- 172.21.15.57:0/4170586351 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f61480c6c00 con 0x7f61480c8e30 2023-10-30T14:51:38.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.612+0000 7f613e7fc640 1 -- 172.21.15.57:0/4170586351 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f613800cc80 con 0x7f61480c8e30 2023-10-30T14:51:38.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.612+0000 7f613e7fc640 1 --2- 172.21.15.57:0/4170586351 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f612c0664a0 0x7f612c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.612+0000 7f613e7fc640 1 -- 172.21.15.57:0/4170586351 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f61380848a0 con 0x7f61480c8e30 2023-10-30T14:51:38.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.612+0000 7f613e7fc640 1 --2- 172.21.15.57:0/4170586351 >> [v2:172.21.15.57:6824/1700292,v1:172.21.15.57:6825/1700292] conn(0x7f612c06f640 0x7f612c071ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.612+0000 7f613e7fc640 1 -- 172.21.15.57:0/4170586351 --> [v2:172.21.15.57:6824/1700292,v1:172.21.15.57:6825/1700292] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f612c072190 con 0x7f612c06f640 2023-10-30T14:51:38.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.612+0000 7f613e7fc640 1 -- 172.21.15.57:0/4170586351 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61380892a0 con 0x7f61480c8e30 2023-10-30T14:51:38.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.612+0000 7f614deda640 1 --2- 172.21.15.57:0/4170586351 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f612c0664a0 0x7f612c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.612+0000 7f614e6db640 1 --2- 172.21.15.57:0/4170586351 >> [v2:172.21.15.57:6824/1700292,v1:172.21.15.57:6825/1700292] conn(0x7f612c06f640 0x7f612c071ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.612+0000 7f614e6db640 1 --2- 172.21.15.57:0/4170586351 >> [v2:172.21.15.57:6824/1700292,v1:172.21.15.57:6825/1700292] conn(0x7f612c06f640 0x7f612c071ac0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:38.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.612+0000 7f614deda640 1 --2- 172.21.15.57:0/4170586351 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f612c0664a0 0x7f612c068960 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f61440b2000 tx=0x7f61440aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:38.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.616+0000 7f613e7fc640 1 -- 172.21.15.57:0/4170586351 <== osd.2 v2:172.21.15.57:6824/1700292 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f612c072190 con 0x7f612c06f640 2023-10-30T14:51:38.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.628+0000 7f615013e640 1 -- 172.21.15.57:0/4170586351 --> [v2:172.21.15.57:6824/1700292,v1:172.21.15.57:6825/1700292] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f61480c6920 con 0x7f612c06f640 2023-10-30T14:51:38.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.628+0000 7f613e7fc640 1 -- 172.21.15.57:0/4170586351 <== osd.2 v2:172.21.15.57:6824/1700292 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f61400070f0 con 0x7f612c06f640 2023-10-30T14:51:38.633 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-30T14:51:38.634 INFO:teuthology.orchestra.run.smithi057.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T14:51:38.634 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-30T14:51:38.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.628+0000 7f615013e640 1 -- 172.21.15.57:0/4170586351 >> [v2:172.21.15.57:6824/1700292,v1:172.21.15.57:6825/1700292] conn(0x7f612c06f640 msgr2=0x7f612c071ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.628+0000 7f615013e640 1 --2- 172.21.15.57:0/4170586351 >> [v2:172.21.15.57:6824/1700292,v1:172.21.15.57:6825/1700292] conn(0x7f612c06f640 0x7f612c071ac0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.628+0000 7f615013e640 1 -- 172.21.15.57:0/4170586351 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f612c0664a0 msgr2=0x7f612c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.628+0000 7f615013e640 1 --2- 172.21.15.57:0/4170586351 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f612c0664a0 0x7f612c068960 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f61440b2000 tx=0x7f61440aa070 comp rx=0 tx=0).stop 2023-10-30T14:51:38.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.628+0000 7f615013e640 1 -- 172.21.15.57:0/4170586351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61480c8e30 msgr2=0x7f61481ef950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.628+0000 7f615013e640 1 --2- 172.21.15.57:0/4170586351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61480c8e30 0x7f61481ef950 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f613800e840 tx=0x7f613800ec40 comp rx=0 tx=0).stop 2023-10-30T14:51:38.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.628+0000 7f614e6db640 1 -- 172.21.15.57:0/4170586351 reap_dead start 2023-10-30T14:51:38.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.628+0000 7f615013e640 1 -- 172.21.15.57:0/4170586351 shutdown_connections 2023-10-30T14:51:38.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.628+0000 7f615013e640 1 -- 172.21.15.57:0/4170586351 >> 172.21.15.57:0/4170586351 conn(0x7f614806afa0 msgr2=0x7f61480716c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:38.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.628+0000 7f615013e640 1 -- 172.21.15.57:0/4170586351 shutdown_connections 2023-10-30T14:51:38.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.628+0000 7f615013e640 1 -- 172.21.15.57:0/4170586351 wait complete. 2023-10-30T14:51:38.652 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-30T14:51:38.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.792+0000 7f8b8566b640 1 Processor -- start 2023-10-30T14:51:38.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.792+0000 7f8b8566b640 1 -- start start 2023-10-30T14:51:38.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.792+0000 7f8b8566b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8b8015d840 0x7f8b800ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.796+0000 7f8b8566b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b800c7b00 con 0x7f8b80153aa0 2023-10-30T14:51:38.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.796+0000 7f8b8566b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b800c2050 con 0x7f8b800cd420 2023-10-30T14:51:38.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.796+0000 7f8b8566b640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b80070a50 con 0x7f8b8015d840 2023-10-30T14:51:38.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.796+0000 7f8b77fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8b8015d840 0x7f8b800ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.796+0000 7f8b7effd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f8b80153aa0 0x7f8b800701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41858/0 (socket says 172.21.15.57:41858) 2023-10-30T14:51:38.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.796+0000 7f8b7effd640 1 -- 172.21.15.57:0/2350413672 learned_addr learned my addr 172.21.15.57:0/2350413672 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:51:38.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.796+0000 7f8b77fff640 1 -- 172.21.15.57:0/2350413672 >> v1:172.21.15.88:6789/0 conn(0x7f8b800cd420 legacy=0x7f8b8006b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.796+0000 7f8b77fff640 1 -- 172.21.15.57:0/2350413672 >> v1:172.21.15.57:6789/0 conn(0x7f8b80153aa0 legacy=0x7f8b800701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.796+0000 7f8b77fff640 1 -- 172.21.15.57:0/2350413672 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b800c2330 con 0x7f8b8015d840 2023-10-30T14:51:38.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.796+0000 7f8b7e7fc640 1 -- 172.21.15.57:0/2350413672 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108762381 0 0) 0x7f8b800c7b00 con 0x7f8b80153aa0 2023-10-30T14:51:38.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.796+0000 7f8b77fff640 1 --2- 172.21.15.57:0/2350413672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b8015d840 0x7f8b800ccee0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f8b680097c0 tx=0x7f8b6801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b97e89dd624c93 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:38.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.796+0000 7f8b7e7fc640 1 -- 172.21.15.57:0/2350413672 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b680a8970 con 0x7f8b8015d840 2023-10-30T14:51:38.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.796+0000 7f8b7e7fc640 1 -- 172.21.15.57:0/2350413672 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b680a8ad0 con 0x7f8b8015d840 2023-10-30T14:51:38.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.796+0000 7f8b7e7fc640 1 -- 172.21.15.57:0/2350413672 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b680a8e60 con 0x7f8b8015d840 2023-10-30T14:51:38.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b8566b640 1 -- 172.21.15.57:0/2350413672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b8015d840 msgr2=0x7f8b800ccee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b8566b640 1 --2- 172.21.15.57:0/2350413672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b8015d840 0x7f8b800ccee0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f8b680097c0 tx=0x7f8b6801ec70 comp rx=0 tx=0).stop 2023-10-30T14:51:38.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b8566b640 1 -- 172.21.15.57:0/2350413672 shutdown_connections 2023-10-30T14:51:38.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b8566b640 1 --2- 172.21.15.57:0/2350413672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b8015d840 0x7f8b800ccee0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b8566b640 1 -- 172.21.15.57:0/2350413672 >> 172.21.15.57:0/2350413672 conn(0x7f8b80071fb0 msgr2=0x7f8b800723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:38.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b8566b640 1 -- 172.21.15.57:0/2350413672 shutdown_connections 2023-10-30T14:51:38.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b8566b640 1 -- 172.21.15.57:0/2350413672 wait complete. 2023-10-30T14:51:38.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b8566b640 1 Processor -- start 2023-10-30T14:51:38.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b8566b640 1 -- start start 2023-10-30T14:51:38.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b8566b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b800cd420 0x7f8b801ef3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b8566b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b80153aa0 0x7f8b80155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b8566b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b8015d840 0x7f8b800c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b8566b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b80157b30 con 0x7f8b80153aa0 2023-10-30T14:51:38.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b8566b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b801ef8e0 con 0x7f8b8015d840 2023-10-30T14:51:38.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b8566b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b800cc510 con 0x7f8b800cd420 2023-10-30T14:51:38.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b77fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b80153aa0 0x7f8b80155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b77fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b80153aa0 0x7f8b80155df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56726/0 (socket says 172.21.15.57:56726) 2023-10-30T14:51:38.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b77fff640 1 -- 172.21.15.57:0/3899237342 learned_addr learned my addr 172.21.15.57:0/3899237342 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:38.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b7effd640 1 --2- 172.21.15.57:0/3899237342 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b800cd420 0x7f8b801ef3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b7f7fe640 1 --2- 172.21.15.57:0/3899237342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b8015d840 0x7f8b800c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b77fff640 1 -- 172.21.15.57:0/3899237342 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b800cd420 msgr2=0x7f8b801ef3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b77fff640 1 --2- 172.21.15.57:0/3899237342 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b800cd420 0x7f8b801ef3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b77fff640 1 -- 172.21.15.57:0/3899237342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b8015d840 msgr2=0x7f8b800c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b77fff640 1 --2- 172.21.15.57:0/3899237342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b8015d840 0x7f8b800c9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b77fff640 1 -- 172.21.15.57:0/3899237342 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b68009420 con 0x7f8b80153aa0 2023-10-30T14:51:38.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.800+0000 7f8b7effd640 1 --2- 172.21.15.57:0/3899237342 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b800cd420 0x7f8b801ef3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:38.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.804+0000 7f8b77fff640 1 --2- 172.21.15.57:0/3899237342 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b80153aa0 0x7f8b80155df0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f8b680097c0 tx=0x7f8b6801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:38.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.804+0000 7f8b777fe640 1 -- 172.21.15.57:0/3899237342 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b680b9630 con 0x7f8b80153aa0 2023-10-30T14:51:38.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.804+0000 7f8b777fe640 1 -- 172.21.15.57:0/3899237342 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b680b9790 con 0x7f8b80153aa0 2023-10-30T14:51:38.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.804+0000 7f8b777fe640 1 -- 172.21.15.57:0/3899237342 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b680b9b40 con 0x7f8b80153aa0 2023-10-30T14:51:38.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.804+0000 7f8b8566b640 1 -- 172.21.15.57:0/3899237342 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b800cc7d0 con 0x7f8b80153aa0 2023-10-30T14:51:38.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.804+0000 7f8b8566b640 1 -- 172.21.15.57:0/3899237342 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b800c8310 con 0x7f8b80153aa0 2023-10-30T14:51:38.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.804+0000 7f8b8566b640 1 -- 172.21.15.57:0/3899237342 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b8004e170 con 0x7f8b80153aa0 2023-10-30T14:51:38.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.804+0000 7f8b777fe640 1 -- 172.21.15.57:0/3899237342 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8b680b7040 con 0x7f8b80153aa0 2023-10-30T14:51:38.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.804+0000 7f8b777fe640 1 --2- 172.21.15.57:0/3899237342 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8b600664d0 0x7f8b60068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:38.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.804+0000 7f8b777fe640 1 -- 172.21.15.57:0/3899237342 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f8b681224f0 con 0x7f8b80153aa0 2023-10-30T14:51:38.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.804+0000 7f8b7effd640 1 --2- 172.21.15.57:0/3899237342 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8b600664d0 0x7f8b60068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:38.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.808+0000 7f8b7effd640 1 --2- 172.21.15.57:0/3899237342 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8b600664d0 0x7f8b60068990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8b70000f70 tx=0x7f8b70007c60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:38.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.808+0000 7f8b777fe640 1 -- 172.21.15.57:0/3899237342 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8b680ef2e0 con 0x7f8b80153aa0 2023-10-30T14:51:38.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.968+0000 7f8b8566b640 1 -- 172.21.15.57:0/3899237342 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b800c28f0 con 0x7f8b600664d0 2023-10-30T14:51:38.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.968+0000 7f8b777fe640 1 -- 172.21.15.57:0/3899237342 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8b70007270 con 0x7f8b600664d0 2023-10-30T14:51:38.973 INFO:teuthology.orchestra.run.smithi057.stderr:instructed osd(s) 2 to deep-scrub 2023-10-30T14:51:38.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.972+0000 7f8b8566b640 1 -- 172.21.15.57:0/3899237342 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8b600664d0 msgr2=0x7f8b60068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.972+0000 7f8b8566b640 1 --2- 172.21.15.57:0/3899237342 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8b600664d0 0x7f8b60068990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8b70000f70 tx=0x7f8b70007c60 comp rx=0 tx=0).stop 2023-10-30T14:51:38.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.972+0000 7f8b8566b640 1 -- 172.21.15.57:0/3899237342 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b80153aa0 msgr2=0x7f8b80155df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:38.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.972+0000 7f8b8566b640 1 --2- 172.21.15.57:0/3899237342 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b80153aa0 0x7f8b80155df0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f8b680097c0 tx=0x7f8b6801ba10 comp rx=0 tx=0).stop 2023-10-30T14:51:38.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.972+0000 7f8b8566b640 1 -- 172.21.15.57:0/3899237342 shutdown_connections 2023-10-30T14:51:38.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.972+0000 7f8b8566b640 1 --2- 172.21.15.57:0/3899237342 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f8b600664d0 0x7f8b60068990 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.972+0000 7f8b8566b640 1 --2- 172.21.15.57:0/3899237342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b8015d840 0x7f8b800c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.972+0000 7f8b8566b640 1 --2- 172.21.15.57:0/3899237342 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b80153aa0 0x7f8b80155df0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.972+0000 7f8b8566b640 1 --2- 172.21.15.57:0/3899237342 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b800cd420 0x7f8b801ef3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:38.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.972+0000 7f8b8566b640 1 -- 172.21.15.57:0/3899237342 >> 172.21.15.57:0/3899237342 conn(0x7f8b80071fb0 msgr2=0x7f8b80152410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:38.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.972+0000 7f8b8566b640 1 -- 172.21.15.57:0/3899237342 shutdown_connections 2023-10-30T14:51:38.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:38.972+0000 7f8b8566b640 1 -- 172.21.15.57:0/3899237342 wait complete. 2023-10-30T14:51:38.995 INFO:tasks.ceph:Scrubbing osd.3 2023-10-30T14:51:38.996 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-30T14:51:39.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.140+0000 7f0e20765640 1 Processor -- start 2023-10-30T14:51:39.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.140+0000 7f0e20765640 1 -- start start 2023-10-30T14:51:39.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.140+0000 7f0e20765640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e181599e0 0x7f0e18153880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.140+0000 7f0e20765640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e1806af00 con 0x7f0e180c4720 2023-10-30T14:51:39.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.140+0000 7f0e20765640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e1806a660 con 0x7f0e180c4340 2023-10-30T14:51:39.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.140+0000 7f0e20765640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e180c4cd0 con 0x7f0e181599e0 2023-10-30T14:51:39.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.140+0000 7f0e1dd00640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f0e180c4720 0x7f0e180ca470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41862/0 (socket says 172.21.15.57:41862) 2023-10-30T14:51:39.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.140+0000 7f0e1dd00640 1 -- 172.21.15.57:0/2360306003 learned_addr learned my addr 172.21.15.57:0/2360306003 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:51:39.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.140+0000 7f0e1ed02640 1 --2- 172.21.15.57:0/2360306003 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e181599e0 0x7f0e18153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.140+0000 7f0e1d4ff640 1 -- 172.21.15.57:0/2360306003 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2494660142 0 0) 0x7f0e1806af00 con 0x7f0e180c4720 2023-10-30T14:51:39.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.140+0000 7f0e1d4ff640 1 -- 172.21.15.57:0/2360306003 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e04003530 con 0x7f0e180c4720 2023-10-30T14:51:39.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.140+0000 7f0e1d4ff640 1 -- 172.21.15.57:0/2360306003 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442021189 0 0) 0x7f0e1806a660 con 0x7f0e180c4340 2023-10-30T14:51:39.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.140+0000 7f0e1ed02640 1 -- 172.21.15.57:0/2360306003 >> v1:172.21.15.88:6789/0 conn(0x7f0e180c4340 legacy=0x7f0e18069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.140+0000 7f0e1ed02640 1 -- 172.21.15.57:0/2360306003 >> v1:172.21.15.57:6789/0 conn(0x7f0e180c4720 legacy=0x7f0e180ca470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.140+0000 7f0e1ed02640 1 -- 172.21.15.57:0/2360306003 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1806a940 con 0x7f0e181599e0 2023-10-30T14:51:39.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.144+0000 7f0e1ed02640 1 --2- 172.21.15.57:0/2360306003 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e181599e0 0x7f0e18153880 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0e10009dc0 tx=0x7f0e1001c6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c509c40ef43d314 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:39.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.144+0000 7f0e1d4ff640 1 -- 172.21.15.57:0/2360306003 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e080016a0 con 0x7f0e181599e0 2023-10-30T14:51:39.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.144+0000 7f0e1d4ff640 1 -- 172.21.15.57:0/2360306003 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e100a6490 con 0x7f0e181599e0 2023-10-30T14:51:39.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.144+0000 7f0e1d4ff640 1 -- 172.21.15.57:0/2360306003 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e100a6800 con 0x7f0e181599e0 2023-10-30T14:51:39.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.144+0000 7f0e20765640 1 -- 172.21.15.57:0/2360306003 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e181599e0 msgr2=0x7f0e18153880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.144+0000 7f0e20765640 1 --2- 172.21.15.57:0/2360306003 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e181599e0 0x7f0e18153880 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0e10009dc0 tx=0x7f0e1001c6f0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.144+0000 7f0e20765640 1 -- 172.21.15.57:0/2360306003 shutdown_connections 2023-10-30T14:51:39.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.144+0000 7f0e20765640 1 --2- 172.21.15.57:0/2360306003 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e181599e0 0x7f0e18153880 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.144+0000 7f0e20765640 1 -- 172.21.15.57:0/2360306003 >> 172.21.15.57:0/2360306003 conn(0x7f0e180cf1c0 msgr2=0x7f0e180cf5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:39.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.144+0000 7f0e20765640 1 -- 172.21.15.57:0/2360306003 shutdown_connections 2023-10-30T14:51:39.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.144+0000 7f0e20765640 1 -- 172.21.15.57:0/2360306003 wait complete. 2023-10-30T14:51:39.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e20765640 1 Processor -- start 2023-10-30T14:51:39.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e20765640 1 -- start start 2023-10-30T14:51:39.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e20765640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e180c4340 0x7f0e18155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e20765640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e180c4720 0x7f0e1814b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e20765640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e181599e0 0x7f0e181f2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e20765640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e18155090 con 0x7f0e181599e0 2023-10-30T14:51:39.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e20765640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1814a650 con 0x7f0e180c4720 2023-10-30T14:51:39.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e20765640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e181542e0 con 0x7f0e180c4340 2023-10-30T14:51:39.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e1ed02640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e181599e0 0x7f0e181f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e1ed02640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e181599e0 0x7f0e181f2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56728/0 (socket says 172.21.15.57:56728) 2023-10-30T14:51:39.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e1ed02640 1 -- 172.21.15.57:0/1259389233 learned_addr learned my addr 172.21.15.57:0/1259389233 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:39.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e1e501640 1 --2- 172.21.15.57:0/1259389233 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e180c4340 0x7f0e18155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e1dd00640 1 --2- 172.21.15.57:0/1259389233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e180c4720 0x7f0e1814b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e1ed02640 1 -- 172.21.15.57:0/1259389233 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e180c4340 msgr2=0x7f0e18155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e1ed02640 1 --2- 172.21.15.57:0/1259389233 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e180c4340 0x7f0e18155f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e1ed02640 1 -- 172.21.15.57:0/1259389233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e180c4720 msgr2=0x7f0e1814b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e1ed02640 1 --2- 172.21.15.57:0/1259389233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e180c4720 0x7f0e1814b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e1ed02640 1 -- 172.21.15.57:0/1259389233 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e10009a70 con 0x7f0e181599e0 2023-10-30T14:51:39.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e1ed02640 1 --2- 172.21.15.57:0/1259389233 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e181599e0 0x7f0e181f2640 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f0e1001a320 tx=0x7f0e10019490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:39.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e0effd640 1 -- 172.21.15.57:0/1259389233 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e100b7070 con 0x7f0e181599e0 2023-10-30T14:51:39.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e0effd640 1 -- 172.21.15.57:0/1259389233 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e10004310 con 0x7f0e181599e0 2023-10-30T14:51:39.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e0effd640 1 -- 172.21.15.57:0/1259389233 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e100046a0 con 0x7f0e181599e0 2023-10-30T14:51:39.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e20765640 1 -- 172.21.15.57:0/1259389233 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e181545c0 con 0x7f0e181599e0 2023-10-30T14:51:39.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.148+0000 7f0e20765640 1 -- 172.21.15.57:0/1259389233 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e1814abd0 con 0x7f0e181599e0 2023-10-30T14:51:39.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.152+0000 7f0e20765640 1 -- 172.21.15.57:0/1259389233 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0e180c9f50 con 0x7f0e181599e0 2023-10-30T14:51:39.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.152+0000 7f0e0effd640 1 -- 172.21.15.57:0/1259389233 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0e100a6600 con 0x7f0e181599e0 2023-10-30T14:51:39.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.152+0000 7f0e0effd640 1 --2- 172.21.15.57:0/1259389233 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0dfc0664d0 0x7f0dfc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.152+0000 7f0e0effd640 1 -- 172.21.15.57:0/1259389233 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f0e1011ef80 con 0x7f0e181599e0 2023-10-30T14:51:39.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.152+0000 7f0e0effd640 1 --2- 172.21.15.57:0/1259389233 >> [v2:172.21.15.57:6816/2893420631,v1:172.21.15.57:6817/2893420631] conn(0x7f0dfc06f670 0x7f0dfc071af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.152+0000 7f0e0effd640 1 -- 172.21.15.57:0/1259389233 --> [v2:172.21.15.57:6816/2893420631,v1:172.21.15.57:6817/2893420631] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0dfc0721c0 con 0x7f0dfc06f670 2023-10-30T14:51:39.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.152+0000 7f0e0effd640 1 -- 172.21.15.57:0/1259389233 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e1011f2f0 con 0x7f0e181599e0 2023-10-30T14:51:39.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.152+0000 7f0e1e501640 1 --2- 172.21.15.57:0/1259389233 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0dfc0664d0 0x7f0dfc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.152+0000 7f0e1dd00640 1 --2- 172.21.15.57:0/1259389233 >> [v2:172.21.15.57:6816/2893420631,v1:172.21.15.57:6817/2893420631] conn(0x7f0dfc06f670 0x7f0dfc071af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.152+0000 7f0e1dd00640 1 --2- 172.21.15.57:0/1259389233 >> [v2:172.21.15.57:6816/2893420631,v1:172.21.15.57:6817/2893420631] conn(0x7f0dfc06f670 0x7f0dfc071af0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:39.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.152+0000 7f0e1e501640 1 --2- 172.21.15.57:0/1259389233 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0dfc0664d0 0x7f0dfc068990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f0e14007fd0 tx=0x7f0e14009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:39.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.156+0000 7f0e0effd640 1 -- 172.21.15.57:0/1259389233 <== osd.3 v2:172.21.15.57:6816/2893420631 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f0dfc0721c0 con 0x7f0dfc06f670 2023-10-30T14:51:39.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.168+0000 7f0e20765640 1 -- 172.21.15.57:0/1259389233 --> [v2:172.21.15.57:6816/2893420631,v1:172.21.15.57:6817/2893420631] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0e180cca80 con 0x7f0dfc06f670 2023-10-30T14:51:39.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.168+0000 7f0e0effd640 1 -- 172.21.15.57:0/1259389233 <== osd.3 v2:172.21.15.57:6816/2893420631 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f0e180cca80 con 0x7f0dfc06f670 2023-10-30T14:51:39.172 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-30T14:51:39.172 INFO:teuthology.orchestra.run.smithi057.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T14:51:39.173 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-30T14:51:39.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.168+0000 7f0e20765640 1 -- 172.21.15.57:0/1259389233 >> [v2:172.21.15.57:6816/2893420631,v1:172.21.15.57:6817/2893420631] conn(0x7f0dfc06f670 msgr2=0x7f0dfc071af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.168+0000 7f0e20765640 1 --2- 172.21.15.57:0/1259389233 >> [v2:172.21.15.57:6816/2893420631,v1:172.21.15.57:6817/2893420631] conn(0x7f0dfc06f670 0x7f0dfc071af0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.168+0000 7f0e20765640 1 -- 172.21.15.57:0/1259389233 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0dfc0664d0 msgr2=0x7f0dfc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.168+0000 7f0e20765640 1 --2- 172.21.15.57:0/1259389233 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f0dfc0664d0 0x7f0dfc068990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f0e14007fd0 tx=0x7f0e14009040 comp rx=0 tx=0).stop 2023-10-30T14:51:39.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.168+0000 7f0e20765640 1 -- 172.21.15.57:0/1259389233 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e181599e0 msgr2=0x7f0e181f2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.168+0000 7f0e20765640 1 --2- 172.21.15.57:0/1259389233 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e181599e0 0x7f0e181f2640 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f0e1001a320 tx=0x7f0e10019490 comp rx=0 tx=0).stop 2023-10-30T14:51:39.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.168+0000 7f0e1ed02640 1 -- 172.21.15.57:0/1259389233 reap_dead start 2023-10-30T14:51:39.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.168+0000 7f0e20765640 1 -- 172.21.15.57:0/1259389233 shutdown_connections 2023-10-30T14:51:39.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.168+0000 7f0e20765640 1 -- 172.21.15.57:0/1259389233 >> 172.21.15.57:0/1259389233 conn(0x7f0e180cf1c0 msgr2=0x7f0e18158740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:39.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.168+0000 7f0e20765640 1 -- 172.21.15.57:0/1259389233 shutdown_connections 2023-10-30T14:51:39.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.168+0000 7f0e20765640 1 -- 172.21.15.57:0/1259389233 wait complete. 2023-10-30T14:51:39.190 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-30T14:51:39.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.332+0000 7f6aee9a3640 1 Processor -- start 2023-10-30T14:51:39.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.332+0000 7f6aee9a3640 1 -- start start 2023-10-30T14:51:39.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.332+0000 7f6aee9a3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ae80c8b00 0x7f6ae80c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.332+0000 7f6aee9a3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6ae815cdf0 0x7f6ae80c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.332+0000 7f6aee9a3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae80b27f0 con 0x7f6ae80b22a0 2023-10-30T14:51:39.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.332+0000 7f6aee9a3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ae815d370 con 0x7f6ae815cdf0 2023-10-30T14:51:39.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.332+0000 7f6aee9a3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae815d630 con 0x7f6ae80c8b00 2023-10-30T14:51:39.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.332+0000 7f6aed9a1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ae80c8b00 0x7f6ae80c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.332+0000 7f6aed1a0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6ae815cdf0 0x7f6ae80c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.332+0000 7f6aed1a0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6ae815cdf0 0x7f6ae80c9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:57470/0 (socket says 172.21.15.57:57470) 2023-10-30T14:51:39.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.332+0000 7f6aed1a0640 1 -- 172.21.15.57:0/2596289481 learned_addr learned my addr 172.21.15.57:0/2596289481 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:39.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.332+0000 7f6aed9a1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ae80c8b00 0x7f6ae80c8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:44352/0 (socket says 172.21.15.57:44352) 2023-10-30T14:51:39.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.332+0000 7f6aed1a0640 1 -- 172.21.15.57:0/2596289481 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ae80c8b00 msgr2=0x7f6ae80c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:39.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.332+0000 7f6aed1a0640 1 --2- 172.21.15.57:0/2596289481 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ae80c8b00 0x7f6ae80c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.332+0000 7f6aed1a0640 1 -- 172.21.15.57:0/2596289481 >> v1:172.21.15.88:6789/0 conn(0x7f6ae80b22a0 legacy=0x7f6ae80b3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.332+0000 7f6aed1a0640 1 -- 172.21.15.57:0/2596289481 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae815d910 con 0x7f6ae815cdf0 2023-10-30T14:51:39.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.332+0000 7f6aed9a1640 1 --2- 172.21.15.57:0/2596289481 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ae80c8b00 0x7f6ae80c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:51:39.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.336+0000 7f6aed1a0640 1 --2- 172.21.15.57:0/2596289481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ae815cdf0 0x7f6ae80c9e30 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009870 tx=0x7f6ad801ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=769a08eb1b1896d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:39.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.340+0000 7f6aec99f640 1 -- 172.21.15.57:0/2596289481 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad80a8a30 con 0x7f6ae815cdf0 2023-10-30T14:51:39.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.340+0000 7f6aec99f640 1 -- 172.21.15.57:0/2596289481 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad80a8b90 con 0x7f6ae815cdf0 2023-10-30T14:51:39.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.340+0000 7f6aec99f640 1 -- 172.21.15.57:0/2596289481 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad80b0440 con 0x7f6ae815cdf0 2023-10-30T14:51:39.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.340+0000 7f6aee9a3640 1 -- 172.21.15.57:0/2596289481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ae815cdf0 msgr2=0x7f6ae80c9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.340+0000 7f6aee9a3640 1 --2- 172.21.15.57:0/2596289481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ae815cdf0 0x7f6ae80c9e30 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009870 tx=0x7f6ad801ed30 comp rx=0 tx=0).stop 2023-10-30T14:51:39.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.340+0000 7f6aee9a3640 1 -- 172.21.15.57:0/2596289481 shutdown_connections 2023-10-30T14:51:39.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.340+0000 7f6aee9a3640 1 --2- 172.21.15.57:0/2596289481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ae815cdf0 0x7f6ae80c9e30 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.340+0000 7f6aee9a3640 1 --2- 172.21.15.57:0/2596289481 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ae80c8b00 0x7f6ae80c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.340+0000 7f6aee9a3640 1 -- 172.21.15.57:0/2596289481 >> 172.21.15.57:0/2596289481 conn(0x7f6ae806bed0 msgr2=0x7f6ae806c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:39.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.340+0000 7f6aee9a3640 1 -- 172.21.15.57:0/2596289481 shutdown_connections 2023-10-30T14:51:39.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.340+0000 7f6aee9a3640 1 -- 172.21.15.57:0/2596289481 wait complete. 2023-10-30T14:51:39.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aee9a3640 1 Processor -- start 2023-10-30T14:51:39.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aee9a3640 1 -- start start 2023-10-30T14:51:39.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aee9a3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ae80b22a0 0x7f6ae81f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aee9a3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ae80c8b00 0x7f6ae8155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aee9a3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ae815cdf0 0x7f6ae8154510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aee9a3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae8150ef0 con 0x7f6ae815cdf0 2023-10-30T14:51:39.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aee9a3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae81f2e10 con 0x7f6ae80c8b00 2023-10-30T14:51:39.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aee9a3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae8150c10 con 0x7f6ae80b22a0 2023-10-30T14:51:39.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aee1a2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ae815cdf0 0x7f6ae8154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aee1a2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ae815cdf0 0x7f6ae8154510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56734/0 (socket says 172.21.15.57:56734) 2023-10-30T14:51:39.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aee1a2640 1 -- 172.21.15.57:0/1150545795 learned_addr learned my addr 172.21.15.57:0/1150545795 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:39.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aed9a1640 1 --2- 172.21.15.57:0/1150545795 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ae80b22a0 0x7f6ae81f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aed1a0640 1 --2- 172.21.15.57:0/1150545795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ae80c8b00 0x7f6ae8155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aee1a2640 1 -- 172.21.15.57:0/1150545795 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ae80b22a0 msgr2=0x7f6ae81f28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aee1a2640 1 --2- 172.21.15.57:0/1150545795 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ae80b22a0 0x7f6ae81f28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aee1a2640 1 -- 172.21.15.57:0/1150545795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ae80c8b00 msgr2=0x7f6ae8155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aee1a2640 1 --2- 172.21.15.57:0/1150545795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ae80c8b00 0x7f6ae8155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aee1a2640 1 -- 172.21.15.57:0/1150545795 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae4009420 con 0x7f6ae815cdf0 2023-10-30T14:51:39.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aed1a0640 1 --2- 172.21.15.57:0/1150545795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ae80c8b00 0x7f6ae8155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:39.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aee1a2640 1 --2- 172.21.15.57:0/1150545795 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ae815cdf0 0x7f6ae8154510 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000c730 tx=0x7f6ae000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:39.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6ade7fc640 1 -- 172.21.15.57:0/1150545795 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0003b80 con 0x7f6ae815cdf0 2023-10-30T14:51:39.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6ade7fc640 1 -- 172.21.15.57:0/1150545795 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae0003ce0 con 0x7f6ae815cdf0 2023-10-30T14:51:39.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6ade7fc640 1 -- 172.21.15.57:0/1150545795 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae000aa30 con 0x7f6ae815cdf0 2023-10-30T14:51:39.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.344+0000 7f6aee9a3640 1 -- 172.21.15.57:0/1150545795 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad80094d0 con 0x7f6ae815cdf0 2023-10-30T14:51:39.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.348+0000 7f6aee9a3640 1 -- 172.21.15.57:0/1150545795 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae8155150 con 0x7f6ae815cdf0 2023-10-30T14:51:39.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.348+0000 7f6aee9a3640 1 -- 172.21.15.57:0/1150545795 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae804e170 con 0x7f6ae815cdf0 2023-10-30T14:51:39.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.348+0000 7f6ade7fc640 1 -- 172.21.15.57:0/1150545795 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6ae000ab90 con 0x7f6ae815cdf0 2023-10-30T14:51:39.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.348+0000 7f6ade7fc640 1 --2- 172.21.15.57:0/1150545795 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6acc0664d0 0x7f6acc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.348+0000 7f6ade7fc640 1 -- 172.21.15.57:0/1150545795 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f6ae0084280 con 0x7f6ae815cdf0 2023-10-30T14:51:39.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.348+0000 7f6aed9a1640 1 --2- 172.21.15.57:0/1150545795 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6acc0664d0 0x7f6acc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.352+0000 7f6aed9a1640 1 --2- 172.21.15.57:0/1150545795 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6acc0664d0 0x7f6acc068990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4009cf0 tx=0x7f6ae4009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:39.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.352+0000 7f6ade7fc640 1 -- 172.21.15.57:0/1150545795 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6ae0050fe0 con 0x7f6ae815cdf0 2023-10-30T14:51:39.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.520+0000 7f6aee9a3640 1 -- 172.21.15.57:0/1150545795 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ae80b2a40 con 0x7f6acc0664d0 2023-10-30T14:51:39.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.520+0000 7f6ade7fc640 1 -- 172.21.15.57:0/1150545795 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6ae4009940 con 0x7f6acc0664d0 2023-10-30T14:51:39.524 INFO:teuthology.orchestra.run.smithi057.stderr:instructed osd(s) 3 to deep-scrub 2023-10-30T14:51:39.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.520+0000 7f6aee9a3640 1 -- 172.21.15.57:0/1150545795 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6acc0664d0 msgr2=0x7f6acc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.520+0000 7f6aee9a3640 1 --2- 172.21.15.57:0/1150545795 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6acc0664d0 0x7f6acc068990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4009cf0 tx=0x7f6ae4009190 comp rx=0 tx=0).stop 2023-10-30T14:51:39.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.520+0000 7f6aee9a3640 1 -- 172.21.15.57:0/1150545795 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ae815cdf0 msgr2=0x7f6ae8154510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.520+0000 7f6aee9a3640 1 --2- 172.21.15.57:0/1150545795 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ae815cdf0 0x7f6ae8154510 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000c730 tx=0x7f6ae000cbf0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.520+0000 7f6aee9a3640 1 -- 172.21.15.57:0/1150545795 shutdown_connections 2023-10-30T14:51:39.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.520+0000 7f6aee9a3640 1 --2- 172.21.15.57:0/1150545795 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6acc0664d0 0x7f6acc068990 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.520+0000 7f6aee9a3640 1 --2- 172.21.15.57:0/1150545795 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ae815cdf0 0x7f6ae8154510 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.520+0000 7f6aee9a3640 1 --2- 172.21.15.57:0/1150545795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ae80c8b00 0x7f6ae8155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.520+0000 7f6aee9a3640 1 --2- 172.21.15.57:0/1150545795 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ae80b22a0 0x7f6ae81f28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.520+0000 7f6aee9a3640 1 -- 172.21.15.57:0/1150545795 >> 172.21.15.57:0/1150545795 conn(0x7f6ae806bed0 msgr2=0x7f6ae815b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:39.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.520+0000 7f6aee9a3640 1 -- 172.21.15.57:0/1150545795 shutdown_connections 2023-10-30T14:51:39.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.520+0000 7f6aee9a3640 1 -- 172.21.15.57:0/1150545795 wait complete. 2023-10-30T14:51:39.545 INFO:tasks.ceph:Scrubbing osd.4 2023-10-30T14:51:39.546 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-30T14:51:39.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.688+0000 7f6bf49b5640 1 Processor -- start 2023-10-30T14:51:39.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.688+0000 7f6bf49b5640 1 -- start start 2023-10-30T14:51:39.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bf49b5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6bf0153920 0x7f6bf006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bf49b5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf00ca7c0 con 0x7f6bf006af10 2023-10-30T14:51:39.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bf49b5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf00caaa0 con 0x7f6bf00c9910 2023-10-30T14:51:39.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bf49b5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bf00cad80 con 0x7f6bf0153920 2023-10-30T14:51:39.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6beffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f6bf006af10 0x7f6bf00cb460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41876/0 (socket says 172.21.15.57:41876) 2023-10-30T14:51:39.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6beffff640 1 -- 172.21.15.57:0/1158007554 learned_addr learned my addr 172.21.15.57:0/1158007554 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:51:39.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bef7fe640 1 --2- 172.21.15.57:0/1158007554 >> v2:172.21.15.88:3300/0 conn(0x7f6bf0153920 0x7f6bf006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bee7fc640 1 -- 172.21.15.57:0/1158007554 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393493515 0 0) 0x7f6bf00ca7c0 con 0x7f6bf006af10 2023-10-30T14:51:39.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bee7fc640 1 -- 172.21.15.57:0/1158007554 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bd8003530 con 0x7f6bf006af10 2023-10-30T14:51:39.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bee7fc640 1 -- 172.21.15.57:0/1158007554 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1964286200 0 0) 0x7f6bd8003530 con 0x7f6bf006af10 2023-10-30T14:51:39.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bee7fc640 1 -- 172.21.15.57:0/1158007554 >> v2:172.21.15.88:3300/0 conn(0x7f6bf0153920 msgr2=0x7f6bf006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:39.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bee7fc640 1 --2- 172.21.15.57:0/1158007554 >> v2:172.21.15.88:3300/0 conn(0x7f6bf0153920 0x7f6bf006c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bee7fc640 1 -- 172.21.15.57:0/1158007554 >> v1:172.21.15.88:6789/0 conn(0x7f6bf00c9910 legacy=0x7f6bf006a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bee7fc640 1 -- 172.21.15.57:0/1158007554 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf015a2c0 con 0x7f6bf006af10 2023-10-30T14:51:39.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bef7fe640 1 --2- 172.21.15.57:0/1158007554 >> v2:172.21.15.88:3300/0 conn(0x7f6bf0153920 0x7f6bf006c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:51:39.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bee7fc640 1 -- 172.21.15.57:0/1158007554 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f6be8001a00 con 0x7f6bf006af10 2023-10-30T14:51:39.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bee7fc640 1 -- 172.21.15.57:0/1158007554 >> v1:172.21.15.57:6789/0 conn(0x7f6bf006af10 legacy=0x7f6bf00cb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bee7fc640 1 --2- 172.21.15.57:0/1158007554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bd80039d0 0x7f6bd8003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bee7fc640 1 --2- 172.21.15.57:0/1158007554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bd80047d0 0x7f6bd8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bee7fc640 1 --2- 172.21.15.57:0/1158007554 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bd8005190 0x7f6bd800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bee7fc640 1 -- 172.21.15.57:0/1158007554 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf00cad80 con 0x7f6bd80039d0 2023-10-30T14:51:39.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bee7fc640 1 -- 172.21.15.57:0/1158007554 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd800c090 con 0x7f6bd80047d0 2023-10-30T14:51:39.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bee7fc640 1 -- 172.21.15.57:0/1158007554 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd800c250 con 0x7f6bd8005190 2023-10-30T14:51:39.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bef7fe640 1 --2- 172.21.15.57:0/1158007554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bd80039d0 0x7f6bd8003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6beeffd640 1 --2- 172.21.15.57:0/1158007554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bd80047d0 0x7f6bd8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6beffff640 1 --2- 172.21.15.57:0/1158007554 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bd8005190 0x7f6bd800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bef7fe640 1 -- 172.21.15.57:0/1158007554 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bd8005190 msgr2=0x7f6bd800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:39.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bef7fe640 1 --2- 172.21.15.57:0/1158007554 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bd8005190 0x7f6bd800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bef7fe640 1 -- 172.21.15.57:0/1158007554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bd80047d0 msgr2=0x7f6bd8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:39.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bef7fe640 1 --2- 172.21.15.57:0/1158007554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bd80047d0 0x7f6bd8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6bef7fe640 1 -- 172.21.15.57:0/1158007554 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd8003530 con 0x7f6bd80039d0 2023-10-30T14:51:39.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6beffff640 1 --2- 172.21.15.57:0/1158007554 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bd8005190 0x7f6bd800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:39.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.692+0000 7f6beffff640 1 -- 172.21.15.57:0/1158007554 reap_dead start 2023-10-30T14:51:39.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.696+0000 7f6bef7fe640 1 --2- 172.21.15.57:0/1158007554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bd80039d0 0x7f6bd8003df0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f6be0009cb0 tx=0x7f6be001cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40c8d3acfe60000f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:39.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.696+0000 7f6bee7fc640 1 -- 172.21.15.57:0/1158007554 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be00a5770 con 0x7f6bd80039d0 2023-10-30T14:51:39.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.696+0000 7f6bee7fc640 1 -- 172.21.15.57:0/1158007554 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be00a58d0 con 0x7f6bd80039d0 2023-10-30T14:51:39.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.696+0000 7f6bee7fc640 1 -- 172.21.15.57:0/1158007554 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be00a5cc0 con 0x7f6bd80039d0 2023-10-30T14:51:39.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.696+0000 7f6bf49b5640 1 -- 172.21.15.57:0/1158007554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bd80039d0 msgr2=0x7f6bd8003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.696+0000 7f6bf49b5640 1 --2- 172.21.15.57:0/1158007554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bd80039d0 0x7f6bd8003df0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f6be0009cb0 tx=0x7f6be001cbc0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.696+0000 7f6bf49b5640 1 -- 172.21.15.57:0/1158007554 shutdown_connections 2023-10-30T14:51:39.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.696+0000 7f6bf49b5640 1 --2- 172.21.15.57:0/1158007554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bd80039d0 0x7f6bd8003df0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.696+0000 7f6bf49b5640 1 -- 172.21.15.57:0/1158007554 >> 172.21.15.57:0/1158007554 conn(0x7f6bf00cf6d0 msgr2=0x7f6bf006be30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:39.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.696+0000 7f6bf49b5640 1 -- 172.21.15.57:0/1158007554 shutdown_connections 2023-10-30T14:51:39.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.696+0000 7f6bf49b5640 1 -- 172.21.15.57:0/1158007554 wait complete. 2023-10-30T14:51:39.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6bf49b5640 1 Processor -- start 2023-10-30T14:51:39.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6bf49b5640 1 -- start start 2023-10-30T14:51:39.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6bf49b5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bd80039d0 0x7f6bf00ce140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6bf49b5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bf01e6790 0x7f6bf00b24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6bf49b5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bf015ad40 0x7f6bf015b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6bf49b5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf015ced0 con 0x7f6bf015ad40 2023-10-30T14:51:39.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6bf49b5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf01f57e0 con 0x7f6bf01e6790 2023-10-30T14:51:39.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6bf49b5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf015b730 con 0x7f6bd80039d0 2023-10-30T14:51:39.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6beffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bf015ad40 0x7f6bf015b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6beeffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bf01e6790 0x7f6bf00b24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6beffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bf015ad40 0x7f6bf015b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56750/0 (socket says 172.21.15.57:56750) 2023-10-30T14:51:39.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6beffff640 1 -- 172.21.15.57:0/2933415055 learned_addr learned my addr 172.21.15.57:0/2933415055 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:39.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6bef7fe640 1 --2- 172.21.15.57:0/2933415055 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bd80039d0 0x7f6bf00ce140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6beffff640 1 -- 172.21.15.57:0/2933415055 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bd80039d0 msgr2=0x7f6bf00ce140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6beffff640 1 --2- 172.21.15.57:0/2933415055 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6bd80039d0 0x7f6bf00ce140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6beffff640 1 -- 172.21.15.57:0/2933415055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bf01e6790 msgr2=0x7f6bf00b24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6beffff640 1 --2- 172.21.15.57:0/2933415055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bf01e6790 0x7f6bf00b24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6beffff640 1 -- 172.21.15.57:0/2933415055 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf015a860 con 0x7f6bf015ad40 2023-10-30T14:51:39.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6beffff640 1 --2- 172.21.15.57:0/2933415055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bf015ad40 0x7f6bf015b1f0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f6bf00b1e30 tx=0x7f6be800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:39.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.700+0000 7f6bd7fff640 1 -- 172.21.15.57:0/2933415055 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be800eb30 con 0x7f6bf015ad40 2023-10-30T14:51:39.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.704+0000 7f6bd7fff640 1 -- 172.21.15.57:0/2933415055 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be800ec90 con 0x7f6bf015ad40 2023-10-30T14:51:39.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.704+0000 7f6bf49b5640 1 -- 172.21.15.57:0/2933415055 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf01f5a30 con 0x7f6bf015ad40 2023-10-30T14:51:39.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.704+0000 7f6bd7fff640 1 -- 172.21.15.57:0/2933415055 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8014480 con 0x7f6bf015ad40 2023-10-30T14:51:39.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.704+0000 7f6bf49b5640 1 -- 172.21.15.57:0/2933415055 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf01f5ff0 con 0x7f6bf015ad40 2023-10-30T14:51:39.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.704+0000 7f6bf49b5640 1 -- 172.21.15.57:0/2933415055 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6bf0153d00 con 0x7f6bf015ad40 2023-10-30T14:51:39.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.704+0000 7f6bd7fff640 1 -- 172.21.15.57:0/2933415055 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6be801d3b0 con 0x7f6bf015ad40 2023-10-30T14:51:39.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.704+0000 7f6bd7fff640 1 --2- 172.21.15.57:0/2933415055 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6bdc0664d0 0x7f6bdc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.704+0000 7f6bd7fff640 1 -- 172.21.15.57:0/2933415055 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f6be80843d0 con 0x7f6bf015ad40 2023-10-30T14:51:39.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.704+0000 7f6bd7fff640 1 --2- 172.21.15.57:0/2933415055 >> [v2:172.21.15.88:6805/2784816485,v1:172.21.15.88:6807/2784816485] conn(0x7f6bdc06f670 0x7f6bdc071af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.704+0000 7f6bd7fff640 1 -- 172.21.15.57:0/2933415055 --> [v2:172.21.15.88:6805/2784816485,v1:172.21.15.88:6807/2784816485] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6bdc0721c0 con 0x7f6bdc06f670 2023-10-30T14:51:39.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.704+0000 7f6bd7fff640 1 -- 172.21.15.57:0/2933415055 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6be80869b0 con 0x7f6bf015ad40 2023-10-30T14:51:39.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.704+0000 7f6bef7fe640 1 --2- 172.21.15.57:0/2933415055 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6bdc0664d0 0x7f6bdc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.708+0000 7f6beeffd640 1 --2- 172.21.15.57:0/2933415055 >> [v2:172.21.15.88:6805/2784816485,v1:172.21.15.88:6807/2784816485] conn(0x7f6bdc06f670 0x7f6bdc071af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.708+0000 7f6bef7fe640 1 --2- 172.21.15.57:0/2933415055 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6bdc0664d0 0x7f6bdc068990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f6be00af000 tx=0x7f6be00046d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:39.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.708+0000 7f6beeffd640 1 --2- 172.21.15.57:0/2933415055 >> [v2:172.21.15.88:6805/2784816485,v1:172.21.15.88:6807/2784816485] conn(0x7f6bdc06f670 0x7f6bdc071af0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:39.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.708+0000 7f6bd7fff640 1 -- 172.21.15.57:0/2933415055 <== osd.4 v2:172.21.15.88:6805/2784816485 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f6bdc0721c0 con 0x7f6bdc06f670 2023-10-30T14:51:39.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.720+0000 7f6bf49b5640 1 -- 172.21.15.57:0/2933415055 --> [v2:172.21.15.88:6805/2784816485,v1:172.21.15.88:6807/2784816485] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6bf015aa20 con 0x7f6bdc06f670 2023-10-30T14:51:39.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.724+0000 7f6bd7fff640 1 -- 172.21.15.57:0/2933415055 <== osd.4 v2:172.21.15.88:6805/2784816485 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f6bf015aa20 con 0x7f6bdc06f670 2023-10-30T14:51:39.728 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-30T14:51:39.728 INFO:teuthology.orchestra.run.smithi057.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T14:51:39.728 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-30T14:51:39.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.724+0000 7f6bf49b5640 1 -- 172.21.15.57:0/2933415055 >> [v2:172.21.15.88:6805/2784816485,v1:172.21.15.88:6807/2784816485] conn(0x7f6bdc06f670 msgr2=0x7f6bdc071af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.724+0000 7f6bf49b5640 1 --2- 172.21.15.57:0/2933415055 >> [v2:172.21.15.88:6805/2784816485,v1:172.21.15.88:6807/2784816485] conn(0x7f6bdc06f670 0x7f6bdc071af0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.724+0000 7f6bf49b5640 1 -- 172.21.15.57:0/2933415055 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6bdc0664d0 msgr2=0x7f6bdc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.724+0000 7f6bf49b5640 1 --2- 172.21.15.57:0/2933415055 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f6bdc0664d0 0x7f6bdc068990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f6be00af000 tx=0x7f6be00046d0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.724+0000 7f6bf49b5640 1 -- 172.21.15.57:0/2933415055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bf015ad40 msgr2=0x7f6bf015b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.724+0000 7f6bf49b5640 1 --2- 172.21.15.57:0/2933415055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bf015ad40 0x7f6bf015b1f0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f6bf00b1e30 tx=0x7f6be800e530 comp rx=0 tx=0).stop 2023-10-30T14:51:39.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.724+0000 7f6beffff640 1 -- 172.21.15.57:0/2933415055 reap_dead start 2023-10-30T14:51:39.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.724+0000 7f6bf49b5640 1 -- 172.21.15.57:0/2933415055 shutdown_connections 2023-10-30T14:51:39.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.724+0000 7f6bf49b5640 1 -- 172.21.15.57:0/2933415055 >> 172.21.15.57:0/2933415055 conn(0x7f6bf00cf6d0 msgr2=0x7f6bf014dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:39.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.724+0000 7f6bf49b5640 1 -- 172.21.15.57:0/2933415055 shutdown_connections 2023-10-30T14:51:39.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.724+0000 7f6bf49b5640 1 -- 172.21.15.57:0/2933415055 wait complete. 2023-10-30T14:51:39.749 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-30T14:51:39.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5953ed6640 1 Processor -- start 2023-10-30T14:51:39.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5953ed6640 1 -- start start 2023-10-30T14:51:39.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5953ed6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f594c15cf90 0x7f594c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5953ed6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594c159f20 con 0x7f594c0c4a10 2023-10-30T14:51:39.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5953ed6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594c0c47e0 con 0x7f594c153c80 2023-10-30T14:51:39.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5953ed6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594c15dec0 con 0x7f594c15cf90 2023-10-30T14:51:39.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5951471640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f594c0c4a10 0x7f594c0ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41882/0 (socket says 172.21.15.57:41882) 2023-10-30T14:51:39.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5951471640 1 -- 172.21.15.57:0/3166582482 learned_addr learned my addr 172.21.15.57:0/3166582482 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:51:39.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5952473640 1 --2- 172.21.15.57:0/3166582482 >> v2:172.21.15.88:3300/0 conn(0x7f594c15cf90 0x7f594c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5950c70640 1 -- 172.21.15.57:0/3166582482 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155192163 0 0) 0x7f594c159f20 con 0x7f594c0c4a10 2023-10-30T14:51:39.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5950c70640 1 -- 172.21.15.57:0/3166582482 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5938003530 con 0x7f594c0c4a10 2023-10-30T14:51:39.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5950c70640 1 -- 172.21.15.57:0/3166582482 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1156787370 0 0) 0x7f5938003530 con 0x7f594c0c4a10 2023-10-30T14:51:39.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5950c70640 1 -- 172.21.15.57:0/3166582482 >> v2:172.21.15.88:3300/0 conn(0x7f594c15cf90 msgr2=0x7f594c0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:39.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5950c70640 1 --2- 172.21.15.57:0/3166582482 >> v2:172.21.15.88:3300/0 conn(0x7f594c15cf90 0x7f594c0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5950c70640 1 -- 172.21.15.57:0/3166582482 >> v1:172.21.15.88:6789/0 conn(0x7f594c153c80 legacy=0x7f594c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5950c70640 1 -- 172.21.15.57:0/3166582482 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c15d620 con 0x7f594c0c4a10 2023-10-30T14:51:39.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5950c70640 1 -- 172.21.15.57:0/3166582482 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7f593c001680 con 0x7f594c0c4a10 2023-10-30T14:51:39.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5950c70640 1 -- 172.21.15.57:0/3166582482 >> v1:172.21.15.57:6789/0 conn(0x7f594c0c4a10 legacy=0x7f594c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5950c70640 1 --2- 172.21.15.57:0/3166582482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5938003980 0x7f5938003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5950c70640 1 --2- 172.21.15.57:0/3166582482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59380047d0 0x7f5938004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5950c70640 1 --2- 172.21.15.57:0/3166582482 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5938005190 0x7f593800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5950c70640 1 -- 172.21.15.57:0/3166582482 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c15dec0 con 0x7f59380047d0 2023-10-30T14:51:39.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5950c70640 1 -- 172.21.15.57:0/3166582482 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593800c040 con 0x7f5938003980 2023-10-30T14:51:39.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5950c70640 1 -- 172.21.15.57:0/3166582482 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593800c200 con 0x7f5938005190 2023-10-30T14:51:39.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5951471640 1 --2- 172.21.15.57:0/3166582482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59380047d0 0x7f5938004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5951c72640 1 --2- 172.21.15.57:0/3166582482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5938003980 0x7f5938003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.892+0000 7f5952473640 1 --2- 172.21.15.57:0/3166582482 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5938005190 0x7f593800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.896+0000 7f5951471640 1 -- 172.21.15.57:0/3166582482 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5938005190 msgr2=0x7f593800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:39.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.896+0000 7f5951471640 1 --2- 172.21.15.57:0/3166582482 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5938005190 0x7f593800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.896+0000 7f5951471640 1 -- 172.21.15.57:0/3166582482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5938003980 msgr2=0x7f5938003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:39.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.896+0000 7f5951471640 1 --2- 172.21.15.57:0/3166582482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5938003980 0x7f5938003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.896+0000 7f5951471640 1 -- 172.21.15.57:0/3166582482 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5938003530 con 0x7f59380047d0 2023-10-30T14:51:39.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.896+0000 7f5951c72640 1 --2- 172.21.15.57:0/3166582482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5938003980 0x7f5938003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:39.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.896+0000 7f5952473640 1 -- 172.21.15.57:0/3166582482 reap_dead start 2023-10-30T14:51:39.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.896+0000 7f5951471640 1 --2- 172.21.15.57:0/3166582482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59380047d0 0x7f5938004c50 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f593c00c210 tx=0x7f593c01cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84ce317466eafb4c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:39.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.896+0000 7f5950c70640 1 -- 172.21.15.57:0/3166582482 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c005ae0 con 0x7f59380047d0 2023-10-30T14:51:39.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.896+0000 7f5950c70640 1 -- 172.21.15.57:0/3166582482 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593c005c40 con 0x7f59380047d0 2023-10-30T14:51:39.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.896+0000 7f5950c70640 1 -- 172.21.15.57:0/3166582482 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c0a5ae0 con 0x7f59380047d0 2023-10-30T14:51:39.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.896+0000 7f5953ed6640 1 -- 172.21.15.57:0/3166582482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59380047d0 msgr2=0x7f5938004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.896+0000 7f5953ed6640 1 --2- 172.21.15.57:0/3166582482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59380047d0 0x7f5938004c50 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f593c00c210 tx=0x7f593c01cba0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.896+0000 7f5953ed6640 1 -- 172.21.15.57:0/3166582482 shutdown_connections 2023-10-30T14:51:39.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.896+0000 7f5953ed6640 1 --2- 172.21.15.57:0/3166582482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59380047d0 0x7f5938004c50 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.896+0000 7f5953ed6640 1 -- 172.21.15.57:0/3166582482 >> 172.21.15.57:0/3166582482 conn(0x7f594c0715c0 msgr2=0x7f594c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:39.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.896+0000 7f5953ed6640 1 -- 172.21.15.57:0/3166582482 shutdown_connections 2023-10-30T14:51:39.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.896+0000 7f5953ed6640 1 -- 172.21.15.57:0/3166582482 wait complete. 2023-10-30T14:51:39.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5953ed6640 1 Processor -- start 2023-10-30T14:51:39.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5953ed6640 1 -- start start 2023-10-30T14:51:39.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5953ed6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59380047d0 0x7f594c14faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5953ed6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f594c1f2450 0x7f594c149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5953ed6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f594c15bef0 0x7f594c15c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5953ed6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c155450 con 0x7f59380047d0 2023-10-30T14:51:39.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5953ed6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c155c40 con 0x7f594c1f2450 2023-10-30T14:51:39.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5953ed6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594c14a470 con 0x7f594c15bef0 2023-10-30T14:51:39.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5951c72640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59380047d0 0x7f594c14faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5951c72640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59380047d0 0x7f594c14faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56774/0 (socket says 172.21.15.57:56774) 2023-10-30T14:51:39.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5951c72640 1 -- 172.21.15.57:0/2174766962 learned_addr learned my addr 172.21.15.57:0/2174766962 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:39.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5951471640 1 --2- 172.21.15.57:0/2174766962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f594c1f2450 0x7f594c149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5952473640 1 --2- 172.21.15.57:0/2174766962 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f594c15bef0 0x7f594c15c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5951c72640 1 -- 172.21.15.57:0/2174766962 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f594c15bef0 msgr2=0x7f594c15c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5951c72640 1 --2- 172.21.15.57:0/2174766962 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f594c15bef0 0x7f594c15c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5951c72640 1 -- 172.21.15.57:0/2174766962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f594c1f2450 msgr2=0x7f594c149f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:39.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5951c72640 1 --2- 172.21.15.57:0/2174766962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f594c1f2450 0x7f594c149f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:39.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5951c72640 1 -- 172.21.15.57:0/2174766962 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c1f2bb0 con 0x7f59380047d0 2023-10-30T14:51:39.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5951c72640 1 --2- 172.21.15.57:0/2174766962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59380047d0 0x7f594c14faf0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f594800e480 tx=0x7f594800e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:39.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f59427fc640 1 -- 172.21.15.57:0/2174766962 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59480052e0 con 0x7f59380047d0 2023-10-30T14:51:39.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f59427fc640 1 -- 172.21.15.57:0/2174766962 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5948005440 con 0x7f59380047d0 2023-10-30T14:51:39.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f59427fc640 1 -- 172.21.15.57:0/2174766962 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5948003840 con 0x7f59380047d0 2023-10-30T14:51:39.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5953ed6640 1 -- 172.21.15.57:0/2174766962 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594c1f2e90 con 0x7f59380047d0 2023-10-30T14:51:39.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.900+0000 7f5953ed6640 1 -- 172.21.15.57:0/2174766962 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594c1558d0 con 0x7f59380047d0 2023-10-30T14:51:39.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.904+0000 7f5953ed6640 1 -- 172.21.15.57:0/2174766962 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f594c155710 con 0x7f59380047d0 2023-10-30T14:51:39.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.904+0000 7f59427fc640 1 -- 172.21.15.57:0/2174766962 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f59480039a0 con 0x7f59380047d0 2023-10-30T14:51:39.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.904+0000 7f59427fc640 1 --2- 172.21.15.57:0/2174766962 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f59300664d0 0x7f5930068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:39.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.904+0000 7f59427fc640 1 -- 172.21.15.57:0/2174766962 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f59480842c0 con 0x7f59380047d0 2023-10-30T14:51:39.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.904+0000 7f5951471640 1 --2- 172.21.15.57:0/2174766962 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f59300664d0 0x7f5930068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:39.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.904+0000 7f5951471640 1 --2- 172.21.15.57:0/2174766962 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f59300664d0 0x7f5930068990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f593c0ab000 tx=0x7f593c0b1040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:39.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:39.908+0000 7f59427fc640 1 -- 172.21.15.57:0/2174766962 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f59480510b0 con 0x7f59380047d0 2023-10-30T14:51:40.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.064+0000 7f5953ed6640 1 -- 172.21.15.57:0/2174766962 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f594c15dbe0 con 0x7f59300664d0 2023-10-30T14:51:40.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.068+0000 7f59427fc640 1 -- 172.21.15.57:0/2174766962 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f593c001580 con 0x7f59300664d0 2023-10-30T14:51:40.071 INFO:teuthology.orchestra.run.smithi057.stderr:instructed osd(s) 4 to deep-scrub 2023-10-30T14:51:40.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.068+0000 7f5953ed6640 1 -- 172.21.15.57:0/2174766962 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f59300664d0 msgr2=0x7f5930068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.068+0000 7f5953ed6640 1 --2- 172.21.15.57:0/2174766962 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f59300664d0 0x7f5930068990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f593c0ab000 tx=0x7f593c0b1040 comp rx=0 tx=0).stop 2023-10-30T14:51:40.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.068+0000 7f5953ed6640 1 -- 172.21.15.57:0/2174766962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59380047d0 msgr2=0x7f594c14faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.068+0000 7f5953ed6640 1 --2- 172.21.15.57:0/2174766962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59380047d0 0x7f594c14faf0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f594800e480 tx=0x7f594800e940 comp rx=0 tx=0).stop 2023-10-30T14:51:40.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.068+0000 7f5953ed6640 1 -- 172.21.15.57:0/2174766962 shutdown_connections 2023-10-30T14:51:40.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.068+0000 7f5953ed6640 1 --2- 172.21.15.57:0/2174766962 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f59300664d0 0x7f5930068990 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.068+0000 7f5953ed6640 1 --2- 172.21.15.57:0/2174766962 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f594c15bef0 0x7f594c15c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.068+0000 7f5953ed6640 1 --2- 172.21.15.57:0/2174766962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f594c1f2450 0x7f594c149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.068+0000 7f5953ed6640 1 --2- 172.21.15.57:0/2174766962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59380047d0 0x7f594c14faf0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.068+0000 7f5953ed6640 1 -- 172.21.15.57:0/2174766962 >> 172.21.15.57:0/2174766962 conn(0x7f594c0715c0 msgr2=0x7f594c150f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:40.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.068+0000 7f5953ed6640 1 -- 172.21.15.57:0/2174766962 shutdown_connections 2023-10-30T14:51:40.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.068+0000 7f5953ed6640 1 -- 172.21.15.57:0/2174766962 wait complete. 2023-10-30T14:51:40.092 INFO:tasks.ceph:Scrubbing osd.5 2023-10-30T14:51:40.093 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-30T14:51:40.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.240+0000 7eff2793e640 1 Processor -- start 2023-10-30T14:51:40.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.240+0000 7eff2793e640 1 -- start start 2023-10-30T14:51:40.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.240+0000 7eff2793e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff200c4340 0x7eff201599e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.240+0000 7eff2793e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff200c4720 0x7eff20069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.240+0000 7eff2793e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7eff200ca470 0x7eff200ca920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.240+0000 7eff2793e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff200c4cd0 con 0x7eff200ca470 2023-10-30T14:51:40.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.240+0000 7eff2793e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff2006af00 con 0x7eff200c4720 2023-10-30T14:51:40.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.240+0000 7eff2793e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff2006a3d0 con 0x7eff200c4340 2023-10-30T14:51:40.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.240+0000 7eff25edb640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7eff200ca470 0x7eff200ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.240+0000 7eff25edb640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7eff200ca470 0x7eff200ca920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56786/0 (socket says 172.21.15.57:56786) 2023-10-30T14:51:40.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.240+0000 7eff25edb640 1 -- 172.21.15.57:0/3795628229 learned_addr learned my addr 172.21.15.57:0/3795628229 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:40.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.240+0000 7eff256da640 1 --2- 172.21.15.57:0/3795628229 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff200c4340 0x7eff201599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.240+0000 7eff24ed9640 1 --2- 172.21.15.57:0/3795628229 >> v2:172.21.15.88:3300/0 conn(0x7eff200c4720 0x7eff20069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.240+0000 7eff25edb640 1 -- 172.21.15.57:0/3795628229 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff200c4340 msgr2=0x7eff201599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:40.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.240+0000 7eff25edb640 1 --2- 172.21.15.57:0/3795628229 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff200c4340 0x7eff201599e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.240+0000 7eff25edb640 1 -- 172.21.15.57:0/3795628229 >> v2:172.21.15.88:3300/0 conn(0x7eff200c4720 msgr2=0x7eff20069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:40.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.240+0000 7eff25edb640 1 --2- 172.21.15.57:0/3795628229 >> v2:172.21.15.88:3300/0 conn(0x7eff200c4720 0x7eff20069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.240+0000 7eff25edb640 1 -- 172.21.15.57:0/3795628229 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2006a660 con 0x7eff200ca470 2023-10-30T14:51:40.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.244+0000 7eff25edb640 1 --2- 172.21.15.57:0/3795628229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff200ca470 0x7eff200ca920 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7eff2006d080 tx=0x7eff1801c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7340e4ff863d91 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:40.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.244+0000 7eff17fff640 1 -- 172.21.15.57:0/3795628229 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff180a64e0 con 0x7eff200ca470 2023-10-30T14:51:40.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.244+0000 7eff17fff640 1 -- 172.21.15.57:0/3795628229 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff180a6640 con 0x7eff200ca470 2023-10-30T14:51:40.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.244+0000 7eff17fff640 1 -- 172.21.15.57:0/3795628229 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff180a69d0 con 0x7eff200ca470 2023-10-30T14:51:40.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.244+0000 7eff2793e640 1 -- 172.21.15.57:0/3795628229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff200ca470 msgr2=0x7eff200ca920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.244+0000 7eff2793e640 1 --2- 172.21.15.57:0/3795628229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff200ca470 0x7eff200ca920 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7eff2006d080 tx=0x7eff1801c650 comp rx=0 tx=0).stop 2023-10-30T14:51:40.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.244+0000 7eff2793e640 1 -- 172.21.15.57:0/3795628229 shutdown_connections 2023-10-30T14:51:40.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.244+0000 7eff2793e640 1 --2- 172.21.15.57:0/3795628229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff200ca470 0x7eff200ca920 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.244+0000 7eff2793e640 1 --2- 172.21.15.57:0/3795628229 >> v2:172.21.15.88:3300/0 conn(0x7eff200c4720 0x7eff20069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.244+0000 7eff2793e640 1 --2- 172.21.15.57:0/3795628229 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff200c4340 0x7eff201599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.244+0000 7eff2793e640 1 -- 172.21.15.57:0/3795628229 >> 172.21.15.57:0/3795628229 conn(0x7eff200cf1c0 msgr2=0x7eff200cf5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:40.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.244+0000 7eff2793e640 1 -- 172.21.15.57:0/3795628229 shutdown_connections 2023-10-30T14:51:40.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.244+0000 7eff2793e640 1 -- 172.21.15.57:0/3795628229 wait complete. 2023-10-30T14:51:40.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff2793e640 1 Processor -- start 2023-10-30T14:51:40.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff2793e640 1 -- start start 2023-10-30T14:51:40.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff2793e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff200c4340 0x7eff201f5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff2793e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff200c4720 0x7eff20163610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff2793e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff200ca470 0x7eff2014b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff2793e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff20155cc0 con 0x7eff200c4340 2023-10-30T14:51:40.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff2793e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff200c8cf0 con 0x7eff200c4720 2023-10-30T14:51:40.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff2793e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff20154790 con 0x7eff200ca470 2023-10-30T14:51:40.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff256da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff200c4340 0x7eff201f5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff256da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff200c4340 0x7eff201f5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56798/0 (socket says 172.21.15.57:56798) 2023-10-30T14:51:40.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff256da640 1 -- 172.21.15.57:0/3424054911 learned_addr learned my addr 172.21.15.57:0/3424054911 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:40.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff25edb640 1 --2- 172.21.15.57:0/3424054911 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff200ca470 0x7eff2014b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff24ed9640 1 --2- 172.21.15.57:0/3424054911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff200c4720 0x7eff20163610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff256da640 1 -- 172.21.15.57:0/3424054911 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff200ca470 msgr2=0x7eff2014b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff256da640 1 --2- 172.21.15.57:0/3424054911 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff200ca470 0x7eff2014b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff256da640 1 -- 172.21.15.57:0/3424054911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff200c4720 msgr2=0x7eff20163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff256da640 1 --2- 172.21.15.57:0/3424054911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff200c4720 0x7eff20163610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff256da640 1 -- 172.21.15.57:0/3424054911 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff18009b30 con 0x7eff200c4340 2023-10-30T14:51:40.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff25edb640 1 --2- 172.21.15.57:0/3424054911 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff200ca470 0x7eff2014b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:40.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff256da640 1 --2- 172.21.15.57:0/3424054911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff200c4340 0x7eff201f5180 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7eff1c00d800 tx=0x7eff1c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:40.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff15ffb640 1 -- 172.21.15.57:0/3424054911 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c0096a0 con 0x7eff200c4340 2023-10-30T14:51:40.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff15ffb640 1 -- 172.21.15.57:0/3424054911 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1c009800 con 0x7eff200c4340 2023-10-30T14:51:40.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff15ffb640 1 -- 172.21.15.57:0/3424054911 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c009990 con 0x7eff200c4340 2023-10-30T14:51:40.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff2793e640 1 -- 172.21.15.57:0/3424054911 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff20154a70 con 0x7eff200c4340 2023-10-30T14:51:40.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.248+0000 7eff2793e640 1 -- 172.21.15.57:0/3424054911 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff20155670 con 0x7eff200c4340 2023-10-30T14:51:40.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.252+0000 7eff2793e640 1 -- 172.21.15.57:0/3424054911 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7eff200c9f50 con 0x7eff200c4340 2023-10-30T14:51:40.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.252+0000 7eff15ffb640 1 -- 172.21.15.57:0/3424054911 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7eff1c009af0 con 0x7eff200c4340 2023-10-30T14:51:40.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.252+0000 7eff15ffb640 1 --2- 172.21.15.57:0/3424054911 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7eff040664a0 0x7eff04068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.252+0000 7eff15ffb640 1 -- 172.21.15.57:0/3424054911 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7eff1c084ae0 con 0x7eff200c4340 2023-10-30T14:51:40.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.252+0000 7eff15ffb640 1 --2- 172.21.15.57:0/3424054911 >> [v2:172.21.15.88:6800/2682449812,v1:172.21.15.88:6801/2682449812] conn(0x7eff0406f640 0x7eff04071ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.252+0000 7eff15ffb640 1 -- 172.21.15.57:0/3424054911 --> [v2:172.21.15.88:6800/2682449812,v1:172.21.15.88:6801/2682449812] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff04072190 con 0x7eff0406f640 2023-10-30T14:51:40.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.252+0000 7eff15ffb640 1 -- 172.21.15.57:0/3424054911 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff1c0892a0 con 0x7eff200c4340 2023-10-30T14:51:40.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.252+0000 7eff24ed9640 1 --2- 172.21.15.57:0/3424054911 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7eff040664a0 0x7eff04068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.252+0000 7eff25edb640 1 --2- 172.21.15.57:0/3424054911 >> [v2:172.21.15.88:6800/2682449812,v1:172.21.15.88:6801/2682449812] conn(0x7eff0406f640 0x7eff04071ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.252+0000 7eff24ed9640 1 --2- 172.21.15.57:0/3424054911 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7eff040664a0 0x7eff04068960 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7eff20157ba0 tx=0x7eff10008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:40.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.252+0000 7eff25edb640 1 --2- 172.21.15.57:0/3424054911 >> [v2:172.21.15.88:6800/2682449812,v1:172.21.15.88:6801/2682449812] conn(0x7eff0406f640 0x7eff04071ac0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:40.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.256+0000 7eff15ffb640 1 -- 172.21.15.57:0/3424054911 <== osd.5 v2:172.21.15.88:6800/2682449812 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7eff04072190 con 0x7eff0406f640 2023-10-30T14:51:40.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.268+0000 7eff2793e640 1 -- 172.21.15.57:0/3424054911 --> [v2:172.21.15.88:6800/2682449812,v1:172.21.15.88:6801/2682449812] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7eff200725b0 con 0x7eff0406f640 2023-10-30T14:51:40.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.268+0000 7eff15ffb640 1 -- 172.21.15.57:0/3424054911 <== osd.5 v2:172.21.15.88:6800/2682449812 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7eff200725b0 con 0x7eff0406f640 2023-10-30T14:51:40.274 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-30T14:51:40.274 INFO:teuthology.orchestra.run.smithi057.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T14:51:40.274 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-30T14:51:40.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.268+0000 7eff2793e640 1 -- 172.21.15.57:0/3424054911 >> [v2:172.21.15.88:6800/2682449812,v1:172.21.15.88:6801/2682449812] conn(0x7eff0406f640 msgr2=0x7eff04071ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.268+0000 7eff2793e640 1 --2- 172.21.15.57:0/3424054911 >> [v2:172.21.15.88:6800/2682449812,v1:172.21.15.88:6801/2682449812] conn(0x7eff0406f640 0x7eff04071ac0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.268+0000 7eff2793e640 1 -- 172.21.15.57:0/3424054911 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7eff040664a0 msgr2=0x7eff04068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.268+0000 7eff2793e640 1 --2- 172.21.15.57:0/3424054911 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7eff040664a0 0x7eff04068960 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7eff20157ba0 tx=0x7eff10008040 comp rx=0 tx=0).stop 2023-10-30T14:51:40.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.268+0000 7eff2793e640 1 -- 172.21.15.57:0/3424054911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff200c4340 msgr2=0x7eff201f5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.268+0000 7eff2793e640 1 --2- 172.21.15.57:0/3424054911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff200c4340 0x7eff201f5180 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7eff1c00d800 tx=0x7eff1c00dcc0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.268+0000 7eff25edb640 1 -- 172.21.15.57:0/3424054911 reap_dead start 2023-10-30T14:51:40.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.268+0000 7eff2793e640 1 -- 172.21.15.57:0/3424054911 shutdown_connections 2023-10-30T14:51:40.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.268+0000 7eff2793e640 1 -- 172.21.15.57:0/3424054911 >> 172.21.15.57:0/3424054911 conn(0x7eff200cf1c0 msgr2=0x7eff2014fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:40.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.268+0000 7eff2793e640 1 -- 172.21.15.57:0/3424054911 shutdown_connections 2023-10-30T14:51:40.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.268+0000 7eff2793e640 1 -- 172.21.15.57:0/3424054911 wait complete. 2023-10-30T14:51:40.292 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-30T14:51:40.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.432+0000 7f55b13a0640 1 Processor -- start 2023-10-30T14:51:40.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.432+0000 7f55b13a0640 1 -- start start 2023-10-30T14:51:40.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.436+0000 7f55b13a0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f55ac0b3950 0x7f55ac0c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.436+0000 7f55b13a0640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55ac0b27f0 con 0x7f55ac15cdf0 2023-10-30T14:51:40.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.436+0000 7f55b13a0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55ac15d370 con 0x7f55ac0c8b00 2023-10-30T14:51:40.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.436+0000 7f55b13a0640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55ac15d630 con 0x7f55ac0b3950 2023-10-30T14:51:40.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.436+0000 7f55ab7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f55ac15cdf0 0x7f55ac0b3240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41890/0 (socket says 172.21.15.57:41890) 2023-10-30T14:51:40.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.436+0000 7f55ab7fe640 1 -- 172.21.15.57:0/2380307233 learned_addr learned my addr 172.21.15.57:0/2380307233 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:51:40.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.436+0000 7f55b0b9f640 1 --2- 172.21.15.57:0/2380307233 >> v2:172.21.15.57:3300/0 conn(0x7f55ac0b3950 0x7f55ac0c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.436+0000 7f55aaffd640 1 -- 172.21.15.57:0/2380307233 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373753483 0 0) 0x7f55ac0b27f0 con 0x7f55ac15cdf0 2023-10-30T14:51:40.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.436+0000 7f55aaffd640 1 -- 172.21.15.57:0/2380307233 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5594003530 con 0x7f55ac15cdf0 2023-10-30T14:51:40.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.436+0000 7f55b0b9f640 1 -- 172.21.15.57:0/2380307233 >> v1:172.21.15.88:6789/0 conn(0x7f55ac0c8b00 legacy=0x7f55ac0c9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.436+0000 7f55b0b9f640 1 -- 172.21.15.57:0/2380307233 >> v1:172.21.15.57:6789/0 conn(0x7f55ac15cdf0 legacy=0x7f55ac0b3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.436+0000 7f55b0b9f640 1 -- 172.21.15.57:0/2380307233 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ac15d910 con 0x7f55ac0b3950 2023-10-30T14:51:40.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.440+0000 7f55b0b9f640 1 --2- 172.21.15.57:0/2380307233 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55ac0b3950 0x7f55ac0c8ee0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f55a4009dc0 tx=0x7f55a401f120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65530bdf49163ba3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:40.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.440+0000 7f55aaffd640 1 -- 172.21.15.57:0/2380307233 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a40af600 con 0x7f55ac0b3950 2023-10-30T14:51:40.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.440+0000 7f55aaffd640 1 -- 172.21.15.57:0/2380307233 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a40ab050 con 0x7f55ac0b3950 2023-10-30T14:51:40.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.440+0000 7f55aaffd640 1 -- 172.21.15.57:0/2380307233 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a40af980 con 0x7f55ac0b3950 2023-10-30T14:51:40.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.440+0000 7f55b13a0640 1 -- 172.21.15.57:0/2380307233 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55ac0b3950 msgr2=0x7f55ac0c8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.440+0000 7f55b13a0640 1 --2- 172.21.15.57:0/2380307233 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55ac0b3950 0x7f55ac0c8ee0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f55a4009dc0 tx=0x7f55a401f120 comp rx=0 tx=0).stop 2023-10-30T14:51:40.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.440+0000 7f55b13a0640 1 -- 172.21.15.57:0/2380307233 shutdown_connections 2023-10-30T14:51:40.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.440+0000 7f55b13a0640 1 --2- 172.21.15.57:0/2380307233 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55ac0b3950 0x7f55ac0c8ee0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.440+0000 7f55b13a0640 1 -- 172.21.15.57:0/2380307233 >> 172.21.15.57:0/2380307233 conn(0x7f55ac06bed0 msgr2=0x7f55ac06c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:40.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.440+0000 7f55b13a0640 1 -- 172.21.15.57:0/2380307233 shutdown_connections 2023-10-30T14:51:40.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.440+0000 7f55b13a0640 1 -- 172.21.15.57:0/2380307233 wait complete. 2023-10-30T14:51:40.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.440+0000 7f55b13a0640 1 Processor -- start 2023-10-30T14:51:40.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.440+0000 7f55b13a0640 1 -- start start 2023-10-30T14:51:40.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55b13a0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55ac0b3950 0x7f55ac1ef1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55b13a0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f55ac0c8b00 0x7f55ac1567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55b13a0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55ac15cdf0 0x7f55ac157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55b13a0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac0c5370 con 0x7f55ac0b3950 2023-10-30T14:51:40.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55b13a0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac0c56f0 con 0x7f55ac15cdf0 2023-10-30T14:51:40.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55b13a0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac0c5cb0 con 0x7f55ac0c8b00 2023-10-30T14:51:40.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55abfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55ac0b3950 0x7f55ac1ef1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55abfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55ac0b3950 0x7f55ac1ef1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56828/0 (socket says 172.21.15.57:56828) 2023-10-30T14:51:40.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55abfff640 1 -- 172.21.15.57:0/1840769276 learned_addr learned my addr 172.21.15.57:0/1840769276 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:40.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55ab7fe640 1 --2- 172.21.15.57:0/1840769276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f55ac0c8b00 0x7f55ac1567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55b0b9f640 1 --2- 172.21.15.57:0/1840769276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55ac15cdf0 0x7f55ac157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55abfff640 1 -- 172.21.15.57:0/1840769276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f55ac0c8b00 msgr2=0x7f55ac1567d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55abfff640 1 --2- 172.21.15.57:0/1840769276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f55ac0c8b00 0x7f55ac1567d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55abfff640 1 -- 172.21.15.57:0/1840769276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55ac15cdf0 msgr2=0x7f55ac157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55abfff640 1 --2- 172.21.15.57:0/1840769276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55ac15cdf0 0x7f55ac157e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55abfff640 1 -- 172.21.15.57:0/1840769276 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a4009a70 con 0x7f55ac0b3950 2023-10-30T14:51:40.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55ab7fe640 1 --2- 172.21.15.57:0/1840769276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f55ac0c8b00 0x7f55ac1567d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:40.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55abfff640 1 --2- 172.21.15.57:0/1840769276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55ac0b3950 0x7f55ac1ef1d0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f559c00c500 tx=0x7f559c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:40.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55a8ff9640 1 -- 172.21.15.57:0/1840769276 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c005220 con 0x7f55ac0b3950 2023-10-30T14:51:40.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55a8ff9640 1 -- 172.21.15.57:0/1840769276 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f559c005380 con 0x7f55ac0b3950 2023-10-30T14:51:40.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55a8ff9640 1 -- 172.21.15.57:0/1840769276 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c007270 con 0x7f55ac0b3950 2023-10-30T14:51:40.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55b13a0640 1 -- 172.21.15.57:0/1840769276 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55ac15ba10 con 0x7f55ac0b3950 2023-10-30T14:51:40.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55b13a0640 1 -- 172.21.15.57:0/1840769276 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55ac15bfb0 con 0x7f55ac0b3950 2023-10-30T14:51:40.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55b13a0640 1 -- 172.21.15.57:0/1840769276 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55ac04e170 con 0x7f55ac0b3950 2023-10-30T14:51:40.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.444+0000 7f55a8ff9640 1 -- 172.21.15.57:0/1840769276 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f559c0073d0 con 0x7f55ac0b3950 2023-10-30T14:51:40.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.448+0000 7f55a8ff9640 1 --2- 172.21.15.57:0/1840769276 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f55980664a0 0x7f5598068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.448+0000 7f55a8ff9640 1 -- 172.21.15.57:0/1840769276 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f559c083f00 con 0x7f55ac0b3950 2023-10-30T14:51:40.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.448+0000 7f55ab7fe640 1 --2- 172.21.15.57:0/1840769276 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f55980664a0 0x7f5598068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.448+0000 7f55ab7fe640 1 --2- 172.21.15.57:0/1840769276 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f55980664a0 0x7f5598068960 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f55ac15c580 tx=0x7f55a0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:40.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.452+0000 7f55a8ff9640 1 -- 172.21.15.57:0/1840769276 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f559c050c70 con 0x7f55ac0b3950 2023-10-30T14:51:40.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.612+0000 7f55b13a0640 1 -- 172.21.15.57:0/1840769276 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f55ac0b2a40 con 0x7f55980664a0 2023-10-30T14:51:40.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.612+0000 7f55a8ff9640 1 -- 172.21.15.57:0/1840769276 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f55a0001480 con 0x7f55980664a0 2023-10-30T14:51:40.618 INFO:teuthology.orchestra.run.smithi057.stderr:instructed osd(s) 5 to deep-scrub 2023-10-30T14:51:40.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.616+0000 7f55b13a0640 1 -- 172.21.15.57:0/1840769276 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f55980664a0 msgr2=0x7f5598068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.616+0000 7f55b13a0640 1 --2- 172.21.15.57:0/1840769276 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f55980664a0 0x7f5598068960 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f55ac15c580 tx=0x7f55a0009040 comp rx=0 tx=0).stop 2023-10-30T14:51:40.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.616+0000 7f55b13a0640 1 -- 172.21.15.57:0/1840769276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55ac0b3950 msgr2=0x7f55ac1ef1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.616+0000 7f55b13a0640 1 --2- 172.21.15.57:0/1840769276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55ac0b3950 0x7f55ac1ef1d0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f559c00c500 tx=0x7f559c00c9c0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.616+0000 7f55b13a0640 1 -- 172.21.15.57:0/1840769276 shutdown_connections 2023-10-30T14:51:40.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.616+0000 7f55b13a0640 1 --2- 172.21.15.57:0/1840769276 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f55980664a0 0x7f5598068960 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.616+0000 7f55b13a0640 1 --2- 172.21.15.57:0/1840769276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55ac15cdf0 0x7f55ac157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.616+0000 7f55b13a0640 1 --2- 172.21.15.57:0/1840769276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f55ac0c8b00 0x7f55ac1567d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.616+0000 7f55b13a0640 1 --2- 172.21.15.57:0/1840769276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55ac0b3950 0x7f55ac1ef1d0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.616+0000 7f55b13a0640 1 -- 172.21.15.57:0/1840769276 >> 172.21.15.57:0/1840769276 conn(0x7f55ac06bed0 msgr2=0x7f55ac153750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:40.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.616+0000 7f55b13a0640 1 -- 172.21.15.57:0/1840769276 shutdown_connections 2023-10-30T14:51:40.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.616+0000 7f55b13a0640 1 -- 172.21.15.57:0/1840769276 wait complete. 2023-10-30T14:51:40.644 INFO:tasks.ceph:Scrubbing osd.6 2023-10-30T14:51:40.645 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-30T14:51:40.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.792+0000 7f110230f640 1 Processor -- start 2023-10-30T14:51:40.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.792+0000 7f110230f640 1 -- start start 2023-10-30T14:51:40.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.792+0000 7f110230f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc06fb20 0x7f10fc0c7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.792+0000 7f110230f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f10fc153bd0 0x7f10fc0c20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.792+0000 7f110230f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10fc15ded0 con 0x7f10fc06ff00 2023-10-30T14:51:40.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.792+0000 7f110230f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10fc06ee00 con 0x7f10fc153bd0 2023-10-30T14:51:40.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.792+0000 7f110230f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc06b130 con 0x7f10fc06fb20 2023-10-30T14:51:40.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.792+0000 7f11008ac640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f10fc153bd0 0x7f10fc0c20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.792+0000 7f10fb7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f10fc06ff00 0x7f10fc1534c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41906/0 (socket says 172.21.15.57:41906) 2023-10-30T14:51:40.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.792+0000 7f10fb7fe640 1 -- 172.21.15.57:0/2954467369 learned_addr learned my addr 172.21.15.57:0/2954467369 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:51:40.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.792+0000 7f10fbfff640 1 --2- 172.21.15.57:0/2954467369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc06fb20 0x7f10fc0c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.792+0000 7f11008ac640 1 -- 172.21.15.57:0/2954467369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc06fb20 msgr2=0x7f10fc0c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:40.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.792+0000 7f11008ac640 1 --2- 172.21.15.57:0/2954467369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc06fb20 0x7f10fc0c7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.792+0000 7f11008ac640 1 -- 172.21.15.57:0/2954467369 >> v1:172.21.15.57:6789/0 conn(0x7f10fc06ff00 legacy=0x7f10fc1534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.792+0000 7f11008ac640 1 -- 172.21.15.57:0/2954467369 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10fc06cff0 con 0x7f10fc153bd0 2023-10-30T14:51:40.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.792+0000 7f10faffd640 1 -- 172.21.15.57:0/2954467369 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59759872 0 0) 0x7f10fc15ded0 con 0x7f10fc06ff00 2023-10-30T14:51:40.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.796+0000 7f11008ac640 1 --2- 172.21.15.57:0/2954467369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10fc153bd0 0x7f10fc0c20f0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f10f4009d90 tx=0x7f10f401c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=752dbac945cedb30 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:40.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.796+0000 7f10faffd640 1 -- 172.21.15.57:0/2954467369 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f40a5780 con 0x7f10fc153bd0 2023-10-30T14:51:40.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.796+0000 7f10faffd640 1 -- 172.21.15.57:0/2954467369 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f40a58e0 con 0x7f10fc153bd0 2023-10-30T14:51:40.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.796+0000 7f10faffd640 1 -- 172.21.15.57:0/2954467369 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f40a5c70 con 0x7f10fc153bd0 2023-10-30T14:51:40.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.796+0000 7f110230f640 1 -- 172.21.15.57:0/2954467369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10fc153bd0 msgr2=0x7f10fc0c20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.796+0000 7f110230f640 1 --2- 172.21.15.57:0/2954467369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10fc153bd0 0x7f10fc0c20f0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f10f4009d90 tx=0x7f10f401c8c0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.796+0000 7f110230f640 1 -- 172.21.15.57:0/2954467369 shutdown_connections 2023-10-30T14:51:40.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.796+0000 7f110230f640 1 --2- 172.21.15.57:0/2954467369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10fc153bd0 0x7f10fc0c20f0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.796+0000 7f110230f640 1 --2- 172.21.15.57:0/2954467369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc06fb20 0x7f10fc0c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.796+0000 7f110230f640 1 -- 172.21.15.57:0/2954467369 >> 172.21.15.57:0/2954467369 conn(0x7f10fc0cd010 msgr2=0x7f10fc0cd420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:40.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.796+0000 7f110230f640 1 -- 172.21.15.57:0/2954467369 shutdown_connections 2023-10-30T14:51:40.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.796+0000 7f110230f640 1 -- 172.21.15.57:0/2954467369 wait complete. 2023-10-30T14:51:40.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.796+0000 7f110230f640 1 Processor -- start 2023-10-30T14:51:40.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.796+0000 7f110230f640 1 -- start start 2023-10-30T14:51:40.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f110230f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10fc06fb20 0x7f10fc1f1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f110230f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc06ff00 0x7f10fc152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f110230f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10fc153bd0 0x7f10fc14b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f110230f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc150e10 con 0x7f10fc06fb20 2023-10-30T14:51:40.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f110230f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc1f2300 con 0x7f10fc153bd0 2023-10-30T14:51:40.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f110230f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc1f25e0 con 0x7f10fc06ff00 2023-10-30T14:51:40.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f10fbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10fc06fb20 0x7f10fc1f1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f10fbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10fc06fb20 0x7f10fc1f1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56848/0 (socket says 172.21.15.57:56848) 2023-10-30T14:51:40.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f10fbfff640 1 -- 172.21.15.57:0/430985494 learned_addr learned my addr 172.21.15.57:0/430985494 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:40.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f10fb7fe640 1 --2- 172.21.15.57:0/430985494 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc06ff00 0x7f10fc152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f11008ac640 1 --2- 172.21.15.57:0/430985494 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10fc153bd0 0x7f10fc14b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f10fbfff640 1 -- 172.21.15.57:0/430985494 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc06ff00 msgr2=0x7f10fc152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f10fbfff640 1 --2- 172.21.15.57:0/430985494 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc06ff00 0x7f10fc152850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f10fbfff640 1 -- 172.21.15.57:0/430985494 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10fc153bd0 msgr2=0x7f10fc14b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f10fbfff640 1 --2- 172.21.15.57:0/430985494 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10fc153bd0 0x7f10fc14b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f10fbfff640 1 -- 172.21.15.57:0/430985494 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f4009a70 con 0x7f10fc06fb20 2023-10-30T14:51:40.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f10fb7fe640 1 --2- 172.21.15.57:0/430985494 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc06ff00 0x7f10fc152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:40.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f10fbfff640 1 --2- 172.21.15.57:0/430985494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10fc06fb20 0x7f10fc1f1ae0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f10ec00e930 tx=0x7f10ec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:40.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f10f8ff9640 1 -- 172.21.15.57:0/430985494 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ec009690 con 0x7f10fc06fb20 2023-10-30T14:51:40.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f10f8ff9640 1 -- 172.21.15.57:0/430985494 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10ec00ca00 con 0x7f10fc06fb20 2023-10-30T14:51:40.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f10f8ff9640 1 -- 172.21.15.57:0/430985494 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ec00cd70 con 0x7f10fc06fb20 2023-10-30T14:51:40.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f110230f640 1 -- 172.21.15.57:0/430985494 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10fc1f28c0 con 0x7f10fc06fb20 2023-10-30T14:51:40.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f110230f640 1 -- 172.21.15.57:0/430985494 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10fc151340 con 0x7f10fc06fb20 2023-10-30T14:51:40.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f110230f640 1 -- 172.21.15.57:0/430985494 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f10fc06e740 con 0x7f10fc06fb20 2023-10-30T14:51:40.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.800+0000 7f10f8ff9640 1 -- 172.21.15.57:0/430985494 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f10ec013470 con 0x7f10fc06fb20 2023-10-30T14:51:40.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.804+0000 7f10f8ff9640 1 --2- 172.21.15.57:0/430985494 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f10e80664a0 0x7f10e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.804+0000 7f10f8ff9640 1 -- 172.21.15.57:0/430985494 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f10ec0848f0 con 0x7f10fc06fb20 2023-10-30T14:51:40.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.804+0000 7f10f8ff9640 1 --2- 172.21.15.57:0/430985494 >> [v2:172.21.15.88:6819/3340693897,v1:172.21.15.88:6822/3340693897] conn(0x7f10e806f640 0x7f10e8071ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.804+0000 7f10f8ff9640 1 -- 172.21.15.57:0/430985494 --> [v2:172.21.15.88:6819/3340693897,v1:172.21.15.88:6822/3340693897] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10e8072190 con 0x7f10e806f640 2023-10-30T14:51:40.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.804+0000 7f10f8ff9640 1 -- 172.21.15.57:0/430985494 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10ec0892a0 con 0x7f10fc06fb20 2023-10-30T14:51:40.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.804+0000 7f10fb7fe640 1 --2- 172.21.15.57:0/430985494 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f10e80664a0 0x7f10e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.804+0000 7f11008ac640 1 --2- 172.21.15.57:0/430985494 >> [v2:172.21.15.88:6819/3340693897,v1:172.21.15.88:6822/3340693897] conn(0x7f10e806f640 0x7f10e8071ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.804+0000 7f10fb7fe640 1 --2- 172.21.15.57:0/430985494 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f10e80664a0 0x7f10e8068960 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0c8430 tx=0x7f10f0007c40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:40.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.804+0000 7f11008ac640 1 --2- 172.21.15.57:0/430985494 >> [v2:172.21.15.88:6819/3340693897,v1:172.21.15.88:6822/3340693897] conn(0x7f10e806f640 0x7f10e8071ac0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:40.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.804+0000 7f10f8ff9640 1 -- 172.21.15.57:0/430985494 <== osd.6 v2:172.21.15.88:6819/3340693897 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f10e8072190 con 0x7f10e806f640 2023-10-30T14:51:40.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.820+0000 7f110230f640 1 -- 172.21.15.57:0/430985494 --> [v2:172.21.15.88:6819/3340693897,v1:172.21.15.88:6822/3340693897] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f10fc06e460 con 0x7f10e806f640 2023-10-30T14:51:40.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.820+0000 7f10f8ff9640 1 -- 172.21.15.57:0/430985494 <== osd.6 v2:172.21.15.88:6819/3340693897 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f10f40a5580 con 0x7f10e806f640 2023-10-30T14:51:40.824 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-30T14:51:40.824 INFO:teuthology.orchestra.run.smithi057.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T14:51:40.825 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-30T14:51:40.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.820+0000 7f110230f640 1 -- 172.21.15.57:0/430985494 >> [v2:172.21.15.88:6819/3340693897,v1:172.21.15.88:6822/3340693897] conn(0x7f10e806f640 msgr2=0x7f10e8071ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.820+0000 7f110230f640 1 --2- 172.21.15.57:0/430985494 >> [v2:172.21.15.88:6819/3340693897,v1:172.21.15.88:6822/3340693897] conn(0x7f10e806f640 0x7f10e8071ac0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.820+0000 7f110230f640 1 -- 172.21.15.57:0/430985494 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f10e80664a0 msgr2=0x7f10e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.820+0000 7f110230f640 1 --2- 172.21.15.57:0/430985494 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f10e80664a0 0x7f10e8068960 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0c8430 tx=0x7f10f0007c40 comp rx=0 tx=0).stop 2023-10-30T14:51:40.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.820+0000 7f110230f640 1 -- 172.21.15.57:0/430985494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10fc06fb20 msgr2=0x7f10fc1f1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.820+0000 7f110230f640 1 --2- 172.21.15.57:0/430985494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10fc06fb20 0x7f10fc1f1ae0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f10ec00e930 tx=0x7f10ec00edf0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.820+0000 7f11008ac640 1 -- 172.21.15.57:0/430985494 reap_dead start 2023-10-30T14:51:40.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.820+0000 7f110230f640 1 -- 172.21.15.57:0/430985494 shutdown_connections 2023-10-30T14:51:40.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.820+0000 7f110230f640 1 -- 172.21.15.57:0/430985494 >> 172.21.15.57:0/430985494 conn(0x7f10fc0cd010 msgr2=0x7f10fc14c550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:40.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.820+0000 7f110230f640 1 -- 172.21.15.57:0/430985494 shutdown_connections 2023-10-30T14:51:40.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.820+0000 7f110230f640 1 -- 172.21.15.57:0/430985494 wait complete. 2023-10-30T14:51:40.843 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-30T14:51:40.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe4a1879640 1 Processor -- start 2023-10-30T14:51:40.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe4a1879640 1 -- start start 2023-10-30T14:51:40.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe4a1879640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe49c153c80 0x7fe49c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe4a1879640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe49c0c4a10 0x7fe49c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe4a1879640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49c159f20 con 0x7fe49c15a3b0 2023-10-30T14:51:40.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe4a1879640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe49c15dec0 con 0x7fe49c153c80 2023-10-30T14:51:40.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe4a1879640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c1599a0 con 0x7fe49c0c4a10 2023-10-30T14:51:40.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49b7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fe49c15a3b0 0x7fe49c0ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41920/0 (socket says 172.21.15.57:41920) 2023-10-30T14:51:40.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49b7fe640 1 -- 172.21.15.57:0/1914163640 learned_addr learned my addr 172.21.15.57:0/1914163640 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:51:40.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe493fff640 1 --2- 172.21.15.57:0/1914163640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe49c0c4a10 0x7fe49c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49affd640 1 --2- 172.21.15.57:0/1914163640 >> v2:172.21.15.88:3300/0 conn(0x7fe49c153c80 0x7fe49c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49a7fc640 1 -- 172.21.15.57:0/1914163640 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171714120 0 0) 0x7fe49c159f20 con 0x7fe49c15a3b0 2023-10-30T14:51:40.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49a7fc640 1 -- 172.21.15.57:0/1914163640 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe488003530 con 0x7fe49c15a3b0 2023-10-30T14:51:40.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49a7fc640 1 -- 172.21.15.57:0/1914163640 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2947497743 0 0) 0x7fe488003530 con 0x7fe49c15a3b0 2023-10-30T14:51:40.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49a7fc640 1 -- 172.21.15.57:0/1914163640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe49c0c4a10 msgr2=0x7fe49c0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:40.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49a7fc640 1 --2- 172.21.15.57:0/1914163640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe49c0c4a10 0x7fe49c0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49a7fc640 1 -- 172.21.15.57:0/1914163640 >> v2:172.21.15.88:3300/0 conn(0x7fe49c153c80 msgr2=0x7fe49c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:40.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49a7fc640 1 --2- 172.21.15.57:0/1914163640 >> v2:172.21.15.88:3300/0 conn(0x7fe49c153c80 0x7fe49c15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49a7fc640 1 -- 172.21.15.57:0/1914163640 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe49c0c47e0 con 0x7fe49c15a3b0 2023-10-30T14:51:40.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49a7fc640 1 -- 172.21.15.57:0/1914163640 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fe494001a00 con 0x7fe49c15a3b0 2023-10-30T14:51:40.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe493fff640 1 --2- 172.21.15.57:0/1914163640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe49c0c4a10 0x7fe49c0c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:51:40.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49a7fc640 1 -- 172.21.15.57:0/1914163640 >> v1:172.21.15.57:6789/0 conn(0x7fe49c15a3b0 legacy=0x7fe49c0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:40.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49a7fc640 1 --2- 172.21.15.57:0/1914163640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe488003980 0x7fe488003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49a7fc640 1 --2- 172.21.15.57:0/1914163640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe49c0c4a10 0x7fe4880047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49a7fc640 1 --2- 172.21.15.57:0/1914163640 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe488004d10 0x7fe488005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:40.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49a7fc640 1 -- 172.21.15.57:0/1914163640 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c15dec0 con 0x7fe488004d10 2023-10-30T14:51:40.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49a7fc640 1 -- 172.21.15.57:0/1914163640 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c1599a0 con 0x7fe488003980 2023-10-30T14:51:40.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49a7fc640 1 -- 172.21.15.57:0/1914163640 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48800b990 con 0x7fe49c0c4a10 2023-10-30T14:51:40.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49b7fe640 1 --2- 172.21.15.57:0/1914163640 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe488004d10 0x7fe488005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe49affd640 1 --2- 172.21.15.57:0/1914163640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe488003980 0x7fe488003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.984+0000 7fe493fff640 1 --2- 172.21.15.57:0/1914163640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe49c0c4a10 0x7fe4880047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:40.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.988+0000 7fe49b7fe640 1 -- 172.21.15.57:0/1914163640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe49c0c4a10 msgr2=0x7fe4880047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:40.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.988+0000 7fe49b7fe640 1 --2- 172.21.15.57:0/1914163640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe49c0c4a10 0x7fe4880047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.988+0000 7fe49b7fe640 1 -- 172.21.15.57:0/1914163640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe488003980 msgr2=0x7fe488003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:40.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.988+0000 7fe49b7fe640 1 --2- 172.21.15.57:0/1914163640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe488003980 0x7fe488003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:40.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.988+0000 7fe49b7fe640 1 -- 172.21.15.57:0/1914163640 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe484009420 con 0x7fe488004d10 2023-10-30T14:51:41.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.988+0000 7fe49b7fe640 1 --2- 172.21.15.57:0/1914163640 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe488004d10 0x7fe488005190 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fe49400a860 tx=0x7fe49401d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8270ddd9f628beb6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:41.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.988+0000 7fe49a7fc640 1 -- 172.21.15.57:0/1914163640 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4940ad630 con 0x7fe488004d10 2023-10-30T14:51:41.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.988+0000 7fe49a7fc640 1 -- 172.21.15.57:0/1914163640 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4940a8070 con 0x7fe488004d10 2023-10-30T14:51:41.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.988+0000 7fe49a7fc640 1 -- 172.21.15.57:0/1914163640 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4940ad9f0 con 0x7fe488004d10 2023-10-30T14:51:41.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.988+0000 7fe4a1879640 1 -- 172.21.15.57:0/1914163640 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe488004d10 msgr2=0x7fe488005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.988+0000 7fe4a1879640 1 --2- 172.21.15.57:0/1914163640 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe488004d10 0x7fe488005190 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fe49400a860 tx=0x7fe49401d360 comp rx=0 tx=0).stop 2023-10-30T14:51:41.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.988+0000 7fe49b7fe640 1 -- 172.21.15.57:0/1914163640 reap_dead start 2023-10-30T14:51:41.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe4a1879640 1 -- 172.21.15.57:0/1914163640 shutdown_connections 2023-10-30T14:51:41.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe4a1879640 1 -- 172.21.15.57:0/1914163640 >> 172.21.15.57:0/1914163640 conn(0x7fe49c0715c0 msgr2=0x7fe49c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:41.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe4a1879640 1 -- 172.21.15.57:0/1914163640 shutdown_connections 2023-10-30T14:51:41.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe4a1879640 1 -- 172.21.15.57:0/1914163640 wait complete. 2023-10-30T14:51:41.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe4a1879640 1 Processor -- start 2023-10-30T14:51:41.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe4a1879640 1 -- start start 2023-10-30T14:51:41.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe4a1879640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe488004d10 0x7fe49c1ef3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe4a1879640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe49c0cdb20 0x7fe49c15bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe4a1879640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe49c0cdf80 0x7fe49c15c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe4a1879640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c1ef930 con 0x7fe49c0cdf80 2023-10-30T14:51:41.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe4a1879640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c0ce560 con 0x7fe49c0cdb20 2023-10-30T14:51:41.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe4a1879640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c15d0d0 con 0x7fe488004d10 2023-10-30T14:51:41.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe49b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe49c0cdf80 0x7fe49c15c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe49b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe49c0cdf80 0x7fe49c15c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56866/0 (socket says 172.21.15.57:56866) 2023-10-30T14:51:41.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe49b7fe640 1 -- 172.21.15.57:0/2963880430 learned_addr learned my addr 172.21.15.57:0/2963880430 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:41.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe49affd640 1 --2- 172.21.15.57:0/2963880430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe488004d10 0x7fe49c1ef3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe493fff640 1 --2- 172.21.15.57:0/2963880430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe49c0cdb20 0x7fe49c15bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe49b7fe640 1 -- 172.21.15.57:0/2963880430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe488004d10 msgr2=0x7fe49c1ef3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe49b7fe640 1 --2- 172.21.15.57:0/2963880430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe488004d10 0x7fe49c1ef3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe49b7fe640 1 -- 172.21.15.57:0/2963880430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe49c0cdb20 msgr2=0x7fe49c15bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe49b7fe640 1 --2- 172.21.15.57:0/2963880430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe49c0cdb20 0x7fe49c15bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe49b7fe640 1 -- 172.21.15.57:0/2963880430 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe49c0cd1c0 con 0x7fe49c0cdf80 2023-10-30T14:51:41.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.992+0000 7fe49b7fe640 1 --2- 172.21.15.57:0/2963880430 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe49c0cdf80 0x7fe49c15c2e0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fe49401cdd0 tx=0x7fe488005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:41.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.996+0000 7fe498ff9640 1 -- 172.21.15.57:0/2963880430 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4940017c0 con 0x7fe49c0cdf80 2023-10-30T14:51:41.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.996+0000 7fe498ff9640 1 -- 172.21.15.57:0/2963880430 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe494001920 con 0x7fe49c0cdf80 2023-10-30T14:51:41.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.996+0000 7fe498ff9640 1 -- 172.21.15.57:0/2963880430 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe494004d70 con 0x7fe49c0cdf80 2023-10-30T14:51:41.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.996+0000 7fe4a1879640 1 -- 172.21.15.57:0/2963880430 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe49c0cd4a0 con 0x7fe49c0cdf80 2023-10-30T14:51:41.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.996+0000 7fe4a1879640 1 -- 172.21.15.57:0/2963880430 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe49c0b4f60 con 0x7fe49c0cdf80 2023-10-30T14:51:41.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.996+0000 7fe4a1879640 1 -- 172.21.15.57:0/2963880430 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe49c04e170 con 0x7fe49c0cdf80 2023-10-30T14:51:41.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.996+0000 7fe498ff9640 1 -- 172.21.15.57:0/2963880430 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe4940aa030 con 0x7fe49c0cdf80 2023-10-30T14:51:41.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.996+0000 7fe498ff9640 1 --2- 172.21.15.57:0/2963880430 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe47c0664d0 0x7fe47c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.996+0000 7fe498ff9640 1 -- 172.21.15.57:0/2963880430 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fe49411e330 con 0x7fe49c0cdf80 2023-10-30T14:51:41.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:40.996+0000 7fe49affd640 1 --2- 172.21.15.57:0/2963880430 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe47c0664d0 0x7fe47c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.000+0000 7fe49affd640 1 --2- 172.21.15.57:0/2963880430 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe47c0664d0 0x7fe47c068990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe48c009be0 tx=0x7fe48c009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:41.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.000+0000 7fe498ff9640 1 -- 172.21.15.57:0/2963880430 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe4940eb000 con 0x7fe49c0cdf80 2023-10-30T14:51:41.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.164+0000 7fe4a1879640 1 -- 172.21.15.57:0/2963880430 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fe49c15d900 con 0x7fe47c0664d0 2023-10-30T14:51:41.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.164+0000 7fe498ff9640 1 -- 172.21.15.57:0/2963880430 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe48c019070 con 0x7fe47c0664d0 2023-10-30T14:51:41.167 INFO:teuthology.orchestra.run.smithi057.stderr:instructed osd(s) 6 to deep-scrub 2023-10-30T14:51:41.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.164+0000 7fe4a1879640 1 -- 172.21.15.57:0/2963880430 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe47c0664d0 msgr2=0x7fe47c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.164+0000 7fe4a1879640 1 --2- 172.21.15.57:0/2963880430 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe47c0664d0 0x7fe47c068990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe48c009be0 tx=0x7fe48c009190 comp rx=0 tx=0).stop 2023-10-30T14:51:41.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.164+0000 7fe4a1879640 1 -- 172.21.15.57:0/2963880430 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe49c0cdf80 msgr2=0x7fe49c15c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.164+0000 7fe4a1879640 1 --2- 172.21.15.57:0/2963880430 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe49c0cdf80 0x7fe49c15c2e0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fe49401cdd0 tx=0x7fe488005800 comp rx=0 tx=0).stop 2023-10-30T14:51:41.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.164+0000 7fe4a1879640 1 -- 172.21.15.57:0/2963880430 shutdown_connections 2023-10-30T14:51:41.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.164+0000 7fe4a1879640 1 --2- 172.21.15.57:0/2963880430 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe47c0664d0 0x7fe47c068990 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.164+0000 7fe4a1879640 1 --2- 172.21.15.57:0/2963880430 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe49c0cdf80 0x7fe49c15c2e0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.164+0000 7fe4a1879640 1 --2- 172.21.15.57:0/2963880430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe49c0cdb20 0x7fe49c15bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.164+0000 7fe4a1879640 1 --2- 172.21.15.57:0/2963880430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe488004d10 0x7fe49c1ef3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.164+0000 7fe4a1879640 1 -- 172.21.15.57:0/2963880430 >> 172.21.15.57:0/2963880430 conn(0x7fe49c0715c0 msgr2=0x7fe49c150ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:41.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.164+0000 7fe4a1879640 1 -- 172.21.15.57:0/2963880430 shutdown_connections 2023-10-30T14:51:41.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.164+0000 7fe4a1879640 1 -- 172.21.15.57:0/2963880430 wait complete. 2023-10-30T14:51:41.193 INFO:tasks.ceph:Scrubbing osd.7 2023-10-30T14:51:41.193 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-30T14:51:41.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.340+0000 7fd148646640 1 Processor -- start 2023-10-30T14:51:41.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.340+0000 7fd148646640 1 -- start start 2023-10-30T14:51:41.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.344+0000 7fd148646640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1400c4340 0x7fd1401599e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.344+0000 7fd148646640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd1400c4720 0x7fd140069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.344+0000 7fd148646640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd14006af00 con 0x7fd1400ca470 2023-10-30T14:51:41.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.344+0000 7fd148646640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1400c4cd0 con 0x7fd1400c4720 2023-10-30T14:51:41.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.344+0000 7fd148646640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14006a3d0 con 0x7fd1400c4340 2023-10-30T14:51:41.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.344+0000 7fd1463e2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1400c4340 0x7fd1401599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.344+0000 7fd1463e2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1400c4340 0x7fd1401599e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:44450/0 (socket says 172.21.15.57:44450) 2023-10-30T14:51:41.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.344+0000 7fd145be1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd1400c4720 0x7fd140069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.344+0000 7fd1463e2640 1 -- 172.21.15.57:0/1523848878 learned_addr learned my addr 172.21.15.57:0/1523848878 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:41.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.344+0000 7fd1463e2640 1 -- 172.21.15.57:0/1523848878 >> v2:172.21.15.88:3300/0 conn(0x7fd1400c4720 msgr2=0x7fd140069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:41.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.344+0000 7fd1463e2640 1 --2- 172.21.15.57:0/1523848878 >> v2:172.21.15.88:3300/0 conn(0x7fd1400c4720 0x7fd140069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.344+0000 7fd1463e2640 1 -- 172.21.15.57:0/1523848878 >> v1:172.21.15.88:6789/0 conn(0x7fd1400ca470 legacy=0x7fd140153880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.344+0000 7fd1463e2640 1 -- 172.21.15.57:0/1523848878 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14006a660 con 0x7fd1400c4340 2023-10-30T14:51:41.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.348+0000 7fd1463e2640 1 --2- 172.21.15.57:0/1523848878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1400c4340 0x7fd1401599e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009870 tx=0x7fd13c01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e30a883404c1b60e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:41.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.348+0000 7fd1453e0640 1 -- 172.21.15.57:0/1523848878 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c0ab680 con 0x7fd1400c4340 2023-10-30T14:51:41.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.348+0000 7fd1453e0640 1 -- 172.21.15.57:0/1523848878 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13c0ab7e0 con 0x7fd1400c4340 2023-10-30T14:51:41.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.348+0000 7fd1453e0640 1 -- 172.21.15.57:0/1523848878 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c0abb70 con 0x7fd1400c4340 2023-10-30T14:51:41.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.348+0000 7fd148646640 1 -- 172.21.15.57:0/1523848878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1400c4340 msgr2=0x7fd1401599e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.348+0000 7fd148646640 1 --2- 172.21.15.57:0/1523848878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1400c4340 0x7fd1401599e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009870 tx=0x7fd13c01c480 comp rx=0 tx=0).stop 2023-10-30T14:51:41.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.348+0000 7fd148646640 1 -- 172.21.15.57:0/1523848878 shutdown_connections 2023-10-30T14:51:41.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.348+0000 7fd148646640 1 --2- 172.21.15.57:0/1523848878 >> v2:172.21.15.88:3300/0 conn(0x7fd1400c4720 0x7fd140069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.348+0000 7fd148646640 1 --2- 172.21.15.57:0/1523848878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1400c4340 0x7fd1401599e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.348+0000 7fd148646640 1 -- 172.21.15.57:0/1523848878 >> 172.21.15.57:0/1523848878 conn(0x7fd1400cf1c0 msgr2=0x7fd1400cf5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:41.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.348+0000 7fd148646640 1 -- 172.21.15.57:0/1523848878 shutdown_connections 2023-10-30T14:51:41.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.348+0000 7fd148646640 1 -- 172.21.15.57:0/1523848878 wait complete. 2023-10-30T14:51:41.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd148646640 1 Processor -- start 2023-10-30T14:51:41.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd148646640 1 -- start start 2023-10-30T14:51:41.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd148646640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1400c4340 0x7fd14014cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd148646640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1400c4720 0x7fd140149d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd148646640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1400ca470 0x7fd140157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd148646640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd140149590 con 0x7fd1400c4720 2023-10-30T14:51:41.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd148646640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14014e030 con 0x7fd1400ca470 2023-10-30T14:51:41.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd148646640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd140149ac0 con 0x7fd1400c4340 2023-10-30T14:51:41.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd145be1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1400c4720 0x7fd140149d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd145be1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1400c4720 0x7fd140149d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56880/0 (socket says 172.21.15.57:56880) 2023-10-30T14:51:41.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd145be1640 1 -- 172.21.15.57:0/1088671932 learned_addr learned my addr 172.21.15.57:0/1088671932 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:41.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd1463e2640 1 --2- 172.21.15.57:0/1088671932 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1400c4340 0x7fd14014cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd146be3640 1 --2- 172.21.15.57:0/1088671932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1400ca470 0x7fd140157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd145be1640 1 -- 172.21.15.57:0/1088671932 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1400c4340 msgr2=0x7fd14014cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd145be1640 1 --2- 172.21.15.57:0/1088671932 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1400c4340 0x7fd14014cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd145be1640 1 -- 172.21.15.57:0/1088671932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1400ca470 msgr2=0x7fd140157680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd145be1640 1 --2- 172.21.15.57:0/1088671932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1400ca470 0x7fd140157680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd145be1640 1 -- 172.21.15.57:0/1088671932 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13c0094d0 con 0x7fd1400c4720 2023-10-30T14:51:41.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd146be3640 1 --2- 172.21.15.57:0/1088671932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1400ca470 0x7fd140157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:51:41.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd145be1640 1 --2- 172.21.15.57:0/1088671932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1400c4720 0x7fd140149d90 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fd13000e840 tx=0x7fd13000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:41.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd136ffd640 1 -- 172.21.15.57:0/1088671932 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd130009510 con 0x7fd1400c4720 2023-10-30T14:51:41.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd136ffd640 1 -- 172.21.15.57:0/1088671932 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13000ee90 con 0x7fd1400c4720 2023-10-30T14:51:41.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd148646640 1 -- 172.21.15.57:0/1088671932 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1400c6170 con 0x7fd1400c4720 2023-10-30T14:51:41.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd136ffd640 1 -- 172.21.15.57:0/1088671932 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13000cb80 con 0x7fd1400c4720 2023-10-30T14:51:41.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd148646640 1 -- 172.21.15.57:0/1088671932 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1401f5be0 con 0x7fd1400c4720 2023-10-30T14:51:41.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.352+0000 7fd148646640 1 -- 172.21.15.57:0/1088671932 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd1400c9c70 con 0x7fd1400c4720 2023-10-30T14:51:41.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.356+0000 7fd136ffd640 1 -- 172.21.15.57:0/1088671932 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd13000cce0 con 0x7fd1400c4720 2023-10-30T14:51:41.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.356+0000 7fd136ffd640 1 --2- 172.21.15.57:0/1088671932 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd1240664d0 0x7fd124068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.356+0000 7fd136ffd640 1 -- 172.21.15.57:0/1088671932 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fd130084530 con 0x7fd1400c4720 2023-10-30T14:51:41.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.356+0000 7fd136ffd640 1 --2- 172.21.15.57:0/1088671932 >> [v2:172.21.15.88:6816/2862111199,v1:172.21.15.88:6817/2862111199] conn(0x7fd12406f670 0x7fd124071af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.356+0000 7fd136ffd640 1 -- 172.21.15.57:0/1088671932 --> [v2:172.21.15.88:6816/2862111199,v1:172.21.15.88:6817/2862111199] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd1240721c0 con 0x7fd12406f670 2023-10-30T14:51:41.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.356+0000 7fd136ffd640 1 -- 172.21.15.57:0/1088671932 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd130086b10 con 0x7fd1400c4720 2023-10-30T14:51:41.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.356+0000 7fd1463e2640 1 --2- 172.21.15.57:0/1088671932 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd1240664d0 0x7fd124068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.356+0000 7fd146be3640 1 --2- 172.21.15.57:0/1088671932 >> [v2:172.21.15.88:6816/2862111199,v1:172.21.15.88:6817/2862111199] conn(0x7fd12406f670 0x7fd124071af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.356+0000 7fd1463e2640 1 --2- 172.21.15.57:0/1088671932 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd1240664d0 0x7fd124068990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd13c0af000 tx=0x7fd13c0b6040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:41.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.356+0000 7fd146be3640 1 --2- 172.21.15.57:0/1088671932 >> [v2:172.21.15.88:6816/2862111199,v1:172.21.15.88:6817/2862111199] conn(0x7fd12406f670 0x7fd124071af0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:41.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.360+0000 7fd136ffd640 1 -- 172.21.15.57:0/1088671932 <== osd.7 v2:172.21.15.88:6816/2862111199 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fd1240721c0 con 0x7fd12406f670 2023-10-30T14:51:41.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.372+0000 7fd148646640 1 -- 172.21.15.57:0/1088671932 --> [v2:172.21.15.88:6816/2862111199,v1:172.21.15.88:6817/2862111199] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd1400725b0 con 0x7fd12406f670 2023-10-30T14:51:41.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.372+0000 7fd136ffd640 1 -- 172.21.15.57:0/1088671932 <== osd.7 v2:172.21.15.88:6816/2862111199 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fd1400725b0 con 0x7fd12406f670 2023-10-30T14:51:41.377 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-30T14:51:41.377 INFO:teuthology.orchestra.run.smithi057.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T14:51:41.377 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-30T14:51:41.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.372+0000 7fd148646640 1 -- 172.21.15.57:0/1088671932 >> [v2:172.21.15.88:6816/2862111199,v1:172.21.15.88:6817/2862111199] conn(0x7fd12406f670 msgr2=0x7fd124071af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.372+0000 7fd148646640 1 --2- 172.21.15.57:0/1088671932 >> [v2:172.21.15.88:6816/2862111199,v1:172.21.15.88:6817/2862111199] conn(0x7fd12406f670 0x7fd124071af0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.372+0000 7fd148646640 1 -- 172.21.15.57:0/1088671932 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd1240664d0 msgr2=0x7fd124068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.372+0000 7fd148646640 1 --2- 172.21.15.57:0/1088671932 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fd1240664d0 0x7fd124068990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd13c0af000 tx=0x7fd13c0b6040 comp rx=0 tx=0).stop 2023-10-30T14:51:41.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.372+0000 7fd148646640 1 -- 172.21.15.57:0/1088671932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1400c4720 msgr2=0x7fd140149d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.372+0000 7fd148646640 1 --2- 172.21.15.57:0/1088671932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1400c4720 0x7fd140149d90 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fd13000e840 tx=0x7fd13000ec40 comp rx=0 tx=0).stop 2023-10-30T14:51:41.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.372+0000 7fd146be3640 1 -- 172.21.15.57:0/1088671932 reap_dead start 2023-10-30T14:51:41.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.372+0000 7fd148646640 1 -- 172.21.15.57:0/1088671932 shutdown_connections 2023-10-30T14:51:41.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.372+0000 7fd148646640 1 -- 172.21.15.57:0/1088671932 >> 172.21.15.57:0/1088671932 conn(0x7fd1400cf1c0 msgr2=0x7fd140150940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:41.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.372+0000 7fd148646640 1 -- 172.21.15.57:0/1088671932 shutdown_connections 2023-10-30T14:51:41.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.372+0000 7fd148646640 1 -- 172.21.15.57:0/1088671932 wait complete. 2023-10-30T14:51:41.395 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-30T14:51:41.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.536+0000 7f34f3805640 1 Processor -- start 2023-10-30T14:51:41.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.536+0000 7f34f3805640 1 -- start start 2023-10-30T14:51:41.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.536+0000 7f34f3805640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34ec0c5ba0 0x7f34ec0c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.536+0000 7f34f3805640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f34ec0c9b40 0x7f34ec1597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.536+0000 7f34f3805640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ec15cda0 con 0x7f34ec0c5f80 2023-10-30T14:51:41.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.536+0000 7f34f3805640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34ec15d990 con 0x7f34ec0c9b40 2023-10-30T14:51:41.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.536+0000 7f34f3805640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ec153f60 con 0x7f34ec0c5ba0 2023-10-30T14:51:41.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.536+0000 7f34f15a1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34ec0c5ba0 0x7f34ec0c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.536+0000 7f34f1da2640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f34ec0c9b40 0x7f34ec1597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.536+0000 7f34f15a1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34ec0c5ba0 0x7f34ec0c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:44476/0 (socket says 172.21.15.57:44476) 2023-10-30T14:51:41.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.536+0000 7f34f15a1640 1 -- 172.21.15.57:0/401398480 learned_addr learned my addr 172.21.15.57:0/401398480 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:41.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.536+0000 7f34f1da2640 1 -- 172.21.15.57:0/401398480 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34ec0c5ba0 msgr2=0x7f34ec0c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:41.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.536+0000 7f34f1da2640 1 --2- 172.21.15.57:0/401398480 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34ec0c5ba0 0x7f34ec0c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.536+0000 7f34f1da2640 1 -- 172.21.15.57:0/401398480 >> v1:172.21.15.88:6789/0 conn(0x7f34ec0c5f80 legacy=0x7f34ec069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.536+0000 7f34f1da2640 1 -- 172.21.15.57:0/401398480 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ec06f490 con 0x7f34ec0c9b40 2023-10-30T14:51:41.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.536+0000 7f34f15a1640 1 --2- 172.21.15.57:0/401398480 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34ec0c5ba0 0x7f34ec0c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:51:41.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.540+0000 7f34f1da2640 1 --2- 172.21.15.57:0/401398480 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34ec0c9b40 0x7f34ec1597b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f34e4009dc0 tx=0x7f34e401c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7fd7ce4c631c10e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:41.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.540+0000 7f34e3fff640 1 -- 172.21.15.57:0/401398480 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e40a57f0 con 0x7f34ec0c9b40 2023-10-30T14:51:41.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.540+0000 7f34e3fff640 1 -- 172.21.15.57:0/401398480 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e40a5950 con 0x7f34ec0c9b40 2023-10-30T14:51:41.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.540+0000 7f34e3fff640 1 -- 172.21.15.57:0/401398480 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e40a5ce0 con 0x7f34ec0c9b40 2023-10-30T14:51:41.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.540+0000 7f34f3805640 1 -- 172.21.15.57:0/401398480 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34ec0c9b40 msgr2=0x7f34ec1597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.540+0000 7f34f3805640 1 --2- 172.21.15.57:0/401398480 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34ec0c9b40 0x7f34ec1597b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f34e4009dc0 tx=0x7f34e401c930 comp rx=0 tx=0).stop 2023-10-30T14:51:41.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.540+0000 7f34f3805640 1 -- 172.21.15.57:0/401398480 shutdown_connections 2023-10-30T14:51:41.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.540+0000 7f34f3805640 1 --2- 172.21.15.57:0/401398480 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34ec0c9b40 0x7f34ec1597b0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.540+0000 7f34f3805640 1 --2- 172.21.15.57:0/401398480 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34ec0c5ba0 0x7f34ec0c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.540+0000 7f34f3805640 1 -- 172.21.15.57:0/401398480 >> 172.21.15.57:0/401398480 conn(0x7f34ec06afa0 msgr2=0x7f34ec06b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:41.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.540+0000 7f34f3805640 1 -- 172.21.15.57:0/401398480 shutdown_connections 2023-10-30T14:51:41.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.540+0000 7f34f3805640 1 -- 172.21.15.57:0/401398480 wait complete. 2023-10-30T14:51:41.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f3805640 1 Processor -- start 2023-10-30T14:51:41.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f3805640 1 -- start start 2023-10-30T14:51:41.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f3805640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34ec0c5ba0 0x7f34ec1575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f3805640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34ec0c5f80 0x7f34ec14cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f3805640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34ec0c9b40 0x7f34ec14fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f3805640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ec14eef0 con 0x7f34ec0c5f80 2023-10-30T14:51:41.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f3805640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ec1f22e0 con 0x7f34ec0c5ba0 2023-10-30T14:51:41.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f3805640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ec1f25a0 con 0x7f34ec0c9b40 2023-10-30T14:51:41.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f0da0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34ec0c5f80 0x7f34ec14cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f0da0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34ec0c5f80 0x7f34ec14cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56888/0 (socket says 172.21.15.57:56888) 2023-10-30T14:51:41.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f0da0640 1 -- 172.21.15.57:0/2690675424 learned_addr learned my addr 172.21.15.57:0/2690675424 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:41.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f15a1640 1 --2- 172.21.15.57:0/2690675424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34ec0c5ba0 0x7f34ec1575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f1da2640 1 --2- 172.21.15.57:0/2690675424 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34ec0c9b40 0x7f34ec14fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f0da0640 1 -- 172.21.15.57:0/2690675424 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34ec0c9b40 msgr2=0x7f34ec14fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f0da0640 1 --2- 172.21.15.57:0/2690675424 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34ec0c9b40 0x7f34ec14fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f0da0640 1 -- 172.21.15.57:0/2690675424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34ec0c5ba0 msgr2=0x7f34ec1575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f0da0640 1 --2- 172.21.15.57:0/2690675424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34ec0c5ba0 0x7f34ec1575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f0da0640 1 -- 172.21.15.57:0/2690675424 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e80094d0 con 0x7f34ec0c5f80 2023-10-30T14:51:41.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f0da0640 1 --2- 172.21.15.57:0/2690675424 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34ec0c5f80 0x7f34ec14cb30 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f34dc00c4d0 tx=0x7f34dc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:41.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34e1ffb640 1 -- 172.21.15.57:0/2690675424 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34dc0052d0 con 0x7f34ec0c5f80 2023-10-30T14:51:41.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34e1ffb640 1 -- 172.21.15.57:0/2690675424 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34dc005430 con 0x7f34ec0c5f80 2023-10-30T14:51:41.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34e1ffb640 1 -- 172.21.15.57:0/2690675424 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34dc012830 con 0x7f34ec0c5f80 2023-10-30T14:51:41.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f3805640 1 -- 172.21.15.57:0/2690675424 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e4009a70 con 0x7f34ec0c5f80 2023-10-30T14:51:41.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.544+0000 7f34f3805640 1 -- 172.21.15.57:0/2690675424 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34ec14f420 con 0x7f34ec0c5f80 2023-10-30T14:51:41.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.548+0000 7f34f3805640 1 -- 172.21.15.57:0/2690675424 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34ec04e170 con 0x7f34ec0c5f80 2023-10-30T14:51:41.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.548+0000 7f34e1ffb640 1 -- 172.21.15.57:0/2690675424 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f34dc012990 con 0x7f34ec0c5f80 2023-10-30T14:51:41.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.548+0000 7f34e1ffb640 1 --2- 172.21.15.57:0/2690675424 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f34d00664d0 0x7f34d0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.548+0000 7f34e1ffb640 1 -- 172.21.15.57:0/2690675424 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f34dc083e80 con 0x7f34ec0c5f80 2023-10-30T14:51:41.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.548+0000 7f34f15a1640 1 --2- 172.21.15.57:0/2690675424 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f34d00664d0 0x7f34d0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.552+0000 7f34f15a1640 1 --2- 172.21.15.57:0/2690675424 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f34d00664d0 0x7f34d0068990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f34e8006cb0 tx=0x7f34e8009240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:41.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.552+0000 7f34e1ffb640 1 -- 172.21.15.57:0/2690675424 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f34dc050d20 con 0x7f34ec0c5f80 2023-10-30T14:51:41.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.708+0000 7f34f3805640 1 -- 172.21.15.57:0/2690675424 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f34ec06f130 con 0x7f34d00664d0 2023-10-30T14:51:41.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.708+0000 7f34e1ffb640 1 -- 172.21.15.57:0/2690675424 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f34e8009960 con 0x7f34d00664d0 2023-10-30T14:51:41.713 INFO:teuthology.orchestra.run.smithi057.stderr:instructed osd(s) 7 to deep-scrub 2023-10-30T14:51:41.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.712+0000 7f34f3805640 1 -- 172.21.15.57:0/2690675424 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f34d00664d0 msgr2=0x7f34d0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.712+0000 7f34f3805640 1 --2- 172.21.15.57:0/2690675424 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f34d00664d0 0x7f34d0068990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f34e8006cb0 tx=0x7f34e8009240 comp rx=0 tx=0).stop 2023-10-30T14:51:41.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.712+0000 7f34f3805640 1 -- 172.21.15.57:0/2690675424 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34ec0c5f80 msgr2=0x7f34ec14cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.712+0000 7f34f3805640 1 --2- 172.21.15.57:0/2690675424 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34ec0c5f80 0x7f34ec14cb30 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f34dc00c4d0 tx=0x7f34dc00c990 comp rx=0 tx=0).stop 2023-10-30T14:51:41.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.712+0000 7f34f3805640 1 -- 172.21.15.57:0/2690675424 shutdown_connections 2023-10-30T14:51:41.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.712+0000 7f34f3805640 1 --2- 172.21.15.57:0/2690675424 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f34d00664d0 0x7f34d0068990 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.712+0000 7f34f3805640 1 --2- 172.21.15.57:0/2690675424 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34ec0c9b40 0x7f34ec14fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.712+0000 7f34f3805640 1 --2- 172.21.15.57:0/2690675424 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34ec0c5f80 0x7f34ec14cb30 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.712+0000 7f34f3805640 1 --2- 172.21.15.57:0/2690675424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34ec0c5ba0 0x7f34ec1575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.712+0000 7f34f3805640 1 -- 172.21.15.57:0/2690675424 >> 172.21.15.57:0/2690675424 conn(0x7f34ec06afa0 msgr2=0x7f34ec0717f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:41.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.712+0000 7f34f3805640 1 -- 172.21.15.57:0/2690675424 shutdown_connections 2023-10-30T14:51:41.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.712+0000 7f34f3805640 1 -- 172.21.15.57:0/2690675424 wait complete. 2023-10-30T14:51:41.734 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:51:41.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.872+0000 7fcdf1a91640 1 Processor -- start 2023-10-30T14:51:41.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.872+0000 7fcdf1a91640 1 -- start start 2023-10-30T14:51:41.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdf1a91640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcdec0701a0 0x7fcdec15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdf1a91640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdec0c2aa0 con 0x7fcdec06b2f0 2023-10-30T14:51:41.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdf1a91640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdec0708e0 con 0x7fcdec1537f0 2023-10-30T14:51:41.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdf1a91640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec0c3ab0 con 0x7fcdec0701a0 2023-10-30T14:51:41.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdebfff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fcdec06b2f0 0x7fcdec1641e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:41926/0 (socket says 172.21.15.57:41926) 2023-10-30T14:51:41.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdebfff640 1 -- 172.21.15.57:0/2378755792 learned_addr learned my addr 172.21.15.57:0/2378755792 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:51:41.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdeaffd640 1 --2- 172.21.15.57:0/2378755792 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcdec0701a0 0x7fcdec15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdea7fc640 1 -- 172.21.15.57:0/2378755792 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448489787 0 0) 0x7fcdec0c2aa0 con 0x7fcdec06b2f0 2023-10-30T14:51:41.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdea7fc640 1 -- 172.21.15.57:0/2378755792 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdd4003530 con 0x7fcdec06b2f0 2023-10-30T14:51:41.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdea7fc640 1 -- 172.21.15.57:0/2378755792 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3096203183 0 0) 0x7fcdd4003530 con 0x7fcdec06b2f0 2023-10-30T14:51:41.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdea7fc640 1 -- 172.21.15.57:0/2378755792 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcdec0701a0 msgr2=0x7fcdec15a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:41.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdea7fc640 1 --2- 172.21.15.57:0/2378755792 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcdec0701a0 0x7fcdec15a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdea7fc640 1 -- 172.21.15.57:0/2378755792 >> v1:172.21.15.88:6789/0 conn(0x7fcdec1537f0 legacy=0x7fcdec0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdea7fc640 1 -- 172.21.15.57:0/2378755792 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec0c1eb0 con 0x7fcdec06b2f0 2023-10-30T14:51:41.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdea7fc640 1 -- 172.21.15.57:0/2378755792 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fcde4001a00 con 0x7fcdec06b2f0 2023-10-30T14:51:41.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdea7fc640 1 -- 172.21.15.57:0/2378755792 >> v1:172.21.15.57:6789/0 conn(0x7fcdec06b2f0 legacy=0x7fcdec1641e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdea7fc640 1 --2- 172.21.15.57:0/2378755792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdd40039b0 0x7fcdd4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdea7fc640 1 --2- 172.21.15.57:0/2378755792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdd4004360 0x7fcdd4004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdea7fc640 1 --2- 172.21.15.57:0/2378755792 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcdec0701a0 0x7fcdd4005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdea7fc640 1 -- 172.21.15.57:0/2378755792 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd400b990 con 0x7fcdd40039b0 2023-10-30T14:51:41.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdea7fc640 1 -- 172.21.15.57:0/2378755792 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd400bb00 con 0x7fcdd4004360 2023-10-30T14:51:41.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdea7fc640 1 -- 172.21.15.57:0/2378755792 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd400bcc0 con 0x7fcdec0701a0 2023-10-30T14:51:41.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdeb7fe640 1 --2- 172.21.15.57:0/2378755792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdd40039b0 0x7fcdd4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdeaffd640 1 --2- 172.21.15.57:0/2378755792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdd4004360 0x7fcdd4004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdebfff640 1 --2- 172.21.15.57:0/2378755792 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcdec0701a0 0x7fcdd4005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdeb7fe640 1 -- 172.21.15.57:0/2378755792 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcdec0701a0 msgr2=0x7fcdd4005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:41.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdeb7fe640 1 --2- 172.21.15.57:0/2378755792 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcdec0701a0 0x7fcdd4005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdeb7fe640 1 -- 172.21.15.57:0/2378755792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdd4004360 msgr2=0x7fcdd4004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:51:41.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdeb7fe640 1 --2- 172.21.15.57:0/2378755792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdd4004360 0x7fcdd4004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.876+0000 7fcdeb7fe640 1 -- 172.21.15.57:0/2378755792 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd4003530 con 0x7fcdd40039b0 2023-10-30T14:51:41.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.880+0000 7fcdeb7fe640 1 --2- 172.21.15.57:0/2378755792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdd40039b0 0x7fcdd4003d90 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fcddc001860 tx=0x7fcddc01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6e943c87d209e64 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:41.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.880+0000 7fcdea7fc640 1 -- 172.21.15.57:0/2378755792 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc0a8890 con 0x7fcdd40039b0 2023-10-30T14:51:41.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.880+0000 7fcdea7fc640 1 -- 172.21.15.57:0/2378755792 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcddc0a89f0 con 0x7fcdd40039b0 2023-10-30T14:51:41.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.880+0000 7fcdea7fc640 1 -- 172.21.15.57:0/2378755792 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc0a8da0 con 0x7fcdd40039b0 2023-10-30T14:51:41.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.880+0000 7fcdf1a91640 1 -- 172.21.15.57:0/2378755792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdd40039b0 msgr2=0x7fcdd4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.880+0000 7fcdf1a91640 1 --2- 172.21.15.57:0/2378755792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdd40039b0 0x7fcdd4003d90 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fcddc001860 tx=0x7fcddc01ec70 comp rx=0 tx=0).stop 2023-10-30T14:51:41.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.880+0000 7fcdebfff640 1 -- 172.21.15.57:0/2378755792 reap_dead start 2023-10-30T14:51:41.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.880+0000 7fcdf1a91640 1 -- 172.21.15.57:0/2378755792 shutdown_connections 2023-10-30T14:51:41.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.880+0000 7fcdf1a91640 1 -- 172.21.15.57:0/2378755792 >> 172.21.15.57:0/2378755792 conn(0x7fcdec071fb0 msgr2=0x7fcdec0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:41.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.880+0000 7fcdf1a91640 1 -- 172.21.15.57:0/2378755792 shutdown_connections 2023-10-30T14:51:41.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.880+0000 7fcdf1a91640 1 -- 172.21.15.57:0/2378755792 wait complete. 2023-10-30T14:51:41.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.880+0000 7fcdf1a91640 1 Processor -- start 2023-10-30T14:51:41.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdf1a91640 1 -- start start 2023-10-30T14:51:41.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdf1a91640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdd40039b0 0x7fcdec0c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdf1a91640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdec0c42d0 0x7fcdec1522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdf1a91640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcdec159480 0x7fcdec1f21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdf1a91640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec159c50 con 0x7fcdec0c42d0 2023-10-30T14:51:41.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdf1a91640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec1efb60 con 0x7fcdd40039b0 2023-10-30T14:51:41.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdf1a91640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec0c50a0 con 0x7fcdec159480 2023-10-30T14:51:41.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdeaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdec0c42d0 0x7fcdec1522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdeaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdec0c42d0 0x7fcdec1522e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56916/0 (socket says 172.21.15.57:56916) 2023-10-30T14:51:41.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdeaffd640 1 -- 172.21.15.57:0/795955108 learned_addr learned my addr 172.21.15.57:0/795955108 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:51:41.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdeb7fe640 1 --2- 172.21.15.57:0/795955108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdd40039b0 0x7fcdec0c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdebfff640 1 --2- 172.21.15.57:0/795955108 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcdec159480 0x7fcdec1f21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdeaffd640 1 -- 172.21.15.57:0/795955108 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcdec159480 msgr2=0x7fcdec1f21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdeaffd640 1 --2- 172.21.15.57:0/795955108 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcdec159480 0x7fcdec1f21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdeaffd640 1 -- 172.21.15.57:0/795955108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdd40039b0 msgr2=0x7fcdec0c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:41.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdeaffd640 1 --2- 172.21.15.57:0/795955108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdd40039b0 0x7fcdec0c4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:41.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdeaffd640 1 -- 172.21.15.57:0/795955108 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec159970 con 0x7fcdec0c42d0 2023-10-30T14:51:41.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdeaffd640 1 --2- 172.21.15.57:0/795955108 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdec0c42d0 0x7fcdec1522e0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fcde00119d0 tx=0x7fcde0011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:41.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdd3fff640 1 -- 172.21.15.57:0/795955108 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde000fb50 con 0x7fcdec0c42d0 2023-10-30T14:51:41.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdd3fff640 1 -- 172.21.15.57:0/795955108 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde000fcb0 con 0x7fcdec0c42d0 2023-10-30T14:51:41.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdd3fff640 1 -- 172.21.15.57:0/795955108 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde0014440 con 0x7fcdec0c42d0 2023-10-30T14:51:41.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdf1a91640 1 -- 172.21.15.57:0/795955108 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdec1eeda0 con 0x7fcdec0c42d0 2023-10-30T14:51:41.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdf1a91640 1 -- 172.21.15.57:0/795955108 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdec1ef360 con 0x7fcdec0c42d0 2023-10-30T14:51:41.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.884+0000 7fcdf1a91640 1 -- 172.21.15.57:0/795955108 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdec04e170 con 0x7fcdec0c42d0 2023-10-30T14:51:41.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.888+0000 7fcdd3fff640 1 -- 172.21.15.57:0/795955108 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcde00145a0 con 0x7fcdec0c42d0 2023-10-30T14:51:41.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.888+0000 7fcdd3fff640 1 --2- 172.21.15.57:0/795955108 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcdd80664d0 0x7fcdd8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:51:41.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.888+0000 7fcdd3fff640 1 -- 172.21.15.57:0/795955108 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fcde0084da0 con 0x7fcdec0c42d0 2023-10-30T14:51:41.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.888+0000 7fcdeb7fe640 1 --2- 172.21.15.57:0/795955108 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcdd80664d0 0x7fcdd8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:51:41.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.888+0000 7fcdeb7fe640 1 --2- 172.21.15.57:0/795955108 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcdd80664d0 0x7fcdd8068990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fcddc0b2000 tx=0x7fcddc0aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:51:41.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:41.892+0000 7fcdd3fff640 1 -- 172.21.15.57:0/795955108 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcde0051bc0 con 0x7fcdec0c42d0 2023-10-30T14:51:42.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:42.048+0000 7fcdf1a91640 1 -- 172.21.15.57:0/795955108 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcdec0c2450 con 0x7fcdd80664d0 2023-10-30T14:51:42.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:42.072+0000 7fcdd3fff640 1 -- 172.21.15.57:0/795955108 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+293506 (secure 0 0 0) 0x7fcddc0067e0 con 0x7fcdd80664d0 2023-10-30T14:51:42.077 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:51:42.078 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-30T14:51:42.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:42.072+0000 7fcdf1a91640 1 -- 172.21.15.57:0/795955108 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcdd80664d0 msgr2=0x7fcdd8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:42.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:42.072+0000 7fcdf1a91640 1 --2- 172.21.15.57:0/795955108 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcdd80664d0 0x7fcdd8068990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fcddc0b2000 tx=0x7fcddc0aa070 comp rx=0 tx=0).stop 2023-10-30T14:51:42.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:42.072+0000 7fcdf1a91640 1 -- 172.21.15.57:0/795955108 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdec0c42d0 msgr2=0x7fcdec1522e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:51:42.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:42.072+0000 7fcdf1a91640 1 --2- 172.21.15.57:0/795955108 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdec0c42d0 0x7fcdec1522e0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fcde00119d0 tx=0x7fcde0011e90 comp rx=0 tx=0).stop 2023-10-30T14:51:42.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:42.072+0000 7fcdf1a91640 1 -- 172.21.15.57:0/795955108 shutdown_connections 2023-10-30T14:51:42.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:42.072+0000 7fcdf1a91640 1 --2- 172.21.15.57:0/795955108 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcdd80664d0 0x7fcdd8068990 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:42.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:42.072+0000 7fcdf1a91640 1 --2- 172.21.15.57:0/795955108 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcdec159480 0x7fcdec1f21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:42.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:42.072+0000 7fcdf1a91640 1 --2- 172.21.15.57:0/795955108 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdec0c42d0 0x7fcdec1522e0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:42.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:42.072+0000 7fcdf1a91640 1 --2- 172.21.15.57:0/795955108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdd40039b0 0x7fcdec0c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:51:42.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:42.072+0000 7fcdf1a91640 1 -- 172.21.15.57:0/795955108 >> 172.21.15.57:0/795955108 conn(0x7fcdec071fb0 msgr2=0x7fcdec14fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:51:42.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:42.076+0000 7fcdf1a91640 1 -- 172.21.15.57:0/795955108 shutdown_connections 2023-10-30T14:51:42.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:51:42.076+0000 7fcdf1a91640 1 -- 172.21.15.57:0/795955108 wait complete. 2023-10-30T14:51:42.098 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":1195,"stamp":"2023-10-30T14:51:41.319848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1743967717,"num_objects":33080,"num_object_clones":14708,"num_object_copies":66160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33080,"num_whiteouts":12864,"num_read":10750,"num_read_kb":3497830,"num_write":98282,"num_write_kb":2874363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1128,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110403,"ondisk_log_size":110403,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":4520388,"kb_used_data":3487108,"kb_used_omap":12,"kb_used_meta":1033203,"kb_avail":750454332,"statfs":{"total":773094113280,"available":768465235968,"internally_reserved":0,"allocated":3570798592,"data_stored":3492635900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":1058000441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,10,3,5,33,26,16,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":83,"apply_latency_ms":83,"commit_latency_ns":83000000,"apply_latency_ns":83000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36593402,"num_objects":493,"num_object_clones":228,"num_object_copies":986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":493,"num_whiteouts":116,"num_read":25,"num_read_kb":4933,"num_write":1172,"num_write_kb":36475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004300"},"pg_stats":[{"pgid":"3.3f","version":"67'1292","reported_seq":3254,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.636670+0000","last_change":"2023-10-30T14:23:31.082534+0000","last_active":"2023-10-30T14:51:34.636670+0000","last_peered":"2023-10-30T14:51:34.636670+0000","last_clean":"2023-10-30T14:51:34.636670+0000","last_became_active":"2023-10-30T14:14:04.650882+0000","last_became_peered":"2023-10-30T14:14:04.650882+0000","last_unstale":"2023-10-30T14:51:34.636670+0000","last_undegraded":"2023-10-30T14:51:34.636670+0000","last_fullsized":"2023-10-30T14:51:34.636670+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1292,"log_dups_size":0,"ondisk_log_size":1292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:46:41.132286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010330941,"stat_sum":{"num_bytes":16885684,"num_objects":463,"num_object_clones":217,"num_object_copies":926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":463,"num_whiteouts":185,"num_read":62,"num_read_kb":8246,"num_write":1079,"num_write_kb":18499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"67'1284","reported_seq":3239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.598421+0000","last_change":"2023-10-30T14:23:30.771600+0000","last_active":"2023-10-30T14:51:34.598421+0000","last_peered":"2023-10-30T14:51:34.598421+0000","last_clean":"2023-10-30T14:51:34.598421+0000","last_became_active":"2023-10-30T14:14:04.587942+0000","last_became_peered":"2023-10-30T14:14:04.587942+0000","last_unstale":"2023-10-30T14:51:34.598421+0000","last_undegraded":"2023-10-30T14:51:34.598421+0000","last_fullsized":"2023-10-30T14:51:34.598421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1284,"log_dups_size":0,"ondisk_log_size":1284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:54:24.389508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002850184,"stat_sum":{"num_bytes":13093874,"num_objects":485,"num_object_clones":221,"num_object_copies":970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":196,"num_read":59,"num_read_kb":8220,"num_write":1070,"num_write_kb":14962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"67'1307","reported_seq":3270,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.601860+0000","last_change":"2023-10-30T14:23:30.776754+0000","last_active":"2023-10-30T14:51:34.601860+0000","last_peered":"2023-10-30T14:51:34.601860+0000","last_clean":"2023-10-30T14:51:34.601860+0000","last_became_active":"2023-10-30T14:14:04.590700+0000","last_became_peered":"2023-10-30T14:14:04.590700+0000","last_unstale":"2023-10-30T14:51:34.601860+0000","last_undegraded":"2023-10-30T14:51:34.601860+0000","last_fullsized":"2023-10-30T14:51:34.601860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1307,"log_dups_size":0,"ondisk_log_size":1307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:11:25.956171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042861139999999997,"stat_sum":{"num_bytes":12610897,"num_objects":491,"num_object_clones":227,"num_object_copies":982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":204,"num_read":73,"num_read_kb":16409,"num_write":1065,"num_write_kb":22118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"67'1308","reported_seq":3268,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.602504+0000","last_change":"2023-10-30T14:23:31.067598+0000","last_active":"2023-10-30T14:51:34.602504+0000","last_peered":"2023-10-30T14:51:34.602504+0000","last_clean":"2023-10-30T14:51:34.602504+0000","last_became_active":"2023-10-30T14:14:45.661677+0000","last_became_peered":"2023-10-30T14:14:45.661677+0000","last_unstale":"2023-10-30T14:51:34.602504+0000","last_undegraded":"2023-10-30T14:51:34.602504+0000","last_fullsized":"2023-10-30T14:51:34.602504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1308,"log_dups_size":0,"ondisk_log_size":1308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:33.531763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003131572,"stat_sum":{"num_bytes":8809488,"num_objects":483,"num_object_clones":217,"num_object_copies":966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":483,"num_whiteouts":190,"num_read":63,"num_read_kb":69,"num_write":1091,"num_write_kb":15023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"67'1316","reported_seq":3276,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.619277+0000","last_change":"2023-10-30T14:23:31.072132+0000","last_active":"2023-10-30T14:51:34.619277+0000","last_peered":"2023-10-30T14:51:34.619277+0000","last_clean":"2023-10-30T14:51:34.619277+0000","last_became_active":"2023-10-30T14:14:04.644683+0000","last_became_peered":"2023-10-30T14:14:04.644683+0000","last_unstale":"2023-10-30T14:51:34.619277+0000","last_undegraded":"2023-10-30T14:51:34.619277+0000","last_fullsized":"2023-10-30T14:51:34.619277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1316,"log_dups_size":0,"ondisk_log_size":1316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:39.271356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052462430000000003,"stat_sum":{"num_bytes":8863750,"num_objects":486,"num_object_clones":224,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":200,"num_read":70,"num_read_kb":4116,"num_write":1084,"num_write_kb":10111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'1291","reported_seq":3259,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.597626+0000","last_change":"2023-10-30T14:23:30.777057+0000","last_active":"2023-10-30T14:51:34.597626+0000","last_peered":"2023-10-30T14:51:34.597626+0000","last_clean":"2023-10-30T14:51:34.597626+0000","last_became_active":"2023-10-30T14:14:04.619335+0000","last_became_peered":"2023-10-30T14:14:04.619335+0000","last_unstale":"2023-10-30T14:51:34.597626+0000","last_undegraded":"2023-10-30T14:51:34.597626+0000","last_fullsized":"2023-10-30T14:51:34.597626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1291,"log_dups_size":0,"ondisk_log_size":1291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:16:48.356144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062283859999999998,"stat_sum":{"num_bytes":20638046,"num_objects":479,"num_object_clones":221,"num_object_copies":958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":195,"num_read":75,"num_read_kb":12254,"num_write":1080,"num_write_kb":22208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"67'1299","reported_seq":3248,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.633250+0000","last_change":"2023-10-30T14:23:31.080820+0000","last_active":"2023-10-30T14:51:34.633250+0000","last_peered":"2023-10-30T14:51:34.633250+0000","last_clean":"2023-10-30T14:51:34.633250+0000","last_became_active":"2023-10-30T14:14:04.641323+0000","last_became_peered":"2023-10-30T14:14:04.641323+0000","last_unstale":"2023-10-30T14:51:34.633250+0000","last_undegraded":"2023-10-30T14:51:34.633250+0000","last_fullsized":"2023-10-30T14:51:34.633250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1299,"log_dups_size":0,"ondisk_log_size":1299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:11.764013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011228897,"stat_sum":{"num_bytes":14710085,"num_objects":493,"num_object_clones":224,"num_object_copies":986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":493,"num_whiteouts":202,"num_read":59,"num_read_kb":8203,"num_write":1083,"num_write_kb":18203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'1249","reported_seq":3205,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.510182+0000","last_change":"2023-10-30T14:23:31.067811+0000","last_active":"2023-10-30T14:51:34.510182+0000","last_peered":"2023-10-30T14:51:34.510182+0000","last_clean":"2023-10-30T14:51:34.510182+0000","last_became_active":"2023-10-30T14:14:04.618312+0000","last_became_peered":"2023-10-30T14:14:04.618312+0000","last_unstale":"2023-10-30T14:51:34.510182+0000","last_undegraded":"2023-10-30T14:51:34.510182+0000","last_fullsized":"2023-10-30T14:51:34.510182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1249,"log_dups_size":0,"ondisk_log_size":1249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:57:22.752126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002849426,"stat_sum":{"num_bytes":12938947,"num_objects":477,"num_object_clones":218,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":194,"num_read":60,"num_read_kb":8209,"num_write":1031,"num_write_kb":15209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"67'1347","reported_seq":3294,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.612750+0000","last_change":"2023-10-30T14:23:31.083143+0000","last_active":"2023-10-30T14:51:34.612750+0000","last_peered":"2023-10-30T14:51:34.612750+0000","last_clean":"2023-10-30T14:51:34.612750+0000","last_became_active":"2023-10-30T14:14:04.647361+0000","last_became_peered":"2023-10-30T14:14:04.647361+0000","last_unstale":"2023-10-30T14:51:34.612750+0000","last_undegraded":"2023-10-30T14:51:34.612750+0000","last_fullsized":"2023-10-30T14:51:34.612750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1347,"log_dups_size":0,"ondisk_log_size":1347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:06:53.616905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010459703000000001,"stat_sum":{"num_bytes":8547707,"num_objects":513,"num_object_clones":237,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":513,"num_whiteouts":209,"num_read":70,"num_read_kb":8267,"num_write":1124,"num_write_kb":13960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"67'1383","reported_seq":3284,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.629891+0000","last_change":"2023-10-30T14:23:31.075673+0000","last_active":"2023-10-30T14:51:34.629891+0000","last_peered":"2023-10-30T14:51:34.629891+0000","last_clean":"2023-10-30T14:51:34.629891+0000","last_became_active":"2023-10-30T14:14:04.630959+0000","last_became_peered":"2023-10-30T14:14:04.630959+0000","last_unstale":"2023-10-30T14:51:34.629891+0000","last_undegraded":"2023-10-30T14:51:34.629891+0000","last_fullsized":"2023-10-30T14:51:34.629891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1383,"log_dups_size":0,"ondisk_log_size":1383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:36.246017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079325130000000004,"stat_sum":{"num_bytes":12417236,"num_objects":545,"num_object_clones":289,"num_object_copies":1090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":545,"num_whiteouts":194,"num_read":76,"num_read_kb":4189,"num_write":1096,"num_write_kb":14000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"67'1303","reported_seq":3245,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.390151+0000","last_change":"2023-10-30T14:23:31.075101+0000","last_active":"2023-10-30T14:51:34.390151+0000","last_peered":"2023-10-30T14:51:34.390151+0000","last_clean":"2023-10-30T14:51:34.390151+0000","last_became_active":"2023-10-30T14:14:04.612653+0000","last_became_peered":"2023-10-30T14:14:04.612653+0000","last_unstale":"2023-10-30T14:51:34.390151+0000","last_undegraded":"2023-10-30T14:51:34.390151+0000","last_fullsized":"2023-10-30T14:51:34.390151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1303,"log_dups_size":0,"ondisk_log_size":1303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:34:44.855573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079098320000000003,"stat_sum":{"num_bytes":17851951,"num_objects":522,"num_object_clones":235,"num_object_copies":1044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":204,"num_read":63,"num_read_kb":9809,"num_write":1080,"num_write_kb":19222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"67'1293","reported_seq":3247,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.621215+0000","last_change":"2023-10-30T14:23:30.957983+0000","last_active":"2023-10-30T14:51:34.621215+0000","last_peered":"2023-10-30T14:51:34.621215+0000","last_clean":"2023-10-30T14:51:34.621215+0000","last_became_active":"2023-10-30T14:14:04.621135+0000","last_became_peered":"2023-10-30T14:14:04.621135+0000","last_unstale":"2023-10-30T14:51:34.621215+0000","last_undegraded":"2023-10-30T14:51:34.621215+0000","last_fullsized":"2023-10-30T14:51:34.621215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1293,"log_dups_size":0,"ondisk_log_size":1293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:04:43.303589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025042409999999999,"stat_sum":{"num_bytes":14360375,"num_objects":506,"num_object_clones":227,"num_object_copies":1012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":195,"num_read":67,"num_read_kb":6339,"num_write":1082,"num_write_kb":20012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"67'1280","reported_seq":3262,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.428666+0000","last_change":"2023-10-30T14:23:31.070450+0000","last_active":"2023-10-30T14:51:34.428666+0000","last_peered":"2023-10-30T14:51:34.428666+0000","last_clean":"2023-10-30T14:51:34.428666+0000","last_became_active":"2023-10-30T14:14:04.616142+0000","last_became_peered":"2023-10-30T14:14:04.616142+0000","last_unstale":"2023-10-30T14:51:34.428666+0000","last_undegraded":"2023-10-30T14:51:34.428666+0000","last_fullsized":"2023-10-30T14:51:34.428666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1280,"log_dups_size":0,"ondisk_log_size":1280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:46:20.636160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049623779999999999,"stat_sum":{"num_bytes":9254692,"num_objects":486,"num_object_clones":220,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":190,"num_read":88,"num_read_kb":8286,"num_write":1073,"num_write_kb":19495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"67'1319","reported_seq":3293,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.579191+0000","last_change":"2023-10-30T14:23:30.775687+0000","last_active":"2023-10-30T14:51:34.579191+0000","last_peered":"2023-10-30T14:51:34.579191+0000","last_clean":"2023-10-30T14:51:34.579191+0000","last_became_active":"2023-10-30T14:14:04.588452+0000","last_became_peered":"2023-10-30T14:14:04.588452+0000","last_unstale":"2023-10-30T14:51:34.579191+0000","last_undegraded":"2023-10-30T14:51:34.579191+0000","last_fullsized":"2023-10-30T14:51:34.579191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1319,"log_dups_size":0,"ondisk_log_size":1319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:40:26.637820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046351760000000004,"stat_sum":{"num_bytes":12959775,"num_objects":471,"num_object_clones":217,"num_object_copies":942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":471,"num_whiteouts":194,"num_read":76,"num_read_kb":7792,"num_write":1107,"num_write_kb":17259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"67'1323","reported_seq":3287,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.357896+0000","last_change":"2023-10-30T14:23:30.770743+0000","last_active":"2023-10-30T14:51:34.357896+0000","last_peered":"2023-10-30T14:51:34.357896+0000","last_clean":"2023-10-30T14:51:34.357896+0000","last_became_active":"2023-10-30T14:14:04.620781+0000","last_became_peered":"2023-10-30T14:14:04.620781+0000","last_unstale":"2023-10-30T14:51:34.357896+0000","last_undegraded":"2023-10-30T14:51:34.357896+0000","last_fullsized":"2023-10-30T14:51:34.357896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1323,"log_dups_size":0,"ondisk_log_size":1323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:09.260750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002316706,"stat_sum":{"num_bytes":21445753,"num_objects":489,"num_object_clones":220,"num_object_copies":978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":196,"num_read":69,"num_read_kb":13296,"num_write":1109,"num_write_kb":27203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"67'1315","reported_seq":3296,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.626407+0000","last_change":"2023-10-30T14:23:30.774013+0000","last_active":"2023-10-30T14:51:34.626407+0000","last_peered":"2023-10-30T14:51:34.626407+0000","last_clean":"2023-10-30T14:51:34.626407+0000","last_became_active":"2023-10-30T14:14:04.608190+0000","last_became_peered":"2023-10-30T14:14:04.608190+0000","last_unstale":"2023-10-30T14:51:34.626407+0000","last_undegraded":"2023-10-30T14:51:34.626407+0000","last_fullsized":"2023-10-30T14:51:34.626407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1315,"log_dups_size":0,"ondisk_log_size":1315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:44:40.113536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035618450000000001,"stat_sum":{"num_bytes":31901819,"num_objects":484,"num_object_clones":219,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":197,"num_read":85,"num_read_kb":23305,"num_write":1093,"num_write_kb":37349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"67'1361","reported_seq":3320,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.400557+0000","last_change":"2023-10-30T14:23:30.779367+0000","last_active":"2023-10-30T14:51:34.400557+0000","last_peered":"2023-10-30T14:51:34.400557+0000","last_clean":"2023-10-30T14:51:34.400557+0000","last_became_active":"2023-10-30T14:14:04.641306+0000","last_became_peered":"2023-10-30T14:14:04.641306+0000","last_unstale":"2023-10-30T14:51:34.400557+0000","last_undegraded":"2023-10-30T14:51:34.400557+0000","last_fullsized":"2023-10-30T14:51:34.400557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1361,"log_dups_size":0,"ondisk_log_size":1361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:53:12.985530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066189990000000004,"stat_sum":{"num_bytes":4095483,"num_objects":513,"num_object_clones":236,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":513,"num_whiteouts":206,"num_read":80,"num_read_kb":4151,"num_write":1131,"num_write_kb":9615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"67'1244","reported_seq":3224,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.632843+0000","last_change":"2023-10-30T14:23:31.067340+0000","last_active":"2023-10-30T14:51:34.632843+0000","last_peered":"2023-10-30T14:51:34.632843+0000","last_clean":"2023-10-30T14:51:34.632843+0000","last_became_active":"2023-10-30T14:14:04.613644+0000","last_became_peered":"2023-10-30T14:14:04.613644+0000","last_unstale":"2023-10-30T14:51:34.632843+0000","last_undegraded":"2023-10-30T14:51:34.632843+0000","last_fullsized":"2023-10-30T14:51:34.632843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1244,"log_dups_size":0,"ondisk_log_size":1244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:50:40.405415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030699740000000001,"stat_sum":{"num_bytes":12521692,"num_objects":445,"num_object_clones":206,"num_object_copies":890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":445,"num_whiteouts":180,"num_read":71,"num_read_kb":8233,"num_write":1044,"num_write_kb":14252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"67'1303","reported_seq":3274,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.612554+0000","last_change":"2023-10-30T14:23:31.082413+0000","last_active":"2023-10-30T14:51:34.612554+0000","last_peered":"2023-10-30T14:51:34.612554+0000","last_clean":"2023-10-30T14:51:34.612554+0000","last_became_active":"2023-10-30T14:14:04.649850+0000","last_became_peered":"2023-10-30T14:14:04.649850+0000","last_unstale":"2023-10-30T14:51:34.612554+0000","last_undegraded":"2023-10-30T14:51:34.612554+0000","last_fullsized":"2023-10-30T14:51:34.612554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1303,"log_dups_size":0,"ondisk_log_size":1303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:27:56.738513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012599246,"stat_sum":{"num_bytes":26075472,"num_objects":477,"num_object_clones":221,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":193,"num_read":77,"num_read_kb":24658,"num_write":1080,"num_write_kb":32086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"67'1349","reported_seq":3283,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.513372+0000","last_change":"2023-10-30T14:23:31.074724+0000","last_active":"2023-10-30T14:51:34.513372+0000","last_peered":"2023-10-30T14:51:34.513372+0000","last_clean":"2023-10-30T14:51:34.513372+0000","last_became_active":"2023-10-30T14:14:04.625042+0000","last_became_peered":"2023-10-30T14:14:04.625042+0000","last_unstale":"2023-10-30T14:51:34.513372+0000","last_undegraded":"2023-10-30T14:51:34.513372+0000","last_fullsized":"2023-10-30T14:51:34.513372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1349,"log_dups_size":0,"ondisk_log_size":1349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:45.303841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082351539999999997,"stat_sum":{"num_bytes":12784426,"num_objects":514,"num_object_clones":242,"num_object_copies":1028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":514,"num_whiteouts":214,"num_read":61,"num_read_kb":10278,"num_write":1107,"num_write_kb":27149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"67'1278","reported_seq":3242,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.429500+0000","last_change":"2023-10-30T14:23:30.775419+0000","last_active":"2023-10-30T14:51:34.429500+0000","last_peered":"2023-10-30T14:51:34.429500+0000","last_clean":"2023-10-30T14:51:34.429500+0000","last_became_active":"2023-10-30T14:14:04.633586+0000","last_became_peered":"2023-10-30T14:14:04.633586+0000","last_unstale":"2023-10-30T14:51:34.429500+0000","last_undegraded":"2023-10-30T14:51:34.429500+0000","last_fullsized":"2023-10-30T14:51:34.429500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1278,"log_dups_size":0,"ondisk_log_size":1278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:19:06.449779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043616920000000003,"stat_sum":{"num_bytes":12342414,"num_objects":462,"num_object_clones":214,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":462,"num_whiteouts":191,"num_read":63,"num_read_kb":8213,"num_write":1061,"num_write_kb":14018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"67'1382","reported_seq":3353,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.405542+0000","last_change":"2023-10-30T14:23:30.777885+0000","last_active":"2023-10-30T14:51:34.405542+0000","last_peered":"2023-10-30T14:51:34.405542+0000","last_clean":"2023-10-30T14:51:34.405542+0000","last_became_active":"2023-10-30T14:14:04.642244+0000","last_became_peered":"2023-10-30T14:14:04.642244+0000","last_unstale":"2023-10-30T14:51:34.405542+0000","last_undegraded":"2023-10-30T14:51:34.405542+0000","last_fullsized":"2023-10-30T14:51:34.405542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1382,"log_dups_size":0,"ondisk_log_size":1382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:25.186063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065088710000000003,"stat_sum":{"num_bytes":26075380,"num_objects":515,"num_object_clones":234,"num_object_copies":1030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":515,"num_whiteouts":211,"num_read":90,"num_read_kb":18405,"num_write":1133,"num_write_kb":31429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'1310","reported_seq":3291,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.509400+0000","last_change":"2023-10-30T14:23:31.070902+0000","last_active":"2023-10-30T14:51:34.509400+0000","last_peered":"2023-10-30T14:51:34.509400+0000","last_clean":"2023-10-30T14:51:34.509400+0000","last_became_active":"2023-10-30T14:14:04.620343+0000","last_became_peered":"2023-10-30T14:14:04.620343+0000","last_unstale":"2023-10-30T14:51:34.509400+0000","last_undegraded":"2023-10-30T14:51:34.509400+0000","last_fullsized":"2023-10-30T14:51:34.509400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1310,"log_dups_size":0,"ondisk_log_size":1310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:45:58.152823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048974149999999996,"stat_sum":{"num_bytes":12481557,"num_objects":484,"num_object_clones":220,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":200,"num_read":86,"num_read_kb":8232,"num_write":1085,"num_write_kb":18010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'1363","reported_seq":3306,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.582590+0000","last_change":"2023-10-30T14:23:30.774642+0000","last_active":"2023-10-30T14:51:34.582590+0000","last_peered":"2023-10-30T14:51:34.582590+0000","last_clean":"2023-10-30T14:51:34.582590+0000","last_became_active":"2023-10-30T14:14:04.609720+0000","last_became_peered":"2023-10-30T14:14:04.609720+0000","last_unstale":"2023-10-30T14:51:34.582590+0000","last_undegraded":"2023-10-30T14:51:34.582590+0000","last_fullsized":"2023-10-30T14:51:34.582590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1363,"log_dups_size":0,"ondisk_log_size":1363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:10:58.430266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040008739999999997,"stat_sum":{"num_bytes":31023805,"num_objects":535,"num_object_clones":248,"num_object_copies":1070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":535,"num_whiteouts":230,"num_read":75,"num_read_kb":20512,"num_write":1099,"num_write_kb":32005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"67'1218","reported_seq":3196,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.509946+0000","last_change":"2023-10-30T14:23:31.069745+0000","last_active":"2023-10-30T14:51:34.509946+0000","last_peered":"2023-10-30T14:51:34.509946+0000","last_clean":"2023-10-30T14:51:34.509946+0000","last_became_active":"2023-10-30T14:14:04.620217+0000","last_became_peered":"2023-10-30T14:14:04.620217+0000","last_unstale":"2023-10-30T14:51:34.509946+0000","last_undegraded":"2023-10-30T14:51:34.509946+0000","last_fullsized":"2023-10-30T14:51:34.509946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1218,"log_dups_size":0,"ondisk_log_size":1218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:36:29.759579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044491130000000002,"stat_sum":{"num_bytes":20631584,"num_objects":455,"num_object_clones":209,"num_object_copies":910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":455,"num_whiteouts":183,"num_read":71,"num_read_kb":15956,"num_write":997,"num_write_kb":22437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"67'1362","reported_seq":3345,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.596301+0000","last_change":"2023-10-30T14:23:31.075832+0000","last_active":"2023-10-30T14:51:34.596301+0000","last_peered":"2023-10-30T14:51:34.596301+0000","last_clean":"2023-10-30T14:51:34.596301+0000","last_became_active":"2023-10-30T14:14:45.669878+0000","last_became_peered":"2023-10-30T14:14:45.669878+0000","last_unstale":"2023-10-30T14:51:34.596301+0000","last_undegraded":"2023-10-30T14:51:34.596301+0000","last_fullsized":"2023-10-30T14:51:34.596301+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1362,"log_dups_size":0,"ondisk_log_size":1362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:53:43.313641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0077997409999999998,"stat_sum":{"num_bytes":25414870,"num_objects":479,"num_object_clones":220,"num_object_copies":958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":194,"num_read":80,"num_read_kb":18178,"num_write":1147,"num_write_kb":27446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"67'1314","reported_seq":3281,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.632800+0000","last_change":"2023-10-30T14:23:30.773367+0000","last_active":"2023-10-30T14:51:34.632800+0000","last_peered":"2023-10-30T14:51:34.632800+0000","last_clean":"2023-10-30T14:51:34.632800+0000","last_became_active":"2023-10-30T14:14:04.597590+0000","last_became_peered":"2023-10-30T14:14:04.597590+0000","last_unstale":"2023-10-30T14:51:34.632800+0000","last_undegraded":"2023-10-30T14:51:34.632800+0000","last_fullsized":"2023-10-30T14:51:34.632800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1314,"log_dups_size":0,"ondisk_log_size":1314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:28:58.351077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044830970000000001,"stat_sum":{"num_bytes":8769781,"num_objects":485,"num_object_clones":224,"num_object_copies":970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":197,"num_read":75,"num_read_kb":4156,"num_write":1100,"num_write_kb":11840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"67'1189","reported_seq":3148,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.623243+0000","last_change":"2023-10-30T14:23:31.070930+0000","last_active":"2023-10-30T14:51:34.623243+0000","last_peered":"2023-10-30T14:51:34.623243+0000","last_clean":"2023-10-30T14:51:34.623243+0000","last_became_active":"2023-10-30T14:14:04.623902+0000","last_became_peered":"2023-10-30T14:14:04.623902+0000","last_unstale":"2023-10-30T14:51:34.623243+0000","last_undegraded":"2023-10-30T14:51:34.623243+0000","last_fullsized":"2023-10-30T14:51:34.623243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1189,"log_dups_size":0,"ondisk_log_size":1189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:13:07.040210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051343489999999999,"stat_sum":{"num_bytes":21110293,"num_objects":460,"num_object_clones":215,"num_object_copies":920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":190,"num_read":58,"num_read_kb":12498,"num_write":969,"num_write_kb":25426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"67'1344","reported_seq":3292,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.583468+0000","last_change":"2023-10-30T14:23:30.775060+0000","last_active":"2023-10-30T14:51:34.583468+0000","last_peered":"2023-10-30T14:51:34.583468+0000","last_clean":"2023-10-30T14:51:34.583468+0000","last_became_active":"2023-10-30T14:14:04.597556+0000","last_became_peered":"2023-10-30T14:14:04.597556+0000","last_unstale":"2023-10-30T14:51:34.583468+0000","last_undegraded":"2023-10-30T14:51:34.583468+0000","last_fullsized":"2023-10-30T14:51:34.583468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1344,"log_dups_size":0,"ondisk_log_size":1344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:42.236111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040069729999999996,"stat_sum":{"num_bytes":25689725,"num_objects":524,"num_object_clones":241,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":212,"num_read":73,"num_read_kb":17618,"num_write":1103,"num_write_kb":27397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"67'1324","reported_seq":3291,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.645005+0000","last_change":"2023-10-30T14:23:30.779406+0000","last_active":"2023-10-30T14:51:34.645005+0000","last_peered":"2023-10-30T14:51:34.645005+0000","last_clean":"2023-10-30T14:51:34.645005+0000","last_became_active":"2023-10-30T14:14:04.618430+0000","last_became_peered":"2023-10-30T14:14:04.618430+0000","last_unstale":"2023-10-30T14:51:34.645005+0000","last_undegraded":"2023-10-30T14:51:34.645005+0000","last_fullsized":"2023-10-30T14:51:34.645005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1324,"log_dups_size":0,"ondisk_log_size":1324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:44:00.464702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074984149999999996,"stat_sum":{"num_bytes":16403179,"num_objects":481,"num_object_clones":225,"num_object_copies":962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":199,"num_read":75,"num_read_kb":12323,"num_write":1105,"num_write_kb":17671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"67'1436","reported_seq":3390,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.639763+0000","last_change":"2023-10-30T14:23:31.067667+0000","last_active":"2023-10-30T14:51:34.639763+0000","last_peered":"2023-10-30T14:51:34.639763+0000","last_clean":"2023-10-30T14:51:34.639763+0000","last_became_active":"2023-10-30T14:14:04.606340+0000","last_became_peered":"2023-10-30T14:14:04.606340+0000","last_unstale":"2023-10-30T14:51:34.639763+0000","last_undegraded":"2023-10-30T14:51:34.639763+0000","last_fullsized":"2023-10-30T14:51:34.639763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1436,"log_dups_size":0,"ondisk_log_size":1436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:41:14.948036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035094890000000002,"stat_sum":{"num_bytes":22361827,"num_objects":534,"num_object_clones":245,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":534,"num_whiteouts":212,"num_read":82,"num_read_kb":9614,"num_write":1202,"num_write_kb":32875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"67'1347","reported_seq":3323,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.648255+0000","last_change":"2023-10-30T14:23:31.074353+0000","last_active":"2023-10-30T14:51:34.648255+0000","last_peered":"2023-10-30T14:51:34.648255+0000","last_clean":"2023-10-30T14:51:34.648255+0000","last_became_active":"2023-10-30T14:14:04.612588+0000","last_became_peered":"2023-10-30T14:14:04.612588+0000","last_unstale":"2023-10-30T14:51:34.648255+0000","last_undegraded":"2023-10-30T14:51:34.648255+0000","last_fullsized":"2023-10-30T14:51:34.648255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1347,"log_dups_size":0,"ondisk_log_size":1347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:56:59.526944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084570800000000005,"stat_sum":{"num_bytes":13290865,"num_objects":484,"num_object_clones":224,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":199,"num_read":83,"num_read_kb":8248,"num_write":1123,"num_write_kb":15099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"67'1290","reported_seq":3256,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.613115+0000","last_change":"2023-10-30T14:23:30.955845+0000","last_active":"2023-10-30T14:51:34.613115+0000","last_peered":"2023-10-30T14:51:34.613115+0000","last_clean":"2023-10-30T14:51:34.613115+0000","last_became_active":"2023-10-30T14:14:04.607683+0000","last_became_peered":"2023-10-30T14:14:04.607683+0000","last_unstale":"2023-10-30T14:51:34.613115+0000","last_undegraded":"2023-10-30T14:51:34.613115+0000","last_fullsized":"2023-10-30T14:51:34.613115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1290,"log_dups_size":0,"ondisk_log_size":1290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:41.822862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001856688,"stat_sum":{"num_bytes":22393964,"num_objects":481,"num_object_clones":222,"num_object_copies":962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":202,"num_read":74,"num_read_kb":12305,"num_write":1053,"num_write_kb":25643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'669","reported_seq":838,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.646665+0000","last_change":"2023-10-30T14:14:03.732084+0000","last_active":"2023-10-30T14:51:34.646665+0000","last_peered":"2023-10-30T14:51:34.646665+0000","last_clean":"2023-10-30T14:51:34.646665+0000","last_became_active":"2023-10-30T14:14:03.731740+0000","last_became_peered":"2023-10-30T14:14:03.731740+0000","last_unstale":"2023-10-30T14:51:34.646665+0000","last_undegraded":"2023-10-30T14:51:34.646665+0000","last_fullsized":"2023-10-30T14:51:34.646665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":669,"log_dups_size":0,"ondisk_log_size":669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:34:27.462477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49204997,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":149,"num_read_kb":143937,"num_write":691,"num_write_kb":73569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"67'1325","reported_seq":3292,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.620707+0000","last_change":"2023-10-30T14:23:30.776069+0000","last_active":"2023-10-30T14:51:34.620707+0000","last_peered":"2023-10-30T14:51:34.620707+0000","last_clean":"2023-10-30T14:51:34.620707+0000","last_became_active":"2023-10-30T14:14:04.621526+0000","last_became_peered":"2023-10-30T14:14:04.621526+0000","last_unstale":"2023-10-30T14:51:34.620707+0000","last_undegraded":"2023-10-30T14:51:34.620707+0000","last_fullsized":"2023-10-30T14:51:34.620707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1325,"log_dups_size":0,"ondisk_log_size":1325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:59:59.926386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047814620000000002,"stat_sum":{"num_bytes":12429609,"num_objects":495,"num_object_clones":226,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":189,"num_read":79,"num_read_kb":8245,"num_write":1124,"num_write_kb":15289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"67'1480","reported_seq":1961,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.252764+0000","last_change":"2023-10-30T14:14:03.730789+0000","last_active":"2023-10-30T14:51:30.252764+0000","last_peered":"2023-10-30T14:51:30.252764+0000","last_clean":"2023-10-30T14:51:30.252764+0000","last_became_active":"2023-10-30T14:14:03.729386+0000","last_became_peered":"2023-10-30T14:14:03.729386+0000","last_unstale":"2023-10-30T14:51:30.252764+0000","last_undegraded":"2023-10-30T14:51:30.252764+0000","last_fullsized":"2023-10-30T14:51:30.252764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1480,"log_dups_size":0,"ondisk_log_size":1480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29761637,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":457,"num_read_kb":164028,"num_write":1504,"num_write_kb":71675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'762","reported_seq":1326,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.994280+0000","last_change":"2023-10-30T14:14:03.724379+0000","last_active":"2023-10-30T14:51:25.994280+0000","last_peered":"2023-10-30T14:51:25.994280+0000","last_clean":"2023-10-30T14:51:25.994280+0000","last_became_active":"2023-10-30T14:14:03.724078+0000","last_became_peered":"2023-10-30T14:14:03.724078+0000","last_unstale":"2023-10-30T14:51:25.994280+0000","last_undegraded":"2023-10-30T14:51:25.994280+0000","last_fullsized":"2023-10-30T14:51:25.994280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":762,"log_dups_size":0,"ondisk_log_size":762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:10:17.829899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30316903,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":547,"num_read_kb":63175,"num_write":794,"num_write_kb":47477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"67'1458","reported_seq":3389,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.648896+0000","last_change":"2023-10-30T14:23:31.068462+0000","last_active":"2023-10-30T14:51:34.648896+0000","last_peered":"2023-10-30T14:51:34.648896+0000","last_clean":"2023-10-30T14:51:34.648896+0000","last_became_active":"2023-10-30T14:14:04.597099+0000","last_became_peered":"2023-10-30T14:14:04.597099+0000","last_unstale":"2023-10-30T14:51:34.648896+0000","last_undegraded":"2023-10-30T14:51:34.648896+0000","last_fullsized":"2023-10-30T14:51:34.648896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1458,"log_dups_size":0,"ondisk_log_size":1458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:04:00.818307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033627510000000002,"stat_sum":{"num_bytes":15535325,"num_objects":552,"num_object_clones":252,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":552,"num_whiteouts":227,"num_read":69,"num_read_kb":8270,"num_write":1194,"num_write_kb":25187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"67'1580","reported_seq":1743,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:27.462264+0000","last_change":"2023-10-30T14:14:03.721030+0000","last_active":"2023-10-30T14:51:27.462264+0000","last_peered":"2023-10-30T14:51:27.462264+0000","last_clean":"2023-10-30T14:51:27.462264+0000","last_became_active":"2023-10-30T14:14:03.719816+0000","last_became_peered":"2023-10-30T14:14:03.719816+0000","last_unstale":"2023-10-30T14:51:27.462264+0000","last_undegraded":"2023-10-30T14:51:27.462264+0000","last_fullsized":"2023-10-30T14:51:27.462264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1580,"log_dups_size":0,"ondisk_log_size":1580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:43:50.150156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35358900,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":164,"num_read_kb":135701,"num_write":1593,"num_write_kb":77235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"67'1291","reported_seq":3256,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.346238+0000","last_change":"2023-10-30T14:23:30.771109+0000","last_active":"2023-10-30T14:51:34.346238+0000","last_peered":"2023-10-30T14:51:34.346238+0000","last_clean":"2023-10-30T14:51:34.346238+0000","last_became_active":"2023-10-30T14:14:04.599323+0000","last_became_peered":"2023-10-30T14:14:04.599323+0000","last_unstale":"2023-10-30T14:51:34.346238+0000","last_undegraded":"2023-10-30T14:51:34.346238+0000","last_fullsized":"2023-10-30T14:51:34.346238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1291,"log_dups_size":0,"ondisk_log_size":1291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:27.656629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025427000000000002,"stat_sum":{"num_bytes":25885700,"num_objects":482,"num_object_clones":224,"num_object_copies":964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":193,"num_read":75,"num_read_kb":21385,"num_write":1056,"num_write_kb":37196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"67'640","reported_seq":767,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.998751+0000","last_change":"2023-10-30T14:14:03.738694+0000","last_active":"2023-10-30T14:51:30.998751+0000","last_peered":"2023-10-30T14:51:30.998751+0000","last_clean":"2023-10-30T14:51:30.998751+0000","last_became_active":"2023-10-30T14:14:03.738132+0000","last_became_peered":"2023-10-30T14:14:03.738132+0000","last_unstale":"2023-10-30T14:51:30.998751+0000","last_undegraded":"2023-10-30T14:51:30.998751+0000","last_fullsized":"2023-10-30T14:51:30.998751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:46:59.692900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360218,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":99,"num_read_kb":53823,"num_write":691,"num_write_kb":33991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"67'1308","reported_seq":3285,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.428808+0000","last_change":"2023-10-30T14:23:30.778832+0000","last_active":"2023-10-30T14:51:34.428808+0000","last_peered":"2023-10-30T14:51:34.428808+0000","last_clean":"2023-10-30T14:51:34.428808+0000","last_became_active":"2023-10-30T14:14:04.648816+0000","last_became_peered":"2023-10-30T14:14:04.648816+0000","last_unstale":"2023-10-30T14:51:34.428808+0000","last_undegraded":"2023-10-30T14:51:34.428808+0000","last_fullsized":"2023-10-30T14:51:34.428808+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1308,"log_dups_size":0,"ondisk_log_size":1308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:20.078590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069848840000000002,"stat_sum":{"num_bytes":12168759,"num_objects":462,"num_object_clones":214,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":462,"num_whiteouts":185,"num_read":77,"num_read_kb":8211,"num_write":1090,"num_write_kb":14663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"67'1050","reported_seq":1187,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.470634+0000","last_change":"2023-10-30T14:14:03.726112+0000","last_active":"2023-10-30T14:51:29.470634+0000","last_peered":"2023-10-30T14:51:29.470634+0000","last_clean":"2023-10-30T14:51:29.470634+0000","last_became_active":"2023-10-30T14:14:03.725272+0000","last_became_peered":"2023-10-30T14:14:03.725272+0000","last_unstale":"2023-10-30T14:51:29.470634+0000","last_undegraded":"2023-10-30T14:51:29.470634+0000","last_fullsized":"2023-10-30T14:51:29.470634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1050,"log_dups_size":0,"ondisk_log_size":1050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:37:20.764294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":123,"num_read_kb":86587,"num_write":1104,"num_write_kb":54796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"67'1405","reported_seq":3324,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.628156+0000","last_change":"2023-10-30T14:23:30.776486+0000","last_active":"2023-10-30T14:51:34.628156+0000","last_peered":"2023-10-30T14:51:34.628156+0000","last_clean":"2023-10-30T14:51:34.628156+0000","last_became_active":"2023-10-30T14:14:04.609697+0000","last_became_peered":"2023-10-30T14:14:04.609697+0000","last_unstale":"2023-10-30T14:51:34.628156+0000","last_undegraded":"2023-10-30T14:51:34.628156+0000","last_fullsized":"2023-10-30T14:51:34.628156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1405,"log_dups_size":0,"ondisk_log_size":1405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:45:56.464708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043395700000000001,"stat_sum":{"num_bytes":12484119,"num_objects":557,"num_object_clones":263,"num_object_copies":1114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":557,"num_whiteouts":236,"num_read":68,"num_read_kb":4261,"num_write":1150,"num_write_kb":13862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"67'1008","reported_seq":1169,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.518670+0000","last_change":"2023-10-30T14:14:03.727871+0000","last_active":"2023-10-30T14:51:34.518670+0000","last_peered":"2023-10-30T14:51:34.518670+0000","last_clean":"2023-10-30T14:51:34.518670+0000","last_became_active":"2023-10-30T14:14:03.727422+0000","last_became_peered":"2023-10-30T14:14:03.727422+0000","last_unstale":"2023-10-30T14:51:34.518670+0000","last_undegraded":"2023-10-30T14:51:34.518670+0000","last_fullsized":"2023-10-30T14:51:34.518670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1008,"log_dups_size":0,"ondisk_log_size":1008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:54.782533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389280,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":161,"num_read_kb":61498,"num_write":1088,"num_write_kb":30346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1502","reported_seq":3375,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.618047+0000","last_change":"2023-10-30T14:23:30.957793+0000","last_active":"2023-10-30T14:51:34.618047+0000","last_peered":"2023-10-30T14:51:34.618047+0000","last_clean":"2023-10-30T14:51:34.618047+0000","last_became_active":"2023-10-30T14:14:04.645333+0000","last_became_peered":"2023-10-30T14:14:04.645333+0000","last_unstale":"2023-10-30T14:51:34.618047+0000","last_undegraded":"2023-10-30T14:51:34.618047+0000","last_fullsized":"2023-10-30T14:51:34.618047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1502,"log_dups_size":0,"ondisk_log_size":1502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:13:02.147712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002301474,"stat_sum":{"num_bytes":21718535,"num_objects":601,"num_object_clones":320,"num_object_copies":1202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":601,"num_whiteouts":226,"num_read":78,"num_read_kb":5632,"num_write":1155,"num_write_kb":28163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"67'579","reported_seq":766,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:27.713419+0000","last_change":"2023-10-30T14:14:03.725554+0000","last_active":"2023-10-30T14:51:27.713419+0000","last_peered":"2023-10-30T14:51:27.713419+0000","last_clean":"2023-10-30T14:51:27.713419+0000","last_became_active":"2023-10-30T14:14:03.724806+0000","last_became_peered":"2023-10-30T14:14:03.724806+0000","last_unstale":"2023-10-30T14:51:27.713419+0000","last_undegraded":"2023-10-30T14:51:27.713419+0000","last_fullsized":"2023-10-30T14:51:27.713419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:37.823561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360800,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":192,"num_read_kb":131840,"num_write":657,"num_write_kb":54408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"67'1328","reported_seq":3273,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.607998+0000","last_change":"2023-10-30T14:23:31.073841+0000","last_active":"2023-10-30T14:51:34.607998+0000","last_peered":"2023-10-30T14:51:34.607998+0000","last_clean":"2023-10-30T14:51:34.607998+0000","last_became_active":"2023-10-30T14:14:04.646543+0000","last_became_peered":"2023-10-30T14:14:04.646543+0000","last_unstale":"2023-10-30T14:51:34.607998+0000","last_undegraded":"2023-10-30T14:51:34.607998+0000","last_fullsized":"2023-10-30T14:51:34.607998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1328,"log_dups_size":0,"ondisk_log_size":1328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:30:29.655398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00744474,"stat_sum":{"num_bytes":22194385,"num_objects":506,"num_object_clones":234,"num_object_copies":1012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":207,"num_read":65,"num_read_kb":16406,"num_write":1096,"num_write_kb":24097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"67'684","reported_seq":859,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.631660+0000","last_change":"2023-10-30T14:14:03.717903+0000","last_active":"2023-10-30T14:51:29.631660+0000","last_peered":"2023-10-30T14:51:29.631660+0000","last_clean":"2023-10-30T14:51:29.631660+0000","last_became_active":"2023-10-30T14:14:03.716941+0000","last_became_peered":"2023-10-30T14:14:03.716941+0000","last_unstale":"2023-10-30T14:51:29.631660+0000","last_undegraded":"2023-10-30T14:51:29.631660+0000","last_fullsized":"2023-10-30T14:51:29.631660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:58.178166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777250,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":201,"num_read_kb":82705,"num_write":762,"num_write_kb":38720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"67'1418","reported_seq":3354,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.643287+0000","last_change":"2023-10-30T14:23:30.775738+0000","last_active":"2023-10-30T14:51:34.643287+0000","last_peered":"2023-10-30T14:51:34.643287+0000","last_clean":"2023-10-30T14:51:34.643287+0000","last_became_active":"2023-10-30T14:14:04.641494+0000","last_became_peered":"2023-10-30T14:14:04.641494+0000","last_unstale":"2023-10-30T14:51:34.643287+0000","last_undegraded":"2023-10-30T14:51:34.643287+0000","last_fullsized":"2023-10-30T14:51:34.643287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1418,"log_dups_size":0,"ondisk_log_size":1418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:44.138968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039664019999999999,"stat_sum":{"num_bytes":26425057,"num_objects":539,"num_object_clones":248,"num_object_copies":1078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":208,"num_read":69,"num_read_kb":14198,"num_write":1182,"num_write_kb":36722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"67'304","reported_seq":425,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.499654+0000","last_change":"2023-10-30T14:14:03.729626+0000","last_active":"2023-10-30T14:51:34.499654+0000","last_peered":"2023-10-30T14:51:34.499654+0000","last_clean":"2023-10-30T14:51:34.499654+0000","last_became_active":"2023-10-30T14:14:03.728053+0000","last_became_peered":"2023-10-30T14:14:03.728053+0000","last_unstale":"2023-10-30T14:51:34.499654+0000","last_undegraded":"2023-10-30T14:51:34.499654+0000","last_fullsized":"2023-10-30T14:51:34.499654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:44:53.362066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":90,"num_read_kb":49506,"num_write":347,"num_write_kb":29524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1406","reported_seq":3367,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.520672+0000","last_change":"2023-10-30T14:23:31.078295+0000","last_active":"2023-10-30T14:51:34.520672+0000","last_peered":"2023-10-30T14:51:34.520672+0000","last_clean":"2023-10-30T14:51:34.520672+0000","last_became_active":"2023-10-30T14:14:04.649896+0000","last_became_peered":"2023-10-30T14:14:04.649896+0000","last_unstale":"2023-10-30T14:51:34.520672+0000","last_undegraded":"2023-10-30T14:51:34.520672+0000","last_fullsized":"2023-10-30T14:51:34.520672+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1406,"log_dups_size":0,"ondisk_log_size":1406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:07:44.011495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010792094,"stat_sum":{"num_bytes":9892654,"num_objects":516,"num_object_clones":241,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":516,"num_whiteouts":210,"num_read":88,"num_read_kb":2154,"num_write":1169,"num_write_kb":11317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"67'714","reported_seq":850,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:32.363555+0000","last_change":"2023-10-30T14:14:03.736824+0000","last_active":"2023-10-30T14:51:32.363555+0000","last_peered":"2023-10-30T14:51:32.363555+0000","last_clean":"2023-10-30T14:51:32.363555+0000","last_became_active":"2023-10-30T14:14:03.736155+0000","last_became_peered":"2023-10-30T14:14:03.736155+0000","last_unstale":"2023-10-30T14:51:32.363555+0000","last_undegraded":"2023-10-30T14:51:32.363555+0000","last_fullsized":"2023-10-30T14:51:32.363555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:35.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":104,"num_read_kb":67382,"num_write":741,"num_write_kb":42078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"67'1271","reported_seq":3243,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.517989+0000","last_change":"2023-10-30T14:23:31.076487+0000","last_active":"2023-10-30T14:51:34.517989+0000","last_peered":"2023-10-30T14:51:34.517989+0000","last_clean":"2023-10-30T14:51:34.517989+0000","last_became_active":"2023-10-30T14:14:04.614328+0000","last_became_peered":"2023-10-30T14:14:04.614328+0000","last_unstale":"2023-10-30T14:51:34.517989+0000","last_undegraded":"2023-10-30T14:51:34.517989+0000","last_fullsized":"2023-10-30T14:51:34.517989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1271,"log_dups_size":0,"ondisk_log_size":1271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:04:54.384597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010029286,"stat_sum":{"num_bytes":9325803,"num_objects":478,"num_object_clones":220,"num_object_copies":956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":189,"num_read":77,"num_read_kb":4169,"num_write":1060,"num_write_kb":11660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"67'1215","reported_seq":1399,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.630666+0000","last_change":"2023-10-30T14:14:03.700690+0000","last_active":"2023-10-30T14:51:29.630666+0000","last_peered":"2023-10-30T14:51:29.630666+0000","last_clean":"2023-10-30T14:51:29.630666+0000","last_became_active":"2023-10-30T14:14:03.700352+0000","last_became_peered":"2023-10-30T14:14:03.700352+0000","last_unstale":"2023-10-30T14:51:29.630666+0000","last_undegraded":"2023-10-30T14:51:29.630666+0000","last_fullsized":"2023-10-30T14:51:29.630666+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1215,"log_dups_size":0,"ondisk_log_size":1215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:03:01.103359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":252,"num_read_kb":55463,"num_write":1290,"num_write_kb":26283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"67'1371","reported_seq":3330,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.611548+0000","last_change":"2023-10-30T14:23:31.079218+0000","last_active":"2023-10-30T14:51:34.611548+0000","last_peered":"2023-10-30T14:51:34.611548+0000","last_clean":"2023-10-30T14:51:34.611548+0000","last_became_active":"2023-10-30T14:14:04.646639+0000","last_became_peered":"2023-10-30T14:14:04.646639+0000","last_unstale":"2023-10-30T14:51:34.611548+0000","last_undegraded":"2023-10-30T14:51:34.611548+0000","last_fullsized":"2023-10-30T14:51:34.611548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1371,"log_dups_size":0,"ondisk_log_size":1371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:32.477580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010379724,"stat_sum":{"num_bytes":13632563,"num_objects":508,"num_object_clones":233,"num_object_copies":1016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":508,"num_whiteouts":207,"num_read":77,"num_read_kb":12340,"num_write":1126,"num_write_kb":24370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'643","reported_seq":768,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.271430+0000","last_change":"2023-10-30T14:14:03.710717+0000","last_active":"2023-10-30T14:51:25.271430+0000","last_peered":"2023-10-30T14:51:25.271430+0000","last_clean":"2023-10-30T14:51:25.271430+0000","last_became_active":"2023-10-30T14:14:03.707950+0000","last_became_peered":"2023-10-30T14:14:03.707950+0000","last_unstale":"2023-10-30T14:51:25.271430+0000","last_undegraded":"2023-10-30T14:51:25.271430+0000","last_fullsized":"2023-10-30T14:51:25.271430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:29:41.232902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971542,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":82,"num_read_kb":57693,"num_write":882,"num_write_kb":42351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"67'1262","reported_seq":3208,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.637549+0000","last_change":"2023-10-30T14:23:31.080966+0000","last_active":"2023-10-30T14:51:34.637549+0000","last_peered":"2023-10-30T14:51:34.637549+0000","last_clean":"2023-10-30T14:51:34.637549+0000","last_became_active":"2023-10-30T14:14:04.644761+0000","last_became_peered":"2023-10-30T14:14:04.644761+0000","last_unstale":"2023-10-30T14:51:34.637549+0000","last_undegraded":"2023-10-30T14:51:34.637549+0000","last_fullsized":"2023-10-30T14:51:34.637549+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1262,"log_dups_size":0,"ondisk_log_size":1262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:55:03.102431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011171970999999999,"stat_sum":{"num_bytes":12945563,"num_objects":503,"num_object_clones":231,"num_object_copies":1006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":213,"num_read":62,"num_read_kb":4133,"num_write":1006,"num_write_kb":18895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'539","reported_seq":670,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.434150+0000","last_change":"2023-10-30T14:14:03.705567+0000","last_active":"2023-10-30T14:51:34.434150+0000","last_peered":"2023-10-30T14:51:34.434150+0000","last_clean":"2023-10-30T14:51:34.434150+0000","last_became_active":"2023-10-30T14:14:03.705186+0000","last_became_peered":"2023-10-30T14:14:03.705186+0000","last_unstale":"2023-10-30T14:51:34.434150+0000","last_undegraded":"2023-10-30T14:51:34.434150+0000","last_fullsized":"2023-10-30T14:51:34.434150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:14.609243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":102,"num_read_kb":90592,"num_write":580,"num_write_kb":50591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"67'1327","reported_seq":3272,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.610193+0000","last_change":"2023-10-30T14:23:31.083260+0000","last_active":"2023-10-30T14:51:34.610193+0000","last_peered":"2023-10-30T14:51:34.610193+0000","last_clean":"2023-10-30T14:51:34.610193+0000","last_became_active":"2023-10-30T14:14:04.646785+0000","last_became_peered":"2023-10-30T14:14:04.646785+0000","last_unstale":"2023-10-30T14:51:34.610193+0000","last_undegraded":"2023-10-30T14:51:34.610193+0000","last_fullsized":"2023-10-30T14:51:34.610193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1327,"log_dups_size":0,"ondisk_log_size":1327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:52:21.536104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010111184,"stat_sum":{"num_bytes":17483229,"num_objects":512,"num_object_clones":242,"num_object_copies":1024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":512,"num_whiteouts":218,"num_read":72,"num_read_kb":10698,"num_write":1044,"num_write_kb":23548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"67'600","reported_seq":750,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:32.359797+0000","last_change":"2023-10-30T14:14:03.693349+0000","last_active":"2023-10-30T14:51:32.359797+0000","last_peered":"2023-10-30T14:51:32.359797+0000","last_clean":"2023-10-30T14:51:32.359797+0000","last_became_active":"2023-10-30T14:14:03.692966+0000","last_became_peered":"2023-10-30T14:14:03.692966+0000","last_unstale":"2023-10-30T14:51:32.359797+0000","last_undegraded":"2023-10-30T14:51:32.359797+0000","last_fullsized":"2023-10-30T14:51:32.359797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:30:28.800214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16780852,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":126,"num_read_kb":110211,"num_write":641,"num_write_kb":50166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'1343","reported_seq":3317,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.627210+0000","last_change":"2023-10-30T14:23:30.778622+0000","last_active":"2023-10-30T14:51:34.627210+0000","last_peered":"2023-10-30T14:51:34.627210+0000","last_clean":"2023-10-30T14:51:34.627210+0000","last_became_active":"2023-10-30T14:14:04.615925+0000","last_became_peered":"2023-10-30T14:14:04.615925+0000","last_unstale":"2023-10-30T14:51:34.627210+0000","last_undegraded":"2023-10-30T14:51:34.627210+0000","last_fullsized":"2023-10-30T14:51:34.627210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1343,"log_dups_size":0,"ondisk_log_size":1343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:54.760495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068317569999999999,"stat_sum":{"num_bytes":4343182,"num_objects":487,"num_object_clones":225,"num_object_copies":974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":206,"num_read":84,"num_read_kb":56,"num_write":1107,"num_write_kb":8908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"67'667","reported_seq":805,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.244959+0000","last_change":"2023-10-30T14:14:03.727659+0000","last_active":"2023-10-30T14:51:25.244959+0000","last_peered":"2023-10-30T14:51:25.244959+0000","last_clean":"2023-10-30T14:51:25.244959+0000","last_became_active":"2023-10-30T14:14:03.727409+0000","last_became_peered":"2023-10-30T14:14:03.727409+0000","last_unstale":"2023-10-30T14:51:25.244959+0000","last_undegraded":"2023-10-30T14:51:25.244959+0000","last_fullsized":"2023-10-30T14:51:25.244959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:24.210635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166406,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":118,"num_read_kb":66012,"num_write":697,"num_write_kb":50302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1295","reported_seq":3267,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.637178+0000","last_change":"2023-10-30T14:23:31.077576+0000","last_active":"2023-10-30T14:51:34.637178+0000","last_peered":"2023-10-30T14:51:34.637178+0000","last_clean":"2023-10-30T14:51:34.637178+0000","last_became_active":"2023-10-30T14:14:45.818032+0000","last_became_peered":"2023-10-30T14:14:45.818032+0000","last_unstale":"2023-10-30T14:51:34.637178+0000","last_undegraded":"2023-10-30T14:51:34.637178+0000","last_fullsized":"2023-10-30T14:51:34.637178+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:54:03.288491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010655906,"stat_sum":{"num_bytes":23800594,"num_objects":485,"num_object_clones":224,"num_object_copies":970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":201,"num_read":74,"num_read_kb":13429,"num_write":1078,"num_write_kb":29347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":229,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T14:49:57.449030+0000","last_change":"2023-10-30T14:13:54.251572+0000","last_active":"2023-10-30T14:49:57.449030+0000","last_peered":"2023-10-30T14:49:57.449030+0000","last_clean":"2023-10-30T14:49:57.449030+0000","last_became_active":"2023-10-30T14:13:54.250907+0000","last_became_peered":"2023-10-30T14:13:54.250907+0000","last_unstale":"2023-10-30T14:49:57.449030+0000","last_undegraded":"2023-10-30T14:49:57.449030+0000","last_fullsized":"2023-10-30T14:49:57.449030+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:13:53.220267+0000","last_clean_scrub_stamp":"2023-10-30T14:13:53.220267+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:44.590852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"67'727","reported_seq":871,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.519767+0000","last_change":"2023-10-30T14:14:03.729034+0000","last_active":"2023-10-30T14:51:34.519767+0000","last_peered":"2023-10-30T14:51:34.519767+0000","last_clean":"2023-10-30T14:51:34.519767+0000","last_became_active":"2023-10-30T14:14:03.728293+0000","last_became_peered":"2023-10-30T14:14:03.728293+0000","last_unstale":"2023-10-30T14:51:34.519767+0000","last_undegraded":"2023-10-30T14:51:34.519767+0000","last_fullsized":"2023-10-30T14:51:34.519767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":727,"log_dups_size":0,"ondisk_log_size":727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:49.209729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165954,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":121,"num_read_kb":78313,"num_write":800,"num_write_kb":42232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"67'1267","reported_seq":3229,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.632290+0000","last_change":"2023-10-30T14:23:31.067976+0000","last_active":"2023-10-30T14:51:34.632290+0000","last_peered":"2023-10-30T14:51:34.632290+0000","last_clean":"2023-10-30T14:51:34.632290+0000","last_became_active":"2023-10-30T14:14:46.451620+0000","last_became_peered":"2023-10-30T14:14:46.451620+0000","last_unstale":"2023-10-30T14:51:34.632290+0000","last_undegraded":"2023-10-30T14:51:34.632290+0000","last_fullsized":"2023-10-30T14:51:34.632290+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1267,"log_dups_size":0,"ondisk_log_size":1267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:10.312892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034002070000000001,"stat_sum":{"num_bytes":12539401,"num_objects":475,"num_object_clones":216,"num_object_copies":950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":185,"num_read":65,"num_read_kb":8203,"num_write":1054,"num_write_kb":18334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"67'823","reported_seq":1009,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.517181+0000","last_change":"2023-10-30T14:14:03.732909+0000","last_active":"2023-10-30T14:51:29.517181+0000","last_peered":"2023-10-30T14:51:29.517181+0000","last_clean":"2023-10-30T14:51:29.517181+0000","last_became_active":"2023-10-30T14:14:03.732199+0000","last_became_peered":"2023-10-30T14:14:03.732199+0000","last_unstale":"2023-10-30T14:51:29.517181+0000","last_undegraded":"2023-10-30T14:51:29.517181+0000","last_fullsized":"2023-10-30T14:51:29.517181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":823,"log_dups_size":0,"ondisk_log_size":823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:38.117750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":198,"num_read_kb":136195,"num_write":895,"num_write_kb":59080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"67'1270","reported_seq":3231,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.636422+0000","last_change":"2023-10-30T14:23:30.775385+0000","last_active":"2023-10-30T14:51:34.636422+0000","last_peered":"2023-10-30T14:51:34.636422+0000","last_clean":"2023-10-30T14:51:34.636422+0000","last_became_active":"2023-10-30T14:14:04.619990+0000","last_became_peered":"2023-10-30T14:14:04.619990+0000","last_unstale":"2023-10-30T14:51:34.636422+0000","last_undegraded":"2023-10-30T14:51:34.636422+0000","last_fullsized":"2023-10-30T14:51:34.636422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1270,"log_dups_size":0,"ondisk_log_size":1270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:15:42.711461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045963250000000001,"stat_sum":{"num_bytes":8686325,"num_objects":475,"num_object_clones":219,"num_object_copies":950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":195,"num_read":63,"num_read_kb":1576,"num_write":1040,"num_write_kb":11254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"67'1755","reported_seq":2862,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.536635+0000","last_change":"2023-10-30T14:14:03.702862+0000","last_active":"2023-10-30T14:51:34.536635+0000","last_peered":"2023-10-30T14:51:34.536635+0000","last_clean":"2023-10-30T14:51:34.536635+0000","last_became_active":"2023-10-30T14:14:03.702513+0000","last_became_peered":"2023-10-30T14:14:03.702513+0000","last_unstale":"2023-10-30T14:51:34.536635+0000","last_undegraded":"2023-10-30T14:51:34.536635+0000","last_fullsized":"2023-10-30T14:51:34.536635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1755,"log_dups_size":0,"ondisk_log_size":1755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:25.499390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27782952,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":1112,"num_read_kb":113704,"num_write":1846,"num_write_kb":62106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"67'1334","reported_seq":3285,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.430294+0000","last_change":"2023-10-30T14:23:30.770719+0000","last_active":"2023-10-30T14:51:34.430294+0000","last_peered":"2023-10-30T14:51:34.430294+0000","last_clean":"2023-10-30T14:51:34.430294+0000","last_became_active":"2023-10-30T14:14:04.589087+0000","last_became_peered":"2023-10-30T14:14:04.589087+0000","last_unstale":"2023-10-30T14:51:34.430294+0000","last_undegraded":"2023-10-30T14:51:34.430294+0000","last_fullsized":"2023-10-30T14:51:34.430294+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1334,"log_dups_size":0,"ondisk_log_size":1334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:44:54.038742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022871390000000001,"stat_sum":{"num_bytes":30609501,"num_objects":512,"num_object_clones":244,"num_object_copies":1024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":512,"num_whiteouts":220,"num_read":78,"num_read_kb":20511,"num_write":1068,"num_write_kb":36447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"67'635","reported_seq":785,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:32.468831+0000","last_change":"2023-10-30T14:14:03.730048+0000","last_active":"2023-10-30T14:51:32.468831+0000","last_peered":"2023-10-30T14:51:32.468831+0000","last_clean":"2023-10-30T14:51:32.468831+0000","last_became_active":"2023-10-30T14:14:03.728925+0000","last_became_peered":"2023-10-30T14:14:03.728925+0000","last_unstale":"2023-10-30T14:51:32.468831+0000","last_undegraded":"2023-10-30T14:51:32.468831+0000","last_fullsized":"2023-10-30T14:51:32.468831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:32:05.056039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":120,"num_read_kb":131794,"num_write":661,"num_write_kb":62442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"67'1363","reported_seq":3313,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.645184+0000","last_change":"2023-10-30T14:23:30.956956+0000","last_active":"2023-10-30T14:51:34.645184+0000","last_peered":"2023-10-30T14:51:34.645184+0000","last_clean":"2023-10-30T14:51:34.645184+0000","last_became_active":"2023-10-30T14:14:04.607377+0000","last_became_peered":"2023-10-30T14:14:04.607377+0000","last_unstale":"2023-10-30T14:51:34.645184+0000","last_undegraded":"2023-10-30T14:51:34.645184+0000","last_fullsized":"2023-10-30T14:51:34.645184+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1363,"log_dups_size":0,"ondisk_log_size":1363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:58:23.634836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022021179999999999,"stat_sum":{"num_bytes":18168327,"num_objects":503,"num_object_clones":232,"num_object_copies":1006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":207,"num_read":65,"num_read_kb":9172,"num_write":1122,"num_write_kb":20929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"67'883","reported_seq":1021,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.428984+0000","last_change":"2023-10-30T14:14:03.693753+0000","last_active":"2023-10-30T14:51:34.428984+0000","last_peered":"2023-10-30T14:51:34.428984+0000","last_clean":"2023-10-30T14:51:34.428984+0000","last_became_active":"2023-10-30T14:14:03.692621+0000","last_became_peered":"2023-10-30T14:14:03.692621+0000","last_unstale":"2023-10-30T14:51:34.428984+0000","last_undegraded":"2023-10-30T14:51:34.428984+0000","last_fullsized":"2023-10-30T14:51:34.428984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":883,"log_dups_size":0,"ondisk_log_size":883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:32:28.577584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":107,"num_read_kb":66236,"num_write":924,"num_write_kb":62670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'1349","reported_seq":3277,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.395863+0000","last_change":"2023-10-30T14:23:30.780258+0000","last_active":"2023-10-30T14:51:34.395863+0000","last_peered":"2023-10-30T14:51:34.395863+0000","last_clean":"2023-10-30T14:51:34.395863+0000","last_became_active":"2023-10-30T14:14:04.613780+0000","last_became_peered":"2023-10-30T14:14:04.613780+0000","last_unstale":"2023-10-30T14:51:34.395863+0000","last_undegraded":"2023-10-30T14:51:34.395863+0000","last_fullsized":"2023-10-30T14:51:34.395863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1349,"log_dups_size":0,"ondisk_log_size":1349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:07:05.556883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065285270000000001,"stat_sum":{"num_bytes":12750235,"num_objects":524,"num_object_clones":251,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":229,"num_read":63,"num_read_kb":8216,"num_write":1081,"num_write_kb":18064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"67'722","reported_seq":885,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:33.739146+0000","last_change":"2023-10-30T14:14:03.740213+0000","last_active":"2023-10-30T14:51:33.739146+0000","last_peered":"2023-10-30T14:51:33.739146+0000","last_clean":"2023-10-30T14:51:33.739146+0000","last_became_active":"2023-10-30T14:14:03.739650+0000","last_became_peered":"2023-10-30T14:14:03.739650+0000","last_unstale":"2023-10-30T14:51:33.739146+0000","last_undegraded":"2023-10-30T14:51:33.739146+0000","last_fullsized":"2023-10-30T14:51:33.739146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":722,"log_dups_size":0,"ondisk_log_size":722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:04:54.120757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777238,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":143,"num_read_kb":109192,"num_write":780,"num_write_kb":46441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"67'1336","reported_seq":3292,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.645098+0000","last_change":"2023-10-30T14:23:30.771719+0000","last_active":"2023-10-30T14:51:34.645098+0000","last_peered":"2023-10-30T14:51:34.645098+0000","last_clean":"2023-10-30T14:51:34.645098+0000","last_became_active":"2023-10-30T14:14:04.590138+0000","last_became_peered":"2023-10-30T14:14:04.590138+0000","last_unstale":"2023-10-30T14:51:34.645098+0000","last_undegraded":"2023-10-30T14:51:34.645098+0000","last_fullsized":"2023-10-30T14:51:34.645098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1336,"log_dups_size":0,"ondisk_log_size":1336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:49.779956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031364370000000002,"stat_sum":{"num_bytes":9254236,"num_objects":503,"num_object_clones":233,"num_object_copies":1006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":207,"num_read":72,"num_read_kb":4145,"num_write":1099,"num_write_kb":19935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"67'773","reported_seq":915,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:32.360961+0000","last_change":"2023-10-30T14:14:03.721450+0000","last_active":"2023-10-30T14:51:32.360961+0000","last_peered":"2023-10-30T14:51:32.360961+0000","last_clean":"2023-10-30T14:51:32.360961+0000","last_became_active":"2023-10-30T14:14:03.721234+0000","last_became_peered":"2023-10-30T14:14:03.721234+0000","last_unstale":"2023-10-30T14:51:32.360961+0000","last_undegraded":"2023-10-30T14:51:32.360961+0000","last_fullsized":"2023-10-30T14:51:32.360961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:26.959676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":136,"num_read_kb":37589,"num_write":1315,"num_write_kb":40351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"67'1265","reported_seq":3219,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.370008+0000","last_change":"2023-10-30T14:23:31.070447+0000","last_active":"2023-10-30T14:51:34.370008+0000","last_peered":"2023-10-30T14:51:34.370008+0000","last_clean":"2023-10-30T14:51:34.370008+0000","last_became_active":"2023-10-30T14:14:04.633643+0000","last_became_peered":"2023-10-30T14:14:04.633643+0000","last_unstale":"2023-10-30T14:51:34.370008+0000","last_undegraded":"2023-10-30T14:51:34.370008+0000","last_fullsized":"2023-10-30T14:51:34.370008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1265,"log_dups_size":0,"ondisk_log_size":1265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:53:41.463549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048916599999999999,"stat_sum":{"num_bytes":9341707,"num_objects":489,"num_object_clones":229,"num_object_copies":978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":206,"num_read":67,"num_read_kb":4121,"num_write":1047,"num_write_kb":15591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"67'442","reported_seq":579,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.512517+0000","last_change":"2023-10-30T14:14:03.738557+0000","last_active":"2023-10-30T14:51:34.512517+0000","last_peered":"2023-10-30T14:51:34.512517+0000","last_clean":"2023-10-30T14:51:34.512517+0000","last_became_active":"2023-10-30T14:14:03.737801+0000","last_became_peered":"2023-10-30T14:14:03.737801+0000","last_unstale":"2023-10-30T14:51:34.512517+0000","last_undegraded":"2023-10-30T14:51:34.512517+0000","last_fullsized":"2023-10-30T14:51:34.512517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":442,"log_dups_size":0,"ondisk_log_size":442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:26.938343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":132,"num_read_kb":45549,"num_write":534,"num_write_kb":26063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"67'1256","reported_seq":3228,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.513243+0000","last_change":"2023-10-30T14:23:30.776079+0000","last_active":"2023-10-30T14:51:34.513243+0000","last_peered":"2023-10-30T14:51:34.513243+0000","last_clean":"2023-10-30T14:51:34.513243+0000","last_became_active":"2023-10-30T14:14:04.588826+0000","last_became_peered":"2023-10-30T14:14:04.588826+0000","last_unstale":"2023-10-30T14:51:34.513243+0000","last_undegraded":"2023-10-30T14:51:34.513243+0000","last_fullsized":"2023-10-30T14:51:34.513243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1256,"log_dups_size":0,"ondisk_log_size":1256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:27.236524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040592670000000001,"stat_sum":{"num_bytes":8300341,"num_objects":453,"num_object_clones":211,"num_object_copies":906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":453,"num_whiteouts":181,"num_read":67,"num_read_kb":4125,"num_write":1053,"num_write_kb":10742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"67'431","reported_seq":544,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.425046+0000","last_change":"2023-10-30T14:14:03.733836+0000","last_active":"2023-10-30T14:51:34.425046+0000","last_peered":"2023-10-30T14:51:34.425046+0000","last_clean":"2023-10-30T14:51:34.425046+0000","last_became_active":"2023-10-30T14:14:03.732912+0000","last_became_peered":"2023-10-30T14:14:03.732912+0000","last_unstale":"2023-10-30T14:51:34.425046+0000","last_undegraded":"2023-10-30T14:51:34.425046+0000","last_fullsized":"2023-10-30T14:51:34.425046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:50.332738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":78,"num_read_kb":41242,"num_write":429,"num_write_kb":21093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"67'1290","reported_seq":3258,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.375789+0000","last_change":"2023-10-30T14:23:30.957473+0000","last_active":"2023-10-30T14:51:34.375789+0000","last_peered":"2023-10-30T14:51:34.375789+0000","last_clean":"2023-10-30T14:51:34.375789+0000","last_became_active":"2023-10-30T14:14:04.609835+0000","last_became_peered":"2023-10-30T14:14:04.609835+0000","last_unstale":"2023-10-30T14:51:34.375789+0000","last_undegraded":"2023-10-30T14:51:34.375789+0000","last_fullsized":"2023-10-30T14:51:34.375789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1290,"log_dups_size":0,"ondisk_log_size":1290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:29:36.272312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023958389999999999,"stat_sum":{"num_bytes":25940957,"num_objects":470,"num_object_clones":215,"num_object_copies":940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":192,"num_read":67,"num_read_kb":20495,"num_write":1067,"num_write_kb":27713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"67'868","reported_seq":1009,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:25.228830+0000","last_change":"2023-10-30T14:14:03.732539+0000","last_active":"2023-10-30T14:51:25.228830+0000","last_peered":"2023-10-30T14:51:25.228830+0000","last_clean":"2023-10-30T14:51:25.228830+0000","last_became_active":"2023-10-30T14:14:03.732265+0000","last_became_peered":"2023-10-30T14:14:03.732265+0000","last_unstale":"2023-10-30T14:51:25.228830+0000","last_undegraded":"2023-10-30T14:51:25.228830+0000","last_fullsized":"2023-10-30T14:51:25.228830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":868,"log_dups_size":0,"ondisk_log_size":868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:02:00.929612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":128,"num_read_kb":49868,"num_write":925,"num_write_kb":26024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"67'1359","reported_seq":3337,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.511045+0000","last_change":"2023-10-30T14:23:31.069537+0000","last_active":"2023-10-30T14:51:34.511045+0000","last_peered":"2023-10-30T14:51:34.511045+0000","last_clean":"2023-10-30T14:51:34.511045+0000","last_became_active":"2023-10-30T14:14:04.594998+0000","last_became_peered":"2023-10-30T14:14:04.594998+0000","last_unstale":"2023-10-30T14:51:34.511045+0000","last_undegraded":"2023-10-30T14:51:34.511045+0000","last_fullsized":"2023-10-30T14:51:34.511045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1359,"log_dups_size":0,"ondisk_log_size":1359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:29:28.794439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00432118,"stat_sum":{"num_bytes":12343269,"num_objects":466,"num_object_clones":216,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":186,"num_read":78,"num_read_kb":8212,"num_write":1132,"num_write_kb":18560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'817","reported_seq":963,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:31.000251+0000","last_change":"2023-10-30T14:14:03.703744+0000","last_active":"2023-10-30T14:51:31.000251+0000","last_peered":"2023-10-30T14:51:31.000251+0000","last_clean":"2023-10-30T14:51:31.000251+0000","last_became_active":"2023-10-30T14:14:03.701759+0000","last_became_peered":"2023-10-30T14:14:03.701759+0000","last_unstale":"2023-10-30T14:51:31.000251+0000","last_undegraded":"2023-10-30T14:51:31.000251+0000","last_fullsized":"2023-10-30T14:51:31.000251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":817,"log_dups_size":0,"ondisk_log_size":817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:22:27.948007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":122,"num_read_kb":94942,"num_write":1183,"num_write_kb":51323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"67'1241","reported_seq":3220,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.589022+0000","last_change":"2023-10-30T14:23:31.071315+0000","last_active":"2023-10-30T14:51:34.589022+0000","last_peered":"2023-10-30T14:51:34.589022+0000","last_clean":"2023-10-30T14:51:34.589022+0000","last_became_active":"2023-10-30T14:14:04.600100+0000","last_became_peered":"2023-10-30T14:14:04.600100+0000","last_unstale":"2023-10-30T14:51:34.589022+0000","last_undegraded":"2023-10-30T14:51:34.589022+0000","last_fullsized":"2023-10-30T14:51:34.589022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1241,"log_dups_size":0,"ondisk_log_size":1241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:30.476350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052227630000000001,"stat_sum":{"num_bytes":16881451,"num_objects":447,"num_object_clones":209,"num_object_copies":894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":182,"num_read":73,"num_read_kb":11653,"num_write":1021,"num_write_kb":18895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"67'773","reported_seq":906,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:30.997876+0000","last_change":"2023-10-30T14:14:03.741268+0000","last_active":"2023-10-30T14:51:30.997876+0000","last_peered":"2023-10-30T14:51:30.997876+0000","last_clean":"2023-10-30T14:51:30.997876+0000","last_became_active":"2023-10-30T14:14:03.740648+0000","last_became_peered":"2023-10-30T14:14:03.740648+0000","last_unstale":"2023-10-30T14:51:30.997876+0000","last_undegraded":"2023-10-30T14:51:30.997876+0000","last_fullsized":"2023-10-30T14:51:30.997876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:30.799148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":98,"num_read_kb":107365,"num_write":798,"num_write_kb":50458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"67'1295","reported_seq":3276,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.356137+0000","last_change":"2023-10-30T14:23:31.073589+0000","last_active":"2023-10-30T14:51:34.356137+0000","last_peered":"2023-10-30T14:51:34.356137+0000","last_clean":"2023-10-30T14:51:34.356137+0000","last_became_active":"2023-10-30T14:14:04.622921+0000","last_became_peered":"2023-10-30T14:14:04.622921+0000","last_unstale":"2023-10-30T14:51:34.356137+0000","last_undegraded":"2023-10-30T14:51:34.356137+0000","last_fullsized":"2023-10-30T14:51:34.356137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:04:46.396711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0071219910000000003,"stat_sum":{"num_bytes":20799446,"num_objects":465,"num_object_clones":210,"num_object_copies":930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":465,"num_whiteouts":183,"num_read":75,"num_read_kb":12323,"num_write":1101,"num_write_kb":26545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"67'1332","reported_seq":3286,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.606774+0000","last_change":"2023-10-30T14:23:30.779989+0000","last_active":"2023-10-30T14:51:34.606774+0000","last_peered":"2023-10-30T14:51:34.606774+0000","last_clean":"2023-10-30T14:51:34.606774+0000","last_became_active":"2023-10-30T14:14:04.621885+0000","last_became_peered":"2023-10-30T14:14:04.621885+0000","last_unstale":"2023-10-30T14:51:34.606774+0000","last_undegraded":"2023-10-30T14:51:34.606774+0000","last_fullsized":"2023-10-30T14:51:34.606774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1332,"log_dups_size":0,"ondisk_log_size":1332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:59:52.839990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067851329999999996,"stat_sum":{"num_bytes":13483294,"num_objects":487,"num_object_clones":228,"num_object_copies":974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":191,"num_read":68,"num_read_kb":8221,"num_write":1123,"num_write_kb":20773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"67'469","reported_seq":619,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.021221+0000","last_change":"2023-10-30T14:14:03.740897+0000","last_active":"2023-10-30T14:51:34.021221+0000","last_peered":"2023-10-30T14:51:34.021221+0000","last_clean":"2023-10-30T14:51:34.021221+0000","last_became_active":"2023-10-30T14:14:03.740247+0000","last_became_peered":"2023-10-30T14:14:03.740247+0000","last_unstale":"2023-10-30T14:51:34.021221+0000","last_undegraded":"2023-10-30T14:51:34.021221+0000","last_fullsized":"2023-10-30T14:51:34.021221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:48:29.601978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":149,"num_read_kb":82851,"num_write":526,"num_write_kb":42289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'1418","reported_seq":3361,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.595418+0000","last_change":"2023-10-30T14:23:30.776236+0000","last_active":"2023-10-30T14:51:34.595418+0000","last_peered":"2023-10-30T14:51:34.595418+0000","last_clean":"2023-10-30T14:51:34.595418+0000","last_became_active":"2023-10-30T14:14:04.623638+0000","last_became_peered":"2023-10-30T14:14:04.623638+0000","last_unstale":"2023-10-30T14:51:34.595418+0000","last_undegraded":"2023-10-30T14:51:34.595418+0000","last_fullsized":"2023-10-30T14:51:34.595418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1418,"log_dups_size":0,"ondisk_log_size":1418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:37:06.313676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049828509999999999,"stat_sum":{"num_bytes":25534330,"num_objects":546,"num_object_clones":248,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":546,"num_whiteouts":220,"num_read":77,"num_read_kb":19614,"num_write":1187,"num_write_kb":31851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"67'1028","reported_seq":1181,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.519728+0000","last_change":"2023-10-30T14:14:03.733233+0000","last_active":"2023-10-30T14:51:34.519728+0000","last_peered":"2023-10-30T14:51:34.519728+0000","last_clean":"2023-10-30T14:51:34.519728+0000","last_became_active":"2023-10-30T14:14:03.732538+0000","last_became_peered":"2023-10-30T14:14:03.732538+0000","last_unstale":"2023-10-30T14:51:34.519728+0000","last_undegraded":"2023-10-30T14:51:34.519728+0000","last_fullsized":"2023-10-30T14:51:34.519728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1028,"log_dups_size":0,"ondisk_log_size":1028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:25.696173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":137,"num_read_kb":150323,"num_write":1072,"num_write_kb":62947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1288","reported_seq":3238,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.774893+0000","last_change":"2023-10-30T14:23:30.776451+0000","last_active":"2023-10-30T14:51:34.774893+0000","last_peered":"2023-10-30T14:51:34.774893+0000","last_clean":"2023-10-30T14:51:34.774893+0000","last_became_active":"2023-10-30T14:14:04.611773+0000","last_became_peered":"2023-10-30T14:14:04.611773+0000","last_unstale":"2023-10-30T14:51:34.774893+0000","last_undegraded":"2023-10-30T14:51:34.774893+0000","last_fullsized":"2023-10-30T14:51:34.774893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1288,"log_dups_size":0,"ondisk_log_size":1288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:55.244396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044924040000000002,"stat_sum":{"num_bytes":26364015,"num_objects":513,"num_object_clones":233,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":513,"num_whiteouts":205,"num_read":66,"num_read_kb":19602,"num_write":1073,"num_write_kb":27528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"67'733","reported_seq":874,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.433821+0000","last_change":"2023-10-30T14:14:03.703988+0000","last_active":"2023-10-30T14:51:34.433821+0000","last_peered":"2023-10-30T14:51:34.433821+0000","last_clean":"2023-10-30T14:51:34.433821+0000","last_became_active":"2023-10-30T14:14:03.703691+0000","last_became_peered":"2023-10-30T14:14:03.703691+0000","last_unstale":"2023-10-30T14:51:34.433821+0000","last_undegraded":"2023-10-30T14:51:34.433821+0000","last_fullsized":"2023-10-30T14:51:34.433821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":733,"log_dups_size":0,"ondisk_log_size":733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:34.538541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":102,"num_read_kb":111696,"num_write":764,"num_write_kb":58666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"67'1338","reported_seq":3272,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.432547+0000","last_change":"2023-10-30T14:23:31.069185+0000","last_active":"2023-10-30T14:51:34.432547+0000","last_peered":"2023-10-30T14:51:34.432547+0000","last_clean":"2023-10-30T14:51:34.432547+0000","last_became_active":"2023-10-30T14:14:04.613420+0000","last_became_peered":"2023-10-30T14:14:04.613420+0000","last_unstale":"2023-10-30T14:51:34.432547+0000","last_undegraded":"2023-10-30T14:51:34.432547+0000","last_fullsized":"2023-10-30T14:51:34.432547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1338,"log_dups_size":0,"ondisk_log_size":1338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:51:35.517831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004010128,"stat_sum":{"num_bytes":13027616,"num_objects":509,"num_object_clones":238,"num_object_copies":1018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":509,"num_whiteouts":202,"num_read":55,"num_read_kb":8221,"num_write":1122,"num_write_kb":14511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"67'639","reported_seq":800,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:33.640569+0000","last_change":"2023-10-30T14:14:03.737319+0000","last_active":"2023-10-30T14:51:33.640569+0000","last_peered":"2023-10-30T14:51:33.640569+0000","last_clean":"2023-10-30T14:51:33.640569+0000","last_became_active":"2023-10-30T14:14:03.736964+0000","last_became_peered":"2023-10-30T14:14:03.736964+0000","last_unstale":"2023-10-30T14:51:33.640569+0000","last_undegraded":"2023-10-30T14:51:33.640569+0000","last_fullsized":"2023-10-30T14:51:33.640569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:41.469809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":182,"num_read_kb":58134,"num_write":696,"num_write_kb":34123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1055045924,"num_objects":31721,"num_object_clones":14708,"num_object_copies":63442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31721,"num_whiteouts":12864,"num_read":4602,"num_read_kb":662584,"num_write":69913,"num_write_kb":1350127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2186289152,"data_stored":2111320648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84634,"ondisk_log_size":84634,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":688331425,"num_objects":1357,"num_object_clones":0,"num_object_copies":2714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":6032,"num_read_kb":2835146,"num_write":28210,"num_write_kb":1521820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1128,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1377132544,"data_stored":1376990268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25652,"ondisk_log_size":25652,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575306,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623332,"kb_used_data":488092,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":93748508,"statfs":{"total":96636764160,"available":95998472192,"internally_reserved":0,"allocated":499806208,"data_stored":488954176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":138475979},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.077,"15min":0.997},"min":{"1min":0.51700000000000002,"5min":0.46300000000000002,"15min":0.45500000000000002},"max":{"1min":6.3040000000000003,"5min":6.399,"15min":6.399},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.2370000000000001,"5min":0.97799999999999998,"15min":0.94399999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":6.4969999999999999,"5min":6.4969999999999999,"15min":6.4969999999999999},"last":1.087}]},{"osd":1,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":0.92500000000000004,"15min":0.93000000000000005},"min":{"1min":0.61599999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":6.1390000000000002,"5min":6.1390000000000002,"15min":6.1390000000000002},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.96599999999999997,"15min":0.99399999999999999},"min":{"1min":0.58699999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":3.8460000000000001,"5min":6.1920000000000002,"15min":6.1920000000000002},"last":0.66700000000000004}]},{"osd":2,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.0449999999999999,"15min":0.96899999999999997},"min":{"1min":0.70099999999999996,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.5739999999999998,"5min":6.5739999999999998,"15min":6.5739999999999998},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.115,"15min":1.01},"min":{"1min":0.81599999999999995,"5min":0.54500000000000004,"15min":0.5},"max":{"1min":6.2350000000000003,"5min":6.5149999999999997,"15min":6.5149999999999997},"last":1.2070000000000001}]},{"osd":3,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.86199999999999999,"15min":0.85599999999999998},"min":{"1min":0.56200000000000006,"5min":0.54800000000000004,"15min":0.53800000000000003},"max":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":4.3230000000000004},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.121,"15min":1.056},"min":{"1min":0.66800000000000004,"5min":0.64800000000000002,"15min":0.54300000000000004},"max":{"1min":3.157,"5min":6.2670000000000003,"15min":6.2670000000000003},"last":1.036}]},{"osd":4,"last update":"Mon Oct 30 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.95599999999999996,"15min":0.95699999999999996},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.54800000000000004},"max":{"1min":4.7409999999999997,"5min":4.7409999999999997,"15min":4.7409999999999997},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.036,"15min":0.97399999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":8.0719999999999992,"5min":8.0719999999999992,"15min":8.0719999999999992},"last":1.125}]},{"osd":5,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3280000000000001,"15min":1.1759999999999999},"min":{"1min":0.78100000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":10.692,"5min":10.692,"15min":10.692},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.3939999999999999,"15min":1.2869999999999999},"min":{"1min":0.72099999999999997,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":6.6429999999999998,"5min":6.6429999999999998,"15min":6.6429999999999998},"last":1.2709999999999999}]},{"osd":6,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.135,"15min":1.129},"min":{"1min":0.748,"5min":0.71599999999999997,"15min":0.62},"max":{"1min":5.2160000000000002,"5min":5.2160000000000002,"15min":5.2160000000000002},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.1639999999999999,"15min":1.107},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":4.367,"5min":4.367,"15min":4.367},"last":1.296}]}]},{"osd":6,"up_from":13,"seq":55834575306,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513220,"kb_used_data":387708,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":93858620,"statfs":{"total":96636764160,"available":96111226880,"internally_reserved":0,"allocated":397012992,"data_stored":389718204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":128514503},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,32,25],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.82899999999999996,"15min":0.83199999999999996},"min":{"1min":0.503,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.216,"5min":1.288,"15min":1.2929999999999999},"last":9.7949999999999999},{"interface":"front","average":{"1min":0.997,"5min":0.90500000000000003,"15min":0.88800000000000001},"min":{"1min":0.68600000000000005,"5min":0.435,"15min":0.435},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.391},"last":9.0180000000000007}]},{"osd":1,"last update":"Mon Oct 30 14:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.89700000000000002,"15min":0.85599999999999998},"min":{"1min":0.66000000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.135,"5min":8,"15min":8},"last":10.169},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.94199999999999995,"15min":0.96599999999999997},"min":{"1min":0.67300000000000004,"5min":0.53400000000000003,"15min":0.318},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.514},"last":8.4169999999999998}]},{"osd":2,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.89300000000000002,"15min":0.89000000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.40999999999999998},"max":{"1min":1.232,"5min":1.2410000000000001,"15min":1.617},"last":5.3819999999999997},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.92900000000000005,"15min":0.94299999999999995},"min":{"1min":0.63400000000000001,"5min":0.59999999999999998,"15min":0.49299999999999999},"max":{"1min":1.181,"5min":1.3149999999999999,"15min":1.4119999999999999},"last":4.8949999999999996}]},{"osd":3,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":0.997,"15min":0.98099999999999998},"min":{"1min":0.73999999999999999,"5min":0.57599999999999996,"15min":0.47799999999999998},"max":{"1min":1.3700000000000001,"5min":1.3720000000000001,"15min":1.6890000000000001},"last":14.827999999999999},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.95499999999999996,"15min":0.96099999999999997},"min":{"1min":0.61499999999999999,"5min":0.59199999999999997,"15min":0.48899999999999999},"max":{"1min":1.286,"5min":1.286,"15min":1.472},"last":4.9279999999999999}]},{"osd":4,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.0309999999999999,"15min":1.032},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.59899999999999998},"max":{"1min":1.1830000000000001,"5min":3.8399999999999999,"15min":3.8399999999999999},"last":3.444},{"interface":"front","average":{"1min":1.016,"5min":1.0269999999999999,"15min":1.04},"min":{"1min":0.69999999999999996,"5min":0.62,"15min":0.435},"max":{"1min":1.341,"5min":1.4730000000000001,"15min":1.645},"last":3.6419999999999999}]},{"osd":5,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":1.0149999999999999,"15min":0.97299999999999998},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.51100000000000001},"max":{"1min":1.1659999999999999,"5min":11.728,"15min":11.728},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.1000000000000001,"15min":1.0920000000000001},"min":{"1min":0.72899999999999998,"5min":0.51700000000000002,"15min":0.51400000000000001},"max":{"1min":1.2310000000000001,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":28.178999999999998}]},{"osd":7,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.224,"15min":1.137},"min":{"1min":0.76400000000000001,"5min":0.755,"15min":0.65200000000000002},"max":{"1min":1.268,"5min":18.041,"15min":18.041},"last":9.7609999999999992},{"interface":"front","average":{"1min":1.181,"5min":1.345,"15min":1.228},"min":{"1min":0.98699999999999999,"5min":0.79700000000000004,"15min":0.56000000000000005},"max":{"1min":1.579,"5min":20.375,"15min":20.375},"last":53.113}]}]},{"osd":5,"up_from":13,"seq":55834575306,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674348,"kb_used_data":539108,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":93697492,"statfs":{"total":96636764160,"available":95946231808,"internally_reserved":0,"allocated":552046592,"data_stored":540347463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":138475974},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":0.83699999999999997,"15min":0.82099999999999995},"min":{"1min":0.54400000000000004,"5min":0.43099999999999999,"15min":0.40100000000000002},"max":{"1min":7.04,"5min":7.04,"15min":7.04},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.131,"5min":0.876,"15min":0.85899999999999999},"min":{"1min":0.53500000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":5.4720000000000004,"5min":5.4720000000000004,"15min":5.4720000000000004},"last":0.75}]},{"osd":1,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83199999999999996,"15min":0.88},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.284,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.83699999999999997,"15min":0.85099999999999998},"min":{"1min":0.57999999999999996,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":4.1630000000000003,"5min":4.1630000000000003,"15min":4.1630000000000003},"last":0.97499999999999998}]},{"osd":2,"last update":"Mon Oct 30 14:51:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.92300000000000004,"15min":0.95999999999999996},"min":{"1min":0.61299999999999999,"5min":0.503,"15min":0.44800000000000001},"max":{"1min":1.3320000000000001,"5min":1.3640000000000001,"15min":2.044},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.087,"15min":1.024},"min":{"1min":0.81899999999999995,"5min":0.69099999999999995,"15min":0.51600000000000001},"max":{"1min":1.425,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":0.94299999999999995}]},{"osd":3,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.85599999999999998,"15min":0.93899999999999995},"min":{"1min":0.623,"5min":0.58399999999999996,"15min":0.56499999999999995},"max":{"1min":1.1000000000000001,"5min":1.2010000000000001,"15min":1.4550000000000001},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.03,"15min":0.97199999999999998},"min":{"1min":0.75,"5min":0.623,"15min":0.52100000000000002},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.4059999999999999},"last":1.2410000000000001}]},{"osd":4,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.149,"15min":1.0469999999999999},"min":{"1min":0.622,"5min":0.62,"15min":0.59099999999999997},"max":{"1min":8.7360000000000007,"5min":8.7360000000000007,"15min":8.7360000000000007},"last":0.80900000000000005},{"interface":"front","average":{"1min":2.335,"5min":1.244,"15min":1.0640000000000001},"min":{"1min":0.621,"5min":0.61199999999999999,"15min":0.59399999999999997},"max":{"1min":23.640000000000001,"5min":23.640000000000001,"15min":23.640000000000001},"last":1.089}]},{"osd":6,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1339999999999999,"15min":1.0700000000000001},"min":{"1min":0.80100000000000005,"5min":0.73399999999999999,"15min":0.55500000000000005},"max":{"1min":1.546,"5min":1.7470000000000001,"15min":2.306},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.194,"15min":1.127},"min":{"1min":0.878,"5min":0.875,"15min":0.57899999999999996},"max":{"1min":1.575,"5min":1.7569999999999999,"15min":2.774},"last":1.343}]},{"osd":7,"last update":"Mon Oct 30 14:50:48 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.99399999999999999,"15min":1.0629999999999999},"min":{"1min":0.71899999999999997,"5min":0.63,"15min":0.40799999999999997},"max":{"1min":1.198,"5min":1.3100000000000001,"15min":3.0739999999999998},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.036,"5min":1.129,"15min":1.1739999999999999},"min":{"1min":0.87,"5min":0.87,"15min":0.77600000000000002},"max":{"1min":1.244,"5min":1.53,"15min":2.1339999999999999},"last":1.052}]}]},{"osd":4,"up_from":13,"seq":55834575306,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":615064,"kb_used_data":479824,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":93756776,"statfs":{"total":96636764160,"available":96006938624,"internally_reserved":0,"allocated":491339776,"data_stored":479836699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":138475974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,10,3,5,1,1,16,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.0109999999999999,"15min":0.876},"min":{"1min":0.53300000000000003,"5min":0.48199999999999998,"15min":0.27900000000000003},"max":{"1min":3.222,"5min":7.1870000000000003,"15min":7.1870000000000003},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.099,"5min":1.107,"15min":0.95099999999999996},"min":{"1min":0.68899999999999995,"5min":0.48199999999999998,"15min":0.33900000000000002},"max":{"1min":3.3010000000000002,"5min":10.864000000000001,"15min":10.864000000000001},"last":0.98199999999999998}]},{"osd":1,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.04,"15min":0.95199999999999996},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.50700000000000001},"max":{"1min":3.3780000000000001,"5min":7.6980000000000004,"15min":7.6980000000000004},"last":1.016},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.91300000000000003,"15min":0.83999999999999997},"min":{"1min":0.58699999999999997,"5min":0.48399999999999999,"15min":0.36599999999999999},"max":{"1min":2.1789999999999998,"5min":5.5259999999999998,"15min":5.5259999999999998},"last":1.798}]},{"osd":2,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0229999999999999,"15min":0.96199999999999997},"min":{"1min":0.64300000000000002,"5min":0.57499999999999996,"15min":0.40899999999999997},"max":{"1min":2.2839999999999998,"5min":5.423,"15min":5.423},"last":1.756},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.254,"15min":1.0940000000000001},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.54400000000000004},"max":{"1min":3.4500000000000002,"5min":13.221,"15min":13.221},"last":1.0449999999999999}]},{"osd":3,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.96099999999999997,"15min":0.86199999999999999},"min":{"1min":0.626,"5min":0.54800000000000004,"15min":0.46899999999999997},"max":{"1min":7.5620000000000003,"5min":7.5620000000000003,"15min":7.5620000000000003},"last":0.871},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.0860000000000001,"15min":1.0029999999999999},"min":{"1min":0.71199999999999997,"5min":0.65500000000000003,"15min":0.497},"max":{"1min":7.7370000000000001,"5min":7.7370000000000001,"15min":7.7370000000000001},"last":0.91900000000000004}]},{"osd":5,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.091,"15min":0.97599999999999998},"min":{"1min":0.72499999999999998,"5min":0.64200000000000002,"15min":0.317},"max":{"1min":7.6360000000000001,"5min":7.6360000000000001,"15min":7.6360000000000001},"last":1.833},{"interface":"front","average":{"1min":1.159,"5min":1.2609999999999999,"15min":1.145},"min":{"1min":0.628,"5min":0.628,"15min":0.442},"max":{"1min":3.6389999999999998,"5min":7.2599999999999998,"15min":7.2599999999999998},"last":1.1040000000000001}]},{"osd":6,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1910000000000001,"15min":1.1020000000000001},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.39200000000000002},"max":{"1min":2.379,"5min":5.4729999999999999,"15min":5.4729999999999999},"last":2.8500000000000001},{"interface":"front","average":{"1min":1.381,"5min":1.2969999999999999,"15min":1.1830000000000001},"min":{"1min":0.84299999999999997,"5min":0.73999999999999999,"15min":0.47899999999999998},"max":{"1min":6.5270000000000001,"5min":10.102,"15min":10.102},"last":1.0760000000000001}]},{"osd":7,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.1120000000000001,"15min":1.042},"min":{"1min":0.75700000000000001,"5min":0.69099999999999995,"15min":0.40100000000000002},"max":{"1min":9.1240000000000006,"5min":9.1240000000000006,"15min":9.1240000000000006},"last":13.026999999999999},{"interface":"front","average":{"1min":1.423,"5min":1.2809999999999999,"15min":1.2130000000000001},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.52900000000000003},"max":{"1min":7.0490000000000004,"5min":7.0490000000000004,"15min":7.0490000000000004},"last":1.8939999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575304,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":654872,"kb_used_data":507152,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":93716968,"statfs":{"total":96636764160,"available":95966175232,"internally_reserved":0,"allocated":519323648,"data_stored":507786843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":151255496},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.53600000000000003,"15min":0.48299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.1130000000000004,"5min":4.1130000000000004,"15min":4.1130000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.54200000000000004,"15min":0.51200000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.91100000000000003,"5min":4.5190000000000001,"15min":4.5190000000000001},"last":0.48799999999999999}]},{"osd":1,"last update":"Mon Oct 30 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.58999999999999997,"15min":0.54400000000000004},"min":{"1min":0.33100000000000002,"5min":0.317,"15min":0.314},"max":{"1min":3.988,"5min":3.988,"15min":3.988},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.56399999999999995,"15min":0.54900000000000004},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.29999999999999999},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.41399999999999998}]},{"osd":2,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.65600000000000003,"15min":0.56100000000000005},"min":{"1min":0.31,"5min":0.29199999999999998,"15min":0.27600000000000002},"max":{"1min":1.0740000000000001,"5min":8.6519999999999992,"15min":8.6519999999999992},"last":0.437},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.73999999999999999,"15min":0.63700000000000001},"min":{"1min":0.42199999999999999,"5min":0.41999999999999998,"15min":0.313},"max":{"1min":1.1040000000000001,"5min":7.0949999999999998,"15min":7.0949999999999998},"last":0.504}]},{"osd":4,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.83399999999999996,"15min":0.82099999999999995},"min":{"1min":0.54600000000000004,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.1220000000000001,"5min":4.5730000000000004,"15min":4.5730000000000004},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.90200000000000002,"15min":0.85299999999999998},"min":{"1min":0.51500000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":4.0910000000000002,"5min":4.4720000000000004,"15min":4.4720000000000004},"last":0.68999999999999995}]},{"osd":5,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.0029999999999999,"15min":0.90000000000000002},"min":{"1min":0.72799999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.296,"5min":14.974,"15min":14.974},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.042,"5min":0.96699999999999997,"15min":0.873},"min":{"1min":0.746,"5min":0.52600000000000002,"15min":0.46600000000000003},"max":{"1min":4.0629999999999997,"5min":5.907,"15min":5.907},"last":0.63900000000000001}]},{"osd":6,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.01,"15min":0.91900000000000004},"min":{"1min":0.70899999999999996,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":5.8140000000000001,"5min":5.8300000000000001,"15min":5.8300000000000001},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.159,"5min":1.073,"15min":0.96299999999999997},"min":{"1min":0.70799999999999996,"5min":0.60199999999999998,"15min":0.55100000000000005},"max":{"1min":6.0300000000000002,"5min":6.0300000000000002,"15min":6.0300000000000002},"last":1.248}]},{"osd":7,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":1.115,"15min":0.995},"min":{"1min":0.67600000000000005,"5min":0.66800000000000004,"15min":0.61699999999999999},"max":{"1min":1.2549999999999999,"5min":16.282,"15min":16.282},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.123,"15min":0.97199999999999998},"min":{"1min":0.70999999999999996,"5min":0.627,"15min":0.58099999999999996},"max":{"1min":1.6519999999999999,"5min":16.096,"15min":16.096},"last":1.161}]}]},{"osd":2,"up_from":13,"seq":55834575306,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":418108,"kb_used_data":310644,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":93953732,"statfs":{"total":96636764160,"available":96208621568,"internally_reserved":0,"allocated":318099456,"data_stored":311268353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.45300000000000001,"15min":0.48699999999999999},"min":{"1min":0.30099999999999999,"5min":0.27400000000000002,"15min":0.26300000000000001},"max":{"1min":0.67800000000000005,"5min":0.79100000000000004,"15min":13.526},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.50900000000000001,"15min":0.51300000000000001},"min":{"1min":0.38300000000000001,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":1.2190000000000001},"last":0.51100000000000001}]},{"osd":1,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46300000000000002,"15min":0.47199999999999998},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65500000000000003,"5min":0.67200000000000004,"15min":1.5469999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54400000000000004,"15min":0.56000000000000005},"min":{"1min":0.38700000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73499999999999999,"5min":0.79300000000000004,"15min":1.452},"last":0.53000000000000003}]},{"osd":3,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58799999999999997,"15min":0.60699999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81299999999999994,"5min":0.95399999999999996,"15min":1.9390000000000001},"last":0.44},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.53900000000000003},"min":{"1min":0.35899999999999999,"5min":0.35499999999999998,"15min":0.33100000000000002},"max":{"1min":0.78500000000000003,"5min":0.79300000000000004,"15min":1.913},"last":0.48899999999999999}]},{"osd":4,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.81699999999999995,"15min":0.81499999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.44900000000000001},"max":{"1min":1.0309999999999999,"5min":4.2229999999999999,"15min":4.2229999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.91600000000000004,"15min":0.91000000000000003},"min":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.59599999999999997},"max":{"1min":1.081,"5min":4.2439999999999998,"15min":4.2439999999999998},"last":0.84099999999999997}]},{"osd":5,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.98999999999999999,"15min":0.86399999999999999},"min":{"1min":0.51900000000000002,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0620000000000001,"5min":22.285,"15min":22.285},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.82899999999999996,"15min":0.83399999999999996},"min":{"1min":0.58199999999999996,"5min":0.56000000000000005,"15min":0.51900000000000002},"max":{"1min":1.1319999999999999,"5min":2.3999999999999999,"15min":2.742},"last":0.80000000000000004}]},{"osd":6,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.86099999999999999,"15min":0.91500000000000004},"min":{"1min":0.65000000000000002,"5min":0.58799999999999997,"15min":0.54500000000000004},"max":{"1min":1.0489999999999999,"5min":2.911,"15min":27.731000000000002},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.872,"15min":0.874},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.1679999999999999,"5min":1.9099999999999999,"15min":2.0009999999999999},"last":0.71399999999999997}]},{"osd":7,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84299999999999997,"15min":0.91600000000000004},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.56499999999999995},"max":{"1min":1.105,"5min":1.2390000000000001,"15min":18.870999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.90400000000000003,"15min":0.96399999999999997},"min":{"1min":0.66600000000000004,"5min":0.61499999999999999,"15min":0.55500000000000005},"max":{"1min":1.0860000000000001,"5min":3.3809999999999998,"15min":18.356999999999999},"last":0.875}]}]},{"osd":1,"up_from":13,"seq":55834575305,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468404,"kb_used_data":351212,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":93903436,"statfs":{"total":96636764160,"available":96157118464,"internally_reserved":0,"allocated":359641088,"data_stored":352841256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":119994823},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.47399999999999998,"15min":0.44800000000000001},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.68200000000000005,"5min":2.4889999999999999,"15min":2.4889999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.505,"5min":0.51600000000000001,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71199999999999997,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.58299999999999996}]},{"osd":2,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.64500000000000002,"15min":0.56699999999999995},"min":{"1min":0.318,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79000000000000004,"5min":8.3160000000000007,"15min":8.3160000000000007},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.54000000000000004,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.27200000000000002,"15min":0.26800000000000002},"max":{"1min":0.61399999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.628}]},{"osd":3,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.61099999999999999,"15min":0.59599999999999997},"min":{"1min":0.38800000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84399999999999997,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.623,"5min":0.64900000000000002,"15min":0.63600000000000001},"min":{"1min":0.439,"5min":0.38100000000000001,"15min":0.36499999999999999},"max":{"1min":0.89200000000000002,"5min":3.54,"15min":3.54},"last":0.69399999999999995}]},{"osd":4,"last update":"Mon Oct 30 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.92000000000000004,"15min":0.83999999999999997},"min":{"1min":0.505,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.92600000000000005,"5min":10.525,"15min":10.525},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.94899999999999995,"15min":0.83899999999999997},"min":{"1min":0.68400000000000005,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.997,"5min":10.571,"15min":10.571},"last":0.93200000000000005}]},{"osd":5,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.88900000000000001,"15min":0.89100000000000001},"min":{"1min":0.628,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.0289999999999999,"5min":3.4969999999999999,"15min":3.4969999999999999},"last":1.01},{"interface":"front","average":{"1min":0.876,"5min":0.84399999999999997,"15min":0.79200000000000004},"min":{"1min":0.72399999999999998,"5min":0.56000000000000005,"15min":0.49099999999999999},"max":{"1min":1.1739999999999999,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.98099999999999998}]},{"osd":6,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.80900000000000005},"min":{"1min":0.58399999999999996,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.1040000000000001,"5min":1.875,"15min":2.4710000000000001},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.88900000000000001,"15min":0.83399999999999996},"min":{"1min":0.57399999999999995,"5min":0.53100000000000003,"15min":0.51700000000000002},"max":{"1min":1.2130000000000001,"5min":5.3680000000000003,"15min":5.3680000000000003},"last":1.0680000000000001}]},{"osd":7,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":1.081,"15min":0.91600000000000004},"min":{"1min":0.59599999999999997,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0249999999999999,"5min":19.885000000000002,"15min":19.885000000000002},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.86499999999999999,"5min":1.054,"15min":0.93200000000000005},"min":{"1min":0.60099999999999998,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.127,"5min":13.401999999999999,"15min":13.401999999999999},"last":0.94399999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575307,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553040,"kb_used_data":423368,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":93818800,"statfs":{"total":96636764160,"available":96070451200,"internally_reserved":0,"allocated":433528832,"data_stored":421882906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132774342},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45900000000000002,"15min":0.44800000000000001},"min":{"1min":0.32200000000000001,"5min":0.26800000000000002,"15min":0.22500000000000001},"max":{"1min":0.749,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.53300000000000003,"15min":0.503},"min":{"1min":0.36699999999999999,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":0.68999999999999995,"5min":3.367,"15min":3.367},"last":0.35899999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58199999999999996,"15min":0.53600000000000003},"min":{"1min":0.44400000000000001,"5min":0.378,"15min":0.30499999999999999},"max":{"1min":0.751,"5min":3.222,"15min":3.222},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53200000000000003,"15min":0.56000000000000005},"min":{"1min":0.40000000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":1.095},"last":0.50600000000000001}]},{"osd":3,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.56599999999999995,"15min":0.60199999999999998},"min":{"1min":0.34000000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.63300000000000001,"5min":6.4980000000000002,"15min":6.4980000000000002},"last":0.436},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57499999999999996,"15min":0.52900000000000003},"min":{"1min":0.40400000000000003,"5min":0.40100000000000002,"15min":0.28000000000000003},"max":{"1min":0.71699999999999997,"5min":2.024,"15min":2.024},"last":0.47299999999999998}]},{"osd":4,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.78800000000000003,"15min":0.82599999999999996},"min":{"1min":0.57599999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.95399999999999996,"5min":2.0659999999999998,"15min":2.8380000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81399999999999995,"15min":0.85999999999999999},"min":{"1min":0.65100000000000002,"5min":0.56899999999999995,"15min":0.53900000000000003},"max":{"1min":0.97699999999999998,"5min":2.1000000000000001,"15min":5.8959999999999999},"last":0.76800000000000002}]},{"osd":5,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.872,"15min":0.86099999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.46300000000000002},"max":{"1min":1.212,"5min":1.278,"15min":6.4480000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.92000000000000004,"15min":0.85599999999999998},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.54000000000000004},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":2.6869999999999998},"last":0.81000000000000005}]},{"osd":6,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.83899999999999997,"15min":0.84699999999999998},"min":{"1min":0.64800000000000002,"5min":0.55300000000000005,"15min":0.54400000000000004},"max":{"1min":1.133,"5min":1.2150000000000001,"15min":4.9980000000000002},"last":0.629},{"interface":"front","average":{"1min":0.877,"5min":0.85499999999999998,"15min":0.86099999999999999},"min":{"1min":0.65300000000000002,"5min":0.55100000000000005,"15min":0.54000000000000004},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.5980000000000001},"last":0.70499999999999996}]},{"osd":7,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.86799999999999999,"15min":0.84699999999999998},"min":{"1min":0.63700000000000001,"5min":0.57899999999999996,"15min":0.499},"max":{"1min":1.1120000000000001,"5min":3.6589999999999998,"15min":3.6589999999999998},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.93100000000000005,"15min":0.88400000000000001},"min":{"1min":0.76300000000000001,"5min":0.625,"15min":0.54100000000000004},"max":{"1min":1.1579999999999999,"5min":1.262,"15min":2.6429999999999998},"last":0.72399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":109068288,"data_stored":109052564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163602432,"data_stored":163582366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":115884032,"data_stored":115864160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":207986688,"data_stored":207968820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":153219072,"data_stored":153204613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":215736320,"data_stored":215715446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":211091456,"data_stored":211074246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":200544256,"data_stored":200528053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323084288,"data_stored":311837872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":195256320,"data_stored":188856788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":201433088,"data_stored":195002091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310554624,"data_stored":299415921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":337338368,"data_stored":326229984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":335527936,"data_stored":324229915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":185139200,"data_stored":178241856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":297955328,"data_stored":287506221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:51:42.106 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.106 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.107 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.107 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.108 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.108 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.108 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.109 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.109 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.109 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.110 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.110 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.111 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.111 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.111 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.112 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.112 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.112 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.113 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.113 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.113 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.114 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.114 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.114 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.115 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.115 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.116 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.116 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.116 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.117 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.117 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.117 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.118 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.118 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.119 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.119 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.119 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.120 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.120 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.121 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.121 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.121 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.122 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.122 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.122 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.123 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.123 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.124 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.124 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.124 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.125 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.125 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.126 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.126 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.126 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.127 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.127 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.127 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.128 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.128 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.129 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.129 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.129 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.130 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.130 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-30T14:13:53.220267+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=13, tm_sec=53, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.130 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.131 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.131 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.131 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.132 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.132 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.133 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.133 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.133 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.134 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.134 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.134 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.135 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.135 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.136 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.136 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.136 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.137 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.137 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.138 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.138 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.138 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.139 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.139 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.139 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.140 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.140 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.140 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.141 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.141 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.141 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.142 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:51:42.142 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T14:51:42.170 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:51:42.174 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:52:02.144 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:52:02.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.319+0000 7feffd49c640 1 Processor -- start 2023-10-30T14:52:02.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.319+0000 7feffd49c640 1 -- start start 2023-10-30T14:52:02.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.323+0000 7feffd49c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7feff80c4a10 0x7feff815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:02.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.323+0000 7feffd49c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff8159f20 con 0x7feff80c42a0 2023-10-30T14:52:02.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.323+0000 7feffd49c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff815d620 con 0x7feff8153c80 2023-10-30T14:52:02.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.323+0000 7feffd49c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff815dec0 con 0x7feff80c4a10 2023-10-30T14:52:02.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.323+0000 7feff6ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7feff8153c80 0x7feff815a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.57:33338/0 (socket says 172.21.15.57:33338) 2023-10-30T14:52:02.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.323+0000 7feff6ffd640 1 -- 172.21.15.57:0/1279037630 learned_addr learned my addr 172.21.15.57:0/1279037630 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:52:02.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.323+0000 7feff67fc640 1 --2- 172.21.15.57:0/1279037630 >> v2:172.21.15.57:3300/0 conn(0x7feff80c4a10 0x7feff815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:02.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.323+0000 7feff5ffb640 1 -- 172.21.15.57:0/1279037630 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411136837 0 0) 0x7feff815d620 con 0x7feff8153c80 2023-10-30T14:52:02.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.323+0000 7feff5ffb640 1 -- 172.21.15.57:0/1279037630 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe0003530 con 0x7feff8153c80 2023-10-30T14:52:02.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.323+0000 7feff67fc640 1 -- 172.21.15.57:0/1279037630 >> v1:172.21.15.88:6789/0 conn(0x7feff8153c80 legacy=0x7feff815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:02.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.323+0000 7feff67fc640 1 -- 172.21.15.57:0/1279037630 >> v1:172.21.15.57:6789/0 conn(0x7feff80c42a0 legacy=0x7feff80ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:52:02.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.323+0000 7feff67fc640 1 -- 172.21.15.57:0/1279037630 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff815d900 con 0x7feff80c4a10 2023-10-30T14:52:02.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.327+0000 7feff67fc640 1 --2- 172.21.15.57:0/1279037630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feff80c4a10 0x7feff815cf90 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fefec0097c0 tx=0x7fefec01ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cbd63f558e73d40 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:02.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.327+0000 7feff5ffb640 1 -- 172.21.15.57:0/1279037630 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec0a8930 con 0x7feff80c4a10 2023-10-30T14:52:02.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.327+0000 7feff5ffb640 1 -- 172.21.15.57:0/1279037630 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefec0a8a90 con 0x7feff80c4a10 2023-10-30T14:52:02.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.327+0000 7feff5ffb640 1 -- 172.21.15.57:0/1279037630 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec0a8e00 con 0x7feff80c4a10 2023-10-30T14:52:02.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.327+0000 7feffd49c640 1 -- 172.21.15.57:0/1279037630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feff80c4a10 msgr2=0x7feff815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:02.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.327+0000 7feffd49c640 1 --2- 172.21.15.57:0/1279037630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feff80c4a10 0x7feff815cf90 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fefec0097c0 tx=0x7fefec01ec30 comp rx=0 tx=0).stop 2023-10-30T14:52:02.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.327+0000 7feffd49c640 1 -- 172.21.15.57:0/1279037630 shutdown_connections 2023-10-30T14:52:02.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.327+0000 7feffd49c640 1 --2- 172.21.15.57:0/1279037630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feff80c4a10 0x7feff815cf90 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:02.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.327+0000 7feffd49c640 1 -- 172.21.15.57:0/1279037630 >> 172.21.15.57:0/1279037630 conn(0x7feff80715c0 msgr2=0x7feff80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:02.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.327+0000 7feffd49c640 1 -- 172.21.15.57:0/1279037630 shutdown_connections 2023-10-30T14:52:02.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.327+0000 7feffd49c640 1 -- 172.21.15.57:0/1279037630 wait complete. 2023-10-30T14:52:02.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.327+0000 7feffd49c640 1 Processor -- start 2023-10-30T14:52:02.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.327+0000 7feffd49c640 1 -- start start 2023-10-30T14:52:02.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feffd49c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feff80c42a0 0x7feff8146830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:02.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feffd49c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feff80c4a10 0x7feff815b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:02.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feffd49c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feff8153c80 0x7feff81ef000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:02.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feffd49c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff815bc20 con 0x7feff8153c80 2023-10-30T14:52:02.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feffd49c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff81ef540 con 0x7feff80c4a10 2023-10-30T14:52:02.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feffd49c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff815b250 con 0x7feff80c42a0 2023-10-30T14:52:02.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feff67fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feff80c4a10 0x7feff815b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:02.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feff77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feff8153c80 0x7feff81ef000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:02.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feff6ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feff80c42a0 0x7feff8146830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:02.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feff67fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feff80c4a10 0x7feff815b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.57:34088/0 (socket says 172.21.15.57:34088) 2023-10-30T14:52:02.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feff67fc640 1 -- 172.21.15.57:0/2325738851 learned_addr learned my addr 172.21.15.57:0/2325738851 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:52:02.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feff77fe640 1 -- 172.21.15.57:0/2325738851 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feff80c42a0 msgr2=0x7feff8146830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:02.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feff77fe640 1 --2- 172.21.15.57:0/2325738851 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feff80c42a0 0x7feff8146830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:02.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feff77fe640 1 -- 172.21.15.57:0/2325738851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feff80c4a10 msgr2=0x7feff815b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:02.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feff77fe640 1 --2- 172.21.15.57:0/2325738851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feff80c4a10 0x7feff815b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:02.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feff77fe640 1 -- 172.21.15.57:0/2325738851 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec009420 con 0x7feff8153c80 2023-10-30T14:52:02.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feff77fe640 1 --2- 172.21.15.57:0/2325738851 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feff8153c80 0x7feff81ef000 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7feff000c780 tx=0x7feff000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:02.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7fefdf7fe640 1 -- 172.21.15.57:0/2325738851 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff000aa50 con 0x7feff8153c80 2023-10-30T14:52:02.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7fefdf7fe640 1 -- 172.21.15.57:0/2325738851 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff000abb0 con 0x7feff8153c80 2023-10-30T14:52:02.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7fefdf7fe640 1 -- 172.21.15.57:0/2325738851 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff000ce30 con 0x7feff8153c80 2023-10-30T14:52:02.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feffd49c640 1 -- 172.21.15.57:0/2325738851 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff81480a0 con 0x7feff8153c80 2023-10-30T14:52:02.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feffd49c640 1 -- 172.21.15.57:0/2325738851 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff815ac00 con 0x7feff8153c80 2023-10-30T14:52:02.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.331+0000 7feffd49c640 1 -- 172.21.15.57:0/2325738851 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff815b3c0 con 0x7feff8153c80 2023-10-30T14:52:02.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.335+0000 7fefdf7fe640 1 -- 172.21.15.57:0/2325738851 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feff0012420 con 0x7feff8153c80 2023-10-30T14:52:02.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.335+0000 7fefdf7fe640 1 --2- 172.21.15.57:0/2325738851 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fefe40664a0 0x7fefe4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:02.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.335+0000 7fefdf7fe640 1 -- 172.21.15.57:0/2325738851 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7feff00844f0 con 0x7feff8153c80 2023-10-30T14:52:02.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.335+0000 7feff6ffd640 1 --2- 172.21.15.57:0/2325738851 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fefe40664a0 0x7fefe4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:02.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.335+0000 7feff6ffd640 1 --2- 172.21.15.57:0/2325738851 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fefe40664a0 0x7fefe4068960 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fefe80011f0 tx=0x7fefe8007d40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:02.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.339+0000 7fefdf7fe640 1 -- 172.21.15.57:0/2325738851 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feff0051310 con 0x7feff8153c80 2023-10-30T14:52:02.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.595+0000 7feffd49c640 1 -- 172.21.15.57:0/2325738851 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feff8159bb0 con 0x7fefe40664a0 2023-10-30T14:52:02.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.619+0000 7fefdf7fe640 1 -- 172.21.15.57:0/2325738851 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+292967 (secure 0 0 0) 0x7fefe8005860 con 0x7fefe40664a0 2023-10-30T14:52:02.626 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:52:02.628 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-30T14:52:02.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.623+0000 7feffd49c640 1 -- 172.21.15.57:0/2325738851 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fefe40664a0 msgr2=0x7fefe4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:02.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.623+0000 7feffd49c640 1 --2- 172.21.15.57:0/2325738851 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fefe40664a0 0x7fefe4068960 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fefe80011f0 tx=0x7fefe8007d40 comp rx=0 tx=0).stop 2023-10-30T14:52:02.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.623+0000 7feffd49c640 1 -- 172.21.15.57:0/2325738851 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feff8153c80 msgr2=0x7feff81ef000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:02.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.623+0000 7feffd49c640 1 --2- 172.21.15.57:0/2325738851 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feff8153c80 0x7feff81ef000 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7feff000c780 tx=0x7feff000cc40 comp rx=0 tx=0).stop 2023-10-30T14:52:02.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.623+0000 7feffd49c640 1 -- 172.21.15.57:0/2325738851 shutdown_connections 2023-10-30T14:52:02.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.623+0000 7feffd49c640 1 --2- 172.21.15.57:0/2325738851 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fefe40664a0 0x7fefe4068960 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:02.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.623+0000 7feffd49c640 1 --2- 172.21.15.57:0/2325738851 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feff8153c80 0x7feff81ef000 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:02.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.623+0000 7feffd49c640 1 --2- 172.21.15.57:0/2325738851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feff80c4a10 0x7feff815b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:02.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.623+0000 7feffd49c640 1 --2- 172.21.15.57:0/2325738851 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feff80c42a0 0x7feff8146830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:02.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.623+0000 7feffd49c640 1 -- 172.21.15.57:0/2325738851 >> 172.21.15.57:0/2325738851 conn(0x7feff80715c0 msgr2=0x7feff8148970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:02.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.623+0000 7feffd49c640 1 -- 172.21.15.57:0/2325738851 shutdown_connections 2023-10-30T14:52:02.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:02.623+0000 7feffd49c640 1 -- 172.21.15.57:0/2325738851 wait complete. 2023-10-30T14:52:02.657 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":1205,"stamp":"2023-10-30T14:52:01.325993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1745330878,"num_objects":33080,"num_object_clones":14708,"num_object_copies":66160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33080,"num_whiteouts":12864,"num_read":10750,"num_read_kb":3497830,"num_write":98307,"num_write_kb":2875704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1127,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13231903,"num_omap_keys":22455,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118662,"ondisk_log_size":118662,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":4535280,"kb_used_data":3489520,"kb_used_omap":12,"kb_used_meta":1045683,"kb_avail":750439440,"statfs":{"total":773094113280,"available":768449986560,"internally_reserved":0,"allocated":3573268480,"data_stored":3495107372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":1070779961},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":334973,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4775800,"num_omap_keys":8537,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003834"},"pg_stats":[{"pgid":"3.3f","version":"67'1292","reported_seq":3255,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:37.470671+0000","last_change":"2023-10-30T14:23:31.082534+0000","last_active":"2023-10-30T14:51:37.470671+0000","last_peered":"2023-10-30T14:51:37.470671+0000","last_clean":"2023-10-30T14:51:37.470671+0000","last_became_active":"2023-10-30T14:14:04.650882+0000","last_became_peered":"2023-10-30T14:14:04.650882+0000","last_unstale":"2023-10-30T14:51:37.470671+0000","last_undegraded":"2023-10-30T14:51:37.470671+0000","last_fullsized":"2023-10-30T14:51:37.470671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1292,"log_dups_size":0,"ondisk_log_size":1292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010330941,"stat_sum":{"num_bytes":16885684,"num_objects":463,"num_object_clones":217,"num_object_copies":926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":463,"num_whiteouts":185,"num_read":62,"num_read_kb":8246,"num_write":1079,"num_write_kb":18499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"67'1284","reported_seq":3240,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:37.094946+0000","last_change":"2023-10-30T14:23:30.771600+0000","last_active":"2023-10-30T14:51:37.094946+0000","last_peered":"2023-10-30T14:51:37.094946+0000","last_clean":"2023-10-30T14:51:37.094946+0000","last_became_active":"2023-10-30T14:14:04.587942+0000","last_became_peered":"2023-10-30T14:14:04.587942+0000","last_unstale":"2023-10-30T14:51:37.094946+0000","last_undegraded":"2023-10-30T14:51:37.094946+0000","last_fullsized":"2023-10-30T14:51:37.094946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1284,"log_dups_size":0,"ondisk_log_size":1284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002850184,"stat_sum":{"num_bytes":13093874,"num_objects":485,"num_object_clones":221,"num_object_copies":970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":196,"num_read":59,"num_read_kb":8220,"num_write":1070,"num_write_kb":14962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"67'1307","reported_seq":3271,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:36.997710+0000","last_change":"2023-10-30T14:23:30.776754+0000","last_active":"2023-10-30T14:51:36.997710+0000","last_peered":"2023-10-30T14:51:36.997710+0000","last_clean":"2023-10-30T14:51:36.997710+0000","last_became_active":"2023-10-30T14:14:04.590700+0000","last_became_peered":"2023-10-30T14:14:04.590700+0000","last_unstale":"2023-10-30T14:51:36.997710+0000","last_undegraded":"2023-10-30T14:51:36.997710+0000","last_fullsized":"2023-10-30T14:51:36.997710+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1307,"log_dups_size":0,"ondisk_log_size":1307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042861139999999997,"stat_sum":{"num_bytes":12610897,"num_objects":491,"num_object_clones":227,"num_object_copies":982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":204,"num_read":73,"num_read_kb":16409,"num_write":1065,"num_write_kb":22118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"67'1308","reported_seq":3269,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.636998+0000","last_change":"2023-10-30T14:23:31.067598+0000","last_active":"2023-10-30T14:51:34.636998+0000","last_peered":"2023-10-30T14:51:34.636998+0000","last_clean":"2023-10-30T14:51:34.636998+0000","last_became_active":"2023-10-30T14:14:45.661677+0000","last_became_peered":"2023-10-30T14:14:45.661677+0000","last_unstale":"2023-10-30T14:51:34.636998+0000","last_undegraded":"2023-10-30T14:51:34.636998+0000","last_fullsized":"2023-10-30T14:51:34.636998+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1308,"log_dups_size":0,"ondisk_log_size":1308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003131572,"stat_sum":{"num_bytes":8809488,"num_objects":483,"num_object_clones":217,"num_object_copies":966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":483,"num_whiteouts":190,"num_read":63,"num_read_kb":69,"num_write":1091,"num_write_kb":15023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"67'1772","reported_seq":3760,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:46.289666+0000","last_change":"2023-10-30T14:51:46.289749+0000","last_active":"2023-10-30T14:51:46.289666+0000","last_peered":"2023-10-30T14:51:46.289666+0000","last_clean":"2023-10-30T14:51:46.289666+0000","last_became_active":"2023-10-30T14:14:04.644683+0000","last_became_peered":"2023-10-30T14:14:04.644683+0000","last_unstale":"2023-10-30T14:51:46.289666+0000","last_undegraded":"2023-10-30T14:51:46.289666+0000","last_fullsized":"2023-10-30T14:51:46.289666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1772","last_scrub_stamp":"2023-10-30T14:51:46.289574+0000","last_deep_scrub":"67'1772","last_deep_scrub_stamp":"2023-10-30T14:51:46.289574+0000","last_clean_scrub_stamp":"2023-10-30T14:51:46.289574+0000","objects_scrubbed":486,"log_size":1772,"log_dups_size":0,"ondisk_log_size":1772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:44:45.481060+0000","scrub_duration":0.82105794700000001,"objects_trimmed":0,"snaptrim_duration":0.0052462430000000003,"stat_sum":{"num_bytes":8863750,"num_objects":486,"num_object_clones":224,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":200,"num_read":70,"num_read_kb":4116,"num_write":1084,"num_write_kb":10111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'1291","reported_seq":3260,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:37.102881+0000","last_change":"2023-10-30T14:23:30.777057+0000","last_active":"2023-10-30T14:51:37.102881+0000","last_peered":"2023-10-30T14:51:37.102881+0000","last_clean":"2023-10-30T14:51:37.102881+0000","last_became_active":"2023-10-30T14:14:04.619335+0000","last_became_peered":"2023-10-30T14:14:04.619335+0000","last_unstale":"2023-10-30T14:51:37.102881+0000","last_undegraded":"2023-10-30T14:51:37.102881+0000","last_fullsized":"2023-10-30T14:51:37.102881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1291,"log_dups_size":0,"ondisk_log_size":1291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062283859999999998,"stat_sum":{"num_bytes":20638046,"num_objects":479,"num_object_clones":221,"num_object_copies":958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":195,"num_read":75,"num_read_kb":12254,"num_write":1080,"num_write_kb":22208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"67'1299","reported_seq":3249,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:37.470436+0000","last_change":"2023-10-30T14:23:31.080820+0000","last_active":"2023-10-30T14:51:37.470436+0000","last_peered":"2023-10-30T14:51:37.470436+0000","last_clean":"2023-10-30T14:51:37.470436+0000","last_became_active":"2023-10-30T14:14:04.641323+0000","last_became_peered":"2023-10-30T14:14:04.641323+0000","last_unstale":"2023-10-30T14:51:37.470436+0000","last_undegraded":"2023-10-30T14:51:37.470436+0000","last_fullsized":"2023-10-30T14:51:37.470436+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1299,"log_dups_size":0,"ondisk_log_size":1299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011228897,"stat_sum":{"num_bytes":14710085,"num_objects":493,"num_object_clones":224,"num_object_copies":986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":493,"num_whiteouts":202,"num_read":59,"num_read_kb":8203,"num_write":1083,"num_write_kb":18203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'1701","reported_seq":3685,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:54.076926+0000","last_change":"2023-10-30T14:51:54.076987+0000","last_active":"2023-10-30T14:51:54.076926+0000","last_peered":"2023-10-30T14:51:54.076926+0000","last_clean":"2023-10-30T14:51:54.076926+0000","last_became_active":"2023-10-30T14:14:04.618312+0000","last_became_peered":"2023-10-30T14:14:04.618312+0000","last_unstale":"2023-10-30T14:51:54.076926+0000","last_undegraded":"2023-10-30T14:51:54.076926+0000","last_fullsized":"2023-10-30T14:51:54.076926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1701","last_scrub_stamp":"2023-10-30T14:51:54.076855+0000","last_deep_scrub":"67'1701","last_deep_scrub_stamp":"2023-10-30T14:51:54.076855+0000","last_clean_scrub_stamp":"2023-10-30T14:51:54.076855+0000","objects_scrubbed":477,"log_size":1701,"log_dups_size":0,"ondisk_log_size":1701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:59:06.379444+0000","scrub_duration":0.66338523199999999,"objects_trimmed":0,"snaptrim_duration":0.002849426,"stat_sum":{"num_bytes":12938947,"num_objects":477,"num_object_clones":218,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":194,"num_read":60,"num_read_kb":8209,"num_write":1031,"num_write_kb":15209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"67'1347","reported_seq":3295,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.640833+0000","last_change":"2023-10-30T14:23:31.083143+0000","last_active":"2023-10-30T14:51:34.640833+0000","last_peered":"2023-10-30T14:51:34.640833+0000","last_clean":"2023-10-30T14:51:34.640833+0000","last_became_active":"2023-10-30T14:14:04.647361+0000","last_became_peered":"2023-10-30T14:14:04.647361+0000","last_unstale":"2023-10-30T14:51:34.640833+0000","last_undegraded":"2023-10-30T14:51:34.640833+0000","last_fullsized":"2023-10-30T14:51:34.640833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1347,"log_dups_size":0,"ondisk_log_size":1347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010459703000000001,"stat_sum":{"num_bytes":8547707,"num_objects":513,"num_object_clones":237,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":513,"num_whiteouts":209,"num_read":70,"num_read_kb":8267,"num_write":1124,"num_write_kb":13960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"67'1383","reported_seq":3285,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:35.077385+0000","last_change":"2023-10-30T14:23:31.075673+0000","last_active":"2023-10-30T14:51:35.077385+0000","last_peered":"2023-10-30T14:51:35.077385+0000","last_clean":"2023-10-30T14:51:35.077385+0000","last_became_active":"2023-10-30T14:14:04.630959+0000","last_became_peered":"2023-10-30T14:14:04.630959+0000","last_unstale":"2023-10-30T14:51:35.077385+0000","last_undegraded":"2023-10-30T14:51:35.077385+0000","last_fullsized":"2023-10-30T14:51:35.077385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1383,"log_dups_size":0,"ondisk_log_size":1383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079325130000000004,"stat_sum":{"num_bytes":12417236,"num_objects":545,"num_object_clones":289,"num_object_copies":1090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":545,"num_whiteouts":194,"num_read":76,"num_read_kb":4189,"num_write":1096,"num_write_kb":14000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"67'1303","reported_seq":3246,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.421282+0000","last_change":"2023-10-30T14:23:31.075101+0000","last_active":"2023-10-30T14:51:34.421282+0000","last_peered":"2023-10-30T14:51:34.421282+0000","last_clean":"2023-10-30T14:51:34.421282+0000","last_became_active":"2023-10-30T14:14:04.612653+0000","last_became_peered":"2023-10-30T14:14:04.612653+0000","last_unstale":"2023-10-30T14:51:34.421282+0000","last_undegraded":"2023-10-30T14:51:34.421282+0000","last_fullsized":"2023-10-30T14:51:34.421282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1303,"log_dups_size":0,"ondisk_log_size":1303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079098320000000003,"stat_sum":{"num_bytes":17851951,"num_objects":522,"num_object_clones":235,"num_object_copies":1044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":204,"num_read":63,"num_read_kb":9809,"num_write":1080,"num_write_kb":19222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"67'1293","reported_seq":3248,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.635551+0000","last_change":"2023-10-30T14:23:30.957983+0000","last_active":"2023-10-30T14:51:34.635551+0000","last_peered":"2023-10-30T14:51:34.635551+0000","last_clean":"2023-10-30T14:51:34.635551+0000","last_became_active":"2023-10-30T14:14:04.621135+0000","last_became_peered":"2023-10-30T14:14:04.621135+0000","last_unstale":"2023-10-30T14:51:34.635551+0000","last_undegraded":"2023-10-30T14:51:34.635551+0000","last_fullsized":"2023-10-30T14:51:34.635551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1293,"log_dups_size":0,"ondisk_log_size":1293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025042409999999999,"stat_sum":{"num_bytes":14360375,"num_objects":506,"num_object_clones":227,"num_object_copies":1012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":195,"num_read":67,"num_read_kb":6339,"num_write":1082,"num_write_kb":20012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"67'1280","reported_seq":3263,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.593417+0000","last_change":"2023-10-30T14:23:31.070450+0000","last_active":"2023-10-30T14:51:34.593417+0000","last_peered":"2023-10-30T14:51:34.593417+0000","last_clean":"2023-10-30T14:51:34.593417+0000","last_became_active":"2023-10-30T14:14:04.616142+0000","last_became_peered":"2023-10-30T14:14:04.616142+0000","last_unstale":"2023-10-30T14:51:34.593417+0000","last_undegraded":"2023-10-30T14:51:34.593417+0000","last_fullsized":"2023-10-30T14:51:34.593417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1280,"log_dups_size":0,"ondisk_log_size":1280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049623779999999999,"stat_sum":{"num_bytes":9254692,"num_objects":486,"num_object_clones":220,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":190,"num_read":88,"num_read_kb":8286,"num_write":1073,"num_write_kb":19495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"67'1319","reported_seq":3294,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:36.997636+0000","last_change":"2023-10-30T14:23:30.775687+0000","last_active":"2023-10-30T14:51:36.997636+0000","last_peered":"2023-10-30T14:51:36.997636+0000","last_clean":"2023-10-30T14:51:36.997636+0000","last_became_active":"2023-10-30T14:14:04.588452+0000","last_became_peered":"2023-10-30T14:14:04.588452+0000","last_unstale":"2023-10-30T14:51:36.997636+0000","last_undegraded":"2023-10-30T14:51:36.997636+0000","last_fullsized":"2023-10-30T14:51:36.997636+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1319,"log_dups_size":0,"ondisk_log_size":1319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046351760000000004,"stat_sum":{"num_bytes":12959775,"num_objects":471,"num_object_clones":217,"num_object_copies":942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":471,"num_whiteouts":194,"num_read":76,"num_read_kb":7792,"num_write":1107,"num_write_kb":17259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"67'1323","reported_seq":3288,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.357992+0000","last_change":"2023-10-30T14:23:30.770743+0000","last_active":"2023-10-30T14:51:34.357992+0000","last_peered":"2023-10-30T14:51:34.357992+0000","last_clean":"2023-10-30T14:51:34.357992+0000","last_became_active":"2023-10-30T14:14:04.620781+0000","last_became_peered":"2023-10-30T14:14:04.620781+0000","last_unstale":"2023-10-30T14:51:34.357992+0000","last_undegraded":"2023-10-30T14:51:34.357992+0000","last_fullsized":"2023-10-30T14:51:34.357992+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1323,"log_dups_size":0,"ondisk_log_size":1323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002316706,"stat_sum":{"num_bytes":21445753,"num_objects":489,"num_object_clones":220,"num_object_copies":978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":196,"num_read":69,"num_read_kb":13296,"num_write":1109,"num_write_kb":27203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"67'1773","reported_seq":3783,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:52.488244+0000","last_change":"2023-10-30T14:51:52.488306+0000","last_active":"2023-10-30T14:51:52.488244+0000","last_peered":"2023-10-30T14:51:52.488244+0000","last_clean":"2023-10-30T14:51:52.488244+0000","last_became_active":"2023-10-30T14:14:04.608190+0000","last_became_peered":"2023-10-30T14:14:04.608190+0000","last_unstale":"2023-10-30T14:51:52.488244+0000","last_undegraded":"2023-10-30T14:51:52.488244+0000","last_fullsized":"2023-10-30T14:51:52.488244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1773","last_scrub_stamp":"2023-10-30T14:51:52.488153+0000","last_deep_scrub":"67'1773","last_deep_scrub_stamp":"2023-10-30T14:51:52.488153+0000","last_clean_scrub_stamp":"2023-10-30T14:51:52.488153+0000","objects_scrubbed":484,"log_size":1773,"log_dups_size":0,"ondisk_log_size":1773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:40.330537+0000","scrub_duration":0.90559973500000002,"objects_trimmed":0,"snaptrim_duration":0.0035618450000000001,"stat_sum":{"num_bytes":31901819,"num_objects":484,"num_object_clones":219,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":197,"num_read":85,"num_read_kb":23305,"num_write":1093,"num_write_kb":37349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"67'1361","reported_seq":3321,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.511956+0000","last_change":"2023-10-30T14:23:30.779367+0000","last_active":"2023-10-30T14:51:34.511956+0000","last_peered":"2023-10-30T14:51:34.511956+0000","last_clean":"2023-10-30T14:51:34.511956+0000","last_became_active":"2023-10-30T14:14:04.641306+0000","last_became_peered":"2023-10-30T14:14:04.641306+0000","last_unstale":"2023-10-30T14:51:34.511956+0000","last_undegraded":"2023-10-30T14:51:34.511956+0000","last_fullsized":"2023-10-30T14:51:34.511956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1361,"log_dups_size":0,"ondisk_log_size":1361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066189990000000004,"stat_sum":{"num_bytes":4095483,"num_objects":513,"num_object_clones":236,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":513,"num_whiteouts":206,"num_read":80,"num_read_kb":4151,"num_write":1131,"num_write_kb":9615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"67'1656","reported_seq":3664,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:52.398678+0000","last_change":"2023-10-30T14:51:52.398752+0000","last_active":"2023-10-30T14:51:52.398678+0000","last_peered":"2023-10-30T14:51:52.398678+0000","last_clean":"2023-10-30T14:51:52.398678+0000","last_became_active":"2023-10-30T14:14:04.613644+0000","last_became_peered":"2023-10-30T14:14:04.613644+0000","last_unstale":"2023-10-30T14:51:52.398678+0000","last_undegraded":"2023-10-30T14:51:52.398678+0000","last_fullsized":"2023-10-30T14:51:52.398678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1656","last_scrub_stamp":"2023-10-30T14:51:52.398591+0000","last_deep_scrub":"67'1656","last_deep_scrub_stamp":"2023-10-30T14:51:52.398591+0000","last_clean_scrub_stamp":"2023-10-30T14:51:52.398591+0000","objects_scrubbed":445,"log_size":1656,"log_dups_size":0,"ondisk_log_size":1656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:58.033004+0000","scrub_duration":0.76830485800000003,"objects_trimmed":0,"snaptrim_duration":0.0030699740000000001,"stat_sum":{"num_bytes":12521692,"num_objects":445,"num_object_clones":206,"num_object_copies":890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":445,"num_whiteouts":180,"num_read":71,"num_read_kb":8233,"num_write":1044,"num_write_kb":14252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"67'1303","reported_seq":3275,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.637468+0000","last_change":"2023-10-30T14:23:31.082413+0000","last_active":"2023-10-30T14:51:34.637468+0000","last_peered":"2023-10-30T14:51:34.637468+0000","last_clean":"2023-10-30T14:51:34.637468+0000","last_became_active":"2023-10-30T14:14:04.649850+0000","last_became_peered":"2023-10-30T14:14:04.649850+0000","last_unstale":"2023-10-30T14:51:34.637468+0000","last_undegraded":"2023-10-30T14:51:34.637468+0000","last_fullsized":"2023-10-30T14:51:34.637468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1303,"log_dups_size":0,"ondisk_log_size":1303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012599246,"stat_sum":{"num_bytes":26075472,"num_objects":477,"num_object_clones":221,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":193,"num_read":77,"num_read_kb":24658,"num_write":1080,"num_write_kb":32086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"67'1349","reported_seq":3284,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.621087+0000","last_change":"2023-10-30T14:23:31.074724+0000","last_active":"2023-10-30T14:51:34.621087+0000","last_peered":"2023-10-30T14:51:34.621087+0000","last_clean":"2023-10-30T14:51:34.621087+0000","last_became_active":"2023-10-30T14:14:04.625042+0000","last_became_peered":"2023-10-30T14:14:04.625042+0000","last_unstale":"2023-10-30T14:51:34.621087+0000","last_undegraded":"2023-10-30T14:51:34.621087+0000","last_fullsized":"2023-10-30T14:51:34.621087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1349,"log_dups_size":0,"ondisk_log_size":1349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082351539999999997,"stat_sum":{"num_bytes":12784426,"num_objects":514,"num_object_clones":242,"num_object_copies":1028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":514,"num_whiteouts":214,"num_read":61,"num_read_kb":10278,"num_write":1107,"num_write_kb":27149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"67'1278","reported_seq":3243,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.519419+0000","last_change":"2023-10-30T14:23:30.775419+0000","last_active":"2023-10-30T14:51:34.519419+0000","last_peered":"2023-10-30T14:51:34.519419+0000","last_clean":"2023-10-30T14:51:34.519419+0000","last_became_active":"2023-10-30T14:14:04.633586+0000","last_became_peered":"2023-10-30T14:14:04.633586+0000","last_unstale":"2023-10-30T14:51:34.519419+0000","last_undegraded":"2023-10-30T14:51:34.519419+0000","last_fullsized":"2023-10-30T14:51:34.519419+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1278,"log_dups_size":0,"ondisk_log_size":1278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043616920000000003,"stat_sum":{"num_bytes":12342414,"num_objects":462,"num_object_clones":214,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":462,"num_whiteouts":191,"num_read":63,"num_read_kb":8213,"num_write":1061,"num_write_kb":14018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"67'1382","reported_seq":3354,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.633179+0000","last_change":"2023-10-30T14:23:30.777885+0000","last_active":"2023-10-30T14:51:34.633179+0000","last_peered":"2023-10-30T14:51:34.633179+0000","last_clean":"2023-10-30T14:51:34.633179+0000","last_became_active":"2023-10-30T14:14:04.642244+0000","last_became_peered":"2023-10-30T14:14:04.642244+0000","last_unstale":"2023-10-30T14:51:34.633179+0000","last_undegraded":"2023-10-30T14:51:34.633179+0000","last_fullsized":"2023-10-30T14:51:34.633179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1382,"log_dups_size":0,"ondisk_log_size":1382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065088710000000003,"stat_sum":{"num_bytes":26075380,"num_objects":515,"num_object_clones":234,"num_object_copies":1030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":515,"num_whiteouts":211,"num_read":90,"num_read_kb":18405,"num_write":1133,"num_write_kb":31429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'1774","reported_seq":3783,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:55.288835+0000","last_change":"2023-10-30T14:51:55.288911+0000","last_active":"2023-10-30T14:51:55.288835+0000","last_peered":"2023-10-30T14:51:55.288835+0000","last_clean":"2023-10-30T14:51:55.288835+0000","last_became_active":"2023-10-30T14:14:04.620343+0000","last_became_peered":"2023-10-30T14:14:04.620343+0000","last_unstale":"2023-10-30T14:51:55.288835+0000","last_undegraded":"2023-10-30T14:51:55.288835+0000","last_fullsized":"2023-10-30T14:51:55.288835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1774","last_scrub_stamp":"2023-10-30T14:51:55.288738+0000","last_deep_scrub":"67'1774","last_deep_scrub_stamp":"2023-10-30T14:51:55.288738+0000","last_clean_scrub_stamp":"2023-10-30T14:51:55.288738+0000","objects_scrubbed":484,"log_size":1774,"log_dups_size":0,"ondisk_log_size":1774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:15:30.753588+0000","scrub_duration":0.87706562700000001,"objects_trimmed":0,"snaptrim_duration":0.0048974149999999996,"stat_sum":{"num_bytes":12481557,"num_objects":484,"num_object_clones":220,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":200,"num_read":86,"num_read_kb":8232,"num_write":1085,"num_write_kb":18010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'1870","reported_seq":3843,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:49.871302+0000","last_change":"2023-10-30T14:51:49.871411+0000","last_active":"2023-10-30T14:51:49.871302+0000","last_peered":"2023-10-30T14:51:49.871302+0000","last_clean":"2023-10-30T14:51:49.871302+0000","last_became_active":"2023-10-30T14:14:04.609720+0000","last_became_peered":"2023-10-30T14:14:04.609720+0000","last_unstale":"2023-10-30T14:51:49.871302+0000","last_undegraded":"2023-10-30T14:51:49.871302+0000","last_fullsized":"2023-10-30T14:51:49.871302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1870","last_scrub_stamp":"2023-10-30T14:51:49.871181+0000","last_deep_scrub":"67'1870","last_deep_scrub_stamp":"2023-10-30T14:51:49.871181+0000","last_clean_scrub_stamp":"2023-10-30T14:51:49.871181+0000","objects_scrubbed":535,"log_size":1870,"log_dups_size":0,"ondisk_log_size":1870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:47:19.781621+0000","scrub_duration":1.1862301479999999,"objects_trimmed":0,"snaptrim_duration":0.0040008739999999997,"stat_sum":{"num_bytes":31023805,"num_objects":535,"num_object_clones":248,"num_object_copies":1070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":535,"num_whiteouts":230,"num_read":75,"num_read_kb":20512,"num_write":1099,"num_write_kb":32005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"67'1218","reported_seq":3197,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.510081+0000","last_change":"2023-10-30T14:23:31.069745+0000","last_active":"2023-10-30T14:51:34.510081+0000","last_peered":"2023-10-30T14:51:34.510081+0000","last_clean":"2023-10-30T14:51:34.510081+0000","last_became_active":"2023-10-30T14:14:04.620217+0000","last_became_peered":"2023-10-30T14:14:04.620217+0000","last_unstale":"2023-10-30T14:51:34.510081+0000","last_undegraded":"2023-10-30T14:51:34.510081+0000","last_fullsized":"2023-10-30T14:51:34.510081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1218,"log_dups_size":0,"ondisk_log_size":1218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044491130000000002,"stat_sum":{"num_bytes":20631584,"num_objects":455,"num_object_clones":209,"num_object_copies":910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":455,"num_whiteouts":183,"num_read":71,"num_read_kb":15956,"num_write":997,"num_write_kb":22437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"67'1362","reported_seq":3346,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.633454+0000","last_change":"2023-10-30T14:23:31.075832+0000","last_active":"2023-10-30T14:51:34.633454+0000","last_peered":"2023-10-30T14:51:34.633454+0000","last_clean":"2023-10-30T14:51:34.633454+0000","last_became_active":"2023-10-30T14:14:45.669878+0000","last_became_peered":"2023-10-30T14:14:45.669878+0000","last_unstale":"2023-10-30T14:51:34.633454+0000","last_undegraded":"2023-10-30T14:51:34.633454+0000","last_fullsized":"2023-10-30T14:51:34.633454+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1362,"log_dups_size":0,"ondisk_log_size":1362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0077997409999999998,"stat_sum":{"num_bytes":25414870,"num_objects":479,"num_object_clones":220,"num_object_copies":958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":194,"num_read":80,"num_read_kb":18178,"num_write":1147,"num_write_kb":27446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"67'1314","reported_seq":3282,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.646304+0000","last_change":"2023-10-30T14:23:30.773367+0000","last_active":"2023-10-30T14:51:34.646304+0000","last_peered":"2023-10-30T14:51:34.646304+0000","last_clean":"2023-10-30T14:51:34.646304+0000","last_became_active":"2023-10-30T14:14:04.597590+0000","last_became_peered":"2023-10-30T14:14:04.597590+0000","last_unstale":"2023-10-30T14:51:34.646304+0000","last_undegraded":"2023-10-30T14:51:34.646304+0000","last_fullsized":"2023-10-30T14:51:34.646304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1314,"log_dups_size":0,"ondisk_log_size":1314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044830970000000001,"stat_sum":{"num_bytes":8769781,"num_objects":485,"num_object_clones":224,"num_object_copies":970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":197,"num_read":75,"num_read_kb":4156,"num_write":1100,"num_write_kb":11840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"67'1189","reported_seq":3149,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.631806+0000","last_change":"2023-10-30T14:23:31.070930+0000","last_active":"2023-10-30T14:51:34.631806+0000","last_peered":"2023-10-30T14:51:34.631806+0000","last_clean":"2023-10-30T14:51:34.631806+0000","last_became_active":"2023-10-30T14:14:04.623902+0000","last_became_peered":"2023-10-30T14:14:04.623902+0000","last_unstale":"2023-10-30T14:51:34.631806+0000","last_undegraded":"2023-10-30T14:51:34.631806+0000","last_fullsized":"2023-10-30T14:51:34.631806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1189,"log_dups_size":0,"ondisk_log_size":1189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051343489999999999,"stat_sum":{"num_bytes":21110293,"num_objects":460,"num_object_clones":215,"num_object_copies":920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":190,"num_read":58,"num_read_kb":12498,"num_write":969,"num_write_kb":25426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"67'1344","reported_seq":3294,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:37.891461+0000","last_change":"2023-10-30T14:23:30.775060+0000","last_active":"2023-10-30T14:51:37.891461+0000","last_peered":"2023-10-30T14:51:37.891461+0000","last_clean":"2023-10-30T14:51:37.891461+0000","last_became_active":"2023-10-30T14:14:04.597556+0000","last_became_peered":"2023-10-30T14:14:04.597556+0000","last_unstale":"2023-10-30T14:51:37.891461+0000","last_undegraded":"2023-10-30T14:51:37.891461+0000","last_fullsized":"2023-10-30T14:51:37.891461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1344,"log_dups_size":0,"ondisk_log_size":1344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040069729999999996,"stat_sum":{"num_bytes":25689725,"num_objects":524,"num_object_clones":241,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":212,"num_read":73,"num_read_kb":17618,"num_write":1103,"num_write_kb":27397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"67'1324","reported_seq":3292,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:37.102925+0000","last_change":"2023-10-30T14:23:30.779406+0000","last_active":"2023-10-30T14:51:37.102925+0000","last_peered":"2023-10-30T14:51:37.102925+0000","last_clean":"2023-10-30T14:51:37.102925+0000","last_became_active":"2023-10-30T14:14:04.618430+0000","last_became_peered":"2023-10-30T14:14:04.618430+0000","last_unstale":"2023-10-30T14:51:37.102925+0000","last_undegraded":"2023-10-30T14:51:37.102925+0000","last_fullsized":"2023-10-30T14:51:37.102925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1324,"log_dups_size":0,"ondisk_log_size":1324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074984149999999996,"stat_sum":{"num_bytes":16403179,"num_objects":481,"num_object_clones":225,"num_object_copies":962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":199,"num_read":75,"num_read_kb":12323,"num_write":1105,"num_write_kb":17671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"67'1948","reported_seq":3932,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:55.392487+0000","last_change":"2023-10-30T14:51:55.392537+0000","last_active":"2023-10-30T14:51:55.392487+0000","last_peered":"2023-10-30T14:51:55.392487+0000","last_clean":"2023-10-30T14:51:55.392487+0000","last_became_active":"2023-10-30T14:14:04.606340+0000","last_became_peered":"2023-10-30T14:14:04.606340+0000","last_unstale":"2023-10-30T14:51:55.392487+0000","last_undegraded":"2023-10-30T14:51:55.392487+0000","last_fullsized":"2023-10-30T14:51:55.392487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1948","last_scrub_stamp":"2023-10-30T14:51:55.392413+0000","last_deep_scrub":"67'1948","last_deep_scrub_stamp":"2023-10-30T14:51:55.392413+0000","last_clean_scrub_stamp":"2023-10-30T14:51:55.392413+0000","objects_scrubbed":534,"log_size":1948,"log_dups_size":0,"ondisk_log_size":1948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:02:49.595530+0000","scrub_duration":0.84988978299999995,"objects_trimmed":0,"snaptrim_duration":0.0035094890000000002,"stat_sum":{"num_bytes":22361827,"num_objects":534,"num_object_clones":245,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":534,"num_whiteouts":212,"num_read":82,"num_read_kb":9614,"num_write":1202,"num_write_kb":32875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"67'1347","reported_seq":3324,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:37.051174+0000","last_change":"2023-10-30T14:23:31.074353+0000","last_active":"2023-10-30T14:51:37.051174+0000","last_peered":"2023-10-30T14:51:37.051174+0000","last_clean":"2023-10-30T14:51:37.051174+0000","last_became_active":"2023-10-30T14:14:04.612588+0000","last_became_peered":"2023-10-30T14:14:04.612588+0000","last_unstale":"2023-10-30T14:51:37.051174+0000","last_undegraded":"2023-10-30T14:51:37.051174+0000","last_fullsized":"2023-10-30T14:51:37.051174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1347,"log_dups_size":0,"ondisk_log_size":1347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084570800000000005,"stat_sum":{"num_bytes":13290865,"num_objects":484,"num_object_clones":224,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":199,"num_read":83,"num_read_kb":8248,"num_write":1123,"num_write_kb":15099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"67'1742","reported_seq":3737,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:52.927433+0000","last_change":"2023-10-30T14:51:52.927568+0000","last_active":"2023-10-30T14:51:52.927433+0000","last_peered":"2023-10-30T14:51:52.927433+0000","last_clean":"2023-10-30T14:51:52.927433+0000","last_became_active":"2023-10-30T14:14:04.607683+0000","last_became_peered":"2023-10-30T14:14:04.607683+0000","last_unstale":"2023-10-30T14:51:52.927433+0000","last_undegraded":"2023-10-30T14:51:52.927433+0000","last_fullsized":"2023-10-30T14:51:52.927433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1742","last_scrub_stamp":"2023-10-30T14:51:52.927213+0000","last_deep_scrub":"67'1742","last_deep_scrub_stamp":"2023-10-30T14:51:52.927213+0000","last_clean_scrub_stamp":"2023-10-30T14:51:52.927213+0000","objects_scrubbed":481,"log_size":1742,"log_dups_size":0,"ondisk_log_size":1742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:57:18.481646+0000","scrub_duration":0.84554383499999997,"objects_trimmed":0,"snaptrim_duration":0.001856688,"stat_sum":{"num_bytes":22393964,"num_objects":481,"num_object_clones":222,"num_object_copies":962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":202,"num_read":74,"num_read_kb":12305,"num_write":1053,"num_write_kb":25643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'710","reported_seq":889,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:54.430131+0000","last_change":"2023-10-30T14:51:46.684252+0000","last_active":"2023-10-30T14:51:54.430131+0000","last_peered":"2023-10-30T14:51:54.430131+0000","last_clean":"2023-10-30T14:51:54.430131+0000","last_became_active":"2023-10-30T14:14:03.731740+0000","last_became_peered":"2023-10-30T14:14:03.731740+0000","last_unstale":"2023-10-30T14:51:54.430131+0000","last_undegraded":"2023-10-30T14:51:54.430131+0000","last_fullsized":"2023-10-30T14:51:54.430131+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'706","last_scrub_stamp":"2023-10-30T14:51:46.684108+0000","last_deep_scrub":"67'706","last_deep_scrub_stamp":"2023-10-30T14:51:46.684108+0000","last_clean_scrub_stamp":"2023-10-30T14:51:46.684108+0000","objects_scrubbed":43,"log_size":710,"log_dups_size":0,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:53:22.342613+0000","scrub_duration":0.17419201400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50568158,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":149,"num_read_kb":143937,"num_write":707,"num_write_kb":74908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":402982,"num_omap_keys":749,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"67'1325","reported_seq":3293,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:36.997687+0000","last_change":"2023-10-30T14:23:30.776069+0000","last_active":"2023-10-30T14:51:36.997687+0000","last_peered":"2023-10-30T14:51:36.997687+0000","last_clean":"2023-10-30T14:51:36.997687+0000","last_became_active":"2023-10-30T14:14:04.621526+0000","last_became_peered":"2023-10-30T14:14:04.621526+0000","last_unstale":"2023-10-30T14:51:36.997687+0000","last_undegraded":"2023-10-30T14:51:36.997687+0000","last_fullsized":"2023-10-30T14:51:36.997687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1325,"log_dups_size":0,"ondisk_log_size":1325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047814620000000002,"stat_sum":{"num_bytes":12429609,"num_objects":495,"num_object_clones":226,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":189,"num_read":79,"num_read_kb":8245,"num_write":1124,"num_write_kb":15289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"67'1505","reported_seq":1996,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:43.210550+0000","last_change":"2023-10-30T14:51:43.210666+0000","last_active":"2023-10-30T14:51:43.210550+0000","last_peered":"2023-10-30T14:51:43.210550+0000","last_clean":"2023-10-30T14:51:43.210550+0000","last_became_active":"2023-10-30T14:14:03.729386+0000","last_became_peered":"2023-10-30T14:14:03.729386+0000","last_unstale":"2023-10-30T14:51:43.210550+0000","last_undegraded":"2023-10-30T14:51:43.210550+0000","last_fullsized":"2023-10-30T14:51:43.210550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1505","last_scrub_stamp":"2023-10-30T14:51:43.210412+0000","last_deep_scrub":"67'1505","last_deep_scrub_stamp":"2023-10-30T14:51:43.210412+0000","last_clean_scrub_stamp":"2023-10-30T14:51:43.210412+0000","objects_scrubbed":46,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:18:14.848623+0000","scrub_duration":0.17059465400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29761637,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":457,"num_read_kb":164028,"num_write":1504,"num_write_kb":71675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":396614,"num_omap_keys":698,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'787","reported_seq":1361,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:45.617219+0000","last_change":"2023-10-30T14:51:45.617294+0000","last_active":"2023-10-30T14:51:45.617219+0000","last_peered":"2023-10-30T14:51:45.617219+0000","last_clean":"2023-10-30T14:51:45.617219+0000","last_became_active":"2023-10-30T14:14:03.724078+0000","last_became_peered":"2023-10-30T14:14:03.724078+0000","last_unstale":"2023-10-30T14:51:45.617219+0000","last_undegraded":"2023-10-30T14:51:45.617219+0000","last_fullsized":"2023-10-30T14:51:45.617219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'787","last_scrub_stamp":"2023-10-30T14:51:45.617096+0000","last_deep_scrub":"67'787","last_deep_scrub_stamp":"2023-10-30T14:51:45.617096+0000","last_clean_scrub_stamp":"2023-10-30T14:51:45.617096+0000","objects_scrubbed":42,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:36:59.792994+0000","scrub_duration":0.123030087,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30316903,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":547,"num_read_kb":63175,"num_write":794,"num_write_kb":47477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":452186,"num_omap_keys":609,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"67'1965","reported_seq":3927,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:50.284892+0000","last_change":"2023-10-30T14:51:50.284976+0000","last_active":"2023-10-30T14:51:50.284892+0000","last_peered":"2023-10-30T14:51:50.284892+0000","last_clean":"2023-10-30T14:51:50.284892+0000","last_became_active":"2023-10-30T14:14:04.597099+0000","last_became_peered":"2023-10-30T14:14:04.597099+0000","last_unstale":"2023-10-30T14:51:50.284892+0000","last_undegraded":"2023-10-30T14:51:50.284892+0000","last_fullsized":"2023-10-30T14:51:50.284892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1965","last_scrub_stamp":"2023-10-30T14:51:50.284800+0000","last_deep_scrub":"67'1965","last_deep_scrub_stamp":"2023-10-30T14:51:50.284800+0000","last_clean_scrub_stamp":"2023-10-30T14:51:50.284800+0000","objects_scrubbed":552,"log_size":1965,"log_dups_size":0,"ondisk_log_size":1965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:16:10.415430+0000","scrub_duration":0.75260475800000004,"objects_trimmed":0,"snaptrim_duration":0.0033627510000000002,"stat_sum":{"num_bytes":15535325,"num_objects":552,"num_object_clones":252,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":552,"num_whiteouts":227,"num_read":69,"num_read_kb":8270,"num_write":1194,"num_write_kb":25187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"67'1603","reported_seq":1776,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:41.944331+0000","last_change":"2023-10-30T14:51:41.944482+0000","last_active":"2023-10-30T14:51:41.944331+0000","last_peered":"2023-10-30T14:51:41.944331+0000","last_clean":"2023-10-30T14:51:41.944331+0000","last_became_active":"2023-10-30T14:14:03.719816+0000","last_became_peered":"2023-10-30T14:14:03.719816+0000","last_unstale":"2023-10-30T14:51:41.944331+0000","last_undegraded":"2023-10-30T14:51:41.944331+0000","last_fullsized":"2023-10-30T14:51:41.944331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1603","last_scrub_stamp":"2023-10-30T14:51:41.944113+0000","last_deep_scrub":"67'1603","last_deep_scrub_stamp":"2023-10-30T14:51:41.944113+0000","last_clean_scrub_stamp":"2023-10-30T14:51:41.944113+0000","objects_scrubbed":49,"log_size":1603,"log_dups_size":0,"ondisk_log_size":1603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:29.585178+0000","scrub_duration":0.217796778,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35358900,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":164,"num_read_kb":135701,"num_write":1593,"num_write_kb":77235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":466020,"num_omap_keys":829,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"67'1291","reported_seq":3257,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.428260+0000","last_change":"2023-10-30T14:23:30.771109+0000","last_active":"2023-10-30T14:51:34.428260+0000","last_peered":"2023-10-30T14:51:34.428260+0000","last_clean":"2023-10-30T14:51:34.428260+0000","last_became_active":"2023-10-30T14:14:04.599323+0000","last_became_peered":"2023-10-30T14:14:04.599323+0000","last_unstale":"2023-10-30T14:51:34.428260+0000","last_undegraded":"2023-10-30T14:51:34.428260+0000","last_fullsized":"2023-10-30T14:51:34.428260+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1291,"log_dups_size":0,"ondisk_log_size":1291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025427000000000002,"stat_sum":{"num_bytes":25885700,"num_objects":482,"num_object_clones":224,"num_object_copies":964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":193,"num_read":75,"num_read_kb":21385,"num_write":1056,"num_write_kb":37196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"67'660","reported_seq":797,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:46.206639+0000","last_change":"2023-10-30T14:51:46.206708+0000","last_active":"2023-10-30T14:51:46.206639+0000","last_peered":"2023-10-30T14:51:46.206639+0000","last_clean":"2023-10-30T14:51:46.206639+0000","last_became_active":"2023-10-30T14:14:03.738132+0000","last_became_peered":"2023-10-30T14:14:03.738132+0000","last_unstale":"2023-10-30T14:51:46.206639+0000","last_undegraded":"2023-10-30T14:51:46.206639+0000","last_fullsized":"2023-10-30T14:51:46.206639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'660","last_scrub_stamp":"2023-10-30T14:51:46.206546+0000","last_deep_scrub":"67'660","last_deep_scrub_stamp":"2023-10-30T14:51:46.206546+0000","last_clean_scrub_stamp":"2023-10-30T14:51:46.206546+0000","objects_scrubbed":43,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:02:40.190575+0000","scrub_duration":0.151344961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360218,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":99,"num_read_kb":53823,"num_write":691,"num_write_kb":33991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":515776,"num_omap_keys":890,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"67'1308","reported_seq":3286,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.609888+0000","last_change":"2023-10-30T14:23:30.778832+0000","last_active":"2023-10-30T14:51:34.609888+0000","last_peered":"2023-10-30T14:51:34.609888+0000","last_clean":"2023-10-30T14:51:34.609888+0000","last_became_active":"2023-10-30T14:14:04.648816+0000","last_became_peered":"2023-10-30T14:14:04.648816+0000","last_unstale":"2023-10-30T14:51:34.609888+0000","last_undegraded":"2023-10-30T14:51:34.609888+0000","last_fullsized":"2023-10-30T14:51:34.609888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1308,"log_dups_size":0,"ondisk_log_size":1308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069848840000000002,"stat_sum":{"num_bytes":12168759,"num_objects":462,"num_object_clones":214,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":462,"num_whiteouts":185,"num_read":77,"num_read_kb":8211,"num_write":1090,"num_write_kb":14663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"67'1070","reported_seq":1217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:47.771730+0000","last_change":"2023-10-30T14:51:47.771870+0000","last_active":"2023-10-30T14:51:47.771730+0000","last_peered":"2023-10-30T14:51:47.771730+0000","last_clean":"2023-10-30T14:51:47.771730+0000","last_became_active":"2023-10-30T14:14:03.725272+0000","last_became_peered":"2023-10-30T14:14:03.725272+0000","last_unstale":"2023-10-30T14:51:47.771730+0000","last_undegraded":"2023-10-30T14:51:47.771730+0000","last_fullsized":"2023-10-30T14:51:47.771730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1070","last_scrub_stamp":"2023-10-30T14:51:47.771555+0000","last_deep_scrub":"67'1070","last_deep_scrub_stamp":"2023-10-30T14:51:47.771555+0000","last_clean_scrub_stamp":"2023-10-30T14:51:47.771555+0000","objects_scrubbed":41,"log_size":1070,"log_dups_size":0,"ondisk_log_size":1070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:13:17.687697+0000","scrub_duration":0.128806527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":123,"num_read_kb":86587,"num_write":1104,"num_write_kb":54796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":577700,"num_omap_keys":1131,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"67'1405","reported_seq":3325,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.632364+0000","last_change":"2023-10-30T14:23:30.776486+0000","last_active":"2023-10-30T14:51:34.632364+0000","last_peered":"2023-10-30T14:51:34.632364+0000","last_clean":"2023-10-30T14:51:34.632364+0000","last_became_active":"2023-10-30T14:14:04.609697+0000","last_became_peered":"2023-10-30T14:14:04.609697+0000","last_unstale":"2023-10-30T14:51:34.632364+0000","last_undegraded":"2023-10-30T14:51:34.632364+0000","last_fullsized":"2023-10-30T14:51:34.632364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1405,"log_dups_size":0,"ondisk_log_size":1405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043395700000000001,"stat_sum":{"num_bytes":12484119,"num_objects":557,"num_object_clones":263,"num_object_copies":1114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":557,"num_whiteouts":236,"num_read":68,"num_read_kb":4261,"num_write":1150,"num_write_kb":13862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"67'1026","reported_seq":1197,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:50.429870+0000","last_change":"2023-10-30T14:51:50.429966+0000","last_active":"2023-10-30T14:51:50.429870+0000","last_peered":"2023-10-30T14:51:50.429870+0000","last_clean":"2023-10-30T14:51:50.429870+0000","last_became_active":"2023-10-30T14:14:03.727422+0000","last_became_peered":"2023-10-30T14:14:03.727422+0000","last_unstale":"2023-10-30T14:51:50.429870+0000","last_undegraded":"2023-10-30T14:51:50.429870+0000","last_fullsized":"2023-10-30T14:51:50.429870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1026","last_scrub_stamp":"2023-10-30T14:51:50.429770+0000","last_deep_scrub":"67'1026","last_deep_scrub_stamp":"2023-10-30T14:51:50.429770+0000","last_clean_scrub_stamp":"2023-10-30T14:51:50.429770+0000","objects_scrubbed":43,"log_size":1026,"log_dups_size":0,"ondisk_log_size":1026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:53.510519+0000","scrub_duration":0.107061642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389280,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":161,"num_read_kb":61498,"num_write":1088,"num_write_kb":30346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":619408,"num_omap_keys":1098,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'2081","reported_seq":3987,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:54.238323+0000","last_change":"2023-10-30T14:51:54.238386+0000","last_active":"2023-10-30T14:51:54.238323+0000","last_peered":"2023-10-30T14:51:54.238323+0000","last_clean":"2023-10-30T14:51:54.238323+0000","last_became_active":"2023-10-30T14:14:04.645333+0000","last_became_peered":"2023-10-30T14:14:04.645333+0000","last_unstale":"2023-10-30T14:51:54.238323+0000","last_undegraded":"2023-10-30T14:51:54.238323+0000","last_fullsized":"2023-10-30T14:51:54.238323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2081","last_scrub_stamp":"2023-10-30T14:51:54.238230+0000","last_deep_scrub":"67'2081","last_deep_scrub_stamp":"2023-10-30T14:51:54.238230+0000","last_clean_scrub_stamp":"2023-10-30T14:51:54.238230+0000","objects_scrubbed":601,"log_size":2081,"log_dups_size":0,"ondisk_log_size":2081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:44.996187+0000","scrub_duration":1.177356566,"objects_trimmed":0,"snaptrim_duration":0.002301474,"stat_sum":{"num_bytes":21718535,"num_objects":601,"num_object_clones":320,"num_object_copies":1202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":601,"num_whiteouts":226,"num_read":78,"num_read_kb":5632,"num_write":1155,"num_write_kb":28163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"67'606","reported_seq":804,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:51.267795+0000","last_change":"2023-10-30T14:51:51.267795+0000","last_active":"2023-10-30T14:51:51.267795+0000","last_peered":"2023-10-30T14:51:51.267795+0000","last_clean":"2023-10-30T14:51:51.267795+0000","last_became_active":"2023-10-30T14:14:03.724806+0000","last_became_peered":"2023-10-30T14:14:03.724806+0000","last_unstale":"2023-10-30T14:51:51.267795+0000","last_undegraded":"2023-10-30T14:51:51.267795+0000","last_fullsized":"2023-10-30T14:51:51.267795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'605","last_scrub_stamp":"2023-10-30T14:51:51.267479+0000","last_deep_scrub":"67'605","last_deep_scrub_stamp":"2023-10-30T14:51:51.267479+0000","last_clean_scrub_stamp":"2023-10-30T14:51:51.267479+0000","objects_scrubbed":50,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:02:26.648751+0000","scrub_duration":0.15457963999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360800,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":192,"num_read_kb":131840,"num_write":659,"num_write_kb":54410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":552351,"num_omap_keys":1032,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"67'1328","reported_seq":3274,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.609160+0000","last_change":"2023-10-30T14:23:31.073841+0000","last_active":"2023-10-30T14:51:34.609160+0000","last_peered":"2023-10-30T14:51:34.609160+0000","last_clean":"2023-10-30T14:51:34.609160+0000","last_became_active":"2023-10-30T14:14:04.646543+0000","last_became_peered":"2023-10-30T14:14:04.646543+0000","last_unstale":"2023-10-30T14:51:34.609160+0000","last_undegraded":"2023-10-30T14:51:34.609160+0000","last_fullsized":"2023-10-30T14:51:34.609160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1328,"log_dups_size":0,"ondisk_log_size":1328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00744474,"stat_sum":{"num_bytes":22194385,"num_objects":506,"num_object_clones":234,"num_object_copies":1012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":207,"num_read":65,"num_read_kb":16406,"num_write":1096,"num_write_kb":24097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"67'705","reported_seq":891,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:50.491455+0000","last_change":"2023-10-30T14:51:50.491497+0000","last_active":"2023-10-30T14:51:50.491455+0000","last_peered":"2023-10-30T14:51:50.491455+0000","last_clean":"2023-10-30T14:51:50.491455+0000","last_became_active":"2023-10-30T14:14:03.716941+0000","last_became_peered":"2023-10-30T14:14:03.716941+0000","last_unstale":"2023-10-30T14:51:50.491455+0000","last_undegraded":"2023-10-30T14:51:50.491455+0000","last_fullsized":"2023-10-30T14:51:50.491455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'705","last_scrub_stamp":"2023-10-30T14:51:50.491382+0000","last_deep_scrub":"67'705","last_deep_scrub_stamp":"2023-10-30T14:51:50.491382+0000","last_clean_scrub_stamp":"2023-10-30T14:51:50.491382+0000","objects_scrubbed":50,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:52:06.985963+0000","scrub_duration":0.118685916,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777250,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":201,"num_read_kb":82705,"num_write":762,"num_write_kb":38720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":647476,"num_omap_keys":1125,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"67'1705","reported_seq":3649,"reported_epoch":67,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-30T14:51:56.998756+0000","last_change":"2023-10-30T14:51:56.444538+0000","last_active":"2023-10-30T14:51:56.998756+0000","last_peered":"2023-10-30T14:51:56.998756+0000","last_clean":"2023-10-30T14:51:56.998756+0000","last_became_active":"2023-10-30T14:14:04.641494+0000","last_became_peered":"2023-10-30T14:14:04.641494+0000","last_unstale":"2023-10-30T14:51:56.998756+0000","last_undegraded":"2023-10-30T14:51:56.998756+0000","last_fullsized":"2023-10-30T14:51:56.998756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":296,"log_size":1705,"log_dups_size":0,"ondisk_log_size":1705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039664019999999999,"stat_sum":{"num_bytes":26425057,"num_objects":539,"num_object_clones":248,"num_object_copies":1078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":208,"num_read":69,"num_read_kb":14198,"num_write":1182,"num_write_kb":36722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"67'327","reported_seq":458,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:44.816766+0000","last_change":"2023-10-30T14:51:44.816862+0000","last_active":"2023-10-30T14:51:44.816766+0000","last_peered":"2023-10-30T14:51:44.816766+0000","last_clean":"2023-10-30T14:51:44.816766+0000","last_became_active":"2023-10-30T14:14:03.728053+0000","last_became_peered":"2023-10-30T14:14:03.728053+0000","last_unstale":"2023-10-30T14:51:44.816766+0000","last_undegraded":"2023-10-30T14:51:44.816766+0000","last_fullsized":"2023-10-30T14:51:44.816766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'327","last_scrub_stamp":"2023-10-30T14:51:44.816631+0000","last_deep_scrub":"67'327","last_deep_scrub_stamp":"2023-10-30T14:51:44.816631+0000","last_clean_scrub_stamp":"2023-10-30T14:51:44.816631+0000","objects_scrubbed":38,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:44:07.678724+0000","scrub_duration":0.092522458000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":90,"num_read_kb":49506,"num_write":347,"num_write_kb":29524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":346610,"num_omap_keys":678,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1406","reported_seq":3368,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.590359+0000","last_change":"2023-10-30T14:23:31.078295+0000","last_active":"2023-10-30T14:51:34.590359+0000","last_peered":"2023-10-30T14:51:34.590359+0000","last_clean":"2023-10-30T14:51:34.590359+0000","last_became_active":"2023-10-30T14:14:04.649896+0000","last_became_peered":"2023-10-30T14:14:04.649896+0000","last_unstale":"2023-10-30T14:51:34.590359+0000","last_undegraded":"2023-10-30T14:51:34.590359+0000","last_fullsized":"2023-10-30T14:51:34.590359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1406,"log_dups_size":0,"ondisk_log_size":1406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010792094,"stat_sum":{"num_bytes":9892654,"num_objects":516,"num_object_clones":241,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":516,"num_whiteouts":210,"num_read":88,"num_read_kb":2154,"num_write":1169,"num_write_kb":11317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"67'732","reported_seq":878,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:54.255960+0000","last_change":"2023-10-30T14:51:54.256020+0000","last_active":"2023-10-30T14:51:54.255960+0000","last_peered":"2023-10-30T14:51:54.255960+0000","last_clean":"2023-10-30T14:51:54.255960+0000","last_became_active":"2023-10-30T14:14:03.736155+0000","last_became_peered":"2023-10-30T14:14:03.736155+0000","last_unstale":"2023-10-30T14:51:54.255960+0000","last_undegraded":"2023-10-30T14:51:54.255960+0000","last_fullsized":"2023-10-30T14:51:54.255960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'732","last_scrub_stamp":"2023-10-30T14:51:54.255862+0000","last_deep_scrub":"67'732","last_deep_scrub_stamp":"2023-10-30T14:51:54.255862+0000","last_clean_scrub_stamp":"2023-10-30T14:51:54.255862+0000","objects_scrubbed":38,"log_size":732,"log_dups_size":0,"ondisk_log_size":732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:23:50.725664+0000","scrub_duration":0.094160410999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":104,"num_read_kb":67382,"num_write":743,"num_write_kb":42078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":416827,"num_omap_keys":710,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"67'1271","reported_seq":3244,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.587520+0000","last_change":"2023-10-30T14:23:31.076487+0000","last_active":"2023-10-30T14:51:34.587520+0000","last_peered":"2023-10-30T14:51:34.587520+0000","last_clean":"2023-10-30T14:51:34.587520+0000","last_became_active":"2023-10-30T14:14:04.614328+0000","last_became_peered":"2023-10-30T14:14:04.614328+0000","last_unstale":"2023-10-30T14:51:34.587520+0000","last_undegraded":"2023-10-30T14:51:34.587520+0000","last_fullsized":"2023-10-30T14:51:34.587520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1271,"log_dups_size":0,"ondisk_log_size":1271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010029286,"stat_sum":{"num_bytes":9325803,"num_objects":478,"num_object_clones":220,"num_object_copies":956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":189,"num_read":77,"num_read_kb":4169,"num_write":1060,"num_write_kb":11660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"67'1235","reported_seq":1429,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:46.739900+0000","last_change":"2023-10-30T14:51:46.739956+0000","last_active":"2023-10-30T14:51:46.739900+0000","last_peered":"2023-10-30T14:51:46.739900+0000","last_clean":"2023-10-30T14:51:46.739900+0000","last_became_active":"2023-10-30T14:14:03.700352+0000","last_became_peered":"2023-10-30T14:14:03.700352+0000","last_unstale":"2023-10-30T14:51:46.739900+0000","last_undegraded":"2023-10-30T14:51:46.739900+0000","last_fullsized":"2023-10-30T14:51:46.739900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1235","last_scrub_stamp":"2023-10-30T14:51:46.739823+0000","last_deep_scrub":"67'1235","last_deep_scrub_stamp":"2023-10-30T14:51:46.739823+0000","last_clean_scrub_stamp":"2023-10-30T14:51:46.739823+0000","objects_scrubbed":44,"log_size":1235,"log_dups_size":0,"ondisk_log_size":1235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:04:42.377958+0000","scrub_duration":0.069929302999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":252,"num_read_kb":55463,"num_write":1290,"num_write_kb":26283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":593045,"num_omap_keys":836,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"67'1851","reported_seq":3839,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:45.460493+0000","last_change":"2023-10-30T14:51:45.460580+0000","last_active":"2023-10-30T14:51:45.460493+0000","last_peered":"2023-10-30T14:51:45.460493+0000","last_clean":"2023-10-30T14:51:45.460493+0000","last_became_active":"2023-10-30T14:14:04.646639+0000","last_became_peered":"2023-10-30T14:14:04.646639+0000","last_unstale":"2023-10-30T14:51:45.460493+0000","last_undegraded":"2023-10-30T14:51:45.460493+0000","last_fullsized":"2023-10-30T14:51:45.460493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1851","last_scrub_stamp":"2023-10-30T14:51:45.460385+0000","last_deep_scrub":"67'1851","last_deep_scrub_stamp":"2023-10-30T14:51:45.460385+0000","last_clean_scrub_stamp":"2023-10-30T14:51:45.460385+0000","objects_scrubbed":508,"log_size":1851,"log_dups_size":0,"ondisk_log_size":1851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:46.502926+0000","scrub_duration":0.96639256900000003,"objects_trimmed":0,"snaptrim_duration":0.010379724,"stat_sum":{"num_bytes":13632563,"num_objects":508,"num_object_clones":233,"num_object_copies":1016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":508,"num_whiteouts":207,"num_read":77,"num_read_kb":12340,"num_write":1126,"num_write_kb":24370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'661","reported_seq":796,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:45.259670+0000","last_change":"2023-10-30T14:51:45.259757+0000","last_active":"2023-10-30T14:51:45.259670+0000","last_peered":"2023-10-30T14:51:45.259670+0000","last_clean":"2023-10-30T14:51:45.259670+0000","last_became_active":"2023-10-30T14:14:03.707950+0000","last_became_peered":"2023-10-30T14:14:03.707950+0000","last_unstale":"2023-10-30T14:51:45.259670+0000","last_undegraded":"2023-10-30T14:51:45.259670+0000","last_fullsized":"2023-10-30T14:51:45.259670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'661","last_scrub_stamp":"2023-10-30T14:51:45.259579+0000","last_deep_scrub":"67'661","last_deep_scrub_stamp":"2023-10-30T14:51:45.259579+0000","last_clean_scrub_stamp":"2023-10-30T14:51:45.259579+0000","objects_scrubbed":32,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:37:24.581339+0000","scrub_duration":0.102681681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971542,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":82,"num_read_kb":57693,"num_write":882,"num_write_kb":42351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":303757,"num_omap_keys":462,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"67'1262","reported_seq":3209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.644322+0000","last_change":"2023-10-30T14:23:31.080966+0000","last_active":"2023-10-30T14:51:34.644322+0000","last_peered":"2023-10-30T14:51:34.644322+0000","last_clean":"2023-10-30T14:51:34.644322+0000","last_became_active":"2023-10-30T14:14:04.644761+0000","last_became_peered":"2023-10-30T14:14:04.644761+0000","last_unstale":"2023-10-30T14:51:34.644322+0000","last_undegraded":"2023-10-30T14:51:34.644322+0000","last_fullsized":"2023-10-30T14:51:34.644322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1262,"log_dups_size":0,"ondisk_log_size":1262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011171970999999999,"stat_sum":{"num_bytes":12945563,"num_objects":503,"num_object_clones":231,"num_object_copies":1006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":213,"num_read":62,"num_read_kb":4133,"num_write":1006,"num_write_kb":18895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'558","reported_seq":699,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:48.767892+0000","last_change":"2023-10-30T14:51:48.767990+0000","last_active":"2023-10-30T14:51:48.767892+0000","last_peered":"2023-10-30T14:51:48.767892+0000","last_clean":"2023-10-30T14:51:48.767892+0000","last_became_active":"2023-10-30T14:14:03.705186+0000","last_became_peered":"2023-10-30T14:14:03.705186+0000","last_unstale":"2023-10-30T14:51:48.767892+0000","last_undegraded":"2023-10-30T14:51:48.767892+0000","last_fullsized":"2023-10-30T14:51:48.767892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'558","last_scrub_stamp":"2023-10-30T14:51:48.767760+0000","last_deep_scrub":"67'558","last_deep_scrub_stamp":"2023-10-30T14:51:48.767760+0000","last_clean_scrub_stamp":"2023-10-30T14:51:48.767760+0000","objects_scrubbed":43,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:37:06.574920+0000","scrub_duration":0.12212297900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":102,"num_read_kb":90592,"num_write":581,"num_write_kb":50591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":484629,"num_omap_keys":764,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"67'1327","reported_seq":3273,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.614000+0000","last_change":"2023-10-30T14:23:31.083260+0000","last_active":"2023-10-30T14:51:34.614000+0000","last_peered":"2023-10-30T14:51:34.614000+0000","last_clean":"2023-10-30T14:51:34.614000+0000","last_became_active":"2023-10-30T14:14:04.646785+0000","last_became_peered":"2023-10-30T14:14:04.646785+0000","last_unstale":"2023-10-30T14:51:34.614000+0000","last_undegraded":"2023-10-30T14:51:34.614000+0000","last_fullsized":"2023-10-30T14:51:34.614000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1327,"log_dups_size":0,"ondisk_log_size":1327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010111184,"stat_sum":{"num_bytes":17483229,"num_objects":512,"num_object_clones":242,"num_object_copies":1024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":512,"num_whiteouts":218,"num_read":72,"num_read_kb":10698,"num_write":1044,"num_write_kb":23548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"67'622","reported_seq":782,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:44.214200+0000","last_change":"2023-10-30T14:51:44.214254+0000","last_active":"2023-10-30T14:51:44.214200+0000","last_peered":"2023-10-30T14:51:44.214200+0000","last_clean":"2023-10-30T14:51:44.214200+0000","last_became_active":"2023-10-30T14:14:03.692966+0000","last_became_peered":"2023-10-30T14:14:03.692966+0000","last_unstale":"2023-10-30T14:51:44.214200+0000","last_undegraded":"2023-10-30T14:51:44.214200+0000","last_fullsized":"2023-10-30T14:51:44.214200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'622","last_scrub_stamp":"2023-10-30T14:51:44.214112+0000","last_deep_scrub":"67'622","last_deep_scrub_stamp":"2023-10-30T14:51:44.214112+0000","last_clean_scrub_stamp":"2023-10-30T14:51:44.214112+0000","objects_scrubbed":36,"log_size":622,"log_dups_size":0,"ondisk_log_size":622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:10:21.572387+0000","scrub_duration":0.085160906999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16780852,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":126,"num_read_kb":110211,"num_write":641,"num_write_kb":50166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":448077,"num_omap_keys":692,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'1343","reported_seq":3318,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.638166+0000","last_change":"2023-10-30T14:23:30.778622+0000","last_active":"2023-10-30T14:51:34.638166+0000","last_peered":"2023-10-30T14:51:34.638166+0000","last_clean":"2023-10-30T14:51:34.638166+0000","last_became_active":"2023-10-30T14:14:04.615925+0000","last_became_peered":"2023-10-30T14:14:04.615925+0000","last_unstale":"2023-10-30T14:51:34.638166+0000","last_undegraded":"2023-10-30T14:51:34.638166+0000","last_fullsized":"2023-10-30T14:51:34.638166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1343,"log_dups_size":0,"ondisk_log_size":1343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068317569999999999,"stat_sum":{"num_bytes":4343182,"num_objects":487,"num_object_clones":225,"num_object_copies":974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":206,"num_read":84,"num_read_kb":56,"num_write":1107,"num_write_kb":8908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"67'683","reported_seq":831,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:39.782505+0000","last_change":"2023-10-30T14:51:39.782590+0000","last_active":"2023-10-30T14:51:39.782505+0000","last_peered":"2023-10-30T14:51:39.782505+0000","last_clean":"2023-10-30T14:51:39.782505+0000","last_became_active":"2023-10-30T14:14:03.727409+0000","last_became_peered":"2023-10-30T14:14:03.727409+0000","last_unstale":"2023-10-30T14:51:39.782505+0000","last_undegraded":"2023-10-30T14:51:39.782505+0000","last_fullsized":"2023-10-30T14:51:39.782505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'683","last_scrub_stamp":"2023-10-30T14:51:39.782417+0000","last_deep_scrub":"67'683","last_deep_scrub_stamp":"2023-10-30T14:51:39.782417+0000","last_clean_scrub_stamp":"2023-10-30T14:51:39.782417+0000","objects_scrubbed":36,"log_size":683,"log_dups_size":0,"ondisk_log_size":683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:14:03.723383+0000","scrub_duration":0.097535790999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166406,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":118,"num_read_kb":66012,"num_write":697,"num_write_kb":50302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":407093,"num_omap_keys":513,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1295","reported_seq":3268,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.640923+0000","last_change":"2023-10-30T14:23:31.077576+0000","last_active":"2023-10-30T14:51:34.640923+0000","last_peered":"2023-10-30T14:51:34.640923+0000","last_clean":"2023-10-30T14:51:34.640923+0000","last_became_active":"2023-10-30T14:14:45.818032+0000","last_became_peered":"2023-10-30T14:14:45.818032+0000","last_unstale":"2023-10-30T14:51:34.640923+0000","last_undegraded":"2023-10-30T14:51:34.640923+0000","last_fullsized":"2023-10-30T14:51:34.640923+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010655906,"stat_sum":{"num_bytes":23800594,"num_objects":485,"num_object_clones":224,"num_object_copies":970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":201,"num_read":74,"num_read_kb":13429,"num_write":1078,"num_write_kb":29347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"67'119","reported_seq":241,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:50.399571+0000","last_change":"2023-10-30T14:51:50.399571+0000","last_active":"2023-10-30T14:51:50.399571+0000","last_peered":"2023-10-30T14:51:50.399571+0000","last_clean":"2023-10-30T14:51:50.399571+0000","last_became_active":"2023-10-30T14:13:54.250907+0000","last_became_peered":"2023-10-30T14:13:54.250907+0000","last_unstale":"2023-10-30T14:51:50.399571+0000","last_undegraded":"2023-10-30T14:51:50.399571+0000","last_fullsized":"2023-10-30T14:51:50.399571+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'119","last_scrub_stamp":"2023-10-30T14:51:50.399169+0000","last_deep_scrub":"67'119","last_deep_scrub_stamp":"2023-10-30T14:51:50.399169+0000","last_clean_scrub_stamp":"2023-10-30T14:51:50.399169+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:18.007665+0000","scrub_duration":0.0099470610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"67'751","reported_seq":905,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:45.869660+0000","last_change":"2023-10-30T14:51:45.869788+0000","last_active":"2023-10-30T14:51:45.869660+0000","last_peered":"2023-10-30T14:51:45.869660+0000","last_clean":"2023-10-30T14:51:45.869660+0000","last_became_active":"2023-10-30T14:14:03.728293+0000","last_became_peered":"2023-10-30T14:14:03.728293+0000","last_unstale":"2023-10-30T14:51:45.869660+0000","last_undegraded":"2023-10-30T14:51:45.869660+0000","last_fullsized":"2023-10-30T14:51:45.869660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'751","last_scrub_stamp":"2023-10-30T14:51:45.869434+0000","last_deep_scrub":"67'751","last_deep_scrub_stamp":"2023-10-30T14:51:45.869434+0000","last_clean_scrub_stamp":"2023-10-30T14:51:45.869434+0000","objects_scrubbed":46,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:31.381332+0000","scrub_duration":0.14853830900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165954,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":121,"num_read_kb":78313,"num_write":800,"num_write_kb":42232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":509463,"num_omap_keys":728,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"67'1553","reported_seq":3526,"reported_epoch":67,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-30T14:51:57.088875+0000","last_change":"2023-10-30T14:51:56.589735+0000","last_active":"2023-10-30T14:51:57.088875+0000","last_peered":"2023-10-30T14:51:57.088875+0000","last_clean":"2023-10-30T14:51:57.088875+0000","last_became_active":"2023-10-30T14:14:46.451620+0000","last_became_peered":"2023-10-30T14:14:46.451620+0000","last_unstale":"2023-10-30T14:51:57.088875+0000","last_undegraded":"2023-10-30T14:51:57.088875+0000","last_fullsized":"2023-10-30T14:51:57.088875+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":294,"log_size":1553,"log_dups_size":0,"ondisk_log_size":1553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034002070000000001,"stat_sum":{"num_bytes":12539401,"num_objects":475,"num_object_clones":216,"num_object_copies":950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":185,"num_read":65,"num_read_kb":8203,"num_write":1054,"num_write_kb":18334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"67'823","reported_seq":1010,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:29.609162+0000","last_change":"2023-10-30T14:14:03.732909+0000","last_active":"2023-10-30T14:51:29.609162+0000","last_peered":"2023-10-30T14:51:29.609162+0000","last_clean":"2023-10-30T14:51:29.609162+0000","last_became_active":"2023-10-30T14:14:03.732199+0000","last_became_peered":"2023-10-30T14:14:03.732199+0000","last_unstale":"2023-10-30T14:51:29.609162+0000","last_undegraded":"2023-10-30T14:51:29.609162+0000","last_fullsized":"2023-10-30T14:51:29.609162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":823,"log_dups_size":0,"ondisk_log_size":823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":198,"num_read_kb":136195,"num_write":895,"num_write_kb":59080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"67'1737","reported_seq":3726,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:56.490661+0000","last_change":"2023-10-30T14:51:56.490720+0000","last_active":"2023-10-30T14:51:56.490661+0000","last_peered":"2023-10-30T14:51:56.490661+0000","last_clean":"2023-10-30T14:51:56.490661+0000","last_became_active":"2023-10-30T14:14:04.619990+0000","last_became_peered":"2023-10-30T14:14:04.619990+0000","last_unstale":"2023-10-30T14:51:56.490661+0000","last_undegraded":"2023-10-30T14:51:56.490661+0000","last_fullsized":"2023-10-30T14:51:56.490661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1737","last_scrub_stamp":"2023-10-30T14:51:56.490587+0000","last_deep_scrub":"67'1737","last_deep_scrub_stamp":"2023-10-30T14:51:56.490587+0000","last_clean_scrub_stamp":"2023-10-30T14:51:56.490587+0000","objects_scrubbed":475,"log_size":1737,"log_dups_size":0,"ondisk_log_size":1737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:42:50.032821+0000","scrub_duration":1.0169323969999999,"objects_trimmed":0,"snaptrim_duration":0.0045963250000000001,"stat_sum":{"num_bytes":8686325,"num_objects":475,"num_object_clones":219,"num_object_copies":950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":195,"num_read":63,"num_read_kb":1576,"num_write":1040,"num_write_kb":11254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"67'1782","reported_seq":2900,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:47.811285+0000","last_change":"2023-10-30T14:51:47.811373+0000","last_active":"2023-10-30T14:51:47.811285+0000","last_peered":"2023-10-30T14:51:47.811285+0000","last_clean":"2023-10-30T14:51:47.811285+0000","last_became_active":"2023-10-30T14:14:03.702513+0000","last_became_peered":"2023-10-30T14:14:03.702513+0000","last_unstale":"2023-10-30T14:51:47.811285+0000","last_undegraded":"2023-10-30T14:51:47.811285+0000","last_fullsized":"2023-10-30T14:51:47.811285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1782","last_scrub_stamp":"2023-10-30T14:51:47.811149+0000","last_deep_scrub":"67'1782","last_deep_scrub_stamp":"2023-10-30T14:51:47.811149+0000","last_clean_scrub_stamp":"2023-10-30T14:51:47.811149+0000","objects_scrubbed":55,"log_size":1782,"log_dups_size":0,"ondisk_log_size":1782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:45:13.943363+0000","scrub_duration":0.120464197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27782952,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":1112,"num_read_kb":113704,"num_write":1847,"num_write_kb":62106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":736919,"num_omap_keys":1275,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"67'1334","reported_seq":3286,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.581275+0000","last_change":"2023-10-30T14:23:30.770719+0000","last_active":"2023-10-30T14:51:34.581275+0000","last_peered":"2023-10-30T14:51:34.581275+0000","last_clean":"2023-10-30T14:51:34.581275+0000","last_became_active":"2023-10-30T14:14:04.589087+0000","last_became_peered":"2023-10-30T14:14:04.589087+0000","last_unstale":"2023-10-30T14:51:34.581275+0000","last_undegraded":"2023-10-30T14:51:34.581275+0000","last_fullsized":"2023-10-30T14:51:34.581275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1334,"log_dups_size":0,"ondisk_log_size":1334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022871390000000001,"stat_sum":{"num_bytes":30609501,"num_objects":512,"num_object_clones":244,"num_object_copies":1024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":512,"num_whiteouts":220,"num_read":78,"num_read_kb":20511,"num_write":1068,"num_write_kb":36447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"67'656","reported_seq":816,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:42.807191+0000","last_change":"2023-10-30T14:51:42.807310+0000","last_active":"2023-10-30T14:51:42.807191+0000","last_peered":"2023-10-30T14:51:42.807191+0000","last_clean":"2023-10-30T14:51:42.807191+0000","last_became_active":"2023-10-30T14:14:03.728925+0000","last_became_peered":"2023-10-30T14:14:03.728925+0000","last_unstale":"2023-10-30T14:51:42.807191+0000","last_undegraded":"2023-10-30T14:51:42.807191+0000","last_fullsized":"2023-10-30T14:51:42.807191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'656","last_scrub_stamp":"2023-10-30T14:51:42.806981+0000","last_deep_scrub":"67'656","last_deep_scrub_stamp":"2023-10-30T14:51:42.806981+0000","last_clean_scrub_stamp":"2023-10-30T14:51:42.806981+0000","objects_scrubbed":40,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:27:15.123985+0000","scrub_duration":0.119997573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":120,"num_read_kb":131794,"num_write":661,"num_write_kb":62442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":535152,"num_omap_keys":914,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"67'1843","reported_seq":3822,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:49.103177+0000","last_change":"2023-10-30T14:51:49.103277+0000","last_active":"2023-10-30T14:51:49.103177+0000","last_peered":"2023-10-30T14:51:49.103177+0000","last_clean":"2023-10-30T14:51:49.103177+0000","last_became_active":"2023-10-30T14:14:04.607377+0000","last_became_peered":"2023-10-30T14:14:04.607377+0000","last_unstale":"2023-10-30T14:51:49.103177+0000","last_undegraded":"2023-10-30T14:51:49.103177+0000","last_fullsized":"2023-10-30T14:51:49.103177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1843","last_scrub_stamp":"2023-10-30T14:51:49.103084+0000","last_deep_scrub":"67'1843","last_deep_scrub_stamp":"2023-10-30T14:51:49.103084+0000","last_clean_scrub_stamp":"2023-10-30T14:51:49.103084+0000","objects_scrubbed":503,"log_size":1843,"log_dups_size":0,"ondisk_log_size":1843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:54:01.899046+0000","scrub_duration":1.0628055430000001,"objects_trimmed":0,"snaptrim_duration":0.0022021179999999999,"stat_sum":{"num_bytes":18168327,"num_objects":503,"num_object_clones":232,"num_object_copies":1006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":207,"num_read":65,"num_read_kb":9172,"num_write":1122,"num_write_kb":20929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"67'909","reported_seq":1057,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:52.423853+0000","last_change":"2023-10-30T14:51:52.423913+0000","last_active":"2023-10-30T14:51:52.423853+0000","last_peered":"2023-10-30T14:51:52.423853+0000","last_clean":"2023-10-30T14:51:52.423853+0000","last_became_active":"2023-10-30T14:14:03.692621+0000","last_became_peered":"2023-10-30T14:14:03.692621+0000","last_unstale":"2023-10-30T14:51:52.423853+0000","last_undegraded":"2023-10-30T14:51:52.423853+0000","last_fullsized":"2023-10-30T14:51:52.423853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'909","last_scrub_stamp":"2023-10-30T14:51:52.423719+0000","last_deep_scrub":"67'909","last_deep_scrub_stamp":"2023-10-30T14:51:52.423719+0000","last_clean_scrub_stamp":"2023-10-30T14:51:52.423719+0000","objects_scrubbed":43,"log_size":909,"log_dups_size":0,"ondisk_log_size":909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:15:28.937000+0000","scrub_duration":0.14411457899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":107,"num_read_kb":66236,"num_write":924,"num_write_kb":62670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":536879,"num_omap_keys":1021,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'1349","reported_seq":3279,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:40.619073+0000","last_change":"2023-10-30T14:23:30.780258+0000","last_active":"2023-10-30T14:51:40.619073+0000","last_peered":"2023-10-30T14:51:40.619073+0000","last_clean":"2023-10-30T14:51:40.619073+0000","last_became_active":"2023-10-30T14:14:04.613780+0000","last_became_peered":"2023-10-30T14:14:04.613780+0000","last_unstale":"2023-10-30T14:51:40.619073+0000","last_undegraded":"2023-10-30T14:51:40.619073+0000","last_fullsized":"2023-10-30T14:51:40.619073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1349,"log_dups_size":0,"ondisk_log_size":1349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065285270000000001,"stat_sum":{"num_bytes":12750235,"num_objects":524,"num_object_clones":251,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":229,"num_read":63,"num_read_kb":8216,"num_write":1081,"num_write_kb":18064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"67'722","reported_seq":886,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:37.470630+0000","last_change":"2023-10-30T14:14:03.740213+0000","last_active":"2023-10-30T14:51:37.470630+0000","last_peered":"2023-10-30T14:51:37.470630+0000","last_clean":"2023-10-30T14:51:37.470630+0000","last_became_active":"2023-10-30T14:14:03.739650+0000","last_became_peered":"2023-10-30T14:14:03.739650+0000","last_unstale":"2023-10-30T14:51:37.470630+0000","last_undegraded":"2023-10-30T14:51:37.470630+0000","last_fullsized":"2023-10-30T14:51:37.470630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":722,"log_dups_size":0,"ondisk_log_size":722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777238,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":143,"num_read_kb":109192,"num_write":780,"num_write_kb":46441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"67'1336","reported_seq":3293,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:37.094896+0000","last_change":"2023-10-30T14:23:30.771719+0000","last_active":"2023-10-30T14:51:37.094896+0000","last_peered":"2023-10-30T14:51:37.094896+0000","last_clean":"2023-10-30T14:51:37.094896+0000","last_became_active":"2023-10-30T14:14:04.590138+0000","last_became_peered":"2023-10-30T14:14:04.590138+0000","last_unstale":"2023-10-30T14:51:37.094896+0000","last_undegraded":"2023-10-30T14:51:37.094896+0000","last_fullsized":"2023-10-30T14:51:37.094896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1336,"log_dups_size":0,"ondisk_log_size":1336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031364370000000002,"stat_sum":{"num_bytes":9254236,"num_objects":503,"num_object_clones":233,"num_object_copies":1006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":207,"num_read":72,"num_read_kb":4145,"num_write":1099,"num_write_kb":19935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"67'792","reported_seq":944,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:43.351494+0000","last_change":"2023-10-30T14:51:43.351549+0000","last_active":"2023-10-30T14:51:43.351494+0000","last_peered":"2023-10-30T14:51:43.351494+0000","last_clean":"2023-10-30T14:51:43.351494+0000","last_became_active":"2023-10-30T14:14:03.721234+0000","last_became_peered":"2023-10-30T14:14:03.721234+0000","last_unstale":"2023-10-30T14:51:43.351494+0000","last_undegraded":"2023-10-30T14:51:43.351494+0000","last_fullsized":"2023-10-30T14:51:43.351494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'792","last_scrub_stamp":"2023-10-30T14:51:43.351414+0000","last_deep_scrub":"67'792","last_deep_scrub_stamp":"2023-10-30T14:51:43.351414+0000","last_clean_scrub_stamp":"2023-10-30T14:51:43.351414+0000","objects_scrubbed":34,"log_size":792,"log_dups_size":0,"ondisk_log_size":792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:49:31.872445+0000","scrub_duration":0.195047427,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":136,"num_read_kb":37589,"num_write":1318,"num_write_kb":40351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":332687,"num_omap_keys":642,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"67'1265","reported_seq":3220,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.636860+0000","last_change":"2023-10-30T14:23:31.070447+0000","last_active":"2023-10-30T14:51:34.636860+0000","last_peered":"2023-10-30T14:51:34.636860+0000","last_clean":"2023-10-30T14:51:34.636860+0000","last_became_active":"2023-10-30T14:14:04.633643+0000","last_became_peered":"2023-10-30T14:14:04.633643+0000","last_unstale":"2023-10-30T14:51:34.636860+0000","last_undegraded":"2023-10-30T14:51:34.636860+0000","last_fullsized":"2023-10-30T14:51:34.636860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1265,"log_dups_size":0,"ondisk_log_size":1265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048916599999999999,"stat_sum":{"num_bytes":9341707,"num_objects":489,"num_object_clones":229,"num_object_copies":978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":206,"num_read":67,"num_read_kb":4121,"num_write":1047,"num_write_kb":15591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"67'464","reported_seq":611,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:44.149414+0000","last_change":"2023-10-30T14:51:44.149516+0000","last_active":"2023-10-30T14:51:44.149414+0000","last_peered":"2023-10-30T14:51:44.149414+0000","last_clean":"2023-10-30T14:51:44.149414+0000","last_became_active":"2023-10-30T14:14:03.737801+0000","last_became_peered":"2023-10-30T14:14:03.737801+0000","last_unstale":"2023-10-30T14:51:44.149414+0000","last_undegraded":"2023-10-30T14:51:44.149414+0000","last_fullsized":"2023-10-30T14:51:44.149414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'464","last_scrub_stamp":"2023-10-30T14:51:44.149257+0000","last_deep_scrub":"67'464","last_deep_scrub_stamp":"2023-10-30T14:51:44.149257+0000","last_clean_scrub_stamp":"2023-10-30T14:51:44.149257+0000","objects_scrubbed":47,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:41:08.565700+0000","scrub_duration":0.15054411200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":132,"num_read_kb":45549,"num_write":534,"num_write_kb":26063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":658163,"num_omap_keys":1051,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"67'1256","reported_seq":3229,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.640809+0000","last_change":"2023-10-30T14:23:30.776079+0000","last_active":"2023-10-30T14:51:34.640809+0000","last_peered":"2023-10-30T14:51:34.640809+0000","last_clean":"2023-10-30T14:51:34.640809+0000","last_became_active":"2023-10-30T14:14:04.588826+0000","last_became_peered":"2023-10-30T14:14:04.588826+0000","last_unstale":"2023-10-30T14:51:34.640809+0000","last_undegraded":"2023-10-30T14:51:34.640809+0000","last_fullsized":"2023-10-30T14:51:34.640809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1256,"log_dups_size":0,"ondisk_log_size":1256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040592670000000001,"stat_sum":{"num_bytes":8300341,"num_objects":453,"num_object_clones":211,"num_object_copies":906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":453,"num_whiteouts":181,"num_read":67,"num_read_kb":4125,"num_write":1053,"num_write_kb":10742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"67'452","reported_seq":574,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:55.355279+0000","last_change":"2023-10-30T14:51:55.355412+0000","last_active":"2023-10-30T14:51:55.355279+0000","last_peered":"2023-10-30T14:51:55.355279+0000","last_clean":"2023-10-30T14:51:55.355279+0000","last_became_active":"2023-10-30T14:14:03.732912+0000","last_became_peered":"2023-10-30T14:14:03.732912+0000","last_unstale":"2023-10-30T14:51:55.355279+0000","last_undegraded":"2023-10-30T14:51:55.355279+0000","last_fullsized":"2023-10-30T14:51:55.355279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'452","last_scrub_stamp":"2023-10-30T14:51:55.355171+0000","last_deep_scrub":"67'452","last_deep_scrub_stamp":"2023-10-30T14:51:55.355171+0000","last_clean_scrub_stamp":"2023-10-30T14:51:55.355171+0000","objects_scrubbed":23,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:16.834277+0000","scrub_duration":0.040337339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":78,"num_read_kb":41242,"num_write":429,"num_write_kb":21093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":203611,"num_omap_keys":381,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"67'1735","reported_seq":3730,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:49.980818+0000","last_change":"2023-10-30T14:51:49.980998+0000","last_active":"2023-10-30T14:51:49.980818+0000","last_peered":"2023-10-30T14:51:49.980818+0000","last_clean":"2023-10-30T14:51:49.980818+0000","last_became_active":"2023-10-30T14:14:04.609835+0000","last_became_peered":"2023-10-30T14:14:04.609835+0000","last_unstale":"2023-10-30T14:51:49.980818+0000","last_undegraded":"2023-10-30T14:51:49.980818+0000","last_fullsized":"2023-10-30T14:51:49.980818+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1735","last_scrub_stamp":"2023-10-30T14:51:49.980600+0000","last_deep_scrub":"67'1735","last_deep_scrub_stamp":"2023-10-30T14:51:49.980600+0000","last_clean_scrub_stamp":"2023-10-30T14:51:49.980600+0000","objects_scrubbed":470,"log_size":1735,"log_dups_size":0,"ondisk_log_size":1735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:25:11.080101+0000","scrub_duration":0.905012963,"objects_trimmed":0,"snaptrim_duration":0.0023958389999999999,"stat_sum":{"num_bytes":25940957,"num_objects":470,"num_object_clones":215,"num_object_copies":940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":192,"num_read":67,"num_read_kb":20495,"num_write":1067,"num_write_kb":27713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"67'884","reported_seq":1035,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:40.201508+0000","last_change":"2023-10-30T14:51:40.201605+0000","last_active":"2023-10-30T14:51:40.201508+0000","last_peered":"2023-10-30T14:51:40.201508+0000","last_clean":"2023-10-30T14:51:40.201508+0000","last_became_active":"2023-10-30T14:14:03.732265+0000","last_became_peered":"2023-10-30T14:14:03.732265+0000","last_unstale":"2023-10-30T14:51:40.201508+0000","last_undegraded":"2023-10-30T14:51:40.201508+0000","last_fullsized":"2023-10-30T14:51:40.201508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'884","last_scrub_stamp":"2023-10-30T14:51:40.201423+0000","last_deep_scrub":"67'884","last_deep_scrub_stamp":"2023-10-30T14:51:40.201423+0000","last_clean_scrub_stamp":"2023-10-30T14:51:40.201423+0000","objects_scrubbed":41,"log_size":884,"log_dups_size":0,"ondisk_log_size":884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:01:53.178520+0000","scrub_duration":0.062050998000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":128,"num_read_kb":49868,"num_write":925,"num_write_kb":26024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":535963,"num_omap_keys":904,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"67'1787","reported_seq":3793,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:49.263007+0000","last_change":"2023-10-30T14:51:49.263072+0000","last_active":"2023-10-30T14:51:49.263007+0000","last_peered":"2023-10-30T14:51:49.263007+0000","last_clean":"2023-10-30T14:51:49.263007+0000","last_became_active":"2023-10-30T14:14:04.594998+0000","last_became_peered":"2023-10-30T14:14:04.594998+0000","last_unstale":"2023-10-30T14:51:49.263007+0000","last_undegraded":"2023-10-30T14:51:49.263007+0000","last_fullsized":"2023-10-30T14:51:49.263007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1787","last_scrub_stamp":"2023-10-30T14:51:49.262934+0000","last_deep_scrub":"67'1787","last_deep_scrub_stamp":"2023-10-30T14:51:49.262934+0000","last_clean_scrub_stamp":"2023-10-30T14:51:49.262934+0000","objects_scrubbed":466,"log_size":1787,"log_dups_size":0,"ondisk_log_size":1787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:53:31.315621+0000","scrub_duration":0.73282899099999999,"objects_trimmed":0,"snaptrim_duration":0.00432118,"stat_sum":{"num_bytes":12343269,"num_objects":466,"num_object_clones":216,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":186,"num_read":78,"num_read_kb":8212,"num_write":1132,"num_write_kb":18560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'841","reported_seq":997,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:41.239916+0000","last_change":"2023-10-30T14:51:41.239981+0000","last_active":"2023-10-30T14:51:41.239916+0000","last_peered":"2023-10-30T14:51:41.239916+0000","last_clean":"2023-10-30T14:51:41.239916+0000","last_became_active":"2023-10-30T14:14:03.701759+0000","last_became_peered":"2023-10-30T14:14:03.701759+0000","last_unstale":"2023-10-30T14:51:41.239916+0000","last_undegraded":"2023-10-30T14:51:41.239916+0000","last_fullsized":"2023-10-30T14:51:41.239916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'841","last_scrub_stamp":"2023-10-30T14:51:41.239826+0000","last_deep_scrub":"67'841","last_deep_scrub_stamp":"2023-10-30T14:51:41.239826+0000","last_clean_scrub_stamp":"2023-10-30T14:51:41.239826+0000","objects_scrubbed":45,"log_size":841,"log_dups_size":0,"ondisk_log_size":841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:13:10.834903+0000","scrub_duration":0.095062992999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":122,"num_read_kb":94942,"num_write":1183,"num_write_kb":51323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":515565,"num_omap_keys":927,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"67'1241","reported_seq":3221,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.645956+0000","last_change":"2023-10-30T14:23:31.071315+0000","last_active":"2023-10-30T14:51:34.645956+0000","last_peered":"2023-10-30T14:51:34.645956+0000","last_clean":"2023-10-30T14:51:34.645956+0000","last_became_active":"2023-10-30T14:14:04.600100+0000","last_became_peered":"2023-10-30T14:14:04.600100+0000","last_unstale":"2023-10-30T14:51:34.645956+0000","last_undegraded":"2023-10-30T14:51:34.645956+0000","last_fullsized":"2023-10-30T14:51:34.645956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1241,"log_dups_size":0,"ondisk_log_size":1241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052227630000000001,"stat_sum":{"num_bytes":16881451,"num_objects":447,"num_object_clones":209,"num_object_copies":894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":182,"num_read":73,"num_read_kb":11653,"num_write":1021,"num_write_kb":18895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"67'773","reported_seq":907,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:32.363249+0000","last_change":"2023-10-30T14:14:03.741268+0000","last_active":"2023-10-30T14:51:32.363249+0000","last_peered":"2023-10-30T14:51:32.363249+0000","last_clean":"2023-10-30T14:51:32.363249+0000","last_became_active":"2023-10-30T14:14:03.740648+0000","last_became_peered":"2023-10-30T14:14:03.740648+0000","last_unstale":"2023-10-30T14:51:32.363249+0000","last_undegraded":"2023-10-30T14:51:32.363249+0000","last_fullsized":"2023-10-30T14:51:32.363249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":98,"num_read_kb":107365,"num_write":798,"num_write_kb":50458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"67'1295","reported_seq":3277,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.429644+0000","last_change":"2023-10-30T14:23:31.073589+0000","last_active":"2023-10-30T14:51:34.429644+0000","last_peered":"2023-10-30T14:51:34.429644+0000","last_clean":"2023-10-30T14:51:34.429644+0000","last_became_active":"2023-10-30T14:14:04.622921+0000","last_became_peered":"2023-10-30T14:14:04.622921+0000","last_unstale":"2023-10-30T14:51:34.429644+0000","last_undegraded":"2023-10-30T14:51:34.429644+0000","last_fullsized":"2023-10-30T14:51:34.429644+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0071219910000000003,"stat_sum":{"num_bytes":20799446,"num_objects":465,"num_object_clones":210,"num_object_copies":930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":465,"num_whiteouts":183,"num_read":75,"num_read_kb":12323,"num_write":1101,"num_write_kb":26545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"67'1332","reported_seq":3287,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:37.103057+0000","last_change":"2023-10-30T14:23:30.779989+0000","last_active":"2023-10-30T14:51:37.103057+0000","last_peered":"2023-10-30T14:51:37.103057+0000","last_clean":"2023-10-30T14:51:37.103057+0000","last_became_active":"2023-10-30T14:14:04.621885+0000","last_became_peered":"2023-10-30T14:14:04.621885+0000","last_unstale":"2023-10-30T14:51:37.103057+0000","last_undegraded":"2023-10-30T14:51:37.103057+0000","last_fullsized":"2023-10-30T14:51:37.103057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1332,"log_dups_size":0,"ondisk_log_size":1332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067851329999999996,"stat_sum":{"num_bytes":13483294,"num_objects":487,"num_object_clones":228,"num_object_copies":974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":191,"num_read":68,"num_read_kb":8221,"num_write":1123,"num_write_kb":20773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"67'469","reported_seq":620,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:37.470540+0000","last_change":"2023-10-30T14:14:03.740897+0000","last_active":"2023-10-30T14:51:37.470540+0000","last_peered":"2023-10-30T14:51:37.470540+0000","last_clean":"2023-10-30T14:51:37.470540+0000","last_became_active":"2023-10-30T14:14:03.740247+0000","last_became_peered":"2023-10-30T14:14:03.740247+0000","last_unstale":"2023-10-30T14:51:37.470540+0000","last_undegraded":"2023-10-30T14:51:37.470540+0000","last_fullsized":"2023-10-30T14:51:37.470540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":149,"num_read_kb":82851,"num_write":526,"num_write_kb":42289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'1418","reported_seq":3362,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:37.103014+0000","last_change":"2023-10-30T14:23:30.776236+0000","last_active":"2023-10-30T14:51:37.103014+0000","last_peered":"2023-10-30T14:51:37.103014+0000","last_clean":"2023-10-30T14:51:37.103014+0000","last_became_active":"2023-10-30T14:14:04.623638+0000","last_became_peered":"2023-10-30T14:14:04.623638+0000","last_unstale":"2023-10-30T14:51:37.103014+0000","last_undegraded":"2023-10-30T14:51:37.103014+0000","last_fullsized":"2023-10-30T14:51:37.103014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1418,"log_dups_size":0,"ondisk_log_size":1418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049828509999999999,"stat_sum":{"num_bytes":25534330,"num_objects":546,"num_object_clones":248,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":546,"num_whiteouts":220,"num_read":77,"num_read_kb":19614,"num_write":1187,"num_write_kb":31851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"67'1028","reported_seq":1182,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:37.470589+0000","last_change":"2023-10-30T14:14:03.733233+0000","last_active":"2023-10-30T14:51:37.470589+0000","last_peered":"2023-10-30T14:51:37.470589+0000","last_clean":"2023-10-30T14:51:37.470589+0000","last_became_active":"2023-10-30T14:14:03.732538+0000","last_became_peered":"2023-10-30T14:14:03.732538+0000","last_unstale":"2023-10-30T14:51:37.470589+0000","last_undegraded":"2023-10-30T14:51:37.470589+0000","last_fullsized":"2023-10-30T14:51:37.470589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:02.656444+0000","last_clean_scrub_stamp":"2023-10-30T14:14:02.656444+0000","objects_scrubbed":0,"log_size":1028,"log_dups_size":0,"ondisk_log_size":1028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":137,"num_read_kb":150323,"num_write":1072,"num_write_kb":62947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1288","reported_seq":3239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:36.997729+0000","last_change":"2023-10-30T14:23:30.776451+0000","last_active":"2023-10-30T14:51:36.997729+0000","last_peered":"2023-10-30T14:51:36.997729+0000","last_clean":"2023-10-30T14:51:36.997729+0000","last_became_active":"2023-10-30T14:14:04.611773+0000","last_became_peered":"2023-10-30T14:14:04.611773+0000","last_unstale":"2023-10-30T14:51:36.997729+0000","last_undegraded":"2023-10-30T14:51:36.997729+0000","last_fullsized":"2023-10-30T14:51:36.997729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1288,"log_dups_size":0,"ondisk_log_size":1288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044924040000000002,"stat_sum":{"num_bytes":26364015,"num_objects":513,"num_object_clones":233,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":513,"num_whiteouts":205,"num_read":66,"num_read_kb":19602,"num_write":1073,"num_write_kb":27528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"67'747","reported_seq":898,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:40.646903+0000","last_change":"2023-10-30T14:51:40.646962+0000","last_active":"2023-10-30T14:51:40.646903+0000","last_peered":"2023-10-30T14:51:40.646903+0000","last_clean":"2023-10-30T14:51:40.646903+0000","last_became_active":"2023-10-30T14:14:03.703691+0000","last_became_peered":"2023-10-30T14:14:03.703691+0000","last_unstale":"2023-10-30T14:51:40.646903+0000","last_undegraded":"2023-10-30T14:51:40.646903+0000","last_fullsized":"2023-10-30T14:51:40.646903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'747","last_scrub_stamp":"2023-10-30T14:51:40.646821+0000","last_deep_scrub":"67'747","last_deep_scrub_stamp":"2023-10-30T14:51:40.646821+0000","last_clean_scrub_stamp":"2023-10-30T14:51:40.646821+0000","objects_scrubbed":37,"log_size":747,"log_dups_size":0,"ondisk_log_size":747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:10:48.146914+0000","scrub_duration":0.052199267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":102,"num_read_kb":111696,"num_write":764,"num_write_kb":58666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":552830,"num_omap_keys":982,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"67'1338","reported_seq":3273,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.514878+0000","last_change":"2023-10-30T14:23:31.069185+0000","last_active":"2023-10-30T14:51:34.514878+0000","last_peered":"2023-10-30T14:51:34.514878+0000","last_clean":"2023-10-30T14:51:34.514878+0000","last_became_active":"2023-10-30T14:14:04.613420+0000","last_became_peered":"2023-10-30T14:14:04.613420+0000","last_unstale":"2023-10-30T14:51:34.514878+0000","last_undegraded":"2023-10-30T14:51:34.514878+0000","last_fullsized":"2023-10-30T14:51:34.514878+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1338,"log_dups_size":0,"ondisk_log_size":1338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004010128,"stat_sum":{"num_bytes":13027616,"num_objects":509,"num_object_clones":238,"num_object_copies":1018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":509,"num_whiteouts":202,"num_read":55,"num_read_kb":8221,"num_write":1122,"num_write_kb":14511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"67'654","reported_seq":825,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:38.523338+0000","last_change":"2023-10-30T14:51:38.523396+0000","last_active":"2023-10-30T14:51:38.523338+0000","last_peered":"2023-10-30T14:51:38.523338+0000","last_clean":"2023-10-30T14:51:38.523338+0000","last_became_active":"2023-10-30T14:14:03.736964+0000","last_became_peered":"2023-10-30T14:14:03.736964+0000","last_unstale":"2023-10-30T14:51:38.523338+0000","last_undegraded":"2023-10-30T14:51:38.523338+0000","last_fullsized":"2023-10-30T14:51:38.523338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'654","last_scrub_stamp":"2023-10-30T14:51:38.523259+0000","last_deep_scrub":"67'654","last_deep_scrub_stamp":"2023-10-30T14:51:38.523259+0000","last_clean_scrub_stamp":"2023-10-30T14:51:38.523259+0000","objects_scrubbed":39,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:31:19.574364+0000","scrub_duration":0.063641798999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":182,"num_read_kb":58134,"num_write":696,"num_write_kb":34123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":484120,"num_omap_keys":814,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1055045924,"num_objects":31721,"num_object_clones":14708,"num_object_copies":63442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31721,"num_whiteouts":12864,"num_read":4602,"num_read_kb":662584,"num_write":69913,"num_write_kb":1350127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2186289152,"data_stored":2111320648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92306,"ondisk_log_size":92306,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":689694586,"num_objects":1357,"num_object_clones":0,"num_object_copies":2714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":6032,"num_read_kb":2835146,"num_write":28235,"num_write_kb":1523161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1127,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13231903,"num_omap_keys":22455,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1379532800,"data_stored":1379389172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26237,"ondisk_log_size":26237,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575310,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632732,"kb_used_data":489172,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93739108,"statfs":{"total":96636764160,"available":95988846592,"internally_reserved":0,"allocated":500912128,"data_stored":490062487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":146995659},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.077,"15min":0.997},"min":{"1min":0.51700000000000002,"5min":0.46300000000000002,"15min":0.45500000000000002},"max":{"1min":6.3040000000000003,"5min":6.399,"15min":6.399},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":0.97799999999999998,"15min":0.94399999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":6.4969999999999999,"5min":6.4969999999999999,"15min":6.4969999999999999},"last":0.84099999999999997}]},{"osd":1,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":0.92500000000000004,"15min":0.93000000000000005},"min":{"1min":0.61599999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":6.1390000000000002,"5min":6.1390000000000002,"15min":6.1390000000000002},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.96599999999999997,"15min":0.99399999999999999},"min":{"1min":0.58699999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":3.8460000000000001,"5min":6.1920000000000002,"15min":6.1920000000000002},"last":0.70099999999999996}]},{"osd":2,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.0449999999999999,"15min":0.96899999999999997},"min":{"1min":0.70099999999999996,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.5739999999999998,"5min":6.5739999999999998,"15min":6.5739999999999998},"last":1.01},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.115,"15min":1.01},"min":{"1min":0.81599999999999995,"5min":0.54500000000000004,"15min":0.5},"max":{"1min":6.2350000000000003,"5min":6.5149999999999997,"15min":6.5149999999999997},"last":1.085}]},{"osd":3,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.86199999999999999,"15min":0.85599999999999998},"min":{"1min":0.56200000000000006,"5min":0.54800000000000004,"15min":0.53800000000000003},"max":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":4.3230000000000004},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.121,"15min":1.056},"min":{"1min":0.66800000000000004,"5min":0.64800000000000002,"15min":0.54300000000000004},"max":{"1min":3.157,"5min":6.2670000000000003,"15min":6.2670000000000003},"last":1.2929999999999999}]},{"osd":4,"last update":"Mon Oct 30 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.95599999999999996,"15min":0.95699999999999996},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.54800000000000004},"max":{"1min":4.7409999999999997,"5min":4.7409999999999997,"15min":4.7409999999999997},"last":1.155},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.036,"15min":0.97399999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":8.0719999999999992,"5min":8.0719999999999992,"15min":8.0719999999999992},"last":1.25}]},{"osd":5,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3280000000000001,"15min":1.1759999999999999},"min":{"1min":0.78100000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":10.692,"5min":10.692,"15min":10.692},"last":1.369},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.3939999999999999,"15min":1.2869999999999999},"min":{"1min":0.72099999999999997,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":6.6429999999999998,"5min":6.6429999999999998,"15min":6.6429999999999998},"last":1.4390000000000001}]},{"osd":6,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.135,"15min":1.129},"min":{"1min":0.748,"5min":0.71599999999999997,"15min":0.62},"max":{"1min":5.2160000000000002,"5min":5.2160000000000002,"15min":5.2160000000000002},"last":1.327},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.1639999999999999,"15min":1.107},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":4.367,"5min":4.367,"15min":4.367},"last":1.4019999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575310,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513220,"kb_used_data":387708,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":93858620,"statfs":{"total":96636764160,"available":96111226880,"internally_reserved":0,"allocated":397012992,"data_stored":389718204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":128514503},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.82899999999999996,"15min":0.83199999999999996},"min":{"1min":0.503,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.216,"5min":1.288,"15min":1.2929999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.997,"5min":0.90500000000000003,"15min":0.88800000000000001},"min":{"1min":0.68600000000000005,"5min":0.435,"15min":0.435},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.391},"last":0.44}]},{"osd":1,"last update":"Mon Oct 30 14:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.89700000000000002,"15min":0.85599999999999998},"min":{"1min":0.66000000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.135,"5min":8,"15min":8},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.94199999999999995,"15min":0.96599999999999997},"min":{"1min":0.67300000000000004,"5min":0.53400000000000003,"15min":0.318},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.514},"last":0.61199999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.89300000000000002,"15min":0.89000000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.40999999999999998},"max":{"1min":1.232,"5min":1.2410000000000001,"15min":1.617},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.92900000000000005,"15min":0.94299999999999995},"min":{"1min":0.63400000000000001,"5min":0.59999999999999998,"15min":0.49299999999999999},"max":{"1min":1.181,"5min":1.3149999999999999,"15min":1.4119999999999999},"last":0.496}]},{"osd":3,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":0.997,"15min":0.98099999999999998},"min":{"1min":0.73999999999999999,"5min":0.57599999999999996,"15min":0.47799999999999998},"max":{"1min":1.3700000000000001,"5min":1.3720000000000001,"15min":1.6890000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.95499999999999996,"15min":0.96099999999999997},"min":{"1min":0.61499999999999999,"5min":0.59199999999999997,"15min":0.48899999999999999},"max":{"1min":1.286,"5min":1.286,"15min":1.472},"last":0.64000000000000001}]},{"osd":4,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.0309999999999999,"15min":1.032},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.59899999999999998},"max":{"1min":1.1830000000000001,"5min":3.8399999999999999,"15min":3.8399999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":1.016,"5min":1.0269999999999999,"15min":1.04},"min":{"1min":0.69999999999999996,"5min":0.62,"15min":0.435},"max":{"1min":1.341,"5min":1.4730000000000001,"15min":1.645},"last":0.54700000000000004}]},{"osd":5,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":1.0149999999999999,"15min":0.97299999999999998},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.51100000000000001},"max":{"1min":1.1659999999999999,"5min":11.728,"15min":11.728},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.1000000000000001,"15min":1.0920000000000001},"min":{"1min":0.72899999999999998,"5min":0.51700000000000002,"15min":0.51400000000000001},"max":{"1min":1.2310000000000001,"5min":9.7759999999999998,"15min":9.7759999999999998},"last":0.70099999999999996}]},{"osd":7,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.224,"15min":1.137},"min":{"1min":0.76400000000000001,"5min":0.755,"15min":0.65200000000000002},"max":{"1min":1.268,"5min":18.041,"15min":18.041},"last":0.752},{"interface":"front","average":{"1min":1.181,"5min":1.345,"15min":1.228},"min":{"1min":0.98699999999999999,"5min":0.79700000000000004,"15min":0.56000000000000005},"max":{"1min":1.579,"5min":20.375,"15min":20.375},"last":0.84999999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575310,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682668,"kb_used_data":539108,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93689172,"statfs":{"total":96636764160,"available":95937712128,"internally_reserved":0,"allocated":552046592,"data_stored":540347463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":146995654},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:51 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.84399999999999997,"15min":0.81399999999999995},"min":{"1min":0.47699999999999998,"5min":0.43099999999999999,"15min":0.40100000000000002},"max":{"1min":1.2070000000000001,"5min":7.04,"15min":7.04},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.89500000000000002,"15min":0.84999999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.202,"5min":5.4720000000000004,"15min":5.4720000000000004},"last":0.38800000000000001}]},{"osd":1,"last update":"Mon Oct 30 14:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.873,"15min":0.879},"min":{"1min":0.53400000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.2390000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.84299999999999997,"15min":0.84499999999999997},"min":{"1min":0.53500000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0569999999999999,"5min":4.1630000000000003,"15min":4.1630000000000003},"last":0.44900000000000001}]},{"osd":2,"last update":"Mon Oct 30 14:51:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.92300000000000004,"15min":0.95999999999999996},"min":{"1min":0.61299999999999999,"5min":0.503,"15min":0.44800000000000001},"max":{"1min":1.3320000000000001,"5min":1.3640000000000001,"15min":2.044},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.087,"15min":1.024},"min":{"1min":0.81899999999999995,"5min":0.69099999999999995,"15min":0.51600000000000001},"max":{"1min":1.425,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":0.93899999999999995}]},{"osd":3,"last update":"Mon Oct 30 14:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.83199999999999996,"15min":0.92400000000000004},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0840000000000001,"5min":1.2010000000000001,"15min":1.4550000000000001},"last":0.628},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.0669999999999999,"15min":0.98399999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.52100000000000002},"max":{"1min":1.3919999999999999,"5min":1.3959999999999999,"15min":1.4059999999999999},"last":1.0029999999999999}]},{"osd":4,"last update":"Mon Oct 30 14:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.1160000000000001,"15min":1.032},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.59099999999999997},"max":{"1min":1.046,"5min":8.7360000000000007,"15min":8.7360000000000007},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.129,"5min":1.27,"15min":1.0760000000000001},"min":{"1min":0.73199999999999998,"5min":0.61199999999999999,"15min":0.59399999999999997},"max":{"1min":1.5980000000000001,"5min":23.640000000000001,"15min":23.640000000000001},"last":1.0600000000000001}]},{"osd":6,"last update":"Mon Oct 30 14:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1539999999999999,"15min":1.0800000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.55500000000000005},"max":{"1min":1.657,"5min":1.7470000000000001,"15min":2.306},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.3,"5min":1.2330000000000001,"15min":1.139},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.57899999999999996},"max":{"1min":1.6120000000000001,"5min":1.7569999999999999,"15min":2.774},"last":1.2050000000000001}]},{"osd":7,"last update":"Mon Oct 30 14:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0029999999999999,"15min":1.0569999999999999},"min":{"1min":0.66700000000000004,"5min":0.64300000000000002,"15min":0.40799999999999997},"max":{"1min":1.532,"5min":1.532,"15min":3.0739999999999998},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0940000000000001,"15min":1.163},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.77600000000000002},"max":{"1min":1.1970000000000001,"5min":1.53,"15min":2.1339999999999999},"last":0.90400000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575310,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627544,"kb_used_data":479824,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":93744296,"statfs":{"total":96636764160,"available":95994159104,"internally_reserved":0,"allocated":491339776,"data_stored":479836699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151255494},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.0109999999999999,"15min":0.876},"min":{"1min":0.53300000000000003,"5min":0.48199999999999998,"15min":0.27900000000000003},"max":{"1min":3.222,"5min":7.1870000000000003,"15min":7.1870000000000003},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.099,"5min":1.107,"15min":0.95099999999999996},"min":{"1min":0.68899999999999995,"5min":0.48199999999999998,"15min":0.33900000000000002},"max":{"1min":3.3010000000000002,"5min":10.864000000000001,"15min":10.864000000000001},"last":0.81000000000000005}]},{"osd":1,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.04,"15min":0.95199999999999996},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.50700000000000001},"max":{"1min":3.3780000000000001,"5min":7.6980000000000004,"15min":7.6980000000000004},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.91300000000000003,"15min":0.83999999999999997},"min":{"1min":0.58699999999999997,"5min":0.48399999999999999,"15min":0.36599999999999999},"max":{"1min":2.1789999999999998,"5min":5.5259999999999998,"15min":5.5259999999999998},"last":0.64800000000000002}]},{"osd":2,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0229999999999999,"15min":0.96199999999999997},"min":{"1min":0.64300000000000002,"5min":0.57499999999999996,"15min":0.40899999999999997},"max":{"1min":2.2839999999999998,"5min":5.423,"15min":5.423},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.254,"15min":1.0940000000000001},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.54400000000000004},"max":{"1min":3.4500000000000002,"5min":13.221,"15min":13.221},"last":1.087}]},{"osd":3,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.96099999999999997,"15min":0.86199999999999999},"min":{"1min":0.626,"5min":0.54800000000000004,"15min":0.46899999999999997},"max":{"1min":7.5620000000000003,"5min":7.5620000000000003,"15min":7.5620000000000003},"last":0.68500000000000005},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.0860000000000001,"15min":1.0029999999999999},"min":{"1min":0.71199999999999997,"5min":0.65500000000000003,"15min":0.497},"max":{"1min":7.7370000000000001,"5min":7.7370000000000001,"15min":7.7370000000000001},"last":0.91700000000000004}]},{"osd":5,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.091,"15min":0.97599999999999998},"min":{"1min":0.72499999999999998,"5min":0.64200000000000002,"15min":0.317},"max":{"1min":7.6360000000000001,"5min":7.6360000000000001,"15min":7.6360000000000001},"last":1.008},{"interface":"front","average":{"1min":1.159,"5min":1.2609999999999999,"15min":1.145},"min":{"1min":0.628,"5min":0.628,"15min":0.442},"max":{"1min":3.6389999999999998,"5min":7.2599999999999998,"15min":7.2599999999999998},"last":1.147}]},{"osd":6,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1910000000000001,"15min":1.1020000000000001},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.39200000000000002},"max":{"1min":2.379,"5min":5.4729999999999999,"15min":5.4729999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.2969999999999999,"15min":1.1830000000000001},"min":{"1min":0.84299999999999997,"5min":0.73999999999999999,"15min":0.47899999999999998},"max":{"1min":6.5270000000000001,"5min":10.102,"15min":10.102},"last":1.1910000000000001}]},{"osd":7,"last update":"Mon Oct 30 14:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.1120000000000001,"15min":1.042},"min":{"1min":0.75700000000000001,"5min":0.69099999999999995,"15min":0.40100000000000002},"max":{"1min":9.1240000000000006,"5min":9.1240000000000006,"15min":9.1240000000000006},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.423,"5min":1.2809999999999999,"15min":1.2130000000000001},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.52900000000000003},"max":{"1min":7.0490000000000004,"5min":7.0490000000000004,"15min":7.0490000000000004},"last":1.04}]}]},{"osd":3,"up_from":13,"seq":55834575308,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":652044,"kb_used_data":508484,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93719796,"statfs":{"total":96636764160,"available":95969071104,"internally_reserved":0,"allocated":520687616,"data_stored":509150004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":146995656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.53600000000000003,"15min":0.48299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.1130000000000004,"5min":4.1130000000000004,"15min":4.1130000000000004},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.54200000000000004,"15min":0.51200000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.91100000000000003,"5min":4.5190000000000001,"15min":4.5190000000000001},"last":0.435}]},{"osd":1,"last update":"Mon Oct 30 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.58999999999999997,"15min":0.54400000000000004},"min":{"1min":0.33100000000000002,"5min":0.317,"15min":0.314},"max":{"1min":3.988,"5min":3.988,"15min":3.988},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.56399999999999995,"15min":0.54900000000000004},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.29999999999999999},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.48799999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.65600000000000003,"15min":0.56100000000000005},"min":{"1min":0.31,"5min":0.29199999999999998,"15min":0.27600000000000002},"max":{"1min":1.0740000000000001,"5min":8.6519999999999992,"15min":8.6519999999999992},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.73999999999999999,"15min":0.63700000000000001},"min":{"1min":0.42199999999999999,"5min":0.41999999999999998,"15min":0.313},"max":{"1min":1.1040000000000001,"5min":7.0949999999999998,"15min":7.0949999999999998},"last":0.65400000000000003}]},{"osd":4,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.83399999999999996,"15min":0.82099999999999995},"min":{"1min":0.54600000000000004,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.1220000000000001,"5min":4.5730000000000004,"15min":4.5730000000000004},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.90200000000000002,"15min":0.85299999999999998},"min":{"1min":0.51500000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":4.0910000000000002,"5min":4.4720000000000004,"15min":4.4720000000000004},"last":0.67000000000000004}]},{"osd":5,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.0029999999999999,"15min":0.90000000000000002},"min":{"1min":0.72799999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.296,"5min":14.974,"15min":14.974},"last":0.627},{"interface":"front","average":{"1min":1.042,"5min":0.96699999999999997,"15min":0.873},"min":{"1min":0.746,"5min":0.52600000000000002,"15min":0.46600000000000003},"max":{"1min":4.0629999999999997,"5min":5.907,"15min":5.907},"last":0.71899999999999997}]},{"osd":6,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.01,"15min":0.91900000000000004},"min":{"1min":0.70899999999999996,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":5.8140000000000001,"5min":5.8300000000000001,"15min":5.8300000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.159,"5min":1.073,"15min":0.96299999999999997},"min":{"1min":0.70799999999999996,"5min":0.60199999999999998,"15min":0.55100000000000005},"max":{"1min":6.0300000000000002,"5min":6.0300000000000002,"15min":6.0300000000000002},"last":0.90900000000000003}]},{"osd":7,"last update":"Mon Oct 30 14:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":1.115,"15min":0.995},"min":{"1min":0.67600000000000005,"5min":0.66800000000000004,"15min":0.61699999999999999},"max":{"1min":1.2549999999999999,"5min":16.282,"15min":16.282},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.123,"15min":0.97199999999999998},"min":{"1min":0.70999999999999996,"5min":0.627,"15min":0.58099999999999996},"max":{"1min":1.6519999999999999,"5min":16.096,"15min":16.096},"last":0.86299999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575310,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409788,"kb_used_data":310644,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":93962052,"statfs":{"total":96636764160,"available":96217141248,"internally_reserved":0,"allocated":318099456,"data_stored":311268353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101513670},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.45300000000000001,"15min":0.48699999999999999},"min":{"1min":0.30099999999999999,"5min":0.27400000000000002,"15min":0.26300000000000001},"max":{"1min":0.67800000000000005,"5min":0.79100000000000004,"15min":13.526},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.50900000000000001,"15min":0.51300000000000001},"min":{"1min":0.38300000000000001,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":1.2190000000000001},"last":0.46100000000000002}]},{"osd":1,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46300000000000002,"15min":0.47199999999999998},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65500000000000003,"5min":0.67200000000000004,"15min":1.5469999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54400000000000004,"15min":0.56000000000000005},"min":{"1min":0.38700000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73499999999999999,"5min":0.79300000000000004,"15min":1.452},"last":0.47999999999999998}]},{"osd":3,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58799999999999997,"15min":0.60699999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81299999999999994,"5min":0.95399999999999996,"15min":1.9390000000000001},"last":0.438},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.53900000000000003},"min":{"1min":0.35899999999999999,"5min":0.35499999999999998,"15min":0.33100000000000002},"max":{"1min":0.78500000000000003,"5min":0.79300000000000004,"15min":1.913},"last":0.37}]},{"osd":4,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.81699999999999995,"15min":0.81499999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.44900000000000001},"max":{"1min":1.0309999999999999,"5min":4.2229999999999999,"15min":4.2229999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.91600000000000004,"15min":0.91000000000000003},"min":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.59599999999999997},"max":{"1min":1.081,"5min":4.2439999999999998,"15min":4.2439999999999998},"last":0.751}]},{"osd":5,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.98999999999999999,"15min":0.86399999999999999},"min":{"1min":0.51900000000000002,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0620000000000001,"5min":22.285,"15min":22.285},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.82899999999999996,"15min":0.83399999999999996},"min":{"1min":0.58199999999999996,"5min":0.56000000000000005,"15min":0.51900000000000002},"max":{"1min":1.1319999999999999,"5min":2.3999999999999999,"15min":2.742},"last":0.64200000000000002}]},{"osd":6,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.86099999999999999,"15min":0.91500000000000004},"min":{"1min":0.65000000000000002,"5min":0.58799999999999997,"15min":0.54500000000000004},"max":{"1min":1.0489999999999999,"5min":2.911,"15min":27.731000000000002},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.872,"15min":0.874},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.1679999999999999,"5min":1.9099999999999999,"15min":2.0009999999999999},"last":0.61299999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84299999999999997,"15min":0.91600000000000004},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.56499999999999995},"max":{"1min":1.105,"5min":1.2390000000000001,"15min":18.870999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.90400000000000003,"15min":0.96399999999999997},"min":{"1min":0.66600000000000004,"5min":0.61499999999999999,"15min":0.55500000000000005},"max":{"1min":1.0860000000000001,"5min":3.3809999999999998,"15min":18.356999999999999},"last":0.79100000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575309,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472564,"kb_used_data":351212,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":93899276,"statfs":{"total":96636764160,"available":96152858624,"internally_reserved":0,"allocated":359641088,"data_stored":352841256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.47399999999999998,"15min":0.44800000000000001},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.68200000000000005,"5min":2.4889999999999999,"15min":2.4889999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.51600000000000001,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71199999999999997,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.61699999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.64500000000000002,"15min":0.56699999999999995},"min":{"1min":0.318,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79000000000000004,"5min":8.3160000000000007,"15min":8.3160000000000007},"last":0.501},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.54000000000000004,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.27200000000000002,"15min":0.26800000000000002},"max":{"1min":0.61399999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.40100000000000002}]},{"osd":3,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.61099999999999999,"15min":0.59599999999999997},"min":{"1min":0.38800000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84399999999999997,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.379},{"interface":"front","average":{"1min":0.623,"5min":0.64900000000000002,"15min":0.63600000000000001},"min":{"1min":0.439,"5min":0.38100000000000001,"15min":0.36499999999999999},"max":{"1min":0.89200000000000002,"5min":3.54,"15min":3.54},"last":0.70199999999999996}]},{"osd":4,"last update":"Mon Oct 30 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.92000000000000004,"15min":0.83999999999999997},"min":{"1min":0.505,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.92600000000000005,"5min":10.525,"15min":10.525},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.94899999999999995,"15min":0.83899999999999997},"min":{"1min":0.68400000000000005,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.997,"5min":10.571,"15min":10.571},"last":0.65300000000000002}]},{"osd":5,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.88900000000000001,"15min":0.89100000000000001},"min":{"1min":0.628,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.0289999999999999,"5min":3.4969999999999999,"15min":3.4969999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.876,"5min":0.84399999999999997,"15min":0.79200000000000004},"min":{"1min":0.72399999999999998,"5min":0.56000000000000005,"15min":0.49099999999999999},"max":{"1min":1.1739999999999999,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.66900000000000004}]},{"osd":6,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.80900000000000005},"min":{"1min":0.58399999999999996,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.1040000000000001,"5min":1.875,"15min":2.4710000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.88900000000000001,"15min":0.83399999999999996},"min":{"1min":0.57399999999999995,"5min":0.53100000000000003,"15min":0.51700000000000002},"max":{"1min":1.2130000000000001,"5min":5.3680000000000003,"15min":5.3680000000000003},"last":0.68600000000000005}]},{"osd":7,"last update":"Mon Oct 30 14:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":1.081,"15min":0.91600000000000004},"min":{"1min":0.59599999999999997,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0249999999999999,"5min":19.885000000000002,"15min":19.885000000000002},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.86499999999999999,"5min":1.054,"15min":0.93200000000000005},"min":{"1min":0.60099999999999998,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.127,"5min":13.401999999999999,"15min":13.401999999999999},"last":0.63}]}]},{"osd":0,"up_from":13,"seq":55834575311,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544720,"kb_used_data":423368,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":93827120,"statfs":{"total":96636764160,"available":96078970880,"internally_reserved":0,"allocated":433528832,"data_stored":421882906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124254662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45900000000000002,"15min":0.44800000000000001},"min":{"1min":0.32200000000000001,"5min":0.26800000000000002,"15min":0.22500000000000001},"max":{"1min":0.749,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.53300000000000003,"15min":0.503},"min":{"1min":0.36699999999999999,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":0.68999999999999995,"5min":3.367,"15min":3.367},"last":0.313}]},{"osd":2,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58199999999999996,"15min":0.53600000000000003},"min":{"1min":0.44400000000000001,"5min":0.378,"15min":0.30499999999999999},"max":{"1min":0.751,"5min":3.222,"15min":3.222},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53200000000000003,"15min":0.56000000000000005},"min":{"1min":0.40000000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":1.095},"last":0.51500000000000001}]},{"osd":3,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.56599999999999995,"15min":0.60199999999999998},"min":{"1min":0.34000000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.63300000000000001,"5min":6.4980000000000002,"15min":6.4980000000000002},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57499999999999996,"15min":0.52900000000000003},"min":{"1min":0.40400000000000003,"5min":0.40100000000000002,"15min":0.28000000000000003},"max":{"1min":0.71699999999999997,"5min":2.024,"15min":2.024},"last":0.55100000000000005}]},{"osd":4,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.78800000000000003,"15min":0.82599999999999996},"min":{"1min":0.57599999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.95399999999999996,"5min":2.0659999999999998,"15min":2.8380000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81399999999999995,"15min":0.85999999999999999},"min":{"1min":0.65100000000000002,"5min":0.56899999999999995,"15min":0.53900000000000003},"max":{"1min":0.97699999999999998,"5min":2.1000000000000001,"15min":5.8959999999999999},"last":0.59399999999999997}]},{"osd":5,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.872,"15min":0.86099999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.46300000000000002},"max":{"1min":1.212,"5min":1.278,"15min":6.4480000000000004},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.92000000000000004,"15min":0.85599999999999998},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.54000000000000004},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":2.6869999999999998},"last":0.90700000000000003}]},{"osd":6,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.83899999999999997,"15min":0.84699999999999998},"min":{"1min":0.64800000000000002,"5min":0.55300000000000005,"15min":0.54400000000000004},"max":{"1min":1.133,"5min":1.2150000000000001,"15min":4.9980000000000002},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.877,"5min":0.85499999999999998,"15min":0.86099999999999999},"min":{"1min":0.65300000000000002,"5min":0.55100000000000005,"15min":0.54000000000000004},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.5980000000000001},"last":0.78100000000000003}]},{"osd":7,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.86799999999999999,"15min":0.84699999999999998},"min":{"1min":0.63700000000000001,"5min":0.57899999999999996,"15min":0.499},"max":{"1min":1.1120000000000001,"5min":3.6589999999999998,"15min":3.6589999999999998},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.93100000000000005,"15min":0.88400000000000001},"min":{"1min":0.76300000000000001,"5min":0.625,"15min":0.54100000000000004},"max":{"1min":1.1579999999999999,"5min":1.262,"15min":2.6429999999999998},"last":0.89200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":109068288,"data_stored":109052564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163602432,"data_stored":163582366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":115884032,"data_stored":115864160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":209350656,"data_stored":209331981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":153219072,"data_stored":153204613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":215736320,"data_stored":215715446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":211091456,"data_stored":211074246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":201580544,"data_stored":201563796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323084288,"data_stored":311837872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":195256320,"data_stored":188856788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":201433088,"data_stored":195002091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310554624,"data_stored":299415921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":337338368,"data_stored":326229984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":335527936,"data_stored":324229915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":185139200,"data_stored":178241856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":297955328,"data_stored":287506221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:52:02.715 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.715 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.716 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.716 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.717 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.717 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.718 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.718 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.718 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.719 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.719 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.720 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.720 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.721 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.721 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.722 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.722 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.723 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.723 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.724 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.724 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.725 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.725 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.726 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.726 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.726 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.727 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.727 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.728 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.728 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.729 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.729 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.729 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.730 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.730 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.731 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.731 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.732 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.732 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.732 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.733 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.733 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.733 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.734 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.734 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.735 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.735 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.735 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.736 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.736 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.737 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.737 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2023-10-30T14:14:02.656444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=2, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.737 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.738 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:02.738 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T14:52:12.226 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:52:12.231 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:52:22.740 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:52:22.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.883+0000 7f87c1972640 1 Processor -- start 2023-10-30T14:52:22.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.883+0000 7f87c1972640 1 -- start start 2023-10-30T14:52:22.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.887+0000 7f87c1972640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f87bc153aa0 0x7f87bc15d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:22.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.887+0000 7f87c1972640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f87bc0701e0 0x7f87bc0ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:22.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.887+0000 7f87c1972640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87bc0cd420 0x7f87bc06b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:22.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.887+0000 7f87c1972640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87bc070a50 con 0x7f87bc0701e0 2023-10-30T14:52:22.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.887+0000 7f87c1972640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87bc15dd80 con 0x7f87bc153aa0 2023-10-30T14:52:22.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.887+0000 7f87c1972640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc0c7b00 con 0x7f87bc0cd420 2023-10-30T14:52:22.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.887+0000 7f87ba7fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f87bc0701e0 0x7f87bc0ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:22.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.887+0000 7f87ba7fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f87bc0701e0 0x7f87bc0ccee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39044/0 (socket says 172.21.15.57:39044) 2023-10-30T14:52:22.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.887+0000 7f87ba7fc640 1 -- 172.21.15.57:0/3376023188 learned_addr learned my addr 172.21.15.57:0/3376023188 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:52:22.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.887+0000 7f87baffd640 1 --2- 172.21.15.57:0/3376023188 >> v2:172.21.15.88:3300/0 conn(0x7f87bc153aa0 0x7f87bc15d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:22.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.887+0000 7f87bb7fe640 1 --2- 172.21.15.57:0/3376023188 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87bc0cd420 0x7f87bc06b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:22.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.887+0000 7f87ba7fc640 1 -- 172.21.15.57:0/3376023188 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87bc0cd420 msgr2=0x7f87bc06b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:52:22.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.887+0000 7f87ba7fc640 1 --2- 172.21.15.57:0/3376023188 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87bc0cd420 0x7f87bc06b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:22.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.887+0000 7f87ba7fc640 1 -- 172.21.15.57:0/3376023188 >> v2:172.21.15.88:3300/0 conn(0x7f87bc153aa0 msgr2=0x7f87bc15d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:52:22.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.887+0000 7f87ba7fc640 1 --2- 172.21.15.57:0/3376023188 >> v2:172.21.15.88:3300/0 conn(0x7f87bc153aa0 0x7f87bc15d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:22.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.887+0000 7f87ba7fc640 1 -- 172.21.15.57:0/3376023188 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87bc0c2050 con 0x7f87bc0701e0 2023-10-30T14:52:22.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.891+0000 7f87ba7fc640 1 --2- 172.21.15.57:0/3376023188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87bc0701e0 0x7f87bc0ccee0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f87bc0caae0 tx=0x7f87b001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f82d7e4504d19ff server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:22.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.891+0000 7f87b9ffb640 1 -- 172.21.15.57:0/3376023188 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b00a89b0 con 0x7f87bc0701e0 2023-10-30T14:52:22.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.891+0000 7f87b9ffb640 1 -- 172.21.15.57:0/3376023188 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b00a8b10 con 0x7f87bc0701e0 2023-10-30T14:52:22.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.891+0000 7f87b9ffb640 1 -- 172.21.15.57:0/3376023188 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b00a8ea0 con 0x7f87bc0701e0 2023-10-30T14:52:22.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.891+0000 7f87c1972640 1 -- 172.21.15.57:0/3376023188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87bc0701e0 msgr2=0x7f87bc0ccee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:22.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.891+0000 7f87c1972640 1 --2- 172.21.15.57:0/3376023188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87bc0701e0 0x7f87bc0ccee0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f87bc0caae0 tx=0x7f87b001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:52:22.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.891+0000 7f87c1972640 1 -- 172.21.15.57:0/3376023188 shutdown_connections 2023-10-30T14:52:22.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.891+0000 7f87c1972640 1 --2- 172.21.15.57:0/3376023188 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87bc0cd420 0x7f87bc06b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:22.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.891+0000 7f87c1972640 1 --2- 172.21.15.57:0/3376023188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87bc0701e0 0x7f87bc0ccee0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:22.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.891+0000 7f87c1972640 1 --2- 172.21.15.57:0/3376023188 >> v2:172.21.15.88:3300/0 conn(0x7f87bc153aa0 0x7f87bc15d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:22.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.891+0000 7f87c1972640 1 -- 172.21.15.57:0/3376023188 >> 172.21.15.57:0/3376023188 conn(0x7f87bc071fb0 msgr2=0x7f87bc0723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:22.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.891+0000 7f87c1972640 1 -- 172.21.15.57:0/3376023188 shutdown_connections 2023-10-30T14:52:22.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.891+0000 7f87c1972640 1 -- 172.21.15.57:0/3376023188 wait complete. 2023-10-30T14:52:22.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.891+0000 7f87c1972640 1 Processor -- start 2023-10-30T14:52:22.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.891+0000 7f87c1972640 1 -- start start 2023-10-30T14:52:22.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87c1972640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87bc0701e0 0x7f87bc15aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:22.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87c1972640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87bc0cd420 0x7f87bc0c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:22.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87c1972640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87bc153aa0 0x7f87bc149960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:22.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87c1972640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc157dc0 con 0x7f87bc153aa0 2023-10-30T14:52:22.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87c1972640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc1598c0 con 0x7f87bc0cd420 2023-10-30T14:52:22.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87c1972640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc15d110 con 0x7f87bc0701e0 2023-10-30T14:52:22.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87bb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87bc153aa0 0x7f87bc149960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:22.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87bb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87bc153aa0 0x7f87bc149960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39050/0 (socket says 172.21.15.57:39050) 2023-10-30T14:52:22.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87bb7fe640 1 -- 172.21.15.57:0/1488582336 learned_addr learned my addr 172.21.15.57:0/1488582336 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:52:22.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87ba7fc640 1 --2- 172.21.15.57:0/1488582336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87bc0cd420 0x7f87bc0c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:22.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87baffd640 1 --2- 172.21.15.57:0/1488582336 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87bc0701e0 0x7f87bc15aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:22.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87bb7fe640 1 -- 172.21.15.57:0/1488582336 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87bc0701e0 msgr2=0x7f87bc15aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:22.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87bb7fe640 1 --2- 172.21.15.57:0/1488582336 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87bc0701e0 0x7f87bc15aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:22.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87bb7fe640 1 -- 172.21.15.57:0/1488582336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87bc0cd420 msgr2=0x7f87bc0c8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:22.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87bb7fe640 1 --2- 172.21.15.57:0/1488582336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87bc0cd420 0x7f87bc0c8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:22.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87bb7fe640 1 -- 172.21.15.57:0/1488582336 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b00094d0 con 0x7f87bc153aa0 2023-10-30T14:52:22.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87baffd640 1 --2- 172.21.15.57:0/1488582336 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87bc0701e0 0x7f87bc15aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:52:22.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87bb7fe640 1 --2- 172.21.15.57:0/1488582336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87bc153aa0 0x7f87bc149960 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f87b400eed0 tx=0x7f87b400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:22.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87a3fff640 1 -- 172.21.15.57:0/1488582336 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b4010040 con 0x7f87bc153aa0 2023-10-30T14:52:22.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87a3fff640 1 -- 172.21.15.57:0/1488582336 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b4014400 con 0x7f87bc153aa0 2023-10-30T14:52:22.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87a3fff640 1 -- 172.21.15.57:0/1488582336 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b4009ea0 con 0x7f87bc153aa0 2023-10-30T14:52:22.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87c1972640 1 -- 172.21.15.57:0/1488582336 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87bc0c7930 con 0x7f87bc153aa0 2023-10-30T14:52:22.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87c1972640 1 -- 172.21.15.57:0/1488582336 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87bc157ae0 con 0x7f87bc153aa0 2023-10-30T14:52:22.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87c1972640 1 -- 172.21.15.57:0/1488582336 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87bc04e170 con 0x7f87bc153aa0 2023-10-30T14:52:22.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.895+0000 7f87a3fff640 1 -- 172.21.15.57:0/1488582336 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f87b40147b0 con 0x7f87bc153aa0 2023-10-30T14:52:22.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.899+0000 7f87a3fff640 1 --2- 172.21.15.57:0/1488582336 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f87a80664a0 0x7f87a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:22.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.899+0000 7f87a3fff640 1 -- 172.21.15.57:0/1488582336 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f87b4084be0 con 0x7f87bc153aa0 2023-10-30T14:52:22.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.899+0000 7f87baffd640 1 --2- 172.21.15.57:0/1488582336 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f87a80664a0 0x7f87a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:22.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.899+0000 7f87baffd640 1 --2- 172.21.15.57:0/1488582336 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f87a80664a0 0x7f87a8068960 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f87ac009b10 tx=0x7f87ac008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:22.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:22.903+0000 7f87a3fff640 1 -- 172.21.15.57:0/1488582336 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f87b40519d0 con 0x7f87bc153aa0 2023-10-30T14:52:23.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:23.055+0000 7f87c1972640 1 -- 172.21.15.57:0/1488582336 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f87bc0c2330 con 0x7f87a80664a0 2023-10-30T14:52:23.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:23.075+0000 7f87a3fff640 1 -- 172.21.15.57:0/1488582336 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+295797 (secure 0 0 0) 0x7f87ac004030 con 0x7f87a80664a0 2023-10-30T14:52:23.082 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:52:23.084 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-30T14:52:23.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:23.079+0000 7f87c1972640 1 -- 172.21.15.57:0/1488582336 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f87a80664a0 msgr2=0x7f87a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:23.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:23.079+0000 7f87c1972640 1 --2- 172.21.15.57:0/1488582336 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f87a80664a0 0x7f87a8068960 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f87ac009b10 tx=0x7f87ac008040 comp rx=0 tx=0).stop 2023-10-30T14:52:23.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:23.079+0000 7f87c1972640 1 -- 172.21.15.57:0/1488582336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87bc153aa0 msgr2=0x7f87bc149960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:23.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:23.079+0000 7f87c1972640 1 --2- 172.21.15.57:0/1488582336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87bc153aa0 0x7f87bc149960 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f87b400eed0 tx=0x7f87b400b480 comp rx=0 tx=0).stop 2023-10-30T14:52:23.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:23.079+0000 7f87c1972640 1 -- 172.21.15.57:0/1488582336 shutdown_connections 2023-10-30T14:52:23.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:23.079+0000 7f87c1972640 1 --2- 172.21.15.57:0/1488582336 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f87a80664a0 0x7f87a8068960 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:23.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:23.079+0000 7f87c1972640 1 --2- 172.21.15.57:0/1488582336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87bc153aa0 0x7f87bc149960 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:23.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:23.079+0000 7f87c1972640 1 --2- 172.21.15.57:0/1488582336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87bc0cd420 0x7f87bc0c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:23.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:23.079+0000 7f87c1972640 1 --2- 172.21.15.57:0/1488582336 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87bc0701e0 0x7f87bc15aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:23.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:23.079+0000 7f87c1972640 1 -- 172.21.15.57:0/1488582336 >> 172.21.15.57:0/1488582336 conn(0x7f87bc071fb0 msgr2=0x7f87bc0ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:23.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:23.079+0000 7f87c1972640 1 -- 172.21.15.57:0/1488582336 shutdown_connections 2023-10-30T14:52:23.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:23.079+0000 7f87c1972640 1 -- 172.21.15.57:0/1488582336 wait complete. 2023-10-30T14:52:23.102 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":1215,"stamp":"2023-10-30T14:52:21.332639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1745330878,"num_objects":33081,"num_object_clones":14708,"num_object_copies":66162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33081,"num_whiteouts":12823,"num_read":10750,"num_read_kb":3497830,"num_write":98408,"num_write_kb":2875712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1128,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16321744,"num_omap_keys":27609,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137534,"ondisk_log_size":137534,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":4600496,"kb_used_data":3489520,"kb_used_omap":12,"kb_used_meta":1110899,"kb_avail":750374224,"statfs":{"total":773094113280,"available":768383205376,"internally_reserved":0,"allocated":3573268480,"data_stored":3495107372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":1137561145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1220237,"num_omap_keys":2121,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004036"},"pg_stats":[{"pgid":"3.3f","version":"67'1292","reported_seq":3255,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:37.470671+0000","last_change":"2023-10-30T14:23:31.082534+0000","last_active":"2023-10-30T14:51:37.470671+0000","last_peered":"2023-10-30T14:51:37.470671+0000","last_clean":"2023-10-30T14:51:37.470671+0000","last_became_active":"2023-10-30T14:14:04.650882+0000","last_became_peered":"2023-10-30T14:14:04.650882+0000","last_unstale":"2023-10-30T14:51:37.470671+0000","last_undegraded":"2023-10-30T14:51:37.470671+0000","last_fullsized":"2023-10-30T14:51:37.470671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1292,"log_dups_size":0,"ondisk_log_size":1292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010330941,"stat_sum":{"num_bytes":16885684,"num_objects":463,"num_object_clones":217,"num_object_copies":926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":463,"num_whiteouts":185,"num_read":62,"num_read_kb":8246,"num_write":1079,"num_write_kb":18499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"67'1769","reported_seq":3752,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:12.785605+0000","last_change":"2023-10-30T14:52:12.785667+0000","last_active":"2023-10-30T14:52:12.785605+0000","last_peered":"2023-10-30T14:52:12.785605+0000","last_clean":"2023-10-30T14:52:12.785605+0000","last_became_active":"2023-10-30T14:14:04.587942+0000","last_became_peered":"2023-10-30T14:14:04.587942+0000","last_unstale":"2023-10-30T14:52:12.785605+0000","last_undegraded":"2023-10-30T14:52:12.785605+0000","last_fullsized":"2023-10-30T14:52:12.785605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1769","last_scrub_stamp":"2023-10-30T14:52:12.785532+0000","last_deep_scrub":"67'1769","last_deep_scrub_stamp":"2023-10-30T14:52:12.785532+0000","last_clean_scrub_stamp":"2023-10-30T14:52:12.785532+0000","objects_scrubbed":485,"log_size":1769,"log_dups_size":0,"ondisk_log_size":1769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:48:56.925914+0000","scrub_duration":0.54729001300000002,"objects_trimmed":0,"snaptrim_duration":0.002850184,"stat_sum":{"num_bytes":13093874,"num_objects":485,"num_object_clones":221,"num_object_copies":970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":193,"num_read":59,"num_read_kb":8220,"num_write":1078,"num_write_kb":14962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"67'1798","reported_seq":3790,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:10.917457+0000","last_change":"2023-10-30T14:52:10.917502+0000","last_active":"2023-10-30T14:52:10.917457+0000","last_peered":"2023-10-30T14:52:10.917457+0000","last_clean":"2023-10-30T14:52:10.917457+0000","last_became_active":"2023-10-30T14:14:04.590700+0000","last_became_peered":"2023-10-30T14:14:04.590700+0000","last_unstale":"2023-10-30T14:52:10.917457+0000","last_undegraded":"2023-10-30T14:52:10.917457+0000","last_fullsized":"2023-10-30T14:52:10.917457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1798","last_scrub_stamp":"2023-10-30T14:52:10.917377+0000","last_deep_scrub":"67'1798","last_deep_scrub_stamp":"2023-10-30T14:52:10.917377+0000","last_clean_scrub_stamp":"2023-10-30T14:52:10.917377+0000","objects_scrubbed":491,"log_size":1798,"log_dups_size":0,"ondisk_log_size":1798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:07:29.339766+0000","scrub_duration":0.60249001999999996,"objects_trimmed":0,"snaptrim_duration":0.0042861139999999997,"stat_sum":{"num_bytes":12610897,"num_objects":491,"num_object_clones":227,"num_object_copies":982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":203,"num_read":73,"num_read_kb":16409,"num_write":1067,"num_write_kb":22118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"67'1784","reported_seq":3772,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:00.217825+0000","last_change":"2023-10-30T14:52:00.217920+0000","last_active":"2023-10-30T14:52:00.217825+0000","last_peered":"2023-10-30T14:52:00.217825+0000","last_clean":"2023-10-30T14:52:00.217825+0000","last_became_active":"2023-10-30T14:14:45.661677+0000","last_became_peered":"2023-10-30T14:14:45.661677+0000","last_unstale":"2023-10-30T14:52:00.217825+0000","last_undegraded":"2023-10-30T14:52:00.217825+0000","last_fullsized":"2023-10-30T14:52:00.217825+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1784","last_scrub_stamp":"2023-10-30T14:52:00.217714+0000","last_deep_scrub":"67'1784","last_deep_scrub_stamp":"2023-10-30T14:52:00.217714+0000","last_clean_scrub_stamp":"2023-10-30T14:52:00.217714+0000","objects_scrubbed":483,"log_size":1784,"log_dups_size":0,"ondisk_log_size":1784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:15:45.674235+0000","scrub_duration":0.72428234999999996,"objects_trimmed":0,"snaptrim_duration":0.003131572,"stat_sum":{"num_bytes":8809488,"num_objects":483,"num_object_clones":217,"num_object_copies":966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":483,"num_whiteouts":190,"num_read":63,"num_read_kb":69,"num_write":1091,"num_write_kb":15023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"67'1773","reported_seq":3761,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:46.289749+0000","last_change":"2023-10-30T14:51:46.289749+0000","last_active":"2023-10-30T14:51:46.289749+0000","last_peered":"2023-10-30T14:51:46.289749+0000","last_clean":"2023-10-30T14:51:46.289749+0000","last_became_active":"2023-10-30T14:14:04.644683+0000","last_became_peered":"2023-10-30T14:14:04.644683+0000","last_unstale":"2023-10-30T14:51:46.289749+0000","last_undegraded":"2023-10-30T14:51:46.289749+0000","last_fullsized":"2023-10-30T14:51:46.289749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1772","last_scrub_stamp":"2023-10-30T14:51:46.289574+0000","last_deep_scrub":"67'1772","last_deep_scrub_stamp":"2023-10-30T14:51:46.289574+0000","last_clean_scrub_stamp":"2023-10-30T14:51:46.289574+0000","objects_scrubbed":486,"log_size":1773,"log_dups_size":0,"ondisk_log_size":1773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:44:45.481060+0000","scrub_duration":0.82105794700000001,"objects_trimmed":0,"snaptrim_duration":0.0052462430000000003,"stat_sum":{"num_bytes":8863750,"num_objects":486,"num_object_clones":224,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":199,"num_read":70,"num_read_kb":4116,"num_write":1086,"num_write_kb":10111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'1770","reported_seq":3766,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:16.578543+0000","last_change":"2023-10-30T14:52:16.578648+0000","last_active":"2023-10-30T14:52:16.578543+0000","last_peered":"2023-10-30T14:52:16.578543+0000","last_clean":"2023-10-30T14:52:16.578543+0000","last_became_active":"2023-10-30T14:14:04.619335+0000","last_became_peered":"2023-10-30T14:14:04.619335+0000","last_unstale":"2023-10-30T14:52:16.578543+0000","last_undegraded":"2023-10-30T14:52:16.578543+0000","last_fullsized":"2023-10-30T14:52:16.578543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1770","last_scrub_stamp":"2023-10-30T14:52:16.578428+0000","last_deep_scrub":"67'1770","last_deep_scrub_stamp":"2023-10-30T14:52:16.578428+0000","last_clean_scrub_stamp":"2023-10-30T14:52:16.578428+0000","objects_scrubbed":479,"log_size":1770,"log_dups_size":0,"ondisk_log_size":1770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:43:02.459839+0000","scrub_duration":0.74945407600000002,"objects_trimmed":0,"snaptrim_duration":0.0062283859999999998,"stat_sum":{"num_bytes":20638046,"num_objects":479,"num_object_clones":221,"num_object_copies":958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":195,"num_read":75,"num_read_kb":12254,"num_write":1080,"num_write_kb":22208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"67'1792","reported_seq":3770,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:16.074406+0000","last_change":"2023-10-30T14:52:16.074486+0000","last_active":"2023-10-30T14:52:16.074406+0000","last_peered":"2023-10-30T14:52:16.074406+0000","last_clean":"2023-10-30T14:52:16.074406+0000","last_became_active":"2023-10-30T14:14:04.641323+0000","last_became_peered":"2023-10-30T14:14:04.641323+0000","last_unstale":"2023-10-30T14:52:16.074406+0000","last_undegraded":"2023-10-30T14:52:16.074406+0000","last_fullsized":"2023-10-30T14:52:16.074406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1792","last_scrub_stamp":"2023-10-30T14:52:16.074315+0000","last_deep_scrub":"67'1792","last_deep_scrub_stamp":"2023-10-30T14:52:16.074315+0000","last_clean_scrub_stamp":"2023-10-30T14:52:16.074315+0000","objects_scrubbed":493,"log_size":1792,"log_dups_size":0,"ondisk_log_size":1792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:12:43.677442+0000","scrub_duration":0.89714960899999996,"objects_trimmed":0,"snaptrim_duration":0.011228897,"stat_sum":{"num_bytes":14710085,"num_objects":493,"num_object_clones":224,"num_object_copies":986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":493,"num_whiteouts":202,"num_read":59,"num_read_kb":8203,"num_write":1083,"num_write_kb":18203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'1702","reported_seq":3686,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:54.076987+0000","last_change":"2023-10-30T14:51:54.076987+0000","last_active":"2023-10-30T14:51:54.076987+0000","last_peered":"2023-10-30T14:51:54.076987+0000","last_clean":"2023-10-30T14:51:54.076987+0000","last_became_active":"2023-10-30T14:14:04.618312+0000","last_became_peered":"2023-10-30T14:14:04.618312+0000","last_unstale":"2023-10-30T14:51:54.076987+0000","last_undegraded":"2023-10-30T14:51:54.076987+0000","last_fullsized":"2023-10-30T14:51:54.076987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1701","last_scrub_stamp":"2023-10-30T14:51:54.076855+0000","last_deep_scrub":"67'1701","last_deep_scrub_stamp":"2023-10-30T14:51:54.076855+0000","last_clean_scrub_stamp":"2023-10-30T14:51:54.076855+0000","objects_scrubbed":477,"log_size":1702,"log_dups_size":0,"ondisk_log_size":1702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:59:06.379444+0000","scrub_duration":0.66338523199999999,"objects_trimmed":0,"snaptrim_duration":0.002849426,"stat_sum":{"num_bytes":12938947,"num_objects":477,"num_object_clones":218,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":193,"num_read":60,"num_read_kb":8209,"num_write":1033,"num_write_kb":15209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"67'1347","reported_seq":3295,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.640833+0000","last_change":"2023-10-30T14:23:31.083143+0000","last_active":"2023-10-30T14:51:34.640833+0000","last_peered":"2023-10-30T14:51:34.640833+0000","last_clean":"2023-10-30T14:51:34.640833+0000","last_became_active":"2023-10-30T14:14:04.647361+0000","last_became_peered":"2023-10-30T14:14:04.647361+0000","last_unstale":"2023-10-30T14:51:34.640833+0000","last_undegraded":"2023-10-30T14:51:34.640833+0000","last_fullsized":"2023-10-30T14:51:34.640833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1347,"log_dups_size":0,"ondisk_log_size":1347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010459703000000001,"stat_sum":{"num_bytes":8547707,"num_objects":513,"num_object_clones":237,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":513,"num_whiteouts":209,"num_read":70,"num_read_kb":8267,"num_write":1124,"num_write_kb":13960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"67'1927","reported_seq":3859,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:05.832025+0000","last_change":"2023-10-30T14:52:05.832091+0000","last_active":"2023-10-30T14:52:05.832025+0000","last_peered":"2023-10-30T14:52:05.832025+0000","last_clean":"2023-10-30T14:52:05.832025+0000","last_became_active":"2023-10-30T14:14:04.630959+0000","last_became_peered":"2023-10-30T14:14:04.630959+0000","last_unstale":"2023-10-30T14:52:05.832025+0000","last_undegraded":"2023-10-30T14:52:05.832025+0000","last_fullsized":"2023-10-30T14:52:05.832025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1927","last_scrub_stamp":"2023-10-30T14:52:05.831888+0000","last_deep_scrub":"67'1927","last_deep_scrub_stamp":"2023-10-30T14:52:05.831888+0000","last_clean_scrub_stamp":"2023-10-30T14:52:05.831888+0000","objects_scrubbed":545,"log_size":1927,"log_dups_size":0,"ondisk_log_size":1927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:33:14.719732+0000","scrub_duration":1.5552255019999999,"objects_trimmed":0,"snaptrim_duration":0.0079325130000000004,"stat_sum":{"num_bytes":12417236,"num_objects":545,"num_object_clones":289,"num_object_copies":1090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":545,"num_whiteouts":193,"num_read":76,"num_read_kb":4189,"num_write":1098,"num_write_kb":14000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"67'1825","reported_seq":3797,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:12.042783+0000","last_change":"2023-10-30T14:52:12.042829+0000","last_active":"2023-10-30T14:52:12.042783+0000","last_peered":"2023-10-30T14:52:12.042783+0000","last_clean":"2023-10-30T14:52:12.042783+0000","last_became_active":"2023-10-30T14:14:04.612653+0000","last_became_peered":"2023-10-30T14:14:04.612653+0000","last_unstale":"2023-10-30T14:52:12.042783+0000","last_undegraded":"2023-10-30T14:52:12.042783+0000","last_fullsized":"2023-10-30T14:52:12.042783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1825","last_scrub_stamp":"2023-10-30T14:52:12.042703+0000","last_deep_scrub":"67'1825","last_deep_scrub_stamp":"2023-10-30T14:52:12.042703+0000","last_clean_scrub_stamp":"2023-10-30T14:52:12.042703+0000","objects_scrubbed":522,"log_size":1825,"log_dups_size":0,"ondisk_log_size":1825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:54:08.506135+0000","scrub_duration":0.74874808599999998,"objects_trimmed":0,"snaptrim_duration":0.0079098320000000003,"stat_sum":{"num_bytes":17851951,"num_objects":522,"num_object_clones":235,"num_object_copies":1044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":202,"num_read":63,"num_read_kb":9809,"num_write":1084,"num_write_kb":19222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"67'1788","reported_seq":3771,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:59.430627+0000","last_change":"2023-10-30T14:51:58.995889+0000","last_active":"2023-10-30T14:51:59.430627+0000","last_peered":"2023-10-30T14:51:59.430627+0000","last_clean":"2023-10-30T14:51:59.430627+0000","last_became_active":"2023-10-30T14:14:04.621135+0000","last_became_peered":"2023-10-30T14:14:04.621135+0000","last_unstale":"2023-10-30T14:51:59.430627+0000","last_undegraded":"2023-10-30T14:51:59.430627+0000","last_fullsized":"2023-10-30T14:51:59.430627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1785","last_scrub_stamp":"2023-10-30T14:51:58.995736+0000","last_deep_scrub":"67'1785","last_deep_scrub_stamp":"2023-10-30T14:51:58.995736+0000","last_clean_scrub_stamp":"2023-10-30T14:51:58.995736+0000","objects_scrubbed":506,"log_size":1788,"log_dups_size":0,"ondisk_log_size":1788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:30:07.331904+0000","scrub_duration":0.92494544599999995,"objects_trimmed":0,"snaptrim_duration":0.0025042409999999999,"stat_sum":{"num_bytes":14360375,"num_objects":506,"num_object_clones":227,"num_object_copies":1012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":193,"num_read":67,"num_read_kb":6339,"num_write":1088,"num_write_kb":20012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"67'1759","reported_seq":3769,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:03.476954+0000","last_change":"2023-10-30T14:52:03.477051+0000","last_active":"2023-10-30T14:52:03.476954+0000","last_peered":"2023-10-30T14:52:03.476954+0000","last_clean":"2023-10-30T14:52:03.476954+0000","last_became_active":"2023-10-30T14:14:04.616142+0000","last_became_peered":"2023-10-30T14:14:04.616142+0000","last_unstale":"2023-10-30T14:52:03.476954+0000","last_undegraded":"2023-10-30T14:52:03.476954+0000","last_fullsized":"2023-10-30T14:52:03.476954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1759","last_scrub_stamp":"2023-10-30T14:52:03.476870+0000","last_deep_scrub":"67'1759","last_deep_scrub_stamp":"2023-10-30T14:52:03.476870+0000","last_clean_scrub_stamp":"2023-10-30T14:52:03.476870+0000","objects_scrubbed":486,"log_size":1759,"log_dups_size":0,"ondisk_log_size":1759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:32:02.837333+0000","scrub_duration":1.022971694,"objects_trimmed":0,"snaptrim_duration":0.0049623779999999999,"stat_sum":{"num_bytes":9254692,"num_objects":486,"num_object_clones":220,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":190,"num_read":88,"num_read_kb":8286,"num_write":1073,"num_write_kb":19495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"67'1790","reported_seq":3792,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:15.904032+0000","last_change":"2023-10-30T14:52:15.904089+0000","last_active":"2023-10-30T14:52:15.904032+0000","last_peered":"2023-10-30T14:52:15.904032+0000","last_clean":"2023-10-30T14:52:15.904032+0000","last_became_active":"2023-10-30T14:14:04.588452+0000","last_became_peered":"2023-10-30T14:14:04.588452+0000","last_unstale":"2023-10-30T14:52:15.904032+0000","last_undegraded":"2023-10-30T14:52:15.904032+0000","last_fullsized":"2023-10-30T14:52:15.904032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1790","last_scrub_stamp":"2023-10-30T14:52:15.903959+0000","last_deep_scrub":"67'1790","last_deep_scrub_stamp":"2023-10-30T14:52:15.903959+0000","last_clean_scrub_stamp":"2023-10-30T14:52:15.903959+0000","objects_scrubbed":471,"log_size":1790,"log_dups_size":0,"ondisk_log_size":1790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:36:49.496291+0000","scrub_duration":0.67904057500000004,"objects_trimmed":0,"snaptrim_duration":0.0046351760000000004,"stat_sum":{"num_bytes":12959775,"num_objects":471,"num_object_clones":217,"num_object_copies":942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":471,"num_whiteouts":194,"num_read":76,"num_read_kb":7792,"num_write":1107,"num_write_kb":17259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"67'1812","reported_seq":3804,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:05.304605+0000","last_change":"2023-10-30T14:52:05.304672+0000","last_active":"2023-10-30T14:52:05.304605+0000","last_peered":"2023-10-30T14:52:05.304605+0000","last_clean":"2023-10-30T14:52:05.304605+0000","last_became_active":"2023-10-30T14:14:04.620781+0000","last_became_peered":"2023-10-30T14:14:04.620781+0000","last_unstale":"2023-10-30T14:52:05.304605+0000","last_undegraded":"2023-10-30T14:52:05.304605+0000","last_fullsized":"2023-10-30T14:52:05.304605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1812","last_scrub_stamp":"2023-10-30T14:52:05.304527+0000","last_deep_scrub":"67'1812","last_deep_scrub_stamp":"2023-10-30T14:52:05.304527+0000","last_clean_scrub_stamp":"2023-10-30T14:52:05.304527+0000","objects_scrubbed":489,"log_size":1812,"log_dups_size":0,"ondisk_log_size":1812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:00:19.753698+0000","scrub_duration":0.98796119900000001,"objects_trimmed":0,"snaptrim_duration":0.002316706,"stat_sum":{"num_bytes":21445753,"num_objects":489,"num_object_clones":220,"num_object_copies":978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":196,"num_read":69,"num_read_kb":13296,"num_write":1109,"num_write_kb":27203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"67'1773","reported_seq":3783,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:52.488244+0000","last_change":"2023-10-30T14:51:52.488306+0000","last_active":"2023-10-30T14:51:52.488244+0000","last_peered":"2023-10-30T14:51:52.488244+0000","last_clean":"2023-10-30T14:51:52.488244+0000","last_became_active":"2023-10-30T14:14:04.608190+0000","last_became_peered":"2023-10-30T14:14:04.608190+0000","last_unstale":"2023-10-30T14:51:52.488244+0000","last_undegraded":"2023-10-30T14:51:52.488244+0000","last_fullsized":"2023-10-30T14:51:52.488244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1773","last_scrub_stamp":"2023-10-30T14:51:52.488153+0000","last_deep_scrub":"67'1773","last_deep_scrub_stamp":"2023-10-30T14:51:52.488153+0000","last_clean_scrub_stamp":"2023-10-30T14:51:52.488153+0000","objects_scrubbed":484,"log_size":1773,"log_dups_size":0,"ondisk_log_size":1773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:40.330537+0000","scrub_duration":0.90559973500000002,"objects_trimmed":0,"snaptrim_duration":0.0035618450000000001,"stat_sum":{"num_bytes":31901819,"num_objects":484,"num_object_clones":219,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":197,"num_read":85,"num_read_kb":23305,"num_write":1093,"num_write_kb":37349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"67'1874","reported_seq":3863,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:07.522300+0000","last_change":"2023-10-30T14:52:07.522383+0000","last_active":"2023-10-30T14:52:07.522300+0000","last_peered":"2023-10-30T14:52:07.522300+0000","last_clean":"2023-10-30T14:52:07.522300+0000","last_became_active":"2023-10-30T14:14:04.641306+0000","last_became_peered":"2023-10-30T14:14:04.641306+0000","last_unstale":"2023-10-30T14:52:07.522300+0000","last_undegraded":"2023-10-30T14:52:07.522300+0000","last_fullsized":"2023-10-30T14:52:07.522300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1874","last_scrub_stamp":"2023-10-30T14:52:07.522201+0000","last_deep_scrub":"67'1874","last_deep_scrub_stamp":"2023-10-30T14:52:07.522201+0000","last_clean_scrub_stamp":"2023-10-30T14:52:07.522201+0000","objects_scrubbed":513,"log_size":1874,"log_dups_size":0,"ondisk_log_size":1874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:12.283985+0000","scrub_duration":0.80925856399999996,"objects_trimmed":0,"snaptrim_duration":0.0066189990000000004,"stat_sum":{"num_bytes":4095483,"num_objects":513,"num_object_clones":236,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":513,"num_whiteouts":206,"num_read":80,"num_read_kb":4151,"num_write":1131,"num_write_kb":9615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"67'1656","reported_seq":3664,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:52.398678+0000","last_change":"2023-10-30T14:51:52.398752+0000","last_active":"2023-10-30T14:51:52.398678+0000","last_peered":"2023-10-30T14:51:52.398678+0000","last_clean":"2023-10-30T14:51:52.398678+0000","last_became_active":"2023-10-30T14:14:04.613644+0000","last_became_peered":"2023-10-30T14:14:04.613644+0000","last_unstale":"2023-10-30T14:51:52.398678+0000","last_undegraded":"2023-10-30T14:51:52.398678+0000","last_fullsized":"2023-10-30T14:51:52.398678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1656","last_scrub_stamp":"2023-10-30T14:51:52.398591+0000","last_deep_scrub":"67'1656","last_deep_scrub_stamp":"2023-10-30T14:51:52.398591+0000","last_clean_scrub_stamp":"2023-10-30T14:51:52.398591+0000","objects_scrubbed":445,"log_size":1656,"log_dups_size":0,"ondisk_log_size":1656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:58.033004+0000","scrub_duration":0.76830485800000003,"objects_trimmed":0,"snaptrim_duration":0.0030699740000000001,"stat_sum":{"num_bytes":12521692,"num_objects":445,"num_object_clones":206,"num_object_copies":890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":445,"num_whiteouts":180,"num_read":71,"num_read_kb":8233,"num_write":1044,"num_write_kb":14252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"67'1304","reported_seq":3276,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:41.714303+0000","last_change":"2023-10-30T14:23:31.082413+0000","last_active":"2023-10-30T14:51:41.714303+0000","last_peered":"2023-10-30T14:51:41.714303+0000","last_clean":"2023-10-30T14:51:41.714303+0000","last_became_active":"2023-10-30T14:14:04.649850+0000","last_became_peered":"2023-10-30T14:14:04.649850+0000","last_unstale":"2023-10-30T14:51:41.714303+0000","last_undegraded":"2023-10-30T14:51:41.714303+0000","last_fullsized":"2023-10-30T14:51:41.714303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1304,"log_dups_size":0,"ondisk_log_size":1304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012599246,"stat_sum":{"num_bytes":26075472,"num_objects":477,"num_object_clones":221,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":192,"num_read":77,"num_read_kb":24658,"num_write":1082,"num_write_kb":32086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"67'1861","reported_seq":3824,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:04.219517+0000","last_change":"2023-10-30T14:52:04.219630+0000","last_active":"2023-10-30T14:52:04.219517+0000","last_peered":"2023-10-30T14:52:04.219517+0000","last_clean":"2023-10-30T14:52:04.219517+0000","last_became_active":"2023-10-30T14:14:04.625042+0000","last_became_peered":"2023-10-30T14:14:04.625042+0000","last_unstale":"2023-10-30T14:52:04.219517+0000","last_undegraded":"2023-10-30T14:52:04.219517+0000","last_fullsized":"2023-10-30T14:52:04.219517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1861","last_scrub_stamp":"2023-10-30T14:52:04.219419+0000","last_deep_scrub":"67'1861","last_deep_scrub_stamp":"2023-10-30T14:52:04.219419+0000","last_clean_scrub_stamp":"2023-10-30T14:52:04.219419+0000","objects_scrubbed":514,"log_size":1861,"log_dups_size":0,"ondisk_log_size":1861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:43:26.153599+0000","scrub_duration":0.918695398,"objects_trimmed":0,"snaptrim_duration":0.0082351539999999997,"stat_sum":{"num_bytes":12784426,"num_objects":514,"num_object_clones":242,"num_object_copies":1028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":514,"num_whiteouts":214,"num_read":61,"num_read_kb":10278,"num_write":1107,"num_write_kb":27149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"67'1278","reported_seq":3243,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.519419+0000","last_change":"2023-10-30T14:23:30.775419+0000","last_active":"2023-10-30T14:51:34.519419+0000","last_peered":"2023-10-30T14:51:34.519419+0000","last_clean":"2023-10-30T14:51:34.519419+0000","last_became_active":"2023-10-30T14:14:04.633586+0000","last_became_peered":"2023-10-30T14:14:04.633586+0000","last_unstale":"2023-10-30T14:51:34.519419+0000","last_undegraded":"2023-10-30T14:51:34.519419+0000","last_fullsized":"2023-10-30T14:51:34.519419+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1278,"log_dups_size":0,"ondisk_log_size":1278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043616920000000003,"stat_sum":{"num_bytes":12342414,"num_objects":462,"num_object_clones":214,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":462,"num_whiteouts":191,"num_read":63,"num_read_kb":8213,"num_write":1061,"num_write_kb":14018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"67'1897","reported_seq":3898,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:15.835635+0000","last_change":"2023-10-30T14:52:15.835702+0000","last_active":"2023-10-30T14:52:15.835635+0000","last_peered":"2023-10-30T14:52:15.835635+0000","last_clean":"2023-10-30T14:52:15.835635+0000","last_became_active":"2023-10-30T14:14:04.642244+0000","last_became_peered":"2023-10-30T14:14:04.642244+0000","last_unstale":"2023-10-30T14:52:15.835635+0000","last_undegraded":"2023-10-30T14:52:15.835635+0000","last_fullsized":"2023-10-30T14:52:15.835635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1897","last_scrub_stamp":"2023-10-30T14:52:15.835546+0000","last_deep_scrub":"67'1897","last_deep_scrub_stamp":"2023-10-30T14:52:15.835546+0000","last_clean_scrub_stamp":"2023-10-30T14:52:15.835546+0000","objects_scrubbed":515,"log_size":1897,"log_dups_size":0,"ondisk_log_size":1897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:22:58.232388+0000","scrub_duration":1.035771956,"objects_trimmed":0,"snaptrim_duration":0.0065088710000000003,"stat_sum":{"num_bytes":26075380,"num_objects":515,"num_object_clones":234,"num_object_copies":1030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":515,"num_whiteouts":210,"num_read":90,"num_read_kb":18405,"num_write":1137,"num_write_kb":31429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'1775","reported_seq":3784,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:55.288911+0000","last_change":"2023-10-30T14:51:55.288911+0000","last_active":"2023-10-30T14:51:55.288911+0000","last_peered":"2023-10-30T14:51:55.288911+0000","last_clean":"2023-10-30T14:51:55.288911+0000","last_became_active":"2023-10-30T14:14:04.620343+0000","last_became_peered":"2023-10-30T14:14:04.620343+0000","last_unstale":"2023-10-30T14:51:55.288911+0000","last_undegraded":"2023-10-30T14:51:55.288911+0000","last_fullsized":"2023-10-30T14:51:55.288911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1774","last_scrub_stamp":"2023-10-30T14:51:55.288738+0000","last_deep_scrub":"67'1774","last_deep_scrub_stamp":"2023-10-30T14:51:55.288738+0000","last_clean_scrub_stamp":"2023-10-30T14:51:55.288738+0000","objects_scrubbed":484,"log_size":1775,"log_dups_size":0,"ondisk_log_size":1775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:15:30.753588+0000","scrub_duration":0.87706562700000001,"objects_trimmed":0,"snaptrim_duration":0.0048974149999999996,"stat_sum":{"num_bytes":12481557,"num_objects":484,"num_object_clones":220,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":199,"num_read":86,"num_read_kb":8232,"num_write":1087,"num_write_kb":18010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'1871","reported_seq":3844,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:49.871411+0000","last_change":"2023-10-30T14:51:49.871411+0000","last_active":"2023-10-30T14:51:49.871411+0000","last_peered":"2023-10-30T14:51:49.871411+0000","last_clean":"2023-10-30T14:51:49.871411+0000","last_became_active":"2023-10-30T14:14:04.609720+0000","last_became_peered":"2023-10-30T14:14:04.609720+0000","last_unstale":"2023-10-30T14:51:49.871411+0000","last_undegraded":"2023-10-30T14:51:49.871411+0000","last_fullsized":"2023-10-30T14:51:49.871411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1870","last_scrub_stamp":"2023-10-30T14:51:49.871181+0000","last_deep_scrub":"67'1870","last_deep_scrub_stamp":"2023-10-30T14:51:49.871181+0000","last_clean_scrub_stamp":"2023-10-30T14:51:49.871181+0000","objects_scrubbed":535,"log_size":1871,"log_dups_size":0,"ondisk_log_size":1871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:47:19.781621+0000","scrub_duration":1.1862301479999999,"objects_trimmed":0,"snaptrim_duration":0.0040008739999999997,"stat_sum":{"num_bytes":31023805,"num_objects":535,"num_object_clones":248,"num_object_copies":1070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":535,"num_whiteouts":229,"num_read":75,"num_read_kb":20512,"num_write":1101,"num_write_kb":32005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"67'1673","reported_seq":3679,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:07.073416+0000","last_change":"2023-10-30T14:52:07.073470+0000","last_active":"2023-10-30T14:52:07.073416+0000","last_peered":"2023-10-30T14:52:07.073416+0000","last_clean":"2023-10-30T14:52:07.073416+0000","last_became_active":"2023-10-30T14:14:04.620217+0000","last_became_peered":"2023-10-30T14:14:04.620217+0000","last_unstale":"2023-10-30T14:52:07.073416+0000","last_undegraded":"2023-10-30T14:52:07.073416+0000","last_fullsized":"2023-10-30T14:52:07.073416+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1673","last_scrub_stamp":"2023-10-30T14:52:07.073343+0000","last_deep_scrub":"67'1673","last_deep_scrub_stamp":"2023-10-30T14:52:07.073343+0000","last_clean_scrub_stamp":"2023-10-30T14:52:07.073343+0000","objects_scrubbed":455,"log_size":1673,"log_dups_size":0,"ondisk_log_size":1673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:59:35.579646+0000","scrub_duration":0.660174924,"objects_trimmed":0,"snaptrim_duration":0.0044491130000000002,"stat_sum":{"num_bytes":20631584,"num_objects":455,"num_object_clones":209,"num_object_copies":910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":455,"num_whiteouts":179,"num_read":71,"num_read_kb":15956,"num_write":1007,"num_write_kb":22437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"67'1841","reported_seq":3852,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:16.109899+0000","last_change":"2023-10-30T14:52:16.109981+0000","last_active":"2023-10-30T14:52:16.109899+0000","last_peered":"2023-10-30T14:52:16.109899+0000","last_clean":"2023-10-30T14:52:16.109899+0000","last_became_active":"2023-10-30T14:14:45.669878+0000","last_became_peered":"2023-10-30T14:14:45.669878+0000","last_unstale":"2023-10-30T14:52:16.109899+0000","last_undegraded":"2023-10-30T14:52:16.109899+0000","last_fullsized":"2023-10-30T14:52:16.109899+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1841","last_scrub_stamp":"2023-10-30T14:52:16.109813+0000","last_deep_scrub":"67'1841","last_deep_scrub_stamp":"2023-10-30T14:52:16.109813+0000","last_clean_scrub_stamp":"2023-10-30T14:52:16.109813+0000","objects_scrubbed":479,"log_size":1841,"log_dups_size":0,"ondisk_log_size":1841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:33:56.104550+0000","scrub_duration":0.80770848900000003,"objects_trimmed":0,"snaptrim_duration":0.0077997409999999998,"stat_sum":{"num_bytes":25414870,"num_objects":479,"num_object_clones":220,"num_object_copies":958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":194,"num_read":80,"num_read_kb":18178,"num_write":1147,"num_write_kb":27446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"67'1799","reported_seq":3794,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:05.950875+0000","last_change":"2023-10-30T14:52:05.950967+0000","last_active":"2023-10-30T14:52:05.950875+0000","last_peered":"2023-10-30T14:52:05.950875+0000","last_clean":"2023-10-30T14:52:05.950875+0000","last_became_active":"2023-10-30T14:14:04.597590+0000","last_became_peered":"2023-10-30T14:14:04.597590+0000","last_unstale":"2023-10-30T14:52:05.950875+0000","last_undegraded":"2023-10-30T14:52:05.950875+0000","last_fullsized":"2023-10-30T14:52:05.950875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1799","last_scrub_stamp":"2023-10-30T14:52:05.950769+0000","last_deep_scrub":"67'1799","last_deep_scrub_stamp":"2023-10-30T14:52:05.950769+0000","last_clean_scrub_stamp":"2023-10-30T14:52:05.950769+0000","objects_scrubbed":485,"log_size":1799,"log_dups_size":0,"ondisk_log_size":1799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:54.947034+0000","scrub_duration":0.61979928200000001,"objects_trimmed":0,"snaptrim_duration":0.0044830970000000001,"stat_sum":{"num_bytes":8769781,"num_objects":485,"num_object_clones":224,"num_object_copies":970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":197,"num_read":75,"num_read_kb":4156,"num_write":1100,"num_write_kb":11840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"67'1646","reported_seq":3633,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:02.338544+0000","last_change":"2023-10-30T14:52:02.338628+0000","last_active":"2023-10-30T14:52:02.338544+0000","last_peered":"2023-10-30T14:52:02.338544+0000","last_clean":"2023-10-30T14:52:02.338544+0000","last_became_active":"2023-10-30T14:14:04.623902+0000","last_became_peered":"2023-10-30T14:14:04.623902+0000","last_unstale":"2023-10-30T14:52:02.338544+0000","last_undegraded":"2023-10-30T14:52:02.338544+0000","last_fullsized":"2023-10-30T14:52:02.338544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1646","last_scrub_stamp":"2023-10-30T14:52:02.338437+0000","last_deep_scrub":"67'1646","last_deep_scrub_stamp":"2023-10-30T14:52:02.338437+0000","last_clean_scrub_stamp":"2023-10-30T14:52:02.338437+0000","objects_scrubbed":460,"log_size":1646,"log_dups_size":0,"ondisk_log_size":1646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:25:36.251158+0000","scrub_duration":0.90177968600000002,"objects_trimmed":0,"snaptrim_duration":0.0051343489999999999,"stat_sum":{"num_bytes":21110293,"num_objects":460,"num_object_clones":215,"num_object_copies":920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":188,"num_read":58,"num_read_kb":12498,"num_write":973,"num_write_kb":25426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"67'1857","reported_seq":3836,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:00.215600+0000","last_change":"2023-10-30T14:52:00.215683+0000","last_active":"2023-10-30T14:52:00.215600+0000","last_peered":"2023-10-30T14:52:00.215600+0000","last_clean":"2023-10-30T14:52:00.215600+0000","last_became_active":"2023-10-30T14:14:04.597556+0000","last_became_peered":"2023-10-30T14:14:04.597556+0000","last_unstale":"2023-10-30T14:52:00.215600+0000","last_undegraded":"2023-10-30T14:52:00.215600+0000","last_fullsized":"2023-10-30T14:52:00.215600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1857","last_scrub_stamp":"2023-10-30T14:52:00.215512+0000","last_deep_scrub":"67'1857","last_deep_scrub_stamp":"2023-10-30T14:52:00.215512+0000","last_clean_scrub_stamp":"2023-10-30T14:52:00.215512+0000","objects_scrubbed":524,"log_size":1857,"log_dups_size":0,"ondisk_log_size":1857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:49:29.093253+0000","scrub_duration":0.81083191899999996,"objects_trimmed":0,"snaptrim_duration":0.0040069729999999996,"stat_sum":{"num_bytes":25689725,"num_objects":524,"num_object_clones":241,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":212,"num_read":73,"num_read_kb":17618,"num_write":1103,"num_write_kb":27397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"67'1805","reported_seq":3800,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:08.853004+0000","last_change":"2023-10-30T14:52:08.853100+0000","last_active":"2023-10-30T14:52:08.853004+0000","last_peered":"2023-10-30T14:52:08.853004+0000","last_clean":"2023-10-30T14:52:08.853004+0000","last_became_active":"2023-10-30T14:14:04.618430+0000","last_became_peered":"2023-10-30T14:14:04.618430+0000","last_unstale":"2023-10-30T14:52:08.853004+0000","last_undegraded":"2023-10-30T14:52:08.853004+0000","last_fullsized":"2023-10-30T14:52:08.853004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1805","last_scrub_stamp":"2023-10-30T14:52:08.852862+0000","last_deep_scrub":"67'1805","last_deep_scrub_stamp":"2023-10-30T14:52:08.852862+0000","last_clean_scrub_stamp":"2023-10-30T14:52:08.852862+0000","objects_scrubbed":481,"log_size":1805,"log_dups_size":0,"ondisk_log_size":1805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:03:36.648223+0000","scrub_duration":1.0990789080000001,"objects_trimmed":0,"snaptrim_duration":0.0074984149999999996,"stat_sum":{"num_bytes":16403179,"num_objects":481,"num_object_clones":225,"num_object_copies":962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":199,"num_read":75,"num_read_kb":12323,"num_write":1105,"num_write_kb":17671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"67'1948","reported_seq":3932,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:55.392487+0000","last_change":"2023-10-30T14:51:55.392537+0000","last_active":"2023-10-30T14:51:55.392487+0000","last_peered":"2023-10-30T14:51:55.392487+0000","last_clean":"2023-10-30T14:51:55.392487+0000","last_became_active":"2023-10-30T14:14:04.606340+0000","last_became_peered":"2023-10-30T14:14:04.606340+0000","last_unstale":"2023-10-30T14:51:55.392487+0000","last_undegraded":"2023-10-30T14:51:55.392487+0000","last_fullsized":"2023-10-30T14:51:55.392487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1948","last_scrub_stamp":"2023-10-30T14:51:55.392413+0000","last_deep_scrub":"67'1948","last_deep_scrub_stamp":"2023-10-30T14:51:55.392413+0000","last_clean_scrub_stamp":"2023-10-30T14:51:55.392413+0000","objects_scrubbed":534,"log_size":1948,"log_dups_size":0,"ondisk_log_size":1948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:02:49.595530+0000","scrub_duration":0.84988978299999995,"objects_trimmed":0,"snaptrim_duration":0.0035094890000000002,"stat_sum":{"num_bytes":22361827,"num_objects":534,"num_object_clones":245,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":534,"num_whiteouts":212,"num_read":82,"num_read_kb":9614,"num_write":1202,"num_write_kb":32875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"67'1825","reported_seq":3829,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:03.419633+0000","last_change":"2023-10-30T14:52:03.419758+0000","last_active":"2023-10-30T14:52:03.419633+0000","last_peered":"2023-10-30T14:52:03.419633+0000","last_clean":"2023-10-30T14:52:03.419633+0000","last_became_active":"2023-10-30T14:14:04.612588+0000","last_became_peered":"2023-10-30T14:14:04.612588+0000","last_unstale":"2023-10-30T14:52:03.419633+0000","last_undegraded":"2023-10-30T14:52:03.419633+0000","last_fullsized":"2023-10-30T14:52:03.419633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1825","last_scrub_stamp":"2023-10-30T14:52:03.419531+0000","last_deep_scrub":"67'1825","last_deep_scrub_stamp":"2023-10-30T14:52:03.419531+0000","last_clean_scrub_stamp":"2023-10-30T14:52:03.419531+0000","objects_scrubbed":484,"log_size":1825,"log_dups_size":0,"ondisk_log_size":1825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:07:15.268230+0000","scrub_duration":1.1307163710000001,"objects_trimmed":0,"snaptrim_duration":0.0084570800000000005,"stat_sum":{"num_bytes":13290865,"num_objects":484,"num_object_clones":224,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":198,"num_read":83,"num_read_kb":8248,"num_write":1125,"num_write_kb":15099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"67'1743","reported_seq":3738,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:52.927568+0000","last_change":"2023-10-30T14:51:52.927568+0000","last_active":"2023-10-30T14:51:52.927568+0000","last_peered":"2023-10-30T14:51:52.927568+0000","last_clean":"2023-10-30T14:51:52.927568+0000","last_became_active":"2023-10-30T14:14:04.607683+0000","last_became_peered":"2023-10-30T14:14:04.607683+0000","last_unstale":"2023-10-30T14:51:52.927568+0000","last_undegraded":"2023-10-30T14:51:52.927568+0000","last_fullsized":"2023-10-30T14:51:52.927568+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1742","last_scrub_stamp":"2023-10-30T14:51:52.927213+0000","last_deep_scrub":"67'1742","last_deep_scrub_stamp":"2023-10-30T14:51:52.927213+0000","last_clean_scrub_stamp":"2023-10-30T14:51:52.927213+0000","objects_scrubbed":481,"log_size":1743,"log_dups_size":0,"ondisk_log_size":1743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:57:18.481646+0000","scrub_duration":0.84554383499999997,"objects_trimmed":0,"snaptrim_duration":0.001856688,"stat_sum":{"num_bytes":22393964,"num_objects":481,"num_object_clones":222,"num_object_copies":962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":201,"num_read":74,"num_read_kb":12305,"num_write":1055,"num_write_kb":25643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'710","reported_seq":889,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:54.430131+0000","last_change":"2023-10-30T14:51:46.684252+0000","last_active":"2023-10-30T14:51:54.430131+0000","last_peered":"2023-10-30T14:51:54.430131+0000","last_clean":"2023-10-30T14:51:54.430131+0000","last_became_active":"2023-10-30T14:14:03.731740+0000","last_became_peered":"2023-10-30T14:14:03.731740+0000","last_unstale":"2023-10-30T14:51:54.430131+0000","last_undegraded":"2023-10-30T14:51:54.430131+0000","last_fullsized":"2023-10-30T14:51:54.430131+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'706","last_scrub_stamp":"2023-10-30T14:51:46.684108+0000","last_deep_scrub":"67'706","last_deep_scrub_stamp":"2023-10-30T14:51:46.684108+0000","last_clean_scrub_stamp":"2023-10-30T14:51:46.684108+0000","objects_scrubbed":43,"log_size":710,"log_dups_size":0,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:53:22.342613+0000","scrub_duration":0.17419201400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50568158,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":149,"num_read_kb":143937,"num_write":707,"num_write_kb":74908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":402982,"num_omap_keys":749,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"67'1820","reported_seq":3816,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:11.899370+0000","last_change":"2023-10-30T14:52:11.899417+0000","last_active":"2023-10-30T14:52:11.899370+0000","last_peered":"2023-10-30T14:52:11.899370+0000","last_clean":"2023-10-30T14:52:11.899370+0000","last_became_active":"2023-10-30T14:14:04.621526+0000","last_became_peered":"2023-10-30T14:14:04.621526+0000","last_unstale":"2023-10-30T14:52:11.899370+0000","last_undegraded":"2023-10-30T14:52:11.899370+0000","last_fullsized":"2023-10-30T14:52:11.899370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1820","last_scrub_stamp":"2023-10-30T14:52:11.899293+0000","last_deep_scrub":"67'1820","last_deep_scrub_stamp":"2023-10-30T14:52:11.899293+0000","last_clean_scrub_stamp":"2023-10-30T14:52:11.899293+0000","objects_scrubbed":495,"log_size":1820,"log_dups_size":0,"ondisk_log_size":1820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:14:22.312179+0000","scrub_duration":0.63139072299999999,"objects_trimmed":0,"snaptrim_duration":0.0047814620000000002,"stat_sum":{"num_bytes":12429609,"num_objects":495,"num_object_clones":226,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":188,"num_read":79,"num_read_kb":8245,"num_write":1126,"num_write_kb":15289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"67'1505","reported_seq":1996,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:43.210550+0000","last_change":"2023-10-30T14:51:43.210666+0000","last_active":"2023-10-30T14:51:43.210550+0000","last_peered":"2023-10-30T14:51:43.210550+0000","last_clean":"2023-10-30T14:51:43.210550+0000","last_became_active":"2023-10-30T14:14:03.729386+0000","last_became_peered":"2023-10-30T14:14:03.729386+0000","last_unstale":"2023-10-30T14:51:43.210550+0000","last_undegraded":"2023-10-30T14:51:43.210550+0000","last_fullsized":"2023-10-30T14:51:43.210550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1505","last_scrub_stamp":"2023-10-30T14:51:43.210412+0000","last_deep_scrub":"67'1505","last_deep_scrub_stamp":"2023-10-30T14:51:43.210412+0000","last_clean_scrub_stamp":"2023-10-30T14:51:43.210412+0000","objects_scrubbed":46,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:18:14.848623+0000","scrub_duration":0.17059465400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29761637,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":457,"num_read_kb":164028,"num_write":1504,"num_write_kb":71675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":396614,"num_omap_keys":698,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'787","reported_seq":1361,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:45.617219+0000","last_change":"2023-10-30T14:51:45.617294+0000","last_active":"2023-10-30T14:51:45.617219+0000","last_peered":"2023-10-30T14:51:45.617219+0000","last_clean":"2023-10-30T14:51:45.617219+0000","last_became_active":"2023-10-30T14:14:03.724078+0000","last_became_peered":"2023-10-30T14:14:03.724078+0000","last_unstale":"2023-10-30T14:51:45.617219+0000","last_undegraded":"2023-10-30T14:51:45.617219+0000","last_fullsized":"2023-10-30T14:51:45.617219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'787","last_scrub_stamp":"2023-10-30T14:51:45.617096+0000","last_deep_scrub":"67'787","last_deep_scrub_stamp":"2023-10-30T14:51:45.617096+0000","last_clean_scrub_stamp":"2023-10-30T14:51:45.617096+0000","objects_scrubbed":42,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:36:59.792994+0000","scrub_duration":0.123030087,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30316903,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":547,"num_read_kb":63175,"num_write":794,"num_write_kb":47477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":452186,"num_omap_keys":609,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"67'1967","reported_seq":3929,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:59.432213+0000","last_change":"2023-10-30T14:51:50.284976+0000","last_active":"2023-10-30T14:51:59.432213+0000","last_peered":"2023-10-30T14:51:59.432213+0000","last_clean":"2023-10-30T14:51:59.432213+0000","last_became_active":"2023-10-30T14:14:04.597099+0000","last_became_peered":"2023-10-30T14:14:04.597099+0000","last_unstale":"2023-10-30T14:51:59.432213+0000","last_undegraded":"2023-10-30T14:51:59.432213+0000","last_fullsized":"2023-10-30T14:51:59.432213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1965","last_scrub_stamp":"2023-10-30T14:51:50.284800+0000","last_deep_scrub":"67'1965","last_deep_scrub_stamp":"2023-10-30T14:51:50.284800+0000","last_clean_scrub_stamp":"2023-10-30T14:51:50.284800+0000","objects_scrubbed":552,"log_size":1967,"log_dups_size":0,"ondisk_log_size":1967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:16:10.415430+0000","scrub_duration":0.75260475800000004,"objects_trimmed":0,"snaptrim_duration":0.0033627510000000002,"stat_sum":{"num_bytes":15535325,"num_objects":552,"num_object_clones":252,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":552,"num_whiteouts":225,"num_read":69,"num_read_kb":8270,"num_write":1198,"num_write_kb":25187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"67'1603","reported_seq":1776,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:41.944331+0000","last_change":"2023-10-30T14:51:41.944482+0000","last_active":"2023-10-30T14:51:41.944331+0000","last_peered":"2023-10-30T14:51:41.944331+0000","last_clean":"2023-10-30T14:51:41.944331+0000","last_became_active":"2023-10-30T14:14:03.719816+0000","last_became_peered":"2023-10-30T14:14:03.719816+0000","last_unstale":"2023-10-30T14:51:41.944331+0000","last_undegraded":"2023-10-30T14:51:41.944331+0000","last_fullsized":"2023-10-30T14:51:41.944331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1603","last_scrub_stamp":"2023-10-30T14:51:41.944113+0000","last_deep_scrub":"67'1603","last_deep_scrub_stamp":"2023-10-30T14:51:41.944113+0000","last_clean_scrub_stamp":"2023-10-30T14:51:41.944113+0000","objects_scrubbed":49,"log_size":1603,"log_dups_size":0,"ondisk_log_size":1603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:29.585178+0000","scrub_duration":0.217796778,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35358900,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":164,"num_read_kb":135701,"num_write":1593,"num_write_kb":77235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":466020,"num_omap_keys":829,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"67'1292","reported_seq":3258,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:38.430509+0000","last_change":"2023-10-30T14:23:30.771109+0000","last_active":"2023-10-30T14:51:38.430509+0000","last_peered":"2023-10-30T14:51:38.430509+0000","last_clean":"2023-10-30T14:51:38.430509+0000","last_became_active":"2023-10-30T14:14:04.599323+0000","last_became_peered":"2023-10-30T14:14:04.599323+0000","last_unstale":"2023-10-30T14:51:38.430509+0000","last_undegraded":"2023-10-30T14:51:38.430509+0000","last_fullsized":"2023-10-30T14:51:38.430509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1292,"log_dups_size":0,"ondisk_log_size":1292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025427000000000002,"stat_sum":{"num_bytes":25885700,"num_objects":482,"num_object_clones":224,"num_object_copies":964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":193,"num_read":75,"num_read_kb":21385,"num_write":1058,"num_write_kb":37196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"67'660","reported_seq":797,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:46.206639+0000","last_change":"2023-10-30T14:51:46.206708+0000","last_active":"2023-10-30T14:51:46.206639+0000","last_peered":"2023-10-30T14:51:46.206639+0000","last_clean":"2023-10-30T14:51:46.206639+0000","last_became_active":"2023-10-30T14:14:03.738132+0000","last_became_peered":"2023-10-30T14:14:03.738132+0000","last_unstale":"2023-10-30T14:51:46.206639+0000","last_undegraded":"2023-10-30T14:51:46.206639+0000","last_fullsized":"2023-10-30T14:51:46.206639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'660","last_scrub_stamp":"2023-10-30T14:51:46.206546+0000","last_deep_scrub":"67'660","last_deep_scrub_stamp":"2023-10-30T14:51:46.206546+0000","last_clean_scrub_stamp":"2023-10-30T14:51:46.206546+0000","objects_scrubbed":43,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:02:40.190575+0000","scrub_duration":0.151344961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360218,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":99,"num_read_kb":53823,"num_write":691,"num_write_kb":33991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":515776,"num_omap_keys":890,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"67'1308","reported_seq":3286,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.609888+0000","last_change":"2023-10-30T14:23:30.778832+0000","last_active":"2023-10-30T14:51:34.609888+0000","last_peered":"2023-10-30T14:51:34.609888+0000","last_clean":"2023-10-30T14:51:34.609888+0000","last_became_active":"2023-10-30T14:14:04.648816+0000","last_became_peered":"2023-10-30T14:14:04.648816+0000","last_unstale":"2023-10-30T14:51:34.609888+0000","last_undegraded":"2023-10-30T14:51:34.609888+0000","last_fullsized":"2023-10-30T14:51:34.609888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1308,"log_dups_size":0,"ondisk_log_size":1308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069848840000000002,"stat_sum":{"num_bytes":12168759,"num_objects":462,"num_object_clones":214,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":462,"num_whiteouts":185,"num_read":77,"num_read_kb":8211,"num_write":1090,"num_write_kb":14663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"67'1070","reported_seq":1217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:47.771730+0000","last_change":"2023-10-30T14:51:47.771870+0000","last_active":"2023-10-30T14:51:47.771730+0000","last_peered":"2023-10-30T14:51:47.771730+0000","last_clean":"2023-10-30T14:51:47.771730+0000","last_became_active":"2023-10-30T14:14:03.725272+0000","last_became_peered":"2023-10-30T14:14:03.725272+0000","last_unstale":"2023-10-30T14:51:47.771730+0000","last_undegraded":"2023-10-30T14:51:47.771730+0000","last_fullsized":"2023-10-30T14:51:47.771730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1070","last_scrub_stamp":"2023-10-30T14:51:47.771555+0000","last_deep_scrub":"67'1070","last_deep_scrub_stamp":"2023-10-30T14:51:47.771555+0000","last_clean_scrub_stamp":"2023-10-30T14:51:47.771555+0000","objects_scrubbed":41,"log_size":1070,"log_dups_size":0,"ondisk_log_size":1070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:13:17.687697+0000","scrub_duration":0.128806527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":123,"num_read_kb":86587,"num_write":1104,"num_write_kb":54796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":577700,"num_omap_keys":1131,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"67'1952","reported_seq":3902,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:04.197699+0000","last_change":"2023-10-30T14:52:04.197786+0000","last_active":"2023-10-30T14:52:04.197699+0000","last_peered":"2023-10-30T14:52:04.197699+0000","last_clean":"2023-10-30T14:52:04.197699+0000","last_became_active":"2023-10-30T14:14:04.609697+0000","last_became_peered":"2023-10-30T14:14:04.609697+0000","last_unstale":"2023-10-30T14:52:04.197699+0000","last_undegraded":"2023-10-30T14:52:04.197699+0000","last_fullsized":"2023-10-30T14:52:04.197699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1952","last_scrub_stamp":"2023-10-30T14:52:04.197582+0000","last_deep_scrub":"67'1952","last_deep_scrub_stamp":"2023-10-30T14:52:04.197582+0000","last_clean_scrub_stamp":"2023-10-30T14:52:04.197582+0000","objects_scrubbed":557,"log_size":1952,"log_dups_size":0,"ondisk_log_size":1952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:29:33.135073+0000","scrub_duration":0.89282557900000004,"objects_trimmed":0,"snaptrim_duration":0.0043395700000000001,"stat_sum":{"num_bytes":12484119,"num_objects":557,"num_object_clones":263,"num_object_copies":1114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":557,"num_whiteouts":236,"num_read":68,"num_read_kb":4261,"num_write":1150,"num_write_kb":13862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"67'1026","reported_seq":1197,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:50.429870+0000","last_change":"2023-10-30T14:51:50.429966+0000","last_active":"2023-10-30T14:51:50.429870+0000","last_peered":"2023-10-30T14:51:50.429870+0000","last_clean":"2023-10-30T14:51:50.429870+0000","last_became_active":"2023-10-30T14:14:03.727422+0000","last_became_peered":"2023-10-30T14:14:03.727422+0000","last_unstale":"2023-10-30T14:51:50.429870+0000","last_undegraded":"2023-10-30T14:51:50.429870+0000","last_fullsized":"2023-10-30T14:51:50.429870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1026","last_scrub_stamp":"2023-10-30T14:51:50.429770+0000","last_deep_scrub":"67'1026","last_deep_scrub_stamp":"2023-10-30T14:51:50.429770+0000","last_clean_scrub_stamp":"2023-10-30T14:51:50.429770+0000","objects_scrubbed":43,"log_size":1026,"log_dups_size":0,"ondisk_log_size":1026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:53.510519+0000","scrub_duration":0.107061642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389280,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":161,"num_read_kb":61498,"num_write":1088,"num_write_kb":30346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":619408,"num_omap_keys":1098,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'2082","reported_seq":3988,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:54.238386+0000","last_change":"2023-10-30T14:51:54.238386+0000","last_active":"2023-10-30T14:51:54.238386+0000","last_peered":"2023-10-30T14:51:54.238386+0000","last_clean":"2023-10-30T14:51:54.238386+0000","last_became_active":"2023-10-30T14:14:04.645333+0000","last_became_peered":"2023-10-30T14:14:04.645333+0000","last_unstale":"2023-10-30T14:51:54.238386+0000","last_undegraded":"2023-10-30T14:51:54.238386+0000","last_fullsized":"2023-10-30T14:51:54.238386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2081","last_scrub_stamp":"2023-10-30T14:51:54.238230+0000","last_deep_scrub":"67'2081","last_deep_scrub_stamp":"2023-10-30T14:51:54.238230+0000","last_clean_scrub_stamp":"2023-10-30T14:51:54.238230+0000","objects_scrubbed":601,"log_size":2082,"log_dups_size":0,"ondisk_log_size":2082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:44.996187+0000","scrub_duration":1.177356566,"objects_trimmed":0,"snaptrim_duration":0.002301474,"stat_sum":{"num_bytes":21718535,"num_objects":601,"num_object_clones":320,"num_object_copies":1202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":601,"num_whiteouts":225,"num_read":78,"num_read_kb":5632,"num_write":1157,"num_write_kb":28163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"67'606","reported_seq":804,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:51.267795+0000","last_change":"2023-10-30T14:51:51.267795+0000","last_active":"2023-10-30T14:51:51.267795+0000","last_peered":"2023-10-30T14:51:51.267795+0000","last_clean":"2023-10-30T14:51:51.267795+0000","last_became_active":"2023-10-30T14:14:03.724806+0000","last_became_peered":"2023-10-30T14:14:03.724806+0000","last_unstale":"2023-10-30T14:51:51.267795+0000","last_undegraded":"2023-10-30T14:51:51.267795+0000","last_fullsized":"2023-10-30T14:51:51.267795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'605","last_scrub_stamp":"2023-10-30T14:51:51.267479+0000","last_deep_scrub":"67'605","last_deep_scrub_stamp":"2023-10-30T14:51:51.267479+0000","last_clean_scrub_stamp":"2023-10-30T14:51:51.267479+0000","objects_scrubbed":50,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:02:26.648751+0000","scrub_duration":0.15457963999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360800,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":192,"num_read_kb":131840,"num_write":659,"num_write_kb":54410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":552351,"num_omap_keys":1032,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"67'1329","reported_seq":3275,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:41.714187+0000","last_change":"2023-10-30T14:23:31.073841+0000","last_active":"2023-10-30T14:51:41.714187+0000","last_peered":"2023-10-30T14:51:41.714187+0000","last_clean":"2023-10-30T14:51:41.714187+0000","last_became_active":"2023-10-30T14:14:04.646543+0000","last_became_peered":"2023-10-30T14:14:04.646543+0000","last_unstale":"2023-10-30T14:51:41.714187+0000","last_undegraded":"2023-10-30T14:51:41.714187+0000","last_fullsized":"2023-10-30T14:51:41.714187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1329,"log_dups_size":0,"ondisk_log_size":1329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00744474,"stat_sum":{"num_bytes":22194385,"num_objects":506,"num_object_clones":234,"num_object_copies":1012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":206,"num_read":65,"num_read_kb":16406,"num_write":1098,"num_write_kb":24097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"67'705","reported_seq":891,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:50.491455+0000","last_change":"2023-10-30T14:51:50.491497+0000","last_active":"2023-10-30T14:51:50.491455+0000","last_peered":"2023-10-30T14:51:50.491455+0000","last_clean":"2023-10-30T14:51:50.491455+0000","last_became_active":"2023-10-30T14:14:03.716941+0000","last_became_peered":"2023-10-30T14:14:03.716941+0000","last_unstale":"2023-10-30T14:51:50.491455+0000","last_undegraded":"2023-10-30T14:51:50.491455+0000","last_fullsized":"2023-10-30T14:51:50.491455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'705","last_scrub_stamp":"2023-10-30T14:51:50.491382+0000","last_deep_scrub":"67'705","last_deep_scrub_stamp":"2023-10-30T14:51:50.491382+0000","last_clean_scrub_stamp":"2023-10-30T14:51:50.491382+0000","objects_scrubbed":50,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:52:06.985963+0000","scrub_duration":0.118685916,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777250,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":201,"num_read_kb":82705,"num_write":762,"num_write_kb":38720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":647476,"num_omap_keys":1125,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"67'1949","reported_seq":3916,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:59.435052+0000","last_change":"2023-10-30T14:51:57.741625+0000","last_active":"2023-10-30T14:51:59.435052+0000","last_peered":"2023-10-30T14:51:59.435052+0000","last_clean":"2023-10-30T14:51:59.435052+0000","last_became_active":"2023-10-30T14:14:04.641494+0000","last_became_peered":"2023-10-30T14:14:04.641494+0000","last_unstale":"2023-10-30T14:51:59.435052+0000","last_undegraded":"2023-10-30T14:51:59.435052+0000","last_fullsized":"2023-10-30T14:51:59.435052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1947","last_scrub_stamp":"2023-10-30T14:51:57.741484+0000","last_deep_scrub":"67'1947","last_deep_scrub_stamp":"2023-10-30T14:51:57.741484+0000","last_clean_scrub_stamp":"2023-10-30T14:51:57.741484+0000","objects_scrubbed":539,"log_size":1949,"log_dups_size":0,"ondisk_log_size":1949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:12:05.541916+0000","scrub_duration":1.29683427,"objects_trimmed":0,"snaptrim_duration":0.0039664019999999999,"stat_sum":{"num_bytes":26425057,"num_objects":539,"num_object_clones":248,"num_object_copies":1078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":207,"num_read":69,"num_read_kb":14198,"num_write":1186,"num_write_kb":36722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"67'327","reported_seq":458,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:44.816766+0000","last_change":"2023-10-30T14:51:44.816862+0000","last_active":"2023-10-30T14:51:44.816766+0000","last_peered":"2023-10-30T14:51:44.816766+0000","last_clean":"2023-10-30T14:51:44.816766+0000","last_became_active":"2023-10-30T14:14:03.728053+0000","last_became_peered":"2023-10-30T14:14:03.728053+0000","last_unstale":"2023-10-30T14:51:44.816766+0000","last_undegraded":"2023-10-30T14:51:44.816766+0000","last_fullsized":"2023-10-30T14:51:44.816766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'327","last_scrub_stamp":"2023-10-30T14:51:44.816631+0000","last_deep_scrub":"67'327","last_deep_scrub_stamp":"2023-10-30T14:51:44.816631+0000","last_clean_scrub_stamp":"2023-10-30T14:51:44.816631+0000","objects_scrubbed":38,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:44:07.678724+0000","scrub_duration":0.092522458000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":90,"num_read_kb":49506,"num_write":347,"num_write_kb":29524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":346610,"num_omap_keys":678,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1407","reported_seq":3369,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:41.714179+0000","last_change":"2023-10-30T14:23:31.078295+0000","last_active":"2023-10-30T14:51:41.714179+0000","last_peered":"2023-10-30T14:51:41.714179+0000","last_clean":"2023-10-30T14:51:41.714179+0000","last_became_active":"2023-10-30T14:14:04.649896+0000","last_became_peered":"2023-10-30T14:14:04.649896+0000","last_unstale":"2023-10-30T14:51:41.714179+0000","last_undegraded":"2023-10-30T14:51:41.714179+0000","last_fullsized":"2023-10-30T14:51:41.714179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1407,"log_dups_size":0,"ondisk_log_size":1407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010792094,"stat_sum":{"num_bytes":9892654,"num_objects":516,"num_object_clones":241,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":516,"num_whiteouts":209,"num_read":88,"num_read_kb":2154,"num_write":1171,"num_write_kb":11317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"67'732","reported_seq":878,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:54.255960+0000","last_change":"2023-10-30T14:51:54.256020+0000","last_active":"2023-10-30T14:51:54.255960+0000","last_peered":"2023-10-30T14:51:54.255960+0000","last_clean":"2023-10-30T14:51:54.255960+0000","last_became_active":"2023-10-30T14:14:03.736155+0000","last_became_peered":"2023-10-30T14:14:03.736155+0000","last_unstale":"2023-10-30T14:51:54.255960+0000","last_undegraded":"2023-10-30T14:51:54.255960+0000","last_fullsized":"2023-10-30T14:51:54.255960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'732","last_scrub_stamp":"2023-10-30T14:51:54.255862+0000","last_deep_scrub":"67'732","last_deep_scrub_stamp":"2023-10-30T14:51:54.255862+0000","last_clean_scrub_stamp":"2023-10-30T14:51:54.255862+0000","objects_scrubbed":38,"log_size":732,"log_dups_size":0,"ondisk_log_size":732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:23:50.725664+0000","scrub_duration":0.094160410999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":104,"num_read_kb":67382,"num_write":743,"num_write_kb":42078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":416827,"num_omap_keys":710,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"67'1749","reported_seq":3749,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:07.048071+0000","last_change":"2023-10-30T14:52:07.048156+0000","last_active":"2023-10-30T14:52:07.048071+0000","last_peered":"2023-10-30T14:52:07.048071+0000","last_clean":"2023-10-30T14:52:07.048071+0000","last_became_active":"2023-10-30T14:14:04.614328+0000","last_became_peered":"2023-10-30T14:14:04.614328+0000","last_unstale":"2023-10-30T14:52:07.048071+0000","last_undegraded":"2023-10-30T14:52:07.048071+0000","last_fullsized":"2023-10-30T14:52:07.048071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1749","last_scrub_stamp":"2023-10-30T14:52:07.047963+0000","last_deep_scrub":"67'1749","last_deep_scrub_stamp":"2023-10-30T14:52:07.047963+0000","last_clean_scrub_stamp":"2023-10-30T14:52:07.047963+0000","objects_scrubbed":478,"log_size":1749,"log_dups_size":0,"ondisk_log_size":1749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:24:27.684849+0000","scrub_duration":0.80605112800000001,"objects_trimmed":0,"snaptrim_duration":0.010029286,"stat_sum":{"num_bytes":9325803,"num_objects":478,"num_object_clones":220,"num_object_copies":956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":188,"num_read":77,"num_read_kb":4169,"num_write":1062,"num_write_kb":11660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"67'1235","reported_seq":1429,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:46.739900+0000","last_change":"2023-10-30T14:51:46.739956+0000","last_active":"2023-10-30T14:51:46.739900+0000","last_peered":"2023-10-30T14:51:46.739900+0000","last_clean":"2023-10-30T14:51:46.739900+0000","last_became_active":"2023-10-30T14:14:03.700352+0000","last_became_peered":"2023-10-30T14:14:03.700352+0000","last_unstale":"2023-10-30T14:51:46.739900+0000","last_undegraded":"2023-10-30T14:51:46.739900+0000","last_fullsized":"2023-10-30T14:51:46.739900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1235","last_scrub_stamp":"2023-10-30T14:51:46.739823+0000","last_deep_scrub":"67'1235","last_deep_scrub_stamp":"2023-10-30T14:51:46.739823+0000","last_clean_scrub_stamp":"2023-10-30T14:51:46.739823+0000","objects_scrubbed":44,"log_size":1235,"log_dups_size":0,"ondisk_log_size":1235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:04:42.377958+0000","scrub_duration":0.069929302999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":252,"num_read_kb":55463,"num_write":1290,"num_write_kb":26283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":593045,"num_omap_keys":836,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"67'1852","reported_seq":3840,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:45.460580+0000","last_change":"2023-10-30T14:51:45.460580+0000","last_active":"2023-10-30T14:51:45.460580+0000","last_peered":"2023-10-30T14:51:45.460580+0000","last_clean":"2023-10-30T14:51:45.460580+0000","last_became_active":"2023-10-30T14:14:04.646639+0000","last_became_peered":"2023-10-30T14:14:04.646639+0000","last_unstale":"2023-10-30T14:51:45.460580+0000","last_undegraded":"2023-10-30T14:51:45.460580+0000","last_fullsized":"2023-10-30T14:51:45.460580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1851","last_scrub_stamp":"2023-10-30T14:51:45.460385+0000","last_deep_scrub":"67'1851","last_deep_scrub_stamp":"2023-10-30T14:51:45.460385+0000","last_clean_scrub_stamp":"2023-10-30T14:51:45.460385+0000","objects_scrubbed":508,"log_size":1852,"log_dups_size":0,"ondisk_log_size":1852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:46.502926+0000","scrub_duration":0.96639256900000003,"objects_trimmed":0,"snaptrim_duration":0.010379724,"stat_sum":{"num_bytes":13632563,"num_objects":508,"num_object_clones":233,"num_object_copies":1016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":508,"num_whiteouts":206,"num_read":77,"num_read_kb":12340,"num_write":1128,"num_write_kb":24370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'661","reported_seq":796,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:45.259670+0000","last_change":"2023-10-30T14:51:45.259757+0000","last_active":"2023-10-30T14:51:45.259670+0000","last_peered":"2023-10-30T14:51:45.259670+0000","last_clean":"2023-10-30T14:51:45.259670+0000","last_became_active":"2023-10-30T14:14:03.707950+0000","last_became_peered":"2023-10-30T14:14:03.707950+0000","last_unstale":"2023-10-30T14:51:45.259670+0000","last_undegraded":"2023-10-30T14:51:45.259670+0000","last_fullsized":"2023-10-30T14:51:45.259670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'661","last_scrub_stamp":"2023-10-30T14:51:45.259579+0000","last_deep_scrub":"67'661","last_deep_scrub_stamp":"2023-10-30T14:51:45.259579+0000","last_clean_scrub_stamp":"2023-10-30T14:51:45.259579+0000","objects_scrubbed":32,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:37:24.581339+0000","scrub_duration":0.102681681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971542,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":82,"num_read_kb":57693,"num_write":882,"num_write_kb":42351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":303757,"num_omap_keys":462,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"67'1765","reported_seq":3740,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:17.071746+0000","last_change":"2023-10-30T14:52:17.071813+0000","last_active":"2023-10-30T14:52:17.071746+0000","last_peered":"2023-10-30T14:52:17.071746+0000","last_clean":"2023-10-30T14:52:17.071746+0000","last_became_active":"2023-10-30T14:14:04.644761+0000","last_became_peered":"2023-10-30T14:14:04.644761+0000","last_unstale":"2023-10-30T14:52:17.071746+0000","last_undegraded":"2023-10-30T14:52:17.071746+0000","last_fullsized":"2023-10-30T14:52:17.071746+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1765","last_scrub_stamp":"2023-10-30T14:52:17.071658+0000","last_deep_scrub":"67'1765","last_deep_scrub_stamp":"2023-10-30T14:52:17.071658+0000","last_clean_scrub_stamp":"2023-10-30T14:52:17.071658+0000","objects_scrubbed":503,"log_size":1765,"log_dups_size":0,"ondisk_log_size":1765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:17:38.995398+0000","scrub_duration":0.84598015100000001,"objects_trimmed":0,"snaptrim_duration":0.011171970999999999,"stat_sum":{"num_bytes":12945563,"num_objects":503,"num_object_clones":231,"num_object_copies":1006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":213,"num_read":62,"num_read_kb":4133,"num_write":1006,"num_write_kb":18895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'558","reported_seq":699,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:48.767892+0000","last_change":"2023-10-30T14:51:48.767990+0000","last_active":"2023-10-30T14:51:48.767892+0000","last_peered":"2023-10-30T14:51:48.767892+0000","last_clean":"2023-10-30T14:51:48.767892+0000","last_became_active":"2023-10-30T14:14:03.705186+0000","last_became_peered":"2023-10-30T14:14:03.705186+0000","last_unstale":"2023-10-30T14:51:48.767892+0000","last_undegraded":"2023-10-30T14:51:48.767892+0000","last_fullsized":"2023-10-30T14:51:48.767892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'558","last_scrub_stamp":"2023-10-30T14:51:48.767760+0000","last_deep_scrub":"67'558","last_deep_scrub_stamp":"2023-10-30T14:51:48.767760+0000","last_clean_scrub_stamp":"2023-10-30T14:51:48.767760+0000","objects_scrubbed":43,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:37:06.574920+0000","scrub_duration":0.12212297900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":102,"num_read_kb":90592,"num_write":581,"num_write_kb":50591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":484629,"num_omap_keys":764,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"67'1327","reported_seq":3273,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.614000+0000","last_change":"2023-10-30T14:23:31.083260+0000","last_active":"2023-10-30T14:51:34.614000+0000","last_peered":"2023-10-30T14:51:34.614000+0000","last_clean":"2023-10-30T14:51:34.614000+0000","last_became_active":"2023-10-30T14:14:04.646785+0000","last_became_peered":"2023-10-30T14:14:04.646785+0000","last_unstale":"2023-10-30T14:51:34.614000+0000","last_undegraded":"2023-10-30T14:51:34.614000+0000","last_fullsized":"2023-10-30T14:51:34.614000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1327,"log_dups_size":0,"ondisk_log_size":1327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010111184,"stat_sum":{"num_bytes":17483229,"num_objects":512,"num_object_clones":242,"num_object_copies":1024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":512,"num_whiteouts":218,"num_read":72,"num_read_kb":10698,"num_write":1044,"num_write_kb":23548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"67'622","reported_seq":782,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:44.214200+0000","last_change":"2023-10-30T14:51:44.214254+0000","last_active":"2023-10-30T14:51:44.214200+0000","last_peered":"2023-10-30T14:51:44.214200+0000","last_clean":"2023-10-30T14:51:44.214200+0000","last_became_active":"2023-10-30T14:14:03.692966+0000","last_became_peered":"2023-10-30T14:14:03.692966+0000","last_unstale":"2023-10-30T14:51:44.214200+0000","last_undegraded":"2023-10-30T14:51:44.214200+0000","last_fullsized":"2023-10-30T14:51:44.214200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'622","last_scrub_stamp":"2023-10-30T14:51:44.214112+0000","last_deep_scrub":"67'622","last_deep_scrub_stamp":"2023-10-30T14:51:44.214112+0000","last_clean_scrub_stamp":"2023-10-30T14:51:44.214112+0000","objects_scrubbed":36,"log_size":622,"log_dups_size":0,"ondisk_log_size":622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:10:21.572387+0000","scrub_duration":0.085160906999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16780852,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":126,"num_read_kb":110211,"num_write":641,"num_write_kb":50166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":448077,"num_omap_keys":692,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'1826","reported_seq":3828,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:00.576393+0000","last_change":"2023-10-30T14:52:00.576470+0000","last_active":"2023-10-30T14:52:00.576393+0000","last_peered":"2023-10-30T14:52:00.576393+0000","last_clean":"2023-10-30T14:52:00.576393+0000","last_became_active":"2023-10-30T14:14:04.615925+0000","last_became_peered":"2023-10-30T14:14:04.615925+0000","last_unstale":"2023-10-30T14:52:00.576393+0000","last_undegraded":"2023-10-30T14:52:00.576393+0000","last_fullsized":"2023-10-30T14:52:00.576393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1826","last_scrub_stamp":"2023-10-30T14:52:00.576300+0000","last_deep_scrub":"67'1826","last_deep_scrub_stamp":"2023-10-30T14:52:00.576300+0000","last_clean_scrub_stamp":"2023-10-30T14:52:00.576300+0000","objects_scrubbed":487,"log_size":1826,"log_dups_size":0,"ondisk_log_size":1826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:01:05.409471+0000","scrub_duration":1.040647087,"objects_trimmed":0,"snaptrim_duration":0.0068317569999999999,"stat_sum":{"num_bytes":4343182,"num_objects":487,"num_object_clones":225,"num_object_copies":974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":205,"num_read":84,"num_read_kb":56,"num_write":1109,"num_write_kb":8908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"67'683","reported_seq":831,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:39.782505+0000","last_change":"2023-10-30T14:51:39.782590+0000","last_active":"2023-10-30T14:51:39.782505+0000","last_peered":"2023-10-30T14:51:39.782505+0000","last_clean":"2023-10-30T14:51:39.782505+0000","last_became_active":"2023-10-30T14:14:03.727409+0000","last_became_peered":"2023-10-30T14:14:03.727409+0000","last_unstale":"2023-10-30T14:51:39.782505+0000","last_undegraded":"2023-10-30T14:51:39.782505+0000","last_fullsized":"2023-10-30T14:51:39.782505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'683","last_scrub_stamp":"2023-10-30T14:51:39.782417+0000","last_deep_scrub":"67'683","last_deep_scrub_stamp":"2023-10-30T14:51:39.782417+0000","last_clean_scrub_stamp":"2023-10-30T14:51:39.782417+0000","objects_scrubbed":36,"log_size":683,"log_dups_size":0,"ondisk_log_size":683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:14:03.723383+0000","scrub_duration":0.097535790999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166406,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":118,"num_read_kb":66012,"num_write":697,"num_write_kb":50302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":407093,"num_omap_keys":513,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1295","reported_seq":3268,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:34.640923+0000","last_change":"2023-10-30T14:23:31.077576+0000","last_active":"2023-10-30T14:51:34.640923+0000","last_peered":"2023-10-30T14:51:34.640923+0000","last_clean":"2023-10-30T14:51:34.640923+0000","last_became_active":"2023-10-30T14:14:45.818032+0000","last_became_peered":"2023-10-30T14:14:45.818032+0000","last_unstale":"2023-10-30T14:51:34.640923+0000","last_undegraded":"2023-10-30T14:51:34.640923+0000","last_fullsized":"2023-10-30T14:51:34.640923+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T14:14:03.671464+0000","last_clean_scrub_stamp":"2023-10-30T14:14:03.671464+0000","objects_scrubbed":0,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010655906,"stat_sum":{"num_bytes":23800594,"num_objects":485,"num_object_clones":224,"num_object_copies":970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":201,"num_read":74,"num_read_kb":13429,"num_write":1078,"num_write_kb":29347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"67'119","reported_seq":241,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:50.399571+0000","last_change":"2023-10-30T14:51:50.399571+0000","last_active":"2023-10-30T14:51:50.399571+0000","last_peered":"2023-10-30T14:51:50.399571+0000","last_clean":"2023-10-30T14:51:50.399571+0000","last_became_active":"2023-10-30T14:13:54.250907+0000","last_became_peered":"2023-10-30T14:13:54.250907+0000","last_unstale":"2023-10-30T14:51:50.399571+0000","last_undegraded":"2023-10-30T14:51:50.399571+0000","last_fullsized":"2023-10-30T14:51:50.399571+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'119","last_scrub_stamp":"2023-10-30T14:51:50.399169+0000","last_deep_scrub":"67'119","last_deep_scrub_stamp":"2023-10-30T14:51:50.399169+0000","last_clean_scrub_stamp":"2023-10-30T14:51:50.399169+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:18.007665+0000","scrub_duration":0.0099470610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"67'751","reported_seq":905,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:45.869660+0000","last_change":"2023-10-30T14:51:45.869788+0000","last_active":"2023-10-30T14:51:45.869660+0000","last_peered":"2023-10-30T14:51:45.869660+0000","last_clean":"2023-10-30T14:51:45.869660+0000","last_became_active":"2023-10-30T14:14:03.728293+0000","last_became_peered":"2023-10-30T14:14:03.728293+0000","last_unstale":"2023-10-30T14:51:45.869660+0000","last_undegraded":"2023-10-30T14:51:45.869660+0000","last_fullsized":"2023-10-30T14:51:45.869660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'751","last_scrub_stamp":"2023-10-30T14:51:45.869434+0000","last_deep_scrub":"67'751","last_deep_scrub_stamp":"2023-10-30T14:51:45.869434+0000","last_clean_scrub_stamp":"2023-10-30T14:51:45.869434+0000","objects_scrubbed":46,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:31.381332+0000","scrub_duration":0.14853830900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165954,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":121,"num_read_kb":78313,"num_write":800,"num_write_kb":42232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":509463,"num_omap_keys":728,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"67'1729","reported_seq":3720,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:57.388643+0000","last_change":"2023-10-30T14:51:57.388643+0000","last_active":"2023-10-30T14:51:57.388643+0000","last_peered":"2023-10-30T14:51:57.388643+0000","last_clean":"2023-10-30T14:51:57.388643+0000","last_became_active":"2023-10-30T14:14:46.451620+0000","last_became_peered":"2023-10-30T14:14:46.451620+0000","last_unstale":"2023-10-30T14:51:57.388643+0000","last_undegraded":"2023-10-30T14:51:57.388643+0000","last_fullsized":"2023-10-30T14:51:57.388643+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1728","last_scrub_stamp":"2023-10-30T14:51:57.388517+0000","last_deep_scrub":"67'1728","last_deep_scrub_stamp":"2023-10-30T14:51:57.388517+0000","last_clean_scrub_stamp":"2023-10-30T14:51:57.388517+0000","objects_scrubbed":475,"log_size":1729,"log_dups_size":0,"ondisk_log_size":1729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:32:53.174606+0000","scrub_duration":0.79869517700000003,"objects_trimmed":0,"snaptrim_duration":0.0034002070000000001,"stat_sum":{"num_bytes":12539401,"num_objects":475,"num_object_clones":216,"num_object_copies":950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":184,"num_read":65,"num_read_kb":8203,"num_write":1056,"num_write_kb":18334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"67'873","reported_seq":1071,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:59.533102+0000","last_change":"2023-10-30T14:51:59.533102+0000","last_active":"2023-10-30T14:51:59.533102+0000","last_peered":"2023-10-30T14:51:59.533102+0000","last_clean":"2023-10-30T14:51:59.533102+0000","last_became_active":"2023-10-30T14:14:03.732199+0000","last_became_peered":"2023-10-30T14:14:03.732199+0000","last_unstale":"2023-10-30T14:51:59.533102+0000","last_undegraded":"2023-10-30T14:51:59.533102+0000","last_fullsized":"2023-10-30T14:51:59.533102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'873","last_scrub_stamp":"2023-10-30T14:51:59.532898+0000","last_deep_scrub":"67'873","last_deep_scrub_stamp":"2023-10-30T14:51:59.532898+0000","last_clean_scrub_stamp":"2023-10-30T14:51:59.532898+0000","objects_scrubbed":51,"log_size":873,"log_dups_size":0,"ondisk_log_size":873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:48:56.595717+0000","scrub_duration":0.15135636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":198,"num_read_kb":136195,"num_write":895,"num_write_kb":59080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":750104,"num_omap_keys":1295,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"67'1738","reported_seq":3727,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:56.490720+0000","last_change":"2023-10-30T14:51:56.490720+0000","last_active":"2023-10-30T14:51:56.490720+0000","last_peered":"2023-10-30T14:51:56.490720+0000","last_clean":"2023-10-30T14:51:56.490720+0000","last_became_active":"2023-10-30T14:14:04.619990+0000","last_became_peered":"2023-10-30T14:14:04.619990+0000","last_unstale":"2023-10-30T14:51:56.490720+0000","last_undegraded":"2023-10-30T14:51:56.490720+0000","last_fullsized":"2023-10-30T14:51:56.490720+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1737","last_scrub_stamp":"2023-10-30T14:51:56.490587+0000","last_deep_scrub":"67'1737","last_deep_scrub_stamp":"2023-10-30T14:51:56.490587+0000","last_clean_scrub_stamp":"2023-10-30T14:51:56.490587+0000","objects_scrubbed":475,"log_size":1738,"log_dups_size":0,"ondisk_log_size":1738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:42:50.032821+0000","scrub_duration":1.0169323969999999,"objects_trimmed":0,"snaptrim_duration":0.0045963250000000001,"stat_sum":{"num_bytes":8686325,"num_objects":475,"num_object_clones":219,"num_object_copies":950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":194,"num_read":63,"num_read_kb":1576,"num_write":1042,"num_write_kb":11254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"67'1782","reported_seq":2900,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:47.811285+0000","last_change":"2023-10-30T14:51:47.811373+0000","last_active":"2023-10-30T14:51:47.811285+0000","last_peered":"2023-10-30T14:51:47.811285+0000","last_clean":"2023-10-30T14:51:47.811285+0000","last_became_active":"2023-10-30T14:14:03.702513+0000","last_became_peered":"2023-10-30T14:14:03.702513+0000","last_unstale":"2023-10-30T14:51:47.811285+0000","last_undegraded":"2023-10-30T14:51:47.811285+0000","last_fullsized":"2023-10-30T14:51:47.811285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1782","last_scrub_stamp":"2023-10-30T14:51:47.811149+0000","last_deep_scrub":"67'1782","last_deep_scrub_stamp":"2023-10-30T14:51:47.811149+0000","last_clean_scrub_stamp":"2023-10-30T14:51:47.811149+0000","objects_scrubbed":55,"log_size":1782,"log_dups_size":0,"ondisk_log_size":1782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:45:13.943363+0000","scrub_duration":0.120464197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27782952,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":1112,"num_read_kb":113704,"num_write":1847,"num_write_kb":62106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":736919,"num_omap_keys":1275,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"67'1840","reported_seq":3821,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:59.429919+0000","last_change":"2023-10-30T14:51:58.917710+0000","last_active":"2023-10-30T14:51:59.429919+0000","last_peered":"2023-10-30T14:51:59.429919+0000","last_clean":"2023-10-30T14:51:59.429919+0000","last_became_active":"2023-10-30T14:14:04.589087+0000","last_became_peered":"2023-10-30T14:14:04.589087+0000","last_unstale":"2023-10-30T14:51:59.429919+0000","last_undegraded":"2023-10-30T14:51:59.429919+0000","last_fullsized":"2023-10-30T14:51:59.429919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1839","last_scrub_stamp":"2023-10-30T14:51:58.917576+0000","last_deep_scrub":"67'1839","last_deep_scrub_stamp":"2023-10-30T14:51:58.917576+0000","last_clean_scrub_stamp":"2023-10-30T14:51:58.917576+0000","objects_scrubbed":512,"log_size":1840,"log_dups_size":0,"ondisk_log_size":1840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:52:42.720787+0000","scrub_duration":0.62564701700000003,"objects_trimmed":0,"snaptrim_duration":0.0022871390000000001,"stat_sum":{"num_bytes":30609501,"num_objects":512,"num_object_clones":244,"num_object_copies":1024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":512,"num_whiteouts":220,"num_read":78,"num_read_kb":20511,"num_write":1070,"num_write_kb":36447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"67'656","reported_seq":816,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:42.807191+0000","last_change":"2023-10-30T14:51:42.807310+0000","last_active":"2023-10-30T14:51:42.807191+0000","last_peered":"2023-10-30T14:51:42.807191+0000","last_clean":"2023-10-30T14:51:42.807191+0000","last_became_active":"2023-10-30T14:14:03.728925+0000","last_became_peered":"2023-10-30T14:14:03.728925+0000","last_unstale":"2023-10-30T14:51:42.807191+0000","last_undegraded":"2023-10-30T14:51:42.807191+0000","last_fullsized":"2023-10-30T14:51:42.807191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'656","last_scrub_stamp":"2023-10-30T14:51:42.806981+0000","last_deep_scrub":"67'656","last_deep_scrub_stamp":"2023-10-30T14:51:42.806981+0000","last_clean_scrub_stamp":"2023-10-30T14:51:42.806981+0000","objects_scrubbed":40,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:27:15.123985+0000","scrub_duration":0.119997573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":120,"num_read_kb":131794,"num_write":661,"num_write_kb":62442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":535152,"num_omap_keys":914,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"67'1844","reported_seq":3823,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:49.103277+0000","last_change":"2023-10-30T14:51:49.103277+0000","last_active":"2023-10-30T14:51:49.103277+0000","last_peered":"2023-10-30T14:51:49.103277+0000","last_clean":"2023-10-30T14:51:49.103277+0000","last_became_active":"2023-10-30T14:14:04.607377+0000","last_became_peered":"2023-10-30T14:14:04.607377+0000","last_unstale":"2023-10-30T14:51:49.103277+0000","last_undegraded":"2023-10-30T14:51:49.103277+0000","last_fullsized":"2023-10-30T14:51:49.103277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1843","last_scrub_stamp":"2023-10-30T14:51:49.103084+0000","last_deep_scrub":"67'1843","last_deep_scrub_stamp":"2023-10-30T14:51:49.103084+0000","last_clean_scrub_stamp":"2023-10-30T14:51:49.103084+0000","objects_scrubbed":503,"log_size":1844,"log_dups_size":0,"ondisk_log_size":1844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:54:01.899046+0000","scrub_duration":1.0628055430000001,"objects_trimmed":0,"snaptrim_duration":0.0022021179999999999,"stat_sum":{"num_bytes":18168327,"num_objects":503,"num_object_clones":232,"num_object_copies":1006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":206,"num_read":65,"num_read_kb":9172,"num_write":1124,"num_write_kb":20929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"67'909","reported_seq":1057,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:52.423853+0000","last_change":"2023-10-30T14:51:52.423913+0000","last_active":"2023-10-30T14:51:52.423853+0000","last_peered":"2023-10-30T14:51:52.423853+0000","last_clean":"2023-10-30T14:51:52.423853+0000","last_became_active":"2023-10-30T14:14:03.692621+0000","last_became_peered":"2023-10-30T14:14:03.692621+0000","last_unstale":"2023-10-30T14:51:52.423853+0000","last_undegraded":"2023-10-30T14:51:52.423853+0000","last_fullsized":"2023-10-30T14:51:52.423853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'909","last_scrub_stamp":"2023-10-30T14:51:52.423719+0000","last_deep_scrub":"67'909","last_deep_scrub_stamp":"2023-10-30T14:51:52.423719+0000","last_clean_scrub_stamp":"2023-10-30T14:51:52.423719+0000","objects_scrubbed":43,"log_size":909,"log_dups_size":0,"ondisk_log_size":909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:15:28.937000+0000","scrub_duration":0.14411457899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":107,"num_read_kb":66236,"num_write":924,"num_write_kb":62670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":536879,"num_omap_keys":1021,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'1866","reported_seq":3825,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:58.425229+0000","last_change":"2023-10-30T14:51:58.425326+0000","last_active":"2023-10-30T14:51:58.425229+0000","last_peered":"2023-10-30T14:51:58.425229+0000","last_clean":"2023-10-30T14:51:58.425229+0000","last_became_active":"2023-10-30T14:14:04.613780+0000","last_became_peered":"2023-10-30T14:14:04.613780+0000","last_unstale":"2023-10-30T14:51:58.425229+0000","last_undegraded":"2023-10-30T14:51:58.425229+0000","last_fullsized":"2023-10-30T14:51:58.425229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1866","last_scrub_stamp":"2023-10-30T14:51:58.425122+0000","last_deep_scrub":"67'1866","last_deep_scrub_stamp":"2023-10-30T14:51:58.425122+0000","last_clean_scrub_stamp":"2023-10-30T14:51:58.425122+0000","objects_scrubbed":524,"log_size":1866,"log_dups_size":0,"ondisk_log_size":1866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:08:18.913623+0000","scrub_duration":0.94003744199999995,"objects_trimmed":0,"snaptrim_duration":0.0065285270000000001,"stat_sum":{"num_bytes":12750235,"num_objects":524,"num_object_clones":251,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":229,"num_read":63,"num_read_kb":8216,"num_write":1081,"num_write_kb":18064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"67'773","reported_seq":947,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:14.347808+0000","last_change":"2023-10-30T14:52:14.347925+0000","last_active":"2023-10-30T14:52:14.347808+0000","last_peered":"2023-10-30T14:52:14.347808+0000","last_clean":"2023-10-30T14:52:14.347808+0000","last_became_active":"2023-10-30T14:14:03.739650+0000","last_became_peered":"2023-10-30T14:14:03.739650+0000","last_unstale":"2023-10-30T14:52:14.347808+0000","last_undegraded":"2023-10-30T14:52:14.347808+0000","last_fullsized":"2023-10-30T14:52:14.347808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'773","last_scrub_stamp":"2023-10-30T14:52:14.347619+0000","last_deep_scrub":"67'773","last_deep_scrub_stamp":"2023-10-30T14:52:14.347619+0000","last_clean_scrub_stamp":"2023-10-30T14:52:14.347619+0000","objects_scrubbed":51,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:19:10.812358+0000","scrub_duration":0.145645301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777238,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":143,"num_read_kb":109192,"num_write":780,"num_write_kb":46441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":672517,"num_omap_keys":1187,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"67'1830","reported_seq":3815,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:00.930337+0000","last_change":"2023-10-30T14:52:00.930388+0000","last_active":"2023-10-30T14:52:00.930337+0000","last_peered":"2023-10-30T14:52:00.930337+0000","last_clean":"2023-10-30T14:52:00.930337+0000","last_became_active":"2023-10-30T14:14:04.590138+0000","last_became_peered":"2023-10-30T14:14:04.590138+0000","last_unstale":"2023-10-30T14:52:00.930337+0000","last_undegraded":"2023-10-30T14:52:00.930337+0000","last_fullsized":"2023-10-30T14:52:00.930337+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1830","last_scrub_stamp":"2023-10-30T14:52:00.930259+0000","last_deep_scrub":"67'1830","last_deep_scrub_stamp":"2023-10-30T14:52:00.930259+0000","last_clean_scrub_stamp":"2023-10-30T14:52:00.930259+0000","objects_scrubbed":503,"log_size":1830,"log_dups_size":0,"ondisk_log_size":1830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:27.603127+0000","scrub_duration":0.57889067100000002,"objects_trimmed":0,"snaptrim_duration":0.0031364370000000002,"stat_sum":{"num_bytes":9254236,"num_objects":503,"num_object_clones":233,"num_object_copies":1006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":207,"num_read":72,"num_read_kb":4145,"num_write":1099,"num_write_kb":19935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"67'793","reported_seq":945,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:43.351549+0000","last_change":"2023-10-30T14:51:43.351549+0000","last_active":"2023-10-30T14:51:43.351549+0000","last_peered":"2023-10-30T14:51:43.351549+0000","last_clean":"2023-10-30T14:51:43.351549+0000","last_became_active":"2023-10-30T14:14:03.721234+0000","last_became_peered":"2023-10-30T14:14:03.721234+0000","last_unstale":"2023-10-30T14:51:43.351549+0000","last_undegraded":"2023-10-30T14:51:43.351549+0000","last_fullsized":"2023-10-30T14:51:43.351549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'792","last_scrub_stamp":"2023-10-30T14:51:43.351414+0000","last_deep_scrub":"67'792","last_deep_scrub_stamp":"2023-10-30T14:51:43.351414+0000","last_clean_scrub_stamp":"2023-10-30T14:51:43.351414+0000","objects_scrubbed":34,"log_size":793,"log_dups_size":0,"ondisk_log_size":793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:49:31.872445+0000","scrub_duration":0.195047427,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":136,"num_read_kb":37589,"num_write":1319,"num_write_kb":40351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":332687,"num_omap_keys":642,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"67'1754","reported_seq":3736,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:06.360812+0000","last_change":"2023-10-30T14:52:06.360878+0000","last_active":"2023-10-30T14:52:06.360812+0000","last_peered":"2023-10-30T14:52:06.360812+0000","last_clean":"2023-10-30T14:52:06.360812+0000","last_became_active":"2023-10-30T14:14:04.633643+0000","last_became_peered":"2023-10-30T14:14:04.633643+0000","last_unstale":"2023-10-30T14:52:06.360812+0000","last_undegraded":"2023-10-30T14:52:06.360812+0000","last_fullsized":"2023-10-30T14:52:06.360812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1754","last_scrub_stamp":"2023-10-30T14:52:06.360742+0000","last_deep_scrub":"67'1754","last_deep_scrub_stamp":"2023-10-30T14:52:06.360742+0000","last_clean_scrub_stamp":"2023-10-30T14:52:06.360742+0000","objects_scrubbed":489,"log_size":1754,"log_dups_size":0,"ondisk_log_size":1754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:27.423144+0000","scrub_duration":0.91191191599999999,"objects_trimmed":0,"snaptrim_duration":0.0048916599999999999,"stat_sum":{"num_bytes":9341707,"num_objects":489,"num_object_clones":229,"num_object_copies":978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":205,"num_read":67,"num_read_kb":4121,"num_write":1049,"num_write_kb":15591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"67'464","reported_seq":611,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:44.149414+0000","last_change":"2023-10-30T14:51:44.149516+0000","last_active":"2023-10-30T14:51:44.149414+0000","last_peered":"2023-10-30T14:51:44.149414+0000","last_clean":"2023-10-30T14:51:44.149414+0000","last_became_active":"2023-10-30T14:14:03.737801+0000","last_became_peered":"2023-10-30T14:14:03.737801+0000","last_unstale":"2023-10-30T14:51:44.149414+0000","last_undegraded":"2023-10-30T14:51:44.149414+0000","last_fullsized":"2023-10-30T14:51:44.149414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'464","last_scrub_stamp":"2023-10-30T14:51:44.149257+0000","last_deep_scrub":"67'464","last_deep_scrub_stamp":"2023-10-30T14:51:44.149257+0000","last_clean_scrub_stamp":"2023-10-30T14:51:44.149257+0000","objects_scrubbed":47,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:41:08.565700+0000","scrub_duration":0.15054411200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":132,"num_read_kb":45549,"num_write":534,"num_write_kb":26063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":658163,"num_omap_keys":1051,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"67'1704","reported_seq":3703,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:02.952731+0000","last_change":"2023-10-30T14:52:02.952808+0000","last_active":"2023-10-30T14:52:02.952731+0000","last_peered":"2023-10-30T14:52:02.952731+0000","last_clean":"2023-10-30T14:52:02.952731+0000","last_became_active":"2023-10-30T14:14:04.588826+0000","last_became_peered":"2023-10-30T14:14:04.588826+0000","last_unstale":"2023-10-30T14:52:02.952731+0000","last_undegraded":"2023-10-30T14:52:02.952731+0000","last_fullsized":"2023-10-30T14:52:02.952731+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1704","last_scrub_stamp":"2023-10-30T14:52:02.952646+0000","last_deep_scrub":"67'1704","last_deep_scrub_stamp":"2023-10-30T14:52:02.952646+0000","last_clean_scrub_stamp":"2023-10-30T14:52:02.952646+0000","objects_scrubbed":453,"log_size":1704,"log_dups_size":0,"ondisk_log_size":1704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:53:48.955190+0000","scrub_duration":0.60434387099999998,"objects_trimmed":0,"snaptrim_duration":0.0040592670000000001,"stat_sum":{"num_bytes":8300341,"num_objects":453,"num_object_clones":211,"num_object_copies":906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":453,"num_whiteouts":180,"num_read":67,"num_read_kb":4125,"num_write":1055,"num_write_kb":10742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"67'452","reported_seq":574,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:55.355279+0000","last_change":"2023-10-30T14:51:55.355412+0000","last_active":"2023-10-30T14:51:55.355279+0000","last_peered":"2023-10-30T14:51:55.355279+0000","last_clean":"2023-10-30T14:51:55.355279+0000","last_became_active":"2023-10-30T14:14:03.732912+0000","last_became_peered":"2023-10-30T14:14:03.732912+0000","last_unstale":"2023-10-30T14:51:55.355279+0000","last_undegraded":"2023-10-30T14:51:55.355279+0000","last_fullsized":"2023-10-30T14:51:55.355279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'452","last_scrub_stamp":"2023-10-30T14:51:55.355171+0000","last_deep_scrub":"67'452","last_deep_scrub_stamp":"2023-10-30T14:51:55.355171+0000","last_clean_scrub_stamp":"2023-10-30T14:51:55.355171+0000","objects_scrubbed":23,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:16.834277+0000","scrub_duration":0.040337339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":78,"num_read_kb":41242,"num_write":429,"num_write_kb":21093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":203611,"num_omap_keys":381,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"67'1735","reported_seq":3730,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:49.980818+0000","last_change":"2023-10-30T14:51:49.980998+0000","last_active":"2023-10-30T14:51:49.980818+0000","last_peered":"2023-10-30T14:51:49.980818+0000","last_clean":"2023-10-30T14:51:49.980818+0000","last_became_active":"2023-10-30T14:14:04.609835+0000","last_became_peered":"2023-10-30T14:14:04.609835+0000","last_unstale":"2023-10-30T14:51:49.980818+0000","last_undegraded":"2023-10-30T14:51:49.980818+0000","last_fullsized":"2023-10-30T14:51:49.980818+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1735","last_scrub_stamp":"2023-10-30T14:51:49.980600+0000","last_deep_scrub":"67'1735","last_deep_scrub_stamp":"2023-10-30T14:51:49.980600+0000","last_clean_scrub_stamp":"2023-10-30T14:51:49.980600+0000","objects_scrubbed":470,"log_size":1735,"log_dups_size":0,"ondisk_log_size":1735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:25:11.080101+0000","scrub_duration":0.905012963,"objects_trimmed":0,"snaptrim_duration":0.0023958389999999999,"stat_sum":{"num_bytes":25940957,"num_objects":470,"num_object_clones":215,"num_object_copies":940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":192,"num_read":67,"num_read_kb":20495,"num_write":1067,"num_write_kb":27713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"67'884","reported_seq":1035,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:40.201508+0000","last_change":"2023-10-30T14:51:40.201605+0000","last_active":"2023-10-30T14:51:40.201508+0000","last_peered":"2023-10-30T14:51:40.201508+0000","last_clean":"2023-10-30T14:51:40.201508+0000","last_became_active":"2023-10-30T14:14:03.732265+0000","last_became_peered":"2023-10-30T14:14:03.732265+0000","last_unstale":"2023-10-30T14:51:40.201508+0000","last_undegraded":"2023-10-30T14:51:40.201508+0000","last_fullsized":"2023-10-30T14:51:40.201508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'884","last_scrub_stamp":"2023-10-30T14:51:40.201423+0000","last_deep_scrub":"67'884","last_deep_scrub_stamp":"2023-10-30T14:51:40.201423+0000","last_clean_scrub_stamp":"2023-10-30T14:51:40.201423+0000","objects_scrubbed":41,"log_size":884,"log_dups_size":0,"ondisk_log_size":884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:01:53.178520+0000","scrub_duration":0.062050998000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":128,"num_read_kb":49868,"num_write":925,"num_write_kb":26024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":535963,"num_omap_keys":904,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"67'1787","reported_seq":3793,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:49.263007+0000","last_change":"2023-10-30T14:51:49.263072+0000","last_active":"2023-10-30T14:51:49.263007+0000","last_peered":"2023-10-30T14:51:49.263007+0000","last_clean":"2023-10-30T14:51:49.263007+0000","last_became_active":"2023-10-30T14:14:04.594998+0000","last_became_peered":"2023-10-30T14:14:04.594998+0000","last_unstale":"2023-10-30T14:51:49.263007+0000","last_undegraded":"2023-10-30T14:51:49.263007+0000","last_fullsized":"2023-10-30T14:51:49.263007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1787","last_scrub_stamp":"2023-10-30T14:51:49.262934+0000","last_deep_scrub":"67'1787","last_deep_scrub_stamp":"2023-10-30T14:51:49.262934+0000","last_clean_scrub_stamp":"2023-10-30T14:51:49.262934+0000","objects_scrubbed":466,"log_size":1787,"log_dups_size":0,"ondisk_log_size":1787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:53:31.315621+0000","scrub_duration":0.73282899099999999,"objects_trimmed":0,"snaptrim_duration":0.00432118,"stat_sum":{"num_bytes":12343269,"num_objects":466,"num_object_clones":216,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":186,"num_read":78,"num_read_kb":8212,"num_write":1132,"num_write_kb":18560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'841","reported_seq":997,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:41.239916+0000","last_change":"2023-10-30T14:51:41.239981+0000","last_active":"2023-10-30T14:51:41.239916+0000","last_peered":"2023-10-30T14:51:41.239916+0000","last_clean":"2023-10-30T14:51:41.239916+0000","last_became_active":"2023-10-30T14:14:03.701759+0000","last_became_peered":"2023-10-30T14:14:03.701759+0000","last_unstale":"2023-10-30T14:51:41.239916+0000","last_undegraded":"2023-10-30T14:51:41.239916+0000","last_fullsized":"2023-10-30T14:51:41.239916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'841","last_scrub_stamp":"2023-10-30T14:51:41.239826+0000","last_deep_scrub":"67'841","last_deep_scrub_stamp":"2023-10-30T14:51:41.239826+0000","last_clean_scrub_stamp":"2023-10-30T14:51:41.239826+0000","objects_scrubbed":45,"log_size":841,"log_dups_size":0,"ondisk_log_size":841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:13:10.834903+0000","scrub_duration":0.095062992999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":122,"num_read_kb":94942,"num_write":1183,"num_write_kb":51323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":515565,"num_omap_keys":927,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"67'1674","reported_seq":3680,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:58.026778+0000","last_change":"2023-10-30T14:51:58.026837+0000","last_active":"2023-10-30T14:51:58.026778+0000","last_peered":"2023-10-30T14:51:58.026778+0000","last_clean":"2023-10-30T14:51:58.026778+0000","last_became_active":"2023-10-30T14:14:04.600100+0000","last_became_peered":"2023-10-30T14:14:04.600100+0000","last_unstale":"2023-10-30T14:51:58.026778+0000","last_undegraded":"2023-10-30T14:51:58.026778+0000","last_fullsized":"2023-10-30T14:51:58.026778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1674","last_scrub_stamp":"2023-10-30T14:51:58.026700+0000","last_deep_scrub":"67'1674","last_deep_scrub_stamp":"2023-10-30T14:51:58.026700+0000","last_clean_scrub_stamp":"2023-10-30T14:51:58.026700+0000","objects_scrubbed":447,"log_size":1674,"log_dups_size":0,"ondisk_log_size":1674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:30:55.078348+0000","scrub_duration":0.55760237800000001,"objects_trimmed":0,"snaptrim_duration":0.0052227630000000001,"stat_sum":{"num_bytes":16881451,"num_objects":447,"num_object_clones":209,"num_object_copies":894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":182,"num_read":73,"num_read_kb":11653,"num_write":1021,"num_write_kb":18895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"67'806","reported_seq":949,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:05.507781+0000","last_change":"2023-10-30T14:52:05.507868+0000","last_active":"2023-10-30T14:52:05.507781+0000","last_peered":"2023-10-30T14:52:05.507781+0000","last_clean":"2023-10-30T14:52:05.507781+0000","last_became_active":"2023-10-30T14:14:03.740648+0000","last_became_peered":"2023-10-30T14:14:03.740648+0000","last_unstale":"2023-10-30T14:52:05.507781+0000","last_undegraded":"2023-10-30T14:52:05.507781+0000","last_fullsized":"2023-10-30T14:52:05.507781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'806","last_scrub_stamp":"2023-10-30T14:52:05.507677+0000","last_deep_scrub":"67'806","last_deep_scrub_stamp":"2023-10-30T14:52:05.507677+0000","last_clean_scrub_stamp":"2023-10-30T14:52:05.507677+0000","objects_scrubbed":33,"log_size":806,"log_dups_size":0,"ondisk_log_size":806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:47:47.824486+0000","scrub_duration":0.134657734,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":98,"num_read_kb":107365,"num_write":798,"num_write_kb":50458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":418172,"num_omap_keys":644,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"67'1756","reported_seq":3765,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:01.140797+0000","last_change":"2023-10-30T14:52:01.140862+0000","last_active":"2023-10-30T14:52:01.140797+0000","last_peered":"2023-10-30T14:52:01.140797+0000","last_clean":"2023-10-30T14:52:01.140797+0000","last_became_active":"2023-10-30T14:14:04.622921+0000","last_became_peered":"2023-10-30T14:14:04.622921+0000","last_unstale":"2023-10-30T14:52:01.140797+0000","last_undegraded":"2023-10-30T14:52:01.140797+0000","last_fullsized":"2023-10-30T14:52:01.140797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1756","last_scrub_stamp":"2023-10-30T14:52:01.140687+0000","last_deep_scrub":"67'1756","last_deep_scrub_stamp":"2023-10-30T14:52:01.140687+0000","last_clean_scrub_stamp":"2023-10-30T14:52:01.140687+0000","objects_scrubbed":465,"log_size":1756,"log_dups_size":0,"ondisk_log_size":1756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:42:01.063269+0000","scrub_duration":0.83106379799999996,"objects_trimmed":0,"snaptrim_duration":0.0071219910000000003,"stat_sum":{"num_bytes":20799446,"num_objects":465,"num_object_clones":210,"num_object_copies":930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":465,"num_whiteouts":183,"num_read":75,"num_read_kb":12323,"num_write":1101,"num_write_kb":26545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"67'1810","reported_seq":3792,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:59.448185+0000","last_change":"2023-10-30T14:51:59.448289+0000","last_active":"2023-10-30T14:51:59.448185+0000","last_peered":"2023-10-30T14:51:59.448185+0000","last_clean":"2023-10-30T14:51:59.448185+0000","last_became_active":"2023-10-30T14:14:04.621885+0000","last_became_peered":"2023-10-30T14:14:04.621885+0000","last_unstale":"2023-10-30T14:51:59.448185+0000","last_undegraded":"2023-10-30T14:51:59.448185+0000","last_fullsized":"2023-10-30T14:51:59.448185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1810","last_scrub_stamp":"2023-10-30T14:51:59.448092+0000","last_deep_scrub":"67'1810","last_deep_scrub_stamp":"2023-10-30T14:51:59.448092+0000","last_clean_scrub_stamp":"2023-10-30T14:51:59.448092+0000","objects_scrubbed":487,"log_size":1810,"log_dups_size":0,"ondisk_log_size":1810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:36:33.157702+0000","scrub_duration":0.92442001200000001,"objects_trimmed":0,"snaptrim_duration":0.0067851329999999996,"stat_sum":{"num_bytes":13483294,"num_objects":487,"num_object_clones":228,"num_object_copies":974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":191,"num_read":68,"num_read_kb":8221,"num_write":1123,"num_write_kb":20773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"67'517","reported_seq":678,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:02.491572+0000","last_change":"2023-10-30T14:52:02.491644+0000","last_active":"2023-10-30T14:52:02.491572+0000","last_peered":"2023-10-30T14:52:02.491572+0000","last_clean":"2023-10-30T14:52:02.491572+0000","last_became_active":"2023-10-30T14:14:03.740247+0000","last_became_peered":"2023-10-30T14:14:03.740247+0000","last_unstale":"2023-10-30T14:52:02.491572+0000","last_undegraded":"2023-10-30T14:52:02.491572+0000","last_fullsized":"2023-10-30T14:52:02.491572+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'517","last_scrub_stamp":"2023-10-30T14:52:02.491486+0000","last_deep_scrub":"67'517","last_deep_scrub_stamp":"2023-10-30T14:52:02.491486+0000","last_clean_scrub_stamp":"2023-10-30T14:52:02.491486+0000","objects_scrubbed":49,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:33:41.383440+0000","scrub_duration":0.160540247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":149,"num_read_kb":82851,"num_write":526,"num_write_kb":42289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":701328,"num_omap_keys":1094,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'1964","reported_seq":3938,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:05.869580+0000","last_change":"2023-10-30T14:52:05.869633+0000","last_active":"2023-10-30T14:52:05.869580+0000","last_peered":"2023-10-30T14:52:05.869580+0000","last_clean":"2023-10-30T14:52:05.869580+0000","last_became_active":"2023-10-30T14:14:04.623638+0000","last_became_peered":"2023-10-30T14:14:04.623638+0000","last_unstale":"2023-10-30T14:52:05.869580+0000","last_undegraded":"2023-10-30T14:52:05.869580+0000","last_fullsized":"2023-10-30T14:52:05.869580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1964","last_scrub_stamp":"2023-10-30T14:52:05.869480+0000","last_deep_scrub":"67'1964","last_deep_scrub_stamp":"2023-10-30T14:52:05.869480+0000","last_clean_scrub_stamp":"2023-10-30T14:52:05.869480+0000","objects_scrubbed":546,"log_size":1964,"log_dups_size":0,"ondisk_log_size":1964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:19:44.576287+0000","scrub_duration":1.2098437230000001,"objects_trimmed":0,"snaptrim_duration":0.0049828509999999999,"stat_sum":{"num_bytes":25534330,"num_objects":546,"num_object_clones":248,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":546,"num_whiteouts":220,"num_read":77,"num_read_kb":19614,"num_write":1187,"num_write_kb":31851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"67'1076","reported_seq":1239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:13.386027+0000","last_change":"2023-10-30T14:52:13.386187+0000","last_active":"2023-10-30T14:52:13.386027+0000","last_peered":"2023-10-30T14:52:13.386027+0000","last_clean":"2023-10-30T14:52:13.386027+0000","last_became_active":"2023-10-30T14:14:03.732538+0000","last_became_peered":"2023-10-30T14:14:03.732538+0000","last_unstale":"2023-10-30T14:52:13.386027+0000","last_undegraded":"2023-10-30T14:52:13.386027+0000","last_fullsized":"2023-10-30T14:52:13.386027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1076","last_scrub_stamp":"2023-10-30T14:52:13.385874+0000","last_deep_scrub":"67'1076","last_deep_scrub_stamp":"2023-10-30T14:52:13.385874+0000","last_clean_scrub_stamp":"2023-10-30T14:52:13.385874+0000","objects_scrubbed":48,"log_size":1076,"log_dups_size":0,"ondisk_log_size":1076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:51:41.237586+0000","scrub_duration":0.18064516899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":137,"num_read_kb":150323,"num_write":1072,"num_write_kb":62947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":547720,"num_omap_keys":934,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1793","reported_seq":3773,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:02.330358+0000","last_change":"2023-10-30T14:52:02.330428+0000","last_active":"2023-10-30T14:52:02.330358+0000","last_peered":"2023-10-30T14:52:02.330358+0000","last_clean":"2023-10-30T14:52:02.330358+0000","last_became_active":"2023-10-30T14:14:04.611773+0000","last_became_peered":"2023-10-30T14:14:04.611773+0000","last_unstale":"2023-10-30T14:52:02.330358+0000","last_undegraded":"2023-10-30T14:52:02.330358+0000","last_fullsized":"2023-10-30T14:52:02.330358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1793","last_scrub_stamp":"2023-10-30T14:52:02.330281+0000","last_deep_scrub":"67'1793","last_deep_scrub_stamp":"2023-10-30T14:52:02.330281+0000","last_clean_scrub_stamp":"2023-10-30T14:52:02.330281+0000","objects_scrubbed":513,"log_size":1793,"log_dups_size":0,"ondisk_log_size":1793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:30.490023+0000","scrub_duration":0.96875353099999995,"objects_trimmed":0,"snaptrim_duration":0.0044924040000000002,"stat_sum":{"num_bytes":26364015,"num_objects":513,"num_object_clones":233,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":513,"num_whiteouts":203,"num_read":66,"num_read_kb":19602,"num_write":1077,"num_write_kb":27528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"67'747","reported_seq":899,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:40.646962+0000","last_change":"2023-10-30T14:51:40.646962+0000","last_active":"2023-10-30T14:51:40.646962+0000","last_peered":"2023-10-30T14:51:40.646962+0000","last_clean":"2023-10-30T14:51:40.646962+0000","last_became_active":"2023-10-30T14:14:03.703691+0000","last_became_peered":"2023-10-30T14:14:03.703691+0000","last_unstale":"2023-10-30T14:51:40.646962+0000","last_undegraded":"2023-10-30T14:51:40.646962+0000","last_fullsized":"2023-10-30T14:51:40.646962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'747","last_scrub_stamp":"2023-10-30T14:51:40.646821+0000","last_deep_scrub":"67'747","last_deep_scrub_stamp":"2023-10-30T14:51:40.646821+0000","last_clean_scrub_stamp":"2023-10-30T14:51:40.646821+0000","objects_scrubbed":37,"log_size":747,"log_dups_size":0,"ondisk_log_size":747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:10:48.146914+0000","scrub_duration":0.052199267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":102,"num_read_kb":111696,"num_write":764,"num_write_kb":58666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":552830,"num_omap_keys":982,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"67'1847","reported_seq":3811,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:05.940136+0000","last_change":"2023-10-30T14:52:05.940136+0000","last_active":"2023-10-30T14:52:05.940136+0000","last_peered":"2023-10-30T14:52:05.940136+0000","last_clean":"2023-10-30T14:52:05.940136+0000","last_became_active":"2023-10-30T14:14:04.613420+0000","last_became_peered":"2023-10-30T14:14:04.613420+0000","last_unstale":"2023-10-30T14:52:05.940136+0000","last_undegraded":"2023-10-30T14:52:05.940136+0000","last_fullsized":"2023-10-30T14:52:05.940136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1847","last_scrub_stamp":"2023-10-30T14:52:05.939976+0000","last_deep_scrub":"67'1847","last_deep_scrub_stamp":"2023-10-30T14:52:05.939976+0000","last_clean_scrub_stamp":"2023-10-30T14:52:05.939976+0000","objects_scrubbed":509,"log_size":1847,"log_dups_size":0,"ondisk_log_size":1847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:23:03.638601+0000","scrub_duration":1.492893977,"objects_trimmed":0,"snaptrim_duration":0.004010128,"stat_sum":{"num_bytes":13027616,"num_objects":509,"num_object_clones":238,"num_object_copies":1018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":509,"num_whiteouts":201,"num_read":55,"num_read_kb":8221,"num_write":1124,"num_write_kb":14511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"67'656","reported_seq":827,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:59.435536+0000","last_change":"2023-10-30T14:51:38.523396+0000","last_active":"2023-10-30T14:51:59.435536+0000","last_peered":"2023-10-30T14:51:59.435536+0000","last_clean":"2023-10-30T14:51:59.435536+0000","last_became_active":"2023-10-30T14:14:03.736964+0000","last_became_peered":"2023-10-30T14:14:03.736964+0000","last_unstale":"2023-10-30T14:51:59.435536+0000","last_undegraded":"2023-10-30T14:51:59.435536+0000","last_fullsized":"2023-10-30T14:51:59.435536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'654","last_scrub_stamp":"2023-10-30T14:51:38.523259+0000","last_deep_scrub":"67'654","last_deep_scrub_stamp":"2023-10-30T14:51:38.523259+0000","last_clean_scrub_stamp":"2023-10-30T14:51:38.523259+0000","objects_scrubbed":39,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:31:19.574364+0000","scrub_duration":0.063641798999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":182,"num_read_kb":58134,"num_write":700,"num_write_kb":34131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":484120,"num_omap_keys":814,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1055045924,"num_objects":31721,"num_object_clones":14708,"num_object_copies":63442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31721,"num_whiteouts":12823,"num_read":4602,"num_read_kb":662584,"num_write":70009,"num_write_kb":1350127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2186289152,"data_stored":2111320648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110945,"ondisk_log_size":110945,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":689694586,"num_objects":1358,"num_object_clones":0,"num_object_copies":2716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1358,"num_whiteouts":0,"num_read":6032,"num_read_kb":2835146,"num_write":28240,"num_write_kb":1523169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1128,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16321744,"num_omap_keys":27609,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1379532800,"data_stored":1379389172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26470,"ondisk_log_size":26470,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575314,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":641052,"kb_used_data":489172,"kb_used_omap":1,"kb_used_meta":151870,"kb_avail":93730788,"statfs":{"total":96636764160,"available":95980326912,"internally_reserved":0,"allocated":500912128,"data_stored":490062487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":155515339},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.077,"15min":0.997},"min":{"1min":0.51700000000000002,"5min":0.46300000000000002,"15min":0.45500000000000002},"max":{"1min":6.3040000000000003,"5min":6.399,"15min":6.399},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":0.97799999999999998,"15min":0.94399999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":6.4969999999999999,"5min":6.4969999999999999,"15min":6.4969999999999999},"last":0.88200000000000001}]},{"osd":1,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":0.92500000000000004,"15min":0.93000000000000005},"min":{"1min":0.61599999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":6.1390000000000002,"5min":6.1390000000000002,"15min":6.1390000000000002},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.96599999999999997,"15min":0.99399999999999999},"min":{"1min":0.58699999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":3.8460000000000001,"5min":6.1920000000000002,"15min":6.1920000000000002},"last":0.74299999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.0449999999999999,"15min":0.96899999999999997},"min":{"1min":0.70099999999999996,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":6.5739999999999998,"5min":6.5739999999999998,"15min":6.5739999999999998},"last":1.054},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.115,"15min":1.01},"min":{"1min":0.81599999999999995,"5min":0.54500000000000004,"15min":0.5},"max":{"1min":6.2350000000000003,"5min":6.5149999999999997,"15min":6.5149999999999997},"last":0.68600000000000005}]},{"osd":3,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.86199999999999999,"15min":0.85599999999999998},"min":{"1min":0.56200000000000006,"5min":0.54800000000000004,"15min":0.53800000000000003},"max":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":4.3230000000000004},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.121,"15min":1.056},"min":{"1min":0.66800000000000004,"5min":0.64800000000000002,"15min":0.54300000000000004},"max":{"1min":3.157,"5min":6.2670000000000003,"15min":6.2670000000000003},"last":0.84099999999999997}]},{"osd":4,"last update":"Mon Oct 30 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.95599999999999996,"15min":0.95699999999999996},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.54800000000000004},"max":{"1min":4.7409999999999997,"5min":4.7409999999999997,"15min":4.7409999999999997},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.036,"15min":0.97399999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":8.0719999999999992,"5min":8.0719999999999992,"15min":8.0719999999999992},"last":0.86099999999999999}]},{"osd":5,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3280000000000001,"15min":1.1759999999999999},"min":{"1min":0.78100000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":10.692,"5min":10.692,"15min":10.692},"last":0.62},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.3939999999999999,"15min":1.2869999999999999},"min":{"1min":0.72099999999999997,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":6.6429999999999998,"5min":6.6429999999999998,"15min":6.6429999999999998},"last":0.92300000000000004}]},{"osd":6,"last update":"Mon Oct 30 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.135,"15min":1.129},"min":{"1min":0.748,"5min":0.71599999999999997,"15min":0.62},"max":{"1min":5.2160000000000002,"5min":5.2160000000000002,"15min":5.2160000000000002},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.1639999999999999,"15min":1.107},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":4.367,"5min":4.367,"15min":4.367},"last":0.81399999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575314,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513220,"kb_used_data":387708,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":93858620,"statfs":{"total":96636764160,"available":96111226880,"internally_reserved":0,"allocated":397012992,"data_stored":389718204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":128514503},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":0.90200000000000002,"15min":0.84999999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":9.7949999999999999,"5min":9.7949999999999999,"15min":9.7949999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":1.23,"5min":0.97899999999999998,"15min":0.91500000000000004},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":9.0180000000000007,"5min":9.0180000000000007,"15min":9.0180000000000007},"last":0.88100000000000001}]},{"osd":1,"last update":"Mon Oct 30 14:52:12 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":0.97499999999999998,"15min":0.879},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":10.169,"5min":10.169,"15min":10.169},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.216,"5min":1,"15min":0.97599999999999998},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.318},"max":{"1min":8.4169999999999998,"5min":8.4169999999999998,"15min":8.4169999999999998},"last":0.72099999999999997}]},{"osd":2,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.90900000000000003,"15min":0.89700000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":5.3819999999999997,"5min":5.3819999999999997,"15min":5.3819999999999997},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.1559999999999999,"5min":0.98399999999999999,"15min":0.95399999999999996},"min":{"1min":0.496,"5min":0.496,"15min":0.49299999999999999},"max":{"1min":4.8949999999999996,"5min":4.8949999999999996,"15min":4.8949999999999996},"last":0.96399999999999997}]},{"osd":3,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":1.639,"5min":1.1379999999999999,"15min":1.0149999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":14.827999999999999,"5min":14.827999999999999,"15min":14.827999999999999},"last":1.04},{"interface":"front","average":{"1min":1.0920000000000001,"5min":0.98399999999999999,"15min":0.97399999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":4.9279999999999999,"5min":4.9279999999999999,"15min":4.9279999999999999},"last":0.80800000000000005}]},{"osd":4,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.056,"15min":1.0369999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":3.7549999999999999,"5min":3.8399999999999999,"15min":3.8399999999999999},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.24,"5min":1.0780000000000001,"15min":1.0569999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.6419999999999999,"5min":3.6419999999999999,"15min":3.6419999999999999},"last":1.091}]},{"osd":5,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.0389999999999999,"15min":0.97899999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.2120000000000002,"5min":11.728,"15min":11.728},"last":0.98499999999999999},{"interface":"front","average":{"1min":2.4289999999999998,"5min":1.3879999999999999,"15min":1.171},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":28.178999999999998,"5min":28.178999999999998,"15min":28.178999999999998},"last":1.0609999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.3080000000000001,"15min":1.165},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":9.7609999999999992,"5min":18.041,"15min":18.041},"last":1.145},{"interface":"front","average":{"1min":3.593,"5min":1.835,"15min":1.393},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":53.113,"5min":53.113,"15min":53.113},"last":1.206}]}]},{"osd":5,"up_from":13,"seq":55834575314,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704876,"kb_used_data":539108,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93666964,"statfs":{"total":96636764160,"available":95914971136,"internally_reserved":0,"allocated":552046592,"data_stored":540347463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:51 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.84399999999999997,"15min":0.81399999999999995},"min":{"1min":0.47699999999999998,"5min":0.43099999999999999,"15min":0.40100000000000002},"max":{"1min":1.2070000000000001,"5min":7.04,"15min":7.04},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.89500000000000002,"15min":0.84999999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.202,"5min":5.4720000000000004,"15min":5.4720000000000004},"last":0.76600000000000001}]},{"osd":1,"last update":"Mon Oct 30 14:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.873,"15min":0.879},"min":{"1min":0.53400000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.2390000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.84299999999999997,"15min":0.84499999999999997},"min":{"1min":0.53500000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0569999999999999,"5min":4.1630000000000003,"15min":4.1630000000000003},"last":0.68500000000000005}]},{"osd":2,"last update":"Mon Oct 30 14:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.93300000000000005,"15min":0.95399999999999996},"min":{"1min":0.54900000000000004,"5min":0.53100000000000003,"15min":0.44800000000000001},"max":{"1min":1.224,"5min":1.3640000000000001,"15min":2.044},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0980000000000001,"15min":1.0249999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.53800000000000003},"max":{"1min":1.3919999999999999,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":1.1259999999999999}]},{"osd":3,"last update":"Mon Oct 30 14:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.83199999999999996,"15min":0.92400000000000004},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0840000000000001,"5min":1.2010000000000001,"15min":1.4550000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.0669999999999999,"15min":0.98399999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.52100000000000002},"max":{"1min":1.3919999999999999,"5min":1.3959999999999999,"15min":1.4059999999999999},"last":1.175}]},{"osd":4,"last update":"Mon Oct 30 14:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.1160000000000001,"15min":1.032},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.59099999999999997},"max":{"1min":1.046,"5min":8.7360000000000007,"15min":8.7360000000000007},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.129,"5min":1.27,"15min":1.0760000000000001},"min":{"1min":0.73199999999999998,"5min":0.61199999999999999,"15min":0.59399999999999997},"max":{"1min":1.5980000000000001,"5min":23.640000000000001,"15min":23.640000000000001},"last":0.96199999999999997}]},{"osd":6,"last update":"Mon Oct 30 14:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1539999999999999,"15min":1.0800000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.55500000000000005},"max":{"1min":1.657,"5min":1.7470000000000001,"15min":2.306},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.3,"5min":1.2330000000000001,"15min":1.139},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.57899999999999996},"max":{"1min":1.6120000000000001,"5min":1.7569999999999999,"15min":2.774},"last":1.222}]},{"osd":7,"last update":"Mon Oct 30 14:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0029999999999999,"15min":1.0569999999999999},"min":{"1min":0.66700000000000004,"5min":0.64300000000000002,"15min":0.40799999999999997},"max":{"1min":1.532,"5min":1.532,"15min":3.0739999999999998},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0940000000000001,"15min":1.163},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.77600000000000002},"max":{"1min":1.1970000000000001,"5min":1.53,"15min":2.1339999999999999},"last":0.98899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575314,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":641432,"kb_used_data":479824,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":93730408,"statfs":{"total":96636764160,"available":95979937792,"internally_reserved":0,"allocated":491339776,"data_stored":479836699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":165476806},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.97299999999999998,"15min":0.87},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.141,"5min":7.1870000000000003,"15min":7.1870000000000003},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.80900000000000005,"5min":1.0940000000000001,"15min":0.95699999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.1180000000000001,"5min":10.864000000000001,"15min":10.864000000000001},"last":0.94299999999999995}]},{"osd":1,"last update":"Mon Oct 30 14:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":1.028,"15min":0.95299999999999996},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.331,"5min":7.6980000000000004,"15min":7.6980000000000004},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.90800000000000003,"15min":0.84399999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.36599999999999999},"max":{"1min":1.798,"5min":5.5259999999999998,"15min":5.5259999999999998},"last":0.755}]},{"osd":2,"last update":"Mon Oct 30 14:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.997,"15min":0.95999999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.756,"5min":5.423,"15min":5.423},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.2549999999999999,"15min":1.0960000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.325,"5min":13.221,"15min":13.221},"last":1.022}]},{"osd":3,"last update":"Mon Oct 30 14:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.95599999999999996,"15min":0.86199999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.96799999999999997,"5min":7.5620000000000003,"15min":7.5620000000000003},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.0800000000000001,"15min":0.996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.171,"5min":7.7370000000000001,"15min":7.7370000000000001},"last":1.1479999999999999}]},{"osd":5,"last update":"Mon Oct 30 14:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":1.042,"15min":0.96899999999999997},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.833,"5min":7.6360000000000001,"15min":7.6360000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.236,"15min":1.1399999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.399,"5min":7.2599999999999998,"15min":7.2599999999999998},"last":0.86299999999999999}]},{"osd":6,"last update":"Mon Oct 30 14:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.163,"15min":1.103},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.8500000000000001,"5min":5.4729999999999999,"15min":5.4729999999999999},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.016,"5min":1.2709999999999999,"15min":1.1719999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.454,"5min":10.102,"15min":10.102},"last":1.3380000000000001}]},{"osd":7,"last update":"Mon Oct 30 14:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.218,"15min":1.0820000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":13.026999999999999,"5min":13.026999999999999,"15min":13.026999999999999},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.236,"15min":1.208},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.8939999999999999,"5min":7.0490000000000004,"15min":7.0490000000000004},"last":1.4510000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575312,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":656204,"kb_used_data":508484,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":93715636,"statfs":{"total":96636764160,"available":95964811264,"internally_reserved":0,"allocated":520687616,"data_stored":509150004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":151255496},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:52:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.505,"15min":0.48399999999999999},"min":{"1min":0.30099999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.54900000000000004,"5min":4.1130000000000004,"15min":4.1130000000000004},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53300000000000003,"15min":0.51100000000000001},"min":{"1min":0.28899999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.72199999999999998,"5min":4.5190000000000001,"15min":4.5190000000000001},"last":0.623}]},{"osd":1,"last update":"Mon Oct 30 14:52:15 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.57799999999999996,"15min":0.54600000000000004},"min":{"1min":0.35999999999999999,"5min":0.317,"15min":0.314},"max":{"1min":0.98899999999999999,"5min":3.988,"15min":3.988},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56399999999999995,"15min":0.54900000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.29999999999999999},"max":{"1min":1.859,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":1.859}]},{"osd":2,"last update":"Mon Oct 30 14:52:07 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56699999999999995,"15min":0.56499999999999995},"min":{"1min":0.42299999999999999,"5min":0.29199999999999998,"15min":0.27600000000000002},"max":{"1min":0.77900000000000003,"5min":3.0590000000000002,"15min":8.6519999999999992},"last":1.2130000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.68300000000000005,"15min":0.63600000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.313},"max":{"1min":0.83999999999999997,"5min":7.0949999999999998,"15min":7.0949999999999998},"last":1.988}]},{"osd":4,"last update":"Mon Oct 30 14:52:07 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.80500000000000005,"15min":0.81599999999999995},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.50900000000000001},"max":{"1min":0.93899999999999995,"5min":4.5730000000000004,"15min":4.5730000000000004},"last":1.7729999999999999},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.872,"15min":0.84399999999999997},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.48899999999999999},"max":{"1min":1.0569999999999999,"5min":4.4720000000000004,"15min":4.4720000000000004},"last":1.0389999999999999}]},{"osd":5,"last update":"Mon Oct 30 14:52:07 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.98999999999999999,"15min":0.89400000000000002},"min":{"1min":0.56499999999999995,"5min":0.46899999999999997,"15min":0.46800000000000003},"max":{"1min":1.0029999999999999,"5min":14.974,"15min":14.974},"last":1.8380000000000001},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.94399999999999995,"15min":0.871},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.46600000000000003},"max":{"1min":1.1100000000000001,"5min":5.907,"15min":5.907},"last":1.3600000000000001}]},{"osd":6,"last update":"Mon Oct 30 14:52:07 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.96999999999999997,"15min":0.92000000000000004},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.2070000000000001,"5min":5.8300000000000001,"15min":5.8300000000000001},"last":1.893},{"interface":"front","average":{"1min":0.91400000000000003,"5min":1.018,"15min":0.96299999999999997},"min":{"1min":0.64200000000000002,"5min":0.60199999999999998,"15min":0.55100000000000005},"max":{"1min":1.248,"5min":6.0300000000000002,"15min":6.0300000000000002},"last":1.9490000000000001}]},{"osd":7,"last update":"Mon Oct 30 14:52:07 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.91200000000000003,"15min":0.99099999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.0529999999999999,"5min":4.6159999999999997,"15min":16.282},"last":1.8160000000000001},{"interface":"front","average":{"1min":0.871,"5min":0.96199999999999997,"15min":0.96799999999999997},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.58099999999999996},"max":{"1min":1.161,"5min":5.7770000000000001,"15min":16.096},"last":1.915}]}]},{"osd":2,"up_from":13,"seq":55834575314,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422268,"kb_used_data":310644,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":93949572,"statfs":{"total":96636764160,"available":96204361728,"internally_reserved":0,"allocated":318099456,"data_stored":311268353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114293190},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.45300000000000001,"15min":0.48699999999999999},"min":{"1min":0.30099999999999999,"5min":0.27400000000000002,"15min":0.26300000000000001},"max":{"1min":0.67800000000000005,"5min":0.79100000000000004,"15min":13.526},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.50900000000000001,"15min":0.51300000000000001},"min":{"1min":0.38300000000000001,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":1.2190000000000001},"last":0.36299999999999999}]},{"osd":1,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46300000000000002,"15min":0.47199999999999998},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65500000000000003,"5min":0.67200000000000004,"15min":1.5469999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54400000000000004,"15min":0.56000000000000005},"min":{"1min":0.38700000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73499999999999999,"5min":0.79300000000000004,"15min":1.452},"last":0.503}]},{"osd":3,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58799999999999997,"15min":0.60699999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81299999999999994,"5min":0.95399999999999996,"15min":1.9390000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.53900000000000003},"min":{"1min":0.35899999999999999,"5min":0.35499999999999998,"15min":0.33100000000000002},"max":{"1min":0.78500000000000003,"5min":0.79300000000000004,"15min":1.913},"last":0.54200000000000004}]},{"osd":4,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.81699999999999995,"15min":0.81499999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.44900000000000001},"max":{"1min":1.0309999999999999,"5min":4.2229999999999999,"15min":4.2229999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.91600000000000004,"15min":0.91000000000000003},"min":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.59599999999999997},"max":{"1min":1.081,"5min":4.2439999999999998,"15min":4.2439999999999998},"last":0.59699999999999998}]},{"osd":5,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.98999999999999999,"15min":0.86399999999999999},"min":{"1min":0.51900000000000002,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0620000000000001,"5min":22.285,"15min":22.285},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.82899999999999996,"15min":0.83399999999999996},"min":{"1min":0.58199999999999996,"5min":0.56000000000000005,"15min":0.51900000000000002},"max":{"1min":1.1319999999999999,"5min":2.3999999999999999,"15min":2.742},"last":0.65600000000000003}]},{"osd":6,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.86099999999999999,"15min":0.91500000000000004},"min":{"1min":0.65000000000000002,"5min":0.58799999999999997,"15min":0.54500000000000004},"max":{"1min":1.0489999999999999,"5min":2.911,"15min":27.731000000000002},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.872,"15min":0.874},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.1679999999999999,"5min":1.9099999999999999,"15min":2.0009999999999999},"last":0.63900000000000001}]},{"osd":7,"last update":"Mon Oct 30 14:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84299999999999997,"15min":0.91600000000000004},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.56499999999999995},"max":{"1min":1.105,"5min":1.2390000000000001,"15min":18.870999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.90400000000000003,"15min":0.96399999999999997},"min":{"1min":0.66600000000000004,"5min":0.61499999999999999,"15min":0.55500000000000005},"max":{"1min":1.0860000000000001,"5min":3.3809999999999998,"15min":18.356999999999999},"last":0.74099999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575313,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468404,"kb_used_data":351212,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":93903436,"statfs":{"total":96636764160,"available":96157118464,"internally_reserved":0,"allocated":359641088,"data_stored":352841256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":119994823},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.47199999999999998,"15min":0.44800000000000001},"min":{"1min":0.29399999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.65500000000000003,"5min":2.4889999999999999,"15min":2.4889999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.51000000000000001,"15min":0.54100000000000004},"min":{"1min":0.376,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.61699999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.42999999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.64800000000000002,"15min":0.56399999999999995},"min":{"1min":0.32700000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69199999999999995,"5min":8.3160000000000007,"15min":8.3160000000000007},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.53400000000000003,"15min":0.48999999999999999},"min":{"1min":0.32500000000000001,"5min":0.27200000000000002,"15min":0.26800000000000002},"max":{"1min":0.628,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.41399999999999998}]},{"osd":3,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.60499999999999998,"15min":0.59299999999999997},"min":{"1min":0.371,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.70999999999999996,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.498},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.64700000000000002,"15min":0.63200000000000001},"min":{"1min":0.436,"5min":0.38100000000000001,"15min":0.36499999999999999},"max":{"1min":0.73299999999999998,"5min":3.54,"15min":3.54},"last":0.52900000000000003}]},{"osd":4,"last update":"Mon Oct 30 14:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.90800000000000003,"15min":0.82199999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.0209999999999999,"5min":10.525,"15min":10.525},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.96199999999999997,"15min":0.84099999999999997},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1910000000000001,"5min":10.571,"15min":10.571},"last":0.99199999999999999}]},{"osd":5,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.876,"15min":0.88400000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.169,"5min":3.4969999999999999,"15min":3.4969999999999999},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.872,"5min":0.85699999999999998,"15min":0.79800000000000004},"min":{"1min":0.63500000000000001,"5min":0.56000000000000005,"15min":0.49099999999999999},"max":{"1min":1.2,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.90300000000000002}]},{"osd":6,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.80500000000000005,"15min":0.80000000000000004},"min":{"1min":0.54200000000000004,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0820000000000001,"5min":1.875,"15min":1.875},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.89000000000000001,"15min":0.83299999999999996},"min":{"1min":0.59799999999999998,"5min":0.53100000000000003,"15min":0.51700000000000002},"max":{"1min":1.1439999999999999,"5min":5.3680000000000003,"15min":5.3680000000000003},"last":0.755}]},{"osd":7,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":1.085,"15min":0.91200000000000003},"min":{"1min":0.626,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0600000000000001,"5min":19.885000000000002,"15min":19.885000000000002},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.87,"5min":1.0469999999999999,"15min":0.93100000000000005},"min":{"1min":0.61499999999999999,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.177,"5min":13.401999999999999,"15min":13.401999999999999},"last":0.84799999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575315,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553040,"kb_used_data":423368,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":93818800,"statfs":{"total":96636764160,"available":96070451200,"internally_reserved":0,"allocated":433528832,"data_stored":421882906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132774342},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45900000000000002,"15min":0.44800000000000001},"min":{"1min":0.32200000000000001,"5min":0.26800000000000002,"15min":0.22500000000000001},"max":{"1min":0.749,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.314},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.53300000000000003,"15min":0.503},"min":{"1min":0.36699999999999999,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":0.68999999999999995,"5min":3.367,"15min":3.367},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58199999999999996,"15min":0.53600000000000003},"min":{"1min":0.44400000000000001,"5min":0.378,"15min":0.30499999999999999},"max":{"1min":0.751,"5min":3.222,"15min":3.222},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53200000000000003,"15min":0.56000000000000005},"min":{"1min":0.40000000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":1.095},"last":0.37}]},{"osd":3,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.56599999999999995,"15min":0.60199999999999998},"min":{"1min":0.34000000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.63300000000000001,"5min":6.4980000000000002,"15min":6.4980000000000002},"last":0.44},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57499999999999996,"15min":0.52900000000000003},"min":{"1min":0.40400000000000003,"5min":0.40100000000000002,"15min":0.28000000000000003},"max":{"1min":0.71699999999999997,"5min":2.024,"15min":2.024},"last":0.48699999999999999}]},{"osd":4,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.78800000000000003,"15min":0.82599999999999996},"min":{"1min":0.57599999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.95399999999999996,"5min":2.0659999999999998,"15min":2.8380000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81399999999999995,"15min":0.85999999999999999},"min":{"1min":0.65100000000000002,"5min":0.56899999999999995,"15min":0.53900000000000003},"max":{"1min":0.97699999999999998,"5min":2.1000000000000001,"15min":5.8959999999999999},"last":0.70199999999999996}]},{"osd":5,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.872,"15min":0.86099999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.46300000000000002},"max":{"1min":1.212,"5min":1.278,"15min":6.4480000000000004},"last":0.62},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.92000000000000004,"15min":0.85599999999999998},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.54000000000000004},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":2.6869999999999998},"last":0.81000000000000005}]},{"osd":6,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.83899999999999997,"15min":0.84699999999999998},"min":{"1min":0.64800000000000002,"5min":0.55300000000000005,"15min":0.54400000000000004},"max":{"1min":1.133,"5min":1.2150000000000001,"15min":4.9980000000000002},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.877,"5min":0.85499999999999998,"15min":0.86099999999999999},"min":{"1min":0.65300000000000002,"5min":0.55100000000000005,"15min":0.54000000000000004},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.5980000000000001},"last":0.76500000000000001}]},{"osd":7,"last update":"Mon Oct 30 14:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.86799999999999999,"15min":0.84699999999999998},"min":{"1min":0.63700000000000001,"5min":0.57899999999999996,"15min":0.499},"max":{"1min":1.1120000000000001,"5min":3.6589999999999998,"15min":3.6589999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.93100000000000005,"15min":0.88400000000000001},"min":{"1min":0.76300000000000001,"5min":0.625,"15min":0.54100000000000004},"max":{"1min":1.1579999999999999,"5min":1.262,"15min":2.6429999999999998},"last":0.875}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":109068288,"data_stored":109052564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163602432,"data_stored":163582366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":115884032,"data_stored":115864160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":209350656,"data_stored":209331981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":153219072,"data_stored":153204613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":215736320,"data_stored":215715446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":211091456,"data_stored":211074246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":201580544,"data_stored":201563796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323084288,"data_stored":311837872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":195256320,"data_stored":188856788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":201433088,"data_stored":195002091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310554624,"data_stored":299415921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":337338368,"data_stored":326229984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":335527936,"data_stored":324229915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":185139200,"data_stored":178241856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":297955328,"data_stored":287506221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:52:23.109 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:23.110 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:23.110 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:23.111 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:23.111 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:23.112 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:23.112 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:23.113 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:23.113 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:23.114 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-30T14:14:03.671464+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=14, tm_sec=3, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=51, tm_sec=36, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:52:23.114 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T14:52:42.293 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:52:42.298 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:52:43.117 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:52:43.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.267+0000 7f8794f3e640 1 Processor -- start 2023-10-30T14:52:43.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.267+0000 7f8794f3e640 1 -- start start 2023-10-30T14:52:43.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.267+0000 7f8794f3e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8790153c80 0x7f879015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:43.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.267+0000 7f8794f3e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87900c4a10 0x7f87900c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:43.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.267+0000 7f8794f3e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8790159f20 con 0x7f879015a3b0 2023-10-30T14:52:43.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.267+0000 7f8794f3e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f879015dec0 con 0x7f8790153c80 2023-10-30T14:52:43.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.267+0000 7f8794f3e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87901599a0 con 0x7f87900c4a10 2023-10-30T14:52:43.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.267+0000 7f878e59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8790153c80 0x7f879015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:43.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.267+0000 7f878e59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8790153c80 0x7f879015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44992/0 (socket says 172.21.15.57:44992) 2023-10-30T14:52:43.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.267+0000 7f878e59c640 1 -- 172.21.15.57:0/1693461786 learned_addr learned my addr 172.21.15.57:0/1693461786 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:52:43.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.267+0000 7f8787fff640 1 --2- 172.21.15.57:0/1693461786 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87900c4a10 0x7f87900c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:43.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.267+0000 7f878e59c640 1 -- 172.21.15.57:0/1693461786 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87900c4a10 msgr2=0x7f87900c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:52:43.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.267+0000 7f878e59c640 1 --2- 172.21.15.57:0/1693461786 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87900c4a10 0x7f87900c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.267+0000 7f878e59c640 1 -- 172.21.15.57:0/1693461786 >> v1:172.21.15.88:6789/0 conn(0x7f879015a3b0 legacy=0x7f87900ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:43.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.267+0000 7f878e59c640 1 -- 172.21.15.57:0/1693461786 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87900c47e0 con 0x7f8790153c80 2023-10-30T14:52:43.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.271+0000 7f878e59c640 1 --2- 172.21.15.57:0/1693461786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8790153c80 0x7f879015cf90 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f8780009870 tx=0x7f878001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c01fd9eb55bb947d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:43.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.271+0000 7f878dd9b640 1 -- 172.21.15.57:0/1693461786 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87800a8a30 con 0x7f8790153c80 2023-10-30T14:52:43.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.271+0000 7f878dd9b640 1 -- 172.21.15.57:0/1693461786 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87800a8b90 con 0x7f8790153c80 2023-10-30T14:52:43.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.271+0000 7f878dd9b640 1 -- 172.21.15.57:0/1693461786 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87800b0440 con 0x7f8790153c80 2023-10-30T14:52:43.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.271+0000 7f8794f3e640 1 -- 172.21.15.57:0/1693461786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8790153c80 msgr2=0x7f879015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:43.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.271+0000 7f8794f3e640 1 --2- 172.21.15.57:0/1693461786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8790153c80 0x7f879015cf90 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f8780009870 tx=0x7f878001ed30 comp rx=0 tx=0).stop 2023-10-30T14:52:43.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.271+0000 7f8794f3e640 1 -- 172.21.15.57:0/1693461786 shutdown_connections 2023-10-30T14:52:43.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.271+0000 7f8794f3e640 1 --2- 172.21.15.57:0/1693461786 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87900c4a10 0x7f87900c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.271+0000 7f8794f3e640 1 --2- 172.21.15.57:0/1693461786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8790153c80 0x7f879015cf90 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.271+0000 7f8794f3e640 1 -- 172.21.15.57:0/1693461786 >> 172.21.15.57:0/1693461786 conn(0x7f87900715c0 msgr2=0x7f87900719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:43.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.271+0000 7f8794f3e640 1 -- 172.21.15.57:0/1693461786 shutdown_connections 2023-10-30T14:52:43.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.271+0000 7f8794f3e640 1 -- 172.21.15.57:0/1693461786 wait complete. 2023-10-30T14:52:43.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f8794f3e640 1 Processor -- start 2023-10-30T14:52:43.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f8794f3e640 1 -- start start 2023-10-30T14:52:43.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f8794f3e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87900c4a10 0x7f879014ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:43.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f8794f3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8790153c80 0x7f87900b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:43.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f8794f3e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f879015a3b0 0x7f87900ce450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:43.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f8794f3e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87900ce990 con 0x7f87900c4a10 2023-10-30T14:52:43.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f8794f3e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87900b26b0 con 0x7f8790153c80 2023-10-30T14:52:43.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f8794f3e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87900cdd20 con 0x7f879015a3b0 2023-10-30T14:52:43.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f878e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87900c4a10 0x7f879014ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:43.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f878e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87900c4a10 0x7f879014ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44996/0 (socket says 172.21.15.57:44996) 2023-10-30T14:52:43.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f878e59c640 1 -- 172.21.15.57:0/2844908246 learned_addr learned my addr 172.21.15.57:0/2844908246 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:52:43.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f8787fff640 1 --2- 172.21.15.57:0/2844908246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8790153c80 0x7f87900b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:43.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f878ed9d640 1 --2- 172.21.15.57:0/2844908246 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f879015a3b0 0x7f87900ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:43.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f878e59c640 1 -- 172.21.15.57:0/2844908246 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f879015a3b0 msgr2=0x7f87900ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:43.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f878e59c640 1 --2- 172.21.15.57:0/2844908246 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f879015a3b0 0x7f87900ce450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f878e59c640 1 -- 172.21.15.57:0/2844908246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8790153c80 msgr2=0x7f87900b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:43.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f878e59c640 1 --2- 172.21.15.57:0/2844908246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8790153c80 0x7f87900b2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f878e59c640 1 -- 172.21.15.57:0/2844908246 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87800094d0 con 0x7f87900c4a10 2023-10-30T14:52:43.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f878e59c640 1 --2- 172.21.15.57:0/2844908246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87900c4a10 0x7f879014ef80 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f87800b3040 tx=0x7f878001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:43.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f8786ffd640 1 -- 172.21.15.57:0/2844908246 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87800ba600 con 0x7f87900c4a10 2023-10-30T14:52:43.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f8786ffd640 1 -- 172.21.15.57:0/2844908246 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87800ba780 con 0x7f87900c4a10 2023-10-30T14:52:43.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f8786ffd640 1 -- 172.21.15.57:0/2844908246 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87800bab30 con 0x7f87900c4a10 2023-10-30T14:52:43.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f8794f3e640 1 -- 172.21.15.57:0/2844908246 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87900cf400 con 0x7f87900c4a10 2023-10-30T14:52:43.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.275+0000 7f8794f3e640 1 -- 172.21.15.57:0/2844908246 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87900cf680 con 0x7f87900c4a10 2023-10-30T14:52:43.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.279+0000 7f8794f3e640 1 -- 172.21.15.57:0/2844908246 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f879004e170 con 0x7f87900c4a10 2023-10-30T14:52:43.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.279+0000 7f8786ffd640 1 -- 172.21.15.57:0/2844908246 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f87800bac90 con 0x7f87900c4a10 2023-10-30T14:52:43.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.279+0000 7f8786ffd640 1 --2- 172.21.15.57:0/2844908246 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f87700664a0 0x7f8770068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:43.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.279+0000 7f8786ffd640 1 -- 172.21.15.57:0/2844908246 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f8780122c50 con 0x7f87900c4a10 2023-10-30T14:52:43.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.279+0000 7f8787fff640 1 --2- 172.21.15.57:0/2844908246 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f87700664a0 0x7f8770068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:43.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.279+0000 7f8787fff640 1 --2- 172.21.15.57:0/2844908246 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f87700664a0 0x7f8770068960 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f87900ceff0 tx=0x7f8778008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:43.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.283+0000 7f8786ffd640 1 -- 172.21.15.57:0/2844908246 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f87800ef920 con 0x7f87900c4a10 2023-10-30T14:52:43.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.439+0000 7f8794f3e640 1 -- 172.21.15.57:0/2844908246 --> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f879015d620 con 0x7f87700664a0 2023-10-30T14:52:43.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.463+0000 7f8786ffd640 1 -- 172.21.15.57:0/2844908246 <== mgr.4111 v2:172.21.15.88:6832/3881513431 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+296044 (secure 0 0 0) 0x7f877800b200 con 0x7f87700664a0 2023-10-30T14:52:43.469 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:52:43.471 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-30T14:52:43.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.463+0000 7f8794f3e640 1 -- 172.21.15.57:0/2844908246 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f87700664a0 msgr2=0x7f8770068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:43.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.463+0000 7f8794f3e640 1 --2- 172.21.15.57:0/2844908246 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f87700664a0 0x7f8770068960 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f87900ceff0 tx=0x7f8778008040 comp rx=0 tx=0).stop 2023-10-30T14:52:43.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.463+0000 7f8794f3e640 1 -- 172.21.15.57:0/2844908246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87900c4a10 msgr2=0x7f879014ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:43.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.463+0000 7f8794f3e640 1 --2- 172.21.15.57:0/2844908246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87900c4a10 0x7f879014ef80 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f87800b3040 tx=0x7f878001bad0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.463+0000 7f8794f3e640 1 -- 172.21.15.57:0/2844908246 shutdown_connections 2023-10-30T14:52:43.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.463+0000 7f8794f3e640 1 --2- 172.21.15.57:0/2844908246 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f87700664a0 0x7f8770068960 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.463+0000 7f8794f3e640 1 --2- 172.21.15.57:0/2844908246 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f879015a3b0 0x7f87900ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.463+0000 7f8794f3e640 1 --2- 172.21.15.57:0/2844908246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8790153c80 0x7f87900b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.463+0000 7f8794f3e640 1 --2- 172.21.15.57:0/2844908246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87900c4a10 0x7f879014ef80 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.463+0000 7f8794f3e640 1 -- 172.21.15.57:0/2844908246 >> 172.21.15.57:0/2844908246 conn(0x7f87900715c0 msgr2=0x7f8790152fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:43.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.463+0000 7f8794f3e640 1 -- 172.21.15.57:0/2844908246 shutdown_connections 2023-10-30T14:52:43.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.463+0000 7f8794f3e640 1 -- 172.21.15.57:0/2844908246 wait complete. 2023-10-30T14:52:43.489 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":1226,"stamp":"2023-10-30T14:52:43.340697+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1745330878,"num_objects":33081,"num_object_clones":14708,"num_object_copies":66162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33081,"num_whiteouts":12823,"num_read":10750,"num_read_kb":3497830,"num_write":98408,"num_write_kb":2875712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1128,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16321744,"num_omap_keys":27609,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142409,"ondisk_log_size":142409,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":4618544,"kb_used_data":3489520,"kb_used_omap":12,"kb_used_meta":1128947,"kb_avail":750356176,"statfs":{"total":773094113280,"available":768364724224,"internally_reserved":0,"allocated":3573268480,"data_stored":3495107372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":1156042297},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004336"},"pg_stats":[{"pgid":"3.3f","version":"67'1755","reported_seq":3744,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:24.972573+0000","last_change":"2023-10-30T14:52:24.972830+0000","last_active":"2023-10-30T14:52:24.972573+0000","last_peered":"2023-10-30T14:52:24.972573+0000","last_clean":"2023-10-30T14:52:24.972573+0000","last_became_active":"2023-10-30T14:14:04.650882+0000","last_became_peered":"2023-10-30T14:14:04.650882+0000","last_unstale":"2023-10-30T14:52:24.972573+0000","last_undegraded":"2023-10-30T14:52:24.972573+0000","last_fullsized":"2023-10-30T14:52:24.972573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1755","last_scrub_stamp":"2023-10-30T14:52:24.972443+0000","last_deep_scrub":"67'1755","last_deep_scrub_stamp":"2023-10-30T14:52:24.972443+0000","last_clean_scrub_stamp":"2023-10-30T14:52:24.972443+0000","objects_scrubbed":463,"log_size":1755,"log_dups_size":0,"ondisk_log_size":1755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:52:24.103372+0000","scrub_duration":0.813438255,"objects_trimmed":0,"snaptrim_duration":0.010330941,"stat_sum":{"num_bytes":16885684,"num_objects":463,"num_object_clones":217,"num_object_copies":926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":463,"num_whiteouts":185,"num_read":62,"num_read_kb":8246,"num_write":1079,"num_write_kb":18499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"67'1769","reported_seq":3752,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:12.785605+0000","last_change":"2023-10-30T14:52:12.785667+0000","last_active":"2023-10-30T14:52:12.785605+0000","last_peered":"2023-10-30T14:52:12.785605+0000","last_clean":"2023-10-30T14:52:12.785605+0000","last_became_active":"2023-10-30T14:14:04.587942+0000","last_became_peered":"2023-10-30T14:14:04.587942+0000","last_unstale":"2023-10-30T14:52:12.785605+0000","last_undegraded":"2023-10-30T14:52:12.785605+0000","last_fullsized":"2023-10-30T14:52:12.785605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1769","last_scrub_stamp":"2023-10-30T14:52:12.785532+0000","last_deep_scrub":"67'1769","last_deep_scrub_stamp":"2023-10-30T14:52:12.785532+0000","last_clean_scrub_stamp":"2023-10-30T14:52:12.785532+0000","objects_scrubbed":485,"log_size":1769,"log_dups_size":0,"ondisk_log_size":1769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:48:56.925914+0000","scrub_duration":0.54729001300000002,"objects_trimmed":0,"snaptrim_duration":0.002850184,"stat_sum":{"num_bytes":13093874,"num_objects":485,"num_object_clones":221,"num_object_copies":970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":193,"num_read":59,"num_read_kb":8220,"num_write":1078,"num_write_kb":14962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"67'1798","reported_seq":3790,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:10.917457+0000","last_change":"2023-10-30T14:52:10.917502+0000","last_active":"2023-10-30T14:52:10.917457+0000","last_peered":"2023-10-30T14:52:10.917457+0000","last_clean":"2023-10-30T14:52:10.917457+0000","last_became_active":"2023-10-30T14:14:04.590700+0000","last_became_peered":"2023-10-30T14:14:04.590700+0000","last_unstale":"2023-10-30T14:52:10.917457+0000","last_undegraded":"2023-10-30T14:52:10.917457+0000","last_fullsized":"2023-10-30T14:52:10.917457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1798","last_scrub_stamp":"2023-10-30T14:52:10.917377+0000","last_deep_scrub":"67'1798","last_deep_scrub_stamp":"2023-10-30T14:52:10.917377+0000","last_clean_scrub_stamp":"2023-10-30T14:52:10.917377+0000","objects_scrubbed":491,"log_size":1798,"log_dups_size":0,"ondisk_log_size":1798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:07:29.339766+0000","scrub_duration":0.60249001999999996,"objects_trimmed":0,"snaptrim_duration":0.0042861139999999997,"stat_sum":{"num_bytes":12610897,"num_objects":491,"num_object_clones":227,"num_object_copies":982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":203,"num_read":73,"num_read_kb":16409,"num_write":1067,"num_write_kb":22118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"67'1784","reported_seq":3772,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:00.217825+0000","last_change":"2023-10-30T14:52:00.217920+0000","last_active":"2023-10-30T14:52:00.217825+0000","last_peered":"2023-10-30T14:52:00.217825+0000","last_clean":"2023-10-30T14:52:00.217825+0000","last_became_active":"2023-10-30T14:14:45.661677+0000","last_became_peered":"2023-10-30T14:14:45.661677+0000","last_unstale":"2023-10-30T14:52:00.217825+0000","last_undegraded":"2023-10-30T14:52:00.217825+0000","last_fullsized":"2023-10-30T14:52:00.217825+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1784","last_scrub_stamp":"2023-10-30T14:52:00.217714+0000","last_deep_scrub":"67'1784","last_deep_scrub_stamp":"2023-10-30T14:52:00.217714+0000","last_clean_scrub_stamp":"2023-10-30T14:52:00.217714+0000","objects_scrubbed":483,"log_size":1784,"log_dups_size":0,"ondisk_log_size":1784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:15:45.674235+0000","scrub_duration":0.72428234999999996,"objects_trimmed":0,"snaptrim_duration":0.003131572,"stat_sum":{"num_bytes":8809488,"num_objects":483,"num_object_clones":217,"num_object_copies":966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":483,"num_whiteouts":190,"num_read":63,"num_read_kb":69,"num_write":1091,"num_write_kb":15023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"67'1773","reported_seq":3761,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:46.289749+0000","last_change":"2023-10-30T14:51:46.289749+0000","last_active":"2023-10-30T14:51:46.289749+0000","last_peered":"2023-10-30T14:51:46.289749+0000","last_clean":"2023-10-30T14:51:46.289749+0000","last_became_active":"2023-10-30T14:14:04.644683+0000","last_became_peered":"2023-10-30T14:14:04.644683+0000","last_unstale":"2023-10-30T14:51:46.289749+0000","last_undegraded":"2023-10-30T14:51:46.289749+0000","last_fullsized":"2023-10-30T14:51:46.289749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1772","last_scrub_stamp":"2023-10-30T14:51:46.289574+0000","last_deep_scrub":"67'1772","last_deep_scrub_stamp":"2023-10-30T14:51:46.289574+0000","last_clean_scrub_stamp":"2023-10-30T14:51:46.289574+0000","objects_scrubbed":486,"log_size":1773,"log_dups_size":0,"ondisk_log_size":1773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:44:45.481060+0000","scrub_duration":0.82105794700000001,"objects_trimmed":0,"snaptrim_duration":0.0052462430000000003,"stat_sum":{"num_bytes":8863750,"num_objects":486,"num_object_clones":224,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":199,"num_read":70,"num_read_kb":4116,"num_write":1086,"num_write_kb":10111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'1770","reported_seq":3766,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:16.578543+0000","last_change":"2023-10-30T14:52:16.578648+0000","last_active":"2023-10-30T14:52:16.578543+0000","last_peered":"2023-10-30T14:52:16.578543+0000","last_clean":"2023-10-30T14:52:16.578543+0000","last_became_active":"2023-10-30T14:14:04.619335+0000","last_became_peered":"2023-10-30T14:14:04.619335+0000","last_unstale":"2023-10-30T14:52:16.578543+0000","last_undegraded":"2023-10-30T14:52:16.578543+0000","last_fullsized":"2023-10-30T14:52:16.578543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1770","last_scrub_stamp":"2023-10-30T14:52:16.578428+0000","last_deep_scrub":"67'1770","last_deep_scrub_stamp":"2023-10-30T14:52:16.578428+0000","last_clean_scrub_stamp":"2023-10-30T14:52:16.578428+0000","objects_scrubbed":479,"log_size":1770,"log_dups_size":0,"ondisk_log_size":1770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:43:02.459839+0000","scrub_duration":0.74945407600000002,"objects_trimmed":0,"snaptrim_duration":0.0062283859999999998,"stat_sum":{"num_bytes":20638046,"num_objects":479,"num_object_clones":221,"num_object_copies":958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":195,"num_read":75,"num_read_kb":12254,"num_write":1080,"num_write_kb":22208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"67'1792","reported_seq":3770,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:16.074406+0000","last_change":"2023-10-30T14:52:16.074486+0000","last_active":"2023-10-30T14:52:16.074406+0000","last_peered":"2023-10-30T14:52:16.074406+0000","last_clean":"2023-10-30T14:52:16.074406+0000","last_became_active":"2023-10-30T14:14:04.641323+0000","last_became_peered":"2023-10-30T14:14:04.641323+0000","last_unstale":"2023-10-30T14:52:16.074406+0000","last_undegraded":"2023-10-30T14:52:16.074406+0000","last_fullsized":"2023-10-30T14:52:16.074406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1792","last_scrub_stamp":"2023-10-30T14:52:16.074315+0000","last_deep_scrub":"67'1792","last_deep_scrub_stamp":"2023-10-30T14:52:16.074315+0000","last_clean_scrub_stamp":"2023-10-30T14:52:16.074315+0000","objects_scrubbed":493,"log_size":1792,"log_dups_size":0,"ondisk_log_size":1792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:12:43.677442+0000","scrub_duration":0.89714960899999996,"objects_trimmed":0,"snaptrim_duration":0.011228897,"stat_sum":{"num_bytes":14710085,"num_objects":493,"num_object_clones":224,"num_object_copies":986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":493,"num_whiteouts":202,"num_read":59,"num_read_kb":8203,"num_write":1083,"num_write_kb":18203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'1702","reported_seq":3686,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:54.076987+0000","last_change":"2023-10-30T14:51:54.076987+0000","last_active":"2023-10-30T14:51:54.076987+0000","last_peered":"2023-10-30T14:51:54.076987+0000","last_clean":"2023-10-30T14:51:54.076987+0000","last_became_active":"2023-10-30T14:14:04.618312+0000","last_became_peered":"2023-10-30T14:14:04.618312+0000","last_unstale":"2023-10-30T14:51:54.076987+0000","last_undegraded":"2023-10-30T14:51:54.076987+0000","last_fullsized":"2023-10-30T14:51:54.076987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1701","last_scrub_stamp":"2023-10-30T14:51:54.076855+0000","last_deep_scrub":"67'1701","last_deep_scrub_stamp":"2023-10-30T14:51:54.076855+0000","last_clean_scrub_stamp":"2023-10-30T14:51:54.076855+0000","objects_scrubbed":477,"log_size":1702,"log_dups_size":0,"ondisk_log_size":1702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:59:06.379444+0000","scrub_duration":0.66338523199999999,"objects_trimmed":0,"snaptrim_duration":0.002849426,"stat_sum":{"num_bytes":12938947,"num_objects":477,"num_object_clones":218,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":193,"num_read":60,"num_read_kb":8209,"num_write":1033,"num_write_kb":15209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"67'1860","reported_seq":3836,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:27.055426+0000","last_change":"2023-10-30T14:52:27.055503+0000","last_active":"2023-10-30T14:52:27.055426+0000","last_peered":"2023-10-30T14:52:27.055426+0000","last_clean":"2023-10-30T14:52:27.055426+0000","last_became_active":"2023-10-30T14:14:04.647361+0000","last_became_peered":"2023-10-30T14:14:04.647361+0000","last_unstale":"2023-10-30T14:52:27.055426+0000","last_undegraded":"2023-10-30T14:52:27.055426+0000","last_fullsized":"2023-10-30T14:52:27.055426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1860","last_scrub_stamp":"2023-10-30T14:52:27.055336+0000","last_deep_scrub":"67'1860","last_deep_scrub_stamp":"2023-10-30T14:52:27.055336+0000","last_clean_scrub_stamp":"2023-10-30T14:52:27.055336+0000","objects_scrubbed":513,"log_size":1860,"log_dups_size":0,"ondisk_log_size":1860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:39.292518+0000","scrub_duration":0.88049944599999996,"objects_trimmed":0,"snaptrim_duration":0.010459703000000001,"stat_sum":{"num_bytes":8547707,"num_objects":513,"num_object_clones":237,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":513,"num_whiteouts":209,"num_read":70,"num_read_kb":8267,"num_write":1124,"num_write_kb":13960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"67'1927","reported_seq":3859,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:05.832025+0000","last_change":"2023-10-30T14:52:05.832091+0000","last_active":"2023-10-30T14:52:05.832025+0000","last_peered":"2023-10-30T14:52:05.832025+0000","last_clean":"2023-10-30T14:52:05.832025+0000","last_became_active":"2023-10-30T14:14:04.630959+0000","last_became_peered":"2023-10-30T14:14:04.630959+0000","last_unstale":"2023-10-30T14:52:05.832025+0000","last_undegraded":"2023-10-30T14:52:05.832025+0000","last_fullsized":"2023-10-30T14:52:05.832025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1927","last_scrub_stamp":"2023-10-30T14:52:05.831888+0000","last_deep_scrub":"67'1927","last_deep_scrub_stamp":"2023-10-30T14:52:05.831888+0000","last_clean_scrub_stamp":"2023-10-30T14:52:05.831888+0000","objects_scrubbed":545,"log_size":1927,"log_dups_size":0,"ondisk_log_size":1927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:33:14.719732+0000","scrub_duration":1.5552255019999999,"objects_trimmed":0,"snaptrim_duration":0.0079325130000000004,"stat_sum":{"num_bytes":12417236,"num_objects":545,"num_object_clones":289,"num_object_copies":1090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":545,"num_whiteouts":193,"num_read":76,"num_read_kb":4189,"num_write":1098,"num_write_kb":14000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"67'1825","reported_seq":3797,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:12.042783+0000","last_change":"2023-10-30T14:52:12.042829+0000","last_active":"2023-10-30T14:52:12.042783+0000","last_peered":"2023-10-30T14:52:12.042783+0000","last_clean":"2023-10-30T14:52:12.042783+0000","last_became_active":"2023-10-30T14:14:04.612653+0000","last_became_peered":"2023-10-30T14:14:04.612653+0000","last_unstale":"2023-10-30T14:52:12.042783+0000","last_undegraded":"2023-10-30T14:52:12.042783+0000","last_fullsized":"2023-10-30T14:52:12.042783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1825","last_scrub_stamp":"2023-10-30T14:52:12.042703+0000","last_deep_scrub":"67'1825","last_deep_scrub_stamp":"2023-10-30T14:52:12.042703+0000","last_clean_scrub_stamp":"2023-10-30T14:52:12.042703+0000","objects_scrubbed":522,"log_size":1825,"log_dups_size":0,"ondisk_log_size":1825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:54:08.506135+0000","scrub_duration":0.74874808599999998,"objects_trimmed":0,"snaptrim_duration":0.0079098320000000003,"stat_sum":{"num_bytes":17851951,"num_objects":522,"num_object_clones":235,"num_object_copies":1044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":202,"num_read":63,"num_read_kb":9809,"num_write":1084,"num_write_kb":19222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"67'1788","reported_seq":3771,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:59.430627+0000","last_change":"2023-10-30T14:51:58.995889+0000","last_active":"2023-10-30T14:51:59.430627+0000","last_peered":"2023-10-30T14:51:59.430627+0000","last_clean":"2023-10-30T14:51:59.430627+0000","last_became_active":"2023-10-30T14:14:04.621135+0000","last_became_peered":"2023-10-30T14:14:04.621135+0000","last_unstale":"2023-10-30T14:51:59.430627+0000","last_undegraded":"2023-10-30T14:51:59.430627+0000","last_fullsized":"2023-10-30T14:51:59.430627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1785","last_scrub_stamp":"2023-10-30T14:51:58.995736+0000","last_deep_scrub":"67'1785","last_deep_scrub_stamp":"2023-10-30T14:51:58.995736+0000","last_clean_scrub_stamp":"2023-10-30T14:51:58.995736+0000","objects_scrubbed":506,"log_size":1788,"log_dups_size":0,"ondisk_log_size":1788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:30:07.331904+0000","scrub_duration":0.92494544599999995,"objects_trimmed":0,"snaptrim_duration":0.0025042409999999999,"stat_sum":{"num_bytes":14360375,"num_objects":506,"num_object_clones":227,"num_object_copies":1012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":193,"num_read":67,"num_read_kb":6339,"num_write":1088,"num_write_kb":20012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"67'1759","reported_seq":3769,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:03.476954+0000","last_change":"2023-10-30T14:52:03.477051+0000","last_active":"2023-10-30T14:52:03.476954+0000","last_peered":"2023-10-30T14:52:03.476954+0000","last_clean":"2023-10-30T14:52:03.476954+0000","last_became_active":"2023-10-30T14:14:04.616142+0000","last_became_peered":"2023-10-30T14:14:04.616142+0000","last_unstale":"2023-10-30T14:52:03.476954+0000","last_undegraded":"2023-10-30T14:52:03.476954+0000","last_fullsized":"2023-10-30T14:52:03.476954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1759","last_scrub_stamp":"2023-10-30T14:52:03.476870+0000","last_deep_scrub":"67'1759","last_deep_scrub_stamp":"2023-10-30T14:52:03.476870+0000","last_clean_scrub_stamp":"2023-10-30T14:52:03.476870+0000","objects_scrubbed":486,"log_size":1759,"log_dups_size":0,"ondisk_log_size":1759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:32:02.837333+0000","scrub_duration":1.022971694,"objects_trimmed":0,"snaptrim_duration":0.0049623779999999999,"stat_sum":{"num_bytes":9254692,"num_objects":486,"num_object_clones":220,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":190,"num_read":88,"num_read_kb":8286,"num_write":1073,"num_write_kb":19495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"67'1790","reported_seq":3792,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:15.904032+0000","last_change":"2023-10-30T14:52:15.904089+0000","last_active":"2023-10-30T14:52:15.904032+0000","last_peered":"2023-10-30T14:52:15.904032+0000","last_clean":"2023-10-30T14:52:15.904032+0000","last_became_active":"2023-10-30T14:14:04.588452+0000","last_became_peered":"2023-10-30T14:14:04.588452+0000","last_unstale":"2023-10-30T14:52:15.904032+0000","last_undegraded":"2023-10-30T14:52:15.904032+0000","last_fullsized":"2023-10-30T14:52:15.904032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1790","last_scrub_stamp":"2023-10-30T14:52:15.903959+0000","last_deep_scrub":"67'1790","last_deep_scrub_stamp":"2023-10-30T14:52:15.903959+0000","last_clean_scrub_stamp":"2023-10-30T14:52:15.903959+0000","objects_scrubbed":471,"log_size":1790,"log_dups_size":0,"ondisk_log_size":1790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:36:49.496291+0000","scrub_duration":0.67904057500000004,"objects_trimmed":0,"snaptrim_duration":0.0046351760000000004,"stat_sum":{"num_bytes":12959775,"num_objects":471,"num_object_clones":217,"num_object_copies":942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":471,"num_whiteouts":194,"num_read":76,"num_read_kb":7792,"num_write":1107,"num_write_kb":17259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"67'1812","reported_seq":3804,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:05.304605+0000","last_change":"2023-10-30T14:52:05.304672+0000","last_active":"2023-10-30T14:52:05.304605+0000","last_peered":"2023-10-30T14:52:05.304605+0000","last_clean":"2023-10-30T14:52:05.304605+0000","last_became_active":"2023-10-30T14:14:04.620781+0000","last_became_peered":"2023-10-30T14:14:04.620781+0000","last_unstale":"2023-10-30T14:52:05.304605+0000","last_undegraded":"2023-10-30T14:52:05.304605+0000","last_fullsized":"2023-10-30T14:52:05.304605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1812","last_scrub_stamp":"2023-10-30T14:52:05.304527+0000","last_deep_scrub":"67'1812","last_deep_scrub_stamp":"2023-10-30T14:52:05.304527+0000","last_clean_scrub_stamp":"2023-10-30T14:52:05.304527+0000","objects_scrubbed":489,"log_size":1812,"log_dups_size":0,"ondisk_log_size":1812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:00:19.753698+0000","scrub_duration":0.98796119900000001,"objects_trimmed":0,"snaptrim_duration":0.002316706,"stat_sum":{"num_bytes":21445753,"num_objects":489,"num_object_clones":220,"num_object_copies":978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":196,"num_read":69,"num_read_kb":13296,"num_write":1109,"num_write_kb":27203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"67'1773","reported_seq":3783,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:52.488244+0000","last_change":"2023-10-30T14:51:52.488306+0000","last_active":"2023-10-30T14:51:52.488244+0000","last_peered":"2023-10-30T14:51:52.488244+0000","last_clean":"2023-10-30T14:51:52.488244+0000","last_became_active":"2023-10-30T14:14:04.608190+0000","last_became_peered":"2023-10-30T14:14:04.608190+0000","last_unstale":"2023-10-30T14:51:52.488244+0000","last_undegraded":"2023-10-30T14:51:52.488244+0000","last_fullsized":"2023-10-30T14:51:52.488244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1773","last_scrub_stamp":"2023-10-30T14:51:52.488153+0000","last_deep_scrub":"67'1773","last_deep_scrub_stamp":"2023-10-30T14:51:52.488153+0000","last_clean_scrub_stamp":"2023-10-30T14:51:52.488153+0000","objects_scrubbed":484,"log_size":1773,"log_dups_size":0,"ondisk_log_size":1773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:40.330537+0000","scrub_duration":0.90559973500000002,"objects_trimmed":0,"snaptrim_duration":0.0035618450000000001,"stat_sum":{"num_bytes":31901819,"num_objects":484,"num_object_clones":219,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":197,"num_read":85,"num_read_kb":23305,"num_write":1093,"num_write_kb":37349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"67'1874","reported_seq":3863,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:07.522300+0000","last_change":"2023-10-30T14:52:07.522383+0000","last_active":"2023-10-30T14:52:07.522300+0000","last_peered":"2023-10-30T14:52:07.522300+0000","last_clean":"2023-10-30T14:52:07.522300+0000","last_became_active":"2023-10-30T14:14:04.641306+0000","last_became_peered":"2023-10-30T14:14:04.641306+0000","last_unstale":"2023-10-30T14:52:07.522300+0000","last_undegraded":"2023-10-30T14:52:07.522300+0000","last_fullsized":"2023-10-30T14:52:07.522300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1874","last_scrub_stamp":"2023-10-30T14:52:07.522201+0000","last_deep_scrub":"67'1874","last_deep_scrub_stamp":"2023-10-30T14:52:07.522201+0000","last_clean_scrub_stamp":"2023-10-30T14:52:07.522201+0000","objects_scrubbed":513,"log_size":1874,"log_dups_size":0,"ondisk_log_size":1874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:12.283985+0000","scrub_duration":0.80925856399999996,"objects_trimmed":0,"snaptrim_duration":0.0066189990000000004,"stat_sum":{"num_bytes":4095483,"num_objects":513,"num_object_clones":236,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":513,"num_whiteouts":206,"num_read":80,"num_read_kb":4151,"num_write":1131,"num_write_kb":9615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"67'1656","reported_seq":3664,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:52.398678+0000","last_change":"2023-10-30T14:51:52.398752+0000","last_active":"2023-10-30T14:51:52.398678+0000","last_peered":"2023-10-30T14:51:52.398678+0000","last_clean":"2023-10-30T14:51:52.398678+0000","last_became_active":"2023-10-30T14:14:04.613644+0000","last_became_peered":"2023-10-30T14:14:04.613644+0000","last_unstale":"2023-10-30T14:51:52.398678+0000","last_undegraded":"2023-10-30T14:51:52.398678+0000","last_fullsized":"2023-10-30T14:51:52.398678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1656","last_scrub_stamp":"2023-10-30T14:51:52.398591+0000","last_deep_scrub":"67'1656","last_deep_scrub_stamp":"2023-10-30T14:51:52.398591+0000","last_clean_scrub_stamp":"2023-10-30T14:51:52.398591+0000","objects_scrubbed":445,"log_size":1656,"log_dups_size":0,"ondisk_log_size":1656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:58.033004+0000","scrub_duration":0.76830485800000003,"objects_trimmed":0,"snaptrim_duration":0.0030699740000000001,"stat_sum":{"num_bytes":12521692,"num_objects":445,"num_object_clones":206,"num_object_copies":890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":445,"num_whiteouts":180,"num_read":71,"num_read_kb":8233,"num_write":1044,"num_write_kb":14252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"67'1781","reported_seq":3781,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:40.155073+0000","last_change":"2023-10-30T14:52:40.155151+0000","last_active":"2023-10-30T14:52:40.155073+0000","last_peered":"2023-10-30T14:52:40.155073+0000","last_clean":"2023-10-30T14:52:40.155073+0000","last_became_active":"2023-10-30T14:14:04.649850+0000","last_became_peered":"2023-10-30T14:14:04.649850+0000","last_unstale":"2023-10-30T14:52:40.155073+0000","last_undegraded":"2023-10-30T14:52:40.155073+0000","last_fullsized":"2023-10-30T14:52:40.155073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1781","last_scrub_stamp":"2023-10-30T14:52:40.154949+0000","last_deep_scrub":"67'1781","last_deep_scrub_stamp":"2023-10-30T14:52:40.154949+0000","last_clean_scrub_stamp":"2023-10-30T14:52:40.154949+0000","objects_scrubbed":477,"log_size":1781,"log_dups_size":0,"ondisk_log_size":1781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:39:01.401764+0000","scrub_duration":0.92641234400000005,"objects_trimmed":0,"snaptrim_duration":0.012599246,"stat_sum":{"num_bytes":26075472,"num_objects":477,"num_object_clones":221,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":192,"num_read":77,"num_read_kb":24658,"num_write":1082,"num_write_kb":32086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"67'1861","reported_seq":3825,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:04.219630+0000","last_change":"2023-10-30T14:52:04.219630+0000","last_active":"2023-10-30T14:52:04.219630+0000","last_peered":"2023-10-30T14:52:04.219630+0000","last_clean":"2023-10-30T14:52:04.219630+0000","last_became_active":"2023-10-30T14:14:04.625042+0000","last_became_peered":"2023-10-30T14:14:04.625042+0000","last_unstale":"2023-10-30T14:52:04.219630+0000","last_undegraded":"2023-10-30T14:52:04.219630+0000","last_fullsized":"2023-10-30T14:52:04.219630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1861","last_scrub_stamp":"2023-10-30T14:52:04.219419+0000","last_deep_scrub":"67'1861","last_deep_scrub_stamp":"2023-10-30T14:52:04.219419+0000","last_clean_scrub_stamp":"2023-10-30T14:52:04.219419+0000","objects_scrubbed":514,"log_size":1861,"log_dups_size":0,"ondisk_log_size":1861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:43:26.153599+0000","scrub_duration":0.918695398,"objects_trimmed":0,"snaptrim_duration":0.0082351539999999997,"stat_sum":{"num_bytes":12784426,"num_objects":514,"num_object_clones":242,"num_object_copies":1028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":514,"num_whiteouts":214,"num_read":61,"num_read_kb":10278,"num_write":1107,"num_write_kb":27149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"67'1740","reported_seq":3732,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:28.040279+0000","last_change":"2023-10-30T14:52:28.040339+0000","last_active":"2023-10-30T14:52:28.040279+0000","last_peered":"2023-10-30T14:52:28.040279+0000","last_clean":"2023-10-30T14:52:28.040279+0000","last_became_active":"2023-10-30T14:14:04.633586+0000","last_became_peered":"2023-10-30T14:14:04.633586+0000","last_unstale":"2023-10-30T14:52:28.040279+0000","last_undegraded":"2023-10-30T14:52:28.040279+0000","last_fullsized":"2023-10-30T14:52:28.040279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1740","last_scrub_stamp":"2023-10-30T14:52:28.040176+0000","last_deep_scrub":"67'1740","last_deep_scrub_stamp":"2023-10-30T14:52:28.040176+0000","last_clean_scrub_stamp":"2023-10-30T14:52:28.040176+0000","objects_scrubbed":462,"log_size":1740,"log_dups_size":0,"ondisk_log_size":1740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:47:20.479253+0000","scrub_duration":0.88723452899999999,"objects_trimmed":0,"snaptrim_duration":0.0043616920000000003,"stat_sum":{"num_bytes":12342414,"num_objects":462,"num_object_clones":214,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":462,"num_whiteouts":191,"num_read":63,"num_read_kb":8213,"num_write":1061,"num_write_kb":14018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"67'1897","reported_seq":3899,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:15.835702+0000","last_change":"2023-10-30T14:52:15.835702+0000","last_active":"2023-10-30T14:52:15.835702+0000","last_peered":"2023-10-30T14:52:15.835702+0000","last_clean":"2023-10-30T14:52:15.835702+0000","last_became_active":"2023-10-30T14:14:04.642244+0000","last_became_peered":"2023-10-30T14:14:04.642244+0000","last_unstale":"2023-10-30T14:52:15.835702+0000","last_undegraded":"2023-10-30T14:52:15.835702+0000","last_fullsized":"2023-10-30T14:52:15.835702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1897","last_scrub_stamp":"2023-10-30T14:52:15.835546+0000","last_deep_scrub":"67'1897","last_deep_scrub_stamp":"2023-10-30T14:52:15.835546+0000","last_clean_scrub_stamp":"2023-10-30T14:52:15.835546+0000","objects_scrubbed":515,"log_size":1897,"log_dups_size":0,"ondisk_log_size":1897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:22:58.232388+0000","scrub_duration":1.035771956,"objects_trimmed":0,"snaptrim_duration":0.0065088710000000003,"stat_sum":{"num_bytes":26075380,"num_objects":515,"num_object_clones":234,"num_object_copies":1030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":515,"num_whiteouts":210,"num_read":90,"num_read_kb":18405,"num_write":1137,"num_write_kb":31429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'1775","reported_seq":3785,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:59.437615+0000","last_change":"2023-10-30T14:51:55.288911+0000","last_active":"2023-10-30T14:51:59.437615+0000","last_peered":"2023-10-30T14:51:59.437615+0000","last_clean":"2023-10-30T14:51:59.437615+0000","last_became_active":"2023-10-30T14:14:04.620343+0000","last_became_peered":"2023-10-30T14:14:04.620343+0000","last_unstale":"2023-10-30T14:51:59.437615+0000","last_undegraded":"2023-10-30T14:51:59.437615+0000","last_fullsized":"2023-10-30T14:51:59.437615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1774","last_scrub_stamp":"2023-10-30T14:51:55.288738+0000","last_deep_scrub":"67'1774","last_deep_scrub_stamp":"2023-10-30T14:51:55.288738+0000","last_clean_scrub_stamp":"2023-10-30T14:51:55.288738+0000","objects_scrubbed":484,"log_size":1775,"log_dups_size":0,"ondisk_log_size":1775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:15:30.753588+0000","scrub_duration":0.87706562700000001,"objects_trimmed":0,"snaptrim_duration":0.0048974149999999996,"stat_sum":{"num_bytes":12481557,"num_objects":484,"num_object_clones":220,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":199,"num_read":86,"num_read_kb":8232,"num_write":1087,"num_write_kb":18010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'1871","reported_seq":3844,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:49.871411+0000","last_change":"2023-10-30T14:51:49.871411+0000","last_active":"2023-10-30T14:51:49.871411+0000","last_peered":"2023-10-30T14:51:49.871411+0000","last_clean":"2023-10-30T14:51:49.871411+0000","last_became_active":"2023-10-30T14:14:04.609720+0000","last_became_peered":"2023-10-30T14:14:04.609720+0000","last_unstale":"2023-10-30T14:51:49.871411+0000","last_undegraded":"2023-10-30T14:51:49.871411+0000","last_fullsized":"2023-10-30T14:51:49.871411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1870","last_scrub_stamp":"2023-10-30T14:51:49.871181+0000","last_deep_scrub":"67'1870","last_deep_scrub_stamp":"2023-10-30T14:51:49.871181+0000","last_clean_scrub_stamp":"2023-10-30T14:51:49.871181+0000","objects_scrubbed":535,"log_size":1871,"log_dups_size":0,"ondisk_log_size":1871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:47:19.781621+0000","scrub_duration":1.1862301479999999,"objects_trimmed":0,"snaptrim_duration":0.0040008739999999997,"stat_sum":{"num_bytes":31023805,"num_objects":535,"num_object_clones":248,"num_object_copies":1070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":535,"num_whiteouts":229,"num_read":75,"num_read_kb":20512,"num_write":1101,"num_write_kb":32005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"67'1673","reported_seq":3679,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:07.073416+0000","last_change":"2023-10-30T14:52:07.073470+0000","last_active":"2023-10-30T14:52:07.073416+0000","last_peered":"2023-10-30T14:52:07.073416+0000","last_clean":"2023-10-30T14:52:07.073416+0000","last_became_active":"2023-10-30T14:14:04.620217+0000","last_became_peered":"2023-10-30T14:14:04.620217+0000","last_unstale":"2023-10-30T14:52:07.073416+0000","last_undegraded":"2023-10-30T14:52:07.073416+0000","last_fullsized":"2023-10-30T14:52:07.073416+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1673","last_scrub_stamp":"2023-10-30T14:52:07.073343+0000","last_deep_scrub":"67'1673","last_deep_scrub_stamp":"2023-10-30T14:52:07.073343+0000","last_clean_scrub_stamp":"2023-10-30T14:52:07.073343+0000","objects_scrubbed":455,"log_size":1673,"log_dups_size":0,"ondisk_log_size":1673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:59:35.579646+0000","scrub_duration":0.660174924,"objects_trimmed":0,"snaptrim_duration":0.0044491130000000002,"stat_sum":{"num_bytes":20631584,"num_objects":455,"num_object_clones":209,"num_object_copies":910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":455,"num_whiteouts":179,"num_read":71,"num_read_kb":15956,"num_write":1007,"num_write_kb":22437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"67'1841","reported_seq":3852,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:16.109899+0000","last_change":"2023-10-30T14:52:16.109981+0000","last_active":"2023-10-30T14:52:16.109899+0000","last_peered":"2023-10-30T14:52:16.109899+0000","last_clean":"2023-10-30T14:52:16.109899+0000","last_became_active":"2023-10-30T14:14:45.669878+0000","last_became_peered":"2023-10-30T14:14:45.669878+0000","last_unstale":"2023-10-30T14:52:16.109899+0000","last_undegraded":"2023-10-30T14:52:16.109899+0000","last_fullsized":"2023-10-30T14:52:16.109899+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1841","last_scrub_stamp":"2023-10-30T14:52:16.109813+0000","last_deep_scrub":"67'1841","last_deep_scrub_stamp":"2023-10-30T14:52:16.109813+0000","last_clean_scrub_stamp":"2023-10-30T14:52:16.109813+0000","objects_scrubbed":479,"log_size":1841,"log_dups_size":0,"ondisk_log_size":1841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:33:56.104550+0000","scrub_duration":0.80770848900000003,"objects_trimmed":0,"snaptrim_duration":0.0077997409999999998,"stat_sum":{"num_bytes":25414870,"num_objects":479,"num_object_clones":220,"num_object_copies":958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":194,"num_read":80,"num_read_kb":18178,"num_write":1147,"num_write_kb":27446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"67'1799","reported_seq":3794,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:05.950875+0000","last_change":"2023-10-30T14:52:05.950967+0000","last_active":"2023-10-30T14:52:05.950875+0000","last_peered":"2023-10-30T14:52:05.950875+0000","last_clean":"2023-10-30T14:52:05.950875+0000","last_became_active":"2023-10-30T14:14:04.597590+0000","last_became_peered":"2023-10-30T14:14:04.597590+0000","last_unstale":"2023-10-30T14:52:05.950875+0000","last_undegraded":"2023-10-30T14:52:05.950875+0000","last_fullsized":"2023-10-30T14:52:05.950875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1799","last_scrub_stamp":"2023-10-30T14:52:05.950769+0000","last_deep_scrub":"67'1799","last_deep_scrub_stamp":"2023-10-30T14:52:05.950769+0000","last_clean_scrub_stamp":"2023-10-30T14:52:05.950769+0000","objects_scrubbed":485,"log_size":1799,"log_dups_size":0,"ondisk_log_size":1799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:54.947034+0000","scrub_duration":0.61979928200000001,"objects_trimmed":0,"snaptrim_duration":0.0044830970000000001,"stat_sum":{"num_bytes":8769781,"num_objects":485,"num_object_clones":224,"num_object_copies":970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":197,"num_read":75,"num_read_kb":4156,"num_write":1100,"num_write_kb":11840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"67'1646","reported_seq":3633,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:02.338544+0000","last_change":"2023-10-30T14:52:02.338628+0000","last_active":"2023-10-30T14:52:02.338544+0000","last_peered":"2023-10-30T14:52:02.338544+0000","last_clean":"2023-10-30T14:52:02.338544+0000","last_became_active":"2023-10-30T14:14:04.623902+0000","last_became_peered":"2023-10-30T14:14:04.623902+0000","last_unstale":"2023-10-30T14:52:02.338544+0000","last_undegraded":"2023-10-30T14:52:02.338544+0000","last_fullsized":"2023-10-30T14:52:02.338544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1646","last_scrub_stamp":"2023-10-30T14:52:02.338437+0000","last_deep_scrub":"67'1646","last_deep_scrub_stamp":"2023-10-30T14:52:02.338437+0000","last_clean_scrub_stamp":"2023-10-30T14:52:02.338437+0000","objects_scrubbed":460,"log_size":1646,"log_dups_size":0,"ondisk_log_size":1646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:25:36.251158+0000","scrub_duration":0.90177968600000002,"objects_trimmed":0,"snaptrim_duration":0.0051343489999999999,"stat_sum":{"num_bytes":21110293,"num_objects":460,"num_object_clones":215,"num_object_copies":920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":188,"num_read":58,"num_read_kb":12498,"num_write":973,"num_write_kb":25426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"67'1857","reported_seq":3836,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:00.215600+0000","last_change":"2023-10-30T14:52:00.215683+0000","last_active":"2023-10-30T14:52:00.215600+0000","last_peered":"2023-10-30T14:52:00.215600+0000","last_clean":"2023-10-30T14:52:00.215600+0000","last_became_active":"2023-10-30T14:14:04.597556+0000","last_became_peered":"2023-10-30T14:14:04.597556+0000","last_unstale":"2023-10-30T14:52:00.215600+0000","last_undegraded":"2023-10-30T14:52:00.215600+0000","last_fullsized":"2023-10-30T14:52:00.215600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1857","last_scrub_stamp":"2023-10-30T14:52:00.215512+0000","last_deep_scrub":"67'1857","last_deep_scrub_stamp":"2023-10-30T14:52:00.215512+0000","last_clean_scrub_stamp":"2023-10-30T14:52:00.215512+0000","objects_scrubbed":524,"log_size":1857,"log_dups_size":0,"ondisk_log_size":1857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:49:29.093253+0000","scrub_duration":0.81083191899999996,"objects_trimmed":0,"snaptrim_duration":0.0040069729999999996,"stat_sum":{"num_bytes":25689725,"num_objects":524,"num_object_clones":241,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":212,"num_read":73,"num_read_kb":17618,"num_write":1103,"num_write_kb":27397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"67'1805","reported_seq":3800,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:08.853004+0000","last_change":"2023-10-30T14:52:08.853100+0000","last_active":"2023-10-30T14:52:08.853004+0000","last_peered":"2023-10-30T14:52:08.853004+0000","last_clean":"2023-10-30T14:52:08.853004+0000","last_became_active":"2023-10-30T14:14:04.618430+0000","last_became_peered":"2023-10-30T14:14:04.618430+0000","last_unstale":"2023-10-30T14:52:08.853004+0000","last_undegraded":"2023-10-30T14:52:08.853004+0000","last_fullsized":"2023-10-30T14:52:08.853004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1805","last_scrub_stamp":"2023-10-30T14:52:08.852862+0000","last_deep_scrub":"67'1805","last_deep_scrub_stamp":"2023-10-30T14:52:08.852862+0000","last_clean_scrub_stamp":"2023-10-30T14:52:08.852862+0000","objects_scrubbed":481,"log_size":1805,"log_dups_size":0,"ondisk_log_size":1805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:03:36.648223+0000","scrub_duration":1.0990789080000001,"objects_trimmed":0,"snaptrim_duration":0.0074984149999999996,"stat_sum":{"num_bytes":16403179,"num_objects":481,"num_object_clones":225,"num_object_copies":962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":199,"num_read":75,"num_read_kb":12323,"num_write":1105,"num_write_kb":17671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"67'1948","reported_seq":3932,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:55.392487+0000","last_change":"2023-10-30T14:51:55.392537+0000","last_active":"2023-10-30T14:51:55.392487+0000","last_peered":"2023-10-30T14:51:55.392487+0000","last_clean":"2023-10-30T14:51:55.392487+0000","last_became_active":"2023-10-30T14:14:04.606340+0000","last_became_peered":"2023-10-30T14:14:04.606340+0000","last_unstale":"2023-10-30T14:51:55.392487+0000","last_undegraded":"2023-10-30T14:51:55.392487+0000","last_fullsized":"2023-10-30T14:51:55.392487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1948","last_scrub_stamp":"2023-10-30T14:51:55.392413+0000","last_deep_scrub":"67'1948","last_deep_scrub_stamp":"2023-10-30T14:51:55.392413+0000","last_clean_scrub_stamp":"2023-10-30T14:51:55.392413+0000","objects_scrubbed":534,"log_size":1948,"log_dups_size":0,"ondisk_log_size":1948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:02:49.595530+0000","scrub_duration":0.84988978299999995,"objects_trimmed":0,"snaptrim_duration":0.0035094890000000002,"stat_sum":{"num_bytes":22361827,"num_objects":534,"num_object_clones":245,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":534,"num_whiteouts":212,"num_read":82,"num_read_kb":9614,"num_write":1202,"num_write_kb":32875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"67'1825","reported_seq":3829,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:03.419633+0000","last_change":"2023-10-30T14:52:03.419758+0000","last_active":"2023-10-30T14:52:03.419633+0000","last_peered":"2023-10-30T14:52:03.419633+0000","last_clean":"2023-10-30T14:52:03.419633+0000","last_became_active":"2023-10-30T14:14:04.612588+0000","last_became_peered":"2023-10-30T14:14:04.612588+0000","last_unstale":"2023-10-30T14:52:03.419633+0000","last_undegraded":"2023-10-30T14:52:03.419633+0000","last_fullsized":"2023-10-30T14:52:03.419633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1825","last_scrub_stamp":"2023-10-30T14:52:03.419531+0000","last_deep_scrub":"67'1825","last_deep_scrub_stamp":"2023-10-30T14:52:03.419531+0000","last_clean_scrub_stamp":"2023-10-30T14:52:03.419531+0000","objects_scrubbed":484,"log_size":1825,"log_dups_size":0,"ondisk_log_size":1825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:07:15.268230+0000","scrub_duration":1.1307163710000001,"objects_trimmed":0,"snaptrim_duration":0.0084570800000000005,"stat_sum":{"num_bytes":13290865,"num_objects":484,"num_object_clones":224,"num_object_copies":968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":198,"num_read":83,"num_read_kb":8248,"num_write":1125,"num_write_kb":15099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"67'1743","reported_seq":3738,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:52.927568+0000","last_change":"2023-10-30T14:51:52.927568+0000","last_active":"2023-10-30T14:51:52.927568+0000","last_peered":"2023-10-30T14:51:52.927568+0000","last_clean":"2023-10-30T14:51:52.927568+0000","last_became_active":"2023-10-30T14:14:04.607683+0000","last_became_peered":"2023-10-30T14:14:04.607683+0000","last_unstale":"2023-10-30T14:51:52.927568+0000","last_undegraded":"2023-10-30T14:51:52.927568+0000","last_fullsized":"2023-10-30T14:51:52.927568+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1742","last_scrub_stamp":"2023-10-30T14:51:52.927213+0000","last_deep_scrub":"67'1742","last_deep_scrub_stamp":"2023-10-30T14:51:52.927213+0000","last_clean_scrub_stamp":"2023-10-30T14:51:52.927213+0000","objects_scrubbed":481,"log_size":1743,"log_dups_size":0,"ondisk_log_size":1743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:57:18.481646+0000","scrub_duration":0.84554383499999997,"objects_trimmed":0,"snaptrim_duration":0.001856688,"stat_sum":{"num_bytes":22393964,"num_objects":481,"num_object_clones":222,"num_object_copies":962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":201,"num_read":74,"num_read_kb":12305,"num_write":1055,"num_write_kb":25643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'710","reported_seq":889,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:54.430131+0000","last_change":"2023-10-30T14:51:46.684252+0000","last_active":"2023-10-30T14:51:54.430131+0000","last_peered":"2023-10-30T14:51:54.430131+0000","last_clean":"2023-10-30T14:51:54.430131+0000","last_became_active":"2023-10-30T14:14:03.731740+0000","last_became_peered":"2023-10-30T14:14:03.731740+0000","last_unstale":"2023-10-30T14:51:54.430131+0000","last_undegraded":"2023-10-30T14:51:54.430131+0000","last_fullsized":"2023-10-30T14:51:54.430131+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'706","last_scrub_stamp":"2023-10-30T14:51:46.684108+0000","last_deep_scrub":"67'706","last_deep_scrub_stamp":"2023-10-30T14:51:46.684108+0000","last_clean_scrub_stamp":"2023-10-30T14:51:46.684108+0000","objects_scrubbed":43,"log_size":710,"log_dups_size":0,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:53:22.342613+0000","scrub_duration":0.17419201400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50568158,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":149,"num_read_kb":143937,"num_write":707,"num_write_kb":74908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":402982,"num_omap_keys":749,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"67'1820","reported_seq":3816,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:11.899370+0000","last_change":"2023-10-30T14:52:11.899417+0000","last_active":"2023-10-30T14:52:11.899370+0000","last_peered":"2023-10-30T14:52:11.899370+0000","last_clean":"2023-10-30T14:52:11.899370+0000","last_became_active":"2023-10-30T14:14:04.621526+0000","last_became_peered":"2023-10-30T14:14:04.621526+0000","last_unstale":"2023-10-30T14:52:11.899370+0000","last_undegraded":"2023-10-30T14:52:11.899370+0000","last_fullsized":"2023-10-30T14:52:11.899370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1820","last_scrub_stamp":"2023-10-30T14:52:11.899293+0000","last_deep_scrub":"67'1820","last_deep_scrub_stamp":"2023-10-30T14:52:11.899293+0000","last_clean_scrub_stamp":"2023-10-30T14:52:11.899293+0000","objects_scrubbed":495,"log_size":1820,"log_dups_size":0,"ondisk_log_size":1820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:14:22.312179+0000","scrub_duration":0.63139072299999999,"objects_trimmed":0,"snaptrim_duration":0.0047814620000000002,"stat_sum":{"num_bytes":12429609,"num_objects":495,"num_object_clones":226,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":188,"num_read":79,"num_read_kb":8245,"num_write":1126,"num_write_kb":15289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"67'1505","reported_seq":1996,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:43.210550+0000","last_change":"2023-10-30T14:51:43.210666+0000","last_active":"2023-10-30T14:51:43.210550+0000","last_peered":"2023-10-30T14:51:43.210550+0000","last_clean":"2023-10-30T14:51:43.210550+0000","last_became_active":"2023-10-30T14:14:03.729386+0000","last_became_peered":"2023-10-30T14:14:03.729386+0000","last_unstale":"2023-10-30T14:51:43.210550+0000","last_undegraded":"2023-10-30T14:51:43.210550+0000","last_fullsized":"2023-10-30T14:51:43.210550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1505","last_scrub_stamp":"2023-10-30T14:51:43.210412+0000","last_deep_scrub":"67'1505","last_deep_scrub_stamp":"2023-10-30T14:51:43.210412+0000","last_clean_scrub_stamp":"2023-10-30T14:51:43.210412+0000","objects_scrubbed":46,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:18:14.848623+0000","scrub_duration":0.17059465400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29761637,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":457,"num_read_kb":164028,"num_write":1504,"num_write_kb":71675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":396614,"num_omap_keys":698,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'787","reported_seq":1361,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:45.617219+0000","last_change":"2023-10-30T14:51:45.617294+0000","last_active":"2023-10-30T14:51:45.617219+0000","last_peered":"2023-10-30T14:51:45.617219+0000","last_clean":"2023-10-30T14:51:45.617219+0000","last_became_active":"2023-10-30T14:14:03.724078+0000","last_became_peered":"2023-10-30T14:14:03.724078+0000","last_unstale":"2023-10-30T14:51:45.617219+0000","last_undegraded":"2023-10-30T14:51:45.617219+0000","last_fullsized":"2023-10-30T14:51:45.617219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'787","last_scrub_stamp":"2023-10-30T14:51:45.617096+0000","last_deep_scrub":"67'787","last_deep_scrub_stamp":"2023-10-30T14:51:45.617096+0000","last_clean_scrub_stamp":"2023-10-30T14:51:45.617096+0000","objects_scrubbed":42,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:36:59.792994+0000","scrub_duration":0.123030087,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30316903,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":547,"num_read_kb":63175,"num_write":794,"num_write_kb":47477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":452186,"num_omap_keys":609,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"67'1967","reported_seq":3929,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:59.432213+0000","last_change":"2023-10-30T14:51:50.284976+0000","last_active":"2023-10-30T14:51:59.432213+0000","last_peered":"2023-10-30T14:51:59.432213+0000","last_clean":"2023-10-30T14:51:59.432213+0000","last_became_active":"2023-10-30T14:14:04.597099+0000","last_became_peered":"2023-10-30T14:14:04.597099+0000","last_unstale":"2023-10-30T14:51:59.432213+0000","last_undegraded":"2023-10-30T14:51:59.432213+0000","last_fullsized":"2023-10-30T14:51:59.432213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1965","last_scrub_stamp":"2023-10-30T14:51:50.284800+0000","last_deep_scrub":"67'1965","last_deep_scrub_stamp":"2023-10-30T14:51:50.284800+0000","last_clean_scrub_stamp":"2023-10-30T14:51:50.284800+0000","objects_scrubbed":552,"log_size":1967,"log_dups_size":0,"ondisk_log_size":1967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:16:10.415430+0000","scrub_duration":0.75260475800000004,"objects_trimmed":0,"snaptrim_duration":0.0033627510000000002,"stat_sum":{"num_bytes":15535325,"num_objects":552,"num_object_clones":252,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":552,"num_whiteouts":225,"num_read":69,"num_read_kb":8270,"num_write":1198,"num_write_kb":25187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"67'1603","reported_seq":1776,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:41.944331+0000","last_change":"2023-10-30T14:51:41.944482+0000","last_active":"2023-10-30T14:51:41.944331+0000","last_peered":"2023-10-30T14:51:41.944331+0000","last_clean":"2023-10-30T14:51:41.944331+0000","last_became_active":"2023-10-30T14:14:03.719816+0000","last_became_peered":"2023-10-30T14:14:03.719816+0000","last_unstale":"2023-10-30T14:51:41.944331+0000","last_undegraded":"2023-10-30T14:51:41.944331+0000","last_fullsized":"2023-10-30T14:51:41.944331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1603","last_scrub_stamp":"2023-10-30T14:51:41.944113+0000","last_deep_scrub":"67'1603","last_deep_scrub_stamp":"2023-10-30T14:51:41.944113+0000","last_clean_scrub_stamp":"2023-10-30T14:51:41.944113+0000","objects_scrubbed":49,"log_size":1603,"log_dups_size":0,"ondisk_log_size":1603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:29.585178+0000","scrub_duration":0.217796778,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35358900,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":164,"num_read_kb":135701,"num_write":1593,"num_write_kb":77235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":466020,"num_omap_keys":829,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"67'1773","reported_seq":3766,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:18.838161+0000","last_change":"2023-10-30T14:52:18.838247+0000","last_active":"2023-10-30T14:52:18.838161+0000","last_peered":"2023-10-30T14:52:18.838161+0000","last_clean":"2023-10-30T14:52:18.838161+0000","last_became_active":"2023-10-30T14:14:04.599323+0000","last_became_peered":"2023-10-30T14:14:04.599323+0000","last_unstale":"2023-10-30T14:52:18.838161+0000","last_undegraded":"2023-10-30T14:52:18.838161+0000","last_fullsized":"2023-10-30T14:52:18.838161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1773","last_scrub_stamp":"2023-10-30T14:52:18.838073+0000","last_deep_scrub":"67'1773","last_deep_scrub_stamp":"2023-10-30T14:52:18.838073+0000","last_clean_scrub_stamp":"2023-10-30T14:52:18.838073+0000","objects_scrubbed":482,"log_size":1773,"log_dups_size":0,"ondisk_log_size":1773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:14.205947+0000","scrub_duration":0.586753258,"objects_trimmed":0,"snaptrim_duration":0.0025427000000000002,"stat_sum":{"num_bytes":25885700,"num_objects":482,"num_object_clones":224,"num_object_copies":964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":193,"num_read":75,"num_read_kb":21385,"num_write":1058,"num_write_kb":37196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"67'660","reported_seq":797,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:46.206639+0000","last_change":"2023-10-30T14:51:46.206708+0000","last_active":"2023-10-30T14:51:46.206639+0000","last_peered":"2023-10-30T14:51:46.206639+0000","last_clean":"2023-10-30T14:51:46.206639+0000","last_became_active":"2023-10-30T14:14:03.738132+0000","last_became_peered":"2023-10-30T14:14:03.738132+0000","last_unstale":"2023-10-30T14:51:46.206639+0000","last_undegraded":"2023-10-30T14:51:46.206639+0000","last_fullsized":"2023-10-30T14:51:46.206639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'660","last_scrub_stamp":"2023-10-30T14:51:46.206546+0000","last_deep_scrub":"67'660","last_deep_scrub_stamp":"2023-10-30T14:51:46.206546+0000","last_clean_scrub_stamp":"2023-10-30T14:51:46.206546+0000","objects_scrubbed":43,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:02:40.190575+0000","scrub_duration":0.151344961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360218,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":99,"num_read_kb":53823,"num_write":691,"num_write_kb":33991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":515776,"num_omap_keys":890,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"67'1770","reported_seq":3774,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:18.734930+0000","last_change":"2023-10-30T14:52:18.734997+0000","last_active":"2023-10-30T14:52:18.734930+0000","last_peered":"2023-10-30T14:52:18.734930+0000","last_clean":"2023-10-30T14:52:18.734930+0000","last_became_active":"2023-10-30T14:14:04.648816+0000","last_became_peered":"2023-10-30T14:14:04.648816+0000","last_unstale":"2023-10-30T14:52:18.734930+0000","last_undegraded":"2023-10-30T14:52:18.734930+0000","last_fullsized":"2023-10-30T14:52:18.734930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1770","last_scrub_stamp":"2023-10-30T14:52:18.734849+0000","last_deep_scrub":"67'1770","last_deep_scrub_stamp":"2023-10-30T14:52:18.734849+0000","last_clean_scrub_stamp":"2023-10-30T14:52:18.734849+0000","objects_scrubbed":462,"log_size":1770,"log_dups_size":0,"ondisk_log_size":1770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:58:28.197143+0000","scrub_duration":0.901014129,"objects_trimmed":0,"snaptrim_duration":0.0069848840000000002,"stat_sum":{"num_bytes":12168759,"num_objects":462,"num_object_clones":214,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":462,"num_whiteouts":185,"num_read":77,"num_read_kb":8211,"num_write":1090,"num_write_kb":14663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"67'1070","reported_seq":1217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:47.771730+0000","last_change":"2023-10-30T14:51:47.771870+0000","last_active":"2023-10-30T14:51:47.771730+0000","last_peered":"2023-10-30T14:51:47.771730+0000","last_clean":"2023-10-30T14:51:47.771730+0000","last_became_active":"2023-10-30T14:14:03.725272+0000","last_became_peered":"2023-10-30T14:14:03.725272+0000","last_unstale":"2023-10-30T14:51:47.771730+0000","last_undegraded":"2023-10-30T14:51:47.771730+0000","last_fullsized":"2023-10-30T14:51:47.771730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1070","last_scrub_stamp":"2023-10-30T14:51:47.771555+0000","last_deep_scrub":"67'1070","last_deep_scrub_stamp":"2023-10-30T14:51:47.771555+0000","last_clean_scrub_stamp":"2023-10-30T14:51:47.771555+0000","objects_scrubbed":41,"log_size":1070,"log_dups_size":0,"ondisk_log_size":1070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:13:17.687697+0000","scrub_duration":0.128806527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":123,"num_read_kb":86587,"num_write":1104,"num_write_kb":54796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":577700,"num_omap_keys":1131,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"67'1952","reported_seq":3902,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:04.197699+0000","last_change":"2023-10-30T14:52:04.197786+0000","last_active":"2023-10-30T14:52:04.197699+0000","last_peered":"2023-10-30T14:52:04.197699+0000","last_clean":"2023-10-30T14:52:04.197699+0000","last_became_active":"2023-10-30T14:14:04.609697+0000","last_became_peered":"2023-10-30T14:14:04.609697+0000","last_unstale":"2023-10-30T14:52:04.197699+0000","last_undegraded":"2023-10-30T14:52:04.197699+0000","last_fullsized":"2023-10-30T14:52:04.197699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1952","last_scrub_stamp":"2023-10-30T14:52:04.197582+0000","last_deep_scrub":"67'1952","last_deep_scrub_stamp":"2023-10-30T14:52:04.197582+0000","last_clean_scrub_stamp":"2023-10-30T14:52:04.197582+0000","objects_scrubbed":557,"log_size":1952,"log_dups_size":0,"ondisk_log_size":1952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:29:33.135073+0000","scrub_duration":0.89282557900000004,"objects_trimmed":0,"snaptrim_duration":0.0043395700000000001,"stat_sum":{"num_bytes":12484119,"num_objects":557,"num_object_clones":263,"num_object_copies":1114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":557,"num_whiteouts":236,"num_read":68,"num_read_kb":4261,"num_write":1150,"num_write_kb":13862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"67'1026","reported_seq":1197,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:50.429870+0000","last_change":"2023-10-30T14:51:50.429966+0000","last_active":"2023-10-30T14:51:50.429870+0000","last_peered":"2023-10-30T14:51:50.429870+0000","last_clean":"2023-10-30T14:51:50.429870+0000","last_became_active":"2023-10-30T14:14:03.727422+0000","last_became_peered":"2023-10-30T14:14:03.727422+0000","last_unstale":"2023-10-30T14:51:50.429870+0000","last_undegraded":"2023-10-30T14:51:50.429870+0000","last_fullsized":"2023-10-30T14:51:50.429870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1026","last_scrub_stamp":"2023-10-30T14:51:50.429770+0000","last_deep_scrub":"67'1026","last_deep_scrub_stamp":"2023-10-30T14:51:50.429770+0000","last_clean_scrub_stamp":"2023-10-30T14:51:50.429770+0000","objects_scrubbed":43,"log_size":1026,"log_dups_size":0,"ondisk_log_size":1026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:53.510519+0000","scrub_duration":0.107061642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389280,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":161,"num_read_kb":61498,"num_write":1088,"num_write_kb":30346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":619408,"num_omap_keys":1098,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'2082","reported_seq":3988,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:54.238386+0000","last_change":"2023-10-30T14:51:54.238386+0000","last_active":"2023-10-30T14:51:54.238386+0000","last_peered":"2023-10-30T14:51:54.238386+0000","last_clean":"2023-10-30T14:51:54.238386+0000","last_became_active":"2023-10-30T14:14:04.645333+0000","last_became_peered":"2023-10-30T14:14:04.645333+0000","last_unstale":"2023-10-30T14:51:54.238386+0000","last_undegraded":"2023-10-30T14:51:54.238386+0000","last_fullsized":"2023-10-30T14:51:54.238386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2081","last_scrub_stamp":"2023-10-30T14:51:54.238230+0000","last_deep_scrub":"67'2081","last_deep_scrub_stamp":"2023-10-30T14:51:54.238230+0000","last_clean_scrub_stamp":"2023-10-30T14:51:54.238230+0000","objects_scrubbed":601,"log_size":2082,"log_dups_size":0,"ondisk_log_size":2082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:44.996187+0000","scrub_duration":1.177356566,"objects_trimmed":0,"snaptrim_duration":0.002301474,"stat_sum":{"num_bytes":21718535,"num_objects":601,"num_object_clones":320,"num_object_copies":1202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":601,"num_whiteouts":225,"num_read":78,"num_read_kb":5632,"num_write":1157,"num_write_kb":28163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"67'606","reported_seq":804,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:51.267795+0000","last_change":"2023-10-30T14:51:51.267795+0000","last_active":"2023-10-30T14:51:51.267795+0000","last_peered":"2023-10-30T14:51:51.267795+0000","last_clean":"2023-10-30T14:51:51.267795+0000","last_became_active":"2023-10-30T14:14:03.724806+0000","last_became_peered":"2023-10-30T14:14:03.724806+0000","last_unstale":"2023-10-30T14:51:51.267795+0000","last_undegraded":"2023-10-30T14:51:51.267795+0000","last_fullsized":"2023-10-30T14:51:51.267795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'605","last_scrub_stamp":"2023-10-30T14:51:51.267479+0000","last_deep_scrub":"67'605","last_deep_scrub_stamp":"2023-10-30T14:51:51.267479+0000","last_clean_scrub_stamp":"2023-10-30T14:51:51.267479+0000","objects_scrubbed":50,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:02:26.648751+0000","scrub_duration":0.15457963999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360800,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":192,"num_read_kb":131840,"num_write":659,"num_write_kb":54410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":552351,"num_omap_keys":1032,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"67'1834","reported_seq":3808,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:20.314142+0000","last_change":"2023-10-30T14:52:20.314262+0000","last_active":"2023-10-30T14:52:20.314142+0000","last_peered":"2023-10-30T14:52:20.314142+0000","last_clean":"2023-10-30T14:52:20.314142+0000","last_became_active":"2023-10-30T14:14:04.646543+0000","last_became_peered":"2023-10-30T14:14:04.646543+0000","last_unstale":"2023-10-30T14:52:20.314142+0000","last_undegraded":"2023-10-30T14:52:20.314142+0000","last_fullsized":"2023-10-30T14:52:20.314142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1834","last_scrub_stamp":"2023-10-30T14:52:20.313970+0000","last_deep_scrub":"67'1834","last_deep_scrub_stamp":"2023-10-30T14:52:20.313970+0000","last_clean_scrub_stamp":"2023-10-30T14:52:20.313970+0000","objects_scrubbed":506,"log_size":1834,"log_dups_size":0,"ondisk_log_size":1834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:00:44.288290+0000","scrub_duration":1.1382562679999999,"objects_trimmed":0,"snaptrim_duration":0.00744474,"stat_sum":{"num_bytes":22194385,"num_objects":506,"num_object_clones":234,"num_object_copies":1012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":206,"num_read":65,"num_read_kb":16406,"num_write":1098,"num_write_kb":24097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"67'705","reported_seq":891,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:50.491455+0000","last_change":"2023-10-30T14:51:50.491497+0000","last_active":"2023-10-30T14:51:50.491455+0000","last_peered":"2023-10-30T14:51:50.491455+0000","last_clean":"2023-10-30T14:51:50.491455+0000","last_became_active":"2023-10-30T14:14:03.716941+0000","last_became_peered":"2023-10-30T14:14:03.716941+0000","last_unstale":"2023-10-30T14:51:50.491455+0000","last_undegraded":"2023-10-30T14:51:50.491455+0000","last_fullsized":"2023-10-30T14:51:50.491455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'705","last_scrub_stamp":"2023-10-30T14:51:50.491382+0000","last_deep_scrub":"67'705","last_deep_scrub_stamp":"2023-10-30T14:51:50.491382+0000","last_clean_scrub_stamp":"2023-10-30T14:51:50.491382+0000","objects_scrubbed":50,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:52:06.985963+0000","scrub_duration":0.118685916,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777250,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":201,"num_read_kb":82705,"num_write":762,"num_write_kb":38720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":647476,"num_omap_keys":1125,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"67'1949","reported_seq":3916,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:59.435052+0000","last_change":"2023-10-30T14:51:57.741625+0000","last_active":"2023-10-30T14:51:59.435052+0000","last_peered":"2023-10-30T14:51:59.435052+0000","last_clean":"2023-10-30T14:51:59.435052+0000","last_became_active":"2023-10-30T14:14:04.641494+0000","last_became_peered":"2023-10-30T14:14:04.641494+0000","last_unstale":"2023-10-30T14:51:59.435052+0000","last_undegraded":"2023-10-30T14:51:59.435052+0000","last_fullsized":"2023-10-30T14:51:59.435052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1947","last_scrub_stamp":"2023-10-30T14:51:57.741484+0000","last_deep_scrub":"67'1947","last_deep_scrub_stamp":"2023-10-30T14:51:57.741484+0000","last_clean_scrub_stamp":"2023-10-30T14:51:57.741484+0000","objects_scrubbed":539,"log_size":1949,"log_dups_size":0,"ondisk_log_size":1949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:12:05.541916+0000","scrub_duration":1.29683427,"objects_trimmed":0,"snaptrim_duration":0.0039664019999999999,"stat_sum":{"num_bytes":26425057,"num_objects":539,"num_object_clones":248,"num_object_copies":1078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":207,"num_read":69,"num_read_kb":14198,"num_write":1186,"num_write_kb":36722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"67'327","reported_seq":458,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:44.816766+0000","last_change":"2023-10-30T14:51:44.816862+0000","last_active":"2023-10-30T14:51:44.816766+0000","last_peered":"2023-10-30T14:51:44.816766+0000","last_clean":"2023-10-30T14:51:44.816766+0000","last_became_active":"2023-10-30T14:14:03.728053+0000","last_became_peered":"2023-10-30T14:14:03.728053+0000","last_unstale":"2023-10-30T14:51:44.816766+0000","last_undegraded":"2023-10-30T14:51:44.816766+0000","last_fullsized":"2023-10-30T14:51:44.816766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'327","last_scrub_stamp":"2023-10-30T14:51:44.816631+0000","last_deep_scrub":"67'327","last_deep_scrub_stamp":"2023-10-30T14:51:44.816631+0000","last_clean_scrub_stamp":"2023-10-30T14:51:44.816631+0000","objects_scrubbed":38,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:44:07.678724+0000","scrub_duration":0.092522458000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":90,"num_read_kb":49506,"num_write":347,"num_write_kb":29524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":346610,"num_omap_keys":678,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1922","reported_seq":3913,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:23.119066+0000","last_change":"2023-10-30T14:52:23.119154+0000","last_active":"2023-10-30T14:52:23.119066+0000","last_peered":"2023-10-30T14:52:23.119066+0000","last_clean":"2023-10-30T14:52:23.119066+0000","last_became_active":"2023-10-30T14:14:04.649896+0000","last_became_peered":"2023-10-30T14:14:04.649896+0000","last_unstale":"2023-10-30T14:52:23.119066+0000","last_undegraded":"2023-10-30T14:52:23.119066+0000","last_fullsized":"2023-10-30T14:52:23.119066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1922","last_scrub_stamp":"2023-10-30T14:52:23.118940+0000","last_deep_scrub":"67'1922","last_deep_scrub_stamp":"2023-10-30T14:52:23.118940+0000","last_clean_scrub_stamp":"2023-10-30T14:52:23.118940+0000","objects_scrubbed":516,"log_size":1922,"log_dups_size":0,"ondisk_log_size":1922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:19:11.163368+0000","scrub_duration":0.99494067500000005,"objects_trimmed":0,"snaptrim_duration":0.010792094,"stat_sum":{"num_bytes":9892654,"num_objects":516,"num_object_clones":241,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":516,"num_whiteouts":209,"num_read":88,"num_read_kb":2154,"num_write":1171,"num_write_kb":11317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"67'732","reported_seq":878,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:54.255960+0000","last_change":"2023-10-30T14:51:54.256020+0000","last_active":"2023-10-30T14:51:54.255960+0000","last_peered":"2023-10-30T14:51:54.255960+0000","last_clean":"2023-10-30T14:51:54.255960+0000","last_became_active":"2023-10-30T14:14:03.736155+0000","last_became_peered":"2023-10-30T14:14:03.736155+0000","last_unstale":"2023-10-30T14:51:54.255960+0000","last_undegraded":"2023-10-30T14:51:54.255960+0000","last_fullsized":"2023-10-30T14:51:54.255960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'732","last_scrub_stamp":"2023-10-30T14:51:54.255862+0000","last_deep_scrub":"67'732","last_deep_scrub_stamp":"2023-10-30T14:51:54.255862+0000","last_clean_scrub_stamp":"2023-10-30T14:51:54.255862+0000","objects_scrubbed":38,"log_size":732,"log_dups_size":0,"ondisk_log_size":732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:23:50.725664+0000","scrub_duration":0.094160410999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":104,"num_read_kb":67382,"num_write":743,"num_write_kb":42078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":416827,"num_omap_keys":710,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"67'1749","reported_seq":3749,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:07.048071+0000","last_change":"2023-10-30T14:52:07.048156+0000","last_active":"2023-10-30T14:52:07.048071+0000","last_peered":"2023-10-30T14:52:07.048071+0000","last_clean":"2023-10-30T14:52:07.048071+0000","last_became_active":"2023-10-30T14:14:04.614328+0000","last_became_peered":"2023-10-30T14:14:04.614328+0000","last_unstale":"2023-10-30T14:52:07.048071+0000","last_undegraded":"2023-10-30T14:52:07.048071+0000","last_fullsized":"2023-10-30T14:52:07.048071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1749","last_scrub_stamp":"2023-10-30T14:52:07.047963+0000","last_deep_scrub":"67'1749","last_deep_scrub_stamp":"2023-10-30T14:52:07.047963+0000","last_clean_scrub_stamp":"2023-10-30T14:52:07.047963+0000","objects_scrubbed":478,"log_size":1749,"log_dups_size":0,"ondisk_log_size":1749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:24:27.684849+0000","scrub_duration":0.80605112800000001,"objects_trimmed":0,"snaptrim_duration":0.010029286,"stat_sum":{"num_bytes":9325803,"num_objects":478,"num_object_clones":220,"num_object_copies":956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":188,"num_read":77,"num_read_kb":4169,"num_write":1062,"num_write_kb":11660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"67'1235","reported_seq":1429,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:46.739900+0000","last_change":"2023-10-30T14:51:46.739956+0000","last_active":"2023-10-30T14:51:46.739900+0000","last_peered":"2023-10-30T14:51:46.739900+0000","last_clean":"2023-10-30T14:51:46.739900+0000","last_became_active":"2023-10-30T14:14:03.700352+0000","last_became_peered":"2023-10-30T14:14:03.700352+0000","last_unstale":"2023-10-30T14:51:46.739900+0000","last_undegraded":"2023-10-30T14:51:46.739900+0000","last_fullsized":"2023-10-30T14:51:46.739900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1235","last_scrub_stamp":"2023-10-30T14:51:46.739823+0000","last_deep_scrub":"67'1235","last_deep_scrub_stamp":"2023-10-30T14:51:46.739823+0000","last_clean_scrub_stamp":"2023-10-30T14:51:46.739823+0000","objects_scrubbed":44,"log_size":1235,"log_dups_size":0,"ondisk_log_size":1235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:04:42.377958+0000","scrub_duration":0.069929302999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":252,"num_read_kb":55463,"num_write":1290,"num_write_kb":26283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":593045,"num_omap_keys":836,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"67'1852","reported_seq":3840,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:45.460580+0000","last_change":"2023-10-30T14:51:45.460580+0000","last_active":"2023-10-30T14:51:45.460580+0000","last_peered":"2023-10-30T14:51:45.460580+0000","last_clean":"2023-10-30T14:51:45.460580+0000","last_became_active":"2023-10-30T14:14:04.646639+0000","last_became_peered":"2023-10-30T14:14:04.646639+0000","last_unstale":"2023-10-30T14:51:45.460580+0000","last_undegraded":"2023-10-30T14:51:45.460580+0000","last_fullsized":"2023-10-30T14:51:45.460580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1851","last_scrub_stamp":"2023-10-30T14:51:45.460385+0000","last_deep_scrub":"67'1851","last_deep_scrub_stamp":"2023-10-30T14:51:45.460385+0000","last_clean_scrub_stamp":"2023-10-30T14:51:45.460385+0000","objects_scrubbed":508,"log_size":1852,"log_dups_size":0,"ondisk_log_size":1852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:46.502926+0000","scrub_duration":0.96639256900000003,"objects_trimmed":0,"snaptrim_duration":0.010379724,"stat_sum":{"num_bytes":13632563,"num_objects":508,"num_object_clones":233,"num_object_copies":1016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":508,"num_whiteouts":206,"num_read":77,"num_read_kb":12340,"num_write":1128,"num_write_kb":24370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'661","reported_seq":797,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:45.259757+0000","last_change":"2023-10-30T14:51:45.259757+0000","last_active":"2023-10-30T14:51:45.259757+0000","last_peered":"2023-10-30T14:51:45.259757+0000","last_clean":"2023-10-30T14:51:45.259757+0000","last_became_active":"2023-10-30T14:14:03.707950+0000","last_became_peered":"2023-10-30T14:14:03.707950+0000","last_unstale":"2023-10-30T14:51:45.259757+0000","last_undegraded":"2023-10-30T14:51:45.259757+0000","last_fullsized":"2023-10-30T14:51:45.259757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'661","last_scrub_stamp":"2023-10-30T14:51:45.259579+0000","last_deep_scrub":"67'661","last_deep_scrub_stamp":"2023-10-30T14:51:45.259579+0000","last_clean_scrub_stamp":"2023-10-30T14:51:45.259579+0000","objects_scrubbed":32,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:37:24.581339+0000","scrub_duration":0.102681681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971542,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":82,"num_read_kb":57693,"num_write":882,"num_write_kb":42351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":303757,"num_omap_keys":462,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"67'1765","reported_seq":3741,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:17.071813+0000","last_change":"2023-10-30T14:52:17.071813+0000","last_active":"2023-10-30T14:52:17.071813+0000","last_peered":"2023-10-30T14:52:17.071813+0000","last_clean":"2023-10-30T14:52:17.071813+0000","last_became_active":"2023-10-30T14:14:04.644761+0000","last_became_peered":"2023-10-30T14:14:04.644761+0000","last_unstale":"2023-10-30T14:52:17.071813+0000","last_undegraded":"2023-10-30T14:52:17.071813+0000","last_fullsized":"2023-10-30T14:52:17.071813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1765","last_scrub_stamp":"2023-10-30T14:52:17.071658+0000","last_deep_scrub":"67'1765","last_deep_scrub_stamp":"2023-10-30T14:52:17.071658+0000","last_clean_scrub_stamp":"2023-10-30T14:52:17.071658+0000","objects_scrubbed":503,"log_size":1765,"log_dups_size":0,"ondisk_log_size":1765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:17:38.995398+0000","scrub_duration":0.84598015100000001,"objects_trimmed":0,"snaptrim_duration":0.011171970999999999,"stat_sum":{"num_bytes":12945563,"num_objects":503,"num_object_clones":231,"num_object_copies":1006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":213,"num_read":62,"num_read_kb":4133,"num_write":1006,"num_write_kb":18895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'558","reported_seq":700,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:48.767990+0000","last_change":"2023-10-30T14:51:48.767990+0000","last_active":"2023-10-30T14:51:48.767990+0000","last_peered":"2023-10-30T14:51:48.767990+0000","last_clean":"2023-10-30T14:51:48.767990+0000","last_became_active":"2023-10-30T14:14:03.705186+0000","last_became_peered":"2023-10-30T14:14:03.705186+0000","last_unstale":"2023-10-30T14:51:48.767990+0000","last_undegraded":"2023-10-30T14:51:48.767990+0000","last_fullsized":"2023-10-30T14:51:48.767990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'558","last_scrub_stamp":"2023-10-30T14:51:48.767760+0000","last_deep_scrub":"67'558","last_deep_scrub_stamp":"2023-10-30T14:51:48.767760+0000","last_clean_scrub_stamp":"2023-10-30T14:51:48.767760+0000","objects_scrubbed":43,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:37:06.574920+0000","scrub_duration":0.12212297900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":102,"num_read_kb":90592,"num_write":581,"num_write_kb":50591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":484629,"num_omap_keys":764,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"67'1839","reported_seq":3814,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:35.160623+0000","last_change":"2023-10-30T14:52:35.160714+0000","last_active":"2023-10-30T14:52:35.160623+0000","last_peered":"2023-10-30T14:52:35.160623+0000","last_clean":"2023-10-30T14:52:35.160623+0000","last_became_active":"2023-10-30T14:14:04.646785+0000","last_became_peered":"2023-10-30T14:14:04.646785+0000","last_unstale":"2023-10-30T14:52:35.160623+0000","last_undegraded":"2023-10-30T14:52:35.160623+0000","last_fullsized":"2023-10-30T14:52:35.160623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1839","last_scrub_stamp":"2023-10-30T14:52:35.160510+0000","last_deep_scrub":"67'1839","last_deep_scrub_stamp":"2023-10-30T14:52:35.160510+0000","last_clean_scrub_stamp":"2023-10-30T14:52:35.160510+0000","objects_scrubbed":512,"log_size":1839,"log_dups_size":0,"ondisk_log_size":1839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:11.288483+0000","scrub_duration":0.98213454600000005,"objects_trimmed":0,"snaptrim_duration":0.010111184,"stat_sum":{"num_bytes":17483229,"num_objects":512,"num_object_clones":242,"num_object_copies":1024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":512,"num_whiteouts":218,"num_read":72,"num_read_kb":10698,"num_write":1044,"num_write_kb":23548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"67'622","reported_seq":783,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:44.214254+0000","last_change":"2023-10-30T14:51:44.214254+0000","last_active":"2023-10-30T14:51:44.214254+0000","last_peered":"2023-10-30T14:51:44.214254+0000","last_clean":"2023-10-30T14:51:44.214254+0000","last_became_active":"2023-10-30T14:14:03.692966+0000","last_became_peered":"2023-10-30T14:14:03.692966+0000","last_unstale":"2023-10-30T14:51:44.214254+0000","last_undegraded":"2023-10-30T14:51:44.214254+0000","last_fullsized":"2023-10-30T14:51:44.214254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'622","last_scrub_stamp":"2023-10-30T14:51:44.214112+0000","last_deep_scrub":"67'622","last_deep_scrub_stamp":"2023-10-30T14:51:44.214112+0000","last_clean_scrub_stamp":"2023-10-30T14:51:44.214112+0000","objects_scrubbed":36,"log_size":622,"log_dups_size":0,"ondisk_log_size":622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:10:21.572387+0000","scrub_duration":0.085160906999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16780852,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":126,"num_read_kb":110211,"num_write":641,"num_write_kb":50166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":448077,"num_omap_keys":692,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'1826","reported_seq":3829,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:00.576470+0000","last_change":"2023-10-30T14:52:00.576470+0000","last_active":"2023-10-30T14:52:00.576470+0000","last_peered":"2023-10-30T14:52:00.576470+0000","last_clean":"2023-10-30T14:52:00.576470+0000","last_became_active":"2023-10-30T14:14:04.615925+0000","last_became_peered":"2023-10-30T14:14:04.615925+0000","last_unstale":"2023-10-30T14:52:00.576470+0000","last_undegraded":"2023-10-30T14:52:00.576470+0000","last_fullsized":"2023-10-30T14:52:00.576470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1826","last_scrub_stamp":"2023-10-30T14:52:00.576300+0000","last_deep_scrub":"67'1826","last_deep_scrub_stamp":"2023-10-30T14:52:00.576300+0000","last_clean_scrub_stamp":"2023-10-30T14:52:00.576300+0000","objects_scrubbed":487,"log_size":1826,"log_dups_size":0,"ondisk_log_size":1826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:01:05.409471+0000","scrub_duration":1.040647087,"objects_trimmed":0,"snaptrim_duration":0.0068317569999999999,"stat_sum":{"num_bytes":4343182,"num_objects":487,"num_object_clones":225,"num_object_copies":974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":205,"num_read":84,"num_read_kb":56,"num_write":1109,"num_write_kb":8908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"67'683","reported_seq":832,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:39.782590+0000","last_change":"2023-10-30T14:51:39.782590+0000","last_active":"2023-10-30T14:51:39.782590+0000","last_peered":"2023-10-30T14:51:39.782590+0000","last_clean":"2023-10-30T14:51:39.782590+0000","last_became_active":"2023-10-30T14:14:03.727409+0000","last_became_peered":"2023-10-30T14:14:03.727409+0000","last_unstale":"2023-10-30T14:51:39.782590+0000","last_undegraded":"2023-10-30T14:51:39.782590+0000","last_fullsized":"2023-10-30T14:51:39.782590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'683","last_scrub_stamp":"2023-10-30T14:51:39.782417+0000","last_deep_scrub":"67'683","last_deep_scrub_stamp":"2023-10-30T14:51:39.782417+0000","last_clean_scrub_stamp":"2023-10-30T14:51:39.782417+0000","objects_scrubbed":36,"log_size":683,"log_dups_size":0,"ondisk_log_size":683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:14:03.723383+0000","scrub_duration":0.097535790999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166406,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":118,"num_read_kb":66012,"num_write":697,"num_write_kb":50302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":407093,"num_omap_keys":513,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1780","reported_seq":3781,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:22.255477+0000","last_change":"2023-10-30T14:52:22.255477+0000","last_active":"2023-10-30T14:52:22.255477+0000","last_peered":"2023-10-30T14:52:22.255477+0000","last_clean":"2023-10-30T14:52:22.255477+0000","last_became_active":"2023-10-30T14:14:45.818032+0000","last_became_peered":"2023-10-30T14:14:45.818032+0000","last_unstale":"2023-10-30T14:52:22.255477+0000","last_undegraded":"2023-10-30T14:52:22.255477+0000","last_fullsized":"2023-10-30T14:52:22.255477+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1780","last_scrub_stamp":"2023-10-30T14:52:22.255274+0000","last_deep_scrub":"67'1780","last_deep_scrub_stamp":"2023-10-30T14:52:22.255274+0000","last_clean_scrub_stamp":"2023-10-30T14:52:22.255274+0000","objects_scrubbed":485,"log_size":1780,"log_dups_size":0,"ondisk_log_size":1780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:19:32.919320+0000","scrub_duration":1.120569916,"objects_trimmed":0,"snaptrim_duration":0.010655906,"stat_sum":{"num_bytes":23800594,"num_objects":485,"num_object_clones":224,"num_object_copies":970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":201,"num_read":74,"num_read_kb":13429,"num_write":1078,"num_write_kb":29347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"67'119","reported_seq":241,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:50.399571+0000","last_change":"2023-10-30T14:51:50.399571+0000","last_active":"2023-10-30T14:51:50.399571+0000","last_peered":"2023-10-30T14:51:50.399571+0000","last_clean":"2023-10-30T14:51:50.399571+0000","last_became_active":"2023-10-30T14:13:54.250907+0000","last_became_peered":"2023-10-30T14:13:54.250907+0000","last_unstale":"2023-10-30T14:51:50.399571+0000","last_undegraded":"2023-10-30T14:51:50.399571+0000","last_fullsized":"2023-10-30T14:51:50.399571+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'119","last_scrub_stamp":"2023-10-30T14:51:50.399169+0000","last_deep_scrub":"67'119","last_deep_scrub_stamp":"2023-10-30T14:51:50.399169+0000","last_clean_scrub_stamp":"2023-10-30T14:51:50.399169+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:18.007665+0000","scrub_duration":0.0099470610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"67'751","reported_seq":905,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:45.869660+0000","last_change":"2023-10-30T14:51:45.869788+0000","last_active":"2023-10-30T14:51:45.869660+0000","last_peered":"2023-10-30T14:51:45.869660+0000","last_clean":"2023-10-30T14:51:45.869660+0000","last_became_active":"2023-10-30T14:14:03.728293+0000","last_became_peered":"2023-10-30T14:14:03.728293+0000","last_unstale":"2023-10-30T14:51:45.869660+0000","last_undegraded":"2023-10-30T14:51:45.869660+0000","last_fullsized":"2023-10-30T14:51:45.869660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'751","last_scrub_stamp":"2023-10-30T14:51:45.869434+0000","last_deep_scrub":"67'751","last_deep_scrub_stamp":"2023-10-30T14:51:45.869434+0000","last_clean_scrub_stamp":"2023-10-30T14:51:45.869434+0000","objects_scrubbed":46,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:31.381332+0000","scrub_duration":0.14853830900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165954,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":121,"num_read_kb":78313,"num_write":800,"num_write_kb":42232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":509463,"num_omap_keys":728,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"67'1729","reported_seq":3720,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:57.388643+0000","last_change":"2023-10-30T14:51:57.388643+0000","last_active":"2023-10-30T14:51:57.388643+0000","last_peered":"2023-10-30T14:51:57.388643+0000","last_clean":"2023-10-30T14:51:57.388643+0000","last_became_active":"2023-10-30T14:14:46.451620+0000","last_became_peered":"2023-10-30T14:14:46.451620+0000","last_unstale":"2023-10-30T14:51:57.388643+0000","last_undegraded":"2023-10-30T14:51:57.388643+0000","last_fullsized":"2023-10-30T14:51:57.388643+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1728","last_scrub_stamp":"2023-10-30T14:51:57.388517+0000","last_deep_scrub":"67'1728","last_deep_scrub_stamp":"2023-10-30T14:51:57.388517+0000","last_clean_scrub_stamp":"2023-10-30T14:51:57.388517+0000","objects_scrubbed":475,"log_size":1729,"log_dups_size":0,"ondisk_log_size":1729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:32:53.174606+0000","scrub_duration":0.79869517700000003,"objects_trimmed":0,"snaptrim_duration":0.0034002070000000001,"stat_sum":{"num_bytes":12539401,"num_objects":475,"num_object_clones":216,"num_object_copies":950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":184,"num_read":65,"num_read_kb":8203,"num_write":1056,"num_write_kb":18334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"67'873","reported_seq":1071,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:59.533102+0000","last_change":"2023-10-30T14:51:59.533102+0000","last_active":"2023-10-30T14:51:59.533102+0000","last_peered":"2023-10-30T14:51:59.533102+0000","last_clean":"2023-10-30T14:51:59.533102+0000","last_became_active":"2023-10-30T14:14:03.732199+0000","last_became_peered":"2023-10-30T14:14:03.732199+0000","last_unstale":"2023-10-30T14:51:59.533102+0000","last_undegraded":"2023-10-30T14:51:59.533102+0000","last_fullsized":"2023-10-30T14:51:59.533102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'873","last_scrub_stamp":"2023-10-30T14:51:59.532898+0000","last_deep_scrub":"67'873","last_deep_scrub_stamp":"2023-10-30T14:51:59.532898+0000","last_clean_scrub_stamp":"2023-10-30T14:51:59.532898+0000","objects_scrubbed":51,"log_size":873,"log_dups_size":0,"ondisk_log_size":873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:48:56.595717+0000","scrub_duration":0.15135636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":198,"num_read_kb":136195,"num_write":895,"num_write_kb":59080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":750104,"num_omap_keys":1295,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"67'1738","reported_seq":3727,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:56.490720+0000","last_change":"2023-10-30T14:51:56.490720+0000","last_active":"2023-10-30T14:51:56.490720+0000","last_peered":"2023-10-30T14:51:56.490720+0000","last_clean":"2023-10-30T14:51:56.490720+0000","last_became_active":"2023-10-30T14:14:04.619990+0000","last_became_peered":"2023-10-30T14:14:04.619990+0000","last_unstale":"2023-10-30T14:51:56.490720+0000","last_undegraded":"2023-10-30T14:51:56.490720+0000","last_fullsized":"2023-10-30T14:51:56.490720+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1737","last_scrub_stamp":"2023-10-30T14:51:56.490587+0000","last_deep_scrub":"67'1737","last_deep_scrub_stamp":"2023-10-30T14:51:56.490587+0000","last_clean_scrub_stamp":"2023-10-30T14:51:56.490587+0000","objects_scrubbed":475,"log_size":1738,"log_dups_size":0,"ondisk_log_size":1738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:42:50.032821+0000","scrub_duration":1.0169323969999999,"objects_trimmed":0,"snaptrim_duration":0.0045963250000000001,"stat_sum":{"num_bytes":8686325,"num_objects":475,"num_object_clones":219,"num_object_copies":950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":194,"num_read":63,"num_read_kb":1576,"num_write":1042,"num_write_kb":11254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"67'1782","reported_seq":2900,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:47.811285+0000","last_change":"2023-10-30T14:51:47.811373+0000","last_active":"2023-10-30T14:51:47.811285+0000","last_peered":"2023-10-30T14:51:47.811285+0000","last_clean":"2023-10-30T14:51:47.811285+0000","last_became_active":"2023-10-30T14:14:03.702513+0000","last_became_peered":"2023-10-30T14:14:03.702513+0000","last_unstale":"2023-10-30T14:51:47.811285+0000","last_undegraded":"2023-10-30T14:51:47.811285+0000","last_fullsized":"2023-10-30T14:51:47.811285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1782","last_scrub_stamp":"2023-10-30T14:51:47.811149+0000","last_deep_scrub":"67'1782","last_deep_scrub_stamp":"2023-10-30T14:51:47.811149+0000","last_clean_scrub_stamp":"2023-10-30T14:51:47.811149+0000","objects_scrubbed":55,"log_size":1782,"log_dups_size":0,"ondisk_log_size":1782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:45:13.943363+0000","scrub_duration":0.120464197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27782952,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":1112,"num_read_kb":113704,"num_write":1847,"num_write_kb":62106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":736919,"num_omap_keys":1275,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"67'1840","reported_seq":3821,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:59.429919+0000","last_change":"2023-10-30T14:51:58.917710+0000","last_active":"2023-10-30T14:51:59.429919+0000","last_peered":"2023-10-30T14:51:59.429919+0000","last_clean":"2023-10-30T14:51:59.429919+0000","last_became_active":"2023-10-30T14:14:04.589087+0000","last_became_peered":"2023-10-30T14:14:04.589087+0000","last_unstale":"2023-10-30T14:51:59.429919+0000","last_undegraded":"2023-10-30T14:51:59.429919+0000","last_fullsized":"2023-10-30T14:51:59.429919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1839","last_scrub_stamp":"2023-10-30T14:51:58.917576+0000","last_deep_scrub":"67'1839","last_deep_scrub_stamp":"2023-10-30T14:51:58.917576+0000","last_clean_scrub_stamp":"2023-10-30T14:51:58.917576+0000","objects_scrubbed":512,"log_size":1840,"log_dups_size":0,"ondisk_log_size":1840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:52:42.720787+0000","scrub_duration":0.62564701700000003,"objects_trimmed":0,"snaptrim_duration":0.0022871390000000001,"stat_sum":{"num_bytes":30609501,"num_objects":512,"num_object_clones":244,"num_object_copies":1024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":512,"num_whiteouts":220,"num_read":78,"num_read_kb":20511,"num_write":1070,"num_write_kb":36447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"67'656","reported_seq":816,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:42.807191+0000","last_change":"2023-10-30T14:51:42.807310+0000","last_active":"2023-10-30T14:51:42.807191+0000","last_peered":"2023-10-30T14:51:42.807191+0000","last_clean":"2023-10-30T14:51:42.807191+0000","last_became_active":"2023-10-30T14:14:03.728925+0000","last_became_peered":"2023-10-30T14:14:03.728925+0000","last_unstale":"2023-10-30T14:51:42.807191+0000","last_undegraded":"2023-10-30T14:51:42.807191+0000","last_fullsized":"2023-10-30T14:51:42.807191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'656","last_scrub_stamp":"2023-10-30T14:51:42.806981+0000","last_deep_scrub":"67'656","last_deep_scrub_stamp":"2023-10-30T14:51:42.806981+0000","last_clean_scrub_stamp":"2023-10-30T14:51:42.806981+0000","objects_scrubbed":40,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:27:15.123985+0000","scrub_duration":0.119997573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":120,"num_read_kb":131794,"num_write":661,"num_write_kb":62442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":535152,"num_omap_keys":914,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"67'1844","reported_seq":3823,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:49.103277+0000","last_change":"2023-10-30T14:51:49.103277+0000","last_active":"2023-10-30T14:51:49.103277+0000","last_peered":"2023-10-30T14:51:49.103277+0000","last_clean":"2023-10-30T14:51:49.103277+0000","last_became_active":"2023-10-30T14:14:04.607377+0000","last_became_peered":"2023-10-30T14:14:04.607377+0000","last_unstale":"2023-10-30T14:51:49.103277+0000","last_undegraded":"2023-10-30T14:51:49.103277+0000","last_fullsized":"2023-10-30T14:51:49.103277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1843","last_scrub_stamp":"2023-10-30T14:51:49.103084+0000","last_deep_scrub":"67'1843","last_deep_scrub_stamp":"2023-10-30T14:51:49.103084+0000","last_clean_scrub_stamp":"2023-10-30T14:51:49.103084+0000","objects_scrubbed":503,"log_size":1844,"log_dups_size":0,"ondisk_log_size":1844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:54:01.899046+0000","scrub_duration":1.0628055430000001,"objects_trimmed":0,"snaptrim_duration":0.0022021179999999999,"stat_sum":{"num_bytes":18168327,"num_objects":503,"num_object_clones":232,"num_object_copies":1006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":206,"num_read":65,"num_read_kb":9172,"num_write":1124,"num_write_kb":20929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"67'909","reported_seq":1057,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:52.423853+0000","last_change":"2023-10-30T14:51:52.423913+0000","last_active":"2023-10-30T14:51:52.423853+0000","last_peered":"2023-10-30T14:51:52.423853+0000","last_clean":"2023-10-30T14:51:52.423853+0000","last_became_active":"2023-10-30T14:14:03.692621+0000","last_became_peered":"2023-10-30T14:14:03.692621+0000","last_unstale":"2023-10-30T14:51:52.423853+0000","last_undegraded":"2023-10-30T14:51:52.423853+0000","last_fullsized":"2023-10-30T14:51:52.423853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'909","last_scrub_stamp":"2023-10-30T14:51:52.423719+0000","last_deep_scrub":"67'909","last_deep_scrub_stamp":"2023-10-30T14:51:52.423719+0000","last_clean_scrub_stamp":"2023-10-30T14:51:52.423719+0000","objects_scrubbed":43,"log_size":909,"log_dups_size":0,"ondisk_log_size":909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:15:28.937000+0000","scrub_duration":0.14411457899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":107,"num_read_kb":66236,"num_write":924,"num_write_kb":62670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":536879,"num_omap_keys":1021,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'1866","reported_seq":3825,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:58.425229+0000","last_change":"2023-10-30T14:51:58.425326+0000","last_active":"2023-10-30T14:51:58.425229+0000","last_peered":"2023-10-30T14:51:58.425229+0000","last_clean":"2023-10-30T14:51:58.425229+0000","last_became_active":"2023-10-30T14:14:04.613780+0000","last_became_peered":"2023-10-30T14:14:04.613780+0000","last_unstale":"2023-10-30T14:51:58.425229+0000","last_undegraded":"2023-10-30T14:51:58.425229+0000","last_fullsized":"2023-10-30T14:51:58.425229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1866","last_scrub_stamp":"2023-10-30T14:51:58.425122+0000","last_deep_scrub":"67'1866","last_deep_scrub_stamp":"2023-10-30T14:51:58.425122+0000","last_clean_scrub_stamp":"2023-10-30T14:51:58.425122+0000","objects_scrubbed":524,"log_size":1866,"log_dups_size":0,"ondisk_log_size":1866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:08:18.913623+0000","scrub_duration":0.94003744199999995,"objects_trimmed":0,"snaptrim_duration":0.0065285270000000001,"stat_sum":{"num_bytes":12750235,"num_objects":524,"num_object_clones":251,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":229,"num_read":63,"num_read_kb":8216,"num_write":1081,"num_write_kb":18064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"67'773","reported_seq":947,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:14.347808+0000","last_change":"2023-10-30T14:52:14.347925+0000","last_active":"2023-10-30T14:52:14.347808+0000","last_peered":"2023-10-30T14:52:14.347808+0000","last_clean":"2023-10-30T14:52:14.347808+0000","last_became_active":"2023-10-30T14:14:03.739650+0000","last_became_peered":"2023-10-30T14:14:03.739650+0000","last_unstale":"2023-10-30T14:52:14.347808+0000","last_undegraded":"2023-10-30T14:52:14.347808+0000","last_fullsized":"2023-10-30T14:52:14.347808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'773","last_scrub_stamp":"2023-10-30T14:52:14.347619+0000","last_deep_scrub":"67'773","last_deep_scrub_stamp":"2023-10-30T14:52:14.347619+0000","last_clean_scrub_stamp":"2023-10-30T14:52:14.347619+0000","objects_scrubbed":51,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:19:10.812358+0000","scrub_duration":0.145645301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777238,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":143,"num_read_kb":109192,"num_write":780,"num_write_kb":46441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":672517,"num_omap_keys":1187,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"67'1830","reported_seq":3815,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:00.930337+0000","last_change":"2023-10-30T14:52:00.930388+0000","last_active":"2023-10-30T14:52:00.930337+0000","last_peered":"2023-10-30T14:52:00.930337+0000","last_clean":"2023-10-30T14:52:00.930337+0000","last_became_active":"2023-10-30T14:14:04.590138+0000","last_became_peered":"2023-10-30T14:14:04.590138+0000","last_unstale":"2023-10-30T14:52:00.930337+0000","last_undegraded":"2023-10-30T14:52:00.930337+0000","last_fullsized":"2023-10-30T14:52:00.930337+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1830","last_scrub_stamp":"2023-10-30T14:52:00.930259+0000","last_deep_scrub":"67'1830","last_deep_scrub_stamp":"2023-10-30T14:52:00.930259+0000","last_clean_scrub_stamp":"2023-10-30T14:52:00.930259+0000","objects_scrubbed":503,"log_size":1830,"log_dups_size":0,"ondisk_log_size":1830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:27.603127+0000","scrub_duration":0.57889067100000002,"objects_trimmed":0,"snaptrim_duration":0.0031364370000000002,"stat_sum":{"num_bytes":9254236,"num_objects":503,"num_object_clones":233,"num_object_copies":1006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":207,"num_read":72,"num_read_kb":4145,"num_write":1099,"num_write_kb":19935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"67'793","reported_seq":945,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:43.351549+0000","last_change":"2023-10-30T14:51:43.351549+0000","last_active":"2023-10-30T14:51:43.351549+0000","last_peered":"2023-10-30T14:51:43.351549+0000","last_clean":"2023-10-30T14:51:43.351549+0000","last_became_active":"2023-10-30T14:14:03.721234+0000","last_became_peered":"2023-10-30T14:14:03.721234+0000","last_unstale":"2023-10-30T14:51:43.351549+0000","last_undegraded":"2023-10-30T14:51:43.351549+0000","last_fullsized":"2023-10-30T14:51:43.351549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'792","last_scrub_stamp":"2023-10-30T14:51:43.351414+0000","last_deep_scrub":"67'792","last_deep_scrub_stamp":"2023-10-30T14:51:43.351414+0000","last_clean_scrub_stamp":"2023-10-30T14:51:43.351414+0000","objects_scrubbed":34,"log_size":793,"log_dups_size":0,"ondisk_log_size":793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:49:31.872445+0000","scrub_duration":0.195047427,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":136,"num_read_kb":37589,"num_write":1319,"num_write_kb":40351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":332687,"num_omap_keys":642,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"67'1754","reported_seq":3736,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:06.360812+0000","last_change":"2023-10-30T14:52:06.360878+0000","last_active":"2023-10-30T14:52:06.360812+0000","last_peered":"2023-10-30T14:52:06.360812+0000","last_clean":"2023-10-30T14:52:06.360812+0000","last_became_active":"2023-10-30T14:14:04.633643+0000","last_became_peered":"2023-10-30T14:14:04.633643+0000","last_unstale":"2023-10-30T14:52:06.360812+0000","last_undegraded":"2023-10-30T14:52:06.360812+0000","last_fullsized":"2023-10-30T14:52:06.360812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1754","last_scrub_stamp":"2023-10-30T14:52:06.360742+0000","last_deep_scrub":"67'1754","last_deep_scrub_stamp":"2023-10-30T14:52:06.360742+0000","last_clean_scrub_stamp":"2023-10-30T14:52:06.360742+0000","objects_scrubbed":489,"log_size":1754,"log_dups_size":0,"ondisk_log_size":1754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:27.423144+0000","scrub_duration":0.91191191599999999,"objects_trimmed":0,"snaptrim_duration":0.0048916599999999999,"stat_sum":{"num_bytes":9341707,"num_objects":489,"num_object_clones":229,"num_object_copies":978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":205,"num_read":67,"num_read_kb":4121,"num_write":1049,"num_write_kb":15591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"67'464","reported_seq":611,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:44.149414+0000","last_change":"2023-10-30T14:51:44.149516+0000","last_active":"2023-10-30T14:51:44.149414+0000","last_peered":"2023-10-30T14:51:44.149414+0000","last_clean":"2023-10-30T14:51:44.149414+0000","last_became_active":"2023-10-30T14:14:03.737801+0000","last_became_peered":"2023-10-30T14:14:03.737801+0000","last_unstale":"2023-10-30T14:51:44.149414+0000","last_undegraded":"2023-10-30T14:51:44.149414+0000","last_fullsized":"2023-10-30T14:51:44.149414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'464","last_scrub_stamp":"2023-10-30T14:51:44.149257+0000","last_deep_scrub":"67'464","last_deep_scrub_stamp":"2023-10-30T14:51:44.149257+0000","last_clean_scrub_stamp":"2023-10-30T14:51:44.149257+0000","objects_scrubbed":47,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:41:08.565700+0000","scrub_duration":0.15054411200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":132,"num_read_kb":45549,"num_write":534,"num_write_kb":26063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":658163,"num_omap_keys":1051,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"67'1704","reported_seq":3703,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:02.952731+0000","last_change":"2023-10-30T14:52:02.952808+0000","last_active":"2023-10-30T14:52:02.952731+0000","last_peered":"2023-10-30T14:52:02.952731+0000","last_clean":"2023-10-30T14:52:02.952731+0000","last_became_active":"2023-10-30T14:14:04.588826+0000","last_became_peered":"2023-10-30T14:14:04.588826+0000","last_unstale":"2023-10-30T14:52:02.952731+0000","last_undegraded":"2023-10-30T14:52:02.952731+0000","last_fullsized":"2023-10-30T14:52:02.952731+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1704","last_scrub_stamp":"2023-10-30T14:52:02.952646+0000","last_deep_scrub":"67'1704","last_deep_scrub_stamp":"2023-10-30T14:52:02.952646+0000","last_clean_scrub_stamp":"2023-10-30T14:52:02.952646+0000","objects_scrubbed":453,"log_size":1704,"log_dups_size":0,"ondisk_log_size":1704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:53:48.955190+0000","scrub_duration":0.60434387099999998,"objects_trimmed":0,"snaptrim_duration":0.0040592670000000001,"stat_sum":{"num_bytes":8300341,"num_objects":453,"num_object_clones":211,"num_object_copies":906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":453,"num_whiteouts":180,"num_read":67,"num_read_kb":4125,"num_write":1055,"num_write_kb":10742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"67'452","reported_seq":574,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:55.355279+0000","last_change":"2023-10-30T14:51:55.355412+0000","last_active":"2023-10-30T14:51:55.355279+0000","last_peered":"2023-10-30T14:51:55.355279+0000","last_clean":"2023-10-30T14:51:55.355279+0000","last_became_active":"2023-10-30T14:14:03.732912+0000","last_became_peered":"2023-10-30T14:14:03.732912+0000","last_unstale":"2023-10-30T14:51:55.355279+0000","last_undegraded":"2023-10-30T14:51:55.355279+0000","last_fullsized":"2023-10-30T14:51:55.355279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'452","last_scrub_stamp":"2023-10-30T14:51:55.355171+0000","last_deep_scrub":"67'452","last_deep_scrub_stamp":"2023-10-30T14:51:55.355171+0000","last_clean_scrub_stamp":"2023-10-30T14:51:55.355171+0000","objects_scrubbed":23,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:16.834277+0000","scrub_duration":0.040337339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":78,"num_read_kb":41242,"num_write":429,"num_write_kb":21093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":203611,"num_omap_keys":381,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"67'1735","reported_seq":3730,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:49.980818+0000","last_change":"2023-10-30T14:51:49.980998+0000","last_active":"2023-10-30T14:51:49.980818+0000","last_peered":"2023-10-30T14:51:49.980818+0000","last_clean":"2023-10-30T14:51:49.980818+0000","last_became_active":"2023-10-30T14:14:04.609835+0000","last_became_peered":"2023-10-30T14:14:04.609835+0000","last_unstale":"2023-10-30T14:51:49.980818+0000","last_undegraded":"2023-10-30T14:51:49.980818+0000","last_fullsized":"2023-10-30T14:51:49.980818+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1735","last_scrub_stamp":"2023-10-30T14:51:49.980600+0000","last_deep_scrub":"67'1735","last_deep_scrub_stamp":"2023-10-30T14:51:49.980600+0000","last_clean_scrub_stamp":"2023-10-30T14:51:49.980600+0000","objects_scrubbed":470,"log_size":1735,"log_dups_size":0,"ondisk_log_size":1735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:25:11.080101+0000","scrub_duration":0.905012963,"objects_trimmed":0,"snaptrim_duration":0.0023958389999999999,"stat_sum":{"num_bytes":25940957,"num_objects":470,"num_object_clones":215,"num_object_copies":940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":192,"num_read":67,"num_read_kb":20495,"num_write":1067,"num_write_kb":27713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"67'884","reported_seq":1035,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:40.201508+0000","last_change":"2023-10-30T14:51:40.201605+0000","last_active":"2023-10-30T14:51:40.201508+0000","last_peered":"2023-10-30T14:51:40.201508+0000","last_clean":"2023-10-30T14:51:40.201508+0000","last_became_active":"2023-10-30T14:14:03.732265+0000","last_became_peered":"2023-10-30T14:14:03.732265+0000","last_unstale":"2023-10-30T14:51:40.201508+0000","last_undegraded":"2023-10-30T14:51:40.201508+0000","last_fullsized":"2023-10-30T14:51:40.201508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'884","last_scrub_stamp":"2023-10-30T14:51:40.201423+0000","last_deep_scrub":"67'884","last_deep_scrub_stamp":"2023-10-30T14:51:40.201423+0000","last_clean_scrub_stamp":"2023-10-30T14:51:40.201423+0000","objects_scrubbed":41,"log_size":884,"log_dups_size":0,"ondisk_log_size":884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:01:53.178520+0000","scrub_duration":0.062050998000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":128,"num_read_kb":49868,"num_write":925,"num_write_kb":26024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":535963,"num_omap_keys":904,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"67'1787","reported_seq":3793,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:49.263007+0000","last_change":"2023-10-30T14:51:49.263072+0000","last_active":"2023-10-30T14:51:49.263007+0000","last_peered":"2023-10-30T14:51:49.263007+0000","last_clean":"2023-10-30T14:51:49.263007+0000","last_became_active":"2023-10-30T14:14:04.594998+0000","last_became_peered":"2023-10-30T14:14:04.594998+0000","last_unstale":"2023-10-30T14:51:49.263007+0000","last_undegraded":"2023-10-30T14:51:49.263007+0000","last_fullsized":"2023-10-30T14:51:49.263007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1787","last_scrub_stamp":"2023-10-30T14:51:49.262934+0000","last_deep_scrub":"67'1787","last_deep_scrub_stamp":"2023-10-30T14:51:49.262934+0000","last_clean_scrub_stamp":"2023-10-30T14:51:49.262934+0000","objects_scrubbed":466,"log_size":1787,"log_dups_size":0,"ondisk_log_size":1787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:53:31.315621+0000","scrub_duration":0.73282899099999999,"objects_trimmed":0,"snaptrim_duration":0.00432118,"stat_sum":{"num_bytes":12343269,"num_objects":466,"num_object_clones":216,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":186,"num_read":78,"num_read_kb":8212,"num_write":1132,"num_write_kb":18560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'841","reported_seq":997,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:41.239916+0000","last_change":"2023-10-30T14:51:41.239981+0000","last_active":"2023-10-30T14:51:41.239916+0000","last_peered":"2023-10-30T14:51:41.239916+0000","last_clean":"2023-10-30T14:51:41.239916+0000","last_became_active":"2023-10-30T14:14:03.701759+0000","last_became_peered":"2023-10-30T14:14:03.701759+0000","last_unstale":"2023-10-30T14:51:41.239916+0000","last_undegraded":"2023-10-30T14:51:41.239916+0000","last_fullsized":"2023-10-30T14:51:41.239916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'841","last_scrub_stamp":"2023-10-30T14:51:41.239826+0000","last_deep_scrub":"67'841","last_deep_scrub_stamp":"2023-10-30T14:51:41.239826+0000","last_clean_scrub_stamp":"2023-10-30T14:51:41.239826+0000","objects_scrubbed":45,"log_size":841,"log_dups_size":0,"ondisk_log_size":841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:13:10.834903+0000","scrub_duration":0.095062992999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":122,"num_read_kb":94942,"num_write":1183,"num_write_kb":51323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":515565,"num_omap_keys":927,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"67'1674","reported_seq":3681,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:58.026837+0000","last_change":"2023-10-30T14:51:58.026837+0000","last_active":"2023-10-30T14:51:58.026837+0000","last_peered":"2023-10-30T14:51:58.026837+0000","last_clean":"2023-10-30T14:51:58.026837+0000","last_became_active":"2023-10-30T14:14:04.600100+0000","last_became_peered":"2023-10-30T14:14:04.600100+0000","last_unstale":"2023-10-30T14:51:58.026837+0000","last_undegraded":"2023-10-30T14:51:58.026837+0000","last_fullsized":"2023-10-30T14:51:58.026837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1674","last_scrub_stamp":"2023-10-30T14:51:58.026700+0000","last_deep_scrub":"67'1674","last_deep_scrub_stamp":"2023-10-30T14:51:58.026700+0000","last_clean_scrub_stamp":"2023-10-30T14:51:58.026700+0000","objects_scrubbed":447,"log_size":1674,"log_dups_size":0,"ondisk_log_size":1674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:30:55.078348+0000","scrub_duration":0.55760237800000001,"objects_trimmed":0,"snaptrim_duration":0.0052227630000000001,"stat_sum":{"num_bytes":16881451,"num_objects":447,"num_object_clones":209,"num_object_copies":894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":182,"num_read":73,"num_read_kb":11653,"num_write":1021,"num_write_kb":18895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"67'806","reported_seq":950,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:05.507868+0000","last_change":"2023-10-30T14:52:05.507868+0000","last_active":"2023-10-30T14:52:05.507868+0000","last_peered":"2023-10-30T14:52:05.507868+0000","last_clean":"2023-10-30T14:52:05.507868+0000","last_became_active":"2023-10-30T14:14:03.740648+0000","last_became_peered":"2023-10-30T14:14:03.740648+0000","last_unstale":"2023-10-30T14:52:05.507868+0000","last_undegraded":"2023-10-30T14:52:05.507868+0000","last_fullsized":"2023-10-30T14:52:05.507868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'806","last_scrub_stamp":"2023-10-30T14:52:05.507677+0000","last_deep_scrub":"67'806","last_deep_scrub_stamp":"2023-10-30T14:52:05.507677+0000","last_clean_scrub_stamp":"2023-10-30T14:52:05.507677+0000","objects_scrubbed":33,"log_size":806,"log_dups_size":0,"ondisk_log_size":806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:47:47.824486+0000","scrub_duration":0.134657734,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":98,"num_read_kb":107365,"num_write":798,"num_write_kb":50458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":418172,"num_omap_keys":644,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"67'1756","reported_seq":3765,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:01.140797+0000","last_change":"2023-10-30T14:52:01.140862+0000","last_active":"2023-10-30T14:52:01.140797+0000","last_peered":"2023-10-30T14:52:01.140797+0000","last_clean":"2023-10-30T14:52:01.140797+0000","last_became_active":"2023-10-30T14:14:04.622921+0000","last_became_peered":"2023-10-30T14:14:04.622921+0000","last_unstale":"2023-10-30T14:52:01.140797+0000","last_undegraded":"2023-10-30T14:52:01.140797+0000","last_fullsized":"2023-10-30T14:52:01.140797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1756","last_scrub_stamp":"2023-10-30T14:52:01.140687+0000","last_deep_scrub":"67'1756","last_deep_scrub_stamp":"2023-10-30T14:52:01.140687+0000","last_clean_scrub_stamp":"2023-10-30T14:52:01.140687+0000","objects_scrubbed":465,"log_size":1756,"log_dups_size":0,"ondisk_log_size":1756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:42:01.063269+0000","scrub_duration":0.83106379799999996,"objects_trimmed":0,"snaptrim_duration":0.0071219910000000003,"stat_sum":{"num_bytes":20799446,"num_objects":465,"num_object_clones":210,"num_object_copies":930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":465,"num_whiteouts":183,"num_read":75,"num_read_kb":12323,"num_write":1101,"num_write_kb":26545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"67'1810","reported_seq":3792,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:59.448185+0000","last_change":"2023-10-30T14:51:59.448289+0000","last_active":"2023-10-30T14:51:59.448185+0000","last_peered":"2023-10-30T14:51:59.448185+0000","last_clean":"2023-10-30T14:51:59.448185+0000","last_became_active":"2023-10-30T14:14:04.621885+0000","last_became_peered":"2023-10-30T14:14:04.621885+0000","last_unstale":"2023-10-30T14:51:59.448185+0000","last_undegraded":"2023-10-30T14:51:59.448185+0000","last_fullsized":"2023-10-30T14:51:59.448185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1810","last_scrub_stamp":"2023-10-30T14:51:59.448092+0000","last_deep_scrub":"67'1810","last_deep_scrub_stamp":"2023-10-30T14:51:59.448092+0000","last_clean_scrub_stamp":"2023-10-30T14:51:59.448092+0000","objects_scrubbed":487,"log_size":1810,"log_dups_size":0,"ondisk_log_size":1810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:36:33.157702+0000","scrub_duration":0.92442001200000001,"objects_trimmed":0,"snaptrim_duration":0.0067851329999999996,"stat_sum":{"num_bytes":13483294,"num_objects":487,"num_object_clones":228,"num_object_copies":974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":191,"num_read":68,"num_read_kb":8221,"num_write":1123,"num_write_kb":20773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"67'517","reported_seq":678,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:02.491572+0000","last_change":"2023-10-30T14:52:02.491644+0000","last_active":"2023-10-30T14:52:02.491572+0000","last_peered":"2023-10-30T14:52:02.491572+0000","last_clean":"2023-10-30T14:52:02.491572+0000","last_became_active":"2023-10-30T14:14:03.740247+0000","last_became_peered":"2023-10-30T14:14:03.740247+0000","last_unstale":"2023-10-30T14:52:02.491572+0000","last_undegraded":"2023-10-30T14:52:02.491572+0000","last_fullsized":"2023-10-30T14:52:02.491572+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'517","last_scrub_stamp":"2023-10-30T14:52:02.491486+0000","last_deep_scrub":"67'517","last_deep_scrub_stamp":"2023-10-30T14:52:02.491486+0000","last_clean_scrub_stamp":"2023-10-30T14:52:02.491486+0000","objects_scrubbed":49,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:33:41.383440+0000","scrub_duration":0.160540247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":149,"num_read_kb":82851,"num_write":526,"num_write_kb":42289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":701328,"num_omap_keys":1094,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'1964","reported_seq":3938,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:05.869580+0000","last_change":"2023-10-30T14:52:05.869633+0000","last_active":"2023-10-30T14:52:05.869580+0000","last_peered":"2023-10-30T14:52:05.869580+0000","last_clean":"2023-10-30T14:52:05.869580+0000","last_became_active":"2023-10-30T14:14:04.623638+0000","last_became_peered":"2023-10-30T14:14:04.623638+0000","last_unstale":"2023-10-30T14:52:05.869580+0000","last_undegraded":"2023-10-30T14:52:05.869580+0000","last_fullsized":"2023-10-30T14:52:05.869580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1964","last_scrub_stamp":"2023-10-30T14:52:05.869480+0000","last_deep_scrub":"67'1964","last_deep_scrub_stamp":"2023-10-30T14:52:05.869480+0000","last_clean_scrub_stamp":"2023-10-30T14:52:05.869480+0000","objects_scrubbed":546,"log_size":1964,"log_dups_size":0,"ondisk_log_size":1964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:19:44.576287+0000","scrub_duration":1.2098437230000001,"objects_trimmed":0,"snaptrim_duration":0.0049828509999999999,"stat_sum":{"num_bytes":25534330,"num_objects":546,"num_object_clones":248,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":546,"num_whiteouts":220,"num_read":77,"num_read_kb":19614,"num_write":1187,"num_write_kb":31851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"67'1076","reported_seq":1239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:13.386027+0000","last_change":"2023-10-30T14:52:13.386187+0000","last_active":"2023-10-30T14:52:13.386027+0000","last_peered":"2023-10-30T14:52:13.386027+0000","last_clean":"2023-10-30T14:52:13.386027+0000","last_became_active":"2023-10-30T14:14:03.732538+0000","last_became_peered":"2023-10-30T14:14:03.732538+0000","last_unstale":"2023-10-30T14:52:13.386027+0000","last_undegraded":"2023-10-30T14:52:13.386027+0000","last_fullsized":"2023-10-30T14:52:13.386027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1076","last_scrub_stamp":"2023-10-30T14:52:13.385874+0000","last_deep_scrub":"67'1076","last_deep_scrub_stamp":"2023-10-30T14:52:13.385874+0000","last_clean_scrub_stamp":"2023-10-30T14:52:13.385874+0000","objects_scrubbed":48,"log_size":1076,"log_dups_size":0,"ondisk_log_size":1076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:51:41.237586+0000","scrub_duration":0.18064516899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":137,"num_read_kb":150323,"num_write":1072,"num_write_kb":62947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":547720,"num_omap_keys":934,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1793","reported_seq":3773,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:02.330358+0000","last_change":"2023-10-30T14:52:02.330428+0000","last_active":"2023-10-30T14:52:02.330358+0000","last_peered":"2023-10-30T14:52:02.330358+0000","last_clean":"2023-10-30T14:52:02.330358+0000","last_became_active":"2023-10-30T14:14:04.611773+0000","last_became_peered":"2023-10-30T14:14:04.611773+0000","last_unstale":"2023-10-30T14:52:02.330358+0000","last_undegraded":"2023-10-30T14:52:02.330358+0000","last_fullsized":"2023-10-30T14:52:02.330358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1793","last_scrub_stamp":"2023-10-30T14:52:02.330281+0000","last_deep_scrub":"67'1793","last_deep_scrub_stamp":"2023-10-30T14:52:02.330281+0000","last_clean_scrub_stamp":"2023-10-30T14:52:02.330281+0000","objects_scrubbed":513,"log_size":1793,"log_dups_size":0,"ondisk_log_size":1793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:30.490023+0000","scrub_duration":0.96875353099999995,"objects_trimmed":0,"snaptrim_duration":0.0044924040000000002,"stat_sum":{"num_bytes":26364015,"num_objects":513,"num_object_clones":233,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":513,"num_whiteouts":203,"num_read":66,"num_read_kb":19602,"num_write":1077,"num_write_kb":27528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"67'747","reported_seq":899,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:40.646962+0000","last_change":"2023-10-30T14:51:40.646962+0000","last_active":"2023-10-30T14:51:40.646962+0000","last_peered":"2023-10-30T14:51:40.646962+0000","last_clean":"2023-10-30T14:51:40.646962+0000","last_became_active":"2023-10-30T14:14:03.703691+0000","last_became_peered":"2023-10-30T14:14:03.703691+0000","last_unstale":"2023-10-30T14:51:40.646962+0000","last_undegraded":"2023-10-30T14:51:40.646962+0000","last_fullsized":"2023-10-30T14:51:40.646962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'747","last_scrub_stamp":"2023-10-30T14:51:40.646821+0000","last_deep_scrub":"67'747","last_deep_scrub_stamp":"2023-10-30T14:51:40.646821+0000","last_clean_scrub_stamp":"2023-10-30T14:51:40.646821+0000","objects_scrubbed":37,"log_size":747,"log_dups_size":0,"ondisk_log_size":747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:10:48.146914+0000","scrub_duration":0.052199267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":102,"num_read_kb":111696,"num_write":764,"num_write_kb":58666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":552830,"num_omap_keys":982,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"67'1847","reported_seq":3811,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:52:05.940136+0000","last_change":"2023-10-30T14:52:05.940136+0000","last_active":"2023-10-30T14:52:05.940136+0000","last_peered":"2023-10-30T14:52:05.940136+0000","last_clean":"2023-10-30T14:52:05.940136+0000","last_became_active":"2023-10-30T14:14:04.613420+0000","last_became_peered":"2023-10-30T14:14:04.613420+0000","last_unstale":"2023-10-30T14:52:05.940136+0000","last_undegraded":"2023-10-30T14:52:05.940136+0000","last_fullsized":"2023-10-30T14:52:05.940136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1847","last_scrub_stamp":"2023-10-30T14:52:05.939976+0000","last_deep_scrub":"67'1847","last_deep_scrub_stamp":"2023-10-30T14:52:05.939976+0000","last_clean_scrub_stamp":"2023-10-30T14:52:05.939976+0000","objects_scrubbed":509,"log_size":1847,"log_dups_size":0,"ondisk_log_size":1847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:23:03.638601+0000","scrub_duration":1.492893977,"objects_trimmed":0,"snaptrim_duration":0.004010128,"stat_sum":{"num_bytes":13027616,"num_objects":509,"num_object_clones":238,"num_object_copies":1018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":509,"num_whiteouts":201,"num_read":55,"num_read_kb":8221,"num_write":1124,"num_write_kb":14511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"67'656","reported_seq":827,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T14:51:59.435536+0000","last_change":"2023-10-30T14:51:38.523396+0000","last_active":"2023-10-30T14:51:59.435536+0000","last_peered":"2023-10-30T14:51:59.435536+0000","last_clean":"2023-10-30T14:51:59.435536+0000","last_became_active":"2023-10-30T14:14:03.736964+0000","last_became_peered":"2023-10-30T14:14:03.736964+0000","last_unstale":"2023-10-30T14:51:59.435536+0000","last_undegraded":"2023-10-30T14:51:59.435536+0000","last_fullsized":"2023-10-30T14:51:59.435536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'654","last_scrub_stamp":"2023-10-30T14:51:38.523259+0000","last_deep_scrub":"67'654","last_deep_scrub_stamp":"2023-10-30T14:51:38.523259+0000","last_clean_scrub_stamp":"2023-10-30T14:51:38.523259+0000","objects_scrubbed":39,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:31:19.574364+0000","scrub_duration":0.063641798999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":182,"num_read_kb":58134,"num_write":700,"num_write_kb":34131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":484120,"num_omap_keys":814,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1055045924,"num_objects":31721,"num_object_clones":14708,"num_object_copies":63442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31721,"num_whiteouts":12823,"num_read":4602,"num_read_kb":662584,"num_write":70009,"num_write_kb":1350127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2186289152,"data_stored":2111320648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115820,"ondisk_log_size":115820,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":689694586,"num_objects":1358,"num_object_clones":0,"num_object_copies":2716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1358,"num_whiteouts":0,"num_read":6032,"num_read_kb":2835146,"num_write":28240,"num_write_kb":1523169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1128,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16321744,"num_omap_keys":27609,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1379532800,"data_stored":1379389172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26470,"ondisk_log_size":26470,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575319,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":654940,"kb_used_data":489172,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93716900,"statfs":{"total":96636764160,"available":95966105600,"internally_reserved":0,"allocated":500912128,"data_stored":490062487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":169736651},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:52:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":1.038,"15min":0.98399999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.1539999999999999,"5min":6.399,"15min":6.399},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.997,"15min":0.94099999999999995},"min":{"1min":0.41499999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.474,"5min":6.4969999999999999,"15min":6.4969999999999999},"last":1.038}]},{"osd":1,"last update":"Mon Oct 30 14:52:21 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.94099999999999995,"15min":0.92900000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.6140000000000001,"5min":6.1390000000000002,"15min":6.1390000000000002},"last":1.2170000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.91700000000000004,"15min":0.97399999999999998},"min":{"1min":0.49399999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.3049999999999999,"5min":6.1920000000000002,"15min":6.1920000000000002},"last":0.60099999999999998}]},{"osd":2,"last update":"Mon Oct 30 14:52:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0669999999999999,"15min":0.97299999999999998},"min":{"1min":0.52300000000000002,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.623,"5min":6.5739999999999998,"15min":6.5739999999999998},"last":1.351},{"interface":"front","average":{"1min":0.999,"5min":1.121,"15min":1.018},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.4810000000000001,"5min":6.5149999999999997,"15min":6.5149999999999997},"last":1.2669999999999999}]},{"osd":3,"last update":"Mon Oct 30 14:52:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.86399999999999999,"15min":0.85199999999999998},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.53800000000000003},"max":{"1min":1.2130000000000001,"5min":4.3230000000000004,"15min":4.3230000000000004},"last":1.395},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.093,"15min":1.0620000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.3129999999999999,"5min":6.2670000000000003,"15min":6.2670000000000003},"last":1.492}]},{"osd":4,"last update":"Mon Oct 30 14:52:21 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.96199999999999997,"15min":0.95199999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.478,"5min":4.7409999999999997,"15min":4.7409999999999997},"last":0.754},{"interface":"front","average":{"1min":1.056,"5min":1.05,"15min":0.98399999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.5620000000000001,"5min":8.0719999999999992,"15min":8.0719999999999992},"last":0.90500000000000003}]},{"osd":5,"last update":"Mon Oct 30 14:52:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.2989999999999999,"15min":1.181},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":2.1019999999999999,"5min":10.692,"15min":10.692},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.385,"15min":1.29},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":2.0590000000000002,"5min":6.6429999999999998,"15min":6.6429999999999998},"last":1.5880000000000001}]},{"osd":6,"last update":"Mon Oct 30 14:52:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.139,"15min":1.1200000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.5760000000000001,"5min":5.2160000000000002,"15min":5.2160000000000002},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.1910000000000001,"15min":1.119},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.927,"5min":4.367,"15min":4.367},"last":1.8799999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575319,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513220,"kb_used_data":387708,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":93858620,"statfs":{"total":96636764160,"available":96111226880,"internally_reserved":0,"allocated":397012992,"data_stored":389718204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":128514503},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":0.90200000000000002,"15min":0.84999999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":9.7949999999999999,"5min":9.7949999999999999,"15min":9.7949999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.23,"5min":0.97899999999999998,"15min":0.91500000000000004},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":9.0180000000000007,"5min":9.0180000000000007,"15min":9.0180000000000007},"last":1.0880000000000001}]},{"osd":1,"last update":"Mon Oct 30 14:52:12 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":0.97499999999999998,"15min":0.879},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":10.169,"5min":10.169,"15min":10.169},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.216,"5min":1,"15min":0.97599999999999998},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.318},"max":{"1min":8.4169999999999998,"5min":8.4169999999999998,"15min":8.4169999999999998},"last":1.1160000000000001}]},{"osd":2,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.90900000000000003,"15min":0.89700000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":5.3819999999999997,"5min":5.3819999999999997,"15min":5.3819999999999997},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":0.98399999999999999,"15min":0.95399999999999996},"min":{"1min":0.496,"5min":0.496,"15min":0.49299999999999999},"max":{"1min":4.8949999999999996,"5min":4.8949999999999996,"15min":4.8949999999999996},"last":0.81699999999999995}]},{"osd":3,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":1.639,"5min":1.1379999999999999,"15min":1.0149999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":14.827999999999999,"5min":14.827999999999999,"15min":14.827999999999999},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.0920000000000001,"5min":0.98399999999999999,"15min":0.97399999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":4.9279999999999999,"5min":4.9279999999999999,"15min":4.9279999999999999},"last":1.0469999999999999}]},{"osd":4,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.056,"15min":1.0369999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":3.7549999999999999,"5min":3.8399999999999999,"15min":3.8399999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.24,"5min":1.0780000000000001,"15min":1.0569999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.6419999999999999,"5min":3.6419999999999999,"15min":3.6419999999999999},"last":0.99099999999999999}]},{"osd":5,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.0389999999999999,"15min":0.97899999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.2120000000000002,"5min":11.728,"15min":11.728},"last":1.0209999999999999},{"interface":"front","average":{"1min":2.4289999999999998,"5min":1.3879999999999999,"15min":1.171},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":28.178999999999998,"5min":28.178999999999998,"15min":28.178999999999998},"last":1.1659999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.3080000000000001,"15min":1.165},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":9.7609999999999992,"5min":18.041,"15min":18.041},"last":1.0700000000000001},{"interface":"front","average":{"1min":3.593,"5min":1.835,"15min":1.393},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":53.113,"5min":53.113,"15min":53.113},"last":1.2130000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575319,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692396,"kb_used_data":539108,"kb_used_omap":1,"kb_used_meta":153278,"kb_avail":93679444,"statfs":{"total":96636764160,"available":95927750656,"internally_reserved":0,"allocated":552046592,"data_stored":540347463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":156957126},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:51:51 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.84399999999999997,"15min":0.81399999999999995},"min":{"1min":0.47699999999999998,"5min":0.43099999999999999,"15min":0.40100000000000002},"max":{"1min":1.2070000000000001,"5min":7.04,"15min":7.04},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.89500000000000002,"15min":0.84999999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.202,"5min":5.4720000000000004,"15min":5.4720000000000004},"last":1.103}]},{"osd":1,"last update":"Mon Oct 30 14:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.873,"15min":0.879},"min":{"1min":0.53400000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.2390000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.84299999999999997,"15min":0.84499999999999997},"min":{"1min":0.53500000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0569999999999999,"5min":4.1630000000000003,"15min":4.1630000000000003},"last":0.81299999999999994}]},{"osd":2,"last update":"Mon Oct 30 14:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.93300000000000005,"15min":0.95399999999999996},"min":{"1min":0.54900000000000004,"5min":0.53100000000000003,"15min":0.44800000000000001},"max":{"1min":1.224,"5min":1.3640000000000001,"15min":2.044},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0980000000000001,"15min":1.0249999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.53800000000000003},"max":{"1min":1.3919999999999999,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":1.161}]},{"osd":3,"last update":"Mon Oct 30 14:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.83199999999999996,"15min":0.92400000000000004},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0840000000000001,"5min":1.2010000000000001,"15min":1.4550000000000001},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.0669999999999999,"15min":0.98399999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.52100000000000002},"max":{"1min":1.3919999999999999,"5min":1.3959999999999999,"15min":1.4059999999999999},"last":1.2549999999999999}]},{"osd":4,"last update":"Mon Oct 30 14:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.1160000000000001,"15min":1.032},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.59099999999999997},"max":{"1min":1.046,"5min":8.7360000000000007,"15min":8.7360000000000007},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.129,"5min":1.27,"15min":1.0760000000000001},"min":{"1min":0.73199999999999998,"5min":0.61199999999999999,"15min":0.59399999999999997},"max":{"1min":1.5980000000000001,"5min":23.640000000000001,"15min":23.640000000000001},"last":1.3440000000000001}]},{"osd":6,"last update":"Mon Oct 30 14:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1539999999999999,"15min":1.0800000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.55500000000000005},"max":{"1min":1.657,"5min":1.7470000000000001,"15min":2.306},"last":1.21},{"interface":"front","average":{"1min":1.3,"5min":1.2330000000000001,"15min":1.139},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.57899999999999996},"max":{"1min":1.6120000000000001,"5min":1.7569999999999999,"15min":2.774},"last":1.3}]},{"osd":7,"last update":"Mon Oct 30 14:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0029999999999999,"15min":1.0569999999999999},"min":{"1min":0.66700000000000004,"5min":0.64300000000000002,"15min":0.40799999999999997},"max":{"1min":1.532,"5min":1.532,"15min":3.0739999999999998},"last":1.069},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0940000000000001,"15min":1.163},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.77600000000000002},"max":{"1min":1.1970000000000001,"5min":1.53,"15min":2.1339999999999999},"last":1.131}]}]},{"osd":4,"up_from":13,"seq":55834575319,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645592,"kb_used_data":479824,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93726248,"statfs":{"total":96636764160,"available":95975677952,"internally_reserved":0,"allocated":491339776,"data_stored":479836699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.97299999999999998,"15min":0.87},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.141,"5min":7.1870000000000003,"15min":7.1870000000000003},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.80900000000000005,"5min":1.0940000000000001,"15min":0.95699999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.1180000000000001,"5min":10.864000000000001,"15min":10.864000000000001},"last":1.0389999999999999}]},{"osd":1,"last update":"Mon Oct 30 14:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":1.028,"15min":0.95299999999999996},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.331,"5min":7.6980000000000004,"15min":7.6980000000000004},"last":1.1830000000000001},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.90800000000000003,"15min":0.84399999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.36599999999999999},"max":{"1min":1.798,"5min":5.5259999999999998,"15min":5.5259999999999998},"last":0.96199999999999997}]},{"osd":2,"last update":"Mon Oct 30 14:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.997,"15min":0.95999999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.756,"5min":5.423,"15min":5.423},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.2549999999999999,"15min":1.0960000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.325,"5min":13.221,"15min":13.221},"last":1.2829999999999999}]},{"osd":3,"last update":"Mon Oct 30 14:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.95599999999999996,"15min":0.86199999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.96799999999999997,"5min":7.5620000000000003,"15min":7.5620000000000003},"last":0.755},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.0800000000000001,"15min":0.996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.171,"5min":7.7370000000000001,"15min":7.7370000000000001},"last":0.90900000000000003}]},{"osd":5,"last update":"Mon Oct 30 14:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":1.042,"15min":0.96899999999999997},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.833,"5min":7.6360000000000001,"15min":7.6360000000000001},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.236,"15min":1.1399999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.399,"5min":7.2599999999999998,"15min":7.2599999999999998},"last":1.355}]},{"osd":6,"last update":"Mon Oct 30 14:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.163,"15min":1.103},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.8500000000000001,"5min":5.4729999999999999,"15min":5.4729999999999999},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.016,"5min":1.2709999999999999,"15min":1.1719999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.454,"5min":10.102,"15min":10.102},"last":1.405}]},{"osd":7,"last update":"Mon Oct 30 14:52:10 2023","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.218,"15min":1.0820000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":13.026999999999999,"5min":13.026999999999999,"15min":13.026999999999999},"last":1.107},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.236,"15min":1.208},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.8939999999999999,"5min":7.0490000000000004,"15min":7.0490000000000004},"last":1.242}]}]},{"osd":3,"up_from":13,"seq":55834575317,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":660364,"kb_used_data":508484,"kb_used_omap":1,"kb_used_meta":151870,"kb_avail":93711476,"statfs":{"total":96636764160,"available":95960551424,"internally_reserved":0,"allocated":520687616,"data_stored":509150004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":155515336},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:52:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.505,"15min":0.48399999999999999},"min":{"1min":0.30099999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.54900000000000004,"5min":4.1130000000000004,"15min":4.1130000000000004},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53300000000000003,"15min":0.51100000000000001},"min":{"1min":0.28899999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.72199999999999998,"5min":4.5190000000000001,"15min":4.5190000000000001},"last":0.65700000000000003}]},{"osd":1,"last update":"Mon Oct 30 14:52:15 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.57799999999999996,"15min":0.54600000000000004},"min":{"1min":0.35999999999999999,"5min":0.317,"15min":0.314},"max":{"1min":0.98899999999999999,"5min":3.988,"15min":3.988},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56399999999999995,"15min":0.54900000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.29999999999999999},"max":{"1min":1.859,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.73699999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:52:07 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56699999999999995,"15min":0.56499999999999995},"min":{"1min":0.42299999999999999,"5min":0.29199999999999998,"15min":0.27600000000000002},"max":{"1min":0.77900000000000003,"5min":3.0590000000000002,"15min":8.6519999999999992},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.68300000000000005,"15min":0.63600000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.313},"max":{"1min":0.83999999999999997,"5min":7.0949999999999998,"15min":7.0949999999999998},"last":0.753}]},{"osd":4,"last update":"Mon Oct 30 14:52:07 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.80500000000000005,"15min":0.81599999999999995},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.50900000000000001},"max":{"1min":0.93899999999999995,"5min":4.5730000000000004,"15min":4.5730000000000004},"last":1.1080000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.872,"15min":0.84399999999999997},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.48899999999999999},"max":{"1min":1.0569999999999999,"5min":4.4720000000000004,"15min":4.4720000000000004},"last":0.82199999999999995}]},{"osd":5,"last update":"Mon Oct 30 14:52:07 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.98999999999999999,"15min":0.89400000000000002},"min":{"1min":0.56499999999999995,"5min":0.46899999999999997,"15min":0.46800000000000003},"max":{"1min":1.0029999999999999,"5min":14.974,"15min":14.974},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.94399999999999995,"15min":0.871},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.46600000000000003},"max":{"1min":1.1100000000000001,"5min":5.907,"15min":5.907},"last":0.97399999999999998}]},{"osd":6,"last update":"Mon Oct 30 14:52:07 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.96999999999999997,"15min":0.92000000000000004},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.2070000000000001,"5min":5.8300000000000001,"15min":5.8300000000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.91400000000000003,"5min":1.018,"15min":0.96299999999999997},"min":{"1min":0.64200000000000002,"5min":0.60199999999999998,"15min":0.55100000000000005},"max":{"1min":1.248,"5min":6.0300000000000002,"15min":6.0300000000000002},"last":0.997}]},{"osd":7,"last update":"Mon Oct 30 14:52:07 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.91200000000000003,"15min":0.99099999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.0529999999999999,"5min":4.6159999999999997,"15min":16.282},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.871,"5min":0.96199999999999997,"15min":0.96799999999999997},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.58099999999999996},"max":{"1min":1.161,"5min":5.7770000000000001,"15min":16.096},"last":0.96299999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575319,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422268,"kb_used_data":310644,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":93949572,"statfs":{"total":96636764160,"available":96204361728,"internally_reserved":0,"allocated":318099456,"data_stored":311268353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114293190},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:52:23 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.45200000000000001,"15min":0.44800000000000001},"min":{"1min":0.29199999999999998,"5min":0.27400000000000002,"15min":0.26300000000000001},"max":{"1min":0.56599999999999995,"5min":0.79100000000000004,"15min":0.82599999999999996},"last":0.435},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51900000000000002,"15min":0.51300000000000001},"min":{"1min":0.36299999999999999,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":0.73399999999999999,"5min":0.80900000000000005,"15min":1.2190000000000001},"last":0.46600000000000003}]},{"osd":1,"last update":"Mon Oct 30 14:52:23 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.46100000000000002,"15min":0.46600000000000003},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28299999999999997},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":1.5469999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55000000000000004,"15min":0.55700000000000005},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":1.452},"last":0.41199999999999998}]},{"osd":3,"last update":"Mon Oct 30 14:52:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58399999999999996,"15min":0.59999999999999998},"min":{"1min":0.438,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.871,"5min":0.871,"15min":1.506},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.51500000000000001,"15min":0.53000000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.33100000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.496}]},{"osd":4,"last update":"Mon Oct 30 14:52:23 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.80700000000000005,"15min":0.80400000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.44900000000000001},"max":{"1min":1.0740000000000001,"5min":4.2229999999999999,"15min":4.2229999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.90700000000000003,"15min":0.90200000000000002},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.101,"5min":4.2439999999999998,"15min":4.2439999999999998},"last":0.89100000000000001}]},{"osd":5,"last update":"Mon Oct 30 14:52:23 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.96299999999999997,"15min":0.85499999999999998},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.97999999999999998,"5min":22.285,"15min":22.285},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.82899999999999996,"15min":0.82599999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0529999999999999,"5min":2.3999999999999999,"15min":2.742},"last":0.92700000000000005}]},{"osd":6,"last update":"Mon Oct 30 14:52:23 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.85799999999999998,"15min":0.84099999999999997},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.54600000000000004},"max":{"1min":1.276,"5min":2.911,"15min":2.911},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.874,"15min":0.86699999999999999},"min":{"1min":0.58499999999999996,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.99299999999999999,"5min":1.9099999999999999,"15min":2.0009999999999999},"last":0.70299999999999996}]},{"osd":7,"last update":"Mon Oct 30 14:52:23 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.84799999999999998,"15min":0.86099999999999999},"min":{"1min":0.65100000000000002,"5min":0.60299999999999998,"15min":0.57299999999999995},"max":{"1min":1.1639999999999999,"5min":1.2390000000000001,"15min":1.2989999999999999},"last":0.873},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.90100000000000002,"15min":0.90700000000000003},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.149,"5min":3.3809999999999998,"15min":3.3809999999999998},"last":0.59499999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575318,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472564,"kb_used_data":351212,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":93899276,"statfs":{"total":96636764160,"available":96152858624,"internally_reserved":0,"allocated":359641088,"data_stored":352841256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.47199999999999998,"15min":0.44800000000000001},"min":{"1min":0.29399999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.65500000000000003,"5min":2.4889999999999999,"15min":2.4889999999999999},"last":0.437},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.51000000000000001,"15min":0.54100000000000004},"min":{"1min":0.376,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.61699999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.46999999999999997}]},{"osd":2,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.64800000000000002,"15min":0.56399999999999995},"min":{"1min":0.32700000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69199999999999995,"5min":8.3160000000000007,"15min":8.3160000000000007},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.53400000000000003,"15min":0.48999999999999999},"min":{"1min":0.32500000000000001,"5min":0.27200000000000002,"15min":0.26800000000000002},"max":{"1min":0.628,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.38200000000000001}]},{"osd":3,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.60499999999999998,"15min":0.59299999999999997},"min":{"1min":0.371,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.70999999999999996,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.64700000000000002,"15min":0.63200000000000001},"min":{"1min":0.436,"5min":0.38100000000000001,"15min":0.36499999999999999},"max":{"1min":0.73299999999999998,"5min":3.54,"15min":3.54},"last":0.56399999999999995}]},{"osd":4,"last update":"Mon Oct 30 14:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.90800000000000003,"15min":0.82199999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.0209999999999999,"5min":10.525,"15min":10.525},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.96199999999999997,"15min":0.84099999999999997},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1910000000000001,"5min":10.571,"15min":10.571},"last":0.92600000000000005}]},{"osd":5,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.876,"15min":0.88400000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.169,"5min":3.4969999999999999,"15min":3.4969999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.872,"5min":0.85699999999999998,"15min":0.79800000000000004},"min":{"1min":0.63500000000000001,"5min":0.56000000000000005,"15min":0.49099999999999999},"max":{"1min":1.2,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.83599999999999997}]},{"osd":6,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.80500000000000005,"15min":0.80000000000000004},"min":{"1min":0.54200000000000004,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0820000000000001,"5min":1.875,"15min":1.875},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.89000000000000001,"15min":0.83299999999999996},"min":{"1min":0.59799999999999998,"5min":0.53100000000000003,"15min":0.51700000000000002},"max":{"1min":1.1439999999999999,"5min":5.3680000000000003,"15min":5.3680000000000003},"last":0.80800000000000005}]},{"osd":7,"last update":"Mon Oct 30 14:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":1.085,"15min":0.91200000000000003},"min":{"1min":0.626,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0600000000000001,"5min":19.885000000000002,"15min":19.885000000000002},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.87,"5min":1.0469999999999999,"15min":0.93100000000000005},"min":{"1min":0.61499999999999999,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.177,"5min":13.401999999999999,"15min":13.401999999999999},"last":0.872}]}]},{"osd":0,"up_from":13,"seq":55834575320,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557200,"kb_used_data":423368,"kb_used_omap":1,"kb_used_meta":133822,"kb_avail":93814640,"statfs":{"total":96636764160,"available":96066191360,"internally_reserved":0,"allocated":433528832,"data_stored":421882906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":137034182},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 14:52:19 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45500000000000002,"15min":0.44900000000000001},"min":{"1min":0.29899999999999999,"5min":0.28899999999999998,"15min":0.22500000000000001},"max":{"1min":0.61299999999999999,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.441,"5min":0.496,"15min":0.502},"min":{"1min":0.313,"5min":0.25700000000000001,"15min":0.23799999999999999},"max":{"1min":1.1659999999999999,"5min":2.8849999999999998,"15min":3.367},"last":0.41599999999999998}]},{"osd":2,"last update":"Mon Oct 30 14:52:19 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57999999999999996,"15min":0.53900000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.30499999999999999},"max":{"1min":0.73099999999999998,"5min":3.222,"15min":3.222},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.53000000000000003,"15min":0.55800000000000005},"min":{"1min":0.28599999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.81599999999999995,"5min":0.85799999999999998,"15min":1.095},"last":0.314}]},{"osd":3,"last update":"Mon Oct 30 14:52:19 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.56999999999999995,"15min":0.59799999999999998},"min":{"1min":0.34100000000000003,"5min":0.32000000000000001,"15min":0.29899999999999999},"max":{"1min":1.5389999999999999,"5min":6.4980000000000002,"15min":6.4980000000000002},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.56200000000000006,"15min":0.53100000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.28000000000000003},"max":{"1min":0.73699999999999999,"5min":2.024,"15min":2.024},"last":0.39300000000000002}]},{"osd":4,"last update":"Mon Oct 30 14:52:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.81699999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.0109999999999999,"5min":2.0659999999999998,"15min":2.8380000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.79500000000000004,"15min":0.85299999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.93700000000000006,"5min":2.1000000000000001,"15min":5.8959999999999999},"last":0.56599999999999995}]},{"osd":5,"last update":"Mon Oct 30 14:52:19 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.86099999999999999,"15min":0.85599999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.46300000000000002},"max":{"1min":1.242,"5min":1.278,"15min":6.4480000000000004},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.91900000000000004,"15min":0.86199999999999999},"min":{"1min":0.56599999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":2.6869999999999998},"last":0.79100000000000004}]},{"osd":6,"last update":"Mon Oct 30 14:52:19 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.81000000000000005,"15min":0.84199999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.0880000000000001,"5min":1.2150000000000001,"15min":4.9980000000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.85699999999999998,"15min":0.86199999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5980000000000001},"last":0.73999999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:52:19 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.85699999999999998,"15min":0.84699999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0629999999999999,"5min":3.6589999999999998,"15min":3.6589999999999998},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.92800000000000005,"15min":0.88300000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54100000000000004},"max":{"1min":1.28,"5min":1.28,"15min":2.6429999999999998},"last":0.77600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":109068288,"data_stored":109052564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163602432,"data_stored":163582366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":115884032,"data_stored":115864160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":209350656,"data_stored":209331981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":153219072,"data_stored":153204613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":215736320,"data_stored":215715446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":211091456,"data_stored":211074246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":201580544,"data_stored":201563796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323084288,"data_stored":311837872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":195256320,"data_stored":188856788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":201433088,"data_stored":195002091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310554624,"data_stored":299415921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":337338368,"data_stored":326229984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":335527936,"data_stored":324229915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":185139200,"data_stored":178241856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":297955328,"data_stored":287506221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:52:43.500 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-30T14:52:43.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.663+0000 7fbb4511d640 1 Processor -- start 2023-10-30T14:52:43.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.667+0000 7fbb4511d640 1 -- start start 2023-10-30T14:52:43.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.667+0000 7fbb4511d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb40153c40 0x7fbb4006d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:43.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.667+0000 7fbb4511d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbb4006dfc0 0x7fbb40069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:43.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.667+0000 7fbb4511d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbb40071ac0 0x7fbb4006a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:43.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.667+0000 7fbb4511d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb4006d8d0 con 0x7fbb4006dfc0 2023-10-30T14:52:43.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.667+0000 7fbb4511d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb4015cf10 con 0x7fbb40071ac0 2023-10-30T14:52:43.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.667+0000 7fbb4511d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb400722b0 con 0x7fbb40153c40 2023-10-30T14:52:43.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.667+0000 7fbb37fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbb4006dfc0 0x7fbb40069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:43.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.667+0000 7fbb37fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbb4006dfc0 0x7fbb40069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45010/0 (socket says 172.21.15.57:45010) 2023-10-30T14:52:43.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.667+0000 7fbb37fff640 1 -- 172.21.15.57:0/1964716524 learned_addr learned my addr 172.21.15.57:0/1964716524 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:52:43.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.667+0000 7fbb3ed9d640 1 --2- 172.21.15.57:0/1964716524 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb40153c40 0x7fbb4006d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:43.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.667+0000 7fbb3f59e640 1 --2- 172.21.15.57:0/1964716524 >> v2:172.21.15.88:3300/0 conn(0x7fbb40071ac0 0x7fbb4006a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:43.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.667+0000 7fbb37fff640 1 -- 172.21.15.57:0/1964716524 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb40153c40 msgr2=0x7fbb4006d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:52:43.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.667+0000 7fbb37fff640 1 --2- 172.21.15.57:0/1964716524 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb40153c40 0x7fbb4006d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.667+0000 7fbb37fff640 1 -- 172.21.15.57:0/1964716524 >> v2:172.21.15.88:3300/0 conn(0x7fbb40071ac0 msgr2=0x7fbb4006a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:52:43.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.667+0000 7fbb37fff640 1 --2- 172.21.15.57:0/1964716524 >> v2:172.21.15.88:3300/0 conn(0x7fbb40071ac0 0x7fbb4006a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.667+0000 7fbb37fff640 1 -- 172.21.15.57:0/1964716524 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb40072590 con 0x7fbb4006dfc0 2023-10-30T14:52:43.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.667+0000 7fbb3f59e640 1 --2- 172.21.15.57:0/1964716524 >> v2:172.21.15.88:3300/0 conn(0x7fbb40071ac0 0x7fbb4006a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:52:43.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.671+0000 7fbb37fff640 1 --2- 172.21.15.57:0/1964716524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb4006dfc0 0x7fbb40069e90 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fbb4006c040 tx=0x7fbb2801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=912e96eb4eb4830a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:43.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.671+0000 7fbb3e59c640 1 -- 172.21.15.57:0/1964716524 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb280a89b0 con 0x7fbb4006dfc0 2023-10-30T14:52:43.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.671+0000 7fbb3e59c640 1 -- 172.21.15.57:0/1964716524 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb280a8b10 con 0x7fbb4006dfc0 2023-10-30T14:52:43.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.671+0000 7fbb3e59c640 1 -- 172.21.15.57:0/1964716524 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb280a8ea0 con 0x7fbb4006dfc0 2023-10-30T14:52:43.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.671+0000 7fbb4511d640 1 -- 172.21.15.57:0/1964716524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb4006dfc0 msgr2=0x7fbb40069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:43.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.671+0000 7fbb4511d640 1 --2- 172.21.15.57:0/1964716524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb4006dfc0 0x7fbb40069e90 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fbb4006c040 tx=0x7fbb2801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.671+0000 7fbb4511d640 1 -- 172.21.15.57:0/1964716524 shutdown_connections 2023-10-30T14:52:43.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.671+0000 7fbb4511d640 1 --2- 172.21.15.57:0/1964716524 >> v2:172.21.15.88:3300/0 conn(0x7fbb40071ac0 0x7fbb4006a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.671+0000 7fbb4511d640 1 --2- 172.21.15.57:0/1964716524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb4006dfc0 0x7fbb40069e90 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.671+0000 7fbb4511d640 1 --2- 172.21.15.57:0/1964716524 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb40153c40 0x7fbb4006d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.671+0000 7fbb4511d640 1 -- 172.21.15.57:0/1964716524 >> 172.21.15.57:0/1964716524 conn(0x7fbb4006afa0 msgr2=0x7fbb4006b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:43.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.671+0000 7fbb4511d640 1 -- 172.21.15.57:0/1964716524 shutdown_connections 2023-10-30T14:52:43.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.671+0000 7fbb4511d640 1 -- 172.21.15.57:0/1964716524 wait complete. 2023-10-30T14:52:43.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.671+0000 7fbb4511d640 1 Processor -- start 2023-10-30T14:52:43.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.671+0000 7fbb4511d640 1 -- start start 2023-10-30T14:52:43.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb4511d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb4006dfc0 0x7fbb4013df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:43.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb4511d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb40071ac0 0x7fbb401e2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:43.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb4511d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb40153c40 0x7fbb40152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:43.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb4511d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4013e4b0 con 0x7fbb4006dfc0 2023-10-30T14:52:43.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb4511d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb401e33e0 con 0x7fbb40153c40 2023-10-30T14:52:43.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb4511d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb401f26a0 con 0x7fbb40071ac0 2023-10-30T14:52:43.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb3ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb4006dfc0 0x7fbb4013df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:43.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb3ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb4006dfc0 0x7fbb4013df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45026/0 (socket says 172.21.15.57:45026) 2023-10-30T14:52:43.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb3ed9d640 1 -- 172.21.15.57:0/775024970 learned_addr learned my addr 172.21.15.57:0/775024970 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:52:43.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb37fff640 1 --2- 172.21.15.57:0/775024970 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb40071ac0 0x7fbb401e2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:43.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb3f59e640 1 --2- 172.21.15.57:0/775024970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb40153c40 0x7fbb40152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:43.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb3ed9d640 1 -- 172.21.15.57:0/775024970 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb40071ac0 msgr2=0x7fbb401e2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:43.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb3ed9d640 1 --2- 172.21.15.57:0/775024970 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb40071ac0 0x7fbb401e2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb3ed9d640 1 -- 172.21.15.57:0/775024970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb40153c40 msgr2=0x7fbb40152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:43.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb3ed9d640 1 --2- 172.21.15.57:0/775024970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb40153c40 0x7fbb40152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb3ed9d640 1 -- 172.21.15.57:0/775024970 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb280094d0 con 0x7fbb4006dfc0 2023-10-30T14:52:43.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb3ed9d640 1 --2- 172.21.15.57:0/775024970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb4006dfc0 0x7fbb4013df70 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000d830 tx=0x7fbb3000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:43.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb377fe640 1 -- 172.21.15.57:0/775024970 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3000b6c0 con 0x7fbb4006dfc0 2023-10-30T14:52:43.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb377fe640 1 -- 172.21.15.57:0/775024970 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3000b820 con 0x7fbb4006dfc0 2023-10-30T14:52:43.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb4511d640 1 -- 172.21.15.57:0/775024970 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb401e3650 con 0x7fbb4006dfc0 2023-10-30T14:52:43.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb377fe640 1 -- 172.21.15.57:0/775024970 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3000bb90 con 0x7fbb4006dfc0 2023-10-30T14:52:43.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb4511d640 1 -- 172.21.15.57:0/775024970 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb4015c630 con 0x7fbb4006dfc0 2023-10-30T14:52:43.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb4511d640 1 -- 172.21.15.57:0/775024970 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb4004e170 con 0x7fbb4006dfc0 2023-10-30T14:52:43.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.675+0000 7fbb377fe640 1 -- 172.21.15.57:0/775024970 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbb3001e460 con 0x7fbb4006dfc0 2023-10-30T14:52:43.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.679+0000 7fbb377fe640 1 --2- 172.21.15.57:0/775024970 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbb200664a0 0x7fbb20068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:43.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.679+0000 7fbb377fe640 1 -- 172.21.15.57:0/775024970 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fbb30084d80 con 0x7fbb4006dfc0 2023-10-30T14:52:43.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.679+0000 7fbb37fff640 1 --2- 172.21.15.57:0/775024970 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbb200664a0 0x7fbb20068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:43.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.679+0000 7fbb37fff640 1 --2- 172.21.15.57:0/775024970 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbb200664a0 0x7fbb20068960 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fbb280b2000 tx=0x7fbb280aa070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:43.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.683+0000 7fbb377fe640 1 -- 172.21.15.57:0/775024970 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbb30051ae0 con 0x7fbb4006dfc0 2023-10-30T14:52:43.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.843+0000 7fbb4511d640 1 -- 172.21.15.57:0/775024970 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fbb4006dda0 con 0x7fbb4006dfc0 2023-10-30T14:52:43.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.847+0000 7fbb377fe640 1 -- 172.21.15.57:0/775024970 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) v1 ==== 123+0+0 (secure 0 0 0) 0x7fbb30055100 con 0x7fbb4006dfc0 2023-10-30T14:52:43.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.847+0000 7fbb377fe640 1 -- 172.21.15.57:0/775024970 <== mon.0 v2:172.21.15.57:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fbb30089070 con 0x7fbb4006dfc0 2023-10-30T14:52:43.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.851+0000 7fbb4511d640 1 -- 172.21.15.57:0/775024970 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbb200664a0 msgr2=0x7fbb20068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:43.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.851+0000 7fbb4511d640 1 --2- 172.21.15.57:0/775024970 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbb200664a0 0x7fbb20068960 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fbb280b2000 tx=0x7fbb280aa070 comp rx=0 tx=0).stop 2023-10-30T14:52:43.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.851+0000 7fbb4511d640 1 -- 172.21.15.57:0/775024970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb4006dfc0 msgr2=0x7fbb4013df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:43.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.851+0000 7fbb4511d640 1 --2- 172.21.15.57:0/775024970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb4006dfc0 0x7fbb4013df70 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000d830 tx=0x7fbb3000dcf0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.851+0000 7fbb4511d640 1 -- 172.21.15.57:0/775024970 shutdown_connections 2023-10-30T14:52:43.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.851+0000 7fbb4511d640 1 --2- 172.21.15.57:0/775024970 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fbb200664a0 0x7fbb20068960 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.851+0000 7fbb4511d640 1 --2- 172.21.15.57:0/775024970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb40153c40 0x7fbb40152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.851+0000 7fbb4511d640 1 --2- 172.21.15.57:0/775024970 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb40071ac0 0x7fbb401e2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.851+0000 7fbb4511d640 1 --2- 172.21.15.57:0/775024970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb4006dfc0 0x7fbb4013df70 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:43.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.851+0000 7fbb4511d640 1 -- 172.21.15.57:0/775024970 >> 172.21.15.57:0/775024970 conn(0x7fbb4006afa0 msgr2=0x7fbb4014e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:43.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.851+0000 7fbb4511d640 1 -- 172.21.15.57:0/775024970 shutdown_connections 2023-10-30T14:52:43.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:43.851+0000 7fbb4511d640 1 -- 172.21.15.57:0/775024970 wait complete. 2023-10-30T14:52:43.875 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-30T14:52:43.876 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:52:44.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.015+0000 7f61d15ee640 1 Processor -- start 2023-10-30T14:52:44.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.015+0000 7f61d15ee640 1 -- start start 2023-10-30T14:52:44.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.015+0000 7f61d15ee640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f61cc0c4a10 0x7f61cc15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.015+0000 7f61d15ee640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f61cc0c42a0 0x7f61cc0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.015+0000 7f61d15ee640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61cc159f20 con 0x7f61cc153c80 2023-10-30T14:52:44.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.015+0000 7f61d15ee640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61cc15dec0 con 0x7f61cc0c4a10 2023-10-30T14:52:44.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.015+0000 7f61d15ee640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61cc1599a0 con 0x7f61cc0c42a0 2023-10-30T14:52:44.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.015+0000 7f61caffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f61cc153c80 0x7f61cc15a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:59946/0 (socket says 172.21.15.57:59946) 2023-10-30T14:52:44.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.015+0000 7f61caffd640 1 -- 172.21.15.57:0/4104241778 learned_addr learned my addr 172.21.15.57:0/4104241778 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:52:44.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.015+0000 7f61c3fff640 1 --2- 172.21.15.57:0/4104241778 >> v2:172.21.15.57:3300/0 conn(0x7f61cc0c4a10 0x7f61cc15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.015+0000 7f61cb7fe640 1 --2- 172.21.15.57:0/4104241778 >> v2:172.21.15.88:3300/0 conn(0x7f61cc0c42a0 0x7f61cc0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.015+0000 7f61ca7fc640 1 -- 172.21.15.57:0/4104241778 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809127817 0 0) 0x7f61cc159f20 con 0x7f61cc153c80 2023-10-30T14:52:44.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.015+0000 7f61ca7fc640 1 -- 172.21.15.57:0/4104241778 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61b8003530 con 0x7f61cc153c80 2023-10-30T14:52:44.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.015+0000 7f61c3fff640 1 -- 172.21.15.57:0/4104241778 >> v2:172.21.15.88:3300/0 conn(0x7f61cc0c42a0 msgr2=0x7f61cc0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:52:44.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.015+0000 7f61c3fff640 1 --2- 172.21.15.57:0/4104241778 >> v2:172.21.15.88:3300/0 conn(0x7f61cc0c42a0 0x7f61cc0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.015+0000 7f61c3fff640 1 -- 172.21.15.57:0/4104241778 >> v1:172.21.15.57:6789/0 conn(0x7f61cc153c80 legacy=0x7f61cc15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:44.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.015+0000 7f61c3fff640 1 -- 172.21.15.57:0/4104241778 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61cc15d620 con 0x7f61cc0c4a10 2023-10-30T14:52:44.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.015+0000 7f61cb7fe640 1 --2- 172.21.15.57:0/4104241778 >> v2:172.21.15.88:3300/0 conn(0x7f61cc0c42a0 0x7f61cc0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:52:44.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.019+0000 7f61c3fff640 1 --2- 172.21.15.57:0/4104241778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61cc0c4a10 0x7f61cc15cf90 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f61cc0b1d40 tx=0x7f61b401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f00364571d591a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:44.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.019+0000 7f61ca7fc640 1 -- 172.21.15.57:0/4104241778 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b40a8950 con 0x7f61cc0c4a10 2023-10-30T14:52:44.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.019+0000 7f61ca7fc640 1 -- 172.21.15.57:0/4104241778 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f61b40a8ab0 con 0x7f61cc0c4a10 2023-10-30T14:52:44.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.019+0000 7f61ca7fc640 1 -- 172.21.15.57:0/4104241778 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b40a8e20 con 0x7f61cc0c4a10 2023-10-30T14:52:44.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.019+0000 7f61d15ee640 1 -- 172.21.15.57:0/4104241778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61cc0c4a10 msgr2=0x7f61cc15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:44.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.019+0000 7f61d15ee640 1 --2- 172.21.15.57:0/4104241778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61cc0c4a10 0x7f61cc15cf90 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f61cc0b1d40 tx=0x7f61b401ec50 comp rx=0 tx=0).stop 2023-10-30T14:52:44.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.019+0000 7f61d15ee640 1 -- 172.21.15.57:0/4104241778 shutdown_connections 2023-10-30T14:52:44.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.019+0000 7f61d15ee640 1 --2- 172.21.15.57:0/4104241778 >> v2:172.21.15.88:3300/0 conn(0x7f61cc0c42a0 0x7f61cc0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.019+0000 7f61d15ee640 1 --2- 172.21.15.57:0/4104241778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61cc0c4a10 0x7f61cc15cf90 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.019+0000 7f61d15ee640 1 -- 172.21.15.57:0/4104241778 >> 172.21.15.57:0/4104241778 conn(0x7f61cc0715c0 msgr2=0x7f61cc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:44.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.019+0000 7f61d15ee640 1 -- 172.21.15.57:0/4104241778 shutdown_connections 2023-10-30T14:52:44.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.019+0000 7f61d15ee640 1 -- 172.21.15.57:0/4104241778 wait complete. 2023-10-30T14:52:44.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61d15ee640 1 Processor -- start 2023-10-30T14:52:44.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61d15ee640 1 -- start start 2023-10-30T14:52:44.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61d15ee640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61cc0c42a0 0x7f61cc15b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61d15ee640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61cc0c4a10 0x7f61cc15bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61d15ee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61cc153c80 0x7f61cc149aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61d15ee640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc1f2f70 con 0x7f61cc0c4a10 2023-10-30T14:52:44.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61d15ee640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc1f29f0 con 0x7f61cc0c42a0 2023-10-30T14:52:44.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61d15ee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc1f2c90 con 0x7f61cc153c80 2023-10-30T14:52:44.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61c3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61cc0c4a10 0x7f61cc15bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61c3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61cc0c4a10 0x7f61cc15bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33138/0 (socket says 172.21.15.57:33138) 2023-10-30T14:52:44.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61c3fff640 1 -- 172.21.15.57:0/1600220984 learned_addr learned my addr 172.21.15.57:0/1600220984 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:52:44.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61caffd640 1 --2- 172.21.15.57:0/1600220984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61cc0c42a0 0x7f61cc15b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61cb7fe640 1 --2- 172.21.15.57:0/1600220984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61cc153c80 0x7f61cc149aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61c3fff640 1 -- 172.21.15.57:0/1600220984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61cc153c80 msgr2=0x7f61cc149aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:44.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61c3fff640 1 --2- 172.21.15.57:0/1600220984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61cc153c80 0x7f61cc149aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61c3fff640 1 -- 172.21.15.57:0/1600220984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61cc0c42a0 msgr2=0x7f61cc15b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:44.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61c3fff640 1 --2- 172.21.15.57:0/1600220984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61cc0c42a0 0x7f61cc15b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61c3fff640 1 -- 172.21.15.57:0/1600220984 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b4009420 con 0x7f61cc0c4a10 2023-10-30T14:52:44.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61c3fff640 1 --2- 172.21.15.57:0/1600220984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61cc0c4a10 0x7f61cc15bd20 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f61b4000c10 tx=0x7f61b401b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:44.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61c37fe640 1 -- 172.21.15.57:0/1600220984 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b40b1e00 con 0x7f61cc0c4a10 2023-10-30T14:52:44.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61c37fe640 1 -- 172.21.15.57:0/1600220984 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f61b40b9410 con 0x7f61cc0c4a10 2023-10-30T14:52:44.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61c37fe640 1 -- 172.21.15.57:0/1600220984 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b40b97e0 con 0x7f61cc0c4a10 2023-10-30T14:52:44.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61d15ee640 1 -- 172.21.15.57:0/1600220984 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61cc15ad00 con 0x7f61cc0c4a10 2023-10-30T14:52:44.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.023+0000 7f61d15ee640 1 -- 172.21.15.57:0/1600220984 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61cc15b2c0 con 0x7f61cc0c4a10 2023-10-30T14:52:44.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.027+0000 7f61d15ee640 1 -- 172.21.15.57:0/1600220984 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61cc15ab60 con 0x7f61cc0c4a10 2023-10-30T14:52:44.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.027+0000 7f61c37fe640 1 -- 172.21.15.57:0/1600220984 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f61b40b7040 con 0x7f61cc0c4a10 2023-10-30T14:52:44.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.027+0000 7f61c37fe640 1 --2- 172.21.15.57:0/1600220984 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f61ac0664d0 0x7f61ac068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.027+0000 7f61c37fe640 1 -- 172.21.15.57:0/1600220984 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f61b4122350 con 0x7f61cc0c4a10 2023-10-30T14:52:44.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.027+0000 7f61caffd640 1 --2- 172.21.15.57:0/1600220984 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f61ac0664d0 0x7f61ac068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.027+0000 7f61caffd640 1 --2- 172.21.15.57:0/1600220984 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f61ac0664d0 0x7f61ac068990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f61bc005890 tx=0x7f61bc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:44.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.031+0000 7f61c37fe640 1 -- 172.21.15.57:0/1600220984 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f61cc15ab60 con 0x7f61cc0c4a10 2023-10-30T14:52:44.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.219+0000 7f61d15ee640 1 -- 172.21.15.57:0/1600220984 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61cc15ab60 con 0x7f61cc0c4a10 2023-10-30T14:52:44.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.219+0000 7f61c37fe640 1 -- 172.21.15.57:0/1600220984 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+6502 (secure 0 0 0) 0x7f61cc15ab60 con 0x7f61cc0c4a10 2023-10-30T14:52:44.226 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:52:44.226 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":8211,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/3751121680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":3751121680},{"type":"v1","addr":"172.21.15.57:6836","nonce":3751121680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:51:19.061797+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8058":{"gid":8058,"name":"e","rank":0,"incarnation":185,"state":"up:active","state_seq":38,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:52:44.228 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 190 2023-10-30T14:52:44.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.223+0000 7f61d15ee640 1 -- 172.21.15.57:0/1600220984 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f61ac0664d0 msgr2=0x7f61ac068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:44.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.223+0000 7f61d15ee640 1 --2- 172.21.15.57:0/1600220984 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f61ac0664d0 0x7f61ac068990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f61bc005890 tx=0x7f61bc009040 comp rx=0 tx=0).stop 2023-10-30T14:52:44.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.223+0000 7f61d15ee640 1 -- 172.21.15.57:0/1600220984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61cc0c4a10 msgr2=0x7f61cc15bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:44.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.223+0000 7f61d15ee640 1 --2- 172.21.15.57:0/1600220984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61cc0c4a10 0x7f61cc15bd20 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f61b4000c10 tx=0x7f61b401b9f0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.223+0000 7f61d15ee640 1 -- 172.21.15.57:0/1600220984 shutdown_connections 2023-10-30T14:52:44.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.223+0000 7f61d15ee640 1 --2- 172.21.15.57:0/1600220984 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f61ac0664d0 0x7f61ac068990 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.223+0000 7f61d15ee640 1 --2- 172.21.15.57:0/1600220984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61cc153c80 0x7f61cc149aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.223+0000 7f61d15ee640 1 --2- 172.21.15.57:0/1600220984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61cc0c4a10 0x7f61cc15bd20 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.223+0000 7f61d15ee640 1 --2- 172.21.15.57:0/1600220984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61cc0c42a0 0x7f61cc15b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.223+0000 7f61d15ee640 1 -- 172.21.15.57:0/1600220984 >> 172.21.15.57:0/1600220984 conn(0x7f61cc0715c0 msgr2=0x7f61cc157810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:44.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.223+0000 7f61d15ee640 1 -- 172.21.15.57:0/1600220984 shutdown_connections 2023-10-30T14:52:44.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.223+0000 7f61d15ee640 1 -- 172.21.15.57:0/1600220984 wait complete. 2023-10-30T14:52:44.246 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:52:44.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.383+0000 7fc10d479640 1 Processor -- start 2023-10-30T14:52:44.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.383+0000 7fc10d479640 1 -- start start 2023-10-30T14:52:44.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc10d479640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc108153b00 0x7fc10815a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc10d479640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc10815a670 con 0x7fc10815d1e0 2023-10-30T14:52:44.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc10d479640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc10815cd50 con 0x7fc1080c5690 2023-10-30T14:52:44.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc10d479640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc108153ee0 con 0x7fc108153b00 2023-10-30T14:52:44.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc107fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc10815d1e0 0x7fc108163fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:59954/0 (socket says 172.21.15.57:59954) 2023-10-30T14:52:44.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc107fff640 1 -- 172.21.15.57:0/1310642925 learned_addr learned my addr 172.21.15.57:0/1310642925 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:52:44.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc106ffd640 1 --2- 172.21.15.57:0/1310642925 >> v2:172.21.15.88:3300/0 conn(0x7fc108153b00 0x7fc10815a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc105ffb640 1 -- 172.21.15.57:0/1310642925 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 840589931 0 0) 0x7fc1000014a0 con 0x7fc10815d1e0 2023-10-30T14:52:44.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc105ffb640 1 -- 172.21.15.57:0/1310642925 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0f0003530 con 0x7fc10815d1e0 2023-10-30T14:52:44.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc105ffb640 1 -- 172.21.15.57:0/1310642925 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3519223363 0 0) 0x7fc0f0003530 con 0x7fc10815d1e0 2023-10-30T14:52:44.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc105ffb640 1 -- 172.21.15.57:0/1310642925 >> v2:172.21.15.88:3300/0 conn(0x7fc108153b00 msgr2=0x7fc10815a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:52:44.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc105ffb640 1 --2- 172.21.15.57:0/1310642925 >> v2:172.21.15.88:3300/0 conn(0x7fc108153b00 0x7fc10815a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc105ffb640 1 -- 172.21.15.57:0/1310642925 >> v1:172.21.15.88:6789/0 conn(0x7fc1080c5690 legacy=0x7fc1080cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:44.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc105ffb640 1 -- 172.21.15.57:0/1310642925 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1080b39b0 con 0x7fc10815d1e0 2023-10-30T14:52:44.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc105ffb640 1 -- 172.21.15.57:0/1310642925 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fc100001b90 con 0x7fc10815d1e0 2023-10-30T14:52:44.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc105ffb640 1 -- 172.21.15.57:0/1310642925 >> v1:172.21.15.57:6789/0 conn(0x7fc10815d1e0 legacy=0x7fc108163fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:44.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc105ffb640 1 --2- 172.21.15.57:0/1310642925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f0003980 0x7fc0f0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc105ffb640 1 --2- 172.21.15.57:0/1310642925 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f00047d0 0x7fc0f0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc105ffb640 1 --2- 172.21.15.57:0/1310642925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0f0005190 0x7fc0f000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc105ffb640 1 -- 172.21.15.57:0/1310642925 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc108153ee0 con 0x7fc0f0003980 2023-10-30T14:52:44.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc105ffb640 1 -- 172.21.15.57:0/1310642925 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f000c030 con 0x7fc0f0005190 2023-10-30T14:52:44.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc105ffb640 1 -- 172.21.15.57:0/1310642925 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f000c1f0 con 0x7fc0f00047d0 2023-10-30T14:52:44.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc106ffd640 1 --2- 172.21.15.57:0/1310642925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f0003980 0x7fc0f0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc107fff640 1 --2- 172.21.15.57:0/1310642925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0f0005190 0x7fc0f000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc1067fc640 1 --2- 172.21.15.57:0/1310642925 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f00047d0 0x7fc0f0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc106ffd640 1 -- 172.21.15.57:0/1310642925 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f00047d0 msgr2=0x7fc0f0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:52:44.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc106ffd640 1 --2- 172.21.15.57:0/1310642925 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0f00047d0 0x7fc0f0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc106ffd640 1 -- 172.21.15.57:0/1310642925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0f0005190 msgr2=0x7fc0f000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:52:44.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc106ffd640 1 --2- 172.21.15.57:0/1310642925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0f0005190 0x7fc0f000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc106ffd640 1 -- 172.21.15.57:0/1310642925 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f0003530 con 0x7fc0f0003980 2023-10-30T14:52:44.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc107fff640 1 --2- 172.21.15.57:0/1310642925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0f0005190 0x7fc0f000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:52:44.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.387+0000 7fc107fff640 1 -- 172.21.15.57:0/1310642925 reap_dead start 2023-10-30T14:52:44.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.391+0000 7fc106ffd640 1 --2- 172.21.15.57:0/1310642925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f0003980 0x7fc0f0003da0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8009ae0 tx=0x7fc0f801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be75541f180cff2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:44.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.391+0000 7fc105ffb640 1 -- 172.21.15.57:0/1310642925 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f80a5530 con 0x7fc0f0003980 2023-10-30T14:52:44.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.391+0000 7fc105ffb640 1 -- 172.21.15.57:0/1310642925 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fc0f80a5690 con 0x7fc0f0003980 2023-10-30T14:52:44.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.391+0000 7fc105ffb640 1 -- 172.21.15.57:0/1310642925 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f80a5a60 con 0x7fc0f0003980 2023-10-30T14:52:44.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.391+0000 7fc10d479640 1 -- 172.21.15.57:0/1310642925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f0003980 msgr2=0x7fc0f0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:44.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.391+0000 7fc10d479640 1 --2- 172.21.15.57:0/1310642925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f0003980 0x7fc0f0003da0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8009ae0 tx=0x7fc0f801ca50 comp rx=0 tx=0).stop 2023-10-30T14:52:44.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.391+0000 7fc10d479640 1 -- 172.21.15.57:0/1310642925 shutdown_connections 2023-10-30T14:52:44.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.391+0000 7fc10d479640 1 --2- 172.21.15.57:0/1310642925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f0003980 0x7fc0f0003da0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.391+0000 7fc10d479640 1 -- 172.21.15.57:0/1310642925 >> 172.21.15.57:0/1310642925 conn(0x7fc10806cef0 msgr2=0x7fc10806d300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:44.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.391+0000 7fc10d479640 1 -- 172.21.15.57:0/1310642925 shutdown_connections 2023-10-30T14:52:44.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.391+0000 7fc10d479640 1 -- 172.21.15.57:0/1310642925 wait complete. 2023-10-30T14:52:44.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc10d479640 1 Processor -- start 2023-10-30T14:52:44.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc10d479640 1 -- start start 2023-10-30T14:52:44.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc10d479640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f0003980 0x7fc10815bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc10d479640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc10815c620 0x7fc108150080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc10d479640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1080c6c40 0x7fc1080c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc10d479640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1081505c0 con 0x7fc0f0003980 2023-10-30T14:52:44.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc10d479640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10815aac0 con 0x7fc1080c6c40 2023-10-30T14:52:44.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc10d479640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1081f2820 con 0x7fc10815c620 2023-10-30T14:52:44.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc106ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f0003980 0x7fc10815bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc106ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f0003980 0x7fc10815bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33144/0 (socket says 172.21.15.57:33144) 2023-10-30T14:52:44.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc106ffd640 1 -- 172.21.15.57:0/2300411435 learned_addr learned my addr 172.21.15.57:0/2300411435 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:52:44.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc1067fc640 1 --2- 172.21.15.57:0/2300411435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc10815c620 0x7fc108150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc107fff640 1 --2- 172.21.15.57:0/2300411435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1080c6c40 0x7fc1080c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc106ffd640 1 -- 172.21.15.57:0/2300411435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc10815c620 msgr2=0x7fc108150080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:44.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc106ffd640 1 --2- 172.21.15.57:0/2300411435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc10815c620 0x7fc108150080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc106ffd640 1 -- 172.21.15.57:0/2300411435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1080c6c40 msgr2=0x7fc1080c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:44.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc106ffd640 1 --2- 172.21.15.57:0/2300411435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1080c6c40 0x7fc1080c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc106ffd640 1 -- 172.21.15.57:0/2300411435 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1081f2b00 con 0x7fc0f0003980 2023-10-30T14:52:44.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc106ffd640 1 --2- 172.21.15.57:0/2300411435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f0003980 0x7fc10815bfc0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fc0f80b0040 tx=0x7fc0f80197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:44.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc0ef7fe640 1 -- 172.21.15.57:0/2300411435 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f80a5530 con 0x7fc0f0003980 2023-10-30T14:52:44.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc0ef7fe640 1 -- 172.21.15.57:0/2300411435 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fc0f8004220 con 0x7fc0f0003980 2023-10-30T14:52:44.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc10d479640 1 -- 172.21.15.57:0/2300411435 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1081f2de0 con 0x7fc0f0003980 2023-10-30T14:52:44.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc0ef7fe640 1 -- 172.21.15.57:0/2300411435 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8003620 con 0x7fc0f0003980 2023-10-30T14:52:44.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc10d479640 1 -- 172.21.15.57:0/2300411435 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1081f33a0 con 0x7fc0f0003980 2023-10-30T14:52:44.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.395+0000 7fc10d479640 1 -- 172.21.15.57:0/2300411435 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc10804e170 con 0x7fc0f0003980 2023-10-30T14:52:44.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.399+0000 7fc0ef7fe640 1 -- 172.21.15.57:0/2300411435 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc0f80b4070 con 0x7fc0f0003980 2023-10-30T14:52:44.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.399+0000 7fc0ef7fe640 1 --2- 172.21.15.57:0/2300411435 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0f40664d0 0x7fc0f4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.399+0000 7fc0ef7fe640 1 -- 172.21.15.57:0/2300411435 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7fc0f811e2b0 con 0x7fc0f0003980 2023-10-30T14:52:44.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.399+0000 7fc1067fc640 1 --2- 172.21.15.57:0/2300411435 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0f40664d0 0x7fc0f4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.399+0000 7fc1067fc640 1 --2- 172.21.15.57:0/2300411435 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0f40664d0 0x7fc0f4068990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fc1081594f0 tx=0x7fc0fc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:44.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.403+0000 7fc0ef7fe640 1 -- 172.21.15.57:0/2300411435 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc0f80eaef0 con 0x7fc0f0003980 2023-10-30T14:52:44.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.591+0000 7fc10d479640 1 -- 172.21.15.57:0/2300411435 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1081509c0 con 0x7fc0f0003980 2023-10-30T14:52:44.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.595+0000 7fc0ef7fe640 1 -- 172.21.15.57:0/2300411435 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+6502 (secure 0 0 0) 0x7fc0f80b2090 con 0x7fc0f0003980 2023-10-30T14:52:44.600 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:52:44.600 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3394468178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3394468178},{"type":"v1","addr":"172.21.15.88:6835","nonce":3394468178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1227890752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1227890752},{"type":"v1","addr":"172.21.15.88:6837","nonce":1227890752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6716,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1078372885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1078372885},{"type":"v1","addr":"172.21.15.88:6839","nonce":1078372885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":170},{"gid":7377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/697507447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":697507447},{"type":"v1","addr":"172.21.15.57:6834","nonce":697507447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":156},{"gid":8211,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/3751121680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":3751121680},{"type":"v1","addr":"172.21.15.57:6836","nonce":3751121680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T14:14:04.087067+0000","modified":"2023-10-30T14:51:19.061797+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8058":{"gid":8058,"name":"e","rank":0,"incarnation":185,"state":"up:active","state_seq":38,"addr":"172.21.15.57:6837/2594912006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2594912006},{"type":"v1","addr":"172.21.15.57:6837","nonce":2594912006}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:52:44.601 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 190 2023-10-30T14:52:44.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.595+0000 7fc10d479640 1 -- 172.21.15.57:0/2300411435 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0f40664d0 msgr2=0x7fc0f4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:44.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.595+0000 7fc10d479640 1 --2- 172.21.15.57:0/2300411435 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0f40664d0 0x7fc0f4068990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fc1081594f0 tx=0x7fc0fc009040 comp rx=0 tx=0).stop 2023-10-30T14:52:44.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.595+0000 7fc10d479640 1 -- 172.21.15.57:0/2300411435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f0003980 msgr2=0x7fc10815bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:44.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.595+0000 7fc10d479640 1 --2- 172.21.15.57:0/2300411435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f0003980 0x7fc10815bfc0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fc0f80b0040 tx=0x7fc0f80197f0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.595+0000 7fc10d479640 1 -- 172.21.15.57:0/2300411435 shutdown_connections 2023-10-30T14:52:44.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.595+0000 7fc10d479640 1 --2- 172.21.15.57:0/2300411435 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fc0f40664d0 0x7fc0f4068990 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.595+0000 7fc10d479640 1 --2- 172.21.15.57:0/2300411435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1080c6c40 0x7fc1080c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.595+0000 7fc10d479640 1 --2- 172.21.15.57:0/2300411435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc10815c620 0x7fc108150080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.595+0000 7fc10d479640 1 --2- 172.21.15.57:0/2300411435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc0f0003980 0x7fc10815bfc0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.595+0000 7fc10d479640 1 -- 172.21.15.57:0/2300411435 >> 172.21.15.57:0/2300411435 conn(0x7fc10806cef0 msgr2=0x7fc1080cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:44.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.595+0000 7fc10d479640 1 -- 172.21.15.57:0/2300411435 shutdown_connections 2023-10-30T14:52:44.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.595+0000 7fc10d479640 1 -- 172.21.15.57:0/2300411435 wait complete. 2023-10-30T14:52:44.620 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:52:44.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.759+0000 7f4db7817640 1 Processor -- start 2023-10-30T14:52:44.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.763+0000 7f4db7817640 1 -- start start 2023-10-30T14:52:44.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.763+0000 7f4db7817640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4db0153c80 0x7f4db015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.763+0000 7f4db7817640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4db00c4a10 0x7f4db00c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.763+0000 7f4db7817640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db0159f20 con 0x7f4db015a3b0 2023-10-30T14:52:44.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.763+0000 7f4db7817640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db015dec0 con 0x7f4db0153c80 2023-10-30T14:52:44.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.763+0000 7f4db7817640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db01599a0 con 0x7f4db00c4a10 2023-10-30T14:52:44.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.763+0000 7f4db55b3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4db0153c80 0x7f4db015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.763+0000 7f4db55b3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4db0153c80 0x7f4db015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33148/0 (socket says 172.21.15.57:33148) 2023-10-30T14:52:44.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.763+0000 7f4db55b3640 1 -- 172.21.15.57:0/3179757476 learned_addr learned my addr 172.21.15.57:0/3179757476 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:52:44.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.763+0000 7f4db4db2640 1 --2- 172.21.15.57:0/3179757476 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4db00c4a10 0x7f4db00c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.763+0000 7f4db55b3640 1 -- 172.21.15.57:0/3179757476 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4db00c4a10 msgr2=0x7f4db00c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:52:44.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.763+0000 7f4db55b3640 1 --2- 172.21.15.57:0/3179757476 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4db00c4a10 0x7f4db00c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.763+0000 7f4db55b3640 1 -- 172.21.15.57:0/3179757476 >> v1:172.21.15.88:6789/0 conn(0x7f4db015a3b0 legacy=0x7f4db00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:44.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.763+0000 7f4db55b3640 1 -- 172.21.15.57:0/3179757476 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db00c47e0 con 0x7f4db0153c80 2023-10-30T14:52:44.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.767+0000 7f4db55b3640 1 --2- 172.21.15.57:0/3179757476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4db0153c80 0x7f4db015cf90 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f4dac009870 tx=0x7f4dac01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14cceb9a5fcc8126 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:44.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.767+0000 7f4da7fff640 1 -- 172.21.15.57:0/3179757476 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac0a8a30 con 0x7f4db0153c80 2023-10-30T14:52:44.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.767+0000 7f4da7fff640 1 -- 172.21.15.57:0/3179757476 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f4dac0a8b90 con 0x7f4db0153c80 2023-10-30T14:52:44.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.767+0000 7f4da7fff640 1 -- 172.21.15.57:0/3179757476 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac0b0440 con 0x7f4db0153c80 2023-10-30T14:52:44.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.767+0000 7f4db7817640 1 -- 172.21.15.57:0/3179757476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4db0153c80 msgr2=0x7f4db015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:44.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.767+0000 7f4db7817640 1 --2- 172.21.15.57:0/3179757476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4db0153c80 0x7f4db015cf90 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f4dac009870 tx=0x7f4dac01ed30 comp rx=0 tx=0).stop 2023-10-30T14:52:44.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.767+0000 7f4db7817640 1 -- 172.21.15.57:0/3179757476 shutdown_connections 2023-10-30T14:52:44.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.767+0000 7f4db7817640 1 --2- 172.21.15.57:0/3179757476 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4db00c4a10 0x7f4db00c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.767+0000 7f4db7817640 1 --2- 172.21.15.57:0/3179757476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4db0153c80 0x7f4db015cf90 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.767+0000 7f4db7817640 1 -- 172.21.15.57:0/3179757476 >> 172.21.15.57:0/3179757476 conn(0x7f4db00715c0 msgr2=0x7f4db00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:44.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.767+0000 7f4db7817640 1 -- 172.21.15.57:0/3179757476 shutdown_connections 2023-10-30T14:52:44.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.767+0000 7f4db7817640 1 -- 172.21.15.57:0/3179757476 wait complete. 2023-10-30T14:52:44.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db7817640 1 Processor -- start 2023-10-30T14:52:44.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db7817640 1 -- start start 2023-10-30T14:52:44.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db7817640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4db00c4a10 0x7f4db0149ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db7817640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4db0153c80 0x7f4db015bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db7817640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4db015a3b0 0x7f4db015c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db7817640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db0155050 con 0x7f4db00c4a10 2023-10-30T14:52:44.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db7817640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db01ef950 con 0x7f4db0153c80 2023-10-30T14:52:44.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db7817640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db00c9450 con 0x7f4db015a3b0 2023-10-30T14:52:44.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db55b3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4db00c4a10 0x7f4db0149ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db55b3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4db00c4a10 0x7f4db0149ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33154/0 (socket says 172.21.15.57:33154) 2023-10-30T14:52:44.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db55b3640 1 -- 172.21.15.57:0/1907458688 learned_addr learned my addr 172.21.15.57:0/1907458688 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:52:44.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db4db2640 1 --2- 172.21.15.57:0/1907458688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4db0153c80 0x7f4db015bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db5db4640 1 --2- 172.21.15.57:0/1907458688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4db015a3b0 0x7f4db015c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db55b3640 1 -- 172.21.15.57:0/1907458688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4db015a3b0 msgr2=0x7f4db015c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:44.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db55b3640 1 --2- 172.21.15.57:0/1907458688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4db015a3b0 0x7f4db015c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db55b3640 1 -- 172.21.15.57:0/1907458688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4db0153c80 msgr2=0x7f4db015bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:44.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db55b3640 1 --2- 172.21.15.57:0/1907458688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4db0153c80 0x7f4db015bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db55b3640 1 -- 172.21.15.57:0/1907458688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac0094d0 con 0x7f4db00c4a10 2023-10-30T14:52:44.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db4db2640 1 --2- 172.21.15.57:0/1907458688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4db0153c80 0x7f4db015bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:52:44.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db55b3640 1 --2- 172.21.15.57:0/1907458688 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4db00c4a10 0x7f4db0149ea0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f4dac0b3040 tx=0x7f4dac01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:44.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4da5ffb640 1 -- 172.21.15.57:0/1907458688 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac0be070 con 0x7f4db00c4a10 2023-10-30T14:52:44.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4da5ffb640 1 -- 172.21.15.57:0/1907458688 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f4dac0ba400 con 0x7f4db00c4a10 2023-10-30T14:52:44.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db7817640 1 -- 172.21.15.57:0/1907458688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db0154440 con 0x7f4db00c4a10 2023-10-30T14:52:44.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4da5ffb640 1 -- 172.21.15.57:0/1907458688 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac0ba7f0 con 0x7f4db00c4a10 2023-10-30T14:52:44.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db7817640 1 -- 172.21.15.57:0/1907458688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db0154a00 con 0x7f4db00c4a10 2023-10-30T14:52:44.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.771+0000 7f4db7817640 1 -- 172.21.15.57:0/1907458688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db004e170 con 0x7f4db00c4a10 2023-10-30T14:52:44.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.775+0000 7f4da5ffb640 1 -- 172.21.15.57:0/1907458688 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4dac0c3410 con 0x7f4db00c4a10 2023-10-30T14:52:44.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.775+0000 7f4da5ffb640 1 --2- 172.21.15.57:0/1907458688 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4d940664a0 0x7f4d94068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:44.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.775+0000 7f4da5ffb640 1 -- 172.21.15.57:0/1907458688 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f4dac122230 con 0x7f4db00c4a10 2023-10-30T14:52:44.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.775+0000 7f4db4db2640 1 --2- 172.21.15.57:0/1907458688 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4d940664a0 0x7f4d94068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:44.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.775+0000 7f4db4db2640 1 --2- 172.21.15.57:0/1907458688 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4d940664a0 0x7f4d94068960 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4db01e92b0 tx=0x7f4da0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:44.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.779+0000 7f4da5ffb640 1 -- 172.21.15.57:0/1907458688 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4dac0eef90 con 0x7f4db00c4a10 2023-10-30T14:52:44.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.939+0000 7f4db7817640 1 -- 172.21.15.57:0/1907458688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4db00b1b20 con 0x7f4db00c4a10 2023-10-30T14:52:44.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.939+0000 7f4da5ffb640 1 -- 172.21.15.57:0/1907458688 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+19036 (secure 0 0 0) 0x7f4dac0f25b0 con 0x7f4db00c4a10 2023-10-30T14:52:44.944 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-30T14:52:44.944 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":67,"fsid":"e4ade530-0fca-4330-9824-3adb64478f80","created":"2023-10-30T14:13:39.852554+0000","modified":"2023-10-30T14:50:50.290577+0000","last_up_change":"2023-10-30T14:13:51.171287+0000","last_in_change":"2023-10-30T14:13:42.112020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T14:13:52.574639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T14:14:02.600122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T14:14:03.069429+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":53,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"febf18a3-5dae-4a27-855b-c0ec470c8488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6801","nonce":3287367848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6803","nonce":3287367848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6807","nonce":3287367848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":3287367848},{"type":"v1","addr":"172.21.15.57:6805","nonce":3287367848}]},"public_addr":"172.21.15.57:6801/3287367848","cluster_addr":"172.21.15.57:6803/3287367848","heartbeat_back_addr":"172.21.15.57:6807/3287367848","heartbeat_front_addr":"172.21.15.57:6805/3287367848","state":["exists","up"]},{"osd":1,"uuid":"049efcb1-8448-4df9-9193-22747f733904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6809","nonce":3340797836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6811","nonce":3340797836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6815","nonce":3340797836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3340797836},{"type":"v1","addr":"172.21.15.57:6813","nonce":3340797836}]},"public_addr":"172.21.15.57:6809/3340797836","cluster_addr":"172.21.15.57:6811/3340797836","heartbeat_back_addr":"172.21.15.57:6815/3340797836","heartbeat_front_addr":"172.21.15.57:6813/3340797836","state":["exists","up"]},{"osd":2,"uuid":"1da21529-8703-49f1-841e-e79fd29ab5b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6825","nonce":1700292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6827","nonce":1700292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6831","nonce":1700292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1700292},{"type":"v1","addr":"172.21.15.57:6829","nonce":1700292}]},"public_addr":"172.21.15.57:6825/1700292","cluster_addr":"172.21.15.57:6827/1700292","heartbeat_back_addr":"172.21.15.57:6831/1700292","heartbeat_front_addr":"172.21.15.57:6829/1700292","state":["exists","up"]},{"osd":3,"uuid":"e4b3c017-ffee-4175-9aae-f33482d1ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6817","nonce":2893420631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6819","nonce":2893420631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6823","nonce":2893420631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2893420631},{"type":"v1","addr":"172.21.15.57:6821","nonce":2893420631}]},"public_addr":"172.21.15.57:6817/2893420631","cluster_addr":"172.21.15.57:6819/2893420631","heartbeat_back_addr":"172.21.15.57:6823/2893420631","heartbeat_front_addr":"172.21.15.57:6821/2893420631","state":["exists","up"]},{"osd":4,"uuid":"1332a9e0-38b6-4d83-b2d5-f0cd2306e983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6807","nonce":2784816485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6811","nonce":2784816485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6815","nonce":2784816485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2784816485},{"type":"v1","addr":"172.21.15.88:6813","nonce":2784816485}]},"public_addr":"172.21.15.88:6807/2784816485","cluster_addr":"172.21.15.88:6811/2784816485","heartbeat_back_addr":"172.21.15.88:6815/2784816485","heartbeat_front_addr":"172.21.15.88:6813/2784816485","state":["exists","up"]},{"osd":5,"uuid":"d826e2f9-75cc-46db-b03c-03146644128a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6801","nonce":2682449812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6803","nonce":2682449812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6810","nonce":2682449812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":2682449812},{"type":"v1","addr":"172.21.15.88:6806","nonce":2682449812}]},"public_addr":"172.21.15.88:6801/2682449812","cluster_addr":"172.21.15.88:6803/2682449812","heartbeat_back_addr":"172.21.15.88:6810/2682449812","heartbeat_front_addr":"172.21.15.88:6806/2682449812","state":["exists","up"]},{"osd":6,"uuid":"fe6fce6c-0b9a-4c80-b21c-4c17db861adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6822","nonce":3340693897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6827","nonce":3340693897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6831","nonce":3340693897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3340693897},{"type":"v1","addr":"172.21.15.88:6829","nonce":3340693897}]},"public_addr":"172.21.15.88:6822/3340693897","cluster_addr":"172.21.15.88:6827/3340693897","heartbeat_back_addr":"172.21.15.88:6831/3340693897","heartbeat_front_addr":"172.21.15.88:6829/3340693897","state":["exists","up"]},{"osd":7,"uuid":"82de08f0-87e3-43fc-8fdf-79409a22ed2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6817","nonce":2862111199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6820","nonce":2862111199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6825","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6826","nonce":2862111199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6821","nonce":2862111199},{"type":"v1","addr":"172.21.15.88:6823","nonce":2862111199}]},"public_addr":"172.21.15.88:6817/2862111199","cluster_addr":"172.21.15.88:6820/2862111199","heartbeat_back_addr":"172.21.15.88:6826/2862111199","heartbeat_front_addr":"172.21.15.88:6823/2862111199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.634920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.773993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.725825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.761452+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.694274+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.650718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T14:13:47.769535+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6835/2613636524":"2023-10-31T14:50:50.290535+0000","172.21.15.57:6837/816502245":"2023-10-31T14:48:30.252448+0000","172.21.15.88:6838/1744808283":"2023-10-31T14:44:20.161100+0000","172.21.15.57:6834/2321408149":"2023-10-31T14:42:35.131496+0000","172.21.15.57:6833/816502245":"2023-10-31T14:48:30.252448+0000","172.21.15.57:6834/2912050965":"2023-10-31T14:23:54.764165+0000","172.21.15.57:6837/1988100930":"2023-10-31T14:20:34.693623+0000","172.21.15.88:6839/2945854794":"2023-10-31T14:18:54.654873+0000","172.21.15.57:6835/2427081813":"2023-10-31T14:26:24.805473+0000","172.21.15.57:6833/1988100930":"2023-10-31T14:20:34.693623+0000","172.21.15.88:6839/358521496":"2023-10-31T14:30:09.871507+0000","172.21.15.57:6836/2693548867":"2023-10-31T14:17:44.635085+0000","172.21.15.57:6832/2912050965":"2023-10-31T14:23:54.764165+0000","172.21.15.57:6832/3701655528":"2023-10-31T14:16:49.618077+0000","172.21.15.88:6838/358521496":"2023-10-31T14:30:09.871507+0000","172.21.15.57:6837/901968115":"2023-10-31T14:14:59.588353+0000","172.21.15.88:6839/1744808283":"2023-10-31T14:44:20.161100+0000","172.21.15.88:6838/2945854794":"2023-10-31T14:18:54.654873+0000","172.21.15.57:6834/3701655528":"2023-10-31T14:16:49.618077+0000","172.21.15.57:6835/2733323524":"2023-10-31T14:32:54.940701+0000","172.21.15.57:6836/2733323524":"2023-10-31T14:32:54.940701+0000","172.21.15.57:6835/2693548867":"2023-10-31T14:17:44.635085+0000","172.21.15.57:6832/2321408149":"2023-10-31T14:42:35.131496+0000","172.21.15.57:6833/862088802":"2023-10-31T14:27:49.829937+0000","172.21.15.88:6838/3604033338":"2023-10-31T14:35:29.980317+0000","172.21.15.57:6836/2613636524":"2023-10-31T14:50:50.290535+0000","172.21.15.88:6839/2390289388":"2023-10-31T14:39:45.086742+0000","172.21.15.57:6837/862088802":"2023-10-31T14:27:49.829937+0000","172.21.15.88:6838/2390289388":"2023-10-31T14:39:45.086742+0000","172.21.15.57:6832/2940934016":"2023-10-31T14:29:09.852381+0000","172.21.15.57:6834/2940934016":"2023-10-31T14:29:09.852381+0000","172.21.15.88:6839/3604033338":"2023-10-31T14:35:29.980317+0000","172.21.15.57:6833/901968115":"2023-10-31T14:14:59.588353+0000","172.21.15.57:6836/2427081813":"2023-10-31T14:26:24.805473+0000","172.21.15.57:6833/1843528309":"2023-10-31T14:38:15.025313+0000","172.21.15.57:6837/1843528309":"2023-10-31T14:38:15.025313+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:52:44.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.939+0000 7f4db7817640 1 -- 172.21.15.57:0/1907458688 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4d940664a0 msgr2=0x7f4d94068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:44.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.939+0000 7f4db7817640 1 --2- 172.21.15.57:0/1907458688 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4d940664a0 0x7f4d94068960 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4db01e92b0 tx=0x7f4da0008040 comp rx=0 tx=0).stop 2023-10-30T14:52:44.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.939+0000 7f4db7817640 1 -- 172.21.15.57:0/1907458688 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4db00c4a10 msgr2=0x7f4db0149ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:44.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.939+0000 7f4db7817640 1 --2- 172.21.15.57:0/1907458688 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4db00c4a10 0x7f4db0149ea0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f4dac0b3040 tx=0x7f4dac01bad0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.939+0000 7f4db7817640 1 -- 172.21.15.57:0/1907458688 shutdown_connections 2023-10-30T14:52:44.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.939+0000 7f4db7817640 1 --2- 172.21.15.57:0/1907458688 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f4d940664a0 0x7f4d94068960 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.939+0000 7f4db7817640 1 --2- 172.21.15.57:0/1907458688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4db015a3b0 0x7f4db015c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.939+0000 7f4db7817640 1 --2- 172.21.15.57:0/1907458688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4db0153c80 0x7f4db015bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.939+0000 7f4db7817640 1 --2- 172.21.15.57:0/1907458688 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4db00c4a10 0x7f4db0149ea0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:44.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.939+0000 7f4db7817640 1 -- 172.21.15.57:0/1907458688 >> 172.21.15.57:0/1907458688 conn(0x7f4db00715c0 msgr2=0x7f4db01589f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:44.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.939+0000 7f4db7817640 1 -- 172.21.15.57:0/1907458688 shutdown_connections 2023-10-30T14:52:44.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:44.939+0000 7f4db7817640 1 -- 172.21.15.57:0/1907458688 wait complete. 2023-10-30T14:52:44.966 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-30T14:52:45.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.107+0000 7f7e3971d640 1 Processor -- start 2023-10-30T14:52:45.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.107+0000 7f7e3971d640 1 -- start start 2023-10-30T14:52:45.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.107+0000 7f7e3971d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7e34151840 0x7f7e340c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:45.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.107+0000 7f7e3971d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7e34151c20 0x7f7e34072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:45.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.107+0000 7f7e3971d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e340cb8b0 0x7f7e34072540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:45.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.107+0000 7f7e3971d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e340c5030 con 0x7f7e34151c20 2023-10-30T14:52:45.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.107+0000 7f7e3971d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e34157830 con 0x7f7e34151840 2023-10-30T14:52:45.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.107+0000 7f7e3971d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e34157b10 con 0x7f7e340cb8b0 2023-10-30T14:52:45.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.107+0000 7f7e327fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7e34151c20 0x7f7e34072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:45.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.107+0000 7f7e327fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7e34151c20 0x7f7e34072000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33168/0 (socket says 172.21.15.57:33168) 2023-10-30T14:52:45.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.107+0000 7f7e327fc640 1 -- 172.21.15.57:0/2267720568 learned_addr learned my addr 172.21.15.57:0/2267720568 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:52:45.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.107+0000 7f7e32ffd640 1 --2- 172.21.15.57:0/2267720568 >> v2:172.21.15.88:3300/0 conn(0x7f7e34151840 0x7f7e340c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:45.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.107+0000 7f7e337fe640 1 --2- 172.21.15.57:0/2267720568 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e340cb8b0 0x7f7e34072540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:45.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.107+0000 7f7e327fc640 1 -- 172.21.15.57:0/2267720568 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e340cb8b0 msgr2=0x7f7e34072540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:52:45.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.107+0000 7f7e327fc640 1 --2- 172.21.15.57:0/2267720568 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e340cb8b0 0x7f7e34072540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:45.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.107+0000 7f7e327fc640 1 -- 172.21.15.57:0/2267720568 >> v2:172.21.15.88:3300/0 conn(0x7f7e34151840 msgr2=0x7f7e340c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:52:45.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.107+0000 7f7e327fc640 1 --2- 172.21.15.57:0/2267720568 >> v2:172.21.15.88:3300/0 conn(0x7f7e34151840 0x7f7e340c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:45.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.107+0000 7f7e327fc640 1 -- 172.21.15.57:0/2267720568 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e34157df0 con 0x7f7e34151c20 2023-10-30T14:52:45.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e327fc640 1 --2- 172.21.15.57:0/2267720568 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e34151c20 0x7f7e34072000 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f7e34072b40 tx=0x7f7e2801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60181c2e9d659bc1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:45.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e31ffb640 1 -- 172.21.15.57:0/2267720568 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e280a89b0 con 0x7f7e34151c20 2023-10-30T14:52:45.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e31ffb640 1 -- 172.21.15.57:0/2267720568 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f7e280a8b10 con 0x7f7e34151c20 2023-10-30T14:52:45.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e31ffb640 1 -- 172.21.15.57:0/2267720568 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e280b0440 con 0x7f7e34151c20 2023-10-30T14:52:45.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 -- 172.21.15.57:0/2267720568 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e34151c20 msgr2=0x7f7e34072000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:45.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 --2- 172.21.15.57:0/2267720568 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e34151c20 0x7f7e34072000 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f7e34072b40 tx=0x7f7e2801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:52:45.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 -- 172.21.15.57:0/2267720568 shutdown_connections 2023-10-30T14:52:45.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 --2- 172.21.15.57:0/2267720568 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e340cb8b0 0x7f7e34072540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:45.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 --2- 172.21.15.57:0/2267720568 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e34151c20 0x7f7e34072000 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:45.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 --2- 172.21.15.57:0/2267720568 >> v2:172.21.15.88:3300/0 conn(0x7f7e34151840 0x7f7e340c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:45.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 -- 172.21.15.57:0/2267720568 >> 172.21.15.57:0/2267720568 conn(0x7f7e340cac70 msgr2=0x7f7e340cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:45.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 -- 172.21.15.57:0/2267720568 shutdown_connections 2023-10-30T14:52:45.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 -- 172.21.15.57:0/2267720568 wait complete. 2023-10-30T14:52:45.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 Processor -- start 2023-10-30T14:52:45.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 -- start start 2023-10-30T14:52:45.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e340cb8b0 0x7f7e341f2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:45.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e34151840 0x7f7e34145290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:45.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e34151c20 0x7f7e3414d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:45.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e341ef270 con 0x7f7e34151840 2023-10-30T14:52:45.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e341f2d60 con 0x7f7e34151c20 2023-10-30T14:52:45.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e341ee8f0 con 0x7f7e340cb8b0 2023-10-30T14:52:45.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e327fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e34151840 0x7f7e34145290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:45.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e327fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e34151840 0x7f7e34145290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33184/0 (socket says 172.21.15.57:33184) 2023-10-30T14:52:45.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e327fc640 1 -- 172.21.15.57:0/2405794184 learned_addr learned my addr 172.21.15.57:0/2405794184 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:52:45.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e32ffd640 1 --2- 172.21.15.57:0/2405794184 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e340cb8b0 0x7f7e341f2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:45.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e337fe640 1 --2- 172.21.15.57:0/2405794184 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e34151c20 0x7f7e3414d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:45.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e327fc640 1 -- 172.21.15.57:0/2405794184 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e340cb8b0 msgr2=0x7f7e341f2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:45.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e327fc640 1 --2- 172.21.15.57:0/2405794184 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e340cb8b0 0x7f7e341f2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:45.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e327fc640 1 -- 172.21.15.57:0/2405794184 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e34151c20 msgr2=0x7f7e3414d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:45.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e327fc640 1 --2- 172.21.15.57:0/2405794184 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e34151c20 0x7f7e3414d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:45.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e327fc640 1 -- 172.21.15.57:0/2405794184 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e280094d0 con 0x7f7e34151840 2023-10-30T14:52:45.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e327fc640 1 --2- 172.21.15.57:0/2405794184 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e34151840 0x7f7e34145290 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f7e2801d3a0 tx=0x7f7e2801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:45.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e1b7fe640 1 -- 172.21.15.57:0/2405794184 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e280b0d60 con 0x7f7e34151840 2023-10-30T14:52:45.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e1b7fe640 1 -- 172.21.15.57:0/2405794184 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f7e280bb460 con 0x7f7e34151840 2023-10-30T14:52:45.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e1b7fe640 1 -- 172.21.15.57:0/2405794184 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e280bb830 con 0x7f7e34151840 2023-10-30T14:52:45.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 -- 172.21.15.57:0/2405794184 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e341dec30 con 0x7f7e34151840 2023-10-30T14:52:45.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 -- 172.21.15.57:0/2405794184 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e341eebb0 con 0x7f7e34151840 2023-10-30T14:52:45.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e3971d640 1 -- 172.21.15.57:0/2405794184 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e3404e170 con 0x7f7e34151840 2023-10-30T14:52:45.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.115+0000 7f7e1b7fe640 1 -- 172.21.15.57:0/2405794184 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7e280b83f0 con 0x7f7e34151840 2023-10-30T14:52:45.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.119+0000 7f7e1b7fe640 1 --2- 172.21.15.57:0/2405794184 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7e200664d0 0x7f7e20068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:45.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.119+0000 7f7e1b7fe640 1 -- 172.21.15.57:0/2405794184 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7373+0+0 (secure 0 0 0) 0x7f7e28121f20 con 0x7f7e34151840 2023-10-30T14:52:45.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.119+0000 7f7e32ffd640 1 --2- 172.21.15.57:0/2405794184 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7e200664d0 0x7f7e20068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:45.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.119+0000 7f7e32ffd640 1 --2- 172.21.15.57:0/2405794184 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7e200664d0 0x7f7e20068990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009b60 tx=0x7f7e24009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:45.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.123+0000 7f7e1b7fe640 1 -- 172.21.15.57:0/2405794184 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7e280eec10 con 0x7f7e34151840 2023-10-30T14:52:45.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.311+0000 7f7e3971d640 1 -- 172.21.15.57:0/2405794184 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7e34066510 con 0x7f7e34151840 2023-10-30T14:52:45.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.315+0000 7f7e1b7fe640 1 -- 172.21.15.57:0/2405794184 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v191) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7e280b5070 con 0x7f7e34151840 2023-10-30T14:52:45.321 INFO:teuthology.orchestra.run.smithi057.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-30T14:52:45.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.319+0000 7f7e3971d640 1 -- 172.21.15.57:0/2405794184 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7e200664d0 msgr2=0x7f7e20068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:45.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.319+0000 7f7e3971d640 1 --2- 172.21.15.57:0/2405794184 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7e200664d0 0x7f7e20068990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009b60 tx=0x7f7e24009190 comp rx=0 tx=0).stop 2023-10-30T14:52:45.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.319+0000 7f7e3971d640 1 -- 172.21.15.57:0/2405794184 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e34151840 msgr2=0x7f7e34145290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:45.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.319+0000 7f7e3971d640 1 --2- 172.21.15.57:0/2405794184 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e34151840 0x7f7e34145290 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f7e2801d3a0 tx=0x7f7e2801ba90 comp rx=0 tx=0).stop 2023-10-30T14:52:45.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.319+0000 7f7e3971d640 1 -- 172.21.15.57:0/2405794184 shutdown_connections 2023-10-30T14:52:45.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.319+0000 7f7e3971d640 1 --2- 172.21.15.57:0/2405794184 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f7e200664d0 0x7f7e20068990 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:45.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.319+0000 7f7e3971d640 1 --2- 172.21.15.57:0/2405794184 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e34151c20 0x7f7e3414d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:45.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.319+0000 7f7e3971d640 1 --2- 172.21.15.57:0/2405794184 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e34151840 0x7f7e34145290 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:45.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.319+0000 7f7e3971d640 1 --2- 172.21.15.57:0/2405794184 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e340cb8b0 0x7f7e341f2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:45.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.319+0000 7f7e3971d640 1 -- 172.21.15.57:0/2405794184 >> 172.21.15.57:0/2405794184 conn(0x7f7e340cac70 msgr2=0x7f7e34155590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:45.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.319+0000 7f7e3971d640 1 -- 172.21.15.57:0/2405794184 shutdown_connections 2023-10-30T14:52:45.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.319+0000 7f7e3971d640 1 -- 172.21.15.57:0/2405794184 wait complete. 2023-10-30T14:52:45.345 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-30T14:52:45.407 INFO:tasks.ceph.mds.e.smithi057.stdout:starting mds.e at 2023-10-30T14:52:45.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.491+0000 7f3132e54640 1 Processor -- start 2023-10-30T14:52:45.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.491+0000 7f3132e54640 1 -- start start 2023-10-30T14:52:45.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.491+0000 7f3132e54640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f312c156eb0 0x7f312c06cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:45.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.491+0000 7f3132e54640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f312c157290 0x7f312c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:45.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.491+0000 7f3132e54640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312c070360 con 0x7f312c0c8330 2023-10-30T14:52:45.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.491+0000 7f3132e54640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f312c06f4f0 con 0x7f312c157290 2023-10-30T14:52:45.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.491+0000 7f3132e54640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312c06b2e0 con 0x7f312c156eb0 2023-10-30T14:52:45.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.491+0000 7f3130bf0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f312c156eb0 0x7f312c06cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:45.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.491+0000 7f312bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f312c157290 0x7f312c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:45.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.491+0000 7f3130bf0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f312c156eb0 0x7f312c06cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.57:42146/0 (socket says 172.21.15.57:42146) 2023-10-30T14:52:45.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.491+0000 7f3130bf0640 1 -- 172.21.15.57:0/3198895122 learned_addr learned my addr 172.21.15.57:0/3198895122 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:52:45.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.495+0000 7f3130bf0640 1 -- 172.21.15.57:0/3198895122 >> v2:172.21.15.88:3300/0 conn(0x7f312c157290 msgr2=0x7f312c06efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:52:45.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.495+0000 7f3130bf0640 1 --2- 172.21.15.57:0/3198895122 >> v2:172.21.15.88:3300/0 conn(0x7f312c157290 0x7f312c06efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:45.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.495+0000 7f3130bf0640 1 -- 172.21.15.57:0/3198895122 >> v1:172.21.15.88:6789/0 conn(0x7f312c0c8330 legacy=0x7f312c0c8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:45.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.495+0000 7f3130bf0640 1 -- 172.21.15.57:0/3198895122 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c06fac0 con 0x7f312c156eb0 2023-10-30T14:52:45.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.495+0000 7f312bfff640 1 --2- 172.21.15.57:0/3198895122 >> v2:172.21.15.88:3300/0 conn(0x7f312c157290 0x7f312c06efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:52:45.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.495+0000 7f3130bf0640 1 --2- 172.21.15.57:0/3198895122 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f312c156eb0 0x7f312c06cb70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3120009870 tx=0x7f312001c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f122e0828d0be04 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:45.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.495+0000 7f312b7fe640 1 -- 172.21.15.57:0/3198895122 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31200ab650 con 0x7f312c156eb0 2023-10-30T14:52:45.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.495+0000 7f312b7fe640 1 -- 172.21.15.57:0/3198895122 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f31200ab7b0 con 0x7f312c156eb0 2023-10-30T14:52:45.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.495+0000 7f312b7fe640 1 -- 172.21.15.57:0/3198895122 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31200abb60 con 0x7f312c156eb0 2023-10-30T14:52:45.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3132e54640 1 -- 172.21.15.57:0/3198895122 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f312c156eb0 msgr2=0x7f312c06cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:45.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3132e54640 1 --2- 172.21.15.57:0/3198895122 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f312c156eb0 0x7f312c06cb70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3120009870 tx=0x7f312001c540 comp rx=0 tx=0).stop 2023-10-30T14:52:45.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3132e54640 1 -- 172.21.15.57:0/3198895122 shutdown_connections 2023-10-30T14:52:45.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3132e54640 1 --2- 172.21.15.57:0/3198895122 >> v2:172.21.15.88:3300/0 conn(0x7f312c157290 0x7f312c06efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:45.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3132e54640 1 --2- 172.21.15.57:0/3198895122 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f312c156eb0 0x7f312c06cb70 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:45.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3132e54640 1 -- 172.21.15.57:0/3198895122 >> 172.21.15.57:0/3198895122 conn(0x7f312c0723f0 msgr2=0x7f312c072800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:45.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3132e54640 1 -- 172.21.15.57:0/3198895122 shutdown_connections 2023-10-30T14:52:45.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3132e54640 1 -- 172.21.15.57:0/3198895122 wait complete. 2023-10-30T14:52:45.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3132e54640 1 Processor -- start 2023-10-30T14:52:45.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3132e54640 1 -- start start 2023-10-30T14:52:45.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3132e54640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f312c0c8330 0x7f312c146d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:45.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3132e54640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f312c156eb0 0x7f312c1f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:45.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3132e54640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f312c157290 0x7f312c1558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:45.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3132e54640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c14f250 con 0x7f312c0c8330 2023-10-30T14:52:45.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3132e54640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c152720 con 0x7f312c156eb0 2023-10-30T14:52:45.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3132e54640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312c0cb210 con 0x7f312c157290 2023-10-30T14:52:45.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3130bf0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f312c0c8330 0x7f312c146d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:45.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3130bf0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f312c0c8330 0x7f312c146d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33198/0 (socket says 172.21.15.57:33198) 2023-10-30T14:52:45.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3130bf0640 1 -- 172.21.15.57:0/2228428478 learned_addr learned my addr 172.21.15.57:0/2228428478 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:52:45.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f31313f1640 1 --2- 172.21.15.57:0/2228428478 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f312c157290 0x7f312c1558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:45.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f312bfff640 1 --2- 172.21.15.57:0/2228428478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f312c156eb0 0x7f312c1f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:45.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3130bf0640 1 -- 172.21.15.57:0/2228428478 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f312c157290 msgr2=0x7f312c1558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:45.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3130bf0640 1 --2- 172.21.15.57:0/2228428478 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f312c157290 0x7f312c1558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:45.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3130bf0640 1 -- 172.21.15.57:0/2228428478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f312c156eb0 msgr2=0x7f312c1f5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:45.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3130bf0640 1 --2- 172.21.15.57:0/2228428478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f312c156eb0 0x7f312c1f5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:45.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3130bf0640 1 -- 172.21.15.57:0/2228428478 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3118009420 con 0x7f312c0c8330 2023-10-30T14:52:45.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.499+0000 7f3130bf0640 1 --2- 172.21.15.57:0/2228428478 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f312c0c8330 0x7f312c146d40 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f31200ad040 tx=0x7f312001ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:45.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.503+0000 7f31297fa640 1 -- 172.21.15.57:0/2228428478 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31200ab650 con 0x7f312c0c8330 2023-10-30T14:52:45.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.503+0000 7f31297fa640 1 -- 172.21.15.57:0/2228428478 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f31200b4840 con 0x7f312c0c8330 2023-10-30T14:52:45.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.503+0000 7f31297fa640 1 -- 172.21.15.57:0/2228428478 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31200b4c10 con 0x7f312c0c8330 2023-10-30T14:52:45.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.503+0000 7f3132e54640 1 -- 172.21.15.57:0/2228428478 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31200094d0 con 0x7f312c0c8330 2023-10-30T14:52:45.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.503+0000 7f3132e54640 1 -- 172.21.15.57:0/2228428478 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f312c1f64e0 con 0x7f312c0c8330 2023-10-30T14:52:45.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.503+0000 7f3132e54640 1 -- 172.21.15.57:0/2228428478 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312c04e170 con 0x7f312c0c8330 2023-10-30T14:52:45.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.503+0000 7f31297fa640 1 -- 172.21.15.57:0/2228428478 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f31200b3380 con 0x7f312c0c8330 2023-10-30T14:52:45.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.503+0000 7f31297fa640 1 --2- 172.21.15.57:0/2228428478 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f31100664a0 0x7f3110068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:45.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.503+0000 7f31297fa640 1 -- 172.21.15.57:0/2228428478 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7459+0+0 (secure 0 0 0) 0x7f312011ec30 con 0x7f312c0c8330 2023-10-30T14:52:45.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.503+0000 7f312bfff640 1 --2- 172.21.15.57:0/2228428478 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f31100664a0 0x7f3110068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:45.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.507+0000 7f312bfff640 1 --2- 172.21.15.57:0/2228428478 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f31100664a0 0x7f3110068960 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f312c164060 tx=0x7f3118008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:45.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.507+0000 7f31297fa640 1 -- 172.21.15.57:0/2228428478 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f31200eb9c0 con 0x7f312c0c8330 2023-10-30T14:52:45.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:45.699+0000 7f3132e54640 1 -- 172.21.15.57:0/2228428478 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f312c0cdf20 con 0x7f312c0c8330 2023-10-30T14:52:46.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.399+0000 7f31297fa640 1 -- 172.21.15.57:0/2228428478 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v192) v1 ==== 104+0+0 (secure 0 0 0) 0x7f31200eefe0 con 0x7f312c0c8330 2023-10-30T14:52:46.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.403+0000 7f3132e54640 1 -- 172.21.15.57:0/2228428478 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f31100664a0 msgr2=0x7f3110068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:46.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.403+0000 7f3132e54640 1 --2- 172.21.15.57:0/2228428478 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f31100664a0 0x7f3110068960 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f312c164060 tx=0x7f3118008040 comp rx=0 tx=0).stop 2023-10-30T14:52:46.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.403+0000 7f3132e54640 1 -- 172.21.15.57:0/2228428478 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f312c0c8330 msgr2=0x7f312c146d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:46.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.403+0000 7f3132e54640 1 --2- 172.21.15.57:0/2228428478 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f312c0c8330 0x7f312c146d40 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f31200ad040 tx=0x7f312001ad80 comp rx=0 tx=0).stop 2023-10-30T14:52:46.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.403+0000 7f3132e54640 1 -- 172.21.15.57:0/2228428478 shutdown_connections 2023-10-30T14:52:46.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.403+0000 7f3132e54640 1 --2- 172.21.15.57:0/2228428478 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7f31100664a0 0x7f3110068960 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:46.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.403+0000 7f3132e54640 1 --2- 172.21.15.57:0/2228428478 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f312c157290 0x7f312c1558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:46.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.403+0000 7f3132e54640 1 --2- 172.21.15.57:0/2228428478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f312c156eb0 0x7f312c1f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:46.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.403+0000 7f3132e54640 1 --2- 172.21.15.57:0/2228428478 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f312c0c8330 0x7f312c146d40 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:46.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.403+0000 7f3132e54640 1 -- 172.21.15.57:0/2228428478 >> 172.21.15.57:0/2228428478 conn(0x7f312c0723f0 msgr2=0x7f312c1589b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:46.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.403+0000 7f3132e54640 1 -- 172.21.15.57:0/2228428478 shutdown_connections 2023-10-30T14:52:46.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.403+0000 7f3132e54640 1 -- 172.21.15.57:0/2228428478 wait complete. 2023-10-30T14:52:46.430 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-30T14:52:46.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.583+0000 7fe35fbaa640 1 Processor -- start 2023-10-30T14:52:46.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.583+0000 7fe35fbaa640 1 -- start start 2023-10-30T14:52:46.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.587+0000 7fe35fbaa640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe3580c5630 0x7fe3580c7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:46.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.587+0000 7fe35fbaa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe35815b310 0x7fe3580c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:46.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.587+0000 7fe35fbaa640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3580c71b0 con 0x7fe358151780 2023-10-30T14:52:46.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.587+0000 7fe35fbaa640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe358071c90 con 0x7fe3580c5630 2023-10-30T14:52:46.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.587+0000 7fe35fbaa640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3580c5c40 con 0x7fe35815b310 2023-10-30T14:52:46.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.587+0000 7fe35d145640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe3580c5630 0x7fe3580c7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:46.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.587+0000 7fe35d145640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe3580c5630 0x7fe3580c7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33212/0 (socket says 172.21.15.57:33212) 2023-10-30T14:52:46.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.587+0000 7fe35d145640 1 -- 172.21.15.57:0/942506513 learned_addr learned my addr 172.21.15.57:0/942506513 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:52:46.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.587+0000 7fe35e147640 1 --2- 172.21.15.57:0/942506513 >> v2:172.21.15.88:3300/0 conn(0x7fe35815b310 0x7fe3580c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:46.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.587+0000 7fe35d145640 1 -- 172.21.15.57:0/942506513 >> v2:172.21.15.88:3300/0 conn(0x7fe35815b310 msgr2=0x7fe3580c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:52:46.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.587+0000 7fe35d145640 1 --2- 172.21.15.57:0/942506513 >> v2:172.21.15.88:3300/0 conn(0x7fe35815b310 0x7fe3580c7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:46.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.587+0000 7fe35d145640 1 -- 172.21.15.57:0/942506513 >> v1:172.21.15.88:6789/0 conn(0x7fe358151780 legacy=0x7fe35815ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:52:46.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.587+0000 7fe35d145640 1 -- 172.21.15.57:0/942506513 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35806fd60 con 0x7fe3580c5630 2023-10-30T14:52:46.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.587+0000 7fe35d145640 1 --2- 172.21.15.57:0/942506513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3580c5630 0x7fe3580c7640 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fe3580c80c0 tx=0x7fe34801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee18554746afc1e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:46.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35c944640 1 -- 172.21.15.57:0/942506513 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3480aa610 con 0x7fe3580c5630 2023-10-30T14:52:46.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35c944640 1 -- 172.21.15.57:0/942506513 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fe3480a6040 con 0x7fe3580c5630 2023-10-30T14:52:46.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35c944640 1 -- 172.21.15.57:0/942506513 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3480aa9b0 con 0x7fe3580c5630 2023-10-30T14:52:46.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35fbaa640 1 -- 172.21.15.57:0/942506513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3580c5630 msgr2=0x7fe3580c7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:46.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35fbaa640 1 --2- 172.21.15.57:0/942506513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3580c5630 0x7fe3580c7640 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fe3580c80c0 tx=0x7fe34801c490 comp rx=0 tx=0).stop 2023-10-30T14:52:46.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35fbaa640 1 -- 172.21.15.57:0/942506513 shutdown_connections 2023-10-30T14:52:46.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35fbaa640 1 --2- 172.21.15.57:0/942506513 >> v2:172.21.15.88:3300/0 conn(0x7fe35815b310 0x7fe3580c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:46.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35fbaa640 1 --2- 172.21.15.57:0/942506513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3580c5630 0x7fe3580c7640 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:46.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35fbaa640 1 -- 172.21.15.57:0/942506513 >> 172.21.15.57:0/942506513 conn(0x7fe35806a6e0 msgr2=0x7fe35806d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:46.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35fbaa640 1 -- 172.21.15.57:0/942506513 shutdown_connections 2023-10-30T14:52:46.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35fbaa640 1 -- 172.21.15.57:0/942506513 wait complete. 2023-10-30T14:52:46.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35fbaa640 1 Processor -- start 2023-10-30T14:52:46.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35fbaa640 1 -- start start 2023-10-30T14:52:46.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35fbaa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3580c5630 0x7fe358160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:46.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35fbaa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe358151780 0x7fe358160aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:46.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35fbaa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe35815b310 0x7fe3580c03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:46.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35fbaa640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3580c9de0 con 0x7fe3580c5630 2023-10-30T14:52:46.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35fbaa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3580bff40 con 0x7fe358151780 2023-10-30T14:52:46.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35fbaa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3580c9540 con 0x7fe35815b310 2023-10-30T14:52:46.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35d946640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3580c5630 0x7fe358160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:46.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35d946640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3580c5630 0x7fe358160560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33214/0 (socket says 172.21.15.57:33214) 2023-10-30T14:52:46.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35d946640 1 -- 172.21.15.57:0/1820653319 learned_addr learned my addr 172.21.15.57:0/1820653319 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:52:46.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35d145640 1 --2- 172.21.15.57:0/1820653319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe358151780 0x7fe358160aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:46.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.591+0000 7fe35e147640 1 --2- 172.21.15.57:0/1820653319 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe35815b310 0x7fe3580c03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:46.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.595+0000 7fe35d946640 1 -- 172.21.15.57:0/1820653319 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe35815b310 msgr2=0x7fe3580c03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:46.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.595+0000 7fe35d946640 1 --2- 172.21.15.57:0/1820653319 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe35815b310 0x7fe3580c03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:46.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.595+0000 7fe35d946640 1 -- 172.21.15.57:0/1820653319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe358151780 msgr2=0x7fe358160aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:46.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.595+0000 7fe35d946640 1 --2- 172.21.15.57:0/1820653319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe358151780 0x7fe358160aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:46.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.595+0000 7fe35d946640 1 -- 172.21.15.57:0/1820653319 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3480094d0 con 0x7fe3580c5630 2023-10-30T14:52:46.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.595+0000 7fe35d946640 1 --2- 172.21.15.57:0/1820653319 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3580c5630 0x7fe358160560 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fe35400c450 tx=0x7fe35400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:46.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.595+0000 7fe34e7fc640 1 -- 172.21.15.57:0/1820653319 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe354012830 con 0x7fe3580c5630 2023-10-30T14:52:46.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.595+0000 7fe34e7fc640 1 -- 172.21.15.57:0/1820653319 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fe3540129b0 con 0x7fe3580c5630 2023-10-30T14:52:46.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.595+0000 7fe35fbaa640 1 -- 172.21.15.57:0/1820653319 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3580c9820 con 0x7fe3580c5630 2023-10-30T14:52:46.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.595+0000 7fe34e7fc640 1 -- 172.21.15.57:0/1820653319 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe354012d80 con 0x7fe3580c5630 2023-10-30T14:52:46.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.595+0000 7fe35fbaa640 1 -- 172.21.15.57:0/1820653319 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3580bfbd0 con 0x7fe3580c5630 2023-10-30T14:52:46.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.595+0000 7fe35fbaa640 1 -- 172.21.15.57:0/1820653319 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe35804e170 con 0x7fe3580c5630 2023-10-30T14:52:46.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.595+0000 7fe34e7fc640 1 -- 172.21.15.57:0/1820653319 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe35401c460 con 0x7fe3580c5630 2023-10-30T14:52:46.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.595+0000 7fe34e7fc640 1 --2- 172.21.15.57:0/1820653319 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe33c0664a0 0x7fe33c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:46.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.595+0000 7fe34e7fc640 1 -- 172.21.15.57:0/1820653319 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7459+0+0 (secure 0 0 0) 0x7fe354084480 con 0x7fe3580c5630 2023-10-30T14:52:46.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.599+0000 7fe35d145640 1 --2- 172.21.15.57:0/1820653319 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe33c0664a0 0x7fe33c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:46.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.599+0000 7fe35d145640 1 --2- 172.21.15.57:0/1820653319 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe33c0664a0 0x7fe33c068960 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fe3480ae000 tx=0x7fe3480b5040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:46.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.599+0000 7fe34e7fc640 1 -- 172.21.15.57:0/1820653319 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe354051110 con 0x7fe3580c5630 2023-10-30T14:52:46.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:46.763+0000 7fe35fbaa640 1 -- 172.21.15.57:0/1820653319 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe35806bc60 con 0x7fe3580c5630 2023-10-30T14:52:47.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.415+0000 7fe34e7fc640 1 -- 172.21.15.57:0/1820653319 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v69) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe354053040 con 0x7fe3580c5630 2023-10-30T14:52:47.415 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' removed 2023-10-30T14:52:47.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.415+0000 7fe35fbaa640 1 -- 172.21.15.57:0/1820653319 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe33c0664a0 msgr2=0x7fe33c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:47.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.415+0000 7fe35fbaa640 1 --2- 172.21.15.57:0/1820653319 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe33c0664a0 0x7fe33c068960 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fe3480ae000 tx=0x7fe3480b5040 comp rx=0 tx=0).stop 2023-10-30T14:52:47.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.415+0000 7fe35fbaa640 1 -- 172.21.15.57:0/1820653319 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3580c5630 msgr2=0x7fe358160560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:47.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.415+0000 7fe35fbaa640 1 --2- 172.21.15.57:0/1820653319 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3580c5630 0x7fe358160560 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fe35400c450 tx=0x7fe35400c910 comp rx=0 tx=0).stop 2023-10-30T14:52:47.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.415+0000 7fe35fbaa640 1 -- 172.21.15.57:0/1820653319 shutdown_connections 2023-10-30T14:52:47.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.415+0000 7fe35fbaa640 1 --2- 172.21.15.57:0/1820653319 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fe33c0664a0 0x7fe33c068960 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:47.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.415+0000 7fe35fbaa640 1 --2- 172.21.15.57:0/1820653319 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe35815b310 0x7fe3580c03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:47.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.415+0000 7fe35fbaa640 1 --2- 172.21.15.57:0/1820653319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe358151780 0x7fe358160aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:47.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.415+0000 7fe35fbaa640 1 --2- 172.21.15.57:0/1820653319 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3580c5630 0x7fe358160560 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:47.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.415+0000 7fe35fbaa640 1 -- 172.21.15.57:0/1820653319 >> 172.21.15.57:0/1820653319 conn(0x7fe35806a6e0 msgr2=0x7fe358150820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:47.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.415+0000 7fe35fbaa640 1 -- 172.21.15.57:0/1820653319 shutdown_connections 2023-10-30T14:52:47.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.415+0000 7fe35fbaa640 1 -- 172.21.15.57:0/1820653319 wait complete. 2023-10-30T14:52:47.457 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-30T14:52:47.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfac291640 1 Processor -- start 2023-10-30T14:52:47.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfac291640 1 -- start start 2023-10-30T14:52:47.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfac291640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcfa40c0eb0 0x7fcfa4169650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:47.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfac291640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa40c04f0 con 0x7fcfa4157eb0 2023-10-30T14:52:47.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfac291640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa406b740 con 0x7fcfa4158290 2023-10-30T14:52:47.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfac291640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa406c2e0 con 0x7fcfa40c0eb0 2023-10-30T14:52:47.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfaa02d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fcfa4157eb0 0x7fcfa406bbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:59968/0 (socket says 172.21.15.57:59968) 2023-10-30T14:52:47.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfaa02d640 1 -- 172.21.15.57:0/2744519536 learned_addr learned my addr 172.21.15.57:0/2744519536 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-10-30T14:52:47.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfaa82e640 1 --2- 172.21.15.57:0/2744519536 >> v2:172.21.15.88:3300/0 conn(0x7fcfa40c0eb0 0x7fcfa4169650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:47.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfa902b640 1 -- 172.21.15.57:0/2744519536 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88877258 0 0) 0x7fcfa40c04f0 con 0x7fcfa4157eb0 2023-10-30T14:52:47.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfa902b640 1 -- 172.21.15.57:0/2744519536 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf90003530 con 0x7fcfa4157eb0 2023-10-30T14:52:47.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfa902b640 1 -- 172.21.15.57:0/2744519536 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2232778166 0 0) 0x7fcf90003530 con 0x7fcfa4157eb0 2023-10-30T14:52:47.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfa902b640 1 -- 172.21.15.57:0/2744519536 >> v2:172.21.15.88:3300/0 conn(0x7fcfa40c0eb0 msgr2=0x7fcfa4169650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:52:47.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfa902b640 1 --2- 172.21.15.57:0/2744519536 >> v2:172.21.15.88:3300/0 conn(0x7fcfa40c0eb0 0x7fcfa4169650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:47.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfa902b640 1 -- 172.21.15.57:0/2744519536 >> v1:172.21.15.88:6789/0 conn(0x7fcfa4158290 legacy=0x7fcfa40c07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:47.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfa902b640 1 -- 172.21.15.57:0/2744519536 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa4162190 con 0x7fcfa4157eb0 2023-10-30T14:52:47.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfa902b640 1 -- 172.21.15.57:0/2744519536 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3525795191 0 0) 0x7fcfa0001680 con 0x7fcfa4157eb0 2023-10-30T14:52:47.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfa902b640 1 -- 172.21.15.57:0/2744519536 >> v1:172.21.15.57:6789/0 conn(0x7fcfa4157eb0 legacy=0x7fcfa406bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:47.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfa902b640 1 --2- 172.21.15.57:0/2744519536 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf90003980 0x7fcf90003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:47.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfa902b640 1 --2- 172.21.15.57:0/2744519536 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf900047d0 0x7fcf90004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:47.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfa902b640 1 --2- 172.21.15.57:0/2744519536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf90005190 0x7fcf9000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:47.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfa902b640 1 -- 172.21.15.57:0/2744519536 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa406c2e0 con 0x7fcf90003980 2023-10-30T14:52:47.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfa902b640 1 -- 172.21.15.57:0/2744519536 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9000c130 con 0x7fcf90005190 2023-10-30T14:52:47.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfa902b640 1 -- 172.21.15.57:0/2744519536 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9000c2f0 con 0x7fcf900047d0 2023-10-30T14:52:47.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfaa02d640 1 --2- 172.21.15.57:0/2744519536 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf90003980 0x7fcf90003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:47.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfa982c640 1 --2- 172.21.15.57:0/2744519536 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf900047d0 0x7fcf90004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:47.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.655+0000 7fcfaa82e640 1 --2- 172.21.15.57:0/2744519536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf90005190 0x7fcf9000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:47.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.659+0000 7fcfaa02d640 1 -- 172.21.15.57:0/2744519536 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf900047d0 msgr2=0x7fcf90004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:52:47.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.659+0000 7fcfaa02d640 1 --2- 172.21.15.57:0/2744519536 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf900047d0 0x7fcf90004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:47.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.659+0000 7fcfaa02d640 1 -- 172.21.15.57:0/2744519536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf90005190 msgr2=0x7fcf9000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:52:47.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.659+0000 7fcfaa02d640 1 --2- 172.21.15.57:0/2744519536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf90005190 0x7fcf9000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:47.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.659+0000 7fcfaa02d640 1 -- 172.21.15.57:0/2744519536 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf90003530 con 0x7fcf90003980 2023-10-30T14:52:47.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.659+0000 7fcfaa82e640 1 --2- 172.21.15.57:0/2744519536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf90005190 0x7fcf9000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:52:47.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.659+0000 7fcfaa82e640 1 -- 172.21.15.57:0/2744519536 reap_dead start 2023-10-30T14:52:47.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.659+0000 7fcfaa02d640 1 --2- 172.21.15.57:0/2744519536 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf90003980 0x7fcf90003da0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000c240 tx=0x7fcfa001cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff4fa1088c6fffa0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:47.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.659+0000 7fcfa902b640 1 -- 172.21.15.57:0/2744519536 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0005ae0 con 0x7fcf90003980 2023-10-30T14:52:47.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.659+0000 7fcfa902b640 1 -- 172.21.15.57:0/2744519536 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fcfa0005c40 con 0x7fcf90003980 2023-10-30T14:52:47.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.659+0000 7fcfa902b640 1 -- 172.21.15.57:0/2744519536 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa00a5b10 con 0x7fcf90003980 2023-10-30T14:52:47.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.659+0000 7fcfac291640 1 -- 172.21.15.57:0/2744519536 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf90003980 msgr2=0x7fcf90003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:47.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.659+0000 7fcfac291640 1 --2- 172.21.15.57:0/2744519536 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf90003980 0x7fcf90003da0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000c240 tx=0x7fcfa001cbd0 comp rx=0 tx=0).stop 2023-10-30T14:52:47.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.659+0000 7fcfac291640 1 -- 172.21.15.57:0/2744519536 shutdown_connections 2023-10-30T14:52:47.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.659+0000 7fcfac291640 1 --2- 172.21.15.57:0/2744519536 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf90003980 0x7fcf90003da0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:47.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.659+0000 7fcfac291640 1 -- 172.21.15.57:0/2744519536 >> 172.21.15.57:0/2744519536 conn(0x7fcfa406e150 msgr2=0x7fcfa4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:47.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.659+0000 7fcfac291640 1 -- 172.21.15.57:0/2744519536 shutdown_connections 2023-10-30T14:52:47.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfac291640 1 -- 172.21.15.57:0/2744519536 wait complete. 2023-10-30T14:52:47.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfac291640 1 Processor -- start 2023-10-30T14:52:47.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfac291640 1 -- start start 2023-10-30T14:52:47.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfac291640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf90003980 0x7fcfa4071900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:47.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfac291640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcfa4071e40 0x7fcfa40c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:47.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfac291640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfa40c5280 0x7fcfa40c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:47.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfac291640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa40cb120 con 0x7fcf90003980 2023-10-30T14:52:47.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfac291640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa415fd30 con 0x7fcfa40c5280 2023-10-30T14:52:47.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfac291640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa414e030 con 0x7fcfa4071e40 2023-10-30T14:52:47.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfaa02d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf90003980 0x7fcfa4071900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:47.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfaa02d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf90003980 0x7fcfa4071900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33224/0 (socket says 172.21.15.57:33224) 2023-10-30T14:52:47.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfaa02d640 1 -- 172.21.15.57:0/2105835009 learned_addr learned my addr 172.21.15.57:0/2105835009 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-30T14:52:47.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfa982c640 1 --2- 172.21.15.57:0/2105835009 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcfa4071e40 0x7fcfa40c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:47.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfaa82e640 1 --2- 172.21.15.57:0/2105835009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfa40c5280 0x7fcfa40c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:47.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfaa02d640 1 -- 172.21.15.57:0/2105835009 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcfa4071e40 msgr2=0x7fcfa40c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:47.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfaa02d640 1 --2- 172.21.15.57:0/2105835009 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcfa4071e40 0x7fcfa40c94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:47.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfaa02d640 1 -- 172.21.15.57:0/2105835009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfa40c5280 msgr2=0x7fcfa40c9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:47.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfaa02d640 1 --2- 172.21.15.57:0/2105835009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfa40c5280 0x7fcfa40c9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:47.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfaa02d640 1 -- 172.21.15.57:0/2105835009 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa40722d0 con 0x7fcf90003980 2023-10-30T14:52:47.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfaa82e640 1 --2- 172.21.15.57:0/2105835009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfa40c5280 0x7fcfa40c9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:52:47.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcfaa02d640 1 --2- 172.21.15.57:0/2105835009 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf90003980 0x7fcfa4071900 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000c240 tx=0x7fcf90005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:47.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcf9affd640 1 -- 172.21.15.57:0/2105835009 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa00b4070 con 0x7fcf90003980 2023-10-30T14:52:47.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcf9affd640 1 -- 172.21.15.57:0/2105835009 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fcfa0001540 con 0x7fcf90003980 2023-10-30T14:52:47.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.663+0000 7fcf9affd640 1 -- 172.21.15.57:0/2105835009 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa00a5af0 con 0x7fcf90003980 2023-10-30T14:52:47.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.667+0000 7fcfac291640 1 -- 172.21.15.57:0/2105835009 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa415ff40 con 0x7fcf90003980 2023-10-30T14:52:47.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.667+0000 7fcfac291640 1 -- 172.21.15.57:0/2105835009 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa40c8cc0 con 0x7fcf90003980 2023-10-30T14:52:47.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.667+0000 7fcfac291640 1 -- 172.21.15.57:0/2105835009 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa404e170 con 0x7fcf90003980 2023-10-30T14:52:47.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.667+0000 7fcf9affd640 1 -- 172.21.15.57:0/2105835009 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcfa00a5c50 con 0x7fcf90003980 2023-10-30T14:52:47.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.667+0000 7fcf9affd640 1 --2- 172.21.15.57:0/2105835009 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcf880664d0 0x7fcf88068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:52:47.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.667+0000 7fcf9affd640 1 -- 172.21.15.57:0/2105835009 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7021+0+0 (secure 0 0 0) 0x7fcfa011e2f0 con 0x7fcf90003980 2023-10-30T14:52:47.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.667+0000 7fcfa982c640 1 --2- 172.21.15.57:0/2105835009 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcf880664d0 0x7fcf88068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:52:47.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.671+0000 7fcfa982c640 1 --2- 172.21.15.57:0/2105835009 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcf880664d0 0x7fcf88068990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fcf94007fd0 tx=0x7fcf94009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:52:47.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.671+0000 7fcf9affd640 1 -- 172.21.15.57:0/2105835009 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcfa00eb1b0 con 0x7fcf90003980 2023-10-30T14:52:47.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:47.831+0000 7fcfac291640 1 -- 172.21.15.57:0/2105835009 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcfa40c6590 con 0x7fcf90003980 2023-10-30T14:52:48.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:48.423+0000 7fcf9affd640 1 -- 172.21.15.57:0/2105835009 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v70) v1 ==== 169+0+0 (secure 0 0 0) 0x7fcfa00ee7d0 con 0x7fcf90003980 2023-10-30T14:52:48.426 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' removed 2023-10-30T14:52:48.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:48.427+0000 7fcfac291640 1 -- 172.21.15.57:0/2105835009 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcf880664d0 msgr2=0x7fcf88068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:48.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:48.427+0000 7fcfac291640 1 --2- 172.21.15.57:0/2105835009 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcf880664d0 0x7fcf88068990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fcf94007fd0 tx=0x7fcf94009040 comp rx=0 tx=0).stop 2023-10-30T14:52:48.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:48.427+0000 7fcfac291640 1 -- 172.21.15.57:0/2105835009 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf90003980 msgr2=0x7fcfa4071900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:52:48.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:48.427+0000 7fcfac291640 1 --2- 172.21.15.57:0/2105835009 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf90003980 0x7fcfa4071900 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000c240 tx=0x7fcf90005800 comp rx=0 tx=0).stop 2023-10-30T14:52:48.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:48.427+0000 7fcfac291640 1 -- 172.21.15.57:0/2105835009 shutdown_connections 2023-10-30T14:52:48.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:48.427+0000 7fcfac291640 1 --2- 172.21.15.57:0/2105835009 >> [v2:172.21.15.88:6832/3881513431,v1:172.21.15.88:6833/3881513431] conn(0x7fcf880664d0 0x7fcf88068990 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:48.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:48.427+0000 7fcfac291640 1 --2- 172.21.15.57:0/2105835009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfa40c5280 0x7fcfa40c9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:48.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:48.427+0000 7fcfac291640 1 --2- 172.21.15.57:0/2105835009 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcfa4071e40 0x7fcfa40c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:48.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:48.427+0000 7fcfac291640 1 --2- 172.21.15.57:0/2105835009 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf90003980 0x7fcfa4071900 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:52:48.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:48.427+0000 7fcfac291640 1 -- 172.21.15.57:0/2105835009 >> 172.21.15.57:0/2105835009 conn(0x7fcfa406e150 msgr2=0x7fcfa4155aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:52:48.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:48.427+0000 7fcfac291640 1 -- 172.21.15.57:0/2105835009 shutdown_connections 2023-10-30T14:52:48.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-30T14:52:48.427+0000 7fcfac291640 1 -- 172.21.15.57:0/2105835009 wait complete. 2023-10-30T14:52:48.449 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-30T14:52:48.449 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T14:52:48.450 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:52:48.506 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T14:52:48.507 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T14:52:48.507 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:52:48.558 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T14:52:48.559 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T14:52:48.559 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:52:48.618 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T14:52:48.619 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-30T14:52:48.619 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:52:48.633 INFO:tasks.ceph.mds.b:Stopped 2023-10-30T14:52:48.633 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T14:52:48.634 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:52:48.689 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T14:52:48.689 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T14:52:48.690 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:52:48.745 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T14:52:48.745 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-30T14:52:48.746 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-30T14:52:48.747 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:52:48.837 INFO:tasks.ceph.osd.0:Stopped 2023-10-30T14:52:48.838 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-30T14:52:48.838 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:52:48.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2023-10-30T14:52:48.949 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~0s 2023-10-30T14:52:54.940 INFO:tasks.ceph.osd.1:Stopped 2023-10-30T14:52:54.941 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-30T14:52:54.942 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:52:55.035 INFO:tasks.ceph.osd.2:Stopped 2023-10-30T14:52:55.035 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-30T14:52:55.036 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:52:55.057 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-30T14:52:55.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2023-10-30T14:52:55.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f has been restored 2023-10-30T14:53:00.965 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-10-30T14:53:00.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-30T14:53:01.138 INFO:tasks.ceph.osd.3:Stopped 2023-10-30T14:53:01.138 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-30T14:53:01.139 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:53:01.204 INFO:tasks.ceph.osd.4:Stopped 2023-10-30T14:53:01.205 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-30T14:53:01.205 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:53:06.672 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-10-30T14:53:06.672 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2023-10-30T14:53:07.307 INFO:tasks.ceph.osd.5:Stopped 2023-10-30T14:53:07.308 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-30T14:53:07.308 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:53:07.387 INFO:tasks.ceph.osd.6:Stopped 2023-10-30T14:53:07.387 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-30T14:53:07.388 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:53:12.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-10-30T14:53:12.177 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-10-30T14:53:12.431 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:53:12.435 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:53:13.490 INFO:tasks.ceph.osd.7:Stopped 2023-10-30T14:53:13.491 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-30T14:53:13.491 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-30T14:53:13.491 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:53:13.549 INFO:tasks.ceph.mgr.y:Stopped 2023-10-30T14:53:13.549 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-30T14:53:13.550 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:53:13.580 INFO:tasks.ceph.mgr.x:Stopped 2023-10-30T14:53:13.580 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-30T14:53:13.581 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T14:53:13.581 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:53:13.604 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T14:53:13.604 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-30T14:53:13.604 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:53:13.639 INFO:tasks.ceph.mon.b:Stopped 2023-10-30T14:53:13.639 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-30T14:53:13.640 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:53:13.695 INFO:tasks.ceph.mon.c:Stopped 2023-10-30T14:53:13.695 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-30T14:53:13.696 DEBUG:teuthology.orchestra.run.smithi057:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v 'Replacing daemon mds' | head -n 1 2023-10-30T14:53:13.738 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi057.front.sepia.ceph.com 2023-10-30T14:53:13.738 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-30T14:53:14.039 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi057.front.sepia.ceph.com 2023-10-30T14:53:14.039 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-30T14:53:14.256 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi057.front.sepia.ceph.com 2023-10-30T14:53:14.257 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-30T14:53:14.445 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi057.front.sepia.ceph.com 2023-10-30T14:53:14.446 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-30T14:53:14.626 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-30T14:53:14.627 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-30T14:53:14.911 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-30T14:53:14.912 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-30T14:53:15.123 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-30T14:53:15.124 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-30T14:53:15.269 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-30T14:53:15.269 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-30T14:53:15.449 INFO:tasks.ceph:Archiving mon data... 2023-10-30T14:53:15.452 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441053/data/mon.a.tgz 2023-10-30T14:53:15.452 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2023-10-30T14:53:15.462 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.YRoBICUxTw 2023-10-30T14:53:15.462 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.YRoBICUxTw 2023-10-30T14:53:15.613 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.YRoBICUxTw 2023-10-30T14:53:15.695 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.YRoBICUxTw is 431KB 2023-10-30T14:53:15.760 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.YRoBICUxTw 2023-10-30T14:53:15.770 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441053/data/mon.b.tgz 2023-10-30T14:53:15.770 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2023-10-30T14:53:15.776 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.DZNOYI1ZHN 2023-10-30T14:53:15.777 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.DZNOYI1ZHN 2023-10-30T14:53:15.920 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.DZNOYI1ZHN 2023-10-30T14:53:15.984 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.DZNOYI1ZHN is 441KB 2023-10-30T14:53:16.047 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.DZNOYI1ZHN 2023-10-30T14:53:16.052 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441053/data/mon.c.tgz 2023-10-30T14:53:16.053 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2023-10-30T14:53:16.101 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.xySDQvTveG 2023-10-30T14:53:16.102 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.xySDQvTveG 2023-10-30T14:53:16.249 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.xySDQvTveG 2023-10-30T14:53:16.312 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.xySDQvTveG is 441KB 2023-10-30T14:53:16.371 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.xySDQvTveG 2023-10-30T14:53:16.376 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-30T14:53:16.377 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T14:53:16.381 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T14:53:16.434 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-30T14:53:16.434 DEBUG:teuthology.orchestra.run.smithi057:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T14:53:16.445 DEBUG:teuthology.orchestra.run.smithi088:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T14:53:16.486 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-30T14:53:16.507 INFO:teuthology.orchestra.run.smithi088.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-30T14:53:16.511 INFO:tasks.ceph:Archiving crash dumps... 2023-10-30T14:53:16.513 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441053/remote/smithi057/crash 2023-10-30T14:53:16.514 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T14:53:16.556 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441053/remote/smithi088/crash 2023-10-30T14:53:16.557 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T14:53:16.569 INFO:tasks.ceph:Shutting down logrotate 2023-10-30T14:53:16.570 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:53:16.605 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:53:16.653 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-30T14:53:16.719 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-30T14:53:16.733 INFO:tasks.ceph:Compressing logs... 2023-10-30T14:53:16.733 DEBUG:teuthology.orchestra.run.smithi057:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T14:53:16.769 DEBUG:teuthology.orchestra.run.smithi088:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T14:53:16.787 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44506.log 2023-10-30T14:53:16.788 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47745.log 2023-10-30T14:53:16.789 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43904.log 2023-10-30T14:53:16.790 INFO:teuthology.orchestra.run.smithi057.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44506.log.gz 2023-10-30T14:53:16.790 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.47745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56015.log 2023-10-30T14:53:16.791 INFO:teuthology.orchestra.run.smithi057.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47745.log.gz 2023-10-30T14:53:16.792 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.43904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46568.log 2023-10-30T14:53:16.792 INFO:teuthology.orchestra.run.smithi057.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43904.log.gz 2023-10-30T14:53:16.793 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58060.log 2023-10-30T14:53:16.793 INFO:teuthology.orchestra.run.smithi057.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56015.log.gz 2023-10-30T14:53:16.794 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46568.log: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.46568.log.gz 2023-10-30T14:53:16.795 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.62987.log 2023-10-30T14:53:16.796 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44303.log 2023-10-30T14:53:16.797 INFO:teuthology.orchestra.run.smithi057.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58060.log.gz 2023-10-30T14:53:16.797 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62987.log: gzip -5 88.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.62987.log.gz 2023-10-30T14:53:16.797 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.46460.log 2023-10-30T14:53:16.799 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38675.log 2023-10-30T14:53:16.799 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38747.log 2023-10-30T14:53:16.800 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38699.log 2023-10-30T14:53:16.800 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38675.log: /var/log/ceph/ceph-client.admin.38747.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-30T14:53:16.800 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38699.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38675.log.gz 2023-10-30T14:53:16.801 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38747.log.gz 2023-10-30T14:53:16.801 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38699.log.gz 2023-10-30T14:53:16.801 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-30T14:53:16.801 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38795.log 2023-10-30T14:53:16.802 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-30T14:53:16.802 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-30T14:53:16.802 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38795.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2023-10-30T14:53:16.802 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38795.log.gz 2023-10-30T14:53:16.803 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T14:53:16.805 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-30T14:53:16.805 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T14:53:16.805 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-10-30T14:53:16.808 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46131.log 2023-10-30T14:53:16.809 INFO:teuthology.orchestra.run.smithi057.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44303.log.gz 2023-10-30T14:53:16.809 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46460.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46460.log.gz 2023-10-30T14:53:16.809 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52838.log 2023-10-30T14:53:16.810 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61192.log 2023-10-30T14:53:16.810 INFO:teuthology.orchestra.run.smithi057.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46131.log.gz 2023-10-30T14:53:16.810 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52838.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52838.log.gz -5 2023-10-30T14:53:16.811 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.62358.log 2023-10-30T14:53:16.811 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61192.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61192.log.gz 2023-10-30T14:53:16.811 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52592.log 2023-10-30T14:53:16.811 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62358.log: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.62358.log.gz 2023-10-30T14:53:16.812 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51378.log 2023-10-30T14:53:16.812 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61324.log 2023-10-30T14:53:16.812 INFO:teuthology.orchestra.run.smithi057.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52592.log.gz 2023-10-30T14:53:16.812 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.51378.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51378.log.gz 2023-10-30T14:53:16.813 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50431.log 2023-10-30T14:53:16.813 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61324.log: gzip 89.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.61324.log.gz --verbose 2023-10-30T14:53:16.813 INFO:teuthology.orchestra.run.smithi057.stderr: -- /var/log/ceph/ceph-client.admin.57643.log 2023-10-30T14:53:16.814 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50431.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50431.log.gz 2023-10-30T14:53:16.814 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45011.log 2023-10-30T14:53:16.814 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53852.log 2023-10-30T14:53:16.814 INFO:teuthology.orchestra.run.smithi057.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57643.log.gz 2023-10-30T14:53:16.815 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45011.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45011.log.gz 2023-10-30T14:53:16.815 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46434.log 2023-10-30T14:53:16.818 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph.audit.log: 94.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T14:53:16.819 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38651.log 2023-10-30T14:53:16.819 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mds.d.log: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T14:53:16.819 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-10-30T14:53:16.820 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38651.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38651.log.gz 2023-10-30T14:53:16.820 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-30T14:53:16.820 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mds.b.log: 94.3% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-10-30T14:53:16.820 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-30T14:53:16.823 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mon.b.log: 94.3% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-10-30T14:53:16.824 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38723.log 2023-10-30T14:53:16.825 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55501.log 2023-10-30T14:53:16.826 INFO:teuthology.orchestra.run.smithi057.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53852.log.gz 2023-10-30T14:53:16.826 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37957.log 2023-10-30T14:53:16.826 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46434.log.gz 2023-10-30T14:53:16.826 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55501.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55501.log.gz 2023-10-30T14:53:16.827 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59921.log 2023-10-30T14:53:16.827 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.37957.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-30T14:53:16.827 INFO:teuthology.orchestra.run.smithi057.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.37957.log.gz 2023-10-30T14:53:16.828 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59921.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59921.log.gz 2023-10-30T14:53:16.828 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57095.log 2023-10-30T14:53:16.828 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62722.log 2023-10-30T14:53:16.828 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57095.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57095.log.gz 2023-10-30T14:53:16.829 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56100.log 2023-10-30T14:53:16.829 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62722.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62722.log.gz 2023-10-30T14:53:16.829 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.56620.log 2023-10-30T14:53:16.829 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56100.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.56100.log.gz 2023-10-30T14:53:16.830 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61541.log 2023-10-30T14:53:16.830 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56620.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56620.log.gz 2023-10-30T14:53:16.830 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43876.log 2023-10-30T14:53:16.831 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61541.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61541.log.gz 2023-10-30T14:53:16.831 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59153.log 2023-10-30T14:53:16.831 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.43876.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43876.log.gz 2023-10-30T14:53:16.831 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60265.log 2023-10-30T14:53:16.832 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44291.log 87.7% 2023-10-30T14:53:16.832 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59153.log.gz 2023-10-30T14:53:16.832 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53986.log 2023-10-30T14:53:16.833 INFO:teuthology.orchestra.run.smithi057.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60265.log.gz 2023-10-30T14:53:16.833 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44291.log: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.44291.log.gz 2023-10-30T14:53:16.833 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53719.log 2023-10-30T14:53:16.833 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53986.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53986.log.gz -5 2023-10-30T14:53:16.834 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49054.log 2023-10-30T14:53:16.834 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38771.log 2023-10-30T14:53:16.838 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50958.log 2023-10-30T14:53:16.838 INFO:teuthology.orchestra.run.smithi057.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53719.log.gz 2023-10-30T14:53:16.838 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.49054.log: gzip -5 88.4% --verbose -- replaced with /var/log/ceph/ceph-client.admin.49054.log.gz -- 2023-10-30T14:53:16.839 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.58767.log 2023-10-30T14:53:16.839 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50958.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.50958.log.gz 2023-10-30T14:53:16.839 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46349.log 2023-10-30T14:53:16.840 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58767.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58767.log.gz 2023-10-30T14:53:16.840 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60161.log 2023-10-30T14:53:16.840 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46349.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46349.log.gz 2023-10-30T14:53:16.840 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62777.log 2023-10-30T14:53:16.841 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60161.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60161.log.gz 2023-10-30T14:53:16.841 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44300.log 2023-10-30T14:53:16.841 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62777.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62777.log.gz 2023-10-30T14:53:16.842 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58140.log 2023-10-30T14:53:16.842 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44300.log: 95.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-30T14:53:16.842 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50457.log 2023-10-30T14:53:16.842 INFO:teuthology.orchestra.run.smithi057.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44300.log.gz 2023-10-30T14:53:16.843 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58140.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58140.log.gz 2023-10-30T14:53:16.843 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52251.log 2023-10-30T14:53:16.843 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50457.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50457.log.gz 2023-10-30T14:53:16.844 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47549.log 2023-10-30T14:53:16.844 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52251.log: gzip -5 --verbose 88.5% -- -- replaced with /var/log/ceph/ceph-client.admin.52251.log.gz 2023-10-30T14:53:16.844 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.53372.log 2023-10-30T14:53:16.844 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.47549.log: gzip -5 --verbose 88.2% -- -- replaced with /var/log/ceph/ceph-client.admin.47549.log.gz 2023-10-30T14:53:16.845 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.54012.log 2023-10-30T14:53:16.847 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38723.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38723.log.gz 2023-10-30T14:53:16.847 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38627.log 2023-10-30T14:53:16.847 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38771.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38771.log.gz 2023-10-30T14:53:16.851 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56752.log 2023-10-30T14:53:16.851 INFO:teuthology.orchestra.run.smithi057.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53372.log.gz 2023-10-30T14:53:16.851 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54012.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54012.log.gz 2023-10-30T14:53:16.851 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55711.log 2023-10-30T14:53:16.852 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56752.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56752.log.gz 2023-10-30T14:53:16.852 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48438.log 2023-10-30T14:53:16.852 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55711.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55711.log.gz 2023-10-30T14:53:16.853 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59687.log 2023-10-30T14:53:16.853 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48177.log 2023-10-30T14:53:16.853 INFO:teuthology.orchestra.run.smithi057.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48438.log.gz 2023-10-30T14:53:16.853 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59687.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59687.log.gz 2023-10-30T14:53:16.854 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-30T14:53:16.854 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48177.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48177.log.gz 2023-10-30T14:53:16.854 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60343.log 2023-10-30T14:53:16.854 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50353.log 2023-10-30T14:53:16.855 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56152.log 2023-10-30T14:53:16.855 INFO:teuthology.orchestra.run.smithi057.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60343.log.gz 2023-10-30T14:53:16.855 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52890.log 2023-10-30T14:53:16.856 INFO:teuthology.orchestra.run.smithi057.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50353.log.gz 2023-10-30T14:53:16.856 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50562.log 2023-10-30T14:53:16.856 INFO:teuthology.orchestra.run.smithi057.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.56152.log.gz 2023-10-30T14:53:16.856 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52890.log: gzip -5 88.4% --verbose -- replaced with /var/log/ceph/ceph-client.admin.52890.log.gz -- 2023-10-30T14:53:16.857 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.53128.log 2023-10-30T14:53:16.857 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46236.log 2023-10-30T14:53:16.857 INFO:teuthology.orchestra.run.smithi057.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50562.log.gz 2023-10-30T14:53:16.860 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53128.log: gzip -5 --verbose 88.7% -- -- replaced with /var/log/ceph/ceph-client.admin.53128.log.gz 2023-10-30T14:53:16.861 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.43956.log 2023-10-30T14:53:16.861 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46236.log: gzip -5 88.8% --verbose -- replaced with /var/log/ceph/ceph-client.admin.46236.log.gz 2023-10-30T14:53:16.861 INFO:teuthology.orchestra.run.smithi057.stderr: -- /var/log/ceph/ceph-client.admin.55737.log 2023-10-30T14:53:16.862 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.43956.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43956.log.gz 2023-10-30T14:53:16.862 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61708.log 2023-10-30T14:53:16.862 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61593.log 2023-10-30T14:53:16.862 INFO:teuthology.orchestra.run.smithi057.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55737.log.gz 2023-10-30T14:53:16.863 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61708.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61708.log.gz 2023-10-30T14:53:16.863 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53208.log 2023-10-30T14:53:16.863 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61593.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61593.log.gz 2023-10-30T14:53:16.864 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44197.log 2023-10-30T14:53:16.864 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53208.log: gzip 88.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.53208.log.gz 2023-10-30T14:53:16.864 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47487.log 2023-10-30T14:53:16.864 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44197.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44197.log.gz 2023-10-30T14:53:16.865 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52864.log 2023-10-30T14:53:16.865 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.47487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58873.log 2023-10-30T14:53:16.865 INFO:teuthology.orchestra.run.smithi057.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.47487.log.gz 2023-10-30T14:53:16.865 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52864.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52864.log.gz 2023-10-30T14:53:16.866 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58244.log 2023-10-30T14:53:16.867 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38627.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38627.log.gz 2023-10-30T14:53:16.870 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52510.log 2023-10-30T14:53:16.870 INFO:teuthology.orchestra.run.smithi057.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58873.log.gz 2023-10-30T14:53:16.870 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55198.log 2023-10-30T14:53:16.871 INFO:teuthology.orchestra.run.smithi057.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58244.log.gz 2023-10-30T14:53:16.871 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61435.log 2023-10-30T14:53:16.871 INFO:teuthology.orchestra.run.smithi057.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52510.log.gz 2023-10-30T14:53:16.871 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50986.log 2023-10-30T14:53:16.872 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55198.log.gz 2023-10-30T14:53:16.872 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61435.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61435.log.gz 2023-10-30T14:53:16.872 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56700.log 2023-10-30T14:53:16.873 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48289.log 2023-10-30T14:53:16.873 INFO:teuthology.orchestra.run.smithi057.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50986.log.gz 2023-10-30T14:53:16.873 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52562.log 2023-10-30T14:53:16.873 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56700.log.gz 2023-10-30T14:53:16.874 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48237.log 2023-10-30T14:53:16.874 INFO:teuthology.orchestra.run.smithi057.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48289.log.gz 2023-10-30T14:53:16.874 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52562.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52562.log.gz 2023-10-30T14:53:16.877 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57341.log 2023-10-30T14:53:16.877 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45652.log 2023-10-30T14:53:16.877 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48237.log.gz 2023-10-30T14:53:16.878 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50932.log 2023-10-30T14:53:16.878 INFO:teuthology.orchestra.run.smithi057.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57341.log.gz 2023-10-30T14:53:16.878 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46016.log 2023-10-30T14:53:16.878 INFO:teuthology.orchestra.run.smithi057.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45652.log.gz 2023-10-30T14:53:16.879 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50932.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50932.log.gz 2023-10-30T14:53:16.879 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60291.log 2023-10-30T14:53:16.879 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51064.log 2023-10-30T14:53:16.879 INFO:teuthology.orchestra.run.smithi057.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46016.log.gz 2023-10-30T14:53:16.880 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57313.log 2023-10-30T14:53:16.880 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60291.log.gz 2023-10-30T14:53:16.882 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.51064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46068.log 2023-10-30T14:53:16.883 INFO:teuthology.orchestra.run.smithi057.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51064.log.gz 2023-10-30T14:53:16.883 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57313.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57313.log.gz 2023-10-30T14:53:16.883 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51272.log 2023-10-30T14:53:16.883 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46068.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46068.log.gz 2023-10-30T14:53:16.884 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52364.log 2023-10-30T14:53:16.884 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.51272.log: gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.51272.log.gz 2023-10-30T14:53:16.884 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.56475.log 2023-10-30T14:53:16.884 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52364.log: gzip 87.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.52364.log.gz --verbose 2023-10-30T14:53:16.885 INFO:teuthology.orchestra.run.smithi057.stderr: -- /var/log/ceph/ceph-client.admin.60135.log 2023-10-30T14:53:16.886 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56475.log: gzip 88.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.56475.log.gz 2023-10-30T14:53:16.887 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51246.log 2023-10-30T14:53:16.887 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60135.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.60135.log.gz 2023-10-30T14:53:16.887 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48572.log 2023-10-30T14:53:16.887 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.51246.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51246.log.gz 2023-10-30T14:53:16.888 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58977.log 2023-10-30T14:53:16.888 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48572.log: gzip -5 87.9% --verbose -- replaced with /var/log/ceph/ceph-client.admin.48572.log.gz -- 2023-10-30T14:53:16.888 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.58192.log 2023-10-30T14:53:16.890 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55553.log 2023-10-30T14:53:16.890 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58977.log.gz 2023-10-30T14:53:16.891 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58192.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58192.log.gz 2023-10-30T14:53:16.891 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52732.log 2023-10-30T14:53:16.891 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55553.log: gzip -5 --verbose 88.2% -- -- replaced with /var/log/ceph/ceph-client.admin.55553.log.gz 2023-10-30T14:53:16.891 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.52916.log 2023-10-30T14:53:16.892 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46183.log 2023-10-30T14:53:16.892 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52732.log.gz 2023-10-30T14:53:16.894 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62132.log 2023-10-30T14:53:16.894 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52916.log.gz 2023-10-30T14:53:16.894 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60496.log 2023-10-30T14:53:16.894 INFO:teuthology.orchestra.run.smithi057.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46183.log.gz 2023-10-30T14:53:16.895 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62132.log: gzip -5 87.5% --verbose -- replaced with /var/log/ceph/ceph-client.admin.62132.log.gz -- 2023-10-30T14:53:16.895 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.62410.log 2023-10-30T14:53:16.895 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54559.log 2023-10-30T14:53:16.896 INFO:teuthology.orchestra.run.smithi057.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60496.log.gz 2023-10-30T14:53:16.897 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45141.log 2023-10-30T14:53:16.897 INFO:teuthology.orchestra.run.smithi057.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62410.log.gz 2023-10-30T14:53:16.897 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58530.log 2023-10-30T14:53:16.898 INFO:teuthology.orchestra.run.smithi057.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54559.log.gz 2023-10-30T14:53:16.898 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45141.log: gzip 88.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.45141.log.gz 2023-10-30T14:53:16.898 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47828.log 2023-10-30T14:53:16.900 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58530.log: gzip -5 --verbose -- 89.4% /var/log/ceph/ceph-client.admin.61245.log 2023-10-30T14:53:16.900 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58530.log.gz 2023-10-30T14:53:16.900 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.47828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52995.log 2023-10-30T14:53:16.901 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47828.log.gz 2023-10-30T14:53:16.901 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61245.log: gzip 88.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.61245.log.gz 2023-10-30T14:53:16.901 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48897.log 2023-10-30T14:53:16.902 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52995.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52995.log.gz 2023-10-30T14:53:16.903 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.56986.log 2023-10-30T14:53:16.903 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48897.log: gzip -5 --verbose -- 88.4% /var/log/ceph/ceph-client.admin.53505.log -- replaced with /var/log/ceph/ceph-client.admin.48897.log.gz 2023-10-30T14:53:16.903 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-30T14:53:16.904 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51918.log 2023-10-30T14:53:16.904 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56986.log.gz 2023-10-30T14:53:16.905 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53505.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53505.log.gz 2023-10-30T14:53:16.905 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61812.log 2023-10-30T14:53:16.906 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.51918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62857.log 2023-10-30T14:53:16.906 INFO:teuthology.orchestra.run.smithi057.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51918.log.gz 2023-10-30T14:53:16.906 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61812.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.61812.log.gz 2023-10-30T14:53:16.907 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44171.log 2023-10-30T14:53:16.908 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62857.log: gzip -5 --verbose -- 89.3% /var/log/ceph/ceph-client.admin.48042.log 2023-10-30T14:53:16.908 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62857.log.gz 2023-10-30T14:53:16.908 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44171.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44171.log.gz 2023-10-30T14:53:16.909 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58086.log 2023-10-30T14:53:16.910 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48042.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48042.log.gzgzip 2023-10-30T14:53:16.910 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.56646.log 2023-10-30T14:53:16.910 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46375.log 2023-10-30T14:53:16.910 INFO:teuthology.orchestra.run.smithi057.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58086.log.gz 2023-10-30T14:53:16.911 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56646.log: gzip -5 --verbose 88.7% -- -- replaced with /var/log/ceph/ceph-client.admin.56646.log.gz /var/log/ceph/ceph-client.admin.55423.log 2023-10-30T14:53:16.912 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-30T14:53:16.912 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46375.log: gzip -5 --verbose 88.1% -- -- replaced with /var/log/ceph/ceph-client.admin.46375.log.gz /var/log/ceph/ceph-client.admin.53533.log 2023-10-30T14:53:16.912 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-30T14:53:16.913 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55423.log: gzip -5 88.7% --verbose -- replaced with /var/log/ceph/ceph-client.admin.55423.log.gz -- 2023-10-30T14:53:16.914 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.53401.log 2023-10-30T14:53:16.914 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53533.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53533.log.gz 2023-10-30T14:53:16.914 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60414.log 2023-10-30T14:53:16.915 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53401.log: gzip -5 --verbose -- 87.5% /var/log/ceph/ceph-osd.2.log 2023-10-30T14:53:16.916 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53401.log.gz 2023-10-30T14:53:16.916 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49331.log 2023-10-30T14:53:16.916 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60414.log.gz 2023-10-30T14:53:16.917 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55685.log 2023-10-30T14:53:16.917 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.49331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61619.log 2023-10-30T14:53:16.918 INFO:teuthology.orchestra.run.smithi057.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49331.log.gz 2023-10-30T14:53:16.919 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55685.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55685.log.gz 2023-10-30T14:53:16.919 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49223.log 2023-10-30T14:53:16.920 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61619.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61619.log.gz 2023-10-30T14:53:16.920 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44496.log 2023-10-30T14:53:16.921 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.49223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56672.log 2023-10-30T14:53:16.921 INFO:teuthology.orchestra.run.smithi057.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49223.log.gz 2023-10-30T14:53:16.922 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51220.log 2023-10-30T14:53:16.922 INFO:teuthology.orchestra.run.smithi057.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44496.log.gz 2023-10-30T14:53:16.923 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54664.log 2023-10-30T14:53:16.924 INFO:teuthology.orchestra.run.smithi057.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56672.log.gz 2023-10-30T14:53:16.924 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.51220.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51220.log.gz 2023-10-30T14:53:16.925 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60317.log 2023-10-30T14:53:16.926 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61276.log 2023-10-30T14:53:16.926 INFO:teuthology.orchestra.run.smithi057.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54664.log.gz 2023-10-30T14:53:16.927 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60919.log 2023-10-30T14:53:16.927 INFO:teuthology.orchestra.run.smithi057.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60317.log.gz 2023-10-30T14:53:16.928 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61276.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61276.log.gz 2023-10-30T14:53:16.928 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44709.log 2023-10-30T14:53:16.929 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58112.log 2023-10-30T14:53:16.929 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60919.log.gz 2023-10-30T14:53:16.930 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44709.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44709.log.gz 2023-10-30T14:53:16.930 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44491.log 2023-10-30T14:53:16.931 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58112.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58112.log.gz 2023-10-30T14:53:16.932 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49171.log 2023-10-30T14:53:16.932 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55040.log 2023-10-30T14:53:16.933 INFO:teuthology.orchestra.run.smithi057.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44491.log.gz 2023-10-30T14:53:16.934 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.49171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48315.log 2023-10-30T14:53:16.934 INFO:teuthology.orchestra.run.smithi057.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49171.log.gz 2023-10-30T14:53:16.935 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55040.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55040.log.gz 2023-10-30T14:53:16.935 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57259.log 2023-10-30T14:53:16.936 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45834.log 2023-10-30T14:53:16.936 INFO:teuthology.orchestra.run.smithi057.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48315.log.gz 2023-10-30T14:53:16.937 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57259.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57259.log.gz 2023-10-30T14:53:16.937 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50194.log 2023-10-30T14:53:16.938 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48652.log 2023-10-30T14:53:16.939 INFO:teuthology.orchestra.run.smithi057.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45834.log.gz 2023-10-30T14:53:16.940 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51970.log 2023-10-30T14:53:16.940 INFO:teuthology.orchestra.run.smithi057.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50194.log.gz 2023-10-30T14:53:16.941 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62280.log 2023-10-30T14:53:16.941 INFO:teuthology.orchestra.run.smithi057.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48652.log.gz 2023-10-30T14:53:16.942 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.51970.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51970.log.gz 2023-10-30T14:53:16.942 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53234.log 2023-10-30T14:53:16.943 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62184.log 2023-10-30T14:53:16.944 INFO:teuthology.orchestra.run.smithi057.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62280.log.gz 2023-10-30T14:53:16.944 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53234.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53234.log.gz 2023-10-30T14:53:16.945 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57483.log 2023-10-30T14:53:16.946 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58715.log 2023-10-30T14:53:16.946 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62184.log.gz 2023-10-30T14:53:16.947 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57483.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57483.log.gz 2023-10-30T14:53:16.947 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48949.log 2023-10-30T14:53:16.948 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56209.log 2023-10-30T14:53:16.948 INFO:teuthology.orchestra.run.smithi057.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58715.log.gz 2023-10-30T14:53:16.949 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48949.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48949.log.gz 2023-10-30T14:53:16.950 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62961.log 2023-10-30T14:53:16.950 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46594.log 2023-10-30T14:53:16.951 INFO:teuthology.orchestra.run.smithi057.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.56209.log.gz 2023-10-30T14:53:16.952 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62961.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62961.log.gz 2023-10-30T14:53:16.952 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-30T14:53:16.953 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59740.log 2023-10-30T14:53:16.953 INFO:teuthology.orchestra.run.smithi057.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46594.log.gz 2023-10-30T14:53:16.954 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44710.log 2023-10-30T14:53:16.955 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47653.log 2023-10-30T14:53:16.955 INFO:teuthology.orchestra.run.smithi057.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59740.log.gz 2023-10-30T14:53:16.956 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57930.log 2023-10-30T14:53:16.957 INFO:teuthology.orchestra.run.smithi057.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44710.log.gz 2023-10-30T14:53:16.957 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.47653.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47653.log.gz 2023-10-30T14:53:16.958 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50723.log 2023-10-30T14:53:16.958 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49140.log 2023-10-30T14:53:16.959 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57930.log.gz 2023-10-30T14:53:16.960 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50723.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50723.log.gz 2023-10-30T14:53:16.960 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50749.log 2023-10-30T14:53:16.961 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.49140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46542.log 2023-10-30T14:53:16.962 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49140.log.gz 2023-10-30T14:53:16.962 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50749.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50749.log.gz 2023-10-30T14:53:16.963 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48546.log 2023-10-30T14:53:16.963 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50854.log 2023-10-30T14:53:16.964 INFO:teuthology.orchestra.run.smithi057.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46542.log.gz 2023-10-30T14:53:16.965 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48546.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48546.log.gz 2023-10-30T14:53:16.965 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57452.log 2023-10-30T14:53:16.966 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47800.log 2023-10-30T14:53:16.966 INFO:teuthology.orchestra.run.smithi057.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50854.log.gz 2023-10-30T14:53:16.967 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57452.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57452.log.gz 2023-10-30T14:53:16.967 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58636.log 2023-10-30T14:53:16.968 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.47800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50483.log 2023-10-30T14:53:16.969 INFO:teuthology.orchestra.run.smithi057.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47800.log.gz 2023-10-30T14:53:16.970 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58636.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58636.log.gz 2023-10-30T14:53:16.970 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46209.log 2023-10-30T14:53:16.971 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54092.log 2023-10-30T14:53:16.971 INFO:teuthology.orchestra.run.smithi057.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50483.log.gz 2023-10-30T14:53:16.972 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46209.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46209.log.gz 2023-10-30T14:53:16.972 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46486.log 2023-10-30T14:53:16.973 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59555.log 2023-10-30T14:53:16.974 INFO:teuthology.orchestra.run.smithi057.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54092.log.gz 2023-10-30T14:53:16.974 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46486.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46486.log.gz 2023-10-30T14:53:16.975 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54803.log 2023-10-30T14:53:16.976 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59451.log 2023-10-30T14:53:16.976 INFO:teuthology.orchestra.run.smithi057.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59555.log.gz 2023-10-30T14:53:16.977 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54803.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54803.log.gz 2023-10-30T14:53:16.977 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61946.log 2023-10-30T14:53:16.978 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54038.log 2023-10-30T14:53:16.979 INFO:teuthology.orchestra.run.smithi057.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59451.log.gz 2023-10-30T14:53:16.979 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61946.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61946.log.gz 2023-10-30T14:53:16.980 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62592.log 2023-10-30T14:53:16.980 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45219.log 2023-10-30T14:53:16.981 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54038.log.gz 2023-10-30T14:53:16.982 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62592.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62592.log.gz 2023-10-30T14:53:16.982 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47883.log 2023-10-30T14:53:16.983 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61166.log 2023-10-30T14:53:16.984 INFO:teuthology.orchestra.run.smithi057.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45219.log.gz 2023-10-30T14:53:16.984 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.47883.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47883.log.gz 2023-10-30T14:53:16.985 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59999.log 2023-10-30T14:53:16.986 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59101.log 2023-10-30T14:53:16.986 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61166.log.gz 2023-10-30T14:53:16.987 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59999.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59999.log.gz 2023-10-30T14:53:16.987 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45678.log 2023-10-30T14:53:16.988 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60239.log 2023-10-30T14:53:16.989 INFO:teuthology.orchestra.run.smithi057.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.59101.log.gz 2023-10-30T14:53:16.989 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45678.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45678.log.gz 2023-10-30T14:53:16.990 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55607.log 2023-10-30T14:53:16.990 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55092.log 2023-10-30T14:53:16.991 INFO:teuthology.orchestra.run.smithi057.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60239.log.gz 2023-10-30T14:53:16.992 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55607.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55607.log.gz 2023-10-30T14:53:16.992 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58297.log 2023-10-30T14:53:16.993 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47987.log 2023-10-30T14:53:16.994 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55092.log.gz 2023-10-30T14:53:16.994 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58297.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58297.log.gz 2023-10-30T14:53:16.995 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61920.log 2023-10-30T14:53:16.996 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.47987.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47987.log.gz 2023-10-30T14:53:16.996 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62566.log 2023-10-30T14:53:16.997 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53072.log 2023-10-30T14:53:16.998 INFO:teuthology.orchestra.run.smithi057.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61920.log.gz 2023-10-30T14:53:16.999 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62566.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62566.log.gz 2023-10-30T14:53:16.999 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62436.log 2023-10-30T14:53:17.000 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53072.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53072.log.gz 2023-10-30T14:53:17.001 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60109.log 2023-10-30T14:53:17.002 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62436.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62436.log.gz 2023-10-30T14:53:17.002 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62935.log 2023-10-30T14:53:17.004 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60109.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60109.log.gz 2023-10-30T14:53:17.004 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59797.log 2023-10-30T14:53:17.005 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62935.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62935.log.gz 2023-10-30T14:53:17.005 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50379.log 2023-10-30T14:53:17.007 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61513.log 2023-10-30T14:53:17.007 INFO:teuthology.orchestra.run.smithi057.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59797.log.gz 2023-10-30T14:53:17.008 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50379.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50379.log.gz 2023-10-30T14:53:17.008 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60387.log 2023-10-30T14:53:17.009 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61513.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61513.log.gz 2023-10-30T14:53:17.010 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52223.log 2023-10-30T14:53:17.011 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46512.log 2023-10-30T14:53:17.012 INFO:teuthology.orchestra.run.smithi057.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60387.log.gz 2023-10-30T14:53:17.013 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52223.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52223.log.gz 2023-10-30T14:53:17.013 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44926.log 2023-10-30T14:53:17.014 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46512.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46512.log.gz 2023-10-30T14:53:17.014 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55370.log 2023-10-30T14:53:17.016 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44926.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44926.log.gz 2023-10-30T14:53:17.016 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56960.log 2023-10-30T14:53:17.017 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55370.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55370.log.gz 2023-10-30T14:53:17.018 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59056.log 2023-10-30T14:53:17.019 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60025.log 89.1% 2023-10-30T14:53:17.019 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56960.log.gz 2023-10-30T14:53:17.021 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60468.log 2023-10-30T14:53:17.021 INFO:teuthology.orchestra.run.smithi057.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59056.log.gz 2023-10-30T14:53:17.022 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61786.log 2023-10-30T14:53:17.022 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60025.log.gz 2023-10-30T14:53:17.024 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60468.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60468.log.gz 2023-10-30T14:53:17.024 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56181.log 2023-10-30T14:53:17.025 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61786.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61786.log.gzgzip 2023-10-30T14:53:17.025 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53879.log 2023-10-30T14:53:17.027 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56181.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56181.log.gz 2023-10-30T14:53:17.027 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51116.log 2023-10-30T14:53:17.028 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44299.log 2023-10-30T14:53:17.029 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53879.log.gz 2023-10-30T14:53:17.029 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.51116.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51116.log.gz 2023-10-30T14:53:17.030 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43770.log 2023-10-30T14:53:17.031 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38066.log 2023-10-30T14:53:17.031 INFO:teuthology.orchestra.run.smithi057.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44299.log.gz 2023-10-30T14:53:17.032 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.43770.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43770.log.gz 2023-10-30T14:53:17.033 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43930.log 2023-10-30T14:53:17.034 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.38066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53663.log 2023-10-30T14:53:17.034 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38066.log.gz 2023-10-30T14:53:17.035 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.43930.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43930.log.gz 2023-10-30T14:53:17.035 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53559.log 2023-10-30T14:53:17.037 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53663.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53663.log.gz 2023-10-30T14:53:17.037 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56908.log 2023-10-30T14:53:17.038 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53559.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53559.log.gz 2023-10-30T14:53:17.039 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59030.log 2023-10-30T14:53:17.040 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56908.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56908.log.gz 2023-10-30T14:53:17.040 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62909.log 2023-10-30T14:53:17.041 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59030.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59030.log.gz 2023-10-30T14:53:17.042 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60551.log 2023-10-30T14:53:17.043 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62909.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62909.log.gz 2023-10-30T14:53:17.044 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50697.log 2023-10-30T14:53:17.045 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60551.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.60551.log.gz 2023-10-30T14:53:17.045 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57695.log 2023-10-30T14:53:17.046 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50697.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50697.log.gz 2023-10-30T14:53:17.047 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54222.log 2023-10-30T14:53:17.048 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57695.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57695.log.gz 2023-10-30T14:53:17.048 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55014.log 2023-10-30T14:53:17.050 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54222.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54222.log.gz 2023-10-30T14:53:17.050 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45037.log 2023-10-30T14:53:17.051 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55014.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55014.log.gz 2023-10-30T14:53:17.052 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54432.log 2023-10-30T14:53:17.052 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53907.log 2023-10-30T14:53:17.053 INFO:teuthology.orchestra.run.smithi057.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45037.log.gz 2023-10-30T14:53:17.054 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54432.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54432.log.gz 2023-10-30T14:53:17.054 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57591.log 2023-10-30T14:53:17.055 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55885.log 2023-10-30T14:53:17.056 INFO:teuthology.orchestra.run.smithi057.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53907.log.gz 2023-10-30T14:53:17.056 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57591.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57591.log.gz 2023-10-30T14:53:17.057 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52618.log 2023-10-30T14:53:17.058 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58582.log 2023-10-30T14:53:17.058 INFO:teuthology.orchestra.run.smithi057.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55885.log.gz 2023-10-30T14:53:17.059 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52618.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52618.log.gz 2023-10-30T14:53:17.059 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61409.log 2023-10-30T14:53:17.060 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48408.log 2023-10-30T14:53:17.061 INFO:teuthology.orchestra.run.smithi057.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58582.log.gz 2023-10-30T14:53:17.061 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61409.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61409.log.gz 2023-10-30T14:53:17.062 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52786.log 2023-10-30T14:53:17.062 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53102.log 2023-10-30T14:53:17.063 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48408.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48408.log.gz 2023-10-30T14:53:17.064 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52786.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52786.log.gz 2023-10-30T14:53:17.064 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55066.log 2023-10-30T14:53:17.065 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.53102.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.59895.log 2023-10-30T14:53:17.066 INFO:teuthology.orchestra.run.smithi057.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53102.log.gz 2023-10-30T14:53:17.066 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55066.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55066.log.gz 2023-10-30T14:53:17.067 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44711.log 2023-10-30T14:53:17.068 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49448.log 2023-10-30T14:53:17.068 INFO:teuthology.orchestra.run.smithi057.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59895.log.gz 2023-10-30T14:53:17.069 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44711.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44711.log.gz 2023-10-30T14:53:17.069 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46314.log 2023-10-30T14:53:17.070 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.49448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59259.log 2023-10-30T14:53:17.071 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49448.log.gz 2023-10-30T14:53:17.071 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46314.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46314.log.gz 2023-10-30T14:53:17.072 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45325.log 2023-10-30T14:53:17.072 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58008.log 2023-10-30T14:53:17.073 INFO:teuthology.orchestra.run.smithi057.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59259.log.gz 2023-10-30T14:53:17.074 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45089.log 2023-10-30T14:53:17.074 INFO:teuthology.orchestra.run.smithi057.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.45325.log.gz 2023-10-30T14:53:17.075 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56313.log 2023-10-30T14:53:17.075 INFO:teuthology.orchestra.run.smithi057.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58008.log.gz 2023-10-30T14:53:17.076 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45089.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45089.log.gz 2023-10-30T14:53:17.077 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45756.log 2023-10-30T14:53:17.077 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50669.log 2023-10-30T14:53:17.078 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56313.log.gz 2023-10-30T14:53:17.079 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45756.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45756.log.gz 2023-10-30T14:53:17.079 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43568.log 2023-10-30T14:53:17.080 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45626.log 2023-10-30T14:53:17.081 INFO:teuthology.orchestra.run.smithi057.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50669.log.gz 2023-10-30T14:53:17.081 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.43568.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43568.log.gz 2023-10-30T14:53:17.082 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54988.log 2023-10-30T14:53:17.083 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44719.log 2023-10-30T14:53:17.083 INFO:teuthology.orchestra.run.smithi057.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45626.log.gz 2023-10-30T14:53:17.084 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54988.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54988.log.gz 2023-10-30T14:53:17.085 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52022.log 2023-10-30T14:53:17.085 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45964.log 2023-10-30T14:53:17.086 INFO:teuthology.orchestra.run.smithi057.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44719.log.gz 2023-10-30T14:53:17.087 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52022.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52022.log.gz 2023-10-30T14:53:17.087 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48209.log 2023-10-30T14:53:17.088 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59179.log 2023-10-30T14:53:17.089 INFO:teuthology.orchestra.run.smithi057.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45964.log.gz 2023-10-30T14:53:17.089 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48209.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48209.log.gz 2023-10-30T14:53:17.090 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59609.log 2023-10-30T14:53:17.091 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57393.log 2023-10-30T14:53:17.091 INFO:teuthology.orchestra.run.smithi057.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59179.log.gz 2023-10-30T14:53:17.092 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59609.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59609.log.gz 2023-10-30T14:53:17.092 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53427.log 2023-10-30T14:53:17.093 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57393.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-30T14:53:17.094 INFO:teuthology.orchestra.run.smithi057.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57393.log.gz 2023-10-30T14:53:17.094 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53427.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53427.log.gz 2023-10-30T14:53:17.095 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44034.log 2023-10-30T14:53:17.095 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43713.log 2023-10-30T14:53:17.096 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44034.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44034.log.gz 2023-10-30T14:53:17.097 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46105.log 2023-10-30T14:53:17.098 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.43713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48089.log 2023-10-30T14:53:17.098 INFO:teuthology.orchestra.run.smithi057.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43713.log.gz 2023-10-30T14:53:17.099 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46105.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46105.log.gz 2023-10-30T14:53:17.100 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50325.log 2023-10-30T14:53:17.100 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48089.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48089.log.gz 2023-10-30T14:53:17.101 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45860.log 2023-10-30T14:53:17.102 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44985.log 2023-10-30T14:53:17.103 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50325.log.gz 2023-10-30T14:53:17.103 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45860.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45860.log.gz 2023-10-30T14:53:17.104 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58349.log 2023-10-30T14:53:17.105 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44093.log 2023-10-30T14:53:17.105 INFO:teuthology.orchestra.run.smithi057.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44985.log.gz 2023-10-30T14:53:17.106 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58349.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58349.log.gz 2023-10-30T14:53:17.106 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63032.log 2023-10-30T14:53:17.107 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62670.log 2023-10-30T14:53:17.108 INFO:teuthology.orchestra.run.smithi057.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44093.log.gz 2023-10-30T14:53:17.108 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.63032.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63032.log.gz 2023-10-30T14:53:17.109 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56934.log 2023-10-30T14:53:17.109 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57181.log 2023-10-30T14:53:17.110 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56934.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62670.log.gz 2023-10-30T14:53:17.111 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56934.log.gz 2023-10-30T14:53:17.111 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46624.log 2023-10-30T14:53:17.112 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59635.log 2023-10-30T14:53:17.113 INFO:teuthology.orchestra.run.smithi057.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57181.log.gz 2023-10-30T14:53:17.114 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46624.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46624.log.gz 2023-10-30T14:53:17.114 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50824.log 2023-10-30T14:53:17.115 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54118.log 2023-10-30T14:53:17.115 INFO:teuthology.orchestra.run.smithi057.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59635.log.gz 2023-10-30T14:53:17.116 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50824.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50824.log.gz 2023-10-30T14:53:17.116 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54170.log 2023-10-30T14:53:17.117 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49001.log 2023-10-30T14:53:17.118 INFO:teuthology.orchestra.run.smithi057.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54118.log.gz 2023-10-30T14:53:17.119 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54170.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54170.log.gz 2023-10-30T14:53:17.119 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56444.log 2023-10-30T14:53:17.120 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.49001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55633.log 2023-10-30T14:53:17.120 INFO:teuthology.orchestra.run.smithi057.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49001.log.gz 2023-10-30T14:53:17.121 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56444.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56444.log.gz 2023-10-30T14:53:17.121 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45193.log 2023-10-30T14:53:17.122 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60633.log 2023-10-30T14:53:17.123 INFO:teuthology.orchestra.run.smithi057.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55633.log.gz 2023-10-30T14:53:17.123 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45193.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45193.log.gz 2023-10-30T14:53:17.124 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58663.log 2023-10-30T14:53:17.125 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56804.log 2023-10-30T14:53:17.126 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60633.log.gz 2023-10-30T14:53:17.127 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58663.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58663.log.gz 2023-10-30T14:53:17.127 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54716.log 2023-10-30T14:53:17.128 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59422.log 2023-10-30T14:53:17.128 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56804.log.gz 2023-10-30T14:53:17.129 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54716.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54716.log.gz 2023-10-30T14:53:17.129 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47692.log 2023-10-30T14:53:17.130 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59422.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59422.log.gz 2023-10-30T14:53:17.131 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54770.log 2023-10-30T14:53:17.132 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.47692.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47692.log.gz 2023-10-30T14:53:17.133 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48464.log 2023-10-30T14:53:17.134 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54770.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54770.log.gz 2023-10-30T14:53:17.134 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55146.log 2023-10-30T14:53:17.136 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48464.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48464.log.gz 2023-10-30T14:53:17.136 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57066.log 2023-10-30T14:53:17.137 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55146.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55146.log.gz 2023-10-30T14:53:17.138 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56339.log 2023-10-30T14:53:17.138 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57066.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57066.log.gz 2023-10-30T14:53:17.139 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53024.log 2023-10-30T14:53:17.140 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56339.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56339.log.gz 2023-10-30T14:53:17.141 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60442.log 2023-10-30T14:53:17.142 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53024.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53024.log.gz 2023-10-30T14:53:17.142 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58951.log 2023-10-30T14:53:17.143 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60442.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60442.log.gz 2023-10-30T14:53:17.144 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45604.log 2023-10-30T14:53:17.145 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58951.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58951.log.gz 2023-10-30T14:53:17.145 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59713.log 2023-10-30T14:53:17.146 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.0.45604.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.45604.log.gz 2023-10-30T14:53:17.147 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58689.log 2023-10-30T14:53:17.148 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59713.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59713.log.gz 2023-10-30T14:53:17.149 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-30T14:53:17.150 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58689.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58689.log.gz 2023-10-30T14:53:17.150 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58375.log 2023-10-30T14:53:17.151 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55449.log 2023-10-30T14:53:17.153 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56778.log 2023-10-30T14:53:17.154 INFO:teuthology.orchestra.run.smithi057.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58375.log.gz 2023-10-30T14:53:17.154 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55449.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55449.log.gz 2023-10-30T14:53:17.155 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44900.log 2023-10-30T14:53:17.156 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54196.log 2023-10-30T14:53:17.157 INFO:teuthology.orchestra.run.smithi057.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.56778.log.gz 2023-10-30T14:53:17.158 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44900.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44900.log.gz 2023-10-30T14:53:17.158 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.57143.log 2023-10-30T14:53:17.159 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54196.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54196.log.gz 2023-10-30T14:53:17.160 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52110.log 2023-10-30T14:53:17.161 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57143.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57143.log.gz 2023-10-30T14:53:17.161 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52338.log 2023-10-30T14:53:17.163 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62644.log 2023-10-30T14:53:17.163 INFO:teuthology.orchestra.run.smithi057.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52110.log.gz 2023-10-30T14:53:17.164 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52338.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52338.log.gz 2023-10-30T14:53:17.164 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57617.log 2023-10-30T14:53:17.166 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56041.log 2023-10-30T14:53:17.166 INFO:teuthology.orchestra.run.smithi057.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62644.log.gz 2023-10-30T14:53:17.167 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44060.log 2023-10-30T14:53:17.167 INFO:teuthology.orchestra.run.smithi057.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57617.log.gz 2023-10-30T14:53:17.169 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56041.log: gzip -5 --verbose -- 88.7% /var/log/ceph/ceph-client.admin.54406.log 2023-10-30T14:53:17.169 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56041.log.gz 2023-10-30T14:53:17.170 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53182.log 2023-10-30T14:53:17.170 INFO:teuthology.orchestra.run.smithi057.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44060.log.gz 2023-10-30T14:53:17.172 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59233.log 2023-10-30T14:53:17.172 INFO:teuthology.orchestra.run.smithi057.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54406.log.gz 2023-10-30T14:53:17.173 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55989.log 2023-10-30T14:53:17.173 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53182.log.gz 2023-10-30T14:53:17.175 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59285.log 2023-10-30T14:53:17.175 INFO:teuthology.orchestra.run.smithi057.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59233.log.gz 2023-10-30T14:53:17.176 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55989.log: gzip 89.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.55989.log.gz --verbose 2023-10-30T14:53:17.176 INFO:teuthology.orchestra.run.smithi057.stderr: -- /var/log/ceph/ceph-client.admin.54484.log 2023-10-30T14:53:17.178 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49277.log 2023-10-30T14:53:17.178 INFO:teuthology.orchestra.run.smithi057.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59285.log.gz 2023-10-30T14:53:17.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-10-30T14:53:17.179 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58430.log 2023-10-30T14:53:17.180 INFO:teuthology.orchestra.run.smithi057.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54484.log.gz 2023-10-30T14:53:17.181 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.49277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62306.log 2023-10-30T14:53:17.181 INFO:teuthology.orchestra.run.smithi057.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49277.log.gz 2023-10-30T14:53:17.182 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58430.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58430.log.gz -5 2023-10-30T14:53:17.182 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.57852.log 2023-10-30T14:53:17.184 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44223.log 2023-10-30T14:53:17.184 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62306.log.gz 2023-10-30T14:53:17.185 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57852.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57852.log.gz 2023-10-30T14:53:17.186 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56594.log 2023-10-30T14:53:17.186 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44223.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44223.log.gz 2023-10-30T14:53:17.187 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56882.log 2023-10-30T14:53:17.189 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51194.log 2023-10-30T14:53:17.189 INFO:teuthology.orchestra.run.smithi057.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56594.log.gz 2023-10-30T14:53:17.190 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56882.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56882.log.gz 2023-10-30T14:53:17.190 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45938.log 2023-10-30T14:53:17.192 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.51194.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51194.log.gz 2023-10-30T14:53:17.192 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62384.log 2023-10-30T14:53:17.194 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59825.log 2023-10-30T14:53:17.194 INFO:teuthology.orchestra.run.smithi057.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45938.log.gz 2023-10-30T14:53:17.195 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48013.log 2023-10-30T14:53:17.195 INFO:teuthology.orchestra.run.smithi057.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62384.log.gz 2023-10-30T14:53:17.197 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59825.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59825.log.gz 2023-10-30T14:53:17.197 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55963.log 2023-10-30T14:53:17.198 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55172.log 2023-10-30T14:53:17.198 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48013.log.gz 2023-10-30T14:53:17.200 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55963.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55963.log.gz 2023-10-30T14:53:17.201 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55172.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55172.log.gz 2023-10-30T14:53:17.202 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45912.log 2023-10-30T14:53:17.203 INFO:teuthology.orchestra.run.smithi088.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-30T14:53:17.204 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60815.log 2023-10-30T14:53:17.205 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45912.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45912.log.gz -5 2023-10-30T14:53:17.205 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.57669.log 2023-10-30T14:53:17.206 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45115.log 2023-10-30T14:53:17.206 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60815.log.gz 2023-10-30T14:53:17.208 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57669.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57669.log.gz 2023-10-30T14:53:17.209 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45115.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45115.log.gz 2023-10-30T14:53:17.210 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44145.log 2023-10-30T14:53:17.211 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51350.log 2023-10-30T14:53:17.213 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58401.log 2023-10-30T14:53:17.213 INFO:teuthology.orchestra.run.smithi057.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44145.log.gz 2023-10-30T14:53:17.215 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.51350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58925.log 2023-10-30T14:53:17.215 INFO:teuthology.orchestra.run.smithi057.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51350.log.gz 2023-10-30T14:53:17.216 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58401.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58401.log.gz 2023-10-30T14:53:17.216 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44249.log 2023-10-30T14:53:17.217 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61218.log 2023-10-30T14:53:17.218 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58925.log.gz 2023-10-30T14:53:17.219 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44249.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44249.log.gz 2023-10-30T14:53:17.220 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61487.log 2023-10-30T14:53:17.220 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61218.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61218.log.gz 2023-10-30T14:53:17.221 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61085.log 2023-10-30T14:53:17.223 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53286.log 2023-10-30T14:53:17.223 INFO:teuthology.orchestra.run.smithi057.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61487.log.gz 2023-10-30T14:53:17.224 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61085.log: gzip -5 87.5% --verbose -- replaced with /var/log/ceph/ceph-client.admin.61085.log.gz -- 2023-10-30T14:53:17.225 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.56287.log 2023-10-30T14:53:17.226 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47909.log 2023-10-30T14:53:17.226 INFO:teuthology.orchestra.run.smithi057.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53286.log.gz 2023-10-30T14:53:17.227 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56287.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56287.log.gz 2023-10-30T14:53:17.228 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54326.log 2023-10-30T14:53:17.228 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.47909.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47909.log.gz 2023-10-30T14:53:17.229 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48341.log 2023-10-30T14:53:17.231 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54326.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54326.log.gz 2023-10-30T14:53:17.231 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57426.log 2023-10-30T14:53:17.233 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48341.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48341.log.gzgzip 2023-10-30T14:53:17.233 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.56830.log 2023-10-30T14:53:17.234 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53933.log 2023-10-30T14:53:17.235 INFO:teuthology.orchestra.run.smithi057.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57426.log.gz 2023-10-30T14:53:17.236 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56830.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56830.log.gz 2023-10-30T14:53:17.236 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57512.log 2023-10-30T14:53:17.237 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53933.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53933.log.gz 2023-10-30T14:53:17.238 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52969.log 2023-10-30T14:53:17.239 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57512.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.57512.log.gz -5 2023-10-30T14:53:17.239 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.62805.log 2023-10-30T14:53:17.241 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58556.log 2023-10-30T14:53:17.241 INFO:teuthology.orchestra.run.smithi057.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52969.log.gz 2023-10-30T14:53:17.242 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62805.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62805.log.gz 2023-10-30T14:53:17.242 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58741.log 2023-10-30T14:53:17.244 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52760.log 2023-10-30T14:53:17.244 INFO:teuthology.orchestra.run.smithi057.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58556.log.gz 2023-10-30T14:53:17.245 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58741.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58741.log.gz 2023-10-30T14:53:17.246 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52760.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52760.log.gz 2023-10-30T14:53:17.247 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53479.log 2023-10-30T14:53:17.248 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62540.log 2023-10-30T14:53:17.250 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53345.log 2023-10-30T14:53:17.250 INFO:teuthology.orchestra.run.smithi057.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53479.log.gz 2023-10-30T14:53:17.251 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62618.log 2023-10-30T14:53:17.252 INFO:teuthology.orchestra.run.smithi057.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62540.log.gz 2023-10-30T14:53:17.253 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53345.log: gzip 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53345.log.gz 2023-10-30T14:53:17.253 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54638.log 2023-10-30T14:53:17.255 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62618.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62618.log.gz 2023-10-30T14:53:17.255 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45271.log 2023-10-30T14:53:17.256 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54638.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54638.log.gz 2023-10-30T14:53:17.256 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61030.log 2023-10-30T14:53:17.258 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44119.log 2023-10-30T14:53:17.258 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45271.log.gz 2023-10-30T14:53:17.259 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61030.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61030.log.gz 2023-10-30T14:53:17.259 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58504.log 2023-10-30T14:53:17.260 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44119.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44119.log.gz 2023-10-30T14:53:17.261 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48923.log 2023-10-30T14:53:17.263 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58504.log: gzip 89.5% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.58504.log.gz -- 2023-10-30T14:53:17.263 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.51996.log 2023-10-30T14:53:17.264 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48923.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48923.log.gz 2023-10-30T14:53:17.264 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53767.log 2023-10-30T14:53:17.266 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.51996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56261.log 2023-10-30T14:53:17.266 INFO:teuthology.orchestra.run.smithi057.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51996.log.gz 2023-10-30T14:53:17.267 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53767.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53767.log.gz 2023-10-30T14:53:17.269 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56261.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56261.log.gz 2023-10-30T14:53:17.269 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T14:53:17.271 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59583.log 2023-10-30T14:53:17.272 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43687.log 2023-10-30T14:53:17.274 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59583.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59583.log.gz 2023-10-30T14:53:17.275 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.43687.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43687.log.gz 2023-10-30T14:53:17.276 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48678.log 2023-10-30T14:53:17.277 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44296.log 2023-10-30T14:53:17.278 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48678.log: 87.6% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T14:53:17.279 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48678.log.gz 2023-10-30T14:53:17.280 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44296.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44296.log.gz 2023-10-30T14:53:17.281 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59127.log 2023-10-30T14:53:17.282 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62080.log 2023-10-30T14:53:17.283 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61760.log 2023-10-30T14:53:17.284 INFO:teuthology.orchestra.run.smithi057.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59127.log.gz 2023-10-30T14:53:17.285 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62080.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62080.log.gz 2023-10-30T14:53:17.285 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50299.log 2023-10-30T14:53:17.287 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56856.log 2023-10-30T14:53:17.287 INFO:teuthology.orchestra.run.smithi057.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61760.log.gz 2023-10-30T14:53:17.288 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50299.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50299.log.gz 2023-10-30T14:53:17.290 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56856.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56856.log.gz 2023-10-30T14:53:17.290 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56503.log 2023-10-30T14:53:17.292 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54690.log 2023-10-30T14:53:17.293 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55396.log 2023-10-30T14:53:17.294 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56503.log.gz 2023-10-30T14:53:17.295 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54690.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54690.log.gz 2023-10-30T14:53:17.296 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55396.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55396.log.gz 2023-10-30T14:53:17.297 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51324.log 2023-10-30T14:53:17.299 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49305.log 2023-10-30T14:53:17.300 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.51324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53260.log 2023-10-30T14:53:17.300 INFO:teuthology.orchestra.run.smithi057.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51324.log.gz 2023-10-30T14:53:17.301 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.49305.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49305.log.gz 2023-10-30T14:53:17.302 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52393.log 2023-10-30T14:53:17.303 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49197.log 2023-10-30T14:53:17.303 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53260.log.gz 2023-10-30T14:53:17.304 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52393.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52393.log.gz 2023-10-30T14:53:17.305 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57539.log 2023-10-30T14:53:17.306 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.49197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54248.log 2023-10-30T14:53:17.306 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49197.log.gz 2023-10-30T14:53:17.308 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57956.log 2023-10-30T14:53:17.308 INFO:teuthology.orchestra.run.smithi057.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57539.log.gz 2023-10-30T14:53:17.309 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44285.log 2023-10-30T14:53:17.309 INFO:teuthology.orchestra.run.smithi057.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54248.log.gz 2023-10-30T14:53:17.310 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57956.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57956.log.gz 2023-10-30T14:53:17.311 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58845.log 2023-10-30T14:53:17.312 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46405.log 2023-10-30T14:53:17.312 INFO:teuthology.orchestra.run.smithi057.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44285.log.gz 2023-10-30T14:53:17.314 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57723.log 2023-10-30T14:53:17.314 INFO:teuthology.orchestra.run.smithi057.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58845.log.gz 2023-10-30T14:53:17.315 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46405.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46405.log.gz 2023-10-30T14:53:17.315 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51168.log 2023-10-30T14:53:17.317 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57723.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57723.log.gz 2023-10-30T14:53:17.317 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61973.log 2023-10-30T14:53:17.318 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.51168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55527.log 2023-10-30T14:53:17.319 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51168.log.gz 2023-10-30T14:53:17.320 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61973.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61973.log.gz 2023-10-30T14:53:17.321 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55527.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55527.log.gz 2023-10-30T14:53:17.322 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54586.log 2023-10-30T14:53:17.323 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57285.log 2023-10-30T14:53:17.324 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54586.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54586.log.gz 2023-10-30T14:53:17.325 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53156.log 2023-10-30T14:53:17.326 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55911.log 2023-10-30T14:53:17.327 INFO:teuthology.orchestra.run.smithi057.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57285.log.gz 2023-10-30T14:53:17.328 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53156.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53156.log.gz 2023-10-30T14:53:17.329 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55911.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55911.log.gz 2023-10-30T14:53:17.330 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51012.log 2023-10-30T14:53:17.331 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60945.log 2023-10-30T14:53:17.333 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.51012.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51012.log.gz 2023-10-30T14:53:17.333 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60978.log 2023-10-30T14:53:17.334 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47857.log 2023-10-30T14:53:17.335 INFO:teuthology.orchestra.run.smithi057.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60945.log.gz 2023-10-30T14:53:17.336 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60978.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60978.log.gz 2023-10-30T14:53:17.337 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.47857.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47857.log.gz 2023-10-30T14:53:17.337 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57878.log 2023-10-30T14:53:17.339 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61567.log 2023-10-30T14:53:17.340 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57878.log: 88.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.57878.log.gz 2023-10-30T14:53:17.341 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54612.log 2023-10-30T14:53:17.342 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57038.log 2023-10-30T14:53:17.343 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61567.log.gz 2023-10-30T14:53:17.344 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44702.log 2023-10-30T14:53:17.344 INFO:teuthology.orchestra.run.smithi057.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54612.log.gz 2023-10-30T14:53:17.345 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57038.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.57038.log.gz 2023-10-30T14:53:17.345 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53453.log 2023-10-30T14:53:17.347 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53793.log 2023-10-30T14:53:17.347 INFO:teuthology.orchestra.run.smithi057.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44702.log.gz 2023-10-30T14:53:17.349 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53453.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53453.log.gz 2023-10-30T14:53:17.350 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53793.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53793.log.gz 2023-10-30T14:53:17.350 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57982.log 2023-10-30T14:53:17.352 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54378.log 2023-10-30T14:53:17.353 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57982.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.57982.log.gz 2023-10-30T14:53:17.354 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60083.log 2023-10-30T14:53:17.355 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49114.log 2023-10-30T14:53:17.355 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54378.log.gz 2023-10-30T14:53:17.356 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60083.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60083.log.gz 2023-10-30T14:53:17.357 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.58323.log 2023-10-30T14:53:17.358 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.49114.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49114.log.gz 2023-10-30T14:53:17.358 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48626.log 2023-10-30T14:53:17.360 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44718.log 2023-10-30T14:53:17.360 INFO:teuthology.orchestra.run.smithi057.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58323.log.gz 2023-10-30T14:53:17.361 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58478.log 2023-10-30T14:53:17.361 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48626.log.gz 2023-10-30T14:53:17.363 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56235.log 2023-10-30T14:53:17.363 INFO:teuthology.orchestra.run.smithi057.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44718.log.gz 2023-10-30T14:53:17.364 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52049.log 2023-10-30T14:53:17.364 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58478.log.gz 2023-10-30T14:53:17.366 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62488.log 2023-10-30T14:53:17.366 INFO:teuthology.orchestra.run.smithi057.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56235.log.gz 2023-10-30T14:53:17.367 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52049.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52049.log.gz 2023-10-30T14:53:17.368 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62488.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62488.log.gz 2023-10-30T14:53:17.369 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43848.log 2023-10-30T14:53:17.371 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52282.log 2023-10-30T14:53:17.372 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.43848.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.43848.log.gz 2023-10-30T14:53:17.373 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43511.log 2023-10-30T14:53:17.374 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54855.log 2023-10-30T14:53:17.374 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52282.log.gz 2023-10-30T14:53:17.375 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.43511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55253.log 2023-10-30T14:53:17.376 INFO:teuthology.orchestra.run.smithi057.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43511.log.gz 2023-10-30T14:53:17.377 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54855.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54855.log.gz -5 2023-10-30T14:53:17.377 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45730.log 2023-10-30T14:53:17.378 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49357.log 2023-10-30T14:53:17.379 INFO:teuthology.orchestra.run.smithi057.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55253.log.gz 2023-10-30T14:53:17.380 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45730.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45730.log.gz 2023-10-30T14:53:17.381 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.49357.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49357.log.gz 2023-10-30T14:53:17.382 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56365.log 2023-10-30T14:53:17.384 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62052.log 2023-10-30T14:53:17.385 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56365.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56365.log.gz 2023-10-30T14:53:17.385 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50536.log 2023-10-30T14:53:17.387 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62052.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62052.log.gz 2023-10-30T14:53:17.387 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61461.log 2023-10-30T14:53:17.388 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50536.log: gzip -5 89.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.50536.log.gz /var/log/ceph/ceph-client.admin.59766.log 2023-10-30T14:53:17.389 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-30T14:53:17.390 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59661.log 2023-10-30T14:53:17.390 INFO:teuthology.orchestra.run.smithi057.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61461.log.gz 2023-10-30T14:53:17.391 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59766.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.59766.log.gz 2023-10-30T14:53:17.391 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54300.log 2023-10-30T14:53:17.393 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62106.log 2023-10-30T14:53:17.393 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59661.log.gz 2023-10-30T14:53:17.394 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54300.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54300.log.gz 2023-10-30T14:53:17.394 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45595.log 2023-10-30T14:53:17.396 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62106.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62106.log.gz 2023-10-30T14:53:17.396 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54144.log 2023-10-30T14:53:17.397 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.0.45595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.45595.log.gz 2023-10-30T14:53:17.397 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50589.log 2023-10-30T14:53:17.398 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54144.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54144.log.gz 2023-10-30T14:53:17.399 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59529.log 2023-10-30T14:53:17.400 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50589.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50589.log.gz 2023-10-30T14:53:17.401 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62696.log 2023-10-30T14:53:17.402 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56392.log 2023-10-30T14:53:17.402 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59529.log.gz 2023-10-30T14:53:17.403 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62696.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62696.log.gz 2023-10-30T14:53:17.405 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56392.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56392.log.gz 2023-10-30T14:53:17.405 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55659.log 2023-10-30T14:53:17.407 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45245.log 2023-10-30T14:53:17.408 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55659.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55659.log.gz 2023-10-30T14:53:17.409 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57751.log 2023-10-30T14:53:17.410 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60789.log 2023-10-30T14:53:17.411 INFO:teuthology.orchestra.run.smithi057.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45245.log.gz 2023-10-30T14:53:17.411 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57751.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57751.log.gz 2023-10-30T14:53:17.412 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57777.log 2023-10-30T14:53:17.413 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44503.log 2023-10-30T14:53:17.414 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60789.log.gz 2023-10-30T14:53:17.415 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55766.log 2023-10-30T14:53:17.415 INFO:teuthology.orchestra.run.smithi057.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57777.log.gz 2023-10-30T14:53:17.416 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53319.log 2023-10-30T14:53:17.417 INFO:teuthology.orchestra.run.smithi057.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44503.log.gz 2023-10-30T14:53:17.418 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48263.log 2023-10-30T14:53:17.418 INFO:teuthology.orchestra.run.smithi057.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55766.log.gz 2023-10-30T14:53:17.419 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56726.log 2023-10-30T14:53:17.420 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53319.log.gz 2023-10-30T14:53:17.421 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48263.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48263.log.gz 2023-10-30T14:53:17.422 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56726.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56726.log.gz 2023-10-30T14:53:17.423 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50778.log 2023-10-30T14:53:17.424 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50641.log 2023-10-30T14:53:17.426 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60522.log 2023-10-30T14:53:17.426 INFO:teuthology.orchestra.run.smithi057.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50778.log.gz 2023-10-30T14:53:17.427 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59973.log 2023-10-30T14:53:17.427 INFO:teuthology.orchestra.run.smithi057.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50641.log.gz 2023-10-30T14:53:17.428 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60522.log: gzip -5 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60522.log.gz 2023-10-30T14:53:17.429 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.56568.log 2023-10-30T14:53:17.430 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59973.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59973.log.gz 2023-10-30T14:53:17.431 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57233.log 2023-10-30T14:53:17.431 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56568.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56568.log.gz 2023-10-30T14:53:17.432 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58793.log 2023-10-30T14:53:17.434 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57233.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57233.log.gz -5 2023-10-30T14:53:17.434 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.61734.log 2023-10-30T14:53:17.435 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50168.log 2023-10-30T14:53:17.435 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58793.log.gz 2023-10-30T14:53:17.437 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62254.log 2023-10-30T14:53:17.437 INFO:teuthology.orchestra.run.smithi057.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61734.log.gz 2023-10-30T14:53:17.438 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50247.log 2023-10-30T14:53:17.439 INFO:teuthology.orchestra.run.smithi057.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50168.log.gz 2023-10-30T14:53:17.440 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62254.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62254.log.gz 2023-10-30T14:53:17.440 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52703.log 2023-10-30T14:53:17.441 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50247.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50247.log.gz 2023-10-30T14:53:17.442 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60187.log 2023-10-30T14:53:17.443 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52703.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52703.log.gz 2023-10-30T14:53:17.444 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62228.log 2023-10-30T14:53:17.445 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60187.log: gzip -5 88.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.60187.log.gz /var/log/ceph/ceph.audit.log 2023-10-30T14:53:17.445 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-30T14:53:17.447 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52536.log 2023-10-30T14:53:17.447 INFO:teuthology.orchestra.run.smithi057.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62228.log.gz 2023-10-30T14:53:17.448 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-10-30T14:53:17.450 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52536.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52536.log.gz 2023-10-30T14:53:17.450 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-mds.c.log: 89.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T14:53:17.454 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58608.log 2023-10-30T14:53:17.456 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56067.log 2023-10-30T14:53:17.458 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58608.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58608.log.gz 2023-10-30T14:53:17.459 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56067.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56067.log.gz 2023-10-30T14:53:17.459 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53689.log 2023-10-30T14:53:17.461 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50906.log 2023-10-30T14:53:17.462 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53689.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53689.log.gz 2023-10-30T14:53:17.463 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50906.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50906.log.gz 2023-10-30T14:53:17.464 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61679.log 2023-10-30T14:53:17.466 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-10-30T14:53:17.467 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61679.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61679.log.gz 2023-10-30T14:53:17.470 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62514.log 2023-10-30T14:53:17.473 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62158.log 2023-10-30T14:53:17.474 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58034.log 2023-10-30T14:53:17.474 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62514.log.gz 2023-10-30T14:53:17.476 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47961.log 2023-10-30T14:53:17.476 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62158.log.gz 2023-10-30T14:53:17.482 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58034.log: gzip -5 88.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.58034.log.gz /var/log/ceph/ceph-client.admin.61004.log 2023-10-30T14:53:17.482 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-30T14:53:17.483 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61004.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61004.log.gz 2023-10-30T14:53:17.483 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.47961.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47961.log.gz 2023-10-30T14:53:17.483 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44302.log 2023-10-30T14:53:17.484 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44302.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44302.log.gz 2023-10-30T14:53:17.485 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48975.log 2023-10-30T14:53:17.488 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52076.log 2023-10-30T14:53:17.489 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48975.log.gz 2023-10-30T14:53:17.491 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44287.log 2023-10-30T14:53:17.493 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52076.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52076.log.gz 2023-10-30T14:53:17.496 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47575.log 2023-10-30T14:53:17.499 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.47575.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47575.log.gz 2023-10-30T14:53:17.500 INFO:teuthology.orchestra.run.smithi057.stderr: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50615.log 2023-10-30T14:53:17.503 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47601.log 2023-10-30T14:53:17.503 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44287.log.gz 2023-10-30T14:53:17.504 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50615.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50615.log.gz 2023-10-30T14:53:17.506 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.47601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54936.log 2023-10-30T14:53:17.509 INFO:teuthology.orchestra.run.smithi057.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49251.log 2023-10-30T14:53:17.510 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54936.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54936.log.gz 2023-10-30T14:53:17.510 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47601.log.gz 2023-10-30T14:53:17.513 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.49251.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49251.log.gz 2023-10-30T14:53:17.514 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61652.log 2023-10-30T14:53:17.517 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55812.log 2023-10-30T14:53:17.518 INFO:teuthology.orchestra.run.smithi057.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61652.log.gz 2023-10-30T14:53:17.519 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49415.log 2023-10-30T14:53:17.526 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55812.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55812.log.gz 2023-10-30T14:53:17.526 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.49415.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49415.log.gz 2023-10-30T14:53:17.526 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44510.log 2023-10-30T14:53:17.527 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44510.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44510.log.gz 2023-10-30T14:53:17.527 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54829.log 2023-10-30T14:53:17.530 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54829.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54829.log.gz 2023-10-30T14:53:17.530 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54458.log 2023-10-30T14:53:17.531 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50405.log 2023-10-30T14:53:17.532 INFO:teuthology.orchestra.run.smithi057.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54458.log.gz 2023-10-30T14:53:17.534 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44701.log 2023-10-30T14:53:17.540 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50405.log: /var/log/ceph/ceph-client.admin.44701.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50405.log.gz 2023-10-30T14:53:17.540 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44511.log 2023-10-30T14:53:17.541 INFO:teuthology.orchestra.run.smithi057.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44701.log.gz 2023-10-30T14:53:17.542 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56418.log 2023-10-30T14:53:17.546 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44511.log: /var/log/ceph/ceph-client.admin.56418.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56418.log.gz 2023-10-30T14:53:17.547 INFO:teuthology.orchestra.run.smithi057.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47774.log 2023-10-30T14:53:17.548 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44511.log.gz 2023-10-30T14:53:17.551 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.47774.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47774.log.gz 2023-10-30T14:53:17.551 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45481.log 2023-10-30T14:53:17.554 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45481.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45481.log.gz 2023-10-30T14:53:17.555 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55120.log 2023-10-30T14:53:17.558 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55120.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55120.log.gz 2023-10-30T14:53:17.558 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52677.log 2023-10-30T14:53:17.561 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52677.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52677.log.gz 2023-10-30T14:53:17.561 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45782.log 2023-10-30T14:53:17.563 INFO:teuthology.orchestra.run.smithi088.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-30T14:53:17.564 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45782.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45782.log.gz 2023-10-30T14:53:17.565 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55344.log 2023-10-30T14:53:17.568 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55344.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55344.log.gz 2023-10-30T14:53:17.568 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59396.log 2023-10-30T14:53:17.575 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59396.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59396.log.gz 2023-10-30T14:53:17.575 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57904.log 2023-10-30T14:53:17.575 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57904.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57904.log.gz 2023-10-30T14:53:17.576 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57565.log 2023-10-30T14:53:17.576 INFO:teuthology.orchestra.run.smithi088.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-30T14:53:17.578 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57565.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57565.log.gz 2023-10-30T14:53:17.579 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62748.log 2023-10-30T14:53:17.582 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62748.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62748.log.gz 2023-10-30T14:53:17.582 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59205.log 2023-10-30T14:53:17.585 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59205.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59205.log.gz 2023-10-30T14:53:17.586 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45704.log 2023-10-30T14:53:17.589 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45704.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45704.log.gz 2023-10-30T14:53:17.589 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43744.log 2023-10-30T14:53:17.592 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.43744.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43744.log.gz 2023-10-30T14:53:17.592 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47718.log 2023-10-30T14:53:17.595 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.47718.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47718.log.gz 2023-10-30T14:53:17.596 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51298.log 2023-10-30T14:53:17.599 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.51298.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51298.log.gz 2023-10-30T14:53:17.599 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58899.log 2023-10-30T14:53:17.602 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62833.log 2023-10-30T14:53:17.602 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58899.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58899.log.gz 2023-10-30T14:53:17.605 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62833.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62833.log.gz 2023-10-30T14:53:17.606 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62004.log 2023-10-30T14:53:17.609 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62004.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62004.log.gz 2023-10-30T14:53:17.609 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59004.log 2023-10-30T14:53:17.612 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59004.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59004.log.gz 2023-10-30T14:53:17.612 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48520.log 2023-10-30T14:53:17.615 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48520.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48520.log.gz 2023-10-30T14:53:17.616 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61866.log 2023-10-30T14:53:17.619 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61866.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61866.log.gz 2023-10-30T14:53:17.619 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52942.log 2023-10-30T14:53:17.622 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52942.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52942.log.gz 2023-10-30T14:53:17.623 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43982.log 2023-10-30T14:53:17.626 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.43982.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43982.log.gz 2023-10-30T14:53:17.626 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54744.log 2023-10-30T14:53:17.629 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54744.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54744.log.gz 2023-10-30T14:53:17.629 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54274.log 2023-10-30T14:53:17.632 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54274.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54274.log.gz 2023-10-30T14:53:17.633 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44509.log 2023-10-30T14:53:17.636 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44509.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44509.log.gz 2023-10-30T14:53:17.636 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38192.log 2023-10-30T14:53:17.639 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.38192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58166.log 2023-10-30T14:53:17.639 INFO:teuthology.orchestra.run.smithi057.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38192.log.gz 2023-10-30T14:53:17.641 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47935.log 2023-10-30T14:53:17.643 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58166.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58166.log.gz 2023-10-30T14:53:17.646 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.47935.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47935.log.gz 2023-10-30T14:53:17.646 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46042.log 2023-10-30T14:53:17.649 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46042.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46042.log.gz 2023-10-30T14:53:17.649 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45886.log 2023-10-30T14:53:17.652 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45886.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45886.log.gz 2023-10-30T14:53:17.653 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54513.log 2023-10-30T14:53:17.657 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54513.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54513.log.gz 2023-10-30T14:53:17.658 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63058.log 2023-10-30T14:53:17.661 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.63058.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63058.log.gz 2023-10-30T14:53:17.661 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57367.log 2023-10-30T14:53:17.664 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57367.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57367.log.gz 2023-10-30T14:53:17.665 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48367.log 2023-10-30T14:53:17.668 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48367.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48367.log.gz 2023-10-30T14:53:17.668 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52308.log 2023-10-30T14:53:17.672 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50509.log 2023-10-30T14:53:17.672 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52308.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52308.log.gz 2023-10-30T14:53:17.672 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46262.log 2023-10-30T14:53:17.673 INFO:teuthology.orchestra.run.smithi057.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50509.log.gz 2023-10-30T14:53:17.676 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46262.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46262.log.gz 2023-10-30T14:53:17.676 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54066.log 2023-10-30T14:53:17.679 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54066.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54066.log.gz 2023-10-30T14:53:17.679 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.33107.log 2023-10-30T14:53:17.682 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph.tmp-client.admin.33107.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.33107.log.gz 2023-10-30T14:53:17.682 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59337.log 2023-10-30T14:53:17.685 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59337.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59337.log.gz 2023-10-30T14:53:17.686 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60867.log 2023-10-30T14:53:17.689 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60867.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60867.log.gz 2023-10-30T14:53:17.689 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58270.log 2023-10-30T14:53:17.692 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58270.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58270.log.gz 2023-10-30T14:53:17.692 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59477.log 2023-10-30T14:53:17.695 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59477.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59477.log.gz 2023-10-30T14:53:17.696 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48598.log 2023-10-30T14:53:17.699 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48598.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48598.log.gz 2023-10-30T14:53:17.699 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61838.log 2023-10-30T14:53:17.703 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61838.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.61838.log.gz 2023-10-30T14:53:17.703 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51038.log 2023-10-30T14:53:17.705 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.51038.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51038.log.gz 2023-10-30T14:53:17.706 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47521.log 2023-10-30T14:53:17.709 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.47521.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47521.log.gz 2023-10-30T14:53:17.709 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45808.log 2023-10-30T14:53:17.712 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45808.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45808.log.gz 2023-10-30T14:53:17.712 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45299.log 2023-10-30T14:53:17.715 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43822.log 2023-10-30T14:53:17.715 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45299.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45299.log.gz 2023-10-30T14:53:17.718 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.43822.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43822.log.gz 2023-10-30T14:53:17.719 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46157.log 2023-10-30T14:53:17.722 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46157.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46157.log.gz 2023-10-30T14:53:17.722 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54884.log 2023-10-30T14:53:17.725 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54884.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54884.log.gz 2023-10-30T14:53:17.725 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57207.log 2023-10-30T14:53:17.727 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61140.log 2023-10-30T14:53:17.730 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57207.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57207.log.gz 2023-10-30T14:53:17.731 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51142.log 2023-10-30T14:53:17.731 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61140.log.gz 2023-10-30T14:53:17.734 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.51142.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51142.log.gz 2023-10-30T14:53:17.735 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60893.log 2023-10-30T14:53:17.737 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53585.log 2023-10-30T14:53:17.738 INFO:teuthology.orchestra.run.smithi057.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60893.log.gz 2023-10-30T14:53:17.740 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49027.log 2023-10-30T14:53:17.740 INFO:teuthology.orchestra.run.smithi057.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53585.log.gz 2023-10-30T14:53:17.743 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.49027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58819.log 2023-10-30T14:53:17.744 INFO:teuthology.orchestra.run.smithi057.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49027.log.gz 2023-10-30T14:53:17.747 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44959.log 2023-10-30T14:53:17.748 INFO:teuthology.orchestra.run.smithi057.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58819.log.gz 2023-10-30T14:53:17.750 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44691.log 2023-10-30T14:53:17.751 INFO:teuthology.orchestra.run.smithi057.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44959.log.gz 2023-10-30T14:53:17.757 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52651.log 2023-10-30T14:53:17.757 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52651.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52651.log.gz 2023-10-30T14:53:17.758 INFO:teuthology.orchestra.run.smithi057.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44691.log.gz 2023-10-30T14:53:17.758 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61114.log 2023-10-30T14:53:17.762 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61114.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61114.log.gz 2023-10-30T14:53:17.762 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60710.log 2023-10-30T14:53:17.765 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60710.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60710.log.gz 2023-10-30T14:53:17.766 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46288.log 2023-10-30T14:53:17.768 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.46288.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46288.log.gz 2023-10-30T14:53:17.769 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53637.log 2023-10-30T14:53:17.772 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53637.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53637.log.gz 2023-10-30T14:53:17.772 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53960.log 2023-10-30T14:53:17.775 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53960.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53960.log.gz 2023-10-30T14:53:17.775 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55579.log 2023-10-30T14:53:17.778 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55579.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55579.log.gz 2023-10-30T14:53:17.779 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53826.log 2023-10-30T14:53:17.782 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53826.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53826.log.gz 2023-10-30T14:53:17.782 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56126.log 2023-10-30T14:53:17.787 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60841.log 2023-10-30T14:53:17.787 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.56126.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56126.log.gz 2023-10-30T14:53:17.787 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51892.log 2023-10-30T14:53:17.788 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60841.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60841.log.gz 2023-10-30T14:53:17.791 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.51892.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51090.log 2023-10-30T14:53:17.791 INFO:teuthology.orchestra.run.smithi057.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51892.log.gz 2023-10-30T14:53:17.794 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.51090.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51090.log.gz 2023-10-30T14:53:17.795 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60763.log 2023-10-30T14:53:17.797 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61894.log 2023-10-30T14:53:17.798 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60763.log.gz 2023-10-30T14:53:17.801 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60662.log 2023-10-30T14:53:17.801 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50273.log 2023-10-30T14:53:17.803 INFO:teuthology.orchestra.run.smithi057.stderr: 89.4%/var/log/ceph/ceph-client.admin.60662.log: -- replaced with /var/log/ceph/ceph-client.admin.61894.log.gz 2023-10-30T14:53:17.810 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55226.log 2023-10-30T14:53:17.810 INFO:teuthology.orchestra.run.smithi057.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60662.log.gz 2023-10-30T14:53:17.811 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55226.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55226.log.gz 2023-10-30T14:53:17.811 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53611.log 2023-10-30T14:53:17.812 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.53611.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45426.log 2023-10-30T14:53:17.813 INFO:teuthology.orchestra.run.smithi057.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50273.log.gz 2023-10-30T14:53:17.813 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53611.log.gz 2023-10-30T14:53:17.814 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58218.log 2023-10-30T14:53:17.815 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51944.log 2023-10-30T14:53:17.816 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.58218.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.45426.log.gz 2023-10-30T14:53:17.819 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.51944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48493.log 2023-10-30T14:53:17.821 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51944.log.gz 2023-10-30T14:53:17.822 INFO:teuthology.orchestra.run.smithi057.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58218.log.gz 2023-10-30T14:53:17.823 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60607.log 2023-10-30T14:53:17.828 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60607.log: /var/log/ceph/ceph-client.admin.48493.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52197.log 2023-10-30T14:53:17.828 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48493.log.gz 2023-10-30T14:53:17.829 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60607.log.gz 2023-10-30T14:53:17.832 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.52197.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.57012.log 2023-10-30T14:53:17.835 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.57012.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52197.log.gz 2023-10-30T14:53:17.836 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50221.log 2023-10-30T14:53:17.836 INFO:teuthology.orchestra.run.smithi057.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57012.log.gz 2023-10-30T14:53:17.839 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.50221.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43796.log 2023-10-30T14:53:17.842 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.43796.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43796.log.gz 2023-10-30T14:53:17.844 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59947.log 2023-10-30T14:53:17.844 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50221.log.gz 2023-10-30T14:53:17.850 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48753.log 2023-10-30T14:53:17.850 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48753.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48753.log.gz 2023-10-30T14:53:17.851 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44497.log 2023-10-30T14:53:17.852 INFO:teuthology.orchestra.run.smithi057.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59947.log.gz 2023-10-30T14:53:17.855 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44497.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44497.log.gz 2023-10-30T14:53:17.856 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59869.log 2023-10-30T14:53:17.859 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59869.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59869.log.gz 2023-10-30T14:53:17.859 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47627.log 2023-10-30T14:53:17.862 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.47627.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47627.log.gz 2023-10-30T14:53:17.862 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60580.log 2023-10-30T14:53:17.865 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60580.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60580.log.gz 2023-10-30T14:53:17.866 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2023-10-30T14:53:17.871 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44008.log 2023-10-30T14:53:17.876 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59503.log 2023-10-30T14:53:17.877 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.44008.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44008.log.gz 2023-10-30T14:53:17.883 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62462.log 2023-10-30T14:53:17.887 INFO:teuthology.orchestra.run.smithi057.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59503.log.gz 2023-10-30T14:53:17.888 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59370.log 2023-10-30T14:53:17.889 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62462.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62462.log.gz 2023-10-30T14:53:17.890 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59370.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59370.log.gz 2023-10-30T14:53:17.891 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61059.log 2023-10-30T14:53:17.897 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.61059.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61059.log.gz 2023-10-30T14:53:17.898 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52136.log 2023-10-30T14:53:17.900 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59311.log 2023-10-30T14:53:17.901 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60054.log 2023-10-30T14:53:17.903 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.59311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62883.log 2023-10-30T14:53:17.903 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59311.log.gz 2023-10-30T14:53:17.904 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60054.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52136.log.gz 2023-10-30T14:53:17.904 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60054.log.gz 2023-10-30T14:53:17.907 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55318.log 2023-10-30T14:53:17.909 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62883.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62883.log.gz 2023-10-30T14:53:17.913 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55318.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55318.log.gz 2023-10-30T14:53:17.913 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45063.log 2023-10-30T14:53:17.919 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55937.log 2023-10-30T14:53:17.920 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45063.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45063.log.gz 2023-10-30T14:53:17.925 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55937.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55937.log.gz 2023-10-30T14:53:17.926 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45376.log 2023-10-30T14:53:17.931 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49088.log 2023-10-30T14:53:17.933 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45376.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.45376.log.gz 2023-10-30T14:53:17.935 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54910.log 2023-10-30T14:53:17.937 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.49088.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49088.log.gz 2023-10-30T14:53:17.939 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52162.log 2023-10-30T14:53:17.944 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54910.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54910.log.gz 2023-10-30T14:53:17.945 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55475.log 2023-10-30T14:53:17.945 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52162.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52162.log.gz 2023-10-30T14:53:17.947 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50880.log 2023-10-30T14:53:17.949 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55475.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55475.log.gz 2023-10-30T14:53:17.956 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.50880.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50880.log.gz 2023-10-30T14:53:17.957 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43604.log 2023-10-30T14:53:17.959 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54352.log 2023-10-30T14:53:17.961 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.43604.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43604.log.gz 2023-10-30T14:53:17.967 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45990.log 2023-10-30T14:53:17.969 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54352.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54352.log.gz 2023-10-30T14:53:17.971 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48871.log 2023-10-30T14:53:17.973 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45990.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45990.log.gz 2023-10-30T14:53:17.977 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48871.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48871.log.gz 2023-10-30T14:53:17.978 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52458.log 2023-10-30T14:53:17.983 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52812.log 2023-10-30T14:53:17.985 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52458.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52458.log.gz 2023-10-30T14:53:17.987 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62332.log 2023-10-30T14:53:17.989 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52812.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52812.log.gz 2023-10-30T14:53:17.991 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52484.log 2023-10-30T14:53:17.996 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.62332.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62332.log.gz 2023-10-30T14:53:17.997 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.52484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45167.log 2023-10-30T14:53:17.997 INFO:teuthology.orchestra.run.smithi057.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52484.log.gz 2023-10-30T14:53:17.999 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48707.log 2023-10-30T14:53:18.001 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.45167.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45167.log.gz 2023-10-30T14:53:18.003 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54962.log 2023-10-30T14:53:18.005 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.48707.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48707.log.gz 2023-10-30T14:53:18.009 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.54962.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54962.log.gz 2023-10-30T14:53:18.010 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55292.log 2023-10-30T14:53:18.015 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60213.log 2023-10-30T14:53:18.017 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.55292.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55292.log.gz 2023-10-30T14:53:18.021 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.60213.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60213.log.gz 2023-10-30T14:53:19.035 INFO:teuthology.orchestra.run.smithi057.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-30T14:53:23.797 INFO:teuthology.orchestra.run.smithi057.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-30T14:53:24.454 INFO:teuthology.orchestra.run.smithi088.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-30T14:53:24.815 INFO:teuthology.orchestra.run.smithi088.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-30T14:53:24.825 INFO:teuthology.orchestra.run.smithi088.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-30T14:53:24.868 INFO:teuthology.orchestra.run.smithi088.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-30T14:53:25.452 INFO:teuthology.orchestra.run.smithi057.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-30T14:53:26.778 INFO:teuthology.orchestra.run.smithi057.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-30T14:53:27.143 INFO:teuthology.orchestra.run.smithi057.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-30T14:53:40.358 INFO:teuthology.orchestra.run.smithi088.stderr: 94.1% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2023-10-30T14:53:40.359 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-10-30T14:53:40.359 INFO:teuthology.orchestra.run.smithi088.stderr:real 0m23.579s 2023-10-30T14:53:40.359 INFO:teuthology.orchestra.run.smithi088.stderr:user 0m55.391s 2023-10-30T14:53:40.360 INFO:teuthology.orchestra.run.smithi088.stderr:sys 0m1.907s 2023-10-30T14:54:01.853 INFO:teuthology.orchestra.run.smithi057.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-10-30T14:54:06.136 INFO:teuthology.orchestra.run.smithi057.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2023-10-30T14:54:12.564 INFO:teuthology.orchestra.run.smithi057.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-10-30T14:54:12.566 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-30T14:54:12.566 INFO:teuthology.orchestra.run.smithi057.stderr:real 0m55.793s 2023-10-30T14:54:12.567 INFO:teuthology.orchestra.run.smithi057.stderr:user 2m53.143s 2023-10-30T14:54:12.567 INFO:teuthology.orchestra.run.smithi057.stderr:sys 0m13.588s 2023-10-30T14:54:12.568 INFO:tasks.ceph:Archiving logs... 2023-10-30T14:54:12.569 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/log/ceph to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441053/remote/smithi057/log 2023-10-30T14:54:12.570 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T14:54:31.642 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/log/ceph to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441053/remote/smithi088/log 2023-10-30T14:54:31.651 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T14:54:37.692 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-30T14:54:37.704 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-30T14:54:37.705 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T14:54:37.709 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T14:54:37.775 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-30T14:54:37.785 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-30T14:54:37.785 DEBUG:teuthology.orchestra.run.smithi057:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T14:54:37.790 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T14:54:37.802 INFO:teuthology.orchestra.run.smithi088.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T14:54:37.803 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================== 2023-10-30T14:54:37.803 INFO:teuthology.orchestra.run.smithi088.stdout:*hv01.front.sepi 44.4.53.2 2 u 46 64 377 0.092 -2.762 0.312 2023-10-30T14:54:37.803 INFO:teuthology.orchestra.run.smithi088.stdout:+hv02.front.sepi 63.231.80.2 3 u 35 64 377 0.092 -1.253 0.351 2023-10-30T14:54:37.803 INFO:teuthology.orchestra.run.smithi088.stdout:+hv03.front.sepi 74.6.168.72 3 u 40 64 377 0.094 -1.471 0.271 2023-10-30T14:54:37.804 INFO:teuthology.orchestra.run.smithi088.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-30T14:54:37.805 INFO:teuthology.orchestra.run.smithi057.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T14:54:37.805 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================== 2023-10-30T14:54:37.806 INFO:teuthology.orchestra.run.smithi057.stdout:*hv01.front.sepi 44.4.53.2 2 u 32 128 377 0.120 -3.206 0.759 2023-10-30T14:54:37.806 INFO:teuthology.orchestra.run.smithi057.stdout:+hv02.front.sepi 63.231.80.2 3 u 46 128 377 0.102 -1.552 0.654 2023-10-30T14:54:37.806 INFO:teuthology.orchestra.run.smithi057.stdout:+hv03.front.sepi 74.6.168.72 3 u 42 128 377 0.112 -2.435 1.175 2023-10-30T14:54:37.807 INFO:teuthology.orchestra.run.smithi057.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-30T14:54:37.807 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-30T14:54:37.816 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-30T14:54:37.817 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-30T14:54:37.828 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-30T14:54:37.837 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-30T14:54:37.846 INFO:teuthology.task.internal:Duration was 3189.853665 seconds 2023-10-30T14:54:37.847 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-30T14:54:37.855 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-30T14:54:37.856 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T14:54:37.860 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T14:54:37.956 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-30T14:54:37.957 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi057.front.sepia.ceph.com 2023-10-30T14:54:37.957 DEBUG:teuthology.orchestra.run.smithi057:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T14:54:37.980 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2023-10-30T14:54:37.981 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T14:54:37.990 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-30T14:54:37.991 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T14:54:38.029 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T14:54:38.067 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-30T14:54:38.067 DEBUG:teuthology.orchestra.run.smithi057:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T14:54:38.118 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T14:54:38.203 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-30T14:54:38.214 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-30T14:54:38.215 DEBUG:teuthology.orchestra.run.smithi057:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T14:54:38.236 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T14:54:38.249 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-30T14:54:38.259 DEBUG:teuthology.orchestra.run.smithi057:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T14:54:38.281 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T14:54:38.297 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = core 2023-10-30T14:54:38.303 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2023-10-30T14:54:38.329 DEBUG:teuthology.orchestra.run.smithi057:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T14:54:38.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T14:54:38.366 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T14:54:38.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T14:54:38.377 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-30T14:54:38.388 INFO:teuthology.task.internal:Transferring archived files... 2023-10-30T14:54:38.389 DEBUG:teuthology.misc:Transferring archived files from smithi057:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441053/remote/smithi057 2023-10-30T14:54:38.390 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T14:54:38.433 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441053/remote/smithi088 2023-10-30T14:54:38.433 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T14:54:38.451 INFO:teuthology.task.internal:Removing archive directory... 2023-10-30T14:54:38.452 DEBUG:teuthology.orchestra.run.smithi057:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T14:54:38.473 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T14:54:38.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-30T14:54:38.508 INFO:teuthology.task.internal:Not uploading archives. 2023-10-30T14:54:38.509 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-30T14:54:38.517 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-30T14:54:38.518 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T14:54:38.525 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T14:54:38.531 INFO:teuthology.orchestra.run.smithi057.stdout: 657970 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 14:54 /home/ubuntu/cephtest 2023-10-30T14:54:38.545 INFO:teuthology.orchestra.run.smithi088.stdout: 524292 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 14:54 /home/ubuntu/cephtest 2023-10-30T14:54:38.546 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-30T14:54:38.556 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-30T14:54:38.575 INFO:teuthology.nuke:Checking targets against current locks 2023-10-30T14:54:38.599 DEBUG:teuthology.nuke:shortname: smithi057 2023-10-30T14:54:38.600 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T14:54:38.625 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441053', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 13:50:53.741350', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIMwmje95SbNyfN/QhIqW+xTGxgbFqqhqVuHX1ojoZwDT7O/oIBYa4ThMFPh2Waxbow/mVMbY8cM9aUazaL1ENE='} 2023-10-30T14:54:38.628 DEBUG:teuthology.nuke:shortname: smithi088 2023-10-30T14:54:38.629 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T14:54:38.654 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441053', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 13:50:53.739374', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPPr4j6c4Elj4JMggLxzU+oXj+Zbox5W/SOrDnjdPhsyNdfo/+FImaEbo+jin9aqDlqe5Rnspl/1O6/Th177wnE='} 2023-10-30T14:54:38.719 INFO:teuthology.orchestra.console.smithi057:Power off 2023-10-30T14:54:38.720 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T14:54:38.772 INFO:teuthology.orchestra.console.smithi088:Power off 2023-10-30T14:54:38.773 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T14:54:38.805 DEBUG:teuthology.orchestra.console.smithi057:power off output: Chassis Power Control: Down/Off 2023-10-30T14:54:38.806 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T14:54:38.815 DEBUG:teuthology.orchestra.console.smithi088:power off output: Chassis Power Control: Down/Off 2023-10-30T14:54:38.816 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T14:54:38.829 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2023-10-30T14:54:38.838 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2023-10-30T14:54:42.831 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T14:54:42.840 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T14:54:42.942 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2023-10-30T14:54:42.950 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2023-10-30T14:54:46.946 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T14:54:46.955 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T14:54:47.058 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is off 2023-10-30T14:54:47.065 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off 2023-10-30T14:54:47.160 INFO:teuthology.orchestra.console.smithi057:Power off completed 2023-10-30T14:54:47.167 INFO:teuthology.orchestra.console.smithi088:Power off completed 2023-10-30T14:54:47.274 INFO:teuthology.run:Summary data: client.0-kernel-sha1: eacf00cc51e204f401b4d5d4f84825a68f373c74 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} duration: 3189.8536648750305 failure_reason: 'Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi057 with status 128: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh''' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=df3faa8d3a3b47ceb97651a5cf1753a5 status: fail success: false 2023-10-30T14:54:47.274 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T14:54:47.344 INFO:teuthology.run:FAIL